2024-03-29T09:12:45.345 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T09:12:45.345 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_vshankar@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} --name vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi --verbose --archive /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195 -- /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/orig.config.yaml 2024-03-29T09:12:45.393 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T09:12:45.483 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195 branch: wip-vshankar-testing-20240326.105515-reef description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} email: vshankar@redhat.com first_in_suite: false job_id: '7628195' kernel: client: branch: testing flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-vshankar-testing-20240326.105515-reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: d0e392d9aee635f2b65768c460c7a151ead7a5e2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d0e392d9aee635f2b65768c460c7a151ead7a5e2 kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240326.105515-reef sha1: d0e392d9aee635f2b65768c460c7a151ead7a5e2 owner: scheduled_vshankar@teuthology priority: 120 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 7327 sha1: d0e392d9aee635f2b65768c460c7a151ead7a5e2 sleep_before_teardown: 0 subset: 990/1024 suite: fs suite_branch: wip-vshankar-testing-20240326.105515-reef suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d0e392d9aee635f2b65768c460c7a151ead7a5e2 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIF8rEDuGqw7u1MAn7aIckiCIMAIgcK8ExXmWCwNekr63BcoLpA25vbhmE2qT/oUqU/zqamOdxeKlOruHPG4l8s= smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFpoTDgkUa4ldKvUIt0bpX7pXwcd199Mcl3Y1HX+RceoaNKFSo7mIUmoYKdUUWIiRlhtPgolkPbAw/7/+jHMc64= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-28_11:21:59 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T09:12:45.483 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa; will attempt to use it 2024-03-29T09:12:45.484 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa/tasks 2024-03-29T09:12:45.484 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T09:12:45.485 INFO:teuthology.task.internal:Checking packages... 2024-03-29T09:12:45.509 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd0e392d9aee635f2b65768c460c7a151ead7a5e2' 2024-03-29T09:12:45.509 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T09:12:45.509 INFO:teuthology.packaging:ref: None 2024-03-29T09:12:45.509 INFO:teuthology.packaging:tag: None 2024-03-29T09:12:45.509 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240326.105515-reef 2024-03-29T09:12:45.509 INFO:teuthology.packaging:sha1: d0e392d9aee635f2b65768c460c7a151ead7a5e2 2024-03-29T09:12:45.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-20240326.105515-reef 2024-03-29T09:12:45.725 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1015.gd0e392d9 2024-03-29T09:12:45.726 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T09:12:45.733 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T09:12:45.733 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T09:12:45.741 INFO:teuthology.task.internal:Saving configuration 2024-03-29T09:12:45.758 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T09:12:45.770 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T09:12:45.794 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 09:07:13.369330', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIF8rEDuGqw7u1MAn7aIckiCIMAIgcK8ExXmWCwNekr63BcoLpA25vbhmE2qT/oUqU/zqamOdxeKlOruHPG4l8s='} 2024-03-29T09:12:45.814 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 09:07:13.367518', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFpoTDgkUa4ldKvUIt0bpX7pXwcd199Mcl3Y1HX+RceoaNKFSo7mIUmoYKdUUWIiRlhtPgolkPbAw/7/+jHMc64='} 2024-03-29T09:12:45.814 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T09:12:45.821 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-29T09:12:45.821 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-29T09:12:45.821 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T09:12:45.889 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5eda77d0d0>, signals=[15]) 2024-03-29T09:12:45.890 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T09:12:45.896 INFO:teuthology.task.internal:Opening connections... 2024-03-29T09:12:45.896 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T09:12:45.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:12:45.973 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T09:12:45.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:12:46.044 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T09:12:46.050 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-03-29T09:12:46.076 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-03-29T09:12:46.076 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-03-29T09:12:46.139 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2024-03-29T09:12:46.139 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="8" 2024-03-29T09:12:46.139 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2024-03-29T09:12:46.139 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2024-03-29T09:12:46.139 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="8" 2024-03-29T09:12:46.139 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el8" 2024-03-29T09:12:46.139 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-29T09:12:46.139 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2024-03-29T09:12:46.139 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-29T09:12:46.139 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2024-03-29T09:12:46.140 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T09:12:46.140 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-29T09:12:46.140 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T09:12:46.140 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-03-29T09:12:46.164 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2024-03-29T09:12:46.186 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2024-03-29T09:12:46.186 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2024-03-29T09:12:46.246 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream" 2024-03-29T09:12:46.246 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="8" 2024-03-29T09:12:46.246 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos" 2024-03-29T09:12:46.246 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora" 2024-03-29T09:12:46.246 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="8" 2024-03-29T09:12:46.246 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el8" 2024-03-29T09:12:46.247 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-29T09:12:46.247 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2024-03-29T09:12:46.247 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-29T09:12:46.247 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/" 2024-03-29T09:12:46.247 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T09:12:46.247 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-29T09:12:46.247 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T09:12:46.247 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2024-03-29T09:12:46.272 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T09:12:46.281 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T09:12:46.286 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T09:12:46.287 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-03-29T09:12:46.289 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2024-03-29T09:12:46.309 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T09:12:46.314 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T09:12:46.314 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-03-29T09:12:46.352 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2024-03-29T09:12:46.380 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T09:12:46.434 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T09:12:46.447 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-29T09:12:46.448 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-03-29T09:12:46.448 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-03-29T09:12:46.448 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T09:12:46.448 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T09:12:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T09:12:46.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2024-03-29T09:12:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T09:12:46.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2024-03-29T09:12:46.783 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-29T09:12:46.783 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-29T09:12:46.784 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-03-29T09:12:46.794 DEBUG:teuthology.task.kernel:sha1 for client.1 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-29T09:12:46.794 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-29T09:12:46.794 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-03-29T09:12:46.805 INFO:teuthology.orchestra.run.smithi012.stdout:4.18.0-489.el8.x86_64 2024-03-29T09:12:46.806 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-29T09:12:46.806 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-29T09:12:46.806 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum upgrade -y linux-firmware 2024-03-29T09:12:46.815 INFO:teuthology.orchestra.run.smithi159.stdout:4.18.0-489.el8.x86_64 2024-03-29T09:12:46.816 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-29T09:12:46.816 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-29T09:12:46.816 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum upgrade -y linux-firmware 2024-03-29T09:21:04.152 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - AppStream 58 kB/s | 28 MB 08:14 2024-03-29T09:21:04.324 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 8 - AppStream 58 kB/s | 28 MB 08:14 2024-03-29T09:21:10.460 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - BaseOS 7.7 MB/s | 10 MB 00:01 2024-03-29T09:21:12.436 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - Extras 51 kB/s | 18 kB 00:00 2024-03-29T09:21:12.715 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 8 - BaseOS 3.3 MB/s | 10 MB 00:03 2024-03-29T09:21:12.728 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - Extras common packages 40 kB/s | 7.5 kB 00:00 2024-03-29T09:21:12.996 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-03-29T09:21:14.241 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-03-29T09:21:14.657 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 8 - Extras 76 kB/s | 18 kB 00:00 2024-03-29T09:21:15.057 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 7.5 kB 00:00 2024-03-29T09:21:15.326 INFO:teuthology.orchestra.run.smithi012.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-03-29T09:21:16.118 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2024-03-29T09:21:18.013 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 411 kB/s | 24 kB 00:00 2024-03-29T09:21:20.110 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 550 kB/s | 24 kB 00:00 2024-03-29T09:21:21.135 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-29T09:21:21.135 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-29T09:21:21.135 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-29T09:21:21.135 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-29T09:21:21.135 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2024-03-29T09:21:21.136 INFO:teuthology.orchestra.run.smithi159.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 baseos 355 M 2024-03-29T09:21:21.136 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:21:21.136 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-29T09:21:21.136 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-29T09:21:21.136 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 1 Package 2024-03-29T09:21:21.136 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:21:21.136 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 355 M 2024-03-29T09:21:21.136 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-29T09:21:23.421 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-29T09:21:23.421 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-29T09:21:23.421 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2024-03-29T09:21:23.421 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-29T09:21:23.422 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-03-29T09:21:23.422 INFO:teuthology.orchestra.run.smithi012.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 baseos 355 M 2024-03-29T09:21:23.422 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:21:23.422 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-29T09:21:23.422 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-29T09:21:23.422 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 1 Package 2024-03-29T09:21:23.422 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:21:23.422 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 355 M 2024-03-29T09:21:23.423 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-29T09:21:36.631 INFO:teuthology.orchestra.run.smithi159.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 23 MB/s | 355 MB 00:15 2024-03-29T09:21:36.631 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-29T09:21:36.631 INFO:teuthology.orchestra.run.smithi159.stdout:Total 23 MB/s | 355 MB 00:15 2024-03-29T09:21:38.225 INFO:teuthology.orchestra.run.smithi012.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 24 MB/s | 355 MB 00:14 2024-03-29T09:21:38.225 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-29T09:21:38.225 INFO:teuthology.orchestra.run.smithi012.stdout:Total 24 MB/s | 355 MB 00:14 2024-03-29T09:21:38.730 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-29T09:21:38.796 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-29T09:21:38.796 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-29T09:21:40.448 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-29T09:21:40.495 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-29T09:21:40.500 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-29T09:21:40.966 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-29T09:21:40.990 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-29T09:21:42.784 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-29T09:21:42.807 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-29T09:22:03.863 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-29T09:22:04.235 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-29T09:22:04.464 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-29T09:22:04.707 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-29T09:22:04.707 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-29T09:22:05.203 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-29T09:22:05.203 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:22:05.203 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2024-03-29T09:22:05.203 INFO:teuthology.orchestra.run.smithi159.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-03-29T09:22:05.203 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:22:05.204 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-29T09:22:05.382 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.1... 2024-03-29T09:22:05.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-29T09:22:05.605 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ 2024-03-29T09:22:05.606 DEBUG:teuthology.orchestra.run.smithi159:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ --input-file=- 2024-03-29T09:22:05.608 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi159.front.sepia.ceph.com to complete... 2024-03-29T09:22:07.172 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-29T09:22:07.543 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-29T09:22:07.755 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-29T09:22:08.010 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-29T09:22:08.010 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-29T09:22:08.515 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-29T09:22:08.515 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:22:08.515 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-03-29T09:22:08.515 INFO:teuthology.orchestra.run.smithi012.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-03-29T09:22:08.515 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:22:08.516 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-29T09:22:08.731 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-29T09:22:08.731 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-29T09:22:08.945 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ 2024-03-29T09:22:08.945 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ --input-file=- 2024-03-29T09:22:08.947 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi012.front.sepia.ceph.com to complete... 2024-03-29T09:22:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29 09:22:26 URL:https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [401601944/401601944] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-29T09:22:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:FINISHED --2024-03-29 09:22:26-- 2024-03-29T09:22:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:Total wall clock time: 18s 2024-03-29T09:22:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:Downloaded: 1 files, 383M in 17s (22.3 MB/s) 2024-03-29T09:22:26.480 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-29T09:22:26.480 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-29T09:22:28.325 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying... ######################################## 2024-03-29T09:22:28.371 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing... ######################################## 2024-03-29T09:22:28.487 INFO:teuthology.orchestra.run.smithi012.stdout:Updating / installing... 2024-03-29T09:23:04.569 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29 09:23:04 URL:https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [401601944/401601944] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-29T09:23:04.569 INFO:teuthology.orchestra.run.smithi159.stderr:FINISHED --2024-03-29 09:23:04-- 2024-03-29T09:23:04.569 INFO:teuthology.orchestra.run.smithi159.stderr:Total wall clock time: 59s 2024-03-29T09:23:04.569 INFO:teuthology.orchestra.run.smithi159.stderr:Downloaded: 1 files, 383M in 59s (6.54 MB/s) 2024-03-29T09:23:04.573 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.1... 2024-03-29T09:23:04.573 DEBUG:teuthology.orchestra.run.smithi159:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-29T09:23:06.317 INFO:teuthology.orchestra.run.smithi159.stdout:Verifying... ######################################## 2024-03-29T09:23:06.360 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing... ######################################## 2024-03-29T09:23:06.470 INFO:teuthology.orchestra.run.smithi159.stdout:Updating / installing... 2024-03-29T09:23:11.514 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-29T09:23:15.989 INFO:teuthology.orchestra.run.smithi012.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T09:23:36.546 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-29T09:23:36.546 DEBUG:teuthology.orchestra.run.smithi012:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-29T09:23:36.607 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-29T09:23:36.607 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-29T09:23:36.607 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-29T09:23:36.607 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-29T09:23:36.607 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-29T09:23:36.607 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-29T09:23:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-29T09:23:36.609 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-29T09:23:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-29T09:23:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-29T09:23:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-29T09:23:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-29T09:23:36.614 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-29T09:23:36.615 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-29T09:23:36.616 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-29T09:23:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-29T09:23:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-29T09:23:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-29T09:23:36.620 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-29T09:23:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-29T09:23:36.622 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-29T09:23:36.623 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-29T09:23:36.624 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-29T09:23:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-29T09:23:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-29T09:23:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-29T09:23:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-29T09:23:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-29T09:23:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-29T09:23:36.631 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-29T09:23:36.632 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-29T09:23:36.633 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-29T09:23:36.634 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-29T09:23:36.635 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-29T09:23:36.636 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-29T09:23:36.637 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-29T09:23:36.638 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-29T09:23:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-29T09:23:36.640 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-29T09:23:36.641 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-29T09:23:36.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-29T09:23:36.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-29T09:23:36.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-29T09:23:36.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-29T09:23:36.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-29T09:23:36.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-29T09:23:36.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-29T09:23:36.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-29T09:23:36.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-29T09:23:36.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-29T09:23:36.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-29T09:23:36.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-29T09:23:36.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-29T09:23:36.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-29T09:23:36.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-29T09:23:36.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-29T09:23:36.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-29T09:23:36.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-29T09:23:36.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-29T09:23:36.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-29T09:23:36.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-29T09:23:36.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-29T09:23:36.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-29T09:23:36.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-29T09:23:36.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-29T09:23:36.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-29T09:23:36.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-29T09:23:36.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-29T09:23:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-29T09:23:36.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-29T09:23:36.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-29T09:23:36.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-29T09:23:36.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-29T09:23:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-29T09:23:36.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-29T09:23:36.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-29T09:23:36.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-29T09:23:36.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-29T09:23:36.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-29T09:23:36.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-29T09:23:36.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-29T09:23:36.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-29T09:23:36.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-29T09:23:36.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-29T09:23:36.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-29T09:23:36.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-29T09:23:36.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-29T09:23:36.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-29T09:23:36.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-29T09:23:36.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-29T09:23:36.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-29T09:23:36.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-29T09:23:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-29T09:23:36.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-29T09:23:36.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-29T09:23:36.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-29T09:23:36.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-29T09:23:36.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-29T09:23:36.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-29T09:23:36.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-29T09:23:36.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-29T09:23:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-29T09:23:36.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-29T09:23:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-29T09:23:36.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-29T09:23:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-29T09:23:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-29T09:23:36.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-29T09:23:36.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-29T09:23:36.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-29T09:23:36.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-29T09:23:36.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-29T09:23:36.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-29T09:23:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-29T09:23:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-29T09:23:36.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-29T09:23:36.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-29T09:23:36.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-29T09:23:36.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-29T09:23:36.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-29T09:23:36.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-29T09:23:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-29T09:23:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-29T09:23:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-29T09:23:36.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-29T09:23:36.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-29T09:23:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-29T09:23:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-29T09:23:36.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-29T09:23:36.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-29T09:23:36.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-29T09:23:36.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-29T09:23:36.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-29T09:23:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-29T09:23:36.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-29T09:23:36.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-29T09:23:36.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-29T09:23:36.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-29T09:23:36.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-29T09:23:36.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-29T09:23:36.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-29T09:23:36.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-29T09:23:36.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-29T09:23:36.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-29T09:23:36.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-29T09:23:36.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-29T09:23:36.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-29T09:23:36.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-29T09:23:36.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-29T09:23:36.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-29T09:23:36.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-29T09:23:36.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-29T09:23:36.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-29T09:23:36.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-29T09:23:36.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-29T09:23:36.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-29T09:23:36.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-29T09:23:36.749 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-29T09:23:36.749 DEBUG:teuthology.orchestra.run.smithi012:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-29T09:23:36.777 INFO:teuthology.orchestra.run.smithi012.stdout:postinstall scriptlet (using /bin/sh): 2024-03-29T09:23:36.777 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-29T09:23:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-29T09:23:36.777 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-03-29T09:23:36.777 INFO:teuthology.orchestra.run.smithi012.stdout:for file in vmlinuz System.map config; do 2024-03-29T09:23:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-29T09:23:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-29T09:23:36.777 INFO:teuthology.orchestra.run.smithi012.stdout: fi 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:done 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-29T09:23:36.778 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-03-29T09:23:36.779 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2024-03-29T09:23:36.859 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2024-03-29T09:23:36.859 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.02 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 129.el8 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Group : System Environment/Base 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 9265586 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Relocations : (not relocatable) 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : CentOS BuildSystem 2024-03-29T09:23:36.860 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2024-03-29T09:23:36.861 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T09:23:36.861 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2024-03-29T09:23:36.861 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2024-03-29T09:23:36.861 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:23:36.861 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T09:23:36.861 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T09:23:36.861 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T09:23:36.861 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2024-03-29T09:23:36.861 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:23:36.861 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T09:23:36.862 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T09:23:36.862 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 6.8.0-g1df0c7e8a235 2024-03-29T09:23:36.862 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T09:23:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-03-29T09:23:43.669 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-03-29T09:23:43.670 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-03-29T09:23:43.703 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.OL3Xh4MMas 2024-03-29T09:23:43.703 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OL3Xh4MMas 2024-03-29T09:23:43.776 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.OL3Xh4MMas 2024-03-29T09:23:43.944 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.OL3Xh4MMas is 6KB 2024-03-29T09:23:43.992 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.OL3Xh4MMas 2024-03-29T09:23:44.010 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T09:23:44.076 INFO:teuthology.orchestra.run.smithi012.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-29T09:23:44.076 INFO:teuthology.orchestra.run.smithi012.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-29T09:23:44.076 INFO:teuthology.orchestra.run.smithi012.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-29T09:23:44.076 INFO:teuthology.orchestra.run.smithi012.stdout:a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235.conf 2024-03-29T09:23:44.078 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235 2024-03-29T09:23:44.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-03-29T09:23:46.885 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-29T09:23:51.079 INFO:teuthology.orchestra.run.smithi159.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T09:24:11.064 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi159.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-29T09:24:11.064 DEBUG:teuthology.orchestra.run.smithi159:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-29T09:24:11.127 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-29T09:24:11.128 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-29T09:24:11.129 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-29T09:24:11.130 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-29T09:24:11.131 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-29T09:24:11.132 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-29T09:24:11.133 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-29T09:24:11.134 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-29T09:24:11.135 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-29T09:24:11.136 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-29T09:24:11.137 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-29T09:24:11.138 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-29T09:24:11.139 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-29T09:24:11.140 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-29T09:24:11.141 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-29T09:24:11.142 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-29T09:24:11.143 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-29T09:24:11.144 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-29T09:24:11.145 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-29T09:24:11.146 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-29T09:24:11.147 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-29T09:24:11.148 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-29T09:24:11.149 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-29T09:24:11.150 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-29T09:24:11.151 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-29T09:24:11.152 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-29T09:24:11.153 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-29T09:24:11.154 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-29T09:24:11.155 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-29T09:24:11.156 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-29T09:24:11.157 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-29T09:24:11.158 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-29T09:24:11.159 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-29T09:24:11.160 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-29T09:24:11.161 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-29T09:24:11.162 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-29T09:24:11.163 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-29T09:24:11.164 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-29T09:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-29T09:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-29T09:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-29T09:24:11.168 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-29T09:24:11.169 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-29T09:24:11.170 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-29T09:24:11.171 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-29T09:24:11.172 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-29T09:24:11.173 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-29T09:24:11.174 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-29T09:24:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-29T09:24:11.176 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-29T09:24:11.177 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-29T09:24:11.178 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-29T09:24:11.179 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-29T09:24:11.180 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-29T09:24:11.181 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-29T09:24:11.182 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-29T09:24:11.183 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-29T09:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-29T09:24:11.185 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-29T09:24:11.186 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-29T09:24:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-29T09:24:11.188 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-29T09:24:11.189 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-29T09:24:11.190 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-29T09:24:11.191 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-29T09:24:11.192 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-29T09:24:11.193 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-29T09:24:11.194 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-29T09:24:11.195 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-29T09:24:11.196 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-29T09:24:11.197 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-29T09:24:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-29T09:24:11.199 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-29T09:24:11.200 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-29T09:24:11.201 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-29T09:24:11.202 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-29T09:24:11.203 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-29T09:24:11.204 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-29T09:24:11.205 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-29T09:24:11.206 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-29T09:24:11.207 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-29T09:24:11.208 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-29T09:24:11.209 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-29T09:24:11.210 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-29T09:24:11.211 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-29T09:24:11.212 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-29T09:24:11.213 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-29T09:24:11.214 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-29T09:24:11.215 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-29T09:24:11.216 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-29T09:24:11.217 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-29T09:24:11.218 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-29T09:24:11.219 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-29T09:24:11.220 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-29T09:24:11.221 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-29T09:24:11.222 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-29T09:24:11.223 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-29T09:24:11.224 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-29T09:24:11.225 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-29T09:24:11.226 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-29T09:24:11.227 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-29T09:24:11.228 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-29T09:24:11.229 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-29T09:24:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-29T09:24:11.231 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-29T09:24:11.231 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-29T09:24:11.231 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-29T09:24:11.231 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-29T09:24:11.231 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-29T09:24:11.231 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-29T09:24:11.231 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-29T09:24:11.231 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-29T09:24:11.231 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-29T09:24:11.231 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-29T09:24:11.232 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-29T09:24:11.233 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-29T09:24:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-29T09:24:11.235 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-29T09:24:11.236 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-29T09:24:11.237 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-29T09:24:11.238 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-29T09:24:11.239 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-29T09:24:11.240 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-29T09:24:11.241 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-29T09:24:11.242 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-29T09:24:11.243 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-29T09:24:11.244 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-29T09:24:11.245 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-29T09:24:11.246 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-29T09:24:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-29T09:24:11.248 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-29T09:24:11.249 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-29T09:24:11.250 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-29T09:24:11.251 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-29T09:24:11.252 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-29T09:24:11.253 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-29T09:24:11.254 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-29T09:24:11.255 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-29T09:24:11.256 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-29T09:24:11.257 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-29T09:24:11.258 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-29T09:24:11.259 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-29T09:24:11.260 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-29T09:24:11.261 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-29T09:24:11.262 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-29T09:24:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-29T09:24:11.264 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-29T09:24:11.265 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-29T09:24:11.265 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-29T09:24:11.265 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-29T09:24:11.265 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-29T09:24:11.265 INFO:teuthology.orchestra.run.smithi159.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-29T09:24:11.268 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-29T09:24:11.269 DEBUG:teuthology.orchestra.run.smithi159:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout:postinstall scriptlet (using /bin/sh): 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout:fi 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout:for file in vmlinuz System.map config; do 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout: fi 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout:done 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-29T09:24:11.294 INFO:teuthology.orchestra.run.smithi159.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-29T09:24:11.295 INFO:teuthology.orchestra.run.smithi159.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-29T09:24:11.295 INFO:teuthology.orchestra.run.smithi159.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-29T09:24:11.295 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-29T09:24:11.295 INFO:teuthology.orchestra.run.smithi159.stdout:fi 2024-03-29T09:24:11.295 INFO:teuthology.orchestra.run.smithi159.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-29T09:24:11.295 INFO:teuthology.orchestra.run.smithi159.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-29T09:24:11.295 INFO:teuthology.orchestra.run.smithi159.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-29T09:24:11.295 INFO:teuthology.orchestra.run.smithi159.stdout:fi 2024-03-29T09:24:11.296 DEBUG:teuthology.orchestra.run.smithi159:> sudo rpm -qi grub2-tools 2024-03-29T09:24:11.372 INFO:teuthology.orchestra.run.smithi159.stdout:Name : grub2-tools 2024-03-29T09:24:11.372 INFO:teuthology.orchestra.run.smithi159.stdout:Epoch : 1 2024-03-29T09:24:11.372 INFO:teuthology.orchestra.run.smithi159.stdout:Version : 2.02 2024-03-29T09:24:11.372 INFO:teuthology.orchestra.run.smithi159.stdout:Release : 129.el8 2024-03-29T09:24:11.372 INFO:teuthology.orchestra.run.smithi159.stdout:Architecture: x86_64 2024-03-29T09:24:11.372 INFO:teuthology.orchestra.run.smithi159.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Group : System Environment/Base 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Size : 9265586 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:License : GPLv3+ 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Relocations : (not relocatable) 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Packager : CentOS BuildSystem 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Vendor : CentOS 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Summary : Support tools for GRUB. 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:Description : 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:24:11.373 INFO:teuthology.orchestra.run.smithi159.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T09:24:11.374 INFO:teuthology.orchestra.run.smithi159.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T09:24:11.374 INFO:teuthology.orchestra.run.smithi159.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T09:24:11.374 INFO:teuthology.orchestra.run.smithi159.stdout:hardware devices. 2024-03-29T09:24:11.374 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:24:11.374 INFO:teuthology.orchestra.run.smithi159.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T09:24:11.375 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T09:24:11.375 INFO:teuthology.task.kernel:Updating grub on smithi159 to boot 6.8.0-g1df0c7e8a235 2024-03-29T09:24:11.375 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T09:24:14.236 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-29T09:24:14.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-29T09:24:14.237 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:24:15.969 INFO:teuthology.orchestra.run.smithi159.stderr:Generating grub configuration file ... 2024-03-29T09:24:18.488 INFO:teuthology.orchestra.run.smithi159.stderr:done 2024-03-29T09:24:18.489 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-03-29T09:24:18.558 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.gcG7PttRn7 2024-03-29T09:24:18.559 DEBUG:teuthology.orchestra.run.smithi159:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gcG7PttRn7 2024-03-29T09:24:18.632 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.gcG7PttRn7 2024-03-29T09:24:18.809 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.gcG7PttRn7 is 6KB 2024-03-29T09:24:18.861 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.gcG7PttRn7 2024-03-29T09:24:18.879 DEBUG:teuthology.orchestra.run.smithi159:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T09:24:18.950 INFO:teuthology.orchestra.run.smithi159.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-29T09:24:18.950 INFO:teuthology.orchestra.run.smithi159.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-29T09:24:18.950 INFO:teuthology.orchestra.run.smithi159.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-29T09:24:18.950 INFO:teuthology.orchestra.run.smithi159.stdout:a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235.conf 2024-03-29T09:24:18.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235 2024-03-29T09:24:19.094 DEBUG:teuthology.orchestra.run.smithi159:> sudo shutdown -r now 2024-03-29T09:24:32.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-29T09:24:41.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-29T09:24:41.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:24:49.102 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-29T09:24:49.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-29T09:24:49.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:25:00.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-29T09:25:12.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-29T09:25:12.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:25:15.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-29T09:25:23.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-03-29T09:25:30.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-29T09:25:30.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:25:30.549 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-03-29T09:25:30.962 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-29T09:25:30.962 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-03-29T09:25:30.962 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-03-29T09:25:31.006 INFO:teuthology.orchestra.run.smithi012.stdout:6.8.0-g1df0c7e8a235 2024-03-29T09:25:31.006 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-29T09:25:31.006 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T09:25:31.007 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T09:25:32.008 DEBUG:teuthology.parallel:result is None 2024-03-29T09:25:32.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-29T09:25:32.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:26:32.618 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T09:26:44.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-29T09:26:44.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:26:44.974 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-03-29T09:26:45.451 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-29T09:26:45.451 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.8.0-g1df0c7e8a235"... 2024-03-29T09:26:45.452 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-03-29T09:26:45.497 INFO:teuthology.orchestra.run.smithi159.stdout:6.8.0-g1df0c7e8a235 2024-03-29T09:26:45.498 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-29T09:26:45.498 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T09:26:45.498 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T09:26:46.498 DEBUG:teuthology.parallel:result is None 2024-03-29T09:26:46.499 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T09:26:46.512 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T09:26:46.512 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T09:26:46.517 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T09:26:46.566 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T09:26:46.573 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T09:26:46.580 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T09:26:46.580 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T09:26:46.584 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T09:26:46.642 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T09:26:46.648 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T09:26:46.648 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T09:26:46.652 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T09:26:46.779 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T09:26:46.793 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T09:26:46.810 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T09:26:46.846 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T09:26:46.850 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T09:26:46.857 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T09:26:46.857 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T09:26:46.861 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T09:26:46.929 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T09:26:46.938 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T09:26:46.938 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T09:26:46.941 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T09:26:46.985 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T09:26:47.093 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T09:26:47.173 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T09:26:47.239 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T09:26:47.306 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:26:47.306 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T09:26:47.373 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T09:26:47.444 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T09:26:47.520 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T09:26:47.592 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T09:26:47.658 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:26:47.658 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T09:26:47.724 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-03-29T09:26:47.728 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2024-03-29T09:26:47.847 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T09:26:47.876 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T09:26:48.305 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T09:26:48.313 INFO:teuthology.task.internal:Starting timer... 2024-03-29T09:26:48.313 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T09:26:48.323 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T09:26:48.331 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-29T09:26:48.331 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2024-03-29T09:26:48.445 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2024-03-29T09:26:48.449 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2024-03-29T09:26:48.567 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: 2024-03-29T09:26:48.570 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T09:26:48.571 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2024-03-29T09:26:48.626 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2024-03-29T09:26:48.627 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2024-03-29T09:26:48.676 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2024-03-29T09:26:48.676 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2024-03-29T09:26:48.676 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T09:26:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T09:26:48.748 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T09:26:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T09:26:48.817 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T09:26:48.817 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2024-03-29T09:26:48.887 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2024-03-29T09:26:48.955 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T09:26:48.966 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T09:26:48.972 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T09:26:49.096 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T09:26:49.111 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T09:26:49.112 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T09:42:16.270 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi159.front.sepia.ceph.com')] 2024-03-29T09:42:16.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-29T09:42:16.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:42:16.378 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-03-29T09:42:16.492 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-29T09:42:16.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-29T09:42:16.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T09:42:16.583 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-03-29T09:42:16.698 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-29T09:42:16.698 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T09:42:16.711 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T09:42:16.711 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T09:42:16.712 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T09:42:16.717 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T09:42:16.718 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T09:42:16.790 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T09:42:16.810 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T09:42:16.823 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T09:42:16.841 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T09:42:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2024-03-29T09:42:16.929 INFO:teuthology.orchestra.run.smithi159.stderr:sudo: ntpd: command not found 2024-03-29T09:42:16.934 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2024-03-29T09:42:16.955 INFO:teuthology.orchestra.run.smithi159.stdout:506 Cannot talk to daemon 2024-03-29T09:42:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T09:42:16.987 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T09:42:16.997 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T09:42:17.018 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T09:42:17.083 INFO:teuthology.orchestra.run.smithi012.stderr:bash: ntpq: command not found 2024-03-29T09:42:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T09:42:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-03-29T09:42:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T09:42:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T09:42:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T09:42:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T09:42:17.103 INFO:teuthology.orchestra.run.smithi159.stderr:bash: ntpq: command not found 2024-03-29T09:42:17.111 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T09:42:17.111 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-03-29T09:42:17.111 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T09:42:17.111 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T09:42:17.112 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T09:42:17.112 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T09:42:17.113 INFO:teuthology.run_tasks:Running task install... 2024-03-29T09:42:17.121 DEBUG:teuthology.task.install:project ceph 2024-03-29T09:42:17.121 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'}} 2024-03-29T09:42:17.121 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'} 2024-03-29T09:42:17.121 INFO:teuthology.task.install:Using flavor: default 2024-03-29T09:42:17.127 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T09:42:17.127 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-03-29T09:42:17.128 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2', 'tag': None, 'wait_for_package': False} 2024-03-29T09:42:17.128 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d0e392d9aee635f2b65768c460c7a151ead7a5e2 2024-03-29T09:42:17.130 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2', 'tag': None, 'wait_for_package': False} 2024-03-29T09:42:17.130 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d0e392d9aee635f2b65768c460c7a151ead7a5e2 2024-03-29T09:42:17.275 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/ 2024-03-29T09:42:17.275 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1015.gd0e392d9 2024-03-29T09:42:17.370 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/ 2024-03-29T09:42:17.370 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1015.gd0e392d9 2024-03-29T09:42:17.388 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T09:42:17.388 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:42:17.388 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T09:42:17.455 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-29T09:42:17.456 DEBUG:teuthology.orchestra.run.smithi159:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d0e392d9aee635f2b65768c460c7a151ead7a5e2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T09:42:17.479 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T09:42:17.479 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:42:17.480 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T09:42:17.549 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-29T09:42:17.549 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d0e392d9aee635f2b65768c460c7a151ead7a5e2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T09:42:17.563 DEBUG:teuthology.orchestra.run.smithi159:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T09:42:17.654 DEBUG:teuthology.orchestra.run.smithi159:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T09:42:17.659 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T09:42:17.720 INFO:teuthology.orchestra.run.smithi159.stdout:check_obsoletes = 1 2024-03-29T09:42:17.723 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum clean all 2024-03-29T09:42:17.752 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T09:42:17.818 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2024-03-29T09:42:17.821 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2024-03-29T09:42:18.059 INFO:teuthology.orchestra.run.smithi159.stdout:76 files removed 2024-03-29T09:42:18.096 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-29T09:42:18.179 INFO:teuthology.orchestra.run.smithi012.stdout:76 files removed 2024-03-29T09:42:18.216 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-29T09:42:19.017 INFO:teuthology.orchestra.run.smithi159.stdout:ceph packages for x86_64 215 kB/s | 80 kB 00:00 2024-03-29T09:42:19.157 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 217 kB/s | 80 kB 00:00 2024-03-29T09:42:19.375 INFO:teuthology.orchestra.run.smithi159.stdout:ceph noarch packages 31 kB/s | 9.9 kB 00:00 2024-03-29T09:42:19.566 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 27 kB/s | 9.9 kB 00:00 2024-03-29T09:42:19.634 INFO:teuthology.orchestra.run.smithi159.stdout:ceph source packages 6.4 kB/s | 1.5 kB 00:00 2024-03-29T09:42:19.951 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 4.2 kB/s | 1.5 kB 00:00 2024-03-29T09:50:40.733 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - AppStream 57 kB/s | 28 MB 08:21 2024-03-29T09:50:46.203 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 8 - AppStream 56 kB/s | 28 MB 08:26 2024-03-29T09:50:47.025 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-03-29T09:50:48.917 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - Extras 91 kB/s | 18 kB 00:00 2024-03-29T09:50:49.318 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 7.5 kB 00:00 2024-03-29T09:50:50.944 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - PowerTools 3.2 MB/s | 4.9 MB 00:01 2024-03-29T09:50:52.070 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-03-29T09:50:52.427 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 8 - BaseOS 18 MB/s | 10 MB 00:00 2024-03-29T09:50:53.888 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 16 MB 00:01 2024-03-29T09:50:54.444 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 8 - Extras 73 kB/s | 18 kB 00:00 2024-03-29T09:50:54.845 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 8 - Extras common packages 27 kB/s | 7.5 kB 00:00 2024-03-29T09:50:56.571 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 8 - PowerTools 3.0 MB/s | 4.9 MB 00:01 2024-03-29T09:50:57.740 INFO:teuthology.orchestra.run.smithi012.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-03-29T09:50:58.363 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 349 kB/s | 24 kB 00:00 2024-03-29T09:50:59.340 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-03-29T09:51:02.156 INFO:teuthology.orchestra.run.smithi159.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-29T09:51:02.157 INFO:teuthology.orchestra.run.smithi159.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-29T09:51:02.167 INFO:teuthology.orchestra.run.smithi159.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2024-03-29T09:51:02.168 INFO:teuthology.orchestra.run.smithi159.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2024-03-29T09:51:02.168 INFO:teuthology.orchestra.run.smithi159.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2024-03-29T09:51:02.169 INFO:teuthology.orchestra.run.smithi159.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-29T09:51:02.171 INFO:teuthology.orchestra.run.smithi159.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-29T09:51:02.173 INFO:teuthology.orchestra.run.smithi159.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-29T09:51:02.174 INFO:teuthology.orchestra.run.smithi159.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-29T09:51:02.259 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-29T09:51:02.274 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-29T09:51:02.274 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-29T09:51:02.274 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-29T09:51:02.274 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-29T09:51:02.274 INFO:teuthology.orchestra.run.smithi159.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2024-03-29T09:51:02.274 INFO:teuthology.orchestra.run.smithi159.stdout: ceph x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 6.4 k 2024-03-29T09:51:02.274 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 5.5 M 2024-03-29T09:51:02.274 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 854 k 2024-03-29T09:51:02.274 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 166 k 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 1.6 M 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 138 k 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 1.8 M 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 7.4 M 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 49 k 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 13 M 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 47 M 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 263 k 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 173 k 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-mirror x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 261 k 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-top noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 18 k 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-devel x86_64 0.190-2.el8 baseos 62 k 2024-03-29T09:51:02.275 INFO:teuthology.orchestra.run.smithi159.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 31 k 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 778 k 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 139 k 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-devel x86_64 2.32.1-46.el8 baseos 100 k 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-devel x86_64 1:1.1.1k-12.el8 baseos 3.2 M 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql x86_64 10.23-2.module_el8+572+929c87ac appstream 1.6 M 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-contrib x86_64 10.23-2.module_el8+572+929c87ac appstream 901 k 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-server x86_64 10.23-2.module_el8+572+929c87ac appstream 5.1 M 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 197 k 2024-03-29T09:51:02.276 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 371 k 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 364 k 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 106 k 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 95 k 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.5 M 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 181 k 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager x86_64 1:1.40.16-15.el8 baseos 2.3 M 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-libnm x86_64 1:1.40.16-15.el8 baseos 1.9 M 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-team x86_64 1:1.40.16-15.el8 baseos 161 k 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-tui x86_64 1:1.40.16-15.el8 baseos 356 k 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: acl x86_64 2.2.53-3.el8 baseos 83 k 2024-03-29T09:51:02.277 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-debuginfod-client x86_64 0.190-2.el8 baseos 77 k 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf x86_64 0.190-2.el8 baseos 233 k 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libs x86_64 0.190-2.el8 baseos 305 k 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: libacl x86_64 2.2.53-3.el8 baseos 35 k 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: libacl-devel x86_64 2.2.53-3.el8 baseos 85 k 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: libblkid x86_64 2.32.1-46.el8 baseos 221 k 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: libfdisk x86_64 2.32.1-46.el8 baseos 254 k 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: libmount x86_64 2.32.1-46.el8 baseos 237 k 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.7 M 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.6 M 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: libsmartcols x86_64 2.32.1-46.el8 baseos 180 k 2024-03-29T09:51:02.278 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid x86_64 2.32.1-46.el8 baseos 99 k 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: openssl x86_64 1:1.1.1k-12.el8 baseos 738 k 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-libs x86_64 1:1.1.1k-12.el8 baseos 1.5 M 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux x86_64 2.32.1-46.el8 baseos 2.6 M 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-user x86_64 2.32.1-46.el8 baseos 103 k 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: xfsdump x86_64 3.1.8-7.el8 baseos 334 k 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 22 M 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 23 k 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 2.3 M 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 245 k 2024-03-29T09:51:02.279 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 4.8 M 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 18 M 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 15 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 24 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 142 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-29T09:51:02.280 INFO:teuthology.orchestra.run.smithi159.stdout: libpq x86_64 13.5-1.el8 appstream 198 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 474 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 6.5 M 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-29T09:51:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 45 k 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 129 k 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-29T09:51:02.282 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-29T09:51:02.283 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-03-29T09:51:02.284 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-29T09:51:02.285 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout: uuid x86_64 1.6.2-43.el8 appstream 64 k 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout:Enabling module streams: 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql 10 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-29T09:51:02.286 INFO:teuthology.orchestra.run.smithi159.stdout:Install 129 Packages 2024-03-29T09:51:02.287 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 24 Packages 2024-03-29T09:51:02.287 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:51:02.287 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 220 M 2024-03-29T09:51:02.287 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-29T09:51:03.247 INFO:teuthology.orchestra.run.smithi159.stdout:(1/153): ceph-18.2.2-1015.gd0e392d9.el8.x86_64. 30 kB/s | 6.4 kB 00:00 2024-03-29T09:51:03.773 INFO:teuthology.orchestra.run.smithi159.stdout:(2/153): ceph-fuse-18.2.2-1015.gd0e392d9.el8.x8 1.6 MB/s | 854 kB 00:00 2024-03-29T09:51:03.881 INFO:teuthology.orchestra.run.smithi159.stdout:(3/153): ceph-immutable-object-cache-18.2.2-101 1.5 MB/s | 166 kB 00:00 2024-03-29T09:51:04.032 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 347 kB/s | 24 kB 00:00 2024-03-29T09:51:04.506 INFO:teuthology.orchestra.run.smithi159.stdout:(4/153): ceph-mds-18.2.2-1015.gd0e392d9.el8.x86 3.7 MB/s | 2.3 MB 00:00 2024-03-29T09:51:04.732 INFO:teuthology.orchestra.run.smithi159.stdout:(5/153): ceph-base-18.2.2-1015.gd0e392d9.el8.x8 3.2 MB/s | 5.5 MB 00:01 2024-03-29T09:51:04.824 INFO:teuthology.orchestra.run.smithi159.stdout:(6/153): ceph-mgr-18.2.2-1015.gd0e392d9.el8.x86 4.9 MB/s | 1.6 MB 00:00 2024-03-29T09:51:05.374 INFO:teuthology.orchestra.run.smithi159.stdout:(7/153): ceph-mon-18.2.2-1015.gd0e392d9.el8.x86 7.5 MB/s | 4.8 MB 00:00 2024-03-29T09:51:06.916 INFO:teuthology.orchestra.run.smithi159.stdout:(8/153): ceph-common-18.2.2-1015.gd0e392d9.el8. 5.6 MB/s | 22 MB 00:03 2024-03-29T09:51:07.100 INFO:teuthology.orchestra.run.smithi159.stdout:(9/153): ceph-radosgw-18.2.2-1015.gd0e392d9.el8 7.6 MB/s | 13 MB 00:01 2024-03-29T09:51:07.125 INFO:teuthology.orchestra.run.smithi159.stdout:(10/153): ceph-selinux-18.2.2-1015.gd0e392d9.el 117 kB/s | 24 kB 00:00 2024-03-29T09:51:07.375 INFO:teuthology.orchestra.run.smithi159.stdout:(11/153): ceph-osd-18.2.2-1015.gd0e392d9.el8.x8 7.2 MB/s | 18 MB 00:02 2024-03-29T09:51:07.400 INFO:teuthology.orchestra.run.smithi159.stdout:(12/153): cephfs-mirror-18.2.2-1015.gd0e392d9.e 948 kB/s | 261 kB 00:00 2024-03-29T09:51:07.442 INFO:teuthology.orchestra.run.smithi159.stdout:(13/153): libcephfs-devel-18.2.2-1015.gd0e392d9 465 kB/s | 31 kB 00:00 2024-03-29T09:51:07.526 INFO:teuthology.orchestra.run.smithi159.stdout:(14/153): libcephsqlite-18.2.2-1015.gd0e392d9.e 1.7 MB/s | 142 kB 00:00 2024-03-29T09:51:07.576 INFO:teuthology.orchestra.run.smithi159.stdout:(15/153): libcephfs2-18.2.2-1015.gd0e392d9.el8. 4.3 MB/s | 778 kB 00:00 2024-03-29T09:51:07.602 INFO:teuthology.orchestra.run.smithi159.stdout:(16/153): librados-devel-18.2.2-1015.gd0e392d9. 1.8 MB/s | 139 kB 00:00 2024-03-29T09:51:07.752 INFO:teuthology.orchestra.run.smithi159.stdout:(17/153): libradosstriper1-18.2.2-1015.gd0e392d 2.6 MB/s | 474 kB 00:00 2024-03-29T09:51:07.869 INFO:teuthology.orchestra.run.smithi159.stdout:(18/153): python3-ceph-argparse-18.2.2-1015.gd0 383 kB/s | 45 kB 00:00 2024-03-29T09:51:07.917 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-29T09:51:07.917 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-29T09:51:07.927 INFO:teuthology.orchestra.run.smithi012.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2024-03-29T09:51:07.928 INFO:teuthology.orchestra.run.smithi012.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2024-03-29T09:51:07.929 INFO:teuthology.orchestra.run.smithi012.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2024-03-29T09:51:07.930 INFO:teuthology.orchestra.run.smithi012.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-29T09:51:07.931 INFO:teuthology.orchestra.run.smithi012.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-29T09:51:07.933 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-29T09:51:07.934 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-29T09:51:07.977 INFO:teuthology.orchestra.run.smithi159.stdout:(19/153): python3-ceph-common-18.2.2-1015.gd0e3 1.2 MB/s | 129 kB 00:00 2024-03-29T09:51:08.025 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-29T09:51:08.041 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-29T09:51:08.042 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-29T09:51:08.042 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-29T09:51:08.042 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-29T09:51:08.043 INFO:teuthology.orchestra.run.smithi012.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2024-03-29T09:51:08.044 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 6.4 k 2024-03-29T09:51:08.044 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 5.5 M 2024-03-29T09:51:08.044 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 854 k 2024-03-29T09:51:08.044 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 166 k 2024-03-29T09:51:08.044 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 1.6 M 2024-03-29T09:51:08.044 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 138 k 2024-03-29T09:51:08.044 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 1.8 M 2024-03-29T09:51:08.044 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 7.4 M 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 49 k 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 13 M 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 47 M 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 263 k 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 173 k 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 261 k 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 18 k 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel x86_64 0.190-2.el8 baseos 62 k 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 31 k 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 778 k 2024-03-29T09:51:08.045 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 139 k 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel x86_64 2.32.1-46.el8 baseos 100 k 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel x86_64 1:1.1.1k-12.el8 baseos 3.2 M 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql x86_64 10.23-2.module_el8+572+929c87ac appstream 1.6 M 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib x86_64 10.23-2.module_el8+572+929c87ac appstream 901 k 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server x86_64 10.23-2.module_el8+572+929c87ac appstream 5.1 M 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 197 k 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 371 k 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 364 k 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 106 k 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 95 k 2024-03-29T09:51:08.046 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.5 M 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 181 k 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager x86_64 1:1.40.16-15.el8 baseos 2.3 M 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm x86_64 1:1.40.16-15.el8 baseos 1.9 M 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team x86_64 1:1.40.16-15.el8 baseos 161 k 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui x86_64 1:1.40.16-15.el8 baseos 356 k 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: acl x86_64 2.2.53-3.el8 baseos 83 k 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client x86_64 0.190-2.el8 baseos 77 k 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf x86_64 0.190-2.el8 baseos 233 k 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs x86_64 0.190-2.el8 baseos 305 k 2024-03-29T09:51:08.047 INFO:teuthology.orchestra.run.smithi012.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: libacl x86_64 2.2.53-3.el8 baseos 35 k 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-devel x86_64 2.2.53-3.el8 baseos 85 k 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: libblkid x86_64 2.32.1-46.el8 baseos 221 k 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: libfdisk x86_64 2.32.1-46.el8 baseos 254 k 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: libmount x86_64 2.32.1-46.el8 baseos 237 k 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.7 M 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.6 M 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: libsmartcols x86_64 2.32.1-46.el8 baseos 180 k 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid x86_64 2.32.1-46.el8 baseos 99 k 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: openssl x86_64 1:1.1.1k-12.el8 baseos 738 k 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs x86_64 1:1.1.1k-12.el8 baseos 1.5 M 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux x86_64 2.32.1-46.el8 baseos 2.6 M 2024-03-29T09:51:08.048 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-user x86_64 2.32.1-46.el8 baseos 103 k 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: xfsdump x86_64 3.1.8-7.el8 baseos 334 k 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 22 M 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 23 k 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 2.3 M 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 245 k 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 4.8 M 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 18 M 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 15 k 2024-03-29T09:51:08.049 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 24 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 142 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: libpq x86_64 13.5-1.el8 appstream 198 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-03-29T09:51:08.050 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-29T09:51:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 474 k 2024-03-29T09:51:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-29T09:51:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 6.5 M 2024-03-29T09:51:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2024-03-29T09:51:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2024-03-29T09:51:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-29T09:51:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2024-03-29T09:51:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2024-03-29T09:51:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-29T09:51:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 45 k 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 129 k 2024-03-29T09:51:08.052 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-29T09:51:08.053 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-29T09:51:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-29T09:51:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-29T09:51:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-29T09:51:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-03-29T09:51:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-03-29T09:51:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-03-29T09:51:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-03-29T09:51:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-29T09:51:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-29T09:51:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-29T09:51:08.055 INFO:teuthology.orchestra.run.smithi012.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: uuid x86_64 1.6.2-43.el8 appstream 64 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-29T09:51:08.056 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2024-03-29T09:51:08.057 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling module streams: 2024-03-29T09:51:08.057 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql 10 2024-03-29T09:51:08.057 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:51:08.057 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-29T09:51:08.057 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-29T09:51:08.057 INFO:teuthology.orchestra.run.smithi012.stdout:Install 129 Packages 2024-03-29T09:51:08.057 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 24 Packages 2024-03-29T09:51:08.057 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:51:08.058 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 220 M 2024-03-29T09:51:08.058 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-29T09:51:08.078 INFO:teuthology.orchestra.run.smithi159.stdout:(20/153): python3-cephfs-18.2.2-1015.gd0e392d9. 1.9 MB/s | 197 kB 00:00 2024-03-29T09:51:08.253 INFO:teuthology.orchestra.run.smithi159.stdout:(21/153): python3-rados-18.2.2-1015.gd0e392d9.e 2.1 MB/s | 371 kB 00:00 2024-03-29T09:51:08.362 INFO:teuthology.orchestra.run.smithi159.stdout:(22/153): python3-rbd-18.2.2-1015.gd0e392d9.el8 3.3 MB/s | 364 kB 00:00 2024-03-29T09:51:08.445 INFO:teuthology.orchestra.run.smithi159.stdout:(23/153): librgw2-18.2.2-1015.gd0e392d9.el8.x86 7.8 MB/s | 6.5 MB 00:00 2024-03-29T09:51:08.462 INFO:teuthology.orchestra.run.smithi159.stdout:(24/153): python3-rgw-18.2.2-1015.gd0e392d9.el8 1.0 MB/s | 106 kB 00:00 2024-03-29T09:51:08.529 INFO:teuthology.orchestra.run.smithi159.stdout:(25/153): rbd-fuse-18.2.2-1015.gd0e392d9.el8.x8 1.1 MB/s | 95 kB 00:00 2024-03-29T09:51:08.638 INFO:teuthology.orchestra.run.smithi159.stdout:(26/153): rbd-nbd-18.2.2-1015.gd0e392d9.el8.x86 1.6 MB/s | 181 kB 00:00 2024-03-29T09:51:08.721 INFO:teuthology.orchestra.run.smithi159.stdout:(27/153): ceph-grafana-dashboards-18.2.2-1015.g 279 kB/s | 23 kB 00:00 2024-03-29T09:51:08.830 INFO:teuthology.orchestra.run.smithi159.stdout:(28/153): ceph-mgr-cephadm-18.2.2-1015.gd0e392d 1.2 MB/s | 138 kB 00:00 2024-03-29T09:51:09.041 INFO:teuthology.orchestra.run.smithi012.stdout:(1/153): ceph-18.2.2-1015.gd0e392d9.el8.x86_64. 20 kB/s | 6.4 kB 00:00 2024-03-29T09:51:09.305 INFO:teuthology.orchestra.run.smithi159.stdout:(29/153): ceph-mgr-dashboard-18.2.2-1015.gd0e39 3.8 MB/s | 1.8 MB 00:00 2024-03-29T09:51:09.364 INFO:teuthology.orchestra.run.smithi159.stdout:(30/153): rbd-mirror-18.2.2-1015.gd0e392d9.el8. 3.8 MB/s | 3.5 MB 00:00 2024-03-29T09:51:09.481 INFO:teuthology.orchestra.run.smithi159.stdout:(31/153): ceph-mgr-modules-core-18.2.2-1015.gd0 2.1 MB/s | 245 kB 00:00 2024-03-29T09:51:09.598 INFO:teuthology.orchestra.run.smithi159.stdout:(32/153): ceph-mgr-rook-18.2.2-1015.gd0e392d9.e 422 kB/s | 49 kB 00:00 2024-03-29T09:51:09.715 INFO:teuthology.orchestra.run.smithi159.stdout:(33/153): ceph-prometheus-alerts-18.2.2-1015.gd 126 kB/s | 15 kB 00:00 2024-03-29T09:51:09.840 INFO:teuthology.orchestra.run.smithi159.stdout:(34/153): ceph-volume-18.2.2-1015.gd0e392d9.el8 2.1 MB/s | 263 kB 00:00 2024-03-29T09:51:09.949 INFO:teuthology.orchestra.run.smithi159.stdout:(35/153): cephadm-18.2.2-1015.gd0e392d9.el8.noa 1.6 MB/s | 173 kB 00:00 2024-03-29T09:51:10.057 INFO:teuthology.orchestra.run.smithi159.stdout:(36/153): cephfs-top-18.2.2-1015.gd0e392d9.el8. 167 kB/s | 18 kB 00:00 2024-03-29T09:51:10.058 INFO:teuthology.orchestra.run.smithi012.stdout:(2/153): ceph-fuse-18.2.2-1015.gd0e392d9.el8.x8 840 kB/s | 854 kB 00:01 2024-03-29T09:51:10.291 INFO:teuthology.orchestra.run.smithi012.stdout:(3/153): ceph-immutable-object-cache-18.2.2-101 712 kB/s | 166 kB 00:00 2024-03-29T09:51:10.883 INFO:teuthology.orchestra.run.smithi159.stdout:(37/153): ceph-mgr-diskprediction-local-18.2.2- 4.7 MB/s | 7.4 MB 00:01 2024-03-29T09:51:10.925 INFO:teuthology.orchestra.run.smithi159.stdout:(38/153): automake-1.16.1-8.el8.noarch.rpm 853 kB/s | 740 kB 00:00 2024-03-29T09:51:11.058 INFO:teuthology.orchestra.run.smithi159.stdout:(39/153): flex-2.6.1-9.el8.x86_64.rpm 2.3 MB/s | 320 kB 00:00 2024-03-29T09:51:11.175 INFO:teuthology.orchestra.run.smithi159.stdout:(40/153): jq-1.6-8.el8.x86_64.rpm 1.7 MB/s | 203 kB 00:00 2024-03-29T09:51:11.309 INFO:teuthology.orchestra.run.smithi159.stdout:(41/153): libpq-13.5-1.el8.x86_64.rpm 1.4 MB/s | 198 kB 00:00 2024-03-29T09:51:11.484 INFO:teuthology.orchestra.run.smithi159.stdout:(42/153): librdkafka-1.6.1-1.el8.x86_64.rpm 3.8 MB/s | 677 kB 00:00 2024-03-29T09:51:11.543 INFO:teuthology.orchestra.run.smithi159.stdout:(43/153): bison-3.0.4-10.el8.x86_64.rpm 1.0 MB/s | 688 kB 00:00 2024-03-29T09:51:11.643 INFO:teuthology.orchestra.run.smithi159.stdout:(44/153): libtool-2.4.6-25.el8.x86_64.rpm 4.4 MB/s | 709 kB 00:00 2024-03-29T09:51:11.718 INFO:teuthology.orchestra.run.smithi159.stdout:(45/153): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2024-03-29T09:51:11.752 INFO:teuthology.orchestra.run.smithi159.stdout:(46/153): oniguruma-6.8.2-3.el8.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-03-29T09:51:11.916 INFO:teuthology.orchestra.run.smithi012.stdout:(4/153): ceph-base-18.2.2-1015.gd0e392d9.el8.x8 1.7 MB/s | 5.5 MB 00:03 2024-03-29T09:51:12.102 INFO:teuthology.orchestra.run.smithi159.stdout:(47/153): openblas-threads-0.3.15-4.el8.x86_64. 14 MB/s | 4.9 MB 00:00 2024-03-29T09:51:12.170 INFO:teuthology.orchestra.run.smithi159.stdout:(48/153): openblas-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2024-03-29T09:51:12.241 INFO:teuthology.orchestra.run.smithi012.stdout:(5/153): ceph-mds-18.2.2-1015.gd0e392d9.el8.x86 1.2 MB/s | 2.3 MB 00:01 2024-03-29T09:51:12.342 INFO:teuthology.orchestra.run.smithi012.stdout:(6/153): ceph-mgr-18.2.2-1015.gd0e392d9.el8.x86 3.7 MB/s | 1.6 MB 00:00 2024-03-29T09:51:12.695 INFO:teuthology.orchestra.run.smithi159.stdout:(49/153): ceph-test-18.2.2-1015.gd0e392d9.el8.x 8.5 MB/s | 47 MB 00:05 2024-03-29T09:51:12.737 INFO:teuthology.orchestra.run.smithi159.stdout:(50/153): postgresql-contrib-10.23-2.module_el8 1.6 MB/s | 901 kB 00:00 2024-03-29T09:51:12.779 INFO:teuthology.orchestra.run.smithi159.stdout:(51/153): postgresql-10.23-2.module_el8+572+929 2.3 MB/s | 1.6 MB 00:00 2024-03-29T09:51:12.904 INFO:teuthology.orchestra.run.smithi159.stdout:(52/153): python3-jinja2-2.10.1-4.el8.noarch.rp 4.5 MB/s | 578 kB 00:00 2024-03-29T09:51:13.021 INFO:teuthology.orchestra.run.smithi159.stdout:(53/153): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2024-03-29T09:51:13.046 INFO:teuthology.orchestra.run.smithi159.stdout:(54/153): python3-jsonpatch-1.21-2.el8.noarch.r 187 kB/s | 27 kB 00:00 2024-03-29T09:51:13.122 INFO:teuthology.orchestra.run.smithi159.stdout:(55/153): python3-jsonpointer-1.10-11.el8.noarc 200 kB/s | 20 kB 00:00 2024-03-29T09:51:13.139 INFO:teuthology.orchestra.run.smithi159.stdout:(56/153): python3-mako-1.0.6-14.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2024-03-29T09:51:13.222 INFO:teuthology.orchestra.run.smithi159.stdout:(57/153): python3-markupsafe-0.23-19.el8.x86_64 387 kB/s | 39 kB 00:00 2024-03-29T09:51:13.348 INFO:teuthology.orchestra.run.smithi159.stdout:(58/153): python3-numpy-1.14.3-10.el8.x86_64.rp 18 MB/s | 3.7 MB 00:00 2024-03-29T09:51:13.365 INFO:teuthology.orchestra.run.smithi159.stdout:(59/153): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.5 MB/s | 225 kB 00:00 2024-03-29T09:51:13.448 INFO:teuthology.orchestra.run.smithi159.stdout:(60/153): python3-prettytable-0.7.2-14.el8.noar 438 kB/s | 44 kB 00:00 2024-03-29T09:51:13.474 INFO:teuthology.orchestra.run.smithi159.stdout:(61/153): python3-pyOpenSSL-19.0.0-1.el8.noarch 952 kB/s | 103 kB 00:00 2024-03-29T09:51:13.549 INFO:teuthology.orchestra.run.smithi159.stdout:(62/153): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-03-29T09:51:13.574 INFO:teuthology.orchestra.run.smithi159.stdout:(63/153): python3-pyasn1-modules-0.3.7-6.el8.no 1.1 MB/s | 110 kB 00:00 2024-03-29T09:51:13.651 INFO:teuthology.orchestra.run.smithi159.stdout:(64/153): python3-pytz-2017.2-11.el8.noarch.rpm 570 kB/s | 57 kB 00:00 2024-03-29T09:51:13.758 INFO:teuthology.orchestra.run.smithi159.stdout:(65/153): postgresql-server-10.23-2.module_el8+ 4.8 MB/s | 5.1 MB 00:01 2024-03-29T09:51:13.817 INFO:teuthology.orchestra.run.smithi159.stdout:(66/153): python3-werkzeug-0.12.2-4.el8.noarch. 2.7 MB/s | 457 kB 00:00 2024-03-29T09:51:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:(67/153): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2024-03-29T09:51:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:(68/153): uuid-1.6.2-43.el8.x86_64.rpm 511 kB/s | 64 kB 00:00 2024-03-29T09:51:14.051 INFO:teuthology.orchestra.run.smithi159.stdout:(69/153): xmlstarlet-1.6.1-20.el8.x86_64.rpm 530 kB/s | 71 kB 00:00 2024-03-29T09:51:14.226 INFO:teuthology.orchestra.run.smithi159.stdout:(70/153): python3-scipy-1.0.0-21.module_el8+762 22 MB/s | 14 MB 00:00 2024-03-29T09:51:14.277 INFO:teuthology.orchestra.run.smithi159.stdout:(71/153): elfutils-libelf-devel-0.190-2.el8.x86 186 kB/s | 62 kB 00:00 2024-03-29T09:51:14.352 INFO:teuthology.orchestra.run.smithi159.stdout:(72/153): fuse-2.9.7-19.el8.x86_64.rpm 277 kB/s | 83 kB 00:00 2024-03-29T09:51:14.427 INFO:teuthology.orchestra.run.smithi159.stdout:(73/153): libaio-devel-0.3.112-1.el8.x86_64.rpm 249 kB/s | 19 kB 00:00 2024-03-29T09:51:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:(74/153): keyutils-libs-devel-1.5.10-9.el8.x86_ 151 kB/s | 48 kB 00:00 2024-03-29T09:51:14.744 INFO:teuthology.orchestra.run.smithi159.stdout:(75/153): krb5-devel-1.18.2-22.el8.x86_64.rpm 1.2 MB/s | 561 kB 00:00 2024-03-29T09:51:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:(76/153): libcom_err-devel-1.45.6-5.el8.x86_64. 108 kB/s | 39 kB 00:00 2024-03-29T09:51:14.862 INFO:teuthology.orchestra.run.smithi159.stdout:(77/153): libgfortran-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 645 kB 00:00 2024-03-29T09:51:14.937 INFO:teuthology.orchestra.run.smithi159.stdout:(78/153): libquadmath-8.5.0-21.el8.x86_64.rpm 897 kB/s | 172 kB 00:00 2024-03-29T09:51:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:(79/153): libselinux-devel-2.9-8.el8.x86_64.rpm 889 kB/s | 200 kB 00:00 2024-03-29T09:51:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:(80/153): libsepol-devel-2.9-3.el8.x86_64.rpm 434 kB/s | 87 kB 00:00 2024-03-29T09:51:15.163 INFO:teuthology.orchestra.run.smithi159.stdout:(81/153): libuuid-devel-2.32.1-46.el8.x86_64.rp 1.3 MB/s | 100 kB 00:00 2024-03-29T09:51:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:(82/153): libverto-devel-0.3.2-2.el8.x86_64.rpm 240 kB/s | 18 kB 00:00 2024-03-29T09:51:15.233 INFO:teuthology.orchestra.run.smithi012.stdout:(7/153): ceph-osd-18.2.2-1015.gd0e392d9.el8.x86 6.4 MB/s | 18 MB 00:02 2024-03-29T09:51:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:(83/153): librabbitmq-0.9.0-5.el8.x86_64.rpm 104 kB/s | 47 kB 00:00 2024-03-29T09:51:15.272 INFO:teuthology.orchestra.run.smithi159.stdout:(84/153): libzstd-devel-1.4.4-1.el8.x86_64.rpm 402 kB/s | 44 kB 00:00 2024-03-29T09:51:15.383 INFO:teuthology.orchestra.run.smithi012.stdout:(8/153): ceph-mon-18.2.2-1015.gd0e392d9.el8.x86 1.5 MB/s | 4.8 MB 00:03 2024-03-29T09:51:15.397 INFO:teuthology.orchestra.run.smithi159.stdout:(85/153): pcre2-utf16-10.32-3.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2024-03-29T09:51:15.475 INFO:teuthology.orchestra.run.smithi012.stdout:(9/153): ceph-selinux-18.2.2-1015.gd0e392d9.el8 266 kB/s | 24 kB 00:00 2024-03-29T09:51:15.564 INFO:teuthology.orchestra.run.smithi159.stdout:(86/153): pcre2-utf32-10.32-3.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2024-03-29T09:51:15.598 INFO:teuthology.orchestra.run.smithi159.stdout:(87/153): pcre2-devel-10.32-3.el8.x86_64.rpm 1.6 MB/s | 605 kB 00:00 2024-03-29T09:51:15.707 INFO:teuthology.orchestra.run.smithi159.stdout:(88/153): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.7 MB/s | 249 kB 00:00 2024-03-29T09:51:15.774 INFO:teuthology.orchestra.run.smithi159.stdout:(89/153): openssl-devel-1.1.1k-12.el8.x86_64.rp 5.7 MB/s | 3.2 MB 00:00 2024-03-29T09:51:15.815 INFO:teuthology.orchestra.run.smithi159.stdout:(90/153): python3-jwt-1.6.1-2.el8.noarch.rpm 393 kB/s | 43 kB 00:00 2024-03-29T09:51:15.945 INFO:teuthology.orchestra.run.smithi159.stdout:(91/153): python3-cryptography-3.2.1-7.el8.x86_ 1.8 MB/s | 641 kB 00:00 2024-03-29T09:51:16.058 INFO:teuthology.orchestra.run.smithi159.stdout:(92/153): python3-pycparser-2.14-14.el8.noarch. 451 kB/s | 109 kB 00:00 2024-03-29T09:51:16.083 INFO:teuthology.orchestra.run.smithi159.stdout:(93/153): python3-oauthlib-2.1.0-1.el8.noarch.r 501 kB/s | 155 kB 00:00 2024-03-29T09:51:16.108 INFO:teuthology.orchestra.run.smithi159.stdout:(94/153): python3-requests-oauthlib-1.0.0-1.el8 262 kB/s | 43 kB 00:00 2024-03-29T09:51:16.184 INFO:teuthology.orchestra.run.smithi159.stdout:(95/153): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 594 kB/s | 74 kB 00:00 2024-03-29T09:51:16.234 INFO:teuthology.orchestra.run.smithi159.stdout:(96/153): zlib-devel-1.2.11-21.el8.x86_64.rpm 464 kB/s | 58 kB 00:00 2024-03-29T09:51:16.309 INFO:teuthology.orchestra.run.smithi159.stdout:(97/153): python3-packaging-16.8-9.el8.noarch.r 241 kB/s | 48 kB 00:00 2024-03-29T09:51:16.393 INFO:teuthology.orchestra.run.smithi159.stdout:(98/153): liboath-2.6.2-3.el8.x86_64.rpm 372 kB/s | 59 kB 00:00 2024-03-29T09:51:16.460 INFO:teuthology.orchestra.run.smithi159.stdout:(99/153): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2024-03-29T09:51:16.477 INFO:teuthology.orchestra.run.smithi159.stdout:(100/153): libunwind-1.3.1-3.el8.x86_64.rpm 449 kB/s | 75 kB 00:00 2024-03-29T09:51:16.519 INFO:teuthology.orchestra.run.smithi159.stdout:(101/153): python3-bcrypt-3.1.6-2.el8.1.x86_64. 747 kB/s | 44 kB 00:00 2024-03-29T09:51:16.552 INFO:teuthology.orchestra.run.smithi159.stdout:(102/153): python3-asyncssh-2.7.0-2.el8.noarch. 2.9 MB/s | 476 kB 00:00 2024-03-29T09:51:16.578 INFO:teuthology.orchestra.run.smithi159.stdout:(103/153): python3-cachetools-3.1.1-4.el8.noarc 564 kB/s | 33 kB 00:00 2024-03-29T09:51:16.603 INFO:teuthology.orchestra.run.smithi159.stdout:(104/153): python3-beautifulsoup4-4.6.3-2.el8.1 1.4 MB/s | 185 kB 00:00 2024-03-29T09:51:16.628 INFO:teuthology.orchestra.run.smithi159.stdout:(105/153): python3-certifi-2018.10.15-7.el8.noa 217 kB/s | 16 kB 00:00 2024-03-29T09:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout:(106/153): python3-cheroot-8.5.2-1.el8.noarch.r 2.2 MB/s | 173 kB 00:00 2024-03-29T09:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout:(107/153): python3-google-auth-1.1.1-10.el8.noa 1.4 MB/s | 81 kB 00:00 2024-03-29T09:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout:(108/153): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2024-03-29T09:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout:(109/153): python3-cherrypy-18.4.0-1.el8.noarch 3.0 MB/s | 384 kB 00:00 2024-03-29T09:51:16.755 INFO:teuthology.orchestra.run.smithi159.stdout:(110/153): python3-jaraco-functools-2.0-4.el8.n 262 kB/s | 18 kB 00:00 2024-03-29T09:51:16.788 INFO:teuthology.orchestra.run.smithi159.stdout:(111/153): python3-logutils-0.3.5-11.el8.noarch 835 kB/s | 49 kB 00:00 2024-03-29T09:51:16.839 INFO:teuthology.orchestra.run.smithi159.stdout:(112/153): python3-more-itertools-7.2.0-3.el8.n 702 kB/s | 59 kB 00:00 2024-03-29T09:51:16.864 INFO:teuthology.orchestra.run.smithi159.stdout:(113/153): python3-natsort-7.1.1-2.el8.noarch.r 802 kB/s | 60 kB 00:00 2024-03-29T09:51:16.898 INFO:teuthology.orchestra.run.smithi159.stdout:(114/153): python3-kubernetes-11.0.0-6.el8.noar 6.3 MB/s | 1.2 MB 00:00 2024-03-29T09:51:16.923 INFO:teuthology.orchestra.run.smithi159.stdout:(115/153): python3-portend-2.6-1.el8.noarch.rpm 265 kB/s | 16 kB 00:00 2024-03-29T09:51:16.948 INFO:teuthology.orchestra.run.smithi159.stdout:(116/153): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2024-03-29T09:51:16.974 INFO:teuthology.orchestra.run.smithi159.stdout:(117/153): python3-repoze-lru-0.7-6.el8.noarch. 446 kB/s | 34 kB 00:00 2024-03-29T09:51:16.999 INFO:teuthology.orchestra.run.smithi159.stdout:(118/153): python3-routes-2.4.1-12.el8.noarch.r 2.6 MB/s | 196 kB 00:00 2024-03-29T09:51:17.024 INFO:teuthology.orchestra.run.smithi159.stdout:(119/153): python3-rsa-4.9-2.el8.noarch.rpm 854 kB/s | 64 kB 00:00 2024-03-29T09:51:17.041 INFO:teuthology.orchestra.run.smithi159.stdout:(120/153): python3-simplegeneric-0.8.1-17.el8.n 278 kB/s | 19 kB 00:00 2024-03-29T09:51:17.058 INFO:teuthology.orchestra.run.smithi159.stdout:(121/153): python3-singledispatch-3.4.0.3-18.el 406 kB/s | 24 kB 00:00 2024-03-29T09:51:17.084 INFO:teuthology.orchestra.run.smithi159.stdout:(122/153): python3-tempora-1.14.1-5.el8.noarch. 497 kB/s | 29 kB 00:00 2024-03-29T09:51:17.109 INFO:teuthology.orchestra.run.smithi159.stdout:(123/153): python3-trustme-0.6.0-4.el8.noarch.r 411 kB/s | 27 kB 00:00 2024-03-29T09:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout:(124/153): python3-waitress-1.4.3-1.el8.noarch. 3.0 MB/s | 255 kB 00:00 2024-03-29T09:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout:(125/153): python3-websocket-client-0.56.0-5.el 1.0 MB/s | 61 kB 00:00 2024-03-29T09:51:17.193 INFO:teuthology.orchestra.run.smithi159.stdout:(126/153): python3-webob-1.8.5-1.el8.1.noarch.r 2.3 MB/s | 251 kB 00:00 2024-03-29T09:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout:(127/153): python3-webtest-2.0.33-1.el8.noarch. 1.2 MB/s | 85 kB 00:00 2024-03-29T09:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout:(128/153): python3-zc-lockfile-2.0-2.el8.noarch 341 kB/s | 23 kB 00:00 2024-03-29T09:51:17.352 INFO:teuthology.orchestra.run.smithi159.stdout:(129/153): thrift-0.13.0-2.el8.x86_64.rpm 11 MB/s | 1.7 MB 00:00 2024-03-29T09:51:17.536 INFO:teuthology.orchestra.run.smithi159.stdout:(130/153): NetworkManager-1.40.16-15.el8.x86_64 12 MB/s | 2.3 MB 00:00 2024-03-29T09:51:17.703 INFO:teuthology.orchestra.run.smithi159.stdout:(131/153): NetworkManager-libnm-1.40.16-15.el8. 11 MB/s | 1.9 MB 00:00 2024-03-29T09:51:17.716 INFO:teuthology.orchestra.run.smithi012.stdout:(10/153): ceph-radosgw-18.2.2-1015.gd0e392d9.el 5.3 MB/s | 13 MB 00:02 2024-03-29T09:51:17.761 INFO:teuthology.orchestra.run.smithi159.stdout:(132/153): NetworkManager-team-1.40.16-15.el8.x 2.7 MB/s | 161 kB 00:00 2024-03-29T09:51:18.253 INFO:teuthology.orchestra.run.smithi159.stdout:(133/153): NetworkManager-tui-1.40.16-15.el8.x8 724 kB/s | 356 kB 00:00 2024-03-29T09:51:18.312 INFO:teuthology.orchestra.run.smithi159.stdout:(134/153): librados2-18.2.2-1015.gd0e392d9.el8. 3.3 MB/s | 3.7 MB 00:01 2024-03-29T09:51:18.337 INFO:teuthology.orchestra.run.smithi159.stdout:(135/153): acl-2.2.53-3.el8.x86_64.rpm 995 kB/s | 83 kB 00:00 2024-03-29T09:51:18.362 INFO:teuthology.orchestra.run.smithi159.stdout:(136/153): elfutils-debuginfod-client-0.190-2.e 1.5 MB/s | 77 kB 00:00 2024-03-29T09:51:18.392 INFO:teuthology.orchestra.run.smithi012.stdout:(11/153): cephfs-mirror-18.2.2-1015.gd0e392d9.e 387 kB/s | 261 kB 00:00 2024-03-29T09:51:18.396 INFO:teuthology.orchestra.run.smithi159.stdout:(137/153): elfutils-libelf-0.190-2.el8.x86_64.r 3.9 MB/s | 233 kB 00:00 2024-03-29T09:51:18.446 INFO:teuthology.orchestra.run.smithi159.stdout:(138/153): elfutils-libs-0.190-2.el8.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-03-29T09:51:18.477 INFO:teuthology.orchestra.run.smithi012.stdout:(12/153): libcephfs-devel-18.2.2-1015.gd0e392d9 363 kB/s | 31 kB 00:00 2024-03-29T09:51:18.513 INFO:teuthology.orchestra.run.smithi159.stdout:(139/153): iproute-6.2.0-5.el8.x86_64.rpm 7.1 MB/s | 854 kB 00:00 2024-03-29T09:51:18.588 INFO:teuthology.orchestra.run.smithi159.stdout:(140/153): libacl-2.2.53-3.el8.x86_64.rpm 246 kB/s | 35 kB 00:00 2024-03-29T09:51:18.728 INFO:teuthology.orchestra.run.smithi012.stdout:(13/153): libcephfs2-18.2.2-1015.gd0e392d9.el8. 3.0 MB/s | 778 kB 00:00 2024-03-29T09:51:18.739 INFO:teuthology.orchestra.run.smithi159.stdout:(141/153): libblkid-2.32.1-46.el8.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-03-29T09:51:18.781 INFO:teuthology.orchestra.run.smithi159.stdout:(142/153): libfdisk-2.32.1-46.el8.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-29T09:51:18.820 INFO:teuthology.orchestra.run.smithi012.stdout:(14/153): libcephsqlite-18.2.2-1015.gd0e392d9.e 1.5 MB/s | 142 kB 00:00 2024-03-29T09:51:18.864 INFO:teuthology.orchestra.run.smithi159.stdout:(143/153): librbd1-18.2.2-1015.gd0e392d9.el8.x8 2.2 MB/s | 3.6 MB 00:01 2024-03-29T09:51:18.898 INFO:teuthology.orchestra.run.smithi159.stdout:(144/153): libmount-2.32.1-46.el8.x86_64.rpm 2.0 MB/s | 237 kB 00:00 2024-03-29T09:51:18.903 INFO:teuthology.orchestra.run.smithi012.stdout:(15/153): librados-devel-18.2.2-1015.gd0e392d9. 1.6 MB/s | 139 kB 00:00 2024-03-29T09:51:18.915 INFO:teuthology.orchestra.run.smithi159.stdout:(145/153): libsmartcols-2.32.1-46.el8.x86_64.rp 3.5 MB/s | 180 kB 00:00 2024-03-29T09:51:18.949 INFO:teuthology.orchestra.run.smithi159.stdout:(146/153): libuuid-2.32.1-46.el8.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-03-29T09:51:18.990 INFO:teuthology.orchestra.run.smithi159.stdout:(147/153): openssl-1.1.1k-12.el8.x86_64.rpm 9.6 MB/s | 738 kB 00:00 2024-03-29T09:51:19.070 INFO:teuthology.orchestra.run.smithi012.stdout:(16/153): libradosstriper1-18.2.2-1015.gd0e392d 2.8 MB/s | 474 kB 00:00 2024-03-29T09:51:19.141 INFO:teuthology.orchestra.run.smithi159.stdout:(148/153): libacl-devel-2.2.53-3.el8.x86_64.rpm 135 kB/s | 85 kB 00:00 2024-03-29T09:51:19.191 INFO:teuthology.orchestra.run.smithi159.stdout:(149/153): util-linux-2.32.1-46.el8.x86_64.rpm 13 MB/s | 2.6 MB 00:00 2024-03-29T09:51:19.216 INFO:teuthology.orchestra.run.smithi159.stdout:(150/153): util-linux-user-2.32.1-46.el8.x86_64 1.3 MB/s | 103 kB 00:00 2024-03-29T09:51:19.242 INFO:teuthology.orchestra.run.smithi159.stdout:(151/153): xfsdump-3.1.8-7.el8.x86_64.rpm 6.5 MB/s | 334 kB 00:00 2024-03-29T09:51:19.283 INFO:teuthology.orchestra.run.smithi159.stdout:(152/153): openssl-libs-1.1.1k-12.el8.x86_64.rp 4.4 MB/s | 1.5 MB 00:00 2024-03-29T09:51:19.334 INFO:teuthology.orchestra.run.smithi159.stdout:(153/153): xfsprogs-5.0.0-12.el8.x86_64.rpm 9.3 MB/s | 1.1 MB 00:00 2024-03-29T09:51:19.337 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-29T09:51:19.337 INFO:teuthology.orchestra.run.smithi159.stdout:Total 13 MB/s | 220 MB 00:17 2024-03-29T09:51:20.579 INFO:teuthology.orchestra.run.smithi012.stdout:(17/153): librgw2-18.2.2-1015.gd0e392d9.el8.x86 4.3 MB/s | 6.5 MB 00:01 2024-03-29T09:51:20.704 INFO:teuthology.orchestra.run.smithi012.stdout:(18/153): python3-ceph-argparse-18.2.2-1015.gd0 358 kB/s | 45 kB 00:00 2024-03-29T09:51:20.821 INFO:teuthology.orchestra.run.smithi012.stdout:(19/153): python3-ceph-common-18.2.2-1015.gd0e3 1.1 MB/s | 129 kB 00:00 2024-03-29T09:51:20.913 INFO:teuthology.orchestra.run.smithi012.stdout:(20/153): python3-cephfs-18.2.2-1015.gd0e392d9. 2.1 MB/s | 197 kB 00:00 2024-03-29T09:51:21.013 INFO:teuthology.orchestra.run.smithi012.stdout:(21/153): python3-rados-18.2.2-1015.gd0e392d9.e 3.6 MB/s | 371 kB 00:00 2024-03-29T09:51:21.113 INFO:teuthology.orchestra.run.smithi012.stdout:(22/153): python3-rbd-18.2.2-1015.gd0e392d9.el8 3.6 MB/s | 364 kB 00:00 2024-03-29T09:51:21.214 INFO:teuthology.orchestra.run.smithi012.stdout:(23/153): python3-rgw-18.2.2-1015.gd0e392d9.el8 1.0 MB/s | 106 kB 00:00 2024-03-29T09:51:21.306 INFO:teuthology.orchestra.run.smithi012.stdout:(24/153): rbd-fuse-18.2.2-1015.gd0e392d9.el8.x8 1.0 MB/s | 95 kB 00:00 2024-03-29T09:51:21.334 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-29T09:51:21.714 INFO:teuthology.orchestra.run.smithi012.stdout:(25/153): ceph-common-18.2.2-1015.gd0e392d9.el8 1.7 MB/s | 22 MB 00:12 2024-03-29T09:51:21.804 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-29T09:51:21.804 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-29T09:51:21.848 INFO:teuthology.orchestra.run.smithi012.stdout:(26/153): rbd-nbd-18.2.2-1015.gd0e392d9.el8.x86 1.3 MB/s | 181 kB 00:00 2024-03-29T09:51:21.940 INFO:teuthology.orchestra.run.smithi012.stdout:(27/153): ceph-grafana-dashboards-18.2.2-1015.g 254 kB/s | 23 kB 00:00 2024-03-29T09:51:22.024 INFO:teuthology.orchestra.run.smithi012.stdout:(28/153): ceph-mgr-cephadm-18.2.2-1015.gd0e392d 1.6 MB/s | 138 kB 00:00 2024-03-29T09:51:22.082 INFO:teuthology.orchestra.run.smithi012.stdout:(29/153): rbd-mirror-18.2.2-1015.gd0e392d9.el8. 4.5 MB/s | 3.5 MB 00:00 2024-03-29T09:51:22.690 INFO:teuthology.orchestra.run.smithi012.stdout:(30/153): ceph-mgr-dashboard-18.2.2-1015.gd0e39 2.7 MB/s | 1.8 MB 00:00 2024-03-29T09:51:22.816 INFO:teuthology.orchestra.run.smithi012.stdout:(31/153): ceph-mgr-modules-core-18.2.2-1015.gd0 1.9 MB/s | 245 kB 00:00 2024-03-29T09:51:22.883 INFO:teuthology.orchestra.run.smithi012.stdout:(32/153): ceph-mgr-rook-18.2.2-1015.gd0e392d9.e 738 kB/s | 49 kB 00:00 2024-03-29T09:51:22.950 INFO:teuthology.orchestra.run.smithi012.stdout:(33/153): ceph-prometheus-alerts-18.2.2-1015.gd 222 kB/s | 15 kB 00:00 2024-03-29T09:51:23.067 INFO:teuthology.orchestra.run.smithi012.stdout:(34/153): ceph-volume-18.2.2-1015.gd0e392d9.el8 2.2 MB/s | 263 kB 00:00 2024-03-29T09:51:23.133 INFO:teuthology.orchestra.run.smithi012.stdout:(35/153): cephadm-18.2.2-1015.gd0e392d9.el8.noa 2.5 MB/s | 173 kB 00:00 2024-03-29T09:51:23.201 INFO:teuthology.orchestra.run.smithi012.stdout:(36/153): cephfs-top-18.2.2-1015.gd0e392d9.el8. 271 kB/s | 18 kB 00:00 2024-03-29T09:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-29T09:51:23.386 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-29T09:51:23.393 INFO:teuthology.orchestra.run.smithi012.stdout:(37/153): ceph-mgr-diskprediction-local-18.2.2- 5.6 MB/s | 7.4 MB 00:01 2024-03-29T09:51:23.859 INFO:teuthology.orchestra.run.smithi012.stdout:(38/153): automake-1.16.1-8.el8.noarch.rpm 1.1 MB/s | 740 kB 00:00 2024-03-29T09:51:23.985 INFO:teuthology.orchestra.run.smithi012.stdout:(39/153): flex-2.6.1-9.el8.x86_64.rpm 2.5 MB/s | 320 kB 00:00 2024-03-29T09:51:24.052 INFO:teuthology.orchestra.run.smithi012.stdout:(40/153): bison-3.0.4-10.el8.x86_64.rpm 1.0 MB/s | 688 kB 00:00 2024-03-29T09:51:24.102 INFO:teuthology.orchestra.run.smithi012.stdout:(41/153): jq-1.6-8.el8.x86_64.rpm 1.7 MB/s | 203 kB 00:00 2024-03-29T09:51:24.161 INFO:teuthology.orchestra.run.smithi012.stdout:(42/153): libpq-13.5-1.el8.x86_64.rpm 1.8 MB/s | 198 kB 00:00 2024-03-29T09:51:24.253 INFO:teuthology.orchestra.run.smithi012.stdout:(43/153): librdkafka-1.6.1-1.el8.x86_64.rpm 4.4 MB/s | 677 kB 00:00 2024-03-29T09:51:24.320 INFO:teuthology.orchestra.run.smithi012.stdout:(44/153): libtool-2.4.6-25.el8.x86_64.rpm 4.4 MB/s | 709 kB 00:00 2024-03-29T09:51:24.370 INFO:teuthology.orchestra.run.smithi012.stdout:(45/153): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.2 MB/s | 259 kB 00:00 2024-03-29T09:51:24.428 INFO:teuthology.orchestra.run.smithi012.stdout:(46/153): oniguruma-6.8.2-3.el8.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-03-29T09:51:24.729 INFO:teuthology.orchestra.run.smithi012.stdout:(47/153): openblas-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-03-29T09:51:24.812 INFO:teuthology.orchestra.run.smithi012.stdout:(48/153): openblas-threads-0.3.15-4.el8.x86_64. 13 MB/s | 4.9 MB 00:00 2024-03-29T09:51:24.854 INFO:teuthology.orchestra.run.smithi012.stdout:(49/153): postgresql-10.23-2.module_el8+572+929 13 MB/s | 1.6 MB 00:00 2024-03-29T09:51:24.930 INFO:teuthology.orchestra.run.smithi012.stdout:(50/153): postgresql-contrib-10.23-2.module_el8 7.5 MB/s | 901 kB 00:00 2024-03-29T09:51:25.114 INFO:teuthology.orchestra.run.smithi012.stdout:(51/153): postgresql-server-10.23-2.module_el8+ 20 MB/s | 5.1 MB 00:00 2024-03-29T09:51:25.197 INFO:teuthology.orchestra.run.smithi012.stdout:(52/153): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T09:51:25.230 INFO:teuthology.orchestra.run.smithi012.stdout:(53/153): python3-jinja2-2.10.1-4.el8.noarch.rp 4.8 MB/s | 578 kB 00:00 2024-03-29T09:51:25.298 INFO:teuthology.orchestra.run.smithi012.stdout:(54/153): python3-jsonpatch-1.21-2.el8.noarch.r 265 kB/s | 27 kB 00:00 2024-03-29T09:51:25.323 INFO:teuthology.orchestra.run.smithi012.stdout:(55/153): python3-jsonpointer-1.10-11.el8.noarc 216 kB/s | 20 kB 00:00 2024-03-29T09:51:25.719 INFO:teuthology.orchestra.run.smithi012.stdout:(56/153): python3-mako-1.0.6-14.el8.noarch.rpm 372 kB/s | 157 kB 00:00 2024-03-29T09:51:25.774 INFO:teuthology.orchestra.run.smithi012.stdout:(57/153): python3-markupsafe-0.23-19.el8.x86_64 86 kB/s | 39 kB 00:00 2024-03-29T09:51:25.908 INFO:teuthology.orchestra.run.smithi012.stdout:(58/153): python3-numpy-1.14.3-10.el8.x86_64.rp 20 MB/s | 3.7 MB 00:00 2024-03-29T09:51:25.924 INFO:teuthology.orchestra.run.smithi012.stdout:(59/153): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.5 MB/s | 225 kB 00:00 2024-03-29T09:51:26.000 INFO:teuthology.orchestra.run.smithi012.stdout:(60/153): python3-prettytable-0.7.2-14.el8.noar 480 kB/s | 44 kB 00:00 2024-03-29T09:51:26.025 INFO:teuthology.orchestra.run.smithi012.stdout:(61/153): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.0 MB/s | 103 kB 00:00 2024-03-29T09:51:26.100 INFO:teuthology.orchestra.run.smithi012.stdout:(62/153): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-03-29T09:51:26.125 INFO:teuthology.orchestra.run.smithi012.stdout:(63/153): python3-pyasn1-modules-0.3.7-6.el8.no 1.1 MB/s | 110 kB 00:00 2024-03-29T09:51:26.193 INFO:teuthology.orchestra.run.smithi012.stdout:(64/153): python3-pytz-2017.2-11.el8.noarch.rpm 620 kB/s | 57 kB 00:00 2024-03-29T09:51:26.301 INFO:teuthology.orchestra.run.smithi012.stdout:(65/153): python3-werkzeug-0.12.2-4.el8.noarch. 4.1 MB/s | 457 kB 00:00 2024-03-29T09:51:26.402 INFO:teuthology.orchestra.run.smithi012.stdout:(66/153): socat-1.7.4.1-1.el8.x86_64.rpm 3.2 MB/s | 323 kB 00:00 2024-03-29T09:51:26.502 INFO:teuthology.orchestra.run.smithi012.stdout:(67/153): uuid-1.6.2-43.el8.x86_64.rpm 642 kB/s | 64 kB 00:00 2024-03-29T09:51:26.694 INFO:teuthology.orchestra.run.smithi012.stdout:(68/153): python3-scipy-1.0.0-21.module_el8+762 25 MB/s | 14 MB 00:00 2024-03-29T09:51:26.711 INFO:teuthology.orchestra.run.smithi012.stdout:(69/153): xmlstarlet-1.6.1-20.el8.x86_64.rpm 338 kB/s | 71 kB 00:00 2024-03-29T09:51:26.836 INFO:teuthology.orchestra.run.smithi012.stdout:(70/153): elfutils-libelf-devel-0.190-2.el8.x86 439 kB/s | 62 kB 00:00 2024-03-29T09:51:26.853 INFO:teuthology.orchestra.run.smithi012.stdout:(71/153): fuse-2.9.7-19.el8.x86_64.rpm 587 kB/s | 83 kB 00:00 2024-03-29T09:51:26.887 INFO:teuthology.orchestra.run.smithi012.stdout:(72/153): keyutils-libs-devel-1.5.10-9.el8.x86_ 962 kB/s | 48 kB 00:00 2024-03-29T09:51:26.937 INFO:teuthology.orchestra.run.smithi012.stdout:(73/153): libaio-devel-0.3.112-1.el8.x86_64.rpm 374 kB/s | 19 kB 00:00 2024-03-29T09:51:26.987 INFO:teuthology.orchestra.run.smithi012.stdout:(74/153): krb5-devel-1.18.2-22.el8.x86_64.rpm 4.1 MB/s | 561 kB 00:00 2024-03-29T09:51:27.012 INFO:teuthology.orchestra.run.smithi012.stdout:(75/153): libcom_err-devel-1.45.6-5.el8.x86_64. 516 kB/s | 39 kB 00:00 2024-03-29T09:51:27.063 INFO:teuthology.orchestra.run.smithi012.stdout:(76/153): libgfortran-8.5.0-21.el8.x86_64.rpm 8.4 MB/s | 645 kB 00:00 2024-03-29T09:51:27.105 INFO:teuthology.orchestra.run.smithi012.stdout:(77/153): libquadmath-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 172 kB 00:00 2024-03-29T09:51:27.130 INFO:teuthology.orchestra.run.smithi012.stdout:(78/153): librabbitmq-0.9.0-5.el8.x86_64.rpm 707 kB/s | 47 kB 00:00 2024-03-29T09:51:27.164 INFO:teuthology.orchestra.run.smithi012.stdout:(79/153): libselinux-devel-2.9-8.el8.x86_64.rpm 3.3 MB/s | 200 kB 00:00 2024-03-29T09:51:27.181 INFO:teuthology.orchestra.run.smithi012.stdout:(80/153): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-03-29T09:51:27.206 INFO:teuthology.orchestra.run.smithi012.stdout:(81/153): libuuid-devel-2.32.1-46.el8.x86_64.rp 2.4 MB/s | 100 kB 00:00 2024-03-29T09:51:27.231 INFO:teuthology.orchestra.run.smithi012.stdout:(82/153): libverto-devel-0.3.2-2.el8.x86_64.rpm 360 kB/s | 18 kB 00:00 2024-03-29T09:51:27.256 INFO:teuthology.orchestra.run.smithi012.stdout:(83/153): libzstd-devel-1.4.4-1.el8.x86_64.rpm 875 kB/s | 44 kB 00:00 2024-03-29T09:51:27.340 INFO:teuthology.orchestra.run.smithi012.stdout:(84/153): pcre2-devel-10.32-3.el8.x86_64.rpm 7.1 MB/s | 605 kB 00:00 2024-03-29T09:51:27.390 INFO:teuthology.orchestra.run.smithi012.stdout:(85/153): pcre2-utf16-10.32-3.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2024-03-29T09:51:27.441 INFO:teuthology.orchestra.run.smithi012.stdout:(86/153): openssl-devel-1.1.1k-12.el8.x86_64.rp 15 MB/s | 3.2 MB 00:00 2024-03-29T09:51:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:(87/153): pcre2-utf32-10.32-3.el8.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T09:51:27.500 INFO:teuthology.orchestra.run.smithi012.stdout:(88/153): python3-cffi-1.11.5-6.el8.x86_64.rpm 4.2 MB/s | 249 kB 00:00 2024-03-29T09:51:27.533 INFO:teuthology.orchestra.run.smithi012.stdout:(89/153): python3-cryptography-3.2.1-7.el8.x86_ 8.3 MB/s | 641 kB 00:00 2024-03-29T09:51:27.558 INFO:teuthology.orchestra.run.smithi012.stdout:(90/153): python3-jwt-1.6.1-2.el8.noarch.rpm 730 kB/s | 43 kB 00:00 2024-03-29T09:51:27.575 INFO:teuthology.orchestra.run.smithi012.stdout:(91/153): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2024-03-29T09:51:27.601 INFO:teuthology.orchestra.run.smithi012.stdout:(92/153): python3-pycparser-2.14-14.el8.noarch. 2.5 MB/s | 109 kB 00:00 2024-03-29T09:51:27.618 INFO:teuthology.orchestra.run.smithi012.stdout:(93/153): python3-requests-oauthlib-1.0.0-1.el8 1.0 MB/s | 43 kB 00:00 2024-03-29T09:51:27.651 INFO:teuthology.orchestra.run.smithi012.stdout:(94/153): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 1.4 MB/s | 74 kB 00:00 2024-03-29T09:51:27.668 INFO:teuthology.orchestra.run.smithi012.stdout:(95/153): zlib-devel-1.2.11-21.el8.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-03-29T09:51:27.860 INFO:teuthology.orchestra.run.smithi012.stdout:(96/153): python3-packaging-16.8-9.el8.noarch.r 231 kB/s | 48 kB 00:00 2024-03-29T09:51:28.494 INFO:teuthology.orchestra.run.smithi012.stdout:(97/153): ceph-test-18.2.2-1015.gd0e392d9.el8.x 3.6 MB/s | 47 MB 00:13 2024-03-29T09:51:28.511 INFO:teuthology.orchestra.run.smithi012.stdout:(98/153): liboath-2.6.2-3.el8.x86_64.rpm 90 kB/s | 59 kB 00:00 2024-03-29T09:51:28.544 INFO:teuthology.orchestra.run.smithi012.stdout:(99/153): gperftools-libs-2.7-9.el8.x86_64.rpm 350 kB/s | 306 kB 00:00 2024-03-29T09:51:28.662 INFO:teuthology.orchestra.run.smithi012.stdout:(100/153): python3-bcrypt-3.1.6-2.el8.1.x86_64. 374 kB/s | 44 kB 00:00 2024-03-29T09:51:28.803 INFO:teuthology.orchestra.run.smithi012.stdout:(101/153): python3-beautifulsoup4-4.6.3-2.el8.1 1.3 MB/s | 185 kB 00:00 2024-03-29T09:51:28.895 INFO:teuthology.orchestra.run.smithi012.stdout:(102/153): python3-cachetools-3.1.1-4.el8.noarc 360 kB/s | 33 kB 00:00 2024-03-29T09:51:28.912 INFO:teuthology.orchestra.run.smithi012.stdout:(103/153): libunwind-1.3.1-3.el8.x86_64.rpm 180 kB/s | 75 kB 00:00 2024-03-29T09:51:28.937 INFO:teuthology.orchestra.run.smithi012.stdout:(104/153): python3-asyncssh-2.7.0-2.el8.noarch. 1.1 MB/s | 476 kB 00:00 2024-03-29T09:51:29.005 INFO:teuthology.orchestra.run.smithi012.stdout:(105/153): python3-certifi-2018.10.15-7.el8.noa 149 kB/s | 16 kB 00:00 2024-03-29T09:51:29.096 INFO:teuthology.orchestra.run.smithi012.stdout:(106/153): python3-cherrypy-18.4.0-1.el8.noarch 2.4 MB/s | 384 kB 00:00 2024-03-29T09:51:29.130 INFO:teuthology.orchestra.run.smithi012.stdout:(107/153): python3-google-auth-1.1.1-10.el8.noa 649 kB/s | 81 kB 00:00 2024-03-29T09:51:29.200 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-29T09:51:29.214 INFO:teuthology.orchestra.run.smithi012.stdout:(108/153): python3-cheroot-8.5.2-1.el8.noarch.r 573 kB/s | 173 kB 00:00 2024-03-29T09:51:29.239 INFO:teuthology.orchestra.run.smithi012.stdout:(109/153): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2024-03-29T09:51:29.264 INFO:teuthology.orchestra.run.smithi012.stdout:(110/153): python3-jaraco-functools-2.0-4.el8.n 131 kB/s | 18 kB 00:00 2024-03-29T09:51:29.356 INFO:teuthology.orchestra.run.smithi012.stdout:(111/153): python3-logutils-0.3.5-11.el8.noarch 418 kB/s | 49 kB 00:00 2024-03-29T09:51:29.382 INFO:teuthology.orchestra.run.smithi012.stdout:(112/153): python3-more-itertools-7.2.0-3.el8.n 501 kB/s | 59 kB 00:00 2024-03-29T09:51:29.406 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/1 2024-03-29T09:51:29.440 INFO:teuthology.orchestra.run.smithi012.stdout:(113/153): python3-kubernetes-11.0.0-6.el8.noar 5.2 MB/s | 1.2 MB 00:00 2024-03-29T09:51:29.474 INFO:teuthology.orchestra.run.smithi012.stdout:(114/153): python3-natsort-7.1.1-2.el8.noarch.r 515 kB/s | 60 kB 00:00 2024-03-29T09:51:29.533 INFO:teuthology.orchestra.run.smithi012.stdout:(115/153): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2024-03-29T09:51:29.558 INFO:teuthology.orchestra.run.smithi012.stdout:(116/153): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2024-03-29T09:51:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:(117/153): python3-repoze-lru-0.7-6.el8.noarch. 309 kB/s | 34 kB 00:00 2024-03-29T09:51:29.633 INFO:teuthology.orchestra.run.smithi012.stdout:(118/153): python3-routes-2.4.1-12.el8.noarch.r 1.9 MB/s | 196 kB 00:00 2024-03-29T09:51:29.634 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-03-29T09:51:29.658 INFO:teuthology.orchestra.run.smithi012.stdout:(119/153): python3-rsa-4.9-2.el8.noarch.rpm 641 kB/s | 64 kB 00:00 2024-03-29T09:51:29.692 INFO:teuthology.orchestra.run.smithi012.stdout:(120/153): python3-simplegeneric-0.8.1-17.el8.n 172 kB/s | 19 kB 00:00 2024-03-29T09:51:29.726 INFO:teuthology.orchestra.run.smithi012.stdout:(121/153): python3-singledispatch-3.4.0.3-18.el 258 kB/s | 24 kB 00:00 2024-03-29T09:51:29.768 INFO:teuthology.orchestra.run.smithi012.stdout:(122/153): python3-tempora-1.14.1-5.el8.noarch. 268 kB/s | 29 kB 00:00 2024-03-29T09:51:29.793 INFO:teuthology.orchestra.run.smithi012.stdout:(123/153): python3-trustme-0.6.0-4.el8.noarch.r 273 kB/s | 27 kB 00:00 2024-03-29T09:51:29.835 INFO:teuthology.orchestra.run.smithi012.stdout:(124/153): python3-waitress-1.4.3-1.el8.noarch. 2.3 MB/s | 255 kB 00:00 2024-03-29T09:51:29.872 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-03-29T09:51:29.911 INFO:teuthology.orchestra.run.smithi012.stdout:(125/153): python3-websocket-client-0.56.0-5.el 524 kB/s | 61 kB 00:00 2024-03-29T09:51:29.936 INFO:teuthology.orchestra.run.smithi012.stdout:(126/153): python3-webob-1.8.5-1.el8.1.noarch.r 1.5 MB/s | 251 kB 00:00 2024-03-29T09:51:29.961 INFO:teuthology.orchestra.run.smithi012.stdout:(127/153): python3-webtest-2.0.33-1.el8.noarch. 680 kB/s | 85 kB 00:00 2024-03-29T09:51:30.011 INFO:teuthology.orchestra.run.smithi012.stdout:(128/153): python3-zc-lockfile-2.0-2.el8.noarch 229 kB/s | 23 kB 00:00 2024-03-29T09:51:30.046 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/177 2024-03-29T09:51:30.171 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libuuid-2.32.1-46.el8.x86_64 3/177 2024-03-29T09:51:30.280 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libuuid-2.32.1-46.el8.x86_64 3/177 2024-03-29T09:51:30.311 INFO:teuthology.orchestra.run.smithi012.stdout:(129/153): thrift-0.13.0-2.el8.x86_64.rpm 4.6 MB/s | 1.7 MB 00:00 2024-03-29T09:51:30.454 INFO:teuthology.orchestra.run.smithi012.stdout:(130/153): librados2-18.2.2-1015.gd0e392d9.el8. 7.5 MB/s | 3.7 MB 00:00 2024-03-29T09:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libblkid-2.32.1-46.el8.x86_64 4/177 2024-03-29T09:51:30.561 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libblkid-2.32.1-46.el8.x86_64 4/177 2024-03-29T09:51:30.611 INFO:teuthology.orchestra.run.smithi012.stdout:(131/153): NetworkManager-1.40.16-15.el8.x86_64 7.6 MB/s | 2.3 MB 00:00 2024-03-29T09:51:30.688 INFO:teuthology.orchestra.run.smithi012.stdout:(132/153): librbd1-18.2.2-1015.gd0e392d9.el8.x8 5.3 MB/s | 3.6 MB 00:00 2024-03-29T09:51:30.729 INFO:teuthology.orchestra.run.smithi012.stdout:(133/153): NetworkManager-libnm-1.40.16-15.el8. 6.9 MB/s | 1.9 MB 00:00 2024-03-29T09:51:30.755 INFO:teuthology.orchestra.run.smithi012.stdout:(134/153): NetworkManager-team-1.40.16-15.el8.x 1.1 MB/s | 161 kB 00:00 2024-03-29T09:51:30.780 INFO:teuthology.orchestra.run.smithi012.stdout:(135/153): acl-2.2.53-3.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2024-03-29T09:51:30.785 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/177 2024-03-29T09:51:30.805 INFO:teuthology.orchestra.run.smithi012.stdout:(136/153): elfutils-debuginfod-client-0.190-2.e 1.5 MB/s | 77 kB 00:00 2024-03-29T09:51:30.830 INFO:teuthology.orchestra.run.smithi012.stdout:(137/153): elfutils-libelf-0.190-2.el8.x86_64.r 4.6 MB/s | 233 kB 00:00 2024-03-29T09:51:30.856 INFO:teuthology.orchestra.run.smithi012.stdout:(138/153): elfutils-libs-0.190-2.el8.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-03-29T09:51:30.906 INFO:teuthology.orchestra.run.smithi012.stdout:(139/153): iproute-6.2.0-5.el8.x86_64.rpm 11 MB/s | 854 kB 00:00 2024-03-29T09:51:30.931 INFO:teuthology.orchestra.run.smithi012.stdout:(140/153): libacl-2.2.53-3.el8.x86_64.rpm 462 kB/s | 35 kB 00:00 2024-03-29T09:51:30.965 INFO:teuthology.orchestra.run.smithi012.stdout:(141/153): NetworkManager-tui-1.40.16-15.el8.x8 1.3 MB/s | 356 kB 00:00 2024-03-29T09:51:30.990 INFO:teuthology.orchestra.run.smithi012.stdout:(142/153): libacl-devel-2.2.53-3.el8.x86_64.rpm 1.0 MB/s | 85 kB 00:00 2024-03-29T09:51:31.015 INFO:teuthology.orchestra.run.smithi012.stdout:(143/153): libblkid-2.32.1-46.el8.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-03-29T09:51:31.027 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 6/177 2024-03-29T09:51:31.040 INFO:teuthology.orchestra.run.smithi012.stdout:(144/153): libfdisk-2.32.1-46.el8.x86_64.rpm 3.3 MB/s | 254 kB 00:00 2024-03-29T09:51:31.066 INFO:teuthology.orchestra.run.smithi012.stdout:(145/153): libmount-2.32.1-46.el8.x86_64.rpm 3.1 MB/s | 237 kB 00:00 2024-03-29T09:51:31.091 INFO:teuthology.orchestra.run.smithi012.stdout:(146/153): libsmartcols-2.32.1-46.el8.x86_64.rp 2.3 MB/s | 180 kB 00:00 2024-03-29T09:51:31.116 INFO:teuthology.orchestra.run.smithi012.stdout:(147/153): libuuid-2.32.1-46.el8.x86_64.rpm 1.3 MB/s | 99 kB 00:00 2024-03-29T09:51:31.183 INFO:teuthology.orchestra.run.smithi012.stdout:(148/153): openssl-1.1.1k-12.el8.x86_64.rpm 6.2 MB/s | 738 kB 00:00 2024-03-29T09:51:31.242 INFO:teuthology.orchestra.run.smithi012.stdout:(149/153): openssl-libs-1.1.1k-12.el8.x86_64.rp 9.8 MB/s | 1.5 MB 00:00 2024-03-29T09:51:31.267 INFO:teuthology.orchestra.run.smithi012.stdout:(150/153): util-linux-user-2.32.1-46.el8.x86_64 1.2 MB/s | 103 kB 00:00 2024-03-29T09:51:31.309 INFO:teuthology.orchestra.run.smithi012.stdout:(151/153): xfsdump-3.1.8-7.el8.x86_64.rpm 4.9 MB/s | 334 kB 00:00 2024-03-29T09:51:31.368 INFO:teuthology.orchestra.run.smithi012.stdout:(152/153): util-linux-2.32.1-46.el8.x86_64.rpm 10 MB/s | 2.6 MB 00:00 2024-03-29T09:51:31.400 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/177 2024-03-29T09:51:31.401 INFO:teuthology.orchestra.run.smithi012.stdout:(153/153): xfsprogs-5.0.0-12.el8.x86_64.rpm 8.1 MB/s | 1.1 MB 00:00 2024-03-29T09:51:31.404 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-29T09:51:31.404 INFO:teuthology.orchestra.run.smithi012.stdout:Total 9.4 MB/s | 220 MB 00:23 2024-03-29T09:51:31.693 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : elfutils-libelf-0.190-2.el8.x86_64 8/177 2024-03-29T09:51:31.895 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-03-29T09:51:32.342 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-03-29T09:51:32.512 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-03-29T09:51:32.932 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-03-29T09:51:33.129 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-03-29T09:51:33.279 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-03-29T09:51:33.420 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-29T09:51:33.503 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-03-29T09:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-03-29T09:51:33.914 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-29T09:51:33.914 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-29T09:51:33.927 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 13/177 2024-03-29T09:51:34.162 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-03-29T09:51:34.246 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-03-29T09:51:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libpq-13.5-1.el8.x86_64 15/177 2024-03-29T09:51:34.693 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 16/177 2024-03-29T09:51:35.155 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/177 2024-03-29T09:51:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 18/177 2024-03-29T09:51:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-03-29T09:51:35.570 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-29T09:51:35.574 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-29T09:51:35.672 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-03-29T09:51:35.878 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 20/177 2024-03-29T09:51:35.956 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 20/177 2024-03-29T09:51:36.494 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libacl-2.2.53-3.el8.x86_64 21/177 2024-03-29T09:51:36.720 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-03-29T09:51:36.799 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-03-29T09:51:37.080 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-29T09:51:37.334 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-29T09:51:37.806 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-29T09:51:38.036 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/177 2024-03-29T09:51:38.227 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 25/177 2024-03-29T09:51:38.386 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 26/177 2024-03-29T09:51:38.539 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 26/177 2024-03-29T09:51:38.728 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 27/177 2024-03-29T09:51:39.394 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 27/177 2024-03-29T09:51:39.595 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-0.3.15-4.el8.x86_64 28/177 2024-03-29T09:51:40.269 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 28/177 2024-03-29T09:51:40.469 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-03-29T09:51:41.280 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-03-29T09:51:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-29T09:51:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/1 2024-03-29T09:51:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 30/177 2024-03-29T09:51:42.186 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 31/177 2024-03-29T09:51:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-03-29T09:51:42.368 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 32/177 2024-03-29T09:51:42.469 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-03-29T09:51:42.613 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/177 2024-03-29T09:51:42.663 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/177 2024-03-29T09:51:42.806 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libuuid-2.32.1-46.el8.x86_64 3/177 2024-03-29T09:51:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8. 34/177 2024-03-29T09:51:42.905 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libuuid-2.32.1-46.el8.x86_64 3/177 2024-03-29T09:51:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el 35/177 2024-03-29T09:51:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libblkid-2.32.1-46.el8.x86_64 4/177 2024-03-29T09:51:43.217 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_6 36/177 2024-03-29T09:51:43.233 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libblkid-2.32.1-46.el8.x86_64 4/177 2024-03-29T09:51:43.475 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/177 2024-03-29T09:51:43.654 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 6/177 2024-03-29T09:51:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/177 2024-03-29T09:51:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libelf-0.190-2.el8.x86_64 8/177 2024-03-29T09:51:44.277 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 37/177 2024-03-29T09:51:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-03-29T09:51:44.639 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 38/177 2024-03-29T09:51:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-03-29T09:51:44.937 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 39/177 2024-03-29T09:51:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-03-29T09:51:45.169 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-03-29T09:51:45.464 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-03-29T09:51:45.639 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-03-29T09:51:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-03-29T09:51:45.956 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-03-29T09:51:46.074 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-03-29T09:51:46.427 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 13/177 2024-03-29T09:51:46.664 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-03-29T09:51:46.764 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-03-29T09:51:46.949 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpq-13.5-1.el8.x86_64 15/177 2024-03-29T09:51:47.182 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 16/177 2024-03-29T09:51:47.227 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-03-29T09:51:47.522 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 41/177 2024-03-29T09:51:47.597 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/177 2024-03-29T09:51:47.742 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 42/177 2024-03-29T09:51:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 18/177 2024-03-29T09:51:47.949 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rsa-4.9-2.el8.noarch 43/177 2024-03-29T09:51:47.964 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-03-29T09:51:48.091 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-03-29T09:51:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 44/177 2024-03-29T09:51:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 20/177 2024-03-29T09:51:48.330 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 20/177 2024-03-29T09:51:48.363 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_6 45/177 2024-03-29T09:51:48.557 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : elfutils-libs-0.190-2.el8.x86_64 46/177 2024-03-29T09:51:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el8.x86_64 47/177 2024-03-29T09:51:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libacl-2.2.53-3.el8.x86_64 21/177 2024-03-29T09:51:48.985 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libfdisk-2.32.1-46.el8.x86_64 48/177 2024-03-29T09:51:49.047 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-03-29T09:51:49.073 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libfdisk-2.32.1-46.el8.x86_64 48/177 2024-03-29T09:51:49.114 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-03-29T09:51:49.243 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libmount-2.32.1-46.el8.x86_64 49/177 2024-03-29T09:51:49.332 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libmount-2.32.1-46.el8.x86_64 49/177 2024-03-29T09:51:49.408 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-29T09:51:49.504 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libuuid-devel-2.32.1-46.el8.x86_64 50/177 2024-03-29T09:51:49.633 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-29T09:51:49.697 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 51/177 2024-03-29T09:51:49.893 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-03-29T09:51:50.118 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-29T09:51:50.338 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/177 2024-03-29T09:51:50.463 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-03-29T09:51:50.504 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 25/177 2024-03-29T09:51:50.680 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 26/177 2024-03-29T09:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : util-linux-2.32.1-46.el8.x86_64 53/177 2024-03-29T09:51:50.780 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: util-linux-2.32.1-46.el8.x86_64 53/177 2024-03-29T09:51:50.865 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 26/177 2024-03-29T09:51:50.984 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 54/177 2024-03-29T09:51:51.038 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 27/177 2024-03-29T09:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 55/177 2024-03-29T09:51:51.501 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 56/177 2024-03-29T09:51:51.648 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 57/177 2024-03-29T09:51:51.735 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 27/177 2024-03-29T09:51:51.788 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 58/177 2024-03-29T09:51:51.997 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 59/177 2024-03-29T09:51:52.013 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.15-4.el8.x86_64 28/177 2024-03-29T09:51:52.253 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 60/177 2024-03-29T09:51:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 61/177 2024-03-29T09:51:52.621 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 62/177 2024-03-29T09:51:52.731 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 28/177 2024-03-29T09:51:52.824 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 63/177 2024-03-29T09:51:52.938 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-03-29T09:51:52.997 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 64/177 2024-03-29T09:51:53.182 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 65/177 2024-03-29T09:51:53.335 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-portend-2.6-1.el8.noarch 66/177 2024-03-29T09:51:53.510 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 67/177 2024-03-29T09:51:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 68/177 2024-03-29T09:51:53.799 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-03-29T09:51:54.012 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 69/177 2024-03-29T09:51:54.142 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 30/177 2024-03-29T09:51:54.231 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 70/177 2024-03-29T09:51:54.331 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 31/177 2024-03-29T09:51:54.487 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 32/177 2024-03-29T09:51:54.564 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 71/177 2024-03-29T09:51:54.685 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/177 2024-03-29T09:51:54.819 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 72/177 2024-03-29T09:51:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8. 34/177 2024-03-29T09:51:55.018 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el 35/177 2024-03-29T09:51:55.047 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 73/177 2024-03-29T09:51:55.291 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_6 36/177 2024-03-29T09:51:55.869 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 74/177 2024-03-29T09:51:56.059 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 75/177 2024-03-29T09:51:56.153 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 75/177 2024-03-29T09:51:56.373 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 76/177 2024-03-29T09:51:56.430 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 37/177 2024-03-29T09:51:56.627 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/177 2024-03-29T09:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 78/177 2024-03-29T09:51:56.892 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 38/177 2024-03-29T09:51:57.202 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 39/177 2024-03-29T09:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 79/177 2024-03-29T09:51:57.428 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-03-29T09:51:57.515 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 80/177 2024-03-29T09:51:57.724 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 81/177 2024-03-29T09:51:58.012 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 82/177 2024-03-29T09:51:58.234 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 83/177 2024-03-29T09:51:59.510 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 84/177 2024-03-29T09:51:59.610 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-03-29T09:51:59.876 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/177 2024-03-29T09:51:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 41/177 2024-03-29T09:52:00.084 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 86/177 2024-03-29T09:52:00.113 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 42/177 2024-03-29T09:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el8.noarch 43/177 2024-03-29T09:52:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 87/177 2024-03-29T09:52:00.502 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 44/177 2024-03-29T09:52:00.673 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 88/177 2024-03-29T09:52:00.730 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_6 45/177 2024-03-29T09:52:00.929 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libs-0.190-2.el8.x86_64 46/177 2024-03-29T09:52:00.936 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 89/177 2024-03-29T09:52:01.119 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el8.x86_64 47/177 2024-03-29T09:52:01.196 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 90/177 2024-03-29T09:52:01.311 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libfdisk-2.32.1-46.el8.x86_64 48/177 2024-03-29T09:52:01.411 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libfdisk-2.32.1-46.el8.x86_64 48/177 2024-03-29T09:52:01.430 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 91/177 2024-03-29T09:52:01.603 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libmount-2.32.1-46.el8.x86_64 49/177 2024-03-29T09:52:01.688 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libmount-2.32.1-46.el8.x86_64 49/177 2024-03-29T09:52:01.824 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 92/177 2024-03-29T09:52:01.924 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libuuid-devel-2.32.1-46.el8.x86_64 50/177 2024-03-29T09:52:02.064 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 93/177 2024-03-29T09:52:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 51/177 2024-03-29T09:52:02.235 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 94/177 2024-03-29T09:52:02.311 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-03-29T09:52:02.444 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 95/177 2024-03-29T09:52:02.918 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-03-29T09:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 96/177 2024-03-29T09:52:03.196 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 97/177 2024-03-29T09:52:03.203 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-2.32.1-46.el8.x86_64 53/177 2024-03-29T09:52:03.230 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: util-linux-2.32.1-46.el8.x86_64 53/177 2024-03-29T09:52:03.374 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 98/177 2024-03-29T09:52:03.419 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 54/177 2024-03-29T09:52:03.576 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 99/177 2024-03-29T09:52:03.689 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 55/177 2024-03-29T09:52:03.844 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 100/177 2024-03-29T09:52:03.885 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 56/177 2024-03-29T09:52:04.061 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fuse-2.9.7-19.el8.x86_64 101/177 2024-03-29T09:52:04.107 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 57/177 2024-03-29T09:52:04.257 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 58/177 2024-03-29T09:52:04.311 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 102/177 2024-03-29T09:52:04.443 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 59/177 2024-03-29T09:52:04.507 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : uuid-1.6.2-43.el8.x86_64 103/177 2024-03-29T09:52:04.599 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 103/177 2024-03-29T09:52:04.636 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 60/177 2024-03-29T09:52:04.767 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 104/177 2024-03-29T09:52:04.867 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 61/177 2024-03-29T09:52:05.089 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 62/177 2024-03-29T09:52:05.243 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 63/177 2024-03-29T09:52:05.418 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 64/177 2024-03-29T09:52:05.593 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 65/177 2024-03-29T09:52:05.729 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-2.6-1.el8.noarch 66/177 2024-03-29T09:52:05.890 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 67/177 2024-03-29T09:52:06.082 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 68/177 2024-03-29T09:52:06.336 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 69/177 2024-03-29T09:52:06.528 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 70/177 2024-03-29T09:52:06.864 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 71/177 2024-03-29T09:52:07.112 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 72/177 2024-03-29T09:52:07.351 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 73/177 2024-03-29T09:52:07.594 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-03-29T09:52:08.083 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-03-29T09:52:08.190 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 74/177 2024-03-29T09:52:08.426 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 75/177 2024-03-29T09:52:08.507 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 75/177 2024-03-29T09:52:08.708 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 76/177 2024-03-29T09:52:08.762 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-03-29T09:52:08.930 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/177 2024-03-29T09:52:09.045 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 106/177 2024-03-29T09:52:09.139 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 78/177 2024-03-29T09:52:09.253 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 106/177 2024-03-29T09:52:09.473 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 107/177 2024-03-29T09:52:09.541 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 79/177 2024-03-29T09:52:09.800 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 80/177 2024-03-29T09:52:10.004 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 81/177 2024-03-29T09:52:10.282 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 82/177 2024-03-29T09:52:10.570 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 83/177 2024-03-29T09:52:11.873 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 84/177 2024-03-29T09:52:12.256 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/177 2024-03-29T09:52:12.452 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 86/177 2024-03-29T09:52:12.730 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 87/177 2024-03-29T09:52:13.070 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 88/177 2024-03-29T09:52:13.445 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 89/177 2024-03-29T09:52:13.689 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 90/177 2024-03-29T09:52:13.931 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 91/177 2024-03-29T09:52:14.279 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 92/177 2024-03-29T09:52:14.489 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 93/177 2024-03-29T09:52:14.635 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 94/177 2024-03-29T09:52:14.849 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 95/177 2024-03-29T09:52:15.188 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 96/177 2024-03-29T09:52:15.405 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 97/177 2024-03-29T09:52:15.541 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 98/177 2024-03-29T09:52:15.752 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 99/177 2024-03-29T09:52:16.120 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 100/177 2024-03-29T09:52:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.7-19.el8.x86_64 101/177 2024-03-29T09:52:16.538 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 102/177 2024-03-29T09:52:16.724 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : uuid-1.6.2-43.el8.x86_64 103/177 2024-03-29T09:52:16.818 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 103/177 2024-03-29T09:52:16.958 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 104/177 2024-03-29T09:52:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-03-29T09:52:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-03-29T09:52:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-03-29T09:52:21.601 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 106/177 2024-03-29T09:52:21.795 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 106/177 2024-03-29T09:52:21.989 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 107/177 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 107/177 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /sys 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /proc 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /dev 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /run 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /mnt 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /var/tmp 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /home 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /tmp 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /dev 2024-03-29T09:52:30.572 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:52:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 108/177 2024-03-29T09:52:32.704 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 108/177 2024-03-29T09:52:32.975 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 109/177 2024-03-29T09:52:33.238 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 109/177 2024-03-29T09:52:33.453 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 110/177 2024-03-29T09:52:33.991 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 110/177 2024-03-29T09:52:34.195 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 111/177 2024-03-29T09:52:34.244 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 111/177 2024-03-29T09:52:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 112/177 2024-03-29T09:52:34.700 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 113/177 2024-03-29T09:52:34.853 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 114/177 2024-03-29T09:52:34.935 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 114/177 2024-03-29T09:52:35.247 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : jq-1.6-8.el8.x86_64 115/177 2024-03-29T09:52:35.478 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : automake-1.16.1-8.el8.noarch 116/177 2024-03-29T09:52:36.153 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 117/177 2024-03-29T09:52:36.369 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 117/177 2024-03-29T09:52:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.e 118/177 2024-03-29T09:52:36.711 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9. 119/177 2024-03-29T09:52:36.927 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 120/177 2024-03-29T09:52:37.190 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 120/177 2024-03-29T09:52:37.399 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 121/177 2024-03-29T09:52:38.479 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 121/177 2024-03-29T09:52:38.652 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 122/177 2024-03-29T09:52:38.780 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 122/177 2024-03-29T09:52:39.188 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el 123/177 2024-03-29T09:52:39.399 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 124/177 2024-03-29T09:52:39.466 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 124/177 2024-03-29T09:52:39.685 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 125/177 2024-03-29T09:52:39.711 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 125/177 2024-03-29T09:52:39.957 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 126/177 2024-03-29T09:52:40.152 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/177 2024-03-29T09:52:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 107/177 2024-03-29T09:52:43.876 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2024-03-29T09:52:43.876 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2024-03-29T09:52:43.876 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /dev 2024-03-29T09:52:43.876 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /run 2024-03-29T09:52:43.876 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2024-03-29T09:52:43.876 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2024-03-29T09:52:43.876 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2024-03-29T09:52:43.876 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2024-03-29T09:52:43.876 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /dev 2024-03-29T09:52:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:52:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 108/177 2024-03-29T09:52:44.987 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/177 2024-03-29T09:52:46.115 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 108/177 2024-03-29T09:52:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 109/177 2024-03-29T09:52:46.539 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 128/177 2024-03-29T09:52:46.638 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 109/177 2024-03-29T09:52:46.782 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 129/177 2024-03-29T09:52:46.834 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 110/177 2024-03-29T09:52:46.842 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 129/177 2024-03-29T09:52:47.060 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 130/177 2024-03-29T09:52:47.129 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 130/177 2024-03-29T09:52:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 131/177 2024-03-29T09:52:47.402 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 110/177 2024-03-29T09:52:47.634 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 111/177 2024-03-29T09:52:47.684 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 111/177 2024-03-29T09:52:47.775 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 131/177 2024-03-29T09:52:47.885 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 112/177 2024-03-29T09:52:47.993 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-03-29T09:52:48.131 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 113/177 2024-03-29T09:52:48.275 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-03-29T09:52:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 114/177 2024-03-29T09:52:48.385 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 114/177 2024-03-29T09:52:48.621 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-contrib-10.23-2.module_el8+572+929c87 133/177 2024-03-29T09:52:48.725 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : jq-1.6-8.el8.x86_64 115/177 2024-03-29T09:52:48.950 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : automake-1.16.1-8.el8.noarch 116/177 2024-03-29T09:52:49.574 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 117/177 2024-03-29T09:52:49.780 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 117/177 2024-03-29T09:52:49.894 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 134/177 2024-03-29T09:52:49.935 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.e 118/177 2024-03-29T09:52:50.137 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9. 119/177 2024-03-29T09:52:50.189 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openssl-devel-1:1.1.1k-12.el8.x86_64 135/177 2024-03-29T09:52:50.383 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 120/177 2024-03-29T09:52:50.416 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : elfutils-libelf-devel-0.190-2.el8.x86_64 136/177 2024-03-29T09:52:50.598 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : util-linux-user-2.32.1-46.el8.x86_64 137/177 2024-03-29T09:52:50.662 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 120/177 2024-03-29T09:52:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 138/177 2024-03-29T09:52:50.868 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 121/177 2024-03-29T09:52:50.971 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_ 139/177 2024-03-29T09:52:51.225 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-team-1:1.40.16-15.el8.x86_64 140/177 2024-03-29T09:52:51.435 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-tui-1:1.40.16-15.el8.x86_64 141/177 2024-03-29T09:52:51.694 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : acl-2.2.53-3.el8.x86_64 142/177 2024-03-29T09:52:51.868 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libacl-devel-2.2.53-3.el8.x86_64 143/177 2024-03-29T09:52:52.015 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 121/177 2024-03-29T09:52:52.208 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 122/177 2024-03-29T09:52:52.341 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 122/177 2024-03-29T09:52:52.799 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el 123/177 2024-03-29T09:52:53.027 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 124/177 2024-03-29T09:52:53.057 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-29T09:52:53.096 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 124/177 2024-03-29T09:52:53.274 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 125/177 2024-03-29T09:52:53.300 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 125/177 2024-03-29T09:52:53.317 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-29T09:52:53.405 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-29T09:52:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 126/177 2024-03-29T09:52:53.622 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : xfsdump-3.1.8-7.el8.x86_64 145/177 2024-03-29T09:52:53.716 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/177 2024-03-29T09:52:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 146/177 2024-03-29T09:52:54.059 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 147/177 2024-03-29T09:52:54.408 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 148/177 2024-03-29T09:52:54.810 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 149/177 2024-03-29T09:52:55.109 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : openssl-1:1.1.1k-12.el8.x86_64 150/177 2024-03-29T09:52:55.303 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 151/177 2024-03-29T09:52:55.558 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flex-2.6.1-9.el8.x86_64 152/177 2024-03-29T09:52:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 152/177 2024-03-29T09:52:55.900 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : bison-3.0.4-10.el8.x86_64 153/177 2024-03-29T09:52:55.926 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 153/177 2024-03-29T09:52:56.060 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-tui-1:1.40.16-4.el8.x86_64 154/177 2024-03-29T09:52:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 155/177 2024-03-29T09:52:56.468 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 156/177 2024-03-29T09:52:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : openssl-1:1.1.1k-9.el8.x86_64 157/177 2024-03-29T09:52:56.859 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 158/177 2024-03-29T09:52:56.859 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-03-29T09:52:57.138 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-03-29T09:52:57.138 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-03-29T09:52:57.398 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-03-29T09:52:57.398 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 161/177 2024-03-29T09:52:57.704 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 161/177 2024-03-29T09:52:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : acl-2.2.53-1.el8.x86_64 162/177 2024-03-29T09:52:58.023 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libacl-devel-2.2.53-1.el8.x86_64 163/177 2024-03-29T09:52:58.235 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 164/177 2024-03-29T09:52:58.410 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 165/177 2024-03-29T09:52:58.643 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 166/177 2024-03-29T09:52:58.643 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 167/177 2024-03-29T09:52:58.855 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/177 2024-03-29T09:52:58.945 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 167/177 2024-03-29T09:52:58.945 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 168/177 2024-03-29T09:52:59.194 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 168/177 2024-03-29T09:52:59.195 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 169/177 2024-03-29T09:52:59.411 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 169/177 2024-03-29T09:52:59.568 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-team-1:1.40.16-4.el8.x86_64 170/177 2024-03-29T09:52:59.590 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-29T09:52:59.591 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-29T09:53:00.142 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-29T09:53:00.142 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-03-29T09:53:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-03-29T09:53:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 173/177 2024-03-29T09:53:00.459 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 128/177 2024-03-29T09:53:00.594 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 173/177 2024-03-29T09:53:00.594 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-03-29T09:53:00.683 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 129/177 2024-03-29T09:53:00.746 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 129/177 2024-03-29T09:53:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-03-29T09:53:00.956 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 130/177 2024-03-29T09:53:00.967 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 175/177 2024-03-29T09:53:01.027 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 130/177 2024-03-29T09:53:01.119 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libacl-2.2.53-1.el8.x86_64 176/177 2024-03-29T09:53:01.119 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-03-29T09:53:01.256 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 131/177 2024-03-29T09:53:01.645 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-03-29T09:53:01.645 INFO:teuthology.orchestra.run.smithi159.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-29T09:53:01.645 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:01.677 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 131/177 2024-03-29T09:53:01.923 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-03-29T09:53:02.221 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-03-29T09:53:02.388 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 1/177 2024-03-29T09:53:02.388 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 2/177 2024-03-29T09:53:02.388 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 3/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 4/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 5/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 6/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 7/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 8/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 9/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 13/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_ 14/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 15/177 2024-03-29T09:53:02.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 16/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_6 17/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 18/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el 20/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8. 21/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_6 22/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 23/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 24/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 25/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 26/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 27/177 2024-03-29T09:53:02.390 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 28/177 2024-03-29T09:53:02.404 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9. 29/177 2024-03-29T09:53:02.404 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 30/177 2024-03-29T09:53:02.404 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 31/177 2024-03-29T09:53:02.404 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 32/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el 33/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 34/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.e 35/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 36/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 37/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 38/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : jq-1.6-8.el8.x86_64 42/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/177 2024-03-29T09:53:02.405 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 44/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 47/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 50/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-contrib-10.23-2.module_el8+572+929c87 51/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-server-10.23-2.module_el8+572+929c87a 52/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 54/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/177 2024-03-29T09:53:02.406 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 65/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 66/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/177 2024-03-29T09:53:02.407 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libelf-devel-0.190-2.el8.x86_64 71/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 72/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 77/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 78/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 79/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 80/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libuuid-devel-2.32.1-46.el8.x86_64 82/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/177 2024-03-29T09:53:02.408 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 84/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8.x86_64 85/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 86/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 87/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 88/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 89/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 90/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 91/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 92/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 93/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 94/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 95/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 96/177 2024-03-29T09:53:02.409 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 97/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 98/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 99/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 100/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 101/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 102/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 103/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 104/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 105/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 106/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 107/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 108/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 109/177 2024-03-29T09:53:02.410 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 110/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 111/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 112/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 113/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 114/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 115/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-portend-2.6-1.el8.noarch 116/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 117/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 118/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 119/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 120/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 121/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 122/177 2024-03-29T09:53:02.411 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 123/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 124/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 125/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 126/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 127/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 128/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 129/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 130/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/177 2024-03-29T09:53:02.412 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-1:1.40.16-15.el8.x86_64 134/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-1:1.40.16-4.el8.x86_64 135/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 136/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 137/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-team-1:1.40.16-15.el8.x86_64 138/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-team-1:1.40.16-4.el8.x86_64 139/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-tui-1:1.40.16-15.el8.x86_64 140/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-tui-1:1.40.16-4.el8.x86_64 141/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : acl-2.2.53-3.el8.x86_64 142/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : acl-2.2.53-1.el8.x86_64 143/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el8.x86_64 144/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 145/177 2024-03-29T09:53:02.413 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libelf-0.190-2.el8.x86_64 146/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 147/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libs-0.190-2.el8.x86_64 148/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 149/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 150/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 151/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libacl-2.2.53-3.el8.x86_64 152/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libacl-2.2.53-1.el8.x86_64 153/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libacl-devel-2.2.53-3.el8.x86_64 154/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libacl-devel-2.2.53-1.el8.x86_64 155/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libblkid-2.32.1-46.el8.x86_64 156/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 157/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libfdisk-2.32.1-46.el8.x86_64 158/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 159/177 2024-03-29T09:53:02.414 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libmount-2.32.1-46.el8.x86_64 160/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 161/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libsmartcols-2.32.1-46.el8.x86_64 162/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 163/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libuuid-2.32.1-46.el8.x86_64 164/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 165/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-1:1.1.1k-12.el8.x86_64 166/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-1:1.1.1k-9.el8.x86_64 167/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8.x86_64 168/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-libs-1:1.1.1k-9.el8.x86_64 169/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-2.32.1-46.el8.x86_64 170/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 171/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-user-2.32.1-46.el8.x86_64 172/177 2024-03-29T09:53:02.415 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 173/177 2024-03-29T09:53:02.416 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsdump-3.1.8-7.el8.x86_64 174/177 2024-03-29T09:53:02.416 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 175/177 2024-03-29T09:53:02.416 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 176/177 2024-03-29T09:53:02.590 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-contrib-10.23-2.module_el8+572+929c87 133/177 2024-03-29T09:53:03.955 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 134/177 2024-03-29T09:53:04.362 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openssl-devel-1:1.1.1k-12.el8.x86_64 135/177 2024-03-29T09:53:04.571 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : elfutils-libelf-devel-0.190-2.el8.x86_64 136/177 2024-03-29T09:53:04.787 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-user-2.32.1-46.el8.x86_64 137/177 2024-03-29T09:53:04.977 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 138/177 2024-03-29T09:53:05.175 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_ 139/177 2024-03-29T09:53:05.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 177/177 2024-03-29T09:53:05.308 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:05.308 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2024-03-29T09:53:05.308 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-1:1.40.16-15.el8.x86_64 2024-03-29T09:53:05.308 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 2024-03-29T09:53:05.308 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-team-1:1.40.16-15.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-tui-1:1.40.16-15.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: acl-2.2.53-3.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-debuginfod-client-0.190-2.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-0.190-2.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libs-0.190-2.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: iproute-6.2.0-5.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: libacl-2.2.53-3.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: libacl-devel-2.2.53-3.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: libblkid-2.32.1-46.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: libfdisk-2.32.1-46.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: libmount-2.32.1-46.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.309 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: libsmartcols-2.32.1-46.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-2.32.1-46.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-1:1.1.1k-12.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-libs-1:1.1.1k-12.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-2.32.1-46.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-user-2.32.1-46.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: xfsdump-3.1.8-7.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: automake-1.16.1-8.el8.noarch 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: bison-3.0.4-10.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.310 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.311 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.312 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:05.312 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.312 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.312 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.312 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:05.312 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:05.312 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.312 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:05.312 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-devel-0.190-2.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: flex-2.6.1-9.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: fuse-2.9.7-19.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: jq-1.6-8.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: krb5-devel-1.18.2-22.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-03-29T09:53:05.313 INFO:teuthology.orchestra.run.smithi159.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: libpq-13.5-1.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: libselinux-devel-2.9-8.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-devel-2.32.1-46.el8.x86_64 2024-03-29T09:53:05.314 INFO:teuthology.orchestra.run.smithi159.stdout: libverto-devel-0.3.2-2.el8.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-0.3.15-4.el8.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-devel-1:1.1.1k-12.el8.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-devel-10.32-3.el8.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-utf16-10.32-3.el8.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-utf32-10.32-3.el8.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-contrib-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-server-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-29T09:53:05.315 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-29T09:53:05.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-29T09:53:05.317 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend-2.6-1.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-29T09:53:05.318 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-29T09:53:05.319 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-29T09:53:05.320 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.320 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.320 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:05.320 INFO:teuthology.orchestra.run.smithi159.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-29T09:53:05.320 INFO:teuthology.orchestra.run.smithi159.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-29T09:53:05.320 INFO:teuthology.orchestra.run.smithi159.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-29T09:53:05.320 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-29T09:53:05.320 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-29T09:53:05.321 INFO:teuthology.orchestra.run.smithi159.stdout: zlib-devel-1.2.11-21.el8.x86_64 2024-03-29T09:53:05.321 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:05.321 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-29T09:53:05.389 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-team-1:1.40.16-15.el8.x86_64 140/177 2024-03-29T09:53:05.616 DEBUG:teuthology.parallel:result is None 2024-03-29T09:53:05.641 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-tui-1:1.40.16-15.el8.x86_64 141/177 2024-03-29T09:53:05.911 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : acl-2.2.53-3.el8.x86_64 142/177 2024-03-29T09:53:06.081 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libacl-devel-2.2.53-3.el8.x86_64 143/177 2024-03-29T09:53:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-29T09:53:07.622 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-29T09:53:07.718 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-29T09:53:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : xfsdump-3.1.8-7.el8.x86_64 145/177 2024-03-29T09:53:08.114 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 146/177 2024-03-29T09:53:08.356 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 147/177 2024-03-29T09:53:08.704 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 148/177 2024-03-29T09:53:09.138 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 149/177 2024-03-29T09:53:09.364 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-1:1.1.1k-12.el8.x86_64 150/177 2024-03-29T09:53:09.546 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 151/177 2024-03-29T09:53:09.795 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flex-2.6.1-9.el8.x86_64 152/177 2024-03-29T09:53:09.905 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 152/177 2024-03-29T09:53:10.203 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : bison-3.0.4-10.el8.x86_64 153/177 2024-03-29T09:53:10.231 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 153/177 2024-03-29T09:53:10.364 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-tui-1:1.40.16-4.el8.x86_64 154/177 2024-03-29T09:53:10.522 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 155/177 2024-03-29T09:53:10.685 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 156/177 2024-03-29T09:53:10.914 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-1:1.1.1k-9.el8.x86_64 157/177 2024-03-29T09:53:11.052 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 158/177 2024-03-29T09:53:11.053 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-03-29T09:53:11.317 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-03-29T09:53:11.317 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-03-29T09:53:11.561 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-03-29T09:53:11.562 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 161/177 2024-03-29T09:53:11.825 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 161/177 2024-03-29T09:53:12.006 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : acl-2.2.53-1.el8.x86_64 162/177 2024-03-29T09:53:12.144 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libacl-devel-2.2.53-1.el8.x86_64 163/177 2024-03-29T09:53:12.347 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 164/177 2024-03-29T09:53:12.522 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 165/177 2024-03-29T09:53:12.747 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 166/177 2024-03-29T09:53:12.747 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 167/177 2024-03-29T09:53:12.982 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 167/177 2024-03-29T09:53:12.982 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 168/177 2024-03-29T09:53:13.257 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 168/177 2024-03-29T09:53:13.257 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 169/177 2024-03-29T09:53:13.482 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 169/177 2024-03-29T09:53:13.614 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-team-1:1.40.16-4.el8.x86_64 170/177 2024-03-29T09:53:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-29T09:53:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-29T09:53:14.177 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-29T09:53:14.177 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-03-29T09:53:14.440 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-03-29T09:53:14.440 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 173/177 2024-03-29T09:53:14.656 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 173/177 2024-03-29T09:53:14.656 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-03-29T09:53:14.890 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-03-29T09:53:15.088 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 175/177 2024-03-29T09:53:15.240 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libacl-2.2.53-1.el8.x86_64 176/177 2024-03-29T09:53:15.240 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-03-29T09:53:15.729 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-03-29T09:53:15.729 INFO:teuthology.orchestra.run.smithi012.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-29T09:53:15.729 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 1/177 2024-03-29T09:53:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 2/177 2024-03-29T09:53:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 3/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 4/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 5/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 6/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 7/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 8/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 9/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 13/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_ 14/177 2024-03-29T09:53:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 15/177 2024-03-29T09:53:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 16/177 2024-03-29T09:53:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_6 17/177 2024-03-29T09:53:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 18/177 2024-03-29T09:53:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-03-29T09:53:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el 20/177 2024-03-29T09:53:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8. 21/177 2024-03-29T09:53:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_6 22/177 2024-03-29T09:53:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 23/177 2024-03-29T09:53:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 24/177 2024-03-29T09:53:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 25/177 2024-03-29T09:53:16.497 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 26/177 2024-03-29T09:53:16.497 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 27/177 2024-03-29T09:53:16.497 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 28/177 2024-03-29T09:53:16.497 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9. 29/177 2024-03-29T09:53:16.497 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 30/177 2024-03-29T09:53:16.497 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 31/177 2024-03-29T09:53:16.497 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 32/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el 33/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 34/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.e 35/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 36/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 37/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 38/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : jq-1.6-8.el8.x86_64 42/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 44/177 2024-03-29T09:53:16.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 47/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 50/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-contrib-10.23-2.module_el8+572+929c87 51/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-server-10.23-2.module_el8+572+929c87a 52/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 54/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/177 2024-03-29T09:53:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 65/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 66/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-devel-0.190-2.el8.x86_64 71/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 72/177 2024-03-29T09:53:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 77/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 78/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 79/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 80/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-devel-2.32.1-46.el8.x86_64 82/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 84/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8.x86_64 85/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 86/177 2024-03-29T09:53:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 87/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 88/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 89/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 90/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 91/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 92/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 93/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 94/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 95/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 96/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 97/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 98/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 99/177 2024-03-29T09:53:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 100/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 101/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 102/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 103/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 104/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 105/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 106/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 107/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 108/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 109/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 110/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 111/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 112/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 113/177 2024-03-29T09:53:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 114/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 115/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-2.6-1.el8.noarch 116/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 117/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 118/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 119/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 120/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 121/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 122/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 123/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 124/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 125/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 126/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 127/177 2024-03-29T09:53:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 128/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 129/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 130/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.40.16-15.el8.x86_64 134/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.40.16-4.el8.x86_64 135/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 136/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 137/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.40.16-15.el8.x86_64 138/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.40.16-4.el8.x86_64 139/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.40.16-15.el8.x86_64 140/177 2024-03-29T09:53:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.40.16-4.el8.x86_64 141/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : acl-2.2.53-3.el8.x86_64 142/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : acl-2.2.53-1.el8.x86_64 143/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el8.x86_64 144/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 145/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.190-2.el8.x86_64 146/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 147/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.190-2.el8.x86_64 148/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 149/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 150/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 151/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-2.2.53-3.el8.x86_64 152/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-2.2.53-1.el8.x86_64 153/177 2024-03-29T09:53:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-devel-2.2.53-3.el8.x86_64 154/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-devel-2.2.53-1.el8.x86_64 155/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libblkid-2.32.1-46.el8.x86_64 156/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 157/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfdisk-2.32.1-46.el8.x86_64 158/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 159/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libmount-2.32.1-46.el8.x86_64 160/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 161/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsmartcols-2.32.1-46.el8.x86_64 162/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 163/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-2.32.1-46.el8.x86_64 164/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 165/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:1.1.1k-12.el8.x86_64 166/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:1.1.1k-9.el8.x86_64 167/177 2024-03-29T09:53:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8.x86_64 168/177 2024-03-29T09:53:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:1.1.1k-9.el8.x86_64 169/177 2024-03-29T09:53:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-2.32.1-46.el8.x86_64 170/177 2024-03-29T09:53:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 171/177 2024-03-29T09:53:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-user-2.32.1-46.el8.x86_64 172/177 2024-03-29T09:53:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 173/177 2024-03-29T09:53:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsdump-3.1.8-7.el8.x86_64 174/177 2024-03-29T09:53:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 175/177 2024-03-29T09:53:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 176/177 2024-03-29T09:53:19.287 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 177/177 2024-03-29T09:53:19.287 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:19.287 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-1:1.40.16-15.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team-1:1.40.16-15.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui-1:1.40.16-15.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: acl-2.2.53-3.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client-0.190-2.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-0.190-2.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs-0.190-2.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: iproute-6.2.0-5.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-2.2.53-3.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-devel-2.2.53-3.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: libblkid-2.32.1-46.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: libfdisk-2.32.1-46.el8.x86_64 2024-03-29T09:53:19.288 INFO:teuthology.orchestra.run.smithi012.stdout: libmount-2.32.1-46.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: libsmartcols-2.32.1-46.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-2.32.1-46.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-1:1.1.1k-12.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs-1:1.1.1k-12.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-2.32.1-46.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-user-2.32.1-46.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: xfsdump-3.1.8-7.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: automake-1.16.1-8.el8.noarch 2024-03-29T09:53:19.289 INFO:teuthology.orchestra.run.smithi012.stdout: bison-3.0.4-10.el8.x86_64 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:19.290 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel-0.190-2.el8.x86_64 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: flex-2.6.1-9.el8.x86_64 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: fuse-2.9.7-19.el8.x86_64 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-29T09:53:19.291 INFO:teuthology.orchestra.run.smithi012.stdout: jq-1.6-8.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: krb5-devel-1.18.2-22.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: libpq-13.5-1.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-29T09:53:19.292 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: libselinux-devel-2.9-8.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel-2.32.1-46.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: libverto-devel-0.3.2-2.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.15-4.el8.x86_64 2024-03-29T09:53:19.293 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel-1:1.1.1k-12.el8.x86_64 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: pcre2-devel-10.32-3.el8.x86_64 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: pcre2-utf16-10.32-3.el8.x86_64 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: pcre2-utf32-10.32-3.el8.x86_64 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.294 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-29T09:53:19.295 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-2.6-1.el8.noarch 2024-03-29T09:53:19.296 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-03-29T09:53:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-03-29T09:53:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-29T09:53:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-29T09:53:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-29T09:53:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-29T09:53:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-29T09:53:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel-1.2.11-21.el8.x86_64 2024-03-29T09:53:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-29T09:53:19.609 DEBUG:teuthology.parallel:result is None 2024-03-29T09:53:19.610 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d0e392d9aee635f2b65768c460c7a151ead7a5e2 2024-03-29T09:53:19.751 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T09:53:19.806 INFO:teuthology.orchestra.run.smithi012.stdout:18.2.2-1015.gd0e392d9.el8 2024-03-29T09:53:19.806 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1015.gd0e392d9.el8 2024-03-29T09:53:19.806 INFO:teuthology.task.install:The correct ceph version 18.2.2-1015.gd0e392d9 is installed. 2024-03-29T09:53:19.808 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d0e392d9aee635f2b65768c460c7a151ead7a5e2 2024-03-29T09:53:19.947 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T09:53:20.008 INFO:teuthology.orchestra.run.smithi159.stdout:18.2.2-1015.gd0e392d9.el8 2024-03-29T09:53:20.008 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1015.gd0e392d9.el8 2024-03-29T09:53:20.008 INFO:teuthology.task.install:The correct ceph version 18.2.2-1015.gd0e392d9 is installed. 2024-03-29T09:53:20.010 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T09:53:20.010 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:20.010 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T09:53:20.082 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:20.082 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T09:53:20.156 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T09:53:20.157 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:20.157 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T09:53:20.229 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T09:53:20.335 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:20.336 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T09:53:20.404 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T09:53:20.511 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T09:53:20.512 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:20.512 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T09:53:20.582 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T09:53:20.691 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:20.691 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T09:53:20.758 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T09:53:20.863 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T09:53:20.864 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:20.864 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T09:53:20.934 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T09:53:21.042 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:21.042 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T09:53:21.109 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T09:53:21.214 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T09:53:21.276 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T09:53:21.276 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /var/log/ceph 2024-03-29T09:53:21.280 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /var/log/ceph 2024-03-29T09:53:21.347 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T09:53:21.347 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T09:53:21.352 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T09:53:21.419 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T09:53:21.419 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T09:53:21.423 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T09:53:21.497 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-29T09:53:21.497 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-29T09:53:21.497 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-29T09:53:21.497 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:21.497 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-29T09:53:21.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-29T09:53:21.749 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-29T09:53:21.819 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:21.820 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-29T09:53:21.864 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-29T09:53:22.079 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-29T09:53:22.151 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T09:53:22.151 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-29T09:53:22.152 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195', 'branch': 'wip-vshankar-testing-20240326.105515-reef', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7628195', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20240326.105515-reef'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20240326.105515-reef', 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 120, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 7327, 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2', 'sleep_before_teardown': 0, 'subset': '990/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20240326.105515-reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIF8rEDuGqw7u1MAn7aIckiCIMAIgcK8ExXmWCwNekr63BcoLpA25vbhmE2qT/oUqU/zqamOdxeKlOruHPG4l8s=', 'smithi159.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFpoTDgkUa4ldKvUIt0bpX7pXwcd199Mcl3Y1HX+RceoaNKFSo7mIUmoYKdUUWIiRlhtPgolkPbAw/7/+jHMc64='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-28_11:21:59', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T09:53:22.152 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T09:53:22.156 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T09:53:22.200 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T09:53:22.243 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T09:53:22.309 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:22.310 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T09:53:22.354 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T09:53:22.354 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-03-29T09:53:22.440 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T09:53:22.440 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T09:53:22.440 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 448 Links: 1 2024-03-29T09:53:22.440 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T09:53:22.440 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T09:53:22.440 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-29 09:53:15.955514116 +0000 2024-03-29T09:53:22.440 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-29 09:37:32.523777320 +0000 2024-03-29T09:53:22.440 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-29 09:37:32.523777320 +0000 2024-03-29T09:53:22.441 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-29 09:37:32.498777519 +0000 2024-03-29T09:53:22.441 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T09:53:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-29T09:53:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-29T09:53:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000284157 s, 1.8 MB/s 2024-03-29T09:53:22.549 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T09:53:22.600 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-03-29T09:53:22.647 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T09:53:22.647 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T09:53:22.647 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 454 Links: 1 2024-03-29T09:53:22.647 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T09:53:22.647 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T09:53:22.647 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-29 09:53:15.955514116 +0000 2024-03-29T09:53:22.647 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-29 09:37:32.995773562 +0000 2024-03-29T09:53:22.647 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-29 09:37:32.995773562 +0000 2024-03-29T09:53:22.648 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-29 09:37:32.967773785 +0000 2024-03-29T09:53:22.648 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T09:53:22.717 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-29T09:53:22.717 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-29T09:53:22.717 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00028183 s, 1.8 MB/s 2024-03-29T09:53:22.720 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T09:53:22.768 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-03-29T09:53:22.815 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T09:53:22.815 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T09:53:22.815 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 460 Links: 1 2024-03-29T09:53:22.815 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T09:53:22.815 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T09:53:22.816 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-29 09:53:15.955514116 +0000 2024-03-29T09:53:22.816 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-29 09:37:33.483769676 +0000 2024-03-29T09:53:22.816 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-29 09:37:33.483769676 +0000 2024-03-29T09:53:22.816 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-29 09:37:33.456769891 +0000 2024-03-29T09:53:22.816 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T09:53:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-29T09:53:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-29T09:53:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00027986 s, 1.8 MB/s 2024-03-29T09:53:22.884 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T09:53:22.932 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-03-29T09:53:22.980 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T09:53:22.980 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T09:53:22.980 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 466 Links: 1 2024-03-29T09:53:22.980 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T09:53:22.980 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T09:53:22.980 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-29 09:53:15.955514116 +0000 2024-03-29T09:53:22.980 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-29 09:37:33.973765775 +0000 2024-03-29T09:53:22.980 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-29 09:37:33.973765775 +0000 2024-03-29T09:53:22.980 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-29 09:37:33.948765974 +0000 2024-03-29T09:53:22.981 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T09:53:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-29T09:53:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-29T09:53:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00029485 s, 1.7 MB/s 2024-03-29T09:53:23.049 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T09:53:23.099 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T09:53:23.099 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:23.099 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T09:53:23.147 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T09:53:23.147 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_1 2024-03-29T09:53:23.193 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T09:53:23.194 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T09:53:23.194 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 448 Links: 1 2024-03-29T09:53:23.194 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T09:53:23.194 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T09:53:23.194 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-29 09:53:01.855981264 +0000 2024-03-29T09:53:23.194 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-29 09:37:22.995656979 +0000 2024-03-29T09:53:23.194 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-29 09:37:22.995656979 +0000 2024-03-29T09:53:23.194 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-29 09:37:22.971657432 +0000 2024-03-29T09:53:23.194 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T09:53:23.257 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-29T09:53:23.258 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-29T09:53:23.258 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000366638 s, 1.4 MB/s 2024-03-29T09:53:23.260 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T09:53:23.306 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_2 2024-03-29T09:53:23.351 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T09:53:23.351 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T09:53:23.351 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 454 Links: 1 2024-03-29T09:53:23.351 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T09:53:23.351 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T09:53:23.351 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-29 09:53:01.855981264 +0000 2024-03-29T09:53:23.351 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-29 09:37:23.446648475 +0000 2024-03-29T09:53:23.351 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-29 09:37:23.446648475 +0000 2024-03-29T09:53:23.351 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-29 09:37:23.422648928 +0000 2024-03-29T09:53:23.352 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T09:53:23.454 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-29T09:53:23.454 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-29T09:53:23.454 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.00028425 s, 1.8 MB/s 2024-03-29T09:53:23.456 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T09:53:23.503 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_3 2024-03-29T09:53:23.548 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T09:53:23.548 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T09:53:23.548 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 460 Links: 1 2024-03-29T09:53:23.549 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T09:53:23.549 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T09:53:23.549 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-29 09:53:01.855981264 +0000 2024-03-29T09:53:23.549 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-29 09:37:23.930639348 +0000 2024-03-29T09:53:23.549 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-29 09:37:23.930639348 +0000 2024-03-29T09:53:23.549 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-29 09:37:23.905639820 +0000 2024-03-29T09:53:23.549 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T09:53:23.652 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-29T09:53:23.652 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-29T09:53:23.652 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000293191 s, 1.7 MB/s 2024-03-29T09:53:23.654 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T09:53:23.700 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_4 2024-03-29T09:53:23.785 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T09:53:23.786 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T09:53:23.786 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 466 Links: 1 2024-03-29T09:53:23.786 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T09:53:23.786 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T09:53:23.786 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-29 09:53:01.855981264 +0000 2024-03-29T09:53:23.786 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-29 09:37:24.410630297 +0000 2024-03-29T09:53:23.786 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-29 09:37:24.410630297 +0000 2024-03-29T09:53:23.786 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-29 09:37:24.385630768 +0000 2024-03-29T09:53:23.787 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T09:53:23.889 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-29T09:53:23.889 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-29T09:53:23.889 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000296298 s, 1.7 MB/s 2024-03-29T09:53:23.892 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T09:53:23.938 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T09:53:23.938 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T09:53:23.938 INFO:tasks.ceph:Generating config... 2024-03-29T09:53:23.939 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-29T09:53:23.939 INFO:tasks.ceph:[client] debug client = 20 2024-03-29T09:53:23.940 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T09:53:23.940 INFO:tasks.ceph:[client] pid file = 2024-03-29T09:53:23.940 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-29T09:53:23.940 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-29T09:53:23.940 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-29T09:53:23.940 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-29T09:53:23.940 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-29T09:53:23.940 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-29T09:53:23.940 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-29T09:53:23.940 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-29T09:53:23.940 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-29T09:53:23.940 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-29T09:53:23.940 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-29T09:53:23.940 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-29T09:53:23.941 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-29T09:53:23.941 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-29T09:53:23.941 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-03-29T09:53:23.941 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T09:53:23.941 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T09:53:23.941 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T09:53:23.941 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T09:53:23.941 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T09:53:23.941 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-29T09:53:23.941 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-29T09:53:23.941 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-29T09:53:23.941 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-29T09:53:23.941 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T09:53:23.941 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T09:53:23.941 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-29T09:53:23.941 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-29T09:53:23.942 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T09:53:23.942 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T09:53:23.942 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-29T09:53:23.942 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T09:53:23.942 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T09:53:23.942 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T09:53:23.942 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T09:53:23.942 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T09:53:23.942 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-29T09:53:23.942 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T09:53:23.942 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T09:53:24.042 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T09:53:24.049 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T09:53:24.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T09:53:24.219 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.12'), ('mon.b', '172.21.15.159'), ('mon.c', '[v2:172.21.15.159:3301,v1:172.21.15.159:6790]')] 2024-03-29T09:53:24.219 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.12,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-29T09:53:24.220 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:24.220 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T09:53:24.266 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.12 --add b 172.21.15.159 --addv c '[v2:172.21.15.159:3301,v1:172.21.15.159:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:24.399 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:24.399 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: generated fsid 216a15e0-b7ec-465c-8851-ed202d429a6c 2024-03-29T09:53:24.399 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = pacific 2024-03-29T09:53:24.399 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 0 2024-03-29T09:53:24.399 INFO:teuthology.orchestra.run.smithi012.stdout:fsid 216a15e0-b7ec-465c-8851-ed202d429a6c 2024-03-29T09:53:24.399 INFO:teuthology.orchestra.run.smithi012.stdout:last_changed 2024-03-29T09:53:24.400635+0000 2024-03-29T09:53:24.400 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-03-29T09:53:24.400635+0000 2024-03-29T09:53:24.400 INFO:teuthology.orchestra.run.smithi012.stdout:min_mon_release 16 (pacific) 2024-03-29T09:53:24.400 INFO:teuthology.orchestra.run.smithi012.stdout:election_strategy: 1 2024-03-29T09:53:24.400 INFO:teuthology.orchestra.run.smithi012.stdout:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-29T09:53:24.400 INFO:teuthology.orchestra.run.smithi012.stdout:1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.b 2024-03-29T09:53:24.400 INFO:teuthology.orchestra.run.smithi012.stdout:2: [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] mon.c 2024-03-29T09:53:24.400 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T09:53:24.402 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T09:53:24.447 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 216a15e0-b7ec-465c-8851-ed202d429a6c... 2024-03-29T09:53:24.448 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T09:53:24.491 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T09:53:24.604 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2024-03-29T09:53:24.605 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2024-03-29T09:53:24.606 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2024-03-29T09:53:24.607 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = "172.21.15.12,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]" 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on skipped message = False 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = 216a15e0-b7ec-465c-8851-ed202d429a6c 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2024-03-29T09:53:24.608 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2024-03-29T09:53:24.609 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: bdev async discard = True 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: bdev enable discard = True 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore allocator = bitmap 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore block size = 96636764160 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore fsck on mount = True 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluefs = 1/20 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluestore = 1/20 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: debug rocksdb = 4/10 2024-03-29T09:53:24.610 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd full ratio = 0.9 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd nearfull ratio = 0.8 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: osd failsafe full ratio = 0.95 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: osd objectstore = bluestore 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: osd op complaint time = 180 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: debug client = 20 2024-03-29T09:53:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: debug finisher = 20 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.612 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: mon op complaint time = 120 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2024-03-29T09:53:24.613 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout: client mount timeout = 600 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout: debug client = 20 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = "" 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout: rados mon op timeout = 900 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout: rados osd op timeout = 900 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.b] 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.c] 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.a] 2024-03-29T09:53:24.614 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.c] 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.b] 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.d] 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout:[mds] 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout: debug mds = 20 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout: debug mds balancer = 20 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout: mds debug frag = True 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout: mds debug scatterstat = True 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout: mds op complaint time = 180 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout: mds verify scatter = True 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout: osd op complaint time = 180 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout: rados mon op timeout = 900 2024-03-29T09:53:24.615 INFO:teuthology.orchestra.run.smithi012.stdout: rados osd op timeout = 900 2024-03-29T09:53:24.616 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2024-03-29T09:53:24.616 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2024-03-29T09:53:24.616 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T09:53:24.616 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2024-03-29T09:53:24.616 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.616 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2024-03-29T09:53:24.616 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.616 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2024-03-29T09:53:24.616 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.616 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.617 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: mon_allow_pool_size_one = true 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2024-03-29T09:53:24.618 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = "172.21.15.12,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]" 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on skipped message = False 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = 216a15e0-b7ec-465c-8851-ed202d429a6c 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2024-03-29T09:53:24.619 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock profile = high_recovery_ops 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.620 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: bdev async discard = True 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: bdev enable discard = True 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore allocator = bitmap 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 1/20 2024-03-29T09:53:24.621 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 1/20 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 4/10 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2024-03-29T09:53:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: debug client = 20 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: debug finisher = 20 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-03-29T09:53:24.623 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T09:53:24.624 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: # 1m isn't quite enough 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: mon_down_mkfs_grace = 2m 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_filestore_osds = false 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: mon op complaint time = 120 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: client mount timeout = 600 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: debug client = 20 2024-03-29T09:53:24.625 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = "" 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 900 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 900 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.b] 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.c] 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.d] 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout:[mds] 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds = 20 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds balancer = 20 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-29T09:53:24.626 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug frag = True 2024-03-29T09:53:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug scatterstat = True 2024-03-29T09:53:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: mds op complaint time = 180 2024-03-29T09:53:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: mds verify scatter = True 2024-03-29T09:53:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2024-03-29T09:53:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 900 2024-03-29T09:53:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 900 2024-03-29T09:53:24.633 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T09:53:24.633 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T09:53:24.747 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T09:53:24.747 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:24.747 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T09:53:24.793 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:24.793 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T09:53:24.879 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T09:53:24.879 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:24.880 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T09:53:24.880 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T09:53:25.013 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:25.013 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:25.097 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T09:53:25.097 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:25.097 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T09:53:25.098 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T09:53:25.188 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:25.188 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:25.272 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T09:53:25.272 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T09:53:25.273 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T09:53:25.398 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T09:53:25.404 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T09:53:25.531 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T09:53:25.538 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T09:53:25.538 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-29T09:53:25.657 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-29T09:53:25.662 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-29T09:53:25.729 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-29T09:53:25.848 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-29T09:53:25.853 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-29T09:53:25.920 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-29T09:53:26.037 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-29T09:53:26.043 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-29T09:53:26.109 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-29T09:53:26.225 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-29T09:53:26.230 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-29T09:53:26.297 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T09:53:26.297 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T09:53:26.394 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T09:53:26.424 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-29T09:53:26.521 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-29T09:53:26.551 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T09:53:26.643 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T09:53:26.672 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-29T09:53:26.764 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-29T09:53:26.793 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T09:53:26.794 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T09:53:26.794 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T09:53:26.864 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T09:53:26.864 INFO:tasks.ceph:role: osd.0 2024-03-29T09:53:26.864 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T09:53:26.864 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T09:53:26.938 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T09:53:26.938 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T09:53:26.938 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T09:53:26.938 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-29T09:53:26.938 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T09:53:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-03-29T09:53:26.939 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T09:53:26.939 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-29T09:53:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T09:53:26.939 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T09:53:27.383 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-03-29T09:53:27.388 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-03-29T09:53:27.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T09:53:27.592 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T09:53:27.669 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T09:53:27.737 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T09:53:27.737 INFO:tasks.ceph:role: osd.1 2024-03-29T09:53:27.737 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T09:53:27.737 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T09:53:27.810 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T09:53:27.810 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T09:53:27.810 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T09:53:27.810 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-29T09:53:27.810 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T09:53:27.810 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-03-29T09:53:27.811 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T09:53:27.811 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-29T09:53:27.811 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T09:53:27.811 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T09:53:28.214 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-03-29T09:53:28.218 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-03-29T09:53:28.218 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T09:53:28.357 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T09:53:28.430 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T09:53:28.496 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T09:53:28.496 INFO:tasks.ceph:role: osd.2 2024-03-29T09:53:28.496 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T09:53:28.496 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T09:53:28.571 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T09:53:28.571 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T09:53:28.571 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T09:53:28.571 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-29T09:53:28.571 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T09:53:28.571 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-03-29T09:53:28.571 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T09:53:28.571 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-29T09:53:28.571 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T09:53:28.571 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T09:53:29.020 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-03-29T09:53:29.024 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-03-29T09:53:29.024 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T09:53:29.177 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T09:53:29.252 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T09:53:29.319 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T09:53:29.320 INFO:tasks.ceph:role: osd.3 2024-03-29T09:53:29.320 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T09:53:29.320 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T09:53:29.394 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T09:53:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T09:53:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T09:53:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-29T09:53:29.394 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T09:53:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-03-29T09:53:29.394 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T09:53:29.395 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-29T09:53:29.395 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T09:53:29.395 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T09:53:29.834 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-03-29T09:53:29.838 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-03-29T09:53:29.839 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T09:53:30.002 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T09:53:30.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:30.194+0000 7fd9ab6a9680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T09:53:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:30.194+0000 7fd9ab6a9680 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T09:53:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:30.194+0000 7fd9ab6a9680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T09:53:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:30.194+0000 7fd9ab6a9680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T09:53:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:30.195+0000 7fd9ab6a9680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T09:53:30.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:30.197+0000 7fd9ab6a9680 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T09:53:33.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T09:53:33.164 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:33.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:33.271+0000 7f0a78814680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T09:53:33.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:33.272+0000 7f0a78814680 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T09:53:33.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:33.272+0000 7f0a78814680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T09:53:33.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:33.272+0000 7f0a78814680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T09:53:33.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:33.272+0000 7f0a78814680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T09:53:33.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:33.274+0000 7f0a78814680 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T09:53:36.180 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T09:53:36.252 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:36.359+0000 7fb4191cf680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T09:53:36.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:36.359+0000 7fb4191cf680 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T09:53:36.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:36.359+0000 7fb4191cf680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T09:53:36.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:36.359+0000 7fb4191cf680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T09:53:36.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:36.359+0000 7fb4191cf680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T09:53:36.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:36.361+0000 7fb4191cf680 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T09:53:39.245 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T09:53:39.317 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:39.424+0000 7f0c100a0680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T09:53:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:39.424+0000 7f0c100a0680 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T09:53:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:39.424+0000 7f0c100a0680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T09:53:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:39.424+0000 7f0c100a0680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T09:53:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:39.424+0000 7f0c100a0680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T09:53:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:53:39.426+0000 7f0c100a0680 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T09:53:42.321 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T09:53:42.392 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T09:53:42.466 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T09:53:42.466 INFO:tasks.ceph:role: osd.4 2024-03-29T09:53:42.466 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T09:53:42.466 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T09:53:42.542 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T09:53:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T09:53:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T09:53:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-29T09:53:42.543 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T09:53:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-29T09:53:42.543 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T09:53:42.543 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-29T09:53:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T09:53:42.543 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T09:53:42.979 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-29T09:53:42.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-29T09:53:42.983 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T09:53:43.189 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T09:53:43.264 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T09:53:43.330 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T09:53:43.330 INFO:tasks.ceph:role: osd.5 2024-03-29T09:53:43.330 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T09:53:43.330 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T09:53:43.402 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T09:53:43.402 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T09:53:43.403 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T09:53:43.403 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-29T09:53:43.403 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T09:53:43.403 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-29T09:53:43.403 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T09:53:43.403 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-29T09:53:43.403 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T09:53:43.403 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T09:53:43.821 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-29T09:53:43.825 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-29T09:53:43.825 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T09:53:43.962 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T09:53:44.035 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T09:53:44.102 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T09:53:44.102 INFO:tasks.ceph:role: osd.6 2024-03-29T09:53:44.102 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T09:53:44.102 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T09:53:44.176 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T09:53:44.176 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T09:53:44.176 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T09:53:44.176 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-29T09:53:44.176 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T09:53:44.177 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-29T09:53:44.177 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T09:53:44.177 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-29T09:53:44.177 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T09:53:44.177 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T09:53:44.641 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-29T09:53:44.645 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-29T09:53:44.646 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T09:53:44.835 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T09:53:44.908 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T09:53:44.974 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T09:53:44.974 INFO:tasks.ceph:role: osd.7 2024-03-29T09:53:44.974 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T09:53:44.974 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T09:53:45.047 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T09:53:45.047 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T09:53:45.047 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T09:53:45.047 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-29T09:53:45.047 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T09:53:45.047 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-29T09:53:45.048 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T09:53:45.048 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-29T09:53:45.048 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T09:53:45.048 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T09:53:45.481 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-29T09:53:45.484 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-29T09:53:45.485 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T09:53:45.623 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T09:53:45.696 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:45.816 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:45.817+0000 7fde1c0e2680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T09:53:45.816 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:45.817+0000 7fde1c0e2680 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T09:53:45.817 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:45.817+0000 7fde1c0e2680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T09:53:45.817 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:45.818+0000 7fde1c0e2680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T09:53:45.817 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:45.818+0000 7fde1c0e2680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T09:53:45.818 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:45.820+0000 7fde1c0e2680 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T09:53:48.704 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T09:53:48.771 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:48.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:48.873+0000 7fc4b620c680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T09:53:48.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:48.874+0000 7fc4b620c680 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T09:53:48.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:48.874+0000 7fc4b620c680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T09:53:48.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:48.874+0000 7fc4b620c680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T09:53:48.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:48.874+0000 7fc4b620c680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T09:53:48.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:48.875+0000 7fc4b620c680 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T09:53:51.755 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T09:53:51.823 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:51.924 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:51.925+0000 7f860523c680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T09:53:51.924 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:51.925+0000 7f860523c680 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T09:53:51.924 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:51.925+0000 7f860523c680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T09:53:51.924 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:51.925+0000 7f860523c680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T09:53:51.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:51.925+0000 7f860523c680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T09:53:51.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:51.927+0000 7f860523c680 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T09:53:52.153 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:53:52.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:53:54.820 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T09:53:54.890 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:53:54.993 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:54.993+0000 7f241f638680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T09:53:54.993 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:54.993+0000 7f241f638680 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T09:53:54.993 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:54.993+0000 7f241f638680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T09:53:54.993 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:54.993+0000 7f241f638680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T09:53:54.994 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:54.993+0000 7f241f638680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T09:53:54.994 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:53:54.995+0000 7f241f638680 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T09:53:57.889 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T09:53:57.958 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T09:53:57.958 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:57.959 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T09:53:58.034 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:58.034 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-29T09:53:58.102 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:58.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-29T09:53:58.210 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:58.210 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T09:53:58.316 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:58.316 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T09:53:58.422 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:58.422 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T09:53:58.528 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:58.528 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T09:53:58.635 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:58.635 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T09:53:58.705 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:58.705 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-29T09:53:58.812 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:58.812 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-29T09:53:58.919 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:58.919 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T09:53:59.024 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:59.025 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T09:53:59.130 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:59.130 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T09:53:59.237 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:59.237 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T09:53:59.345 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:59.346 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T09:53:59.393 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:53:59.393 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-29T09:53:59.479 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:59.479 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T09:53:59.523 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:53:59.523 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-29T09:53:59.607 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T09:53:59.607 DEBUG:teuthology.orchestra.run.smithi012:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T09:53:59.612 DEBUG:teuthology.orchestra.run.smithi159:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.y] 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw== 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.a] 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg== 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.c] 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg== 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw== 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w== 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2024-03-29T09:53:59.673 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A== 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.3] 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g== 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.x] 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ== 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.b] 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw== 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.d] 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA== 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.4] 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ== 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.5] 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw== 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.6] 2024-03-29T09:53:59.674 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ== 2024-03-29T09:53:59.675 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.7] 2024-03-29T09:53:59.675 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w== 2024-03-29T09:53:59.675 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-03-29T09:53:59.675 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA== 2024-03-29T09:53:59.675 INFO:teuthology.orchestra.run.smithi012.stdout:[client.2] 2024-03-29T09:53:59.675 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ== 2024-03-29T09:53:59.675 INFO:teuthology.orchestra.run.smithi012.stdout:[client.1] 2024-03-29T09:53:59.675 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA== 2024-03-29T09:53:59.675 INFO:teuthology.orchestra.run.smithi012.stdout:[client.3] 2024-03-29T09:53:59.675 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw== 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.y] 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw== 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg== 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg== 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw== 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w== 2024-03-29T09:53:59.711 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A== 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.3] 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g== 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.x] 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ== 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw== 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.d] 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA== 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.4] 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ== 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.5] 2024-03-29T09:53:59.712 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw== 2024-03-29T09:53:59.713 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.6] 2024-03-29T09:53:59.713 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ== 2024-03-29T09:53:59.713 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.7] 2024-03-29T09:53:59.713 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w== 2024-03-29T09:53:59.713 INFO:teuthology.orchestra.run.smithi159.stdout:[client.0] 2024-03-29T09:53:59.713 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA== 2024-03-29T09:53:59.713 INFO:teuthology.orchestra.run.smithi159.stdout:[client.2] 2024-03-29T09:53:59.713 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ== 2024-03-29T09:53:59.713 INFO:teuthology.orchestra.run.smithi159.stdout:[client.1] 2024-03-29T09:53:59.713 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA== 2024-03-29T09:53:59.714 INFO:teuthology.orchestra.run.smithi159.stdout:[client.3] 2024-03-29T09:53:59.714 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw== 2024-03-29T09:53:59.715 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T09:53:59.720 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T09:53:59.861 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T09:53:59.866 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T09:53:59.973 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T09:53:59.977 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T09:54:00.086 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.090 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.195 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.199 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.305 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.309 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.413 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.417 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.522 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T09:54:00.526 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T09:54:00.630 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T09:54:00.634 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T09:54:00.744 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T09:54:00.748 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-29T09:54:00.857 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.861 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.966 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:00.970 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:01.074 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:01.078 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:01.187 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:01.191 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T09:54:01.296 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T09:54:01.301 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T09:54:01.410 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T09:54:01.415 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T09:54:01.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T09:54:01.527 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T09:54:01.636 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T09:54:01.640 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T09:54:01.749 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T09:54:01.749 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T09:54:01.816 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T09:54:01.944 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T09:54:02.011 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T09:54:02.078 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T09:54:02.198 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T09:54:02.264 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T09:54:02.330 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T09:54:02.450 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T09:54:02.517 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:54:02.521 DEBUG:teuthology.orchestra.run.smithi159:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T09:54:02.565 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T09:54:02.565 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T09:54:02.565 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T09:54:02.607 INFO:tasks.ceph.mon.a:Started 2024-03-29T09:54:02.607 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T09:54:02.608 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T09:54:02.611 INFO:tasks.ceph.mon.b:Started 2024-03-29T09:54:02.611 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T09:54:02.612 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T09:54:02.617 INFO:tasks.ceph.mon.c:Started 2024-03-29T09:54:02.617 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T09:54:02.618 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T09:54:02.618 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T09:54:02.622 INFO:tasks.ceph.mgr.y:Started 2024-03-29T09:54:02.622 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T09:54:02.622 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T09:54:02.626 INFO:tasks.ceph.mgr.x:Started 2024-03-29T09:54:02.627 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T09:54:02.627 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T09:54:02.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.838+0000 7f7fbf200700 1 Processor -- start 2024-03-29T09:54:02.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.838+0000 7f7fbf200700 1 -- start start 2024-03-29T09:54:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.839+0000 7f7fbf200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fc00d04c0 0x7f7fc00b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.839+0000 7f7fbf200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc00b6a00 con 0x7f7fc015eb10 2024-03-29T09:54:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.839+0000 7f7fbf200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc015f060 con 0x7f7fc00d08a0 2024-03-29T09:54:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.839+0000 7f7fbf200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc015f340 con 0x7f7fc00d04c0 2024-03-29T09:54:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.839+0000 7f7fbd400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7fc015eb10 0x7f7fc00b62f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42066/0 (socket says 172.21.15.12:42066) 2024-03-29T09:54:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.839+0000 7f7fbd400700 1 -- 172.21.15.12:0/3975460853 learned_addr learned my addr 172.21.15.12:0/3975460853 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:02.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927571595 0 0) 0x7f7fc00b6a00 con 0x7f7fc015eb10 2024-03-29T09:54:02.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbde00700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fc00d04c0 0x7f7fc00b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:02.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa40035b0 con 0x7f7fc015eb10 2024-03-29T09:54:02.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51239146 0 0) 0x7f7fc015f060 con 0x7f7fc00d08a0 2024-03-29T09:54:02.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc00b6a00 con 0x7f7fc00d08a0 2024-03-29T09:54:02.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4254568353 0 0) 0x7f7fa40035b0 con 0x7f7fc015eb10 2024-03-29T09:54:02.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fc00d04c0 msgr2=0x7f7fc00b8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:02.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fc00d04c0 0x7f7fc00b8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:02.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 >> v1:172.21.15.159:6789/0 conn(0x7f7fc00d08a0 legacy=0x7f7fc015e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:02.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc015f620 con 0x7f7fc015eb10 2024-03-29T09:54:02.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f7fac001670 con 0x7f7fc015eb10 2024-03-29T09:54:02.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 >> v1:172.21.15.12:6789/0 conn(0x7f7fc015eb10 legacy=0x7f7fc00b62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:02.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fa4003bc0 0x7f7fa4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:02.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fa4004850 0x7f7fa4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:02.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4005210 0x7f7fa400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:02.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc015f340 con 0x7f7fa4004850 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa400c0e0 con 0x7f7fa4003bc0 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa400c270 con 0x7f7fa4005210 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbde00700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fc00d04c0 0x7f7fc00b8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.840+0000 7f7fbd400700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fa4004850 0x7f7fa4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbe800700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4005210 0x7f7fa400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbde00700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fa4003bc0 0x7f7fa4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbd400700 1 -- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4005210 msgr2=0x7f7fa400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbd400700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4005210 0x7f7fa400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbd400700 1 -- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fa4003bc0 msgr2=0x7f7fa4003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbd400700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fa4003bc0 0x7f7fa4003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbd400700 1 -- 172.21.15.12:0/3975460853 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa40035b0 con 0x7f7fa4004850 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbd400700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fa4004850 0x7f7fa4004cd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00c210 tx=0x7f7fa4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af047f03e44e83df server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:02.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac00cd50 con 0x7f7fa4004850 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac005960 con 0x7f7fa4004850 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.841+0000 7f7fbca00700 1 -- 172.21.15.12:0/3975460853 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac005d10 con 0x7f7fa4004850 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.842+0000 7f7fbf200700 1 -- 172.21.15.12:0/3975460853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fa4004850 msgr2=0x7f7fa4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.842+0000 7f7fbf200700 1 --2- 172.21.15.12:0/3975460853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fa4004850 0x7f7fa4004cd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00c210 tx=0x7f7fa4005880 comp rx=0 tx=0).stop 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.842+0000 7f7fbe800700 1 -- 172.21.15.12:0/3975460853 reap_dead start 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.842+0000 7f7fbf200700 1 -- 172.21.15.12:0/3975460853 shutdown_connections 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.842+0000 7f7fbf200700 1 -- 172.21.15.12:0/3975460853 >> 172.21.15.12:0/3975460853 conn(0x7f7fc00668e0 msgr2=0x7f7fc0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.842+0000 7f7fbf200700 1 -- 172.21.15.12:0/3975460853 shutdown_connections 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.842+0000 7f7fbf200700 1 -- 172.21.15.12:0/3975460853 wait complete. 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.844+0000 7f7fbf200700 1 Processor -- start 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.844+0000 7f7fbf200700 1 -- start start 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.845+0000 7f7fbf200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4004850 0x7f7fc01561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.845+0000 7f7fbf200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fc00cae10 0x7f7fc01f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.845+0000 7f7fbf200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fc00cc7e0 0x7f7fc00ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.845+0000 7f7fbf200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00d1ee0 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.845+0000 7f7fbf200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00d4190 con 0x7f7fc00cae10 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.845+0000 7f7fbf200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc0155f40 con 0x7f7fa4004850 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fc00cc7e0 0x7f7fc00ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fc00cc7e0 0x7f7fc00ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60764/0 (socket says 172.21.15.12:60764) 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbe800700 1 -- 172.21.15.12:0/3564745651 learned_addr learned my addr 172.21.15.12:0/3564745651 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbd400700 1 --2- 172.21.15.12:0/3564745651 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fc00cae10 0x7f7fc01f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbde00700 1 --2- 172.21.15.12:0/3564745651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4004850 0x7f7fc01561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbe800700 1 -- 172.21.15.12:0/3564745651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4004850 msgr2=0x7f7fc01561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbe800700 1 --2- 172.21.15.12:0/3564745651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4004850 0x7f7fc01561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:02.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbe800700 1 -- 172.21.15.12:0/3564745651 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fc00cae10 msgr2=0x7f7fc01f2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:02.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbe800700 1 --2- 172.21.15.12:0/3564745651 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fc00cae10 0x7f7fc01f2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:02.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbe800700 1 -- 172.21.15.12:0/3564745651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc00d3bd0 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbd400700 1 --2- 172.21.15.12:0/3564745651 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fc00cae10 0x7f7fc01f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:02.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.846+0000 7f7fbe800700 1 --2- 172.21.15.12:0/3564745651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fc00cc7e0 0x7f7fc00ccc70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00b5550 tx=0x7f7fb800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:02.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.847+0000 7f7fb6000700 1 -- 172.21.15.12:0/3564745651 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8003cd0 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.847+0000 7f7fb6000700 1 -- 172.21.15.12:0/3564745651 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb8003e50 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.847+0000 7f7fb6000700 1 -- 172.21.15.12:0/3564745651 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb800dc00 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.847+0000 7f7fbf200700 1 -- 172.21.15.12:0/3564745651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc00d3eb0 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.847+0000 7f7fbf200700 1 -- 172.21.15.12:0/3564745651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc00cb2f0 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.847+0000 7f7fb6000700 1 -- 172.21.15.12:0/3564745651 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7fb80053d0 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.847+0000 7f7fb6000700 1 -- 172.21.15.12:0/3564745651 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7fb801d760 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.848+0000 7f7fbf200700 1 -- 172.21.15.12:0/3564745651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc004ea80 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.851+0000 7f7fb6000700 1 -- 172.21.15.12:0/3564745651 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f7fb80057c0 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.933+0000 7f7fbf200700 1 -- 172.21.15.12:0/3564745651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7fc00b4630 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.936+0000 7f7fb6000700 1 -- 172.21.15.12:0/3564745651 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7fb800dd60 con 0x7f7fc00cc7e0 2024-03-29T09:54:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.937+0000 7f7fbf200700 1 -- 172.21.15.12:0/3564745651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fc00cc7e0 msgr2=0x7f7fc00ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.937+0000 7f7fbf200700 1 --2- 172.21.15.12:0/3564745651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fc00cc7e0 0x7f7fc00ccc70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00b5550 tx=0x7f7fb800d430 comp rx=0 tx=0).stop 2024-03-29T09:54:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.937+0000 7f7fbf200700 1 -- 172.21.15.12:0/3564745651 shutdown_connections 2024-03-29T09:54:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.937+0000 7f7fbf200700 1 --2- 172.21.15.12:0/3564745651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fc00cc7e0 0x7f7fc00ccc70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.938+0000 7f7fbf200700 1 --2- 172.21.15.12:0/3564745651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4004850 0x7f7fc01561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.938+0000 7f7fbf200700 1 --2- 172.21.15.12:0/3564745651 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fc00cae10 0x7f7fc01f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.938+0000 7f7fbf200700 1 -- 172.21.15.12:0/3564745651 >> 172.21.15.12:0/3564745651 conn(0x7f7fc00668e0 msgr2=0x7f7fc00b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.938+0000 7f7fbf200700 1 -- 172.21.15.12:0/3564745651 shutdown_connections 2024-03-29T09:54:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:02.938+0000 7f7fbf200700 1 -- 172.21.15.12:0/3564745651 wait complete. 2024-03-29T09:54:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2024-03-29T09:54:02.954 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T09:54:02.954 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T09:54:02.955 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:54:02.955 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T09:54:02.985 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:02.986+0000 7f68dcca0240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T09:54:03.022 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:54:03.022 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T09:54:03.132 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:03.133+0000 7fa044cc8240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T09:54:03.140 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:54:03.140 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T09:54:03.246 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:03.246+0000 7f68dcca0240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T09:54:03.260 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:54:03.260 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T09:54:03.340 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:03.340+0000 7fa044cc8240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T09:54:03.367 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:54:03.367 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T09:54:03.433 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:54:03.433 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T09:54:03.539 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:54:03.539 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T09:54:03.649 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:54:03.649 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T09:54:03.700 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:03.701+0000 7f68dcca0240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T09:54:03.756 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new cf9b5216-0622-4882-90da-b5e81467c283 0 2024-03-29T09:54:03.797 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:03.797+0000 7fa044cc8240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T09:54:03.986 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.986+0000 7fb3e2c00700 1 Processor -- start 2024-03-29T09:54:03.986 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.986+0000 7fb3e2c00700 1 -- start start 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3e2c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb3dc0d16a0 0x7fb3dc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3e2c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb3dc0d2370 0x7fb3dc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3e2c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc161420 con 0x7fb3dc15d9d0 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3e2c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3dc0d4340 con 0x7fb3dc0d2370 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3e2c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3dc15edc0 con 0x7fb3dc0d16a0 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3db400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb3dc0d16a0 0x7fb3dc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3db400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb3dc0d16a0 0x7fb3dc0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:50452/0 (socket says 172.21.15.159:50452) 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3db400700 1 -- 172.21.15.159:0/537158900 learned_addr learned my addr 172.21.15.159:0/537158900 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3e0c00700 1 --2- 172.21.15.159:0/537158900 >> v2:172.21.15.12:3300/0 conn(0x7fb3dc0d2370 0x7fb3dc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3db400700 1 -- 172.21.15.159:0/537158900 >> v2:172.21.15.12:3300/0 conn(0x7fb3dc0d2370 msgr2=0x7fb3dc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3db400700 1 --2- 172.21.15.159:0/537158900 >> v2:172.21.15.12:3300/0 conn(0x7fb3dc0d2370 0x7fb3dc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3db400700 1 -- 172.21.15.159:0/537158900 >> v1:172.21.15.12:6789/0 conn(0x7fb3dc15d9d0 legacy=0x7fb3dc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:03.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.988+0000 7fb3db400700 1 -- 172.21.15.159:0/537158900 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc161700 con 0x7fb3dc0d16a0 2024-03-29T09:54:03.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.989+0000 7fb3db400700 1 --2- 172.21.15.159:0/537158900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d16a0 0x7fb3dc0d1e30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0d12f0 tx=0x7fb3c8009bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7c07038f71054bd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:03.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.989+0000 7fb3daa00700 1 -- 172.21.15.159:0/537158900 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8011830 con 0x7fb3dc0d16a0 2024-03-29T09:54:03.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.989+0000 7fb3daa00700 1 -- 172.21.15.159:0/537158900 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c8011990 con 0x7fb3dc0d16a0 2024-03-29T09:54:03.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.989+0000 7fb3daa00700 1 -- 172.21.15.159:0/537158900 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8011d40 con 0x7fb3dc0d16a0 2024-03-29T09:54:03.991 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.989+0000 7fb3e2c00700 1 -- 172.21.15.159:0/537158900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d16a0 msgr2=0x7fb3dc0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:03.991 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.989+0000 7fb3e2c00700 1 --2- 172.21.15.159:0/537158900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d16a0 0x7fb3dc0d1e30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0d12f0 tx=0x7fb3c8009bc0 comp rx=0 tx=0).stop 2024-03-29T09:54:03.991 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.989+0000 7fb3e2c00700 1 -- 172.21.15.159:0/537158900 shutdown_connections 2024-03-29T09:54:03.991 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.989+0000 7fb3e2c00700 1 --2- 172.21.15.159:0/537158900 >> v2:172.21.15.12:3300/0 conn(0x7fb3dc0d2370 0x7fb3dc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:03.991 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.989+0000 7fb3e2c00700 1 --2- 172.21.15.159:0/537158900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d16a0 0x7fb3dc0d1e30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:03.991 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.989+0000 7fb3e2c00700 1 -- 172.21.15.159:0/537158900 >> 172.21.15.159:0/537158900 conn(0x7fb3dc08aff0 msgr2=0x7fb3dc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:03.991 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.990+0000 7fb3e2c00700 1 -- 172.21.15.159:0/537158900 shutdown_connections 2024-03-29T09:54:03.991 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.990+0000 7fb3e2c00700 1 -- 172.21.15.159:0/537158900 wait complete. 2024-03-29T09:54:03.992 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.992+0000 7fb3e2c00700 1 Processor -- start 2024-03-29T09:54:03.992 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.992+0000 7fb3e2c00700 1 -- start start 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3e2c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3dc0d16a0 0x7fb3dc0cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3e2c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d2370 0x7fb3dc14f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3e2c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3dc15d9d0 0x7fb3dc15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3e2c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0cb850 con 0x7fb3dc15d9d0 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3e2c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0cbb30 con 0x7fb3dc0d2370 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3e2c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0cbe10 con 0x7fb3dc0d16a0 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3db400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d2370 0x7fb3dc14f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3dbe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3dc0d16a0 0x7fb3dc0cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3db400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d2370 0x7fb3dc14f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:50464/0 (socket says 172.21.15.159:50464) 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3db400700 1 -- 172.21.15.159:0/1893136651 learned_addr learned my addr 172.21.15.159:0/1893136651 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.994+0000 7fb3e0c00700 1 --2- 172.21.15.159:0/1893136651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3dc15d9d0 0x7fb3dc15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.995+0000 7fb3db400700 1 -- 172.21.15.159:0/1893136651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3dc0d16a0 msgr2=0x7fb3dc0cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:03.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.995+0000 7fb3db400700 1 --2- 172.21.15.159:0/1893136651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3dc0d16a0 0x7fb3dc0cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.995+0000 7fb3db400700 1 -- 172.21.15.159:0/1893136651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3dc15d9d0 msgr2=0x7fb3dc15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.995+0000 7fb3db400700 1 --2- 172.21.15.159:0/1893136651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3dc15d9d0 0x7fb3dc15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.995+0000 7fb3db400700 1 -- 172.21.15.159:0/1893136651 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c80094f0 con 0x7fb3dc0d2370 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.995+0000 7fb3db400700 1 --2- 172.21.15.159:0/1893136651 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d2370 0x7fb3dc14f8f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8013000 tx=0x7fb3c8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.995+0000 7fb3d3e00700 1 -- 172.21.15.159:0/1893136651 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c801a910 con 0x7fb3dc0d2370 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.995+0000 7fb3d3e00700 1 -- 172.21.15.159:0/1893136651 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c801f070 con 0x7fb3dc0d2370 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.995+0000 7fb3d3e00700 1 -- 172.21.15.159:0/1893136651 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c801ac90 con 0x7fb3dc0d2370 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.995+0000 7fb3e2c00700 1 -- 172.21.15.159:0/1893136651 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc0cc0f0 con 0x7fb3dc0d2370 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.996+0000 7fb3e2c00700 1 -- 172.21.15.159:0/1893136651 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc0cb540 con 0x7fb3dc0d2370 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.996+0000 7fb3d3e00700 1 -- 172.21.15.159:0/1893136651 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb3c802e750 con 0x7fb3dc0d2370 2024-03-29T09:54:03.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.996+0000 7fb3d3e00700 1 -- 172.21.15.159:0/1893136651 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb3c801b630 con 0x7fb3dc0d2370 2024-03-29T09:54:03.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.997+0000 7fb3e2c00700 1 -- 172.21.15.159:0/1893136651 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3dc0d1ca0 con 0x7fb3dc0d2370 2024-03-29T09:54:03.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:03.999+0000 7fb3d3e00700 1 -- 172.21.15.159:0/1893136651 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb3c8021070 con 0x7fb3dc0d2370 2024-03-29T09:54:04.086 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.086+0000 7fb3e2c00700 1 -- 172.21.15.159:0/1893136651 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cf9b5216-0622-4882-90da-b5e81467c283", "id": 0} v 0) v1 -- 0x7fb3dc06afa0 con 0x7fb3dc0d2370 2024-03-29T09:54:04.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.094+0000 7fb3d3e00700 1 -- 172.21.15.159:0/1893136651 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cf9b5216-0622-4882-90da-b5e81467c283", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb3c8015020 con 0x7fb3dc0d2370 2024-03-29T09:54:04.094 INFO:teuthology.orchestra.run.smithi159.stdout:0 2024-03-29T09:54:04.095 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.095+0000 7fb3e2c00700 1 -- 172.21.15.159:0/1893136651 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d2370 msgr2=0x7fb3dc14f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.095 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.095+0000 7fb3e2c00700 1 --2- 172.21.15.159:0/1893136651 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d2370 0x7fb3dc14f8f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8013000 tx=0x7fb3c8004750 comp rx=0 tx=0).stop 2024-03-29T09:54:04.095 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.095+0000 7fb3e2c00700 1 -- 172.21.15.159:0/1893136651 shutdown_connections 2024-03-29T09:54:04.095 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.095+0000 7fb3e2c00700 1 --2- 172.21.15.159:0/1893136651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3dc15d9d0 0x7fb3dc15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.095 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.095+0000 7fb3e2c00700 1 --2- 172.21.15.159:0/1893136651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3dc0d16a0 0x7fb3dc0cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.095 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.095+0000 7fb3e2c00700 1 --2- 172.21.15.159:0/1893136651 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3dc0d2370 0x7fb3dc14f8f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.095 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.095+0000 7fb3e2c00700 1 -- 172.21.15.159:0/1893136651 >> 172.21.15.159:0/1893136651 conn(0x7fb3dc08aff0 msgr2=0x7fb3dc0b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:04.096 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.095+0000 7fb3e2c00700 1 -- 172.21.15.159:0/1893136651 shutdown_connections 2024-03-29T09:54:04.096 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.095+0000 7fb3e2c00700 1 -- 172.21.15.159:0/1893136651 wait complete. 2024-03-29T09:54:04.114 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 1607d220-054e-43c6-8e00-45696c8702c0 1 2024-03-29T09:54:04.304 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.304+0000 7f71fa200700 1 Processor -- start 2024-03-29T09:54:04.304 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.304+0000 7f71fa200700 1 -- start start 2024-03-29T09:54:04.306 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.305+0000 7f71fa200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f71f415d9d0 0x7f71f40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.306 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.305+0000 7f71fa200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d16a0 0x7f71f40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.306 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71fa200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f4161420 con 0x7f71f40d2370 2024-03-29T09:54:04.306 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71fa200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f40d2820 con 0x7f71f415d9d0 2024-03-29T09:54:04.307 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71fa200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f4161700 con 0x7f71f40d16a0 2024-03-29T09:54:04.307 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71f2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d16a0 0x7f71f40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.307 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71f2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d16a0 0x7f71f40d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:55828/0 (socket says 172.21.15.159:55828) 2024-03-29T09:54:04.307 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71f2a00700 1 -- 172.21.15.159:0/2655412499 learned_addr learned my addr 172.21.15.159:0/2655412499 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:04.307 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71f3400700 1 --2- 172.21.15.159:0/2655412499 >> v2:172.21.15.12:3300/0 conn(0x7f71f415d9d0 0x7f71f40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.307 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71f2a00700 1 -- 172.21.15.159:0/2655412499 >> v2:172.21.15.12:3300/0 conn(0x7f71f415d9d0 msgr2=0x7f71f40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:04.307 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71f2a00700 1 --2- 172.21.15.159:0/2655412499 >> v2:172.21.15.12:3300/0 conn(0x7f71f415d9d0 0x7f71f40d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.307 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71f2a00700 1 -- 172.21.15.159:0/2655412499 >> v1:172.21.15.12:6789/0 conn(0x7f71f40d2370 legacy=0x7f71f415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.308 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71f2a00700 1 -- 172.21.15.159:0/2655412499 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f41619e0 con 0x7f71f40d16a0 2024-03-29T09:54:04.308 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.306+0000 7f71f2a00700 1 --2- 172.21.15.159:0/2655412499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d16a0 0x7f71f40d1e30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71e0006fd0 tx=0x7f71e0009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac49b50b84f5f5bd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:04.309 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.307+0000 7f71f2000700 1 -- 172.21.15.159:0/2655412499 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e0005e30 con 0x7f71f40d16a0 2024-03-29T09:54:04.309 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.307+0000 7f71f2000700 1 -- 172.21.15.159:0/2655412499 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e0004030 con 0x7f71f40d16a0 2024-03-29T09:54:04.309 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.307+0000 7f71f2000700 1 -- 172.21.15.159:0/2655412499 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e00043e0 con 0x7f71f40d16a0 2024-03-29T09:54:04.310 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.307+0000 7f71fa200700 1 -- 172.21.15.159:0/2655412499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d16a0 msgr2=0x7f71f40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.310 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.307+0000 7f71fa200700 1 --2- 172.21.15.159:0/2655412499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d16a0 0x7f71f40d1e30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71e0006fd0 tx=0x7f71e0009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.310 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.307+0000 7f71fa200700 1 -- 172.21.15.159:0/2655412499 shutdown_connections 2024-03-29T09:54:04.310 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.307+0000 7f71fa200700 1 --2- 172.21.15.159:0/2655412499 >> v2:172.21.15.12:3300/0 conn(0x7f71f415d9d0 0x7f71f40d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.310 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.307+0000 7f71fa200700 1 --2- 172.21.15.159:0/2655412499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d16a0 0x7f71f40d1e30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.310 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.307+0000 7f71fa200700 1 -- 172.21.15.159:0/2655412499 >> 172.21.15.159:0/2655412499 conn(0x7f71f408aff0 msgr2=0x7f71f40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:04.310 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.308+0000 7f71fa200700 1 -- 172.21.15.159:0/2655412499 shutdown_connections 2024-03-29T09:54:04.311 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.308+0000 7f71fa200700 1 -- 172.21.15.159:0/2655412499 wait complete. 2024-03-29T09:54:04.315 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.310+0000 7f71fa200700 1 Processor -- start 2024-03-29T09:54:04.315 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.310+0000 7f71fa200700 1 -- start start 2024-03-29T09:54:04.315 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.311+0000 7f71fa200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f40d16a0 0x7f71f415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.315 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.311+0000 7f71fa200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d2370 0x7f71f40cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.315 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.311+0000 7f71fa200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f415d9d0 0x7f71f415c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.315 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.311+0000 7f71fa200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40cd5e0 con 0x7f71f40d16a0 2024-03-29T09:54:04.316 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.311+0000 7f71fa200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40b8cb0 con 0x7f71f415d9d0 2024-03-29T09:54:04.316 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.311+0000 7f71fa200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40b70d0 con 0x7f71f40d2370 2024-03-29T09:54:04.316 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.311+0000 7f71f2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d2370 0x7f71f40cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.316 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f40d16a0 0x7f71f415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.316 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d2370 0x7f71f40cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:55834/0 (socket says 172.21.15.159:55834) 2024-03-29T09:54:04.316 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f3e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f415d9d0 0x7f71f415c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.316 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f2a00700 1 -- 172.21.15.159:0/1145754511 learned_addr learned my addr 172.21.15.159:0/1145754511 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:04.317 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f3400700 1 -- 172.21.15.159:0/1145754511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d2370 msgr2=0x7f71f40cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.317 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f3400700 1 --2- 172.21.15.159:0/1145754511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d2370 0x7f71f40cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.317 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f3400700 1 -- 172.21.15.159:0/1145754511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f415d9d0 msgr2=0x7f71f415c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.317 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f3400700 1 --2- 172.21.15.159:0/1145754511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f415d9d0 0x7f71f415c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.317 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f3400700 1 -- 172.21.15.159:0/1145754511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e0009500 con 0x7f71f40d16a0 2024-03-29T09:54:04.317 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f2a00700 1 --2- 172.21.15.159:0/1145754511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d2370 0x7f71f40cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:04.318 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.312+0000 7f71f3400700 1 --2- 172.21.15.159:0/1145754511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f40d16a0 0x7f71f415cb00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71e400e870 tx=0x7f71e400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:04.318 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.313+0000 7f71eb400700 1 -- 172.21.15.159:0/1145754511 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4013680 con 0x7f71f40d16a0 2024-03-29T09:54:04.318 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.313+0000 7f71eb400700 1 -- 172.21.15.159:0/1145754511 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e40137e0 con 0x7f71f40d16a0 2024-03-29T09:54:04.318 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.313+0000 7f71fa200700 1 -- 172.21.15.159:0/1145754511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f40b73b0 con 0x7f71f40d16a0 2024-03-29T09:54:04.318 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.313+0000 7f71eb400700 1 -- 172.21.15.159:0/1145754511 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4013b90 con 0x7f71f40d16a0 2024-03-29T09:54:04.318 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.313+0000 7f71fa200700 1 -- 172.21.15.159:0/1145754511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f40ca690 con 0x7f71f40d16a0 2024-03-29T09:54:04.318 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.313+0000 7f71eb400700 1 -- 172.21.15.159:0/1145754511 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f71e4005a20 con 0x7f71f40d16a0 2024-03-29T09:54:04.318 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.313+0000 7f71eb400700 1 -- 172.21.15.159:0/1145754511 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f71e401c800 con 0x7f71f40d16a0 2024-03-29T09:54:04.319 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.314+0000 7f71fa200700 1 -- 172.21.15.159:0/1145754511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f40cace0 con 0x7f71f40d16a0 2024-03-29T09:54:04.319 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.316+0000 7f71eb400700 1 -- 172.21.15.159:0/1145754511 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f71e4021060 con 0x7f71f40d16a0 2024-03-29T09:54:04.334 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:04.334+0000 7f68dcca0240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T09:54:04.401 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.401+0000 7f71fa200700 1 -- 172.21.15.159:0/1145754511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1607d220-054e-43c6-8e00-45696c8702c0", "id": 1} v 0) v1 -- 0x7f71f40d1ca0 con 0x7f71f40d16a0 2024-03-29T09:54:04.406 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.406+0000 7f71eb400700 1 -- 172.21.15.159:0/1145754511 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1607d220-054e-43c6-8e00-45696c8702c0", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f71e4017070 con 0x7f71f40d16a0 2024-03-29T09:54:04.406 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-03-29T09:54:04.408 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.408+0000 7f71fa200700 1 -- 172.21.15.159:0/1145754511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f40d16a0 msgr2=0x7f71f415cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.408 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.408+0000 7f71fa200700 1 --2- 172.21.15.159:0/1145754511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f40d16a0 0x7f71f415cb00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71e400e870 tx=0x7f71e400eb80 comp rx=0 tx=0).stop 2024-03-29T09:54:04.408 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.408+0000 7f71fa200700 1 -- 172.21.15.159:0/1145754511 shutdown_connections 2024-03-29T09:54:04.408 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.408+0000 7f71fa200700 1 --2- 172.21.15.159:0/1145754511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f415d9d0 0x7f71f415c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.408 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.408+0000 7f71fa200700 1 --2- 172.21.15.159:0/1145754511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f40d16a0 0x7f71f415cb00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.409 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.408+0000 7f71fa200700 1 --2- 172.21.15.159:0/1145754511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f40d2370 0x7f71f40cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.409 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.408+0000 7f71fa200700 1 -- 172.21.15.159:0/1145754511 >> 172.21.15.159:0/1145754511 conn(0x7f71f408aff0 msgr2=0x7f71f40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:04.409 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.408+0000 7f71fa200700 1 -- 172.21.15.159:0/1145754511 shutdown_connections 2024-03-29T09:54:04.409 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.408+0000 7f71fa200700 1 -- 172.21.15.159:0/1145754511 wait complete. 2024-03-29T09:54:04.425 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 623142ba-90d6-4b24-8677-0058a5a01056 2 2024-03-29T09:54:04.436 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:04.436+0000 7fa044cc8240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T09:54:04.598 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:04.599+0000 7f68dcca0240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T09:54:04.617 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.617+0000 7fadaa800700 1 Processor -- start 2024-03-29T09:54:04.617 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.618+0000 7fadaa800700 1 -- start start 2024-03-29T09:54:04.619 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.619+0000 7fadaa800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada415ef70 0x7fada40b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.619 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.619+0000 7fadaa800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada4161150 con 0x7fada4160d70 2024-03-29T09:54:04.619 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.619+0000 7fadaa800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada406dba0 con 0x7fada406a790 2024-03-29T09:54:04.619 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.619+0000 7fadaa800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada40b4cd0 con 0x7fada415ef70 2024-03-29T09:54:04.619 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.619+0000 7fada3e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fada406a790 0x7fada406e080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:39152/0 (socket says 172.21.15.159:39152) 2024-03-29T09:54:04.619 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.619+0000 7fada2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada415ef70 0x7fada40b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.619 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.619+0000 7fada3e00700 1 -- 172.21.15.159:0/146880442 learned_addr learned my addr 172.21.15.159:0/146880442 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-29T09:54:04.621 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.619+0000 7fada2000700 1 -- 172.21.15.159:0/146880442 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518820870 0 0) 0x7fada406dba0 con 0x7fada406a790 2024-03-29T09:54:04.621 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.620+0000 7fada2a00700 1 -- 172.21.15.159:0/146880442 >> v1:172.21.15.159:6789/0 conn(0x7fada406a790 legacy=0x7fada406e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.621 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.620+0000 7fada2a00700 1 -- 172.21.15.159:0/146880442 >> v1:172.21.15.12:6789/0 conn(0x7fada4160d70 legacy=0x7fada415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.620+0000 7fada2a00700 1 -- 172.21.15.159:0/146880442 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40b4ee0 con 0x7fada415ef70 2024-03-29T09:54:04.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.620+0000 7fada2a00700 1 --2- 172.21.15.159:0/146880442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada415ef70 0x7fada40b3200 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fad90009870 tx=0x7fad90009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a79d05cdcd61e48 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:04.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.620+0000 7fada2000700 1 -- 172.21.15.159:0/146880442 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90011830 con 0x7fada415ef70 2024-03-29T09:54:04.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.620+0000 7fada2000700 1 -- 172.21.15.159:0/146880442 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad90011990 con 0x7fada415ef70 2024-03-29T09:54:04.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.620+0000 7fada2000700 1 -- 172.21.15.159:0/146880442 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90011d20 con 0x7fada415ef70 2024-03-29T09:54:04.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.621+0000 7fadaa800700 1 -- 172.21.15.159:0/146880442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada415ef70 msgr2=0x7fada40b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.621+0000 7fadaa800700 1 --2- 172.21.15.159:0/146880442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada415ef70 0x7fada40b3200 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fad90009870 tx=0x7fad90009b80 comp rx=0 tx=0).stop 2024-03-29T09:54:04.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.621+0000 7fadaa800700 1 -- 172.21.15.159:0/146880442 shutdown_connections 2024-03-29T09:54:04.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.621+0000 7fadaa800700 1 --2- 172.21.15.159:0/146880442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada415ef70 0x7fada40b3200 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.621+0000 7fadaa800700 1 -- 172.21.15.159:0/146880442 >> 172.21.15.159:0/146880442 conn(0x7fada40b5720 msgr2=0x7fada40b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:04.623 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.621+0000 7fadaa800700 1 -- 172.21.15.159:0/146880442 shutdown_connections 2024-03-29T09:54:04.623 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.621+0000 7fadaa800700 1 -- 172.21.15.159:0/146880442 wait complete. 2024-03-29T09:54:04.625 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.623+0000 7fadaa800700 1 Processor -- start 2024-03-29T09:54:04.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.623+0000 7fadaa800700 1 -- start start 2024-03-29T09:54:04.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fadaa800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fada406a790 0x7fada41f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fadaa800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fada415ef70 0x7fada409a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fadaa800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada4160d70 0x7fada40adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fadaa800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada41f6560 con 0x7fada406a790 2024-03-29T09:54:04.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fadaa800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada41f67b0 con 0x7fada415ef70 2024-03-29T09:54:04.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fadaa800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada41f6a90 con 0x7fada4160d70 2024-03-29T09:54:04.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fada3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada4160d70 0x7fada40adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fada3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada4160d70 0x7fada40adcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:55852/0 (socket says 172.21.15.159:55852) 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fada3e00700 1 -- 172.21.15.159:0/360620405 learned_addr learned my addr 172.21.15.159:0/360620405 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fada2a00700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fada415ef70 0x7fada409a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.624+0000 7fada3400700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fada406a790 0x7fada41f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fada2a00700 1 -- 172.21.15.159:0/360620405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada4160d70 msgr2=0x7fada40adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fada2a00700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada4160d70 0x7fada40adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fada2a00700 1 -- 172.21.15.159:0/360620405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fada406a790 msgr2=0x7fada41f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fada2a00700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fada406a790 0x7fada41f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fada2a00700 1 -- 172.21.15.159:0/360620405 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad900094a0 con 0x7fada415ef70 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fada3400700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fada406a790 0x7fada41f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fada3e00700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada4160d70 0x7fada40adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fada2a00700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fada415ef70 0x7fada409a2f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fad90016000 tx=0x7fad90004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fad9b400700 1 -- 172.21.15.159:0/360620405 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad900038a0 con 0x7fada415ef70 2024-03-29T09:54:04.628 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fad9b400700 1 -- 172.21.15.159:0/360620405 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9001f070 con 0x7fada415ef70 2024-03-29T09:54:04.628 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fad9b400700 1 -- 172.21.15.159:0/360620405 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90003c20 con 0x7fada415ef70 2024-03-29T09:54:04.628 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fadaa800700 1 -- 172.21.15.159:0/360620405 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada41f6d70 con 0x7fada415ef70 2024-03-29T09:54:04.628 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.625+0000 7fadaa800700 1 -- 172.21.15.159:0/360620405 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada41f3d30 con 0x7fada415ef70 2024-03-29T09:54:04.628 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.626+0000 7fad9b400700 1 -- 172.21.15.159:0/360620405 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fad9001a730 con 0x7fada415ef70 2024-03-29T09:54:04.628 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.626+0000 7fad9b400700 1 -- 172.21.15.159:0/360620405 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fad90021070 con 0x7fada415ef70 2024-03-29T09:54:04.628 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.626+0000 7fadaa800700 1 -- 172.21.15.159:0/360620405 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada4067720 con 0x7fada415ef70 2024-03-29T09:54:04.630 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.630+0000 7fad9b400700 1 -- 172.21.15.159:0/360620405 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fad9002a020 con 0x7fada415ef70 2024-03-29T09:54:04.708 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:04.708+0000 7fa044cc8240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T09:54:04.717 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.717+0000 7fadaa800700 1 -- 172.21.15.159:0/360620405 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "623142ba-90d6-4b24-8677-0058a5a01056", "id": 2} v 0) v1 -- 0x7fada40b5ff0 con 0x7fada415ef70 2024-03-29T09:54:04.725 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.725+0000 7fad9b400700 1 -- 172.21.15.159:0/360620405 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "623142ba-90d6-4b24-8677-0058a5a01056", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fad90015020 con 0x7fada415ef70 2024-03-29T09:54:04.725 INFO:teuthology.orchestra.run.smithi159.stdout:2 2024-03-29T09:54:04.726 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.726+0000 7fadaa800700 1 -- 172.21.15.159:0/360620405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fada415ef70 msgr2=0x7fada409a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.727 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.726+0000 7fadaa800700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fada415ef70 0x7fada409a2f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fad90016000 tx=0x7fad90004750 comp rx=0 tx=0).stop 2024-03-29T09:54:04.727 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.726+0000 7fadaa800700 1 -- 172.21.15.159:0/360620405 shutdown_connections 2024-03-29T09:54:04.727 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.726+0000 7fadaa800700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fada4160d70 0x7fada40adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.727 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.726+0000 7fadaa800700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fada406a790 0x7fada41f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.727 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.726+0000 7fadaa800700 1 --2- 172.21.15.159:0/360620405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fada415ef70 0x7fada409a2f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.727 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.726+0000 7fadaa800700 1 -- 172.21.15.159:0/360620405 >> 172.21.15.159:0/360620405 conn(0x7fada40b5720 msgr2=0x7fada415ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:04.727 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.727+0000 7fadaa800700 1 -- 172.21.15.159:0/360620405 shutdown_connections 2024-03-29T09:54:04.727 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.727+0000 7fadaa800700 1 -- 172.21.15.159:0/360620405 wait complete. 2024-03-29T09:54:04.743 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 82460961-6482-46b5-8419-0bb9824c7a6f 3 2024-03-29T09:54:04.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.978+0000 7fe8bca00700 1 Processor -- start 2024-03-29T09:54:04.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.979+0000 7fe8bca00700 1 -- start start 2024-03-29T09:54:04.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8bca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe8b815d710 0x7fe8b80d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8bca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b80d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8bca00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b80cf640 con 0x7fe8b80d1b70 2024-03-29T09:54:04.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8bca00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b80cf220 con 0x7fe8b815d710 2024-03-29T09:54:04.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8bca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b80ce5a0 con 0x7fe8b80d2020 2024-03-29T09:54:04.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8b6800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b80d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8b6800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b80d0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:54134/0 (socket says 172.21.15.159:54134) 2024-03-29T09:54:04.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8b6800700 1 -- 172.21.15.159:0/2884351030 learned_addr learned my addr 172.21.15.159:0/2884351030 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:04.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8b5e00700 1 --2- 172.21.15.159:0/2884351030 >> v2:172.21.15.12:3300/0 conn(0x7fe8b815d710 0x7fe8b80d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8b6800700 1 -- 172.21.15.159:0/2884351030 >> v2:172.21.15.12:3300/0 conn(0x7fe8b815d710 msgr2=0x7fe8b80d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:04.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8b6800700 1 --2- 172.21.15.159:0/2884351030 >> v2:172.21.15.12:3300/0 conn(0x7fe8b815d710 0x7fe8b80d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.980+0000 7fe8b6800700 1 -- 172.21.15.159:0/2884351030 >> v1:172.21.15.12:6789/0 conn(0x7fe8b80d1b70 legacy=0x7fe8b80cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.981+0000 7fe8b6800700 1 -- 172.21.15.159:0/2884351030 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b80ce880 con 0x7fe8b80d2020 2024-03-29T09:54:04.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.981+0000 7fe8b6800700 1 --2- 172.21.15.159:0/2884351030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b80d0620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0009e50 tx=0x7fe8b000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=168e4728538aa3ff server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:04.982 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.981+0000 7fe8b4a00700 1 -- 172.21.15.159:0/2884351030 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b000e070 con 0x7fe8b80d2020 2024-03-29T09:54:04.983 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.981+0000 7fe8b4a00700 1 -- 172.21.15.159:0/2884351030 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b000b910 con 0x7fe8b80d2020 2024-03-29T09:54:04.983 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.981+0000 7fe8b4a00700 1 -- 172.21.15.159:0/2884351030 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b000bcc0 con 0x7fe8b80d2020 2024-03-29T09:54:04.983 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.982+0000 7fe8bca00700 1 -- 172.21.15.159:0/2884351030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 msgr2=0x7fe8b80d0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.983 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.982+0000 7fe8bca00700 1 --2- 172.21.15.159:0/2884351030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b80d0620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0009e50 tx=0x7fe8b000b330 comp rx=0 tx=0).stop 2024-03-29T09:54:04.983 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.982+0000 7fe8bca00700 1 -- 172.21.15.159:0/2884351030 shutdown_connections 2024-03-29T09:54:04.983 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.982+0000 7fe8bca00700 1 --2- 172.21.15.159:0/2884351030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b80d0620 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.983 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.982+0000 7fe8bca00700 1 --2- 172.21.15.159:0/2884351030 >> v2:172.21.15.12:3300/0 conn(0x7fe8b815d710 0x7fe8b80d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.983 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.982+0000 7fe8bca00700 1 -- 172.21.15.159:0/2884351030 >> 172.21.15.159:0/2884351030 conn(0x7fe8b808aff0 msgr2=0x7fe8b80d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:04.983 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.982+0000 7fe8bca00700 1 -- 172.21.15.159:0/2884351030 shutdown_connections 2024-03-29T09:54:04.983 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.982+0000 7fe8bca00700 1 -- 172.21.15.159:0/2884351030 wait complete. 2024-03-29T09:54:04.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.985+0000 7fe8bca00700 1 Processor -- start 2024-03-29T09:54:04.985 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.985+0000 7fe8bca00700 1 -- start start 2024-03-29T09:54:04.988 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.986+0000 7fe8bca00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8b80d1b70 0x7fe8b80c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.988 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.986+0000 7fe8bca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b81f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.988 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.986+0000 7fe8bca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8b815d710 0x7fe8b8146e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:04.988 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.986+0000 7fe8bca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b81f2ff0 con 0x7fe8b815d710 2024-03-29T09:54:04.988 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.986+0000 7fe8bca00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b8147360 con 0x7fe8b80d1b70 2024-03-29T09:54:04.988 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.986+0000 7fe8bca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b80c8ac0 con 0x7fe8b80d2020 2024-03-29T09:54:04.988 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8b80d1b70 0x7fe8b80c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.988 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b81f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.988 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8b80d1b70 0x7fe8b80c9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:47032/0 (socket says 172.21.15.159:47032) 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5e00700 1 -- 172.21.15.159:0/1697956760 learned_addr learned my addr 172.21.15.159:0/1697956760 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b6800700 1 --2- 172.21.15.159:0/1697956760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8b815d710 0x7fe8b8146e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5e00700 1 -- 172.21.15.159:0/1697956760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 msgr2=0x7fe8b81f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5e00700 1 --2- 172.21.15.159:0/1697956760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b81f2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5e00700 1 -- 172.21.15.159:0/1697956760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8b815d710 msgr2=0x7fe8b8146e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5e00700 1 --2- 172.21.15.159:0/1697956760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8b815d710 0x7fe8b8146e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5e00700 1 -- 172.21.15.159:0/1697956760 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b0009b30 con 0x7fe8b80d1b70 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5400700 1 --2- 172.21.15.159:0/1697956760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b81f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.987+0000 7fe8b5e00700 1 --2- 172.21.15.159:0/1697956760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8b80d1b70 0x7fe8b80c9b60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe8a800e8a0 tx=0x7fe8a800ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.988+0000 7fe8ae000700 1 -- 172.21.15.159:0/1697956760 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8009590 con 0x7fe8b80d1b70 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.988+0000 7fe8ae000700 1 -- 172.21.15.159:0/1697956760 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a8013680 con 0x7fe8b80d1b70 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.988+0000 7fe8bca00700 1 -- 172.21.15.159:0/1697956760 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b80c8da0 con 0x7fe8b80d1b70 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.988+0000 7fe8ae000700 1 -- 172.21.15.159:0/1697956760 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a80139f0 con 0x7fe8b80d1b70 2024-03-29T09:54:04.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.988+0000 7fe8bca00700 1 -- 172.21.15.159:0/1697956760 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b80c9360 con 0x7fe8b80d1b70 2024-03-29T09:54:04.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.988+0000 7fe8ae000700 1 -- 172.21.15.159:0/1697956760 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe8a8005a20 con 0x7fe8b80d1b70 2024-03-29T09:54:04.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.988+0000 7fe8ae000700 1 -- 172.21.15.159:0/1697956760 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fe8a8005cd0 con 0x7fe8b80d1b70 2024-03-29T09:54:04.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.989+0000 7fe8bca00700 1 -- 172.21.15.159:0/1697956760 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b8067e30 con 0x7fe8b80d1b70 2024-03-29T09:54:04.991 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:04.991+0000 7fe8ae000700 1 -- 172.21.15.159:0/1697956760 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fe8a8021060 con 0x7fe8b80d1b70 2024-03-29T09:54:05.075 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.075+0000 7fe8bca00700 1 -- 172.21.15.159:0/1697956760 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "82460961-6482-46b5-8419-0bb9824c7a6f", "id": 3} v 0) v1 -- 0x7fe8b80b9510 con 0x7fe8b80d1b70 2024-03-29T09:54:05.080 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:05.081+0000 7f68dcca0240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T09:54:05.083 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.083+0000 7fe8ae000700 1 -- 172.21.15.159:0/1697956760 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "82460961-6482-46b5-8419-0bb9824c7a6f", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe8a8029020 con 0x7fe8b80d1b70 2024-03-29T09:54:05.084 INFO:teuthology.orchestra.run.smithi159.stdout:3 2024-03-29T09:54:05.085 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.084+0000 7fe8bca00700 1 -- 172.21.15.159:0/1697956760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8b80d1b70 msgr2=0x7fe8b80c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.085 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.085+0000 7fe8bca00700 1 --2- 172.21.15.159:0/1697956760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8b80d1b70 0x7fe8b80c9b60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe8a800e8a0 tx=0x7fe8a800ebb0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.085 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.085+0000 7fe8bca00700 1 -- 172.21.15.159:0/1697956760 shutdown_connections 2024-03-29T09:54:05.085 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.085+0000 7fe8bca00700 1 --2- 172.21.15.159:0/1697956760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8b815d710 0x7fe8b8146e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.085 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.085+0000 7fe8bca00700 1 --2- 172.21.15.159:0/1697956760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8b80d1b70 0x7fe8b80c9b60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.085 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.085+0000 7fe8bca00700 1 --2- 172.21.15.159:0/1697956760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8b80d2020 0x7fe8b81f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.085 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.085+0000 7fe8bca00700 1 -- 172.21.15.159:0/1697956760 >> 172.21.15.159:0/1697956760 conn(0x7fe8b808aff0 msgr2=0x7fe8b80b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:05.085 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.085+0000 7fe8bca00700 1 -- 172.21.15.159:0/1697956760 shutdown_connections 2024-03-29T09:54:05.085 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.085+0000 7fe8bca00700 1 -- 172.21.15.159:0/1697956760 wait complete. 2024-03-29T09:54:05.103 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 0bed68a5-f7af-452a-98b2-f26f1b394375 4 2024-03-29T09:54:05.206 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:05.206+0000 7fa044cc8240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T09:54:05.292 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.292+0000 7f680fa00700 1 Processor -- start 2024-03-29T09:54:05.292 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.292+0000 7f680fa00700 1 -- start start 2024-03-29T09:54:05.293 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.293+0000 7f680fa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68080d16a0 0x7f68080d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.293 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.293+0000 7f680fa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6808161420 con 0x7f680815d9d0 2024-03-29T09:54:05.293 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.293+0000 7f680fa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6808161700 con 0x7f68080d2370 2024-03-29T09:54:05.293 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.293+0000 7f680fa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68080d4340 con 0x7f68080d16a0 2024-03-29T09:54:05.293 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.293+0000 7f680da00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f68080d2370 0x7f68080ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:50488/0 (socket says 172.21.15.159:50488) 2024-03-29T09:54:05.293 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.293+0000 7f680da00700 1 -- 172.21.15.159:0/1387902456 learned_addr learned my addr 172.21.15.159:0/1387902456 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-29T09:54:05.293 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.293+0000 7f6807e00700 1 --2- 172.21.15.159:0/1387902456 >> v2:172.21.15.12:3300/0 conn(0x7f68080d16a0 0x7f68080d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.294 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.294+0000 7f6807400700 1 -- 172.21.15.159:0/1387902456 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494119063 0 0) 0x7f6808161700 con 0x7f68080d2370 2024-03-29T09:54:05.294 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.294+0000 7f6807400700 1 -- 172.21.15.159:0/1387902456 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f00035b0 con 0x7f68080d2370 2024-03-29T09:54:05.295 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.294+0000 7f6807e00700 1 -- 172.21.15.159:0/1387902456 >> v1:172.21.15.159:6789/0 conn(0x7f68080d2370 legacy=0x7f68080ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.295 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.294+0000 7f6807e00700 1 -- 172.21.15.159:0/1387902456 >> v1:172.21.15.12:6789/0 conn(0x7f680815d9d0 legacy=0x7f68080d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.295 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.294+0000 7f6807e00700 1 -- 172.21.15.159:0/1387902456 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68081619e0 con 0x7f68080d16a0 2024-03-29T09:54:05.295 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.294+0000 7f6807e00700 1 --2- 172.21.15.159:0/1387902456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 0x7f68080d1e30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009820 tx=0x7f67f4009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123f4be6aa23387f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:05.295 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.295+0000 7f6807400700 1 -- 172.21.15.159:0/1387902456 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4005e30 con 0x7f68080d16a0 2024-03-29T09:54:05.295 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.295+0000 7f6807400700 1 -- 172.21.15.159:0/1387902456 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f4006030 con 0x7f68080d16a0 2024-03-29T09:54:05.295 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.295+0000 7f6807400700 1 -- 172.21.15.159:0/1387902456 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f40063c0 con 0x7f68080d16a0 2024-03-29T09:54:05.296 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.295+0000 7f680fa00700 1 -- 172.21.15.159:0/1387902456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 msgr2=0x7f68080d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.296 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.295+0000 7f680fa00700 1 --2- 172.21.15.159:0/1387902456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 0x7f68080d1e30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009820 tx=0x7f67f4009b30 comp rx=0 tx=0).stop 2024-03-29T09:54:05.296 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.295+0000 7f680fa00700 1 -- 172.21.15.159:0/1387902456 shutdown_connections 2024-03-29T09:54:05.296 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.295+0000 7f680fa00700 1 --2- 172.21.15.159:0/1387902456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 0x7f68080d1e30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.296 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.295+0000 7f680fa00700 1 -- 172.21.15.159:0/1387902456 >> 172.21.15.159:0/1387902456 conn(0x7f680808aff0 msgr2=0x7f68080d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:05.296 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.295+0000 7f680fa00700 1 -- 172.21.15.159:0/1387902456 shutdown_connections 2024-03-29T09:54:05.296 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.295+0000 7f680fa00700 1 -- 172.21.15.159:0/1387902456 wait complete. 2024-03-29T09:54:05.297 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.297+0000 7f680fa00700 1 Processor -- start 2024-03-29T09:54:05.297 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.297+0000 7f680fa00700 1 -- start start 2024-03-29T09:54:05.299 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f680fa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 0x7f680815ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.299 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f680fa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68080d2370 0x7f68080b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f680fa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f680815d9d0 0x7f68080cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f680fa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680815f6e0 con 0x7f68080d16a0 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f680fa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680815f9c0 con 0x7f680815d9d0 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f680fa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680815fca0 con 0x7f68080d2370 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f680d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 0x7f680815ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f680da00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f680815d9d0 0x7f68080cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f680d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 0x7f680815ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.159:46622/0 (socket says 172.21.15.159:46622) 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f680d000700 1 -- 172.21.15.159:0/374396519 learned_addr learned my addr 172.21.15.159:0/374396519 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.298+0000 7f6807e00700 1 --2- 172.21.15.159:0/374396519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68080d2370 0x7f68080b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.299+0000 7f680d000700 1 -- 172.21.15.159:0/374396519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68080d2370 msgr2=0x7f68080b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.299+0000 7f680d000700 1 --2- 172.21.15.159:0/374396519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68080d2370 0x7f68080b4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.299+0000 7f680d000700 1 -- 172.21.15.159:0/374396519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f680815d9d0 msgr2=0x7f68080cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.299+0000 7f680d000700 1 --2- 172.21.15.159:0/374396519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f680815d9d0 0x7f68080cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.300 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.299+0000 7f680d000700 1 -- 172.21.15.159:0/374396519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f4009430 con 0x7f68080d16a0 2024-03-29T09:54:05.301 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.299+0000 7f680d000700 1 --2- 172.21.15.159:0/374396519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 0x7f680815ee50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c450 tx=0x7f67fc00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:05.301 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.299+0000 7f6804c00700 1 -- 172.21.15.159:0/374396519 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0052d0 con 0x7f68080d16a0 2024-03-29T09:54:05.301 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.299+0000 7f6804c00700 1 -- 172.21.15.159:0/374396519 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc005430 con 0x7f68080d16a0 2024-03-29T09:54:05.301 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.299+0000 7f6804c00700 1 -- 172.21.15.159:0/374396519 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0038c0 con 0x7f68080d16a0 2024-03-29T09:54:05.301 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.299+0000 7f680fa00700 1 -- 172.21.15.159:0/374396519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68080ca690 con 0x7f68080d16a0 2024-03-29T09:54:05.301 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.300+0000 7f680fa00700 1 -- 172.21.15.159:0/374396519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680815f390 con 0x7f68080d16a0 2024-03-29T09:54:05.301 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.300+0000 7f6804c00700 1 -- 172.21.15.159:0/374396519 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f67fc003d20 con 0x7f68080d16a0 2024-03-29T09:54:05.301 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.300+0000 7f6804c00700 1 -- 172.21.15.159:0/374396519 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f67fc011cc0 con 0x7f68080d16a0 2024-03-29T09:54:05.301 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.301+0000 7f680fa00700 1 -- 172.21.15.159:0/374396519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68080d1ca0 con 0x7f68080d16a0 2024-03-29T09:54:05.305 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.305+0000 7f6804c00700 1 -- 172.21.15.159:0/374396519 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f67fc017070 con 0x7f68080d16a0 2024-03-29T09:54:05.391 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.392+0000 7f680fa00700 1 -- 172.21.15.159:0/374396519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0bed68a5-f7af-452a-98b2-f26f1b394375", "id": 4} v 0) v1 -- 0x7f68080cabb0 con 0x7f68080d16a0 2024-03-29T09:54:05.399 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.399+0000 7f6804c00700 1 -- 172.21.15.159:0/374396519 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0bed68a5-f7af-452a-98b2-f26f1b394375", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f68080cabb0 con 0x7f68080d16a0 2024-03-29T09:54:05.399 INFO:teuthology.orchestra.run.smithi159.stdout:4 2024-03-29T09:54:05.400 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.400+0000 7f680fa00700 1 -- 172.21.15.159:0/374396519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 msgr2=0x7f680815ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.400 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.400+0000 7f680fa00700 1 --2- 172.21.15.159:0/374396519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 0x7f680815ee50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c450 tx=0x7f67fc00c760 comp rx=0 tx=0).stop 2024-03-29T09:54:05.400 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.400+0000 7f680fa00700 1 -- 172.21.15.159:0/374396519 shutdown_connections 2024-03-29T09:54:05.400 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.400+0000 7f680fa00700 1 --2- 172.21.15.159:0/374396519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f680815d9d0 0x7f68080cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.400 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.400+0000 7f680fa00700 1 --2- 172.21.15.159:0/374396519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68080d16a0 0x7f680815ee50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.401 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.400+0000 7f680fa00700 1 --2- 172.21.15.159:0/374396519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68080d2370 0x7f68080b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.401 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.400+0000 7f680fa00700 1 -- 172.21.15.159:0/374396519 >> 172.21.15.159:0/374396519 conn(0x7f680808aff0 msgr2=0x7f68080b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:05.401 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.400+0000 7f680fa00700 1 -- 172.21.15.159:0/374396519 shutdown_connections 2024-03-29T09:54:05.401 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.400+0000 7f680fa00700 1 -- 172.21.15.159:0/374396519 wait complete. 2024-03-29T09:54:05.417 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new b7c21a7a-17e5-4a72-af50-2200c49f3c88 5 2024-03-29T09:54:05.605 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.605+0000 7f2fd1400700 1 Processor -- start 2024-03-29T09:54:05.605 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.605+0000 7f2fd1400700 1 -- start start 2024-03-29T09:54:05.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.606+0000 7f2fd1400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2fcc0d16a0 0x7f2fcc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.606+0000 7f2fd1400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc161420 con 0x7f2fcc15d9d0 2024-03-29T09:54:05.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.607+0000 7f2fd1400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc161700 con 0x7f2fcc0d2370 2024-03-29T09:54:05.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.607+0000 7f2fd1400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc0d4340 con 0x7f2fcc0d16a0 2024-03-29T09:54:05.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.607+0000 7f2fc9e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2fcc0d16a0 0x7f2fcc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.607+0000 7f2fc9e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2fcc0d16a0 0x7f2fcc0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:47052/0 (socket says 172.21.15.159:47052) 2024-03-29T09:54:05.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.607+0000 7f2fcb200700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f2fcc0d2370 0x7f2fcc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:50494/0 (socket says 172.21.15.159:50494) 2024-03-29T09:54:05.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.607+0000 7f2fc9e00700 1 -- 172.21.15.159:0/366936200 learned_addr learned my addr 172.21.15.159:0/366936200 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:05.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.607+0000 7f2fc9e00700 1 -- 172.21.15.159:0/366936200 >> v1:172.21.15.159:6789/0 conn(0x7f2fcc0d2370 legacy=0x7f2fcc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.607+0000 7f2fc9e00700 1 -- 172.21.15.159:0/366936200 >> v1:172.21.15.12:6789/0 conn(0x7f2fcc15d9d0 legacy=0x7f2fcc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.607+0000 7f2fc9e00700 1 -- 172.21.15.159:0/366936200 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc1619e0 con 0x7f2fcc0d16a0 2024-03-29T09:54:05.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.607+0000 7f2fc9e00700 1 --2- 172.21.15.159:0/366936200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fcc0d16a0 0x7f2fcc0d1e30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80098e0 tx=0x7f2fb8009bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e55360dbf4ce5f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:05.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.608+0000 7f2fc9400700 1 -- 172.21.15.159:0/366936200 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800b8a0 con 0x7f2fcc0d16a0 2024-03-29T09:54:05.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.608+0000 7f2fc9400700 1 -- 172.21.15.159:0/366936200 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb800ba00 con 0x7f2fcc0d16a0 2024-03-29T09:54:05.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.608+0000 7f2fc9400700 1 -- 172.21.15.159:0/366936200 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800bdb0 con 0x7f2fcc0d16a0 2024-03-29T09:54:05.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.608+0000 7f2fd1400700 1 -- 172.21.15.159:0/366936200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fcc0d16a0 msgr2=0x7f2fcc0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.608+0000 7f2fd1400700 1 --2- 172.21.15.159:0/366936200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fcc0d16a0 0x7f2fcc0d1e30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80098e0 tx=0x7f2fb8009bf0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.608+0000 7f2fd1400700 1 -- 172.21.15.159:0/366936200 shutdown_connections 2024-03-29T09:54:05.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.608+0000 7f2fd1400700 1 --2- 172.21.15.159:0/366936200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fcc0d16a0 0x7f2fcc0d1e30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.608+0000 7f2fd1400700 1 -- 172.21.15.159:0/366936200 >> 172.21.15.159:0/366936200 conn(0x7f2fcc08aff0 msgr2=0x7f2fcc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:05.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.608+0000 7f2fd1400700 1 -- 172.21.15.159:0/366936200 shutdown_connections 2024-03-29T09:54:05.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.609+0000 7f2fd1400700 1 -- 172.21.15.159:0/366936200 wait complete. 2024-03-29T09:54:05.610 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.610+0000 7f2fd1400700 1 Processor -- start 2024-03-29T09:54:05.610 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.610+0000 7f2fd1400700 1 -- start start 2024-03-29T09:54:05.611 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.611+0000 7f2fd1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fcc0d16a0 0x7f2fcc1f5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.611 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.611+0000 7f2fd1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fcc0d2370 0x7f2fcc0cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.611 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.611+0000 7f2fd1400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fcc15d9d0 0x7f2fcc0cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.611+0000 7f2fd1400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0b60c0 con 0x7f2fcc0d16a0 2024-03-29T09:54:05.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.611+0000 7f2fd1400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0b63a0 con 0x7f2fcc15d9d0 2024-03-29T09:54:05.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.611+0000 7f2fd1400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0b6680 con 0x7f2fcc0d2370 2024-03-29T09:54:05.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fc9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fcc0d2370 0x7f2fcc0cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fcb200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fcc15d9d0 0x7f2fcc0cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fcc0d16a0 0x7f2fcc1f5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fcc0d16a0 0x7f2fcc1f5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.159:46630/0 (socket says 172.21.15.159:46630) 2024-03-29T09:54:05.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fca800700 1 -- 172.21.15.159:0/542984450 learned_addr learned my addr 172.21.15.159:0/542984450 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fc9e00700 1 -- 172.21.15.159:0/542984450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fcc15d9d0 msgr2=0x7f2fcc0cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fc9e00700 1 --2- 172.21.15.159:0/542984450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fcc15d9d0 0x7f2fcc0cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fc9e00700 1 -- 172.21.15.159:0/542984450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fcc0d16a0 msgr2=0x7f2fcc1f5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fc9e00700 1 --2- 172.21.15.159:0/542984450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fcc0d16a0 0x7f2fcc1f5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fc9e00700 1 -- 172.21.15.159:0/542984450 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb80094f0 con 0x7f2fcc0d2370 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fca800700 1 --2- 172.21.15.159:0/542984450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fcc0d16a0 0x7f2fcc1f5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.612+0000 7f2fc9e00700 1 --2- 172.21.15.159:0/542984450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fcc0d2370 0x7f2fcc0cbeb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8016000 tx=0x7f2fb8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.613+0000 7f2fc2a00700 1 -- 172.21.15.159:0/542984450 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8011cd0 con 0x7f2fcc0d2370 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.613+0000 7f2fc2a00700 1 -- 172.21.15.159:0/542984450 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb801f070 con 0x7f2fcc0d2370 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.613+0000 7f2fc2a00700 1 -- 172.21.15.159:0/542984450 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80266c0 con 0x7f2fcc0d2370 2024-03-29T09:54:05.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.613+0000 7f2fd1400700 1 -- 172.21.15.159:0/542984450 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc0b6960 con 0x7f2fcc0d2370 2024-03-29T09:54:05.615 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.613+0000 7f2fd1400700 1 -- 172.21.15.159:0/542984450 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc0cc3f0 con 0x7f2fcc0d2370 2024-03-29T09:54:05.615 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.613+0000 7f2fc2a00700 1 -- 172.21.15.159:0/542984450 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2fb8026b20 con 0x7f2fcc0d2370 2024-03-29T09:54:05.615 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.613+0000 7f2fc2a00700 1 -- 172.21.15.159:0/542984450 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f2fb8021070 con 0x7f2fcc0d2370 2024-03-29T09:54:05.615 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.614+0000 7f2fd1400700 1 -- 172.21.15.159:0/542984450 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc0d1ca0 con 0x7f2fcc0d2370 2024-03-29T09:54:05.616 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.617+0000 7f2fc2a00700 1 -- 172.21.15.159:0/542984450 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f2fb802b020 con 0x7f2fcc0d2370 2024-03-29T09:54:05.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.703+0000 7f2fd1400700 1 -- 172.21.15.159:0/542984450 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b7c21a7a-17e5-4a72-af50-2200c49f3c88", "id": 5} v 0) v1 -- 0x7f2fcc0b6e80 con 0x7f2fcc0d2370 2024-03-29T09:54:05.710 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.711+0000 7f2fc2a00700 1 -- 172.21.15.159:0/542984450 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7c21a7a-17e5-4a72-af50-2200c49f3c88", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2fcc0b6e80 con 0x7f2fcc0d2370 2024-03-29T09:54:05.711 INFO:teuthology.orchestra.run.smithi159.stdout:5 2024-03-29T09:54:05.712 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.712+0000 7f2fd1400700 1 -- 172.21.15.159:0/542984450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fcc0d2370 msgr2=0x7f2fcc0cbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.712 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.712+0000 7f2fd1400700 1 --2- 172.21.15.159:0/542984450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fcc0d2370 0x7f2fcc0cbeb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8016000 tx=0x7f2fb8004750 comp rx=0 tx=0).stop 2024-03-29T09:54:05.712 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.712+0000 7f2fd1400700 1 -- 172.21.15.159:0/542984450 shutdown_connections 2024-03-29T09:54:05.712 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.712+0000 7f2fd1400700 1 --2- 172.21.15.159:0/542984450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fcc15d9d0 0x7f2fcc0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.712 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.712+0000 7f2fd1400700 1 --2- 172.21.15.159:0/542984450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fcc0d16a0 0x7f2fcc1f5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.712 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.712+0000 7f2fd1400700 1 --2- 172.21.15.159:0/542984450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fcc0d2370 0x7f2fcc0cbeb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.712 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.712+0000 7f2fd1400700 1 -- 172.21.15.159:0/542984450 >> 172.21.15.159:0/542984450 conn(0x7f2fcc08aff0 msgr2=0x7f2fcc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:05.712 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.712+0000 7f2fd1400700 1 -- 172.21.15.159:0/542984450 shutdown_connections 2024-03-29T09:54:05.713 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.712+0000 7f2fd1400700 1 -- 172.21.15.159:0/542984450 wait complete. 2024-03-29T09:54:05.729 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 7f90aabb-c34c-4700-8bfa-a1f4d4549820 6 2024-03-29T09:54:05.923 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.923+0000 7f7e8fa00700 1 Processor -- start 2024-03-29T09:54:05.923 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.923+0000 7f7e8fa00700 1 -- start start 2024-03-29T09:54:05.924 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.924+0000 7f7e8fa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e8815d710 0x7f7e880d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.924+0000 7f7e8fa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7e880d1ff0 0x7f7e880d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.924+0000 7f7e8fa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e880cf640 con 0x7f7e880d1b70 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.924+0000 7f7e8fa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e880cf220 con 0x7f7e880d1ff0 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.924+0000 7f7e8fa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e880ce5a0 con 0x7f7e8815d710 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.924+0000 7f7e8d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e8815d710 0x7f7e880d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.924+0000 7f7e8d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e8815d710 0x7f7e880d12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:54166/0 (socket says 172.21.15.159:54166) 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.924+0000 7f7e8d000700 1 -- 172.21.15.159:0/620709302 learned_addr learned my addr 172.21.15.159:0/620709302 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.924+0000 7f7e8da00700 1 --2- 172.21.15.159:0/620709302 >> v2:172.21.15.159:3300/0 conn(0x7f7e880d1ff0 0x7f7e880d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.925+0000 7f7e8d000700 1 -- 172.21.15.159:0/620709302 >> v2:172.21.15.159:3300/0 conn(0x7f7e880d1ff0 msgr2=0x7f7e880d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.925+0000 7f7e8d000700 1 --2- 172.21.15.159:0/620709302 >> v2:172.21.15.159:3300/0 conn(0x7f7e880d1ff0 0x7f7e880d0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.925+0000 7f7e8d000700 1 -- 172.21.15.159:0/620709302 >> v1:172.21.15.12:6789/0 conn(0x7f7e880d1b70 legacy=0x7f7e880cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.925+0000 7f7e8d000700 1 -- 172.21.15.159:0/620709302 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e880ce880 con 0x7f7e8815d710 2024-03-29T09:54:05.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.925+0000 7f7e8d000700 1 --2- 172.21.15.159:0/620709302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e8815d710 0x7f7e880d12f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0098f0 tx=0x7f7e7c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=403e0947d0af8da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.925+0000 7f7e87400700 1 -- 172.21.15.159:0/620709302 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c011830 con 0x7f7e8815d710 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.925+0000 7f7e87400700 1 -- 172.21.15.159:0/620709302 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c011990 con 0x7f7e8815d710 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.925+0000 7f7e87400700 1 -- 172.21.15.159:0/620709302 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c011d20 con 0x7f7e8815d710 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.926+0000 7f7e8fa00700 1 -- 172.21.15.159:0/620709302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e8815d710 msgr2=0x7f7e880d12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.926+0000 7f7e8fa00700 1 --2- 172.21.15.159:0/620709302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e8815d710 0x7f7e880d12f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0098f0 tx=0x7f7e7c009c00 comp rx=0 tx=0).stop 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.926+0000 7f7e8fa00700 1 -- 172.21.15.159:0/620709302 shutdown_connections 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.926+0000 7f7e8fa00700 1 --2- 172.21.15.159:0/620709302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e8815d710 0x7f7e880d12f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.926+0000 7f7e8fa00700 1 --2- 172.21.15.159:0/620709302 >> v2:172.21.15.159:3300/0 conn(0x7f7e880d1ff0 0x7f7e880d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.926+0000 7f7e8fa00700 1 -- 172.21.15.159:0/620709302 >> 172.21.15.159:0/620709302 conn(0x7f7e8808aff0 msgr2=0x7f7e880d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.926+0000 7f7e8fa00700 1 -- 172.21.15.159:0/620709302 shutdown_connections 2024-03-29T09:54:05.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.926+0000 7f7e8fa00700 1 -- 172.21.15.159:0/620709302 wait complete. 2024-03-29T09:54:05.928 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.928+0000 7f7e8fa00700 1 Processor -- start 2024-03-29T09:54:05.928 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.928+0000 7f7e8fa00700 1 -- start start 2024-03-29T09:54:05.930 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e8fa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e880d1b70 0x7f7e881f6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.930 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e8fa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e880d1ff0 0x7f7e880b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.930 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e8fa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e8815d710 0x7f7e880b7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e8fa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e880b8ae0 con 0x7f7e8815d710 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e8fa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e881f6460 con 0x7f7e880d1ff0 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e8fa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e880b7c40 con 0x7f7e880d1b70 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e8d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e880d1b70 0x7f7e881f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e8d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e880d1b70 0x7f7e881f6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:54170/0 (socket says 172.21.15.159:54170) 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e8d000700 1 -- 172.21.15.159:0/1226451462 learned_addr learned my addr 172.21.15.159:0/1226451462 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e87e00700 1 --2- 172.21.15.159:0/1226451462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e880d1ff0 0x7f7e880b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.929+0000 7f7e8da00700 1 --2- 172.21.15.159:0/1226451462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e8815d710 0x7f7e880b7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e8d000700 1 -- 172.21.15.159:0/1226451462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e880d1ff0 msgr2=0x7f7e880b71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e8d000700 1 --2- 172.21.15.159:0/1226451462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e880d1ff0 0x7f7e880b71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e8d000700 1 -- 172.21.15.159:0/1226451462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e8815d710 msgr2=0x7f7e880b7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e8d000700 1 --2- 172.21.15.159:0/1226451462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e8815d710 0x7f7e880b7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e8d000700 1 -- 172.21.15.159:0/1226451462 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c009550 con 0x7f7e880d1b70 2024-03-29T09:54:05.931 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e8d000700 1 --2- 172.21.15.159:0/1226451462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e880d1b70 0x7f7e881f6910 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c014040 tx=0x7f7e7c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:05.932 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e84c00700 1 -- 172.21.15.159:0/1226451462 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c004400 con 0x7f7e880d1b70 2024-03-29T09:54:05.932 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e84c00700 1 -- 172.21.15.159:0/1226451462 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c01f070 con 0x7f7e880d1b70 2024-03-29T09:54:05.932 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e8fa00700 1 -- 172.21.15.159:0/1226451462 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e880b7ed0 con 0x7f7e880d1b70 2024-03-29T09:54:05.932 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e84c00700 1 -- 172.21.15.159:0/1226451462 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0038e0 con 0x7f7e880d1b70 2024-03-29T09:54:05.932 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.930+0000 7f7e8fa00700 1 -- 172.21.15.159:0/1226451462 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e880b8490 con 0x7f7e880d1b70 2024-03-29T09:54:05.932 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.931+0000 7f7e84c00700 1 -- 172.21.15.159:0/1226451462 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7e7c021030 con 0x7f7e880d1b70 2024-03-29T09:54:05.932 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.931+0000 7f7e84c00700 1 -- 172.21.15.159:0/1226451462 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f7e7c004580 con 0x7f7e880d1b70 2024-03-29T09:54:05.932 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.931+0000 7f7e8fa00700 1 -- 172.21.15.159:0/1226451462 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e881f42a0 con 0x7f7e880d1b70 2024-03-29T09:54:05.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:05.934+0000 7f7e84c00700 1 -- 172.21.15.159:0/1226451462 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f7e7c015020 con 0x7f7e880d1b70 2024-03-29T09:54:05.936 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:05.937+0000 7f68dcca0240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T09:54:06.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.019+0000 7f7e8fa00700 1 -- 172.21.15.159:0/1226451462 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7f90aabb-c34c-4700-8bfa-a1f4d4549820", "id": 6} v 0) v1 -- 0x7f7e88067e30 con 0x7f7e880d1b70 2024-03-29T09:54:06.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.024+0000 7f7e84c00700 1 -- 172.21.15.159:0/1226451462 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f90aabb-c34c-4700-8bfa-a1f4d4549820", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7e7c003d40 con 0x7f7e880d1b70 2024-03-29T09:54:06.023 INFO:teuthology.orchestra.run.smithi159.stdout:6 2024-03-29T09:54:06.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.025+0000 7f7e8fa00700 1 -- 172.21.15.159:0/1226451462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e880d1b70 msgr2=0x7f7e881f6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.025+0000 7f7e8fa00700 1 --2- 172.21.15.159:0/1226451462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e880d1b70 0x7f7e881f6910 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c014040 tx=0x7f7e7c004750 comp rx=0 tx=0).stop 2024-03-29T09:54:06.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.025+0000 7f7e8fa00700 1 -- 172.21.15.159:0/1226451462 shutdown_connections 2024-03-29T09:54:06.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.025+0000 7f7e8fa00700 1 --2- 172.21.15.159:0/1226451462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e8815d710 0x7f7e880b7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.025+0000 7f7e8fa00700 1 --2- 172.21.15.159:0/1226451462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e880d1b70 0x7f7e881f6910 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.025+0000 7f7e8fa00700 1 --2- 172.21.15.159:0/1226451462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e880d1ff0 0x7f7e880b71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.025+0000 7f7e8fa00700 1 -- 172.21.15.159:0/1226451462 >> 172.21.15.159:0/1226451462 conn(0x7f7e8808aff0 msgr2=0x7f7e880cfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:06.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.025+0000 7f7e8fa00700 1 -- 172.21.15.159:0/1226451462 shutdown_connections 2024-03-29T09:54:06.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.025+0000 7f7e8fa00700 1 -- 172.21.15.159:0/1226451462 wait complete. 2024-03-29T09:54:06.033 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:06.034+0000 7f68dcca0240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T09:54:06.042 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new fa2eb995-510c-426b-9027-c75b1aaa9360 7 2024-03-29T09:54:06.076 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:06.076+0000 7fa044cc8240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T09:54:06.174 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:06.174+0000 7fa044cc8240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T09:54:06.211 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:06.212+0000 7f68dcca0240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T09:54:06.230 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.231+0000 7fe1fca00700 1 Processor -- start 2024-03-29T09:54:06.231 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.231+0000 7fe1fca00700 1 -- start start 2024-03-29T09:54:06.233 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.232+0000 7fe1fca00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe1f815d9d0 0x7fe1f80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:06.233 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.232+0000 7fe1fca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe1f80d16a0 0x7fe1f80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:06.233 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.232+0000 7fe1fca00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f8161420 con 0x7fe1f80d2370 2024-03-29T09:54:06.233 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.232+0000 7fe1fca00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f80d27e0 con 0x7fe1f80d16a0 2024-03-29T09:54:06.233 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.232+0000 7fe1fca00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f8161700 con 0x7fe1f815d9d0 2024-03-29T09:54:06.233 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.232+0000 7fe1f5e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe1f815d9d0 0x7fe1f80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:06.233 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.232+0000 7fe1f6800700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fe1f80d2370 0x7fe1f815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:50506/0 (socket says 172.21.15.159:50506) 2024-03-29T09:54:06.233 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.232+0000 7fe1f6800700 1 -- 172.21.15.159:0/2230357443 learned_addr learned my addr 172.21.15.159:0/2230357443 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-29T09:54:06.233 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.232+0000 7fe1f5400700 1 --2- 172.21.15.159:0/2230357443 >> v2:172.21.15.12:3300/0 conn(0x7fe1f80d16a0 0x7fe1f80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:06.233 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.233+0000 7fe1f5e00700 1 -- 172.21.15.159:0/2230357443 >> v2:172.21.15.12:3300/0 conn(0x7fe1f80d16a0 msgr2=0x7fe1f80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:06.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.233+0000 7fe1f5e00700 1 --2- 172.21.15.159:0/2230357443 >> v2:172.21.15.12:3300/0 conn(0x7fe1f80d16a0 0x7fe1f80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.233+0000 7fe1f5e00700 1 -- 172.21.15.159:0/2230357443 >> v1:172.21.15.159:6789/0 conn(0x7fe1f80d2370 legacy=0x7fe1f815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.233+0000 7fe1f5e00700 1 -- 172.21.15.159:0/2230357443 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f81619e0 con 0x7fe1f815d9d0 2024-03-29T09:54:06.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.233+0000 7fe1f4a00700 1 -- 172.21.15.159:0/2230357443 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437400200 0 0) 0x7fe1f8161420 con 0x7fe1f80d2370 2024-03-29T09:54:06.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.233+0000 7fe1f5e00700 1 --2- 172.21.15.159:0/2230357443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 0x7fe1f80d3c30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80d0ff0 tx=0x7fe1e8009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5837153eafccc24c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:06.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.233+0000 7fe1f4a00700 1 -- 172.21.15.159:0/2230357443 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8011830 con 0x7fe1f815d9d0 2024-03-29T09:54:06.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.233+0000 7fe1f4a00700 1 -- 172.21.15.159:0/2230357443 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e8011990 con 0x7fe1f815d9d0 2024-03-29T09:54:06.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.233+0000 7fe1f4a00700 1 -- 172.21.15.159:0/2230357443 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8011d20 con 0x7fe1f815d9d0 2024-03-29T09:54:06.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.234+0000 7fe1fca00700 1 -- 172.21.15.159:0/2230357443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 msgr2=0x7fe1f80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.234+0000 7fe1fca00700 1 --2- 172.21.15.159:0/2230357443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 0x7fe1f80d3c30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80d0ff0 tx=0x7fe1e8009b20 comp rx=0 tx=0).stop 2024-03-29T09:54:06.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.234+0000 7fe1fca00700 1 -- 172.21.15.159:0/2230357443 shutdown_connections 2024-03-29T09:54:06.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.234+0000 7fe1fca00700 1 --2- 172.21.15.159:0/2230357443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 0x7fe1f80d3c30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.234+0000 7fe1fca00700 1 --2- 172.21.15.159:0/2230357443 >> v2:172.21.15.12:3300/0 conn(0x7fe1f80d16a0 0x7fe1f80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.236 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.234+0000 7fe1fca00700 1 -- 172.21.15.159:0/2230357443 >> 172.21.15.159:0/2230357443 conn(0x7fe1f808aff0 msgr2=0x7fe1f80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:06.236 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.234+0000 7fe1fca00700 1 -- 172.21.15.159:0/2230357443 shutdown_connections 2024-03-29T09:54:06.236 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.234+0000 7fe1fca00700 1 -- 172.21.15.159:0/2230357443 wait complete. 2024-03-29T09:54:06.236 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.236+0000 7fe1fca00700 1 Processor -- start 2024-03-29T09:54:06.236 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.236+0000 7fe1fca00700 1 -- start start 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1fca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1f80d16a0 0x7fe1f8154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1fca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1f80d2370 0x7fe1f81e9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1fca00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 0x7fe1f81553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1fca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f81e94c0 con 0x7fe1f80d16a0 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1fca00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f81e88b0 con 0x7fe1f815d9d0 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1fca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f81e8b90 con 0x7fe1f80d2370 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f6800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 0x7fe1f81553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1f80d2370 0x7fe1f81e9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1f80d2370 0x7fe1f81e9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:54180/0 (socket says 172.21.15.159:54180) 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f6800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 0x7fe1f81553f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:47098/0 (socket says 172.21.15.159:47098) 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f5400700 1 -- 172.21.15.159:0/1048702326 learned_addr learned my addr 172.21.15.159:0/1048702326 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-29T09:54:06.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f5e00700 1 --2- 172.21.15.159:0/1048702326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1f80d16a0 0x7fe1f8154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f5400700 1 -- 172.21.15.159:0/1048702326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 msgr2=0x7fe1f81553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f5400700 1 --2- 172.21.15.159:0/1048702326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 0x7fe1f81553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f5400700 1 -- 172.21.15.159:0/1048702326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1f80d16a0 msgr2=0x7fe1f8154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f5400700 1 --2- 172.21.15.159:0/1048702326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1f80d16a0 0x7fe1f8154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f5400700 1 -- 172.21.15.159:0/1048702326 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e80094a0 con 0x7fe1f80d2370 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.237+0000 7fe1f6800700 1 --2- 172.21.15.159:0/1048702326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 0x7fe1f81553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.238+0000 7fe1f5400700 1 --2- 172.21.15.159:0/1048702326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1f80d2370 0x7fe1f81e9950 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400e960 tx=0x7fe1e400ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.238+0000 7fe1ee000700 1 -- 172.21.15.159:0/1048702326 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e40096f0 con 0x7fe1f80d2370 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.238+0000 7fe1ee000700 1 -- 172.21.15.159:0/1048702326 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e4013680 con 0x7fe1f80d2370 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.238+0000 7fe1fca00700 1 -- 172.21.15.159:0/1048702326 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f81e8e70 con 0x7fe1f80d2370 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.238+0000 7fe1ee000700 1 -- 172.21.15.159:0/1048702326 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4013a30 con 0x7fe1f80d2370 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.238+0000 7fe1fca00700 1 -- 172.21.15.159:0/1048702326 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f815e940 con 0x7fe1f80d2370 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.238+0000 7fe1ee000700 1 -- 172.21.15.159:0/1048702326 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe1e401c450 con 0x7fe1f80d2370 2024-03-29T09:54:06.240 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.238+0000 7fe1ee000700 1 -- 172.21.15.159:0/1048702326 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fe1e400cc20 con 0x7fe1f80d2370 2024-03-29T09:54:06.241 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.239+0000 7fe1fca00700 1 -- 172.21.15.159:0/1048702326 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f80d1ca0 con 0x7fe1f80d2370 2024-03-29T09:54:06.241 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.241+0000 7fe1ee000700 1 -- 172.21.15.159:0/1048702326 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fe1e4021060 con 0x7fe1f80d2370 2024-03-29T09:54:06.328 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.328+0000 7fe1fca00700 1 -- 172.21.15.159:0/1048702326 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "fa2eb995-510c-426b-9027-c75b1aaa9360", "id": 7} v 0) v1 -- 0x7fe1f806afa0 con 0x7fe1f80d2370 2024-03-29T09:54:06.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.332+0000 7fe1ee000700 1 -- 172.21.15.159:0/1048702326 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fa2eb995-510c-426b-9027-c75b1aaa9360", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe1e4017070 con 0x7fe1f80d2370 2024-03-29T09:54:06.332 INFO:teuthology.orchestra.run.smithi159.stdout:7 2024-03-29T09:54:06.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.333+0000 7fe1fca00700 1 -- 172.21.15.159:0/1048702326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1f80d2370 msgr2=0x7fe1f81e9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.333+0000 7fe1fca00700 1 --2- 172.21.15.159:0/1048702326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1f80d2370 0x7fe1f81e9950 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400e960 tx=0x7fe1e400ec70 comp rx=0 tx=0).stop 2024-03-29T09:54:06.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.334+0000 7fe1fca00700 1 -- 172.21.15.159:0/1048702326 shutdown_connections 2024-03-29T09:54:06.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.334+0000 7fe1fca00700 1 --2- 172.21.15.159:0/1048702326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1f815d9d0 0x7fe1f81553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.334+0000 7fe1fca00700 1 --2- 172.21.15.159:0/1048702326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1f80d16a0 0x7fe1f8154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.334+0000 7fe1fca00700 1 --2- 172.21.15.159:0/1048702326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1f80d2370 0x7fe1f81e9950 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.334+0000 7fe1fca00700 1 -- 172.21.15.159:0/1048702326 >> 172.21.15.159:0/1048702326 conn(0x7fe1f808aff0 msgr2=0x7fe1f80b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:06.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.334+0000 7fe1fca00700 1 -- 172.21.15.159:0/1048702326 shutdown_connections 2024-03-29T09:54:06.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-29T09:54:06.334+0000 7fe1fca00700 1 -- 172.21.15.159:0/1048702326 wait complete. 2024-03-29T09:54:06.350 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T09:54:06.350 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T09:54:06.352 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:06.353+0000 7fa044cc8240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T09:54:06.354 INFO:tasks.ceph.osd.0:Started 2024-03-29T09:54:06.354 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T09:54:06.355 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T09:54:06.358 INFO:tasks.ceph.osd.1:Started 2024-03-29T09:54:06.358 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T09:54:06.358 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T09:54:06.363 INFO:tasks.ceph.osd.2:Started 2024-03-29T09:54:06.363 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T09:54:06.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T09:54:06.368 INFO:tasks.ceph.osd.3:Started 2024-03-29T09:54:06.368 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T09:54:06.368 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T09:54:06.372 INFO:tasks.ceph.osd.4:Started 2024-03-29T09:54:06.372 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T09:54:06.372 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T09:54:06.376 INFO:tasks.ceph.osd.5:Started 2024-03-29T09:54:06.376 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T09:54:06.376 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T09:54:06.380 INFO:tasks.ceph.osd.6:Started 2024-03-29T09:54:06.380 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T09:54:06.380 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T09:54:06.384 INFO:tasks.ceph.osd.7:Started 2024-03-29T09:54:06.385 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:06.458 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:06.458+0000 7f68dcca0240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T09:54:06.593 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:06.594+0000 7f68dcca0240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T09:54:06.602 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:06.602+0000 7fa044cc8240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T09:54:06.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.653+0000 7fd9de600700 1 Processor -- start 2024-03-29T09:54:06.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.653+0000 7fd9de600700 1 -- start start 2024-03-29T09:54:06.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.654+0000 7fd9de600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd9d80d18b0 0x7fd9d80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:06.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.654+0000 7fd9de600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d8161590 0x7fd9d8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:06.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.654+0000 7fd9de600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d80ceda0 con 0x7fd9d8160f10 2024-03-29T09:54:06.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.654+0000 7fd9de600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d80d1d30 con 0x7fd9d80d18b0 2024-03-29T09:54:06.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.654+0000 7fd9de600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d8161f60 con 0x7fd9d8161590 2024-03-29T09:54:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d7400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd9d8160f10 0x7fd9d80d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42094/0 (socket says 172.21.15.12:42094) 2024-03-29T09:54:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d7400700 1 -- 172.21.15.12:0/802580139 learned_addr learned my addr 172.21.15.12:0/802580139 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d7e00700 1 --2- 172.21.15.12:0/802580139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d8161590 0x7fd9d8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d6a00700 1 --2- 172.21.15.12:0/802580139 >> v2:172.21.15.159:3300/0 conn(0x7fd9d80d18b0 0x7fd9d80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d6000700 1 -- 172.21.15.12:0/802580139 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717099000 0 0) 0x7fd9d80ceda0 con 0x7fd9d8160f10 2024-03-29T09:54:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d6000700 1 -- 172.21.15.12:0/802580139 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9bc0035b0 con 0x7fd9d8160f10 2024-03-29T09:54:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d7e00700 1 -- 172.21.15.12:0/802580139 >> v2:172.21.15.159:3300/0 conn(0x7fd9d80d18b0 msgr2=0x7fd9d80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d7e00700 1 --2- 172.21.15.12:0/802580139 >> v2:172.21.15.159:3300/0 conn(0x7fd9d80d18b0 0x7fd9d80cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d7e00700 1 -- 172.21.15.12:0/802580139 >> v1:172.21.15.12:6789/0 conn(0x7fd9d8160f10 legacy=0x7fd9d80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d7e00700 1 -- 172.21.15.12:0/802580139 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d81612f0 con 0x7fd9d8161590 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.655+0000 7fd9d7e00700 1 --2- 172.21.15.12:0/802580139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d8161590 0x7fd9d8161a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009d90 tx=0x7fd9d000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30b671cff0fb0b64 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.656+0000 7fd9d6000700 1 -- 172.21.15.12:0/802580139 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d000e070 con 0x7fd9d8161590 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.656+0000 7fd9d6000700 1 -- 172.21.15.12:0/802580139 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d000bb20 con 0x7fd9d8161590 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.656+0000 7fd9d6000700 1 -- 172.21.15.12:0/802580139 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0012430 con 0x7fd9d8161590 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.656+0000 7fd9de600700 1 -- 172.21.15.12:0/802580139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d8161590 msgr2=0x7fd9d8161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.656+0000 7fd9de600700 1 --2- 172.21.15.12:0/802580139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d8161590 0x7fd9d8161a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009d90 tx=0x7fd9d000b330 comp rx=0 tx=0).stop 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.657+0000 7fd9de600700 1 -- 172.21.15.12:0/802580139 shutdown_connections 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.657+0000 7fd9de600700 1 --2- 172.21.15.12:0/802580139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d8161590 0x7fd9d8161a20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.657+0000 7fd9de600700 1 --2- 172.21.15.12:0/802580139 >> v2:172.21.15.159:3300/0 conn(0x7fd9d80d18b0 0x7fd9d80cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.657+0000 7fd9de600700 1 -- 172.21.15.12:0/802580139 >> 172.21.15.12:0/802580139 conn(0x7fd9d808aff0 msgr2=0x7fd9d80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:06.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.657+0000 7fd9de600700 1 -- 172.21.15.12:0/802580139 shutdown_connections 2024-03-29T09:54:06.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.657+0000 7fd9de600700 1 -- 172.21.15.12:0/802580139 wait complete. 2024-03-29T09:54:06.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.658+0000 7fd9de600700 1 Processor -- start 2024-03-29T09:54:06.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.659+0000 7fd9de600700 1 -- start start 2024-03-29T09:54:06.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.659+0000 7fd9de600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d80d18b0 0x7fd9d8158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:06.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.659+0000 7fd9de600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9d8160f10 0x7fd9d80ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:06.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.659+0000 7fd9de600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9d8161590 0x7fd9d80cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:06.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.659+0000 7fd9de600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d809e3e0 con 0x7fd9d8161590 2024-03-29T09:54:06.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.659+0000 7fd9de600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d815f710 con 0x7fd9d8160f10 2024-03-29T09:54:06.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.659+0000 7fd9de600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d815f9f0 con 0x7fd9d80d18b0 2024-03-29T09:54:06.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d80d18b0 0x7fd9d8158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:06.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d80d18b0 0x7fd9d8158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:55216/0 (socket says 172.21.15.12:55216) 2024-03-29T09:54:06.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d7400700 1 -- 172.21.15.12:0/1280899191 learned_addr learned my addr 172.21.15.12:0/1280899191 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:06.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d7e00700 1 --2- 172.21.15.12:0/1280899191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9d8161590 0x7fd9d80cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:06.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d6a00700 1 --2- 172.21.15.12:0/1280899191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9d8160f10 0x7fd9d80ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:06.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d7400700 1 -- 172.21.15.12:0/1280899191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9d8160f10 msgr2=0x7fd9d80ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d7400700 1 --2- 172.21.15.12:0/1280899191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9d8160f10 0x7fd9d80ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d7400700 1 -- 172.21.15.12:0/1280899191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9d8161590 msgr2=0x7fd9d80cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d7400700 1 --2- 172.21.15.12:0/1280899191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9d8161590 0x7fd9d80cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d7400700 1 -- 172.21.15.12:0/1280899191 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0009a70 con 0x7fd9d80d18b0 2024-03-29T09:54:06.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.660+0000 7fd9d7e00700 1 --2- 172.21.15.12:0/1280899191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9d8161590 0x7fd9d80cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:06.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.661+0000 7fd9d7400700 1 --2- 172.21.15.12:0/1280899191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d80d18b0 0x7fd9d8158fc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8001eb0 tx=0x7fd9c800c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:06.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.661+0000 7fd9cf400700 1 -- 172.21.15.12:0/1280899191 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8005d70 con 0x7fd9d80d18b0 2024-03-29T09:54:06.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.661+0000 7fd9cf400700 1 -- 172.21.15.12:0/1280899191 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c80070d0 con 0x7fd9d80d18b0 2024-03-29T09:54:06.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.661+0000 7fd9cf400700 1 -- 172.21.15.12:0/1280899191 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8007460 con 0x7fd9d80d18b0 2024-03-29T09:54:06.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.661+0000 7fd9de600700 1 -- 172.21.15.12:0/1280899191 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d815fcd0 con 0x7fd9d80d18b0 2024-03-29T09:54:06.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.661+0000 7fd9de600700 1 -- 172.21.15.12:0/1280899191 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d8160290 con 0x7fd9d80d18b0 2024-03-29T09:54:06.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.661+0000 7fd9cf400700 1 -- 172.21.15.12:0/1280899191 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd9c8012400 con 0x7fd9d80d18b0 2024-03-29T09:54:06.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.661+0000 7fd9cf400700 1 -- 172.21.15.12:0/1280899191 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd9c8012d80 con 0x7fd9d80d18b0 2024-03-29T09:54:06.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.662+0000 7fd9de600700 1 -- 172.21.15.12:0/1280899191 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d8067720 con 0x7fd9d80d18b0 2024-03-29T09:54:06.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.665+0000 7fd9cf400700 1 -- 172.21.15.12:0/1280899191 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fd9c8021060 con 0x7fd9d80d18b0 2024-03-29T09:54:06.697 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:06.698+0000 7fa044cc8240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T09:54:06.699 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:06.700+0000 7f68dcca0240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T09:54:06.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.740+0000 7fd9de600700 1 -- 172.21.15.12:0/1280899191 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9d8160730 con 0x7fd9d80d18b0 2024-03-29T09:54:06.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.741+0000 7fd9cf400700 1 -- 172.21.15.12:0/1280899191 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fd9c8017070 con 0x7fd9d80d18b0 2024-03-29T09:54:06.741 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:06.741 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:06.330559+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.743+0000 7fd9de600700 1 -- 172.21.15.12:0/1280899191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d80d18b0 msgr2=0x7fd9d8158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.743+0000 7fd9de600700 1 --2- 172.21.15.12:0/1280899191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d80d18b0 0x7fd9d8158fc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8001eb0 tx=0x7fd9c800c7b0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.743+0000 7fd9de600700 1 -- 172.21.15.12:0/1280899191 shutdown_connections 2024-03-29T09:54:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.743+0000 7fd9de600700 1 --2- 172.21.15.12:0/1280899191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9d8161590 0x7fd9d80cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.743+0000 7fd9de600700 1 --2- 172.21.15.12:0/1280899191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9d80d18b0 0x7fd9d8158fc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.743+0000 7fd9de600700 1 --2- 172.21.15.12:0/1280899191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9d8160f10 0x7fd9d80ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.743+0000 7fd9de600700 1 -- 172.21.15.12:0/1280899191 >> 172.21.15.12:0/1280899191 conn(0x7fd9d808aff0 msgr2=0x7fd9d815c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.743+0000 7fd9de600700 1 -- 172.21.15.12:0/1280899191 shutdown_connections 2024-03-29T09:54:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:06.743+0000 7fd9de600700 1 -- 172.21.15.12:0/1280899191 wait complete. 2024-03-29T09:54:06.767 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T09:54:06.768 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T09:54:06.798 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:06.798+0000 7fa044cc8240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T09:54:06.812 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:06.813+0000 7f68dcca0240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T09:54:06.899 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:06.899+0000 7fa044cc8240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T09:54:06.931 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:06.932+0000 7f68dcca0240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T09:54:07.003 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:07.004+0000 7fa044cc8240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T09:54:07.056 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-29T09:54:07.056+0000 7f5f2b251680 -1 Falling back to public interface 2024-03-29T09:54:07.071 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-29T09:54:07.072+0000 7faf2a044680 -1 Falling back to public interface 2024-03-29T09:54:07.072 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-03-29T09:54:07.073+0000 7f367f1b2680 -1 Falling back to public interface 2024-03-29T09:54:07.095 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-03-29T09:54:07.096+0000 7f16dadd1680 -1 Falling back to public interface 2024-03-29T09:54:07.098 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-29T09:54:07.098+0000 7fe85afb1680 -1 Falling back to public interface 2024-03-29T09:54:07.098 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-29T09:54:07.098+0000 7fabc96f1680 -1 Falling back to public interface 2024-03-29T09:54:07.108 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-03-29T09:54:07.108+0000 7fad39856680 -1 Falling back to public interface 2024-03-29T09:54:07.114 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-03-29T09:54:07.112+0000 7fc92ac18680 -1 Falling back to public interface 2024-03-29T09:54:07.195 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:07.196+0000 7f68dcca0240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T09:54:07.257 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:07.257+0000 7fa044cc8240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T09:54:07.460 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:07.460+0000 7f68dcca0240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T09:54:07.520 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:07.520+0000 7fa044cc8240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T09:54:07.572 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:07.670 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:07.670+0000 7f68dcca0240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T09:54:07.720 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:07.720+0000 7fa044cc8240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T09:54:07.750 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.751+0000 7fda54600700 1 Processor -- start 2024-03-29T09:54:07.750 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.751+0000 7fda54600700 1 -- start start 2024-03-29T09:54:07.753 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda54600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda4c06d580 0x7fda4c15d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:07.753 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda54600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4c0d16d0 con 0x7fda4c0cde60 2024-03-29T09:54:07.753 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda54600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4c06c990 con 0x7fda4c0ce7d0 2024-03-29T09:54:07.753 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda54600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4c0d1950 con 0x7fda4c06d580 2024-03-29T09:54:07.753 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda51200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fda4c0cde60 0x7fda4c06ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42106/0 (socket says 172.21.15.12:42106) 2024-03-29T09:54:07.753 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda51200700 1 -- 172.21.15.12:0/1179141128 learned_addr learned my addr 172.21.15.12:0/1179141128 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:07.753 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda52600700 1 --2- 172.21.15.12:0/1179141128 >> v2:172.21.15.12:3300/0 conn(0x7fda4c06d580 0x7fda4c15d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:07.754 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda4be00700 1 -- 172.21.15.12:0/1179141128 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620703074 0 0) 0x7fda4c0d16d0 con 0x7fda4c0cde60 2024-03-29T09:54:07.754 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda4be00700 1 -- 172.21.15.12:0/1179141128 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda340035b0 con 0x7fda4c0cde60 2024-03-29T09:54:07.754 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda4be00700 1 -- 172.21.15.12:0/1179141128 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250235030 0 0) 0x7fda4c06c990 con 0x7fda4c0ce7d0 2024-03-29T09:54:07.754 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda4be00700 1 -- 172.21.15.12:0/1179141128 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda4c0d16d0 con 0x7fda4c0ce7d0 2024-03-29T09:54:07.754 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda52600700 1 -- 172.21.15.12:0/1179141128 >> v1:172.21.15.159:6789/0 conn(0x7fda4c0ce7d0 legacy=0x7fda4c0b59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:07.754 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda52600700 1 -- 172.21.15.12:0/1179141128 >> v1:172.21.15.12:6789/0 conn(0x7fda4c0cde60 legacy=0x7fda4c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:07.754 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.753+0000 7fda52600700 1 -- 172.21.15.12:0/1179141128 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c0d1c30 con 0x7fda4c06d580 2024-03-29T09:54:07.754 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.754+0000 7fda52600700 1 --2- 172.21.15.12:0/1179141128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c06d580 0x7fda4c15d170 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fda44009d90 tx=0x7fda4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b406e60592ddd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:07.755 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.754+0000 7fda4be00700 1 -- 172.21.15.12:0/1179141128 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4400e070 con 0x7fda4c06d580 2024-03-29T09:54:07.755 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.754+0000 7fda4be00700 1 -- 172.21.15.12:0/1179141128 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4400bb20 con 0x7fda4c06d580 2024-03-29T09:54:07.755 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.754+0000 7fda4be00700 1 -- 172.21.15.12:0/1179141128 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44012430 con 0x7fda4c06d580 2024-03-29T09:54:07.755 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.755+0000 7fda54600700 1 -- 172.21.15.12:0/1179141128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c06d580 msgr2=0x7fda4c15d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:07.755 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.755+0000 7fda54600700 1 --2- 172.21.15.12:0/1179141128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c06d580 0x7fda4c15d170 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fda44009d90 tx=0x7fda4400b330 comp rx=0 tx=0).stop 2024-03-29T09:54:07.755 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.755+0000 7fda54600700 1 -- 172.21.15.12:0/1179141128 shutdown_connections 2024-03-29T09:54:07.756 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.755+0000 7fda54600700 1 --2- 172.21.15.12:0/1179141128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c06d580 0x7fda4c15d170 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:07.756 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.755+0000 7fda54600700 1 -- 172.21.15.12:0/1179141128 >> 172.21.15.12:0/1179141128 conn(0x7fda4c0d3520 msgr2=0x7fda4c0d3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:07.756 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.755+0000 7fda54600700 1 -- 172.21.15.12:0/1179141128 shutdown_connections 2024-03-29T09:54:07.756 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.755+0000 7fda54600700 1 -- 172.21.15.12:0/1179141128 wait complete. 2024-03-29T09:54:07.756 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.757+0000 7fda54600700 1 Processor -- start 2024-03-29T09:54:07.756 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.757+0000 7fda54600700 1 -- start start 2024-03-29T09:54:07.758 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda54600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda4c06d580 0x7fda4c1d3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda54600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c0cde60 0x7fda4c09d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda54600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda4c0ce7d0 0x7fda4c06b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda54600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c0b67f0 con 0x7fda4c0cde60 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda54600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c0b5c70 con 0x7fda4c06d580 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda54600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c0b5f50 con 0x7fda4c0ce7d0 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda51200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c0cde60 0x7fda4c09d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda51200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c0cde60 0x7fda4c09d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60824/0 (socket says 172.21.15.12:60824) 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda51200700 1 -- 172.21.15.12:0/767468581 learned_addr learned my addr 172.21.15.12:0/767468581 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda51c00700 1 --2- 172.21.15.12:0/767468581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda4c06d580 0x7fda4c1d3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.758+0000 7fda52600700 1 --2- 172.21.15.12:0/767468581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda4c0ce7d0 0x7fda4c06b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:07.759 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda51200700 1 -- 172.21.15.12:0/767468581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda4c0ce7d0 msgr2=0x7fda4c06b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:07.760 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda51200700 1 --2- 172.21.15.12:0/767468581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda4c0ce7d0 0x7fda4c06b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:07.760 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda51200700 1 -- 172.21.15.12:0/767468581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda4c06d580 msgr2=0x7fda4c1d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:07.760 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda51200700 1 --2- 172.21.15.12:0/767468581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda4c06d580 0x7fda4c1d3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:07.760 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda51200700 1 -- 172.21.15.12:0/767468581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda44009a70 con 0x7fda4c0cde60 2024-03-29T09:54:07.760 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda51200700 1 --2- 172.21.15.12:0/767468581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c0cde60 0x7fda4c09d200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c700 tx=0x7fda3800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:07.760 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda49600700 1 -- 172.21.15.12:0/767468581 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38005d70 con 0x7fda4c0cde60 2024-03-29T09:54:07.760 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda49600700 1 -- 172.21.15.12:0/767468581 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda380070d0 con 0x7fda4c0cde60 2024-03-29T09:54:07.760 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda49600700 1 -- 172.21.15.12:0/767468581 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38007440 con 0x7fda4c0cde60 2024-03-29T09:54:07.760 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda54600700 1 -- 172.21.15.12:0/767468581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c0b6230 con 0x7fda4c0cde60 2024-03-29T09:54:07.761 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.759+0000 7fda54600700 1 -- 172.21.15.12:0/767468581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c09fbc0 con 0x7fda4c0cde60 2024-03-29T09:54:07.761 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.760+0000 7fda49600700 1 -- 172.21.15.12:0/767468581 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fda3801c440 con 0x7fda4c0cde60 2024-03-29T09:54:07.761 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.760+0000 7fda49600700 1 -- 172.21.15.12:0/767468581 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fda3801cdc0 con 0x7fda4c0cde60 2024-03-29T09:54:07.761 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.760+0000 7fda54600700 1 -- 172.21.15.12:0/767468581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4c04ea80 con 0x7fda4c0cde60 2024-03-29T09:54:07.762 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.763+0000 7fda49600700 1 -- 172.21.15.12:0/767468581 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fda38021060 con 0x7fda4c0cde60 2024-03-29T09:54:07.768 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:07.768+0000 7f68dcca0240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T09:54:07.811 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:07.811+0000 7fa044cc8240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T09:54:07.837 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.838+0000 7fda54600700 1 -- 172.21.15.12:0/767468581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda4c06b170 con 0x7fda4c0cde60 2024-03-29T09:54:07.838 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.839+0000 7fda49600700 1 -- 172.21.15.12:0/767468581 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fda38017070 con 0x7fda4c0cde60 2024-03-29T09:54:07.838 INFO:teuthology.misc.health.smithi012.stdout: 2024-03-29T09:54:07.838 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":10,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:06.330559+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:07.839 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.840+0000 7fda54600700 1 -- 172.21.15.12:0/767468581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c0cde60 msgr2=0x7fda4c09d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:07.839 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.840+0000 7fda54600700 1 --2- 172.21.15.12:0/767468581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c0cde60 0x7fda4c09d200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c700 tx=0x7fda3800ca10 comp rx=0 tx=0).stop 2024-03-29T09:54:07.840 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.840+0000 7fda54600700 1 -- 172.21.15.12:0/767468581 shutdown_connections 2024-03-29T09:54:07.840 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.840+0000 7fda54600700 1 --2- 172.21.15.12:0/767468581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda4c0ce7d0 0x7fda4c06b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:07.840 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.840+0000 7fda54600700 1 --2- 172.21.15.12:0/767468581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda4c06d580 0x7fda4c1d3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:07.840 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.840+0000 7fda54600700 1 --2- 172.21.15.12:0/767468581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda4c0cde60 0x7fda4c09d200 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:07.840 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.840+0000 7fda54600700 1 -- 172.21.15.12:0/767468581 >> 172.21.15.12:0/767468581 conn(0x7fda4c0d3520 msgr2=0x7fda4c0b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:07.840 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.840+0000 7fda54600700 1 -- 172.21.15.12:0/767468581 shutdown_connections 2024-03-29T09:54:07.840 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:07.840+0000 7fda54600700 1 -- 172.21.15.12:0/767468581 wait complete. 2024-03-29T09:54:07.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T09:54:08.622 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:08.622+0000 7f68dcca0240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T09:54:08.630 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:08.630+0000 7fa044cc8240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T09:54:08.839 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:08.839+0000 7fa044cc8240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T09:54:08.840 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:08.841+0000 7f68dcca0240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T09:54:09.040 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:09.040+0000 7fa044cc8240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T09:54:09.050 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:09.051+0000 7f68dcca0240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T09:54:09.553 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:09.553+0000 7fa044cc8240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T09:54:09.601 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:09.601+0000 7f68dcca0240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T09:54:09.762 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:09.763+0000 7fa044cc8240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T09:54:09.821 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:09.821+0000 7f68dcca0240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T09:54:10.285 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:10.285+0000 7fa044cc8240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T09:54:10.394 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:10.394+0000 7f68dcca0240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T09:54:10.417 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:10.418+0000 7fa044cc8240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T09:54:10.531 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:54:10.532+0000 7f68dcca0240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T09:54:10.563 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-29T09:54:10.563+0000 7f5f2b251680 -1 osd.4 0 log_to_monitors true 2024-03-29T09:54:10.599 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-03-29T09:54:10.600+0000 7f367f1b2680 -1 osd.2 0 log_to_monitors true 2024-03-29T09:54:10.641 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-29T09:54:10.641+0000 7fabc96f1680 -1 osd.6 0 log_to_monitors true 2024-03-29T09:54:10.661 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-03-29T09:54:10.661+0000 7fad39856680 -1 osd.1 0 log_to_monitors true 2024-03-29T09:54:10.684 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-03-29T09:54:10.685+0000 7fc92ac18680 -1 osd.3 0 log_to_monitors true 2024-03-29T09:54:10.858 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-29T09:54:10.858+0000 7faf2a044680 -1 osd.7 0 log_to_monitors true 2024-03-29T09:54:10.890 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-03-29T09:54:10.890+0000 7f16dadd1680 -1 osd.0 0 log_to_monitors true 2024-03-29T09:54:10.891 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-29T09:54:10.892+0000 7fe85afb1680 -1 osd.5 0 log_to_monitors true 2024-03-29T09:54:11.440 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:11.440+0000 7f9ffb200700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T09:54:11.440 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:54:11.440+0000 7f9ffb200700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T09:54:14.660 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:14.802 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-29T09:54:14.802+0000 7f5f1a400700 -1 osd.4 0 waiting for initial osdmap 2024-03-29T09:54:14.814 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-29T09:54:14.814+0000 7f5f0e800700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T09:54:14.816 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-29T09:54:14.817+0000 7fabb6a00700 -1 osd.6 0 waiting for initial osdmap 2024-03-29T09:54:14.817 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-03-29T09:54:14.817+0000 7f366c400700 -1 osd.2 0 waiting for initial osdmap 2024-03-29T09:54:14.830 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-03-29T09:54:14.828+0000 7f3662600700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T09:54:14.830 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-03-29T09:54:14.830+0000 7f16c8000700 -1 osd.0 0 waiting for initial osdmap 2024-03-29T09:54:14.836 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-29T09:54:14.836+0000 7fabacc00700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T09:54:14.845 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-03-29T09:54:14.844+0000 7f16be200700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T09:54:14.854 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-29T09:54:14.854+0000 7fe848200700 -1 osd.5 0 waiting for initial osdmap 2024-03-29T09:54:14.860 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-29T09:54:14.860+0000 7fe83e400700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T09:54:14.866 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-29T09:54:14.866+0000 7faf17400700 -1 osd.7 0 waiting for initial osdmap 2024-03-29T09:54:14.876 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-29T09:54:14.876+0000 7faf0d600700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T09:54:14.883 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-03-29T09:54:14.883+0000 7fc918000700 -1 osd.3 0 waiting for initial osdmap 2024-03-29T09:54:14.893 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-03-29T09:54:14.892+0000 7fc90e200700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T09:54:14.893 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-03-29T09:54:14.893+0000 7fad26c00700 -1 osd.1 0 waiting for initial osdmap 2024-03-29T09:54:14.900 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-03-29T09:54:14.900+0000 7fad1ce00700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-29T09:54:14.925 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.924+0000 7fb958200700 1 Processor -- start 2024-03-29T09:54:14.925 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.924+0000 7fb958200700 1 -- start start 2024-03-29T09:54:14.926 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.925+0000 7fb958200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb9500b5670 0x7fb9500b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:14.926 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.925+0000 7fb958200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9500b4140 con 0x7fb95015e9f0 2024-03-29T09:54:14.926 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.925+0000 7fb958200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9500cdef0 con 0x7fb95015edd0 2024-03-29T09:54:14.926 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.925+0000 7fb958200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9500ce1d0 con 0x7fb9500b5670 2024-03-29T09:54:14.926 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.926+0000 7fb955800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb95015e9f0 0x7fb9500b4f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46690/0 (socket says 172.21.15.12:46690) 2024-03-29T09:54:14.926 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.926+0000 7fb955800700 1 -- 172.21.15.12:0/275628174 learned_addr learned my addr 172.21.15.12:0/275628174 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:14.928 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.926+0000 7fb954e00700 1 --2- 172.21.15.12:0/275628174 >> v2:172.21.15.159:3300/0 conn(0x7fb9500b5670 0x7fb9500b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:14.928 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.926+0000 7fb94fe00700 1 -- 172.21.15.12:0/275628174 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587007609 0 0) 0x7fb9500b4140 con 0x7fb95015e9f0 2024-03-29T09:54:14.928 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.926+0000 7fb94fe00700 1 -- 172.21.15.12:0/275628174 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9380035b0 con 0x7fb95015e9f0 2024-03-29T09:54:14.928 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.926+0000 7fb94fe00700 1 -- 172.21.15.12:0/275628174 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072360705 0 0) 0x7fb9500cdef0 con 0x7fb95015edd0 2024-03-29T09:54:14.928 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.926+0000 7fb94fe00700 1 -- 172.21.15.12:0/275628174 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9500b4140 con 0x7fb95015edd0 2024-03-29T09:54:14.928 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.926+0000 7fb954e00700 1 -- 172.21.15.12:0/275628174 >> v1:172.21.15.159:6789/0 conn(0x7fb95015edd0 legacy=0x7fb9500d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:14.928 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.926+0000 7fb954e00700 1 -- 172.21.15.12:0/275628174 >> v1:172.21.15.12:6789/0 conn(0x7fb95015e9f0 legacy=0x7fb9500b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.926+0000 7fb954e00700 1 -- 172.21.15.12:0/275628174 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9500d17f0 con 0x7fb9500b5670 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb954e00700 1 --2- 172.21.15.12:0/275628174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9500b5670 0x7fb9500b4420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009840 tx=0x7fb93c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3105d9fe2a7593d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb94fe00700 1 -- 172.21.15.12:0/275628174 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c011830 con 0x7fb9500b5670 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb94fe00700 1 -- 172.21.15.12:0/275628174 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c011990 con 0x7fb9500b5670 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb94fe00700 1 -- 172.21.15.12:0/275628174 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c011d20 con 0x7fb9500b5670 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb958200700 1 -- 172.21.15.12:0/275628174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9500b5670 msgr2=0x7fb9500b4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb958200700 1 --2- 172.21.15.12:0/275628174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9500b5670 0x7fb9500b4420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009840 tx=0x7fb93c009b50 comp rx=0 tx=0).stop 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb958200700 1 -- 172.21.15.12:0/275628174 shutdown_connections 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb958200700 1 --2- 172.21.15.12:0/275628174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9500b5670 0x7fb9500b4420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb958200700 1 -- 172.21.15.12:0/275628174 >> 172.21.15.12:0/275628174 conn(0x7fb95008aff0 msgr2=0x7fb9500d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb958200700 1 -- 172.21.15.12:0/275628174 shutdown_connections 2024-03-29T09:54:14.929 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.927+0000 7fb958200700 1 -- 172.21.15.12:0/275628174 wait complete. 2024-03-29T09:54:14.930 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.930+0000 7fb958200700 1 Processor -- start 2024-03-29T09:54:14.930 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.930+0000 7fb958200700 1 -- start start 2024-03-29T09:54:14.933 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.931+0000 7fb958200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9500b5670 0x7fb9500b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:14.933 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.931+0000 7fb958200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb95015e9f0 0x7fb9500b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:14.933 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.931+0000 7fb958200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb95015edd0 0x7fb9500b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:14.933 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.931+0000 7fb958200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9500b6e90 con 0x7fb9500b5670 2024-03-29T09:54:14.933 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.931+0000 7fb958200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9500b5f70 con 0x7fb95015e9f0 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.931+0000 7fb958200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9500b1470 con 0x7fb95015edd0 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.931+0000 7fb955800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9500b5670 0x7fb9500b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.931+0000 7fb955800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9500b5670 0x7fb9500b0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39098/0 (socket says 172.21.15.12:39098) 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.931+0000 7fb955800700 1 -- 172.21.15.12:0/4056524897 learned_addr learned my addr 172.21.15.12:0/4056524897 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb954e00700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb95015e9f0 0x7fb9500b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb956200700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb95015edd0 0x7fb9500b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb955800700 1 -- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb95015edd0 msgr2=0x7fb9500b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb955800700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb95015edd0 0x7fb9500b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb955800700 1 -- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb95015e9f0 msgr2=0x7fb9500b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb955800700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb95015e9f0 0x7fb9500b72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb955800700 1 -- 172.21.15.12:0/4056524897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93c0094a0 con 0x7fb9500b5670 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb954e00700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb95015e9f0 0x7fb9500b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb956200700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb95015edd0 0x7fb9500b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:14.934 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb955800700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9500b5670 0x7fb9500b0f30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb94400c4d0 tx=0x7fb94400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:14.935 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb94d600700 1 -- 172.21.15.12:0/4056524897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944005d70 con 0x7fb9500b5670 2024-03-29T09:54:14.935 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb94d600700 1 -- 172.21.15.12:0/4056524897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9440070d0 con 0x7fb9500b5670 2024-03-29T09:54:14.935 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb958200700 1 -- 172.21.15.12:0/4056524897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9500b6210 con 0x7fb9500b5670 2024-03-29T09:54:14.935 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.932+0000 7fb94d600700 1 -- 172.21.15.12:0/4056524897 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944007480 con 0x7fb9500b5670 2024-03-29T09:54:14.935 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.933+0000 7fb958200700 1 -- 172.21.15.12:0/4056524897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9500b67d0 con 0x7fb9500b5670 2024-03-29T09:54:14.936 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.934+0000 7fb958200700 1 -- 172.21.15.12:0/4056524897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9500d4420 con 0x7fb9500b5670 2024-03-29T09:54:14.936 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.935+0000 7fb94d600700 1 -- 172.21.15.12:0/4056524897 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7fb9440075e0 con 0x7fb9500b5670 2024-03-29T09:54:14.936 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.935+0000 7fb94d600700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb93006ac00 0x7fb93006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:14.936 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.935+0000 7fb94d600700 1 -- 172.21.15.12:0/4056524897 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fb944087950 con 0x7fb9500b5670 2024-03-29T09:54:14.936 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.936+0000 7fb954e00700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb93006ac00 0x7fb93006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:14.937 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.936+0000 7fb954e00700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb93006ac00 0x7fb93006d0c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9500b1b10 tx=0x7fb93c01c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:14.941 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:14.941+0000 7fb94d600700 1 -- 172.21.15.12:0/4056524897 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb944053a30 con 0x7fb9500b5670 2024-03-29T09:54:15.156 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.156+0000 7fb958200700 1 -- 172.21.15.12:0/4056524897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9500b7100 con 0x7fb9500b5670 2024-03-29T09:54:15.157 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.156+0000 7fb94d600700 1 -- 172.21.15.12:0/4056524897 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb944017020 con 0x7fb9500b5670 2024-03-29T09:54:15.157 INFO:teuthology.misc.health.smithi012.stdout: 2024-03-29T09:54:15.157 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":12,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:12.484774+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:15.160 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 -- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb93006ac00 msgr2=0x7fb93006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:15.160 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb93006ac00 0x7fb93006d0c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9500b1b10 tx=0x7fb93c01c040 comp rx=0 tx=0).stop 2024-03-29T09:54:15.160 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 -- 172.21.15.12:0/4056524897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9500b5670 msgr2=0x7fb9500b0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:15.160 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9500b5670 0x7fb9500b0f30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb94400c4d0 tx=0x7fb94400c890 comp rx=0 tx=0).stop 2024-03-29T09:54:15.160 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 -- 172.21.15.12:0/4056524897 shutdown_connections 2024-03-29T09:54:15.160 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb93006ac00 0x7fb93006d0c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:15.160 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb95015edd0 0x7fb9500b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:15.161 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9500b5670 0x7fb9500b0f30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:15.161 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 --2- 172.21.15.12:0/4056524897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb95015e9f0 0x7fb9500b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:15.161 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 -- 172.21.15.12:0/4056524897 >> 172.21.15.12:0/4056524897 conn(0x7fb95008aff0 msgr2=0x7fb9500c7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:15.161 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 -- 172.21.15.12:0/4056524897 shutdown_connections 2024-03-29T09:54:15.161 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:15.159+0000 7fb958200700 1 -- 172.21.15.12:0/4056524897 wait complete. 2024-03-29T09:54:15.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T09:54:21.981 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:22.167 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.167+0000 7fdacbe00700 1 Processor -- start 2024-03-29T09:54:22.167 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.167+0000 7fdacbe00700 1 -- start start 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.168+0000 7fdacbe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdac415e9f0 0x7fdac40b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.168+0000 7fdacbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdac415edd0 0x7fdac40b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.168+0000 7fdacbe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdac40cef80 0x7fdac40b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.168+0000 7fdacbe00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdac40b4140 con 0x7fdac415edd0 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.168+0000 7fdacbe00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdac40cdef0 con 0x7fdac415e9f0 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.168+0000 7fdacbe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac40ce1d0 con 0x7fdac40cef80 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdac415edd0 0x7fdac40b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdac415edd0 0x7fdac40b4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57994/0 (socket says 172.21.15.12:57994) 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac8a00700 1 -- 172.21.15.12:0/1175485086 learned_addr learned my addr 172.21.15.12:0/1175485086 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac9400700 1 --2- 172.21.15.12:0/1175485086 >> v2:172.21.15.159:3300/0 conn(0x7fdac415e9f0 0x7fdac40b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac9e00700 1 --2- 172.21.15.12:0/1175485086 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdac40cef80 0x7fdac40b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.171 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac8a00700 1 -- 172.21.15.12:0/1175485086 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdac40cef80 msgr2=0x7fdac40b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac8a00700 1 --2- 172.21.15.12:0/1175485086 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdac40cef80 0x7fdac40b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac8a00700 1 -- 172.21.15.12:0/1175485086 >> v2:172.21.15.159:3300/0 conn(0x7fdac415e9f0 msgr2=0x7fdac40b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac8a00700 1 --2- 172.21.15.12:0/1175485086 >> v2:172.21.15.159:3300/0 conn(0x7fdac415e9f0 0x7fdac40b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac8a00700 1 -- 172.21.15.12:0/1175485086 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac40d17f0 con 0x7fdac415edd0 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.169+0000 7fdac8a00700 1 --2- 172.21.15.12:0/1175485086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 0x7fdac40b4f60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdac408daf0 tx=0x7fdab0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fed087205d76b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdac3e00700 1 -- 172.21.15.12:0/1175485086 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0011830 con 0x7fdac415edd0 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdac3e00700 1 -- 172.21.15.12:0/1175485086 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab0011990 con 0x7fdac415edd0 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdac3e00700 1 -- 172.21.15.12:0/1175485086 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0011d20 con 0x7fdac415edd0 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdacbe00700 1 -- 172.21.15.12:0/1175485086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 msgr2=0x7fdac40b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdacbe00700 1 --2- 172.21.15.12:0/1175485086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 0x7fdac40b4f60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdac408daf0 tx=0x7fdab0009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdacbe00700 1 -- 172.21.15.12:0/1175485086 shutdown_connections 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdacbe00700 1 --2- 172.21.15.12:0/1175485086 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdac40cef80 0x7fdac40b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdacbe00700 1 --2- 172.21.15.12:0/1175485086 >> v2:172.21.15.159:3300/0 conn(0x7fdac415e9f0 0x7fdac40b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdacbe00700 1 --2- 172.21.15.12:0/1175485086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 0x7fdac40b4f60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdacbe00700 1 -- 172.21.15.12:0/1175485086 >> 172.21.15.12:0/1175485086 conn(0x7fdac408aff0 msgr2=0x7fdac40d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:22.172 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdacbe00700 1 -- 172.21.15.12:0/1175485086 shutdown_connections 2024-03-29T09:54:22.173 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.170+0000 7fdacbe00700 1 -- 172.21.15.12:0/1175485086 wait complete. 2024-03-29T09:54:22.174 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.172+0000 7fdacbe00700 1 Processor -- start 2024-03-29T09:54:22.174 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.172+0000 7fdacbe00700 1 -- start start 2024-03-29T09:54:22.174 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.173+0000 7fdacbe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdac40cef80 0x7fdac4152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.174 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.173+0000 7fdacbe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdac415e9f0 0x7fdac40c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.174 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.173+0000 7fdacbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 0x7fdac40ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.173+0000 7fdacbe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac40c9020 con 0x7fdac415edd0 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.173+0000 7fdacbe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac40b1bd0 con 0x7fdac415e9f0 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.173+0000 7fdacbe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac40c8410 con 0x7fdac40cef80 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.173+0000 7fdac9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 0x7fdac40ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.173+0000 7fdac9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 0x7fdac40ccaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58008/0 (socket says 172.21.15.12:58008) 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.173+0000 7fdac9e00700 1 -- 172.21.15.12:0/920593669 learned_addr learned my addr 172.21.15.12:0/920593669 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.173+0000 7fdac8a00700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdac415e9f0 0x7fdac40c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdac9400700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdac40cef80 0x7fdac4152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdac9e00700 1 -- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdac40cef80 msgr2=0x7fdac4152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdac9e00700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdac40cef80 0x7fdac4152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdac9e00700 1 -- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdac415e9f0 msgr2=0x7fdac40c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdac9e00700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdac415e9f0 0x7fdac40c9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdac9e00700 1 -- 172.21.15.12:0/920593669 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab0009550 con 0x7fdac415edd0 2024-03-29T09:54:22.175 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdac9e00700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 0x7fdac40ccaa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00ef00 tx=0x7fdabc00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:22.176 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdac1600700 1 -- 172.21.15.12:0/920593669 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdabc010040 con 0x7fdac415edd0 2024-03-29T09:54:22.176 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdac1600700 1 -- 172.21.15.12:0/920593669 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdabc00bc80 con 0x7fdac415edd0 2024-03-29T09:54:22.176 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdac1600700 1 -- 172.21.15.12:0/920593669 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdabc009e40 con 0x7fdac415edd0 2024-03-29T09:54:22.176 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.174+0000 7fdacbe00700 1 -- 172.21.15.12:0/920593669 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac40c86f0 con 0x7fdac415edd0 2024-03-29T09:54:22.177 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.175+0000 7fdacbe00700 1 -- 172.21.15.12:0/920593669 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac40c8cb0 con 0x7fdac415edd0 2024-03-29T09:54:22.177 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.175+0000 7fdacbe00700 1 -- 172.21.15.12:0/920593669 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac40d4420 con 0x7fdac415edd0 2024-03-29T09:54:22.177 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.177+0000 7fdac1600700 1 -- 172.21.15.12:0/920593669 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdabc014400 con 0x7fdac415edd0 2024-03-29T09:54:22.178 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.177+0000 7fdac1600700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdaa406ac90 0x7fdaa406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.178 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.178+0000 7fdac1600700 1 -- 172.21.15.12:0/920593669 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdabc089360 con 0x7fdac415edd0 2024-03-29T09:54:22.178 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.178+0000 7fdac9400700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdaa406ac90 0x7fdaa406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.179 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.179+0000 7fdac9400700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdaa406ac90 0x7fdaa406d150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009be0 tx=0x7fdab8009210 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:22.183 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.183+0000 7fdac1600700 1 -- 172.21.15.12:0/920593669 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdabc054880 con 0x7fdac415edd0 2024-03-29T09:54:22.303 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:54:22.307 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:54:22.398 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.398+0000 7fdacbe00700 1 -- 172.21.15.12:0/920593669 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdac40c93b0 con 0x7fdac415edd0 2024-03-29T09:54:22.415 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.399+0000 7fdac1600700 1 -- 172.21.15.12:0/920593669 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12382 (secure 0 0 0) 0x7fdabc057ea0 con 0x7fdac415edd0 2024-03-29T09:54:22.416 INFO:teuthology.misc.health.smithi012.stdout: 2024-03-29T09:54:22.416 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":16,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:18.853550+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:22.417 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 -- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdaa406ac90 msgr2=0x7fdaa406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.417 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdaa406ac90 0x7fdaa406d150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009be0 tx=0x7fdab8009210 comp rx=0 tx=0).stop 2024-03-29T09:54:22.418 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 -- 172.21.15.12:0/920593669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 msgr2=0x7fdac40ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.418 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 0x7fdac40ccaa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00ef00 tx=0x7fdabc00b390 comp rx=0 tx=0).stop 2024-03-29T09:54:22.418 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 -- 172.21.15.12:0/920593669 shutdown_connections 2024-03-29T09:54:22.418 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdaa406ac90 0x7fdaa406d150 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.418 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac415edd0 0x7fdac40ccaa0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.418 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdac40cef80 0x7fdac4152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.418 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 --2- 172.21.15.12:0/920593669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdac415e9f0 0x7fdac40c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.418 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 -- 172.21.15.12:0/920593669 >> 172.21.15.12:0/920593669 conn(0x7fdac408aff0 msgr2=0x7fdac415d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:22.418 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.401+0000 7fdacbe00700 1 -- 172.21.15.12:0/920593669 shutdown_connections 2024-03-29T09:54:22.418 INFO:teuthology.misc.health.smithi012.stderr:2024-03-29T09:54:22.402+0000 7fdacbe00700 1 -- 172.21.15.12:0/920593669 wait complete. 2024-03-29T09:54:22.419 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-29T09:54:22.419 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-29T09:54:22.419 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T09:54:22.419 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T09:54:22.462 INFO:tasks.ceph.mds.a:Started 2024-03-29T09:54:22.463 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T09:54:22.463 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T09:54:22.466 INFO:tasks.ceph.mds.c:Started 2024-03-29T09:54:22.466 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T09:54:22.467 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T09:54:22.471 INFO:tasks.ceph.mds.b:Started 2024-03-29T09:54:22.471 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T09:54:22.471 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T09:54:22.474 INFO:tasks.ceph.mds.d:Started 2024-03-29T09:54:22.475 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-29T09:54:22.475 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:22.613 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T09:54:22.615 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T09:54:22.617 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T09:54:22.617 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T09:54:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.683+0000 7f5e47200700 1 Processor -- start 2024-03-29T09:54:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.683+0000 7f5e47200700 1 -- start start 2024-03-29T09:54:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.684+0000 7f5e47200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5e40160ed0 0x7f5e400d3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.684+0000 7f5e47200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e400d22a0 0x7f5e400b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.684+0000 7f5e47200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e400cebd0 con 0x7f5e400d4200 2024-03-29T09:54:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.684+0000 7f5e47200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e401612b0 con 0x7f5e40160ed0 2024-03-29T09:54:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.684+0000 7f5e47200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400b3280 con 0x7f5e400d22a0 2024-03-29T09:54:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e3f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5e400d4200 0x7f5e400d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44010/0 (socket says 172.21.15.12:44010) 2024-03-29T09:54:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e3f400700 1 -- 172.21.15.12:0/3943760269 learned_addr learned my addr 172.21.15.12:0/3943760269 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e3fe00700 1 --2- 172.21.15.12:0/3943760269 >> v2:172.21.15.159:3300/0 conn(0x7f5e40160ed0 0x7f5e400d3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e45200700 1 --2- 172.21.15.12:0/3943760269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e400d22a0 0x7f5e400b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e3ea00700 1 -- 172.21.15.12:0/3943760269 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820369661 0 0) 0x7f5e400cebd0 con 0x7f5e400d4200 2024-03-29T09:54:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e3ea00700 1 -- 172.21.15.12:0/3943760269 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e240035b0 con 0x7f5e400d4200 2024-03-29T09:54:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e45200700 1 -- 172.21.15.12:0/3943760269 >> v2:172.21.15.159:3300/0 conn(0x7f5e40160ed0 msgr2=0x7f5e400d3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e45200700 1 --2- 172.21.15.12:0/3943760269 >> v2:172.21.15.159:3300/0 conn(0x7f5e40160ed0 0x7f5e400d3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e45200700 1 -- 172.21.15.12:0/3943760269 >> v1:172.21.15.12:6789/0 conn(0x7f5e400d4200 legacy=0x7f5e400d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e45200700 1 -- 172.21.15.12:0/3943760269 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e400b2cf0 con 0x7f5e400d22a0 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e3fe00700 1 --2- 172.21.15.12:0/3943760269 >> v2:172.21.15.159:3300/0 conn(0x7f5e40160ed0 0x7f5e400d3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.685+0000 7f5e45200700 1 --2- 172.21.15.12:0/3943760269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e400d22a0 0x7f5e400b3710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009d90 tx=0x7f5e3800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb3af286e92890fa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e3ea00700 1 -- 172.21.15.12:0/3943760269 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3800e070 con 0x7f5e400d22a0 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e3ea00700 1 -- 172.21.15.12:0/3943760269 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3800bb20 con 0x7f5e400d22a0 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e3ea00700 1 -- 172.21.15.12:0/3943760269 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38012430 con 0x7f5e400d22a0 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e47200700 1 -- 172.21.15.12:0/3943760269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e400d22a0 msgr2=0x7f5e400b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e47200700 1 --2- 172.21.15.12:0/3943760269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e400d22a0 0x7f5e400b3710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009d90 tx=0x7f5e3800b330 comp rx=0 tx=0).stop 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e47200700 1 -- 172.21.15.12:0/3943760269 shutdown_connections 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e47200700 1 --2- 172.21.15.12:0/3943760269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e400d22a0 0x7f5e400b3710 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e47200700 1 --2- 172.21.15.12:0/3943760269 >> v2:172.21.15.159:3300/0 conn(0x7f5e40160ed0 0x7f5e400d3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e47200700 1 -- 172.21.15.12:0/3943760269 >> 172.21.15.12:0/3943760269 conn(0x7f5e4008aff0 msgr2=0x7f5e400d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e47200700 1 -- 172.21.15.12:0/3943760269 shutdown_connections 2024-03-29T09:54:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.686+0000 7f5e47200700 1 -- 172.21.15.12:0/3943760269 wait complete. 2024-03-29T09:54:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.688+0000 7f5e47200700 1 Processor -- start 2024-03-29T09:54:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.688+0000 7f5e47200700 1 -- start start 2024-03-29T09:54:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e47200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e400d22a0 0x7f5e40151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e47200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e400d4200 0x7f5e400cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e47200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e40160ed0 0x7f5e400b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e47200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400b7540 con 0x7f5e400d22a0 2024-03-29T09:54:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e47200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400b7110 con 0x7f5e400d4200 2024-03-29T09:54:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e47200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400b6870 con 0x7f5e40160ed0 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e3fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e400d22a0 0x7f5e40151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e3fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e400d22a0 0x7f5e40151b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58046/0 (socket says 172.21.15.12:58046) 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e3fe00700 1 -- 172.21.15.12:0/3115783918 learned_addr learned my addr 172.21.15.12:0/3115783918 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e3f400700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e400d4200 0x7f5e400cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e45200700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e40160ed0 0x7f5e400b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e3fe00700 1 -- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e40160ed0 msgr2=0x7f5e400b6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e3fe00700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e40160ed0 0x7f5e400b6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e3fe00700 1 -- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e400d4200 msgr2=0x7f5e400cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e3fe00700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e400d4200 0x7f5e400cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.690+0000 7f5e3fe00700 1 -- 172.21.15.12:0/3115783918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e300094a0 con 0x7f5e400d22a0 2024-03-29T09:54:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.691+0000 7f5e3fe00700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e400d22a0 0x7f5e40151b30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000ebe0 tx=0x7f5e3000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:22.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.691+0000 7f5e37e00700 1 -- 172.21.15.12:0/3115783918 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30009760 con 0x7f5e400d22a0 2024-03-29T09:54:22.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.691+0000 7f5e37e00700 1 -- 172.21.15.12:0/3115783918 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3000c850 con 0x7f5e400d22a0 2024-03-29T09:54:22.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.691+0000 7f5e47200700 1 -- 172.21.15.12:0/3115783918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e38009a70 con 0x7f5e400d22a0 2024-03-29T09:54:22.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.691+0000 7f5e37e00700 1 -- 172.21.15.12:0/3115783918 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3000cbe0 con 0x7f5e400d22a0 2024-03-29T09:54:22.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.691+0000 7f5e47200700 1 -- 172.21.15.12:0/3115783918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e400b6e30 con 0x7f5e400d22a0 2024-03-29T09:54:22.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.692+0000 7f5e47200700 1 -- 172.21.15.12:0/3115783918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e400d1a00 con 0x7f5e400d22a0 2024-03-29T09:54:22.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.693+0000 7f5e37e00700 1 -- 172.21.15.12:0/3115783918 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5e3000cd40 con 0x7f5e400d22a0 2024-03-29T09:54:22.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.694+0000 7f5e37e00700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e2006ac90 0x7f5e2006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.695+0000 7f5e37e00700 1 -- 172.21.15.12:0/3115783918 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5e30089400 con 0x7f5e400d22a0 2024-03-29T09:54:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.695+0000 7f5e3f400700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e2006ac90 0x7f5e2006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.695+0000 7f5e3f400700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e2006ac90 0x7f5e2006d150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5e400caad0 tx=0x7f5e2c007c70 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.699+0000 7f5e37e00700 1 -- 172.21.15.12:0/3115783918 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e30054890 con 0x7f5e400d22a0 2024-03-29T09:54:22.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.917+0000 7f5e47200700 1 -- 172.21.15.12:0/3115783918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e40067e30 con 0x7f5e400d22a0 2024-03-29T09:54:22.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.918+0000 7f5e37e00700 1 -- 172.21.15.12:0/3115783918 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12382 (secure 0 0 0) 0x7f5e30057eb0 con 0x7f5e400d22a0 2024-03-29T09:54:22.918 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:22.918 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":16,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:18.853550+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 -- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e2006ac90 msgr2=0x7f5e2006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e2006ac90 0x7f5e2006d150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5e400caad0 tx=0x7f5e2c007c70 comp rx=0 tx=0).stop 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 -- 172.21.15.12:0/3115783918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e400d22a0 msgr2=0x7f5e40151b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e400d22a0 0x7f5e40151b30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000ebe0 tx=0x7f5e3000eef0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 -- 172.21.15.12:0/3115783918 shutdown_connections 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e2006ac90 0x7f5e2006d150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e40160ed0 0x7f5e400b6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e400d22a0 0x7f5e40151b30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 --2- 172.21.15.12:0/3115783918 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e400d4200 0x7f5e400cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 -- 172.21.15.12:0/3115783918 >> 172.21.15.12:0/3115783918 conn(0x7f5e4008aff0 msgr2=0x7f5e4006c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 -- 172.21.15.12:0/3115783918 shutdown_connections 2024-03-29T09:54:22.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:22.920+0000 7f5e47200700 1 -- 172.21.15.12:0/3115783918 wait complete. 2024-03-29T09:54:22.938 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:54:22.939 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:54:23.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.139+0000 7f501f000700 1 Processor -- start 2024-03-29T09:54:23.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.139+0000 7f501f000700 1 -- start start 2024-03-29T09:54:23.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.140+0000 7f501f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f501815d9d0 0x7f50180d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.140+0000 7f501f000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f50180d16a0 0x7f50180d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.140+0000 7f501f000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5018161420 con 0x7f50180d2370 2024-03-29T09:54:23.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.140+0000 7f501f000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50180d2820 con 0x7f50180d16a0 2024-03-29T09:54:23.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.140+0000 7f501f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5018161700 con 0x7f501815d9d0 2024-03-29T09:54:23.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.140+0000 7f501d000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f50180d2370 0x7f501815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44016/0 (socket says 172.21.15.12:44016) 2024-03-29T09:54:23.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.140+0000 7f501d000700 1 -- 172.21.15.12:0/3582092441 learned_addr learned my addr 172.21.15.12:0/3582092441 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:23.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.141+0000 7f5017e00700 1 --2- 172.21.15.12:0/3582092441 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f501815d9d0 0x7f50180d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.141+0000 7f5017400700 1 --2- 172.21.15.12:0/3582092441 >> v2:172.21.15.159:3300/0 conn(0x7f50180d16a0 0x7f50180d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.141+0000 7f5016a00700 1 -- 172.21.15.12:0/3582092441 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588173150 0 0) 0x7f5018161420 con 0x7f50180d2370 2024-03-29T09:54:23.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.141+0000 7f5016a00700 1 -- 172.21.15.12:0/3582092441 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ffc0035b0 con 0x7f50180d2370 2024-03-29T09:54:23.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.141+0000 7f5016a00700 1 -- 172.21.15.12:0/3582092441 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534028532 0 0) 0x7f4ffc0035b0 con 0x7f50180d2370 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.141+0000 7f5017400700 1 -- 172.21.15.12:0/3582092441 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f501815d9d0 msgr2=0x7f50180d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.141+0000 7f5017400700 1 --2- 172.21.15.12:0/3582092441 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f501815d9d0 0x7f50180d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.141+0000 7f5017400700 1 -- 172.21.15.12:0/3582092441 >> v1:172.21.15.12:6789/0 conn(0x7f50180d2370 legacy=0x7f501815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.141+0000 7f5017400700 1 -- 172.21.15.12:0/3582092441 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50181619e0 con 0x7f50180d16a0 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f5017400700 1 --2- 172.21.15.12:0/3582092441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50180d16a0 0x7f50180d1e30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5004009820 tx=0x7f5004009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd39881ac34a4a28 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f5016a00700 1 -- 172.21.15.12:0/3582092441 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50100019f0 con 0x7f50180d16a0 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f5016a00700 1 -- 172.21.15.12:0/3582092441 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5004005c30 con 0x7f50180d16a0 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f5016a00700 1 -- 172.21.15.12:0/3582092441 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004004030 con 0x7f50180d16a0 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f501f000700 1 -- 172.21.15.12:0/3582092441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50180d16a0 msgr2=0x7f50180d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f501f000700 1 --2- 172.21.15.12:0/3582092441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50180d16a0 0x7f50180d1e30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5004009820 tx=0x7f5004009b30 comp rx=0 tx=0).stop 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f501f000700 1 -- 172.21.15.12:0/3582092441 shutdown_connections 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f501f000700 1 --2- 172.21.15.12:0/3582092441 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f501815d9d0 0x7f50180d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f501f000700 1 --2- 172.21.15.12:0/3582092441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50180d16a0 0x7f50180d1e30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f501f000700 1 -- 172.21.15.12:0/3582092441 >> 172.21.15.12:0/3582092441 conn(0x7f501808aff0 msgr2=0x7f50180d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:23.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f501f000700 1 -- 172.21.15.12:0/3582092441 shutdown_connections 2024-03-29T09:54:23.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.142+0000 7f501f000700 1 -- 172.21.15.12:0/3582092441 wait complete. 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.144+0000 7f501f000700 1 Processor -- start 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.144+0000 7f501f000700 1 -- start start 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.145+0000 7f501f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50180d16a0 0x7f50181f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.145+0000 7f501f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50180d2370 0x7f50180b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.145+0000 7f501f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501815d9d0 0x7f50180b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.145+0000 7f501f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50181f7ef0 con 0x7f501815d9d0 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.145+0000 7f501f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5018158ca0 con 0x7f50180d2370 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.145+0000 7f501f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50181f4040 con 0x7f50180d16a0 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.145+0000 7f501d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501815d9d0 0x7f50180b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.145+0000 7f501d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501815d9d0 0x7f50180b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58060/0 (socket says 172.21.15.12:58060) 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f501d000700 1 -- 172.21.15.12:0/1985376335 learned_addr learned my addr 172.21.15.12:0/1985376335 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f5017400700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50180d2370 0x7f50180b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f5017e00700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50180d16a0 0x7f50181f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f501d000700 1 -- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50180d16a0 msgr2=0x7f50181f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f501d000700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50180d16a0 0x7f50181f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f501d000700 1 -- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50180d2370 msgr2=0x7f50180b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f501d000700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50180d2370 0x7f50180b49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f501d000700 1 -- 172.21.15.12:0/1985376335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5004009430 con 0x7f501815d9d0 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f501d000700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501815d9d0 0x7f50180b4f00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f501000cca0 tx=0x7f501000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f500fe00700 1 -- 172.21.15.12:0/1985376335 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501000ab70 con 0x7f501815d9d0 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f500fe00700 1 -- 172.21.15.12:0/1985376335 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501000acf0 con 0x7f501815d9d0 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f500fe00700 1 -- 172.21.15.12:0/1985376335 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010007310 con 0x7f501815d9d0 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.146+0000 7f501f000700 1 -- 172.21.15.12:0/1985376335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50181f4320 con 0x7f501815d9d0 2024-03-29T09:54:23.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.147+0000 7f501f000700 1 -- 172.21.15.12:0/1985376335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501815de50 con 0x7f501815d9d0 2024-03-29T09:54:23.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.148+0000 7f501f000700 1 -- 172.21.15.12:0/1985376335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50180d1ca0 con 0x7f501815d9d0 2024-03-29T09:54:23.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.149+0000 7f500fe00700 1 -- 172.21.15.12:0/1985376335 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5010007470 con 0x7f501815d9d0 2024-03-29T09:54:23.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.150+0000 7f500fe00700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ff806ac90 0x7f4ff806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.150+0000 7f500fe00700 1 -- 172.21.15.12:0/1985376335 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5010088a80 con 0x7f501815d9d0 2024-03-29T09:54:23.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.150+0000 7f5017e00700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ff806ac90 0x7f4ff806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.151+0000 7f5017e00700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ff806ac90 0x7f4ff806d150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5008009b40 tx=0x7f5008009210 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:23.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.155+0000 7f500fe00700 1 -- 172.21.15.12:0/1985376335 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5010054040 con 0x7f501815d9d0 2024-03-29T09:54:23.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.370+0000 7f501f000700 1 -- 172.21.15.12:0/1985376335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f501815e070 con 0x7f501815d9d0 2024-03-29T09:54:23.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.370+0000 7f500fe00700 1 -- 172.21.15.12:0/1985376335 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5010018020 con 0x7f501815d9d0 2024-03-29T09:54:23.371 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:54:23.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.372+0000 7f501f000700 1 -- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ff806ac90 msgr2=0x7f4ff806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.372+0000 7f501f000700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ff806ac90 0x7f4ff806d150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5008009b40 tx=0x7f5008009210 comp rx=0 tx=0).stop 2024-03-29T09:54:23.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.372+0000 7f501f000700 1 -- 172.21.15.12:0/1985376335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501815d9d0 msgr2=0x7f50180b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.372+0000 7f501f000700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501815d9d0 0x7f50180b4f00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f501000cca0 tx=0x7f501000a060 comp rx=0 tx=0).stop 2024-03-29T09:54:23.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.373+0000 7f501f000700 1 -- 172.21.15.12:0/1985376335 shutdown_connections 2024-03-29T09:54:23.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.373+0000 7f501f000700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ff806ac90 0x7f4ff806d150 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.373+0000 7f501f000700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501815d9d0 0x7f50180b4f00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.373+0000 7f501f000700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50180d16a0 0x7f50181f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.373+0000 7f501f000700 1 --2- 172.21.15.12:0/1985376335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50180d2370 0x7f50180b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.373+0000 7f501f000700 1 -- 172.21.15.12:0/1985376335 >> 172.21.15.12:0/1985376335 conn(0x7f501808aff0 msgr2=0x7f50180b19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:23.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.373+0000 7f501f000700 1 -- 172.21.15.12:0/1985376335 shutdown_connections 2024-03-29T09:54:23.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.373+0000 7f501f000700 1 -- 172.21.15.12:0/1985376335 wait complete. 2024-03-29T09:54:23.391 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:54:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.586+0000 7f9304c00700 1 Processor -- start 2024-03-29T09:54:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.586+0000 7f9304c00700 1 -- start start 2024-03-29T09:54:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.587+0000 7f9304c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9300160f10 0x7f93000d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f9304c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f93000cf230 0x7f93000d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f9304c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93000ceda0 con 0x7f9300161590 2024-03-29T09:54:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f9304c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93000d23d0 con 0x7f93000cf230 2024-03-29T09:54:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f9304c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93001612f0 con 0x7f9300160f10 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fd400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f93000cf230 0x7f93000d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fde00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9300160f10 0x7f93000d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fd400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f93000cf230 0x7f93000d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58068/0 (socket says 172.21.15.12:58068) 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fe800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9300161590 0x7f93000b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44020/0 (socket says 172.21.15.12:44020) 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fd400700 1 -- 172.21.15.12:0/2288399125 learned_addr learned my addr 172.21.15.12:0/2288399125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fd400700 1 -- 172.21.15.12:0/2288399125 >> v2:172.21.15.159:3300/0 conn(0x7f9300160f10 msgr2=0x7f93000d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fd400700 1 --2- 172.21.15.12:0/2288399125 >> v2:172.21.15.159:3300/0 conn(0x7f9300160f10 0x7f93000d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fd400700 1 -- 172.21.15.12:0/2288399125 >> v1:172.21.15.12:6789/0 conn(0x7f9300161590 legacy=0x7f93000b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fd400700 1 -- 172.21.15.12:0/2288399125 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93000ce470 con 0x7f93000cf230 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fde00700 1 --2- 172.21.15.12:0/2288399125 >> v2:172.21.15.159:3300/0 conn(0x7f9300160f10 0x7f93000d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.588+0000 7f92fca00700 1 -- 172.21.15.12:0/2288399125 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247901704 0 0) 0x7f93000ceda0 con 0x7f9300161590 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.589+0000 7f92fd400700 1 --2- 172.21.15.12:0/2288399125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93000cf230 0x7f93000d1e90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f93000d1080 tx=0x7f92ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80527a9b4ccf3a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.589+0000 7f92fca00700 1 -- 172.21.15.12:0/2288399125 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec011830 con 0x7f93000cf230 2024-03-29T09:54:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.589+0000 7f92fca00700 1 -- 172.21.15.12:0/2288399125 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec011990 con 0x7f93000cf230 2024-03-29T09:54:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.589+0000 7f92fca00700 1 -- 172.21.15.12:0/2288399125 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec011d20 con 0x7f93000cf230 2024-03-29T09:54:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.589+0000 7f9304c00700 1 -- 172.21.15.12:0/2288399125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93000cf230 msgr2=0x7f93000d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.589+0000 7f9304c00700 1 --2- 172.21.15.12:0/2288399125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93000cf230 0x7f93000d1e90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f93000d1080 tx=0x7f92ec009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.589+0000 7f9304c00700 1 -- 172.21.15.12:0/2288399125 shutdown_connections 2024-03-29T09:54:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.589+0000 7f9304c00700 1 --2- 172.21.15.12:0/2288399125 >> v2:172.21.15.159:3300/0 conn(0x7f9300160f10 0x7f93000d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.589+0000 7f9304c00700 1 --2- 172.21.15.12:0/2288399125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93000cf230 0x7f93000d1e90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.589+0000 7f9304c00700 1 -- 172.21.15.12:0/2288399125 >> 172.21.15.12:0/2288399125 conn(0x7f930008aff0 msgr2=0x7f93000d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.590+0000 7f9304c00700 1 -- 172.21.15.12:0/2288399125 shutdown_connections 2024-03-29T09:54:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.590+0000 7f9304c00700 1 -- 172.21.15.12:0/2288399125 wait complete. 2024-03-29T09:54:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.591+0000 7f9304c00700 1 Processor -- start 2024-03-29T09:54:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.591+0000 7f9304c00700 1 -- start start 2024-03-29T09:54:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.592+0000 7f9304c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93000cf230 0x7f93001516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.592+0000 7f9304c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9300160f10 0x7f93000b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.592+0000 7f9304c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9300161590 0x7f93001f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.592+0000 7f9304c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9300157c00 con 0x7f9300161590 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.592+0000 7f9304c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93001581c0 con 0x7f9300160f10 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.592+0000 7f9304c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9300151c10 con 0x7f93000cf230 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.592+0000 7f92fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9300161590 0x7f93001f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9300161590 0x7f93001f6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58084/0 (socket says 172.21.15.12:58084) 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fe800700 1 -- 172.21.15.12:0/3961674778 learned_addr learned my addr 172.21.15.12:0/3961674778 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fd400700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9300160f10 0x7f93000b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fde00700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93000cf230 0x7f93001516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fe800700 1 -- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93000cf230 msgr2=0x7f93001516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fe800700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93000cf230 0x7f93001516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fe800700 1 -- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9300160f10 msgr2=0x7f93000b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fe800700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9300160f10 0x7f93000b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fe800700 1 -- 172.21.15.12:0/3961674778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f00094a0 con 0x7f9300161590 2024-03-29T09:54:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fd400700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9300160f10 0x7f93000b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fde00700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93000cf230 0x7f93001516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92fe800700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9300161590 0x7f93001f6ff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f92f800cba0 tx=0x7f92f800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92f6000700 1 -- 172.21.15.12:0/3961674778 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f800ab80 con 0x7f9300161590 2024-03-29T09:54:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92f6000700 1 -- 172.21.15.12:0/3961674778 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f800ad00 con 0x7f9300161590 2024-03-29T09:54:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f9304c00700 1 -- 172.21.15.12:0/3961674778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ec009550 con 0x7f9300161590 2024-03-29T09:54:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.593+0000 7f92f6000700 1 -- 172.21.15.12:0/3961674778 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f8007310 con 0x7f9300161590 2024-03-29T09:54:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.594+0000 7f9304c00700 1 -- 172.21.15.12:0/3961674778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93000b6f10 con 0x7f9300161590 2024-03-29T09:54:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.595+0000 7f9304c00700 1 -- 172.21.15.12:0/3961674778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9300067720 con 0x7f9300161590 2024-03-29T09:54:23.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.595+0000 7f92f6000700 1 -- 172.21.15.12:0/3961674778 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f92f8007470 con 0x7f9300161590 2024-03-29T09:54:23.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.597+0000 7f92f6000700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92e006ac90 0x7f92e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:23.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.598+0000 7f92f6000700 1 -- 172.21.15.12:0/3961674778 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f92f8088ee0 con 0x7f9300161590 2024-03-29T09:54:23.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.598+0000 7f92fde00700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92e006ac90 0x7f92e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:23.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.599+0000 7f92fde00700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92e006ac90 0x7f92e006d150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f92f0009c50 tx=0x7f92f0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:23.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.602+0000 7f92f6000700 1 -- 172.21.15.12:0/3961674778 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92f8054370 con 0x7f9300161590 2024-03-29T09:54:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.865+0000 7f9304c00700 1 -- 172.21.15.12:0/3961674778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f930006bf30 con 0x7f9300161590 2024-03-29T09:54:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.865+0000 7f92f6000700 1 -- 172.21.15.12:0/3961674778 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3530 (secure 0 0 0) 0x7f92f8018090 con 0x7f9300161590 2024-03-29T09:54:23.866 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:23.866 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/4093246304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4093246304},{"type":"v1","addr":"172.21.15.12:6833","nonce":4093246304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1631416176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1631416176},{"type":"v1","addr":"172.21.15.159:6836","nonce":1631416176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3828834681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3828834681},{"type":"v1","addr":"172.21.15.159:6837","nonce":3828834681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3922451359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3922451359},{"type":"v1","addr":"172.21.15.12:6835","nonce":3922451359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 -- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92e006ac90 msgr2=0x7f92e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92e006ac90 0x7f92e006d150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f92f0009c50 tx=0x7f92f0008040 comp rx=0 tx=0).stop 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 -- 172.21.15.12:0/3961674778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9300161590 msgr2=0x7f93001f6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9300161590 0x7f93001f6ff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f92f800cba0 tx=0x7f92f800cf60 comp rx=0 tx=0).stop 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 -- 172.21.15.12:0/3961674778 shutdown_connections 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92e006ac90 0x7f92e006d150 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9300161590 0x7f93001f6ff0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93000cf230 0x7f93001516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 --2- 172.21.15.12:0/3961674778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9300160f10 0x7f93000b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 -- 172.21.15.12:0/3961674778 >> 172.21.15.12:0/3961674778 conn(0x7f930008aff0 msgr2=0x7f93000b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 -- 172.21.15.12:0/3961674778 shutdown_connections 2024-03-29T09:54:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:23.868+0000 7f9304c00700 1 -- 172.21.15.12:0/3961674778 wait complete. 2024-03-29T09:54:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 3 2024-03-29T09:54:23.887 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:24.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.083+0000 7f77efc00700 1 Processor -- start 2024-03-29T09:54:24.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.083+0000 7f77efc00700 1 -- start start 2024-03-29T09:54:24.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.084+0000 7f77efc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f77e80d18b0 0x7f77e80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.084+0000 7f77efc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77e8161590 0x7f77e8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.084+0000 7f77efc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e80ceda0 con 0x7f77e8160f10 2024-03-29T09:54:24.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.084+0000 7f77efc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e80d1d30 con 0x7f77e8161590 2024-03-29T09:54:24.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.084+0000 7f77efc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e8161f60 con 0x7f77e80d18b0 2024-03-29T09:54:24.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.085+0000 7f77edc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77e8161590 0x7f77e8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.085+0000 7f77ed200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f77e8160f10 0x7f77e80d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44032/0 (socket says 172.21.15.12:44032) 2024-03-29T09:54:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.085+0000 7f77ed200700 1 -- 172.21.15.12:0/2761061581 learned_addr learned my addr 172.21.15.12:0/2761061581 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.085+0000 7f77e7e00700 1 --2- 172.21.15.12:0/2761061581 >> v2:172.21.15.159:3300/0 conn(0x7f77e80d18b0 0x7f77e80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.085+0000 7f77edc00700 1 -- 172.21.15.12:0/2761061581 >> v2:172.21.15.159:3300/0 conn(0x7f77e80d18b0 msgr2=0x7f77e80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.085+0000 7f77edc00700 1 --2- 172.21.15.12:0/2761061581 >> v2:172.21.15.159:3300/0 conn(0x7f77e80d18b0 0x7f77e80cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.085+0000 7f77edc00700 1 -- 172.21.15.12:0/2761061581 >> v1:172.21.15.12:6789/0 conn(0x7f77e8160f10 legacy=0x7f77e80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.085+0000 7f77edc00700 1 -- 172.21.15.12:0/2761061581 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e81612f0 con 0x7f77e8161590 2024-03-29T09:54:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.085+0000 7f77e7400700 1 -- 172.21.15.12:0/2761061581 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376130041 0 0) 0x7f77e80ceda0 con 0x7f77e8160f10 2024-03-29T09:54:24.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.085+0000 7f77edc00700 1 --2- 172.21.15.12:0/2761061581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e8161590 0x7f77e8161a20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f77e808ca30 tx=0x7f77e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5f97161bf4af70 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:24.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77e7400700 1 -- 172.21.15.12:0/2761061581 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e000e070 con 0x7f77e8161590 2024-03-29T09:54:24.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77e7400700 1 -- 172.21.15.12:0/2761061581 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e000bb20 con 0x7f77e8161590 2024-03-29T09:54:24.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77e7400700 1 -- 172.21.15.12:0/2761061581 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0012430 con 0x7f77e8161590 2024-03-29T09:54:24.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77efc00700 1 -- 172.21.15.12:0/2761061581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e8161590 msgr2=0x7f77e8161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77efc00700 1 --2- 172.21.15.12:0/2761061581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e8161590 0x7f77e8161a20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f77e808ca30 tx=0x7f77e000b330 comp rx=0 tx=0).stop 2024-03-29T09:54:24.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77efc00700 1 -- 172.21.15.12:0/2761061581 shutdown_connections 2024-03-29T09:54:24.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77efc00700 1 --2- 172.21.15.12:0/2761061581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e8161590 0x7f77e8161a20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77efc00700 1 --2- 172.21.15.12:0/2761061581 >> v2:172.21.15.159:3300/0 conn(0x7f77e80d18b0 0x7f77e80cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77efc00700 1 -- 172.21.15.12:0/2761061581 >> 172.21.15.12:0/2761061581 conn(0x7f77e808aff0 msgr2=0x7f77e80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:24.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77efc00700 1 -- 172.21.15.12:0/2761061581 shutdown_connections 2024-03-29T09:54:24.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.086+0000 7f77efc00700 1 -- 172.21.15.12:0/2761061581 wait complete. 2024-03-29T09:54:24.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.088+0000 7f77efc00700 1 Processor -- start 2024-03-29T09:54:24.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.088+0000 7f77efc00700 1 -- start start 2024-03-29T09:54:24.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.089+0000 7f77efc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e80d18b0 0x7f77e8155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.089+0000 7f77efc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77e8160f10 0x7f77e80b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.089+0000 7f77efc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77e8161590 0x7f77e80b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.089+0000 7f77efc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80b78c0 con 0x7f77e80d18b0 2024-03-29T09:54:24.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.089+0000 7f77efc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e8155d50 con 0x7f77e8160f10 2024-03-29T09:54:24.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.089+0000 7f77efc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80b6c40 con 0x7f77e8161590 2024-03-29T09:54:24.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.089+0000 7f77ed200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e80d18b0 0x7f77e8155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77ed200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e80d18b0 0x7f77e8155810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58094/0 (socket says 172.21.15.12:58094) 2024-03-29T09:54:24.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77ed200700 1 -- 172.21.15.12:0/2625504024 learned_addr learned my addr 172.21.15.12:0/2625504024 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:24.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77e7e00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77e8160f10 0x7f77e80b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77edc00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77e8161590 0x7f77e80b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77ed200700 1 -- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77e8161590 msgr2=0x7f77e80b8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77ed200700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77e8161590 0x7f77e80b8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77ed200700 1 -- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77e8160f10 msgr2=0x7f77e80b6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77ed200700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77e8160f10 0x7f77e80b6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77ed200700 1 -- 172.21.15.12:0/2625504024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e0009a70 con 0x7f77e80d18b0 2024-03-29T09:54:24.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77edc00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77e8161590 0x7f77e80b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:24.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77ed200700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e80d18b0 0x7f77e8155810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00c550 tx=0x7f77dc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:24.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.090+0000 7f77e4c00700 1 -- 172.21.15.12:0/2625504024 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc004c80 con 0x7f77e80d18b0 2024-03-29T09:54:24.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.091+0000 7f77e4c00700 1 -- 172.21.15.12:0/2625504024 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc004e00 con 0x7f77e80d18b0 2024-03-29T09:54:24.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.091+0000 7f77e4c00700 1 -- 172.21.15.12:0/2625504024 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc007310 con 0x7f77e80d18b0 2024-03-29T09:54:24.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.091+0000 7f77efc00700 1 -- 172.21.15.12:0/2625504024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e80b6f20 con 0x7f77e80d18b0 2024-03-29T09:54:24.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.091+0000 7f77efc00700 1 -- 172.21.15.12:0/2625504024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e80b74e0 con 0x7f77e80d18b0 2024-03-29T09:54:24.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.092+0000 7f77efc00700 1 -- 172.21.15.12:0/2625504024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e8067720 con 0x7f77e80d18b0 2024-03-29T09:54:24.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.093+0000 7f77e4c00700 1 -- 172.21.15.12:0/2625504024 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f77dc007470 con 0x7f77e80d18b0 2024-03-29T09:54:24.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.094+0000 7f77e4c00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f77c806ac90 0x7f77c806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.094+0000 7f77e4c00700 1 -- 172.21.15.12:0/2625504024 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f77dc088d20 con 0x7f77e80d18b0 2024-03-29T09:54:24.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.094+0000 7f77e7e00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f77c806ac90 0x7f77c806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.095+0000 7f77e7e00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f77c806ac90 0x7f77c806d150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77e8158f20 tx=0x7f77d4009210 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:24.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.098+0000 7f77e4c00700 1 -- 172.21.15.12:0/2625504024 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f77dc0541b0 con 0x7f77e80d18b0 2024-03-29T09:54:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.313+0000 7f77efc00700 1 -- 172.21.15.12:0/2625504024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77e80b88d0 con 0x7f77e80d18b0 2024-03-29T09:54:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.314+0000 7f77e4c00700 1 -- 172.21.15.12:0/2625504024 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12382 (secure 0 0 0) 0x7f77dc0577d0 con 0x7f77e80d18b0 2024-03-29T09:54:24.314 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:24.315 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":16,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:18.853550+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.316+0000 7f77efc00700 1 -- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f77c806ac90 msgr2=0x7f77c806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.316+0000 7f77efc00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f77c806ac90 0x7f77c806d150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77e8158f20 tx=0x7f77d4009210 comp rx=0 tx=0).stop 2024-03-29T09:54:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.316+0000 7f77efc00700 1 -- 172.21.15.12:0/2625504024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e80d18b0 msgr2=0x7f77e8155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.316+0000 7f77efc00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e80d18b0 0x7f77e8155810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00c550 tx=0x7f77dc00c860 comp rx=0 tx=0).stop 2024-03-29T09:54:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.316+0000 7f77efc00700 1 -- 172.21.15.12:0/2625504024 shutdown_connections 2024-03-29T09:54:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.316+0000 7f77efc00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f77c806ac90 0x7f77c806d150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.316+0000 7f77efc00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77e8161590 0x7f77e80b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.316+0000 7f77efc00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77e80d18b0 0x7f77e8155810 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.316+0000 7f77efc00700 1 --2- 172.21.15.12:0/2625504024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77e8160f10 0x7f77e80b6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.316+0000 7f77efc00700 1 -- 172.21.15.12:0/2625504024 >> 172.21.15.12:0/2625504024 conn(0x7f77e808aff0 msgr2=0x7f77e80cc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:24.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.317+0000 7f77efc00700 1 -- 172.21.15.12:0/2625504024 shutdown_connections 2024-03-29T09:54:24.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.317+0000 7f77efc00700 1 -- 172.21.15.12:0/2625504024 wait complete. 2024-03-29T09:54:24.335 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:54:24.335 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:54:24.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.534+0000 7f10b8800700 1 Processor -- start 2024-03-29T09:54:24.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.534+0000 7f10b8800700 1 -- start start 2024-03-29T09:54:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.535+0000 7f10b8800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f10b0157f90 0x7f10b00b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.535+0000 7f10b8800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b01613e0 0x7f10b0161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.535+0000 7f10b8800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b00b3070 con 0x7f10b00b3420 2024-03-29T09:54:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.535+0000 7f10b8800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b0158370 con 0x7f10b0157f90 2024-03-29T09:54:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.535+0000 7f10b8800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b00b2710 con 0x7f10b01613e0 2024-03-29T09:54:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b6800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f10b00b3420 0x7f10b00cd020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44044/0 (socket says 172.21.15.12:44044) 2024-03-29T09:54:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b6800700 1 -- 172.21.15.12:0/2611676819 learned_addr learned my addr 172.21.15.12:0/2611676819 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b5e00700 1 --2- 172.21.15.12:0/2611676819 >> v2:172.21.15.12:3300/0 conn(0x7f10b0157f90 0x7f10b00b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b5400700 1 --2- 172.21.15.12:0/2611676819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b01613e0 0x7f10b0161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b4a00700 1 -- 172.21.15.12:0/2611676819 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076062832 0 0) 0x7f10ac001490 con 0x7f10b00b3420 2024-03-29T09:54:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b4a00700 1 -- 172.21.15.12:0/2611676819 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10980035b0 con 0x7f10b00b3420 2024-03-29T09:54:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b5e00700 1 -- 172.21.15.12:0/2611676819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b01613e0 msgr2=0x7f10b0161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b5e00700 1 --2- 172.21.15.12:0/2611676819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b01613e0 0x7f10b0161860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b5e00700 1 -- 172.21.15.12:0/2611676819 >> v1:172.21.15.12:6789/0 conn(0x7f10b00b3420 legacy=0x7f10b00cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b5e00700 1 -- 172.21.15.12:0/2611676819 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b00b2070 con 0x7f10b0157f90 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.536+0000 7f10b5e00700 1 --2- 172.21.15.12:0/2611676819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b0157f90 0x7f10b00b2b30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009840 tx=0x7f10a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70460525c8e1aa65 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.537+0000 7f10b4a00700 1 -- 172.21.15.12:0/2611676819 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4011830 con 0x7f10b0157f90 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.537+0000 7f10b4a00700 1 -- 172.21.15.12:0/2611676819 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a4011990 con 0x7f10b0157f90 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.537+0000 7f10b4a00700 1 -- 172.21.15.12:0/2611676819 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4011d20 con 0x7f10b0157f90 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.537+0000 7f10b8800700 1 -- 172.21.15.12:0/2611676819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b0157f90 msgr2=0x7f10b00b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.537+0000 7f10b8800700 1 --2- 172.21.15.12:0/2611676819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b0157f90 0x7f10b00b2b30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009840 tx=0x7f10a4009b50 comp rx=0 tx=0).stop 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.537+0000 7f10b8800700 1 -- 172.21.15.12:0/2611676819 shutdown_connections 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.537+0000 7f10b8800700 1 --2- 172.21.15.12:0/2611676819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b0157f90 0x7f10b00b2b30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.537+0000 7f10b8800700 1 --2- 172.21.15.12:0/2611676819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b01613e0 0x7f10b0161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.537+0000 7f10b8800700 1 -- 172.21.15.12:0/2611676819 >> 172.21.15.12:0/2611676819 conn(0x7f10b00b0d70 msgr2=0x7f10b00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.538+0000 7f10b8800700 1 -- 172.21.15.12:0/2611676819 shutdown_connections 2024-03-29T09:54:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.538+0000 7f10b8800700 1 -- 172.21.15.12:0/2611676819 wait complete. 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.539+0000 7f10b8800700 1 Processor -- start 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.539+0000 7f10b8800700 1 -- start start 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.540+0000 7f10b8800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b00b3420 0x7f10b01f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.540+0000 7f10b8800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b0157f90 0x7f10b00cf640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.540+0000 7f10b8800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b01613e0 0x7f10b015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.540+0000 7f10b8800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b00ca260 con 0x7f10b01613e0 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.540+0000 7f10b8800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b01f3090 con 0x7f10b0157f90 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.540+0000 7f10b8800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b00ca680 con 0x7f10b00b3420 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b01613e0 0x7f10b015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b01613e0 0x7f10b015f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58100/0 (socket says 172.21.15.12:58100) 2024-03-29T09:54:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b6800700 1 -- 172.21.15.12:0/2100530934 learned_addr learned my addr 172.21.15.12:0/2100530934 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b5e00700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b00b3420 0x7f10b01f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b5400700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b0157f90 0x7f10b00cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b6800700 1 -- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b00b3420 msgr2=0x7f10b01f2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b6800700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b00b3420 0x7f10b01f2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b6800700 1 -- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b0157f90 msgr2=0x7f10b00cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b6800700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b0157f90 0x7f10b00cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b6800700 1 -- 172.21.15.12:0/2100530934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a40094a0 con 0x7f10b01613e0 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.541+0000 7f10b6800700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b01613e0 0x7f10b015f9a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00cb10 tx=0x7f10ac00ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.542+0000 7f10aa000700 1 -- 172.21.15.12:0/2100530934 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac00a960 con 0x7f10b01613e0 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.542+0000 7f10aa000700 1 -- 172.21.15.12:0/2100530934 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ac00aac0 con 0x7f10b01613e0 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.542+0000 7f10b8800700 1 -- 172.21.15.12:0/2100530934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b00c95e0 con 0x7f10b01613e0 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.542+0000 7f10aa000700 1 -- 172.21.15.12:0/2100530934 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac00ae50 con 0x7f10b01613e0 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.542+0000 7f10b8800700 1 -- 172.21.15.12:0/2100530934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b00c9ba0 con 0x7f10b01613e0 2024-03-29T09:54:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.543+0000 7f10b8800700 1 -- 172.21.15.12:0/2100530934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b00ca3a0 con 0x7f10b01613e0 2024-03-29T09:54:24.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.544+0000 7f10aa000700 1 -- 172.21.15.12:0/2100530934 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f10ac01b3b0 con 0x7f10b01613e0 2024-03-29T09:54:24.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.545+0000 7f10aa000700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f109406ac90 0x7f109406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.545+0000 7f10aa000700 1 -- 172.21.15.12:0/2100530934 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f10ac088ab0 con 0x7f10b01613e0 2024-03-29T09:54:24.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.545+0000 7f10b5e00700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f109406ac90 0x7f109406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.546+0000 7f10b5e00700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f109406ac90 0x7f109406d150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f10a4000c00 tx=0x7f10a4008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:24.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.549+0000 7f10aa000700 1 -- 172.21.15.12:0/2100530934 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f10b00ca3a0 con 0x7f10b01613e0 2024-03-29T09:54:24.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.762+0000 7f10b8800700 1 -- 172.21.15.12:0/2100530934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10b00b3070 con 0x7f10b01613e0 2024-03-29T09:54:24.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.763+0000 7f10aa000700 1 -- 172.21.15.12:0/2100530934 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10b00b3070 con 0x7f10b01613e0 2024-03-29T09:54:24.764 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:54:24.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.765+0000 7f10b8800700 1 -- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f109406ac90 msgr2=0x7f109406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.765+0000 7f10b8800700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f109406ac90 0x7f109406d150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f10a4000c00 tx=0x7f10a4008030 comp rx=0 tx=0).stop 2024-03-29T09:54:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.765+0000 7f10b8800700 1 -- 172.21.15.12:0/2100530934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b01613e0 msgr2=0x7f10b015f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.765+0000 7f10b8800700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b01613e0 0x7f10b015f9a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00cb10 tx=0x7f10ac00ce20 comp rx=0 tx=0).stop 2024-03-29T09:54:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.766+0000 7f10b8800700 1 -- 172.21.15.12:0/2100530934 shutdown_connections 2024-03-29T09:54:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.766+0000 7f10b8800700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f109406ac90 0x7f109406d150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.766+0000 7f10b8800700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10b01613e0 0x7f10b015f9a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.766+0000 7f10b8800700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b00b3420 0x7f10b01f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.766+0000 7f10b8800700 1 --2- 172.21.15.12:0/2100530934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b0157f90 0x7f10b00cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.766+0000 7f10b8800700 1 -- 172.21.15.12:0/2100530934 >> 172.21.15.12:0/2100530934 conn(0x7f10b00b0d70 msgr2=0x7f10b00cd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.766+0000 7f10b8800700 1 -- 172.21.15.12:0/2100530934 shutdown_connections 2024-03-29T09:54:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.766+0000 7f10b8800700 1 -- 172.21.15.12:0/2100530934 wait complete. 2024-03-29T09:54:24.784 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T09:54:24.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.982+0000 7ff92e800700 1 Processor -- start 2024-03-29T09:54:24.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.982+0000 7ff92e800700 1 -- start start 2024-03-29T09:54:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.983+0000 7ff92e800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 0x7ff9280d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.983+0000 7ff92e800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff9280cf230 0x7ff9280d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.983+0000 7ff92e800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9280ceda0 con 0x7ff928161590 2024-03-29T09:54:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.983+0000 7ff92e800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9280d23d0 con 0x7ff9280cf230 2024-03-29T09:54:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.983+0000 7ff92e800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9281612f0 con 0x7ff928160f10 2024-03-29T09:54:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.983+0000 7ff926a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff9280cf230 0x7ff9280d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.983+0000 7ff927400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 0x7ff9280d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.983+0000 7ff927400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 0x7ff9280d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:44208/0 (socket says 172.21.15.12:44208) 2024-03-29T09:54:24.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.983+0000 7ff926a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff9280cf230 0x7ff9280d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:40822/0 (socket says 172.21.15.12:40822) 2024-03-29T09:54:24.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.983+0000 7ff927400700 1 -- 172.21.15.12:0/928812022 learned_addr learned my addr 172.21.15.12:0/928812022 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.984+0000 7ff927400700 1 -- 172.21.15.12:0/928812022 >> v2:172.21.15.159:3300/0 conn(0x7ff9280cf230 msgr2=0x7ff9280d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.984+0000 7ff927400700 1 --2- 172.21.15.12:0/928812022 >> v2:172.21.15.159:3300/0 conn(0x7ff9280cf230 0x7ff9280d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.984+0000 7ff927400700 1 -- 172.21.15.12:0/928812022 >> v1:172.21.15.159:6789/0 conn(0x7ff928161590 legacy=0x7ff9280b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.984+0000 7ff927400700 1 -- 172.21.15.12:0/928812022 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9280ce470 con 0x7ff928160f10 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.984+0000 7ff926a00700 1 --2- 172.21.15.12:0/928812022 >> v2:172.21.15.159:3300/0 conn(0x7ff9280cf230 0x7ff9280d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.984+0000 7ff927400700 1 --2- 172.21.15.12:0/928812022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 0x7ff9280d18b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff9180098f0 tx=0x7ff918009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cce2d61ccd261e09 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff926000700 1 -- 172.21.15.12:0/928812022 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918011830 con 0x7ff928160f10 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff926000700 1 -- 172.21.15.12:0/928812022 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff918011990 con 0x7ff928160f10 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff926000700 1 -- 172.21.15.12:0/928812022 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918011d20 con 0x7ff928160f10 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff92e800700 1 -- 172.21.15.12:0/928812022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 msgr2=0x7ff9280d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff92e800700 1 --2- 172.21.15.12:0/928812022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 0x7ff9280d18b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff9180098f0 tx=0x7ff918009c00 comp rx=0 tx=0).stop 2024-03-29T09:54:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff92e800700 1 -- 172.21.15.12:0/928812022 shutdown_connections 2024-03-29T09:54:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff92e800700 1 --2- 172.21.15.12:0/928812022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 0x7ff9280d18b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff92e800700 1 --2- 172.21.15.12:0/928812022 >> v2:172.21.15.159:3300/0 conn(0x7ff9280cf230 0x7ff9280d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff92e800700 1 -- 172.21.15.12:0/928812022 >> 172.21.15.12:0/928812022 conn(0x7ff92808aff0 msgr2=0x7ff9280d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff92e800700 1 -- 172.21.15.12:0/928812022 shutdown_connections 2024-03-29T09:54:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.985+0000 7ff92e800700 1 -- 172.21.15.12:0/928812022 wait complete. 2024-03-29T09:54:24.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.987+0000 7ff92e800700 1 Processor -- start 2024-03-29T09:54:24.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.987+0000 7ff92e800700 1 -- start start 2024-03-29T09:54:24.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.988+0000 7ff92e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9280cf230 0x7ff928153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.988+0000 7ff92e800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 0x7ff928150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.988+0000 7ff92e800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff928161590 0x7ff928155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.988+0000 7ff92e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92815edc0 con 0x7ff9280cf230 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.988+0000 7ff92e800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9281542b0 con 0x7ff928161590 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.988+0000 7ff92e800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92815e420 con 0x7ff928160f10 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.988+0000 7ff927400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9280cf230 0x7ff928153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.989+0000 7ff926a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 0x7ff928150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.989+0000 7ff927400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9280cf230 0x7ff928153c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58108/0 (socket says 172.21.15.12:58108) 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.989+0000 7ff927400700 1 -- 172.21.15.12:0/799539855 learned_addr learned my addr 172.21.15.12:0/799539855 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.989+0000 7ff927e00700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff928161590 0x7ff928155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.989+0000 7ff927400700 1 -- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 msgr2=0x7ff928150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.989+0000 7ff927400700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 0x7ff928150fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.989+0000 7ff927400700 1 -- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff928161590 msgr2=0x7ff928155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.989+0000 7ff927400700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff928161590 0x7ff928155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.989+0000 7ff927400700 1 -- 172.21.15.12:0/799539855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9140094a0 con 0x7ff9280cf230 2024-03-29T09:54:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.989+0000 7ff927400700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9280cf230 0x7ff928153c20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff918009c40 tx=0x7ff918004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.990+0000 7ff91f400700 1 -- 172.21.15.12:0/799539855 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918004590 con 0x7ff9280cf230 2024-03-29T09:54:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.990+0000 7ff91f400700 1 -- 172.21.15.12:0/799539855 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91801e070 con 0x7ff9280cf230 2024-03-29T09:54:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.990+0000 7ff91f400700 1 -- 172.21.15.12:0/799539855 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9180038e0 con 0x7ff9280cf230 2024-03-29T09:54:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.990+0000 7ff92e800700 1 -- 172.21.15.12:0/799539855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff918009550 con 0x7ff9280cf230 2024-03-29T09:54:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.990+0000 7ff92e800700 1 -- 172.21.15.12:0/799539855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92815e9e0 con 0x7ff9280cf230 2024-03-29T09:54:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.991+0000 7ff92e800700 1 -- 172.21.15.12:0/799539855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92815ef00 con 0x7ff9280cf230 2024-03-29T09:54:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.991+0000 7ff91f400700 1 -- 172.21.15.12:0/799539855 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff918003a40 con 0x7ff9280cf230 2024-03-29T09:54:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.992+0000 7ff91f400700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff90806ac90 0x7ff90806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.992+0000 7ff91f400700 1 -- 172.21.15.12:0/799539855 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff918089e10 con 0x7ff9280cf230 2024-03-29T09:54:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.993+0000 7ff926a00700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff90806ac90 0x7ff90806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:24.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.993+0000 7ff926a00700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff90806ac90 0x7ff90806d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff9280b9310 tx=0x7ff914009210 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:24.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:24.997+0000 7ff91f400700 1 -- 172.21.15.12:0/799539855 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff91808d030 con 0x7ff9280cf230 2024-03-29T09:54:25.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.216+0000 7ff92e800700 1 -- 172.21.15.12:0/799539855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff9280b4950 con 0x7ff9280cf230 2024-03-29T09:54:25.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.217+0000 7ff91f400700 1 -- 172.21.15.12:0/799539855 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff918055330 con 0x7ff9280cf230 2024-03-29T09:54:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:25.217 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T09:54:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:54:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T09:54:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T09:54:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:54:25.217 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 -- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff90806ac90 msgr2=0x7ff90806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff90806ac90 0x7ff90806d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff9280b9310 tx=0x7ff914009210 comp rx=0 tx=0).stop 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 -- 172.21.15.12:0/799539855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9280cf230 msgr2=0x7ff928153c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9280cf230 0x7ff928153c20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff918009c40 tx=0x7ff918004750 comp rx=0 tx=0).stop 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 -- 172.21.15.12:0/799539855 shutdown_connections 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff90806ac90 0x7ff90806d150 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff928161590 0x7ff928155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9280cf230 0x7ff928153c20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 --2- 172.21.15.12:0/799539855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff928160f10 0x7ff928150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 -- 172.21.15.12:0/799539855 >> 172.21.15.12:0/799539855 conn(0x7ff92808aff0 msgr2=0x7ff9280cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:25.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 -- 172.21.15.12:0/799539855 shutdown_connections 2024-03-29T09:54:25.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.219+0000 7ff92e800700 1 -- 172.21.15.12:0/799539855 wait complete. 2024-03-29T09:54:25.237 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T09:54:25.238 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T09:54:25.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.439+0000 7f2d1da00700 1 Processor -- start 2024-03-29T09:54:25.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.439+0000 7f2d1da00700 1 -- start start 2024-03-29T09:54:25.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.440+0000 7f2d1da00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2d180ced30 0x7f2d180d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:25.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.440+0000 7f2d1da00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180d2710 0x7f2d1815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:25.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.440+0000 7f2d1da00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d180ce8a0 con 0x7f2d1815ddf0 2024-03-29T09:54:25.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.440+0000 7f2d1da00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1815ec60 con 0x7f2d180ced30 2024-03-29T09:54:25.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.440+0000 7f2d1da00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180cdc90 con 0x7f2d180d2710 2024-03-29T09:54:25.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.440+0000 7f2d16a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2d1815ddf0 0x7f2d1815e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44060/0 (socket says 172.21.15.12:44060) 2024-03-29T09:54:25.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.440+0000 7f2d16a00700 1 -- 172.21.15.12:0/4061311207 learned_addr learned my addr 172.21.15.12:0/4061311207 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d16000700 1 --2- 172.21.15.12:0/4061311207 >> v2:172.21.15.159:3300/0 conn(0x7f2d180ced30 0x7f2d180d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d17400700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180d2710 0x7f2d1815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928843727 0 0) 0x7f2d180ce8a0 con 0x7f2d1815ddf0 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cfc0035b0 con 0x7f2d1815ddf0 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4063475978 0 0) 0x7f2cfc0035b0 con 0x7f2d1815ddf0 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180d2710 msgr2=0x7f2d1815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180d2710 0x7f2d1815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 >> v2:172.21.15.159:3300/0 conn(0x7f2d180ced30 msgr2=0x7f2d180d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 --2- 172.21.15.12:0/4061311207 >> v2:172.21.15.159:3300/0 conn(0x7f2d180ced30 0x7f2d180d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d180cdf70 con 0x7f2d1815ddf0 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f2d08001670 con 0x7f2d1815ddf0 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d17400700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180d2710 0x7f2d1815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 >> v1:172.21.15.12:6789/0 conn(0x7f2d1815ddf0 legacy=0x7f2d1815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:25.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc003a00 0x7f2cfc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180d2710 0x7f2cfc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc004d90 0x7f2cfc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1815ec60 con 0x7f2cfc004d90 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180cdc90 con 0x7f2cfc003a00 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.441+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc00ba10 con 0x7f2d180d2710 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d17400700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc004d90 0x7f2cfc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d16a00700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc003a00 0x7f2cfc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d16000700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180d2710 0x7f2cfc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d17400700 1 -- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180d2710 msgr2=0x7f2cfc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d17400700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180d2710 0x7f2cfc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d17400700 1 -- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc003a00 msgr2=0x7f2cfc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d17400700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc003a00 0x7f2cfc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d17400700 1 -- 172.21.15.12:0/4061311207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d10009a70 con 0x7f2cfc004d90 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d17400700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc004d90 0x7f2cfc005210 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000e380 tx=0x7f2cfc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58bed24b4cc6cdf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1000ee20 con 0x7f2cfc004d90 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1000f040 con 0x7f2cfc004d90 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.442+0000 7f2d15600700 1 -- 172.21.15.12:0/4061311207 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d100146a0 con 0x7f2cfc004d90 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.443+0000 7f2d1da00700 1 -- 172.21.15.12:0/4061311207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc004d90 msgr2=0x7f2cfc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.443+0000 7f2d1da00700 1 --2- 172.21.15.12:0/4061311207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc004d90 0x7f2cfc005210 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000e380 tx=0x7f2cfc005880 comp rx=0 tx=0).stop 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.443+0000 7f2d17400700 1 -- 172.21.15.12:0/4061311207 reap_dead start 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.443+0000 7f2d1da00700 1 -- 172.21.15.12:0/4061311207 shutdown_connections 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.443+0000 7f2d1da00700 1 -- 172.21.15.12:0/4061311207 >> 172.21.15.12:0/4061311207 conn(0x7f2d1808aff0 msgr2=0x7f2d180d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.443+0000 7f2d1da00700 1 -- 172.21.15.12:0/4061311207 shutdown_connections 2024-03-29T09:54:25.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.443+0000 7f2d1da00700 1 -- 172.21.15.12:0/4061311207 wait complete. 2024-03-29T09:54:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.445+0000 7f2d1da00700 1 Processor -- start 2024-03-29T09:54:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.445+0000 7f2d1da00700 1 -- start start 2024-03-29T09:54:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d1da00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc004d90 0x7f2d1806cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d1da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d1814b400 0x7f2d1814b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d1da00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180b4ae0 0x7f2d180b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d1da00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180ca930 con 0x7f2d1814b400 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d1da00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180cac10 con 0x7f2cfc004d90 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d1da00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180caef0 con 0x7f2d180b4ae0 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d16000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d1814b400 0x7f2d1814b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d16000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d1814b400 0x7f2d1814b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58126/0 (socket says 172.21.15.12:58126) 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d16a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc004d90 0x7f2d1806cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d16000700 1 -- 172.21.15.12:0/2264933205 learned_addr learned my addr 172.21.15.12:0/2264933205 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.446+0000 7f2d17400700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180b4ae0 0x7f2d180b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d16000700 1 -- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180b4ae0 msgr2=0x7f2d180b4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d16000700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180b4ae0 0x7f2d180b4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d16000700 1 -- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc004d90 msgr2=0x7f2d1806cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d16000700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc004d90 0x7f2d1806cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d16000700 1 -- 172.21.15.12:0/2264933205 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d180cb1d0 con 0x7f2d1814b400 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d16a00700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc004d90 0x7f2d1806cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d16000700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d1814b400 0x7f2d1814b880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d04011aa0 tx=0x7f2d04011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d0ea00700 1 -- 172.21.15.12:0/2264933205 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0400f940 con 0x7f2d1814b400 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d0ea00700 1 -- 172.21.15.12:0/2264933205 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0400faa0 con 0x7f2d1814b400 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d0ea00700 1 -- 172.21.15.12:0/2264933205 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0400fe50 con 0x7f2d1814b400 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d1da00700 1 -- 172.21.15.12:0/2264933205 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d180cb4b0 con 0x7f2d1814b400 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.447+0000 7f2d1da00700 1 -- 172.21.15.12:0/2264933205 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1806d0c0 con 0x7f2d1814b400 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.448+0000 7f2d1da00700 1 -- 172.21.15.12:0/2264933205 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d180cb6d0 con 0x7f2d1814b400 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.449+0000 7f2d0ea00700 1 -- 172.21.15.12:0/2264933205 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2d04014440 con 0x7f2d1814b400 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.450+0000 7f2d0ea00700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf806ac90 0x7f2cf806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.450+0000 7f2d0ea00700 1 -- 172.21.15.12:0/2264933205 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2d04089750 con 0x7f2d1814b400 2024-03-29T09:54:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.451+0000 7f2d16a00700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf806ac90 0x7f2cf806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.451+0000 7f2d16a00700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf806ac90 0x7f2cf806d150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2d08001bf0 tx=0x7f2d08001dd0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.455+0000 7f2d0ea00700 1 -- 172.21.15.12:0/2264933205 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d04054be0 con 0x7f2d1814b400 2024-03-29T09:54:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.669+0000 7f2d1da00700 1 -- 172.21.15.12:0/2264933205 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2d18067720 con 0x7f2d1814b400 2024-03-29T09:54:25.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.887+0000 7f2d0ea00700 1 -- 172.21.15.12:0/2264933205 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2d04058200 con 0x7f2d1814b400 2024-03-29T09:54:25.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.890+0000 7f2d1da00700 1 -- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf806ac90 msgr2=0x7f2cf806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:25.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.890+0000 7f2d1da00700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf806ac90 0x7f2cf806d150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2d08001bf0 tx=0x7f2d08001dd0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.890+0000 7f2d1da00700 1 -- 172.21.15.12:0/2264933205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d1814b400 msgr2=0x7f2d1814b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.890+0000 7f2d1da00700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d1814b400 0x7f2d1814b880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d04011aa0 tx=0x7f2d04011db0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.891+0000 7f2d1da00700 1 -- 172.21.15.12:0/2264933205 shutdown_connections 2024-03-29T09:54:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.891+0000 7f2d1da00700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf806ac90 0x7f2cf806d150 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.891+0000 7f2d1da00700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d180b4ae0 0x7f2d180b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.891+0000 7f2d1da00700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc004d90 0x7f2d1806cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.891+0000 7f2d1da00700 1 --2- 172.21.15.12:0/2264933205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d1814b400 0x7f2d1814b880 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.891+0000 7f2d1da00700 1 -- 172.21.15.12:0/2264933205 >> 172.21.15.12:0/2264933205 conn(0x7f2d1808aff0 msgr2=0x7f2d180d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.891+0000 7f2d1da00700 1 -- 172.21.15.12:0/2264933205 shutdown_connections 2024-03-29T09:54:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:25.891+0000 7f2d1da00700 1 -- 172.21.15.12:0/2264933205 wait complete. 2024-03-29T09:54:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T09:54:25.915 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T09:54:26.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.119+0000 7fc221600700 1 Processor -- start 2024-03-29T09:54:26.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.119+0000 7fc221600700 1 -- start start 2024-03-29T09:54:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.120+0000 7fc221600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c15d9d0 0x7fc21c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc221600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc21c0d16a0 0x7fc21c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc221600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc21c0d2370 0x7fc21c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc221600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21c0d2820 con 0x7fc21c0d16a0 2024-03-29T09:54:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc221600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21c161420 con 0x7fc21c0d2370 2024-03-29T09:54:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc221600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c161700 con 0x7fc21c15d9d0 2024-03-29T09:54:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc21a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc21c0d16a0 0x7fc21c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc21a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc21c0d16a0 0x7fc21c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58142/0 (socket says 172.21.15.12:58142) 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc21a000700 1 -- 172.21.15.12:0/3306998809 learned_addr learned my addr 172.21.15.12:0/3306998809 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc21aa00700 1 --2- 172.21.15.12:0/3306998809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c15d9d0 0x7fc21c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc21b400700 1 --2- 172.21.15.12:0/3306998809 >> v2:172.21.15.159:3300/0 conn(0x7fc21c0d2370 0x7fc21c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc21a000700 1 -- 172.21.15.12:0/3306998809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c15d9d0 msgr2=0x7fc21c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc21a000700 1 --2- 172.21.15.12:0/3306998809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c15d9d0 0x7fc21c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc21a000700 1 -- 172.21.15.12:0/3306998809 >> v2:172.21.15.159:3300/0 conn(0x7fc21c0d2370 msgr2=0x7fc21c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc21a000700 1 --2- 172.21.15.12:0/3306998809 >> v2:172.21.15.159:3300/0 conn(0x7fc21c0d2370 0x7fc21c15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.121+0000 7fc21a000700 1 -- 172.21.15.12:0/3306998809 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c1619e0 con 0x7fc21c0d16a0 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc21a000700 1 --2- 172.21.15.12:0/3306998809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d16a0 0x7fc21c0d1e30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc21c0cdc00 tx=0x7fc208009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9858b84f701abb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc219600700 1 -- 172.21.15.12:0/3306998809 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208005e30 con 0x7fc21c0d16a0 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc219600700 1 -- 172.21.15.12:0/3306998809 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc208004030 con 0x7fc21c0d16a0 2024-03-29T09:54:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc219600700 1 -- 172.21.15.12:0/3306998809 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2080043e0 con 0x7fc21c0d16a0 2024-03-29T09:54:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc221600700 1 -- 172.21.15.12:0/3306998809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d16a0 msgr2=0x7fc21c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc221600700 1 --2- 172.21.15.12:0/3306998809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d16a0 0x7fc21c0d1e30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc21c0cdc00 tx=0x7fc208009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc221600700 1 -- 172.21.15.12:0/3306998809 shutdown_connections 2024-03-29T09:54:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc221600700 1 --2- 172.21.15.12:0/3306998809 >> v2:172.21.15.159:3300/0 conn(0x7fc21c0d2370 0x7fc21c15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc221600700 1 --2- 172.21.15.12:0/3306998809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c15d9d0 0x7fc21c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc221600700 1 --2- 172.21.15.12:0/3306998809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d16a0 0x7fc21c0d1e30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc221600700 1 -- 172.21.15.12:0/3306998809 >> 172.21.15.12:0/3306998809 conn(0x7fc21c08aff0 msgr2=0x7fc21c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.122+0000 7fc221600700 1 -- 172.21.15.12:0/3306998809 shutdown_connections 2024-03-29T09:54:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.123+0000 7fc221600700 1 -- 172.21.15.12:0/3306998809 wait complete. 2024-03-29T09:54:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.124+0000 7fc221600700 1 Processor -- start 2024-03-29T09:54:26.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.124+0000 7fc221600700 1 -- start start 2024-03-29T09:54:26.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.125+0000 7fc221600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c0d16a0 0x7fc21c1e82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:26.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.125+0000 7fc221600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d2370 0x7fc21c1e9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:26.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.125+0000 7fc221600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc21c15d9d0 0x7fc21c0b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:26.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.125+0000 7fc221600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c1e9bb0 con 0x7fc21c0d2370 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.125+0000 7fc221600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c1e77f0 con 0x7fc21c15d9d0 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.125+0000 7fc221600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c1e7ad0 con 0x7fc21c0d16a0 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d2370 0x7fc21c1e9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d2370 0x7fc21c1e9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58156/0 (socket says 172.21.15.12:58156) 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21a000700 1 -- 172.21.15.12:0/2135804858 learned_addr learned my addr 172.21.15.12:0/2135804858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21aa00700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c0d16a0 0x7fc21c1e82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21b400700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc21c15d9d0 0x7fc21c0b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21a000700 1 -- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c0d16a0 msgr2=0x7fc21c1e82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21a000700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c0d16a0 0x7fc21c1e82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21a000700 1 -- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc21c15d9d0 msgr2=0x7fc21c0b09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21a000700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc21c15d9d0 0x7fc21c0b09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21a000700 1 -- 172.21.15.12:0/2135804858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc208009500 con 0x7fc21c0d2370 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21b400700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc21c15d9d0 0x7fc21c0b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.126+0000 7fc21a000700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d2370 0x7fc21c1e9160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc208009130 tx=0x7fc208004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.127+0000 7fc212a00700 1 -- 172.21.15.12:0/2135804858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208003d20 con 0x7fc21c0d2370 2024-03-29T09:54:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.127+0000 7fc212a00700 1 -- 172.21.15.12:0/2135804858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc208003e80 con 0x7fc21c0d2370 2024-03-29T09:54:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.127+0000 7fc212a00700 1 -- 172.21.15.12:0/2135804858 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208011660 con 0x7fc21c0d2370 2024-03-29T09:54:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.127+0000 7fc221600700 1 -- 172.21.15.12:0/2135804858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21c1e7db0 con 0x7fc21c0d2370 2024-03-29T09:54:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.127+0000 7fc221600700 1 -- 172.21.15.12:0/2135804858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21c15cd90 con 0x7fc21c0d2370 2024-03-29T09:54:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.128+0000 7fc221600700 1 -- 172.21.15.12:0/2135804858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21c0d1ca0 con 0x7fc21c0d2370 2024-03-29T09:54:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.129+0000 7fc212a00700 1 -- 172.21.15.12:0/2135804858 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc2080117c0 con 0x7fc21c0d2370 2024-03-29T09:54:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.130+0000 7fc212a00700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1fc06ac90 0x7fc1fc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.130+0000 7fc212a00700 1 -- 172.21.15.12:0/2135804858 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc20808a7a0 con 0x7fc21c0d2370 2024-03-29T09:54:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.130+0000 7fc21aa00700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1fc06ac90 0x7fc1fc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.131+0000 7fc21aa00700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1fc06ac90 0x7fc1fc06d150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc20c006b20 tx=0x7fc20c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.134+0000 7fc212a00700 1 -- 172.21.15.12:0/2135804858 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc208055bd0 con 0x7fc21c0d2370 2024-03-29T09:54:26.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.345+0000 7fc221600700 1 -- 172.21.15.12:0/2135804858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc21c067720 con 0x7fc21c0d2370 2024-03-29T09:54:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.899+0000 7fc212a00700 1 -- 172.21.15.12:0/2135804858 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc2080591f0 con 0x7fc21c0d2370 2024-03-29T09:54:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.905+0000 7fc221600700 1 -- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1fc06ac90 msgr2=0x7fc1fc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.905+0000 7fc221600700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1fc06ac90 0x7fc1fc06d150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc20c006b20 tx=0x7fc20c008040 comp rx=0 tx=0).stop 2024-03-29T09:54:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.905+0000 7fc221600700 1 -- 172.21.15.12:0/2135804858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d2370 msgr2=0x7fc21c1e9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.905+0000 7fc221600700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d2370 0x7fc21c1e9160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc208009130 tx=0x7fc208004750 comp rx=0 tx=0).stop 2024-03-29T09:54:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.906+0000 7fc221600700 1 -- 172.21.15.12:0/2135804858 shutdown_connections 2024-03-29T09:54:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.906+0000 7fc221600700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1fc06ac90 0x7fc1fc06d150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.906+0000 7fc221600700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc21c15d9d0 0x7fc21c0b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.906+0000 7fc221600700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c0d16a0 0x7fc21c1e82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.906+0000 7fc221600700 1 --2- 172.21.15.12:0/2135804858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc21c0d2370 0x7fc21c1e9160 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc208009130 tx=0x7fc208004750 comp rx=0 tx=0).stop 2024-03-29T09:54:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.906+0000 7fc221600700 1 -- 172.21.15.12:0/2135804858 >> 172.21.15.12:0/2135804858 conn(0x7fc21c08aff0 msgr2=0x7fc21c0d39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.907+0000 7fc221600700 1 -- 172.21.15.12:0/2135804858 shutdown_connections 2024-03-29T09:54:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:26.907+0000 7fc221600700 1 -- 172.21.15.12:0/2135804858 wait complete. 2024-03-29T09:54:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T09:54:26.944 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T09:54:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.148+0000 7f8471200700 1 Processor -- start 2024-03-29T09:54:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.148+0000 7f8471200700 1 -- start start 2024-03-29T09:54:27.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.149+0000 7f8471200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f846c15d9d0 0x7f846c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.149+0000 7f8471200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f846c0d16a0 0x7f846c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.149+0000 7f8471200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846c161420 con 0x7f846c0d2370 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.149+0000 7f8471200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f846c0d27e0 con 0x7f846c15d9d0 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.149+0000 7f8471200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f846c161700 con 0x7f846c0d16a0 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.149+0000 7f8469e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f846c15d9d0 0x7f846c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.149+0000 7f8469e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f846c15d9d0 0x7f846c0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58168/0 (socket says 172.21.15.12:58168) 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.149+0000 7f8469e00700 1 -- 172.21.15.12:0/2264448483 learned_addr learned my addr 172.21.15.12:0/2264448483 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.149+0000 7f8469400700 1 --2- 172.21.15.12:0/2264448483 >> v2:172.21.15.159:3300/0 conn(0x7f846c0d16a0 0x7f846c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.150+0000 7f8469e00700 1 -- 172.21.15.12:0/2264448483 >> v2:172.21.15.159:3300/0 conn(0x7f846c0d16a0 msgr2=0x7f846c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.150+0000 7f8469e00700 1 --2- 172.21.15.12:0/2264448483 >> v2:172.21.15.159:3300/0 conn(0x7f846c0d16a0 0x7f846c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.150+0000 7f8469e00700 1 -- 172.21.15.12:0/2264448483 >> v1:172.21.15.159:6789/0 conn(0x7f846c0d2370 legacy=0x7f846c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.150+0000 7f8469e00700 1 -- 172.21.15.12:0/2264448483 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c1619e0 con 0x7f846c15d9d0 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.150+0000 7f8469e00700 1 --2- 172.21.15.12:0/2264448483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c15d9d0 0x7f846c0d3c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f846c0d12f0 tx=0x7f845c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cefb9ce298268f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.150+0000 7f8468a00700 1 -- 172.21.15.12:0/2264448483 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c00b870 con 0x7f846c15d9d0 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.150+0000 7f8468a00700 1 -- 172.21.15.12:0/2264448483 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845c00b9d0 con 0x7f846c15d9d0 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.150+0000 7f8468a00700 1 -- 172.21.15.12:0/2264448483 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c00bd60 con 0x7f846c15d9d0 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.151+0000 7f8471200700 1 -- 172.21.15.12:0/2264448483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c15d9d0 msgr2=0x7f846c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.151+0000 7f8471200700 1 --2- 172.21.15.12:0/2264448483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c15d9d0 0x7f846c0d3c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f846c0d12f0 tx=0x7f845c009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.151+0000 7f8471200700 1 -- 172.21.15.12:0/2264448483 shutdown_connections 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.151+0000 7f8471200700 1 --2- 172.21.15.12:0/2264448483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c15d9d0 0x7f846c0d3c30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.151+0000 7f8471200700 1 --2- 172.21.15.12:0/2264448483 >> v2:172.21.15.159:3300/0 conn(0x7f846c0d16a0 0x7f846c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.151+0000 7f8471200700 1 -- 172.21.15.12:0/2264448483 >> 172.21.15.12:0/2264448483 conn(0x7f846c08aff0 msgr2=0x7f846c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.151+0000 7f8471200700 1 -- 172.21.15.12:0/2264448483 shutdown_connections 2024-03-29T09:54:27.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.151+0000 7f8471200700 1 -- 172.21.15.12:0/2264448483 wait complete. 2024-03-29T09:54:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.153+0000 7f8471200700 1 Processor -- start 2024-03-29T09:54:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.153+0000 7f8471200700 1 -- start start 2024-03-29T09:54:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8471200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c0d16a0 0x7f846c0b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8471200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0d2370 0x7f846c1580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8471200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f846c15d9d0 0x7f846c0b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8471200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c0b6fd0 con 0x7f846c0d16a0 2024-03-29T09:54:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8471200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c0b79a0 con 0x7f846c15d9d0 2024-03-29T09:54:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8471200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846c0b63c0 con 0x7f846c0d2370 2024-03-29T09:54:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8469e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c0d16a0 0x7f846c0b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8469e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c0d16a0 0x7f846c0b7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58182/0 (socket says 172.21.15.12:58182) 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8469e00700 1 -- 172.21.15.12:0/2699793660 learned_addr learned my addr 172.21.15.12:0/2699793660 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f846a800700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f846c15d9d0 0x7f846c0b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8469400700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0d2370 0x7f846c1580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8469e00700 1 -- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0d2370 msgr2=0x7f846c1580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8469e00700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0d2370 0x7f846c1580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8469e00700 1 -- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f846c15d9d0 msgr2=0x7f846c0b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8469e00700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f846c15d9d0 0x7f846c0b8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.154+0000 7f8469e00700 1 -- 172.21.15.12:0/2699793660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c009550 con 0x7f846c0d16a0 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.155+0000 7f8469e00700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c0d16a0 0x7f846c0b7460 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f845c000c00 tx=0x7f845c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.155+0000 7f8462000700 1 -- 172.21.15.12:0/2699793660 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c004520 con 0x7f846c0d16a0 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.155+0000 7f8462000700 1 -- 172.21.15.12:0/2699793660 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845c01e070 con 0x7f846c0d16a0 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.155+0000 7f8471200700 1 -- 172.21.15.12:0/2699793660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c0b66a0 con 0x7f846c0d16a0 2024-03-29T09:54:27.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.155+0000 7f8462000700 1 -- 172.21.15.12:0/2699793660 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c0038a0 con 0x7f846c0d16a0 2024-03-29T09:54:27.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.155+0000 7f8471200700 1 -- 172.21.15.12:0/2699793660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846c0b6c60 con 0x7f846c0d16a0 2024-03-29T09:54:27.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.156+0000 7f8471200700 1 -- 172.21.15.12:0/2699793660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846c0d1ca0 con 0x7f846c0d16a0 2024-03-29T09:54:27.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.157+0000 7f8462000700 1 -- 172.21.15.12:0/2699793660 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f845c003a00 con 0x7f846c0d16a0 2024-03-29T09:54:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.158+0000 7f8462000700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f844c06ac90 0x7f844c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.159+0000 7f8462000700 1 -- 172.21.15.12:0/2699793660 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f845c020070 con 0x7f846c0d16a0 2024-03-29T09:54:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.159+0000 7f8469400700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f844c06ac90 0x7f844c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.160+0000 7f8469400700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f844c06ac90 0x7f844c06d150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f846c0cdc00 tx=0x7f8458008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:27.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.162+0000 7f8462000700 1 -- 172.21.15.12:0/2699793660 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f845c055fc0 con 0x7f846c0d16a0 2024-03-29T09:54:27.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.422+0000 7f8471200700 1 -- 172.21.15.12:0/2699793660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f846c06b9b0 con 0x7f846c0d16a0 2024-03-29T09:54:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.432+0000 7f8462000700 1 -- 172.21.15.12:0/2699793660 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f845c0595e0 con 0x7f846c0d16a0 2024-03-29T09:54:27.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.437+0000 7f8471200700 1 -- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f844c06ac90 msgr2=0x7f844c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.437+0000 7f8471200700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f844c06ac90 0x7f844c06d150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f846c0cdc00 tx=0x7f8458008040 comp rx=0 tx=0).stop 2024-03-29T09:54:27.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.437+0000 7f8471200700 1 -- 172.21.15.12:0/2699793660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c0d16a0 msgr2=0x7f846c0b7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.437+0000 7f8471200700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c0d16a0 0x7f846c0b7460 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f845c000c00 tx=0x7f845c004750 comp rx=0 tx=0).stop 2024-03-29T09:54:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.443+0000 7f8471200700 1 -- 172.21.15.12:0/2699793660 shutdown_connections 2024-03-29T09:54:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.443+0000 7f8471200700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f844c06ac90 0x7f844c06d150 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.443+0000 7f8471200700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f846c15d9d0 0x7f846c0b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.443+0000 7f8471200700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f846c0d16a0 0x7f846c0b7460 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.443+0000 7f8471200700 1 --2- 172.21.15.12:0/2699793660 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0d2370 0x7f846c1580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.443+0000 7f8471200700 1 -- 172.21.15.12:0/2699793660 >> 172.21.15.12:0/2699793660 conn(0x7f846c08aff0 msgr2=0x7f846c15bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:27.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.447+0000 7f8471200700 1 -- 172.21.15.12:0/2699793660 shutdown_connections 2024-03-29T09:54:27.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.448+0000 7f8471200700 1 -- 172.21.15.12:0/2699793660 wait complete. 2024-03-29T09:54:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 2 and data pool 3 2024-03-29T09:54:27.483 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:27.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.679+0000 7f4e97200700 1 Processor -- start 2024-03-29T09:54:27.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.679+0000 7f4e97200700 1 -- start start 2024-03-29T09:54:27.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.680+0000 7f4e97200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4e90160f10 0x7f4e900d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.680+0000 7f4e97200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e900ceda0 con 0x7f4e90161590 2024-03-29T09:54:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.680+0000 7f4e97200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e901612f0 con 0x7f4e900cf230 2024-03-29T09:54:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.680+0000 7f4e97200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e900ce470 con 0x7f4e90160f10 2024-03-29T09:54:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.680+0000 7f4e95200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4e90161590 0x7f4e900b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44074/0 (socket says 172.21.15.12:44074) 2024-03-29T09:54:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.680+0000 7f4e95200700 1 -- 172.21.15.12:0/3317367018 learned_addr learned my addr 172.21.15.12:0/3317367018 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.680+0000 7f4e8fe00700 1 --2- 172.21.15.12:0/3317367018 >> v2:172.21.15.159:3300/0 conn(0x7f4e90160f10 0x7f4e900d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.681+0000 7f4e8ea00700 1 -- 172.21.15.12:0/3317367018 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580315420 0 0) 0x7f4e900ceda0 con 0x7f4e90161590 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.681+0000 7f4e8ea00700 1 -- 172.21.15.12:0/3317367018 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e740035b0 con 0x7f4e90161590 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.681+0000 7f4e8fe00700 1 -- 172.21.15.12:0/3317367018 >> v1:172.21.15.159:6789/0 conn(0x7f4e900cf230 legacy=0x7f4e900d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.681+0000 7f4e8fe00700 1 -- 172.21.15.12:0/3317367018 >> v1:172.21.15.12:6789/0 conn(0x7f4e90161590 legacy=0x7f4e900b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.681+0000 7f4e8fe00700 1 -- 172.21.15.12:0/3317367018 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e900ce750 con 0x7f4e90160f10 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.681+0000 7f4e8fe00700 1 --2- 172.21.15.12:0/3317367018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e90160f10 0x7f4e900d18b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4e80009840 tx=0x7f4e80009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4403c353964322c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.681+0000 7f4e8ea00700 1 -- 172.21.15.12:0/3317367018 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e80011830 con 0x7f4e90160f10 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.681+0000 7f4e8ea00700 1 -- 172.21.15.12:0/3317367018 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e80011990 con 0x7f4e90160f10 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.681+0000 7f4e8ea00700 1 -- 172.21.15.12:0/3317367018 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e80011d00 con 0x7f4e90160f10 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.682+0000 7f4e97200700 1 -- 172.21.15.12:0/3317367018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e90160f10 msgr2=0x7f4e900d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.682+0000 7f4e97200700 1 --2- 172.21.15.12:0/3317367018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e90160f10 0x7f4e900d18b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4e80009840 tx=0x7f4e80009b50 comp rx=0 tx=0).stop 2024-03-29T09:54:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.682+0000 7f4e97200700 1 -- 172.21.15.12:0/3317367018 shutdown_connections 2024-03-29T09:54:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.682+0000 7f4e97200700 1 --2- 172.21.15.12:0/3317367018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e90160f10 0x7f4e900d18b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.682+0000 7f4e97200700 1 -- 172.21.15.12:0/3317367018 >> 172.21.15.12:0/3317367018 conn(0x7f4e9008aff0 msgr2=0x7f4e900d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.682+0000 7f4e97200700 1 -- 172.21.15.12:0/3317367018 shutdown_connections 2024-03-29T09:54:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.682+0000 7f4e97200700 1 -- 172.21.15.12:0/3317367018 wait complete. 2024-03-29T09:54:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.684+0000 7f4e97200700 1 Processor -- start 2024-03-29T09:54:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.684+0000 7f4e97200700 1 -- start start 2024-03-29T09:54:27.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e97200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e900cf230 0x7f4e90157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e97200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e90160f10 0x7f4e90150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e97200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e90161590 0x7f4e90151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e97200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900b6500 con 0x7f4e90161590 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e97200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e90152d30 con 0x7f4e90160f10 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e97200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900b84f0 con 0x7f4e900cf230 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e95200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e90161590 0x7f4e90151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e95200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e90161590 0x7f4e90151070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58190/0 (socket says 172.21.15.12:58190) 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e95200700 1 -- 172.21.15.12:0/295987783 learned_addr learned my addr 172.21.15.12:0/295987783 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e8f400700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e90160f10 0x7f4e90150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e8fe00700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e900cf230 0x7f4e90157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e95200700 1 -- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e900cf230 msgr2=0x7f4e90157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e95200700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e900cf230 0x7f4e90157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e95200700 1 -- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e90160f10 msgr2=0x7f4e90150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e95200700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e90160f10 0x7f4e90150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.685+0000 7f4e95200700 1 -- 172.21.15.12:0/295987783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e800094a0 con 0x7f4e90161590 2024-03-29T09:54:27.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.686+0000 7f4e95200700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e90161590 0x7f4e90151070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800caa0 tx=0x7f4e8800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:27.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.686+0000 7f4e87e00700 1 -- 172.21.15.12:0/295987783 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8800a9e0 con 0x7f4e90161590 2024-03-29T09:54:27.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.686+0000 7f4e87e00700 1 -- 172.21.15.12:0/295987783 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8800ab40 con 0x7f4e90161590 2024-03-29T09:54:27.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.686+0000 7f4e87e00700 1 -- 172.21.15.12:0/295987783 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88012400 con 0x7f4e90161590 2024-03-29T09:54:27.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.686+0000 7f4e97200700 1 -- 172.21.15.12:0/295987783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e900b87d0 con 0x7f4e90161590 2024-03-29T09:54:27.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.686+0000 7f4e97200700 1 -- 172.21.15.12:0/295987783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e900b6120 con 0x7f4e90161590 2024-03-29T09:54:27.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.687+0000 7f4e97200700 1 -- 172.21.15.12:0/295987783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e900b6640 con 0x7f4e90161590 2024-03-29T09:54:27.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.688+0000 7f4e87e00700 1 -- 172.21.15.12:0/295987783 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4e8801b3b0 con 0x7f4e90161590 2024-03-29T09:54:27.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.689+0000 7f4e87e00700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e7006ac90 0x7f4e7006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:27.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.689+0000 7f4e87e00700 1 -- 172.21.15.12:0/295987783 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e88088f20 con 0x7f4e90161590 2024-03-29T09:54:27.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.690+0000 7f4e8fe00700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e7006ac90 0x7f4e7006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.690+0000 7f4e8fe00700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e7006ac90 0x7f4e7006d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e80000c00 tx=0x7f4e80008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.694+0000 7f4e87e00700 1 -- 172.21.15.12:0/295987783 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e880541b0 con 0x7f4e90161590 2024-03-29T09:54:27.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.910+0000 7f4e97200700 1 -- 172.21.15.12:0/295987783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e90067720 con 0x7f4e90161590 2024-03-29T09:54:27.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.911+0000 7f4e87e00700 1 -- 172.21.15.12:0/295987783 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16050 (secure 0 0 0) 0x7f4e880577d0 con 0x7f4e90161590 2024-03-29T09:54:27.912 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:27.912 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":19,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:27.425184+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 -- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e7006ac90 msgr2=0x7f4e7006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e7006ac90 0x7f4e7006d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e80000c00 tx=0x7f4e80008030 comp rx=0 tx=0).stop 2024-03-29T09:54:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 -- 172.21.15.12:0/295987783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e90161590 msgr2=0x7f4e90151070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e90161590 0x7f4e90151070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800caa0 tx=0x7f4e8800cdb0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 -- 172.21.15.12:0/295987783 shutdown_connections 2024-03-29T09:54:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e7006ac90 0x7f4e7006d150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e90161590 0x7f4e90151070 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e900cf230 0x7f4e90157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 --2- 172.21.15.12:0/295987783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e90160f10 0x7f4e90150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 -- 172.21.15.12:0/295987783 >> 172.21.15.12:0/295987783 conn(0x7f4e9008aff0 msgr2=0x7f4e900caeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 -- 172.21.15.12:0/295987783 shutdown_connections 2024-03-29T09:54:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:27.914+0000 7f4e97200700 1 -- 172.21.15.12:0/295987783 wait complete. 2024-03-29T09:54:27.932 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:28.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.128+0000 7fb907800700 1 Processor -- start 2024-03-29T09:54:28.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.128+0000 7fb907800700 1 -- start start 2024-03-29T09:54:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.129+0000 7fb907800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9000cb3f0 0x7fb9000ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.129+0000 7fb907800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb9000ac600 0x7fb90006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.129+0000 7fb907800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90015fd50 con 0x7fb90015b5d0 2024-03-29T09:54:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.129+0000 7fb907800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90015b9b0 con 0x7fb9000ac600 2024-03-29T09:54:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.129+0000 7fb907800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000aca70 con 0x7fb9000cb3f0 2024-03-29T09:54:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb905800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb9000ac600 0x7fb90006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb905800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb9000ac600 0x7fb90006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58196/0 (socket says 172.21.15.12:58196) 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb905800700 1 -- 172.21.15.12:0/174458687 learned_addr learned my addr 172.21.15.12:0/174458687 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb904e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb90015b5d0 0x7fb9000ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44082/0 (socket says 172.21.15.12:44082) 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb8ffe00700 1 --2- 172.21.15.12:0/174458687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9000cb3f0 0x7fb9000ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb905800700 1 -- 172.21.15.12:0/174458687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9000cb3f0 msgr2=0x7fb9000ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb905800700 1 --2- 172.21.15.12:0/174458687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9000cb3f0 0x7fb9000ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb905800700 1 -- 172.21.15.12:0/174458687 >> v1:172.21.15.12:6789/0 conn(0x7fb90015b5d0 legacy=0x7fb9000ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb905800700 1 -- 172.21.15.12:0/174458687 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90015f140 con 0x7fb9000ac600 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb8ff400700 1 -- 172.21.15.12:0/174458687 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866908123 0 0) 0x7fb90015fd50 con 0x7fb90015b5d0 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.130+0000 7fb905800700 1 --2- 172.21.15.12:0/174458687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9000ac600 0x7fb90006c9d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8009e50 tx=0x7fb8f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9539e900c4fb41e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.131+0000 7fb8ff400700 1 -- 172.21.15.12:0/174458687 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f800e070 con 0x7fb9000ac600 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.131+0000 7fb8ff400700 1 -- 172.21.15.12:0/174458687 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f800b9c0 con 0x7fb9000ac600 2024-03-29T09:54:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.131+0000 7fb8ff400700 1 -- 172.21.15.12:0/174458687 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f800bd70 con 0x7fb9000ac600 2024-03-29T09:54:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.131+0000 7fb907800700 1 -- 172.21.15.12:0/174458687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9000ac600 msgr2=0x7fb90006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.131+0000 7fb907800700 1 --2- 172.21.15.12:0/174458687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9000ac600 0x7fb90006c9d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8009e50 tx=0x7fb8f800b330 comp rx=0 tx=0).stop 2024-03-29T09:54:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.131+0000 7fb907800700 1 -- 172.21.15.12:0/174458687 shutdown_connections 2024-03-29T09:54:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.131+0000 7fb907800700 1 --2- 172.21.15.12:0/174458687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9000ac600 0x7fb90006c9d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.131+0000 7fb907800700 1 --2- 172.21.15.12:0/174458687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9000cb3f0 0x7fb9000ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.131+0000 7fb907800700 1 -- 172.21.15.12:0/174458687 >> 172.21.15.12:0/174458687 conn(0x7fb9000cf190 msgr2=0x7fb9000cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.131+0000 7fb907800700 1 -- 172.21.15.12:0/174458687 shutdown_connections 2024-03-29T09:54:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.132+0000 7fb907800700 1 -- 172.21.15.12:0/174458687 wait complete. 2024-03-29T09:54:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.133+0000 7fb907800700 1 Processor -- start 2024-03-29T09:54:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.133+0000 7fb907800700 1 -- start start 2024-03-29T09:54:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.134+0000 7fb907800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9000ac600 0x7fb90015e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.134+0000 7fb907800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb90015b5d0 0x7fb9000b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.134+0000 7fb907800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9000c9fb0 0x7fb9000b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.134+0000 7fb907800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9001535c0 con 0x7fb90015b5d0 2024-03-29T09:54:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.134+0000 7fb907800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000c9b20 con 0x7fb9000c9fb0 2024-03-29T09:54:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.134+0000 7fb907800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000c8f10 con 0x7fb9000ac600 2024-03-29T09:54:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb8ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb90015b5d0 0x7fb9000b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb8ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb90015b5d0 0x7fb9000b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58202/0 (socket says 172.21.15.12:58202) 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb8ffe00700 1 -- 172.21.15.12:0/1240146610 learned_addr learned my addr 172.21.15.12:0/1240146610 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb904e00700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9000ac600 0x7fb90015e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb905800700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9000c9fb0 0x7fb9000b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb8ffe00700 1 -- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9000ac600 msgr2=0x7fb90015e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb8ffe00700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9000ac600 0x7fb90015e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb8ffe00700 1 -- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9000c9fb0 msgr2=0x7fb9000b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb8ffe00700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9000c9fb0 0x7fb9000b2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb8ffe00700 1 -- 172.21.15.12:0/1240146610 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f8009b30 con 0x7fb90015b5d0 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.135+0000 7fb8ffe00700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb90015b5d0 0x7fb9000b1ed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00e810 tx=0x7fb8ec00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.136+0000 7fb8fcc00700 1 -- 172.21.15.12:0/1240146610 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec00c070 con 0x7fb90015b5d0 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.136+0000 7fb8fcc00700 1 -- 172.21.15.12:0/1240146610 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec013610 con 0x7fb90015b5d0 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.136+0000 7fb8fcc00700 1 -- 172.21.15.12:0/1240146610 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec0139c0 con 0x7fb90015b5d0 2024-03-29T09:54:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.136+0000 7fb907800700 1 -- 172.21.15.12:0/1240146610 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9000c91f0 con 0x7fb90015b5d0 2024-03-29T09:54:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.136+0000 7fb907800700 1 -- 172.21.15.12:0/1240146610 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9000c97b0 con 0x7fb90015b5d0 2024-03-29T09:54:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.137+0000 7fb907800700 1 -- 172.21.15.12:0/1240146610 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb900066ff0 con 0x7fb90015b5d0 2024-03-29T09:54:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.138+0000 7fb8fcc00700 1 -- 172.21.15.12:0/1240146610 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb8ec013b20 con 0x7fb90015b5d0 2024-03-29T09:54:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.138+0000 7fb8fcc00700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb8e006ac90 0x7fb8e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.138+0000 7fb8fcc00700 1 -- 172.21.15.12:0/1240146610 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8ec0890d0 con 0x7fb90015b5d0 2024-03-29T09:54:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.139+0000 7fb904e00700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb8e006ac90 0x7fb8e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.140+0000 7fb904e00700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb8e006ac90 0x7fb8e006d150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4000f60 tx=0x7fb8f4007e70 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.143+0000 7fb8fcc00700 1 -- 172.21.15.12:0/1240146610 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8ec054220 con 0x7fb90015b5d0 2024-03-29T09:54:28.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.355+0000 7fb907800700 1 -- 172.21.15.12:0/1240146610 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb900067720 con 0x7fb90015b5d0 2024-03-29T09:54:28.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.356+0000 7fb8fcc00700 1 -- 172.21.15.12:0/1240146610 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16050 (secure 0 0 0) 0x7fb8ec057840 con 0x7fb90015b5d0 2024-03-29T09:54:28.357 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:28.357 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":19,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:27.425184+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:28.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.358+0000 7fb907800700 1 -- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb8e006ac90 msgr2=0x7fb8e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.358+0000 7fb907800700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb8e006ac90 0x7fb8e006d150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4000f60 tx=0x7fb8f4007e70 comp rx=0 tx=0).stop 2024-03-29T09:54:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.358+0000 7fb907800700 1 -- 172.21.15.12:0/1240146610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb90015b5d0 msgr2=0x7fb9000b1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.358+0000 7fb907800700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb90015b5d0 0x7fb9000b1ed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00e810 tx=0x7fb8ec00eb20 comp rx=0 tx=0).stop 2024-03-29T09:54:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.359+0000 7fb907800700 1 -- 172.21.15.12:0/1240146610 shutdown_connections 2024-03-29T09:54:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.359+0000 7fb907800700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb8e006ac90 0x7fb8e006d150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.359+0000 7fb907800700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9000c9fb0 0x7fb9000b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.359+0000 7fb907800700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9000ac600 0x7fb90015e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.359+0000 7fb907800700 1 --2- 172.21.15.12:0/1240146610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb90015b5d0 0x7fb9000b1ed0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.359+0000 7fb907800700 1 -- 172.21.15.12:0/1240146610 >> 172.21.15.12:0/1240146610 conn(0x7fb9000cf190 msgr2=0x7fb90015cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.359+0000 7fb907800700 1 -- 172.21.15.12:0/1240146610 shutdown_connections 2024-03-29T09:54:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.359+0000 7fb907800700 1 -- 172.21.15.12:0/1240146610 wait complete. 2024-03-29T09:54:28.377 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T09:54:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.579+0000 7f2c9b000700 1 Processor -- start 2024-03-29T09:54:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.580+0000 7f2c9b000700 1 -- start start 2024-03-29T09:54:28.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.581+0000 7f2c9b000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2c940d16a0 0x7f2c940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.581+0000 7f2c9b000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c94161420 con 0x7f2c940d2370 2024-03-29T09:54:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.581+0000 7f2c9b000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c94161700 con 0x7f2c9415d9d0 2024-03-29T09:54:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.581+0000 7f2c9b000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c940d4340 con 0x7f2c940d16a0 2024-03-29T09:54:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.581+0000 7f2c99000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2c940d2370 0x7f2c940ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44096/0 (socket says 172.21.15.12:44096) 2024-03-29T09:54:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.581+0000 7f2c99000700 1 -- 172.21.15.12:0/1747946728 learned_addr learned my addr 172.21.15.12:0/1747946728 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.581+0000 7f2c93400700 1 --2- 172.21.15.12:0/1747946728 >> v2:172.21.15.159:3300/0 conn(0x7f2c940d16a0 0x7f2c940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841389707 0 0) 0x7f2c94161420 con 0x7f2c940d2370 2024-03-29T09:54:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c780035b0 con 0x7f2c940d2370 2024-03-29T09:54:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036626158 0 0) 0x7f2c94161700 con 0x7f2c9415d9d0 2024-03-29T09:54:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c94161420 con 0x7f2c9415d9d0 2024-03-29T09:54:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4096547508 0 0) 0x7f2c780035b0 con 0x7f2c940d2370 2024-03-29T09:54:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 >> v2:172.21.15.159:3300/0 conn(0x7f2c940d16a0 msgr2=0x7f2c940d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 --2- 172.21.15.12:0/1747946728 >> v2:172.21.15.159:3300/0 conn(0x7f2c940d16a0 0x7f2c940d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 >> v1:172.21.15.159:6789/0 conn(0x7f2c9415d9d0 legacy=0x7f2c940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c941619e0 con 0x7f2c940d2370 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f2c8c0019f0 con 0x7f2c940d2370 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 >> v1:172.21.15.12:6789/0 conn(0x7f2c940d2370 legacy=0x7f2c940ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c78003b90 0x7f2c78003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c78004850 0x7f2c78004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c7800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7800c060 con 0x7f2c78005210 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7800c1d0 con 0x7f2c78004850 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7800c390 con 0x7f2c78003b90 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c93400700 1 --2- 172.21.15.12:0/1747946728 >> v2:172.21.15.159:3300/0 conn(0x7f2c940d16a0 0x7f2c940d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.582+0000 7f2c99000700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c7800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c93e00700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c78003b90 0x7f2c78003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c93400700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c78004850 0x7f2c78004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c99000700 1 -- 172.21.15.12:0/1747946728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c78003b90 msgr2=0x7f2c78003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c99000700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c78003b90 0x7f2c78003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c99000700 1 -- 172.21.15.12:0/1747946728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c78004850 msgr2=0x7f2c78004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c99000700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c78004850 0x7f2c78004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c99000700 1 -- 172.21.15.12:0/1747946728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c780035b0 con 0x7f2c78005210 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c99000700 1 -- 172.21.15.12:0/1747946728 reap_dead start 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c99000700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c7800ba10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00a7d0 tx=0x7f2c78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c97d8adf7c81877 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c012600 con 0x7f2c78005210 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c0127a0 con 0x7f2c78005210 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.583+0000 7f2c92a00700 1 -- 172.21.15.12:0/1747946728 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c012b50 con 0x7f2c78005210 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.584+0000 7f2c9b000700 1 -- 172.21.15.12:0/1747946728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 msgr2=0x7f2c7800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.584+0000 7f2c9b000700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c7800ba10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00a7d0 tx=0x7f2c78005880 comp rx=0 tx=0).stop 2024-03-29T09:54:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.584+0000 7f2c9b000700 1 -- 172.21.15.12:0/1747946728 shutdown_connections 2024-03-29T09:54:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.584+0000 7f2c9b000700 1 --2- 172.21.15.12:0/1747946728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c7800ba10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.584+0000 7f2c9b000700 1 -- 172.21.15.12:0/1747946728 >> 172.21.15.12:0/1747946728 conn(0x7f2c9408aff0 msgr2=0x7f2c940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.584+0000 7f2c9b000700 1 -- 172.21.15.12:0/1747946728 shutdown_connections 2024-03-29T09:54:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.584+0000 7f2c9b000700 1 -- 172.21.15.12:0/1747946728 wait complete. 2024-03-29T09:54:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.586+0000 7f2c9b000700 1 Processor -- start 2024-03-29T09:54:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.586+0000 7f2c9b000700 1 -- start start 2024-03-29T09:54:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c9b000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c94158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c9b000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c941600f0 0x7f2c940b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c9b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c940b43b0 0x7f2c940b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c9b000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c940b4d70 con 0x7f2c78005210 2024-03-29T09:54:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c9b000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c940b7e90 con 0x7f2c941600f0 2024-03-29T09:54:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c9b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c940b7310 con 0x7f2c940b43b0 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c93e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c94158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c93e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c94158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58222/0 (socket says 172.21.15.12:58222) 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c93e00700 1 -- 172.21.15.12:0/2697484866 learned_addr learned my addr 172.21.15.12:0/2697484866 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c93400700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c941600f0 0x7f2c940b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.587+0000 7f2c99000700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c940b43b0 0x7f2c940b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c93e00700 1 -- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c940b43b0 msgr2=0x7f2c940b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c93e00700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c940b43b0 0x7f2c940b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c93e00700 1 -- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c941600f0 msgr2=0x7f2c940b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c93e00700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c941600f0 0x7f2c940b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c93e00700 1 -- 172.21.15.12:0/2697484866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c940b75f0 con 0x7f2c78005210 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c99000700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c940b43b0 0x7f2c940b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c93e00700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c94158980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400e7b0 tx=0x7f2c8400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c8be00700 1 -- 172.21.15.12:0/2697484866 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c840052e0 con 0x7f2c78005210 2024-03-29T09:54:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c8be00700 1 -- 172.21.15.12:0/2697484866 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c84005440 con 0x7f2c78005210 2024-03-29T09:54:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c8be00700 1 -- 172.21.15.12:0/2697484866 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c840038c0 con 0x7f2c78005210 2024-03-29T09:54:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c9b000700 1 -- 172.21.15.12:0/2697484866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c940b78d0 con 0x7f2c78005210 2024-03-29T09:54:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.588+0000 7f2c9b000700 1 -- 172.21.15.12:0/2697484866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c94158ec0 con 0x7f2c78005210 2024-03-29T09:54:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.589+0000 7f2c9b000700 1 -- 172.21.15.12:0/2697484866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c94067720 con 0x7f2c78005210 2024-03-29T09:54:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.590+0000 7f2c8be00700 1 -- 172.21.15.12:0/2697484866 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2c84003a20 con 0x7f2c78005210 2024-03-29T09:54:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.591+0000 7f2c8be00700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c7406acc0 0x7f2c7406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.591+0000 7f2c8be00700 1 -- 172.21.15.12:0/2697484866 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c84088c70 con 0x7f2c78005210 2024-03-29T09:54:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.591+0000 7f2c93400700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c7406acc0 0x7f2c7406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.592+0000 7f2c93400700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c7406acc0 0x7f2c7406d180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c940a1420 tx=0x7f2c80008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:28.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.596+0000 7f2c8be00700 1 -- 172.21.15.12:0/2697484866 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2c84053dc0 con 0x7f2c78005210 2024-03-29T09:54:28.634 INFO:tasks.ceph.mds.c.smithi012.stderr:2024-03-29T09:54:28.633+0000 7fb31ea00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T09:54:28.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:28.859+0000 7f2c9b000700 1 -- 172.21.15.12:0/2697484866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2c94161010 con 0x7f2c78005210 2024-03-29T09:54:29.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.435+0000 7f2c8be00700 1 -- 172.21.15.12:0/2697484866 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2c840190b0 con 0x7f2c78005210 2024-03-29T09:54:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 -- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c7406acc0 msgr2=0x7f2c7406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c7406acc0 0x7f2c7406d180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c940a1420 tx=0x7f2c80008040 comp rx=0 tx=0).stop 2024-03-29T09:54:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 -- 172.21.15.12:0/2697484866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 msgr2=0x7f2c94158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c94158980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400e7b0 tx=0x7f2c8400eac0 comp rx=0 tx=0).stop 2024-03-29T09:54:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 -- 172.21.15.12:0/2697484866 shutdown_connections 2024-03-29T09:54:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c7406acc0 0x7f2c7406d180 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c940b43b0 0x7f2c940b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c78005210 0x7f2c94158980 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 --2- 172.21.15.12:0/2697484866 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c941600f0 0x7f2c940b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 -- 172.21.15.12:0/2697484866 >> 172.21.15.12:0/2697484866 conn(0x7f2c9408aff0 msgr2=0x7f2c940ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 -- 172.21.15.12:0/2697484866 shutdown_connections 2024-03-29T09:54:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.438+0000 7f2c9b000700 1 -- 172.21.15.12:0/2697484866 wait complete. 2024-03-29T09:54:29.460 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T09:54:29.460 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T09:54:29.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.664+0000 7fe68bc00700 1 Processor -- start 2024-03-29T09:54:29.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.664+0000 7fe68bc00700 1 -- start start 2024-03-29T09:54:29.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.665+0000 7fe68bc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe68415d9d0 0x7fe6840d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:29.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.665+0000 7fe68bc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6840d2540 0x7fe6840ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:29.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.665+0000 7fe68bc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe684161420 con 0x7fe6840d16a0 2024-03-29T09:54:29.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.665+0000 7fe68bc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe68415edc0 con 0x7fe68415d9d0 2024-03-29T09:54:29.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.665+0000 7fe68bc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe684161700 con 0x7fe6840d2540 2024-03-29T09:54:29.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.665+0000 7fe689200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe68415d9d0 0x7fe6840d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:29.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.665+0000 7fe683e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe6840d16a0 0x7fe6840d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58080/0 (socket says 172.21.15.12:58080) 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.665+0000 7fe683e00700 1 -- 172.21.15.12:0/1148813937 learned_addr learned my addr 172.21.15.12:0/1148813937 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.666+0000 7fe689c00700 1 --2- 172.21.15.12:0/1148813937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6840d2540 0x7fe6840ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.666+0000 7fe689200700 1 -- 172.21.15.12:0/1148813937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6840d2540 msgr2=0x7fe6840ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.666+0000 7fe689200700 1 --2- 172.21.15.12:0/1148813937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6840d2540 0x7fe6840ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.666+0000 7fe689200700 1 -- 172.21.15.12:0/1148813937 >> v1:172.21.15.12:6789/0 conn(0x7fe6840d16a0 legacy=0x7fe6840d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.666+0000 7fe689200700 1 -- 172.21.15.12:0/1148813937 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6841619e0 con 0x7fe68415d9d0 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.666+0000 7fe683400700 1 -- 172.21.15.12:0/1148813937 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454473624 0 0) 0x7fe684161420 con 0x7fe6840d16a0 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.666+0000 7fe689200700 1 --2- 172.21.15.12:0/1148813937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe68415d9d0 0x7fe6840d3c30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe678009840 tx=0x7fe678009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4dca593309aea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.666+0000 7fe683400700 1 -- 172.21.15.12:0/1148813937 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678011830 con 0x7fe68415d9d0 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.666+0000 7fe683400700 1 -- 172.21.15.12:0/1148813937 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe678011990 con 0x7fe68415d9d0 2024-03-29T09:54:29.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.666+0000 7fe683400700 1 -- 172.21.15.12:0/1148813937 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678011d20 con 0x7fe68415d9d0 2024-03-29T09:54:29.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.667+0000 7fe68bc00700 1 -- 172.21.15.12:0/1148813937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe68415d9d0 msgr2=0x7fe6840d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:29.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.667+0000 7fe68bc00700 1 --2- 172.21.15.12:0/1148813937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe68415d9d0 0x7fe6840d3c30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe678009840 tx=0x7fe678009b50 comp rx=0 tx=0).stop 2024-03-29T09:54:29.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.667+0000 7fe68bc00700 1 -- 172.21.15.12:0/1148813937 shutdown_connections 2024-03-29T09:54:29.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.667+0000 7fe68bc00700 1 --2- 172.21.15.12:0/1148813937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6840d2540 0x7fe6840ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:29.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.667+0000 7fe68bc00700 1 --2- 172.21.15.12:0/1148813937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe68415d9d0 0x7fe6840d3c30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:29.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.667+0000 7fe68bc00700 1 -- 172.21.15.12:0/1148813937 >> 172.21.15.12:0/1148813937 conn(0x7fe68408aff0 msgr2=0x7fe6840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:29.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.667+0000 7fe68bc00700 1 -- 172.21.15.12:0/1148813937 shutdown_connections 2024-03-29T09:54:29.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.667+0000 7fe68bc00700 1 -- 172.21.15.12:0/1148813937 wait complete. 2024-03-29T09:54:29.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.669+0000 7fe68bc00700 1 Processor -- start 2024-03-29T09:54:29.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.669+0000 7fe68bc00700 1 -- start start 2024-03-29T09:54:29.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.670+0000 7fe68bc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6840d16a0 0x7fe6841529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:29.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.670+0000 7fe68bc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6840d2540 0x7fe6840cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:29.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.670+0000 7fe68bc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe68415d9d0 0x7fe6840b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:29.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.670+0000 7fe68bc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840cb840 con 0x7fe6840d16a0 2024-03-29T09:54:29.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.670+0000 7fe68bc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840cc1a0 con 0x7fe6840d2540 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.670+0000 7fe68bc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840ccc00 con 0x7fe68415d9d0 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.670+0000 7fe689200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6840d16a0 0x7fe6841529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.670+0000 7fe689200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6840d16a0 0x7fe6841529b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40426/0 (socket says 172.21.15.12:40426) 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.670+0000 7fe689200700 1 -- 172.21.15.12:0/972759385 learned_addr learned my addr 172.21.15.12:0/972759385 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe689c00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe68415d9d0 0x7fe6840b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe683e00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6840d2540 0x7fe6840cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe689200700 1 -- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe68415d9d0 msgr2=0x7fe6840b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe689200700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe68415d9d0 0x7fe6840b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe689200700 1 -- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6840d2540 msgr2=0x7fe6840cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe689200700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6840d2540 0x7fe6840cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:29.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe689200700 1 -- 172.21.15.12:0/972759385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780094a0 con 0x7fe6840d16a0 2024-03-29T09:54:29.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe683e00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6840d2540 0x7fe6840cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:29.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe689200700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6840d16a0 0x7fe6841529b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe6780090b0 tx=0x7fe678004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:29.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe680c00700 1 -- 172.21.15.12:0/972759385 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6780038a0 con 0x7fe6840d16a0 2024-03-29T09:54:29.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe680c00700 1 -- 172.21.15.12:0/972759385 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67801e070 con 0x7fe6840d16a0 2024-03-29T09:54:29.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe680c00700 1 -- 172.21.15.12:0/972759385 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678003c00 con 0x7fe6840d16a0 2024-03-29T09:54:29.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe68bc00700 1 -- 172.21.15.12:0/972759385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6840cb180 con 0x7fe6840d16a0 2024-03-29T09:54:29.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.671+0000 7fe68bc00700 1 -- 172.21.15.12:0/972759385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6840cc790 con 0x7fe6840d16a0 2024-03-29T09:54:29.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.672+0000 7fe68bc00700 1 -- 172.21.15.12:0/972759385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6840d1ca0 con 0x7fe6840d16a0 2024-03-29T09:54:29.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.673+0000 7fe680c00700 1 -- 172.21.15.12:0/972759385 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe67801a420 con 0x7fe6840d16a0 2024-03-29T09:54:29.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.674+0000 7fe680c00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe66406ac90 0x7fe66406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:29.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.674+0000 7fe680c00700 1 -- 172.21.15.12:0/972759385 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe67808a490 con 0x7fe6840d16a0 2024-03-29T09:54:29.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.674+0000 7fe683e00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe66406ac90 0x7fe66406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:29.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.675+0000 7fe683e00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe66406ac90 0x7fe66406d150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe684072350 tx=0x7fe670009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:29.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.678+0000 7fe680c00700 1 -- 172.21.15.12:0/972759385 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe678055660 con 0x7fe6840d16a0 2024-03-29T09:54:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:29.939+0000 7fe68bc00700 1 -- 172.21.15.12:0/972759385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe6840cbab0 con 0x7fe6840d16a0 2024-03-29T09:54:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.900+0000 7fe680c00700 1 -- 172.21.15.12:0/972759385 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe678058c80 con 0x7fe6840d16a0 2024-03-29T09:54:30.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.903+0000 7fe68bc00700 1 -- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe66406ac90 msgr2=0x7fe66406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:30.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.903+0000 7fe68bc00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe66406ac90 0x7fe66406d150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe684072350 tx=0x7fe670009040 comp rx=0 tx=0).stop 2024-03-29T09:54:30.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.903+0000 7fe68bc00700 1 -- 172.21.15.12:0/972759385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6840d16a0 msgr2=0x7fe6841529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:30.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.903+0000 7fe68bc00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6840d16a0 0x7fe6841529b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe6780090b0 tx=0x7fe678004750 comp rx=0 tx=0).stop 2024-03-29T09:54:30.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.903+0000 7fe68bc00700 1 -- 172.21.15.12:0/972759385 shutdown_connections 2024-03-29T09:54:30.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.903+0000 7fe68bc00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe66406ac90 0x7fe66406d150 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:30.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.903+0000 7fe68bc00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe68415d9d0 0x7fe6840b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:30.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.903+0000 7fe68bc00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6840d16a0 0x7fe6841529b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:30.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.903+0000 7fe68bc00700 1 --2- 172.21.15.12:0/972759385 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6840d2540 0x7fe6840cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:30.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.904+0000 7fe68bc00700 1 -- 172.21.15.12:0/972759385 >> 172.21.15.12:0/972759385 conn(0x7fe68408aff0 msgr2=0x7fe6840b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:30.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.904+0000 7fe68bc00700 1 -- 172.21.15.12:0/972759385 shutdown_connections 2024-03-29T09:54:30.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:30.904+0000 7fe68bc00700 1 -- 172.21.15.12:0/972759385 wait complete. 2024-03-29T09:54:30.925 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T09:54:30.925 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:54:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.127+0000 7fe302400700 1 Processor -- start 2024-03-29T09:54:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.127+0000 7fe302400700 1 -- start start 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe302400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2fc0d18b0 0x7fe2fc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe302400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe2fc161590 0x7fe2fc161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe302400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc0ceda0 con 0x7fe2fc160f10 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe302400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2fc0d1d30 con 0x7fe2fc0d18b0 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe302400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2fc161f60 con 0x7fe2fc161590 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2faa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2fc0d18b0 0x7fe2fc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2faa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2fc0d18b0 0x7fe2fc0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40442/0 (socket says 172.21.15.12:40442) 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2faa00700 1 -- 172.21.15.12:0/3433294606 learned_addr learned my addr 172.21.15.12:0/3433294606 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2fbe00700 1 --2- 172.21.15.12:0/3433294606 >> v2:172.21.15.159:3300/0 conn(0x7fe2fc161590 0x7fe2fc161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2faa00700 1 -- 172.21.15.12:0/3433294606 >> v2:172.21.15.159:3300/0 conn(0x7fe2fc161590 msgr2=0x7fe2fc161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2faa00700 1 --2- 172.21.15.12:0/3433294606 >> v2:172.21.15.159:3300/0 conn(0x7fe2fc161590 0x7fe2fc161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2faa00700 1 -- 172.21.15.12:0/3433294606 >> v1:172.21.15.159:6789/0 conn(0x7fe2fc160f10 legacy=0x7fe2fc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2faa00700 1 -- 172.21.15.12:0/3433294606 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc1612f0 con 0x7fe2fc0d18b0 2024-03-29T09:54:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2faa00700 1 --2- 172.21.15.12:0/3433294606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc0d18b0 0x7fe2fc0cf230 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0d1030 tx=0x7fe2e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba79e7a748ca7963 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2fa000700 1 -- 172.21.15.12:0/3433294606 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8004230 con 0x7fe2fc0d18b0 2024-03-29T09:54:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2fa000700 1 -- 172.21.15.12:0/3433294606 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e8004390 con 0x7fe2fc0d18b0 2024-03-29T09:54:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.128+0000 7fe2fa000700 1 -- 172.21.15.12:0/3433294606 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e80036a0 con 0x7fe2fc0d18b0 2024-03-29T09:54:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.129+0000 7fe302400700 1 -- 172.21.15.12:0/3433294606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc0d18b0 msgr2=0x7fe2fc0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.129+0000 7fe302400700 1 --2- 172.21.15.12:0/3433294606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc0d18b0 0x7fe2fc0cf230 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0d1030 tx=0x7fe2e8009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.129+0000 7fe302400700 1 -- 172.21.15.12:0/3433294606 shutdown_connections 2024-03-29T09:54:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.129+0000 7fe302400700 1 --2- 172.21.15.12:0/3433294606 >> v2:172.21.15.159:3300/0 conn(0x7fe2fc161590 0x7fe2fc161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.129+0000 7fe302400700 1 --2- 172.21.15.12:0/3433294606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc0d18b0 0x7fe2fc0cf230 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.129+0000 7fe302400700 1 -- 172.21.15.12:0/3433294606 >> 172.21.15.12:0/3433294606 conn(0x7fe2fc08aff0 msgr2=0x7fe2fc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.129+0000 7fe302400700 1 -- 172.21.15.12:0/3433294606 shutdown_connections 2024-03-29T09:54:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.129+0000 7fe302400700 1 -- 172.21.15.12:0/3433294606 wait complete. 2024-03-29T09:54:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.131+0000 7fe302400700 1 Processor -- start 2024-03-29T09:54:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe302400700 1 -- start start 2024-03-29T09:54:31.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe302400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2fc0d18b0 0x7fe2fc1f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe302400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc160f10 0x7fe2fc1f6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe302400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc161590 0x7fe2fc09d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe302400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc09f600 con 0x7fe2fc160f10 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe302400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc1e8740 con 0x7fe2fc0d18b0 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe302400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc1f4c70 con 0x7fe2fc161590 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe2faa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc160f10 0x7fe2fc1f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe2faa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc160f10 0x7fe2fc1f6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40454/0 (socket says 172.21.15.12:40454) 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe2faa00700 1 -- 172.21.15.12:0/4106540823 learned_addr learned my addr 172.21.15.12:0/4106540823 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe2fb400700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2fc0d18b0 0x7fe2fc1f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe2fbe00700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc161590 0x7fe2fc09d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe2faa00700 1 -- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc161590 msgr2=0x7fe2fc09d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.133+0000 7fe2faa00700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc161590 0x7fe2fc09d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.134+0000 7fe2faa00700 1 -- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2fc0d18b0 msgr2=0x7fe2fc1f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.134+0000 7fe2faa00700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2fc0d18b0 0x7fe2fc1f4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.134+0000 7fe2faa00700 1 -- 172.21.15.12:0/4106540823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e8009550 con 0x7fe2fc160f10 2024-03-29T09:54:31.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.134+0000 7fe2faa00700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc160f10 0x7fe2fc1f6380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009130 tx=0x7fe2e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:31.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.134+0000 7fe2f3400700 1 -- 172.21.15.12:0/4106540823 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8003e20 con 0x7fe2fc160f10 2024-03-29T09:54:31.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.134+0000 7fe2f3400700 1 -- 172.21.15.12:0/4106540823 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e8011630 con 0x7fe2fc160f10 2024-03-29T09:54:31.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.134+0000 7fe2f3400700 1 -- 172.21.15.12:0/4106540823 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e80119a0 con 0x7fe2fc160f10 2024-03-29T09:54:31.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.134+0000 7fe302400700 1 -- 172.21.15.12:0/4106540823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc1f68c0 con 0x7fe2fc160f10 2024-03-29T09:54:31.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.134+0000 7fe302400700 1 -- 172.21.15.12:0/4106540823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc09fb10 con 0x7fe2fc160f10 2024-03-29T09:54:31.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.135+0000 7fe302400700 1 -- 172.21.15.12:0/4106540823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2fc067720 con 0x7fe2fc160f10 2024-03-29T09:54:31.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.136+0000 7fe2f3400700 1 -- 172.21.15.12:0/4106540823 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe2e8011b00 con 0x7fe2fc160f10 2024-03-29T09:54:31.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.137+0000 7fe2f3400700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ace0 0x7fe2dc06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.137+0000 7fe2f3400700 1 -- 172.21.15.12:0/4106540823 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2e808a940 con 0x7fe2fc160f10 2024-03-29T09:54:31.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.137+0000 7fe2fb400700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ace0 0x7fe2dc06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.138+0000 7fe2fb400700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ace0 0x7fe2dc06d1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009960 tx=0x7fe2ec0090e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:31.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.141+0000 7fe2f3400700 1 -- 172.21.15.12:0/4106540823 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2e8055b10 con 0x7fe2fc160f10 2024-03-29T09:54:31.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.399+0000 7fe302400700 1 -- 172.21.15.12:0/4106540823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2fc0ce1b0 con 0x7fe2fc160f10 2024-03-29T09:54:31.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.399+0000 7fe2f3400700 1 -- 172.21.15.12:0/4106540823 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7fe2e8011db0 con 0x7fe2fc160f10 2024-03-29T09:54:31.400 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:31.400 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/4093246304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4093246304},{"type":"v1","addr":"172.21.15.12:6833","nonce":4093246304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1631416176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1631416176},{"type":"v1","addr":"172.21.15.159:6836","nonce":1631416176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3828834681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3828834681},{"type":"v1","addr":"172.21.15.159:6837","nonce":3828834681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:54:27.425500+0000","modified":"2024-03-29T09:54:30.895316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3922451359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3922451359},{"type":"v1","addr":"172.21.15.12:6835","nonce":3922451359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-29T09:54:31.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 -- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ace0 msgr2=0x7fe2dc06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ace0 0x7fe2dc06d1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009960 tx=0x7fe2ec0090e0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 -- 172.21.15.12:0/4106540823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc160f10 msgr2=0x7fe2fc1f6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc160f10 0x7fe2fc1f6380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009130 tx=0x7fe2e8004750 comp rx=0 tx=0).stop 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 -- 172.21.15.12:0/4106540823 shutdown_connections 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ace0 0x7fe2dc06d1a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc161590 0x7fe2fc09d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2fc0d18b0 0x7fe2fc1f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 --2- 172.21.15.12:0/4106540823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc160f10 0x7fe2fc1f6380 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 -- 172.21.15.12:0/4106540823 >> 172.21.15.12:0/4106540823 conn(0x7fe2fc08aff0 msgr2=0x7fe2fc0caee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 -- 172.21.15.12:0/4106540823 shutdown_connections 2024-03-29T09:54:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.402+0000 7fe302400700 1 -- 172.21.15.12:0/4106540823 wait complete. 2024-03-29T09:54:31.404 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-03-29T09:54:31.420 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.616+0000 7fbcb5e00700 1 Processor -- start 2024-03-29T09:54:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.617+0000 7fbcb5e00700 1 -- start start 2024-03-29T09:54:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.618+0000 7fbcb5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00d18b0 0x7fbcb00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.618+0000 7fbcb5e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb00ceda0 con 0x7fbcb0160f10 2024-03-29T09:54:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.618+0000 7fbcb5e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb01612f0 con 0x7fbcb0161590 2024-03-29T09:54:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.618+0000 7fbcb5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb00d1d30 con 0x7fbcb00d18b0 2024-03-29T09:54:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.618+0000 7fbcaea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbcb0160f10 0x7fbcb00d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58090/0 (socket says 172.21.15.12:58090) 2024-03-29T09:54:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.618+0000 7fbcaea00700 1 -- 172.21.15.12:0/3367156957 learned_addr learned my addr 172.21.15.12:0/3367156957 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.618+0000 7fbcae000700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00d18b0 0x7fbcb00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.618+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779061197 0 0) 0x7fbcb00ceda0 con 0x7fbcb0160f10 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.618+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc940035b0 con 0x7fbcb0160f10 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 730683268 0 0) 0x7fbc940035b0 con 0x7fbcb0160f10 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00d18b0 msgr2=0x7fbcb00cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00d18b0 0x7fbcb00cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 >> v1:172.21.15.159:6789/0 conn(0x7fbcb0161590 legacy=0x7fbcb00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb00ce470 con 0x7fbcb0160f10 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fbca0001670 con 0x7fbcb0160f10 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 >> v1:172.21.15.12:6789/0 conn(0x7fbcb0160f10 legacy=0x7fbcb00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc94003a30 0x7fbc94003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc940043e0 0x7fbc940048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00d18b0 0x7fbc940051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb00d1d30 con 0x7fbc94003a30 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9400bac0 con 0x7fbc940043e0 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9400bc80 con 0x7fbcb00d18b0 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcaea00700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc94003a30 0x7fbc94003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcae000700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc940043e0 0x7fbc940048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.619+0000 7fbcaf400700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00d18b0 0x7fbc940051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcaea00700 1 -- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00d18b0 msgr2=0x7fbc940051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcaea00700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00d18b0 0x7fbc940051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcaea00700 1 -- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc940043e0 msgr2=0x7fbc940048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcaea00700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc940043e0 0x7fbc940048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcaea00700 1 -- 172.21.15.12:0/3367156957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc940035b0 con 0x7fbc94003a30 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcaf400700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00d18b0 0x7fbc940051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcaea00700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc94003a30 0x7fbc94003e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbca000c2f0 tx=0x7fbc94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b8104f658e6cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca000ce30 con 0x7fbc94003a30 2024-03-29T09:54:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca0005960 con 0x7fbc94003a30 2024-03-29T09:54:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcad600700 1 -- 172.21.15.12:0/3367156957 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca0005cf0 con 0x7fbc94003a30 2024-03-29T09:54:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcb5e00700 1 -- 172.21.15.12:0/3367156957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc94003a30 msgr2=0x7fbc94003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.620+0000 7fbcb5e00700 1 --2- 172.21.15.12:0/3367156957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc94003a30 0x7fbc94003e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbca000c2f0 tx=0x7fbc94005880 comp rx=0 tx=0).stop 2024-03-29T09:54:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.621+0000 7fbcaf400700 1 -- 172.21.15.12:0/3367156957 reap_dead start 2024-03-29T09:54:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.621+0000 7fbcb5e00700 1 -- 172.21.15.12:0/3367156957 shutdown_connections 2024-03-29T09:54:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.621+0000 7fbcb5e00700 1 -- 172.21.15.12:0/3367156957 >> 172.21.15.12:0/3367156957 conn(0x7fbcb008aff0 msgr2=0x7fbcb00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.621+0000 7fbcb5e00700 1 -- 172.21.15.12:0/3367156957 shutdown_connections 2024-03-29T09:54:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.621+0000 7fbcb5e00700 1 -- 172.21.15.12:0/3367156957 wait complete. 2024-03-29T09:54:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.622+0000 7fbcb5e00700 1 Processor -- start 2024-03-29T09:54:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcb5e00700 1 -- start start 2024-03-29T09:54:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcb5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc94003a30 0x7fbcb00c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcb5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcb015a2d0 0x7fbcb00b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcb5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00ca6e0 0x7fbcb00cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcb5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb00b8f20 con 0x7fbcb015a2d0 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcb5e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb00cb0b0 con 0x7fbc94003a30 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcb5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb00b9120 con 0x7fbcb00ca6e0 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcb015a2d0 0x7fbcb00b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcb015a2d0 0x7fbcb00b89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40484/0 (socket says 172.21.15.12:40484) 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcae000700 1 -- 172.21.15.12:0/2905140272 learned_addr learned my addr 172.21.15.12:0/2905140272 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcaea00700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc94003a30 0x7fbcb00c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcaf400700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00ca6e0 0x7fbcb00cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcae000700 1 -- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00ca6e0 msgr2=0x7fbcb00cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcae000700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00ca6e0 0x7fbcb00cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcae000700 1 -- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc94003a30 msgr2=0x7fbcb00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcae000700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc94003a30 0x7fbcb00c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcae000700 1 -- 172.21.15.12:0/2905140272 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb015d490 con 0x7fbcb015a2d0 2024-03-29T09:54:31.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.624+0000 7fbcae000700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcb015a2d0 0x7fbcb00b89e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c011aa0 tx=0x7fbc9c011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:31.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.625+0000 7fbca7400700 1 -- 172.21.15.12:0/2905140272 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c011ee0 con 0x7fbcb015a2d0 2024-03-29T09:54:31.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.625+0000 7fbca7400700 1 -- 172.21.15.12:0/2905140272 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9c00f940 con 0x7fbcb015a2d0 2024-03-29T09:54:31.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.625+0000 7fbca7400700 1 -- 172.21.15.12:0/2905140272 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c00fcf0 con 0x7fbcb015a2d0 2024-03-29T09:54:31.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.625+0000 7fbcb5e00700 1 -- 172.21.15.12:0/2905140272 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb015b1f0 con 0x7fbcb015a2d0 2024-03-29T09:54:31.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.625+0000 7fbcb5e00700 1 -- 172.21.15.12:0/2905140272 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb015d730 con 0x7fbcb015a2d0 2024-03-29T09:54:31.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.627+0000 7fbcb5e00700 1 -- 172.21.15.12:0/2905140272 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb0067720 con 0x7fbcb015a2d0 2024-03-29T09:54:31.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.627+0000 7fbca7400700 1 -- 172.21.15.12:0/2905140272 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbc9c014440 con 0x7fbcb015a2d0 2024-03-29T09:54:31.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.627+0000 7fbca7400700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc9006ac90 0x7fbc9006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:31.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.628+0000 7fbca7400700 1 -- 172.21.15.12:0/2905140272 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc9c0898d0 con 0x7fbcb015a2d0 2024-03-29T09:54:31.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.628+0000 7fbcaea00700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc9006ac90 0x7fbc9006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:31.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.630+0000 7fbcaea00700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc9006ac90 0x7fbc9006d150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbca0008fd0 tx=0x7fbca0007000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:31.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.633+0000 7fbca7400700 1 -- 172.21.15.12:0/2905140272 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc9c054b60 con 0x7fbcb015a2d0 2024-03-29T09:54:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.848+0000 7fbcb5e00700 1 -- 172.21.15.12:0/2905140272 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcb015d140 con 0x7fbcb015a2d0 2024-03-29T09:54:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.849+0000 7fbca7400700 1 -- 172.21.15.12:0/2905140272 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7fbc9c022020 con 0x7fbcb015a2d0 2024-03-29T09:54:31.849 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:31.849 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 -- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc9006ac90 msgr2=0x7fbc9006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc9006ac90 0x7fbc9006d150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbca0008fd0 tx=0x7fbca0007000 comp rx=0 tx=0).stop 2024-03-29T09:54:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 -- 172.21.15.12:0/2905140272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcb015a2d0 msgr2=0x7fbcb00b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcb015a2d0 0x7fbcb00b89e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c011aa0 tx=0x7fbc9c011db0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 -- 172.21.15.12:0/2905140272 shutdown_connections 2024-03-29T09:54:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc9006ac90 0x7fbc9006d150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcb00ca6e0 0x7fbcb00cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc94003a30 0x7fbcb00c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 --2- 172.21.15.12:0/2905140272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcb015a2d0 0x7fbcb00b89e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 -- 172.21.15.12:0/2905140272 >> 172.21.15.12:0/2905140272 conn(0x7fbcb008aff0 msgr2=0x7fbcb015bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:31.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 -- 172.21.15.12:0/2905140272 shutdown_connections 2024-03-29T09:54:31.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:31.852+0000 7fbcb5e00700 1 -- 172.21.15.12:0/2905140272 wait complete. 2024-03-29T09:54:31.870 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T09:54:31.870 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T09:54:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.068+0000 7fd85bc00700 1 Processor -- start 2024-03-29T09:54:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.068+0000 7fd85bc00700 1 -- start start 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.069+0000 7fd85bc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd8540d18b0 0x7fd8540cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.069+0000 7fd85bc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd854161590 0x7fd854161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.069+0000 7fd85bc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8540ceda0 con 0x7fd854160f10 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.069+0000 7fd85bc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8540d1d30 con 0x7fd854161590 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.069+0000 7fd85bc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd854161f60 con 0x7fd8540d18b0 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.070+0000 7fd859c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd854161590 0x7fd854161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.070+0000 7fd859c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd854161590 0x7fd854161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40498/0 (socket says 172.21.15.12:40498) 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.070+0000 7fd859c00700 1 -- 172.21.15.12:0/1411777419 learned_addr learned my addr 172.21.15.12:0/1411777419 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.070+0000 7fd853e00700 1 --2- 172.21.15.12:0/1411777419 >> v2:172.21.15.159:3300/0 conn(0x7fd8540d18b0 0x7fd8540cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.070+0000 7fd859c00700 1 -- 172.21.15.12:0/1411777419 >> v2:172.21.15.159:3300/0 conn(0x7fd8540d18b0 msgr2=0x7fd8540cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.070+0000 7fd859c00700 1 --2- 172.21.15.12:0/1411777419 >> v2:172.21.15.159:3300/0 conn(0x7fd8540d18b0 0x7fd8540cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.070+0000 7fd859c00700 1 -- 172.21.15.12:0/1411777419 >> v1:172.21.15.159:6789/0 conn(0x7fd854160f10 legacy=0x7fd8540d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.070+0000 7fd859c00700 1 -- 172.21.15.12:0/1411777419 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8541612f0 con 0x7fd854161590 2024-03-29T09:54:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.070+0000 7fd859c00700 1 --2- 172.21.15.12:0/1411777419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 0x7fd854161a20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd85408ca30 tx=0x7fd84c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f240e9293e2e818 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd853400700 1 -- 172.21.15.12:0/1411777419 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c00e070 con 0x7fd854161590 2024-03-29T09:54:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd853400700 1 -- 172.21.15.12:0/1411777419 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84c003a20 con 0x7fd854161590 2024-03-29T09:54:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd853400700 1 -- 172.21.15.12:0/1411777419 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c003dd0 con 0x7fd854161590 2024-03-29T09:54:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd85bc00700 1 -- 172.21.15.12:0/1411777419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 msgr2=0x7fd854161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd85bc00700 1 --2- 172.21.15.12:0/1411777419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 0x7fd854161a20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd85408ca30 tx=0x7fd84c00b330 comp rx=0 tx=0).stop 2024-03-29T09:54:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd85bc00700 1 -- 172.21.15.12:0/1411777419 shutdown_connections 2024-03-29T09:54:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd85bc00700 1 --2- 172.21.15.12:0/1411777419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 0x7fd854161a20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd85bc00700 1 --2- 172.21.15.12:0/1411777419 >> v2:172.21.15.159:3300/0 conn(0x7fd8540d18b0 0x7fd8540cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd85bc00700 1 -- 172.21.15.12:0/1411777419 >> 172.21.15.12:0/1411777419 conn(0x7fd85408aff0 msgr2=0x7fd8540d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd85bc00700 1 -- 172.21.15.12:0/1411777419 shutdown_connections 2024-03-29T09:54:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.071+0000 7fd85bc00700 1 -- 172.21.15.12:0/1411777419 wait complete. 2024-03-29T09:54:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.073+0000 7fd85bc00700 1 Processor -- start 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.073+0000 7fd85bc00700 1 -- start start 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.074+0000 7fd85bc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8540d18b0 0x7fd85415b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.074+0000 7fd85bc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd854160f10 0x7fd85409e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.074+0000 7fd85bc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 0x7fd8541e6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.074+0000 7fd85bc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85409f9c0 con 0x7fd854161590 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.074+0000 7fd85bc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85415b620 con 0x7fd854160f10 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.074+0000 7fd85bc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85409edb0 con 0x7fd8540d18b0 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.074+0000 7fd859c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 0x7fd8541e6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.074+0000 7fd859c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 0x7fd8541e6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40502/0 (socket says 172.21.15.12:40502) 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.074+0000 7fd859c00700 1 -- 172.21.15.12:0/4236325783 learned_addr learned my addr 172.21.15.12:0/4236325783 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.074+0000 7fd859200700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8540d18b0 0x7fd85415b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd853e00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd854160f10 0x7fd85409e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd859c00700 1 -- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8540d18b0 msgr2=0x7fd85415b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd859c00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8540d18b0 0x7fd85415b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd859c00700 1 -- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd854160f10 msgr2=0x7fd85409e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd859c00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd854160f10 0x7fd85409e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd859c00700 1 -- 172.21.15.12:0/4236325783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c009b30 con 0x7fd854161590 2024-03-29T09:54:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd859c00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 0x7fd8541e6db0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd84c015040 tx=0x7fd84c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd850c00700 1 -- 172.21.15.12:0/4236325783 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c0094d0 con 0x7fd854161590 2024-03-29T09:54:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd850c00700 1 -- 172.21.15.12:0/4236325783 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84c004240 con 0x7fd854161590 2024-03-29T09:54:32.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd850c00700 1 -- 172.21.15.12:0/4236325783 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c004630 con 0x7fd854161590 2024-03-29T09:54:32.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.075+0000 7fd85bc00700 1 -- 172.21.15.12:0/4236325783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85409f090 con 0x7fd854161590 2024-03-29T09:54:32.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.076+0000 7fd85bc00700 1 -- 172.21.15.12:0/4236325783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85409f650 con 0x7fd854161590 2024-03-29T09:54:32.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.077+0000 7fd85bc00700 1 -- 172.21.15.12:0/4236325783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85409fb30 con 0x7fd854161590 2024-03-29T09:54:32.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.078+0000 7fd850c00700 1 -- 172.21.15.12:0/4236325783 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd84c019070 con 0x7fd854161590 2024-03-29T09:54:32.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.078+0000 7fd850c00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406ac90 0x7fd83406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.079+0000 7fd850c00700 1 -- 172.21.15.12:0/4236325783 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd84c08a190 con 0x7fd854161590 2024-03-29T09:54:32.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.079+0000 7fd859200700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406ac90 0x7fd83406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.079+0000 7fd859200700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406ac90 0x7fd83406d150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd848009960 tx=0x7fd8480090e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:32.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.083+0000 7fd850c00700 1 -- 172.21.15.12:0/4236325783 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd84c058df0 con 0x7fd854161590 2024-03-29T09:54:32.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.293+0000 7fd85bc00700 1 -- 172.21.15.12:0/4236325783 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8540ce750 con 0x7fd83406ac90 2024-03-29T09:54:32.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.302+0000 7fd850c00700 1 -- 172.21.15.12:0/4236325783 <== mgr.4114 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+253963 (secure 0 0 0) 0x7fd848009e30 con 0x7fd83406ac90 2024-03-29T09:54:32.308 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:32.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.306+0000 7fd85bc00700 1 -- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406ac90 msgr2=0x7fd83406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406ac90 0x7fd83406d150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd848009960 tx=0x7fd8480090e0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 -- 172.21.15.12:0/4236325783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 msgr2=0x7fd8541e6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 0x7fd8541e6db0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd84c015040 tx=0x7fd84c004ad0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 -- 172.21.15.12:0/4236325783 shutdown_connections 2024-03-29T09:54:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406ac90 0x7fd83406d150 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd854161590 0x7fd8541e6db0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8540d18b0 0x7fd85415b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 --2- 172.21.15.12:0/4236325783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd854160f10 0x7fd85409e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 -- 172.21.15.12:0/4236325783 >> 172.21.15.12:0/4236325783 conn(0x7fd85408aff0 msgr2=0x7fd8540caac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 -- 172.21.15.12:0/4236325783 shutdown_connections 2024-03-29T09:54:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.307+0000 7fd85bc00700 1 -- 172.21.15.12:0/4236325783 wait complete. 2024-03-29T09:54:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T09:54:32.326 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-03-29T09:54:30.437452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219200,"kb_used_data":2384,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755520,"statfs":{"total":773094113280,"available":772869652480,"internally_reserved":0,"allocated":2441216,"data_stored":1551382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.548121"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.890716+0000","last_change":"2024-03-29T09:54:25.890716+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.890716+0000","last_undegraded":"2024-03-29T09:54:25.890716+0000","last_fullsized":"2024-03-29T09:54:25.890716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:40:59.105279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.896609+0000","last_change":"2024-03-29T09:54:25.896609+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.896609+0000","last_undegraded":"2024-03-29T09:54:25.896609+0000","last_fullsized":"2024-03-29T09:54:25.896609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:37.644975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.896268+0000","last_change":"2024-03-29T09:54:25.896268+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.896268+0000","last_undegraded":"2024-03-29T09:54:25.896268+0000","last_fullsized":"2024-03-29T09:54:25.896268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.894286+0000","last_change":"2024-03-29T09:54:25.894286+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.894286+0000","last_undegraded":"2024-03-29T09:54:25.894286+0000","last_fullsized":"2024-03-29T09:54:25.894286+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:02.185001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.896595+0000","last_change":"2024-03-29T09:54:25.896595+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.896595+0000","last_undegraded":"2024-03-29T09:54:25.896595+0000","last_fullsized":"2024-03-29T09:54:25.896595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:37:17.462538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.893077+0000","last_change":"2024-03-29T09:54:25.893077+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.893077+0000","last_undegraded":"2024-03-29T09:54:25.893077+0000","last_fullsized":"2024-03-29T09:54:25.893077+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.896568+0000","last_change":"2024-03-29T09:54:25.896568+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.896568+0000","last_undegraded":"2024-03-29T09:54:25.896568+0000","last_fullsized":"2024-03-29T09:54:25.896568+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.900316+0000","last_change":"2024-03-29T09:54:25.900316+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.900316+0000","last_undegraded":"2024-03-29T09:54:25.900316+0000","last_fullsized":"2024-03-29T09:54:25.900316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.894202+0000","last_change":"2024-03-29T09:54:25.894202+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.894202+0000","last_undegraded":"2024-03-29T09:54:25.894202+0000","last_fullsized":"2024-03-29T09:54:25.894202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.900847+0000","last_change":"2024-03-29T09:54:25.900847+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.900847+0000","last_undegraded":"2024-03-29T09:54:25.900847+0000","last_fullsized":"2024-03-29T09:54:25.900847+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.897844+0000","last_change":"2024-03-29T09:54:25.897844+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.897844+0000","last_undegraded":"2024-03-29T09:54:25.897844+0000","last_fullsized":"2024-03-29T09:54:25.897844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:02.185001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.893321+0000","last_change":"2024-03-29T09:54:25.893321+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.893321+0000","last_undegraded":"2024-03-29T09:54:25.893321+0000","last_fullsized":"2024-03-29T09:54:25.893321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.890754+0000","last_change":"2024-03-29T09:54:25.890754+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.890754+0000","last_undegraded":"2024-03-29T09:54:25.890754+0000","last_fullsized":"2024-03-29T09:54:25.890754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:49:24.339314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.901505+0000","last_change":"2024-03-29T09:54:25.901505+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.901505+0000","last_undegraded":"2024-03-29T09:54:25.901505+0000","last_fullsized":"2024-03-29T09:54:25.901505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.893168+0000","last_change":"2024-03-29T09:54:25.893168+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.893168+0000","last_undegraded":"2024-03-29T09:54:25.893168+0000","last_fullsized":"2024-03-29T09:54:25.893168+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.904890+0000","last_change":"2024-03-29T09:54:25.904890+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.904890+0000","last_undegraded":"2024-03-29T09:54:25.904890+0000","last_fullsized":"2024-03-29T09:54:25.904890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.896240+0000","last_change":"2024-03-29T09:54:25.896240+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.896240+0000","last_undegraded":"2024-03-29T09:54:25.896240+0000","last_fullsized":"2024-03-29T09:54:25.896240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:53.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.897535+0000","last_change":"2024-03-29T09:54:25.897535+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.897535+0000","last_undegraded":"2024-03-29T09:54:25.897535+0000","last_fullsized":"2024-03-29T09:54:25.897535+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.900343+0000","last_change":"2024-03-29T09:54:25.900343+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.900343+0000","last_undegraded":"2024-03-29T09:54:25.900343+0000","last_fullsized":"2024-03-29T09:54:25.900343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.905083+0000","last_change":"2024-03-29T09:54:25.905083+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.905083+0000","last_undegraded":"2024-03-29T09:54:25.905083+0000","last_fullsized":"2024-03-29T09:54:25.905083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.900700+0000","last_change":"2024-03-29T09:54:25.900700+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.900700+0000","last_undegraded":"2024-03-29T09:54:25.900700+0000","last_fullsized":"2024-03-29T09:54:25.900700+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":161,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-03-29T09:54:21.498721+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T09:54:21.498721+0000","last_peered":"2024-03-29T09:54:21.498721+0000","last_clean":"2024-03-29T09:54:21.498721+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T09:54:21.498721+0000","last_undegraded":"2024-03-29T09:54:21.498721+0000","last_fullsized":"2024-03-29T09:54:21.498721+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.893783+0000","last_change":"2024-03-29T09:54:25.893783+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.893783+0000","last_undegraded":"2024-03-29T09:54:25.893783+0000","last_fullsized":"2024-03-29T09:54:25.893783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.895591+0000","last_change":"2024-03-29T09:54:25.895591+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.895591+0000","last_undegraded":"2024-03-29T09:54:25.895591+0000","last_fullsized":"2024-03-29T09:54:25.895591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:38.666641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.897543+0000","last_change":"2024-03-29T09:54:25.897543+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.897543+0000","last_undegraded":"2024-03-29T09:54:25.897543+0000","last_fullsized":"2024-03-29T09:54:25.897543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:15:56.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.893146+0000","last_change":"2024-03-29T09:54:25.893146+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.893146+0000","last_undegraded":"2024-03-29T09:54:25.893146+0000","last_fullsized":"2024-03-29T09:54:25.893146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.894496+0000","last_change":"2024-03-29T09:54:25.894496+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.894496+0000","last_undegraded":"2024-03-29T09:54:25.894496+0000","last_fullsized":"2024-03-29T09:54:25.894496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.896634+0000","last_change":"2024-03-29T09:54:25.896634+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.896634+0000","last_undegraded":"2024-03-29T09:54:25.896634+0000","last_fullsized":"2024-03-29T09:54:25.896634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.896970+0000","last_change":"2024-03-29T09:54:25.896970+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.896970+0000","last_undegraded":"2024-03-29T09:54:25.896970+0000","last_fullsized":"2024-03-29T09:54:25.896970+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:38.666641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.894872+0000","last_change":"2024-03-29T09:54:25.894872+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.894872+0000","last_undegraded":"2024-03-29T09:54:25.894872+0000","last_fullsized":"2024-03-29T09:54:25.894872+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:15:56.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.904921+0000","last_change":"2024-03-29T09:54:25.904921+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.904921+0000","last_undegraded":"2024-03-29T09:54:25.904921+0000","last_fullsized":"2024-03-29T09:54:25.904921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:37:17.462538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.896568+0000","last_change":"2024-03-29T09:54:25.896568+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.896568+0000","last_undegraded":"2024-03-29T09:54:25.896568+0000","last_fullsized":"2024-03-29T09:54:25.896568+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-29T09:54:25.907255+0000","last_change":"2024-03-29T09:54:25.907255+0000","last_active":"2024-03-29T09:54:25.881798+0000","last_peered":"2024-03-29T09:54:25.881798+0000","last_clean":"2024-03-29T09:54:25.881798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:25.907255+0000","last_undegraded":"2024-03-29T09:54:25.907255+0000","last_fullsized":"2024-03-29T09:54:25.907255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T09:54:26.888423+0000","last_change":"2024-03-29T09:54:26.888423+0000","last_active":"2024-03-29T09:54:26.888423+0000","last_peered":"2024-03-29T09:54:26.888423+0000","last_clean":"2024-03-29T09:54:26.888423+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T09:54:26.888423+0000","last_undegraded":"2024-03-29T09:54:26.888423+0000","last_fullsized":"2024-03-29T09:54:26.888423+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28376,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343464,"statfs":{"total":96636764160,"available":96607707136,"internally_reserved":0,"allocated":749568,"data_stored":636099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":757760,"data_stored":640897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T09:54:32.333 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T09:54:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.565+0000 7f1610e00700 1 Processor -- start 2024-03-29T09:54:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.565+0000 7f1610e00700 1 -- start start 2024-03-29T09:54:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.566+0000 7f1610e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f160c0d18b0 0x7f160c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1610e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c161590 0x7f160c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1610e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c0ceda0 con 0x7f160c160f10 2024-03-29T09:54:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1610e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160c0d1d30 con 0x7f160c0d18b0 2024-03-29T09:54:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1610e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160c161f60 con 0x7f160c161590 2024-03-29T09:54:32.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1609400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f160c0d18b0 0x7f160c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1609400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f160c0d18b0 0x7f160c0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46654/0 (socket says 172.21.15.12:46654) 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1609400700 1 -- 172.21.15.12:0/2621167128 learned_addr learned my addr 172.21.15.12:0/2621167128 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f160a800700 1 --2- 172.21.15.12:0/2621167128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c161590 0x7f160c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1609400700 1 -- 172.21.15.12:0/2621167128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c161590 msgr2=0x7f160c161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1609400700 1 --2- 172.21.15.12:0/2621167128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c161590 0x7f160c161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1609400700 1 -- 172.21.15.12:0/2621167128 >> v1:172.21.15.159:6789/0 conn(0x7f160c160f10 legacy=0x7f160c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f1609400700 1 -- 172.21.15.12:0/2621167128 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c1612f0 con 0x7f160c0d18b0 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.567+0000 7f160a800700 1 --2- 172.21.15.12:0/2621167128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c161590 0x7f160c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.568+0000 7f1609400700 1 --2- 172.21.15.12:0/2621167128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160c0d18b0 0x7f160c0cf230 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009920 tx=0x7f15f8009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=439c751d1b28fd55 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.568+0000 7f1608a00700 1 -- 172.21.15.12:0/2621167128 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8011830 con 0x7f160c0d18b0 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.568+0000 7f1608a00700 1 -- 172.21.15.12:0/2621167128 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f8011990 con 0x7f160c0d18b0 2024-03-29T09:54:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.568+0000 7f1608a00700 1 -- 172.21.15.12:0/2621167128 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8011d20 con 0x7f160c0d18b0 2024-03-29T09:54:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.569+0000 7f1610e00700 1 -- 172.21.15.12:0/2621167128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160c0d18b0 msgr2=0x7f160c0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.569+0000 7f1610e00700 1 --2- 172.21.15.12:0/2621167128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160c0d18b0 0x7f160c0cf230 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009920 tx=0x7f15f8009c30 comp rx=0 tx=0).stop 2024-03-29T09:54:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.569+0000 7f1610e00700 1 -- 172.21.15.12:0/2621167128 shutdown_connections 2024-03-29T09:54:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.569+0000 7f1610e00700 1 --2- 172.21.15.12:0/2621167128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c161590 0x7f160c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.569+0000 7f1610e00700 1 --2- 172.21.15.12:0/2621167128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160c0d18b0 0x7f160c0cf230 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.569+0000 7f1610e00700 1 -- 172.21.15.12:0/2621167128 >> 172.21.15.12:0/2621167128 conn(0x7f160c08aff0 msgr2=0x7f160c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:32.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.569+0000 7f1610e00700 1 -- 172.21.15.12:0/2621167128 shutdown_connections 2024-03-29T09:54:32.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.569+0000 7f1610e00700 1 -- 172.21.15.12:0/2621167128 wait complete. 2024-03-29T09:54:32.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.571+0000 7f1610e00700 1 Processor -- start 2024-03-29T09:54:32.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.571+0000 7f1610e00700 1 -- start start 2024-03-29T09:54:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1610e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160c0d18b0 0x7f160c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1610e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c160f10 0x7f160c1f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1610e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160c161590 0x7f160c0b8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1610e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c1f4ad0 con 0x7f160c0d18b0 2024-03-29T09:54:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1610e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c1f3ec0 con 0x7f160c161590 2024-03-29T09:54:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1610e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160c1f41a0 con 0x7f160c160f10 2024-03-29T09:54:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1609e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160c0d18b0 0x7f160c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1609e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160c0d18b0 0x7f160c0b7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40504/0 (socket says 172.21.15.12:40504) 2024-03-29T09:54:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1609e00700 1 -- 172.21.15.12:0/2603192645 learned_addr learned my addr 172.21.15.12:0/2603192645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1609400700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c160f10 0x7f160c1f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f160a800700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160c161590 0x7f160c0b8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1609e00700 1 -- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c160f10 msgr2=0x7f160c1f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1609e00700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c160f10 0x7f160c1f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1609e00700 1 -- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160c161590 msgr2=0x7f160c0b8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1609e00700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160c161590 0x7f160c0b8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1609e00700 1 -- 172.21.15.12:0/2603192645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1604009a70 con 0x7f160c0d18b0 2024-03-29T09:54:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.572+0000 7f1609400700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c160f10 0x7f160c1f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.573+0000 7f1609e00700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160c0d18b0 0x7f160c0b7090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00c450 tx=0x7f15fc00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.573+0000 7f1602000700 1 -- 172.21.15.12:0/2603192645 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc0052d0 con 0x7f160c0d18b0 2024-03-29T09:54:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.573+0000 7f1602000700 1 -- 172.21.15.12:0/2603192645 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15fc005430 con 0x7f160c0d18b0 2024-03-29T09:54:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.573+0000 7f1610e00700 1 -- 172.21.15.12:0/2603192645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f8009550 con 0x7f160c0d18b0 2024-03-29T09:54:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.573+0000 7f1602000700 1 -- 172.21.15.12:0/2603192645 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc0038c0 con 0x7f160c0d18b0 2024-03-29T09:54:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.573+0000 7f1610e00700 1 -- 172.21.15.12:0/2603192645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f160c1f4760 con 0x7f160c0d18b0 2024-03-29T09:54:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.574+0000 7f1610e00700 1 -- 172.21.15.12:0/2603192645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160c067720 con 0x7f160c0d18b0 2024-03-29T09:54:32.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.575+0000 7f1602000700 1 -- 172.21.15.12:0/2603192645 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f15fc003a20 con 0x7f160c0d18b0 2024-03-29T09:54:32.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.576+0000 7f1602000700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15ec06ac90 0x7f15ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:32.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.577+0000 7f1602000700 1 -- 172.21.15.12:0/2603192645 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15fc088640 con 0x7f160c0d18b0 2024-03-29T09:54:32.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.577+0000 7f1609400700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15ec06ac90 0x7f15ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.578+0000 7f1609400700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15ec06ac90 0x7f15ec06d150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15f8014040 tx=0x7f15f8008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.580+0000 7f1602000700 1 -- 172.21.15.12:0/2603192645 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15fc053810 con 0x7f160c0d18b0 2024-03-29T09:54:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.790+0000 7f1610e00700 1 -- 172.21.15.12:0/2603192645 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f160c0ce750 con 0x7f15ec06ac90 2024-03-29T09:54:32.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.800+0000 7f1602000700 1 -- 172.21.15.12:0/2603192645 <== mgr.4114 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264693 (secure 0 0 0) 0x7f15f8011c00 con 0x7f15ec06ac90 2024-03-29T09:54:32.804 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 -- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15ec06ac90 msgr2=0x7f15ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15ec06ac90 0x7f15ec06d150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15f8014040 tx=0x7f15f8008030 comp rx=0 tx=0).stop 2024-03-29T09:54:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 -- 172.21.15.12:0/2603192645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160c0d18b0 msgr2=0x7f160c0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160c0d18b0 0x7f160c0b7090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00c450 tx=0x7f15fc00c760 comp rx=0 tx=0).stop 2024-03-29T09:54:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 -- 172.21.15.12:0/2603192645 shutdown_connections 2024-03-29T09:54:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15ec06ac90 0x7f15ec06d150 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160c161590 0x7f160c0b8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160c0d18b0 0x7f160c0b7090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 --2- 172.21.15.12:0/2603192645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160c160f10 0x7f160c1f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 -- 172.21.15.12:0/2603192645 >> 172.21.15.12:0/2603192645 conn(0x7f160c08aff0 msgr2=0x7f160c0d03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:32.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 -- 172.21.15.12:0/2603192645 shutdown_connections 2024-03-29T09:54:32.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:32.805+0000 7f1610e00700 1 -- 172.21.15.12:0/2603192645 wait complete. 2024-03-29T09:54:32.807 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T09:54:32.823 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-29T09:54:32.437851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219064,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755656,"statfs":{"total":773094113280,"available":772869791744,"internally_reserved":0,"allocated":2887680,"data_stored":1751625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8],"upper_bound":16},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001529"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674620+0000","last_change":"2024-03-29T09:54:27.472798+0000","last_active":"2024-03-29T09:54:28.674620+0000","last_peered":"2024-03-29T09:54:28.674620+0000","last_clean":"2024-03-29T09:54:28.674620+0000","last_became_active":"2024-03-29T09:54:27.468781+0000","last_became_peered":"2024-03-29T09:54:27.468781+0000","last_unstale":"2024-03-29T09:54:28.674620+0000","last_undegraded":"2024-03-29T09:54:28.674620+0000","last_fullsized":"2024-03-29T09:54:28.674620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:10.721926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572841+0000","last_change":"2024-03-29T09:54:27.456722+0000","last_active":"2024-03-29T09:54:28.572841+0000","last_peered":"2024-03-29T09:54:28.572841+0000","last_clean":"2024-03-29T09:54:28.572841+0000","last_became_active":"2024-03-29T09:54:27.456334+0000","last_became_peered":"2024-03-29T09:54:27.456334+0000","last_unstale":"2024-03-29T09:54:28.572841+0000","last_undegraded":"2024-03-29T09:54:28.572841+0000","last_fullsized":"2024-03-29T09:54:28.572841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:52.185097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674299+0000","last_change":"2024-03-29T09:54:27.455943+0000","last_active":"2024-03-29T09:54:28.674299+0000","last_peered":"2024-03-29T09:54:28.674299+0000","last_clean":"2024-03-29T09:54:28.674299+0000","last_became_active":"2024-03-29T09:54:27.455551+0000","last_became_peered":"2024-03-29T09:54:27.455551+0000","last_unstale":"2024-03-29T09:54:28.674299+0000","last_undegraded":"2024-03-29T09:54:28.674299+0000","last_fullsized":"2024-03-29T09:54:28.674299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:03.191627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438591+0000","last_change":"2024-03-29T09:54:27.445755+0000","last_active":"2024-03-29T09:54:28.438591+0000","last_peered":"2024-03-29T09:54:28.438591+0000","last_clean":"2024-03-29T09:54:28.438591+0000","last_became_active":"2024-03-29T09:54:27.444623+0000","last_became_peered":"2024-03-29T09:54:27.444623+0000","last_unstale":"2024-03-29T09:54:28.438591+0000","last_undegraded":"2024-03-29T09:54:28.438591+0000","last_fullsized":"2024-03-29T09:54:28.438591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:19.141206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674074+0000","last_change":"2024-03-29T09:54:27.468817+0000","last_active":"2024-03-29T09:54:28.674074+0000","last_peered":"2024-03-29T09:54:28.674074+0000","last_clean":"2024-03-29T09:54:28.674074+0000","last_became_active":"2024-03-29T09:54:27.467049+0000","last_became_peered":"2024-03-29T09:54:27.467049+0000","last_unstale":"2024-03-29T09:54:28.674074+0000","last_undegraded":"2024-03-29T09:54:28.674074+0000","last_fullsized":"2024-03-29T09:54:28.674074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:44:20.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437373+0000","last_change":"2024-03-29T09:54:27.453526+0000","last_active":"2024-03-29T09:54:28.437373+0000","last_peered":"2024-03-29T09:54:28.437373+0000","last_clean":"2024-03-29T09:54:28.437373+0000","last_became_active":"2024-03-29T09:54:27.452090+0000","last_became_peered":"2024-03-29T09:54:27.452090+0000","last_unstale":"2024-03-29T09:54:28.437373+0000","last_undegraded":"2024-03-29T09:54:28.437373+0000","last_fullsized":"2024-03-29T09:54:28.437373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:30.050849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673999+0000","last_change":"2024-03-29T09:54:27.471016+0000","last_active":"2024-03-29T09:54:28.673999+0000","last_peered":"2024-03-29T09:54:28.673999+0000","last_clean":"2024-03-29T09:54:28.673999+0000","last_became_active":"2024-03-29T09:54:27.470657+0000","last_became_peered":"2024-03-29T09:54:27.470657+0000","last_unstale":"2024-03-29T09:54:28.673999+0000","last_undegraded":"2024-03-29T09:54:28.673999+0000","last_fullsized":"2024-03-29T09:54:28.673999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:56.949701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673522+0000","last_change":"2024-03-29T09:54:27.458451+0000","last_active":"2024-03-29T09:54:28.673522+0000","last_peered":"2024-03-29T09:54:28.673522+0000","last_clean":"2024-03-29T09:54:28.673522+0000","last_became_active":"2024-03-29T09:54:27.458110+0000","last_became_peered":"2024-03-29T09:54:27.458110+0000","last_unstale":"2024-03-29T09:54:28.673522+0000","last_undegraded":"2024-03-29T09:54:28.673522+0000","last_fullsized":"2024-03-29T09:54:28.673522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:57.886383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674535+0000","last_change":"2024-03-29T09:54:27.471858+0000","last_active":"2024-03-29T09:54:28.674535+0000","last_peered":"2024-03-29T09:54:28.674535+0000","last_clean":"2024-03-29T09:54:28.674535+0000","last_became_active":"2024-03-29T09:54:27.469164+0000","last_became_peered":"2024-03-29T09:54:27.469164+0000","last_unstale":"2024-03-29T09:54:28.674535+0000","last_undegraded":"2024-03-29T09:54:28.674535+0000","last_fullsized":"2024-03-29T09:54:28.674535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:44:42.873042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.439014+0000","last_change":"2024-03-29T09:54:27.442987+0000","last_active":"2024-03-29T09:54:28.439014+0000","last_peered":"2024-03-29T09:54:28.439014+0000","last_clean":"2024-03-29T09:54:28.439014+0000","last_became_active":"2024-03-29T09:54:27.442294+0000","last_became_peered":"2024-03-29T09:54:27.442294+0000","last_unstale":"2024-03-29T09:54:28.439014+0000","last_undegraded":"2024-03-29T09:54:28.439014+0000","last_fullsized":"2024-03-29T09:54:28.439014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438771+0000","last_change":"2024-03-29T09:54:27.442621+0000","last_active":"2024-03-29T09:54:28.438771+0000","last_peered":"2024-03-29T09:54:28.438771+0000","last_clean":"2024-03-29T09:54:28.438771+0000","last_became_active":"2024-03-29T09:54:27.442259+0000","last_became_peered":"2024-03-29T09:54:27.442259+0000","last_unstale":"2024-03-29T09:54:28.438771+0000","last_undegraded":"2024-03-29T09:54:28.438771+0000","last_fullsized":"2024-03-29T09:54:28.438771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436677+0000","last_change":"2024-03-29T09:54:27.453502+0000","last_active":"2024-03-29T09:54:28.436677+0000","last_peered":"2024-03-29T09:54:28.436677+0000","last_clean":"2024-03-29T09:54:28.436677+0000","last_became_active":"2024-03-29T09:54:27.453015+0000","last_became_peered":"2024-03-29T09:54:27.453015+0000","last_unstale":"2024-03-29T09:54:28.436677+0000","last_undegraded":"2024-03-29T09:54:28.436677+0000","last_fullsized":"2024-03-29T09:54:28.436677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:54.979913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674480+0000","last_change":"2024-03-29T09:54:27.451715+0000","last_active":"2024-03-29T09:54:28.674480+0000","last_peered":"2024-03-29T09:54:28.674480+0000","last_clean":"2024-03-29T09:54:28.674480+0000","last_became_active":"2024-03-29T09:54:27.451432+0000","last_became_peered":"2024-03-29T09:54:27.451432+0000","last_unstale":"2024-03-29T09:54:28.674480+0000","last_undegraded":"2024-03-29T09:54:28.674480+0000","last_fullsized":"2024-03-29T09:54:28.674480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674068+0000","last_change":"2024-03-29T09:54:27.460653+0000","last_active":"2024-03-29T09:54:28.674068+0000","last_peered":"2024-03-29T09:54:28.674068+0000","last_clean":"2024-03-29T09:54:28.674068+0000","last_became_active":"2024-03-29T09:54:27.459198+0000","last_became_peered":"2024-03-29T09:54:27.459198+0000","last_unstale":"2024-03-29T09:54:28.674068+0000","last_undegraded":"2024-03-29T09:54:28.674068+0000","last_fullsized":"2024-03-29T09:54:28.674068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:27:59.103915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572329+0000","last_change":"2024-03-29T09:54:27.457510+0000","last_active":"2024-03-29T09:54:28.572329+0000","last_peered":"2024-03-29T09:54:28.572329+0000","last_clean":"2024-03-29T09:54:28.572329+0000","last_became_active":"2024-03-29T09:54:27.456319+0000","last_became_peered":"2024-03-29T09:54:27.456319+0000","last_unstale":"2024-03-29T09:54:28.572329+0000","last_undegraded":"2024-03-29T09:54:28.572329+0000","last_fullsized":"2024-03-29T09:54:28.572329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437168+0000","last_change":"2024-03-29T09:54:27.460545+0000","last_active":"2024-03-29T09:54:28.437168+0000","last_peered":"2024-03-29T09:54:28.437168+0000","last_clean":"2024-03-29T09:54:28.437168+0000","last_became_active":"2024-03-29T09:54:27.460152+0000","last_became_peered":"2024-03-29T09:54:27.460152+0000","last_unstale":"2024-03-29T09:54:28.437168+0000","last_undegraded":"2024-03-29T09:54:28.437168+0000","last_fullsized":"2024-03-29T09:54:28.437168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:52.185097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438011+0000","last_change":"2024-03-29T09:54:27.469947+0000","last_active":"2024-03-29T09:54:28.438011+0000","last_peered":"2024-03-29T09:54:28.438011+0000","last_clean":"2024-03-29T09:54:28.438011+0000","last_became_active":"2024-03-29T09:54:27.469649+0000","last_became_peered":"2024-03-29T09:54:27.469649+0000","last_unstale":"2024-03-29T09:54:28.438011+0000","last_undegraded":"2024-03-29T09:54:28.438011+0000","last_fullsized":"2024-03-29T09:54:28.438011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:44:42.873042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.511192+0000","last_change":"2024-03-29T09:54:27.438206+0000","last_active":"2024-03-29T09:54:28.511192+0000","last_peered":"2024-03-29T09:54:28.511192+0000","last_clean":"2024-03-29T09:54:28.511192+0000","last_became_active":"2024-03-29T09:54:27.437957+0000","last_became_peered":"2024-03-29T09:54:27.437957+0000","last_unstale":"2024-03-29T09:54:28.511192+0000","last_undegraded":"2024-03-29T09:54:28.511192+0000","last_fullsized":"2024-03-29T09:54:28.511192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:03.191627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674238+0000","last_change":"2024-03-29T09:54:27.466200+0000","last_active":"2024-03-29T09:54:28.674238+0000","last_peered":"2024-03-29T09:54:28.674238+0000","last_clean":"2024-03-29T09:54:28.674238+0000","last_became_active":"2024-03-29T09:54:27.465886+0000","last_became_peered":"2024-03-29T09:54:27.465886+0000","last_unstale":"2024-03-29T09:54:28.674238+0000","last_undegraded":"2024-03-29T09:54:28.674238+0000","last_fullsized":"2024-03-29T09:54:28.674238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:57.886383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.459838+0000","last_change":"2024-03-29T09:54:26.929518+0000","last_active":"2024-03-29T09:54:27.459838+0000","last_peered":"2024-03-29T09:54:27.459838+0000","last_clean":"2024-03-29T09:54:27.459838+0000","last_became_active":"2024-03-29T09:54:26.929253+0000","last_became_peered":"2024-03-29T09:54:26.929253+0000","last_unstale":"2024-03-29T09:54:27.459838+0000","last_undegraded":"2024-03-29T09:54:27.459838+0000","last_fullsized":"2024-03-29T09:54:27.459838+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:40:59.105279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674699+0000","last_change":"2024-03-29T09:54:27.454266+0000","last_active":"2024-03-29T09:54:28.674699+0000","last_peered":"2024-03-29T09:54:28.674699+0000","last_clean":"2024-03-29T09:54:28.674699+0000","last_became_active":"2024-03-29T09:54:27.454005+0000","last_became_peered":"2024-03-29T09:54:27.454005+0000","last_unstale":"2024-03-29T09:54:28.674699+0000","last_undegraded":"2024-03-29T09:54:28.674699+0000","last_fullsized":"2024-03-29T09:54:28.674699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:02.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.467235+0000","last_change":"2024-03-29T09:54:26.926684+0000","last_active":"2024-03-29T09:54:27.467235+0000","last_peered":"2024-03-29T09:54:27.467235+0000","last_clean":"2024-03-29T09:54:27.467235+0000","last_became_active":"2024-03-29T09:54:26.926350+0000","last_became_peered":"2024-03-29T09:54:26.926350+0000","last_unstale":"2024-03-29T09:54:27.467235+0000","last_undegraded":"2024-03-29T09:54:27.467235+0000","last_fullsized":"2024-03-29T09:54:27.467235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:37.644975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673888+0000","last_change":"2024-03-29T09:54:27.452090+0000","last_active":"2024-03-29T09:54:28.673888+0000","last_peered":"2024-03-29T09:54:28.673888+0000","last_clean":"2024-03-29T09:54:28.673888+0000","last_became_active":"2024-03-29T09:54:27.451361+0000","last_became_peered":"2024-03-29T09:54:27.451361+0000","last_unstale":"2024-03-29T09:54:28.673888+0000","last_undegraded":"2024-03-29T09:54:28.673888+0000","last_fullsized":"2024-03-29T09:54:28.673888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:25:15.310717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.468152+0000","last_change":"2024-03-29T09:54:26.924715+0000","last_active":"2024-03-29T09:54:27.468152+0000","last_peered":"2024-03-29T09:54:27.468152+0000","last_clean":"2024-03-29T09:54:27.468152+0000","last_became_active":"2024-03-29T09:54:26.924430+0000","last_became_peered":"2024-03-29T09:54:26.924430+0000","last_unstale":"2024-03-29T09:54:27.468152+0000","last_undegraded":"2024-03-29T09:54:27.468152+0000","last_fullsized":"2024-03-29T09:54:27.468152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438051+0000","last_change":"2024-03-29T09:54:27.455410+0000","last_active":"2024-03-29T09:54:28.438051+0000","last_peered":"2024-03-29T09:54:28.438051+0000","last_clean":"2024-03-29T09:54:28.438051+0000","last_became_active":"2024-03-29T09:54:27.453992+0000","last_became_peered":"2024-03-29T09:54:27.453992+0000","last_unstale":"2024-03-29T09:54:28.438051+0000","last_undegraded":"2024-03-29T09:54:28.438051+0000","last_fullsized":"2024-03-29T09:54:28.438051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:57.886383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.469487+0000","last_change":"2024-03-29T09:54:26.931418+0000","last_active":"2024-03-29T09:54:27.469487+0000","last_peered":"2024-03-29T09:54:27.469487+0000","last_clean":"2024-03-29T09:54:27.469487+0000","last_became_active":"2024-03-29T09:54:26.931148+0000","last_became_peered":"2024-03-29T09:54:26.931148+0000","last_unstale":"2024-03-29T09:54:27.469487+0000","last_undegraded":"2024-03-29T09:54:27.469487+0000","last_fullsized":"2024-03-29T09:54:27.469487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:02.185001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437930+0000","last_change":"2024-03-29T09:54:27.454343+0000","last_active":"2024-03-29T09:54:28.437930+0000","last_peered":"2024-03-29T09:54:28.437930+0000","last_clean":"2024-03-29T09:54:28.437930+0000","last_became_active":"2024-03-29T09:54:27.454012+0000","last_became_peered":"2024-03-29T09:54:27.454012+0000","last_unstale":"2024-03-29T09:54:28.437930+0000","last_undegraded":"2024-03-29T09:54:28.437930+0000","last_fullsized":"2024-03-29T09:54:28.437930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:50.524833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.467317+0000","last_change":"2024-03-29T09:54:26.929141+0000","last_active":"2024-03-29T09:54:27.467317+0000","last_peered":"2024-03-29T09:54:27.467317+0000","last_clean":"2024-03-29T09:54:27.467317+0000","last_became_active":"2024-03-29T09:54:26.928876+0000","last_became_peered":"2024-03-29T09:54:26.928876+0000","last_unstale":"2024-03-29T09:54:27.467317+0000","last_undegraded":"2024-03-29T09:54:27.467317+0000","last_fullsized":"2024-03-29T09:54:27.467317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:37:17.462538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436160+0000","last_change":"2024-03-29T09:54:27.448632+0000","last_active":"2024-03-29T09:54:28.436160+0000","last_peered":"2024-03-29T09:54:28.436160+0000","last_clean":"2024-03-29T09:54:28.436160+0000","last_became_active":"2024-03-29T09:54:27.448165+0000","last_became_peered":"2024-03-29T09:54:27.448165+0000","last_unstale":"2024-03-29T09:54:28.436160+0000","last_undegraded":"2024-03-29T09:54:28.436160+0000","last_fullsized":"2024-03-29T09:54:28.436160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:58.125922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.455407+0000","last_change":"2024-03-29T09:54:26.916444+0000","last_active":"2024-03-29T09:54:27.455407+0000","last_peered":"2024-03-29T09:54:27.455407+0000","last_clean":"2024-03-29T09:54:27.455407+0000","last_became_active":"2024-03-29T09:54:26.916132+0000","last_became_peered":"2024-03-29T09:54:26.916132+0000","last_unstale":"2024-03-29T09:54:27.455407+0000","last_undegraded":"2024-03-29T09:54:27.455407+0000","last_fullsized":"2024-03-29T09:54:27.455407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674190+0000","last_change":"2024-03-29T09:54:27.464314+0000","last_active":"2024-03-29T09:54:28.674190+0000","last_peered":"2024-03-29T09:54:28.674190+0000","last_clean":"2024-03-29T09:54:28.674190+0000","last_became_active":"2024-03-29T09:54:27.463702+0000","last_became_peered":"2024-03-29T09:54:27.463702+0000","last_unstale":"2024-03-29T09:54:28.674190+0000","last_undegraded":"2024-03-29T09:54:28.674190+0000","last_fullsized":"2024-03-29T09:54:28.674190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.444809+0000","last_change":"2024-03-29T09:54:26.921122+0000","last_active":"2024-03-29T09:54:27.444809+0000","last_peered":"2024-03-29T09:54:27.444809+0000","last_clean":"2024-03-29T09:54:27.444809+0000","last_became_active":"2024-03-29T09:54:26.919924+0000","last_became_peered":"2024-03-29T09:54:26.919924+0000","last_unstale":"2024-03-29T09:54:27.444809+0000","last_undegraded":"2024-03-29T09:54:27.444809+0000","last_fullsized":"2024-03-29T09:54:27.444809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673522+0000","last_change":"2024-03-29T09:54:27.465683+0000","last_active":"2024-03-29T09:54:28.673522+0000","last_peered":"2024-03-29T09:54:28.673522+0000","last_clean":"2024-03-29T09:54:28.673522+0000","last_became_active":"2024-03-29T09:54:27.465379+0000","last_became_peered":"2024-03-29T09:54:27.465379+0000","last_unstale":"2024-03-29T09:54:28.673522+0000","last_undegraded":"2024-03-29T09:54:28.673522+0000","last_fullsized":"2024-03-29T09:54:28.673522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:52.185097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.434322+0000","last_change":"2024-03-29T09:54:26.918838+0000","last_active":"2024-03-29T09:54:27.434322+0000","last_peered":"2024-03-29T09:54:27.434322+0000","last_clean":"2024-03-29T09:54:27.434322+0000","last_became_active":"2024-03-29T09:54:26.918185+0000","last_became_peered":"2024-03-29T09:54:26.918185+0000","last_unstale":"2024-03-29T09:54:27.434322+0000","last_undegraded":"2024-03-29T09:54:27.434322+0000","last_fullsized":"2024-03-29T09:54:27.434322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572430+0000","last_change":"2024-03-29T09:54:27.449379+0000","last_active":"2024-03-29T09:54:28.572430+0000","last_peered":"2024-03-29T09:54:28.572430+0000","last_clean":"2024-03-29T09:54:28.572430+0000","last_became_active":"2024-03-29T09:54:27.448935+0000","last_became_peered":"2024-03-29T09:54:27.448935+0000","last_unstale":"2024-03-29T09:54:28.572430+0000","last_undegraded":"2024-03-29T09:54:28.572430+0000","last_fullsized":"2024-03-29T09:54:28.572430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.434616+0000","last_change":"2024-03-29T09:54:26.920134+0000","last_active":"2024-03-29T09:54:27.434616+0000","last_peered":"2024-03-29T09:54:27.434616+0000","last_clean":"2024-03-29T09:54:27.434616+0000","last_became_active":"2024-03-29T09:54:26.919158+0000","last_became_peered":"2024-03-29T09:54:26.919158+0000","last_unstale":"2024-03-29T09:54:27.434616+0000","last_undegraded":"2024-03-29T09:54:27.434616+0000","last_fullsized":"2024-03-29T09:54:27.434616+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438287+0000","last_change":"2024-03-29T09:54:27.467877+0000","last_active":"2024-03-29T09:54:28.438287+0000","last_peered":"2024-03-29T09:54:28.438287+0000","last_clean":"2024-03-29T09:54:28.438287+0000","last_became_active":"2024-03-29T09:54:27.467347+0000","last_became_peered":"2024-03-29T09:54:27.467347+0000","last_unstale":"2024-03-29T09:54:28.438287+0000","last_undegraded":"2024-03-29T09:54:28.438287+0000","last_fullsized":"2024-03-29T09:54:28.438287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.437566+0000","last_change":"2024-03-29T09:54:26.911930+0000","last_active":"2024-03-29T09:54:27.437566+0000","last_peered":"2024-03-29T09:54:27.437566+0000","last_clean":"2024-03-29T09:54:27.437566+0000","last_became_active":"2024-03-29T09:54:26.911549+0000","last_became_peered":"2024-03-29T09:54:26.911549+0000","last_unstale":"2024-03-29T09:54:27.437566+0000","last_undegraded":"2024-03-29T09:54:27.437566+0000","last_fullsized":"2024-03-29T09:54:27.437566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674175+0000","last_change":"2024-03-29T09:54:27.451917+0000","last_active":"2024-03-29T09:54:28.674175+0000","last_peered":"2024-03-29T09:54:28.674175+0000","last_clean":"2024-03-29T09:54:28.674175+0000","last_became_active":"2024-03-29T09:54:27.450981+0000","last_became_peered":"2024-03-29T09:54:27.450981+0000","last_unstale":"2024-03-29T09:54:28.674175+0000","last_undegraded":"2024-03-29T09:54:28.674175+0000","last_fullsized":"2024-03-29T09:54:28.674175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:41:21.776144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.470049+0000","last_change":"2024-03-29T09:54:26.917969+0000","last_active":"2024-03-29T09:54:27.470049+0000","last_peered":"2024-03-29T09:54:27.470049+0000","last_clean":"2024-03-29T09:54:27.470049+0000","last_became_active":"2024-03-29T09:54:26.916331+0000","last_became_peered":"2024-03-29T09:54:26.916331+0000","last_unstale":"2024-03-29T09:54:27.470049+0000","last_undegraded":"2024-03-29T09:54:27.470049+0000","last_fullsized":"2024-03-29T09:54:27.470049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:02.185001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436752+0000","last_change":"2024-03-29T09:54:27.465694+0000","last_active":"2024-03-29T09:54:28.436752+0000","last_peered":"2024-03-29T09:54:28.436752+0000","last_clean":"2024-03-29T09:54:28.436752+0000","last_became_active":"2024-03-29T09:54:27.465420+0000","last_became_peered":"2024-03-29T09:54:27.465420+0000","last_unstale":"2024-03-29T09:54:28.436752+0000","last_undegraded":"2024-03-29T09:54:28.436752+0000","last_fullsized":"2024-03-29T09:54:28.436752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:39.673266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.474851+0000","last_change":"2024-03-29T09:54:26.925541+0000","last_active":"2024-03-29T09:54:27.474851+0000","last_peered":"2024-03-29T09:54:27.474851+0000","last_clean":"2024-03-29T09:54:27.474851+0000","last_became_active":"2024-03-29T09:54:26.925012+0000","last_became_peered":"2024-03-29T09:54:26.925012+0000","last_unstale":"2024-03-29T09:54:27.474851+0000","last_undegraded":"2024-03-29T09:54:27.474851+0000","last_fullsized":"2024-03-29T09:54:27.474851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674348+0000","last_change":"2024-03-29T09:54:27.470254+0000","last_active":"2024-03-29T09:54:28.674348+0000","last_peered":"2024-03-29T09:54:28.674348+0000","last_clean":"2024-03-29T09:54:28.674348+0000","last_became_active":"2024-03-29T09:54:27.468546+0000","last_became_peered":"2024-03-29T09:54:27.468546+0000","last_unstale":"2024-03-29T09:54:28.674348+0000","last_undegraded":"2024-03-29T09:54:28.674348+0000","last_fullsized":"2024-03-29T09:54:28.674348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:30.050849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.464632+0000","last_change":"2024-03-29T09:54:26.924638+0000","last_active":"2024-03-29T09:54:27.464632+0000","last_peered":"2024-03-29T09:54:27.464632+0000","last_clean":"2024-03-29T09:54:27.464632+0000","last_became_active":"2024-03-29T09:54:26.923692+0000","last_became_peered":"2024-03-29T09:54:26.923692+0000","last_unstale":"2024-03-29T09:54:27.464632+0000","last_undegraded":"2024-03-29T09:54:27.464632+0000","last_fullsized":"2024-03-29T09:54:27.464632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:49:24.339314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673965+0000","last_change":"2024-03-29T09:54:27.465512+0000","last_active":"2024-03-29T09:54:28.673965+0000","last_peered":"2024-03-29T09:54:28.673965+0000","last_clean":"2024-03-29T09:54:28.673965+0000","last_became_active":"2024-03-29T09:54:27.465258+0000","last_became_peered":"2024-03-29T09:54:27.465258+0000","last_unstale":"2024-03-29T09:54:28.673965+0000","last_undegraded":"2024-03-29T09:54:28.673965+0000","last_fullsized":"2024-03-29T09:54:28.673965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.437301+0000","last_change":"2024-03-29T09:54:26.922511+0000","last_active":"2024-03-29T09:54:27.437301+0000","last_peered":"2024-03-29T09:54:27.437301+0000","last_clean":"2024-03-29T09:54:27.437301+0000","last_became_active":"2024-03-29T09:54:26.922182+0000","last_became_peered":"2024-03-29T09:54:26.922182+0000","last_unstale":"2024-03-29T09:54:27.437301+0000","last_undegraded":"2024-03-29T09:54:27.437301+0000","last_fullsized":"2024-03-29T09:54:27.437301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674259+0000","last_change":"2024-03-29T09:54:27.472857+0000","last_active":"2024-03-29T09:54:28.674259+0000","last_peered":"2024-03-29T09:54:28.674259+0000","last_clean":"2024-03-29T09:54:28.674259+0000","last_became_active":"2024-03-29T09:54:27.472482+0000","last_became_peered":"2024-03-29T09:54:27.472482+0000","last_unstale":"2024-03-29T09:54:28.674259+0000","last_undegraded":"2024-03-29T09:54:28.674259+0000","last_fullsized":"2024-03-29T09:54:28.674259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:18.345522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.432778+0000","last_change":"2024-03-29T09:54:26.920268+0000","last_active":"2024-03-29T09:54:27.432778+0000","last_peered":"2024-03-29T09:54:27.432778+0000","last_clean":"2024-03-29T09:54:27.432778+0000","last_became_active":"2024-03-29T09:54:26.919956+0000","last_became_peered":"2024-03-29T09:54:26.919956+0000","last_unstale":"2024-03-29T09:54:27.432778+0000","last_undegraded":"2024-03-29T09:54:27.432778+0000","last_fullsized":"2024-03-29T09:54:27.432778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438219+0000","last_change":"2024-03-29T09:54:27.457604+0000","last_active":"2024-03-29T09:54:28.438219+0000","last_peered":"2024-03-29T09:54:28.438219+0000","last_clean":"2024-03-29T09:54:28.438219+0000","last_became_active":"2024-03-29T09:54:27.457197+0000","last_became_peered":"2024-03-29T09:54:27.457197+0000","last_unstale":"2024-03-29T09:54:28.438219+0000","last_undegraded":"2024-03-29T09:54:28.438219+0000","last_fullsized":"2024-03-29T09:54:28.438219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:25:15.310717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.464670+0000","last_change":"2024-03-29T09:54:26.921804+0000","last_active":"2024-03-29T09:54:27.464670+0000","last_peered":"2024-03-29T09:54:27.464670+0000","last_clean":"2024-03-29T09:54:27.464670+0000","last_became_active":"2024-03-29T09:54:26.921487+0000","last_became_peered":"2024-03-29T09:54:26.921487+0000","last_unstale":"2024-03-29T09:54:27.464670+0000","last_undegraded":"2024-03-29T09:54:27.464670+0000","last_fullsized":"2024-03-29T09:54:27.464670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674124+0000","last_change":"2024-03-29T09:54:27.470978+0000","last_active":"2024-03-29T09:54:28.674124+0000","last_peered":"2024-03-29T09:54:28.674124+0000","last_clean":"2024-03-29T09:54:28.674124+0000","last_became_active":"2024-03-29T09:54:27.470690+0000","last_became_peered":"2024-03-29T09:54:27.470690+0000","last_unstale":"2024-03-29T09:54:28.674124+0000","last_undegraded":"2024-03-29T09:54:28.674124+0000","last_fullsized":"2024-03-29T09:54:28.674124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:19.141206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.456285+0000","last_change":"2024-03-29T09:54:26.928327+0000","last_active":"2024-03-29T09:54:27.456285+0000","last_peered":"2024-03-29T09:54:27.456285+0000","last_clean":"2024-03-29T09:54:27.456285+0000","last_became_active":"2024-03-29T09:54:26.927938+0000","last_became_peered":"2024-03-29T09:54:26.927938+0000","last_unstale":"2024-03-29T09:54:27.456285+0000","last_undegraded":"2024-03-29T09:54:27.456285+0000","last_fullsized":"2024-03-29T09:54:27.456285+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:53.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674582+0000","last_change":"2024-03-29T09:54:27.466328+0000","last_active":"2024-03-29T09:54:28.674582+0000","last_peered":"2024-03-29T09:54:28.674582+0000","last_clean":"2024-03-29T09:54:28.674582+0000","last_became_active":"2024-03-29T09:54:27.465869+0000","last_became_peered":"2024-03-29T09:54:27.465869+0000","last_unstale":"2024-03-29T09:54:28.674582+0000","last_undegraded":"2024-03-29T09:54:28.674582+0000","last_fullsized":"2024-03-29T09:54:28.674582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:50.524833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.438039+0000","last_change":"2024-03-29T09:54:26.925276+0000","last_active":"2024-03-29T09:54:27.438039+0000","last_peered":"2024-03-29T09:54:27.438039+0000","last_clean":"2024-03-29T09:54:27.438039+0000","last_became_active":"2024-03-29T09:54:26.924980+0000","last_became_peered":"2024-03-29T09:54:26.924980+0000","last_unstale":"2024-03-29T09:54:27.438039+0000","last_undegraded":"2024-03-29T09:54:27.438039+0000","last_fullsized":"2024-03-29T09:54:27.438039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438030+0000","last_change":"2024-03-29T09:54:27.447713+0000","last_active":"2024-03-29T09:54:28.438030+0000","last_peered":"2024-03-29T09:54:28.438030+0000","last_clean":"2024-03-29T09:54:28.438030+0000","last_became_active":"2024-03-29T09:54:27.446552+0000","last_became_peered":"2024-03-29T09:54:27.446552+0000","last_unstale":"2024-03-29T09:54:28.438030+0000","last_undegraded":"2024-03-29T09:54:28.438030+0000","last_fullsized":"2024-03-29T09:54:28.438030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:31.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.437304+0000","last_change":"2024-03-29T09:54:26.911470+0000","last_active":"2024-03-29T09:54:27.437304+0000","last_peered":"2024-03-29T09:54:27.437304+0000","last_clean":"2024-03-29T09:54:27.437304+0000","last_became_active":"2024-03-29T09:54:26.911092+0000","last_became_peered":"2024-03-29T09:54:26.911092+0000","last_unstale":"2024-03-29T09:54:27.437304+0000","last_undegraded":"2024-03-29T09:54:27.437304+0000","last_fullsized":"2024-03-29T09:54:27.437304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436533+0000","last_change":"2024-03-29T09:54:27.459676+0000","last_active":"2024-03-29T09:54:28.436533+0000","last_peered":"2024-03-29T09:54:28.436533+0000","last_clean":"2024-03-29T09:54:28.436533+0000","last_became_active":"2024-03-29T09:54:27.459375+0000","last_became_peered":"2024-03-29T09:54:27.459375+0000","last_unstale":"2024-03-29T09:54:28.436533+0000","last_undegraded":"2024-03-29T09:54:28.436533+0000","last_fullsized":"2024-03-29T09:54:28.436533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:09.734996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572306+0000","last_change":"2024-03-29T09:54:27.446597+0000","last_active":"2024-03-29T09:54:28.572306+0000","last_peered":"2024-03-29T09:54:28.572306+0000","last_clean":"2024-03-29T09:54:28.572306+0000","last_became_active":"2024-03-29T09:54:27.444170+0000","last_became_peered":"2024-03-29T09:54:27.444170+0000","last_unstale":"2024-03-29T09:54:28.572306+0000","last_undegraded":"2024-03-29T09:54:28.572306+0000","last_fullsized":"2024-03-29T09:54:28.572306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.432316+0000","last_change":"2024-03-29T09:54:26.923022+0000","last_active":"2024-03-29T09:54:27.432316+0000","last_peered":"2024-03-29T09:54:27.432316+0000","last_clean":"2024-03-29T09:54:27.432316+0000","last_became_active":"2024-03-29T09:54:26.922558+0000","last_became_peered":"2024-03-29T09:54:26.922558+0000","last_unstale":"2024-03-29T09:54:27.432316+0000","last_undegraded":"2024-03-29T09:54:27.432316+0000","last_fullsized":"2024-03-29T09:54:27.432316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674355+0000","last_change":"2024-03-29T09:54:27.452598+0000","last_active":"2024-03-29T09:54:28.674355+0000","last_peered":"2024-03-29T09:54:28.674355+0000","last_clean":"2024-03-29T09:54:28.674355+0000","last_became_active":"2024-03-29T09:54:27.452331+0000","last_became_peered":"2024-03-29T09:54:27.452331+0000","last_unstale":"2024-03-29T09:54:28.674355+0000","last_undegraded":"2024-03-29T09:54:28.674355+0000","last_fullsized":"2024-03-29T09:54:28.674355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:50.524833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.433351+0000","last_change":"2024-03-29T09:54:26.916736+0000","last_active":"2024-03-29T09:54:27.433351+0000","last_peered":"2024-03-29T09:54:27.433351+0000","last_clean":"2024-03-29T09:54:27.433351+0000","last_became_active":"2024-03-29T09:54:26.916439+0000","last_became_peered":"2024-03-29T09:54:26.916439+0000","last_unstale":"2024-03-29T09:54:27.433351+0000","last_undegraded":"2024-03-29T09:54:27.433351+0000","last_fullsized":"2024-03-29T09:54:27.433351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.453076+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T09:54:27.453076+0000","last_peered":"2024-03-29T09:54:27.453076+0000","last_clean":"2024-03-29T09:54:27.453076+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T09:54:27.453076+0000","last_undegraded":"2024-03-29T09:54:27.453076+0000","last_fullsized":"2024-03-29T09:54:27.453076+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673776+0000","last_change":"2024-03-29T09:54:27.462932+0000","last_active":"2024-03-29T09:54:28.673776+0000","last_peered":"2024-03-29T09:54:28.673776+0000","last_clean":"2024-03-29T09:54:28.673776+0000","last_became_active":"2024-03-29T09:54:27.462665+0000","last_became_peered":"2024-03-29T09:54:27.462665+0000","last_unstale":"2024-03-29T09:54:28.673776+0000","last_undegraded":"2024-03-29T09:54:28.673776+0000","last_fullsized":"2024-03-29T09:54:28.673776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:02.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.453904+0000","last_change":"2024-03-29T09:54:26.924232+0000","last_active":"2024-03-29T09:54:27.453904+0000","last_peered":"2024-03-29T09:54:27.453904+0000","last_clean":"2024-03-29T09:54:27.453904+0000","last_became_active":"2024-03-29T09:54:26.923740+0000","last_became_peered":"2024-03-29T09:54:26.923740+0000","last_unstale":"2024-03-29T09:54:27.453904+0000","last_undegraded":"2024-03-29T09:54:27.453904+0000","last_fullsized":"2024-03-29T09:54:27.453904+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.457620+0000","last_change":"2024-03-29T09:54:26.929540+0000","last_active":"2024-03-29T09:54:27.457620+0000","last_peered":"2024-03-29T09:54:27.457620+0000","last_clean":"2024-03-29T09:54:27.457620+0000","last_became_active":"2024-03-29T09:54:26.928753+0000","last_became_peered":"2024-03-29T09:54:26.928753+0000","last_unstale":"2024-03-29T09:54:27.457620+0000","last_undegraded":"2024-03-29T09:54:27.457620+0000","last_fullsized":"2024-03-29T09:54:27.457620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:38.666641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572622+0000","last_change":"2024-03-29T09:54:27.449508+0000","last_active":"2024-03-29T09:54:28.572622+0000","last_peered":"2024-03-29T09:54:28.572622+0000","last_clean":"2024-03-29T09:54:28.572622+0000","last_became_active":"2024-03-29T09:54:27.449177+0000","last_became_peered":"2024-03-29T09:54:27.449177+0000","last_unstale":"2024-03-29T09:54:28.572622+0000","last_undegraded":"2024-03-29T09:54:28.572622+0000","last_fullsized":"2024-03-29T09:54:28.572622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.437867+0000","last_change":"2024-03-29T09:54:26.909694+0000","last_active":"2024-03-29T09:54:27.437867+0000","last_peered":"2024-03-29T09:54:27.437867+0000","last_clean":"2024-03-29T09:54:27.437867+0000","last_became_active":"2024-03-29T09:54:26.909172+0000","last_became_peered":"2024-03-29T09:54:26.909172+0000","last_unstale":"2024-03-29T09:54:27.437867+0000","last_undegraded":"2024-03-29T09:54:27.437867+0000","last_fullsized":"2024-03-29T09:54:27.437867+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:15:56.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674814+0000","last_change":"2024-03-29T09:54:27.466321+0000","last_active":"2024-03-29T09:54:28.674814+0000","last_peered":"2024-03-29T09:54:28.674814+0000","last_clean":"2024-03-29T09:54:28.674814+0000","last_became_active":"2024-03-29T09:54:27.465544+0000","last_became_peered":"2024-03-29T09:54:27.465544+0000","last_unstale":"2024-03-29T09:54:28.674814+0000","last_undegraded":"2024-03-29T09:54:28.674814+0000","last_fullsized":"2024-03-29T09:54:28.674814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.450346+0000","last_change":"2024-03-29T09:54:26.917641+0000","last_active":"2024-03-29T09:54:27.450346+0000","last_peered":"2024-03-29T09:54:27.450346+0000","last_clean":"2024-03-29T09:54:27.450346+0000","last_became_active":"2024-03-29T09:54:26.915808+0000","last_became_peered":"2024-03-29T09:54:26.915808+0000","last_unstale":"2024-03-29T09:54:27.450346+0000","last_undegraded":"2024-03-29T09:54:27.450346+0000","last_fullsized":"2024-03-29T09:54:27.450346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674304+0000","last_change":"2024-03-29T09:54:27.459303+0000","last_active":"2024-03-29T09:54:28.674304+0000","last_peered":"2024-03-29T09:54:28.674304+0000","last_clean":"2024-03-29T09:54:28.674304+0000","last_became_active":"2024-03-29T09:54:27.458902+0000","last_became_peered":"2024-03-29T09:54:27.458902+0000","last_unstale":"2024-03-29T09:54:28.674304+0000","last_undegraded":"2024-03-29T09:54:28.674304+0000","last_fullsized":"2024-03-29T09:54:28.674304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:25.651718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.454803+0000","last_change":"2024-03-29T09:54:26.929171+0000","last_active":"2024-03-29T09:54:27.454803+0000","last_peered":"2024-03-29T09:54:27.454803+0000","last_clean":"2024-03-29T09:54:27.454803+0000","last_became_active":"2024-03-29T09:54:26.928796+0000","last_became_peered":"2024-03-29T09:54:26.928796+0000","last_unstale":"2024-03-29T09:54:27.454803+0000","last_undegraded":"2024-03-29T09:54:27.454803+0000","last_fullsized":"2024-03-29T09:54:27.454803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436200+0000","last_change":"2024-03-29T09:54:27.453529+0000","last_active":"2024-03-29T09:54:28.436200+0000","last_peered":"2024-03-29T09:54:28.436200+0000","last_clean":"2024-03-29T09:54:28.436200+0000","last_became_active":"2024-03-29T09:54:27.453153+0000","last_became_peered":"2024-03-29T09:54:27.453153+0000","last_unstale":"2024-03-29T09:54:28.436200+0000","last_undegraded":"2024-03-29T09:54:28.436200+0000","last_fullsized":"2024-03-29T09:54:28.436200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:25.068710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.438223+0000","last_change":"2024-03-29T09:54:26.916239+0000","last_active":"2024-03-29T09:54:27.438223+0000","last_peered":"2024-03-29T09:54:27.438223+0000","last_clean":"2024-03-29T09:54:27.438223+0000","last_became_active":"2024-03-29T09:54:26.915960+0000","last_became_peered":"2024-03-29T09:54:26.915960+0000","last_unstale":"2024-03-29T09:54:27.438223+0000","last_undegraded":"2024-03-29T09:54:27.438223+0000","last_fullsized":"2024-03-29T09:54:27.438223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673647+0000","last_change":"2024-03-29T09:54:27.448090+0000","last_active":"2024-03-29T09:54:28.673647+0000","last_peered":"2024-03-29T09:54:28.673647+0000","last_clean":"2024-03-29T09:54:28.673647+0000","last_became_active":"2024-03-29T09:54:27.447614+0000","last_became_peered":"2024-03-29T09:54:27.447614+0000","last_unstale":"2024-03-29T09:54:28.673647+0000","last_undegraded":"2024-03-29T09:54:28.673647+0000","last_fullsized":"2024-03-29T09:54:28.673647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:25:15.310717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.438747+0000","last_change":"2024-03-29T09:54:26.919921+0000","last_active":"2024-03-29T09:54:27.438747+0000","last_peered":"2024-03-29T09:54:27.438747+0000","last_clean":"2024-03-29T09:54:27.438747+0000","last_became_active":"2024-03-29T09:54:26.919609+0000","last_became_peered":"2024-03-29T09:54:26.919609+0000","last_unstale":"2024-03-29T09:54:27.438747+0000","last_undegraded":"2024-03-29T09:54:27.438747+0000","last_fullsized":"2024-03-29T09:54:27.438747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:38.666641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674643+0000","last_change":"2024-03-29T09:54:27.458246+0000","last_active":"2024-03-29T09:54:28.674643+0000","last_peered":"2024-03-29T09:54:28.674643+0000","last_clean":"2024-03-29T09:54:28.674643+0000","last_became_active":"2024-03-29T09:54:27.457985+0000","last_became_peered":"2024-03-29T09:54:27.457985+0000","last_unstale":"2024-03-29T09:54:28.674643+0000","last_undegraded":"2024-03-29T09:54:28.674643+0000","last_fullsized":"2024-03-29T09:54:28.674643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:30.050849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.469210+0000","last_change":"2024-03-29T09:54:26.927045+0000","last_active":"2024-03-29T09:54:27.469210+0000","last_peered":"2024-03-29T09:54:27.469210+0000","last_clean":"2024-03-29T09:54:27.469210+0000","last_became_active":"2024-03-29T09:54:26.926750+0000","last_became_peered":"2024-03-29T09:54:26.926750+0000","last_unstale":"2024-03-29T09:54:27.469210+0000","last_undegraded":"2024-03-29T09:54:27.469210+0000","last_fullsized":"2024-03-29T09:54:27.469210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:15:56.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437955+0000","last_change":"2024-03-29T09:54:27.447647+0000","last_active":"2024-03-29T09:54:28.437955+0000","last_peered":"2024-03-29T09:54:28.437955+0000","last_clean":"2024-03-29T09:54:28.437955+0000","last_became_active":"2024-03-29T09:54:27.447371+0000","last_became_peered":"2024-03-29T09:54:27.447371+0000","last_unstale":"2024-03-29T09:54:28.437955+0000","last_undegraded":"2024-03-29T09:54:28.437955+0000","last_fullsized":"2024-03-29T09:54:28.437955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.432327+0000","last_change":"2024-03-29T09:54:26.929260+0000","last_active":"2024-03-29T09:54:27.432327+0000","last_peered":"2024-03-29T09:54:27.432327+0000","last_clean":"2024-03-29T09:54:27.432327+0000","last_became_active":"2024-03-29T09:54:26.928909+0000","last_became_peered":"2024-03-29T09:54:26.928909+0000","last_unstale":"2024-03-29T09:54:27.432327+0000","last_undegraded":"2024-03-29T09:54:27.432327+0000","last_fullsized":"2024-03-29T09:54:27.432327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:37:17.462538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674364+0000","last_change":"2024-03-29T09:54:27.471604+0000","last_active":"2024-03-29T09:54:28.674364+0000","last_peered":"2024-03-29T09:54:28.674364+0000","last_clean":"2024-03-29T09:54:28.674364+0000","last_became_active":"2024-03-29T09:54:27.471197+0000","last_became_peered":"2024-03-29T09:54:27.471197+0000","last_unstale":"2024-03-29T09:54:28.674364+0000","last_undegraded":"2024-03-29T09:54:28.674364+0000","last_fullsized":"2024-03-29T09:54:28.674364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:25.712202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.463137+0000","last_change":"2024-03-29T09:54:26.916668+0000","last_active":"2024-03-29T09:54:27.463137+0000","last_peered":"2024-03-29T09:54:27.463137+0000","last_clean":"2024-03-29T09:54:27.463137+0000","last_became_active":"2024-03-29T09:54:26.916233+0000","last_became_peered":"2024-03-29T09:54:26.916233+0000","last_unstale":"2024-03-29T09:54:27.463137+0000","last_undegraded":"2024-03-29T09:54:27.463137+0000","last_fullsized":"2024-03-29T09:54:27.463137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438328+0000","last_change":"2024-03-29T09:54:27.444803+0000","last_active":"2024-03-29T09:54:28.438328+0000","last_peered":"2024-03-29T09:54:28.438328+0000","last_clean":"2024-03-29T09:54:28.438328+0000","last_became_active":"2024-03-29T09:54:27.444104+0000","last_became_peered":"2024-03-29T09:54:27.444104+0000","last_unstale":"2024-03-29T09:54:28.438328+0000","last_undegraded":"2024-03-29T09:54:28.438328+0000","last_fullsized":"2024-03-29T09:54:28.438328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:25.712202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.464351+0000","last_change":"2024-03-29T09:54:26.923513+0000","last_active":"2024-03-29T09:54:27.464351+0000","last_peered":"2024-03-29T09:54:27.464351+0000","last_clean":"2024-03-29T09:54:27.464351+0000","last_became_active":"2024-03-29T09:54:26.922870+0000","last_became_peered":"2024-03-29T09:54:26.922870+0000","last_unstale":"2024-03-29T09:54:27.464351+0000","last_undegraded":"2024-03-29T09:54:27.464351+0000","last_fullsized":"2024-03-29T09:54:27.464351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674452+0000","last_change":"2024-03-29T09:54:27.471588+0000","last_active":"2024-03-29T09:54:28.674452+0000","last_peered":"2024-03-29T09:54:28.674452+0000","last_clean":"2024-03-29T09:54:28.674452+0000","last_became_active":"2024-03-29T09:54:27.468491+0000","last_became_peered":"2024-03-29T09:54:27.468491+0000","last_unstale":"2024-03-29T09:54:28.674452+0000","last_undegraded":"2024-03-29T09:54:28.674452+0000","last_fullsized":"2024-03-29T09:54:28.674452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:31.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438168+0000","last_change":"2024-03-29T09:54:27.461434+0000","last_active":"2024-03-29T09:54:28.438168+0000","last_peered":"2024-03-29T09:54:28.438168+0000","last_clean":"2024-03-29T09:54:28.438168+0000","last_became_active":"2024-03-29T09:54:27.461095+0000","last_became_peered":"2024-03-29T09:54:27.461095+0000","last_unstale":"2024-03-29T09:54:28.438168+0000","last_undegraded":"2024-03-29T09:54:28.438168+0000","last_fullsized":"2024-03-29T09:54:28.438168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:02.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.511270+0000","last_change":"2024-03-29T09:54:27.445834+0000","last_active":"2024-03-29T09:54:28.511270+0000","last_peered":"2024-03-29T09:54:28.511270+0000","last_clean":"2024-03-29T09:54:28.511270+0000","last_became_active":"2024-03-29T09:54:27.444597+0000","last_became_peered":"2024-03-29T09:54:27.444597+0000","last_unstale":"2024-03-29T09:54:28.511270+0000","last_undegraded":"2024-03-29T09:54:28.511270+0000","last_fullsized":"2024-03-29T09:54:28.511270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:36.310545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437472+0000","last_change":"2024-03-29T09:54:27.449523+0000","last_active":"2024-03-29T09:54:28.437472+0000","last_peered":"2024-03-29T09:54:28.437472+0000","last_clean":"2024-03-29T09:54:28.437472+0000","last_became_active":"2024-03-29T09:54:27.447299+0000","last_became_peered":"2024-03-29T09:54:27.447299+0000","last_unstale":"2024-03-29T09:54:28.437472+0000","last_undegraded":"2024-03-29T09:54:28.437472+0000","last_fullsized":"2024-03-29T09:54:28.437472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:18.345522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674066+0000","last_change":"2024-03-29T09:54:27.464458+0000","last_active":"2024-03-29T09:54:28.674066+0000","last_peered":"2024-03-29T09:54:28.674066+0000","last_clean":"2024-03-29T09:54:28.674066+0000","last_became_active":"2024-03-29T09:54:27.464146+0000","last_became_peered":"2024-03-29T09:54:27.464146+0000","last_unstale":"2024-03-29T09:54:28.674066+0000","last_undegraded":"2024-03-29T09:54:28.674066+0000","last_fullsized":"2024-03-29T09:54:28.674066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674683+0000","last_change":"2024-03-29T09:54:27.463860+0000","last_active":"2024-03-29T09:54:28.674683+0000","last_peered":"2024-03-29T09:54:28.674683+0000","last_clean":"2024-03-29T09:54:28.674683+0000","last_became_active":"2024-03-29T09:54:27.461514+0000","last_became_peered":"2024-03-29T09:54:27.461514+0000","last_unstale":"2024-03-29T09:54:28.674683+0000","last_undegraded":"2024-03-29T09:54:28.674683+0000","last_fullsized":"2024-03-29T09:54:28.674683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572614+0000","last_change":"2024-03-29T09:54:27.457584+0000","last_active":"2024-03-29T09:54:28.572614+0000","last_peered":"2024-03-29T09:54:28.572614+0000","last_clean":"2024-03-29T09:54:28.572614+0000","last_became_active":"2024-03-29T09:54:27.457325+0000","last_became_peered":"2024-03-29T09:54:27.457325+0000","last_unstale":"2024-03-29T09:54:28.572614+0000","last_undegraded":"2024-03-29T09:54:28.572614+0000","last_fullsized":"2024-03-29T09:54:28.572614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.439001+0000","last_change":"2024-03-29T09:54:27.467951+0000","last_active":"2024-03-29T09:54:28.439001+0000","last_peered":"2024-03-29T09:54:28.439001+0000","last_clean":"2024-03-29T09:54:28.439001+0000","last_became_active":"2024-03-29T09:54:27.466080+0000","last_became_peered":"2024-03-29T09:54:27.466080+0000","last_unstale":"2024-03-29T09:54:28.439001+0000","last_undegraded":"2024-03-29T09:54:28.439001+0000","last_fullsized":"2024-03-29T09:54:28.439001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674709+0000","last_change":"2024-03-29T09:54:27.460513+0000","last_active":"2024-03-29T09:54:28.674709+0000","last_peered":"2024-03-29T09:54:28.674709+0000","last_clean":"2024-03-29T09:54:28.674709+0000","last_became_active":"2024-03-29T09:54:27.460231+0000","last_became_peered":"2024-03-29T09:54:27.460231+0000","last_unstale":"2024-03-29T09:54:28.674709+0000","last_undegraded":"2024-03-29T09:54:28.674709+0000","last_fullsized":"2024-03-29T09:54:28.674709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437313+0000","last_change":"2024-03-29T09:54:27.462171+0000","last_active":"2024-03-29T09:54:28.437313+0000","last_peered":"2024-03-29T09:54:28.437313+0000","last_clean":"2024-03-29T09:54:28.437313+0000","last_became_active":"2024-03-29T09:54:27.461066+0000","last_became_peered":"2024-03-29T09:54:27.461066+0000","last_unstale":"2024-03-29T09:54:28.437313+0000","last_undegraded":"2024-03-29T09:54:28.437313+0000","last_fullsized":"2024-03-29T09:54:28.437313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:44:20.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674108+0000","last_change":"2024-03-29T09:54:27.456049+0000","last_active":"2024-03-29T09:54:28.674108+0000","last_peered":"2024-03-29T09:54:28.674108+0000","last_clean":"2024-03-29T09:54:28.674108+0000","last_became_active":"2024-03-29T09:54:27.454521+0000","last_became_peered":"2024-03-29T09:54:27.454521+0000","last_unstale":"2024-03-29T09:54:28.674108+0000","last_undegraded":"2024-03-29T09:54:28.674108+0000","last_fullsized":"2024-03-29T09:54:28.674108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437418+0000","last_change":"2024-03-29T09:54:27.465472+0000","last_active":"2024-03-29T09:54:28.437418+0000","last_peered":"2024-03-29T09:54:28.437418+0000","last_clean":"2024-03-29T09:54:28.437418+0000","last_became_active":"2024-03-29T09:54:27.465180+0000","last_became_peered":"2024-03-29T09:54:27.465180+0000","last_unstale":"2024-03-29T09:54:28.437418+0000","last_undegraded":"2024-03-29T09:54:28.437418+0000","last_fullsized":"2024-03-29T09:54:28.437418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674044+0000","last_change":"2024-03-29T09:54:27.466113+0000","last_active":"2024-03-29T09:54:28.674044+0000","last_peered":"2024-03-29T09:54:28.674044+0000","last_clean":"2024-03-29T09:54:28.674044+0000","last_became_active":"2024-03-29T09:54:27.465912+0000","last_became_peered":"2024-03-29T09:54:27.465912+0000","last_unstale":"2024-03-29T09:54:28.674044+0000","last_undegraded":"2024-03-29T09:54:28.674044+0000","last_fullsized":"2024-03-29T09:54:28.674044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438778+0000","last_change":"2024-03-29T09:54:27.447415+0000","last_active":"2024-03-29T09:54:28.438778+0000","last_peered":"2024-03-29T09:54:28.438778+0000","last_clean":"2024-03-29T09:54:28.438778+0000","last_became_active":"2024-03-29T09:54:27.446985+0000","last_became_peered":"2024-03-29T09:54:27.446985+0000","last_unstale":"2024-03-29T09:54:28.438778+0000","last_undegraded":"2024-03-29T09:54:28.438778+0000","last_fullsized":"2024-03-29T09:54:28.438778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4080000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1960000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8279999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":806912,"data_stored":660885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T09:54:32.828 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T09:54:32.828 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-29T09:54:32.829 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-29T09:54:32.829 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-29T09:54:32.829 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-29T09:54:33.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.062+0000 7f279a000700 1 Processor -- start 2024-03-29T09:54:33.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.062+0000 7f279a000700 1 -- start start 2024-03-29T09:54:33.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.063+0000 7f279a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2794160f10 0x7f27940d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.063+0000 7f279a000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27940ceda0 con 0x7f2794161590 2024-03-29T09:54:33.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.063+0000 7f279a000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27941612f0 con 0x7f27940cf230 2024-03-29T09:54:33.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.063+0000 7f279a000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27940ce470 con 0x7f2794160f10 2024-03-29T09:54:33.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.063+0000 7f2793400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2794160f10 0x7f27940d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:33.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.063+0000 7f2793e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2794161590 0x7f27940b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58098/0 (socket says 172.21.15.12:58098) 2024-03-29T09:54:33.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.063+0000 7f2793e00700 1 -- 172.21.15.12:0/3848604089 learned_addr learned my addr 172.21.15.12:0/3848604089 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:33.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.064+0000 7f2793400700 1 -- 172.21.15.12:0/3848604089 >> v1:172.21.15.159:6789/0 conn(0x7f27940cf230 legacy=0x7f27940d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.064+0000 7f2792000700 1 -- 172.21.15.12:0/3848604089 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048443909 0 0) 0x7f27940ceda0 con 0x7f2794161590 2024-03-29T09:54:33.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.064+0000 7f2793400700 1 -- 172.21.15.12:0/3848604089 >> v1:172.21.15.12:6789/0 conn(0x7f2794161590 legacy=0x7f27940b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.064+0000 7f2793400700 1 -- 172.21.15.12:0/3848604089 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27940ce750 con 0x7f2794160f10 2024-03-29T09:54:33.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.064+0000 7f2793400700 1 --2- 172.21.15.12:0/3848604089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2794160f10 0x7f27940d18b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2784009840 tx=0x7f2784009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c556a80163086e32 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:33.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.064+0000 7f2792000700 1 -- 172.21.15.12:0/3848604089 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784011830 con 0x7f2794160f10 2024-03-29T09:54:33.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.064+0000 7f2792000700 1 -- 172.21.15.12:0/3848604089 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2784011990 con 0x7f2794160f10 2024-03-29T09:54:33.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.064+0000 7f2792000700 1 -- 172.21.15.12:0/3848604089 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784011d20 con 0x7f2794160f10 2024-03-29T09:54:33.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.065+0000 7f279a000700 1 -- 172.21.15.12:0/3848604089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2794160f10 msgr2=0x7f27940d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.065+0000 7f279a000700 1 --2- 172.21.15.12:0/3848604089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2794160f10 0x7f27940d18b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2784009840 tx=0x7f2784009b50 comp rx=0 tx=0).stop 2024-03-29T09:54:33.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.065+0000 7f279a000700 1 -- 172.21.15.12:0/3848604089 shutdown_connections 2024-03-29T09:54:33.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.065+0000 7f279a000700 1 --2- 172.21.15.12:0/3848604089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2794160f10 0x7f27940d18b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.065+0000 7f279a000700 1 -- 172.21.15.12:0/3848604089 >> 172.21.15.12:0/3848604089 conn(0x7f279408aff0 msgr2=0x7f27940d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:33.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.065+0000 7f279a000700 1 -- 172.21.15.12:0/3848604089 shutdown_connections 2024-03-29T09:54:33.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.065+0000 7f279a000700 1 -- 172.21.15.12:0/3848604089 wait complete. 2024-03-29T09:54:33.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.067+0000 7f279a000700 1 Processor -- start 2024-03-29T09:54:33.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.067+0000 7f279a000700 1 -- start start 2024-03-29T09:54:33.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f279a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27940cf230 0x7f2794158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f279a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2794160f10 0x7f279415b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f279a000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2794161590 0x7f279415ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f279a000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27941e7040 con 0x7f27940cf230 2024-03-29T09:54:33.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f279a000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2794159260 con 0x7f2794161590 2024-03-29T09:54:33.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f279a000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f279415df40 con 0x7f2794160f10 2024-03-29T09:54:33.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f2793400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27940cf230 0x7f2794158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f2793400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27940cf230 0x7f2794158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40530/0 (socket says 172.21.15.12:40530) 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f2793400700 1 -- 172.21.15.12:0/3441779501 learned_addr learned my addr 172.21.15.12:0/3441779501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f2793400700 1 -- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2794160f10 msgr2=0x7f279415b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f2793400700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2794160f10 0x7f279415b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f2793400700 1 -- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2794161590 msgr2=0x7f279415ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f2793400700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2794161590 0x7f279415ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.068+0000 7f2793400700 1 -- 172.21.15.12:0/3441779501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27840094a0 con 0x7f27940cf230 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.069+0000 7f2793400700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27940cf230 0x7f2794158d20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27840090b0 tx=0x7f2784004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.069+0000 7f278b400700 1 -- 172.21.15.12:0/3441779501 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278401a9c0 con 0x7f27940cf230 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.069+0000 7f278b400700 1 -- 172.21.15.12:0/3441779501 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f278401ab20 con 0x7f27940cf230 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.069+0000 7f279a000700 1 -- 172.21.15.12:0/3441779501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279415e220 con 0x7f27940cf230 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.069+0000 7f278b400700 1 -- 172.21.15.12:0/3441779501 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784024420 con 0x7f27940cf230 2024-03-29T09:54:33.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.069+0000 7f279a000700 1 -- 172.21.15.12:0/3441779501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279415e7e0 con 0x7f27940cf230 2024-03-29T09:54:33.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.070+0000 7f279a000700 1 -- 172.21.15.12:0/3441779501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2794067720 con 0x7f27940cf230 2024-03-29T09:54:33.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.071+0000 7f278b400700 1 -- 172.21.15.12:0/3441779501 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2784022070 con 0x7f27940cf230 2024-03-29T09:54:33.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.072+0000 7f278b400700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f277406ac90 0x7f277406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.072+0000 7f278b400700 1 -- 172.21.15.12:0/3441779501 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f278408a790 con 0x7f27940cf230 2024-03-29T09:54:33.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.072+0000 7f2792a00700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f277406ac90 0x7f277406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:33.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.073+0000 7f2792a00700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f277406ac90 0x7f277406d150 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f27941f6880 tx=0x7f2780007a40 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:33.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.076+0000 7f278b400700 1 -- 172.21.15.12:0/3441779501 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2784055960 con 0x7f27940cf230 2024-03-29T09:54:33.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.349+0000 7f279a000700 1 -- 172.21.15.12:0/3441779501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f279415e9d0 con 0x7f27940cf230 2024-03-29T09:54:33.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.352+0000 7f278b400700 1 -- 172.21.15.12:0/3441779501 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170932 (secure 0 0 0) 0x7f2784029090 con 0x7f27940cf230 2024-03-29T09:54:33.354 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:33.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.355+0000 7f279a000700 1 -- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f277406ac90 msgr2=0x7f277406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f277406ac90 0x7f277406d150 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f27941f6880 tx=0x7f2780007a40 comp rx=0 tx=0).stop 2024-03-29T09:54:33.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 -- 172.21.15.12:0/3441779501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27940cf230 msgr2=0x7f2794158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27940cf230 0x7f2794158d20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27840090b0 tx=0x7f2784004750 comp rx=0 tx=0).stop 2024-03-29T09:54:33.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 -- 172.21.15.12:0/3441779501 shutdown_connections 2024-03-29T09:54:33.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f277406ac90 0x7f277406d150 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2794161590 0x7f279415ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27940cf230 0x7f2794158d20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 --2- 172.21.15.12:0/3441779501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2794160f10 0x7f279415b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 -- 172.21.15.12:0/3441779501 >> 172.21.15.12:0/3441779501 conn(0x7f279408aff0 msgr2=0x7f27940ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:33.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 -- 172.21.15.12:0/3441779501 shutdown_connections 2024-03-29T09:54:33.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.356+0000 7f279a000700 1 -- 172.21.15.12:0/3441779501 wait complete. 2024-03-29T09:54:33.373 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"active_gid":4114,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":43444},{"type":"v1","addr":"172.21.15.159:6833","nonce":43444}]},"active_addr":"172.21.15.159:6833/43444","active_change":"2024-03-29T09:54:10.422221+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4100,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":249193272}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":1756693282}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":795330822}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":3395907487}]}]} 2024-03-29T09:54:33.378 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-29T09:54:33.378 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-29T09:54:33.378 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:33.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.575+0000 7fad59200700 1 Processor -- start 2024-03-29T09:54:33.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.575+0000 7fad59200700 1 -- start start 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.576+0000 7fad59200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 0x7fad540d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad59200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fad54161590 0x7fad54161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad59200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad540ceda0 con 0x7fad540cf230 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad59200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad54161f80 con 0x7fad54161590 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad59200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad541612f0 con 0x7fad54160f10 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad52800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fad54161590 0x7fad54161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad52800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fad54161590 0x7fad54161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40542/0 (socket says 172.21.15.12:40542) 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad52800700 1 -- 172.21.15.12:0/3645574906 learned_addr learned my addr 172.21.15.12:0/3645574906 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad51e00700 1 --2- 172.21.15.12:0/3645574906 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 0x7fad540d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad52800700 1 -- 172.21.15.12:0/3645574906 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 msgr2=0x7fad540d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad52800700 1 --2- 172.21.15.12:0/3645574906 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 0x7fad540d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad52800700 1 -- 172.21.15.12:0/3645574906 >> v1:172.21.15.159:6789/0 conn(0x7fad540cf230 legacy=0x7fad540d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad52800700 1 -- 172.21.15.12:0/3645574906 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540ce470 con 0x7fad54161590 2024-03-29T09:54:33.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.577+0000 7fad51e00700 1 --2- 172.21.15.12:0/3645574906 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 0x7fad540d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:33.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad52800700 1 --2- 172.21.15.12:0/3645574906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad54161590 0x7fad54161a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fad4c009e80 tx=0x7fad4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4439a75f3ac72f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad50a00700 1 -- 172.21.15.12:0/3645574906 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c00e070 con 0x7fad54161590 2024-03-29T09:54:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad50a00700 1 -- 172.21.15.12:0/3645574906 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c00b980 con 0x7fad54161590 2024-03-29T09:54:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad50a00700 1 -- 172.21.15.12:0/3645574906 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c00bd30 con 0x7fad54161590 2024-03-29T09:54:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad59200700 1 -- 172.21.15.12:0/3645574906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad54161590 msgr2=0x7fad54161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad59200700 1 --2- 172.21.15.12:0/3645574906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad54161590 0x7fad54161a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fad4c009e80 tx=0x7fad4c00b330 comp rx=0 tx=0).stop 2024-03-29T09:54:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad59200700 1 -- 172.21.15.12:0/3645574906 shutdown_connections 2024-03-29T09:54:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad59200700 1 --2- 172.21.15.12:0/3645574906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad54161590 0x7fad54161a40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad59200700 1 --2- 172.21.15.12:0/3645574906 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 0x7fad540d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad59200700 1 -- 172.21.15.12:0/3645574906 >> 172.21.15.12:0/3645574906 conn(0x7fad5408aff0 msgr2=0x7fad540d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.578+0000 7fad59200700 1 -- 172.21.15.12:0/3645574906 shutdown_connections 2024-03-29T09:54:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.579+0000 7fad59200700 1 -- 172.21.15.12:0/3645574906 wait complete. 2024-03-29T09:54:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.580+0000 7fad59200700 1 Processor -- start 2024-03-29T09:54:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.580+0000 7fad59200700 1 -- start start 2024-03-29T09:54:33.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.581+0000 7fad59200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad540cf230 0x7fad541587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.581+0000 7fad59200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 0x7fad5415f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.581+0000 7fad59200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad54161590 0x7fad5415fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.581+0000 7fad59200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad540b71d0 con 0x7fad540cf230 2024-03-29T09:54:33.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.581+0000 7fad59200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5415e540 con 0x7fad54161590 2024-03-29T09:54:33.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.581+0000 7fad59200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5415e820 con 0x7fad54160f10 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.581+0000 7fad51e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad540cf230 0x7fad541587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.581+0000 7fad51e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad540cf230 0x7fad541587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40548/0 (socket says 172.21.15.12:40548) 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.581+0000 7fad51e00700 1 -- 172.21.15.12:0/2458913268 learned_addr learned my addr 172.21.15.12:0/2458913268 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad52800700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad54161590 0x7fad5415fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad51400700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 0x7fad5415f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad51e00700 1 -- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 msgr2=0x7fad5415f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad51e00700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 0x7fad5415f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad51e00700 1 -- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad54161590 msgr2=0x7fad5415fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad51e00700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad54161590 0x7fad5415fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad51e00700 1 -- 172.21.15.12:0/2458913268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c009b30 con 0x7fad540cf230 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad51e00700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad540cf230 0x7fad541587d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fad4400e760 tx=0x7fad4400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad4a000700 1 -- 172.21.15.12:0/2458913268 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44013610 con 0x7fad540cf230 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad4a000700 1 -- 172.21.15.12:0/2458913268 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad44013770 con 0x7fad540cf230 2024-03-29T09:54:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad4a000700 1 -- 172.21.15.12:0/2458913268 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44013b20 con 0x7fad540cf230 2024-03-29T09:54:33.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad59200700 1 -- 172.21.15.12:0/2458913268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5415eb00 con 0x7fad540cf230 2024-03-29T09:54:33.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.582+0000 7fad59200700 1 -- 172.21.15.12:0/2458913268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5415e160 con 0x7fad540cf230 2024-03-29T09:54:33.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.583+0000 7fad59200700 1 -- 172.21.15.12:0/2458913268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad540b74b0 con 0x7fad540cf230 2024-03-29T09:54:33.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.584+0000 7fad4a000700 1 -- 172.21.15.12:0/2458913268 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fad44013c80 con 0x7fad540cf230 2024-03-29T09:54:33.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.585+0000 7fad4a000700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad3406ac90 0x7fad3406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:33.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.585+0000 7fad4a000700 1 -- 172.21.15.12:0/2458913268 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad44088f80 con 0x7fad540cf230 2024-03-29T09:54:33.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.585+0000 7fad51400700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad3406ac90 0x7fad3406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:33.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.586+0000 7fad51400700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad3406ac90 0x7fad3406d150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fad5415b770 tx=0x7fad40009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:33.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.589+0000 7fad4a000700 1 -- 172.21.15.12:0/2458913268 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fad440540c0 con 0x7fad540cf230 2024-03-29T09:54:33.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.802+0000 7fad59200700 1 -- 172.21.15.12:0/2458913268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad54067720 con 0x7fad540cf230 2024-03-29T09:54:33.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.803+0000 7fad4a000700 1 -- 172.21.15.12:0/2458913268 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7fad440576e0 con 0x7fad540cf230 2024-03-29T09:54:33.803 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:33.803 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:33.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.805+0000 7fad59200700 1 -- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad3406ac90 msgr2=0x7fad3406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.805+0000 7fad59200700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad3406ac90 0x7fad3406d150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fad5415b770 tx=0x7fad40009040 comp rx=0 tx=0).stop 2024-03-29T09:54:33.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.806+0000 7fad59200700 1 -- 172.21.15.12:0/2458913268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad540cf230 msgr2=0x7fad541587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:33.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.806+0000 7fad59200700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad540cf230 0x7fad541587d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fad4400e760 tx=0x7fad4400eb20 comp rx=0 tx=0).stop 2024-03-29T09:54:33.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.806+0000 7fad59200700 1 -- 172.21.15.12:0/2458913268 shutdown_connections 2024-03-29T09:54:33.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.806+0000 7fad59200700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad3406ac90 0x7fad3406d150 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.806+0000 7fad59200700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad54161590 0x7fad5415fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.806+0000 7fad59200700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad540cf230 0x7fad541587d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.806+0000 7fad59200700 1 --2- 172.21.15.12:0/2458913268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad54160f10 0x7fad5415f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:33.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.806+0000 7fad59200700 1 -- 172.21.15.12:0/2458913268 >> 172.21.15.12:0/2458913268 conn(0x7fad5408aff0 msgr2=0x7fad540cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:33.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.806+0000 7fad59200700 1 -- 172.21.15.12:0/2458913268 shutdown_connections 2024-03-29T09:54:33.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:33.806+0000 7fad59200700 1 -- 172.21.15.12:0/2458913268 wait complete. 2024-03-29T09:54:33.823 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-29T09:54:33.823 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:34.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.022+0000 7f0a7ea00700 1 Processor -- start 2024-03-29T09:54:34.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.022+0000 7f0a7ea00700 1 -- start start 2024-03-29T09:54:34.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.023+0000 7f0a7ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a78160f10 0x7f0a780d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.023+0000 7f0a7ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0a780cf230 0x7f0a780d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.023+0000 7f0a7ea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a780ceda0 con 0x7f0a78161590 2024-03-29T09:54:34.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.023+0000 7f0a7ea00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a780d23d0 con 0x7f0a780cf230 2024-03-29T09:54:34.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.023+0000 7f0a7ea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a781612f0 con 0x7f0a78160f10 2024-03-29T09:54:34.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.023+0000 7f0a7ca00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0a78161590 0x7f0a780b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58106/0 (socket says 172.21.15.12:58106) 2024-03-29T09:54:34.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.023+0000 7f0a7ca00700 1 -- 172.21.15.12:0/3367506937 learned_addr learned my addr 172.21.15.12:0/3367506937 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984121522 0 0) 0x7f0a780ceda0 con 0x7f0a78161590 2024-03-29T09:54:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a77400700 1 --2- 172.21.15.12:0/3367506937 >> v2:172.21.15.159:3300/0 conn(0x7f0a780cf230 0x7f0a780d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a77e00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a78160f10 0x7f0a780d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a5c0035b0 con 0x7f0a78161590 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1499660010 0 0) 0x7f0a5c0035b0 con 0x7f0a78161590 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a78160f10 msgr2=0x7f0a780d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a78160f10 0x7f0a780d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 >> v2:172.21.15.159:3300/0 conn(0x7f0a780cf230 msgr2=0x7f0a780d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 --2- 172.21.15.12:0/3367506937 >> v2:172.21.15.159:3300/0 conn(0x7f0a780cf230 0x7f0a780d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a780ce470 con 0x7f0a78161590 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0a700019f0 con 0x7f0a78161590 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 >> v1:172.21.15.12:6789/0 conn(0x7f0a78161590 legacy=0x7f0a780b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c003a00 0x7f0a5c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a78160f10 0x7f0a5c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c004d90 0x7f0a5c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a780d23d0 con 0x7f0a5c003a00 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a781612f0 con 0x7f0a5c004d90 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c00ba10 con 0x7f0a78160f10 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.024+0000 7f0a77e00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c003a00 0x7f0a5c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a7ca00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c004d90 0x7f0a5c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a77400700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a78160f10 0x7f0a5c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a77e00700 1 -- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a78160f10 msgr2=0x7f0a5c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a77e00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a78160f10 0x7f0a5c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a77e00700 1 -- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c004d90 msgr2=0x7f0a5c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a77e00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c004d90 0x7f0a5c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a77e00700 1 -- 172.21.15.12:0/3367506937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c0035b0 con 0x7f0a5c003a00 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a7ca00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c004d90 0x7f0a5c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a77e00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c003a00 0x7f0a5c003e20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009b60 tx=0x7f0a5c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6402ffdff1cf4595 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6800e6c0 con 0x7f0a5c003a00 2024-03-29T09:54:34.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6800e820 con 0x7f0a5c003a00 2024-03-29T09:54:34.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.025+0000 7f0a76a00700 1 -- 172.21.15.12:0/3367506937 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6800eb90 con 0x7f0a5c003a00 2024-03-29T09:54:34.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.026+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3367506937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c003a00 msgr2=0x7f0a5c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.026+0000 7f0a7ea00700 1 --2- 172.21.15.12:0/3367506937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c003a00 0x7f0a5c003e20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009b60 tx=0x7f0a5c005880 comp rx=0 tx=0).stop 2024-03-29T09:54:34.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.026+0000 7f0a7ca00700 1 -- 172.21.15.12:0/3367506937 reap_dead start 2024-03-29T09:54:34.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.026+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3367506937 shutdown_connections 2024-03-29T09:54:34.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.026+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3367506937 >> 172.21.15.12:0/3367506937 conn(0x7f0a7808aff0 msgr2=0x7f0a780d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.026+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3367506937 shutdown_connections 2024-03-29T09:54:34.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.026+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3367506937 wait complete. 2024-03-29T09:54:34.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.028+0000 7f0a7ea00700 1 Processor -- start 2024-03-29T09:54:34.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.028+0000 7f0a7ea00700 1 -- start start 2024-03-29T09:54:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c003a00 0x7f0a781529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a781f3d60 0x7f0a780b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a780cc1a0 0x7f0a780cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a781f3350 con 0x7f0a780cc1a0 2024-03-29T09:54:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a780ccb70 con 0x7f0a5c003a00 2024-03-29T09:54:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a781565f0 con 0x7f0a781f3d60 2024-03-29T09:54:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a780cc1a0 0x7f0a780cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a780cc1a0 0x7f0a780cc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40564/0 (socket says 172.21.15.12:40564) 2024-03-29T09:54:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ca00700 1 -- 172.21.15.12:0/3419357752 learned_addr learned my addr 172.21.15.12:0/3419357752 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a77e00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c003a00 0x7f0a781529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a77400700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a781f3d60 0x7f0a780b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ca00700 1 -- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a781f3d60 msgr2=0x7f0a780b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ca00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a781f3d60 0x7f0a780b3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ca00700 1 -- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c003a00 msgr2=0x7f0a781529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ca00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c003a00 0x7f0a781529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a7ca00700 1 -- 172.21.15.12:0/3419357752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a78157030 con 0x7f0a780cc1a0 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.029+0000 7f0a77400700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a781f3d60 0x7f0a780b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.030+0000 7f0a7ca00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a780cc1a0 0x7f0a780cc630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0a70000f80 tx=0x7f0a7000e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.030+0000 7f0a6fe00700 1 -- 172.21.15.12:0/3419357752 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7000ecd0 con 0x7f0a780cc1a0 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.030+0000 7f0a6fe00700 1 -- 172.21.15.12:0/3419357752 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7000ee50 con 0x7f0a780cc1a0 2024-03-29T09:54:34.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.030+0000 7f0a6fe00700 1 -- 172.21.15.12:0/3419357752 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a70013660 con 0x7f0a780cc1a0 2024-03-29T09:54:34.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.030+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3419357752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a78157310 con 0x7f0a780cc1a0 2024-03-29T09:54:34.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.030+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3419357752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a781f3560 con 0x7f0a780cc1a0 2024-03-29T09:54:34.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.031+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3419357752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a78157830 con 0x7f0a780cc1a0 2024-03-29T09:54:34.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.032+0000 7f0a6fe00700 1 -- 172.21.15.12:0/3419357752 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0a700137c0 con 0x7f0a780cc1a0 2024-03-29T09:54:34.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.033+0000 7f0a6fe00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a5806ac90 0x7f0a5806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.033+0000 7f0a6fe00700 1 -- 172.21.15.12:0/3419357752 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a70089ad0 con 0x7f0a780cc1a0 2024-03-29T09:54:34.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.033+0000 7f0a77e00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a5806ac90 0x7f0a5806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.034+0000 7f0a77e00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a5806ac90 0x7f0a5806d150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a680090b0 tx=0x7f0a6801f040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.037+0000 7f0a6fe00700 1 -- 172.21.15.12:0/3419357752 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0a78157830 con 0x7f0a780cc1a0 2024-03-29T09:54:34.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.252+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3419357752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a78157830 con 0x7f0a780cc1a0 2024-03-29T09:54:34.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.253+0000 7f0a6fe00700 1 -- 172.21.15.12:0/3419357752 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f0a70081e60 con 0x7f0a780cc1a0 2024-03-29T09:54:34.253 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:34.253 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:34.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a5806ac90 msgr2=0x7f0a5806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a5806ac90 0x7f0a5806d150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a680090b0 tx=0x7f0a6801f040 comp rx=0 tx=0).stop 2024-03-29T09:54:34.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3419357752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a780cc1a0 msgr2=0x7f0a780cc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a780cc1a0 0x7f0a780cc630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0a70000f80 tx=0x7f0a7000e440 comp rx=0 tx=0).stop 2024-03-29T09:54:34.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3419357752 shutdown_connections 2024-03-29T09:54:34.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a5806ac90 0x7f0a5806d150 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a780cc1a0 0x7f0a780cc630 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c003a00 0x7f0a781529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 --2- 172.21.15.12:0/3419357752 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a781f3d60 0x7f0a780b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3419357752 >> 172.21.15.12:0/3419357752 conn(0x7f0a7808aff0 msgr2=0x7f0a780b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.255+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3419357752 shutdown_connections 2024-03-29T09:54:34.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.256+0000 7f0a7ea00700 1 -- 172.21.15.12:0/3419357752 wait complete. 2024-03-29T09:54:34.274 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-29T09:54:34.274 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-29T09:54:34.275 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-29T09:54:34.275 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-29T09:54:34.275 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-29T09:54:34.276 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-29T09:54:34.276 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-29T09:54:34.277 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-29T09:54:34.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.643+0000 7fd6d5400700 1 Processor -- start 2024-03-29T09:54:34.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.643+0000 7fd6d5400700 1 -- start start 2024-03-29T09:54:34.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.645+0000 7fd6d5400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6d015bcf0 0x7fd6d006c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.647+0000 7fd6ce800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6d015bcf0 0x7fd6d006c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.647+0000 7fd6ce800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6d015bcf0 0x7fd6d006c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40568/0 (socket says 172.21.15.12:40568) 2024-03-29T09:54:34.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.647+0000 7fd6cde00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd6d0161f60 0x7fd6d0070750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58116/0 (socket says 172.21.15.12:58116) 2024-03-29T09:54:34.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.648+0000 7f1610400700 1 Processor -- start 2024-03-29T09:54:34.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.648+0000 7f1610400700 1 -- start start 2024-03-29T09:54:34.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.646+0000 7f3118a00700 1 Processor -- start 2024-03-29T09:54:34.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.646+0000 7f3118a00700 1 -- start start 2024-03-29T09:54:34.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.648+0000 7f3118a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f31141657c0 0x7f3114165c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.648+0000 7f3118a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f31140d1a00 0x7f31140d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.648+0000 7f3118a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31140d1570 con 0x7f311415f830 2024-03-29T09:54:34.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.648+0000 7f3118a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31141661b0 con 0x7f31141657c0 2024-03-29T09:54:34.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.648+0000 7f3118a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311415fc10 con 0x7f31140d1a00 2024-03-29T09:54:34.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7f3111400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f31141657c0 0x7f3114165c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7fe426a00700 1 Processor -- start 2024-03-29T09:54:34.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7fd6d5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0070e60 0x7fd6d00b2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7fd6d5400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d006d080 con 0x7fd6d0161f60 2024-03-29T09:54:34.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7fd6d5400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d006d360 con 0x7fd6d015bcf0 2024-03-29T09:54:34.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7fd6d5400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d006d640 con 0x7fd6d0070e60 2024-03-29T09:54:34.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7fe426a00700 1 -- start start 2024-03-29T09:54:34.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7f3112800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f31140d1a00 0x7f31140d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7f3112800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f31140d1a00 0x7f31140d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46698/0 (socket says 172.21.15.12:46698) 2024-03-29T09:54:34.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7f3112800700 1 -- 172.21.15.12:0/319610336 learned_addr learned my addr 172.21.15.12:0/319610336 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7fd6ce800700 1 -- 172.21.15.12:0/575784305 learned_addr learned my addr 172.21.15.12:0/575784305 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.650+0000 7f3112800700 1 -- 172.21.15.12:0/319610336 >> v2:172.21.15.12:3300/0 conn(0x7f31141657c0 msgr2=0x7f3114165c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7f1610400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f16081597f0 0x7f16080b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.649+0000 7f1610400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160806dc60 0x7f160806e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.650+0000 7f3112800700 1 --2- 172.21.15.12:0/319610336 >> v2:172.21.15.12:3300/0 conn(0x7f31141657c0 0x7f3114165c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.650+0000 7f3112800700 1 -- 172.21.15.12:0/319610336 >> v1:172.21.15.159:6789/0 conn(0x7f311415f830 legacy=0x7f31140cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.650+0000 7f3112800700 1 -- 172.21.15.12:0/319610336 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31140d06b0 con 0x7f31140d1a00 2024-03-29T09:54:34.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7f3111400700 1 --2- 172.21.15.12:0/319610336 >> v2:172.21.15.12:3300/0 conn(0x7f31141657c0 0x7f3114165c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:34.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7fd6cf200700 1 --2- 172.21.15.12:0/575784305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0070e60 0x7fd6d00b2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7fd6ce800700 1 -- 172.21.15.12:0/575784305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0070e60 msgr2=0x7fd6d00b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.650+0000 7f160da00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f16081597f0 0x7f16080b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7f1610400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16080b2970 con 0x7f160815d4d0 2024-03-29T09:54:34.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7f1610400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16080b2780 con 0x7f16081597f0 2024-03-29T09:54:34.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7f1610400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16080b2c50 con 0x7f160806dc60 2024-03-29T09:54:34.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7f160d000700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f160815d4d0 0x7f16080b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46408/0 (socket says 172.21.15.12:46408) 2024-03-29T09:54:34.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7f160d000700 1 -- 172.21.15.12:0/859524802 learned_addr learned my addr 172.21.15.12:0/859524802 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:34.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7f160e400700 1 --2- 172.21.15.12:0/859524802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160806dc60 0x7f160806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7f3112800700 1 --2- 172.21.15.12:0/319610336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31140d1a00 0x7f31140d1e90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31140d14e0 tx=0x7f310c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c2e646162c7473e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7fd6ce800700 1 --2- 172.21.15.12:0/575784305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0070e60 0x7fd6d00b2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7fd6ce800700 1 -- 172.21.15.12:0/575784305 >> v1:172.21.15.12:6789/0 conn(0x7fd6d0161f60 legacy=0x7fd6d0070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7fd6ce800700 1 -- 172.21.15.12:0/575784305 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d006d920 con 0x7fd6d015bcf0 2024-03-29T09:54:34.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7f1607e00700 1 -- 172.21.15.12:0/859524802 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364819385 0 0) 0x7f16080b2970 con 0x7f160815d4d0 2024-03-29T09:54:34.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.651+0000 7f1607e00700 1 -- 172.21.15.12:0/859524802 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f00035b0 con 0x7f160815d4d0 2024-03-29T09:54:34.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.652+0000 7fd6cf200700 1 --2- 172.21.15.12:0/575784305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0070e60 0x7fd6d00b2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:34.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.652+0000 7f160e400700 1 -- 172.21.15.12:0/859524802 >> v2:172.21.15.159:3300/0 conn(0x7f16081597f0 msgr2=0x7f16080b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.652+0000 7f160e400700 1 --2- 172.21.15.12:0/859524802 >> v2:172.21.15.159:3300/0 conn(0x7f16081597f0 0x7f16080b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.652+0000 7f160e400700 1 -- 172.21.15.12:0/859524802 >> v1:172.21.15.159:6789/0 conn(0x7f160815d4d0 legacy=0x7f16080b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.652+0000 7f160e400700 1 -- 172.21.15.12:0/859524802 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16080b2f30 con 0x7f160806dc60 2024-03-29T09:54:34.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.652+0000 7fe426a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe41803e950 0x7fe418006c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7fe426a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe418057330 con 0x7fe4180543a0 2024-03-29T09:54:34.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7fe426a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe418007310 con 0x7fe4180e3240 2024-03-29T09:54:34.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7fe426a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe418007bb0 con 0x7fe41803e950 2024-03-29T09:54:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.652+0000 7fd6ce800700 1 --2- 172.21.15.12:0/575784305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d015bcf0 0x7fd6d006c710 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80098f0 tx=0x7fd6c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b83840720951314 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7f3110a00700 1 -- 172.21.15.12:0/319610336 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c00e070 con 0x7f31140d1a00 2024-03-29T09:54:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7f160e400700 1 --2- 172.21.15.12:0/859524802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160806dc60 0x7f160806e110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1600009d90 tx=0x7f160000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d01fd469f2e90a9d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7f3110a00700 1 -- 172.21.15.12:0/319610336 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c003a20 con 0x7f31140d1a00 2024-03-29T09:54:34.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7f3110a00700 1 -- 172.21.15.12:0/319610336 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c003dd0 con 0x7f31140d1a00 2024-03-29T09:54:34.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7fe425600700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fe4180e3240 0x7fe418053c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46418/0 (socket says 172.21.15.12:46418) 2024-03-29T09:54:34.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7f1607e00700 1 -- 172.21.15.12:0/859524802 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160000e070 con 0x7f160806dc60 2024-03-29T09:54:34.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7f1607e00700 1 -- 172.21.15.12:0/859524802 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160000bb20 con 0x7f160806dc60 2024-03-29T09:54:34.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7f1607e00700 1 -- 172.21.15.12:0/859524802 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1600012430 con 0x7f160806dc60 2024-03-29T09:54:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7fe425600700 1 -- 172.21.15.12:0/3190459714 learned_addr learned my addr 172.21.15.12:0/3190459714 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7fd6cd400700 1 -- 172.21.15.12:0/575784305 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c800b8a0 con 0x7fd6d015bcf0 2024-03-29T09:54:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7fd6cd400700 1 -- 172.21.15.12:0/575784305 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c800ba00 con 0x7fd6d015bcf0 2024-03-29T09:54:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.653+0000 7fd6cd400700 1 -- 172.21.15.12:0/575784305 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c800bd90 con 0x7fd6d015bcf0 2024-03-29T09:54:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.654+0000 7fe426000700 1 --2- 172.21.15.12:0/3190459714 >> v2:172.21.15.159:3300/0 conn(0x7fe41803e950 0x7fe418006c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.654+0000 7f3118a00700 1 -- 172.21.15.12:0/319610336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31140d1a00 msgr2=0x7f31140d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.654+0000 7f3118a00700 1 --2- 172.21.15.12:0/319610336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31140d1a00 0x7f31140d1e90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31140d14e0 tx=0x7f310c00b330 comp rx=0 tx=0).stop 2024-03-29T09:54:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.654+0000 7fe41fe00700 1 -- 172.21.15.12:0/3190459714 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030304702 0 0) 0x7fe418007310 con 0x7fe4180e3240 2024-03-29T09:54:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.654+0000 7fe41fe00700 1 -- 172.21.15.12:0/3190459714 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4080035b0 con 0x7fe4180e3240 2024-03-29T09:54:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.654+0000 7f1610400700 1 -- 172.21.15.12:0/859524802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160806dc60 msgr2=0x7f160806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.654+0000 7f3118a00700 1 -- 172.21.15.12:0/319610336 shutdown_connections 2024-03-29T09:54:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.654+0000 7f3118a00700 1 --2- 172.21.15.12:0/319610336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31140d1a00 0x7f31140d1e90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.654+0000 7f3118a00700 1 --2- 172.21.15.12:0/319610336 >> v2:172.21.15.12:3300/0 conn(0x7f31141657c0 0x7f3114165c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.654+0000 7f3118a00700 1 -- 172.21.15.12:0/319610336 >> 172.21.15.12:0/319610336 conn(0x7f311408aff0 msgr2=0x7f31140d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7f1610400700 1 --2- 172.21.15.12:0/859524802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160806dc60 0x7f160806e110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1600009d90 tx=0x7f160000b330 comp rx=0 tx=0).stop 2024-03-29T09:54:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7f1610400700 1 -- 172.21.15.12:0/859524802 shutdown_connections 2024-03-29T09:54:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7f3118a00700 1 -- 172.21.15.12:0/319610336 shutdown_connections 2024-03-29T09:54:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7f1610400700 1 --2- 172.21.15.12:0/859524802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f160806dc60 0x7f160806e110 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7fe426000700 1 -- 172.21.15.12:0/3190459714 >> v1:172.21.15.159:6789/0 conn(0x7fe4180e3240 legacy=0x7fe418053c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7f1610400700 1 --2- 172.21.15.12:0/859524802 >> v2:172.21.15.159:3300/0 conn(0x7f16081597f0 0x7f16080b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7f1610400700 1 -- 172.21.15.12:0/859524802 >> 172.21.15.12:0/859524802 conn(0x7f16080b0d70 msgr2=0x7f16080b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7fe41fe00700 1 -- 172.21.15.12:0/3190459714 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1388151998 0 0) 0x7fe4080035b0 con 0x7fe4180e3240 2024-03-29T09:54:34.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7fe426000700 1 -- 172.21.15.12:0/3190459714 >> v1:172.21.15.12:6789/0 conn(0x7fe4180543a0 legacy=0x7fe4180577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7fe426000700 1 -- 172.21.15.12:0/3190459714 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4180075f0 con 0x7fe41803e950 2024-03-29T09:54:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.656+0000 7f1610400700 1 -- 172.21.15.12:0/859524802 shutdown_connections 2024-03-29T09:54:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.656+0000 7fe426000700 1 --2- 172.21.15.12:0/3190459714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe41803e950 0x7fe418006c10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe42006a090 tx=0x7fe420068df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0ab570b7f15725b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.656+0000 7f23bda00700 1 Processor -- start 2024-03-29T09:54:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7fd6d5400700 1 -- 172.21.15.12:0/575784305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d015bcf0 msgr2=0x7fd6d006c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7fd6d5400700 1 --2- 172.21.15.12:0/575784305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d015bcf0 0x7fd6d006c710 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80098f0 tx=0x7fd6c8009c00 comp rx=0 tx=0).stop 2024-03-29T09:54:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7fd6d5400700 1 -- 172.21.15.12:0/575784305 shutdown_connections 2024-03-29T09:54:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7fd6d5400700 1 --2- 172.21.15.12:0/575784305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0070e60 0x7fd6d00b2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7fd6d5400700 1 --2- 172.21.15.12:0/575784305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d015bcf0 0x7fd6d006c710 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80098f0 tx=0x7fd6c8009c00 comp rx=0 tx=0).stop 2024-03-29T09:54:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7fd6d5400700 1 -- 172.21.15.12:0/575784305 >> 172.21.15.12:0/575784305 conn(0x7fd6d00b0c70 msgr2=0x7fd6d00b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.655+0000 7f3118a00700 1 -- 172.21.15.12:0/319610336 wait complete. 2024-03-29T09:54:34.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.657+0000 7fd6d5400700 1 -- 172.21.15.12:0/575784305 shutdown_connections 2024-03-29T09:54:34.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.657+0000 7fe41fe00700 1 -- 172.21.15.12:0/3190459714 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414001670 con 0x7fe41803e950 2024-03-29T09:54:34.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.657+0000 7fe41fe00700 1 -- 172.21.15.12:0/3190459714 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42004f2d0 con 0x7fe41803e950 2024-03-29T09:54:34.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.657+0000 7fe41fe00700 1 -- 172.21.15.12:0/3190459714 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe420067a00 con 0x7fe41803e950 2024-03-29T09:54:34.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.658+0000 7fe426a00700 1 -- 172.21.15.12:0/3190459714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe41803e950 msgr2=0x7fe418006c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.658+0000 7f3118a00700 1 Processor -- start 2024-03-29T09:54:34.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.658+0000 7fe426a00700 1 --2- 172.21.15.12:0/3190459714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe41803e950 0x7fe418006c10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe42006a090 tx=0x7fe420068df0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.659+0000 7f3118a00700 1 -- start start 2024-03-29T09:54:34.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.656+0000 7f1610400700 1 -- 172.21.15.12:0/859524802 wait complete. 2024-03-29T09:54:34.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.659+0000 7f1610400700 1 Processor -- start 2024-03-29T09:54:34.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.659+0000 7fe426a00700 1 -- 172.21.15.12:0/3190459714 shutdown_connections 2024-03-29T09:54:34.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.659+0000 7fe426a00700 1 --2- 172.21.15.12:0/3190459714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe41803e950 0x7fe418006c10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.659+0000 7fe426a00700 1 -- 172.21.15.12:0/3190459714 >> 172.21.15.12:0/3190459714 conn(0x7fe418055a40 msgr2=0x7fe418055e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.659+0000 7ff87f200700 1 Processor -- start 2024-03-29T09:54:34.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.659+0000 7ff87f200700 1 -- start start 2024-03-29T09:54:34.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.659+0000 7fe426a00700 1 -- 172.21.15.12:0/3190459714 shutdown_connections 2024-03-29T09:54:34.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.660+0000 7f1610400700 1 -- start start 2024-03-29T09:54:34.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.660+0000 7fe426a00700 1 -- 172.21.15.12:0/3190459714 wait complete. 2024-03-29T09:54:34.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.660+0000 7f3118a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31140d1a00 0x7f3114150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.660+0000 7f3118a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f311415f830 0x7f311409dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.660+0000 7f3118a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31141657c0 0x7f31141516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.660+0000 7f3118a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311409e430 con 0x7f31141657c0 2024-03-29T09:54:34.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.660+0000 7f3118a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31140c89d0 con 0x7f311415f830 2024-03-29T09:54:34.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.660+0000 7f3118a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31140c99e0 con 0x7f31140d1a00 2024-03-29T09:54:34.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.656+0000 7f23bda00700 1 -- start start 2024-03-29T09:54:34.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.658+0000 7f23bda00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f23b806e810 0x7f23b80b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.658+0000 7f23bda00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f23b8072910 0x7f23b80cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.658+0000 7f23bda00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80cc140 con 0x7f23b815d810 2024-03-29T09:54:34.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.658+0000 7f23bda00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b815df80 con 0x7f23b806e810 2024-03-29T09:54:34.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.658+0000 7f23bda00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b815e260 con 0x7f23b8072910 2024-03-29T09:54:34.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.661+0000 7f3111e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31140d1a00 0x7f3114150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.661+0000 7f3111400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f311415f830 0x7f311409dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.661+0000 7f3111400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f311415f830 0x7f311409dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46730/0 (socket says 172.21.15.12:46730) 2024-03-29T09:54:34.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.661+0000 7f3111400700 1 -- 172.21.15.12:0/3073957065 learned_addr learned my addr 172.21.15.12:0/3073957065 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.661+0000 7f3112800700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31141657c0 0x7f31141516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.658+0000 7fd6d5400700 1 -- 172.21.15.12:0/575784305 wait complete. 2024-03-29T09:54:34.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7f3111400700 1 -- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31140d1a00 msgr2=0x7f3114150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7f3111400700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31140d1a00 0x7f3114150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7f3111400700 1 -- 172.21.15.12:0/3073957065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31141657c0 msgr2=0x7f31141516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7f3111400700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31141657c0 0x7f31141516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7f3111400700 1 -- 172.21.15.12:0/3073957065 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c009b30 con 0x7f311415f830 2024-03-29T09:54:34.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7ff87f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8780e3250 0x7ff8780537a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7f1610400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160806dc60 0x7f160814e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7f3111e00700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31140d1a00 0x7f3114150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:34.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7ff87f200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff878053ce0 0x7ff878058000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7ff87f200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff878006e80 con 0x7ff878054340 2024-03-29T09:54:34.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7ff87f200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff878006c70 con 0x7ff878053ce0 2024-03-29T09:54:34.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7ff87f200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff878007c40 con 0x7ff8780e3250 2024-03-29T09:54:34.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.661+0000 7fd6d5400700 1 Processor -- start 2024-03-29T09:54:34.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7fd6d5400700 1 -- start start 2024-03-29T09:54:34.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7f1610400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16081597f0 0x7f160809c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f23b7400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f23b8072910 0x7f23b80cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f1610400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160815d4d0 0x7f160809cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f23b7400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f23b8072910 0x7f23b80cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46746/0 (socket says 172.21.15.12:46746) 2024-03-29T09:54:34.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f23b7400700 1 -- 172.21.15.12:0/571599327 learned_addr learned my addr 172.21.15.12:0/571599327 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.662+0000 7fe426a00700 1 Processor -- start 2024-03-29T09:54:34.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7ff87de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8780e3250 0x7ff8780537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7ff87de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8780e3250 0x7ff8780537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51338/0 (socket says 172.21.15.12:51338) 2024-03-29T09:54:34.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7ff87de00700 1 -- 172.21.15.12:0/4222643032 learned_addr learned my addr 172.21.15.12:0/4222643032 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f3111400700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f311415f830 0x7f311409dba0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f310000c820 tx=0x7f310000cbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7fe426a00700 1 -- start start 2024-03-29T09:54:34.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f160da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160806dc60 0x7f160814e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f310a000700 1 -- 172.21.15.12:0/3073957065 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100005c20 con 0x7f311415f830 2024-03-29T09:54:34.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7ff87de00700 1 -- 172.21.15.12:0/4222643032 >> v2:172.21.15.12:3300/0 conn(0x7ff878053ce0 msgr2=0x7ff878058000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7ff87de00700 1 --2- 172.21.15.12:0/4222643032 >> v2:172.21.15.12:3300/0 conn(0x7ff878053ce0 0x7ff878058000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7fb0a7e00700 1 Processor -- start 2024-03-29T09:54:34.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7fb0a7e00700 1 -- start start 2024-03-29T09:54:34.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f160da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160806dc60 0x7f160814e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40600/0 (socket says 172.21.15.12:40600) 2024-03-29T09:54:34.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f160e400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160815d4d0 0x7f160809cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f160e400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160815d4d0 0x7f160809cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46762/0 (socket says 172.21.15.12:46762) 2024-03-29T09:54:34.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f160d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16081597f0 0x7f160809c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7ff87de00700 1 -- 172.21.15.12:0/4222643032 >> v1:172.21.15.12:6789/0 conn(0x7ff878054340 legacy=0x7ff87803c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7ff87de00700 1 -- 172.21.15.12:0/4222643032 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff878007160 con 0x7ff8780e3250 2024-03-29T09:54:34.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.664+0000 7f160d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16081597f0 0x7f160809c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51346/0 (socket says 172.21.15.12:51346) 2024-03-29T09:54:34.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.664+0000 7f160da00700 1 -- 172.21.15.12:0/368949574 learned_addr learned my addr 172.21.15.12:0/368949574 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f310a000700 1 -- 172.21.15.12:0/3073957065 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3100005d80 con 0x7f311415f830 2024-03-29T09:54:34.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.663+0000 7f310a000700 1 -- 172.21.15.12:0/3073957065 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100011800 con 0x7f311415f830 2024-03-29T09:54:34.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.664+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31140c8c20 con 0x7f311415f830 2024-03-29T09:54:34.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.664+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31140c91e0 con 0x7f311415f830 2024-03-29T09:54:34.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.664+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1608099c10 con 0x7f160806dc60 2024-03-29T09:54:34.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.664+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160814e7d0 con 0x7f160815d4d0 2024-03-29T09:54:34.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.664+0000 7f23b5600700 1 -- 172.21.15.12:0/571599327 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718743602 0 0) 0x7f23b80cc140 con 0x7f23b815d810 2024-03-29T09:54:34.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.664+0000 7f23b5600700 1 -- 172.21.15.12:0/571599327 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f239c0035b0 con 0x7f23b815d810 2024-03-29T09:54:34.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.664+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160809d190 con 0x7f16081597f0 2024-03-29T09:54:34.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.664+0000 7f23b6000700 1 --2- 172.21.15.12:0/571599327 >> v2:172.21.15.12:3300/0 conn(0x7f23b806e810 0x7f23b80b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7ff87de00700 1 --2- 172.21.15.12:0/4222643032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8780e3250 0x7ff8780537a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff8700098f0 tx=0x7ff870009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c221cd1269b8d4d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f31140d0c70 con 0x7f311415f830 2024-03-29T09:54:34.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f23b7400700 1 -- 172.21.15.12:0/571599327 >> v2:172.21.15.12:3300/0 conn(0x7f23b806e810 msgr2=0x7f23b80b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f23b7400700 1 --2- 172.21.15.12:0/571599327 >> v2:172.21.15.12:3300/0 conn(0x7f23b806e810 0x7f23b80b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f23b7400700 1 -- 172.21.15.12:0/571599327 >> v1:172.21.15.12:6789/0 conn(0x7f23b815d810 legacy=0x7f23b806e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f23b7400700 1 -- 172.21.15.12:0/571599327 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b80cbdd0 con 0x7f23b8072910 2024-03-29T09:54:34.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f23b6000700 1 --2- 172.21.15.12:0/571599327 >> v2:172.21.15.12:3300/0 conn(0x7f23b806e810 0x7f23b80b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:34.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7fe426a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe41803e950 0x7fe4180ddc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f160d000700 1 -- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160815d4d0 msgr2=0x7f160809cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f160d000700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160815d4d0 0x7f160809cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f160d000700 1 -- 172.21.15.12:0/368949574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160806dc60 msgr2=0x7f160814e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f160d000700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160806dc60 0x7f160814e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7f160d000700 1 -- 172.21.15.12:0/368949574 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1600009a70 con 0x7f16081597f0 2024-03-29T09:54:34.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7ff87ca00700 1 -- 172.21.15.12:0/4222643032 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff870004230 con 0x7ff8780e3250 2024-03-29T09:54:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7ff87ca00700 1 -- 172.21.15.12:0/4222643032 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff870004390 con 0x7ff8780e3250 2024-03-29T09:54:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.665+0000 7ff87ca00700 1 -- 172.21.15.12:0/4222643032 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8700036a0 con 0x7ff8780e3250 2024-03-29T09:54:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7f23b7400700 1 --2- 172.21.15.12:0/571599327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23b8072910 0x7f23b80cc5d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f23b815e540 tx=0x7f23b000c5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e4009cea38833ee server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7fe426a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4180543a0 0x7fe4180e0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7f160da00700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160806dc60 0x7f160814e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7f160e400700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f160815d4d0 0x7f160809cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7fd6d5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d0070e60 0x7fd6d015a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7fd6d5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0161f60 0x7fd6d0158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7fd6d5400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d015a5c0 0x7fd6d0158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7fd6d5400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d015b3e0 con 0x7fd6d0070e60 2024-03-29T09:54:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7fd6d5400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d015ab40 con 0x7fd6d015a5c0 2024-03-29T09:54:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7fd6d5400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d015ae20 con 0x7fd6d0161f60 2024-03-29T09:54:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7fe426a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4180e3240 0x7fe4180e1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7f23b5600700 1 -- 172.21.15.12:0/571599327 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b000cda0 con 0x7f23b8072910 2024-03-29T09:54:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7f23b5600700 1 -- 172.21.15.12:0/571599327 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b000a140 con 0x7f23b8072910 2024-03-29T09:54:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7f23b5600700 1 -- 172.21.15.12:0/571599327 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b000a4d0 con 0x7f23b8072910 2024-03-29T09:54:34.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7fe425600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe41803e950 0x7fe4180ddc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7fe424c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4180543a0 0x7fe4180e0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fe425600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe41803e950 0x7fe4180ddc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51352/0 (socket says 172.21.15.12:51352) 2024-03-29T09:54:34.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fe424c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4180543a0 0x7fe4180e0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40612/0 (socket says 172.21.15.12:40612) 2024-03-29T09:54:34.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7ff87f200700 1 -- 172.21.15.12:0/4222643032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8780e3250 msgr2=0x7ff8780537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.666+0000 7ff87f200700 1 --2- 172.21.15.12:0/4222643032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8780e3250 0x7ff8780537a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff8700098f0 tx=0x7ff870009c00 comp rx=0 tx=0).stop 2024-03-29T09:54:34.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fe425600700 1 -- 172.21.15.12:0/268053801 learned_addr learned my addr 172.21.15.12:0/268053801 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fd6cf200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d015a5c0 0x7fd6d0158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe418168440 con 0x7fe4180543a0 2024-03-29T09:54:34.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fe426000700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4180e3240 0x7fe4180e1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41803a480 con 0x7fe4180e3240 2024-03-29T09:54:34.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fd6cf200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d015a5c0 0x7fd6d0158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46772/0 (socket says 172.21.15.12:46772) 2024-03-29T09:54:34.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fd6cf200700 1 -- 172.21.15.12:0/3208169596 learned_addr learned my addr 172.21.15.12:0/3208169596 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fd6cde00700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0161f60 0x7fd6d0158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41803a760 con 0x7fe41803e950 2024-03-29T09:54:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7fd6ce800700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d0070e60 0x7fd6d015a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7f23bda00700 1 -- 172.21.15.12:0/571599327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23b8072910 msgr2=0x7f23b80cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.667+0000 7f23bda00700 1 --2- 172.21.15.12:0/571599327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23b8072910 0x7f23b80cc5d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f23b815e540 tx=0x7f23b000c5c0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fd6cde00700 1 -- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d015a5c0 msgr2=0x7fd6d0158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fb0a7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a8159400 0x7fb0a80b3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fd6cde00700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d015a5c0 0x7fd6d0158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fd6cde00700 1 -- 172.21.15.12:0/3208169596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d0070e60 msgr2=0x7fd6d015a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fd6cde00700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d0070e60 0x7fd6d015a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fd6cde00700 1 -- 172.21.15.12:0/3208169596 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c8009550 con 0x7fd6d0161f60 2024-03-29T09:54:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7f160d000700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16081597f0 0x7f160809c710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f15f400c390 tx=0x7f15f400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7ff87f200700 1 -- 172.21.15.12:0/4222643032 shutdown_connections 2024-03-29T09:54:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7ff87f200700 1 --2- 172.21.15.12:0/4222643032 >> v2:172.21.15.12:3300/0 conn(0x7ff878053ce0 0x7ff878058000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7ff87f200700 1 --2- 172.21.15.12:0/4222643032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8780e3250 0x7ff8780537a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7ff87f200700 1 -- 172.21.15.12:0/4222643032 >> 172.21.15.12:0/4222643032 conn(0x7ff878058f40 msgr2=0x7ff878059350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7ff87f200700 1 -- 172.21.15.12:0/4222643032 shutdown_connections 2024-03-29T09:54:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7ff87f200700 1 -- 172.21.15.12:0/4222643032 wait complete. 2024-03-29T09:54:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fb0a7e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a80b4370 con 0x7fb0a80b3eb0 2024-03-29T09:54:34.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7f1605600700 1 -- 172.21.15.12:0/368949574 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f4004a70 con 0x7f16081597f0 2024-03-29T09:54:34.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7f1605600700 1 -- 172.21.15.12:0/368949574 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f4004bd0 con 0x7f16081597f0 2024-03-29T09:54:34.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7f1605600700 1 -- 172.21.15.12:0/368949574 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f40070d0 con 0x7f16081597f0 2024-03-29T09:54:34.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1608099270 con 0x7f16081597f0 2024-03-29T09:54:34.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7f23bda00700 1 -- 172.21.15.12:0/571599327 shutdown_connections 2024-03-29T09:54:34.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7f23bda00700 1 --2- 172.21.15.12:0/571599327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23b8072910 0x7f23b80cc5d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7f23bda00700 1 --2- 172.21.15.12:0/571599327 >> v2:172.21.15.12:3300/0 conn(0x7f23b806e810 0x7f23b80b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7f23bda00700 1 -- 172.21.15.12:0/571599327 >> 172.21.15.12:0/571599327 conn(0x7f23b8070f00 msgr2=0x7f23b8071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1608099830 con 0x7f16081597f0 2024-03-29T09:54:34.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fb0a7e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a815d2d0 con 0x7fb0a806cf30 2024-03-29T09:54:34.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fb0a7e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a80b4650 con 0x7fb0a8159400 2024-03-29T09:54:34.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7fb0a6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a8159400 0x7fb0a80b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fe426000700 1 -- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe41803e950 msgr2=0x7fe4180ddc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fe426000700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe41803e950 0x7fe4180ddc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fe426000700 1 -- 172.21.15.12:0/268053801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4180543a0 msgr2=0x7fe4180e0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7fe426000700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4180543a0 0x7fe4180e0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7fe426000700 1 -- 172.21.15.12:0/268053801 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4200686c0 con 0x7fe4180e3240 2024-03-29T09:54:34.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7fe424c00700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4180543a0 0x7fe4180e0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:34.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7fe426000700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4180e3240 0x7fe4180e1160 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe420064450 tx=0x7fe4200685b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7fb0a6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a8159400 0x7fb0a80b3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51372/0 (socket says 172.21.15.12:51372) 2024-03-29T09:54:34.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7fb0a6a00700 1 -- 172.21.15.12:0/2616128427 learned_addr learned my addr 172.21.15.12:0/2616128427 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7fe41d600700 1 -- 172.21.15.12:0/268053801 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42009b070 con 0x7fe4180e3240 2024-03-29T09:54:34.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.669+0000 7fe41d600700 1 -- 172.21.15.12:0/268053801 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42006ae60 con 0x7fe4180e3240 2024-03-29T09:54:34.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.670+0000 7fe41d600700 1 -- 172.21.15.12:0/268053801 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe420094660 con 0x7fe4180e3240 2024-03-29T09:54:34.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.670+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41803aa40 con 0x7fe4180e3240 2024-03-29T09:54:34.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.670+0000 7fb0a6a00700 1 -- 172.21.15.12:0/2616128427 >> v1:172.21.15.159:6789/0 conn(0x7fb0a806cf30 legacy=0x7fb0a80ccec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T09:54:34.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.670+0000 7fb0a6a00700 1 -- 172.21.15.12:0/2616128427 >> v1:172.21.15.12:6789/0 conn(0x7fb0a80b3eb0 legacy=0x7fb0a806c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.670+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f15e4000fc0 con 0x7f16081597f0 2024-03-29T09:54:34.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fd6cf200700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d015a5c0 0x7fd6d0158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:34.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7fd6ce800700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d0070e60 0x7fd6d015a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:34.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.671+0000 7fb0a6a00700 1 -- 172.21.15.12:0/2616128427 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a815d5b0 con 0x7fb0a8159400 2024-03-29T09:54:34.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.668+0000 7f310a000700 1 -- 172.21.15.12:0/3073957065 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3100011960 con 0x7f311415f830 2024-03-29T09:54:34.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.670+0000 7f310a000700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30f406ac90 0x7f30f406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.670+0000 7f310a000700 1 -- 172.21.15.12:0/3073957065 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3100089700 con 0x7f311415f830 2024-03-29T09:54:34.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.670+0000 7f310a000700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] conn(0x7f30f4072e30 0x7f30f4075250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.670+0000 7f310a000700 1 -- 172.21.15.12:0/3073957065 --> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30f4075900 con 0x7f30f4072e30 2024-03-29T09:54:34.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.671+0000 7f310a000700 1 -- 172.21.15.12:0/3073957065 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3100089b50 con 0x7f311415f830 2024-03-29T09:54:34.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.671+0000 7f3112800700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] conn(0x7f30f4072e30 0x7f30f4075250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.671+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41803b000 con 0x7fe4180e3240 2024-03-29T09:54:34.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.671+0000 7fd6cde00700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0161f60 0x7fd6d0158020 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd6d006c230 tx=0x7fd6c000cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.671+0000 7ff87f200700 1 Processor -- start 2024-03-29T09:54:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7fb0a6a00700 1 --2- 172.21.15.12:0/2616128427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a8159400 0x7fb0a80b3970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000d180 tx=0x7fb0a000d490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59c514e14ca8d05d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7ff87f200700 1 -- start start 2024-03-29T09:54:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.671+0000 7f23bda00700 1 -- 172.21.15.12:0/571599327 shutdown_connections 2024-03-29T09:54:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7fa698e00700 1 Processor -- start 2024-03-29T09:54:34.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7fd6c6a00700 1 -- 172.21.15.12:0/3208169596 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c000a9e0 con 0x7fd6d0161f60 2024-03-29T09:54:34.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7fd6c6a00700 1 -- 172.21.15.12:0/3208169596 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c000ab40 con 0x7fd6d0161f60 2024-03-29T09:54:34.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7fd6c6a00700 1 -- 172.21.15.12:0/3208169596 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0007070 con 0x7fd6d0161f60 2024-03-29T09:54:34.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7f3111e00700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30f406ac90 0x7f30f406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7fd6d5400700 1 -- 172.21.15.12:0/3208169596 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d006e2a0 con 0x7fd6d0161f60 2024-03-29T09:54:34.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7fb0a5600700 1 -- 172.21.15.12:0/2616128427 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0010070 con 0x7fb0a8159400 2024-03-29T09:54:34.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7fb0a5600700 1 -- 172.21.15.12:0/2616128427 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a000dba0 con 0x7fb0a8159400 2024-03-29T09:54:34.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7fb0a5600700 1 -- 172.21.15.12:0/2616128427 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0003a20 con 0x7fb0a8159400 2024-03-29T09:54:34.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7f23bda00700 1 -- 172.21.15.12:0/571599327 wait complete. 2024-03-29T09:54:34.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.672+0000 7fd6d5400700 1 -- 172.21.15.12:0/3208169596 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d0152980 con 0x7fd6d0161f60 2024-03-29T09:54:34.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.673+0000 7f3112800700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] conn(0x7f30f4072e30 0x7f30f4075250 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.673+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe418005840 con 0x7fe4180e3240 2024-03-29T09:54:34.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.673+0000 7fb0a7e00700 1 -- 172.21.15.12:0/2616128427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a8159400 msgr2=0x7fb0a80b3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.673+0000 7fb0a7e00700 1 --2- 172.21.15.12:0/2616128427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a8159400 0x7fb0a80b3970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000d180 tx=0x7fb0a000d490 comp rx=0 tx=0).stop 2024-03-29T09:54:34.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.674+0000 7ff87f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff878053ce0 0x7ff8780d2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.674+0000 7fa698e00700 1 -- start start 2024-03-29T09:54:34.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.674+0000 7ff87f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff878054340 0x7ff87803ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.674+0000 7ff87f200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8780e3250 0x7ff87803b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.674+0000 7ff87de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff878053ce0 0x7ff8780d2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.675+0000 7ff87de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff878053ce0 0x7ff8780d2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51380/0 (socket says 172.21.15.12:51380) 2024-03-29T09:54:34.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.674+0000 7f3111e00700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30f406ac90 0x7f30f406d150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3104003fc0 tx=0x7f3104009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.675+0000 7f310a000700 1 -- 172.21.15.12:0/3073957065 <== osd.1 v2:172.21.15.12:6816/43909 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f30f4075900 con 0x7f30f4072e30 2024-03-29T09:54:34.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.675+0000 7fd6d5400700 1 -- 172.21.15.12:0/3208169596 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd6d006e120 con 0x7fd6d0161f60 2024-03-29T09:54:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.675+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8780d8a70 con 0x7ff878054340 2024-03-29T09:54:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.675+0000 7f1605600700 1 -- 172.21.15.12:0/368949574 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f15f4007230 con 0x7f16081597f0 2024-03-29T09:54:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f1605600700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15e806ad60 0x7f15e806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f1605600700 1 -- 172.21.15.12:0/368949574 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15f40891e0 con 0x7f16081597f0 2024-03-29T09:54:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f1605600700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] conn(0x7f15e8072f80 0x7f15e80753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f1605600700 1 -- 172.21.15.12:0/368949574 --> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15e8075a50 con 0x7f15e8072f80 2024-03-29T09:54:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f1605600700 1 -- 172.21.15.12:0/368949574 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15f4089590 con 0x7f16081597f0 2024-03-29T09:54:34.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7fd6c6a00700 1 -- 172.21.15.12:0/3208169596 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd6c00071d0 con 0x7fd6d0161f60 2024-03-29T09:54:34.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.675+0000 7fe41d600700 1 -- 172.21.15.12:0/268053801 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe4200947c0 con 0x7fe4180e3240 2024-03-29T09:54:34.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7fe41d600700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe40006ad60 0x7fe40006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7fe41d600700 1 -- 172.21.15.12:0/268053801 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe42008f020 con 0x7fe4180e3240 2024-03-29T09:54:34.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7fe41d600700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] conn(0x7fe400072f00 0x7fe400075320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7fe41d600700 1 -- 172.21.15.12:0/268053801 --> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4000759d0 con 0x7fe400072f00 2024-03-29T09:54:34.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7fe425600700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe40006ad60 0x7fe40006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.675+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff878052540 con 0x7ff8780e3250 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.675+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8780529d0 con 0x7ff878053ce0 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.675+0000 7ff87de00700 1 -- 172.21.15.12:0/3097006509 learned_addr learned my addr 172.21.15.12:0/3097006509 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.675+0000 7ff87d400700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff878054340 0x7ff87803ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7ff87de00700 1 -- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8780e3250 msgr2=0x7ff87803b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7ff87de00700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8780e3250 0x7ff87803b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7ff87de00700 1 -- 172.21.15.12:0/3097006509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff878054340 msgr2=0x7ff87803ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7ff87de00700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff878054340 0x7ff87803ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7ff87de00700 1 -- 172.21.15.12:0/3097006509 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff870009550 con 0x7ff878053ce0 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7ff87de00700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff878053ce0 0x7ff8780d2d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff878006db0 tx=0x7ff870004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7ff876000700 1 -- 172.21.15.12:0/3097006509 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff870003e90 con 0x7ff878053ce0 2024-03-29T09:54:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7ff876000700 1 -- 172.21.15.12:0/3097006509 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87000b660 con 0x7ff878053ce0 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7ff876000700 1 -- 172.21.15.12:0/3097006509 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87000b9f0 con 0x7ff878053ce0 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff878051c10 con 0x7ff878053ce0 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8780521d0 con 0x7ff878053ce0 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7fa698e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 0x7fa6940b2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7fa698e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa694163190 con 0x7fa6940b3760 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7fa698e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa694162580 con 0x7fa6940cc370 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7fa698e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa694162860 con 0x7fa6940ce500 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7fa691e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 0x7fa6940b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7fa691e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 0x7fa6940b2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51392/0 (socket says 172.21.15.12:51392) 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7fa691e00700 1 -- 172.21.15.12:0/2543897863 learned_addr learned my addr 172.21.15.12:0/2543897863 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7fa691e00700 1 -- 172.21.15.12:0/2543897863 >> v1:172.21.15.159:6789/0 conn(0x7fa6940cc370 legacy=0x7fa6940ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7fa691e00700 1 -- 172.21.15.12:0/2543897863 >> v1:172.21.15.12:6789/0 conn(0x7fa6940b3760 legacy=0x7fa694165ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T09:54:34.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.677+0000 7fa691e00700 1 -- 172.21.15.12:0/2543897863 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa694162b40 con 0x7fa6940ce500 2024-03-29T09:54:34.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fe424c00700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] conn(0x7fe400072f00 0x7fe400075320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7f160da00700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15e806ad60 0x7f15e806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fd6c6a00700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd6b006ad60 0x7fd6b006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fa691e00700 1 --2- 172.21.15.12:0/2543897863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 0x7fa6940b2070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009e80 tx=0x7fa68c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db3683f8c34e0253 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fd6c6a00700 1 -- 172.21.15.12:0/3208169596 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6c0089030 con 0x7fd6d0161f60 2024-03-29T09:54:34.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fb0a7e00700 1 -- 172.21.15.12:0/2616128427 shutdown_connections 2024-03-29T09:54:34.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fb0a7e00700 1 --2- 172.21.15.12:0/2616128427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a8159400 0x7fb0a80b3970 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7f23bda00700 1 Processor -- start 2024-03-29T09:54:34.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.676+0000 7f23bda00700 1 -- start start 2024-03-29T09:54:34.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f23bda00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23b806e810 0x7f23b81ff570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f23bda00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23b8072910 0x7f23b80b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f23bda00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b815d810 0x7f23b80b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff878007720 con 0x7ff878053ce0 2024-03-29T09:54:34.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f23bda00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b80b0990 con 0x7f23b815d810 2024-03-29T09:54:34.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f23bda00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b80af8f0 con 0x7f23b806e810 2024-03-29T09:54:34.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f23bda00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b80afbd0 con 0x7f23b8072910 2024-03-29T09:54:34.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.678+0000 7f23b6a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23b806e810 0x7f23b81ff570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7f23b6a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23b806e810 0x7f23b81ff570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46788/0 (socket says 172.21.15.12:46788) 2024-03-29T09:54:34.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7f23b6a00700 1 -- 172.21.15.12:0/2336990357 learned_addr learned my addr 172.21.15.12:0/2336990357 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fe425600700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe40006ad60 0x7fe40006d220 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe4140011e0 tx=0x7fe414007a80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7f23b6000700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23b8072910 0x7f23b80b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7f23b7400700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b815d810 0x7f23b80b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fa690a00700 1 -- 172.21.15.12:0/2543897863 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c00e070 con 0x7fa6940ce500 2024-03-29T09:54:34.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fa690a00700 1 -- 172.21.15.12:0/2543897863 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68c003a20 con 0x7fa6940ce500 2024-03-29T09:54:34.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fa690a00700 1 -- 172.21.15.12:0/2543897863 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c003dd0 con 0x7fa6940ce500 2024-03-29T09:54:34.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.679+0000 7fb0a7e00700 1 -- 172.21.15.12:0/2616128427 >> 172.21.15.12:0/2616128427 conn(0x7fb0a806df90 msgr2=0x7fb0a806a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7fe41d600700 1 -- 172.21.15.12:0/268053801 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4200cef40 con 0x7fe4180e3240 2024-03-29T09:54:34.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7f23b6000700 1 -- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23b806e810 msgr2=0x7f23b81ff570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7f23b6000700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23b806e810 0x7f23b81ff570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7f23b6000700 1 -- 172.21.15.12:0/2336990357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b815d810 msgr2=0x7f23b80b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7f23b6000700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b815d810 0x7f23b80b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7f23b6000700 1 -- 172.21.15.12:0/2336990357 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b000c050 con 0x7f23b8072910 2024-03-29T09:54:34.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7fb0a7e00700 1 -- 172.21.15.12:0/2616128427 shutdown_connections 2024-03-29T09:54:34.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7fb0a7e00700 1 -- 172.21.15.12:0/2616128427 wait complete. 2024-03-29T09:54:34.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7fe424c00700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] conn(0x7fe400072f00 0x7fe400075320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7f23b6000700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23b8072910 0x7f23b80b11f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f23a400e810 tx=0x7f23a400eb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.681+0000 7f160e400700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] conn(0x7f15e8072f80 0x7f15e80753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.681+0000 7f23aea00700 1 -- 172.21.15.12:0/2336990357 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a400c070 con 0x7f23b8072910 2024-03-29T09:54:34.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.681+0000 7f23aea00700 1 -- 172.21.15.12:0/2336990357 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a4013610 con 0x7f23b8072910 2024-03-29T09:54:34.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.681+0000 7f23aea00700 1 -- 172.21.15.12:0/2336990357 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a40139c0 con 0x7f23b8072910 2024-03-29T09:54:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.681+0000 7fd6c6a00700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] conn(0x7fd6b0072f80 0x7fd6b00753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.681+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b80afeb0 con 0x7f23b8072910 2024-03-29T09:54:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.681+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b80b0470 con 0x7f23b8072910 2024-03-29T09:54:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.682+0000 7f160da00700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15e806ad60 0x7f15e806d220 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f15fc009cf0 tx=0x7f15fc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7fa698e00700 1 -- 172.21.15.12:0/2543897863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 msgr2=0x7fa6940b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.680+0000 7fa698e00700 1 --2- 172.21.15.12:0/2543897863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 0x7fa6940b2070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009e80 tx=0x7fa68c00b330 comp rx=0 tx=0).stop 2024-03-29T09:54:34.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.681+0000 7fa698e00700 1 -- 172.21.15.12:0/2543897863 shutdown_connections 2024-03-29T09:54:34.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.681+0000 7fa698e00700 1 --2- 172.21.15.12:0/2543897863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 0x7fa6940b2070 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.681+0000 7fa698e00700 1 -- 172.21.15.12:0/2543897863 >> 172.21.15.12:0/2543897863 conn(0x7fa6940b0d70 msgr2=0x7fa6940b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.682+0000 7fd6ce800700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd6b006ad60 0x7fd6b006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.682+0000 7fe41d600700 1 -- 172.21.15.12:0/268053801 <== osd.3 v2:172.21.15.12:6820/43910 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe4000759d0 con 0x7fe400072f00 2024-03-29T09:54:34.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.682+0000 7fd6c6a00700 1 -- 172.21.15.12:0/3208169596 --> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6b0075a50 con 0x7fd6b0072f80 2024-03-29T09:54:34.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.683+0000 7fd6c6a00700 1 -- 172.21.15.12:0/3208169596 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6c008b5f0 con 0x7fd6d0161f60 2024-03-29T09:54:34.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.682+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f23b815dbf0 con 0x7f23b8072910 2024-03-29T09:54:34.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.683+0000 7f160e400700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] conn(0x7f15e8072f80 0x7f15e80753a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.684+0000 7fb0a7e00700 1 Processor -- start 2024-03-29T09:54:34.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.684+0000 7fd6ce800700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd6b006ad60 0x7fd6b006d220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0074510 tx=0x7fd6c8008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.684+0000 7fb0a7e00700 1 -- start start 2024-03-29T09:54:34.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.684+0000 7fd6cf200700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] conn(0x7fd6b0072f80 0x7fd6b00753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.684+0000 7ff876000700 1 -- 172.21.15.12:0/3097006509 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff870022070 con 0x7ff878053ce0 2024-03-29T09:54:34.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.685+0000 7f1605600700 1 -- 172.21.15.12:0/368949574 <== osd.0 v2:172.21.15.12:6808/43908 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f15e8075a50 con 0x7f15e8072f80 2024-03-29T09:54:34.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.685+0000 7fd6cf200700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] conn(0x7fd6b0072f80 0x7fd6b00753a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.685+0000 7ff876000700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff86006ac90 0x7ff86006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.685+0000 7ff876000700 1 -- 172.21.15.12:0/3097006509 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff87008b590 con 0x7ff878053ce0 2024-03-29T09:54:34.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.686+0000 7ff876000700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] conn(0x7ff860072cc0 0x7ff8600750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.686+0000 7ff876000700 1 -- 172.21.15.12:0/3097006509 --> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff860075790 con 0x7ff860072cc0 2024-03-29T09:54:34.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.686+0000 7ff876000700 1 -- 172.21.15.12:0/3097006509 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff87008b900 con 0x7ff878053ce0 2024-03-29T09:54:34.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.686+0000 7fa698e00700 1 -- 172.21.15.12:0/2543897863 shutdown_connections 2024-03-29T09:54:34.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.686+0000 7fa698e00700 1 -- 172.21.15.12:0/2543897863 wait complete. 2024-03-29T09:54:34.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.686+0000 7ff87d400700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff86006ac90 0x7ff86006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.686+0000 7ff87e800700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] conn(0x7ff860072cc0 0x7ff8600750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.686+0000 7fb0a7e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0a806cf30 0x7fb0a814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.686+0000 7f23aea00700 1 -- 172.21.15.12:0/2336990357 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f23a4013b20 con 0x7f23b8072910 2024-03-29T09:54:34.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.686+0000 7fb0a7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a80b3eb0 0x7fb0a8147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0a8159400 0x7fb0a81f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a814f510 con 0x7fb0a8159400 2024-03-29T09:54:34.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a8147d90 con 0x7fb0a806cf30 2024-03-29T09:54:34.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a81505e0 con 0x7fb0a80b3eb0 2024-03-29T09:54:34.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a6a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0a806cf30 0x7fb0a814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0a8159400 0x7fb0a81f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0a8159400 0x7fb0a81f3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40640/0 (socket says 172.21.15.12:40640) 2024-03-29T09:54:34.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7400700 1 -- 172.21.15.12:0/3597597593 learned_addr learned my addr 172.21.15.12:0/3597597593 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a6000700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a80b3eb0 0x7fb0a8147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7ff87d400700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff86006ac90 0x7ff86006d150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff8780e05a0 tx=0x7ff86c0090e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7400700 1 -- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a80b3eb0 msgr2=0x7fb0a8147850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7400700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0a80b3eb0 0x7fb0a8147850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7400700 1 -- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0a806cf30 msgr2=0x7fb0a814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.687+0000 7fb0a7400700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0a806cf30 0x7fb0a814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.688+0000 7fb0a7400700 1 -- 172.21.15.12:0/3597597593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a000bd40 con 0x7fb0a8159400 2024-03-29T09:54:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.688+0000 7fb0a6a00700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0a806cf30 0x7fb0a814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.688+0000 7fd6c6a00700 1 -- 172.21.15.12:0/3208169596 <== osd.2 v2:172.21.15.12:6800/43907 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd6b0075a50 con 0x7fd6b0072f80 2024-03-29T09:54:34.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.688+0000 7f23aea00700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f239806ace0 0x7f239806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.689+0000 7fa698e00700 1 Processor -- start 2024-03-29T09:54:34.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.688+0000 7fb0a7400700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0a8159400 0x7fb0a81f3160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb09400b4a0 tx=0x7fb09400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.688+0000 7f23aea00700 1 -- 172.21.15.12:0/2336990357 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23a40895b0 con 0x7f23b8072910 2024-03-29T09:54:34.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.689+0000 7f23aea00700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] conn(0x7f2398072f00 0x7f2398075320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.689+0000 7f23aea00700 1 -- 172.21.15.12:0/2336990357 --> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23980759d0 con 0x7f2398072f00 2024-03-29T09:54:34.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.689+0000 7f23aea00700 1 -- 172.21.15.12:0/2336990357 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23a4089920 con 0x7f23b8072910 2024-03-29T09:54:34.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.689+0000 7fb09ea00700 1 -- 172.21.15.12:0/3597597593 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094011830 con 0x7fb0a8159400 2024-03-29T09:54:34.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.689+0000 7f23b7400700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] conn(0x7f2398072f00 0x7f2398075320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.690+0000 7fb09ea00700 1 -- 172.21.15.12:0/3597597593 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0940119b0 con 0x7fb0a8159400 2024-03-29T09:54:34.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.690+0000 7fb09ea00700 1 -- 172.21.15.12:0/3597597593 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094011d60 con 0x7fb0a8159400 2024-03-29T09:54:34.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.690+0000 7f23b7400700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] conn(0x7f2398072f00 0x7f2398075320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.690+0000 7f23b6a00700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f239806ace0 0x7f239806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.689+0000 7fa698e00700 1 -- start start 2024-03-29T09:54:34.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.690+0000 7fa698e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6940b3760 0x7fa694154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.690+0000 7fa698e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6940cc370 0x7fa69409c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.690+0000 7fa698e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 0x7fa6941f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.690+0000 7fa698e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69409cbe0 con 0x7fa6940b3760 2024-03-29T09:54:34.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.690+0000 7fa698e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941f7500 con 0x7fa6940cc370 2024-03-29T09:54:34.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.690+0000 7fa698e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69415eed0 con 0x7fa6940ce500 2024-03-29T09:54:34.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.691+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a814f6f0 con 0x7fb0a8159400 2024-03-29T09:54:34.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.691+0000 7fa692800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 0x7fa6941f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.691+0000 7ff87e800700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] conn(0x7ff860072cc0 0x7ff8600750e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.691+0000 7fa691e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6940b3760 0x7fa694154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.691+0000 7fa691400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6940cc370 0x7fa69409c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.691+0000 7fa691e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6940b3760 0x7fa694154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40650/0 (socket says 172.21.15.12:40650) 2024-03-29T09:54:34.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.691+0000 7fa691e00700 1 -- 172.21.15.12:0/196952069 learned_addr learned my addr 172.21.15.12:0/196952069 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:34.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.692+0000 7f23b6a00700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f239806ace0 0x7f239806d1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f23a80011e0 tx=0x7f23a8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.691+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a814fcb0 con 0x7fb0a8159400 2024-03-29T09:54:34.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.692+0000 7fa691e00700 1 -- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 msgr2=0x7fa6941f6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.692+0000 7ff876000700 1 -- 172.21.15.12:0/3097006509 <== osd.4 v2:172.21.15.159:6800/44146 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff860075790 con 0x7ff860072cc0 2024-03-29T09:54:34.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.692+0000 7f23aea00700 1 -- 172.21.15.12:0/2336990357 <== osd.5 v2:172.21.15.159:6816/44148 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f23980759d0 con 0x7f2398072f00 2024-03-29T09:54:34.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.692+0000 7fa691e00700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940ce500 0x7fa6941f6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.692+0000 7fa691e00700 1 -- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6940cc370 msgr2=0x7fa69409c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.692+0000 7fa691e00700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6940cc370 0x7fa69409c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.692+0000 7fa691e00700 1 -- 172.21.15.12:0/196952069 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c009b30 con 0x7fa6940b3760 2024-03-29T09:54:34.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.692+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb0a81597e0 con 0x7fb0a8159400 2024-03-29T09:54:34.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.693+0000 7fa691400700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6940cc370 0x7fa69409c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:34.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.693+0000 7fa691e00700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6940b3760 0x7fa694154610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa68c015040 tx=0x7fa68c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.694+0000 7fa68a000700 1 -- 172.21.15.12:0/196952069 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c0092e0 con 0x7fa6940b3760 2024-03-29T09:54:34.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.694+0000 7fa68a000700 1 -- 172.21.15.12:0/196952069 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68c004240 con 0x7fa6940b3760 2024-03-29T09:54:34.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.694+0000 7fa68a000700 1 -- 172.21.15.12:0/196952069 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c004610 con 0x7fa6940b3760 2024-03-29T09:54:34.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.694+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69409cdf0 con 0x7fa6940b3760 2024-03-29T09:54:34.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.694+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69415f190 con 0x7fa6940b3760 2024-03-29T09:54:34.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.695+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa6940b3b90 con 0x7fa6940b3760 2024-03-29T09:54:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.696+0000 7fb09ea00700 1 -- 172.21.15.12:0/3597597593 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb09401a410 con 0x7fb0a8159400 2024-03-29T09:54:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.698+0000 7fb09ea00700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb08806acc0 0x7fb08806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.698+0000 7fb09ea00700 1 -- 172.21.15.12:0/3597597593 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb094088d40 con 0x7fb0a8159400 2024-03-29T09:54:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.699+0000 7fb09ea00700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] conn(0x7fb088072e60 0x7fb088075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.699+0000 7fb09ea00700 1 -- 172.21.15.12:0/3597597593 --> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb088075930 con 0x7fb088072e60 2024-03-29T09:54:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.699+0000 7fb0a6a00700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb08806acc0 0x7fb08806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.699+0000 7fb09ea00700 1 -- 172.21.15.12:0/3597597593 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb09408d030 con 0x7fb0a8159400 2024-03-29T09:54:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.699+0000 7fa68a000700 1 -- 172.21.15.12:0/196952069 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa68c019070 con 0x7fa6940b3760 2024-03-29T09:54:34.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.699+0000 7fb0a6a00700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb08806acc0 0x7fb08806d180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0000f80 tx=0x7fb0a0012000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.700+0000 7fb0a6000700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] conn(0x7fb088072e60 0x7fb088075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.700+0000 7fa68a000700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa67406ace0 0x7fa67406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.701+0000 7fb0a6000700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] conn(0x7fb088072e60 0x7fb088075280 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.701+0000 7fa68a000700 1 -- 172.21.15.12:0/196952069 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa68c08a800 con 0x7fa6940b3760 2024-03-29T09:54:34.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.701+0000 7fa691400700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa67406ace0 0x7fa67406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.702+0000 7fa68a000700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] conn(0x7fa674072f00 0x7fa674075320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:34.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.702+0000 7fb09ea00700 1 -- 172.21.15.12:0/3597597593 <== osd.6 v2:172.21.15.159:6817/44159 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb088075930 con 0x7fb088072e60 2024-03-29T09:54:34.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.702+0000 7fa68a000700 1 -- 172.21.15.12:0/196952069 --> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6740759d0 con 0x7fa674072f00 2024-03-29T09:54:34.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.702+0000 7fa68a000700 1 -- 172.21.15.12:0/196952069 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa68c08ab70 con 0x7fa6940b3760 2024-03-29T09:54:34.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.703+0000 7fa692800700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] conn(0x7fa674072f00 0x7fa674075320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:34.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.703+0000 7fa691400700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa67406ace0 0x7fa67406d1a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa6941fab10 tx=0x7fa680009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.703+0000 7fa692800700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] conn(0x7fa674072f00 0x7fa674075320 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:34.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.704+0000 7fa68a000700 1 -- 172.21.15.12:0/196952069 <== osd.7 v2:172.21.15.159:6808/44147 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa6740759d0 con 0x7fa674072f00 2024-03-29T09:54:34.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.714+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 --> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f31140d1200 con 0x7f30f4072e30 2024-03-29T09:54:34.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.714+0000 7f310a000700 1 -- 172.21.15.12:0/3073957065 <== osd.1 v2:172.21.15.12:6816/43909 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f310c0092e0 con 0x7f30f4072e30 2024-03-29T09:54:34.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 >> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] conn(0x7f30f4072e30 msgr2=0x7f30f4075250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3118a00700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] conn(0x7f30f4072e30 0x7f30f4075250 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30f406ac90 msgr2=0x7f30f406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3118a00700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30f406ac90 0x7f30f406d150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3104003fc0 tx=0x7f3104009040 comp rx=0 tx=0).stop 2024-03-29T09:54:34.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f311415f830 msgr2=0x7f311409dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3118a00700 1 --2- 172.21.15.12:0/3073957065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f311415f830 0x7f311409dba0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f310000c820 tx=0x7f310000cbe0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3112800700 1 -- 172.21.15.12:0/3073957065 reap_dead start 2024-03-29T09:54:34.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 shutdown_connections 2024-03-29T09:54:34.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 >> 172.21.15.12:0/3073957065 conn(0x7f311408aff0 msgr2=0x7f31140ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 shutdown_connections 2024-03-29T09:54:34.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.719+0000 7f3118a00700 1 -- 172.21.15.12:0/3073957065 wait complete. 2024-03-29T09:54:34.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.721+0000 7fd6d5400700 1 -- 172.21.15.12:0/3208169596 --> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd6d015c0d0 con 0x7fd6b0072f80 2024-03-29T09:54:34.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.722+0000 7fd6c6a00700 1 -- 172.21.15.12:0/3208169596 <== osd.2 v2:172.21.15.12:6800/43907 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd6bc00c040 con 0x7fd6b0072f80 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.720+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 --> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f15e4002d40 con 0x7f15e8072f80 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.721+0000 7f1605600700 1 -- 172.21.15.12:0/368949574 <== osd.0 v2:172.21.15.12:6808/43908 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f15e4002d40 con 0x7f15e8072f80 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.721+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 >> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] conn(0x7f15e8072f80 msgr2=0x7f15e80753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.721+0000 7f1610400700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] conn(0x7f15e8072f80 0x7f15e80753a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15e806ad60 msgr2=0x7f15e806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7f1610400700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15e806ad60 0x7f15e806d220 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f15fc009cf0 tx=0x7f15fc008040 comp rx=0 tx=0).stop 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16081597f0 msgr2=0x7f160809c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7f1610400700 1 --2- 172.21.15.12:0/368949574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16081597f0 0x7f160809c710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f15f400c390 tx=0x7f15f400c6a0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7f160e400700 1 -- 172.21.15.12:0/368949574 reap_dead start 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 shutdown_connections 2024-03-29T09:54:34.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 >> 172.21.15.12:0/368949574 conn(0x7f16080b0d70 msgr2=0x7f16080af450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7fd6bbe00700 1 -- 172.21.15.12:0/3208169596 >> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] conn(0x7fd6b0072f80 msgr2=0x7fd6b00753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7fd6bbe00700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] conn(0x7fd6b0072f80 0x7fd6b00753a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7fd6bbe00700 1 -- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd6b006ad60 msgr2=0x7fd6b006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7fd6bbe00700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd6b006ad60 0x7fd6b006d220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0074510 tx=0x7fd6c8008030 comp rx=0 tx=0).stop 2024-03-29T09:54:34.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7fd6bbe00700 1 -- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0161f60 msgr2=0x7fd6d0158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7fd6bbe00700 1 --2- 172.21.15.12:0/3208169596 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d0161f60 0x7fd6d0158020 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd6d006c230 tx=0x7fd6c000cd50 comp rx=0 tx=0).stop 2024-03-29T09:54:34.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7fd6cf200700 1 -- 172.21.15.12:0/3208169596 reap_dead start 2024-03-29T09:54:34.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7fd6bbe00700 1 -- 172.21.15.12:0/3208169596 shutdown_connections 2024-03-29T09:54:34.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7fd6bbe00700 1 -- 172.21.15.12:0/3208169596 >> 172.21.15.12:0/3208169596 conn(0x7fd6d00b0c70 msgr2=0x7fd6d00b03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.724+0000 7fd6bbe00700 1 -- 172.21.15.12:0/3208169596 shutdown_connections 2024-03-29T09:54:34.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.724+0000 7fd6bbe00700 1 -- 172.21.15.12:0/3208169596 wait complete. 2024-03-29T09:54:34.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.723+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 --> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe4180de180 con 0x7fe400072f00 2024-03-29T09:54:34.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.724+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 shutdown_connections 2024-03-29T09:54:34.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.724+0000 7f1610400700 1 -- 172.21.15.12:0/368949574 wait complete. 2024-03-29T09:54:34.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.725+0000 7fe41d600700 1 -- 172.21.15.12:0/268053801 <== osd.3 v2:172.21.15.12:6820/43910 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe4180de180 con 0x7fe400072f00 2024-03-29T09:54:34.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.726+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 >> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] conn(0x7fe400072f00 msgr2=0x7fe400075320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.726+0000 7fe426a00700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] conn(0x7fe400072f00 0x7fe400075320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.726+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe40006ad60 msgr2=0x7fe40006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.726+0000 7fe426a00700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe40006ad60 0x7fe40006d220 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe4140011e0 tx=0x7fe414007a80 comp rx=0 tx=0).stop 2024-03-29T09:54:34.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.726+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4180e3240 msgr2=0x7fe4180e1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.726+0000 7fe426a00700 1 --2- 172.21.15.12:0/268053801 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4180e3240 0x7fe4180e1160 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe420064450 tx=0x7fe4200685b0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.726+0000 7fe426000700 1 -- 172.21.15.12:0/268053801 reap_dead start 2024-03-29T09:54:34.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.727+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 --> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff878054150 con 0x7ff860072cc0 2024-03-29T09:54:34.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.727+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 shutdown_connections 2024-03-29T09:54:34.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.727+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 >> 172.21.15.12:0/268053801 conn(0x7fe418055a40 msgr2=0x7fe41804e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.727+0000 7ff876000700 1 -- 172.21.15.12:0/3097006509 <== osd.4 v2:172.21.15.159:6800/44146 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff88004fa10 con 0x7ff860072cc0 2024-03-29T09:54:34.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.728+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] conn(0x7ff860072cc0 msgr2=0x7ff8600750e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.728+0000 7ff87f200700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] conn(0x7ff860072cc0 0x7ff8600750e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.728+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff86006ac90 msgr2=0x7ff86006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.728+0000 7ff87f200700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff86006ac90 0x7ff86006d150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff8780e05a0 tx=0x7ff86c0090e0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.728+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff878053ce0 msgr2=0x7ff8780d2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.728+0000 7ff87f200700 1 --2- 172.21.15.12:0/3097006509 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff878053ce0 0x7ff8780d2d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff878006db0 tx=0x7ff870004750 comp rx=0 tx=0).stop 2024-03-29T09:54:34.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.728+0000 7ff87e800700 1 -- 172.21.15.12:0/3097006509 reap_dead start 2024-03-29T09:54:34.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.728+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 shutdown_connections 2024-03-29T09:54:34.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.728+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 >> 172.21.15.12:0/3097006509 conn(0x7ff878058f40 msgr2=0x7ff87804e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.729+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 shutdown_connections 2024-03-29T09:54:34.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.731+0000 7ff87f200700 1 -- 172.21.15.12:0/3097006509 wait complete. 2024-03-29T09:54:34.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.731+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 shutdown_connections 2024-03-29T09:54:34.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.731+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 --> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f23b80ae890 con 0x7f2398072f00 2024-03-29T09:54:34.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.732+0000 7fe426a00700 1 -- 172.21.15.12:0/268053801 wait complete. 2024-03-29T09:54:34.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.732+0000 7f23aea00700 1 -- 172.21.15.12:0/2336990357 <== osd.5 v2:172.21.15.159:6816/44148 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f23b0007990 con 0x7f2398072f00 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.735+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] conn(0x7f2398072f00 msgr2=0x7f2398075320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.735+0000 7f23bda00700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] conn(0x7f2398072f00 0x7f2398075320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.735+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f239806ace0 msgr2=0x7f239806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.735+0000 7f23bda00700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f239806ace0 0x7f239806d1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f23a80011e0 tx=0x7f23a8009040 comp rx=0 tx=0).stop 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.736+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23b8072910 msgr2=0x7f23b80b11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.736+0000 7f23bda00700 1 --2- 172.21.15.12:0/2336990357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23b8072910 0x7f23b80b11f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f23a400e810 tx=0x7f23a400eb20 comp rx=0 tx=0).stop 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.738+0000 7f23b7400700 1 -- 172.21.15.12:0/2336990357 reap_dead start 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.738+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 shutdown_connections 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.738+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 >> 172.21.15.12:0/2336990357 conn(0x7f23b8070f00 msgr2=0x7f23b81587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.739+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 shutdown_connections 2024-03-29T09:54:34.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.739+0000 7f23bda00700 1 -- 172.21.15.12:0/2336990357 wait complete. 2024-03-29T09:54:34.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.742+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 --> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa694067720 con 0x7fa674072f00 2024-03-29T09:54:34.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.743+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 --> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb0a804ea80 con 0x7fb088072e60 2024-03-29T09:54:34.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.743+0000 7fa68a000700 1 -- 172.21.15.12:0/196952069 <== osd.7 v2:172.21.15.159:6808/44147 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa694067720 con 0x7fa674072f00 2024-03-29T09:54:34.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.743+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] conn(0x7fa674072f00 msgr2=0x7fa674075320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.743+0000 7fa698e00700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] conn(0x7fa674072f00 0x7fa674075320 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.744+0000 7fb09ea00700 1 -- 172.21.15.12:0/3597597593 <== osd.6 v2:172.21.15.159:6817/44159 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb0a804ea80 con 0x7fb088072e60 2024-03-29T09:54:34.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.744+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa67406ace0 msgr2=0x7fa67406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.744+0000 7fa698e00700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa67406ace0 0x7fa67406d1a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa6941fab10 tx=0x7fa680009040 comp rx=0 tx=0).stop 2024-03-29T09:54:34.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.744+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6940b3760 msgr2=0x7fa694154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.744+0000 7fa698e00700 1 --2- 172.21.15.12:0/196952069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6940b3760 0x7fa694154610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa68c015040 tx=0x7fa68c004ad0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.744+0000 7fa692800700 1 -- 172.21.15.12:0/196952069 reap_dead start 2024-03-29T09:54:34.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.744+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 shutdown_connections 2024-03-29T09:54:34.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.744+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 >> 172.21.15.12:0/196952069 conn(0x7fa6940b0d70 msgr2=0x7fa6940cb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.745+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] conn(0x7fb088072e60 msgr2=0x7fb088075280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.745+0000 7fb0a7e00700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] conn(0x7fb088072e60 0x7fb088075280 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.745+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb08806acc0 msgr2=0x7fb08806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.745+0000 7fb0a7e00700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb08806acc0 0x7fb08806d180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0000f80 tx=0x7fb0a0012000 comp rx=0 tx=0).stop 2024-03-29T09:54:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.745+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0a8159400 msgr2=0x7fb0a81f3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.745+0000 7fb0a7e00700 1 --2- 172.21.15.12:0/3597597593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0a8159400 0x7fb0a81f3160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb09400b4a0 tx=0x7fb09400b7b0 comp rx=0 tx=0).stop 2024-03-29T09:54:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.745+0000 7fb0a7400700 1 -- 172.21.15.12:0/3597597593 reap_dead start 2024-03-29T09:54:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.746+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 shutdown_connections 2024-03-29T09:54:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.746+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 >> 172.21.15.12:0/3597597593 conn(0x7fb0a806df90 msgr2=0x7fb0a80adb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.746+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 shutdown_connections 2024-03-29T09:54:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.746+0000 7fa698e00700 1 -- 172.21.15.12:0/196952069 wait complete. 2024-03-29T09:54:34.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.747+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 shutdown_connections 2024-03-29T09:54:34.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:34.747+0000 7fb0a7e00700 1 -- 172.21.15.12:0/3597597593 wait complete. 2024-03-29T09:54:34.764 INFO:teuthology.orchestra.run.smithi012.stdout:55834574853 2024-03-29T09:54:34.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T09:54:34.765 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:34.765 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-29T09:54:34.766 INFO:teuthology.orchestra.run.smithi012.stdout:55834574855 2024-03-29T09:54:34.766 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T09:54:34.767 INFO:teuthology.orchestra.run.smithi012.stdout:55834574855 2024-03-29T09:54:34.767 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-29T09:54:34.770 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:34.770 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-29T09:54:34.776 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:34.776 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-29T09:54:34.777 INFO:teuthology.orchestra.run.smithi012.stdout:55834574855 2024-03-29T09:54:34.777 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-29T09:54:34.778 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:34.778 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-29T09:54:35.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.118+0000 7f4363200700 1 Processor -- start 2024-03-29T09:54:35.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.118+0000 7f4363200700 1 -- start start 2024-03-29T09:54:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.121+0000 7f4363200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f435c053790 0x7f435c053c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.121+0000 7f4363200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c03ee20 con 0x7f435c0063d0 2024-03-29T09:54:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.122+0000 7f4363200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c054150 con 0x7f435c0e32c0 2024-03-29T09:54:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.122+0000 7f4363200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435c0578f0 con 0x7f435c053790 2024-03-29T09:54:35.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.121+0000 7fdc75800700 1 Processor -- start 2024-03-29T09:54:35.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.121+0000 7fdc75800700 1 -- start start 2024-03-29T09:54:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.122+0000 7f4361e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f435c0e32c0 0x7f435c058700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46458/0 (socket says 172.21.15.12:46458) 2024-03-29T09:54:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.122+0000 7f4361e00700 1 -- 172.21.15.12:0/3548341623 learned_addr learned my addr 172.21.15.12:0/3548341623 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.123+0000 7f4361400700 1 --2- 172.21.15.12:0/3548341623 >> v2:172.21.15.159:3300/0 conn(0x7f435c053790 0x7f435c053c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.123+0000 7fdc75800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdc680e32f0 0x7fdc68052c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.123+0000 7fdc75800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc680e36d0 con 0x7fdc6804ffa0 2024-03-29T09:54:35.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.123+0000 7fdc75800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc680536c0 con 0x7fdc680531b0 2024-03-29T09:54:35.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.123+0000 7fdc75800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc680539a0 con 0x7fdc680e32f0 2024-03-29T09:54:35.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.123+0000 7fdc6ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdc680e32f0 0x7fdc68052c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.123+0000 7fdc6ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdc680e32f0 0x7fdc68052c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46826/0 (socket says 172.21.15.12:46826) 2024-03-29T09:54:35.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.123+0000 7fdc6ea00700 1 -- 172.21.15.12:0/131145763 learned_addr learned my addr 172.21.15.12:0/131145763 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.123+0000 7f4360a00700 1 -- 172.21.15.12:0/3548341623 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217362678 0 0) 0x7f4354001110 con 0x7f435c0e32c0 2024-03-29T09:54:35.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.124+0000 7f4361400700 1 -- 172.21.15.12:0/3548341623 >> v1:172.21.15.159:6789/0 conn(0x7f435c0e32c0 legacy=0x7f435c058700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.124+0000 7f4361400700 1 -- 172.21.15.12:0/3548341623 >> v1:172.21.15.12:6789/0 conn(0x7f435c0063d0 legacy=0x7f435c006860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:35.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.125+0000 7f4361400700 1 -- 172.21.15.12:0/3548341623 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c03e860 con 0x7f435c053790 2024-03-29T09:54:35.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.125+0000 7f4361400700 1 --2- 172.21.15.12:0/3548341623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c053790 0x7f435c053c10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f436406a090 tx=0x7f4364068df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28fe231f656ff2f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.125+0000 7fdc6d600700 1 -- 172.21.15.12:0/131145763 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930304061 0 0) 0x7fdc680536c0 con 0x7fdc680531b0 2024-03-29T09:54:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.125+0000 7fdc6d600700 1 -- 172.21.15.12:0/131145763 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc540035b0 con 0x7fdc680531b0 2024-03-29T09:54:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.125+0000 7fdc6ea00700 1 -- 172.21.15.12:0/131145763 >> v1:172.21.15.159:6789/0 conn(0x7fdc680531b0 legacy=0x7fdc6804f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.125+0000 7fdc6ea00700 1 -- 172.21.15.12:0/131145763 >> v1:172.21.15.12:6789/0 conn(0x7fdc6804ffa0 legacy=0x7fdc68037bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.125+0000 7fdc6ea00700 1 -- 172.21.15.12:0/131145763 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc68053c80 con 0x7fdc680e32f0 2024-03-29T09:54:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.126+0000 7fdc6ea00700 1 --2- 172.21.15.12:0/131145763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc680e32f0 0x7fdc68052c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc70062370 tx=0x7fdc70068df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23c95716dfc45802 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.126+0000 7f4360a00700 1 -- 172.21.15.12:0/3548341623 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436408e030 con 0x7f435c053790 2024-03-29T09:54:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.126+0000 7f4360a00700 1 -- 172.21.15.12:0/3548341623 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436404f060 con 0x7f435c053790 2024-03-29T09:54:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.126+0000 7f4360a00700 1 -- 172.21.15.12:0/3548341623 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436406a790 con 0x7f435c053790 2024-03-29T09:54:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.126+0000 7fdc6d600700 1 -- 172.21.15.12:0/131145763 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7008e030 con 0x7fdc680e32f0 2024-03-29T09:54:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.126+0000 7fdc6d600700 1 -- 172.21.15.12:0/131145763 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7004f060 con 0x7fdc680e32f0 2024-03-29T09:54:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.126+0000 7fdc6d600700 1 -- 172.21.15.12:0/131145763 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7006a790 con 0x7fdc680e32f0 2024-03-29T09:54:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7f4363200700 1 -- 172.21.15.12:0/3548341623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c053790 msgr2=0x7f435c053c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7f4363200700 1 --2- 172.21.15.12:0/3548341623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c053790 0x7f435c053c10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f436406a090 tx=0x7f4364068df0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7fdc75800700 1 -- 172.21.15.12:0/131145763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc680e32f0 msgr2=0x7fdc68052c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7fdc75800700 1 --2- 172.21.15.12:0/131145763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc680e32f0 0x7fdc68052c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc70062370 tx=0x7fdc70068df0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7fe416200700 1 Processor -- start 2024-03-29T09:54:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7f4363200700 1 -- 172.21.15.12:0/3548341623 shutdown_connections 2024-03-29T09:54:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7f4363200700 1 --2- 172.21.15.12:0/3548341623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c053790 0x7f435c053c10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7f4363200700 1 -- 172.21.15.12:0/3548341623 >> 172.21.15.12:0/3548341623 conn(0x7f435c0559c0 msgr2=0x7f435c055da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7f3cb8a00700 1 Processor -- start 2024-03-29T09:54:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7fdc75800700 1 -- 172.21.15.12:0/131145763 shutdown_connections 2024-03-29T09:54:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7fdc75800700 1 --2- 172.21.15.12:0/131145763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc680e32f0 0x7fdc68052c70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.127+0000 7fdc75800700 1 -- 172.21.15.12:0/131145763 >> 172.21.15.12:0/131145763 conn(0x7fdc68009fd0 msgr2=0x7fdc6800a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.128+0000 7f3cb8a00700 1 -- start start 2024-03-29T09:54:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.129+0000 7f4363200700 1 -- 172.21.15.12:0/3548341623 shutdown_connections 2024-03-29T09:54:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.129+0000 7f4363200700 1 -- 172.21.15.12:0/3548341623 wait complete. 2024-03-29T09:54:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.128+0000 7fe416200700 1 -- start start 2024-03-29T09:54:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.129+0000 7fdc75800700 1 -- 172.21.15.12:0/131145763 shutdown_connections 2024-03-29T09:54:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.129+0000 7fdc75800700 1 -- 172.21.15.12:0/131145763 wait complete. 2024-03-29T09:54:35.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.130+0000 7f3cb8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac033f60 0x7f3cac0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.130+0000 7f3cb8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3cac0481c0 0x7f3cac0eb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.130+0000 7f3cb8a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3cac047c70 0x7f3cac005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.130+0000 7f3cb8a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac04fb30 con 0x7f3cac0481c0 2024-03-29T09:54:35.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.130+0000 7f3cb8a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac050260 con 0x7f3cac047c70 2024-03-29T09:54:35.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.130+0000 7f3cb8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac04fdd0 con 0x7f3cac033f60 2024-03-29T09:54:35.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.132+0000 7f4363200700 1 Processor -- start 2024-03-29T09:54:35.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.132+0000 7fdc75800700 1 Processor -- start 2024-03-29T09:54:35.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.132+0000 7f3cb2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3cac0481c0 0x7f3cac0eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.132+0000 7f3cb2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3cac0481c0 0x7f3cac0eb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40658/0 (socket says 172.21.15.12:40658) 2024-03-29T09:54:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.132+0000 7f3cb2a00700 1 -- 172.21.15.12:0/1893043289 learned_addr learned my addr 172.21.15.12:0/1893043289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.132+0000 7f3cb3e00700 1 --2- 172.21.15.12:0/1893043289 >> v2:172.21.15.159:3300/0 conn(0x7f3cac047c70 0x7f3cac005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.133+0000 7f3cb2a00700 1 -- 172.21.15.12:0/1893043289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac033f60 msgr2=0x7f3cac0eb1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T09:54:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.133+0000 7f3cb2a00700 1 --2- 172.21.15.12:0/1893043289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac033f60 0x7f3cac0eb1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.133+0000 7f3cb2a00700 1 -- 172.21.15.12:0/1893043289 >> v2:172.21.15.159:3300/0 conn(0x7f3cac047c70 msgr2=0x7f3cac005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.133+0000 7f3cb2a00700 1 --2- 172.21.15.12:0/1893043289 >> v2:172.21.15.159:3300/0 conn(0x7f3cac047c70 0x7f3cac005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.133+0000 7f3cb2a00700 1 -- 172.21.15.12:0/1893043289 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac0ec070 con 0x7f3cac0481c0 2024-03-29T09:54:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.133+0000 7fdc75800700 1 -- start start 2024-03-29T09:54:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.133+0000 7f3cb2a00700 1 --2- 172.21.15.12:0/1893043289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cac0481c0 0x7f3cac0eb710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3cac034530 tx=0x7f3ca0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f4fc6f240014e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.131+0000 7fe416200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe408033f60 0x7fe4080eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.131+0000 7fe416200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4080481c0 0x7fe4080eb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.131+0000 7fe416200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40804fdd0 con 0x7fe408047c70 2024-03-29T09:54:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.131+0000 7fe416200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40804fb30 con 0x7fe408033f60 2024-03-29T09:54:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.131+0000 7fe416200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe408050260 con 0x7fe4080481c0 2024-03-29T09:54:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.131+0000 7fe414e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe408033f60 0x7fe4080eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.132+0000 7fe415800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe408047c70 0x7fe408005ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58186/0 (socket says 172.21.15.12:58186) 2024-03-29T09:54:35.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.132+0000 7fe415800700 1 -- 172.21.15.12:0/895694488 learned_addr learned my addr 172.21.15.12:0/895694488 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:35.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.134+0000 7f3cb2000700 1 -- 172.21.15.12:0/1893043289 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca000b870 con 0x7f3cac0481c0 2024-03-29T09:54:35.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.132+0000 7fe40fe00700 1 --2- 172.21.15.12:0/895694488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4080481c0 0x7fe4080eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.133+0000 7fe40fe00700 1 -- 172.21.15.12:0/895694488 >> v2:172.21.15.12:3300/0 conn(0x7fe408033f60 msgr2=0x7fe4080eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:35.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.133+0000 7fe40fe00700 1 --2- 172.21.15.12:0/895694488 >> v2:172.21.15.12:3300/0 conn(0x7fe408033f60 0x7fe4080eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.133+0000 7fe40fe00700 1 -- 172.21.15.12:0/895694488 >> v1:172.21.15.12:6789/0 conn(0x7fe408047c70 legacy=0x7fe408005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:35.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.134+0000 7fe40fe00700 1 -- 172.21.15.12:0/895694488 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4080ec070 con 0x7fe4080481c0 2024-03-29T09:54:35.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.134+0000 7f3cb2000700 1 -- 172.21.15.12:0/1893043289 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca000b9d0 con 0x7f3cac0481c0 2024-03-29T09:54:35.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.134+0000 7f3cb2000700 1 -- 172.21.15.12:0/1893043289 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca000b870 con 0x7f3cac0481c0 2024-03-29T09:54:35.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.135+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1893043289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cac0481c0 msgr2=0x7f3cac0eb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.135+0000 7f3cb8a00700 1 --2- 172.21.15.12:0/1893043289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cac0481c0 0x7f3cac0eb710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3cac034530 tx=0x7f3ca0009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1893043289 shutdown_connections 2024-03-29T09:54:35.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7f3cb8a00700 1 --2- 172.21.15.12:0/1893043289 >> v2:172.21.15.159:3300/0 conn(0x7f3cac047c70 0x7f3cac005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7f3cb8a00700 1 --2- 172.21.15.12:0/1893043289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac033f60 0x7f3cac0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7f3cb8a00700 1 --2- 172.21.15.12:0/1893043289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cac0481c0 0x7f3cac0eb710 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1893043289 >> 172.21.15.12:0/1893043289 conn(0x7f3cac054b20 msgr2=0x7f3cac054f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.134+0000 7fe40fe00700 1 --2- 172.21.15.12:0/895694488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4080481c0 0x7fe4080eb710 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009840 tx=0x7fe3fc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b14cd9cb93acda61 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.134+0000 7fe414e00700 1 --2- 172.21.15.12:0/895694488 >> v2:172.21.15.12:3300/0 conn(0x7fe408033f60 0x7fe4080eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:35.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.135+0000 7fe40f400700 1 -- 172.21.15.12:0/895694488 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc011830 con 0x7fe4080481c0 2024-03-29T09:54:35.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.135+0000 7fe40f400700 1 -- 172.21.15.12:0/895694488 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc011990 con 0x7fe4080481c0 2024-03-29T09:54:35.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.135+0000 7fe40f400700 1 -- 172.21.15.12:0/895694488 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc011d20 con 0x7fe4080481c0 2024-03-29T09:54:35.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7fe416200700 1 -- 172.21.15.12:0/895694488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4080481c0 msgr2=0x7fe4080eb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7fe416200700 1 --2- 172.21.15.12:0/895694488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4080481c0 0x7fe4080eb710 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009840 tx=0x7fe3fc009b50 comp rx=0 tx=0).stop 2024-03-29T09:54:35.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7fe416200700 1 -- 172.21.15.12:0/895694488 shutdown_connections 2024-03-29T09:54:35.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7fe416200700 1 --2- 172.21.15.12:0/895694488 >> v2:172.21.15.12:3300/0 conn(0x7fe408033f60 0x7fe4080eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7fe416200700 1 --2- 172.21.15.12:0/895694488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4080481c0 0x7fe4080eb710 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.136+0000 7fe416200700 1 -- 172.21.15.12:0/895694488 >> 172.21.15.12:0/895694488 conn(0x7fe408054b20 msgr2=0x7fe408054f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.137+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1893043289 shutdown_connections 2024-03-29T09:54:35.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.137+0000 7fdc75800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc6804ffa0 0x7fdc680d6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.137+0000 7fdc75800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc680531b0 0x7fdc68034ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.137+0000 7fdc75800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc680e32f0 0x7fdc68035430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.137+0000 7fdc6ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc6804ffa0 0x7fdc680d6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.137+0000 7fdc6ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc6804ffa0 0x7fdc680d6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46844/0 (socket says 172.21.15.12:46844) 2024-03-29T09:54:35.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.137+0000 7fe416200700 1 -- 172.21.15.12:0/895694488 shutdown_connections 2024-03-29T09:54:35.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.137+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1893043289 wait complete. 2024-03-29T09:54:35.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.138+0000 7f28d7200700 1 Processor -- start 2024-03-29T09:54:35.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.137+0000 7fdc6e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc680531b0 0x7fdc68034ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.138+0000 7fdc6e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc680531b0 0x7fdc68034ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40664/0 (socket says 172.21.15.12:40664) 2024-03-29T09:54:35.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.138+0000 7fdc6f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc680e32f0 0x7fdc68035430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.138+0000 7fdc75800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680e5530 con 0x7fdc680531b0 2024-03-29T09:54:35.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.138+0000 7fdc6f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc680e32f0 0x7fdc68035430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51472/0 (socket says 172.21.15.12:51472) 2024-03-29T09:54:35.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.138+0000 7fdc6f400700 1 -- 172.21.15.12:0/2236488987 learned_addr learned my addr 172.21.15.12:0/2236488987 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.138+0000 7fdc75800700 1 -- 172.21.15.12:0/2236488987 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680e5810 con 0x7fdc6804ffa0 2024-03-29T09:54:35.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.139+0000 7fdc75800700 1 -- 172.21.15.12:0/2236488987 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680e5af0 con 0x7fdc680e32f0 2024-03-29T09:54:35.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.138+0000 7f28d7200700 1 -- start start 2024-03-29T09:54:35.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.140+0000 7f28d7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28d00d2540 0x7f28d01687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.140+0000 7f28d7200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d0161420 con 0x7f28d00d16a0 2024-03-29T09:54:35.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.140+0000 7f28d7200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d0161700 con 0x7f28d015d9d0 2024-03-29T09:54:35.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.140+0000 7f28d7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d00d4340 con 0x7f28d00d2540 2024-03-29T09:54:35.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.137+0000 7fe416200700 1 -- 172.21.15.12:0/895694488 wait complete. 2024-03-29T09:54:35.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.140+0000 7fe416200700 1 Processor -- start 2024-03-29T09:54:35.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.140+0000 7f28cf400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f28d00d16a0 0x7f28d00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58190/0 (socket says 172.21.15.12:58190) 2024-03-29T09:54:35.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.140+0000 7f28cf400700 1 -- 172.21.15.12:0/2698207163 learned_addr learned my addr 172.21.15.12:0/2698207163 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:35.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.141+0000 7fe416200700 1 -- start start 2024-03-29T09:54:35.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.141+0000 7f28d5200700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28d00d2540 0x7f28d01687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.141+0000 7fdc6f400700 1 -- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc6804ffa0 msgr2=0x7fdc680d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.141+0000 7fdc6f400700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc6804ffa0 0x7fdc680d6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9456789 0 0) 0x7f28d0161700 con 0x7f28d015d9d0 2024-03-29T09:54:35.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b40035b0 con 0x7f28d015d9d0 2024-03-29T09:54:35.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192527365 0 0) 0x7f28d0161420 con 0x7f28d00d16a0 2024-03-29T09:54:35.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d0161700 con 0x7f28d00d16a0 2024-03-29T09:54:35.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3740195370 0 0) 0x7f28b40035b0 con 0x7f28d015d9d0 2024-03-29T09:54:35.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28d00d2540 msgr2=0x7f28d01687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:35.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28d00d2540 0x7f28d01687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 >> v1:172.21.15.12:6789/0 conn(0x7f28d00d16a0 legacy=0x7f28d00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d01619e0 con 0x7f28d015d9d0 2024-03-29T09:54:35.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f28c0001670 con 0x7f28d015d9d0 2024-03-29T09:54:35.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.142+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 >> v1:172.21.15.159:6789/0 conn(0x7f28d015d9d0 legacy=0x7f28d00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f28cea00700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b4004bf0 0x7f28b4004ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7fdc6f400700 1 -- 172.21.15.12:0/2236488987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc680531b0 msgr2=0x7fdc68034ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7fdc6f400700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc680531b0 0x7fdc68034ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7fdc6f400700 1 -- 172.21.15.12:0/2236488987 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc700686c0 con 0x7fdc680e32f0 2024-03-29T09:54:35.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f28cea00700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28b40037f0 0x7f28b4003c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f4363200700 1 -- start start 2024-03-29T09:54:35.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f28cea00700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28b40041b0 0x7f28b400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b400c000 con 0x7f28b40041b0 2024-03-29T09:54:35.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b400c170 con 0x7f28b40037f0 2024-03-29T09:54:35.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f28cfe00700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b4004bf0 0x7f28b4004ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b400c330 con 0x7f28b4004bf0 2024-03-29T09:54:35.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f3cb8a00700 1 Processor -- start 2024-03-29T09:54:35.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f28d5200700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28d00d2540 0x7f28d01687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:35.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7fe416200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe408033f60 0x7fe4080ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.143+0000 7f28cf400700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28b40037f0 0x7f28b4003c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7fe416200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe408047c70 0x7fe4080352e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7fe414e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe408033f60 0x7fe4080ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7f28cf400700 1 -- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b4004bf0 msgr2=0x7f28b4004ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:35.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7f28cf400700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b4004bf0 0x7f28b4004ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7f3cb8a00700 1 -- start start 2024-03-29T09:54:35.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7fe414e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe408033f60 0x7fe4080ea8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51502/0 (socket says 172.21.15.12:51502) 2024-03-29T09:54:35.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7f28cf400700 1 -- 172.21.15.12:0/2698207163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28b40041b0 msgr2=0x7f28b400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:35.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7f28cf400700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28b40041b0 0x7f28b400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7fe40fe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe408047c70 0x7fe4080352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7f28cf400700 1 -- 172.21.15.12:0/2698207163 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b40035b0 con 0x7f28b40037f0 2024-03-29T09:54:35.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.144+0000 7fe40fe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe408047c70 0x7fe4080352e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46866/0 (socket says 172.21.15.12:46866) 2024-03-29T09:54:35.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.145+0000 7fdc6e000700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc680531b0 0x7fdc68034ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:35.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.145+0000 7fe414e00700 1 -- 172.21.15.12:0/1465324041 learned_addr learned my addr 172.21.15.12:0/1465324041 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.145+0000 7fe416200700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4080481c0 0x7fe408185330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.145+0000 7f28cf400700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28b40037f0 0x7f28b4003c70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f28d00d1ae0 tx=0x7f28b4005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c7a41934ae6d498 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.145+0000 7fdc6f400700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc680e32f0 0x7fdc68035430 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00c410 tx=0x7fdc5c00c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.145+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc004f40 con 0x7f28b40037f0 2024-03-29T09:54:35.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.145+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408035820 con 0x7fe4080481c0 2024-03-29T09:54:35.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.145+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28bc005a70 con 0x7f28b40037f0 2024-03-29T09:54:35.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.145+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408185870 con 0x7fe408047c70 2024-03-29T09:54:35.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.146+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080e3e30 con 0x7fe408033f60 2024-03-29T09:54:35.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.146+0000 7f28cea00700 1 -- 172.21.15.12:0/2698207163 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc005e20 con 0x7f28b40037f0 2024-03-29T09:54:35.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.146+0000 7fe414e00700 1 -- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe408047c70 msgr2=0x7fe4080352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.146+0000 7fe414e00700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe408047c70 0x7fe4080352e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.146+0000 7fe414e00700 1 -- 172.21.15.12:0/1465324041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4080481c0 msgr2=0x7fe408185330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.146+0000 7fe414e00700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4080481c0 0x7fe408185330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.146+0000 7fe414e00700 1 -- 172.21.15.12:0/1465324041 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4040094a0 con 0x7fe408033f60 2024-03-29T09:54:35.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.146+0000 7f4363200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c0063d0 0x7f435c04e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7f4363200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f435c053790 0x7f435c0d5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7f4363200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f435c0e32c0 0x7f435c0dc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7f4363200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c0e3d30 con 0x7f435c0e32c0 2024-03-29T09:54:35.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7f4363200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c04eff0 con 0x7f435c0063d0 2024-03-29T09:54:35.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7f28d7200700 1 -- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28b40037f0 msgr2=0x7f28b4003c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7f28d7200700 1 --2- 172.21.15.12:0/2698207163 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28b40037f0 0x7f28b4003c70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f28d00d1ae0 tx=0x7f28b4005880 comp rx=0 tx=0).stop 2024-03-29T09:54:35.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7f28d5200700 1 -- 172.21.15.12:0/2698207163 reap_dead start 2024-03-29T09:54:35.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7f4363200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c04ea30 con 0x7f435c053790 2024-03-29T09:54:35.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7f28d7200700 1 -- 172.21.15.12:0/2698207163 shutdown_connections 2024-03-29T09:54:35.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7f28d7200700 1 -- 172.21.15.12:0/2698207163 >> 172.21.15.12:0/2698207163 conn(0x7f28d008aff0 msgr2=0x7f28d00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fdc66a00700 1 -- 172.21.15.12:0/2236488987 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c0052d0 con 0x7fdc680e32f0 2024-03-29T09:54:35.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fe414e00700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe408033f60 0x7fe4080ea8c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe40400ebe0 tx=0x7fe40400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fe40cc00700 1 -- 172.21.15.12:0/1465324041 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40400c850 con 0x7fe408033f60 2024-03-29T09:54:35.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fe40cc00700 1 -- 172.21.15.12:0/1465324041 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40400c9b0 con 0x7fe408033f60 2024-03-29T09:54:35.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fe40cc00700 1 -- 172.21.15.12:0/1465324041 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40400cd40 con 0x7fe408033f60 2024-03-29T09:54:35.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3fc0094a0 con 0x7fe408033f60 2024-03-29T09:54:35.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4080e43d0 con 0x7fe408033f60 2024-03-29T09:54:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fdc66a00700 1 -- 172.21.15.12:0/2236488987 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5c005450 con 0x7fdc680e32f0 2024-03-29T09:54:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fdc66a00700 1 -- 172.21.15.12:0/2236488987 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c0038c0 con 0x7fdc680e32f0 2024-03-29T09:54:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fdc75800700 1 -- 172.21.15.12:0/2236488987 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc680e5dd0 con 0x7fdc680e32f0 2024-03-29T09:54:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.147+0000 7fdc75800700 1 -- 172.21.15.12:0/2236488987 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc680e65d0 con 0x7fdc680e32f0 2024-03-29T09:54:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.148+0000 7f4361e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c0063d0 0x7f435c04e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.148+0000 7f4361400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f435c053790 0x7f435c0d5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.148+0000 7f3cb8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cac033f60 0x7f3cac0d87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.148+0000 7f3cb8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac047c70 0x7f3cac04ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.148+0000 7f3cb8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cac0481c0 0x7f3cac038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.148+0000 7f3cb8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac03a0f0 con 0x7f3cac033f60 2024-03-29T09:54:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.148+0000 7f3cb8a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac049400 con 0x7f3cac0481c0 2024-03-29T09:54:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.148+0000 7f3cb8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0496e0 con 0x7f3cac047c70 2024-03-29T09:54:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.149+0000 7f28d7200700 1 -- 172.21.15.12:0/2698207163 shutdown_connections 2024-03-29T09:54:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.148+0000 7f4361e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c0063d0 0x7f435c04e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46880/0 (socket says 172.21.15.12:46880) 2024-03-29T09:54:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.148+0000 7f4361e00700 1 -- 172.21.15.12:0/3707603928 learned_addr learned my addr 172.21.15.12:0/3707603928 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.149+0000 7f4362800700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f435c0e32c0 0x7f435c0dc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.149+0000 7f4361e00700 1 -- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f435c053790 msgr2=0x7f435c0d5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.149+0000 7f4361e00700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f435c053790 0x7f435c0d5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.149+0000 7f4361e00700 1 -- 172.21.15.12:0/3707603928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f435c0e32c0 msgr2=0x7f435c0dc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.149+0000 7f4361e00700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f435c0e32c0 0x7f435c0dc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.149+0000 7f4361e00700 1 -- 172.21.15.12:0/3707603928 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43640686c0 con 0x7f435c0063d0 2024-03-29T09:54:35.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.151+0000 7f4361e00700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c0063d0 0x7f435c04e450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f435400c730 tx=0x7f435400caf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.150+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4080053f0 con 0x7fe408033f60 2024-03-29T09:54:35.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.151+0000 7fe40cc00700 1 -- 172.21.15.12:0/1465324041 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe404013470 con 0x7fe408033f60 2024-03-29T09:54:35.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.150+0000 7f3cb3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cac033f60 0x7f3cac0d87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.150+0000 7f3cb2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac047c70 0x7f3cac04ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.150+0000 7f3cb2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac047c70 0x7f3cac04ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51508/0 (socket says 172.21.15.12:51508) 2024-03-29T09:54:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.150+0000 7f3cb2a00700 1 -- 172.21.15.12:0/1723956403 learned_addr learned my addr 172.21.15.12:0/1723956403 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.151+0000 7f3cb2a00700 1 -- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cac0481c0 msgr2=0x7f3cac038cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.151+0000 7f3cb2a00700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cac0481c0 0x7f3cac038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.151+0000 7f3cb2a00700 1 -- 172.21.15.12:0/1723956403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cac033f60 msgr2=0x7f3cac0d87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.151+0000 7f3cb2a00700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cac033f60 0x7f3cac0d87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.151+0000 7f3cb2a00700 1 -- 172.21.15.12:0/1723956403 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca0009550 con 0x7f3cac047c70 2024-03-29T09:54:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.151+0000 7f3cb3400700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cac033f60 0x7f3cac0d87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.152+0000 7f3cb2a00700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac047c70 0x7f3cac04ad40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009840 tx=0x7f3ca0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.152+0000 7f435a000700 1 -- 172.21.15.12:0/3707603928 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354004c80 con 0x7f435c0063d0 2024-03-29T09:54:35.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.152+0000 7f435a000700 1 -- 172.21.15.12:0/3707603928 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4354004de0 con 0x7f435c0063d0 2024-03-29T09:54:35.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.152+0000 7f435a000700 1 -- 172.21.15.12:0/3707603928 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43540072f0 con 0x7f435c0063d0 2024-03-29T09:54:35.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.149+0000 7f28d7200700 1 -- 172.21.15.12:0/2698207163 wait complete. 2024-03-29T09:54:35.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.152+0000 7f4363200700 1 -- 172.21.15.12:0/3707603928 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c04ed10 con 0x7f435c0063d0 2024-03-29T09:54:35.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.151+0000 7fdc5be00700 1 -- 172.21.15.12:0/2236488987 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc440052f0 con 0x7fdc680e32f0 2024-03-29T09:54:35.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.153+0000 7f28d7200700 1 Processor -- start 2024-03-29T09:54:35.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.153+0000 7fe40cc00700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3f006ad60 0x7fe3f006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.153+0000 7f28d7200700 1 -- start start 2024-03-29T09:54:35.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.153+0000 7fe40cc00700 1 -- 172.21.15.12:0/1465324041 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4040893d0 con 0x7fe408033f60 2024-03-29T09:54:35.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.154+0000 7f4363200700 1 -- 172.21.15.12:0/3707603928 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c0d57b0 con 0x7f435c0063d0 2024-03-29T09:54:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.154+0000 7fe40fe00700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3f006ad60 0x7fe3f006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.155+0000 7fe40fe00700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3f006ad60 0x7fe3f006d220 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe408048620 tx=0x7fe3fc008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.155+0000 7f28d7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b40037f0 0x7f28d01fd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.154+0000 7fdc66a00700 1 -- 172.21.15.12:0/2236488987 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdc5c003a20 con 0x7fdc680e32f0 2024-03-29T09:54:35.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.155+0000 7f28d7200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28d00cb570 0x7f28d00cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f28d7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d01ffe90 0x7f28d0200310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f28cfe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b40037f0 0x7f28d01fd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f28cfe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b40037f0 0x7f28d01fd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51518/0 (socket says 172.21.15.12:51518) 2024-03-29T09:54:35.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.155+0000 7fdc66a00700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5006ad60 0x7fdc5006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7fdc66a00700 1 -- 172.21.15.12:0/2236488987 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc5c088870 con 0x7fdc680e32f0 2024-03-29T09:54:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f4363200700 1 -- 172.21.15.12:0/3707603928 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435c0053f0 con 0x7f435c0063d0 2024-03-29T09:54:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f28cf400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28d00cb570 0x7f28d00cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f28d5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d01ffe90 0x7f28d0200310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f28cf400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28d00cb570 0x7f28d00cb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46896/0 (socket says 172.21.15.12:46896) 2024-03-29T09:54:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f28cf400700 1 -- 172.21.15.12:0/411260980 learned_addr learned my addr 172.21.15.12:0/411260980 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d01fd640 con 0x7f28d01ffe90 2024-03-29T09:54:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7fdc6ea00700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5006ad60 0x7fdc5006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d00cbf30 con 0x7f28d00cb570 2024-03-29T09:54:35.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.156+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d01e7400 con 0x7f28b40037f0 2024-03-29T09:54:35.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.157+0000 7f28cfe00700 1 -- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28d00cb570 msgr2=0x7f28d00cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.157+0000 7f28cfe00700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28d00cb570 0x7f28d00cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.157+0000 7fdc6ea00700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5006ad60 0x7fdc5006d220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdc68007a80 tx=0x7fdc7006d000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.157+0000 7f28cfe00700 1 -- 172.21.15.12:0/411260980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d01ffe90 msgr2=0x7f28d0200310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.157+0000 7f28cfe00700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d01ffe90 0x7f28d0200310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.157+0000 7f28cfe00700 1 -- 172.21.15.12:0/411260980 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d01e76e0 con 0x7f28b40037f0 2024-03-29T09:54:35.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.157+0000 7f28d5200700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d01ffe90 0x7f28d0200310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:35.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.158+0000 7f435a000700 1 -- 172.21.15.12:0/3707603928 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4354007450 con 0x7f435c0063d0 2024-03-29T09:54:35.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.158+0000 7f28cfe00700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b40037f0 0x7f28d01fd100 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28c000e7b0 tx=0x7f28c000eac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.158+0000 7fdc66a00700 1 -- 172.21.15.12:0/2236488987 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdc5c053b80 con 0x7fdc680e32f0 2024-03-29T09:54:35.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.159+0000 7f435a000700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f434406ace0 0x7f434406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.159+0000 7f435a000700 1 -- 172.21.15.12:0/3707603928 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4354088fb0 con 0x7f435c0063d0 2024-03-29T09:54:35.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.159+0000 7f4361400700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f434406ace0 0x7f434406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.160+0000 7f4361400700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f434406ace0 0x7f434406d1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f435c0086e0 tx=0x7f436406e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.163+0000 7f28c7e00700 1 -- 172.21.15.12:0/411260980 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c00052e0 con 0x7f28b40037f0 2024-03-29T09:54:35.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.163+0000 7f28c7e00700 1 -- 172.21.15.12:0/411260980 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c0005440 con 0x7f28b40037f0 2024-03-29T09:54:35.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.164+0000 7fe40cc00700 1 -- 172.21.15.12:0/1465324041 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe404054590 con 0x7fe408033f60 2024-03-29T09:54:35.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.164+0000 7f3ca7400700 1 -- 172.21.15.12:0/1723956403 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0011a30 con 0x7f3cac047c70 2024-03-29T09:54:35.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.163+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d01e79c0 con 0x7f28b40037f0 2024-03-29T09:54:35.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.164+0000 7f3ca7400700 1 -- 172.21.15.12:0/1723956403 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca001f070 con 0x7f3cac047c70 2024-03-29T09:54:35.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.164+0000 7f3ca7400700 1 -- 172.21.15.12:0/1723956403 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0011dd0 con 0x7f3cac047c70 2024-03-29T09:54:35.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.164+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d0200850 con 0x7f28b40037f0 2024-03-29T09:54:35.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.164+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1723956403 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cac04d930 con 0x7f3cac047c70 2024-03-29T09:54:35.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.164+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1723956403 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac04def0 con 0x7f3cac047c70 2024-03-29T09:54:35.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.166+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1723956403 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cac04e110 con 0x7f3cac047c70 2024-03-29T09:54:35.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.167+0000 7f435a000700 1 -- 172.21.15.12:0/3707603928 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4354054170 con 0x7f435c0063d0 2024-03-29T09:54:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.166+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d00d1ca0 con 0x7f28b40037f0 2024-03-29T09:54:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.166+0000 7f28c7e00700 1 -- 172.21.15.12:0/411260980 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c00038c0 con 0x7f28b40037f0 2024-03-29T09:54:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.168+0000 7f28c7e00700 1 -- 172.21.15.12:0/411260980 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f28c0003a60 con 0x7f28b40037f0 2024-03-29T09:54:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.170+0000 7f28c7e00700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f28b006ad60 0x7f28b006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.171+0000 7f28c7e00700 1 -- 172.21.15.12:0/411260980 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28c0088ac0 con 0x7f28b40037f0 2024-03-29T09:54:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.169+0000 7f3ca7400700 1 -- 172.21.15.12:0/1723956403 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3ca0021030 con 0x7f3cac047c70 2024-03-29T09:54:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.170+0000 7f3ca7400700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c9406ad60 0x7f3c9406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.170+0000 7f3ca7400700 1 -- 172.21.15.12:0/1723956403 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ca008ada0 con 0x7f3cac047c70 2024-03-29T09:54:35.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.171+0000 7f28cf400700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f28b006ad60 0x7f28b006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.172+0000 7f28cf400700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f28b006ad60 0x7f28b006d220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28d0162400 tx=0x7f28bc00a000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.172+0000 7f3cb3400700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c9406ad60 0x7f3c9406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.173+0000 7f3ca7400700 1 -- 172.21.15.12:0/1723956403 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ca0056080 con 0x7f3cac047c70 2024-03-29T09:54:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.173+0000 7f3cb3400700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c9406ad60 0x7f3c9406d220 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009910 tx=0x7f3ca8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.178+0000 7f28c7e00700 1 -- 172.21.15.12:0/411260980 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f28c008ae60 con 0x7f28b40037f0 2024-03-29T09:54:35.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.203+0000 7ff008800700 1 Processor -- start 2024-03-29T09:54:35.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.204+0000 7ff008800700 1 -- start start 2024-03-29T09:54:35.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.206+0000 7ff008800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff00015d810 0x7ff0000b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.206+0000 7ff008800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0000cc140 con 0x7ff00006e810 2024-03-29T09:54:35.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.206+0000 7ff008800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00015df80 con 0x7ff000072910 2024-03-29T09:54:35.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.206+0000 7ff008800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00015e260 con 0x7ff00015d810 2024-03-29T09:54:35.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.207+0000 7ff006800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff00006e810 0x7ff0000ce470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58194/0 (socket says 172.21.15.12:58194) 2024-03-29T09:54:35.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.207+0000 7f81f8a00700 1 Processor -- start 2024-03-29T09:54:35.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.208+0000 7ff006800700 1 -- 172.21.15.12:0/3817055532 learned_addr learned my addr 172.21.15.12:0/3817055532 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:35.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.208+0000 7ff005e00700 1 --2- 172.21.15.12:0/3817055532 >> v2:172.21.15.12:3300/0 conn(0x7ff00015d810 0x7ff0000b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.208+0000 7f81f8a00700 1 -- start start 2024-03-29T09:54:35.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.209+0000 7ff004a00700 1 -- 172.21.15.12:0/3817055532 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748903185 0 0) 0x7ff00015df80 con 0x7ff000072910 2024-03-29T09:54:35.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.209+0000 7ff004a00700 1 -- 172.21.15.12:0/3817055532 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe80035b0 con 0x7ff000072910 2024-03-29T09:54:35.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.209+0000 7ff004a00700 1 -- 172.21.15.12:0/3817055532 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656639995 0 0) 0x7ff0000cc140 con 0x7ff00006e810 2024-03-29T09:54:35.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.209+0000 7ff004a00700 1 -- 172.21.15.12:0/3817055532 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff00015df80 con 0x7ff00006e810 2024-03-29T09:54:35.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.210+0000 7ff005e00700 1 -- 172.21.15.12:0/3817055532 >> v1:172.21.15.159:6789/0 conn(0x7ff000072910 legacy=0x7ff00006e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.210+0000 7ff005e00700 1 -- 172.21.15.12:0/3817055532 >> v1:172.21.15.12:6789/0 conn(0x7ff00006e810 legacy=0x7ff0000ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.210+0000 7ff005e00700 1 -- 172.21.15.12:0/3817055532 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0000cbdd0 con 0x7ff00015d810 2024-03-29T09:54:35.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.210+0000 7f81f8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f81ec03a510 0x7f81ec0068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.210+0000 7f81f8a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f81ec0537b0 0x7f81ec053c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.210+0000 7ff005e00700 1 --2- 172.21.15.12:0/3817055532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff00015d810 0x7ff0000b4240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feff0009870 tx=0x7feff0009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224d1bf9df9b691a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.211+0000 7f81f8a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec053320 con 0x7f81ec04f7e0 2024-03-29T09:54:35.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.211+0000 7ff004a00700 1 -- 172.21.15.12:0/3817055532 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0011b40 con 0x7ff00015d810 2024-03-29T09:54:35.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.211+0000 7f81f8a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81ec0071f0 con 0x7f81ec03a510 2024-03-29T09:54:35.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.211+0000 7f81f8a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81ec0074d0 con 0x7f81ec0537b0 2024-03-29T09:54:35.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.211+0000 7f81f1400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f81ec03a510 0x7f81ec0068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.211+0000 7f81f1400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f81ec03a510 0x7f81ec0068c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40722/0 (socket says 172.21.15.12:40722) 2024-03-29T09:54:35.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.211+0000 7f81f1400700 1 -- 172.21.15.12:0/2032870235 learned_addr learned my addr 172.21.15.12:0/2032870235 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.211+0000 7ff004a00700 1 -- 172.21.15.12:0/3817055532 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff0011ca0 con 0x7ff00015d810 2024-03-29T09:54:35.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.211+0000 7ff004a00700 1 -- 172.21.15.12:0/3817055532 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0011b40 con 0x7ff00015d810 2024-03-29T09:54:35.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.212+0000 7ff008800700 1 -- 172.21.15.12:0/3817055532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff00015d810 msgr2=0x7ff0000b4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.212+0000 7ff008800700 1 --2- 172.21.15.12:0/3817055532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff00015d810 0x7ff0000b4240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feff0009870 tx=0x7feff0009b80 comp rx=0 tx=0).stop 2024-03-29T09:54:35.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.212+0000 7f81f2800700 1 --2- 172.21.15.12:0/2032870235 >> v2:172.21.15.159:3300/0 conn(0x7f81ec0537b0 0x7f81ec053c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.213+0000 7ff008800700 1 -- 172.21.15.12:0/3817055532 shutdown_connections 2024-03-29T09:54:35.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.213+0000 7ff008800700 1 --2- 172.21.15.12:0/3817055532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff00015d810 0x7ff0000b4240 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.213+0000 7ff008800700 1 -- 172.21.15.12:0/3817055532 >> 172.21.15.12:0/3817055532 conn(0x7ff000070f00 msgr2=0x7ff000071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.213+0000 7f81f2800700 1 -- 172.21.15.12:0/2032870235 >> v2:172.21.15.12:3300/0 conn(0x7f81ec03a510 msgr2=0x7f81ec0068c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:35.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.213+0000 7f81f2800700 1 --2- 172.21.15.12:0/2032870235 >> v2:172.21.15.12:3300/0 conn(0x7f81ec03a510 0x7f81ec0068c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.214+0000 7ff008800700 1 -- 172.21.15.12:0/3817055532 shutdown_connections 2024-03-29T09:54:35.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.213+0000 7f81f2800700 1 -- 172.21.15.12:0/2032870235 >> v1:172.21.15.12:6789/0 conn(0x7f81ec04f7e0 legacy=0x7f81ec04fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.213+0000 7f81f2800700 1 -- 172.21.15.12:0/2032870235 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec0077b0 con 0x7f81ec0537b0 2024-03-29T09:54:35.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.214+0000 7f81f2800700 1 --2- 172.21.15.12:0/2032870235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec0537b0 0x7f81ec053c60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f81f4050300 tx=0x7f81f406a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cffe98e267d77ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.214+0000 7f81f0a00700 1 -- 172.21.15.12:0/2032870235 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f408e030 con 0x7f81ec0537b0 2024-03-29T09:54:35.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.214+0000 7f81f0a00700 1 -- 172.21.15.12:0/2032870235 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f404f060 con 0x7f81ec0537b0 2024-03-29T09:54:35.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.214+0000 7f705a800700 1 Processor -- start 2024-03-29T09:54:35.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.215+0000 7f81f0a00700 1 -- 172.21.15.12:0/2032870235 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f406a790 con 0x7f81ec0537b0 2024-03-29T09:54:35.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.215+0000 7f705a800700 1 -- start start 2024-03-29T09:54:35.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.215+0000 7f81f8a00700 1 -- 172.21.15.12:0/2032870235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec0537b0 msgr2=0x7f81ec053c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.215+0000 7f81f8a00700 1 --2- 172.21.15.12:0/2032870235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec0537b0 0x7f81ec053c60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f81f4050300 tx=0x7f81f406a660 comp rx=0 tx=0).stop 2024-03-29T09:54:35.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.214+0000 7ff008800700 1 -- 172.21.15.12:0/3817055532 wait complete. 2024-03-29T09:54:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.217+0000 7ff008800700 1 Processor -- start 2024-03-29T09:54:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.217+0000 7ff008800700 1 -- start start 2024-03-29T09:54:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.216+0000 7f705a800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f70540724c0 0x7f70540cf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.216+0000 7f705a800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f70540ac3a0 0x7f70540ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.216+0000 7f705a800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705406df90 con 0x7f705415d7a0 2024-03-29T09:54:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.216+0000 7f705a800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705406eba0 con 0x7f70540724c0 2024-03-29T09:54:35.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.216+0000 7f705a800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705406e270 con 0x7f70540ac3a0 2024-03-29T09:54:35.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.218+0000 7f81f8a00700 1 -- 172.21.15.12:0/2032870235 shutdown_connections 2024-03-29T09:54:35.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.218+0000 7f81f8a00700 1 --2- 172.21.15.12:0/2032870235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec0537b0 0x7f81ec053c60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.218+0000 7f81f8a00700 1 --2- 172.21.15.12:0/2032870235 >> v2:172.21.15.12:3300/0 conn(0x7f81ec03a510 0x7f81ec0068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.218+0000 7f81f8a00700 1 -- 172.21.15.12:0/2032870235 >> 172.21.15.12:0/2032870235 conn(0x7f81ec054b10 msgr2=0x7f81ec054f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.216+0000 7f7053400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f705415d7a0 0x7f70540cea70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46494/0 (socket says 172.21.15.12:46494) 2024-03-29T09:54:35.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.217+0000 7f7053400700 1 -- 172.21.15.12:0/98567556 learned_addr learned my addr 172.21.15.12:0/98567556 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:35.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.218+0000 7f7053e00700 1 --2- 172.21.15.12:0/98567556 >> v2:172.21.15.159:3300/0 conn(0x7f70540ac3a0 0x7f70540ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.219+0000 7f7053e00700 1 -- 172.21.15.12:0/98567556 >> v2:172.21.15.12:3300/0 conn(0x7f70540724c0 msgr2=0x7f70540cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:35.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.219+0000 7f7053e00700 1 --2- 172.21.15.12:0/98567556 >> v2:172.21.15.12:3300/0 conn(0x7f70540724c0 0x7f70540cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.219+0000 7f7053e00700 1 -- 172.21.15.12:0/98567556 >> v1:172.21.15.159:6789/0 conn(0x7f705415d7a0 legacy=0x7f70540cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.219+0000 7f7053e00700 1 -- 172.21.15.12:0/98567556 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705406e550 con 0x7f70540ac3a0 2024-03-29T09:54:35.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.219+0000 7f7052000700 1 -- 172.21.15.12:0/98567556 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297460326 0 0) 0x7f705406df90 con 0x7f705415d7a0 2024-03-29T09:54:35.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.219+0000 7f7053e00700 1 --2- 172.21.15.12:0/98567556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70540ac3a0 0x7f70540ac850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7054070d10 tx=0x7f703c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=132e02269666a7f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.219+0000 7ff008800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff00006e810 0x7ff0001ef030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.219+0000 7f7052000700 1 -- 172.21.15.12:0/98567556 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c011830 con 0x7f70540ac3a0 2024-03-29T09:54:35.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.219+0000 7f7052000700 1 -- 172.21.15.12:0/98567556 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c011990 con 0x7f70540ac3a0 2024-03-29T09:54:35.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.220+0000 7f7052000700 1 -- 172.21.15.12:0/98567556 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c011d20 con 0x7f70540ac3a0 2024-03-29T09:54:35.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.220+0000 7ff008800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff000072910 0x7ff0001f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.220+0000 7ff008800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff00015d810 0x7ff0001e41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.220+0000 7ff008800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0001ef570 con 0x7ff00006e810 2024-03-29T09:54:35.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.220+0000 7ff008800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0001f2310 con 0x7ff000072910 2024-03-29T09:54:35.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.220+0000 7f705a800700 1 -- 172.21.15.12:0/98567556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70540ac3a0 msgr2=0x7f70540ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.220+0000 7f705a800700 1 --2- 172.21.15.12:0/98567556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70540ac3a0 0x7f70540ac850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7054070d10 tx=0x7f703c009b20 comp rx=0 tx=0).stop 2024-03-29T09:54:35.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7f81f8a00700 1 -- 172.21.15.12:0/2032870235 shutdown_connections 2024-03-29T09:54:35.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7ff008800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0000af4b0 con 0x7ff00015d810 2024-03-29T09:54:35.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7ff005400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff000072910 0x7ff0001f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7ff005400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff000072910 0x7ff0001f1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46926/0 (socket says 172.21.15.12:46926) 2024-03-29T09:54:35.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7ff005400700 1 -- 172.21.15.12:0/1506264234 learned_addr learned my addr 172.21.15.12:0/1506264234 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7f705a800700 1 -- 172.21.15.12:0/98567556 shutdown_connections 2024-03-29T09:54:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7f705a800700 1 --2- 172.21.15.12:0/98567556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70540ac3a0 0x7f70540ac850 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7f705a800700 1 --2- 172.21.15.12:0/98567556 >> v2:172.21.15.12:3300/0 conn(0x7f70540724c0 0x7f70540cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7f705a800700 1 -- 172.21.15.12:0/98567556 >> 172.21.15.12:0/98567556 conn(0x7f7054070f00 msgr2=0x7f7054071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7ff006800700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff00015d810 0x7ff0001e41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.222+0000 7ff005e00700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff00006e810 0x7ff0001ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.222+0000 7f705a800700 1 -- 172.21.15.12:0/98567556 shutdown_connections 2024-03-29T09:54:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.222+0000 7ff006800700 1 -- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff000072910 msgr2=0x7ff0001f1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.223+0000 7ff006800700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff000072910 0x7ff0001f1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.223+0000 7ff006800700 1 -- 172.21.15.12:0/1506264234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff00006e810 msgr2=0x7ff0001ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.223+0000 7ff006800700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff00006e810 0x7ff0001ef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.223+0000 7ff006800700 1 -- 172.21.15.12:0/1506264234 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff00094a0 con 0x7ff00015d810 2024-03-29T09:54:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.223+0000 7ff005e00700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff00006e810 0x7ff0001ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.221+0000 7f81f8a00700 1 -- 172.21.15.12:0/2032870235 wait complete. 2024-03-29T09:54:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.223+0000 7ff006800700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff00015d810 0x7ff0001e41d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feff400c3e0 tx=0x7feff400c6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.223+0000 7f81f8a00700 1 Processor -- start 2024-03-29T09:54:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.224+0000 7f81f8a00700 1 -- start start 2024-03-29T09:54:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.224+0000 7feffa000700 1 -- 172.21.15.12:0/1506264234 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4004c80 con 0x7ff00015d810 2024-03-29T09:54:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.224+0000 7feffa000700 1 -- 172.21.15.12:0/1506264234 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff4004de0 con 0x7ff00015d810 2024-03-29T09:54:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.224+0000 7feffa000700 1 -- 172.21.15.12:0/1506264234 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff40072f0 con 0x7ff00015d810 2024-03-29T09:54:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.224+0000 7ff008800700 1 -- 172.21.15.12:0/1506264234 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0000af790 con 0x7ff00015d810 2024-03-29T09:54:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.224+0000 7ff008800700 1 -- 172.21.15.12:0/1506264234 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0000afd50 con 0x7ff00015d810 2024-03-29T09:54:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.224+0000 7f705a800700 1 -- 172.21.15.12:0/98567556 wait complete. 2024-03-29T09:54:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.226+0000 7f81f8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81ec03a510 0x7f81ec182bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.226+0000 7f81f8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec04f7e0 0x7f81ec04e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.226+0000 7f81f8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81ec0537b0 0x7f81ec036c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.227+0000 7f81f1400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec04f7e0 0x7f81ec04e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.227+0000 7f81f1400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec04f7e0 0x7f81ec04e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46940/0 (socket says 172.21.15.12:46940) 2024-03-29T09:54:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.227+0000 7f81f2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81ec0537b0 0x7f81ec036c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.227+0000 7f81f2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81ec0537b0 0x7f81ec036c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40750/0 (socket says 172.21.15.12:40750) 2024-03-29T09:54:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.227+0000 7f81f1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81ec03a510 0x7f81ec182bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.228+0000 7f81f1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81ec03a510 0x7f81ec182bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51536/0 (socket says 172.21.15.12:51536) 2024-03-29T09:54:35.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.228+0000 7f81f1e00700 1 -- 172.21.15.12:0/2881525120 learned_addr learned my addr 172.21.15.12:0/2881525120 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.228+0000 7f705a800700 1 Processor -- start 2024-03-29T09:54:35.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.228+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec037170 con 0x7f81ec0537b0 2024-03-29T09:54:35.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.228+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec04f070 con 0x7f81ec04f7e0 2024-03-29T09:54:35.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.226+0000 7ff008800700 1 -- 172.21.15.12:0/1506264234 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00006b7d0 con 0x7ff00015d810 2024-03-29T09:54:35.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.228+0000 7feffa000700 1 -- 172.21.15.12:0/1506264234 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7feff4007450 con 0x7ff00015d810 2024-03-29T09:54:35.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.228+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec04eab0 con 0x7f81ec03a510 2024-03-29T09:54:35.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.229+0000 7f81f1e00700 1 -- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec04f7e0 msgr2=0x7f81ec04e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.229+0000 7f81f1e00700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec04f7e0 0x7f81ec04e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.229+0000 7f81f1e00700 1 -- 172.21.15.12:0/2881525120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81ec0537b0 msgr2=0x7f81ec036c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.229+0000 7f81f1e00700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81ec0537b0 0x7f81ec036c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.229+0000 7f81f1e00700 1 -- 172.21.15.12:0/2881525120 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f40686c0 con 0x7f81ec03a510 2024-03-29T09:54:35.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.229+0000 7f81f2800700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81ec0537b0 0x7f81ec036c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:35.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.229+0000 7f81f1400700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec04f7e0 0x7f81ec04e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:35.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.230+0000 7f81f1e00700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81ec03a510 0x7f81ec182bd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f81e400c4e0 tx=0x7f81e400c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.230+0000 7f705a800700 1 -- start start 2024-03-29T09:54:35.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.230+0000 7feffa000700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefe406ac90 0x7fefe406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.230+0000 7feffa000700 1 -- 172.21.15.12:0/1506264234 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feff4088690 con 0x7ff00015d810 2024-03-29T09:54:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.231+0000 7f705a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70540724c0 0x7f70541eeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.231+0000 7f705a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70540ac3a0 0x7f7054156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.231+0000 7f705a800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f705415d7a0 0x7f70541517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.231+0000 7f705a800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70541efc20 con 0x7f70540ac3a0 2024-03-29T09:54:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.231+0000 7f705a800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70541f0000 con 0x7f705415d7a0 2024-03-29T09:54:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.231+0000 7f705a800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70541ef380 con 0x7f70540724c0 2024-03-29T09:54:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f705415d7a0 0x7f70541517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f705415d7a0 0x7f70541517e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46950/0 (socket says 172.21.15.12:46950) 2024-03-29T09:54:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053e00700 1 -- 172.21.15.12:0/2381364132 learned_addr learned my addr 172.21.15.12:0/2381364132 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053400700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70540724c0 0x7f70541eeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.231+0000 7ff005e00700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefe406ac90 0x7fefe406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053e00700 1 -- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70540724c0 msgr2=0x7f70541eeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053e00700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70540724c0 0x7f70541eeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053e00700 1 -- 172.21.15.12:0/2381364132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70540ac3a0 msgr2=0x7f7054156830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T09:54:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053e00700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70540ac3a0 0x7f7054156830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053e00700 1 -- 172.21.15.12:0/2381364132 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c0094a0 con 0x7f705415d7a0 2024-03-29T09:54:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053400700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70540724c0 0x7f70541eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.232+0000 7f7053e00700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f705415d7a0 0x7f70541517e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f705406e3b0 tx=0x7f703c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.233+0000 7ff005e00700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefe406ac90 0x7fefe406d150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feff0000c00 tx=0x7feff0008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.233+0000 7f704b400700 1 -- 172.21.15.12:0/2381364132 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c004230 con 0x7f705415d7a0 2024-03-29T09:54:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.233+0000 7f704b400700 1 -- 172.21.15.12:0/2381364132 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c0043b0 con 0x7f705415d7a0 2024-03-29T09:54:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.233+0000 7f704b400700 1 -- 172.21.15.12:0/2381364132 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c01a790 con 0x7f705415d7a0 2024-03-29T09:54:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.233+0000 7f705a800700 1 -- 172.21.15.12:0/2381364132 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70541ef660 con 0x7f705415d7a0 2024-03-29T09:54:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.233+0000 7f705a800700 1 -- 172.21.15.12:0/2381364132 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7054156d70 con 0x7f705415d7a0 2024-03-29T09:54:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.233+0000 7f81ea000700 1 -- 172.21.15.12:0/2881525120 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e40052d0 con 0x7f81ec03a510 2024-03-29T09:54:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.233+0000 7f81ea000700 1 -- 172.21.15.12:0/2881525120 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e4005430 con 0x7f81ec03a510 2024-03-29T09:54:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.233+0000 7f81ea000700 1 -- 172.21.15.12:0/2881525120 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e40038c0 con 0x7f81ec03a510 2024-03-29T09:54:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.233+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ec0e35a0 con 0x7f81ec03a510 2024-03-29T09:54:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.234+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ec04c720 con 0x7f81ec03a510 2024-03-29T09:54:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.234+0000 7feffa000700 1 -- 172.21.15.12:0/1506264234 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feff4053870 con 0x7ff00015d810 2024-03-29T09:54:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.235+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ec04f1e0 con 0x7f81ec03a510 2024-03-29T09:54:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.236+0000 7f705a800700 1 -- 172.21.15.12:0/2381364132 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705406b7d0 con 0x7f705415d7a0 2024-03-29T09:54:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.236+0000 7f81ea000700 1 -- 172.21.15.12:0/2881525120 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f81e4003a20 con 0x7f81ec03a510 2024-03-29T09:54:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.238+0000 7f81ea000700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81d406ad60 0x7f81d406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.238+0000 7f81ea000700 1 -- 172.21.15.12:0/2881525120 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81e40887e0 con 0x7f81ec03a510 2024-03-29T09:54:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.238+0000 7f704b400700 1 -- 172.21.15.12:0/2381364132 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f703c022070 con 0x7f705415d7a0 2024-03-29T09:54:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.239+0000 7f704b400700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f703006acc0 0x7f703006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.239+0000 7f704b400700 1 -- 172.21.15.12:0/2381364132 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f703c08a820 con 0x7f705415d7a0 2024-03-29T09:54:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.240+0000 7f81f1400700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81d406ad60 0x7f81d406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.240+0000 7f7053400700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f703006acc0 0x7f703006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.241+0000 7f7053400700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f703006acc0 0x7f703006d180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f704c000f60 tx=0x7f704c001500 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.241+0000 7f81f1400700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81d406ad60 0x7f81d406d220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f81ec04b0f0 tx=0x7f81e0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.242+0000 7f704b400700 1 -- 172.21.15.12:0/2381364132 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f703c055b90 con 0x7f705415d7a0 2024-03-29T09:54:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.242+0000 7f81ea000700 1 -- 172.21.15.12:0/2881525120 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81e4053a30 con 0x7f81ec03a510 2024-03-29T09:54:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.636+0000 7fdc5be00700 1 -- 172.21.15.12:0/2236488987 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fdc44005870 con 0x7fdc680e32f0 2024-03-29T09:54:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.640+0000 7fdc66a00700 1 -- 172.21.15.12:0/2236488987 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdc5c017020 con 0x7fdc680e32f0 2024-03-29T09:54:35.640 INFO:teuthology.orchestra.run.smithi012.stdout:55834574853 2024-03-29T09:54:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.645+0000 7fdc5be00700 1 -- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5006ad60 msgr2=0x7fdc5006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.645+0000 7fdc5be00700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5006ad60 0x7fdc5006d220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdc68007a80 tx=0x7fdc7006d000 comp rx=0 tx=0).stop 2024-03-29T09:54:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.645+0000 7fdc5be00700 1 -- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc680e32f0 msgr2=0x7fdc68035430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.645+0000 7fdc5be00700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc680e32f0 0x7fdc68035430 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00c410 tx=0x7fdc5c00c720 comp rx=0 tx=0).stop 2024-03-29T09:54:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.645+0000 7fdc5be00700 1 -- 172.21.15.12:0/2236488987 shutdown_connections 2024-03-29T09:54:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.645+0000 7fdc5be00700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5006ad60 0x7fdc5006d220 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.645+0000 7fdc5be00700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc680e32f0 0x7fdc68035430 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.645+0000 7fdc5be00700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc6804ffa0 0x7fdc680d6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.645+0000 7fdc5be00700 1 --2- 172.21.15.12:0/2236488987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc680531b0 0x7fdc68034ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.645+0000 7fdc5be00700 1 -- 172.21.15.12:0/2236488987 >> 172.21.15.12:0/2236488987 conn(0x7fdc68009fd0 msgr2=0x7fdc68038050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.646+0000 7fdc5be00700 1 -- 172.21.15.12:0/2236488987 shutdown_connections 2024-03-29T09:54:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.646+0000 7fdc5be00700 1 -- 172.21.15.12:0/2236488987 wait complete. 2024-03-29T09:54:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.648+0000 7f4363200700 1 -- 172.21.15.12:0/3707603928 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f435c054150 con 0x7f435c0063d0 2024-03-29T09:54:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.649+0000 7f435a000700 1 -- 172.21.15.12:0/3707603928 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f435c054150 con 0x7f435c0063d0 2024-03-29T09:54:35.649 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2024-03-29T09:54:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.653+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe4080527f0 con 0x7fe408033f60 2024-03-29T09:54:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.653+0000 7f4363200700 1 -- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f434406ace0 msgr2=0x7f434406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.653+0000 7f4363200700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f434406ace0 0x7f434406d1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f435c0086e0 tx=0x7f436406e000 comp rx=0 tx=0).stop 2024-03-29T09:54:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.653+0000 7f4363200700 1 -- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c0063d0 msgr2=0x7f435c04e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.653+0000 7f4363200700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c0063d0 0x7f435c04e450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f435400c730 tx=0x7f435400caf0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.654+0000 7f4363200700 1 -- 172.21.15.12:0/3707603928 shutdown_connections 2024-03-29T09:54:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.654+0000 7f4363200700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f434406ace0 0x7f434406d1a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.654+0000 7f4363200700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f435c0e32c0 0x7f435c0dc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.654+0000 7f4363200700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f435c0063d0 0x7f435c04e450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.654+0000 7f4363200700 1 --2- 172.21.15.12:0/3707603928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f435c053790 0x7f435c0d5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.654+0000 7f4363200700 1 -- 172.21.15.12:0/3707603928 >> 172.21.15.12:0/3707603928 conn(0x7f435c0559c0 msgr2=0x7f435c0511e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.654+0000 7fe40cc00700 1 -- 172.21.15.12:0/1465324041 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe40400cea0 con 0x7fe408033f60 2024-03-29T09:54:35.654 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.655+0000 7f4363200700 1 -- 172.21.15.12:0/3707603928 shutdown_connections 2024-03-29T09:54:35.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.655+0000 7f4363200700 1 -- 172.21.15.12:0/3707603928 wait complete. 2024-03-29T09:54:35.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.658+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3f006ad60 msgr2=0x7fe3f006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.658+0000 7fe416200700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3f006ad60 0x7fe3f006d220 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe408048620 tx=0x7fe3fc008030 comp rx=0 tx=0).stop 2024-03-29T09:54:35.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.658+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe408033f60 msgr2=0x7fe4080ea8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.658+0000 7fe416200700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe408033f60 0x7fe4080ea8c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe40400ebe0 tx=0x7fe40400eef0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.658+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 shutdown_connections 2024-03-29T09:54:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.658+0000 7fe416200700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3f006ad60 0x7fe3f006d220 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.658+0000 7fe416200700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4080481c0 0x7fe408185330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.658+0000 7fe416200700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe408033f60 0x7fe4080ea8c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.658+0000 7fe416200700 1 --2- 172.21.15.12:0/1465324041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe408047c70 0x7fe4080352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.658+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 >> 172.21.15.12:0/1465324041 conn(0x7fe408054b20 msgr2=0x7fe408036e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.659+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 shutdown_connections 2024-03-29T09:54:35.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.659+0000 7fe416200700 1 -- 172.21.15.12:0/1465324041 wait complete. 2024-03-29T09:54:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.663+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f28d01e7ee0 con 0x7f28b40037f0 2024-03-29T09:54:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.663+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1723956403 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3cac049820 con 0x7f3cac047c70 2024-03-29T09:54:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.663+0000 7f3ca7400700 1 -- 172.21.15.12:0/1723956403 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3cac049820 con 0x7f3cac047c70 2024-03-29T09:54:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.664+0000 7f28c7e00700 1 -- 172.21.15.12:0/411260980 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f28d01e7ee0 con 0x7f28b40037f0 2024-03-29T09:54:35.663 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:35.667 INFO:teuthology.orchestra.run.smithi012.stdout:55834574853 2024-03-29T09:54:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.668+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c9406ad60 msgr2=0x7f3c9406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.668+0000 7f3cb8a00700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c9406ad60 0x7f3c9406d220 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009910 tx=0x7f3ca8008040 comp rx=0 tx=0).stop 2024-03-29T09:54:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.668+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac047c70 msgr2=0x7f3cac04ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.668+0000 7f3cb8a00700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac047c70 0x7f3cac04ad40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009840 tx=0x7f3ca0004750 comp rx=0 tx=0).stop 2024-03-29T09:54:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.669+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1723956403 shutdown_connections 2024-03-29T09:54:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.669+0000 7f3cb8a00700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c9406ad60 0x7f3c9406d220 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.669+0000 7f3cb8a00700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cac0481c0 0x7f3cac038cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.669+0000 7f3cb8a00700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cac033f60 0x7f3cac0d87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.669+0000 7f3cb8a00700 1 --2- 172.21.15.12:0/1723956403 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cac047c70 0x7f3cac04ad40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.669+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1723956403 >> 172.21.15.12:0/1723956403 conn(0x7f3cac054b20 msgr2=0x7f3cac034560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.669+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1723956403 shutdown_connections 2024-03-29T09:54:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.669+0000 7f3cb8a00700 1 -- 172.21.15.12:0/1723956403 wait complete. 2024-03-29T09:54:35.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.670+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f28b006ad60 msgr2=0x7f28b006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.670+0000 7f28d7200700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f28b006ad60 0x7f28b006d220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28d0162400 tx=0x7f28bc00a000 comp rx=0 tx=0).stop 2024-03-29T09:54:35.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.670+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b40037f0 msgr2=0x7f28d01fd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.670+0000 7f28d7200700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b40037f0 0x7f28d01fd100 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28c000e7b0 tx=0x7f28c000eac0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.671+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 shutdown_connections 2024-03-29T09:54:35.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.671+0000 7f28d7200700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f28b006ad60 0x7f28b006d220 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.671+0000 7f28d7200700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d01ffe90 0x7f28d0200310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.671+0000 7f28d7200700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28b40037f0 0x7f28d01fd100 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.671+0000 7f28d7200700 1 --2- 172.21.15.12:0/411260980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28d00cb570 0x7f28d00cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.671+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 >> 172.21.15.12:0/411260980 conn(0x7f28d008aff0 msgr2=0x7f28d015ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.672+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 shutdown_connections 2024-03-29T09:54:35.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.672+0000 7f28d7200700 1 -- 172.21.15.12:0/411260980 wait complete. 2024-03-29T09:54:35.681 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-03-29T09:54:35.688 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2024-03-29T09:54:35.697 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2024-03-29T09:54:35.697 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2024-03-29T09:54:35.700 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-03-29T09:54:35.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.708+0000 7f705a800700 1 -- 172.21.15.12:0/2381364132 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f70541efdb0 con 0x7f705415d7a0 2024-03-29T09:54:35.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.709+0000 7f704b400700 1 -- 172.21.15.12:0/2381364132 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f703c0591b0 con 0x7f705415d7a0 2024-03-29T09:54:35.708 INFO:teuthology.orchestra.run.smithi012.stdout:55834574853 2024-03-29T09:54:35.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 -- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f703006acc0 msgr2=0x7f703006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f703006acc0 0x7f703006d180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f704c000f60 tx=0x7f704c001500 comp rx=0 tx=0).stop 2024-03-29T09:54:35.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 -- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f705415d7a0 msgr2=0x7f70541517e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f705415d7a0 0x7f70541517e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f705406e3b0 tx=0x7f703c004750 comp rx=0 tx=0).stop 2024-03-29T09:54:35.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 -- 172.21.15.12:0/2381364132 shutdown_connections 2024-03-29T09:54:35.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f703006acc0 0x7f703006d180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f705415d7a0 0x7f70541517e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70540724c0 0x7f70541eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 --2- 172.21.15.12:0/2381364132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70540ac3a0 0x7f7054156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 -- 172.21.15.12:0/2381364132 >> 172.21.15.12:0/2381364132 conn(0x7f7054070f00 msgr2=0x7f70540b1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 -- 172.21.15.12:0/2381364132 shutdown_connections 2024-03-29T09:54:35.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.712+0000 7f705a800700 1 -- 172.21.15.12:0/2381364132 wait complete. 2024-03-29T09:54:35.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.719+0000 7ff008800700 1 -- 172.21.15.12:0/1506264234 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff000067720 con 0x7ff00015d810 2024-03-29T09:54:35.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.720+0000 7feffa000700 1 -- 172.21.15.12:0/1506264234 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feff4007700 con 0x7ff00015d810 2024-03-29T09:54:35.719 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:35.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.723+0000 7ff008800700 1 -- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefe406ac90 msgr2=0x7fefe406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.723+0000 7ff008800700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefe406ac90 0x7fefe406d150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feff0000c00 tx=0x7feff0008030 comp rx=0 tx=0).stop 2024-03-29T09:54:35.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.723+0000 7ff008800700 1 -- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff00015d810 msgr2=0x7ff0001e41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.723+0000 7ff008800700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff00015d810 0x7ff0001e41d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feff400c3e0 tx=0x7feff400c6f0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.723+0000 7ff008800700 1 -- 172.21.15.12:0/1506264234 shutdown_connections 2024-03-29T09:54:35.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.723+0000 7ff008800700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefe406ac90 0x7fefe406d150 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.723+0000 7ff008800700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff00015d810 0x7ff0001e41d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.723+0000 7ff008800700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff00006e810 0x7ff0001ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.723+0000 7ff008800700 1 --2- 172.21.15.12:0/1506264234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff000072910 0x7ff0001f1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.723+0000 7ff008800700 1 -- 172.21.15.12:0/1506264234 >> 172.21.15.12:0/1506264234 conn(0x7ff000070f00 msgr2=0x7ff0000adb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.724+0000 7ff008800700 1 -- 172.21.15.12:0/1506264234 shutdown_connections 2024-03-29T09:54:35.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.724+0000 7ff008800700 1 -- 172.21.15.12:0/1506264234 wait complete. 2024-03-29T09:54:35.730 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-03-29T09:54:35.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.732+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f81ec04ebf0 con 0x7f81ec03a510 2024-03-29T09:54:35.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.733+0000 7f81ea000700 1 -- 172.21.15.12:0/2881525120 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f81ec04ebf0 con 0x7f81ec03a510 2024-03-29T09:54:35.732 INFO:teuthology.orchestra.run.smithi012.stdout:55834574853 2024-03-29T09:54:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.735+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81d406ad60 msgr2=0x7f81d406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.735+0000 7f81f8a00700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81d406ad60 0x7f81d406d220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f81ec04b0f0 tx=0x7f81e0008040 comp rx=0 tx=0).stop 2024-03-29T09:54:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.735+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81ec03a510 msgr2=0x7f81ec182bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.735+0000 7f81f8a00700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81ec03a510 0x7f81ec182bd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f81e400c4e0 tx=0x7f81e400c8a0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.735+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 shutdown_connections 2024-03-29T09:54:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.735+0000 7f81f8a00700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81d406ad60 0x7f81d406d220 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.735+0000 7f81f8a00700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81ec0537b0 0x7f81ec036c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.735+0000 7f81f8a00700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81ec03a510 0x7f81ec182bd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.735+0000 7f81f8a00700 1 --2- 172.21.15.12:0/2881525120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81ec04f7e0 0x7f81ec04e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:35.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.735+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 >> 172.21.15.12:0/2881525120 conn(0x7f81ec054b10 msgr2=0x7f81ec054ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:35.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.736+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 shutdown_connections 2024-03-29T09:54:35.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:35.736+0000 7f81f8a00700 1 -- 172.21.15.12:0/2881525120 wait complete. 2024-03-29T09:54:35.742 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-03-29T09:54:35.755 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-03-29T09:54:36.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-29T09:54:36.690 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T09:54:36.697 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T09:54:36.698 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-29T09:54:36.701 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-29T09:54:36.731 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-29T09:54:36.742 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-29T09:54:36.756 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-29T09:54:36.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.000+0000 7f560cc00700 1 Processor -- start 2024-03-29T09:54:37.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.001+0000 7f560cc00700 1 -- start start 2024-03-29T09:54:37.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.005+0000 7f560cc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f560004f790 0x7f56000df400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.005+0000 7f560cc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56000543d0 con 0x7f560004fb70 2024-03-29T09:54:37.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.006+0000 7f560cc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56000537c0 con 0x7f56000532e0 2024-03-29T09:54:37.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.006+0000 7f560cc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5600053aa0 con 0x7f560004f790 2024-03-29T09:54:37.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.007+0000 7f5606800700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f56000532e0 0x7f56000061c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46496/0 (socket says 172.21.15.12:46496) 2024-03-29T09:54:37.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.007+0000 7f5606800700 1 -- 172.21.15.12:0/3422488326 learned_addr learned my addr 172.21.15.12:0/3422488326 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:37.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.007+0000 7f5605e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f560004fb70 0x7f560004ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58210/0 (socket says 172.21.15.12:58210) 2024-03-29T09:54:37.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.007+0000 7f5605400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f560004f790 0x7f56000df400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.009+0000 7f5605400700 1 -- 172.21.15.12:0/3422488326 >> v1:172.21.15.159:6789/0 conn(0x7f56000532e0 legacy=0x7f56000061c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.009+0000 7f5605400700 1 -- 172.21.15.12:0/3422488326 >> v1:172.21.15.12:6789/0 conn(0x7f560004fb70 legacy=0x7f560004ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.009+0000 7f5605400700 1 -- 172.21.15.12:0/3422488326 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5600053d80 con 0x7f560004f790 2024-03-29T09:54:37.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.010+0000 7f5605400700 1 --2- 172.21.15.12:0/3422488326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560004f790 0x7f56000df400 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f55f8007360 tx=0x7f55f8007670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fc02fdde61ef8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.010+0000 7f5604a00700 1 -- 172.21.15.12:0/3422488326 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f800a580 con 0x7f560004f790 2024-03-29T09:54:37.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.010+0000 7f5604a00700 1 -- 172.21.15.12:0/3422488326 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f800a6e0 con 0x7f560004f790 2024-03-29T09:54:37.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.010+0000 7f5604a00700 1 -- 172.21.15.12:0/3422488326 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8004230 con 0x7f560004f790 2024-03-29T09:54:37.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.012+0000 7f560cc00700 1 -- 172.21.15.12:0/3422488326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560004f790 msgr2=0x7f56000df400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.012+0000 7f560cc00700 1 --2- 172.21.15.12:0/3422488326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560004f790 0x7f56000df400 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f55f8007360 tx=0x7f55f8007670 comp rx=0 tx=0).stop 2024-03-29T09:54:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.012+0000 7f560cc00700 1 -- 172.21.15.12:0/3422488326 shutdown_connections 2024-03-29T09:54:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.012+0000 7f560cc00700 1 --2- 172.21.15.12:0/3422488326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560004f790 0x7f56000df400 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.013+0000 7f560cc00700 1 -- 172.21.15.12:0/3422488326 >> 172.21.15.12:0/3422488326 conn(0x7f5600054b10 msgr2=0x7f5600054f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.016+0000 7f560cc00700 1 -- 172.21.15.12:0/3422488326 shutdown_connections 2024-03-29T09:54:37.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.016+0000 7f560cc00700 1 -- 172.21.15.12:0/3422488326 wait complete. 2024-03-29T09:54:37.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.020+0000 7f560cc00700 1 Processor -- start 2024-03-29T09:54:37.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.020+0000 7f560cc00700 1 -- start start 2024-03-29T09:54:37.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.024+0000 7f560cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f560004f790 0x7f56001854f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.024+0000 7f560cc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560004fb70 0x7f56000dbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.024+0000 7f5605e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f560004f790 0x7f56001854f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.025+0000 7f560cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56000532e0 0x7f56000dc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.025+0000 7f560cc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5600185e10 con 0x7f56000532e0 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.025+0000 7f560cc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56001860f0 con 0x7f560004fb70 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.025+0000 7f560cc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56001863d0 con 0x7f560004f790 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.025+0000 7f5605e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f560004f790 0x7f56001854f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51546/0 (socket says 172.21.15.12:51546) 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.025+0000 7f5605e00700 1 -- 172.21.15.12:0/4266475902 learned_addr learned my addr 172.21.15.12:0/4266475902 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.025+0000 7f5605400700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560004fb70 0x7f56000dbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.025+0000 7f5606800700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56000532e0 0x7f56000dc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.026+0000 7f5606800700 1 -- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f560004f790 msgr2=0x7f56001854f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.026+0000 7f5606800700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f560004f790 0x7f56001854f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.026+0000 7f5606800700 1 -- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560004fb70 msgr2=0x7f56000dbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.026+0000 7f5606800700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560004fb70 0x7f56000dbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.026+0000 7f5606800700 1 -- 172.21.15.12:0/4266475902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f8007040 con 0x7f56000532e0 2024-03-29T09:54:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.027+0000 7f5606800700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56000532e0 0x7f56000dc210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5608000eb0 tx=0x7f5608068110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.027+0000 7f55fe000700 1 -- 172.21.15.12:0/4266475902 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560808e070 con 0x7f56000532e0 2024-03-29T09:54:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.027+0000 7f560cc00700 1 -- 172.21.15.12:0/4266475902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56001866b0 con 0x7f56000532e0 2024-03-29T09:54:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.028+0000 7f560cc00700 1 -- 172.21.15.12:0/4266475902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56000de9e0 con 0x7f56000532e0 2024-03-29T09:54:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.030+0000 7f560cc00700 1 -- 172.21.15.12:0/4266475902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56000053e0 con 0x7f56000532e0 2024-03-29T09:54:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.032+0000 7f55fe000700 1 -- 172.21.15.12:0/4266475902 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560806bc80 con 0x7f56000532e0 2024-03-29T09:54:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.032+0000 7f55fe000700 1 -- 172.21.15.12:0/4266475902 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5608093420 con 0x7f56000532e0 2024-03-29T09:54:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.034+0000 7f55fe000700 1 -- 172.21.15.12:0/4266475902 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5608093580 con 0x7f56000532e0 2024-03-29T09:54:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.035+0000 7f55fe000700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f55e806acb0 0x7f55e806d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.039+0000 7f55fe000700 1 -- 172.21.15.12:0/4266475902 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56080d21d0 con 0x7f56000532e0 2024-03-29T09:54:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.039+0000 7f5605e00700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f55e806acb0 0x7f55e806d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.043+0000 7f5605e00700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f55e806acb0 0x7f55e806d170 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f55f4001380 tx=0x7f55f4001310 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.047+0000 7f55fe000700 1 -- 172.21.15.12:0/4266475902 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f56080ce5a0 con 0x7f56000532e0 2024-03-29T09:54:37.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.080+0000 7f3c42800700 1 Processor -- start 2024-03-29T09:54:37.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.080+0000 7f3c42800700 1 -- start start 2024-03-29T09:54:37.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.082+0000 7f3c42800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c15d810 0x7f3c3c0b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.082+0000 7f3c42800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3c3c072910 0x7f3c3c0cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.082+0000 7f3c42800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3c3c0724c0 0x7f3c3c06e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.084+0000 7f3c42800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c3c0cc140 con 0x7f3c3c072910 2024-03-29T09:54:37.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.084+0000 7f3c42800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c3c15df80 con 0x7f3c3c0724c0 2024-03-29T09:54:37.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.084+0000 7f3c42800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c15e260 con 0x7f3c3c15d810 2024-03-29T09:54:37.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.086+0000 7f3c3aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3c3c072910 0x7f3c3c0cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.086+0000 7f3c3aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3c3c072910 0x7f3c3c0cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40794/0 (socket says 172.21.15.12:40794) 2024-03-29T09:54:37.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.086+0000 7f3c3aa00700 1 -- 172.21.15.12:0/959515236 learned_addr learned my addr 172.21.15.12:0/959515236 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.086+0000 7f3c3be00700 1 --2- 172.21.15.12:0/959515236 >> v2:172.21.15.159:3300/0 conn(0x7f3c3c0724c0 0x7f3c3c06e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.086+0000 7f3c3b400700 1 --2- 172.21.15.12:0/959515236 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c15d810 0x7f3c3c0b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.082+0000 7f3be6600700 1 Processor -- start 2024-03-29T09:54:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.082+0000 7f3be6600700 1 -- start start 2024-03-29T09:54:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.084+0000 7f3be6600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be015f730 0x7f3be00721c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.084+0000 7f3be6600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3be00d0d00 0x7f3be0161ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.084+0000 7f3be6600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be0162e80 con 0x7f3be0162410 2024-03-29T09:54:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.084+0000 7f3be6600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be0071d30 con 0x7f3be00d0d00 2024-03-29T09:54:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.084+0000 7f3be6600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be01628c0 con 0x7f3be015f730 2024-03-29T09:54:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.085+0000 7f3bdea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3be00d0d00 0x7f3be0161ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.085+0000 7f3bdea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3be00d0d00 0x7f3be0161ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40780/0 (socket says 172.21.15.12:40780) 2024-03-29T09:54:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.085+0000 7f3bdea00700 1 -- 172.21.15.12:0/3018245233 learned_addr learned my addr 172.21.15.12:0/3018245233 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.087+0000 7f3bdf400700 1 --2- 172.21.15.12:0/3018245233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be015f730 0x7f3be00721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.087+0000 7f3bdea00700 1 -- 172.21.15.12:0/3018245233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be015f730 msgr2=0x7f3be00721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.087+0000 7f3bdea00700 1 --2- 172.21.15.12:0/3018245233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be015f730 0x7f3be00721c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.087+0000 7f3bdea00700 1 -- 172.21.15.12:0/3018245233 >> v1:172.21.15.12:6789/0 conn(0x7f3be0162410 legacy=0x7f3be006a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.088+0000 7f3dce200700 1 Processor -- start 2024-03-29T09:54:37.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.088+0000 7f3c3aa00700 1 -- 172.21.15.12:0/959515236 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c15d810 msgr2=0x7f3c3c0b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.088+0000 7f3dce200700 1 -- start start 2024-03-29T09:54:37.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.088+0000 7f3bdea00700 1 -- 172.21.15.12:0/3018245233 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be0162ba0 con 0x7f3be00d0d00 2024-03-29T09:54:37.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.089+0000 7f3c3aa00700 1 --2- 172.21.15.12:0/959515236 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c15d810 0x7f3c3c0b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.089+0000 7f3c3aa00700 1 -- 172.21.15.12:0/959515236 >> v2:172.21.15.159:3300/0 conn(0x7f3c3c0724c0 msgr2=0x7f3c3c06e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.089+0000 7f3c3aa00700 1 --2- 172.21.15.12:0/959515236 >> v2:172.21.15.159:3300/0 conn(0x7f3c3c0724c0 0x7f3c3c06e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.089+0000 7f3c3aa00700 1 -- 172.21.15.12:0/959515236 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c0cbdd0 con 0x7f3c3c072910 2024-03-29T09:54:37.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.089+0000 7f3c3b400700 1 --2- 172.21.15.12:0/959515236 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c15d810 0x7f3c3c0b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:37.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.089+0000 7f3bdea00700 1 --2- 172.21.15.12:0/3018245233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3be00d0d00 0x7f3be0161ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009920 tx=0x7f3bd0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcff5046c7e45894 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.090+0000 7f3bde000700 1 -- 172.21.15.12:0/3018245233 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0004230 con 0x7f3be00d0d00 2024-03-29T09:54:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.090+0000 7f3bde000700 1 -- 172.21.15.12:0/3018245233 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd0004390 con 0x7f3be00d0d00 2024-03-29T09:54:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.090+0000 7f3bde000700 1 -- 172.21.15.12:0/3018245233 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd00036a0 con 0x7f3be00d0d00 2024-03-29T09:54:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.090+0000 7f3c3aa00700 1 --2- 172.21.15.12:0/959515236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c3c072910 0x7f3c3c0cc5d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c06de10 tx=0x7f3c24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2231665329a4b35d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.091+0000 7f3be6600700 1 -- 172.21.15.12:0/3018245233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3be00d0d00 msgr2=0x7f3be0161ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.091+0000 7f3be6600700 1 --2- 172.21.15.12:0/3018245233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3be00d0d00 0x7f3be0161ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009920 tx=0x7f3bd0009c30 comp rx=0 tx=0).stop 2024-03-29T09:54:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.091+0000 7f3be6600700 1 -- 172.21.15.12:0/3018245233 shutdown_connections 2024-03-29T09:54:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.091+0000 7f3be6600700 1 --2- 172.21.15.12:0/3018245233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be015f730 0x7f3be00721c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.091+0000 7f3be6600700 1 --2- 172.21.15.12:0/3018245233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3be00d0d00 0x7f3be0161ed0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.091+0000 7f3be6600700 1 -- 172.21.15.12:0/3018245233 >> 172.21.15.12:0/3018245233 conn(0x7f3be00d0090 msgr2=0x7f3be00d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.091+0000 7f3c3a000700 1 -- 172.21.15.12:0/959515236 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24011830 con 0x7f3c3c072910 2024-03-29T09:54:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.092+0000 7f3c3a000700 1 -- 172.21.15.12:0/959515236 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c24011990 con 0x7f3c3c072910 2024-03-29T09:54:37.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.090+0000 7f3dce200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc80cfbc0 0x7f3dc80cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.090+0000 7f3dce200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3dc80ce2d0 0x7f3dc80ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.090+0000 7f3dce200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc806e620 con 0x7f3dc806ca60 2024-03-29T09:54:37.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.090+0000 7f3dce200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc80ceca0 con 0x7f3dc80ce2d0 2024-03-29T09:54:37.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.090+0000 7f3dce200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc80b2a80 con 0x7f3dc80cfbc0 2024-03-29T09:54:37.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.092+0000 7f3c3a000700 1 -- 172.21.15.12:0/959515236 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24011d20 con 0x7f3c3c072910 2024-03-29T09:54:37.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.092+0000 7f3be6600700 1 -- 172.21.15.12:0/3018245233 shutdown_connections 2024-03-29T09:54:37.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.093+0000 7f3dc7400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f3dc806ca60 0x7f3dc815d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46500/0 (socket says 172.21.15.12:46500) 2024-03-29T09:54:37.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.093+0000 7f3dc7400700 1 -- 172.21.15.12:0/893436885 learned_addr learned my addr 172.21.15.12:0/893436885 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:37.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.093+0000 7f3dc6a00700 1 --2- 172.21.15.12:0/893436885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc80cfbc0 0x7f3dc80cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.093+0000 7f3c42800700 1 -- 172.21.15.12:0/959515236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c3c072910 msgr2=0x7f3c3c0cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.093+0000 7f3c42800700 1 --2- 172.21.15.12:0/959515236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c3c072910 0x7f3c3c0cc5d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c06de10 tx=0x7f3c24009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.092+0000 7f3be6600700 1 -- 172.21.15.12:0/3018245233 wait complete. 2024-03-29T09:54:37.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3be6600700 1 Processor -- start 2024-03-29T09:54:37.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.093+0000 7f3dc7e00700 1 --2- 172.21.15.12:0/893436885 >> v2:172.21.15.12:3300/0 conn(0x7f3dc80ce2d0 0x7f3dc80ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3c42800700 1 -- 172.21.15.12:0/959515236 shutdown_connections 2024-03-29T09:54:37.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3c42800700 1 --2- 172.21.15.12:0/959515236 >> v2:172.21.15.159:3300/0 conn(0x7f3c3c0724c0 0x7f3c3c06e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3c42800700 1 --2- 172.21.15.12:0/959515236 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c15d810 0x7f3c3c0b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3c42800700 1 --2- 172.21.15.12:0/959515236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c3c072910 0x7f3c3c0cc5d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3c42800700 1 -- 172.21.15.12:0/959515236 >> 172.21.15.12:0/959515236 conn(0x7f3c3c070f00 msgr2=0x7f3c3c071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3c42800700 1 -- 172.21.15.12:0/959515236 shutdown_connections 2024-03-29T09:54:37.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.094+0000 7f3dc6000700 1 -- 172.21.15.12:0/893436885 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082426108 0 0) 0x7f3dc806e620 con 0x7f3dc806ca60 2024-03-29T09:54:37.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3c42800700 1 -- 172.21.15.12:0/959515236 wait complete. 2024-03-29T09:54:37.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.096+0000 7f3be6600700 1 -- start start 2024-03-29T09:54:37.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.098+0000 7f3be6600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3be00d0d00 0x7f3be00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.098+0000 7f3be6600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be0162410 0x7f3be013f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.098+0000 7f3be6600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3be00ccd30 0x7f3be00cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3dc6000700 1 -- 172.21.15.12:0/893436885 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dac0035b0 con 0x7f3dc806ca60 2024-03-29T09:54:37.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3be6600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be0139370 con 0x7f3be00d0d00 2024-03-29T09:54:37.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3bdea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be0162410 0x7f3be013f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3bdf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3be00d0d00 0x7f3be00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3bdea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be0162410 0x7f3be013f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51576/0 (socket says 172.21.15.12:51576) 2024-03-29T09:54:37.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3bdea00700 1 -- 172.21.15.12:0/2648657997 learned_addr learned my addr 172.21.15.12:0/2648657997 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3dc7e00700 1 -- 172.21.15.12:0/893436885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc80cfbc0 msgr2=0x7f3dc80cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3dc7e00700 1 --2- 172.21.15.12:0/893436885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc80cfbc0 0x7f3dc80cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.095+0000 7f3dc7e00700 1 -- 172.21.15.12:0/893436885 >> v1:172.21.15.159:6789/0 conn(0x7f3dc806ca60 legacy=0x7f3dc815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.096+0000 7f3dc7e00700 1 -- 172.21.15.12:0/893436885 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc806e900 con 0x7f3dc80ce2d0 2024-03-29T09:54:37.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.096+0000 7f3dc7e00700 1 --2- 172.21.15.12:0/893436885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dc80ce2d0 0x7f3dc80ce760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0009d90 tx=0x7f3dc000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce4377a025dd354 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.097+0000 7f3dc6000700 1 -- 172.21.15.12:0/893436885 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc000e070 con 0x7f3dc80ce2d0 2024-03-29T09:54:37.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.097+0000 7f3dc6000700 1 -- 172.21.15.12:0/893436885 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc000bb20 con 0x7f3dc80ce2d0 2024-03-29T09:54:37.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.097+0000 7f3dc6000700 1 -- 172.21.15.12:0/893436885 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc000e070 con 0x7f3dc80ce2d0 2024-03-29T09:54:37.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3dce200700 1 -- 172.21.15.12:0/893436885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dc80ce2d0 msgr2=0x7f3dc80ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3dce200700 1 --2- 172.21.15.12:0/893436885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dc80ce2d0 0x7f3dc80ce760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0009d90 tx=0x7f3dc000b330 comp rx=0 tx=0).stop 2024-03-29T09:54:37.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f50e6200700 1 Processor -- start 2024-03-29T09:54:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3dce200700 1 -- 172.21.15.12:0/893436885 shutdown_connections 2024-03-29T09:54:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3dce200700 1 --2- 172.21.15.12:0/893436885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dc80ce2d0 0x7f3dc80ce760 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.100+0000 7f50e6200700 1 -- start start 2024-03-29T09:54:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3dce200700 1 --2- 172.21.15.12:0/893436885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc80cfbc0 0x7f3dc80cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3dce200700 1 -- 172.21.15.12:0/893436885 >> 172.21.15.12:0/893436885 conn(0x7f3dc8070f00 msgr2=0x7f3dc8071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.099+0000 7f3be6600700 1 -- 172.21.15.12:0/2648657997 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be01631e0 con 0x7f3be00ccd30 2024-03-29T09:54:37.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.100+0000 7f3be6600700 1 -- 172.21.15.12:0/2648657997 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be0140d90 con 0x7f3be0162410 2024-03-29T09:54:37.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.101+0000 7f3dce200700 1 -- 172.21.15.12:0/893436885 shutdown_connections 2024-03-29T09:54:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.100+0000 7f3bdfe00700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3be00ccd30 0x7f3be00cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.101+0000 7f3bdf400700 1 -- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be0162410 msgr2=0x7f3be013f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.101+0000 7f3bdf400700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be0162410 0x7f3be013f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.101+0000 7f3bdf400700 1 -- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3be00ccd30 msgr2=0x7f3be00cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.101+0000 7f3bdf400700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3be00ccd30 0x7f3be00cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.101+0000 7f3bdf400700 1 -- 172.21.15.12:0/2648657997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd0009550 con 0x7f3be00d0d00 2024-03-29T09:54:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.101+0000 7f3dce200700 1 -- 172.21.15.12:0/893436885 wait complete. 2024-03-29T09:54:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.101+0000 7f3c42800700 1 Processor -- start 2024-03-29T09:54:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.102+0000 7f50e6200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f50e015d710 0x7f50e00d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.103+0000 7f50e6200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e00cf640 con 0x7f50e00d1b70 2024-03-29T09:54:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.103+0000 7f50e6200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e00d0d30 con 0x7f50e00d2020 2024-03-29T09:54:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.103+0000 7f50e6200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e00cf220 con 0x7f50e015d710 2024-03-29T09:54:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.103+0000 7f50dfe00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f50e00d2020 0x7f50e00d0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46502/0 (socket says 172.21.15.12:46502) 2024-03-29T09:54:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.103+0000 7f50dfe00700 1 -- 172.21.15.12:0/1441605100 learned_addr learned my addr 172.21.15.12:0/1441605100 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.102+0000 7f3c42800700 1 -- start start 2024-03-29T09:54:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.103+0000 7f3c42800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c0724c0 0x7f3c3c14cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.103+0000 7f3c42800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c3c072910 0x7f3c3c0ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.103+0000 7f3c42800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c3c15d810 0x7f3c3c0ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603984133 0 0) 0x7f50e00d0d30 con 0x7f50e00d2020 2024-03-29T09:54:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.103+0000 7f3c42800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c1500d0 con 0x7f3c3c15d810 2024-03-29T09:54:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c40035b0 con 0x7f50e00d2020 2024-03-29T09:54:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f3c3b400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c0724c0 0x7f3c3c14cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f3c3aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c3c072910 0x7f3c3c0ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f50df400700 1 --2- 172.21.15.12:0/1441605100 >> v2:172.21.15.12:3300/0 conn(0x7f50e015d710 0x7f50e00d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f3c3b400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c0724c0 0x7f3c3c14cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51592/0 (socket says 172.21.15.12:51592) 2024-03-29T09:54:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f3dce200700 1 Processor -- start 2024-03-29T09:54:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f3c3b400700 1 -- 172.21.15.12:0/3778682365 learned_addr learned my addr 172.21.15.12:0/3778682365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3740769791 0 0) 0x7f50c40035b0 con 0x7f50e00d2020 2024-03-29T09:54:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 >> v2:172.21.15.12:3300/0 conn(0x7f50e015d710 msgr2=0x7f50e00d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f50de000700 1 --2- 172.21.15.12:0/1441605100 >> v2:172.21.15.12:3300/0 conn(0x7f50e015d710 0x7f50e00d12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 >> v1:172.21.15.12:6789/0 conn(0x7f50e00d1b70 legacy=0x7f50e00cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e00ce5a0 con 0x7f50e00d2020 2024-03-29T09:54:37.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c1f70a0 con 0x7f3c3c072910 2024-03-29T09:54:37.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.104+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c0af8a0 con 0x7f3c3c0724c0 2024-03-29T09:54:37.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f3c3b400700 1 -- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c3c072910 msgr2=0x7f3c3c0ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f3c3b400700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c3c072910 0x7f3c3c0ad2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f50c8001670 con 0x7f50e00d2020 2024-03-29T09:54:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 >> v1:172.21.15.159:6789/0 conn(0x7f50e00d2020 legacy=0x7f50e00d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f50de000700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50c4003a50 0x7f50c4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f50de000700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50c4004850 0x7f50c4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f50de000700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 0x7f50c400ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00cf220 con 0x7f50c40053c0 2024-03-29T09:54:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c400c310 con 0x7f50c4004850 2024-03-29T09:54:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c400c4d0 con 0x7f50c4003a50 2024-03-29T09:54:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f3bdf400700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3be00d0d00 0x7f3be00cf330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800efc0 tx=0x7f3bd800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f3c3b400700 1 -- 172.21.15.12:0/3778682365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c3c15d810 msgr2=0x7f3c3c0ad7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f3c3b400700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c3c15d810 0x7f3c3c0ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f3c3b400700 1 -- 172.21.15.12:0/3778682365 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c24009550 con 0x7f3c3c0724c0 2024-03-29T09:54:37.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f3bd7400700 1 -- 172.21.15.12:0/2648657997 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd800cdc0 con 0x7f3be00d0d00 2024-03-29T09:54:37.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f50dfe00700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 0x7f50c400ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f3bd7400700 1 -- 172.21.15.12:0/2648657997 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd800f040 con 0x7f3be00d0d00 2024-03-29T09:54:37.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f3bd7400700 1 -- 172.21.15.12:0/2648657997 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd80146a0 con 0x7f3be00d0d00 2024-03-29T09:54:37.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f3be6600700 1 -- 172.21.15.12:0/2648657997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be0141070 con 0x7f3be00d0d00 2024-03-29T09:54:37.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f50dea00700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50c4004850 0x7f50c4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f3c3b400700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c0724c0 0x7f3c3c14cee0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009ed0 tx=0x7f3c340084e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f50dfe00700 1 -- 172.21.15.12:0/1441605100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50c4003a50 msgr2=0x7f50c4003e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T09:54:37.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f50dfe00700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50c4003a50 0x7f50c4003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f50dfe00700 1 -- 172.21.15.12:0/1441605100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50c4004850 msgr2=0x7f50c4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f50dfe00700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50c4004850 0x7f50c4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f50dfe00700 1 -- 172.21.15.12:0/1441605100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c40035b0 con 0x7f50c40053c0 2024-03-29T09:54:37.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f50dea00700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50c4004850 0x7f50c4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:37.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f3be6600700 1 -- 172.21.15.12:0/2648657997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be0163710 con 0x7f3be00d0d00 2024-03-29T09:54:37.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f50dfe00700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 0x7f50c400ba30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f50c800c210 tx=0x7f50c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676a07a92693642c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f50dfe00700 1 -- 172.21.15.12:0/1441605100 reap_dead start 2024-03-29T09:54:37.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f3c33400700 1 -- 172.21.15.12:0/3778682365 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c34010070 con 0x7f3c3c0724c0 2024-03-29T09:54:37.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f3c33400700 1 -- 172.21.15.12:0/3778682365 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3400be40 con 0x7f3c3c0724c0 2024-03-29T09:54:37.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f3c33400700 1 -- 172.21.15.12:0/3778682365 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c34008d30 con 0x7f3c3c0724c0 2024-03-29T09:54:37.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3c0afb80 con 0x7f3c3c0724c0 2024-03-29T09:54:37.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.105+0000 7f3dce200700 1 -- start start 2024-03-29T09:54:37.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f3dce200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc806ca60 0x7f3dc8149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f3dce200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dc80ce2d0 0x7f3dc80ada90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.106+0000 7f3dce200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3dc80cfbc0 0x7f3dc81495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f3dce200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc81f2b80 con 0x7f3dc80ce2d0 2024-03-29T09:54:37.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f3dce200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc81f2e60 con 0x7f3dc80cfbc0 2024-03-29T09:54:37.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f3dce200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc81f3140 con 0x7f3dc806ca60 2024-03-29T09:54:37.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.107+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3c0b0140 con 0x7f3c3c0724c0 2024-03-29T09:54:37.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.109+0000 7f3bd4c00700 1 -- 172.21.15.12:0/2648657997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb40052f0 con 0x7f3be00d0d00 2024-03-29T09:54:37.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.109+0000 7f3dc7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc806ca60 0x7f3dc8149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.109+0000 7f3dc7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc806ca60 0x7f3dc8149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51596/0 (socket says 172.21.15.12:51596) 2024-03-29T09:54:37.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.109+0000 7f3dc7400700 1 -- 172.21.15.12:0/1279355937 learned_addr learned my addr 172.21.15.12:0/1279355937 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.109+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3c06b7d0 con 0x7f3c3c0724c0 2024-03-29T09:54:37.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.110+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c800cdb0 con 0x7f50c40053c0 2024-03-29T09:54:37.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.110+0000 7f3dc7e00700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3dc80cfbc0 0x7f3dc81495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.110+0000 7f3dc7400700 1 -- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3dc80cfbc0 msgr2=0x7f3dc81495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.110+0000 7f3dc7400700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3dc80cfbc0 0x7f3dc81495f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.110+0000 7f3dc7400700 1 -- 172.21.15.12:0/1279355937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dc80ce2d0 msgr2=0x7f3dc80ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.110+0000 7f3dc7400700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dc80ce2d0 0x7f3dc80ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.110+0000 7f3dc7400700 1 -- 172.21.15.12:0/1279355937 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc0009a70 con 0x7f3dc806ca60 2024-03-29T09:54:37.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.110+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c8005960 con 0x7f50c40053c0 2024-03-29T09:54:37.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.110+0000 7f50de000700 1 -- 172.21.15.12:0/1441605100 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8005cd0 con 0x7f50c40053c0 2024-03-29T09:54:37.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.111+0000 7f3dc7400700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc806ca60 0x7f3dc8149e80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3db400c700 tx=0x7f3db400cac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.111+0000 7f50e6200700 1 -- 172.21.15.12:0/1441605100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 msgr2=0x7f50c400ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.111+0000 7f50e6200700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 0x7f50c400ba30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f50c800c210 tx=0x7f50c4005880 comp rx=0 tx=0).stop 2024-03-29T09:54:37.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.112+0000 7f50e6200700 1 -- 172.21.15.12:0/1441605100 shutdown_connections 2024-03-29T09:54:37.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.112+0000 7f50e6200700 1 --2- 172.21.15.12:0/1441605100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 0x7f50c400ba30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.112+0000 7f50e6200700 1 -- 172.21.15.12:0/1441605100 >> 172.21.15.12:0/1441605100 conn(0x7f50e008aff0 msgr2=0x7f50e00d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.112+0000 7f50e6200700 1 -- 172.21.15.12:0/1441605100 shutdown_connections 2024-03-29T09:54:37.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.112+0000 7f50e6200700 1 -- 172.21.15.12:0/1441605100 wait complete. 2024-03-29T09:54:37.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.112+0000 7f3bd7400700 1 -- 172.21.15.12:0/2648657997 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3bd8014800 con 0x7f3be00d0d00 2024-03-29T09:54:37.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.114+0000 7f3bd7400700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3bc006ac90 0x7f3bc006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.113+0000 7f3c33400700 1 -- 172.21.15.12:0/3778682365 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3c34014070 con 0x7f3c3c0724c0 2024-03-29T09:54:37.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.115+0000 7f3c33400700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c1806ad60 0x7f3c1806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.115+0000 7f3c33400700 1 -- 172.21.15.12:0/3778682365 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c34089230 con 0x7f3c3c0724c0 2024-03-29T09:54:37.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.114+0000 7f3bd7400700 1 -- 172.21.15.12:0/2648657997 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bd808a520 con 0x7f3be00d0d00 2024-03-29T09:54:37.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.116+0000 7f3dbf400700 1 -- 172.21.15.12:0/1279355937 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db4005d70 con 0x7f3dc806ca60 2024-03-29T09:54:37.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.116+0000 7f3dbf400700 1 -- 172.21.15.12:0/1279355937 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db40070d0 con 0x7f3dc806ca60 2024-03-29T09:54:37.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.116+0000 7f3dbf400700 1 -- 172.21.15.12:0/1279355937 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db4007460 con 0x7f3dc806ca60 2024-03-29T09:54:37.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.116+0000 7f3dce200700 1 -- 172.21.15.12:0/1279355937 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc81f3420 con 0x7f3dc806ca60 2024-03-29T09:54:37.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.116+0000 7f3c3aa00700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c1806ad60 0x7f3c1806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.116+0000 7f50e6200700 1 Processor -- start 2024-03-29T09:54:37.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.117+0000 7f3bdea00700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3bc006ac90 0x7f3bc006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.117+0000 7f3dce200700 1 -- 172.21.15.12:0/1279355937 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc81f2830 con 0x7f3dc806ca60 2024-03-29T09:54:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.117+0000 7f50e6200700 1 -- start start 2024-03-29T09:54:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.118+0000 7f3c33400700 1 -- 172.21.15.12:0/3778682365 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3c34054480 con 0x7f3c3c0724c0 2024-03-29T09:54:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.120+0000 7f3dbf400700 1 -- 172.21.15.12:0/1279355937 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3db40075c0 con 0x7f3dc806ca60 2024-03-29T09:54:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.120+0000 7f3c3aa00700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c1806ad60 0x7f3c1806d220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3c24000c00 tx=0x7f3c2401d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.120+0000 7f50e6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 0x7f50e0160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.121+0000 7f50e6200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50e0156370 0x7f50e01f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.121+0000 7f50e6200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50e00b45e0 0x7f50e00b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.121+0000 7f50e6200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00b4fb0 con 0x7f50c40053c0 2024-03-29T09:54:37.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.121+0000 7f50e6200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e0158b20 con 0x7f50e0156370 2024-03-29T09:54:37.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.121+0000 7f50e6200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e01f32a0 con 0x7f50e00b45e0 2024-03-29T09:54:37.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.121+0000 7f3dce200700 1 -- 172.21.15.12:0/1279355937 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da00052f0 con 0x7f3dc806ca60 2024-03-29T09:54:37.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.121+0000 7f50df400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 0x7f50e0160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f50df400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 0x7f50e0160690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40878/0 (socket says 172.21.15.12:40878) 2024-03-29T09:54:37.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f50dfe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50e00b45e0 0x7f50e00b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f50dfe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50e00b45e0 0x7f50e00b4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51604/0 (socket says 172.21.15.12:51604) 2024-03-29T09:54:37.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f50dfe00700 1 -- 172.21.15.12:0/793904374 learned_addr learned my addr 172.21.15.12:0/793904374 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f3bd7400700 1 -- 172.21.15.12:0/2648657997 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3bd80556e0 con 0x7f3be00d0d00 2024-03-29T09:54:37.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f3bdea00700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3bc006ac90 0x7f3bc006d150 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0000c00 tx=0x7f3bd0008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f50dea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50e0156370 0x7f50e01f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f3dbf400700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3da806ace0 0x7f3da806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.123+0000 7f3dbf400700 1 -- 172.21.15.12:0/1279355937 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3db40888b0 con 0x7f3dc806ca60 2024-03-29T09:54:37.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f50dfe00700 1 -- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50e0156370 msgr2=0x7f50e01f06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f50dfe00700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50e0156370 0x7f50e01f06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f50dfe00700 1 -- 172.21.15.12:0/793904374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 msgr2=0x7f50e0160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.122+0000 7f50dfe00700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 0x7f50e0160690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.123+0000 7f50dfe00700 1 -- 172.21.15.12:0/793904374 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e01f3560 con 0x7f50e00b45e0 2024-03-29T09:54:37.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.123+0000 7f50dfe00700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50e00b45e0 0x7f50e00b4a70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f50e008bc90 tx=0x7f50c8001930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.123+0000 7f3dc6a00700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3da806ace0 0x7f3da806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.124+0000 7f3dc6a00700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3da806ace0 0x7f3da806d1a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3db8006b20 tx=0x7f3db8006ab0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.127+0000 7f50d7400700 1 -- 172.21.15.12:0/793904374 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8004c80 con 0x7f50e00b45e0 2024-03-29T09:54:37.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.127+0000 7f50d7400700 1 -- 172.21.15.12:0/793904374 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c80019a0 con 0x7f50e00b45e0 2024-03-29T09:54:37.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.127+0000 7f3dbf400700 1 -- 172.21.15.12:0/1279355937 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3db4053b60 con 0x7f3dc806ca60 2024-03-29T09:54:37.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.127+0000 7f50e6200700 1 -- 172.21.15.12:0/793904374 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e01f3840 con 0x7f50e00b45e0 2024-03-29T09:54:37.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.128+0000 7f50e6200700 1 -- 172.21.15.12:0/793904374 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e01f3e00 con 0x7f50e00b45e0 2024-03-29T09:54:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.128+0000 7f50d7400700 1 -- 172.21.15.12:0/793904374 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8001b30 con 0x7f50e00b45e0 2024-03-29T09:54:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.131+0000 7f50e6200700 1 -- 172.21.15.12:0/793904374 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b80052f0 con 0x7f50e00b45e0 2024-03-29T09:54:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.132+0000 7f50d7400700 1 -- 172.21.15.12:0/793904374 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f50c8007070 con 0x7f50e00b45e0 2024-03-29T09:54:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.133+0000 7f50d7400700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50bc06ac90 0x7f50bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.134+0000 7f50d7400700 1 -- 172.21.15.12:0/793904374 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50c80590a0 con 0x7f50e00b45e0 2024-03-29T09:54:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.134+0000 7f50df400700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50bc06ac90 0x7f50bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.135+0000 7f50df400700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50bc06ac90 0x7f50bc06d150 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f50e00684a0 tx=0x7f50d8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.140+0000 7f50d7400700 1 -- 172.21.15.12:0/793904374 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f50c8055470 con 0x7f50e00b45e0 2024-03-29T09:54:37.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.158+0000 7f6dfbc00700 1 Processor -- start 2024-03-29T09:54:37.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.158+0000 7f6dfbc00700 1 -- start start 2024-03-29T09:54:37.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.160+0000 7f6dfbc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6df40ad3b0 0x7f6df41684a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.160+0000 7f6dfbc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df40cf890 con 0x7f6df415d7a0 2024-03-29T09:54:37.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.161+0000 7f6dfbc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df406df90 con 0x7f6df40724c0 2024-03-29T09:54:37.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.161+0000 7f6dfbc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df406eba0 con 0x7f6df40ad3b0 2024-03-29T09:54:37.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.161+0000 7f6df9c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6df40ad3b0 0x7f6df41684a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.161+0000 7f6df9c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6df40ad3b0 0x7f6df41684a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:47024/0 (socket says 172.21.15.12:47024) 2024-03-29T09:54:37.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.161+0000 7f6df9c00700 1 -- 172.21.15.12:0/3690446417 learned_addr learned my addr 172.21.15.12:0/3690446417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.161+0000 7f6df9c00700 1 -- 172.21.15.12:0/3690446417 >> v1:172.21.15.159:6789/0 conn(0x7f6df40724c0 legacy=0x7f6df40cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.161+0000 7f6df9c00700 1 -- 172.21.15.12:0/3690446417 >> v1:172.21.15.12:6789/0 conn(0x7f6df415d7a0 legacy=0x7f6df40cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.161+0000 7f6df9c00700 1 -- 172.21.15.12:0/3690446417 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df406e270 con 0x7f6df40ad3b0 2024-03-29T09:54:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.162+0000 7f6df9c00700 1 --2- 172.21.15.12:0/3690446417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 0x7f6df41684a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc0098f0 tx=0x7f6ddc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2683a74180a5883d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.163+0000 7f6df3400700 1 -- 172.21.15.12:0/3690446417 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc004230 con 0x7f6df40ad3b0 2024-03-29T09:54:37.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.163+0000 7f6df3400700 1 -- 172.21.15.12:0/3690446417 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ddc004390 con 0x7f6df40ad3b0 2024-03-29T09:54:37.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.163+0000 7f6df3400700 1 -- 172.21.15.12:0/3690446417 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc0036a0 con 0x7f6df40ad3b0 2024-03-29T09:54:37.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.164+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3690446417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 msgr2=0x7f6df41684a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.164+0000 7f6dfbc00700 1 --2- 172.21.15.12:0/3690446417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 0x7f6df41684a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc0098f0 tx=0x7f6ddc009c00 comp rx=0 tx=0).stop 2024-03-29T09:54:37.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.164+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3690446417 shutdown_connections 2024-03-29T09:54:37.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.164+0000 7f6dfbc00700 1 --2- 172.21.15.12:0/3690446417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 0x7f6df41684a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.165+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3690446417 >> 172.21.15.12:0/3690446417 conn(0x7f6df4070f00 msgr2=0x7f6df4071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.165+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3690446417 shutdown_connections 2024-03-29T09:54:37.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.165+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3690446417 wait complete. 2024-03-29T09:54:37.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.168+0000 7f24fa400700 1 Processor -- start 2024-03-29T09:54:37.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.168+0000 7f24fa400700 1 -- start start 2024-03-29T09:54:37.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.170+0000 7f6dfbc00700 1 Processor -- start 2024-03-29T09:54:37.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.170+0000 7f6dfbc00700 1 -- start start 2024-03-29T09:54:37.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.171+0000 7f24fa400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f24f415d810 0x7f24f40b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.171+0000 7f24f3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f24f415d810 0x7f24f40b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.171+0000 7f24f2a00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f24f4072910 0x7f24f406e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46522/0 (socket says 172.21.15.12:46522) 2024-03-29T09:54:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f24fa400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f40cc140 con 0x7f24f406e810 2024-03-29T09:54:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f24fa400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f415df80 con 0x7f24f4072910 2024-03-29T09:54:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f24fa400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f415e260 con 0x7f24f415d810 2024-03-29T09:54:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f24f2a00700 1 -- 172.21.15.12:0/3276310193 learned_addr learned my addr 172.21.15.12:0/3276310193 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:37.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607745247 0 0) 0x7f24f415df80 con 0x7f24f4072910 2024-03-29T09:54:37.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d80035b0 con 0x7f24f4072910 2024-03-29T09:54:37.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617812700 0 0) 0x7f24f40cc140 con 0x7f24f406e810 2024-03-29T09:54:37.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f6dfbc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6df40724c0 0x7f6df41f8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f6dfbc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 0x7f6df4098550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f6dfbc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6df415d7a0 0x7f6df41f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f6dfbc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df4099370 con 0x7f6df415d7a0 2024-03-29T09:54:37.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f6dfbc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df4099650 con 0x7f6df40ad3b0 2024-03-29T09:54:37.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.172+0000 7f6dfbc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df4099930 con 0x7f6df40724c0 2024-03-29T09:54:37.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.173+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f415df80 con 0x7f24f406e810 2024-03-29T09:54:37.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.173+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2715767277 0 0) 0x7f24d80035b0 con 0x7f24f4072910 2024-03-29T09:54:37.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.173+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 >> v2:172.21.15.12:3300/0 conn(0x7f24f415d810 msgr2=0x7f24f40b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.173+0000 7f6df9200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6df40724c0 0x7f6df41f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.173+0000 7f6df3e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 0x7f6df4098550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.173+0000 7f6df3e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 0x7f6df4098550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:47040/0 (socket says 172.21.15.12:47040) 2024-03-29T09:54:37.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.173+0000 7f6df3e00700 1 -- 172.21.15.12:0/3589634233 learned_addr learned my addr 172.21.15.12:0/3589634233 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 --2- 172.21.15.12:0/3276310193 >> v2:172.21.15.12:3300/0 conn(0x7f24f415d810 0x7f24f40b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 >> v1:172.21.15.12:6789/0 conn(0x7f24f406e810 legacy=0x7f24f40ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f40cbdd0 con 0x7f24f4072910 2024-03-29T09:54:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f24e4001670 con 0x7f24f4072910 2024-03-29T09:54:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 >> v1:172.21.15.159:6789/0 conn(0x7f24f4072910 legacy=0x7f24f406e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f6df9c00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6df415d7a0 0x7f6df41f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8003b90 0x7f24d8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24d8004850 0x7f24d8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24d8005210 0x7f24d8010020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec0015b0 con 0x7f24d8003b90 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f415e260 con 0x7f24d8005210 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d8010560 con 0x7f24d8004850 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f3400700 1 --2- 172.21.15.12:0/3276310193 >> v2:172.21.15.12:3300/0 conn(0x7f24f415d810 0x7f24f40b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f24f3e00700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24d8005210 0x7f24d8010020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.175+0000 7f24f3400700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8003b90 0x7f24d8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.175+0000 7f24f3e00700 1 -- 172.21.15.12:0/3276310193 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24d8004850 msgr2=0x7f24d8004cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.175+0000 7f24f3e00700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24d8004850 0x7f24d8004cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.175+0000 7f24f3e00700 1 -- 172.21.15.12:0/3276310193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8003b90 msgr2=0x7f24d8003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.175+0000 7f24f3e00700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8003b90 0x7f24d8003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.175+0000 7f24f3e00700 1 -- 172.21.15.12:0/3276310193 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d80035b0 con 0x7f24d8005210 2024-03-29T09:54:37.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f6df3e00700 1 -- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6df40724c0 msgr2=0x7f6df41f8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f6df3e00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6df40724c0 0x7f6df41f8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f6df3e00700 1 -- 172.21.15.12:0/3589634233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6df415d7a0 msgr2=0x7f6df41f7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.174+0000 7f6df3e00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6df415d7a0 0x7f6df41f7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.175+0000 7f6df3e00700 1 -- 172.21.15.12:0/3589634233 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc009550 con 0x7f6df40ad3b0 2024-03-29T09:54:37.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.176+0000 7f24f3e00700 1 -- 172.21.15.12:0/3276310193 reap_dead start 2024-03-29T09:54:37.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.176+0000 7f24f3400700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8003b90 0x7f24d8003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:37.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.175+0000 7f6df3e00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 0x7f6df4098550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6de400c8c0 tx=0x7f6de400cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.176+0000 7f6df0c00700 1 -- 172.21.15.12:0/3589634233 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de4003e10 con 0x7f6df40ad3b0 2024-03-29T09:54:37.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.176+0000 7f24f3e00700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24d8005210 0x7f24d8010020 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00a770 tx=0x7f24d8005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58b580d005630db7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.176+0000 7f6df0c00700 1 -- 172.21.15.12:0/3589634233 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de400ce60 con 0x7f6df40ad3b0 2024-03-29T09:54:37.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.176+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3589634233 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df4099c10 con 0x7f6df40ad3b0 2024-03-29T09:54:37.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.176+0000 7f6df0c00700 1 -- 172.21.15.12:0/3589634233 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de40052b0 con 0x7f6df40ad3b0 2024-03-29T09:54:37.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.177+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3589634233 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df41f8cb0 con 0x7f6df40ad3b0 2024-03-29T09:54:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.177+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec00a210 con 0x7f24d8005210 2024-03-29T09:54:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.177+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ec0052f0 con 0x7f24d8005210 2024-03-29T09:54:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.177+0000 7f24f2000700 1 -- 172.21.15.12:0/3276310193 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec0056a0 con 0x7f24d8005210 2024-03-29T09:54:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.178+0000 7f24fa400700 1 -- 172.21.15.12:0/3276310193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24d8005210 msgr2=0x7f24d8010020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.178+0000 7f24fa400700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24d8005210 0x7f24d8010020 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00a770 tx=0x7f24d8005880 comp rx=0 tx=0).stop 2024-03-29T09:54:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.178+0000 7f24fa400700 1 -- 172.21.15.12:0/3276310193 shutdown_connections 2024-03-29T09:54:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.178+0000 7f24fa400700 1 --2- 172.21.15.12:0/3276310193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24d8005210 0x7f24d8010020 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.178+0000 7f24fa400700 1 -- 172.21.15.12:0/3276310193 >> 172.21.15.12:0/3276310193 conn(0x7f24f4070f00 msgr2=0x7f24f4071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.178+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3589634233 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df406b7d0 con 0x7f6df40ad3b0 2024-03-29T09:54:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.179+0000 7f24fa400700 1 -- 172.21.15.12:0/3276310193 shutdown_connections 2024-03-29T09:54:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.179+0000 7f24fa400700 1 -- 172.21.15.12:0/3276310193 wait complete. 2024-03-29T09:54:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.183+0000 7f24fa400700 1 Processor -- start 2024-03-29T09:54:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.182+0000 7f6df0c00700 1 -- 172.21.15.12:0/3589634233 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6de4005410 con 0x7f6df40ad3b0 2024-03-29T09:54:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.183+0000 7f24fa400700 1 -- start start 2024-03-29T09:54:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.185+0000 7f6df0c00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6dd006ace0 0x7f6dd006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.185+0000 7f6df0c00700 1 -- 172.21.15.12:0/3589634233 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6de4088fa0 con 0x7f6df40ad3b0 2024-03-29T09:54:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.185+0000 7f6df9200700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6dd006ace0 0x7f6dd006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.186+0000 7f24fa400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8005210 0x7f24f4154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.186+0000 7f24fa400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24f414cc50 0x7f24f414d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.186+0000 7f24fa400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24f4152e20 0x7f24f41532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.186+0000 7f24fa400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f41521d0 con 0x7f24d8005210 2024-03-29T09:54:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.186+0000 7f6df0c00700 1 -- 172.21.15.12:0/3589634233 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6de4054160 con 0x7f6df40ad3b0 2024-03-29T09:54:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.186+0000 7f24fa400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f41526d0 con 0x7f24f4152e20 2024-03-29T09:54:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.186+0000 7f24fa400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f414e290 con 0x7f24f414cc50 2024-03-29T09:54:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.187+0000 7f24f2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24f414cc50 0x7f24f414d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.187+0000 7f24f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8005210 0x7f24f4154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.187+0000 7f24f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8005210 0x7f24f4154a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40918/0 (socket says 172.21.15.12:40918) 2024-03-29T09:54:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.187+0000 7f24f3400700 1 -- 172.21.15.12:0/2083521303 learned_addr learned my addr 172.21.15.12:0/2083521303 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.187+0000 7f24f3e00700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24f4152e20 0x7f24f41532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.188+0000 7f24f3e00700 1 -- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24f414cc50 msgr2=0x7f24f414d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.188+0000 7f24f3e00700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24f414cc50 0x7f24f414d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.188+0000 7f24f3e00700 1 -- 172.21.15.12:0/2083521303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8005210 msgr2=0x7f24f4154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.188+0000 7f6df9200700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6dd006ace0 0x7f6dd006d1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009870 tx=0x7f6dec008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.188+0000 7f24f3e00700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8005210 0x7f24f4154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.188+0000 7f24f3e00700 1 -- 172.21.15.12:0/2083521303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f4152990 con 0x7f24f4152e20 2024-03-29T09:54:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.188+0000 7f24f3400700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8005210 0x7f24f4154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.189+0000 7f24f3e00700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24f4152e20 0x7f24f41532a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f24f406f270 tx=0x7f24ec001ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.189+0000 7f24eb400700 1 -- 172.21.15.12:0/2083521303 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec00a210 con 0x7f24f4152e20 2024-03-29T09:54:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.189+0000 7f24eb400700 1 -- 172.21.15.12:0/2083521303 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ec005460 con 0x7f24f4152e20 2024-03-29T09:54:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.189+0000 7f24eb400700 1 -- 172.21.15.12:0/2083521303 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec0057f0 con 0x7f24f4152e20 2024-03-29T09:54:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.189+0000 7f24fa400700 1 -- 172.21.15.12:0/2083521303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f4154f60 con 0x7f24f4152e20 2024-03-29T09:54:37.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.190+0000 7f24fa400700 1 -- 172.21.15.12:0/2083521303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f414dcd0 con 0x7f24f4152e20 2024-03-29T09:54:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.193+0000 7f24eb400700 1 -- 172.21.15.12:0/2083521303 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f24ec012440 con 0x7f24f4152e20 2024-03-29T09:54:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.195+0000 7f24fa400700 1 -- 172.21.15.12:0/2083521303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f406b7d0 con 0x7f24f4152e20 2024-03-29T09:54:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.195+0000 7f24eb400700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24d006ace0 0x7f24d006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.195+0000 7f2878e00700 1 Processor -- start 2024-03-29T09:54:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.196+0000 7f24eb400700 1 -- 172.21.15.12:0/2083521303 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24ec059e40 con 0x7f24f4152e20 2024-03-29T09:54:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.196+0000 7f2878e00700 1 -- start start 2024-03-29T09:54:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.196+0000 7f24f3400700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24d006ace0 0x7f24d006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.197+0000 7f24f3400700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24d006ace0 0x7f24d006d1a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f24dc014470 tx=0x7f24dc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.199+0000 7f2878e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f287415af30 0x7f287415b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.199+0000 7f2878e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f287406da80 0x7f287415e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.200+0000 7f2871e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f287415af30 0x7f287415b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.200+0000 7f2871e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f287415af30 0x7f287415b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:47068/0 (socket says 172.21.15.12:47068) 2024-03-29T09:54:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.200+0000 7f2872800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f287406da80 0x7f287415e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.200+0000 7f2871e00700 1 -- 172.21.15.12:0/97691388 learned_addr learned my addr 172.21.15.12:0/97691388 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.200+0000 7f2878e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287406eae0 con 0x7f28740b4a50 2024-03-29T09:54:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.200+0000 7f2878e00700 1 -- 172.21.15.12:0/97691388 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287406f660 con 0x7f287406da80 2024-03-29T09:54:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.200+0000 7f2878e00700 1 -- 172.21.15.12:0/97691388 --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287406faf0 con 0x7f287415af30 2024-03-29T09:54:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.201+0000 7f24eb400700 1 -- 172.21.15.12:0/2083521303 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f24ec056210 con 0x7f24f4152e20 2024-03-29T09:54:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.201+0000 7f2872800700 1 -- 172.21.15.12:0/97691388 >> v2:172.21.15.159:3300/0 conn(0x7f287415af30 msgr2=0x7f287415b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.201+0000 7f2872800700 1 --2- 172.21.15.12:0/97691388 >> v2:172.21.15.159:3300/0 conn(0x7f287415af30 0x7f287415b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.201+0000 7f2872800700 1 -- 172.21.15.12:0/97691388 >> v1:172.21.15.12:6789/0 conn(0x7f28740b4a50 legacy=0x7f287406d370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T09:54:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.201+0000 7f2872800700 1 -- 172.21.15.12:0/97691388 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287406e150 con 0x7f287406da80 2024-03-29T09:54:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.201+0000 7f2872800700 1 --2- 172.21.15.12:0/97691388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f287406da80 0x7f287415e6f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f287406fda0 tx=0x7f286c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855ee4ca9c841662 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.202+0000 7f2871400700 1 -- 172.21.15.12:0/97691388 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c004230 con 0x7f287406da80 2024-03-29T09:54:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.202+0000 7f2871400700 1 -- 172.21.15.12:0/97691388 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286c004390 con 0x7f287406da80 2024-03-29T09:54:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.202+0000 7f2871400700 1 -- 172.21.15.12:0/97691388 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c0036a0 con 0x7f287406da80 2024-03-29T09:54:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.203+0000 7f2878e00700 1 -- 172.21.15.12:0/97691388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f287406da80 msgr2=0x7f287415e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.203+0000 7f2878e00700 1 --2- 172.21.15.12:0/97691388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f287406da80 0x7f287415e6f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f287406fda0 tx=0x7f286c009b20 comp rx=0 tx=0).stop 2024-03-29T09:54:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.204+0000 7f2878e00700 1 -- 172.21.15.12:0/97691388 shutdown_connections 2024-03-29T09:54:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.204+0000 7f2878e00700 1 --2- 172.21.15.12:0/97691388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f287406da80 0x7f287415e6f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.204+0000 7f2878e00700 1 --2- 172.21.15.12:0/97691388 >> v2:172.21.15.159:3300/0 conn(0x7f287415af30 0x7f287415b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.204+0000 7f2878e00700 1 -- 172.21.15.12:0/97691388 >> 172.21.15.12:0/97691388 conn(0x7f2874071fb0 msgr2=0x7f287406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.204+0000 7f2878e00700 1 -- 172.21.15.12:0/97691388 shutdown_connections 2024-03-29T09:54:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.204+0000 7f2878e00700 1 -- 172.21.15.12:0/97691388 wait complete. 2024-03-29T09:54:37.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.208+0000 7f2878e00700 1 Processor -- start 2024-03-29T09:54:37.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.210+0000 7f2878e00700 1 -- start start 2024-03-29T09:54:37.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.212+0000 7f2878e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f287406da80 0x7f2874150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.212+0000 7f2878e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28740b4a50 0x7f287415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.212+0000 7f2878e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f287415af30 0x7f28740cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.212+0000 7f2878e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28741513d0 con 0x7f287406da80 2024-03-29T09:54:37.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.212+0000 7f2878e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28740ccb50 con 0x7f287415af30 2024-03-29T09:54:37.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.212+0000 7f2878e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28740cce30 con 0x7f28740b4a50 2024-03-29T09:54:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.213+0000 7f2872800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f287415af30 0x7f28740cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.213+0000 7f2869400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28740b4a50 0x7f287415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.213+0000 7f2872800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f287415af30 0x7f28740cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:47084/0 (socket says 172.21.15.12:47084) 2024-03-29T09:54:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.213+0000 7f2872800700 1 -- 172.21.15.12:0/3089378654 learned_addr learned my addr 172.21.15.12:0/3089378654 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.213+0000 7f2872800700 1 -- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28740b4a50 msgr2=0x7f287415ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.213+0000 7f2872800700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28740b4a50 0x7f287415ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.213+0000 7f2872800700 1 -- 172.21.15.12:0/3089378654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f287406da80 msgr2=0x7f2874150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.213+0000 7f2872800700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f287406da80 0x7f2874150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.213+0000 7f2872800700 1 -- 172.21.15.12:0/3089378654 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c0094a0 con 0x7f287415af30 2024-03-29T09:54:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.213+0000 7f2869400700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28740b4a50 0x7f287415ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:37.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.216+0000 7f2872800700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f287415af30 0x7f28740cdbf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f286c009b60 tx=0x7f286c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.216+0000 7f286aa00700 1 -- 172.21.15.12:0/3089378654 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c0045e0 con 0x7f287415af30 2024-03-29T09:54:37.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.216+0000 7f286aa00700 1 -- 172.21.15.12:0/3089378654 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286c003e90 con 0x7f287415af30 2024-03-29T09:54:37.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.216+0000 7f286aa00700 1 -- 172.21.15.12:0/3089378654 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c011870 con 0x7f287415af30 2024-03-29T09:54:37.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.217+0000 7f2878e00700 1 -- 172.21.15.12:0/3089378654 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28740cd110 con 0x7f287415af30 2024-03-29T09:54:37.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.217+0000 7f2878e00700 1 -- 172.21.15.12:0/3089378654 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28740cd6d0 con 0x7f287415af30 2024-03-29T09:54:37.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.219+0000 7f2878e00700 1 -- 172.21.15.12:0/3089378654 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2874067720 con 0x7f287415af30 2024-03-29T09:54:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.222+0000 7f286aa00700 1 -- 172.21.15.12:0/3089378654 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f286c018050 con 0x7f287415af30 2024-03-29T09:54:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.223+0000 7f286aa00700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f285006ac90 0x7f285006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.224+0000 7f286aa00700 1 -- 172.21.15.12:0/3089378654 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f286c059a80 con 0x7f287415af30 2024-03-29T09:54:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.224+0000 7f2871e00700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f285006ac90 0x7f285006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.225+0000 7f286aa00700 1 -- 172.21.15.12:0/3089378654 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f286c055e50 con 0x7f287415af30 2024-03-29T09:54:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.226+0000 7f2871e00700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f285006ac90 0x7f285006d150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f285c0097f0 tx=0x7f285c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.520+0000 7f560cc00700 1 -- 172.21.15.12:0/4266475902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5600052c80 con 0x7f56000532e0 2024-03-29T09:54:37.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.521+0000 7f55fe000700 1 -- 172.21.15.12:0/4266475902 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f56080ce740 con 0x7f56000532e0 2024-03-29T09:54:37.521 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:37.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.525+0000 7f560cc00700 1 -- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f55e806acb0 msgr2=0x7f55e806d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.525+0000 7f560cc00700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f55e806acb0 0x7f55e806d170 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f55f4001380 tx=0x7f55f4001310 comp rx=0 tx=0).stop 2024-03-29T09:54:37.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.526+0000 7f560cc00700 1 -- 172.21.15.12:0/4266475902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56000532e0 msgr2=0x7f56000dc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.526+0000 7f560cc00700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56000532e0 0x7f56000dc210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5608000eb0 tx=0x7f5608068110 comp rx=0 tx=0).stop 2024-03-29T09:54:37.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.526+0000 7f560cc00700 1 -- 172.21.15.12:0/4266475902 shutdown_connections 2024-03-29T09:54:37.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.527+0000 7f560cc00700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f55e806acb0 0x7f55e806d170 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.527+0000 7f560cc00700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56000532e0 0x7f56000dc210 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.527+0000 7f560cc00700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f560004f790 0x7f56001854f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.527+0000 7f560cc00700 1 --2- 172.21.15.12:0/4266475902 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560004fb70 0x7f56000dbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.527+0000 7f560cc00700 1 -- 172.21.15.12:0/4266475902 >> 172.21.15.12:0/4266475902 conn(0x7f5600054b10 msgr2=0x7f56000dfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.527+0000 7f560cc00700 1 -- 172.21.15.12:0/4266475902 shutdown_connections 2024-03-29T09:54:37.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.528+0000 7f560cc00700 1 -- 172.21.15.12:0/4266475902 wait complete. 2024-03-29T09:54:37.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-03-29T09:54:37.559 DEBUG:teuthology.parallel:result is None 2024-03-29T09:54:37.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.579+0000 7f3bd4c00700 1 -- 172.21.15.12:0/2648657997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3bb4005870 con 0x7f3be00d0d00 2024-03-29T09:54:37.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.580+0000 7f3bd7400700 1 -- 172.21.15.12:0/2648657997 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3bd8058d00 con 0x7f3be00d0d00 2024-03-29T09:54:37.579 INFO:teuthology.orchestra.run.smithi012.stdout:55834574853 2024-03-29T09:54:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.582+0000 7f3bd4c00700 1 -- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3bc006ac90 msgr2=0x7f3bc006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.582+0000 7f3bd4c00700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3bc006ac90 0x7f3bc006d150 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0000c00 tx=0x7f3bd0008030 comp rx=0 tx=0).stop 2024-03-29T09:54:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.582+0000 7f3bd4c00700 1 -- 172.21.15.12:0/2648657997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3be00d0d00 msgr2=0x7f3be00cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.582+0000 7f3bd4c00700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3be00d0d00 0x7f3be00cf330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800efc0 tx=0x7f3bd800c3a0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.583+0000 7f3bd4c00700 1 -- 172.21.15.12:0/2648657997 shutdown_connections 2024-03-29T09:54:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.583+0000 7f3bd4c00700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3bc006ac90 0x7f3bc006d150 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.583+0000 7f3bd4c00700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3be00ccd30 0x7f3be00cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.583+0000 7f3bd4c00700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3be00d0d00 0x7f3be00cf330 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.583+0000 7f3bd4c00700 1 --2- 172.21.15.12:0/2648657997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3be0162410 0x7f3be013f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.583+0000 7f3bd4c00700 1 -- 172.21.15.12:0/2648657997 >> 172.21.15.12:0/2648657997 conn(0x7f3be00d0090 msgr2=0x7f3be0161720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.583+0000 7f3bd4c00700 1 -- 172.21.15.12:0/2648657997 shutdown_connections 2024-03-29T09:54:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.583+0000 7f3bd4c00700 1 -- 172.21.15.12:0/2648657997 wait complete. 2024-03-29T09:54:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.592+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3c3c067720 con 0x7f3c3c0724c0 2024-03-29T09:54:37.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.596+0000 7f3c33400700 1 -- 172.21.15.12:0/3778682365 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3c34057aa0 con 0x7f3c3c0724c0 2024-03-29T09:54:37.597 INFO:teuthology.orchestra.run.smithi012.stdout:55834574855 2024-03-29T09:54:37.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.600+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c1806ad60 msgr2=0x7f3c1806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.600+0000 7f3c42800700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c1806ad60 0x7f3c1806d220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3c24000c00 tx=0x7f3c2401d040 comp rx=0 tx=0).stop 2024-03-29T09:54:37.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.600+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c0724c0 msgr2=0x7f3c3c14cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.600+0000 7f3c42800700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c0724c0 0x7f3c3c14cee0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009ed0 tx=0x7f3c340084e0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.600+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 shutdown_connections 2024-03-29T09:54:37.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.600+0000 7f3c42800700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3c1806ad60 0x7f3c1806d220 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.600+0000 7f3c42800700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c3c15d810 0x7f3c3c0ad7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.600+0000 7f3c42800700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c3c0724c0 0x7f3c3c14cee0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.600+0000 7f3c42800700 1 --2- 172.21.15.12:0/3778682365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c3c072910 0x7f3c3c0ad2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.600+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 >> 172.21.15.12:0/3778682365 conn(0x7f3c3c070f00 msgr2=0x7f3c3c0b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.601+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 shutdown_connections 2024-03-29T09:54:37.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.601+0000 7f3c42800700 1 -- 172.21.15.12:0/3778682365 wait complete. 2024-03-29T09:54:37.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2024-03-29T09:54:37.608 DEBUG:teuthology.parallel:result is None 2024-03-29T09:54:37.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.622+0000 7f50e6200700 1 -- 172.21.15.12:0/793904374 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f50b8005870 con 0x7f50e00b45e0 2024-03-29T09:54:37.622 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-03-29T09:54:37.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.623+0000 7f3dce200700 1 -- 172.21.15.12:0/1279355937 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3da0005870 con 0x7f3dc806ca60 2024-03-29T09:54:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.623+0000 7f50d7400700 1 -- 172.21.15.12:0/793904374 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f50c8023030 con 0x7f50e00b45e0 2024-03-29T09:54:37.623 DEBUG:teuthology.parallel:result is None 2024-03-29T09:54:37.623 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.623+0000 7f3dbf400700 1 -- 172.21.15.12:0/1279355937 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3db4017090 con 0x7f3dc806ca60 2024-03-29T09:54:37.624 INFO:teuthology.orchestra.run.smithi012.stdout:55834574855 2024-03-29T09:54:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.626+0000 7f50e6200700 1 -- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50bc06ac90 msgr2=0x7f50bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.626+0000 7f50e6200700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50bc06ac90 0x7f50bc06d150 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f50e00684a0 tx=0x7f50d8008040 comp rx=0 tx=0).stop 2024-03-29T09:54:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.626+0000 7f50e6200700 1 -- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50e00b45e0 msgr2=0x7f50e00b4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.626+0000 7f50e6200700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50e00b45e0 0x7f50e00b4a70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f50e008bc90 tx=0x7f50c8001930 comp rx=0 tx=0).stop 2024-03-29T09:54:37.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f50e6200700 1 -- 172.21.15.12:0/793904374 shutdown_connections 2024-03-29T09:54:37.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f50e6200700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50bc06ac90 0x7f50bc06d150 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f50e6200700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50e00b45e0 0x7f50e00b4a70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f50e6200700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50c40053c0 0x7f50e0160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f50e6200700 1 --2- 172.21.15.12:0/793904374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50e0156370 0x7f50e01f06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f50e6200700 1 -- 172.21.15.12:0/793904374 >> 172.21.15.12:0/793904374 conn(0x7f50e008aff0 msgr2=0x7f50e00b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f50e6200700 1 -- 172.21.15.12:0/793904374 shutdown_connections 2024-03-29T09:54:37.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f3dce200700 1 -- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3da806ace0 msgr2=0x7f3da806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f3dce200700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3da806ace0 0x7f3da806d1a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3db8006b20 tx=0x7f3db8006ab0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f3dce200700 1 -- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc806ca60 msgr2=0x7f3dc8149e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f3dce200700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc806ca60 0x7f3dc8149e80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3db400c700 tx=0x7f3db400cac0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f50e6200700 1 -- 172.21.15.12:0/793904374 wait complete. 2024-03-29T09:54:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f3dce200700 1 -- 172.21.15.12:0/1279355937 shutdown_connections 2024-03-29T09:54:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f3dce200700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3da806ace0 0x7f3da806d1a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f3dce200700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3dc80cfbc0 0x7f3dc81495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f3dce200700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3dc806ca60 0x7f3dc8149e80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.627+0000 7f3dce200700 1 --2- 172.21.15.12:0/1279355937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dc80ce2d0 0x7f3dc80ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.628+0000 7f3dce200700 1 -- 172.21.15.12:0/1279355937 >> 172.21.15.12:0/1279355937 conn(0x7f3dc8070f00 msgr2=0x7f3dc814d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.628+0000 7f3dce200700 1 -- 172.21.15.12:0/1279355937 shutdown_connections 2024-03-29T09:54:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.628+0000 7f3dce200700 1 -- 172.21.15.12:0/1279355937 wait complete. 2024-03-29T09:54:37.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.644+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3589634233 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f6df409a130 con 0x7f6df40ad3b0 2024-03-29T09:54:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.645+0000 7f6df0c00700 1 -- 172.21.15.12:0/3589634233 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6df409a130 con 0x7f6df40ad3b0 2024-03-29T09:54:37.645 INFO:teuthology.orchestra.run.smithi012.stdout:55834574855 2024-03-29T09:54:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.648+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6dd006ace0 msgr2=0x7f6dd006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.648+0000 7f6dfbc00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6dd006ace0 0x7f6dd006d1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009870 tx=0x7f6dec008040 comp rx=0 tx=0).stop 2024-03-29T09:54:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.649+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 msgr2=0x7f6df4098550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.649+0000 7f6dfbc00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 0x7f6df4098550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6de400c8c0 tx=0x7f6de400cc80 comp rx=0 tx=0).stop 2024-03-29T09:54:37.649 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-03-29T09:54:37.649 DEBUG:teuthology.parallel:result is None 2024-03-29T09:54:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.649+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3589634233 shutdown_connections 2024-03-29T09:54:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.649+0000 7f6dfbc00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6dd006ace0 0x7f6dd006d1a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.649+0000 7f6dfbc00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6df415d7a0 0x7f6df41f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.649+0000 7f6dfbc00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6df40724c0 0x7f6df41f8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.649+0000 7f6dfbc00700 1 --2- 172.21.15.12:0/3589634233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df40ad3b0 0x7f6df4098550 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.649+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3589634233 >> 172.21.15.12:0/3589634233 conn(0x7f6df4070f00 msgr2=0x7f6df41565b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.649+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3589634233 shutdown_connections 2024-03-29T09:54:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.649+0000 7f6dfbc00700 1 -- 172.21.15.12:0/3589634233 wait complete. 2024-03-29T09:54:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.655+0000 7f24fa400700 1 -- 172.21.15.12:0/2083521303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f24f4152b20 con 0x7f24f4152e20 2024-03-29T09:54:37.655 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.655+0000 7f24eb400700 1 -- 172.21.15.12:0/2083521303 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f24ec021030 con 0x7f24f4152e20 2024-03-29T09:54:37.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-03-29T09:54:37.659 DEBUG:teuthology.parallel:result is None 2024-03-29T09:54:37.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 -- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24d006ace0 msgr2=0x7f24d006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24d006ace0 0x7f24d006d1a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f24dc014470 tx=0x7f24dc008040 comp rx=0 tx=0).stop 2024-03-29T09:54:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 -- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24f4152e20 msgr2=0x7f24f41532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24f4152e20 0x7f24f41532a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f24f406f270 tx=0x7f24ec001ad0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 -- 172.21.15.12:0/2083521303 shutdown_connections 2024-03-29T09:54:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24d006ace0 0x7f24d006d1a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24f4152e20 0x7f24f41532a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24d8005210 0x7f24f4154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 --2- 172.21.15.12:0/2083521303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24f414cc50 0x7f24f414d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 -- 172.21.15.12:0/2083521303 >> 172.21.15.12:0/2083521303 conn(0x7f24f4070f00 msgr2=0x7f24f40b1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 -- 172.21.15.12:0/2083521303 shutdown_connections 2024-03-29T09:54:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.659+0000 7f24fa400700 1 -- 172.21.15.12:0/2083521303 wait complete. 2024-03-29T09:54:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.664+0000 7f2878e00700 1 -- 172.21.15.12:0/3089378654 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f287415e260 con 0x7f287415af30 2024-03-29T09:54:37.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.664+0000 7f286aa00700 1 -- 172.21.15.12:0/3089378654 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f286c016070 con 0x7f287415af30 2024-03-29T09:54:37.664 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2024-03-29T09:54:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 -- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f285006ac90 msgr2=0x7f285006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f285006ac90 0x7f285006d150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f285c0097f0 tx=0x7f285c008040 comp rx=0 tx=0).stop 2024-03-29T09:54:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 -- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f287415af30 msgr2=0x7f28740cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f287415af30 0x7f28740cdbf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f286c009b60 tx=0x7f286c004750 comp rx=0 tx=0).stop 2024-03-29T09:54:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 -- 172.21.15.12:0/3089378654 shutdown_connections 2024-03-29T09:54:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f285006ac90 0x7f285006d150 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f287415af30 0x7f28740cdbf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f287406da80 0x7f2874150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 --2- 172.21.15.12:0/3089378654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28740b4a50 0x7f287415ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 -- 172.21.15.12:0/3089378654 >> 172.21.15.12:0/3089378654 conn(0x7f2874071fb0 msgr2=0x7f287406d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 -- 172.21.15.12:0/3089378654 shutdown_connections 2024-03-29T09:54:37.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.667+0000 7f2878e00700 1 -- 172.21.15.12:0/3089378654 wait complete. 2024-03-29T09:54:37.669 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-03-29T09:54:37.669 DEBUG:teuthology.parallel:result is None 2024-03-29T09:54:37.679 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-03-29T09:54:37.679 DEBUG:teuthology.parallel:result is None 2024-03-29T09:54:37.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-03-29T09:54:37.686 DEBUG:teuthology.parallel:result is None 2024-03-29T09:54:37.686 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-29T09:54:37.686 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T09:54:37.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.924+0000 7fd342200700 1 Processor -- start 2024-03-29T09:54:37.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.924+0000 7fd342200700 1 -- start start 2024-03-29T09:54:37.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.926+0000 7fd342200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd33c15d670 0x7fd33c0ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.926+0000 7fd342200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd33c15da50 0x7fd33c0b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.926+0000 7fd342200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0b3890 con 0x7fd33c0b3000 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.926+0000 7fd342200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c0acac0 con 0x7fd33c15d670 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.926+0000 7fd342200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c159f50 con 0x7fd33c15da50 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.926+0000 7fd33b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd33c15d670 0x7fd33c0ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.926+0000 7fd33b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd33c15d670 0x7fd33c0ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40942/0 (socket says 172.21.15.12:40942) 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.926+0000 7fd33b400700 1 -- 172.21.15.12:0/581598158 learned_addr learned my addr 172.21.15.12:0/581598158 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.926+0000 7fd33aa00700 1 --2- 172.21.15.12:0/581598158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd33c15da50 0x7fd33c0b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.927+0000 7fd33b400700 1 -- 172.21.15.12:0/581598158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd33c15da50 msgr2=0x7fd33c0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.927+0000 7fd33b400700 1 --2- 172.21.15.12:0/581598158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd33c15da50 0x7fd33c0b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.927+0000 7fd33b400700 1 -- 172.21.15.12:0/581598158 >> v1:172.21.15.159:6789/0 conn(0x7fd33c0b3000 legacy=0x7fd33c06dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.927+0000 7fd33b400700 1 -- 172.21.15.12:0/581598158 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0b35b0 con 0x7fd33c15d670 2024-03-29T09:54:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.927+0000 7fd33b400700 1 --2- 172.21.15.12:0/581598158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15d670 0x7fd33c0ac580 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0098f0 tx=0x7fd32c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9340c7ec370a3274 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.927+0000 7fd33a000700 1 -- 172.21.15.12:0/581598158 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c011830 con 0x7fd33c15d670 2024-03-29T09:54:37.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.927+0000 7fd33a000700 1 -- 172.21.15.12:0/581598158 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c011990 con 0x7fd33c15d670 2024-03-29T09:54:37.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.927+0000 7fd33a000700 1 -- 172.21.15.12:0/581598158 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c011d20 con 0x7fd33c15d670 2024-03-29T09:54:37.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.928+0000 7fd342200700 1 -- 172.21.15.12:0/581598158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15d670 msgr2=0x7fd33c0ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.928+0000 7fd342200700 1 --2- 172.21.15.12:0/581598158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15d670 0x7fd33c0ac580 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0098f0 tx=0x7fd32c009c00 comp rx=0 tx=0).stop 2024-03-29T09:54:37.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.928+0000 7fd342200700 1 -- 172.21.15.12:0/581598158 shutdown_connections 2024-03-29T09:54:37.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.928+0000 7fd342200700 1 --2- 172.21.15.12:0/581598158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15d670 0x7fd33c0ac580 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.928+0000 7fd342200700 1 --2- 172.21.15.12:0/581598158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd33c15da50 0x7fd33c0b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.928+0000 7fd342200700 1 -- 172.21.15.12:0/581598158 >> 172.21.15.12:0/581598158 conn(0x7fd33c0cf190 msgr2=0x7fd33c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:37.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.928+0000 7fd342200700 1 -- 172.21.15.12:0/581598158 shutdown_connections 2024-03-29T09:54:37.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.928+0000 7fd342200700 1 -- 172.21.15.12:0/581598158 wait complete. 2024-03-29T09:54:37.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.930+0000 7fd342200700 1 Processor -- start 2024-03-29T09:54:37.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.930+0000 7fd342200700 1 -- start start 2024-03-29T09:54:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd342200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd33c0b3000 0x7fd33c141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd342200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd33c15d670 0x7fd33c1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd342200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15da50 0x7fd33c152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd342200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c142940 con 0x7fd33c15da50 2024-03-29T09:54:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd342200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c142130 con 0x7fd33c15d670 2024-03-29T09:54:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd342200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c098ea0 con 0x7fd33c0b3000 2024-03-29T09:54:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd33be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15da50 0x7fd33c152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd33aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd33c15d670 0x7fd33c1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd33be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15da50 0x7fd33c152cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40948/0 (socket says 172.21.15.12:40948) 2024-03-29T09:54:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd33be00700 1 -- 172.21.15.12:0/4263084327 learned_addr learned my addr 172.21.15.12:0/4263084327 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.931+0000 7fd33b400700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd33c0b3000 0x7fd33c141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.932+0000 7fd33be00700 1 -- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd33c0b3000 msgr2=0x7fd33c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.932+0000 7fd33be00700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd33c0b3000 0x7fd33c141aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.932+0000 7fd33be00700 1 -- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd33c15d670 msgr2=0x7fd33c1527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.932+0000 7fd33be00700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd33c15d670 0x7fd33c1527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.932+0000 7fd33be00700 1 -- 172.21.15.12:0/4263084327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c009550 con 0x7fd33c15da50 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.932+0000 7fd33be00700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15da50 0x7fd33c152cf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd33400cc50 tx=0x7fd33400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.932+0000 7fd333400700 1 -- 172.21.15.12:0/4263084327 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd334003c00 con 0x7fd33c15da50 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.932+0000 7fd333400700 1 -- 172.21.15.12:0/4263084327 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd334003d60 con 0x7fd33c15da50 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.932+0000 7fd342200700 1 -- 172.21.15.12:0/4263084327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c142380 con 0x7fd33c15da50 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.932+0000 7fd333400700 1 -- 172.21.15.12:0/4263084327 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33400aab0 con 0x7fd33c15da50 2024-03-29T09:54:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.933+0000 7fd342200700 1 -- 172.21.15.12:0/4263084327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c096820 con 0x7fd33c15da50 2024-03-29T09:54:37.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.934+0000 7fd342200700 1 -- 172.21.15.12:0/4263084327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c096680 con 0x7fd33c15da50 2024-03-29T09:54:37.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.935+0000 7fd333400700 1 -- 172.21.15.12:0/4263084327 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd33400ac10 con 0x7fd33c15da50 2024-03-29T09:54:37.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.935+0000 7fd333400700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd31c06ac90 0x7fd31c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:37.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.936+0000 7fd333400700 1 -- 172.21.15.12:0/4263084327 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd334089310 con 0x7fd33c15da50 2024-03-29T09:54:37.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.936+0000 7fd33b400700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd31c06ac90 0x7fd31c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.937+0000 7fd33b400700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd31c06ac90 0x7fd31c06d150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd32c000c00 tx=0x7fd32c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:37.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:37.940+0000 7fd333400700 1 -- 172.21.15.12:0/4263084327 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd33c096680 con 0x7fd33c15da50 2024-03-29T09:54:38.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.148+0000 7fd342200700 1 -- 172.21.15.12:0/4263084327 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd33c06d9c0 con 0x7fd31c06ac90 2024-03-29T09:54:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.157+0000 7fd333400700 1 -- 172.21.15.12:0/4263084327 <== mgr.4114 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264558 (secure 0 0 0) 0x7fd32c011b80 con 0x7fd31c06ac90 2024-03-29T09:54:38.161 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.161+0000 7fd342200700 1 -- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd31c06ac90 msgr2=0x7fd31c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.161+0000 7fd342200700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd31c06ac90 0x7fd31c06d150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd32c000c00 tx=0x7fd32c008030 comp rx=0 tx=0).stop 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.161+0000 7fd342200700 1 -- 172.21.15.12:0/4263084327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15da50 msgr2=0x7fd33c152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.161+0000 7fd342200700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15da50 0x7fd33c152cf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd33400cc50 tx=0x7fd33400cf60 comp rx=0 tx=0).stop 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.161+0000 7fd342200700 1 -- 172.21.15.12:0/4263084327 shutdown_connections 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.161+0000 7fd342200700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd31c06ac90 0x7fd31c06d150 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.161+0000 7fd342200700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd33c15da50 0x7fd33c152cf0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.161+0000 7fd342200700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd33c0b3000 0x7fd33c141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.161+0000 7fd342200700 1 --2- 172.21.15.12:0/4263084327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd33c15d670 0x7fd33c1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.161+0000 7fd342200700 1 -- 172.21.15.12:0/4263084327 >> 172.21.15.12:0/4263084327 conn(0x7fd33c0cf190 msgr2=0x7fd33c15c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.162+0000 7fd342200700 1 -- 172.21.15.12:0/4263084327 shutdown_connections 2024-03-29T09:54:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.162+0000 7fd342200700 1 -- 172.21.15.12:0/4263084327 wait complete. 2024-03-29T09:54:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T09:54:38.179 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-29T09:54:36.438873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219064,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755656,"statfs":{"total":773094113280,"available":772869791744,"internally_reserved":0,"allocated":2887680,"data_stored":1751625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.009251"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674620+0000","last_change":"2024-03-29T09:54:27.472798+0000","last_active":"2024-03-29T09:54:28.674620+0000","last_peered":"2024-03-29T09:54:28.674620+0000","last_clean":"2024-03-29T09:54:28.674620+0000","last_became_active":"2024-03-29T09:54:27.468781+0000","last_became_peered":"2024-03-29T09:54:27.468781+0000","last_unstale":"2024-03-29T09:54:28.674620+0000","last_undegraded":"2024-03-29T09:54:28.674620+0000","last_fullsized":"2024-03-29T09:54:28.674620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:10.721926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572841+0000","last_change":"2024-03-29T09:54:27.456722+0000","last_active":"2024-03-29T09:54:28.572841+0000","last_peered":"2024-03-29T09:54:28.572841+0000","last_clean":"2024-03-29T09:54:28.572841+0000","last_became_active":"2024-03-29T09:54:27.456334+0000","last_became_peered":"2024-03-29T09:54:27.456334+0000","last_unstale":"2024-03-29T09:54:28.572841+0000","last_undegraded":"2024-03-29T09:54:28.572841+0000","last_fullsized":"2024-03-29T09:54:28.572841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:52.185097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674299+0000","last_change":"2024-03-29T09:54:27.455943+0000","last_active":"2024-03-29T09:54:28.674299+0000","last_peered":"2024-03-29T09:54:28.674299+0000","last_clean":"2024-03-29T09:54:28.674299+0000","last_became_active":"2024-03-29T09:54:27.455551+0000","last_became_peered":"2024-03-29T09:54:27.455551+0000","last_unstale":"2024-03-29T09:54:28.674299+0000","last_undegraded":"2024-03-29T09:54:28.674299+0000","last_fullsized":"2024-03-29T09:54:28.674299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:03.191627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438591+0000","last_change":"2024-03-29T09:54:27.445755+0000","last_active":"2024-03-29T09:54:28.438591+0000","last_peered":"2024-03-29T09:54:28.438591+0000","last_clean":"2024-03-29T09:54:28.438591+0000","last_became_active":"2024-03-29T09:54:27.444623+0000","last_became_peered":"2024-03-29T09:54:27.444623+0000","last_unstale":"2024-03-29T09:54:28.438591+0000","last_undegraded":"2024-03-29T09:54:28.438591+0000","last_fullsized":"2024-03-29T09:54:28.438591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:19.141206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674074+0000","last_change":"2024-03-29T09:54:27.468817+0000","last_active":"2024-03-29T09:54:28.674074+0000","last_peered":"2024-03-29T09:54:28.674074+0000","last_clean":"2024-03-29T09:54:28.674074+0000","last_became_active":"2024-03-29T09:54:27.467049+0000","last_became_peered":"2024-03-29T09:54:27.467049+0000","last_unstale":"2024-03-29T09:54:28.674074+0000","last_undegraded":"2024-03-29T09:54:28.674074+0000","last_fullsized":"2024-03-29T09:54:28.674074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:44:20.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437373+0000","last_change":"2024-03-29T09:54:27.453526+0000","last_active":"2024-03-29T09:54:28.437373+0000","last_peered":"2024-03-29T09:54:28.437373+0000","last_clean":"2024-03-29T09:54:28.437373+0000","last_became_active":"2024-03-29T09:54:27.452090+0000","last_became_peered":"2024-03-29T09:54:27.452090+0000","last_unstale":"2024-03-29T09:54:28.437373+0000","last_undegraded":"2024-03-29T09:54:28.437373+0000","last_fullsized":"2024-03-29T09:54:28.437373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:30.050849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673999+0000","last_change":"2024-03-29T09:54:27.471016+0000","last_active":"2024-03-29T09:54:28.673999+0000","last_peered":"2024-03-29T09:54:28.673999+0000","last_clean":"2024-03-29T09:54:28.673999+0000","last_became_active":"2024-03-29T09:54:27.470657+0000","last_became_peered":"2024-03-29T09:54:27.470657+0000","last_unstale":"2024-03-29T09:54:28.673999+0000","last_undegraded":"2024-03-29T09:54:28.673999+0000","last_fullsized":"2024-03-29T09:54:28.673999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:56.949701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673522+0000","last_change":"2024-03-29T09:54:27.458451+0000","last_active":"2024-03-29T09:54:28.673522+0000","last_peered":"2024-03-29T09:54:28.673522+0000","last_clean":"2024-03-29T09:54:28.673522+0000","last_became_active":"2024-03-29T09:54:27.458110+0000","last_became_peered":"2024-03-29T09:54:27.458110+0000","last_unstale":"2024-03-29T09:54:28.673522+0000","last_undegraded":"2024-03-29T09:54:28.673522+0000","last_fullsized":"2024-03-29T09:54:28.673522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:57.886383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674535+0000","last_change":"2024-03-29T09:54:27.471858+0000","last_active":"2024-03-29T09:54:28.674535+0000","last_peered":"2024-03-29T09:54:28.674535+0000","last_clean":"2024-03-29T09:54:28.674535+0000","last_became_active":"2024-03-29T09:54:27.469164+0000","last_became_peered":"2024-03-29T09:54:27.469164+0000","last_unstale":"2024-03-29T09:54:28.674535+0000","last_undegraded":"2024-03-29T09:54:28.674535+0000","last_fullsized":"2024-03-29T09:54:28.674535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:44:42.873042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.439014+0000","last_change":"2024-03-29T09:54:27.442987+0000","last_active":"2024-03-29T09:54:28.439014+0000","last_peered":"2024-03-29T09:54:28.439014+0000","last_clean":"2024-03-29T09:54:28.439014+0000","last_became_active":"2024-03-29T09:54:27.442294+0000","last_became_peered":"2024-03-29T09:54:27.442294+0000","last_unstale":"2024-03-29T09:54:28.439014+0000","last_undegraded":"2024-03-29T09:54:28.439014+0000","last_fullsized":"2024-03-29T09:54:28.439014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438771+0000","last_change":"2024-03-29T09:54:27.442621+0000","last_active":"2024-03-29T09:54:28.438771+0000","last_peered":"2024-03-29T09:54:28.438771+0000","last_clean":"2024-03-29T09:54:28.438771+0000","last_became_active":"2024-03-29T09:54:27.442259+0000","last_became_peered":"2024-03-29T09:54:27.442259+0000","last_unstale":"2024-03-29T09:54:28.438771+0000","last_undegraded":"2024-03-29T09:54:28.438771+0000","last_fullsized":"2024-03-29T09:54:28.438771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436677+0000","last_change":"2024-03-29T09:54:27.453502+0000","last_active":"2024-03-29T09:54:28.436677+0000","last_peered":"2024-03-29T09:54:28.436677+0000","last_clean":"2024-03-29T09:54:28.436677+0000","last_became_active":"2024-03-29T09:54:27.453015+0000","last_became_peered":"2024-03-29T09:54:27.453015+0000","last_unstale":"2024-03-29T09:54:28.436677+0000","last_undegraded":"2024-03-29T09:54:28.436677+0000","last_fullsized":"2024-03-29T09:54:28.436677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:54.979913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674480+0000","last_change":"2024-03-29T09:54:27.451715+0000","last_active":"2024-03-29T09:54:28.674480+0000","last_peered":"2024-03-29T09:54:28.674480+0000","last_clean":"2024-03-29T09:54:28.674480+0000","last_became_active":"2024-03-29T09:54:27.451432+0000","last_became_peered":"2024-03-29T09:54:27.451432+0000","last_unstale":"2024-03-29T09:54:28.674480+0000","last_undegraded":"2024-03-29T09:54:28.674480+0000","last_fullsized":"2024-03-29T09:54:28.674480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674068+0000","last_change":"2024-03-29T09:54:27.460653+0000","last_active":"2024-03-29T09:54:28.674068+0000","last_peered":"2024-03-29T09:54:28.674068+0000","last_clean":"2024-03-29T09:54:28.674068+0000","last_became_active":"2024-03-29T09:54:27.459198+0000","last_became_peered":"2024-03-29T09:54:27.459198+0000","last_unstale":"2024-03-29T09:54:28.674068+0000","last_undegraded":"2024-03-29T09:54:28.674068+0000","last_fullsized":"2024-03-29T09:54:28.674068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:27:59.103915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572329+0000","last_change":"2024-03-29T09:54:27.457510+0000","last_active":"2024-03-29T09:54:28.572329+0000","last_peered":"2024-03-29T09:54:28.572329+0000","last_clean":"2024-03-29T09:54:28.572329+0000","last_became_active":"2024-03-29T09:54:27.456319+0000","last_became_peered":"2024-03-29T09:54:27.456319+0000","last_unstale":"2024-03-29T09:54:28.572329+0000","last_undegraded":"2024-03-29T09:54:28.572329+0000","last_fullsized":"2024-03-29T09:54:28.572329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437168+0000","last_change":"2024-03-29T09:54:27.460545+0000","last_active":"2024-03-29T09:54:28.437168+0000","last_peered":"2024-03-29T09:54:28.437168+0000","last_clean":"2024-03-29T09:54:28.437168+0000","last_became_active":"2024-03-29T09:54:27.460152+0000","last_became_peered":"2024-03-29T09:54:27.460152+0000","last_unstale":"2024-03-29T09:54:28.437168+0000","last_undegraded":"2024-03-29T09:54:28.437168+0000","last_fullsized":"2024-03-29T09:54:28.437168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:52.185097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438011+0000","last_change":"2024-03-29T09:54:27.469947+0000","last_active":"2024-03-29T09:54:28.438011+0000","last_peered":"2024-03-29T09:54:28.438011+0000","last_clean":"2024-03-29T09:54:28.438011+0000","last_became_active":"2024-03-29T09:54:27.469649+0000","last_became_peered":"2024-03-29T09:54:27.469649+0000","last_unstale":"2024-03-29T09:54:28.438011+0000","last_undegraded":"2024-03-29T09:54:28.438011+0000","last_fullsized":"2024-03-29T09:54:28.438011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:44:42.873042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.511192+0000","last_change":"2024-03-29T09:54:27.438206+0000","last_active":"2024-03-29T09:54:28.511192+0000","last_peered":"2024-03-29T09:54:28.511192+0000","last_clean":"2024-03-29T09:54:28.511192+0000","last_became_active":"2024-03-29T09:54:27.437957+0000","last_became_peered":"2024-03-29T09:54:27.437957+0000","last_unstale":"2024-03-29T09:54:28.511192+0000","last_undegraded":"2024-03-29T09:54:28.511192+0000","last_fullsized":"2024-03-29T09:54:28.511192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:03.191627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674238+0000","last_change":"2024-03-29T09:54:27.466200+0000","last_active":"2024-03-29T09:54:28.674238+0000","last_peered":"2024-03-29T09:54:28.674238+0000","last_clean":"2024-03-29T09:54:28.674238+0000","last_became_active":"2024-03-29T09:54:27.465886+0000","last_became_peered":"2024-03-29T09:54:27.465886+0000","last_unstale":"2024-03-29T09:54:28.674238+0000","last_undegraded":"2024-03-29T09:54:28.674238+0000","last_fullsized":"2024-03-29T09:54:28.674238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:57.886383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.459838+0000","last_change":"2024-03-29T09:54:26.929518+0000","last_active":"2024-03-29T09:54:27.459838+0000","last_peered":"2024-03-29T09:54:27.459838+0000","last_clean":"2024-03-29T09:54:27.459838+0000","last_became_active":"2024-03-29T09:54:26.929253+0000","last_became_peered":"2024-03-29T09:54:26.929253+0000","last_unstale":"2024-03-29T09:54:27.459838+0000","last_undegraded":"2024-03-29T09:54:27.459838+0000","last_fullsized":"2024-03-29T09:54:27.459838+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:40:59.105279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674699+0000","last_change":"2024-03-29T09:54:27.454266+0000","last_active":"2024-03-29T09:54:28.674699+0000","last_peered":"2024-03-29T09:54:28.674699+0000","last_clean":"2024-03-29T09:54:28.674699+0000","last_became_active":"2024-03-29T09:54:27.454005+0000","last_became_peered":"2024-03-29T09:54:27.454005+0000","last_unstale":"2024-03-29T09:54:28.674699+0000","last_undegraded":"2024-03-29T09:54:28.674699+0000","last_fullsized":"2024-03-29T09:54:28.674699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:02.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.467235+0000","last_change":"2024-03-29T09:54:26.926684+0000","last_active":"2024-03-29T09:54:27.467235+0000","last_peered":"2024-03-29T09:54:27.467235+0000","last_clean":"2024-03-29T09:54:27.467235+0000","last_became_active":"2024-03-29T09:54:26.926350+0000","last_became_peered":"2024-03-29T09:54:26.926350+0000","last_unstale":"2024-03-29T09:54:27.467235+0000","last_undegraded":"2024-03-29T09:54:27.467235+0000","last_fullsized":"2024-03-29T09:54:27.467235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:37.644975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673888+0000","last_change":"2024-03-29T09:54:27.452090+0000","last_active":"2024-03-29T09:54:28.673888+0000","last_peered":"2024-03-29T09:54:28.673888+0000","last_clean":"2024-03-29T09:54:28.673888+0000","last_became_active":"2024-03-29T09:54:27.451361+0000","last_became_peered":"2024-03-29T09:54:27.451361+0000","last_unstale":"2024-03-29T09:54:28.673888+0000","last_undegraded":"2024-03-29T09:54:28.673888+0000","last_fullsized":"2024-03-29T09:54:28.673888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:25:15.310717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.468152+0000","last_change":"2024-03-29T09:54:26.924715+0000","last_active":"2024-03-29T09:54:27.468152+0000","last_peered":"2024-03-29T09:54:27.468152+0000","last_clean":"2024-03-29T09:54:27.468152+0000","last_became_active":"2024-03-29T09:54:26.924430+0000","last_became_peered":"2024-03-29T09:54:26.924430+0000","last_unstale":"2024-03-29T09:54:27.468152+0000","last_undegraded":"2024-03-29T09:54:27.468152+0000","last_fullsized":"2024-03-29T09:54:27.468152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438051+0000","last_change":"2024-03-29T09:54:27.455410+0000","last_active":"2024-03-29T09:54:28.438051+0000","last_peered":"2024-03-29T09:54:28.438051+0000","last_clean":"2024-03-29T09:54:28.438051+0000","last_became_active":"2024-03-29T09:54:27.453992+0000","last_became_peered":"2024-03-29T09:54:27.453992+0000","last_unstale":"2024-03-29T09:54:28.438051+0000","last_undegraded":"2024-03-29T09:54:28.438051+0000","last_fullsized":"2024-03-29T09:54:28.438051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:57.886383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.469487+0000","last_change":"2024-03-29T09:54:26.931418+0000","last_active":"2024-03-29T09:54:27.469487+0000","last_peered":"2024-03-29T09:54:27.469487+0000","last_clean":"2024-03-29T09:54:27.469487+0000","last_became_active":"2024-03-29T09:54:26.931148+0000","last_became_peered":"2024-03-29T09:54:26.931148+0000","last_unstale":"2024-03-29T09:54:27.469487+0000","last_undegraded":"2024-03-29T09:54:27.469487+0000","last_fullsized":"2024-03-29T09:54:27.469487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:02.185001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437930+0000","last_change":"2024-03-29T09:54:27.454343+0000","last_active":"2024-03-29T09:54:28.437930+0000","last_peered":"2024-03-29T09:54:28.437930+0000","last_clean":"2024-03-29T09:54:28.437930+0000","last_became_active":"2024-03-29T09:54:27.454012+0000","last_became_peered":"2024-03-29T09:54:27.454012+0000","last_unstale":"2024-03-29T09:54:28.437930+0000","last_undegraded":"2024-03-29T09:54:28.437930+0000","last_fullsized":"2024-03-29T09:54:28.437930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:50.524833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.467317+0000","last_change":"2024-03-29T09:54:26.929141+0000","last_active":"2024-03-29T09:54:27.467317+0000","last_peered":"2024-03-29T09:54:27.467317+0000","last_clean":"2024-03-29T09:54:27.467317+0000","last_became_active":"2024-03-29T09:54:26.928876+0000","last_became_peered":"2024-03-29T09:54:26.928876+0000","last_unstale":"2024-03-29T09:54:27.467317+0000","last_undegraded":"2024-03-29T09:54:27.467317+0000","last_fullsized":"2024-03-29T09:54:27.467317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:37:17.462538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436160+0000","last_change":"2024-03-29T09:54:27.448632+0000","last_active":"2024-03-29T09:54:28.436160+0000","last_peered":"2024-03-29T09:54:28.436160+0000","last_clean":"2024-03-29T09:54:28.436160+0000","last_became_active":"2024-03-29T09:54:27.448165+0000","last_became_peered":"2024-03-29T09:54:27.448165+0000","last_unstale":"2024-03-29T09:54:28.436160+0000","last_undegraded":"2024-03-29T09:54:28.436160+0000","last_fullsized":"2024-03-29T09:54:28.436160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:58.125922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.455407+0000","last_change":"2024-03-29T09:54:26.916444+0000","last_active":"2024-03-29T09:54:27.455407+0000","last_peered":"2024-03-29T09:54:27.455407+0000","last_clean":"2024-03-29T09:54:27.455407+0000","last_became_active":"2024-03-29T09:54:26.916132+0000","last_became_peered":"2024-03-29T09:54:26.916132+0000","last_unstale":"2024-03-29T09:54:27.455407+0000","last_undegraded":"2024-03-29T09:54:27.455407+0000","last_fullsized":"2024-03-29T09:54:27.455407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674190+0000","last_change":"2024-03-29T09:54:27.464314+0000","last_active":"2024-03-29T09:54:28.674190+0000","last_peered":"2024-03-29T09:54:28.674190+0000","last_clean":"2024-03-29T09:54:28.674190+0000","last_became_active":"2024-03-29T09:54:27.463702+0000","last_became_peered":"2024-03-29T09:54:27.463702+0000","last_unstale":"2024-03-29T09:54:28.674190+0000","last_undegraded":"2024-03-29T09:54:28.674190+0000","last_fullsized":"2024-03-29T09:54:28.674190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.444809+0000","last_change":"2024-03-29T09:54:26.921122+0000","last_active":"2024-03-29T09:54:27.444809+0000","last_peered":"2024-03-29T09:54:27.444809+0000","last_clean":"2024-03-29T09:54:27.444809+0000","last_became_active":"2024-03-29T09:54:26.919924+0000","last_became_peered":"2024-03-29T09:54:26.919924+0000","last_unstale":"2024-03-29T09:54:27.444809+0000","last_undegraded":"2024-03-29T09:54:27.444809+0000","last_fullsized":"2024-03-29T09:54:27.444809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673522+0000","last_change":"2024-03-29T09:54:27.465683+0000","last_active":"2024-03-29T09:54:28.673522+0000","last_peered":"2024-03-29T09:54:28.673522+0000","last_clean":"2024-03-29T09:54:28.673522+0000","last_became_active":"2024-03-29T09:54:27.465379+0000","last_became_peered":"2024-03-29T09:54:27.465379+0000","last_unstale":"2024-03-29T09:54:28.673522+0000","last_undegraded":"2024-03-29T09:54:28.673522+0000","last_fullsized":"2024-03-29T09:54:28.673522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:52.185097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.434322+0000","last_change":"2024-03-29T09:54:26.918838+0000","last_active":"2024-03-29T09:54:27.434322+0000","last_peered":"2024-03-29T09:54:27.434322+0000","last_clean":"2024-03-29T09:54:27.434322+0000","last_became_active":"2024-03-29T09:54:26.918185+0000","last_became_peered":"2024-03-29T09:54:26.918185+0000","last_unstale":"2024-03-29T09:54:27.434322+0000","last_undegraded":"2024-03-29T09:54:27.434322+0000","last_fullsized":"2024-03-29T09:54:27.434322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572430+0000","last_change":"2024-03-29T09:54:27.449379+0000","last_active":"2024-03-29T09:54:28.572430+0000","last_peered":"2024-03-29T09:54:28.572430+0000","last_clean":"2024-03-29T09:54:28.572430+0000","last_became_active":"2024-03-29T09:54:27.448935+0000","last_became_peered":"2024-03-29T09:54:27.448935+0000","last_unstale":"2024-03-29T09:54:28.572430+0000","last_undegraded":"2024-03-29T09:54:28.572430+0000","last_fullsized":"2024-03-29T09:54:28.572430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.434616+0000","last_change":"2024-03-29T09:54:26.920134+0000","last_active":"2024-03-29T09:54:27.434616+0000","last_peered":"2024-03-29T09:54:27.434616+0000","last_clean":"2024-03-29T09:54:27.434616+0000","last_became_active":"2024-03-29T09:54:26.919158+0000","last_became_peered":"2024-03-29T09:54:26.919158+0000","last_unstale":"2024-03-29T09:54:27.434616+0000","last_undegraded":"2024-03-29T09:54:27.434616+0000","last_fullsized":"2024-03-29T09:54:27.434616+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438287+0000","last_change":"2024-03-29T09:54:27.467877+0000","last_active":"2024-03-29T09:54:28.438287+0000","last_peered":"2024-03-29T09:54:28.438287+0000","last_clean":"2024-03-29T09:54:28.438287+0000","last_became_active":"2024-03-29T09:54:27.467347+0000","last_became_peered":"2024-03-29T09:54:27.467347+0000","last_unstale":"2024-03-29T09:54:28.438287+0000","last_undegraded":"2024-03-29T09:54:28.438287+0000","last_fullsized":"2024-03-29T09:54:28.438287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.437566+0000","last_change":"2024-03-29T09:54:26.911930+0000","last_active":"2024-03-29T09:54:27.437566+0000","last_peered":"2024-03-29T09:54:27.437566+0000","last_clean":"2024-03-29T09:54:27.437566+0000","last_became_active":"2024-03-29T09:54:26.911549+0000","last_became_peered":"2024-03-29T09:54:26.911549+0000","last_unstale":"2024-03-29T09:54:27.437566+0000","last_undegraded":"2024-03-29T09:54:27.437566+0000","last_fullsized":"2024-03-29T09:54:27.437566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674175+0000","last_change":"2024-03-29T09:54:27.451917+0000","last_active":"2024-03-29T09:54:28.674175+0000","last_peered":"2024-03-29T09:54:28.674175+0000","last_clean":"2024-03-29T09:54:28.674175+0000","last_became_active":"2024-03-29T09:54:27.450981+0000","last_became_peered":"2024-03-29T09:54:27.450981+0000","last_unstale":"2024-03-29T09:54:28.674175+0000","last_undegraded":"2024-03-29T09:54:28.674175+0000","last_fullsized":"2024-03-29T09:54:28.674175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:41:21.776144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.470049+0000","last_change":"2024-03-29T09:54:26.917969+0000","last_active":"2024-03-29T09:54:27.470049+0000","last_peered":"2024-03-29T09:54:27.470049+0000","last_clean":"2024-03-29T09:54:27.470049+0000","last_became_active":"2024-03-29T09:54:26.916331+0000","last_became_peered":"2024-03-29T09:54:26.916331+0000","last_unstale":"2024-03-29T09:54:27.470049+0000","last_undegraded":"2024-03-29T09:54:27.470049+0000","last_fullsized":"2024-03-29T09:54:27.470049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:02.185001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436752+0000","last_change":"2024-03-29T09:54:27.465694+0000","last_active":"2024-03-29T09:54:28.436752+0000","last_peered":"2024-03-29T09:54:28.436752+0000","last_clean":"2024-03-29T09:54:28.436752+0000","last_became_active":"2024-03-29T09:54:27.465420+0000","last_became_peered":"2024-03-29T09:54:27.465420+0000","last_unstale":"2024-03-29T09:54:28.436752+0000","last_undegraded":"2024-03-29T09:54:28.436752+0000","last_fullsized":"2024-03-29T09:54:28.436752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:39.673266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.474851+0000","last_change":"2024-03-29T09:54:26.925541+0000","last_active":"2024-03-29T09:54:27.474851+0000","last_peered":"2024-03-29T09:54:27.474851+0000","last_clean":"2024-03-29T09:54:27.474851+0000","last_became_active":"2024-03-29T09:54:26.925012+0000","last_became_peered":"2024-03-29T09:54:26.925012+0000","last_unstale":"2024-03-29T09:54:27.474851+0000","last_undegraded":"2024-03-29T09:54:27.474851+0000","last_fullsized":"2024-03-29T09:54:27.474851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674348+0000","last_change":"2024-03-29T09:54:27.470254+0000","last_active":"2024-03-29T09:54:28.674348+0000","last_peered":"2024-03-29T09:54:28.674348+0000","last_clean":"2024-03-29T09:54:28.674348+0000","last_became_active":"2024-03-29T09:54:27.468546+0000","last_became_peered":"2024-03-29T09:54:27.468546+0000","last_unstale":"2024-03-29T09:54:28.674348+0000","last_undegraded":"2024-03-29T09:54:28.674348+0000","last_fullsized":"2024-03-29T09:54:28.674348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:30.050849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.464632+0000","last_change":"2024-03-29T09:54:26.924638+0000","last_active":"2024-03-29T09:54:27.464632+0000","last_peered":"2024-03-29T09:54:27.464632+0000","last_clean":"2024-03-29T09:54:27.464632+0000","last_became_active":"2024-03-29T09:54:26.923692+0000","last_became_peered":"2024-03-29T09:54:26.923692+0000","last_unstale":"2024-03-29T09:54:27.464632+0000","last_undegraded":"2024-03-29T09:54:27.464632+0000","last_fullsized":"2024-03-29T09:54:27.464632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:49:24.339314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673965+0000","last_change":"2024-03-29T09:54:27.465512+0000","last_active":"2024-03-29T09:54:28.673965+0000","last_peered":"2024-03-29T09:54:28.673965+0000","last_clean":"2024-03-29T09:54:28.673965+0000","last_became_active":"2024-03-29T09:54:27.465258+0000","last_became_peered":"2024-03-29T09:54:27.465258+0000","last_unstale":"2024-03-29T09:54:28.673965+0000","last_undegraded":"2024-03-29T09:54:28.673965+0000","last_fullsized":"2024-03-29T09:54:28.673965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.437301+0000","last_change":"2024-03-29T09:54:26.922511+0000","last_active":"2024-03-29T09:54:27.437301+0000","last_peered":"2024-03-29T09:54:27.437301+0000","last_clean":"2024-03-29T09:54:27.437301+0000","last_became_active":"2024-03-29T09:54:26.922182+0000","last_became_peered":"2024-03-29T09:54:26.922182+0000","last_unstale":"2024-03-29T09:54:27.437301+0000","last_undegraded":"2024-03-29T09:54:27.437301+0000","last_fullsized":"2024-03-29T09:54:27.437301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674259+0000","last_change":"2024-03-29T09:54:27.472857+0000","last_active":"2024-03-29T09:54:28.674259+0000","last_peered":"2024-03-29T09:54:28.674259+0000","last_clean":"2024-03-29T09:54:28.674259+0000","last_became_active":"2024-03-29T09:54:27.472482+0000","last_became_peered":"2024-03-29T09:54:27.472482+0000","last_unstale":"2024-03-29T09:54:28.674259+0000","last_undegraded":"2024-03-29T09:54:28.674259+0000","last_fullsized":"2024-03-29T09:54:28.674259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:18.345522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.432778+0000","last_change":"2024-03-29T09:54:26.920268+0000","last_active":"2024-03-29T09:54:27.432778+0000","last_peered":"2024-03-29T09:54:27.432778+0000","last_clean":"2024-03-29T09:54:27.432778+0000","last_became_active":"2024-03-29T09:54:26.919956+0000","last_became_peered":"2024-03-29T09:54:26.919956+0000","last_unstale":"2024-03-29T09:54:27.432778+0000","last_undegraded":"2024-03-29T09:54:27.432778+0000","last_fullsized":"2024-03-29T09:54:27.432778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438219+0000","last_change":"2024-03-29T09:54:27.457604+0000","last_active":"2024-03-29T09:54:28.438219+0000","last_peered":"2024-03-29T09:54:28.438219+0000","last_clean":"2024-03-29T09:54:28.438219+0000","last_became_active":"2024-03-29T09:54:27.457197+0000","last_became_peered":"2024-03-29T09:54:27.457197+0000","last_unstale":"2024-03-29T09:54:28.438219+0000","last_undegraded":"2024-03-29T09:54:28.438219+0000","last_fullsized":"2024-03-29T09:54:28.438219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:25:15.310717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.464670+0000","last_change":"2024-03-29T09:54:26.921804+0000","last_active":"2024-03-29T09:54:27.464670+0000","last_peered":"2024-03-29T09:54:27.464670+0000","last_clean":"2024-03-29T09:54:27.464670+0000","last_became_active":"2024-03-29T09:54:26.921487+0000","last_became_peered":"2024-03-29T09:54:26.921487+0000","last_unstale":"2024-03-29T09:54:27.464670+0000","last_undegraded":"2024-03-29T09:54:27.464670+0000","last_fullsized":"2024-03-29T09:54:27.464670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674124+0000","last_change":"2024-03-29T09:54:27.470978+0000","last_active":"2024-03-29T09:54:28.674124+0000","last_peered":"2024-03-29T09:54:28.674124+0000","last_clean":"2024-03-29T09:54:28.674124+0000","last_became_active":"2024-03-29T09:54:27.470690+0000","last_became_peered":"2024-03-29T09:54:27.470690+0000","last_unstale":"2024-03-29T09:54:28.674124+0000","last_undegraded":"2024-03-29T09:54:28.674124+0000","last_fullsized":"2024-03-29T09:54:28.674124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:19.141206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.456285+0000","last_change":"2024-03-29T09:54:26.928327+0000","last_active":"2024-03-29T09:54:27.456285+0000","last_peered":"2024-03-29T09:54:27.456285+0000","last_clean":"2024-03-29T09:54:27.456285+0000","last_became_active":"2024-03-29T09:54:26.927938+0000","last_became_peered":"2024-03-29T09:54:26.927938+0000","last_unstale":"2024-03-29T09:54:27.456285+0000","last_undegraded":"2024-03-29T09:54:27.456285+0000","last_fullsized":"2024-03-29T09:54:27.456285+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:53.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674582+0000","last_change":"2024-03-29T09:54:27.466328+0000","last_active":"2024-03-29T09:54:28.674582+0000","last_peered":"2024-03-29T09:54:28.674582+0000","last_clean":"2024-03-29T09:54:28.674582+0000","last_became_active":"2024-03-29T09:54:27.465869+0000","last_became_peered":"2024-03-29T09:54:27.465869+0000","last_unstale":"2024-03-29T09:54:28.674582+0000","last_undegraded":"2024-03-29T09:54:28.674582+0000","last_fullsized":"2024-03-29T09:54:28.674582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:50.524833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.438039+0000","last_change":"2024-03-29T09:54:26.925276+0000","last_active":"2024-03-29T09:54:27.438039+0000","last_peered":"2024-03-29T09:54:27.438039+0000","last_clean":"2024-03-29T09:54:27.438039+0000","last_became_active":"2024-03-29T09:54:26.924980+0000","last_became_peered":"2024-03-29T09:54:26.924980+0000","last_unstale":"2024-03-29T09:54:27.438039+0000","last_undegraded":"2024-03-29T09:54:27.438039+0000","last_fullsized":"2024-03-29T09:54:27.438039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438030+0000","last_change":"2024-03-29T09:54:27.447713+0000","last_active":"2024-03-29T09:54:28.438030+0000","last_peered":"2024-03-29T09:54:28.438030+0000","last_clean":"2024-03-29T09:54:28.438030+0000","last_became_active":"2024-03-29T09:54:27.446552+0000","last_became_peered":"2024-03-29T09:54:27.446552+0000","last_unstale":"2024-03-29T09:54:28.438030+0000","last_undegraded":"2024-03-29T09:54:28.438030+0000","last_fullsized":"2024-03-29T09:54:28.438030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:31.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.437304+0000","last_change":"2024-03-29T09:54:26.911470+0000","last_active":"2024-03-29T09:54:27.437304+0000","last_peered":"2024-03-29T09:54:27.437304+0000","last_clean":"2024-03-29T09:54:27.437304+0000","last_became_active":"2024-03-29T09:54:26.911092+0000","last_became_peered":"2024-03-29T09:54:26.911092+0000","last_unstale":"2024-03-29T09:54:27.437304+0000","last_undegraded":"2024-03-29T09:54:27.437304+0000","last_fullsized":"2024-03-29T09:54:27.437304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436533+0000","last_change":"2024-03-29T09:54:27.459676+0000","last_active":"2024-03-29T09:54:28.436533+0000","last_peered":"2024-03-29T09:54:28.436533+0000","last_clean":"2024-03-29T09:54:28.436533+0000","last_became_active":"2024-03-29T09:54:27.459375+0000","last_became_peered":"2024-03-29T09:54:27.459375+0000","last_unstale":"2024-03-29T09:54:28.436533+0000","last_undegraded":"2024-03-29T09:54:28.436533+0000","last_fullsized":"2024-03-29T09:54:28.436533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:09.734996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572306+0000","last_change":"2024-03-29T09:54:27.446597+0000","last_active":"2024-03-29T09:54:28.572306+0000","last_peered":"2024-03-29T09:54:28.572306+0000","last_clean":"2024-03-29T09:54:28.572306+0000","last_became_active":"2024-03-29T09:54:27.444170+0000","last_became_peered":"2024-03-29T09:54:27.444170+0000","last_unstale":"2024-03-29T09:54:28.572306+0000","last_undegraded":"2024-03-29T09:54:28.572306+0000","last_fullsized":"2024-03-29T09:54:28.572306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.432316+0000","last_change":"2024-03-29T09:54:26.923022+0000","last_active":"2024-03-29T09:54:27.432316+0000","last_peered":"2024-03-29T09:54:27.432316+0000","last_clean":"2024-03-29T09:54:27.432316+0000","last_became_active":"2024-03-29T09:54:26.922558+0000","last_became_peered":"2024-03-29T09:54:26.922558+0000","last_unstale":"2024-03-29T09:54:27.432316+0000","last_undegraded":"2024-03-29T09:54:27.432316+0000","last_fullsized":"2024-03-29T09:54:27.432316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674355+0000","last_change":"2024-03-29T09:54:27.452598+0000","last_active":"2024-03-29T09:54:28.674355+0000","last_peered":"2024-03-29T09:54:28.674355+0000","last_clean":"2024-03-29T09:54:28.674355+0000","last_became_active":"2024-03-29T09:54:27.452331+0000","last_became_peered":"2024-03-29T09:54:27.452331+0000","last_unstale":"2024-03-29T09:54:28.674355+0000","last_undegraded":"2024-03-29T09:54:28.674355+0000","last_fullsized":"2024-03-29T09:54:28.674355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:50.524833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.433351+0000","last_change":"2024-03-29T09:54:26.916736+0000","last_active":"2024-03-29T09:54:27.433351+0000","last_peered":"2024-03-29T09:54:27.433351+0000","last_clean":"2024-03-29T09:54:27.433351+0000","last_became_active":"2024-03-29T09:54:26.916439+0000","last_became_peered":"2024-03-29T09:54:26.916439+0000","last_unstale":"2024-03-29T09:54:27.433351+0000","last_undegraded":"2024-03-29T09:54:27.433351+0000","last_fullsized":"2024-03-29T09:54:27.433351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.453076+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T09:54:27.453076+0000","last_peered":"2024-03-29T09:54:27.453076+0000","last_clean":"2024-03-29T09:54:27.453076+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T09:54:27.453076+0000","last_undegraded":"2024-03-29T09:54:27.453076+0000","last_fullsized":"2024-03-29T09:54:27.453076+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673776+0000","last_change":"2024-03-29T09:54:27.462932+0000","last_active":"2024-03-29T09:54:28.673776+0000","last_peered":"2024-03-29T09:54:28.673776+0000","last_clean":"2024-03-29T09:54:28.673776+0000","last_became_active":"2024-03-29T09:54:27.462665+0000","last_became_peered":"2024-03-29T09:54:27.462665+0000","last_unstale":"2024-03-29T09:54:28.673776+0000","last_undegraded":"2024-03-29T09:54:28.673776+0000","last_fullsized":"2024-03-29T09:54:28.673776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:02.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.453904+0000","last_change":"2024-03-29T09:54:26.924232+0000","last_active":"2024-03-29T09:54:27.453904+0000","last_peered":"2024-03-29T09:54:27.453904+0000","last_clean":"2024-03-29T09:54:27.453904+0000","last_became_active":"2024-03-29T09:54:26.923740+0000","last_became_peered":"2024-03-29T09:54:26.923740+0000","last_unstale":"2024-03-29T09:54:27.453904+0000","last_undegraded":"2024-03-29T09:54:27.453904+0000","last_fullsized":"2024-03-29T09:54:27.453904+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.457620+0000","last_change":"2024-03-29T09:54:26.929540+0000","last_active":"2024-03-29T09:54:27.457620+0000","last_peered":"2024-03-29T09:54:27.457620+0000","last_clean":"2024-03-29T09:54:27.457620+0000","last_became_active":"2024-03-29T09:54:26.928753+0000","last_became_peered":"2024-03-29T09:54:26.928753+0000","last_unstale":"2024-03-29T09:54:27.457620+0000","last_undegraded":"2024-03-29T09:54:27.457620+0000","last_fullsized":"2024-03-29T09:54:27.457620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:38.666641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572622+0000","last_change":"2024-03-29T09:54:27.449508+0000","last_active":"2024-03-29T09:54:28.572622+0000","last_peered":"2024-03-29T09:54:28.572622+0000","last_clean":"2024-03-29T09:54:28.572622+0000","last_became_active":"2024-03-29T09:54:27.449177+0000","last_became_peered":"2024-03-29T09:54:27.449177+0000","last_unstale":"2024-03-29T09:54:28.572622+0000","last_undegraded":"2024-03-29T09:54:28.572622+0000","last_fullsized":"2024-03-29T09:54:28.572622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572750+0000","last_change":"2024-03-29T09:54:26.909694+0000","last_active":"2024-03-29T09:54:28.572750+0000","last_peered":"2024-03-29T09:54:28.572750+0000","last_clean":"2024-03-29T09:54:28.572750+0000","last_became_active":"2024-03-29T09:54:26.909172+0000","last_became_peered":"2024-03-29T09:54:26.909172+0000","last_unstale":"2024-03-29T09:54:28.572750+0000","last_undegraded":"2024-03-29T09:54:28.572750+0000","last_fullsized":"2024-03-29T09:54:28.572750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:15:56.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674814+0000","last_change":"2024-03-29T09:54:27.466321+0000","last_active":"2024-03-29T09:54:28.674814+0000","last_peered":"2024-03-29T09:54:28.674814+0000","last_clean":"2024-03-29T09:54:28.674814+0000","last_became_active":"2024-03-29T09:54:27.465544+0000","last_became_peered":"2024-03-29T09:54:27.465544+0000","last_unstale":"2024-03-29T09:54:28.674814+0000","last_undegraded":"2024-03-29T09:54:28.674814+0000","last_fullsized":"2024-03-29T09:54:28.674814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.450346+0000","last_change":"2024-03-29T09:54:26.917641+0000","last_active":"2024-03-29T09:54:27.450346+0000","last_peered":"2024-03-29T09:54:27.450346+0000","last_clean":"2024-03-29T09:54:27.450346+0000","last_became_active":"2024-03-29T09:54:26.915808+0000","last_became_peered":"2024-03-29T09:54:26.915808+0000","last_unstale":"2024-03-29T09:54:27.450346+0000","last_undegraded":"2024-03-29T09:54:27.450346+0000","last_fullsized":"2024-03-29T09:54:27.450346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674304+0000","last_change":"2024-03-29T09:54:27.459303+0000","last_active":"2024-03-29T09:54:28.674304+0000","last_peered":"2024-03-29T09:54:28.674304+0000","last_clean":"2024-03-29T09:54:28.674304+0000","last_became_active":"2024-03-29T09:54:27.458902+0000","last_became_peered":"2024-03-29T09:54:27.458902+0000","last_unstale":"2024-03-29T09:54:28.674304+0000","last_undegraded":"2024-03-29T09:54:28.674304+0000","last_fullsized":"2024-03-29T09:54:28.674304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:25.651718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.454803+0000","last_change":"2024-03-29T09:54:26.929171+0000","last_active":"2024-03-29T09:54:27.454803+0000","last_peered":"2024-03-29T09:54:27.454803+0000","last_clean":"2024-03-29T09:54:27.454803+0000","last_became_active":"2024-03-29T09:54:26.928796+0000","last_became_peered":"2024-03-29T09:54:26.928796+0000","last_unstale":"2024-03-29T09:54:27.454803+0000","last_undegraded":"2024-03-29T09:54:27.454803+0000","last_fullsized":"2024-03-29T09:54:27.454803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436200+0000","last_change":"2024-03-29T09:54:27.453529+0000","last_active":"2024-03-29T09:54:28.436200+0000","last_peered":"2024-03-29T09:54:28.436200+0000","last_clean":"2024-03-29T09:54:28.436200+0000","last_became_active":"2024-03-29T09:54:27.453153+0000","last_became_peered":"2024-03-29T09:54:27.453153+0000","last_unstale":"2024-03-29T09:54:28.436200+0000","last_undegraded":"2024-03-29T09:54:28.436200+0000","last_fullsized":"2024-03-29T09:54:28.436200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:25.068710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.438223+0000","last_change":"2024-03-29T09:54:26.916239+0000","last_active":"2024-03-29T09:54:27.438223+0000","last_peered":"2024-03-29T09:54:27.438223+0000","last_clean":"2024-03-29T09:54:27.438223+0000","last_became_active":"2024-03-29T09:54:26.915960+0000","last_became_peered":"2024-03-29T09:54:26.915960+0000","last_unstale":"2024-03-29T09:54:27.438223+0000","last_undegraded":"2024-03-29T09:54:27.438223+0000","last_fullsized":"2024-03-29T09:54:27.438223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673647+0000","last_change":"2024-03-29T09:54:27.448090+0000","last_active":"2024-03-29T09:54:28.673647+0000","last_peered":"2024-03-29T09:54:28.673647+0000","last_clean":"2024-03-29T09:54:28.673647+0000","last_became_active":"2024-03-29T09:54:27.447614+0000","last_became_peered":"2024-03-29T09:54:27.447614+0000","last_unstale":"2024-03-29T09:54:28.673647+0000","last_undegraded":"2024-03-29T09:54:28.673647+0000","last_fullsized":"2024-03-29T09:54:28.673647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:25:15.310717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.438747+0000","last_change":"2024-03-29T09:54:26.919921+0000","last_active":"2024-03-29T09:54:27.438747+0000","last_peered":"2024-03-29T09:54:27.438747+0000","last_clean":"2024-03-29T09:54:27.438747+0000","last_became_active":"2024-03-29T09:54:26.919609+0000","last_became_peered":"2024-03-29T09:54:26.919609+0000","last_unstale":"2024-03-29T09:54:27.438747+0000","last_undegraded":"2024-03-29T09:54:27.438747+0000","last_fullsized":"2024-03-29T09:54:27.438747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:38.666641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674643+0000","last_change":"2024-03-29T09:54:27.458246+0000","last_active":"2024-03-29T09:54:28.674643+0000","last_peered":"2024-03-29T09:54:28.674643+0000","last_clean":"2024-03-29T09:54:28.674643+0000","last_became_active":"2024-03-29T09:54:27.457985+0000","last_became_peered":"2024-03-29T09:54:27.457985+0000","last_unstale":"2024-03-29T09:54:28.674643+0000","last_undegraded":"2024-03-29T09:54:28.674643+0000","last_fullsized":"2024-03-29T09:54:28.674643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:30.050849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.469210+0000","last_change":"2024-03-29T09:54:26.927045+0000","last_active":"2024-03-29T09:54:27.469210+0000","last_peered":"2024-03-29T09:54:27.469210+0000","last_clean":"2024-03-29T09:54:27.469210+0000","last_became_active":"2024-03-29T09:54:26.926750+0000","last_became_peered":"2024-03-29T09:54:26.926750+0000","last_unstale":"2024-03-29T09:54:27.469210+0000","last_undegraded":"2024-03-29T09:54:27.469210+0000","last_fullsized":"2024-03-29T09:54:27.469210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:15:56.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437955+0000","last_change":"2024-03-29T09:54:27.447647+0000","last_active":"2024-03-29T09:54:28.437955+0000","last_peered":"2024-03-29T09:54:28.437955+0000","last_clean":"2024-03-29T09:54:28.437955+0000","last_became_active":"2024-03-29T09:54:27.447371+0000","last_became_peered":"2024-03-29T09:54:27.447371+0000","last_unstale":"2024-03-29T09:54:28.437955+0000","last_undegraded":"2024-03-29T09:54:28.437955+0000","last_fullsized":"2024-03-29T09:54:28.437955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.432327+0000","last_change":"2024-03-29T09:54:26.929260+0000","last_active":"2024-03-29T09:54:27.432327+0000","last_peered":"2024-03-29T09:54:27.432327+0000","last_clean":"2024-03-29T09:54:27.432327+0000","last_became_active":"2024-03-29T09:54:26.928909+0000","last_became_peered":"2024-03-29T09:54:26.928909+0000","last_unstale":"2024-03-29T09:54:27.432327+0000","last_undegraded":"2024-03-29T09:54:27.432327+0000","last_fullsized":"2024-03-29T09:54:27.432327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:37:17.462538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674364+0000","last_change":"2024-03-29T09:54:27.471604+0000","last_active":"2024-03-29T09:54:28.674364+0000","last_peered":"2024-03-29T09:54:28.674364+0000","last_clean":"2024-03-29T09:54:28.674364+0000","last_became_active":"2024-03-29T09:54:27.471197+0000","last_became_peered":"2024-03-29T09:54:27.471197+0000","last_unstale":"2024-03-29T09:54:28.674364+0000","last_undegraded":"2024-03-29T09:54:28.674364+0000","last_fullsized":"2024-03-29T09:54:28.674364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:25.712202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.463137+0000","last_change":"2024-03-29T09:54:26.916668+0000","last_active":"2024-03-29T09:54:27.463137+0000","last_peered":"2024-03-29T09:54:27.463137+0000","last_clean":"2024-03-29T09:54:27.463137+0000","last_became_active":"2024-03-29T09:54:26.916233+0000","last_became_peered":"2024-03-29T09:54:26.916233+0000","last_unstale":"2024-03-29T09:54:27.463137+0000","last_undegraded":"2024-03-29T09:54:27.463137+0000","last_fullsized":"2024-03-29T09:54:27.463137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438328+0000","last_change":"2024-03-29T09:54:27.444803+0000","last_active":"2024-03-29T09:54:28.438328+0000","last_peered":"2024-03-29T09:54:28.438328+0000","last_clean":"2024-03-29T09:54:28.438328+0000","last_became_active":"2024-03-29T09:54:27.444104+0000","last_became_peered":"2024-03-29T09:54:27.444104+0000","last_unstale":"2024-03-29T09:54:28.438328+0000","last_undegraded":"2024-03-29T09:54:28.438328+0000","last_fullsized":"2024-03-29T09:54:28.438328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:25.712202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.464351+0000","last_change":"2024-03-29T09:54:26.923513+0000","last_active":"2024-03-29T09:54:27.464351+0000","last_peered":"2024-03-29T09:54:27.464351+0000","last_clean":"2024-03-29T09:54:27.464351+0000","last_became_active":"2024-03-29T09:54:26.922870+0000","last_became_peered":"2024-03-29T09:54:26.922870+0000","last_unstale":"2024-03-29T09:54:27.464351+0000","last_undegraded":"2024-03-29T09:54:27.464351+0000","last_fullsized":"2024-03-29T09:54:27.464351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674452+0000","last_change":"2024-03-29T09:54:27.471588+0000","last_active":"2024-03-29T09:54:28.674452+0000","last_peered":"2024-03-29T09:54:28.674452+0000","last_clean":"2024-03-29T09:54:28.674452+0000","last_became_active":"2024-03-29T09:54:27.468491+0000","last_became_peered":"2024-03-29T09:54:27.468491+0000","last_unstale":"2024-03-29T09:54:28.674452+0000","last_undegraded":"2024-03-29T09:54:28.674452+0000","last_fullsized":"2024-03-29T09:54:28.674452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:31.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438168+0000","last_change":"2024-03-29T09:54:27.461434+0000","last_active":"2024-03-29T09:54:28.438168+0000","last_peered":"2024-03-29T09:54:28.438168+0000","last_clean":"2024-03-29T09:54:28.438168+0000","last_became_active":"2024-03-29T09:54:27.461095+0000","last_became_peered":"2024-03-29T09:54:27.461095+0000","last_unstale":"2024-03-29T09:54:28.438168+0000","last_undegraded":"2024-03-29T09:54:28.438168+0000","last_fullsized":"2024-03-29T09:54:28.438168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:02.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.511270+0000","last_change":"2024-03-29T09:54:27.445834+0000","last_active":"2024-03-29T09:54:28.511270+0000","last_peered":"2024-03-29T09:54:28.511270+0000","last_clean":"2024-03-29T09:54:28.511270+0000","last_became_active":"2024-03-29T09:54:27.444597+0000","last_became_peered":"2024-03-29T09:54:27.444597+0000","last_unstale":"2024-03-29T09:54:28.511270+0000","last_undegraded":"2024-03-29T09:54:28.511270+0000","last_fullsized":"2024-03-29T09:54:28.511270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:36.310545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437472+0000","last_change":"2024-03-29T09:54:27.449523+0000","last_active":"2024-03-29T09:54:28.437472+0000","last_peered":"2024-03-29T09:54:28.437472+0000","last_clean":"2024-03-29T09:54:28.437472+0000","last_became_active":"2024-03-29T09:54:27.447299+0000","last_became_peered":"2024-03-29T09:54:27.447299+0000","last_unstale":"2024-03-29T09:54:28.437472+0000","last_undegraded":"2024-03-29T09:54:28.437472+0000","last_fullsized":"2024-03-29T09:54:28.437472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:18.345522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674066+0000","last_change":"2024-03-29T09:54:27.464458+0000","last_active":"2024-03-29T09:54:28.674066+0000","last_peered":"2024-03-29T09:54:28.674066+0000","last_clean":"2024-03-29T09:54:28.674066+0000","last_became_active":"2024-03-29T09:54:27.464146+0000","last_became_peered":"2024-03-29T09:54:27.464146+0000","last_unstale":"2024-03-29T09:54:28.674066+0000","last_undegraded":"2024-03-29T09:54:28.674066+0000","last_fullsized":"2024-03-29T09:54:28.674066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674683+0000","last_change":"2024-03-29T09:54:27.463860+0000","last_active":"2024-03-29T09:54:28.674683+0000","last_peered":"2024-03-29T09:54:28.674683+0000","last_clean":"2024-03-29T09:54:28.674683+0000","last_became_active":"2024-03-29T09:54:27.461514+0000","last_became_peered":"2024-03-29T09:54:27.461514+0000","last_unstale":"2024-03-29T09:54:28.674683+0000","last_undegraded":"2024-03-29T09:54:28.674683+0000","last_fullsized":"2024-03-29T09:54:28.674683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572614+0000","last_change":"2024-03-29T09:54:27.457584+0000","last_active":"2024-03-29T09:54:28.572614+0000","last_peered":"2024-03-29T09:54:28.572614+0000","last_clean":"2024-03-29T09:54:28.572614+0000","last_became_active":"2024-03-29T09:54:27.457325+0000","last_became_peered":"2024-03-29T09:54:27.457325+0000","last_unstale":"2024-03-29T09:54:28.572614+0000","last_undegraded":"2024-03-29T09:54:28.572614+0000","last_fullsized":"2024-03-29T09:54:28.572614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.439001+0000","last_change":"2024-03-29T09:54:27.467951+0000","last_active":"2024-03-29T09:54:28.439001+0000","last_peered":"2024-03-29T09:54:28.439001+0000","last_clean":"2024-03-29T09:54:28.439001+0000","last_became_active":"2024-03-29T09:54:27.466080+0000","last_became_peered":"2024-03-29T09:54:27.466080+0000","last_unstale":"2024-03-29T09:54:28.439001+0000","last_undegraded":"2024-03-29T09:54:28.439001+0000","last_fullsized":"2024-03-29T09:54:28.439001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674709+0000","last_change":"2024-03-29T09:54:27.460513+0000","last_active":"2024-03-29T09:54:28.674709+0000","last_peered":"2024-03-29T09:54:28.674709+0000","last_clean":"2024-03-29T09:54:28.674709+0000","last_became_active":"2024-03-29T09:54:27.460231+0000","last_became_peered":"2024-03-29T09:54:27.460231+0000","last_unstale":"2024-03-29T09:54:28.674709+0000","last_undegraded":"2024-03-29T09:54:28.674709+0000","last_fullsized":"2024-03-29T09:54:28.674709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437313+0000","last_change":"2024-03-29T09:54:27.462171+0000","last_active":"2024-03-29T09:54:28.437313+0000","last_peered":"2024-03-29T09:54:28.437313+0000","last_clean":"2024-03-29T09:54:28.437313+0000","last_became_active":"2024-03-29T09:54:27.461066+0000","last_became_peered":"2024-03-29T09:54:27.461066+0000","last_unstale":"2024-03-29T09:54:28.437313+0000","last_undegraded":"2024-03-29T09:54:28.437313+0000","last_fullsized":"2024-03-29T09:54:28.437313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:44:20.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674108+0000","last_change":"2024-03-29T09:54:27.456049+0000","last_active":"2024-03-29T09:54:28.674108+0000","last_peered":"2024-03-29T09:54:28.674108+0000","last_clean":"2024-03-29T09:54:28.674108+0000","last_became_active":"2024-03-29T09:54:27.454521+0000","last_became_peered":"2024-03-29T09:54:27.454521+0000","last_unstale":"2024-03-29T09:54:28.674108+0000","last_undegraded":"2024-03-29T09:54:28.674108+0000","last_fullsized":"2024-03-29T09:54:28.674108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437418+0000","last_change":"2024-03-29T09:54:27.465472+0000","last_active":"2024-03-29T09:54:28.437418+0000","last_peered":"2024-03-29T09:54:28.437418+0000","last_clean":"2024-03-29T09:54:28.437418+0000","last_became_active":"2024-03-29T09:54:27.465180+0000","last_became_peered":"2024-03-29T09:54:27.465180+0000","last_unstale":"2024-03-29T09:54:28.437418+0000","last_undegraded":"2024-03-29T09:54:28.437418+0000","last_fullsized":"2024-03-29T09:54:28.437418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674044+0000","last_change":"2024-03-29T09:54:27.466113+0000","last_active":"2024-03-29T09:54:28.674044+0000","last_peered":"2024-03-29T09:54:28.674044+0000","last_clean":"2024-03-29T09:54:28.674044+0000","last_became_active":"2024-03-29T09:54:27.465912+0000","last_became_peered":"2024-03-29T09:54:27.465912+0000","last_unstale":"2024-03-29T09:54:28.674044+0000","last_undegraded":"2024-03-29T09:54:28.674044+0000","last_fullsized":"2024-03-29T09:54:28.674044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438778+0000","last_change":"2024-03-29T09:54:27.447415+0000","last_active":"2024-03-29T09:54:28.438778+0000","last_peered":"2024-03-29T09:54:28.438778+0000","last_clean":"2024-03-29T09:54:28.438778+0000","last_became_active":"2024-03-29T09:54:27.446985+0000","last_became_peered":"2024-03-29T09:54:27.446985+0000","last_unstale":"2024-03-29T09:54:28.438778+0000","last_undegraded":"2024-03-29T09:54:28.438778+0000","last_fullsized":"2024-03-29T09:54:28.438778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":806912,"data_stored":660885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1840000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3319999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4550000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T09:54:38.183 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T09:54:38.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.417+0000 7fde11800700 1 Processor -- start 2024-03-29T09:54:38.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.418+0000 7fde11800700 1 -- start start 2024-03-29T09:54:38.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.419+0000 7fde11800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fde0c160f10 0x7fde0c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.419+0000 7fde11800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fde0c0cf230 0x7fde0c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.419+0000 7fde11800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c0ceda0 con 0x7fde0c161590 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.419+0000 7fde11800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c0d23d0 con 0x7fde0c0cf230 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.419+0000 7fde11800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c1612f0 con 0x7fde0c160f10 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fde0c0cf230 0x7fde0c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fde0c0cf230 0x7fde0c0d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40960/0 (socket says 172.21.15.12:40960) 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fde0c161590 0x7fde0c0b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58238/0 (socket says 172.21.15.12:58238) 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0a000700 1 -- 172.21.15.12:0/3340484348 learned_addr learned my addr 172.21.15.12:0/3340484348 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0aa00700 1 --2- 172.21.15.12:0/3340484348 >> v2:172.21.15.159:3300/0 conn(0x7fde0c160f10 0x7fde0c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0a000700 1 -- 172.21.15.12:0/3340484348 >> v2:172.21.15.159:3300/0 conn(0x7fde0c160f10 msgr2=0x7fde0c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0a000700 1 --2- 172.21.15.12:0/3340484348 >> v2:172.21.15.159:3300/0 conn(0x7fde0c160f10 0x7fde0c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0a000700 1 -- 172.21.15.12:0/3340484348 >> v1:172.21.15.12:6789/0 conn(0x7fde0c161590 legacy=0x7fde0c0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0a000700 1 -- 172.21.15.12:0/3340484348 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c0ce470 con 0x7fde0c0cf230 2024-03-29T09:54:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0aa00700 1 --2- 172.21.15.12:0/3340484348 >> v2:172.21.15.159:3300/0 conn(0x7fde0c160f10 0x7fde0c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:38.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde0a000700 1 --2- 172.21.15.12:0/3340484348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 0x7fde0c0d1e90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fde0c0d1080 tx=0x7fddf8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d489ecb40438d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.420+0000 7fde09600700 1 -- 172.21.15.12:0/3340484348 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf800b870 con 0x7fde0c0cf230 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.421+0000 7fde09600700 1 -- 172.21.15.12:0/3340484348 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf800b9d0 con 0x7fde0c0cf230 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.421+0000 7fde09600700 1 -- 172.21.15.12:0/3340484348 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf800bd60 con 0x7fde0c0cf230 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.421+0000 7fde11800700 1 -- 172.21.15.12:0/3340484348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 msgr2=0x7fde0c0d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.421+0000 7fde11800700 1 --2- 172.21.15.12:0/3340484348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 0x7fde0c0d1e90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fde0c0d1080 tx=0x7fddf8009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.421+0000 7fde11800700 1 -- 172.21.15.12:0/3340484348 shutdown_connections 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.421+0000 7fde11800700 1 --2- 172.21.15.12:0/3340484348 >> v2:172.21.15.159:3300/0 conn(0x7fde0c160f10 0x7fde0c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.421+0000 7fde11800700 1 --2- 172.21.15.12:0/3340484348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 0x7fde0c0d1e90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.421+0000 7fde11800700 1 -- 172.21.15.12:0/3340484348 >> 172.21.15.12:0/3340484348 conn(0x7fde0c08aff0 msgr2=0x7fde0c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.421+0000 7fde11800700 1 -- 172.21.15.12:0/3340484348 shutdown_connections 2024-03-29T09:54:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.421+0000 7fde11800700 1 -- 172.21.15.12:0/3340484348 wait complete. 2024-03-29T09:54:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.423+0000 7fde11800700 1 Processor -- start 2024-03-29T09:54:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.423+0000 7fde11800700 1 -- start start 2024-03-29T09:54:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde11800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 0x7fde0c1f7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde11800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde0c160f10 0x7fde0c1550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde11800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde0c161590 0x7fde0c15f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde11800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c1569a0 con 0x7fde0c0cf230 2024-03-29T09:54:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde11800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c154b90 con 0x7fde0c160f10 2024-03-29T09:54:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde11800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c155900 con 0x7fde0c161590 2024-03-29T09:54:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde0aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 0x7fde0c1f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde0aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 0x7fde0c1f7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40970/0 (socket says 172.21.15.12:40970) 2024-03-29T09:54:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde0aa00700 1 -- 172.21.15.12:0/1195922401 learned_addr learned my addr 172.21.15.12:0/1195922401 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde0b400700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde0c161590 0x7fde0c15f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.424+0000 7fde0a000700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde0c160f10 0x7fde0c1550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde0aa00700 1 -- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde0c161590 msgr2=0x7fde0c15f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde0aa00700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde0c161590 0x7fde0c15f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde0aa00700 1 -- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde0c160f10 msgr2=0x7fde0c1550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde0aa00700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde0c160f10 0x7fde0c1550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde0aa00700 1 -- 172.21.15.12:0/1195922401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8009550 con 0x7fde0c0cf230 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde0aa00700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 0x7fde0c1f7500 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00c7f0 tx=0x7fddfc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde02a00700 1 -- 172.21.15.12:0/1195922401 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc011800 con 0x7fde0c0cf230 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde02a00700 1 -- 172.21.15.12:0/1195922401 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc011960 con 0x7fde0c0cf230 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde11800700 1 -- 172.21.15.12:0/1195922401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c155be0 con 0x7fde0c0cf230 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde02a00700 1 -- 172.21.15.12:0/1195922401 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc011d10 con 0x7fde0c0cf230 2024-03-29T09:54:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.425+0000 7fde11800700 1 -- 172.21.15.12:0/1195922401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c1561a0 con 0x7fde0c0cf230 2024-03-29T09:54:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.426+0000 7fde11800700 1 -- 172.21.15.12:0/1195922401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c1566c0 con 0x7fde0c0cf230 2024-03-29T09:54:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.427+0000 7fde02a00700 1 -- 172.21.15.12:0/1195922401 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fddfc018360 con 0x7fde0c0cf230 2024-03-29T09:54:38.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.429+0000 7fde02a00700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddec06ac90 0x7fddec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.429+0000 7fde02a00700 1 -- 172.21.15.12:0/1195922401 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddfc0892b0 con 0x7fde0c0cf230 2024-03-29T09:54:38.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.429+0000 7fde0a000700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddec06ac90 0x7fddec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.430+0000 7fde0a000700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddec06ac90 0x7fddec06d150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fddf8014040 tx=0x7fddf8008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.434+0000 7fde02a00700 1 -- 172.21.15.12:0/1195922401 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fde0c1566c0 con 0x7fde0c0cf230 2024-03-29T09:54:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.643+0000 7fde11800700 1 -- 172.21.15.12:0/1195922401 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fde0c0cf6b0 con 0x7fddec06ac90 2024-03-29T09:54:38.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.652+0000 7fde02a00700 1 -- 172.21.15.12:0/1195922401 <== mgr.4114 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264571 (secure 0 0 0) 0x7fddf800bbc0 con 0x7fddec06ac90 2024-03-29T09:54:38.655 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 -- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddec06ac90 msgr2=0x7fddec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddec06ac90 0x7fddec06d150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fddf8014040 tx=0x7fddf8008030 comp rx=0 tx=0).stop 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 -- 172.21.15.12:0/1195922401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 msgr2=0x7fde0c1f7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 0x7fde0c1f7500 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00c7f0 tx=0x7fddfc00cbb0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 -- 172.21.15.12:0/1195922401 shutdown_connections 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddec06ac90 0x7fddec06d150 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde0c161590 0x7fde0c15f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde0c0cf230 0x7fde0c1f7500 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 --2- 172.21.15.12:0/1195922401 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde0c160f10 0x7fde0c1550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 -- 172.21.15.12:0/1195922401 >> 172.21.15.12:0/1195922401 conn(0x7fde0c08aff0 msgr2=0x7fde0c160150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.656+0000 7fde11800700 1 -- 172.21.15.12:0/1195922401 shutdown_connections 2024-03-29T09:54:38.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.657+0000 7fde11800700 1 -- 172.21.15.12:0/1195922401 wait complete. 2024-03-29T09:54:38.659 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T09:54:38.675 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-29T09:54:38.439432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219064,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755656,"statfs":{"total":773094113280,"available":772869791744,"internally_reserved":0,"allocated":2887680,"data_stored":1751625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003931"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674620+0000","last_change":"2024-03-29T09:54:27.472798+0000","last_active":"2024-03-29T09:54:28.674620+0000","last_peered":"2024-03-29T09:54:28.674620+0000","last_clean":"2024-03-29T09:54:28.674620+0000","last_became_active":"2024-03-29T09:54:27.468781+0000","last_became_peered":"2024-03-29T09:54:27.468781+0000","last_unstale":"2024-03-29T09:54:28.674620+0000","last_undegraded":"2024-03-29T09:54:28.674620+0000","last_fullsized":"2024-03-29T09:54:28.674620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:10.721926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572841+0000","last_change":"2024-03-29T09:54:27.456722+0000","last_active":"2024-03-29T09:54:28.572841+0000","last_peered":"2024-03-29T09:54:28.572841+0000","last_clean":"2024-03-29T09:54:28.572841+0000","last_became_active":"2024-03-29T09:54:27.456334+0000","last_became_peered":"2024-03-29T09:54:27.456334+0000","last_unstale":"2024-03-29T09:54:28.572841+0000","last_undegraded":"2024-03-29T09:54:28.572841+0000","last_fullsized":"2024-03-29T09:54:28.572841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:52.185097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674299+0000","last_change":"2024-03-29T09:54:27.455943+0000","last_active":"2024-03-29T09:54:28.674299+0000","last_peered":"2024-03-29T09:54:28.674299+0000","last_clean":"2024-03-29T09:54:28.674299+0000","last_became_active":"2024-03-29T09:54:27.455551+0000","last_became_peered":"2024-03-29T09:54:27.455551+0000","last_unstale":"2024-03-29T09:54:28.674299+0000","last_undegraded":"2024-03-29T09:54:28.674299+0000","last_fullsized":"2024-03-29T09:54:28.674299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:03.191627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438591+0000","last_change":"2024-03-29T09:54:27.445755+0000","last_active":"2024-03-29T09:54:28.438591+0000","last_peered":"2024-03-29T09:54:28.438591+0000","last_clean":"2024-03-29T09:54:28.438591+0000","last_became_active":"2024-03-29T09:54:27.444623+0000","last_became_peered":"2024-03-29T09:54:27.444623+0000","last_unstale":"2024-03-29T09:54:28.438591+0000","last_undegraded":"2024-03-29T09:54:28.438591+0000","last_fullsized":"2024-03-29T09:54:28.438591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:19.141206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674074+0000","last_change":"2024-03-29T09:54:27.468817+0000","last_active":"2024-03-29T09:54:28.674074+0000","last_peered":"2024-03-29T09:54:28.674074+0000","last_clean":"2024-03-29T09:54:28.674074+0000","last_became_active":"2024-03-29T09:54:27.467049+0000","last_became_peered":"2024-03-29T09:54:27.467049+0000","last_unstale":"2024-03-29T09:54:28.674074+0000","last_undegraded":"2024-03-29T09:54:28.674074+0000","last_fullsized":"2024-03-29T09:54:28.674074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:44:20.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437373+0000","last_change":"2024-03-29T09:54:27.453526+0000","last_active":"2024-03-29T09:54:28.437373+0000","last_peered":"2024-03-29T09:54:28.437373+0000","last_clean":"2024-03-29T09:54:28.437373+0000","last_became_active":"2024-03-29T09:54:27.452090+0000","last_became_peered":"2024-03-29T09:54:27.452090+0000","last_unstale":"2024-03-29T09:54:28.437373+0000","last_undegraded":"2024-03-29T09:54:28.437373+0000","last_fullsized":"2024-03-29T09:54:28.437373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:30.050849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673999+0000","last_change":"2024-03-29T09:54:27.471016+0000","last_active":"2024-03-29T09:54:28.673999+0000","last_peered":"2024-03-29T09:54:28.673999+0000","last_clean":"2024-03-29T09:54:28.673999+0000","last_became_active":"2024-03-29T09:54:27.470657+0000","last_became_peered":"2024-03-29T09:54:27.470657+0000","last_unstale":"2024-03-29T09:54:28.673999+0000","last_undegraded":"2024-03-29T09:54:28.673999+0000","last_fullsized":"2024-03-29T09:54:28.673999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:56.949701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673522+0000","last_change":"2024-03-29T09:54:27.458451+0000","last_active":"2024-03-29T09:54:28.673522+0000","last_peered":"2024-03-29T09:54:28.673522+0000","last_clean":"2024-03-29T09:54:28.673522+0000","last_became_active":"2024-03-29T09:54:27.458110+0000","last_became_peered":"2024-03-29T09:54:27.458110+0000","last_unstale":"2024-03-29T09:54:28.673522+0000","last_undegraded":"2024-03-29T09:54:28.673522+0000","last_fullsized":"2024-03-29T09:54:28.673522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:57.886383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674535+0000","last_change":"2024-03-29T09:54:27.471858+0000","last_active":"2024-03-29T09:54:28.674535+0000","last_peered":"2024-03-29T09:54:28.674535+0000","last_clean":"2024-03-29T09:54:28.674535+0000","last_became_active":"2024-03-29T09:54:27.469164+0000","last_became_peered":"2024-03-29T09:54:27.469164+0000","last_unstale":"2024-03-29T09:54:28.674535+0000","last_undegraded":"2024-03-29T09:54:28.674535+0000","last_fullsized":"2024-03-29T09:54:28.674535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:44:42.873042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.439014+0000","last_change":"2024-03-29T09:54:27.442987+0000","last_active":"2024-03-29T09:54:28.439014+0000","last_peered":"2024-03-29T09:54:28.439014+0000","last_clean":"2024-03-29T09:54:28.439014+0000","last_became_active":"2024-03-29T09:54:27.442294+0000","last_became_peered":"2024-03-29T09:54:27.442294+0000","last_unstale":"2024-03-29T09:54:28.439014+0000","last_undegraded":"2024-03-29T09:54:28.439014+0000","last_fullsized":"2024-03-29T09:54:28.439014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438771+0000","last_change":"2024-03-29T09:54:27.442621+0000","last_active":"2024-03-29T09:54:28.438771+0000","last_peered":"2024-03-29T09:54:28.438771+0000","last_clean":"2024-03-29T09:54:28.438771+0000","last_became_active":"2024-03-29T09:54:27.442259+0000","last_became_peered":"2024-03-29T09:54:27.442259+0000","last_unstale":"2024-03-29T09:54:28.438771+0000","last_undegraded":"2024-03-29T09:54:28.438771+0000","last_fullsized":"2024-03-29T09:54:28.438771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436677+0000","last_change":"2024-03-29T09:54:27.453502+0000","last_active":"2024-03-29T09:54:28.436677+0000","last_peered":"2024-03-29T09:54:28.436677+0000","last_clean":"2024-03-29T09:54:28.436677+0000","last_became_active":"2024-03-29T09:54:27.453015+0000","last_became_peered":"2024-03-29T09:54:27.453015+0000","last_unstale":"2024-03-29T09:54:28.436677+0000","last_undegraded":"2024-03-29T09:54:28.436677+0000","last_fullsized":"2024-03-29T09:54:28.436677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:54.979913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674480+0000","last_change":"2024-03-29T09:54:27.451715+0000","last_active":"2024-03-29T09:54:28.674480+0000","last_peered":"2024-03-29T09:54:28.674480+0000","last_clean":"2024-03-29T09:54:28.674480+0000","last_became_active":"2024-03-29T09:54:27.451432+0000","last_became_peered":"2024-03-29T09:54:27.451432+0000","last_unstale":"2024-03-29T09:54:28.674480+0000","last_undegraded":"2024-03-29T09:54:28.674480+0000","last_fullsized":"2024-03-29T09:54:28.674480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674068+0000","last_change":"2024-03-29T09:54:27.460653+0000","last_active":"2024-03-29T09:54:28.674068+0000","last_peered":"2024-03-29T09:54:28.674068+0000","last_clean":"2024-03-29T09:54:28.674068+0000","last_became_active":"2024-03-29T09:54:27.459198+0000","last_became_peered":"2024-03-29T09:54:27.459198+0000","last_unstale":"2024-03-29T09:54:28.674068+0000","last_undegraded":"2024-03-29T09:54:28.674068+0000","last_fullsized":"2024-03-29T09:54:28.674068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:27:59.103915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572329+0000","last_change":"2024-03-29T09:54:27.457510+0000","last_active":"2024-03-29T09:54:28.572329+0000","last_peered":"2024-03-29T09:54:28.572329+0000","last_clean":"2024-03-29T09:54:28.572329+0000","last_became_active":"2024-03-29T09:54:27.456319+0000","last_became_peered":"2024-03-29T09:54:27.456319+0000","last_unstale":"2024-03-29T09:54:28.572329+0000","last_undegraded":"2024-03-29T09:54:28.572329+0000","last_fullsized":"2024-03-29T09:54:28.572329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437168+0000","last_change":"2024-03-29T09:54:27.460545+0000","last_active":"2024-03-29T09:54:28.437168+0000","last_peered":"2024-03-29T09:54:28.437168+0000","last_clean":"2024-03-29T09:54:28.437168+0000","last_became_active":"2024-03-29T09:54:27.460152+0000","last_became_peered":"2024-03-29T09:54:27.460152+0000","last_unstale":"2024-03-29T09:54:28.437168+0000","last_undegraded":"2024-03-29T09:54:28.437168+0000","last_fullsized":"2024-03-29T09:54:28.437168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:52.185097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438011+0000","last_change":"2024-03-29T09:54:27.469947+0000","last_active":"2024-03-29T09:54:28.438011+0000","last_peered":"2024-03-29T09:54:28.438011+0000","last_clean":"2024-03-29T09:54:28.438011+0000","last_became_active":"2024-03-29T09:54:27.469649+0000","last_became_peered":"2024-03-29T09:54:27.469649+0000","last_unstale":"2024-03-29T09:54:28.438011+0000","last_undegraded":"2024-03-29T09:54:28.438011+0000","last_fullsized":"2024-03-29T09:54:28.438011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:44:42.873042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.511192+0000","last_change":"2024-03-29T09:54:27.438206+0000","last_active":"2024-03-29T09:54:28.511192+0000","last_peered":"2024-03-29T09:54:28.511192+0000","last_clean":"2024-03-29T09:54:28.511192+0000","last_became_active":"2024-03-29T09:54:27.437957+0000","last_became_peered":"2024-03-29T09:54:27.437957+0000","last_unstale":"2024-03-29T09:54:28.511192+0000","last_undegraded":"2024-03-29T09:54:28.511192+0000","last_fullsized":"2024-03-29T09:54:28.511192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:03.191627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674238+0000","last_change":"2024-03-29T09:54:27.466200+0000","last_active":"2024-03-29T09:54:28.674238+0000","last_peered":"2024-03-29T09:54:28.674238+0000","last_clean":"2024-03-29T09:54:28.674238+0000","last_became_active":"2024-03-29T09:54:27.465886+0000","last_became_peered":"2024-03-29T09:54:27.465886+0000","last_unstale":"2024-03-29T09:54:28.674238+0000","last_undegraded":"2024-03-29T09:54:28.674238+0000","last_fullsized":"2024-03-29T09:54:28.674238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:57.886383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.459838+0000","last_change":"2024-03-29T09:54:26.929518+0000","last_active":"2024-03-29T09:54:27.459838+0000","last_peered":"2024-03-29T09:54:27.459838+0000","last_clean":"2024-03-29T09:54:27.459838+0000","last_became_active":"2024-03-29T09:54:26.929253+0000","last_became_peered":"2024-03-29T09:54:26.929253+0000","last_unstale":"2024-03-29T09:54:27.459838+0000","last_undegraded":"2024-03-29T09:54:27.459838+0000","last_fullsized":"2024-03-29T09:54:27.459838+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:40:59.105279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674699+0000","last_change":"2024-03-29T09:54:27.454266+0000","last_active":"2024-03-29T09:54:28.674699+0000","last_peered":"2024-03-29T09:54:28.674699+0000","last_clean":"2024-03-29T09:54:28.674699+0000","last_became_active":"2024-03-29T09:54:27.454005+0000","last_became_peered":"2024-03-29T09:54:27.454005+0000","last_unstale":"2024-03-29T09:54:28.674699+0000","last_undegraded":"2024-03-29T09:54:28.674699+0000","last_fullsized":"2024-03-29T09:54:28.674699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:02.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.467235+0000","last_change":"2024-03-29T09:54:26.926684+0000","last_active":"2024-03-29T09:54:27.467235+0000","last_peered":"2024-03-29T09:54:27.467235+0000","last_clean":"2024-03-29T09:54:27.467235+0000","last_became_active":"2024-03-29T09:54:26.926350+0000","last_became_peered":"2024-03-29T09:54:26.926350+0000","last_unstale":"2024-03-29T09:54:27.467235+0000","last_undegraded":"2024-03-29T09:54:27.467235+0000","last_fullsized":"2024-03-29T09:54:27.467235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:37.644975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673888+0000","last_change":"2024-03-29T09:54:27.452090+0000","last_active":"2024-03-29T09:54:28.673888+0000","last_peered":"2024-03-29T09:54:28.673888+0000","last_clean":"2024-03-29T09:54:28.673888+0000","last_became_active":"2024-03-29T09:54:27.451361+0000","last_became_peered":"2024-03-29T09:54:27.451361+0000","last_unstale":"2024-03-29T09:54:28.673888+0000","last_undegraded":"2024-03-29T09:54:28.673888+0000","last_fullsized":"2024-03-29T09:54:28.673888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:25:15.310717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.468152+0000","last_change":"2024-03-29T09:54:26.924715+0000","last_active":"2024-03-29T09:54:27.468152+0000","last_peered":"2024-03-29T09:54:27.468152+0000","last_clean":"2024-03-29T09:54:27.468152+0000","last_became_active":"2024-03-29T09:54:26.924430+0000","last_became_peered":"2024-03-29T09:54:26.924430+0000","last_unstale":"2024-03-29T09:54:27.468152+0000","last_undegraded":"2024-03-29T09:54:27.468152+0000","last_fullsized":"2024-03-29T09:54:27.468152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438051+0000","last_change":"2024-03-29T09:54:27.455410+0000","last_active":"2024-03-29T09:54:28.438051+0000","last_peered":"2024-03-29T09:54:28.438051+0000","last_clean":"2024-03-29T09:54:28.438051+0000","last_became_active":"2024-03-29T09:54:27.453992+0000","last_became_peered":"2024-03-29T09:54:27.453992+0000","last_unstale":"2024-03-29T09:54:28.438051+0000","last_undegraded":"2024-03-29T09:54:28.438051+0000","last_fullsized":"2024-03-29T09:54:28.438051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:57.886383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.469487+0000","last_change":"2024-03-29T09:54:26.931418+0000","last_active":"2024-03-29T09:54:27.469487+0000","last_peered":"2024-03-29T09:54:27.469487+0000","last_clean":"2024-03-29T09:54:27.469487+0000","last_became_active":"2024-03-29T09:54:26.931148+0000","last_became_peered":"2024-03-29T09:54:26.931148+0000","last_unstale":"2024-03-29T09:54:27.469487+0000","last_undegraded":"2024-03-29T09:54:27.469487+0000","last_fullsized":"2024-03-29T09:54:27.469487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:02.185001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437930+0000","last_change":"2024-03-29T09:54:27.454343+0000","last_active":"2024-03-29T09:54:28.437930+0000","last_peered":"2024-03-29T09:54:28.437930+0000","last_clean":"2024-03-29T09:54:28.437930+0000","last_became_active":"2024-03-29T09:54:27.454012+0000","last_became_peered":"2024-03-29T09:54:27.454012+0000","last_unstale":"2024-03-29T09:54:28.437930+0000","last_undegraded":"2024-03-29T09:54:28.437930+0000","last_fullsized":"2024-03-29T09:54:28.437930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:50.524833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.467317+0000","last_change":"2024-03-29T09:54:26.929141+0000","last_active":"2024-03-29T09:54:27.467317+0000","last_peered":"2024-03-29T09:54:27.467317+0000","last_clean":"2024-03-29T09:54:27.467317+0000","last_became_active":"2024-03-29T09:54:26.928876+0000","last_became_peered":"2024-03-29T09:54:26.928876+0000","last_unstale":"2024-03-29T09:54:27.467317+0000","last_undegraded":"2024-03-29T09:54:27.467317+0000","last_fullsized":"2024-03-29T09:54:27.467317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:37:17.462538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436160+0000","last_change":"2024-03-29T09:54:27.448632+0000","last_active":"2024-03-29T09:54:28.436160+0000","last_peered":"2024-03-29T09:54:28.436160+0000","last_clean":"2024-03-29T09:54:28.436160+0000","last_became_active":"2024-03-29T09:54:27.448165+0000","last_became_peered":"2024-03-29T09:54:27.448165+0000","last_unstale":"2024-03-29T09:54:28.436160+0000","last_undegraded":"2024-03-29T09:54:28.436160+0000","last_fullsized":"2024-03-29T09:54:28.436160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:58.125922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.455407+0000","last_change":"2024-03-29T09:54:26.916444+0000","last_active":"2024-03-29T09:54:27.455407+0000","last_peered":"2024-03-29T09:54:27.455407+0000","last_clean":"2024-03-29T09:54:27.455407+0000","last_became_active":"2024-03-29T09:54:26.916132+0000","last_became_peered":"2024-03-29T09:54:26.916132+0000","last_unstale":"2024-03-29T09:54:27.455407+0000","last_undegraded":"2024-03-29T09:54:27.455407+0000","last_fullsized":"2024-03-29T09:54:27.455407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674190+0000","last_change":"2024-03-29T09:54:27.464314+0000","last_active":"2024-03-29T09:54:28.674190+0000","last_peered":"2024-03-29T09:54:28.674190+0000","last_clean":"2024-03-29T09:54:28.674190+0000","last_became_active":"2024-03-29T09:54:27.463702+0000","last_became_peered":"2024-03-29T09:54:27.463702+0000","last_unstale":"2024-03-29T09:54:28.674190+0000","last_undegraded":"2024-03-29T09:54:28.674190+0000","last_fullsized":"2024-03-29T09:54:28.674190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.444809+0000","last_change":"2024-03-29T09:54:26.921122+0000","last_active":"2024-03-29T09:54:27.444809+0000","last_peered":"2024-03-29T09:54:27.444809+0000","last_clean":"2024-03-29T09:54:27.444809+0000","last_became_active":"2024-03-29T09:54:26.919924+0000","last_became_peered":"2024-03-29T09:54:26.919924+0000","last_unstale":"2024-03-29T09:54:27.444809+0000","last_undegraded":"2024-03-29T09:54:27.444809+0000","last_fullsized":"2024-03-29T09:54:27.444809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673522+0000","last_change":"2024-03-29T09:54:27.465683+0000","last_active":"2024-03-29T09:54:28.673522+0000","last_peered":"2024-03-29T09:54:28.673522+0000","last_clean":"2024-03-29T09:54:28.673522+0000","last_became_active":"2024-03-29T09:54:27.465379+0000","last_became_peered":"2024-03-29T09:54:27.465379+0000","last_unstale":"2024-03-29T09:54:28.673522+0000","last_undegraded":"2024-03-29T09:54:28.673522+0000","last_fullsized":"2024-03-29T09:54:28.673522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:52.185097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.434322+0000","last_change":"2024-03-29T09:54:26.918838+0000","last_active":"2024-03-29T09:54:27.434322+0000","last_peered":"2024-03-29T09:54:27.434322+0000","last_clean":"2024-03-29T09:54:27.434322+0000","last_became_active":"2024-03-29T09:54:26.918185+0000","last_became_peered":"2024-03-29T09:54:26.918185+0000","last_unstale":"2024-03-29T09:54:27.434322+0000","last_undegraded":"2024-03-29T09:54:27.434322+0000","last_fullsized":"2024-03-29T09:54:27.434322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572430+0000","last_change":"2024-03-29T09:54:27.449379+0000","last_active":"2024-03-29T09:54:28.572430+0000","last_peered":"2024-03-29T09:54:28.572430+0000","last_clean":"2024-03-29T09:54:28.572430+0000","last_became_active":"2024-03-29T09:54:27.448935+0000","last_became_peered":"2024-03-29T09:54:27.448935+0000","last_unstale":"2024-03-29T09:54:28.572430+0000","last_undegraded":"2024-03-29T09:54:28.572430+0000","last_fullsized":"2024-03-29T09:54:28.572430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.434616+0000","last_change":"2024-03-29T09:54:26.920134+0000","last_active":"2024-03-29T09:54:27.434616+0000","last_peered":"2024-03-29T09:54:27.434616+0000","last_clean":"2024-03-29T09:54:27.434616+0000","last_became_active":"2024-03-29T09:54:26.919158+0000","last_became_peered":"2024-03-29T09:54:26.919158+0000","last_unstale":"2024-03-29T09:54:27.434616+0000","last_undegraded":"2024-03-29T09:54:27.434616+0000","last_fullsized":"2024-03-29T09:54:27.434616+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438287+0000","last_change":"2024-03-29T09:54:27.467877+0000","last_active":"2024-03-29T09:54:28.438287+0000","last_peered":"2024-03-29T09:54:28.438287+0000","last_clean":"2024-03-29T09:54:28.438287+0000","last_became_active":"2024-03-29T09:54:27.467347+0000","last_became_peered":"2024-03-29T09:54:27.467347+0000","last_unstale":"2024-03-29T09:54:28.438287+0000","last_undegraded":"2024-03-29T09:54:28.438287+0000","last_fullsized":"2024-03-29T09:54:28.438287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.437566+0000","last_change":"2024-03-29T09:54:26.911930+0000","last_active":"2024-03-29T09:54:27.437566+0000","last_peered":"2024-03-29T09:54:27.437566+0000","last_clean":"2024-03-29T09:54:27.437566+0000","last_became_active":"2024-03-29T09:54:26.911549+0000","last_became_peered":"2024-03-29T09:54:26.911549+0000","last_unstale":"2024-03-29T09:54:27.437566+0000","last_undegraded":"2024-03-29T09:54:27.437566+0000","last_fullsized":"2024-03-29T09:54:27.437566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674175+0000","last_change":"2024-03-29T09:54:27.451917+0000","last_active":"2024-03-29T09:54:28.674175+0000","last_peered":"2024-03-29T09:54:28.674175+0000","last_clean":"2024-03-29T09:54:28.674175+0000","last_became_active":"2024-03-29T09:54:27.450981+0000","last_became_peered":"2024-03-29T09:54:27.450981+0000","last_unstale":"2024-03-29T09:54:28.674175+0000","last_undegraded":"2024-03-29T09:54:28.674175+0000","last_fullsized":"2024-03-29T09:54:28.674175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:41:21.776144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.470049+0000","last_change":"2024-03-29T09:54:26.917969+0000","last_active":"2024-03-29T09:54:27.470049+0000","last_peered":"2024-03-29T09:54:27.470049+0000","last_clean":"2024-03-29T09:54:27.470049+0000","last_became_active":"2024-03-29T09:54:26.916331+0000","last_became_peered":"2024-03-29T09:54:26.916331+0000","last_unstale":"2024-03-29T09:54:27.470049+0000","last_undegraded":"2024-03-29T09:54:27.470049+0000","last_fullsized":"2024-03-29T09:54:27.470049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:02.185001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436752+0000","last_change":"2024-03-29T09:54:27.465694+0000","last_active":"2024-03-29T09:54:28.436752+0000","last_peered":"2024-03-29T09:54:28.436752+0000","last_clean":"2024-03-29T09:54:28.436752+0000","last_became_active":"2024-03-29T09:54:27.465420+0000","last_became_peered":"2024-03-29T09:54:27.465420+0000","last_unstale":"2024-03-29T09:54:28.436752+0000","last_undegraded":"2024-03-29T09:54:28.436752+0000","last_fullsized":"2024-03-29T09:54:28.436752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:39.673266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.474851+0000","last_change":"2024-03-29T09:54:26.925541+0000","last_active":"2024-03-29T09:54:27.474851+0000","last_peered":"2024-03-29T09:54:27.474851+0000","last_clean":"2024-03-29T09:54:27.474851+0000","last_became_active":"2024-03-29T09:54:26.925012+0000","last_became_peered":"2024-03-29T09:54:26.925012+0000","last_unstale":"2024-03-29T09:54:27.474851+0000","last_undegraded":"2024-03-29T09:54:27.474851+0000","last_fullsized":"2024-03-29T09:54:27.474851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674348+0000","last_change":"2024-03-29T09:54:27.470254+0000","last_active":"2024-03-29T09:54:28.674348+0000","last_peered":"2024-03-29T09:54:28.674348+0000","last_clean":"2024-03-29T09:54:28.674348+0000","last_became_active":"2024-03-29T09:54:27.468546+0000","last_became_peered":"2024-03-29T09:54:27.468546+0000","last_unstale":"2024-03-29T09:54:28.674348+0000","last_undegraded":"2024-03-29T09:54:28.674348+0000","last_fullsized":"2024-03-29T09:54:28.674348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:30.050849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.464632+0000","last_change":"2024-03-29T09:54:26.924638+0000","last_active":"2024-03-29T09:54:27.464632+0000","last_peered":"2024-03-29T09:54:27.464632+0000","last_clean":"2024-03-29T09:54:27.464632+0000","last_became_active":"2024-03-29T09:54:26.923692+0000","last_became_peered":"2024-03-29T09:54:26.923692+0000","last_unstale":"2024-03-29T09:54:27.464632+0000","last_undegraded":"2024-03-29T09:54:27.464632+0000","last_fullsized":"2024-03-29T09:54:27.464632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:49:24.339314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673965+0000","last_change":"2024-03-29T09:54:27.465512+0000","last_active":"2024-03-29T09:54:28.673965+0000","last_peered":"2024-03-29T09:54:28.673965+0000","last_clean":"2024-03-29T09:54:28.673965+0000","last_became_active":"2024-03-29T09:54:27.465258+0000","last_became_peered":"2024-03-29T09:54:27.465258+0000","last_unstale":"2024-03-29T09:54:28.673965+0000","last_undegraded":"2024-03-29T09:54:28.673965+0000","last_fullsized":"2024-03-29T09:54:28.673965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.437301+0000","last_change":"2024-03-29T09:54:26.922511+0000","last_active":"2024-03-29T09:54:27.437301+0000","last_peered":"2024-03-29T09:54:27.437301+0000","last_clean":"2024-03-29T09:54:27.437301+0000","last_became_active":"2024-03-29T09:54:26.922182+0000","last_became_peered":"2024-03-29T09:54:26.922182+0000","last_unstale":"2024-03-29T09:54:27.437301+0000","last_undegraded":"2024-03-29T09:54:27.437301+0000","last_fullsized":"2024-03-29T09:54:27.437301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674259+0000","last_change":"2024-03-29T09:54:27.472857+0000","last_active":"2024-03-29T09:54:28.674259+0000","last_peered":"2024-03-29T09:54:28.674259+0000","last_clean":"2024-03-29T09:54:28.674259+0000","last_became_active":"2024-03-29T09:54:27.472482+0000","last_became_peered":"2024-03-29T09:54:27.472482+0000","last_unstale":"2024-03-29T09:54:28.674259+0000","last_undegraded":"2024-03-29T09:54:28.674259+0000","last_fullsized":"2024-03-29T09:54:28.674259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:18.345522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.432778+0000","last_change":"2024-03-29T09:54:26.920268+0000","last_active":"2024-03-29T09:54:27.432778+0000","last_peered":"2024-03-29T09:54:27.432778+0000","last_clean":"2024-03-29T09:54:27.432778+0000","last_became_active":"2024-03-29T09:54:26.919956+0000","last_became_peered":"2024-03-29T09:54:26.919956+0000","last_unstale":"2024-03-29T09:54:27.432778+0000","last_undegraded":"2024-03-29T09:54:27.432778+0000","last_fullsized":"2024-03-29T09:54:27.432778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438219+0000","last_change":"2024-03-29T09:54:27.457604+0000","last_active":"2024-03-29T09:54:28.438219+0000","last_peered":"2024-03-29T09:54:28.438219+0000","last_clean":"2024-03-29T09:54:28.438219+0000","last_became_active":"2024-03-29T09:54:27.457197+0000","last_became_peered":"2024-03-29T09:54:27.457197+0000","last_unstale":"2024-03-29T09:54:28.438219+0000","last_undegraded":"2024-03-29T09:54:28.438219+0000","last_fullsized":"2024-03-29T09:54:28.438219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:25:15.310717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.464670+0000","last_change":"2024-03-29T09:54:26.921804+0000","last_active":"2024-03-29T09:54:27.464670+0000","last_peered":"2024-03-29T09:54:27.464670+0000","last_clean":"2024-03-29T09:54:27.464670+0000","last_became_active":"2024-03-29T09:54:26.921487+0000","last_became_peered":"2024-03-29T09:54:26.921487+0000","last_unstale":"2024-03-29T09:54:27.464670+0000","last_undegraded":"2024-03-29T09:54:27.464670+0000","last_fullsized":"2024-03-29T09:54:27.464670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674124+0000","last_change":"2024-03-29T09:54:27.470978+0000","last_active":"2024-03-29T09:54:28.674124+0000","last_peered":"2024-03-29T09:54:28.674124+0000","last_clean":"2024-03-29T09:54:28.674124+0000","last_became_active":"2024-03-29T09:54:27.470690+0000","last_became_peered":"2024-03-29T09:54:27.470690+0000","last_unstale":"2024-03-29T09:54:28.674124+0000","last_undegraded":"2024-03-29T09:54:28.674124+0000","last_fullsized":"2024-03-29T09:54:28.674124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:19.141206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.456285+0000","last_change":"2024-03-29T09:54:26.928327+0000","last_active":"2024-03-29T09:54:27.456285+0000","last_peered":"2024-03-29T09:54:27.456285+0000","last_clean":"2024-03-29T09:54:27.456285+0000","last_became_active":"2024-03-29T09:54:26.927938+0000","last_became_peered":"2024-03-29T09:54:26.927938+0000","last_unstale":"2024-03-29T09:54:27.456285+0000","last_undegraded":"2024-03-29T09:54:27.456285+0000","last_fullsized":"2024-03-29T09:54:27.456285+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:53.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674582+0000","last_change":"2024-03-29T09:54:27.466328+0000","last_active":"2024-03-29T09:54:28.674582+0000","last_peered":"2024-03-29T09:54:28.674582+0000","last_clean":"2024-03-29T09:54:28.674582+0000","last_became_active":"2024-03-29T09:54:27.465869+0000","last_became_peered":"2024-03-29T09:54:27.465869+0000","last_unstale":"2024-03-29T09:54:28.674582+0000","last_undegraded":"2024-03-29T09:54:28.674582+0000","last_fullsized":"2024-03-29T09:54:28.674582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:50.524833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.438039+0000","last_change":"2024-03-29T09:54:26.925276+0000","last_active":"2024-03-29T09:54:27.438039+0000","last_peered":"2024-03-29T09:54:27.438039+0000","last_clean":"2024-03-29T09:54:27.438039+0000","last_became_active":"2024-03-29T09:54:26.924980+0000","last_became_peered":"2024-03-29T09:54:26.924980+0000","last_unstale":"2024-03-29T09:54:27.438039+0000","last_undegraded":"2024-03-29T09:54:27.438039+0000","last_fullsized":"2024-03-29T09:54:27.438039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438030+0000","last_change":"2024-03-29T09:54:27.447713+0000","last_active":"2024-03-29T09:54:28.438030+0000","last_peered":"2024-03-29T09:54:28.438030+0000","last_clean":"2024-03-29T09:54:28.438030+0000","last_became_active":"2024-03-29T09:54:27.446552+0000","last_became_peered":"2024-03-29T09:54:27.446552+0000","last_unstale":"2024-03-29T09:54:28.438030+0000","last_undegraded":"2024-03-29T09:54:28.438030+0000","last_fullsized":"2024-03-29T09:54:28.438030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:31.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.437304+0000","last_change":"2024-03-29T09:54:26.911470+0000","last_active":"2024-03-29T09:54:27.437304+0000","last_peered":"2024-03-29T09:54:27.437304+0000","last_clean":"2024-03-29T09:54:27.437304+0000","last_became_active":"2024-03-29T09:54:26.911092+0000","last_became_peered":"2024-03-29T09:54:26.911092+0000","last_unstale":"2024-03-29T09:54:27.437304+0000","last_undegraded":"2024-03-29T09:54:27.437304+0000","last_fullsized":"2024-03-29T09:54:27.437304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436533+0000","last_change":"2024-03-29T09:54:27.459676+0000","last_active":"2024-03-29T09:54:28.436533+0000","last_peered":"2024-03-29T09:54:28.436533+0000","last_clean":"2024-03-29T09:54:28.436533+0000","last_became_active":"2024-03-29T09:54:27.459375+0000","last_became_peered":"2024-03-29T09:54:27.459375+0000","last_unstale":"2024-03-29T09:54:28.436533+0000","last_undegraded":"2024-03-29T09:54:28.436533+0000","last_fullsized":"2024-03-29T09:54:28.436533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:09.734996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572306+0000","last_change":"2024-03-29T09:54:27.446597+0000","last_active":"2024-03-29T09:54:28.572306+0000","last_peered":"2024-03-29T09:54:28.572306+0000","last_clean":"2024-03-29T09:54:28.572306+0000","last_became_active":"2024-03-29T09:54:27.444170+0000","last_became_peered":"2024-03-29T09:54:27.444170+0000","last_unstale":"2024-03-29T09:54:28.572306+0000","last_undegraded":"2024-03-29T09:54:28.572306+0000","last_fullsized":"2024-03-29T09:54:28.572306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.432316+0000","last_change":"2024-03-29T09:54:26.923022+0000","last_active":"2024-03-29T09:54:27.432316+0000","last_peered":"2024-03-29T09:54:27.432316+0000","last_clean":"2024-03-29T09:54:27.432316+0000","last_became_active":"2024-03-29T09:54:26.922558+0000","last_became_peered":"2024-03-29T09:54:26.922558+0000","last_unstale":"2024-03-29T09:54:27.432316+0000","last_undegraded":"2024-03-29T09:54:27.432316+0000","last_fullsized":"2024-03-29T09:54:27.432316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674355+0000","last_change":"2024-03-29T09:54:27.452598+0000","last_active":"2024-03-29T09:54:28.674355+0000","last_peered":"2024-03-29T09:54:28.674355+0000","last_clean":"2024-03-29T09:54:28.674355+0000","last_became_active":"2024-03-29T09:54:27.452331+0000","last_became_peered":"2024-03-29T09:54:27.452331+0000","last_unstale":"2024-03-29T09:54:28.674355+0000","last_undegraded":"2024-03-29T09:54:28.674355+0000","last_fullsized":"2024-03-29T09:54:28.674355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:50.524833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.433351+0000","last_change":"2024-03-29T09:54:26.916736+0000","last_active":"2024-03-29T09:54:27.433351+0000","last_peered":"2024-03-29T09:54:27.433351+0000","last_clean":"2024-03-29T09:54:27.433351+0000","last_became_active":"2024-03-29T09:54:26.916439+0000","last_became_peered":"2024-03-29T09:54:26.916439+0000","last_unstale":"2024-03-29T09:54:27.433351+0000","last_undegraded":"2024-03-29T09:54:27.433351+0000","last_fullsized":"2024-03-29T09:54:27.433351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.453076+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T09:54:27.453076+0000","last_peered":"2024-03-29T09:54:27.453076+0000","last_clean":"2024-03-29T09:54:27.453076+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T09:54:27.453076+0000","last_undegraded":"2024-03-29T09:54:27.453076+0000","last_fullsized":"2024-03-29T09:54:27.453076+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673776+0000","last_change":"2024-03-29T09:54:27.462932+0000","last_active":"2024-03-29T09:54:28.673776+0000","last_peered":"2024-03-29T09:54:28.673776+0000","last_clean":"2024-03-29T09:54:28.673776+0000","last_became_active":"2024-03-29T09:54:27.462665+0000","last_became_peered":"2024-03-29T09:54:27.462665+0000","last_unstale":"2024-03-29T09:54:28.673776+0000","last_undegraded":"2024-03-29T09:54:28.673776+0000","last_fullsized":"2024-03-29T09:54:28.673776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:02.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.453904+0000","last_change":"2024-03-29T09:54:26.924232+0000","last_active":"2024-03-29T09:54:27.453904+0000","last_peered":"2024-03-29T09:54:27.453904+0000","last_clean":"2024-03-29T09:54:27.453904+0000","last_became_active":"2024-03-29T09:54:26.923740+0000","last_became_peered":"2024-03-29T09:54:26.923740+0000","last_unstale":"2024-03-29T09:54:27.453904+0000","last_undegraded":"2024-03-29T09:54:27.453904+0000","last_fullsized":"2024-03-29T09:54:27.453904+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:53.973287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.457620+0000","last_change":"2024-03-29T09:54:26.929540+0000","last_active":"2024-03-29T09:54:27.457620+0000","last_peered":"2024-03-29T09:54:27.457620+0000","last_clean":"2024-03-29T09:54:27.457620+0000","last_became_active":"2024-03-29T09:54:26.928753+0000","last_became_peered":"2024-03-29T09:54:26.928753+0000","last_unstale":"2024-03-29T09:54:27.457620+0000","last_undegraded":"2024-03-29T09:54:27.457620+0000","last_fullsized":"2024-03-29T09:54:27.457620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:38.666641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572622+0000","last_change":"2024-03-29T09:54:27.449508+0000","last_active":"2024-03-29T09:54:28.572622+0000","last_peered":"2024-03-29T09:54:28.572622+0000","last_clean":"2024-03-29T09:54:28.572622+0000","last_became_active":"2024-03-29T09:54:27.449177+0000","last_became_peered":"2024-03-29T09:54:27.449177+0000","last_unstale":"2024-03-29T09:54:28.572622+0000","last_undegraded":"2024-03-29T09:54:28.572622+0000","last_fullsized":"2024-03-29T09:54:28.572622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572750+0000","last_change":"2024-03-29T09:54:26.909694+0000","last_active":"2024-03-29T09:54:28.572750+0000","last_peered":"2024-03-29T09:54:28.572750+0000","last_clean":"2024-03-29T09:54:28.572750+0000","last_became_active":"2024-03-29T09:54:26.909172+0000","last_became_peered":"2024-03-29T09:54:26.909172+0000","last_unstale":"2024-03-29T09:54:28.572750+0000","last_undegraded":"2024-03-29T09:54:28.572750+0000","last_fullsized":"2024-03-29T09:54:28.572750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:15:56.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674814+0000","last_change":"2024-03-29T09:54:27.466321+0000","last_active":"2024-03-29T09:54:28.674814+0000","last_peered":"2024-03-29T09:54:28.674814+0000","last_clean":"2024-03-29T09:54:28.674814+0000","last_became_active":"2024-03-29T09:54:27.465544+0000","last_became_peered":"2024-03-29T09:54:27.465544+0000","last_unstale":"2024-03-29T09:54:28.674814+0000","last_undegraded":"2024-03-29T09:54:28.674814+0000","last_fullsized":"2024-03-29T09:54:28.674814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:35.545393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.450346+0000","last_change":"2024-03-29T09:54:26.917641+0000","last_active":"2024-03-29T09:54:27.450346+0000","last_peered":"2024-03-29T09:54:27.450346+0000","last_clean":"2024-03-29T09:54:27.450346+0000","last_became_active":"2024-03-29T09:54:26.915808+0000","last_became_peered":"2024-03-29T09:54:26.915808+0000","last_unstale":"2024-03-29T09:54:27.450346+0000","last_undegraded":"2024-03-29T09:54:27.450346+0000","last_fullsized":"2024-03-29T09:54:27.450346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:57.119297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674304+0000","last_change":"2024-03-29T09:54:27.459303+0000","last_active":"2024-03-29T09:54:28.674304+0000","last_peered":"2024-03-29T09:54:28.674304+0000","last_clean":"2024-03-29T09:54:28.674304+0000","last_became_active":"2024-03-29T09:54:27.458902+0000","last_became_peered":"2024-03-29T09:54:27.458902+0000","last_unstale":"2024-03-29T09:54:28.674304+0000","last_undegraded":"2024-03-29T09:54:28.674304+0000","last_fullsized":"2024-03-29T09:54:28.674304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:25.651718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.454803+0000","last_change":"2024-03-29T09:54:26.929171+0000","last_active":"2024-03-29T09:54:27.454803+0000","last_peered":"2024-03-29T09:54:27.454803+0000","last_clean":"2024-03-29T09:54:27.454803+0000","last_became_active":"2024-03-29T09:54:26.928796+0000","last_became_peered":"2024-03-29T09:54:26.928796+0000","last_unstale":"2024-03-29T09:54:27.454803+0000","last_undegraded":"2024-03-29T09:54:27.454803+0000","last_fullsized":"2024-03-29T09:54:27.454803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.436200+0000","last_change":"2024-03-29T09:54:27.453529+0000","last_active":"2024-03-29T09:54:28.436200+0000","last_peered":"2024-03-29T09:54:28.436200+0000","last_clean":"2024-03-29T09:54:28.436200+0000","last_became_active":"2024-03-29T09:54:27.453153+0000","last_became_peered":"2024-03-29T09:54:27.453153+0000","last_unstale":"2024-03-29T09:54:28.436200+0000","last_undegraded":"2024-03-29T09:54:28.436200+0000","last_fullsized":"2024-03-29T09:54:28.436200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:25.068710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.438223+0000","last_change":"2024-03-29T09:54:26.916239+0000","last_active":"2024-03-29T09:54:27.438223+0000","last_peered":"2024-03-29T09:54:27.438223+0000","last_clean":"2024-03-29T09:54:27.438223+0000","last_became_active":"2024-03-29T09:54:26.915960+0000","last_became_peered":"2024-03-29T09:54:26.915960+0000","last_unstale":"2024-03-29T09:54:27.438223+0000","last_undegraded":"2024-03-29T09:54:27.438223+0000","last_fullsized":"2024-03-29T09:54:27.438223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:05.666298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.673647+0000","last_change":"2024-03-29T09:54:27.448090+0000","last_active":"2024-03-29T09:54:28.673647+0000","last_peered":"2024-03-29T09:54:28.673647+0000","last_clean":"2024-03-29T09:54:28.673647+0000","last_became_active":"2024-03-29T09:54:27.447614+0000","last_became_peered":"2024-03-29T09:54:27.447614+0000","last_unstale":"2024-03-29T09:54:28.673647+0000","last_undegraded":"2024-03-29T09:54:28.673647+0000","last_fullsized":"2024-03-29T09:54:28.673647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:25:15.310717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.438747+0000","last_change":"2024-03-29T09:54:26.919921+0000","last_active":"2024-03-29T09:54:27.438747+0000","last_peered":"2024-03-29T09:54:27.438747+0000","last_clean":"2024-03-29T09:54:27.438747+0000","last_became_active":"2024-03-29T09:54:26.919609+0000","last_became_peered":"2024-03-29T09:54:26.919609+0000","last_unstale":"2024-03-29T09:54:27.438747+0000","last_undegraded":"2024-03-29T09:54:27.438747+0000","last_fullsized":"2024-03-29T09:54:27.438747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:47:38.666641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674643+0000","last_change":"2024-03-29T09:54:27.458246+0000","last_active":"2024-03-29T09:54:28.674643+0000","last_peered":"2024-03-29T09:54:28.674643+0000","last_clean":"2024-03-29T09:54:28.674643+0000","last_became_active":"2024-03-29T09:54:27.457985+0000","last_became_peered":"2024-03-29T09:54:27.457985+0000","last_unstale":"2024-03-29T09:54:28.674643+0000","last_undegraded":"2024-03-29T09:54:28.674643+0000","last_fullsized":"2024-03-29T09:54:28.674643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:30.050849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.469210+0000","last_change":"2024-03-29T09:54:26.927045+0000","last_active":"2024-03-29T09:54:27.469210+0000","last_peered":"2024-03-29T09:54:27.469210+0000","last_clean":"2024-03-29T09:54:27.469210+0000","last_became_active":"2024-03-29T09:54:26.926750+0000","last_became_peered":"2024-03-29T09:54:26.926750+0000","last_unstale":"2024-03-29T09:54:27.469210+0000","last_undegraded":"2024-03-29T09:54:27.469210+0000","last_fullsized":"2024-03-29T09:54:27.469210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:15:56.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437955+0000","last_change":"2024-03-29T09:54:27.447647+0000","last_active":"2024-03-29T09:54:28.437955+0000","last_peered":"2024-03-29T09:54:28.437955+0000","last_clean":"2024-03-29T09:54:28.437955+0000","last_became_active":"2024-03-29T09:54:27.447371+0000","last_became_peered":"2024-03-29T09:54:27.447371+0000","last_unstale":"2024-03-29T09:54:28.437955+0000","last_undegraded":"2024-03-29T09:54:28.437955+0000","last_fullsized":"2024-03-29T09:54:28.437955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:39.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.432327+0000","last_change":"2024-03-29T09:54:26.929260+0000","last_active":"2024-03-29T09:54:27.432327+0000","last_peered":"2024-03-29T09:54:27.432327+0000","last_clean":"2024-03-29T09:54:27.432327+0000","last_became_active":"2024-03-29T09:54:26.928909+0000","last_became_peered":"2024-03-29T09:54:26.928909+0000","last_unstale":"2024-03-29T09:54:27.432327+0000","last_undegraded":"2024-03-29T09:54:27.432327+0000","last_fullsized":"2024-03-29T09:54:27.432327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:37:17.462538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674364+0000","last_change":"2024-03-29T09:54:27.471604+0000","last_active":"2024-03-29T09:54:28.674364+0000","last_peered":"2024-03-29T09:54:28.674364+0000","last_clean":"2024-03-29T09:54:28.674364+0000","last_became_active":"2024-03-29T09:54:27.471197+0000","last_became_peered":"2024-03-29T09:54:27.471197+0000","last_unstale":"2024-03-29T09:54:28.674364+0000","last_undegraded":"2024-03-29T09:54:28.674364+0000","last_fullsized":"2024-03-29T09:54:28.674364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:25.712202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.463137+0000","last_change":"2024-03-29T09:54:26.916668+0000","last_active":"2024-03-29T09:54:27.463137+0000","last_peered":"2024-03-29T09:54:27.463137+0000","last_clean":"2024-03-29T09:54:27.463137+0000","last_became_active":"2024-03-29T09:54:26.916233+0000","last_became_peered":"2024-03-29T09:54:26.916233+0000","last_unstale":"2024-03-29T09:54:27.463137+0000","last_undegraded":"2024-03-29T09:54:27.463137+0000","last_fullsized":"2024-03-29T09:54:27.463137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438328+0000","last_change":"2024-03-29T09:54:27.444803+0000","last_active":"2024-03-29T09:54:28.438328+0000","last_peered":"2024-03-29T09:54:28.438328+0000","last_clean":"2024-03-29T09:54:28.438328+0000","last_became_active":"2024-03-29T09:54:27.444104+0000","last_became_peered":"2024-03-29T09:54:27.444104+0000","last_unstale":"2024-03-29T09:54:28.438328+0000","last_undegraded":"2024-03-29T09:54:28.438328+0000","last_fullsized":"2024-03-29T09:54:28.438328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:25.712202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-29T09:54:27.464351+0000","last_change":"2024-03-29T09:54:26.923513+0000","last_active":"2024-03-29T09:54:27.464351+0000","last_peered":"2024-03-29T09:54:27.464351+0000","last_clean":"2024-03-29T09:54:27.464351+0000","last_became_active":"2024-03-29T09:54:26.922870+0000","last_became_peered":"2024-03-29T09:54:26.922870+0000","last_unstale":"2024-03-29T09:54:27.464351+0000","last_undegraded":"2024-03-29T09:54:27.464351+0000","last_fullsized":"2024-03-29T09:54:27.464351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:25.881798+0000","last_clean_scrub_stamp":"2024-03-29T09:54:25.881798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:08.728371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674452+0000","last_change":"2024-03-29T09:54:27.471588+0000","last_active":"2024-03-29T09:54:28.674452+0000","last_peered":"2024-03-29T09:54:28.674452+0000","last_clean":"2024-03-29T09:54:28.674452+0000","last_became_active":"2024-03-29T09:54:27.468491+0000","last_became_peered":"2024-03-29T09:54:27.468491+0000","last_unstale":"2024-03-29T09:54:28.674452+0000","last_undegraded":"2024-03-29T09:54:28.674452+0000","last_fullsized":"2024-03-29T09:54:28.674452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:31.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438168+0000","last_change":"2024-03-29T09:54:27.461434+0000","last_active":"2024-03-29T09:54:28.438168+0000","last_peered":"2024-03-29T09:54:28.438168+0000","last_clean":"2024-03-29T09:54:28.438168+0000","last_became_active":"2024-03-29T09:54:27.461095+0000","last_became_peered":"2024-03-29T09:54:27.461095+0000","last_unstale":"2024-03-29T09:54:28.438168+0000","last_undegraded":"2024-03-29T09:54:28.438168+0000","last_fullsized":"2024-03-29T09:54:28.438168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:14:02.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.511270+0000","last_change":"2024-03-29T09:54:27.445834+0000","last_active":"2024-03-29T09:54:28.511270+0000","last_peered":"2024-03-29T09:54:28.511270+0000","last_clean":"2024-03-29T09:54:28.511270+0000","last_became_active":"2024-03-29T09:54:27.444597+0000","last_became_peered":"2024-03-29T09:54:27.444597+0000","last_unstale":"2024-03-29T09:54:28.511270+0000","last_undegraded":"2024-03-29T09:54:28.511270+0000","last_fullsized":"2024-03-29T09:54:28.511270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:36.310545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437472+0000","last_change":"2024-03-29T09:54:27.449523+0000","last_active":"2024-03-29T09:54:28.437472+0000","last_peered":"2024-03-29T09:54:28.437472+0000","last_clean":"2024-03-29T09:54:28.437472+0000","last_became_active":"2024-03-29T09:54:27.447299+0000","last_became_peered":"2024-03-29T09:54:27.447299+0000","last_unstale":"2024-03-29T09:54:28.437472+0000","last_undegraded":"2024-03-29T09:54:28.437472+0000","last_fullsized":"2024-03-29T09:54:28.437472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:18.345522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674066+0000","last_change":"2024-03-29T09:54:27.464458+0000","last_active":"2024-03-29T09:54:28.674066+0000","last_peered":"2024-03-29T09:54:28.674066+0000","last_clean":"2024-03-29T09:54:28.674066+0000","last_became_active":"2024-03-29T09:54:27.464146+0000","last_became_peered":"2024-03-29T09:54:27.464146+0000","last_unstale":"2024-03-29T09:54:28.674066+0000","last_undegraded":"2024-03-29T09:54:28.674066+0000","last_fullsized":"2024-03-29T09:54:28.674066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674683+0000","last_change":"2024-03-29T09:54:27.463860+0000","last_active":"2024-03-29T09:54:28.674683+0000","last_peered":"2024-03-29T09:54:28.674683+0000","last_clean":"2024-03-29T09:54:28.674683+0000","last_became_active":"2024-03-29T09:54:27.461514+0000","last_became_peered":"2024-03-29T09:54:27.461514+0000","last_unstale":"2024-03-29T09:54:28.674683+0000","last_undegraded":"2024-03-29T09:54:28.674683+0000","last_fullsized":"2024-03-29T09:54:28.674683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.572614+0000","last_change":"2024-03-29T09:54:27.457584+0000","last_active":"2024-03-29T09:54:28.572614+0000","last_peered":"2024-03-29T09:54:28.572614+0000","last_clean":"2024-03-29T09:54:28.572614+0000","last_became_active":"2024-03-29T09:54:27.457325+0000","last_became_peered":"2024-03-29T09:54:27.457325+0000","last_unstale":"2024-03-29T09:54:28.572614+0000","last_undegraded":"2024-03-29T09:54:28.572614+0000","last_fullsized":"2024-03-29T09:54:28.572614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.439001+0000","last_change":"2024-03-29T09:54:27.467951+0000","last_active":"2024-03-29T09:54:28.439001+0000","last_peered":"2024-03-29T09:54:28.439001+0000","last_clean":"2024-03-29T09:54:28.439001+0000","last_became_active":"2024-03-29T09:54:27.466080+0000","last_became_peered":"2024-03-29T09:54:27.466080+0000","last_unstale":"2024-03-29T09:54:28.439001+0000","last_undegraded":"2024-03-29T09:54:28.439001+0000","last_fullsized":"2024-03-29T09:54:28.439001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674709+0000","last_change":"2024-03-29T09:54:27.460513+0000","last_active":"2024-03-29T09:54:28.674709+0000","last_peered":"2024-03-29T09:54:28.674709+0000","last_clean":"2024-03-29T09:54:28.674709+0000","last_became_active":"2024-03-29T09:54:27.460231+0000","last_became_peered":"2024-03-29T09:54:27.460231+0000","last_unstale":"2024-03-29T09:54:28.674709+0000","last_undegraded":"2024-03-29T09:54:28.674709+0000","last_fullsized":"2024-03-29T09:54:28.674709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.746737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437313+0000","last_change":"2024-03-29T09:54:27.462171+0000","last_active":"2024-03-29T09:54:28.437313+0000","last_peered":"2024-03-29T09:54:28.437313+0000","last_clean":"2024-03-29T09:54:28.437313+0000","last_became_active":"2024-03-29T09:54:27.461066+0000","last_became_peered":"2024-03-29T09:54:27.461066+0000","last_unstale":"2024-03-29T09:54:28.437313+0000","last_undegraded":"2024-03-29T09:54:28.437313+0000","last_fullsized":"2024-03-29T09:54:28.437313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:44:20.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674108+0000","last_change":"2024-03-29T09:54:27.456049+0000","last_active":"2024-03-29T09:54:28.674108+0000","last_peered":"2024-03-29T09:54:28.674108+0000","last_clean":"2024-03-29T09:54:28.674108+0000","last_became_active":"2024-03-29T09:54:27.454521+0000","last_became_peered":"2024-03-29T09:54:27.454521+0000","last_unstale":"2024-03-29T09:54:28.674108+0000","last_undegraded":"2024-03-29T09:54:28.674108+0000","last_fullsized":"2024-03-29T09:54:28.674108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.437418+0000","last_change":"2024-03-29T09:54:27.465472+0000","last_active":"2024-03-29T09:54:28.437418+0000","last_peered":"2024-03-29T09:54:28.437418+0000","last_clean":"2024-03-29T09:54:28.437418+0000","last_became_active":"2024-03-29T09:54:27.465180+0000","last_became_peered":"2024-03-29T09:54:27.465180+0000","last_unstale":"2024-03-29T09:54:28.437418+0000","last_undegraded":"2024-03-29T09:54:28.437418+0000","last_fullsized":"2024-03-29T09:54:28.437418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:14.464634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.674044+0000","last_change":"2024-03-29T09:54:27.466113+0000","last_active":"2024-03-29T09:54:28.674044+0000","last_peered":"2024-03-29T09:54:28.674044+0000","last_clean":"2024-03-29T09:54:28.674044+0000","last_became_active":"2024-03-29T09:54:27.465912+0000","last_became_peered":"2024-03-29T09:54:27.465912+0000","last_unstale":"2024-03-29T09:54:28.674044+0000","last_undegraded":"2024-03-29T09:54:28.674044+0000","last_fullsized":"2024-03-29T09:54:28.674044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-29T09:54:28.438778+0000","last_change":"2024-03-29T09:54:27.447415+0000","last_active":"2024-03-29T09:54:28.438778+0000","last_peered":"2024-03-29T09:54:28.438778+0000","last_clean":"2024-03-29T09:54:28.438778+0000","last_became_active":"2024-03-29T09:54:27.446985+0000","last_became_peered":"2024-03-29T09:54:27.446985+0000","last_unstale":"2024-03-29T09:54:28.438778+0000","last_undegraded":"2024-03-29T09:54:28.438778+0000","last_fullsized":"2024-03-29T09:54:28.438778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:26.888423+0000","last_clean_scrub_stamp":"2024-03-29T09:54:26.888423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:10:32.184687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.252}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":806912,"data_stored":660885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T09:54:38.679 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-29T09:54:38.679 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-29T09:54:38.679 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-29T09:54:38.680 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-29T09:54:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.910+0000 7f55bf000700 1 Processor -- start 2024-03-29T09:54:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.911+0000 7f55bf000700 1 -- start start 2024-03-29T09:54:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.912+0000 7f55bf000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f55b80ceec0 0x7f55b81612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.912+0000 7f55bf000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b8161c50 con 0x7f55b8161800 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.912+0000 7f55bf000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b80d30d0 con 0x7f55b815d640 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.912+0000 7f55bf000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b80b6840 con 0x7f55b80ceec0 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.912+0000 7f55bd000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f55b8161800 0x7f55b80b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54786/0 (socket says 172.21.15.12:54786) 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.912+0000 7f55bd000700 1 -- 172.21.15.12:0/2035059077 learned_addr learned my addr 172.21.15.12:0/2035059077 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.912+0000 7f55b7400700 1 --2- 172.21.15.12:0/2035059077 >> v2:172.21.15.159:3300/0 conn(0x7f55b80ceec0 0x7f55b81612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.912+0000 7f55b6a00700 1 -- 172.21.15.12:0/2035059077 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122888050 0 0) 0x7f55b0001490 con 0x7f55b8161800 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.912+0000 7f55b6a00700 1 -- 172.21.15.12:0/2035059077 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c0035b0 con 0x7f55b8161800 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.913+0000 7f55b7400700 1 -- 172.21.15.12:0/2035059077 >> v1:172.21.15.159:6789/0 conn(0x7f55b815d640 legacy=0x7f55b80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.913+0000 7f55b7400700 1 -- 172.21.15.12:0/2035059077 >> v1:172.21.15.12:6789/0 conn(0x7f55b8161800 legacy=0x7f55b80b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.913+0000 7f55b7400700 1 -- 172.21.15.12:0/2035059077 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b80d2b30 con 0x7f55b80ceec0 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.913+0000 7f55b7400700 1 --2- 172.21.15.12:0/2035059077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55b80ceec0 0x7f55b81612c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009840 tx=0x7f55a4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dc5df907c9a4385 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.913+0000 7f55b6a00700 1 -- 172.21.15.12:0/2035059077 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4011830 con 0x7f55b80ceec0 2024-03-29T09:54:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.913+0000 7f55b6a00700 1 -- 172.21.15.12:0/2035059077 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a4011990 con 0x7f55b80ceec0 2024-03-29T09:54:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.913+0000 7f55b6a00700 1 -- 172.21.15.12:0/2035059077 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4011d20 con 0x7f55b80ceec0 2024-03-29T09:54:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.914+0000 7f55bf000700 1 -- 172.21.15.12:0/2035059077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55b80ceec0 msgr2=0x7f55b81612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.914+0000 7f55bf000700 1 --2- 172.21.15.12:0/2035059077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55b80ceec0 0x7f55b81612c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009840 tx=0x7f55a4009b50 comp rx=0 tx=0).stop 2024-03-29T09:54:38.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.914+0000 7f55bf000700 1 -- 172.21.15.12:0/2035059077 shutdown_connections 2024-03-29T09:54:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.914+0000 7f55bf000700 1 --2- 172.21.15.12:0/2035059077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55b80ceec0 0x7f55b81612c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.914+0000 7f55bf000700 1 -- 172.21.15.12:0/2035059077 >> 172.21.15.12:0/2035059077 conn(0x7f55b808aff0 msgr2=0x7f55b80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.914+0000 7f55bf000700 1 -- 172.21.15.12:0/2035059077 shutdown_connections 2024-03-29T09:54:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.914+0000 7f55bf000700 1 -- 172.21.15.12:0/2035059077 wait complete. 2024-03-29T09:54:38.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.916+0000 7f55bf000700 1 Processor -- start 2024-03-29T09:54:38.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.916+0000 7f55bf000700 1 -- start start 2024-03-29T09:54:38.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bf000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55b80ceec0 0x7f55b81f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bf000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55b815d640 0x7f55b80cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bf000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55b8161800 0x7f55b80cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bf000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b81f3dd0 con 0x7f55b8161800 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bf000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80ce5b0 con 0x7f55b80ceec0 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bf000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80cd9a0 con 0x7f55b815d640 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55b8161800 0x7f55b80cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55b8161800 0x7f55b80cb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60574/0 (socket says 172.21.15.12:60574) 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bd000700 1 -- 172.21.15.12:0/2209502977 learned_addr learned my addr 172.21.15.12:0/2209502977 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55b7400700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55b815d640 0x7f55b80cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55b7e00700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55b80ceec0 0x7f55b81f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bd000700 1 -- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55b815d640 msgr2=0x7f55b80cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bd000700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55b815d640 0x7f55b80cd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bd000700 1 -- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55b80ceec0 msgr2=0x7f55b81f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bd000700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55b80ceec0 0x7f55b81f3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.917+0000 7f55bd000700 1 -- 172.21.15.12:0/2209502977 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a40094a0 con 0x7f55b8161800 2024-03-29T09:54:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.918+0000 7f55bd000700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55b8161800 0x7f55b80cb3d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f55b000cb40 tx=0x7f55b000ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.918+0000 7f55afe00700 1 -- 172.21.15.12:0/2209502977 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b000a9e0 con 0x7f55b8161800 2024-03-29T09:54:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.918+0000 7f55afe00700 1 -- 172.21.15.12:0/2209502977 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b000ab40 con 0x7f55b8161800 2024-03-29T09:54:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.918+0000 7f55bf000700 1 -- 172.21.15.12:0/2209502977 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b80cdc80 con 0x7f55b8161800 2024-03-29T09:54:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.918+0000 7f55afe00700 1 -- 172.21.15.12:0/2209502977 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b0012400 con 0x7f55b8161800 2024-03-29T09:54:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.918+0000 7f55bf000700 1 -- 172.21.15.12:0/2209502977 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b80ce240 con 0x7f55b8161800 2024-03-29T09:54:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.919+0000 7f55bf000700 1 -- 172.21.15.12:0/2209502977 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b80cd710 con 0x7f55b8161800 2024-03-29T09:54:38.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.920+0000 7f55afe00700 1 -- 172.21.15.12:0/2209502977 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f55b001b3b0 con 0x7f55b8161800 2024-03-29T09:54:38.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.921+0000 7f55afe00700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f559806ac90 0x7f559806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:38.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.922+0000 7f55afe00700 1 -- 172.21.15.12:0/2209502977 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55b0088d80 con 0x7f55b8161800 2024-03-29T09:54:38.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.922+0000 7f55b7e00700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f559806ac90 0x7f559806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.923+0000 7f55b7e00700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f559806ac90 0x7f559806d150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f55a8000f60 tx=0x7f55a8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:38.926+0000 7f55afe00700 1 -- 172.21.15.12:0/2209502977 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55b0053f70 con 0x7f55b8161800 2024-03-29T09:54:39.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.185+0000 7f55bf000700 1 -- 172.21.15.12:0/2209502977 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f55b8161c50 con 0x7f55b8161800 2024-03-29T09:54:39.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.186+0000 7f55afe00700 1 -- 172.21.15.12:0/2209502977 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f55b0017040 con 0x7f55b8161800 2024-03-29T09:54:39.185 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:39.185 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-29T09:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 -- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f559806ac90 msgr2=0x7f559806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f559806ac90 0x7f559806d150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f55a8000f60 tx=0x7f55a8009040 comp rx=0 tx=0).stop 2024-03-29T09:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 -- 172.21.15.12:0/2209502977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55b8161800 msgr2=0x7f55b80cb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55b8161800 0x7f55b80cb3d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f55b000cb40 tx=0x7f55b000ce50 comp rx=0 tx=0).stop 2024-03-29T09:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 -- 172.21.15.12:0/2209502977 shutdown_connections 2024-03-29T09:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f559806ac90 0x7f559806d150 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55b8161800 0x7f55b80cb3d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55b80ceec0 0x7f55b81f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 --2- 172.21.15.12:0/2209502977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55b815d640 0x7f55b80cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 -- 172.21.15.12:0/2209502977 >> 172.21.15.12:0/2209502977 conn(0x7f55b808aff0 msgr2=0x7f55b80b8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:39.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 -- 172.21.15.12:0/2209502977 shutdown_connections 2024-03-29T09:54:39.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.188+0000 7f55bf000700 1 -- 172.21.15.12:0/2209502977 wait complete. 2024-03-29T09:54:39.206 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-29T09:54:39.207 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.404+0000 7f2586400700 1 Processor -- start 2024-03-29T09:54:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.405+0000 7f2586400700 1 -- start start 2024-03-29T09:54:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.406+0000 7f2586400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 0x7f25800d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.406+0000 7f2586400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2580161590 0x7f2580161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.406+0000 7f2586400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25800ceda0 con 0x7f25800cf230 2024-03-29T09:54:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.406+0000 7f2586400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2580161f80 con 0x7f2580161590 2024-03-29T09:54:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.406+0000 7f2586400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25801612f0 con 0x7f2580160f10 2024-03-29T09:54:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.406+0000 7f257fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2580161590 0x7f2580161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.406+0000 7f257fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2580161590 0x7f2580161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60580/0 (socket says 172.21.15.12:60580) 2024-03-29T09:54:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.406+0000 7f257fe00700 1 -- 172.21.15.12:0/2559802292 learned_addr learned my addr 172.21.15.12:0/2559802292 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.406+0000 7f257f400700 1 --2- 172.21.15.12:0/2559802292 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 0x7f25800d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.407+0000 7f257fe00700 1 -- 172.21.15.12:0/2559802292 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 msgr2=0x7f25800d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.407+0000 7f257fe00700 1 --2- 172.21.15.12:0/2559802292 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 0x7f25800d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.407+0000 7f257fe00700 1 -- 172.21.15.12:0/2559802292 >> v1:172.21.15.159:6789/0 conn(0x7f25800cf230 legacy=0x7f25800d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.407+0000 7f257fe00700 1 -- 172.21.15.12:0/2559802292 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25800ce470 con 0x7f2580161590 2024-03-29T09:54:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.407+0000 7f257fe00700 1 --2- 172.21.15.12:0/2559802292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 0x7f2580161a40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2578009e80 tx=0x7f257800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282dc525091d9835 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.407+0000 7f257e000700 1 -- 172.21.15.12:0/2559802292 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257800e070 con 0x7f2580161590 2024-03-29T09:54:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.407+0000 7f257e000700 1 -- 172.21.15.12:0/2559802292 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257800b8d0 con 0x7f2580161590 2024-03-29T09:54:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.407+0000 7f257e000700 1 -- 172.21.15.12:0/2559802292 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257800bc80 con 0x7f2580161590 2024-03-29T09:54:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.408+0000 7f2586400700 1 -- 172.21.15.12:0/2559802292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 msgr2=0x7f2580161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.408+0000 7f2586400700 1 --2- 172.21.15.12:0/2559802292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 0x7f2580161a40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2578009e80 tx=0x7f257800b330 comp rx=0 tx=0).stop 2024-03-29T09:54:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.408+0000 7f2586400700 1 -- 172.21.15.12:0/2559802292 shutdown_connections 2024-03-29T09:54:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.408+0000 7f2586400700 1 --2- 172.21.15.12:0/2559802292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 0x7f2580161a40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.408+0000 7f2586400700 1 --2- 172.21.15.12:0/2559802292 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 0x7f25800d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.408+0000 7f2586400700 1 -- 172.21.15.12:0/2559802292 >> 172.21.15.12:0/2559802292 conn(0x7f258008aff0 msgr2=0x7f25800d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.408+0000 7f2586400700 1 -- 172.21.15.12:0/2559802292 shutdown_connections 2024-03-29T09:54:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.408+0000 7f2586400700 1 -- 172.21.15.12:0/2559802292 wait complete. 2024-03-29T09:54:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.410+0000 7f2586400700 1 Processor -- start 2024-03-29T09:54:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.410+0000 7f2586400700 1 -- start start 2024-03-29T09:54:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f2586400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25800cf230 0x7f2580158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f2586400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 0x7f25801e7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f2586400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 0x7f25801e8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f2586400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2580158b60 con 0x7f2580161590 2024-03-29T09:54:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f2586400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25800b75a0 con 0x7f25800cf230 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f2586400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25800b6f90 con 0x7f2580160f10 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 0x7f25801e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 0x7f25801e8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60582/0 (socket says 172.21.15.12:60582) 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257fe00700 1 -- 172.21.15.12:0/575879298 learned_addr learned my addr 172.21.15.12:0/575879298 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257ea00700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 0x7f25801e7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257f400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25800cf230 0x7f2580158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257fe00700 1 -- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 msgr2=0x7f25801e7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257fe00700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 0x7f25801e7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257fe00700 1 -- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25800cf230 msgr2=0x7f2580158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257fe00700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25800cf230 0x7f2580158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257fe00700 1 -- 172.21.15.12:0/575879298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2578009b30 con 0x7f2580161590 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257ea00700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 0x7f25801e7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.412+0000 7f257f400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25800cf230 0x7f2580158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.413+0000 7f257fe00700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 0x7f25801e8360 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f258008c570 tx=0x7f2578004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:39.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.413+0000 7f2577400700 1 -- 172.21.15.12:0/575879298 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25780092e0 con 0x7f2580161590 2024-03-29T09:54:39.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.413+0000 7f2577400700 1 -- 172.21.15.12:0/575879298 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2578004960 con 0x7f2580161590 2024-03-29T09:54:39.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.413+0000 7f2577400700 1 -- 172.21.15.12:0/575879298 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257801c680 con 0x7f2580161590 2024-03-29T09:54:39.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.413+0000 7f2586400700 1 -- 172.21.15.12:0/575879298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258015fdc0 con 0x7f2580161590 2024-03-29T09:54:39.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.413+0000 7f2586400700 1 -- 172.21.15.12:0/575879298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25800b7230 con 0x7f2580161590 2024-03-29T09:54:39.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.414+0000 7f2586400700 1 -- 172.21.15.12:0/575879298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2580067720 con 0x7f2580161590 2024-03-29T09:54:39.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.415+0000 7f2577400700 1 -- 172.21.15.12:0/575879298 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f257801c7e0 con 0x7f2580161590 2024-03-29T09:54:39.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.416+0000 7f2577400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f256006ac90 0x7f256006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.416+0000 7f2577400700 1 -- 172.21.15.12:0/575879298 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f257808ad50 con 0x7f2580161590 2024-03-29T09:54:39.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.416+0000 7f257f400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f256006ac90 0x7f256006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.417+0000 7f257f400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f256006ac90 0x7f256006d150 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2570009ac0 tx=0x7f2570008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.420+0000 7f2577400700 1 -- 172.21.15.12:0/575879298 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2578055f20 con 0x7f2580161590 2024-03-29T09:54:39.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.634+0000 7f2586400700 1 -- 172.21.15.12:0/575879298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25800b76e0 con 0x7f2580161590 2024-03-29T09:54:39.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.636+0000 7f2577400700 1 -- 172.21.15.12:0/575879298 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f2578087ec0 con 0x7f2580161590 2024-03-29T09:54:39.635 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:39.635 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:39.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 -- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f256006ac90 msgr2=0x7f256006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f256006ac90 0x7f256006d150 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2570009ac0 tx=0x7f2570008040 comp rx=0 tx=0).stop 2024-03-29T09:54:39.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 -- 172.21.15.12:0/575879298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 msgr2=0x7f25801e8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 0x7f25801e8360 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f258008c570 tx=0x7f2578004ad0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 -- 172.21.15.12:0/575879298 shutdown_connections 2024-03-29T09:54:39.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f256006ac90 0x7f256006d150 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2580161590 0x7f25801e8360 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25800cf230 0x7f2580158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 --2- 172.21.15.12:0/575879298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2580160f10 0x7f25801e7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 -- 172.21.15.12:0/575879298 >> 172.21.15.12:0/575879298 conn(0x7f258008aff0 msgr2=0x7f258006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:39.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 -- 172.21.15.12:0/575879298 shutdown_connections 2024-03-29T09:54:39.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.638+0000 7f2586400700 1 -- 172.21.15.12:0/575879298 wait complete. 2024-03-29T09:54:39.656 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:54:39.656 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:54:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.857+0000 7f8a91000700 1 Processor -- start 2024-03-29T09:54:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.857+0000 7f8a91000700 1 -- start start 2024-03-29T09:54:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.858+0000 7f8a91000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8a8c0d2540 0x7f8a8c1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.858+0000 7f8a91000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a8c161420 con 0x7f8a8c0d16a0 2024-03-29T09:54:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.858+0000 7f8a91000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a8c161700 con 0x7f8a8c15d9d0 2024-03-29T09:54:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.858+0000 7f8a91000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8c0d4340 con 0x7f8a8c0d2540 2024-03-29T09:54:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a89400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8a8c0d16a0 0x7f8a8c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54794/0 (socket says 172.21.15.12:54794) 2024-03-29T09:54:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a89400700 1 -- 172.21.15.12:0/3706799963 learned_addr learned my addr 172.21.15.12:0/3706799963 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a8a800700 1 --2- 172.21.15.12:0/3706799963 >> v2:172.21.15.159:3300/0 conn(0x7f8a8c0d2540 0x7f8a8c1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222979474 0 0) 0x7f8a8c161420 con 0x7f8a8c0d16a0 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a700035b0 con 0x7f8a8c0d16a0 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000636084 0 0) 0x7f8a8c161700 con 0x7f8a8c15d9d0 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a8c161420 con 0x7f8a8c15d9d0 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 572597151 0 0) 0x7f8a700035b0 con 0x7f8a8c0d16a0 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 >> v2:172.21.15.159:3300/0 conn(0x7f8a8c0d2540 msgr2=0x7f8a8c1687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a88a00700 1 --2- 172.21.15.12:0/3706799963 >> v2:172.21.15.159:3300/0 conn(0x7f8a8c0d2540 0x7f8a8c1687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 >> v1:172.21.15.159:6789/0 conn(0x7f8a8c15d9d0 legacy=0x7f8a8c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c1619e0 con 0x7f8a8c0d16a0 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.859+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f8a78005120 con 0x7f8a8c0d16a0 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a8a800700 1 --2- 172.21.15.12:0/3706799963 >> v2:172.21.15.159:3300/0 conn(0x7f8a8c0d2540 0x7f8a8c1687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 >> v1:172.21.15.12:6789/0 conn(0x7f8a8c0d16a0 legacy=0x7f8a8c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a88a00700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a70003be0 0x7f8a70003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a88a00700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a70004850 0x7f8a70004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a88a00700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a70005260 0x7f8a7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7000c160 con 0x7f8a70003be0 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7000c2d0 con 0x7f8a70004850 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7000c490 con 0x7f8a70005260 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a89e00700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a70003be0 0x7f8a70003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a89400700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a70004850 0x7f8a70004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a8a800700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a70005260 0x7f8a7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a89e00700 1 -- 172.21.15.12:0/3706799963 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a70005260 msgr2=0x7f8a7000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a89e00700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a70005260 0x7f8a7000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a89e00700 1 -- 172.21.15.12:0/3706799963 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a70004850 msgr2=0x7f8a70004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a89e00700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a70004850 0x7f8a70004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a89e00700 1 -- 172.21.15.12:0/3706799963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a700035b0 con 0x7f8a70003be0 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.860+0000 7f8a8a800700 1 -- 172.21.15.12:0/3706799963 reap_dead start 2024-03-29T09:54:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.861+0000 7f8a89e00700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a70003be0 0x7f8a70003fe0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c001940 tx=0x7f8a70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150ac0dd95cd6fc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.861+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c001430 con 0x7f8a70003be0 2024-03-29T09:54:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.861+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7c001d70 con 0x7f8a70003be0 2024-03-29T09:54:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.861+0000 7f8a88a00700 1 -- 172.21.15.12:0/3706799963 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c005510 con 0x7f8a70003be0 2024-03-29T09:54:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.861+0000 7f8a91000700 1 -- 172.21.15.12:0/3706799963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a70003be0 msgr2=0x7f8a70003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.861+0000 7f8a91000700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a70003be0 0x7f8a70003fe0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c001940 tx=0x7f8a70005880 comp rx=0 tx=0).stop 2024-03-29T09:54:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.861+0000 7f8a91000700 1 -- 172.21.15.12:0/3706799963 shutdown_connections 2024-03-29T09:54:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.861+0000 7f8a91000700 1 --2- 172.21.15.12:0/3706799963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a70003be0 0x7f8a70003fe0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.861+0000 7f8a91000700 1 -- 172.21.15.12:0/3706799963 >> 172.21.15.12:0/3706799963 conn(0x7f8a8c08aff0 msgr2=0x7f8a8c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.861+0000 7f8a91000700 1 -- 172.21.15.12:0/3706799963 shutdown_connections 2024-03-29T09:54:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.862+0000 7f8a91000700 1 -- 172.21.15.12:0/3706799963 wait complete. 2024-03-29T09:54:39.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.863+0000 7f8a91000700 1 Processor -- start 2024-03-29T09:54:39.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.863+0000 7f8a91000700 1 -- start start 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.864+0000 7f8a91000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a70003be0 0x7f8a8c158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.864+0000 7f8a91000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a8c15eaf0 0x7f8a8c0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.864+0000 7f8a91000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a8c0cb5f0 0x7f8a8c0b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.864+0000 7f8a91000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c15ef50 con 0x7f8a8c0cb5f0 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.864+0000 7f8a91000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c0cbaf0 con 0x7f8a8c15eaf0 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.864+0000 7f8a91000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c15f1a0 con 0x7f8a70003be0 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.864+0000 7f8a8a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a8c0cb5f0 0x7f8a8c0b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.864+0000 7f8a8a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a8c0cb5f0 0x7f8a8c0b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60602/0 (socket says 172.21.15.12:60602) 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.864+0000 7f8a8a800700 1 -- 172.21.15.12:0/2192736323 learned_addr learned my addr 172.21.15.12:0/2192736323 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a89e00700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a70003be0 0x7f8a8c158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a89400700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a8c15eaf0 0x7f8a8c0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a8a800700 1 -- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a70003be0 msgr2=0x7f8a8c158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a8a800700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a70003be0 0x7f8a8c158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a8a800700 1 -- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a8c15eaf0 msgr2=0x7f8a8c0cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a8a800700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a8c15eaf0 0x7f8a8c0cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a8a800700 1 -- 172.21.15.12:0/2192736323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c15f480 con 0x7f8a8c0cb5f0 2024-03-29T09:54:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a8a800700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a8c0cb5f0 0x7f8a8c0b4860 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0d2f40 tx=0x7f8a84009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a82000700 1 -- 172.21.15.12:0/2192736323 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8400e070 con 0x7f8a8c0cb5f0 2024-03-29T09:54:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a82000700 1 -- 172.21.15.12:0/2192736323 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a84003c30 con 0x7f8a8c0cb5f0 2024-03-29T09:54:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a82000700 1 -- 172.21.15.12:0/2192736323 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84003dc0 con 0x7f8a8c0cb5f0 2024-03-29T09:54:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.865+0000 7f8a91000700 1 -- 172.21.15.12:0/2192736323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8c15f760 con 0x7f8a8c0cb5f0 2024-03-29T09:54:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.866+0000 7f8a91000700 1 -- 172.21.15.12:0/2192736323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8c0b6020 con 0x7f8a8c0cb5f0 2024-03-29T09:54:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.867+0000 7f8a91000700 1 -- 172.21.15.12:0/2192736323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8c15f950 con 0x7f8a8c0cb5f0 2024-03-29T09:54:39.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.867+0000 7f8a82000700 1 -- 172.21.15.12:0/2192736323 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8a840119b0 con 0x7f8a8c0cb5f0 2024-03-29T09:54:39.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.868+0000 7f8a82000700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a6c06ac90 0x7f8a6c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:39.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.868+0000 7f8a82000700 1 -- 172.21.15.12:0/2192736323 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a840898b0 con 0x7f8a8c0cb5f0 2024-03-29T09:54:39.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.869+0000 7f8a89e00700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a6c06ac90 0x7f8a6c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:39.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.869+0000 7f8a89e00700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a6c06ac90 0x7f8a6c06d150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c005040 tx=0x7f8a7c00a000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:39.873+0000 7f8a82000700 1 -- 172.21.15.12:0/2192736323 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a840549f0 con 0x7f8a8c0cb5f0 2024-03-29T09:54:40.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.083+0000 7f8a91000700 1 -- 172.21.15.12:0/2192736323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a8c067720 con 0x7f8a8c0cb5f0 2024-03-29T09:54:40.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.084+0000 7f8a82000700 1 -- 172.21.15.12:0/2192736323 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a84058010 con 0x7f8a8c0cb5f0 2024-03-29T09:54:40.084 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.086+0000 7f8a91000700 1 -- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a6c06ac90 msgr2=0x7f8a6c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.086+0000 7f8a91000700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a6c06ac90 0x7f8a6c06d150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c005040 tx=0x7f8a7c00a000 comp rx=0 tx=0).stop 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.086+0000 7f8a91000700 1 -- 172.21.15.12:0/2192736323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a8c0cb5f0 msgr2=0x7f8a8c0b4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.086+0000 7f8a91000700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a8c0cb5f0 0x7f8a8c0b4860 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0d2f40 tx=0x7f8a84009f90 comp rx=0 tx=0).stop 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.086+0000 7f8a91000700 1 -- 172.21.15.12:0/2192736323 shutdown_connections 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.086+0000 7f8a91000700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a6c06ac90 0x7f8a6c06d150 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.086+0000 7f8a91000700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a8c0cb5f0 0x7f8a8c0b4860 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.086+0000 7f8a91000700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a70003be0 0x7f8a8c158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.086+0000 7f8a91000700 1 --2- 172.21.15.12:0/2192736323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a8c15eaf0 0x7f8a8c0cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.087+0000 7f8a91000700 1 -- 172.21.15.12:0/2192736323 >> 172.21.15.12:0/2192736323 conn(0x7f8a8c08aff0 msgr2=0x7f8a8c0b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.087+0000 7f8a91000700 1 -- 172.21.15.12:0/2192736323 shutdown_connections 2024-03-29T09:54:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.087+0000 7f8a91000700 1 -- 172.21.15.12:0/2192736323 wait complete. 2024-03-29T09:54:40.104 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:54:40.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.305+0000 7f1ac6a00700 1 Processor -- start 2024-03-29T09:54:40.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.305+0000 7f1ac6a00700 1 -- start start 2024-03-29T09:54:40.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1ac6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ac015d9d0 0x7f1ac00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1ac6a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1ac00d16a0 0x7f1ac00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1ac6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac00d2370 0x7f1ac015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1ac6a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac00d2820 con 0x7f1ac015d9d0 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1ac6a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac0161420 con 0x7f1ac00d16a0 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1ac6a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0161700 con 0x7f1ac00d2370 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1abfe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ac015d9d0 0x7f1ac00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1abfe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ac015d9d0 0x7f1ac00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60606/0 (socket says 172.21.15.12:60606) 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1abfe00700 1 -- 172.21.15.12:0/3996021888 learned_addr learned my addr 172.21.15.12:0/3996021888 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1abf400700 1 --2- 172.21.15.12:0/3996021888 >> v2:172.21.15.159:3300/0 conn(0x7f1ac00d16a0 0x7f1ac00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.306+0000 7f1ac4a00700 1 --2- 172.21.15.12:0/3996021888 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac00d2370 0x7f1ac015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.307+0000 7f1abfe00700 1 -- 172.21.15.12:0/3996021888 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac00d2370 msgr2=0x7f1ac015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.307+0000 7f1abfe00700 1 --2- 172.21.15.12:0/3996021888 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac00d2370 0x7f1ac015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.307+0000 7f1abfe00700 1 -- 172.21.15.12:0/3996021888 >> v2:172.21.15.159:3300/0 conn(0x7f1ac00d16a0 msgr2=0x7f1ac00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.307+0000 7f1abfe00700 1 --2- 172.21.15.12:0/3996021888 >> v2:172.21.15.159:3300/0 conn(0x7f1ac00d16a0 0x7f1ac00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.307+0000 7f1abfe00700 1 -- 172.21.15.12:0/3996021888 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac01619e0 con 0x7f1ac015d9d0 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.307+0000 7f1abf400700 1 --2- 172.21.15.12:0/3996021888 >> v2:172.21.15.159:3300/0 conn(0x7f1ac00d16a0 0x7f1ac00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.307+0000 7f1abfe00700 1 --2- 172.21.15.12:0/3996021888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 0x7f1ac00d3c30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00d12f0 tx=0x7f1ab0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ce7fed0d28a22c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.307+0000 7f1abea00700 1 -- 172.21.15.12:0/3996021888 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab000b870 con 0x7f1ac015d9d0 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.307+0000 7f1abea00700 1 -- 172.21.15.12:0/3996021888 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab000b9d0 con 0x7f1ac015d9d0 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.307+0000 7f1abea00700 1 -- 172.21.15.12:0/3996021888 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab000bd60 con 0x7f1ac015d9d0 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.308+0000 7f1ac6a00700 1 -- 172.21.15.12:0/3996021888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 msgr2=0x7f1ac00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.308+0000 7f1ac6a00700 1 --2- 172.21.15.12:0/3996021888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 0x7f1ac00d3c30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00d12f0 tx=0x7f1ab0009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.308+0000 7f1ac6a00700 1 -- 172.21.15.12:0/3996021888 shutdown_connections 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.308+0000 7f1ac6a00700 1 --2- 172.21.15.12:0/3996021888 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac00d2370 0x7f1ac015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.308+0000 7f1ac6a00700 1 --2- 172.21.15.12:0/3996021888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 0x7f1ac00d3c30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.308+0000 7f1ac6a00700 1 --2- 172.21.15.12:0/3996021888 >> v2:172.21.15.159:3300/0 conn(0x7f1ac00d16a0 0x7f1ac00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.308+0000 7f1ac6a00700 1 -- 172.21.15.12:0/3996021888 >> 172.21.15.12:0/3996021888 conn(0x7f1ac008aff0 msgr2=0x7f1ac00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:40.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.308+0000 7f1ac6a00700 1 -- 172.21.15.12:0/3996021888 shutdown_connections 2024-03-29T09:54:40.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.308+0000 7f1ac6a00700 1 -- 172.21.15.12:0/3996021888 wait complete. 2024-03-29T09:54:40.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.309+0000 7f1ac6a00700 1 Processor -- start 2024-03-29T09:54:40.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.310+0000 7f1ac6a00700 1 -- start start 2024-03-29T09:54:40.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.310+0000 7f1ac6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac00d16a0 0x7f1ac01f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.310+0000 7f1ac6a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ac00d2370 0x7f1ac014d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 0x7f1ac00cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac6a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac01f6d80 con 0x7f1ac015d9d0 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac6a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac01f7340 con 0x7f1ac00d2370 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac6a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac01f7ca0 con 0x7f1ac00d16a0 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 0x7f1ac00cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 0x7f1ac00cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60608/0 (socket says 172.21.15.12:60608) 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac4a00700 1 -- 172.21.15.12:0/2342818400 learned_addr learned my addr 172.21.15.12:0/2342818400 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1abfe00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac00d16a0 0x7f1ac01f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1abf400700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ac00d2370 0x7f1ac014d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac4a00700 1 -- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac00d16a0 msgr2=0x7f1ac01f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac4a00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac00d16a0 0x7f1ac01f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac4a00700 1 -- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ac00d2370 msgr2=0x7f1ac014d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac4a00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ac00d2370 0x7f1ac014d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.311+0000 7f1ac4a00700 1 -- 172.21.15.12:0/2342818400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab0009550 con 0x7f1ac015d9d0 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.312+0000 7f1ac4a00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 0x7f1ac00cd440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8007e40 tx=0x7f1ab800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:40.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.312+0000 7f1ab7e00700 1 -- 172.21.15.12:0/2342818400 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab800ed30 con 0x7f1ac015d9d0 2024-03-29T09:54:40.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.312+0000 7f1ab7e00700 1 -- 172.21.15.12:0/2342818400 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab8014420 con 0x7f1ac015d9d0 2024-03-29T09:54:40.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.312+0000 7f1ab7e00700 1 -- 172.21.15.12:0/2342818400 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab80147d0 con 0x7f1ac015d9d0 2024-03-29T09:54:40.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.312+0000 7f1ac6a00700 1 -- 172.21.15.12:0/2342818400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac014d800 con 0x7f1ac015d9d0 2024-03-29T09:54:40.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.312+0000 7f1ac6a00700 1 -- 172.21.15.12:0/2342818400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac0161030 con 0x7f1ac015d9d0 2024-03-29T09:54:40.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.313+0000 7f1ac6a00700 1 -- 172.21.15.12:0/2342818400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac00d1ca0 con 0x7f1ac015d9d0 2024-03-29T09:54:40.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.314+0000 7f1ab7e00700 1 -- 172.21.15.12:0/2342818400 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1ab8014930 con 0x7f1ac015d9d0 2024-03-29T09:54:40.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.315+0000 7f1ab7e00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1aa006ac90 0x7f1aa006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.315+0000 7f1ab7e00700 1 -- 172.21.15.12:0/2342818400 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ab8089a50 con 0x7f1ac015d9d0 2024-03-29T09:54:40.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.315+0000 7f1abfe00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1aa006ac90 0x7f1aa006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.316+0000 7f1abfe00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1aa006ac90 0x7f1aa006d150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0014040 tx=0x7f1ab0008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:40.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.319+0000 7f1ab7e00700 1 -- 172.21.15.12:0/2342818400 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ab8054b90 con 0x7f1ac015d9d0 2024-03-29T09:54:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.532+0000 7f1ac6a00700 1 -- 172.21.15.12:0/2342818400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1ac01f7480 con 0x7f1ac015d9d0 2024-03-29T09:54:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.533+0000 7f1ab7e00700 1 -- 172.21.15.12:0/2342818400 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1ac01f7480 con 0x7f1ac015d9d0 2024-03-29T09:54:40.532 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:54:40.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 -- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1aa006ac90 msgr2=0x7f1aa006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1aa006ac90 0x7f1aa006d150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0014040 tx=0x7f1ab0008030 comp rx=0 tx=0).stop 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 -- 172.21.15.12:0/2342818400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 msgr2=0x7f1ac00cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 0x7f1ac00cd440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8007e40 tx=0x7f1ab800e3c0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 -- 172.21.15.12:0/2342818400 shutdown_connections 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1aa006ac90 0x7f1aa006d150 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac015d9d0 0x7f1ac00cd440 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac00d16a0 0x7f1ac01f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 --2- 172.21.15.12:0/2342818400 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ac00d2370 0x7f1ac014d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 -- 172.21.15.12:0/2342818400 >> 172.21.15.12:0/2342818400 conn(0x7f1ac008aff0 msgr2=0x7f1ac00ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 -- 172.21.15.12:0/2342818400 shutdown_connections 2024-03-29T09:54:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.535+0000 7f1ac6a00700 1 -- 172.21.15.12:0/2342818400 wait complete. 2024-03-29T09:54:40.553 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.754+0000 7f4719200700 1 Processor -- start 2024-03-29T09:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.754+0000 7f4719200700 1 -- start start 2024-03-29T09:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.755+0000 7f4719200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47140d16a0 0x7f47140d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.755+0000 7f4719200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f47140d2370 0x7f47140ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.755+0000 7f4719200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4714161420 con 0x7f471415d9d0 2024-03-29T09:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.755+0000 7f4719200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47140d2820 con 0x7f47140d2370 2024-03-29T09:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.755+0000 7f4719200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47140d4340 con 0x7f47140d16a0 2024-03-29T09:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.756+0000 7f4712800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f47140d2370 0x7f47140ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.756+0000 7f4712800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f47140d2370 0x7f47140ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60618/0 (socket says 172.21.15.12:60618) 2024-03-29T09:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.756+0000 7f4712800700 1 -- 172.21.15.12:0/959599902 learned_addr learned my addr 172.21.15.12:0/959599902 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.756+0000 7f4711400700 1 --2- 172.21.15.12:0/959599902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47140d16a0 0x7f47140d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.756+0000 7f4712800700 1 -- 172.21.15.12:0/959599902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47140d16a0 msgr2=0x7f47140d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.756+0000 7f4712800700 1 --2- 172.21.15.12:0/959599902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47140d16a0 0x7f47140d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.756+0000 7f4712800700 1 -- 172.21.15.12:0/959599902 >> v1:172.21.15.159:6789/0 conn(0x7f471415d9d0 legacy=0x7f47140d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.756+0000 7f4712800700 1 -- 172.21.15.12:0/959599902 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4714161700 con 0x7f47140d2370 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.756+0000 7f4712800700 1 --2- 172.21.15.12:0/959599902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47140d2370 0x7f47140ce360 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f470c009e80 tx=0x7f470c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce48c8805fa71abb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4710a00700 1 -- 172.21.15.12:0/959599902 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c00e070 con 0x7f47140d2370 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4710a00700 1 -- 172.21.15.12:0/959599902 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c003a20 con 0x7f47140d2370 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4710a00700 1 -- 172.21.15.12:0/959599902 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c003dd0 con 0x7f47140d2370 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4719200700 1 -- 172.21.15.12:0/959599902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47140d2370 msgr2=0x7f47140ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4719200700 1 --2- 172.21.15.12:0/959599902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47140d2370 0x7f47140ce360 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f470c009e80 tx=0x7f470c00b330 comp rx=0 tx=0).stop 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4719200700 1 -- 172.21.15.12:0/959599902 shutdown_connections 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4719200700 1 --2- 172.21.15.12:0/959599902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47140d2370 0x7f47140ce360 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4719200700 1 --2- 172.21.15.12:0/959599902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47140d16a0 0x7f47140d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4719200700 1 -- 172.21.15.12:0/959599902 >> 172.21.15.12:0/959599902 conn(0x7f471408aff0 msgr2=0x7f47140d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4719200700 1 -- 172.21.15.12:0/959599902 shutdown_connections 2024-03-29T09:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.757+0000 7f4719200700 1 -- 172.21.15.12:0/959599902 wait complete. 2024-03-29T09:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.759+0000 7f4719200700 1 Processor -- start 2024-03-29T09:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.759+0000 7f4719200700 1 -- start start 2024-03-29T09:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.760+0000 7f4719200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47140d16a0 0x7f47140cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.760+0000 7f4719200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47140d2370 0x7f47141e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.760+0000 7f4719200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471415d9d0 0x7f47141e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.760+0000 7f4719200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47141e9020 con 0x7f471415d9d0 2024-03-29T09:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.760+0000 7f4719200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47141e77c0 con 0x7f47140d16a0 2024-03-29T09:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.760+0000 7f4719200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47141e7a40 con 0x7f47140d2370 2024-03-29T09:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4712800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471415d9d0 0x7f47141e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4712800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471415d9d0 0x7f47141e7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60624/0 (socket says 172.21.15.12:60624) 2024-03-29T09:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4711400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47140d2370 0x7f47141e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4712800700 1 -- 172.21.15.12:0/3854631103 learned_addr learned my addr 172.21.15.12:0/3854631103 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4711e00700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47140d16a0 0x7f47140cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4712800700 1 -- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47140d2370 msgr2=0x7f47141e8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4712800700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47140d2370 0x7f47141e8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4712800700 1 -- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47140d16a0 msgr2=0x7f47140cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4712800700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47140d16a0 0x7f47140cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4712800700 1 -- 172.21.15.12:0/3854631103 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c009b30 con 0x7f471415d9d0 2024-03-29T09:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4712800700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471415d9d0 0x7f47141e7280 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f471408c230 tx=0x7f470c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f470a000700 1 -- 172.21.15.12:0/3854631103 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c0092e0 con 0x7f471415d9d0 2024-03-29T09:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f470a000700 1 -- 172.21.15.12:0/3854631103 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c0094f0 con 0x7f471415d9d0 2024-03-29T09:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f470a000700 1 -- 172.21.15.12:0/3854631103 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c004460 con 0x7f471415d9d0 2024-03-29T09:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.761+0000 7f4719200700 1 -- 172.21.15.12:0/3854631103 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47141e7d20 con 0x7f471415d9d0 2024-03-29T09:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.762+0000 7f4719200700 1 -- 172.21.15.12:0/3854631103 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47141e82e0 con 0x7f471415d9d0 2024-03-29T09:54:40.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.763+0000 7f4719200700 1 -- 172.21.15.12:0/3854631103 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47140d1ca0 con 0x7f471415d9d0 2024-03-29T09:54:40.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.763+0000 7f470a000700 1 -- 172.21.15.12:0/3854631103 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f470c019070 con 0x7f471415d9d0 2024-03-29T09:54:40.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.764+0000 7f470a000700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f46f406ac90 0x7f46f406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:40.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.764+0000 7f470a000700 1 -- 172.21.15.12:0/3854631103 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f470c08a140 con 0x7f471415d9d0 2024-03-29T09:54:40.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.765+0000 7f4711e00700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f46f406ac90 0x7f46f406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:40.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.765+0000 7f4711e00700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f46f406ac90 0x7f46f406d150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4704009960 tx=0x7f47040090e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.769+0000 7f470a000700 1 -- 172.21.15.12:0/3854631103 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f470c058da0 con 0x7f471415d9d0 2024-03-29T09:54:40.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.985+0000 7f4719200700 1 -- 172.21.15.12:0/3854631103 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f47140d3310 con 0x7f471415d9d0 2024-03-29T09:54:40.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.986+0000 7f470a000700 1 -- 172.21.15.12:0/3854631103 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f470c058930 con 0x7f471415d9d0 2024-03-29T09:54:40.985 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:54:40.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 -- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f46f406ac90 msgr2=0x7f46f406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f46f406ac90 0x7f46f406d150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4704009960 tx=0x7f47040090e0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 -- 172.21.15.12:0/3854631103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471415d9d0 msgr2=0x7f47141e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:40.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471415d9d0 0x7f47141e7280 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f471408c230 tx=0x7f470c004ad0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 -- 172.21.15.12:0/3854631103 shutdown_connections 2024-03-29T09:54:40.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f46f406ac90 0x7f46f406d150 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471415d9d0 0x7f47141e7280 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47140d16a0 0x7f47140cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 --2- 172.21.15.12:0/3854631103 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47140d2370 0x7f47141e8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:40.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 -- 172.21.15.12:0/3854631103 >> 172.21.15.12:0/3854631103 conn(0x7f471408aff0 msgr2=0x7f47140b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:40.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 -- 172.21.15.12:0/3854631103 shutdown_connections 2024-03-29T09:54:40.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:40.988+0000 7f4719200700 1 -- 172.21.15.12:0/3854631103 wait complete. 2024-03-29T09:54:41.006 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:54:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.203+0000 7fa65a400700 1 Processor -- start 2024-03-29T09:54:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.203+0000 7fa65a400700 1 -- start start 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.204+0000 7fa65a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6540d18b0 0x7fa6540cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.204+0000 7fa65a400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa654161590 0x7fa654161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.204+0000 7fa65a400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6540ceda0 con 0x7fa654160f10 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.204+0000 7fa65a400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6540d1d30 con 0x7fa654161590 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.204+0000 7fa65a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa654161f60 con 0x7fa6540d18b0 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa653400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa654160f10 0x7fa6540d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54802/0 (socket says 172.21.15.12:54802) 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa653400700 1 -- 172.21.15.12:0/1276181765 learned_addr learned my addr 172.21.15.12:0/1276181765 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa653e00700 1 --2- 172.21.15.12:0/1276181765 >> v2:172.21.15.159:3300/0 conn(0x7fa654161590 0x7fa654161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa652a00700 1 --2- 172.21.15.12:0/1276181765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6540d18b0 0x7fa6540cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa652000700 1 -- 172.21.15.12:0/1276181765 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021829495 0 0) 0x7fa6540ceda0 con 0x7fa654160f10 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa652000700 1 -- 172.21.15.12:0/1276181765 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6380035b0 con 0x7fa654160f10 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa652a00700 1 -- 172.21.15.12:0/1276181765 >> v2:172.21.15.159:3300/0 conn(0x7fa654161590 msgr2=0x7fa654161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa652a00700 1 --2- 172.21.15.12:0/1276181765 >> v2:172.21.15.159:3300/0 conn(0x7fa654161590 0x7fa654161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa652a00700 1 -- 172.21.15.12:0/1276181765 >> v1:172.21.15.12:6789/0 conn(0x7fa654160f10 legacy=0x7fa6540d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa652a00700 1 -- 172.21.15.12:0/1276181765 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6541612f0 con 0x7fa6540d18b0 2024-03-29T09:54:41.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.205+0000 7fa653e00700 1 --2- 172.21.15.12:0/1276181765 >> v2:172.21.15.159:3300/0 conn(0x7fa654161590 0x7fa654161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:41.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.206+0000 7fa652a00700 1 --2- 172.21.15.12:0/1276181765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6540d18b0 0x7fa6540cf230 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa640009870 tx=0x7fa640009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d59a40a7ae9556 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:41.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.206+0000 7fa652000700 1 -- 172.21.15.12:0/1276181765 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640011830 con 0x7fa6540d18b0 2024-03-29T09:54:41.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.206+0000 7fa652000700 1 -- 172.21.15.12:0/1276181765 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa640011990 con 0x7fa6540d18b0 2024-03-29T09:54:41.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.206+0000 7fa652000700 1 -- 172.21.15.12:0/1276181765 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640011d20 con 0x7fa6540d18b0 2024-03-29T09:54:41.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.206+0000 7fa65a400700 1 -- 172.21.15.12:0/1276181765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6540d18b0 msgr2=0x7fa6540cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.206+0000 7fa65a400700 1 --2- 172.21.15.12:0/1276181765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6540d18b0 0x7fa6540cf230 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa640009870 tx=0x7fa640009b80 comp rx=0 tx=0).stop 2024-03-29T09:54:41.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.207+0000 7fa65a400700 1 -- 172.21.15.12:0/1276181765 shutdown_connections 2024-03-29T09:54:41.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.207+0000 7fa65a400700 1 --2- 172.21.15.12:0/1276181765 >> v2:172.21.15.159:3300/0 conn(0x7fa654161590 0x7fa654161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.207+0000 7fa65a400700 1 --2- 172.21.15.12:0/1276181765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6540d18b0 0x7fa6540cf230 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.207+0000 7fa65a400700 1 -- 172.21.15.12:0/1276181765 >> 172.21.15.12:0/1276181765 conn(0x7fa65408aff0 msgr2=0x7fa6540d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:41.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.207+0000 7fa65a400700 1 -- 172.21.15.12:0/1276181765 shutdown_connections 2024-03-29T09:54:41.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.207+0000 7fa65a400700 1 -- 172.21.15.12:0/1276181765 wait complete. 2024-03-29T09:54:41.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.208+0000 7fa65a400700 1 Processor -- start 2024-03-29T09:54:41.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.209+0000 7fa65a400700 1 -- start start 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa65a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6540d18b0 0x7fa654160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa65a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa654160f10 0x7fa65414ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa65a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654161590 0x7fa6540b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa65a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65414bf50 con 0x7fa654161590 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa65a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65415e700 con 0x7fa6540d18b0 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa65a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65415eb90 con 0x7fa654160f10 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa653e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654161590 0x7fa6540b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa653e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654161590 0x7fa6540b8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60628/0 (socket says 172.21.15.12:60628) 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa653e00700 1 -- 172.21.15.12:0/881740981 learned_addr learned my addr 172.21.15.12:0/881740981 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa652a00700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa654160f10 0x7fa65414ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa653400700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6540d18b0 0x7fa654160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa653e00700 1 -- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa654160f10 msgr2=0x7fa65414ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa653e00700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa654160f10 0x7fa65414ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa653e00700 1 -- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6540d18b0 msgr2=0x7fa654160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa653e00700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6540d18b0 0x7fa654160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.210+0000 7fa653e00700 1 -- 172.21.15.12:0/881740981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c009a70 con 0x7fa654161590 2024-03-29T09:54:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.211+0000 7fa653e00700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654161590 0x7fa6540b8220 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00efc0 tx=0x7fa64c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.211+0000 7fa64b400700 1 -- 172.21.15.12:0/881740981 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00ca90 con 0x7fa654161590 2024-03-29T09:54:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.211+0000 7fa64b400700 1 -- 172.21.15.12:0/881740981 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c00f040 con 0x7fa654161590 2024-03-29T09:54:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.211+0000 7fa64b400700 1 -- 172.21.15.12:0/881740981 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00ce30 con 0x7fa654161590 2024-03-29T09:54:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.211+0000 7fa65a400700 1 -- 172.21.15.12:0/881740981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6400094a0 con 0x7fa654161590 2024-03-29T09:54:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.211+0000 7fa65a400700 1 -- 172.21.15.12:0/881740981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65415e0b0 con 0x7fa654161590 2024-03-29T09:54:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.212+0000 7fa65a400700 1 -- 172.21.15.12:0/881740981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa654067720 con 0x7fa654161590 2024-03-29T09:54:41.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.213+0000 7fa64b400700 1 -- 172.21.15.12:0/881740981 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa64c014480 con 0x7fa654161590 2024-03-29T09:54:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.214+0000 7fa64b400700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa63406ac90 0x7fa63406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.214+0000 7fa64b400700 1 -- 172.21.15.12:0/881740981 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa64c089aa0 con 0x7fa654161590 2024-03-29T09:54:41.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.215+0000 7fa653400700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa63406ac90 0x7fa63406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.215+0000 7fa653400700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa63406ac90 0x7fa63406d150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa644007fd0 tx=0x7fa644009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:41.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.218+0000 7fa64b400700 1 -- 172.21.15.12:0/881740981 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa64c054be0 con 0x7fa654161590 2024-03-29T09:54:41.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.481+0000 7fa65a400700 1 -- 172.21.15.12:0/881740981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa65415e840 con 0x7fa654161590 2024-03-29T09:54:41.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.482+0000 7fa64b400700 1 -- 172.21.15.12:0/881740981 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7fa64c009cd0 con 0x7fa654161590 2024-03-29T09:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/4093246304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4093246304},{"type":"v1","addr":"172.21.15.12:6833","nonce":4093246304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1631416176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1631416176},{"type":"v1","addr":"172.21.15.159:6836","nonce":1631416176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3828834681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3828834681},{"type":"v1","addr":"172.21.15.159:6837","nonce":3828834681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:54:27.425500+0000","modified":"2024-03-29T09:54:30.895316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3922451359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3922451359},{"type":"v1","addr":"172.21.15.12:6835","nonce":3922451359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-29T09:54:41.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 -- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa63406ac90 msgr2=0x7fa63406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa63406ac90 0x7fa63406d150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa644007fd0 tx=0x7fa644009040 comp rx=0 tx=0).stop 2024-03-29T09:54:41.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 -- 172.21.15.12:0/881740981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654161590 msgr2=0x7fa6540b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654161590 0x7fa6540b8220 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00efc0 tx=0x7fa64c00c3a0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 -- 172.21.15.12:0/881740981 shutdown_connections 2024-03-29T09:54:41.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa63406ac90 0x7fa63406d150 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654161590 0x7fa6540b8220 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6540d18b0 0x7fa654160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 --2- 172.21.15.12:0/881740981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa654160f10 0x7fa65414ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 -- 172.21.15.12:0/881740981 >> 172.21.15.12:0/881740981 conn(0x7fa65408aff0 msgr2=0x7fa65415bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:41.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 -- 172.21.15.12:0/881740981 shutdown_connections 2024-03-29T09:54:41.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.484+0000 7fa65a400700 1 -- 172.21.15.12:0/881740981 wait complete. 2024-03-29T09:54:41.485 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-03-29T09:54:41.502 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:41.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.699+0000 7fd553800700 1 Processor -- start 2024-03-29T09:54:41.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.699+0000 7fd553800700 1 -- start start 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.700+0000 7fd553800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd54c0d18b0 0x7fd54c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.700+0000 7fd553800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c161590 0x7fd54c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.700+0000 7fd553800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0ceda0 con 0x7fd54c160f10 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.700+0000 7fd553800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c0d1d30 con 0x7fd54c0d18b0 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.700+0000 7fd553800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c161f60 con 0x7fd54c161590 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd54be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd54c0d18b0 0x7fd54c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd550e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd54c160f10 0x7fd54c0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54818/0 (socket says 172.21.15.12:54818) 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd550e00700 1 -- 172.21.15.12:0/1646160085 learned_addr learned my addr 172.21.15.12:0/1646160085 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd551800700 1 --2- 172.21.15.12:0/1646160085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c161590 0x7fd54c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd54b400700 1 -- 172.21.15.12:0/1646160085 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161971304 0 0) 0x7fd54c0ceda0 con 0x7fd54c160f10 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd54be00700 1 -- 172.21.15.12:0/1646160085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c161590 msgr2=0x7fd54c161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd54be00700 1 --2- 172.21.15.12:0/1646160085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c161590 0x7fd54c161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd54be00700 1 -- 172.21.15.12:0/1646160085 >> v1:172.21.15.12:6789/0 conn(0x7fd54c160f10 legacy=0x7fd54c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd54be00700 1 -- 172.21.15.12:0/1646160085 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c1612f0 con 0x7fd54c0d18b0 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd551800700 1 --2- 172.21.15.12:0/1646160085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c161590 0x7fd54c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.701+0000 7fd54be00700 1 --2- 172.21.15.12:0/1646160085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 0x7fd54c0cf230 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd538009870 tx=0x7fd538009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08399375d757963 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd54b400700 1 -- 172.21.15.12:0/1646160085 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd538011830 con 0x7fd54c0d18b0 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd54b400700 1 -- 172.21.15.12:0/1646160085 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd538011990 con 0x7fd54c0d18b0 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd54b400700 1 -- 172.21.15.12:0/1646160085 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd538011d20 con 0x7fd54c0d18b0 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd553800700 1 -- 172.21.15.12:0/1646160085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 msgr2=0x7fd54c0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd553800700 1 --2- 172.21.15.12:0/1646160085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 0x7fd54c0cf230 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd538009870 tx=0x7fd538009b80 comp rx=0 tx=0).stop 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd553800700 1 -- 172.21.15.12:0/1646160085 shutdown_connections 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd553800700 1 --2- 172.21.15.12:0/1646160085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c161590 0x7fd54c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd553800700 1 --2- 172.21.15.12:0/1646160085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 0x7fd54c0cf230 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd553800700 1 -- 172.21.15.12:0/1646160085 >> 172.21.15.12:0/1646160085 conn(0x7fd54c08aff0 msgr2=0x7fd54c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd553800700 1 -- 172.21.15.12:0/1646160085 shutdown_connections 2024-03-29T09:54:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.702+0000 7fd553800700 1 -- 172.21.15.12:0/1646160085 wait complete. 2024-03-29T09:54:41.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.704+0000 7fd553800700 1 Processor -- start 2024-03-29T09:54:41.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.704+0000 7fd553800700 1 -- start start 2024-03-29T09:54:41.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.705+0000 7fd553800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 0x7fd54c0a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.705+0000 7fd553800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c160f10 0x7fd54c158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.705+0000 7fd553800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd54c161590 0x7fd54c15fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.705+0000 7fd553800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0a0250 con 0x7fd54c0d18b0 2024-03-29T09:54:41.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.705+0000 7fd553800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0b83a0 con 0x7fd54c161590 2024-03-29T09:54:41.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.705+0000 7fd553800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c158b60 con 0x7fd54c160f10 2024-03-29T09:54:41.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.705+0000 7fd550e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 0x7fd54c0a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd550e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 0x7fd54c0a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60644/0 (socket says 172.21.15.12:60644) 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd550e00700 1 -- 172.21.15.12:0/213394892 learned_addr learned my addr 172.21.15.12:0/213394892 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd551800700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd54c161590 0x7fd54c15fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd54be00700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c160f10 0x7fd54c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd550e00700 1 -- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c160f10 msgr2=0x7fd54c158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd550e00700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c160f10 0x7fd54c158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd550e00700 1 -- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd54c161590 msgr2=0x7fd54c15fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd550e00700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd54c161590 0x7fd54c15fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd550e00700 1 -- 172.21.15.12:0/213394892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5380094a0 con 0x7fd54c0d18b0 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd550e00700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 0x7fd54c0a0670 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd54000c580 tx=0x7fd54000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.706+0000 7fd548c00700 1 -- 172.21.15.12:0/213394892 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540004c80 con 0x7fd54c0d18b0 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.707+0000 7fd548c00700 1 -- 172.21.15.12:0/213394892 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd540004e00 con 0x7fd54c0d18b0 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.707+0000 7fd553800700 1 -- 172.21.15.12:0/213394892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c09f5d0 con 0x7fd54c0d18b0 2024-03-29T09:54:41.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.707+0000 7fd548c00700 1 -- 172.21.15.12:0/213394892 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540007310 con 0x7fd54c0d18b0 2024-03-29T09:54:41.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.707+0000 7fd553800700 1 -- 172.21.15.12:0/213394892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c09fb90 con 0x7fd54c0d18b0 2024-03-29T09:54:41.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.708+0000 7fd553800700 1 -- 172.21.15.12:0/213394892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c067720 con 0x7fd54c0d18b0 2024-03-29T09:54:41.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.709+0000 7fd548c00700 1 -- 172.21.15.12:0/213394892 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd540007470 con 0x7fd54c0d18b0 2024-03-29T09:54:41.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.710+0000 7fd548c00700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd52c06ac90 0x7fd52c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:41.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.710+0000 7fd548c00700 1 -- 172.21.15.12:0/213394892 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd540088e20 con 0x7fd54c0d18b0 2024-03-29T09:54:41.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.710+0000 7fd54be00700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd52c06ac90 0x7fd52c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:41.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.711+0000 7fd54be00700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd52c06ac90 0x7fd52c06d150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd5380090b0 tx=0x7fd53801c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.715+0000 7fd548c00700 1 -- 172.21.15.12:0/213394892 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd540054090 con 0x7fd54c0d18b0 2024-03-29T09:54:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.930+0000 7fd553800700 1 -- 172.21.15.12:0/213394892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd54c0b5da0 con 0x7fd54c0d18b0 2024-03-29T09:54:41.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.931+0000 7fd548c00700 1 -- 172.21.15.12:0/213394892 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7fd5400576b0 con 0x7fd54c0d18b0 2024-03-29T09:54:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:41.931 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 -- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd52c06ac90 msgr2=0x7fd52c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd52c06ac90 0x7fd52c06d150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd5380090b0 tx=0x7fd53801c040 comp rx=0 tx=0).stop 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 -- 172.21.15.12:0/213394892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 msgr2=0x7fd54c0a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 0x7fd54c0a0670 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd54000c580 tx=0x7fd54000c890 comp rx=0 tx=0).stop 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 -- 172.21.15.12:0/213394892 shutdown_connections 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd52c06ac90 0x7fd52c06d150 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd54c161590 0x7fd54c15fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd54c0d18b0 0x7fd54c0a0670 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 --2- 172.21.15.12:0/213394892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd54c160f10 0x7fd54c158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 -- 172.21.15.12:0/213394892 >> 172.21.15.12:0/213394892 conn(0x7fd54c08aff0 msgr2=0x7fd54c0cbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 -- 172.21.15.12:0/213394892 shutdown_connections 2024-03-29T09:54:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:41.934+0000 7fd553800700 1 -- 172.21.15.12:0/213394892 wait complete. 2024-03-29T09:54:41.952 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:54:41.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:54:42.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.155+0000 7f29c9400700 1 Processor -- start 2024-03-29T09:54:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.155+0000 7f29c9400700 1 -- start start 2024-03-29T09:54:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.156+0000 7f29c9400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f29c40d16a0 0x7f29c40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.156+0000 7f29c9400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c4161420 con 0x7f29c40d2370 2024-03-29T09:54:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.156+0000 7f29c9400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c4161700 con 0x7f29c415d9d0 2024-03-29T09:54:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.156+0000 7f29c9400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c40d4340 con 0x7f29c40d16a0 2024-03-29T09:54:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.156+0000 7f29c3200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f29c40d2370 0x7f29c40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54832/0 (socket says 172.21.15.12:54832) 2024-03-29T09:54:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.156+0000 7f29c3200700 1 -- 172.21.15.12:0/709954276 learned_addr learned my addr 172.21.15.12:0/709954276 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.156+0000 7f29c1e00700 1 --2- 172.21.15.12:0/709954276 >> v2:172.21.15.159:3300/0 conn(0x7f29c40d16a0 0x7f29c40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.157+0000 7f29c1400700 1 -- 172.21.15.12:0/709954276 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758889539 0 0) 0x7f29c4161420 con 0x7f29c40d2370 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.157+0000 7f29c1400700 1 -- 172.21.15.12:0/709954276 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a80035b0 con 0x7f29c40d2370 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.157+0000 7f29c1e00700 1 -- 172.21.15.12:0/709954276 >> v1:172.21.15.159:6789/0 conn(0x7f29c415d9d0 legacy=0x7f29c40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.157+0000 7f29c1400700 1 -- 172.21.15.12:0/709954276 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570492168 0 0) 0x7f29c4161700 con 0x7f29c415d9d0 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.157+0000 7f29c1e00700 1 -- 172.21.15.12:0/709954276 >> v1:172.21.15.12:6789/0 conn(0x7f29c40d2370 legacy=0x7f29c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.157+0000 7f29c1e00700 1 -- 172.21.15.12:0/709954276 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c41619e0 con 0x7f29c40d16a0 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.157+0000 7f29c1e00700 1 --2- 172.21.15.12:0/709954276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29c40d16a0 0x7f29c40d1e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009820 tx=0x7f29b0009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=927d227560d07845 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.158+0000 7f29c1400700 1 -- 172.21.15.12:0/709954276 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b0005e30 con 0x7f29c40d16a0 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.158+0000 7f29c1400700 1 -- 172.21.15.12:0/709954276 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b0006030 con 0x7f29c40d16a0 2024-03-29T09:54:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.158+0000 7f29c1400700 1 -- 172.21.15.12:0/709954276 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b00063c0 con 0x7f29c40d16a0 2024-03-29T09:54:42.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.158+0000 7f29c9400700 1 -- 172.21.15.12:0/709954276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29c40d16a0 msgr2=0x7f29c40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.158+0000 7f29c9400700 1 --2- 172.21.15.12:0/709954276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29c40d16a0 0x7f29c40d1e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009820 tx=0x7f29b0009b30 comp rx=0 tx=0).stop 2024-03-29T09:54:42.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.158+0000 7f29c9400700 1 -- 172.21.15.12:0/709954276 shutdown_connections 2024-03-29T09:54:42.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.158+0000 7f29c9400700 1 --2- 172.21.15.12:0/709954276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29c40d16a0 0x7f29c40d1e30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.158+0000 7f29c9400700 1 -- 172.21.15.12:0/709954276 >> 172.21.15.12:0/709954276 conn(0x7f29c408aff0 msgr2=0x7f29c40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:42.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.158+0000 7f29c9400700 1 -- 172.21.15.12:0/709954276 shutdown_connections 2024-03-29T09:54:42.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.158+0000 7f29c9400700 1 -- 172.21.15.12:0/709954276 wait complete. 2024-03-29T09:54:42.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.160+0000 7f29c9400700 1 Processor -- start 2024-03-29T09:54:42.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.160+0000 7f29c9400700 1 -- start start 2024-03-29T09:54:42.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.161+0000 7f29c9400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29c40d16a0 0x7f29c40ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.161+0000 7f29c9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29c40d2370 0x7f29c40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.161+0000 7f29c9400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29c415d9d0 0x7f29c4158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.161+0000 7f29c9400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c41592d0 con 0x7f29c40d2370 2024-03-29T09:54:42.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.161+0000 7f29c9400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40cac60 con 0x7f29c415d9d0 2024-03-29T09:54:42.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.161+0000 7f29c9400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40caf40 con 0x7f29c40d16a0 2024-03-29T09:54:42.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.161+0000 7f29c1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29c40d2370 0x7f29c40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.161+0000 7f29c1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29c40d2370 0x7f29c40cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60654/0 (socket says 172.21.15.12:60654) 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.161+0000 7f29c1e00700 1 -- 172.21.15.12:0/3661709214 learned_addr learned my addr 172.21.15.12:0/3661709214 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.162+0000 7f29c2800700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29c40d16a0 0x7f29c40ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.162+0000 7f29c3200700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29c415d9d0 0x7f29c4158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.162+0000 7f29c1e00700 1 -- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29c40d16a0 msgr2=0x7f29c40ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.162+0000 7f29c1e00700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29c40d16a0 0x7f29c40ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.162+0000 7f29c1e00700 1 -- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29c415d9d0 msgr2=0x7f29c4158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.162+0000 7f29c1e00700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29c415d9d0 0x7f29c4158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.162+0000 7f29c1e00700 1 -- 172.21.15.12:0/3661709214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b0009430 con 0x7f29c40d2370 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.162+0000 7f29c1e00700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29c40d2370 0x7f29c40cbd00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f29b0013040 tx=0x7f29b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.163+0000 7f29baa00700 1 -- 172.21.15.12:0/3661709214 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b00067f0 con 0x7f29c40d2370 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.163+0000 7f29baa00700 1 -- 172.21.15.12:0/3661709214 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b0020070 con 0x7f29c40d2370 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.163+0000 7f29c9400700 1 -- 172.21.15.12:0/3661709214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c40cb220 con 0x7f29c40d2370 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.163+0000 7f29baa00700 1 -- 172.21.15.12:0/3661709214 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b0025670 con 0x7f29c40d2370 2024-03-29T09:54:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.163+0000 7f29c9400700 1 -- 172.21.15.12:0/3661709214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c40cb7e0 con 0x7f29c40d2370 2024-03-29T09:54:42.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.164+0000 7f29c9400700 1 -- 172.21.15.12:0/3661709214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c40d1ca0 con 0x7f29c40d2370 2024-03-29T09:54:42.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.165+0000 7f29baa00700 1 -- 172.21.15.12:0/3661709214 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f29b0022030 con 0x7f29c40d2370 2024-03-29T09:54:42.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.166+0000 7f29baa00700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f29a406ac90 0x7f29a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.166+0000 7f29baa00700 1 -- 172.21.15.12:0/3661709214 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29b008a6f0 con 0x7f29c40d2370 2024-03-29T09:54:42.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.166+0000 7f29c2800700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f29a406ac90 0x7f29a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.167+0000 7f29c2800700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f29a406ac90 0x7f29a406d150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f29b4000f60 tx=0x7f29b4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:42.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.170+0000 7f29baa00700 1 -- 172.21.15.12:0/3661709214 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f29b0055960 con 0x7f29c40d2370 2024-03-29T09:54:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.386+0000 7f29c9400700 1 -- 172.21.15.12:0/3661709214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29c4067720 con 0x7f29c40d2370 2024-03-29T09:54:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.386+0000 7f29baa00700 1 -- 172.21.15.12:0/3661709214 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29b0058f80 con 0x7f29c40d2370 2024-03-29T09:54:42.386 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:54:42.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 -- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f29a406ac90 msgr2=0x7f29a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f29a406ac90 0x7f29a406d150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f29b4000f60 tx=0x7f29b4009040 comp rx=0 tx=0).stop 2024-03-29T09:54:42.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 -- 172.21.15.12:0/3661709214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29c40d2370 msgr2=0x7f29c40cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29c40d2370 0x7f29c40cbd00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f29b0013040 tx=0x7f29b0004750 comp rx=0 tx=0).stop 2024-03-29T09:54:42.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 -- 172.21.15.12:0/3661709214 shutdown_connections 2024-03-29T09:54:42.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f29a406ac90 0x7f29a406d150 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29c415d9d0 0x7f29c4158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29c40d16a0 0x7f29c40ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 --2- 172.21.15.12:0/3661709214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29c40d2370 0x7f29c40cbd00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 -- 172.21.15.12:0/3661709214 >> 172.21.15.12:0/3661709214 conn(0x7f29c408aff0 msgr2=0x7f29c40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 -- 172.21.15.12:0/3661709214 shutdown_connections 2024-03-29T09:54:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.389+0000 7f29c9400700 1 -- 172.21.15.12:0/3661709214 wait complete. 2024-03-29T09:54:42.406 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:54:42.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.607+0000 7f8954c00700 1 Processor -- start 2024-03-29T09:54:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.607+0000 7f8954c00700 1 -- start start 2024-03-29T09:54:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.608+0000 7f8954c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f895015d9d0 0x7f89500d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.608+0000 7f8954c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f89500d16a0 0x7f89500d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.608+0000 7f8954c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8950161420 con 0x7f89500d2370 2024-03-29T09:54:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.608+0000 7f8954c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89500d27e0 con 0x7f89500d16a0 2024-03-29T09:54:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.608+0000 7f8954c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8950161700 con 0x7f895015d9d0 2024-03-29T09:54:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.608+0000 7f894d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f89500d16a0 0x7f89500d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.608+0000 7f894e800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f89500d2370 0x7f895015ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54844/0 (socket says 172.21.15.12:54844) 2024-03-29T09:54:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.608+0000 7f894e800700 1 -- 172.21.15.12:0/2760223663 learned_addr learned my addr 172.21.15.12:0/2760223663 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.608+0000 7f894de00700 1 --2- 172.21.15.12:0/2760223663 >> v2:172.21.15.159:3300/0 conn(0x7f895015d9d0 0x7f89500d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.609+0000 7f894d400700 1 -- 172.21.15.12:0/2760223663 >> v2:172.21.15.159:3300/0 conn(0x7f895015d9d0 msgr2=0x7f89500d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.609+0000 7f894d400700 1 --2- 172.21.15.12:0/2760223663 >> v2:172.21.15.159:3300/0 conn(0x7f895015d9d0 0x7f89500d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.609+0000 7f894d400700 1 -- 172.21.15.12:0/2760223663 >> v1:172.21.15.12:6789/0 conn(0x7f89500d2370 legacy=0x7f895015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.609+0000 7f894d400700 1 -- 172.21.15.12:0/2760223663 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89501619e0 con 0x7f89500d16a0 2024-03-29T09:54:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.609+0000 7f894ca00700 1 -- 172.21.15.12:0/2760223663 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405996488 0 0) 0x7f8950161420 con 0x7f89500d2370 2024-03-29T09:54:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.609+0000 7f894d400700 1 --2- 172.21.15.12:0/2760223663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d16a0 0x7f89500d1e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f89500d0ff0 tx=0x7f893c009b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cef6fa9835bbb7b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.609+0000 7f894ca00700 1 -- 172.21.15.12:0/2760223663 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c011830 con 0x7f89500d16a0 2024-03-29T09:54:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.609+0000 7f894ca00700 1 -- 172.21.15.12:0/2760223663 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c011990 con 0x7f89500d16a0 2024-03-29T09:54:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.609+0000 7f894ca00700 1 -- 172.21.15.12:0/2760223663 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c011d40 con 0x7f89500d16a0 2024-03-29T09:54:42.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.610+0000 7f8954c00700 1 -- 172.21.15.12:0/2760223663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d16a0 msgr2=0x7f89500d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.610+0000 7f8954c00700 1 --2- 172.21.15.12:0/2760223663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d16a0 0x7f89500d1e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f89500d0ff0 tx=0x7f893c009b10 comp rx=0 tx=0).stop 2024-03-29T09:54:42.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.610+0000 7f8954c00700 1 -- 172.21.15.12:0/2760223663 shutdown_connections 2024-03-29T09:54:42.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.610+0000 7f8954c00700 1 --2- 172.21.15.12:0/2760223663 >> v2:172.21.15.159:3300/0 conn(0x7f895015d9d0 0x7f89500d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.610+0000 7f8954c00700 1 --2- 172.21.15.12:0/2760223663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d16a0 0x7f89500d1e30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.610+0000 7f8954c00700 1 -- 172.21.15.12:0/2760223663 >> 172.21.15.12:0/2760223663 conn(0x7f895008aff0 msgr2=0x7f89500d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:42.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.610+0000 7f8954c00700 1 -- 172.21.15.12:0/2760223663 shutdown_connections 2024-03-29T09:54:42.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.610+0000 7f8954c00700 1 -- 172.21.15.12:0/2760223663 wait complete. 2024-03-29T09:54:42.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.612+0000 7f8954c00700 1 Processor -- start 2024-03-29T09:54:42.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.612+0000 7f8954c00700 1 -- start start 2024-03-29T09:54:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f8954c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500d16a0 0x7f89501f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f8954c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d2370 0x7f89500b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f8954c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895015d9d0 0x7f89500cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f8954c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89500b6a50 con 0x7f89500d2370 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f8954c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89500b6d30 con 0x7f895015d9d0 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f8954c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89500b7010 con 0x7f89500d16a0 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d2370 0x7f89500b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d2370 0x7f89500b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60674/0 (socket says 172.21.15.12:60674) 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894d400700 1 -- 172.21.15.12:0/2457163573 learned_addr learned my addr 172.21.15.12:0/2457163573 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894e800700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895015d9d0 0x7f89500cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894de00700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500d16a0 0x7f89501f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894d400700 1 -- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500d16a0 msgr2=0x7f89501f6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894d400700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500d16a0 0x7f89501f6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894d400700 1 -- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895015d9d0 msgr2=0x7f89500cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894d400700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895015d9d0 0x7f89500cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894d400700 1 -- 172.21.15.12:0/2457163573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c009440 con 0x7f89500d2370 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894de00700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500d16a0 0x7f89501f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.613+0000 7f894d400700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d2370 0x7f89500b6020 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f893c0090b0 tx=0x7f893c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.614+0000 7f8946000700 1 -- 172.21.15.12:0/2457163573 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c01a730 con 0x7f89500d2370 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.614+0000 7f8946000700 1 -- 172.21.15.12:0/2457163573 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c01a890 con 0x7f89500d2370 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.614+0000 7f8954c00700 1 -- 172.21.15.12:0/2457163573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89500b72f0 con 0x7f89500d2370 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.614+0000 7f8946000700 1 -- 172.21.15.12:0/2457163573 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c01ac00 con 0x7f89500d2370 2024-03-29T09:54:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.614+0000 7f8954c00700 1 -- 172.21.15.12:0/2457163573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89500b8a40 con 0x7f89500d2370 2024-03-29T09:54:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.615+0000 7f8954c00700 1 -- 172.21.15.12:0/2457163573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89500d1ca0 con 0x7f89500d2370 2024-03-29T09:54:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.616+0000 7f8946000700 1 -- 172.21.15.12:0/2457163573 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f893c022070 con 0x7f89500d2370 2024-03-29T09:54:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.617+0000 7f8946000700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f893006ac90 0x7f893006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.617+0000 7f8946000700 1 -- 172.21.15.12:0/2457163573 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f893c08a3b0 con 0x7f89500d2370 2024-03-29T09:54:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.617+0000 7f894de00700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f893006ac90 0x7f893006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.618+0000 7f894de00700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f893006ac90 0x7f893006d150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8950161580 tx=0x7f8940008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:42.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.621+0000 7f8946000700 1 -- 172.21.15.12:0/2457163573 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f893c0556a0 con 0x7f89500d2370 2024-03-29T09:54:42.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.833+0000 7f8954c00700 1 -- 172.21.15.12:0/2457163573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89500b7810 con 0x7f89500d2370 2024-03-29T09:54:42.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.833+0000 7f8946000700 1 -- 172.21.15.12:0/2457163573 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89500b7810 con 0x7f89500d2370 2024-03-29T09:54:42.833 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:54:42.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 -- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f893006ac90 msgr2=0x7f893006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f893006ac90 0x7f893006d150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8950161580 tx=0x7f8940008040 comp rx=0 tx=0).stop 2024-03-29T09:54:42.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 -- 172.21.15.12:0/2457163573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d2370 msgr2=0x7f89500b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:42.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d2370 0x7f89500b6020 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f893c0090b0 tx=0x7f893c004750 comp rx=0 tx=0).stop 2024-03-29T09:54:42.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 -- 172.21.15.12:0/2457163573 shutdown_connections 2024-03-29T09:54:42.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f893006ac90 0x7f893006d150 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895015d9d0 0x7f89500cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500d16a0 0x7f89501f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 --2- 172.21.15.12:0/2457163573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89500d2370 0x7f89500b6020 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:42.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 -- 172.21.15.12:0/2457163573 >> 172.21.15.12:0/2457163573 conn(0x7f895008aff0 msgr2=0x7f89500cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:42.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 -- 172.21.15.12:0/2457163573 shutdown_connections 2024-03-29T09:54:42.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:42.835+0000 7f8954c00700 1 -- 172.21.15.12:0/2457163573 wait complete. 2024-03-29T09:54:42.853 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:54:43.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.056+0000 7f246c600700 1 Processor -- start 2024-03-29T09:54:43.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.056+0000 7f246c600700 1 -- start start 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.057+0000 7f246c600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f246415d9d0 0x7f24640d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.057+0000 7f246c600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f24640d2540 0x7f24640ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.057+0000 7f246c600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2464161420 con 0x7f24640d16a0 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.057+0000 7f246c600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246415edc0 con 0x7f246415d9d0 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.057+0000 7f246c600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2464161700 con 0x7f24640d2540 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.057+0000 7f2469c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f246415d9d0 0x7f24640d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f2469c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f246415d9d0 0x7f24640d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60682/0 (socket says 172.21.15.12:60682) 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f2469200700 1 --1- 172.21.15.12:0/3656981399 >> v1:172.21.15.12:6789/0 conn(0x7f24640d16a0 0x7f24640d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54854/0 (socket says 172.21.15.12:54854) 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f2469c00700 1 -- 172.21.15.12:0/3656981399 learned_addr learned my addr 172.21.15.12:0/3656981399 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f246a600700 1 --2- 172.21.15.12:0/3656981399 >> v2:172.21.15.159:3300/0 conn(0x7f24640d2540 0x7f24640ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f2469c00700 1 -- 172.21.15.12:0/3656981399 >> v2:172.21.15.159:3300/0 conn(0x7f24640d2540 msgr2=0x7f24640ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f2469c00700 1 --2- 172.21.15.12:0/3656981399 >> v2:172.21.15.159:3300/0 conn(0x7f24640d2540 0x7f24640ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f2469c00700 1 -- 172.21.15.12:0/3656981399 >> v1:172.21.15.12:6789/0 conn(0x7f24640d16a0 legacy=0x7f24640d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f2469c00700 1 -- 172.21.15.12:0/3656981399 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24641619e0 con 0x7f246415d9d0 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f2463e00700 1 -- 172.21.15.12:0/3656981399 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833928278 0 0) 0x7f2464161420 con 0x7f24640d16a0 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f2469c00700 1 --2- 172.21.15.12:0/3656981399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246415d9d0 0x7f24640d3c30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f246408ca10 tx=0x7f2458009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdec5043537a9bd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.058+0000 7f2463e00700 1 -- 172.21.15.12:0/3656981399 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245800b870 con 0x7f246415d9d0 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.059+0000 7f2463e00700 1 -- 172.21.15.12:0/3656981399 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245800b9d0 con 0x7f246415d9d0 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.059+0000 7f2463e00700 1 -- 172.21.15.12:0/3656981399 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245800bd60 con 0x7f246415d9d0 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.059+0000 7f246c600700 1 -- 172.21.15.12:0/3656981399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246415d9d0 msgr2=0x7f24640d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.059+0000 7f246c600700 1 --2- 172.21.15.12:0/3656981399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246415d9d0 0x7f24640d3c30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f246408ca10 tx=0x7f2458009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.059+0000 7f246c600700 1 -- 172.21.15.12:0/3656981399 shutdown_connections 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.059+0000 7f246c600700 1 --2- 172.21.15.12:0/3656981399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246415d9d0 0x7f24640d3c30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.059+0000 7f246c600700 1 --2- 172.21.15.12:0/3656981399 >> v2:172.21.15.159:3300/0 conn(0x7f24640d2540 0x7f24640ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.059+0000 7f246c600700 1 -- 172.21.15.12:0/3656981399 >> 172.21.15.12:0/3656981399 conn(0x7f246408aff0 msgr2=0x7f24640d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.059+0000 7f246c600700 1 -- 172.21.15.12:0/3656981399 shutdown_connections 2024-03-29T09:54:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.059+0000 7f246c600700 1 -- 172.21.15.12:0/3656981399 wait complete. 2024-03-29T09:54:43.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.061+0000 7f246c600700 1 Processor -- start 2024-03-29T09:54:43.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.061+0000 7f246c600700 1 -- start start 2024-03-29T09:54:43.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f246c600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24640d16a0 0x7f24640b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f246c600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24640d2540 0x7f246415c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f246c600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246415d9d0 0x7f246415cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f246c600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24640b6580 con 0x7f246415d9d0 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f246c600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24640ca700 con 0x7f24640d16a0 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f246c600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24641e9a80 con 0x7f24640d2540 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f2469c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24640d16a0 0x7f24640b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f246a600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246415d9d0 0x7f246415cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f2469c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24640d16a0 0x7f24640b5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:41842/0 (socket says 172.21.15.12:41842) 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f2469c00700 1 -- 172.21.15.12:0/3739687556 learned_addr learned my addr 172.21.15.12:0/3739687556 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.062+0000 7f2469200700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24640d2540 0x7f246415c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f246a600700 1 -- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24640d2540 msgr2=0x7f246415c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f246a600700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24640d2540 0x7f246415c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f246a600700 1 -- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24640d16a0 msgr2=0x7f24640b5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f246a600700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24640d16a0 0x7f24640b5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f246a600700 1 -- 172.21.15.12:0/3739687556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2458009550 con 0x7f246415d9d0 2024-03-29T09:54:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f2469c00700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24640d16a0 0x7f24640b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:54:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f246a600700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246415d9d0 0x7f246415cbd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f245c007d90 tx=0x7f245c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f2461600700 1 -- 172.21.15.12:0/3739687556 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c018070 con 0x7f246415d9d0 2024-03-29T09:54:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f2461600700 1 -- 172.21.15.12:0/3739687556 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245c00eb30 con 0x7f246415d9d0 2024-03-29T09:54:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f246c600700 1 -- 172.21.15.12:0/3739687556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24641e9d60 con 0x7f246415d9d0 2024-03-29T09:54:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f2461600700 1 -- 172.21.15.12:0/3739687556 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00eec0 con 0x7f246415d9d0 2024-03-29T09:54:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.063+0000 7f246c600700 1 -- 172.21.15.12:0/3739687556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24640b5670 con 0x7f246415d9d0 2024-03-29T09:54:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.064+0000 7f246c600700 1 -- 172.21.15.12:0/3739687556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24640d1ca0 con 0x7f246415d9d0 2024-03-29T09:54:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.065+0000 7f2461600700 1 -- 172.21.15.12:0/3739687556 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f245c014420 con 0x7f246415d9d0 2024-03-29T09:54:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.066+0000 7f2461600700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f244406ac90 0x7f244406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.066+0000 7f2461600700 1 -- 172.21.15.12:0/3739687556 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f245c0890e0 con 0x7f246415d9d0 2024-03-29T09:54:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.067+0000 7f2469c00700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f244406ac90 0x7f244406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.067+0000 7f2469c00700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f244406ac90 0x7f244406d150 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2458009c10 tx=0x7f2458008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:43.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.071+0000 7f2461600700 1 -- 172.21.15.12:0/3739687556 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f245c0bc580 con 0x7f246415d9d0 2024-03-29T09:54:43.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.285+0000 7f246c600700 1 -- 172.21.15.12:0/3739687556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24640b5830 con 0x7f246415d9d0 2024-03-29T09:54:43.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.285+0000 7f2461600700 1 -- 172.21.15.12:0/3739687556 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f245c054220 con 0x7f246415d9d0 2024-03-29T09:54:43.285 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 -- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f244406ac90 msgr2=0x7f244406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f244406ac90 0x7f244406d150 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2458009c10 tx=0x7f2458008030 comp rx=0 tx=0).stop 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 -- 172.21.15.12:0/3739687556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246415d9d0 msgr2=0x7f246415cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246415d9d0 0x7f246415cbd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f245c007d90 tx=0x7f245c00e3c0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 -- 172.21.15.12:0/3739687556 shutdown_connections 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f244406ac90 0x7f244406d150 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246415d9d0 0x7f246415cbd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24640d16a0 0x7f24640b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 --2- 172.21.15.12:0/3739687556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24640d2540 0x7f246415c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 -- 172.21.15.12:0/3739687556 >> 172.21.15.12:0/3739687556 conn(0x7f246408aff0 msgr2=0x7f24640d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 -- 172.21.15.12:0/3739687556 shutdown_connections 2024-03-29T09:54:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.288+0000 7f246c600700 1 -- 172.21.15.12:0/3739687556 wait complete. 2024-03-29T09:54:43.306 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:54:43.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.508+0000 7f2d98c00700 1 Processor -- start 2024-03-29T09:54:43.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.508+0000 7f2d98c00700 1 -- start start 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.509+0000 7f2d98c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2d94160f10 0x7f2d940d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.509+0000 7f2d98c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d940cf230 0x7f2d940d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.509+0000 7f2d98c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d94161590 0x7f2d94161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.509+0000 7f2d98c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d94161f60 con 0x7f2d940cf230 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.509+0000 7f2d98c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d940ceda0 con 0x7f2d94160f10 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.509+0000 7f2d98c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d940d23d0 con 0x7f2d94161590 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d91400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d940cf230 0x7f2d940d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d91400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d940cf230 0x7f2d940d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60686/0 (socket says 172.21.15.12:60686) 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d91400700 1 -- 172.21.15.12:0/1611400602 learned_addr learned my addr 172.21.15.12:0/1611400602 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d92800700 1 --2- 172.21.15.12:0/1611400602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d94161590 0x7f2d94161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d91e00700 1 --2- 172.21.15.12:0/1611400602 >> v2:172.21.15.159:3300/0 conn(0x7f2d94160f10 0x7f2d940d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d91400700 1 -- 172.21.15.12:0/1611400602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d94161590 msgr2=0x7f2d94161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:43.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d91400700 1 --2- 172.21.15.12:0/1611400602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d94161590 0x7f2d94161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d91400700 1 -- 172.21.15.12:0/1611400602 >> v2:172.21.15.159:3300/0 conn(0x7f2d94160f10 msgr2=0x7f2d940d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:43.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d91400700 1 --2- 172.21.15.12:0/1611400602 >> v2:172.21.15.159:3300/0 conn(0x7f2d94160f10 0x7f2d940d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d91400700 1 -- 172.21.15.12:0/1611400602 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d941612f0 con 0x7f2d940cf230 2024-03-29T09:54:43.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.510+0000 7f2d91400700 1 --2- 172.21.15.12:0/1611400602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d940cf230 0x7f2d940d1e90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2d940d1080 tx=0x7f2d80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7af0181465ecf9a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:43.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d90a00700 1 -- 172.21.15.12:0/1611400602 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8000b870 con 0x7f2d940cf230 2024-03-29T09:54:43.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d90a00700 1 -- 172.21.15.12:0/1611400602 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8000b9d0 con 0x7f2d940cf230 2024-03-29T09:54:43.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d90a00700 1 -- 172.21.15.12:0/1611400602 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8000bd60 con 0x7f2d940cf230 2024-03-29T09:54:43.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d98c00700 1 -- 172.21.15.12:0/1611400602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d940cf230 msgr2=0x7f2d940d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d98c00700 1 --2- 172.21.15.12:0/1611400602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d940cf230 0x7f2d940d1e90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2d940d1080 tx=0x7f2d80009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d98c00700 1 -- 172.21.15.12:0/1611400602 shutdown_connections 2024-03-29T09:54:43.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d98c00700 1 --2- 172.21.15.12:0/1611400602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d94161590 0x7f2d94161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d98c00700 1 --2- 172.21.15.12:0/1611400602 >> v2:172.21.15.159:3300/0 conn(0x7f2d94160f10 0x7f2d940d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d98c00700 1 --2- 172.21.15.12:0/1611400602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d940cf230 0x7f2d940d1e90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d98c00700 1 -- 172.21.15.12:0/1611400602 >> 172.21.15.12:0/1611400602 conn(0x7f2d9408aff0 msgr2=0x7f2d940d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:43.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d98c00700 1 -- 172.21.15.12:0/1611400602 shutdown_connections 2024-03-29T09:54:43.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.511+0000 7f2d98c00700 1 -- 172.21.15.12:0/1611400602 wait complete. 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.513+0000 7f2d98c00700 1 Processor -- start 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.513+0000 7f2d98c00700 1 -- start start 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d98c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d940cf230 0x7f2d9414ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d98c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d94160f10 0x7f2d9415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d98c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d94161590 0x7f2d9415f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d98c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d941568d0 con 0x7f2d94160f10 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d98c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9415fd00 con 0x7f2d940cf230 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d98c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9415ffe0 con 0x7f2d94161590 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d91400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d94160f10 0x7f2d9415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d91400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d94160f10 0x7f2d9415ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60698/0 (socket says 172.21.15.12:60698) 2024-03-29T09:54:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d91400700 1 -- 172.21.15.12:0/1706804268 learned_addr learned my addr 172.21.15.12:0/1706804268 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d92800700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d94161590 0x7f2d9415f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.514+0000 7f2d91e00700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d940cf230 0x7f2d9414ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.515+0000 7f2d91400700 1 -- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d94161590 msgr2=0x7f2d9415f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.515+0000 7f2d91400700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d94161590 0x7f2d9415f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.515+0000 7f2d91400700 1 -- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d940cf230 msgr2=0x7f2d9414ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.515+0000 7f2d91400700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d940cf230 0x7f2d9414ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.515+0000 7f2d91400700 1 -- 172.21.15.12:0/1706804268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d80009550 con 0x7f2d94160f10 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.515+0000 7f2d91400700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d94160f10 0x7f2d9415ef90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2d80000c00 tx=0x7f2d80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.515+0000 7f2d8a000700 1 -- 172.21.15.12:0/1706804268 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d80004310 con 0x7f2d94160f10 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.515+0000 7f2d8a000700 1 -- 172.21.15.12:0/1706804268 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8001e070 con 0x7f2d94160f10 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.515+0000 7f2d8a000700 1 -- 172.21.15.12:0/1706804268 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d800036a0 con 0x7f2d94160f10 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.515+0000 7f2d98c00700 1 -- 172.21.15.12:0/1706804268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d94156280 con 0x7f2d94160f10 2024-03-29T09:54:43.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.516+0000 7f2d98c00700 1 -- 172.21.15.12:0/1706804268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d9415fa10 con 0x7f2d94160f10 2024-03-29T09:54:43.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.516+0000 7f2d98c00700 1 -- 172.21.15.12:0/1706804268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d94156a80 con 0x7f2d94160f10 2024-03-29T09:54:43.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.517+0000 7f2d8a000700 1 -- 172.21.15.12:0/1706804268 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2d80020030 con 0x7f2d94160f10 2024-03-29T09:54:43.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.518+0000 7f2d8a000700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2d7406acc0 0x7f2d7406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:43.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.518+0000 7f2d8a000700 1 -- 172.21.15.12:0/1706804268 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d8008a1b0 con 0x7f2d94160f10 2024-03-29T09:54:43.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.519+0000 7f2d91e00700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2d7406acc0 0x7f2d7406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:43.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.519+0000 7f2d91e00700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2d7406acc0 0x7f2d7406d180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2d84006b20 tx=0x7f2d84008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:43.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.523+0000 7f2d8a000700 1 -- 172.21.15.12:0/1706804268 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d94156a80 con 0x7f2d94160f10 2024-03-29T09:54:43.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.781+0000 7f2d98c00700 1 -- 172.21.15.12:0/1706804268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d94067720 con 0x7f2d94160f10 2024-03-29T09:54:43.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.781+0000 7f2d8a000700 1 -- 172.21.15.12:0/1706804268 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7f2d80055380 con 0x7f2d94160f10 2024-03-29T09:54:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:43.781 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/4093246304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4093246304},{"type":"v1","addr":"172.21.15.12:6833","nonce":4093246304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1631416176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1631416176},{"type":"v1","addr":"172.21.15.159:6836","nonce":1631416176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3828834681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3828834681},{"type":"v1","addr":"172.21.15.159:6837","nonce":3828834681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:54:27.425500+0000","modified":"2024-03-29T09:54:30.895316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3922451359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3922451359},{"type":"v1","addr":"172.21.15.12:6835","nonce":3922451359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 -- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2d7406acc0 msgr2=0x7f2d7406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2d7406acc0 0x7f2d7406d180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2d84006b20 tx=0x7f2d84008040 comp rx=0 tx=0).stop 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 -- 172.21.15.12:0/1706804268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d94160f10 msgr2=0x7f2d9415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d94160f10 0x7f2d9415ef90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2d80000c00 tx=0x7f2d80004750 comp rx=0 tx=0).stop 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 -- 172.21.15.12:0/1706804268 shutdown_connections 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2d7406acc0 0x7f2d7406d180 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d94161590 0x7f2d9415f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d940cf230 0x7f2d9414ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 --2- 172.21.15.12:0/1706804268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d94160f10 0x7f2d9415ef90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 -- 172.21.15.12:0/1706804268 >> 172.21.15.12:0/1706804268 conn(0x7f2d9408aff0 msgr2=0x7f2d940cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 -- 172.21.15.12:0/1706804268 shutdown_connections 2024-03-29T09:54:43.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.784+0000 7f2d98c00700 1 -- 172.21.15.12:0/1706804268 wait complete. 2024-03-29T09:54:43.785 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-03-29T09:54:43.801 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:43.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.999+0000 7f1566a00700 1 Processor -- start 2024-03-29T09:54:43.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:43.999+0000 7f1566a00700 1 -- start start 2024-03-29T09:54:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.000+0000 7f1566a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1560161ca0 0x7f15600cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.000+0000 7f1566a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15600ceda0 con 0x7f1560160f10 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.000+0000 7f1566a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15601612f0 con 0x7f15600d18b0 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.000+0000 7f1566a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15600d1d30 con 0x7f1560161ca0 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.001+0000 7f1564a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1560161ca0 0x7f15600cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.001+0000 7f155fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1560160f10 0x7f15600d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54862/0 (socket says 172.21.15.12:54862) 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.001+0000 7f155fe00700 1 -- 172.21.15.12:0/1370021358 learned_addr learned my addr 172.21.15.12:0/1370021358 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.001+0000 7f1564a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1560161ca0 0x7f15600cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60712/0 (socket says 172.21.15.12:60712) 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.001+0000 7f1564a00700 1 -- 172.21.15.12:0/1370021358 >> v1:172.21.15.159:6789/0 conn(0x7f15600d18b0 legacy=0x7f1560161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.001+0000 7f1564a00700 1 -- 172.21.15.12:0/1370021358 >> v1:172.21.15.12:6789/0 conn(0x7f1560160f10 legacy=0x7f15600d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.001+0000 7f155ea00700 1 -- 172.21.15.12:0/1370021358 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263998526 0 0) 0x7f15600ceda0 con 0x7f1560160f10 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.001+0000 7f1564a00700 1 -- 172.21.15.12:0/1370021358 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15600ce470 con 0x7f1560161ca0 2024-03-29T09:54:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.001+0000 7f1564a00700 1 --2- 172.21.15.12:0/1370021358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1560161ca0 0x7f15600cf230 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1558009d90 tx=0x7f155800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ce4bcc0c6dff1b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:44.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.002+0000 7f155ea00700 1 -- 172.21.15.12:0/1370021358 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155800e070 con 0x7f1560161ca0 2024-03-29T09:54:44.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.002+0000 7f155ea00700 1 -- 172.21.15.12:0/1370021358 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155800bbd0 con 0x7f1560161ca0 2024-03-29T09:54:44.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.002+0000 7f155ea00700 1 -- 172.21.15.12:0/1370021358 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1558012430 con 0x7f1560161ca0 2024-03-29T09:54:44.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.002+0000 7f1566a00700 1 -- 172.21.15.12:0/1370021358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1560161ca0 msgr2=0x7f15600cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.002+0000 7f1566a00700 1 --2- 172.21.15.12:0/1370021358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1560161ca0 0x7f15600cf230 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1558009d90 tx=0x7f155800b330 comp rx=0 tx=0).stop 2024-03-29T09:54:44.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.002+0000 7f1566a00700 1 -- 172.21.15.12:0/1370021358 shutdown_connections 2024-03-29T09:54:44.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.002+0000 7f1566a00700 1 --2- 172.21.15.12:0/1370021358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1560161ca0 0x7f15600cf230 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.002+0000 7f1566a00700 1 -- 172.21.15.12:0/1370021358 >> 172.21.15.12:0/1370021358 conn(0x7f156008aff0 msgr2=0x7f15600d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:44.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.002+0000 7f1566a00700 1 -- 172.21.15.12:0/1370021358 shutdown_connections 2024-03-29T09:54:44.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.002+0000 7f1566a00700 1 -- 172.21.15.12:0/1370021358 wait complete. 2024-03-29T09:54:44.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.004+0000 7f1566a00700 1 Processor -- start 2024-03-29T09:54:44.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.004+0000 7f1566a00700 1 -- start start 2024-03-29T09:54:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f1566a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15600d18b0 0x7f15601588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f1566a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1560160f10 0x7f15601e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f1566a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1560161ca0 0x7f15600b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f1566a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15601e87a0 con 0x7f15600d18b0 2024-03-29T09:54:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f1566a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15601e8310 con 0x7f1560160f10 2024-03-29T09:54:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f1566a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15601e7700 con 0x7f1560161ca0 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f155fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15600d18b0 0x7f15601588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f155fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15600d18b0 0x7f15601588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60720/0 (socket says 172.21.15.12:60720) 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f155fe00700 1 -- 172.21.15.12:0/2215760762 learned_addr learned my addr 172.21.15.12:0/2215760762 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f155f400700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1560160f10 0x7f15601e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.005+0000 7f1564a00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1560161ca0 0x7f15600b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f155fe00700 1 -- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1560161ca0 msgr2=0x7f15600b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f155fe00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1560161ca0 0x7f15600b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f155fe00700 1 -- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1560160f10 msgr2=0x7f15601e6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f155fe00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1560160f10 0x7f15601e6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f155fe00700 1 -- 172.21.15.12:0/2215760762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1558009a70 con 0x7f15600d18b0 2024-03-29T09:54:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f155fe00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15600d18b0 0x7f15601588d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f155000c550 tx=0x7f155000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:44.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f1557e00700 1 -- 172.21.15.12:0/2215760762 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550004c80 con 0x7f15600d18b0 2024-03-29T09:54:44.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f1557e00700 1 -- 172.21.15.12:0/2215760762 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1550004de0 con 0x7f15600d18b0 2024-03-29T09:54:44.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f1566a00700 1 -- 172.21.15.12:0/2215760762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15601e79e0 con 0x7f15600d18b0 2024-03-29T09:54:44.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f1557e00700 1 -- 172.21.15.12:0/2215760762 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550007310 con 0x7f15600d18b0 2024-03-29T09:54:44.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.006+0000 7f1566a00700 1 -- 172.21.15.12:0/2215760762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15601e7fa0 con 0x7f15600d18b0 2024-03-29T09:54:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.007+0000 7f1566a00700 1 -- 172.21.15.12:0/2215760762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1560067720 con 0x7f15600d18b0 2024-03-29T09:54:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.008+0000 7f1557e00700 1 -- 172.21.15.12:0/2215760762 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1550007470 con 0x7f15600d18b0 2024-03-29T09:54:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.009+0000 7f1557e00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f154006ac90 0x7f154006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.009+0000 7f1557e00700 1 -- 172.21.15.12:0/2215760762 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1550088ed0 con 0x7f15600d18b0 2024-03-29T09:54:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.010+0000 7f155f400700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f154006ac90 0x7f154006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.010+0000 7f155f400700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f154006ac90 0x7f154006d150 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f156006cd80 tx=0x7f154c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.014+0000 7f1557e00700 1 -- 172.21.15.12:0/2215760762 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15500540a0 con 0x7f15600d18b0 2024-03-29T09:54:44.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.230+0000 7f1566a00700 1 -- 172.21.15.12:0/2215760762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15601e85f0 con 0x7f15600d18b0 2024-03-29T09:54:44.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.231+0000 7f1557e00700 1 -- 172.21.15.12:0/2215760762 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f15500576c0 con 0x7f15600d18b0 2024-03-29T09:54:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:44.231 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.233+0000 7f1566a00700 1 -- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f154006ac90 msgr2=0x7f154006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.233+0000 7f1566a00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f154006ac90 0x7f154006d150 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f156006cd80 tx=0x7f154c009040 comp rx=0 tx=0).stop 2024-03-29T09:54:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.233+0000 7f1566a00700 1 -- 172.21.15.12:0/2215760762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15600d18b0 msgr2=0x7f15601588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.233+0000 7f1566a00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15600d18b0 0x7f15601588d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f155000c550 tx=0x7f155000c860 comp rx=0 tx=0).stop 2024-03-29T09:54:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.234+0000 7f1566a00700 1 -- 172.21.15.12:0/2215760762 shutdown_connections 2024-03-29T09:54:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.234+0000 7f1566a00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f154006ac90 0x7f154006d150 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.234+0000 7f1566a00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1560161ca0 0x7f15600b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.234+0000 7f1566a00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15600d18b0 0x7f15601588d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.234+0000 7f1566a00700 1 --2- 172.21.15.12:0/2215760762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1560160f10 0x7f15601e6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.234+0000 7f1566a00700 1 -- 172.21.15.12:0/2215760762 >> 172.21.15.12:0/2215760762 conn(0x7f156008aff0 msgr2=0x7f15600d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.234+0000 7f1566a00700 1 -- 172.21.15.12:0/2215760762 shutdown_connections 2024-03-29T09:54:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.234+0000 7f1566a00700 1 -- 172.21.15.12:0/2215760762 wait complete. 2024-03-29T09:54:44.252 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:54:44.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.449+0000 7fd844600700 1 Processor -- start 2024-03-29T09:54:44.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.449+0000 7fd844600700 1 -- start start 2024-03-29T09:54:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.450+0000 7fd844600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd83c161ca0 0x7fd83c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.450+0000 7fd844600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c0ceda0 con 0x7fd83c0d18b0 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.450+0000 7fd844600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c1612f0 con 0x7fd83c160f10 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.450+0000 7fd844600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83c0d1d30 con 0x7fd83c161ca0 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.450+0000 7fd841200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd83c0d18b0 0x7fd83c161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54864/0 (socket says 172.21.15.12:54864) 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.450+0000 7fd841200700 1 -- 172.21.15.12:0/3205504495 learned_addr learned my addr 172.21.15.12:0/3205504495 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.450+0000 7fd842600700 1 --2- 172.21.15.12:0/3205504495 >> v2:172.21.15.12:3300/0 conn(0x7fd83c161ca0 0x7fd83c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.451+0000 7fd83be00700 1 -- 172.21.15.12:0/3205504495 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580340378 0 0) 0x7fd83c0ceda0 con 0x7fd83c0d18b0 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.451+0000 7fd83be00700 1 -- 172.21.15.12:0/3205504495 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8240035b0 con 0x7fd83c0d18b0 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.451+0000 7fd842600700 1 -- 172.21.15.12:0/3205504495 >> v1:172.21.15.159:6789/0 conn(0x7fd83c160f10 legacy=0x7fd83c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.451+0000 7fd842600700 1 -- 172.21.15.12:0/3205504495 >> v1:172.21.15.12:6789/0 conn(0x7fd83c0d18b0 legacy=0x7fd83c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.451+0000 7fd842600700 1 -- 172.21.15.12:0/3205504495 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c0ce470 con 0x7fd83c161ca0 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.451+0000 7fd842600700 1 --2- 172.21.15.12:0/3205504495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c161ca0 0x7fd83c0cf230 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd834009d90 tx=0x7fd83400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5619bcf18f219139 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.451+0000 7fd83be00700 1 -- 172.21.15.12:0/3205504495 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83400e070 con 0x7fd83c161ca0 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.451+0000 7fd83be00700 1 -- 172.21.15.12:0/3205504495 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83400bb20 con 0x7fd83c161ca0 2024-03-29T09:54:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.451+0000 7fd83be00700 1 -- 172.21.15.12:0/3205504495 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834012430 con 0x7fd83c161ca0 2024-03-29T09:54:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.452+0000 7fd844600700 1 -- 172.21.15.12:0/3205504495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c161ca0 msgr2=0x7fd83c0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.452+0000 7fd844600700 1 --2- 172.21.15.12:0/3205504495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c161ca0 0x7fd83c0cf230 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd834009d90 tx=0x7fd83400b330 comp rx=0 tx=0).stop 2024-03-29T09:54:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.452+0000 7fd844600700 1 -- 172.21.15.12:0/3205504495 shutdown_connections 2024-03-29T09:54:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.452+0000 7fd844600700 1 --2- 172.21.15.12:0/3205504495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c161ca0 0x7fd83c0cf230 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.452+0000 7fd844600700 1 -- 172.21.15.12:0/3205504495 >> 172.21.15.12:0/3205504495 conn(0x7fd83c08aff0 msgr2=0x7fd83c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.452+0000 7fd844600700 1 -- 172.21.15.12:0/3205504495 shutdown_connections 2024-03-29T09:54:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.452+0000 7fd844600700 1 -- 172.21.15.12:0/3205504495 wait complete. 2024-03-29T09:54:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.454+0000 7fd844600700 1 Processor -- start 2024-03-29T09:54:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.454+0000 7fd844600700 1 -- start start 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd844600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd83c0d18b0 0x7fd83c14eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd844600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c160f10 0x7fd83c09e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd844600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd83c161ca0 0x7fd83c1f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd844600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c14f060 con 0x7fd83c160f10 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd844600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c09eef0 con 0x7fd83c0d18b0 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd844600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c09f1d0 con 0x7fd83c161ca0 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd841200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c160f10 0x7fd83c09e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd841200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c160f10 0x7fd83c09e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60746/0 (socket says 172.21.15.12:60746) 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd841200700 1 -- 172.21.15.12:0/3071854539 learned_addr learned my addr 172.21.15.12:0/3071854539 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd841c00700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd83c0d18b0 0x7fd83c14eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd842600700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd83c161ca0 0x7fd83c1f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd841200700 1 -- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd83c161ca0 msgr2=0x7fd83c1f6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd841200700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd83c161ca0 0x7fd83c1f6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd841200700 1 -- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd83c0d18b0 msgr2=0x7fd83c14eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd841200700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd83c0d18b0 0x7fd83c14eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.456+0000 7fd841200700 1 -- 172.21.15.12:0/3071854539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd834009a70 con 0x7fd83c160f10 2024-03-29T09:54:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.457+0000 7fd841200700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c160f10 0x7fd83c09e4e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd828001eb0 tx=0x7fd82800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.457+0000 7fd839600700 1 -- 172.21.15.12:0/3071854539 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd828005d70 con 0x7fd83c160f10 2024-03-29T09:54:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.457+0000 7fd839600700 1 -- 172.21.15.12:0/3071854539 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8280070d0 con 0x7fd83c160f10 2024-03-29T09:54:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.457+0000 7fd844600700 1 -- 172.21.15.12:0/3071854539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83c09f4b0 con 0x7fd83c160f10 2024-03-29T09:54:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.457+0000 7fd839600700 1 -- 172.21.15.12:0/3071854539 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd828007460 con 0x7fd83c160f10 2024-03-29T09:54:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.457+0000 7fd844600700 1 -- 172.21.15.12:0/3071854539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83c09fa70 con 0x7fd83c160f10 2024-03-29T09:54:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.458+0000 7fd844600700 1 -- 172.21.15.12:0/3071854539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83c067720 con 0x7fd83c160f10 2024-03-29T09:54:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.459+0000 7fd839600700 1 -- 172.21.15.12:0/3071854539 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd8280075c0 con 0x7fd83c160f10 2024-03-29T09:54:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.460+0000 7fd839600700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd81c06ac90 0x7fd81c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.460+0000 7fd839600700 1 -- 172.21.15.12:0/3071854539 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8280887b0 con 0x7fd83c160f10 2024-03-29T09:54:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.460+0000 7fd841c00700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd81c06ac90 0x7fd81c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:44.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.461+0000 7fd841c00700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd81c06ac90 0x7fd81c06d150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd830007fd0 tx=0x7fd830009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.464+0000 7fd839600700 1 -- 172.21.15.12:0/3071854539 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8280538f0 con 0x7fd83c160f10 2024-03-29T09:54:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.726+0000 7fd844600700 1 -- 172.21.15.12:0/3071854539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd83c0cced0 con 0x7fd83c160f10 2024-03-29T09:54:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.726+0000 7fd839600700 1 -- 172.21.15.12:0/3071854539 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7fd828056f10 con 0x7fd83c160f10 2024-03-29T09:54:44.726 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:44.726 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/4093246304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4093246304},{"type":"v1","addr":"172.21.15.12:6833","nonce":4093246304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1631416176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1631416176},{"type":"v1","addr":"172.21.15.159:6836","nonce":1631416176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3828834681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3828834681},{"type":"v1","addr":"172.21.15.159:6837","nonce":3828834681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:54:27.425500+0000","modified":"2024-03-29T09:54:30.895316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3922451359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3922451359},{"type":"v1","addr":"172.21.15.12:6835","nonce":3922451359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-29T09:54:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 -- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd81c06ac90 msgr2=0x7fd81c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd81c06ac90 0x7fd81c06d150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd830007fd0 tx=0x7fd830009040 comp rx=0 tx=0).stop 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 -- 172.21.15.12:0/3071854539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c160f10 msgr2=0x7fd83c09e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c160f10 0x7fd83c09e4e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd828001eb0 tx=0x7fd82800c7b0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 -- 172.21.15.12:0/3071854539 shutdown_connections 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd81c06ac90 0x7fd81c06d150 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd83c161ca0 0x7fd83c1f6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd83c0d18b0 0x7fd83c14eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 --2- 172.21.15.12:0/3071854539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c160f10 0x7fd83c09e4e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 -- 172.21.15.12:0/3071854539 >> 172.21.15.12:0/3071854539 conn(0x7fd83c08aff0 msgr2=0x7fd83c0d2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 -- 172.21.15.12:0/3071854539 shutdown_connections 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:44.729+0000 7fd844600700 1 -- 172.21.15.12:0/3071854539 wait complete. 2024-03-29T09:54:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-03-29T09:54:45.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T09:54:27.425500+0000', 'modified': '2024-03-29T09:54:30.895316+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6835/3922451359', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6834', 'nonce': 3922451359}, {'type': 'v1', 'addr': '172.21.15.12:6835', 'nonce': 3922451359}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T09:54:45.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T09:54:45.149 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-29T09:54:45.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.352+0000 7f9baaa00700 1 Processor -- start 2024-03-29T09:54:45.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.352+0000 7f9baaa00700 1 -- start start 2024-03-29T09:54:45.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.353+0000 7f9baaa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9ba40cd5c0 0x7f9ba415e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.353+0000 7f9baaa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ba406f310 0x7f9ba406f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.353+0000 7f9baaa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba415eda0 con 0x7f9ba415cfe0 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.353+0000 7f9baaa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba406fce0 con 0x7f9ba406f310 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.353+0000 7f9baaa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba4099dd0 con 0x7f9ba40cd5c0 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.353+0000 7f9ba3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ba406f310 0x7f9ba406f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.353+0000 7f9ba3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ba406f310 0x7f9ba406f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60748/0 (socket says 172.21.15.12:60748) 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.353+0000 7f9ba3400700 1 -- 172.21.15.12:0/638284469 learned_addr learned my addr 172.21.15.12:0/638284469 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.354+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/638284469 >> v2:172.21.15.159:3300/0 conn(0x7f9ba40cd5c0 0x7f9ba415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.354+0000 7f9ba3400700 1 -- 172.21.15.12:0/638284469 >> v2:172.21.15.159:3300/0 conn(0x7f9ba40cd5c0 msgr2=0x7f9ba415e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.354+0000 7f9ba3400700 1 --2- 172.21.15.12:0/638284469 >> v2:172.21.15.159:3300/0 conn(0x7f9ba40cd5c0 0x7f9ba415e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.354+0000 7f9ba3400700 1 -- 172.21.15.12:0/638284469 >> v1:172.21.15.159:6789/0 conn(0x7f9ba415cfe0 legacy=0x7f9ba415d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.354+0000 7f9ba3400700 1 -- 172.21.15.12:0/638284469 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba409a0b0 con 0x7f9ba406f310 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.354+0000 7f9ba3400700 1 --2- 172.21.15.12:0/638284469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 0x7f9ba406f7a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9b900098f0 tx=0x7f9b90009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25de84675013a4ea server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.354+0000 7f9ba2a00700 1 -- 172.21.15.12:0/638284469 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b90004230 con 0x7f9ba406f310 2024-03-29T09:54:45.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.354+0000 7f9ba2a00700 1 -- 172.21.15.12:0/638284469 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b90004390 con 0x7f9ba406f310 2024-03-29T09:54:45.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.354+0000 7f9ba2a00700 1 -- 172.21.15.12:0/638284469 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b900036a0 con 0x7f9ba406f310 2024-03-29T09:54:45.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.355+0000 7f9baaa00700 1 -- 172.21.15.12:0/638284469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 msgr2=0x7f9ba406f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:45.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.355+0000 7f9baaa00700 1 --2- 172.21.15.12:0/638284469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 0x7f9ba406f7a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9b900098f0 tx=0x7f9b90009c00 comp rx=0 tx=0).stop 2024-03-29T09:54:45.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.355+0000 7f9baaa00700 1 -- 172.21.15.12:0/638284469 shutdown_connections 2024-03-29T09:54:45.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.355+0000 7f9baaa00700 1 --2- 172.21.15.12:0/638284469 >> v2:172.21.15.159:3300/0 conn(0x7f9ba40cd5c0 0x7f9ba415e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.355+0000 7f9baaa00700 1 --2- 172.21.15.12:0/638284469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 0x7f9ba406f7a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.355+0000 7f9baaa00700 1 -- 172.21.15.12:0/638284469 >> 172.21.15.12:0/638284469 conn(0x7f9ba40d1370 msgr2=0x7f9ba40d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:45.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.355+0000 7f9baaa00700 1 -- 172.21.15.12:0/638284469 shutdown_connections 2024-03-29T09:54:45.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.355+0000 7f9baaa00700 1 -- 172.21.15.12:0/638284469 wait complete. 2024-03-29T09:54:45.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.357+0000 7f9baaa00700 1 Processor -- start 2024-03-29T09:54:45.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.357+0000 7f9baaa00700 1 -- start start 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9baaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 0x7f9ba41549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9baaa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ba40cd5c0 0x7f9ba415af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9baaa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ba415cfe0 0x7f9ba4158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9baaa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba415bcc0 con 0x7f9ba406f310 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9baaa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba41f3660 con 0x7f9ba415cfe0 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9baaa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba415b4d0 con 0x7f9ba40cd5c0 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9ba3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 0x7f9ba41549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9ba3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 0x7f9ba41549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60750/0 (socket says 172.21.15.12:60750) 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9ba3e00700 1 -- 172.21.15.12:0/2828662810 learned_addr learned my addr 172.21.15.12:0/2828662810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9ba3400700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ba40cd5c0 0x7f9ba415af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9ba8a00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ba415cfe0 0x7f9ba4158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9ba3e00700 1 -- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ba40cd5c0 msgr2=0x7f9ba415af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ba40cd5c0 0x7f9ba415af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9ba3e00700 1 -- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ba415cfe0 msgr2=0x7f9ba4158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:45.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ba415cfe0 0x7f9ba4158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.358+0000 7f9ba3e00700 1 -- 172.21.15.12:0/2828662810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b90009550 con 0x7f9ba406f310 2024-03-29T09:54:45.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.359+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 0x7f9ba41549d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400e810 tx=0x7f9b9400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:45.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.359+0000 7f9b9be00700 1 -- 172.21.15.12:0/2828662810 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9400c070 con 0x7f9ba406f310 2024-03-29T09:54:45.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.359+0000 7f9b9be00700 1 -- 172.21.15.12:0/2828662810 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b94013610 con 0x7f9ba406f310 2024-03-29T09:54:45.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.359+0000 7f9b9be00700 1 -- 172.21.15.12:0/2828662810 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b940139c0 con 0x7f9ba406f310 2024-03-29T09:54:45.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.359+0000 7f9baaa00700 1 -- 172.21.15.12:0/2828662810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba4158680 con 0x7f9ba406f310 2024-03-29T09:54:45.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.359+0000 7f9baaa00700 1 -- 172.21.15.12:0/2828662810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba415b9e0 con 0x7f9ba406f310 2024-03-29T09:54:45.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.361+0000 7f9b9be00700 1 -- 172.21.15.12:0/2828662810 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9b94013b20 con 0x7f9ba406f310 2024-03-29T09:54:45.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.362+0000 7f9b9be00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9b8406ac90 0x7f9b8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.362+0000 7f9b9be00700 1 -- 172.21.15.12:0/2828662810 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b94089100 con 0x7f9ba406f310 2024-03-29T09:54:45.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.362+0000 7f9ba3400700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9b8406ac90 0x7f9b8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.363+0000 7f9ba3400700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9b8406ac90 0x7f9b8406d150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009130 tx=0x7f9b90008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:45.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.364+0000 7f9bbe4d4b80 1 Processor -- start 2024-03-29T09:54:45.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.364+0000 7f9bbe4d4b80 1 -- start start 2024-03-29T09:54:45.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.365+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c80025830 0x557c80025c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.365+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fffba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.365+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c7fff6b60 0x557c80022a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c7ff70650 con 0x557c7fff8be0 2024-03-29T09:54:45.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c7ff51c30 con 0x557c80025830 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c7ff4f430 con 0x557c7fff6b60 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9ba3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fffba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9ba3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fffba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60754/0 (socket says 172.21.15.12:60754) 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9ba3400700 1 -- 172.21.15.12:0/3009635472 learned_addr learned my addr 172.21.15.12:0/3009635472 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/3009635472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c80025830 0x557c80025c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9ba8a00700 1 --2- 172.21.15.12:0/3009635472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c7fff6b60 0x557c80022a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9ba3400700 1 -- 172.21.15.12:0/3009635472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c7fff6b60 msgr2=0x557c80022a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9ba3400700 1 --2- 172.21.15.12:0/3009635472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c7fff6b60 0x557c80022a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9ba3400700 1 -- 172.21.15.12:0/3009635472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c80025830 msgr2=0x557c80025c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9ba3400700 1 --2- 172.21.15.12:0/3009635472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c80025830 0x557c80025c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.366+0000 7f9ba3400700 1 -- 172.21.15.12:0/3009635472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c7fd3e790 con 0x557c7fff8be0 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9ba3400700 1 --2- 172.21.15.12:0/3009635472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fffba10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9b90000c00 tx=0x7f9b90010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbfd32936814e728 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9b8fe00700 1 -- 172.21.15.12:0/3009635472 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b90020070 con 0x557c7fff8be0 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9b8fe00700 1 -- 172.21.15.12:0/3009635472 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b90003cb0 con 0x557c7fff8be0 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9b8fe00700 1 -- 172.21.15.12:0/3009635472 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b90012860 con 0x557c7fff8be0 2024-03-29T09:54:45.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/3009635472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 msgr2=0x557c7fffba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:45.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/3009635472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fffba10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9b90000c00 tx=0x7f9b90010000 comp rx=0 tx=0).stop 2024-03-29T09:54:45.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/3009635472 shutdown_connections 2024-03-29T09:54:45.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/3009635472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c7fff6b60 0x557c80022a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/3009635472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c80025830 0x557c80025c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/3009635472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fffba10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.367+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/3009635472 >> 172.21.15.12:0/3009635472 conn(0x557c7ff596a0 msgr2=0x557c80040640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:45.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.368+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/3009635472 shutdown_connections 2024-03-29T09:54:45.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.368+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/3009635472 wait complete. 2024-03-29T09:54:45.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.368+0000 7f9bbe4d4b80 1 -- start start 2024-03-29T09:54:45.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.369+0000 7f9bbe4d4b80 1 Processor -- start 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.369+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c80012fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.369+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 0x557c80027830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.369+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.369+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c7ff4ad20 con 0x557c7fff6b60 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.369+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c7fee6770 con 0x557c7fff8be0 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.369+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c7ff096f0 con 0x557c80025830 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9bbe4d4b80 10 client.0 _ll_register_callbacks cb 0x557c7fef05e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c80012fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c80012fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60758/0 (socket says 172.21.15.12:60758) 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3e00700 1 -- 172.21.15.12:0/4292712994 learned_addr learned my addr 172.21.15.12:0/4292712994 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3e00700 1 -- 172.21.15.12:0/4292712994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 msgr2=0x557c7fff9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3400700 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 0x557c80027830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3e00700 1 -- 172.21.15.12:0/4292712994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 msgr2=0x557c80027830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 0x557c80027830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3e00700 1 -- 172.21.15.12:0/4292712994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba409a0b0 con 0x557c7fff6b60 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3400700 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 0x557c80027830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c80012fc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9b94057c10 tx=0x7f9b94058450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0314a0b2725ca19 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9b8cc00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9b8cc00700 1 -- 172.21.15.12:0/4292712994 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94060070 con 0x557c7fff6b60 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.370+0000 7f9b8cc00700 1 -- 172.21.15.12:0/4292712994 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b9405bd40 con 0x557c7fff6b60 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/4292712994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557c7ffed920 con 0x557c7fff6b60 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9b8cc00700 1 -- 172.21.15.12:0/4292712994 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b94060300 con 0x557c7fff6b60 2024-03-29T09:54:45.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9bbe4d4b80 10 client.4668 fetch_fsmap learned FSMap version 7 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/4292712994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x557c7fe2b0a0 con 0x557c7fff6b60 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9b8cc00700 1 -- 172.21.15.12:0/4292712994 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557c7ffed920 con 0x557c7fff6b60 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9b8cc00700 1 -- 172.21.15.12:0/4292712994 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 7) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9b940750d0 con 0x557c7fff6b60 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9bbe4d4b80 10 client.4668 fetch_fsmap finished waiting for FSMap version 7 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9bbe4d4b80 10 client.4668 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9bbe4d4b80 20 client.4668 populate_metadata read hostname 'smithi012' 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.12:6834/3922451359,v1:172.21.15.12:6835/3922451359] conn(0x557c80042b60 0x557c80015950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9bbe4d4b80 4 client.4668 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/4292712994 --> [v2:172.21.15.12:6834/3922451359,v1:172.21.15.12:6835/3922451359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c7ffca6f0 con 0x557c80042b60 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.371+0000 7f9ba3400700 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.12:6834/3922451359,v1:172.21.15.12:6835/3922451359] conn(0x557c80042b60 0x557c80015950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.372+0000 7f9b8cc00700 10 client.4668 ms_handle_connect on v2:172.21.15.12:6834/3922451359 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.372+0000 7f9ba3400700 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.12:6834/3922451359,v1:172.21.15.12:6835/3922451359] conn(0x557c80042b60 0x557c80015950 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fff5d42fc8a7be90 server_cookie=93119105804c1e7d in_seq=0 out_seq=0 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9b8cc00700 1 -- 172.21.15.12:0/4292712994 <== mds.0 v2:172.21.15.12:6834/3922451359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557c7ffca6f0 con 0x557c80042b60 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9b8cc00700 10 client.4668 handle_command_reply: tid=0 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 client.4668 shutdown 2024-03-29T09:54:45.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/4292712994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 msgr2=0x557c80012fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c80012fc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9b94057c10 tx=0x7f9b94058450 comp rx=0 tx=0).stop 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/4292712994 shutdown_connections 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.12:6834/3922451359,v1:172.21.15.12:6835/3922451359] conn(0x557c80042b60 0x557c80015950 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c80012fc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/4292712994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 0x557c80027830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/4292712994 >> 172.21.15.12:0/4292712994 conn(0x557c7ff596a0 msgr2=0x557c7fff9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9b8cc00700 0 client.4668 ms_handle_reset on v2:172.21.15.12:6834/3922451359 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/4292712994 shutdown_connections 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/4292712994 wait complete. 2024-03-29T09:54:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.373+0000 7f9bbe4d4b80 20 client.4668 trim_cache size 0 max 16384 2024-03-29T09:54:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.391+0000 7f9bbe4d4b80 1 Processor -- start 2024-03-29T09:54:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.392+0000 7f9bbe4d4b80 1 -- start start 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.392+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c7fff8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.392+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 0x557c7fff9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.392+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.392+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c7fd3f390 con 0x557c7fff6b60 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.392+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c80028b00 con 0x557c7fff8be0 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.392+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c80040d40 con 0x557c80025830 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.392+0000 7f9ba3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c7fff8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c7fff8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60768/0 (socket says 172.21.15.12:60768) 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba3e00700 1 -- 172.21.15.12:0/1407374390 learned_addr learned my addr 172.21.15.12:0/1407374390 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba8a00700 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba3400700 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 0x557c7fff9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba3e00700 1 -- 172.21.15.12:0/1407374390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 msgr2=0x557c7fff5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba3e00700 1 -- 172.21.15.12:0/1407374390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 msgr2=0x557c7fff9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 0x557c7fff9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba3e00700 1 -- 172.21.15.12:0/1407374390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c7ff11010 con 0x557c7fff6b60 2024-03-29T09:54:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba8a00700 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba3400700 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 0x557c7fff9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:54:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c7fff8fc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9b94057bc0 tx=0x7f9b9401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d89659238b26855 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9b99200700 1 -- 172.21.15.12:0/1407374390 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9405bd40 con 0x557c7fff6b60 2024-03-29T09:54:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9b99200700 1 -- 172.21.15.12:0/1407374390 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b94060300 con 0x557c7fff6b60 2024-03-29T09:54:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.393+0000 7f9b99200700 1 -- 172.21.15.12:0/1407374390 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94023bb0 con 0x557c7fff6b60 2024-03-29T09:54:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.394+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1407374390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 msgr2=0x557c7fff8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.394+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c7fff8fc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9b94057bc0 tx=0x7f9b9401e000 comp rx=0 tx=0).stop 2024-03-29T09:54:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.394+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1407374390 shutdown_connections 2024-03-29T09:54:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.394+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.394+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff6b60 0x557c7fff8fc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.394+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/1407374390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff8be0 0x557c7fff9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.394+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1407374390 >> 172.21.15.12:0/1407374390 conn(0x557c7ff596a0 msgr2=0x557c7fff1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.394+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1407374390 shutdown_connections 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.394+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1407374390 wait complete. 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.395+0000 7f9bbe4d4b80 1 -- start start 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.395+0000 7f9bbe4d4b80 1 Processor -- start 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff6b60 0x557c80020b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fff8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9bbe4d4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c80021a30 con 0x557c7fff8be0 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8003e520 con 0x557c7fff6b60 2024-03-29T09:54:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9bbe4d4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c800a81a0 con 0x557c80025830 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9bbe4d4b80 10 client.0 _ll_register_callbacks cb 0x557c7fef05e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9ba3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fff8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9ba3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fff8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60770/0 (socket says 172.21.15.12:60770) 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9ba3400700 1 -- 172.21.15.12:0/1394539438 learned_addr learned my addr 172.21.15.12:0/1394539438 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff6b60 0x557c80020b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9ba8a00700 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9ba3400700 1 -- 172.21.15.12:0/1394539438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 msgr2=0x557c7fff9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9ba3400700 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9ba3400700 1 -- 172.21.15.12:0/1394539438 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff6b60 msgr2=0x557c80020b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9ba3400700 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff6b60 0x557c80020b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.396+0000 7f9ba3400700 1 -- 172.21.15.12:0/1394539438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c7fe2b0a0 con 0x557c7fff8be0 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9ba3400700 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fff8fc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9b900042a0 tx=0x7f9b900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0fd439a22e69bf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9b8cc00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9b8cc00700 1 -- 172.21.15.12:0/1394539438 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9003dc60 con 0x557c7fff8be0 2024-03-29T09:54:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9b8cc00700 1 -- 172.21.15.12:0/1394539438 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b9003ddc0 con 0x557c7fff8be0 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1394539438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557c80037520 con 0x557c7fff8be0 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9b8cc00700 1 -- 172.21.15.12:0/1394539438 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b9003d030 con 0x557c7fff8be0 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9bbe4d4b80 10 client.4674 fetch_fsmap learned FSMap version 7 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1394539438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x557c7fd9f1d0 con 0x557c7fff8be0 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9b8cc00700 1 -- 172.21.15.12:0/1394539438 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557c80037520 con 0x557c7fff8be0 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9b8cc00700 1 -- 172.21.15.12:0/1394539438 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 7) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9b9003d190 con 0x557c7fff8be0 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9bbe4d4b80 10 client.4674 fetch_fsmap finished waiting for FSMap version 7 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9bbe4d4b80 10 client.4674 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9bbe4d4b80 20 client.4674 populate_metadata read hostname 'smithi012' 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.12:6834/3922451359,v1:172.21.15.12:6835/3922451359] conn(0x557c800a7290 0x557c800a7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9bbe4d4b80 4 client.4674 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.397+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1394539438 --> [v2:172.21.15.12:6834/3922451359,v1:172.21.15.12:6835/3922451359] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f9b90003cb0 con 0x557c800a7290 2024-03-29T09:54:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.398+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.12:6834/3922451359,v1:172.21.15.12:6835/3922451359] conn(0x557c800a7290 0x557c800a7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.398+0000 7f9ba3e00700 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.12:6834/3922451359,v1:172.21.15.12:6835/3922451359] conn(0x557c800a7290 0x557c800a7670 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=364998ceda78d3d2 server_cookie=73e5331b49699793 in_seq=0 out_seq=0 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.398+0000 7f9b8cc00700 10 client.4674 ms_handle_connect on v2:172.21.15.12:6834/3922451359 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.398+0000 7f9b8cc00700 1 -- 172.21.15.12:0/1394539438 <== mds.0 v2:172.21.15.12:6834/3922451359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x7f9b90003cb0 con 0x557c800a7290 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.398+0000 7f9b8cc00700 10 client.4674 handle_command_reply: tid=0 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.398+0000 7f9bbe4d4b80 1 client.4674 shutdown 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.398+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1394539438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 msgr2=0x557c7fff8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.398+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fff8fc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9b900042a0 tx=0x7f9b900047c0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1394539438 shutdown_connections 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.12:6834/3922451359,v1:172.21.15.12:6835/3922451359] conn(0x557c800a7290 0x557c800a7670 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x557c80025830 0x557c7fff9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x557c7fff6b60 0x557c80020b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9b8cc00700 0 client.4674 ms_handle_reset on v2:172.21.15.12:6834/3922451359 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9bbe4d4b80 1 --2- 172.21.15.12:0/1394539438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c7fff8be0 0x557c7fff8fc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1394539438 >> 172.21.15.12:0/1394539438 conn(0x557c7ff596a0 msgr2=0x557c7fff1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1394539438 shutdown_connections 2024-03-29T09:54:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9bbe4d4b80 1 -- 172.21.15.12:0/1394539438 wait complete. 2024-03-29T09:54:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9bbe4d4b80 20 client.4674 trim_cache size 0 max 16384 2024-03-29T09:54:45.403 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T09:54:45.403 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T09:54:45.403 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T09:54:45.403 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 4282, 2024-03-29T09:54:45.403 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T09:54:45.403 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T09:54:45.403 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T09:54:45.404 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 17.970188936, 2024-03-29T09:54:45.404 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 7, 2024-03-29T09:54:45.404 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 20, 2024-03-29T09:54:45.404 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 20, 2024-03-29T09:54:45.404 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 22.778489647000001 2024-03-29T09:54:45.404 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T09:54:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9baaa00700 1 -- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9b8406ac90 msgr2=0x7f9b8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9baaa00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9b8406ac90 0x7f9b8406d150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009130 tx=0x7f9b90008030 comp rx=0 tx=0).stop 2024-03-29T09:54:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9baaa00700 1 -- 172.21.15.12:0/2828662810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 msgr2=0x7f9ba41549d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.399+0000 7f9baaa00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 0x7f9ba41549d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400e810 tx=0x7f9b9400eb20 comp rx=0 tx=0).stop 2024-03-29T09:54:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.400+0000 7f9baaa00700 1 -- 172.21.15.12:0/2828662810 shutdown_connections 2024-03-29T09:54:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.400+0000 7f9baaa00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9b8406ac90 0x7f9b8406d150 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.400+0000 7f9baaa00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ba415cfe0 0x7f9ba4158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.400+0000 7f9baaa00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba406f310 0x7f9ba41549d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.400+0000 7f9baaa00700 1 --2- 172.21.15.12:0/2828662810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ba40cd5c0 0x7f9ba415af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.400+0000 7f9baaa00700 1 -- 172.21.15.12:0/2828662810 >> 172.21.15.12:0/2828662810 conn(0x7f9ba40d1370 msgr2=0x7f9ba40d1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.400+0000 7f9baaa00700 1 -- 172.21.15.12:0/2828662810 shutdown_connections 2024-03-29T09:54:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:45.400+0000 7f9baaa00700 1 -- 172.21.15.12:0/2828662810 wait complete. 2024-03-29T09:54:45.416 INFO:teuthology.run_tasks:Running task sequential... 2024-03-29T09:54:45.425 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-29T09:54:45.425 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-29T09:54:45.426 INFO:teuthology.task.print:Enabling mgr modules 2024-03-29T09:54:45.426 INFO:teuthology.run_tasks:Running task kclient... 2024-03-29T09:54:45.456 INFO:tasks.kclient:Mounting kernel clients... 2024-03-29T09:54:45.456 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-29T09:54:45.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:45.456 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T09:54:45.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:45.502 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-03-29T09:54:45.612 INFO:teuthology.orchestra.run.smithi012.stderr:Cannot find device "ceph-brx" 2024-03-29T09:54:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T09:54:45.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:45.615 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T09:54:45.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:45.660 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-03-29T09:54:45.770 INFO:teuthology.orchestra.run.smithi012.stderr:Cannot find device "ceph-brx" 2024-03-29T09:54:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T09:54:45.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:45.772 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-03-29T09:54:45.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:45.824 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link delete ceph-brx 2024-03-29T09:54:45.897 INFO:teuthology.orchestra.run.smithi159.stderr:Cannot find device "ceph-brx" 2024-03-29T09:54:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T09:54:45.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:45.900 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-03-29T09:54:45.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:45.946 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link delete ceph-brx 2024-03-29T09:54:46.055 INFO:teuthology.orchestra.run.smithi159.stderr:Cannot find device "ceph-brx" 2024-03-29T09:54:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T09:54:46.058 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-29T09:54:46.059 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-29T09:54:46.059 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T09:54:46.059 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T09:54:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.256+0000 7fe9b3000700 1 Processor -- start 2024-03-29T09:54:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.257+0000 7fe9b3000700 1 -- start start 2024-03-29T09:54:46.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.258+0000 7fe9b3000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9ac15b5d0 0x7fe9ac0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.258+0000 7fe9b3000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9ac0ac7d0 0x7fe9ac06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.258+0000 7fe9b3000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac15fd50 con 0x7fe9ac0cb3f0 2024-03-29T09:54:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.258+0000 7fe9b3000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9ac15b9b0 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.258+0000 7fe9b3000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac15f140 con 0x7fe9ac15b5d0 2024-03-29T09:54:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.258+0000 7fe9b1000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9ac0ac7d0 0x7fe9ac06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.258+0000 7fe9b1000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9ac0ac7d0 0x7fe9ac06c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60780/0 (socket says 172.21.15.12:60780) 2024-03-29T09:54:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.258+0000 7fe9b1000700 1 -- 172.21.15.12:0/882312742 learned_addr learned my addr 172.21.15.12:0/882312742 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.258+0000 7fe9ab400700 1 --1- 172.21.15.12:0/882312742 >> v1:172.21.15.159:6789/0 conn(0x7fe9ac0cb3f0 0x7fe9ac0ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:52662/0 (socket says 172.21.15.12:52662) 2024-03-29T09:54:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.258+0000 7fe9abe00700 1 --2- 172.21.15.12:0/882312742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9ac15b5d0 0x7fe9ac0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.259+0000 7fe9b1000700 1 -- 172.21.15.12:0/882312742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9ac15b5d0 msgr2=0x7fe9ac0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.259+0000 7fe9b1000700 1 --2- 172.21.15.12:0/882312742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9ac15b5d0 0x7fe9ac0ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.259+0000 7fe9b1000700 1 -- 172.21.15.12:0/882312742 >> v1:172.21.15.159:6789/0 conn(0x7fe9ac0cb3f0 legacy=0x7fe9ac0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.259+0000 7fe9b1000700 1 -- 172.21.15.12:0/882312742 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac15f420 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.259+0000 7fe9aaa00700 1 -- 172.21.15.12:0/882312742 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912840369 0 0) 0x7fe9ac15fd50 con 0x7fe9ac0cb3f0 2024-03-29T09:54:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.259+0000 7fe9b1000700 1 --2- 172.21.15.12:0/882312742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 0x7fe9ac06c9d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4009e50 tx=0x7fe9a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c898c5b6a958e881 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.259+0000 7fe9aaa00700 1 -- 172.21.15.12:0/882312742 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a400e070 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.259+0000 7fe9aaa00700 1 -- 172.21.15.12:0/882312742 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a400b910 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.259+0000 7fe9aaa00700 1 -- 172.21.15.12:0/882312742 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a400bcc0 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.260+0000 7fe9b3000700 1 -- 172.21.15.12:0/882312742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 msgr2=0x7fe9ac06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:46.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.260+0000 7fe9b3000700 1 --2- 172.21.15.12:0/882312742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 0x7fe9ac06c9d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4009e50 tx=0x7fe9a400b330 comp rx=0 tx=0).stop 2024-03-29T09:54:46.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.260+0000 7fe9b3000700 1 -- 172.21.15.12:0/882312742 shutdown_connections 2024-03-29T09:54:46.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.260+0000 7fe9b3000700 1 --2- 172.21.15.12:0/882312742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 0x7fe9ac06c9d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:46.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.260+0000 7fe9b3000700 1 --2- 172.21.15.12:0/882312742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9ac15b5d0 0x7fe9ac0ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:46.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.260+0000 7fe9b3000700 1 -- 172.21.15.12:0/882312742 >> 172.21.15.12:0/882312742 conn(0x7fe9ac0cf190 msgr2=0x7fe9ac0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:46.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.260+0000 7fe9b3000700 1 -- 172.21.15.12:0/882312742 shutdown_connections 2024-03-29T09:54:46.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.260+0000 7fe9b3000700 1 -- 172.21.15.12:0/882312742 wait complete. 2024-03-29T09:54:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.262+0000 7fe9b3000700 1 Processor -- start 2024-03-29T09:54:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.262+0000 7fe9b3000700 1 -- start start 2024-03-29T09:54:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9b3000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 0x7fe9ac06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9b3000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9ac15b5d0 0x7fe9ac155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9b3000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9ac1556d0 0x7fe9ac145290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9b3000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac156440 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9b3000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac155e80 con 0x7fe9ac15b5d0 2024-03-29T09:54:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9b3000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac156160 con 0x7fe9ac1556d0 2024-03-29T09:54:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9abe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 0x7fe9ac06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9abe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 0x7fe9ac06d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60792/0 (socket says 172.21.15.12:60792) 2024-03-29T09:54:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9abe00700 1 -- 172.21.15.12:0/2447358281 learned_addr learned my addr 172.21.15.12:0/2447358281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9ab400700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9ac15b5d0 0x7fe9ac155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9b1000700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9ac1556d0 0x7fe9ac145290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9abe00700 1 -- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9ac1556d0 msgr2=0x7fe9ac145290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9abe00700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9ac1556d0 0x7fe9ac145290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9abe00700 1 -- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9ac15b5d0 msgr2=0x7fe9ac155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9abe00700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9ac15b5d0 0x7fe9ac155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.263+0000 7fe9abe00700 1 -- 172.21.15.12:0/2447358281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a4009b30 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.264+0000 7fe9abe00700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 0x7fe9ac06d1e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe99c00e870 tx=0x7fe99c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.264+0000 7fe9a3e00700 1 -- 172.21.15.12:0/2447358281 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c009590 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.264+0000 7fe9a3e00700 1 -- 172.21.15.12:0/2447358281 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99c013680 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.264+0000 7fe9a3e00700 1 -- 172.21.15.12:0/2447358281 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c0139f0 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.264+0000 7fe9b3000700 1 -- 172.21.15.12:0/2447358281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ac157510 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.264+0000 7fe9b3000700 1 -- 172.21.15.12:0/2447358281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ac155b40 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.265+0000 7fe9b3000700 1 -- 172.21.15.12:0/2447358281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ac0acf40 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.266+0000 7fe9a3e00700 1 -- 172.21.15.12:0/2447358281 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe99c013b50 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.267+0000 7fe9a3e00700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe98c06ac90 0x7fe98c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:46.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.267+0000 7fe9a3e00700 1 -- 172.21.15.12:0/2447358281 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe99c0891b0 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.267+0000 7fe9ab400700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe98c06ac90 0x7fe98c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:46.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.268+0000 7fe9ab400700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe98c06ac90 0x7fe98c06d150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac1f4d20 tx=0x7fe998007de0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:46.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.272+0000 7fe9a3e00700 1 -- 172.21.15.12:0/2447358281 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe99c054380 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.531+0000 7fe9b3000700 1 -- 172.21.15.12:0/2447358281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe9ac0ad270 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.532+0000 7fe9a3e00700 1 -- 172.21.15.12:0/2447358281 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe99c0579a0 con 0x7fe9ac0ac7d0 2024-03-29T09:54:46.531 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 -- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe98c06ac90 msgr2=0x7fe98c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe98c06ac90 0x7fe98c06d150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac1f4d20 tx=0x7fe998007de0 comp rx=0 tx=0).stop 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 -- 172.21.15.12:0/2447358281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 msgr2=0x7fe9ac06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 0x7fe9ac06d1e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe99c00e870 tx=0x7fe99c00eb80 comp rx=0 tx=0).stop 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 -- 172.21.15.12:0/2447358281 shutdown_connections 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe98c06ac90 0x7fe98c06d150 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9ac1556d0 0x7fe9ac145290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ac0ac7d0 0x7fe9ac06d1e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 --2- 172.21.15.12:0/2447358281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9ac15b5d0 0x7fe9ac155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 -- 172.21.15.12:0/2447358281 >> 172.21.15.12:0/2447358281 conn(0x7fe9ac0cf190 msgr2=0x7fe9ac15d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 -- 172.21.15.12:0/2447358281 shutdown_connections 2024-03-29T09:54:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:46.534+0000 7fe9b3000700 1 -- 172.21.15.12:0/2447358281 wait complete. 2024-03-29T09:54:46.552 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-29T09:54:46.552 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T09:54:46.552 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T09:54:46.552 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T09:54:46.552 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T09:54:46.552 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-29T09:54:46.552 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-29T09:54:46.552 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T09:54:46.553 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T09:54:46.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:46.553 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T09:54:46.597 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T09:54:46.597 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T09:54:46.597 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T09:54:46.597 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T09:54:46.598 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 41431sec preferred_lft 41431sec 2024-03-29T09:54:46.599 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T09:54:46.599 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:46.599 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:46.599 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:46.599 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T09:54:46.600 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-29T09:54:46.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:46.600 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T09:54:46.600 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T09:54:46.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add name ceph-brx type bridge 2024-03-29T09:54:46.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr flush dev ceph-brx 2024-03-29T09:54:46.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set ceph-brx up 2024-03-29T09:54:46.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-29T09:54:46.601 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T09:54:47.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:47.043 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T09:54:47.146 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T09:54:47.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:47.149 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T09:54:47.223 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T09:54:47.223 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T09:54:47.223 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T09:54:47.223 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T09:54:47.223 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T09:54:47.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:47.224 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T09:54:47.224 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T09:54:47.224 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T09:54:47.224 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T09:54:47.224 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T09:54:47.225 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T09:54:47.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:47.878 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T09:54:47.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:47.924 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-03-29T09:54:48.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:48.010 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T09:54:48.010 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T09:54:48.010 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T09:54:48.010 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-29T09:54:48.010 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T09:54:48.187 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-29T09:54:48.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:48.187 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T09:54:48.188 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T09:54:48.188 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-29T09:54:48.188 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-29T09:54:48.188 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-29T09:54:48.188 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-29T09:54:48.188 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-29T09:54:48.188 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T09:54:48.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:48.446 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T09:54:48.446 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T09:54:48.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T09:54:48.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.0 master ceph-brx 2024-03-29T09:54:48.446 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T09:54:48.613 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T09:54:48.613 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T09:54:48.657 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T09:54:48.657 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T09:54:48.658 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T09:54:48.741 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T09:54:48.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:48.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:48.883 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:48.883 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T09:54:49.480 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:49.480 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T09:54:49.480 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T09:54:49.480 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T09:54:49.480 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T09:54:49.486 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T09:54:49.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:49.486 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T09:54:49.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:49.554 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:49.669 INFO:teuthology.orchestra.run.smithi012.stdout:client4508 2024-03-29T09:54:49.669 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:54:49.669 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:54:49.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:49.713 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:49.853 INFO:teuthology.orchestra.run.smithi012.stdout:client4508 2024-03-29T09:54:49.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:49.854 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:49.958 INFO:teuthology.orchestra.run.smithi012.stdout:client4508 2024-03-29T09:54:49.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:49.959 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/216a15e0-b7ec-465c-8851-ed202d429a6c.client4508/status') 2024-03-29T09:54:50.077 INFO:teuthology.orchestra.run.smithi012.stdout:instance: client.4508 (3)172.21.15.12:0/1036487171 2024-03-29T09:54:50.077 INFO:teuthology.orchestra.run.smithi012.stdout:blocklisted: false 2024-03-29T09:54:50.077 INFO:teuthology.orchestra.run.smithi012.stderr:0+1 records in 2024-03-29T09:54:50.077 INFO:teuthology.orchestra.run.smithi012.stderr:0+1 records out 2024-03-29T09:54:50.077 INFO:teuthology.orchestra.run.smithi012.stderr:70 bytes copied, 6.9067e-05 s, 1.0 MB/s 2024-03-29T09:54:50.080 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-03-29T09:54:50.080 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-29T09:54:50.080 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-29T09:54:50.080 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T09:54:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.278+0000 7fa0cf000700 1 Processor -- start 2024-03-29T09:54:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.278+0000 7fa0cf000700 1 -- start start 2024-03-29T09:54:50.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0cf000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa0c8160f10 0x7fa0c80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0cf000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0c80cf230 0x7fa0c80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0cf000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0c8161590 0x7fa0c8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0cf000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c8161f60 con 0x7fa0c80cf230 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0cf000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c80ceda0 con 0x7fa0c8160f10 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0cf000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80d23d0 con 0x7fa0c8161590 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0c7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0c80cf230 0x7fa0c80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0c7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0c80cf230 0x7fa0c80d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39210/0 (socket says 172.21.15.12:39210) 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0c7400700 1 -- 172.21.15.12:0/3685651709 learned_addr learned my addr 172.21.15.12:0/3685651709 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0cd000700 1 --2- 172.21.15.12:0/3685651709 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0c8161590 0x7fa0c8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.279+0000 7fa0c7e00700 1 --2- 172.21.15.12:0/3685651709 >> v2:172.21.15.159:3300/0 conn(0x7fa0c8160f10 0x7fa0c80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.280+0000 7fa0c7400700 1 -- 172.21.15.12:0/3685651709 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0c8161590 msgr2=0x7fa0c8161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.280+0000 7fa0c7400700 1 --2- 172.21.15.12:0/3685651709 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0c8161590 0x7fa0c8161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.280+0000 7fa0c7400700 1 -- 172.21.15.12:0/3685651709 >> v2:172.21.15.159:3300/0 conn(0x7fa0c8160f10 msgr2=0x7fa0c80d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:50.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.280+0000 7fa0c7400700 1 --2- 172.21.15.12:0/3685651709 >> v2:172.21.15.159:3300/0 conn(0x7fa0c8160f10 0x7fa0c80d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.280+0000 7fa0c7400700 1 -- 172.21.15.12:0/3685651709 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c81612f0 con 0x7fa0c80cf230 2024-03-29T09:54:50.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.280+0000 7fa0c7400700 1 --2- 172.21.15.12:0/3685651709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c80cf230 0x7fa0c80d1e90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa0c80d1080 tx=0x7fa0b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58c91510901cf8a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:50.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.280+0000 7fa0c6a00700 1 -- 172.21.15.12:0/3685651709 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b400b870 con 0x7fa0c80cf230 2024-03-29T09:54:50.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.280+0000 7fa0c6a00700 1 -- 172.21.15.12:0/3685651709 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b400b9d0 con 0x7fa0c80cf230 2024-03-29T09:54:50.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.280+0000 7fa0c6a00700 1 -- 172.21.15.12:0/3685651709 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b400bd60 con 0x7fa0c80cf230 2024-03-29T09:54:50.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.281+0000 7fa0cf000700 1 -- 172.21.15.12:0/3685651709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c80cf230 msgr2=0x7fa0c80d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:50.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.281+0000 7fa0cf000700 1 --2- 172.21.15.12:0/3685651709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c80cf230 0x7fa0c80d1e90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa0c80d1080 tx=0x7fa0b4009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.281+0000 7fa0cf000700 1 -- 172.21.15.12:0/3685651709 shutdown_connections 2024-03-29T09:54:50.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.281+0000 7fa0cf000700 1 --2- 172.21.15.12:0/3685651709 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0c8161590 0x7fa0c8161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.281+0000 7fa0cf000700 1 --2- 172.21.15.12:0/3685651709 >> v2:172.21.15.159:3300/0 conn(0x7fa0c8160f10 0x7fa0c80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.281+0000 7fa0cf000700 1 --2- 172.21.15.12:0/3685651709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c80cf230 0x7fa0c80d1e90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.281+0000 7fa0cf000700 1 -- 172.21.15.12:0/3685651709 >> 172.21.15.12:0/3685651709 conn(0x7fa0c808aff0 msgr2=0x7fa0c80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:50.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.281+0000 7fa0cf000700 1 -- 172.21.15.12:0/3685651709 shutdown_connections 2024-03-29T09:54:50.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.281+0000 7fa0cf000700 1 -- 172.21.15.12:0/3685651709 wait complete. 2024-03-29T09:54:50.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.283+0000 7fa0cf000700 1 Processor -- start 2024-03-29T09:54:50.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.283+0000 7fa0cf000700 1 -- start start 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0cf000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0c80cf230 0x7fa0c814e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0cf000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0c8160f10 0x7fa0c815b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0cf000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c8161590 0x7fa0c815ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0cf000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80b8ae0 con 0x7fa0c8161590 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0cf000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80b8dc0 con 0x7fa0c80cf230 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0cf000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80b90a0 con 0x7fa0c8160f10 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0cd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c8161590 0x7fa0c815ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0cd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c8161590 0x7fa0c815ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39212/0 (socket says 172.21.15.12:39212) 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0cd000700 1 -- 172.21.15.12:0/118538003 learned_addr learned my addr 172.21.15.12:0/118538003 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0c7400700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0c8160f10 0x7fa0c815b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:50.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.284+0000 7fa0c7e00700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0c80cf230 0x7fa0c814e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0cd000700 1 -- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0c8160f10 msgr2=0x7fa0c815b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0cd000700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0c8160f10 0x7fa0c815b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0cd000700 1 -- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0c80cf230 msgr2=0x7fa0c814e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0cd000700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0c80cf230 0x7fa0c814e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0cd000700 1 -- 172.21.15.12:0/118538003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b4009550 con 0x7fa0c8161590 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0cd000700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c8161590 0x7fa0c815ba00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0007e60 tx=0x7fa0c000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0bfe00700 1 -- 172.21.15.12:0/118538003 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c000ed30 con 0x7fa0c8161590 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0bfe00700 1 -- 172.21.15.12:0/118538003 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c0014420 con 0x7fa0c8161590 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0bfe00700 1 -- 172.21.15.12:0/118538003 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00147d0 con 0x7fa0c8161590 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0cf000700 1 -- 172.21.15.12:0/118538003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c81f3060 con 0x7fa0c8161590 2024-03-29T09:54:50.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.285+0000 7fa0cf000700 1 -- 172.21.15.12:0/118538003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c80b7c40 con 0x7fa0c8161590 2024-03-29T09:54:50.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.286+0000 7fa0cf000700 1 -- 172.21.15.12:0/118538003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c81f3250 con 0x7fa0c8161590 2024-03-29T09:54:50.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.287+0000 7fa0bfe00700 1 -- 172.21.15.12:0/118538003 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa0c0014930 con 0x7fa0c8161590 2024-03-29T09:54:50.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.288+0000 7fa0bfe00700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa0a806ac90 0x7fa0a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:50.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.288+0000 7fa0bfe00700 1 -- 172.21.15.12:0/118538003 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0c0089960 con 0x7fa0c8161590 2024-03-29T09:54:50.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.288+0000 7fa0c7e00700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa0a806ac90 0x7fa0a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:50.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.289+0000 7fa0c7e00700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa0a806ac90 0x7fa0a806d150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8006b20 tx=0x7fa0b8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.293+0000 7fa0bfe00700 1 -- 172.21.15.12:0/118538003 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa0c0054aa0 con 0x7fa0c8161590 2024-03-29T09:54:50.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.555+0000 7fa0cf000700 1 -- 172.21.15.12:0/118538003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa0c8067720 con 0x7fa0c8161590 2024-03-29T09:54:50.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.555+0000 7fa0bfe00700 1 -- 172.21.15.12:0/118538003 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa0c00580c0 con 0x7fa0c8161590 2024-03-29T09:54:50.555 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T09:54:50.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 -- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa0a806ac90 msgr2=0x7fa0a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:50.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa0a806ac90 0x7fa0a806d150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8006b20 tx=0x7fa0b8008040 comp rx=0 tx=0).stop 2024-03-29T09:54:50.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 -- 172.21.15.12:0/118538003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c8161590 msgr2=0x7fa0c815ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:50.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c8161590 0x7fa0c815ba00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0007e60 tx=0x7fa0c000e3c0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 -- 172.21.15.12:0/118538003 shutdown_connections 2024-03-29T09:54:50.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa0a806ac90 0x7fa0a806d150 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c8161590 0x7fa0c815ba00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0c80cf230 0x7fa0c814e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 --2- 172.21.15.12:0/118538003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0c8160f10 0x7fa0c815b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:50.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 -- 172.21.15.12:0/118538003 >> 172.21.15.12:0/118538003 conn(0x7fa0c808aff0 msgr2=0x7fa0c80ca790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:50.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 -- 172.21.15.12:0/118538003 shutdown_connections 2024-03-29T09:54:50.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:50.558+0000 7fa0cf000700 1 -- 172.21.15.12:0/118538003 wait complete. 2024-03-29T09:54:50.575 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-29T09:54:50.575 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T09:54:50.575 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T09:54:50.575 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-29T09:54:50.575 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-29T09:54:50.575 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-29T09:54:50.575 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-29T09:54:50.576 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T09:54:50.576 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-29T09:54:50.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:50.576 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T09:54:50.621 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T09:54:50.621 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T09:54:50.622 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T09:54:50.622 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:50.622 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T09:54:50.622 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:50.622 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:50.622 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:50.622 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T09:54:50.622 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:50.623 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:50.623 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T09:54:50.623 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T09:54:50.623 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:50.623 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T09:54:50.623 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T09:54:50.623 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 41427sec preferred_lft 41427sec 2024-03-29T09:54:50.623 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T09:54:50.623 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:50.623 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T09:54:50.624 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:50.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:50.625 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T09:54:50.726 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T09:54:50.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:50.729 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T09:54:50.775 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T09:54:50.775 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T09:54:50.775 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T09:54:50.775 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T09:54:50.775 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T09:54:50.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:50.776 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T09:54:50.776 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T09:54:50.776 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T09:54:50.776 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T09:54:50.776 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T09:54:50.776 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T09:54:50.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:50.941 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T09:54:50.985 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T09:54:50.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:50.987 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-03-29T09:54:51.070 INFO:teuthology.orchestra.run.smithi012.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-29T09:54:51.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:51.072 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T09:54:51.072 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T09:54:51.072 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T09:54:51.072 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-29T09:54:51.072 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T09:54:51.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:51.247 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 9e:d4:62:66:53:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:51.314 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::9cd4:62ff:fe66:53ee/64 scope link 2024-03-29T09:54:51.315 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:51.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:51.317 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-29T09:54:51.384 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T09:54:51.384 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T09:54:51.384 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T09:54:51.384 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:51.384 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T09:54:51.384 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:51.384 INFO:teuthology.orchestra.run.smithi012.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T09:54:51.385 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 9e:d4:62:66:53:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-29T09:54:51.385 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-29T09:54:51.385 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:51.385 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::9cd4:62ff:fe66:53ee/64 scope link 2024-03-29T09:54:51.385 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:51.387 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-29T09:54:51.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:51.387 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T09:54:51.387 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T09:54:51.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-29T09:54:51.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-29T09:54:51.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-29T09:54:51.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-29T09:54:51.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-29T09:54:51.388 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T09:54:51.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:51.628 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T09:54:51.628 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T09:54:51.628 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.1 up 2024-03-29T09:54:51.628 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.1 master ceph-brx 2024-03-29T09:54:51.628 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T09:54:51.795 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T09:54:51.795 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-29T09:54:51.839 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-29T09:54:51.839 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T09:54:51.839 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-29T09:54:51.924 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-03-29T09:54:51.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:51.924 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:52.067 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:52.067 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T09:54:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T09:54:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T09:54:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 2@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T09:54:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T09:54:52.101 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T09:54:52.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:52.101 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-29T09:54:52.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:52.168 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:52.252 INFO:teuthology.orchestra.run.smithi012.stdout:client4642 2024-03-29T09:54:52.252 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:54:52.252 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:54:52.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:52.296 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:52.420 INFO:teuthology.orchestra.run.smithi012.stdout:client4642 2024-03-29T09:54:52.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:52.420 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:52.504 INFO:teuthology.orchestra.run.smithi012.stdout:client4642 2024-03-29T09:54:52.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:52.505 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/216a15e0-b7ec-465c-8851-ed202d429a6c.client4642/status') 2024-03-29T09:54:52.513 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:54:52.555 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:54:52.611 INFO:teuthology.orchestra.run.smithi012.stdout:instance: client.4642 (3)172.21.15.12:0/2229392814 2024-03-29T09:54:52.611 INFO:teuthology.orchestra.run.smithi012.stdout:blocklisted: false 2024-03-29T09:54:52.611 INFO:teuthology.orchestra.run.smithi012.stderr:0+1 records in 2024-03-29T09:54:52.611 INFO:teuthology.orchestra.run.smithi012.stderr:0+1 records out 2024-03-29T09:54:52.612 INFO:teuthology.orchestra.run.smithi012.stderr:70 bytes copied, 5.6509e-05 s, 1.2 MB/s 2024-03-29T09:54:52.614 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-03-29T09:54:52.614 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-29T09:54:52.614 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-29T09:54:52.614 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T09:54:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.813+0000 7f178f200700 1 Processor -- start 2024-03-29T09:54:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.813+0000 7f178f200700 1 -- start start 2024-03-29T09:54:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.814+0000 7f178f200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f179015d5d0 0x7f17900d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.814+0000 7f178f200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f17900d2870 0x7f17900d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.814+0000 7f178f200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17900d0c40 con 0x7f17900d0830 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.814+0000 7f178f200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f179015d9b0 con 0x7f179015d5d0 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.814+0000 7f178f200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17900b2860 con 0x7f17900d2870 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.814+0000 7f178de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f179015d5d0 0x7f17900d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.815+0000 7f178e800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f17900d0830 0x7f17900b63e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52438/0 (socket says 172.21.15.12:52438) 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.815+0000 7f178e800700 1 -- 172.21.15.12:0/1871162428 learned_addr learned my addr 172.21.15.12:0/1871162428 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.815+0000 7f178d400700 1 --2- 172.21.15.12:0/1871162428 >> v2:172.21.15.159:3300/0 conn(0x7f17900d2870 0x7f17900d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.815+0000 7f178de00700 1 -- 172.21.15.12:0/1871162428 >> v2:172.21.15.159:3300/0 conn(0x7f17900d2870 msgr2=0x7f17900d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.815+0000 7f178de00700 1 --2- 172.21.15.12:0/1871162428 >> v2:172.21.15.159:3300/0 conn(0x7f17900d2870 0x7f17900d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.815+0000 7f178de00700 1 -- 172.21.15.12:0/1871162428 >> v1:172.21.15.12:6789/0 conn(0x7f17900d0830 legacy=0x7f17900b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.815+0000 7f178de00700 1 -- 172.21.15.12:0/1871162428 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17900d2dc0 con 0x7f179015d5d0 2024-03-29T09:54:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.815+0000 7f178de00700 1 --2- 172.21.15.12:0/1871162428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179015d5d0 0x7f17900d3e60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f17900b4dd0 tx=0x7f1780009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c0ad36425601c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178ca00700 1 -- 172.21.15.12:0/1871162428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780011830 con 0x7f179015d5d0 2024-03-29T09:54:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178ca00700 1 -- 172.21.15.12:0/1871162428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1780011990 con 0x7f179015d5d0 2024-03-29T09:54:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178ca00700 1 -- 172.21.15.12:0/1871162428 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780011d20 con 0x7f179015d5d0 2024-03-29T09:54:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178f200700 1 -- 172.21.15.12:0/1871162428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179015d5d0 msgr2=0x7f17900d3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178f200700 1 --2- 172.21.15.12:0/1871162428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179015d5d0 0x7f17900d3e60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f17900b4dd0 tx=0x7f1780009b20 comp rx=0 tx=0).stop 2024-03-29T09:54:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178f200700 1 -- 172.21.15.12:0/1871162428 shutdown_connections 2024-03-29T09:54:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178f200700 1 --2- 172.21.15.12:0/1871162428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179015d5d0 0x7f17900d3e60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178f200700 1 --2- 172.21.15.12:0/1871162428 >> v2:172.21.15.159:3300/0 conn(0x7f17900d2870 0x7f17900d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178f200700 1 -- 172.21.15.12:0/1871162428 >> 172.21.15.12:0/1871162428 conn(0x7f17900668e0 msgr2=0x7f1790066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178f200700 1 -- 172.21.15.12:0/1871162428 shutdown_connections 2024-03-29T09:54:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.816+0000 7f178f200700 1 -- 172.21.15.12:0/1871162428 wait complete. 2024-03-29T09:54:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.818+0000 7f178f200700 1 Processor -- start 2024-03-29T09:54:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.818+0000 7f178f200700 1 -- start start 2024-03-29T09:54:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.819+0000 7f178f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17900d0830 0x7f17901f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.819+0000 7f178f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17900d2870 0x7f17900cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.819+0000 7f178f200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f179015d5d0 0x7f17900ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.819+0000 7f178f200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17901f5db0 con 0x7f17900d0830 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.819+0000 7f178f200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17901f51a0 con 0x7f179015d5d0 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.819+0000 7f178f200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17901f5480 con 0x7f17900d2870 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.819+0000 7f178de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17900d0830 0x7f17901f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.819+0000 7f178de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17900d0830 0x7f17901f48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39222/0 (socket says 172.21.15.12:39222) 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.819+0000 7f178de00700 1 -- 172.21.15.12:0/4196399592 learned_addr learned my addr 172.21.15.12:0/4196399592 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.820+0000 7f178d400700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17900d2870 0x7f17900cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.820+0000 7f178e800700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f179015d5d0 0x7f17900ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.820+0000 7f178de00700 1 -- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17900d2870 msgr2=0x7f17900cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.820+0000 7f178de00700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17900d2870 0x7f17900cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.820+0000 7f178de00700 1 -- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f179015d5d0 msgr2=0x7f17900ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:52.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.820+0000 7f178de00700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f179015d5d0 0x7f17900ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:52.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.820+0000 7f178de00700 1 -- 172.21.15.12:0/4196399592 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17800094a0 con 0x7f17900d0830 2024-03-29T09:54:52.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.820+0000 7f178de00700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17900d0830 0x7f17901f48a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1780000c00 tx=0x7f1780004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:52.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.821+0000 7f1786000700 1 -- 172.21.15.12:0/4196399592 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17800036a0 con 0x7f17900d0830 2024-03-29T09:54:52.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.821+0000 7f1786000700 1 -- 172.21.15.12:0/4196399592 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178001f070 con 0x7f17900d0830 2024-03-29T09:54:52.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.821+0000 7f178f200700 1 -- 172.21.15.12:0/4196399592 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17901f5760 con 0x7f17900d0830 2024-03-29T09:54:52.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.821+0000 7f1786000700 1 -- 172.21.15.12:0/4196399592 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780003a40 con 0x7f17900d0830 2024-03-29T09:54:52.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.821+0000 7f178f200700 1 -- 172.21.15.12:0/4196399592 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17901f4de0 con 0x7f17900d0830 2024-03-29T09:54:52.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.822+0000 7f178f200700 1 -- 172.21.15.12:0/4196399592 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17901f4f70 con 0x7f17900d0830 2024-03-29T09:54:52.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.823+0000 7f1786000700 1 -- 172.21.15.12:0/4196399592 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1780003ba0 con 0x7f17900d0830 2024-03-29T09:54:52.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.824+0000 7f1786000700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177006ac90 0x7f177006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:52.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.824+0000 7f1786000700 1 -- 172.21.15.12:0/4196399592 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1780021070 con 0x7f17900d0830 2024-03-29T09:54:52.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.825+0000 7f178d400700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177006ac90 0x7f177006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:52.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.826+0000 7f178d400700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177006ac90 0x7f177006d150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f17901f6220 tx=0x7f177c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:52.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:52.829+0000 7f1786000700 1 -- 172.21.15.12:0/4196399592 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1780055dc0 con 0x7f17900d0830 2024-03-29T09:54:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.092+0000 7f178f200700 1 -- 172.21.15.12:0/4196399592 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f17901f6090 con 0x7f17900d0830 2024-03-29T09:54:53.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.092+0000 7f1786000700 1 -- 172.21.15.12:0/4196399592 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f178002b020 con 0x7f17900d0830 2024-03-29T09:54:53.146 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T09:54:53.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 -- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177006ac90 msgr2=0x7f177006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:53.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177006ac90 0x7f177006d150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f17901f6220 tx=0x7f177c008040 comp rx=0 tx=0).stop 2024-03-29T09:54:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 -- 172.21.15.12:0/4196399592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17900d0830 msgr2=0x7f17901f48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17900d0830 0x7f17901f48a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1780000c00 tx=0x7f1780004750 comp rx=0 tx=0).stop 2024-03-29T09:54:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 -- 172.21.15.12:0/4196399592 shutdown_connections 2024-03-29T09:54:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177006ac90 0x7f177006d150 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f179015d5d0 0x7f17900ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17900d0830 0x7f17901f48a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 --2- 172.21.15.12:0/4196399592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17900d2870 0x7f17900cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 -- 172.21.15.12:0/4196399592 >> 172.21.15.12:0/4196399592 conn(0x7f17900668e0 msgr2=0x7f17900b63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 -- 172.21.15.12:0/4196399592 shutdown_connections 2024-03-29T09:54:53.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:53.095+0000 7f178f200700 1 -- 172.21.15.12:0/4196399592 wait complete. 2024-03-29T09:54:53.149 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-29T09:54:53.149 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T09:54:53.149 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-03-29T09:54:53.149 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-29T09:54:53.149 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-29T09:54:53.149 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-29T09:54:53.149 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-29T09:54:53.149 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T09:54:53.149 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-29T09:54:53.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:53.149 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-03-29T09:54:53.200 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T09:54:53.200 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T09:54:53.200 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T09:54:53.200 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:53.200 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-03-29T09:54:53.200 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 41479sec preferred_lft 41479sec 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link 2024-03-29T09:54:53.201 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:53.202 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-29T09:54:53.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:53.202 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-03-29T09:54:53.202 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-03-29T09:54:53.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link add name ceph-brx type bridge 2024-03-29T09:54:53.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip addr flush dev ceph-brx 2024-03-29T09:54:53.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set ceph-brx up 2024-03-29T09:54:53.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-29T09:54:53.202 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-03-29T09:54:53.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:53.650 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T09:54:53.753 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-03-29T09:54:53.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:53.756 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-03-29T09:54:53.830 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-03-29T09:54:53.830 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T09:54:53.830 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-29T09:54:53.830 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-29T09:54:53.831 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T09:54:53.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:53.831 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-03-29T09:54:53.831 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-03-29T09:54:53.831 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-29T09:54:53.831 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-29T09:54:53.831 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-29T09:54:53.831 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-03-29T09:54:54.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:54.485 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-03-29T09:54:54.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:54.531 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list-id 2024-03-29T09:54:54.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:54.576 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-03-29T09:54:54.576 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-03-29T09:54:54.576 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T09:54:54.576 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-29T09:54:54.576 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-03-29T09:54:54.752 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-29T09:54:54.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:54.752 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-03-29T09:54:54.752 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-03-29T09:54:54.752 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-29T09:54:54.752 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-29T09:54:54.752 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-29T09:54:54.752 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-29T09:54:54.752 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-29T09:54:54.752 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-03-29T09:54:55.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:55.019 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-03-29T09:54:55.019 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-03-29T09:54:55.019 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-03-29T09:54:55.019 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set dev brx.0 master ceph-brx 2024-03-29T09:54:55.019 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-03-29T09:54:55.185 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T09:54:55.185 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-29T09:54:55.229 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-29T09:54:55.230 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T09:54:55.230 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-29T09:54:55.315 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-03-29T09:54:55.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:55.315 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:55.457 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:55.458 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T09:54:56.071 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:56.071 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T09:54:56.071 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T09:54:56.071 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with new device syntax: 1@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T09:54:56.071 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T09:54:56.078 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T09:54:56.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:56.079 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-29T09:54:56.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:56.147 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:56.250 INFO:teuthology.orchestra.run.smithi159.stdout:client4695 2024-03-29T09:54:56.250 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:54:56.250 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:54:56.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:56.295 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:56.420 INFO:teuthology.orchestra.run.smithi159.stdout:client4695 2024-03-29T09:54:56.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:56.420 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:56.505 INFO:teuthology.orchestra.run.smithi159.stdout:client4695 2024-03-29T09:54:56.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:56.506 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/216a15e0-b7ec-465c-8851-ed202d429a6c.client4695/status') 2024-03-29T09:54:56.609 INFO:teuthology.orchestra.run.smithi159.stdout:instance: client.4695 (3)172.21.15.159:0/2874282617 2024-03-29T09:54:56.610 INFO:teuthology.orchestra.run.smithi159.stdout:blocklisted: false 2024-03-29T09:54:56.610 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records in 2024-03-29T09:54:56.610 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records out 2024-03-29T09:54:56.610 INFO:teuthology.orchestra.run.smithi159.stderr:71 bytes copied, 9.5551e-05 s, 743 kB/s 2024-03-29T09:54:56.613 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-03-29T09:54:56.613 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-29T09:54:56.613 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-29T09:54:56.613 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T09:54:56.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.820+0000 7fac29c00700 1 Processor -- start 2024-03-29T09:54:56.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.821+0000 7fac29c00700 1 -- start start 2024-03-29T09:54:56.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.822+0000 7fac29c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac2415d5d0 0x7fac240d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.822+0000 7fac29c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fac240d2870 0x7fac240d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.822+0000 7fac29c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac240d0c40 con 0x7fac240d0830 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.822+0000 7fac29c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac2415d9b0 con 0x7fac2415d5d0 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.822+0000 7fac29c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac240b2860 con 0x7fac240d2870 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.822+0000 7fac23e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac2415d5d0 0x7fac240d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.822+0000 7fac23e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac2415d5d0 0x7fac240d3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39226/0 (socket says 172.21.15.12:39226) 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.822+0000 7fac23e00700 1 -- 172.21.15.12:0/1601902050 learned_addr learned my addr 172.21.15.12:0/1601902050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.822+0000 7fac23400700 1 --2- 172.21.15.12:0/1601902050 >> v2:172.21.15.159:3300/0 conn(0x7fac240d2870 0x7fac240d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.823+0000 7fac23e00700 1 -- 172.21.15.12:0/1601902050 >> v2:172.21.15.159:3300/0 conn(0x7fac240d2870 msgr2=0x7fac240d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.823+0000 7fac23e00700 1 --2- 172.21.15.12:0/1601902050 >> v2:172.21.15.159:3300/0 conn(0x7fac240d2870 0x7fac240d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.823+0000 7fac23e00700 1 -- 172.21.15.12:0/1601902050 >> v1:172.21.15.159:6789/0 conn(0x7fac240d0830 legacy=0x7fac240b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.823+0000 7fac23e00700 1 -- 172.21.15.12:0/1601902050 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac240d2dc0 con 0x7fac2415d5d0 2024-03-29T09:54:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.823+0000 7fac23e00700 1 --2- 172.21.15.12:0/1601902050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 0x7fac240d3e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fac240b4dd0 tx=0x7fac14009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f4d5ad601fe4ae server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.823+0000 7fac22a00700 1 -- 172.21.15.12:0/1601902050 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14011830 con 0x7fac2415d5d0 2024-03-29T09:54:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.823+0000 7fac22a00700 1 -- 172.21.15.12:0/1601902050 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac14011990 con 0x7fac2415d5d0 2024-03-29T09:54:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.823+0000 7fac22a00700 1 -- 172.21.15.12:0/1601902050 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14011d20 con 0x7fac2415d5d0 2024-03-29T09:54:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.824+0000 7fac29c00700 1 -- 172.21.15.12:0/1601902050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 msgr2=0x7fac240d3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.824+0000 7fac29c00700 1 --2- 172.21.15.12:0/1601902050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 0x7fac240d3e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fac240b4dd0 tx=0x7fac14009bd0 comp rx=0 tx=0).stop 2024-03-29T09:54:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.824+0000 7fac29c00700 1 -- 172.21.15.12:0/1601902050 shutdown_connections 2024-03-29T09:54:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.824+0000 7fac29c00700 1 --2- 172.21.15.12:0/1601902050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 0x7fac240d3e60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.824+0000 7fac29c00700 1 --2- 172.21.15.12:0/1601902050 >> v2:172.21.15.159:3300/0 conn(0x7fac240d2870 0x7fac240d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:56.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.824+0000 7fac29c00700 1 -- 172.21.15.12:0/1601902050 >> 172.21.15.12:0/1601902050 conn(0x7fac240668e0 msgr2=0x7fac24066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:56.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.824+0000 7fac29c00700 1 -- 172.21.15.12:0/1601902050 shutdown_connections 2024-03-29T09:54:56.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.824+0000 7fac29c00700 1 -- 172.21.15.12:0/1601902050 wait complete. 2024-03-29T09:54:56.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.825+0000 7fac29c00700 1 Processor -- start 2024-03-29T09:54:56.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.826+0000 7fac29c00700 1 -- start start 2024-03-29T09:54:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.826+0000 7fac29c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac240d0830 0x7fac240c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac240d2870 0x7fac240b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 0x7fac241f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac240b4400 con 0x7fac2415d5d0 2024-03-29T09:54:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac241f2ba0 con 0x7fac240d0830 2024-03-29T09:54:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac240c8190 con 0x7fac240d2870 2024-03-29T09:54:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 0x7fac241f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac23e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac240d0830 0x7fac240c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 0x7fac241f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39238/0 (socket says 172.21.15.12:39238) 2024-03-29T09:54:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29200700 1 -- 172.21.15.12:0/2847342074 learned_addr learned my addr 172.21.15.12:0/2847342074 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac23400700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac240d2870 0x7fac240b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29200700 1 -- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac240d2870 msgr2=0x7fac240b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29200700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac240d2870 0x7fac240b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29200700 1 -- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac240d0830 msgr2=0x7fac240c6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29200700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac240d0830 0x7fac240c6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.827+0000 7fac29200700 1 -- 172.21.15.12:0/2847342074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac14009550 con 0x7fac2415d5d0 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.828+0000 7fac29200700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 0x7fac241f2660 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00cb80 tx=0x7fac1c00ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.828+0000 7fac1be00700 1 -- 172.21.15.12:0/2847342074 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1c003c00 con 0x7fac2415d5d0 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.828+0000 7fac1be00700 1 -- 172.21.15.12:0/2847342074 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1c003d60 con 0x7fac2415d5d0 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.828+0000 7fac29c00700 1 -- 172.21.15.12:0/2847342074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac240c7580 con 0x7fac2415d5d0 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.828+0000 7fac1be00700 1 -- 172.21.15.12:0/2847342074 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1c00aaa0 con 0x7fac2415d5d0 2024-03-29T09:54:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.828+0000 7fac29c00700 1 -- 172.21.15.12:0/2847342074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac240c7b40 con 0x7fac2415d5d0 2024-03-29T09:54:56.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.829+0000 7fac29c00700 1 -- 172.21.15.12:0/2847342074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac240c8340 con 0x7fac2415d5d0 2024-03-29T09:54:56.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.830+0000 7fac1be00700 1 -- 172.21.15.12:0/2847342074 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fac1c00ac00 con 0x7fac2415d5d0 2024-03-29T09:54:56.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.831+0000 7fac1be00700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac0406ac90 0x7fac0406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:56.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.831+0000 7fac1be00700 1 -- 172.21.15.12:0/2847342074 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac1c0892a0 con 0x7fac2415d5d0 2024-03-29T09:54:56.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.831+0000 7fac23e00700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac0406ac90 0x7fac0406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:56.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.832+0000 7fac23e00700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac0406ac90 0x7fac0406d150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fac14009c10 tx=0x7fac14008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:56.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:56.835+0000 7fac1be00700 1 -- 172.21.15.12:0/2847342074 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac240c8340 con 0x7fac2415d5d0 2024-03-29T09:54:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.098+0000 7fac29c00700 1 -- 172.21.15.12:0/2847342074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fac240c8300 con 0x7fac2415d5d0 2024-03-29T09:54:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.098+0000 7fac1be00700 1 -- 172.21.15.12:0/2847342074 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fac1c0543e0 con 0x7fac2415d5d0 2024-03-29T09:54:57.101 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T09:54:57.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 -- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac0406ac90 msgr2=0x7fac0406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:57.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac0406ac90 0x7fac0406d150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fac14009c10 tx=0x7fac14008030 comp rx=0 tx=0).stop 2024-03-29T09:54:57.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 -- 172.21.15.12:0/2847342074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 msgr2=0x7fac241f2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:57.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 0x7fac241f2660 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00cb80 tx=0x7fac1c00ce90 comp rx=0 tx=0).stop 2024-03-29T09:54:57.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 -- 172.21.15.12:0/2847342074 shutdown_connections 2024-03-29T09:54:57.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac0406ac90 0x7fac0406d150 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:57.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac2415d5d0 0x7fac241f2660 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:57.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac240d0830 0x7fac240c6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:57.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 --2- 172.21.15.12:0/2847342074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac240d2870 0x7fac240b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:57.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 -- 172.21.15.12:0/2847342074 >> 172.21.15.12:0/2847342074 conn(0x7fac240668e0 msgr2=0x7fac240b63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:57.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 -- 172.21.15.12:0/2847342074 shutdown_connections 2024-03-29T09:54:57.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:57.101+0000 7fac29c00700 1 -- 172.21.15.12:0/2847342074 wait complete. 2024-03-29T09:54:57.118 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-29T09:54:57.118 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T09:54:57.119 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-03-29T09:54:57.119 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-29T09:54:57.119 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-29T09:54:57.119 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-29T09:54:57.119 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-29T09:54:57.119 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T09:54:57.119 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-29T09:54:57.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:57.119 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-03-29T09:54:57.167 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T09:54:57.167 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T09:54:57.167 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T09:54:57.167 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.167 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 41475sec preferred_lft 41475sec 2024-03-29T09:54:57.168 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ba:c9:12:af:9a:22 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::6ce7:c7ff:fee5:a0ab/64 scope link 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ba:c9:12:af:9a:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::b8c9:12ff:feaf:9a22/64 scope link 2024-03-29T09:54:57.169 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:57.170 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T09:54:57.236 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-03-29T09:54:57.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:57.239 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-03-29T09:54:57.285 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-03-29T09:54:57.286 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T09:54:57.286 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-29T09:54:57.286 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-29T09:54:57.286 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T09:54:57.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:57.287 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-03-29T09:54:57.287 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-03-29T09:54:57.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-29T09:54:57.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-29T09:54:57.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-29T09:54:57.287 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-03-29T09:54:57.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:57.449 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-03-29T09:54:57.495 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-29T09:54:57.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:57.496 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list-id 2024-03-29T09:54:57.580 INFO:teuthology.orchestra.run.smithi159.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-29T09:54:57.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:57.581 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-03-29T09:54:57.581 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-03-29T09:54:57.581 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T09:54:57.581 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-29T09:54:57.581 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-03-29T09:54:57.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:57.757 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-29T09:54:57.825 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether b6:82:4a:66:25:af brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::b482:4aff:fe66:25af/64 scope link 2024-03-29T09:54:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:57.829 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-29T09:54:57.896 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T09:54:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T09:54:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T09:54:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-03-29T09:54:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.896 INFO:teuthology.orchestra.run.smithi159.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T09:54:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether b6:82:4a:66:25:af brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-29T09:54:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-29T09:54:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::b482:4aff:fe66:25af/64 scope link 2024-03-29T09:54:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:54:57.899 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-29T09:54:57.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:57.900 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-03-29T09:54:57.900 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-03-29T09:54:57.900 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-29T09:54:57.900 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-29T09:54:57.900 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-29T09:54:57.900 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-29T09:54:57.900 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-29T09:54:57.900 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-03-29T09:54:58.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:58.141 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-03-29T09:54:58.141 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-03-29T09:54:58.141 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.1 up 2024-03-29T09:54:58.141 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set dev brx.1 master ceph-brx 2024-03-29T09:54:58.142 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-03-29T09:54:58.306 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T09:54:58.306 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-29T09:54:58.351 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-29T09:54:58.351 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T09:54:58.351 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-29T09:54:58.435 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-03-29T09:54:58.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:58.435 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:58.575 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:58.575 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T09:54:58.604 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:54:58.604 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T09:54:58.605 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T09:54:58.605 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with new device syntax: 3@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T09:54:58.605 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T09:54:58.610 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T09:54:58.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:58.611 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-29T09:54:58.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:58.677 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:58.762 INFO:teuthology.orchestra.run.smithi159.stdout:client4523 2024-03-29T09:54:58.763 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:54:58.763 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:54:58.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:58.807 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:58.933 INFO:teuthology.orchestra.run.smithi159.stdout:client4523 2024-03-29T09:54:58.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:58.934 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:54:59.019 INFO:teuthology.orchestra.run.smithi159.stdout:client4523 2024-03-29T09:54:59.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:54:59.020 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/216a15e0-b7ec-465c-8851-ed202d429a6c.client4523/status') 2024-03-29T09:54:59.123 INFO:teuthology.orchestra.run.smithi159.stdout:instance: client.4523 (3)192.168.0.2:0/4249637343 2024-03-29T09:54:59.123 INFO:teuthology.orchestra.run.smithi159.stdout:blocklisted: false 2024-03-29T09:54:59.123 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records in 2024-03-29T09:54:59.123 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records out 2024-03-29T09:54:59.123 INFO:teuthology.orchestra.run.smithi159.stderr:69 bytes copied, 5.112e-05 s, 1.3 MB/s 2024-03-29T09:54:59.126 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-29T09:54:59.161 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:54:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.372+0000 7ff6e8a00700 1 Processor -- start 2024-03-29T09:54:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.373+0000 7ff6e8a00700 1 -- start start 2024-03-29T09:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.374+0000 7ff6e8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e409aba0 0x7ff6e406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.374+0000 7ff6e8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6e406c480 0x7ff6e40ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.374+0000 7ff6e8a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e406bb40 con 0x7ff6e415d710 2024-03-29T09:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.374+0000 7ff6e8a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e406c910 con 0x7ff6e406c480 2024-03-29T09:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.374+0000 7ff6e8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4099b00 con 0x7ff6e409aba0 2024-03-29T09:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.374+0000 7ff6e2800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6e406c480 0x7ff6e40ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.374+0000 7ff6e1e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff6e415d710 0x7ff6e40d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33586/0 (socket says 172.21.15.12:33586) 2024-03-29T09:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.374+0000 7ff6e1e00700 1 -- 172.21.15.12:0/1684522312 learned_addr learned my addr 172.21.15.12:0/1684522312 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.374+0000 7ff6e1400700 1 --2- 172.21.15.12:0/1684522312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e409aba0 0x7ff6e406bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.375+0000 7ff6e2800700 1 -- 172.21.15.12:0/1684522312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e409aba0 msgr2=0x7ff6e406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.375+0000 7ff6e2800700 1 --2- 172.21.15.12:0/1684522312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e409aba0 0x7ff6e406bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.375+0000 7ff6e2800700 1 -- 172.21.15.12:0/1684522312 >> v1:172.21.15.12:6789/0 conn(0x7ff6e415d710 legacy=0x7ff6e40d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.375+0000 7ff6e2800700 1 -- 172.21.15.12:0/1684522312 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4099de0 con 0x7ff6e406c480 2024-03-29T09:54:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.375+0000 7ff6e0a00700 1 -- 172.21.15.12:0/1684522312 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133218915 0 0) 0x7ff6e406bb40 con 0x7ff6e415d710 2024-03-29T09:54:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.375+0000 7ff6e2800700 1 --2- 172.21.15.12:0/1684522312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e406c480 0x7ff6e40ccbe0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009d90 tx=0x7ff6dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadce3dfac789dae server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.375+0000 7ff6e0a00700 1 -- 172.21.15.12:0/1684522312 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00e070 con 0x7ff6e406c480 2024-03-29T09:54:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.375+0000 7ff6e0a00700 1 -- 172.21.15.12:0/1684522312 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc00bb20 con 0x7ff6e406c480 2024-03-29T09:54:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.375+0000 7ff6e0a00700 1 -- 172.21.15.12:0/1684522312 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc012430 con 0x7ff6e406c480 2024-03-29T09:54:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.376+0000 7ff6e8a00700 1 -- 172.21.15.12:0/1684522312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e406c480 msgr2=0x7ff6e40ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.376+0000 7ff6e8a00700 1 --2- 172.21.15.12:0/1684522312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e406c480 0x7ff6e40ccbe0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009d90 tx=0x7ff6dc00b330 comp rx=0 tx=0).stop 2024-03-29T09:54:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.376+0000 7ff6e8a00700 1 -- 172.21.15.12:0/1684522312 shutdown_connections 2024-03-29T09:54:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.376+0000 7ff6e8a00700 1 --2- 172.21.15.12:0/1684522312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e406c480 0x7ff6e40ccbe0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.376+0000 7ff6e8a00700 1 --2- 172.21.15.12:0/1684522312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e409aba0 0x7ff6e406bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.376+0000 7ff6e8a00700 1 -- 172.21.15.12:0/1684522312 >> 172.21.15.12:0/1684522312 conn(0x7ff6e40d13e0 msgr2=0x7ff6e40d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.376+0000 7ff6e8a00700 1 -- 172.21.15.12:0/1684522312 shutdown_connections 2024-03-29T09:54:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.376+0000 7ff6e8a00700 1 -- 172.21.15.12:0/1684522312 wait complete. 2024-03-29T09:54:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.378+0000 7ff6e8a00700 1 Processor -- start 2024-03-29T09:54:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.378+0000 7ff6e8a00700 1 -- start start 2024-03-29T09:54:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e406c480 0x7ff6e41474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e409aba0 0x7ff6e40b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e415d710 0x7ff6e40b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e41472b0 con 0x7ff6e409aba0 2024-03-29T09:54:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e8a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4157880 con 0x7ff6e415d710 2024-03-29T09:54:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e41575a0 con 0x7ff6e406c480 2024-03-29T09:54:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e409aba0 0x7ff6e40b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e409aba0 0x7ff6e40b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37602/0 (socket says 172.21.15.12:37602) 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e1400700 1 -- 172.21.15.12:0/3704426006 learned_addr learned my addr 172.21.15.12:0/3704426006 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e1e00700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e406c480 0x7ff6e41474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e2800700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e415d710 0x7ff6e40b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e1400700 1 -- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e406c480 msgr2=0x7ff6e41474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e1400700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e406c480 0x7ff6e41474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e1400700 1 -- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e415d710 msgr2=0x7ff6e40b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e1400700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e415d710 0x7ff6e40b5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.379+0000 7ff6e1400700 1 -- 172.21.15.12:0/3704426006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc009a70 con 0x7ff6e409aba0 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.380+0000 7ff6e1400700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e409aba0 0x7ff6e40b5290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000e980 tx=0x7ff6d000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.380+0000 7ff6da000700 1 -- 172.21.15.12:0/3704426006 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0009710 con 0x7ff6e409aba0 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.380+0000 7ff6da000700 1 -- 172.21.15.12:0/3704426006 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d000c690 con 0x7ff6e409aba0 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.380+0000 7ff6da000700 1 -- 172.21.15.12:0/3704426006 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d000ca40 con 0x7ff6e409aba0 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.380+0000 7ff6e8a00700 1 -- 172.21.15.12:0/3704426006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e4159f90 con 0x7ff6e409aba0 2024-03-29T09:54:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.380+0000 7ff6e8a00700 1 -- 172.21.15.12:0/3704426006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e4147a30 con 0x7ff6e409aba0 2024-03-29T09:54:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.381+0000 7ff6e8a00700 1 -- 172.21.15.12:0/3704426006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e415a4b0 con 0x7ff6e409aba0 2024-03-29T09:54:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.382+0000 7ff6da000700 1 -- 172.21.15.12:0/3704426006 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff6d000cba0 con 0x7ff6e409aba0 2024-03-29T09:54:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.383+0000 7ff6da000700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6c406acc0 0x7ff6c406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.383+0000 7ff6da000700 1 -- 172.21.15.12:0/3704426006 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6d00893d0 con 0x7ff6e409aba0 2024-03-29T09:54:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.384+0000 7ff6e1e00700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6c406acc0 0x7ff6c406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.384+0000 7ff6e1e00700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6c406acc0 0x7ff6c406d180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4000f60 tx=0x7ff6d4007e70 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:59.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.388+0000 7ff6da000700 1 -- 172.21.15.12:0/3704426006 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6d0054510 con 0x7ff6e409aba0 2024-03-29T09:54:59.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.607+0000 7ff6e8a00700 1 -- 172.21.15.12:0/3704426006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6e4067720 con 0x7ff6e409aba0 2024-03-29T09:54:59.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.608+0000 7ff6da000700 1 -- 172.21.15.12:0/3704426006 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7ff6d0057b30 con 0x7ff6e409aba0 2024-03-29T09:54:59.608 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:54:59.608 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 -- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6c406acc0 msgr2=0x7ff6c406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6c406acc0 0x7ff6c406d180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4000f60 tx=0x7ff6d4007e70 comp rx=0 tx=0).stop 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 -- 172.21.15.12:0/3704426006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e409aba0 msgr2=0x7ff6e40b5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e409aba0 0x7ff6e40b5290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000e980 tx=0x7ff6d000ec90 comp rx=0 tx=0).stop 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 -- 172.21.15.12:0/3704426006 shutdown_connections 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6c406acc0 0x7ff6c406d180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e415d710 0x7ff6e40b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e406c480 0x7ff6e41474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 --2- 172.21.15.12:0/3704426006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6e409aba0 0x7ff6e40b5290 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 -- 172.21.15.12:0/3704426006 >> 172.21.15.12:0/3704426006 conn(0x7ff6e40d13e0 msgr2=0x7ff6e40ae9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 -- 172.21.15.12:0/3704426006 shutdown_connections 2024-03-29T09:54:59.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.611+0000 7ff6e8a00700 1 -- 172.21.15.12:0/3704426006 wait complete. 2024-03-29T09:54:59.629 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:54:59.630 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:54:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.831+0000 7ff286400700 1 Processor -- start 2024-03-29T09:54:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.831+0000 7ff286400700 1 -- start start 2024-03-29T09:54:59.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.832+0000 7ff286400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff28015d9d0 0x7ff2800d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.832+0000 7ff286400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff2800d16a0 0x7ff2800d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.832+0000 7ff286400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff280161420 con 0x7ff2800d2370 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.832+0000 7ff286400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2800d2820 con 0x7ff2800d16a0 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.832+0000 7ff286400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff280161700 con 0x7ff28015d9d0 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.832+0000 7ff27ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff2800d16a0 0x7ff2800d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.832+0000 7ff27fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff2800d2370 0x7ff28015ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33598/0 (socket says 172.21.15.12:33598) 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.832+0000 7ff27fe00700 1 -- 172.21.15.12:0/2173642063 learned_addr learned my addr 172.21.15.12:0/2173642063 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.832+0000 7ff27f400700 1 --2- 172.21.15.12:0/2173642063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff28015d9d0 0x7ff2800d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.833+0000 7ff27ea00700 1 -- 172.21.15.12:0/2173642063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff28015d9d0 msgr2=0x7ff2800d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.833+0000 7ff27ea00700 1 --2- 172.21.15.12:0/2173642063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff28015d9d0 0x7ff2800d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.833+0000 7ff27ea00700 1 -- 172.21.15.12:0/2173642063 >> v1:172.21.15.12:6789/0 conn(0x7ff2800d2370 legacy=0x7ff28015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.833+0000 7ff27ea00700 1 -- 172.21.15.12:0/2173642063 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2801619e0 con 0x7ff2800d16a0 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.833+0000 7ff27e000700 1 -- 172.21.15.12:0/2173642063 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376798237 0 0) 0x7ff280161420 con 0x7ff2800d2370 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.833+0000 7ff27ea00700 1 --2- 172.21.15.12:0/2173642063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2800d16a0 0x7ff2800d1e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009820 tx=0x7ff26c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffba19e82727e04 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.833+0000 7ff27e000700 1 -- 172.21.15.12:0/2173642063 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c005e30 con 0x7ff2800d16a0 2024-03-29T09:54:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.833+0000 7ff27e000700 1 -- 172.21.15.12:0/2173642063 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c006030 con 0x7ff2800d16a0 2024-03-29T09:54:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.833+0000 7ff27e000700 1 -- 172.21.15.12:0/2173642063 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c0063e0 con 0x7ff2800d16a0 2024-03-29T09:54:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.834+0000 7ff286400700 1 -- 172.21.15.12:0/2173642063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2800d16a0 msgr2=0x7ff2800d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.834+0000 7ff286400700 1 --2- 172.21.15.12:0/2173642063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2800d16a0 0x7ff2800d1e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009820 tx=0x7ff26c009b30 comp rx=0 tx=0).stop 2024-03-29T09:54:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.834+0000 7ff286400700 1 -- 172.21.15.12:0/2173642063 shutdown_connections 2024-03-29T09:54:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.834+0000 7ff286400700 1 --2- 172.21.15.12:0/2173642063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff28015d9d0 0x7ff2800d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.834+0000 7ff286400700 1 --2- 172.21.15.12:0/2173642063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2800d16a0 0x7ff2800d1e30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.834+0000 7ff286400700 1 -- 172.21.15.12:0/2173642063 >> 172.21.15.12:0/2173642063 conn(0x7ff28008aff0 msgr2=0x7ff2800d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:54:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.834+0000 7ff286400700 1 -- 172.21.15.12:0/2173642063 shutdown_connections 2024-03-29T09:54:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.834+0000 7ff286400700 1 -- 172.21.15.12:0/2173642063 wait complete. 2024-03-29T09:54:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.836+0000 7ff286400700 1 Processor -- start 2024-03-29T09:54:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.836+0000 7ff286400700 1 -- start start 2024-03-29T09:54:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff286400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2800d16a0 0x7ff28015f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff286400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2800d2370 0x7ff2801f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff286400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff28015d9d0 0x7ff2801f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff286400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800b62c0 con 0x7ff28015d9d0 2024-03-29T09:54:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff286400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28015f680 con 0x7ff2800d16a0 2024-03-29T09:54:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff286400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff280160a10 con 0x7ff2800d2370 2024-03-29T09:54:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff27fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff28015d9d0 0x7ff2801f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff27fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff28015d9d0 0x7ff2801f3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37628/0 (socket says 172.21.15.12:37628) 2024-03-29T09:54:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff27fe00700 1 -- 172.21.15.12:0/2277523368 learned_addr learned my addr 172.21.15.12:0/2277523368 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff27ea00700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2800d2370 0x7ff2801f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.837+0000 7ff27f400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2800d16a0 0x7ff28015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff27fe00700 1 -- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2800d2370 msgr2=0x7ff2801f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff27fe00700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2800d2370 0x7ff2801f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff27fe00700 1 -- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2800d16a0 msgr2=0x7ff28015f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff27fe00700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2800d16a0 0x7ff28015f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff27fe00700 1 -- 172.21.15.12:0/2277523368 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c009430 con 0x7ff28015d9d0 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff27f400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2800d16a0 0x7ff28015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff27fe00700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff28015d9d0 0x7ff2801f3df0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff278002230 tx=0x7ff27800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff277400700 1 -- 172.21.15.12:0/2277523368 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27800ab70 con 0x7ff28015d9d0 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff277400700 1 -- 172.21.15.12:0/2277523368 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27800acf0 con 0x7ff28015d9d0 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff286400700 1 -- 172.21.15.12:0/2277523368 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2800b6510 con 0x7ff28015d9d0 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff277400700 1 -- 172.21.15.12:0/2277523368 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff278007310 con 0x7ff28015d9d0 2024-03-29T09:54:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.838+0000 7ff286400700 1 -- 172.21.15.12:0/2277523368 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2800b6ad0 con 0x7ff28015d9d0 2024-03-29T09:54:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.839+0000 7ff286400700 1 -- 172.21.15.12:0/2277523368 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2800d1ca0 con 0x7ff28015d9d0 2024-03-29T09:54:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.840+0000 7ff277400700 1 -- 172.21.15.12:0/2277523368 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff278007470 con 0x7ff28015d9d0 2024-03-29T09:54:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.841+0000 7ff277400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff26006ac90 0x7ff26006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:54:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.841+0000 7ff277400700 1 -- 172.21.15.12:0/2277523368 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff278089190 con 0x7ff28015d9d0 2024-03-29T09:54:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.841+0000 7ff27f400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff26006ac90 0x7ff26006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:54:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.842+0000 7ff27f400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff26006ac90 0x7ff26006d150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff270007a50 tx=0x7ff270008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:54:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:54:59.846+0000 7ff277400700 1 -- 172.21.15.12:0/2277523368 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff278054360 con 0x7ff28015d9d0 2024-03-29T09:55:00.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.061+0000 7ff286400700 1 -- 172.21.15.12:0/2277523368 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2800b6cf0 con 0x7ff28015d9d0 2024-03-29T09:55:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.061+0000 7ff277400700 1 -- 172.21.15.12:0/2277523368 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff27800ae60 con 0x7ff28015d9d0 2024-03-29T09:55:00.061 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:55:00.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.063+0000 7ff286400700 1 -- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff26006ac90 msgr2=0x7ff26006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.063+0000 7ff286400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff26006ac90 0x7ff26006d150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff270007a50 tx=0x7ff270008040 comp rx=0 tx=0).stop 2024-03-29T09:55:00.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.063+0000 7ff286400700 1 -- 172.21.15.12:0/2277523368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff28015d9d0 msgr2=0x7ff2801f3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.063+0000 7ff286400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff28015d9d0 0x7ff2801f3df0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff278002230 tx=0x7ff27800cf50 comp rx=0 tx=0).stop 2024-03-29T09:55:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.063+0000 7ff286400700 1 -- 172.21.15.12:0/2277523368 shutdown_connections 2024-03-29T09:55:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.063+0000 7ff286400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff26006ac90 0x7ff26006d150 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.063+0000 7ff286400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff28015d9d0 0x7ff2801f3df0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.063+0000 7ff286400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2800d16a0 0x7ff28015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.063+0000 7ff286400700 1 --2- 172.21.15.12:0/2277523368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2800d2370 0x7ff2801f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.063+0000 7ff286400700 1 -- 172.21.15.12:0/2277523368 >> 172.21.15.12:0/2277523368 conn(0x7ff28008aff0 msgr2=0x7ff28015c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.064+0000 7ff286400700 1 -- 172.21.15.12:0/2277523368 shutdown_connections 2024-03-29T09:55:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.064+0000 7ff286400700 1 -- 172.21.15.12:0/2277523368 wait complete. 2024-03-29T09:55:00.082 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:55:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.283+0000 7f299b600700 1 Processor -- start 2024-03-29T09:55:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.283+0000 7f299b600700 1 -- start start 2024-03-29T09:55:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.284+0000 7f299b600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 0x7f29940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.284+0000 7f299b600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f29940d2370 0x7f29940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.284+0000 7f299b600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2994161420 con 0x7f299415d9d0 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.284+0000 7f299b600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29940d2820 con 0x7f29940d2370 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.284+0000 7f299b600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29940d4340 con 0x7f29940d16a0 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.284+0000 7f2998c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f299415d9d0 0x7f29940d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33606/0 (socket says 172.21.15.12:33606) 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.284+0000 7f2998c00700 1 -- 172.21.15.12:0/277001227 learned_addr learned my addr 172.21.15.12:0/277001227 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.285+0000 7f2993e00700 1 --2- 172.21.15.12:0/277001227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 0x7f29940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.285+0000 7f2999600700 1 --2- 172.21.15.12:0/277001227 >> v2:172.21.15.159:3300/0 conn(0x7f29940d2370 0x7f29940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.285+0000 7f2993400700 1 -- 172.21.15.12:0/277001227 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152739316 0 0) 0x7f2994161420 con 0x7f299415d9d0 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.285+0000 7f2993400700 1 -- 172.21.15.12:0/277001227 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f297c0035b0 con 0x7f299415d9d0 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.285+0000 7f2999600700 1 -- 172.21.15.12:0/277001227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 msgr2=0x7f29940d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.285+0000 7f2999600700 1 --2- 172.21.15.12:0/277001227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 0x7f29940d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.285+0000 7f2999600700 1 -- 172.21.15.12:0/277001227 >> v1:172.21.15.12:6789/0 conn(0x7f299415d9d0 legacy=0x7f29940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.285+0000 7f2999600700 1 -- 172.21.15.12:0/277001227 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2994161700 con 0x7f29940d2370 2024-03-29T09:55:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.285+0000 7f2993e00700 1 --2- 172.21.15.12:0/277001227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 0x7f29940d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f2999600700 1 --2- 172.21.15.12:0/277001227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29940d2370 0x7f29940ce360 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f298c009dc0 tx=0x7f298c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=446830ae17086910 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f2993400700 1 -- 172.21.15.12:0/277001227 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c00e070 con 0x7f29940d2370 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f2993400700 1 -- 172.21.15.12:0/277001227 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c00bae0 con 0x7f29940d2370 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f2993400700 1 -- 172.21.15.12:0/277001227 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c00be90 con 0x7f29940d2370 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f299b600700 1 -- 172.21.15.12:0/277001227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29940d2370 msgr2=0x7f29940ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f299b600700 1 --2- 172.21.15.12:0/277001227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29940d2370 0x7f29940ce360 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f298c009dc0 tx=0x7f298c00b330 comp rx=0 tx=0).stop 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f299b600700 1 -- 172.21.15.12:0/277001227 shutdown_connections 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f299b600700 1 --2- 172.21.15.12:0/277001227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29940d2370 0x7f29940ce360 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f299b600700 1 --2- 172.21.15.12:0/277001227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 0x7f29940d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f299b600700 1 -- 172.21.15.12:0/277001227 >> 172.21.15.12:0/277001227 conn(0x7f299408aff0 msgr2=0x7f29940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.286+0000 7f299b600700 1 -- 172.21.15.12:0/277001227 shutdown_connections 2024-03-29T09:55:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.287+0000 7f299b600700 1 -- 172.21.15.12:0/277001227 wait complete. 2024-03-29T09:55:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.288+0000 7f299b600700 1 Processor -- start 2024-03-29T09:55:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.288+0000 7f299b600700 1 -- start start 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.289+0000 7f299b600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 0x7f299414ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.289+0000 7f299b600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29940d2370 0x7f29940b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.289+0000 7f299b600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f299415d9d0 0x7f29940b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.289+0000 7f299b600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299415fbb0 con 0x7f29940d2370 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.289+0000 7f299b600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29941f37c0 con 0x7f299415d9d0 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.289+0000 7f299b600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29940b7640 con 0x7f29940d16a0 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2993e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29940d2370 0x7f29940b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2993e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29940d2370 0x7f29940b6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37634/0 (socket says 172.21.15.12:37634) 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2993e00700 1 -- 172.21.15.12:0/1808444535 learned_addr learned my addr 172.21.15.12:0/1808444535 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2998c00700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 0x7f299414ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2999600700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f299415d9d0 0x7f29940b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2993e00700 1 -- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 msgr2=0x7f299414ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2993e00700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 0x7f299414ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2993e00700 1 -- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f299415d9d0 msgr2=0x7f29940b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2993e00700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f299415d9d0 0x7f29940b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2993e00700 1 -- 172.21.15.12:0/1808444535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2980009430 con 0x7f29940d2370 2024-03-29T09:55:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2998c00700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 0x7f299414ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.290+0000 7f2993e00700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29940d2370 0x7f29940b6f40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f298000ebe0 tx=0x7f298000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.291+0000 7f2990c00700 1 -- 172.21.15.12:0/1808444535 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2980005a20 con 0x7f29940d2370 2024-03-29T09:55:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.291+0000 7f2990c00700 1 -- 172.21.15.12:0/1808444535 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2980005b80 con 0x7f29940d2370 2024-03-29T09:55:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.291+0000 7f2990c00700 1 -- 172.21.15.12:0/1808444535 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298000c850 con 0x7f29940d2370 2024-03-29T09:55:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.291+0000 7f299b600700 1 -- 172.21.15.12:0/1808444535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298c009a70 con 0x7f29940d2370 2024-03-29T09:55:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.291+0000 7f299b600700 1 -- 172.21.15.12:0/1808444535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29940b7c00 con 0x7f29940d2370 2024-03-29T09:55:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.292+0000 7f299b600700 1 -- 172.21.15.12:0/1808444535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29940d1ca0 con 0x7f29940d2370 2024-03-29T09:55:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.293+0000 7f2990c00700 1 -- 172.21.15.12:0/1808444535 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f298000c9b0 con 0x7f29940d2370 2024-03-29T09:55:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.294+0000 7f2990c00700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f297406ac90 0x7f297406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.294+0000 7f2990c00700 1 -- 172.21.15.12:0/1808444535 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2980089760 con 0x7f29940d2370 2024-03-29T09:55:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.294+0000 7f2998c00700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f297406ac90 0x7f297406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.295+0000 7f2998c00700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f297406ac90 0x7f297406d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29880011e0 tx=0x7f2988009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.298+0000 7f2990c00700 1 -- 172.21.15.12:0/1808444535 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2980054810 con 0x7f29940d2370 2024-03-29T09:55:00.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.512+0000 7f299b600700 1 -- 172.21.15.12:0/1808444535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f29940d4080 con 0x7f29940d2370 2024-03-29T09:55:00.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.513+0000 7f2990c00700 1 -- 172.21.15.12:0/1808444535 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2980020020 con 0x7f29940d2370 2024-03-29T09:55:00.512 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:55:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 -- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f297406ac90 msgr2=0x7f297406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f297406ac90 0x7f297406d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29880011e0 tx=0x7f2988009040 comp rx=0 tx=0).stop 2024-03-29T09:55:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 -- 172.21.15.12:0/1808444535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29940d2370 msgr2=0x7f29940b6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29940d2370 0x7f29940b6f40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f298000ebe0 tx=0x7f298000eef0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 -- 172.21.15.12:0/1808444535 shutdown_connections 2024-03-29T09:55:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f297406ac90 0x7f297406d150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f299415d9d0 0x7f29940b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29940d16a0 0x7f299414ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 --2- 172.21.15.12:0/1808444535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29940d2370 0x7f29940b6f40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 -- 172.21.15.12:0/1808444535 >> 172.21.15.12:0/1808444535 conn(0x7f299408aff0 msgr2=0x7f299415ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:00.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 -- 172.21.15.12:0/1808444535 shutdown_connections 2024-03-29T09:55:00.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.515+0000 7f299b600700 1 -- 172.21.15.12:0/1808444535 wait complete. 2024-03-29T09:55:00.533 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:55:00.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.733+0000 7f9ede400700 1 Processor -- start 2024-03-29T09:55:00.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.733+0000 7f9ede400700 1 -- start start 2024-03-29T09:55:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.734+0000 7f9ede400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed815d9d0 0x7f9ed80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.734+0000 7f9ede400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9ed80d16a0 0x7f9ed80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.734+0000 7f9ede400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed8161420 con 0x7f9ed80d2370 2024-03-29T09:55:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.734+0000 7f9ede400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed80d2820 con 0x7f9ed80d16a0 2024-03-29T09:55:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.734+0000 7f9ede400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed8161700 con 0x7f9ed815d9d0 2024-03-29T09:55:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed815d9d0 0x7f9ed80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed6a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9ed80d16a0 0x7f9ed80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed7e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9ed80d2370 0x7f9ed815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:39758/0 (socket says 172.21.15.12:39758) 2024-03-29T09:55:00.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed7e00700 1 -- 172.21.15.12:0/3980402173 learned_addr learned my addr 172.21.15.12:0/3980402173 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:00.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed6000700 1 -- 172.21.15.12:0/3980402173 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235877729 0 0) 0x7f9ed8161420 con 0x7f9ed80d2370 2024-03-29T09:55:00.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed6000700 1 -- 172.21.15.12:0/3980402173 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ebc0035b0 con 0x7f9ed80d2370 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed6a00700 1 -- 172.21.15.12:0/3980402173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed815d9d0 msgr2=0x7f9ed80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed6a00700 1 --2- 172.21.15.12:0/3980402173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed815d9d0 0x7f9ed80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed6a00700 1 -- 172.21.15.12:0/3980402173 >> v1:172.21.15.159:6789/0 conn(0x7f9ed80d2370 legacy=0x7f9ed815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed6a00700 1 -- 172.21.15.12:0/3980402173 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed81619e0 con 0x7f9ed80d16a0 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.735+0000 7f9ed7400700 1 --2- 172.21.15.12:0/3980402173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed815d9d0 0x7f9ed80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.736+0000 7f9ed6a00700 1 --2- 172.21.15.12:0/3980402173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed80d16a0 0x7f9ed80d1e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009820 tx=0x7f9ec4009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc3fbee549201140 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.736+0000 7f9ed6000700 1 -- 172.21.15.12:0/3980402173 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4005e30 con 0x7f9ed80d16a0 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.736+0000 7f9ed6000700 1 -- 172.21.15.12:0/3980402173 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec4006030 con 0x7f9ed80d16a0 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.736+0000 7f9ed6000700 1 -- 172.21.15.12:0/3980402173 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec40063c0 con 0x7f9ed80d16a0 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.736+0000 7f9ede400700 1 -- 172.21.15.12:0/3980402173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed80d16a0 msgr2=0x7f9ed80d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.736+0000 7f9ede400700 1 --2- 172.21.15.12:0/3980402173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed80d16a0 0x7f9ed80d1e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009820 tx=0x7f9ec4009b30 comp rx=0 tx=0).stop 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.737+0000 7f9ede400700 1 -- 172.21.15.12:0/3980402173 shutdown_connections 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.737+0000 7f9ede400700 1 --2- 172.21.15.12:0/3980402173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed815d9d0 0x7f9ed80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.737+0000 7f9ede400700 1 --2- 172.21.15.12:0/3980402173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed80d16a0 0x7f9ed80d1e30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.737+0000 7f9ede400700 1 -- 172.21.15.12:0/3980402173 >> 172.21.15.12:0/3980402173 conn(0x7f9ed808aff0 msgr2=0x7f9ed80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:00.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.737+0000 7f9ede400700 1 -- 172.21.15.12:0/3980402173 shutdown_connections 2024-03-29T09:55:00.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.737+0000 7f9ede400700 1 -- 172.21.15.12:0/3980402173 wait complete. 2024-03-29T09:55:00.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.738+0000 7f9ede400700 1 Processor -- start 2024-03-29T09:55:00.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.738+0000 7f9ede400700 1 -- start start 2024-03-29T09:55:00.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.739+0000 7f9ede400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed80d16a0 0x7f9ed8152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.739+0000 7f9ede400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed80d2370 0x7f9ed8151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.739+0000 7f9ede400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed815d9d0 0x7f9ed80c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.739+0000 7f9ede400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed8153210 con 0x7f9ed80d2370 2024-03-29T09:55:00.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.739+0000 7f9ede400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed8152190 con 0x7f9ed815d9d0 2024-03-29T09:55:00.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.739+0000 7f9ede400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed80b61c0 con 0x7f9ed80d16a0 2024-03-29T09:55:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed80d2370 0x7f9ed8151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed80d2370 0x7f9ed8151c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37642/0 (socket says 172.21.15.12:37642) 2024-03-29T09:55:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed6a00700 1 -- 172.21.15.12:0/131845959 learned_addr learned my addr 172.21.15.12:0/131845959 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed7e00700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed815d9d0 0x7f9ed80c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed6a00700 1 -- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed80d16a0 msgr2=0x7f9ed8152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed7400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed80d16a0 0x7f9ed8152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed6a00700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed80d16a0 0x7f9ed8152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed6a00700 1 -- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed815d9d0 msgr2=0x7f9ed80c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed6a00700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed815d9d0 0x7f9ed80c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed6a00700 1 -- 172.21.15.12:0/131845959 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec80094a0 con 0x7f9ed80d2370 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed7400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed80d16a0 0x7f9ed8152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.740+0000 7f9ed6a00700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed80d2370 0x7f9ed8151c50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4012000 tx=0x7f9ec4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.741+0000 7f9ecf400700 1 -- 172.21.15.12:0/131845959 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec40063a0 con 0x7f9ed80d2370 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.741+0000 7f9ecf400700 1 -- 172.21.15.12:0/131845959 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec4020070 con 0x7f9ed80d2370 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.741+0000 7f9ede400700 1 -- 172.21.15.12:0/131845959 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec4009430 con 0x7f9ed80d2370 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.741+0000 7f9ecf400700 1 -- 172.21.15.12:0/131845959 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4006720 con 0x7f9ed80d2370 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.741+0000 7f9ede400700 1 -- 172.21.15.12:0/131845959 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed80b6780 con 0x7f9ed80d2370 2024-03-29T09:55:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.742+0000 7f9ede400700 1 -- 172.21.15.12:0/131845959 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed80d1ca0 con 0x7f9ed80d2370 2024-03-29T09:55:00.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.743+0000 7f9ecf400700 1 -- 172.21.15.12:0/131845959 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9ec4022030 con 0x7f9ed80d2370 2024-03-29T09:55:00.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.743+0000 7f9ecf400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb806ac90 0x7f9eb806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:00.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.744+0000 7f9ecf400700 1 -- 172.21.15.12:0/131845959 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ec408a860 con 0x7f9ed80d2370 2024-03-29T09:55:00.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.744+0000 7f9ed7400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb806ac90 0x7f9eb806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:00.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.745+0000 7f9ed7400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb806ac90 0x7f9eb806d150 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009c50 tx=0x7f9ec8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:00.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.748+0000 7f9ecf400700 1 -- 172.21.15.12:0/131845959 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ec4055a30 con 0x7f9ed80d2370 2024-03-29T09:55:00.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.960+0000 7f9ede400700 1 -- 172.21.15.12:0/131845959 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9ed8067720 con 0x7f9ed80d2370 2024-03-29T09:55:00.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.960+0000 7f9ecf400700 1 -- 172.21.15.12:0/131845959 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9ec4059050 con 0x7f9ed80d2370 2024-03-29T09:55:00.960 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:55:00.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.962+0000 7f9ede400700 1 -- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb806ac90 msgr2=0x7f9eb806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.962+0000 7f9ede400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb806ac90 0x7f9eb806d150 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009c50 tx=0x7f9ec8008040 comp rx=0 tx=0).stop 2024-03-29T09:55:00.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.962+0000 7f9ede400700 1 -- 172.21.15.12:0/131845959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed80d2370 msgr2=0x7f9ed8151c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:00.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.962+0000 7f9ede400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed80d2370 0x7f9ed8151c50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4012000 tx=0x7f9ec4004750 comp rx=0 tx=0).stop 2024-03-29T09:55:00.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.962+0000 7f9ede400700 1 -- 172.21.15.12:0/131845959 shutdown_connections 2024-03-29T09:55:00.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.962+0000 7f9ede400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb806ac90 0x7f9eb806d150 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.962+0000 7f9ede400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed815d9d0 0x7f9ed80c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.962+0000 7f9ede400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed80d16a0 0x7f9ed8152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.962+0000 7f9ede400700 1 --2- 172.21.15.12:0/131845959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed80d2370 0x7f9ed8151c50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:00.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.962+0000 7f9ede400700 1 -- 172.21.15.12:0/131845959 >> 172.21.15.12:0/131845959 conn(0x7f9ed808aff0 msgr2=0x7f9ed80b1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:00.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.963+0000 7f9ede400700 1 -- 172.21.15.12:0/131845959 shutdown_connections 2024-03-29T09:55:00.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:00.963+0000 7f9ede400700 1 -- 172.21.15.12:0/131845959 wait complete. 2024-03-29T09:55:00.981 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:01.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.178+0000 7f0e5f000700 1 Processor -- start 2024-03-29T09:55:01.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.178+0000 7f0e5f000700 1 -- start start 2024-03-29T09:55:01.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.179+0000 7f0e5f000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0e58161ca0 0x7f0e580cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.179+0000 7f0e5f000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e580ceda0 con 0x7f0e58160f10 2024-03-29T09:55:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.179+0000 7f0e5f000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e581612f0 con 0x7f0e580d18b0 2024-03-29T09:55:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.179+0000 7f0e5f000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e580d1d30 con 0x7f0e58161ca0 2024-03-29T09:55:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.179+0000 7f0e57e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0e58160f10 0x7f0e580d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33610/0 (socket says 172.21.15.12:33610) 2024-03-29T09:55:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.179+0000 7f0e57e00700 1 -- 172.21.15.12:0/3399489780 learned_addr learned my addr 172.21.15.12:0/3399489780 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.179+0000 7f0e5d000700 1 --2- 172.21.15.12:0/3399489780 >> v2:172.21.15.159:3300/0 conn(0x7f0e58161ca0 0x7f0e580cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881617487 0 0) 0x7f0e580ceda0 con 0x7f0e58160f10 2024-03-29T09:55:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e3c0035b0 con 0x7f0e58160f10 2024-03-29T09:55:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2862539891 0 0) 0x7f0e3c0035b0 con 0x7f0e58160f10 2024-03-29T09:55:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 >> v2:172.21.15.159:3300/0 conn(0x7f0e58161ca0 msgr2=0x7f0e580cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 --2- 172.21.15.12:0/3399489780 >> v2:172.21.15.159:3300/0 conn(0x7f0e58161ca0 0x7f0e580cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 >> v1:172.21.15.159:6789/0 conn(0x7f0e580d18b0 legacy=0x7f0e58161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e580ce470 con 0x7f0e58160f10 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0e48001670 con 0x7f0e58160f10 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 >> v1:172.21.15.12:6789/0 conn(0x7f0e58160f10 legacy=0x7f0e580d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e3c003a00 0x7f0e3c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e3c004850 0x7f0e3c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3c005210 0x7f0e3c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e580d1d30 con 0x7f0e3c005210 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c00c1b0 con 0x7f0e3c004850 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c00c370 con 0x7f0e3c003a00 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e5d000700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3c005210 0x7f0e3c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.180+0000 7f0e57400700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e3c004850 0x7f0e3c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e57e00700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e3c003a00 0x7f0e3c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e5d000700 1 -- 172.21.15.12:0/3399489780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e3c003a00 msgr2=0x7f0e3c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e5d000700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e3c003a00 0x7f0e3c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e5d000700 1 -- 172.21.15.12:0/3399489780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e3c004850 msgr2=0x7f0e3c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e5d000700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e3c004850 0x7f0e3c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e5d000700 1 -- 172.21.15.12:0/3399489780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c0035b0 con 0x7f0e3c005210 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e5d000700 1 -- 172.21.15.12:0/3399489780 reap_dead start 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e5d000700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3c005210 0x7f0e3c00ba10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009fc0 tx=0x7f0e3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a26c18d64574f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50003a20 con 0x7f0e3c005210 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5000f040 con 0x7f0e3c005210 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.181+0000 7f0e56a00700 1 -- 172.21.15.12:0/3399489780 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50003da0 con 0x7f0e3c005210 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.182+0000 7f0e5f000700 1 -- 172.21.15.12:0/3399489780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3c005210 msgr2=0x7f0e3c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.182+0000 7f0e5f000700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3c005210 0x7f0e3c00ba10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009fc0 tx=0x7f0e3c005880 comp rx=0 tx=0).stop 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.182+0000 7f0e5f000700 1 -- 172.21.15.12:0/3399489780 shutdown_connections 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.182+0000 7f0e5f000700 1 --2- 172.21.15.12:0/3399489780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3c005210 0x7f0e3c00ba10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.182+0000 7f0e5f000700 1 -- 172.21.15.12:0/3399489780 >> 172.21.15.12:0/3399489780 conn(0x7f0e5808aff0 msgr2=0x7f0e580d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.182+0000 7f0e5f000700 1 -- 172.21.15.12:0/3399489780 shutdown_connections 2024-03-29T09:55:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.182+0000 7f0e5f000700 1 -- 172.21.15.12:0/3399489780 wait complete. 2024-03-29T09:55:01.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.184+0000 7f0e5f000700 1 Processor -- start 2024-03-29T09:55:01.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.184+0000 7f0e5f000700 1 -- start start 2024-03-29T09:55:01.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e5f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e3c005210 0x7f0e581f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e5f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e581f6530 0x7f0e581f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e5f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e580b7060 0x7f0e580b74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e5f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e581f6ef0 con 0x7f0e580b7060 2024-03-29T09:55:01.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e5f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e581f1330 con 0x7f0e581f6530 2024-03-29T09:55:01.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e5f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e581f0d00 con 0x7f0e3c005210 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e5d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e580b7060 0x7f0e580b74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e5d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e580b7060 0x7f0e580b74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37660/0 (socket says 172.21.15.12:37660) 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e5d000700 1 -- 172.21.15.12:0/534308239 learned_addr learned my addr 172.21.15.12:0/534308239 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e57400700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e581f6530 0x7f0e581f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.185+0000 7f0e57e00700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e3c005210 0x7f0e581f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e5d000700 1 -- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e3c005210 msgr2=0x7f0e581f03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e5d000700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e3c005210 0x7f0e581f03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e5d000700 1 -- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e581f6530 msgr2=0x7f0e581f69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e5d000700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e581f6530 0x7f0e581f69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e5d000700 1 -- 172.21.15.12:0/534308239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e580b7dd0 con 0x7f0e580b7060 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e5d000700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e580b7060 0x7f0e580b74e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000e700 tx=0x7f0e50009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e4fe00700 1 -- 172.21.15.12:0/534308239 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50007000 con 0x7f0e580b7060 2024-03-29T09:55:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e4fe00700 1 -- 172.21.15.12:0/534308239 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e50003e10 con 0x7f0e580b7060 2024-03-29T09:55:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e5f000700 1 -- 172.21.15.12:0/534308239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e581f0fc0 con 0x7f0e580b7060 2024-03-29T09:55:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e4fe00700 1 -- 172.21.15.12:0/534308239 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50004490 con 0x7f0e580b7060 2024-03-29T09:55:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.186+0000 7f0e5f000700 1 -- 172.21.15.12:0/534308239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e580b7a20 con 0x7f0e580b7060 2024-03-29T09:55:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.187+0000 7f0e5f000700 1 -- 172.21.15.12:0/534308239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e581f1180 con 0x7f0e580b7060 2024-03-29T09:55:01.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.188+0000 7f0e4fe00700 1 -- 172.21.15.12:0/534308239 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0e500045f0 con 0x7f0e580b7060 2024-03-29T09:55:01.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.189+0000 7f0e4fe00700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0e3806ad10 0x7f0e3806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.189+0000 7f0e4fe00700 1 -- 172.21.15.12:0/534308239 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e5008a5d0 con 0x7f0e580b7060 2024-03-29T09:55:01.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.190+0000 7f0e57e00700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0e3806ad10 0x7f0e3806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.190+0000 7f0e57e00700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0e3806ad10 0x7f0e3806d1d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0e48001200 tx=0x7f0e48007040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.193+0000 7f0e4fe00700 1 -- 172.21.15.12:0/534308239 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0e50055820 con 0x7f0e580b7060 2024-03-29T09:55:01.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.412+0000 7f0e5f000700 1 -- 172.21.15.12:0/534308239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e58067720 con 0x7f0e580b7060 2024-03-29T09:55:01.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.413+0000 7f0e4fe00700 1 -- 172.21.15.12:0/534308239 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f0e50058e40 con 0x7f0e580b7060 2024-03-29T09:55:01.413 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:01.413 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 -- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0e3806ad10 msgr2=0x7f0e3806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0e3806ad10 0x7f0e3806d1d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0e48001200 tx=0x7f0e48007040 comp rx=0 tx=0).stop 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 -- 172.21.15.12:0/534308239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e580b7060 msgr2=0x7f0e580b74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e580b7060 0x7f0e580b74e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000e700 tx=0x7f0e50009750 comp rx=0 tx=0).stop 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 -- 172.21.15.12:0/534308239 shutdown_connections 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0e3806ad10 0x7f0e3806d1d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e580b7060 0x7f0e580b74e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e3c005210 0x7f0e581f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 --2- 172.21.15.12:0/534308239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e581f6530 0x7f0e581f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 -- 172.21.15.12:0/534308239 >> 172.21.15.12:0/534308239 conn(0x7f0e5808aff0 msgr2=0x7f0e5806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 -- 172.21.15.12:0/534308239 shutdown_connections 2024-03-29T09:55:01.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.416+0000 7f0e5f000700 1 -- 172.21.15.12:0/534308239 wait complete. 2024-03-29T09:55:01.434 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:55:01.435 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:55:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.635+0000 7fb1ce200700 1 Processor -- start 2024-03-29T09:55:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.635+0000 7fb1ce200700 1 -- start start 2024-03-29T09:55:01.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.636+0000 7fb1ce200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d16a0 0x7fb1c80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.636+0000 7fb1ce200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb1c80d2370 0x7fb1c80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.636+0000 7fb1ce200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c8161420 con 0x7fb1c815d9d0 2024-03-29T09:55:01.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.636+0000 7fb1ce200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c80d2820 con 0x7fb1c80d2370 2024-03-29T09:55:01.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.636+0000 7fb1ce200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80d4340 con 0x7fb1c80d16a0 2024-03-29T09:55:01.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.636+0000 7fb1c7400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb1c815d9d0 0x7fb1c80d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33612/0 (socket says 172.21.15.12:33612) 2024-03-29T09:55:01.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.636+0000 7fb1c7400700 1 -- 172.21.15.12:0/2710385549 learned_addr learned my addr 172.21.15.12:0/2710385549 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:01.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.637+0000 7fb1c7e00700 1 --2- 172.21.15.12:0/2710385549 >> v2:172.21.15.159:3300/0 conn(0x7fb1c80d2370 0x7fb1c80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.637+0000 7fb1c6a00700 1 --2- 172.21.15.12:0/2710385549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d16a0 0x7fb1c80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.637+0000 7fb1c6000700 1 -- 172.21.15.12:0/2710385549 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291573949 0 0) 0x7fb1c8161420 con 0x7fb1c815d9d0 2024-03-29T09:55:01.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.637+0000 7fb1c6000700 1 -- 172.21.15.12:0/2710385549 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ac0035b0 con 0x7fb1c815d9d0 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.637+0000 7fb1c6a00700 1 -- 172.21.15.12:0/2710385549 >> v2:172.21.15.159:3300/0 conn(0x7fb1c80d2370 msgr2=0x7fb1c80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.637+0000 7fb1c6a00700 1 --2- 172.21.15.12:0/2710385549 >> v2:172.21.15.159:3300/0 conn(0x7fb1c80d2370 0x7fb1c80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.637+0000 7fb1c6a00700 1 -- 172.21.15.12:0/2710385549 >> v1:172.21.15.12:6789/0 conn(0x7fb1c815d9d0 legacy=0x7fb1c80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.637+0000 7fb1c6a00700 1 -- 172.21.15.12:0/2710385549 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c8161700 con 0x7fb1c80d16a0 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.637+0000 7fb1c7e00700 1 --2- 172.21.15.12:0/2710385549 >> v2:172.21.15.159:3300/0 conn(0x7fb1c80d2370 0x7fb1c80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.638+0000 7fb1c6a00700 1 --2- 172.21.15.12:0/2710385549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d16a0 0x7fb1c80d1e30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009820 tx=0x7fb1b4009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dadc315bf0ebc994 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.638+0000 7fb1c6000700 1 -- 172.21.15.12:0/2710385549 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4005e30 con 0x7fb1c80d16a0 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.638+0000 7fb1c6000700 1 -- 172.21.15.12:0/2710385549 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b4006030 con 0x7fb1c80d16a0 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.638+0000 7fb1c6000700 1 -- 172.21.15.12:0/2710385549 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b40063a0 con 0x7fb1c80d16a0 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.638+0000 7fb1ce200700 1 -- 172.21.15.12:0/2710385549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d16a0 msgr2=0x7fb1c80d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.638+0000 7fb1ce200700 1 --2- 172.21.15.12:0/2710385549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d16a0 0x7fb1c80d1e30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009820 tx=0x7fb1b4009b30 comp rx=0 tx=0).stop 2024-03-29T09:55:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.638+0000 7fb1ce200700 1 -- 172.21.15.12:0/2710385549 shutdown_connections 2024-03-29T09:55:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.638+0000 7fb1ce200700 1 --2- 172.21.15.12:0/2710385549 >> v2:172.21.15.159:3300/0 conn(0x7fb1c80d2370 0x7fb1c80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.638+0000 7fb1ce200700 1 --2- 172.21.15.12:0/2710385549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d16a0 0x7fb1c80d1e30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.638+0000 7fb1ce200700 1 -- 172.21.15.12:0/2710385549 >> 172.21.15.12:0/2710385549 conn(0x7fb1c808aff0 msgr2=0x7fb1c80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.639+0000 7fb1ce200700 1 -- 172.21.15.12:0/2710385549 shutdown_connections 2024-03-29T09:55:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.639+0000 7fb1ce200700 1 -- 172.21.15.12:0/2710385549 wait complete. 2024-03-29T09:55:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.640+0000 7fb1ce200700 1 Processor -- start 2024-03-29T09:55:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.640+0000 7fb1ce200700 1 -- start start 2024-03-29T09:55:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.641+0000 7fb1ce200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1c80d16a0 0x7fb1c815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.641+0000 7fb1ce200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d2370 0x7fb1c814e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.641+0000 7fb1ce200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1c815d9d0 0x7fb1c80b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.641+0000 7fb1ce200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80cc7a0 con 0x7fb1c80d16a0 2024-03-29T09:55:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.641+0000 7fb1ce200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c8160ef0 con 0x7fb1c815d9d0 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.641+0000 7fb1ce200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c814ef20 con 0x7fb1c80d2370 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1c80d16a0 0x7fb1c815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1c80d16a0 0x7fb1c815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37674/0 (socket says 172.21.15.12:37674) 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7400700 1 -- 172.21.15.12:0/2729638667 learned_addr learned my addr 172.21.15.12:0/2729638667 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c6a00700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d2370 0x7fb1c814e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7e00700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1c815d9d0 0x7fb1c80b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7400700 1 -- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d2370 msgr2=0x7fb1c814e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7400700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d2370 0x7fb1c814e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7400700 1 -- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1c815d9d0 msgr2=0x7fb1c80b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7400700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1c815d9d0 0x7fb1c80b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7400700 1 -- 172.21.15.12:0/2729638667 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c0009a70 con 0x7fb1c80d16a0 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7e00700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1c815d9d0 0x7fb1c80b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.642+0000 7fb1c7400700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1c80d16a0 0x7fb1c815ddb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800c450 tx=0x7fb1b800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:01.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.643+0000 7fb1bf400700 1 -- 172.21.15.12:0/2729638667 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8005d70 con 0x7fb1c80d16a0 2024-03-29T09:55:01.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.643+0000 7fb1bf400700 1 -- 172.21.15.12:0/2729638667 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b80070d0 con 0x7fb1c80d16a0 2024-03-29T09:55:01.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.643+0000 7fb1ce200700 1 -- 172.21.15.12:0/2729638667 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b4009430 con 0x7fb1c80d16a0 2024-03-29T09:55:01.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.643+0000 7fb1bf400700 1 -- 172.21.15.12:0/2729638667 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8007440 con 0x7fb1c80d16a0 2024-03-29T09:55:01.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.643+0000 7fb1ce200700 1 -- 172.21.15.12:0/2729638667 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c80cbe00 con 0x7fb1c80d16a0 2024-03-29T09:55:01.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.644+0000 7fb1ce200700 1 -- 172.21.15.12:0/2729638667 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c80d1ca0 con 0x7fb1c80d16a0 2024-03-29T09:55:01.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.645+0000 7fb1bf400700 1 -- 172.21.15.12:0/2729638667 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb1b80075a0 con 0x7fb1c80d16a0 2024-03-29T09:55:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.646+0000 7fb1bf400700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb1a806acc0 0x7fb1a806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.646+0000 7fb1bf400700 1 -- 172.21.15.12:0/2729638667 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1b80886b0 con 0x7fb1c80d16a0 2024-03-29T09:55:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.647+0000 7fb1c6a00700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb1a806acc0 0x7fb1a806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.647+0000 7fb1c6a00700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb1a806acc0 0x7fb1a806d180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4000c00 tx=0x7fb1b4008070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:01.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.651+0000 7fb1bf400700 1 -- 172.21.15.12:0/2729638667 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb1b80538a0 con 0x7fb1c80d16a0 2024-03-29T09:55:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.868+0000 7fb1ce200700 1 -- 172.21.15.12:0/2729638667 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb1c8067720 con 0x7fb1c80d16a0 2024-03-29T09:55:01.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.868+0000 7fb1bf400700 1 -- 172.21.15.12:0/2729638667 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1b8056ec0 con 0x7fb1c80d16a0 2024-03-29T09:55:01.868 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:55:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 -- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb1a806acc0 msgr2=0x7fb1a806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb1a806acc0 0x7fb1a806d180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4000c00 tx=0x7fb1b4008070 comp rx=0 tx=0).stop 2024-03-29T09:55:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 -- 172.21.15.12:0/2729638667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1c80d16a0 msgr2=0x7fb1c815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1c80d16a0 0x7fb1c815ddb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800c450 tx=0x7fb1b800c760 comp rx=0 tx=0).stop 2024-03-29T09:55:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 -- 172.21.15.12:0/2729638667 shutdown_connections 2024-03-29T09:55:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb1a806acc0 0x7fb1a806d180 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1c815d9d0 0x7fb1c80b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1c80d16a0 0x7fb1c815ddb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 --2- 172.21.15.12:0/2729638667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1c80d2370 0x7fb1c814e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 -- 172.21.15.12:0/2729638667 >> 172.21.15.12:0/2729638667 conn(0x7fb1c808aff0 msgr2=0x7fb1c806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 -- 172.21.15.12:0/2729638667 shutdown_connections 2024-03-29T09:55:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:01.870+0000 7fb1ce200700 1 -- 172.21.15.12:0/2729638667 wait complete. 2024-03-29T09:55:01.888 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:55:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.090+0000 7f89d1000700 1 Processor -- start 2024-03-29T09:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.090+0000 7f89d1000700 1 -- start start 2024-03-29T09:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.091+0000 7f89d1000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f89cc15d9d0 0x7f89cc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.091+0000 7f89d1000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89cc0d16a0 0x7f89cc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.091+0000 7f89d1000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc161420 con 0x7f89cc0d2370 2024-03-29T09:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.091+0000 7f89d1000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89cc0d2820 con 0x7f89cc15d9d0 2024-03-29T09:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.091+0000 7f89d1000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc161700 con 0x7f89cc0d16a0 2024-03-29T09:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.092+0000 7f89c9e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f89cc15d9d0 0x7f89cc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.092+0000 7f89c9e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f89cc15d9d0 0x7f89cc0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:59436/0 (socket says 172.21.15.12:59436) 2024-03-29T09:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.092+0000 7f89c9e00700 1 -- 172.21.15.12:0/3889176549 learned_addr learned my addr 172.21.15.12:0/3889176549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.092+0000 7f89c9400700 1 --2- 172.21.15.12:0/3889176549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89cc0d16a0 0x7f89cc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.092+0000 7f89c9e00700 1 -- 172.21.15.12:0/3889176549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89cc0d16a0 msgr2=0x7f89cc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.092+0000 7f89c9e00700 1 --2- 172.21.15.12:0/3889176549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89cc0d16a0 0x7f89cc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.092+0000 7f89c9e00700 1 -- 172.21.15.12:0/3889176549 >> v1:172.21.15.159:6789/0 conn(0x7f89cc0d2370 legacy=0x7f89cc15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.092+0000 7f89c9e00700 1 -- 172.21.15.12:0/3889176549 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc1619e0 con 0x7f89cc15d9d0 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.093+0000 7f89c9e00700 1 --2- 172.21.15.12:0/3889176549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89cc15d9d0 0x7f89cc0d3c30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0098f0 tx=0x7f89bc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98cebaedf56f909 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.093+0000 7f89c8a00700 1 -- 172.21.15.12:0/3889176549 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc011830 con 0x7f89cc15d9d0 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.093+0000 7f89c8a00700 1 -- 172.21.15.12:0/3889176549 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89bc011990 con 0x7f89cc15d9d0 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.093+0000 7f89c8a00700 1 -- 172.21.15.12:0/3889176549 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc011d20 con 0x7f89cc15d9d0 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.093+0000 7f89d1000700 1 -- 172.21.15.12:0/3889176549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89cc15d9d0 msgr2=0x7f89cc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.093+0000 7f89d1000700 1 --2- 172.21.15.12:0/3889176549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89cc15d9d0 0x7f89cc0d3c30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0098f0 tx=0x7f89bc009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.094+0000 7f89d1000700 1 -- 172.21.15.12:0/3889176549 shutdown_connections 2024-03-29T09:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.094+0000 7f89d1000700 1 --2- 172.21.15.12:0/3889176549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89cc15d9d0 0x7f89cc0d3c30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.094+0000 7f89d1000700 1 --2- 172.21.15.12:0/3889176549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89cc0d16a0 0x7f89cc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.094+0000 7f89d1000700 1 -- 172.21.15.12:0/3889176549 >> 172.21.15.12:0/3889176549 conn(0x7f89cc08aff0 msgr2=0x7f89cc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.094+0000 7f89d1000700 1 -- 172.21.15.12:0/3889176549 shutdown_connections 2024-03-29T09:55:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.094+0000 7f89d1000700 1 -- 172.21.15.12:0/3889176549 wait complete. 2024-03-29T09:55:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.095+0000 7f89d1000700 1 Processor -- start 2024-03-29T09:55:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.096+0000 7f89d1000700 1 -- start start 2024-03-29T09:55:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.096+0000 7f89d1000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89cc0d16a0 0x7f89cc155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.096+0000 7f89d1000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89cc0d2370 0x7f89cc0b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.096+0000 7f89d1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89cc15d9d0 0x7f89cc159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.096+0000 7f89d1000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc15e810 con 0x7f89cc15d9d0 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.096+0000 7f89d1000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc0b8df0 con 0x7f89cc0d2370 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.096+0000 7f89d1000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc15dee0 con 0x7f89cc0d16a0 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89ca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89cc15d9d0 0x7f89cc159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89c9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89cc0d16a0 0x7f89cc155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89ca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89cc15d9d0 0x7f89cc159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37682/0 (socket says 172.21.15.12:37682) 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89ca800700 1 -- 172.21.15.12:0/768994886 learned_addr learned my addr 172.21.15.12:0/768994886 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89c9400700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89cc0d2370 0x7f89cc0b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89ca800700 1 -- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89cc0d16a0 msgr2=0x7f89cc155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89ca800700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89cc0d16a0 0x7f89cc155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89ca800700 1 -- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89cc0d2370 msgr2=0x7f89cc0b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89ca800700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89cc0d2370 0x7f89cc0b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89ca800700 1 -- 172.21.15.12:0/768994886 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc009550 con 0x7f89cc15d9d0 2024-03-29T09:55:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.097+0000 7f89ca800700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89cc15d9d0 0x7f89cc159c50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f89c400cc50 tx=0x7f89c400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.098+0000 7f89c2000700 1 -- 172.21.15.12:0/768994886 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c4003c00 con 0x7f89cc15d9d0 2024-03-29T09:55:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.098+0000 7f89c2000700 1 -- 172.21.15.12:0/768994886 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c4003d60 con 0x7f89cc15d9d0 2024-03-29T09:55:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.098+0000 7f89c2000700 1 -- 172.21.15.12:0/768994886 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c400aab0 con 0x7f89cc15d9d0 2024-03-29T09:55:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.098+0000 7f89d1000700 1 -- 172.21.15.12:0/768994886 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc15e1c0 con 0x7f89cc15d9d0 2024-03-29T09:55:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.098+0000 7f89d1000700 1 -- 172.21.15.12:0/768994886 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc155560 con 0x7f89cc15d9d0 2024-03-29T09:55:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.099+0000 7f89d1000700 1 -- 172.21.15.12:0/768994886 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89cc0d1ca0 con 0x7f89cc15d9d0 2024-03-29T09:55:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.100+0000 7f89c2000700 1 -- 172.21.15.12:0/768994886 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f89c400ac10 con 0x7f89cc15d9d0 2024-03-29T09:55:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.100+0000 7f89c2000700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f89ac06ac90 0x7f89ac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.101+0000 7f89c2000700 1 -- 172.21.15.12:0/768994886 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89c40892a0 con 0x7f89cc15d9d0 2024-03-29T09:55:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.101+0000 7f89c9e00700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f89ac06ac90 0x7f89ac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.102+0000 7f89c9e00700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f89ac06ac90 0x7f89ac06d150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f89bc014040 tx=0x7f89bc008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.105+0000 7f89c2000700 1 -- 172.21.15.12:0/768994886 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f89c4054470 con 0x7f89cc15d9d0 2024-03-29T09:55:02.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.320+0000 7f89d1000700 1 -- 172.21.15.12:0/768994886 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89cc067720 con 0x7f89cc15d9d0 2024-03-29T09:55:02.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.320+0000 7f89c2000700 1 -- 172.21.15.12:0/768994886 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89c4057a90 con 0x7f89cc15d9d0 2024-03-29T09:55:02.320 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 -- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f89ac06ac90 msgr2=0x7f89ac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f89ac06ac90 0x7f89ac06d150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f89bc014040 tx=0x7f89bc008030 comp rx=0 tx=0).stop 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 -- 172.21.15.12:0/768994886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89cc15d9d0 msgr2=0x7f89cc159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89cc15d9d0 0x7f89cc159c50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f89c400cc50 tx=0x7f89c400cf60 comp rx=0 tx=0).stop 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 -- 172.21.15.12:0/768994886 shutdown_connections 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f89ac06ac90 0x7f89ac06d150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89cc15d9d0 0x7f89cc159c50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89cc0d16a0 0x7f89cc155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 --2- 172.21.15.12:0/768994886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89cc0d2370 0x7f89cc0b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 -- 172.21.15.12:0/768994886 >> 172.21.15.12:0/768994886 conn(0x7f89cc08aff0 msgr2=0x7f89cc0b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 -- 172.21.15.12:0/768994886 shutdown_connections 2024-03-29T09:55:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.323+0000 7f89d1000700 1 -- 172.21.15.12:0/768994886 wait complete. 2024-03-29T09:55:02.340 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:55:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.541+0000 7f1805a00700 1 Processor -- start 2024-03-29T09:55:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.541+0000 7f1805a00700 1 -- start start 2024-03-29T09:55:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.542+0000 7f1805a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f180015d990 0x7f18000d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.542+0000 7f1805a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180009a750 0x7f180006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.542+0000 7f1805a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18000d0970 con 0x7f1800099a60 2024-03-29T09:55:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.542+0000 7f1805a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180006c020 con 0x7f180015d990 2024-03-29T09:55:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.542+0000 7f1805a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18000c81a0 con 0x7f180009a750 2024-03-29T09:55:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.542+0000 7f17ff400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180009a750 0x7f180006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.543+0000 7f17fe000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f180015d990 0x7f18000d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.543+0000 7f17fe000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f180015d990 0x7f18000d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:59454/0 (socket says 172.21.15.12:59454) 2024-03-29T09:55:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.543+0000 7f17fe000700 1 -- 172.21.15.12:0/690774821 learned_addr learned my addr 172.21.15.12:0/690774821 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.543+0000 7f17ff400700 1 -- 172.21.15.12:0/690774821 >> v2:172.21.15.159:3300/0 conn(0x7f180015d990 msgr2=0x7f18000d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.543+0000 7f17ff400700 1 --2- 172.21.15.12:0/690774821 >> v2:172.21.15.159:3300/0 conn(0x7f180015d990 0x7f18000d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.543+0000 7f17ff400700 1 -- 172.21.15.12:0/690774821 >> v1:172.21.15.159:6789/0 conn(0x7f1800099a60 legacy=0x7f1800099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.543+0000 7f17ff400700 1 -- 172.21.15.12:0/690774821 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18000d0690 con 0x7f180009a750 2024-03-29T09:55:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.543+0000 7f17fe000700 1 --2- 172.21.15.12:0/690774821 >> v2:172.21.15.159:3300/0 conn(0x7f180015d990 0x7f18000d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.543+0000 7f17ff400700 1 --2- 172.21.15.12:0/690774821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180009a750 0x7f180006c4b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009d90 tx=0x7f17f800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46d5208e0945624d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.544+0000 7f17fd600700 1 -- 172.21.15.12:0/690774821 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f800e070 con 0x7f180009a750 2024-03-29T09:55:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.544+0000 7f17fd600700 1 -- 172.21.15.12:0/690774821 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f800bb20 con 0x7f180009a750 2024-03-29T09:55:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.544+0000 7f17fd600700 1 -- 172.21.15.12:0/690774821 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8012430 con 0x7f180009a750 2024-03-29T09:55:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.544+0000 7f1805a00700 1 -- 172.21.15.12:0/690774821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180009a750 msgr2=0x7f180006c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.544+0000 7f1805a00700 1 --2- 172.21.15.12:0/690774821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180009a750 0x7f180006c4b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009d90 tx=0x7f17f800b330 comp rx=0 tx=0).stop 2024-03-29T09:55:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.545+0000 7f1805a00700 1 -- 172.21.15.12:0/690774821 shutdown_connections 2024-03-29T09:55:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.545+0000 7f1805a00700 1 --2- 172.21.15.12:0/690774821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180009a750 0x7f180006c4b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.545+0000 7f1805a00700 1 --2- 172.21.15.12:0/690774821 >> v2:172.21.15.159:3300/0 conn(0x7f180015d990 0x7f18000d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.545+0000 7f1805a00700 1 -- 172.21.15.12:0/690774821 >> 172.21.15.12:0/690774821 conn(0x7f18000b2e20 msgr2=0x7f18000b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.545+0000 7f1805a00700 1 -- 172.21.15.12:0/690774821 shutdown_connections 2024-03-29T09:55:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.545+0000 7f1805a00700 1 -- 172.21.15.12:0/690774821 wait complete. 2024-03-29T09:55:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.546+0000 7f1805a00700 1 Processor -- start 2024-03-29T09:55:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.547+0000 7f1805a00700 1 -- start start 2024-03-29T09:55:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.547+0000 7f1805a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1800099a60 0x7f1800157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.547+0000 7f1805a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f180009a750 0x7f180015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.547+0000 7f1805a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180015d990 0x7f180015e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.547+0000 7f1805a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18000c9920 con 0x7f180009a750 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.547+0000 7f1805a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1800160150 con 0x7f1800099a60 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.547+0000 7f1805a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180015e8e0 con 0x7f180015d990 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.548+0000 7f17fe000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f180009a750 0x7f180015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.548+0000 7f17fe000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f180009a750 0x7f180015dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37696/0 (socket says 172.21.15.12:37696) 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.548+0000 7f17fe000700 1 -- 172.21.15.12:0/1849954682 learned_addr learned my addr 172.21.15.12:0/1849954682 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.548+0000 7f17ff400700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180015d990 0x7f180015e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.548+0000 7f17fea00700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1800099a60 0x7f1800157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.548+0000 7f17fe000700 1 -- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180015d990 msgr2=0x7f180015e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.548+0000 7f17fe000700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180015d990 0x7f180015e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.548+0000 7f17fe000700 1 -- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1800099a60 msgr2=0x7f1800157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.548+0000 7f17fe000700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1800099a60 0x7f1800157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.548+0000 7f17fe000700 1 -- 172.21.15.12:0/1849954682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec009550 con 0x7f180009a750 2024-03-29T09:55:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.549+0000 7f17fe000700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f180009a750 0x7f180015dd70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00e9b0 tx=0x7f17ec00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.549+0000 7f17f6a00700 1 -- 172.21.15.12:0/1849954682 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec00ee50 con 0x7f180009a750 2024-03-29T09:55:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.549+0000 7f17f6a00700 1 -- 172.21.15.12:0/1849954682 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17ec00c850 con 0x7f180009a750 2024-03-29T09:55:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.549+0000 7f1805a00700 1 -- 172.21.15.12:0/1849954682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f8009a70 con 0x7f180009a750 2024-03-29T09:55:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.549+0000 7f17f6a00700 1 -- 172.21.15.12:0/1849954682 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec00cbc0 con 0x7f180009a750 2024-03-29T09:55:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.549+0000 7f1805a00700 1 -- 172.21.15.12:0/1849954682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180015eea0 con 0x7f180009a750 2024-03-29T09:55:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.550+0000 7f1805a00700 1 -- 172.21.15.12:0/1849954682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1800067720 con 0x7f180009a750 2024-03-29T09:55:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.551+0000 7f17f6a00700 1 -- 172.21.15.12:0/1849954682 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f17ec00cd20 con 0x7f180009a750 2024-03-29T09:55:02.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.552+0000 7f17f6a00700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f17e006ac90 0x7f17e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.552+0000 7f17f6a00700 1 -- 172.21.15.12:0/1849954682 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17ec0892f0 con 0x7f180009a750 2024-03-29T09:55:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.552+0000 7f17fea00700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f17e006ac90 0x7f17e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.553+0000 7f17fea00700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f17e006ac90 0x7f17e006d150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1800070300 tx=0x7f17f0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.557+0000 7f17f6a00700 1 -- 172.21.15.12:0/1849954682 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17ec0544c0 con 0x7f180009a750 2024-03-29T09:55:02.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.772+0000 7f1805a00700 1 -- 172.21.15.12:0/1849954682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f180009ac30 con 0x7f180009a750 2024-03-29T09:55:02.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.772+0000 7f17f6a00700 1 -- 172.21.15.12:0/1849954682 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f17ec057ae0 con 0x7f180009a750 2024-03-29T09:55:02.772 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:55:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 -- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f17e006ac90 msgr2=0x7f17e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f17e006ac90 0x7f17e006d150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1800070300 tx=0x7f17f0009040 comp rx=0 tx=0).stop 2024-03-29T09:55:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 -- 172.21.15.12:0/1849954682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f180009a750 msgr2=0x7f180015dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f180009a750 0x7f180015dd70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00e9b0 tx=0x7f17ec00ecc0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 -- 172.21.15.12:0/1849954682 shutdown_connections 2024-03-29T09:55:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f17e006ac90 0x7f17e006d150 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f180015d990 0x7f180015e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1800099a60 0x7f1800157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 --2- 172.21.15.12:0/1849954682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f180009a750 0x7f180015dd70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 -- 172.21.15.12:0/1849954682 >> 172.21.15.12:0/1849954682 conn(0x7f18000b2e20 msgr2=0x7f18000cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 -- 172.21.15.12:0/1849954682 shutdown_connections 2024-03-29T09:55:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.775+0000 7f1805a00700 1 -- 172.21.15.12:0/1849954682 wait complete. 2024-03-29T09:55:02.792 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:02.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.990+0000 7f229c400700 1 Processor -- start 2024-03-29T09:55:02.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.990+0000 7f229c400700 1 -- start start 2024-03-29T09:55:02.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.991+0000 7f229c400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22941619d0 0x7f22941685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.991+0000 7f229c400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22940d30d0 con 0x7f229415d640 2024-03-29T09:55:02.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.991+0000 7f229c400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22940b6840 con 0x7f22940ceec0 2024-03-29T09:55:02.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.991+0000 7f229c400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22940d2b30 con 0x7f22941619d0 2024-03-29T09:55:02.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.991+0000 7f229a400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22941619d0 0x7f22941685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.991+0000 7f2299a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f229415d640 0x7f22940d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33616/0 (socket says 172.21.15.12:33616) 2024-03-29T09:55:02.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.992+0000 7f2299a00700 1 -- 172.21.15.12:0/2038003142 learned_addr learned my addr 172.21.15.12:0/2038003142 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:02.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.992+0000 7f229a400700 1 -- 172.21.15.12:0/2038003142 >> v1:172.21.15.159:6789/0 conn(0x7f22940ceec0 legacy=0x7f22941612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.992+0000 7f229a400700 1 -- 172.21.15.12:0/2038003142 >> v1:172.21.15.12:6789/0 conn(0x7f229415d640 legacy=0x7f22940d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.992+0000 7f229a400700 1 -- 172.21.15.12:0/2038003142 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22940d2df0 con 0x7f22941619d0 2024-03-29T09:55:02.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.992+0000 7f229a400700 1 --2- 172.21.15.12:0/2038003142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22941619d0 0x7f22941685f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f228c009d90 tx=0x7f228c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b215278c4d6d3fe server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:02.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.992+0000 7f2293e00700 1 -- 172.21.15.12:0/2038003142 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c00e070 con 0x7f22941619d0 2024-03-29T09:55:02.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.993+0000 7f2293e00700 1 -- 172.21.15.12:0/2038003142 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c00bbd0 con 0x7f22941619d0 2024-03-29T09:55:02.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.993+0000 7f2293e00700 1 -- 172.21.15.12:0/2038003142 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c012430 con 0x7f22941619d0 2024-03-29T09:55:02.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.993+0000 7f229c400700 1 -- 172.21.15.12:0/2038003142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22941619d0 msgr2=0x7f22941685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.993+0000 7f229c400700 1 --2- 172.21.15.12:0/2038003142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22941619d0 0x7f22941685f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f228c009d90 tx=0x7f228c00b330 comp rx=0 tx=0).stop 2024-03-29T09:55:02.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.993+0000 7f229c400700 1 -- 172.21.15.12:0/2038003142 shutdown_connections 2024-03-29T09:55:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.993+0000 7f229c400700 1 --2- 172.21.15.12:0/2038003142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22941619d0 0x7f22941685f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.993+0000 7f229c400700 1 -- 172.21.15.12:0/2038003142 >> 172.21.15.12:0/2038003142 conn(0x7f229408aff0 msgr2=0x7f22940d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.993+0000 7f229c400700 1 -- 172.21.15.12:0/2038003142 shutdown_connections 2024-03-29T09:55:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.993+0000 7f229c400700 1 -- 172.21.15.12:0/2038003142 wait complete. 2024-03-29T09:55:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.995+0000 7f229c400700 1 Processor -- start 2024-03-29T09:55:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.995+0000 7f229c400700 1 -- start start 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f229c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22940ceec0 0x7f229415bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f229c400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229415d640 0x7f22941e8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f229c400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f22941619d0 0x7f22941e8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f229c400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22940a03b0 con 0x7f229415d640 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f229c400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229409e330 con 0x7f22941619d0 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f229c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229409e610 con 0x7f22940ceec0 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f2299000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229415d640 0x7f22941e8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f2299000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229415d640 0x7f22941e8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37710/0 (socket says 172.21.15.12:37710) 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f2299000700 1 -- 172.21.15.12:0/4166593321 learned_addr learned my addr 172.21.15.12:0/4166593321 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f229a400700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f22941619d0 0x7f22941e8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.996+0000 7f2299a00700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22940ceec0 0x7f229415bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f2299000700 1 -- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22940ceec0 msgr2=0x7f229415bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f2299000700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22940ceec0 0x7f229415bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f2299000700 1 -- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f22941619d0 msgr2=0x7f22941e8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f2299000700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f22941619d0 0x7f22941e8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f2299000700 1 -- 172.21.15.12:0/4166593321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c009a70 con 0x7f229415d640 2024-03-29T09:55:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f2299000700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229415d640 0x7f22941e8110 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f228000c4e0 tx=0x7f228000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f2291600700 1 -- 172.21.15.12:0/4166593321 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22800052d0 con 0x7f229415d640 2024-03-29T09:55:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f2291600700 1 -- 172.21.15.12:0/4166593321 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2280005430 con 0x7f229415d640 2024-03-29T09:55:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f229c400700 1 -- 172.21.15.12:0/4166593321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229409e8f0 con 0x7f229415d640 2024-03-29T09:55:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f2291600700 1 -- 172.21.15.12:0/4166593321 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22800038c0 con 0x7f229415d640 2024-03-29T09:55:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.997+0000 7f229c400700 1 -- 172.21.15.12:0/4166593321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229409ff40 con 0x7f229415d640 2024-03-29T09:55:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.998+0000 7f229c400700 1 -- 172.21.15.12:0/4166593321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22940a01c0 con 0x7f229415d640 2024-03-29T09:55:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:02.999+0000 7f2291600700 1 -- 172.21.15.12:0/4166593321 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2280003a20 con 0x7f229415d640 2024-03-29T09:55:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.000+0000 7f2291600700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227406ac90 0x7f227406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.000+0000 7f2291600700 1 -- 172.21.15.12:0/4166593321 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2280088760 con 0x7f229415d640 2024-03-29T09:55:03.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.001+0000 7f2299a00700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227406ac90 0x7f227406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.001+0000 7f2299a00700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227406ac90 0x7f227406d150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2288005910 tx=0x7f2288009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.005+0000 7f2291600700 1 -- 172.21.15.12:0/4166593321 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f22800538a0 con 0x7f229415d640 2024-03-29T09:55:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.219+0000 7f229c400700 1 -- 172.21.15.12:0/4166593321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2294067720 con 0x7f229415d640 2024-03-29T09:55:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.220+0000 7f2291600700 1 -- 172.21.15.12:0/4166593321 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f2280056ec0 con 0x7f229415d640 2024-03-29T09:55:03.220 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:03.220 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 -- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227406ac90 msgr2=0x7f227406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227406ac90 0x7f227406d150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2288005910 tx=0x7f2288009040 comp rx=0 tx=0).stop 2024-03-29T09:55:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 -- 172.21.15.12:0/4166593321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229415d640 msgr2=0x7f22941e8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229415d640 0x7f22941e8110 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f228000c4e0 tx=0x7f228000c7f0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 -- 172.21.15.12:0/4166593321 shutdown_connections 2024-03-29T09:55:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227406ac90 0x7f227406d150 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f22941619d0 0x7f22941e8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22940ceec0 0x7f229415bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 --2- 172.21.15.12:0/4166593321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229415d640 0x7f22941e8110 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 -- 172.21.15.12:0/4166593321 >> 172.21.15.12:0/4166593321 conn(0x7f229408aff0 msgr2=0x7f22940b8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 -- 172.21.15.12:0/4166593321 shutdown_connections 2024-03-29T09:55:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.222+0000 7f229c400700 1 -- 172.21.15.12:0/4166593321 wait complete. 2024-03-29T09:55:03.240 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:55:03.240 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:55:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.441+0000 7f75e5200700 1 Processor -- start 2024-03-29T09:55:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.441+0000 7f75e5200700 1 -- start start 2024-03-29T09:55:03.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.442+0000 7f75e5200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f75e015d9d0 0x7f75e00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.442+0000 7f75e5200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e00d16a0 0x7f75e00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.442+0000 7f75e5200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e0161420 con 0x7f75e00d2370 2024-03-29T09:55:03.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.442+0000 7f75e5200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e00d2820 con 0x7f75e015d9d0 2024-03-29T09:55:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.442+0000 7f75e5200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0161700 con 0x7f75e00d16a0 2024-03-29T09:55:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.443+0000 7f75dde00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f75e015d9d0 0x7f75e00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.443+0000 7f75dde00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f75e015d9d0 0x7f75e00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37722/0 (socket says 172.21.15.12:37722) 2024-03-29T09:55:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.443+0000 7f75dde00700 1 -- 172.21.15.12:0/1717481349 learned_addr learned my addr 172.21.15.12:0/1717481349 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.443+0000 7f75dd400700 1 --2- 172.21.15.12:0/1717481349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e00d16a0 0x7f75e00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.443+0000 7f75dde00700 1 -- 172.21.15.12:0/1717481349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e00d16a0 msgr2=0x7f75e00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.443+0000 7f75dde00700 1 --2- 172.21.15.12:0/1717481349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e00d16a0 0x7f75e00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.443+0000 7f75dde00700 1 -- 172.21.15.12:0/1717481349 >> v1:172.21.15.159:6789/0 conn(0x7f75e00d2370 legacy=0x7f75e015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.443+0000 7f75dde00700 1 -- 172.21.15.12:0/1717481349 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e01619e0 con 0x7f75e015d9d0 2024-03-29T09:55:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.443+0000 7f75dd400700 1 --2- 172.21.15.12:0/1717481349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e00d16a0 0x7f75e00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75dde00700 1 --2- 172.21.15.12:0/1717481349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e015d9d0 0x7f75e00d3c30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f75d00098f0 tx=0x7f75d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1abd0a5f76f43b2f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75dca00700 1 -- 172.21.15.12:0/1717481349 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d000b8a0 con 0x7f75e015d9d0 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75dca00700 1 -- 172.21.15.12:0/1717481349 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d000ba00 con 0x7f75e015d9d0 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75dca00700 1 -- 172.21.15.12:0/1717481349 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d000bd90 con 0x7f75e015d9d0 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75e5200700 1 -- 172.21.15.12:0/1717481349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e015d9d0 msgr2=0x7f75e00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75e5200700 1 --2- 172.21.15.12:0/1717481349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e015d9d0 0x7f75e00d3c30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f75d00098f0 tx=0x7f75d0009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75e5200700 1 -- 172.21.15.12:0/1717481349 shutdown_connections 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75e5200700 1 --2- 172.21.15.12:0/1717481349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e015d9d0 0x7f75e00d3c30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75e5200700 1 --2- 172.21.15.12:0/1717481349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e00d16a0 0x7f75e00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75e5200700 1 -- 172.21.15.12:0/1717481349 >> 172.21.15.12:0/1717481349 conn(0x7f75e008aff0 msgr2=0x7f75e00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.444+0000 7f75e5200700 1 -- 172.21.15.12:0/1717481349 shutdown_connections 2024-03-29T09:55:03.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.445+0000 7f75e5200700 1 -- 172.21.15.12:0/1717481349 wait complete. 2024-03-29T09:55:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.446+0000 7f75e5200700 1 Processor -- start 2024-03-29T09:55:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.446+0000 7f75e5200700 1 -- start start 2024-03-29T09:55:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.447+0000 7f75e5200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75e00d16a0 0x7f75e0155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.447+0000 7f75e5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e00d2370 0x7f75e00b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.447+0000 7f75e5200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e015d9d0 0x7f75e0159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.447+0000 7f75e5200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e015e810 con 0x7f75e00d2370 2024-03-29T09:55:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.447+0000 7f75e5200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e00b8df0 con 0x7f75e00d16a0 2024-03-29T09:55:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.447+0000 7f75e5200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e015dee0 con 0x7f75e015d9d0 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.447+0000 7f75dd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e00d2370 0x7f75e00b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.447+0000 7f75dd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e00d2370 0x7f75e00b8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37734/0 (socket says 172.21.15.12:37734) 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.447+0000 7f75dd400700 1 -- 172.21.15.12:0/3358438227 learned_addr learned my addr 172.21.15.12:0/3358438227 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.448+0000 7f75dde00700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75e00d16a0 0x7f75e0155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.448+0000 7f75de800700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e015d9d0 0x7f75e0159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.448+0000 7f75dd400700 1 -- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e015d9d0 msgr2=0x7f75e0159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.448+0000 7f75dd400700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e015d9d0 0x7f75e0159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.448+0000 7f75dd400700 1 -- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75e00d16a0 msgr2=0x7f75e0155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.448+0000 7f75dd400700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75e00d16a0 0x7f75e0155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.448+0000 7f75dd400700 1 -- 172.21.15.12:0/3358438227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d0009550 con 0x7f75e00d2370 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.448+0000 7f75dd400700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e00d2370 0x7f75e00b8480 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00e810 tx=0x7f75cc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.448+0000 7f75d6000700 1 -- 172.21.15.12:0/3358438227 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc00c070 con 0x7f75e00d2370 2024-03-29T09:55:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.449+0000 7f75d6000700 1 -- 172.21.15.12:0/3358438227 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75cc005c20 con 0x7f75e00d2370 2024-03-29T09:55:03.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.449+0000 7f75d6000700 1 -- 172.21.15.12:0/3358438227 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc006030 con 0x7f75e00d2370 2024-03-29T09:55:03.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.449+0000 7f75e5200700 1 -- 172.21.15.12:0/3358438227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e015e1c0 con 0x7f75e00d2370 2024-03-29T09:55:03.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.449+0000 7f75e5200700 1 -- 172.21.15.12:0/3358438227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e0155560 con 0x7f75e00d2370 2024-03-29T09:55:03.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.450+0000 7f75e5200700 1 -- 172.21.15.12:0/3358438227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e00d1ca0 con 0x7f75e00d2370 2024-03-29T09:55:03.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.451+0000 7f75d6000700 1 -- 172.21.15.12:0/3358438227 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f75cc006190 con 0x7f75e00d2370 2024-03-29T09:55:03.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.451+0000 7f75d6000700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75c006ac90 0x7f75c006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.452+0000 7f75d6000700 1 -- 172.21.15.12:0/3358438227 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75cc088fc0 con 0x7f75e00d2370 2024-03-29T09:55:03.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.452+0000 7f75dde00700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75c006ac90 0x7f75c006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.453+0000 7f75dde00700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75c006ac90 0x7f75c006d150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f75d0014040 tx=0x7f75d0008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:03.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.456+0000 7f75d6000700 1 -- 172.21.15.12:0/3358438227 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f75cc054190 con 0x7f75e00d2370 2024-03-29T09:55:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.667+0000 7f75e5200700 1 -- 172.21.15.12:0/3358438227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75e0067720 con 0x7f75e00d2370 2024-03-29T09:55:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.668+0000 7f75d6000700 1 -- 172.21.15.12:0/3358438227 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75cc0577b0 con 0x7f75e00d2370 2024-03-29T09:55:03.667 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:55:03.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 -- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75c006ac90 msgr2=0x7f75c006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75c006ac90 0x7f75c006d150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f75d0014040 tx=0x7f75d0008030 comp rx=0 tx=0).stop 2024-03-29T09:55:03.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 -- 172.21.15.12:0/3358438227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e00d2370 msgr2=0x7f75e00b8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e00d2370 0x7f75e00b8480 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00e810 tx=0x7f75cc00eb20 comp rx=0 tx=0).stop 2024-03-29T09:55:03.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 -- 172.21.15.12:0/3358438227 shutdown_connections 2024-03-29T09:55:03.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75c006ac90 0x7f75c006d150 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75e015d9d0 0x7f75e0159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75e00d16a0 0x7f75e0155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 --2- 172.21.15.12:0/3358438227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75e00d2370 0x7f75e00b8480 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 -- 172.21.15.12:0/3358438227 >> 172.21.15.12:0/3358438227 conn(0x7f75e008aff0 msgr2=0x7f75e00b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 -- 172.21.15.12:0/3358438227 shutdown_connections 2024-03-29T09:55:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.670+0000 7f75e5200700 1 -- 172.21.15.12:0/3358438227 wait complete. 2024-03-29T09:55:03.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:55:03.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.890+0000 7fc9e1600700 1 Processor -- start 2024-03-29T09:55:03.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.890+0000 7fc9e1600700 1 -- start start 2024-03-29T09:55:03.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.891+0000 7fc9e1600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc15d9d0 0x7fc9dc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.891+0000 7fc9e1600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc9dc0d16a0 0x7fc9dc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.891+0000 7fc9e1600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc161420 con 0x7fc9dc0d2370 2024-03-29T09:55:03.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.891+0000 7fc9e1600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9dc0d2820 con 0x7fc9dc0d16a0 2024-03-29T09:55:03.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.891+0000 7fc9e1600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc161700 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.891+0000 7fc9db400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc9dc0d2370 0x7fc9dc15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33624/0 (socket says 172.21.15.12:33624) 2024-03-29T09:55:03.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.891+0000 7fc9db400700 1 -- 172.21.15.12:0/1320831838 learned_addr learned my addr 172.21.15.12:0/1320831838 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:03.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.891+0000 7fc9daa00700 1 --2- 172.21.15.12:0/1320831838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc15d9d0 0x7fc9dc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.891+0000 7fc9da000700 1 --2- 172.21.15.12:0/1320831838 >> v2:172.21.15.159:3300/0 conn(0x7fc9dc0d16a0 0x7fc9dc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.892+0000 7fc9d9600700 1 -- 172.21.15.12:0/1320831838 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099944683 0 0) 0x7fc9dc161420 con 0x7fc9dc0d2370 2024-03-29T09:55:03.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.892+0000 7fc9d9600700 1 -- 172.21.15.12:0/1320831838 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c00035b0 con 0x7fc9dc0d2370 2024-03-29T09:55:03.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.892+0000 7fc9daa00700 1 -- 172.21.15.12:0/1320831838 >> v2:172.21.15.159:3300/0 conn(0x7fc9dc0d16a0 msgr2=0x7fc9dc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:03.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.892+0000 7fc9daa00700 1 --2- 172.21.15.12:0/1320831838 >> v2:172.21.15.159:3300/0 conn(0x7fc9dc0d16a0 0x7fc9dc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.892+0000 7fc9daa00700 1 -- 172.21.15.12:0/1320831838 >> v1:172.21.15.12:6789/0 conn(0x7fc9dc0d2370 legacy=0x7fc9dc15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.892+0000 7fc9d9600700 1 -- 172.21.15.12:0/1320831838 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3009321439 0 0) 0x7fc9c00035b0 con 0x7fc9dc0d2370 2024-03-29T09:55:03.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.892+0000 7fc9daa00700 1 -- 172.21.15.12:0/1320831838 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc1619e0 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.892+0000 7fc9da000700 1 --2- 172.21.15.12:0/1320831838 >> v2:172.21.15.159:3300/0 conn(0x7fc9dc0d16a0 0x7fc9dc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.892+0000 7fc9daa00700 1 --2- 172.21.15.12:0/1320831838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc15d9d0 0x7fc9dc0d3c30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc009840 tx=0x7fc9cc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6a0fbaf0b5605b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.893+0000 7fc9d9600700 1 -- 172.21.15.12:0/1320831838 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d40019f0 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.893+0000 7fc9d9600700 1 -- 172.21.15.12:0/1320831838 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9cc004030 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.893+0000 7fc9d9600700 1 -- 172.21.15.12:0/1320831838 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc0043c0 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.893+0000 7fc9e1600700 1 -- 172.21.15.12:0/1320831838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc15d9d0 msgr2=0x7fc9dc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.893+0000 7fc9e1600700 1 --2- 172.21.15.12:0/1320831838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc15d9d0 0x7fc9dc0d3c30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc009840 tx=0x7fc9cc009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.893+0000 7fc9e1600700 1 -- 172.21.15.12:0/1320831838 shutdown_connections 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.893+0000 7fc9e1600700 1 --2- 172.21.15.12:0/1320831838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc15d9d0 0x7fc9dc0d3c30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.893+0000 7fc9e1600700 1 --2- 172.21.15.12:0/1320831838 >> v2:172.21.15.159:3300/0 conn(0x7fc9dc0d16a0 0x7fc9dc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.893+0000 7fc9e1600700 1 -- 172.21.15.12:0/1320831838 >> 172.21.15.12:0/1320831838 conn(0x7fc9dc08aff0 msgr2=0x7fc9dc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.893+0000 7fc9e1600700 1 -- 172.21.15.12:0/1320831838 shutdown_connections 2024-03-29T09:55:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.894+0000 7fc9e1600700 1 -- 172.21.15.12:0/1320831838 wait complete. 2024-03-29T09:55:03.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.895+0000 7fc9e1600700 1 Processor -- start 2024-03-29T09:55:03.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.895+0000 7fc9e1600700 1 -- start start 2024-03-29T09:55:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.896+0000 7fc9e1600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc0d16a0 0x7fc9dc14e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.896+0000 7fc9e1600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9dc0d2370 0x7fc9dc0b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.896+0000 7fc9e1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9dc15d9d0 0x7fc9dc0b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.896+0000 7fc9e1600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc15e5a0 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.896+0000 7fc9e1600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0b7b40 con 0x7fc9dc0d2370 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.896+0000 7fc9e1600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0b7860 con 0x7fc9dc0d16a0 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9dc15d9d0 0x7fc9dc0b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9daa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc0d16a0 0x7fc9dc14e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9daa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc0d16a0 0x7fc9dc14e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:40388/0 (socket says 172.21.15.12:40388) 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9daa00700 1 -- 172.21.15.12:0/3783528209 learned_addr learned my addr 172.21.15.12:0/3783528209 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9da000700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9dc0d2370 0x7fc9dc0b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9db400700 1 -- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc0d16a0 msgr2=0x7fc9dc14e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9db400700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc0d16a0 0x7fc9dc14e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9db400700 1 -- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9dc0d2370 msgr2=0x7fc9dc0b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9db400700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9dc0d2370 0x7fc9dc0b6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9db400700 1 -- 172.21.15.12:0/3783528209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c8009430 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.897+0000 7fc9da000700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9dc0d2370 0x7fc9dc0b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.898+0000 7fc9db400700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9dc15d9d0 0x7fc9dc0b49d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4001560 tx=0x7fc9d400ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.898+0000 7fc9d2a00700 1 -- 172.21.15.12:0/3783528209 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d400ab00 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.898+0000 7fc9d2a00700 1 -- 172.21.15.12:0/3783528209 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d400ac60 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.898+0000 7fc9d2a00700 1 -- 172.21.15.12:0/3783528209 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d40070d0 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.898+0000 7fc9e1600700 1 -- 172.21.15.12:0/3783528209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9cc0094a0 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.898+0000 7fc9e1600700 1 -- 172.21.15.12:0/3783528209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9dc15e230 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.899+0000 7fc9e1600700 1 -- 172.21.15.12:0/3783528209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9dc0d1ca0 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.900+0000 7fc9d2a00700 1 -- 172.21.15.12:0/3783528209 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc9d4007230 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.901+0000 7fc9d2a00700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc9bc06ac90 0x7fc9bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:03.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.901+0000 7fc9d2a00700 1 -- 172.21.15.12:0/3783528209 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9d4088f00 con 0x7fc9dc15d9d0 2024-03-29T09:55:03.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.902+0000 7fc9daa00700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc9bc06ac90 0x7fc9bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.902+0000 7fc9daa00700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc9bc06ac90 0x7fc9bc06d150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc000c00 tx=0x7fc9cc008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:03.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:03.906+0000 7fc9d2a00700 1 -- 172.21.15.12:0/3783528209 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc9d40540e0 con 0x7fc9dc15d9d0 2024-03-29T09:55:04.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.122+0000 7fc9e1600700 1 -- 172.21.15.12:0/3783528209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc9dc15ddb0 con 0x7fc9dc15d9d0 2024-03-29T09:55:04.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.123+0000 7fc9d2a00700 1 -- 172.21.15.12:0/3783528209 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc9dc15ddb0 con 0x7fc9dc15d9d0 2024-03-29T09:55:04.123 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:55:04.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.125+0000 7fc9e1600700 1 -- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc9bc06ac90 msgr2=0x7fc9bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.125+0000 7fc9e1600700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc9bc06ac90 0x7fc9bc06d150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc000c00 tx=0x7fc9cc008030 comp rx=0 tx=0).stop 2024-03-29T09:55:04.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.125+0000 7fc9e1600700 1 -- 172.21.15.12:0/3783528209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9dc15d9d0 msgr2=0x7fc9dc0b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.125+0000 7fc9e1600700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9dc15d9d0 0x7fc9dc0b49d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4001560 tx=0x7fc9d400ce90 comp rx=0 tx=0).stop 2024-03-29T09:55:04.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.125+0000 7fc9e1600700 1 -- 172.21.15.12:0/3783528209 shutdown_connections 2024-03-29T09:55:04.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.125+0000 7fc9e1600700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc9bc06ac90 0x7fc9bc06d150 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.125+0000 7fc9e1600700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9dc15d9d0 0x7fc9dc0b49d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.125+0000 7fc9e1600700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9dc0d16a0 0x7fc9dc14e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.125+0000 7fc9e1600700 1 --2- 172.21.15.12:0/3783528209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9dc0d2370 0x7fc9dc0b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.125+0000 7fc9e1600700 1 -- 172.21.15.12:0/3783528209 >> 172.21.15.12:0/3783528209 conn(0x7fc9dc08aff0 msgr2=0x7fc9dc0b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:04.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.126+0000 7fc9e1600700 1 -- 172.21.15.12:0/3783528209 shutdown_connections 2024-03-29T09:55:04.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.126+0000 7fc9e1600700 1 -- 172.21.15.12:0/3783528209 wait complete. 2024-03-29T09:55:04.143 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:55:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.344+0000 7f5f04400700 1 Processor -- start 2024-03-29T09:55:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.344+0000 7f5f04400700 1 -- start start 2024-03-29T09:55:04.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.345+0000 7f5f04400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5efc15d9d0 0x7f5efc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.345+0000 7f5f04400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 0x7f5efc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.345+0000 7f5f04400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc161420 con 0x7f5efc0d2370 2024-03-29T09:55:04.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.345+0000 7f5f04400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5efc0d2820 con 0x7f5efc15d9d0 2024-03-29T09:55:04.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.345+0000 7f5f04400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc161700 con 0x7f5efc0d16a0 2024-03-29T09:55:04.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.345+0000 7f5f01a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5efc15d9d0 0x7f5efc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.346+0000 7f5f01a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5efc15d9d0 0x7f5efc0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37760/0 (socket says 172.21.15.12:37760) 2024-03-29T09:55:04.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.346+0000 7f5f01a00700 1 -- 172.21.15.12:0/3684784977 learned_addr learned my addr 172.21.15.12:0/3684784977 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:04.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.346+0000 7f5f01000700 1 --2- 172.21.15.12:0/3684784977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 0x7f5efc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.346+0000 7f5f01a00700 1 -- 172.21.15.12:0/3684784977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 msgr2=0x7f5efc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:04.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.346+0000 7f5f01a00700 1 --2- 172.21.15.12:0/3684784977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 0x7f5efc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.346+0000 7f5f01a00700 1 -- 172.21.15.12:0/3684784977 >> v1:172.21.15.159:6789/0 conn(0x7f5efc0d2370 legacy=0x7f5efc15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.346+0000 7f5f01a00700 1 -- 172.21.15.12:0/3684784977 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc1619e0 con 0x7f5efc15d9d0 2024-03-29T09:55:04.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.346+0000 7f5f01000700 1 --2- 172.21.15.12:0/3684784977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 0x7f5efc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.346+0000 7f5f01a00700 1 --2- 172.21.15.12:0/3684784977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 0x7f5efc0d3c30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5ef00098f0 tx=0x7f5ef0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05de10f05507d6f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.346+0000 7f5efbe00700 1 -- 172.21.15.12:0/3684784977 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef000b8a0 con 0x7f5efc15d9d0 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.347+0000 7f5efbe00700 1 -- 172.21.15.12:0/3684784977 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef000ba00 con 0x7f5efc15d9d0 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.347+0000 7f5efbe00700 1 -- 172.21.15.12:0/3684784977 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef000bd90 con 0x7f5efc15d9d0 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.347+0000 7f5f04400700 1 -- 172.21.15.12:0/3684784977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 msgr2=0x7f5efc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.347+0000 7f5f04400700 1 --2- 172.21.15.12:0/3684784977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 0x7f5efc0d3c30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5ef00098f0 tx=0x7f5ef0009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.347+0000 7f5f04400700 1 -- 172.21.15.12:0/3684784977 shutdown_connections 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.347+0000 7f5f04400700 1 --2- 172.21.15.12:0/3684784977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 0x7f5efc0d3c30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.347+0000 7f5f04400700 1 --2- 172.21.15.12:0/3684784977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 0x7f5efc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.347+0000 7f5f04400700 1 -- 172.21.15.12:0/3684784977 >> 172.21.15.12:0/3684784977 conn(0x7f5efc08aff0 msgr2=0x7f5efc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.347+0000 7f5f04400700 1 -- 172.21.15.12:0/3684784977 shutdown_connections 2024-03-29T09:55:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.347+0000 7f5f04400700 1 -- 172.21.15.12:0/3684784977 wait complete. 2024-03-29T09:55:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.349+0000 7f5f04400700 1 Processor -- start 2024-03-29T09:55:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.349+0000 7f5f04400700 1 -- start start 2024-03-29T09:55:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f04400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 0x7f5efc155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f04400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5efc0d2370 0x7f5efc0b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f04400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 0x7f5efc159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f04400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc15e810 con 0x7f5efc15d9d0 2024-03-29T09:55:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f04400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc0b8df0 con 0x7f5efc0d2370 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f04400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc15dee0 con 0x7f5efc0d16a0 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f02400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 0x7f5efc159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f01a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 0x7f5efc155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f02400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 0x7f5efc159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37772/0 (socket says 172.21.15.12:37772) 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f02400700 1 -- 172.21.15.12:0/3170256763 learned_addr learned my addr 172.21.15.12:0/3170256763 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.350+0000 7f5f01000700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5efc0d2370 0x7f5efc0b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5f02400700 1 -- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 msgr2=0x7f5efc155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5f02400700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 0x7f5efc155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5f02400700 1 -- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5efc0d2370 msgr2=0x7f5efc0b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5f02400700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5efc0d2370 0x7f5efc0b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5f02400700 1 -- 172.21.15.12:0/3170256763 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef0009550 con 0x7f5efc15d9d0 2024-03-29T09:55:04.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5f02400700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 0x7f5efc159c50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400cc50 tx=0x7f5ef400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:04.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5ef9600700 1 -- 172.21.15.12:0/3170256763 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4003c00 con 0x7f5efc15d9d0 2024-03-29T09:55:04.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5ef9600700 1 -- 172.21.15.12:0/3170256763 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef4003d60 con 0x7f5efc15d9d0 2024-03-29T09:55:04.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5ef9600700 1 -- 172.21.15.12:0/3170256763 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef400aab0 con 0x7f5efc15d9d0 2024-03-29T09:55:04.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5f04400700 1 -- 172.21.15.12:0/3170256763 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5efc15e1c0 con 0x7f5efc15d9d0 2024-03-29T09:55:04.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.351+0000 7f5f04400700 1 -- 172.21.15.12:0/3170256763 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5efc155560 con 0x7f5efc15d9d0 2024-03-29T09:55:04.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.353+0000 7f5f04400700 1 -- 172.21.15.12:0/3170256763 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5efc0d1ca0 con 0x7f5efc15d9d0 2024-03-29T09:55:04.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.354+0000 7f5ef9600700 1 -- 172.21.15.12:0/3170256763 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5ef400ac10 con 0x7f5efc15d9d0 2024-03-29T09:55:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.354+0000 7f5ef9600700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5edc06ac90 0x7f5edc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.355+0000 7f5ef9600700 1 -- 172.21.15.12:0/3170256763 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ef4089200 con 0x7f5efc15d9d0 2024-03-29T09:55:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.355+0000 7f5f01a00700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5edc06ac90 0x7f5edc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.356+0000 7f5f01a00700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5edc06ac90 0x7f5edc06d150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0014040 tx=0x7f5ef0008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:04.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.359+0000 7f5ef9600700 1 -- 172.21.15.12:0/3170256763 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ef4054470 con 0x7f5efc15d9d0 2024-03-29T09:55:04.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.574+0000 7f5f04400700 1 -- 172.21.15.12:0/3170256763 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5efc067720 con 0x7f5efc15d9d0 2024-03-29T09:55:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.574+0000 7f5ef9600700 1 -- 172.21.15.12:0/3170256763 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ef4057a90 con 0x7f5efc15d9d0 2024-03-29T09:55:04.574 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:55:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 -- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5edc06ac90 msgr2=0x7f5edc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5edc06ac90 0x7f5edc06d150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0014040 tx=0x7f5ef0008030 comp rx=0 tx=0).stop 2024-03-29T09:55:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 -- 172.21.15.12:0/3170256763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 msgr2=0x7f5efc159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 0x7f5efc159c50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400cc50 tx=0x7f5ef400cf60 comp rx=0 tx=0).stop 2024-03-29T09:55:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 -- 172.21.15.12:0/3170256763 shutdown_connections 2024-03-29T09:55:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5edc06ac90 0x7f5edc06d150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5efc15d9d0 0x7f5efc159c50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5efc0d16a0 0x7f5efc155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 --2- 172.21.15.12:0/3170256763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5efc0d2370 0x7f5efc0b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 -- 172.21.15.12:0/3170256763 >> 172.21.15.12:0/3170256763 conn(0x7f5efc08aff0 msgr2=0x7f5efc0b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 -- 172.21.15.12:0/3170256763 shutdown_connections 2024-03-29T09:55:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.577+0000 7f5f04400700 1 -- 172.21.15.12:0/3170256763 wait complete. 2024-03-29T09:55:04.594 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:04.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.793+0000 7f3319c00700 1 Processor -- start 2024-03-29T09:55:04.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.793+0000 7f3319c00700 1 -- start start 2024-03-29T09:55:04.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3319c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3314160f10 0x7f33140d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3319c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f33140cf230 0x7f33140d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3319c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3314161590 0x7f3314161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3319c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3314161f60 con 0x7f33140cf230 2024-03-29T09:55:04.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3319c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33140ceda0 con 0x7f3314161590 2024-03-29T09:55:04.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3319c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33140d23d0 con 0x7f3314160f10 2024-03-29T09:55:04.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3312000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f33140cf230 0x7f33140d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3312000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f33140cf230 0x7f33140d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37774/0 (socket says 172.21.15.12:37774) 2024-03-29T09:55:04.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3312000700 1 -- 172.21.15.12:0/616850035 learned_addr learned my addr 172.21.15.12:0/616850035 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3313400700 1 --2- 172.21.15.12:0/616850035 >> v2:172.21.15.159:3300/0 conn(0x7f3314161590 0x7f3314161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.794+0000 7f3312a00700 1 --2- 172.21.15.12:0/616850035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3314160f10 0x7f33140d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.795+0000 7f3312000700 1 -- 172.21.15.12:0/616850035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3314160f10 msgr2=0x7f33140d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.795+0000 7f3312000700 1 --2- 172.21.15.12:0/616850035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3314160f10 0x7f33140d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.795+0000 7f3312000700 1 -- 172.21.15.12:0/616850035 >> v2:172.21.15.159:3300/0 conn(0x7f3314161590 msgr2=0x7f3314161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.795+0000 7f3312000700 1 --2- 172.21.15.12:0/616850035 >> v2:172.21.15.159:3300/0 conn(0x7f3314161590 0x7f3314161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.795+0000 7f3312000700 1 -- 172.21.15.12:0/616850035 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33141612f0 con 0x7f33140cf230 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.795+0000 7f3312000700 1 --2- 172.21.15.12:0/616850035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 0x7f33140d1e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f33140cfad0 tx=0x7f3300009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf865b09bda7bba server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.795+0000 7f3311600700 1 -- 172.21.15.12:0/616850035 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330000b870 con 0x7f33140cf230 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.795+0000 7f3311600700 1 -- 172.21.15.12:0/616850035 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330000b9d0 con 0x7f33140cf230 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.795+0000 7f3311600700 1 -- 172.21.15.12:0/616850035 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330000bd60 con 0x7f33140cf230 2024-03-29T09:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.796+0000 7f3319c00700 1 -- 172.21.15.12:0/616850035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 msgr2=0x7f33140d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.796+0000 7f3319c00700 1 --2- 172.21.15.12:0/616850035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 0x7f33140d1e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f33140cfad0 tx=0x7f3300009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.796+0000 7f3319c00700 1 -- 172.21.15.12:0/616850035 shutdown_connections 2024-03-29T09:55:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.796+0000 7f3319c00700 1 --2- 172.21.15.12:0/616850035 >> v2:172.21.15.159:3300/0 conn(0x7f3314161590 0x7f3314161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.796+0000 7f3319c00700 1 --2- 172.21.15.12:0/616850035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3314160f10 0x7f33140d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.796+0000 7f3319c00700 1 --2- 172.21.15.12:0/616850035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 0x7f33140d1e90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.796+0000 7f3319c00700 1 -- 172.21.15.12:0/616850035 >> 172.21.15.12:0/616850035 conn(0x7f331408aff0 msgr2=0x7f33140d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.796+0000 7f3319c00700 1 -- 172.21.15.12:0/616850035 shutdown_connections 2024-03-29T09:55:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.796+0000 7f3319c00700 1 -- 172.21.15.12:0/616850035 wait complete. 2024-03-29T09:55:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.798+0000 7f3319c00700 1 Processor -- start 2024-03-29T09:55:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.798+0000 7f3319c00700 1 -- start start 2024-03-29T09:55:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3319c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 0x7f33141555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3319c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3314160f10 0x7f33141f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3319c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3314161590 0x7f33141f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3319c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331409e540 con 0x7f33140cf230 2024-03-29T09:55:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3319c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331409f5c0 con 0x7f3314160f10 2024-03-29T09:55:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3319c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33140b4820 con 0x7f3314161590 2024-03-29T09:55:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3312a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 0x7f33141555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3312a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 0x7f33141555f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37788/0 (socket says 172.21.15.12:37788) 2024-03-29T09:55:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3312a00700 1 -- 172.21.15.12:0/1378136090 learned_addr learned my addr 172.21.15.12:0/1378136090 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3312000700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3314160f10 0x7f33141f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.799+0000 7f3313400700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3314161590 0x7f33141f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f3312a00700 1 -- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3314161590 msgr2=0x7f33141f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f3312a00700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3314161590 0x7f33141f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f3312a00700 1 -- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3314160f10 msgr2=0x7f33141f6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f3312a00700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3314160f10 0x7f33141f6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f3312a00700 1 -- 172.21.15.12:0/1378136090 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3300009550 con 0x7f33140cf230 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f3312a00700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 0x7f33141555f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f330400d830 tx=0x7f330400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f330aa00700 1 -- 172.21.15.12:0/1378136090 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33040125f0 con 0x7f33140cf230 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f330aa00700 1 -- 172.21.15.12:0/1378136090 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3304012750 con 0x7f33140cf230 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f330aa00700 1 -- 172.21.15.12:0/1378136090 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304012b00 con 0x7f33140cf230 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f3319c00700 1 -- 172.21.15.12:0/1378136090 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331409e790 con 0x7f33140cf230 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.800+0000 7f3319c00700 1 -- 172.21.15.12:0/1378136090 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331409ef70 con 0x7f33140cf230 2024-03-29T09:55:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.801+0000 7f3319c00700 1 -- 172.21.15.12:0/1378136090 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3314067720 con 0x7f33140cf230 2024-03-29T09:55:04.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.802+0000 7f330aa00700 1 -- 172.21.15.12:0/1378136090 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3304012c60 con 0x7f33140cf230 2024-03-29T09:55:04.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.803+0000 7f330aa00700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32f406ac90 0x7f32f406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:04.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.803+0000 7f330aa00700 1 -- 172.21.15.12:0/1378136090 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33040893e0 con 0x7f33140cf230 2024-03-29T09:55:04.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.804+0000 7f3312000700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32f406ac90 0x7f32f406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:04.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.804+0000 7f3312000700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32f406ac90 0x7f32f406d150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3300014040 tx=0x7f3300008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:04.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:04.807+0000 7f330aa00700 1 -- 172.21.15.12:0/1378136090 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3304054520 con 0x7f33140cf230 2024-03-29T09:55:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.026+0000 7f3319c00700 1 -- 172.21.15.12:0/1378136090 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33140d0ad0 con 0x7f33140cf230 2024-03-29T09:55:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.028+0000 7f330aa00700 1 -- 172.21.15.12:0/1378136090 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f3304057b40 con 0x7f33140cf230 2024-03-29T09:55:05.027 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:05.027 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 -- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32f406ac90 msgr2=0x7f32f406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32f406ac90 0x7f32f406d150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3300014040 tx=0x7f3300008030 comp rx=0 tx=0).stop 2024-03-29T09:55:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 -- 172.21.15.12:0/1378136090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 msgr2=0x7f33141555f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 0x7f33141555f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f330400d830 tx=0x7f330400db40 comp rx=0 tx=0).stop 2024-03-29T09:55:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 -- 172.21.15.12:0/1378136090 shutdown_connections 2024-03-29T09:55:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32f406ac90 0x7f32f406d150 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3314161590 0x7f33141f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33140cf230 0x7f33141555f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 --2- 172.21.15.12:0/1378136090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3314160f10 0x7f33141f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 -- 172.21.15.12:0/1378136090 >> 172.21.15.12:0/1378136090 conn(0x7f331408aff0 msgr2=0x7f33140d1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 -- 172.21.15.12:0/1378136090 shutdown_connections 2024-03-29T09:55:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.030+0000 7f3319c00700 1 -- 172.21.15.12:0/1378136090 wait complete. 2024-03-29T09:55:05.047 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:55:05.048 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:55:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.248+0000 7fe24d200700 1 Processor -- start 2024-03-29T09:55:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.248+0000 7fe24d200700 1 -- start start 2024-03-29T09:55:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.249+0000 7fe24d200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe24815d9d0 0x7fe2480d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.249+0000 7fe24d200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2480d16a0 0x7fe2480d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.249+0000 7fe24d200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe248161420 con 0x7fe2480d2370 2024-03-29T09:55:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.249+0000 7fe24d200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2480d27e0 con 0x7fe2480d16a0 2024-03-29T09:55:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.249+0000 7fe24d200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe248161700 con 0x7fe24815d9d0 2024-03-29T09:55:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.250+0000 7fe245400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2480d16a0 0x7fe2480d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.250+0000 7fe245400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2480d16a0 0x7fe2480d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37790/0 (socket says 172.21.15.12:37790) 2024-03-29T09:55:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.250+0000 7fe245400700 1 -- 172.21.15.12:0/1329202875 learned_addr learned my addr 172.21.15.12:0/1329202875 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.250+0000 7fe245e00700 1 --2- 172.21.15.12:0/1329202875 >> v2:172.21.15.159:3300/0 conn(0x7fe24815d9d0 0x7fe2480d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.250+0000 7fe245400700 1 -- 172.21.15.12:0/1329202875 >> v2:172.21.15.159:3300/0 conn(0x7fe24815d9d0 msgr2=0x7fe2480d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.250+0000 7fe245400700 1 --2- 172.21.15.12:0/1329202875 >> v2:172.21.15.159:3300/0 conn(0x7fe24815d9d0 0x7fe2480d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.250+0000 7fe245400700 1 -- 172.21.15.12:0/1329202875 >> v1:172.21.15.159:6789/0 conn(0x7fe2480d2370 legacy=0x7fe24815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.250+0000 7fe245400700 1 -- 172.21.15.12:0/1329202875 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2481619e0 con 0x7fe2480d16a0 2024-03-29T09:55:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe245400700 1 --2- 172.21.15.12:0/1329202875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 0x7fe2480d1e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe2480d0ff0 tx=0x7fe234009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360d41966865b3bf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe244a00700 1 -- 172.21.15.12:0/1329202875 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234005e30 con 0x7fe2480d16a0 2024-03-29T09:55:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe244a00700 1 -- 172.21.15.12:0/1329202875 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe234004030 con 0x7fe2480d16a0 2024-03-29T09:55:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe244a00700 1 -- 172.21.15.12:0/1329202875 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2340043e0 con 0x7fe2480d16a0 2024-03-29T09:55:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe24d200700 1 -- 172.21.15.12:0/1329202875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 msgr2=0x7fe2480d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe24d200700 1 --2- 172.21.15.12:0/1329202875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 0x7fe2480d1e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe2480d0ff0 tx=0x7fe234009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe24d200700 1 -- 172.21.15.12:0/1329202875 shutdown_connections 2024-03-29T09:55:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe24d200700 1 --2- 172.21.15.12:0/1329202875 >> v2:172.21.15.159:3300/0 conn(0x7fe24815d9d0 0x7fe2480d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe24d200700 1 --2- 172.21.15.12:0/1329202875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 0x7fe2480d1e30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe24d200700 1 -- 172.21.15.12:0/1329202875 >> 172.21.15.12:0/1329202875 conn(0x7fe24808aff0 msgr2=0x7fe2480d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.251+0000 7fe24d200700 1 -- 172.21.15.12:0/1329202875 shutdown_connections 2024-03-29T09:55:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.252+0000 7fe24d200700 1 -- 172.21.15.12:0/1329202875 wait complete. 2024-03-29T09:55:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.253+0000 7fe24d200700 1 Processor -- start 2024-03-29T09:55:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.253+0000 7fe24d200700 1 -- start start 2024-03-29T09:55:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.254+0000 7fe24d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 0x7fe2480ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.254+0000 7fe24d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2480d2370 0x7fe2480b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.254+0000 7fe24d200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24815d9d0 0x7fe248159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.254+0000 7fe24d200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480b6f70 con 0x7fe2480d16a0 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.254+0000 7fe24d200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480b71a0 con 0x7fe24815d9d0 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.254+0000 7fe24d200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480b7480 con 0x7fe2480d2370 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.254+0000 7fe245e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 0x7fe2480ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.254+0000 7fe245e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 0x7fe2480ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37792/0 (socket says 172.21.15.12:37792) 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.254+0000 7fe245e00700 1 -- 172.21.15.12:0/152227081 learned_addr learned my addr 172.21.15.12:0/152227081 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.255+0000 7fe245400700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2480d2370 0x7fe2480b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.255+0000 7fe246800700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24815d9d0 0x7fe248159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.255+0000 7fe245e00700 1 -- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2480d2370 msgr2=0x7fe2480b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.255+0000 7fe245e00700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2480d2370 0x7fe2480b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.255+0000 7fe245e00700 1 -- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24815d9d0 msgr2=0x7fe248159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.255+0000 7fe245e00700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24815d9d0 0x7fe248159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.255+0000 7fe245e00700 1 -- 172.21.15.12:0/152227081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe234009500 con 0x7fe2480d16a0 2024-03-29T09:55:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.255+0000 7fe246800700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24815d9d0 0x7fe248159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.255+0000 7fe245e00700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 0x7fe2480ccb60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe23800e870 tx=0x7fe23800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.255+0000 7fe23e000700 1 -- 172.21.15.12:0/152227081 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe238009590 con 0x7fe2480d16a0 2024-03-29T09:55:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.256+0000 7fe23e000700 1 -- 172.21.15.12:0/152227081 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe238013680 con 0x7fe2480d16a0 2024-03-29T09:55:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.256+0000 7fe23e000700 1 -- 172.21.15.12:0/152227081 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2380139f0 con 0x7fe2480d16a0 2024-03-29T09:55:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.256+0000 7fe24d200700 1 -- 172.21.15.12:0/152227081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2480b7760 con 0x7fe2480d16a0 2024-03-29T09:55:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.256+0000 7fe24d200700 1 -- 172.21.15.12:0/152227081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2480b7d20 con 0x7fe2480d16a0 2024-03-29T09:55:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.257+0000 7fe24d200700 1 -- 172.21.15.12:0/152227081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2480d1ca0 con 0x7fe2480d16a0 2024-03-29T09:55:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.258+0000 7fe23e000700 1 -- 172.21.15.12:0/152227081 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe238013b50 con 0x7fe2480d16a0 2024-03-29T09:55:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.259+0000 7fe23e000700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe22806ac90 0x7fe22806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.259+0000 7fe23e000700 1 -- 172.21.15.12:0/152227081 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe238089250 con 0x7fe2480d16a0 2024-03-29T09:55:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.259+0000 7fe245400700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe22806ac90 0x7fe22806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.260+0000 7fe245400700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe22806ac90 0x7fe22806d150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe2480d1ae0 tx=0x7fe234008070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.263+0000 7fe23e000700 1 -- 172.21.15.12:0/152227081 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe238054430 con 0x7fe2480d16a0 2024-03-29T09:55:05.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.477+0000 7fe24d200700 1 -- 172.21.15.12:0/152227081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe248067720 con 0x7fe2480d16a0 2024-03-29T09:55:05.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.477+0000 7fe23e000700 1 -- 172.21.15.12:0/152227081 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe238057a50 con 0x7fe2480d16a0 2024-03-29T09:55:05.477 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 -- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe22806ac90 msgr2=0x7fe22806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe22806ac90 0x7fe22806d150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe2480d1ae0 tx=0x7fe234008070 comp rx=0 tx=0).stop 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 -- 172.21.15.12:0/152227081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 msgr2=0x7fe2480ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 0x7fe2480ccb60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe23800e870 tx=0x7fe23800eb80 comp rx=0 tx=0).stop 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 -- 172.21.15.12:0/152227081 shutdown_connections 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe22806ac90 0x7fe22806d150 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24815d9d0 0x7fe248159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2480d16a0 0x7fe2480ccb60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 --2- 172.21.15.12:0/152227081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2480d2370 0x7fe2480b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 -- 172.21.15.12:0/152227081 >> 172.21.15.12:0/152227081 conn(0x7fe24808aff0 msgr2=0x7fe2480b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 -- 172.21.15.12:0/152227081 shutdown_connections 2024-03-29T09:55:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.480+0000 7fe24d200700 1 -- 172.21.15.12:0/152227081 wait complete. 2024-03-29T09:55:05.498 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:55:05.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.699+0000 7fdcd0400700 1 Processor -- start 2024-03-29T09:55:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.699+0000 7fdcd0400700 1 -- start start 2024-03-29T09:55:05.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.700+0000 7fdcd0400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcc815d9d0 0x7fdcc80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.700+0000 7fdcd0400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc8161420 con 0x7fdcc80d16a0 2024-03-29T09:55:05.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.700+0000 7fdcd0400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc8161700 con 0x7fdcc80d2540 2024-03-29T09:55:05.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.700+0000 7fdcd0400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc815edc0 con 0x7fdcc815d9d0 2024-03-29T09:55:05.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.700+0000 7fdccda00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcc815d9d0 0x7fdcc80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.700+0000 7fdccd000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdcc80d16a0 0x7fdcc80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33636/0 (socket says 172.21.15.12:33636) 2024-03-29T09:55:05.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.700+0000 7fdccd000700 1 -- 172.21.15.12:0/1579161714 learned_addr learned my addr 172.21.15.12:0/1579161714 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:05.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.701+0000 7fdccda00700 1 -- 172.21.15.12:0/1579161714 >> v1:172.21.15.159:6789/0 conn(0x7fdcc80d2540 legacy=0x7fdcc80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:05.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.701+0000 7fdcc7e00700 1 -- 172.21.15.12:0/1579161714 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569663272 0 0) 0x7fdcc8161420 con 0x7fdcc80d16a0 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.701+0000 7fdccda00700 1 -- 172.21.15.12:0/1579161714 >> v1:172.21.15.12:6789/0 conn(0x7fdcc80d16a0 legacy=0x7fdcc80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.701+0000 7fdccda00700 1 -- 172.21.15.12:0/1579161714 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc81619e0 con 0x7fdcc815d9d0 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.701+0000 7fdccda00700 1 --2- 172.21.15.12:0/1579161714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc815d9d0 0x7fdcc80d3c30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009840 tx=0x7fdcbc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3850e703add277a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.701+0000 7fdcc7e00700 1 -- 172.21.15.12:0/1579161714 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc011830 con 0x7fdcc815d9d0 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.701+0000 7fdcc7e00700 1 -- 172.21.15.12:0/1579161714 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcbc011990 con 0x7fdcc815d9d0 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.701+0000 7fdcc7e00700 1 -- 172.21.15.12:0/1579161714 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc011d20 con 0x7fdcc815d9d0 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.702+0000 7fdcd0400700 1 -- 172.21.15.12:0/1579161714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc815d9d0 msgr2=0x7fdcc80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.702+0000 7fdcd0400700 1 --2- 172.21.15.12:0/1579161714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc815d9d0 0x7fdcc80d3c30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009840 tx=0x7fdcbc009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.702+0000 7fdcd0400700 1 -- 172.21.15.12:0/1579161714 shutdown_connections 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.702+0000 7fdcd0400700 1 --2- 172.21.15.12:0/1579161714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc815d9d0 0x7fdcc80d3c30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.702+0000 7fdcd0400700 1 -- 172.21.15.12:0/1579161714 >> 172.21.15.12:0/1579161714 conn(0x7fdcc808aff0 msgr2=0x7fdcc80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:05.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.702+0000 7fdcd0400700 1 -- 172.21.15.12:0/1579161714 shutdown_connections 2024-03-29T09:55:05.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.702+0000 7fdcd0400700 1 -- 172.21.15.12:0/1579161714 wait complete. 2024-03-29T09:55:05.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.704+0000 7fdcd0400700 1 Processor -- start 2024-03-29T09:55:05.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.704+0000 7fdcd0400700 1 -- start start 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdcd0400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc80d16a0 0x7fdcc80b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdcd0400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc80d2540 0x7fdcc80b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdcd0400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc815d9d0 0x7fdcc815c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdcd0400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80b6f40 con 0x7fdcc80d2540 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdcd0400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80b7cb0 con 0x7fdcc815d9d0 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdcd0400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80b7130 con 0x7fdcc80d16a0 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdccd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc80d2540 0x7fdcc80b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdccd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc80d2540 0x7fdcc80b6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37810/0 (socket says 172.21.15.12:37810) 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdccd000700 1 -- 172.21.15.12:0/844760735 learned_addr learned my addr 172.21.15.12:0/844760735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdccda00700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc80d16a0 0x7fdcc80b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.705+0000 7fdcce400700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc815d9d0 0x7fdcc815c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdccd000700 1 -- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc80d16a0 msgr2=0x7fdcc80b88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdccd000700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc80d16a0 0x7fdcc80b88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdccd000700 1 -- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc815d9d0 msgr2=0x7fdcc815c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdccd000700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc815d9d0 0x7fdcc815c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdccd000700 1 -- 172.21.15.12:0/844760735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcbc0094a0 con 0x7fdcc80d2540 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdccd000700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc80d2540 0x7fdcc80b6a00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400c780 tx=0x7fdcb400ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdcc5600700 1 -- 172.21.15.12:0/844760735 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb40074e0 con 0x7fdcc80d2540 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdcc5600700 1 -- 172.21.15.12:0/844760735 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb4007640 con 0x7fdcc80d2540 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdcc5600700 1 -- 172.21.15.12:0/844760735 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4005a70 con 0x7fdcc80d2540 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdcd0400700 1 -- 172.21.15.12:0/844760735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc80b7410 con 0x7fdcc80d2540 2024-03-29T09:55:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.706+0000 7fdcd0400700 1 -- 172.21.15.12:0/844760735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc80b79d0 con 0x7fdcc80d2540 2024-03-29T09:55:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.707+0000 7fdcd0400700 1 -- 172.21.15.12:0/844760735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc80d1ca0 con 0x7fdcc80d2540 2024-03-29T09:55:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.708+0000 7fdcc5600700 1 -- 172.21.15.12:0/844760735 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdcb4005bd0 con 0x7fdcc80d2540 2024-03-29T09:55:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.709+0000 7fdcc5600700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdca806ac90 0x7fdca806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.709+0000 7fdcc5600700 1 -- 172.21.15.12:0/844760735 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdcb4088680 con 0x7fdcc80d2540 2024-03-29T09:55:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.709+0000 7fdccda00700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdca806ac90 0x7fdca806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.710+0000 7fdccda00700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdca806ac90 0x7fdca806d150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc0090b0 tx=0x7fdcbc01c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:05.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.714+0000 7fdcc5600700 1 -- 172.21.15.12:0/844760735 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdcb4053900 con 0x7fdcc80d2540 2024-03-29T09:55:05.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.928+0000 7fdcd0400700 1 -- 172.21.15.12:0/844760735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdcc81f39f0 con 0x7fdcc80d2540 2024-03-29T09:55:05.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.928+0000 7fdcc5600700 1 -- 172.21.15.12:0/844760735 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdcc81f39f0 con 0x7fdcc80d2540 2024-03-29T09:55:05.928 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:55:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 -- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdca806ac90 msgr2=0x7fdca806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdca806ac90 0x7fdca806d150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc0090b0 tx=0x7fdcbc01c040 comp rx=0 tx=0).stop 2024-03-29T09:55:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 -- 172.21.15.12:0/844760735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc80d2540 msgr2=0x7fdcc80b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc80d2540 0x7fdcc80b6a00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400c780 tx=0x7fdcb400ca90 comp rx=0 tx=0).stop 2024-03-29T09:55:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 -- 172.21.15.12:0/844760735 shutdown_connections 2024-03-29T09:55:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdca806ac90 0x7fdca806d150 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc815d9d0 0x7fdcc815c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc80d16a0 0x7fdcc80b88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 --2- 172.21.15.12:0/844760735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcc80d2540 0x7fdcc80b6a00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 -- 172.21.15.12:0/844760735 >> 172.21.15.12:0/844760735 conn(0x7fdcc808aff0 msgr2=0x7fdcc815e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:05.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 -- 172.21.15.12:0/844760735 shutdown_connections 2024-03-29T09:55:05.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:05.931+0000 7fdcd0400700 1 -- 172.21.15.12:0/844760735 wait complete. 2024-03-29T09:55:05.949 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:55:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.151+0000 7f0ee6e00700 1 Processor -- start 2024-03-29T09:55:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.151+0000 7f0ee6e00700 1 -- start start 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.152+0000 7f0ee6e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0ee015d9d0 0x7f0ee00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.152+0000 7f0ee6e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ee00d16a0 0x7f0ee00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.152+0000 7f0ee6e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ee00d2370 0x7f0ee015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.152+0000 7f0ee6e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee00d2820 con 0x7f0ee00d2370 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.152+0000 7f0ee6e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee0161420 con 0x7f0ee015d9d0 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.152+0000 7f0ee6e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee0161700 con 0x7f0ee00d16a0 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0ee4e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ee00d2370 0x7f0ee015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0ee4e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ee00d2370 0x7f0ee015ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37824/0 (socket says 172.21.15.12:37824) 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0ee4e00700 1 -- 172.21.15.12:0/4155542478 learned_addr learned my addr 172.21.15.12:0/4155542478 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0edfe00700 1 --2- 172.21.15.12:0/4155542478 >> v2:172.21.15.159:3300/0 conn(0x7f0ee015d9d0 0x7f0ee00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0edf400700 1 --2- 172.21.15.12:0/4155542478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ee00d16a0 0x7f0ee00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0ee4e00700 1 -- 172.21.15.12:0/4155542478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ee00d16a0 msgr2=0x7f0ee00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0ee4e00700 1 --2- 172.21.15.12:0/4155542478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ee00d16a0 0x7f0ee00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0ee4e00700 1 -- 172.21.15.12:0/4155542478 >> v2:172.21.15.159:3300/0 conn(0x7f0ee015d9d0 msgr2=0x7f0ee00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:06.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0ee4e00700 1 --2- 172.21.15.12:0/4155542478 >> v2:172.21.15.159:3300/0 conn(0x7f0ee015d9d0 0x7f0ee00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0ee4e00700 1 -- 172.21.15.12:0/4155542478 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee01619e0 con 0x7f0ee00d2370 2024-03-29T09:55:06.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.153+0000 7f0ee4e00700 1 --2- 172.21.15.12:0/4155542478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d2370 0x7f0ee015ddb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00d0ff0 tx=0x7f0ed800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca7bf66ff716498 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0edea00700 1 -- 172.21.15.12:0/4155542478 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed800e070 con 0x7f0ee00d2370 2024-03-29T09:55:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0edea00700 1 -- 172.21.15.12:0/4155542478 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed800b910 con 0x7f0ee00d2370 2024-03-29T09:55:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0edea00700 1 -- 172.21.15.12:0/4155542478 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed800bcc0 con 0x7f0ee00d2370 2024-03-29T09:55:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4155542478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d2370 msgr2=0x7f0ee015ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0ee6e00700 1 --2- 172.21.15.12:0/4155542478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d2370 0x7f0ee015ddb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00d0ff0 tx=0x7f0ed800b330 comp rx=0 tx=0).stop 2024-03-29T09:55:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4155542478 shutdown_connections 2024-03-29T09:55:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0ee6e00700 1 --2- 172.21.15.12:0/4155542478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d2370 0x7f0ee015ddb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0ee6e00700 1 --2- 172.21.15.12:0/4155542478 >> v2:172.21.15.159:3300/0 conn(0x7f0ee015d9d0 0x7f0ee00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0ee6e00700 1 --2- 172.21.15.12:0/4155542478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ee00d16a0 0x7f0ee00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4155542478 >> 172.21.15.12:0/4155542478 conn(0x7f0ee008aff0 msgr2=0x7f0ee00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:06.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4155542478 shutdown_connections 2024-03-29T09:55:06.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.154+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4155542478 wait complete. 2024-03-29T09:55:06.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.156+0000 7f0ee6e00700 1 Processor -- start 2024-03-29T09:55:06.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.156+0000 7f0ee6e00700 1 -- start start 2024-03-29T09:55:06.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.157+0000 7f0ee6e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d16a0 0x7f0ee00b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.157+0000 7f0ee6e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ee00d2370 0x7f0ee01f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.157+0000 7f0ee6e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ee015d9d0 0x7f0ee00cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.157+0000 7f0ee6e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee00b8140 con 0x7f0ee00d16a0 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.157+0000 7f0ee6e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee01f3ca0 con 0x7f0ee015d9d0 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.157+0000 7f0ee6e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee01f3f80 con 0x7f0ee00d2370 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.157+0000 7f0edfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d16a0 0x7f0ee00b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.157+0000 7f0edfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d16a0 0x7f0ee00b7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37840/0 (socket says 172.21.15.12:37840) 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.157+0000 7f0edfe00700 1 -- 172.21.15.12:0/4116882735 learned_addr learned my addr 172.21.15.12:0/4116882735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0edf400700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ee00d2370 0x7f0ee01f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0ee4e00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ee015d9d0 0x7f0ee00cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0edfe00700 1 -- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ee00d2370 msgr2=0x7f0ee01f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0edfe00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ee00d2370 0x7f0ee01f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0edfe00700 1 -- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ee015d9d0 msgr2=0x7f0ee00cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:06.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0edfe00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ee015d9d0 0x7f0ee00cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0edfe00700 1 -- 172.21.15.12:0/4116882735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed8009b30 con 0x7f0ee00d16a0 2024-03-29T09:55:06.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0edfe00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d16a0 0x7f0ee00b7c00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000e870 tx=0x7f0ed000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:06.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0ed7e00700 1 -- 172.21.15.12:0/4116882735 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0009590 con 0x7f0ee00d16a0 2024-03-29T09:55:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0ed7e00700 1 -- 172.21.15.12:0/4116882735 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed0013680 con 0x7f0ee00d16a0 2024-03-29T09:55:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0ed7e00700 1 -- 172.21.15.12:0/4116882735 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed00139f0 con 0x7f0ee00d16a0 2024-03-29T09:55:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.158+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4116882735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee00b61a0 con 0x7f0ee00d16a0 2024-03-29T09:55:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.159+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4116882735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee00cd230 con 0x7f0ee00d16a0 2024-03-29T09:55:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.160+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4116882735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee00b67f0 con 0x7f0ee00d16a0 2024-03-29T09:55:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.161+0000 7f0ed7e00700 1 -- 172.21.15.12:0/4116882735 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0ed0013b50 con 0x7f0ee00d16a0 2024-03-29T09:55:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.161+0000 7f0ed7e00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ec006ac90 0x7f0ec006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.162+0000 7f0ed7e00700 1 -- 172.21.15.12:0/4116882735 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ed00894b0 con 0x7f0ee00d16a0 2024-03-29T09:55:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.162+0000 7f0edf400700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ec006ac90 0x7f0ec006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.163+0000 7f0edf400700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ec006ac90 0x7f0ec006d150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0ee01f0dd0 tx=0x7f0ecc0091c0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:06.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.166+0000 7f0ed7e00700 1 -- 172.21.15.12:0/4116882735 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ed0054720 con 0x7f0ee00d16a0 2024-03-29T09:55:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.379+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4116882735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0ee00d1ca0 con 0x7f0ee00d16a0 2024-03-29T09:55:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.380+0000 7f0ed7e00700 1 -- 172.21.15.12:0/4116882735 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0ed0057d40 con 0x7f0ee00d16a0 2024-03-29T09:55:06.379 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:55:06.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ec006ac90 msgr2=0x7f0ec006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ec006ac90 0x7f0ec006d150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0ee01f0dd0 tx=0x7f0ecc0091c0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4116882735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d16a0 msgr2=0x7f0ee00b7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d16a0 0x7f0ee00b7c00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000e870 tx=0x7f0ed000eb80 comp rx=0 tx=0).stop 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4116882735 shutdown_connections 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ec006ac90 0x7f0ec006d150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ee015d9d0 0x7f0ee00cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ee00d16a0 0x7f0ee00b7c00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 --2- 172.21.15.12:0/4116882735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ee00d2370 0x7f0ee01f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4116882735 >> 172.21.15.12:0/4116882735 conn(0x7f0ee008aff0 msgr2=0x7f0ee00ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4116882735 shutdown_connections 2024-03-29T09:55:06.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.382+0000 7f0ee6e00700 1 -- 172.21.15.12:0/4116882735 wait complete. 2024-03-29T09:55:06.427 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2024-03-29T09:55:06.428 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T09:55:06.428 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-03-29T09:55:06.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.627+0000 7f78df000700 1 Processor -- start 2024-03-29T09:55:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.627+0000 7f78df000700 1 -- start start 2024-03-29T09:55:06.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.628+0000 7f78df000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f78d815da50 0x7f78d80ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.628+0000 7f78df000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d80cec70 0x7f78d80d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.628+0000 7f78df000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d80d24e0 con 0x7f78d80d0b20 2024-03-29T09:55:06.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.628+0000 7f78df000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d80b95d0 con 0x7f78d815da50 2024-03-29T09:55:06.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.628+0000 7f78df000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d815de30 con 0x7f78d80cec70 2024-03-29T09:55:06.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.628+0000 7f78dd000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f78d80d0b20 0x7f78d80b52e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33638/0 (socket says 172.21.15.12:33638) 2024-03-29T09:55:06.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.628+0000 7f78dd000700 1 -- 172.21.15.12:0/729922713 learned_addr learned my addr 172.21.15.12:0/729922713 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:06.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.629+0000 7f78d7400700 1 --2- 172.21.15.12:0/729922713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d80cec70 0x7f78d80d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.629+0000 7f78d7e00700 1 --2- 172.21.15.12:0/729922713 >> v2:172.21.15.159:3300/0 conn(0x7f78d815da50 0x7f78d80ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.629+0000 7f78d6a00700 1 -- 172.21.15.12:0/729922713 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605262025 0 0) 0x7f78d80d24e0 con 0x7f78d80d0b20 2024-03-29T09:55:06.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.629+0000 7f78d6a00700 1 -- 172.21.15.12:0/729922713 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78bc0035b0 con 0x7f78d80d0b20 2024-03-29T09:55:06.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.629+0000 7f78d7400700 1 -- 172.21.15.12:0/729922713 >> v2:172.21.15.159:3300/0 conn(0x7f78d815da50 msgr2=0x7f78d80ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:06.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.629+0000 7f78d7400700 1 --2- 172.21.15.12:0/729922713 >> v2:172.21.15.159:3300/0 conn(0x7f78d815da50 0x7f78d80ce730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.629+0000 7f78d7400700 1 -- 172.21.15.12:0/729922713 >> v1:172.21.15.12:6789/0 conn(0x7f78d80d0b20 legacy=0x7f78d80b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:06.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.629+0000 7f78d7400700 1 -- 172.21.15.12:0/729922713 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80d27c0 con 0x7f78d80cec70 2024-03-29T09:55:06.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.629+0000 7f78d7400700 1 --2- 172.21.15.12:0/729922713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d80cec70 0x7f78d80d05e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009840 tx=0x7f78c4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59ecc5b9a7a0b89b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78d6a00700 1 -- 172.21.15.12:0/729922713 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4011830 con 0x7f78d80cec70 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78d6a00700 1 -- 172.21.15.12:0/729922713 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c4011990 con 0x7f78d80cec70 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78d6a00700 1 -- 172.21.15.12:0/729922713 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4011d00 con 0x7f78d80cec70 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78df000700 1 -- 172.21.15.12:0/729922713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d80cec70 msgr2=0x7f78d80d05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78df000700 1 --2- 172.21.15.12:0/729922713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d80cec70 0x7f78d80d05e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009840 tx=0x7f78c4009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78df000700 1 -- 172.21.15.12:0/729922713 shutdown_connections 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78df000700 1 --2- 172.21.15.12:0/729922713 >> v2:172.21.15.159:3300/0 conn(0x7f78d815da50 0x7f78d80ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78df000700 1 --2- 172.21.15.12:0/729922713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d80cec70 0x7f78d80d05e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78df000700 1 -- 172.21.15.12:0/729922713 >> 172.21.15.12:0/729922713 conn(0x7f78d80666a0 msgr2=0x7f78d8066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78df000700 1 -- 172.21.15.12:0/729922713 shutdown_connections 2024-03-29T09:55:06.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.630+0000 7f78df000700 1 -- 172.21.15.12:0/729922713 wait complete. 2024-03-29T09:55:06.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.632+0000 7f78df000700 1 Processor -- start 2024-03-29T09:55:06.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.632+0000 7f78df000700 1 -- start start 2024-03-29T09:55:06.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.633+0000 7f78df000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d80cec70 0x7f78d80c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.633+0000 7f78df000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d80d0b20 0x7f78d81e7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.633+0000 7f78df000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d815da50 0x7f78d80c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.633+0000 7f78df000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d81e75f0 con 0x7f78d80d0b20 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.633+0000 7f78df000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d81e6740 con 0x7f78d80cec70 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.633+0000 7f78df000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d81e69e0 con 0x7f78d815da50 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78d7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d80d0b20 0x7f78d81e7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78d7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d80d0b20 0x7f78d81e7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37846/0 (socket says 172.21.15.12:37846) 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78d7400700 1 -- 172.21.15.12:0/954772337 learned_addr learned my addr 172.21.15.12:0/954772337 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78d7e00700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d80cec70 0x7f78d80c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78dd000700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d815da50 0x7f78d80c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78d7400700 1 -- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d815da50 msgr2=0x7f78d80c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78d7400700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d815da50 0x7f78d80c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78d7400700 1 -- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d80cec70 msgr2=0x7f78d80c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:06.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78d7400700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d80cec70 0x7f78d80c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:06.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78d7400700 1 -- 172.21.15.12:0/954772337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c40094a0 con 0x7f78d80d0b20 2024-03-29T09:55:06.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.634+0000 7f78d7400700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d80d0b20 0x7f78d81e7a80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f78c40090b0 tx=0x7f78c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.635+0000 7f78cfe00700 1 -- 172.21.15.12:0/954772337 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4004550 con 0x7f78d80d0b20 2024-03-29T09:55:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.635+0000 7f78cfe00700 1 -- 172.21.15.12:0/954772337 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c401f070 con 0x7f78d80d0b20 2024-03-29T09:55:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.635+0000 7f78cfe00700 1 -- 172.21.15.12:0/954772337 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c40038c0 con 0x7f78d80d0b20 2024-03-29T09:55:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.635+0000 7f78df000700 1 -- 172.21.15.12:0/954772337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d81e6cc0 con 0x7f78d80d0b20 2024-03-29T09:55:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.635+0000 7f78df000700 1 -- 172.21.15.12:0/954772337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d81e7280 con 0x7f78d80d0b20 2024-03-29T09:55:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.636+0000 7f78cd600700 1 -- 172.21.15.12:0/954772337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d81e7760 con 0x7f78d80d0b20 2024-03-29T09:55:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.637+0000 7f78cfe00700 1 -- 172.21.15.12:0/954772337 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f78c4003a20 con 0x7f78d80d0b20 2024-03-29T09:55:06.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.638+0000 7f78cfe00700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f78b806ac90 0x7f78b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:06.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.638+0000 7f78cfe00700 1 -- 172.21.15.12:0/954772337 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78c4021070 con 0x7f78d80d0b20 2024-03-29T09:55:06.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.638+0000 7f78d7e00700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f78b806ac90 0x7f78b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:06.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.639+0000 7f78d7e00700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f78b806ac90 0x7f78b806d150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f78d8154780 tx=0x7f78c8009210 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:06.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.642+0000 7f78cfe00700 1 -- 172.21.15.12:0/954772337 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f78c4055c40 con 0x7f78d80d0b20 2024-03-29T09:55:06.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:06.949+0000 7f78cd600700 1 -- 172.21.15.12:0/954772337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f78d8067720 con 0x7f78d80d0b20 2024-03-29T09:55:07.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.025+0000 7f78cfe00700 1 -- 172.21.15.12:0/954772337 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v55) v1 ==== 161+0+0 (secure 0 0 0) 0x7f78c402b0e0 con 0x7f78d80d0b20 2024-03-29T09:55:07.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 -- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f78b806ac90 msgr2=0x7f78b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:07.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f78b806ac90 0x7f78b806d150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f78d8154780 tx=0x7f78c8009210 comp rx=0 tx=0).stop 2024-03-29T09:55:07.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 -- 172.21.15.12:0/954772337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d80d0b20 msgr2=0x7f78d81e7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:07.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d80d0b20 0x7f78d81e7a80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f78c40090b0 tx=0x7f78c4004750 comp rx=0 tx=0).stop 2024-03-29T09:55:07.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 -- 172.21.15.12:0/954772337 shutdown_connections 2024-03-29T09:55:07.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f78b806ac90 0x7f78b806d150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d815da50 0x7f78d80c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d80cec70 0x7f78d80c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 --2- 172.21.15.12:0/954772337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d80d0b20 0x7f78d81e7a80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 -- 172.21.15.12:0/954772337 >> 172.21.15.12:0/954772337 conn(0x7f78d80666a0 msgr2=0x7f78d8154970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:07.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 -- 172.21.15.12:0/954772337 shutdown_connections 2024-03-29T09:55:07.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.027+0000 7f78cd600700 1 -- 172.21.15.12:0/954772337 wait complete. 2024-03-29T09:55:07.045 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T09:55:07.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.245+0000 7f5ce5a00700 1 Processor -- start 2024-03-29T09:55:07.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.245+0000 7f5ce5a00700 1 -- start start 2024-03-29T09:55:07.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.246+0000 7f5ce5a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 0x7f5ce00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:07.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.246+0000 7f5ce5a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5ce00d2540 0x7f5ce00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:07.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.246+0000 7f5ce5a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce0161420 con 0x7f5ce00d16a0 2024-03-29T09:55:07.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.246+0000 7f5ce5a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce015edc0 con 0x7f5ce00d2540 2024-03-29T09:55:07.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.246+0000 7f5ce5a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce0161700 con 0x7f5ce015d9d0 2024-03-29T09:55:07.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cdea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 0x7f5ce00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:07.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cdf400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5ce00d2540 0x7f5ce00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:07.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cdf400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5ce00d2540 0x7f5ce00ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:59598/0 (socket says 172.21.15.12:59598) 2024-03-29T09:55:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cdea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 0x7f5ce00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:40510/0 (socket says 172.21.15.12:40510) 2024-03-29T09:55:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cde000700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f5ce00d16a0 0x7f5ce00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:39828/0 (socket says 172.21.15.12:39828) 2024-03-29T09:55:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cdf400700 1 -- 172.21.15.12:0/169980424 learned_addr learned my addr 172.21.15.12:0/169980424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cdea00700 1 -- 172.21.15.12:0/169980424 >> v2:172.21.15.159:3300/0 conn(0x7f5ce00d2540 msgr2=0x7f5ce00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cdea00700 1 --2- 172.21.15.12:0/169980424 >> v2:172.21.15.159:3300/0 conn(0x7f5ce00d2540 0x7f5ce00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cdea00700 1 -- 172.21.15.12:0/169980424 >> v1:172.21.15.159:6789/0 conn(0x7f5ce00d16a0 legacy=0x7f5ce00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cdd600700 1 -- 172.21.15.12:0/169980424 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272427577 0 0) 0x7f5ce0161420 con 0x7f5ce00d16a0 2024-03-29T09:55:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.247+0000 7f5cdea00700 1 -- 172.21.15.12:0/169980424 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce01619e0 con 0x7f5ce015d9d0 2024-03-29T09:55:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.248+0000 7f5cdea00700 1 --2- 172.21.15.12:0/169980424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 0x7f5ce00d3c30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009870 tx=0x7f5cd0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e91debffb270482 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.248+0000 7f5cdd600700 1 -- 172.21.15.12:0/169980424 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0011830 con 0x7f5ce015d9d0 2024-03-29T09:55:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.248+0000 7f5cdd600700 1 -- 172.21.15.12:0/169980424 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd0011990 con 0x7f5ce015d9d0 2024-03-29T09:55:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.248+0000 7f5cdd600700 1 -- 172.21.15.12:0/169980424 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0011d20 con 0x7f5ce015d9d0 2024-03-29T09:55:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.249+0000 7f5ce5a00700 1 -- 172.21.15.12:0/169980424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 msgr2=0x7f5ce00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.249+0000 7f5ce5a00700 1 --2- 172.21.15.12:0/169980424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 0x7f5ce00d3c30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009870 tx=0x7f5cd0009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.249+0000 7f5ce5a00700 1 -- 172.21.15.12:0/169980424 shutdown_connections 2024-03-29T09:55:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.249+0000 7f5ce5a00700 1 --2- 172.21.15.12:0/169980424 >> v2:172.21.15.159:3300/0 conn(0x7f5ce00d2540 0x7f5ce00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.249+0000 7f5ce5a00700 1 --2- 172.21.15.12:0/169980424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 0x7f5ce00d3c30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.249+0000 7f5ce5a00700 1 -- 172.21.15.12:0/169980424 >> 172.21.15.12:0/169980424 conn(0x7f5ce008aff0 msgr2=0x7f5ce00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.249+0000 7f5ce5a00700 1 -- 172.21.15.12:0/169980424 shutdown_connections 2024-03-29T09:55:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.249+0000 7f5ce5a00700 1 -- 172.21.15.12:0/169980424 wait complete. 2024-03-29T09:55:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.250+0000 7f5ce5a00700 1 Processor -- start 2024-03-29T09:55:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.251+0000 7f5ce5a00700 1 -- start start 2024-03-29T09:55:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.251+0000 7f5ce5a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce00d16a0 0x7f5ce00a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.251+0000 7f5ce5a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ce00d2540 0x7f5ce00b7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.251+0000 7f5ce5a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 0x7f5ce00b75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.251+0000 7f5ce5a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce009d3f0 con 0x7f5ce00d16a0 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.251+0000 7f5ce5a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce00b8480 con 0x7f5ce00d2540 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.251+0000 7f5ce5a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce00b7be0 con 0x7f5ce015d9d0 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cdea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce00d16a0 0x7f5ce00a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cdea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce00d16a0 0x7f5ce00a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37848/0 (socket says 172.21.15.12:37848) 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cdea00700 1 -- 172.21.15.12:0/3145835593 learned_addr learned my addr 172.21.15.12:0/3145835593 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cde000700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ce00d2540 0x7f5ce00b7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cdf400700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 0x7f5ce00b75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cdea00700 1 -- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 msgr2=0x7f5ce00b75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cdea00700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 0x7f5ce00b75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cdea00700 1 -- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ce00d2540 msgr2=0x7f5ce00b7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cdea00700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ce00d2540 0x7f5ce00b7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cdea00700 1 -- 172.21.15.12:0/3145835593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd00094a0 con 0x7f5ce00d16a0 2024-03-29T09:55:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cde000700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ce00d2540 0x7f5ce00b7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.252+0000 7f5cdea00700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce00d16a0 0x7f5ce00a0a50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0014040 tx=0x7f5cd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.253+0000 7f5cd6a00700 1 -- 172.21.15.12:0/3145835593 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd00038a0 con 0x7f5ce00d16a0 2024-03-29T09:55:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.253+0000 7f5cd6a00700 1 -- 172.21.15.12:0/3145835593 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd001e070 con 0x7f5ce00d16a0 2024-03-29T09:55:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.253+0000 7f5cd6a00700 1 -- 172.21.15.12:0/3145835593 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0003c60 con 0x7f5ce00d16a0 2024-03-29T09:55:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.253+0000 7f5ce5a00700 1 -- 172.21.15.12:0/3145835593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce00b7ec0 con 0x7f5ce00d16a0 2024-03-29T09:55:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.253+0000 7f5ce5a00700 1 -- 172.21.15.12:0/3145835593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce01f3900 con 0x7f5ce00d16a0 2024-03-29T09:55:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.254+0000 7f5ce5a00700 1 -- 172.21.15.12:0/3145835593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce00d1ca0 con 0x7f5ce00d16a0 2024-03-29T09:55:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.255+0000 7f5cd6a00700 1 -- 172.21.15.12:0/3145835593 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5cd002d450 con 0x7f5ce00d16a0 2024-03-29T09:55:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.256+0000 7f5cd6a00700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cc006ac90 0x7f5cc006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.256+0000 7f5cd6a00700 1 -- 172.21.15.12:0/3145835593 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cd008a3b0 con 0x7f5ce00d16a0 2024-03-29T09:55:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.256+0000 7f5cde000700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cc006ac90 0x7f5cc006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.257+0000 7f5cde000700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cc006ac90 0x7f5cc006d150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0160ae0 tx=0x7f5ccc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.260+0000 7f5cd6a00700 1 -- 172.21.15.12:0/3145835593 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5cd00555a0 con 0x7f5ce00d16a0 2024-03-29T09:55:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.479+0000 7f5ce5a00700 1 -- 172.21.15.12:0/3145835593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5ce00d14c0 con 0x7f5ce00d16a0 2024-03-29T09:55:07.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.483+0000 7f5cd6a00700 1 -- 172.21.15.12:0/3145835593 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5cd0058bc0 con 0x7f5ce00d16a0 2024-03-29T09:55:07.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.485+0000 7f5ce5a00700 1 -- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cc006ac90 msgr2=0x7f5cc006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.485+0000 7f5ce5a00700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cc006ac90 0x7f5cc006d150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0160ae0 tx=0x7f5ccc009040 comp rx=0 tx=0).stop 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.485+0000 7f5ce5a00700 1 -- 172.21.15.12:0/3145835593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce00d16a0 msgr2=0x7f5ce00a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.485+0000 7f5ce5a00700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce00d16a0 0x7f5ce00a0a50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0014040 tx=0x7f5cd0004750 comp rx=0 tx=0).stop 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.485+0000 7f5ce5a00700 1 -- 172.21.15.12:0/3145835593 shutdown_connections 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.486+0000 7f5ce5a00700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cc006ac90 0x7f5cc006d150 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.486+0000 7f5ce5a00700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ce015d9d0 0x7f5ce00b75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.486+0000 7f5ce5a00700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce00d16a0 0x7f5ce00a0a50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.486+0000 7f5ce5a00700 1 --2- 172.21.15.12:0/3145835593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ce00d2540 0x7f5ce00b7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.486+0000 7f5ce5a00700 1 -- 172.21.15.12:0/3145835593 >> 172.21.15.12:0/3145835593 conn(0x7f5ce008aff0 msgr2=0x7f5ce00b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.486+0000 7f5ce5a00700 1 -- 172.21.15.12:0/3145835593 shutdown_connections 2024-03-29T09:55:07.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:07.486+0000 7f5ce5a00700 1 -- 172.21.15.12:0/3145835593 wait complete. 2024-03-29T09:55:07.503 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T09:55:07.503 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T09:55:07.591 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T09:55:07.684 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T09:55:07.685 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:55:07.685 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:07.729 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:55:07.729 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:07.814 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:55:07.814 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:07.899 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:08.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.138+0000 7f6dc5600700 1 Processor -- start 2024-03-29T09:55:08.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.138+0000 7f6dc5600700 1 -- start start 2024-03-29T09:55:08.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.139+0000 7f6dc5600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6dc00cf3c0 0x7f6dc00d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.139+0000 7f6dc5600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc015db40 0x7f6dc00cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.139+0000 7f6dc5600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc00d1330 con 0x7f6dc015d760 2024-03-29T09:55:08.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.139+0000 7f6dc5600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc00d1fb0 con 0x7f6dc00cf3c0 2024-03-29T09:55:08.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.139+0000 7f6dc5600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc00d1610 con 0x7f6dc015db40 2024-03-29T09:55:08.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.140+0000 7f6dbf400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc015db40 0x7f6dc00cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:08.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.140+0000 7f6dbe000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6dc00cf3c0 0x7f6dc00d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:08.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.140+0000 7f6dbf400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc015db40 0x7f6dc00cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:40526/0 (socket says 172.21.15.12:40526) 2024-03-29T09:55:08.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.140+0000 7f6dbf400700 1 -- 172.21.15.12:0/2482280493 learned_addr learned my addr 172.21.15.12:0/2482280493 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:08.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.140+0000 7f6dbf400700 1 -- 172.21.15.12:0/2482280493 >> v2:172.21.15.159:3300/0 conn(0x7f6dc00cf3c0 msgr2=0x7f6dc00d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:08.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.140+0000 7f6dbf400700 1 --2- 172.21.15.12:0/2482280493 >> v2:172.21.15.159:3300/0 conn(0x7f6dc00cf3c0 0x7f6dc00d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.140+0000 7f6dbf400700 1 -- 172.21.15.12:0/2482280493 >> v1:172.21.15.159:6789/0 conn(0x7f6dc015d760 legacy=0x7f6dc00cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.140+0000 7f6dbf400700 1 -- 172.21.15.12:0/2482280493 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc00d18f0 con 0x7f6dc015db40 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.140+0000 7f6dbe000700 1 --2- 172.21.15.12:0/2482280493 >> v2:172.21.15.159:3300/0 conn(0x7f6dc00cf3c0 0x7f6dc00d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.141+0000 7f6dbf400700 1 --2- 172.21.15.12:0/2482280493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc015db40 0x7f6dc00cffd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009e50 tx=0x7f6db800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dffa0707afe5271 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.141+0000 7f6dbd600700 1 -- 172.21.15.12:0/2482280493 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800e070 con 0x7f6dc015db40 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.141+0000 7f6dbd600700 1 -- 172.21.15.12:0/2482280493 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db800b910 con 0x7f6dc015db40 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.141+0000 7f6dbd600700 1 -- 172.21.15.12:0/2482280493 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800bcc0 con 0x7f6dc015db40 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.142+0000 7f6dc5600700 1 -- 172.21.15.12:0/2482280493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc015db40 msgr2=0x7f6dc00cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.142+0000 7f6dc5600700 1 --2- 172.21.15.12:0/2482280493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc015db40 0x7f6dc00cffd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009e50 tx=0x7f6db800b330 comp rx=0 tx=0).stop 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.142+0000 7f6dc5600700 1 -- 172.21.15.12:0/2482280493 shutdown_connections 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.142+0000 7f6dc5600700 1 --2- 172.21.15.12:0/2482280493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc015db40 0x7f6dc00cffd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.142+0000 7f6dc5600700 1 --2- 172.21.15.12:0/2482280493 >> v2:172.21.15.159:3300/0 conn(0x7f6dc00cf3c0 0x7f6dc00d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.142+0000 7f6dc5600700 1 -- 172.21.15.12:0/2482280493 >> 172.21.15.12:0/2482280493 conn(0x7f6dc008aff0 msgr2=0x7f6dc00d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.142+0000 7f6dc5600700 1 -- 172.21.15.12:0/2482280493 shutdown_connections 2024-03-29T09:55:08.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.142+0000 7f6dc5600700 1 -- 172.21.15.12:0/2482280493 wait complete. 2024-03-29T09:55:08.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.143+0000 7f6dc5600700 1 Processor -- start 2024-03-29T09:55:08.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.144+0000 7f6dc5600700 1 -- start start 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dc5600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc00cf3c0 0x7f6dc01f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dc5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc015d760 0x7f6dc00b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dc5600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6dc015db40 0x7f6dc00ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dc5600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc00b6740 con 0x7f6dc015d760 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dc5600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc00c7fa0 con 0x7f6dc015db40 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dc5600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc00c8280 con 0x7f6dc00cf3c0 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dbe000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc015d760 0x7f6dc00b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dbe000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc015d760 0x7f6dc00b6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37856/0 (socket says 172.21.15.12:37856) 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dbe000700 1 -- 172.21.15.12:0/387621395 learned_addr learned my addr 172.21.15.12:0/387621395 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dbe000700 1 -- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc00cf3c0 msgr2=0x7f6dc01f2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dbe000700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc00cf3c0 0x7f6dc01f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dbe000700 1 -- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6dc015db40 msgr2=0x7f6dc00ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dbe000700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6dc015db40 0x7f6dc00ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dbe000700 1 -- 172.21.15.12:0/387621395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac0094a0 con 0x7f6dc015d760 2024-03-29T09:55:08.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.145+0000 7f6dbe000700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc015d760 0x7f6dc00b6200 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6dac00ebe0 tx=0x7f6dac009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:08.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.146+0000 7f6db6a00700 1 -- 172.21.15.12:0/387621395 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac00c8b0 con 0x7f6dc015d760 2024-03-29T09:55:08.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.146+0000 7f6db6a00700 1 -- 172.21.15.12:0/387621395 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dac00ca10 con 0x7f6dc015d760 2024-03-29T09:55:08.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.146+0000 7f6db6a00700 1 -- 172.21.15.12:0/387621395 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac00cd80 con 0x7f6dc015d760 2024-03-29T09:55:08.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.146+0000 7f6dc5600700 1 -- 172.21.15.12:0/387621395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db8009b30 con 0x7f6dc015d760 2024-03-29T09:55:08.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.146+0000 7f6dc5600700 1 -- 172.21.15.12:0/387621395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc00c8840 con 0x7f6dc015d760 2024-03-29T09:55:08.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.147+0000 7f6dc5600700 1 -- 172.21.15.12:0/387621395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc0067720 con 0x7f6dc015d760 2024-03-29T09:55:08.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.148+0000 7f6db6a00700 1 -- 172.21.15.12:0/387621395 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6dac013470 con 0x7f6dc015d760 2024-03-29T09:55:08.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.149+0000 7f6db6a00700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6da006ac90 0x7f6da006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.149+0000 7f6db6a00700 1 -- 172.21.15.12:0/387621395 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6dac089560 con 0x7f6dc015d760 2024-03-29T09:55:08.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.149+0000 7f6dbea00700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6da006ac90 0x7f6da006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:08.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.150+0000 7f6dbea00700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6da006ac90 0x7f6da006d150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6db0000f60 tx=0x7f6db0007a40 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:08.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.153+0000 7f6db6a00700 1 -- 172.21.15.12:0/387621395 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6dac0546a0 con 0x7f6dc015d760 2024-03-29T09:55:08.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.367+0000 7f6dc5600700 1 -- 172.21.15.12:0/387621395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dc00b9460 con 0x7f6dc015d760 2024-03-29T09:55:08.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.368+0000 7f6db6a00700 1 -- 172.21.15.12:0/387621395 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f6dac057cc0 con 0x7f6dc015d760 2024-03-29T09:55:08.368 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:08.368 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.370+0000 7f6dc5600700 1 -- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6da006ac90 msgr2=0x7f6da006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.370+0000 7f6dc5600700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6da006ac90 0x7f6da006d150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6db0000f60 tx=0x7f6db0007a40 comp rx=0 tx=0).stop 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.370+0000 7f6dc5600700 1 -- 172.21.15.12:0/387621395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc015d760 msgr2=0x7f6dc00b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.370+0000 7f6dc5600700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc015d760 0x7f6dc00b6200 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6dac00ebe0 tx=0x7f6dac009e40 comp rx=0 tx=0).stop 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.371+0000 7f6dc5600700 1 -- 172.21.15.12:0/387621395 shutdown_connections 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.371+0000 7f6dc5600700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6da006ac90 0x7f6da006d150 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.371+0000 7f6dc5600700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6dc015db40 0x7f6dc00ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.371+0000 7f6dc5600700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc00cf3c0 0x7f6dc01f2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.371+0000 7f6dc5600700 1 --2- 172.21.15.12:0/387621395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc015d760 0x7f6dc00b6200 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.371+0000 7f6dc5600700 1 -- 172.21.15.12:0/387621395 >> 172.21.15.12:0/387621395 conn(0x7f6dc008aff0 msgr2=0x7f6dc00b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.371+0000 7f6dc5600700 1 -- 172.21.15.12:0/387621395 shutdown_connections 2024-03-29T09:55:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.371+0000 7f6dc5600700 1 -- 172.21.15.12:0/387621395 wait complete. 2024-03-29T09:55:08.388 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:55:08.389 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:55:08.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.590+0000 7f0ce6200700 1 Processor -- start 2024-03-29T09:55:08.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.590+0000 7f0ce6200700 1 -- start start 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.591+0000 7f0ce6200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0ce00d16a0 0x7f0ce00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.591+0000 7f0ce6200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ce00d2370 0x7f0ce00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.591+0000 7f0ce6200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce0161420 con 0x7f0ce015d9d0 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.591+0000 7f0ce6200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce00d4340 con 0x7f0ce00d2370 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.591+0000 7f0ce6200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce015edc0 con 0x7f0ce00d16a0 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.591+0000 7f0cdfe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ce00d2370 0x7f0ce00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.591+0000 7f0cdfe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ce00d2370 0x7f0ce00ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37866/0 (socket says 172.21.15.12:37866) 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.592+0000 7f0cdfe00700 1 -- 172.21.15.12:0/822883070 learned_addr learned my addr 172.21.15.12:0/822883070 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.592+0000 7f0cdea00700 1 --2- 172.21.15.12:0/822883070 >> v2:172.21.15.159:3300/0 conn(0x7f0ce00d16a0 0x7f0ce00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.592+0000 7f0cdfe00700 1 -- 172.21.15.12:0/822883070 >> v2:172.21.15.159:3300/0 conn(0x7f0ce00d16a0 msgr2=0x7f0ce00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.592+0000 7f0cdfe00700 1 --2- 172.21.15.12:0/822883070 >> v2:172.21.15.159:3300/0 conn(0x7f0ce00d16a0 0x7f0ce00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.592+0000 7f0cdfe00700 1 -- 172.21.15.12:0/822883070 >> v1:172.21.15.159:6789/0 conn(0x7f0ce015d9d0 legacy=0x7f0ce00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.592+0000 7f0cdfe00700 1 -- 172.21.15.12:0/822883070 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce0161700 con 0x7f0ce00d2370 2024-03-29T09:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.592+0000 7f0cdfe00700 1 --2- 172.21.15.12:0/822883070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 0x7f0ce00ce360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00d12f0 tx=0x7f0cd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d682279ea06a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.592+0000 7f0cde000700 1 -- 172.21.15.12:0/822883070 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd800e070 con 0x7f0ce00d2370 2024-03-29T09:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.592+0000 7f0cde000700 1 -- 172.21.15.12:0/822883070 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd8003a20 con 0x7f0ce00d2370 2024-03-29T09:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.592+0000 7f0cde000700 1 -- 172.21.15.12:0/822883070 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8003dd0 con 0x7f0ce00d2370 2024-03-29T09:55:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.593+0000 7f0ce6200700 1 -- 172.21.15.12:0/822883070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 msgr2=0x7f0ce00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.593+0000 7f0ce6200700 1 --2- 172.21.15.12:0/822883070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 0x7f0ce00ce360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00d12f0 tx=0x7f0cd800b330 comp rx=0 tx=0).stop 2024-03-29T09:55:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.593+0000 7f0ce6200700 1 -- 172.21.15.12:0/822883070 shutdown_connections 2024-03-29T09:55:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.593+0000 7f0ce6200700 1 --2- 172.21.15.12:0/822883070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 0x7f0ce00ce360 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.593+0000 7f0ce6200700 1 --2- 172.21.15.12:0/822883070 >> v2:172.21.15.159:3300/0 conn(0x7f0ce00d16a0 0x7f0ce00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.593+0000 7f0ce6200700 1 -- 172.21.15.12:0/822883070 >> 172.21.15.12:0/822883070 conn(0x7f0ce008aff0 msgr2=0x7f0ce00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:08.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.593+0000 7f0ce6200700 1 -- 172.21.15.12:0/822883070 shutdown_connections 2024-03-29T09:55:08.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.593+0000 7f0ce6200700 1 -- 172.21.15.12:0/822883070 wait complete. 2024-03-29T09:55:08.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.595+0000 7f0ce6200700 1 Processor -- start 2024-03-29T09:55:08.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.595+0000 7f0ce6200700 1 -- start start 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.596+0000 7f0ce6200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ce00d16a0 0x7f0ce015c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.596+0000 7f0ce6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 0x7f0ce00a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.596+0000 7f0ce6200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ce015d9d0 0x7f0ce015baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.596+0000 7f0ce6200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce00b8970 con 0x7f0ce00d2370 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.596+0000 7f0ce6200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce00ca5f0 con 0x7f0ce00d16a0 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.596+0000 7f0ce6200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce00b80d0 con 0x7f0ce015d9d0 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.596+0000 7f0cdea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 0x7f0ce00a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.596+0000 7f0cdea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 0x7f0ce00a0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37872/0 (socket says 172.21.15.12:37872) 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.596+0000 7f0cdea00700 1 -- 172.21.15.12:0/3704360847 learned_addr learned my addr 172.21.15.12:0/3704360847 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.596+0000 7f0cdf400700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ce00d16a0 0x7f0ce015c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cdfe00700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ce015d9d0 0x7f0ce015baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cdea00700 1 -- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ce015d9d0 msgr2=0x7f0ce015baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cdea00700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ce015d9d0 0x7f0ce015baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cdea00700 1 -- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ce00d16a0 msgr2=0x7f0ce015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cdea00700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ce00d16a0 0x7f0ce015c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cdea00700 1 -- 172.21.15.12:0/3704360847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd8009b30 con 0x7f0ce00d2370 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cdfe00700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ce015d9d0 0x7f0ce015baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cdea00700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 0x7f0ce00a0a30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00e840 tx=0x7f0ccc00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cd7400700 1 -- 172.21.15.12:0/3704360847 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc013610 con 0x7f0ce00d2370 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cd7400700 1 -- 172.21.15.12:0/3704360847 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc013770 con 0x7f0ce00d2370 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0cd7400700 1 -- 172.21.15.12:0/3704360847 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc013b20 con 0x7f0ce00d2370 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0ce6200700 1 -- 172.21.15.12:0/3704360847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce00b83b0 con 0x7f0ce00d2370 2024-03-29T09:55:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.597+0000 7f0ce6200700 1 -- 172.21.15.12:0/3704360847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce00ca820 con 0x7f0ce00d2370 2024-03-29T09:55:08.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.598+0000 7f0ce6200700 1 -- 172.21.15.12:0/3704360847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce00d1ca0 con 0x7f0ce00d2370 2024-03-29T09:55:08.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.599+0000 7f0cd7400700 1 -- 172.21.15.12:0/3704360847 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0ccc013c80 con 0x7f0ce00d2370 2024-03-29T09:55:08.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.600+0000 7f0cd7400700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0cc006ac90 0x7f0cc006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:08.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.600+0000 7f0cd7400700 1 -- 172.21.15.12:0/3704360847 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ccc089110 con 0x7f0ce00d2370 2024-03-29T09:55:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.600+0000 7f0cdf400700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0cc006ac90 0x7f0cc006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.601+0000 7f0cdf400700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0cc006ac90 0x7f0cc006d150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009870 tx=0x7f0cd0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:08.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.604+0000 7f0cd7400700 1 -- 172.21.15.12:0/3704360847 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ccc0541c0 con 0x7f0ce00d2370 2024-03-29T09:55:08.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.822+0000 7f0ce6200700 1 -- 172.21.15.12:0/3704360847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ce00caa10 con 0x7f0ce00d2370 2024-03-29T09:55:08.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.823+0000 7f0cd7400700 1 -- 172.21.15.12:0/3704360847 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ccc0577e0 con 0x7f0ce00d2370 2024-03-29T09:55:08.822 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:55:08.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 -- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0cc006ac90 msgr2=0x7f0cc006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0cc006ac90 0x7f0cc006d150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009870 tx=0x7f0cd0008040 comp rx=0 tx=0).stop 2024-03-29T09:55:08.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 -- 172.21.15.12:0/3704360847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 msgr2=0x7f0ce00a0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:08.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 0x7f0ce00a0a30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00e840 tx=0x7f0ccc00ec00 comp rx=0 tx=0).stop 2024-03-29T09:55:08.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 -- 172.21.15.12:0/3704360847 shutdown_connections 2024-03-29T09:55:08.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0cc006ac90 0x7f0cc006d150 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ce015d9d0 0x7f0ce015baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ce00d16a0 0x7f0ce015c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 --2- 172.21.15.12:0/3704360847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ce00d2370 0x7f0ce00a0a30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:08.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 -- 172.21.15.12:0/3704360847 >> 172.21.15.12:0/3704360847 conn(0x7f0ce008aff0 msgr2=0x7f0ce015e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:08.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 -- 172.21.15.12:0/3704360847 shutdown_connections 2024-03-29T09:55:08.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:08.825+0000 7f0ce6200700 1 -- 172.21.15.12:0/3704360847 wait complete. 2024-03-29T09:55:08.844 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:55:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.045+0000 7fddf5e00700 1 Processor -- start 2024-03-29T09:55:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.045+0000 7fddf5e00700 1 -- start start 2024-03-29T09:55:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddf5e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fddf015d9d0 0x7fddf00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddf5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fddf00d16a0 0x7fddf00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddf5e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf0161420 con 0x7fddf00d2370 2024-03-29T09:55:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddf5e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf00d27e0 con 0x7fddf00d16a0 2024-03-29T09:55:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddf5e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf0161700 con 0x7fddf015d9d0 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddee000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fddf00d16a0 0x7fddf00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddee000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fddf00d16a0 0x7fddf00d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40240/0 (socket says 172.21.15.12:40240) 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddee000700 1 -- 172.21.15.12:0/1684820081 learned_addr learned my addr 172.21.15.12:0/1684820081 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddeea00700 1 --2- 172.21.15.12:0/1684820081 >> v2:172.21.15.159:3300/0 conn(0x7fddf015d9d0 0x7fddf00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddee000700 1 -- 172.21.15.12:0/1684820081 >> v2:172.21.15.159:3300/0 conn(0x7fddf015d9d0 msgr2=0x7fddf00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddee000700 1 --2- 172.21.15.12:0/1684820081 >> v2:172.21.15.159:3300/0 conn(0x7fddf015d9d0 0x7fddf00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddee000700 1 -- 172.21.15.12:0/1684820081 >> v1:172.21.15.159:6789/0 conn(0x7fddf00d2370 legacy=0x7fddf015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.047+0000 7fddee000700 1 -- 172.21.15.12:0/1684820081 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf01619e0 con 0x7fddf00d16a0 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.048+0000 7fddee000700 1 --2- 172.21.15.12:0/1684820081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d16a0 0x7fddf00d1e30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fddf00d0ff0 tx=0x7fdddc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e468aa8b158a299b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.048+0000 7fdded600700 1 -- 172.21.15.12:0/1684820081 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc005e30 con 0x7fddf00d16a0 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.048+0000 7fdded600700 1 -- 172.21.15.12:0/1684820081 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc004030 con 0x7fddf00d16a0 2024-03-29T09:55:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.048+0000 7fdded600700 1 -- 172.21.15.12:0/1684820081 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc0043e0 con 0x7fddf00d16a0 2024-03-29T09:55:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.048+0000 7fddf5e00700 1 -- 172.21.15.12:0/1684820081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d16a0 msgr2=0x7fddf00d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.048+0000 7fddf5e00700 1 --2- 172.21.15.12:0/1684820081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d16a0 0x7fddf00d1e30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fddf00d0ff0 tx=0x7fdddc009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.049+0000 7fddf5e00700 1 -- 172.21.15.12:0/1684820081 shutdown_connections 2024-03-29T09:55:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.049+0000 7fddf5e00700 1 --2- 172.21.15.12:0/1684820081 >> v2:172.21.15.159:3300/0 conn(0x7fddf015d9d0 0x7fddf00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.049+0000 7fddf5e00700 1 --2- 172.21.15.12:0/1684820081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d16a0 0x7fddf00d1e30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.049+0000 7fddf5e00700 1 -- 172.21.15.12:0/1684820081 >> 172.21.15.12:0/1684820081 conn(0x7fddf008aff0 msgr2=0x7fddf00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.049+0000 7fddf5e00700 1 -- 172.21.15.12:0/1684820081 shutdown_connections 2024-03-29T09:55:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.049+0000 7fddf5e00700 1 -- 172.21.15.12:0/1684820081 wait complete. 2024-03-29T09:55:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.050+0000 7fddf5e00700 1 Processor -- start 2024-03-29T09:55:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.051+0000 7fddf5e00700 1 -- start start 2024-03-29T09:55:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.051+0000 7fddf5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddf00d16a0 0x7fddf0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.051+0000 7fddf5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d2370 0x7fddf00cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.051+0000 7fddf5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddf015d9d0 0x7fddf00b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddf5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf00cc510 con 0x7fddf00d2370 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddf5e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf00cb900 con 0x7fddf00d16a0 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddf5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf00cbbe0 con 0x7fddf015d9d0 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddee000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d2370 0x7fddf00cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddee000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d2370 0x7fddf00cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40252/0 (socket says 172.21.15.12:40252) 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddee000700 1 -- 172.21.15.12:0/3501811962 learned_addr learned my addr 172.21.15.12:0/3501811962 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddeea00700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddf00d16a0 0x7fddf0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddef400700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddf015d9d0 0x7fddf00b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddee000700 1 -- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddf015d9d0 msgr2=0x7fddf00b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddee000700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddf015d9d0 0x7fddf00b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddee000700 1 -- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddf00d16a0 msgr2=0x7fddf0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddee000700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddf00d16a0 0x7fddf0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddee000700 1 -- 172.21.15.12:0/3501811962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc009500 con 0x7fddf00d2370 2024-03-29T09:55:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.052+0000 7fddef400700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddf015d9d0 0x7fddf00b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.053+0000 7fddee000700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d2370 0x7fddf00cb000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009130 tx=0x7fdddc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.053+0000 7fdde7400700 1 -- 172.21.15.12:0/3501811962 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc003d20 con 0x7fddf00d2370 2024-03-29T09:55:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.053+0000 7fdde7400700 1 -- 172.21.15.12:0/3501811962 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc003e80 con 0x7fddf00d2370 2024-03-29T09:55:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.053+0000 7fdde7400700 1 -- 172.21.15.12:0/3501811962 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc011660 con 0x7fddf00d2370 2024-03-29T09:55:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.053+0000 7fddf5e00700 1 -- 172.21.15.12:0/3501811962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf00cbec0 con 0x7fddf00d2370 2024-03-29T09:55:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.053+0000 7fddf5e00700 1 -- 172.21.15.12:0/3501811962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf00cb540 con 0x7fddf00d2370 2024-03-29T09:55:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.054+0000 7fddf5e00700 1 -- 172.21.15.12:0/3501811962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf00cb6d0 con 0x7fddf00d2370 2024-03-29T09:55:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.055+0000 7fdde7400700 1 -- 172.21.15.12:0/3501811962 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdddc0117c0 con 0x7fddf00d2370 2024-03-29T09:55:09.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.056+0000 7fdde7400700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddd006ac90 0x7fddd006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.056+0000 7fdde7400700 1 -- 172.21.15.12:0/3501811962 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdddc08a540 con 0x7fddf00d2370 2024-03-29T09:55:09.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.057+0000 7fddeea00700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddd006ac90 0x7fddd006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.058+0000 7fddeea00700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddd006ac90 0x7fddd006d150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fddf00c9ca0 tx=0x7fdde0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.061+0000 7fdde7400700 1 -- 172.21.15.12:0/3501811962 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdddc055710 con 0x7fddf00d2370 2024-03-29T09:55:09.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.275+0000 7fddf5e00700 1 -- 172.21.15.12:0/3501811962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fddf00d1ca0 con 0x7fddf00d2370 2024-03-29T09:55:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.275+0000 7fdde7400700 1 -- 172.21.15.12:0/3501811962 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdddc058d30 con 0x7fddf00d2370 2024-03-29T09:55:09.275 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:55:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 -- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddd006ac90 msgr2=0x7fddd006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddd006ac90 0x7fddd006d150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fddf00c9ca0 tx=0x7fdde0008040 comp rx=0 tx=0).stop 2024-03-29T09:55:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 -- 172.21.15.12:0/3501811962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d2370 msgr2=0x7fddf00cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d2370 0x7fddf00cb000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009130 tx=0x7fdddc004750 comp rx=0 tx=0).stop 2024-03-29T09:55:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 -- 172.21.15.12:0/3501811962 shutdown_connections 2024-03-29T09:55:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddd006ac90 0x7fddd006d150 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddf015d9d0 0x7fddf00b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddf00d16a0 0x7fddf0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 --2- 172.21.15.12:0/3501811962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf00d2370 0x7fddf00cb000 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 -- 172.21.15.12:0/3501811962 >> 172.21.15.12:0/3501811962 conn(0x7fddf008aff0 msgr2=0x7fddf00b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:09.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 -- 172.21.15.12:0/3501811962 shutdown_connections 2024-03-29T09:55:09.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.278+0000 7fddf5e00700 1 -- 172.21.15.12:0/3501811962 wait complete. 2024-03-29T09:55:09.295 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:55:09.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.496+0000 7fe232e00700 1 Processor -- start 2024-03-29T09:55:09.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.496+0000 7fe232e00700 1 -- start start 2024-03-29T09:55:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.497+0000 7fe232e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe22c15d9d0 0x7fe22c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.497+0000 7fe232e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe22c0d16a0 0x7fe22c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.497+0000 7fe232e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c161420 con 0x7fe22c0d2370 2024-03-29T09:55:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.497+0000 7fe232e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22c0d2820 con 0x7fe22c15d9d0 2024-03-29T09:55:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.497+0000 7fe232e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c161700 con 0x7fe22c0d16a0 2024-03-29T09:55:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe230e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe22c0d2370 0x7fe22c15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54440/0 (socket says 172.21.15.12:54440) 2024-03-29T09:55:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe230e00700 1 -- 172.21.15.12:0/2436485782 learned_addr learned my addr 172.21.15.12:0/2436485782 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe22be00700 1 --2- 172.21.15.12:0/2436485782 >> v2:172.21.15.159:3300/0 conn(0x7fe22c15d9d0 0x7fe22c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe22b400700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe22c0d16a0 0x7fe22c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008195304 0 0) 0x7fe22c161420 con 0x7fe22c0d2370 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2100035b0 con 0x7fe22c0d2370 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4019568077 0 0) 0x7fe2100035b0 con 0x7fe22c0d2370 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe22c0d16a0 msgr2=0x7fe22c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe22aa00700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe22c0d16a0 0x7fe22c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 >> v2:172.21.15.159:3300/0 conn(0x7fe22c15d9d0 msgr2=0x7fe22c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe22aa00700 1 --2- 172.21.15.12:0/2436485782 >> v2:172.21.15.159:3300/0 conn(0x7fe22c15d9d0 0x7fe22c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.498+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c1619e0 con 0x7fe22c0d2370 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fe2240019f0 con 0x7fe22c0d2370 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 >> v1:172.21.15.12:6789/0 conn(0x7fe22c0d2370 legacy=0x7fe22c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe22aa00700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe210003a00 0x7fe210003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe22aa00700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe22c0d16a0 0x7fe210004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe22aa00700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe210004d90 0x7fe210005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0d2820 con 0x7fe210004d90 2024-03-29T09:55:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c161700 con 0x7fe210003a00 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21000ba10 con 0x7fe22c0d16a0 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe230e00700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe210004d90 0x7fe210005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe22be00700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe210003a00 0x7fe210003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe22b400700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe22c0d16a0 0x7fe210004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe230e00700 1 -- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe22c0d16a0 msgr2=0x7fe210004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe230e00700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe22c0d16a0 0x7fe210004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe230e00700 1 -- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe210003a00 msgr2=0x7fe210003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe230e00700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe210003a00 0x7fe210003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.499+0000 7fe230e00700 1 -- 172.21.15.12:0/2436485782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2100035b0 con 0x7fe210004d90 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.500+0000 7fe230e00700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe210004d90 0x7fe210005210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe22400a860 tx=0x7fe210005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b525e8a6c3080cd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.500+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224012600 con 0x7fe210004d90 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.500+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe224012780 con 0x7fe210004d90 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.500+0000 7fe22aa00700 1 -- 172.21.15.12:0/2436485782 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224012b30 con 0x7fe210004d90 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.500+0000 7fe232e00700 1 -- 172.21.15.12:0/2436485782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe210004d90 msgr2=0x7fe210005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.500+0000 7fe232e00700 1 --2- 172.21.15.12:0/2436485782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe210004d90 0x7fe210005210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe22400a860 tx=0x7fe210005880 comp rx=0 tx=0).stop 2024-03-29T09:55:09.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.500+0000 7fe230e00700 1 -- 172.21.15.12:0/2436485782 reap_dead start 2024-03-29T09:55:09.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.500+0000 7fe232e00700 1 -- 172.21.15.12:0/2436485782 shutdown_connections 2024-03-29T09:55:09.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.500+0000 7fe232e00700 1 -- 172.21.15.12:0/2436485782 >> 172.21.15.12:0/2436485782 conn(0x7fe22c08aff0 msgr2=0x7fe22c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:09.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.501+0000 7fe232e00700 1 -- 172.21.15.12:0/2436485782 shutdown_connections 2024-03-29T09:55:09.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.501+0000 7fe232e00700 1 -- 172.21.15.12:0/2436485782 wait complete. 2024-03-29T09:55:09.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.502+0000 7fe232e00700 1 Processor -- start 2024-03-29T09:55:09.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.502+0000 7fe232e00700 1 -- start start 2024-03-29T09:55:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.503+0000 7fe232e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe210004d90 0x7fe22c151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.503+0000 7fe232e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe22c1525b0 0x7fe22c0b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.503+0000 7fe232e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe22c0b79c0 0x7fe22c0b4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.503+0000 7fe232e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0b7530 con 0x7fe22c1525b0 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.503+0000 7fe232e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0b6920 con 0x7fe22c0b79c0 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.503+0000 7fe232e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0b6c00 con 0x7fe210004d90 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe22b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe22c1525b0 0x7fe22c0b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe22b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe22c1525b0 0x7fe22c0b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40268/0 (socket says 172.21.15.12:40268) 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe22b400700 1 -- 172.21.15.12:0/139396447 learned_addr learned my addr 172.21.15.12:0/139396447 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe22be00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe210004d90 0x7fe22c151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe230e00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe22c0b79c0 0x7fe22c0b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe22b400700 1 -- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe210004d90 msgr2=0x7fe22c151aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe22b400700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe210004d90 0x7fe22c151aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe22b400700 1 -- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe22c0b79c0 msgr2=0x7fe22c0b4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe22b400700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe22c0b79c0 0x7fe22c0b4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe22b400700 1 -- 172.21.15.12:0/139396447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c0b6ee0 con 0x7fe22c1525b0 2024-03-29T09:55:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe230e00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe22c0b79c0 0x7fe22c0b4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.504+0000 7fe22b400700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe22c1525b0 0x7fe22c0b6020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe2180119a0 tx=0x7fe218011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.505+0000 7fe223e00700 1 -- 172.21.15.12:0/139396447 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218011ee0 con 0x7fe22c1525b0 2024-03-29T09:55:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.505+0000 7fe223e00700 1 -- 172.21.15.12:0/139396447 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe218005a20 con 0x7fe22c1525b0 2024-03-29T09:55:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.505+0000 7fe223e00700 1 -- 172.21.15.12:0/139396447 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218005df0 con 0x7fe22c1525b0 2024-03-29T09:55:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.505+0000 7fe232e00700 1 -- 172.21.15.12:0/139396447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c0b71c0 con 0x7fe22c1525b0 2024-03-29T09:55:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.505+0000 7fe232e00700 1 -- 172.21.15.12:0/139396447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c0b4b80 con 0x7fe22c1525b0 2024-03-29T09:55:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.506+0000 7fe232e00700 1 -- 172.21.15.12:0/139396447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22c0d1ca0 con 0x7fe22c1525b0 2024-03-29T09:55:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.507+0000 7fe223e00700 1 -- 172.21.15.12:0/139396447 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe21800f940 con 0x7fe22c1525b0 2024-03-29T09:55:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.508+0000 7fe223e00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe20c06ac90 0x7fe20c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.508+0000 7fe223e00700 1 -- 172.21.15.12:0/139396447 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe218089780 con 0x7fe22c1525b0 2024-03-29T09:55:09.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.508+0000 7fe22be00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe20c06ac90 0x7fe20c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.509+0000 7fe22be00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe20c06ac90 0x7fe20c06d150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe21c009c60 tx=0x7fe21c009210 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:09.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.513+0000 7fe223e00700 1 -- 172.21.15.12:0/139396447 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2180548c0 con 0x7fe22c1525b0 2024-03-29T09:55:09.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.725+0000 7fe232e00700 1 -- 172.21.15.12:0/139396447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe22c067720 con 0x7fe22c1525b0 2024-03-29T09:55:09.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.725+0000 7fe223e00700 1 -- 172.21.15.12:0/139396447 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe218057ee0 con 0x7fe22c1525b0 2024-03-29T09:55:09.725 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 -- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe20c06ac90 msgr2=0x7fe20c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe20c06ac90 0x7fe20c06d150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe21c009c60 tx=0x7fe21c009210 comp rx=0 tx=0).stop 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 -- 172.21.15.12:0/139396447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe22c1525b0 msgr2=0x7fe22c0b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe22c1525b0 0x7fe22c0b6020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe2180119a0 tx=0x7fe218011d60 comp rx=0 tx=0).stop 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 -- 172.21.15.12:0/139396447 shutdown_connections 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe20c06ac90 0x7fe20c06d150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe22c0b79c0 0x7fe22c0b4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe210004d90 0x7fe22c151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 --2- 172.21.15.12:0/139396447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe22c1525b0 0x7fe22c0b6020 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 -- 172.21.15.12:0/139396447 >> 172.21.15.12:0/139396447 conn(0x7fe22c08aff0 msgr2=0x7fe22c0b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 -- 172.21.15.12:0/139396447 shutdown_connections 2024-03-29T09:55:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.728+0000 7fe232e00700 1 -- 172.21.15.12:0/139396447 wait complete. 2024-03-29T09:55:09.746 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T09:55:09.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.946+0000 7fcba9600700 1 Processor -- start 2024-03-29T09:55:09.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.946+0000 7fcba9600700 1 -- start start 2024-03-29T09:55:09.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.947+0000 7fcba9600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcba40d16a0 0x7fcba40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.948+0000 7fcba9600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba4161420 con 0x7fcba40d2370 2024-03-29T09:55:09.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.948+0000 7fcba9600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba4161700 con 0x7fcba415d9d0 2024-03-29T09:55:09.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.948+0000 7fcba9600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba40d4340 con 0x7fcba40d16a0 2024-03-29T09:55:09.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.948+0000 7fcba3400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcba40d2370 0x7fcba40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54452/0 (socket says 172.21.15.12:54452) 2024-03-29T09:55:09.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.948+0000 7fcba3400700 1 -- 172.21.15.12:0/1998323583 learned_addr learned my addr 172.21.15.12:0/1998323583 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:09.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.948+0000 7fcba2000700 1 --2- 172.21.15.12:0/1998323583 >> v2:172.21.15.159:3300/0 conn(0x7fcba40d16a0 0x7fcba40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.948+0000 7fcba1600700 1 -- 172.21.15.12:0/1998323583 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182254110 0 0) 0x7fcba4161420 con 0x7fcba40d2370 2024-03-29T09:55:09.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.948+0000 7fcba1600700 1 -- 172.21.15.12:0/1998323583 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb880035b0 con 0x7fcba40d2370 2024-03-29T09:55:09.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.948+0000 7fcba2000700 1 -- 172.21.15.12:0/1998323583 >> v1:172.21.15.159:6789/0 conn(0x7fcba415d9d0 legacy=0x7fcba40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.949+0000 7fcba2000700 1 -- 172.21.15.12:0/1998323583 >> v1:172.21.15.12:6789/0 conn(0x7fcba40d2370 legacy=0x7fcba40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.949+0000 7fcba2000700 1 -- 172.21.15.12:0/1998323583 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba41619e0 con 0x7fcba40d16a0 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.949+0000 7fcba2000700 1 --2- 172.21.15.12:0/1998323583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcba40d16a0 0x7fcba40d1e30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009820 tx=0x7fcb90009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e465ffca37644c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.949+0000 7fcba1600700 1 -- 172.21.15.12:0/1998323583 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90005e30 con 0x7fcba40d16a0 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.949+0000 7fcba1600700 1 -- 172.21.15.12:0/1998323583 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb90006030 con 0x7fcba40d16a0 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.949+0000 7fcba1600700 1 -- 172.21.15.12:0/1998323583 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb900063c0 con 0x7fcba40d16a0 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.950+0000 7fcba9600700 1 -- 172.21.15.12:0/1998323583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcba40d16a0 msgr2=0x7fcba40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.950+0000 7fcba9600700 1 --2- 172.21.15.12:0/1998323583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcba40d16a0 0x7fcba40d1e30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009820 tx=0x7fcb90009b30 comp rx=0 tx=0).stop 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.950+0000 7fcba9600700 1 -- 172.21.15.12:0/1998323583 shutdown_connections 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.950+0000 7fcba9600700 1 --2- 172.21.15.12:0/1998323583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcba40d16a0 0x7fcba40d1e30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.950+0000 7fcba9600700 1 -- 172.21.15.12:0/1998323583 >> 172.21.15.12:0/1998323583 conn(0x7fcba408aff0 msgr2=0x7fcba40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.950+0000 7fcba9600700 1 -- 172.21.15.12:0/1998323583 shutdown_connections 2024-03-29T09:55:09.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.950+0000 7fcba9600700 1 -- 172.21.15.12:0/1998323583 wait complete. 2024-03-29T09:55:09.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.951+0000 7fcba9600700 1 Processor -- start 2024-03-29T09:55:09.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.952+0000 7fcba9600700 1 -- start start 2024-03-29T09:55:09.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.952+0000 7fcba9600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcba40d16a0 0x7fcba40b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.952+0000 7fcba9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba40d2370 0x7fcba40b8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.952+0000 7fcba9600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcba415d9d0 0x7fcba41607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.952+0000 7fcba9600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40b76d0 con 0x7fcba40d2370 2024-03-29T09:55:09.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.952+0000 7fcba9600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40b7110 con 0x7fcba40d16a0 2024-03-29T09:55:09.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.952+0000 7fcba9600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40b73f0 con 0x7fcba415d9d0 2024-03-29T09:55:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba40d2370 0x7fcba40b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba40d2370 0x7fcba40b8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40272/0 (socket says 172.21.15.12:40272) 2024-03-29T09:55:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2000700 1 -- 172.21.15.12:0/3820349976 learned_addr learned my addr 172.21.15.12:0/3820349976 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2000700 1 -- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcba415d9d0 msgr2=0x7fcba41607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2000700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcba415d9d0 0x7fcba41607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2a00700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcba40d16a0 0x7fcba40b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2000700 1 -- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcba40d16a0 msgr2=0x7fcba40b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2000700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcba40d16a0 0x7fcba40b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2000700 1 -- 172.21.15.12:0/3820349976 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb90009430 con 0x7fcba40d2370 2024-03-29T09:55:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2a00700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcba40d16a0 0x7fcba40b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.953+0000 7fcba2000700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba40d2370 0x7fcba40b8440 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcb90000c00 tx=0x7fcb90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.954+0000 7fcb9aa00700 1 -- 172.21.15.12:0/3820349976 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90004240 con 0x7fcba40d2370 2024-03-29T09:55:09.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.954+0000 7fcb9aa00700 1 -- 172.21.15.12:0/3820349976 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb900043a0 con 0x7fcba40d2370 2024-03-29T09:55:09.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.954+0000 7fcb9aa00700 1 -- 172.21.15.12:0/3820349976 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb900036a0 con 0x7fcba40d2370 2024-03-29T09:55:09.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.954+0000 7fcba9600700 1 -- 172.21.15.12:0/3820349976 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba415df50 con 0x7fcba40d2370 2024-03-29T09:55:09.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.954+0000 7fcba9600700 1 -- 172.21.15.12:0/3820349976 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba40b8980 con 0x7fcba40d2370 2024-03-29T09:55:09.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.955+0000 7fcba9600700 1 -- 172.21.15.12:0/3820349976 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba40d1ca0 con 0x7fcba40d2370 2024-03-29T09:55:09.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.956+0000 7fcb9aa00700 1 -- 172.21.15.12:0/3820349976 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcb90022070 con 0x7fcba40d2370 2024-03-29T09:55:09.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.956+0000 7fcb9aa00700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcb8406ac90 0x7fcb8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:09.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.957+0000 7fcb9aa00700 1 -- 172.21.15.12:0/3820349976 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb9008ab80 con 0x7fcba40d2370 2024-03-29T09:55:09.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.957+0000 7fcba2a00700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcb8406ac90 0x7fcb8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:09.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.958+0000 7fcba2a00700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcb8406ac90 0x7fcb8406d150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcb94000f60 tx=0x7fcb94009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:09.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:09.961+0000 7fcb9aa00700 1 -- 172.21.15.12:0/3820349976 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb90055d50 con 0x7fcba40d2370 2024-03-29T09:55:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.188+0000 7fcba9600700 1 -- 172.21.15.12:0/3820349976 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcba415ddb0 con 0x7fcba40d2370 2024-03-29T09:55:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.188+0000 7fcb9aa00700 1 -- 172.21.15.12:0/3820349976 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fcba415ddb0 con 0x7fcba40d2370 2024-03-29T09:55:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 -- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcb8406ac90 msgr2=0x7fcb8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcb8406ac90 0x7fcb8406d150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcb94000f60 tx=0x7fcb94009040 comp rx=0 tx=0).stop 2024-03-29T09:55:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 -- 172.21.15.12:0/3820349976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba40d2370 msgr2=0x7fcba40b8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba40d2370 0x7fcba40b8440 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcb90000c00 tx=0x7fcb90004750 comp rx=0 tx=0).stop 2024-03-29T09:55:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 -- 172.21.15.12:0/3820349976 shutdown_connections 2024-03-29T09:55:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcb8406ac90 0x7fcb8406d150 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcba415d9d0 0x7fcba41607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcba40d16a0 0x7fcba40b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 --2- 172.21.15.12:0/3820349976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba40d2370 0x7fcba40b8440 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 -- 172.21.15.12:0/3820349976 >> 172.21.15.12:0/3820349976 conn(0x7fcba408aff0 msgr2=0x7fcba40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.190+0000 7fcba9600700 1 -- 172.21.15.12:0/3820349976 shutdown_connections 2024-03-29T09:55:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.191+0000 7fcba9600700 1 -- 172.21.15.12:0/3820349976 wait complete. 2024-03-29T09:55:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-29T09:55:10.208 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T09:55:10.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:10.208 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-29T09:55:10.281 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T09:55:10.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:10.282 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T09:55:10.330 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:55:10.331 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:10.381 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:55:10.382 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:10.429 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:55:10.429 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:10.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:10.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.714+0000 7f5acb400700 1 Processor -- start 2024-03-29T09:55:10.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.715+0000 7f5acb400700 1 -- start start 2024-03-29T09:55:10.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5acb400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40cb3f0 0x7f5ac40ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:10.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5acb400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5ac40ac600 0x7f5ac406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5acb400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac415fd50 con 0x7f5ac415b5d0 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5acb400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac415b9b0 con 0x7f5ac40ac600 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5acb400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac40aca70 con 0x7f5ac40cb3f0 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5ac9400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5ac40ac600 0x7f5ac406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5ac3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40cb3f0 0x7f5ac40ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5ac3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40cb3f0 0x7f5ac40ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:55934/0 (socket says 172.21.15.12:55934) 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5ac9400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5ac40ac600 0x7f5ac406c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:45646/0 (socket says 172.21.15.12:45646) 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5ac3e00700 1 -- 172.21.15.12:0/3102178488 learned_addr learned my addr 172.21.15.12:0/3102178488 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.716+0000 7f5ac8a00700 1 --1- 172.21.15.12:0/3102178488 >> v1:172.21.15.159:6789/0 conn(0x7f5ac415b5d0 0x7f5ac40ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46060/0 (socket says 172.21.15.12:46060) 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.717+0000 7f5ac3e00700 1 -- 172.21.15.12:0/3102178488 >> v2:172.21.15.159:3300/0 conn(0x7f5ac40ac600 msgr2=0x7f5ac406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.717+0000 7f5ac3e00700 1 --2- 172.21.15.12:0/3102178488 >> v2:172.21.15.159:3300/0 conn(0x7f5ac40ac600 0x7f5ac406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.717+0000 7f5ac3e00700 1 -- 172.21.15.12:0/3102178488 >> v1:172.21.15.159:6789/0 conn(0x7f5ac415b5d0 legacy=0x7f5ac40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:10.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.717+0000 7f5ac3e00700 1 -- 172.21.15.12:0/3102178488 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac415f140 con 0x7f5ac40cb3f0 2024-03-29T09:55:10.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.717+0000 7f5ac9400700 1 --2- 172.21.15.12:0/3102178488 >> v2:172.21.15.159:3300/0 conn(0x7f5ac40ac600 0x7f5ac406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:10.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.717+0000 7f5ac3e00700 1 --2- 172.21.15.12:0/3102178488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40cb3f0 0x7f5ac40ac0c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00098f0 tx=0x7f5ab0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af970347085708d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:10.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.717+0000 7f5ac3400700 1 -- 172.21.15.12:0/3102178488 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0011830 con 0x7f5ac40cb3f0 2024-03-29T09:55:10.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.717+0000 7f5ac3400700 1 -- 172.21.15.12:0/3102178488 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab0011990 con 0x7f5ac40cb3f0 2024-03-29T09:55:10.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.717+0000 7f5ac3400700 1 -- 172.21.15.12:0/3102178488 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0011d20 con 0x7f5ac40cb3f0 2024-03-29T09:55:10.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.718+0000 7f5acb400700 1 -- 172.21.15.12:0/3102178488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40cb3f0 msgr2=0x7f5ac40ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:10.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.718+0000 7f5acb400700 1 --2- 172.21.15.12:0/3102178488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40cb3f0 0x7f5ac40ac0c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00098f0 tx=0x7f5ab0009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:10.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.718+0000 7f5acb400700 1 -- 172.21.15.12:0/3102178488 shutdown_connections 2024-03-29T09:55:10.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.718+0000 7f5acb400700 1 --2- 172.21.15.12:0/3102178488 >> v2:172.21.15.159:3300/0 conn(0x7f5ac40ac600 0x7f5ac406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.718+0000 7f5acb400700 1 --2- 172.21.15.12:0/3102178488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40cb3f0 0x7f5ac40ac0c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.718+0000 7f5acb400700 1 -- 172.21.15.12:0/3102178488 >> 172.21.15.12:0/3102178488 conn(0x7f5ac40cf190 msgr2=0x7f5ac40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:10.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.718+0000 7f5acb400700 1 -- 172.21.15.12:0/3102178488 shutdown_connections 2024-03-29T09:55:10.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.718+0000 7f5acb400700 1 -- 172.21.15.12:0/3102178488 wait complete. 2024-03-29T09:55:10.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.720+0000 7f5acb400700 1 Processor -- start 2024-03-29T09:55:10.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.720+0000 7f5acb400700 1 -- start start 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5acb400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40ac600 0x7f5ac4157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5acb400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ac415b5d0 0x7f5ac4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5acb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac409af60 0x7f5ac415a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5acb400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac41ef870 con 0x7f5ac409af60 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5acb400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac41eea00 con 0x7f5ac415b5d0 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5acb400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac415ec00 con 0x7f5ac40ac600 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5ac9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac409af60 0x7f5ac415a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5ac9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac409af60 0x7f5ac415a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40282/0 (socket says 172.21.15.12:40282) 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5ac9400700 1 -- 172.21.15.12:0/3615391323 learned_addr learned my addr 172.21.15.12:0/3615391323 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5ac8a00700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40ac600 0x7f5ac4157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:10.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5ac3e00700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ac415b5d0 0x7f5ac4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5ac9400700 1 -- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40ac600 msgr2=0x7f5ac4157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5ac9400700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40ac600 0x7f5ac4157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5ac9400700 1 -- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ac415b5d0 msgr2=0x7f5ac4159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5ac9400700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ac415b5d0 0x7f5ac4159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.721+0000 7f5ac9400700 1 -- 172.21.15.12:0/3615391323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc009a70 con 0x7f5ac409af60 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.722+0000 7f5ac3e00700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ac415b5d0 0x7f5ac4159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.722+0000 7f5ac9400700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac409af60 0x7f5ac415a170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009f00 tx=0x7f5abc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.722+0000 7f5ac0c00700 1 -- 172.21.15.12:0/3615391323 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc009d70 con 0x7f5ac409af60 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.722+0000 7f5ac0c00700 1 -- 172.21.15.12:0/3615391323 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc00f040 con 0x7f5ac409af60 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.722+0000 7f5ac0c00700 1 -- 172.21.15.12:0/3615391323 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc014680 con 0x7f5ac409af60 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.722+0000 7f5acb400700 1 -- 172.21.15.12:0/3615391323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab0009550 con 0x7f5ac409af60 2024-03-29T09:55:10.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.722+0000 7f5acb400700 1 -- 172.21.15.12:0/3615391323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac41eeed0 con 0x7f5ac409af60 2024-03-29T09:55:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.723+0000 7f5acb400700 1 -- 172.21.15.12:0/3615391323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac4067720 con 0x7f5ac409af60 2024-03-29T09:55:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.724+0000 7f5ac0c00700 1 -- 172.21.15.12:0/3615391323 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5abc0147e0 con 0x7f5ac409af60 2024-03-29T09:55:10.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.726+0000 7f5ac0c00700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5aa406ac90 0x7f5aa406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:10.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.726+0000 7f5ac0c00700 1 -- 172.21.15.12:0/3615391323 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5abc08a090 con 0x7f5ac409af60 2024-03-29T09:55:10.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.726+0000 7f5ac8a00700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5aa406ac90 0x7f5aa406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:10.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.727+0000 7f5ac8a00700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5aa406ac90 0x7f5aa406d150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8000f60 tx=0x7f5ab8001520 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:10.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.730+0000 7f5ac0c00700 1 -- 172.21.15.12:0/3615391323 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5abc055260 con 0x7f5ac409af60 2024-03-29T09:55:10.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.943+0000 7f5acb400700 1 -- 172.21.15.12:0/3615391323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ac406ea60 con 0x7f5ac409af60 2024-03-29T09:55:10.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.944+0000 7f5ac0c00700 1 -- 172.21.15.12:0/3615391323 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f5abc058880 con 0x7f5ac409af60 2024-03-29T09:55:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:10.944 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:54:28.427654+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 -- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5aa406ac90 msgr2=0x7f5aa406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5aa406ac90 0x7f5aa406d150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8000f60 tx=0x7f5ab8001520 comp rx=0 tx=0).stop 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 -- 172.21.15.12:0/3615391323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac409af60 msgr2=0x7f5ac415a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac409af60 0x7f5ac415a170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009f00 tx=0x7f5abc00c3a0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 -- 172.21.15.12:0/3615391323 shutdown_connections 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5aa406ac90 0x7f5aa406d150 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac409af60 0x7f5ac415a170 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ac40ac600 0x7f5ac4157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 --2- 172.21.15.12:0/3615391323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ac415b5d0 0x7f5ac4159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 -- 172.21.15.12:0/3615391323 >> 172.21.15.12:0/3615391323 conn(0x7f5ac40cf190 msgr2=0x7f5ac40aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 -- 172.21.15.12:0/3615391323 shutdown_connections 2024-03-29T09:55:10.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:10.947+0000 7f5acb400700 1 -- 172.21.15.12:0/3615391323 wait complete. 2024-03-29T09:55:10.965 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:55:10.966 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:55:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.172+0000 7f1fa1c00700 1 Processor -- start 2024-03-29T09:55:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.172+0000 7f1fa1c00700 1 -- start start 2024-03-29T09:55:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.173+0000 7f1fa1c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1f9c099a60 0x7f1f9c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.173+0000 7f1fa1c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f9c09a750 0x7f1f9c06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.174+0000 7f1fa1c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9c0d0970 con 0x7f1f9c15d990 2024-03-29T09:55:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.174+0000 7f1fa1c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9c06c020 con 0x7f1f9c099a60 2024-03-29T09:55:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.174+0000 7f1fa1c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c0c81a0 con 0x7f1f9c09a750 2024-03-29T09:55:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.174+0000 7f1f9a000700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f1f9c15d990 0x7f1f9c099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46062/0 (socket says 172.21.15.12:46062) 2024-03-29T09:55:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.174+0000 7f1f9aa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1f9c099a60 0x7f1f9c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.174+0000 7f1f9b400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f9c09a750 0x7f1f9c06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.174+0000 7f1f9a000700 1 -- 172.21.15.12:0/1915702708 learned_addr learned my addr 172.21.15.12:0/1915702708 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.175+0000 7f1f99600700 1 -- 172.21.15.12:0/1915702708 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226974646 0 0) 0x7f1f9c0d0970 con 0x7f1f9c15d990 2024-03-29T09:55:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.175+0000 7f1f99600700 1 -- 172.21.15.12:0/1915702708 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f800035b0 con 0x7f1f9c15d990 2024-03-29T09:55:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.175+0000 7f1f9aa00700 1 -- 172.21.15.12:0/1915702708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f9c09a750 msgr2=0x7f1f9c06c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.175+0000 7f1f9aa00700 1 --2- 172.21.15.12:0/1915702708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f9c09a750 0x7f1f9c06c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.175+0000 7f1f9aa00700 1 -- 172.21.15.12:0/1915702708 >> v1:172.21.15.159:6789/0 conn(0x7f1f9c15d990 legacy=0x7f1f9c099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.175+0000 7f1f9aa00700 1 -- 172.21.15.12:0/1915702708 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c0d0690 con 0x7f1f9c099a60 2024-03-29T09:55:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.175+0000 7f1f9aa00700 1 --2- 172.21.15.12:0/1915702708 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f9c099a60 0x7f1f9c0d0060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009840 tx=0x7f1f8c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ddafbe5103455a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1f99600700 1 -- 172.21.15.12:0/1915702708 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c011830 con 0x7f1f9c099a60 2024-03-29T09:55:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1f99600700 1 -- 172.21.15.12:0/1915702708 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c011990 con 0x7f1f9c099a60 2024-03-29T09:55:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1f99600700 1 -- 172.21.15.12:0/1915702708 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c011d20 con 0x7f1f9c099a60 2024-03-29T09:55:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1915702708 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f9c099a60 msgr2=0x7f1f9c0d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1fa1c00700 1 --2- 172.21.15.12:0/1915702708 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f9c099a60 0x7f1f9c0d0060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009840 tx=0x7f1f8c009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1915702708 shutdown_connections 2024-03-29T09:55:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1fa1c00700 1 --2- 172.21.15.12:0/1915702708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f9c09a750 0x7f1f9c06c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1fa1c00700 1 --2- 172.21.15.12:0/1915702708 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f9c099a60 0x7f1f9c0d0060 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1915702708 >> 172.21.15.12:0/1915702708 conn(0x7f1f9c0b2e20 msgr2=0x7f1f9c0b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1915702708 shutdown_connections 2024-03-29T09:55:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.176+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1915702708 wait complete. 2024-03-29T09:55:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.178+0000 7f1fa1c00700 1 Processor -- start 2024-03-29T09:55:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.178+0000 7f1fa1c00700 1 -- start start 2024-03-29T09:55:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.179+0000 7f1fa1c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c099a60 0x7f1f9c15f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.179+0000 7f1fa1c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f9c09a750 0x7f1f9c0ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.179+0000 7f1fa1c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f9c15d990 0x7f1f9c1507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.179+0000 7f1fa1c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c160a30 con 0x7f1f9c099a60 2024-03-29T09:55:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.179+0000 7f1fa1c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c15fe20 con 0x7f1f9c09a750 2024-03-29T09:55:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.179+0000 7f1fa1c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c160100 con 0x7f1f9c15d990 2024-03-29T09:55:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.179+0000 7f1f9aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c099a60 0x7f1f9c15f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.179+0000 7f1f9aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c099a60 0x7f1f9c15f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40292/0 (socket says 172.21.15.12:40292) 2024-03-29T09:55:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.179+0000 7f1f9aa00700 1 -- 172.21.15.12:0/1377442602 learned_addr learned my addr 172.21.15.12:0/1377442602 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.179+0000 7f1f9a000700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f9c09a750 0x7f1f9c0ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1f9b400700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f9c15d990 0x7f1f9c1507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1f9aa00700 1 -- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f9c15d990 msgr2=0x7f1f9c1507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1f9aa00700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f9c15d990 0x7f1f9c1507c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1f9aa00700 1 -- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f9c09a750 msgr2=0x7f1f9c0ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1f9aa00700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f9c09a750 0x7f1f9c0ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1f9aa00700 1 -- 172.21.15.12:0/1377442602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c0094a0 con 0x7f1f9c099a60 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1f9aa00700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c099a60 0x7f1f9c15f7d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009fd0 tx=0x7f1f8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1f92a00700 1 -- 172.21.15.12:0/1377442602 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c004590 con 0x7f1f9c099a60 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1f92a00700 1 -- 172.21.15.12:0/1377442602 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c01e070 con 0x7f1f9c099a60 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1377442602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9c1603e0 con 0x7f1f9c099a60 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.180+0000 7f1f92a00700 1 -- 172.21.15.12:0/1377442602 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c0038e0 con 0x7f1f9c099a60 2024-03-29T09:55:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.181+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1377442602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9c0ca810 con 0x7f1f9c099a60 2024-03-29T09:55:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.182+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1377442602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9c15b8b0 con 0x7f1f9c099a60 2024-03-29T09:55:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.182+0000 7f1f92a00700 1 -- 172.21.15.12:0/1377442602 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1f8c003a40 con 0x7f1f9c099a60 2024-03-29T09:55:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.183+0000 7f1f92a00700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1f7c06ac90 0x7f1f7c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.183+0000 7f1f92a00700 1 -- 172.21.15.12:0/1377442602 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f1f8c020070 con 0x7f1f9c099a60 2024-03-29T09:55:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.184+0000 7f1f9a000700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1f7c06ac90 0x7f1f7c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.185+0000 7f1f9a000700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1f7c06ac90 0x7f1f7c06d150 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0b3550 tx=0x7f1f88007dc0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.188+0000 7f1f92a00700 1 -- 172.21.15.12:0/1377442602 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f8c0556a0 con 0x7f1f9c099a60 2024-03-29T09:55:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.405+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1377442602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f9c0b3b60 con 0x7f1f9c099a60 2024-03-29T09:55:11.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.405+0000 7f1f92a00700 1 -- 172.21.15.12:0/1377442602 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f8c058cc0 con 0x7f1f9c099a60 2024-03-29T09:55:11.405 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:55:11.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.407+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1f7c06ac90 msgr2=0x7f1f7c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.407+0000 7f1fa1c00700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1f7c06ac90 0x7f1f7c06d150 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0b3550 tx=0x7f1f88007dc0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.407+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1377442602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c099a60 msgr2=0x7f1f9c15f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.407+0000 7f1fa1c00700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c099a60 0x7f1f9c15f7d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009fd0 tx=0x7f1f8c004750 comp rx=0 tx=0).stop 2024-03-29T09:55:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.407+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1377442602 shutdown_connections 2024-03-29T09:55:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.407+0000 7f1fa1c00700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1f7c06ac90 0x7f1f7c06d150 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.407+0000 7f1fa1c00700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f9c15d990 0x7f1f9c1507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.407+0000 7f1fa1c00700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c099a60 0x7f1f9c15f7d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.407+0000 7f1fa1c00700 1 --2- 172.21.15.12:0/1377442602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f9c09a750 0x7f1f9c0ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.408+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1377442602 >> 172.21.15.12:0/1377442602 conn(0x7f1f9c0b2e20 msgr2=0x7f1f9c15cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.408+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1377442602 shutdown_connections 2024-03-29T09:55:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.408+0000 7f1fa1c00700 1 -- 172.21.15.12:0/1377442602 wait complete. 2024-03-29T09:55:11.425 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:55:11.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.626+0000 7fe5a3c00700 1 Processor -- start 2024-03-29T09:55:11.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.627+0000 7fe5a3c00700 1 -- start start 2024-03-29T09:55:11.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.628+0000 7fe5a3c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe59c0d16a0 0x7fe59c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.628+0000 7fe5a3c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59c161420 con 0x7fe59c15d9d0 2024-03-29T09:55:11.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.628+0000 7fe5a3c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59c161700 con 0x7fe59c0d2370 2024-03-29T09:55:11.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.628+0000 7fe5a3c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59c0d4340 con 0x7fe59c0d16a0 2024-03-29T09:55:11.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.628+0000 7fe5a1200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe59c15d9d0 0x7fe59c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54462/0 (socket says 172.21.15.12:54462) 2024-03-29T09:55:11.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.628+0000 7fe5a1200700 1 -- 172.21.15.12:0/1714200484 learned_addr learned my addr 172.21.15.12:0/1714200484 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:11.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.628+0000 7fe59be00700 1 --2- 172.21.15.12:0/1714200484 >> v2:172.21.15.159:3300/0 conn(0x7fe59c0d16a0 0x7fe59c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.628+0000 7fe59b400700 1 -- 172.21.15.12:0/1714200484 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161899774 0 0) 0x7fe59c161420 con 0x7fe59c15d9d0 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.629+0000 7fe59b400700 1 -- 172.21.15.12:0/1714200484 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5840035b0 con 0x7fe59c15d9d0 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.629+0000 7fe59b400700 1 -- 172.21.15.12:0/1714200484 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188527140 0 0) 0x7fe59c161700 con 0x7fe59c0d2370 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.629+0000 7fe59b400700 1 -- 172.21.15.12:0/1714200484 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe59c161420 con 0x7fe59c0d2370 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.629+0000 7fe59be00700 1 -- 172.21.15.12:0/1714200484 >> v1:172.21.15.159:6789/0 conn(0x7fe59c0d2370 legacy=0x7fe59c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.629+0000 7fe59be00700 1 -- 172.21.15.12:0/1714200484 >> v1:172.21.15.12:6789/0 conn(0x7fe59c15d9d0 legacy=0x7fe59c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.629+0000 7fe59b400700 1 -- 172.21.15.12:0/1714200484 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3788781916 0 0) 0x7fe5840035b0 con 0x7fe59c15d9d0 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.629+0000 7fe59be00700 1 -- 172.21.15.12:0/1714200484 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c1619e0 con 0x7fe59c0d16a0 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.629+0000 7fe59be00700 1 --2- 172.21.15.12:0/1714200484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59c0d16a0 0x7fe59c0d1e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe588009820 tx=0x7fe588009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9dd38832de0cee server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.629+0000 7fe59b400700 1 -- 172.21.15.12:0/1714200484 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590001670 con 0x7fe59c0d16a0 2024-03-29T09:55:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.630+0000 7fe59b400700 1 -- 172.21.15.12:0/1714200484 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe588005c30 con 0x7fe59c0d16a0 2024-03-29T09:55:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.630+0000 7fe59b400700 1 -- 172.21.15.12:0/1714200484 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe588004030 con 0x7fe59c0d16a0 2024-03-29T09:55:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.630+0000 7fe5a3c00700 1 -- 172.21.15.12:0/1714200484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59c0d16a0 msgr2=0x7fe59c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.630+0000 7fe5a3c00700 1 --2- 172.21.15.12:0/1714200484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59c0d16a0 0x7fe59c0d1e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe588009820 tx=0x7fe588009b30 comp rx=0 tx=0).stop 2024-03-29T09:55:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.630+0000 7fe5a3c00700 1 -- 172.21.15.12:0/1714200484 shutdown_connections 2024-03-29T09:55:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.630+0000 7fe5a3c00700 1 --2- 172.21.15.12:0/1714200484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59c0d16a0 0x7fe59c0d1e30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.630+0000 7fe5a3c00700 1 -- 172.21.15.12:0/1714200484 >> 172.21.15.12:0/1714200484 conn(0x7fe59c08aff0 msgr2=0x7fe59c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.630+0000 7fe5a3c00700 1 -- 172.21.15.12:0/1714200484 shutdown_connections 2024-03-29T09:55:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.630+0000 7fe5a3c00700 1 -- 172.21.15.12:0/1714200484 wait complete. 2024-03-29T09:55:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.632+0000 7fe5a3c00700 1 Processor -- start 2024-03-29T09:55:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.632+0000 7fe5a3c00700 1 -- start start 2024-03-29T09:55:11.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe5a3c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59c0d16a0 0x7fe59c15c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe5a3c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59c0d2370 0x7fe59c0b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe5a3c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe59c15d9d0 0x7fe59c0b8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe5a3c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c15fee0 con 0x7fe59c0d16a0 2024-03-29T09:55:11.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe5a3c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c0cd390 con 0x7fe59c0d2370 2024-03-29T09:55:11.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe5a3c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c0cb0a0 con 0x7fe59c15d9d0 2024-03-29T09:55:11.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe5a1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59c0d16a0 0x7fe59c15c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe5a1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59c0d16a0 0x7fe59c15c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40306/0 (socket says 172.21.15.12:40306) 2024-03-29T09:55:11.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe5a1200700 1 -- 172.21.15.12:0/383963898 learned_addr learned my addr 172.21.15.12:0/383963898 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe59be00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59c0d2370 0x7fe59c0b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.633+0000 7fe5a1c00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe59c15d9d0 0x7fe59c0b8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.634+0000 7fe5a1200700 1 -- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe59c15d9d0 msgr2=0x7fe59c0b8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.634+0000 7fe5a1200700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe59c15d9d0 0x7fe59c0b8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.634+0000 7fe5a1200700 1 -- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59c0d2370 msgr2=0x7fe59c0b80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.634+0000 7fe5a1200700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59c0d2370 0x7fe59c0b80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.634+0000 7fe5a1200700 1 -- 172.21.15.12:0/383963898 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe588009430 con 0x7fe59c0d16a0 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.634+0000 7fe5a1200700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59c0d16a0 0x7fe59c15c160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe59000c6b0 tx=0x7fe59000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.634+0000 7fe598c00700 1 -- 172.21.15.12:0/383963898 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590004c80 con 0x7fe59c0d16a0 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.634+0000 7fe598c00700 1 -- 172.21.15.12:0/383963898 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe590004e00 con 0x7fe59c0d16a0 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.634+0000 7fe598c00700 1 -- 172.21.15.12:0/383963898 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590007310 con 0x7fe59c0d16a0 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.634+0000 7fe5a3c00700 1 -- 172.21.15.12:0/383963898 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59c0cb380 con 0x7fe59c0d16a0 2024-03-29T09:55:11.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.635+0000 7fe5a3c00700 1 -- 172.21.15.12:0/383963898 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59c0cb940 con 0x7fe59c0d16a0 2024-03-29T09:55:11.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.636+0000 7fe5a3c00700 1 -- 172.21.15.12:0/383963898 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59c0d1ca0 con 0x7fe59c0d16a0 2024-03-29T09:55:11.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.637+0000 7fe598c00700 1 -- 172.21.15.12:0/383963898 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe590007470 con 0x7fe59c0d16a0 2024-03-29T09:55:11.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.637+0000 7fe598c00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe57c06ac90 0x7fe57c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:11.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.638+0000 7fe598c00700 1 -- 172.21.15.12:0/383963898 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe5900890f0 con 0x7fe59c0d16a0 2024-03-29T09:55:11.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.638+0000 7fe59be00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe57c06ac90 0x7fe57c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:11.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.639+0000 7fe59be00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe57c06ac90 0x7fe57c06d150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0d1ae0 tx=0x7fe588008070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:11.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.642+0000 7fe598c00700 1 -- 172.21.15.12:0/383963898 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe590054230 con 0x7fe59c0d16a0 2024-03-29T09:55:11.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.854+0000 7fe5a3c00700 1 -- 172.21.15.12:0/383963898 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe59c067720 con 0x7fe59c0d16a0 2024-03-29T09:55:11.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.855+0000 7fe598c00700 1 -- 172.21.15.12:0/383963898 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe590057850 con 0x7fe59c0d16a0 2024-03-29T09:55:11.854 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:55:11.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.857+0000 7fe5a3c00700 1 -- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe57c06ac90 msgr2=0x7fe57c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.857+0000 7fe5a3c00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe57c06ac90 0x7fe57c06d150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0d1ae0 tx=0x7fe588008070 comp rx=0 tx=0).stop 2024-03-29T09:55:11.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.857+0000 7fe5a3c00700 1 -- 172.21.15.12:0/383963898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59c0d16a0 msgr2=0x7fe59c15c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:11.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.857+0000 7fe5a3c00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59c0d16a0 0x7fe59c15c160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe59000c6b0 tx=0x7fe59000c9c0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.857+0000 7fe5a3c00700 1 -- 172.21.15.12:0/383963898 shutdown_connections 2024-03-29T09:55:11.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.857+0000 7fe5a3c00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe57c06ac90 0x7fe57c06d150 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.857+0000 7fe5a3c00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe59c15d9d0 0x7fe59c0b8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.857+0000 7fe5a3c00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59c0d16a0 0x7fe59c15c160 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.857+0000 7fe5a3c00700 1 --2- 172.21.15.12:0/383963898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59c0d2370 0x7fe59c0b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:11.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.857+0000 7fe5a3c00700 1 -- 172.21.15.12:0/383963898 >> 172.21.15.12:0/383963898 conn(0x7fe59c08aff0 msgr2=0x7fe59c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:11.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.858+0000 7fe5a3c00700 1 -- 172.21.15.12:0/383963898 shutdown_connections 2024-03-29T09:55:11.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:11.858+0000 7fe5a3c00700 1 -- 172.21.15.12:0/383963898 wait complete. 2024-03-29T09:55:11.874 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:55:12.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.079+0000 7f8348e00700 1 Processor -- start 2024-03-29T09:55:12.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.079+0000 7f8348e00700 1 -- start start 2024-03-29T09:55:12.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.080+0000 7f8348e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f834415d9d0 0x7f83440d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:12.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.080+0000 7f8348e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8344161420 con 0x7f83440d16a0 2024-03-29T09:55:12.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.080+0000 7f8348e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8344161700 con 0x7f83440d2540 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.080+0000 7f8348e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f834415edc0 con 0x7f834415d9d0 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.080+0000 7f8341e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f834415d9d0 0x7f83440d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.080+0000 7f8341400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f83440d16a0 0x7f83440d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54470/0 (socket says 172.21.15.12:54470) 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.080+0000 7f8341400700 1 -- 172.21.15.12:0/2460268739 learned_addr learned my addr 172.21.15.12:0/2460268739 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.081+0000 7f8341e00700 1 -- 172.21.15.12:0/2460268739 >> v1:172.21.15.159:6789/0 conn(0x7f83440d2540 legacy=0x7f83440ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.081+0000 7f8341e00700 1 -- 172.21.15.12:0/2460268739 >> v1:172.21.15.12:6789/0 conn(0x7f83440d16a0 legacy=0x7f83440d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.081+0000 7f8341e00700 1 -- 172.21.15.12:0/2460268739 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83441619e0 con 0x7f834415d9d0 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.081+0000 7f8340a00700 1 -- 172.21.15.12:0/2460268739 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395276608 0 0) 0x7f8344161420 con 0x7f83440d16a0 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.081+0000 7f8341e00700 1 --2- 172.21.15.12:0/2460268739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 0x7f83440d3c30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8334009840 tx=0x7f8334009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136479cb855eacde server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.081+0000 7f8340a00700 1 -- 172.21.15.12:0/2460268739 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004230 con 0x7f834415d9d0 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.081+0000 7f8340a00700 1 -- 172.21.15.12:0/2460268739 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8334004390 con 0x7f834415d9d0 2024-03-29T09:55:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.081+0000 7f8340a00700 1 -- 172.21.15.12:0/2460268739 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340036a0 con 0x7f834415d9d0 2024-03-29T09:55:12.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.082+0000 7f8348e00700 1 -- 172.21.15.12:0/2460268739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 msgr2=0x7f83440d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.082+0000 7f8348e00700 1 --2- 172.21.15.12:0/2460268739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 0x7f83440d3c30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8334009840 tx=0x7f8334009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:12.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.082+0000 7f8348e00700 1 -- 172.21.15.12:0/2460268739 shutdown_connections 2024-03-29T09:55:12.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.082+0000 7f8348e00700 1 --2- 172.21.15.12:0/2460268739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 0x7f83440d3c30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.082+0000 7f8348e00700 1 -- 172.21.15.12:0/2460268739 >> 172.21.15.12:0/2460268739 conn(0x7f834408aff0 msgr2=0x7f83440d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:12.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.082+0000 7f8348e00700 1 -- 172.21.15.12:0/2460268739 shutdown_connections 2024-03-29T09:55:12.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.082+0000 7f8348e00700 1 -- 172.21.15.12:0/2460268739 wait complete. 2024-03-29T09:55:12.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.084+0000 7f8348e00700 1 Processor -- start 2024-03-29T09:55:12.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.084+0000 7f8348e00700 1 -- start start 2024-03-29T09:55:12.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8348e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d16a0 0x7f834415ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:12.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8348e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440d2540 0x7f83441606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:12.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8348e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 0x7f8344160c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8348e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83440b7d70 con 0x7f834415d9d0 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8348e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83440b7160 con 0x7f83440d2540 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8348e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83440b73d0 con 0x7f83440d16a0 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8342800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 0x7f8344160c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8342800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 0x7f8344160c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40320/0 (socket says 172.21.15.12:40320) 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8342800700 1 -- 172.21.15.12:0/1947347987 learned_addr learned my addr 172.21.15.12:0/1947347987 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8341e00700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d16a0 0x7f834415ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8341400700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440d2540 0x7f83441606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8342800700 1 -- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d16a0 msgr2=0x7f834415ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8342800700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d16a0 0x7f834415ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8342800700 1 -- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440d2540 msgr2=0x7f83441606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8342800700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440d2540 0x7f83441606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.086+0000 7f8342800700 1 -- 172.21.15.12:0/1947347987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83340094a0 con 0x7f834415d9d0 2024-03-29T09:55:12.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.087+0000 7f8342800700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 0x7f8344160c30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f833c00c7b0 tx=0x7f833c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:12.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.087+0000 7f833a000700 1 -- 172.21.15.12:0/1947347987 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c003e10 con 0x7f834415d9d0 2024-03-29T09:55:12.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.087+0000 7f833a000700 1 -- 172.21.15.12:0/1947347987 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c005070 con 0x7f834415d9d0 2024-03-29T09:55:12.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.087+0000 7f833a000700 1 -- 172.21.15.12:0/1947347987 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c005420 con 0x7f834415d9d0 2024-03-29T09:55:12.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.087+0000 7f8348e00700 1 -- 172.21.15.12:0/1947347987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83440b76b0 con 0x7f834415d9d0 2024-03-29T09:55:12.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.087+0000 7f8348e00700 1 -- 172.21.15.12:0/1947347987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834415cf70 con 0x7f834415d9d0 2024-03-29T09:55:12.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.088+0000 7f8348e00700 1 -- 172.21.15.12:0/1947347987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83440d1ca0 con 0x7f834415d9d0 2024-03-29T09:55:12.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.089+0000 7f833a000700 1 -- 172.21.15.12:0/1947347987 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f833c005580 con 0x7f834415d9d0 2024-03-29T09:55:12.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.090+0000 7f833a000700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f832406ac90 0x7f832406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:12.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.090+0000 7f833a000700 1 -- 172.21.15.12:0/1947347987 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f833c0887c0 con 0x7f834415d9d0 2024-03-29T09:55:12.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.090+0000 7f8341e00700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f832406ac90 0x7f832406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:12.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.091+0000 7f8341e00700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f832406ac90 0x7f832406d150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f83340090b0 tx=0x7f8334008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:12.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.094+0000 7f833a000700 1 -- 172.21.15.12:0/1947347987 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f833c0537e0 con 0x7f834415d9d0 2024-03-29T09:55:12.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.314+0000 7f8348e00700 1 -- 172.21.15.12:0/1947347987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f83440b7eb0 con 0x7f834415d9d0 2024-03-29T09:55:12.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.314+0000 7f833a000700 1 -- 172.21.15.12:0/1947347987 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f83440b7eb0 con 0x7f834415d9d0 2024-03-29T09:55:12.314 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 -- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f832406ac90 msgr2=0x7f832406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f832406ac90 0x7f832406d150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f83340090b0 tx=0x7f8334008030 comp rx=0 tx=0).stop 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 -- 172.21.15.12:0/1947347987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 msgr2=0x7f8344160c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 0x7f8344160c30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f833c00c7b0 tx=0x7f833c00cb70 comp rx=0 tx=0).stop 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 -- 172.21.15.12:0/1947347987 shutdown_connections 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f832406ac90 0x7f832406d150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f834415d9d0 0x7f8344160c30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d16a0 0x7f834415ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 --2- 172.21.15.12:0/1947347987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440d2540 0x7f83441606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 -- 172.21.15.12:0/1947347987 >> 172.21.15.12:0/1947347987 conn(0x7f834408aff0 msgr2=0x7f83440c9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 -- 172.21.15.12:0/1947347987 shutdown_connections 2024-03-29T09:55:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.317+0000 7f8348e00700 1 -- 172.21.15.12:0/1947347987 wait complete. 2024-03-29T09:55:12.335 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T09:55:12.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.533+0000 7f67f9600700 1 Processor -- start 2024-03-29T09:55:12.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.533+0000 7f67f9600700 1 -- start start 2024-03-29T09:55:12.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.535+0000 7f67f9600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f40d2540 0x7f67f41687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:12.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.535+0000 7f67f9600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f4161420 con 0x7f67f415d9d0 2024-03-29T09:55:12.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.535+0000 7f67f9600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f4161700 con 0x7f67f40d16a0 2024-03-29T09:55:12.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.535+0000 7f67f9600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f40d4340 con 0x7f67f40d2540 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.535+0000 7f67f2a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f67f415d9d0 0x7f67f40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54474/0 (socket says 172.21.15.12:54474) 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.535+0000 7f67f2a00700 1 -- 172.21.15.12:0/3782534290 learned_addr learned my addr 172.21.15.12:0/3782534290 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.535+0000 7f67f3400700 1 --2- 172.21.15.12:0/3782534290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f40d2540 0x7f67f41687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.535+0000 7f67f1600700 1 -- 172.21.15.12:0/3782534290 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283369573 0 0) 0x7f67f4161420 con 0x7f67f415d9d0 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.535+0000 7f67f1600700 1 -- 172.21.15.12:0/3782534290 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d80035b0 con 0x7f67f415d9d0 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.536+0000 7f67f1600700 1 -- 172.21.15.12:0/3782534290 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199544595 0 0) 0x7f67f4161700 con 0x7f67f40d16a0 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.536+0000 7f67f1600700 1 -- 172.21.15.12:0/3782534290 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f4161420 con 0x7f67f40d16a0 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.536+0000 7f67f1600700 1 -- 172.21.15.12:0/3782534290 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1254430833 0 0) 0x7f67d80035b0 con 0x7f67f415d9d0 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.536+0000 7f67f3400700 1 -- 172.21.15.12:0/3782534290 >> v1:172.21.15.159:6789/0 conn(0x7f67f40d16a0 legacy=0x7f67f40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.536+0000 7f67f3400700 1 -- 172.21.15.12:0/3782534290 >> v1:172.21.15.12:6789/0 conn(0x7f67f415d9d0 legacy=0x7f67f40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.536+0000 7f67f3400700 1 -- 172.21.15.12:0/3782534290 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f41619e0 con 0x7f67f40d2540 2024-03-29T09:55:12.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.536+0000 7f67f3400700 1 --2- 172.21.15.12:0/3782534290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f40d2540 0x7f67f41687b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009dc0 tx=0x7f67ec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f38d39b4ab5dd7a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:12.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.536+0000 7f67f1600700 1 -- 172.21.15.12:0/3782534290 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec00e070 con 0x7f67f40d2540 2024-03-29T09:55:12.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.537+0000 7f67f1600700 1 -- 172.21.15.12:0/3782534290 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ec003a20 con 0x7f67f40d2540 2024-03-29T09:55:12.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.537+0000 7f67f1600700 1 -- 172.21.15.12:0/3782534290 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e4001670 con 0x7f67f40d2540 2024-03-29T09:55:12.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.537+0000 7f67f9600700 1 -- 172.21.15.12:0/3782534290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f40d2540 msgr2=0x7f67f41687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.537+0000 7f67f9600700 1 --2- 172.21.15.12:0/3782534290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f40d2540 0x7f67f41687b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009dc0 tx=0x7f67ec00b330 comp rx=0 tx=0).stop 2024-03-29T09:55:12.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.537+0000 7f67f9600700 1 -- 172.21.15.12:0/3782534290 shutdown_connections 2024-03-29T09:55:12.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.537+0000 7f67f9600700 1 --2- 172.21.15.12:0/3782534290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f40d2540 0x7f67f41687b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.537+0000 7f67f9600700 1 -- 172.21.15.12:0/3782534290 >> 172.21.15.12:0/3782534290 conn(0x7f67f408aff0 msgr2=0x7f67f40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:12.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.537+0000 7f67f9600700 1 -- 172.21.15.12:0/3782534290 shutdown_connections 2024-03-29T09:55:12.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.537+0000 7f67f9600700 1 -- 172.21.15.12:0/3782534290 wait complete. 2024-03-29T09:55:12.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.539+0000 7f67f9600700 1 Processor -- start 2024-03-29T09:55:12.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.539+0000 7f67f9600700 1 -- start start 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.540+0000 7f67f9600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f40d16a0 0x7f67f4153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.540+0000 7f67f9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40d2540 0x7f67f40b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.540+0000 7f67f9600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f415d9d0 0x7f67f40ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.540+0000 7f67f9600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f4154320 con 0x7f67f40d2540 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.540+0000 7f67f9600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f41e8930 con 0x7f67f40d16a0 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.540+0000 7f67f9600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f41e8c10 con 0x7f67f415d9d0 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.540+0000 7f67f2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40d2540 0x7f67f40b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.540+0000 7f67f2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40d2540 0x7f67f40b7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40334/0 (socket says 172.21.15.12:40334) 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.540+0000 7f67f2000700 1 -- 172.21.15.12:0/2080980363 learned_addr learned my addr 172.21.15.12:0/2080980363 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67f2a00700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f40d16a0 0x7f67f4153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:12.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67f3400700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f415d9d0 0x7f67f40ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67f2000700 1 -- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f415d9d0 msgr2=0x7f67f40ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67f2000700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f415d9d0 0x7f67f40ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67f2000700 1 -- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f40d16a0 msgr2=0x7f67f4153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67f2000700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f40d16a0 0x7f67f4153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67f2000700 1 -- 172.21.15.12:0/2080980363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec009a70 con 0x7f67f40d2540 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67f2000700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40d2540 0x7f67f40b7de0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67e000c4f0 tx=0x7f67e000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67eaa00700 1 -- 172.21.15.12:0/2080980363 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e00052d0 con 0x7f67f40d2540 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67eaa00700 1 -- 172.21.15.12:0/2080980363 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e0005430 con 0x7f67f40d2540 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67f9600700 1 -- 172.21.15.12:0/2080980363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f41e8ef0 con 0x7f67f40d2540 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67eaa00700 1 -- 172.21.15.12:0/2080980363 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e00038f0 con 0x7f67f40d2540 2024-03-29T09:55:12.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.541+0000 7f67f9600700 1 -- 172.21.15.12:0/2080980363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f41e94b0 con 0x7f67f40d2540 2024-03-29T09:55:12.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.542+0000 7f67f9600700 1 -- 172.21.15.12:0/2080980363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f40d1ca0 con 0x7f67f40d2540 2024-03-29T09:55:12.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.543+0000 7f67eaa00700 1 -- 172.21.15.12:0/2080980363 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f67e0003a50 con 0x7f67f40d2540 2024-03-29T09:55:12.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.544+0000 7f67eaa00700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67d406ac90 0x7f67d406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:12.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.544+0000 7f67eaa00700 1 -- 172.21.15.12:0/2080980363 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f67e0088980 con 0x7f67f40d2540 2024-03-29T09:55:12.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.545+0000 7f67f2a00700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67d406ac90 0x7f67d406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:12.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.545+0000 7f67f2a00700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67d406ac90 0x7f67d406d150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f67e4007fd0 tx=0x7f67e4007a60 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:12.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.549+0000 7f67eaa00700 1 -- 172.21.15.12:0/2080980363 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67e0053ac0 con 0x7f67f40d2540 2024-03-29T09:55:12.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.778+0000 7f67f9600700 1 -- 172.21.15.12:0/2080980363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f67f4067720 con 0x7f67f40d2540 2024-03-29T09:55:12.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.778+0000 7f67eaa00700 1 -- 172.21.15.12:0/2080980363 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f67e0017080 con 0x7f67f40d2540 2024-03-29T09:55:12.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 -- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67d406ac90 msgr2=0x7f67d406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67d406ac90 0x7f67d406d150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f67e4007fd0 tx=0x7f67e4007a60 comp rx=0 tx=0).stop 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 -- 172.21.15.12:0/2080980363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40d2540 msgr2=0x7f67f40b7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40d2540 0x7f67f40b7de0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67e000c4f0 tx=0x7f67e000c8b0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 -- 172.21.15.12:0/2080980363 shutdown_connections 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67d406ac90 0x7f67d406d150 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67f415d9d0 0x7f67f40ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f40d16a0 0x7f67f4153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 --2- 172.21.15.12:0/2080980363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40d2540 0x7f67f40b7de0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 -- 172.21.15.12:0/2080980363 >> 172.21.15.12:0/2080980363 conn(0x7f67f408aff0 msgr2=0x7f67f40cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 -- 172.21.15.12:0/2080980363 shutdown_connections 2024-03-29T09:55:12.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:12.781+0000 7f67f9600700 1 -- 172.21.15.12:0/2080980363 wait complete. 2024-03-29T09:55:12.782 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-29T09:55:12.798 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-29T09:55:12.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:12.799 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-29T09:55:12.879 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T09:55:12.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:12.879 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-29T09:55:12.985 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:55:12.985 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:13.030 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:55:13.030 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:13.115 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:55:13.115 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:13.200 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.438+0000 7fc1f2a00700 1 Processor -- start 2024-03-29T09:55:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.439+0000 7fc1f2a00700 1 -- start start 2024-03-29T09:55:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1f2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec160f10 0x7fc1ec0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1f2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1ec0cf230 0x7fc1ec0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1f2a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ec0ceda0 con 0x7fc1ec161590 2024-03-29T09:55:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1f2a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ec0d23d0 con 0x7fc1ec0cf230 2024-03-29T09:55:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1f2a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec1612f0 con 0x7fc1ec160f10 2024-03-29T09:55:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1eb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1ec0cf230 0x7fc1ec0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1f0a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc1ec161590 0x7fc1ec0b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54480/0 (socket says 172.21.15.12:54480) 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1f0a00700 1 -- 172.21.15.12:0/4079546560 learned_addr learned my addr 172.21.15.12:0/4079546560 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1ebe00700 1 --2- 172.21.15.12:0/4079546560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec160f10 0x7fc1ec0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1eb400700 1 -- 172.21.15.12:0/4079546560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec160f10 msgr2=0x7fc1ec0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1eb400700 1 --2- 172.21.15.12:0/4079546560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec160f10 0x7fc1ec0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1eb400700 1 -- 172.21.15.12:0/4079546560 >> v1:172.21.15.12:6789/0 conn(0x7fc1ec161590 legacy=0x7fc1ec0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.441+0000 7fc1eb400700 1 -- 172.21.15.12:0/4079546560 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec0ce470 con 0x7fc1ec0cf230 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.441+0000 7fc1ebe00700 1 --2- 172.21.15.12:0/4079546560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec160f10 0x7fc1ec0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.440+0000 7fc1eaa00700 1 -- 172.21.15.12:0/4079546560 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483113352 0 0) 0x7fc1ec0ceda0 con 0x7fc1ec161590 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.441+0000 7fc1eb400700 1 --2- 172.21.15.12:0/4079546560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 0x7fc1ec0d1e90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009840 tx=0x7fc1d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5935a46cbca6b09d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.441+0000 7fc1eaa00700 1 -- 172.21.15.12:0/4079546560 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8011830 con 0x7fc1ec0cf230 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.441+0000 7fc1eaa00700 1 -- 172.21.15.12:0/4079546560 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d8011990 con 0x7fc1ec0cf230 2024-03-29T09:55:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.441+0000 7fc1eaa00700 1 -- 172.21.15.12:0/4079546560 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8011d20 con 0x7fc1ec0cf230 2024-03-29T09:55:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.441+0000 7fc1f2a00700 1 -- 172.21.15.12:0/4079546560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 msgr2=0x7fc1ec0d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.441+0000 7fc1f2a00700 1 --2- 172.21.15.12:0/4079546560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 0x7fc1ec0d1e90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009840 tx=0x7fc1d8009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.442+0000 7fc1f2a00700 1 -- 172.21.15.12:0/4079546560 shutdown_connections 2024-03-29T09:55:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.442+0000 7fc1f2a00700 1 --2- 172.21.15.12:0/4079546560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec160f10 0x7fc1ec0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.442+0000 7fc1f2a00700 1 --2- 172.21.15.12:0/4079546560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 0x7fc1ec0d1e90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.442+0000 7fc1f2a00700 1 -- 172.21.15.12:0/4079546560 >> 172.21.15.12:0/4079546560 conn(0x7fc1ec08aff0 msgr2=0x7fc1ec0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.442+0000 7fc1f2a00700 1 -- 172.21.15.12:0/4079546560 shutdown_connections 2024-03-29T09:55:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.442+0000 7fc1f2a00700 1 -- 172.21.15.12:0/4079546560 wait complete. 2024-03-29T09:55:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.443+0000 7fc1f2a00700 1 Processor -- start 2024-03-29T09:55:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.443+0000 7fc1f2a00700 1 -- start start 2024-03-29T09:55:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.444+0000 7fc1f2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 0x7fc1ec0c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.444+0000 7fc1f2a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1ec160f10 0x7fc1ec0b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.444+0000 7fc1f2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec161590 0x7fc1ec0b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.444+0000 7fc1f2a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec0b8dc0 con 0x7fc1ec0cf230 2024-03-29T09:55:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.444+0000 7fc1f2a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec0b7e90 con 0x7fc1ec160f10 2024-03-29T09:55:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.444+0000 7fc1f2a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec0b8140 con 0x7fc1ec161590 2024-03-29T09:55:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 0x7fc1ec0c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 0x7fc1ec0c9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40350/0 (socket says 172.21.15.12:40350) 2024-03-29T09:55:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1ebe00700 1 -- 172.21.15.12:0/3316487040 learned_addr learned my addr 172.21.15.12:0/3316487040 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1f0a00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec161590 0x7fc1ec0b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1eb400700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1ec160f10 0x7fc1ec0b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1ebe00700 1 -- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec161590 msgr2=0x7fc1ec0b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1ebe00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec161590 0x7fc1ec0b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1ebe00700 1 -- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1ec160f10 msgr2=0x7fc1ec0b7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1ebe00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1ec160f10 0x7fc1ec0b7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1ebe00700 1 -- 172.21.15.12:0/3316487040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d80094a0 con 0x7fc1ec0cf230 2024-03-29T09:55:13.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.445+0000 7fc1ebe00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 0x7fc1ec0c9320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc00e760 tx=0x7fc1dc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:13.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.446+0000 7fc1e3e00700 1 -- 172.21.15.12:0/3316487040 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc013610 con 0x7fc1ec0cf230 2024-03-29T09:55:13.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.446+0000 7fc1e3e00700 1 -- 172.21.15.12:0/3316487040 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc013770 con 0x7fc1ec0cf230 2024-03-29T09:55:13.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.446+0000 7fc1e3e00700 1 -- 172.21.15.12:0/3316487040 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc013b20 con 0x7fc1ec0cf230 2024-03-29T09:55:13.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.446+0000 7fc1f2a00700 1 -- 172.21.15.12:0/3316487040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ec0b8420 con 0x7fc1ec0cf230 2024-03-29T09:55:13.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.446+0000 7fc1f2a00700 1 -- 172.21.15.12:0/3316487040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ec0b89e0 con 0x7fc1ec0cf230 2024-03-29T09:55:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.447+0000 7fc1f2a00700 1 -- 172.21.15.12:0/3316487040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ec156740 con 0x7fc1ec0cf230 2024-03-29T09:55:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.448+0000 7fc1e3e00700 1 -- 172.21.15.12:0/3316487040 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc1dc013c80 con 0x7fc1ec0cf230 2024-03-29T09:55:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.448+0000 7fc1e3e00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1cc06ac90 0x7fc1cc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.449+0000 7fc1e3e00700 1 -- 172.21.15.12:0/3316487040 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc1dc088fb0 con 0x7fc1ec0cf230 2024-03-29T09:55:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.449+0000 7fc1eb400700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1cc06ac90 0x7fc1cc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.450+0000 7fc1eb400700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1cc06ac90 0x7fc1cc06d150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80090b0 tx=0x7fc1d801c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:13.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.453+0000 7fc1e3e00700 1 -- 172.21.15.12:0/3316487040 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc1dc0540f0 con 0x7fc1ec0cf230 2024-03-29T09:55:13.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.668+0000 7fc1f2a00700 1 -- 172.21.15.12:0/3316487040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1ec067720 con 0x7fc1ec0cf230 2024-03-29T09:55:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.669+0000 7fc1e3e00700 1 -- 172.21.15.12:0/3316487040 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16333 (secure 0 0 0) 0x7fc1dc057710 con 0x7fc1ec0cf230 2024-03-29T09:55:13.669 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:13.669 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:55:12.039728+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 -- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1cc06ac90 msgr2=0x7fc1cc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1cc06ac90 0x7fc1cc06d150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80090b0 tx=0x7fc1d801c040 comp rx=0 tx=0).stop 2024-03-29T09:55:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 -- 172.21.15.12:0/3316487040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 msgr2=0x7fc1ec0c9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 0x7fc1ec0c9320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc00e760 tx=0x7fc1dc00eb20 comp rx=0 tx=0).stop 2024-03-29T09:55:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 -- 172.21.15.12:0/3316487040 shutdown_connections 2024-03-29T09:55:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc1cc06ac90 0x7fc1cc06d150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1ec161590 0x7fc1ec0b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1ec0cf230 0x7fc1ec0c9320 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 --2- 172.21.15.12:0/3316487040 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1ec160f10 0x7fc1ec0b7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 -- 172.21.15.12:0/3316487040 >> 172.21.15.12:0/3316487040 conn(0x7fc1ec08aff0 msgr2=0x7fc1ec06c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 -- 172.21.15.12:0/3316487040 shutdown_connections 2024-03-29T09:55:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.672+0000 7fc1f2a00700 1 -- 172.21.15.12:0/3316487040 wait complete. 2024-03-29T09:55:13.690 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:55:13.691 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:55:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.892+0000 7f6bcf600700 1 Processor -- start 2024-03-29T09:55:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.892+0000 7f6bcf600700 1 -- start start 2024-03-29T09:55:13.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.893+0000 7f6bcf600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6bc815d9d0 0x7f6bc80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.893+0000 7f6bcf600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc80d16a0 0x7f6bc80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.893+0000 7f6bcf600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc8161420 con 0x7f6bc80d2370 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.893+0000 7f6bcf600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc80d2820 con 0x7f6bc815d9d0 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.893+0000 7f6bcf600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc8161700 con 0x7f6bc80d16a0 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.893+0000 7f6bccc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6bc815d9d0 0x7f6bc80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.893+0000 7f6bccc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6bc815d9d0 0x7f6bc80d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40360/0 (socket says 172.21.15.12:40360) 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.893+0000 7f6bccc00700 1 -- 172.21.15.12:0/1406566659 learned_addr learned my addr 172.21.15.12:0/1406566659 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.894+0000 7f6bc7e00700 1 --2- 172.21.15.12:0/1406566659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc80d16a0 0x7f6bc80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.894+0000 7f6bccc00700 1 -- 172.21.15.12:0/1406566659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc80d16a0 msgr2=0x7f6bc80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.894+0000 7f6bccc00700 1 --2- 172.21.15.12:0/1406566659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc80d16a0 0x7f6bc80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.894+0000 7f6bccc00700 1 -- 172.21.15.12:0/1406566659 >> v1:172.21.15.159:6789/0 conn(0x7f6bc80d2370 legacy=0x7f6bc815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.894+0000 7f6bccc00700 1 -- 172.21.15.12:0/1406566659 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc81619e0 con 0x7f6bc815d9d0 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.894+0000 7f6bccc00700 1 --2- 172.21.15.12:0/1406566659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 0x7f6bc80d3c30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0098f0 tx=0x7f6bbc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a361d09348043845 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.894+0000 7f6bc7400700 1 -- 172.21.15.12:0/1406566659 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc004230 con 0x7f6bc815d9d0 2024-03-29T09:55:13.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.894+0000 7f6bc7400700 1 -- 172.21.15.12:0/1406566659 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc004390 con 0x7f6bc815d9d0 2024-03-29T09:55:13.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.894+0000 7f6bc7400700 1 -- 172.21.15.12:0/1406566659 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc0036a0 con 0x7f6bc815d9d0 2024-03-29T09:55:13.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.895+0000 7f6bcf600700 1 -- 172.21.15.12:0/1406566659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 msgr2=0x7f6bc80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:13.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.895+0000 7f6bcf600700 1 --2- 172.21.15.12:0/1406566659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 0x7f6bc80d3c30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0098f0 tx=0x7f6bbc009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:13.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.895+0000 7f6bcf600700 1 -- 172.21.15.12:0/1406566659 shutdown_connections 2024-03-29T09:55:13.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.895+0000 7f6bcf600700 1 --2- 172.21.15.12:0/1406566659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 0x7f6bc80d3c30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.895+0000 7f6bcf600700 1 --2- 172.21.15.12:0/1406566659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc80d16a0 0x7f6bc80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.895+0000 7f6bcf600700 1 -- 172.21.15.12:0/1406566659 >> 172.21.15.12:0/1406566659 conn(0x7f6bc808aff0 msgr2=0x7f6bc80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:13.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.895+0000 7f6bcf600700 1 -- 172.21.15.12:0/1406566659 shutdown_connections 2024-03-29T09:55:13.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.895+0000 7f6bcf600700 1 -- 172.21.15.12:0/1406566659 wait complete. 2024-03-29T09:55:13.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.897+0000 7f6bcf600700 1 Processor -- start 2024-03-29T09:55:13.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.897+0000 7f6bcf600700 1 -- start start 2024-03-29T09:55:13.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bcf600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc80d16a0 0x7f6bc81601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bcf600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc80d2370 0x7f6bc80b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bcf600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 0x7f6bc8158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bcf600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc80b74f0 con 0x7f6bc815d9d0 2024-03-29T09:55:13.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bcf600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc80b6870 con 0x7f6bc80d2370 2024-03-29T09:55:13.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bcf600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc80b6b50 con 0x7f6bc80d16a0 2024-03-29T09:55:13.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bcd600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 0x7f6bc8158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bcd600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 0x7f6bc8158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40368/0 (socket says 172.21.15.12:40368) 2024-03-29T09:55:13.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bcd600700 1 -- 172.21.15.12:0/61964580 learned_addr learned my addr 172.21.15.12:0/61964580 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:13.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bc7e00700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc80d2370 0x7f6bc80b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.898+0000 7f6bccc00700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc80d16a0 0x7f6bc81601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bcd600700 1 -- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc80d16a0 msgr2=0x7f6bc81601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:13.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bcd600700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc80d16a0 0x7f6bc81601a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bcd600700 1 -- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc80d2370 msgr2=0x7f6bc80b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:13.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bcd600700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc80d2370 0x7f6bc80b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:13.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bcd600700 1 -- 172.21.15.12:0/61964580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc009550 con 0x7f6bc815d9d0 2024-03-29T09:55:13.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bcd600700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 0x7f6bc8158ca0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6bc000cc10 tx=0x7f6bc000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:13.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bc4c00700 1 -- 172.21.15.12:0/61964580 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc000aa60 con 0x7f6bc815d9d0 2024-03-29T09:55:13.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bc4c00700 1 -- 172.21.15.12:0/61964580 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc000abc0 con 0x7f6bc815d9d0 2024-03-29T09:55:13.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bc4c00700 1 -- 172.21.15.12:0/61964580 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0012420 con 0x7f6bc815d9d0 2024-03-29T09:55:13.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bcf600700 1 -- 172.21.15.12:0/61964580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc80b6e30 con 0x7f6bc815d9d0 2024-03-29T09:55:13.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.899+0000 7f6bcf600700 1 -- 172.21.15.12:0/61964580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc80b6560 con 0x7f6bc815d9d0 2024-03-29T09:55:13.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.900+0000 7f6bcf600700 1 -- 172.21.15.12:0/61964580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc80d1ca0 con 0x7f6bc815d9d0 2024-03-29T09:55:13.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.901+0000 7f6bc4c00700 1 -- 172.21.15.12:0/61964580 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6bc0012580 con 0x7f6bc815d9d0 2024-03-29T09:55:13.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.902+0000 7f6bc4c00700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ba806ac90 0x7f6ba806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:13.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.902+0000 7f6bc4c00700 1 -- 172.21.15.12:0/61964580 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6bc0089780 con 0x7f6bc815d9d0 2024-03-29T09:55:13.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.903+0000 7f6bccc00700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ba806ac90 0x7f6ba806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:13.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.903+0000 7f6bccc00700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ba806ac90 0x7f6ba806d150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc009130 tx=0x7f6bbc008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:13.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:13.907+0000 7f6bc4c00700 1 -- 172.21.15.12:0/61964580 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6bc0054830 con 0x7f6bc815d9d0 2024-03-29T09:55:14.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.119+0000 7f6bcf600700 1 -- 172.21.15.12:0/61964580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6bc80b7630 con 0x7f6bc815d9d0 2024-03-29T09:55:14.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.119+0000 7f6bc4c00700 1 -- 172.21.15.12:0/61964580 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6bc80b7630 con 0x7f6bc815d9d0 2024-03-29T09:55:14.119 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:55:14.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 -- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ba806ac90 msgr2=0x7f6ba806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ba806ac90 0x7f6ba806d150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc009130 tx=0x7f6bbc008030 comp rx=0 tx=0).stop 2024-03-29T09:55:14.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 -- 172.21.15.12:0/61964580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 msgr2=0x7f6bc8158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 0x7f6bc8158ca0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6bc000cc10 tx=0x7f6bc000cf20 comp rx=0 tx=0).stop 2024-03-29T09:55:14.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 -- 172.21.15.12:0/61964580 shutdown_connections 2024-03-29T09:55:14.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ba806ac90 0x7f6ba806d150 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bc815d9d0 0x7f6bc8158ca0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc80d16a0 0x7f6bc81601a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 --2- 172.21.15.12:0/61964580 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc80d2370 0x7f6bc80b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 -- 172.21.15.12:0/61964580 >> 172.21.15.12:0/61964580 conn(0x7f6bc808aff0 msgr2=0x7f6bc80b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:14.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 -- 172.21.15.12:0/61964580 shutdown_connections 2024-03-29T09:55:14.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.122+0000 7f6bcf600700 1 -- 172.21.15.12:0/61964580 wait complete. 2024-03-29T09:55:14.139 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:55:14.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.341+0000 7f619e000700 1 Processor -- start 2024-03-29T09:55:14.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.341+0000 7f619e000700 1 -- start start 2024-03-29T09:55:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.342+0000 7f619e000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f619815d9d0 0x7f61980d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.342+0000 7f619e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61980d2540 0x7f61980ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.342+0000 7f619e000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6198161420 con 0x7f61980d16a0 2024-03-29T09:55:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.342+0000 7f619e000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619815edc0 con 0x7f619815d9d0 2024-03-29T09:55:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.342+0000 7f619e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6198161700 con 0x7f61980d2540 2024-03-29T09:55:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f61980d16a0 0x7f61980d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54488/0 (socket says 172.21.15.12:54488) 2024-03-29T09:55:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196a00700 1 -- 172.21.15.12:0/66754616 learned_addr learned my addr 172.21.15.12:0/66754616 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6197400700 1 --2- 172.21.15.12:0/66754616 >> v2:172.21.15.159:3300/0 conn(0x7f619815d9d0 0x7f61980d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6197e00700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61980d2540 0x7f61980ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249653657 0 0) 0x7f6198161420 con 0x7f61980d16a0 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617c0035b0 con 0x7f61980d16a0 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822895882 0 0) 0x7f617c0035b0 con 0x7f61980d16a0 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61980d2540 msgr2=0x7f61980ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196000700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61980d2540 0x7f61980ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 >> v2:172.21.15.159:3300/0 conn(0x7f619815d9d0 msgr2=0x7f61980d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196000700 1 --2- 172.21.15.12:0/66754616 >> v2:172.21.15.159:3300/0 conn(0x7f619815d9d0 0x7f61980d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61981619e0 con 0x7f61980d16a0 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.343+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f61840017f0 con 0x7f61980d16a0 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 >> v1:172.21.15.12:6789/0 conn(0x7f61980d16a0 legacy=0x7f61980d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196000700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f617c003a00 0x7f617c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196000700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617c004850 0x7f617c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196000700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61980d2540 0x7f617c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6198161700 con 0x7f617c004850 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c00ba10 con 0x7f617c003a00 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617c00bb50 con 0x7f61980d2540 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196a00700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617c004850 0x7f617c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6197e00700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61980d2540 0x7f617c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6197400700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f617c003a00 0x7f617c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196a00700 1 -- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61980d2540 msgr2=0x7f617c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196a00700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61980d2540 0x7f617c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196a00700 1 -- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f617c003a00 msgr2=0x7f617c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196a00700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f617c003a00 0x7f617c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.344+0000 7f6196a00700 1 -- 172.21.15.12:0/66754616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c0035b0 con 0x7f617c004850 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.345+0000 7f6196a00700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617c004850 0x7f617c004cd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f618400c2f0 tx=0x7f617c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93f6423fc94299d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.345+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618400ce30 con 0x7f617c004850 2024-03-29T09:55:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.345+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6184007bf0 con 0x7f617c004850 2024-03-29T09:55:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.345+0000 7f6196000700 1 -- 172.21.15.12:0/66754616 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184007310 con 0x7f617c004850 2024-03-29T09:55:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.345+0000 7f619e000700 1 -- 172.21.15.12:0/66754616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617c004850 msgr2=0x7f617c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.345+0000 7f619e000700 1 --2- 172.21.15.12:0/66754616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617c004850 0x7f617c004cd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f618400c2f0 tx=0x7f617c005880 comp rx=0 tx=0).stop 2024-03-29T09:55:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.345+0000 7f6197e00700 1 -- 172.21.15.12:0/66754616 reap_dead start 2024-03-29T09:55:14.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.345+0000 7f619e000700 1 -- 172.21.15.12:0/66754616 shutdown_connections 2024-03-29T09:55:14.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.345+0000 7f619e000700 1 -- 172.21.15.12:0/66754616 >> 172.21.15.12:0/66754616 conn(0x7f619808aff0 msgr2=0x7f61980d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:14.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.346+0000 7f619e000700 1 -- 172.21.15.12:0/66754616 shutdown_connections 2024-03-29T09:55:14.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.346+0000 7f619e000700 1 -- 172.21.15.12:0/66754616 wait complete. 2024-03-29T09:55:14.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.347+0000 7f619e000700 1 Processor -- start 2024-03-29T09:55:14.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.348+0000 7f619e000700 1 -- start start 2024-03-29T09:55:14.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.348+0000 7f619e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f617c004850 0x7f6198155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.348+0000 7f619e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f619815f5d0 0x7f619815fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f619e000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61980b4ce0 0x7f61980b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f619e000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980ccf70 con 0x7f619815f5d0 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f619e000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980b56a0 con 0x7f61980b4ce0 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f619e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61980cb080 con 0x7f617c004850 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6196a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f619815f5d0 0x7f619815fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6196a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f619815f5d0 0x7f619815fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40398/0 (socket says 172.21.15.12:40398) 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6196a00700 1 -- 172.21.15.12:0/1282163250 learned_addr learned my addr 172.21.15.12:0/1282163250 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6197400700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f617c004850 0x7f6198155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6197e00700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61980b4ce0 0x7f61980b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6196a00700 1 -- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f617c004850 msgr2=0x7f6198155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6196a00700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f617c004850 0x7f6198155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6196a00700 1 -- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61980b4ce0 msgr2=0x7f61980b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6196a00700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61980b4ce0 0x7f61980b5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6196a00700 1 -- 172.21.15.12:0/1282163250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61981558a0 con 0x7f619815f5d0 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.349+0000 7f6196a00700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f619815f5d0 0x7f619815fa50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6184015040 tx=0x7f6184001bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.350+0000 7f618f400700 1 -- 172.21.15.12:0/1282163250 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61840072d0 con 0x7f619815f5d0 2024-03-29T09:55:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.350+0000 7f618f400700 1 -- 172.21.15.12:0/1282163250 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6184007430 con 0x7f619815f5d0 2024-03-29T09:55:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.350+0000 7f619e000700 1 -- 172.21.15.12:0/1282163250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61980ca790 con 0x7f619815f5d0 2024-03-29T09:55:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.350+0000 7f618f400700 1 -- 172.21.15.12:0/1282163250 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61840077c0 con 0x7f619815f5d0 2024-03-29T09:55:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.350+0000 7f619e000700 1 -- 172.21.15.12:0/1282163250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61980caa30 con 0x7f619815f5d0 2024-03-29T09:55:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.351+0000 7f619e000700 1 -- 172.21.15.12:0/1282163250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61980d1ca0 con 0x7f619815f5d0 2024-03-29T09:55:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.352+0000 7f618f400700 1 -- 172.21.15.12:0/1282163250 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f618401e430 con 0x7f619815f5d0 2024-03-29T09:55:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.353+0000 7f618f400700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f617806ac90 0x7f617806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.353+0000 7f618f400700 1 -- 172.21.15.12:0/1282163250 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f618408a5c0 con 0x7f619815f5d0 2024-03-29T09:55:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.353+0000 7f6197400700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f617806ac90 0x7f617806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.354+0000 7f6197400700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f617806ac90 0x7f617806d150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6188009ba0 tx=0x7f6188008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:14.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.357+0000 7f618f400700 1 -- 172.21.15.12:0/1282163250 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6184055700 con 0x7f619815f5d0 2024-03-29T09:55:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.572+0000 7f619e000700 1 -- 172.21.15.12:0/1282163250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6198067720 con 0x7f619815f5d0 2024-03-29T09:55:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.573+0000 7f618f400700 1 -- 172.21.15.12:0/1282163250 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f618400cea0 con 0x7f619815f5d0 2024-03-29T09:55:14.573 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:55:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.575+0000 7f619e000700 1 -- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f617806ac90 msgr2=0x7f617806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.575+0000 7f619e000700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f617806ac90 0x7f617806d150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6188009ba0 tx=0x7f6188008040 comp rx=0 tx=0).stop 2024-03-29T09:55:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.575+0000 7f619e000700 1 -- 172.21.15.12:0/1282163250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f619815f5d0 msgr2=0x7f619815fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.575+0000 7f619e000700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f619815f5d0 0x7f619815fa50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6184015040 tx=0x7f6184001bf0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.575+0000 7f619e000700 1 -- 172.21.15.12:0/1282163250 shutdown_connections 2024-03-29T09:55:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.575+0000 7f619e000700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f617806ac90 0x7f617806d150 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.575+0000 7f619e000700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61980b4ce0 0x7f61980b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.575+0000 7f619e000700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f617c004850 0x7f6198155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.575+0000 7f619e000700 1 --2- 172.21.15.12:0/1282163250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f619815f5d0 0x7f619815fa50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.575+0000 7f619e000700 1 -- 172.21.15.12:0/1282163250 >> 172.21.15.12:0/1282163250 conn(0x7f619808aff0 msgr2=0x7f61980b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.576+0000 7f619e000700 1 -- 172.21.15.12:0/1282163250 shutdown_connections 2024-03-29T09:55:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.576+0000 7f619e000700 1 -- 172.21.15.12:0/1282163250 wait complete. 2024-03-29T09:55:14.594 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:55:14.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.794+0000 7f682c200700 1 Processor -- start 2024-03-29T09:55:14.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.794+0000 7f682c200700 1 -- start start 2024-03-29T09:55:14.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f682c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415dad0 0x7f68240ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f682c200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68240d1630 0x7f68240d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f682c200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f68240d1fd0 0x7f682415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f682c200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68240cfcc0 con 0x7f68240d1630 2024-03-29T09:55:14.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f682c200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68240cd9f0 con 0x7f68240d1fd0 2024-03-29T09:55:14.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f682c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682415f820 con 0x7f682415dad0 2024-03-29T09:55:14.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f6828e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68240d1630 0x7f68240d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f6828e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68240d1630 0x7f68240d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40410/0 (socket says 172.21.15.12:40410) 2024-03-29T09:55:14.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f6828e00700 1 -- 172.21.15.12:0/235074138 learned_addr learned my addr 172.21.15.12:0/235074138 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:14.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f6829800700 1 --2- 172.21.15.12:0/235074138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415dad0 0x7f68240ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.795+0000 7f682a200700 1 --2- 172.21.15.12:0/235074138 >> v2:172.21.15.159:3300/0 conn(0x7f68240d1fd0 0x7f682415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.796+0000 7f6828e00700 1 -- 172.21.15.12:0/235074138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415dad0 msgr2=0x7f68240ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:14.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.796+0000 7f6828e00700 1 --2- 172.21.15.12:0/235074138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415dad0 0x7f68240ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.796+0000 7f6828e00700 1 -- 172.21.15.12:0/235074138 >> v2:172.21.15.159:3300/0 conn(0x7f68240d1fd0 msgr2=0x7f682415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:14.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.796+0000 7f6828e00700 1 --2- 172.21.15.12:0/235074138 >> v2:172.21.15.159:3300/0 conn(0x7f68240d1fd0 0x7f682415deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.796+0000 7f6828e00700 1 -- 172.21.15.12:0/235074138 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68240d2460 con 0x7f68240d1630 2024-03-29T09:55:14.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.796+0000 7f6828e00700 1 --2- 172.21.15.12:0/235074138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1630 0x7f68240d1a90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f68240b95a0 tx=0x7f6810009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b2b9b5c6b54aa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:14.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.796+0000 7f6823e00700 1 -- 172.21.15.12:0/235074138 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681000b870 con 0x7f68240d1630 2024-03-29T09:55:14.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.796+0000 7f6823e00700 1 -- 172.21.15.12:0/235074138 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681000b9d0 con 0x7f68240d1630 2024-03-29T09:55:14.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.796+0000 7f6823e00700 1 -- 172.21.15.12:0/235074138 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681000bd60 con 0x7f68240d1630 2024-03-29T09:55:14.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.797+0000 7f682c200700 1 -- 172.21.15.12:0/235074138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1630 msgr2=0x7f68240d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.797+0000 7f682c200700 1 --2- 172.21.15.12:0/235074138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1630 0x7f68240d1a90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f68240b95a0 tx=0x7f6810009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.797+0000 7f682c200700 1 -- 172.21.15.12:0/235074138 shutdown_connections 2024-03-29T09:55:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.797+0000 7f682c200700 1 --2- 172.21.15.12:0/235074138 >> v2:172.21.15.159:3300/0 conn(0x7f68240d1fd0 0x7f682415deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.797+0000 7f682c200700 1 --2- 172.21.15.12:0/235074138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415dad0 0x7f68240ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.797+0000 7f682c200700 1 --2- 172.21.15.12:0/235074138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1630 0x7f68240d1a90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.797+0000 7f682c200700 1 -- 172.21.15.12:0/235074138 >> 172.21.15.12:0/235074138 conn(0x7f682408aff0 msgr2=0x7f68240d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.797+0000 7f682c200700 1 -- 172.21.15.12:0/235074138 shutdown_connections 2024-03-29T09:55:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.797+0000 7f682c200700 1 -- 172.21.15.12:0/235074138 wait complete. 2024-03-29T09:55:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.799+0000 7f682c200700 1 Processor -- start 2024-03-29T09:55:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.799+0000 7f682c200700 1 -- start start 2024-03-29T09:55:14.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f682c200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68240d1630 0x7f6824154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f682c200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1fd0 0x7f6824155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f682c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415dad0 0x7f68241f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f682c200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6824155df0 con 0x7f68240d1fd0 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f682c200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6824155b10 con 0x7f68240d1630 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f682c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68240b74d0 con 0x7f682415dad0 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f6828e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1fd0 0x7f6824155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f6828e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1fd0 0x7f6824155490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40414/0 (socket says 172.21.15.12:40414) 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f6828e00700 1 -- 172.21.15.12:0/4209799215 learned_addr learned my addr 172.21.15.12:0/4209799215 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f682a200700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415dad0 0x7f68241f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f6829800700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68240d1630 0x7f6824154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f6828e00700 1 -- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415dad0 msgr2=0x7f68241f6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f6828e00700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415dad0 0x7f68241f6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f6828e00700 1 -- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68240d1630 msgr2=0x7f6824154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f6828e00700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68240d1630 0x7f6824154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.800+0000 7f6828e00700 1 -- 172.21.15.12:0/4209799215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6810009550 con 0x7f68240d1fd0 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.801+0000 7f6828e00700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1fd0 0x7f6824155490 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6810000c00 tx=0x7f6810004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:14.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.801+0000 7f6821600700 1 -- 172.21.15.12:0/4209799215 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810004540 con 0x7f68240d1fd0 2024-03-29T09:55:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.801+0000 7f6821600700 1 -- 172.21.15.12:0/4209799215 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68100036a0 con 0x7f68240d1fd0 2024-03-29T09:55:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.801+0000 7f682c200700 1 -- 172.21.15.12:0/4209799215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68240b77b0 con 0x7f68240d1fd0 2024-03-29T09:55:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.801+0000 7f6821600700 1 -- 172.21.15.12:0/4209799215 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810003a50 con 0x7f68240d1fd0 2024-03-29T09:55:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.801+0000 7f682c200700 1 -- 172.21.15.12:0/4209799215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6824152650 con 0x7f68240d1fd0 2024-03-29T09:55:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.802+0000 7f682c200700 1 -- 172.21.15.12:0/4209799215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6824067720 con 0x7f68240d1fd0 2024-03-29T09:55:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.803+0000 7f6821600700 1 -- 172.21.15.12:0/4209799215 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6810021070 con 0x7f68240d1fd0 2024-03-29T09:55:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.804+0000 7f6821600700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406ac90 0x7f680406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.804+0000 7f6821600700 1 -- 172.21.15.12:0/4209799215 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f681008ad40 con 0x7f68240d1fd0 2024-03-29T09:55:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.804+0000 7f6829800700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406ac90 0x7f680406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.805+0000 7f6829800700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406ac90 0x7f680406d150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f681800b9b0 tx=0x7f6818008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:14.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:14.808+0000 7f6821600700 1 -- 172.21.15.12:0/4209799215 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6810055fa0 con 0x7f68240d1fd0 2024-03-29T09:55:15.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.024+0000 7f682c200700 1 -- 172.21.15.12:0/4209799215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f68240b6130 con 0x7f68240d1fd0 2024-03-29T09:55:15.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.025+0000 7f6821600700 1 -- 172.21.15.12:0/4209799215 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6810015020 con 0x7f68240d1fd0 2024-03-29T09:55:15.024 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:55:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 -- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406ac90 msgr2=0x7f680406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406ac90 0x7f680406d150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f681800b9b0 tx=0x7f6818008040 comp rx=0 tx=0).stop 2024-03-29T09:55:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 -- 172.21.15.12:0/4209799215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1fd0 msgr2=0x7f6824155490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1fd0 0x7f6824155490 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6810000c00 tx=0x7f6810004750 comp rx=0 tx=0).stop 2024-03-29T09:55:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 -- 172.21.15.12:0/4209799215 shutdown_connections 2024-03-29T09:55:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406ac90 0x7f680406d150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415dad0 0x7f68241f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68240d1630 0x7f6824154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 --2- 172.21.15.12:0/4209799215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d1fd0 0x7f6824155490 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 -- 172.21.15.12:0/4209799215 >> 172.21.15.12:0/4209799215 conn(0x7f682408aff0 msgr2=0x7f68240ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:15.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 -- 172.21.15.12:0/4209799215 shutdown_connections 2024-03-29T09:55:15.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.027+0000 7f682c200700 1 -- 172.21.15.12:0/4209799215 wait complete. 2024-03-29T09:55:15.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T09:55:15.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.243+0000 7f9a52600700 1 Processor -- start 2024-03-29T09:55:15.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.244+0000 7f9a52600700 1 -- start start 2024-03-29T09:55:15.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.245+0000 7f9a52600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9a4c15d9d0 0x7f9a4c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:15.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.245+0000 7f9a52600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c161420 con 0x7f9a4c0d2540 2024-03-29T09:55:15.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.245+0000 7f9a52600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c161700 con 0x7f9a4c0d16a0 2024-03-29T09:55:15.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.245+0000 7f9a52600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4c15edc0 con 0x7f9a4c15d9d0 2024-03-29T09:55:15.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.245+0000 7f9a4be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9a4c0d2540 0x7f9a4c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54492/0 (socket says 172.21.15.12:54492) 2024-03-29T09:55:15.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.245+0000 7f9a4be00700 1 -- 172.21.15.12:0/2662081083 learned_addr learned my addr 172.21.15.12:0/2662081083 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.245+0000 7f9a4b400700 1 --2- 172.21.15.12:0/2662081083 >> v2:172.21.15.159:3300/0 conn(0x7f9a4c15d9d0 0x7f9a4c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.245+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289624375 0 0) 0x7f9a4c161420 con 0x7f9a4c0d2540 2024-03-29T09:55:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a300035b0 con 0x7f9a4c0d2540 2024-03-29T09:55:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3562699332 0 0) 0x7f9a300035b0 con 0x7f9a4c0d2540 2024-03-29T09:55:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 >> v2:172.21.15.159:3300/0 conn(0x7f9a4c15d9d0 msgr2=0x7f9a4c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 --2- 172.21.15.12:0/2662081083 >> v2:172.21.15.159:3300/0 conn(0x7f9a4c15d9d0 0x7f9a4c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 >> v1:172.21.15.159:6789/0 conn(0x7f9a4c0d16a0 legacy=0x7f9a4c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c1619e0 con 0x7f9a4c0d2540 2024-03-29T09:55:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f9a440019f0 con 0x7f9a4c0d2540 2024-03-29T09:55:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 >> v1:172.21.15.12:6789/0 conn(0x7f9a4c0d2540 legacy=0x7f9a4c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a30003a00 0x7f9a30003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a30004850 0x7f9a30004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a30005210 0x7f9a3000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3000c180 con 0x7f9a30005210 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3000c2f0 con 0x7f9a30004850 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3000c4b0 con 0x7f9a30003a00 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4be00700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a30005210 0x7f9a3000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4b400700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a30003a00 0x7f9a30003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.246+0000 7f9a4aa00700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a30004850 0x7f9a30004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4be00700 1 -- 172.21.15.12:0/2662081083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a30003a00 msgr2=0x7f9a30003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4be00700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a30003a00 0x7f9a30003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4be00700 1 -- 172.21.15.12:0/2662081083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a30004850 msgr2=0x7f9a30004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4be00700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a30004850 0x7f9a30004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4be00700 1 -- 172.21.15.12:0/2662081083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a300035b0 con 0x7f9a30005210 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4aa00700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a30004850 0x7f9a30004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4be00700 1 -- 172.21.15.12:0/2662081083 reap_dead start 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4be00700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a30005210 0x7f9a3000ba10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400a800 tx=0x7f9a30005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1531a9968ac803 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4400a2f0 con 0x7f9a30005210 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a44012620 con 0x7f9a30005210 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.247+0000 7f9a4a000700 1 -- 172.21.15.12:0/2662081083 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a440129d0 con 0x7f9a30005210 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.248+0000 7f9a52600700 1 -- 172.21.15.12:0/2662081083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a30005210 msgr2=0x7f9a3000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.248+0000 7f9a52600700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a30005210 0x7f9a3000ba10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400a800 tx=0x7f9a30005880 comp rx=0 tx=0).stop 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.248+0000 7f9a52600700 1 -- 172.21.15.12:0/2662081083 shutdown_connections 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.248+0000 7f9a52600700 1 --2- 172.21.15.12:0/2662081083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a30005210 0x7f9a3000ba10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.248+0000 7f9a52600700 1 -- 172.21.15.12:0/2662081083 >> 172.21.15.12:0/2662081083 conn(0x7f9a4c08aff0 msgr2=0x7f9a4c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.248+0000 7f9a52600700 1 -- 172.21.15.12:0/2662081083 shutdown_connections 2024-03-29T09:55:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.248+0000 7f9a52600700 1 -- 172.21.15.12:0/2662081083 wait complete. 2024-03-29T09:55:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.250+0000 7f9a52600700 1 Processor -- start 2024-03-29T09:55:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.250+0000 7f9a52600700 1 -- start start 2024-03-29T09:55:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a52600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a30005210 0x7f9a4c158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a52600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a4c160320 0x7f9a4c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a52600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a4c15e2f0 0x7f9a4c0ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a52600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c158fc0 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a52600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0cba60 con 0x7f9a30005210 2024-03-29T09:55:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a52600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0cbd40 con 0x7f9a4c160320 2024-03-29T09:55:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a4be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a4c15e2f0 0x7f9a4c0ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a4be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a4c15e2f0 0x7f9a4c0ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40430/0 (socket says 172.21.15.12:40430) 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a4be00700 1 -- 172.21.15.12:0/301250587 learned_addr learned my addr 172.21.15.12:0/301250587 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a4b400700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a30005210 0x7f9a4c158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.251+0000 7f9a4aa00700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a4c160320 0x7f9a4c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a4be00700 1 -- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a4c160320 msgr2=0x7f9a4c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a4be00700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a4c160320 0x7f9a4c15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a4be00700 1 -- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a30005210 msgr2=0x7f9a4c158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a4be00700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a30005210 0x7f9a4c158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a4be00700 1 -- 172.21.15.12:0/301250587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c0cc020 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a4be00700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a4c15e2f0 0x7f9a4c0ccb00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0d34f0 tx=0x7f9a44001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a43400700 1 -- 172.21.15.12:0/301250587 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a44024070 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a43400700 1 -- 172.21.15.12:0/301250587 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4400a210 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a43400700 1 -- 172.21.15.12:0/301250587 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a440019d0 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a52600700 1 -- 172.21.15.12:0/301250587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4c0cc300 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.252+0000 7f9a52600700 1 -- 172.21.15.12:0/301250587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4c0b6020 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.254+0000 7f9a52600700 1 -- 172.21.15.12:0/301250587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4c0cc820 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.254+0000 7f9a43400700 1 -- 172.21.15.12:0/301250587 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9a4401d420 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.255+0000 7f9a43400700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a2c06ac90 0x7f9a2c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.255+0000 7f9a43400700 1 -- 172.21.15.12:0/301250587 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9a4408a5b0 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.256+0000 7f9a4b400700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a2c06ac90 0x7f9a2c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.256+0000 7f9a4b400700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a2c06ac90 0x7f9a2c06d150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009be0 tx=0x7f9a3c009210 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.260+0000 7f9a43400700 1 -- 172.21.15.12:0/301250587 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9a4c0cc820 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.490+0000 7f9a52600700 1 -- 172.21.15.12:0/301250587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9a4c0cc820 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.491+0000 7f9a43400700 1 -- 172.21.15.12:0/301250587 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9a4c0cc820 con 0x7f9a4c15e2f0 2024-03-29T09:55:15.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 -- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a2c06ac90 msgr2=0x7f9a2c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:15.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a2c06ac90 0x7f9a2c06d150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009be0 tx=0x7f9a3c009210 comp rx=0 tx=0).stop 2024-03-29T09:55:15.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 -- 172.21.15.12:0/301250587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a4c15e2f0 msgr2=0x7f9a4c0ccb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:15.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a4c15e2f0 0x7f9a4c0ccb00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0d34f0 tx=0x7f9a44001d80 comp rx=0 tx=0).stop 2024-03-29T09:55:15.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 -- 172.21.15.12:0/301250587 shutdown_connections 2024-03-29T09:55:15.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a2c06ac90 0x7f9a2c06d150 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a4c15e2f0 0x7f9a4c0ccb00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a30005210 0x7f9a4c158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 --2- 172.21.15.12:0/301250587 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a4c160320 0x7f9a4c15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:15.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 -- 172.21.15.12:0/301250587 >> 172.21.15.12:0/301250587 conn(0x7f9a4c08aff0 msgr2=0x7f9a4c0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:15.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 -- 172.21.15.12:0/301250587 shutdown_connections 2024-03-29T09:55:15.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:15.493+0000 7f9a52600700 1 -- 172.21.15.12:0/301250587 wait complete. 2024-03-29T09:55:15.494 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-29T09:55:15.511 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-03-29T09:55:15.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:15.512 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-03-29T09:55:15.581 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T09:55:15.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:15.582 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-29T09:55:15.627 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:55:15.627 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:15.677 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:55:15.677 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:15.724 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T09:55:15.724 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:15.809 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.008+0000 7f117a600700 1 Processor -- start 2024-03-29T09:55:16.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.008+0000 7f117a600700 1 -- start start 2024-03-29T09:55:16.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.009+0000 7f117a600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11740ac7d0 0x7f1174166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.009+0000 7f117a600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117415fd50 con 0x7f11740cb3f0 2024-03-29T09:55:16.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.009+0000 7f117a600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117415f140 con 0x7f117415b5d0 2024-03-29T09:55:16.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.009+0000 7f117a600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117415b9b0 con 0x7f11740ac7d0 2024-03-29T09:55:16.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.009+0000 7f1172a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f11740cb3f0 0x7f11740ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54496/0 (socket says 172.21.15.12:54496) 2024-03-29T09:55:16.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.009+0000 7f1172a00700 1 -- 172.21.15.12:0/2241256174 learned_addr learned my addr 172.21.15.12:0/2241256174 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:16.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.009+0000 7f1173e00700 1 --2- 172.21.15.12:0/2241256174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11740ac7d0 0x7f1174166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.010+0000 7f1172000700 1 -- 172.21.15.12:0/2241256174 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958158352 0 0) 0x7f117415fd50 con 0x7f11740cb3f0 2024-03-29T09:55:16.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.010+0000 7f1172000700 1 -- 172.21.15.12:0/2241256174 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11580035b0 con 0x7f11740cb3f0 2024-03-29T09:55:16.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.010+0000 7f1172000700 1 -- 172.21.15.12:0/2241256174 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358456476 0 0) 0x7f117415f140 con 0x7f117415b5d0 2024-03-29T09:55:16.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.010+0000 7f1172000700 1 -- 172.21.15.12:0/2241256174 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f117415fd50 con 0x7f117415b5d0 2024-03-29T09:55:16.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.010+0000 7f1173e00700 1 -- 172.21.15.12:0/2241256174 >> v1:172.21.15.159:6789/0 conn(0x7f117415b5d0 legacy=0x7f11740ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.010+0000 7f1173e00700 1 -- 172.21.15.12:0/2241256174 >> v1:172.21.15.12:6789/0 conn(0x7f11740cb3f0 legacy=0x7f11740ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.010+0000 7f1173e00700 1 -- 172.21.15.12:0/2241256174 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117415f420 con 0x7f11740ac7d0 2024-03-29T09:55:16.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.010+0000 7f1173e00700 1 --2- 172.21.15.12:0/2241256174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11740ac7d0 0x7f1174166430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f116c009dc0 tx=0x7f116c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efeebd97860b1c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:16.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.010+0000 7f1172000700 1 -- 172.21.15.12:0/2241256174 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c00e070 con 0x7f11740ac7d0 2024-03-29T09:55:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.011+0000 7f1172000700 1 -- 172.21.15.12:0/2241256174 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c00bb20 con 0x7f11740ac7d0 2024-03-29T09:55:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.011+0000 7f1172000700 1 -- 172.21.15.12:0/2241256174 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c012430 con 0x7f11740ac7d0 2024-03-29T09:55:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.011+0000 7f117a600700 1 -- 172.21.15.12:0/2241256174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11740ac7d0 msgr2=0x7f1174166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.011+0000 7f117a600700 1 --2- 172.21.15.12:0/2241256174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11740ac7d0 0x7f1174166430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f116c009dc0 tx=0x7f116c00b330 comp rx=0 tx=0).stop 2024-03-29T09:55:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.011+0000 7f117a600700 1 -- 172.21.15.12:0/2241256174 shutdown_connections 2024-03-29T09:55:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.011+0000 7f117a600700 1 --2- 172.21.15.12:0/2241256174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11740ac7d0 0x7f1174166430 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.011+0000 7f117a600700 1 -- 172.21.15.12:0/2241256174 >> 172.21.15.12:0/2241256174 conn(0x7f11740cf190 msgr2=0x7f11740cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.011+0000 7f117a600700 1 -- 172.21.15.12:0/2241256174 shutdown_connections 2024-03-29T09:55:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.011+0000 7f117a600700 1 -- 172.21.15.12:0/2241256174 wait complete. 2024-03-29T09:55:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.013+0000 7f117a600700 1 Processor -- start 2024-03-29T09:55:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.013+0000 7f117a600700 1 -- start start 2024-03-29T09:55:16.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.014+0000 7f117a600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11740ac7d0 0x7f11741545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.014+0000 7f117a600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f117415b5d0 0x7f11741f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.014+0000 7f117a600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1174152f30 0x7f1174154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.014+0000 7f117a600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174154af0 con 0x7f117415b5d0 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.014+0000 7f117a600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174153880 con 0x7f11740ac7d0 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.014+0000 7f117a600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1174154120 con 0x7f1174152f30 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.014+0000 7f1172a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f117415b5d0 0x7f11741f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.014+0000 7f1172a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f117415b5d0 0x7f11741f6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40446/0 (socket says 172.21.15.12:40446) 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.015+0000 7f1172a00700 1 -- 172.21.15.12:0/2025604339 learned_addr learned my addr 172.21.15.12:0/2025604339 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.015+0000 7f1173400700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11740ac7d0 0x7f11741545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.015+0000 7f1173e00700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1174152f30 0x7f1174154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.015+0000 7f1172a00700 1 -- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1174152f30 msgr2=0x7f1174154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.015+0000 7f1172a00700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1174152f30 0x7f1174154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.015+0000 7f1172a00700 1 -- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11740ac7d0 msgr2=0x7f11741545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.015+0000 7f1172a00700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11740ac7d0 0x7f11741545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.015+0000 7f1172a00700 1 -- 172.21.15.12:0/2025604339 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c009a70 con 0x7f117415b5d0 2024-03-29T09:55:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.015+0000 7f1172a00700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f117415b5d0 0x7f11741f6130 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f116000c730 tx=0x7f116000ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.016+0000 7f116b400700 1 -- 172.21.15.12:0/2025604339 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160005d70 con 0x7f117415b5d0 2024-03-29T09:55:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.016+0000 7f116b400700 1 -- 172.21.15.12:0/2025604339 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11600070d0 con 0x7f117415b5d0 2024-03-29T09:55:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.016+0000 7f116b400700 1 -- 172.21.15.12:0/2025604339 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160007440 con 0x7f117415b5d0 2024-03-29T09:55:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.016+0000 7f117a600700 1 -- 172.21.15.12:0/2025604339 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11741f6670 con 0x7f117415b5d0 2024-03-29T09:55:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.016+0000 7f117a600700 1 -- 172.21.15.12:0/2025604339 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1174153ad0 con 0x7f117415b5d0 2024-03-29T09:55:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.017+0000 7f117a600700 1 -- 172.21.15.12:0/2025604339 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1174067720 con 0x7f117415b5d0 2024-03-29T09:55:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.018+0000 7f116b400700 1 -- 172.21.15.12:0/2025604339 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f11600075a0 con 0x7f117415b5d0 2024-03-29T09:55:16.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.019+0000 7f116b400700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f115406ace0 0x7f115406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.019+0000 7f116b400700 1 -- 172.21.15.12:0/2025604339 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f11600886a0 con 0x7f117415b5d0 2024-03-29T09:55:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.019+0000 7f1173400700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f115406ace0 0x7f115406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.020+0000 7f1173400700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f115406ace0 0x7f115406d1a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f11640011e0 tx=0x7f1164009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:16.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.023+0000 7f116b400700 1 -- 172.21.15.12:0/2025604339 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1160053910 con 0x7f117415b5d0 2024-03-29T09:55:16.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.235+0000 7f117a600700 1 -- 172.21.15.12:0/2025604339 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1174154260 con 0x7f117415b5d0 2024-03-29T09:55:16.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.237+0000 7f116b400700 1 -- 172.21.15.12:0/2025604339 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f1160056f30 con 0x7f117415b5d0 2024-03-29T09:55:16.236 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:16.237 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:55:12.039728+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 -- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f115406ace0 msgr2=0x7f115406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f115406ace0 0x7f115406d1a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f11640011e0 tx=0x7f1164009040 comp rx=0 tx=0).stop 2024-03-29T09:55:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 -- 172.21.15.12:0/2025604339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f117415b5d0 msgr2=0x7f11741f6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f117415b5d0 0x7f11741f6130 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f116000c730 tx=0x7f116000ca40 comp rx=0 tx=0).stop 2024-03-29T09:55:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 -- 172.21.15.12:0/2025604339 shutdown_connections 2024-03-29T09:55:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f115406ace0 0x7f115406d1a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1174152f30 0x7f1174154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11740ac7d0 0x7f11741545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 --2- 172.21.15.12:0/2025604339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f117415b5d0 0x7f11741f6130 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 -- 172.21.15.12:0/2025604339 >> 172.21.15.12:0/2025604339 conn(0x7f11740cf190 msgr2=0x7f117406c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:16.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 -- 172.21.15.12:0/2025604339 shutdown_connections 2024-03-29T09:55:16.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.239+0000 7f117a600700 1 -- 172.21.15.12:0/2025604339 wait complete. 2024-03-29T09:55:16.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:55:16.258 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:55:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.459+0000 7f4cbc600700 1 Processor -- start 2024-03-29T09:55:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.460+0000 7f4cbc600700 1 -- start start 2024-03-29T09:55:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.461+0000 7f4cbc600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4cb40cbb30 0x7f4cb40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.461+0000 7f4cbc600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb40cc450 con 0x7f4cb40cd480 2024-03-29T09:55:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.461+0000 7f4cbc600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb40cdae0 con 0x7f4cb40cd020 2024-03-29T09:55:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.461+0000 7f4cbc600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb40cd8f0 con 0x7f4cb40cbb30 2024-03-29T09:55:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.461+0000 7f4cba600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4cb40cd480 0x7f4cb406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54512/0 (socket says 172.21.15.12:54512) 2024-03-29T09:55:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.461+0000 7f4cba600700 1 -- 172.21.15.12:0/3299404656 learned_addr learned my addr 172.21.15.12:0/3299404656 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.461+0000 7f4cb9c00700 1 --2- 172.21.15.12:0/3299404656 >> v2:172.21.15.159:3300/0 conn(0x7f4cb40cbb30 0x7f4cb40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.461+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520020222 0 0) 0x7f4cb40cc450 con 0x7f4cb40cd480 2024-03-29T09:55:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c9c0035b0 con 0x7f4cb40cd480 2024-03-29T09:55:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985583898 0 0) 0x7f4cb40cdae0 con 0x7f4cb40cd020 2024-03-29T09:55:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb40cc450 con 0x7f4cb40cd020 2024-03-29T09:55:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1523598695 0 0) 0x7f4c9c0035b0 con 0x7f4cb40cd480 2024-03-29T09:55:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 >> v2:172.21.15.159:3300/0 conn(0x7f4cb40cbb30 msgr2=0x7f4cb40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 --2- 172.21.15.12:0/3299404656 >> v2:172.21.15.159:3300/0 conn(0x7f4cb40cbb30 0x7f4cb40cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 >> v1:172.21.15.159:6789/0 conn(0x7f4cb40cd020 legacy=0x7f4cb40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb40cddc0 con 0x7f4cb40cd480 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f4cac0019f0 con 0x7f4cb40cd480 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 >> v1:172.21.15.12:6789/0 conn(0x7f4cb40cd480 legacy=0x7f4cb406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c9c003be0 0x7f4c9c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c9c004850 0x7f4c9c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4c9c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c00c0e0 con 0x7f4c9c005210 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c00c220 con 0x7f4c9c003be0 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c00c3e0 con 0x7f4c9c004850 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cb9c00700 1 --2- 172.21.15.12:0/3299404656 >> v2:172.21.15.159:3300/0 conn(0x7f4cb40cbb30 0x7f4cb40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.462+0000 7f4cba600700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4c9c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cb9c00700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c9c003be0 0x7f4c9c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cb9200700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c9c004850 0x7f4c9c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cba600700 1 -- 172.21.15.12:0/3299404656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c9c004850 msgr2=0x7f4c9c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cba600700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c9c004850 0x7f4c9c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cba600700 1 -- 172.21.15.12:0/3299404656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c9c003be0 msgr2=0x7f4c9c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cba600700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c9c003be0 0x7f4c9c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cba600700 1 -- 172.21.15.12:0/3299404656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c0035b0 con 0x7f4c9c005210 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cba600700 1 -- 172.21.15.12:0/3299404656 reap_dead start 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cba600700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4c9c00ba10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00a700 tx=0x7f4c9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721f8ad8212c67be server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac0072d0 con 0x7f4c9c005210 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cac007470 con 0x7f4c9c005210 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.463+0000 7f4cb3e00700 1 -- 172.21.15.12:0/3299404656 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac007820 con 0x7f4c9c005210 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.464+0000 7f4cbc600700 1 -- 172.21.15.12:0/3299404656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 msgr2=0x7f4c9c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.464+0000 7f4cbc600700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4c9c00ba10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00a700 tx=0x7f4c9c005880 comp rx=0 tx=0).stop 2024-03-29T09:55:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.464+0000 7f4cbc600700 1 -- 172.21.15.12:0/3299404656 shutdown_connections 2024-03-29T09:55:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.464+0000 7f4cbc600700 1 --2- 172.21.15.12:0/3299404656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4c9c00ba10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.464+0000 7f4cbc600700 1 -- 172.21.15.12:0/3299404656 >> 172.21.15.12:0/3299404656 conn(0x7f4cb40b0d70 msgr2=0x7f4cb40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.464+0000 7f4cbc600700 1 -- 172.21.15.12:0/3299404656 shutdown_connections 2024-03-29T09:55:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.464+0000 7f4cbc600700 1 -- 172.21.15.12:0/3299404656 wait complete. 2024-03-29T09:55:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.466+0000 7f4cbc600700 1 Processor -- start 2024-03-29T09:55:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.466+0000 7f4cbc600700 1 -- start start 2024-03-29T09:55:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cbc600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4cb414f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cbc600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cb40c8480 0x7f4cb40c8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cbc600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb40ca0d0 0x7f4cb40ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cbc600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb40724a0 con 0x7f4c9c005210 2024-03-29T09:55:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cbc600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb4071820 con 0x7f4cb40ca0d0 2024-03-29T09:55:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cbc600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb4071b00 con 0x7f4cb40c8480 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cb9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4cb414f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cb9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4cb414f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40466/0 (socket says 172.21.15.12:40466) 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cb9c00700 1 -- 172.21.15.12:0/1320790142 learned_addr learned my addr 172.21.15.12:0/1320790142 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cba600700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb40ca0d0 0x7f4cb40ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cb9200700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cb40c8480 0x7f4cb40c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cb9c00700 1 -- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cb40c8480 msgr2=0x7f4cb40c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cb9c00700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cb40c8480 0x7f4cb40c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cb9c00700 1 -- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb40ca0d0 msgr2=0x7f4cb40ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cb9c00700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb40ca0d0 0x7f4cb40ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cb9c00700 1 -- 172.21.15.12:0/1320790142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb4071de0 con 0x7f4c9c005210 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.467+0000 7f4cba600700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb40ca0d0 0x7f4cb40ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.468+0000 7f4cb9c00700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4cb414f050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8011ca0 tx=0x7f4ca800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.468+0000 7f4cb1600700 1 -- 172.21.15.12:0/1320790142 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca800fb30 con 0x7f4c9c005210 2024-03-29T09:55:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.468+0000 7f4cb1600700 1 -- 172.21.15.12:0/1320790142 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca800fc90 con 0x7f4c9c005210 2024-03-29T09:55:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.468+0000 7f4cb1600700 1 -- 172.21.15.12:0/1320790142 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8004020 con 0x7f4c9c005210 2024-03-29T09:55:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.468+0000 7f4cbc600700 1 -- 172.21.15.12:0/1320790142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb40720c0 con 0x7f4c9c005210 2024-03-29T09:55:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.468+0000 7f4cbc600700 1 -- 172.21.15.12:0/1320790142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb414f590 con 0x7f4c9c005210 2024-03-29T09:55:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.469+0000 7f4cbc600700 1 -- 172.21.15.12:0/1320790142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb4067720 con 0x7f4c9c005210 2024-03-29T09:55:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.470+0000 7f4cb1600700 1 -- 172.21.15.12:0/1320790142 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4ca8004180 con 0x7f4c9c005210 2024-03-29T09:55:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.471+0000 7f4cb1600700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c9406ac90 0x7f4c9406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.471+0000 7f4cb1600700 1 -- 172.21.15.12:0/1320790142 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4ca8089980 con 0x7f4c9c005210 2024-03-29T09:55:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.471+0000 7f4cb9200700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c9406ac90 0x7f4c9406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.472+0000 7f4cb9200700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c9406ac90 0x7f4c9406d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00078e0 tx=0x7f4ca0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:16.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.475+0000 7f4cb1600700 1 -- 172.21.15.12:0/1320790142 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ca8054ac0 con 0x7f4c9c005210 2024-03-29T09:55:16.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.691+0000 7f4cbc600700 1 -- 172.21.15.12:0/1320790142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4cb4071c40 con 0x7f4c9c005210 2024-03-29T09:55:16.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.691+0000 7f4cb1600700 1 -- 172.21.15.12:0/1320790142 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4cb4071c40 con 0x7f4c9c005210 2024-03-29T09:55:16.691 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:55:16.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 -- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c9406ac90 msgr2=0x7f4c9406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c9406ac90 0x7f4c9406d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00078e0 tx=0x7f4ca0009040 comp rx=0 tx=0).stop 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 -- 172.21.15.12:0/1320790142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 msgr2=0x7f4cb414f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4cb414f050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8011ca0 tx=0x7f4ca800f520 comp rx=0 tx=0).stop 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 -- 172.21.15.12:0/1320790142 shutdown_connections 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c9406ac90 0x7f4c9406d150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb40ca0d0 0x7f4cb40ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c9c005210 0x7f4cb414f050 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 --2- 172.21.15.12:0/1320790142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cb40c8480 0x7f4cb40c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 -- 172.21.15.12:0/1320790142 >> 172.21.15.12:0/1320790142 conn(0x7f4cb40b0d70 msgr2=0x7f4cb40cb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 -- 172.21.15.12:0/1320790142 shutdown_connections 2024-03-29T09:55:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.694+0000 7f4cbc600700 1 -- 172.21.15.12:0/1320790142 wait complete. 2024-03-29T09:55:16.711 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:55:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.912+0000 7f25a9400700 1 Processor -- start 2024-03-29T09:55:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.912+0000 7f25a9400700 1 -- start start 2024-03-29T09:55:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.914+0000 7f25a9400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a40ce230 0x7f25a40ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.914+0000 7f25a9400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f25a40b2070 0x7f25a40b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.914+0000 7f25a9400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a40709c0 con 0x7f25a415d800 2024-03-29T09:55:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.914+0000 7f25a9400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a40b40f0 con 0x7f25a40b2070 2024-03-29T09:55:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.914+0000 7f25a9400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a4070c80 con 0x7f25a40ce230 2024-03-29T09:55:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.914+0000 7f25a1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a40ce230 0x7f25a40ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.914+0000 7f25a3200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f25a40b2070 0x7f25a40b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a3200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f25a40b2070 0x7f25a40b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:45816/0 (socket says 172.21.15.12:45816) 2024-03-29T09:55:16.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a2800700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f25a415d800 0x7f25a40725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46140/0 (socket says 172.21.15.12:46140) 2024-03-29T09:55:16.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a40ce230 0x7f25a40ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:56126/0 (socket says 172.21.15.12:56126) 2024-03-29T09:55:16.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a3200700 1 -- 172.21.15.12:0/1414566834 learned_addr learned my addr 172.21.15.12:0/1414566834 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a1400700 1 -- 172.21.15.12:0/1414566834 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302137706 0 0) 0x7f25a40709c0 con 0x7f25a415d800 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a1400700 1 -- 172.21.15.12:0/1414566834 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25880057c0 con 0x7f25a415d800 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a3200700 1 -- 172.21.15.12:0/1414566834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a40ce230 msgr2=0x7f25a40ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a3200700 1 --2- 172.21.15.12:0/1414566834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a40ce230 0x7f25a40ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a3200700 1 -- 172.21.15.12:0/1414566834 >> v1:172.21.15.159:6789/0 conn(0x7f25a415d800 legacy=0x7f25a40725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a3200700 1 -- 172.21.15.12:0/1414566834 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a4070f60 con 0x7f25a40b2070 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.915+0000 7f25a1e00700 1 --2- 172.21.15.12:0/1414566834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a40ce230 0x7f25a40ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a3200700 1 --2- 172.21.15.12:0/1414566834 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25a40b2070 0x7f25a40b2500 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f259c00af60 tx=0x7f259c00af90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cb06de4933c5247 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a1400700 1 -- 172.21.15.12:0/1414566834 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c00a2d0 con 0x7f25a40b2070 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a1400700 1 -- 172.21.15.12:0/1414566834 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c00c960 con 0x7f25a40b2070 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a1400700 1 -- 172.21.15.12:0/1414566834 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c00cd10 con 0x7f25a40b2070 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a9400700 1 -- 172.21.15.12:0/1414566834 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25a40b2070 msgr2=0x7f25a40b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a9400700 1 --2- 172.21.15.12:0/1414566834 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25a40b2070 0x7f25a40b2500 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f259c00af60 tx=0x7f259c00af90 comp rx=0 tx=0).stop 2024-03-29T09:55:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a9400700 1 -- 172.21.15.12:0/1414566834 shutdown_connections 2024-03-29T09:55:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a9400700 1 --2- 172.21.15.12:0/1414566834 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25a40b2070 0x7f25a40b2500 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a9400700 1 --2- 172.21.15.12:0/1414566834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a40ce230 0x7f25a40ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a9400700 1 -- 172.21.15.12:0/1414566834 >> 172.21.15.12:0/1414566834 conn(0x7f25a40b0d70 msgr2=0x7f25a40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a9400700 1 -- 172.21.15.12:0/1414566834 shutdown_connections 2024-03-29T09:55:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.916+0000 7f25a9400700 1 -- 172.21.15.12:0/1414566834 wait complete. 2024-03-29T09:55:16.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.918+0000 7f25a9400700 1 Processor -- start 2024-03-29T09:55:16.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.918+0000 7f25a9400700 1 -- start start 2024-03-29T09:55:16.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.919+0000 7f25a9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25a40b2070 0x7f25a41ee760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.919+0000 7f25a9400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25a40ce230 0x7f25a4156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.919+0000 7f25a9400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a415d800 0x7f25a4151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.919+0000 7f25a9400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a41eeca0 con 0x7f25a40b2070 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.919+0000 7f25a9400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a406e040 con 0x7f25a40ce230 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.919+0000 7f25a9400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a4158810 con 0x7f25a415d800 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.919+0000 7f25a2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25a40b2070 0x7f25a41ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25a40b2070 0x7f25a41ee760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40480/0 (socket says 172.21.15.12:40480) 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a2800700 1 -- 172.21.15.12:0/1144615251 learned_addr learned my addr 172.21.15.12:0/1144615251 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a3200700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a415d800 0x7f25a4151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a1e00700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25a40ce230 0x7f25a4156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a2800700 1 -- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a415d800 msgr2=0x7f25a4151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a2800700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a415d800 0x7f25a4151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a2800700 1 -- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25a40ce230 msgr2=0x7f25a4156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a2800700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25a40ce230 0x7f25a4156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a2800700 1 -- 172.21.15.12:0/1144615251 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25900094a0 con 0x7f25a40b2070 2024-03-29T09:55:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a2800700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25a40b2070 0x7f25a41ee760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f259400c700 tx=0x7f259400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f259aa00700 1 -- 172.21.15.12:0/1144615251 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594005cb0 con 0x7f25a40b2070 2024-03-29T09:55:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f259aa00700 1 -- 172.21.15.12:0/1144615251 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2594005e10 con 0x7f25a40b2070 2024-03-29T09:55:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f25a9400700 1 -- 172.21.15.12:0/1144615251 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c00ac40 con 0x7f25a40b2070 2024-03-29T09:55:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.920+0000 7f259aa00700 1 -- 172.21.15.12:0/1144615251 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940072d0 con 0x7f25a40b2070 2024-03-29T09:55:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.921+0000 7f25a9400700 1 -- 172.21.15.12:0/1144615251 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a4158dd0 con 0x7f25a40b2070 2024-03-29T09:55:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.921+0000 7f25a9400700 1 -- 172.21.15.12:0/1144615251 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a4067720 con 0x7f25a40b2070 2024-03-29T09:55:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.923+0000 7f259aa00700 1 -- 172.21.15.12:0/1144615251 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2594007430 con 0x7f25a40b2070 2024-03-29T09:55:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.924+0000 7f259aa00700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f258406ac90 0x7f258406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.924+0000 7f259aa00700 1 -- 172.21.15.12:0/1144615251 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2594088e10 con 0x7f25a40b2070 2024-03-29T09:55:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.924+0000 7f25a1e00700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f258406ac90 0x7f258406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.925+0000 7f25a1e00700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f258406ac90 0x7f258406d150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f25a40adc60 tx=0x7f2590008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:16.927+0000 7f259aa00700 1 -- 172.21.15.12:0/1144615251 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2594053f80 con 0x7f25a40b2070 2024-03-29T09:55:17.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.144+0000 7f25a9400700 1 -- 172.21.15.12:0/1144615251 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f25a415b690 con 0x7f25a40b2070 2024-03-29T09:55:17.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.144+0000 7f259aa00700 1 -- 172.21.15.12:0/1144615251 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2594021020 con 0x7f25a40b2070 2024-03-29T09:55:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:55:17.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.146+0000 7f25a9400700 1 -- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f258406ac90 msgr2=0x7f258406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.146+0000 7f25a9400700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f258406ac90 0x7f258406d150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f25a40adc60 tx=0x7f2590008040 comp rx=0 tx=0).stop 2024-03-29T09:55:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.147+0000 7f25a9400700 1 -- 172.21.15.12:0/1144615251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25a40b2070 msgr2=0x7f25a41ee760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.147+0000 7f25a9400700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25a40b2070 0x7f25a41ee760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f259400c700 tx=0x7f259400cac0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.147+0000 7f25a9400700 1 -- 172.21.15.12:0/1144615251 shutdown_connections 2024-03-29T09:55:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.147+0000 7f25a9400700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f258406ac90 0x7f258406d150 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.147+0000 7f25a9400700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25a415d800 0x7f25a4151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.147+0000 7f25a9400700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25a40b2070 0x7f25a41ee760 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.147+0000 7f25a9400700 1 --2- 172.21.15.12:0/1144615251 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25a40ce230 0x7f25a4156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.147+0000 7f25a9400700 1 -- 172.21.15.12:0/1144615251 >> 172.21.15.12:0/1144615251 conn(0x7f25a40b0d70 msgr2=0x7f25a4072610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.147+0000 7f25a9400700 1 -- 172.21.15.12:0/1144615251 shutdown_connections 2024-03-29T09:55:17.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.147+0000 7f25a9400700 1 -- 172.21.15.12:0/1144615251 wait complete. 2024-03-29T09:55:17.164 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:55:17.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.365+0000 7fb6f7200700 1 Processor -- start 2024-03-29T09:55:17.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.366+0000 7fb6f7200700 1 -- start start 2024-03-29T09:55:17.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f7200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6f815a5a0 0x7fb6f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6f80abfc0 0x7fb6f806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f7200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb6f80709f0 0x7fb6f8070e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f7200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f806dd80 con 0x7fb6f815a5a0 2024-03-29T09:55:17.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f7200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f80aeb00 con 0x7fb6f80709f0 2024-03-29T09:55:17.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f80713c0 con 0x7fb6f80abfc0 2024-03-29T09:55:17.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6f815a5a0 0x7fb6f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6f815a5a0 0x7fb6f806e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40488/0 (socket says 172.21.15.12:40488) 2024-03-29T09:55:17.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f5e00700 1 -- 172.21.15.12:0/3362969252 learned_addr learned my addr 172.21.15.12:0/3362969252 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f6800700 1 --2- 172.21.15.12:0/3362969252 >> v2:172.21.15.159:3300/0 conn(0x7fb6f80709f0 0x7fb6f8070e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.367+0000 7fb6f5400700 1 --2- 172.21.15.12:0/3362969252 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6f80abfc0 0x7fb6f806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.368+0000 7fb6f5e00700 1 -- 172.21.15.12:0/3362969252 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6f80abfc0 msgr2=0x7fb6f806d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.368+0000 7fb6f5e00700 1 --2- 172.21.15.12:0/3362969252 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6f80abfc0 0x7fb6f806d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.368+0000 7fb6f5e00700 1 -- 172.21.15.12:0/3362969252 >> v2:172.21.15.159:3300/0 conn(0x7fb6f80709f0 msgr2=0x7fb6f8070e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.368+0000 7fb6f5e00700 1 --2- 172.21.15.12:0/3362969252 >> v2:172.21.15.159:3300/0 conn(0x7fb6f80709f0 0x7fb6f8070e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.368+0000 7fb6f5e00700 1 -- 172.21.15.12:0/3362969252 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f80cda20 con 0x7fb6f815a5a0 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.368+0000 7fb6f6800700 1 --2- 172.21.15.12:0/3362969252 >> v2:172.21.15.159:3300/0 conn(0x7fb6f80709f0 0x7fb6f8070e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.368+0000 7fb6f5e00700 1 --2- 172.21.15.12:0/3362969252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f815a5a0 0x7fb6f806e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb6f80b1d80 tx=0x7fb6e8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20f8138279e6a25 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.368+0000 7fb6f4a00700 1 -- 172.21.15.12:0/3362969252 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e800b860 con 0x7fb6f815a5a0 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.368+0000 7fb6f4a00700 1 -- 172.21.15.12:0/3362969252 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e800b9c0 con 0x7fb6f815a5a0 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.368+0000 7fb6f4a00700 1 -- 172.21.15.12:0/3362969252 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e800bd50 con 0x7fb6f815a5a0 2024-03-29T09:55:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.369+0000 7fb6f7200700 1 -- 172.21.15.12:0/3362969252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f815a5a0 msgr2=0x7fb6f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.369+0000 7fb6f7200700 1 --2- 172.21.15.12:0/3362969252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f815a5a0 0x7fb6f806e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb6f80b1d80 tx=0x7fb6e8009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.369+0000 7fb6f7200700 1 -- 172.21.15.12:0/3362969252 shutdown_connections 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.369+0000 7fb6f7200700 1 --2- 172.21.15.12:0/3362969252 >> v2:172.21.15.159:3300/0 conn(0x7fb6f80709f0 0x7fb6f8070e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.369+0000 7fb6f7200700 1 --2- 172.21.15.12:0/3362969252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f815a5a0 0x7fb6f806e090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.369+0000 7fb6f7200700 1 --2- 172.21.15.12:0/3362969252 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6f80abfc0 0x7fb6f806d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.369+0000 7fb6f7200700 1 -- 172.21.15.12:0/3362969252 >> 172.21.15.12:0/3362969252 conn(0x7fb6f806adc0 msgr2=0x7fb6f80b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.369+0000 7fb6f7200700 1 -- 172.21.15.12:0/3362969252 shutdown_connections 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.369+0000 7fb6f7200700 1 -- 172.21.15.12:0/3362969252 wait complete. 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.371+0000 7fb6f7200700 1 Processor -- start 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.371+0000 7fb6f7200700 1 -- start start 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f7200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6f80709f0 0x7fb6f814c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f80abfc0 0x7fb6f815a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6f815aed0 0x7fb6f814ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f7200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f814f910 con 0x7fb6f80abfc0 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f7200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f815b360 con 0x7fb6f80709f0 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f80c9530 con 0x7fb6f815aed0 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f80abfc0 0x7fb6f815a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f80abfc0 0x7fb6f815a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40490/0 (socket says 172.21.15.12:40490) 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f5400700 1 -- 172.21.15.12:0/1067089556 learned_addr learned my addr 172.21.15.12:0/1067089556 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f5e00700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6f80709f0 0x7fb6f814c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f6800700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6f815aed0 0x7fb6f814ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f5400700 1 -- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6f815aed0 msgr2=0x7fb6f814ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f5400700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6f815aed0 0x7fb6f814ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f5400700 1 -- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6f80709f0 msgr2=0x7fb6f814c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f5400700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6f80709f0 0x7fb6f814c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.372+0000 7fb6f5400700 1 -- 172.21.15.12:0/1067089556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e8009550 con 0x7fb6f80abfc0 2024-03-29T09:55:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.373+0000 7fb6f5400700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f80abfc0 0x7fb6f815a990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400e810 tx=0x7fb6e400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.373+0000 7fb6ee000700 1 -- 172.21.15.12:0/1067089556 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4013610 con 0x7fb6f80abfc0 2024-03-29T09:55:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.373+0000 7fb6ee000700 1 -- 172.21.15.12:0/1067089556 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e4013770 con 0x7fb6f80abfc0 2024-03-29T09:55:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.373+0000 7fb6ee000700 1 -- 172.21.15.12:0/1067089556 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4013b20 con 0x7fb6f80abfc0 2024-03-29T09:55:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.373+0000 7fb6f7200700 1 -- 172.21.15.12:0/1067089556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f80c9810 con 0x7fb6f80abfc0 2024-03-29T09:55:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.373+0000 7fb6f7200700 1 -- 172.21.15.12:0/1067089556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f80c9dd0 con 0x7fb6f80abfc0 2024-03-29T09:55:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.374+0000 7fb6f7200700 1 -- 172.21.15.12:0/1067089556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f804ea80 con 0x7fb6f80abfc0 2024-03-29T09:55:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.375+0000 7fb6ee000700 1 -- 172.21.15.12:0/1067089556 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb6e4013c80 con 0x7fb6f80abfc0 2024-03-29T09:55:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.376+0000 7fb6ee000700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb6d806ac90 0x7fb6d806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.376+0000 7fb6ee000700 1 -- 172.21.15.12:0/1067089556 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb6e4088ff0 con 0x7fb6f80abfc0 2024-03-29T09:55:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.376+0000 7fb6f5e00700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb6d806ac90 0x7fb6d806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.377+0000 7fb6f5e00700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb6d806ac90 0x7fb6d806d150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8000c00 tx=0x7fb6e8008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.380+0000 7fb6ee000700 1 -- 172.21.15.12:0/1067089556 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb6e4054110 con 0x7fb6f80abfc0 2024-03-29T09:55:17.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.594+0000 7fb6f7200700 1 -- 172.21.15.12:0/1067089556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb6f80b1a00 con 0x7fb6f80abfc0 2024-03-29T09:55:17.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.595+0000 7fb6ee000700 1 -- 172.21.15.12:0/1067089556 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb6e4057730 con 0x7fb6f80abfc0 2024-03-29T09:55:17.595 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:55:17.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.597+0000 7fb6f7200700 1 -- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb6d806ac90 msgr2=0x7fb6d806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.597+0000 7fb6f7200700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb6d806ac90 0x7fb6d806d150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8000c00 tx=0x7fb6e8008030 comp rx=0 tx=0).stop 2024-03-29T09:55:17.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.597+0000 7fb6f7200700 1 -- 172.21.15.12:0/1067089556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f80abfc0 msgr2=0x7fb6f815a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.597+0000 7fb6f7200700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f80abfc0 0x7fb6f815a990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400e810 tx=0x7fb6e400eb20 comp rx=0 tx=0).stop 2024-03-29T09:55:17.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.597+0000 7fb6f7200700 1 -- 172.21.15.12:0/1067089556 shutdown_connections 2024-03-29T09:55:17.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.597+0000 7fb6f7200700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb6d806ac90 0x7fb6d806d150 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.597+0000 7fb6f7200700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6f815aed0 0x7fb6f814ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.597+0000 7fb6f7200700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6f80709f0 0x7fb6f814c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.597+0000 7fb6f7200700 1 --2- 172.21.15.12:0/1067089556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6f80abfc0 0x7fb6f815a990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.597+0000 7fb6f7200700 1 -- 172.21.15.12:0/1067089556 >> 172.21.15.12:0/1067089556 conn(0x7fb6f806adc0 msgr2=0x7fb6f80cb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:17.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.598+0000 7fb6f7200700 1 -- 172.21.15.12:0/1067089556 shutdown_connections 2024-03-29T09:55:17.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.598+0000 7fb6f7200700 1 -- 172.21.15.12:0/1067089556 wait complete. 2024-03-29T09:55:17.615 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T09:55:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.814+0000 7f981e600700 1 Processor -- start 2024-03-29T09:55:17.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.814+0000 7f981e600700 1 -- start start 2024-03-29T09:55:17.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.815+0000 7f981e600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f981815d9d0 0x7f98180d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.815+0000 7f981e600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f98180d2540 0x7f98180ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.815+0000 7f981e600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9818161420 con 0x7f98180d16a0 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.815+0000 7f981e600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981815edc0 con 0x7f98180d2540 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.815+0000 7f981e600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9818161700 con 0x7f981815d9d0 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.815+0000 7f9816a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f98180d16a0 0x7f98180d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54528/0 (socket says 172.21.15.12:54528) 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.815+0000 7f9816a00700 1 -- 172.21.15.12:0/4099807417 learned_addr learned my addr 172.21.15.12:0/4099807417 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.815+0000 7f9817400700 1 --2- 172.21.15.12:0/4099807417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f981815d9d0 0x7f98180d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.815+0000 7f9817e00700 1 --2- 172.21.15.12:0/4099807417 >> v2:172.21.15.159:3300/0 conn(0x7f98180d2540 0x7f98180ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.816+0000 7f9816000700 1 -- 172.21.15.12:0/4099807417 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163302304 0 0) 0x7f9818161420 con 0x7f98180d16a0 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.816+0000 7f9816000700 1 -- 172.21.15.12:0/4099807417 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97fc0035b0 con 0x7f98180d16a0 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.816+0000 7f9817400700 1 -- 172.21.15.12:0/4099807417 >> v2:172.21.15.159:3300/0 conn(0x7f98180d2540 msgr2=0x7f98180ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.816+0000 7f9817400700 1 --2- 172.21.15.12:0/4099807417 >> v2:172.21.15.159:3300/0 conn(0x7f98180d2540 0x7f98180ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.816+0000 7f9817400700 1 -- 172.21.15.12:0/4099807417 >> v1:172.21.15.12:6789/0 conn(0x7f98180d16a0 legacy=0x7f98180d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.816+0000 7f9817400700 1 -- 172.21.15.12:0/4099807417 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98181619e0 con 0x7f981815d9d0 2024-03-29T09:55:17.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.816+0000 7f9817e00700 1 --2- 172.21.15.12:0/4099807417 >> v2:172.21.15.159:3300/0 conn(0x7f98180d2540 0x7f98180ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.816+0000 7f9817400700 1 --2- 172.21.15.12:0/4099807417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f981815d9d0 0x7f98180d3c30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9808009870 tx=0x7f9808009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a55366177765bd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f9816000700 1 -- 172.21.15.12:0/4099807417 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9808011830 con 0x7f981815d9d0 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f9816000700 1 -- 172.21.15.12:0/4099807417 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9808011990 con 0x7f981815d9d0 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f9816000700 1 -- 172.21.15.12:0/4099807417 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9808011d20 con 0x7f981815d9d0 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f981e600700 1 -- 172.21.15.12:0/4099807417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f981815d9d0 msgr2=0x7f98180d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f981e600700 1 --2- 172.21.15.12:0/4099807417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f981815d9d0 0x7f98180d3c30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9808009870 tx=0x7f9808009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f981e600700 1 -- 172.21.15.12:0/4099807417 shutdown_connections 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f981e600700 1 --2- 172.21.15.12:0/4099807417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f981815d9d0 0x7f98180d3c30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f981e600700 1 --2- 172.21.15.12:0/4099807417 >> v2:172.21.15.159:3300/0 conn(0x7f98180d2540 0x7f98180ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f981e600700 1 -- 172.21.15.12:0/4099807417 >> 172.21.15.12:0/4099807417 conn(0x7f981808aff0 msgr2=0x7f98180d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f981e600700 1 -- 172.21.15.12:0/4099807417 shutdown_connections 2024-03-29T09:55:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.817+0000 7f981e600700 1 -- 172.21.15.12:0/4099807417 wait complete. 2024-03-29T09:55:17.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.819+0000 7f981e600700 1 Processor -- start 2024-03-29T09:55:17.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.819+0000 7f981e600700 1 -- start start 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.820+0000 7f981e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98180d16a0 0x7f981815c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.820+0000 7f981e600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98180d2540 0x7f98180b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.820+0000 7f981e600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f981815d9d0 0x7f98180b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.820+0000 7f981e600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98180b78b0 con 0x7f98180d16a0 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.820+0000 7f981e600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98180b7b90 con 0x7f981815d9d0 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.820+0000 7f981e600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98180b7e70 con 0x7f98180d2540 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9817400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98180d16a0 0x7f981815c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9817400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98180d16a0 0x7f981815c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40500/0 (socket says 172.21.15.12:40500) 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9817400700 1 -- 172.21.15.12:0/509921412 learned_addr learned my addr 172.21.15.12:0/509921412 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9816a00700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98180d2540 0x7f98180b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9817e00700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f981815d9d0 0x7f98180b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9817400700 1 -- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98180d2540 msgr2=0x7f98180b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9817400700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98180d2540 0x7f98180b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9817400700 1 -- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f981815d9d0 msgr2=0x7f98180b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9817400700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f981815d9d0 0x7f98180b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9817400700 1 -- 172.21.15.12:0/509921412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810009a70 con 0x7f98180d16a0 2024-03-29T09:55:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9816a00700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98180d2540 0x7f98180b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.821+0000 7f9817400700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98180d16a0 0x7f981815c900 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9808014040 tx=0x7f9808004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:17.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.824+0000 7f980f400700 1 -- 172.21.15.12:0/509921412 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98080036a0 con 0x7f98180d16a0 2024-03-29T09:55:17.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.824+0000 7f980f400700 1 -- 172.21.15.12:0/509921412 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980801e070 con 0x7f98180d16a0 2024-03-29T09:55:17.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.824+0000 7f980f400700 1 -- 172.21.15.12:0/509921412 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9808003a20 con 0x7f98180d16a0 2024-03-29T09:55:17.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.824+0000 7f981e600700 1 -- 172.21.15.12:0/509921412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98080094a0 con 0x7f98180d16a0 2024-03-29T09:55:17.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.824+0000 7f981e600700 1 -- 172.21.15.12:0/509921412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98180b75c0 con 0x7f98180d16a0 2024-03-29T09:55:17.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.825+0000 7f981e600700 1 -- 172.21.15.12:0/509921412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98180d1ca0 con 0x7f98180d16a0 2024-03-29T09:55:17.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.826+0000 7f980f400700 1 -- 172.21.15.12:0/509921412 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9808003b80 con 0x7f98180d16a0 2024-03-29T09:55:17.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.826+0000 7f980f400700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97f806ac90 0x7f97f806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:17.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.827+0000 7f980f400700 1 -- 172.21.15.12:0/509921412 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f980808a400 con 0x7f98180d16a0 2024-03-29T09:55:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.827+0000 7f9816a00700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97f806ac90 0x7f97f806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.828+0000 7f9816a00700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97f806ac90 0x7f97f806d150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9804007af0 tx=0x7f9804009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:17.831+0000 7f980f400700 1 -- 172.21.15.12:0/509921412 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9808055540 con 0x7f98180d16a0 2024-03-29T09:55:18.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.057+0000 7f981e600700 1 -- 172.21.15.12:0/509921412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f98180b80b0 con 0x7f98180d16a0 2024-03-29T09:55:18.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.058+0000 7f980f400700 1 -- 172.21.15.12:0/509921412 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9808058b60 con 0x7f98180d16a0 2024-03-29T09:55:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 -- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97f806ac90 msgr2=0x7f97f806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97f806ac90 0x7f97f806d150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9804007af0 tx=0x7f9804009040 comp rx=0 tx=0).stop 2024-03-29T09:55:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 -- 172.21.15.12:0/509921412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98180d16a0 msgr2=0x7f981815c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98180d16a0 0x7f981815c900 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9808014040 tx=0x7f9808004750 comp rx=0 tx=0).stop 2024-03-29T09:55:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 -- 172.21.15.12:0/509921412 shutdown_connections 2024-03-29T09:55:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97f806ac90 0x7f97f806d150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f981815d9d0 0x7f98180b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98180d16a0 0x7f981815c900 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 --2- 172.21.15.12:0/509921412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98180d2540 0x7f98180b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 -- 172.21.15.12:0/509921412 >> 172.21.15.12:0/509921412 conn(0x7f981808aff0 msgr2=0x7f981815ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:18.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.060+0000 7f981e600700 1 -- 172.21.15.12:0/509921412 shutdown_connections 2024-03-29T09:55:18.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.061+0000 7f981e600700 1 -- 172.21.15.12:0/509921412 wait complete. 2024-03-29T09:55:18.061 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-03-29T09:55:18.078 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-03-29T09:55:18.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:18.079 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-03-29T09:55:18.157 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T09:55:18.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:18.158 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-29T09:55:18.204 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T09:55:18.204 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:55:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.402+0000 7f3fccc00700 1 Processor -- start 2024-03-29T09:55:18.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.403+0000 7f3fccc00700 1 -- start start 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.404+0000 7f3fccc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3fc8160f10 0x7f3fc80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.404+0000 7f3fccc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fc80cf230 0x7f3fc80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.404+0000 7f3fccc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc80ceda0 con 0x7f3fc8161590 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.404+0000 7f3fccc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc80d23d0 con 0x7f3fc8160f10 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.404+0000 7f3fccc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc81612f0 con 0x7f3fc80cf230 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.404+0000 7f3fc5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3fc8160f10 0x7f3fc80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.404+0000 7f3fc5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3fc8160f10 0x7f3fc80d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40506/0 (socket says 172.21.15.12:40506) 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.404+0000 7f3fc5e00700 1 -- 172.21.15.12:0/2307635296 learned_addr learned my addr 172.21.15.12:0/2307635296 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.404+0000 7f3fc6800700 1 --1- 172.21.15.12:0/2307635296 >> v1:172.21.15.12:6789/0 conn(0x7f3fc8161590 0x7f3fc80b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54530/0 (socket says 172.21.15.12:54530) 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.404+0000 7f3fc5400700 1 --2- 172.21.15.12:0/2307635296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fc80cf230 0x7f3fc80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.405+0000 7f3fc5e00700 1 -- 172.21.15.12:0/2307635296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fc80cf230 msgr2=0x7f3fc80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.405+0000 7f3fc5e00700 1 --2- 172.21.15.12:0/2307635296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fc80cf230 0x7f3fc80d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.405+0000 7f3fc5e00700 1 -- 172.21.15.12:0/2307635296 >> v1:172.21.15.12:6789/0 conn(0x7f3fc8161590 legacy=0x7f3fc80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:18.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.405+0000 7f3fc5e00700 1 -- 172.21.15.12:0/2307635296 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc80ce470 con 0x7f3fc8160f10 2024-03-29T09:55:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.405+0000 7f3fc5e00700 1 --2- 172.21.15.12:0/2307635296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc8160f10 0x7f3fc80d18b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80098f0 tx=0x7f3fb8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4cb7a63c275433 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.405+0000 7f3fc4a00700 1 -- 172.21.15.12:0/2307635296 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb800b8a0 con 0x7f3fc8160f10 2024-03-29T09:55:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.405+0000 7f3fc4a00700 1 -- 172.21.15.12:0/2307635296 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb800ba00 con 0x7f3fc8160f10 2024-03-29T09:55:18.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.405+0000 7f3fc4a00700 1 -- 172.21.15.12:0/2307635296 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb800bd90 con 0x7f3fc8160f10 2024-03-29T09:55:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.406+0000 7f3fccc00700 1 -- 172.21.15.12:0/2307635296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc8160f10 msgr2=0x7f3fc80d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.406+0000 7f3fccc00700 1 --2- 172.21.15.12:0/2307635296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc8160f10 0x7f3fc80d18b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80098f0 tx=0x7f3fb8009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.406+0000 7f3fccc00700 1 -- 172.21.15.12:0/2307635296 shutdown_connections 2024-03-29T09:55:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.406+0000 7f3fccc00700 1 --2- 172.21.15.12:0/2307635296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc8160f10 0x7f3fc80d18b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.406+0000 7f3fccc00700 1 --2- 172.21.15.12:0/2307635296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fc80cf230 0x7f3fc80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.406+0000 7f3fccc00700 1 -- 172.21.15.12:0/2307635296 >> 172.21.15.12:0/2307635296 conn(0x7f3fc808aff0 msgr2=0x7f3fc80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.406+0000 7f3fccc00700 1 -- 172.21.15.12:0/2307635296 shutdown_connections 2024-03-29T09:55:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.406+0000 7f3fccc00700 1 -- 172.21.15.12:0/2307635296 wait complete. 2024-03-29T09:55:18.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.408+0000 7f3fccc00700 1 Processor -- start 2024-03-29T09:55:18.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.408+0000 7f3fccc00700 1 -- start start 2024-03-29T09:55:18.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.409+0000 7f3fccc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc80cf230 0x7f3fc80b8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:18.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fccc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fc8160f10 0x7f3fc80b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:18.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fccc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fc8161590 0x7f3fc80b76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:18.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fccc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc80b8b60 con 0x7f3fc80cf230 2024-03-29T09:55:18.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fccc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8157900 con 0x7f3fc8160f10 2024-03-29T09:55:18.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fccc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8156470 con 0x7f3fc8161590 2024-03-29T09:55:18.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc80cf230 0x7f3fc80b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc80cf230 0x7f3fc80b8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40510/0 (socket says 172.21.15.12:40510) 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc5e00700 1 -- 172.21.15.12:0/3799398642 learned_addr learned my addr 172.21.15.12:0/3799398642 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc5400700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fc8160f10 0x7f3fc80b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc6800700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fc8161590 0x7f3fc80b76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc5e00700 1 -- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fc8161590 msgr2=0x7f3fc80b76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc5e00700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fc8161590 0x7f3fc80b76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc5e00700 1 -- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fc8160f10 msgr2=0x7f3fc80b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc5e00700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fc8160f10 0x7f3fc80b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc5e00700 1 -- 172.21.15.12:0/3799398642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb8009550 con 0x7f3fc80cf230 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.410+0000 7f3fc5e00700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc80cf230 0x7f3fc80b8620 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009c40 tx=0x7f3fb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.411+0000 7f3fbe000700 1 -- 172.21.15.12:0/3799398642 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80036a0 con 0x7f3fc80cf230 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.411+0000 7f3fbe000700 1 -- 172.21.15.12:0/3799398642 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb801e070 con 0x7f3fc80cf230 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.411+0000 7f3fbe000700 1 -- 172.21.15.12:0/3799398642 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8003a40 con 0x7f3fc80cf230 2024-03-29T09:55:18.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.411+0000 7f3fccc00700 1 -- 172.21.15.12:0/3799398642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc8156fd0 con 0x7f3fc80cf230 2024-03-29T09:55:18.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.411+0000 7f3fccc00700 1 -- 172.21.15.12:0/3799398642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc8157590 con 0x7f3fc80cf230 2024-03-29T09:55:18.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.412+0000 7f3fccc00700 1 -- 172.21.15.12:0/3799398642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc8067720 con 0x7f3fc80cf230 2024-03-29T09:55:18.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.413+0000 7f3fbe000700 1 -- 172.21.15.12:0/3799398642 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3fb8003ba0 con 0x7f3fc80cf230 2024-03-29T09:55:18.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.413+0000 7f3fbe000700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fa806ac90 0x7f3fa806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:18.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.414+0000 7f3fbe000700 1 -- 172.21.15.12:0/3799398642 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f3fb8020070 con 0x7f3fc80cf230 2024-03-29T09:55:18.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.414+0000 7f3fc5400700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fa806ac90 0x7f3fa806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:18.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.415+0000 7f3fc5400700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fa806ac90 0x7f3fa806d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3fc81f6880 tx=0x7f3fb4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.418+0000 7f3fbe000700 1 -- 172.21.15.12:0/3799398642 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3fb8055ad0 con 0x7f3fc80cf230 2024-03-29T09:55:18.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.681+0000 7f3fccc00700 1 -- 172.21.15.12:0/3799398642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc806c160 con 0x7f3fc80cf230 2024-03-29T09:55:18.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.682+0000 7f3fbe000700 1 -- 172.21.15.12:0/3799398642 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7f3fb80590f0 con 0x7f3fc80cf230 2024-03-29T09:55:18.681 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:18.681 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/4093246304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4093246304},{"type":"v1","addr":"172.21.15.12:6833","nonce":4093246304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1631416176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1631416176},{"type":"v1","addr":"172.21.15.159:6836","nonce":1631416176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3828834681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3828834681},{"type":"v1","addr":"172.21.15.159:6837","nonce":3828834681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:54:27.425500+0000","modified":"2024-03-29T09:54:30.895316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3922451359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3922451359},{"type":"v1","addr":"172.21.15.12:6835","nonce":3922451359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-29T09:55:18.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 -- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fa806ac90 msgr2=0x7f3fa806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:18.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fa806ac90 0x7f3fa806d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3fc81f6880 tx=0x7f3fb4008040 comp rx=0 tx=0).stop 2024-03-29T09:55:18.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 -- 172.21.15.12:0/3799398642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc80cf230 msgr2=0x7f3fc80b8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:18.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc80cf230 0x7f3fc80b8620 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009c40 tx=0x7f3fb8004750 comp rx=0 tx=0).stop 2024-03-29T09:55:18.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 -- 172.21.15.12:0/3799398642 shutdown_connections 2024-03-29T09:55:18.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fa806ac90 0x7f3fa806d150 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fc8161590 0x7f3fc80b76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fc80cf230 0x7f3fc80b8620 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 --2- 172.21.15.12:0/3799398642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fc8160f10 0x7f3fc80b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 -- 172.21.15.12:0/3799398642 >> 172.21.15.12:0/3799398642 conn(0x7f3fc808aff0 msgr2=0x7f3fc80caf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.684+0000 7f3fccc00700 1 -- 172.21.15.12:0/3799398642 shutdown_connections 2024-03-29T09:55:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:18.685+0000 7f3fccc00700 1 -- 172.21.15.12:0/3799398642 wait complete. 2024-03-29T09:55:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-03-29T09:55:18.703 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.101+0000 7f2948800700 1 Processor -- start 2024-03-29T09:55:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.101+0000 7f2948800700 1 -- start start 2024-03-29T09:55:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.102+0000 7f2948800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400709c0 0x7f2940070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.102+0000 7f2948800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29400b2180 con 0x7f29400b3af0 2024-03-29T09:55:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.102+0000 7f2948800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294015b100 con 0x7f2940072580 2024-03-29T09:55:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.102+0000 7f2948800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294015b3e0 con 0x7f29400709c0 2024-03-29T09:55:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.102+0000 7f2945e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f29400b3af0 0x7f294015a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47458/0 (socket says 172.21.15.12:47458) 2024-03-29T09:55:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.102+0000 7f2945e00700 1 -- 172.21.15.12:0/884343169 learned_addr learned my addr 172.21.15.12:0/884343169 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2946800700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400709c0 0x7f2940070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247631046 0 0) 0x7f29400b2180 con 0x7f29400b3af0 2024-03-29T09:55:19.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29280035b0 con 0x7f29400b3af0 2024-03-29T09:55:19.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331481325 0 0) 0x7f294015b100 con 0x7f2940072580 2024-03-29T09:55:19.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29400b2180 con 0x7f2940072580 2024-03-29T09:55:19.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3756937936 0 0) 0x7f29280035b0 con 0x7f29400b3af0 2024-03-29T09:55:19.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400709c0 msgr2=0x7f2940070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:19.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400709c0 0x7f2940070e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 >> v1:172.21.15.159:6789/0 conn(0x7f2940072580 legacy=0x7f29400b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29400b23f0 con 0x7f29400b3af0 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f29340015f0 con 0x7f29400b3af0 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 >> v1:172.21.15.12:6789/0 conn(0x7f29400b3af0 legacy=0x7f294015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2928003bc0 0x7f2928003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400709c0 0x7f2928004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 0x7f2928005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294015b3e0 con 0x7f2928004d90 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292800bac0 con 0x7f2928003bc0 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.103+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f292800bc80 con 0x7f29400709c0 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.104+0000 7f2946800700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 0x7f2928005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.104+0000 7f2945400700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400709c0 0x7f2928004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.104+0000 7f2945e00700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2928003bc0 0x7f2928003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.104+0000 7f2946800700 1 -- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400709c0 msgr2=0x7f2928004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:19.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.104+0000 7f2946800700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400709c0 0x7f2928004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.104+0000 7f2946800700 1 -- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2928003bc0 msgr2=0x7f2928003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.104+0000 7f2946800700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2928003bc0 0x7f2928003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.104+0000 7f2946800700 1 -- 172.21.15.12:0/884343169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280035b0 con 0x7f2928004d90 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.104+0000 7f2946800700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 0x7f2928005210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f293c009fc0 tx=0x7f2928005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4bf6e10b4f0ca server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.104+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293c00ec30 con 0x7f2928004d90 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.105+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293c00f040 con 0x7f2928004d90 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.105+0000 7f2944a00700 1 -- 172.21.15.12:0/884343169 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293c014480 con 0x7f2928004d90 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.105+0000 7f2948800700 1 -- 172.21.15.12:0/884343169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 msgr2=0x7f2928005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.105+0000 7f2948800700 1 --2- 172.21.15.12:0/884343169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 0x7f2928005210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f293c009fc0 tx=0x7f2928005880 comp rx=0 tx=0).stop 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.105+0000 7f2946800700 1 -- 172.21.15.12:0/884343169 reap_dead start 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.105+0000 7f2948800700 1 -- 172.21.15.12:0/884343169 shutdown_connections 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.105+0000 7f2948800700 1 -- 172.21.15.12:0/884343169 >> 172.21.15.12:0/884343169 conn(0x7f29400cf190 msgr2=0x7f29400cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.105+0000 7f2948800700 1 -- 172.21.15.12:0/884343169 shutdown_connections 2024-03-29T09:55:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.105+0000 7f2948800700 1 -- 172.21.15.12:0/884343169 wait complete. 2024-03-29T09:55:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.108+0000 7f2948800700 1 Processor -- start 2024-03-29T09:55:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.108+0000 7f2948800700 1 -- start start 2024-03-29T09:55:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2948800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 0x7f29401f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2948800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29400cdff0 0x7f294014de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2948800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400cbba0 0x7f294014e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2948800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29401f3720 con 0x7f2928004d90 2024-03-29T09:55:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2948800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29401580d0 con 0x7f29400cdff0 2024-03-29T09:55:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2948800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2940158650 con 0x7f29400cbba0 2024-03-29T09:55:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2945e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 0x7f29401f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2945e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 0x7f29401f31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39696/0 (socket says 172.21.15.12:39696) 2024-03-29T09:55:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2945e00700 1 -- 172.21.15.12:0/1260934750 learned_addr learned my addr 172.21.15.12:0/1260934750 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2946800700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400cbba0 0x7f294014e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.109+0000 7f2945400700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29400cdff0 0x7f294014de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f2945e00700 1 -- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400cbba0 msgr2=0x7f294014e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f2945e00700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400cbba0 0x7f294014e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f2945e00700 1 -- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29400cdff0 msgr2=0x7f294014de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f2945e00700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29400cdff0 0x7f294014de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f2945e00700 1 -- 172.21.15.12:0/1260934750 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29400ce560 con 0x7f2928004d90 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f2945400700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29400cdff0 0x7f294014de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f2945e00700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 0x7f29401f31e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f293400e7b0 tx=0x7f293400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f293a000700 1 -- 172.21.15.12:0/1260934750 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934012590 con 0x7f2928004d90 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f293a000700 1 -- 172.21.15.12:0/1260934750 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2934012710 con 0x7f2928004d90 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f2948800700 1 -- 172.21.15.12:0/1260934750 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29400ce840 con 0x7f2928004d90 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f293a000700 1 -- 172.21.15.12:0/1260934750 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934012ac0 con 0x7f2928004d90 2024-03-29T09:55:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.110+0000 7f2948800700 1 -- 172.21.15.12:0/1260934750 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294014e9d0 con 0x7f2928004d90 2024-03-29T09:55:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.111+0000 7f2948800700 1 -- 172.21.15.12:0/1260934750 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29400cea30 con 0x7f2928004d90 2024-03-29T09:55:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.112+0000 7f293a000700 1 -- 172.21.15.12:0/1260934750 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2934012c20 con 0x7f2928004d90 2024-03-29T09:55:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.113+0000 7f293a000700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f292406ac90 0x7f292406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.113+0000 7f293a000700 1 -- 172.21.15.12:0/1260934750 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2934089160 con 0x7f2928004d90 2024-03-29T09:55:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.113+0000 7f2945400700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f292406ac90 0x7f292406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.114+0000 7f2945400700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f292406ac90 0x7f292406d150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f29400cf790 tx=0x7f2930009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:19.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.117+0000 7f293a000700 1 -- 172.21.15.12:0/1260934750 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f29400cea30 con 0x7f2928004d90 2024-03-29T09:55:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.332+0000 7f2948800700 1 -- 172.21.15.12:0/1260934750 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29400cea30 con 0x7f2928004d90 2024-03-29T09:55:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.333+0000 7f293a000700 1 -- 172.21.15.12:0/1260934750 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f29400cea30 con 0x7f2928004d90 2024-03-29T09:55:19.333 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:19.333 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:55:12.039728+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 -- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f292406ac90 msgr2=0x7f292406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f292406ac90 0x7f292406d150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f29400cf790 tx=0x7f2930009040 comp rx=0 tx=0).stop 2024-03-29T09:55:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 -- 172.21.15.12:0/1260934750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 msgr2=0x7f29401f31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 0x7f29401f31e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f293400e7b0 tx=0x7f293400eac0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 -- 172.21.15.12:0/1260934750 shutdown_connections 2024-03-29T09:55:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f292406ac90 0x7f292406d150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29400cbba0 0x7f294014e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2928004d90 0x7f29401f31e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 --2- 172.21.15.12:0/1260934750 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29400cdff0 0x7f294014de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 -- 172.21.15.12:0/1260934750 >> 172.21.15.12:0/1260934750 conn(0x7f29400cf190 msgr2=0x7f29400ace20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 -- 172.21.15.12:0/1260934750 shutdown_connections 2024-03-29T09:55:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.336+0000 7f2948800700 1 -- 172.21.15.12:0/1260934750 wait complete. 2024-03-29T09:55:19.354 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:54:25.711250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:54:26.388309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:55:19.354 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:55:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.555+0000 7fd85a400700 1 Processor -- start 2024-03-29T09:55:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.555+0000 7fd85a400700 1 -- start start 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.556+0000 7fd85a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 0x7fd8540d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.556+0000 7fd85a400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd8540d16a0 0x7fd8540d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd85a400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd854161420 con 0x7fd8540d2370 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd85a400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8540d2820 con 0x7fd8540d16a0 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd85a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd854161700 con 0x7fd85415d9d0 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd852a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd8540d16a0 0x7fd8540d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd853e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd8540d2370 0x7fd85415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47470/0 (socket says 172.21.15.12:47470) 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd853e00700 1 -- 172.21.15.12:0/474421519 learned_addr learned my addr 172.21.15.12:0/474421519 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd853400700 1 --2- 172.21.15.12:0/474421519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 0x7fd8540d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd852a00700 1 -- 172.21.15.12:0/474421519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 msgr2=0x7fd8540d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd852a00700 1 --2- 172.21.15.12:0/474421519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 0x7fd8540d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd852a00700 1 -- 172.21.15.12:0/474421519 >> v1:172.21.15.12:6789/0 conn(0x7fd8540d2370 legacy=0x7fd85415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.557+0000 7fd852a00700 1 -- 172.21.15.12:0/474421519 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8541619e0 con 0x7fd8540d16a0 2024-03-29T09:55:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd852a00700 1 --2- 172.21.15.12:0/474421519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d16a0 0x7fd8540d1e30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd840009820 tx=0x7fd840009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49abba182be278f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd852000700 1 -- 172.21.15.12:0/474421519 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd840005e30 con 0x7fd8540d16a0 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd852000700 1 -- 172.21.15.12:0/474421519 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd840006030 con 0x7fd8540d16a0 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd852000700 1 -- 172.21.15.12:0/474421519 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8400063e0 con 0x7fd8540d16a0 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd85a400700 1 -- 172.21.15.12:0/474421519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d16a0 msgr2=0x7fd8540d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd85a400700 1 --2- 172.21.15.12:0/474421519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d16a0 0x7fd8540d1e30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd840009820 tx=0x7fd840009b30 comp rx=0 tx=0).stop 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd85a400700 1 -- 172.21.15.12:0/474421519 shutdown_connections 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd85a400700 1 --2- 172.21.15.12:0/474421519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 0x7fd8540d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd85a400700 1 --2- 172.21.15.12:0/474421519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d16a0 0x7fd8540d1e30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd85a400700 1 -- 172.21.15.12:0/474421519 >> 172.21.15.12:0/474421519 conn(0x7fd85408aff0 msgr2=0x7fd8540d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.558+0000 7fd85a400700 1 -- 172.21.15.12:0/474421519 shutdown_connections 2024-03-29T09:55:19.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.559+0000 7fd85a400700 1 -- 172.21.15.12:0/474421519 wait complete. 2024-03-29T09:55:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.561+0000 7fd85a400700 1 Processor -- start 2024-03-29T09:55:19.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.561+0000 7fd85a400700 1 -- start start 2024-03-29T09:55:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.562+0000 7fd85a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8540d16a0 0x7fd85415f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.562+0000 7fd85a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d2370 0x7fd8540ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.562+0000 7fd85a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 0x7fd854151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.562+0000 7fd85a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85415e2b0 con 0x7fd8540d2370 2024-03-29T09:55:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.562+0000 7fd85a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85415e590 con 0x7fd8540d16a0 2024-03-29T09:55:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.562+0000 7fd85a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85415e870 con 0x7fd85415d9d0 2024-03-29T09:55:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd852a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d2370 0x7fd8540ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd852a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d2370 0x7fd8540ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39716/0 (socket says 172.21.15.12:39716) 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd852a00700 1 -- 172.21.15.12:0/2500098019 learned_addr learned my addr 172.21.15.12:0/2500098019 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd853400700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8540d16a0 0x7fd85415f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd853e00700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 0x7fd854151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd852a00700 1 -- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 msgr2=0x7fd854151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd852a00700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 0x7fd854151350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd852a00700 1 -- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8540d16a0 msgr2=0x7fd85415f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd852a00700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8540d16a0 0x7fd85415f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd852a00700 1 -- 172.21.15.12:0/2500098019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd840009430 con 0x7fd8540d2370 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd853e00700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 0x7fd854151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.563+0000 7fd852a00700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d2370 0x7fd8540ce590 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd840009b70 tx=0x7fd840004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.564+0000 7fd84b400700 1 -- 172.21.15.12:0/2500098019 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd840006890 con 0x7fd8540d2370 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.564+0000 7fd84b400700 1 -- 172.21.15.12:0/2500098019 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd840020070 con 0x7fd8540d2370 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.564+0000 7fd84b400700 1 -- 172.21.15.12:0/2500098019 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd840004260 con 0x7fd8540d2370 2024-03-29T09:55:19.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.564+0000 7fd85a400700 1 -- 172.21.15.12:0/2500098019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85415eb50 con 0x7fd8540d2370 2024-03-29T09:55:19.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.564+0000 7fd85a400700 1 -- 172.21.15.12:0/2500098019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85415f890 con 0x7fd8540d2370 2024-03-29T09:55:19.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.565+0000 7fd85a400700 1 -- 172.21.15.12:0/2500098019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8540d1ca0 con 0x7fd8540d2370 2024-03-29T09:55:19.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.566+0000 7fd84b400700 1 -- 172.21.15.12:0/2500098019 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd840022030 con 0x7fd8540d2370 2024-03-29T09:55:19.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.566+0000 7fd84b400700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406acc0 0x7fd83406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:19.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.567+0000 7fd84b400700 1 -- 172.21.15.12:0/2500098019 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd84008a9b0 con 0x7fd8540d2370 2024-03-29T09:55:19.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.567+0000 7fd853400700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406acc0 0x7fd83406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:19.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.568+0000 7fd853400700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406acc0 0x7fd83406d180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd844006b20 tx=0x7fd844008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:19.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.571+0000 7fd84b400700 1 -- 172.21.15.12:0/2500098019 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd840055af0 con 0x7fd8540d2370 2024-03-29T09:55:19.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.784+0000 7fd85a400700 1 -- 172.21.15.12:0/2500098019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd85415f070 con 0x7fd8540d2370 2024-03-29T09:55:19.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.785+0000 7fd84b400700 1 -- 172.21.15.12:0/2500098019 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd85415f070 con 0x7fd8540d2370 2024-03-29T09:55:19.785 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:55:19.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.787+0000 7fd85a400700 1 -- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406acc0 msgr2=0x7fd83406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.787+0000 7fd85a400700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406acc0 0x7fd83406d180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd844006b20 tx=0x7fd844008040 comp rx=0 tx=0).stop 2024-03-29T09:55:19.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.787+0000 7fd85a400700 1 -- 172.21.15.12:0/2500098019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d2370 msgr2=0x7fd8540ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.787+0000 7fd85a400700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d2370 0x7fd8540ce590 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd840009b70 tx=0x7fd840004750 comp rx=0 tx=0).stop 2024-03-29T09:55:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.787+0000 7fd85a400700 1 -- 172.21.15.12:0/2500098019 shutdown_connections 2024-03-29T09:55:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.787+0000 7fd85a400700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd83406acc0 0x7fd83406d180 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.787+0000 7fd85a400700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd85415d9d0 0x7fd854151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.787+0000 7fd85a400700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8540d16a0 0x7fd85415f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.787+0000 7fd85a400700 1 --2- 172.21.15.12:0/2500098019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8540d2370 0x7fd8540ce590 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.787+0000 7fd85a400700 1 -- 172.21.15.12:0/2500098019 >> 172.21.15.12:0/2500098019 conn(0x7fd85408aff0 msgr2=0x7fd8540b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.788+0000 7fd85a400700 1 -- 172.21.15.12:0/2500098019 shutdown_connections 2024-03-29T09:55:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:19.788+0000 7fd85a400700 1 -- 172.21.15.12:0/2500098019 wait complete. 2024-03-29T09:55:19.805 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T09:55:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.007+0000 7fe3de000700 1 Processor -- start 2024-03-29T09:55:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.007+0000 7fe3de000700 1 -- start start 2024-03-29T09:55:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.009+0000 7fe3de000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe3d806fa20 0x7fe3d815e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.009+0000 7fe3de000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d815eda0 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.009+0000 7fe3de000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d8099dd0 con 0x7fe3d815cfe0 2024-03-29T09:55:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.009+0000 7fe3de000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d809a0b0 con 0x7fe3d806fa20 2024-03-29T09:55:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.009+0000 7fe3d7400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe3d80cd5c0 0x7fe3d806f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47484/0 (socket says 172.21.15.12:47484) 2024-03-29T09:55:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.009+0000 7fe3d7400700 1 -- 172.21.15.12:0/3308196842 learned_addr learned my addr 172.21.15.12:0/3308196842 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.010+0000 7fe3d6a00700 1 --2- 172.21.15.12:0/3308196842 >> v2:172.21.15.159:3300/0 conn(0x7fe3d806fa20 0x7fe3d815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.010+0000 7fe3d6000700 1 -- 172.21.15.12:0/3308196842 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352088112 0 0) 0x7fe3c8001110 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.010+0000 7fe3d6000700 1 -- 172.21.15.12:0/3308196842 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3bc0035b0 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.010+0000 7fe3d6a00700 1 -- 172.21.15.12:0/3308196842 >> v1:172.21.15.159:6789/0 conn(0x7fe3d815cfe0 legacy=0x7fe3d815d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.010+0000 7fe3d6a00700 1 -- 172.21.15.12:0/3308196842 >> v1:172.21.15.12:6789/0 conn(0x7fe3d80cd5c0 legacy=0x7fe3d806f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.010+0000 7fe3d6a00700 1 -- 172.21.15.12:0/3308196842 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d809a390 con 0x7fe3d806fa20 2024-03-29T09:55:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.010+0000 7fe3d6a00700 1 --2- 172.21.15.12:0/3308196842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3d806fa20 0x7fe3d815e860 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009870 tx=0x7fe3c4009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48d643bc329a19a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.011+0000 7fe3d6000700 1 -- 172.21.15.12:0/3308196842 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4011830 con 0x7fe3d806fa20 2024-03-29T09:55:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.011+0000 7fe3d6000700 1 -- 172.21.15.12:0/3308196842 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c4011990 con 0x7fe3d806fa20 2024-03-29T09:55:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.011+0000 7fe3d6000700 1 -- 172.21.15.12:0/3308196842 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4011d20 con 0x7fe3d806fa20 2024-03-29T09:55:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.011+0000 7fe3de000700 1 -- 172.21.15.12:0/3308196842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3d806fa20 msgr2=0x7fe3d815e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.011+0000 7fe3de000700 1 --2- 172.21.15.12:0/3308196842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3d806fa20 0x7fe3d815e860 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009870 tx=0x7fe3c4009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.011+0000 7fe3de000700 1 -- 172.21.15.12:0/3308196842 shutdown_connections 2024-03-29T09:55:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.011+0000 7fe3de000700 1 --2- 172.21.15.12:0/3308196842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3d806fa20 0x7fe3d815e860 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.011+0000 7fe3de000700 1 -- 172.21.15.12:0/3308196842 >> 172.21.15.12:0/3308196842 conn(0x7fe3d80d1370 msgr2=0x7fe3d80d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.011+0000 7fe3de000700 1 -- 172.21.15.12:0/3308196842 shutdown_connections 2024-03-29T09:55:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.012+0000 7fe3de000700 1 -- 172.21.15.12:0/3308196842 wait complete. 2024-03-29T09:55:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.013+0000 7fe3de000700 1 Processor -- start 2024-03-29T09:55:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.013+0000 7fe3de000700 1 -- start start 2024-03-29T09:55:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.014+0000 7fe3de000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3d806fa20 0x7fe3d8154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.014+0000 7fe3de000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d80cd5c0 0x7fe3d80b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.014+0000 7fe3de000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3d815cfe0 0x7fe3d806cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.014+0000 7fe3de000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d8154ac0 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.014+0000 7fe3de000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d80b6600 con 0x7fe3d806fa20 2024-03-29T09:55:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.014+0000 7fe3de000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d80b68e0 con 0x7fe3d815cfe0 2024-03-29T09:55:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.014+0000 7fe3d6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d80cd5c0 0x7fe3d80b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d80cd5c0 0x7fe3d80b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39722/0 (socket says 172.21.15.12:39722) 2024-03-29T09:55:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d6a00700 1 -- 172.21.15.12:0/3520066846 learned_addr learned my addr 172.21.15.12:0/3520066846 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d7400700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3d806fa20 0x7fe3d8154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d7e00700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3d815cfe0 0x7fe3d806cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d6a00700 1 -- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3d815cfe0 msgr2=0x7fe3d806cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d6a00700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3d815cfe0 0x7fe3d806cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d6a00700 1 -- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3d806fa20 msgr2=0x7fe3d8154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d6a00700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3d806fa20 0x7fe3d8154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d6a00700 1 -- 172.21.15.12:0/3520066846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c40094a0 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d7e00700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3d815cfe0 0x7fe3d806cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d7400700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3d806fa20 0x7fe3d8154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3d6a00700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d80cd5c0 0x7fe3d80b2440 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe3c40090b0 tx=0x7fe3c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.015+0000 7fe3cf400700 1 -- 172.21.15.12:0/3520066846 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c40036a0 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.016+0000 7fe3cf400700 1 -- 172.21.15.12:0/3520066846 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c401f070 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.016+0000 7fe3de000700 1 -- 172.21.15.12:0/3520066846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d80b6bc0 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.016+0000 7fe3cf400700 1 -- 172.21.15.12:0/3520066846 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4003a00 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.016+0000 7fe3de000700 1 -- 172.21.15.12:0/3520066846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d80b6220 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.017+0000 7fe3de000700 1 -- 172.21.15.12:0/3520066846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d80b6740 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.018+0000 7fe3cf400700 1 -- 172.21.15.12:0/3520066846 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe3c4003b60 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.019+0000 7fe3cf400700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3b806ac90 0x7fe3b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.019+0000 7fe3cf400700 1 -- 172.21.15.12:0/3520066846 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe3c4021070 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.019+0000 7fe3d7400700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3b806ac90 0x7fe3b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.020+0000 7fe3d7400700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3b806ac90 0x7fe3b806d150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8001270 tx=0x7fe3c8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.023+0000 7fe3cf400700 1 -- 172.21.15.12:0/3520066846 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe3d80b6740 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.239+0000 7fe3de000700 1 -- 172.21.15.12:0/3520066846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe3d80b6740 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.239+0000 7fe3cf400700 1 -- 172.21.15.12:0/3520066846 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe3d80b6740 con 0x7fe3d80cd5c0 2024-03-29T09:55:20.239 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T09:55:20.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 -- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3b806ac90 msgr2=0x7fe3b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3b806ac90 0x7fe3b806d150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8001270 tx=0x7fe3c8009040 comp rx=0 tx=0).stop 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 -- 172.21.15.12:0/3520066846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d80cd5c0 msgr2=0x7fe3d80b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d80cd5c0 0x7fe3d80b2440 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe3c40090b0 tx=0x7fe3c4004750 comp rx=0 tx=0).stop 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 -- 172.21.15.12:0/3520066846 shutdown_connections 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3b806ac90 0x7fe3b806d150 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3d815cfe0 0x7fe3d806cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3d806fa20 0x7fe3d8154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 --2- 172.21.15.12:0/3520066846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3d80cd5c0 0x7fe3d80b2440 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 -- 172.21.15.12:0/3520066846 >> 172.21.15.12:0/3520066846 conn(0x7fe3d80d1370 msgr2=0x7fe3d80cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 -- 172.21.15.12:0/3520066846 shutdown_connections 2024-03-29T09:55:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.242+0000 7fe3de000700 1 -- 172.21.15.12:0/3520066846 wait complete. 2024-03-29T09:55:20.260 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T09:55:20.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.460+0000 7f2292a00700 1 Processor -- start 2024-03-29T09:55:20.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.460+0000 7f2292a00700 1 -- start start 2024-03-29T09:55:20.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.461+0000 7f2292a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f228c15d9d0 0x7f228c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.461+0000 7f2292a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f228c0d16a0 0x7f228c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.461+0000 7f2292a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c161420 con 0x7f228c0d2370 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.461+0000 7f2292a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228c0d2820 con 0x7f228c15d9d0 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.461+0000 7f2292a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c161700 con 0x7f228c0d16a0 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.462+0000 7f228be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f228c15d9d0 0x7f228c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.462+0000 7f228be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f228c15d9d0 0x7f228c0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39738/0 (socket says 172.21.15.12:39738) 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.462+0000 7f228be00700 1 -- 172.21.15.12:0/1234899547 learned_addr learned my addr 172.21.15.12:0/1234899547 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.462+0000 7f228b400700 1 --2- 172.21.15.12:0/1234899547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f228c0d16a0 0x7f228c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.462+0000 7f228be00700 1 -- 172.21.15.12:0/1234899547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f228c0d16a0 msgr2=0x7f228c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.462+0000 7f228be00700 1 --2- 172.21.15.12:0/1234899547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f228c0d16a0 0x7f228c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.462+0000 7f228be00700 1 -- 172.21.15.12:0/1234899547 >> v1:172.21.15.159:6789/0 conn(0x7f228c0d2370 legacy=0x7f228c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:20.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.462+0000 7f228be00700 1 -- 172.21.15.12:0/1234899547 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c1619e0 con 0x7f228c15d9d0 2024-03-29T09:55:20.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.462+0000 7f228be00700 1 --2- 172.21.15.12:0/1234899547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c15d9d0 0x7f228c0d3c30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f227c0098f0 tx=0x7f227c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3739e380977a03 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:20.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f228aa00700 1 -- 172.21.15.12:0/1234899547 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c004230 con 0x7f228c15d9d0 2024-03-29T09:55:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f228aa00700 1 -- 172.21.15.12:0/1234899547 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c004390 con 0x7f228c15d9d0 2024-03-29T09:55:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f228aa00700 1 -- 172.21.15.12:0/1234899547 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0036a0 con 0x7f228c15d9d0 2024-03-29T09:55:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f2292a00700 1 -- 172.21.15.12:0/1234899547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c15d9d0 msgr2=0x7f228c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f2292a00700 1 --2- 172.21.15.12:0/1234899547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c15d9d0 0x7f228c0d3c30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f227c0098f0 tx=0x7f227c009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f2292a00700 1 -- 172.21.15.12:0/1234899547 shutdown_connections 2024-03-29T09:55:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f2292a00700 1 --2- 172.21.15.12:0/1234899547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c15d9d0 0x7f228c0d3c30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f2292a00700 1 --2- 172.21.15.12:0/1234899547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f228c0d16a0 0x7f228c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f2292a00700 1 -- 172.21.15.12:0/1234899547 >> 172.21.15.12:0/1234899547 conn(0x7f228c08aff0 msgr2=0x7f228c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f2292a00700 1 -- 172.21.15.12:0/1234899547 shutdown_connections 2024-03-29T09:55:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.463+0000 7f2292a00700 1 -- 172.21.15.12:0/1234899547 wait complete. 2024-03-29T09:55:20.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.465+0000 7f2292a00700 1 Processor -- start 2024-03-29T09:55:20.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.465+0000 7f2292a00700 1 -- start start 2024-03-29T09:55:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f2292a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c0d16a0 0x7f228c15aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f2292a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f228c0d2370 0x7f228c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f2292a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f228c15d9d0 0x7f228c15e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f2292a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c15e830 con 0x7f228c0d16a0 2024-03-29T09:55:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f2292a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c15eae0 con 0x7f228c15d9d0 2024-03-29T09:55:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f2292a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c15edc0 con 0x7f228c0d2370 2024-03-29T09:55:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f228be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c0d16a0 0x7f228c15aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f228be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c0d16a0 0x7f228c15aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39746/0 (socket says 172.21.15.12:39746) 2024-03-29T09:55:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f228be00700 1 -- 172.21.15.12:0/1193579248 learned_addr learned my addr 172.21.15.12:0/1193579248 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f2290a00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f228c15d9d0 0x7f228c15e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.466+0000 7f228b400700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f228c0d2370 0x7f228c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.467+0000 7f228be00700 1 -- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f228c0d2370 msgr2=0x7f228c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.467+0000 7f228be00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f228c0d2370 0x7f228c15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.467+0000 7f228be00700 1 -- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f228c15d9d0 msgr2=0x7f228c15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.467+0000 7f228be00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f228c15d9d0 0x7f228c15e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.467+0000 7f228be00700 1 -- 172.21.15.12:0/1193579248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c009550 con 0x7f228c0d16a0 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.467+0000 7f228be00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c0d16a0 0x7f228c15aa60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f227c009130 tx=0x7f227c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.467+0000 7f2283e00700 1 -- 172.21.15.12:0/1193579248 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c003cf0 con 0x7f228c0d16a0 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.467+0000 7f2283e00700 1 -- 172.21.15.12:0/1193579248 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c003e50 con 0x7f228c0d16a0 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.467+0000 7f2292a00700 1 -- 172.21.15.12:0/1193579248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c15f0a0 con 0x7f228c0d16a0 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.467+0000 7f2283e00700 1 -- 172.21.15.12:0/1193579248 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c00b8a0 con 0x7f228c0d16a0 2024-03-29T09:55:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.468+0000 7f2292a00700 1 -- 172.21.15.12:0/1193579248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c15f660 con 0x7f228c0d16a0 2024-03-29T09:55:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.469+0000 7f2292a00700 1 -- 172.21.15.12:0/1193579248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c0d1ca0 con 0x7f228c0d16a0 2024-03-29T09:55:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.470+0000 7f2283e00700 1 -- 172.21.15.12:0/1193579248 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f227c00ba00 con 0x7f228c0d16a0 2024-03-29T09:55:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.470+0000 7f2283e00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f226c06ac90 0x7f226c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.470+0000 7f2283e00700 1 -- 172.21.15.12:0/1193579248 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f227c08abe0 con 0x7f228c0d16a0 2024-03-29T09:55:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.471+0000 7f228b400700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f226c06ac90 0x7f226c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.472+0000 7f228b400700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f226c06ac90 0x7f226c06d150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f228c15bb50 tx=0x7f2278008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.475+0000 7f2283e00700 1 -- 172.21.15.12:0/1193579248 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f227c055d20 con 0x7f228c0d16a0 2024-03-29T09:55:20.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.687+0000 7f2292a00700 1 -- 172.21.15.12:0/1193579248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f228c067720 con 0x7f228c0d16a0 2024-03-29T09:55:20.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.688+0000 7f2283e00700 1 -- 172.21.15.12:0/1193579248 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f227c059340 con 0x7f228c0d16a0 2024-03-29T09:55:20.688 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T09:55:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.690+0000 7f2292a00700 1 -- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f226c06ac90 msgr2=0x7f226c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.690+0000 7f2292a00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f226c06ac90 0x7f226c06d150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f228c15bb50 tx=0x7f2278008040 comp rx=0 tx=0).stop 2024-03-29T09:55:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.690+0000 7f2292a00700 1 -- 172.21.15.12:0/1193579248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c0d16a0 msgr2=0x7f228c15aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.690+0000 7f2292a00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c0d16a0 0x7f228c15aa60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f227c009130 tx=0x7f227c004750 comp rx=0 tx=0).stop 2024-03-29T09:55:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.691+0000 7f2292a00700 1 -- 172.21.15.12:0/1193579248 shutdown_connections 2024-03-29T09:55:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.691+0000 7f2292a00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f226c06ac90 0x7f226c06d150 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.691+0000 7f2292a00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f228c15d9d0 0x7f228c15e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.691+0000 7f2292a00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f228c0d16a0 0x7f228c15aa60 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.691+0000 7f2292a00700 1 --2- 172.21.15.12:0/1193579248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f228c0d2370 0x7f228c15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.691+0000 7f2292a00700 1 -- 172.21.15.12:0/1193579248 >> 172.21.15.12:0/1193579248 conn(0x7f228c08aff0 msgr2=0x7f228c0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.691+0000 7f2292a00700 1 -- 172.21.15.12:0/1193579248 shutdown_connections 2024-03-29T09:55:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.691+0000 7f2292a00700 1 -- 172.21.15.12:0/1193579248 wait complete. 2024-03-29T09:55:20.708 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:55:20.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.906+0000 7f60ce000700 1 Processor -- start 2024-03-29T09:55:20.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.906+0000 7f60ce000700 1 -- start start 2024-03-29T09:55:20.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.908+0000 7f60ce000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60c8160f10 0x7f60c80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.908+0000 7f60ce000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f60c80cf230 0x7f60c80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.908+0000 7f60ce000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c80ceda0 con 0x7f60c8161590 2024-03-29T09:55:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.908+0000 7f60ce000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c80d23d0 con 0x7f60c80cf230 2024-03-29T09:55:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.908+0000 7f60ce000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c81612f0 con 0x7f60c8160f10 2024-03-29T09:55:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.908+0000 7f60c6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f60c80cf230 0x7f60c80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.908+0000 7f60c7e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f60c8161590 0x7f60c80b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47492/0 (socket says 172.21.15.12:47492) 2024-03-29T09:55:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.908+0000 7f60c7e00700 1 -- 172.21.15.12:0/1886079306 learned_addr learned my addr 172.21.15.12:0/1886079306 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.908+0000 7f60c7400700 1 --2- 172.21.15.12:0/1886079306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60c8160f10 0x7f60c80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.908+0000 7f60c6a00700 1 -- 172.21.15.12:0/1886079306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60c8160f10 msgr2=0x7f60c80d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.909+0000 7f60c6a00700 1 --2- 172.21.15.12:0/1886079306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60c8160f10 0x7f60c80d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.909+0000 7f60c6a00700 1 -- 172.21.15.12:0/1886079306 >> v1:172.21.15.12:6789/0 conn(0x7f60c8161590 legacy=0x7f60c80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.909+0000 7f60c6a00700 1 -- 172.21.15.12:0/1886079306 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c80ce470 con 0x7f60c80cf230 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.909+0000 7f60c6000700 1 -- 172.21.15.12:0/1886079306 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279275923 0 0) 0x7f60c80ceda0 con 0x7f60c8161590 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.909+0000 7f60c6a00700 1 --2- 172.21.15.12:0/1886079306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c80cf230 0x7f60c80d1e90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009840 tx=0x7f60b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15241ac09620947b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.909+0000 7f60c6000700 1 -- 172.21.15.12:0/1886079306 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4011830 con 0x7f60c80cf230 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.909+0000 7f60c6000700 1 -- 172.21.15.12:0/1886079306 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b4011990 con 0x7f60c80cf230 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.909+0000 7f60c6000700 1 -- 172.21.15.12:0/1886079306 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4011d20 con 0x7f60c80cf230 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.910+0000 7f60ce000700 1 -- 172.21.15.12:0/1886079306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c80cf230 msgr2=0x7f60c80d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.910+0000 7f60ce000700 1 --2- 172.21.15.12:0/1886079306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c80cf230 0x7f60c80d1e90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009840 tx=0x7f60b4009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.910+0000 7f60ce000700 1 -- 172.21.15.12:0/1886079306 shutdown_connections 2024-03-29T09:55:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.910+0000 7f60ce000700 1 --2- 172.21.15.12:0/1886079306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60c8160f10 0x7f60c80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.910+0000 7f60ce000700 1 --2- 172.21.15.12:0/1886079306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c80cf230 0x7f60c80d1e90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.910+0000 7f60ce000700 1 -- 172.21.15.12:0/1886079306 >> 172.21.15.12:0/1886079306 conn(0x7f60c808aff0 msgr2=0x7f60c80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.910+0000 7f60ce000700 1 -- 172.21.15.12:0/1886079306 shutdown_connections 2024-03-29T09:55:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.910+0000 7f60ce000700 1 -- 172.21.15.12:0/1886079306 wait complete. 2024-03-29T09:55:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.911+0000 7f60ce000700 1 Processor -- start 2024-03-29T09:55:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.912+0000 7f60ce000700 1 -- start start 2024-03-29T09:55:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60ce000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60c80cf230 0x7f60c80c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60ce000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60c8160f10 0x7f60c80b63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60ce000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c8161590 0x7f60c81f6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60ce000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c80c9860 con 0x7f60c8161590 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60ce000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c80b6e50 con 0x7f60c80cf230 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60ce000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c80b7130 con 0x7f60c8160f10 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60c7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c8161590 0x7f60c81f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60c7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c8161590 0x7f60c81f6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39750/0 (socket says 172.21.15.12:39750) 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60c7e00700 1 -- 172.21.15.12:0/347640535 learned_addr learned my addr 172.21.15.12:0/347640535 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60c6a00700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60c8160f10 0x7f60c80b63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60c7400700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60c80cf230 0x7f60c80c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60c7e00700 1 -- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60c8160f10 msgr2=0x7f60c80b63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60c7e00700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60c8160f10 0x7f60c80b63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60c7e00700 1 -- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60c80cf230 msgr2=0x7f60c80c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60c7e00700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60c80cf230 0x7f60c80c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.913+0000 7f60c7e00700 1 -- 172.21.15.12:0/347640535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40094a0 con 0x7f60c8161590 2024-03-29T09:55:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.914+0000 7f60c7e00700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c8161590 0x7f60c81f6430 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f60c000cb60 tx=0x7f60c000ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.914+0000 7f60bf400700 1 -- 172.21.15.12:0/347640535 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c000a9e0 con 0x7f60c8161590 2024-03-29T09:55:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.914+0000 7f60bf400700 1 -- 172.21.15.12:0/347640535 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c000ab40 con 0x7f60c8161590 2024-03-29T09:55:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.914+0000 7f60bf400700 1 -- 172.21.15.12:0/347640535 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c00070d0 con 0x7f60c8161590 2024-03-29T09:55:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.914+0000 7f60ce000700 1 -- 172.21.15.12:0/347640535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c80b7410 con 0x7f60c8161590 2024-03-29T09:55:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.914+0000 7f60ce000700 1 -- 172.21.15.12:0/347640535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c80b79d0 con 0x7f60c8161590 2024-03-29T09:55:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.915+0000 7f60ce000700 1 -- 172.21.15.12:0/347640535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c8067720 con 0x7f60c8161590 2024-03-29T09:55:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.916+0000 7f60bf400700 1 -- 172.21.15.12:0/347640535 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f60c0007230 con 0x7f60c8161590 2024-03-29T09:55:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.917+0000 7f60bf400700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f60a806ac90 0x7f60a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.917+0000 7f60bf400700 1 -- 172.21.15.12:0/347640535 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f60c0088dc0 con 0x7f60c8161590 2024-03-29T09:55:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.917+0000 7f60c7400700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f60a806ac90 0x7f60a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.918+0000 7f60c7400700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f60a806ac90 0x7f60a806d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009b40 tx=0x7f60b8009210 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:20.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:20.921+0000 7f60bf400700 1 -- 172.21.15.12:0/347640535 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f60c0053f00 con 0x7f60c8161590 2024-03-29T09:55:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.187+0000 7f60ce000700 1 -- 172.21.15.12:0/347640535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60c815ce20 con 0x7f60c8161590 2024-03-29T09:55:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.188+0000 7f60bf400700 1 -- 172.21.15.12:0/347640535 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7f60c0018070 con 0x7f60c8161590 2024-03-29T09:55:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:21.188 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/4093246304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4093246304},{"type":"v1","addr":"172.21.15.12:6833","nonce":4093246304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1631416176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1631416176},{"type":"v1","addr":"172.21.15.159:6836","nonce":1631416176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3828834681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3828834681},{"type":"v1","addr":"172.21.15.159:6837","nonce":3828834681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:54:27.425500+0000","modified":"2024-03-29T09:54:30.895316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3922451359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3922451359},{"type":"v1","addr":"172.21.15.12:6835","nonce":3922451359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-29T09:55:21.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.190+0000 7f60ce000700 1 -- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f60a806ac90 msgr2=0x7f60a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.190+0000 7f60ce000700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f60a806ac90 0x7f60a806d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009b40 tx=0x7f60b8009210 comp rx=0 tx=0).stop 2024-03-29T09:55:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.191+0000 7f60ce000700 1 -- 172.21.15.12:0/347640535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c8161590 msgr2=0x7f60c81f6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.191+0000 7f60ce000700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c8161590 0x7f60c81f6430 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f60c000cb60 tx=0x7f60c000ce70 comp rx=0 tx=0).stop 2024-03-29T09:55:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.191+0000 7f60ce000700 1 -- 172.21.15.12:0/347640535 shutdown_connections 2024-03-29T09:55:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.191+0000 7f60ce000700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f60a806ac90 0x7f60a806d150 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.191+0000 7f60ce000700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60c8161590 0x7f60c81f6430 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.191+0000 7f60ce000700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60c80cf230 0x7f60c80c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.191+0000 7f60ce000700 1 --2- 172.21.15.12:0/347640535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60c8160f10 0x7f60c80b63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.191+0000 7f60ce000700 1 -- 172.21.15.12:0/347640535 >> 172.21.15.12:0/347640535 conn(0x7f60c808aff0 msgr2=0x7f60c80d1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.191+0000 7f60ce000700 1 -- 172.21.15.12:0/347640535 shutdown_connections 2024-03-29T09:55:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.191+0000 7f60ce000700 1 -- 172.21.15.12:0/347640535 wait complete. 2024-03-29T09:55:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-03-29T09:55:21.208 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:21.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.405+0000 7f15bce00700 1 Processor -- start 2024-03-29T09:55:21.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.406+0000 7f15bce00700 1 -- start start 2024-03-29T09:55:21.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.407+0000 7f15bce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8160f10 0x7f15b80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.407+0000 7f15bce00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15b8161590 0x7f15b8161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.407+0000 7f15bce00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b80ceda0 con 0x7f15b80cf230 2024-03-29T09:55:21.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.407+0000 7f15bce00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b8161f80 con 0x7f15b8161590 2024-03-29T09:55:21.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.407+0000 7f15bce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b81612f0 con 0x7f15b8160f10 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.407+0000 7f15b6800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15b8161590 0x7f15b8161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.407+0000 7f15b6800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15b8161590 0x7f15b8161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39756/0 (socket says 172.21.15.12:39756) 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.407+0000 7f15b6800700 1 -- 172.21.15.12:0/1783826366 learned_addr learned my addr 172.21.15.12:0/1783826366 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.407+0000 7f15b5e00700 1 --2- 172.21.15.12:0/1783826366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8160f10 0x7f15b80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.408+0000 7f15b6800700 1 -- 172.21.15.12:0/1783826366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8160f10 msgr2=0x7f15b80d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.408+0000 7f15b6800700 1 --2- 172.21.15.12:0/1783826366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8160f10 0x7f15b80d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.408+0000 7f15b6800700 1 -- 172.21.15.12:0/1783826366 >> v1:172.21.15.159:6789/0 conn(0x7f15b80cf230 legacy=0x7f15b80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.408+0000 7f15b6800700 1 -- 172.21.15.12:0/1783826366 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b80ce470 con 0x7f15b8161590 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.408+0000 7f15b6800700 1 --2- 172.21.15.12:0/1783826366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b8161590 0x7f15b8161a40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009e80 tx=0x7f15b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31c382b76e9e81c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.408+0000 7f15b4a00700 1 -- 172.21.15.12:0/1783826366 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b000e070 con 0x7f15b8161590 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.408+0000 7f15b4a00700 1 -- 172.21.15.12:0/1783826366 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b000b8d0 con 0x7f15b8161590 2024-03-29T09:55:21.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.408+0000 7f15b4a00700 1 -- 172.21.15.12:0/1783826366 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b000bc80 con 0x7f15b8161590 2024-03-29T09:55:21.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.409+0000 7f15bce00700 1 -- 172.21.15.12:0/1783826366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b8161590 msgr2=0x7f15b8161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.409+0000 7f15bce00700 1 --2- 172.21.15.12:0/1783826366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b8161590 0x7f15b8161a40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009e80 tx=0x7f15b000b330 comp rx=0 tx=0).stop 2024-03-29T09:55:21.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.409+0000 7f15bce00700 1 -- 172.21.15.12:0/1783826366 shutdown_connections 2024-03-29T09:55:21.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.409+0000 7f15bce00700 1 --2- 172.21.15.12:0/1783826366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b8161590 0x7f15b8161a40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.409+0000 7f15bce00700 1 --2- 172.21.15.12:0/1783826366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8160f10 0x7f15b80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.409+0000 7f15bce00700 1 -- 172.21.15.12:0/1783826366 >> 172.21.15.12:0/1783826366 conn(0x7f15b808aff0 msgr2=0x7f15b80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:21.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.409+0000 7f15bce00700 1 -- 172.21.15.12:0/1783826366 shutdown_connections 2024-03-29T09:55:21.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.409+0000 7f15bce00700 1 -- 172.21.15.12:0/1783826366 wait complete. 2024-03-29T09:55:21.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.411+0000 7f15bce00700 1 Processor -- start 2024-03-29T09:55:21.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.411+0000 7f15bce00700 1 -- start start 2024-03-29T09:55:21.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15bce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b80cf230 0x7f15b80b7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15bce00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b8160f10 0x7f15b815f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15bce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8161590 0x7f15b80b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15bce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80b6680 con 0x7f15b80cf230 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15bce00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80b6960 con 0x7f15b8160f10 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15bce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80b6c40 con 0x7f15b8161590 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15b5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b80cf230 0x7f15b80b7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15b5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b80cf230 0x7f15b80b7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39758/0 (socket says 172.21.15.12:39758) 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15b5e00700 1 -- 172.21.15.12:0/4233279962 learned_addr learned my addr 172.21.15.12:0/4233279962 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15b5e00700 1 -- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8161590 msgr2=0x7f15b80b8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15b5e00700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8161590 0x7f15b80b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15b5e00700 1 -- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b8160f10 msgr2=0x7f15b815f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15b5e00700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b8160f10 0x7f15b815f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.412+0000 7f15b5e00700 1 -- 172.21.15.12:0/4233279962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b0009b30 con 0x7f15b80cf230 2024-03-29T09:55:21.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.413+0000 7f15b5e00700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b80cf230 0x7f15b80b7720 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f15a800e870 tx=0x7f15a800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:21.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.413+0000 7f15ae000700 1 -- 172.21.15.12:0/4233279962 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8009590 con 0x7f15b80cf230 2024-03-29T09:55:21.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.413+0000 7f15ae000700 1 -- 172.21.15.12:0/4233279962 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a800ee70 con 0x7f15b80cf230 2024-03-29T09:55:21.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.413+0000 7f15ae000700 1 -- 172.21.15.12:0/4233279962 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a800ca50 con 0x7f15b80cf230 2024-03-29T09:55:21.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.413+0000 7f15bce00700 1 -- 172.21.15.12:0/4233279962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b80b6f20 con 0x7f15b80cf230 2024-03-29T09:55:21.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.413+0000 7f15bce00700 1 -- 172.21.15.12:0/4233279962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b8158bf0 con 0x7f15b80cf230 2024-03-29T09:55:21.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.414+0000 7f15bce00700 1 -- 172.21.15.12:0/4233279962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b8067720 con 0x7f15b80cf230 2024-03-29T09:55:21.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.415+0000 7f15ae000700 1 -- 172.21.15.12:0/4233279962 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f15a800cbb0 con 0x7f15b80cf230 2024-03-29T09:55:21.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.416+0000 7f15ae000700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159806ac90 0x7f159806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.416+0000 7f15ae000700 1 -- 172.21.15.12:0/4233279962 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f15a8089410 con 0x7f15b80cf230 2024-03-29T09:55:21.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.416+0000 7f15b5400700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159806ac90 0x7f159806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:21.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.417+0000 7f15b5400700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159806ac90 0x7f159806d150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f15a4000f60 tx=0x7f15a4007a40 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:21.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.420+0000 7f15ae000700 1 -- 172.21.15.12:0/4233279962 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15a80544c0 con 0x7f15b80cf230 2024-03-29T09:55:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.636+0000 7f15bce00700 1 -- 172.21.15.12:0/4233279962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15b80ccc30 con 0x7f15b80cf230 2024-03-29T09:55:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.637+0000 7f15ae000700 1 -- 172.21.15.12:0/4233279962 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f15a800ce60 con 0x7f15b80cf230 2024-03-29T09:55:21.637 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:21.637 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:55:12.039728+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 -- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159806ac90 msgr2=0x7f159806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159806ac90 0x7f159806d150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f15a4000f60 tx=0x7f15a4007a40 comp rx=0 tx=0).stop 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 -- 172.21.15.12:0/4233279962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b80cf230 msgr2=0x7f15b80b7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b80cf230 0x7f15b80b7720 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f15a800e870 tx=0x7f15a800ec30 comp rx=0 tx=0).stop 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 -- 172.21.15.12:0/4233279962 shutdown_connections 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159806ac90 0x7f159806d150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8161590 0x7f15b80b8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b80cf230 0x7f15b80b7720 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 --2- 172.21.15.12:0/4233279962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b8160f10 0x7f15b815f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 -- 172.21.15.12:0/4233279962 >> 172.21.15.12:0/4233279962 conn(0x7f15b808aff0 msgr2=0x7f15b80b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 -- 172.21.15.12:0/4233279962 shutdown_connections 2024-03-29T09:55:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.640+0000 7f15bce00700 1 -- 172.21.15.12:0/4233279962 wait complete. 2024-03-29T09:55:21.658 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T09:55:21.658 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:55:21.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.855+0000 7fde66200700 1 Processor -- start 2024-03-29T09:55:21.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.855+0000 7fde66200700 1 -- start start 2024-03-29T09:55:21.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.856+0000 7fde66200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fde6015b5d0 0x7fde600ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.856+0000 7fde66200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fde600cb3f0 0x7fde600ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.856+0000 7fde66200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6015fd50 con 0x7fde600ae930 2024-03-29T09:55:21.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.856+0000 7fde66200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6015b9b0 con 0x7fde600cb3f0 2024-03-29T09:55:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.856+0000 7fde66200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6015f140 con 0x7fde6015b5d0 2024-03-29T09:55:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.856+0000 7fde5ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fde600cb3f0 0x7fde600ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.857+0000 7fde5ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fde600cb3f0 0x7fde600ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39772/0 (socket says 172.21.15.12:39772) 2024-03-29T09:55:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.857+0000 7fde5fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fde600ae930 0x7fde60140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47500/0 (socket says 172.21.15.12:47500) 2024-03-29T09:55:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.857+0000 7fde5ea00700 1 -- 172.21.15.12:0/3689173686 learned_addr learned my addr 172.21.15.12:0/3689173686 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.857+0000 7fde5f400700 1 --2- 172.21.15.12:0/3689173686 >> v2:172.21.15.159:3300/0 conn(0x7fde6015b5d0 0x7fde600ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.857+0000 7fde5ea00700 1 -- 172.21.15.12:0/3689173686 >> v2:172.21.15.159:3300/0 conn(0x7fde6015b5d0 msgr2=0x7fde600ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.857+0000 7fde5ea00700 1 --2- 172.21.15.12:0/3689173686 >> v2:172.21.15.159:3300/0 conn(0x7fde6015b5d0 0x7fde600ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.857+0000 7fde5ea00700 1 -- 172.21.15.12:0/3689173686 >> v1:172.21.15.12:6789/0 conn(0x7fde600ae930 legacy=0x7fde60140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.857+0000 7fde5ea00700 1 -- 172.21.15.12:0/3689173686 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6015f420 con 0x7fde600cb3f0 2024-03-29T09:55:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.857+0000 7fde5ea00700 1 --2- 172.21.15.12:0/3689173686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600cb3f0 0x7fde600ac0c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fde6006b8b0 tx=0x7fde4c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc0b48b6c17f1f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.857+0000 7fde5e000700 1 -- 172.21.15.12:0/3689173686 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c00b870 con 0x7fde600cb3f0 2024-03-29T09:55:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.858+0000 7fde5e000700 1 -- 172.21.15.12:0/3689173686 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4c00b9d0 con 0x7fde600cb3f0 2024-03-29T09:55:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.858+0000 7fde5e000700 1 -- 172.21.15.12:0/3689173686 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c00bd60 con 0x7fde600cb3f0 2024-03-29T09:55:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.858+0000 7fde66200700 1 -- 172.21.15.12:0/3689173686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600cb3f0 msgr2=0x7fde600ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.858+0000 7fde66200700 1 --2- 172.21.15.12:0/3689173686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600cb3f0 0x7fde600ac0c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fde6006b8b0 tx=0x7fde4c009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.858+0000 7fde66200700 1 -- 172.21.15.12:0/3689173686 shutdown_connections 2024-03-29T09:55:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.858+0000 7fde66200700 1 --2- 172.21.15.12:0/3689173686 >> v2:172.21.15.159:3300/0 conn(0x7fde6015b5d0 0x7fde600ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.858+0000 7fde66200700 1 --2- 172.21.15.12:0/3689173686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600cb3f0 0x7fde600ac0c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.858+0000 7fde66200700 1 -- 172.21.15.12:0/3689173686 >> 172.21.15.12:0/3689173686 conn(0x7fde600cf190 msgr2=0x7fde600cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.858+0000 7fde66200700 1 -- 172.21.15.12:0/3689173686 shutdown_connections 2024-03-29T09:55:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.858+0000 7fde66200700 1 -- 172.21.15.12:0/3689173686 wait complete. 2024-03-29T09:55:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.860+0000 7fde66200700 1 Processor -- start 2024-03-29T09:55:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.860+0000 7fde66200700 1 -- start start 2024-03-29T09:55:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.861+0000 7fde66200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600ae930 0x7fde600cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.861+0000 7fde66200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde6015b5d0 0x7fde601f1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.861+0000 7fde66200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde600b4150 0x7fde60144580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.861+0000 7fde66200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde600cd7f0 con 0x7fde600ae930 2024-03-29T09:55:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.861+0000 7fde66200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde601537a0 con 0x7fde6015b5d0 2024-03-29T09:55:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.861+0000 7fde66200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde600b45e0 con 0x7fde600b4150 2024-03-29T09:55:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.861+0000 7fde5f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600ae930 0x7fde600cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.861+0000 7fde5f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600ae930 0x7fde600cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39776/0 (socket says 172.21.15.12:39776) 2024-03-29T09:55:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.861+0000 7fde5f400700 1 -- 172.21.15.12:0/2908189119 learned_addr learned my addr 172.21.15.12:0/2908189119 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde5ea00700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde6015b5d0 0x7fde601f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde5f400700 1 -- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde600b4150 msgr2=0x7fde60144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde5f400700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde600b4150 0x7fde60144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde5f400700 1 -- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde6015b5d0 msgr2=0x7fde601f1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde5f400700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde6015b5d0 0x7fde601f1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde5f400700 1 -- 172.21.15.12:0/2908189119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c009550 con 0x7fde600ae930 2024-03-29T09:55:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde5ea00700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde6015b5d0 0x7fde601f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde5f400700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600ae930 0x7fde600cc580 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fde5000e840 tx=0x7fde5000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde57400700 1 -- 172.21.15.12:0/2908189119 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50013610 con 0x7fde600ae930 2024-03-29T09:55:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde57400700 1 -- 172.21.15.12:0/2908189119 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde50013770 con 0x7fde600ae930 2024-03-29T09:55:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde66200700 1 -- 172.21.15.12:0/2908189119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde600ccb70 con 0x7fde600ae930 2024-03-29T09:55:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde57400700 1 -- 172.21.15.12:0/2908189119 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50013b20 con 0x7fde600ae930 2024-03-29T09:55:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.862+0000 7fde66200700 1 -- 172.21.15.12:0/2908189119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde600cd130 con 0x7fde600ae930 2024-03-29T09:55:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.864+0000 7fde66200700 1 -- 172.21.15.12:0/2908189119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde60066ff0 con 0x7fde600ae930 2024-03-29T09:55:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.864+0000 7fde57400700 1 -- 172.21.15.12:0/2908189119 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fde50013c80 con 0x7fde600ae930 2024-03-29T09:55:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.865+0000 7fde57400700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde4006acc0 0x7fde4006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.865+0000 7fde57400700 1 -- 172.21.15.12:0/2908189119 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fde50089070 con 0x7fde600ae930 2024-03-29T09:55:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.866+0000 7fde5ea00700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde4006acc0 0x7fde4006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.866+0000 7fde5ea00700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde4006acc0 0x7fde4006d180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fde6015ef20 tx=0x7fde4c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:21.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:21.870+0000 7fde57400700 1 -- 172.21.15.12:0/2908189119 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fde50054120 con 0x7fde600ae930 2024-03-29T09:55:22.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.130+0000 7fde66200700 1 -- 172.21.15.12:0/2908189119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde60067720 con 0x7fde600ae930 2024-03-29T09:55:22.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.130+0000 7fde57400700 1 -- 172.21.15.12:0/2908189119 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7fde50057740 con 0x7fde600ae930 2024-03-29T09:55:22.130 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:22.131 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/4093246304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4093246304},{"type":"v1","addr":"172.21.15.12:6833","nonce":4093246304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1631416176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1631416176},{"type":"v1","addr":"172.21.15.159:6836","nonce":1631416176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3828834681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3828834681},{"type":"v1","addr":"172.21.15.159:6837","nonce":3828834681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:54:27.425500+0000","modified":"2024-03-29T09:54:30.895316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3922451359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3922451359},{"type":"v1","addr":"172.21.15.12:6835","nonce":3922451359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-29T09:55:22.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 -- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde4006acc0 msgr2=0x7fde4006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde4006acc0 0x7fde4006d180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fde6015ef20 tx=0x7fde4c008030 comp rx=0 tx=0).stop 2024-03-29T09:55:22.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 -- 172.21.15.12:0/2908189119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600ae930 msgr2=0x7fde600cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600ae930 0x7fde600cc580 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fde5000e840 tx=0x7fde5000ec00 comp rx=0 tx=0).stop 2024-03-29T09:55:22.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 -- 172.21.15.12:0/2908189119 shutdown_connections 2024-03-29T09:55:22.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde4006acc0 0x7fde4006d180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde600b4150 0x7fde60144580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde600ae930 0x7fde600cc580 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 --2- 172.21.15.12:0/2908189119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde6015b5d0 0x7fde601f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 -- 172.21.15.12:0/2908189119 >> 172.21.15.12:0/2908189119 conn(0x7fde600cf190 msgr2=0x7fde600aeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:22.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 -- 172.21.15.12:0/2908189119 shutdown_connections 2024-03-29T09:55:22.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.133+0000 7fde66200700 1 -- 172.21.15.12:0/2908189119 wait complete. 2024-03-29T09:55:22.134 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-03-29T09:55:22.151 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:55:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.349+0000 7f1477400700 1 Processor -- start 2024-03-29T09:55:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.349+0000 7f1477400700 1 -- start start 2024-03-29T09:55:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.350+0000 7f1477400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1470160f10 0x7f14700d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.350+0000 7f1477400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700ceda0 con 0x7f1470161590 2024-03-29T09:55:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.350+0000 7f1477400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14701612f0 con 0x7f14700cf230 2024-03-29T09:55:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.350+0000 7f1477400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14700ce470 con 0x7f1470160f10 2024-03-29T09:55:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f1475400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1470161590 0x7f14700b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47510/0 (socket says 172.21.15.12:47510) 2024-03-29T09:55:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f1475400700 1 -- 172.21.15.12:0/2014253061 learned_addr learned my addr 172.21.15.12:0/2014253061 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:22.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f1474a00700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1470160f10 0x7f14700d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252896433 0 0) 0x7f14700ceda0 con 0x7f1470161590 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14580035b0 con 0x7f1470161590 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123580132 0 0) 0x7f14701612f0 con 0x7f14700cf230 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14700ceda0 con 0x7f14700cf230 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2026467955 0 0) 0x7f14580035b0 con 0x7f1470161590 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1470160f10 msgr2=0x7f14700d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f146f400700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1470160f10 0x7f14700d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 >> v1:172.21.15.159:6789/0 conn(0x7f14700cf230 legacy=0x7f14700d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14700ce750 con 0x7f1470161590 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f14680019f0 con 0x7f1470161590 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.351+0000 7f1474a00700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1470160f10 0x7f14700d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 >> v1:172.21.15.12:6789/0 conn(0x7f1470161590 legacy=0x7f14700b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f146f400700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1458003bc0 0x7f1458003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f146f400700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1470160f10 0x7f1458004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f146f400700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1458004d90 0x7f1458005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14700ce470 con 0x7f1458004d90 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145800bb20 con 0x7f1458003bc0 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f145800bce0 con 0x7f1470160f10 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f1474a00700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1458003bc0 0x7f1458003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f1475400700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1458004d90 0x7f1458005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f146fe00700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1470160f10 0x7f1458004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f1475400700 1 -- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1470160f10 msgr2=0x7f1458004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f1475400700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1470160f10 0x7f1458004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f1475400700 1 -- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1458003bc0 msgr2=0x7f1458003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f1475400700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1458003bc0 0x7f1458003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.352+0000 7f1475400700 1 -- 172.21.15.12:0/2014253061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640094a0 con 0x7f1458004d90 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.353+0000 7f1475400700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1458004d90 0x7f1458005210 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f146800a710 tx=0x7f1458005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dce0199086176f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.353+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14680072d0 con 0x7f1458004d90 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.353+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1468007430 con 0x7f1458004d90 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.353+0000 7f146f400700 1 -- 172.21.15.12:0/2014253061 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14680077e0 con 0x7f1458004d90 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.353+0000 7f1477400700 1 -- 172.21.15.12:0/2014253061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1458004d90 msgr2=0x7f1458005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.353+0000 7f1477400700 1 --2- 172.21.15.12:0/2014253061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1458004d90 0x7f1458005210 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f146800a710 tx=0x7f1458005880 comp rx=0 tx=0).stop 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.353+0000 7f1475400700 1 -- 172.21.15.12:0/2014253061 reap_dead start 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.354+0000 7f1477400700 1 -- 172.21.15.12:0/2014253061 shutdown_connections 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.354+0000 7f1477400700 1 -- 172.21.15.12:0/2014253061 >> 172.21.15.12:0/2014253061 conn(0x7f147008aff0 msgr2=0x7f14700d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.354+0000 7f1477400700 1 -- 172.21.15.12:0/2014253061 shutdown_connections 2024-03-29T09:55:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.354+0000 7f1477400700 1 -- 172.21.15.12:0/2014253061 wait complete. 2024-03-29T09:55:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.356+0000 7f1477400700 1 Processor -- start 2024-03-29T09:55:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.356+0000 7f1477400700 1 -- start start 2024-03-29T09:55:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.357+0000 7f1477400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1458004d90 0x7f1470151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.357+0000 7f1477400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f147015a190 0x7f147015a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f1477400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14701e79f0 0x7f14701e7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f1477400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147015ab50 con 0x7f147015a190 2024-03-29T09:55:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f1477400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470153140 con 0x7f14701e79f0 2024-03-29T09:55:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f1477400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1470154150 con 0x7f1458004d90 2024-03-29T09:55:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f146fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f147015a190 0x7f147015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f146fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f147015a190 0x7f147015a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39788/0 (socket says 172.21.15.12:39788) 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f146fe00700 1 -- 172.21.15.12:0/1777211029 learned_addr learned my addr 172.21.15.12:0/1777211029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f1474a00700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1458004d90 0x7f1470151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f146fe00700 1 -- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1458004d90 msgr2=0x7f1470151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f146fe00700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1458004d90 0x7f1470151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f1475400700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14701e79f0 0x7f14701e7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f146fe00700 1 -- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14701e79f0 msgr2=0x7f14701e7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f146fe00700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14701e79f0 0x7f14701e7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f146fe00700 1 -- 172.21.15.12:0/1777211029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1470153390 con 0x7f147015a190 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f1474a00700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1458004d90 0x7f1470151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f1475400700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14701e79f0 0x7f14701e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.358+0000 7f146fe00700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f147015a190 0x7f147015a610 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f145c00f7f0 tx=0x7f145c00fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.359+0000 7f146cc00700 1 -- 172.21.15.12:0/1777211029 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c005320 con 0x7f147015a190 2024-03-29T09:55:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.359+0000 7f146cc00700 1 -- 172.21.15.12:0/1777211029 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c005480 con 0x7f147015a190 2024-03-29T09:55:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.359+0000 7f1477400700 1 -- 172.21.15.12:0/1777211029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1470153670 con 0x7f147015a190 2024-03-29T09:55:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.359+0000 7f146cc00700 1 -- 172.21.15.12:0/1777211029 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c006aa0 con 0x7f147015a190 2024-03-29T09:55:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.359+0000 7f1477400700 1 -- 172.21.15.12:0/1777211029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1470153c30 con 0x7f147015a190 2024-03-29T09:55:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.360+0000 7f1477400700 1 -- 172.21.15.12:0/1777211029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1470067720 con 0x7f147015a190 2024-03-29T09:55:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.361+0000 7f146cc00700 1 -- 172.21.15.12:0/1777211029 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f145c006c00 con 0x7f147015a190 2024-03-29T09:55:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.362+0000 7f146cc00700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f145006ac90 0x7f145006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.362+0000 7f146cc00700 1 -- 172.21.15.12:0/1777211029 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f145c088e10 con 0x7f147015a190 2024-03-29T09:55:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.362+0000 7f1474a00700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f145006ac90 0x7f145006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.363+0000 7f1474a00700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f145006ac90 0x7f145006d150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1464009ce0 tx=0x7f1464008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.366+0000 7f146cc00700 1 -- 172.21.15.12:0/1777211029 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f145c08c030 con 0x7f147015a190 2024-03-29T09:55:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.627+0000 7f1477400700 1 -- 172.21.15.12:0/1777211029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f147008b760 con 0x7f147015a190 2024-03-29T09:55:22.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.627+0000 7f146cc00700 1 -- 172.21.15.12:0/1777211029 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7f145c01a090 con 0x7f147015a190 2024-03-29T09:55:22.627 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:22.627 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/4093246304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4093246304},{"type":"v1","addr":"172.21.15.12:6833","nonce":4093246304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1631416176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1631416176},{"type":"v1","addr":"172.21.15.159:6836","nonce":1631416176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3828834681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3828834681},{"type":"v1","addr":"172.21.15.159:6837","nonce":3828834681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:54:27.425500+0000","modified":"2024-03-29T09:54:30.895316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3922451359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3922451359},{"type":"v1","addr":"172.21.15.12:6835","nonce":3922451359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-29T09:55:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 -- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f145006ac90 msgr2=0x7f145006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f145006ac90 0x7f145006d150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1464009ce0 tx=0x7f1464008040 comp rx=0 tx=0).stop 2024-03-29T09:55:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 -- 172.21.15.12:0/1777211029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f147015a190 msgr2=0x7f147015a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f147015a190 0x7f147015a610 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f145c00f7f0 tx=0x7f145c00fbb0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 -- 172.21.15.12:0/1777211029 shutdown_connections 2024-03-29T09:55:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f145006ac90 0x7f145006d150 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14701e79f0 0x7f14701e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1458004d90 0x7f1470151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 --2- 172.21.15.12:0/1777211029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f147015a190 0x7f147015a610 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 -- 172.21.15.12:0/1777211029 >> 172.21.15.12:0/1777211029 conn(0x7f147008aff0 msgr2=0x7f147015c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 -- 172.21.15.12:0/1777211029 shutdown_connections 2024-03-29T09:55:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.630+0000 7f1477400700 1 -- 172.21.15.12:0/1777211029 wait complete. 2024-03-29T09:55:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-03-29T09:55:22.649 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:22.664 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:55:22.698 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:55:22.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.848+0000 7f4882000700 1 Processor -- start 2024-03-29T09:55:22.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.849+0000 7f4882000700 1 -- start start 2024-03-29T09:55:22.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f4882000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f487c15b5d0 0x7f487c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f4882000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f487c0cb3f0 0x7f487c0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f4882000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487c15fd50 con 0x7f487c0ae930 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f4882000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f487c15b9b0 con 0x7f487c0cb3f0 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f4882000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f487c15f140 con 0x7f487c15b5d0 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f487aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f487c0cb3f0 0x7f487c0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f487aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f487c0cb3f0 0x7f487c0ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39792/0 (socket says 172.21.15.12:39792) 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f487aa00700 1 -- 172.21.15.12:0/2735045267 learned_addr learned my addr 172.21.15.12:0/2735045267 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f487b400700 1 --2- 172.21.15.12:0/2735045267 >> v2:172.21.15.159:3300/0 conn(0x7f487c15b5d0 0x7f487c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f487aa00700 1 -- 172.21.15.12:0/2735045267 >> v2:172.21.15.159:3300/0 conn(0x7f487c15b5d0 msgr2=0x7f487c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f487aa00700 1 --2- 172.21.15.12:0/2735045267 >> v2:172.21.15.159:3300/0 conn(0x7f487c15b5d0 0x7f487c0ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f487aa00700 1 -- 172.21.15.12:0/2735045267 >> v1:172.21.15.159:6789/0 conn(0x7f487c0ae930 legacy=0x7f487c140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f487aa00700 1 -- 172.21.15.12:0/2735045267 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c15f420 con 0x7f487c0cb3f0 2024-03-29T09:55:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.850+0000 7f487b400700 1 --2- 172.21.15.12:0/2735045267 >> v2:172.21.15.159:3300/0 conn(0x7f487c15b5d0 0x7f487c0ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.851+0000 7f487aa00700 1 --2- 172.21.15.12:0/2735045267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0cb3f0 0x7f487c0ac0c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f487c06b8b0 tx=0x7f4868009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db69850e6bbd752 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.851+0000 7f487a000700 1 -- 172.21.15.12:0/2735045267 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4868004230 con 0x7f487c0cb3f0 2024-03-29T09:55:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.851+0000 7f487a000700 1 -- 172.21.15.12:0/2735045267 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4868004390 con 0x7f487c0cb3f0 2024-03-29T09:55:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.851+0000 7f487a000700 1 -- 172.21.15.12:0/2735045267 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48680036a0 con 0x7f487c0cb3f0 2024-03-29T09:55:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.851+0000 7f4882000700 1 -- 172.21.15.12:0/2735045267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0cb3f0 msgr2=0x7f487c0ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.851+0000 7f4882000700 1 --2- 172.21.15.12:0/2735045267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0cb3f0 0x7f487c0ac0c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f487c06b8b0 tx=0x7f4868009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.852+0000 7f4882000700 1 -- 172.21.15.12:0/2735045267 shutdown_connections 2024-03-29T09:55:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.852+0000 7f4882000700 1 --2- 172.21.15.12:0/2735045267 >> v2:172.21.15.159:3300/0 conn(0x7f487c15b5d0 0x7f487c0ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.852+0000 7f4882000700 1 --2- 172.21.15.12:0/2735045267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0cb3f0 0x7f487c0ac0c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.852+0000 7f4882000700 1 -- 172.21.15.12:0/2735045267 >> 172.21.15.12:0/2735045267 conn(0x7f487c0cf190 msgr2=0x7f487c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.852+0000 7f4882000700 1 -- 172.21.15.12:0/2735045267 shutdown_connections 2024-03-29T09:55:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.852+0000 7f4882000700 1 -- 172.21.15.12:0/2735045267 wait complete. 2024-03-29T09:55:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.853+0000 7f4882000700 1 Processor -- start 2024-03-29T09:55:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.854+0000 7f4882000700 1 -- start start 2024-03-29T09:55:22.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.854+0000 7f4882000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f487c0ae930 0x7f487c0ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.854+0000 7f4882000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f487c15b5d0 0x7f487c0b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.854+0000 7f4882000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0afc70 0x7f487c0b0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.854+0000 7f4882000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c0b0640 con 0x7f487c0afc70 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f4882000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c0c99b0 con 0x7f487c15b5d0 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f4882000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487c0c9c90 con 0x7f487c0ae930 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0afc70 0x7f487c0b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0afc70 0x7f487c0b0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39808/0 (socket says 172.21.15.12:39808) 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487be00700 1 -- 172.21.15.12:0/1312243872 learned_addr learned my addr 172.21.15.12:0/1312243872 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487b400700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f487c0ae930 0x7f487c0ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487aa00700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f487c15b5d0 0x7f487c0b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487be00700 1 -- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f487c0ae930 msgr2=0x7f487c0ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487be00700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f487c0ae930 0x7f487c0ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487be00700 1 -- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f487c15b5d0 msgr2=0x7f487c0b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487be00700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f487c15b5d0 0x7f487c0b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:22.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487be00700 1 -- 172.21.15.12:0/1312243872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4868009550 con 0x7f487c0afc70 2024-03-29T09:55:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487b400700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f487c0ae930 0x7f487c0ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487aa00700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f487c15b5d0 0x7f487c0b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.855+0000 7f487be00700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0afc70 0x7f487c0b0100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f487400b970 tx=0x7f487400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.856+0000 7f4873400700 1 -- 172.21.15.12:0/1312243872 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487400c9b0 con 0x7f487c0afc70 2024-03-29T09:55:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.856+0000 7f4873400700 1 -- 172.21.15.12:0/1312243872 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487400cb10 con 0x7f487c0afc70 2024-03-29T09:55:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.856+0000 7f4873400700 1 -- 172.21.15.12:0/1312243872 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874014440 con 0x7f487c0afc70 2024-03-29T09:55:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.856+0000 7f4882000700 1 -- 172.21.15.12:0/1312243872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487c0c9f70 con 0x7f487c0afc70 2024-03-29T09:55:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.856+0000 7f4882000700 1 -- 172.21.15.12:0/1312243872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487c0b3d20 con 0x7f487c0afc70 2024-03-29T09:55:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.857+0000 7f4882000700 1 -- 172.21.15.12:0/1312243872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487c0ca490 con 0x7f487c0afc70 2024-03-29T09:55:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.858+0000 7f4873400700 1 -- 172.21.15.12:0/1312243872 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f487401d3b0 con 0x7f487c0afc70 2024-03-29T09:55:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.859+0000 7f4873400700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f485c06ace0 0x7f485c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.859+0000 7f4873400700 1 -- 172.21.15.12:0/1312243872 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4874088fe0 con 0x7f487c0afc70 2024-03-29T09:55:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.859+0000 7f487b400700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f485c06ace0 0x7f485c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:22.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.860+0000 7f487b400700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f485c06ace0 0x7f485c06d1a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f487c153990 tx=0x7f486c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:22.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:22.864+0000 7f4873400700 1 -- 172.21.15.12:0/1312243872 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f48740bc580 con 0x7f487c0afc70 2024-03-29T09:55:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.083+0000 7f4882000700 1 -- 172.21.15.12:0/1312243872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f487c067720 con 0x7f487c0afc70 2024-03-29T09:55:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.084+0000 7f4873400700 1 -- 172.21.15.12:0/1312243872 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f487c0ca490 con 0x7f487c0afc70 2024-03-29T09:55:23.084 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:23.084 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:55:12.039728+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:54:25.711250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-29T09:54:26.388309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 -- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f485c06ace0 msgr2=0x7f485c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f485c06ace0 0x7f485c06d1a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f487c153990 tx=0x7f486c008040 comp rx=0 tx=0).stop 2024-03-29T09:55:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 -- 172.21.15.12:0/1312243872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0afc70 msgr2=0x7f487c0b0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0afc70 0x7f487c0b0100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f487400b970 tx=0x7f487400bd30 comp rx=0 tx=0).stop 2024-03-29T09:55:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 -- 172.21.15.12:0/1312243872 shutdown_connections 2024-03-29T09:55:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f485c06ace0 0x7f485c06d1a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f487c0afc70 0x7f487c0b0100 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f487c0ae930 0x7f487c0ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 --2- 172.21.15.12:0/1312243872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f487c15b5d0 0x7f487c0b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 -- 172.21.15.12:0/1312243872 >> 172.21.15.12:0/1312243872 conn(0x7f487c0cf190 msgr2=0x7f487c15dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 -- 172.21.15.12:0/1312243872 shutdown_connections 2024-03-29T09:55:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.087+0000 7f4882000700 1 -- 172.21.15.12:0/1312243872 wait complete. 2024-03-29T09:55:23.106 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T09:55:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.305+0000 7f92d0e00700 1 Processor -- start 2024-03-29T09:55:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.305+0000 7f92d0e00700 1 -- start start 2024-03-29T09:55:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.306+0000 7f92d0e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 0x7f92cc15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:23.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.306+0000 7f92d0e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc0ce270 con 0x7f92cc0cee10 2024-03-29T09:55:23.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.306+0000 7f92d0e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc15b600 con 0x7f92cc15f4d0 2024-03-29T09:55:23.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.306+0000 7f92d0e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0d2760 con 0x7f92cc1611b0 2024-03-29T09:55:23.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.307+0000 7f92ca800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f92cc0cee10 0x7f92cc161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47522/0 (socket says 172.21.15.12:47522) 2024-03-29T09:55:23.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.307+0000 7f92ca800700 1 -- 172.21.15.12:0/1963063217 learned_addr learned my addr 172.21.15.12:0/1963063217 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:23.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.307+0000 7f92c9e00700 1 --2- 172.21.15.12:0/1963063217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 0x7f92cc15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:23.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.307+0000 7f92c8a00700 1 -- 172.21.15.12:0/1963063217 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772092296 0 0) 0x7f92cc0ce270 con 0x7f92cc0cee10 2024-03-29T09:55:23.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.307+0000 7f92c8a00700 1 -- 172.21.15.12:0/1963063217 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b00035b0 con 0x7f92cc0cee10 2024-03-29T09:55:23.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.307+0000 7f92c8a00700 1 -- 172.21.15.12:0/1963063217 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241544748 0 0) 0x7f92cc15b600 con 0x7f92cc15f4d0 2024-03-29T09:55:23.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.307+0000 7f92c8a00700 1 -- 172.21.15.12:0/1963063217 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92cc0ce270 con 0x7f92cc15f4d0 2024-03-29T09:55:23.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.307+0000 7f92c9e00700 1 -- 172.21.15.12:0/1963063217 >> v1:172.21.15.159:6789/0 conn(0x7f92cc15f4d0 legacy=0x7f92cc0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.307+0000 7f92c9e00700 1 -- 172.21.15.12:0/1963063217 >> v1:172.21.15.12:6789/0 conn(0x7f92cc0cee10 legacy=0x7f92cc161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.307+0000 7f92c9e00700 1 -- 172.21.15.12:0/1963063217 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc0d2a40 con 0x7f92cc1611b0 2024-03-29T09:55:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.308+0000 7f92c9e00700 1 --2- 172.21.15.12:0/1963063217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 0x7f92cc15ef90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009870 tx=0x7f92bc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=819a7da084f3c261 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.308+0000 7f92c8a00700 1 -- 172.21.15.12:0/1963063217 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc011830 con 0x7f92cc1611b0 2024-03-29T09:55:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.308+0000 7f92c8a00700 1 -- 172.21.15.12:0/1963063217 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc011990 con 0x7f92cc1611b0 2024-03-29T09:55:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.308+0000 7f92c8a00700 1 -- 172.21.15.12:0/1963063217 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc011d20 con 0x7f92cc1611b0 2024-03-29T09:55:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.309+0000 7f92d0e00700 1 -- 172.21.15.12:0/1963063217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 msgr2=0x7f92cc15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.309+0000 7f92d0e00700 1 --2- 172.21.15.12:0/1963063217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 0x7f92cc15ef90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009870 tx=0x7f92bc009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.309+0000 7f92d0e00700 1 -- 172.21.15.12:0/1963063217 shutdown_connections 2024-03-29T09:55:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.309+0000 7f92d0e00700 1 --2- 172.21.15.12:0/1963063217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 0x7f92cc15ef90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.309+0000 7f92d0e00700 1 -- 172.21.15.12:0/1963063217 >> 172.21.15.12:0/1963063217 conn(0x7f92cc08aff0 msgr2=0x7f92cc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.309+0000 7f92d0e00700 1 -- 172.21.15.12:0/1963063217 shutdown_connections 2024-03-29T09:55:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.309+0000 7f92d0e00700 1 -- 172.21.15.12:0/1963063217 wait complete. 2024-03-29T09:55:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.310+0000 7f92d0e00700 1 Processor -- start 2024-03-29T09:55:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.311+0000 7f92d0e00700 1 -- start start 2024-03-29T09:55:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92d0e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92cc0cee10 0x7f92cc1f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92d0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92cc15f4d0 0x7f92cc1f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92d0e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 0x7f92cc0c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92d0e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0cd840 con 0x7f92cc15f4d0 2024-03-29T09:55:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92d0e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc1f32f0 con 0x7f92cc0cee10 2024-03-29T09:55:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92d0e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0c8c60 con 0x7f92cc1611b0 2024-03-29T09:55:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92c9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92cc15f4d0 0x7f92cc1f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92c9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92cc15f4d0 0x7f92cc1f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39814/0 (socket says 172.21.15.12:39814) 2024-03-29T09:55:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92c9400700 1 -- 172.21.15.12:0/2178515239 learned_addr learned my addr 172.21.15.12:0/2178515239 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92c9e00700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92cc0cee10 0x7f92cc1f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92ca800700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 0x7f92cc0c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92c9400700 1 -- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 msgr2=0x7f92cc0c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92c9400700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 0x7f92cc0c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92c9400700 1 -- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92cc0cee10 msgr2=0x7f92cc1f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92c9400700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92cc0cee10 0x7f92cc1f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.312+0000 7f92c9400700 1 -- 172.21.15.12:0/2178515239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc0094a0 con 0x7f92cc15f4d0 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.313+0000 7f92c9400700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92cc15f4d0 0x7f92cc1f2db0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c500 tx=0x7f92b800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.313+0000 7f92c2000700 1 -- 172.21.15.12:0/2178515239 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b80052d0 con 0x7f92cc15f4d0 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.313+0000 7f92c2000700 1 -- 172.21.15.12:0/2178515239 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b8005430 con 0x7f92cc15f4d0 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.313+0000 7f92c2000700 1 -- 172.21.15.12:0/2178515239 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b80038c0 con 0x7f92cc15f4d0 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.313+0000 7f92d0e00700 1 -- 172.21.15.12:0/2178515239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc0ca6a0 con 0x7f92cc15f4d0 2024-03-29T09:55:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.313+0000 7f92d0e00700 1 -- 172.21.15.12:0/2178515239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc0ca940 con 0x7f92cc15f4d0 2024-03-29T09:55:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.314+0000 7f92d0e00700 1 -- 172.21.15.12:0/2178515239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc067720 con 0x7f92cc15f4d0 2024-03-29T09:55:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.315+0000 7f92c2000700 1 -- 172.21.15.12:0/2178515239 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f92b8003a20 con 0x7f92cc15f4d0 2024-03-29T09:55:23.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.316+0000 7f92c2000700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92ac06ac90 0x7f92ac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:23.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.316+0000 7f92c2000700 1 -- 172.21.15.12:0/2178515239 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f92b8088950 con 0x7f92cc15f4d0 2024-03-29T09:55:23.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.317+0000 7f92c9e00700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92ac06ac90 0x7f92ac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:23.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.318+0000 7f92c9e00700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92ac06ac90 0x7f92ac06d150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0097c0 tx=0x7f92bc01c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:23.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.321+0000 7f92c2000700 1 -- 172.21.15.12:0/2178515239 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92b8053ac0 con 0x7f92cc15f4d0 2024-03-29T09:55:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.586+0000 7f92d0e00700 1 -- 172.21.15.12:0/2178515239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f92cc0b5d00 con 0x7f92cc15f4d0 2024-03-29T09:55:23.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T09:55:23.587+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T09:55:23.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.601+0000 7f92c2000700 1 -- 172.21.15.12:0/2178515239 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7f92b8017090 con 0x7f92cc15f4d0 2024-03-29T09:55:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 -- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92ac06ac90 msgr2=0x7f92ac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92ac06ac90 0x7f92ac06d150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0097c0 tx=0x7f92bc01c040 comp rx=0 tx=0).stop 2024-03-29T09:55:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 -- 172.21.15.12:0/2178515239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92cc15f4d0 msgr2=0x7f92cc1f2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92cc15f4d0 0x7f92cc1f2db0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c500 tx=0x7f92b800c810 comp rx=0 tx=0).stop 2024-03-29T09:55:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 -- 172.21.15.12:0/2178515239 shutdown_connections 2024-03-29T09:55:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f92ac06ac90 0x7f92ac06d150 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92cc1611b0 0x7f92cc0c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92cc0cee10 0x7f92cc1f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 --2- 172.21.15.12:0/2178515239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92cc15f4d0 0x7f92cc1f2db0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 -- 172.21.15.12:0/2178515239 >> 172.21.15.12:0/2178515239 conn(0x7f92cc08aff0 msgr2=0x7f92cc06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 -- 172.21.15.12:0/2178515239 shutdown_connections 2024-03-29T09:55:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.604+0000 7f92d0e00700 1 -- 172.21.15.12:0/2178515239 wait complete. 2024-03-29T09:55:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T09:55:23.623 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T09:55:23.674 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T09:55:23.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.827+0000 7f5dafa00700 1 Processor -- start 2024-03-29T09:55:23.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.827+0000 7f5dafa00700 1 -- start start 2024-03-29T09:55:23.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.828+0000 7f5dafa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 0x7f5da815e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:23.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.828+0000 7f5dafa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da815eda0 con 0x7f5da80cd5c0 2024-03-29T09:55:23.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.828+0000 7f5dafa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da8099dd0 con 0x7f5da815cfe0 2024-03-29T09:55:23.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.828+0000 7f5dafa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da809a0b0 con 0x7f5da806fa20 2024-03-29T09:55:23.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.829+0000 7f5dad000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5da80cd5c0 0x7f5da806f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47532/0 (socket says 172.21.15.12:47532) 2024-03-29T09:55:23.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.829+0000 7f5dad000700 1 -- 172.21.15.12:0/1978440128 learned_addr learned my addr 172.21.15.12:0/1978440128 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:23.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.829+0000 7f5da7e00700 1 --2- 172.21.15.12:0/1978440128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 0x7f5da815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:23.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.829+0000 7f5da7400700 1 -- 172.21.15.12:0/1978440128 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364275223 0 0) 0x7f5d9c001110 con 0x7f5da80cd5c0 2024-03-29T09:55:23.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.829+0000 7f5da7400700 1 -- 172.21.15.12:0/1978440128 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d900035b0 con 0x7f5da80cd5c0 2024-03-29T09:55:23.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.829+0000 7f5da7400700 1 -- 172.21.15.12:0/1978440128 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994521173 0 0) 0x7f5da8099dd0 con 0x7f5da815cfe0 2024-03-29T09:55:23.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.829+0000 7f5da7400700 1 -- 172.21.15.12:0/1978440128 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d9c001110 con 0x7f5da815cfe0 2024-03-29T09:55:23.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.829+0000 7f5da7e00700 1 -- 172.21.15.12:0/1978440128 >> v1:172.21.15.159:6789/0 conn(0x7f5da815cfe0 legacy=0x7f5da815d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.829+0000 7f5da7e00700 1 -- 172.21.15.12:0/1978440128 >> v1:172.21.15.12:6789/0 conn(0x7f5da80cd5c0 legacy=0x7f5da806f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.829+0000 7f5da7e00700 1 -- 172.21.15.12:0/1978440128 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da809a390 con 0x7f5da806fa20 2024-03-29T09:55:23.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.830+0000 7f5da7e00700 1 --2- 172.21.15.12:0/1978440128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 0x7f5da815e860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009870 tx=0x7f5d94009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf82e20fd5865a42 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:23.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.830+0000 7f5da7400700 1 -- 172.21.15.12:0/1978440128 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94011830 con 0x7f5da806fa20 2024-03-29T09:55:23.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.830+0000 7f5da7400700 1 -- 172.21.15.12:0/1978440128 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d94011990 con 0x7f5da806fa20 2024-03-29T09:55:23.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.830+0000 7f5da7400700 1 -- 172.21.15.12:0/1978440128 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94011d20 con 0x7f5da806fa20 2024-03-29T09:55:23.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.830+0000 7f5dafa00700 1 -- 172.21.15.12:0/1978440128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 msgr2=0x7f5da815e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.830+0000 7f5dafa00700 1 --2- 172.21.15.12:0/1978440128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 0x7f5da815e860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009870 tx=0x7f5d94009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:23.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.830+0000 7f5dafa00700 1 -- 172.21.15.12:0/1978440128 shutdown_connections 2024-03-29T09:55:23.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.831+0000 7f5dafa00700 1 --2- 172.21.15.12:0/1978440128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 0x7f5da815e860 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.831+0000 7f5dafa00700 1 -- 172.21.15.12:0/1978440128 >> 172.21.15.12:0/1978440128 conn(0x7f5da80d1370 msgr2=0x7f5da80d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:23.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.831+0000 7f5dafa00700 1 -- 172.21.15.12:0/1978440128 shutdown_connections 2024-03-29T09:55:23.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.831+0000 7f5dafa00700 1 -- 172.21.15.12:0/1978440128 wait complete. 2024-03-29T09:55:23.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.833+0000 7f5dafa00700 1 Processor -- start 2024-03-29T09:55:23.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.833+0000 7f5dafa00700 1 -- start start 2024-03-29T09:55:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.834+0000 7f5dafa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 0x7f5da80b5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.834+0000 7f5dafa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5da80cd5c0 0x7f5da80ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.834+0000 7f5dafa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5da815cfe0 0x7f5da80ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.834+0000 7f5dafa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da8154d90 con 0x7f5da80cd5c0 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.834+0000 7f5dafa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da80af510 con 0x7f5da815cfe0 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.834+0000 7f5dafa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da80af7f0 con 0x7f5da806fa20 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5da7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5da80cd5c0 0x7f5da80ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5da7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5da80cd5c0 0x7f5da80ae160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39840/0 (socket says 172.21.15.12:39840) 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5da7e00700 1 -- 172.21.15.12:0/1824957677 learned_addr learned my addr 172.21.15.12:0/1824957677 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5dad000700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 0x7f5da80b5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5dada00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5da815cfe0 0x7f5da80ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5da7e00700 1 -- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 msgr2=0x7f5da80b5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5da7e00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 0x7f5da80b5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5da7e00700 1 -- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5da815cfe0 msgr2=0x7f5da80ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5da7e00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5da815cfe0 0x7f5da80ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5da7e00700 1 -- 172.21.15.12:0/1824957677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940094a0 con 0x7f5da80cd5c0 2024-03-29T09:55:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5dad000700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 0x7f5da80b5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5dada00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5da815cfe0 0x7f5da80ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.835+0000 7f5da7e00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5da80cd5c0 0x7f5da80ae160 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5d940090b0 tx=0x7f5d94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.836+0000 7f5da4c00700 1 -- 172.21.15.12:0/1824957677 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9401a6e0 con 0x7f5da80cd5c0 2024-03-29T09:55:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.836+0000 7f5da4c00700 1 -- 172.21.15.12:0/1824957677 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9401a840 con 0x7f5da80cd5c0 2024-03-29T09:55:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.836+0000 7f5da4c00700 1 -- 172.21.15.12:0/1824957677 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9401abb0 con 0x7f5da80cd5c0 2024-03-29T09:55:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.836+0000 7f5dafa00700 1 -- 172.21.15.12:0/1824957677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da80afad0 con 0x7f5da80cd5c0 2024-03-29T09:55:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.836+0000 7f5dafa00700 1 -- 172.21.15.12:0/1824957677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da80b61e0 con 0x7f5da80cd5c0 2024-03-29T09:55:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.837+0000 7f5dafa00700 1 -- 172.21.15.12:0/1824957677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da80b0bf0 con 0x7f5da80cd5c0 2024-03-29T09:55:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.838+0000 7f5da4c00700 1 -- 172.21.15.12:0/1824957677 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5d94022070 con 0x7f5da80cd5c0 2024-03-29T09:55:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.839+0000 7f5da4c00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d8806ac90 0x7f5d8806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.839+0000 7f5da4c00700 1 -- 172.21.15.12:0/1824957677 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f5d9408a380 con 0x7f5da80cd5c0 2024-03-29T09:55:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.839+0000 7f5dad000700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d8806ac90 0x7f5d8806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.840+0000 7f5dad000700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d8806ac90 0x7f5d8806d150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c000f60 tx=0x7f5d9c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:23.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:23.843+0000 7f5da4c00700 1 -- 172.21.15.12:0/1824957677 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5d940555a0 con 0x7f5da80cd5c0 2024-03-29T09:55:24.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.100+0000 7f5dafa00700 1 -- 172.21.15.12:0/1824957677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5da8067720 con 0x7f5da80cd5c0 2024-03-29T09:55:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.603+0000 7f5da4c00700 1 -- 172.21.15.12:0/1824957677 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5d94058bc0 con 0x7f5da80cd5c0 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 -- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d8806ac90 msgr2=0x7f5d8806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d8806ac90 0x7f5d8806d150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c000f60 tx=0x7f5d9c009040 comp rx=0 tx=0).stop 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 -- 172.21.15.12:0/1824957677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5da80cd5c0 msgr2=0x7f5da80ae160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5da80cd5c0 0x7f5da80ae160 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5d940090b0 tx=0x7f5d94004750 comp rx=0 tx=0).stop 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 -- 172.21.15.12:0/1824957677 shutdown_connections 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d8806ac90 0x7f5d8806d150 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5da815cfe0 0x7f5da80ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5da806fa20 0x7f5da80b5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 --2- 172.21.15.12:0/1824957677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5da80cd5c0 0x7f5da80ae160 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 -- 172.21.15.12:0/1824957677 >> 172.21.15.12:0/1824957677 conn(0x7f5da80d1370 msgr2=0x7f5da806eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 -- 172.21.15.12:0/1824957677 shutdown_connections 2024-03-29T09:55:24.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.607+0000 7f5dafa00700 1 -- 172.21.15.12:0/1824957677 wait complete. 2024-03-29T09:55:24.628 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T09:55:24.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.832+0000 7fb34b800700 1 Processor -- start 2024-03-29T09:55:24.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.833+0000 7fb34b800700 1 -- start start 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb34b800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb3441580c0 0x7fb34406e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb34b800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb34406e7f0 0x7fb34415ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb34b800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34406ec50 con 0x7fb34415b160 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb34b800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb344071570 con 0x7fb3441580c0 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb34b800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb344071990 con 0x7fb34406e7f0 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb349800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb34415b160 0x7fb34415df60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47536/0 (socket says 172.21.15.12:47536) 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb349800700 1 -- 172.21.15.12:0/3986587160 learned_addr learned my addr 172.21.15.12:0/3986587160 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb348e00700 1 --2- 172.21.15.12:0/3986587160 >> v2:172.21.15.159:3300/0 conn(0x7fb3441580c0 0x7fb34406e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb343e00700 1 --2- 172.21.15.12:0/3986587160 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb34406e7f0 0x7fb34415ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb343400700 1 -- 172.21.15.12:0/3986587160 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571561892 0 0) 0x7fb34406ec50 con 0x7fb34415b160 2024-03-29T09:55:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.834+0000 7fb343400700 1 -- 172.21.15.12:0/3986587160 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb32c0035b0 con 0x7fb34415b160 2024-03-29T09:55:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.835+0000 7fb348e00700 1 -- 172.21.15.12:0/3986587160 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb34406e7f0 msgr2=0x7fb34415ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.835+0000 7fb348e00700 1 --2- 172.21.15.12:0/3986587160 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb34406e7f0 0x7fb34415ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.835+0000 7fb348e00700 1 -- 172.21.15.12:0/3986587160 >> v1:172.21.15.12:6789/0 conn(0x7fb34415b160 legacy=0x7fb34415df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.835+0000 7fb343400700 1 -- 172.21.15.12:0/3986587160 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2380922031 0 0) 0x7fb32c0035b0 con 0x7fb34415b160 2024-03-29T09:55:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.835+0000 7fb348e00700 1 -- 172.21.15.12:0/3986587160 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb344070980 con 0x7fb3441580c0 2024-03-29T09:55:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.835+0000 7fb348e00700 1 --2- 172.21.15.12:0/3986587160 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3441580c0 0x7fb34406e2b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb338009870 tx=0x7fb338009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7a66b3a9efd6c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.835+0000 7fb343400700 1 -- 172.21.15.12:0/3986587160 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338011830 con 0x7fb3441580c0 2024-03-29T09:55:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.835+0000 7fb343400700 1 -- 172.21.15.12:0/3986587160 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb338011990 con 0x7fb3441580c0 2024-03-29T09:55:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.835+0000 7fb343400700 1 -- 172.21.15.12:0/3986587160 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338011d20 con 0x7fb3441580c0 2024-03-29T09:55:24.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.836+0000 7fb34b800700 1 -- 172.21.15.12:0/3986587160 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3441580c0 msgr2=0x7fb34406e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:24.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.836+0000 7fb34b800700 1 --2- 172.21.15.12:0/3986587160 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3441580c0 0x7fb34406e2b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb338009870 tx=0x7fb338009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:24.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.836+0000 7fb34b800700 1 -- 172.21.15.12:0/3986587160 shutdown_connections 2024-03-29T09:55:24.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.836+0000 7fb34b800700 1 --2- 172.21.15.12:0/3986587160 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3441580c0 0x7fb34406e2b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.836+0000 7fb34b800700 1 --2- 172.21.15.12:0/3986587160 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb34406e7f0 0x7fb34415ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.836+0000 7fb34b800700 1 -- 172.21.15.12:0/3986587160 >> 172.21.15.12:0/3986587160 conn(0x7fb3440cf210 msgr2=0x7fb3440cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.836+0000 7fb34b800700 1 -- 172.21.15.12:0/3986587160 shutdown_connections 2024-03-29T09:55:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.836+0000 7fb34b800700 1 -- 172.21.15.12:0/3986587160 wait complete. 2024-03-29T09:55:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.838+0000 7fb34b800700 1 Processor -- start 2024-03-29T09:55:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.838+0000 7fb34b800700 1 -- start start 2024-03-29T09:55:24.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb34b800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34406e7f0 0x7fb34415ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:24.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb34b800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3441580c0 0x7fb3440b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:24.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb34b800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb34415b160 0x7fb3440b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:24.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb34b800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb344159de0 con 0x7fb34406e7f0 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb34b800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb344158eb0 con 0x7fb34415b160 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb34b800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb344159160 con 0x7fb3441580c0 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb348e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34406e7f0 0x7fb34415ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb348e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34406e7f0 0x7fb34415ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39854/0 (socket says 172.21.15.12:39854) 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb348e00700 1 -- 172.21.15.12:0/1732298496 learned_addr learned my addr 172.21.15.12:0/1732298496 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb343e00700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3441580c0 0x7fb3440b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.839+0000 7fb349800700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb34415b160 0x7fb3440b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb348e00700 1 -- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3441580c0 msgr2=0x7fb3440b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb348e00700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3441580c0 0x7fb3440b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb348e00700 1 -- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb34415b160 msgr2=0x7fb3440b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb348e00700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb34415b160 0x7fb3440b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb348e00700 1 -- 172.21.15.12:0/1732298496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3380094a0 con 0x7fb34406e7f0 2024-03-29T09:55:24.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb348e00700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34406e7f0 0x7fb34415ca40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb3380090b0 tx=0x7fb338004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:24.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb340c00700 1 -- 172.21.15.12:0/1732298496 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33801a970 con 0x7fb34406e7f0 2024-03-29T09:55:24.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb340c00700 1 -- 172.21.15.12:0/1732298496 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33801aad0 con 0x7fb34406e7f0 2024-03-29T09:55:24.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb34b800700 1 -- 172.21.15.12:0/1732298496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb344159440 con 0x7fb34406e7f0 2024-03-29T09:55:24.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb340c00700 1 -- 172.21.15.12:0/1732298496 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33801ae60 con 0x7fb34406e7f0 2024-03-29T09:55:24.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.840+0000 7fb34b800700 1 -- 172.21.15.12:0/1732298496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb344159a00 con 0x7fb34406e7f0 2024-03-29T09:55:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.841+0000 7fb34b800700 1 -- 172.21.15.12:0/1732298496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3440b4710 con 0x7fb34406e7f0 2024-03-29T09:55:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.842+0000 7fb340c00700 1 -- 172.21.15.12:0/1732298496 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb33802d450 con 0x7fb34406e7f0 2024-03-29T09:55:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.843+0000 7fb340c00700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb32406ac90 0x7fb32406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.843+0000 7fb340c00700 1 -- 172.21.15.12:0/1732298496 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fb33808a760 con 0x7fb34406e7f0 2024-03-29T09:55:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.844+0000 7fb343e00700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb32406ac90 0x7fb32406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.844+0000 7fb343e00700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb32406ac90 0x7fb32406d150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb34415a790 tx=0x7fb330009210 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:24.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:24.848+0000 7fb340c00700 1 -- 172.21.15.12:0/1732298496 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb338055840 con 0x7fb34406e7f0 2024-03-29T09:55:25.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.062+0000 7fb34b800700 1 -- 172.21.15.12:0/1732298496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb344070e70 con 0x7fb34406e7f0 2024-03-29T09:55:25.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.605+0000 7fb340c00700 1 -- 172.21.15.12:0/1732298496 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb3380150d0 con 0x7fb34406e7f0 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 -- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb32406ac90 msgr2=0x7fb32406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb32406ac90 0x7fb32406d150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb34415a790 tx=0x7fb330009210 comp rx=0 tx=0).stop 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 -- 172.21.15.12:0/1732298496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34406e7f0 msgr2=0x7fb34415ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34406e7f0 0x7fb34415ca40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb3380090b0 tx=0x7fb338004750 comp rx=0 tx=0).stop 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 -- 172.21.15.12:0/1732298496 shutdown_connections 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb32406ac90 0x7fb32406d150 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb34415b160 0x7fb3440b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34406e7f0 0x7fb34415ca40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 --2- 172.21.15.12:0/1732298496 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3441580c0 0x7fb3440b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 -- 172.21.15.12:0/1732298496 >> 172.21.15.12:0/1732298496 conn(0x7fb3440cf210 msgr2=0x7fb344071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 -- 172.21.15.12:0/1732298496 shutdown_connections 2024-03-29T09:55:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.610+0000 7fb34b800700 1 -- 172.21.15.12:0/1732298496 wait complete. 2024-03-29T09:55:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T09:55:25.631 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T09:55:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.835+0000 7f1f10c00700 1 Processor -- start 2024-03-29T09:55:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.836+0000 7f1f10c00700 1 -- start start 2024-03-29T09:55:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.837+0000 7f1f10c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f0c0d28e0 0x7f1f0c168680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.837+0000 7f1f10c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0c0ce8a0 con 0x7f1f0c15ddf0 2024-03-29T09:55:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.837+0000 7f1f10c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0c0cdc90 con 0x7f1f0c0ced30 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.837+0000 7f1f10c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c15ec60 con 0x7f1f0c0d28e0 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.837+0000 7f1f09e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1f0c15ddf0 0x7f1f0c15e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47542/0 (socket says 172.21.15.12:47542) 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.837+0000 7f1f09e00700 1 -- 172.21.15.12:0/256110671 learned_addr learned my addr 172.21.15.12:0/256110671 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.837+0000 7f1f0a800700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f0c0d28e0 0x7f1f0c168680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.837+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403428986 0 0) 0x7f1f0c0ce8a0 con 0x7f1f0c15ddf0 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef00035b0 con 0x7f1f0c15ddf0 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015763090 0 0) 0x7f1f0c0cdc90 con 0x7f1f0c0ced30 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f0c0ce8a0 con 0x7f1f0c0ced30 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 471681181 0 0) 0x7f1ef00035b0 con 0x7f1f0c15ddf0 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f0c0d28e0 msgr2=0x7f1f0c168680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f0c0d28e0 0x7f1f0c168680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 >> v1:172.21.15.159:6789/0 conn(0x7f1f0c0ced30 legacy=0x7f1f0c0d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c0cdf70 con 0x7f1f0c15ddf0 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f1efc001670 con 0x7f1f0c15ddf0 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 >> v1:172.21.15.12:6789/0 conn(0x7f1f0c15ddf0 legacy=0x7f1f0c15e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ef0003bc0 0x7f1ef0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ef0004850 0x7f1ef0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ef0005210 0x7f1ef000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c15ec60 con 0x7f1ef0004850 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef000bf80 con 0x7f1ef0003bc0 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef000c140 con 0x7f1ef0005210 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f09400700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ef0004850 0x7f1ef0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.838+0000 7f1f09e00700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ef0003bc0 0x7f1ef0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.839+0000 7f1f0a800700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ef0005210 0x7f1ef000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.839+0000 7f1f09400700 1 -- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ef0005210 msgr2=0x7f1ef000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.839+0000 7f1f09400700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ef0005210 0x7f1ef000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.839+0000 7f1f09400700 1 -- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ef0003bc0 msgr2=0x7f1ef0003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.839+0000 7f1f09400700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ef0003bc0 0x7f1ef0003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.839+0000 7f1f09400700 1 -- 172.21.15.12:0/256110671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef00035b0 con 0x7f1ef0004850 2024-03-29T09:55:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.839+0000 7f1f09400700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ef0004850 0x7f1ef0004cd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8001940 tx=0x7f1ef0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfdbf5c3a4812d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.839+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8001430 con 0x7f1ef0004850 2024-03-29T09:55:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.839+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8001d70 con 0x7f1ef0004850 2024-03-29T09:55:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.839+0000 7f1f08a00700 1 -- 172.21.15.12:0/256110671 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8005510 con 0x7f1ef0004850 2024-03-29T09:55:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.840+0000 7f1f10c00700 1 -- 172.21.15.12:0/256110671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ef0004850 msgr2=0x7f1ef0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.840+0000 7f1f10c00700 1 --2- 172.21.15.12:0/256110671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ef0004850 0x7f1ef0004cd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8001940 tx=0x7f1ef0005880 comp rx=0 tx=0).stop 2024-03-29T09:55:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.840+0000 7f1f0a800700 1 -- 172.21.15.12:0/256110671 reap_dead start 2024-03-29T09:55:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.840+0000 7f1f10c00700 1 -- 172.21.15.12:0/256110671 shutdown_connections 2024-03-29T09:55:25.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.840+0000 7f1f10c00700 1 -- 172.21.15.12:0/256110671 >> 172.21.15.12:0/256110671 conn(0x7f1f0c08aff0 msgr2=0x7f1f0c0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:25.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.840+0000 7f1f10c00700 1 -- 172.21.15.12:0/256110671 shutdown_connections 2024-03-29T09:55:25.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.840+0000 7f1f10c00700 1 -- 172.21.15.12:0/256110671 wait complete. 2024-03-29T09:55:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.842+0000 7f1f10c00700 1 Processor -- start 2024-03-29T09:55:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.842+0000 7f1f10c00700 1 -- start start 2024-03-29T09:55:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f10c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ef0004850 0x7f1f0c0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f10c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c0ca5e0 0x7f1f0c1588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f10c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f0c0b7280 0x7f1f0c0b7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f10c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c0c9c50 con 0x7f1f0c0ca5e0 2024-03-29T09:55:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f10c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c0c8310 con 0x7f1ef0004850 2024-03-29T09:55:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f10c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c0b7c40 con 0x7f1f0c0b7280 2024-03-29T09:55:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f09400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c0ca5e0 0x7f1f0c1588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f09400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c0ca5e0 0x7f1f0c1588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39868/0 (socket says 172.21.15.12:39868) 2024-03-29T09:55:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f09400700 1 -- 172.21.15.12:0/3794351520 learned_addr learned my addr 172.21.15.12:0/3794351520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f09e00700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ef0004850 0x7f1f0c0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.843+0000 7f1f0a800700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f0c0b7280 0x7f1f0c0b7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f09400700 1 -- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f0c0b7280 msgr2=0x7f1f0c0b7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f09400700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f0c0b7280 0x7f1f0c0b7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f09400700 1 -- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ef0004850 msgr2=0x7f1f0c0c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f09400700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ef0004850 0x7f1f0c0c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f09400700 1 -- 172.21.15.12:0/3794351520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c0cb760 con 0x7f1f0c0ca5e0 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f0a800700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f0c0b7280 0x7f1f0c0b7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f09400700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c0ca5e0 0x7f1f0c1588d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80055b0 tx=0x7f1ef80012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f02000700 1 -- 172.21.15.12:0/3794351520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8006c70 con 0x7f1f0c0ca5e0 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f02000700 1 -- 172.21.15.12:0/3794351520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8006dd0 con 0x7f1f0c0ca5e0 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f10c00700 1 -- 172.21.15.12:0/3794351520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0c0cab50 con 0x7f1f0c0ca5e0 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f02000700 1 -- 172.21.15.12:0/3794351520 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8007260 con 0x7f1f0c0ca5e0 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.844+0000 7f1f10c00700 1 -- 172.21.15.12:0/3794351520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0c0cb110 con 0x7f1f0c0ca5e0 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.845+0000 7f1f10c00700 1 -- 172.21.15.12:0/3794351520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f0c0cb8f0 con 0x7f1f0c0ca5e0 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.846+0000 7f1f02000700 1 -- 172.21.15.12:0/3794351520 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1ef80073c0 con 0x7f1f0c0ca5e0 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.847+0000 7f1f02000700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1eec06ac90 0x7f1eec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.848+0000 7f1f02000700 1 -- 172.21.15.12:0/3794351520 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f1ef808a350 con 0x7f1f0c0ca5e0 2024-03-29T09:55:25.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.848+0000 7f1f09e00700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1eec06ac90 0x7f1eec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:25.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.849+0000 7f1f09e00700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1eec06ac90 0x7f1eec06d150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0c7140 tx=0x7f1efc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:25.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:25.851+0000 7f1f02000700 1 -- 172.21.15.12:0/3794351520 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ef80555f0 con 0x7f1f0c0ca5e0 2024-03-29T09:55:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.071+0000 7f1f10c00700 1 -- 172.21.15.12:0/3794351520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1f0c0cf110 con 0x7f1f0c0ca5e0 2024-03-29T09:55:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.611+0000 7f1f02000700 1 -- 172.21.15.12:0/3794351520 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1ef8058c10 con 0x7f1f0c0ca5e0 2024-03-29T09:55:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 -- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1eec06ac90 msgr2=0x7f1eec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1eec06ac90 0x7f1eec06d150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0c7140 tx=0x7f1efc009040 comp rx=0 tx=0).stop 2024-03-29T09:55:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 -- 172.21.15.12:0/3794351520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c0ca5e0 msgr2=0x7f1f0c1588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c0ca5e0 0x7f1f0c1588d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80055b0 tx=0x7f1ef80012d0 comp rx=0 tx=0).stop 2024-03-29T09:55:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 -- 172.21.15.12:0/3794351520 shutdown_connections 2024-03-29T09:55:26.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1eec06ac90 0x7f1eec06d150 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:26.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f0c0b7280 0x7f1f0c0b7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:26.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ef0004850 0x7f1f0c0c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:26.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 --2- 172.21.15.12:0/3794351520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c0ca5e0 0x7f1f0c1588d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:26.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 -- 172.21.15.12:0/3794351520 >> 172.21.15.12:0/3794351520 conn(0x7f1f0c08aff0 msgr2=0x7f1f0c0b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:26.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 -- 172.21.15.12:0/3794351520 shutdown_connections 2024-03-29T09:55:26.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.614+0000 7f1f10c00700 1 -- 172.21.15.12:0/3794351520 wait complete. 2024-03-29T09:55:26.616 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T09:55:26.634 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T09:55:26.634 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T09:55:26.634 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T09:55:26.634 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T09:55:26.635 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T09:55:26.635 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T09:55:26.635 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T09:55:26.635 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T09:55:26.635 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T09:55:26.635 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T09:55:26.636 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T09:55:26.636 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T09:55:26.636 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T09:55:26.636 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T09:55:26.636 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T09:55:26.636 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T09:55:26.651 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T09:55:26.651 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T09:55:26.652 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T09:55:26.652 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T09:55:26.653 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T09:55:26.653 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T09:55:26.654 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T09:55:26.654 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T09:55:26.696 INFO:tasks.ceph.mds.c:Started 2024-03-29T09:55:26.696 DEBUG:teuthology.parallel:result is None 2024-03-29T09:55:26.697 INFO:tasks.ceph.mds.d:Started 2024-03-29T09:55:26.698 DEBUG:teuthology.parallel:result is None 2024-03-29T09:55:26.698 INFO:tasks.ceph.mds.a:Started 2024-03-29T09:55:26.698 DEBUG:teuthology.parallel:result is None 2024-03-29T09:55:26.700 INFO:tasks.ceph.mds.b:Started 2024-03-29T09:55:26.700 DEBUG:teuthology.parallel:result is None 2024-03-29T09:55:26.700 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T09:55:26.842 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T09:55:26.842 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T09:55:26.846 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T09:55:26.847 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T09:55:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.923+0000 7f64af600700 1 Processor -- start 2024-03-29T09:55:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.924+0000 7f64af600700 1 -- start start 2024-03-29T09:55:26.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.925+0000 7f64af600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f64a815ef70 0x7f64a80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:26.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.925+0000 7f64af600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a8161150 con 0x7f64a806a790 2024-03-29T09:55:26.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.925+0000 7f64af600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a806dba0 con 0x7f64a8160d70 2024-03-29T09:55:26.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.925+0000 7f64af600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a80b4cd0 con 0x7f64a815ef70 2024-03-29T09:55:26.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.925+0000 7f64ad600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f64a806a790 0x7f64a806e080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47558/0 (socket says 172.21.15.12:47558) 2024-03-29T09:55:26.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.925+0000 7f64ad600700 1 -- 172.21.15.12:0/982485007 learned_addr learned my addr 172.21.15.12:0/982485007 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:26.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.925+0000 7f64a7e00700 1 --2- 172.21.15.12:0/982485007 >> v2:172.21.15.159:3300/0 conn(0x7f64a815ef70 0x7f64a80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:26.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7400700 1 -- 172.21.15.12:0/982485007 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767303857 0 0) 0x7f64a8161150 con 0x7f64a806a790 2024-03-29T09:55:26.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7400700 1 -- 172.21.15.12:0/982485007 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64900035b0 con 0x7f64a806a790 2024-03-29T09:55:26.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7400700 1 -- 172.21.15.12:0/982485007 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320326970 0 0) 0x7f64a806dba0 con 0x7f64a8160d70 2024-03-29T09:55:26.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7400700 1 -- 172.21.15.12:0/982485007 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a8161150 con 0x7f64a8160d70 2024-03-29T09:55:26.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7e00700 1 -- 172.21.15.12:0/982485007 >> v1:172.21.15.159:6789/0 conn(0x7f64a8160d70 legacy=0x7f64a815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:26.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7e00700 1 -- 172.21.15.12:0/982485007 >> v1:172.21.15.12:6789/0 conn(0x7f64a806a790 legacy=0x7f64a806e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:26.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7e00700 1 -- 172.21.15.12:0/982485007 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a80b4ee0 con 0x7f64a815ef70 2024-03-29T09:55:26.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7e00700 1 --2- 172.21.15.12:0/982485007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64a815ef70 0x7f64a80b3200 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6494009840 tx=0x7f6494009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=950e183fcc7e26f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:26.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7400700 1 -- 172.21.15.12:0/982485007 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494011830 con 0x7f64a815ef70 2024-03-29T09:55:26.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7400700 1 -- 172.21.15.12:0/982485007 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6494011990 con 0x7f64a815ef70 2024-03-29T09:55:26.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.926+0000 7f64a7400700 1 -- 172.21.15.12:0/982485007 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494011d20 con 0x7f64a815ef70 2024-03-29T09:55:26.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.927+0000 7f64af600700 1 -- 172.21.15.12:0/982485007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64a815ef70 msgr2=0x7f64a80b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:26.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.927+0000 7f64af600700 1 --2- 172.21.15.12:0/982485007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64a815ef70 0x7f64a80b3200 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6494009840 tx=0x7f6494009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:26.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.927+0000 7f64af600700 1 -- 172.21.15.12:0/982485007 shutdown_connections 2024-03-29T09:55:26.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.927+0000 7f64af600700 1 --2- 172.21.15.12:0/982485007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64a815ef70 0x7f64a80b3200 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:26.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.927+0000 7f64af600700 1 -- 172.21.15.12:0/982485007 >> 172.21.15.12:0/982485007 conn(0x7f64a80b5720 msgr2=0x7f64a80b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:26.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.927+0000 7f64af600700 1 -- 172.21.15.12:0/982485007 shutdown_connections 2024-03-29T09:55:26.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.927+0000 7f64af600700 1 -- 172.21.15.12:0/982485007 wait complete. 2024-03-29T09:55:26.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.929+0000 7f64af600700 1 Processor -- start 2024-03-29T09:55:26.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.929+0000 7f64af600700 1 -- start start 2024-03-29T09:55:26.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64af600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64a806a790 0x7f64a81f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:26.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64af600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64a815ef70 0x7f64a80b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:26.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64af600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64a8160d70 0x7f64a80af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:26.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64af600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a80b2610 con 0x7f64a8160d70 2024-03-29T09:55:26.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64af600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a81f3090 con 0x7f64a806a790 2024-03-29T09:55:26.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64af600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a80b1c70 con 0x7f64a815ef70 2024-03-29T09:55:26.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64ad600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64a8160d70 0x7f64a80af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:26.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64ad600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64a8160d70 0x7f64a80af930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39910/0 (socket says 172.21.15.12:39910) 2024-03-29T09:55:26.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64ad600700 1 -- 172.21.15.12:0/3544622264 learned_addr learned my addr 172.21.15.12:0/3544622264 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:26.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64acc00700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64a806a790 0x7f64a81f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:26.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64a7e00700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64a815ef70 0x7f64a80b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:26.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64ad600700 1 -- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64a815ef70 msgr2=0x7f64a80b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:26.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64ad600700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64a815ef70 0x7f64a80b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:26.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64ad600700 1 -- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64a806a790 msgr2=0x7f64a81f2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:26.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64ad600700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64a806a790 0x7f64a81f2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:26.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64ad600700 1 -- 172.21.15.12:0/3544622264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64940094a0 con 0x7f64a8160d70 2024-03-29T09:55:26.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64acc00700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64a806a790 0x7f64a81f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:26.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.930+0000 7f64a7e00700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64a815ef70 0x7f64a80b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:26.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.931+0000 7f64ad600700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64a8160d70 0x7f64a80af930 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f64a000cbc0 tx=0x7f64a000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:26.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.931+0000 7f64a4c00700 1 -- 172.21.15.12:0/3544622264 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a000ab50 con 0x7f64a8160d70 2024-03-29T09:55:26.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.931+0000 7f64a4c00700 1 -- 172.21.15.12:0/3544622264 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a000acb0 con 0x7f64a8160d70 2024-03-29T09:55:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.931+0000 7f64a4c00700 1 -- 172.21.15.12:0/3544622264 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0012600 con 0x7f64a8160d70 2024-03-29T09:55:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.931+0000 7f64af600700 1 -- 172.21.15.12:0/3544622264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a80b1f50 con 0x7f64a8160d70 2024-03-29T09:55:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.931+0000 7f64af600700 1 -- 172.21.15.12:0/3544622264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a80b2f70 con 0x7f64a8160d70 2024-03-29T09:55:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.932+0000 7f64af600700 1 -- 172.21.15.12:0/3544622264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a80b2750 con 0x7f64a8160d70 2024-03-29T09:55:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.933+0000 7f64a4c00700 1 -- 172.21.15.12:0/3544622264 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f64a001b3b0 con 0x7f64a8160d70 2024-03-29T09:55:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.934+0000 7f64a4c00700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f648806ac90 0x7f648806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.934+0000 7f64a4c00700 1 -- 172.21.15.12:0/3544622264 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f64a0088d60 con 0x7f64a8160d70 2024-03-29T09:55:26.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.934+0000 7f64acc00700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f648806ac90 0x7f648806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:26.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.935+0000 7f64acc00700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f648806ac90 0x7f648806d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f649c001220 tx=0x7f649c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:26.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:26.938+0000 7f64a4c00700 1 -- 172.21.15.12:0/3544622264 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64a80b2750 con 0x7f64a8160d70 2024-03-29T09:55:27.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.169+0000 7f64af600700 1 -- 172.21.15.12:0/3544622264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f64a80b2750 con 0x7f64a8160d70 2024-03-29T09:55:27.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.896+0000 7f64a4c00700 1 -- 172.21.15.12:0/3544622264 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7f64a80b2750 con 0x7f64a8160d70 2024-03-29T09:55:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 -- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f648806ac90 msgr2=0x7f648806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f648806ac90 0x7f648806d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f649c001220 tx=0x7f649c009040 comp rx=0 tx=0).stop 2024-03-29T09:55:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 -- 172.21.15.12:0/3544622264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64a8160d70 msgr2=0x7f64a80af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64a8160d70 0x7f64a80af930 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f64a000cbc0 tx=0x7f64a000cf80 comp rx=0 tx=0).stop 2024-03-29T09:55:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 -- 172.21.15.12:0/3544622264 shutdown_connections 2024-03-29T09:55:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f648806ac90 0x7f648806d150 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64a8160d70 0x7f64a80af930 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64a806a790 0x7f64a81f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 --2- 172.21.15.12:0/3544622264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64a815ef70 0x7f64a80b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 -- 172.21.15.12:0/3544622264 >> 172.21.15.12:0/3544622264 conn(0x7f64a80b5720 msgr2=0x7f64a806df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 -- 172.21.15.12:0/3544622264 shutdown_connections 2024-03-29T09:55:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:27.899+0000 7f64af600700 1 -- 172.21.15.12:0/3544622264 wait complete. 2024-03-29T09:55:27.900 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T09:55:27.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T09:55:28.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.125+0000 7fb747600700 1 Processor -- start 2024-03-29T09:55:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.125+0000 7fb747600700 1 -- start start 2024-03-29T09:55:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.126+0000 7fb747600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb740161ca0 0x7fb7400cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.126+0000 7fb747600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7400ceda0 con 0x7fb740160f10 2024-03-29T09:55:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.126+0000 7fb747600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7401612f0 con 0x7fb7400d18b0 2024-03-29T09:55:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.126+0000 7fb747600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7400d1d30 con 0x7fb740161ca0 2024-03-29T09:55:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.126+0000 7fb744c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb740160f10 0x7fb7400d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47568/0 (socket says 172.21.15.12:47568) 2024-03-29T09:55:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.126+0000 7fb744c00700 1 -- 172.21.15.12:0/1352997849 learned_addr learned my addr 172.21.15.12:0/1352997849 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.126+0000 7fb745600700 1 --2- 172.21.15.12:0/1352997849 >> v2:172.21.15.159:3300/0 conn(0x7fb740161ca0 0x7fb7400cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9506674 0 0) 0x7fb7400ceda0 con 0x7fb740160f10 2024-03-29T09:55:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7280035b0 con 0x7fb740160f10 2024-03-29T09:55:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1443933927 0 0) 0x7fb7280035b0 con 0x7fb740160f10 2024-03-29T09:55:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 >> v2:172.21.15.159:3300/0 conn(0x7fb740161ca0 msgr2=0x7fb7400cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 --2- 172.21.15.12:0/1352997849 >> v2:172.21.15.159:3300/0 conn(0x7fb740161ca0 0x7fb7400cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 >> v1:172.21.15.159:6789/0 conn(0x7fb7400d18b0 legacy=0x7fb740161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400ce470 con 0x7fb740160f10 2024-03-29T09:55:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fb734001670 con 0x7fb740160f10 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 >> v1:172.21.15.12:6789/0 conn(0x7fb740160f10 legacy=0x7fb7400d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb728003a50 0x7fb728003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb728004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb728005260 0x7fb72800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400d1d30 con 0x7fb728004850 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72800c1b0 con 0x7fb728003a50 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72800c370 con 0x7fb728005260 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb744c00700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb728003a50 0x7fb728003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.127+0000 7fb73fe00700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb728004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb745600700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb728005260 0x7fb72800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb73fe00700 1 -- 172.21.15.12:0/1352997849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb728005260 msgr2=0x7fb72800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb73fe00700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb728005260 0x7fb72800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb73fe00700 1 -- 172.21.15.12:0/1352997849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb728003a50 msgr2=0x7fb728003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb73fe00700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb728003a50 0x7fb728003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb73fe00700 1 -- 172.21.15.12:0/1352997849 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7280035b0 con 0x7fb728004850 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb745600700 1 -- 172.21.15.12:0/1352997849 reap_dead start 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb73fe00700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb728004cd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb72c007fd0 tx=0x7fb728005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85afa8378811284 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0052d0 con 0x7fb728004850 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c005450 con 0x7fb728004850 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.128+0000 7fb73f400700 1 -- 172.21.15.12:0/1352997849 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0038c0 con 0x7fb728004850 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.129+0000 7fb747600700 1 -- 172.21.15.12:0/1352997849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 msgr2=0x7fb728004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.129+0000 7fb747600700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb728004cd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb72c007fd0 tx=0x7fb728005880 comp rx=0 tx=0).stop 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.129+0000 7fb747600700 1 -- 172.21.15.12:0/1352997849 shutdown_connections 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.129+0000 7fb747600700 1 --2- 172.21.15.12:0/1352997849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb728004cd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.129+0000 7fb747600700 1 -- 172.21.15.12:0/1352997849 >> 172.21.15.12:0/1352997849 conn(0x7fb74008aff0 msgr2=0x7fb7400d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.129+0000 7fb747600700 1 -- 172.21.15.12:0/1352997849 shutdown_connections 2024-03-29T09:55:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.129+0000 7fb747600700 1 -- 172.21.15.12:0/1352997849 wait complete. 2024-03-29T09:55:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.131+0000 7fb747600700 1 Processor -- start 2024-03-29T09:55:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.131+0000 7fb747600700 1 -- start start 2024-03-29T09:55:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb747600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb74015a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb747600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb74015fdf0 0x7fb740160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb747600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7400cc850 0x7fb7400cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb747600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400cd210 con 0x7fb728004850 2024-03-29T09:55:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb747600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74015b180 con 0x7fb7400cc850 2024-03-29T09:55:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb747600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74015f090 con 0x7fb74015fdf0 2024-03-29T09:55:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb744c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb74015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb745600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7400cc850 0x7fb7400cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb744c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb74015a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39922/0 (socket says 172.21.15.12:39922) 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb745600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7400cc850 0x7fb7400cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:55454/0 (socket says 172.21.15.12:55454) 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb744c00700 1 -- 172.21.15.12:0/3116301623 learned_addr learned my addr 172.21.15.12:0/3116301623 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.132+0000 7fb73fe00700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb74015fdf0 0x7fb740160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb744c00700 1 -- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb74015fdf0 msgr2=0x7fb740160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb744c00700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb74015fdf0 0x7fb740160270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb744c00700 1 -- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7400cc850 msgr2=0x7fb7400cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb744c00700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7400cc850 0x7fb7400cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb744c00700 1 -- 172.21.15.12:0/3116301623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74015f370 con 0x7fb728004850 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb744c00700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb74015a860 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb734000c00 tx=0x7fb734007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb73cc00700 1 -- 172.21.15.12:0/3116301623 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734011580 con 0x7fb728004850 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb73cc00700 1 -- 172.21.15.12:0/3116301623 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb734011720 con 0x7fb728004850 2024-03-29T09:55:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb747600700 1 -- 172.21.15.12:0/3116301623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74015f650 con 0x7fb728004850 2024-03-29T09:55:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb73cc00700 1 -- 172.21.15.12:0/3116301623 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734011ad0 con 0x7fb728004850 2024-03-29T09:55:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.133+0000 7fb747600700 1 -- 172.21.15.12:0/3116301623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74009d720 con 0x7fb728004850 2024-03-29T09:55:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.134+0000 7fb747600700 1 -- 172.21.15.12:0/3116301623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb740067720 con 0x7fb728004850 2024-03-29T09:55:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.135+0000 7fb73cc00700 1 -- 172.21.15.12:0/3116301623 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb734011c30 con 0x7fb728004850 2024-03-29T09:55:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.136+0000 7fb73cc00700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb72006ac90 0x7fb72006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.136+0000 7fb73cc00700 1 -- 172.21.15.12:0/3116301623 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb7340890e0 con 0x7fb728004850 2024-03-29T09:55:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.136+0000 7fb73fe00700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb72006ac90 0x7fb72006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.137+0000 7fb73fe00700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb72006ac90 0x7fb72006d150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb72c005040 tx=0x7fb72c009030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.141+0000 7fb73cc00700 1 -- 172.21.15.12:0/3116301623 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7340544e0 con 0x7fb728004850 2024-03-29T09:55:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.437+0000 7fb747600700 1 -- 172.21.15.12:0/3116301623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb74008b760 con 0x7fb728004850 2024-03-29T09:55:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.437+0000 7fb73cc00700 1 -- 172.21.15.12:0/3116301623 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fb734057b00 con 0x7fb728004850 2024-03-29T09:55:28.437 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:28.437 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T09:55:28.437 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T09:55:28.437 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.437 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T09:55:28.437 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T09:55:28.437 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.437 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T09:55:28.437 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.438 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.439 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.440 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T09:55:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T09:55:28.442 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.443 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T09:55:28.444 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.445 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T09:55:28.446 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.447 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-mds", 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQC6jwZmWsx9LxAAgs8Ik4jaIfq+fdPekpmG6Q==", 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-mds" 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.448 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-mgr", 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQC6jwZm+dV9LxAAQwKY1tTUWAwBDNpIUYYTNA==", 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-osd", 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQC6jwZmQ959LxAABzmPgkC63JzOtTvAwkjEjw==", 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-osd" 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.449 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-rbd", 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQC6jwZmoul9LxAAcoPdLZWohwhyPRWB60x/Lw==", 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQC6jwZmQ/N9LxAAyNn1Uhx/6qcWLu9mSKsFpw==", 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-rgw", 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQC6jwZmOPx9LxAANscpNN5x+FJZa8SAvR9q0w==", 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:28.451 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T09:55:28.452 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T09:55:28.452 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T09:55:28.452 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T09:55:28.452 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T09:55:28.452 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T09:55:28.452 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.452 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:28.452 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T09:55:28.452 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T09:55:28.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 -- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb72006ac90 msgr2=0x7fb72006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb72006ac90 0x7fb72006d150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb72c005040 tx=0x7fb72c009030 comp rx=0 tx=0).stop 2024-03-29T09:55:28.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 -- 172.21.15.12:0/3116301623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 msgr2=0x7fb74015a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb74015a860 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb734000c00 tx=0x7fb734007570 comp rx=0 tx=0).stop 2024-03-29T09:55:28.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 -- 172.21.15.12:0/3116301623 shutdown_connections 2024-03-29T09:55:28.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb72006ac90 0x7fb72006d150 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7400cc850 0x7fb7400cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb728004850 0x7fb74015a860 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 --2- 172.21.15.12:0/3116301623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb74015fdf0 0x7fb740160270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 -- 172.21.15.12:0/3116301623 >> 172.21.15.12:0/3116301623 conn(0x7fb74008aff0 msgr2=0x7fb74015b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:28.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 -- 172.21.15.12:0/3116301623 shutdown_connections 2024-03-29T09:55:28.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.440+0000 7fb747600700 1 -- 172.21.15.12:0/3116301623 wait complete. 2024-03-29T09:55:28.458 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-29T09:55:28.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.657+0000 7f0fbf600700 1 Processor -- start 2024-03-29T09:55:28.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.657+0000 7f0fbf600700 1 -- start start 2024-03-29T09:55:28.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.658+0000 7f0fbf600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0fb815dad0 0x7f0fb80ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.658+0000 7f0fbf600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0fb80d21d0 0x7f0fb815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.658+0000 7f0fbf600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb815f820 con 0x7f0fb80d1630 2024-03-29T09:55:28.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.658+0000 7f0fbf600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb80cfcc0 con 0x7f0fb815dad0 2024-03-29T09:55:28.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.658+0000 7f0fbf600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb80cd9f0 con 0x7f0fb80d21d0 2024-03-29T09:55:28.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.658+0000 7f0fbcc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0fb815dad0 0x7f0fb80ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.658+0000 7f0fbcc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0fb815dad0 0x7f0fb80ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39930/0 (socket says 172.21.15.12:39930) 2024-03-29T09:55:28.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.658+0000 7f0fbcc00700 1 -- 172.21.15.12:0/3035818875 learned_addr learned my addr 172.21.15.12:0/3035818875 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:28.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.658+0000 7f0fbd600700 1 --2- 172.21.15.12:0/3035818875 >> v2:172.21.15.159:3300/0 conn(0x7f0fb80d21d0 0x7f0fb815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.659+0000 7f0fbcc00700 1 -- 172.21.15.12:0/3035818875 >> v2:172.21.15.159:3300/0 conn(0x7f0fb80d21d0 msgr2=0x7f0fb815fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:28.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.659+0000 7f0fbcc00700 1 --2- 172.21.15.12:0/3035818875 >> v2:172.21.15.159:3300/0 conn(0x7f0fb80d21d0 0x7f0fb815fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.659+0000 7f0fbcc00700 1 -- 172.21.15.12:0/3035818875 >> v1:172.21.15.159:6789/0 conn(0x7f0fb80d1630 legacy=0x7f0fb80d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.659+0000 7f0fbcc00700 1 -- 172.21.15.12:0/3035818875 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb80cf420 con 0x7f0fb815dad0 2024-03-29T09:55:28.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.659+0000 7f0fbcc00700 1 --2- 172.21.15.12:0/3035818875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 0x7f0fb80ced80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0fb808ccb0 tx=0x7f0fac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcfe433d7deb5b73 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:28.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.659+0000 7f0fb7400700 1 -- 172.21.15.12:0/3035818875 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac011830 con 0x7f0fb815dad0 2024-03-29T09:55:28.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.659+0000 7f0fb7400700 1 -- 172.21.15.12:0/3035818875 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac011990 con 0x7f0fb815dad0 2024-03-29T09:55:28.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.659+0000 7f0fb7400700 1 -- 172.21.15.12:0/3035818875 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac011d20 con 0x7f0fb815dad0 2024-03-29T09:55:28.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.660+0000 7f0fbf600700 1 -- 172.21.15.12:0/3035818875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 msgr2=0x7f0fb80ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.660+0000 7f0fbf600700 1 --2- 172.21.15.12:0/3035818875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 0x7f0fb80ced80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0fb808ccb0 tx=0x7f0fac009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.660+0000 7f0fbf600700 1 -- 172.21.15.12:0/3035818875 shutdown_connections 2024-03-29T09:55:28.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.660+0000 7f0fbf600700 1 --2- 172.21.15.12:0/3035818875 >> v2:172.21.15.159:3300/0 conn(0x7f0fb80d21d0 0x7f0fb815fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.660+0000 7f0fbf600700 1 --2- 172.21.15.12:0/3035818875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 0x7f0fb80ced80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.660+0000 7f0fbf600700 1 -- 172.21.15.12:0/3035818875 >> 172.21.15.12:0/3035818875 conn(0x7f0fb808aff0 msgr2=0x7f0fb80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:28.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.660+0000 7f0fbf600700 1 -- 172.21.15.12:0/3035818875 shutdown_connections 2024-03-29T09:55:28.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.660+0000 7f0fbf600700 1 -- 172.21.15.12:0/3035818875 wait complete. 2024-03-29T09:55:28.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.662+0000 7f0fbf600700 1 Processor -- start 2024-03-29T09:55:28.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.662+0000 7f0fbf600700 1 -- start start 2024-03-29T09:55:28.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fbf600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb80d1630 0x7f0fb814dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fbf600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fb80d21d0 0x7f0fb80c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fbf600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 0x7f0fb80c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fbf600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb80c8810 con 0x7f0fb815dad0 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fbf600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb80cd3f0 con 0x7f0fb80d1630 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fbf600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb80c7b90 con 0x7f0fb80d21d0 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fbd600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 0x7f0fb80c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fbd600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 0x7f0fb80c7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39942/0 (socket says 172.21.15.12:39942) 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fbd600700 1 -- 172.21.15.12:0/2320255476 learned_addr learned my addr 172.21.15.12:0/2320255476 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fbcc00700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb80d1630 0x7f0fb814dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.663+0000 7f0fb7e00700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fb80d21d0 0x7f0fb80c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fbd600700 1 -- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fb80d21d0 msgr2=0x7f0fb80c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fbd600700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fb80d21d0 0x7f0fb80c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fbd600700 1 -- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb80d1630 msgr2=0x7f0fb814dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fbd600700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb80d1630 0x7f0fb814dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fbd600700 1 -- 172.21.15.12:0/2320255476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac009550 con 0x7f0fb815dad0 2024-03-29T09:55:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fbcc00700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb80d1630 0x7f0fb814dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fbd600700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 0x7f0fb80c7560 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000efc0 tx=0x7f0fb000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fb4c00700 1 -- 172.21.15.12:0/2320255476 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0009ea0 con 0x7f0fb815dad0 2024-03-29T09:55:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fb4c00700 1 -- 172.21.15.12:0/2320255476 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb000f040 con 0x7f0fb815dad0 2024-03-29T09:55:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fb4c00700 1 -- 172.21.15.12:0/2320255476 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb000ced0 con 0x7f0fb815dad0 2024-03-29T09:55:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fbf600700 1 -- 172.21.15.12:0/2320255476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb80c7e70 con 0x7f0fb815dad0 2024-03-29T09:55:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.664+0000 7f0fbf600700 1 -- 172.21.15.12:0/2320255476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb80c8430 con 0x7f0fb815dad0 2024-03-29T09:55:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.665+0000 7f0fbf600700 1 -- 172.21.15.12:0/2320255476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb8067720 con 0x7f0fb815dad0 2024-03-29T09:55:28.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.666+0000 7f0fb4c00700 1 -- 172.21.15.12:0/2320255476 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0fb0014480 con 0x7f0fb815dad0 2024-03-29T09:55:28.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.667+0000 7f0fb4c00700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9806ac90 0x7f0f9806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:28.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.668+0000 7f0fb4c00700 1 -- 172.21.15.12:0/2320255476 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0fb00897e0 con 0x7f0fb815dad0 2024-03-29T09:55:28.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.668+0000 7f0fbcc00700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9806ac90 0x7f0f9806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:28.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.669+0000 7f0fbcc00700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9806ac90 0x7f0f9806d150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0fac014040 tx=0x7f0fac008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:28.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.672+0000 7f0fb4c00700 1 -- 172.21.15.12:0/2320255476 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0fb0054d00 con 0x7f0fb815dad0 2024-03-29T09:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.973+0000 7f0fbf600700 1 -- 172.21.15.12:0/2320255476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f0fb80cef60 con 0x7f0fb815dad0 2024-03-29T09:55:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.978+0000 7f0fb4c00700 1 -- 172.21.15.12:0/2320255476 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0fb0058320 con 0x7f0fb815dad0 2024-03-29T09:55:28.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 -- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9806ac90 msgr2=0x7f0f9806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9806ac90 0x7f0f9806d150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0fac014040 tx=0x7f0fac008030 comp rx=0 tx=0).stop 2024-03-29T09:55:28.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 -- 172.21.15.12:0/2320255476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 msgr2=0x7f0fb80c7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:28.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 0x7f0fb80c7560 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000efc0 tx=0x7f0fb000c3a0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 -- 172.21.15.12:0/2320255476 shutdown_connections 2024-03-29T09:55:28.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9806ac90 0x7f0f9806d150 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb815dad0 0x7f0fb80c7560 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb80d1630 0x7f0fb814dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 --2- 172.21.15.12:0/2320255476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fb80d21d0 0x7f0fb80c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:28.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 -- 172.21.15.12:0/2320255476 >> 172.21.15.12:0/2320255476 conn(0x7f0fb808aff0 msgr2=0x7f0fb8150b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:28.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 -- 172.21.15.12:0/2320255476 shutdown_connections 2024-03-29T09:55:28.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:28.981+0000 7f0fbf600700 1 -- 172.21.15.12:0/2320255476 wait complete. 2024-03-29T09:55:28.999 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-29T09:55:29.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.198+0000 7effac400700 1 Processor -- start 2024-03-29T09:55:29.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.198+0000 7effac400700 1 -- start start 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.199+0000 7effac400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7effa415d9d0 0x7effa40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.199+0000 7effac400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effa40d16a0 0x7effa40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.199+0000 7effac400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa4161420 con 0x7effa40d2370 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.199+0000 7effac400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa40d2820 con 0x7effa415d9d0 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.199+0000 7effac400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa4161700 con 0x7effa40d16a0 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.199+0000 7effa9a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7effa415d9d0 0x7effa40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.199+0000 7effa9a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7effa415d9d0 0x7effa40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49764/0 (socket says 172.21.15.12:49764) 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.199+0000 7effa9a00700 1 -- 172.21.15.12:0/3700820527 learned_addr learned my addr 172.21.15.12:0/3700820527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.200+0000 7effa9000700 1 --2- 172.21.15.12:0/3700820527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effa40d16a0 0x7effa40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.200+0000 7effa9a00700 1 -- 172.21.15.12:0/3700820527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effa40d16a0 msgr2=0x7effa40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.200+0000 7effa9a00700 1 --2- 172.21.15.12:0/3700820527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effa40d16a0 0x7effa40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.200+0000 7effa9a00700 1 -- 172.21.15.12:0/3700820527 >> v1:172.21.15.12:6789/0 conn(0x7effa40d2370 legacy=0x7effa415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.200+0000 7effa9a00700 1 -- 172.21.15.12:0/3700820527 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa41619e0 con 0x7effa415d9d0 2024-03-29T09:55:29.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.200+0000 7effa9a00700 1 --2- 172.21.15.12:0/3700820527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 0x7effa40d3c30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7eff980098f0 tx=0x7eff98009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e15831db461cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:29.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.200+0000 7effa3e00700 1 -- 172.21.15.12:0/3700820527 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9800b8a0 con 0x7effa415d9d0 2024-03-29T09:55:29.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.200+0000 7effa3e00700 1 -- 172.21.15.12:0/3700820527 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9800ba00 con 0x7effa415d9d0 2024-03-29T09:55:29.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.200+0000 7effa3e00700 1 -- 172.21.15.12:0/3700820527 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9800bd90 con 0x7effa415d9d0 2024-03-29T09:55:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.201+0000 7effac400700 1 -- 172.21.15.12:0/3700820527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 msgr2=0x7effa40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.201+0000 7effac400700 1 --2- 172.21.15.12:0/3700820527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 0x7effa40d3c30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7eff980098f0 tx=0x7eff98009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.201+0000 7effac400700 1 -- 172.21.15.12:0/3700820527 shutdown_connections 2024-03-29T09:55:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.201+0000 7effac400700 1 --2- 172.21.15.12:0/3700820527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 0x7effa40d3c30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.201+0000 7effac400700 1 --2- 172.21.15.12:0/3700820527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effa40d16a0 0x7effa40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.201+0000 7effac400700 1 -- 172.21.15.12:0/3700820527 >> 172.21.15.12:0/3700820527 conn(0x7effa408aff0 msgr2=0x7effa40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.201+0000 7effac400700 1 -- 172.21.15.12:0/3700820527 shutdown_connections 2024-03-29T09:55:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.201+0000 7effac400700 1 -- 172.21.15.12:0/3700820527 wait complete. 2024-03-29T09:55:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.203+0000 7effac400700 1 Processor -- start 2024-03-29T09:55:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.204+0000 7effac400700 1 -- start start 2024-03-29T09:55:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effac400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effa40d16a0 0x7effa4155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effac400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7effa40d2370 0x7effa40b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effac400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 0x7effa4159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effac400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa415e810 con 0x7effa415d9d0 2024-03-29T09:55:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effac400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa40b8df0 con 0x7effa40d2370 2024-03-29T09:55:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effac400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa415dee0 con 0x7effa40d16a0 2024-03-29T09:55:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effaa400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 0x7effa4159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effaa400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 0x7effa4159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49770/0 (socket says 172.21.15.12:49770) 2024-03-29T09:55:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effaa400700 1 -- 172.21.15.12:0/2148269291 learned_addr learned my addr 172.21.15.12:0/2148269291 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effa9a00700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effa40d16a0 0x7effa4155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.205+0000 7effa9000700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7effa40d2370 0x7effa40b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effaa400700 1 -- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effa40d16a0 msgr2=0x7effa4155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effaa400700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effa40d16a0 0x7effa4155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effaa400700 1 -- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7effa40d2370 msgr2=0x7effa40b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effaa400700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7effa40d2370 0x7effa40b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effaa400700 1 -- 172.21.15.12:0/2148269291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98009550 con 0x7effa415d9d0 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effaa400700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 0x7effa4159c50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00cc50 tx=0x7eff9c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effa1600700 1 -- 172.21.15.12:0/2148269291 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c003c00 con 0x7effa415d9d0 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effa1600700 1 -- 172.21.15.12:0/2148269291 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c003d60 con 0x7effa415d9d0 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effac400700 1 -- 172.21.15.12:0/2148269291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa415e1c0 con 0x7effa415d9d0 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effa1600700 1 -- 172.21.15.12:0/2148269291 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c00aab0 con 0x7effa415d9d0 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.206+0000 7effac400700 1 -- 172.21.15.12:0/2148269291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa4155560 con 0x7effa415d9d0 2024-03-29T09:55:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.207+0000 7effac400700 1 -- 172.21.15.12:0/2148269291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa40d1ca0 con 0x7effa415d9d0 2024-03-29T09:55:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.208+0000 7effa1600700 1 -- 172.21.15.12:0/2148269291 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7eff9c00ac10 con 0x7effa415d9d0 2024-03-29T09:55:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.209+0000 7effa1600700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff8406ac90 0x7eff8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.209+0000 7effa1600700 1 -- 172.21.15.12:0/2148269291 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7eff9c088f90 con 0x7effa415d9d0 2024-03-29T09:55:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.210+0000 7effa9a00700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff8406ac90 0x7eff8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.210+0000 7effa9a00700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff8406ac90 0x7eff8406d150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7eff98014040 tx=0x7eff98008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.213+0000 7effa1600700 1 -- 172.21.15.12:0/2148269291 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff9c0544b0 con 0x7effa415d9d0 2024-03-29T09:55:29.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.510+0000 7effac400700 1 -- 172.21.15.12:0/2148269291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7effa4067720 con 0x7effa415d9d0 2024-03-29T09:55:29.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.514+0000 7effa1600700 1 -- 172.21.15.12:0/2148269291 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7eff9c057ad0 con 0x7effa415d9d0 2024-03-29T09:55:29.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.516+0000 7effac400700 1 -- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff8406ac90 msgr2=0x7eff8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:29.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.516+0000 7effac400700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff8406ac90 0x7eff8406d150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7eff98014040 tx=0x7eff98008030 comp rx=0 tx=0).stop 2024-03-29T09:55:29.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.516+0000 7effac400700 1 -- 172.21.15.12:0/2148269291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 msgr2=0x7effa4159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.516+0000 7effac400700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 0x7effa4159c50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00cc50 tx=0x7eff9c00cf60 comp rx=0 tx=0).stop 2024-03-29T09:55:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.517+0000 7effac400700 1 -- 172.21.15.12:0/2148269291 shutdown_connections 2024-03-29T09:55:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.517+0000 7effac400700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff8406ac90 0x7eff8406d150 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.517+0000 7effac400700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effa415d9d0 0x7effa4159c50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.517+0000 7effac400700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effa40d16a0 0x7effa4155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.517+0000 7effac400700 1 --2- 172.21.15.12:0/2148269291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7effa40d2370 0x7effa40b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.517+0000 7effac400700 1 -- 172.21.15.12:0/2148269291 >> 172.21.15.12:0/2148269291 conn(0x7effa408aff0 msgr2=0x7effa40b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.517+0000 7effac400700 1 -- 172.21.15.12:0/2148269291 shutdown_connections 2024-03-29T09:55:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.517+0000 7effac400700 1 -- 172.21.15.12:0/2148269291 wait complete. 2024-03-29T09:55:29.534 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-29T09:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.733+0000 7f76d0c00700 1 Processor -- start 2024-03-29T09:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.734+0000 7f76d0c00700 1 -- start start 2024-03-29T09:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.735+0000 7f76d0c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f76cc15d8a0 0x7f76cc0b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.735+0000 7f76d0c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc0cf1b0 0x7f76cc0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.735+0000 7f76d0c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc0ced20 con 0x7f76cc0d3eb0 2024-03-29T09:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.735+0000 7f76d0c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0d4340 con 0x7f76cc15d8a0 2024-03-29T09:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.735+0000 7f76d0c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0ce6d0 con 0x7f76cc0cf1b0 2024-03-29T09:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.735+0000 7f76c9400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f76cc0d3eb0 0x7f76cc06b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44192/0 (socket says 172.21.15.12:44192) 2024-03-29T09:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.735+0000 7f76c9400700 1 -- 172.21.15.12:0/31310616 learned_addr learned my addr 172.21.15.12:0/31310616 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.735+0000 7f76c9e00700 1 --2- 172.21.15.12:0/31310616 >> v2:172.21.15.159:3300/0 conn(0x7f76cc15d8a0 0x7f76cc0b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.735+0000 7f76ca800700 1 --2- 172.21.15.12:0/31310616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc0cf1b0 0x7f76cc0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.735+0000 7f76c8a00700 1 -- 172.21.15.12:0/31310616 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159395261 0 0) 0x7f76cc0ced20 con 0x7f76cc0d3eb0 2024-03-29T09:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.736+0000 7f76c8a00700 1 -- 172.21.15.12:0/31310616 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b00035b0 con 0x7f76cc0d3eb0 2024-03-29T09:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.736+0000 7f76c9e00700 1 -- 172.21.15.12:0/31310616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc0cf1b0 msgr2=0x7f76cc0cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.736+0000 7f76c9e00700 1 --2- 172.21.15.12:0/31310616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc0cf1b0 0x7f76cc0cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.736+0000 7f76c9e00700 1 -- 172.21.15.12:0/31310616 >> v1:172.21.15.12:6789/0 conn(0x7f76cc0d3eb0 legacy=0x7f76cc06b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.736+0000 7f76c9e00700 1 -- 172.21.15.12:0/31310616 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc0ce9b0 con 0x7f76cc15d8a0 2024-03-29T09:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.736+0000 7f76c9e00700 1 --2- 172.21.15.12:0/31310616 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc15d8a0 0x7f76cc0b2990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009840 tx=0x7f76bc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be10bab38062a289 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.736+0000 7f76c8a00700 1 -- 172.21.15.12:0/31310616 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc011830 con 0x7f76cc15d8a0 2024-03-29T09:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.736+0000 7f76c8a00700 1 -- 172.21.15.12:0/31310616 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76bc011990 con 0x7f76cc15d8a0 2024-03-29T09:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.736+0000 7f76c8a00700 1 -- 172.21.15.12:0/31310616 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc011d20 con 0x7f76cc15d8a0 2024-03-29T09:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.737+0000 7f76d0c00700 1 -- 172.21.15.12:0/31310616 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc15d8a0 msgr2=0x7f76cc0b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.737+0000 7f76d0c00700 1 --2- 172.21.15.12:0/31310616 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc15d8a0 0x7f76cc0b2990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009840 tx=0x7f76bc009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.737+0000 7f76d0c00700 1 -- 172.21.15.12:0/31310616 shutdown_connections 2024-03-29T09:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.737+0000 7f76d0c00700 1 --2- 172.21.15.12:0/31310616 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc0cf1b0 0x7f76cc0cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.737+0000 7f76d0c00700 1 --2- 172.21.15.12:0/31310616 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc15d8a0 0x7f76cc0b2990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.737+0000 7f76d0c00700 1 -- 172.21.15.12:0/31310616 >> 172.21.15.12:0/31310616 conn(0x7f76cc08aff0 msgr2=0x7f76cc0d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.737+0000 7f76d0c00700 1 -- 172.21.15.12:0/31310616 shutdown_connections 2024-03-29T09:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.737+0000 7f76d0c00700 1 -- 172.21.15.12:0/31310616 wait complete. 2024-03-29T09:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.739+0000 7f76d0c00700 1 Processor -- start 2024-03-29T09:55:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.740+0000 7f76d0c00700 1 -- start start 2024-03-29T09:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76d0c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc0cf1b0 0x7f76cc0cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76d0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76cc15d8a0 0x7f76cc0b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76d0c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc1f2990 0x7f76cc1f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76d0c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0b4ff0 con 0x7f76cc15d8a0 2024-03-29T09:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76d0c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0b5ac0 con 0x7f76cc0cf1b0 2024-03-29T09:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76d0c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc1f2e40 con 0x7f76cc1f2990 2024-03-29T09:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76c9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76cc15d8a0 0x7f76cc0b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76c9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76cc15d8a0 0x7f76cc0b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49786/0 (socket says 172.21.15.12:49786) 2024-03-29T09:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76c9400700 1 -- 172.21.15.12:0/1196519971 learned_addr learned my addr 172.21.15.12:0/1196519971 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76c9e00700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc0cf1b0 0x7f76cc0cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.741+0000 7f76ca800700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc1f2990 0x7f76cc1f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76c9400700 1 -- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc1f2990 msgr2=0x7f76cc1f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76c9400700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc1f2990 0x7f76cc1f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76c9400700 1 -- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc0cf1b0 msgr2=0x7f76cc0cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76c9400700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc0cf1b0 0x7f76cc0cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76c9400700 1 -- 172.21.15.12:0/1196519971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc0094a0 con 0x7f76cc15d8a0 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76c9400700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76cc15d8a0 0x7f76cc0b4ab0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f76b8001260 tx=0x7f76b800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76c2000700 1 -- 172.21.15.12:0/1196519971 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80074e0 con 0x7f76cc15d8a0 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76c2000700 1 -- 172.21.15.12:0/1196519971 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b8007640 con 0x7f76cc15d8a0 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76d0c00700 1 -- 172.21.15.12:0/1196519971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc0b5220 con 0x7f76cc15d8a0 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76c2000700 1 -- 172.21.15.12:0/1196519971 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8005a50 con 0x7f76cc15d8a0 2024-03-29T09:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.742+0000 7f76d0c00700 1 -- 172.21.15.12:0/1196519971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc0b57e0 con 0x7f76cc15d8a0 2024-03-29T09:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.743+0000 7f76d0c00700 1 -- 172.21.15.12:0/1196519971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc067720 con 0x7f76cc15d8a0 2024-03-29T09:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.744+0000 7f76c2000700 1 -- 172.21.15.12:0/1196519971 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f76b8005bb0 con 0x7f76cc15d8a0 2024-03-29T09:55:29.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.745+0000 7f76c2000700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76ac06ac90 0x7f76ac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:29.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.745+0000 7f76c2000700 1 -- 172.21.15.12:0/1196519971 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f76b80884d0 con 0x7f76cc15d8a0 2024-03-29T09:55:29.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.746+0000 7f76c9e00700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76ac06ac90 0x7f76ac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:29.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.747+0000 7f76c9e00700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76ac06ac90 0x7f76ac06d150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f76bc000c00 tx=0x7f76bc01c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:29.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:29.750+0000 7f76c2000700 1 -- 172.21.15.12:0/1196519971 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76b80573f0 con 0x7f76cc15d8a0 2024-03-29T09:55:30.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.049+0000 7f76d0c00700 1 -- 172.21.15.12:0/1196519971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f76cc0d1950 con 0x7f76cc15d8a0 2024-03-29T09:55:30.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.053+0000 7f76c2000700 1 -- 172.21.15.12:0/1196519971 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f76b80170a0 con 0x7f76cc15d8a0 2024-03-29T09:55:30.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 -- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76ac06ac90 msgr2=0x7f76ac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76ac06ac90 0x7f76ac06d150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f76bc000c00 tx=0x7f76bc01c040 comp rx=0 tx=0).stop 2024-03-29T09:55:30.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 -- 172.21.15.12:0/1196519971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76cc15d8a0 msgr2=0x7f76cc0b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76cc15d8a0 0x7f76cc0b4ab0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f76b8001260 tx=0x7f76b800c9e0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 -- 172.21.15.12:0/1196519971 shutdown_connections 2024-03-29T09:55:30.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76ac06ac90 0x7f76ac06d150 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc1f2990 0x7f76cc1f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc0cf1b0 0x7f76cc0cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 --2- 172.21.15.12:0/1196519971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76cc15d8a0 0x7f76cc0b4ab0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 -- 172.21.15.12:0/1196519971 >> 172.21.15.12:0/1196519971 conn(0x7f76cc08aff0 msgr2=0x7f76cc15ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:30.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 -- 172.21.15.12:0/1196519971 shutdown_connections 2024-03-29T09:55:30.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.055+0000 7f76d0c00700 1 -- 172.21.15.12:0/1196519971 wait complete. 2024-03-29T09:55:30.073 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-29T09:55:30.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.272+0000 7f617a000700 1 Processor -- start 2024-03-29T09:55:30.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.272+0000 7f617a000700 1 -- start start 2024-03-29T09:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.273+0000 7f617a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f617415a5c0 0x7f61740709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.273+0000 7f617a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f617415dfe0 0x7f61740ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.273+0000 7f617a000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61740cb310 con 0x7f61740ce670 2024-03-29T09:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.273+0000 7f617a000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61740ceb00 con 0x7f617415a5c0 2024-03-29T09:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.273+0000 7f617a000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6174070f00 con 0x7f617415dfe0 2024-03-29T09:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.273+0000 7f6173400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f617415a5c0 0x7f61740709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.273+0000 7f6173400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f617415a5c0 0x7f61740709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49792/0 (socket says 172.21.15.12:49792) 2024-03-29T09:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.273+0000 7f6173400700 1 -- 172.21.15.12:0/2379896914 learned_addr learned my addr 172.21.15.12:0/2379896914 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:30.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.274+0000 7f6172a00700 1 --2- 172.21.15.12:0/2379896914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f617415dfe0 0x7f61740ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.274+0000 7f6173400700 1 -- 172.21.15.12:0/2379896914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f617415dfe0 msgr2=0x7f61740ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:30.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.274+0000 7f6173400700 1 --2- 172.21.15.12:0/2379896914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f617415dfe0 0x7f61740ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.274+0000 7f6173400700 1 -- 172.21.15.12:0/2379896914 >> v1:172.21.15.159:6789/0 conn(0x7f61740ce670 legacy=0x7f61740b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:30.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.274+0000 7f6173400700 1 -- 172.21.15.12:0/2379896914 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61740cca90 con 0x7f617415a5c0 2024-03-29T09:55:30.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.274+0000 7f6173400700 1 --2- 172.21.15.12:0/2379896914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415a5c0 0x7f61740709c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f61640098f0 tx=0x7f6164009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed23e5e9eb21cbc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:30.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.274+0000 7f6172000700 1 -- 172.21.15.12:0/2379896914 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164004230 con 0x7f617415a5c0 2024-03-29T09:55:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.274+0000 7f6172000700 1 -- 172.21.15.12:0/2379896914 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6164004390 con 0x7f617415a5c0 2024-03-29T09:55:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.274+0000 7f6172000700 1 -- 172.21.15.12:0/2379896914 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61640036a0 con 0x7f617415a5c0 2024-03-29T09:55:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.275+0000 7f617a000700 1 -- 172.21.15.12:0/2379896914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415a5c0 msgr2=0x7f61740709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.275+0000 7f617a000700 1 --2- 172.21.15.12:0/2379896914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415a5c0 0x7f61740709c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f61640098f0 tx=0x7f6164009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.275+0000 7f617a000700 1 -- 172.21.15.12:0/2379896914 shutdown_connections 2024-03-29T09:55:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.275+0000 7f617a000700 1 --2- 172.21.15.12:0/2379896914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415a5c0 0x7f61740709c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.275+0000 7f617a000700 1 --2- 172.21.15.12:0/2379896914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f617415dfe0 0x7f61740ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.275+0000 7f617a000700 1 -- 172.21.15.12:0/2379896914 >> 172.21.15.12:0/2379896914 conn(0x7f61740b0d70 msgr2=0x7f61740b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.275+0000 7f617a000700 1 -- 172.21.15.12:0/2379896914 shutdown_connections 2024-03-29T09:55:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.275+0000 7f617a000700 1 -- 172.21.15.12:0/2379896914 wait complete. 2024-03-29T09:55:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.277+0000 7f617a000700 1 Processor -- start 2024-03-29T09:55:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.277+0000 7f617a000700 1 -- start start 2024-03-29T09:55:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.278+0000 7f617a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61740ce670 0x7f6174099700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.278+0000 7f617a000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f617415a5c0 0x7f6174099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.278+0000 7f617a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415dfe0 0x7f61741f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.278+0000 7f617a000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61741569d0 con 0x7f617415dfe0 2024-03-29T09:55:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.278+0000 7f617a000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617409a470 con 0x7f617415a5c0 2024-03-29T09:55:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.278+0000 7f617a000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617409a750 con 0x7f61740ce670 2024-03-29T09:55:30.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6173e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415dfe0 0x7f61741f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6173e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415dfe0 0x7f61741f3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49806/0 (socket says 172.21.15.12:49806) 2024-03-29T09:55:30.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6173e00700 1 -- 172.21.15.12:0/2538919967 learned_addr learned my addr 172.21.15.12:0/2538919967 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:30.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6172a00700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f617415a5c0 0x7f6174099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6173400700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61740ce670 0x7f6174099700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6173e00700 1 -- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61740ce670 msgr2=0x7f6174099700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6173e00700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61740ce670 0x7f6174099700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6173e00700 1 -- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f617415a5c0 msgr2=0x7f6174099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6173e00700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f617415a5c0 0x7f6174099c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6173e00700 1 -- 172.21.15.12:0/2538919967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6164009550 con 0x7f617415dfe0 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f6173e00700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415dfe0 0x7f61741f3a80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f616c00cc10 tx=0x7f616c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f616b400700 1 -- 172.21.15.12:0/2538919967 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c00aa60 con 0x7f617415dfe0 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f616b400700 1 -- 172.21.15.12:0/2538919967 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c00abc0 con 0x7f617415dfe0 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.279+0000 7f616b400700 1 -- 172.21.15.12:0/2538919967 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c012420 con 0x7f617415dfe0 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.280+0000 7f617a000700 1 -- 172.21.15.12:0/2538919967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617409aa30 con 0x7f617415dfe0 2024-03-29T09:55:30.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.280+0000 7f617a000700 1 -- 172.21.15.12:0/2538919967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617409a180 con 0x7f617415dfe0 2024-03-29T09:55:30.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.281+0000 7f617a000700 1 -- 172.21.15.12:0/2538919967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6174156b40 con 0x7f617415dfe0 2024-03-29T09:55:30.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.282+0000 7f616b400700 1 -- 172.21.15.12:0/2538919967 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f616c012580 con 0x7f617415dfe0 2024-03-29T09:55:30.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.282+0000 7f616b400700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f615406ac90 0x7f615406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.282+0000 7f616b400700 1 -- 172.21.15.12:0/2538919967 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f616c0892b0 con 0x7f617415dfe0 2024-03-29T09:55:30.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.283+0000 7f6173400700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f615406ac90 0x7f615406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.284+0000 7f6173400700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f615406ac90 0x7f615406d150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6164009130 tx=0x7f6164008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.287+0000 7f616b400700 1 -- 172.21.15.12:0/2538919967 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f616c0547d0 con 0x7f617415dfe0 2024-03-29T09:55:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.583+0000 7f617a000700 1 -- 172.21.15.12:0/2538919967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f6174067720 con 0x7f617415dfe0 2024-03-29T09:55:30.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.586+0000 7f616b400700 1 -- 172.21.15.12:0/2538919967 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f616c057df0 con 0x7f617415dfe0 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 -- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f615406ac90 msgr2=0x7f615406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f615406ac90 0x7f615406d150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6164009130 tx=0x7f6164008030 comp rx=0 tx=0).stop 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 -- 172.21.15.12:0/2538919967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415dfe0 msgr2=0x7f61741f3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415dfe0 0x7f61741f3a80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f616c00cc10 tx=0x7f616c00cf20 comp rx=0 tx=0).stop 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 -- 172.21.15.12:0/2538919967 shutdown_connections 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f615406ac90 0x7f615406d150 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617415dfe0 0x7f61741f3a80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61740ce670 0x7f6174099700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 --2- 172.21.15.12:0/2538919967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f617415a5c0 0x7f6174099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 -- 172.21.15.12:0/2538919967 >> 172.21.15.12:0/2538919967 conn(0x7f61740b0d70 msgr2=0x7f617415e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 -- 172.21.15.12:0/2538919967 shutdown_connections 2024-03-29T09:55:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.589+0000 7f617a000700 1 -- 172.21.15.12:0/2538919967 wait complete. 2024-03-29T09:55:30.608 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-29T09:55:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.805+0000 7f94f6e00700 1 Processor -- start 2024-03-29T09:55:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.806+0000 7f94f6e00700 1 -- start start 2024-03-29T09:55:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.807+0000 7f94f6e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f94f015d9d0 0x7f94f00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.807+0000 7f94f6e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94f00d2540 0x7f94f00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.807+0000 7f94f6e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f0161420 con 0x7f94f00d16a0 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.807+0000 7f94f6e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f015edc0 con 0x7f94f015d9d0 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.807+0000 7f94f6e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0161700 con 0x7f94f00d2540 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.807+0000 7f94efe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f94f015d9d0 0x7f94f00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.807+0000 7f94efe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f94f015d9d0 0x7f94f00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49808/0 (socket says 172.21.15.12:49808) 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.807+0000 7f94efe00700 1 -- 172.21.15.12:0/1228756683 learned_addr learned my addr 172.21.15.12:0/1228756683 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.807+0000 7f94f4e00700 1 --2- 172.21.15.12:0/1228756683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94f00d2540 0x7f94f00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.808+0000 7f94efe00700 1 -- 172.21.15.12:0/1228756683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94f00d2540 msgr2=0x7f94f00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.808+0000 7f94efe00700 1 --2- 172.21.15.12:0/1228756683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94f00d2540 0x7f94f00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.808+0000 7f94efe00700 1 -- 172.21.15.12:0/1228756683 >> v1:172.21.15.12:6789/0 conn(0x7f94f00d16a0 legacy=0x7f94f00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.808+0000 7f94efe00700 1 -- 172.21.15.12:0/1228756683 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f01619e0 con 0x7f94f015d9d0 2024-03-29T09:55:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.808+0000 7f94eea00700 1 -- 172.21.15.12:0/1228756683 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960510571 0 0) 0x7f94f0161420 con 0x7f94f00d16a0 2024-03-29T09:55:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.808+0000 7f94efe00700 1 --2- 172.21.15.12:0/1228756683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f015d9d0 0x7f94f00d3c30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f94e00098f0 tx=0x7f94e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95d877eb162972a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.808+0000 7f94eea00700 1 -- 172.21.15.12:0/1228756683 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0011830 con 0x7f94f015d9d0 2024-03-29T09:55:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.808+0000 7f94eea00700 1 -- 172.21.15.12:0/1228756683 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e0011990 con 0x7f94f015d9d0 2024-03-29T09:55:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.808+0000 7f94eea00700 1 -- 172.21.15.12:0/1228756683 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0011d20 con 0x7f94f015d9d0 2024-03-29T09:55:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.809+0000 7f94f6e00700 1 -- 172.21.15.12:0/1228756683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f015d9d0 msgr2=0x7f94f00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.809+0000 7f94f6e00700 1 --2- 172.21.15.12:0/1228756683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f015d9d0 0x7f94f00d3c30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f94e00098f0 tx=0x7f94e0009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.809+0000 7f94f6e00700 1 -- 172.21.15.12:0/1228756683 shutdown_connections 2024-03-29T09:55:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.809+0000 7f94f6e00700 1 --2- 172.21.15.12:0/1228756683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94f00d2540 0x7f94f00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.809+0000 7f94f6e00700 1 --2- 172.21.15.12:0/1228756683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f015d9d0 0x7f94f00d3c30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.809+0000 7f94f6e00700 1 -- 172.21.15.12:0/1228756683 >> 172.21.15.12:0/1228756683 conn(0x7f94f008aff0 msgr2=0x7f94f00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.809+0000 7f94f6e00700 1 -- 172.21.15.12:0/1228756683 shutdown_connections 2024-03-29T09:55:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.809+0000 7f94f6e00700 1 -- 172.21.15.12:0/1228756683 wait complete. 2024-03-29T09:55:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.811+0000 7f94f6e00700 1 Processor -- start 2024-03-29T09:55:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.811+0000 7f94f6e00700 1 -- start start 2024-03-29T09:55:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94f6e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94f00d16a0 0x7f94f01529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94f6e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f00d2540 0x7f94f00cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94f6e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94f015d9d0 0x7f94f00b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94f6e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00cb840 con 0x7f94f00d2540 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94f6e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00cc1a0 con 0x7f94f00d16a0 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94f6e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00ccc00 con 0x7f94f015d9d0 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94ef400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f00d2540 0x7f94f00cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94ef400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f00d2540 0x7f94f00cbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49818/0 (socket says 172.21.15.12:49818) 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94ef400700 1 -- 172.21.15.12:0/1672977378 learned_addr learned my addr 172.21.15.12:0/1672977378 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94f4e00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94f015d9d0 0x7f94f00b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94efe00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94f00d16a0 0x7f94f01529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94ef400700 1 -- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94f015d9d0 msgr2=0x7f94f00b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94ef400700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94f015d9d0 0x7f94f00b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94ef400700 1 -- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94f00d16a0 msgr2=0x7f94f01529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94ef400700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94f00d16a0 0x7f94f01529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.812+0000 7f94ef400700 1 -- 172.21.15.12:0/1672977378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e0009550 con 0x7f94f00d2540 2024-03-29T09:55:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.813+0000 7f94ef400700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f00d2540 0x7f94f00cbc60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00c580 tx=0x7f94dc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.813+0000 7f94e7e00700 1 -- 172.21.15.12:0/1672977378 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0054e0 con 0x7f94f00d2540 2024-03-29T09:55:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.813+0000 7f94e7e00700 1 -- 172.21.15.12:0/1672977378 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc003680 con 0x7f94f00d2540 2024-03-29T09:55:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.813+0000 7f94e7e00700 1 -- 172.21.15.12:0/1672977378 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc003a30 con 0x7f94f00d2540 2024-03-29T09:55:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.813+0000 7f94f6e00700 1 -- 172.21.15.12:0/1672977378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f00cb180 con 0x7f94f00d2540 2024-03-29T09:55:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.813+0000 7f94f6e00700 1 -- 172.21.15.12:0/1672977378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f00cc790 con 0x7f94f00d2540 2024-03-29T09:55:30.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.814+0000 7f94f6e00700 1 -- 172.21.15.12:0/1672977378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f00d1ca0 con 0x7f94f00d2540 2024-03-29T09:55:30.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.815+0000 7f94e7e00700 1 -- 172.21.15.12:0/1672977378 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f94dc003b90 con 0x7f94f00d2540 2024-03-29T09:55:30.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.816+0000 7f94e7e00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f94d006ac90 0x7f94d006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:30.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.817+0000 7f94e7e00700 1 -- 172.21.15.12:0/1672977378 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f94dc088490 con 0x7f94f00d2540 2024-03-29T09:55:30.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.817+0000 7f94efe00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f94d006ac90 0x7f94d006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:30.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.818+0000 7f94efe00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f94d006ac90 0x7f94d006d150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f94e0014040 tx=0x7f94e0008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:30.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:30.821+0000 7f94e7e00700 1 -- 172.21.15.12:0/1672977378 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f94dc053920 con 0x7f94f00d2540 2024-03-29T09:55:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.118+0000 7f94f6e00700 1 -- 172.21.15.12:0/1672977378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f94f0067720 con 0x7f94f00d2540 2024-03-29T09:55:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.121+0000 7f94e7e00700 1 -- 172.21.15.12:0/1672977378 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f94dc0170a0 con 0x7f94f00d2540 2024-03-29T09:55:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.123+0000 7f94f6e00700 1 -- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f94d006ac90 msgr2=0x7f94d006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.123+0000 7f94f6e00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f94d006ac90 0x7f94d006d150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f94e0014040 tx=0x7f94e0008030 comp rx=0 tx=0).stop 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.123+0000 7f94f6e00700 1 -- 172.21.15.12:0/1672977378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f00d2540 msgr2=0x7f94f00cbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.123+0000 7f94f6e00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f00d2540 0x7f94f00cbc60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00c580 tx=0x7f94dc00c890 comp rx=0 tx=0).stop 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.124+0000 7f94f6e00700 1 -- 172.21.15.12:0/1672977378 shutdown_connections 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.124+0000 7f94f6e00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f94d006ac90 0x7f94d006d150 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.124+0000 7f94f6e00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94f015d9d0 0x7f94f00b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.124+0000 7f94f6e00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94f00d16a0 0x7f94f01529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.124+0000 7f94f6e00700 1 --2- 172.21.15.12:0/1672977378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94f00d2540 0x7f94f00cbc60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.124+0000 7f94f6e00700 1 -- 172.21.15.12:0/1672977378 >> 172.21.15.12:0/1672977378 conn(0x7f94f008aff0 msgr2=0x7f94f00b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.124+0000 7f94f6e00700 1 -- 172.21.15.12:0/1672977378 shutdown_connections 2024-03-29T09:55:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.124+0000 7f94f6e00700 1 -- 172.21.15.12:0/1672977378 wait complete. 2024-03-29T09:55:31.142 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-29T09:55:31.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.341+0000 7f6a5ae00700 1 Processor -- start 2024-03-29T09:55:31.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.341+0000 7f6a5ae00700 1 -- start start 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a5ae00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6a5415d9d0 0x7f6a540d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a5ae00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d16a0 0x7f6a540d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a5ae00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6a540d2370 0x7f6a5415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a5ae00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a540d2820 con 0x7f6a5415d9d0 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a5ae00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a54161420 con 0x7f6a540d2370 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a5ae00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a54161700 con 0x7f6a540d16a0 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a53e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6a5415d9d0 0x7f6a540d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a53e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6a5415d9d0 0x7f6a540d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49828/0 (socket says 172.21.15.12:49828) 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a53e00700 1 -- 172.21.15.12:0/2363664489 learned_addr learned my addr 172.21.15.12:0/2363664489 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a53400700 1 --2- 172.21.15.12:0/2363664489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d16a0 0x7f6a540d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.342+0000 7f6a58e00700 1 --2- 172.21.15.12:0/2363664489 >> v2:172.21.15.159:3300/0 conn(0x7f6a540d2370 0x7f6a5415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.343+0000 7f6a53e00700 1 -- 172.21.15.12:0/2363664489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d16a0 msgr2=0x7f6a540d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:31.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.343+0000 7f6a53e00700 1 --2- 172.21.15.12:0/2363664489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d16a0 0x7f6a540d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.343+0000 7f6a53e00700 1 -- 172.21.15.12:0/2363664489 >> v2:172.21.15.159:3300/0 conn(0x7f6a540d2370 msgr2=0x7f6a5415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:31.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.343+0000 7f6a53e00700 1 --2- 172.21.15.12:0/2363664489 >> v2:172.21.15.159:3300/0 conn(0x7f6a540d2370 0x7f6a5415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.343+0000 7f6a53e00700 1 -- 172.21.15.12:0/2363664489 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a541619e0 con 0x7f6a5415d9d0 2024-03-29T09:55:31.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.343+0000 7f6a53400700 1 --2- 172.21.15.12:0/2363664489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d16a0 0x7f6a540d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:31.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.343+0000 7f6a53e00700 1 --2- 172.21.15.12:0/2363664489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 0x7f6a540d3c30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6a540cdc00 tx=0x7f6a44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc704c046f5f1548 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:31.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.343+0000 7f6a52a00700 1 -- 172.21.15.12:0/2363664489 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4400b870 con 0x7f6a5415d9d0 2024-03-29T09:55:31.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.343+0000 7f6a52a00700 1 -- 172.21.15.12:0/2363664489 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4400b9d0 con 0x7f6a5415d9d0 2024-03-29T09:55:31.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.343+0000 7f6a52a00700 1 -- 172.21.15.12:0/2363664489 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4400bd60 con 0x7f6a5415d9d0 2024-03-29T09:55:31.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.344+0000 7f6a5ae00700 1 -- 172.21.15.12:0/2363664489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 msgr2=0x7f6a540d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.344+0000 7f6a5ae00700 1 --2- 172.21.15.12:0/2363664489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 0x7f6a540d3c30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6a540cdc00 tx=0x7f6a44009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.344+0000 7f6a5ae00700 1 -- 172.21.15.12:0/2363664489 shutdown_connections 2024-03-29T09:55:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.344+0000 7f6a5ae00700 1 --2- 172.21.15.12:0/2363664489 >> v2:172.21.15.159:3300/0 conn(0x7f6a540d2370 0x7f6a5415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.344+0000 7f6a5ae00700 1 --2- 172.21.15.12:0/2363664489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 0x7f6a540d3c30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.344+0000 7f6a5ae00700 1 --2- 172.21.15.12:0/2363664489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d16a0 0x7f6a540d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.344+0000 7f6a5ae00700 1 -- 172.21.15.12:0/2363664489 >> 172.21.15.12:0/2363664489 conn(0x7f6a5408aff0 msgr2=0x7f6a540d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.344+0000 7f6a5ae00700 1 -- 172.21.15.12:0/2363664489 shutdown_connections 2024-03-29T09:55:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.344+0000 7f6a5ae00700 1 -- 172.21.15.12:0/2363664489 wait complete. 2024-03-29T09:55:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.346+0000 7f6a5ae00700 1 Processor -- start 2024-03-29T09:55:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.346+0000 7f6a5ae00700 1 -- start start 2024-03-29T09:55:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a5ae00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a540d16a0 0x7f6a5415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a5ae00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d2370 0x7f6a540b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a5ae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 0x7f6a541e8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a5ae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a540cb550 con 0x7f6a5415d9d0 2024-03-29T09:55:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a5ae00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a540cb250 con 0x7f6a540d16a0 2024-03-29T09:55:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a5ae00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5415d000 con 0x7f6a540d2370 2024-03-29T09:55:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a58e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 0x7f6a541e8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a58e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 0x7f6a541e8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49840/0 (socket says 172.21.15.12:49840) 2024-03-29T09:55:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a58e00700 1 -- 172.21.15.12:0/1084452638 learned_addr learned my addr 172.21.15.12:0/1084452638 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a53400700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d2370 0x7f6a540b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a53e00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a540d16a0 0x7f6a5415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a58e00700 1 -- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d2370 msgr2=0x7f6a540b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a58e00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d2370 0x7f6a540b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a58e00700 1 -- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a540d16a0 msgr2=0x7f6a5415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a58e00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a540d16a0 0x7f6a5415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a58e00700 1 -- 172.21.15.12:0/1084452638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a44009550 con 0x7f6a5415d9d0 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a53e00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a540d16a0 0x7f6a5415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.347+0000 7f6a53400700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d2370 0x7f6a540b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.348+0000 7f6a58e00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 0x7f6a541e8e40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c007e40 tx=0x7f6a4c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.348+0000 7f6a4be00700 1 -- 172.21.15.12:0/1084452638 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c00eea0 con 0x7f6a5415d9d0 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.348+0000 7f6a4be00700 1 -- 172.21.15.12:0/1084452638 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4c014420 con 0x7f6a5415d9d0 2024-03-29T09:55:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.348+0000 7f6a4be00700 1 -- 172.21.15.12:0/1084452638 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0147b0 con 0x7f6a5415d9d0 2024-03-29T09:55:31.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.348+0000 7f6a5ae00700 1 -- 172.21.15.12:0/1084452638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a540b6f70 con 0x7f6a5415d9d0 2024-03-29T09:55:31.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.348+0000 7f6a5ae00700 1 -- 172.21.15.12:0/1084452638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a540b7520 con 0x7f6a5415d9d0 2024-03-29T09:55:31.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.349+0000 7f6a5ae00700 1 -- 172.21.15.12:0/1084452638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a540d1ca0 con 0x7f6a5415d9d0 2024-03-29T09:55:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.350+0000 7f6a4be00700 1 -- 172.21.15.12:0/1084452638 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6a4c014910 con 0x7f6a5415d9d0 2024-03-29T09:55:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.351+0000 7f6a4be00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a3406ac90 0x7f6a3406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.351+0000 7f6a4be00700 1 -- 172.21.15.12:0/1084452638 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6a4c0896e0 con 0x7f6a5415d9d0 2024-03-29T09:55:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.351+0000 7f6a53e00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a3406ac90 0x7f6a3406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.352+0000 7f6a53e00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a3406ac90 0x7f6a3406d150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009130 tx=0x7f6a44008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.355+0000 7f6a4be00700 1 -- 172.21.15.12:0/1084452638 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a4c054ca0 con 0x7f6a5415d9d0 2024-03-29T09:55:31.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.656+0000 7f6a5ae00700 1 -- 172.21.15.12:0/1084452638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f6a54067720 con 0x7f6a5415d9d0 2024-03-29T09:55:31.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.660+0000 7f6a4be00700 1 -- 172.21.15.12:0/1084452638 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f6a4c0582c0 con 0x7f6a5415d9d0 2024-03-29T09:55:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 -- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a3406ac90 msgr2=0x7f6a3406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a3406ac90 0x7f6a3406d150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009130 tx=0x7f6a44008030 comp rx=0 tx=0).stop 2024-03-29T09:55:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 -- 172.21.15.12:0/1084452638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 msgr2=0x7f6a541e8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 0x7f6a541e8e40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c007e40 tx=0x7f6a4c00e3c0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 -- 172.21.15.12:0/1084452638 shutdown_connections 2024-03-29T09:55:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a3406ac90 0x7f6a3406d150 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a5415d9d0 0x7f6a541e8e40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a540d16a0 0x7f6a5415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 --2- 172.21.15.12:0/1084452638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a540d2370 0x7f6a540b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 -- 172.21.15.12:0/1084452638 >> 172.21.15.12:0/1084452638 conn(0x7f6a5408aff0 msgr2=0x7f6a540b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 -- 172.21.15.12:0/1084452638 shutdown_connections 2024-03-29T09:55:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.663+0000 7f6a5ae00700 1 -- 172.21.15.12:0/1084452638 wait complete. 2024-03-29T09:55:31.681 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:31.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.878+0000 7f20b1800700 1 Processor -- start 2024-03-29T09:55:31.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.879+0000 7f20b1800700 1 -- start start 2024-03-29T09:55:31.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.880+0000 7f20b1800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f20ac0d1ed0 0x7f20ac0cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.880+0000 7f20b1800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac0cfdf0 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.880+0000 7f20b1800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac0d2330 con 0x7f20ac15b660 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.880+0000 7f20b1800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ac099a60 con 0x7f20ac0d1ed0 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.880+0000 7f20ab400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f20ac0cf9a0 0x7f20ac09a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44212/0 (socket says 172.21.15.12:44212) 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.880+0000 7f20ab400700 1 -- 172.21.15.12:0/2069587496 learned_addr learned my addr 172.21.15.12:0/2069587496 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.880+0000 7f20aa000700 1 --2- 172.21.15.12:0/2069587496 >> v2:172.21.15.159:3300/0 conn(0x7f20ac0d1ed0 0x7f20ac0cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20a9600700 1 -- 172.21.15.12:0/2069587496 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309112007 0 0) 0x7f20a4001490 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20a9600700 1 -- 172.21.15.12:0/2069587496 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20900035b0 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20a9600700 1 -- 172.21.15.12:0/2069587496 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265752874 0 0) 0x7f20ac0d2330 con 0x7f20ac15b660 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20a9600700 1 -- 172.21.15.12:0/2069587496 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a4001490 con 0x7f20ac15b660 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20aa000700 1 -- 172.21.15.12:0/2069587496 >> v1:172.21.15.159:6789/0 conn(0x7f20ac15b660 legacy=0x7f20ac0c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20aa000700 1 -- 172.21.15.12:0/2069587496 >> v1:172.21.15.12:6789/0 conn(0x7f20ac0cf9a0 legacy=0x7f20ac09a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20aa000700 1 -- 172.21.15.12:0/2069587496 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac0cce30 con 0x7f20ac0d1ed0 2024-03-29T09:55:31.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20aa000700 1 --2- 172.21.15.12:0/2069587496 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20ac0d1ed0 0x7f20ac0cf460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2098009840 tx=0x7f2098009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6948ad2a9fa08206 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:31.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20a9600700 1 -- 172.21.15.12:0/2069587496 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098011830 con 0x7f20ac0d1ed0 2024-03-29T09:55:31.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20a9600700 1 -- 172.21.15.12:0/2069587496 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2098011990 con 0x7f20ac0d1ed0 2024-03-29T09:55:31.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.881+0000 7f20a9600700 1 -- 172.21.15.12:0/2069587496 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098011d20 con 0x7f20ac0d1ed0 2024-03-29T09:55:31.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.882+0000 7f20b1800700 1 -- 172.21.15.12:0/2069587496 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20ac0d1ed0 msgr2=0x7f20ac0cf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.882+0000 7f20b1800700 1 --2- 172.21.15.12:0/2069587496 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20ac0d1ed0 0x7f20ac0cf460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2098009840 tx=0x7f2098009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:31.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.882+0000 7f20b1800700 1 -- 172.21.15.12:0/2069587496 shutdown_connections 2024-03-29T09:55:31.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.882+0000 7f20b1800700 1 --2- 172.21.15.12:0/2069587496 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20ac0d1ed0 0x7f20ac0cf460 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.882+0000 7f20b1800700 1 -- 172.21.15.12:0/2069587496 >> 172.21.15.12:0/2069587496 conn(0x7f20ac0d13e0 msgr2=0x7f20ac0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:31.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.882+0000 7f20b1800700 1 -- 172.21.15.12:0/2069587496 shutdown_connections 2024-03-29T09:55:31.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.882+0000 7f20b1800700 1 -- 172.21.15.12:0/2069587496 wait complete. 2024-03-29T09:55:31.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.884+0000 7f20b1800700 1 Processor -- start 2024-03-29T09:55:31.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.884+0000 7f20b1800700 1 -- start start 2024-03-29T09:55:31.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.885+0000 7f20b1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20ac0cf9a0 0x7f20ac15cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.885+0000 7f20b1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20ac0d1ed0 0x7f20ac157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.885+0000 7f20b1800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20ac15b660 0x7f20ac1485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.885+0000 7f20b1800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac15d100 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.885+0000 7f20b1800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac15d330 con 0x7f20ac15b660 2024-03-29T09:55:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.885+0000 7f20b1800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac15d610 con 0x7f20ac0d1ed0 2024-03-29T09:55:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.885+0000 7f20aaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20ac0cf9a0 0x7f20ac15cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.885+0000 7f20aaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20ac0cf9a0 0x7f20ac15cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49854/0 (socket says 172.21.15.12:49854) 2024-03-29T09:55:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.885+0000 7f20aaa00700 1 -- 172.21.15.12:0/937709013 learned_addr learned my addr 172.21.15.12:0/937709013 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20ab400700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20ac15b660 0x7f20ac1485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20aa000700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20ac0d1ed0 0x7f20ac157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20aaa00700 1 -- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20ac0d1ed0 msgr2=0x7f20ac157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20aaa00700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20ac0d1ed0 0x7f20ac157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20aaa00700 1 -- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20ac15b660 msgr2=0x7f20ac1485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20aaa00700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20ac15b660 0x7f20ac1485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20aaa00700 1 -- 172.21.15.12:0/937709013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980094a0 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20aaa00700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20ac0cf9a0 0x7f20ac15cbc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f209c00c520 tx=0x7f209c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20a2a00700 1 -- 172.21.15.12:0/937709013 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c0052d0 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20a2a00700 1 -- 172.21.15.12:0/937709013 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c005430 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20b1800700 1 -- 172.21.15.12:0/937709013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac15d8f0 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20a2a00700 1 -- 172.21.15.12:0/937709013 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c0038c0 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.886+0000 7f20b1800700 1 -- 172.21.15.12:0/937709013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac15deb0 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.887+0000 7f20b1800700 1 -- 172.21.15.12:0/937709013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac06e470 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.888+0000 7f20a2a00700 1 -- 172.21.15.12:0/937709013 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f209c003a20 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.889+0000 7f20a2a00700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f208c06ac90 0x7f208c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:31.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.889+0000 7f20a2a00700 1 -- 172.21.15.12:0/937709013 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f209c0885e0 con 0x7f20ac0cf9a0 2024-03-29T09:55:31.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.890+0000 7f20aa000700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f208c06ac90 0x7f208c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:31.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.890+0000 7f20aa000700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f208c06ac90 0x7f208c06d150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2098009fd0 tx=0x7f209801c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:31.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:31.893+0000 7f20a2a00700 1 -- 172.21.15.12:0/937709013 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f209c053a80 con 0x7f20ac0cf9a0 2024-03-29T09:55:32.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.109+0000 7f20b1800700 1 -- 172.21.15.12:0/937709013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20ac067720 con 0x7f20ac0cf9a0 2024-03-29T09:55:32.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.110+0000 7f20a2a00700 1 -- 172.21.15.12:0/937709013 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12389 (secure 0 0 0) 0x7f209c017090 con 0x7f20ac0cf9a0 2024-03-29T09:55:32.110 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:32.110 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":30,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:55:27.889608+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:32.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.112+0000 7f20b1800700 1 -- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f208c06ac90 msgr2=0x7f208c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.112+0000 7f20b1800700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f208c06ac90 0x7f208c06d150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2098009fd0 tx=0x7f209801c040 comp rx=0 tx=0).stop 2024-03-29T09:55:32.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.112+0000 7f20b1800700 1 -- 172.21.15.12:0/937709013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20ac0cf9a0 msgr2=0x7f20ac15cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.112+0000 7f20b1800700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20ac0cf9a0 0x7f20ac15cbc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f209c00c520 tx=0x7f209c00c8e0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.112+0000 7f20b1800700 1 -- 172.21.15.12:0/937709013 shutdown_connections 2024-03-29T09:55:32.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.112+0000 7f20b1800700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f208c06ac90 0x7f208c06d150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.112+0000 7f20b1800700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20ac15b660 0x7f20ac1485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.112+0000 7f20b1800700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20ac0cf9a0 0x7f20ac15cbc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.112+0000 7f20b1800700 1 --2- 172.21.15.12:0/937709013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20ac0d1ed0 0x7f20ac157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.112+0000 7f20b1800700 1 -- 172.21.15.12:0/937709013 >> 172.21.15.12:0/937709013 conn(0x7f20ac0d13e0 msgr2=0x7f20ac0c7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:32.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.113+0000 7f20b1800700 1 -- 172.21.15.12:0/937709013 shutdown_connections 2024-03-29T09:55:32.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.113+0000 7f20b1800700 1 -- 172.21.15.12:0/937709013 wait complete. 2024-03-29T09:55:32.131 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T09:55:32.131 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T09:55:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.332+0000 7f7b7b600700 1 Processor -- start 2024-03-29T09:55:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.332+0000 7f7b7b600700 1 -- start start 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.333+0000 7f7b7b600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b7415a5c0 0x7f7b740709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.333+0000 7f7b7b600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7b7415dfe0 0x7f7b740ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.333+0000 7f7b7b600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b740cb310 con 0x7f7b740ce670 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.333+0000 7f7b7b600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b740ceb00 con 0x7f7b7415dfe0 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.333+0000 7f7b7b600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b74070f00 con 0x7f7b7415a5c0 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.333+0000 7f7b73e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7b7415dfe0 0x7f7b740ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.333+0000 7f7b73e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7b7415dfe0 0x7f7b740ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49858/0 (socket says 172.21.15.12:49858) 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.333+0000 7f7b73e00700 1 -- 172.21.15.12:0/818028873 learned_addr learned my addr 172.21.15.12:0/818028873 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.334+0000 7f7b78c00700 1 --2- 172.21.15.12:0/818028873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b7415a5c0 0x7f7b740709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.334+0000 7f7b73e00700 1 -- 172.21.15.12:0/818028873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b7415a5c0 msgr2=0x7f7b740709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.334+0000 7f7b73e00700 1 --2- 172.21.15.12:0/818028873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b7415a5c0 0x7f7b740709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.334+0000 7f7b73e00700 1 -- 172.21.15.12:0/818028873 >> v1:172.21.15.159:6789/0 conn(0x7f7b740ce670 legacy=0x7f7b740b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.334+0000 7f7b73e00700 1 -- 172.21.15.12:0/818028873 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b740cca90 con 0x7f7b7415dfe0 2024-03-29T09:55:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.334+0000 7f7b78c00700 1 --2- 172.21.15.12:0/818028873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b7415a5c0 0x7f7b740709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.334+0000 7f7b73e00700 1 --2- 172.21.15.12:0/818028873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 0x7f7b740ce130 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7b600098f0 tx=0x7f7b60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc3d1ef66253 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.334+0000 7f7b73400700 1 -- 172.21.15.12:0/818028873 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6000b8a0 con 0x7f7b7415dfe0 2024-03-29T09:55:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.334+0000 7f7b73400700 1 -- 172.21.15.12:0/818028873 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6000ba00 con 0x7f7b7415dfe0 2024-03-29T09:55:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.334+0000 7f7b73400700 1 -- 172.21.15.12:0/818028873 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6000bd90 con 0x7f7b7415dfe0 2024-03-29T09:55:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.335+0000 7f7b7b600700 1 -- 172.21.15.12:0/818028873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 msgr2=0x7f7b740ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.335+0000 7f7b7b600700 1 --2- 172.21.15.12:0/818028873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 0x7f7b740ce130 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7b600098f0 tx=0x7f7b60009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.335+0000 7f7b7b600700 1 -- 172.21.15.12:0/818028873 shutdown_connections 2024-03-29T09:55:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.335+0000 7f7b7b600700 1 --2- 172.21.15.12:0/818028873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b7415a5c0 0x7f7b740709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.335+0000 7f7b7b600700 1 --2- 172.21.15.12:0/818028873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 0x7f7b740ce130 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.335+0000 7f7b7b600700 1 -- 172.21.15.12:0/818028873 >> 172.21.15.12:0/818028873 conn(0x7f7b740b0d70 msgr2=0x7f7b740b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.335+0000 7f7b7b600700 1 -- 172.21.15.12:0/818028873 shutdown_connections 2024-03-29T09:55:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.335+0000 7f7b7b600700 1 -- 172.21.15.12:0/818028873 wait complete. 2024-03-29T09:55:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.337+0000 7f7b7b600700 1 Processor -- start 2024-03-29T09:55:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.337+0000 7f7b7b600700 1 -- start start 2024-03-29T09:55:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b7b600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b740ce670 0x7f7b7406c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b7b600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b7415a5c0 0x7f7b74098a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b7b600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 0x7f7b74152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b7b600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b74098f60 con 0x7f7b7415dfe0 2024-03-29T09:55:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b7b600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b74152650 con 0x7f7b7415a5c0 2024-03-29T09:55:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b7b600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7406cf20 con 0x7f7b740ce670 2024-03-29T09:55:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b79600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 0x7f7b74152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b79600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 0x7f7b74152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49868/0 (socket says 172.21.15.12:49868) 2024-03-29T09:55:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b79600700 1 -- 172.21.15.12:0/1844189516 learned_addr learned my addr 172.21.15.12:0/1844189516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b73e00700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b7415a5c0 0x7f7b74098a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.338+0000 7f7b78c00700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b740ce670 0x7f7b7406c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b79600700 1 -- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b740ce670 msgr2=0x7f7b7406c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b79600700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b740ce670 0x7f7b7406c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b79600700 1 -- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b7415a5c0 msgr2=0x7f7b74098a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b79600700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b7415a5c0 0x7f7b74098a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b79600700 1 -- 172.21.15.12:0/1844189516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b60009550 con 0x7f7b7415dfe0 2024-03-29T09:55:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b79600700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 0x7f7b74152110 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00cc50 tx=0x7f7b6c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b70c00700 1 -- 172.21.15.12:0/1844189516 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c003c00 con 0x7f7b7415dfe0 2024-03-29T09:55:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b70c00700 1 -- 172.21.15.12:0/1844189516 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c003d60 con 0x7f7b7415dfe0 2024-03-29T09:55:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b70c00700 1 -- 172.21.15.12:0/1844189516 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00aab0 con 0x7f7b7415dfe0 2024-03-29T09:55:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b7b600700 1 -- 172.21.15.12:0/1844189516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7406d200 con 0x7f7b7415dfe0 2024-03-29T09:55:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.339+0000 7f7b7b600700 1 -- 172.21.15.12:0/1844189516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7406d7c0 con 0x7f7b7415dfe0 2024-03-29T09:55:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.340+0000 7f7b7b600700 1 -- 172.21.15.12:0/1844189516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b74067720 con 0x7f7b7415dfe0 2024-03-29T09:55:32.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.341+0000 7f7b70c00700 1 -- 172.21.15.12:0/1844189516 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7b6c00ac10 con 0x7f7b7415dfe0 2024-03-29T09:55:32.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.343+0000 7f7b70c00700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b5406ac90 0x7f7b5406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.343+0000 7f7b70c00700 1 -- 172.21.15.12:0/1844189516 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7b6c088fc0 con 0x7f7b7415dfe0 2024-03-29T09:55:32.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.343+0000 7f7b78c00700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b5406ac90 0x7f7b5406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.344+0000 7f7b78c00700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b5406ac90 0x7f7b5406d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7b680099d0 tx=0x7f7b68008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.347+0000 7f7b70c00700 1 -- 172.21.15.12:0/1844189516 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b6c0544e0 con 0x7f7b7415dfe0 2024-03-29T09:55:32.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.562+0000 7f7b7b600700 1 -- 172.21.15.12:0/1844189516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b740ae530 con 0x7f7b7415dfe0 2024-03-29T09:55:32.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.562+0000 7f7b70c00700 1 -- 172.21.15.12:0/1844189516 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b6c057b00 con 0x7f7b7415dfe0 2024-03-29T09:55:32.562 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T09:55:32.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 -- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b5406ac90 msgr2=0x7f7b5406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b5406ac90 0x7f7b5406d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7b680099d0 tx=0x7f7b68008040 comp rx=0 tx=0).stop 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 -- 172.21.15.12:0/1844189516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 msgr2=0x7f7b74152110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 0x7f7b74152110 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00cc50 tx=0x7f7b6c00cf60 comp rx=0 tx=0).stop 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 -- 172.21.15.12:0/1844189516 shutdown_connections 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b5406ac90 0x7f7b5406d150 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b7415dfe0 0x7f7b74152110 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b740ce670 0x7f7b7406c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 --2- 172.21.15.12:0/1844189516 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b7415a5c0 0x7f7b74098a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 -- 172.21.15.12:0/1844189516 >> 172.21.15.12:0/1844189516 conn(0x7f7b740b0d70 msgr2=0x7f7b740af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 -- 172.21.15.12:0/1844189516 shutdown_connections 2024-03-29T09:55:32.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.565+0000 7f7b7b600700 1 -- 172.21.15.12:0/1844189516 wait complete. 2024-03-29T09:55:32.583 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T09:55:32.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.781+0000 7fcd39e00700 1 Processor -- start 2024-03-29T09:55:32.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.781+0000 7fcd39e00700 1 -- start start 2024-03-29T09:55:32.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.782+0000 7fcd39e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340b3af0 0x7fcd3415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.782+0000 7fcd39e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd340b2180 con 0x7fcd340709c0 2024-03-29T09:55:32.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.782+0000 7fcd39e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3415b100 con 0x7fcd34072580 2024-03-29T09:55:32.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.782+0000 7fcd39e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3415b3e0 con 0x7fcd340b3af0 2024-03-29T09:55:32.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd33400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcd340709c0 0x7fcd3406e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44224/0 (socket says 172.21.15.12:44224) 2024-03-29T09:55:32.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd33400700 1 -- 172.21.15.12:0/2687589404 learned_addr learned my addr 172.21.15.12:0/2687589404 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:32.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd32a00700 1 --2- 172.21.15.12:0/2687589404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340b3af0 0x7fcd3415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd31600700 1 -- 172.21.15.12:0/2687589404 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237470553 0 0) 0x7fcd340b2180 con 0x7fcd340709c0 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd31600700 1 -- 172.21.15.12:0/2687589404 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd180035b0 con 0x7fcd340709c0 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd31600700 1 -- 172.21.15.12:0/2687589404 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818035604 0 0) 0x7fcd3415b100 con 0x7fcd34072580 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd31600700 1 -- 172.21.15.12:0/2687589404 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd340b2180 con 0x7fcd34072580 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd32a00700 1 -- 172.21.15.12:0/2687589404 >> v1:172.21.15.159:6789/0 conn(0x7fcd34072580 legacy=0x7fcd340b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd32a00700 1 -- 172.21.15.12:0/2687589404 >> v1:172.21.15.12:6789/0 conn(0x7fcd340709c0 legacy=0x7fcd3406e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd31600700 1 -- 172.21.15.12:0/2687589404 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 647864127 0 0) 0x7fcd180035b0 con 0x7fcd340709c0 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.783+0000 7fcd32a00700 1 -- 172.21.15.12:0/2687589404 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd340b23f0 con 0x7fcd340b3af0 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.784+0000 7fcd32a00700 1 --2- 172.21.15.12:0/2687589404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340b3af0 0x7fcd3415a860 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009870 tx=0x7fcd24009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6c9763eadc1d805 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.784+0000 7fcd31600700 1 -- 172.21.15.12:0/2687589404 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd24011830 con 0x7fcd340b3af0 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.784+0000 7fcd31600700 1 -- 172.21.15.12:0/2687589404 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd24011990 con 0x7fcd340b3af0 2024-03-29T09:55:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.784+0000 7fcd31600700 1 -- 172.21.15.12:0/2687589404 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd24011d20 con 0x7fcd340b3af0 2024-03-29T09:55:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.785+0000 7fcd39e00700 1 -- 172.21.15.12:0/2687589404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340b3af0 msgr2=0x7fcd3415a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.785+0000 7fcd39e00700 1 --2- 172.21.15.12:0/2687589404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340b3af0 0x7fcd3415a860 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009870 tx=0x7fcd24009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.785+0000 7fcd39e00700 1 -- 172.21.15.12:0/2687589404 shutdown_connections 2024-03-29T09:55:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.785+0000 7fcd39e00700 1 --2- 172.21.15.12:0/2687589404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340b3af0 0x7fcd3415a860 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.785+0000 7fcd39e00700 1 -- 172.21.15.12:0/2687589404 >> 172.21.15.12:0/2687589404 conn(0x7fcd340cf190 msgr2=0x7fcd340cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.785+0000 7fcd39e00700 1 -- 172.21.15.12:0/2687589404 shutdown_connections 2024-03-29T09:55:32.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.785+0000 7fcd39e00700 1 -- 172.21.15.12:0/2687589404 wait complete. 2024-03-29T09:55:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.787+0000 7fcd39e00700 1 Processor -- start 2024-03-29T09:55:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.787+0000 7fcd39e00700 1 -- start start 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd39e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340709c0 0x7fcd34147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd39e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd34072580 0x7fcd340c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd39e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd340b3af0 0x7fcd3415c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd39e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd34151d50 con 0x7fcd340b3af0 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd39e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3415a290 con 0x7fcd34072580 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd39e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340c8600 con 0x7fcd340709c0 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd33400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd340b3af0 0x7fcd3415c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd33400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd340b3af0 0x7fcd3415c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49878/0 (socket says 172.21.15.12:49878) 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd33400700 1 -- 172.21.15.12:0/3787966 learned_addr learned my addr 172.21.15.12:0/3787966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd32000700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd34072580 0x7fcd340c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd32a00700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340709c0 0x7fcd34147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd33400700 1 -- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340709c0 msgr2=0x7fcd34147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd33400700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340709c0 0x7fcd34147540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd33400700 1 -- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd34072580 msgr2=0x7fcd340c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:32.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd33400700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd34072580 0x7fcd340c90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:32.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.788+0000 7fcd33400700 1 -- 172.21.15.12:0/3787966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd240094a0 con 0x7fcd340b3af0 2024-03-29T09:55:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.789+0000 7fcd33400700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd340b3af0 0x7fcd3415c890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00cad0 tx=0x7fcd2c00ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.789+0000 7fcd2b400700 1 -- 172.21.15.12:0/3787966 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c00ab00 con 0x7fcd340b3af0 2024-03-29T09:55:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.789+0000 7fcd2b400700 1 -- 172.21.15.12:0/3787966 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c00ac60 con 0x7fcd340b3af0 2024-03-29T09:55:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.789+0000 7fcd2b400700 1 -- 172.21.15.12:0/3787966 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c012400 con 0x7fcd340b3af0 2024-03-29T09:55:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.789+0000 7fcd39e00700 1 -- 172.21.15.12:0/3787966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd340c88e0 con 0x7fcd340b3af0 2024-03-29T09:55:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.789+0000 7fcd39e00700 1 -- 172.21.15.12:0/3787966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd34147a80 con 0x7fcd340b3af0 2024-03-29T09:55:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.790+0000 7fcd39e00700 1 -- 172.21.15.12:0/3787966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd340c8460 con 0x7fcd340b3af0 2024-03-29T09:55:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.792+0000 7fcd2b400700 1 -- 172.21.15.12:0/3787966 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcd2c012580 con 0x7fcd340b3af0 2024-03-29T09:55:32.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.792+0000 7fcd2b400700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcd1406ac90 0x7fcd1406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:32.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.793+0000 7fcd2b400700 1 -- 172.21.15.12:0/3787966 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcd2c089460 con 0x7fcd340b3af0 2024-03-29T09:55:32.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.793+0000 7fcd32a00700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcd1406ac90 0x7fcd1406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:32.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.794+0000 7fcd32a00700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcd1406ac90 0x7fcd1406d150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcd240090b0 tx=0x7fcd2401c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:32.797+0000 7fcd2b400700 1 -- 172.21.15.12:0/3787966 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd340c8460 con 0x7fcd340b3af0 2024-03-29T09:55:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.014+0000 7fcd39e00700 1 -- 172.21.15.12:0/3787966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcd340c8e00 con 0x7fcd340b3af0 2024-03-29T09:55:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.014+0000 7fcd2b400700 1 -- 172.21.15.12:0/3787966 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcd340c8e00 con 0x7fcd340b3af0 2024-03-29T09:55:33.014 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:33.014 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T09:55:33.014 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:33.014 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T09:55:33.014 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T09:55:33.014 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:33.014 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T09:55:33.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.016+0000 7fcd39e00700 1 -- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcd1406ac90 msgr2=0x7fcd1406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.016+0000 7fcd39e00700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcd1406ac90 0x7fcd1406d150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcd240090b0 tx=0x7fcd2401c040 comp rx=0 tx=0).stop 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.017+0000 7fcd39e00700 1 -- 172.21.15.12:0/3787966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd340b3af0 msgr2=0x7fcd3415c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.017+0000 7fcd39e00700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd340b3af0 0x7fcd3415c890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00cad0 tx=0x7fcd2c00ce90 comp rx=0 tx=0).stop 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.017+0000 7fcd39e00700 1 -- 172.21.15.12:0/3787966 shutdown_connections 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.017+0000 7fcd39e00700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcd1406ac90 0x7fcd1406d150 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.017+0000 7fcd39e00700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd340b3af0 0x7fcd3415c890 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.017+0000 7fcd39e00700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd340709c0 0x7fcd34147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.017+0000 7fcd39e00700 1 --2- 172.21.15.12:0/3787966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd34072580 0x7fcd340c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.017+0000 7fcd39e00700 1 -- 172.21.15.12:0/3787966 >> 172.21.15.12:0/3787966 conn(0x7fcd340cf190 msgr2=0x7fcd3406d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.017+0000 7fcd39e00700 1 -- 172.21.15.12:0/3787966 shutdown_connections 2024-03-29T09:55:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.017+0000 7fcd39e00700 1 -- 172.21.15.12:0/3787966 wait complete. 2024-03-29T09:55:33.035 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T09:55:33.035 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T09:55:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.238+0000 7f5437800700 1 Processor -- start 2024-03-29T09:55:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.238+0000 7f5437800700 1 -- start start 2024-03-29T09:55:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.239+0000 7f5437800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f54300ced30 0x7f54300d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.239+0000 7f5437800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f54300d2710 0x7f543015f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.239+0000 7f5437800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54300ce8a0 con 0x7f543015ddf0 2024-03-29T09:55:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.239+0000 7f5437800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f543015ec60 con 0x7f54300d2710 2024-03-29T09:55:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.239+0000 7f5437800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54300d2b60 con 0x7f54300ced30 2024-03-29T09:55:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.239+0000 7f5435800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f54300d2710 0x7f543015f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.239+0000 7f5434e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f543015ddf0 0x7f543015e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44232/0 (socket says 172.21.15.12:44232) 2024-03-29T09:55:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.239+0000 7f5434e00700 1 -- 172.21.15.12:0/2245509972 learned_addr learned my addr 172.21.15.12:0/2245509972 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f542fe00700 1 --2- 172.21.15.12:0/2245509972 >> v2:172.21.15.159:3300/0 conn(0x7f54300ced30 0x7f54300d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f542f400700 1 -- 172.21.15.12:0/2245509972 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314409062 0 0) 0x7f54300ce8a0 con 0x7f543015ddf0 2024-03-29T09:55:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f5435800700 1 -- 172.21.15.12:0/2245509972 >> v2:172.21.15.159:3300/0 conn(0x7f54300ced30 msgr2=0x7f54300d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f5435800700 1 --2- 172.21.15.12:0/2245509972 >> v2:172.21.15.159:3300/0 conn(0x7f54300ced30 0x7f54300d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f5435800700 1 -- 172.21.15.12:0/2245509972 >> v1:172.21.15.12:6789/0 conn(0x7f543015ddf0 legacy=0x7f543015e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f5435800700 1 -- 172.21.15.12:0/2245509972 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54300cdc90 con 0x7f54300d2710 2024-03-29T09:55:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f542fe00700 1 --2- 172.21.15.12:0/2245509972 >> v2:172.21.15.159:3300/0 conn(0x7f54300ced30 0x7f54300d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f5435800700 1 --2- 172.21.15.12:0/2245509972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300d2710 0x7f543015f0f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f54300d0fe0 tx=0x7f542800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e42148befa5d7e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f542f400700 1 -- 172.21.15.12:0/2245509972 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542800e070 con 0x7f54300d2710 2024-03-29T09:55:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f542f400700 1 -- 172.21.15.12:0/2245509972 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542800bbd0 con 0x7f54300d2710 2024-03-29T09:55:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.240+0000 7f542f400700 1 -- 172.21.15.12:0/2245509972 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428012430 con 0x7f54300d2710 2024-03-29T09:55:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.241+0000 7f5437800700 1 -- 172.21.15.12:0/2245509972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300d2710 msgr2=0x7f543015f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.241+0000 7f5437800700 1 --2- 172.21.15.12:0/2245509972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300d2710 0x7f543015f0f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f54300d0fe0 tx=0x7f542800b330 comp rx=0 tx=0).stop 2024-03-29T09:55:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.241+0000 7f5437800700 1 -- 172.21.15.12:0/2245509972 shutdown_connections 2024-03-29T09:55:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.241+0000 7f5437800700 1 --2- 172.21.15.12:0/2245509972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300d2710 0x7f543015f0f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.241+0000 7f5437800700 1 --2- 172.21.15.12:0/2245509972 >> v2:172.21.15.159:3300/0 conn(0x7f54300ced30 0x7f54300d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.241+0000 7f5437800700 1 -- 172.21.15.12:0/2245509972 >> 172.21.15.12:0/2245509972 conn(0x7f543008aff0 msgr2=0x7f54300d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.241+0000 7f5437800700 1 -- 172.21.15.12:0/2245509972 shutdown_connections 2024-03-29T09:55:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.241+0000 7f5437800700 1 -- 172.21.15.12:0/2245509972 wait complete. 2024-03-29T09:55:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.243+0000 7f5437800700 1 Processor -- start 2024-03-29T09:55:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.244+0000 7f5437800700 1 -- start start 2024-03-29T09:55:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f5437800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300ced30 0x7f54300ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f5437800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54300d2710 0x7f5430159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f5437800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f543015ddf0 0x7f54300c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f5437800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54300cb840 con 0x7f54300ced30 2024-03-29T09:55:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f5437800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543015a410 con 0x7f54300d2710 2024-03-29T09:55:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f5437800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54300cabc0 con 0x7f543015ddf0 2024-03-29T09:55:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f5434e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300ced30 0x7f54300ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f5434e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300ced30 0x7f54300ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49898/0 (socket says 172.21.15.12:49898) 2024-03-29T09:55:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f5434e00700 1 -- 172.21.15.12:0/2788256228 learned_addr learned my addr 172.21.15.12:0/2788256228 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f542fe00700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54300d2710 0x7f5430159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.245+0000 7f5435800700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f543015ddf0 0x7f54300c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f5434e00700 1 -- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f543015ddf0 msgr2=0x7f54300c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f5434e00700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f543015ddf0 0x7f54300c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f5434e00700 1 -- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54300d2710 msgr2=0x7f5430159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f5434e00700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54300d2710 0x7f5430159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f5434e00700 1 -- 172.21.15.12:0/2788256228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5428009a70 con 0x7f54300ced30 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f5435800700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f543015ddf0 0x7f54300c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f5434e00700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300ced30 0x7f54300ca5e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f542400c550 tx=0x7f542400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f542cc00700 1 -- 172.21.15.12:0/2788256228 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424004c80 con 0x7f54300ced30 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f542cc00700 1 -- 172.21.15.12:0/2788256228 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5424004e00 con 0x7f54300ced30 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f5437800700 1 -- 172.21.15.12:0/2788256228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54300caea0 con 0x7f54300ced30 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f542cc00700 1 -- 172.21.15.12:0/2788256228 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424007310 con 0x7f54300ced30 2024-03-29T09:55:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.246+0000 7f5437800700 1 -- 172.21.15.12:0/2788256228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54300cb460 con 0x7f54300ced30 2024-03-29T09:55:33.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.247+0000 7f5437800700 1 -- 172.21.15.12:0/2788256228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54300cb980 con 0x7f54300ced30 2024-03-29T09:55:33.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.248+0000 7f542cc00700 1 -- 172.21.15.12:0/2788256228 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5424007470 con 0x7f54300ced30 2024-03-29T09:55:33.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.249+0000 7f542cc00700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541006ac90 0x7f541006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:33.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.249+0000 7f542cc00700 1 -- 172.21.15.12:0/2788256228 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5424088d40 con 0x7f54300ced30 2024-03-29T09:55:33.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.249+0000 7f542fe00700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541006ac90 0x7f541006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:33.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.250+0000 7f542fe00700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541006ac90 0x7f541006d150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f541c006f10 tx=0x7f541c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:33.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.254+0000 7f542cc00700 1 -- 172.21.15.12:0/2788256228 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5424054150 con 0x7f54300ced30 2024-03-29T09:55:33.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.467+0000 7f5437800700 1 -- 172.21.15.12:0/2788256228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5430067720 con 0x7f54300ced30 2024-03-29T09:55:33.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.915+0000 7f542cc00700 1 -- 172.21.15.12:0/2788256228 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5424057770 con 0x7f54300ced30 2024-03-29T09:55:33.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.919+0000 7f5437800700 1 -- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541006ac90 msgr2=0x7f541006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:33.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.919+0000 7f5437800700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541006ac90 0x7f541006d150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f541c006f10 tx=0x7f541c008040 comp rx=0 tx=0).stop 2024-03-29T09:55:33.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.919+0000 7f5437800700 1 -- 172.21.15.12:0/2788256228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300ced30 msgr2=0x7f54300ca5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:33.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.919+0000 7f5437800700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300ced30 0x7f54300ca5e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f542400c550 tx=0x7f542400c860 comp rx=0 tx=0).stop 2024-03-29T09:55:33.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.919+0000 7f5437800700 1 -- 172.21.15.12:0/2788256228 shutdown_connections 2024-03-29T09:55:33.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.919+0000 7f5437800700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541006ac90 0x7f541006d150 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.919+0000 7f5437800700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f543015ddf0 0x7f54300c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.919+0000 7f5437800700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54300ced30 0x7f54300ca5e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.919+0000 7f5437800700 1 --2- 172.21.15.12:0/2788256228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54300d2710 0x7f5430159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:33.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.919+0000 7f5437800700 1 -- 172.21.15.12:0/2788256228 >> 172.21.15.12:0/2788256228 conn(0x7f543008aff0 msgr2=0x7f543015e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:33.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.920+0000 7f5437800700 1 -- 172.21.15.12:0/2788256228 shutdown_connections 2024-03-29T09:55:33.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:33.920+0000 7f5437800700 1 -- 172.21.15.12:0/2788256228 wait complete. 2024-03-29T09:55:33.922 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T09:55:33.941 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T09:55:34.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.147+0000 7f2cca800700 1 Processor -- start 2024-03-29T09:55:34.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.147+0000 7f2cca800700 1 -- start start 2024-03-29T09:55:34.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.148+0000 7f2cca800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cc40d16a0 0x7f2cc40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:34.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.148+0000 7f2cca800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 0x7f2cc40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:34.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.148+0000 7f2cca800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc4161420 con 0x7f2cc415d9d0 2024-03-29T09:55:34.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.148+0000 7f2cca800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc40d2820 con 0x7f2cc40d16a0 2024-03-29T09:55:34.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.148+0000 7f2cca800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40d4340 con 0x7f2cc40d2370 2024-03-29T09:55:34.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cc40d16a0 0x7f2cc40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:34.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc3400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2cc415d9d0 0x7f2cc40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44236/0 (socket says 172.21.15.12:44236) 2024-03-29T09:55:34.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc3400700 1 -- 172.21.15.12:0/3892523631 learned_addr learned my addr 172.21.15.12:0/3892523631 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc3e00700 1 --2- 172.21.15.12:0/3892523631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 0x7f2cc40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc2a00700 1 -- 172.21.15.12:0/3892523631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 msgr2=0x7f2cc40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/3892523631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 0x7f2cc40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc2a00700 1 -- 172.21.15.12:0/3892523631 >> v1:172.21.15.12:6789/0 conn(0x7f2cc415d9d0 legacy=0x7f2cc40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:34.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc2a00700 1 -- 172.21.15.12:0/3892523631 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc4161700 con 0x7f2cc40d16a0 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc3e00700 1 --2- 172.21.15.12:0/3892523631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 0x7f2cc40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc2000700 1 -- 172.21.15.12:0/3892523631 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256047918 0 0) 0x7f2cc4161420 con 0x7f2cc415d9d0 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.149+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/3892523631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 0x7f2cc40d1e30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009830 tx=0x7f2cb0009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b634acb9a22518 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cc2000700 1 -- 172.21.15.12:0/3892523631 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0011830 con 0x7f2cc40d16a0 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cc2000700 1 -- 172.21.15.12:0/3892523631 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb0011990 con 0x7f2cc40d16a0 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cc2000700 1 -- 172.21.15.12:0/3892523631 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0011d40 con 0x7f2cc40d16a0 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cca800700 1 -- 172.21.15.12:0/3892523631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 msgr2=0x7f2cc40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cca800700 1 --2- 172.21.15.12:0/3892523631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 0x7f2cc40d1e30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009830 tx=0x7f2cb0009b40 comp rx=0 tx=0).stop 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cca800700 1 -- 172.21.15.12:0/3892523631 shutdown_connections 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cca800700 1 --2- 172.21.15.12:0/3892523631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 0x7f2cc40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cca800700 1 --2- 172.21.15.12:0/3892523631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 0x7f2cc40d1e30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cca800700 1 -- 172.21.15.12:0/3892523631 >> 172.21.15.12:0/3892523631 conn(0x7f2cc408aff0 msgr2=0x7f2cc40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cca800700 1 -- 172.21.15.12:0/3892523631 shutdown_connections 2024-03-29T09:55:34.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.150+0000 7f2cca800700 1 -- 172.21.15.12:0/3892523631 wait complete. 2024-03-29T09:55:34.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.153+0000 7f2cca800700 1 Processor -- start 2024-03-29T09:55:34.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.153+0000 7f2cca800700 1 -- start start 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.154+0000 7f2cca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 0x7f2cc40b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.154+0000 7f2cca800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 0x7f2cc415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.154+0000 7f2cca800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cc415d9d0 0x7f2cc4158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.154+0000 7f2cca800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc415ee30 con 0x7f2cc40d16a0 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.154+0000 7f2cca800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc415e870 con 0x7f2cc415d9d0 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.154+0000 7f2cca800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc415eb50 con 0x7f2cc40d2370 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.154+0000 7f2cc3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 0x7f2cc40b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.154+0000 7f2cc3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 0x7f2cc40b4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49904/0 (socket says 172.21.15.12:49904) 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.154+0000 7f2cc3400700 1 -- 172.21.15.12:0/2237414006 learned_addr learned my addr 172.21.15.12:0/2237414006 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 0x7f2cc415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cc3e00700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cc415d9d0 0x7f2cc4158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:34.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cc3400700 1 -- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 msgr2=0x7f2cc415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cc3400700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 0x7f2cc415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cc3400700 1 -- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cc415d9d0 msgr2=0x7f2cc4158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cc3400700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cc415d9d0 0x7f2cc4158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cc3400700 1 -- 172.21.15.12:0/2237414006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb0009440 con 0x7f2cc40d16a0 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cc3e00700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cc415d9d0 0x7f2cc4158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cc3400700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 0x7f2cc40b4550 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400c550 tx=0x7f2cb400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cbb400700 1 -- 172.21.15.12:0/2237414006 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4004c80 con 0x7f2cc40d16a0 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cbb400700 1 -- 172.21.15.12:0/2237414006 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb4004e00 con 0x7f2cc40d16a0 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cbb400700 1 -- 172.21.15.12:0/2237414006 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4007310 con 0x7f2cc40d16a0 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.155+0000 7f2cca800700 1 -- 172.21.15.12:0/2237414006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc415f350 con 0x7f2cc40d16a0 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.156+0000 7f2cca800700 1 -- 172.21.15.12:0/2237414006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc40b8d20 con 0x7f2cc40d16a0 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.157+0000 7f2cca800700 1 -- 172.21.15.12:0/2237414006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc40d1ca0 con 0x7f2cc40d16a0 2024-03-29T09:55:34.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.158+0000 7f2cbb400700 1 -- 172.21.15.12:0/2237414006 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2cb4007470 con 0x7f2cc40d16a0 2024-03-29T09:55:34.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.159+0000 7f2cbb400700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ca406ac90 0x7f2ca406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:34.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.159+0000 7f2cbb400700 1 -- 172.21.15.12:0/2237414006 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f2cb4088e30 con 0x7f2cc40d16a0 2024-03-29T09:55:34.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.159+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ca406ac90 0x7f2ca406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:34.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.160+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ca406ac90 0x7f2ca406d150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40d1ae0 tx=0x7f2cb001c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:34.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.164+0000 7f2cbb400700 1 -- 172.21.15.12:0/2237414006 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2cb40541e0 con 0x7f2cc40d16a0 2024-03-29T09:55:34.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.376+0000 7f2cca800700 1 -- 172.21.15.12:0/2237414006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2cc4067720 con 0x7f2cc40d16a0 2024-03-29T09:55:34.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.924+0000 7f2cbb400700 1 -- 172.21.15.12:0/2237414006 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2cb4057800 con 0x7f2cc40d16a0 2024-03-29T09:55:34.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.931+0000 7f2cca800700 1 -- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ca406ac90 msgr2=0x7f2ca406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:34.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.931+0000 7f2cca800700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ca406ac90 0x7f2ca406d150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40d1ae0 tx=0x7f2cb001c040 comp rx=0 tx=0).stop 2024-03-29T09:55:34.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.931+0000 7f2cca800700 1 -- 172.21.15.12:0/2237414006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 msgr2=0x7f2cc40b4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:34.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.931+0000 7f2cca800700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 0x7f2cc40b4550 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400c550 tx=0x7f2cb400c910 comp rx=0 tx=0).stop 2024-03-29T09:55:34.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.933+0000 7f2cca800700 1 -- 172.21.15.12:0/2237414006 shutdown_connections 2024-03-29T09:55:34.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.933+0000 7f2cca800700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ca406ac90 0x7f2ca406d150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:34.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.933+0000 7f2cca800700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cc415d9d0 0x7f2cc4158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:34.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.933+0000 7f2cca800700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc40d16a0 0x7f2cc40b4550 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:34.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.933+0000 7f2cca800700 1 --2- 172.21.15.12:0/2237414006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cc40d2370 0x7f2cc415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:34.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.933+0000 7f2cca800700 1 -- 172.21.15.12:0/2237414006 >> 172.21.15.12:0/2237414006 conn(0x7f2cc408aff0 msgr2=0x7f2cc415ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:34.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.934+0000 7f2cca800700 1 -- 172.21.15.12:0/2237414006 shutdown_connections 2024-03-29T09:55:34.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:34.934+0000 7f2cca800700 1 -- 172.21.15.12:0/2237414006 wait complete. 2024-03-29T09:55:34.936 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T09:55:34.960 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T09:55:35.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.162+0000 7f6afd800700 1 Processor -- start 2024-03-29T09:55:35.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.162+0000 7f6afd800700 1 -- start start 2024-03-29T09:55:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.163+0000 7f6afd800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6af815d9d0 0x7f6af80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.163+0000 7f6afd800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6af80d16a0 0x7f6af80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.163+0000 7f6afd800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6af80d2370 0x7f6af815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.163+0000 7f6afd800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af80d2820 con 0x7f6af80d2370 2024-03-29T09:55:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.163+0000 7f6afd800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af8161420 con 0x7f6af815d9d0 2024-03-29T09:55:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.163+0000 7f6afd800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af8161700 con 0x7f6af80d16a0 2024-03-29T09:55:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.163+0000 7f6af7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6af80d2370 0x7f6af815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6af80d2370 0x7f6af815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49908/0 (socket says 172.21.15.12:49908) 2024-03-29T09:55:35.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af6a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6af815d9d0 0x7f6af80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af7400700 1 -- 172.21.15.12:0/4147687368 learned_addr learned my addr 172.21.15.12:0/4147687368 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:35.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af6000700 1 --2- 172.21.15.12:0/4147687368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6af80d16a0 0x7f6af80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af7400700 1 -- 172.21.15.12:0/4147687368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6af80d16a0 msgr2=0x7f6af80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:35.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af7400700 1 --2- 172.21.15.12:0/4147687368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6af80d16a0 0x7f6af80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af7400700 1 -- 172.21.15.12:0/4147687368 >> v2:172.21.15.159:3300/0 conn(0x7f6af815d9d0 msgr2=0x7f6af80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:35.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af7400700 1 --2- 172.21.15.12:0/4147687368 >> v2:172.21.15.159:3300/0 conn(0x7f6af815d9d0 0x7f6af80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af7400700 1 -- 172.21.15.12:0/4147687368 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af81619e0 con 0x7f6af80d2370 2024-03-29T09:55:35.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af7400700 1 --2- 172.21.15.12:0/4147687368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af80d2370 0x7f6af815ddb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6af80d0ff0 tx=0x7f6af000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33770b9a4b5a2240 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:35.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.164+0000 7f6af5600700 1 -- 172.21.15.12:0/4147687368 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af000e070 con 0x7f6af80d2370 2024-03-29T09:55:35.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6af5600700 1 -- 172.21.15.12:0/4147687368 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af000b910 con 0x7f6af80d2370 2024-03-29T09:55:35.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6af5600700 1 -- 172.21.15.12:0/4147687368 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af000bcc0 con 0x7f6af80d2370 2024-03-29T09:55:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6afd800700 1 -- 172.21.15.12:0/4147687368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af80d2370 msgr2=0x7f6af815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6afd800700 1 --2- 172.21.15.12:0/4147687368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af80d2370 0x7f6af815ddb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6af80d0ff0 tx=0x7f6af000b330 comp rx=0 tx=0).stop 2024-03-29T09:55:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6afd800700 1 -- 172.21.15.12:0/4147687368 shutdown_connections 2024-03-29T09:55:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6afd800700 1 --2- 172.21.15.12:0/4147687368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af80d2370 0x7f6af815ddb0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6afd800700 1 --2- 172.21.15.12:0/4147687368 >> v2:172.21.15.159:3300/0 conn(0x7f6af815d9d0 0x7f6af80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6afd800700 1 --2- 172.21.15.12:0/4147687368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6af80d16a0 0x7f6af80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6afd800700 1 -- 172.21.15.12:0/4147687368 >> 172.21.15.12:0/4147687368 conn(0x7f6af808aff0 msgr2=0x7f6af80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6afd800700 1 -- 172.21.15.12:0/4147687368 shutdown_connections 2024-03-29T09:55:35.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.165+0000 7f6afd800700 1 -- 172.21.15.12:0/4147687368 wait complete. 2024-03-29T09:55:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.167+0000 7f6afd800700 1 Processor -- start 2024-03-29T09:55:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.167+0000 7f6afd800700 1 -- start start 2024-03-29T09:55:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.168+0000 7f6afd800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6af80d16a0 0x7f6af80b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.168+0000 7f6afd800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6af80d2370 0x7f6af81f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.168+0000 7f6afd800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af815d9d0 0x7f6af80cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.168+0000 7f6afd800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80b8140 con 0x7f6af815d9d0 2024-03-29T09:55:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.168+0000 7f6afd800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af81f3ca0 con 0x7f6af80d16a0 2024-03-29T09:55:35.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.168+0000 7f6afd800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af81f3f80 con 0x7f6af80d2370 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.168+0000 7f6af7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af815d9d0 0x7f6af80cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6af7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af815d9d0 0x7f6af80cb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49916/0 (socket says 172.21.15.12:49916) 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6af7400700 1 -- 172.21.15.12:0/1563001997 learned_addr learned my addr 172.21.15.12:0/1563001997 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6af6a00700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6af80d16a0 0x7f6af80b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6af6000700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6af80d2370 0x7f6af81f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6af7400700 1 -- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6af80d2370 msgr2=0x7f6af81f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6af7400700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6af80d2370 0x7f6af81f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6af7400700 1 -- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6af80d16a0 msgr2=0x7f6af80b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6af7400700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6af80d16a0 0x7f6af80b7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6af7400700 1 -- 172.21.15.12:0/1563001997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af0009b30 con 0x7f6af815d9d0 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6af7400700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af815d9d0 0x7f6af80cb640 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6af808c230 tx=0x7f6af0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.169+0000 7f6aeea00700 1 -- 172.21.15.12:0/1563001997 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af00099b0 con 0x7f6af815d9d0 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.170+0000 7f6aeea00700 1 -- 172.21.15.12:0/1563001997 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af000ce80 con 0x7f6af815d9d0 2024-03-29T09:55:35.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.170+0000 7f6aeea00700 1 -- 172.21.15.12:0/1563001997 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0004490 con 0x7f6af815d9d0 2024-03-29T09:55:35.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.170+0000 7f6afd800700 1 -- 172.21.15.12:0/1563001997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af80b61a0 con 0x7f6af815d9d0 2024-03-29T09:55:35.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.170+0000 7f6afd800700 1 -- 172.21.15.12:0/1563001997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af80cd230 con 0x7f6af815d9d0 2024-03-29T09:55:35.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.171+0000 7f6afd800700 1 -- 172.21.15.12:0/1563001997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af80d1ca0 con 0x7f6af815d9d0 2024-03-29T09:55:35.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.172+0000 7f6aeea00700 1 -- 172.21.15.12:0/1563001997 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6af00045f0 con 0x7f6af815d9d0 2024-03-29T09:55:35.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.172+0000 7f6aeea00700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ad806ac90 0x7f6ad806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.173+0000 7f6aeea00700 1 -- 172.21.15.12:0/1563001997 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6af008a5f0 con 0x7f6af815d9d0 2024-03-29T09:55:35.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.173+0000 7f6af6a00700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ad806ac90 0x7f6ad806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.174+0000 7f6af6a00700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ad806ac90 0x7f6ad806d150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009b40 tx=0x7f6ae8009210 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.178+0000 7f6aeea00700 1 -- 172.21.15.12:0/1563001997 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6af0055820 con 0x7f6af815d9d0 2024-03-29T09:55:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.436+0000 7f6afd800700 1 -- 172.21.15.12:0/1563001997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6af80b66c0 con 0x7f6af815d9d0 2024-03-29T09:55:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.444+0000 7f6aeea00700 1 -- 172.21.15.12:0/1563001997 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v11) v1 ==== 172+0+0 (secure 0 0 0) 0x7f6af80b66c0 con 0x7f6af815d9d0 2024-03-29T09:55:35.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.449+0000 7f6afd800700 1 -- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ad806ac90 msgr2=0x7f6ad806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.450+0000 7f6afd800700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ad806ac90 0x7f6ad806d150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009b40 tx=0x7f6ae8009210 comp rx=0 tx=0).stop 2024-03-29T09:55:35.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.450+0000 7f6afd800700 1 -- 172.21.15.12:0/1563001997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af815d9d0 msgr2=0x7f6af80cb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.450+0000 7f6afd800700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af815d9d0 0x7f6af80cb640 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6af808c230 tx=0x7f6af0004ad0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.451+0000 7f6afd800700 1 -- 172.21.15.12:0/1563001997 shutdown_connections 2024-03-29T09:55:35.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.451+0000 7f6afd800700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ad806ac90 0x7f6ad806d150 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.451+0000 7f6afd800700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6af815d9d0 0x7f6af80cb640 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.451+0000 7f6afd800700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6af80d16a0 0x7f6af80b7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.451+0000 7f6afd800700 1 --2- 172.21.15.12:0/1563001997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6af80d2370 0x7f6af81f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.451+0000 7f6afd800700 1 -- 172.21.15.12:0/1563001997 >> 172.21.15.12:0/1563001997 conn(0x7f6af808aff0 msgr2=0x7f6af80ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:35.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.454+0000 7f6afd800700 1 -- 172.21.15.12:0/1563001997 shutdown_connections 2024-03-29T09:55:35.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.454+0000 7f6afd800700 1 -- 172.21.15.12:0/1563001997 wait complete. 2024-03-29T09:55:35.462 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 4 and data pool 5 2024-03-29T09:55:35.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:35.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.687+0000 7fc96ea00700 1 Processor -- start 2024-03-29T09:55:35.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.687+0000 7fc96ea00700 1 -- start start 2024-03-29T09:55:35.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.688+0000 7fc96ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc96815d710 0x7fc9680d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.688+0000 7fc96ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc96809aba0 0x7fc9680d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.688+0000 7fc96ea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96806bb40 con 0x7fc96806bf40 2024-03-29T09:55:35.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.688+0000 7fc96ea00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc968099b00 con 0x7fc96815d710 2024-03-29T09:55:35.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.688+0000 7fc96ea00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc968099de0 con 0x7fc96809aba0 2024-03-29T09:55:35.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.688+0000 7fc967e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc96815d710 0x7fc9680d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.688+0000 7fc967e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc96815d710 0x7fc9680d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49930/0 (socket says 172.21.15.12:49930) 2024-03-29T09:55:35.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.688+0000 7fc967e00700 1 -- 172.21.15.12:0/3284724448 learned_addr learned my addr 172.21.15.12:0/3284724448 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:35.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.688+0000 7fc96ca00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc96806bf40 0x7fc9680ccbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44246/0 (socket says 172.21.15.12:44246) 2024-03-29T09:55:35.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.689+0000 7fc967400700 1 --2- 172.21.15.12:0/3284724448 >> v2:172.21.15.159:3300/0 conn(0x7fc96809aba0 0x7fc9680d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.689+0000 7fc967e00700 1 -- 172.21.15.12:0/3284724448 >> v2:172.21.15.159:3300/0 conn(0x7fc96809aba0 msgr2=0x7fc9680d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:35.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.689+0000 7fc967e00700 1 --2- 172.21.15.12:0/3284724448 >> v2:172.21.15.159:3300/0 conn(0x7fc96809aba0 0x7fc9680d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.689+0000 7fc967e00700 1 -- 172.21.15.12:0/3284724448 >> v1:172.21.15.12:6789/0 conn(0x7fc96806bf40 legacy=0x7fc9680ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.689+0000 7fc967e00700 1 -- 172.21.15.12:0/3284724448 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96809a0c0 con 0x7fc96815d710 2024-03-29T09:55:35.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.689+0000 7fc967e00700 1 --2- 172.21.15.12:0/3284724448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96815d710 0x7fc9680d0130 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc968072100 tx=0x7fc958009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36244c4439b858f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:35.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.689+0000 7fc966a00700 1 -- 172.21.15.12:0/3284724448 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95800b870 con 0x7fc96815d710 2024-03-29T09:55:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.690+0000 7fc966a00700 1 -- 172.21.15.12:0/3284724448 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95800b9d0 con 0x7fc96815d710 2024-03-29T09:55:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.690+0000 7fc966a00700 1 -- 172.21.15.12:0/3284724448 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95800bd60 con 0x7fc96815d710 2024-03-29T09:55:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.690+0000 7fc96ea00700 1 -- 172.21.15.12:0/3284724448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96815d710 msgr2=0x7fc9680d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.690+0000 7fc96ea00700 1 --2- 172.21.15.12:0/3284724448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96815d710 0x7fc9680d0130 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc968072100 tx=0x7fc958009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.690+0000 7fc96ea00700 1 -- 172.21.15.12:0/3284724448 shutdown_connections 2024-03-29T09:55:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.690+0000 7fc96ea00700 1 --2- 172.21.15.12:0/3284724448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96815d710 0x7fc9680d0130 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.690+0000 7fc96ea00700 1 --2- 172.21.15.12:0/3284724448 >> v2:172.21.15.159:3300/0 conn(0x7fc96809aba0 0x7fc9680d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.690+0000 7fc96ea00700 1 -- 172.21.15.12:0/3284724448 >> 172.21.15.12:0/3284724448 conn(0x7fc9680d13e0 msgr2=0x7fc9680d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.690+0000 7fc96ea00700 1 -- 172.21.15.12:0/3284724448 shutdown_connections 2024-03-29T09:55:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.690+0000 7fc96ea00700 1 -- 172.21.15.12:0/3284724448 wait complete. 2024-03-29T09:55:35.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.692+0000 7fc96ea00700 1 Processor -- start 2024-03-29T09:55:35.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.692+0000 7fc96ea00700 1 -- start start 2024-03-29T09:55:35.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.693+0000 7fc96ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96806bf40 0x7fc968157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.693+0000 7fc96ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96809aba0 0x7fc9681f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.693+0000 7fc96ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96815d710 0x7fc96815f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.693+0000 7fc96ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96815efc0 con 0x7fc96806bf40 2024-03-29T09:55:35.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.693+0000 7fc96ea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9681f6a10 con 0x7fc96815d710 2024-03-29T09:55:35.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.693+0000 7fc96ea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc968160cb0 con 0x7fc96809aba0 2024-03-29T09:55:35.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.693+0000 7fc967e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96806bf40 0x7fc968157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.693+0000 7fc967e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96806bf40 0x7fc968157260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49934/0 (socket says 172.21.15.12:49934) 2024-03-29T09:55:35.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.693+0000 7fc967e00700 1 -- 172.21.15.12:0/733863932 learned_addr learned my addr 172.21.15.12:0/733863932 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc967400700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96809aba0 0x7fc9681f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc96ca00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96815d710 0x7fc96815f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc967e00700 1 -- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96809aba0 msgr2=0x7fc9681f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc967e00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96809aba0 0x7fc9681f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc967e00700 1 -- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96815d710 msgr2=0x7fc96815f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc967e00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96815d710 0x7fc96815f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc967e00700 1 -- 172.21.15.12:0/733863932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc958009550 con 0x7fc96806bf40 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc967400700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96809aba0 0x7fc9681f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc96ca00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96815d710 0x7fc96815f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc967e00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96806bf40 0x7fc968157260 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc968072690 tx=0x7fc958004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc95fe00700 1 -- 172.21.15.12:0/733863932 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958004520 con 0x7fc96806bf40 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc95fe00700 1 -- 172.21.15.12:0/733863932 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95801e070 con 0x7fc96806bf40 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc95fe00700 1 -- 172.21.15.12:0/733863932 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9580038a0 con 0x7fc96806bf40 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.694+0000 7fc96ea00700 1 -- 172.21.15.12:0/733863932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9681609d0 con 0x7fc96806bf40 2024-03-29T09:55:35.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.695+0000 7fc96ea00700 1 -- 172.21.15.12:0/733863932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96815ec50 con 0x7fc96806bf40 2024-03-29T09:55:35.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.696+0000 7fc96ea00700 1 -- 172.21.15.12:0/733863932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96815ee10 con 0x7fc96806bf40 2024-03-29T09:55:35.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.696+0000 7fc95fe00700 1 -- 172.21.15.12:0/733863932 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc958003a00 con 0x7fc96806bf40 2024-03-29T09:55:35.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.697+0000 7fc95fe00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc94806acc0 0x7fc94806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:35.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.697+0000 7fc95fe00700 1 -- 172.21.15.12:0/733863932 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc958020070 con 0x7fc96806bf40 2024-03-29T09:55:35.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.698+0000 7fc967400700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc94806acc0 0x7fc94806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.698+0000 7fc967400700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc94806acc0 0x7fc94806d180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc968158240 tx=0x7fc954008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:35.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.702+0000 7fc95fe00700 1 -- 172.21.15.12:0/733863932 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc958055ab0 con 0x7fc96806bf40 2024-03-29T09:55:35.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.919+0000 7fc96ea00700 1 -- 172.21.15.12:0/733863932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9680c81e0 con 0x7fc96806bf40 2024-03-29T09:55:35.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.921+0000 7fc95fe00700 1 -- 172.21.15.12:0/733863932 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16114 (secure 0 0 0) 0x7fc958015090 con 0x7fc96806bf40 2024-03-29T09:55:35.920 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:35.920 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":33,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:55:35.437916+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:55:33.508935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T09:55:34.418367+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 -- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc94806acc0 msgr2=0x7fc94806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc94806acc0 0x7fc94806d180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc968158240 tx=0x7fc954008040 comp rx=0 tx=0).stop 2024-03-29T09:55:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 -- 172.21.15.12:0/733863932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96806bf40 msgr2=0x7fc968157260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96806bf40 0x7fc968157260 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc968072690 tx=0x7fc958004750 comp rx=0 tx=0).stop 2024-03-29T09:55:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 -- 172.21.15.12:0/733863932 shutdown_connections 2024-03-29T09:55:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc94806acc0 0x7fc94806d180 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96815d710 0x7fc96815f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc96806bf40 0x7fc968157260 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 --2- 172.21.15.12:0/733863932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96809aba0 0x7fc9681f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 -- 172.21.15.12:0/733863932 >> 172.21.15.12:0/733863932 conn(0x7fc9680d13e0 msgr2=0x7fc9680b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.923+0000 7fc96ea00700 1 -- 172.21.15.12:0/733863932 shutdown_connections 2024-03-29T09:55:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:35.924+0000 7fc96ea00700 1 -- 172.21.15.12:0/733863932 wait complete. 2024-03-29T09:55:35.942 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:36.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.139+0000 7f3737400700 1 Processor -- start 2024-03-29T09:55:36.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.139+0000 7f3737400700 1 -- start start 2024-03-29T09:55:36.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.140+0000 7f3737400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f373015b5d0 0x7f37300ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.141+0000 7f3737400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f373015fd50 con 0x7f37300cb3f0 2024-03-29T09:55:36.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.141+0000 7f3737400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f373015f140 con 0x7f37300ac7d0 2024-03-29T09:55:36.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.141+0000 7f3737400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373015b9b0 con 0x7f373015b5d0 2024-03-29T09:55:36.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.141+0000 7f372fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f37300cb3f0 0x7f37300ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44258/0 (socket says 172.21.15.12:44258) 2024-03-29T09:55:36.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.141+0000 7f372fe00700 1 -- 172.21.15.12:0/953928475 learned_addr learned my addr 172.21.15.12:0/953928475 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:36.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.141+0000 7f3734a00700 1 --2- 172.21.15.12:0/953928475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f373015b5d0 0x7f37300ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:36.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.141+0000 7f372f400700 1 -- 172.21.15.12:0/953928475 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746160016 0 0) 0x7f373015fd50 con 0x7f37300cb3f0 2024-03-29T09:55:36.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.141+0000 7f372f400700 1 -- 172.21.15.12:0/953928475 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37180035b0 con 0x7f37300cb3f0 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.141+0000 7f372f400700 1 -- 172.21.15.12:0/953928475 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619359794 0 0) 0x7f373015f140 con 0x7f37300ac7d0 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.142+0000 7f372f400700 1 -- 172.21.15.12:0/953928475 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f373015fd50 con 0x7f37300ac7d0 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.142+0000 7f3734a00700 1 -- 172.21.15.12:0/953928475 >> v1:172.21.15.159:6789/0 conn(0x7f37300ac7d0 legacy=0x7f373006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.142+0000 7f3734a00700 1 -- 172.21.15.12:0/953928475 >> v1:172.21.15.12:6789/0 conn(0x7f37300cb3f0 legacy=0x7f37300ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.142+0000 7f3734a00700 1 -- 172.21.15.12:0/953928475 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373015f420 con 0x7f373015b5d0 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.142+0000 7f3734a00700 1 --2- 172.21.15.12:0/953928475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f373015b5d0 0x7f37300ae3f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3724009870 tx=0x7f3724009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2d7f4861490fb8e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.142+0000 7f372f400700 1 -- 172.21.15.12:0/953928475 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724011830 con 0x7f373015b5d0 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.142+0000 7f372f400700 1 -- 172.21.15.12:0/953928475 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3724011990 con 0x7f373015b5d0 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.142+0000 7f372f400700 1 -- 172.21.15.12:0/953928475 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724011d20 con 0x7f373015b5d0 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.143+0000 7f3737400700 1 -- 172.21.15.12:0/953928475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f373015b5d0 msgr2=0x7f37300ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.143+0000 7f3737400700 1 --2- 172.21.15.12:0/953928475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f373015b5d0 0x7f37300ae3f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3724009870 tx=0x7f3724009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.143+0000 7f3737400700 1 -- 172.21.15.12:0/953928475 shutdown_connections 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.143+0000 7f3737400700 1 --2- 172.21.15.12:0/953928475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f373015b5d0 0x7f37300ae3f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.143+0000 7f3737400700 1 -- 172.21.15.12:0/953928475 >> 172.21.15.12:0/953928475 conn(0x7f37300cf190 msgr2=0x7f37300cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.143+0000 7f3737400700 1 -- 172.21.15.12:0/953928475 shutdown_connections 2024-03-29T09:55:36.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.143+0000 7f3737400700 1 -- 172.21.15.12:0/953928475 wait complete. 2024-03-29T09:55:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.145+0000 7f3737400700 1 Processor -- start 2024-03-29T09:55:36.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.145+0000 7f3737400700 1 -- start start 2024-03-29T09:55:36.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f3737400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37300ac7d0 0x7f3730156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f3737400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f373015b5d0 0x7f37301f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f3737400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37300cbba0 0x7f373015a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f3737400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373015d880 con 0x7f373015b5d0 2024-03-29T09:55:36.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f3737400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373015ae00 con 0x7f37300cbba0 2024-03-29T09:55:36.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f3737400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37301f3190 con 0x7f37300ac7d0 2024-03-29T09:55:36.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f372fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f373015b5d0 0x7f37301f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:36.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f372fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f373015b5d0 0x7f37301f2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49948/0 (socket says 172.21.15.12:49948) 2024-03-29T09:55:36.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f372fe00700 1 -- 172.21.15.12:0/2400779584 learned_addr learned my addr 172.21.15.12:0/2400779584 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:36.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f372fe00700 1 -- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37300ac7d0 msgr2=0x7f3730156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:36.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f372fe00700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37300ac7d0 0x7f3730156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f372fe00700 1 -- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37300cbba0 msgr2=0x7f373015a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:36.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f372fe00700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37300cbba0 0x7f373015a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.146+0000 7f372fe00700 1 -- 172.21.15.12:0/2400779584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37240094a0 con 0x7f373015b5d0 2024-03-29T09:55:36.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.147+0000 7f372fe00700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f373015b5d0 0x7f37301f2b70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f371c00c700 tx=0x7f371c00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:36.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.147+0000 7f372cc00700 1 -- 172.21.15.12:0/2400779584 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c005d70 con 0x7f373015b5d0 2024-03-29T09:55:36.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.147+0000 7f372cc00700 1 -- 172.21.15.12:0/2400779584 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c0070d0 con 0x7f373015b5d0 2024-03-29T09:55:36.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.147+0000 7f372cc00700 1 -- 172.21.15.12:0/2400779584 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c007480 con 0x7f373015b5d0 2024-03-29T09:55:36.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.147+0000 7f3737400700 1 -- 172.21.15.12:0/2400779584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37301f3470 con 0x7f373015b5d0 2024-03-29T09:55:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.147+0000 7f3737400700 1 -- 172.21.15.12:0/2400779584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37301f3a30 con 0x7f373015b5d0 2024-03-29T09:55:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.148+0000 7f3737400700 1 -- 172.21.15.12:0/2400779584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3730067720 con 0x7f373015b5d0 2024-03-29T09:55:36.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.149+0000 7f372cc00700 1 -- 172.21.15.12:0/2400779584 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f371c0075e0 con 0x7f373015b5d0 2024-03-29T09:55:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.150+0000 7f372cc00700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f371006ac90 0x7f371006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.150+0000 7f372cc00700 1 -- 172.21.15.12:0/2400779584 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f371c088a50 con 0x7f373015b5d0 2024-03-29T09:55:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.150+0000 7f3734a00700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f371006ac90 0x7f371006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:36.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.151+0000 7f3734a00700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f371006ac90 0x7f371006d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f37240090b0 tx=0x7f3724010000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:36.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.155+0000 7f372cc00700 1 -- 172.21.15.12:0/2400779584 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f371c053ba0 con 0x7f373015b5d0 2024-03-29T09:55:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.369+0000 7f3737400700 1 -- 172.21.15.12:0/2400779584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3730157330 con 0x7f373015b5d0 2024-03-29T09:55:36.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.370+0000 7f372cc00700 1 -- 172.21.15.12:0/2400779584 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16114 (secure 0 0 0) 0x7f371c0571c0 con 0x7f373015b5d0 2024-03-29T09:55:36.370 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:36.370 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":33,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:55:35.437916+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:55:33.508935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T09:55:34.418367+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:36.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.372+0000 7f3737400700 1 -- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f371006ac90 msgr2=0x7f371006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:36.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.372+0000 7f3737400700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f371006ac90 0x7f371006d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f37240090b0 tx=0x7f3724010000 comp rx=0 tx=0).stop 2024-03-29T09:55:36.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.372+0000 7f3737400700 1 -- 172.21.15.12:0/2400779584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f373015b5d0 msgr2=0x7f37301f2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.372+0000 7f3737400700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f373015b5d0 0x7f37301f2b70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f371c00c700 tx=0x7f371c00cac0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.372+0000 7f3737400700 1 -- 172.21.15.12:0/2400779584 shutdown_connections 2024-03-29T09:55:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.372+0000 7f3737400700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f371006ac90 0x7f371006d150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.372+0000 7f3737400700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37300cbba0 0x7f373015a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.372+0000 7f3737400700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37300ac7d0 0x7f3730156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.372+0000 7f3737400700 1 --2- 172.21.15.12:0/2400779584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f373015b5d0 0x7f37301f2b70 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.372+0000 7f3737400700 1 -- 172.21.15.12:0/2400779584 >> 172.21.15.12:0/2400779584 conn(0x7f37300cf190 msgr2=0x7f37300b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.373+0000 7f3737400700 1 -- 172.21.15.12:0/2400779584 shutdown_connections 2024-03-29T09:55:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.373+0000 7f3737400700 1 -- 172.21.15.12:0/2400779584 wait complete. 2024-03-29T09:55:36.390 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T09:55:36.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.594+0000 7f7f56c00700 1 Processor -- start 2024-03-29T09:55:36.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.594+0000 7f7f56c00700 1 -- start start 2024-03-29T09:55:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.595+0000 7f7f56c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f5015d9d0 0x7f7f500d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.595+0000 7f7f56c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7f500d16a0 0x7f7f500d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.595+0000 7f7f56c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f500d2370 0x7f7f5015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.595+0000 7f7f56c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f500d2820 con 0x7f7f5015d9d0 2024-03-29T09:55:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.595+0000 7f7f56c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f50161420 con 0x7f7f500d16a0 2024-03-29T09:55:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.595+0000 7f7f56c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f50161700 con 0x7f7f500d2370 2024-03-29T09:55:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.595+0000 7f7f4fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f5015d9d0 0x7f7f500d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.595+0000 7f7f4fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f5015d9d0 0x7f7f500d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49954/0 (socket says 172.21.15.12:49954) 2024-03-29T09:55:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.595+0000 7f7f4fe00700 1 -- 172.21.15.12:0/541086053 learned_addr learned my addr 172.21.15.12:0/541086053 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.595+0000 7f7f54c00700 1 --2- 172.21.15.12:0/541086053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f500d2370 0x7f7f5015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.596+0000 7f7f4f400700 1 --2- 172.21.15.12:0/541086053 >> v2:172.21.15.159:3300/0 conn(0x7f7f500d16a0 0x7f7f500d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.596+0000 7f7f4fe00700 1 -- 172.21.15.12:0/541086053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f500d2370 msgr2=0x7f7f5015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.596+0000 7f7f4fe00700 1 --2- 172.21.15.12:0/541086053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f500d2370 0x7f7f5015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.596+0000 7f7f4fe00700 1 -- 172.21.15.12:0/541086053 >> v2:172.21.15.159:3300/0 conn(0x7f7f500d16a0 msgr2=0x7f7f500d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.596+0000 7f7f4fe00700 1 --2- 172.21.15.12:0/541086053 >> v2:172.21.15.159:3300/0 conn(0x7f7f500d16a0 0x7f7f500d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.596+0000 7f7f4fe00700 1 -- 172.21.15.12:0/541086053 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f501619e0 con 0x7f7f5015d9d0 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.596+0000 7f7f4fe00700 1 --2- 172.21.15.12:0/541086053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f5015d9d0 0x7f7f500d3c30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7f500d12f0 tx=0x7f7f40009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9f0473d1406b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.596+0000 7f7f4ea00700 1 -- 172.21.15.12:0/541086053 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4000b870 con 0x7f7f5015d9d0 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.596+0000 7f7f4ea00700 1 -- 172.21.15.12:0/541086053 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4000b9d0 con 0x7f7f5015d9d0 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.596+0000 7f7f4ea00700 1 -- 172.21.15.12:0/541086053 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4000bd60 con 0x7f7f5015d9d0 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.597+0000 7f7f56c00700 1 -- 172.21.15.12:0/541086053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f5015d9d0 msgr2=0x7f7f500d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.597+0000 7f7f56c00700 1 --2- 172.21.15.12:0/541086053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f5015d9d0 0x7f7f500d3c30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7f500d12f0 tx=0x7f7f40009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.597+0000 7f7f56c00700 1 -- 172.21.15.12:0/541086053 shutdown_connections 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.597+0000 7f7f56c00700 1 --2- 172.21.15.12:0/541086053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f500d2370 0x7f7f5015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.597+0000 7f7f56c00700 1 --2- 172.21.15.12:0/541086053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f5015d9d0 0x7f7f500d3c30 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.597+0000 7f7f56c00700 1 --2- 172.21.15.12:0/541086053 >> v2:172.21.15.159:3300/0 conn(0x7f7f500d16a0 0x7f7f500d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.597+0000 7f7f56c00700 1 -- 172.21.15.12:0/541086053 >> 172.21.15.12:0/541086053 conn(0x7f7f5008aff0 msgr2=0x7f7f500d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.597+0000 7f7f56c00700 1 -- 172.21.15.12:0/541086053 shutdown_connections 2024-03-29T09:55:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.597+0000 7f7f56c00700 1 -- 172.21.15.12:0/541086053 wait complete. 2024-03-29T09:55:36.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.599+0000 7f7f56c00700 1 Processor -- start 2024-03-29T09:55:36.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.599+0000 7f7f56c00700 1 -- start start 2024-03-29T09:55:36.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f56c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f500d16a0 0x7f7f501f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f56c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f500d2370 0x7f7f5014d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f56c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f5015d9d0 0x7f7f500cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f56c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f501f6d80 con 0x7f7f500d2370 2024-03-29T09:55:36.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f56c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f501f7340 con 0x7f7f5015d9d0 2024-03-29T09:55:36.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f56c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f501f7ca0 con 0x7f7f500d16a0 2024-03-29T09:55:36.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f4f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f500d2370 0x7f7f5014d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f4f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f500d2370 0x7f7f5014d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49964/0 (socket says 172.21.15.12:49964) 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f4f400700 1 -- 172.21.15.12:0/1886933533 learned_addr learned my addr 172.21.15.12:0/1886933533 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f4fe00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f500d16a0 0x7f7f501f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f54c00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f5015d9d0 0x7f7f500cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f4f400700 1 -- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f500d16a0 msgr2=0x7f7f501f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f4f400700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f500d16a0 0x7f7f501f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f4f400700 1 -- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f5015d9d0 msgr2=0x7f7f500cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f4f400700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f5015d9d0 0x7f7f500cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.600+0000 7f7f4f400700 1 -- 172.21.15.12:0/1886933533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f40009550 con 0x7f7f500d2370 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.601+0000 7f7f4f400700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f500d2370 0x7f7f5014d2c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c00e810 tx=0x7f7f3c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.601+0000 7f7f47e00700 1 -- 172.21.15.12:0/1886933533 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c00c070 con 0x7f7f500d2370 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.601+0000 7f7f47e00700 1 -- 172.21.15.12:0/1886933533 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3c013610 con 0x7f7f500d2370 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.601+0000 7f7f47e00700 1 -- 172.21.15.12:0/1886933533 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c0139a0 con 0x7f7f500d2370 2024-03-29T09:55:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.601+0000 7f7f56c00700 1 -- 172.21.15.12:0/1886933533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5014d800 con 0x7f7f500d2370 2024-03-29T09:55:36.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.601+0000 7f7f56c00700 1 -- 172.21.15.12:0/1886933533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f50161030 con 0x7f7f500d2370 2024-03-29T09:55:36.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.602+0000 7f7f56c00700 1 -- 172.21.15.12:0/1886933533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f500d1ca0 con 0x7f7f500d2370 2024-03-29T09:55:36.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.603+0000 7f7f47e00700 1 -- 172.21.15.12:0/1886933533 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7f3c013b00 con 0x7f7f500d2370 2024-03-29T09:55:36.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.604+0000 7f7f47e00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f3006ac90 0x7f7f3006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:36.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.604+0000 7f7f47e00700 1 -- 172.21.15.12:0/1886933533 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f3c088ff0 con 0x7f7f500d2370 2024-03-29T09:55:36.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.604+0000 7f7f4fe00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f3006ac90 0x7f7f3006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:36.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.605+0000 7f7f4fe00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f3006ac90 0x7f7f3006d150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7f40014040 tx=0x7f7f40008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:36.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.608+0000 7f7f47e00700 1 -- 172.21.15.12:0/1886933533 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f3c054150 con 0x7f7f500d2370 2024-03-29T09:55:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:36.870+0000 7f7f56c00700 1 -- 172.21.15.12:0/1886933533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7f50161250 con 0x7f7f500d2370 2024-03-29T09:55:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.450+0000 7f7f47e00700 1 -- 172.21.15.12:0/1886933533 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7f3c013db0 con 0x7f7f500d2370 2024-03-29T09:55:37.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.453+0000 7f7f56c00700 1 -- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f3006ac90 msgr2=0x7f7f3006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:37.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.453+0000 7f7f56c00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f3006ac90 0x7f7f3006d150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7f40014040 tx=0x7f7f40008030 comp rx=0 tx=0).stop 2024-03-29T09:55:37.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.453+0000 7f7f56c00700 1 -- 172.21.15.12:0/1886933533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f500d2370 msgr2=0x7f7f5014d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:37.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.453+0000 7f7f56c00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f500d2370 0x7f7f5014d2c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c00e810 tx=0x7f7f3c00eb20 comp rx=0 tx=0).stop 2024-03-29T09:55:37.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.454+0000 7f7f56c00700 1 -- 172.21.15.12:0/1886933533 shutdown_connections 2024-03-29T09:55:37.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.454+0000 7f7f56c00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f3006ac90 0x7f7f3006d150 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:37.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.454+0000 7f7f56c00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f5015d9d0 0x7f7f500cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:37.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.454+0000 7f7f56c00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f500d16a0 0x7f7f501f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:37.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.454+0000 7f7f56c00700 1 --2- 172.21.15.12:0/1886933533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f500d2370 0x7f7f5014d2c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:37.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.454+0000 7f7f56c00700 1 -- 172.21.15.12:0/1886933533 >> 172.21.15.12:0/1886933533 conn(0x7f7f5008aff0 msgr2=0x7f7f500ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:37.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.454+0000 7f7f56c00700 1 -- 172.21.15.12:0/1886933533 shutdown_connections 2024-03-29T09:55:37.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.454+0000 7f7f56c00700 1 -- 172.21.15.12:0/1886933533 wait complete. 2024-03-29T09:55:37.474 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T09:55:37.474 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T09:55:37.625 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-29T09:55:37.626+0000 7f412e600700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T09:55:37.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.680+0000 7f224e600700 1 Processor -- start 2024-03-29T09:55:37.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.681+0000 7f224e600700 1 -- start start 2024-03-29T09:55:37.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.682+0000 7f224e600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f22480cd020 0x7f22480cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:37.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.682+0000 7f224e600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22480b2070 0x7f22480b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:37.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.682+0000 7f224e600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480cdae0 con 0x7f22480cbb30 2024-03-29T09:55:37.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.682+0000 7f224e600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22480cddc0 con 0x7f22480cd020 2024-03-29T09:55:37.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.682+0000 7f224e600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22480ce0a0 con 0x7f22480b2070 2024-03-29T09:55:37.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.682+0000 7f2247400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f22480cbb30 0x7f22480cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44260/0 (socket says 172.21.15.12:44260) 2024-03-29T09:55:37.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.682+0000 7f2247400700 1 -- 172.21.15.12:0/3649258494 learned_addr learned my addr 172.21.15.12:0/3649258494 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:37.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.682+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629651367 0 0) 0x7f22480cdae0 con 0x7f22480cbb30 2024-03-29T09:55:37.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2247e00700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22480b2070 0x7f22480b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:37.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246a00700 1 --2- 172.21.15.12:0/3649258494 >> v2:172.21.15.159:3300/0 conn(0x7f22480cd020 0x7f22480cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c0035b0 con 0x7f22480cbb30 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4056644647 0 0) 0x7f222c0035b0 con 0x7f22480cbb30 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22480b2070 msgr2=0x7f22480b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22480b2070 0x7f22480b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 >> v2:172.21.15.159:3300/0 conn(0x7f22480cd020 msgr2=0x7f22480cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 --2- 172.21.15.12:0/3649258494 >> v2:172.21.15.159:3300/0 conn(0x7f22480cd020 0x7f22480cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224806dc20 con 0x7f22480cbb30 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f2238001670 con 0x7f22480cbb30 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 >> v1:172.21.15.12:6789/0 conn(0x7f22480cbb30 legacy=0x7f22480cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22480b2070 0x7f222c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f222c003fd0 0x7f222c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f222c004d90 0x7f222c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22480cddc0 con 0x7f222c004d90 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22480ce0a0 con 0x7f222c003fd0 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c00ba10 con 0x7f22480b2070 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.683+0000 7f2247e00700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f222c004d90 0x7f222c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2247400700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22480b2070 0x7f222c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2246a00700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f222c003fd0 0x7f222c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2247e00700 1 -- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22480b2070 msgr2=0x7f222c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2247e00700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22480b2070 0x7f222c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2247e00700 1 -- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f222c003fd0 msgr2=0x7f222c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2247e00700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f222c003fd0 0x7f222c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2247e00700 1 -- 172.21.15.12:0/3649258494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c0035b0 con 0x7f222c004d90 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2247400700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22480b2070 0x7f222c003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2247e00700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f222c004d90 0x7f222c005210 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f224000e170 tx=0x7f222c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56245ab3812b3ea server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224000ec90 con 0x7f222c004d90 2024-03-29T09:55:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224000f040 con 0x7f222c004d90 2024-03-29T09:55:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.684+0000 7f2246000700 1 -- 172.21.15.12:0/3649258494 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240014480 con 0x7f222c004d90 2024-03-29T09:55:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.685+0000 7f224e600700 1 -- 172.21.15.12:0/3649258494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f222c004d90 msgr2=0x7f222c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.685+0000 7f224e600700 1 --2- 172.21.15.12:0/3649258494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f222c004d90 0x7f222c005210 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f224000e170 tx=0x7f222c005880 comp rx=0 tx=0).stop 2024-03-29T09:55:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.685+0000 7f2247e00700 1 -- 172.21.15.12:0/3649258494 reap_dead start 2024-03-29T09:55:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.685+0000 7f224e600700 1 -- 172.21.15.12:0/3649258494 shutdown_connections 2024-03-29T09:55:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.685+0000 7f224e600700 1 -- 172.21.15.12:0/3649258494 >> 172.21.15.12:0/3649258494 conn(0x7f22480b0d70 msgr2=0x7f22480b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.685+0000 7f224e600700 1 -- 172.21.15.12:0/3649258494 shutdown_connections 2024-03-29T09:55:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.685+0000 7f224e600700 1 -- 172.21.15.12:0/3649258494 wait complete. 2024-03-29T09:55:37.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.687+0000 7f224e600700 1 Processor -- start 2024-03-29T09:55:37.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.687+0000 7f224e600700 1 -- start start 2024-03-29T09:55:37.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f224e600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f222c004d90 0x7f2248156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:37.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f224e600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2248098580 0x7f2248098a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:37.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f224e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22480acfb0 0x7f22480ad460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:37.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f224e600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2248098f40 con 0x7f22480acfb0 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f224e600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2248140d50 con 0x7f2248098580 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f224e600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2248140460 con 0x7f222c004d90 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f2247e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22480acfb0 0x7f22480ad460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f2247400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f222c004d90 0x7f2248156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f2246a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2248098580 0x7f2248098a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f2247400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f222c004d90 0x7f2248156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:48132/0 (socket says 172.21.15.12:48132) 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f2247e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22480acfb0 0x7f22480ad460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49984/0 (socket says 172.21.15.12:49984) 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.688+0000 7f2247400700 1 -- 172.21.15.12:0/2272180002 learned_addr learned my addr 172.21.15.12:0/2272180002 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f2247e00700 1 -- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f222c004d90 msgr2=0x7f2248156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f2247e00700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f222c004d90 0x7f2248156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f2247e00700 1 -- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2248098580 msgr2=0x7f2248098a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f2247e00700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2248098580 0x7f2248098a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f2247e00700 1 -- 172.21.15.12:0/2272180002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22480add50 con 0x7f22480acfb0 2024-03-29T09:55:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f2247400700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f222c004d90 0x7f2248156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f2246a00700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2248098580 0x7f2248098a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f2247e00700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22480acfb0 0x7f22480ad460 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2240019000 tx=0x7f2240009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f223f400700 1 -- 172.21.15.12:0/2272180002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240007000 con 0x7f22480acfb0 2024-03-29T09:55:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f223f400700 1 -- 172.21.15.12:0/2272180002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2240003c60 con 0x7f22480acfb0 2024-03-29T09:55:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f223f400700 1 -- 172.21.15.12:0/2272180002 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224000eea0 con 0x7f22480acfb0 2024-03-29T09:55:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f224e600700 1 -- 172.21.15.12:0/2272180002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2248140700 con 0x7f22480acfb0 2024-03-29T09:55:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.689+0000 7f224e600700 1 -- 172.21.15.12:0/2272180002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2248154040 con 0x7f22480acfb0 2024-03-29T09:55:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.690+0000 7f224e600700 1 -- 172.21.15.12:0/2272180002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22480cdae0 con 0x7f22480acfb0 2024-03-29T09:55:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.691+0000 7f223f400700 1 -- 172.21.15.12:0/2272180002 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2240004070 con 0x7f22480acfb0 2024-03-29T09:55:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.693+0000 7f223f400700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f222806ac90 0x7f222806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.693+0000 7f223f400700 1 -- 172.21.15.12:0/2272180002 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f224008e8b0 con 0x7f22480acfb0 2024-03-29T09:55:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.693+0000 7f2247400700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f222806ac90 0x7f222806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.694+0000 7f2247400700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f222806ac90 0x7f222806d150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2238005fd0 tx=0x7f2238007040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:37.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.697+0000 7f223f400700 1 -- 172.21.15.12:0/2272180002 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2240059a80 con 0x7f22480acfb0 2024-03-29T09:55:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:37.961+0000 7f224e600700 1 -- 172.21.15.12:0/2272180002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2248067720 con 0x7f22480acfb0 2024-03-29T09:55:38.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.932+0000 7f223f400700 1 -- 172.21.15.12:0/2272180002 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7f224005d0a0 con 0x7f22480acfb0 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 -- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f222806ac90 msgr2=0x7f222806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f222806ac90 0x7f222806d150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2238005fd0 tx=0x7f2238007040 comp rx=0 tx=0).stop 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 -- 172.21.15.12:0/2272180002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22480acfb0 msgr2=0x7f22480ad460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22480acfb0 0x7f22480ad460 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2240019000 tx=0x7f2240009750 comp rx=0 tx=0).stop 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 -- 172.21.15.12:0/2272180002 shutdown_connections 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f222806ac90 0x7f222806d150 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22480acfb0 0x7f22480ad460 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f222c004d90 0x7f2248156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 --2- 172.21.15.12:0/2272180002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2248098580 0x7f2248098a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 -- 172.21.15.12:0/2272180002 >> 172.21.15.12:0/2272180002 conn(0x7f22480b0d70 msgr2=0x7f224806e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 -- 172.21.15.12:0/2272180002 shutdown_connections 2024-03-29T09:55:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:38.936+0000 7f224e600700 1 -- 172.21.15.12:0/2272180002 wait complete. 2024-03-29T09:55:38.956 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T09:55:38.957 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:55:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.158+0000 7f5840a00700 1 Processor -- start 2024-03-29T09:55:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.159+0000 7f5840a00700 1 -- start start 2024-03-29T09:55:39.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.160+0000 7f5840a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f583c0b3af0 0x7f583c15a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.160+0000 7f5840a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f583c0709c0 0x7f583c070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.160+0000 7f5840a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c0b2180 con 0x7f583c072580 2024-03-29T09:55:39.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.160+0000 7f5840a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583c15b100 con 0x7f583c0709c0 2024-03-29T09:55:39.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.160+0000 7f5840a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583c15b3e0 con 0x7f583c0b3af0 2024-03-29T09:55:39.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.160+0000 7f583a800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f583c0709c0 0x7f583c070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.160+0000 7f5839400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f583c072580 0x7f583c0b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56834/0 (socket says 172.21.15.12:56834) 2024-03-29T09:55:39.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.160+0000 7f5839400700 1 -- 172.21.15.12:0/2183656567 learned_addr learned my addr 172.21.15.12:0/2183656567 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:39.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.160+0000 7f5839e00700 1 --2- 172.21.15.12:0/2183656567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f583c0b3af0 0x7f583c15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.161+0000 7f583a800700 1 -- 172.21.15.12:0/2183656567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f583c0b3af0 msgr2=0x7f583c15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.161+0000 7f583a800700 1 --2- 172.21.15.12:0/2183656567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f583c0b3af0 0x7f583c15a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.161+0000 7f583a800700 1 -- 172.21.15.12:0/2183656567 >> v1:172.21.15.12:6789/0 conn(0x7f583c072580 legacy=0x7f583c0b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.161+0000 7f583a800700 1 -- 172.21.15.12:0/2183656567 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c0b23f0 con 0x7f583c0709c0 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.161+0000 7f583a800700 1 --2- 172.21.15.12:0/2183656567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0709c0 0x7f583c070e30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5834009d90 tx=0x7f583400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5027fd1ca216188e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.161+0000 7f5838a00700 1 -- 172.21.15.12:0/2183656567 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583400bd20 con 0x7f583c0709c0 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.161+0000 7f5838a00700 1 -- 172.21.15.12:0/2183656567 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583400be80 con 0x7f583c0709c0 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.161+0000 7f5838a00700 1 -- 172.21.15.12:0/2183656567 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58340116a0 con 0x7f583c0709c0 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.162+0000 7f5840a00700 1 -- 172.21.15.12:0/2183656567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0709c0 msgr2=0x7f583c070e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.162+0000 7f5840a00700 1 --2- 172.21.15.12:0/2183656567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0709c0 0x7f583c070e30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5834009d90 tx=0x7f583400b330 comp rx=0 tx=0).stop 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.162+0000 7f5840a00700 1 -- 172.21.15.12:0/2183656567 shutdown_connections 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.162+0000 7f5840a00700 1 --2- 172.21.15.12:0/2183656567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0709c0 0x7f583c070e30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.162+0000 7f5840a00700 1 --2- 172.21.15.12:0/2183656567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f583c0b3af0 0x7f583c15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.162+0000 7f5840a00700 1 -- 172.21.15.12:0/2183656567 >> 172.21.15.12:0/2183656567 conn(0x7f583c0cf190 msgr2=0x7f583c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.162+0000 7f5840a00700 1 -- 172.21.15.12:0/2183656567 shutdown_connections 2024-03-29T09:55:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.162+0000 7f5840a00700 1 -- 172.21.15.12:0/2183656567 wait complete. 2024-03-29T09:55:39.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.164+0000 7f5840a00700 1 Processor -- start 2024-03-29T09:55:39.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.164+0000 7f5840a00700 1 -- start start 2024-03-29T09:55:39.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f5840a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f583c0709c0 0x7f583c158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f5840a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f583c072580 0x7f583c142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f5840a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0b3af0 0x7f583c144780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f5840a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0cea30 con 0x7f583c0b3af0 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f5840a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0ce470 con 0x7f583c072580 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f5840a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0ce750 con 0x7f583c0709c0 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f583a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0b3af0 0x7f583c144780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f583a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0b3af0 0x7f583c144780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35446/0 (socket says 172.21.15.12:35446) 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f583a800700 1 -- 172.21.15.12:0/677062811 learned_addr learned my addr 172.21.15.12:0/677062811 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f5839e00700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f583c0709c0 0x7f583c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f5839400700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f583c072580 0x7f583c142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f583a800700 1 -- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f583c0709c0 msgr2=0x7f583c158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f583a800700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f583c0709c0 0x7f583c158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f583a800700 1 -- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f583c072580 msgr2=0x7f583c142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f583a800700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f583c072580 0x7f583c142730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f583a800700 1 -- 172.21.15.12:0/677062811 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5834009a70 con 0x7f583c0b3af0 2024-03-29T09:55:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.165+0000 7f583a800700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0b3af0 0x7f583c144780 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f583c072a30 tx=0x7f5834004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:39.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.166+0000 7f5832000700 1 -- 172.21.15.12:0/677062811 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834003ea0 con 0x7f583c0b3af0 2024-03-29T09:55:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.166+0000 7f5832000700 1 -- 172.21.15.12:0/677062811 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5834004960 con 0x7f583c0b3af0 2024-03-29T09:55:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.166+0000 7f5832000700 1 -- 172.21.15.12:0/677062811 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58340116a0 con 0x7f583c0b3af0 2024-03-29T09:55:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.166+0000 7f5840a00700 1 -- 172.21.15.12:0/677062811 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c158fa0 con 0x7f583c0b3af0 2024-03-29T09:55:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.166+0000 7f5840a00700 1 -- 172.21.15.12:0/677062811 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c14f960 con 0x7f583c0b3af0 2024-03-29T09:55:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.167+0000 7f5840a00700 1 -- 172.21.15.12:0/677062811 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c1594c0 con 0x7f583c0b3af0 2024-03-29T09:55:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.168+0000 7f5832000700 1 -- 172.21.15.12:0/677062811 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5834011800 con 0x7f583c0b3af0 2024-03-29T09:55:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.169+0000 7f5832000700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f581c06ac90 0x7f581c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.169+0000 7f5832000700 1 -- 172.21.15.12:0/677062811 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f583408b220 con 0x7f583c0b3af0 2024-03-29T09:55:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.169+0000 7f5839e00700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f581c06ac90 0x7f581c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.170+0000 7f5839e00700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f581c06ac90 0x7f581c06d150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f582c009b70 tx=0x7f582c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.174+0000 7f5832000700 1 -- 172.21.15.12:0/677062811 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f583c1594c0 con 0x7f583c0b3af0 2024-03-29T09:55:39.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.438+0000 7f5840a00700 1 -- 172.21.15.12:0/677062811 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f583c1594c0 con 0x7f583c0b3af0 2024-03-29T09:55:39.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.438+0000 7f5832000700 1 -- 172.21.15.12:0/677062811 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f583c1594c0 con 0x7f583c0b3af0 2024-03-29T09:55:39.438 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:39.438 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2845880355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2845880355},{"type":"v1","addr":"172.21.15.159:6837","nonce":2845880355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2836227468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2836227468},{"type":"v1","addr":"172.21.15.159:6836","nonce":2836227468}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1863550973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1863550973},{"type":"v1","addr":"172.21.15.12:6834","nonce":1863550973}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:55:35.438233+0000","modified":"2024-03-29T09:55:38.926199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/2013939805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2013939805},{"type":"v1","addr":"172.21.15.12:6835","nonce":2013939805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-29T09:55:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 -- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f581c06ac90 msgr2=0x7f581c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f581c06ac90 0x7f581c06d150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f582c009b70 tx=0x7f582c008040 comp rx=0 tx=0).stop 2024-03-29T09:55:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 -- 172.21.15.12:0/677062811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0b3af0 msgr2=0x7f583c144780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0b3af0 0x7f583c144780 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f583c072a30 tx=0x7f5834004b40 comp rx=0 tx=0).stop 2024-03-29T09:55:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 -- 172.21.15.12:0/677062811 shutdown_connections 2024-03-29T09:55:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f581c06ac90 0x7f581c06d150 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f583c0b3af0 0x7f583c144780 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f583c0709c0 0x7f583c158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 --2- 172.21.15.12:0/677062811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f583c072580 0x7f583c142730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 -- 172.21.15.12:0/677062811 >> 172.21.15.12:0/677062811 conn(0x7f583c0cf190 msgr2=0x7f583c150110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 -- 172.21.15.12:0/677062811 shutdown_connections 2024-03-29T09:55:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.441+0000 7f5840a00700 1 -- 172.21.15.12:0/677062811 wait complete. 2024-03-29T09:55:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-29T09:55:39.459 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T09:55:39.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.657+0000 7f8037400700 1 Processor -- start 2024-03-29T09:55:39.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.657+0000 7f8037400700 1 -- start start 2024-03-29T09:55:39.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.658+0000 7f8037400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8030160f10 0x7f80300d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.658+0000 7f8037400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8030161590 0x7f8030161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.658+0000 7f8037400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80300ceda0 con 0x7f80300cf230 2024-03-29T09:55:39.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.658+0000 7f8037400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8030161f80 con 0x7f8030160f10 2024-03-29T09:55:39.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.658+0000 7f8037400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80301612f0 con 0x7f8030161590 2024-03-29T09:55:39.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.658+0000 7f8034a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8030160f10 0x7f80300d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.658+0000 7f802fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f80300cf230 0x7f80300d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56846/0 (socket says 172.21.15.12:56846) 2024-03-29T09:55:39.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.658+0000 7f802fe00700 1 -- 172.21.15.12:0/143989582 learned_addr learned my addr 172.21.15.12:0/143989582 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:39.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.658+0000 7f8035400700 1 --2- 172.21.15.12:0/143989582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8030161590 0x7f8030161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.659+0000 7f8034a00700 1 -- 172.21.15.12:0/143989582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8030161590 msgr2=0x7f8030161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.659+0000 7f8034a00700 1 --2- 172.21.15.12:0/143989582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8030161590 0x7f8030161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.659+0000 7f8034a00700 1 -- 172.21.15.12:0/143989582 >> v1:172.21.15.12:6789/0 conn(0x7f80300cf230 legacy=0x7f80300d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.659+0000 7f8034a00700 1 -- 172.21.15.12:0/143989582 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80300ce470 con 0x7f8030160f10 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.659+0000 7f8034a00700 1 --2- 172.21.15.12:0/143989582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 0x7f80300d18b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8024009840 tx=0x7f8024009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5949394047f8ee server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.659+0000 7f802f400700 1 -- 172.21.15.12:0/143989582 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8024011830 con 0x7f8030160f10 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.659+0000 7f802f400700 1 -- 172.21.15.12:0/143989582 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8024011990 con 0x7f8030160f10 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.659+0000 7f802f400700 1 -- 172.21.15.12:0/143989582 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8024011d20 con 0x7f8030160f10 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.660+0000 7f8037400700 1 -- 172.21.15.12:0/143989582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 msgr2=0x7f80300d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.660+0000 7f8037400700 1 --2- 172.21.15.12:0/143989582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 0x7f80300d18b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8024009840 tx=0x7f8024009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.660+0000 7f8037400700 1 -- 172.21.15.12:0/143989582 shutdown_connections 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.660+0000 7f8037400700 1 --2- 172.21.15.12:0/143989582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8030161590 0x7f8030161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.660+0000 7f8037400700 1 --2- 172.21.15.12:0/143989582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 0x7f80300d18b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.660+0000 7f8037400700 1 -- 172.21.15.12:0/143989582 >> 172.21.15.12:0/143989582 conn(0x7f803008aff0 msgr2=0x7f80300d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:39.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.660+0000 7f8037400700 1 -- 172.21.15.12:0/143989582 shutdown_connections 2024-03-29T09:55:39.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.660+0000 7f8037400700 1 -- 172.21.15.12:0/143989582 wait complete. 2024-03-29T09:55:39.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.662+0000 7f8037400700 1 Processor -- start 2024-03-29T09:55:39.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.662+0000 7f8037400700 1 -- start start 2024-03-29T09:55:39.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f8037400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80300cf230 0x7f8030155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f8037400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 0x7f8030156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f8037400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8030161590 0x7f8030158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f8037400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8030157dd0 con 0x7f8030160f10 2024-03-29T09:55:39.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f8037400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8030157150 con 0x7f8030161590 2024-03-29T09:55:39.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f8037400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8030157430 con 0x7f80300cf230 2024-03-29T09:55:39.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f802fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 0x7f8030156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f802fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 0x7f8030156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35466/0 (socket says 172.21.15.12:35466) 2024-03-29T09:55:39.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f802fe00700 1 -- 172.21.15.12:0/3745711504 learned_addr learned my addr 172.21.15.12:0/3745711504 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:39.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f8034a00700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80300cf230 0x7f8030155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.663+0000 7f8035400700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8030161590 0x7f8030158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f802fe00700 1 -- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80300cf230 msgr2=0x7f8030155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f802fe00700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80300cf230 0x7f8030155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f802fe00700 1 -- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8030161590 msgr2=0x7f8030158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f802fe00700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8030161590 0x7f8030158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f802fe00700 1 -- 172.21.15.12:0/3745711504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80240094a0 con 0x7f8030160f10 2024-03-29T09:55:39.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f8034a00700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80300cf230 0x7f8030155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:39.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f8035400700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8030161590 0x7f8030158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:39.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f802fe00700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 0x7f8030156650 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c450 tx=0x7f801c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:39.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f802cc00700 1 -- 172.21.15.12:0/3745711504 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c005d70 con 0x7f8030160f10 2024-03-29T09:55:39.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f802cc00700 1 -- 172.21.15.12:0/3745711504 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801c0070d0 con 0x7f8030160f10 2024-03-29T09:55:39.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f8037400700 1 -- 172.21.15.12:0/3745711504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8030157710 con 0x7f8030160f10 2024-03-29T09:55:39.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f802cc00700 1 -- 172.21.15.12:0/3745711504 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c007440 con 0x7f8030160f10 2024-03-29T09:55:39.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.664+0000 7f8037400700 1 -- 172.21.15.12:0/3745711504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803015fcf0 con 0x7f8030160f10 2024-03-29T09:55:39.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.665+0000 7f8037400700 1 -- 172.21.15.12:0/3745711504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8030067720 con 0x7f8030160f10 2024-03-29T09:55:39.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.666+0000 7f802cc00700 1 -- 172.21.15.12:0/3745711504 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f801c0075a0 con 0x7f8030160f10 2024-03-29T09:55:39.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.667+0000 7f802cc00700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f801006ac90 0x7f801006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:39.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.668+0000 7f802cc00700 1 -- 172.21.15.12:0/3745711504 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f801c0886b0 con 0x7f8030160f10 2024-03-29T09:55:39.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.668+0000 7f8034a00700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f801006ac90 0x7f801006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:39.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.669+0000 7f8034a00700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f801006ac90 0x7f801006d150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f80300d4600 tx=0x7f8024008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.672+0000 7f802cc00700 1 -- 172.21.15.12:0/3745711504 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f801c0538a0 con 0x7f8030160f10 2024-03-29T09:55:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.890+0000 7f8037400700 1 -- 172.21.15.12:0/3745711504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8030155fd0 con 0x7f8030160f10 2024-03-29T09:55:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.892+0000 7f802cc00700 1 -- 172.21.15.12:0/3745711504 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16101 (secure 0 0 0) 0x7f801c056ec0 con 0x7f8030160f10 2024-03-29T09:55:39.891 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:39.892 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":34,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T09:55:36.440988+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:55:33.508935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T09:55:34.418367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T09:55:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 -- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f801006ac90 msgr2=0x7f801006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f801006ac90 0x7f801006d150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f80300d4600 tx=0x7f8024008030 comp rx=0 tx=0).stop 2024-03-29T09:55:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 -- 172.21.15.12:0/3745711504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 msgr2=0x7f8030156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 0x7f8030156650 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c450 tx=0x7f801c00c810 comp rx=0 tx=0).stop 2024-03-29T09:55:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 -- 172.21.15.12:0/3745711504 shutdown_connections 2024-03-29T09:55:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f801006ac90 0x7f801006d150 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8030161590 0x7f8030158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80300cf230 0x7f8030155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 --2- 172.21.15.12:0/3745711504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8030160f10 0x7f8030156650 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 -- 172.21.15.12:0/3745711504 >> 172.21.15.12:0/3745711504 conn(0x7f803008aff0 msgr2=0x7f80300ca9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 -- 172.21.15.12:0/3745711504 shutdown_connections 2024-03-29T09:55:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:39.894+0000 7f8037400700 1 -- 172.21.15.12:0/3745711504 wait complete. 2024-03-29T09:55:39.912 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T09:55:39.912 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T09:55:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.110+0000 7f91d7200700 1 Processor -- start 2024-03-29T09:55:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.111+0000 7f91d7200700 1 -- start start 2024-03-29T09:55:40.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.112+0000 7f91d7200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91d0160f10 0x7f91d00d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.112+0000 7f91d7200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f91d00cf230 0x7f91d00d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.112+0000 7f91d7200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d00ceda0 con 0x7f91d0161590 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.112+0000 7f91d7200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d00d23d0 con 0x7f91d0160f10 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.112+0000 7f91d7200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d01612f0 con 0x7f91d00cf230 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.112+0000 7f91cfe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91d0160f10 0x7f91d00d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.112+0000 7f91d5200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f91d0161590 0x7f91d00b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56854/0 (socket says 172.21.15.12:56854) 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.112+0000 7f91cfe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91d0160f10 0x7f91d00d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35468/0 (socket says 172.21.15.12:35468) 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.112+0000 7f91d5200700 1 -- 172.21.15.12:0/2564409033 learned_addr learned my addr 172.21.15.12:0/2564409033 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.112+0000 7f91cf400700 1 --2- 172.21.15.12:0/2564409033 >> v2:172.21.15.159:3300/0 conn(0x7f91d00cf230 0x7f91d00d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.113+0000 7f91cfe00700 1 -- 172.21.15.12:0/2564409033 >> v2:172.21.15.159:3300/0 conn(0x7f91d00cf230 msgr2=0x7f91d00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.113+0000 7f91cfe00700 1 --2- 172.21.15.12:0/2564409033 >> v2:172.21.15.159:3300/0 conn(0x7f91d00cf230 0x7f91d00d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.113+0000 7f91cfe00700 1 -- 172.21.15.12:0/2564409033 >> v1:172.21.15.12:6789/0 conn(0x7f91d0161590 legacy=0x7f91d00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.113+0000 7f91cfe00700 1 -- 172.21.15.12:0/2564409033 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d00ce470 con 0x7f91d0160f10 2024-03-29T09:55:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.113+0000 7f91cea00700 1 -- 172.21.15.12:0/2564409033 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893035580 0 0) 0x7f91d00ceda0 con 0x7f91d0161590 2024-03-29T09:55:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.113+0000 7f91cfe00700 1 --2- 172.21.15.12:0/2564409033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d0160f10 0x7f91d00d18b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f91d008cdb0 tx=0x7f91c0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea09a5af9696756e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.113+0000 7f91cea00700 1 -- 172.21.15.12:0/2564409033 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c0011830 con 0x7f91d0160f10 2024-03-29T09:55:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.113+0000 7f91cea00700 1 -- 172.21.15.12:0/2564409033 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c0011990 con 0x7f91d0160f10 2024-03-29T09:55:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.113+0000 7f91cea00700 1 -- 172.21.15.12:0/2564409033 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c0011d20 con 0x7f91d0160f10 2024-03-29T09:55:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.114+0000 7f91d7200700 1 -- 172.21.15.12:0/2564409033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d0160f10 msgr2=0x7f91d00d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.114+0000 7f91d7200700 1 --2- 172.21.15.12:0/2564409033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d0160f10 0x7f91d00d18b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f91d008cdb0 tx=0x7f91c0009b20 comp rx=0 tx=0).stop 2024-03-29T09:55:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.114+0000 7f91d7200700 1 -- 172.21.15.12:0/2564409033 shutdown_connections 2024-03-29T09:55:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.114+0000 7f91d7200700 1 --2- 172.21.15.12:0/2564409033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d0160f10 0x7f91d00d18b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.114+0000 7f91d7200700 1 --2- 172.21.15.12:0/2564409033 >> v2:172.21.15.159:3300/0 conn(0x7f91d00cf230 0x7f91d00d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.114+0000 7f91d7200700 1 -- 172.21.15.12:0/2564409033 >> 172.21.15.12:0/2564409033 conn(0x7f91d008aff0 msgr2=0x7f91d00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.114+0000 7f91d7200700 1 -- 172.21.15.12:0/2564409033 shutdown_connections 2024-03-29T09:55:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.114+0000 7f91d7200700 1 -- 172.21.15.12:0/2564409033 wait complete. 2024-03-29T09:55:40.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.116+0000 7f91d7200700 1 Processor -- start 2024-03-29T09:55:40.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.116+0000 7f91d7200700 1 -- start start 2024-03-29T09:55:40.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91d7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d00cf230 0x7f91d0147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91d7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91d0160f10 0x7f91d0159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91d7200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d0161590 0x7f91d0156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91d7200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d0147d80 con 0x7f91d00cf230 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91d7200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00b7930 con 0x7f91d0161590 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91d7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00b7c10 con 0x7f91d0160f10 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91cfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d00cf230 0x7f91d0147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91cfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d00cf230 0x7f91d0147840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35474/0 (socket says 172.21.15.12:35474) 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91cfe00700 1 -- 172.21.15.12:0/661768686 learned_addr learned my addr 172.21.15.12:0/661768686 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91cf400700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91d0160f10 0x7f91d0159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91d5200700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d0161590 0x7f91d0156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91cfe00700 1 -- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91d0160f10 msgr2=0x7f91d0159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91cfe00700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91d0160f10 0x7f91d0159420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91cfe00700 1 -- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d0161590 msgr2=0x7f91d0156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91cfe00700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d0161590 0x7f91d0156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91cfe00700 1 -- 172.21.15.12:0/661768686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c00094a0 con 0x7f91d00cf230 2024-03-29T09:55:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.117+0000 7f91d5200700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d0161590 0x7f91d0156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.118+0000 7f91cfe00700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d00cf230 0x7f91d0147840 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f91c0009b60 tx=0x7f91c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.118+0000 7f91c7e00700 1 -- 172.21.15.12:0/661768686 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c00038a0 con 0x7f91d00cf230 2024-03-29T09:55:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.118+0000 7f91c7e00700 1 -- 172.21.15.12:0/661768686 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c001e070 con 0x7f91d00cf230 2024-03-29T09:55:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.118+0000 7f91c7e00700 1 -- 172.21.15.12:0/661768686 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c0003c00 con 0x7f91d00cf230 2024-03-29T09:55:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.118+0000 7f91d7200700 1 -- 172.21.15.12:0/661768686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d00b7ef0 con 0x7f91d00cf230 2024-03-29T09:55:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.118+0000 7f91d7200700 1 -- 172.21.15.12:0/661768686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d00b74a0 con 0x7f91d00cf230 2024-03-29T09:55:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.119+0000 7f91d7200700 1 -- 172.21.15.12:0/661768686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d00b8540 con 0x7f91d00cf230 2024-03-29T09:55:40.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.120+0000 7f91c7e00700 1 -- 172.21.15.12:0/661768686 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f91c002d450 con 0x7f91d00cf230 2024-03-29T09:55:40.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.121+0000 7f91c7e00700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91b006ac90 0x7f91b006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.121+0000 7f91c7e00700 1 -- 172.21.15.12:0/661768686 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91c008a420 con 0x7f91d00cf230 2024-03-29T09:55:40.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.121+0000 7f91cf400700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91b006ac90 0x7f91b006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.122+0000 7f91cf400700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91b006ac90 0x7f91b006d150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f91d0153270 tx=0x7f91bc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:40.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.125+0000 7f91c7e00700 1 -- 172.21.15.12:0/661768686 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91c00555f0 con 0x7f91d00cf230 2024-03-29T09:55:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.335+0000 7f91d7200700 1 -- 172.21.15.12:0/661768686 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f91d00cf6b0 con 0x7f91b006ac90 2024-03-29T09:55:40.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.345+0000 7f91c7e00700 1 -- 172.21.15.12:0/661768686 <== mgr.4114 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288921 (secure 0 0 0) 0x7f91bc00b5c0 con 0x7f91b006ac90 2024-03-29T09:55:40.348 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 -- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91b006ac90 msgr2=0x7f91b006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91b006ac90 0x7f91b006d150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f91d0153270 tx=0x7f91bc008040 comp rx=0 tx=0).stop 2024-03-29T09:55:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 -- 172.21.15.12:0/661768686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d00cf230 msgr2=0x7f91d0147840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d00cf230 0x7f91d0147840 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f91c0009b60 tx=0x7f91c0004750 comp rx=0 tx=0).stop 2024-03-29T09:55:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 -- 172.21.15.12:0/661768686 shutdown_connections 2024-03-29T09:55:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91b006ac90 0x7f91b006d150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d0161590 0x7f91d0156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d00cf230 0x7f91d0147840 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 --2- 172.21.15.12:0/661768686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91d0160f10 0x7f91d0159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 -- 172.21.15.12:0/661768686 >> 172.21.15.12:0/661768686 conn(0x7f91d008aff0 msgr2=0x7f91d00b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 -- 172.21.15.12:0/661768686 shutdown_connections 2024-03-29T09:55:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.349+0000 7f91d7200700 1 -- 172.21.15.12:0/661768686 wait complete. 2024-03-29T09:55:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T09:55:40.367 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-03-29T09:55:38.454010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220320,"kb_used_data":4132,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754400,"statfs":{"total":773094113280,"available":772868505600,"internally_reserved":0,"allocated":4231168,"data_stored":2350497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.538369"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452541+0000","last_change":"2024-03-29T09:55:35.452157+0000","last_active":"2024-03-29T09:55:36.452541+0000","last_peered":"2024-03-29T09:55:36.452541+0000","last_clean":"2024-03-29T09:55:36.452541+0000","last_became_active":"2024-03-29T09:55:35.451841+0000","last_became_peered":"2024-03-29T09:55:35.451841+0000","last_unstale":"2024-03-29T09:55:36.452541+0000","last_undegraded":"2024-03-29T09:55:36.452541+0000","last_fullsized":"2024-03-29T09:55:36.452541+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:02.691272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478215+0000","last_change":"2024-03-29T09:55:35.470292+0000","last_active":"2024-03-29T09:55:35.478215+0000","last_peered":"2024-03-29T09:55:35.478215+0000","last_clean":"2024-03-29T09:55:35.478215+0000","last_became_active":"2024-03-29T09:55:35.470036+0000","last_became_peered":"2024-03-29T09:55:35.470036+0000","last_unstale":"2024-03-29T09:55:35.478215+0000","last_undegraded":"2024-03-29T09:55:35.478215+0000","last_fullsized":"2024-03-29T09:55:35.478215+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:54.134659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475705+0000","last_change":"2024-03-29T09:55:35.466012+0000","last_active":"2024-03-29T09:55:35.475705+0000","last_peered":"2024-03-29T09:55:35.475705+0000","last_clean":"2024-03-29T09:55:35.475705+0000","last_became_active":"2024-03-29T09:55:35.464991+0000","last_became_peered":"2024-03-29T09:55:35.464991+0000","last_unstale":"2024-03-29T09:55:35.475705+0000","last_undegraded":"2024-03-29T09:55:35.475705+0000","last_fullsized":"2024-03-29T09:55:35.475705+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:54.134659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.479827+0000","last_change":"2024-03-29T09:55:35.468730+0000","last_active":"2024-03-29T09:55:35.479827+0000","last_peered":"2024-03-29T09:55:35.479827+0000","last_clean":"2024-03-29T09:55:35.479827+0000","last_became_active":"2024-03-29T09:55:35.468286+0000","last_became_peered":"2024-03-29T09:55:35.468286+0000","last_unstale":"2024-03-29T09:55:35.479827+0000","last_undegraded":"2024-03-29T09:55:35.479827+0000","last_fullsized":"2024-03-29T09:55:35.479827+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:28:49.878736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453619+0000","last_change":"2024-03-29T09:55:35.474980+0000","last_active":"2024-03-29T09:55:36.453619+0000","last_peered":"2024-03-29T09:55:36.453619+0000","last_clean":"2024-03-29T09:55:36.453619+0000","last_became_active":"2024-03-29T09:55:35.473492+0000","last_became_peered":"2024-03-29T09:55:35.473492+0000","last_unstale":"2024-03-29T09:55:36.453619+0000","last_undegraded":"2024-03-29T09:55:36.453619+0000","last_fullsized":"2024-03-29T09:55:36.453619+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:49:22.165389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.476372+0000","last_change":"2024-03-29T09:55:35.462563+0000","last_active":"2024-03-29T09:55:35.476372+0000","last_peered":"2024-03-29T09:55:35.476372+0000","last_clean":"2024-03-29T09:55:35.476372+0000","last_became_active":"2024-03-29T09:55:35.462256+0000","last_became_peered":"2024-03-29T09:55:35.462256+0000","last_unstale":"2024-03-29T09:55:35.476372+0000","last_undegraded":"2024-03-29T09:55:35.476372+0000","last_fullsized":"2024-03-29T09:55:35.476372+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:40:09.967878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452712+0000","last_change":"2024-03-29T09:55:35.472251+0000","last_active":"2024-03-29T09:55:36.452712+0000","last_peered":"2024-03-29T09:55:36.452712+0000","last_clean":"2024-03-29T09:55:36.452712+0000","last_became_active":"2024-03-29T09:55:35.470513+0000","last_became_peered":"2024-03-29T09:55:35.470513+0000","last_unstale":"2024-03-29T09:55:36.452712+0000","last_undegraded":"2024-03-29T09:55:36.452712+0000","last_fullsized":"2024-03-29T09:55:36.452712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:09:14.949848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.451863+0000","last_change":"2024-03-29T09:55:35.463829+0000","last_active":"2024-03-29T09:55:36.451863+0000","last_peered":"2024-03-29T09:55:36.451863+0000","last_clean":"2024-03-29T09:55:36.451863+0000","last_became_active":"2024-03-29T09:55:35.463562+0000","last_became_peered":"2024-03-29T09:55:35.463562+0000","last_unstale":"2024-03-29T09:55:36.451863+0000","last_undegraded":"2024-03-29T09:55:36.451863+0000","last_fullsized":"2024-03-29T09:55:36.451863+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:14:54.261455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.474921+0000","last_change":"2024-03-29T09:55:35.469975+0000","last_active":"2024-03-29T09:55:35.474921+0000","last_peered":"2024-03-29T09:55:35.474921+0000","last_clean":"2024-03-29T09:55:35.474921+0000","last_became_active":"2024-03-29T09:55:35.468396+0000","last_became_peered":"2024-03-29T09:55:35.468396+0000","last_unstale":"2024-03-29T09:55:35.474921+0000","last_undegraded":"2024-03-29T09:55:35.474921+0000","last_fullsized":"2024-03-29T09:55:35.474921+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:36.716035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477516+0000","last_change":"2024-03-29T09:55:35.470609+0000","last_active":"2024-03-29T09:55:35.477516+0000","last_peered":"2024-03-29T09:55:35.477516+0000","last_clean":"2024-03-29T09:55:35.477516+0000","last_became_active":"2024-03-29T09:55:35.470180+0000","last_became_peered":"2024-03-29T09:55:35.470180+0000","last_unstale":"2024-03-29T09:55:35.477516+0000","last_undegraded":"2024-03-29T09:55:35.477516+0000","last_fullsized":"2024-03-29T09:55:35.477516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:06.711276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452914+0000","last_change":"2024-03-29T09:55:35.469739+0000","last_active":"2024-03-29T09:55:36.452914+0000","last_peered":"2024-03-29T09:55:36.452914+0000","last_clean":"2024-03-29T09:55:36.452914+0000","last_became_active":"2024-03-29T09:55:35.469242+0000","last_became_peered":"2024-03-29T09:55:35.469242+0000","last_unstale":"2024-03-29T09:55:36.452914+0000","last_undegraded":"2024-03-29T09:55:36.452914+0000","last_fullsized":"2024-03-29T09:55:36.452914+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:27:17.579514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477982+0000","last_change":"2024-03-29T09:55:35.472836+0000","last_active":"2024-03-29T09:55:35.477982+0000","last_peered":"2024-03-29T09:55:35.477982+0000","last_clean":"2024-03-29T09:55:35.477982+0000","last_became_active":"2024-03-29T09:55:35.471826+0000","last_became_peered":"2024-03-29T09:55:35.471826+0000","last_unstale":"2024-03-29T09:55:35.477982+0000","last_undegraded":"2024-03-29T09:55:35.477982+0000","last_fullsized":"2024-03-29T09:55:35.477982+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:47:12.317720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453636+0000","last_change":"2024-03-29T09:55:35.473546+0000","last_active":"2024-03-29T09:55:36.453636+0000","last_peered":"2024-03-29T09:55:36.453636+0000","last_clean":"2024-03-29T09:55:36.453636+0000","last_became_active":"2024-03-29T09:55:35.473219+0000","last_became_peered":"2024-03-29T09:55:35.473219+0000","last_unstale":"2024-03-29T09:55:36.453636+0000","last_undegraded":"2024-03-29T09:55:36.453636+0000","last_fullsized":"2024-03-29T09:55:36.453636+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:02.691272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477777+0000","last_change":"2024-03-29T09:55:35.469414+0000","last_active":"2024-03-29T09:55:35.477777+0000","last_peered":"2024-03-29T09:55:35.477777+0000","last_clean":"2024-03-29T09:55:35.477777+0000","last_became_active":"2024-03-29T09:55:35.469161+0000","last_became_peered":"2024-03-29T09:55:35.469161+0000","last_unstale":"2024-03-29T09:55:35.477777+0000","last_undegraded":"2024-03-29T09:55:35.477777+0000","last_fullsized":"2024-03-29T09:55:35.477777+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:06:19.955544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.456273+0000","last_change":"2024-03-29T09:55:35.453942+0000","last_active":"2024-03-29T09:55:35.456273+0000","last_peered":"2024-03-29T09:55:35.456273+0000","last_clean":"2024-03-29T09:55:35.456273+0000","last_became_active":"2024-03-29T09:55:35.453580+0000","last_became_peered":"2024-03-29T09:55:35.453580+0000","last_unstale":"2024-03-29T09:55:35.456273+0000","last_undegraded":"2024-03-29T09:55:35.456273+0000","last_fullsized":"2024-03-29T09:55:35.456273+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:09:14.949848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452083+0000","last_change":"2024-03-29T09:55:35.473126+0000","last_active":"2024-03-29T09:55:36.452083+0000","last_peered":"2024-03-29T09:55:36.452083+0000","last_clean":"2024-03-29T09:55:36.452083+0000","last_became_active":"2024-03-29T09:55:35.472592+0000","last_became_peered":"2024-03-29T09:55:35.472592+0000","last_unstale":"2024-03-29T09:55:36.452083+0000","last_undegraded":"2024-03-29T09:55:36.452083+0000","last_fullsized":"2024-03-29T09:55:36.452083+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:39.961948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453075+0000","last_change":"2024-03-29T09:55:35.462151+0000","last_active":"2024-03-29T09:55:36.453075+0000","last_peered":"2024-03-29T09:55:36.453075+0000","last_clean":"2024-03-29T09:55:36.453075+0000","last_became_active":"2024-03-29T09:55:35.461868+0000","last_became_peered":"2024-03-29T09:55:35.461868+0000","last_unstale":"2024-03-29T09:55:36.453075+0000","last_undegraded":"2024-03-29T09:55:36.453075+0000","last_fullsized":"2024-03-29T09:55:36.453075+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:20:12.232627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478198+0000","last_change":"2024-03-29T09:55:35.470126+0000","last_active":"2024-03-29T09:55:35.478198+0000","last_peered":"2024-03-29T09:55:35.478198+0000","last_clean":"2024-03-29T09:55:35.478198+0000","last_became_active":"2024-03-29T09:55:35.469642+0000","last_became_peered":"2024-03-29T09:55:35.469642+0000","last_unstale":"2024-03-29T09:55:35.478198+0000","last_undegraded":"2024-03-29T09:55:35.478198+0000","last_fullsized":"2024-03-29T09:55:35.478198+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:42.191643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478715+0000","last_change":"2024-03-29T09:55:35.471953+0000","last_active":"2024-03-29T09:55:35.478715+0000","last_peered":"2024-03-29T09:55:35.478715+0000","last_clean":"2024-03-29T09:55:35.478715+0000","last_became_active":"2024-03-29T09:55:35.471653+0000","last_became_peered":"2024-03-29T09:55:35.471653+0000","last_unstale":"2024-03-29T09:55:35.478715+0000","last_undegraded":"2024-03-29T09:55:35.478715+0000","last_fullsized":"2024-03-29T09:55:35.478715+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:47:12.317720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453384+0000","last_change":"2024-03-29T09:55:35.463578+0000","last_active":"2024-03-29T09:55:36.453384+0000","last_peered":"2024-03-29T09:55:36.453384+0000","last_clean":"2024-03-29T09:55:36.453384+0000","last_became_active":"2024-03-29T09:55:35.463359+0000","last_became_peered":"2024-03-29T09:55:35.463359+0000","last_unstale":"2024-03-29T09:55:36.453384+0000","last_undegraded":"2024-03-29T09:55:36.453384+0000","last_fullsized":"2024-03-29T09:55:36.453384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:27.138795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.467986+0000","last_change":"2024-03-29T09:55:35.460186+0000","last_active":"2024-03-29T09:55:35.467986+0000","last_peered":"2024-03-29T09:55:35.467986+0000","last_clean":"2024-03-29T09:55:35.467986+0000","last_became_active":"2024-03-29T09:55:35.459966+0000","last_became_peered":"2024-03-29T09:55:35.459966+0000","last_unstale":"2024-03-29T09:55:35.467986+0000","last_undegraded":"2024-03-29T09:55:35.467986+0000","last_fullsized":"2024-03-29T09:55:35.467986+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:36.964655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477717+0000","last_change":"2024-03-29T09:55:35.471369+0000","last_active":"2024-03-29T09:55:35.477717+0000","last_peered":"2024-03-29T09:55:35.477717+0000","last_clean":"2024-03-29T09:55:35.477717+0000","last_became_active":"2024-03-29T09:55:35.471016+0000","last_became_peered":"2024-03-29T09:55:35.471016+0000","last_unstale":"2024-03-29T09:55:35.477717+0000","last_undegraded":"2024-03-29T09:55:35.477717+0000","last_fullsized":"2024-03-29T09:55:35.477717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:08:40.125637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.461067+0000","last_change":"2024-03-29T09:55:35.455198+0000","last_active":"2024-03-29T09:55:35.461067+0000","last_peered":"2024-03-29T09:55:35.461067+0000","last_clean":"2024-03-29T09:55:35.461067+0000","last_became_active":"2024-03-29T09:55:35.454902+0000","last_became_peered":"2024-03-29T09:55:35.454902+0000","last_unstale":"2024-03-29T09:55:35.461067+0000","last_undegraded":"2024-03-29T09:55:35.461067+0000","last_fullsized":"2024-03-29T09:55:35.461067+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:06.711276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.468526+0000","last_change":"2024-03-29T09:55:35.463768+0000","last_active":"2024-03-29T09:55:35.468526+0000","last_peered":"2024-03-29T09:55:35.468526+0000","last_clean":"2024-03-29T09:55:35.468526+0000","last_became_active":"2024-03-29T09:55:35.461341+0000","last_became_peered":"2024-03-29T09:55:35.461341+0000","last_unstale":"2024-03-29T09:55:35.468526+0000","last_undegraded":"2024-03-29T09:55:35.468526+0000","last_fullsized":"2024-03-29T09:55:35.468526+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:59.893866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.474726+0000","last_change":"2024-03-29T09:55:35.467427+0000","last_active":"2024-03-29T09:55:35.474726+0000","last_peered":"2024-03-29T09:55:35.474726+0000","last_clean":"2024-03-29T09:55:35.474726+0000","last_became_active":"2024-03-29T09:55:35.466955+0000","last_became_peered":"2024-03-29T09:55:35.466955+0000","last_unstale":"2024-03-29T09:55:35.474726+0000","last_undegraded":"2024-03-29T09:55:35.474726+0000","last_fullsized":"2024-03-29T09:55:35.474726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:33.902566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478178+0000","last_change":"2024-03-29T09:55:35.469202+0000","last_active":"2024-03-29T09:55:35.478178+0000","last_peered":"2024-03-29T09:55:35.478178+0000","last_clean":"2024-03-29T09:55:35.478178+0000","last_became_active":"2024-03-29T09:55:35.463997+0000","last_became_peered":"2024-03-29T09:55:35.463997+0000","last_unstale":"2024-03-29T09:55:35.478178+0000","last_undegraded":"2024-03-29T09:55:35.478178+0000","last_fullsized":"2024-03-29T09:55:35.478178+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:00:59.504082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.480649+0000","last_change":"2024-03-29T09:55:35.477723+0000","last_active":"2024-03-29T09:55:35.480649+0000","last_peered":"2024-03-29T09:55:35.480649+0000","last_clean":"2024-03-29T09:55:35.480649+0000","last_became_active":"2024-03-29T09:55:35.476438+0000","last_became_peered":"2024-03-29T09:55:35.476438+0000","last_unstale":"2024-03-29T09:55:35.480649+0000","last_undegraded":"2024-03-29T09:55:35.480649+0000","last_fullsized":"2024-03-29T09:55:35.480649+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:36.716035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.469164+0000","last_change":"2024-03-29T09:55:35.459841+0000","last_active":"2024-03-29T09:55:35.469164+0000","last_peered":"2024-03-29T09:55:35.469164+0000","last_clean":"2024-03-29T09:55:35.469164+0000","last_became_active":"2024-03-29T09:55:35.459543+0000","last_became_peered":"2024-03-29T09:55:35.459543+0000","last_unstale":"2024-03-29T09:55:35.469164+0000","last_undegraded":"2024-03-29T09:55:35.469164+0000","last_fullsized":"2024-03-29T09:55:35.469164+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:39.961948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477846+0000","last_change":"2024-03-29T09:55:35.473320+0000","last_active":"2024-03-29T09:55:35.477846+0000","last_peered":"2024-03-29T09:55:35.477846+0000","last_clean":"2024-03-29T09:55:35.477846+0000","last_became_active":"2024-03-29T09:55:35.472941+0000","last_became_peered":"2024-03-29T09:55:35.472941+0000","last_unstale":"2024-03-29T09:55:35.477846+0000","last_undegraded":"2024-03-29T09:55:35.477846+0000","last_fullsized":"2024-03-29T09:55:35.477846+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:44:31.526558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452741+0000","last_change":"2024-03-29T09:55:35.459789+0000","last_active":"2024-03-29T09:55:36.452741+0000","last_peered":"2024-03-29T09:55:36.452741+0000","last_clean":"2024-03-29T09:55:36.452741+0000","last_became_active":"2024-03-29T09:55:35.459495+0000","last_became_peered":"2024-03-29T09:55:35.459495+0000","last_unstale":"2024-03-29T09:55:36.452741+0000","last_undegraded":"2024-03-29T09:55:36.452741+0000","last_fullsized":"2024-03-29T09:55:36.452741+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:08:40.125637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452775+0000","last_change":"2024-03-29T09:55:35.458995+0000","last_active":"2024-03-29T09:55:36.452775+0000","last_peered":"2024-03-29T09:55:36.452775+0000","last_clean":"2024-03-29T09:55:36.452775+0000","last_became_active":"2024-03-29T09:55:35.458702+0000","last_became_peered":"2024-03-29T09:55:35.458702+0000","last_unstale":"2024-03-29T09:55:36.452775+0000","last_undegraded":"2024-03-29T09:55:36.452775+0000","last_fullsized":"2024-03-29T09:55:36.452775+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:27.138795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452371+0000","last_change":"2024-03-29T09:55:35.466771+0000","last_active":"2024-03-29T09:55:36.452371+0000","last_peered":"2024-03-29T09:55:36.452371+0000","last_clean":"2024-03-29T09:55:36.452371+0000","last_became_active":"2024-03-29T09:55:35.466485+0000","last_became_peered":"2024-03-29T09:55:35.466485+0000","last_unstale":"2024-03-29T09:55:36.452371+0000","last_undegraded":"2024-03-29T09:55:36.452371+0000","last_fullsized":"2024-03-29T09:55:36.452371+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:01.829351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.465591+0000","last_change":"2024-03-29T09:55:35.459865+0000","last_active":"2024-03-29T09:55:35.465591+0000","last_peered":"2024-03-29T09:55:35.465591+0000","last_clean":"2024-03-29T09:55:35.465591+0000","last_became_active":"2024-03-29T09:55:35.459581+0000","last_became_peered":"2024-03-29T09:55:35.459581+0000","last_unstale":"2024-03-29T09:55:35.465591+0000","last_undegraded":"2024-03-29T09:55:35.465591+0000","last_fullsized":"2024-03-29T09:55:35.465591+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:36.964655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452656+0000","last_change":"2024-03-29T09:55:34.940588+0000","last_active":"2024-03-29T09:55:36.452656+0000","last_peered":"2024-03-29T09:55:36.452656+0000","last_clean":"2024-03-29T09:55:36.452656+0000","last_became_active":"2024-03-29T09:55:34.940108+0000","last_became_peered":"2024-03-29T09:55:34.940108+0000","last_unstale":"2024-03-29T09:55:36.452656+0000","last_undegraded":"2024-03-29T09:55:36.452656+0000","last_fullsized":"2024-03-29T09:55:36.452656+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:39.068027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452553+0000","last_change":"2024-03-29T09:55:35.464555+0000","last_active":"2024-03-29T09:55:36.452553+0000","last_peered":"2024-03-29T09:55:36.452553+0000","last_clean":"2024-03-29T09:55:36.452553+0000","last_became_active":"2024-03-29T09:55:35.463938+0000","last_became_peered":"2024-03-29T09:55:35.463938+0000","last_unstale":"2024-03-29T09:55:36.452553+0000","last_undegraded":"2024-03-29T09:55:36.452553+0000","last_fullsized":"2024-03-29T09:55:36.452553+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:13.885594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.469026+0000","last_change":"2024-03-29T09:55:34.944943+0000","last_active":"2024-03-29T09:55:35.469026+0000","last_peered":"2024-03-29T09:55:35.469026+0000","last_clean":"2024-03-29T09:55:35.469026+0000","last_became_active":"2024-03-29T09:55:34.944450+0000","last_became_peered":"2024-03-29T09:55:34.944450+0000","last_unstale":"2024-03-29T09:55:35.469026+0000","last_undegraded":"2024-03-29T09:55:35.469026+0000","last_fullsized":"2024-03-29T09:55:35.469026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:01.683697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477841+0000","last_change":"2024-03-29T09:55:35.472922+0000","last_active":"2024-03-29T09:55:35.477841+0000","last_peered":"2024-03-29T09:55:35.477841+0000","last_clean":"2024-03-29T09:55:35.477841+0000","last_became_active":"2024-03-29T09:55:35.472623+0000","last_became_peered":"2024-03-29T09:55:35.472623+0000","last_unstale":"2024-03-29T09:55:35.477841+0000","last_undegraded":"2024-03-29T09:55:35.477841+0000","last_fullsized":"2024-03-29T09:55:35.477841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:14:54.261455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.480991+0000","last_change":"2024-03-29T09:55:34.949340+0000","last_active":"2024-03-29T09:55:35.480991+0000","last_peered":"2024-03-29T09:55:35.480991+0000","last_clean":"2024-03-29T09:55:35.480991+0000","last_became_active":"2024-03-29T09:55:34.948898+0000","last_became_peered":"2024-03-29T09:55:34.948898+0000","last_unstale":"2024-03-29T09:55:35.480991+0000","last_undegraded":"2024-03-29T09:55:35.480991+0000","last_fullsized":"2024-03-29T09:55:35.480991+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:50:30.442405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.481257+0000","last_change":"2024-03-29T09:55:35.478699+0000","last_active":"2024-03-29T09:55:35.481257+0000","last_peered":"2024-03-29T09:55:35.481257+0000","last_clean":"2024-03-29T09:55:35.481257+0000","last_became_active":"2024-03-29T09:55:35.477819+0000","last_became_peered":"2024-03-29T09:55:35.477819+0000","last_unstale":"2024-03-29T09:55:35.481257+0000","last_undegraded":"2024-03-29T09:55:35.481257+0000","last_fullsized":"2024-03-29T09:55:35.481257+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:49:22.165389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453463+0000","last_change":"2024-03-29T09:55:34.941607+0000","last_active":"2024-03-29T09:55:36.453463+0000","last_peered":"2024-03-29T09:55:36.453463+0000","last_clean":"2024-03-29T09:55:36.453463+0000","last_became_active":"2024-03-29T09:55:34.940784+0000","last_became_peered":"2024-03-29T09:55:34.940784+0000","last_unstale":"2024-03-29T09:55:36.453463+0000","last_undegraded":"2024-03-29T09:55:36.453463+0000","last_fullsized":"2024-03-29T09:55:36.453463+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:43.371924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452781+0000","last_change":"2024-03-29T09:55:35.473648+0000","last_active":"2024-03-29T09:55:36.452781+0000","last_peered":"2024-03-29T09:55:36.452781+0000","last_clean":"2024-03-29T09:55:36.452781+0000","last_became_active":"2024-03-29T09:55:35.473366+0000","last_became_peered":"2024-03-29T09:55:35.473366+0000","last_unstale":"2024-03-29T09:55:36.452781+0000","last_undegraded":"2024-03-29T09:55:36.452781+0000","last_fullsized":"2024-03-29T09:55:36.452781+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:16:25.336942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.479100+0000","last_change":"2024-03-29T09:55:34.947769+0000","last_active":"2024-03-29T09:55:35.479100+0000","last_peered":"2024-03-29T09:55:35.479100+0000","last_clean":"2024-03-29T09:55:35.479100+0000","last_became_active":"2024-03-29T09:55:34.947520+0000","last_became_peered":"2024-03-29T09:55:34.947520+0000","last_unstale":"2024-03-29T09:55:35.479100+0000","last_undegraded":"2024-03-29T09:55:35.479100+0000","last_fullsized":"2024-03-29T09:55:35.479100+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.781386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.468431+0000","last_change":"2024-03-29T09:55:35.458483+0000","last_active":"2024-03-29T09:55:35.468431+0000","last_peered":"2024-03-29T09:55:35.468431+0000","last_clean":"2024-03-29T09:55:35.468431+0000","last_became_active":"2024-03-29T09:55:35.458210+0000","last_became_peered":"2024-03-29T09:55:35.458210+0000","last_unstale":"2024-03-29T09:55:35.468431+0000","last_undegraded":"2024-03-29T09:55:35.468431+0000","last_fullsized":"2024-03-29T09:55:35.468431+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:50.835509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.479085+0000","last_change":"2024-03-29T09:55:34.947855+0000","last_active":"2024-03-29T09:55:35.479085+0000","last_peered":"2024-03-29T09:55:35.479085+0000","last_clean":"2024-03-29T09:55:35.479085+0000","last_became_active":"2024-03-29T09:55:34.947601+0000","last_became_peered":"2024-03-29T09:55:34.947601+0000","last_unstale":"2024-03-29T09:55:35.479085+0000","last_undegraded":"2024-03-29T09:55:35.479085+0000","last_fullsized":"2024-03-29T09:55:35.479085+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:22.913621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475275+0000","last_change":"2024-03-29T09:55:35.470802+0000","last_active":"2024-03-29T09:55:35.475275+0000","last_peered":"2024-03-29T09:55:35.475275+0000","last_clean":"2024-03-29T09:55:35.475275+0000","last_became_active":"2024-03-29T09:55:35.470534+0000","last_became_peered":"2024-03-29T09:55:35.470534+0000","last_unstale":"2024-03-29T09:55:35.475275+0000","last_undegraded":"2024-03-29T09:55:35.475275+0000","last_fullsized":"2024-03-29T09:55:35.475275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:06.711276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.445329+0000","last_change":"2024-03-29T09:55:34.947192+0000","last_active":"2024-03-29T09:55:35.445329+0000","last_peered":"2024-03-29T09:55:35.445329+0000","last_clean":"2024-03-29T09:55:35.445329+0000","last_became_active":"2024-03-29T09:55:34.946034+0000","last_became_peered":"2024-03-29T09:55:34.946034+0000","last_unstale":"2024-03-29T09:55:35.445329+0000","last_undegraded":"2024-03-29T09:55:35.445329+0000","last_fullsized":"2024-03-29T09:55:35.445329+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:28:48.871162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.474421+0000","last_change":"2024-03-29T09:55:35.466599+0000","last_active":"2024-03-29T09:55:35.474421+0000","last_peered":"2024-03-29T09:55:35.474421+0000","last_clean":"2024-03-29T09:55:35.474421+0000","last_became_active":"2024-03-29T09:55:35.466276+0000","last_became_peered":"2024-03-29T09:55:35.466276+0000","last_unstale":"2024-03-29T09:55:35.474421+0000","last_undegraded":"2024-03-29T09:55:35.474421+0000","last_fullsized":"2024-03-29T09:55:35.474421+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:13.885594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.454732+0000","last_change":"2024-03-29T09:55:34.934895+0000","last_active":"2024-03-29T09:55:35.454732+0000","last_peered":"2024-03-29T09:55:35.454732+0000","last_clean":"2024-03-29T09:55:35.454732+0000","last_became_active":"2024-03-29T09:55:34.933758+0000","last_became_peered":"2024-03-29T09:55:34.933758+0000","last_unstale":"2024-03-29T09:55:35.454732+0000","last_undegraded":"2024-03-29T09:55:35.454732+0000","last_fullsized":"2024-03-29T09:55:35.454732+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:10:16.221815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475809+0000","last_change":"2024-03-29T09:55:35.468023+0000","last_active":"2024-03-29T09:55:35.475809+0000","last_peered":"2024-03-29T09:55:35.475809+0000","last_clean":"2024-03-29T09:55:35.475809+0000","last_became_active":"2024-03-29T09:55:35.467702+0000","last_became_peered":"2024-03-29T09:55:35.467702+0000","last_unstale":"2024-03-29T09:55:35.475809+0000","last_undegraded":"2024-03-29T09:55:35.475809+0000","last_fullsized":"2024-03-29T09:55:35.475809+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:20:04.154117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.449003+0000","last_change":"2024-03-29T09:55:34.946594+0000","last_active":"2024-03-29T09:55:35.449003+0000","last_peered":"2024-03-29T09:55:35.449003+0000","last_clean":"2024-03-29T09:55:35.449003+0000","last_became_active":"2024-03-29T09:55:34.946263+0000","last_became_peered":"2024-03-29T09:55:34.946263+0000","last_unstale":"2024-03-29T09:55:35.449003+0000","last_undegraded":"2024-03-29T09:55:35.449003+0000","last_fullsized":"2024-03-29T09:55:35.449003+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:32.894992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477883+0000","last_change":"2024-03-29T09:55:35.471254+0000","last_active":"2024-03-29T09:55:35.477883+0000","last_peered":"2024-03-29T09:55:35.477883+0000","last_clean":"2024-03-29T09:55:35.477883+0000","last_became_active":"2024-03-29T09:55:35.471007+0000","last_became_peered":"2024-03-29T09:55:35.471007+0000","last_unstale":"2024-03-29T09:55:35.477883+0000","last_undegraded":"2024-03-29T09:55:35.477883+0000","last_fullsized":"2024-03-29T09:55:35.477883+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:20:04.154117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.451723+0000","last_change":"2024-03-29T09:55:34.950126+0000","last_active":"2024-03-29T09:55:35.451723+0000","last_peered":"2024-03-29T09:55:35.451723+0000","last_clean":"2024-03-29T09:55:35.451723+0000","last_became_active":"2024-03-29T09:55:34.949434+0000","last_became_peered":"2024-03-29T09:55:34.949434+0000","last_unstale":"2024-03-29T09:55:35.451723+0000","last_undegraded":"2024-03-29T09:55:35.451723+0000","last_fullsized":"2024-03-29T09:55:35.451723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:26.131220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477346+0000","last_change":"2024-03-29T09:55:35.468307+0000","last_active":"2024-03-29T09:55:35.477346+0000","last_peered":"2024-03-29T09:55:35.477346+0000","last_clean":"2024-03-29T09:55:35.477346+0000","last_became_active":"2024-03-29T09:55:35.468014+0000","last_became_peered":"2024-03-29T09:55:35.468014+0000","last_unstale":"2024-03-29T09:55:35.477346+0000","last_undegraded":"2024-03-29T09:55:35.477346+0000","last_fullsized":"2024-03-29T09:55:35.477346+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:59.620621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478025+0000","last_change":"2024-03-29T09:55:34.944741+0000","last_active":"2024-03-29T09:55:35.478025+0000","last_peered":"2024-03-29T09:55:35.478025+0000","last_clean":"2024-03-29T09:55:35.478025+0000","last_became_active":"2024-03-29T09:55:34.944219+0000","last_became_peered":"2024-03-29T09:55:34.944219+0000","last_unstale":"2024-03-29T09:55:35.478025+0000","last_undegraded":"2024-03-29T09:55:35.478025+0000","last_fullsized":"2024-03-29T09:55:35.478025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:12.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475729+0000","last_change":"2024-03-29T09:55:35.466268+0000","last_active":"2024-03-29T09:55:35.475729+0000","last_peered":"2024-03-29T09:55:35.475729+0000","last_clean":"2024-03-29T09:55:35.475729+0000","last_became_active":"2024-03-29T09:55:35.465694+0000","last_became_peered":"2024-03-29T09:55:35.465694+0000","last_unstale":"2024-03-29T09:55:35.475729+0000","last_undegraded":"2024-03-29T09:55:35.475729+0000","last_fullsized":"2024-03-29T09:55:35.475729+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:01.829351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.483492+0000","last_change":"2024-03-29T09:55:34.949317+0000","last_active":"2024-03-29T09:55:35.483492+0000","last_peered":"2024-03-29T09:55:35.483492+0000","last_clean":"2024-03-29T09:55:35.483492+0000","last_became_active":"2024-03-29T09:55:34.948798+0000","last_became_peered":"2024-03-29T09:55:34.948798+0000","last_unstale":"2024-03-29T09:55:35.483492+0000","last_undegraded":"2024-03-29T09:55:35.483492+0000","last_fullsized":"2024-03-29T09:55:35.483492+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:23.265666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475934+0000","last_change":"2024-03-29T09:55:35.467623+0000","last_active":"2024-03-29T09:55:35.475934+0000","last_peered":"2024-03-29T09:55:35.475934+0000","last_clean":"2024-03-29T09:55:35.475934+0000","last_became_active":"2024-03-29T09:55:35.467160+0000","last_became_peered":"2024-03-29T09:55:35.467160+0000","last_unstale":"2024-03-29T09:55:35.475934+0000","last_undegraded":"2024-03-29T09:55:35.475934+0000","last_fullsized":"2024-03-29T09:55:35.475934+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:40:33.535101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.473728+0000","last_change":"2024-03-29T09:55:34.942233+0000","last_active":"2024-03-29T09:55:35.473728+0000","last_peered":"2024-03-29T09:55:35.473728+0000","last_clean":"2024-03-29T09:55:35.473728+0000","last_became_active":"2024-03-29T09:55:34.941182+0000","last_became_peered":"2024-03-29T09:55:34.941182+0000","last_unstale":"2024-03-29T09:55:35.473728+0000","last_undegraded":"2024-03-29T09:55:35.473728+0000","last_fullsized":"2024-03-29T09:55:35.473728+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:50:39.393444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.468668+0000","last_change":"2024-03-29T09:55:35.457426+0000","last_active":"2024-03-29T09:55:35.468668+0000","last_peered":"2024-03-29T09:55:35.468668+0000","last_clean":"2024-03-29T09:55:35.468668+0000","last_became_active":"2024-03-29T09:55:35.457168+0000","last_became_peered":"2024-03-29T09:55:35.457168+0000","last_unstale":"2024-03-29T09:55:35.468668+0000","last_undegraded":"2024-03-29T09:55:35.468668+0000","last_fullsized":"2024-03-29T09:55:35.468668+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:50.835509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.457512+0000","last_change":"2024-03-29T09:55:34.933086+0000","last_active":"2024-03-29T09:55:35.457512+0000","last_peered":"2024-03-29T09:55:35.457512+0000","last_clean":"2024-03-29T09:55:35.457512+0000","last_became_active":"2024-03-29T09:55:34.932742+0000","last_became_peered":"2024-03-29T09:55:34.932742+0000","last_unstale":"2024-03-29T09:55:35.457512+0000","last_undegraded":"2024-03-29T09:55:35.457512+0000","last_fullsized":"2024-03-29T09:55:35.457512+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:22.913621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452265+0000","last_change":"2024-03-29T09:55:35.474578+0000","last_active":"2024-03-29T09:55:36.452265+0000","last_peered":"2024-03-29T09:55:36.452265+0000","last_clean":"2024-03-29T09:55:36.452265+0000","last_became_active":"2024-03-29T09:55:35.474051+0000","last_became_peered":"2024-03-29T09:55:35.474051+0000","last_unstale":"2024-03-29T09:55:36.452265+0000","last_undegraded":"2024-03-29T09:55:36.452265+0000","last_fullsized":"2024-03-29T09:55:36.452265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:59.893866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.445431+0000","last_change":"2024-03-29T09:55:34.947164+0000","last_active":"2024-03-29T09:55:35.445431+0000","last_peered":"2024-03-29T09:55:35.445431+0000","last_clean":"2024-03-29T09:55:35.445431+0000","last_became_active":"2024-03-29T09:55:34.946483+0000","last_became_peered":"2024-03-29T09:55:34.946483+0000","last_unstale":"2024-03-29T09:55:35.445431+0000","last_undegraded":"2024-03-29T09:55:35.445431+0000","last_fullsized":"2024-03-29T09:55:35.445431+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:49:21.157814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452390+0000","last_change":"2024-03-29T09:55:35.458353+0000","last_active":"2024-03-29T09:55:36.452390+0000","last_peered":"2024-03-29T09:55:36.452390+0000","last_clean":"2024-03-29T09:55:36.452390+0000","last_became_active":"2024-03-29T09:55:35.456888+0000","last_became_peered":"2024-03-29T09:55:35.456888+0000","last_unstale":"2024-03-29T09:55:36.452390+0000","last_undegraded":"2024-03-29T09:55:36.452390+0000","last_fullsized":"2024-03-29T09:55:36.452390+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:24.273241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.481922+0000","last_change":"2024-03-29T09:55:34.941832+0000","last_active":"2024-03-29T09:55:35.481922+0000","last_peered":"2024-03-29T09:55:35.481922+0000","last_clean":"2024-03-29T09:55:35.481922+0000","last_became_active":"2024-03-29T09:55:34.941309+0000","last_became_peered":"2024-03-29T09:55:34.941309+0000","last_unstale":"2024-03-29T09:55:35.481922+0000","last_undegraded":"2024-03-29T09:55:35.481922+0000","last_fullsized":"2024-03-29T09:55:35.481922+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:40:32.527526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.468145+0000","last_change":"2024-03-29T09:55:35.457654+0000","last_active":"2024-03-29T09:55:35.468145+0000","last_peered":"2024-03-29T09:55:35.468145+0000","last_clean":"2024-03-29T09:55:35.468145+0000","last_became_active":"2024-03-29T09:55:35.457396+0000","last_became_peered":"2024-03-29T09:55:35.457396+0000","last_unstale":"2024-03-29T09:55:35.468145+0000","last_undegraded":"2024-03-29T09:55:35.468145+0000","last_fullsized":"2024-03-29T09:55:35.468145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:28:49.878736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478893+0000","last_change":"2024-03-29T09:55:34.943595+0000","last_active":"2024-03-29T09:55:35.478893+0000","last_peered":"2024-03-29T09:55:35.478893+0000","last_clean":"2024-03-29T09:55:35.478893+0000","last_became_active":"2024-03-29T09:55:34.943312+0000","last_became_peered":"2024-03-29T09:55:34.943312+0000","last_unstale":"2024-03-29T09:55:35.478893+0000","last_undegraded":"2024-03-29T09:55:35.478893+0000","last_fullsized":"2024-03-29T09:55:35.478893+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:06.263255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.469068+0000","last_change":"2024-03-29T09:55:35.463198+0000","last_active":"2024-03-29T09:55:35.469068+0000","last_peered":"2024-03-29T09:55:35.469068+0000","last_clean":"2024-03-29T09:55:35.469068+0000","last_became_active":"2024-03-29T09:55:35.460798+0000","last_became_peered":"2024-03-29T09:55:35.460798+0000","last_unstale":"2024-03-29T09:55:35.469068+0000","last_undegraded":"2024-03-29T09:55:35.469068+0000","last_fullsized":"2024-03-29T09:55:35.469068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:27:17.579514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.473736+0000","last_change":"2024-03-29T09:55:34.941454+0000","last_active":"2024-03-29T09:55:35.473736+0000","last_peered":"2024-03-29T09:55:35.473736+0000","last_clean":"2024-03-29T09:55:35.473736+0000","last_became_active":"2024-03-29T09:55:34.941187+0000","last_became_peered":"2024-03-29T09:55:34.941187+0000","last_unstale":"2024-03-29T09:55:35.473736+0000","last_undegraded":"2024-03-29T09:55:35.473736+0000","last_fullsized":"2024-03-29T09:55:35.473736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:39.068027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.481309+0000","last_change":"2024-03-29T09:55:35.479712+0000","last_active":"2024-03-29T09:55:35.481309+0000","last_peered":"2024-03-29T09:55:35.481309+0000","last_clean":"2024-03-29T09:55:35.481309+0000","last_became_active":"2024-03-29T09:55:35.479404+0000","last_became_peered":"2024-03-29T09:55:35.479404+0000","last_unstale":"2024-03-29T09:55:35.481309+0000","last_undegraded":"2024-03-29T09:55:35.481309+0000","last_fullsized":"2024-03-29T09:55:35.481309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:36.964655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475555+0000","last_change":"2024-03-29T09:55:34.938571+0000","last_active":"2024-03-29T09:55:35.475555+0000","last_peered":"2024-03-29T09:55:35.475555+0000","last_clean":"2024-03-29T09:55:35.475555+0000","last_became_active":"2024-03-29T09:55:34.937776+0000","last_became_peered":"2024-03-29T09:55:34.937776+0000","last_unstale":"2024-03-29T09:55:35.475555+0000","last_undegraded":"2024-03-29T09:55:35.475555+0000","last_fullsized":"2024-03-29T09:55:35.475555+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:06.263255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453539+0000","last_change":"2024-03-29T09:55:35.460515+0000","last_active":"2024-03-29T09:55:36.453539+0000","last_peered":"2024-03-29T09:55:36.453539+0000","last_clean":"2024-03-29T09:55:36.453539+0000","last_became_active":"2024-03-29T09:55:35.460266+0000","last_became_peered":"2024-03-29T09:55:35.460266+0000","last_unstale":"2024-03-29T09:55:36.453539+0000","last_undegraded":"2024-03-29T09:55:36.453539+0000","last_fullsized":"2024-03-29T09:55:36.453539+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:50:31.449980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.461204+0000","last_change":"2024-03-29T09:55:34.948028+0000","last_active":"2024-03-29T09:55:35.461204+0000","last_peered":"2024-03-29T09:55:35.461204+0000","last_clean":"2024-03-29T09:55:35.461204+0000","last_became_active":"2024-03-29T09:55:34.947688+0000","last_became_peered":"2024-03-29T09:55:34.947688+0000","last_unstale":"2024-03-29T09:55:35.461204+0000","last_undegraded":"2024-03-29T09:55:35.461204+0000","last_fullsized":"2024-03-29T09:55:35.461204+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:49:47.226070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453336+0000","last_change":"2024-03-29T09:55:35.468390+0000","last_active":"2024-03-29T09:55:36.453336+0000","last_peered":"2024-03-29T09:55:36.453336+0000","last_clean":"2024-03-29T09:55:36.453336+0000","last_became_active":"2024-03-29T09:55:35.468148+0000","last_became_peered":"2024-03-29T09:55:35.468148+0000","last_unstale":"2024-03-29T09:55:36.453336+0000","last_undegraded":"2024-03-29T09:55:36.453336+0000","last_fullsized":"2024-03-29T09:55:36.453336+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:34.798479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.473579+0000","last_change":"2024-03-29T09:55:34.944643+0000","last_active":"2024-03-29T09:55:35.473579+0000","last_peered":"2024-03-29T09:55:35.473579+0000","last_clean":"2024-03-29T09:55:35.473579+0000","last_became_active":"2024-03-29T09:55:34.943984+0000","last_became_peered":"2024-03-29T09:55:34.943984+0000","last_unstale":"2024-03-29T09:55:35.473579+0000","last_undegraded":"2024-03-29T09:55:35.473579+0000","last_fullsized":"2024-03-29T09:55:35.473579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:12.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453702+0000","last_change":"2024-03-29T09:55:35.466473+0000","last_active":"2024-03-29T09:55:36.453702+0000","last_peered":"2024-03-29T09:55:36.453702+0000","last_clean":"2024-03-29T09:55:36.453702+0000","last_became_active":"2024-03-29T09:55:35.466216+0000","last_became_peered":"2024-03-29T09:55:35.466216+0000","last_unstale":"2024-03-29T09:55:36.453702+0000","last_undegraded":"2024-03-29T09:55:36.453702+0000","last_fullsized":"2024-03-29T09:55:36.453702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:40:33.535101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452541+0000","last_change":"2024-03-29T09:55:34.942491+0000","last_active":"2024-03-29T09:55:36.452541+0000","last_peered":"2024-03-29T09:55:36.452541+0000","last_clean":"2024-03-29T09:55:36.452541+0000","last_became_active":"2024-03-29T09:55:34.941777+0000","last_became_peered":"2024-03-29T09:55:34.941777+0000","last_unstale":"2024-03-29T09:55:36.452541+0000","last_undegraded":"2024-03-29T09:55:36.452541+0000","last_fullsized":"2024-03-29T09:55:36.452541+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:10:16.221815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453018+0000","last_change":"2024-03-29T09:55:35.458455+0000","last_active":"2024-03-29T09:55:36.453018+0000","last_peered":"2024-03-29T09:55:36.453018+0000","last_clean":"2024-03-29T09:55:36.453018+0000","last_became_active":"2024-03-29T09:55:35.458190+0000","last_became_peered":"2024-03-29T09:55:35.458190+0000","last_unstale":"2024-03-29T09:55:36.453018+0000","last_undegraded":"2024-03-29T09:55:36.453018+0000","last_fullsized":"2024-03-29T09:55:36.453018+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:13.885594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.454939+0000","last_change":"2024-03-29T09:55:34.942208+0000","last_active":"2024-03-29T09:55:35.454939+0000","last_peered":"2024-03-29T09:55:35.454939+0000","last_clean":"2024-03-29T09:55:35.454939+0000","last_became_active":"2024-03-29T09:55:34.941888+0000","last_became_peered":"2024-03-29T09:55:34.941888+0000","last_unstale":"2024-03-29T09:55:35.454939+0000","last_undegraded":"2024-03-29T09:55:35.454939+0000","last_fullsized":"2024-03-29T09:55:35.454939+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:41:49.708910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452809+0000","last_change":"2024-03-29T09:55:35.462110+0000","last_active":"2024-03-29T09:55:36.452809+0000","last_peered":"2024-03-29T09:55:36.452809+0000","last_clean":"2024-03-29T09:55:36.452809+0000","last_became_active":"2024-03-29T09:55:35.461820+0000","last_became_peered":"2024-03-29T09:55:35.461820+0000","last_unstale":"2024-03-29T09:55:36.452809+0000","last_undegraded":"2024-03-29T09:55:36.452809+0000","last_fullsized":"2024-03-29T09:55:36.452809+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:33:34.801403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.444737+0000","last_change":"2024-03-29T09:55:34.944551+0000","last_active":"2024-03-29T09:55:35.444737+0000","last_peered":"2024-03-29T09:55:35.444737+0000","last_clean":"2024-03-29T09:55:35.444737+0000","last_became_active":"2024-03-29T09:55:34.944214+0000","last_became_peered":"2024-03-29T09:55:34.944214+0000","last_unstale":"2024-03-29T09:55:35.444737+0000","last_undegraded":"2024-03-29T09:55:35.444737+0000","last_fullsized":"2024-03-29T09:55:35.444737+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:50:30.442405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453466+0000","last_change":"2024-03-29T09:55:35.464636+0000","last_active":"2024-03-29T09:55:36.453466+0000","last_peered":"2024-03-29T09:55:36.453466+0000","last_clean":"2024-03-29T09:55:36.453466+0000","last_became_active":"2024-03-29T09:55:35.463304+0000","last_became_peered":"2024-03-29T09:55:35.463304+0000","last_unstale":"2024-03-29T09:55:36.453466+0000","last_undegraded":"2024-03-29T09:55:36.453466+0000","last_fullsized":"2024-03-29T09:55:36.453466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:24.273241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.470728+0000","last_change":"2024-03-29T09:55:34.936806+0000","last_active":"2024-03-29T09:55:35.470728+0000","last_peered":"2024-03-29T09:55:35.470728+0000","last_clean":"2024-03-29T09:55:35.470728+0000","last_became_active":"2024-03-29T09:55:34.936552+0000","last_became_peered":"2024-03-29T09:55:34.936552+0000","last_unstale":"2024-03-29T09:55:35.470728+0000","last_undegraded":"2024-03-29T09:55:35.470728+0000","last_fullsized":"2024-03-29T09:55:35.470728+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:10:16.221815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475860+0000","last_change":"2024-03-29T09:55:35.469638+0000","last_active":"2024-03-29T09:55:35.475860+0000","last_peered":"2024-03-29T09:55:35.475860+0000","last_clean":"2024-03-29T09:55:35.475860+0000","last_became_active":"2024-03-29T09:55:35.469375+0000","last_became_peered":"2024-03-29T09:55:35.469375+0000","last_unstale":"2024-03-29T09:55:35.475860+0000","last_undegraded":"2024-03-29T09:55:35.475860+0000","last_fullsized":"2024-03-29T09:55:35.475860+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:33:34.801403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.451817+0000","last_change":"2024-03-29T09:55:34.944733+0000","last_active":"2024-03-29T09:55:35.451817+0000","last_peered":"2024-03-29T09:55:35.451817+0000","last_clean":"2024-03-29T09:55:35.451817+0000","last_became_active":"2024-03-29T09:55:34.944161+0000","last_became_peered":"2024-03-29T09:55:34.944161+0000","last_unstale":"2024-03-29T09:55:35.451817+0000","last_undegraded":"2024-03-29T09:55:35.451817+0000","last_fullsized":"2024-03-29T09:55:35.451817+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:23.265666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.469074+0000","last_change":"2024-03-29T09:55:35.464745+0000","last_active":"2024-03-29T09:55:35.469074+0000","last_peered":"2024-03-29T09:55:35.469074+0000","last_clean":"2024-03-29T09:55:35.469074+0000","last_became_active":"2024-03-29T09:55:35.464410+0000","last_became_peered":"2024-03-29T09:55:35.464410+0000","last_unstale":"2024-03-29T09:55:35.469074+0000","last_undegraded":"2024-03-29T09:55:35.469074+0000","last_fullsized":"2024-03-29T09:55:35.469074+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:28:49.878736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.458594+0000","last_change":"2024-03-29T09:55:34.935319+0000","last_active":"2024-03-29T09:55:35.458594+0000","last_peered":"2024-03-29T09:55:35.458594+0000","last_clean":"2024-03-29T09:55:35.458594+0000","last_became_active":"2024-03-29T09:55:34.934799+0000","last_became_peered":"2024-03-29T09:55:34.934799+0000","last_unstale":"2024-03-29T09:55:35.458594+0000","last_undegraded":"2024-03-29T09:55:35.458594+0000","last_fullsized":"2024-03-29T09:55:35.458594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:28.148256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452491+0000","last_change":"2024-03-29T09:55:35.458803+0000","last_active":"2024-03-29T09:55:36.452491+0000","last_peered":"2024-03-29T09:55:36.452491+0000","last_clean":"2024-03-29T09:55:36.452491+0000","last_became_active":"2024-03-29T09:55:35.456734+0000","last_became_peered":"2024-03-29T09:55:35.456734+0000","last_unstale":"2024-03-29T09:55:36.452491+0000","last_undegraded":"2024-03-29T09:55:36.452491+0000","last_fullsized":"2024-03-29T09:55:36.452491+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:50:31.449980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.450005+0000","last_change":"2024-03-29T09:55:34.944828+0000","last_active":"2024-03-29T09:55:35.450005+0000","last_peered":"2024-03-29T09:55:35.450005+0000","last_clean":"2024-03-29T09:55:35.450005+0000","last_became_active":"2024-03-29T09:55:34.944367+0000","last_became_peered":"2024-03-29T09:55:34.944367+0000","last_unstale":"2024-03-29T09:55:35.450005+0000","last_undegraded":"2024-03-29T09:55:35.450005+0000","last_fullsized":"2024-03-29T09:55:35.450005+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:41:49.708910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.469504+0000","last_change":"2024-03-29T09:55:35.461679+0000","last_active":"2024-03-29T09:55:35.469504+0000","last_peered":"2024-03-29T09:55:35.469504+0000","last_clean":"2024-03-29T09:55:35.469504+0000","last_became_active":"2024-03-29T09:55:35.460935+0000","last_became_peered":"2024-03-29T09:55:35.460935+0000","last_unstale":"2024-03-29T09:55:35.469504+0000","last_undegraded":"2024-03-29T09:55:35.469504+0000","last_fullsized":"2024-03-29T09:55:35.469504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:14:54.261455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.457637+0000","last_change":"2024-03-29T09:55:34.931195+0000","last_active":"2024-03-29T09:55:35.457637+0000","last_peered":"2024-03-29T09:55:35.457637+0000","last_clean":"2024-03-29T09:55:35.457637+0000","last_became_active":"2024-03-29T09:55:34.930885+0000","last_became_peered":"2024-03-29T09:55:34.930885+0000","last_unstale":"2024-03-29T09:55:35.457637+0000","last_undegraded":"2024-03-29T09:55:35.457637+0000","last_fullsized":"2024-03-29T09:55:35.457637+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:39.068027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478771+0000","last_change":"2024-03-29T09:55:35.472255+0000","last_active":"2024-03-29T09:55:35.478771+0000","last_peered":"2024-03-29T09:55:35.478771+0000","last_clean":"2024-03-29T09:55:35.478771+0000","last_became_active":"2024-03-29T09:55:35.471964+0000","last_became_peered":"2024-03-29T09:55:35.471964+0000","last_unstale":"2024-03-29T09:55:35.478771+0000","last_undegraded":"2024-03-29T09:55:35.478771+0000","last_fullsized":"2024-03-29T09:55:35.478771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:39.961948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.455006+0000","last_change":"2024-03-29T09:55:34.935632+0000","last_active":"2024-03-29T09:55:35.455006+0000","last_peered":"2024-03-29T09:55:35.455006+0000","last_clean":"2024-03-29T09:55:35.455006+0000","last_became_active":"2024-03-29T09:55:34.935326+0000","last_became_peered":"2024-03-29T09:55:34.935326+0000","last_unstale":"2024-03-29T09:55:35.455006+0000","last_undegraded":"2024-03-29T09:55:35.455006+0000","last_fullsized":"2024-03-29T09:55:35.455006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:28.148256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477733+0000","last_change":"2024-03-29T09:55:35.471480+0000","last_active":"2024-03-29T09:55:35.477733+0000","last_peered":"2024-03-29T09:55:35.477733+0000","last_clean":"2024-03-29T09:55:35.477733+0000","last_became_active":"2024-03-29T09:55:35.471165+0000","last_became_peered":"2024-03-29T09:55:35.471165+0000","last_unstale":"2024-03-29T09:55:35.477733+0000","last_undegraded":"2024-03-29T09:55:35.477733+0000","last_fullsized":"2024-03-29T09:55:35.477733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:20:12.232627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.443725+0000","last_change":"2024-03-29T09:55:34.947995+0000","last_active":"2024-03-29T09:55:35.443725+0000","last_peered":"2024-03-29T09:55:35.443725+0000","last_clean":"2024-03-29T09:55:35.443725+0000","last_became_active":"2024-03-29T09:55:34.947623+0000","last_became_peered":"2024-03-29T09:55:34.947623+0000","last_unstale":"2024-03-29T09:55:35.443725+0000","last_undegraded":"2024-03-29T09:55:35.443725+0000","last_fullsized":"2024-03-29T09:55:35.443725+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:35.708460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452773+0000","last_change":"2024-03-29T09:55:35.466153+0000","last_active":"2024-03-29T09:55:36.452773+0000","last_peered":"2024-03-29T09:55:36.452773+0000","last_clean":"2024-03-29T09:55:36.452773+0000","last_became_active":"2024-03-29T09:55:35.464346+0000","last_became_peered":"2024-03-29T09:55:35.464346+0000","last_unstale":"2024-03-29T09:55:36.452773+0000","last_undegraded":"2024-03-29T09:55:36.452773+0000","last_fullsized":"2024-03-29T09:55:36.452773+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:36.716035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452411+0000","last_change":"2024-03-29T09:55:34.938938+0000","last_active":"2024-03-29T09:55:36.452411+0000","last_peered":"2024-03-29T09:55:36.452411+0000","last_clean":"2024-03-29T09:55:36.452411+0000","last_became_active":"2024-03-29T09:55:34.938456+0000","last_became_peered":"2024-03-29T09:55:34.938456+0000","last_unstale":"2024-03-29T09:55:36.452411+0000","last_undegraded":"2024-03-29T09:55:36.452411+0000","last_fullsized":"2024-03-29T09:55:36.452411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:22.913621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.449414+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T09:55:35.449414+0000","last_peered":"2024-03-29T09:55:35.449414+0000","last_clean":"2024-03-29T09:55:35.449414+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T09:55:35.449414+0000","last_undegraded":"2024-03-29T09:55:35.449414+0000","last_fullsized":"2024-03-29T09:55:35.449414+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":970752,"data_stored":736215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.748},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.51300000000000001}]},{"osd":1,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71399999999999997}]},{"osd":2,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.58599999999999997}]},{"osd":4,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.70199999999999996}]},{"osd":6,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.72299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":6.7610000000000001,"5min":6.7610000000000001,"15min":6.7610000000000001},"last":0.61899999999999999}]},{"osd":1,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.58499999999999996}]},{"osd":2,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.499},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":0.81399999999999995}]},{"osd":3,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":3.1960000000000002,"5min":3.1960000000000002,"15min":3.1960000000000002},"last":0.745},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":4.3760000000000003,"5min":4.3760000000000003,"15min":4.3760000000000003},"last":0.65400000000000003}]},{"osd":4,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":0.79400000000000004}]},{"osd":5,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.8279999999999998,"5min":2.8279999999999998,"15min":2.8279999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.72699999999999998}]},{"osd":7,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.7869999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.76500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":147343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.316},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.38800000000000001}]},{"osd":2,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.282,"5min":4.282,"15min":4.282},"last":0.35199999999999998}]},{"osd":3,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.3200000000000003,"5min":4.3200000000000003,"15min":4.3200000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.44700000000000001}]},{"osd":4,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"last":0.92200000000000004}]},{"osd":5,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":0.84099999999999997}]},{"osd":6,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":4.3899999999999997,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":0.874},{"interface":"front","average":{"1min":0.877,"5min":0.877,"15min":0.877},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.93300000000000005}]},{"osd":7,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.81799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":962560,"data_stored":735106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.44800000000000001}]},{"osd":2,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52400000000000002}]},{"osd":4,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":1.1060000000000001}]},{"osd":5,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.627},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.66900000000000004}]},{"osd":7,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":1.0049999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":145827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.435},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.33100000000000002}]},{"osd":4,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.53100000000000003}]},{"osd":5,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.56299999999999994}]},{"osd":6,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.67600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":147343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.3799999999999999,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.1840000000000002,"5min":4.1840000000000002,"15min":4.1840000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.314}]},{"osd":2,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.5970000000000004,"5min":4.5970000000000004,"15min":4.5970000000000004},"last":0.56399999999999995}]},{"osd":4,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.78400000000000003}]},{"osd":5,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.4180000000000001,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.81899999999999995}]},{"osd":7,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":4.4550000000000001,"5min":4.4550000000000001,"15min":4.4550000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.70099999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":145861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.64800000000000002}]},{"osd":1,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.59399999999999997}]},{"osd":5,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.621},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.61199999999999999}]},{"osd":6,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.97,"5min":1.97,"15min":1.97},"last":0.66600000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.58899999999999997}]},{"osd":1,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.622}]},{"osd":2,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.47899999999999998}]},{"osd":3,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.67300000000000004}]},{"osd":4,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.65500000000000003}]},{"osd":6,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.752},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.77200000000000002}]},{"osd":7,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.71399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T09:55:40.373 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T09:55:40.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.606+0000 7f5a4ee00700 1 Processor -- start 2024-03-29T09:55:40.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.606+0000 7f5a4ee00700 1 -- start start 2024-03-29T09:55:40.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a4ee00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5a4815b660 0x7f5a480d1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a4ee00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5a480c4f30 0x7f5a480cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a4ee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 0x7f5a480cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a4ee00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a48099a60 con 0x7f5a4815b660 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a4ee00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a480cce30 con 0x7f5a480c4f30 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a4ee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a480cd110 con 0x7f5a480c5390 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a47e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5a4815b660 0x7f5a480d1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a47e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5a4815b660 0x7f5a480d1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35490/0 (socket says 172.21.15.12:35490) 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a47e00700 1 -- 172.21.15.12:0/2553223067 learned_addr learned my addr 172.21.15.12:0/2553223067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a47400700 1 --2- 172.21.15.12:0/2553223067 >> v2:172.21.15.159:3300/0 conn(0x7f5a480c4f30 0x7f5a480cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a4ce00700 1 --2- 172.21.15.12:0/2553223067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 0x7f5a480cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a47e00700 1 -- 172.21.15.12:0/2553223067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 msgr2=0x7f5a480cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a47e00700 1 --2- 172.21.15.12:0/2553223067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 0x7f5a480cf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a47e00700 1 -- 172.21.15.12:0/2553223067 >> v2:172.21.15.159:3300/0 conn(0x7f5a480c4f30 msgr2=0x7f5a480cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a47e00700 1 --2- 172.21.15.12:0/2553223067 >> v2:172.21.15.159:3300/0 conn(0x7f5a480c4f30 0x7f5a480cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a47e00700 1 -- 172.21.15.12:0/2553223067 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a48099c70 con 0x7f5a4815b660 2024-03-29T09:55:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a47400700 1 --2- 172.21.15.12:0/2553223067 >> v2:172.21.15.159:3300/0 conn(0x7f5a480c4f30 0x7f5a480cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:40.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.608+0000 7f5a4ce00700 1 --2- 172.21.15.12:0/2553223067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 0x7f5a480cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:40.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.609+0000 7f5a47e00700 1 --2- 172.21.15.12:0/2553223067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 0x7f5a480d1ed0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5a48072100 tx=0x7f5a38009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8930f743b6b15c47 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:40.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.609+0000 7f5a46a00700 1 -- 172.21.15.12:0/2553223067 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3800b870 con 0x7f5a4815b660 2024-03-29T09:55:40.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.609+0000 7f5a46a00700 1 -- 172.21.15.12:0/2553223067 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3800b9d0 con 0x7f5a4815b660 2024-03-29T09:55:40.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.609+0000 7f5a46a00700 1 -- 172.21.15.12:0/2553223067 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3800bd60 con 0x7f5a4815b660 2024-03-29T09:55:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.609+0000 7f5a4ee00700 1 -- 172.21.15.12:0/2553223067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 msgr2=0x7f5a480d1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.609+0000 7f5a4ee00700 1 --2- 172.21.15.12:0/2553223067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 0x7f5a480d1ed0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5a48072100 tx=0x7f5a38009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.610+0000 7f5a4ee00700 1 -- 172.21.15.12:0/2553223067 shutdown_connections 2024-03-29T09:55:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.610+0000 7f5a4ee00700 1 --2- 172.21.15.12:0/2553223067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 0x7f5a480cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.610+0000 7f5a4ee00700 1 --2- 172.21.15.12:0/2553223067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 0x7f5a480d1ed0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.610+0000 7f5a4ee00700 1 --2- 172.21.15.12:0/2553223067 >> v2:172.21.15.159:3300/0 conn(0x7f5a480c4f30 0x7f5a480cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.610+0000 7f5a4ee00700 1 -- 172.21.15.12:0/2553223067 >> 172.21.15.12:0/2553223067 conn(0x7f5a480d13e0 msgr2=0x7f5a480d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.610+0000 7f5a4ee00700 1 -- 172.21.15.12:0/2553223067 shutdown_connections 2024-03-29T09:55:40.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.610+0000 7f5a4ee00700 1 -- 172.21.15.12:0/2553223067 wait complete. 2024-03-29T09:55:40.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.611+0000 7f5a4ee00700 1 Processor -- start 2024-03-29T09:55:40.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.612+0000 7f5a4ee00700 1 -- start start 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.612+0000 7f5a4ee00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a480c4f30 0x7f5a48150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.612+0000 7f5a4ee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 0x7f5a480b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.612+0000 7f5a4ee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 0x7f5a480b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.612+0000 7f5a4ee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a480b5530 con 0x7f5a4815b660 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.612+0000 7f5a4ee00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4815ca20 con 0x7f5a480c4f30 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.612+0000 7f5a4ee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a480b49b0 con 0x7f5a480c5390 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a4ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 0x7f5a480b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a4ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 0x7f5a480b5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35494/0 (socket says 172.21.15.12:35494) 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a4ce00700 1 -- 172.21.15.12:0/4100641378 learned_addr learned my addr 172.21.15.12:0/4100641378 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a47e00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a480c4f30 0x7f5a48150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a47400700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 0x7f5a480b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a4ce00700 1 -- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 msgr2=0x7f5a480b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a4ce00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 0x7f5a480b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a4ce00700 1 -- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a480c4f30 msgr2=0x7f5a48150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a4ce00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a480c4f30 0x7f5a48150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a4ce00700 1 -- 172.21.15.12:0/4100641378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a38009550 con 0x7f5a4815b660 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a47400700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 0x7f5a480b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a47e00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a480c4f30 0x7f5a48150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.613+0000 7f5a4ce00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 0x7f5a480b5bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000cd50 tx=0x7f5a4000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.614+0000 7f5a3fe00700 1 -- 172.21.15.12:0/4100641378 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4000f480 con 0x7f5a4815b660 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.614+0000 7f5a3fe00700 1 -- 172.21.15.12:0/4100641378 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4000f600 con 0x7f5a4815b660 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.614+0000 7f5a3fe00700 1 -- 172.21.15.12:0/4100641378 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4000f9b0 con 0x7f5a4815b660 2024-03-29T09:55:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.614+0000 7f5a4ee00700 1 -- 172.21.15.12:0/4100641378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a480b4c90 con 0x7f5a4815b660 2024-03-29T09:55:40.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.614+0000 7f5a4ee00700 1 -- 172.21.15.12:0/4100641378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a480b5250 con 0x7f5a4815b660 2024-03-29T09:55:40.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.615+0000 7f5a4ee00700 1 -- 172.21.15.12:0/4100641378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4806e470 con 0x7f5a4815b660 2024-03-29T09:55:40.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.616+0000 7f5a3fe00700 1 -- 172.21.15.12:0/4100641378 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5a4000b040 con 0x7f5a4815b660 2024-03-29T09:55:40.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.617+0000 7f5a3fe00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5a2806ac90 0x7f5a2806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:40.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.617+0000 7f5a3fe00700 1 -- 172.21.15.12:0/4100641378 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a40089ac0 con 0x7f5a4815b660 2024-03-29T09:55:40.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.618+0000 7f5a47e00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5a2806ac90 0x7f5a2806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:40.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.618+0000 7f5a47e00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5a2806ac90 0x7f5a2806d150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5a38000c00 tx=0x7f5a38008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:40.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.622+0000 7f5a3fe00700 1 -- 172.21.15.12:0/4100641378 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a40054c90 con 0x7f5a4815b660 2024-03-29T09:55:40.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.831+0000 7f5a4ee00700 1 -- 172.21.15.12:0/4100641378 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a480d01b0 con 0x7f5a2806ac90 2024-03-29T09:55:40.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.841+0000 7f5a3fe00700 1 -- 172.21.15.12:0/4100641378 <== mgr.4114 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288921 (secure 0 0 0) 0x7f5a3800bc40 con 0x7f5a2806ac90 2024-03-29T09:55:40.845 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.844+0000 7f5a4ee00700 1 -- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5a2806ac90 msgr2=0x7f5a2806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.844+0000 7f5a4ee00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5a2806ac90 0x7f5a2806d150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5a38000c00 tx=0x7f5a38008030 comp rx=0 tx=0).stop 2024-03-29T09:55:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.844+0000 7f5a4ee00700 1 -- 172.21.15.12:0/4100641378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 msgr2=0x7f5a480b5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.844+0000 7f5a4ee00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 0x7f5a480b5bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000cd50 tx=0x7f5a4000a560 comp rx=0 tx=0).stop 2024-03-29T09:55:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.845+0000 7f5a4ee00700 1 -- 172.21.15.12:0/4100641378 shutdown_connections 2024-03-29T09:55:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.845+0000 7f5a4ee00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5a2806ac90 0x7f5a2806d150 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.845+0000 7f5a4ee00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a4815b660 0x7f5a480b5bf0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.845+0000 7f5a4ee00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a480c4f30 0x7f5a48150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.845+0000 7f5a4ee00700 1 --2- 172.21.15.12:0/4100641378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a480c5390 0x7f5a480b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.845+0000 7f5a4ee00700 1 -- 172.21.15.12:0/4100641378 >> 172.21.15.12:0/4100641378 conn(0x7f5a480d13e0 msgr2=0x7f5a4806fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.845+0000 7f5a4ee00700 1 -- 172.21.15.12:0/4100641378 shutdown_connections 2024-03-29T09:55:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:40.845+0000 7f5a4ee00700 1 -- 172.21.15.12:0/4100641378 wait complete. 2024-03-29T09:55:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T09:55:40.862 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-03-29T09:55:40.454682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220320,"kb_used_data":4132,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754400,"statfs":{"total":773094113280,"available":772868505600,"internally_reserved":0,"allocated":4231168,"data_stored":2350497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001397"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452541+0000","last_change":"2024-03-29T09:55:35.452157+0000","last_active":"2024-03-29T09:55:36.452541+0000","last_peered":"2024-03-29T09:55:36.452541+0000","last_clean":"2024-03-29T09:55:36.452541+0000","last_became_active":"2024-03-29T09:55:35.451841+0000","last_became_peered":"2024-03-29T09:55:35.451841+0000","last_unstale":"2024-03-29T09:55:36.452541+0000","last_undegraded":"2024-03-29T09:55:36.452541+0000","last_fullsized":"2024-03-29T09:55:36.452541+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:02.691272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478215+0000","last_change":"2024-03-29T09:55:35.470292+0000","last_active":"2024-03-29T09:55:35.478215+0000","last_peered":"2024-03-29T09:55:35.478215+0000","last_clean":"2024-03-29T09:55:35.478215+0000","last_became_active":"2024-03-29T09:55:35.470036+0000","last_became_peered":"2024-03-29T09:55:35.470036+0000","last_unstale":"2024-03-29T09:55:35.478215+0000","last_undegraded":"2024-03-29T09:55:35.478215+0000","last_fullsized":"2024-03-29T09:55:35.478215+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:54.134659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475705+0000","last_change":"2024-03-29T09:55:35.466012+0000","last_active":"2024-03-29T09:55:35.475705+0000","last_peered":"2024-03-29T09:55:35.475705+0000","last_clean":"2024-03-29T09:55:35.475705+0000","last_became_active":"2024-03-29T09:55:35.464991+0000","last_became_peered":"2024-03-29T09:55:35.464991+0000","last_unstale":"2024-03-29T09:55:35.475705+0000","last_undegraded":"2024-03-29T09:55:35.475705+0000","last_fullsized":"2024-03-29T09:55:35.475705+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:54.134659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.479827+0000","last_change":"2024-03-29T09:55:35.468730+0000","last_active":"2024-03-29T09:55:35.479827+0000","last_peered":"2024-03-29T09:55:35.479827+0000","last_clean":"2024-03-29T09:55:35.479827+0000","last_became_active":"2024-03-29T09:55:35.468286+0000","last_became_peered":"2024-03-29T09:55:35.468286+0000","last_unstale":"2024-03-29T09:55:35.479827+0000","last_undegraded":"2024-03-29T09:55:35.479827+0000","last_fullsized":"2024-03-29T09:55:35.479827+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:28:49.878736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453619+0000","last_change":"2024-03-29T09:55:35.474980+0000","last_active":"2024-03-29T09:55:36.453619+0000","last_peered":"2024-03-29T09:55:36.453619+0000","last_clean":"2024-03-29T09:55:36.453619+0000","last_became_active":"2024-03-29T09:55:35.473492+0000","last_became_peered":"2024-03-29T09:55:35.473492+0000","last_unstale":"2024-03-29T09:55:36.453619+0000","last_undegraded":"2024-03-29T09:55:36.453619+0000","last_fullsized":"2024-03-29T09:55:36.453619+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:49:22.165389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.476372+0000","last_change":"2024-03-29T09:55:35.462563+0000","last_active":"2024-03-29T09:55:35.476372+0000","last_peered":"2024-03-29T09:55:35.476372+0000","last_clean":"2024-03-29T09:55:35.476372+0000","last_became_active":"2024-03-29T09:55:35.462256+0000","last_became_peered":"2024-03-29T09:55:35.462256+0000","last_unstale":"2024-03-29T09:55:35.476372+0000","last_undegraded":"2024-03-29T09:55:35.476372+0000","last_fullsized":"2024-03-29T09:55:35.476372+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:40:09.967878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452712+0000","last_change":"2024-03-29T09:55:35.472251+0000","last_active":"2024-03-29T09:55:36.452712+0000","last_peered":"2024-03-29T09:55:36.452712+0000","last_clean":"2024-03-29T09:55:36.452712+0000","last_became_active":"2024-03-29T09:55:35.470513+0000","last_became_peered":"2024-03-29T09:55:35.470513+0000","last_unstale":"2024-03-29T09:55:36.452712+0000","last_undegraded":"2024-03-29T09:55:36.452712+0000","last_fullsized":"2024-03-29T09:55:36.452712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:09:14.949848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.451863+0000","last_change":"2024-03-29T09:55:35.463829+0000","last_active":"2024-03-29T09:55:36.451863+0000","last_peered":"2024-03-29T09:55:36.451863+0000","last_clean":"2024-03-29T09:55:36.451863+0000","last_became_active":"2024-03-29T09:55:35.463562+0000","last_became_peered":"2024-03-29T09:55:35.463562+0000","last_unstale":"2024-03-29T09:55:36.451863+0000","last_undegraded":"2024-03-29T09:55:36.451863+0000","last_fullsized":"2024-03-29T09:55:36.451863+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:14:54.261455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.474921+0000","last_change":"2024-03-29T09:55:35.469975+0000","last_active":"2024-03-29T09:55:35.474921+0000","last_peered":"2024-03-29T09:55:35.474921+0000","last_clean":"2024-03-29T09:55:35.474921+0000","last_became_active":"2024-03-29T09:55:35.468396+0000","last_became_peered":"2024-03-29T09:55:35.468396+0000","last_unstale":"2024-03-29T09:55:35.474921+0000","last_undegraded":"2024-03-29T09:55:35.474921+0000","last_fullsized":"2024-03-29T09:55:35.474921+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:36.716035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477516+0000","last_change":"2024-03-29T09:55:35.470609+0000","last_active":"2024-03-29T09:55:35.477516+0000","last_peered":"2024-03-29T09:55:35.477516+0000","last_clean":"2024-03-29T09:55:35.477516+0000","last_became_active":"2024-03-29T09:55:35.470180+0000","last_became_peered":"2024-03-29T09:55:35.470180+0000","last_unstale":"2024-03-29T09:55:35.477516+0000","last_undegraded":"2024-03-29T09:55:35.477516+0000","last_fullsized":"2024-03-29T09:55:35.477516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:06.711276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452914+0000","last_change":"2024-03-29T09:55:35.469739+0000","last_active":"2024-03-29T09:55:36.452914+0000","last_peered":"2024-03-29T09:55:36.452914+0000","last_clean":"2024-03-29T09:55:36.452914+0000","last_became_active":"2024-03-29T09:55:35.469242+0000","last_became_peered":"2024-03-29T09:55:35.469242+0000","last_unstale":"2024-03-29T09:55:36.452914+0000","last_undegraded":"2024-03-29T09:55:36.452914+0000","last_fullsized":"2024-03-29T09:55:36.452914+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:27:17.579514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477982+0000","last_change":"2024-03-29T09:55:35.472836+0000","last_active":"2024-03-29T09:55:35.477982+0000","last_peered":"2024-03-29T09:55:35.477982+0000","last_clean":"2024-03-29T09:55:35.477982+0000","last_became_active":"2024-03-29T09:55:35.471826+0000","last_became_peered":"2024-03-29T09:55:35.471826+0000","last_unstale":"2024-03-29T09:55:35.477982+0000","last_undegraded":"2024-03-29T09:55:35.477982+0000","last_fullsized":"2024-03-29T09:55:35.477982+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:47:12.317720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453636+0000","last_change":"2024-03-29T09:55:35.473546+0000","last_active":"2024-03-29T09:55:36.453636+0000","last_peered":"2024-03-29T09:55:36.453636+0000","last_clean":"2024-03-29T09:55:36.453636+0000","last_became_active":"2024-03-29T09:55:35.473219+0000","last_became_peered":"2024-03-29T09:55:35.473219+0000","last_unstale":"2024-03-29T09:55:36.453636+0000","last_undegraded":"2024-03-29T09:55:36.453636+0000","last_fullsized":"2024-03-29T09:55:36.453636+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:02.691272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477777+0000","last_change":"2024-03-29T09:55:35.469414+0000","last_active":"2024-03-29T09:55:35.477777+0000","last_peered":"2024-03-29T09:55:35.477777+0000","last_clean":"2024-03-29T09:55:35.477777+0000","last_became_active":"2024-03-29T09:55:35.469161+0000","last_became_peered":"2024-03-29T09:55:35.469161+0000","last_unstale":"2024-03-29T09:55:35.477777+0000","last_undegraded":"2024-03-29T09:55:35.477777+0000","last_fullsized":"2024-03-29T09:55:35.477777+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:06:19.955544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.456273+0000","last_change":"2024-03-29T09:55:35.453942+0000","last_active":"2024-03-29T09:55:35.456273+0000","last_peered":"2024-03-29T09:55:35.456273+0000","last_clean":"2024-03-29T09:55:35.456273+0000","last_became_active":"2024-03-29T09:55:35.453580+0000","last_became_peered":"2024-03-29T09:55:35.453580+0000","last_unstale":"2024-03-29T09:55:35.456273+0000","last_undegraded":"2024-03-29T09:55:35.456273+0000","last_fullsized":"2024-03-29T09:55:35.456273+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:09:14.949848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452083+0000","last_change":"2024-03-29T09:55:35.473126+0000","last_active":"2024-03-29T09:55:36.452083+0000","last_peered":"2024-03-29T09:55:36.452083+0000","last_clean":"2024-03-29T09:55:36.452083+0000","last_became_active":"2024-03-29T09:55:35.472592+0000","last_became_peered":"2024-03-29T09:55:35.472592+0000","last_unstale":"2024-03-29T09:55:36.452083+0000","last_undegraded":"2024-03-29T09:55:36.452083+0000","last_fullsized":"2024-03-29T09:55:36.452083+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:39.961948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453075+0000","last_change":"2024-03-29T09:55:35.462151+0000","last_active":"2024-03-29T09:55:36.453075+0000","last_peered":"2024-03-29T09:55:36.453075+0000","last_clean":"2024-03-29T09:55:36.453075+0000","last_became_active":"2024-03-29T09:55:35.461868+0000","last_became_peered":"2024-03-29T09:55:35.461868+0000","last_unstale":"2024-03-29T09:55:36.453075+0000","last_undegraded":"2024-03-29T09:55:36.453075+0000","last_fullsized":"2024-03-29T09:55:36.453075+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:20:12.232627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478198+0000","last_change":"2024-03-29T09:55:35.470126+0000","last_active":"2024-03-29T09:55:35.478198+0000","last_peered":"2024-03-29T09:55:35.478198+0000","last_clean":"2024-03-29T09:55:35.478198+0000","last_became_active":"2024-03-29T09:55:35.469642+0000","last_became_peered":"2024-03-29T09:55:35.469642+0000","last_unstale":"2024-03-29T09:55:35.478198+0000","last_undegraded":"2024-03-29T09:55:35.478198+0000","last_fullsized":"2024-03-29T09:55:35.478198+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:42.191643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478715+0000","last_change":"2024-03-29T09:55:35.471953+0000","last_active":"2024-03-29T09:55:35.478715+0000","last_peered":"2024-03-29T09:55:35.478715+0000","last_clean":"2024-03-29T09:55:35.478715+0000","last_became_active":"2024-03-29T09:55:35.471653+0000","last_became_peered":"2024-03-29T09:55:35.471653+0000","last_unstale":"2024-03-29T09:55:35.478715+0000","last_undegraded":"2024-03-29T09:55:35.478715+0000","last_fullsized":"2024-03-29T09:55:35.478715+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:47:12.317720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453384+0000","last_change":"2024-03-29T09:55:35.463578+0000","last_active":"2024-03-29T09:55:36.453384+0000","last_peered":"2024-03-29T09:55:36.453384+0000","last_clean":"2024-03-29T09:55:36.453384+0000","last_became_active":"2024-03-29T09:55:35.463359+0000","last_became_peered":"2024-03-29T09:55:35.463359+0000","last_unstale":"2024-03-29T09:55:36.453384+0000","last_undegraded":"2024-03-29T09:55:36.453384+0000","last_fullsized":"2024-03-29T09:55:36.453384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:27.138795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.467986+0000","last_change":"2024-03-29T09:55:35.460186+0000","last_active":"2024-03-29T09:55:35.467986+0000","last_peered":"2024-03-29T09:55:35.467986+0000","last_clean":"2024-03-29T09:55:35.467986+0000","last_became_active":"2024-03-29T09:55:35.459966+0000","last_became_peered":"2024-03-29T09:55:35.459966+0000","last_unstale":"2024-03-29T09:55:35.467986+0000","last_undegraded":"2024-03-29T09:55:35.467986+0000","last_fullsized":"2024-03-29T09:55:35.467986+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:36.964655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477717+0000","last_change":"2024-03-29T09:55:35.471369+0000","last_active":"2024-03-29T09:55:35.477717+0000","last_peered":"2024-03-29T09:55:35.477717+0000","last_clean":"2024-03-29T09:55:35.477717+0000","last_became_active":"2024-03-29T09:55:35.471016+0000","last_became_peered":"2024-03-29T09:55:35.471016+0000","last_unstale":"2024-03-29T09:55:35.477717+0000","last_undegraded":"2024-03-29T09:55:35.477717+0000","last_fullsized":"2024-03-29T09:55:35.477717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:08:40.125637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.461067+0000","last_change":"2024-03-29T09:55:35.455198+0000","last_active":"2024-03-29T09:55:35.461067+0000","last_peered":"2024-03-29T09:55:35.461067+0000","last_clean":"2024-03-29T09:55:35.461067+0000","last_became_active":"2024-03-29T09:55:35.454902+0000","last_became_peered":"2024-03-29T09:55:35.454902+0000","last_unstale":"2024-03-29T09:55:35.461067+0000","last_undegraded":"2024-03-29T09:55:35.461067+0000","last_fullsized":"2024-03-29T09:55:35.461067+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:06.711276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.468526+0000","last_change":"2024-03-29T09:55:35.463768+0000","last_active":"2024-03-29T09:55:35.468526+0000","last_peered":"2024-03-29T09:55:35.468526+0000","last_clean":"2024-03-29T09:55:35.468526+0000","last_became_active":"2024-03-29T09:55:35.461341+0000","last_became_peered":"2024-03-29T09:55:35.461341+0000","last_unstale":"2024-03-29T09:55:35.468526+0000","last_undegraded":"2024-03-29T09:55:35.468526+0000","last_fullsized":"2024-03-29T09:55:35.468526+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:59.893866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.474726+0000","last_change":"2024-03-29T09:55:35.467427+0000","last_active":"2024-03-29T09:55:35.474726+0000","last_peered":"2024-03-29T09:55:35.474726+0000","last_clean":"2024-03-29T09:55:35.474726+0000","last_became_active":"2024-03-29T09:55:35.466955+0000","last_became_peered":"2024-03-29T09:55:35.466955+0000","last_unstale":"2024-03-29T09:55:35.474726+0000","last_undegraded":"2024-03-29T09:55:35.474726+0000","last_fullsized":"2024-03-29T09:55:35.474726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:33.902566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478178+0000","last_change":"2024-03-29T09:55:35.469202+0000","last_active":"2024-03-29T09:55:35.478178+0000","last_peered":"2024-03-29T09:55:35.478178+0000","last_clean":"2024-03-29T09:55:35.478178+0000","last_became_active":"2024-03-29T09:55:35.463997+0000","last_became_peered":"2024-03-29T09:55:35.463997+0000","last_unstale":"2024-03-29T09:55:35.478178+0000","last_undegraded":"2024-03-29T09:55:35.478178+0000","last_fullsized":"2024-03-29T09:55:35.478178+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:00:59.504082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.480649+0000","last_change":"2024-03-29T09:55:35.477723+0000","last_active":"2024-03-29T09:55:35.480649+0000","last_peered":"2024-03-29T09:55:35.480649+0000","last_clean":"2024-03-29T09:55:35.480649+0000","last_became_active":"2024-03-29T09:55:35.476438+0000","last_became_peered":"2024-03-29T09:55:35.476438+0000","last_unstale":"2024-03-29T09:55:35.480649+0000","last_undegraded":"2024-03-29T09:55:35.480649+0000","last_fullsized":"2024-03-29T09:55:35.480649+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:36.716035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.469164+0000","last_change":"2024-03-29T09:55:35.459841+0000","last_active":"2024-03-29T09:55:35.469164+0000","last_peered":"2024-03-29T09:55:35.469164+0000","last_clean":"2024-03-29T09:55:35.469164+0000","last_became_active":"2024-03-29T09:55:35.459543+0000","last_became_peered":"2024-03-29T09:55:35.459543+0000","last_unstale":"2024-03-29T09:55:35.469164+0000","last_undegraded":"2024-03-29T09:55:35.469164+0000","last_fullsized":"2024-03-29T09:55:35.469164+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:39.961948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477846+0000","last_change":"2024-03-29T09:55:35.473320+0000","last_active":"2024-03-29T09:55:35.477846+0000","last_peered":"2024-03-29T09:55:35.477846+0000","last_clean":"2024-03-29T09:55:35.477846+0000","last_became_active":"2024-03-29T09:55:35.472941+0000","last_became_peered":"2024-03-29T09:55:35.472941+0000","last_unstale":"2024-03-29T09:55:35.477846+0000","last_undegraded":"2024-03-29T09:55:35.477846+0000","last_fullsized":"2024-03-29T09:55:35.477846+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:44:31.526558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452741+0000","last_change":"2024-03-29T09:55:35.459789+0000","last_active":"2024-03-29T09:55:36.452741+0000","last_peered":"2024-03-29T09:55:36.452741+0000","last_clean":"2024-03-29T09:55:36.452741+0000","last_became_active":"2024-03-29T09:55:35.459495+0000","last_became_peered":"2024-03-29T09:55:35.459495+0000","last_unstale":"2024-03-29T09:55:36.452741+0000","last_undegraded":"2024-03-29T09:55:36.452741+0000","last_fullsized":"2024-03-29T09:55:36.452741+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:08:40.125637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452775+0000","last_change":"2024-03-29T09:55:35.458995+0000","last_active":"2024-03-29T09:55:36.452775+0000","last_peered":"2024-03-29T09:55:36.452775+0000","last_clean":"2024-03-29T09:55:36.452775+0000","last_became_active":"2024-03-29T09:55:35.458702+0000","last_became_peered":"2024-03-29T09:55:35.458702+0000","last_unstale":"2024-03-29T09:55:36.452775+0000","last_undegraded":"2024-03-29T09:55:36.452775+0000","last_fullsized":"2024-03-29T09:55:36.452775+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:27.138795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452371+0000","last_change":"2024-03-29T09:55:35.466771+0000","last_active":"2024-03-29T09:55:36.452371+0000","last_peered":"2024-03-29T09:55:36.452371+0000","last_clean":"2024-03-29T09:55:36.452371+0000","last_became_active":"2024-03-29T09:55:35.466485+0000","last_became_peered":"2024-03-29T09:55:35.466485+0000","last_unstale":"2024-03-29T09:55:36.452371+0000","last_undegraded":"2024-03-29T09:55:36.452371+0000","last_fullsized":"2024-03-29T09:55:36.452371+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:01.829351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.465591+0000","last_change":"2024-03-29T09:55:35.459865+0000","last_active":"2024-03-29T09:55:35.465591+0000","last_peered":"2024-03-29T09:55:35.465591+0000","last_clean":"2024-03-29T09:55:35.465591+0000","last_became_active":"2024-03-29T09:55:35.459581+0000","last_became_peered":"2024-03-29T09:55:35.459581+0000","last_unstale":"2024-03-29T09:55:35.465591+0000","last_undegraded":"2024-03-29T09:55:35.465591+0000","last_fullsized":"2024-03-29T09:55:35.465591+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:36.964655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452656+0000","last_change":"2024-03-29T09:55:34.940588+0000","last_active":"2024-03-29T09:55:36.452656+0000","last_peered":"2024-03-29T09:55:36.452656+0000","last_clean":"2024-03-29T09:55:36.452656+0000","last_became_active":"2024-03-29T09:55:34.940108+0000","last_became_peered":"2024-03-29T09:55:34.940108+0000","last_unstale":"2024-03-29T09:55:36.452656+0000","last_undegraded":"2024-03-29T09:55:36.452656+0000","last_fullsized":"2024-03-29T09:55:36.452656+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:39.068027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452553+0000","last_change":"2024-03-29T09:55:35.464555+0000","last_active":"2024-03-29T09:55:36.452553+0000","last_peered":"2024-03-29T09:55:36.452553+0000","last_clean":"2024-03-29T09:55:36.452553+0000","last_became_active":"2024-03-29T09:55:35.463938+0000","last_became_peered":"2024-03-29T09:55:35.463938+0000","last_unstale":"2024-03-29T09:55:36.452553+0000","last_undegraded":"2024-03-29T09:55:36.452553+0000","last_fullsized":"2024-03-29T09:55:36.452553+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:13.885594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.469026+0000","last_change":"2024-03-29T09:55:34.944943+0000","last_active":"2024-03-29T09:55:35.469026+0000","last_peered":"2024-03-29T09:55:35.469026+0000","last_clean":"2024-03-29T09:55:35.469026+0000","last_became_active":"2024-03-29T09:55:34.944450+0000","last_became_peered":"2024-03-29T09:55:34.944450+0000","last_unstale":"2024-03-29T09:55:35.469026+0000","last_undegraded":"2024-03-29T09:55:35.469026+0000","last_fullsized":"2024-03-29T09:55:35.469026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:01.683697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477841+0000","last_change":"2024-03-29T09:55:35.472922+0000","last_active":"2024-03-29T09:55:35.477841+0000","last_peered":"2024-03-29T09:55:35.477841+0000","last_clean":"2024-03-29T09:55:35.477841+0000","last_became_active":"2024-03-29T09:55:35.472623+0000","last_became_peered":"2024-03-29T09:55:35.472623+0000","last_unstale":"2024-03-29T09:55:35.477841+0000","last_undegraded":"2024-03-29T09:55:35.477841+0000","last_fullsized":"2024-03-29T09:55:35.477841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:14:54.261455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.480991+0000","last_change":"2024-03-29T09:55:34.949340+0000","last_active":"2024-03-29T09:55:35.480991+0000","last_peered":"2024-03-29T09:55:35.480991+0000","last_clean":"2024-03-29T09:55:35.480991+0000","last_became_active":"2024-03-29T09:55:34.948898+0000","last_became_peered":"2024-03-29T09:55:34.948898+0000","last_unstale":"2024-03-29T09:55:35.480991+0000","last_undegraded":"2024-03-29T09:55:35.480991+0000","last_fullsized":"2024-03-29T09:55:35.480991+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:50:30.442405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.481257+0000","last_change":"2024-03-29T09:55:35.478699+0000","last_active":"2024-03-29T09:55:35.481257+0000","last_peered":"2024-03-29T09:55:35.481257+0000","last_clean":"2024-03-29T09:55:35.481257+0000","last_became_active":"2024-03-29T09:55:35.477819+0000","last_became_peered":"2024-03-29T09:55:35.477819+0000","last_unstale":"2024-03-29T09:55:35.481257+0000","last_undegraded":"2024-03-29T09:55:35.481257+0000","last_fullsized":"2024-03-29T09:55:35.481257+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:49:22.165389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453463+0000","last_change":"2024-03-29T09:55:34.941607+0000","last_active":"2024-03-29T09:55:36.453463+0000","last_peered":"2024-03-29T09:55:36.453463+0000","last_clean":"2024-03-29T09:55:36.453463+0000","last_became_active":"2024-03-29T09:55:34.940784+0000","last_became_peered":"2024-03-29T09:55:34.940784+0000","last_unstale":"2024-03-29T09:55:36.453463+0000","last_undegraded":"2024-03-29T09:55:36.453463+0000","last_fullsized":"2024-03-29T09:55:36.453463+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:43.371924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452781+0000","last_change":"2024-03-29T09:55:35.473648+0000","last_active":"2024-03-29T09:55:36.452781+0000","last_peered":"2024-03-29T09:55:36.452781+0000","last_clean":"2024-03-29T09:55:36.452781+0000","last_became_active":"2024-03-29T09:55:35.473366+0000","last_became_peered":"2024-03-29T09:55:35.473366+0000","last_unstale":"2024-03-29T09:55:36.452781+0000","last_undegraded":"2024-03-29T09:55:36.452781+0000","last_fullsized":"2024-03-29T09:55:36.452781+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:16:25.336942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.479100+0000","last_change":"2024-03-29T09:55:34.947769+0000","last_active":"2024-03-29T09:55:35.479100+0000","last_peered":"2024-03-29T09:55:35.479100+0000","last_clean":"2024-03-29T09:55:35.479100+0000","last_became_active":"2024-03-29T09:55:34.947520+0000","last_became_peered":"2024-03-29T09:55:34.947520+0000","last_unstale":"2024-03-29T09:55:35.479100+0000","last_undegraded":"2024-03-29T09:55:35.479100+0000","last_fullsized":"2024-03-29T09:55:35.479100+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.781386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.468431+0000","last_change":"2024-03-29T09:55:35.458483+0000","last_active":"2024-03-29T09:55:35.468431+0000","last_peered":"2024-03-29T09:55:35.468431+0000","last_clean":"2024-03-29T09:55:35.468431+0000","last_became_active":"2024-03-29T09:55:35.458210+0000","last_became_peered":"2024-03-29T09:55:35.458210+0000","last_unstale":"2024-03-29T09:55:35.468431+0000","last_undegraded":"2024-03-29T09:55:35.468431+0000","last_fullsized":"2024-03-29T09:55:35.468431+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:50.835509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.479085+0000","last_change":"2024-03-29T09:55:34.947855+0000","last_active":"2024-03-29T09:55:35.479085+0000","last_peered":"2024-03-29T09:55:35.479085+0000","last_clean":"2024-03-29T09:55:35.479085+0000","last_became_active":"2024-03-29T09:55:34.947601+0000","last_became_peered":"2024-03-29T09:55:34.947601+0000","last_unstale":"2024-03-29T09:55:35.479085+0000","last_undegraded":"2024-03-29T09:55:35.479085+0000","last_fullsized":"2024-03-29T09:55:35.479085+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:22.913621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475275+0000","last_change":"2024-03-29T09:55:35.470802+0000","last_active":"2024-03-29T09:55:35.475275+0000","last_peered":"2024-03-29T09:55:35.475275+0000","last_clean":"2024-03-29T09:55:35.475275+0000","last_became_active":"2024-03-29T09:55:35.470534+0000","last_became_peered":"2024-03-29T09:55:35.470534+0000","last_unstale":"2024-03-29T09:55:35.475275+0000","last_undegraded":"2024-03-29T09:55:35.475275+0000","last_fullsized":"2024-03-29T09:55:35.475275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:06.711276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.445329+0000","last_change":"2024-03-29T09:55:34.947192+0000","last_active":"2024-03-29T09:55:35.445329+0000","last_peered":"2024-03-29T09:55:35.445329+0000","last_clean":"2024-03-29T09:55:35.445329+0000","last_became_active":"2024-03-29T09:55:34.946034+0000","last_became_peered":"2024-03-29T09:55:34.946034+0000","last_unstale":"2024-03-29T09:55:35.445329+0000","last_undegraded":"2024-03-29T09:55:35.445329+0000","last_fullsized":"2024-03-29T09:55:35.445329+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:28:48.871162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.474421+0000","last_change":"2024-03-29T09:55:35.466599+0000","last_active":"2024-03-29T09:55:35.474421+0000","last_peered":"2024-03-29T09:55:35.474421+0000","last_clean":"2024-03-29T09:55:35.474421+0000","last_became_active":"2024-03-29T09:55:35.466276+0000","last_became_peered":"2024-03-29T09:55:35.466276+0000","last_unstale":"2024-03-29T09:55:35.474421+0000","last_undegraded":"2024-03-29T09:55:35.474421+0000","last_fullsized":"2024-03-29T09:55:35.474421+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:13.885594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.454732+0000","last_change":"2024-03-29T09:55:34.934895+0000","last_active":"2024-03-29T09:55:35.454732+0000","last_peered":"2024-03-29T09:55:35.454732+0000","last_clean":"2024-03-29T09:55:35.454732+0000","last_became_active":"2024-03-29T09:55:34.933758+0000","last_became_peered":"2024-03-29T09:55:34.933758+0000","last_unstale":"2024-03-29T09:55:35.454732+0000","last_undegraded":"2024-03-29T09:55:35.454732+0000","last_fullsized":"2024-03-29T09:55:35.454732+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:10:16.221815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475809+0000","last_change":"2024-03-29T09:55:35.468023+0000","last_active":"2024-03-29T09:55:35.475809+0000","last_peered":"2024-03-29T09:55:35.475809+0000","last_clean":"2024-03-29T09:55:35.475809+0000","last_became_active":"2024-03-29T09:55:35.467702+0000","last_became_peered":"2024-03-29T09:55:35.467702+0000","last_unstale":"2024-03-29T09:55:35.475809+0000","last_undegraded":"2024-03-29T09:55:35.475809+0000","last_fullsized":"2024-03-29T09:55:35.475809+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:20:04.154117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.449003+0000","last_change":"2024-03-29T09:55:34.946594+0000","last_active":"2024-03-29T09:55:35.449003+0000","last_peered":"2024-03-29T09:55:35.449003+0000","last_clean":"2024-03-29T09:55:35.449003+0000","last_became_active":"2024-03-29T09:55:34.946263+0000","last_became_peered":"2024-03-29T09:55:34.946263+0000","last_unstale":"2024-03-29T09:55:35.449003+0000","last_undegraded":"2024-03-29T09:55:35.449003+0000","last_fullsized":"2024-03-29T09:55:35.449003+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:32.894992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477883+0000","last_change":"2024-03-29T09:55:35.471254+0000","last_active":"2024-03-29T09:55:35.477883+0000","last_peered":"2024-03-29T09:55:35.477883+0000","last_clean":"2024-03-29T09:55:35.477883+0000","last_became_active":"2024-03-29T09:55:35.471007+0000","last_became_peered":"2024-03-29T09:55:35.471007+0000","last_unstale":"2024-03-29T09:55:35.477883+0000","last_undegraded":"2024-03-29T09:55:35.477883+0000","last_fullsized":"2024-03-29T09:55:35.477883+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:20:04.154117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.451723+0000","last_change":"2024-03-29T09:55:34.950126+0000","last_active":"2024-03-29T09:55:35.451723+0000","last_peered":"2024-03-29T09:55:35.451723+0000","last_clean":"2024-03-29T09:55:35.451723+0000","last_became_active":"2024-03-29T09:55:34.949434+0000","last_became_peered":"2024-03-29T09:55:34.949434+0000","last_unstale":"2024-03-29T09:55:35.451723+0000","last_undegraded":"2024-03-29T09:55:35.451723+0000","last_fullsized":"2024-03-29T09:55:35.451723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:26.131220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477346+0000","last_change":"2024-03-29T09:55:35.468307+0000","last_active":"2024-03-29T09:55:35.477346+0000","last_peered":"2024-03-29T09:55:35.477346+0000","last_clean":"2024-03-29T09:55:35.477346+0000","last_became_active":"2024-03-29T09:55:35.468014+0000","last_became_peered":"2024-03-29T09:55:35.468014+0000","last_unstale":"2024-03-29T09:55:35.477346+0000","last_undegraded":"2024-03-29T09:55:35.477346+0000","last_fullsized":"2024-03-29T09:55:35.477346+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:08:59.620621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478025+0000","last_change":"2024-03-29T09:55:34.944741+0000","last_active":"2024-03-29T09:55:35.478025+0000","last_peered":"2024-03-29T09:55:35.478025+0000","last_clean":"2024-03-29T09:55:35.478025+0000","last_became_active":"2024-03-29T09:55:34.944219+0000","last_became_peered":"2024-03-29T09:55:34.944219+0000","last_unstale":"2024-03-29T09:55:35.478025+0000","last_undegraded":"2024-03-29T09:55:35.478025+0000","last_fullsized":"2024-03-29T09:55:35.478025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:12.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475729+0000","last_change":"2024-03-29T09:55:35.466268+0000","last_active":"2024-03-29T09:55:35.475729+0000","last_peered":"2024-03-29T09:55:35.475729+0000","last_clean":"2024-03-29T09:55:35.475729+0000","last_became_active":"2024-03-29T09:55:35.465694+0000","last_became_peered":"2024-03-29T09:55:35.465694+0000","last_unstale":"2024-03-29T09:55:35.475729+0000","last_undegraded":"2024-03-29T09:55:35.475729+0000","last_fullsized":"2024-03-29T09:55:35.475729+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:01.829351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.483492+0000","last_change":"2024-03-29T09:55:34.949317+0000","last_active":"2024-03-29T09:55:35.483492+0000","last_peered":"2024-03-29T09:55:35.483492+0000","last_clean":"2024-03-29T09:55:35.483492+0000","last_became_active":"2024-03-29T09:55:34.948798+0000","last_became_peered":"2024-03-29T09:55:34.948798+0000","last_unstale":"2024-03-29T09:55:35.483492+0000","last_undegraded":"2024-03-29T09:55:35.483492+0000","last_fullsized":"2024-03-29T09:55:35.483492+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:23.265666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475934+0000","last_change":"2024-03-29T09:55:35.467623+0000","last_active":"2024-03-29T09:55:35.475934+0000","last_peered":"2024-03-29T09:55:35.475934+0000","last_clean":"2024-03-29T09:55:35.475934+0000","last_became_active":"2024-03-29T09:55:35.467160+0000","last_became_peered":"2024-03-29T09:55:35.467160+0000","last_unstale":"2024-03-29T09:55:35.475934+0000","last_undegraded":"2024-03-29T09:55:35.475934+0000","last_fullsized":"2024-03-29T09:55:35.475934+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:40:33.535101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.473728+0000","last_change":"2024-03-29T09:55:34.942233+0000","last_active":"2024-03-29T09:55:35.473728+0000","last_peered":"2024-03-29T09:55:35.473728+0000","last_clean":"2024-03-29T09:55:35.473728+0000","last_became_active":"2024-03-29T09:55:34.941182+0000","last_became_peered":"2024-03-29T09:55:34.941182+0000","last_unstale":"2024-03-29T09:55:35.473728+0000","last_undegraded":"2024-03-29T09:55:35.473728+0000","last_fullsized":"2024-03-29T09:55:35.473728+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:50:39.393444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.468668+0000","last_change":"2024-03-29T09:55:35.457426+0000","last_active":"2024-03-29T09:55:35.468668+0000","last_peered":"2024-03-29T09:55:35.468668+0000","last_clean":"2024-03-29T09:55:35.468668+0000","last_became_active":"2024-03-29T09:55:35.457168+0000","last_became_peered":"2024-03-29T09:55:35.457168+0000","last_unstale":"2024-03-29T09:55:35.468668+0000","last_undegraded":"2024-03-29T09:55:35.468668+0000","last_fullsized":"2024-03-29T09:55:35.468668+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:50.835509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.457512+0000","last_change":"2024-03-29T09:55:34.933086+0000","last_active":"2024-03-29T09:55:35.457512+0000","last_peered":"2024-03-29T09:55:35.457512+0000","last_clean":"2024-03-29T09:55:35.457512+0000","last_became_active":"2024-03-29T09:55:34.932742+0000","last_became_peered":"2024-03-29T09:55:34.932742+0000","last_unstale":"2024-03-29T09:55:35.457512+0000","last_undegraded":"2024-03-29T09:55:35.457512+0000","last_fullsized":"2024-03-29T09:55:35.457512+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:22.913621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452265+0000","last_change":"2024-03-29T09:55:35.474578+0000","last_active":"2024-03-29T09:55:36.452265+0000","last_peered":"2024-03-29T09:55:36.452265+0000","last_clean":"2024-03-29T09:55:36.452265+0000","last_became_active":"2024-03-29T09:55:35.474051+0000","last_became_peered":"2024-03-29T09:55:35.474051+0000","last_unstale":"2024-03-29T09:55:36.452265+0000","last_undegraded":"2024-03-29T09:55:36.452265+0000","last_fullsized":"2024-03-29T09:55:36.452265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:59.893866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.445431+0000","last_change":"2024-03-29T09:55:34.947164+0000","last_active":"2024-03-29T09:55:35.445431+0000","last_peered":"2024-03-29T09:55:35.445431+0000","last_clean":"2024-03-29T09:55:35.445431+0000","last_became_active":"2024-03-29T09:55:34.946483+0000","last_became_peered":"2024-03-29T09:55:34.946483+0000","last_unstale":"2024-03-29T09:55:35.445431+0000","last_undegraded":"2024-03-29T09:55:35.445431+0000","last_fullsized":"2024-03-29T09:55:35.445431+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:49:21.157814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452390+0000","last_change":"2024-03-29T09:55:35.458353+0000","last_active":"2024-03-29T09:55:36.452390+0000","last_peered":"2024-03-29T09:55:36.452390+0000","last_clean":"2024-03-29T09:55:36.452390+0000","last_became_active":"2024-03-29T09:55:35.456888+0000","last_became_peered":"2024-03-29T09:55:35.456888+0000","last_unstale":"2024-03-29T09:55:36.452390+0000","last_undegraded":"2024-03-29T09:55:36.452390+0000","last_fullsized":"2024-03-29T09:55:36.452390+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:24.273241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.481922+0000","last_change":"2024-03-29T09:55:34.941832+0000","last_active":"2024-03-29T09:55:35.481922+0000","last_peered":"2024-03-29T09:55:35.481922+0000","last_clean":"2024-03-29T09:55:35.481922+0000","last_became_active":"2024-03-29T09:55:34.941309+0000","last_became_peered":"2024-03-29T09:55:34.941309+0000","last_unstale":"2024-03-29T09:55:35.481922+0000","last_undegraded":"2024-03-29T09:55:35.481922+0000","last_fullsized":"2024-03-29T09:55:35.481922+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:40:32.527526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.468145+0000","last_change":"2024-03-29T09:55:35.457654+0000","last_active":"2024-03-29T09:55:35.468145+0000","last_peered":"2024-03-29T09:55:35.468145+0000","last_clean":"2024-03-29T09:55:35.468145+0000","last_became_active":"2024-03-29T09:55:35.457396+0000","last_became_peered":"2024-03-29T09:55:35.457396+0000","last_unstale":"2024-03-29T09:55:35.468145+0000","last_undegraded":"2024-03-29T09:55:35.468145+0000","last_fullsized":"2024-03-29T09:55:35.468145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:28:49.878736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478893+0000","last_change":"2024-03-29T09:55:34.943595+0000","last_active":"2024-03-29T09:55:35.478893+0000","last_peered":"2024-03-29T09:55:35.478893+0000","last_clean":"2024-03-29T09:55:35.478893+0000","last_became_active":"2024-03-29T09:55:34.943312+0000","last_became_peered":"2024-03-29T09:55:34.943312+0000","last_unstale":"2024-03-29T09:55:35.478893+0000","last_undegraded":"2024-03-29T09:55:35.478893+0000","last_fullsized":"2024-03-29T09:55:35.478893+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:06.263255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.469068+0000","last_change":"2024-03-29T09:55:35.463198+0000","last_active":"2024-03-29T09:55:35.469068+0000","last_peered":"2024-03-29T09:55:35.469068+0000","last_clean":"2024-03-29T09:55:35.469068+0000","last_became_active":"2024-03-29T09:55:35.460798+0000","last_became_peered":"2024-03-29T09:55:35.460798+0000","last_unstale":"2024-03-29T09:55:35.469068+0000","last_undegraded":"2024-03-29T09:55:35.469068+0000","last_fullsized":"2024-03-29T09:55:35.469068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:27:17.579514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.473736+0000","last_change":"2024-03-29T09:55:34.941454+0000","last_active":"2024-03-29T09:55:35.473736+0000","last_peered":"2024-03-29T09:55:35.473736+0000","last_clean":"2024-03-29T09:55:35.473736+0000","last_became_active":"2024-03-29T09:55:34.941187+0000","last_became_peered":"2024-03-29T09:55:34.941187+0000","last_unstale":"2024-03-29T09:55:35.473736+0000","last_undegraded":"2024-03-29T09:55:35.473736+0000","last_fullsized":"2024-03-29T09:55:35.473736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:39.068027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.481309+0000","last_change":"2024-03-29T09:55:35.479712+0000","last_active":"2024-03-29T09:55:35.481309+0000","last_peered":"2024-03-29T09:55:35.481309+0000","last_clean":"2024-03-29T09:55:35.481309+0000","last_became_active":"2024-03-29T09:55:35.479404+0000","last_became_peered":"2024-03-29T09:55:35.479404+0000","last_unstale":"2024-03-29T09:55:35.481309+0000","last_undegraded":"2024-03-29T09:55:35.481309+0000","last_fullsized":"2024-03-29T09:55:35.481309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:36.964655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475555+0000","last_change":"2024-03-29T09:55:34.938571+0000","last_active":"2024-03-29T09:55:35.475555+0000","last_peered":"2024-03-29T09:55:35.475555+0000","last_clean":"2024-03-29T09:55:35.475555+0000","last_became_active":"2024-03-29T09:55:34.937776+0000","last_became_peered":"2024-03-29T09:55:34.937776+0000","last_unstale":"2024-03-29T09:55:35.475555+0000","last_undegraded":"2024-03-29T09:55:35.475555+0000","last_fullsized":"2024-03-29T09:55:35.475555+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:06.263255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453539+0000","last_change":"2024-03-29T09:55:35.460515+0000","last_active":"2024-03-29T09:55:36.453539+0000","last_peered":"2024-03-29T09:55:36.453539+0000","last_clean":"2024-03-29T09:55:36.453539+0000","last_became_active":"2024-03-29T09:55:35.460266+0000","last_became_peered":"2024-03-29T09:55:35.460266+0000","last_unstale":"2024-03-29T09:55:36.453539+0000","last_undegraded":"2024-03-29T09:55:36.453539+0000","last_fullsized":"2024-03-29T09:55:36.453539+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:50:31.449980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.461204+0000","last_change":"2024-03-29T09:55:34.948028+0000","last_active":"2024-03-29T09:55:35.461204+0000","last_peered":"2024-03-29T09:55:35.461204+0000","last_clean":"2024-03-29T09:55:35.461204+0000","last_became_active":"2024-03-29T09:55:34.947688+0000","last_became_peered":"2024-03-29T09:55:34.947688+0000","last_unstale":"2024-03-29T09:55:35.461204+0000","last_undegraded":"2024-03-29T09:55:35.461204+0000","last_fullsized":"2024-03-29T09:55:35.461204+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:49:47.226070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453336+0000","last_change":"2024-03-29T09:55:35.468390+0000","last_active":"2024-03-29T09:55:36.453336+0000","last_peered":"2024-03-29T09:55:36.453336+0000","last_clean":"2024-03-29T09:55:36.453336+0000","last_became_active":"2024-03-29T09:55:35.468148+0000","last_became_peered":"2024-03-29T09:55:35.468148+0000","last_unstale":"2024-03-29T09:55:36.453336+0000","last_undegraded":"2024-03-29T09:55:36.453336+0000","last_fullsized":"2024-03-29T09:55:36.453336+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:34.798479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.473579+0000","last_change":"2024-03-29T09:55:34.944643+0000","last_active":"2024-03-29T09:55:35.473579+0000","last_peered":"2024-03-29T09:55:35.473579+0000","last_clean":"2024-03-29T09:55:35.473579+0000","last_became_active":"2024-03-29T09:55:34.943984+0000","last_became_peered":"2024-03-29T09:55:34.943984+0000","last_unstale":"2024-03-29T09:55:35.473579+0000","last_undegraded":"2024-03-29T09:55:35.473579+0000","last_fullsized":"2024-03-29T09:55:35.473579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:12.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453702+0000","last_change":"2024-03-29T09:55:35.466473+0000","last_active":"2024-03-29T09:55:36.453702+0000","last_peered":"2024-03-29T09:55:36.453702+0000","last_clean":"2024-03-29T09:55:36.453702+0000","last_became_active":"2024-03-29T09:55:35.466216+0000","last_became_peered":"2024-03-29T09:55:35.466216+0000","last_unstale":"2024-03-29T09:55:36.453702+0000","last_undegraded":"2024-03-29T09:55:36.453702+0000","last_fullsized":"2024-03-29T09:55:36.453702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:40:33.535101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452541+0000","last_change":"2024-03-29T09:55:34.942491+0000","last_active":"2024-03-29T09:55:36.452541+0000","last_peered":"2024-03-29T09:55:36.452541+0000","last_clean":"2024-03-29T09:55:36.452541+0000","last_became_active":"2024-03-29T09:55:34.941777+0000","last_became_peered":"2024-03-29T09:55:34.941777+0000","last_unstale":"2024-03-29T09:55:36.452541+0000","last_undegraded":"2024-03-29T09:55:36.452541+0000","last_fullsized":"2024-03-29T09:55:36.452541+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:10:16.221815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453018+0000","last_change":"2024-03-29T09:55:35.458455+0000","last_active":"2024-03-29T09:55:36.453018+0000","last_peered":"2024-03-29T09:55:36.453018+0000","last_clean":"2024-03-29T09:55:36.453018+0000","last_became_active":"2024-03-29T09:55:35.458190+0000","last_became_peered":"2024-03-29T09:55:35.458190+0000","last_unstale":"2024-03-29T09:55:36.453018+0000","last_undegraded":"2024-03-29T09:55:36.453018+0000","last_fullsized":"2024-03-29T09:55:36.453018+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:13.885594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.454939+0000","last_change":"2024-03-29T09:55:34.942208+0000","last_active":"2024-03-29T09:55:35.454939+0000","last_peered":"2024-03-29T09:55:35.454939+0000","last_clean":"2024-03-29T09:55:35.454939+0000","last_became_active":"2024-03-29T09:55:34.941888+0000","last_became_peered":"2024-03-29T09:55:34.941888+0000","last_unstale":"2024-03-29T09:55:35.454939+0000","last_undegraded":"2024-03-29T09:55:35.454939+0000","last_fullsized":"2024-03-29T09:55:35.454939+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:41:49.708910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452809+0000","last_change":"2024-03-29T09:55:35.462110+0000","last_active":"2024-03-29T09:55:36.452809+0000","last_peered":"2024-03-29T09:55:36.452809+0000","last_clean":"2024-03-29T09:55:36.452809+0000","last_became_active":"2024-03-29T09:55:35.461820+0000","last_became_peered":"2024-03-29T09:55:35.461820+0000","last_unstale":"2024-03-29T09:55:36.452809+0000","last_undegraded":"2024-03-29T09:55:36.452809+0000","last_fullsized":"2024-03-29T09:55:36.452809+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:33:34.801403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.444737+0000","last_change":"2024-03-29T09:55:34.944551+0000","last_active":"2024-03-29T09:55:35.444737+0000","last_peered":"2024-03-29T09:55:35.444737+0000","last_clean":"2024-03-29T09:55:35.444737+0000","last_became_active":"2024-03-29T09:55:34.944214+0000","last_became_peered":"2024-03-29T09:55:34.944214+0000","last_unstale":"2024-03-29T09:55:35.444737+0000","last_undegraded":"2024-03-29T09:55:35.444737+0000","last_fullsized":"2024-03-29T09:55:35.444737+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:50:30.442405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.453466+0000","last_change":"2024-03-29T09:55:35.464636+0000","last_active":"2024-03-29T09:55:36.453466+0000","last_peered":"2024-03-29T09:55:36.453466+0000","last_clean":"2024-03-29T09:55:36.453466+0000","last_became_active":"2024-03-29T09:55:35.463304+0000","last_became_peered":"2024-03-29T09:55:35.463304+0000","last_unstale":"2024-03-29T09:55:36.453466+0000","last_undegraded":"2024-03-29T09:55:36.453466+0000","last_fullsized":"2024-03-29T09:55:36.453466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:24.273241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.470728+0000","last_change":"2024-03-29T09:55:34.936806+0000","last_active":"2024-03-29T09:55:35.470728+0000","last_peered":"2024-03-29T09:55:35.470728+0000","last_clean":"2024-03-29T09:55:35.470728+0000","last_became_active":"2024-03-29T09:55:34.936552+0000","last_became_peered":"2024-03-29T09:55:34.936552+0000","last_unstale":"2024-03-29T09:55:35.470728+0000","last_undegraded":"2024-03-29T09:55:35.470728+0000","last_fullsized":"2024-03-29T09:55:35.470728+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:10:16.221815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.475860+0000","last_change":"2024-03-29T09:55:35.469638+0000","last_active":"2024-03-29T09:55:35.475860+0000","last_peered":"2024-03-29T09:55:35.475860+0000","last_clean":"2024-03-29T09:55:35.475860+0000","last_became_active":"2024-03-29T09:55:35.469375+0000","last_became_peered":"2024-03-29T09:55:35.469375+0000","last_unstale":"2024-03-29T09:55:35.475860+0000","last_undegraded":"2024-03-29T09:55:35.475860+0000","last_fullsized":"2024-03-29T09:55:35.475860+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:33:34.801403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.451817+0000","last_change":"2024-03-29T09:55:34.944733+0000","last_active":"2024-03-29T09:55:35.451817+0000","last_peered":"2024-03-29T09:55:35.451817+0000","last_clean":"2024-03-29T09:55:35.451817+0000","last_became_active":"2024-03-29T09:55:34.944161+0000","last_became_peered":"2024-03-29T09:55:34.944161+0000","last_unstale":"2024-03-29T09:55:35.451817+0000","last_undegraded":"2024-03-29T09:55:35.451817+0000","last_fullsized":"2024-03-29T09:55:35.451817+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:23.265666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.469074+0000","last_change":"2024-03-29T09:55:35.464745+0000","last_active":"2024-03-29T09:55:35.469074+0000","last_peered":"2024-03-29T09:55:35.469074+0000","last_clean":"2024-03-29T09:55:35.469074+0000","last_became_active":"2024-03-29T09:55:35.464410+0000","last_became_peered":"2024-03-29T09:55:35.464410+0000","last_unstale":"2024-03-29T09:55:35.469074+0000","last_undegraded":"2024-03-29T09:55:35.469074+0000","last_fullsized":"2024-03-29T09:55:35.469074+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:28:49.878736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.458594+0000","last_change":"2024-03-29T09:55:34.935319+0000","last_active":"2024-03-29T09:55:35.458594+0000","last_peered":"2024-03-29T09:55:35.458594+0000","last_clean":"2024-03-29T09:55:35.458594+0000","last_became_active":"2024-03-29T09:55:34.934799+0000","last_became_peered":"2024-03-29T09:55:34.934799+0000","last_unstale":"2024-03-29T09:55:35.458594+0000","last_undegraded":"2024-03-29T09:55:35.458594+0000","last_fullsized":"2024-03-29T09:55:35.458594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:28.148256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452491+0000","last_change":"2024-03-29T09:55:35.458803+0000","last_active":"2024-03-29T09:55:36.452491+0000","last_peered":"2024-03-29T09:55:36.452491+0000","last_clean":"2024-03-29T09:55:36.452491+0000","last_became_active":"2024-03-29T09:55:35.456734+0000","last_became_peered":"2024-03-29T09:55:35.456734+0000","last_unstale":"2024-03-29T09:55:36.452491+0000","last_undegraded":"2024-03-29T09:55:36.452491+0000","last_fullsized":"2024-03-29T09:55:36.452491+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:50:31.449980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.450005+0000","last_change":"2024-03-29T09:55:34.944828+0000","last_active":"2024-03-29T09:55:35.450005+0000","last_peered":"2024-03-29T09:55:35.450005+0000","last_clean":"2024-03-29T09:55:35.450005+0000","last_became_active":"2024-03-29T09:55:34.944367+0000","last_became_peered":"2024-03-29T09:55:34.944367+0000","last_unstale":"2024-03-29T09:55:35.450005+0000","last_undegraded":"2024-03-29T09:55:35.450005+0000","last_fullsized":"2024-03-29T09:55:35.450005+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:41:49.708910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.469504+0000","last_change":"2024-03-29T09:55:35.461679+0000","last_active":"2024-03-29T09:55:35.469504+0000","last_peered":"2024-03-29T09:55:35.469504+0000","last_clean":"2024-03-29T09:55:35.469504+0000","last_became_active":"2024-03-29T09:55:35.460935+0000","last_became_peered":"2024-03-29T09:55:35.460935+0000","last_unstale":"2024-03-29T09:55:35.469504+0000","last_undegraded":"2024-03-29T09:55:35.469504+0000","last_fullsized":"2024-03-29T09:55:35.469504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:14:54.261455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.457637+0000","last_change":"2024-03-29T09:55:34.931195+0000","last_active":"2024-03-29T09:55:35.457637+0000","last_peered":"2024-03-29T09:55:35.457637+0000","last_clean":"2024-03-29T09:55:35.457637+0000","last_became_active":"2024-03-29T09:55:34.930885+0000","last_became_peered":"2024-03-29T09:55:34.930885+0000","last_unstale":"2024-03-29T09:55:35.457637+0000","last_undegraded":"2024-03-29T09:55:35.457637+0000","last_fullsized":"2024-03-29T09:55:35.457637+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:39.068027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.478771+0000","last_change":"2024-03-29T09:55:35.472255+0000","last_active":"2024-03-29T09:55:35.478771+0000","last_peered":"2024-03-29T09:55:35.478771+0000","last_clean":"2024-03-29T09:55:35.478771+0000","last_became_active":"2024-03-29T09:55:35.471964+0000","last_became_peered":"2024-03-29T09:55:35.471964+0000","last_unstale":"2024-03-29T09:55:35.478771+0000","last_undegraded":"2024-03-29T09:55:35.478771+0000","last_fullsized":"2024-03-29T09:55:35.478771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:45:39.961948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.455006+0000","last_change":"2024-03-29T09:55:34.935632+0000","last_active":"2024-03-29T09:55:35.455006+0000","last_peered":"2024-03-29T09:55:35.455006+0000","last_clean":"2024-03-29T09:55:35.455006+0000","last_became_active":"2024-03-29T09:55:34.935326+0000","last_became_peered":"2024-03-29T09:55:34.935326+0000","last_unstale":"2024-03-29T09:55:35.455006+0000","last_undegraded":"2024-03-29T09:55:35.455006+0000","last_fullsized":"2024-03-29T09:55:35.455006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:28.148256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.477733+0000","last_change":"2024-03-29T09:55:35.471480+0000","last_active":"2024-03-29T09:55:35.477733+0000","last_peered":"2024-03-29T09:55:35.477733+0000","last_clean":"2024-03-29T09:55:35.477733+0000","last_became_active":"2024-03-29T09:55:35.471165+0000","last_became_peered":"2024-03-29T09:55:35.471165+0000","last_unstale":"2024-03-29T09:55:35.477733+0000","last_undegraded":"2024-03-29T09:55:35.477733+0000","last_fullsized":"2024-03-29T09:55:35.477733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:20:12.232627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.443725+0000","last_change":"2024-03-29T09:55:34.947995+0000","last_active":"2024-03-29T09:55:35.443725+0000","last_peered":"2024-03-29T09:55:35.443725+0000","last_clean":"2024-03-29T09:55:35.443725+0000","last_became_active":"2024-03-29T09:55:34.947623+0000","last_became_peered":"2024-03-29T09:55:34.947623+0000","last_unstale":"2024-03-29T09:55:35.443725+0000","last_undegraded":"2024-03-29T09:55:35.443725+0000","last_fullsized":"2024-03-29T09:55:35.443725+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:35.708460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452773+0000","last_change":"2024-03-29T09:55:35.466153+0000","last_active":"2024-03-29T09:55:36.452773+0000","last_peered":"2024-03-29T09:55:36.452773+0000","last_clean":"2024-03-29T09:55:36.452773+0000","last_became_active":"2024-03-29T09:55:35.464346+0000","last_became_peered":"2024-03-29T09:55:35.464346+0000","last_unstale":"2024-03-29T09:55:36.452773+0000","last_undegraded":"2024-03-29T09:55:36.452773+0000","last_fullsized":"2024-03-29T09:55:36.452773+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:34.915037+0000","last_clean_scrub_stamp":"2024-03-29T09:55:34.915037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:36.716035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-29T09:55:36.452411+0000","last_change":"2024-03-29T09:55:34.938938+0000","last_active":"2024-03-29T09:55:36.452411+0000","last_peered":"2024-03-29T09:55:36.452411+0000","last_clean":"2024-03-29T09:55:36.452411+0000","last_became_active":"2024-03-29T09:55:34.938456+0000","last_became_peered":"2024-03-29T09:55:34.938456+0000","last_unstale":"2024-03-29T09:55:36.452411+0000","last_undegraded":"2024-03-29T09:55:36.452411+0000","last_fullsized":"2024-03-29T09:55:36.452411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:55:33.907462+0000","last_clean_scrub_stamp":"2024-03-29T09:55:33.907462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:22.913621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-29T09:55:35.449414+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T09:55:35.449414+0000","last_peered":"2024-03-29T09:55:35.449414+0000","last_clean":"2024-03-29T09:55:35.449414+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T09:55:35.449414+0000","last_undegraded":"2024-03-29T09:55:35.449414+0000","last_fullsized":"2024-03-29T09:55:35.449414+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":970752,"data_stored":736215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.748},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.51300000000000001}]},{"osd":1,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71399999999999997}]},{"osd":2,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.58599999999999997}]},{"osd":4,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.70199999999999996}]},{"osd":6,"last update":"Fri Mar 29 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.72299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":6.7610000000000001,"5min":6.7610000000000001,"15min":6.7610000000000001},"last":0.61899999999999999}]},{"osd":1,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.58499999999999996}]},{"osd":2,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.499},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":0.81399999999999995}]},{"osd":3,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":3.1960000000000002,"5min":3.1960000000000002,"15min":3.1960000000000002},"last":0.745},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":4.3760000000000003,"5min":4.3760000000000003,"15min":4.3760000000000003},"last":0.65400000000000003}]},{"osd":4,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":0.79400000000000004}]},{"osd":5,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.8279999999999998,"5min":2.8279999999999998,"15min":2.8279999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.72699999999999998}]},{"osd":7,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.7869999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.76500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":147343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.316},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.38800000000000001}]},{"osd":2,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.282,"5min":4.282,"15min":4.282},"last":0.35199999999999998}]},{"osd":3,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.3200000000000003,"5min":4.3200000000000003,"15min":4.3200000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.44700000000000001}]},{"osd":4,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"last":0.92200000000000004}]},{"osd":5,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":0.84099999999999997}]},{"osd":6,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":4.3899999999999997,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":0.874},{"interface":"front","average":{"1min":0.877,"5min":0.877,"15min":0.877},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.93300000000000005}]},{"osd":7,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.81799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":962560,"data_stored":735106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.44800000000000001}]},{"osd":2,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52400000000000002}]},{"osd":4,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":1.1060000000000001}]},{"osd":5,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.627},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.66900000000000004}]},{"osd":7,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":1.0049999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":145827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.435},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.33100000000000002}]},{"osd":4,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.53100000000000003}]},{"osd":5,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.56299999999999994}]},{"osd":6,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.67600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":147343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.3799999999999999,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.1840000000000002,"5min":4.1840000000000002,"15min":4.1840000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.314}]},{"osd":2,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.5970000000000004,"5min":4.5970000000000004,"15min":4.5970000000000004},"last":0.56399999999999995}]},{"osd":4,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.78400000000000003}]},{"osd":5,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.4180000000000001,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.81899999999999995}]},{"osd":7,"last update":"Fri Mar 29 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":4.4550000000000001,"5min":4.4550000000000001,"15min":4.4550000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.70099999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":145861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.64800000000000002}]},{"osd":1,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.59399999999999997}]},{"osd":5,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.621},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.61199999999999999}]},{"osd":6,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Mar 29 09:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.97,"5min":1.97,"15min":1.97},"last":0.66600000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.58899999999999997}]},{"osd":1,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.622}]},{"osd":2,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.47899999999999998}]},{"osd":3,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.67300000000000004}]},{"osd":4,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.65500000000000003}]},{"osd":6,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.752},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.77200000000000002}]},{"osd":7,"last update":"Fri Mar 29 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.71399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T09:55:40.867 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T09:55:40.867 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T09:55:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.108+0000 7fdddfc00700 1 Processor -- start 2024-03-29T09:55:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.108+0000 7fdddfc00700 1 -- start start 2024-03-29T09:55:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.109+0000 7fdddfc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd80cee10 0x7fddd8161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.109+0000 7fdddfc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd80ce270 con 0x7fddd81611b0 2024-03-29T09:55:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.109+0000 7fdddfc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd815b600 con 0x7fddd80d21f0 2024-03-29T09:55:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.109+0000 7fdddfc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80d2760 con 0x7fddd80cee10 2024-03-29T09:55:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.109+0000 7fdddd200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fddd81611b0 0x7fddd815ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56868/0 (socket says 172.21.15.12:56868) 2024-03-29T09:55:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.109+0000 7fdddd200700 1 -- 172.21.15.12:0/883087316 learned_addr learned my addr 172.21.15.12:0/883087316 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.110+0000 7fddddc00700 1 --2- 172.21.15.12:0/883087316 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd80cee10 0x7fddd8161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.110+0000 7fddd7400700 1 -- 172.21.15.12:0/883087316 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623125337 0 0) 0x7fddd80ce270 con 0x7fddd81611b0 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.110+0000 7fddd7400700 1 -- 172.21.15.12:0/883087316 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc00035b0 con 0x7fddd81611b0 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.110+0000 7fddd7400700 1 -- 172.21.15.12:0/883087316 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964861808 0 0) 0x7fddd815b600 con 0x7fddd80d21f0 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.110+0000 7fddd7400700 1 -- 172.21.15.12:0/883087316 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd80ce270 con 0x7fddd80d21f0 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.110+0000 7fddddc00700 1 -- 172.21.15.12:0/883087316 >> v1:172.21.15.159:6789/0 conn(0x7fddd80d21f0 legacy=0x7fddd80ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.110+0000 7fddddc00700 1 -- 172.21.15.12:0/883087316 >> v1:172.21.15.12:6789/0 conn(0x7fddd81611b0 legacy=0x7fddd815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.110+0000 7fddd7400700 1 -- 172.21.15.12:0/883087316 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3839548719 0 0) 0x7fddc00035b0 con 0x7fddd81611b0 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.110+0000 7fddddc00700 1 -- 172.21.15.12:0/883087316 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd80d2a40 con 0x7fddd80cee10 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.111+0000 7fddddc00700 1 --2- 172.21.15.12:0/883087316 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd80cee10 0x7fddd8161590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009dc0 tx=0x7fddd000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7dff230bfaddef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.111+0000 7fddd7400700 1 -- 172.21.15.12:0/883087316 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd000e070 con 0x7fddd80cee10 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.111+0000 7fddd7400700 1 -- 172.21.15.12:0/883087316 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd0003a20 con 0x7fddd80cee10 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.111+0000 7fddd7400700 1 -- 172.21.15.12:0/883087316 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc001670 con 0x7fddd80cee10 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.111+0000 7fdddfc00700 1 -- 172.21.15.12:0/883087316 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd80cee10 msgr2=0x7fddd8161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.111+0000 7fdddfc00700 1 --2- 172.21.15.12:0/883087316 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd80cee10 0x7fddd8161590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009dc0 tx=0x7fddd000b330 comp rx=0 tx=0).stop 2024-03-29T09:55:41.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.111+0000 7fdddfc00700 1 -- 172.21.15.12:0/883087316 shutdown_connections 2024-03-29T09:55:41.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.111+0000 7fdddfc00700 1 --2- 172.21.15.12:0/883087316 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd80cee10 0x7fddd8161590 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.112+0000 7fdddfc00700 1 -- 172.21.15.12:0/883087316 >> 172.21.15.12:0/883087316 conn(0x7fddd808aff0 msgr2=0x7fddd80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:41.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.112+0000 7fdddfc00700 1 -- 172.21.15.12:0/883087316 shutdown_connections 2024-03-29T09:55:41.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.112+0000 7fdddfc00700 1 -- 172.21.15.12:0/883087316 wait complete. 2024-03-29T09:55:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.113+0000 7fdddfc00700 1 Processor -- start 2024-03-29T09:55:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.113+0000 7fdddfc00700 1 -- start start 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.114+0000 7fdddfc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddd80cee10 0x7fddd8157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.114+0000 7fdddfc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddd80d21f0 0x7fddd80b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.114+0000 7fdddfc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd81611b0 0x7fddd81565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.114+0000 7fdddfc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80b8230 con 0x7fddd80cee10 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.114+0000 7fdddfc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd81e7120 con 0x7fddd80d21f0 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.114+0000 7fdddfc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd81e7400 con 0x7fddd81611b0 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fdddd200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddd80cee10 0x7fddd8157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fdddd200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddd80cee10 0x7fddd8157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35496/0 (socket says 172.21.15.12:35496) 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fdddd200700 1 -- 172.21.15.12:0/331787324 learned_addr learned my addr 172.21.15.12:0/331787324 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fddd7e00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddd80d21f0 0x7fddd80b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fddddc00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd81611b0 0x7fddd81565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fdddd200700 1 -- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd81611b0 msgr2=0x7fddd81565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fdddd200700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd81611b0 0x7fddd81565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fdddd200700 1 -- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddd80d21f0 msgr2=0x7fddd80b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fdddd200700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddd80d21f0 0x7fddd80b7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fdddd200700 1 -- 172.21.15.12:0/331787324 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd0009a70 con 0x7fddd80cee10 2024-03-29T09:55:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fddddc00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd81611b0 0x7fddd81565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.115+0000 7fdddd200700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddd80cee10 0x7fddd8157110 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00c700 tx=0x7fddcc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.116+0000 7fddd4c00700 1 -- 172.21.15.12:0/331787324 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc004c80 con 0x7fddd80cee10 2024-03-29T09:55:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.116+0000 7fddd4c00700 1 -- 172.21.15.12:0/331787324 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc004e00 con 0x7fddd80cee10 2024-03-29T09:55:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.116+0000 7fddd4c00700 1 -- 172.21.15.12:0/331787324 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc007310 con 0x7fddd80cee10 2024-03-29T09:55:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.116+0000 7fdddfc00700 1 -- 172.21.15.12:0/331787324 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd81e76e0 con 0x7fddd80cee10 2024-03-29T09:55:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.116+0000 7fdddfc00700 1 -- 172.21.15.12:0/331787324 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd8156af0 con 0x7fddd80cee10 2024-03-29T09:55:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.117+0000 7fdddfc00700 1 -- 172.21.15.12:0/331787324 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd8067720 con 0x7fddd80cee10 2024-03-29T09:55:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.118+0000 7fddd4c00700 1 -- 172.21.15.12:0/331787324 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fddcc007470 con 0x7fddd80cee10 2024-03-29T09:55:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.119+0000 7fddd4c00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddb806ac90 0x7fddb806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.119+0000 7fddd4c00700 1 -- 172.21.15.12:0/331787324 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddcc089040 con 0x7fddd80cee10 2024-03-29T09:55:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.119+0000 7fddd7e00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddb806ac90 0x7fddb806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.120+0000 7fddd7e00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddb806ac90 0x7fddb806d150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fddd8160440 tx=0x7fddc4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.123+0000 7fddd4c00700 1 -- 172.21.15.12:0/331787324 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fddcc0bc580 con 0x7fddd80cee10 2024-03-29T09:55:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.420+0000 7fdddfc00700 1 -- 172.21.15.12:0/331787324 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fddd80b5d00 con 0x7fddd80cee10 2024-03-29T09:55:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.424+0000 7fddd4c00700 1 -- 172.21.15.12:0/331787324 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+0 (secure 0 0 0) 0x7fddcc054210 con 0x7fddd80cee10 2024-03-29T09:55:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 -- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddb806ac90 msgr2=0x7fddb806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddb806ac90 0x7fddb806d150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fddd8160440 tx=0x7fddc4009040 comp rx=0 tx=0).stop 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 -- 172.21.15.12:0/331787324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddd80cee10 msgr2=0x7fddd8157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddd80cee10 0x7fddd8157110 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00c700 tx=0x7fddcc00ca10 comp rx=0 tx=0).stop 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 -- 172.21.15.12:0/331787324 shutdown_connections 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fddb806ac90 0x7fddb806d150 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddd81611b0 0x7fddd81565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddd80cee10 0x7fddd8157110 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 --2- 172.21.15.12:0/331787324 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddd80d21f0 0x7fddd80b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 -- 172.21.15.12:0/331787324 >> 172.21.15.12:0/331787324 conn(0x7fddd808aff0 msgr2=0x7fddd806ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 -- 172.21.15.12:0/331787324 shutdown_connections 2024-03-29T09:55:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.427+0000 7fdddfc00700 1 -- 172.21.15.12:0/331787324 wait complete. 2024-03-29T09:55:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T09:55:41.444 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T09:55:41.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.643+0000 7ff6a0600700 1 Processor -- start 2024-03-29T09:55:41.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.643+0000 7ff6a0600700 1 -- start start 2024-03-29T09:55:41.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.645+0000 7ff6a0600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff6980ce590 0x7ff6980ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.645+0000 7ff6a0600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6980b6a20 0x7ff6980b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.645+0000 7ff6a0600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6980ceef0 con 0x7ff698160be0 2024-03-29T09:55:41.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.645+0000 7ff6a0600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6980b65d0 con 0x7ff6980ce590 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.645+0000 7ff6a0600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6980d2a80 con 0x7ff6980b6a20 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.645+0000 7ff69dc00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff698160be0 0x7ff6980cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56878/0 (socket says 172.21.15.12:56878) 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.645+0000 7ff69dc00700 1 -- 172.21.15.12:0/1046222676 learned_addr learned my addr 172.21.15.12:0/1046222676 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.645+0000 7ff69d200700 1 --2- 172.21.15.12:0/1046222676 >> v2:172.21.15.159:3300/0 conn(0x7ff6980ce590 0x7ff6980ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.645+0000 7ff69e600700 1 --2- 172.21.15.12:0/1046222676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6980b6a20 0x7ff6980b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.646+0000 7ff697e00700 1 -- 172.21.15.12:0/1046222676 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500916187 0 0) 0x7ff6980ceef0 con 0x7ff698160be0 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.646+0000 7ff697e00700 1 -- 172.21.15.12:0/1046222676 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6800035b0 con 0x7ff698160be0 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.646+0000 7ff69e600700 1 -- 172.21.15.12:0/1046222676 >> v2:172.21.15.159:3300/0 conn(0x7ff6980ce590 msgr2=0x7ff6980ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.646+0000 7ff69e600700 1 --2- 172.21.15.12:0/1046222676 >> v2:172.21.15.159:3300/0 conn(0x7ff6980ce590 0x7ff6980ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.646+0000 7ff69e600700 1 -- 172.21.15.12:0/1046222676 >> v1:172.21.15.12:6789/0 conn(0x7ff698160be0 legacy=0x7ff6980cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.646+0000 7ff69e600700 1 -- 172.21.15.12:0/1046222676 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff698160fc0 con 0x7ff6980b6a20 2024-03-29T09:55:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.646+0000 7ff69d200700 1 --2- 172.21.15.12:0/1046222676 >> v2:172.21.15.159:3300/0 conn(0x7ff6980ce590 0x7ff6980ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:41.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.646+0000 7ff69e600700 1 --2- 172.21.15.12:0/1046222676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6980b6a20 0x7ff6980b6eb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff690009d90 tx=0x7ff69000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef889246a05823b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff697e00700 1 -- 172.21.15.12:0/1046222676 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69000e070 con 0x7ff6980b6a20 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff697e00700 1 -- 172.21.15.12:0/1046222676 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69000bb20 con 0x7ff6980b6a20 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff697e00700 1 -- 172.21.15.12:0/1046222676 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff690012430 con 0x7ff6980b6a20 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff6a0600700 1 -- 172.21.15.12:0/1046222676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6980b6a20 msgr2=0x7ff6980b6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff6a0600700 1 --2- 172.21.15.12:0/1046222676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6980b6a20 0x7ff6980b6eb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff690009d90 tx=0x7ff69000b330 comp rx=0 tx=0).stop 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff6a0600700 1 -- 172.21.15.12:0/1046222676 shutdown_connections 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff6a0600700 1 --2- 172.21.15.12:0/1046222676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6980b6a20 0x7ff6980b6eb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff6a0600700 1 --2- 172.21.15.12:0/1046222676 >> v2:172.21.15.159:3300/0 conn(0x7ff6980ce590 0x7ff6980ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff6a0600700 1 -- 172.21.15.12:0/1046222676 >> 172.21.15.12:0/1046222676 conn(0x7ff69808aff0 msgr2=0x7ff6980d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff6a0600700 1 -- 172.21.15.12:0/1046222676 shutdown_connections 2024-03-29T09:55:41.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.647+0000 7ff6a0600700 1 -- 172.21.15.12:0/1046222676 wait complete. 2024-03-29T09:55:41.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.649+0000 7ff6a0600700 1 Processor -- start 2024-03-29T09:55:41.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.649+0000 7ff6a0600700 1 -- start start 2024-03-29T09:55:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.650+0000 7ff6a0600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6980b6a20 0x7ff6980a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.650+0000 7ff6a0600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6980ce590 0x7ff6980cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.650+0000 7ff6a0600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff698160be0 0x7ff6980cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.650+0000 7ff6a0600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69814b3b0 con 0x7ff6980ce590 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.650+0000 7ff6a0600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6980a11e0 con 0x7ff6980b6a20 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.650+0000 7ff6a0600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6980c7f70 con 0x7ff698160be0 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.650+0000 7ff69d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6980ce590 0x7ff6980cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.650+0000 7ff69d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6980ce590 0x7ff6980cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35500/0 (socket says 172.21.15.12:35500) 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.650+0000 7ff69d200700 1 -- 172.21.15.12:0/877683551 learned_addr learned my addr 172.21.15.12:0/877683551 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.650+0000 7ff69dc00700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6980b6a20 0x7ff6980a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.651+0000 7ff69e600700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff698160be0 0x7ff6980cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.651+0000 7ff69d200700 1 -- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff698160be0 msgr2=0x7ff6980cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.651+0000 7ff69d200700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff698160be0 0x7ff6980cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.651+0000 7ff69d200700 1 -- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6980b6a20 msgr2=0x7ff6980a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.651+0000 7ff69d200700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6980b6a20 0x7ff6980a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.651+0000 7ff69d200700 1 -- 172.21.15.12:0/877683551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6840094a0 con 0x7ff6980ce590 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.651+0000 7ff69d200700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6980ce590 0x7ff6980cce30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff68400ec10 tx=0x7ff68400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.651+0000 7ff695600700 1 -- 172.21.15.12:0/877683551 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68400c850 con 0x7ff6980ce590 2024-03-29T09:55:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.651+0000 7ff695600700 1 -- 172.21.15.12:0/877683551 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68400c9b0 con 0x7ff6980ce590 2024-03-29T09:55:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.652+0000 7ff695600700 1 -- 172.21.15.12:0/877683551 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68400cd20 con 0x7ff6980ce590 2024-03-29T09:55:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.652+0000 7ff6a0600700 1 -- 172.21.15.12:0/877683551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff690009a70 con 0x7ff6980ce590 2024-03-29T09:55:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.652+0000 7ff6a0600700 1 -- 172.21.15.12:0/877683551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6980c8530 con 0x7ff6980ce590 2024-03-29T09:55:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.653+0000 7ff6a0600700 1 -- 172.21.15.12:0/877683551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69814b520 con 0x7ff6980ce590 2024-03-29T09:55:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.654+0000 7ff695600700 1 -- 172.21.15.12:0/877683551 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff684013470 con 0x7ff6980ce590 2024-03-29T09:55:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.655+0000 7ff695600700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff67806ac90 0x7ff67806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.655+0000 7ff695600700 1 -- 172.21.15.12:0/877683551 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff684089a80 con 0x7ff6980ce590 2024-03-29T09:55:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.655+0000 7ff69dc00700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff67806ac90 0x7ff67806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.656+0000 7ff69dc00700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff67806ac90 0x7ff67806d150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff68c007fd0 tx=0x7ff68c007dc0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:41.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.658+0000 7ff695600700 1 -- 172.21.15.12:0/877683551 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff684054c40 con 0x7ff6980ce590 2024-03-29T09:55:41.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.918+0000 7ff6a0600700 1 -- 172.21.15.12:0/877683551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff69814b4f0 con 0x7ff6980ce590 2024-03-29T09:55:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.919+0000 7ff695600700 1 -- 172.21.15.12:0/877683551 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4740 (secure 0 0 0) 0x7ff684017090 con 0x7ff6980ce590 2024-03-29T09:55:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T09:55:41.918 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2845880355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2845880355},{"type":"v1","addr":"172.21.15.159:6837","nonce":2845880355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2836227468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2836227468},{"type":"v1","addr":"172.21.15.159:6836","nonce":2836227468}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1863550973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1863550973},{"type":"v1","addr":"172.21.15.12:6834","nonce":1863550973}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:55:35.438233+0000","modified":"2024-03-29T09:55:38.926199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/2013939805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2013939805},{"type":"v1","addr":"172.21.15.12:6835","nonce":2013939805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-29T09:55:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 -- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff67806ac90 msgr2=0x7ff67806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff67806ac90 0x7ff67806d150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff68c007fd0 tx=0x7ff68c007dc0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 -- 172.21.15.12:0/877683551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6980ce590 msgr2=0x7ff6980cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6980ce590 0x7ff6980cce30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff68400ec10 tx=0x7ff68400ef20 comp rx=0 tx=0).stop 2024-03-29T09:55:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 -- 172.21.15.12:0/877683551 shutdown_connections 2024-03-29T09:55:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff67806ac90 0x7ff67806d150 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff698160be0 0x7ff6980cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6980b6a20 0x7ff6980a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 --2- 172.21.15.12:0/877683551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6980ce590 0x7ff6980cce30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 -- 172.21.15.12:0/877683551 >> 172.21.15.12:0/877683551 conn(0x7ff69808aff0 msgr2=0x7ff6980b8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 -- 172.21.15.12:0/877683551 shutdown_connections 2024-03-29T09:55:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:41.921+0000 7ff6a0600700 1 -- 172.21.15.12:0/877683551 wait complete. 2024-03-29T09:55:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-29T09:55:42.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T09:55:35.438233+0000', 'modified': '2024-03-29T09:55:38.926199+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5049}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5049': {'gid': 5049, 'name': 'a', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.12:6835/2013939805', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6833', 'nonce': 2013939805}, {'type': 'v1', 'addr': '172.21.15.12:6835', 'nonce': 2013939805}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T09:55:42.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T09:55:42.343 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-29T09:55:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.543+0000 7f1e94600700 1 Processor -- start 2024-03-29T09:55:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.544+0000 7f1e94600700 1 -- start start 2024-03-29T09:55:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.545+0000 7f1e94600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1e8c0d16a0 0x7f1e8c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.545+0000 7f1e94600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 0x7f1e8c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.545+0000 7f1e94600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c161420 con 0x7f1e8c15d9d0 2024-03-29T09:55:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.545+0000 7f1e94600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8c0d2820 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.545+0000 7f1e94600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0d4340 con 0x7f1e8c0d2370 2024-03-29T09:55:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.545+0000 7f1e91200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1e8c0d16a0 0x7f1e8c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.545+0000 7f1e92600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 0x7f1e8c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.545+0000 7f1e91200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1e8c0d16a0 0x7f1e8c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:48004/0 (socket says 172.21.15.12:48004) 2024-03-29T09:55:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.545+0000 7f1e91200700 1 -- 172.21.15.12:0/95047779 learned_addr learned my addr 172.21.15.12:0/95047779 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.545+0000 7f1e91c00700 1 --1- 172.21.15.12:0/95047779 >> v1:172.21.15.159:6789/0 conn(0x7f1e8c15d9d0 0x7f1e8c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:38836/0 (socket says 172.21.15.12:38836) 2024-03-29T09:55:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.546+0000 7f1e91200700 1 -- 172.21.15.12:0/95047779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 msgr2=0x7f1e8c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.546+0000 7f1e91200700 1 --2- 172.21.15.12:0/95047779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 0x7f1e8c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.546+0000 7f1e91200700 1 -- 172.21.15.12:0/95047779 >> v1:172.21.15.159:6789/0 conn(0x7f1e8c15d9d0 legacy=0x7f1e8c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.546+0000 7f1e91200700 1 -- 172.21.15.12:0/95047779 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c161700 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.546+0000 7f1e92600700 1 --2- 172.21.15.12:0/95047779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 0x7f1e8c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.546+0000 7f1e8be00700 1 -- 172.21.15.12:0/95047779 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542762403 0 0) 0x7f1e8c161420 con 0x7f1e8c15d9d0 2024-03-29T09:55:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.546+0000 7f1e91200700 1 --2- 172.21.15.12:0/95047779 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e8c0d16a0 0x7f1e8c0d1e30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1e78006fd0 tx=0x7f1e78009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57d48b6726c7a31c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.546+0000 7f1e8be00700 1 -- 172.21.15.12:0/95047779 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e78005e30 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.546+0000 7f1e8be00700 1 -- 172.21.15.12:0/95047779 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e78004030 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.546+0000 7f1e8be00700 1 -- 172.21.15.12:0/95047779 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e780043e0 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.547+0000 7f1e94600700 1 -- 172.21.15.12:0/95047779 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e8c0d16a0 msgr2=0x7f1e8c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.547+0000 7f1e94600700 1 --2- 172.21.15.12:0/95047779 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e8c0d16a0 0x7f1e8c0d1e30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1e78006fd0 tx=0x7f1e78009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.547+0000 7f1e94600700 1 -- 172.21.15.12:0/95047779 shutdown_connections 2024-03-29T09:55:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.547+0000 7f1e94600700 1 --2- 172.21.15.12:0/95047779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 0x7f1e8c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.547+0000 7f1e94600700 1 --2- 172.21.15.12:0/95047779 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e8c0d16a0 0x7f1e8c0d1e30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.547+0000 7f1e94600700 1 -- 172.21.15.12:0/95047779 >> 172.21.15.12:0/95047779 conn(0x7f1e8c08aff0 msgr2=0x7f1e8c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.547+0000 7f1e94600700 1 -- 172.21.15.12:0/95047779 shutdown_connections 2024-03-29T09:55:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.547+0000 7f1e94600700 1 -- 172.21.15.12:0/95047779 wait complete. 2024-03-29T09:55:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.549+0000 7f1e94600700 1 Processor -- start 2024-03-29T09:55:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.549+0000 7f1e94600700 1 -- start start 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e94600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c0d16a0 0x7f1e8c1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e94600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 0x7f1e8c0ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e94600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e8c15d9d0 0x7f1e8c15fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e94600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0cbbd0 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e94600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0cb000 con 0x7f1e8c15d9d0 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e94600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0cc060 con 0x7f1e8c0d2370 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e91c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c0d16a0 0x7f1e8c1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e91c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c0d16a0 0x7f1e8c1565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35510/0 (socket says 172.21.15.12:35510) 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e91c00700 1 -- 172.21.15.12:0/1878553020 learned_addr learned my addr 172.21.15.12:0/1878553020 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e91200700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 0x7f1e8c0ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.550+0000 7f1e92600700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e8c15d9d0 0x7f1e8c15fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e91c00700 1 -- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 msgr2=0x7f1e8c0ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e91c00700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 0x7f1e8c0ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e91c00700 1 -- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e8c15d9d0 msgr2=0x7f1e8c15fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e91c00700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e8c15d9d0 0x7f1e8c15fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e91c00700 1 -- 172.21.15.12:0/1878553020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e84009a70 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e91c00700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c0d16a0 0x7f1e8c1565a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000c550 tx=0x7f1e8000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e89600700 1 -- 172.21.15.12:0/1878553020 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80004c80 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e89600700 1 -- 172.21.15.12:0/1878553020 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e80004e00 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e89600700 1 -- 172.21.15.12:0/1878553020 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80007310 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e94600700 1 -- 172.21.15.12:0/1878553020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e78009500 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.551+0000 7f1e94600700 1 -- 172.21.15.12:0/1878553020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c0cb580 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.553+0000 7f1e89600700 1 -- 172.21.15.12:0/1878553020 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1e80007470 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.554+0000 7f1e89600700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e6c06ac90 0x7f1e6c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.554+0000 7f1e89600700 1 -- 172.21.15.12:0/1878553020 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e80088e90 con 0x7f1e8c0d16a0 2024-03-29T09:55:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.554+0000 7f1e91200700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e6c06ac90 0x7f1e6c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.555+0000 7f1e91200700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e6c06ac90 0x7f1e6c06d150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1e78006fd0 tx=0x7f1e78008070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:42.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.556+0000 7f1ea8157b80 1 Processor -- start 2024-03-29T09:55:42.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.557+0000 7f1ea8157b80 1 -- start start 2024-03-29T09:55:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bc1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53b97a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bbea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53b0c650 con 0x563d53bc1830 2024-03-29T09:55:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53aedc30 con 0x563d53b94be0 2024-03-29T09:55:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d53aeb430 con 0x563d53b92b60 2024-03-29T09:55:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1e91c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bc1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1e91c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bc1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35516/0 (socket says 172.21.15.12:35516) 2024-03-29T09:55:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1e91c00700 1 -- 172.21.15.12:0/3879411263 learned_addr learned my addr 172.21.15.12:0/3879411263 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1e91c00700 1 -- 172.21.15.12:0/3879411263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 msgr2=0x563d53bbea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1e91c00700 1 --2- 172.21.15.12:0/3879411263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bbea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1e91c00700 1 -- 172.21.15.12:0/3879411263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 msgr2=0x563d53b97a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1e91c00700 1 --2- 172.21.15.12:0/3879411263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53b97a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.558+0000 7f1e91c00700 1 -- 172.21.15.12:0/3879411263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d538da790 con 0x563d53bc1830 2024-03-29T09:55:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.559+0000 7f1e91c00700 1 --2- 172.21.15.12:0/3879411263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bc1c10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1e80001eb0 tx=0x7f1e80057900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e47325e4fb666 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:42.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.559+0000 7f1e7e000700 1 -- 172.21.15.12:0/3879411263 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80057dc0 con 0x563d53bc1830 2024-03-29T09:55:42.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.559+0000 7f1e7e000700 1 -- 172.21.15.12:0/3879411263 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8005bbf0 con 0x563d53bc1830 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.559+0000 7f1e7e000700 1 -- 172.21.15.12:0/3879411263 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800601b0 con 0x563d53bc1830 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.560+0000 7f1ea8157b80 1 -- 172.21.15.12:0/3879411263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 msgr2=0x563d53bc1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.560+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/3879411263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bc1c10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1e80001eb0 tx=0x7f1e80057900 comp rx=0 tx=0).stop 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.560+0000 7f1ea8157b80 1 -- 172.21.15.12:0/3879411263 shutdown_connections 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.560+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/3879411263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bbea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.560+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/3879411263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bc1c10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.560+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/3879411263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53b97a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.560+0000 7f1ea8157b80 1 -- 172.21.15.12:0/3879411263 >> 172.21.15.12:0/3879411263 conn(0x563d53af56a0 msgr2=0x563d53bdc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.560+0000 7f1ea8157b80 1 -- 172.21.15.12:0/3879411263 shutdown_connections 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.560+0000 7f1ea8157b80 1 -- 172.21.15.12:0/3879411263 wait complete. 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.560+0000 7f1ea8157b80 1 -- start start 2024-03-29T09:55:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.561+0000 7f1ea8157b80 1 Processor -- start 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bee800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53bddc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bde190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53ae6d20 con 0x563d53bc1830 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53a82770 con 0x563d53b94be0 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d53aa56f0 con 0x563d53b92b60 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1ea8157b80 10 client.0 _ll_register_callbacks cb 0x563d53a8c5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e92600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bde190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e92600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bde190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35520/0 (socket says 172.21.15.12:35520) 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e92600700 1 -- 172.21.15.12:0/2669526826 learned_addr learned my addr 172.21.15.12:0/2669526826 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e91200700 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53bddc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e91c00700 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e92600700 1 -- 172.21.15.12:0/2669526826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 msgr2=0x563d53bee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e92600700 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bee800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e92600700 1 -- 172.21.15.12:0/2669526826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 msgr2=0x563d53bddc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e92600700 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53bddc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e92600700 1 -- 172.21.15.12:0/2669526826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c0cb580 con 0x563d53bc1830 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.562+0000 7f1e91200700 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53bddc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.563+0000 7f1e92600700 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bde190 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1e84011d40 tx=0x7f1e8400f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90c5a958ff01ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.563+0000 7f1e72a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.563+0000 7f1e72a00700 1 -- 172.21.15.12:0/2669526826 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400e040 con 0x563d53bc1830 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.563+0000 7f1ea8157b80 1 -- 172.21.15.12:0/2669526826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d53b89920 con 0x563d53bc1830 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.563+0000 7f1e72a00700 1 -- 172.21.15.12:0/2669526826 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e8400fe30 con 0x563d53bc1830 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.563+0000 7f1e72a00700 1 -- 172.21.15.12:0/2669526826 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e84011ee0 con 0x563d53bc1830 2024-03-29T09:55:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.563+0000 7f1ea8157b80 10 client.5226 fetch_fsmap learned FSMap version 15 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.563+0000 7f1ea8157b80 1 -- 172.21.15.12:0/2669526826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x563d539c70a0 con 0x563d53bc1830 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.564+0000 7f1e72a00700 1 -- 172.21.15.12:0/2669526826 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563d53b89920 con 0x563d53bc1830 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.564+0000 7f1e72a00700 1 -- 172.21.15.12:0/2669526826 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 15) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1e8401b030 con 0x563d53bc1830 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.564+0000 7f1ea8157b80 10 client.5226 fetch_fsmap finished waiting for FSMap version 15 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.564+0000 7f1ea8157b80 10 client.5226 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.564+0000 7f1ea8157b80 20 client.5226 populate_metadata read hostname 'smithi012' 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.564+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.12:6833/2013939805,v1:172.21.15.12:6835/2013939805] conn(0x563d53bbfc70 0x563d53bc3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.564+0000 7f1ea8157b80 4 client.5226 mds_command: new command op to 5049 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.564+0000 7f1ea8157b80 1 -- 172.21.15.12:0/2669526826 --> [v2:172.21.15.12:6833/2013939805,v1:172.21.15.12:6835/2013939805] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d53b666f0 con 0x563d53bbfc70 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.564+0000 7f1e91c00700 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.12:6833/2013939805,v1:172.21.15.12:6835/2013939805] conn(0x563d53bbfc70 0x563d53bc3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.565+0000 7f1e91c00700 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.12:6833/2013939805,v1:172.21.15.12:6835/2013939805] conn(0x563d53bbfc70 0x563d53bc3090 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89246ca948243c11 server_cookie=c55f3def4a3c8f07 in_seq=0 out_seq=0 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.565+0000 7f1e72a00700 10 client.5226 ms_handle_connect on v2:172.21.15.12:6833/2013939805 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.565+0000 7f1e72a00700 1 -- 172.21.15.12:0/2669526826 <== mds.0 v2:172.21.15.12:6833/2013939805 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563d53b666f0 con 0x563d53bbfc70 2024-03-29T09:55:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.565+0000 7f1e72a00700 10 client.5226 handle_command_reply: tid=0 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.565+0000 7f1ea8157b80 1 client.5226 shutdown 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 1 -- 172.21.15.12:0/2669526826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 msgr2=0x563d53bde190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bde190 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1e84011d40 tx=0x7f1e8400f490 comp rx=0 tx=0).stop 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 1 -- 172.21.15.12:0/2669526826 shutdown_connections 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.12:6833/2013939805,v1:172.21.15.12:6835/2013939805] conn(0x563d53bbfc70 0x563d53bc3090 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53bde190 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/2669526826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53bddc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1e72a00700 0 client.5226 ms_handle_reset on v2:172.21.15.12:6833/2013939805 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 1 -- 172.21.15.12:0/2669526826 >> 172.21.15.12:0/2669526826 conn(0x563d53af56a0 msgr2=0x563d53bbddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 1 -- 172.21.15.12:0/2669526826 shutdown_connections 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 1 -- 172.21.15.12:0/2669526826 wait complete. 2024-03-29T09:55:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.566+0000 7f1ea8157b80 20 client.5226 trim_cache size 0 max 16384 2024-03-29T09:55:42.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.584+0000 7f1ea8157b80 1 Processor -- start 2024-03-29T09:55:42.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.584+0000 7f1ea8157b80 1 -- start start 2024-03-29T09:55:42.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bd8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53b98400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53b98940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d538db390 con 0x563d53bc1830 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53b94870 con 0x563d53b94be0 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d53bc1fa0 con 0x563d53b92b60 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1e92600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53b98940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1e92600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53b98940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35534/0 (socket says 172.21.15.12:35534) 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1e92600700 1 -- 172.21.15.12:0/4202204089 learned_addr learned my addr 172.21.15.12:0/4202204089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1e91200700 1 --2- 172.21.15.12:0/4202204089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53b98400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.585+0000 7f1e91c00700 1 --2- 172.21.15.12:0/4202204089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bd8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.586+0000 7f1e92600700 1 -- 172.21.15.12:0/4202204089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 msgr2=0x563d53bd8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:42.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.586+0000 7f1e92600700 1 --2- 172.21.15.12:0/4202204089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bd8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.586+0000 7f1e92600700 1 -- 172.21.15.12:0/4202204089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 msgr2=0x563d53b98400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.586+0000 7f1e92600700 1 --2- 172.21.15.12:0/4202204089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53b98400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.586+0000 7f1e92600700 1 -- 172.21.15.12:0/4202204089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d53aad010 con 0x563d53bc1830 2024-03-29T09:55:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.586+0000 7f1e92600700 1 --2- 172.21.15.12:0/4202204089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53b98940 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009950 tx=0x7f1e84016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf3ad122602984d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.586+0000 7f1e7ea00700 1 -- 172.21.15.12:0/4202204089 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400e040 con 0x563d53bc1830 2024-03-29T09:55:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.586+0000 7f1e7ea00700 1 -- 172.21.15.12:0/4202204089 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8400f040 con 0x563d53bc1830 2024-03-29T09:55:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.586+0000 7f1e7ea00700 1 -- 172.21.15.12:0/4202204089 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400fe50 con 0x563d53bc1830 2024-03-29T09:55:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.587+0000 7f1ea8157b80 1 -- 172.21.15.12:0/4202204089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 msgr2=0x563d53b98940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.587+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/4202204089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53b98940 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009950 tx=0x7f1e84016000 comp rx=0 tx=0).stop 2024-03-29T09:55:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.587+0000 7f1ea8157b80 1 -- 172.21.15.12:0/4202204089 shutdown_connections 2024-03-29T09:55:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.587+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/4202204089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53bc1830 0x563d53b98940 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.587+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/4202204089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bd8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.587+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/4202204089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53b94be0 0x563d53b98400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.587+0000 7f1ea8157b80 1 -- 172.21.15.12:0/4202204089 >> 172.21.15.12:0/4202204089 conn(0x563d53af56a0 msgr2=0x563d53bd9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.587+0000 7f1ea8157b80 1 -- 172.21.15.12:0/4202204089 shutdown_connections 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.587+0000 7f1ea8157b80 1 -- 172.21.15.12:0/4202204089 wait complete. 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.587+0000 7f1ea8157b80 1 -- start start 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.588+0000 7f1ea8157b80 1 Processor -- start 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bbd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53b94be0 0x563d53bf4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1ea8157b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53bc1830 0x563d53bc45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53b96380 con 0x563d53b94be0 2024-03-29T09:55:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53bbdde0 con 0x563d53bc1830 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1ea8157b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d53bf4e00 con 0x563d53b92b60 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1ea8157b80 10 client.0 _ll_register_callbacks cb 0x563d53a8c5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1e91200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53b94be0 0x563d53bf4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1e91200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53b94be0 0x563d53bf4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35546/0 (socket says 172.21.15.12:35546) 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1e91200700 1 -- 172.21.15.12:0/567045751 learned_addr learned my addr 172.21.15.12:0/567045751 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1e91c00700 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bbd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1e92600700 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53bc1830 0x563d53bc45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1e91200700 1 -- 172.21.15.12:0/567045751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 msgr2=0x563d53bbd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1e91200700 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bbd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1e91200700 1 -- 172.21.15.12:0/567045751 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53bc1830 msgr2=0x563d53bc45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1e91200700 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53bc1830 0x563d53bc45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.589+0000 7f1e91200700 1 -- 172.21.15.12:0/567045751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d539c70a0 con 0x563d53b94be0 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1e91200700 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53b94be0 0x563d53bf4640 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1e78004000 tx=0x7f1e780047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7394938270a9180 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1e72a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1e72a00700 1 -- 172.21.15.12:0/567045751 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7801f9b0 con 0x563d53b94be0 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1e72a00700 1 -- 172.21.15.12:0/567045751 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e78024070 con 0x563d53b94be0 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1ea8157b80 1 -- 172.21.15.12:0/567045751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1e8005bbf0 con 0x563d53b94be0 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1e72a00700 1 -- 172.21.15.12:0/567045751 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e7801fb10 con 0x563d53b94be0 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1ea8157b80 10 client.5232 fetch_fsmap learned FSMap version 15 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1ea8157b80 1 -- 172.21.15.12:0/567045751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x563d5393b1d0 con 0x563d53b94be0 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1e72a00700 1 -- 172.21.15.12:0/567045751 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8005bbf0 con 0x563d53b94be0 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1e72a00700 1 -- 172.21.15.12:0/567045751 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 15) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1e78032020 con 0x563d53b94be0 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.590+0000 7f1ea8157b80 10 client.5232 fetch_fsmap finished waiting for FSMap version 15 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1ea8157b80 10 client.5232 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1ea8157b80 20 client.5232 populate_metadata read hostname 'smithi012' 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.12:6833/2013939805,v1:172.21.15.12:6835/2013939805] conn(0x563d53b98400 0x563d53b987e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1ea8157b80 4 client.5232 mds_command: new command op to 5049 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T09:55:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1ea8157b80 1 -- 172.21.15.12:0/567045751 --> [v2:172.21.15.12:6833/2013939805,v1:172.21.15.12:6835/2013939805] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563d53920040 con 0x563d53b98400 2024-03-29T09:55:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1e91c00700 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.12:6833/2013939805,v1:172.21.15.12:6835/2013939805] conn(0x563d53b98400 0x563d53b987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1e91c00700 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.12:6833/2013939805,v1:172.21.15.12:6835/2013939805] conn(0x563d53b98400 0x563d53b987e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71121ba3f362a5a2 server_cookie=29ea1afbe97b6f23 in_seq=0 out_seq=0 2024-03-29T09:55:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1e72a00700 10 client.5232 ms_handle_connect on v2:172.21.15.12:6833/2013939805 2024-03-29T09:55:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1e72a00700 1 -- 172.21.15.12:0/567045751 <== mds.0 v2:172.21.15.12:6833/2013939805 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x563d53920040 con 0x563d53b98400 2024-03-29T09:55:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1e72a00700 10 client.5232 handle_command_reply: tid=0 2024-03-29T09:55:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.591+0000 7f1ea8157b80 1 client.5232 shutdown 2024-03-29T09:55:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 1 -- 172.21.15.12:0/567045751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53b94be0 msgr2=0x563d53bf4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53b94be0 0x563d53bf4640 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1e78004000 tx=0x7f1e780047c0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 1 -- 172.21.15.12:0/567045751 shutdown_connections 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d53bc1830 0x563d53bc45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d53b92b60 0x563d53bbd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.12:6833/2013939805,v1:172.21.15.12:6835/2013939805] conn(0x563d53b98400 0x563d53b987e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 1 --2- 172.21.15.12:0/567045751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d53b94be0 0x563d53bf4640 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 1 -- 172.21.15.12:0/567045751 >> 172.21.15.12:0/567045751 conn(0x563d53af56a0 msgr2=0x563d53bddb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1e72a00700 0 client.5232 ms_handle_reset on v2:172.21.15.12:6833/2013939805 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 1 -- 172.21.15.12:0/567045751 shutdown_connections 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 1 -- 172.21.15.12:0/567045751 wait complete. 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.592+0000 7f1ea8157b80 20 client.5232 trim_cache size 0 max 16384 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T09:55:42.596 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 5049, 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1515298810000001, 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 15, 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 34, 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 34, 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 15.739337772000001 2024-03-29T09:55:42.597 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T09:55:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 -- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e6c06ac90 msgr2=0x7f1e6c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e6c06ac90 0x7f1e6c06d150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1e78006fd0 tx=0x7f1e78008070 comp rx=0 tx=0).stop 2024-03-29T09:55:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 -- 172.21.15.12:0/1878553020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c0d16a0 msgr2=0x7f1e8c1565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c0d16a0 0x7f1e8c1565a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000c550 tx=0x7f1e8000c860 comp rx=0 tx=0).stop 2024-03-29T09:55:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 -- 172.21.15.12:0/1878553020 shutdown_connections 2024-03-29T09:55:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e6c06ac90 0x7f1e6c06d150 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e8c15d9d0 0x7f1e8c15fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c0d16a0 0x7f1e8c1565a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 --2- 172.21.15.12:0/1878553020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e8c0d2370 0x7f1e8c0ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 -- 172.21.15.12:0/1878553020 >> 172.21.15.12:0/1878553020 conn(0x7f1e8c08aff0 msgr2=0x7f1e8c0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 -- 172.21.15.12:0/1878553020 shutdown_connections 2024-03-29T09:55:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.593+0000 7f1e94600700 1 -- 172.21.15.12:0/1878553020 wait complete. 2024-03-29T09:55:42.610 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T09:55:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.808+0000 7f1907800700 1 Processor -- start 2024-03-29T09:55:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.808+0000 7f1907800700 1 -- start start 2024-03-29T09:55:42.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.809+0000 7f1907800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1900160f10 0x7f19000d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.809+0000 7f1907800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f19000cf230 0x7f19000d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.809+0000 7f1907800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19000ceda0 con 0x7f1900161590 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.809+0000 7f1907800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19000d23d0 con 0x7f19000cf230 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.809+0000 7f1907800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19001612f0 con 0x7f1900160f10 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.809+0000 7f18ffe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f19000cf230 0x7f19000d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.809+0000 7f18ffe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f19000cf230 0x7f19000d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35558/0 (socket says 172.21.15.12:35558) 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.809+0000 7f18ffe00700 1 -- 172.21.15.12:0/3370813906 learned_addr learned my addr 172.21.15.12:0/3370813906 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.809+0000 7f1904e00700 1 --2- 172.21.15.12:0/3370813906 >> v2:172.21.15.159:3300/0 conn(0x7f1900160f10 0x7f19000d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.810+0000 7f18ffe00700 1 -- 172.21.15.12:0/3370813906 >> v2:172.21.15.159:3300/0 conn(0x7f1900160f10 msgr2=0x7f19000d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.810+0000 7f18ffe00700 1 --2- 172.21.15.12:0/3370813906 >> v2:172.21.15.159:3300/0 conn(0x7f1900160f10 0x7f19000d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.810+0000 7f18ffe00700 1 -- 172.21.15.12:0/3370813906 >> v1:172.21.15.159:6789/0 conn(0x7f1900161590 legacy=0x7f19000b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.810+0000 7f18ffe00700 1 -- 172.21.15.12:0/3370813906 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19000ce470 con 0x7f19000cf230 2024-03-29T09:55:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.810+0000 7f18ffe00700 1 --2- 172.21.15.12:0/3370813906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19000cf230 0x7f19000d1e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f19000d1080 tx=0x7f18ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2430e33d7b9e4a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.810+0000 7f18ff400700 1 -- 172.21.15.12:0/3370813906 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec011830 con 0x7f19000cf230 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.810+0000 7f18ff400700 1 -- 172.21.15.12:0/3370813906 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ec011990 con 0x7f19000cf230 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.810+0000 7f18ff400700 1 -- 172.21.15.12:0/3370813906 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec011d20 con 0x7f19000cf230 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.811+0000 7f1907800700 1 -- 172.21.15.12:0/3370813906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19000cf230 msgr2=0x7f19000d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.811+0000 7f1907800700 1 --2- 172.21.15.12:0/3370813906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19000cf230 0x7f19000d1e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f19000d1080 tx=0x7f18ec009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.811+0000 7f1907800700 1 -- 172.21.15.12:0/3370813906 shutdown_connections 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.811+0000 7f1907800700 1 --2- 172.21.15.12:0/3370813906 >> v2:172.21.15.159:3300/0 conn(0x7f1900160f10 0x7f19000d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.811+0000 7f1907800700 1 --2- 172.21.15.12:0/3370813906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19000cf230 0x7f19000d1e90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.811+0000 7f1907800700 1 -- 172.21.15.12:0/3370813906 >> 172.21.15.12:0/3370813906 conn(0x7f190008aff0 msgr2=0x7f19000d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.811+0000 7f1907800700 1 -- 172.21.15.12:0/3370813906 shutdown_connections 2024-03-29T09:55:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.811+0000 7f1907800700 1 -- 172.21.15.12:0/3370813906 wait complete. 2024-03-29T09:55:42.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.812+0000 7f1907800700 1 Processor -- start 2024-03-29T09:55:42.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.813+0000 7f1907800700 1 -- start start 2024-03-29T09:55:42.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.813+0000 7f1907800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19000cf230 0x7f19001594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.813+0000 7f1907800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1900160f10 0x7f1900156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.813+0000 7f1907800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1900161590 0x7f1900155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.813+0000 7f1907800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19001599e0 con 0x7f1900161590 2024-03-29T09:55:42.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.813+0000 7f1907800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1900159ef0 con 0x7f19000cf230 2024-03-29T09:55:42.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.813+0000 7f1907800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1900159c10 con 0x7f1900160f10 2024-03-29T09:55:42.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1905800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1900161590 0x7f1900155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1905800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1900161590 0x7f1900155f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35568/0 (socket says 172.21.15.12:35568) 2024-03-29T09:55:42.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1905800700 1 -- 172.21.15.12:0/2509439049 learned_addr learned my addr 172.21.15.12:0/2509439049 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1904e00700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19000cf230 0x7f19001594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f18ffe00700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1900160f10 0x7f1900156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1905800700 1 -- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1900160f10 msgr2=0x7f1900156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1905800700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1900160f10 0x7f1900156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1905800700 1 -- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19000cf230 msgr2=0x7f19001594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1905800700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19000cf230 0x7f19001594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1905800700 1 -- 172.21.15.12:0/2509439049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec009550 con 0x7f1900161590 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1904e00700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19000cf230 0x7f19001594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.814+0000 7f1905800700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1900161590 0x7f1900155f00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f18f800cb50 tx=0x7f18f800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.815+0000 7f18fcc00700 1 -- 172.21.15.12:0/2509439049 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8003c00 con 0x7f1900161590 2024-03-29T09:55:42.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.815+0000 7f18fcc00700 1 -- 172.21.15.12:0/2509439049 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f8003d60 con 0x7f1900161590 2024-03-29T09:55:42.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.815+0000 7f1907800700 1 -- 172.21.15.12:0/2509439049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19000b7500 con 0x7f1900161590 2024-03-29T09:55:42.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.815+0000 7f18fcc00700 1 -- 172.21.15.12:0/2509439049 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f800ab70 con 0x7f1900161590 2024-03-29T09:55:42.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.815+0000 7f1907800700 1 -- 172.21.15.12:0/2509439049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19000b7ac0 con 0x7f1900161590 2024-03-29T09:55:42.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.816+0000 7f1907800700 1 -- 172.21.15.12:0/2509439049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1900067720 con 0x7f1900161590 2024-03-29T09:55:42.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.817+0000 7f18fcc00700 1 -- 172.21.15.12:0/2509439049 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f18f800acd0 con 0x7f1900161590 2024-03-29T09:55:42.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.818+0000 7f18fcc00700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f18e006ac90 0x7f18e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:42.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.818+0000 7f18fcc00700 1 -- 172.21.15.12:0/2509439049 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18f8089380 con 0x7f1900161590 2024-03-29T09:55:42.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.818+0000 7f1904e00700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f18e006ac90 0x7f18e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:42.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.819+0000 7f1904e00700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f18e006ac90 0x7f18e006d150 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f19000ca380 tx=0x7f18f4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:42.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:42.823+0000 7f18fcc00700 1 -- 172.21.15.12:0/2509439049 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18f8054550 con 0x7f1900161590 2024-03-29T09:55:43.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.090+0000 7f1907800700 1 -- 172.21.15.12:0/2509439049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f19000c9a30 con 0x7f1900161590 2024-03-29T09:55:43.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.090+0000 7f18fcc00700 1 -- 172.21.15.12:0/2509439049 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f18f8057b70 con 0x7f1900161590 2024-03-29T09:55:43.090 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T09:55:43.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 -- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f18e006ac90 msgr2=0x7f18e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:43.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f18e006ac90 0x7f18e006d150 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f19000ca380 tx=0x7f18f4008040 comp rx=0 tx=0).stop 2024-03-29T09:55:43.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 -- 172.21.15.12:0/2509439049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1900161590 msgr2=0x7f1900155f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:43.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1900161590 0x7f1900155f00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f18f800cb50 tx=0x7f18f800cf10 comp rx=0 tx=0).stop 2024-03-29T09:55:43.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 -- 172.21.15.12:0/2509439049 shutdown_connections 2024-03-29T09:55:43.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f18e006ac90 0x7f18e006d150 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:43.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1900161590 0x7f1900155f00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:43.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19000cf230 0x7f19001594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:43.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 --2- 172.21.15.12:0/2509439049 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1900160f10 0x7f1900156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:43.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 -- 172.21.15.12:0/2509439049 >> 172.21.15.12:0/2509439049 conn(0x7f190008aff0 msgr2=0x7f19000cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:43.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 -- 172.21.15.12:0/2509439049 shutdown_connections 2024-03-29T09:55:43.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:43.093+0000 7f1907800700 1 -- 172.21.15.12:0/2509439049 wait complete. 2024-03-29T09:55:43.111 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T09:55:43.111 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T09:55:43.111 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T09:55:43.111 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T09:55:43.111 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T09:55:43.112 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T09:55:43.112 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T09:55:43.112 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T09:55:43.112 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T09:55:43.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:43.112 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:55:43.158 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:55:43.159 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T09:55:43.159 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T09:55:43.161 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T09:55:43.161 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T09:55:43.161 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 41374sec preferred_lft 41374sec 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:55:43.162 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T09:55:43.163 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T09:55:43.163 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T09:55:43.163 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:55:43.163 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T09:55:43.163 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T09:55:43.163 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T09:55:43.163 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T09:55:43.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:43.163 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T09:55:43.262 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T09:55:43.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:43.266 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T09:55:43.312 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T09:55:43.312 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T09:55:43.312 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T09:55:43.312 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T09:55:43.313 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T09:55:43.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:43.314 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T09:55:43.314 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T09:55:43.314 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T09:55:43.314 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T09:55:43.314 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T09:55:43.314 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T09:55:43.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:43.479 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T09:55:43.523 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T09:55:43.523 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T09:55:43.524 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T09:55:43.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:43.524 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T09:55:43.632 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T09:55:43.632 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T09:55:43.677 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T09:55:43.677 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T09:55:43.677 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T09:55:43.761 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T09:55:43.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:43.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:55:43.903 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:55:43.903 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T09:55:43.931 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T09:55:43.931 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T09:55:43.932 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T09:55:43.932 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T09:55:43.932 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T09:55:43.937 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T09:55:43.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:43.938 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T09:55:44.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:44.005 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T09:55:44.107 INFO:teuthology.orchestra.run.smithi012.stdout:client4916 2024-03-29T09:55:44.107 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T09:55:44.107 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T09:55:44.152 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T09:55:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.392+0000 7fc4c5e00700 1 Processor -- start 2024-03-29T09:55:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.392+0000 7fc4c5e00700 1 -- start start 2024-03-29T09:55:44.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.393+0000 7fc4c5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 0x7fc4c015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.393+0000 7fc4c5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc4c00cee10 0x7fc4c0161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4c5e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c00ce270 con 0x7fc4c015f4d0 2024-03-29T09:55:44.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4c5e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c015b600 con 0x7fc4c00cee10 2024-03-29T09:55:44.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4c5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c00d2760 con 0x7fc4c01611b0 2024-03-29T09:55:44.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4bf400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc4c00cee10 0x7fc4c0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4bf400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc4c00cee10 0x7fc4c0161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35578/0 (socket says 172.21.15.12:35578) 2024-03-29T09:55:44.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4bf400700 1 -- 172.21.15.12:0/3610230171 learned_addr learned my addr 172.21.15.12:0/3610230171 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:44.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4bea00700 1 --2- 172.21.15.12:0/3610230171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 0x7fc4c015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4bf400700 1 -- 172.21.15.12:0/3610230171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 msgr2=0x7fc4c015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4bf400700 1 --2- 172.21.15.12:0/3610230171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 0x7fc4c015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4bf400700 1 -- 172.21.15.12:0/3610230171 >> v1:172.21.15.159:6789/0 conn(0x7fc4c015f4d0 legacy=0x7fc4c00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.394+0000 7fc4bf400700 1 -- 172.21.15.12:0/3610230171 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c00d2a40 con 0x7fc4c00cee10 2024-03-29T09:55:44.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.395+0000 7fc4bf400700 1 --2- 172.21.15.12:0/3610230171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 0x7fc4c0161590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009e50 tx=0x7fc4b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1ec8874d84b0b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.395+0000 7fc4bd600700 1 -- 172.21.15.12:0/3610230171 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800e070 con 0x7fc4c00cee10 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.395+0000 7fc4bd600700 1 -- 172.21.15.12:0/3610230171 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b800b910 con 0x7fc4c00cee10 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.395+0000 7fc4bd600700 1 -- 172.21.15.12:0/3610230171 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800bcc0 con 0x7fc4c00cee10 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.395+0000 7fc4c5e00700 1 -- 172.21.15.12:0/3610230171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 msgr2=0x7fc4c0161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.395+0000 7fc4c5e00700 1 --2- 172.21.15.12:0/3610230171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 0x7fc4c0161590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009e50 tx=0x7fc4b800b330 comp rx=0 tx=0).stop 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.396+0000 7fc4c5e00700 1 -- 172.21.15.12:0/3610230171 shutdown_connections 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.396+0000 7fc4c5e00700 1 --2- 172.21.15.12:0/3610230171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 0x7fc4c0161590 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.396+0000 7fc4c5e00700 1 --2- 172.21.15.12:0/3610230171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 0x7fc4c015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.396+0000 7fc4c5e00700 1 -- 172.21.15.12:0/3610230171 >> 172.21.15.12:0/3610230171 conn(0x7fc4c008aff0 msgr2=0x7fc4c00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.396+0000 7fc4c5e00700 1 -- 172.21.15.12:0/3610230171 shutdown_connections 2024-03-29T09:55:44.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.396+0000 7fc4c5e00700 1 -- 172.21.15.12:0/3610230171 wait complete. 2024-03-29T09:55:44.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.397+0000 7fc4c5e00700 1 Processor -- start 2024-03-29T09:55:44.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.398+0000 7fc4c5e00700 1 -- start start 2024-03-29T09:55:44.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4c5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 0x7fc4c00b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4c5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4c015f4d0 0x7fc4c01ea550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4c5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 0x7fc4c00c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4c5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c0149630 con 0x7fc4c00cee10 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4c5e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c0149910 con 0x7fc4c015f4d0 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4c5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c0149bf0 con 0x7fc4c01611b0 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 0x7fc4c00b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 0x7fc4c00b6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35584/0 (socket says 172.21.15.12:35584) 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4bea00700 1 -- 172.21.15.12:0/2673673748 learned_addr learned my addr 172.21.15.12:0/2673673748 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4be000700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4c015f4d0 0x7fc4c01ea550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4bf400700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 0x7fc4c00c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4bea00700 1 -- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 msgr2=0x7fc4c00c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:44.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4bea00700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 0x7fc4c00c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4bea00700 1 -- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4c015f4d0 msgr2=0x7fc4c01ea550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:44.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4bea00700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4c015f4d0 0x7fc4c01ea550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4bea00700 1 -- 172.21.15.12:0/2673673748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b8009b30 con 0x7fc4c00cee10 2024-03-29T09:55:44.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.399+0000 7fc4bf400700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 0x7fc4c00c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:44.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.400+0000 7fc4bea00700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 0x7fc4c00b6150 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000e870 tx=0x7fc4b000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:44.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.400+0000 7fc4b7400700 1 -- 172.21.15.12:0/2673673748 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0009590 con 0x7fc4c00cee10 2024-03-29T09:55:44.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.400+0000 7fc4b7400700 1 -- 172.21.15.12:0/2673673748 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b000ee70 con 0x7fc4c00cee10 2024-03-29T09:55:44.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.400+0000 7fc4b7400700 1 -- 172.21.15.12:0/2673673748 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b000ca50 con 0x7fc4c00cee10 2024-03-29T09:55:44.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.400+0000 7fc4c5e00700 1 -- 172.21.15.12:0/2673673748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c0149ed0 con 0x7fc4c00cee10 2024-03-29T09:55:44.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.400+0000 7fc4c5e00700 1 -- 172.21.15.12:0/2673673748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c01eaa90 con 0x7fc4c00cee10 2024-03-29T09:55:44.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.401+0000 7fc4c5e00700 1 -- 172.21.15.12:0/2673673748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c014a3f0 con 0x7fc4c00cee10 2024-03-29T09:55:44.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.402+0000 7fc4b7400700 1 -- 172.21.15.12:0/2673673748 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc4b000cbb0 con 0x7fc4c00cee10 2024-03-29T09:55:44.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.403+0000 7fc4b7400700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc4a006ac90 0x7fc4a006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.403+0000 7fc4b7400700 1 -- 172.21.15.12:0/2673673748 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4b00892e0 con 0x7fc4c00cee10 2024-03-29T09:55:44.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.403+0000 7fc4be000700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc4a006ac90 0x7fc4a006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.404+0000 7fc4be000700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc4a006ac90 0x7fc4a006d150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc4c006cfa0 tx=0x7fc4ac009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:44.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.408+0000 7fc4b7400700 1 -- 172.21.15.12:0/2673673748 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4b00544c0 con 0x7fc4c00cee10 2024-03-29T09:55:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.651+0000 7fc4c5e00700 1 -- 172.21.15.12:0/2673673748 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4c00cdf00 con 0x7fc4a006ac90 2024-03-29T09:55:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.653+0000 7fc4b7400700 1 -- 172.21.15.12:0/2673673748 <== mgr.4114 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc4ac001480 con 0x7fc4a006ac90 2024-03-29T09:55:44.652 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T09:55:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T09:55:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T09:55:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T09:55:44.653 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T09:55:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 -- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc4a006ac90 msgr2=0x7fc4a006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc4a006ac90 0x7fc4a006d150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc4c006cfa0 tx=0x7fc4ac009040 comp rx=0 tx=0).stop 2024-03-29T09:55:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 -- 172.21.15.12:0/2673673748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 msgr2=0x7fc4c00b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 0x7fc4c00b6150 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000e870 tx=0x7fc4b000ec30 comp rx=0 tx=0).stop 2024-03-29T09:55:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 -- 172.21.15.12:0/2673673748 shutdown_connections 2024-03-29T09:55:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc4a006ac90 0x7fc4a006d150 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4c01611b0 0x7fc4c00c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4c00cee10 0x7fc4c00b6150 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 --2- 172.21.15.12:0/2673673748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4c015f4d0 0x7fc4c01ea550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 -- 172.21.15.12:0/2673673748 >> 172.21.15.12:0/2673673748 conn(0x7fc4c008aff0 msgr2=0x7fc4c015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.655+0000 7fc4c5e00700 1 -- 172.21.15.12:0/2673673748 shutdown_connections 2024-03-29T09:55:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.656+0000 7fc4c5e00700 1 -- 172.21.15.12:0/2673673748 wait complete. 2024-03-29T09:55:44.673 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T09:55:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.873+0000 7faea1c00700 1 Processor -- start 2024-03-29T09:55:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.873+0000 7faea1c00700 1 -- start start 2024-03-29T09:55:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.874+0000 7faea1c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c160d70 0x7fae9c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.874+0000 7faea1c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fae9c15ef70 0x7fae9c15e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.875+0000 7faea1c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c161150 con 0x7fae9c06a790 2024-03-29T09:55:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.875+0000 7faea1c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c0b4cd0 con 0x7fae9c15ef70 2024-03-29T09:55:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.875+0000 7faea1c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c06dba0 con 0x7fae9c160d70 2024-03-29T09:55:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.875+0000 7fae9aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c160d70 0x7fae9c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.875+0000 7fae9a000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fae9c15ef70 0x7fae9c15e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.875+0000 7fae9aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c160d70 0x7fae9c0b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:52696/0 (socket says 172.21.15.12:52696) 2024-03-29T09:55:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.875+0000 7fae9aa00700 1 -- 172.21.15.12:0/659439430 learned_addr learned my addr 172.21.15.12:0/659439430 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.875+0000 7fae9a000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fae9c15ef70 0x7fae9c15e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:48110/0 (socket says 172.21.15.12:48110) 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.876+0000 7fae9aa00700 1 -- 172.21.15.12:0/659439430 >> v2:172.21.15.159:3300/0 conn(0x7fae9c15ef70 msgr2=0x7fae9c15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.876+0000 7fae9aa00700 1 --2- 172.21.15.12:0/659439430 >> v2:172.21.15.159:3300/0 conn(0x7fae9c15ef70 0x7fae9c15e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.876+0000 7fae9aa00700 1 -- 172.21.15.12:0/659439430 >> v1:172.21.15.159:6789/0 conn(0x7fae9c06a790 legacy=0x7fae9c1613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.876+0000 7fae9aa00700 1 -- 172.21.15.12:0/659439430 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c15ed30 con 0x7fae9c160d70 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.876+0000 7fae9aa00700 1 --2- 172.21.15.12:0/659439430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c160d70 0x7fae9c0b3200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0098f0 tx=0x7fae8c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=667f6bdc550872f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.876+0000 7fae99600700 1 -- 172.21.15.12:0/659439430 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c011830 con 0x7fae9c160d70 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.876+0000 7fae99600700 1 -- 172.21.15.12:0/659439430 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c011990 con 0x7fae9c160d70 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.876+0000 7fae99600700 1 -- 172.21.15.12:0/659439430 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c011d20 con 0x7fae9c160d70 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.877+0000 7faea1c00700 1 -- 172.21.15.12:0/659439430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c160d70 msgr2=0x7fae9c0b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.877+0000 7faea1c00700 1 --2- 172.21.15.12:0/659439430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c160d70 0x7fae9c0b3200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0098f0 tx=0x7fae8c009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.877+0000 7faea1c00700 1 -- 172.21.15.12:0/659439430 shutdown_connections 2024-03-29T09:55:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.877+0000 7faea1c00700 1 --2- 172.21.15.12:0/659439430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c160d70 0x7fae9c0b3200 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.877+0000 7faea1c00700 1 --2- 172.21.15.12:0/659439430 >> v2:172.21.15.159:3300/0 conn(0x7fae9c15ef70 0x7fae9c15e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.877+0000 7faea1c00700 1 -- 172.21.15.12:0/659439430 >> 172.21.15.12:0/659439430 conn(0x7fae9c0b5720 msgr2=0x7fae9c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.877+0000 7faea1c00700 1 -- 172.21.15.12:0/659439430 shutdown_connections 2024-03-29T09:55:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.877+0000 7faea1c00700 1 -- 172.21.15.12:0/659439430 wait complete. 2024-03-29T09:55:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.879+0000 7faea1c00700 1 Processor -- start 2024-03-29T09:55:44.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.879+0000 7faea1c00700 1 -- start start 2024-03-29T09:55:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7faea1c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae9c06a790 0x7fae9c15d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7faea1c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c15ef70 0x7fae9c09b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7faea1c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae9c160d70 0x7fae9c138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7faea1c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c138250 con 0x7fae9c160d70 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7faea1c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0ca160 con 0x7fae9c06a790 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7faea1c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c1394b0 con 0x7fae9c15ef70 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7fae9b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae9c160d70 0x7fae9c138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7fae9b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae9c160d70 0x7fae9c138670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35600/0 (socket says 172.21.15.12:35600) 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7fae9b400700 1 -- 172.21.15.12:0/3664558649 learned_addr learned my addr 172.21.15.12:0/3664558649 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7fae9a000700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c15ef70 0x7fae9c09b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7fae9aa00700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae9c06a790 0x7fae9c15d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7fae9b400700 1 -- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c15ef70 msgr2=0x7fae9c09b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7fae9b400700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c15ef70 0x7fae9c09b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7fae9b400700 1 -- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae9c06a790 msgr2=0x7fae9c15d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7fae9b400700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae9c06a790 0x7fae9c15d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.880+0000 7fae9b400700 1 -- 172.21.15.12:0/3664558649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c009550 con 0x7fae9c160d70 2024-03-29T09:55:44.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.881+0000 7fae9b400700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae9c160d70 0x7fae9c138670 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fae9400cc50 tx=0x7fae9400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:44.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.881+0000 7fae92a00700 1 -- 172.21.15.12:0/3664558649 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94003c00 con 0x7fae9c160d70 2024-03-29T09:55:44.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.881+0000 7fae92a00700 1 -- 172.21.15.12:0/3664558649 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae94003d60 con 0x7fae9c160d70 2024-03-29T09:55:44.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.881+0000 7fae92a00700 1 -- 172.21.15.12:0/3664558649 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9400aab0 con 0x7fae9c160d70 2024-03-29T09:55:44.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.881+0000 7faea1c00700 1 -- 172.21.15.12:0/3664558649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c1378b0 con 0x7fae9c160d70 2024-03-29T09:55:44.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.881+0000 7faea1c00700 1 -- 172.21.15.12:0/3664558649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c137e70 con 0x7fae9c160d70 2024-03-29T09:55:44.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.882+0000 7faea1c00700 1 -- 172.21.15.12:0/3664558649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c138390 con 0x7fae9c160d70 2024-03-29T09:55:44.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.883+0000 7fae92a00700 1 -- 172.21.15.12:0/3664558649 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fae9400ac10 con 0x7fae9c160d70 2024-03-29T09:55:44.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.884+0000 7fae92a00700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae7c06ac90 0x7fae7c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:44.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.884+0000 7fae92a00700 1 -- 172.21.15.12:0/3664558649 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae94089370 con 0x7fae9c160d70 2024-03-29T09:55:44.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.885+0000 7fae9aa00700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae7c06ac90 0x7fae7c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:44.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.885+0000 7fae9aa00700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae7c06ac90 0x7fae7c06d150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fae8c000c00 tx=0x7fae8c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:44.889+0000 7fae92a00700 1 -- 172.21.15.12:0/3664558649 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae9c138390 con 0x7fae9c160d70 2024-03-29T09:55:45.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.168+0000 7faea1c00700 1 -- 172.21.15.12:0/3664558649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fae9c138390 con 0x7fae9c160d70 2024-03-29T09:55:45.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.949+0000 7fae92a00700 1 -- 172.21.15.12:0/3664558649 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7fae94089080 con 0x7fae9c160d70 2024-03-29T09:55:45.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.952+0000 7fae92a00700 1 -- 172.21.15.12:0/3664558649 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7fae940721c0 con 0x7fae9c160d70 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 -- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae7c06ac90 msgr2=0x7fae7c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae7c06ac90 0x7fae7c06d150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fae8c000c00 tx=0x7fae8c008030 comp rx=0 tx=0).stop 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 -- 172.21.15.12:0/3664558649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae9c160d70 msgr2=0x7fae9c138670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae9c160d70 0x7fae9c138670 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fae9400cc50 tx=0x7fae9400cf60 comp rx=0 tx=0).stop 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 -- 172.21.15.12:0/3664558649 shutdown_connections 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae7c06ac90 0x7fae7c06d150 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae9c160d70 0x7fae9c138670 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae9c06a790 0x7fae9c15d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 --2- 172.21.15.12:0/3664558649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae9c15ef70 0x7fae9c09b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 -- 172.21.15.12:0/3664558649 >> 172.21.15.12:0/3664558649 conn(0x7fae9c0b5720 msgr2=0x7fae9c144de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:45.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 -- 172.21.15.12:0/3664558649 shutdown_connections 2024-03-29T09:55:45.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:45.955+0000 7faea1c00700 1 -- 172.21.15.12:0/3664558649 wait complete. 2024-03-29T09:55:45.977 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T09:55:46.174 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:46.174+0000 7fe2b9d30240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T09:55:46.178 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:46.179+0000 7f8a540ff240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T09:55:46.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.188+0000 7f6984000700 1 Processor -- start 2024-03-29T09:55:46.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.189+0000 7f6984000700 1 -- start start 2024-03-29T09:55:46.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.190+0000 7f6984000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f697c0d16a0 0x7f697c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:46.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.190+0000 7f6984000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c161420 con 0x7f697c15d9d0 2024-03-29T09:55:46.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.190+0000 7f6984000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c161700 con 0x7f697c0d2370 2024-03-29T09:55:46.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.190+0000 7f6984000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697c0d4340 con 0x7f697c0d16a0 2024-03-29T09:55:46.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.190+0000 7f6980c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f697c0d16a0 0x7f697c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:46.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.190+0000 7f6980c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f697c0d16a0 0x7f697c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35634/0 (socket says 172.21.15.12:35634) 2024-03-29T09:55:46.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.190+0000 7f6980c00700 1 -- 172.21.15.12:0/2026197717 learned_addr learned my addr 172.21.15.12:0/2026197717 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:46.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.190+0000 7f6981600700 1 --1- 172.21.15.12:0/2026197717 >> v1:172.21.15.12:6789/0 conn(0x7f697c15d9d0 0x7f697c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56894/0 (socket says 172.21.15.12:56894) 2024-03-29T09:55:46.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.190+0000 7f6980c00700 1 -- 172.21.15.12:0/2026197717 >> v1:172.21.15.159:6789/0 conn(0x7f697c0d2370 legacy=0x7f697c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.191+0000 7f6980c00700 1 -- 172.21.15.12:0/2026197717 >> v1:172.21.15.12:6789/0 conn(0x7f697c15d9d0 legacy=0x7f697c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.191+0000 7f6980c00700 1 -- 172.21.15.12:0/2026197717 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c1619e0 con 0x7f697c0d16a0 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.191+0000 7f6980c00700 1 --2- 172.21.15.12:0/2026197717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 0x7f697c0d1e30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6968006fd0 tx=0x7f6968009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778b0be6b54dbe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.191+0000 7f697be00700 1 -- 172.21.15.12:0/2026197717 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968005e30 con 0x7f697c0d16a0 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.191+0000 7f697be00700 1 -- 172.21.15.12:0/2026197717 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6968004030 con 0x7f697c0d16a0 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.191+0000 7f697be00700 1 -- 172.21.15.12:0/2026197717 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69680043e0 con 0x7f697c0d16a0 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.192+0000 7f6984000700 1 -- 172.21.15.12:0/2026197717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 msgr2=0x7f697c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.192+0000 7f6984000700 1 --2- 172.21.15.12:0/2026197717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 0x7f697c0d1e30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6968006fd0 tx=0x7f6968009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.192+0000 7f6984000700 1 -- 172.21.15.12:0/2026197717 shutdown_connections 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.192+0000 7f6984000700 1 --2- 172.21.15.12:0/2026197717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 0x7f697c0d1e30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.192+0000 7f6984000700 1 -- 172.21.15.12:0/2026197717 >> 172.21.15.12:0/2026197717 conn(0x7f697c08aff0 msgr2=0x7f697c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.192+0000 7f6984000700 1 -- 172.21.15.12:0/2026197717 shutdown_connections 2024-03-29T09:55:46.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.192+0000 7f6984000700 1 -- 172.21.15.12:0/2026197717 wait complete. 2024-03-29T09:55:46.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.194+0000 7f6984000700 1 Processor -- start 2024-03-29T09:55:46.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.194+0000 7f6984000700 1 -- start start 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6984000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 0x7f697c155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6984000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f697c0d2370 0x7f697c0b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6984000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f697c15d9d0 0x7f697c0b4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6984000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c0b8820 con 0x7f697c0d16a0 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6984000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c0b7c10 con 0x7f697c15d9d0 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6984000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697c0b7ef0 con 0x7f697c0d2370 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6981600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 0x7f697c155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6981600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 0x7f697c155d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35648/0 (socket says 172.21.15.12:35648) 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6981600700 1 -- 172.21.15.12:0/1063838713 learned_addr learned my addr 172.21.15.12:0/1063838713 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6982000700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f697c15d9d0 0x7f697c0b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6980c00700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f697c0d2370 0x7f697c0b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:46.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6981600700 1 -- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f697c0d2370 msgr2=0x7f697c0b7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.195+0000 7f6981600700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f697c0d2370 0x7f697c0b7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.196+0000 7f6981600700 1 -- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f697c15d9d0 msgr2=0x7f697c0b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.196+0000 7f6981600700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f697c15d9d0 0x7f697c0b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.196+0000 7f6981600700 1 -- 172.21.15.12:0/1063838713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6968009500 con 0x7f697c0d16a0 2024-03-29T09:55:46.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.196+0000 7f6981600700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 0x7f697c155d90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f697000c410 tx=0x7f697000c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:46.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.196+0000 7f6979600700 1 -- 172.21.15.12:0/1063838713 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6970005cb0 con 0x7f697c0d16a0 2024-03-29T09:55:46.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.196+0000 7f6979600700 1 -- 172.21.15.12:0/1063838713 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6970005e10 con 0x7f697c0d16a0 2024-03-29T09:55:46.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.196+0000 7f6979600700 1 -- 172.21.15.12:0/1063838713 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69700072d0 con 0x7f697c0d16a0 2024-03-29T09:55:46.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.196+0000 7f6984000700 1 -- 172.21.15.12:0/1063838713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697c0b81d0 con 0x7f697c0d16a0 2024-03-29T09:55:46.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.196+0000 7f6984000700 1 -- 172.21.15.12:0/1063838713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697c160c60 con 0x7f697c0d16a0 2024-03-29T09:55:46.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.197+0000 7f6984000700 1 -- 172.21.15.12:0/1063838713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697c0d1ca0 con 0x7f697c0d16a0 2024-03-29T09:55:46.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.198+0000 7f6979600700 1 -- 172.21.15.12:0/1063838713 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7f6970007430 con 0x7f697c0d16a0 2024-03-29T09:55:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.200+0000 7f6979600700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f695c06ace0 0x7f695c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.200+0000 7f6979600700 1 -- 172.21.15.12:0/1063838713 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69700885c0 con 0x7f697c0d16a0 2024-03-29T09:55:46.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.200+0000 7f6980c00700 1 -- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f695c06ace0 msgr2=0x7f695c06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T09:55:46.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.200+0000 7f6980c00700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f695c06ace0 0x7f695c06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T09:55:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.205+0000 7f6979600700 1 -- 172.21.15.12:0/1063838713 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69700537b0 con 0x7f697c0d16a0 2024-03-29T09:55:46.360 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:46.360+0000 7fe2b9d30240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T09:55:46.385 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:46.385+0000 7f8a540ff240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T09:55:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.401+0000 7f6980c00700 1 -- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f695c06ace0 msgr2=0x7f695c06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T09:55:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.401+0000 7f6980c00700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f695c06ace0 0x7f695c06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T09:55:46.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.460+0000 7f6984000700 1 -- 172.21.15.12:0/1063838713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f697c067720 con 0x7f697c0d16a0 2024-03-29T09:55:46.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.465+0000 7f6979600700 1 -- 172.21.15.12:0/1063838713 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6970056dd0 con 0x7f697c0d16a0 2024-03-29T09:55:46.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 -- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f695c06ace0 msgr2=0x7f695c06d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f695c06ace0 0x7f695c06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 -- 172.21.15.12:0/1063838713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 msgr2=0x7f697c155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 0x7f697c155d90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f697000c410 tx=0x7f697000c720 comp rx=0 tx=0).stop 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 -- 172.21.15.12:0/1063838713 shutdown_connections 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f695c06ace0 0x7f695c06d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f697c15d9d0 0x7f697c0b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f697c0d16a0 0x7f697c155d90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 --2- 172.21.15.12:0/1063838713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f697c0d2370 0x7f697c0b7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 -- 172.21.15.12:0/1063838713 >> 172.21.15.12:0/1063838713 conn(0x7f697c08aff0 msgr2=0x7f697c0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 -- 172.21.15.12:0/1063838713 shutdown_connections 2024-03-29T09:55:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.468+0000 7f6984000700 1 -- 172.21.15.12:0/1063838713 wait complete. 2024-03-29T09:55:46.486 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T09:55:46.680 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:46.681+0000 7fe2b9d30240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T09:55:46.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.690+0000 7f9c6ce00700 1 Processor -- start 2024-03-29T09:55:46.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.691+0000 7f9c6ce00700 1 -- start start 2024-03-29T09:55:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.692+0000 7f9c6ce00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9c680cbb30 0x7f9c680cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.692+0000 7f9c6ce00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c680cc450 con 0x7f9c680cd480 2024-03-29T09:55:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.692+0000 7f9c6ce00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c680cdae0 con 0x7f9c680cd020 2024-03-29T09:55:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.692+0000 7f9c6ce00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c680cd8f0 con 0x7f9c680cbb30 2024-03-29T09:55:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.692+0000 7f9c66800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9c680cd480 0x7f9c6806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56908/0 (socket says 172.21.15.12:56908) 2024-03-29T09:55:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.692+0000 7f9c66800700 1 -- 172.21.15.12:0/4255587157 learned_addr learned my addr 172.21.15.12:0/4255587157 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.692+0000 7f9c65e00700 1 --2- 172.21.15.12:0/4255587157 >> v2:172.21.15.159:3300/0 conn(0x7f9c680cbb30 0x7f9c680cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:46.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.693+0000 7f9c64a00700 1 -- 172.21.15.12:0/4255587157 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416133984 0 0) 0x7f9c680cc450 con 0x7f9c680cd480 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.693+0000 7f9c65e00700 1 -- 172.21.15.12:0/4255587157 >> v1:172.21.15.159:6789/0 conn(0x7f9c680cd020 legacy=0x7f9c680b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.693+0000 7f9c65e00700 1 -- 172.21.15.12:0/4255587157 >> v1:172.21.15.12:6789/0 conn(0x7f9c680cd480 legacy=0x7f9c6806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.693+0000 7f9c65e00700 1 -- 172.21.15.12:0/4255587157 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c680cddc0 con 0x7f9c680cbb30 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.693+0000 7f9c65e00700 1 --2- 172.21.15.12:0/4255587157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c680cbb30 0x7f9c680cbf10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009840 tx=0x7f9c58009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74c17fd0445863c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.693+0000 7f9c64a00700 1 -- 172.21.15.12:0/4255587157 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58011830 con 0x7f9c680cbb30 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.693+0000 7f9c64a00700 1 -- 172.21.15.12:0/4255587157 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c58011990 con 0x7f9c680cbb30 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.693+0000 7f9c64a00700 1 -- 172.21.15.12:0/4255587157 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58011d20 con 0x7f9c680cbb30 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.694+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4255587157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c680cbb30 msgr2=0x7f9c680cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.694+0000 7f9c6ce00700 1 --2- 172.21.15.12:0/4255587157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c680cbb30 0x7f9c680cbf10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009840 tx=0x7f9c58009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.694+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4255587157 shutdown_connections 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.694+0000 7f9c6ce00700 1 --2- 172.21.15.12:0/4255587157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c680cbb30 0x7f9c680cbf10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.694+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4255587157 >> 172.21.15.12:0/4255587157 conn(0x7f9c680b0d70 msgr2=0x7f9c680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:46.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.694+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4255587157 shutdown_connections 2024-03-29T09:55:46.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.694+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4255587157 wait complete. 2024-03-29T09:55:46.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.696+0000 7f9c6ce00700 1 Processor -- start 2024-03-29T09:55:46.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.696+0000 7f9c6ce00700 1 -- start start 2024-03-29T09:55:46.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c6ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c680cbb30 0x7f9c6814e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:46.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c6ce00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c680cd020 0x7f9c680cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:46.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c6ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c680cd480 0x7f9c680b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:46.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c6ce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c681560e0 con 0x7f9c680cd480 2024-03-29T09:55:46.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c6ce00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c680b2210 con 0x7f9c680cd020 2024-03-29T09:55:46.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c6ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c680b24f0 con 0x7f9c680cbb30 2024-03-29T09:55:46.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c66800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c680cd480 0x7f9c680b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c66800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c680cd480 0x7f9c680b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35658/0 (socket says 172.21.15.12:35658) 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c66800700 1 -- 172.21.15.12:0/4019250077 learned_addr learned my addr 172.21.15.12:0/4019250077 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c65400700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c680cd020 0x7f9c680cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.697+0000 7f9c65e00700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c680cbb30 0x7f9c6814e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.698+0000 7f9c66800700 1 -- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c680cbb30 msgr2=0x7f9c6814e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.698+0000 7f9c66800700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c680cbb30 0x7f9c6814e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.698+0000 7f9c66800700 1 -- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c680cd020 msgr2=0x7f9c680cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.698+0000 7f9c66800700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c680cd020 0x7f9c680cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.698+0000 7f9c66800700 1 -- 172.21.15.12:0/4019250077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c580094a0 con 0x7f9c680cd480 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.698+0000 7f9c66800700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c680cd480 0x7f9c680b3130 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9c60002230 tx=0x7f9c6000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.698+0000 7f9c5e000700 1 -- 172.21.15.12:0/4019250077 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6000aac0 con 0x7f9c680cd480 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.698+0000 7f9c5e000700 1 -- 172.21.15.12:0/4019250077 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6000ac40 con 0x7f9c680cd480 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.698+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4019250077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c680b27d0 con 0x7f9c680cd480 2024-03-29T09:55:46.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.698+0000 7f9c5e000700 1 -- 172.21.15.12:0/4019250077 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c600070d0 con 0x7f9c680cd480 2024-03-29T09:55:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.699+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4019250077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c680b2d90 con 0x7f9c680cd480 2024-03-29T09:55:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.700+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4019250077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c68067720 con 0x7f9c680cd480 2024-03-29T09:55:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.700+0000 7f9c5e000700 1 -- 172.21.15.12:0/4019250077 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7f9c60007250 con 0x7f9c680cd480 2024-03-29T09:55:46.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.702+0000 7f9c5e000700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c4806ace0 0x7f9c4806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:46.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.702+0000 7f9c5e000700 1 -- 172.21.15.12:0/4019250077 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c60089220 con 0x7f9c680cd480 2024-03-29T09:55:46.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.702+0000 7f9c65e00700 1 -- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c4806ace0 msgr2=0x7f9c4806d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T09:55:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.702+0000 7f9c65e00700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c4806ace0 0x7f9c4806d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T09:55:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.706+0000 7f9c5e000700 1 -- 172.21.15.12:0/4019250077 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9c60054360 con 0x7f9c680cd480 2024-03-29T09:55:46.728 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:46.729+0000 7f8a540ff240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T09:55:46.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.903+0000 7f9c65e00700 1 -- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c4806ace0 msgr2=0x7f9c4806d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T09:55:46.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.903+0000 7f9c65e00700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c4806ace0 0x7f9c4806d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T09:55:46.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.921+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4019250077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9c680ce2e0 con 0x7f9c680cd480 2024-03-29T09:55:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.927+0000 7f9c5e000700 1 -- 172.21.15.12:0/4019250077 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9c60057980 con 0x7f9c680cd480 2024-03-29T09:55:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c4806ace0 msgr2=0x7f9c4806d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T09:55:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c4806ace0 0x7f9c4806d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4019250077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c680cd480 msgr2=0x7f9c680b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c680cd480 0x7f9c680b3130 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9c60002230 tx=0x7f9c6000cf50 comp rx=0 tx=0).stop 2024-03-29T09:55:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4019250077 shutdown_connections 2024-03-29T09:55:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c4806ace0 0x7f9c4806d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c680cd480 0x7f9c680b3130 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c680cbb30 0x7f9c6814e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 --2- 172.21.15.12:0/4019250077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c680cd020 0x7f9c680cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4019250077 >> 172.21.15.12:0/4019250077 conn(0x7f9c680b0d70 msgr2=0x7f9c6815a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:46.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4019250077 shutdown_connections 2024-03-29T09:55:46.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:46.929+0000 7f9c6ce00700 1 -- 172.21.15.12:0/4019250077 wait complete. 2024-03-29T09:55:46.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:46.947 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-29T09:55:46.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:46.997 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2024-03-29T09:55:47.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2024-03-29T09:55:47.215 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:47.215+0000 7fe2b9d30240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T09:55:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.249+0000 7fd520800700 1 Processor -- start 2024-03-29T09:55:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.249+0000 7fd520800700 1 -- start start 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.250+0000 7fd520800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d16a0 0x7fd5180d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.250+0000 7fd520800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd5180d2370 0x7fd5180ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.250+0000 7fd520800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd518161420 con 0x7fd51815d9d0 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.250+0000 7fd520800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5180d2820 con 0x7fd5180d2370 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.250+0000 7fd520800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5180d4340 con 0x7fd5180d16a0 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.250+0000 7fd51de00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd51815d9d0 0x7fd5180d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56910/0 (socket says 172.21.15.12:56910) 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.250+0000 7fd51de00700 1 -- 172.21.15.12:0/4246126130 learned_addr learned my addr 172.21.15.12:0/4246126130 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.251+0000 7fd51d400700 1 --2- 172.21.15.12:0/4246126130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d16a0 0x7fd5180d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.251+0000 7fd51e800700 1 --2- 172.21.15.12:0/4246126130 >> v2:172.21.15.159:3300/0 conn(0x7fd5180d2370 0x7fd5180ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.251+0000 7fd51ca00700 1 -- 172.21.15.12:0/4246126130 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088230943 0 0) 0x7fd518161420 con 0x7fd51815d9d0 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.251+0000 7fd51ca00700 1 -- 172.21.15.12:0/4246126130 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5000035b0 con 0x7fd51815d9d0 2024-03-29T09:55:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.251+0000 7fd51d400700 1 -- 172.21.15.12:0/4246126130 >> v2:172.21.15.159:3300/0 conn(0x7fd5180d2370 msgr2=0x7fd5180ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.251+0000 7fd51d400700 1 --2- 172.21.15.12:0/4246126130 >> v2:172.21.15.159:3300/0 conn(0x7fd5180d2370 0x7fd5180ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.251+0000 7fd51d400700 1 -- 172.21.15.12:0/4246126130 >> v1:172.21.15.12:6789/0 conn(0x7fd51815d9d0 legacy=0x7fd5180d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.251+0000 7fd51d400700 1 -- 172.21.15.12:0/4246126130 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd518161700 con 0x7fd5180d16a0 2024-03-29T09:55:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.251+0000 7fd51e800700 1 --2- 172.21.15.12:0/4246126130 >> v2:172.21.15.159:3300/0 conn(0x7fd5180d2370 0x7fd5180ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.251+0000 7fd51d400700 1 --2- 172.21.15.12:0/4246126130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d16a0 0x7fd5180d1e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd508009820 tx=0x7fd508009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=265060663490ba15 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd51ca00700 1 -- 172.21.15.12:0/4246126130 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508005e30 con 0x7fd5180d16a0 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd51ca00700 1 -- 172.21.15.12:0/4246126130 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd508006030 con 0x7fd5180d16a0 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd51ca00700 1 -- 172.21.15.12:0/4246126130 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5080063c0 con 0x7fd5180d16a0 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd520800700 1 -- 172.21.15.12:0/4246126130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d16a0 msgr2=0x7fd5180d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd520800700 1 --2- 172.21.15.12:0/4246126130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d16a0 0x7fd5180d1e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd508009820 tx=0x7fd508009b30 comp rx=0 tx=0).stop 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd520800700 1 -- 172.21.15.12:0/4246126130 shutdown_connections 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd520800700 1 --2- 172.21.15.12:0/4246126130 >> v2:172.21.15.159:3300/0 conn(0x7fd5180d2370 0x7fd5180ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd520800700 1 --2- 172.21.15.12:0/4246126130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d16a0 0x7fd5180d1e30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd520800700 1 -- 172.21.15.12:0/4246126130 >> 172.21.15.12:0/4246126130 conn(0x7fd51808aff0 msgr2=0x7fd5180d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd520800700 1 -- 172.21.15.12:0/4246126130 shutdown_connections 2024-03-29T09:55:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.252+0000 7fd520800700 1 -- 172.21.15.12:0/4246126130 wait complete. 2024-03-29T09:55:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.254+0000 7fd520800700 1 Processor -- start 2024-03-29T09:55:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.254+0000 7fd520800700 1 -- start start 2024-03-29T09:55:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.255+0000 7fd520800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5180d16a0 0x7fd5180c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:47.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.255+0000 7fd520800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d2370 0x7fd5180b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.255+0000 7fd520800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd51815d9d0 0x7fd5180b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.255+0000 7fd520800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51814ab70 con 0x7fd5180d16a0 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.255+0000 7fd520800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51814ae50 con 0x7fd51815d9d0 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.255+0000 7fd520800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51814b130 con 0x7fd5180d2370 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.255+0000 7fd51de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5180d16a0 0x7fd5180c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd51de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5180d16a0 0x7fd5180c94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35668/0 (socket says 172.21.15.12:35668) 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd51de00700 1 -- 172.21.15.12:0/1013903646 learned_addr learned my addr 172.21.15.12:0/1013903646 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd51d400700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d2370 0x7fd5180b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd51e800700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd51815d9d0 0x7fd5180b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd51de00700 1 -- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d2370 msgr2=0x7fd5180b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd51de00700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d2370 0x7fd5180b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd51de00700 1 -- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd51815d9d0 msgr2=0x7fd5180b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd51de00700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd51815d9d0 0x7fd5180b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd51de00700 1 -- 172.21.15.12:0/1013903646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd514009a70 con 0x7fd5180d16a0 2024-03-29T09:55:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd51de00700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5180d16a0 0x7fd5180c94c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd50c001eb0 tx=0x7fd50c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.256+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c00cea0 con 0x7fd5180d16a0 2024-03-29T09:55:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.257+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50c005d70 con 0x7fd5180d16a0 2024-03-29T09:55:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.257+0000 7fd520800700 1 -- 172.21.15.12:0/1013903646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd508009430 con 0x7fd5180d16a0 2024-03-29T09:55:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.257+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0072f0 con 0x7fd5180d16a0 2024-03-29T09:55:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.257+0000 7fd520800700 1 -- 172.21.15.12:0/1013903646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51814b6f0 con 0x7fd5180d16a0 2024-03-29T09:55:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.258+0000 7fd520800700 1 -- 172.21.15.12:0/1013903646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5180d1ca0 con 0x7fd5180d16a0 2024-03-29T09:55:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.259+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7fd50c007450 con 0x7fd5180d16a0 2024-03-29T09:55:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.260+0000 7fd512000700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.260+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd50c088730 con 0x7fd5180d16a0 2024-03-29T09:55:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.260+0000 7fd51d400700 1 -- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 msgr2=0x7fd4fc06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T09:55:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.260+0000 7fd51d400700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T09:55:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.264+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd50c053910 con 0x7fd5180d16a0 2024-03-29T09:55:47.313 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:47.314+0000 7f8a540ff240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T09:55:47.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.461+0000 7fd51d400700 1 -- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 msgr2=0x7fd4fc06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T09:55:47.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.461+0000 7fd51d400700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T09:55:47.470 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:47.470+0000 7fe2b9d30240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T09:55:47.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.504+0000 7fd520800700 1 -- 172.21.15.12:0/1013903646 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd518161fa0 con 0x7fd4fc06ace0 2024-03-29T09:55:47.578 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:47.578+0000 7f8a540ff240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T09:55:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.862+0000 7fd51d400700 1 -- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 msgr2=0x7fd4fc06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T09:55:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:47.862+0000 7fd51d400700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T09:55:47.927 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:47.927+0000 7fe2b9d30240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T09:55:48.053 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:48.053+0000 7f8a540ff240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T09:55:48.639 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:48.639+0000 7fe2b9d30240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T09:55:48.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:48.663+0000 7fd51d400700 1 -- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 msgr2=0x7fd4fc06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T09:55:48.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:48.663+0000 7fd51d400700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T09:55:48.730 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:48.730+0000 7fe2b9d30240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T09:55:48.791 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:48.792+0000 7f8a540ff240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T09:55:48.886 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:48.886+0000 7f8a540ff240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T09:55:48.890 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:48.891+0000 7fe2b9d30240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T09:55:49.052 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:49.053+0000 7f8a540ff240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T09:55:49.103 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:49.103+0000 7fe2b9d30240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T09:55:49.193 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:49.193+0000 7fe2b9d30240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T09:55:49.268 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:49.269+0000 7f8a540ff240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T09:55:49.290 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:49.291+0000 7fe2b9d30240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T09:55:49.363 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:49.363+0000 7f8a540ff240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T09:55:49.388 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:49.388+0000 7fe2b9d30240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T09:55:49.465 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:49.466+0000 7f8a540ff240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T09:55:49.486 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:49.486+0000 7fe2b9d30240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T09:55:49.566 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:49.567+0000 7f8a540ff240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T09:55:49.667 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:49.668+0000 7f8a540ff240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T09:55:49.695 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:49.695+0000 7fe2b9d30240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T09:55:49.886 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:49.887+0000 7f8a540ff240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T09:55:49.950 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:49.951+0000 7fe2b9d30240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T09:55:50.138 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:50.139+0000 7fe2b9d30240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T09:55:50.147 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:50.147+0000 7f8a540ff240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T09:55:50.229 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:50.230+0000 7fe2b9d30240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T09:55:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:50.265+0000 7fd51d400700 1 -- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 msgr2=0x7fd4fc06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T09:55:50.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:50.265+0000 7fd51d400700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T09:55:50.344 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:50.345+0000 7f8a540ff240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T09:55:50.440 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:50.441+0000 7f8a540ff240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T09:55:50.959 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:50.959+0000 7fe2b9d30240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T09:55:51.168 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:51.168+0000 7fe2b9d30240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T09:55:51.199 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:51.200+0000 7f8a540ff240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T09:55:51.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:51.372+0000 7fe2b9d30240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T09:55:51.416 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:51.417+0000 7f8a540ff240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T09:55:51.631 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:51.631+0000 7f8a540ff240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T09:55:51.843 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:51.843+0000 7fe2b9d30240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T09:55:52.046 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.046+0000 7fe2b9d30240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T09:55:52.124 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:52.125+0000 7f8a540ff240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T09:55:52.335 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:52.336+0000 7f8a540ff240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T09:55:52.485 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.485+0000 7fe2b9d30240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T09:55:52.593 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.593+0000 7fe2b9d30240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T09:55:52.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:52.607+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 7) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fd50c053ce0 con 0x7fd5180d16a0 2024-03-29T09:55:52.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:52.608+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 msgr2=0x7fd4fc06d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T09:55:52.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:52.608+0000 7fd512000700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:52.803 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:52.803+0000 7f8a540ff240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T09:55:52.823 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.823+0000 7fe222400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:55:52.823 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.823+0000 7fe222400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:55:52.823 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.823+0000 7fe222400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:55:52.823 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.823+0000 7fe222400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:55:52.824 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.823+0000 7fe222400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:55:52.824 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.824+0000 7fe226000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:55:52.824 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.824+0000 7fe226000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:55:52.825 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.824+0000 7fe226000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:55:52.825 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.824+0000 7fe226000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:55:52.825 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:55:52.824+0000 7fe226000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:55:52.919 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T09:55:52.920+0000 7f8a540ff240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T09:55:52.919 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:55:52.923 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:55:53.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.612+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 8) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fd50c05f530 con 0x7fd5180d16a0 2024-03-29T09:55:53.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.612+0000 7fd512000700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:53.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.612+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4fc01d470 con 0x7fd4fc06ace0 2024-03-29T09:55:53.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.615+0000 7fd51d400700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:53.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.619+0000 7fd51d400700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd508000c00 tx=0x7fd508008070 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:53.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.737+0000 7fd512000700 1 -- 172.21.15.12:0/1013903646 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd4fc01d470 con 0x7fd4fc06ace0 2024-03-29T09:55:53.737 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/dir0 2024-03-29T09:55:53.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.739+0000 7fd520800700 1 -- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 msgr2=0x7fd4fc06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:53.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.739+0000 7fd520800700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd508000c00 tx=0x7fd508008070 comp rx=0 tx=0).stop 2024-03-29T09:55:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.739+0000 7fd520800700 1 -- 172.21.15.12:0/1013903646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5180d16a0 msgr2=0x7fd5180c94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.739+0000 7fd520800700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5180d16a0 0x7fd5180c94c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd50c001eb0 tx=0x7fd50c00c7b0 comp rx=0 tx=0).stop 2024-03-29T09:55:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.739+0000 7fd520800700 1 -- 172.21.15.12:0/1013903646 shutdown_connections 2024-03-29T09:55:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.739+0000 7fd520800700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4fc06ace0 0x7fd4fc06d1a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.739+0000 7fd520800700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd51815d9d0 0x7fd5180b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.739+0000 7fd520800700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5180d16a0 0x7fd5180c94c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.739+0000 7fd520800700 1 --2- 172.21.15.12:0/1013903646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5180d2370 0x7fd5180b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.739+0000 7fd520800700 1 -- 172.21.15.12:0/1013903646 >> 172.21.15.12:0/1013903646 conn(0x7fd51808aff0 msgr2=0x7fd5180cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.740+0000 7fd520800700 1 -- 172.21.15.12:0/1013903646 shutdown_connections 2024-03-29T09:55:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:53.740+0000 7fd520800700 1 -- 172.21.15.12:0/1013903646 wait complete. 2024-03-29T09:55:53.758 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2024-03-29T09:55:53.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:53.759 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2024-03-29T09:55:53.805 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2024-03-29T09:55:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.007+0000 7fdd7d000700 1 Processor -- start 2024-03-29T09:55:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.007+0000 7fdd7d000700 1 -- start start 2024-03-29T09:55:54.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.008+0000 7fdd7d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d16a0 0x7fdd780d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.008+0000 7fdd7d000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdd780d2370 0x7fdd780ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd7d000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd78161420 con 0x7fdd7815d9d0 2024-03-29T09:55:54.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd7d000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd780d2820 con 0x7fdd780d2370 2024-03-29T09:55:54.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd7d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd780d4340 con 0x7fdd780d16a0 2024-03-29T09:55:54.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd76800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdd780d2370 0x7fdd780ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd75400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d16a0 0x7fdd780d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd75e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fdd7815d9d0 0x7fdd780d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:47988/0 (socket says 172.21.15.12:47988) 2024-03-29T09:55:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd75e00700 1 -- 172.21.15.12:0/975699217 learned_addr learned my addr 172.21.15.12:0/975699217 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd76800700 1 -- 172.21.15.12:0/975699217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d16a0 msgr2=0x7fdd780d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd76800700 1 --2- 172.21.15.12:0/975699217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d16a0 0x7fdd780d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd76800700 1 -- 172.21.15.12:0/975699217 >> v1:172.21.15.159:6789/0 conn(0x7fdd7815d9d0 legacy=0x7fdd780d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd76800700 1 -- 172.21.15.12:0/975699217 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd78161700 con 0x7fdd780d2370 2024-03-29T09:55:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.009+0000 7fdd74a00700 1 -- 172.21.15.12:0/975699217 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221397456 0 0) 0x7fdd78161420 con 0x7fdd7815d9d0 2024-03-29T09:55:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.010+0000 7fdd76800700 1 --2- 172.21.15.12:0/975699217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd780d2370 0x7fdd780ce360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009dc0 tx=0x7fdd7000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66d71af0ccf88cc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.010+0000 7fdd74a00700 1 -- 172.21.15.12:0/975699217 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7000e070 con 0x7fdd780d2370 2024-03-29T09:55:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.010+0000 7fdd74a00700 1 -- 172.21.15.12:0/975699217 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7000bae0 con 0x7fdd780d2370 2024-03-29T09:55:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.010+0000 7fdd74a00700 1 -- 172.21.15.12:0/975699217 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7000be90 con 0x7fdd780d2370 2024-03-29T09:55:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.010+0000 7fdd7d000700 1 -- 172.21.15.12:0/975699217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd780d2370 msgr2=0x7fdd780ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.010+0000 7fdd7d000700 1 --2- 172.21.15.12:0/975699217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd780d2370 0x7fdd780ce360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009dc0 tx=0x7fdd7000b330 comp rx=0 tx=0).stop 2024-03-29T09:55:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.011+0000 7fdd7d000700 1 -- 172.21.15.12:0/975699217 shutdown_connections 2024-03-29T09:55:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.011+0000 7fdd7d000700 1 --2- 172.21.15.12:0/975699217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd780d2370 0x7fdd780ce360 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.011+0000 7fdd7d000700 1 --2- 172.21.15.12:0/975699217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d16a0 0x7fdd780d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.011+0000 7fdd7d000700 1 -- 172.21.15.12:0/975699217 >> 172.21.15.12:0/975699217 conn(0x7fdd7808aff0 msgr2=0x7fdd780d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.011+0000 7fdd7d000700 1 -- 172.21.15.12:0/975699217 shutdown_connections 2024-03-29T09:55:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.011+0000 7fdd7d000700 1 -- 172.21.15.12:0/975699217 wait complete. 2024-03-29T09:55:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.012+0000 7fdd7d000700 1 Processor -- start 2024-03-29T09:55:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.013+0000 7fdd7d000700 1 -- start start 2024-03-29T09:55:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.013+0000 7fdd7d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd780d16a0 0x7fdd781f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.013+0000 7fdd7d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d2370 0x7fdd7815f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.013+0000 7fdd7d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd7815d9d0 0x7fdd780b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd7d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd78160a80 con 0x7fdd7815d9d0 2024-03-29T09:55:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd7d000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd781f64e0 con 0x7fdd780d16a0 2024-03-29T09:55:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd7d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7815fe00 con 0x7fdd780d2370 2024-03-29T09:55:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd76800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd7815d9d0 0x7fdd780b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd76800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd7815d9d0 0x7fdd780b45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39402/0 (socket says 172.21.15.12:39402) 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd76800700 1 -- 172.21.15.12:0/1916944424 learned_addr learned my addr 172.21.15.12:0/1916944424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd75e00700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd780d16a0 0x7fdd781f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd75400700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d2370 0x7fdd7815f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd76800700 1 -- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d2370 msgr2=0x7fdd7815f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd76800700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d2370 0x7fdd7815f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd76800700 1 -- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd780d16a0 msgr2=0x7fdd781f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd76800700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd780d16a0 0x7fdd781f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd76800700 1 -- 172.21.15.12:0/1916944424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd70009a70 con 0x7fdd7815d9d0 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd75400700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d2370 0x7fdd7815f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.014+0000 7fdd75e00700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd780d16a0 0x7fdd781f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:54.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.015+0000 7fdd76800700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd7815d9d0 0x7fdd780b45b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdd7808c230 tx=0x7fdd70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:54.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.015+0000 7fdd6e000700 1 -- 172.21.15.12:0/1916944424 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd70027680 con 0x7fdd7815d9d0 2024-03-29T09:55:54.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.015+0000 7fdd6e000700 1 -- 172.21.15.12:0/1916944424 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd70020070 con 0x7fdd7815d9d0 2024-03-29T09:55:54.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.015+0000 7fdd7d000700 1 -- 172.21.15.12:0/1916944424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd781600e0 con 0x7fdd7815d9d0 2024-03-29T09:55:54.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.015+0000 7fdd6e000700 1 -- 172.21.15.12:0/1916944424 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd70027a20 con 0x7fdd7815d9d0 2024-03-29T09:55:54.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.015+0000 7fdd7d000700 1 -- 172.21.15.12:0/1916944424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd781606a0 con 0x7fdd7815d9d0 2024-03-29T09:55:54.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.016+0000 7fdd7d000700 1 -- 172.21.15.12:0/1916944424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd780d1ca0 con 0x7fdd7815d9d0 2024-03-29T09:55:54.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.017+0000 7fdd6e000700 1 -- 172.21.15.12:0/1916944424 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fdd7001b400 con 0x7fdd7815d9d0 2024-03-29T09:55:54.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.018+0000 7fdd6e000700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd5806ab30 0x7fdd5806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.018+0000 7fdd6e000700 1 -- 172.21.15.12:0/1916944424 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdd70022070 con 0x7fdd7815d9d0 2024-03-29T09:55:54.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.018+0000 7fdd75e00700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd5806ab30 0x7fdd5806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.019+0000 7fdd75e00700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd5806ab30 0x7fdd5806cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdd68000f60 tx=0x7fdd68007d30 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:54.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.023+0000 7fdd6e000700 1 -- 172.21.15.12:0/1916944424 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd70059d20 con 0x7fdd7815d9d0 2024-03-29T09:55:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.261+0000 7fdd7d000700 1 -- 172.21.15.12:0/1916944424 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd78161fa0 con 0x7fdd5806ab30 2024-03-29T09:55:54.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.347+0000 7fdd6e000700 1 -- 172.21.15.12:0/1916944424 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fdd68004870 con 0x7fdd5806ab30 2024-03-29T09:55:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/dir1 2024-03-29T09:55:54.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 -- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd5806ab30 msgr2=0x7fdd5806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:54.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd5806ab30 0x7fdd5806cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdd68000f60 tx=0x7fdd68007d30 comp rx=0 tx=0).stop 2024-03-29T09:55:54.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 -- 172.21.15.12:0/1916944424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd7815d9d0 msgr2=0x7fdd780b45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:54.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd7815d9d0 0x7fdd780b45b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdd7808c230 tx=0x7fdd70004ad0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 -- 172.21.15.12:0/1916944424 shutdown_connections 2024-03-29T09:55:54.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd5806ab30 0x7fdd5806cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd7815d9d0 0x7fdd780b45b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd780d16a0 0x7fdd781f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 --2- 172.21.15.12:0/1916944424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd780d2370 0x7fdd7815f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 -- 172.21.15.12:0/1916944424 >> 172.21.15.12:0/1916944424 conn(0x7fdd7808aff0 msgr2=0x7fdd7815ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:54.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 -- 172.21.15.12:0/1916944424 shutdown_connections 2024-03-29T09:55:54.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.350+0000 7fdd7d000700 1 -- 172.21.15.12:0/1916944424 wait complete. 2024-03-29T09:55:54.368 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2024-03-29T09:55:54.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:54.368 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2024-03-29T09:55:54.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2024-03-29T09:55:54.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.657+0000 7f0572200700 1 Processor -- start 2024-03-29T09:55:54.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.657+0000 7f0572200700 1 -- start start 2024-03-29T09:55:54.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.658+0000 7f0572200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c0cd5c0 0x7f056c15e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.658+0000 7f0572200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f056c15cfe0 0x7f056c15d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.658+0000 7f0572200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c15eda0 con 0x7f056c06f310 2024-03-29T09:55:54.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.658+0000 7f0572200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f056c099dd0 con 0x7f056c15cfe0 2024-03-29T09:55:54.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.658+0000 7f0572200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c09a0b0 con 0x7f056c0cd5c0 2024-03-29T09:55:54.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.659+0000 7f056be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f056c15cfe0 0x7f056c15d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.659+0000 7f056be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f056c15cfe0 0x7f056c15d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39416/0 (socket says 172.21.15.12:39416) 2024-03-29T09:55:54.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.659+0000 7f056be00700 1 -- 172.21.15.12:0/997937548 learned_addr learned my addr 172.21.15.12:0/997937548 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:54.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.659+0000 7f056b400700 1 --2- 172.21.15.12:0/997937548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c0cd5c0 0x7f056c15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.659+0000 7f056b400700 1 -- 172.21.15.12:0/997937548 >> v2:172.21.15.12:3300/0 conn(0x7f056c15cfe0 msgr2=0x7f056c15d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:54.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.659+0000 7f056b400700 1 --2- 172.21.15.12:0/997937548 >> v2:172.21.15.12:3300/0 conn(0x7f056c15cfe0 0x7f056c15d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.659+0000 7f056b400700 1 -- 172.21.15.12:0/997937548 >> v1:172.21.15.159:6789/0 conn(0x7f056c06f310 legacy=0x7f056c06f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:54.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.660+0000 7f056b400700 1 -- 172.21.15.12:0/997937548 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c09a390 con 0x7f056c0cd5c0 2024-03-29T09:55:54.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.660+0000 7f056b400700 1 --2- 172.21.15.12:0/997937548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c0cd5c0 0x7f056c15e860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f055c009840 tx=0x7f055c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d13d52605ce49883 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:54.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.667+0000 7f056a000700 1 -- 172.21.15.12:0/997937548 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c011830 con 0x7f056c0cd5c0 2024-03-29T09:55:54.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.667+0000 7f056a000700 1 -- 172.21.15.12:0/997937548 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055c011990 con 0x7f056c0cd5c0 2024-03-29T09:55:54.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.667+0000 7f056a000700 1 -- 172.21.15.12:0/997937548 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c011d20 con 0x7f056c0cd5c0 2024-03-29T09:55:54.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.668+0000 7f0572200700 1 -- 172.21.15.12:0/997937548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c0cd5c0 msgr2=0x7f056c15e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:54.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.668+0000 7f0572200700 1 --2- 172.21.15.12:0/997937548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c0cd5c0 0x7f056c15e860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f055c009840 tx=0x7f055c009b50 comp rx=0 tx=0).stop 2024-03-29T09:55:54.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.668+0000 7f0572200700 1 -- 172.21.15.12:0/997937548 shutdown_connections 2024-03-29T09:55:54.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.668+0000 7f0572200700 1 --2- 172.21.15.12:0/997937548 >> v2:172.21.15.12:3300/0 conn(0x7f056c15cfe0 0x7f056c15d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.668+0000 7f0572200700 1 --2- 172.21.15.12:0/997937548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c0cd5c0 0x7f056c15e860 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.668+0000 7f0572200700 1 -- 172.21.15.12:0/997937548 >> 172.21.15.12:0/997937548 conn(0x7f056c0d1370 msgr2=0x7f056c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:54.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.668+0000 7f0572200700 1 -- 172.21.15.12:0/997937548 shutdown_connections 2024-03-29T09:55:54.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.668+0000 7f0572200700 1 -- 172.21.15.12:0/997937548 wait complete. 2024-03-29T09:55:54.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.670+0000 7f0572200700 1 Processor -- start 2024-03-29T09:55:54.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.670+0000 7f0572200700 1 -- start start 2024-03-29T09:55:54.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f0572200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f056c06f310 0x7f056c152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f0572200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f056c0cd5c0 0x7f056c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f0572200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c15cfe0 0x7f056c06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f0572200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0c9dd0 con 0x7f056c06f310 2024-03-29T09:55:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f0572200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0c9250 con 0x7f056c0cd5c0 2024-03-29T09:55:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f0572200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0c9530 con 0x7f056c15cfe0 2024-03-29T09:55:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f056b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f056c06f310 0x7f056c152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f056b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f056c06f310 0x7f056c152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39418/0 (socket says 172.21.15.12:39418) 2024-03-29T09:55:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f056b400700 1 -- 172.21.15.12:0/4243107281 learned_addr learned my addr 172.21.15.12:0/4243107281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f056be00700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c15cfe0 0x7f056c06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.671+0000 7f056aa00700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f056c0cd5c0 0x7f056c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.672+0000 7f056b400700 1 -- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c15cfe0 msgr2=0x7f056c06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.672+0000 7f056b400700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c15cfe0 0x7f056c06ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.672+0000 7f056b400700 1 -- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f056c0cd5c0 msgr2=0x7f056c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:54.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.672+0000 7f056b400700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f056c0cd5c0 0x7f056c06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:54.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.672+0000 7f056b400700 1 -- 172.21.15.12:0/4243107281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055c0094a0 con 0x7f056c06f310 2024-03-29T09:55:54.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.672+0000 7f056b400700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f056c06f310 0x7f056c152770 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f055c000c00 tx=0x7f055c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:54.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.672+0000 7f0563400700 1 -- 172.21.15.12:0/4243107281 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c0036a0 con 0x7f056c06f310 2024-03-29T09:55:54.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.672+0000 7f0563400700 1 -- 172.21.15.12:0/4243107281 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055c01f070 con 0x7f056c06f310 2024-03-29T09:55:54.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.672+0000 7f0563400700 1 -- 172.21.15.12:0/4243107281 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c003a60 con 0x7f056c06f310 2024-03-29T09:55:54.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.672+0000 7f0572200700 1 -- 172.21.15.12:0/4243107281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c0c9810 con 0x7f056c06f310 2024-03-29T09:55:54.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.673+0000 7f0572200700 1 -- 172.21.15.12:0/4243107281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c152cb0 con 0x7f056c06f310 2024-03-29T09:55:54.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.674+0000 7f0572200700 1 -- 172.21.15.12:0/4243107281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c067720 con 0x7f056c06f310 2024-03-29T09:55:54.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.675+0000 7f0563400700 1 -- 172.21.15.12:0/4243107281 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f055c003bc0 con 0x7f056c06f310 2024-03-29T09:55:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.675+0000 7f0563400700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f054c06ae00 0x7f054c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.676+0000 7f0563400700 1 -- 172.21.15.12:0/4243107281 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f055c021070 con 0x7f056c06f310 2024-03-29T09:55:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.676+0000 7f056aa00700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f054c06ae00 0x7f054c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.676+0000 7f056aa00700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f054c06ae00 0x7f054c06d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f056c0b7650 tx=0x7f0558009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:54.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.680+0000 7f0563400700 1 -- 172.21.15.12:0/4243107281 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f055c055e40 con 0x7f056c06f310 2024-03-29T09:55:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:54.923+0000 7f0572200700 1 -- 172.21.15.12:0/4243107281 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f056c099a60 con 0x7f054c06ae00 2024-03-29T09:55:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.015+0000 7f0563400700 1 -- 172.21.15.12:0/4243107281 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0558001d60 con 0x7f054c06ae00 2024-03-29T09:55:55.015 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/dir2 2024-03-29T09:55:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.017+0000 7f0572200700 1 -- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f054c06ae00 msgr2=0x7f054c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.017+0000 7f0572200700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f054c06ae00 0x7f054c06d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f056c0b7650 tx=0x7f0558009040 comp rx=0 tx=0).stop 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.017+0000 7f0572200700 1 -- 172.21.15.12:0/4243107281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f056c06f310 msgr2=0x7f056c152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.017+0000 7f0572200700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f056c06f310 0x7f056c152770 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f055c000c00 tx=0x7f055c004750 comp rx=0 tx=0).stop 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.018+0000 7f0572200700 1 -- 172.21.15.12:0/4243107281 shutdown_connections 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.018+0000 7f0572200700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f054c06ae00 0x7f054c06d2c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.018+0000 7f0572200700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f056c15cfe0 0x7f056c06ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.018+0000 7f0572200700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f056c06f310 0x7f056c152770 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.018+0000 7f0572200700 1 --2- 172.21.15.12:0/4243107281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f056c0cd5c0 0x7f056c06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.018+0000 7f0572200700 1 -- 172.21.15.12:0/4243107281 >> 172.21.15.12:0/4243107281 conn(0x7f056c0d1370 msgr2=0x7f056c0d1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.018+0000 7f0572200700 1 -- 172.21.15.12:0/4243107281 shutdown_connections 2024-03-29T09:55:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.018+0000 7f0572200700 1 -- 172.21.15.12:0/4243107281 wait complete. 2024-03-29T09:55:55.036 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2024-03-29T09:55:55.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:55.036 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2024-03-29T09:55:55.082 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2024-03-29T09:55:55.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.284+0000 7f608a200700 1 Processor -- start 2024-03-29T09:55:55.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.285+0000 7f608a200700 1 -- start start 2024-03-29T09:55:55.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.286+0000 7f608a200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f608415d9d0 0x7f60840d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.286+0000 7f608a200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f60840d2540 0x7f60840ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.286+0000 7f608a200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6084161420 con 0x7f60840d16a0 2024-03-29T09:55:55.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.286+0000 7f608a200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f608415edc0 con 0x7f60840d2540 2024-03-29T09:55:55.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.286+0000 7f608a200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6084161700 con 0x7f608415d9d0 2024-03-29T09:55:55.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.286+0000 7f6082a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f60840d16a0 0x7f60840d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60770/0 (socket says 172.21.15.12:60770) 2024-03-29T09:55:55.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.286+0000 7f6082a00700 1 -- 172.21.15.12:0/1250485114 learned_addr learned my addr 172.21.15.12:0/1250485114 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:55.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.286+0000 7f6083400700 1 --2- 172.21.15.12:0/1250485114 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f608415d9d0 0x7f60840d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.286+0000 7f6083e00700 1 --2- 172.21.15.12:0/1250485114 >> v2:172.21.15.159:3300/0 conn(0x7f60840d2540 0x7f60840ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.287+0000 7f6082000700 1 -- 172.21.15.12:0/1250485114 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59558532 0 0) 0x7f6084161420 con 0x7f60840d16a0 2024-03-29T09:55:55.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.287+0000 7f6082000700 1 -- 172.21.15.12:0/1250485114 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60680035b0 con 0x7f60840d16a0 2024-03-29T09:55:55.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.287+0000 7f6083400700 1 -- 172.21.15.12:0/1250485114 >> v2:172.21.15.159:3300/0 conn(0x7f60840d2540 msgr2=0x7f60840ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:55.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.287+0000 7f6083400700 1 --2- 172.21.15.12:0/1250485114 >> v2:172.21.15.159:3300/0 conn(0x7f60840d2540 0x7f60840ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.287+0000 7f6082000700 1 -- 172.21.15.12:0/1250485114 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 971292178 0 0) 0x7f60680035b0 con 0x7f60840d16a0 2024-03-29T09:55:55.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.287+0000 7f6083400700 1 -- 172.21.15.12:0/1250485114 >> v1:172.21.15.12:6789/0 conn(0x7f60840d16a0 legacy=0x7f60840d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.287+0000 7f6083400700 1 -- 172.21.15.12:0/1250485114 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60841619e0 con 0x7f608415d9d0 2024-03-29T09:55:55.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.287+0000 7f6083400700 1 --2- 172.21.15.12:0/1250485114 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f608415d9d0 0x7f60840d3c30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6074009870 tx=0x7f6074009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78af7b54aed7e99c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:55.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f6082000700 1 -- 172.21.15.12:0/1250485114 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60700017f0 con 0x7f608415d9d0 2024-03-29T09:55:55.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f6082000700 1 -- 172.21.15.12:0/1250485114 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6074004030 con 0x7f608415d9d0 2024-03-29T09:55:55.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f6082000700 1 -- 172.21.15.12:0/1250485114 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60740043a0 con 0x7f608415d9d0 2024-03-29T09:55:55.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f608a200700 1 -- 172.21.15.12:0/1250485114 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f608415d9d0 msgr2=0x7f60840d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f608a200700 1 --2- 172.21.15.12:0/1250485114 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f608415d9d0 0x7f60840d3c30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6074009870 tx=0x7f6074009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:55.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f608a200700 1 -- 172.21.15.12:0/1250485114 shutdown_connections 2024-03-29T09:55:55.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f608a200700 1 --2- 172.21.15.12:0/1250485114 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f608415d9d0 0x7f60840d3c30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f608a200700 1 --2- 172.21.15.12:0/1250485114 >> v2:172.21.15.159:3300/0 conn(0x7f60840d2540 0x7f60840ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f608a200700 1 -- 172.21.15.12:0/1250485114 >> 172.21.15.12:0/1250485114 conn(0x7f608408aff0 msgr2=0x7f60840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:55.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f608a200700 1 -- 172.21.15.12:0/1250485114 shutdown_connections 2024-03-29T09:55:55.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.288+0000 7f608a200700 1 -- 172.21.15.12:0/1250485114 wait complete. 2024-03-29T09:55:55.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.290+0000 7f608a200700 1 Processor -- start 2024-03-29T09:55:55.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.290+0000 7f608a200700 1 -- start start 2024-03-29T09:55:55.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.291+0000 7f608a200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60840d16a0 0x7f608415c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.291+0000 7f608a200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60840d2540 0x7f60840b78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.291+0000 7f608a200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608415d9d0 0x7f60840b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.291+0000 7f608a200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840cb8c0 con 0x7f608415d9d0 2024-03-29T09:55:55.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.291+0000 7f608a200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840cacf0 con 0x7f60840d16a0 2024-03-29T09:55:55.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.291+0000 7f608a200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60840caf90 con 0x7f60840d2540 2024-03-29T09:55:55.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.291+0000 7f6083e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608415d9d0 0x7f60840b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.291+0000 7f6083e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608415d9d0 0x7f60840b7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39424/0 (socket says 172.21.15.12:39424) 2024-03-29T09:55:55.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.291+0000 7f6083e00700 1 -- 172.21.15.12:0/1998085200 learned_addr learned my addr 172.21.15.12:0/1998085200 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:55.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f6083400700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60840d16a0 0x7f608415c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f6082a00700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60840d2540 0x7f60840b78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f6083e00700 1 -- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60840d2540 msgr2=0x7f60840b78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f6083e00700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60840d2540 0x7f60840b78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f6083e00700 1 -- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60840d16a0 msgr2=0x7f608415c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f6083e00700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60840d16a0 0x7f608415c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f6083e00700 1 -- 172.21.15.12:0/1998085200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60740094a0 con 0x7f608415d9d0 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f6083400700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60840d16a0 0x7f608415c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f6082a00700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60840d2540 0x7f60840b78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f6083e00700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608415d9d0 0x7f60840b7df0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f607c00ef90 tx=0x7f607c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f607b400700 1 -- 172.21.15.12:0/1998085200 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c009ca0 con 0x7f608415d9d0 2024-03-29T09:55:55.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f607b400700 1 -- 172.21.15.12:0/1998085200 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607c00f040 con 0x7f608415d9d0 2024-03-29T09:55:55.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f607b400700 1 -- 172.21.15.12:0/1998085200 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c014480 con 0x7f608415d9d0 2024-03-29T09:55:55.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f608a200700 1 -- 172.21.15.12:0/1998085200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60840cb270 con 0x7f608415d9d0 2024-03-29T09:55:55.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.292+0000 7f608a200700 1 -- 172.21.15.12:0/1998085200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608415c860 con 0x7f608415d9d0 2024-03-29T09:55:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.294+0000 7f608a200700 1 -- 172.21.15.12:0/1998085200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60840d1ca0 con 0x7f608415d9d0 2024-03-29T09:55:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.294+0000 7f607b400700 1 -- 172.21.15.12:0/1998085200 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f607c0145e0 con 0x7f608415d9d0 2024-03-29T09:55:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.295+0000 7f607b400700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f606406ae00 0x7f606406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.295+0000 7f607b400700 1 -- 172.21.15.12:0/1998085200 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f607c08a4f0 con 0x7f608415d9d0 2024-03-29T09:55:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.296+0000 7f6083400700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f606406ae00 0x7f606406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.296+0000 7f6083400700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f606406ae00 0x7f606406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60740090b0 tx=0x7f6074013040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:55.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.300+0000 7f607b400700 1 -- 172.21.15.12:0/1998085200 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f607c055280 con 0x7f608415d9d0 2024-03-29T09:55:55.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.541+0000 7f608a200700 1 -- 172.21.15.12:0/1998085200 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6084161fa0 con 0x7f606406ae00 2024-03-29T09:55:55.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.625+0000 7f607b400700 1 -- 172.21.15.12:0/1998085200 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f60740220c0 con 0x7f606406ae00 2024-03-29T09:55:55.625 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/dir3 2024-03-29T09:55:55.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.627+0000 7f608a200700 1 -- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f606406ae00 msgr2=0x7f606406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.627+0000 7f608a200700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f606406ae00 0x7f606406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60740090b0 tx=0x7f6074013040 comp rx=0 tx=0).stop 2024-03-29T09:55:55.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.627+0000 7f608a200700 1 -- 172.21.15.12:0/1998085200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608415d9d0 msgr2=0x7f60840b7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.627+0000 7f608a200700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608415d9d0 0x7f60840b7df0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f607c00ef90 tx=0x7f607c00c3a0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.627+0000 7f608a200700 1 -- 172.21.15.12:0/1998085200 shutdown_connections 2024-03-29T09:55:55.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.627+0000 7f608a200700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f606406ae00 0x7f606406d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.627+0000 7f608a200700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608415d9d0 0x7f60840b7df0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.627+0000 7f608a200700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60840d16a0 0x7f608415c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.627+0000 7f608a200700 1 --2- 172.21.15.12:0/1998085200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60840d2540 0x7f60840b78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.627+0000 7f608a200700 1 -- 172.21.15.12:0/1998085200 >> 172.21.15.12:0/1998085200 conn(0x7f608408aff0 msgr2=0x7f608406c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:55.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.628+0000 7f608a200700 1 -- 172.21.15.12:0/1998085200 shutdown_connections 2024-03-29T09:55:55.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.628+0000 7f608a200700 1 -- 172.21.15.12:0/1998085200 wait complete. 2024-03-29T09:55:55.645 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2024-03-29T09:55:55.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:55.645 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2024-03-29T09:55:55.691 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2024-03-29T09:55:55.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.932+0000 7fd314400700 1 Processor -- start 2024-03-29T09:55:55.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.933+0000 7fd314400700 1 -- start start 2024-03-29T09:55:55.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.934+0000 7fd314400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c0ced80 0x7fd30c0d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.934+0000 7fd314400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c0cf1e0 con 0x7fd30c15fcb0 2024-03-29T09:55:55.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.934+0000 7fd314400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c15f820 con 0x7fd30c15dad0 2024-03-29T09:55:55.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.934+0000 7fd314400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c0cfcc0 con 0x7fd30c0ced80 2024-03-29T09:55:55.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.934+0000 7fd312400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd30c15fcb0 0x7fd30c160140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60784/0 (socket says 172.21.15.12:60784) 2024-03-29T09:55:55.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.934+0000 7fd312400700 1 -- 172.21.15.12:0/1864169333 learned_addr learned my addr 172.21.15.12:0/1864169333 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:55.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.934+0000 7fd311000700 1 --2- 172.21.15.12:0/1864169333 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c0ced80 0x7fd30c0d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.934+0000 7fd30be00700 1 -- 172.21.15.12:0/1864169333 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157347885 0 0) 0x7fd304001490 con 0x7fd30c15fcb0 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.935+0000 7fd30be00700 1 -- 172.21.15.12:0/1864169333 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f40035b0 con 0x7fd30c15fcb0 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.935+0000 7fd30be00700 1 -- 172.21.15.12:0/1864169333 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178855530 0 0) 0x7fd30c15f820 con 0x7fd30c15dad0 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.935+0000 7fd30be00700 1 -- 172.21.15.12:0/1864169333 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd304001490 con 0x7fd30c15dad0 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.935+0000 7fd311000700 1 -- 172.21.15.12:0/1864169333 >> v1:172.21.15.159:6789/0 conn(0x7fd30c15dad0 legacy=0x7fd30c0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.935+0000 7fd311000700 1 -- 172.21.15.12:0/1864169333 >> v1:172.21.15.12:6789/0 conn(0x7fd30c15fcb0 legacy=0x7fd30c160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.935+0000 7fd311000700 1 -- 172.21.15.12:0/1864169333 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c0cf420 con 0x7fd30c0ced80 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.935+0000 7fd311000700 1 --2- 172.21.15.12:0/1864169333 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c0ced80 0x7fd30c0d1d40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009870 tx=0x7fd2f8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b8827e0a1c6151f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.936+0000 7fd30be00700 1 -- 172.21.15.12:0/1864169333 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8011830 con 0x7fd30c0ced80 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.936+0000 7fd30be00700 1 -- 172.21.15.12:0/1864169333 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f8011990 con 0x7fd30c0ced80 2024-03-29T09:55:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.936+0000 7fd30be00700 1 -- 172.21.15.12:0/1864169333 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8011d20 con 0x7fd30c0ced80 2024-03-29T09:55:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.936+0000 7fd314400700 1 -- 172.21.15.12:0/1864169333 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c0ced80 msgr2=0x7fd30c0d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.936+0000 7fd314400700 1 --2- 172.21.15.12:0/1864169333 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c0ced80 0x7fd30c0d1d40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009870 tx=0x7fd2f8009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.936+0000 7fd314400700 1 -- 172.21.15.12:0/1864169333 shutdown_connections 2024-03-29T09:55:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.936+0000 7fd314400700 1 --2- 172.21.15.12:0/1864169333 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c0ced80 0x7fd30c0d1d40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.936+0000 7fd314400700 1 -- 172.21.15.12:0/1864169333 >> 172.21.15.12:0/1864169333 conn(0x7fd30c08aff0 msgr2=0x7fd30c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.936+0000 7fd314400700 1 -- 172.21.15.12:0/1864169333 shutdown_connections 2024-03-29T09:55:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.937+0000 7fd314400700 1 -- 172.21.15.12:0/1864169333 wait complete. 2024-03-29T09:55:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.938+0000 7fd314400700 1 Processor -- start 2024-03-29T09:55:55.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.938+0000 7fd314400700 1 -- start start 2024-03-29T09:55:55.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.939+0000 7fd314400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd30c0ced80 0x7fd30c0c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.939+0000 7fd314400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd30c15dad0 0x7fd30c0c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.939+0000 7fd314400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c15fcb0 0x7fd30c0cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.939+0000 7fd314400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c0cb6e0 con 0x7fd30c0ced80 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.939+0000 7fd314400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c0c9ff0 con 0x7fd30c15dad0 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.939+0000 7fd314400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c0cb980 con 0x7fd30c15fcb0 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.939+0000 7fd311a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd30c0ced80 0x7fd30c0c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd311a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd30c0ced80 0x7fd30c0c7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39426/0 (socket says 172.21.15.12:39426) 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd311a00700 1 -- 172.21.15.12:0/3318638925 learned_addr learned my addr 172.21.15.12:0/3318638925 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd311000700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd30c15dad0 0x7fd30c0c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd312400700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c15fcb0 0x7fd30c0cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd311a00700 1 -- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c15fcb0 msgr2=0x7fd30c0cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd311a00700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c15fcb0 0x7fd30c0cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd311a00700 1 -- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd30c15dad0 msgr2=0x7fd30c0c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd311a00700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd30c15dad0 0x7fd30c0c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd311a00700 1 -- 172.21.15.12:0/3318638925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80094a0 con 0x7fd30c0ced80 2024-03-29T09:55:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd312400700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c15fcb0 0x7fd30c0cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.940+0000 7fd311a00700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd30c0ced80 0x7fd30c0c7450 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd30000c520 tx=0x7fd30000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.941+0000 7fd309600700 1 -- 172.21.15.12:0/3318638925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3000052d0 con 0x7fd30c0ced80 2024-03-29T09:55:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.941+0000 7fd309600700 1 -- 172.21.15.12:0/3318638925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd300005430 con 0x7fd30c0ced80 2024-03-29T09:55:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.941+0000 7fd309600700 1 -- 172.21.15.12:0/3318638925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3000038a0 con 0x7fd30c0ced80 2024-03-29T09:55:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.941+0000 7fd314400700 1 -- 172.21.15.12:0/3318638925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30c0cbc60 con 0x7fd30c0ced80 2024-03-29T09:55:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.941+0000 7fd314400700 1 -- 172.21.15.12:0/3318638925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30c0c9ca0 con 0x7fd30c0ced80 2024-03-29T09:55:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.942+0000 7fd314400700 1 -- 172.21.15.12:0/3318638925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30c0cbe50 con 0x7fd30c0ced80 2024-03-29T09:55:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.943+0000 7fd309600700 1 -- 172.21.15.12:0/3318638925 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fd300003a00 con 0x7fd30c0ced80 2024-03-29T09:55:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.944+0000 7fd309600700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2ec06ae00 0x7fd2ec06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.944+0000 7fd309600700 1 -- 172.21.15.12:0/3318638925 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fd300088ba0 con 0x7fd30c0ced80 2024-03-29T09:55:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.944+0000 7fd311000700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2ec06ae00 0x7fd2ec06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.945+0000 7fd311000700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2ec06ae00 0x7fd2ec06d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80097c0 tx=0x7fd2f801c040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:55.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:55.948+0000 7fd309600700 1 -- 172.21.15.12:0/3318638925 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd300053ad0 con 0x7fd30c0ced80 2024-03-29T09:55:56.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.188+0000 7fd314400700 1 -- 172.21.15.12:0/3318638925 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd30c0cf9e0 con 0x7fd2ec06ae00 2024-03-29T09:55:56.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.273+0000 7fd309600700 1 -- 172.21.15.12:0/3318638925 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd2f8011b00 con 0x7fd2ec06ae00 2024-03-29T09:55:56.273 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/dir4 2024-03-29T09:55:56.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.275+0000 7fd314400700 1 -- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2ec06ae00 msgr2=0x7fd2ec06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:56.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.275+0000 7fd314400700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2ec06ae00 0x7fd2ec06d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80097c0 tx=0x7fd2f801c040 comp rx=0 tx=0).stop 2024-03-29T09:55:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.275+0000 7fd314400700 1 -- 172.21.15.12:0/3318638925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd30c0ced80 msgr2=0x7fd30c0c7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.275+0000 7fd314400700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd30c0ced80 0x7fd30c0c7450 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd30000c520 tx=0x7fd30000c830 comp rx=0 tx=0).stop 2024-03-29T09:55:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.275+0000 7fd314400700 1 -- 172.21.15.12:0/3318638925 shutdown_connections 2024-03-29T09:55:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.275+0000 7fd314400700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2ec06ae00 0x7fd2ec06d2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.275+0000 7fd314400700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30c15fcb0 0x7fd30c0cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.275+0000 7fd314400700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd30c0ced80 0x7fd30c0c7450 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.275+0000 7fd314400700 1 --2- 172.21.15.12:0/3318638925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd30c15dad0 0x7fd30c0c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.275+0000 7fd314400700 1 -- 172.21.15.12:0/3318638925 >> 172.21.15.12:0/3318638925 conn(0x7fd30c08aff0 msgr2=0x7fd30c06d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.276+0000 7fd314400700 1 -- 172.21.15.12:0/3318638925 shutdown_connections 2024-03-29T09:55:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.276+0000 7fd314400700 1 -- 172.21.15.12:0/3318638925 wait complete. 2024-03-29T09:55:56.293 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2024-03-29T09:55:56.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:56.293 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2024-03-29T09:55:56.339 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2024-03-29T09:55:56.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.580+0000 7fdf80e00700 1 Processor -- start 2024-03-29T09:55:56.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.580+0000 7fdf80e00700 1 -- start start 2024-03-29T09:55:56.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.581+0000 7fdf80e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdf7c0b3b30 0x7fdf7c0cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:56.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.581+0000 7fdf80e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c0b2070 con 0x7fdf7c157f90 2024-03-29T09:55:56.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.581+0000 7fdf80e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c15e0f0 con 0x7fdf7c0b2b30 2024-03-29T09:55:56.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.581+0000 7fdf80e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7c158370 con 0x7fdf7c0b3b30 2024-03-29T09:55:56.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.581+0000 7fdf79e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdf7c157f90 0x7fdf7c1613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60786/0 (socket says 172.21.15.12:60786) 2024-03-29T09:55:56.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.581+0000 7fdf79e00700 1 -- 172.21.15.12:0/1117398303 learned_addr learned my addr 172.21.15.12:0/1117398303 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:56.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf7a800700 1 --2- 172.21.15.12:0/1117398303 >> v2:172.21.15.159:3300/0 conn(0x7fdf7c0b3b30 0x7fdf7c0cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:56.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486871245 0 0) 0x7fdf7c0b2070 con 0x7fdf7c157f90 2024-03-29T09:55:56.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf600035b0 con 0x7fdf7c157f90 2024-03-29T09:55:56.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 488625090 0 0) 0x7fdf600035b0 con 0x7fdf7c157f90 2024-03-29T09:55:56.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 >> v2:172.21.15.159:3300/0 conn(0x7fdf7c0b3b30 msgr2=0x7fdf7c0cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:56.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 --2- 172.21.15.12:0/1117398303 >> v2:172.21.15.159:3300/0 conn(0x7fdf7c0b3b30 0x7fdf7c0cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 >> v1:172.21.15.159:6789/0 conn(0x7fdf7c0b2b30 legacy=0x7fdf7c0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0b2fe0 con 0x7fdf7c157f90 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fdf6c001670 con 0x7fdf7c157f90 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 >> v1:172.21.15.12:6789/0 conn(0x7fdf7c157f90 legacy=0x7fdf7c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf60003a00 0x7fdf60003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf60004850 0x7fdf60004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.582+0000 7fdf78a00700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf60005210 0x7fdf6000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c158370 con 0x7fdf60005210 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6000c0b0 con 0x7fdf60004850 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6000c270 con 0x7fdf60003a00 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf79400700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf60004850 0x7fdf60004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf7a800700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf60005210 0x7fdf6000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf79e00700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf60003a00 0x7fdf60003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf7a800700 1 -- 172.21.15.12:0/1117398303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf60003a00 msgr2=0x7fdf60003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf7a800700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf60003a00 0x7fdf60003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf7a800700 1 -- 172.21.15.12:0/1117398303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf60004850 msgr2=0x7fdf60004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf7a800700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf60004850 0x7fdf60004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf7a800700 1 -- 172.21.15.12:0/1117398303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf600035b0 con 0x7fdf60005210 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf79400700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf60004850 0x7fdf60004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.583+0000 7fdf7a800700 1 -- 172.21.15.12:0/1117398303 reap_dead start 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.584+0000 7fdf7a800700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf60005210 0x7fdf6000ba10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009fc0 tx=0x7fdf60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8d9ad201299d43 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.584+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74003a20 con 0x7fdf60005210 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.584+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7400f040 con 0x7fdf60005210 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.584+0000 7fdf78a00700 1 -- 172.21.15.12:0/1117398303 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74003de0 con 0x7fdf60005210 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.584+0000 7fdf80e00700 1 -- 172.21.15.12:0/1117398303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf60005210 msgr2=0x7fdf6000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.584+0000 7fdf80e00700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf60005210 0x7fdf6000ba10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009fc0 tx=0x7fdf60005880 comp rx=0 tx=0).stop 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.584+0000 7fdf80e00700 1 -- 172.21.15.12:0/1117398303 shutdown_connections 2024-03-29T09:55:56.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.584+0000 7fdf80e00700 1 --2- 172.21.15.12:0/1117398303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf60005210 0x7fdf6000ba10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.584+0000 7fdf80e00700 1 -- 172.21.15.12:0/1117398303 >> 172.21.15.12:0/1117398303 conn(0x7fdf7c0b0d70 msgr2=0x7fdf7c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:56.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.585+0000 7fdf80e00700 1 -- 172.21.15.12:0/1117398303 shutdown_connections 2024-03-29T09:55:56.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.585+0000 7fdf80e00700 1 -- 172.21.15.12:0/1117398303 wait complete. 2024-03-29T09:55:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.586+0000 7fdf80e00700 1 Processor -- start 2024-03-29T09:55:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.586+0000 7fdf80e00700 1 -- start start 2024-03-29T09:55:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.587+0000 7fdf80e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf60005210 0x7fdf7c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.587+0000 7fdf80e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf7c0aea10 0x7fdf7c15a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.587+0000 7fdf80e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c1584b0 0x7fdf7c155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.587+0000 7fdf80e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0b2710 con 0x7fdf7c1584b0 2024-03-29T09:55:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.587+0000 7fdf80e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c15ac10 con 0x7fdf7c0aea10 2024-03-29T09:55:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.587+0000 7fdf80e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0aef80 con 0x7fdf60005210 2024-03-29T09:55:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf7a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c1584b0 0x7fdf7c155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf7a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c1584b0 0x7fdf7c155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39448/0 (socket says 172.21.15.12:39448) 2024-03-29T09:55:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf7a800700 1 -- 172.21.15.12:0/4266722389 learned_addr learned my addr 172.21.15.12:0/4266722389 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf79e00700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf60005210 0x7fdf7c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf79400700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf7c0aea10 0x7fdf7c15a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf7a800700 1 -- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf60005210 msgr2=0x7fdf7c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf7a800700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf60005210 0x7fdf7c06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf7a800700 1 -- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf7c0aea10 msgr2=0x7fdf7c15a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf7a800700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf7c0aea10 0x7fdf7c15a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf7a800700 1 -- 172.21.15.12:0/4266722389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0af260 con 0x7fdf7c1584b0 2024-03-29T09:55:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf79400700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf7c0aea10 0x7fdf7c15a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:56.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.588+0000 7fdf7a800700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c1584b0 0x7fdf7c155070 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c070950 tx=0x7fdf74009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:56.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.589+0000 7fdf72000700 1 -- 172.21.15.12:0/4266722389 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74003e10 con 0x7fdf7c1584b0 2024-03-29T09:55:56.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.589+0000 7fdf72000700 1 -- 172.21.15.12:0/4266722389 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf74004230 con 0x7fdf7c1584b0 2024-03-29T09:55:56.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.589+0000 7fdf72000700 1 -- 172.21.15.12:0/4266722389 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf740043e0 con 0x7fdf7c1584b0 2024-03-29T09:55:56.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.589+0000 7fdf80e00700 1 -- 172.21.15.12:0/4266722389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c0af540 con 0x7fdf7c1584b0 2024-03-29T09:55:56.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.589+0000 7fdf80e00700 1 -- 172.21.15.12:0/4266722389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c0afb00 con 0x7fdf7c1584b0 2024-03-29T09:55:56.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.590+0000 7fdf80e00700 1 -- 172.21.15.12:0/4266722389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c067720 con 0x7fdf7c1584b0 2024-03-29T09:55:56.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.591+0000 7fdf72000700 1 -- 172.21.15.12:0/4266722389 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fdf74004540 con 0x7fdf7c1584b0 2024-03-29T09:55:56.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.592+0000 7fdf72000700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdf5c06ae00 0x7fdf5c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:56.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.592+0000 7fdf72000700 1 -- 172.21.15.12:0/4266722389 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdf7408ac50 con 0x7fdf7c1584b0 2024-03-29T09:55:56.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.593+0000 7fdf79e00700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdf5c06ae00 0x7fdf5c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:56.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.593+0000 7fdf79e00700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdf5c06ae00 0x7fdf5c06d2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c000c00 tx=0x7fdf6c000ed0 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.596+0000 7fdf72000700 1 -- 172.21.15.12:0/4266722389 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf74055ba0 con 0x7fdf7c1584b0 2024-03-29T09:55:56.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.670+0000 7fdf72000700 1 -- 172.21.15.12:0/4266722389 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fdf740047f0 con 0x7fdf7c1584b0 2024-03-29T09:55:56.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.839+0000 7fdf80e00700 1 -- 172.21.15.12:0/4266722389 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf7c0b2330 con 0x7fdf5c06ae00 2024-03-29T09:55:56.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.934+0000 7fdf72000700 1 -- 172.21.15.12:0/4266722389 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fdf6c0015b0 con 0x7fdf5c06ae00 2024-03-29T09:55:56.934 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/dir5 2024-03-29T09:55:56.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 -- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdf5c06ae00 msgr2=0x7fdf5c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:56.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdf5c06ae00 0x7fdf5c06d2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c000c00 tx=0x7fdf6c000ed0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 -- 172.21.15.12:0/4266722389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c1584b0 msgr2=0x7fdf7c155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:56.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c1584b0 0x7fdf7c155070 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c070950 tx=0x7fdf74009750 comp rx=0 tx=0).stop 2024-03-29T09:55:56.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 -- 172.21.15.12:0/4266722389 shutdown_connections 2024-03-29T09:55:56.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdf5c06ae00 0x7fdf5c06d2c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c1584b0 0x7fdf7c155070 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf60005210 0x7fdf7c06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 --2- 172.21.15.12:0/4266722389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf7c0aea10 0x7fdf7c15a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:56.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 -- 172.21.15.12:0/4266722389 >> 172.21.15.12:0/4266722389 conn(0x7fdf7c0b0d70 msgr2=0x7fdf7c156be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:56.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 -- 172.21.15.12:0/4266722389 shutdown_connections 2024-03-29T09:55:56.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:56.937+0000 7fdf80e00700 1 -- 172.21.15.12:0/4266722389 wait complete. 2024-03-29T09:55:56.955 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2024-03-29T09:55:56.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:56.955 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2024-03-29T09:55:57.001 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2024-03-29T09:55:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.242+0000 7f7dbf200700 1 Processor -- start 2024-03-29T09:55:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.243+0000 7f7dbf200700 1 -- start start 2024-03-29T09:55:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7dbf200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7db8099a60 0x7f7db80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7dbf200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7db815d990 0x7f7db8099e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7dbf200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db80d0970 con 0x7f7db809a750 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7dbf200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db806c020 con 0x7f7db815d990 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7dbf200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db809a380 con 0x7f7db8099a60 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7db7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7db815d990 0x7f7db8099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7db7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7db815d990 0x7f7db8099e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39458/0 (socket says 172.21.15.12:39458) 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7db7400700 1 -- 172.21.15.12:0/952267808 learned_addr learned my addr 172.21.15.12:0/952267808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7db7e00700 1 --2- 172.21.15.12:0/952267808 >> v2:172.21.15.159:3300/0 conn(0x7f7db8099a60 0x7f7db80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7db7400700 1 -- 172.21.15.12:0/952267808 >> v2:172.21.15.159:3300/0 conn(0x7f7db8099a60 msgr2=0x7f7db80d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7db7400700 1 --2- 172.21.15.12:0/952267808 >> v2:172.21.15.159:3300/0 conn(0x7f7db8099a60 0x7f7db80d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.244+0000 7f7db7400700 1 -- 172.21.15.12:0/952267808 >> v1:172.21.15.159:6789/0 conn(0x7f7db809a750 legacy=0x7f7db806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.245+0000 7f7db7400700 1 -- 172.21.15.12:0/952267808 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db80c81a0 con 0x7f7db815d990 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.245+0000 7f7db7400700 1 --2- 172.21.15.12:0/952267808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 0x7f7db8099e40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7db8071f40 tx=0x7f7da4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5589c014375aabc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.245+0000 7f7db6a00700 1 -- 172.21.15.12:0/952267808 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da400b870 con 0x7f7db815d990 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.245+0000 7f7db6a00700 1 -- 172.21.15.12:0/952267808 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da400b9d0 con 0x7f7db815d990 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.245+0000 7f7db6a00700 1 -- 172.21.15.12:0/952267808 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da400bd60 con 0x7f7db815d990 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.245+0000 7f7dbf200700 1 -- 172.21.15.12:0/952267808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 msgr2=0x7f7db8099e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.245+0000 7f7dbf200700 1 --2- 172.21.15.12:0/952267808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 0x7f7db8099e40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7db8071f40 tx=0x7f7da4009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.246+0000 7f7dbf200700 1 -- 172.21.15.12:0/952267808 shutdown_connections 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.246+0000 7f7dbf200700 1 --2- 172.21.15.12:0/952267808 >> v2:172.21.15.159:3300/0 conn(0x7f7db8099a60 0x7f7db80d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.246+0000 7f7dbf200700 1 --2- 172.21.15.12:0/952267808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 0x7f7db8099e40 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.246+0000 7f7dbf200700 1 -- 172.21.15.12:0/952267808 >> 172.21.15.12:0/952267808 conn(0x7f7db80b2e20 msgr2=0x7f7db80b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.246+0000 7f7dbf200700 1 -- 172.21.15.12:0/952267808 shutdown_connections 2024-03-29T09:55:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.246+0000 7f7dbf200700 1 -- 172.21.15.12:0/952267808 wait complete. 2024-03-29T09:55:57.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.247+0000 7f7dbf200700 1 Processor -- start 2024-03-29T09:55:57.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.248+0000 7f7dbf200700 1 -- start start 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.248+0000 7f7dbf200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7db8099a60 0x7f7db815a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbf200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7db809a750 0x7f7db80cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbf200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 0x7f7db80cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbf200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db80ca600 con 0x7f7db815d990 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbf200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db80ca8e0 con 0x7f7db8099a60 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbf200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db80cabc0 con 0x7f7db809a750 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbd200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 0x7f7db80cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbd200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 0x7f7db80cb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39470/0 (socket says 172.21.15.12:39470) 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbd200700 1 -- 172.21.15.12:0/1316368981 learned_addr learned my addr 172.21.15.12:0/1316368981 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7db7e00700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7db8099a60 0x7f7db815a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7db7400700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7db809a750 0x7f7db80cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbd200700 1 -- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7db809a750 msgr2=0x7f7db80cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbd200700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7db809a750 0x7f7db80cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbd200700 1 -- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7db8099a60 msgr2=0x7f7db815a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbd200700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7db8099a60 0x7f7db815a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7dbd200700 1 -- 172.21.15.12:0/1316368981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da4009550 con 0x7f7db815d990 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7db7e00700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7db8099a60 0x7f7db815a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.249+0000 7f7db7400700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7db809a750 0x7f7db80cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.250+0000 7f7dbd200700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 0x7f7db80cb9f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7db000cb50 tx=0x7f7db000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.250+0000 7f7dafe00700 1 -- 172.21.15.12:0/1316368981 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db0003c00 con 0x7f7db815d990 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.250+0000 7f7dafe00700 1 -- 172.21.15.12:0/1316368981 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db0003d60 con 0x7f7db815d990 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.250+0000 7f7dafe00700 1 -- 172.21.15.12:0/1316368981 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db000ab70 con 0x7f7db815d990 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.250+0000 7f7dbf200700 1 -- 172.21.15.12:0/1316368981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db80ccec0 con 0x7f7db815d990 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.250+0000 7f7dbf200700 1 -- 172.21.15.12:0/1316368981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db814d970 con 0x7f7db815d990 2024-03-29T09:55:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.251+0000 7f7dbf200700 1 -- 172.21.15.12:0/1316368981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db80cade0 con 0x7f7db815d990 2024-03-29T09:55:57.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.252+0000 7f7dafe00700 1 -- 172.21.15.12:0/1316368981 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f7db000acd0 con 0x7f7db815d990 2024-03-29T09:55:57.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.253+0000 7f7dafe00700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7d9806ae90 0x7f7d9806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.253+0000 7f7dafe00700 1 -- 172.21.15.12:0/1316368981 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f7db00898a0 con 0x7f7db815d990 2024-03-29T09:55:57.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.253+0000 7f7db7e00700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7d9806ae90 0x7f7d9806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.254+0000 7f7db7e00700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7d9806ae90 0x7f7d9806d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7db80b25c0 tx=0x7f7da8008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:57.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.257+0000 7f7dafe00700 1 -- 172.21.15.12:0/1316368981 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7db0054660 con 0x7f7db815d990 2024-03-29T09:55:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.497+0000 7f7dbf200700 1 -- 172.21.15.12:0/1316368981 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7db80caa50 con 0x7f7d9806ae90 2024-03-29T09:55:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.582+0000 7f7dafe00700 1 -- 172.21.15.12:0/1316368981 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7db80caa50 con 0x7f7d9806ae90 2024-03-29T09:55:57.582 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/dir6 2024-03-29T09:55:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 -- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7d9806ae90 msgr2=0x7f7d9806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7d9806ae90 0x7f7d9806d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7db80b25c0 tx=0x7f7da8008040 comp rx=0 tx=0).stop 2024-03-29T09:55:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 -- 172.21.15.12:0/1316368981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 msgr2=0x7f7db80cb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 0x7f7db80cb9f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7db000cb50 tx=0x7f7db000cf10 comp rx=0 tx=0).stop 2024-03-29T09:55:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 -- 172.21.15.12:0/1316368981 shutdown_connections 2024-03-29T09:55:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7d9806ae90 0x7f7d9806d350 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7db815d990 0x7f7db80cb9f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7db8099a60 0x7f7db815a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 --2- 172.21.15.12:0/1316368981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7db809a750 0x7f7db80cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 -- 172.21.15.12:0/1316368981 >> 172.21.15.12:0/1316368981 conn(0x7f7db80b2e20 msgr2=0x7f7db80b6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:57.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 -- 172.21.15.12:0/1316368981 shutdown_connections 2024-03-29T09:55:57.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.585+0000 7f7dbf200700 1 -- 172.21.15.12:0/1316368981 wait complete. 2024-03-29T09:55:57.602 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2024-03-29T09:55:57.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:57.603 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2024-03-29T09:55:57.649 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2024-03-29T09:55:57.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.852+0000 7f8a49c00700 1 Processor -- start 2024-03-29T09:55:57.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.852+0000 7f8a49c00700 1 -- start start 2024-03-29T09:55:57.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.853+0000 7f8a49c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8a440cee10 0x7f8a44161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.853+0000 7f8a49c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a440ce270 con 0x7f8a440d21f0 2024-03-29T09:55:57.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.853+0000 7f8a49c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4415b600 con 0x7f8a441611b0 2024-03-29T09:55:57.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.853+0000 7f8a49c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a440d2760 con 0x7f8a440cee10 2024-03-29T09:55:57.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.853+0000 7f8a42000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8a440d21f0 0x7f8a440ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60802/0 (socket says 172.21.15.12:60802) 2024-03-29T09:55:57.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.853+0000 7f8a42000700 1 -- 172.21.15.12:0/979479423 learned_addr learned my addr 172.21.15.12:0/979479423 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:57.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a43400700 1 --2- 172.21.15.12:0/979479423 >> v2:172.21.15.159:3300/0 conn(0x7f8a440cee10 0x7f8a44161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326225844 0 0) 0x7f8a440ce270 con 0x7f8a440d21f0 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a280035b0 con 0x7f8a440d21f0 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1234369605 0 0) 0x7f8a280035b0 con 0x7f8a440d21f0 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 >> v2:172.21.15.159:3300/0 conn(0x7f8a440cee10 msgr2=0x7f8a44161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 --2- 172.21.15.12:0/979479423 >> v2:172.21.15.159:3300/0 conn(0x7f8a440cee10 0x7f8a44161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 >> v1:172.21.15.159:6789/0 conn(0x7f8a441611b0 legacy=0x7f8a4415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a440d2a40 con 0x7f8a440d21f0 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f8a30001670 con 0x7f8a440d21f0 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 >> v1:172.21.15.12:6789/0 conn(0x7f8a440d21f0 legacy=0x7f8a440ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a28003a00 0x7f8a28003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a28004400 0x7f8a280048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a280050d0 0x7f8a2800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.854+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a440d2760 con 0x7f8a280050d0 2024-03-29T09:55:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2800c030 con 0x7f8a28004400 2024-03-29T09:55:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2800c1f0 con 0x7f8a28003a00 2024-03-29T09:55:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a43400700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a280050d0 0x7f8a2800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a42000700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a28004400 0x7f8a280048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a42a00700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a28003a00 0x7f8a28003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a43400700 1 -- 172.21.15.12:0/979479423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a28003a00 msgr2=0x7f8a28003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a43400700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a28003a00 0x7f8a28003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a43400700 1 -- 172.21.15.12:0/979479423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a28004400 msgr2=0x7f8a280048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a43400700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a28004400 0x7f8a280048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a43400700 1 -- 172.21.15.12:0/979479423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a280035b0 con 0x7f8a280050d0 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a42a00700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a28003a00 0x7f8a28003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a43400700 1 -- 172.21.15.12:0/979479423 reap_dead start 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.855+0000 7f8a43400700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a280050d0 0x7f8a2800ba10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009fc0 tx=0x7f8a28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab6de9e8cc93b6f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.856+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c003a20 con 0x7f8a280050d0 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.856+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c00f040 con 0x7f8a280050d0 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.856+0000 7f8a41600700 1 -- 172.21.15.12:0/979479423 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c003de0 con 0x7f8a280050d0 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.856+0000 7f8a49c00700 1 -- 172.21.15.12:0/979479423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a280050d0 msgr2=0x7f8a2800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.856+0000 7f8a49c00700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a280050d0 0x7f8a2800ba10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009fc0 tx=0x7f8a28005880 comp rx=0 tx=0).stop 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.856+0000 7f8a49c00700 1 -- 172.21.15.12:0/979479423 shutdown_connections 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.856+0000 7f8a49c00700 1 --2- 172.21.15.12:0/979479423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a280050d0 0x7f8a2800ba10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.856+0000 7f8a49c00700 1 -- 172.21.15.12:0/979479423 >> 172.21.15.12:0/979479423 conn(0x7f8a4408aff0 msgr2=0x7f8a440d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.856+0000 7f8a49c00700 1 -- 172.21.15.12:0/979479423 shutdown_connections 2024-03-29T09:55:57.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.856+0000 7f8a49c00700 1 -- 172.21.15.12:0/979479423 wait complete. 2024-03-29T09:55:57.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.858+0000 7f8a49c00700 1 Processor -- start 2024-03-29T09:55:57.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.858+0000 7f8a49c00700 1 -- start start 2024-03-29T09:55:57.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.859+0000 7f8a49c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a280050d0 0x7f8a4414bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.859+0000 7f8a49c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a440c9b10 0x7f8a440c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.859+0000 7f8a49c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440c8b90 0x7f8a440c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.859+0000 7f8a49c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a440b6150 con 0x7f8a440c8b90 2024-03-29T09:55:57.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.859+0000 7f8a49c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a44157b30 con 0x7f8a280050d0 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.859+0000 7f8a49c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a440cd260 con 0x7f8a440c9b10 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.859+0000 7f8a43400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440c8b90 0x7f8a440c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a43400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440c8b90 0x7f8a440c9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39492/0 (socket says 172.21.15.12:39492) 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a43400700 1 -- 172.21.15.12:0/3672375254 learned_addr learned my addr 172.21.15.12:0/3672375254 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a42a00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a280050d0 0x7f8a4414bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a42000700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a440c9b10 0x7f8a440c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a43400700 1 -- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a440c9b10 msgr2=0x7f8a440c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a43400700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a440c9b10 0x7f8a440c8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a43400700 1 -- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a280050d0 msgr2=0x7f8a4414bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a43400700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a280050d0 0x7f8a4414bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a43400700 1 -- 172.21.15.12:0/3672375254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a441574e0 con 0x7f8a440c8b90 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a42a00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a280050d0 0x7f8a4414bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a43400700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440c8b90 0x7f8a440c9010 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8a441610f0 tx=0x7f8a3c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:57.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.860+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3672375254 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c003e90 con 0x7f8a440c8b90 2024-03-29T09:55:57.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.861+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3672375254 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c004230 con 0x7f8a440c8b90 2024-03-29T09:55:57.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.861+0000 7f8a49c00700 1 -- 172.21.15.12:0/3672375254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a441577c0 con 0x7f8a440c8b90 2024-03-29T09:55:57.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.861+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3672375254 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c0045e0 con 0x7f8a440c8b90 2024-03-29T09:55:57.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.861+0000 7f8a49c00700 1 -- 172.21.15.12:0/3672375254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a440cc0a0 con 0x7f8a440c8b90 2024-03-29T09:55:57.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.862+0000 7f8a49c00700 1 -- 172.21.15.12:0/3672375254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a44157340 con 0x7f8a440c8b90 2024-03-29T09:55:57.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.863+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3672375254 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f8a3c004740 con 0x7f8a440c8b90 2024-03-29T09:55:57.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.863+0000 7f8a3aa00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406ae90 0x7f8a2406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:57.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.863+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3672375254 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f8a3c08ab60 con 0x7f8a440c8b90 2024-03-29T09:55:57.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.864+0000 7f8a42a00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406ae90 0x7f8a2406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:57.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.865+0000 7f8a42a00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406ae90 0x7f8a2406d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8a340011a0 tx=0x7f8a34009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:57.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:57.868+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3672375254 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a44157340 con 0x7f8a440c8b90 2024-03-29T09:55:58.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.110+0000 7f8a49c00700 1 -- 172.21.15.12:0/3672375254 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a440c9ef0 con 0x7f8a2406ae90 2024-03-29T09:55:58.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.196+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3672375254 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8a440c9ef0 con 0x7f8a2406ae90 2024-03-29T09:55:58.196 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/dir7 2024-03-29T09:55:58.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 -- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406ae90 msgr2=0x7f8a2406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:58.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406ae90 0x7f8a2406d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8a340011a0 tx=0x7f8a34009040 comp rx=0 tx=0).stop 2024-03-29T09:55:58.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 -- 172.21.15.12:0/3672375254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440c8b90 msgr2=0x7f8a440c9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440c8b90 0x7f8a440c9010 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8a441610f0 tx=0x7f8a3c009750 comp rx=0 tx=0).stop 2024-03-29T09:55:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 -- 172.21.15.12:0/3672375254 shutdown_connections 2024-03-29T09:55:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406ae90 0x7f8a2406d350 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440c8b90 0x7f8a440c9010 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a280050d0 0x7f8a4414bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 --2- 172.21.15.12:0/3672375254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a440c9b10 0x7f8a440c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 -- 172.21.15.12:0/3672375254 >> 172.21.15.12:0/3672375254 conn(0x7f8a4408aff0 msgr2=0x7f8a440ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 -- 172.21.15.12:0/3672375254 shutdown_connections 2024-03-29T09:55:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.199+0000 7f8a49c00700 1 -- 172.21.15.12:0/3672375254 wait complete. 2024-03-29T09:55:58.216 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2024-03-29T09:55:58.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:58.217 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2024-03-29T09:55:58.263 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2024-03-29T09:55:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.505+0000 7f685d000700 1 Processor -- start 2024-03-29T09:55:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.505+0000 7f685d000700 1 -- start start 2024-03-29T09:55:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.506+0000 7f685d000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f685815d9d0 0x7f68580d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.506+0000 7f685d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68580d16a0 0x7f68580d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f685d000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6858161420 con 0x7f68580d2370 2024-03-29T09:55:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f685d000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68580d27e0 con 0x7f68580d16a0 2024-03-29T09:55:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f685d000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6858161700 con 0x7f685815d9d0 2024-03-29T09:55:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f6855400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68580d16a0 0x7f68580d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f6855400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68580d16a0 0x7f68580d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39504/0 (socket says 172.21.15.12:39504) 2024-03-29T09:55:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f6855400700 1 -- 172.21.15.12:0/2680335478 learned_addr learned my addr 172.21.15.12:0/2680335478 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f6855e00700 1 --2- 172.21.15.12:0/2680335478 >> v2:172.21.15.159:3300/0 conn(0x7f685815d9d0 0x7f68580d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f6855400700 1 -- 172.21.15.12:0/2680335478 >> v2:172.21.15.159:3300/0 conn(0x7f685815d9d0 msgr2=0x7f68580d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:58.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f6855400700 1 --2- 172.21.15.12:0/2680335478 >> v2:172.21.15.159:3300/0 conn(0x7f685815d9d0 0x7f68580d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f6855400700 1 -- 172.21.15.12:0/2680335478 >> v1:172.21.15.159:6789/0 conn(0x7f68580d2370 legacy=0x7f685815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:58.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f6855400700 1 -- 172.21.15.12:0/2680335478 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68581619e0 con 0x7f68580d16a0 2024-03-29T09:55:58.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.507+0000 7f6855e00700 1 --2- 172.21.15.12:0/2680335478 >> v2:172.21.15.159:3300/0 conn(0x7f685815d9d0 0x7f68580d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:58.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.508+0000 7f6855400700 1 --2- 172.21.15.12:0/2680335478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68580d16a0 0x7f68580d1e30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f68580d0ff0 tx=0x7f6844009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496ede95f5f46b48 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:58.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.508+0000 7f6854a00700 1 -- 172.21.15.12:0/2680335478 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6844005e30 con 0x7f68580d16a0 2024-03-29T09:55:58.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.508+0000 7f6854a00700 1 -- 172.21.15.12:0/2680335478 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6844004030 con 0x7f68580d16a0 2024-03-29T09:55:58.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.508+0000 7f6854a00700 1 -- 172.21.15.12:0/2680335478 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68440043e0 con 0x7f68580d16a0 2024-03-29T09:55:58.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.508+0000 7f685d000700 1 -- 172.21.15.12:0/2680335478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68580d16a0 msgr2=0x7f68580d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:58.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.508+0000 7f685d000700 1 --2- 172.21.15.12:0/2680335478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68580d16a0 0x7f68580d1e30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f68580d0ff0 tx=0x7f6844009bd0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.508+0000 7f685d000700 1 -- 172.21.15.12:0/2680335478 shutdown_connections 2024-03-29T09:55:58.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.508+0000 7f685d000700 1 --2- 172.21.15.12:0/2680335478 >> v2:172.21.15.159:3300/0 conn(0x7f685815d9d0 0x7f68580d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.508+0000 7f685d000700 1 --2- 172.21.15.12:0/2680335478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68580d16a0 0x7f68580d1e30 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.508+0000 7f685d000700 1 -- 172.21.15.12:0/2680335478 >> 172.21.15.12:0/2680335478 conn(0x7f685808aff0 msgr2=0x7f68580d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.509+0000 7f685d000700 1 -- 172.21.15.12:0/2680335478 shutdown_connections 2024-03-29T09:55:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.509+0000 7f685d000700 1 -- 172.21.15.12:0/2680335478 wait complete. 2024-03-29T09:55:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.510+0000 7f685d000700 1 Processor -- start 2024-03-29T09:55:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.511+0000 7f685d000700 1 -- start start 2024-03-29T09:55:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.511+0000 7f685d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68580d16a0 0x7f6858155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.511+0000 7f685d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68580d2370 0x7f68580cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f685d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f685815d9d0 0x7f68580b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f685d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580cc510 con 0x7f685815d9d0 2024-03-29T09:55:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f685d000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580cb900 con 0x7f68580d2370 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f685d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68580cbbe0 con 0x7f68580d16a0 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6856800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f685815d9d0 0x7f68580b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6856800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f685815d9d0 0x7f68580b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39508/0 (socket says 172.21.15.12:39508) 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6856800700 1 -- 172.21.15.12:0/3487972365 learned_addr learned my addr 172.21.15.12:0/3487972365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6855400700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68580d2370 0x7f68580cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6855e00700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68580d16a0 0x7f6858155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6856800700 1 -- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68580d16a0 msgr2=0x7f6858155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6856800700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68580d16a0 0x7f6858155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6856800700 1 -- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68580d2370 msgr2=0x7f68580cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6856800700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68580d2370 0x7f68580cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6856800700 1 -- 172.21.15.12:0/3487972365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6844009500 con 0x7f685815d9d0 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.512+0000 7f6856800700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f685815d9d0 0x7f68580b4d00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f685000cc50 tx=0x7f685000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.513+0000 7f684e000700 1 -- 172.21.15.12:0/3487972365 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685000ad80 con 0x7f685815d9d0 2024-03-29T09:55:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.513+0000 7f684e000700 1 -- 172.21.15.12:0/3487972365 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685000ce50 con 0x7f685815d9d0 2024-03-29T09:55:58.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.513+0000 7f684e000700 1 -- 172.21.15.12:0/3487972365 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850003c40 con 0x7f685815d9d0 2024-03-29T09:55:58.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.513+0000 7f685d000700 1 -- 172.21.15.12:0/3487972365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68580cbec0 con 0x7f685815d9d0 2024-03-29T09:55:58.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.513+0000 7f685d000700 1 -- 172.21.15.12:0/3487972365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68580cb540 con 0x7f685815d9d0 2024-03-29T09:55:58.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.514+0000 7f685d000700 1 -- 172.21.15.12:0/3487972365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68580d1ca0 con 0x7f685815d9d0 2024-03-29T09:55:58.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.515+0000 7f684e000700 1 -- 172.21.15.12:0/3487972365 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f68500050d0 con 0x7f685815d9d0 2024-03-29T09:55:58.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.516+0000 7f684e000700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f683806ae90 0x7f683806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.516+0000 7f684e000700 1 -- 172.21.15.12:0/3487972365 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f68500898e0 con 0x7f685815d9d0 2024-03-29T09:55:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.517+0000 7f6855e00700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f683806ae90 0x7f683806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:58.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.517+0000 7f6855e00700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f683806ae90 0x7f683806d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f68580c9ca0 tx=0x7f6848008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:58.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.521+0000 7f684e000700 1 -- 172.21.15.12:0/3487972365 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6850054730 con 0x7f685815d9d0 2024-03-29T09:55:58.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.763+0000 7f685d000700 1 -- 172.21.15.12:0/3487972365 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6858161fa0 con 0x7f683806ae90 2024-03-29T09:55:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.845+0000 7f684e000700 1 -- 172.21.15.12:0/3487972365 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f684800c590 con 0x7f683806ae90 2024-03-29T09:55:58.845 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/dir8 2024-03-29T09:55:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.847+0000 7f685d000700 1 -- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f683806ae90 msgr2=0x7f683806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.847+0000 7f685d000700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f683806ae90 0x7f683806d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f68580c9ca0 tx=0x7f6848008040 comp rx=0 tx=0).stop 2024-03-29T09:55:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.847+0000 7f685d000700 1 -- 172.21.15.12:0/3487972365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f685815d9d0 msgr2=0x7f68580b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.847+0000 7f685d000700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f685815d9d0 0x7f68580b4d00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f685000cc50 tx=0x7f685000a340 comp rx=0 tx=0).stop 2024-03-29T09:55:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.848+0000 7f685d000700 1 -- 172.21.15.12:0/3487972365 shutdown_connections 2024-03-29T09:55:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.848+0000 7f685d000700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f683806ae90 0x7f683806d350 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.848+0000 7f685d000700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f685815d9d0 0x7f68580b4d00 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.848+0000 7f685d000700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68580d16a0 0x7f6858155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.848+0000 7f685d000700 1 --2- 172.21.15.12:0/3487972365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68580d2370 0x7f68580cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.848+0000 7f685d000700 1 -- 172.21.15.12:0/3487972365 >> 172.21.15.12:0/3487972365 conn(0x7f685808aff0 msgr2=0x7f68580b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.848+0000 7f685d000700 1 -- 172.21.15.12:0/3487972365 shutdown_connections 2024-03-29T09:55:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:58.848+0000 7f685d000700 1 -- 172.21.15.12:0/3487972365 wait complete. 2024-03-29T09:55:58.865 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2024-03-29T09:55:58.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T09:55:58.865 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2024-03-29T09:55:58.911 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2024-03-29T09:55:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.114+0000 7f17a1400700 1 Processor -- start 2024-03-29T09:55:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.115+0000 7f17a1400700 1 -- start start 2024-03-29T09:55:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.116+0000 7f17a1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c1513c0 0x7f179c06ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.116+0000 7f17a1400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c15b8c0 con 0x7f179c06b0c0 2024-03-29T09:55:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.116+0000 7f17a1400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c15ee50 con 0x7f179c15b230 2024-03-29T09:55:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.116+0000 7f17a1400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179c15b6c0 con 0x7f179c1513c0 2024-03-29T09:55:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.116+0000 7f1799e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f179c06b0c0 0x7f179c15ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33600/0 (socket says 172.21.15.12:33600) 2024-03-29T09:55:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.116+0000 7f1799e00700 1 -- 172.21.15.12:0/1747043863 learned_addr learned my addr 172.21.15.12:0/1747043863 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:55:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.116+0000 7f179a800700 1 --2- 172.21.15.12:0/1747043863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c1513c0 0x7f179c06ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.116+0000 7f1799400700 1 -- 172.21.15.12:0/1747043863 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030258655 0 0) 0x7f179c15b8c0 con 0x7f179c06b0c0 2024-03-29T09:55:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.117+0000 7f1799400700 1 -- 172.21.15.12:0/1747043863 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17800057c0 con 0x7f179c06b0c0 2024-03-29T09:55:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.117+0000 7f1799400700 1 -- 172.21.15.12:0/1747043863 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626583433 0 0) 0x7f179c15ee50 con 0x7f179c15b230 2024-03-29T09:55:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.117+0000 7f1799400700 1 -- 172.21.15.12:0/1747043863 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c15b8c0 con 0x7f179c15b230 2024-03-29T09:55:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.117+0000 7f179a800700 1 -- 172.21.15.12:0/1747043863 >> v1:172.21.15.159:6789/0 conn(0x7f179c15b230 legacy=0x7f179c0711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.117+0000 7f179a800700 1 -- 172.21.15.12:0/1747043863 >> v1:172.21.15.12:6789/0 conn(0x7f179c06b0c0 legacy=0x7f179c15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.117+0000 7f179a800700 1 -- 172.21.15.12:0/1747043863 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c15e090 con 0x7f179c1513c0 2024-03-29T09:55:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.117+0000 7f179a800700 1 --2- 172.21.15.12:0/1747043863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c1513c0 0x7f179c06ab80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f178c009870 tx=0x7f178c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c76249f8ee6903a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.117+0000 7f1799400700 1 -- 172.21.15.12:0/1747043863 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c011830 con 0x7f179c1513c0 2024-03-29T09:55:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.117+0000 7f1799400700 1 -- 172.21.15.12:0/1747043863 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c011990 con 0x7f179c1513c0 2024-03-29T09:55:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.117+0000 7f1799400700 1 -- 172.21.15.12:0/1747043863 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c011d00 con 0x7f179c1513c0 2024-03-29T09:55:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.118+0000 7f17a1400700 1 -- 172.21.15.12:0/1747043863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c1513c0 msgr2=0x7f179c06ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.118+0000 7f17a1400700 1 --2- 172.21.15.12:0/1747043863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c1513c0 0x7f179c06ab80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f178c009870 tx=0x7f178c009b80 comp rx=0 tx=0).stop 2024-03-29T09:55:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.118+0000 7f17a1400700 1 -- 172.21.15.12:0/1747043863 shutdown_connections 2024-03-29T09:55:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.118+0000 7f17a1400700 1 --2- 172.21.15.12:0/1747043863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c1513c0 0x7f179c06ab80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.118+0000 7f17a1400700 1 -- 172.21.15.12:0/1747043863 >> 172.21.15.12:0/1747043863 conn(0x7f179c0b1230 msgr2=0x7f179c0b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.118+0000 7f17a1400700 1 -- 172.21.15.12:0/1747043863 shutdown_connections 2024-03-29T09:55:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.118+0000 7f17a1400700 1 -- 172.21.15.12:0/1747043863 wait complete. 2024-03-29T09:55:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.120+0000 7f17a1400700 1 Processor -- start 2024-03-29T09:55:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.120+0000 7f17a1400700 1 -- start start 2024-03-29T09:55:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f17a1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c06b0c0 0x7f179c0af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f17a1400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f179c1513c0 0x7f179c1496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f17a1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179c15b230 0x7f179c151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f17a1400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c152d20 con 0x7f179c15b230 2024-03-29T09:55:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f17a1400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c152110 con 0x7f179c1513c0 2024-03-29T09:55:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f17a1400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179c1523f0 con 0x7f179c06b0c0 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f179b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179c15b230 0x7f179c151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f179b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179c15b230 0x7f179c151a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57482/0 (socket says 172.21.15.12:57482) 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f179b200700 1 -- 172.21.15.12:0/1997358497 learned_addr learned my addr 172.21.15.12:0/1997358497 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f179a800700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c06b0c0 0x7f179c0af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.121+0000 7f1799e00700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f179c1513c0 0x7f179c1496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f179b200700 1 -- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c06b0c0 msgr2=0x7f179c0af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f179b200700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c06b0c0 0x7f179c0af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f179b200700 1 -- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f179c1513c0 msgr2=0x7f179c1496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f179b200700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f179c1513c0 0x7f179c1496e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f179b200700 1 -- 172.21.15.12:0/1997358497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0094a0 con 0x7f179c15b230 2024-03-29T09:55:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f179a800700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c06b0c0 0x7f179c0af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:55:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f179b200700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179c15b230 0x7f179c151a10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f179400cbc0 tx=0x7f179400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f1792a00700 1 -- 172.21.15.12:0/1997358497 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1794003c00 con 0x7f179c15b230 2024-03-29T09:55:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f1792a00700 1 -- 172.21.15.12:0/1997358497 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1794003d60 con 0x7f179c15b230 2024-03-29T09:55:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f17a1400700 1 -- 172.21.15.12:0/1997358497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f179c1526d0 con 0x7f179c15b230 2024-03-29T09:55:59.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f1792a00700 1 -- 172.21.15.12:0/1997358497 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179400abb0 con 0x7f179c15b230 2024-03-29T09:55:59.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.122+0000 7f17a1400700 1 -- 172.21.15.12:0/1997358497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179c0ae060 con 0x7f179c15b230 2024-03-29T09:55:59.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.123+0000 7f17a1400700 1 -- 172.21.15.12:0/1997358497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f179c152ed0 con 0x7f179c15b230 2024-03-29T09:55:59.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.124+0000 7f1792a00700 1 -- 172.21.15.12:0/1997358497 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f179400ad10 con 0x7f179c15b230 2024-03-29T09:55:59.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.125+0000 7f1792a00700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177c06ae90 0x7f177c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.125+0000 7f1792a00700 1 -- 172.21.15.12:0/1997358497 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1794089940 con 0x7f179c15b230 2024-03-29T09:55:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.126+0000 7f179a800700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177c06ae90 0x7f177c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.126+0000 7f179a800700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177c06ae90 0x7f177c06d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f178c0090b0 tx=0x7f178c008030 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:59.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.130+0000 7f1792a00700 1 -- 172.21.15.12:0/1997358497 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f179c152ed0 con 0x7f179c15b230 2024-03-29T09:55:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.368+0000 7f17a1400700 1 -- 172.21.15.12:0/1997358497 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f179c152560 con 0x7f177c06ae90 2024-03-29T09:55:59.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.453+0000 7f1792a00700 1 -- 172.21.15.12:0/1997358497 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f179c152560 con 0x7f177c06ae90 2024-03-29T09:55:59.452 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/dir9 2024-03-29T09:55:59.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 -- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177c06ae90 msgr2=0x7f177c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177c06ae90 0x7f177c06d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f178c0090b0 tx=0x7f178c008030 comp rx=0 tx=0).stop 2024-03-29T09:55:59.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 -- 172.21.15.12:0/1997358497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179c15b230 msgr2=0x7f179c151a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179c15b230 0x7f179c151a10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f179400cbc0 tx=0x7f179400cf80 comp rx=0 tx=0).stop 2024-03-29T09:55:59.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 -- 172.21.15.12:0/1997358497 shutdown_connections 2024-03-29T09:55:59.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f177c06ae90 0x7f177c06d350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f179c15b230 0x7f179c151a10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f179c06b0c0 0x7f179c0af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 --2- 172.21.15.12:0/1997358497 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f179c1513c0 0x7f179c1496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 -- 172.21.15.12:0/1997358497 >> 172.21.15.12:0/1997358497 conn(0x7f179c0b1230 msgr2=0x7f179c15a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:59.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 -- 172.21.15.12:0/1997358497 shutdown_connections 2024-03-29T09:55:59.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.455+0000 7f17a1400700 1 -- 172.21.15.12:0/1997358497 wait complete. 2024-03-29T09:55:59.475 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2024-03-29T09:55:59.475 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2024-03-29T09:55:59.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.677+0000 7f478ca00700 1 Processor -- start 2024-03-29T09:55:59.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.677+0000 7f478ca00700 1 -- start start 2024-03-29T09:55:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.678+0000 7f478ca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f478815d9d0 0x7f47880d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.678+0000 7f478ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47880d2540 0x7f47880ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.678+0000 7f478ca00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4788161420 con 0x7f47880d16a0 2024-03-29T09:55:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.678+0000 7f478ca00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478815edc0 con 0x7f478815d9d0 2024-03-29T09:55:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.678+0000 7f478ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4788161700 con 0x7f47880d2540 2024-03-29T09:55:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.678+0000 7f4785e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f478815d9d0 0x7f47880d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.678+0000 7f4785e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f478815d9d0 0x7f47880d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57494/0 (socket says 172.21.15.12:57494) 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.678+0000 7f4785e00700 1 -- 172.21.15.12:0/2629209929 learned_addr learned my addr 172.21.15.12:0/2629209929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.678+0000 7f4786800700 1 --2- 172.21.15.12:0/2629209929 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47880d2540 0x7f47880ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.679+0000 7f4785e00700 1 -- 172.21.15.12:0/2629209929 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47880d2540 msgr2=0x7f47880ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.679+0000 7f4785e00700 1 --2- 172.21.15.12:0/2629209929 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47880d2540 0x7f47880ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.679+0000 7f4785e00700 1 -- 172.21.15.12:0/2629209929 >> v1:172.21.15.159:6789/0 conn(0x7f47880d16a0 legacy=0x7f47880d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.679+0000 7f4785e00700 1 -- 172.21.15.12:0/2629209929 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47881619e0 con 0x7f478815d9d0 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.679+0000 7f4785e00700 1 --2- 172.21.15.12:0/2629209929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f478815d9d0 0x7f47880d3c30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f47780098f0 tx=0x7f4778009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54ad848745309bb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.679+0000 7f4784a00700 1 -- 172.21.15.12:0/2629209929 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778011830 con 0x7f478815d9d0 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.679+0000 7f4784a00700 1 -- 172.21.15.12:0/2629209929 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4778011990 con 0x7f478815d9d0 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.679+0000 7f4784a00700 1 -- 172.21.15.12:0/2629209929 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778011d20 con 0x7f478815d9d0 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.680+0000 7f478ca00700 1 -- 172.21.15.12:0/2629209929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f478815d9d0 msgr2=0x7f47880d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.680+0000 7f478ca00700 1 --2- 172.21.15.12:0/2629209929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f478815d9d0 0x7f47880d3c30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f47780098f0 tx=0x7f4778009c00 comp rx=0 tx=0).stop 2024-03-29T09:55:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.680+0000 7f478ca00700 1 -- 172.21.15.12:0/2629209929 shutdown_connections 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.680+0000 7f478ca00700 1 --2- 172.21.15.12:0/2629209929 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47880d2540 0x7f47880ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.680+0000 7f478ca00700 1 --2- 172.21.15.12:0/2629209929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f478815d9d0 0x7f47880d3c30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.680+0000 7f478ca00700 1 -- 172.21.15.12:0/2629209929 >> 172.21.15.12:0/2629209929 conn(0x7f478808aff0 msgr2=0x7f47880d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.680+0000 7f478ca00700 1 -- 172.21.15.12:0/2629209929 shutdown_connections 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.680+0000 7f478ca00700 1 -- 172.21.15.12:0/2629209929 wait complete. 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.682+0000 7f478ca00700 1 Processor -- start 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.682+0000 7f478ca00700 1 -- start start 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f478ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47880d16a0 0x7f47881e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f478ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47880d2540 0x7f478814f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f478ca00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f478815d9d0 0x7f47880b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f478ca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880cb5a0 con 0x7f47880d16a0 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f478ca00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47881e82e0 con 0x7f478815d9d0 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f478ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47881e85c0 con 0x7f47880d2540 2024-03-29T09:55:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f4785e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47880d16a0 0x7f47881e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f4785e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47880d16a0 0x7f47881e7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57498/0 (socket says 172.21.15.12:57498) 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f4785e00700 1 -- 172.21.15.12:0/439288070 learned_addr learned my addr 172.21.15.12:0/439288070 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f4785400700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47880d2540 0x7f478814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f4786800700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f478815d9d0 0x7f47880b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f4785e00700 1 -- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47880d2540 msgr2=0x7f478814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f4785e00700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47880d2540 0x7f478814f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f4785e00700 1 -- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f478815d9d0 msgr2=0x7f47880b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f4785e00700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f478815d9d0 0x7f47880b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.683+0000 7f4785e00700 1 -- 172.21.15.12:0/439288070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4778009550 con 0x7f47880d16a0 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.684+0000 7f4785e00700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47880d16a0 0x7f47881e7c90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4778009c40 tx=0x7f4778004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.684+0000 7f477e000700 1 -- 172.21.15.12:0/439288070 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778004510 con 0x7f47880d16a0 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.684+0000 7f477e000700 1 -- 172.21.15.12:0/439288070 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477801e070 con 0x7f47880d16a0 2024-03-29T09:55:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.684+0000 7f477e000700 1 -- 172.21.15.12:0/439288070 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47780038e0 con 0x7f47880d16a0 2024-03-29T09:55:59.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.684+0000 7f478ca00700 1 -- 172.21.15.12:0/439288070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47881e88a0 con 0x7f47880d16a0 2024-03-29T09:55:59.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.684+0000 7f478ca00700 1 -- 172.21.15.12:0/439288070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478814fd90 con 0x7f47880d16a0 2024-03-29T09:55:59.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.685+0000 7f478ca00700 1 -- 172.21.15.12:0/439288070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47880d1ca0 con 0x7f47880d16a0 2024-03-29T09:55:59.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.686+0000 7f477e000700 1 -- 172.21.15.12:0/439288070 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f4778003a40 con 0x7f47880d16a0 2024-03-29T09:55:59.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.687+0000 7f477e000700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f476806ae90 0x7f476806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:55:59.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.687+0000 7f477e000700 1 -- 172.21.15.12:0/439288070 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f477808a6d0 con 0x7f47880d16a0 2024-03-29T09:55:59.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.687+0000 7f4785400700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f476806ae90 0x7f476806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:55:59.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.688+0000 7f4785400700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f476806ae90 0x7f476806d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f478815c930 tx=0x7f4774009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:55:59.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.691+0000 7f477e000700 1 -- 172.21.15.12:0/439288070 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f47780554a0 con 0x7f47880d16a0 2024-03-29T09:55:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:55:59.935+0000 7f478ca00700 1 -- 172.21.15.12:0/439288070 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4788161fa0 con 0x7f476806ae90 2024-03-29T09:56:00.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.020+0000 7f477e000700 1 -- 172.21.15.12:0/439288070 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f4774001d70 con 0x7f476806ae90 2024-03-29T09:56:00.020 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/dir0 2024-03-29T09:56:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 -- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f476806ae90 msgr2=0x7f476806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f476806ae90 0x7f476806d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f478815c930 tx=0x7f4774009040 comp rx=0 tx=0).stop 2024-03-29T09:56:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 -- 172.21.15.12:0/439288070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47880d16a0 msgr2=0x7f47881e7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47880d16a0 0x7f47881e7c90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4778009c40 tx=0x7f4778004750 comp rx=0 tx=0).stop 2024-03-29T09:56:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 -- 172.21.15.12:0/439288070 shutdown_connections 2024-03-29T09:56:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f476806ae90 0x7f476806d350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f478815d9d0 0x7f47880b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47880d16a0 0x7f47881e7c90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 --2- 172.21.15.12:0/439288070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47880d2540 0x7f478814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 -- 172.21.15.12:0/439288070 >> 172.21.15.12:0/439288070 conn(0x7f478808aff0 msgr2=0x7f47880b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 -- 172.21.15.12:0/439288070 shutdown_connections 2024-03-29T09:56:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.023+0000 7f478ca00700 1 -- 172.21.15.12:0/439288070 wait complete. 2024-03-29T09:56:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:56:00.031+0000 7fe20b200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:56:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:56:00.031+0000 7fe20b200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:56:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:56:00.031+0000 7fe20b200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:56:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:56:00.031+0000 7fe20b200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:56:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T09:56:00.031+0000 7fe20b200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T09:56:00.043 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2024-03-29T09:56:00.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2024-03-29T09:56:00.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.263+0000 7fe57a400700 1 Processor -- start 2024-03-29T09:56:00.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.263+0000 7fe57a400700 1 -- start start 2024-03-29T09:56:00.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.264+0000 7fe57a400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe5740ce230 0x7fe5740ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.264+0000 7fe57a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5740b2070 0x7fe5740b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.264+0000 7fe57a400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5740709c0 con 0x7fe57415d800 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.264+0000 7fe57a400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5740b40f0 con 0x7fe5740ce230 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.264+0000 7fe57a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe574070c80 con 0x7fe5740b2070 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.264+0000 7fe573e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5740b2070 0x7fe5740b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.264+0000 7fe573400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fe57415d800 0x7fe5740725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:34408/0 (socket says 172.21.15.12:34408) 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.265+0000 7fe573400700 1 -- 172.21.15.12:0/3674367135 learned_addr learned my addr 172.21.15.12:0/3674367135 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.265+0000 7fe572a00700 1 --2- 172.21.15.12:0/3674367135 >> v2:172.21.15.159:3300/0 conn(0x7fe5740ce230 0x7fe5740ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.265+0000 7fe573e00700 1 -- 172.21.15.12:0/3674367135 >> v2:172.21.15.159:3300/0 conn(0x7fe5740ce230 msgr2=0x7fe5740ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.265+0000 7fe573e00700 1 --2- 172.21.15.12:0/3674367135 >> v2:172.21.15.159:3300/0 conn(0x7fe5740ce230 0x7fe5740ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.265+0000 7fe573e00700 1 -- 172.21.15.12:0/3674367135 >> v1:172.21.15.159:6789/0 conn(0x7fe57415d800 legacy=0x7fe5740725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.265+0000 7fe573e00700 1 -- 172.21.15.12:0/3674367135 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe574070f60 con 0x7fe5740b2070 2024-03-29T09:56:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.265+0000 7fe573e00700 1 --2- 172.21.15.12:0/3674367135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5740b2070 0x7fe5740b2500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00bfa0 tx=0x7fe56c00d330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20b019ce901cc605 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.265+0000 7fe572000700 1 -- 172.21.15.12:0/3674367135 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c010070 con 0x7fe5740b2070 2024-03-29T09:56:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.266+0000 7fe572000700 1 -- 172.21.15.12:0/3674367135 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c00da00 con 0x7fe5740b2070 2024-03-29T09:56:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.266+0000 7fe572000700 1 -- 172.21.15.12:0/3674367135 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c00ddb0 con 0x7fe5740b2070 2024-03-29T09:56:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.266+0000 7fe57a400700 1 -- 172.21.15.12:0/3674367135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5740b2070 msgr2=0x7fe5740b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.266+0000 7fe57a400700 1 --2- 172.21.15.12:0/3674367135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5740b2070 0x7fe5740b2500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00bfa0 tx=0x7fe56c00d330 comp rx=0 tx=0).stop 2024-03-29T09:56:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.266+0000 7fe57a400700 1 -- 172.21.15.12:0/3674367135 shutdown_connections 2024-03-29T09:56:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.266+0000 7fe57a400700 1 --2- 172.21.15.12:0/3674367135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5740b2070 0x7fe5740b2500 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.266+0000 7fe57a400700 1 --2- 172.21.15.12:0/3674367135 >> v2:172.21.15.159:3300/0 conn(0x7fe5740ce230 0x7fe5740ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.266+0000 7fe57a400700 1 -- 172.21.15.12:0/3674367135 >> 172.21.15.12:0/3674367135 conn(0x7fe5740b0d70 msgr2=0x7fe5740b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.266+0000 7fe57a400700 1 -- 172.21.15.12:0/3674367135 shutdown_connections 2024-03-29T09:56:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.266+0000 7fe57a400700 1 -- 172.21.15.12:0/3674367135 wait complete. 2024-03-29T09:56:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.268+0000 7fe57a400700 1 Processor -- start 2024-03-29T09:56:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.268+0000 7fe57a400700 1 -- start start 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.269+0000 7fe57a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5740b2070 0x7fe574156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.269+0000 7fe57a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5740ce230 0x7fe574098110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.269+0000 7fe57a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe57415d800 0x7fe574152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.269+0000 7fe57a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57415c080 con 0x7fe5740b2070 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.269+0000 7fe57a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5740ac050 con 0x7fe5740ce230 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.269+0000 7fe57a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57414f4e0 con 0x7fe57415d800 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe573400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5740b2070 0x7fe574156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe573400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5740b2070 0x7fe574156b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57512/0 (socket says 172.21.15.12:57512) 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe573400700 1 -- 172.21.15.12:0/1333475862 learned_addr learned my addr 172.21.15.12:0/1333475862 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe573e00700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe57415d800 0x7fe574152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe572a00700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5740ce230 0x7fe574098110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe573400700 1 -- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe57415d800 msgr2=0x7fe574152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe573400700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe57415d800 0x7fe574152c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe573400700 1 -- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5740ce230 msgr2=0x7fe574098110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe573400700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5740ce230 0x7fe574098110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe573400700 1 -- 172.21.15.12:0/1333475862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c00bc80 con 0x7fe5740b2070 2024-03-29T09:56:00.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.270+0000 7fe572a00700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5740ce230 0x7fe574098110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:56:00.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.271+0000 7fe573400700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5740b2070 0x7fe574156b10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe56400c480 tx=0x7fe56400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:00.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.271+0000 7fe56b400700 1 -- 172.21.15.12:0/1333475862 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe564005d70 con 0x7fe5740b2070 2024-03-29T09:56:00.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.271+0000 7fe56b400700 1 -- 172.21.15.12:0/1333475862 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5640070d0 con 0x7fe5740b2070 2024-03-29T09:56:00.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.271+0000 7fe56b400700 1 -- 172.21.15.12:0/1333475862 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe564007440 con 0x7fe5740b2070 2024-03-29T09:56:00.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.271+0000 7fe57a400700 1 -- 172.21.15.12:0/1333475862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57414f7c0 con 0x7fe5740b2070 2024-03-29T09:56:00.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.271+0000 7fe57a400700 1 -- 172.21.15.12:0/1333475862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57414fd80 con 0x7fe5740b2070 2024-03-29T09:56:00.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.273+0000 7fe57a400700 1 -- 172.21.15.12:0/1333475862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe574067720 con 0x7fe5740b2070 2024-03-29T09:56:00.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.273+0000 7fe56b400700 1 -- 172.21.15.12:0/1333475862 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fe5640075a0 con 0x7fe5740b2070 2024-03-29T09:56:00.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.274+0000 7fe56b400700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe55406ae90 0x7fe55406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.274+0000 7fe56b400700 1 -- 172.21.15.12:0/1333475862 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fe564088a10 con 0x7fe5740b2070 2024-03-29T09:56:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.275+0000 7fe572a00700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe55406ae90 0x7fe55406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.275+0000 7fe572a00700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe55406ae90 0x7fe55406d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe57414b380 tx=0x7fe560008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.279+0000 7fe56b400700 1 -- 172.21.15.12:0/1333475862 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5640539f0 con 0x7fe5740b2070 2024-03-29T09:56:00.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.564+0000 7fe57a400700 1 -- 172.21.15.12:0/1333475862 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5740cd9e0 con 0x7fe55406ae90 2024-03-29T09:56:00.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.630+0000 7fe56b400700 1 -- 172.21.15.12:0/1333475862 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe560009960 con 0x7fe55406ae90 2024-03-29T09:56:00.630 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/dir1 2024-03-29T09:56:00.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 -- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe55406ae90 msgr2=0x7fe55406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe55406ae90 0x7fe55406d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe57414b380 tx=0x7fe560008040 comp rx=0 tx=0).stop 2024-03-29T09:56:00.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 -- 172.21.15.12:0/1333475862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5740b2070 msgr2=0x7fe574156b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5740b2070 0x7fe574156b10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe56400c480 tx=0x7fe56400c790 comp rx=0 tx=0).stop 2024-03-29T09:56:00.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 -- 172.21.15.12:0/1333475862 shutdown_connections 2024-03-29T09:56:00.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe55406ae90 0x7fe55406d350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe57415d800 0x7fe574152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5740b2070 0x7fe574156b10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 --2- 172.21.15.12:0/1333475862 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5740ce230 0x7fe574098110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 -- 172.21.15.12:0/1333475862 >> 172.21.15.12:0/1333475862 conn(0x7fe5740b0d70 msgr2=0x7fe5740cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:00.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 -- 172.21.15.12:0/1333475862 shutdown_connections 2024-03-29T09:56:00.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.633+0000 7fe57a400700 1 -- 172.21.15.12:0/1333475862 wait complete. 2024-03-29T09:56:00.653 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2024-03-29T09:56:00.653 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2024-03-29T09:56:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.878+0000 7f9ca3600700 1 Processor -- start 2024-03-29T09:56:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.878+0000 7f9ca3600700 1 -- start start 2024-03-29T09:56:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.880+0000 7f9ca3600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9c9c0d16a0 0x7f9c9c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.880+0000 7f9ca3600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9c9c0d2370 0x7f9c9c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.880+0000 7f9ca3600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c161420 con 0x7f9c9c15d9d0 2024-03-29T09:56:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.880+0000 7f9ca3600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c0d4340 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.880+0000 7f9ca3600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c15edc0 con 0x7f9c9c0d2370 2024-03-29T09:56:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.880+0000 7f9c9be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9c9c0d16a0 0x7f9c9c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.880+0000 7f9c9be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9c9c0d16a0 0x7f9c9c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57522/0 (socket says 172.21.15.12:57522) 2024-03-29T09:56:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.880+0000 7f9ca0c00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9c9c15d9d0 0x7f9c9c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:34412/0 (socket says 172.21.15.12:34412) 2024-03-29T09:56:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.880+0000 7f9c9be00700 1 -- 172.21.15.12:0/2013789749 learned_addr learned my addr 172.21.15.12:0/2013789749 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.880+0000 7f9ca1600700 1 --2- 172.21.15.12:0/2013789749 >> v2:172.21.15.159:3300/0 conn(0x7f9c9c0d2370 0x7f9c9c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.881+0000 7f9c9be00700 1 -- 172.21.15.12:0/2013789749 >> v2:172.21.15.159:3300/0 conn(0x7f9c9c0d2370 msgr2=0x7f9c9c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:00.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.881+0000 7f9c9be00700 1 --2- 172.21.15.12:0/2013789749 >> v2:172.21.15.159:3300/0 conn(0x7f9c9c0d2370 0x7f9c9c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.881+0000 7f9c9be00700 1 -- 172.21.15.12:0/2013789749 >> v1:172.21.15.159:6789/0 conn(0x7f9c9c15d9d0 legacy=0x7f9c9c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.881+0000 7f9c9be00700 1 -- 172.21.15.12:0/2013789749 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c161700 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.881+0000 7f9c9b400700 1 -- 172.21.15.12:0/2013789749 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265368540 0 0) 0x7f9c9c161420 con 0x7f9c9c15d9d0 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.881+0000 7f9c9be00700 1 --2- 172.21.15.12:0/2013789749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c9c0d16a0 0x7f9c9c0d1e30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c08ca10 tx=0x7f9c88009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db993d3ee4c9433 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.881+0000 7f9c9b400700 1 -- 172.21.15.12:0/2013789749 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88005e30 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.881+0000 7f9c9b400700 1 -- 172.21.15.12:0/2013789749 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c88004030 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.881+0000 7f9c9b400700 1 -- 172.21.15.12:0/2013789749 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c880043e0 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.882+0000 7f9ca3600700 1 -- 172.21.15.12:0/2013789749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c9c0d16a0 msgr2=0x7f9c9c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.882+0000 7f9ca3600700 1 --2- 172.21.15.12:0/2013789749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c9c0d16a0 0x7f9c9c0d1e30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c08ca10 tx=0x7f9c88009bd0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.882+0000 7f9ca3600700 1 -- 172.21.15.12:0/2013789749 shutdown_connections 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.882+0000 7f9ca3600700 1 --2- 172.21.15.12:0/2013789749 >> v2:172.21.15.159:3300/0 conn(0x7f9c9c0d2370 0x7f9c9c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.882+0000 7f9ca3600700 1 --2- 172.21.15.12:0/2013789749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c9c0d16a0 0x7f9c9c0d1e30 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.882+0000 7f9ca3600700 1 -- 172.21.15.12:0/2013789749 >> 172.21.15.12:0/2013789749 conn(0x7f9c9c08aff0 msgr2=0x7f9c9c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.882+0000 7f9ca3600700 1 -- 172.21.15.12:0/2013789749 shutdown_connections 2024-03-29T09:56:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.882+0000 7f9ca3600700 1 -- 172.21.15.12:0/2013789749 wait complete. 2024-03-29T09:56:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.884+0000 7f9ca3600700 1 Processor -- start 2024-03-29T09:56:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.884+0000 7f9ca3600700 1 -- start start 2024-03-29T09:56:00.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.885+0000 7f9ca3600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c9c0d16a0 0x7f9c9c1e8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.885+0000 7f9ca3600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c9c0d2370 0x7f9c9c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.885+0000 7f9ca3600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c9c15d9d0 0x7f9c9c1e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.885+0000 7f9ca3600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c1e9270 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.885+0000 7f9ca3600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c1e7c90 con 0x7f9c9c0d2370 2024-03-29T09:56:00.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.885+0000 7f9ca3600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c1e7f70 con 0x7f9c9c15d9d0 2024-03-29T09:56:00.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.886+0000 7f9ca0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c9c0d16a0 0x7f9c9c1e8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.886+0000 7f9c9be00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c9c0d2370 0x7f9c9c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.886+0000 7f9c9be00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c9c0d2370 0x7f9c9c159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:48758/0 (socket says 172.21.15.12:48758) 2024-03-29T09:56:00.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.886+0000 7f9c9be00700 1 -- 172.21.15.12:0/3848548955 learned_addr learned my addr 172.21.15.12:0/3848548955 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:00.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.886+0000 7f9ca1600700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c9c15d9d0 0x7f9c9c1e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.886+0000 7f9ca0c00700 1 -- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c9c15d9d0 msgr2=0x7f9c9c1e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.886+0000 7f9ca0c00700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c9c15d9d0 0x7f9c9c1e7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.886+0000 7f9ca0c00700 1 -- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c9c0d2370 msgr2=0x7f9c9c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:00.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.886+0000 7f9ca0c00700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c9c0d2370 0x7f9c9c159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:00.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.886+0000 7f9ca0c00700 1 -- 172.21.15.12:0/3848548955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c88009500 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.887+0000 7f9ca0c00700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c9c0d16a0 0x7f9c9c1e8d30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c410 tx=0x7f9c9000c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:00.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.887+0000 7f9c98c00700 1 -- 172.21.15.12:0/3848548955 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90004c80 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.887+0000 7f9c98c00700 1 -- 172.21.15.12:0/3848548955 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c90004de0 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.887+0000 7f9c98c00700 1 -- 172.21.15.12:0/3848548955 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90007310 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.887+0000 7f9ca3600700 1 -- 172.21.15.12:0/3848548955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c1e8250 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.887+0000 7f9ca3600700 1 -- 172.21.15.12:0/3848548955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c1e8810 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.888+0000 7f9ca3600700 1 -- 172.21.15.12:0/3848548955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c0d1ca0 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.889+0000 7f9c98c00700 1 -- 172.21.15.12:0/3848548955 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 11) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9c90007470 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.890+0000 7f9c98c00700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c7c06af50 0x7f9c7c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:00.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.890+0000 7f9c98c00700 1 -- 172.21.15.12:0/3848548955 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9c900894d0 con 0x7f9c9c0d16a0 2024-03-29T09:56:00.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.891+0000 7f9c9be00700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c7c06af50 0x7f9c7c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:00.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.891+0000 7f9c9be00700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c7c06af50 0x7f9c7c06d410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9c88014040 tx=0x7f9c88008070 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:00.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:00.896+0000 7f9c98c00700 1 -- 172.21.15.12:0/3848548955 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9c90054140 con 0x7f9c9c0d16a0 2024-03-29T09:56:01.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.159+0000 7f9ca3600700 1 -- 172.21.15.12:0/3848548955 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c9c161fa0 con 0x7f9c7c06af50 2024-03-29T09:56:01.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.217+0000 7f9c98c00700 1 -- 172.21.15.12:0/3848548955 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9c88021070 con 0x7f9c7c06af50 2024-03-29T09:56:01.217 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/dir2 2024-03-29T09:56:01.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.219+0000 7f9ca3600700 1 -- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c7c06af50 msgr2=0x7f9c7c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.219+0000 7f9ca3600700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c7c06af50 0x7f9c7c06d410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9c88014040 tx=0x7f9c88008070 comp rx=0 tx=0).stop 2024-03-29T09:56:01.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.219+0000 7f9ca3600700 1 -- 172.21.15.12:0/3848548955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c9c0d16a0 msgr2=0x7f9c9c1e8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.219+0000 7f9ca3600700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c9c0d16a0 0x7f9c9c1e8d30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c410 tx=0x7f9c9000c720 comp rx=0 tx=0).stop 2024-03-29T09:56:01.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.220+0000 7f9ca3600700 1 -- 172.21.15.12:0/3848548955 shutdown_connections 2024-03-29T09:56:01.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.220+0000 7f9ca3600700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c7c06af50 0x7f9c7c06d410 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.220+0000 7f9ca3600700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c9c15d9d0 0x7f9c9c1e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.220+0000 7f9ca3600700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c9c0d16a0 0x7f9c9c1e8d30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.220+0000 7f9ca3600700 1 --2- 172.21.15.12:0/3848548955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c9c0d2370 0x7f9c9c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.220+0000 7f9ca3600700 1 -- 172.21.15.12:0/3848548955 >> 172.21.15.12:0/3848548955 conn(0x7f9c9c08aff0 msgr2=0x7f9c9c0c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:01.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.220+0000 7f9ca3600700 1 -- 172.21.15.12:0/3848548955 shutdown_connections 2024-03-29T09:56:01.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.220+0000 7f9ca3600700 1 -- 172.21.15.12:0/3848548955 wait complete. 2024-03-29T09:56:01.237 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2024-03-29T09:56:01.237 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2024-03-29T09:56:01.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.438+0000 7fac23200700 1 Processor -- start 2024-03-29T09:56:01.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.438+0000 7fac23200700 1 -- start start 2024-03-29T09:56:01.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.439+0000 7fac23200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac1c0d2540 0x7fac1c1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.439+0000 7fac23200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1c161420 con 0x7fac1c15d9d0 2024-03-29T09:56:01.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.439+0000 7fac23200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1c161700 con 0x7fac1c0d16a0 2024-03-29T09:56:01.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.439+0000 7fac23200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1c0d4340 con 0x7fac1c0d2540 2024-03-29T09:56:01.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.439+0000 7fac21200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac1c0d2540 0x7fac1c1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.439+0000 7fac1be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fac1c15d9d0 0x7fac1c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33604/0 (socket says 172.21.15.12:33604) 2024-03-29T09:56:01.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.439+0000 7fac1be00700 1 -- 172.21.15.12:0/3471388281 learned_addr learned my addr 172.21.15.12:0/3471388281 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:56:01.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.440+0000 7fac21200700 1 -- 172.21.15.12:0/3471388281 >> v1:172.21.15.159:6789/0 conn(0x7fac1c0d16a0 legacy=0x7fac1c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.440+0000 7fac21200700 1 -- 172.21.15.12:0/3471388281 >> v1:172.21.15.12:6789/0 conn(0x7fac1c15d9d0 legacy=0x7fac1c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.440+0000 7fac21200700 1 -- 172.21.15.12:0/3471388281 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c1619e0 con 0x7fac1c0d2540 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.440+0000 7fac1aa00700 1 -- 172.21.15.12:0/3471388281 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406531973 0 0) 0x7fac1c161700 con 0x7fac1c0d16a0 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.440+0000 7fac21200700 1 --2- 172.21.15.12:0/3471388281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c0d2540 0x7fac1c1687b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fac14009d90 tx=0x7fac1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa74e47716be638 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.440+0000 7fac1aa00700 1 -- 172.21.15.12:0/3471388281 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1400e070 con 0x7fac1c0d2540 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.440+0000 7fac1aa00700 1 -- 172.21.15.12:0/3471388281 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1400bad0 con 0x7fac1c0d2540 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.440+0000 7fac1aa00700 1 -- 172.21.15.12:0/3471388281 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1400be80 con 0x7fac1c0d2540 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.441+0000 7fac23200700 1 -- 172.21.15.12:0/3471388281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c0d2540 msgr2=0x7fac1c1687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.441+0000 7fac23200700 1 --2- 172.21.15.12:0/3471388281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c0d2540 0x7fac1c1687b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fac14009d90 tx=0x7fac1400b330 comp rx=0 tx=0).stop 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.441+0000 7fac23200700 1 -- 172.21.15.12:0/3471388281 shutdown_connections 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.441+0000 7fac23200700 1 --2- 172.21.15.12:0/3471388281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c0d2540 0x7fac1c1687b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.441+0000 7fac23200700 1 -- 172.21.15.12:0/3471388281 >> 172.21.15.12:0/3471388281 conn(0x7fac1c08aff0 msgr2=0x7fac1c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.441+0000 7fac23200700 1 -- 172.21.15.12:0/3471388281 shutdown_connections 2024-03-29T09:56:01.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.441+0000 7fac23200700 1 -- 172.21.15.12:0/3471388281 wait complete. 2024-03-29T09:56:01.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.443+0000 7fac23200700 1 Processor -- start 2024-03-29T09:56:01.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.443+0000 7fac23200700 1 -- start start 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac23200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac1c0d16a0 0x7fac1c153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac23200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac1c0d2540 0x7fac1c1606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac23200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c15d9d0 0x7fac1c0b7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac23200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c1540d0 con 0x7fac1c15d9d0 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac23200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c15e400 con 0x7fac1c0d2540 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac23200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c0b8760 con 0x7fac1c0d16a0 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac21200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c15d9d0 0x7fac1c0b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac21200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c15d9d0 0x7fac1c0b7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57560/0 (socket says 172.21.15.12:57560) 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac1be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac1c0d16a0 0x7fac1c153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac21200700 1 -- 172.21.15.12:0/2614365917 learned_addr learned my addr 172.21.15.12:0/2614365917 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:01.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.444+0000 7fac1b400700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac1c0d2540 0x7fac1c1606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac21200700 1 -- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac1c0d16a0 msgr2=0x7fac1c153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac21200700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac1c0d16a0 0x7fac1c153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac21200700 1 -- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac1c0d2540 msgr2=0x7fac1c1606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac21200700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac1c0d2540 0x7fac1c1606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac21200700 1 -- 172.21.15.12:0/2614365917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac14009a70 con 0x7fac1c15d9d0 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac21200700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c15d9d0 0x7fac1c0b7750 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fac1c08c230 tx=0x7fac14004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac13e00700 1 -- 172.21.15.12:0/2614365917 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14012630 con 0x7fac1c15d9d0 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac13e00700 1 -- 172.21.15.12:0/2614365917 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac14012790 con 0x7fac1c15d9d0 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac23200700 1 -- 172.21.15.12:0/2614365917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c0b8a40 con 0x7fac1c15d9d0 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac13e00700 1 -- 172.21.15.12:0/2614365917 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14012b40 con 0x7fac1c15d9d0 2024-03-29T09:56:01.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.445+0000 7fac23200700 1 -- 172.21.15.12:0/2614365917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c15e090 con 0x7fac1c15d9d0 2024-03-29T09:56:01.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.446+0000 7fac23200700 1 -- 172.21.15.12:0/2614365917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c0d1ca0 con 0x7fac1c15d9d0 2024-03-29T09:56:01.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.447+0000 7fac13e00700 1 -- 172.21.15.12:0/2614365917 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 11) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fac14012ca0 con 0x7fac1c15d9d0 2024-03-29T09:56:01.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.448+0000 7fac13e00700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fabfc06af50 0x7fabfc06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.448+0000 7fac13e00700 1 -- 172.21.15.12:0/2614365917 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fac1408bfb0 con 0x7fac1c15d9d0 2024-03-29T09:56:01.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.449+0000 7fac1be00700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fabfc06af50 0x7fabfc06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.449+0000 7fac1be00700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fabfc06af50 0x7fabfc06d410 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac0c000f60 tx=0x7fac0c004a50 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:01.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.453+0000 7fac13e00700 1 -- 172.21.15.12:0/2614365917 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac14056d30 con 0x7fac1c15d9d0 2024-03-29T09:56:01.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.698+0000 7fac23200700 1 -- 172.21.15.12:0/2614365917 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fac1c0b6360 con 0x7fabfc06af50 2024-03-29T09:56:01.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.755+0000 7fac13e00700 1 -- 172.21.15.12:0/2614365917 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fac1c0b6360 con 0x7fabfc06af50 2024-03-29T09:56:01.755 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/dir3 2024-03-29T09:56:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.757+0000 7fac23200700 1 -- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fabfc06af50 msgr2=0x7fabfc06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.757+0000 7fac23200700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fabfc06af50 0x7fabfc06d410 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac0c000f60 tx=0x7fac0c004a50 comp rx=0 tx=0).stop 2024-03-29T09:56:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.757+0000 7fac23200700 1 -- 172.21.15.12:0/2614365917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c15d9d0 msgr2=0x7fac1c0b7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.757+0000 7fac23200700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c15d9d0 0x7fac1c0b7750 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fac1c08c230 tx=0x7fac14004ad0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.758+0000 7fac23200700 1 -- 172.21.15.12:0/2614365917 shutdown_connections 2024-03-29T09:56:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.758+0000 7fac23200700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fabfc06af50 0x7fabfc06d410 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.758+0000 7fac23200700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1c15d9d0 0x7fac1c0b7750 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.758+0000 7fac23200700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac1c0d16a0 0x7fac1c153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.758+0000 7fac23200700 1 --2- 172.21.15.12:0/2614365917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac1c0d2540 0x7fac1c1606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.758+0000 7fac23200700 1 -- 172.21.15.12:0/2614365917 >> 172.21.15.12:0/2614365917 conn(0x7fac1c08aff0 msgr2=0x7fac1c0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.758+0000 7fac23200700 1 -- 172.21.15.12:0/2614365917 shutdown_connections 2024-03-29T09:56:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.758+0000 7fac23200700 1 -- 172.21.15.12:0/2614365917 wait complete. 2024-03-29T09:56:01.775 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2024-03-29T09:56:01.775 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2024-03-29T09:56:01.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.978+0000 7ff3f1200700 1 Processor -- start 2024-03-29T09:56:01.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.978+0000 7ff3f1200700 1 -- start start 2024-03-29T09:56:01.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.979+0000 7ff3f1200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff3ec15d9d0 0x7ff3ec0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.979+0000 7ff3f1200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 0x7ff3ec0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.979+0000 7ff3f1200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec161420 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.979+0000 7ff3f1200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3ec15edc0 con 0x7ff3ec15d9d0 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.979+0000 7ff3f1200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec161700 con 0x7ff3ec0d2540 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.979+0000 7ff3ea800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 0x7ff3ec0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.979+0000 7ff3ea800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 0x7ff3ec0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:36628/0 (socket says 172.21.15.12:36628) 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.979+0000 7ff3ea800700 1 -- 172.21.15.12:0/1722735035 learned_addr learned my addr 172.21.15.12:0/1722735035 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.979+0000 7ff3e9400700 1 --1- 172.21.15.12:0/1722735035 >> v1:172.21.15.159:6789/0 conn(0x7ff3ec0d16a0 0x7ff3ec0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:34436/0 (socket says 172.21.15.12:34436) 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.979+0000 7ff3e9e00700 1 --2- 172.21.15.12:0/1722735035 >> v2:172.21.15.159:3300/0 conn(0x7ff3ec15d9d0 0x7ff3ec0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.980+0000 7ff3ea800700 1 -- 172.21.15.12:0/1722735035 >> v2:172.21.15.159:3300/0 conn(0x7ff3ec15d9d0 msgr2=0x7ff3ec0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.980+0000 7ff3ea800700 1 --2- 172.21.15.12:0/1722735035 >> v2:172.21.15.159:3300/0 conn(0x7ff3ec15d9d0 0x7ff3ec0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.980+0000 7ff3ea800700 1 -- 172.21.15.12:0/1722735035 >> v1:172.21.15.159:6789/0 conn(0x7ff3ec0d16a0 legacy=0x7ff3ec0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.980+0000 7ff3ea800700 1 -- 172.21.15.12:0/1722735035 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec1619e0 con 0x7ff3ec0d2540 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.980+0000 7ff3e8a00700 1 -- 172.21.15.12:0/1722735035 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200167347 0 0) 0x7ff3ec161420 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.980+0000 7ff3ea800700 1 --2- 172.21.15.12:0/1722735035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 0x7ff3ec0ce360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009e50 tx=0x7ff3e400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1810a6e1fa363daa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.980+0000 7ff3e8a00700 1 -- 172.21.15.12:0/1722735035 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e400e070 con 0x7ff3ec0d2540 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.980+0000 7ff3e8a00700 1 -- 172.21.15.12:0/1722735035 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e400b910 con 0x7ff3ec0d2540 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.980+0000 7ff3e8a00700 1 -- 172.21.15.12:0/1722735035 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e400bcc0 con 0x7ff3ec0d2540 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.981+0000 7ff3f1200700 1 -- 172.21.15.12:0/1722735035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 msgr2=0x7ff3ec0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.981+0000 7ff3f1200700 1 --2- 172.21.15.12:0/1722735035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 0x7ff3ec0ce360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009e50 tx=0x7ff3e400b330 comp rx=0 tx=0).stop 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.981+0000 7ff3f1200700 1 -- 172.21.15.12:0/1722735035 shutdown_connections 2024-03-29T09:56:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.981+0000 7ff3f1200700 1 --2- 172.21.15.12:0/1722735035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 0x7ff3ec0ce360 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.981+0000 7ff3f1200700 1 --2- 172.21.15.12:0/1722735035 >> v2:172.21.15.159:3300/0 conn(0x7ff3ec15d9d0 0x7ff3ec0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.981+0000 7ff3f1200700 1 -- 172.21.15.12:0/1722735035 >> 172.21.15.12:0/1722735035 conn(0x7ff3ec08aff0 msgr2=0x7ff3ec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.981+0000 7ff3f1200700 1 -- 172.21.15.12:0/1722735035 shutdown_connections 2024-03-29T09:56:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.981+0000 7ff3f1200700 1 -- 172.21.15.12:0/1722735035 wait complete. 2024-03-29T09:56:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.983+0000 7ff3f1200700 1 Processor -- start 2024-03-29T09:56:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.984+0000 7ff3f1200700 1 -- start start 2024-03-29T09:56:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3f1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3ec0d16a0 0x7ff3ec0cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3f1200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 0x7ff3ec0b7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3f1200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3ec15d9d0 0x7ff3ec15c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3f1200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0cb4f0 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3f1200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0b81f0 con 0x7ff3ec15d9d0 2024-03-29T09:56:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3f1200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec15c6a0 con 0x7ff3ec0d2540 2024-03-29T09:56:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3e9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3ec0d16a0 0x7ff3ec0cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3e9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3ec0d16a0 0x7ff3ec0cafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57566/0 (socket says 172.21.15.12:57566) 2024-03-29T09:56:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3e9e00700 1 -- 172.21.15.12:0/4007608859 learned_addr learned my addr 172.21.15.12:0/4007608859 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3e9400700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 0x7ff3ec0b7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3ea800700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3ec15d9d0 0x7ff3ec15c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3e9e00700 1 -- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 msgr2=0x7ff3ec0b7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.985+0000 7ff3e9e00700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 0x7ff3ec0b7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.986+0000 7ff3e9e00700 1 -- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3ec15d9d0 msgr2=0x7ff3ec15c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.986+0000 7ff3e9e00700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3ec15d9d0 0x7ff3ec15c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.986+0000 7ff3e9e00700 1 -- 172.21.15.12:0/4007608859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4009b30 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.986+0000 7ff3ea800700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3ec15d9d0 0x7ff3ec15c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T09:56:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.986+0000 7ff3e9e00700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3ec0d16a0 0x7ff3ec0cafb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00e870 tx=0x7ff3dc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.986+0000 7ff3e2000700 1 -- 172.21.15.12:0/4007608859 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc009590 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.986+0000 7ff3e2000700 1 -- 172.21.15.12:0/4007608859 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc00ee70 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.986+0000 7ff3f1200700 1 -- 172.21.15.12:0/4007608859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec0cb700 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.986+0000 7ff3e2000700 1 -- 172.21.15.12:0/4007608859 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc00ca50 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.986+0000 7ff3f1200700 1 -- 172.21.15.12:0/4007608859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec0cbcc0 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.987+0000 7ff3f1200700 1 -- 172.21.15.12:0/4007608859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec0d1ca0 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.988+0000 7ff3e2000700 1 -- 172.21.15.12:0/4007608859 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 11) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff3dc00cbb0 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.989+0000 7ff3e2000700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3cc06af20 0x7ff3cc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.989+0000 7ff3e2000700 1 -- 172.21.15.12:0/4007608859 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff3dc0898d0 con 0x7ff3ec0d16a0 2024-03-29T09:56:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.990+0000 7ff3e9400700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3cc06af20 0x7ff3cc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:01.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.990+0000 7ff3e9400700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3cc06af20 0x7ff3cc06d3e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec0b2290 tx=0x7ff3d8007bd0 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:01.993+0000 7ff3e2000700 1 -- 172.21.15.12:0/4007608859 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff3dc054640 con 0x7ff3ec0d16a0 2024-03-29T09:56:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.238+0000 7ff3f1200700 1 -- 172.21.15.12:0/4007608859 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3ec161fa0 con 0x7ff3cc06af20 2024-03-29T09:56:02.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.301+0000 7ff3e2000700 1 -- 172.21.15.12:0/4007608859 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff3d8007d60 con 0x7ff3cc06af20 2024-03-29T09:56:02.301 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/dir4 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 -- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3cc06af20 msgr2=0x7ff3cc06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3cc06af20 0x7ff3cc06d3e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec0b2290 tx=0x7ff3d8007bd0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 -- 172.21.15.12:0/4007608859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3ec0d16a0 msgr2=0x7ff3ec0cafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3ec0d16a0 0x7ff3ec0cafb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00e870 tx=0x7ff3dc00ec30 comp rx=0 tx=0).stop 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 -- 172.21.15.12:0/4007608859 shutdown_connections 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3cc06af20 0x7ff3cc06d3e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3ec15d9d0 0x7ff3ec15c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3ec0d16a0 0x7ff3ec0cafb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 --2- 172.21.15.12:0/4007608859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3ec0d2540 0x7ff3ec0b7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 -- 172.21.15.12:0/4007608859 >> 172.21.15.12:0/4007608859 conn(0x7ff3ec08aff0 msgr2=0x7ff3ec0d1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 -- 172.21.15.12:0/4007608859 shutdown_connections 2024-03-29T09:56:02.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.304+0000 7ff3f1200700 1 -- 172.21.15.12:0/4007608859 wait complete. 2024-03-29T09:56:02.322 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2024-03-29T09:56:02.322 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2024-03-29T09:56:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.523+0000 7f4ee5800700 1 Processor -- start 2024-03-29T09:56:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.523+0000 7f4ee5800700 1 -- start start 2024-03-29T09:56:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.524+0000 7f4ee5800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4ee00cbb30 0x7f4ee00cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.524+0000 7f4ee5800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee00cc450 con 0x7f4ee00cd480 2024-03-29T09:56:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.524+0000 7f4ee5800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee00cdae0 con 0x7f4ee00cd020 2024-03-29T09:56:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.524+0000 7f4ee5800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee00cd8f0 con 0x7f4ee00cbb30 2024-03-29T09:56:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.525+0000 7f4edf400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4ee00cd480 0x7f4ee006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33608/0 (socket says 172.21.15.12:33608) 2024-03-29T09:56:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.525+0000 7f4edf400700 1 -- 172.21.15.12:0/3063641534 learned_addr learned my addr 172.21.15.12:0/3063641534 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:56:02.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.525+0000 7f4edea00700 1 --2- 172.21.15.12:0/3063641534 >> v2:172.21.15.159:3300/0 conn(0x7f4ee00cbb30 0x7f4ee00cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.525+0000 7f4edd600700 1 -- 172.21.15.12:0/3063641534 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016845203 0 0) 0x7f4ee00cc450 con 0x7f4ee00cd480 2024-03-29T09:56:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.525+0000 7f4edd600700 1 -- 172.21.15.12:0/3063641534 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec40035b0 con 0x7f4ee00cd480 2024-03-29T09:56:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.525+0000 7f4edea00700 1 -- 172.21.15.12:0/3063641534 >> v1:172.21.15.159:6789/0 conn(0x7f4ee00cd020 legacy=0x7f4ee00b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.525+0000 7f4edea00700 1 -- 172.21.15.12:0/3063641534 >> v1:172.21.15.12:6789/0 conn(0x7f4ee00cd480 legacy=0x7f4ee006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.525+0000 7f4edea00700 1 -- 172.21.15.12:0/3063641534 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee00cddc0 con 0x7f4ee00cbb30 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4edea00700 1 --2- 172.21.15.12:0/3063641534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ee00cbb30 0x7f4ee00cbf10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009840 tx=0x7f4ed0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab6a8b98123ddef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4edd600700 1 -- 172.21.15.12:0/3063641534 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0011830 con 0x7f4ee00cbb30 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4edd600700 1 -- 172.21.15.12:0/3063641534 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed0011990 con 0x7f4ee00cbb30 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4edd600700 1 -- 172.21.15.12:0/3063641534 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0011d20 con 0x7f4ee00cbb30 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4ee5800700 1 -- 172.21.15.12:0/3063641534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ee00cbb30 msgr2=0x7f4ee00cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4ee5800700 1 --2- 172.21.15.12:0/3063641534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ee00cbb30 0x7f4ee00cbf10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009840 tx=0x7f4ed0009b50 comp rx=0 tx=0).stop 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4ee5800700 1 -- 172.21.15.12:0/3063641534 shutdown_connections 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4ee5800700 1 --2- 172.21.15.12:0/3063641534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ee00cbb30 0x7f4ee00cbf10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4ee5800700 1 -- 172.21.15.12:0/3063641534 >> 172.21.15.12:0/3063641534 conn(0x7f4ee00b0d70 msgr2=0x7f4ee00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4ee5800700 1 -- 172.21.15.12:0/3063641534 shutdown_connections 2024-03-29T09:56:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.526+0000 7f4ee5800700 1 -- 172.21.15.12:0/3063641534 wait complete. 2024-03-29T09:56:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.529+0000 7f4ee5800700 1 Processor -- start 2024-03-29T09:56:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.529+0000 7f4ee5800700 1 -- start start 2024-03-29T09:56:02.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.530+0000 7f4ee5800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ee00cbb30 0x7f4ee00b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:02.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.530+0000 7f4ee5800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ee00cd020 0x7f4ee00b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:02.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.530+0000 7f4ee5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ee00cd480 0x7f4ee0145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:02.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.530+0000 7f4ee5800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee0145ad0 con 0x7f4ee00cd480 2024-03-29T09:56:02.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.530+0000 7f4ee5800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee015eaf0 con 0x7f4ee00cd020 2024-03-29T09:56:02.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.530+0000 7f4ee5800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee00b2480 con 0x7f4ee00cbb30 2024-03-29T09:56:02.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.530+0000 7f4edf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ee00cd480 0x7f4ee0145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:02.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4edf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ee00cd480 0x7f4ee0145590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57576/0 (socket says 172.21.15.12:57576) 2024-03-29T09:56:02.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4edf400700 1 -- 172.21.15.12:0/349369664 learned_addr learned my addr 172.21.15.12:0/349369664 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:02.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4edea00700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ee00cbb30 0x7f4ee00b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:02.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4ede000700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ee00cd020 0x7f4ee00b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:02.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4edf400700 1 -- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ee00cbb30 msgr2=0x7f4ee00b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:02.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4edf400700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ee00cbb30 0x7f4ee00b1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4edf400700 1 -- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ee00cd020 msgr2=0x7f4ee00b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:02.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4edf400700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ee00cd020 0x7f4ee00b3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4edf400700 1 -- 172.21.15.12:0/349369664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed00094a0 con 0x7f4ee00cd480 2024-03-29T09:56:02.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4edf400700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ee00cd480 0x7f4ee0145590 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4ed800cb40 tx=0x7f4ed800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:02.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4ed6a00700 1 -- 172.21.15.12:0/349369664 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed800a9e0 con 0x7f4ee00cd480 2024-03-29T09:56:02.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.531+0000 7f4ed6a00700 1 -- 172.21.15.12:0/349369664 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed800ab40 con 0x7f4ee00cd480 2024-03-29T09:56:02.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.532+0000 7f4ed6a00700 1 -- 172.21.15.12:0/349369664 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8012400 con 0x7f4ee00cd480 2024-03-29T09:56:02.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.532+0000 7f4ee5800700 1 -- 172.21.15.12:0/349369664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee00b2760 con 0x7f4ee00cd480 2024-03-29T09:56:02.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.532+0000 7f4ee5800700 1 -- 172.21.15.12:0/349369664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee00b2d20 con 0x7f4ee00cd480 2024-03-29T09:56:02.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.533+0000 7f4ee5800700 1 -- 172.21.15.12:0/349369664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee0067720 con 0x7f4ee00cd480 2024-03-29T09:56:02.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.534+0000 7f4ed6a00700 1 -- 172.21.15.12:0/349369664 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 11) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f4ed801b3b0 con 0x7f4ee00cd480 2024-03-29T09:56:02.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.535+0000 7f4ed6a00700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ec006af20 0x7f4ec006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:02.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.535+0000 7f4ed6a00700 1 -- 172.21.15.12:0/349369664 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f4ed80893b0 con 0x7f4ee00cd480 2024-03-29T09:56:02.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.535+0000 7f4edea00700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ec006af20 0x7f4ec006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:02.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.536+0000 7f4edea00700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ec006af20 0x7f4ec006d3e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00090b0 tx=0x7f4ed001c040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.539+0000 7f4ed6a00700 1 -- 172.21.15.12:0/349369664 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ed80541d0 con 0x7f4ee00cd480 2024-03-29T09:56:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.785+0000 7f4ee5800700 1 -- 172.21.15.12:0/349369664 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ee006df00 con 0x7f4ec006af20 2024-03-29T09:56:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.853+0000 7f4ed6a00700 1 -- 172.21.15.12:0/349369664 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f4ed0011b80 con 0x7f4ec006af20 2024-03-29T09:56:02.853 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/dir5 2024-03-29T09:56:02.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.855+0000 7f4ee5800700 1 -- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ec006af20 msgr2=0x7f4ec006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:02.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.855+0000 7f4ee5800700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ec006af20 0x7f4ec006d3e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00090b0 tx=0x7f4ed001c040 comp rx=0 tx=0).stop 2024-03-29T09:56:02.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.855+0000 7f4ee5800700 1 -- 172.21.15.12:0/349369664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ee00cd480 msgr2=0x7f4ee0145590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:02.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.855+0000 7f4ee5800700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ee00cd480 0x7f4ee0145590 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4ed800cb40 tx=0x7f4ed800ce50 comp rx=0 tx=0).stop 2024-03-29T09:56:02.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.855+0000 7f4ee5800700 1 -- 172.21.15.12:0/349369664 shutdown_connections 2024-03-29T09:56:02.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.855+0000 7f4ee5800700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ec006af20 0x7f4ec006d3e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.855+0000 7f4ee5800700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ee00cd480 0x7f4ee0145590 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.855+0000 7f4ee5800700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ee00cbb30 0x7f4ee00b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.855+0000 7f4ee5800700 1 --2- 172.21.15.12:0/349369664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ee00cd020 0x7f4ee00b3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:02.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.855+0000 7f4ee5800700 1 -- 172.21.15.12:0/349369664 >> 172.21.15.12:0/349369664 conn(0x7f4ee00b0d70 msgr2=0x7f4ee006e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:02.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.856+0000 7f4ee5800700 1 -- 172.21.15.12:0/349369664 shutdown_connections 2024-03-29T09:56:02.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:02.856+0000 7f4ee5800700 1 -- 172.21.15.12:0/349369664 wait complete. 2024-03-29T09:56:02.873 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2024-03-29T09:56:02.874 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2024-03-29T09:56:03.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.075+0000 7f4c59600700 1 Processor -- start 2024-03-29T09:56:03.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.075+0000 7f4c59600700 1 -- start start 2024-03-29T09:56:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.076+0000 7f4c59600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4c540d16a0 0x7f4c540d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.076+0000 7f4c59600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c540d2370 0x7f4c540ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.076+0000 7f4c59600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c54161420 con 0x7f4c5415d9d0 2024-03-29T09:56:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.076+0000 7f4c59600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c540d2820 con 0x7f4c540d16a0 2024-03-29T09:56:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.076+0000 7f4c59600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540d4340 con 0x7f4c540d2370 2024-03-29T09:56:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.076+0000 7f4c52a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4c5415d9d0 0x7f4c540d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33620/0 (socket says 172.21.15.12:33620) 2024-03-29T09:56:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.076+0000 7f4c52a00700 1 -- 172.21.15.12:0/883081055 learned_addr learned my addr 172.21.15.12:0/883081055 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:56:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c52000700 1 --2- 172.21.15.12:0/883081055 >> v2:172.21.15.159:3300/0 conn(0x7f4c540d16a0 0x7f4c540d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c53400700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c540d2370 0x7f4c540ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360159622 0 0) 0x7f4c54161420 con 0x7f4c5415d9d0 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c380035b0 con 0x7f4c5415d9d0 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 747937770 0 0) 0x7f4c380035b0 con 0x7f4c5415d9d0 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c540d2370 msgr2=0x7f4c540ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c540d2370 0x7f4c540ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 >> v2:172.21.15.159:3300/0 conn(0x7f4c540d16a0 msgr2=0x7f4c540d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 --2- 172.21.15.12:0/883081055 >> v2:172.21.15.159:3300/0 conn(0x7f4c540d16a0 0x7f4c540d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c54161700 con 0x7f4c5415d9d0 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f4c44001670 con 0x7f4c5415d9d0 2024-03-29T09:56:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 >> v1:172.21.15.12:6789/0 conn(0x7f4c5415d9d0 legacy=0x7f4c540d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c38003a00 0x7f4c38003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c540d2370 0x7f4c38004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c38004d90 0x7f4c38005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540d2820 con 0x7f4c38004d90 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3800ba10 con 0x7f4c38003a00 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3800bb50 con 0x7f4c540d2370 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.077+0000 7f4c52000700 1 --2- 172.21.15.12:0/883081055 >> v2:172.21.15.159:3300/0 conn(0x7f4c540d16a0 0x7f4c540d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c53400700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c38004d90 0x7f4c38005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c52a00700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c38003a00 0x7f4c38003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c53400700 1 -- 172.21.15.12:0/883081055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c540d2370 msgr2=0x7f4c38004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c53400700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c540d2370 0x7f4c38004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c53400700 1 -- 172.21.15.12:0/883081055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c38003a00 msgr2=0x7f4c38003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c53400700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c38003a00 0x7f4c38003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c53400700 1 -- 172.21.15.12:0/883081055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c380035b0 con 0x7f4c38004d90 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c53400700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c38004d90 0x7f4c38005210 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009fc0 tx=0x7f4c38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fabc5a58dcd8350 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00ec30 con 0x7f4c38004d90 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c00f040 con 0x7f4c38004d90 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.078+0000 7f4c51600700 1 -- 172.21.15.12:0/883081055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c014480 con 0x7f4c38004d90 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.079+0000 7f4c59600700 1 -- 172.21.15.12:0/883081055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c38004d90 msgr2=0x7f4c38005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.079+0000 7f4c59600700 1 --2- 172.21.15.12:0/883081055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c38004d90 0x7f4c38005210 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009fc0 tx=0x7f4c38005880 comp rx=0 tx=0).stop 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.079+0000 7f4c53400700 1 -- 172.21.15.12:0/883081055 reap_dead start 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.079+0000 7f4c59600700 1 -- 172.21.15.12:0/883081055 shutdown_connections 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.079+0000 7f4c59600700 1 -- 172.21.15.12:0/883081055 >> 172.21.15.12:0/883081055 conn(0x7f4c5408aff0 msgr2=0x7f4c540d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.079+0000 7f4c59600700 1 -- 172.21.15.12:0/883081055 shutdown_connections 2024-03-29T09:56:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.079+0000 7f4c59600700 1 -- 172.21.15.12:0/883081055 wait complete. 2024-03-29T09:56:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.081+0000 7f4c59600700 1 Processor -- start 2024-03-29T09:56:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.081+0000 7f4c59600700 1 -- start start 2024-03-29T09:56:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.082+0000 7f4c59600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c38004d90 0x7f4c5414e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.082+0000 7f4c59600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c540b8c20 0x7f4c5415f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.082+0000 7f4c59600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c54160a50 0x7f4c540b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.082+0000 7f4c59600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540b76d0 con 0x7f4c540b8c20 2024-03-29T09:56:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.082+0000 7f4c59600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5414ee90 con 0x7f4c54160a50 2024-03-29T09:56:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.082+0000 7f4c59600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5415f9b0 con 0x7f4c38004d90 2024-03-29T09:56:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.082+0000 7f4c52000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c540b8c20 0x7f4c5415f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.082+0000 7f4c52000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c540b8c20 0x7f4c5415f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57598/0 (socket says 172.21.15.12:57598) 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.082+0000 7f4c52000700 1 -- 172.21.15.12:0/2907699744 learned_addr learned my addr 172.21.15.12:0/2907699744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.082+0000 7f4c52a00700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c38004d90 0x7f4c5414e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c53400700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c54160a50 0x7f4c540b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c52000700 1 -- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c38004d90 msgr2=0x7f4c5414e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c52000700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c38004d90 0x7f4c5414e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c52000700 1 -- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c54160a50 msgr2=0x7f4c540b4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c52000700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c54160a50 0x7f4c540b4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c52000700 1 -- 172.21.15.12:0/2907699744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5415fc90 con 0x7f4c540b8c20 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c52000700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c540b8c20 0x7f4c5415f040 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000eb60 tx=0x7f4c4000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c4aa00700 1 -- 172.21.15.12:0/2907699744 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40009e00 con 0x7f4c540b8c20 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c4aa00700 1 -- 172.21.15.12:0/2907699744 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4000c850 con 0x7f4c540b8c20 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c4aa00700 1 -- 172.21.15.12:0/2907699744 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4000cc00 con 0x7f4c540b8c20 2024-03-29T09:56:03.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c59600700 1 -- 172.21.15.12:0/2907699744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5415ff70 con 0x7f4c540b8c20 2024-03-29T09:56:03.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.083+0000 7f4c59600700 1 -- 172.21.15.12:0/2907699744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c54160530 con 0x7f4c540b8c20 2024-03-29T09:56:03.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.084+0000 7f4c59600700 1 -- 172.21.15.12:0/2907699744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c540d1ca0 con 0x7f4c540b8c20 2024-03-29T09:56:03.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.085+0000 7f4c4aa00700 1 -- 172.21.15.12:0/2907699744 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 11) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f4c40013470 con 0x7f4c540b8c20 2024-03-29T09:56:03.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.086+0000 7f4c4aa00700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c3406af20 0x7f4c3406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.086+0000 7f4c4aa00700 1 -- 172.21.15.12:0/2907699744 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f4c40089ce0 con 0x7f4c540b8c20 2024-03-29T09:56:03.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.087+0000 7f4c52a00700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c3406af20 0x7f4c3406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.087+0000 7f4c52a00700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c3406af20 0x7f4c3406d3e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c44001eb0 tx=0x7f4c44009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:03.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.091+0000 7f4c4aa00700 1 -- 172.21.15.12:0/2907699744 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c40058450 con 0x7f4c540b8c20 2024-03-29T09:56:03.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.335+0000 7f4c59600700 1 -- 172.21.15.12:0/2907699744 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c54161fa0 con 0x7f4c3406af20 2024-03-29T09:56:03.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.393+0000 7f4c4aa00700 1 -- 172.21.15.12:0/2907699744 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f4c4400d050 con 0x7f4c3406af20 2024-03-29T09:56:03.393 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/dir6 2024-03-29T09:56:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 -- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c3406af20 msgr2=0x7f4c3406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c3406af20 0x7f4c3406d3e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c44001eb0 tx=0x7f4c44009040 comp rx=0 tx=0).stop 2024-03-29T09:56:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 -- 172.21.15.12:0/2907699744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c540b8c20 msgr2=0x7f4c5415f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c540b8c20 0x7f4c5415f040 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000eb60 tx=0x7f4c4000ee70 comp rx=0 tx=0).stop 2024-03-29T09:56:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 -- 172.21.15.12:0/2907699744 shutdown_connections 2024-03-29T09:56:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c3406af20 0x7f4c3406d3e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c54160a50 0x7f4c540b4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c38004d90 0x7f4c5414e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 --2- 172.21.15.12:0/2907699744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c540b8c20 0x7f4c5415f040 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 -- 172.21.15.12:0/2907699744 >> 172.21.15.12:0/2907699744 conn(0x7f4c5408aff0 msgr2=0x7f4c540ce770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 -- 172.21.15.12:0/2907699744 shutdown_connections 2024-03-29T09:56:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.396+0000 7f4c59600700 1 -- 172.21.15.12:0/2907699744 wait complete. 2024-03-29T09:56:03.414 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2024-03-29T09:56:03.418 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2024-03-29T09:56:03.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.619+0000 7f585ca00700 1 Processor -- start 2024-03-29T09:56:03.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.619+0000 7f585ca00700 1 -- start start 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.620+0000 7f585ca00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f585815d9d0 0x7f58580d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.620+0000 7f585ca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58580d16a0 0x7f58580d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.620+0000 7f585ca00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5858161420 con 0x7f58580d2370 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.620+0000 7f585ca00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58580d27e0 con 0x7f58580d16a0 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.620+0000 7f585ca00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5858161700 con 0x7f585815d9d0 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.620+0000 7f5855400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58580d16a0 0x7f58580d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.620+0000 7f5855400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58580d16a0 0x7f58580d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57600/0 (socket says 172.21.15.12:57600) 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.620+0000 7f5855400700 1 -- 172.21.15.12:0/2511046065 learned_addr learned my addr 172.21.15.12:0/2511046065 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.620+0000 7f5855e00700 1 --2- 172.21.15.12:0/2511046065 >> v2:172.21.15.159:3300/0 conn(0x7f585815d9d0 0x7f58580d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.621+0000 7f5855400700 1 -- 172.21.15.12:0/2511046065 >> v2:172.21.15.159:3300/0 conn(0x7f585815d9d0 msgr2=0x7f58580d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.621+0000 7f5855400700 1 --2- 172.21.15.12:0/2511046065 >> v2:172.21.15.159:3300/0 conn(0x7f585815d9d0 0x7f58580d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.621+0000 7f5855400700 1 -- 172.21.15.12:0/2511046065 >> v1:172.21.15.159:6789/0 conn(0x7f58580d2370 legacy=0x7f585815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.621+0000 7f5855400700 1 -- 172.21.15.12:0/2511046065 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58581619e0 con 0x7f58580d16a0 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.621+0000 7f5855400700 1 --2- 172.21.15.12:0/2511046065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58580d16a0 0x7f58580d1e30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f58580d0ff0 tx=0x7f5844009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182f5eb1bb26cfea server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.621+0000 7f5854a00700 1 -- 172.21.15.12:0/2511046065 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5844011830 con 0x7f58580d16a0 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.622+0000 7f5854a00700 1 -- 172.21.15.12:0/2511046065 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5844011990 con 0x7f58580d16a0 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.622+0000 7f5854a00700 1 -- 172.21.15.12:0/2511046065 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5844011d40 con 0x7f58580d16a0 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.622+0000 7f585ca00700 1 -- 172.21.15.12:0/2511046065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58580d16a0 msgr2=0x7f58580d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.622+0000 7f585ca00700 1 --2- 172.21.15.12:0/2511046065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58580d16a0 0x7f58580d1e30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f58580d0ff0 tx=0x7f5844009bc0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.622+0000 7f585ca00700 1 -- 172.21.15.12:0/2511046065 shutdown_connections 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.622+0000 7f585ca00700 1 --2- 172.21.15.12:0/2511046065 >> v2:172.21.15.159:3300/0 conn(0x7f585815d9d0 0x7f58580d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.622+0000 7f585ca00700 1 --2- 172.21.15.12:0/2511046065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58580d16a0 0x7f58580d1e30 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.622+0000 7f585ca00700 1 -- 172.21.15.12:0/2511046065 >> 172.21.15.12:0/2511046065 conn(0x7f585808aff0 msgr2=0x7f58580d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.622+0000 7f585ca00700 1 -- 172.21.15.12:0/2511046065 shutdown_connections 2024-03-29T09:56:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.622+0000 7f585ca00700 1 -- 172.21.15.12:0/2511046065 wait complete. 2024-03-29T09:56:03.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.624+0000 7f585ca00700 1 Processor -- start 2024-03-29T09:56:03.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.624+0000 7f585ca00700 1 -- start start 2024-03-29T09:56:03.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.625+0000 7f585ca00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58580d16a0 0x7f58580caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.625+0000 7f585ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58580d2370 0x7f58580b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.625+0000 7f585ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585815d9d0 0x7f58580b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.625+0000 7f585ca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5858155fa0 con 0x7f585815d9d0 2024-03-29T09:56:03.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.625+0000 7f585ca00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58580cc980 con 0x7f58580d16a0 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.625+0000 7f585ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58580ccc60 con 0x7f58580d2370 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.625+0000 7f5856800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585815d9d0 0x7f58580b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.625+0000 7f5855e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58580d16a0 0x7f58580caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.625+0000 7f5856800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585815d9d0 0x7f58580b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57606/0 (socket says 172.21.15.12:57606) 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.625+0000 7f5856800700 1 -- 172.21.15.12:0/1458649934 learned_addr learned my addr 172.21.15.12:0/1458649934 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f5855400700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58580d2370 0x7f58580b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f5856800700 1 -- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58580d2370 msgr2=0x7f58580b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f5856800700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58580d2370 0x7f58580b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f5856800700 1 -- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58580d16a0 msgr2=0x7f58580caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f5856800700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58580d16a0 0x7f58580caa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f5856800700 1 -- 172.21.15.12:0/1458649934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58440094f0 con 0x7f585815d9d0 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f5856800700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585815d9d0 0x7f58580b4270 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f585000cc50 tx=0x7f585000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f584e000700 1 -- 172.21.15.12:0/1458649934 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5850003c00 con 0x7f585815d9d0 2024-03-29T09:56:03.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f584e000700 1 -- 172.21.15.12:0/1458649934 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5850003d60 con 0x7f585815d9d0 2024-03-29T09:56:03.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f584e000700 1 -- 172.21.15.12:0/1458649934 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585000aab0 con 0x7f585815d9d0 2024-03-29T09:56:03.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.626+0000 7f585ca00700 1 -- 172.21.15.12:0/1458649934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58580ccf40 con 0x7f585815d9d0 2024-03-29T09:56:03.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.627+0000 7f585ca00700 1 -- 172.21.15.12:0/1458649934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58580cd500 con 0x7f585815d9d0 2024-03-29T09:56:03.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.628+0000 7f585ca00700 1 -- 172.21.15.12:0/1458649934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58580d1ca0 con 0x7f585815d9d0 2024-03-29T09:56:03.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.628+0000 7f584e000700 1 -- 172.21.15.12:0/1458649934 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 11) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f585000ac10 con 0x7f585815d9d0 2024-03-29T09:56:03.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.629+0000 7f584e000700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f583806af20 0x7f583806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:03.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.629+0000 7f584e000700 1 -- 172.21.15.12:0/1458649934 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5850089930 con 0x7f585815d9d0 2024-03-29T09:56:03.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.630+0000 7f5855e00700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f583806af20 0x7f583806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:03.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.630+0000 7f5855e00700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f583806af20 0x7f583806d3e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5848009b40 tx=0x7f5848009210 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:03.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.634+0000 7f584e000700 1 -- 172.21.15.12:0/1458649934 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5850054610 con 0x7f585815d9d0 2024-03-29T09:56:03.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.880+0000 7f585ca00700 1 -- 172.21.15.12:0/1458649934 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5858161fa0 con 0x7f583806af20 2024-03-29T09:56:03.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.941+0000 7f584e000700 1 -- 172.21.15.12:0/1458649934 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5848011810 con 0x7f583806af20 2024-03-29T09:56:03.941 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/dir7 2024-03-29T09:56:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 -- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f583806af20 msgr2=0x7f583806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f583806af20 0x7f583806d3e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5848009b40 tx=0x7f5848009210 comp rx=0 tx=0).stop 2024-03-29T09:56:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 -- 172.21.15.12:0/1458649934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585815d9d0 msgr2=0x7f58580b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585815d9d0 0x7f58580b4270 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f585000cc50 tx=0x7f585000cf60 comp rx=0 tx=0).stop 2024-03-29T09:56:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 -- 172.21.15.12:0/1458649934 shutdown_connections 2024-03-29T09:56:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f583806af20 0x7f583806d3e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585815d9d0 0x7f58580b4270 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58580d16a0 0x7f58580caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 --2- 172.21.15.12:0/1458649934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58580d2370 0x7f58580b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 -- 172.21.15.12:0/1458649934 >> 172.21.15.12:0/1458649934 conn(0x7f585808aff0 msgr2=0x7f585815be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 -- 172.21.15.12:0/1458649934 shutdown_connections 2024-03-29T09:56:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:03.944+0000 7f585ca00700 1 -- 172.21.15.12:0/1458649934 wait complete. 2024-03-29T09:56:03.961 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2024-03-29T09:56:03.961 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2024-03-29T09:56:04.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.163+0000 7fdefdc00700 1 Processor -- start 2024-03-29T09:56:04.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.163+0000 7fdefdc00700 1 -- start start 2024-03-29T09:56:04.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.164+0000 7fdefdc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdef80cbb30 0x7fdef80cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.164+0000 7fdefdc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdef80cd020 0x7fdef80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.164+0000 7fdefdc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdef80b2070 0x7fdef80b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.164+0000 7fdefdc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef80cc450 con 0x7fdef80cbb30 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.164+0000 7fdefdc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef80cdae0 con 0x7fdef80cd020 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.164+0000 7fdefdc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80cddc0 con 0x7fdef80b2070 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.164+0000 7fdef6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdef80cbb30 0x7fdef80cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdef80cbb30 0x7fdef80cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57622/0 (socket says 172.21.15.12:57622) 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef6a00700 1 -- 172.21.15.12:0/4111211073 learned_addr learned my addr 172.21.15.12:0/4111211073 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef6000700 1 --2- 172.21.15.12:0/4111211073 >> v2:172.21.15.159:3300/0 conn(0x7fdef80cd020 0x7fdef80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef7400700 1 --2- 172.21.15.12:0/4111211073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdef80b2070 0x7fdef80b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef6a00700 1 -- 172.21.15.12:0/4111211073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdef80b2070 msgr2=0x7fdef80b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef6a00700 1 --2- 172.21.15.12:0/4111211073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdef80b2070 0x7fdef80b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef6a00700 1 -- 172.21.15.12:0/4111211073 >> v2:172.21.15.159:3300/0 conn(0x7fdef80cd020 msgr2=0x7fdef80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:04.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef6a00700 1 --2- 172.21.15.12:0/4111211073 >> v2:172.21.15.159:3300/0 conn(0x7fdef80cd020 0x7fdef80cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef6a00700 1 -- 172.21.15.12:0/4111211073 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef80ce0a0 con 0x7fdef80cbb30 2024-03-29T09:56:04.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef6000700 1 --2- 172.21.15.12:0/4111211073 >> v2:172.21.15.159:3300/0 conn(0x7fdef80cd020 0x7fdef80cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:56:04.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.165+0000 7fdef6a00700 1 --2- 172.21.15.12:0/4111211073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80cbb30 0x7fdef80cbf10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdef80b1e30 tx=0x7fdee8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549ec21d333a926d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:04.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdef5600700 1 -- 172.21.15.12:0/4111211073 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee800b870 con 0x7fdef80cbb30 2024-03-29T09:56:04.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdef5600700 1 -- 172.21.15.12:0/4111211073 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee800b9d0 con 0x7fdef80cbb30 2024-03-29T09:56:04.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdef5600700 1 -- 172.21.15.12:0/4111211073 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee800bd60 con 0x7fdef80cbb30 2024-03-29T09:56:04.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdefdc00700 1 -- 172.21.15.12:0/4111211073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80cbb30 msgr2=0x7fdef80cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:04.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdefdc00700 1 --2- 172.21.15.12:0/4111211073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80cbb30 0x7fdef80cbf10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdef80b1e30 tx=0x7fdee8009bd0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdefdc00700 1 -- 172.21.15.12:0/4111211073 shutdown_connections 2024-03-29T09:56:04.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdefdc00700 1 --2- 172.21.15.12:0/4111211073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdef80b2070 0x7fdef80b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdefdc00700 1 --2- 172.21.15.12:0/4111211073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80cbb30 0x7fdef80cbf10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdefdc00700 1 --2- 172.21.15.12:0/4111211073 >> v2:172.21.15.159:3300/0 conn(0x7fdef80cd020 0x7fdef80cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdefdc00700 1 -- 172.21.15.12:0/4111211073 >> 172.21.15.12:0/4111211073 conn(0x7fdef80b0d70 msgr2=0x7fdef80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:04.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdefdc00700 1 -- 172.21.15.12:0/4111211073 shutdown_connections 2024-03-29T09:56:04.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.166+0000 7fdefdc00700 1 -- 172.21.15.12:0/4111211073 wait complete. 2024-03-29T09:56:04.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.168+0000 7fdefdc00700 1 Processor -- start 2024-03-29T09:56:04.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.168+0000 7fdefdc00700 1 -- start start 2024-03-29T09:56:04.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.169+0000 7fdefdc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80b2070 0x7fdef8158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.169+0000 7fdefdc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdef80cbb30 0x7fdef80ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.169+0000 7fdefdc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdef80cd020 0x7fdef81579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.169+0000 7fdefdc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80adc00 con 0x7fdef80b2070 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.169+0000 7fdefdc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80ae590 con 0x7fdef80cbb30 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.169+0000 7fdefdc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80acbe0 con 0x7fdef80cd020 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.169+0000 7fdef6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80b2070 0x7fdef8158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.169+0000 7fdef6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80b2070 0x7fdef8158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57626/0 (socket says 172.21.15.12:57626) 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdef6a00700 1 -- 172.21.15.12:0/3411363520 learned_addr learned my addr 172.21.15.12:0/3411363520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdef7400700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdef80cd020 0x7fdef81579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdef6000700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdef80cbb30 0x7fdef80ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdef6a00700 1 -- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdef80cd020 msgr2=0x7fdef81579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdef6a00700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdef80cd020 0x7fdef81579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdef6a00700 1 -- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdef80cbb30 msgr2=0x7fdef80ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdef6a00700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdef80cbb30 0x7fdef80ad6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdef6a00700 1 -- 172.21.15.12:0/3411363520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee8009550 con 0x7fdef80b2070 2024-03-29T09:56:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdef6a00700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80b2070 0x7fdef8158200 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fdee8000c00 tx=0x7fdee8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdeeea00700 1 -- 172.21.15.12:0/3411363520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8004520 con 0x7fdef80b2070 2024-03-29T09:56:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdeeea00700 1 -- 172.21.15.12:0/3411363520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee801e070 con 0x7fdef80b2070 2024-03-29T09:56:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdeeea00700 1 -- 172.21.15.12:0/3411363520 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee80038c0 con 0x7fdef80b2070 2024-03-29T09:56:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.170+0000 7fdefdc00700 1 -- 172.21.15.12:0/3411363520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef80acec0 con 0x7fdef80b2070 2024-03-29T09:56:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.171+0000 7fdefdc00700 1 -- 172.21.15.12:0/3411363520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef81589e0 con 0x7fdef80b2070 2024-03-29T09:56:04.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.172+0000 7fdefdc00700 1 -- 172.21.15.12:0/3411363520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef8067720 con 0x7fdef80b2070 2024-03-29T09:56:04.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.172+0000 7fdeeea00700 1 -- 172.21.15.12:0/3411363520 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 11) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fdee8003a20 con 0x7fdef80b2070 2024-03-29T09:56:04.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.173+0000 7fdeeea00700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fded806af20 0x7fded806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.173+0000 7fdeeea00700 1 -- 172.21.15.12:0/3411363520 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdee8020070 con 0x7fdef80b2070 2024-03-29T09:56:04.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.174+0000 7fdef6000700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fded806af20 0x7fded806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.174+0000 7fdef6000700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fded806af20 0x7fded806d3e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdef815ccb0 tx=0x7fdee4008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:04.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.178+0000 7fdeeea00700 1 -- 172.21.15.12:0/3411363520 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdee8055c50 con 0x7fdef80b2070 2024-03-29T09:56:04.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.426+0000 7fdefdc00700 1 -- 172.21.15.12:0/3411363520 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdef80aca60 con 0x7fded806af20 2024-03-29T09:56:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.488+0000 7fdeeea00700 1 -- 172.21.15.12:0/3411363520 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fdee400c050 con 0x7fded806af20 2024-03-29T09:56:04.488 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/dir8 2024-03-29T09:56:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 -- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fded806af20 msgr2=0x7fded806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fded806af20 0x7fded806d3e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdef815ccb0 tx=0x7fdee4008040 comp rx=0 tx=0).stop 2024-03-29T09:56:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 -- 172.21.15.12:0/3411363520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80b2070 msgr2=0x7fdef8158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80b2070 0x7fdef8158200 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fdee8000c00 tx=0x7fdee8004750 comp rx=0 tx=0).stop 2024-03-29T09:56:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 -- 172.21.15.12:0/3411363520 shutdown_connections 2024-03-29T09:56:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fded806af20 0x7fded806d3e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdef80cd020 0x7fdef81579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdef80b2070 0x7fdef8158200 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 --2- 172.21.15.12:0/3411363520 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdef80cbb30 0x7fdef80ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 -- 172.21.15.12:0/3411363520 >> 172.21.15.12:0/3411363520 conn(0x7fdef80b0d70 msgr2=0x7fdef806e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 -- 172.21.15.12:0/3411363520 shutdown_connections 2024-03-29T09:56:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.491+0000 7fdefdc00700 1 -- 172.21.15.12:0/3411363520 wait complete. 2024-03-29T09:56:04.508 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2024-03-29T09:56:04.508 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2024-03-29T09:56:04.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.709+0000 7f3fc1800700 1 Processor -- start 2024-03-29T09:56:04.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.710+0000 7f3fc1800700 1 -- start start 2024-03-29T09:56:04.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.711+0000 7f3fc1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc0ced80 0x7f3fbc0d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.711+0000 7f3fc1800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0cf1e0 con 0x7f3fbc15dad0 2024-03-29T09:56:04.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.711+0000 7f3fc1800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc15f820 con 0x7f3fbc15fcb0 2024-03-29T09:56:04.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.711+0000 7f3fc1800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0cfcc0 con 0x7f3fbc0ced80 2024-03-29T09:56:04.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.711+0000 7f3fbaa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3fbc15dad0 0x7f3fbc0d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33622/0 (socket says 172.21.15.12:33622) 2024-03-29T09:56:04.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.711+0000 7f3fbaa00700 1 -- 172.21.15.12:0/1612423570 learned_addr learned my addr 172.21.15.12:0/1612423570 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.711+0000 7f3fba000700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc0ced80 0x7f3fbc0d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.711+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633767465 0 0) 0x7f3fac001110 con 0x7f3fbc15dad0 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.711+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa00035b0 con 0x7f3fbc15dad0 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2579049773 0 0) 0x7f3fa00035b0 con 0x7f3fbc15dad0 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc0ced80 msgr2=0x7f3fbc0d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc0ced80 0x7f3fbc0d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 >> v1:172.21.15.159:6789/0 conn(0x7f3fbc15fcb0 legacy=0x7f3fbc160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc0cf420 con 0x7f3fbc15dad0 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f3fac001800 con 0x7f3fbc15dad0 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 >> v1:172.21.15.12:6789/0 conn(0x7f3fbc15dad0 legacy=0x7f3fbc0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fa0003a30 0x7f3fa0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 0x7f3fa0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc0ced80 0x7f3fa0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0cfcc0 con 0x7f3fa0004850 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa000baf0 con 0x7f3fa0003a30 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa000bcb0 con 0x7f3fbc0ced80 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fba000700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 0x7f3fa0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fbaa00700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fa0003a30 0x7f3fa0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.712+0000 7f3fbb400700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc0ced80 0x7f3fa0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fba000700 1 -- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc0ced80 msgr2=0x7f3fa0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fba000700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc0ced80 0x7f3fa0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fba000700 1 -- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fa0003a30 msgr2=0x7f3fa0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fba000700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fa0003a30 0x7f3fa0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fba000700 1 -- 172.21.15.12:0/1612423570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa00035b0 con 0x7f3fa0004850 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fbb400700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc0ced80 0x7f3fa0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:56:04.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fbaa00700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fa0003a30 0x7f3fa0003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fba000700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 0x7f3fa0004cd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8009b30 tx=0x7f3fa0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e337e7c549e0f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa800e6c0 con 0x7f3fa0004850 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa800e820 con 0x7f3fa0004850 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.713+0000 7f3fb9600700 1 -- 172.21.15.12:0/1612423570 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa800eb90 con 0x7f3fa0004850 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.714+0000 7f3fc1800700 1 -- 172.21.15.12:0/1612423570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 msgr2=0x7f3fa0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.714+0000 7f3fc1800700 1 --2- 172.21.15.12:0/1612423570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 0x7f3fa0004cd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8009b30 tx=0x7f3fa0005880 comp rx=0 tx=0).stop 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.714+0000 7f3fbb400700 1 -- 172.21.15.12:0/1612423570 reap_dead start 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.714+0000 7f3fc1800700 1 -- 172.21.15.12:0/1612423570 shutdown_connections 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.714+0000 7f3fc1800700 1 -- 172.21.15.12:0/1612423570 >> 172.21.15.12:0/1612423570 conn(0x7f3fbc08aff0 msgr2=0x7f3fbc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.714+0000 7f3fc1800700 1 -- 172.21.15.12:0/1612423570 shutdown_connections 2024-03-29T09:56:04.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.714+0000 7f3fc1800700 1 -- 172.21.15.12:0/1612423570 wait complete. 2024-03-29T09:56:04.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.716+0000 7f3fc1800700 1 Processor -- start 2024-03-29T09:56:04.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.716+0000 7f3fc1800700 1 -- start start 2024-03-29T09:56:04.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fc1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 0x7f3fbc14eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fc1800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fbc0cc290 0x7f3fbc0b74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fc1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc15bd80 0x7f3fbc15c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fc1800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc15c740 con 0x7f3fa0004850 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fc1800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc1f6d00 con 0x7f3fbc0cc290 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fc1800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0b7e70 con 0x7f3fbc15bd80 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fbaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 0x7f3fbc14eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fbaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 0x7f3fbc14eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57646/0 (socket says 172.21.15.12:57646) 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fbaa00700 1 -- 172.21.15.12:0/3997826392 learned_addr learned my addr 172.21.15.12:0/3997826392 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fbb400700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc15bd80 0x7f3fbc15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fba000700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fbc0cc290 0x7f3fbc0b74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fbaa00700 1 -- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc15bd80 msgr2=0x7f3fbc15c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fbaa00700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc15bd80 0x7f3fbc15c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fbaa00700 1 -- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fbc0cc290 msgr2=0x7f3fbc0b74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fbaa00700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fbc0cc290 0x7f3fbc0b74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.717+0000 7f3fbaa00700 1 -- 172.21.15.12:0/3997826392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc1f6f50 con 0x7f3fa0004850 2024-03-29T09:56:04.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.718+0000 7f3fbaa00700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 0x7f3fbc14eb70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3fac001390 tx=0x7f3fac007690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:04.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.718+0000 7f3fb2a00700 1 -- 172.21.15.12:0/3997826392 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac0077b0 con 0x7f3fa0004850 2024-03-29T09:56:04.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.718+0000 7f3fb2a00700 1 -- 172.21.15.12:0/3997826392 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac00e860 con 0x7f3fa0004850 2024-03-29T09:56:04.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.718+0000 7f3fb2a00700 1 -- 172.21.15.12:0/3997826392 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac00ebd0 con 0x7f3fa0004850 2024-03-29T09:56:04.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.718+0000 7f3fc1800700 1 -- 172.21.15.12:0/3997826392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc1f7230 con 0x7f3fa0004850 2024-03-29T09:56:04.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.718+0000 7f3fc1800700 1 -- 172.21.15.12:0/3997826392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc14f0b0 con 0x7f3fa0004850 2024-03-29T09:56:04.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.719+0000 7f3fc1800700 1 -- 172.21.15.12:0/3997826392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc0ca440 con 0x7f3fa0004850 2024-03-29T09:56:04.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.720+0000 7f3fb2a00700 1 -- 172.21.15.12:0/3997826392 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 11) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f3fac00ed30 con 0x7f3fa0004850 2024-03-29T09:56:04.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.721+0000 7f3fb2a00700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f9c06af50 0x7f3f9c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T09:56:04.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.721+0000 7f3fb2a00700 1 -- 172.21.15.12:0/3997826392 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f3fac089750 con 0x7f3fa0004850 2024-03-29T09:56:04.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.721+0000 7f3fba000700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f9c06af50 0x7f3f9c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T09:56:04.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.722+0000 7f3fba000700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f9c06af50 0x7f3f9c06d410 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3fa80090b0 tx=0x7f3fa8008030 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T09:56:04.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.725+0000 7f3fb2a00700 1 -- 172.21.15.12:0/3997826392 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3fac057f10 con 0x7f3fa0004850 2024-03-29T09:56:04.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:04.973+0000 7f3fc1800700 1 -- 172.21.15.12:0/3997826392 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fbc0cf9e0 con 0x7f3f9c06af50 2024-03-29T09:56:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.034+0000 7f3fb2a00700 1 -- 172.21.15.12:0/3997826392 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3fa800e990 con 0x7f3f9c06af50 2024-03-29T09:56:05.034 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/dir9 2024-03-29T09:56:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 -- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f9c06af50 msgr2=0x7f3f9c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f9c06af50 0x7f3f9c06d410 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3fa80090b0 tx=0x7f3fa8008030 comp rx=0 tx=0).stop 2024-03-29T09:56:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 -- 172.21.15.12:0/3997826392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 msgr2=0x7f3fbc14eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T09:56:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 0x7f3fbc14eb70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3fac001390 tx=0x7f3fac007690 comp rx=0 tx=0).stop 2024-03-29T09:56:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 -- 172.21.15.12:0/3997826392 shutdown_connections 2024-03-29T09:56:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f9c06af50 0x7f3f9c06d410 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3fbc15bd80 0x7f3fbc15c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3fa0004850 0x7f3fbc14eb70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 --2- 172.21.15.12:0/3997826392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3fbc0cc290 0x7f3fbc0b74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T09:56:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 -- 172.21.15.12:0/3997826392 >> 172.21.15.12:0/3997826392 conn(0x7f3fbc08aff0 msgr2=0x7f3fbc160b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T09:56:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 -- 172.21.15.12:0/3997826392 shutdown_connections 2024-03-29T09:56:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T09:56:05.036+0000 7f3fc1800700 1 -- 172.21.15.12:0/3997826392 wait complete. 2024-03-29T09:56:05.054 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2024-03-29T09:56:23.014 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:56:23.019 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:56:53.215 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:56:53.220 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:57:23.328 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:57:23.333 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:57:53.615 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:57:53.620 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:58:23.721 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:58:23.727 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:58:53.860 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:58:53.897 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:59:24.011 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:59:24.016 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:59:54.128 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T09:59:54.134 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:00:00.559 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:00:00.559+0000 7fe206200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:00:00.559 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:00:00.559+0000 7fe206200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:00:00.559 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:00:00.559+0000 7fe206200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:00:00.559 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:00:00.559+0000 7fe206200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:00:00.559 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:00:00.559+0000 7fe206200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:00:24.242 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:00:24.247 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:00:54.364 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:00:54.368 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:01:24.477 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:01:24.482 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:01:54.591 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:01:54.596 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:02:24.705 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:02:24.710 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:02:54.835 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:02:54.840 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:03:24.949 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:03:24.954 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:03:55.061 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:03:55.067 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:04:25.174 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:04:25.180 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:04:55.323 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:04:55.328 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:05:25.439 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:05:25.445 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:05:55.567 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:05:55.572 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:06:21.055 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2024-03-29T10:06:21.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.268+0000 7f4aa0a00700 1 Processor -- start 2024-03-29T10:06:21.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.268+0000 7f4aa0a00700 1 -- start start 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4aa0a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4a9c0ce500 0x7f4a9c0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4aa0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0cc370 0x7f4a9c0ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4aa0a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4a9c0b3760 0x7f4a9c0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4aa0a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a9c163190 con 0x7f4a9c0b3760 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4aa0a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a9c0cee20 con 0x7f4a9c0ce500 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4aa0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c162580 con 0x7f4a9c0cc370 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4a9a800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4a9c0b3760 0x7f4a9c0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4a9a800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4a9c0b3760 0x7f4a9c0b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32958/0 (socket says 172.21.15.12:32958) 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4a9a800700 1 -- 172.21.15.12:0/2632252917 learned_addr learned my addr 172.21.15.12:0/2632252917 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4a99400700 1 --2- 172.21.15.12:0/2632252917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0cc370 0x7f4a9c0ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.270+0000 7f4a99e00700 1 --2- 172.21.15.12:0/2632252917 >> v2:172.21.15.159:3300/0 conn(0x7f4a9c0ce500 0x7f4a9c0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.271+0000 7f4a9a800700 1 -- 172.21.15.12:0/2632252917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0cc370 msgr2=0x7f4a9c0ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:21.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.271+0000 7f4a9a800700 1 --2- 172.21.15.12:0/2632252917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0cc370 0x7f4a9c0ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.271+0000 7f4a9a800700 1 -- 172.21.15.12:0/2632252917 >> v2:172.21.15.159:3300/0 conn(0x7f4a9c0ce500 msgr2=0x7f4a9c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:21.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.271+0000 7f4a9a800700 1 --2- 172.21.15.12:0/2632252917 >> v2:172.21.15.159:3300/0 conn(0x7f4a9c0ce500 0x7f4a9c0b2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.271+0000 7f4a9a800700 1 -- 172.21.15.12:0/2632252917 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c162860 con 0x7f4a9c0b3760 2024-03-29T10:06:21.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.271+0000 7f4a99e00700 1 --2- 172.21.15.12:0/2632252917 >> v2:172.21.15.159:3300/0 conn(0x7f4a9c0ce500 0x7f4a9c0b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:21.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.271+0000 7f4a9a800700 1 --2- 172.21.15.12:0/2632252917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 0x7f4a9c0b3bd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0b1db0 tx=0x7f4a9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1440ee6614c371e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:21.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.271+0000 7f4a98a00700 1 -- 172.21.15.12:0/2632252917 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9400e070 con 0x7f4a9c0b3760 2024-03-29T10:06:21.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.271+0000 7f4a98a00700 1 -- 172.21.15.12:0/2632252917 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9400b910 con 0x7f4a9c0b3760 2024-03-29T10:06:21.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.271+0000 7f4a98a00700 1 -- 172.21.15.12:0/2632252917 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9400bcc0 con 0x7f4a9c0b3760 2024-03-29T10:06:21.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.272+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2632252917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 msgr2=0x7f4a9c0b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:21.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.272+0000 7f4aa0a00700 1 --2- 172.21.15.12:0/2632252917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 0x7f4a9c0b3bd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0b1db0 tx=0x7f4a9400b330 comp rx=0 tx=0).stop 2024-03-29T10:06:21.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.272+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2632252917 shutdown_connections 2024-03-29T10:06:21.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.272+0000 7f4aa0a00700 1 --2- 172.21.15.12:0/2632252917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 0x7f4a9c0b3bd0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.272+0000 7f4aa0a00700 1 --2- 172.21.15.12:0/2632252917 >> v2:172.21.15.159:3300/0 conn(0x7f4a9c0ce500 0x7f4a9c0b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.272+0000 7f4aa0a00700 1 --2- 172.21.15.12:0/2632252917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0cc370 0x7f4a9c0ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.272+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2632252917 >> 172.21.15.12:0/2632252917 conn(0x7f4a9c0b0d70 msgr2=0x7f4a9c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:21.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.272+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2632252917 shutdown_connections 2024-03-29T10:06:21.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.272+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2632252917 wait complete. 2024-03-29T10:06:21.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.274+0000 7f4aa0a00700 1 Processor -- start 2024-03-29T10:06:21.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.274+0000 7f4aa0a00700 1 -- start start 2024-03-29T10:06:21.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4aa0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 0x7f4a9c098020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4aa0a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c0cc370 0x7f4a9c09c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4aa0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0ce500 0x7f4a9c1632d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4aa0a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0adae0 con 0x7f4a9c0b3760 2024-03-29T10:06:21.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4aa0a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0aced0 con 0x7f4a9c0cc370 2024-03-29T10:06:21.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4aa0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0ad1b0 con 0x7f4a9c0ce500 2024-03-29T10:06:21.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4a99e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 0x7f4a9c098020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4a99e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 0x7f4a9c098020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32970/0 (socket says 172.21.15.12:32970) 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4a99e00700 1 -- 172.21.15.12:0/2393416110 learned_addr learned my addr 172.21.15.12:0/2393416110 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4a9a800700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0ce500 0x7f4a9c1632d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.275+0000 7f4a99400700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c0cc370 0x7f4a9c09c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4a99e00700 1 -- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0ce500 msgr2=0x7f4a9c1632d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4a99e00700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0ce500 0x7f4a9c1632d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4a99e00700 1 -- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c0cc370 msgr2=0x7f4a9c09c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4a99e00700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c0cc370 0x7f4a9c09c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4a99e00700 1 -- 172.21.15.12:0/2393416110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a94009b30 con 0x7f4a9c0b3760 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4a99400700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c0cc370 0x7f4a9c09c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4a99e00700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 0x7f4a9c098020 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00e810 tx=0x7f4a8c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4a92000700 1 -- 172.21.15.12:0/2393416110 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c00c020 con 0x7f4a9c0b3760 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4a92000700 1 -- 172.21.15.12:0/2393416110 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8c004250 con 0x7f4a9c0b3760 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2393416110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c0ad490 con 0x7f4a9c0b3760 2024-03-29T10:06:21.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4a92000700 1 -- 172.21.15.12:0/2393416110 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c0036a0 con 0x7f4a9c0b3760 2024-03-29T10:06:21.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.276+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2393416110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c098560 con 0x7f4a9c0b3760 2024-03-29T10:06:21.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.277+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2393416110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c067720 con 0x7f4a9c0b3760 2024-03-29T10:06:21.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.278+0000 7f4a92000700 1 -- 172.21.15.12:0/2393416110 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f4a8c003800 con 0x7f4a9c0b3760 2024-03-29T10:06:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.279+0000 7f4a92000700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a7c06ae90 0x7f4a7c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.279+0000 7f4a92000700 1 -- 172.21.15.12:0/2393416110 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f4a8c089330 con 0x7f4a9c0b3760 2024-03-29T10:06:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.280+0000 7f4a99400700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a7c06ae90 0x7f4a7c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.281+0000 7f4a99400700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a7c06ae90 0x7f4a7c06d350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0b08a0 tx=0x7f4a88008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:21.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.284+0000 7f4a92000700 1 -- 172.21.15.12:0/2393416110 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4a8c053fe0 con 0x7f4a9c0b3760 2024-03-29T10:06:21.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.524+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2393416110 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a9c162e20 con 0x7f4a7c06ae90 2024-03-29T10:06:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.610+0000 7f4a92000700 1 -- 172.21.15.12:0/2393416110 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f4a880098e0 con 0x7f4a7c06ae90 2024-03-29T10:06:21.610 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a7c06ae90 msgr2=0x7f4a7c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a7c06ae90 0x7f4a7c06d350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0b08a0 tx=0x7f4a88008040 comp rx=0 tx=0).stop 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2393416110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 msgr2=0x7f4a9c098020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 0x7f4a9c098020 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00e810 tx=0x7f4a8c00eb20 comp rx=0 tx=0).stop 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2393416110 shutdown_connections 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a7c06ae90 0x7f4a7c06d350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0ce500 0x7f4a9c1632d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0b3760 0x7f4a9c098020 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 --2- 172.21.15.12:0/2393416110 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c0cc370 0x7f4a9c09c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2393416110 >> 172.21.15.12:0/2393416110 conn(0x7f4a9c0b0d70 msgr2=0x7f4a9c0b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2393416110 shutdown_connections 2024-03-29T10:06:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.613+0000 7f4aa0a00700 1 -- 172.21.15.12:0/2393416110 wait complete. 2024-03-29T10:06:21.631 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2024-03-29T10:06:21.631 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2024-03-29T10:06:21.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.833+0000 7f8d3e000700 1 Processor -- start 2024-03-29T10:06:21.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.833+0000 7f8d3e000700 1 -- start start 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.834+0000 7f8d3e000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8d380cd020 0x7f8d380cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.834+0000 7f8d3e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8d380b2070 0x7f8d380b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.834+0000 7f8d3e000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d380cdae0 con 0x7f8d380cbb30 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.834+0000 7f8d3e000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d380cddc0 con 0x7f8d380b2070 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.834+0000 7f8d3e000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d380ce0a0 con 0x7f8d380cd020 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.834+0000 7f8d37e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8d380b2070 0x7f8d380b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.834+0000 7f8d37e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8d380b2070 0x7f8d380b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32984/0 (socket says 172.21.15.12:32984) 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.834+0000 7f8d37e00700 1 -- 172.21.15.12:0/3799458511 learned_addr learned my addr 172.21.15.12:0/3799458511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.835+0000 7f8d37e00700 1 -- 172.21.15.12:0/3799458511 >> v2:172.21.15.159:3300/0 conn(0x7f8d380cd020 msgr2=0x7f8d380cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.835+0000 7f8d36a00700 1 --2- 172.21.15.12:0/3799458511 >> v2:172.21.15.159:3300/0 conn(0x7f8d380cd020 0x7f8d380cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.835+0000 7f8d37e00700 1 --2- 172.21.15.12:0/3799458511 >> v2:172.21.15.159:3300/0 conn(0x7f8d380cd020 0x7f8d380cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.835+0000 7f8d37e00700 1 -- 172.21.15.12:0/3799458511 >> v1:172.21.15.159:6789/0 conn(0x7f8d380cbb30 legacy=0x7f8d380cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:21.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.835+0000 7f8d37e00700 1 -- 172.21.15.12:0/3799458511 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3806dc20 con 0x7f8d380b2070 2024-03-29T10:06:21.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.835+0000 7f8d36a00700 1 --2- 172.21.15.12:0/3799458511 >> v2:172.21.15.159:3300/0 conn(0x7f8d380cd020 0x7f8d380cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:21.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.835+0000 7f8d37e00700 1 --2- 172.21.15.12:0/3799458511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380b2070 0x7f8d380b2500 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8d380b1e30 tx=0x7f8d3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c6d144fe919391 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:21.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.835+0000 7f8d36000700 1 -- 172.21.15.12:0/3799458511 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3000e070 con 0x7f8d380b2070 2024-03-29T10:06:21.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.835+0000 7f8d36000700 1 -- 172.21.15.12:0/3799458511 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d30003a20 con 0x7f8d380b2070 2024-03-29T10:06:21.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.835+0000 7f8d36000700 1 -- 172.21.15.12:0/3799458511 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30003dd0 con 0x7f8d380b2070 2024-03-29T10:06:21.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.836+0000 7f8d3e000700 1 -- 172.21.15.12:0/3799458511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380b2070 msgr2=0x7f8d380b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:21.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.836+0000 7f8d3e000700 1 --2- 172.21.15.12:0/3799458511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380b2070 0x7f8d380b2500 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8d380b1e30 tx=0x7f8d3000b330 comp rx=0 tx=0).stop 2024-03-29T10:06:21.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.836+0000 7f8d3e000700 1 -- 172.21.15.12:0/3799458511 shutdown_connections 2024-03-29T10:06:21.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.836+0000 7f8d3e000700 1 --2- 172.21.15.12:0/3799458511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380b2070 0x7f8d380b2500 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.836+0000 7f8d3e000700 1 --2- 172.21.15.12:0/3799458511 >> v2:172.21.15.159:3300/0 conn(0x7f8d380cd020 0x7f8d380cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.836+0000 7f8d3e000700 1 -- 172.21.15.12:0/3799458511 >> 172.21.15.12:0/3799458511 conn(0x7f8d380b0d70 msgr2=0x7f8d380b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:21.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.836+0000 7f8d3e000700 1 -- 172.21.15.12:0/3799458511 shutdown_connections 2024-03-29T10:06:21.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.836+0000 7f8d3e000700 1 -- 172.21.15.12:0/3799458511 wait complete. 2024-03-29T10:06:21.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.838+0000 7f8d3e000700 1 Processor -- start 2024-03-29T10:06:21.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.838+0000 7f8d3e000700 1 -- start start 2024-03-29T10:06:21.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d3e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d380b2070 0x7f8d381f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d3e000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d380cbb30 0x7f8d381521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d3e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380cd020 0x7f8d3815dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d3e000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d38153820 con 0x7f8d380cd020 2024-03-29T10:06:21.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d3e000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d381f3220 con 0x7f8d380cbb30 2024-03-29T10:06:21.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d3e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d38152c10 con 0x7f8d380b2070 2024-03-29T10:06:21.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d37e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380cd020 0x7f8d3815dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d37e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380cd020 0x7f8d3815dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32986/0 (socket says 172.21.15.12:32986) 2024-03-29T10:06:21.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d37e00700 1 -- 172.21.15.12:0/2226374399 learned_addr learned my addr 172.21.15.12:0/2226374399 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:21.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d36a00700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d380cbb30 0x7f8d381521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d37400700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d380b2070 0x7f8d381f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d37e00700 1 -- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d380b2070 msgr2=0x7f8d381f2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:21.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d37e00700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d380b2070 0x7f8d381f2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d37e00700 1 -- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d380cbb30 msgr2=0x7f8d381521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:21.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d37e00700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d380cbb30 0x7f8d381521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:21.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d37e00700 1 -- 172.21.15.12:0/2226374399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d30009b30 con 0x7f8d380cd020 2024-03-29T10:06:21.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d36a00700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d380cbb30 0x7f8d381521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:21.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.839+0000 7f8d37400700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d380b2070 0x7f8d381f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:21.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.840+0000 7f8d37e00700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380cd020 0x7f8d3815dd30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000b370 tx=0x7f8d30004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:21.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.840+0000 7f8d2f400700 1 -- 172.21.15.12:0/2226374399 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30004500 con 0x7f8d380cd020 2024-03-29T10:06:21.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.840+0000 7f8d2f400700 1 -- 172.21.15.12:0/2226374399 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d30004680 con 0x7f8d380cd020 2024-03-29T10:06:21.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.840+0000 7f8d3e000700 1 -- 172.21.15.12:0/2226374399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d38152ef0 con 0x7f8d380cd020 2024-03-29T10:06:21.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.840+0000 7f8d2f400700 1 -- 172.21.15.12:0/2226374399 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30012430 con 0x7f8d380cd020 2024-03-29T10:06:21.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.840+0000 7f8d3e000700 1 -- 172.21.15.12:0/2226374399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d381534b0 con 0x7f8d380cd020 2024-03-29T10:06:21.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.841+0000 7f8d3e000700 1 -- 172.21.15.12:0/2226374399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d381539d0 con 0x7f8d380cd020 2024-03-29T10:06:21.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.842+0000 7f8d2f400700 1 -- 172.21.15.12:0/2226374399 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f8d30012590 con 0x7f8d380cd020 2024-03-29T10:06:21.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.843+0000 7f8d2f400700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d1806ae90 0x7f8d1806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.843+0000 7f8d2f400700 1 -- 172.21.15.12:0/2226374399 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f8d3008b440 con 0x7f8d380cd020 2024-03-29T10:06:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.843+0000 7f8d37400700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d1806ae90 0x7f8d1806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.844+0000 7f8d37400700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d1806ae90 0x7f8d1806d350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d28003fc0 tx=0x7f8d28009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:21.848+0000 7f8d2f400700 1 -- 172.21.15.12:0/2226374399 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d381539d0 con 0x7f8d380cd020 2024-03-29T10:06:22.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.091+0000 7f8d3e000700 1 -- 172.21.15.12:0/2226374399 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d3815bdd0 con 0x7f8d1806ae90 2024-03-29T10:06:22.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.181+0000 7f8d2f400700 1 -- 172.21.15.12:0/2226374399 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f8d28003bb0 con 0x7f8d1806ae90 2024-03-29T10:06:22.181 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2024-03-29T10:06:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 -- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d1806ae90 msgr2=0x7f8d1806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d1806ae90 0x7f8d1806d350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d28003fc0 tx=0x7f8d28009040 comp rx=0 tx=0).stop 2024-03-29T10:06:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 -- 172.21.15.12:0/2226374399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380cd020 msgr2=0x7f8d3815dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380cd020 0x7f8d3815dd30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000b370 tx=0x7f8d30004ad0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 -- 172.21.15.12:0/2226374399 shutdown_connections 2024-03-29T10:06:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d1806ae90 0x7f8d1806d350 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d380cd020 0x7f8d3815dd30 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d380b2070 0x7f8d381f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 --2- 172.21.15.12:0/2226374399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d380cbb30 0x7f8d381521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 -- 172.21.15.12:0/2226374399 >> 172.21.15.12:0/2226374399 conn(0x7f8d380b0d70 msgr2=0x7f8d380cbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 -- 172.21.15.12:0/2226374399 shutdown_connections 2024-03-29T10:06:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.184+0000 7f8d3e000700 1 -- 172.21.15.12:0/2226374399 wait complete. 2024-03-29T10:06:22.202 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2024-03-29T10:06:22.202 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2024-03-29T10:06:22.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.403+0000 7f35cd800700 1 Processor -- start 2024-03-29T10:06:22.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.403+0000 7f35cd800700 1 -- start start 2024-03-29T10:06:22.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.405+0000 7f35cd800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f35c8099a60 0x7f35c80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.405+0000 7f35cd800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c80d0970 con 0x7f35c815d990 2024-03-29T10:06:22.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.405+0000 7f35cd800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c806c020 con 0x7f35c809a750 2024-03-29T10:06:22.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.405+0000 7f35cd800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c80c81a0 con 0x7f35c8099a60 2024-03-29T10:06:22.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.405+0000 7f35c6000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f35c815d990 0x7f35c8099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52632/0 (socket says 172.21.15.12:52632) 2024-03-29T10:06:22.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.405+0000 7f35c6000700 1 -- 172.21.15.12:0/4120595835 learned_addr learned my addr 172.21.15.12:0/4120595835 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.405+0000 7f35c6a00700 1 --2- 172.21.15.12:0/4120595835 >> v2:172.21.15.159:3300/0 conn(0x7f35c8099a60 0x7f35c80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.405+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433555301 0 0) 0x7f35c80d0970 con 0x7f35c815d990 2024-03-29T10:06:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.405+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35ac0035b0 con 0x7f35c815d990 2024-03-29T10:06:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 144891264 0 0) 0x7f35ac0035b0 con 0x7f35c815d990 2024-03-29T10:06:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 >> v2:172.21.15.159:3300/0 conn(0x7f35c8099a60 msgr2=0x7f35c80d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 --2- 172.21.15.12:0/4120595835 >> v2:172.21.15.159:3300/0 conn(0x7f35c8099a60 0x7f35c80d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 >> v1:172.21.15.159:6789/0 conn(0x7f35c809a750 legacy=0x7f35c806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c80d0690 con 0x7f35c815d990 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f35b4001670 con 0x7f35c815d990 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 >> v1:172.21.15.12:6789/0 conn(0x7f35c815d990 legacy=0x7f35c8099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35ac003a00 0x7f35ac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35ac004850 0x7f35ac004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35ac005210 0x7f35ac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80c81a0 con 0x7f35ac003a00 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac00c160 con 0x7f35ac004850 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac00c320 con 0x7f35ac005210 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c6a00700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35ac003a00 0x7f35ac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c6000700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35ac004850 0x7f35ac004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.406+0000 7f35c7400700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35ac005210 0x7f35ac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c6a00700 1 -- 172.21.15.12:0/4120595835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35ac005210 msgr2=0x7f35ac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c6a00700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35ac005210 0x7f35ac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c6a00700 1 -- 172.21.15.12:0/4120595835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35ac004850 msgr2=0x7f35ac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c6a00700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35ac004850 0x7f35ac004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c6a00700 1 -- 172.21.15.12:0/4120595835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac0035b0 con 0x7f35ac003a00 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c7400700 1 -- 172.21.15.12:0/4120595835 reap_dead start 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c6000700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35ac004850 0x7f35ac004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c6a00700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35ac003a00 0x7f35ac003de0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009b60 tx=0x7f35ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82957be63d9ed3f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b800e6c0 con 0x7f35ac003a00 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b800e820 con 0x7f35ac003a00 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.407+0000 7f35c5600700 1 -- 172.21.15.12:0/4120595835 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b800eb90 con 0x7f35ac003a00 2024-03-29T10:06:22.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.408+0000 7f35cd800700 1 -- 172.21.15.12:0/4120595835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35ac003a00 msgr2=0x7f35ac003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.408+0000 7f35cd800700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35ac003a00 0x7f35ac003de0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009b60 tx=0x7f35ac005880 comp rx=0 tx=0).stop 2024-03-29T10:06:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.408+0000 7f35cd800700 1 -- 172.21.15.12:0/4120595835 shutdown_connections 2024-03-29T10:06:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.408+0000 7f35cd800700 1 --2- 172.21.15.12:0/4120595835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35ac003a00 0x7f35ac003de0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.408+0000 7f35cd800700 1 -- 172.21.15.12:0/4120595835 >> 172.21.15.12:0/4120595835 conn(0x7f35c80b2e20 msgr2=0x7f35c80b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.408+0000 7f35cd800700 1 -- 172.21.15.12:0/4120595835 shutdown_connections 2024-03-29T10:06:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.408+0000 7f35cd800700 1 -- 172.21.15.12:0/4120595835 wait complete. 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.410+0000 7f35cd800700 1 Processor -- start 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.410+0000 7f35cd800700 1 -- start start 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35cd800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35ac003a00 0x7f35c81f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35cd800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35c80cb960 0x7f35c80ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35cd800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c80cee20 0x7f35c814b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35cd800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80cd560 con 0x7f35c80cb960 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35cd800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80cd840 con 0x7f35ac003a00 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35cd800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80cdb20 con 0x7f35c80cee20 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35c6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35c80cb960 0x7f35c80ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35c6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35c80cb960 0x7f35c80ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33018/0 (socket says 172.21.15.12:33018) 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35c6000700 1 -- 172.21.15.12:0/3212301391 learned_addr learned my addr 172.21.15.12:0/3212301391 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35c7400700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c80cee20 0x7f35c814b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35c6a00700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35ac003a00 0x7f35c81f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35c6000700 1 -- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c80cee20 msgr2=0x7f35c814b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35c6000700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c80cee20 0x7f35c814b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35c6000700 1 -- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35ac003a00 msgr2=0x7f35c81f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35c6000700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35ac003a00 0x7f35c81f26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.411+0000 7f35c6000700 1 -- 172.21.15.12:0/3212301391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c80cde00 con 0x7f35c80cb960 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.412+0000 7f35c6000700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35c80cb960 0x7f35c80ce8e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f35b40017a0 tx=0x7f35b400eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.412+0000 7f35bea00700 1 -- 172.21.15.12:0/3212301391 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b4012590 con 0x7f35c80cb960 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.412+0000 7f35bea00700 1 -- 172.21.15.12:0/3212301391 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b40126f0 con 0x7f35c80cb960 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.412+0000 7f35bea00700 1 -- 172.21.15.12:0/3212301391 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b4012a60 con 0x7f35c80cb960 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.412+0000 7f35cd800700 1 -- 172.21.15.12:0/3212301391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c80ce0e0 con 0x7f35c80cb960 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.412+0000 7f35cd800700 1 -- 172.21.15.12:0/3212301391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c814e030 con 0x7f35c80cb960 2024-03-29T10:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.413+0000 7f35cd800700 1 -- 172.21.15.12:0/3212301391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c806ab60 con 0x7f35c80cb960 2024-03-29T10:06:22.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.415+0000 7f35bea00700 1 -- 172.21.15.12:0/3212301391 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f35b4012bc0 con 0x7f35c80cb960 2024-03-29T10:06:22.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.415+0000 7f35bea00700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f35a806aec0 0x7f35a806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.415+0000 7f35bea00700 1 -- 172.21.15.12:0/3212301391 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f35b4089500 con 0x7f35c80cb960 2024-03-29T10:06:22.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.416+0000 7f35c6a00700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f35a806aec0 0x7f35a806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.416+0000 7f35c6a00700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f35a806aec0 0x7f35a806d380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f35b80090b0 tx=0x7f35b8008030 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:22.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.420+0000 7f35bea00700 1 -- 172.21.15.12:0/3212301391 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f35b40542c0 con 0x7f35c80cb960 2024-03-29T10:06:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.664+0000 7f35cd800700 1 -- 172.21.15.12:0/3212301391 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f35c806bcb0 con 0x7f35a806aec0 2024-03-29T10:06:22.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.755+0000 7f35bea00700 1 -- 172.21.15.12:0/3212301391 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f35b8006c60 con 0x7f35a806aec0 2024-03-29T10:06:22.755 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2024-03-29T10:06:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 -- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f35a806aec0 msgr2=0x7f35a806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f35a806aec0 0x7f35a806d380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f35b80090b0 tx=0x7f35b8008030 comp rx=0 tx=0).stop 2024-03-29T10:06:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 -- 172.21.15.12:0/3212301391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35c80cb960 msgr2=0x7f35c80ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35c80cb960 0x7f35c80ce8e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f35b40017a0 tx=0x7f35b400eb10 comp rx=0 tx=0).stop 2024-03-29T10:06:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 -- 172.21.15.12:0/3212301391 shutdown_connections 2024-03-29T10:06:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f35a806aec0 0x7f35a806d380 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c80cee20 0x7f35c814b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35ac003a00 0x7f35c81f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 --2- 172.21.15.12:0/3212301391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35c80cb960 0x7f35c80ce8e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 -- 172.21.15.12:0/3212301391 >> 172.21.15.12:0/3212301391 conn(0x7f35c80b2e20 msgr2=0x7f35c80b5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:22.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 -- 172.21.15.12:0/3212301391 shutdown_connections 2024-03-29T10:06:22.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.758+0000 7f35cd800700 1 -- 172.21.15.12:0/3212301391 wait complete. 2024-03-29T10:06:22.775 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2024-03-29T10:06:22.775 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2024-03-29T10:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.978+0000 7fa730200700 1 Processor -- start 2024-03-29T10:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.978+0000 7fa730200700 1 -- start start 2024-03-29T10:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.979+0000 7fa730200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa7280b3b30 0x7fa7280cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.980+0000 7fa730200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7280b2070 con 0x7fa728157f90 2024-03-29T10:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.980+0000 7fa730200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72815e0f0 con 0x7fa7280b2b30 2024-03-29T10:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.980+0000 7fa730200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa728158370 con 0x7fa7280b3b30 2024-03-29T10:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.980+0000 7fa72d800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa728157f90 0x7fa7281613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52644/0 (socket says 172.21.15.12:52644) 2024-03-29T10:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.980+0000 7fa72d800700 1 -- 172.21.15.12:0/1623593606 learned_addr learned my addr 172.21.15.12:0/1623593606 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.980+0000 7fa72e200700 1 --2- 172.21.15.12:0/1623593606 >> v2:172.21.15.159:3300/0 conn(0x7fa7280b3b30 0x7fa7280cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.980+0000 7fa72e200700 1 -- 172.21.15.12:0/1623593606 >> v1:172.21.15.159:6789/0 conn(0x7fa7280b2b30 legacy=0x7fa7280b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.980+0000 7fa72e200700 1 -- 172.21.15.12:0/1623593606 >> v1:172.21.15.12:6789/0 conn(0x7fa728157f90 legacy=0x7fa7281613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.980+0000 7fa72e200700 1 -- 172.21.15.12:0/1623593606 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7280b2fe0 con 0x7fa7280b3b30 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.981+0000 7fa72e200700 1 --2- 172.21.15.12:0/1623593606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280b3b30 0x7fa7280cd020 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa720009d90 tx=0x7fa72000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc23be546cbdb439 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.981+0000 7fa727e00700 1 -- 172.21.15.12:0/1623593606 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72000e070 con 0x7fa7280b3b30 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.981+0000 7fa727e00700 1 -- 172.21.15.12:0/1623593606 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72000bbd0 con 0x7fa7280b3b30 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.981+0000 7fa727e00700 1 -- 172.21.15.12:0/1623593606 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720012430 con 0x7fa7280b3b30 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.982+0000 7fa730200700 1 -- 172.21.15.12:0/1623593606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280b3b30 msgr2=0x7fa7280cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.982+0000 7fa730200700 1 --2- 172.21.15.12:0/1623593606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280b3b30 0x7fa7280cd020 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa720009d90 tx=0x7fa72000b330 comp rx=0 tx=0).stop 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.982+0000 7fa730200700 1 -- 172.21.15.12:0/1623593606 shutdown_connections 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.982+0000 7fa730200700 1 --2- 172.21.15.12:0/1623593606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280b3b30 0x7fa7280cd020 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.982+0000 7fa730200700 1 -- 172.21.15.12:0/1623593606 >> 172.21.15.12:0/1623593606 conn(0x7fa7280b0d70 msgr2=0x7fa7280b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.982+0000 7fa730200700 1 -- 172.21.15.12:0/1623593606 shutdown_connections 2024-03-29T10:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.982+0000 7fa730200700 1 -- 172.21.15.12:0/1623593606 wait complete. 2024-03-29T10:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.983+0000 7fa730200700 1 Processor -- start 2024-03-29T10:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.984+0000 7fa730200700 1 -- start start 2024-03-29T10:06:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa730200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280b2b30 0x7fa7280ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa730200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7280b3b30 0x7fa7280c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa730200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa728157f90 0x7fa7281531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa730200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280b2710 con 0x7fa728157f90 2024-03-29T10:06:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa730200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280ac590 con 0x7fa7280b2b30 2024-03-29T10:06:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa730200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280c9700 con 0x7fa7280b3b30 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa72e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa728157f90 0x7fa7281531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa72e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa728157f90 0x7fa7281531b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33022/0 (socket says 172.21.15.12:33022) 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa72e200700 1 -- 172.21.15.12:0/567397806 learned_addr learned my addr 172.21.15.12:0/567397806 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa72d800700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280b2b30 0x7fa7280ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa72ce00700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7280b3b30 0x7fa7280c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa72e200700 1 -- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7280b3b30 msgr2=0x7fa7280c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa72e200700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7280b3b30 0x7fa7280c91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa72e200700 1 -- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280b2b30 msgr2=0x7fa7280ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa72e200700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280b2b30 0x7fa7280ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.985+0000 7fa72e200700 1 -- 172.21.15.12:0/567397806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa720009a70 con 0x7fa728157f90 2024-03-29T10:06:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.986+0000 7fa72e200700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa728157f90 0x7fa7281531b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa72806c130 tx=0x7fa720004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.986+0000 7fa725600700 1 -- 172.21.15.12:0/567397806 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720012d20 con 0x7fa728157f90 2024-03-29T10:06:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.986+0000 7fa725600700 1 -- 172.21.15.12:0/567397806 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa720020070 con 0x7fa728157f90 2024-03-29T10:06:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.986+0000 7fa730200700 1 -- 172.21.15.12:0/567397806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7280c9980 con 0x7fa728157f90 2024-03-29T10:06:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.986+0000 7fa725600700 1 -- 172.21.15.12:0/567397806 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720012ea0 con 0x7fa728157f90 2024-03-29T10:06:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.986+0000 7fa730200700 1 -- 172.21.15.12:0/567397806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72815fe60 con 0x7fa728157f90 2024-03-29T10:06:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.987+0000 7fa730200700 1 -- 172.21.15.12:0/567397806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa728067720 con 0x7fa728157f90 2024-03-29T10:06:22.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.988+0000 7fa725600700 1 -- 172.21.15.12:0/567397806 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fa720004280 con 0x7fa728157f90 2024-03-29T10:06:22.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.989+0000 7fa725600700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa70806ae90 0x7fa70806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:22.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.990+0000 7fa725600700 1 -- 172.21.15.12:0/567397806 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa720022070 con 0x7fa728157f90 2024-03-29T10:06:22.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.990+0000 7fa72d800700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa70806ae90 0x7fa70806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.991+0000 7fa72d800700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa70806ae90 0x7fa70806d350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa71c005910 tx=0x7fa71c009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:22.995+0000 7fa725600700 1 -- 172.21.15.12:0/567397806 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7200563d0 con 0x7fa728157f90 2024-03-29T10:06:23.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.240+0000 7fa730200700 1 -- 172.21.15.12:0/567397806 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7280b2330 con 0x7fa70806ae90 2024-03-29T10:06:23.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.332+0000 7fa725600700 1 -- 172.21.15.12:0/567397806 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa71c001580 con 0x7fa70806ae90 2024-03-29T10:06:23.331 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2024-03-29T10:06:23.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 -- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa70806ae90 msgr2=0x7fa70806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:23.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa70806ae90 0x7fa70806d350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa71c005910 tx=0x7fa71c009040 comp rx=0 tx=0).stop 2024-03-29T10:06:23.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 -- 172.21.15.12:0/567397806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa728157f90 msgr2=0x7fa7281531b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:23.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa728157f90 0x7fa7281531b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa72806c130 tx=0x7fa720004ad0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 -- 172.21.15.12:0/567397806 shutdown_connections 2024-03-29T10:06:23.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa70806ae90 0x7fa70806d350 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa728157f90 0x7fa7281531b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280b2b30 0x7fa7280ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 --2- 172.21.15.12:0/567397806 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7280b3b30 0x7fa7280c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 -- 172.21.15.12:0/567397806 >> 172.21.15.12:0/567397806 conn(0x7fa7280b0d70 msgr2=0x7fa72806c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:23.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 -- 172.21.15.12:0/567397806 shutdown_connections 2024-03-29T10:06:23.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.334+0000 7fa730200700 1 -- 172.21.15.12:0/567397806 wait complete. 2024-03-29T10:06:23.352 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2024-03-29T10:06:23.352 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2024-03-29T10:06:23.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.556+0000 7f2d15600700 1 Processor -- start 2024-03-29T10:06:23.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.556+0000 7f2d15600700 1 -- start start 2024-03-29T10:06:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.557+0000 7f2d15600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d1015d9d0 0x7f2d100d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.557+0000 7f2d15600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d100d2540 0x7f2d100ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.557+0000 7f2d15600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d10161420 con 0x7f2d100d16a0 2024-03-29T10:06:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.557+0000 7f2d15600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1015edc0 con 0x7f2d100d2540 2024-03-29T10:06:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.557+0000 7f2d15600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10161700 con 0x7f2d1015d9d0 2024-03-29T10:06:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.557+0000 7f2d0f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d100d2540 0x7f2d100ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.557+0000 7f2d0f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d100d2540 0x7f2d100ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33026/0 (socket says 172.21.15.12:33026) 2024-03-29T10:06:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.557+0000 7f2d0f400700 1 -- 172.21.15.12:0/121105537 learned_addr learned my addr 172.21.15.12:0/121105537 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.557+0000 7f2d0ea00700 1 --2- 172.21.15.12:0/121105537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d1015d9d0 0x7f2d100d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.558+0000 7f2d0f400700 1 -- 172.21.15.12:0/121105537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d1015d9d0 msgr2=0x7f2d100d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.558+0000 7f2d0f400700 1 --2- 172.21.15.12:0/121105537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d1015d9d0 0x7f2d100d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.558+0000 7f2d0f400700 1 -- 172.21.15.12:0/121105537 >> v1:172.21.15.12:6789/0 conn(0x7f2d100d16a0 legacy=0x7f2d100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.558+0000 7f2d0f400700 1 -- 172.21.15.12:0/121105537 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d101619e0 con 0x7f2d100d2540 2024-03-29T10:06:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.558+0000 7f2d0f400700 1 --2- 172.21.15.12:0/121105537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d2540 0x7f2d100ce360 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2d08009e80 tx=0x7f2d0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765eff74d430ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.558+0000 7f2d0d600700 1 -- 172.21.15.12:0/121105537 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800e070 con 0x7f2d100d2540 2024-03-29T10:06:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.558+0000 7f2d0d600700 1 -- 172.21.15.12:0/121105537 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0800b980 con 0x7f2d100d2540 2024-03-29T10:06:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.558+0000 7f2d0d600700 1 -- 172.21.15.12:0/121105537 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800bd30 con 0x7f2d100d2540 2024-03-29T10:06:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.559+0000 7f2d15600700 1 -- 172.21.15.12:0/121105537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d2540 msgr2=0x7f2d100ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.559+0000 7f2d15600700 1 --2- 172.21.15.12:0/121105537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d2540 0x7f2d100ce360 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2d08009e80 tx=0x7f2d0800b330 comp rx=0 tx=0).stop 2024-03-29T10:06:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.559+0000 7f2d15600700 1 -- 172.21.15.12:0/121105537 shutdown_connections 2024-03-29T10:06:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.559+0000 7f2d15600700 1 --2- 172.21.15.12:0/121105537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d2540 0x7f2d100ce360 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.559+0000 7f2d15600700 1 --2- 172.21.15.12:0/121105537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d1015d9d0 0x7f2d100d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.559+0000 7f2d15600700 1 -- 172.21.15.12:0/121105537 >> 172.21.15.12:0/121105537 conn(0x7f2d1008aff0 msgr2=0x7f2d100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.559+0000 7f2d15600700 1 -- 172.21.15.12:0/121105537 shutdown_connections 2024-03-29T10:06:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.559+0000 7f2d15600700 1 -- 172.21.15.12:0/121105537 wait complete. 2024-03-29T10:06:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.561+0000 7f2d15600700 1 Processor -- start 2024-03-29T10:06:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.561+0000 7f2d15600700 1 -- start start 2024-03-29T10:06:23.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d15600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d16a0 0x7f2d10154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:23.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d15600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d100d2540 0x7f2d100b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d15600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d1015d9d0 0x7f2d100b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d15600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100b60c0 con 0x7f2d100d16a0 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d15600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100b63a0 con 0x7f2d1015d9d0 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d15600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100b6680 con 0x7f2d100d2540 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d16a0 0x7f2d10154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d16a0 0x7f2d10154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33028/0 (socket says 172.21.15.12:33028) 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0ea00700 1 -- 172.21.15.12:0/1607250426 learned_addr learned my addr 172.21.15.12:0/1607250426 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0e000700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d100d2540 0x7f2d100b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0f400700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d1015d9d0 0x7f2d100b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0ea00700 1 -- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d100d2540 msgr2=0x7f2d100b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0ea00700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d100d2540 0x7f2d100b83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0ea00700 1 -- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d1015d9d0 msgr2=0x7f2d100b8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0ea00700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d1015d9d0 0x7f2d100b8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0ea00700 1 -- 172.21.15.12:0/1607250426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d08009b30 con 0x7f2d100d16a0 2024-03-29T10:06:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0e000700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d100d2540 0x7f2d100b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.563+0000 7f2d0f400700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d1015d9d0 0x7f2d100b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.564+0000 7f2d0ea00700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d16a0 0x7f2d10154480 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000d800 tx=0x7f2d0000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.564+0000 7f2d06a00700 1 -- 172.21.15.12:0/1607250426 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d000125f0 con 0x7f2d100d16a0 2024-03-29T10:06:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.564+0000 7f2d06a00700 1 -- 172.21.15.12:0/1607250426 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d00012750 con 0x7f2d100d16a0 2024-03-29T10:06:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.564+0000 7f2d15600700 1 -- 172.21.15.12:0/1607250426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d100b6960 con 0x7f2d100d16a0 2024-03-29T10:06:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.564+0000 7f2d06a00700 1 -- 172.21.15.12:0/1607250426 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00012b00 con 0x7f2d100d16a0 2024-03-29T10:06:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.564+0000 7f2d15600700 1 -- 172.21.15.12:0/1607250426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d101549c0 con 0x7f2d100d16a0 2024-03-29T10:06:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.565+0000 7f2d15600700 1 -- 172.21.15.12:0/1607250426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d100d1ca0 con 0x7f2d100d16a0 2024-03-29T10:06:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.566+0000 7f2d06a00700 1 -- 172.21.15.12:0/1607250426 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f2d00012c60 con 0x7f2d100d16a0 2024-03-29T10:06:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.567+0000 7f2d06a00700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf006ae90 0x7f2cf006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.567+0000 7f2d06a00700 1 -- 172.21.15.12:0/1607250426 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f2d00089bf0 con 0x7f2d100d16a0 2024-03-29T10:06:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.567+0000 7f2d0e000700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf006ae90 0x7f2cf006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.568+0000 7f2d0e000700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf006ae90 0x7f2cf006d350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2d10156670 tx=0x7f2cfc009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:23.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.571+0000 7f2d06a00700 1 -- 172.21.15.12:0/1607250426 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d000549b0 con 0x7f2d100d16a0 2024-03-29T10:06:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.816+0000 7f2d15600700 1 -- 172.21.15.12:0/1607250426 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d10161fa0 con 0x7f2cf006ae90 2024-03-29T10:06:23.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.908+0000 7f2d06a00700 1 -- 172.21.15.12:0/1607250426 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f2cfc0013d0 con 0x7f2cf006ae90 2024-03-29T10:06:23.908 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2024-03-29T10:06:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 -- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf006ae90 msgr2=0x7f2cf006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf006ae90 0x7f2cf006d350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2d10156670 tx=0x7f2cfc009040 comp rx=0 tx=0).stop 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 -- 172.21.15.12:0/1607250426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d16a0 msgr2=0x7f2d10154480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d16a0 0x7f2d10154480 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000d800 tx=0x7f2d0000dbc0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 -- 172.21.15.12:0/1607250426 shutdown_connections 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cf006ae90 0x7f2cf006d350 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d1015d9d0 0x7f2d100b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d100d16a0 0x7f2d10154480 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 --2- 172.21.15.12:0/1607250426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d100d2540 0x7f2d100b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 -- 172.21.15.12:0/1607250426 >> 172.21.15.12:0/1607250426 conn(0x7f2d1008aff0 msgr2=0x7f2d1015fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 -- 172.21.15.12:0/1607250426 shutdown_connections 2024-03-29T10:06:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:23.910+0000 7f2d15600700 1 -- 172.21.15.12:0/1607250426 wait complete. 2024-03-29T10:06:23.928 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2024-03-29T10:06:23.928 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2024-03-29T10:06:24.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.131+0000 7f01fa400700 1 Processor -- start 2024-03-29T10:06:24.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.131+0000 7f01fa400700 1 -- start start 2024-03-29T10:06:24.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.132+0000 7f01fa400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f01f40d2540 0x7f01f41687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.132+0000 7f01fa400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f4161420 con 0x7f01f40d16a0 2024-03-29T10:06:24.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.132+0000 7f01fa400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f4161700 con 0x7f01f415d9d0 2024-03-29T10:06:24.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.132+0000 7f01fa400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f40d4340 con 0x7f01f40d2540 2024-03-29T10:06:24.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.132+0000 7f01f3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f01f40d2540 0x7f01f41687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.132+0000 7f01f3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f01f40d2540 0x7f01f41687b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33036/0 (socket says 172.21.15.12:33036) 2024-03-29T10:06:24.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.132+0000 7f01f3e00700 1 -- 172.21.15.12:0/105584091 learned_addr learned my addr 172.21.15.12:0/105584091 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:24.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.133+0000 7f01f3e00700 1 -- 172.21.15.12:0/105584091 >> v1:172.21.15.159:6789/0 conn(0x7f01f415d9d0 legacy=0x7f01f40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:24.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.133+0000 7f01f3e00700 1 -- 172.21.15.12:0/105584091 >> v1:172.21.15.12:6789/0 conn(0x7f01f40d16a0 legacy=0x7f01f40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:24.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.133+0000 7f01f3e00700 1 -- 172.21.15.12:0/105584091 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f41619e0 con 0x7f01f40d2540 2024-03-29T10:06:24.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.133+0000 7f01f3e00700 1 --2- 172.21.15.12:0/105584091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d2540 0x7f01f41687b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009e50 tx=0x7f01ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4aeaaa3dbd2aa0e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:24.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.133+0000 7f01f2000700 1 -- 172.21.15.12:0/105584091 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec00e070 con 0x7f01f40d2540 2024-03-29T10:06:24.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.133+0000 7f01f2000700 1 -- 172.21.15.12:0/105584091 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ec00b9c0 con 0x7f01f40d2540 2024-03-29T10:06:24.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.133+0000 7f01f2000700 1 -- 172.21.15.12:0/105584091 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec00bd70 con 0x7f01f40d2540 2024-03-29T10:06:24.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.134+0000 7f01fa400700 1 -- 172.21.15.12:0/105584091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d2540 msgr2=0x7f01f41687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:24.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.134+0000 7f01fa400700 1 --2- 172.21.15.12:0/105584091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d2540 0x7f01f41687b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009e50 tx=0x7f01ec00b330 comp rx=0 tx=0).stop 2024-03-29T10:06:24.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.134+0000 7f01fa400700 1 -- 172.21.15.12:0/105584091 shutdown_connections 2024-03-29T10:06:24.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.134+0000 7f01fa400700 1 --2- 172.21.15.12:0/105584091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d2540 0x7f01f41687b0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.134+0000 7f01fa400700 1 -- 172.21.15.12:0/105584091 >> 172.21.15.12:0/105584091 conn(0x7f01f408aff0 msgr2=0x7f01f40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:24.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.134+0000 7f01fa400700 1 -- 172.21.15.12:0/105584091 shutdown_connections 2024-03-29T10:06:24.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.134+0000 7f01fa400700 1 -- 172.21.15.12:0/105584091 wait complete. 2024-03-29T10:06:24.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.136+0000 7f01fa400700 1 Processor -- start 2024-03-29T10:06:24.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.136+0000 7f01fa400700 1 -- start start 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01fa400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d16a0 0x7f01f41f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01fa400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01f40d2540 0x7f01f41f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01fa400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01f415d9d0 0x7f01f40b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01fa400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f40b7070 con 0x7f01f40d16a0 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01fa400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f40b7330 con 0x7f01f415d9d0 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01fa400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f40b7610 con 0x7f01f40d2540 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d16a0 0x7f01f41f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d16a0 0x7f01f41f34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33042/0 (socket says 172.21.15.12:33042) 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f3400700 1 -- 172.21.15.12:0/1213527502 learned_addr learned my addr 172.21.15.12:0/1213527502 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f2a00700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01f40d2540 0x7f01f41f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f3e00700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01f415d9d0 0x7f01f40b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f3400700 1 -- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01f40d2540 msgr2=0x7f01f41f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f3400700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01f40d2540 0x7f01f41f6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f3400700 1 -- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01f415d9d0 msgr2=0x7f01f40b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f3400700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01f415d9d0 0x7f01f40b4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f3400700 1 -- 172.21.15.12:0/1213527502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec009b30 con 0x7f01f40d16a0 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f3e00700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01f415d9d0 0x7f01f40b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.137+0000 7f01f2a00700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01f40d2540 0x7f01f41f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.138+0000 7f01f3400700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d16a0 0x7f01f41f34b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f01e400c450 tx=0x7f01e400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.138+0000 7f01eb400700 1 -- 172.21.15.12:0/1213527502 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40052d0 con 0x7f01f40d16a0 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.138+0000 7f01eb400700 1 -- 172.21.15.12:0/1213527502 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e4005430 con 0x7f01f40d16a0 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.138+0000 7f01eb400700 1 -- 172.21.15.12:0/1213527502 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40038c0 con 0x7f01f40d16a0 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.138+0000 7f01fa400700 1 -- 172.21.15.12:0/1213527502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f40b78f0 con 0x7f01f40d16a0 2024-03-29T10:06:24.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.138+0000 7f01fa400700 1 -- 172.21.15.12:0/1213527502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f41f6750 con 0x7f01f40d16a0 2024-03-29T10:06:24.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.139+0000 7f01fa400700 1 -- 172.21.15.12:0/1213527502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f40d1ca0 con 0x7f01f40d16a0 2024-03-29T10:06:24.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.140+0000 7f01eb400700 1 -- 172.21.15.12:0/1213527502 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f01e4003a20 con 0x7f01f40d16a0 2024-03-29T10:06:24.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.141+0000 7f01eb400700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f01d406ae90 0x7f01d406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.141+0000 7f01eb400700 1 -- 172.21.15.12:0/1213527502 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f01e4088b60 con 0x7f01f40d16a0 2024-03-29T10:06:24.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.141+0000 7f01f2a00700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f01d406ae90 0x7f01d406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.142+0000 7f01f2a00700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f01d406ae90 0x7f01d406d350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f01f40cbb00 tx=0x7f01e0009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:24.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.146+0000 7f01eb400700 1 -- 172.21.15.12:0/1213527502 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f01e40539b0 con 0x7f01f40d16a0 2024-03-29T10:06:24.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.389+0000 7f01fa400700 1 -- 172.21.15.12:0/1213527502 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f01f4161fa0 con 0x7f01d406ae90 2024-03-29T10:06:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.476+0000 7f01eb400700 1 -- 172.21.15.12:0/1213527502 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f01e00013d0 con 0x7f01d406ae90 2024-03-29T10:06:24.476 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2024-03-29T10:06:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 -- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f01d406ae90 msgr2=0x7f01d406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f01d406ae90 0x7f01d406d350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f01f40cbb00 tx=0x7f01e0009040 comp rx=0 tx=0).stop 2024-03-29T10:06:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 -- 172.21.15.12:0/1213527502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d16a0 msgr2=0x7f01f41f34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d16a0 0x7f01f41f34b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f01e400c450 tx=0x7f01e400c810 comp rx=0 tx=0).stop 2024-03-29T10:06:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 -- 172.21.15.12:0/1213527502 shutdown_connections 2024-03-29T10:06:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f01d406ae90 0x7f01d406d350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01f415d9d0 0x7f01f40b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01f40d16a0 0x7f01f41f34b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 --2- 172.21.15.12:0/1213527502 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01f40d2540 0x7f01f41f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 -- 172.21.15.12:0/1213527502 >> 172.21.15.12:0/1213527502 conn(0x7f01f408aff0 msgr2=0x7f01f40b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.479+0000 7f01fa400700 1 -- 172.21.15.12:0/1213527502 shutdown_connections 2024-03-29T10:06:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.480+0000 7f01fa400700 1 -- 172.21.15.12:0/1213527502 wait complete. 2024-03-29T10:06:24.496 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2024-03-29T10:06:24.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2024-03-29T10:06:24.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.697+0000 7fa581800700 1 Processor -- start 2024-03-29T10:06:24.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.697+0000 7fa581800700 1 -- start start 2024-03-29T10:06:24.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.698+0000 7fa581800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa57c15d9d0 0x7fa57c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.698+0000 7fa581800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c0d16a0 0x7fa57c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa581800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa57c0d2370 0x7fa57c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa581800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa57c0d2820 con 0x7fa57c0d2370 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa581800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa57c161420 con 0x7fa57c15d9d0 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa581800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c161700 con 0x7fa57c0d16a0 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa57b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa57c0d2370 0x7fa57c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa57b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa57c0d2370 0x7fa57c15ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33058/0 (socket says 172.21.15.12:33058) 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa57b400700 1 -- 172.21.15.12:0/285781503 learned_addr learned my addr 172.21.15.12:0/285781503 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa57b400700 1 -- 172.21.15.12:0/285781503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c0d16a0 msgr2=0x7fa57c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa57b400700 1 --2- 172.21.15.12:0/285781503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c0d16a0 0x7fa57c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa57b400700 1 -- 172.21.15.12:0/285781503 >> v2:172.21.15.159:3300/0 conn(0x7fa57c15d9d0 msgr2=0x7fa57c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa57aa00700 1 --2- 172.21.15.12:0/285781503 >> v2:172.21.15.159:3300/0 conn(0x7fa57c15d9d0 0x7fa57c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa57b400700 1 --2- 172.21.15.12:0/285781503 >> v2:172.21.15.159:3300/0 conn(0x7fa57c15d9d0 0x7fa57c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa57b400700 1 -- 172.21.15.12:0/285781503 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57c1619e0 con 0x7fa57c0d2370 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.699+0000 7fa57aa00700 1 --2- 172.21.15.12:0/285781503 >> v2:172.21.15.159:3300/0 conn(0x7fa57c15d9d0 0x7fa57c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:24.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa57b400700 1 --2- 172.21.15.12:0/285781503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d2370 0x7fa57c15ddb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa57c0d0ff0 tx=0x7fa57400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54ac2442907bace server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:24.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa579600700 1 -- 172.21.15.12:0/285781503 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57400e070 con 0x7fa57c0d2370 2024-03-29T10:06:24.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa579600700 1 -- 172.21.15.12:0/285781503 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57400b9c0 con 0x7fa57c0d2370 2024-03-29T10:06:24.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa579600700 1 -- 172.21.15.12:0/285781503 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57400bd70 con 0x7fa57c0d2370 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa581800700 1 -- 172.21.15.12:0/285781503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d2370 msgr2=0x7fa57c15ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa581800700 1 --2- 172.21.15.12:0/285781503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d2370 0x7fa57c15ddb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa57c0d0ff0 tx=0x7fa57400b330 comp rx=0 tx=0).stop 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa581800700 1 -- 172.21.15.12:0/285781503 shutdown_connections 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa581800700 1 --2- 172.21.15.12:0/285781503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d2370 0x7fa57c15ddb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa581800700 1 --2- 172.21.15.12:0/285781503 >> v2:172.21.15.159:3300/0 conn(0x7fa57c15d9d0 0x7fa57c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa581800700 1 --2- 172.21.15.12:0/285781503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c0d16a0 0x7fa57c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.700+0000 7fa581800700 1 -- 172.21.15.12:0/285781503 >> 172.21.15.12:0/285781503 conn(0x7fa57c08aff0 msgr2=0x7fa57c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.701+0000 7fa581800700 1 -- 172.21.15.12:0/285781503 shutdown_connections 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.701+0000 7fa581800700 1 -- 172.21.15.12:0/285781503 wait complete. 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.702+0000 7fa581800700 1 Processor -- start 2024-03-29T10:06:24.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.702+0000 7fa581800700 1 -- start start 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.703+0000 7fa581800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d16a0 0x7fa57c0b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.703+0000 7fa581800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c0d2370 0x7fa57c15c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.703+0000 7fa581800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c15d9d0 0x7fa57c0b8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.703+0000 7fa581800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c0b8a60 con 0x7fa57c0d16a0 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.703+0000 7fa581800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c0b8d40 con 0x7fa57c15d9d0 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.703+0000 7fa581800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c0b60c0 con 0x7fa57c0d2370 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d16a0 0x7fa57c0b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d16a0 0x7fa57c0b6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33068/0 (socket says 172.21.15.12:33068) 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57aa00700 1 -- 172.21.15.12:0/157241645 learned_addr learned my addr 172.21.15.12:0/157241645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57b400700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c15d9d0 0x7fa57c0b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57a000700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c0d2370 0x7fa57c15c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57aa00700 1 -- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c0d2370 msgr2=0x7fa57c15c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57aa00700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c0d2370 0x7fa57c15c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57aa00700 1 -- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c15d9d0 msgr2=0x7fa57c0b8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57aa00700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c15d9d0 0x7fa57c0b8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57aa00700 1 -- 172.21.15.12:0/157241645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa574009b30 con 0x7fa57c0d16a0 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57a000700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c0d2370 0x7fa57c15c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57b400700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c15d9d0 0x7fa57c0b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.704+0000 7fa57aa00700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d16a0 0x7fa57c0b6ba0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00c7f0 tx=0x7fa56c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.705+0000 7fa572a00700 1 -- 172.21.15.12:0/157241645 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c011800 con 0x7fa57c0d16a0 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.705+0000 7fa572a00700 1 -- 172.21.15.12:0/157241645 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c011960 con 0x7fa57c0d16a0 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.705+0000 7fa572a00700 1 -- 172.21.15.12:0/157241645 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c011d10 con 0x7fa57c0d16a0 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.705+0000 7fa581800700 1 -- 172.21.15.12:0/157241645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57c0b63a0 con 0x7fa57c0d16a0 2024-03-29T10:06:24.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.705+0000 7fa581800700 1 -- 172.21.15.12:0/157241645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57c15ce40 con 0x7fa57c0d16a0 2024-03-29T10:06:24.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.706+0000 7fa581800700 1 -- 172.21.15.12:0/157241645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57c0d1ca0 con 0x7fa57c0d16a0 2024-03-29T10:06:24.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.707+0000 7fa572a00700 1 -- 172.21.15.12:0/157241645 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fa56c018360 con 0x7fa57c0d16a0 2024-03-29T10:06:24.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.708+0000 7fa572a00700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa55c06ae90 0x7fa55c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:24.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.708+0000 7fa572a00700 1 -- 172.21.15.12:0/157241645 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa56c089930 con 0x7fa57c0d16a0 2024-03-29T10:06:24.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.708+0000 7fa57a000700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa55c06ae90 0x7fa55c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:24.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.709+0000 7fa57a000700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa55c06ae90 0x7fa55c06d350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa57c0b1f00 tx=0x7fa568008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:24.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.712+0000 7fa572a00700 1 -- 172.21.15.12:0/157241645 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa56c0545f0 con 0x7fa57c0d16a0 2024-03-29T10:06:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:24.958+0000 7fa581800700 1 -- 172.21.15.12:0/157241645 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa57c161fa0 con 0x7fa55c06ae90 2024-03-29T10:06:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.048+0000 7fa572a00700 1 -- 172.21.15.12:0/157241645 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa568009e70 con 0x7fa55c06ae90 2024-03-29T10:06:25.048 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2024-03-29T10:06:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 -- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa55c06ae90 msgr2=0x7fa55c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa55c06ae90 0x7fa55c06d350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa57c0b1f00 tx=0x7fa568008040 comp rx=0 tx=0).stop 2024-03-29T10:06:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 -- 172.21.15.12:0/157241645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d16a0 msgr2=0x7fa57c0b6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d16a0 0x7fa57c0b6ba0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00c7f0 tx=0x7fa56c00cbb0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 -- 172.21.15.12:0/157241645 shutdown_connections 2024-03-29T10:06:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa55c06ae90 0x7fa55c06d350 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c15d9d0 0x7fa57c0b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa57c0d16a0 0x7fa57c0b6ba0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 --2- 172.21.15.12:0/157241645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c0d2370 0x7fa57c15c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 -- 172.21.15.12:0/157241645 >> 172.21.15.12:0/157241645 conn(0x7fa57c08aff0 msgr2=0x7fa57c0c9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 -- 172.21.15.12:0/157241645 shutdown_connections 2024-03-29T10:06:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.050+0000 7fa581800700 1 -- 172.21.15.12:0/157241645 wait complete. 2024-03-29T10:06:25.067 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2024-03-29T10:06:25.068 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2024-03-29T10:06:25.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.269+0000 7f0689000700 1 Processor -- start 2024-03-29T10:06:25.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.270+0000 7f0689000700 1 -- start start 2024-03-29T10:06:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0689000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f068415d9d0 0x7f06840d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0689000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f06840d16a0 0x7f06840d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0689000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0684161420 con 0x7f06840d2370 2024-03-29T10:06:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0689000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06840d27e0 con 0x7f068415d9d0 2024-03-29T10:06:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0689000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0684161700 con 0x7f06840d16a0 2024-03-29T10:06:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0681e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f068415d9d0 0x7f06840d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0681e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f068415d9d0 0x7f06840d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33072/0 (socket says 172.21.15.12:33072) 2024-03-29T10:06:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0681e00700 1 -- 172.21.15.12:0/3714139125 learned_addr learned my addr 172.21.15.12:0/3714139125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0681e00700 1 -- 172.21.15.12:0/3714139125 >> v2:172.21.15.159:3300/0 conn(0x7f06840d16a0 msgr2=0x7f06840d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0681e00700 1 --2- 172.21.15.12:0/3714139125 >> v2:172.21.15.159:3300/0 conn(0x7f06840d16a0 0x7f06840d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0681e00700 1 -- 172.21.15.12:0/3714139125 >> v1:172.21.15.12:6789/0 conn(0x7f06840d2370 legacy=0x7f068415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.271+0000 7f0681e00700 1 -- 172.21.15.12:0/3714139125 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06841619e0 con 0x7f068415d9d0 2024-03-29T10:06:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.272+0000 7f0681e00700 1 --2- 172.21.15.12:0/3714139125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068415d9d0 0x7f06840d3c30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f06840d12f0 tx=0x7f0674009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af2839a03738e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.272+0000 7f0680a00700 1 -- 172.21.15.12:0/3714139125 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067400b870 con 0x7f068415d9d0 2024-03-29T10:06:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.272+0000 7f0680a00700 1 -- 172.21.15.12:0/3714139125 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067400b9d0 con 0x7f068415d9d0 2024-03-29T10:06:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.272+0000 7f0680a00700 1 -- 172.21.15.12:0/3714139125 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067400bd60 con 0x7f068415d9d0 2024-03-29T10:06:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.273+0000 7f0689000700 1 -- 172.21.15.12:0/3714139125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068415d9d0 msgr2=0x7f06840d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.273+0000 7f0689000700 1 --2- 172.21.15.12:0/3714139125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068415d9d0 0x7f06840d3c30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f06840d12f0 tx=0x7f0674009bd0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.273+0000 7f0689000700 1 -- 172.21.15.12:0/3714139125 shutdown_connections 2024-03-29T10:06:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.273+0000 7f0689000700 1 --2- 172.21.15.12:0/3714139125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068415d9d0 0x7f06840d3c30 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.273+0000 7f0689000700 1 --2- 172.21.15.12:0/3714139125 >> v2:172.21.15.159:3300/0 conn(0x7f06840d16a0 0x7f06840d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.273+0000 7f0689000700 1 -- 172.21.15.12:0/3714139125 >> 172.21.15.12:0/3714139125 conn(0x7f068408aff0 msgr2=0x7f06840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.273+0000 7f0689000700 1 -- 172.21.15.12:0/3714139125 shutdown_connections 2024-03-29T10:06:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.273+0000 7f0689000700 1 -- 172.21.15.12:0/3714139125 wait complete. 2024-03-29T10:06:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.274+0000 7f0689000700 1 Processor -- start 2024-03-29T10:06:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.275+0000 7f0689000700 1 -- start start 2024-03-29T10:06:25.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.275+0000 7f0689000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06840d16a0 0x7f0684155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.275+0000 7f0689000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06840d2370 0x7f06840ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.275+0000 7f0689000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f068415d9d0 0x7f06840ce860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0689000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06841558a0 con 0x7f06840d2370 2024-03-29T10:06:25.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0689000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06840b7ed0 con 0x7f068415d9d0 2024-03-29T10:06:25.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0689000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06840b8fb0 con 0x7f06840d16a0 2024-03-29T10:06:25.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0681400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06840d2370 0x7f06840ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0681400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06840d2370 0x7f06840ce320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33076/0 (socket says 172.21.15.12:33076) 2024-03-29T10:06:25.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0681400700 1 -- 172.21.15.12:0/285870100 learned_addr learned my addr 172.21.15.12:0/285870100 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:25.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0681e00700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06840d16a0 0x7f0684155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0682800700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f068415d9d0 0x7f06840ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0681400700 1 -- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06840d16a0 msgr2=0x7f0684155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0681400700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06840d16a0 0x7f0684155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0681400700 1 -- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f068415d9d0 msgr2=0x7f06840ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0681400700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f068415d9d0 0x7f06840ce860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.276+0000 7f0681400700 1 -- 172.21.15.12:0/285870100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0674009550 con 0x7f06840d2370 2024-03-29T10:06:25.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.277+0000 7f0681400700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06840d2370 0x7f06840ce320 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f067000b550 tx=0x7f067000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:25.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.277+0000 7f067a000700 1 -- 172.21.15.12:0/285870100 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670005aa0 con 0x7f06840d2370 2024-03-29T10:06:25.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.277+0000 7f067a000700 1 -- 172.21.15.12:0/285870100 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0670005c00 con 0x7f06840d2370 2024-03-29T10:06:25.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.277+0000 7f067a000700 1 -- 172.21.15.12:0/285870100 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670011630 con 0x7f06840d2370 2024-03-29T10:06:25.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.277+0000 7f0689000700 1 -- 172.21.15.12:0/285870100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06840b7110 con 0x7f06840d2370 2024-03-29T10:06:25.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.277+0000 7f0689000700 1 -- 172.21.15.12:0/285870100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06840b76d0 con 0x7f06840d2370 2024-03-29T10:06:25.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.278+0000 7f0689000700 1 -- 172.21.15.12:0/285870100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06840d1ca0 con 0x7f06840d2370 2024-03-29T10:06:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.279+0000 7f067a000700 1 -- 172.21.15.12:0/285870100 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f0670011790 con 0x7f06840d2370 2024-03-29T10:06:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.280+0000 7f067a000700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f066406ae90 0x7f066406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.280+0000 7f067a000700 1 -- 172.21.15.12:0/285870100 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f06700893f0 con 0x7f06840d2370 2024-03-29T10:06:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.281+0000 7f0681e00700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f066406ae90 0x7f066406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.281+0000 7f0681e00700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f066406ae90 0x7f066406d350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0674014040 tx=0x7f0674008030 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.285+0000 7f067a000700 1 -- 172.21.15.12:0/285870100 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06700541b0 con 0x7f06840d2370 2024-03-29T10:06:25.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.527+0000 7f0689000700 1 -- 172.21.15.12:0/285870100 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0684161fa0 con 0x7f066406ae90 2024-03-29T10:06:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.614+0000 7f067a000700 1 -- 172.21.15.12:0/285870100 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f067400bb40 con 0x7f066406ae90 2024-03-29T10:06:25.614 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2024-03-29T10:06:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.616+0000 7f0689000700 1 -- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f066406ae90 msgr2=0x7f066406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.616+0000 7f0689000700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f066406ae90 0x7f066406d350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0674014040 tx=0x7f0674008030 comp rx=0 tx=0).stop 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.617+0000 7f0689000700 1 -- 172.21.15.12:0/285870100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06840d2370 msgr2=0x7f06840ce320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.617+0000 7f0689000700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06840d2370 0x7f06840ce320 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f067000b550 tx=0x7f067000b860 comp rx=0 tx=0).stop 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.617+0000 7f0689000700 1 -- 172.21.15.12:0/285870100 shutdown_connections 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.617+0000 7f0689000700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f066406ae90 0x7f066406d350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.617+0000 7f0689000700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f068415d9d0 0x7f06840ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.617+0000 7f0689000700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06840d16a0 0x7f0684155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.617+0000 7f0689000700 1 --2- 172.21.15.12:0/285870100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06840d2370 0x7f06840ce320 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.617+0000 7f0689000700 1 -- 172.21.15.12:0/285870100 >> 172.21.15.12:0/285870100 conn(0x7f068408aff0 msgr2=0x7f068406cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.617+0000 7f0689000700 1 -- 172.21.15.12:0/285870100 shutdown_connections 2024-03-29T10:06:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.617+0000 7f0689000700 1 -- 172.21.15.12:0/285870100 wait complete. 2024-03-29T10:06:25.635 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2024-03-29T10:06:25.635 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2024-03-29T10:06:25.680 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:06:25.684 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:06:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.842+0000 7faffc400700 1 Processor -- start 2024-03-29T10:06:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.842+0000 7faffc400700 1 -- start start 2024-03-29T10:06:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.843+0000 7faffc400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7faff415dad0 0x7faff40ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.843+0000 7faffc400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff40d1630 0x7faff40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.843+0000 7faffc400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff415f820 con 0x7faff40d1fd0 2024-03-29T10:06:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.843+0000 7faffc400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff40cfcc0 con 0x7faff415dad0 2024-03-29T10:06:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.843+0000 7faffc400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff40cd9f0 con 0x7faff40d1630 2024-03-29T10:06:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.843+0000 7faffa400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7faff40d1fd0 0x7faff415deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52684/0 (socket says 172.21.15.12:52684) 2024-03-29T10:06:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faffa400700 1 -- 172.21.15.12:0/2427983668 learned_addr learned my addr 172.21.15.12:0/2427983668 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:25.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973860447 0 0) 0x7faff415f820 con 0x7faff40d1fd0 2024-03-29T10:06:25.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafdc0035b0 con 0x7faff40d1fd0 2024-03-29T10:06:25.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2689250753 0 0) 0x7fafdc0035b0 con 0x7faff40d1fd0 2024-03-29T10:06:25.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff40d1630 msgr2=0x7faff40d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:25.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff40d1630 0x7faff40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 >> v2:172.21.15.159:3300/0 conn(0x7faff415dad0 msgr2=0x7faff40ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:25.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 --2- 172.21.15.12:0/2427983668 >> v2:172.21.15.159:3300/0 conn(0x7faff415dad0 0x7faff40ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff40d2460 con 0x7faff40d1fd0 2024-03-29T10:06:25.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fafec0019f0 con 0x7faff40d1fd0 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 >> v1:172.21.15.12:6789/0 conn(0x7faff40d1fd0 legacy=0x7faff415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.844+0000 7faff3e00700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff40d1630 0x7fafdc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faff3e00700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafdc004110 0x7fafdc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faff3e00700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafdc004de0 0x7fafdc005260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff40cfcc0 con 0x7fafdc004de0 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff40cd9f0 con 0x7fafdc004110 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc00bcd0 con 0x7faff40d1630 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faffa400700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafdc004de0 0x7fafdc005260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faff9a00700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff40d1630 0x7fafdc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faff9000700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafdc004110 0x7fafdc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faffa400700 1 -- 172.21.15.12:0/2427983668 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff40d1630 msgr2=0x7fafdc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:25.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faffa400700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff40d1630 0x7fafdc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faffa400700 1 -- 172.21.15.12:0/2427983668 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafdc004110 msgr2=0x7fafdc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faffa400700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafdc004110 0x7fafdc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.845+0000 7faffa400700 1 -- 172.21.15.12:0/2427983668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc0035b0 con 0x7fafdc004de0 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.846+0000 7faffa400700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafdc004de0 0x7fafdc005260 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fafec00a840 tx=0x7fafdc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c7954db51985a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.846+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec012600 con 0x7fafdc004de0 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.846+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafec012760 con 0x7fafdc004de0 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.846+0000 7faff3e00700 1 -- 172.21.15.12:0/2427983668 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec012af0 con 0x7fafdc004de0 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.846+0000 7faffc400700 1 -- 172.21.15.12:0/2427983668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafdc004de0 msgr2=0x7fafdc005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.846+0000 7faffc400700 1 --2- 172.21.15.12:0/2427983668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafdc004de0 0x7fafdc005260 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fafec00a840 tx=0x7fafdc005880 comp rx=0 tx=0).stop 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.846+0000 7faffa400700 1 -- 172.21.15.12:0/2427983668 reap_dead start 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.846+0000 7faffc400700 1 -- 172.21.15.12:0/2427983668 shutdown_connections 2024-03-29T10:06:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.846+0000 7faffc400700 1 -- 172.21.15.12:0/2427983668 >> 172.21.15.12:0/2427983668 conn(0x7faff408aff0 msgr2=0x7faff40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.846+0000 7faffc400700 1 -- 172.21.15.12:0/2427983668 shutdown_connections 2024-03-29T10:06:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.847+0000 7faffc400700 1 -- 172.21.15.12:0/2427983668 wait complete. 2024-03-29T10:06:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.848+0000 7faffc400700 1 Processor -- start 2024-03-29T10:06:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.848+0000 7faffc400700 1 -- start start 2024-03-29T10:06:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.849+0000 7faffc400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafdc004de0 0x7faff40b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.849+0000 7faffc400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff4141400 0x7faff4146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.849+0000 7faffc400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faff4141860 0x7faff4146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.849+0000 7faffc400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff40b8ef0 con 0x7faff4141860 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.849+0000 7faffc400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff40b91d0 con 0x7fafdc004de0 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.849+0000 7faffc400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff40c99d0 con 0x7faff4141400 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faffa400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faff4141860 0x7faff4146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faffa400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faff4141860 0x7faff4146e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33092/0 (socket says 172.21.15.12:33092) 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faffa400700 1 -- 172.21.15.12:0/1023496345 learned_addr learned my addr 172.21.15.12:0/1023496345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faff9000700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff4141400 0x7faff4146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faff9a00700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafdc004de0 0x7faff40b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faffa400700 1 -- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff4141400 msgr2=0x7faff4146930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faffa400700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff4141400 0x7faff4146930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faffa400700 1 -- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafdc004de0 msgr2=0x7faff40b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faffa400700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafdc004de0 0x7faff40b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faffa400700 1 -- 172.21.15.12:0/1023496345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff40c9cb0 con 0x7faff4141860 2024-03-29T10:06:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.850+0000 7faffa400700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faff4141860 0x7faff4146e70 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fafec001cb0 tx=0x7faff406b0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.851+0000 7faff1600700 1 -- 172.21.15.12:0/1023496345 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec01e600 con 0x7faff4141860 2024-03-29T10:06:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.851+0000 7faff1600700 1 -- 172.21.15.12:0/1023496345 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafec01e760 con 0x7faff4141860 2024-03-29T10:06:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.851+0000 7faffc400700 1 -- 172.21.15.12:0/1023496345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff41f2aa0 con 0x7faff4141860 2024-03-29T10:06:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.851+0000 7faff1600700 1 -- 172.21.15.12:0/1023496345 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec01eaf0 con 0x7faff4141860 2024-03-29T10:06:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.851+0000 7faffc400700 1 -- 172.21.15.12:0/1023496345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff415b560 con 0x7faff4141860 2024-03-29T10:06:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.852+0000 7faffc400700 1 -- 172.21.15.12:0/1023496345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff4067720 con 0x7faff4141860 2024-03-29T10:06:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.853+0000 7faff1600700 1 -- 172.21.15.12:0/1023496345 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fafec019030 con 0x7faff4141860 2024-03-29T10:06:25.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.854+0000 7faff1600700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafd406aec0 0x7fafd406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:25.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.854+0000 7faff1600700 1 -- 172.21.15.12:0/1023496345 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fafec08b2f0 con 0x7faff4141860 2024-03-29T10:06:25.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.854+0000 7faff9a00700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafd406aec0 0x7fafd406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:25.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.855+0000 7faff9a00700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafd406aec0 0x7fafd406d380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fafe8004790 tx=0x7fafe80090e0 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:25.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:25.858+0000 7faff1600700 1 -- 172.21.15.12:0/1023496345 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fafec056140 con 0x7faff4141860 2024-03-29T10:06:26.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.102+0000 7faffc400700 1 -- 172.21.15.12:0/1023496345 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7faff40cf9e0 con 0x7fafd406aec0 2024-03-29T10:06:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.184+0000 7faff1600700 1 -- 172.21.15.12:0/1023496345 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fafe8004050 con 0x7fafd406aec0 2024-03-29T10:06:26.184 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2024-03-29T10:06:26.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 -- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafd406aec0 msgr2=0x7fafd406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:26.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafd406aec0 0x7fafd406d380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fafe8004790 tx=0x7fafe80090e0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 -- 172.21.15.12:0/1023496345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faff4141860 msgr2=0x7faff4146e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:26.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faff4141860 0x7faff4146e70 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fafec001cb0 tx=0x7faff406b0a0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 -- 172.21.15.12:0/1023496345 shutdown_connections 2024-03-29T10:06:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafd406aec0 0x7fafd406d380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faff4141860 0x7faff4146e70 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafdc004de0 0x7faff40b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 --2- 172.21.15.12:0/1023496345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faff4141400 0x7faff4146930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 -- 172.21.15.12:0/1023496345 >> 172.21.15.12:0/1023496345 conn(0x7faff408aff0 msgr2=0x7faff406b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 -- 172.21.15.12:0/1023496345 shutdown_connections 2024-03-29T10:06:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.187+0000 7faffc400700 1 -- 172.21.15.12:0/1023496345 wait complete. 2024-03-29T10:06:26.207 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2024-03-29T10:06:26.207 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2024-03-29T10:06:26.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.408+0000 7fa9b1800700 1 Processor -- start 2024-03-29T10:06:26.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.408+0000 7fa9b1800700 1 -- start start 2024-03-29T10:06:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.409+0000 7fa9b1800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9ac1611b0 0x7fa9ac15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.409+0000 7fa9b1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 0x7fa9ac0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.409+0000 7fa9b1800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0ce270 con 0x7fa9ac0d21f0 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.409+0000 7fa9b1800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac15b600 con 0x7fa9ac1611b0 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.409+0000 7fa9b1800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0d2760 con 0x7fa9ac15f4d0 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.410+0000 7fa9aaa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9ac1611b0 0x7fa9ac15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.410+0000 7fa9aaa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9ac1611b0 0x7fa9ac15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33104/0 (socket says 172.21.15.12:33104) 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.410+0000 7fa9aaa00700 1 -- 172.21.15.12:0/2885346009 learned_addr learned my addr 172.21.15.12:0/2885346009 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.410+0000 7fa9a3e00700 1 --2- 172.21.15.12:0/2885346009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 0x7fa9ac0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.410+0000 7fa9aaa00700 1 -- 172.21.15.12:0/2885346009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 msgr2=0x7fa9ac0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.410+0000 7fa9aaa00700 1 --2- 172.21.15.12:0/2885346009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 0x7fa9ac0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.410+0000 7fa9aaa00700 1 -- 172.21.15.12:0/2885346009 >> v1:172.21.15.159:6789/0 conn(0x7fa9ac0d21f0 legacy=0x7fa9ac161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.410+0000 7fa9aaa00700 1 -- 172.21.15.12:0/2885346009 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac0d2a40 con 0x7fa9ac1611b0 2024-03-29T10:06:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.410+0000 7fa9a3e00700 1 --2- 172.21.15.12:0/2885346009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 0x7fa9ac0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.410+0000 7fa9aaa00700 1 --2- 172.21.15.12:0/2885346009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 0x7fa9ac15ef90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0098f0 tx=0x7fa99c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dca8d5842e3927b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9aa000700 1 -- 172.21.15.12:0/2885346009 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c00b8a0 con 0x7fa9ac1611b0 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9aa000700 1 -- 172.21.15.12:0/2885346009 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99c00ba00 con 0x7fa9ac1611b0 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9aa000700 1 -- 172.21.15.12:0/2885346009 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c00bd90 con 0x7fa9ac1611b0 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9b1800700 1 -- 172.21.15.12:0/2885346009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 msgr2=0x7fa9ac15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9b1800700 1 --2- 172.21.15.12:0/2885346009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 0x7fa9ac15ef90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0098f0 tx=0x7fa99c009c00 comp rx=0 tx=0).stop 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9b1800700 1 -- 172.21.15.12:0/2885346009 shutdown_connections 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9b1800700 1 --2- 172.21.15.12:0/2885346009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 0x7fa9ac15ef90 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9b1800700 1 --2- 172.21.15.12:0/2885346009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 0x7fa9ac0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9b1800700 1 -- 172.21.15.12:0/2885346009 >> 172.21.15.12:0/2885346009 conn(0x7fa9ac08aff0 msgr2=0x7fa9ac0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9b1800700 1 -- 172.21.15.12:0/2885346009 shutdown_connections 2024-03-29T10:06:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.411+0000 7fa9b1800700 1 -- 172.21.15.12:0/2885346009 wait complete. 2024-03-29T10:06:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.413+0000 7fa9b1800700 1 Processor -- start 2024-03-29T10:06:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.413+0000 7fa9b1800700 1 -- start start 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.414+0000 7fa9b1800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9ac0d21f0 0x7fa9ac1f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.414+0000 7fa9b1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 0x7fa9ac1f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.414+0000 7fa9b1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 0x7fa9ac09d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.414+0000 7fa9b1800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0caf10 con 0x7fa9ac1611b0 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.414+0000 7fa9b1800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0ca600 con 0x7fa9ac0d21f0 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.414+0000 7fa9b1800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac09e130 con 0x7fa9ac15f4d0 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9ab400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 0x7fa9ac09d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9ab400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 0x7fa9ac09d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33106/0 (socket says 172.21.15.12:33106) 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9ab400700 1 -- 172.21.15.12:0/2713104219 learned_addr learned my addr 172.21.15.12:0/2713104219 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9a3e00700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 0x7fa9ac1f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9aaa00700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9ac0d21f0 0x7fa9ac1f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9ab400700 1 -- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 msgr2=0x7fa9ac1f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9ab400700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 0x7fa9ac1f69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9ab400700 1 -- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9ac0d21f0 msgr2=0x7fa9ac1f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9ab400700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9ac0d21f0 0x7fa9ac1f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9ab400700 1 -- 172.21.15.12:0/2713104219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c009550 con 0x7fa9ac1611b0 2024-03-29T10:06:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.415+0000 7fa9ab400700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 0x7fa9ac09d3e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400cc50 tx=0x7fa9a400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.416+0000 7fa9a2a00700 1 -- 172.21.15.12:0/2713104219 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a400ad80 con 0x7fa9ac1611b0 2024-03-29T10:06:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.416+0000 7fa9a2a00700 1 -- 172.21.15.12:0/2713104219 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a400ce50 con 0x7fa9ac1611b0 2024-03-29T10:06:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.416+0000 7fa9a2a00700 1 -- 172.21.15.12:0/2713104219 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4003c40 con 0x7fa9ac1611b0 2024-03-29T10:06:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.416+0000 7fa9b1800700 1 -- 172.21.15.12:0/2713104219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac0ca850 con 0x7fa9ac1611b0 2024-03-29T10:06:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.416+0000 7fa9b1800700 1 -- 172.21.15.12:0/2713104219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac09d920 con 0x7fa9ac1611b0 2024-03-29T10:06:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.417+0000 7fa9b1800700 1 -- 172.21.15.12:0/2713104219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac15d610 con 0x7fa9ac1611b0 2024-03-29T10:06:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.418+0000 7fa9a2a00700 1 -- 172.21.15.12:0/2713104219 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fa9a40050d0 con 0x7fa9ac1611b0 2024-03-29T10:06:26.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.418+0000 7fa9a2a00700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa98806ae90 0x7fa98806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:26.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.419+0000 7fa9a2a00700 1 -- 172.21.15.12:0/2713104219 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa9a4089890 con 0x7fa9ac1611b0 2024-03-29T10:06:26.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.419+0000 7fa9aaa00700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa98806ae90 0x7fa98806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:26.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.420+0000 7fa9aaa00700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa98806ae90 0x7fa98806d350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa99c014040 tx=0x7fa99c008030 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:26.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.423+0000 7fa9a2a00700 1 -- 172.21.15.12:0/2713104219 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa9a40546e0 con 0x7fa9ac1611b0 2024-03-29T10:06:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.665+0000 7fa9b1800700 1 -- 172.21.15.12:0/2713104219 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9ac0cdf00 con 0x7fa98806ae90 2024-03-29T10:06:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.754+0000 7fa9a2a00700 1 -- 172.21.15.12:0/2713104219 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa99c00bb70 con 0x7fa98806ae90 2024-03-29T10:06:26.754 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 -- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa98806ae90 msgr2=0x7fa98806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa98806ae90 0x7fa98806d350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa99c014040 tx=0x7fa99c008030 comp rx=0 tx=0).stop 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 -- 172.21.15.12:0/2713104219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 msgr2=0x7fa9ac09d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 0x7fa9ac09d3e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400cc50 tx=0x7fa9a400a340 comp rx=0 tx=0).stop 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 -- 172.21.15.12:0/2713104219 shutdown_connections 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa98806ae90 0x7fa98806d350 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac1611b0 0x7fa9ac09d3e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9ac0d21f0 0x7fa9ac1f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 --2- 172.21.15.12:0/2713104219 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9ac15f4d0 0x7fa9ac1f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 -- 172.21.15.12:0/2713104219 >> 172.21.15.12:0/2713104219 conn(0x7fa9ac08aff0 msgr2=0x7fa9ac15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 -- 172.21.15.12:0/2713104219 shutdown_connections 2024-03-29T10:06:26.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:26.757+0000 7fa9b1800700 1 -- 172.21.15.12:0/2713104219 wait complete. 2024-03-29T10:06:26.775 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2024-03-29T10:06:26.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:26.775 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-03-29T10:06:26.822 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:26.823 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:26.823 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:26.823 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:26.823 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:26.823 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:26.823 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:26.823 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:26.823 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:26.823 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:26.824 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-09_57_00_UTC', 'scheduled-2024-03-29-09_58_00_UTC', 'scheduled-2024-03-29-09_59_00_UTC', 'scheduled-2024-03-29-10_00_00_UTC', 'scheduled-2024-03-29-10_01_00_UTC', 'scheduled-2024-03-29-10_02_00_UTC', 'scheduled-2024-03-29-10_03_00_UTC', 'scheduled-2024-03-29-10_04_00_UTC', 'scheduled-2024-03-29-10_05_00_UTC', 'scheduled-2024-03-29-10_06_00_UTC'] 2024-03-29T10:06:26.824 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2024-03-29T10:06:27.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.026+0000 7fa0b7600700 1 Processor -- start 2024-03-29T10:06:27.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.026+0000 7fa0b7600700 1 -- start start 2024-03-29T10:06:27.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.027+0000 7fa0b7600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0b015d9d0 0x7fa0b00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.027+0000 7fa0b7600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b0161420 con 0x7fa0b00d16a0 2024-03-29T10:06:27.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.027+0000 7fa0b7600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b0161700 con 0x7fa0b00d2540 2024-03-29T10:06:27.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.027+0000 7fa0b7600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b015edc0 con 0x7fa0b015d9d0 2024-03-29T10:06:27.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.028+0000 7fa0b4c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0b015d9d0 0x7fa0b00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.028+0000 7fa0b4c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0b015d9d0 0x7fa0b00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33112/0 (socket says 172.21.15.12:33112) 2024-03-29T10:06:27.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.028+0000 7fa0b4c00700 1 -- 172.21.15.12:0/4196477846 learned_addr learned my addr 172.21.15.12:0/4196477846 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.028+0000 7fa0b4c00700 1 -- 172.21.15.12:0/4196477846 >> v1:172.21.15.159:6789/0 conn(0x7fa0b00d2540 legacy=0x7fa0b00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.028+0000 7fa0b4c00700 1 -- 172.21.15.12:0/4196477846 >> v1:172.21.15.12:6789/0 conn(0x7fa0b00d16a0 legacy=0x7fa0b00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.028+0000 7fa0b4c00700 1 -- 172.21.15.12:0/4196477846 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b01619e0 con 0x7fa0b015d9d0 2024-03-29T10:06:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.028+0000 7fa0b4c00700 1 --2- 172.21.15.12:0/4196477846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b015d9d0 0x7fa0b00d3c30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40098f0 tx=0x7fa0a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82e8519eb9287d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.028+0000 7fa0af400700 1 -- 172.21.15.12:0/4196477846 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a400b8a0 con 0x7fa0b015d9d0 2024-03-29T10:06:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.029+0000 7fa0af400700 1 -- 172.21.15.12:0/4196477846 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a400ba00 con 0x7fa0b015d9d0 2024-03-29T10:06:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.029+0000 7fa0af400700 1 -- 172.21.15.12:0/4196477846 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a400bd90 con 0x7fa0b015d9d0 2024-03-29T10:06:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.029+0000 7fa0b7600700 1 -- 172.21.15.12:0/4196477846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b015d9d0 msgr2=0x7fa0b00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.029+0000 7fa0b7600700 1 --2- 172.21.15.12:0/4196477846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b015d9d0 0x7fa0b00d3c30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40098f0 tx=0x7fa0a4009c00 comp rx=0 tx=0).stop 2024-03-29T10:06:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.029+0000 7fa0b7600700 1 -- 172.21.15.12:0/4196477846 shutdown_connections 2024-03-29T10:06:27.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.029+0000 7fa0b7600700 1 --2- 172.21.15.12:0/4196477846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b015d9d0 0x7fa0b00d3c30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.029+0000 7fa0b7600700 1 -- 172.21.15.12:0/4196477846 >> 172.21.15.12:0/4196477846 conn(0x7fa0b008aff0 msgr2=0x7fa0b00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:27.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.029+0000 7fa0b7600700 1 -- 172.21.15.12:0/4196477846 shutdown_connections 2024-03-29T10:06:27.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.029+0000 7fa0b7600700 1 -- 172.21.15.12:0/4196477846 wait complete. 2024-03-29T10:06:27.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.031+0000 7fa0b7600700 1 Processor -- start 2024-03-29T10:06:27.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.031+0000 7fa0b7600700 1 -- start start 2024-03-29T10:06:27.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0b7600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0b00d16a0 0x7fa0b00b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0b7600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b00d2540 0x7fa0b01e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0b7600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0b015d9d0 0x7fa0b01e77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0b7600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00b3980 con 0x7fa0b00d2540 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0b7600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00b7b20 con 0x7fa0b00d16a0 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0b7600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00b3050 con 0x7fa0b015d9d0 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0afe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b00d2540 0x7fa0b01e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0b4c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0b00d16a0 0x7fa0b00b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0afe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b00d2540 0x7fa0b01e7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33122/0 (socket says 172.21.15.12:33122) 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0afe00700 1 -- 172.21.15.12:0/3434879663 learned_addr learned my addr 172.21.15.12:0/3434879663 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.032+0000 7fa0b5600700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0b015d9d0 0x7fa0b01e77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.033+0000 7fa0afe00700 1 -- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0b015d9d0 msgr2=0x7fa0b01e77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.033+0000 7fa0afe00700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0b015d9d0 0x7fa0b01e77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.033+0000 7fa0afe00700 1 -- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0b00d16a0 msgr2=0x7fa0b00b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.033+0000 7fa0afe00700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0b00d16a0 0x7fa0b00b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.033+0000 7fa0afe00700 1 -- 172.21.15.12:0/3434879663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a4009550 con 0x7fa0b00d2540 2024-03-29T10:06:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.033+0000 7fa0afe00700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b00d2540 0x7fa0b01e7280 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa09c001eb0 tx=0x7fa09c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:27.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.033+0000 7fa0acc00700 1 -- 172.21.15.12:0/3434879663 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c0054e0 con 0x7fa0b00d2540 2024-03-29T10:06:27.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.033+0000 7fa0acc00700 1 -- 172.21.15.12:0/3434879663 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09c003680 con 0x7fa0b00d2540 2024-03-29T10:06:27.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.033+0000 7fa0acc00700 1 -- 172.21.15.12:0/3434879663 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c0039f0 con 0x7fa0b00d2540 2024-03-29T10:06:27.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.033+0000 7fa0b7600700 1 -- 172.21.15.12:0/3434879663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b00b3330 con 0x7fa0b00d2540 2024-03-29T10:06:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.034+0000 7fa0b7600700 1 -- 172.21.15.12:0/3434879663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b00b4350 con 0x7fa0b00d2540 2024-03-29T10:06:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.034+0000 7fa0b7600700 1 -- 172.21.15.12:0/3434879663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b00d1ca0 con 0x7fa0b00d2540 2024-03-29T10:06:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.035+0000 7fa0acc00700 1 -- 172.21.15.12:0/3434879663 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fa09c003b50 con 0x7fa0b00d2540 2024-03-29T10:06:27.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.036+0000 7fa0acc00700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa09006ae90 0x7fa09006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.036+0000 7fa0acc00700 1 -- 172.21.15.12:0/3434879663 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa09c0889a0 con 0x7fa0b00d2540 2024-03-29T10:06:27.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.037+0000 7fa0b4c00700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa09006ae90 0x7fa09006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.038+0000 7fa0b4c00700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa09006ae90 0x7fa09006d350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009840 tx=0x7fa0a4008030 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.041+0000 7fa0acc00700 1 -- 172.21.15.12:0/3434879663 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa09c0538a0 con 0x7fa0b00d2540 2024-03-29T10:06:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.282+0000 7fa0b7600700 1 -- 172.21.15.12:0/3434879663 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa0b0161fa0 con 0x7fa09006ae90 2024-03-29T10:06:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.292+0000 7fa0acc00700 1 -- 172.21.15.12:0/3434879663 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fa0a400bbf0 con 0x7fa09006ae90 2024-03-29T10:06:27.292 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:27.292 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:53", "first": "2024-03-29T09:57:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 10, "pruned_count": 0, "active": false}] 2024-03-29T10:06:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 -- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa09006ae90 msgr2=0x7fa09006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa09006ae90 0x7fa09006d350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009840 tx=0x7fa0a4008030 comp rx=0 tx=0).stop 2024-03-29T10:06:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 -- 172.21.15.12:0/3434879663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b00d2540 msgr2=0x7fa0b01e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b00d2540 0x7fa0b01e7280 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa09c001eb0 tx=0x7fa09c00c7b0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 -- 172.21.15.12:0/3434879663 shutdown_connections 2024-03-29T10:06:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa09006ae90 0x7fa09006d350 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0b015d9d0 0x7fa0b01e77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0b00d16a0 0x7fa0b00b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 --2- 172.21.15.12:0/3434879663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b00d2540 0x7fa0b01e7280 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 -- 172.21.15.12:0/3434879663 >> 172.21.15.12:0/3434879663 conn(0x7fa0b008aff0 msgr2=0x7fa0b015ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 -- 172.21.15.12:0/3434879663 shutdown_connections 2024-03-29T10:06:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.294+0000 7fa0b7600700 1 -- 172.21.15.12:0/3434879663 wait complete. 2024-03-29T10:06:27.312 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:53", "first": "2024-03-29T09:57:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 10, "pruned_count": 0, "active": false}] 2024-03-29T10:06:27.312 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T09:55:53', 'first': '2024-03-29T09:57:00', 'last': '2024-03-29T10:06:00', 'last_pruned': None, 'created_count': 10, 'pruned_count': 0, 'active': False} 2024-03-29T10:06:27.312 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 10 2024-03-29T10:06:27.312 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 10 2024-03-29T10:06:27.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:27.313 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-03-29T10:06:27.358 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:27.358 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:27.358 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:27.358 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:27.358 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:27.358 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:27.358 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:27.359 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:27.359 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:27.359 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:27.359 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:27.360 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-09_56_00_UTC', 'scheduled-2024-03-29-09_57_00_UTC', 'scheduled-2024-03-29-09_58_00_UTC', 'scheduled-2024-03-29-09_59_00_UTC', 'scheduled-2024-03-29-10_00_00_UTC', 'scheduled-2024-03-29-10_01_00_UTC', 'scheduled-2024-03-29-10_02_00_UTC', 'scheduled-2024-03-29-10_03_00_UTC', 'scheduled-2024-03-29-10_04_00_UTC', 'scheduled-2024-03-29-10_05_00_UTC', 'scheduled-2024-03-29-10_06_00_UTC'] 2024-03-29T10:06:27.360 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2024-03-29T10:06:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.600+0000 7f0d91e00700 1 Processor -- start 2024-03-29T10:06:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.600+0000 7f0d91e00700 1 -- start start 2024-03-29T10:06:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.601+0000 7f0d91e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0d8c0d21f0 0x7f0d8c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.602+0000 7f0d91e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c0ce270 con 0x7f0d8c1611b0 2024-03-29T10:06:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.602+0000 7f0d91e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c15b600 con 0x7f0d8c0cec40 2024-03-29T10:06:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.602+0000 7f0d91e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d8c0d2760 con 0x7f0d8c0d21f0 2024-03-29T10:06:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.602+0000 7f0d8aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0d8c1611b0 0x7f0d8c15ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52716/0 (socket says 172.21.15.12:52716) 2024-03-29T10:06:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.602+0000 7f0d8aa00700 1 -- 172.21.15.12:0/2629670450 learned_addr learned my addr 172.21.15.12:0/2629670450 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.602+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469103533 0 0) 0x7f0d8c0ce270 con 0x7f0d8c1611b0 2024-03-29T10:06:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.602+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d700035b0 con 0x7f0d8c1611b0 2024-03-29T10:06:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.602+0000 7f0d8a000700 1 --2- 172.21.15.12:0/2629670450 >> v2:172.21.15.159:3300/0 conn(0x7f0d8c0d21f0 0x7f0d8c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2766597018 0 0) 0x7f0d700035b0 con 0x7f0d8c1611b0 2024-03-29T10:06:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 >> v2:172.21.15.159:3300/0 conn(0x7f0d8c0d21f0 msgr2=0x7f0d8c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 --2- 172.21.15.12:0/2629670450 >> v2:172.21.15.159:3300/0 conn(0x7f0d8c0d21f0 0x7f0d8c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 >> v1:172.21.15.159:6789/0 conn(0x7f0d8c0cec40 legacy=0x7f0d8c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c0d2a40 con 0x7f0d8c1611b0 2024-03-29T10:06:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0d7c001670 con 0x7f0d8c1611b0 2024-03-29T10:06:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 >> v1:172.21.15.12:6789/0 conn(0x7f0d8c1611b0 legacy=0x7f0d8c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d70003a00 0x7f0d70003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d70004850 0x7f0d70004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d70005210 0x7f0d7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c0d2760 con 0x7f0d70005210 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7000c160 con 0x7f0d70004850 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7000c320 con 0x7f0d70003a00 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8a000700 1 --2- 172.21.15.12:0/2629670450 >> v2:172.21.15.159:3300/0 conn(0x7f0d8c0d21f0 0x7f0d8c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8b400700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d70005210 0x7f0d7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8aa00700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d70003a00 0x7f0d70003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8a000700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d70004850 0x7f0d70004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8b400700 1 -- 172.21.15.12:0/2629670450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d70003a00 msgr2=0x7f0d70003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8b400700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d70003a00 0x7f0d70003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8b400700 1 -- 172.21.15.12:0/2629670450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d70004850 msgr2=0x7f0d70004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8b400700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d70004850 0x7f0d70004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8b400700 1 -- 172.21.15.12:0/2629670450 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d700035b0 con 0x7f0d70005210 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8a000700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d70004850 0x7f0d70004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.603+0000 7f0d8aa00700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d70003a00 0x7f0d70003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.604+0000 7f0d8b400700 1 -- 172.21.15.12:0/2629670450 reap_dead start 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.604+0000 7f0d8b400700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d70005210 0x7f0d7000ba10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c240 tx=0x7f0d70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1abcddf9602f33b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.604+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d84001df0 con 0x7f0d70005210 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.604+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8400cdc0 con 0x7f0d70005210 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.604+0000 7f0d89600700 1 -- 172.21.15.12:0/2629670450 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d84003c40 con 0x7f0d70005210 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.604+0000 7f0d91e00700 1 -- 172.21.15.12:0/2629670450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d70005210 msgr2=0x7f0d7000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.604+0000 7f0d91e00700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d70005210 0x7f0d7000ba10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c240 tx=0x7f0d70005880 comp rx=0 tx=0).stop 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.604+0000 7f0d91e00700 1 -- 172.21.15.12:0/2629670450 shutdown_connections 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.604+0000 7f0d91e00700 1 --2- 172.21.15.12:0/2629670450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d70005210 0x7f0d7000ba10 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.604+0000 7f0d91e00700 1 -- 172.21.15.12:0/2629670450 >> 172.21.15.12:0/2629670450 conn(0x7f0d8c08aff0 msgr2=0x7f0d8c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.605+0000 7f0d91e00700 1 -- 172.21.15.12:0/2629670450 shutdown_connections 2024-03-29T10:06:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.605+0000 7f0d91e00700 1 -- 172.21.15.12:0/2629670450 wait complete. 2024-03-29T10:06:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.606+0000 7f0d91e00700 1 Processor -- start 2024-03-29T10:06:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.606+0000 7f0d91e00700 1 -- start start 2024-03-29T10:06:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.607+0000 7f0d91e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d70005210 0x7f0d8c0caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.607+0000 7f0d91e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d8c0c96b0 0x7f0d8c0c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.607+0000 7f0d91e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d8c0b6150 0x7f0d8c0b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.607+0000 7f0d91e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c14b1f0 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.607+0000 7f0d91e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c0ca920 con 0x7f0d8c0b6150 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.607+0000 7f0d91e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c0c88f0 con 0x7f0d70005210 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d8c0c96b0 0x7f0d8c0c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d8c0c96b0 0x7f0d8c0c9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33140/0 (socket says 172.21.15.12:33140) 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8a000700 1 -- 172.21.15.12:0/274442771 learned_addr learned my addr 172.21.15.12:0/274442771 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8aa00700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d70005210 0x7f0d8c0caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8b400700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d8c0b6150 0x7f0d8c0b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8a000700 1 -- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d70005210 msgr2=0x7f0d8c0caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8a000700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d70005210 0x7f0d8c0caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8a000700 1 -- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d8c0b6150 msgr2=0x7f0d8c0b65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8a000700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d8c0b6150 0x7f0d8c0b65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8a000700 1 -- 172.21.15.12:0/274442771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c0c8bd0 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8b400700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d8c0b6150 0x7f0d8c0b65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.608+0000 7f0d8a000700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d8c0c96b0 0x7f0d8c0c9b30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800f510 tx=0x7f0d7800f8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.609+0000 7f0d83400700 1 -- 172.21.15.12:0/274442771 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78014870 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.609+0000 7f0d83400700 1 -- 172.21.15.12:0/274442771 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7800c050 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.609+0000 7f0d91e00700 1 -- 172.21.15.12:0/274442771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c0c8eb0 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.609+0000 7f0d83400700 1 -- 172.21.15.12:0/274442771 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78014bd0 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.609+0000 7f0d91e00700 1 -- 172.21.15.12:0/274442771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c0cb1e0 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.610+0000 7f0d91e00700 1 -- 172.21.15.12:0/274442771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8c0c9500 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.611+0000 7f0d83400700 1 -- 172.21.15.12:0/274442771 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f0d78014d30 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.612+0000 7f0d83400700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d6c06ae90 0x7f0d6c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.612+0000 7f0d83400700 1 -- 172.21.15.12:0/274442771 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f0d78089f60 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.612+0000 7f0d8aa00700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d6c06ae90 0x7f0d6c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.613+0000 7f0d8aa00700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d6c06ae90 0x7f0d6c06d350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c000f60 tx=0x7f0d7c009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.616+0000 7f0d83400700 1 -- 172.21.15.12:0/274442771 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0d78054ca0 con 0x7f0d8c0c96b0 2024-03-29T10:06:27.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.855+0000 7f0d91e00700 1 -- 172.21.15.12:0/274442771 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d8c0cf050 con 0x7f0d6c06ae90 2024-03-29T10:06:27.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.865+0000 7f0d83400700 1 -- 172.21.15.12:0/274442771 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f0d7c001630 con 0x7f0d6c06ae90 2024-03-29T10:06:27.865 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:27.865 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:54", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:27.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 -- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d6c06ae90 msgr2=0x7f0d6c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d6c06ae90 0x7f0d6c06d350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c000f60 tx=0x7f0d7c009040 comp rx=0 tx=0).stop 2024-03-29T10:06:27.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 -- 172.21.15.12:0/274442771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d8c0c96b0 msgr2=0x7f0d8c0c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:27.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d8c0c96b0 0x7f0d8c0c9b30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800f510 tx=0x7f0d7800f8d0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 -- 172.21.15.12:0/274442771 shutdown_connections 2024-03-29T10:06:27.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d6c06ae90 0x7f0d6c06d350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d8c0b6150 0x7f0d8c0b65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d70005210 0x7f0d8c0caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 --2- 172.21.15.12:0/274442771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d8c0c96b0 0x7f0d8c0c9b30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:27.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 -- 172.21.15.12:0/274442771 >> 172.21.15.12:0/274442771 conn(0x7f0d8c08aff0 msgr2=0x7f0d8c06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:27.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 -- 172.21.15.12:0/274442771 shutdown_connections 2024-03-29T10:06:27.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:27.868+0000 7f0d91e00700 1 -- 172.21.15.12:0/274442771 wait complete. 2024-03-29T10:06:27.886 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:54", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:27.887 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T09:55:54', 'first': '2024-03-29T09:56:00', 'last': '2024-03-29T10:06:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-29T10:06:27.887 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-29T10:06:27.887 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-29T10:06:27.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:27.887 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-03-29T10:06:27.934 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:27.934 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:27.934 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:27.934 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:27.934 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:27.935 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:27.935 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:27.935 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:27.935 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:27.935 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:27.935 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:27.936 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-09_56_00_UTC', 'scheduled-2024-03-29-09_57_00_UTC', 'scheduled-2024-03-29-09_58_00_UTC', 'scheduled-2024-03-29-09_59_00_UTC', 'scheduled-2024-03-29-10_00_00_UTC', 'scheduled-2024-03-29-10_01_00_UTC', 'scheduled-2024-03-29-10_02_00_UTC', 'scheduled-2024-03-29-10_03_00_UTC', 'scheduled-2024-03-29-10_04_00_UTC', 'scheduled-2024-03-29-10_05_00_UTC', 'scheduled-2024-03-29-10_06_00_UTC'] 2024-03-29T10:06:27.936 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2024-03-29T10:06:28.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.136+0000 7f0929e00700 1 Processor -- start 2024-03-29T10:06:28.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.136+0000 7f0929e00700 1 -- start start 2024-03-29T10:06:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.137+0000 7f0929e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f092415d9d0 0x7f09240d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.137+0000 7f0929e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09240d2540 0x7f09240ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.137+0000 7f0929e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0924161420 con 0x7f09240d16a0 2024-03-29T10:06:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.137+0000 7f0929e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f092415edc0 con 0x7f092415d9d0 2024-03-29T10:06:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.137+0000 7f0929e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0924161700 con 0x7f09240d2540 2024-03-29T10:06:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.137+0000 7f0922a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f092415d9d0 0x7f09240d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.137+0000 7f0922a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f092415d9d0 0x7f09240d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33156/0 (socket says 172.21.15.12:33156) 2024-03-29T10:06:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.137+0000 7f0922a00700 1 -- 172.21.15.12:0/2175254836 learned_addr learned my addr 172.21.15.12:0/2175254836 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.138+0000 7f0923400700 1 --2- 172.21.15.12:0/2175254836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09240d2540 0x7f09240ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.138+0000 7f0922a00700 1 -- 172.21.15.12:0/2175254836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09240d2540 msgr2=0x7f09240ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.138+0000 7f0922a00700 1 --2- 172.21.15.12:0/2175254836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09240d2540 0x7f09240ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.138+0000 7f0922a00700 1 -- 172.21.15.12:0/2175254836 >> v1:172.21.15.12:6789/0 conn(0x7f09240d16a0 legacy=0x7f09240d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.138+0000 7f0922a00700 1 -- 172.21.15.12:0/2175254836 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09241619e0 con 0x7f092415d9d0 2024-03-29T10:06:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.138+0000 7f0922a00700 1 --2- 172.21.15.12:0/2175254836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f092415d9d0 0x7f09240d3c30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f09140098f0 tx=0x7f0914009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239f2ba1bf56e905 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.138+0000 7f0921600700 1 -- 172.21.15.12:0/2175254836 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091400b8a0 con 0x7f092415d9d0 2024-03-29T10:06:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.138+0000 7f0921600700 1 -- 172.21.15.12:0/2175254836 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091400ba00 con 0x7f092415d9d0 2024-03-29T10:06:28.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.138+0000 7f0921600700 1 -- 172.21.15.12:0/2175254836 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091400bd90 con 0x7f092415d9d0 2024-03-29T10:06:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.139+0000 7f0929e00700 1 -- 172.21.15.12:0/2175254836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f092415d9d0 msgr2=0x7f09240d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.139+0000 7f0929e00700 1 --2- 172.21.15.12:0/2175254836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f092415d9d0 0x7f09240d3c30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f09140098f0 tx=0x7f0914009c00 comp rx=0 tx=0).stop 2024-03-29T10:06:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.139+0000 7f0929e00700 1 -- 172.21.15.12:0/2175254836 shutdown_connections 2024-03-29T10:06:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.139+0000 7f0929e00700 1 --2- 172.21.15.12:0/2175254836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09240d2540 0x7f09240ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.139+0000 7f0929e00700 1 --2- 172.21.15.12:0/2175254836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f092415d9d0 0x7f09240d3c30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.139+0000 7f0929e00700 1 -- 172.21.15.12:0/2175254836 >> 172.21.15.12:0/2175254836 conn(0x7f092408aff0 msgr2=0x7f09240d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.139+0000 7f0929e00700 1 -- 172.21.15.12:0/2175254836 shutdown_connections 2024-03-29T10:06:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.139+0000 7f0929e00700 1 -- 172.21.15.12:0/2175254836 wait complete. 2024-03-29T10:06:28.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.141+0000 7f0929e00700 1 Processor -- start 2024-03-29T10:06:28.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.141+0000 7f0929e00700 1 -- start start 2024-03-29T10:06:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0929e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09240d16a0 0x7f09240b8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0929e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09240d2540 0x7f092414fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0929e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f092415d9d0 0x7f09241e8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0929e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092414ff50 con 0x7f09240d2540 2024-03-29T10:06:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0929e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09240b73c0 con 0x7f09240d16a0 2024-03-29T10:06:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0929e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09241e89a0 con 0x7f092415d9d0 2024-03-29T10:06:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0922000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09240d2540 0x7f092414fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0922000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09240d2540 0x7f092414fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33170/0 (socket says 172.21.15.12:33170) 2024-03-29T10:06:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0922000700 1 -- 172.21.15.12:0/1791507595 learned_addr learned my addr 172.21.15.12:0/1791507595 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0923400700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f092415d9d0 0x7f09241e8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.142+0000 7f0922a00700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09240d16a0 0x7f09240b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f0922000700 1 -- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f092415d9d0 msgr2=0x7f09241e8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f0922000700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f092415d9d0 0x7f09241e8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f0922000700 1 -- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09240d16a0 msgr2=0x7f09240b8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f0922000700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09240d16a0 0x7f09240b8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f0922000700 1 -- 172.21.15.12:0/1791507595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0914009550 con 0x7f09240d2540 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f0922a00700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09240d16a0 0x7f09240b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f0922000700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09240d2540 0x7f092414fa10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f091000c480 tx=0x7f091000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f091b400700 1 -- 172.21.15.12:0/1791507595 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09100054e0 con 0x7f09240d2540 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f091b400700 1 -- 172.21.15.12:0/1791507595 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0910003680 con 0x7f09240d2540 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f0929e00700 1 -- 172.21.15.12:0/1791507595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09240b6ad0 con 0x7f09240d2540 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f091b400700 1 -- 172.21.15.12:0/1791507595 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910003a30 con 0x7f09240d2540 2024-03-29T10:06:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.143+0000 7f0929e00700 1 -- 172.21.15.12:0/1791507595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09240b7050 con 0x7f09240d2540 2024-03-29T10:06:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.144+0000 7f0929e00700 1 -- 172.21.15.12:0/1791507595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09240d1ca0 con 0x7f09240d2540 2024-03-29T10:06:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.146+0000 7f091b400700 1 -- 172.21.15.12:0/1791507595 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f0910003b90 con 0x7f09240d2540 2024-03-29T10:06:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.146+0000 7f091b400700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f090406ae90 0x7f090406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.147+0000 7f091b400700 1 -- 172.21.15.12:0/1791507595 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f0910088650 con 0x7f09240d2540 2024-03-29T10:06:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.147+0000 7f0922a00700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f090406ae90 0x7f090406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.148+0000 7f0922a00700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f090406ae90 0x7f090406d350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0914014040 tx=0x7f0914008030 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:28.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.151+0000 7f091b400700 1 -- 172.21.15.12:0/1791507595 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0910053540 con 0x7f09240d2540 2024-03-29T10:06:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.391+0000 7f0929e00700 1 -- 172.21.15.12:0/1791507595 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0924161fa0 con 0x7f090406ae90 2024-03-29T10:06:28.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.401+0000 7f091b400700 1 -- 172.21.15.12:0/1791507595 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f091400bb70 con 0x7f090406ae90 2024-03-29T10:06:28.401 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:28.401 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:54", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:28.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.403+0000 7f0929e00700 1 -- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f090406ae90 msgr2=0x7f090406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.403+0000 7f0929e00700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f090406ae90 0x7f090406d350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0914014040 tx=0x7f0914008030 comp rx=0 tx=0).stop 2024-03-29T10:06:28.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.403+0000 7f0929e00700 1 -- 172.21.15.12:0/1791507595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09240d2540 msgr2=0x7f092414fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.403+0000 7f0929e00700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09240d2540 0x7f092414fa10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f091000c480 tx=0x7f091000c790 comp rx=0 tx=0).stop 2024-03-29T10:06:28.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.403+0000 7f0929e00700 1 -- 172.21.15.12:0/1791507595 shutdown_connections 2024-03-29T10:06:28.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.403+0000 7f0929e00700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f090406ae90 0x7f090406d350 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.403+0000 7f0929e00700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f092415d9d0 0x7f09241e8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.403+0000 7f0929e00700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09240d16a0 0x7f09240b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.403+0000 7f0929e00700 1 --2- 172.21.15.12:0/1791507595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09240d2540 0x7f092414fa10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.403+0000 7f0929e00700 1 -- 172.21.15.12:0/1791507595 >> 172.21.15.12:0/1791507595 conn(0x7f092408aff0 msgr2=0x7f09240b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:28.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.404+0000 7f0929e00700 1 -- 172.21.15.12:0/1791507595 shutdown_connections 2024-03-29T10:06:28.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.404+0000 7f0929e00700 1 -- 172.21.15.12:0/1791507595 wait complete. 2024-03-29T10:06:28.421 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:54", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:28.421 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T09:55:54', 'first': '2024-03-29T09:56:00', 'last': '2024-03-29T10:06:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-29T10:06:28.422 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-29T10:06:28.422 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-29T10:06:28.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:28.422 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-03-29T10:06:28.468 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:28.468 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:28.468 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:28.468 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:28.468 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:28.468 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:28.469 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:28.469 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:28.469 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:28.469 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:28.469 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:28.470 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-09_56_00_UTC', 'scheduled-2024-03-29-09_57_00_UTC', 'scheduled-2024-03-29-09_58_00_UTC', 'scheduled-2024-03-29-09_59_00_UTC', 'scheduled-2024-03-29-10_00_00_UTC', 'scheduled-2024-03-29-10_01_00_UTC', 'scheduled-2024-03-29-10_02_00_UTC', 'scheduled-2024-03-29-10_03_00_UTC', 'scheduled-2024-03-29-10_04_00_UTC', 'scheduled-2024-03-29-10_05_00_UTC', 'scheduled-2024-03-29-10_06_00_UTC'] 2024-03-29T10:06:28.470 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2024-03-29T10:06:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.670+0000 7f21a8800700 1 Processor -- start 2024-03-29T10:06:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.670+0000 7f21a8800700 1 -- start start 2024-03-29T10:06:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.671+0000 7f21a8800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21a00cee10 0x7f21a0161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.672+0000 7f21a8800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a00ce270 con 0x7f21a01611b0 2024-03-29T10:06:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.672+0000 7f21a8800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a015b600 con 0x7f21a00d21f0 2024-03-29T10:06:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.672+0000 7f21a8800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a00d2760 con 0x7f21a00cee10 2024-03-29T10:06:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.672+0000 7f21a6800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21a00cee10 0x7f21a0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.672+0000 7f21a6800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21a00cee10 0x7f21a0161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33186/0 (socket says 172.21.15.12:33186) 2024-03-29T10:06:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.672+0000 7f21a6800700 1 -- 172.21.15.12:0/236061776 learned_addr learned my addr 172.21.15.12:0/236061776 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.672+0000 7f21a6800700 1 -- 172.21.15.12:0/236061776 >> v1:172.21.15.159:6789/0 conn(0x7f21a00d21f0 legacy=0x7f21a00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.672+0000 7f21a6800700 1 -- 172.21.15.12:0/236061776 >> v1:172.21.15.12:6789/0 conn(0x7f21a01611b0 legacy=0x7f21a015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.672+0000 7f21a6800700 1 -- 172.21.15.12:0/236061776 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a00d2a40 con 0x7f21a00cee10 2024-03-29T10:06:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.672+0000 7f21a6800700 1 --2- 172.21.15.12:0/236061776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00cee10 0x7f21a0161590 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f219c009e50 tx=0x7f219c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af1af18b67727fd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.673+0000 7f21a4a00700 1 -- 172.21.15.12:0/236061776 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c00e070 con 0x7f21a00cee10 2024-03-29T10:06:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.673+0000 7f21a4a00700 1 -- 172.21.15.12:0/236061776 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219c003a20 con 0x7f21a00cee10 2024-03-29T10:06:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.673+0000 7f21a4a00700 1 -- 172.21.15.12:0/236061776 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c003dd0 con 0x7f21a00cee10 2024-03-29T10:06:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.673+0000 7f21a8800700 1 -- 172.21.15.12:0/236061776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00cee10 msgr2=0x7f21a0161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.673+0000 7f21a8800700 1 --2- 172.21.15.12:0/236061776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00cee10 0x7f21a0161590 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f219c009e50 tx=0x7f219c00b330 comp rx=0 tx=0).stop 2024-03-29T10:06:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.673+0000 7f21a8800700 1 -- 172.21.15.12:0/236061776 shutdown_connections 2024-03-29T10:06:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.673+0000 7f21a8800700 1 --2- 172.21.15.12:0/236061776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00cee10 0x7f21a0161590 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.673+0000 7f21a8800700 1 -- 172.21.15.12:0/236061776 >> 172.21.15.12:0/236061776 conn(0x7f21a008aff0 msgr2=0x7f21a00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.673+0000 7f21a8800700 1 -- 172.21.15.12:0/236061776 shutdown_connections 2024-03-29T10:06:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.673+0000 7f21a8800700 1 -- 172.21.15.12:0/236061776 wait complete. 2024-03-29T10:06:28.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.675+0000 7f21a8800700 1 Processor -- start 2024-03-29T10:06:28.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.675+0000 7f21a8800700 1 -- start start 2024-03-29T10:06:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.676+0000 7f21a8800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21a00cee10 0x7f21a00ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.676+0000 7f21a8800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00d21f0 0x7f21a00cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.676+0000 7f21a8800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21a01611b0 0x7f21a00cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.676+0000 7f21a8800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a0151270 con 0x7f21a00d21f0 2024-03-29T10:06:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.676+0000 7f21a8800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a01f3420 con 0x7f21a00cee10 2024-03-29T10:06:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.676+0000 7f21a8800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00cb1c0 con 0x7f21a01611b0 2024-03-29T10:06:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.676+0000 7f21a5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00d21f0 0x7f21a00cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.676+0000 7f21a5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00d21f0 0x7f21a00cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33198/0 (socket says 172.21.15.12:33198) 2024-03-29T10:06:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.676+0000 7f21a5400700 1 -- 172.21.15.12:0/1672416957 learned_addr learned my addr 172.21.15.12:0/1672416957 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f21a5e00700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21a00cee10 0x7f21a00ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f21a6800700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21a01611b0 0x7f21a00cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f21a5400700 1 -- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21a01611b0 msgr2=0x7f21a00cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f21a5400700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21a01611b0 0x7f21a00cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f21a5400700 1 -- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21a00cee10 msgr2=0x7f21a00ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f21a5400700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21a00cee10 0x7f21a00ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f21a5400700 1 -- 172.21.15.12:0/1672416957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c009b30 con 0x7f21a00d21f0 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f21a5e00700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21a00cee10 0x7f21a00ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f21a5400700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00d21f0 0x7f21a00cab40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f219000c4d0 tx=0x7f219000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f219a000700 1 -- 172.21.15.12:0/1672416957 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2190011790 con 0x7f21a00d21f0 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f219a000700 1 -- 172.21.15.12:0/1672416957 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21900118f0 con 0x7f21a00d21f0 2024-03-29T10:06:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f21a8800700 1 -- 172.21.15.12:0/1672416957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a00cb4a0 con 0x7f21a00d21f0 2024-03-29T10:06:28.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.677+0000 7f219a000700 1 -- 172.21.15.12:0/1672416957 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2190011ca0 con 0x7f21a00d21f0 2024-03-29T10:06:28.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.678+0000 7f21a8800700 1 -- 172.21.15.12:0/1672416957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a00cba60 con 0x7f21a00d21f0 2024-03-29T10:06:28.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.679+0000 7f21a8800700 1 -- 172.21.15.12:0/1672416957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a0067720 con 0x7f21a00d21f0 2024-03-29T10:06:28.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.680+0000 7f219a000700 1 -- 172.21.15.12:0/1672416957 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f219001c460 con 0x7f21a00d21f0 2024-03-29T10:06:28.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.681+0000 7f219a000700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f218406ae90 0x7f218406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:28.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.682+0000 7f219a000700 1 -- 172.21.15.12:0/1672416957 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f21900895c0 con 0x7f21a00d21f0 2024-03-29T10:06:28.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.682+0000 7f21a5e00700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f218406ae90 0x7f218406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:28.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.683+0000 7f21a5e00700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f218406ae90 0x7f218406d350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f21a0159ba0 tx=0x7f2194009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:28.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.686+0000 7f219a000700 1 -- 172.21.15.12:0/1672416957 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2190054420 con 0x7f21a00d21f0 2024-03-29T10:06:28.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.928+0000 7f21a8800700 1 -- 172.21.15.12:0/1672416957 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21a015f6a0 con 0x7f218406ae90 2024-03-29T10:06:28.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.938+0000 7f219a000700 1 -- 172.21.15.12:0/1672416957 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f21940050d0 con 0x7f218406ae90 2024-03-29T10:06:28.938 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:28.938 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:55", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:28.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 -- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f218406ae90 msgr2=0x7f218406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f218406ae90 0x7f218406d350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f21a0159ba0 tx=0x7f2194009040 comp rx=0 tx=0).stop 2024-03-29T10:06:28.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 -- 172.21.15.12:0/1672416957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00d21f0 msgr2=0x7f21a00cab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:28.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00d21f0 0x7f21a00cab40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f219000c4d0 tx=0x7f219000c890 comp rx=0 tx=0).stop 2024-03-29T10:06:28.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 -- 172.21.15.12:0/1672416957 shutdown_connections 2024-03-29T10:06:28.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f218406ae90 0x7f218406d350 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21a01611b0 0x7f21a00cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21a00cee10 0x7f21a00ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 --2- 172.21.15.12:0/1672416957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a00d21f0 0x7f21a00cab40 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:28.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 -- 172.21.15.12:0/1672416957 >> 172.21.15.12:0/1672416957 conn(0x7f21a008aff0 msgr2=0x7f21a006c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:28.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 -- 172.21.15.12:0/1672416957 shutdown_connections 2024-03-29T10:06:28.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:28.940+0000 7f21a8800700 1 -- 172.21.15.12:0/1672416957 wait complete. 2024-03-29T10:06:28.958 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:55", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:28.958 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T09:55:55', 'first': '2024-03-29T09:56:00', 'last': '2024-03-29T10:06:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-29T10:06:28.958 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-29T10:06:28.958 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-29T10:06:28.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:28.958 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-03-29T10:06:29.004 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:29.004 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:29.004 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:29.004 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:29.004 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:29.004 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:29.005 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:29.005 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:29.005 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:29.005 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:29.006 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:29.007 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-09_56_00_UTC', 'scheduled-2024-03-29-09_57_00_UTC', 'scheduled-2024-03-29-09_58_00_UTC', 'scheduled-2024-03-29-09_59_00_UTC', 'scheduled-2024-03-29-10_00_00_UTC', 'scheduled-2024-03-29-10_01_00_UTC', 'scheduled-2024-03-29-10_02_00_UTC', 'scheduled-2024-03-29-10_03_00_UTC', 'scheduled-2024-03-29-10_04_00_UTC', 'scheduled-2024-03-29-10_05_00_UTC', 'scheduled-2024-03-29-10_06_00_UTC'] 2024-03-29T10:06:29.007 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2024-03-29T10:06:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.248+0000 7f5ebf000700 1 Processor -- start 2024-03-29T10:06:29.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.248+0000 7f5ebf000700 1 -- start start 2024-03-29T10:06:29.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.249+0000 7f5ebf000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5eb815a5c0 0x7f5eb80709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:29.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.249+0000 7f5ebf000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80cb310 con 0x7f5eb815dfe0 2024-03-29T10:06:29.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.249+0000 7f5ebf000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb8070f00 con 0x7f5eb80ce840 2024-03-29T10:06:29.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.249+0000 7f5ebf000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb80711a0 con 0x7f5eb815a5c0 2024-03-29T10:06:29.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.250+0000 7f5eb7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5eb815a5c0 0x7f5eb80709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:29.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.250+0000 7f5eb7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5eb815a5c0 0x7f5eb80709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53060/0 (socket says 172.21.15.12:53060) 2024-03-29T10:06:29.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.250+0000 7f5eb7e00700 1 -- 172.21.15.12:0/4230067409 learned_addr learned my addr 172.21.15.12:0/4230067409 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.250+0000 7f5eb7e00700 1 -- 172.21.15.12:0/4230067409 >> v1:172.21.15.159:6789/0 conn(0x7f5eb80ce840 legacy=0x7f5eb80cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.250+0000 7f5eb7e00700 1 -- 172.21.15.12:0/4230067409 >> v1:172.21.15.12:6789/0 conn(0x7f5eb815dfe0 legacy=0x7f5eb80ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.250+0000 7f5eb7e00700 1 -- 172.21.15.12:0/4230067409 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8071480 con 0x7f5eb815a5c0 2024-03-29T10:06:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.250+0000 7f5eb7e00700 1 --2- 172.21.15.12:0/4230067409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 0x7f5eb80709c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80098f0 tx=0x7f5ea8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef011496a4162f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.251+0000 7f5eb6a00700 1 -- 172.21.15.12:0/4230067409 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8004230 con 0x7f5eb815a5c0 2024-03-29T10:06:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.251+0000 7f5eb6a00700 1 -- 172.21.15.12:0/4230067409 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea8004390 con 0x7f5eb815a5c0 2024-03-29T10:06:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.251+0000 7f5eb6a00700 1 -- 172.21.15.12:0/4230067409 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea80036a0 con 0x7f5eb815a5c0 2024-03-29T10:06:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.251+0000 7f5ebf000700 1 -- 172.21.15.12:0/4230067409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 msgr2=0x7f5eb80709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.251+0000 7f5ebf000700 1 --2- 172.21.15.12:0/4230067409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 0x7f5eb80709c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80098f0 tx=0x7f5ea8009c00 comp rx=0 tx=0).stop 2024-03-29T10:06:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.251+0000 7f5ebf000700 1 -- 172.21.15.12:0/4230067409 shutdown_connections 2024-03-29T10:06:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.251+0000 7f5ebf000700 1 --2- 172.21.15.12:0/4230067409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 0x7f5eb80709c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.251+0000 7f5ebf000700 1 -- 172.21.15.12:0/4230067409 >> 172.21.15.12:0/4230067409 conn(0x7f5eb80b0d70 msgr2=0x7f5eb80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.251+0000 7f5ebf000700 1 -- 172.21.15.12:0/4230067409 shutdown_connections 2024-03-29T10:06:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.251+0000 7f5ebf000700 1 -- 172.21.15.12:0/4230067409 wait complete. 2024-03-29T10:06:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.253+0000 7f5ebf000700 1 Processor -- start 2024-03-29T10:06:29.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.253+0000 7f5ebf000700 1 -- start start 2024-03-29T10:06:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.254+0000 7f5ebf000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb80ce840 0x7f5eb81485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.254+0000 7f5ebf000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 0x7f5eb80b2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.254+0000 7f5ebf000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb815dfe0 0x7f5eb80b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.254+0000 7f5ebf000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb815c360 con 0x7f5eb815a5c0 2024-03-29T10:06:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.254+0000 7f5ebf000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb8148b20 con 0x7f5eb80ce840 2024-03-29T10:06:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.254+0000 7f5ebf000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb815c640 con 0x7f5eb815dfe0 2024-03-29T10:06:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.254+0000 7f5eb7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 0x7f5eb80b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.254+0000 7f5eb7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 0x7f5eb80b2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53074/0 (socket says 172.21.15.12:53074) 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.254+0000 7f5eb7400700 1 -- 172.21.15.12:0/921514389 learned_addr learned my addr 172.21.15.12:0/921514389 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5eb7e00700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb80ce840 0x7f5eb81485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5ebd000700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb815dfe0 0x7f5eb80b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5eb7400700 1 -- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb815dfe0 msgr2=0x7f5eb80b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5eb7400700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb815dfe0 0x7f5eb80b2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5eb7400700 1 -- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb80ce840 msgr2=0x7f5eb81485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5eb7400700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb80ce840 0x7f5eb81485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5eb7400700 1 -- 172.21.15.12:0/921514389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea8009550 con 0x7f5eb815a5c0 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5ebd000700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb815dfe0 0x7f5eb80b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5eb7400700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 0x7f5eb80b2640 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400c460 tx=0x7f5ea400c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5eafe00700 1 -- 172.21.15.12:0/921514389 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea400cea0 con 0x7f5eb815a5c0 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5eafe00700 1 -- 172.21.15.12:0/921514389 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea40052f0 con 0x7f5eb815a5c0 2024-03-29T10:06:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5eafe00700 1 -- 172.21.15.12:0/921514389 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4006860 con 0x7f5eb815a5c0 2024-03-29T10:06:29.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.255+0000 7f5ebf000700 1 -- 172.21.15.12:0/921514389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb815b6e0 con 0x7f5eb815a5c0 2024-03-29T10:06:29.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.256+0000 7f5ebf000700 1 -- 172.21.15.12:0/921514389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb815bca0 con 0x7f5eb815a5c0 2024-03-29T10:06:29.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.257+0000 7f5ebf000700 1 -- 172.21.15.12:0/921514389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb815c4a0 con 0x7f5eb815a5c0 2024-03-29T10:06:29.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.257+0000 7f5eafe00700 1 -- 172.21.15.12:0/921514389 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f5ea4003680 con 0x7f5eb815a5c0 2024-03-29T10:06:29.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.258+0000 7f5eafe00700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e9806ae90 0x7f5e9806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:29.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.258+0000 7f5eafe00700 1 -- 172.21.15.12:0/921514389 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5ea4088960 con 0x7f5eb815a5c0 2024-03-29T10:06:29.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.259+0000 7f5eb7e00700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e9806ae90 0x7f5e9806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:29.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.260+0000 7f5eb7e00700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e9806ae90 0x7f5e9806d350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8009130 tx=0x7f5ea8008030 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:29.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.263+0000 7f5eafe00700 1 -- 172.21.15.12:0/921514389 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5eb815c4a0 con 0x7f5eb815a5c0 2024-03-29T10:06:29.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.503+0000 7f5ebf000700 1 -- 172.21.15.12:0/921514389 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5eb80cb560 con 0x7f5e9806ae90 2024-03-29T10:06:29.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.514+0000 7f5eafe00700 1 -- 172.21.15.12:0/921514389 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f5ea8006d00 con 0x7f5e9806ae90 2024-03-29T10:06:29.514 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:29.514 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:56", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 -- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e9806ae90 msgr2=0x7f5e9806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e9806ae90 0x7f5e9806d350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8009130 tx=0x7f5ea8008030 comp rx=0 tx=0).stop 2024-03-29T10:06:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 -- 172.21.15.12:0/921514389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 msgr2=0x7f5eb80b2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 0x7f5eb80b2640 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400c460 tx=0x7f5ea400c770 comp rx=0 tx=0).stop 2024-03-29T10:06:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 -- 172.21.15.12:0/921514389 shutdown_connections 2024-03-29T10:06:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5e9806ae90 0x7f5e9806d350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb815dfe0 0x7f5eb80b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb80ce840 0x7f5eb81485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:29.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 --2- 172.21.15.12:0/921514389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb815a5c0 0x7f5eb80b2640 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:29.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 -- 172.21.15.12:0/921514389 >> 172.21.15.12:0/921514389 conn(0x7f5eb80b0d70 msgr2=0x7f5eb815e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:29.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 -- 172.21.15.12:0/921514389 shutdown_connections 2024-03-29T10:06:29.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.517+0000 7f5ebf000700 1 -- 172.21.15.12:0/921514389 wait complete. 2024-03-29T10:06:29.535 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:56", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:29.535 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T09:55:56', 'first': '2024-03-29T09:56:00', 'last': '2024-03-29T10:06:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-29T10:06:29.535 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-29T10:06:29.535 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-29T10:06:29.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:29.535 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-03-29T10:06:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:29.584 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-09_56_00_UTC', 'scheduled-2024-03-29-09_57_00_UTC', 'scheduled-2024-03-29-09_58_00_UTC', 'scheduled-2024-03-29-09_59_00_UTC', 'scheduled-2024-03-29-10_00_00_UTC', 'scheduled-2024-03-29-10_01_00_UTC', 'scheduled-2024-03-29-10_02_00_UTC', 'scheduled-2024-03-29-10_03_00_UTC', 'scheduled-2024-03-29-10_04_00_UTC', 'scheduled-2024-03-29-10_05_00_UTC', 'scheduled-2024-03-29-10_06_00_UTC'] 2024-03-29T10:06:29.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2024-03-29T10:06:29.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.784+0000 7f839a200700 1 Processor -- start 2024-03-29T10:06:29.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.784+0000 7f839a200700 1 -- start start 2024-03-29T10:06:29.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.786+0000 7f839a200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d2540 0x7f83941687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:29.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.786+0000 7f839a200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8394161420 con 0x7f839415d9d0 2024-03-29T10:06:29.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.786+0000 7f839a200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8394161700 con 0x7f83940d16a0 2024-03-29T10:06:29.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.786+0000 7f839a200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83940d4340 con 0x7f83940d2540 2024-03-29T10:06:29.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.786+0000 7f8393400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f839415d9d0 0x7f83940d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54234/0 (socket says 172.21.15.12:54234) 2024-03-29T10:06:29.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.786+0000 7f8393400700 1 -- 172.21.15.12:0/1750396593 learned_addr learned my addr 172.21.15.12:0/1750396593 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:29.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.786+0000 7f8393e00700 1 --2- 172.21.15.12:0/1750396593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d2540 0x7f83941687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:29.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8392000700 1 -- 172.21.15.12:0/1750396593 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216450605 0 0) 0x7f8394161420 con 0x7f839415d9d0 2024-03-29T10:06:29.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8392000700 1 -- 172.21.15.12:0/1750396593 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83780035b0 con 0x7f839415d9d0 2024-03-29T10:06:29.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8392000700 1 -- 172.21.15.12:0/1750396593 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820593509 0 0) 0x7f8394161700 con 0x7f83940d16a0 2024-03-29T10:06:29.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8392000700 1 -- 172.21.15.12:0/1750396593 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8394161420 con 0x7f83940d16a0 2024-03-29T10:06:29.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8393e00700 1 -- 172.21.15.12:0/1750396593 >> v1:172.21.15.159:6789/0 conn(0x7f83940d16a0 legacy=0x7f83940d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8393e00700 1 -- 172.21.15.12:0/1750396593 >> v1:172.21.15.12:6789/0 conn(0x7f839415d9d0 legacy=0x7f83940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8393e00700 1 -- 172.21.15.12:0/1750396593 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83941619e0 con 0x7f83940d2540 2024-03-29T10:06:29.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8393e00700 1 --2- 172.21.15.12:0/1750396593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d2540 0x7f83941687b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f838c009dc0 tx=0x7f838c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b50b4b031ca8fec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:29.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8392000700 1 -- 172.21.15.12:0/1750396593 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c00e070 con 0x7f83940d2540 2024-03-29T10:06:29.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8392000700 1 -- 172.21.15.12:0/1750396593 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838c00bb20 con 0x7f83940d2540 2024-03-29T10:06:29.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.787+0000 7f8392000700 1 -- 172.21.15.12:0/1750396593 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c012430 con 0x7f83940d2540 2024-03-29T10:06:29.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.788+0000 7f839a200700 1 -- 172.21.15.12:0/1750396593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d2540 msgr2=0x7f83941687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.788+0000 7f839a200700 1 --2- 172.21.15.12:0/1750396593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d2540 0x7f83941687b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f838c009dc0 tx=0x7f838c00b330 comp rx=0 tx=0).stop 2024-03-29T10:06:29.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.788+0000 7f839a200700 1 -- 172.21.15.12:0/1750396593 shutdown_connections 2024-03-29T10:06:29.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.788+0000 7f839a200700 1 --2- 172.21.15.12:0/1750396593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d2540 0x7f83941687b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:29.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.788+0000 7f839a200700 1 -- 172.21.15.12:0/1750396593 >> 172.21.15.12:0/1750396593 conn(0x7f839408aff0 msgr2=0x7f83940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:29.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.788+0000 7f839a200700 1 -- 172.21.15.12:0/1750396593 shutdown_connections 2024-03-29T10:06:29.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.788+0000 7f839a200700 1 -- 172.21.15.12:0/1750396593 wait complete. 2024-03-29T10:06:29.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.790+0000 7f839a200700 1 Processor -- start 2024-03-29T10:06:29.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.790+0000 7f839a200700 1 -- start start 2024-03-29T10:06:29.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.791+0000 7f839a200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d16a0 0x7f83940cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:29.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.791+0000 7f839a200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83940d2540 0x7f8394156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:29.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.791+0000 7f839a200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f839415d9d0 0x7f839415fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:29.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.791+0000 7f839a200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83940ca5f0 con 0x7f83940d2540 2024-03-29T10:06:29.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.791+0000 7f839a200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83940cae90 con 0x7f839415d9d0 2024-03-29T10:06:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.791+0000 7f839a200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8394156590 con 0x7f83940d16a0 2024-03-29T10:06:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.791+0000 7f8392a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83940d2540 0x7f8394156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.791+0000 7f8392a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83940d2540 0x7f8394156050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53078/0 (socket says 172.21.15.12:53078) 2024-03-29T10:06:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.791+0000 7f8392a00700 1 -- 172.21.15.12:0/95303044 learned_addr learned my addr 172.21.15.12:0/95303044 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f8393e00700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f839415d9d0 0x7f839415fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f8393400700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d16a0 0x7f83940cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f8392a00700 1 -- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d16a0 msgr2=0x7f83940cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f8392a00700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d16a0 0x7f83940cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f8392a00700 1 -- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f839415d9d0 msgr2=0x7f839415fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f8392a00700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f839415d9d0 0x7f839415fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f8392a00700 1 -- 172.21.15.12:0/95303044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c009a70 con 0x7f83940d2540 2024-03-29T10:06:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f8393e00700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f839415d9d0 0x7f839415fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f8393400700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d16a0 0x7f83940cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f8392a00700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83940d2540 0x7f8394156050 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f838000c520 tx=0x7f838000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f838b400700 1 -- 172.21.15.12:0/95303044 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83800052d0 con 0x7f83940d2540 2024-03-29T10:06:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f838b400700 1 -- 172.21.15.12:0/95303044 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8380005430 con 0x7f83940d2540 2024-03-29T10:06:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f839a200700 1 -- 172.21.15.12:0/95303044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83941608b0 con 0x7f83940d2540 2024-03-29T10:06:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f838b400700 1 -- 172.21.15.12:0/95303044 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83800038f0 con 0x7f83940d2540 2024-03-29T10:06:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.792+0000 7f839a200700 1 -- 172.21.15.12:0/95303044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83940cab20 con 0x7f83940d2540 2024-03-29T10:06:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.793+0000 7f839a200700 1 -- 172.21.15.12:0/95303044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83940d1ca0 con 0x7f83940d2540 2024-03-29T10:06:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.794+0000 7f838b400700 1 -- 172.21.15.12:0/95303044 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f8380003a50 con 0x7f83940d2540 2024-03-29T10:06:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.795+0000 7f838b400700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837406ae90 0x7f837406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.795+0000 7f838b400700 1 -- 172.21.15.12:0/95303044 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f8380088e50 con 0x7f83940d2540 2024-03-29T10:06:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.795+0000 7f8393400700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837406ae90 0x7f837406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.796+0000 7f8393400700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837406ae90 0x7f837406d350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f83840011e0 tx=0x7f8384009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:29.800+0000 7f838b400700 1 -- 172.21.15.12:0/95303044 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8380053b90 con 0x7f83940d2540 2024-03-29T10:06:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.037+0000 7f839a200700 1 -- 172.21.15.12:0/95303044 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83940cace0 con 0x7f837406ae90 2024-03-29T10:06:30.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.047+0000 7f838b400700 1 -- 172.21.15.12:0/95303044 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f83940cace0 con 0x7f837406ae90 2024-03-29T10:06:30.047 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:30.047 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:56", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 -- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837406ae90 msgr2=0x7f837406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837406ae90 0x7f837406d350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f83840011e0 tx=0x7f8384009040 comp rx=0 tx=0).stop 2024-03-29T10:06:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 -- 172.21.15.12:0/95303044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83940d2540 msgr2=0x7f8394156050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83940d2540 0x7f8394156050 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f838000c520 tx=0x7f838000c8e0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 -- 172.21.15.12:0/95303044 shutdown_connections 2024-03-29T10:06:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837406ae90 0x7f837406d350 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f839415d9d0 0x7f839415fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83940d16a0 0x7f83940cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 --2- 172.21.15.12:0/95303044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83940d2540 0x7f8394156050 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 -- 172.21.15.12:0/95303044 >> 172.21.15.12:0/95303044 conn(0x7f839408aff0 msgr2=0x7f83940cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 -- 172.21.15.12:0/95303044 shutdown_connections 2024-03-29T10:06:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.050+0000 7f839a200700 1 -- 172.21.15.12:0/95303044 wait complete. 2024-03-29T10:06:30.068 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:56", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:30.069 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T09:55:56', 'first': '2024-03-29T09:56:00', 'last': '2024-03-29T10:06:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-29T10:06:30.069 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-29T10:06:30.069 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-29T10:06:30.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:30.069 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-03-29T10:06:30.115 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:30.115 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:30.115 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:30.115 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:30.115 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:30.115 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:30.116 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:30.116 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:30.116 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:30.116 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:30.116 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:30.117 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-09_56_00_UTC', 'scheduled-2024-03-29-09_57_00_UTC', 'scheduled-2024-03-29-09_58_00_UTC', 'scheduled-2024-03-29-09_59_00_UTC', 'scheduled-2024-03-29-10_00_00_UTC', 'scheduled-2024-03-29-10_01_00_UTC', 'scheduled-2024-03-29-10_02_00_UTC', 'scheduled-2024-03-29-10_03_00_UTC', 'scheduled-2024-03-29-10_04_00_UTC', 'scheduled-2024-03-29-10_05_00_UTC', 'scheduled-2024-03-29-10_06_00_UTC'] 2024-03-29T10:06:30.117 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2024-03-29T10:06:30.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.357+0000 7f2d01200700 1 Processor -- start 2024-03-29T10:06:30.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.357+0000 7f2d01200700 1 -- start start 2024-03-29T10:06:30.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2d01200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2cfc0cbb30 0x7f2cfc0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2d01200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cd020 0x7f2cfc0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2d01200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cfc0b2070 0x7f2cfc0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2d01200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc0cc450 con 0x7f2cfc0b2070 2024-03-29T10:06:30.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2d01200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc0cdae0 con 0x7f2cfc0cbb30 2024-03-29T10:06:30.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2d01200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc0cddc0 con 0x7f2cfc0cd020 2024-03-29T10:06:30.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2cfa800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cfc0b2070 0x7f2cfc0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2cfa800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cfc0b2070 0x7f2cfc0b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53086/0 (socket says 172.21.15.12:53086) 2024-03-29T10:06:30.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2cfa800700 1 -- 172.21.15.12:0/3224505964 learned_addr learned my addr 172.21.15.12:0/3224505964 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:30.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2cf9e00700 1 --2- 172.21.15.12:0/3224505964 >> v2:172.21.15.159:3300/0 conn(0x7f2cfc0cbb30 0x7f2cfc0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.358+0000 7f2cf9400700 1 --2- 172.21.15.12:0/3224505964 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cd020 0x7f2cfc0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.359+0000 7f2cfa800700 1 -- 172.21.15.12:0/3224505964 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cd020 msgr2=0x7f2cfc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:30.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.359+0000 7f2cfa800700 1 --2- 172.21.15.12:0/3224505964 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cd020 0x7f2cfc0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.359+0000 7f2cfa800700 1 -- 172.21.15.12:0/3224505964 >> v2:172.21.15.159:3300/0 conn(0x7f2cfc0cbb30 msgr2=0x7f2cfc0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:30.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.359+0000 7f2cfa800700 1 --2- 172.21.15.12:0/3224505964 >> v2:172.21.15.159:3300/0 conn(0x7f2cfc0cbb30 0x7f2cfc0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.359+0000 7f2cfa800700 1 -- 172.21.15.12:0/3224505964 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc0ce0a0 con 0x7f2cfc0b2070 2024-03-29T10:06:30.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.359+0000 7f2cf9400700 1 --2- 172.21.15.12:0/3224505964 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cd020 0x7f2cfc0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.359+0000 7f2cfa800700 1 --2- 172.21.15.12:0/3224505964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 0x7f2cfc0b2500 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc06fd00 tx=0x7f2cf400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13fd4a4c64a721 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.359+0000 7f2cf8a00700 1 -- 172.21.15.12:0/3224505964 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf400e070 con 0x7f2cfc0b2070 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.359+0000 7f2cf8a00700 1 -- 172.21.15.12:0/3224505964 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf400b9c0 con 0x7f2cfc0b2070 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.359+0000 7f2cf8a00700 1 -- 172.21.15.12:0/3224505964 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf400bd70 con 0x7f2cfc0b2070 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.360+0000 7f2d01200700 1 -- 172.21.15.12:0/3224505964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 msgr2=0x7f2cfc0b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.360+0000 7f2d01200700 1 --2- 172.21.15.12:0/3224505964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 0x7f2cfc0b2500 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc06fd00 tx=0x7f2cf400b330 comp rx=0 tx=0).stop 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.360+0000 7f2d01200700 1 -- 172.21.15.12:0/3224505964 shutdown_connections 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.360+0000 7f2d01200700 1 --2- 172.21.15.12:0/3224505964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 0x7f2cfc0b2500 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.360+0000 7f2d01200700 1 --2- 172.21.15.12:0/3224505964 >> v2:172.21.15.159:3300/0 conn(0x7f2cfc0cbb30 0x7f2cfc0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.360+0000 7f2d01200700 1 --2- 172.21.15.12:0/3224505964 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cd020 0x7f2cfc0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.360+0000 7f2d01200700 1 -- 172.21.15.12:0/3224505964 >> 172.21.15.12:0/3224505964 conn(0x7f2cfc0b0d70 msgr2=0x7f2cfc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:30.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.360+0000 7f2d01200700 1 -- 172.21.15.12:0/3224505964 shutdown_connections 2024-03-29T10:06:30.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.360+0000 7f2d01200700 1 -- 172.21.15.12:0/3224505964 wait complete. 2024-03-29T10:06:30.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.362+0000 7f2d01200700 1 Processor -- start 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.362+0000 7f2d01200700 1 -- start start 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2d01200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 0x7f2cfc15dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2d01200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cbb30 0x7f2cfc148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2d01200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc0cd020 0x7f2cfc147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2d01200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc149530 con 0x7f2cfc0b2070 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2d01200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc149810 con 0x7f2cfc0cd020 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2d01200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc149af0 con 0x7f2cfc0cbb30 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2cf9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 0x7f2cfc15dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2cf9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 0x7f2cfc15dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53088/0 (socket says 172.21.15.12:53088) 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2cf9e00700 1 -- 172.21.15.12:0/3776008674 learned_addr learned my addr 172.21.15.12:0/3776008674 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2cfa800700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc0cd020 0x7f2cfc147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.363+0000 7f2cf9400700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cbb30 0x7f2cfc148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2cf9e00700 1 -- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cbb30 msgr2=0x7f2cfc148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2cf9e00700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cbb30 0x7f2cfc148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2cf9e00700 1 -- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc0cd020 msgr2=0x7f2cfc147e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2cf9e00700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc0cd020 0x7f2cfc147e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2cf9e00700 1 -- 172.21.15.12:0/3776008674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf4009b30 con 0x7f2cfc0b2070 2024-03-29T10:06:30.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2cf9e00700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 0x7f2cfc15dec0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00d800 tx=0x7f2cec00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:30.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2cf2000700 1 -- 172.21.15.12:0/3776008674 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec0125f0 con 0x7f2cfc0b2070 2024-03-29T10:06:30.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2cf2000700 1 -- 172.21.15.12:0/3776008674 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec012750 con 0x7f2cfc0b2070 2024-03-29T10:06:30.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2d01200700 1 -- 172.21.15.12:0/3776008674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc149dd0 con 0x7f2cfc0b2070 2024-03-29T10:06:30.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2cf2000700 1 -- 172.21.15.12:0/3776008674 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec012ae0 con 0x7f2cfc0b2070 2024-03-29T10:06:30.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.364+0000 7f2d01200700 1 -- 172.21.15.12:0/3776008674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc1f4250 con 0x7f2cfc0b2070 2024-03-29T10:06:30.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.365+0000 7f2d01200700 1 -- 172.21.15.12:0/3776008674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc067720 con 0x7f2cfc0b2070 2024-03-29T10:06:30.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.366+0000 7f2cf2000700 1 -- 172.21.15.12:0/3776008674 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f2cec012c40 con 0x7f2cfc0b2070 2024-03-29T10:06:30.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.367+0000 7f2cf2000700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cdc06ae90 0x7f2cdc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.368+0000 7f2cf2000700 1 -- 172.21.15.12:0/3776008674 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f2cec089780 con 0x7f2cfc0b2070 2024-03-29T10:06:30.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.368+0000 7f2cf9400700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cdc06ae90 0x7f2cdc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.369+0000 7f2cf9400700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cdc06ae90 0x7f2cdc06d350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8006f10 tx=0x7f2ce8008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:30.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.372+0000 7f2cf2000700 1 -- 172.21.15.12:0/3776008674 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2cec0545e0 con 0x7f2cfc0b2070 2024-03-29T10:06:30.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.611+0000 7f2d01200700 1 -- 172.21.15.12:0/3776008674 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2cfc149980 con 0x7f2cdc06ae90 2024-03-29T10:06:30.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.621+0000 7f2cf2000700 1 -- 172.21.15.12:0/3776008674 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f2cfc149980 con 0x7f2cdc06ae90 2024-03-29T10:06:30.621 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:30.621 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:57", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:30.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.623+0000 7f2d01200700 1 -- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cdc06ae90 msgr2=0x7f2cdc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.623+0000 7f2d01200700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cdc06ae90 0x7f2cdc06d350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8006f10 tx=0x7f2ce8008040 comp rx=0 tx=0).stop 2024-03-29T10:06:30.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.623+0000 7f2d01200700 1 -- 172.21.15.12:0/3776008674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 msgr2=0x7f2cfc15dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.623+0000 7f2d01200700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 0x7f2cfc15dec0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00d800 tx=0x7f2cec00db10 comp rx=0 tx=0).stop 2024-03-29T10:06:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.623+0000 7f2d01200700 1 -- 172.21.15.12:0/3776008674 shutdown_connections 2024-03-29T10:06:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.623+0000 7f2d01200700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cdc06ae90 0x7f2cdc06d350 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.623+0000 7f2d01200700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cfc0cd020 0x7f2cfc147e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.623+0000 7f2d01200700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cfc0b2070 0x7f2cfc15dec0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.623+0000 7f2d01200700 1 --2- 172.21.15.12:0/3776008674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cfc0cbb30 0x7f2cfc148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.624+0000 7f2d01200700 1 -- 172.21.15.12:0/3776008674 >> 172.21.15.12:0/3776008674 conn(0x7f2cfc0b0d70 msgr2=0x7f2cfc15a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.624+0000 7f2d01200700 1 -- 172.21.15.12:0/3776008674 shutdown_connections 2024-03-29T10:06:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.624+0000 7f2d01200700 1 -- 172.21.15.12:0/3776008674 wait complete. 2024-03-29T10:06:30.641 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:57", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:30.641 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T09:55:57', 'first': '2024-03-29T09:56:00', 'last': '2024-03-29T10:06:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-29T10:06:30.641 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-29T10:06:30.641 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-29T10:06:30.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:30.641 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-03-29T10:06:30.686 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:30.687 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:30.687 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:30.687 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:30.687 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:30.687 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:30.687 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:30.687 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:30.687 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:30.687 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:30.687 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:30.688 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-09_56_00_UTC', 'scheduled-2024-03-29-09_57_00_UTC', 'scheduled-2024-03-29-09_58_00_UTC', 'scheduled-2024-03-29-09_59_00_UTC', 'scheduled-2024-03-29-10_00_00_UTC', 'scheduled-2024-03-29-10_01_00_UTC', 'scheduled-2024-03-29-10_02_00_UTC', 'scheduled-2024-03-29-10_03_00_UTC', 'scheduled-2024-03-29-10_04_00_UTC', 'scheduled-2024-03-29-10_05_00_UTC', 'scheduled-2024-03-29-10_06_00_UTC'] 2024-03-29T10:06:30.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2024-03-29T10:06:30.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.929+0000 7fa4cee00700 1 Processor -- start 2024-03-29T10:06:30.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.930+0000 7fa4cee00700 1 -- start start 2024-03-29T10:06:30.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.931+0000 7fa4cee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c815a5c0 0x7fa4c80709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.931+0000 7fa4cee00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa4c80ce840 0x7fa4c80cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.931+0000 7fa4cee00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c80cb310 con 0x7fa4c815dfe0 2024-03-29T10:06:30.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.931+0000 7fa4cee00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c8070f00 con 0x7fa4c80ce840 2024-03-29T10:06:30.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.931+0000 7fa4cee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c80711a0 con 0x7fa4c815a5c0 2024-03-29T10:06:30.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.931+0000 7fa4c7400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa4c815dfe0 0x7fa4c80ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54250/0 (socket says 172.21.15.12:54250) 2024-03-29T10:06:30.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.931+0000 7fa4c7400700 1 -- 172.21.15.12:0/3567272815 learned_addr learned my addr 172.21.15.12:0/3567272815 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:30.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.931+0000 7fa4cce00700 1 --2- 172.21.15.12:0/3567272815 >> v2:172.21.15.12:3300/0 conn(0x7fa4c80ce840 0x7fa4c80cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.931+0000 7fa4c7e00700 1 --2- 172.21.15.12:0/3567272815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c815a5c0 0x7fa4c80709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.932+0000 7fa4cce00700 1 -- 172.21.15.12:0/3567272815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c815a5c0 msgr2=0x7fa4c80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:30.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.932+0000 7fa4cce00700 1 --2- 172.21.15.12:0/3567272815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c815a5c0 0x7fa4c80709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.932+0000 7fa4cce00700 1 -- 172.21.15.12:0/3567272815 >> v1:172.21.15.12:6789/0 conn(0x7fa4c815dfe0 legacy=0x7fa4c80ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.932+0000 7fa4cce00700 1 -- 172.21.15.12:0/3567272815 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c8071480 con 0x7fa4c80ce840 2024-03-29T10:06:30.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.932+0000 7fa4cce00700 1 --2- 172.21.15.12:0/3567272815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ce840 0x7fa4c80cba80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0009d90 tx=0x7fa4c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b2970b9eed525d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:30.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.932+0000 7fa4c6a00700 1 -- 172.21.15.12:0/3567272815 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c000e070 con 0x7fa4c80ce840 2024-03-29T10:06:30.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.932+0000 7fa4c6a00700 1 -- 172.21.15.12:0/3567272815 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c000bb20 con 0x7fa4c80ce840 2024-03-29T10:06:30.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.932+0000 7fa4c6a00700 1 -- 172.21.15.12:0/3567272815 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0012430 con 0x7fa4c80ce840 2024-03-29T10:06:30.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.933+0000 7fa4cee00700 1 -- 172.21.15.12:0/3567272815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ce840 msgr2=0x7fa4c80cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.933+0000 7fa4cee00700 1 --2- 172.21.15.12:0/3567272815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ce840 0x7fa4c80cba80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0009d90 tx=0x7fa4c000b330 comp rx=0 tx=0).stop 2024-03-29T10:06:30.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.933+0000 7fa4cee00700 1 -- 172.21.15.12:0/3567272815 shutdown_connections 2024-03-29T10:06:30.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.933+0000 7fa4cee00700 1 --2- 172.21.15.12:0/3567272815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ce840 0x7fa4c80cba80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.933+0000 7fa4cee00700 1 --2- 172.21.15.12:0/3567272815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c815a5c0 0x7fa4c80709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.933+0000 7fa4cee00700 1 -- 172.21.15.12:0/3567272815 >> 172.21.15.12:0/3567272815 conn(0x7fa4c80b0d70 msgr2=0x7fa4c80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:30.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.933+0000 7fa4cee00700 1 -- 172.21.15.12:0/3567272815 shutdown_connections 2024-03-29T10:06:30.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.933+0000 7fa4cee00700 1 -- 172.21.15.12:0/3567272815 wait complete. 2024-03-29T10:06:30.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.935+0000 7fa4cee00700 1 Processor -- start 2024-03-29T10:06:30.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.935+0000 7fa4cee00700 1 -- start start 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4cee00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4c80ce840 0x7fa4c81f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4cee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c815a5c0 0x7fa4c80c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4cee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c815dfe0 0x7fa4c80af770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4cee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c8154760 con 0x7fa4c815a5c0 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4cee00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c8154a40 con 0x7fa4c80ce840 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4cee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c8154d20 con 0x7fa4c815dfe0 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c815a5c0 0x7fa4c80c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c815a5c0 0x7fa4c80c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53098/0 (socket says 172.21.15.12:53098) 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4c7400700 1 -- 172.21.15.12:0/3106165242 learned_addr learned my addr 172.21.15.12:0/3106165242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4c7400700 1 -- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c815dfe0 msgr2=0x7fa4c80af770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4c7400700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c815dfe0 0x7fa4c80af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4c7e00700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4c80ce840 0x7fa4c81f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4c7400700 1 -- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4c80ce840 msgr2=0x7fa4c81f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:30.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4c7400700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4c80ce840 0x7fa4c81f5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:30.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.936+0000 7fa4c7400700 1 -- 172.21.15.12:0/3106165242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c0009a70 con 0x7fa4c815a5c0 2024-03-29T10:06:30.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.937+0000 7fa4c7400700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c815a5c0 0x7fa4c80c83f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400c480 tx=0x7fa4b400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:30.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.937+0000 7fa4bfe00700 1 -- 172.21.15.12:0/3106165242 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4005d70 con 0x7fa4c815a5c0 2024-03-29T10:06:30.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.937+0000 7fa4bfe00700 1 -- 172.21.15.12:0/3106165242 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b40070d0 con 0x7fa4c815a5c0 2024-03-29T10:06:30.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.937+0000 7fa4bfe00700 1 -- 172.21.15.12:0/3106165242 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4007440 con 0x7fa4c815a5c0 2024-03-29T10:06:30.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.937+0000 7fa4cee00700 1 -- 172.21.15.12:0/3106165242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c8155000 con 0x7fa4c815a5c0 2024-03-29T10:06:30.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.937+0000 7fa4cee00700 1 -- 172.21.15.12:0/3106165242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c8156200 con 0x7fa4c815a5c0 2024-03-29T10:06:30.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.938+0000 7fa4cee00700 1 -- 172.21.15.12:0/3106165242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c8155520 con 0x7fa4c815a5c0 2024-03-29T10:06:30.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.939+0000 7fa4bfe00700 1 -- 172.21.15.12:0/3106165242 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fa4b40075a0 con 0x7fa4c815a5c0 2024-03-29T10:06:30.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.940+0000 7fa4bfe00700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa4a806ae90 0x7fa4a806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:30.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.940+0000 7fa4bfe00700 1 -- 172.21.15.12:0/3106165242 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa4b4088bf0 con 0x7fa4c815a5c0 2024-03-29T10:06:30.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.940+0000 7fa4c7e00700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa4a806ae90 0x7fa4a806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:30.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.941+0000 7fa4c7e00700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa4a806ae90 0x7fa4a806d350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009ab0 tx=0x7fa4b8008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:30.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:30.944+0000 7fa4bfe00700 1 -- 172.21.15.12:0/3106165242 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4c8155520 con 0x7fa4c815a5c0 2024-03-29T10:06:31.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.185+0000 7fa4cee00700 1 -- 172.21.15.12:0/3106165242 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4c80cb560 con 0x7fa4a806ae90 2024-03-29T10:06:31.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.194+0000 7fa4bfe00700 1 -- 172.21.15.12:0/3106165242 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fa4b80106e0 con 0x7fa4a806ae90 2024-03-29T10:06:31.194 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:31.194 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:58", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:31.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 -- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa4a806ae90 msgr2=0x7fa4a806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:31.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa4a806ae90 0x7fa4a806d350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009ab0 tx=0x7fa4b8008040 comp rx=0 tx=0).stop 2024-03-29T10:06:31.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 -- 172.21.15.12:0/3106165242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c815a5c0 msgr2=0x7fa4c80c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:31.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c815a5c0 0x7fa4c80c83f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400c480 tx=0x7fa4b400c840 comp rx=0 tx=0).stop 2024-03-29T10:06:31.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 -- 172.21.15.12:0/3106165242 shutdown_connections 2024-03-29T10:06:31.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa4a806ae90 0x7fa4a806d350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c815dfe0 0x7fa4c80af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4c80ce840 0x7fa4c81f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 --2- 172.21.15.12:0/3106165242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c815a5c0 0x7fa4c80c83f0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 -- 172.21.15.12:0/3106165242 >> 172.21.15.12:0/3106165242 conn(0x7fa4c80b0d70 msgr2=0x7fa4c80b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:31.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 -- 172.21.15.12:0/3106165242 shutdown_connections 2024-03-29T10:06:31.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.197+0000 7fa4cee00700 1 -- 172.21.15.12:0/3106165242 wait complete. 2024-03-29T10:06:31.214 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:58", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:31.215 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T09:55:58', 'first': '2024-03-29T09:56:00', 'last': '2024-03-29T10:06:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-29T10:06:31.215 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-29T10:06:31.215 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-29T10:06:31.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:31.215 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-03-29T10:06:31.260 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:31.260 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:31.260 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:31.260 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:31.261 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:31.261 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:31.261 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:31.261 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:31.261 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:31.261 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:31.261 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:31.262 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-09_56_00_UTC', 'scheduled-2024-03-29-09_57_00_UTC', 'scheduled-2024-03-29-09_58_00_UTC', 'scheduled-2024-03-29-09_59_00_UTC', 'scheduled-2024-03-29-10_00_00_UTC', 'scheduled-2024-03-29-10_01_00_UTC', 'scheduled-2024-03-29-10_02_00_UTC', 'scheduled-2024-03-29-10_03_00_UTC', 'scheduled-2024-03-29-10_04_00_UTC', 'scheduled-2024-03-29-10_05_00_UTC', 'scheduled-2024-03-29-10_06_00_UTC'] 2024-03-29T10:06:31.262 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2024-03-29T10:06:31.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.506+0000 7ff61ce00700 1 Processor -- start 2024-03-29T10:06:31.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.506+0000 7ff61ce00700 1 -- start start 2024-03-29T10:06:31.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.507+0000 7ff61ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6180cd5c0 0x7ff61815e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:31.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.507+0000 7ff61ce00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff61806f310 0x7ff61806f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:31.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.507+0000 7ff61ce00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61815eda0 con 0x7ff61815cfe0 2024-03-29T10:06:31.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.507+0000 7ff61ce00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff61806fcb0 con 0x7ff61806f310 2024-03-29T10:06:31.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.507+0000 7ff61ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff618099dd0 con 0x7ff6180cd5c0 2024-03-29T10:06:31.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff615e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6180cd5c0 0x7ff61815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:31.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff615400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff61806f310 0x7ff61806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:31.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff615e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6180cd5c0 0x7ff61815e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51896/0 (socket says 172.21.15.12:51896) 2024-03-29T10:06:31.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff616800700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ff61815cfe0 0x7ff61815d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:54612/0 (socket says 172.21.15.12:54612) 2024-03-29T10:06:31.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff615e00700 1 -- 172.21.15.12:0/635062672 learned_addr learned my addr 172.21.15.12:0/635062672 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:31.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff615e00700 1 -- 172.21.15.12:0/635062672 >> v2:172.21.15.159:3300/0 conn(0x7ff61806f310 msgr2=0x7ff61806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:31.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff615e00700 1 --2- 172.21.15.12:0/635062672 >> v2:172.21.15.159:3300/0 conn(0x7ff61806f310 0x7ff61806f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff615e00700 1 -- 172.21.15.12:0/635062672 >> v1:172.21.15.159:6789/0 conn(0x7ff61815cfe0 legacy=0x7ff61815d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:31.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff615e00700 1 -- 172.21.15.12:0/635062672 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61809a0b0 con 0x7ff6180cd5c0 2024-03-29T10:06:31.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff615400700 1 --2- 172.21.15.12:0/635062672 >> v2:172.21.15.159:3300/0 conn(0x7ff61806f310 0x7ff61806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.508+0000 7ff615e00700 1 --2- 172.21.15.12:0/635062672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6180cd5c0 0x7ff61815e860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff6080098f0 tx=0x7ff608009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3f49357ec1c0957 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff614a00700 1 -- 172.21.15.12:0/635062672 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608011830 con 0x7ff6180cd5c0 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff614a00700 1 -- 172.21.15.12:0/635062672 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608011990 con 0x7ff6180cd5c0 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff614a00700 1 -- 172.21.15.12:0/635062672 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608011d20 con 0x7ff6180cd5c0 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff61ce00700 1 -- 172.21.15.12:0/635062672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6180cd5c0 msgr2=0x7ff61815e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff61ce00700 1 --2- 172.21.15.12:0/635062672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6180cd5c0 0x7ff61815e860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff6080098f0 tx=0x7ff608009c00 comp rx=0 tx=0).stop 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff61ce00700 1 -- 172.21.15.12:0/635062672 shutdown_connections 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff61ce00700 1 --2- 172.21.15.12:0/635062672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6180cd5c0 0x7ff61815e860 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff61ce00700 1 --2- 172.21.15.12:0/635062672 >> v2:172.21.15.159:3300/0 conn(0x7ff61806f310 0x7ff61806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff61ce00700 1 -- 172.21.15.12:0/635062672 >> 172.21.15.12:0/635062672 conn(0x7ff6180d1370 msgr2=0x7ff6180d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff61ce00700 1 -- 172.21.15.12:0/635062672 shutdown_connections 2024-03-29T10:06:31.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.509+0000 7ff61ce00700 1 -- 172.21.15.12:0/635062672 wait complete. 2024-03-29T10:06:31.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.511+0000 7ff61ce00700 1 Processor -- start 2024-03-29T10:06:31.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.511+0000 7ff61ce00700 1 -- start start 2024-03-29T10:06:31.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.512+0000 7ff61ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff61806f310 0x7ff6180b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:31.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.512+0000 7ff61ce00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6180cd5c0 0x7ff6181575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:31.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.512+0000 7ff61ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff61815cfe0 0x7ff61815d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:31.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.512+0000 7ff61ce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff618158090 con 0x7ff61806f310 2024-03-29T10:06:31.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.512+0000 7ff61ce00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180b5de0 con 0x7ff6180cd5c0 2024-03-29T10:06:31.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.512+0000 7ff61ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180b60c0 con 0x7ff61815cfe0 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.512+0000 7ff615e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff61806f310 0x7ff6180b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.512+0000 7ff615e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff61806f310 0x7ff6180b50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53100/0 (socket says 172.21.15.12:53100) 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.512+0000 7ff615e00700 1 -- 172.21.15.12:0/241746744 learned_addr learned my addr 172.21.15.12:0/241746744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff615400700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6180cd5c0 0x7ff6181575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff616800700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff61815cfe0 0x7ff61815d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff615e00700 1 -- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff61815cfe0 msgr2=0x7ff61815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff615e00700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff61815cfe0 0x7ff61815d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff615e00700 1 -- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6180cd5c0 msgr2=0x7ff6181575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff615e00700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6180cd5c0 0x7ff6181575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff615e00700 1 -- 172.21.15.12:0/241746744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6040094a0 con 0x7ff61806f310 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff616800700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff61815cfe0 0x7ff61815d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff615e00700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff61806f310 0x7ff6180b50b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff608009c40 tx=0x7ff608004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:31.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff60e000700 1 -- 172.21.15.12:0/241746744 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608004480 con 0x7ff61806f310 2024-03-29T10:06:31.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff60e000700 1 -- 172.21.15.12:0/241746744 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60801f070 con 0x7ff61806f310 2024-03-29T10:06:31.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.513+0000 7ff61ce00700 1 -- 172.21.15.12:0/241746744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff608009550 con 0x7ff61806f310 2024-03-29T10:06:31.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.514+0000 7ff60e000700 1 -- 172.21.15.12:0/241746744 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6080038e0 con 0x7ff61806f310 2024-03-29T10:06:31.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.514+0000 7ff61ce00700 1 -- 172.21.15.12:0/241746744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6180b6680 con 0x7ff61806f310 2024-03-29T10:06:31.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.515+0000 7ff61ce00700 1 -- 172.21.15.12:0/241746744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff618068240 con 0x7ff61806f310 2024-03-29T10:06:31.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.515+0000 7ff60e000700 1 -- 172.21.15.12:0/241746744 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7ff608021030 con 0x7ff61806f310 2024-03-29T10:06:31.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.517+0000 7ff60e000700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5f806ae90 0x7ff5f806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:31.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.518+0000 7ff60e000700 1 -- 172.21.15.12:0/241746744 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff60808ac00 con 0x7ff61806f310 2024-03-29T10:06:31.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.518+0000 7ff615400700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5f806ae90 0x7ff5f806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:31.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.519+0000 7ff615400700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5f806ae90 0x7ff5f806d350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff604009d70 tx=0x7ff604009210 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:31.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.522+0000 7ff60e000700 1 -- 172.21.15.12:0/241746744 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff608091080 con 0x7ff61806f310 2024-03-29T10:06:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.761+0000 7ff61ce00700 1 -- 172.21.15.12:0/241746744 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff61809a950 con 0x7ff5f806ae90 2024-03-29T10:06:31.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.771+0000 7ff60e000700 1 -- 172.21.15.12:0/241746744 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7ff6040099c0 con 0x7ff5f806ae90 2024-03-29T10:06:31.771 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:31.771 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:58", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:31.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 -- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5f806ae90 msgr2=0x7ff5f806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:31.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5f806ae90 0x7ff5f806d350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff604009d70 tx=0x7ff604009210 comp rx=0 tx=0).stop 2024-03-29T10:06:31.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 -- 172.21.15.12:0/241746744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff61806f310 msgr2=0x7ff6180b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:31.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff61806f310 0x7ff6180b50b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff608009c40 tx=0x7ff608004750 comp rx=0 tx=0).stop 2024-03-29T10:06:31.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 -- 172.21.15.12:0/241746744 shutdown_connections 2024-03-29T10:06:31.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5f806ae90 0x7ff5f806d350 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff61815cfe0 0x7ff61815d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff61806f310 0x7ff6180b50b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 --2- 172.21.15.12:0/241746744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6180cd5c0 0x7ff6181575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:31.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 -- 172.21.15.12:0/241746744 >> 172.21.15.12:0/241746744 conn(0x7ff6180d1370 msgr2=0x7ff6180b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:31.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.773+0000 7ff61ce00700 1 -- 172.21.15.12:0/241746744 shutdown_connections 2024-03-29T10:06:31.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:31.774+0000 7ff61ce00700 1 -- 172.21.15.12:0/241746744 wait complete. 2024-03-29T10:06:31.792 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:58", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:31.792 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T09:55:58', 'first': '2024-03-29T09:56:00', 'last': '2024-03-29T10:06:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-29T10:06:31.792 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-29T10:06:31.792 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-29T10:06:31.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:31.792 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-03-29T10:06:31.838 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:31.838 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:31.838 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:31.838 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:31.838 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:31.838 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:31.838 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:31.838 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:31.838 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:31.839 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:31.839 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:31.840 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-09_56_00_UTC', 'scheduled-2024-03-29-09_57_00_UTC', 'scheduled-2024-03-29-09_58_00_UTC', 'scheduled-2024-03-29-09_59_00_UTC', 'scheduled-2024-03-29-10_00_00_UTC', 'scheduled-2024-03-29-10_01_00_UTC', 'scheduled-2024-03-29-10_02_00_UTC', 'scheduled-2024-03-29-10_03_00_UTC', 'scheduled-2024-03-29-10_04_00_UTC', 'scheduled-2024-03-29-10_05_00_UTC', 'scheduled-2024-03-29-10_06_00_UTC'] 2024-03-29T10:06:31.840 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2024-03-29T10:06:32.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.082+0000 7fd042800700 1 Processor -- start 2024-03-29T10:06:32.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.082+0000 7fd042800700 1 -- start start 2024-03-29T10:06:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.083+0000 7fd042800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd03c1611b0 0x7fd03c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.083+0000 7fd042800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c15f4d0 0x7fd03c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.083+0000 7fd042800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03c0ce270 con 0x7fd03c0d21f0 2024-03-29T10:06:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.083+0000 7fd042800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd03c15b600 con 0x7fd03c1611b0 2024-03-29T10:06:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.083+0000 7fd042800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0d2760 con 0x7fd03c15f4d0 2024-03-29T10:06:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd03c0d21f0 0x7fd03c161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54264/0 (socket says 172.21.15.12:54264) 2024-03-29T10:06:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03be00700 1 -- 172.21.15.12:0/239884311 learned_addr learned my addr 172.21.15.12:0/239884311 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03aa00700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c15f4d0 0x7fd03c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03b400700 1 --2- 172.21.15.12:0/239884311 >> v2:172.21.15.159:3300/0 conn(0x7fd03c1611b0 0x7fd03c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707147783 0 0) 0x7fd03c0ce270 con 0x7fd03c0d21f0 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0200035b0 con 0x7fd03c0d21f0 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3152348468 0 0) 0x7fd0200035b0 con 0x7fd03c0d21f0 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c15f4d0 msgr2=0x7fd03c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03a000700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c15f4d0 0x7fd03c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 >> v2:172.21.15.159:3300/0 conn(0x7fd03c1611b0 msgr2=0x7fd03c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03a000700 1 --2- 172.21.15.12:0/239884311 >> v2:172.21.15.159:3300/0 conn(0x7fd03c1611b0 0x7fd03c15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c0d2a40 con 0x7fd03c0d21f0 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fd0340019f0 con 0x7fd03c0d21f0 2024-03-29T10:06:32.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.084+0000 7fd03b400700 1 --2- 172.21.15.12:0/239884311 >> v2:172.21.15.159:3300/0 conn(0x7fd03c1611b0 0x7fd03c15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 >> v1:172.21.15.12:6789/0 conn(0x7fd03c0d21f0 legacy=0x7fd03c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03a000700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd020003a00 0x7fd020003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03a000700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd020004850 0x7fd020004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03a000700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c15f4d0 0x7fd020005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c15b600 con 0x7fd020003a00 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0d2760 con 0x7fd020004850 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02000ba10 con 0x7fd03c15f4d0 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03b400700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd020003a00 0x7fd020003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03be00700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c15f4d0 0x7fd020005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03aa00700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd020004850 0x7fd020004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03b400700 1 -- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c15f4d0 msgr2=0x7fd020005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03b400700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c15f4d0 0x7fd020005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03b400700 1 -- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd020004850 msgr2=0x7fd020004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03b400700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd020004850 0x7fd020004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03b400700 1 -- 172.21.15.12:0/239884311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0200035b0 con 0x7fd020003a00 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.085+0000 7fd03aa00700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd020004850 0x7fd020004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd03b400700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd020003a00 0x7fd020003e20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd02c009d30 tx=0x7fd020005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3e03b563d5b3e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c009e80 con 0x7fd020003a00 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02c00e9a0 con 0x7fd020003a00 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd03a000700 1 -- 172.21.15.12:0/239884311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c00ed30 con 0x7fd020003a00 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd042800700 1 -- 172.21.15.12:0/239884311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd020003a00 msgr2=0x7fd020003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd042800700 1 --2- 172.21.15.12:0/239884311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd020003a00 0x7fd020003e20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd02c009d30 tx=0x7fd020005880 comp rx=0 tx=0).stop 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd03be00700 1 -- 172.21.15.12:0/239884311 reap_dead start 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd042800700 1 -- 172.21.15.12:0/239884311 shutdown_connections 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd042800700 1 -- 172.21.15.12:0/239884311 >> 172.21.15.12:0/239884311 conn(0x7fd03c08aff0 msgr2=0x7fd03c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd042800700 1 -- 172.21.15.12:0/239884311 shutdown_connections 2024-03-29T10:06:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.086+0000 7fd042800700 1 -- 172.21.15.12:0/239884311 wait complete. 2024-03-29T10:06:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.088+0000 7fd042800700 1 Processor -- start 2024-03-29T10:06:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.088+0000 7fd042800700 1 -- start start 2024-03-29T10:06:32.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.089+0000 7fd042800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd020003a00 0x7fd03c0b8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.089+0000 7fd042800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c0cbb50 0x7fd03c157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.089+0000 7fd042800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0cb280 0x7fd03c1f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.089+0000 7fd042800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0cae60 con 0x7fd03c0cb280 2024-03-29T10:06:32.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.089+0000 7fd042800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c1f3ae0 con 0x7fd020003a00 2024-03-29T10:06:32.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.089+0000 7fd042800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0cc3a0 con 0x7fd03c0cbb50 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0cb280 0x7fd03c1f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0cb280 0x7fd03c1f2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53128/0 (socket says 172.21.15.12:53128) 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03be00700 1 -- 172.21.15.12:0/1597765536 learned_addr learned my addr 172.21.15.12:0/1597765536 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03b400700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd020003a00 0x7fd03c0b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03aa00700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c0cbb50 0x7fd03c157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03be00700 1 -- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c0cbb50 msgr2=0x7fd03c157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03be00700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c0cbb50 0x7fd03c157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03be00700 1 -- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd020003a00 msgr2=0x7fd03c0b8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03be00700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd020003a00 0x7fd03c0b8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03be00700 1 -- 172.21.15.12:0/1597765536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c0cc680 con 0x7fd03c0cb280 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03aa00700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c0cbb50 0x7fd03c157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.090+0000 7fd03be00700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0cb280 0x7fd03c1f2fe0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd034000f80 tx=0x7fd03400a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:32.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.091+0000 7fd033400700 1 -- 172.21.15.12:0/1597765536 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03400ad50 con 0x7fd03c0cb280 2024-03-29T10:06:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.091+0000 7fd033400700 1 -- 172.21.15.12:0/1597765536 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0340133f0 con 0x7fd03c0cb280 2024-03-29T10:06:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.091+0000 7fd033400700 1 -- 172.21.15.12:0/1597765536 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0340137a0 con 0x7fd03c0cb280 2024-03-29T10:06:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.091+0000 7fd042800700 1 -- 172.21.15.12:0/1597765536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03c0ca7a0 con 0x7fd03c0cb280 2024-03-29T10:06:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.091+0000 7fd042800700 1 -- 172.21.15.12:0/1597765536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03c157660 con 0x7fd03c0cb280 2024-03-29T10:06:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.092+0000 7fd042800700 1 -- 172.21.15.12:0/1597765536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03c0ca600 con 0x7fd03c0cb280 2024-03-29T10:06:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.093+0000 7fd033400700 1 -- 172.21.15.12:0/1597765536 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fd034013900 con 0x7fd03c0cb280 2024-03-29T10:06:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.094+0000 7fd033400700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd01c06aec0 0x7fd01c06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.094+0000 7fd033400700 1 -- 172.21.15.12:0/1597765536 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fd034089fe0 con 0x7fd03c0cb280 2024-03-29T10:06:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.094+0000 7fd03b400700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd01c06aec0 0x7fd01c06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.095+0000 7fd03b400700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd01c06aec0 0x7fd01c06d380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0090b0 tx=0x7fd02c01f040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:32.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.098+0000 7fd033400700 1 -- 172.21.15.12:0/1597765536 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd03c0ca600 con 0x7fd03c0cb280 2024-03-29T10:06:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.338+0000 7fd042800700 1 -- 172.21.15.12:0/1597765536 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd03c0cdf00 con 0x7fd01c06aec0 2024-03-29T10:06:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.348+0000 7fd033400700 1 -- 172.21.15.12:0/1597765536 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fd02c00ec10 con 0x7fd01c06aec0 2024-03-29T10:06:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:32.349 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:59", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 -- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd01c06aec0 msgr2=0x7fd01c06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd01c06aec0 0x7fd01c06d380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0090b0 tx=0x7fd02c01f040 comp rx=0 tx=0).stop 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 -- 172.21.15.12:0/1597765536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0cb280 msgr2=0x7fd03c1f2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0cb280 0x7fd03c1f2fe0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd034000f80 tx=0x7fd03400a640 comp rx=0 tx=0).stop 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 -- 172.21.15.12:0/1597765536 shutdown_connections 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd01c06aec0 0x7fd01c06d380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd03c0cb280 0x7fd03c1f2fe0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd020003a00 0x7fd03c0b8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 --2- 172.21.15.12:0/1597765536 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd03c0cbb50 0x7fd03c157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 -- 172.21.15.12:0/1597765536 >> 172.21.15.12:0/1597765536 conn(0x7fd03c08aff0 msgr2=0x7fd03c15f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 -- 172.21.15.12:0/1597765536 shutdown_connections 2024-03-29T10:06:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.351+0000 7fd042800700 1 -- 172.21.15.12:0/1597765536 wait complete. 2024-03-29T10:06:32.369 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T09:55:59", "first": "2024-03-29T09:56:00", "last": "2024-03-29T10:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-29T10:06:32.370 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T09:55:59', 'first': '2024-03-29T09:56:00', 'last': '2024-03-29T10:06:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-29T10:06:32.370 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-29T10:06:32.370 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-29T10:06:32.370 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2024-03-29T10:06:32.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.572+0000 7fa49ba00700 1 Processor -- start 2024-03-29T10:06:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.572+0000 7fa49ba00700 1 -- start start 2024-03-29T10:06:32.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.573+0000 7fa49ba00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa49415d9d0 0x7fa4940d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.573+0000 7fa49ba00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa4940d16a0 0x7fa4940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.573+0000 7fa49ba00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa494161420 con 0x7fa4940d2370 2024-03-29T10:06:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.573+0000 7fa49ba00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4940d27e0 con 0x7fa4940d16a0 2024-03-29T10:06:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.573+0000 7fa49ba00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa494161700 con 0x7fa49415d9d0 2024-03-29T10:06:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.573+0000 7fa493e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa4940d16a0 0x7fa4940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.573+0000 7fa493e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa4940d16a0 0x7fa4940d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53142/0 (socket says 172.21.15.12:53142) 2024-03-29T10:06:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.573+0000 7fa493e00700 1 -- 172.21.15.12:0/1254161657 learned_addr learned my addr 172.21.15.12:0/1254161657 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.573+0000 7fa499000700 1 --2- 172.21.15.12:0/1254161657 >> v2:172.21.15.159:3300/0 conn(0x7fa49415d9d0 0x7fa4940d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.574+0000 7fa493e00700 1 -- 172.21.15.12:0/1254161657 >> v2:172.21.15.159:3300/0 conn(0x7fa49415d9d0 msgr2=0x7fa4940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.574+0000 7fa493e00700 1 --2- 172.21.15.12:0/1254161657 >> v2:172.21.15.159:3300/0 conn(0x7fa49415d9d0 0x7fa4940d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.574+0000 7fa493e00700 1 -- 172.21.15.12:0/1254161657 >> v1:172.21.15.159:6789/0 conn(0x7fa4940d2370 legacy=0x7fa49415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.574+0000 7fa493e00700 1 -- 172.21.15.12:0/1254161657 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4941619e0 con 0x7fa4940d16a0 2024-03-29T10:06:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.574+0000 7fa493e00700 1 --2- 172.21.15.12:0/1254161657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4940d16a0 0x7fa4940d1e30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa4940d0ff0 tx=0x7fa480009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b90d3e2d474727 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.574+0000 7fa493400700 1 -- 172.21.15.12:0/1254161657 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480005e30 con 0x7fa4940d16a0 2024-03-29T10:06:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.574+0000 7fa493400700 1 -- 172.21.15.12:0/1254161657 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480004030 con 0x7fa4940d16a0 2024-03-29T10:06:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.574+0000 7fa493400700 1 -- 172.21.15.12:0/1254161657 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800043e0 con 0x7fa4940d16a0 2024-03-29T10:06:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.575+0000 7fa49ba00700 1 -- 172.21.15.12:0/1254161657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4940d16a0 msgr2=0x7fa4940d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.575+0000 7fa49ba00700 1 --2- 172.21.15.12:0/1254161657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4940d16a0 0x7fa4940d1e30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa4940d0ff0 tx=0x7fa480009bd0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.575+0000 7fa49ba00700 1 -- 172.21.15.12:0/1254161657 shutdown_connections 2024-03-29T10:06:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.575+0000 7fa49ba00700 1 --2- 172.21.15.12:0/1254161657 >> v2:172.21.15.159:3300/0 conn(0x7fa49415d9d0 0x7fa4940d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.575+0000 7fa49ba00700 1 --2- 172.21.15.12:0/1254161657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4940d16a0 0x7fa4940d1e30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.575+0000 7fa49ba00700 1 -- 172.21.15.12:0/1254161657 >> 172.21.15.12:0/1254161657 conn(0x7fa49408aff0 msgr2=0x7fa4940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.575+0000 7fa49ba00700 1 -- 172.21.15.12:0/1254161657 shutdown_connections 2024-03-29T10:06:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.575+0000 7fa49ba00700 1 -- 172.21.15.12:0/1254161657 wait complete. 2024-03-29T10:06:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.577+0000 7fa49ba00700 1 Processor -- start 2024-03-29T10:06:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.577+0000 7fa49ba00700 1 -- start start 2024-03-29T10:06:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.578+0000 7fa49ba00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4940d16a0 0x7fa494155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.578+0000 7fa49ba00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4940d2370 0x7fa4940cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.578+0000 7fa49ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa49415d9d0 0x7fa4940b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.578+0000 7fa49ba00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940cc510 con 0x7fa4940d2370 2024-03-29T10:06:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.578+0000 7fa49ba00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940cb900 con 0x7fa4940d16a0 2024-03-29T10:06:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.578+0000 7fa49ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940cbbe0 con 0x7fa49415d9d0 2024-03-29T10:06:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.578+0000 7fa499000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4940d16a0 0x7fa494155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.578+0000 7fa499000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4940d16a0 0x7fa494155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:39874/0 (socket says 172.21.15.12:39874) 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.578+0000 7fa499000700 1 -- 172.21.15.12:0/786169132 learned_addr learned my addr 172.21.15.12:0/786169132 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.578+0000 7fa493e00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4940d2370 0x7fa4940cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa499a00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa49415d9d0 0x7fa4940b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa493e00700 1 -- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa49415d9d0 msgr2=0x7fa4940b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa493e00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa49415d9d0 0x7fa4940b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa493e00700 1 -- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4940d16a0 msgr2=0x7fa494155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa493e00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4940d16a0 0x7fa494155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa493e00700 1 -- 172.21.15.12:0/786169132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa480009500 con 0x7fa4940d2370 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa493e00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4940d2370 0x7fa4940cb000 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa480009fd0 tx=0x7fa480004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa490c00700 1 -- 172.21.15.12:0/786169132 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480003c70 con 0x7fa4940d2370 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa490c00700 1 -- 172.21.15.12:0/786169132 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480003df0 con 0x7fa4940d2370 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa49ba00700 1 -- 172.21.15.12:0/786169132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4940cbec0 con 0x7fa4940d2370 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.579+0000 7fa490c00700 1 -- 172.21.15.12:0/786169132 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480011660 con 0x7fa4940d2370 2024-03-29T10:06:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.580+0000 7fa49ba00700 1 -- 172.21.15.12:0/786169132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4940cb540 con 0x7fa4940d2370 2024-03-29T10:06:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.581+0000 7fa49ba00700 1 -- 172.21.15.12:0/786169132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4940cb6d0 con 0x7fa4940d2370 2024-03-29T10:06:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.581+0000 7fa490c00700 1 -- 172.21.15.12:0/786169132 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fa4800117c0 con 0x7fa4940d2370 2024-03-29T10:06:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.583+0000 7fa490c00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa47406ae90 0x7fa47406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.583+0000 7fa490c00700 1 -- 172.21.15.12:0/786169132 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa48008a3d0 con 0x7fa4940d2370 2024-03-29T10:06:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.583+0000 7fa499000700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa47406ae90 0x7fa47406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.584+0000 7fa499000700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa47406ae90 0x7fa47406d350 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa4940c9ca0 tx=0x7fa488008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.587+0000 7fa490c00700 1 -- 172.21.15.12:0/786169132 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4800553a0 con 0x7fa4940d2370 2024-03-29T10:06:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.827+0000 7fa49ba00700 1 -- 172.21.15.12:0/786169132 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa494161fa0 con 0x7fa47406ae90 2024-03-29T10:06:32.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.928+0000 7fa490c00700 1 -- 172.21.15.12:0/786169132 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fa488010740 con 0x7fa47406ae90 2024-03-29T10:06:32.928 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/dir0 2024-03-29T10:06:32.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 -- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa47406ae90 msgr2=0x7fa47406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa47406ae90 0x7fa47406d350 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa4940c9ca0 tx=0x7fa488008040 comp rx=0 tx=0).stop 2024-03-29T10:06:32.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 -- 172.21.15.12:0/786169132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4940d2370 msgr2=0x7fa4940cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:32.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4940d2370 0x7fa4940cb000 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa480009fd0 tx=0x7fa480004750 comp rx=0 tx=0).stop 2024-03-29T10:06:32.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 -- 172.21.15.12:0/786169132 shutdown_connections 2024-03-29T10:06:32.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa47406ae90 0x7fa47406d350 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa49415d9d0 0x7fa4940b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4940d16a0 0x7fa494155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 --2- 172.21.15.12:0/786169132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4940d2370 0x7fa4940cb000 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:32.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 -- 172.21.15.12:0/786169132 >> 172.21.15.12:0/786169132 conn(0x7fa49408aff0 msgr2=0x7fa4940b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:32.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 -- 172.21.15.12:0/786169132 shutdown_connections 2024-03-29T10:06:32.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:32.931+0000 7fa49ba00700 1 -- 172.21.15.12:0/786169132 wait complete. 2024-03-29T10:06:32.948 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2024-03-29T10:06:32.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:32.949 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-03-29T10:06:32.994 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:32.994 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:32.994 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:32.994 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:32.995 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:32.995 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:32.995 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:32.996 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:32.996 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:32.996 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:32.997 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:32.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:32.997 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-29-09_57_00_UTC) 2024-03-29T10:06:33.051 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:33.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:33.051 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-29-09_58_00_UTC) 2024-03-29T10:06:33.102 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:33.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:33.102 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-29-09_59_00_UTC) 2024-03-29T10:06:33.153 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:33.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:33.154 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_00_00_UTC) 2024-03-29T10:06:33.204 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:33.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:33.205 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_01_00_UTC) 2024-03-29T10:06:33.255 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:33.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:33.256 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_02_00_UTC) 2024-03-29T10:06:33.307 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:33.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:33.308 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_03_00_UTC) 2024-03-29T10:06:33.363 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:33.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:33.363 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_04_00_UTC) 2024-03-29T10:06:33.415 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:33.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:33.416 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_05_00_UTC) 2024-03-29T10:06:33.467 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:33.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:33.468 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-29-10_06_00_UTC) 2024-03-29T10:06:33.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:33.520 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2024-03-29T10:06:33.568 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2024-03-29T10:06:33.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.770+0000 7f05d2000700 1 Processor -- start 2024-03-29T10:06:33.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.770+0000 7f05d2000700 1 -- start start 2024-03-29T10:06:33.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.771+0000 7f05d2000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f05cc15d9d0 0x7f05cc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:33.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.771+0000 7f05d2000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc161420 con 0x7f05cc0d2540 2024-03-29T10:06:33.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.771+0000 7f05d2000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc161700 con 0x7f05cc0d16a0 2024-03-29T10:06:33.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.771+0000 7f05d2000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05cc15edc0 con 0x7f05cc15d9d0 2024-03-29T10:06:33.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.771+0000 7f05cbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f05cc0d2540 0x7f05cc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54276/0 (socket says 172.21.15.12:54276) 2024-03-29T10:06:33.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.771+0000 7f05cbe00700 1 -- 172.21.15.12:0/2251058425 learned_addr learned my addr 172.21.15.12:0/2251058425 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05cb400700 1 --2- 172.21.15.12:0/2251058425 >> v2:172.21.15.159:3300/0 conn(0x7f05cc15d9d0 0x7f05cc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361116619 0 0) 0x7f05cc161420 con 0x7f05cc0d2540 2024-03-29T10:06:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b00035b0 con 0x7f05cc0d2540 2024-03-29T10:06:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3522941346 0 0) 0x7f05b00035b0 con 0x7f05cc0d2540 2024-03-29T10:06:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 >> v2:172.21.15.159:3300/0 conn(0x7f05cc15d9d0 msgr2=0x7f05cc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 --2- 172.21.15.12:0/2251058425 >> v2:172.21.15.159:3300/0 conn(0x7f05cc15d9d0 0x7f05cc0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 >> v1:172.21.15.159:6789/0 conn(0x7f05cc0d16a0 legacy=0x7f05cc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc1619e0 con 0x7f05cc0d2540 2024-03-29T10:06:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f05c40019f0 con 0x7f05cc0d2540 2024-03-29T10:06:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 >> v1:172.21.15.12:6789/0 conn(0x7f05cc0d2540 legacy=0x7f05cc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05b0003a00 0x7f05b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05b0004400 0x7f05b00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05b0005170 0x7f05b000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b000c080 con 0x7f05b0004400 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b000c1f0 con 0x7f05b0005170 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.772+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b000c3b0 con 0x7f05b0003a00 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.773+0000 7f05caa00700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05b0004400 0x7f05b00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.773+0000 7f05cb400700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05b0003a00 0x7f05b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.773+0000 7f05cbe00700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05b0005170 0x7f05b000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.773+0000 7f05caa00700 1 -- 172.21.15.12:0/2251058425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05b0003a00 msgr2=0x7f05b0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.773+0000 7f05caa00700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05b0003a00 0x7f05b0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.773+0000 7f05caa00700 1 -- 172.21.15.12:0/2251058425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05b0005170 msgr2=0x7f05b000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:33.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.773+0000 7f05caa00700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05b0005170 0x7f05b000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.773+0000 7f05cbe00700 1 -- 172.21.15.12:0/2251058425 reap_dead start 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.773+0000 7f05caa00700 1 -- 172.21.15.12:0/2251058425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b00035b0 con 0x7f05b0004400 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.773+0000 7f05caa00700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05b0004400 0x7f05b00048d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f05b8001900 tx=0x7f05b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27103c8899ac0453 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.774+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b80013a0 con 0x7f05b0004400 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.774+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b80052d0 con 0x7f05b0004400 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.774+0000 7f05ca000700 1 -- 172.21.15.12:0/2251058425 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b80036d0 con 0x7f05b0004400 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.774+0000 7f05d2000700 1 -- 172.21.15.12:0/2251058425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05b0004400 msgr2=0x7f05b00048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.774+0000 7f05d2000700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05b0004400 0x7f05b00048d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f05b8001900 tx=0x7f05b0005880 comp rx=0 tx=0).stop 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.774+0000 7f05d2000700 1 -- 172.21.15.12:0/2251058425 shutdown_connections 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.774+0000 7f05d2000700 1 --2- 172.21.15.12:0/2251058425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05b0004400 0x7f05b00048d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.774+0000 7f05d2000700 1 -- 172.21.15.12:0/2251058425 >> 172.21.15.12:0/2251058425 conn(0x7f05cc08aff0 msgr2=0x7f05cc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.774+0000 7f05d2000700 1 -- 172.21.15.12:0/2251058425 shutdown_connections 2024-03-29T10:06:33.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.774+0000 7f05d2000700 1 -- 172.21.15.12:0/2251058425 wait complete. 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.776+0000 7f05d2000700 1 Processor -- start 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.776+0000 7f05d2000700 1 -- start start 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05d2000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05b0004400 0x7f05cc1f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05d2000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05cc0ca660 0x7f05cc0cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05d2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05cc0cc530 0x7f05cc0cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05d2000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc0ccf00 con 0x7f05cc0cc530 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05d2000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc0b60c0 con 0x7f05cc0ca660 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05d2000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc0b63a0 con 0x7f05b0004400 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05cc0cc530 0x7f05cc0cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05cc0cc530 0x7f05cc0cc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53170/0 (socket says 172.21.15.12:53170) 2024-03-29T10:06:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05cbe00700 1 -- 172.21.15.12:0/456838037 learned_addr learned my addr 172.21.15.12:0/456838037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05cb400700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05b0004400 0x7f05cc1f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.777+0000 7f05caa00700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05cc0ca660 0x7f05cc0cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05cbe00700 1 -- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05b0004400 msgr2=0x7f05cc1f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05cbe00700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05b0004400 0x7f05cc1f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05cbe00700 1 -- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05cc0ca660 msgr2=0x7f05cc0cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05cbe00700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05cc0ca660 0x7f05cc0cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05cbe00700 1 -- 172.21.15.12:0/456838037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc0b6680 con 0x7f05cc0cc530 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05cbe00700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05cc0cc530 0x7f05cc0cc9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f05c4000f80 tx=0x7f05c400a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05c3400700 1 -- 172.21.15.12:0/456838037 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c400adb0 con 0x7f05cc0cc530 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05c3400700 1 -- 172.21.15.12:0/456838037 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c40133f0 con 0x7f05cc0cc530 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05d2000700 1 -- 172.21.15.12:0/456838037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05cc0b6960 con 0x7f05cc0cc530 2024-03-29T10:06:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05c3400700 1 -- 172.21.15.12:0/456838037 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c40137a0 con 0x7f05cc0cc530 2024-03-29T10:06:33.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.778+0000 7f05d2000700 1 -- 172.21.15.12:0/456838037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05cc1f4960 con 0x7f05cc0cc530 2024-03-29T10:06:33.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.779+0000 7f05d2000700 1 -- 172.21.15.12:0/456838037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05cc0b6e80 con 0x7f05cc0cc530 2024-03-29T10:06:33.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.780+0000 7f05c3400700 1 -- 172.21.15.12:0/456838037 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f05c4013900 con 0x7f05cc0cc530 2024-03-29T10:06:33.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.781+0000 7f05c3400700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05ac06aee0 0x7f05ac06d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:33.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.781+0000 7f05c3400700 1 -- 172.21.15.12:0/456838037 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f05c4089e80 con 0x7f05cc0cc530 2024-03-29T10:06:33.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.782+0000 7f05cb400700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05ac06aee0 0x7f05ac06d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:33.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.783+0000 7f05cb400700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05ac06aee0 0x7f05ac06d3a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009c60 tx=0x7f05bc009210 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:33.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:33.786+0000 7f05c3400700 1 -- 172.21.15.12:0/456838037 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f05cc0b6e80 con 0x7f05cc0cc530 2024-03-29T10:06:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.028+0000 7f05d2000700 1 -- 172.21.15.12:0/456838037 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f05cc161fa0 con 0x7f05ac06aee0 2024-03-29T10:06:34.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.131+0000 7f05c3400700 1 -- 172.21.15.12:0/456838037 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f05bc019070 con 0x7f05ac06aee0 2024-03-29T10:06:34.131 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/dir1 2024-03-29T10:06:34.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 -- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05ac06aee0 msgr2=0x7f05ac06d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:34.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05ac06aee0 0x7f05ac06d3a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009c60 tx=0x7f05bc009210 comp rx=0 tx=0).stop 2024-03-29T10:06:34.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 -- 172.21.15.12:0/456838037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05cc0cc530 msgr2=0x7f05cc0cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:34.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05cc0cc530 0x7f05cc0cc9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f05c4000f80 tx=0x7f05c400a640 comp rx=0 tx=0).stop 2024-03-29T10:06:34.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 -- 172.21.15.12:0/456838037 shutdown_connections 2024-03-29T10:06:34.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05ac06aee0 0x7f05ac06d3a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:34.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05cc0cc530 0x7f05cc0cc9c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:34.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05b0004400 0x7f05cc1f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:34.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 --2- 172.21.15.12:0/456838037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05cc0ca660 0x7f05cc0cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:34.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 -- 172.21.15.12:0/456838037 >> 172.21.15.12:0/456838037 conn(0x7f05cc08aff0 msgr2=0x7f05cc0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:34.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 -- 172.21.15.12:0/456838037 shutdown_connections 2024-03-29T10:06:34.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:34.134+0000 7f05d2000700 1 -- 172.21.15.12:0/456838037 wait complete. 2024-03-29T10:06:34.152 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2024-03-29T10:06:34.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.153 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-03-29T10:06:34.199 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:34.199 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:34.199 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:34.199 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:34.199 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:34.199 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:34.199 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:34.199 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:34.199 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:34.200 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:34.200 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:34.201 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:34.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.201 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-09_56_00_UTC) 2024-03-29T10:06:34.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:34.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.296 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-09_57_00_UTC) 2024-03-29T10:06:34.347 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:34.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.348 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-09_58_00_UTC) 2024-03-29T10:06:34.399 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:34.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.400 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-09_59_00_UTC) 2024-03-29T10:06:34.452 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:34.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.453 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_00_00_UTC) 2024-03-29T10:06:34.505 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:34.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.506 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_01_00_UTC) 2024-03-29T10:06:34.558 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:34.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.558 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_02_00_UTC) 2024-03-29T10:06:34.610 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:34.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.610 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_03_00_UTC) 2024-03-29T10:06:34.662 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:34.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.663 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_04_00_UTC) 2024-03-29T10:06:34.714 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:34.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.714 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_05_00_UTC) 2024-03-29T10:06:34.766 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:34.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.767 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-29-10_06_00_UTC) 2024-03-29T10:06:34.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:34.819 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2024-03-29T10:06:34.866 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2024-03-29T10:06:35.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.069+0000 7f2c81800700 1 Processor -- start 2024-03-29T10:06:35.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.069+0000 7f2c81800700 1 -- start start 2024-03-29T10:06:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.070+0000 7f2c81800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2c7c0cd480 0x7f2c7c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.070+0000 7f2c81800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c0cdae0 con 0x7f2c7c0cd020 2024-03-29T10:06:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.070+0000 7f2c81800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c0cddc0 con 0x7f2c7c0cbb30 2024-03-29T10:06:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.070+0000 7f2c81800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7c0ce0a0 con 0x7f2c7c0cd480 2024-03-29T10:06:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.070+0000 7f2c7a000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2c7c0cd020 0x7f2c7c0b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54278/0 (socket says 172.21.15.12:54278) 2024-03-29T10:06:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.070+0000 7f2c7a000700 1 -- 172.21.15.12:0/1497421784 learned_addr learned my addr 172.21.15.12:0/1497421784 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:35.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.071+0000 7f2c7b400700 1 --2- 172.21.15.12:0/1497421784 >> v2:172.21.15.12:3300/0 conn(0x7f2c7c0cd480 0x7f2c7c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:35.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.071+0000 7f2c7b400700 1 -- 172.21.15.12:0/1497421784 >> v1:172.21.15.159:6789/0 conn(0x7f2c7c0cbb30 legacy=0x7f2c7c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:35.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.071+0000 7f2c7b400700 1 -- 172.21.15.12:0/1497421784 >> v1:172.21.15.12:6789/0 conn(0x7f2c7c0cd020 legacy=0x7f2c7c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:35.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.071+0000 7f2c7b400700 1 -- 172.21.15.12:0/1497421784 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c06dc20 con 0x7f2c7c0cd480 2024-03-29T10:06:35.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.071+0000 7f2c7b400700 1 --2- 172.21.15.12:0/1497421784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 0x7f2c7c06e420 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009d90 tx=0x7f2c7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ba1b6e4bd8f9ad server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:35.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.072+0000 7f2c79600700 1 -- 172.21.15.12:0/1497421784 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7400e070 con 0x7f2c7c0cd480 2024-03-29T10:06:35.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.072+0000 7f2c79600700 1 -- 172.21.15.12:0/1497421784 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7400bbd0 con 0x7f2c7c0cd480 2024-03-29T10:06:35.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.072+0000 7f2c79600700 1 -- 172.21.15.12:0/1497421784 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74012430 con 0x7f2c7c0cd480 2024-03-29T10:06:35.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.072+0000 7f2c81800700 1 -- 172.21.15.12:0/1497421784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 msgr2=0x7f2c7c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:35.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.072+0000 7f2c81800700 1 --2- 172.21.15.12:0/1497421784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 0x7f2c7c06e420 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009d90 tx=0x7f2c7400b330 comp rx=0 tx=0).stop 2024-03-29T10:06:35.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.072+0000 7f2c81800700 1 -- 172.21.15.12:0/1497421784 shutdown_connections 2024-03-29T10:06:35.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.072+0000 7f2c81800700 1 --2- 172.21.15.12:0/1497421784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 0x7f2c7c06e420 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:35.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.072+0000 7f2c81800700 1 -- 172.21.15.12:0/1497421784 >> 172.21.15.12:0/1497421784 conn(0x7f2c7c0b0d70 msgr2=0x7f2c7c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:35.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.072+0000 7f2c81800700 1 -- 172.21.15.12:0/1497421784 shutdown_connections 2024-03-29T10:06:35.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.072+0000 7f2c81800700 1 -- 172.21.15.12:0/1497421784 wait complete. 2024-03-29T10:06:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.074+0000 7f2c81800700 1 Processor -- start 2024-03-29T10:06:35.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.074+0000 7f2c81800700 1 -- start start 2024-03-29T10:06:35.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.075+0000 7f2c81800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c7c0cbb30 0x7f2c7c14d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:35.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.075+0000 7f2c81800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c7c0cd020 0x7f2c7c0ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:35.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.075+0000 7f2c81800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 0x7f2c7c0afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:35.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.075+0000 7f2c81800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c14db70 con 0x7f2c7c0cd480 2024-03-29T10:06:35.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.075+0000 7f2c81800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c157e40 con 0x7f2c7c0cbb30 2024-03-29T10:06:35.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.075+0000 7f2c81800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0ca730 con 0x7f2c7c0cd020 2024-03-29T10:06:35.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.075+0000 7f2c7b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 0x7f2c7c0afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:35.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.075+0000 7f2c7b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 0x7f2c7c0afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53186/0 (socket says 172.21.15.12:53186) 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.075+0000 7f2c7b400700 1 -- 172.21.15.12:0/535846373 learned_addr learned my addr 172.21.15.12:0/535846373 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.075+0000 7f2c7aa00700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c7c0cbb30 0x7f2c7c14d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c7a000700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c7c0cd020 0x7f2c7c0ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c7b400700 1 -- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c7c0cd020 msgr2=0x7f2c7c0ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c7b400700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c7c0cd020 0x7f2c7c0ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c7b400700 1 -- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c7c0cbb30 msgr2=0x7f2c7c14d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c7b400700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c7c0cbb30 0x7f2c7c14d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c7b400700 1 -- 172.21.15.12:0/535846373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c74009a70 con 0x7f2c7c0cd480 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c7a000700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c7c0cd020 0x7f2c7c0ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c7b400700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 0x7f2c7c0afa40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400b730 tx=0x7f2c74004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c72a00700 1 -- 172.21.15.12:0/535846373 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74012e00 con 0x7f2c7c0cd480 2024-03-29T10:06:35.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c72a00700 1 -- 172.21.15.12:0/535846373 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c74020070 con 0x7f2c7c0cd480 2024-03-29T10:06:35.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c72a00700 1 -- 172.21.15.12:0/535846373 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c740276c0 con 0x7f2c7c0cd480 2024-03-29T10:06:35.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.076+0000 7f2c81800700 1 -- 172.21.15.12:0/535846373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c0c9d60 con 0x7f2c7c0cd480 2024-03-29T10:06:35.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.077+0000 7f2c81800700 1 -- 172.21.15.12:0/535846373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c0c9430 con 0x7f2c7c0cd480 2024-03-29T10:06:35.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.078+0000 7f2c81800700 1 -- 172.21.15.12:0/535846373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7c0c9ef0 con 0x7f2c7c0cd480 2024-03-29T10:06:35.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.079+0000 7f2c72a00700 1 -- 172.21.15.12:0/535846373 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f2c7401c400 con 0x7f2c7c0cd480 2024-03-29T10:06:35.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.079+0000 7f2c72a00700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c5c06ae90 0x7f2c5c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:35.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.079+0000 7f2c72a00700 1 -- 172.21.15.12:0/535846373 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f2c74022070 con 0x7f2c7c0cd480 2024-03-29T10:06:35.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.080+0000 7f2c7aa00700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c5c06ae90 0x7f2c5c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:35.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.081+0000 7f2c7aa00700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c5c06ae90 0x7f2c5c06d350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c007fd0 tx=0x7f2c6c009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:35.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.084+0000 7f2c72a00700 1 -- 172.21.15.12:0/535846373 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2c74056450 con 0x7f2c7c0cd480 2024-03-29T10:06:35.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.326+0000 7f2c81800700 1 -- 172.21.15.12:0/535846373 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c7c15bdd0 con 0x7f2c5c06ae90 2024-03-29T10:06:35.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.432+0000 7f2c72a00700 1 -- 172.21.15.12:0/535846373 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f2c6c0014c0 con 0x7f2c5c06ae90 2024-03-29T10:06:35.432 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/dir2 2024-03-29T10:06:35.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 -- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c5c06ae90 msgr2=0x7f2c5c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:35.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c5c06ae90 0x7f2c5c06d350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c007fd0 tx=0x7f2c6c009040 comp rx=0 tx=0).stop 2024-03-29T10:06:35.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 -- 172.21.15.12:0/535846373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 msgr2=0x7f2c7c0afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:35.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 0x7f2c7c0afa40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400b730 tx=0x7f2c74004ad0 comp rx=0 tx=0).stop 2024-03-29T10:06:35.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 -- 172.21.15.12:0/535846373 shutdown_connections 2024-03-29T10:06:35.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c5c06ae90 0x7f2c5c06d350 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:35.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c7c0cd480 0x7f2c7c0afa40 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c7c0cbb30 0x7f2c7c14d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 --2- 172.21.15.12:0/535846373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c7c0cd020 0x7f2c7c0ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 -- 172.21.15.12:0/535846373 >> 172.21.15.12:0/535846373 conn(0x7f2c7c0b0d70 msgr2=0x7f2c7c15dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 -- 172.21.15.12:0/535846373 shutdown_connections 2024-03-29T10:06:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:35.435+0000 7f2c81800700 1 -- 172.21.15.12:0/535846373 wait complete. 2024-03-29T10:06:35.453 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2024-03-29T10:06:35.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.453 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-03-29T10:06:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:35.500 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:35.500 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:35.500 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:35.501 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:35.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.501 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-09_56_00_UTC) 2024-03-29T10:06:35.555 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:35.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.555 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-09_57_00_UTC) 2024-03-29T10:06:35.607 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:35.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.607 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-09_58_00_UTC) 2024-03-29T10:06:35.660 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:35.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.660 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-09_59_00_UTC) 2024-03-29T10:06:35.711 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:35.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.712 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_00_00_UTC) 2024-03-29T10:06:35.764 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:35.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.765 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_01_00_UTC) 2024-03-29T10:06:35.816 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:35.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.817 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_02_00_UTC) 2024-03-29T10:06:35.869 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:35.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.870 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_03_00_UTC) 2024-03-29T10:06:35.922 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:35.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.922 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_04_00_UTC) 2024-03-29T10:06:35.975 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:35.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:35.975 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_05_00_UTC) 2024-03-29T10:06:36.027 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:36.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:36.027 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-29-10_06_00_UTC) 2024-03-29T10:06:36.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:36.081 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2024-03-29T10:06:36.128 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2024-03-29T10:06:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.331+0000 7fb759e00700 1 Processor -- start 2024-03-29T10:06:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.331+0000 7fb759e00700 1 -- start start 2024-03-29T10:06:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.332+0000 7fb759e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb7540d16a0 0x7fb7540d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.332+0000 7fb759e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb7540d2370 0x7fb7540ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.332+0000 7fb759e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb754161420 con 0x7fb75415d9d0 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.332+0000 7fb759e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7540d4340 con 0x7fb7540d2370 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.332+0000 7fb759e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb75415edc0 con 0x7fb7540d16a0 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.332+0000 7fb753400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb7540d2370 0x7fb7540ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.332+0000 7fb753400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb7540d2370 0x7fb7540ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53198/0 (socket says 172.21.15.12:53198) 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.333+0000 7fb753400700 1 -- 172.21.15.12:0/3289451461 learned_addr learned my addr 172.21.15.12:0/3289451461 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.333+0000 7fb752000700 1 --2- 172.21.15.12:0/3289451461 >> v2:172.21.15.159:3300/0 conn(0x7fb7540d16a0 0x7fb7540d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.333+0000 7fb753400700 1 -- 172.21.15.12:0/3289451461 >> v2:172.21.15.159:3300/0 conn(0x7fb7540d16a0 msgr2=0x7fb7540d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.333+0000 7fb753400700 1 --2- 172.21.15.12:0/3289451461 >> v2:172.21.15.159:3300/0 conn(0x7fb7540d16a0 0x7fb7540d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.333+0000 7fb753400700 1 -- 172.21.15.12:0/3289451461 >> v1:172.21.15.159:6789/0 conn(0x7fb75415d9d0 legacy=0x7fb7540d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.333+0000 7fb753400700 1 -- 172.21.15.12:0/3289451461 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb754161700 con 0x7fb7540d2370 2024-03-29T10:06:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.333+0000 7fb753400700 1 --2- 172.21.15.12:0/3289451461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d2370 0x7fb7540ce360 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb7540d12f0 tx=0x7fb74c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8442bd44e2271682 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:36.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.333+0000 7fb751600700 1 -- 172.21.15.12:0/3289451461 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c00e070 con 0x7fb7540d2370 2024-03-29T10:06:36.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.334+0000 7fb751600700 1 -- 172.21.15.12:0/3289451461 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74c00b9c0 con 0x7fb7540d2370 2024-03-29T10:06:36.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.334+0000 7fb751600700 1 -- 172.21.15.12:0/3289451461 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c00bd70 con 0x7fb7540d2370 2024-03-29T10:06:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.334+0000 7fb759e00700 1 -- 172.21.15.12:0/3289451461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d2370 msgr2=0x7fb7540ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.334+0000 7fb759e00700 1 --2- 172.21.15.12:0/3289451461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d2370 0x7fb7540ce360 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb7540d12f0 tx=0x7fb74c00b330 comp rx=0 tx=0).stop 2024-03-29T10:06:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.334+0000 7fb759e00700 1 -- 172.21.15.12:0/3289451461 shutdown_connections 2024-03-29T10:06:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.334+0000 7fb759e00700 1 --2- 172.21.15.12:0/3289451461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d2370 0x7fb7540ce360 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.334+0000 7fb759e00700 1 --2- 172.21.15.12:0/3289451461 >> v2:172.21.15.159:3300/0 conn(0x7fb7540d16a0 0x7fb7540d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.334+0000 7fb759e00700 1 -- 172.21.15.12:0/3289451461 >> 172.21.15.12:0/3289451461 conn(0x7fb75408aff0 msgr2=0x7fb7540d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.334+0000 7fb759e00700 1 -- 172.21.15.12:0/3289451461 shutdown_connections 2024-03-29T10:06:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.334+0000 7fb759e00700 1 -- 172.21.15.12:0/3289451461 wait complete. 2024-03-29T10:06:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.336+0000 7fb759e00700 1 Processor -- start 2024-03-29T10:06:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.336+0000 7fb759e00700 1 -- start start 2024-03-29T10:06:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb759e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d16a0 0x7fb75415c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb759e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7540d2370 0x7fb75406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb759e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb75415d9d0 0x7fb75415bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb759e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7541e7e30 con 0x7fb7540d16a0 2024-03-29T10:06:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb759e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7541e8110 con 0x7fb75415d9d0 2024-03-29T10:06:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb759e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7541e83f0 con 0x7fb7540d2370 2024-03-29T10:06:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb752a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d16a0 0x7fb75415c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb752a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d16a0 0x7fb75415c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53210/0 (socket says 172.21.15.12:53210) 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb752a00700 1 -- 172.21.15.12:0/1598995982 learned_addr learned my addr 172.21.15.12:0/1598995982 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb752000700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7540d2370 0x7fb75406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.337+0000 7fb753400700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb75415d9d0 0x7fb75415bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb752a00700 1 -- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7540d2370 msgr2=0x7fb75406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb752a00700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7540d2370 0x7fb75406bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb752a00700 1 -- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb75415d9d0 msgr2=0x7fb75415bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb752a00700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb75415d9d0 0x7fb75415bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb752a00700 1 -- 172.21.15.12:0/1598995982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c009b30 con 0x7fb7540d16a0 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb752a00700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d16a0 0x7fb75415c8b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb74400c450 tx=0x7fb74400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb74b400700 1 -- 172.21.15.12:0/1598995982 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440052d0 con 0x7fb7540d16a0 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb74b400700 1 -- 172.21.15.12:0/1598995982 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb744005430 con 0x7fb7540d16a0 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb759e00700 1 -- 172.21.15.12:0/1598995982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7541e86d0 con 0x7fb7540d16a0 2024-03-29T10:06:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb74b400700 1 -- 172.21.15.12:0/1598995982 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440038c0 con 0x7fb7540d16a0 2024-03-29T10:06:36.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.338+0000 7fb759e00700 1 -- 172.21.15.12:0/1598995982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75415cdf0 con 0x7fb7540d16a0 2024-03-29T10:06:36.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.339+0000 7fb759e00700 1 -- 172.21.15.12:0/1598995982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7540d1ca0 con 0x7fb7540d16a0 2024-03-29T10:06:36.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.340+0000 7fb74b400700 1 -- 172.21.15.12:0/1598995982 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fb744003a20 con 0x7fb7540d16a0 2024-03-29T10:06:36.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.342+0000 7fb74b400700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb73406ae90 0x7fb73406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:36.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.342+0000 7fb74b400700 1 -- 172.21.15.12:0/1598995982 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb744088a30 con 0x7fb7540d16a0 2024-03-29T10:06:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.342+0000 7fb752000700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb73406ae90 0x7fb73406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.343+0000 7fb752000700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb73406ae90 0x7fb73406d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb740006fd0 tx=0x7fb740008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.346+0000 7fb74b400700 1 -- 172.21.15.12:0/1598995982 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb744053920 con 0x7fb7540d16a0 2024-03-29T10:06:36.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.585+0000 7fb759e00700 1 -- 172.21.15.12:0/1598995982 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb754161fa0 con 0x7fb73406ae90 2024-03-29T10:06:36.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.688+0000 7fb74b400700 1 -- 172.21.15.12:0/1598995982 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fb740009e40 con 0x7fb73406ae90 2024-03-29T10:06:36.688 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/dir3 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.690+0000 7fb759e00700 1 -- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb73406ae90 msgr2=0x7fb73406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.690+0000 7fb759e00700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb73406ae90 0x7fb73406d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb740006fd0 tx=0x7fb740008040 comp rx=0 tx=0).stop 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.690+0000 7fb759e00700 1 -- 172.21.15.12:0/1598995982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d16a0 msgr2=0x7fb75415c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.690+0000 7fb759e00700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d16a0 0x7fb75415c8b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb74400c450 tx=0x7fb74400c760 comp rx=0 tx=0).stop 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.691+0000 7fb759e00700 1 -- 172.21.15.12:0/1598995982 shutdown_connections 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.691+0000 7fb759e00700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb73406ae90 0x7fb73406d350 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.691+0000 7fb759e00700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb75415d9d0 0x7fb75415bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.691+0000 7fb759e00700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7540d16a0 0x7fb75415c8b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.691+0000 7fb759e00700 1 --2- 172.21.15.12:0/1598995982 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7540d2370 0x7fb75406bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.691+0000 7fb759e00700 1 -- 172.21.15.12:0/1598995982 >> 172.21.15.12:0/1598995982 conn(0x7fb75408aff0 msgr2=0x7fb7540b0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.691+0000 7fb759e00700 1 -- 172.21.15.12:0/1598995982 shutdown_connections 2024-03-29T10:06:36.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:36.691+0000 7fb759e00700 1 -- 172.21.15.12:0/1598995982 wait complete. 2024-03-29T10:06:36.709 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2024-03-29T10:06:36.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:36.709 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-03-29T10:06:36.754 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:36.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:36.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:36.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:36.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:36.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:36.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:36.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:36.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:36.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:36.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:36.756 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:36.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:36.756 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-09_56_00_UTC) 2024-03-29T10:06:36.851 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:36.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:36.851 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-09_57_00_UTC) 2024-03-29T10:06:36.904 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:36.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:36.904 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-09_58_00_UTC) 2024-03-29T10:06:36.956 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:36.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:36.957 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-09_59_00_UTC) 2024-03-29T10:06:37.009 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:37.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:37.009 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_00_00_UTC) 2024-03-29T10:06:37.063 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:37.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:37.063 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_01_00_UTC) 2024-03-29T10:06:37.184 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:37.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:37.185 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_02_00_UTC) 2024-03-29T10:06:37.304 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:37.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:37.305 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_03_00_UTC) 2024-03-29T10:06:37.394 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:37.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:37.394 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_04_00_UTC) 2024-03-29T10:06:37.455 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:37.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:37.456 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_05_00_UTC) 2024-03-29T10:06:37.507 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:37.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:37.507 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-29-10_06_00_UTC) 2024-03-29T10:06:37.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:37.559 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2024-03-29T10:06:37.606 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2024-03-29T10:06:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.807+0000 7fe302600700 1 Processor -- start 2024-03-29T10:06:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.807+0000 7fe302600700 1 -- start start 2024-03-29T10:06:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.808+0000 7fe302600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2fc15dfe0 0x7fe2fc0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.808+0000 7fe302600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc0ce670 0x7fe2fc0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.808+0000 7fe302600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc0cb310 con 0x7fe2fc15a5c0 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.808+0000 7fe302600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2fc0ceb00 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.808+0000 7fe302600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0711a0 con 0x7fe2fc0ce670 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.808+0000 7fe2faa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2fc15dfe0 0x7fe2fc0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.808+0000 7fe2faa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2fc15dfe0 0x7fe2fc0ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53224/0 (socket says 172.21.15.12:53224) 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.808+0000 7fe2faa00700 1 -- 172.21.15.12:0/2446035158 learned_addr learned my addr 172.21.15.12:0/2446035158 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.808+0000 7fe2fbe00700 1 --2- 172.21.15.12:0/2446035158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc0ce670 0x7fe2fc0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.809+0000 7fe2faa00700 1 -- 172.21.15.12:0/2446035158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc0ce670 msgr2=0x7fe2fc0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.809+0000 7fe2faa00700 1 --2- 172.21.15.12:0/2446035158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc0ce670 0x7fe2fc0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.809+0000 7fe2faa00700 1 -- 172.21.15.12:0/2446035158 >> v1:172.21.15.159:6789/0 conn(0x7fe2fc15a5c0 legacy=0x7fe2fc0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.809+0000 7fe2faa00700 1 -- 172.21.15.12:0/2446035158 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc071480 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.809+0000 7fe2faa00700 1 --2- 172.21.15.12:0/2446035158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 0x7fe2fc0ce130 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe2e80098f0 tx=0x7fe2e8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938da3e5a1d233bc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.809+0000 7fe2fa000700 1 -- 172.21.15.12:0/2446035158 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800f8a0 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.809+0000 7fe2fa000700 1 -- 172.21.15.12:0/2446035158 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e800fa00 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.809+0000 7fe2fa000700 1 -- 172.21.15.12:0/2446035158 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800fd90 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.810+0000 7fe302600700 1 -- 172.21.15.12:0/2446035158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 msgr2=0x7fe2fc0ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.810+0000 7fe302600700 1 --2- 172.21.15.12:0/2446035158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 0x7fe2fc0ce130 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe2e80098f0 tx=0x7fe2e8009c00 comp rx=0 tx=0).stop 2024-03-29T10:06:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.810+0000 7fe302600700 1 -- 172.21.15.12:0/2446035158 shutdown_connections 2024-03-29T10:06:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.810+0000 7fe302600700 1 --2- 172.21.15.12:0/2446035158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc0ce670 0x7fe2fc0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.810+0000 7fe302600700 1 --2- 172.21.15.12:0/2446035158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 0x7fe2fc0ce130 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.810+0000 7fe302600700 1 -- 172.21.15.12:0/2446035158 >> 172.21.15.12:0/2446035158 conn(0x7fe2fc0b0d70 msgr2=0x7fe2fc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.810+0000 7fe302600700 1 -- 172.21.15.12:0/2446035158 shutdown_connections 2024-03-29T10:06:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.810+0000 7fe302600700 1 -- 172.21.15.12:0/2446035158 wait complete. 2024-03-29T10:06:37.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.812+0000 7fe302600700 1 Processor -- start 2024-03-29T10:06:37.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.812+0000 7fe302600700 1 -- start start 2024-03-29T10:06:37.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe302600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc0ce670 0x7fe2fc1486e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe302600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2fc15a5c0 0x7fe2fc154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe302600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 0x7fe2fc0cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe302600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc154fa0 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe302600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc155280 con 0x7fe2fc15a5c0 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe302600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc155560 con 0x7fe2fc0ce670 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe2fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 0x7fe2fc0cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe2fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 0x7fe2fc0cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53240/0 (socket says 172.21.15.12:53240) 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe2fbe00700 1 -- 172.21.15.12:0/305795531 learned_addr learned my addr 172.21.15.12:0/305795531 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe2faa00700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2fc15a5c0 0x7fe2fc154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe2fb400700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc0ce670 0x7fe2fc1486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe2fbe00700 1 -- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc0ce670 msgr2=0x7fe2fc1486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:37.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe2fbe00700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc0ce670 0x7fe2fc1486e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe2fbe00700 1 -- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2fc15a5c0 msgr2=0x7fe2fc154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe2fbe00700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2fc15a5c0 0x7fe2fc154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.814+0000 7fe2fbe00700 1 -- 172.21.15.12:0/305795531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e8009550 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.815+0000 7fe2fbe00700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 0x7fe2fc0cd530 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4007e40 tx=0x7fe2f400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.815+0000 7fe2f3400700 1 -- 172.21.15.12:0/305795531 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f400eea0 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.815+0000 7fe2f3400700 1 -- 172.21.15.12:0/305795531 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f4014420 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.815+0000 7fe2f3400700 1 -- 172.21.15.12:0/305795531 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f40147b0 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.815+0000 7fe302600700 1 -- 172.21.15.12:0/305795531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc155840 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.815+0000 7fe302600700 1 -- 172.21.15.12:0/305795531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc154d50 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.816+0000 7fe302600700 1 -- 172.21.15.12:0/305795531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2fc067720 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.817+0000 7fe2f3400700 1 -- 172.21.15.12:0/305795531 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fe2f4014910 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.818+0000 7fe2f3400700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ae90 0x7fe2dc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.818+0000 7fe2f3400700 1 -- 172.21.15.12:0/305795531 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6983+0+0 (secure 0 0 0) 0x7fe2f408a460 con 0x7fe2fc15dfe0 2024-03-29T10:06:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.818+0000 7fe2fb400700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ae90 0x7fe2dc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.819+0000 7fe2fb400700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ae90 0x7fe2dc06d350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec000f60 tx=0x7fe2ec009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:37.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:37.822+0000 7fe2f3400700 1 -- 172.21.15.12:0/305795531 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2f4054e20 con 0x7fe2fc15dfe0 2024-03-29T10:06:38.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.123+0000 7fe302600700 1 -- 172.21.15.12:0/305795531 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2fc1553f0 con 0x7fe2dc06ae90 2024-03-29T10:06:38.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.229+0000 7fe2f3400700 1 -- 172.21.15.12:0/305795531 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe2fc1553f0 con 0x7fe2dc06ae90 2024-03-29T10:06:38.229 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/dir4 2024-03-29T10:06:38.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 -- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ae90 msgr2=0x7fe2dc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:38.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ae90 0x7fe2dc06d350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec000f60 tx=0x7fe2ec009040 comp rx=0 tx=0).stop 2024-03-29T10:06:38.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 -- 172.21.15.12:0/305795531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 msgr2=0x7fe2fc0cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:38.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 0x7fe2fc0cd530 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4007e40 tx=0x7fe2f400e3c0 comp rx=0 tx=0).stop 2024-03-29T10:06:38.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 -- 172.21.15.12:0/305795531 shutdown_connections 2024-03-29T10:06:38.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2dc06ae90 0x7fe2dc06d350 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:38.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2fc15dfe0 0x7fe2fc0cd530 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:38.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2fc0ce670 0x7fe2fc1486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:38.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 --2- 172.21.15.12:0/305795531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2fc15a5c0 0x7fe2fc154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:38.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 -- 172.21.15.12:0/305795531 >> 172.21.15.12:0/305795531 conn(0x7fe2fc0b0d70 msgr2=0x7fe2fc0cf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:38.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 -- 172.21.15.12:0/305795531 shutdown_connections 2024-03-29T10:06:38.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:38.231+0000 7fe302600700 1 -- 172.21.15.12:0/305795531 wait complete. 2024-03-29T10:06:38.249 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2024-03-29T10:06:38.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.249 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-03-29T10:06:38.294 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:38.294 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:38.294 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:38.295 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:38.295 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:38.295 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:38.295 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:38.295 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:38.295 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:38.295 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:38.295 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:38.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:38.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.296 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-09_56_00_UTC) 2024-03-29T10:06:38.404 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:38.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.405 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-09_57_00_UTC) 2024-03-29T10:06:38.461 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:38.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.461 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-09_58_00_UTC) 2024-03-29T10:06:38.514 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:38.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.514 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-09_59_00_UTC) 2024-03-29T10:06:38.567 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:38.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.567 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_00_00_UTC) 2024-03-29T10:06:38.621 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:38.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.621 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_01_00_UTC) 2024-03-29T10:06:38.685 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:38.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.686 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_02_00_UTC) 2024-03-29T10:06:38.738 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:38.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.738 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_03_00_UTC) 2024-03-29T10:06:38.794 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:38.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.794 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_04_00_UTC) 2024-03-29T10:06:38.847 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:38.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.847 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_05_00_UTC) 2024-03-29T10:06:38.900 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:38.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.900 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-29-10_06_00_UTC) 2024-03-29T10:06:38.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:38.954 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2024-03-29T10:06:39.003 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2024-03-29T10:06:39.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.214+0000 7f4f21e00700 1 Processor -- start 2024-03-29T10:06:39.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.214+0000 7f4f21e00700 1 -- start start 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.216+0000 7f4f21e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0d21f0 0x7f4f1c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.216+0000 7f4f21e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c0ce270 con 0x7f4f1c0cec40 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.216+0000 7f4f21e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c15b600 con 0x7f4f1c1611b0 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.216+0000 7f4f21e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0cf0b0 con 0x7f4f1c0d21f0 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.216+0000 7f4f1b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4f1c0cec40 0x7f4f1c161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36614/0 (socket says 172.21.15.12:36614) 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.216+0000 7f4f1b400700 1 -- 172.21.15.12:0/2896400003 learned_addr learned my addr 172.21.15.12:0/2896400003 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.216+0000 7f4f1a000700 1 --2- 172.21.15.12:0/2896400003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0d21f0 0x7f4f1c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.216+0000 7f4f19600700 1 -- 172.21.15.12:0/2896400003 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648742509 0 0) 0x7f4f1c0ce270 con 0x7f4f1c0cec40 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.216+0000 7f4f19600700 1 -- 172.21.15.12:0/2896400003 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f000035b0 con 0x7f4f1c0cec40 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.217+0000 7f4f19600700 1 -- 172.21.15.12:0/2896400003 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565991419 0 0) 0x7f4f1c15b600 con 0x7f4f1c1611b0 2024-03-29T10:06:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.217+0000 7f4f19600700 1 -- 172.21.15.12:0/2896400003 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f1c0ce270 con 0x7f4f1c1611b0 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.217+0000 7f4f1a000700 1 -- 172.21.15.12:0/2896400003 >> v1:172.21.15.159:6789/0 conn(0x7f4f1c1611b0 legacy=0x7f4f1c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.217+0000 7f4f1a000700 1 -- 172.21.15.12:0/2896400003 >> v1:172.21.15.12:6789/0 conn(0x7f4f1c0cec40 legacy=0x7f4f1c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.217+0000 7f4f1a000700 1 -- 172.21.15.12:0/2896400003 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c0d2760 con 0x7f4f1c0d21f0 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.217+0000 7f4f1a000700 1 --2- 172.21.15.12:0/2896400003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0d21f0 0x7f4f1c0ce700 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009870 tx=0x7f4f08009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1ceee74c9da96fe server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.217+0000 7f4f19600700 1 -- 172.21.15.12:0/2896400003 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08011830 con 0x7f4f1c0d21f0 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.217+0000 7f4f19600700 1 -- 172.21.15.12:0/2896400003 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f08011990 con 0x7f4f1c0d21f0 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.217+0000 7f4f19600700 1 -- 172.21.15.12:0/2896400003 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08011d20 con 0x7f4f1c0d21f0 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.218+0000 7f4f21e00700 1 -- 172.21.15.12:0/2896400003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0d21f0 msgr2=0x7f4f1c0ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.218+0000 7f4f21e00700 1 --2- 172.21.15.12:0/2896400003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0d21f0 0x7f4f1c0ce700 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009870 tx=0x7f4f08009b80 comp rx=0 tx=0).stop 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.218+0000 7f4f21e00700 1 -- 172.21.15.12:0/2896400003 shutdown_connections 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.218+0000 7f4f21e00700 1 --2- 172.21.15.12:0/2896400003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0d21f0 0x7f4f1c0ce700 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.218+0000 7f4f21e00700 1 -- 172.21.15.12:0/2896400003 >> 172.21.15.12:0/2896400003 conn(0x7f4f1c08aff0 msgr2=0x7f4f1c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.218+0000 7f4f21e00700 1 -- 172.21.15.12:0/2896400003 shutdown_connections 2024-03-29T10:06:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.218+0000 7f4f21e00700 1 -- 172.21.15.12:0/2896400003 wait complete. 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.220+0000 7f4f21e00700 1 Processor -- start 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.220+0000 7f4f21e00700 1 -- start start 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f21e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0cec40 0x7f4f1c159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f21e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f1c0d21f0 0x7f4f1c1f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f21e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f1c1611b0 0x7f4f1c0b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f21e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c159870 con 0x7f4f1c1611b0 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f21e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c1f63e0 con 0x7f4f1c0d21f0 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f21e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c1461e0 con 0x7f4f1c0cec40 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f1b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f1c1611b0 0x7f4f1c0b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f1aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0cec40 0x7f4f1c159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f1b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f1c1611b0 0x7f4f1c0b8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51648/0 (socket says 172.21.15.12:51648) 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f1b400700 1 -- 172.21.15.12:0/297276975 learned_addr learned my addr 172.21.15.12:0/297276975 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:39.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f1a000700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f1c0d21f0 0x7f4f1c1f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f1b400700 1 -- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0cec40 msgr2=0x7f4f1c159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f1b400700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0cec40 0x7f4f1c159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f1b400700 1 -- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f1c0d21f0 msgr2=0x7f4f1c1f5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f1b400700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f1c0d21f0 0x7f4f1c1f5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.221+0000 7f4f1b400700 1 -- 172.21.15.12:0/297276975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f080094a0 con 0x7f4f1c1611b0 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.222+0000 7f4f1b400700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f1c1611b0 0x7f4f1c0b8500 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400cb40 tx=0x7f4f1400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.222+0000 7f4f13400700 1 -- 172.21.15.12:0/297276975 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1400a9e0 con 0x7f4f1c1611b0 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.222+0000 7f4f13400700 1 -- 172.21.15.12:0/297276975 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1400ab40 con 0x7f4f1c1611b0 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.222+0000 7f4f13400700 1 -- 172.21.15.12:0/297276975 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14012400 con 0x7f4f1c1611b0 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.222+0000 7f4f21e00700 1 -- 172.21.15.12:0/297276975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c145660 con 0x7f4f1c1611b0 2024-03-29T10:06:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.222+0000 7f4f21e00700 1 -- 172.21.15.12:0/297276975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c145c20 con 0x7f4f1c1611b0 2024-03-29T10:06:39.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.223+0000 7f4f21e00700 1 -- 172.21.15.12:0/297276975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c067720 con 0x7f4f1c1611b0 2024-03-29T10:06:39.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.224+0000 7f4f13400700 1 -- 172.21.15.12:0/297276975 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f4f1401b3b0 con 0x7f4f1c1611b0 2024-03-29T10:06:39.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.225+0000 7f4f13400700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4efc06ae90 0x7f4efc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:39.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.225+0000 7f4f13400700 1 -- 172.21.15.12:0/297276975 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6615+0+0 (secure 0 0 0) 0x7f4f140896d0 con 0x7f4f1c1611b0 2024-03-29T10:06:39.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.226+0000 7f4f1aa00700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4efc06ae90 0x7f4efc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:39.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.227+0000 7f4f1aa00700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4efc06ae90 0x7f4efc06d350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c001220 tx=0x7f4f0c009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:39.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.230+0000 7f4f13400700 1 -- 172.21.15.12:0/297276975 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4f14054290 con 0x7f4f1c1611b0 2024-03-29T10:06:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.482+0000 7f4f21e00700 1 -- 172.21.15.12:0/297276975 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f1c0cdf00 con 0x7f4efc06ae90 2024-03-29T10:06:39.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.607+0000 7f4f13400700 1 -- 172.21.15.12:0/297276975 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f4f0c0014b0 con 0x7f4efc06ae90 2024-03-29T10:06:39.607 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/dir5 2024-03-29T10:06:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 -- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4efc06ae90 msgr2=0x7f4efc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4efc06ae90 0x7f4efc06d350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c001220 tx=0x7f4f0c009040 comp rx=0 tx=0).stop 2024-03-29T10:06:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 -- 172.21.15.12:0/297276975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f1c1611b0 msgr2=0x7f4f1c0b8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f1c1611b0 0x7f4f1c0b8500 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400cb40 tx=0x7f4f1400ce50 comp rx=0 tx=0).stop 2024-03-29T10:06:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 -- 172.21.15.12:0/297276975 shutdown_connections 2024-03-29T10:06:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4efc06ae90 0x7f4efc06d350 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f1c1611b0 0x7f4f1c0b8500 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f1c0cec40 0x7f4f1c159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:39.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 --2- 172.21.15.12:0/297276975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f1c0d21f0 0x7f4f1c1f5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:39.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 -- 172.21.15.12:0/297276975 >> 172.21.15.12:0/297276975 conn(0x7f4f1c08aff0 msgr2=0x7f4f1c15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:39.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 -- 172.21.15.12:0/297276975 shutdown_connections 2024-03-29T10:06:39.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:39.610+0000 7f4f21e00700 1 -- 172.21.15.12:0/297276975 wait complete. 2024-03-29T10:06:39.627 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2024-03-29T10:06:39.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:39.627 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-03-29T10:06:39.673 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:39.673 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:39.673 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:39.673 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:39.673 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:39.673 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:39.673 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:39.673 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:39.673 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:39.673 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:39.674 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:39.674 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:39.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:39.675 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-09_56_00_UTC) 2024-03-29T10:06:39.729 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:39.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:39.729 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-09_57_00_UTC) 2024-03-29T10:06:39.780 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:39.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:39.781 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-09_58_00_UTC) 2024-03-29T10:06:39.832 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:39.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:39.832 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-09_59_00_UTC) 2024-03-29T10:06:39.883 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:39.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:39.883 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_00_00_UTC) 2024-03-29T10:06:39.935 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:39.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:39.935 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_01_00_UTC) 2024-03-29T10:06:39.987 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:39.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:39.987 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_02_00_UTC) 2024-03-29T10:06:40.038 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:40.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:40.038 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_03_00_UTC) 2024-03-29T10:06:40.127 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:40.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:40.127 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_04_00_UTC) 2024-03-29T10:06:40.210 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:40.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:40.211 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_05_00_UTC) 2024-03-29T10:06:40.262 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:40.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:40.262 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-29-10_06_00_UTC) 2024-03-29T10:06:40.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:40.313 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2024-03-29T10:06:40.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2024-03-29T10:06:40.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.572+0000 7f9287000700 1 Processor -- start 2024-03-29T10:06:40.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.572+0000 7f9287000700 1 -- start start 2024-03-29T10:06:40.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.573+0000 7f9287000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f92800d21f0 0x7f92800ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:40.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.573+0000 7f9287000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92800cec40 0x7f9280161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:40.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.573+0000 7f9287000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92800ce270 con 0x7f92801611b0 2024-03-29T10:06:40.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.573+0000 7f9287000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928015b600 con 0x7f92800d21f0 2024-03-29T10:06:40.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.573+0000 7f9287000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92800d2760 con 0x7f92800cec40 2024-03-29T10:06:40.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.573+0000 7f927f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f92800d21f0 0x7f92800ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:40.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.573+0000 7f927f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f92800d21f0 0x7f92800ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51658/0 (socket says 172.21.15.12:51658) 2024-03-29T10:06:40.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.573+0000 7f927f400700 1 -- 172.21.15.12:0/4243107427 learned_addr learned my addr 172.21.15.12:0/4243107427 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:40.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.574+0000 7f9285000700 1 --2- 172.21.15.12:0/4243107427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92800cec40 0x7f9280161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:40.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.574+0000 7f927f400700 1 -- 172.21.15.12:0/4243107427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92800cec40 msgr2=0x7f9280161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:40.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.574+0000 7f927f400700 1 --2- 172.21.15.12:0/4243107427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92800cec40 0x7f9280161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:40.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.574+0000 7f927f400700 1 -- 172.21.15.12:0/4243107427 >> v1:172.21.15.159:6789/0 conn(0x7f92801611b0 legacy=0x7f928015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:40.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.574+0000 7f927f400700 1 -- 172.21.15.12:0/4243107427 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92800d2a40 con 0x7f92800d21f0 2024-03-29T10:06:40.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.574+0000 7f927f400700 1 --2- 172.21.15.12:0/4243107427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 0x7f92800ce700 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f926c0098f0 tx=0x7f926c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5534511c66a265bc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:40.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.574+0000 7f927ea00700 1 -- 172.21.15.12:0/4243107427 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c00b8a0 con 0x7f92800d21f0 2024-03-29T10:06:40.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.574+0000 7f927ea00700 1 -- 172.21.15.12:0/4243107427 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c00ba00 con 0x7f92800d21f0 2024-03-29T10:06:40.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.574+0000 7f927ea00700 1 -- 172.21.15.12:0/4243107427 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c00bd90 con 0x7f92800d21f0 2024-03-29T10:06:40.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.575+0000 7f9287000700 1 -- 172.21.15.12:0/4243107427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 msgr2=0x7f92800ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:40.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.575+0000 7f9287000700 1 --2- 172.21.15.12:0/4243107427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 0x7f92800ce700 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f926c0098f0 tx=0x7f926c009c00 comp rx=0 tx=0).stop 2024-03-29T10:06:40.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.575+0000 7f9287000700 1 -- 172.21.15.12:0/4243107427 shutdown_connections 2024-03-29T10:06:40.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.575+0000 7f9287000700 1 --2- 172.21.15.12:0/4243107427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92800cec40 0x7f9280161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:40.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.575+0000 7f9287000700 1 --2- 172.21.15.12:0/4243107427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 0x7f92800ce700 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:40.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.575+0000 7f9287000700 1 -- 172.21.15.12:0/4243107427 >> 172.21.15.12:0/4243107427 conn(0x7f928008aff0 msgr2=0x7f92800d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:40.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.575+0000 7f9287000700 1 -- 172.21.15.12:0/4243107427 shutdown_connections 2024-03-29T10:06:40.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.575+0000 7f9287000700 1 -- 172.21.15.12:0/4243107427 wait complete. 2024-03-29T10:06:40.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.577+0000 7f9287000700 1 Processor -- start 2024-03-29T10:06:40.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.577+0000 7f9287000700 1 -- start start 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f9287000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92800cec40 0x7f92801e6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f9287000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 0x7f92800b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f9287000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92801611b0 0x7f92800b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f9287000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92800cbae0 con 0x7f92800d21f0 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f9287000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92800cbdc0 con 0x7f92801611b0 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f9287000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92800cc0a0 con 0x7f92800cec40 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f927f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 0x7f92800b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f927f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 0x7f92800b85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51668/0 (socket says 172.21.15.12:51668) 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f927f400700 1 -- 172.21.15.12:0/900245694 learned_addr learned my addr 172.21.15.12:0/900245694 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f927fe00700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92800cec40 0x7f92801e6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f9285000700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92801611b0 0x7f92800b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:40.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f927f400700 1 -- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92800cec40 msgr2=0x7f92801e6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:40.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f927f400700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92800cec40 0x7f92801e6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:40.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f927f400700 1 -- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92801611b0 msgr2=0x7f92800b8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:40.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f927f400700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92801611b0 0x7f92800b8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:40.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.578+0000 7f927f400700 1 -- 172.21.15.12:0/900245694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c009550 con 0x7f92800d21f0 2024-03-29T10:06:40.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.579+0000 7f927f400700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 0x7f92800b85a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f926c000c00 tx=0x7f926c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:40.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.579+0000 7f9277e00700 1 -- 172.21.15.12:0/900245694 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c004480 con 0x7f92800d21f0 2024-03-29T10:06:40.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.579+0000 7f9277e00700 1 -- 172.21.15.12:0/900245694 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c01f070 con 0x7f92800d21f0 2024-03-29T10:06:40.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.579+0000 7f9277e00700 1 -- 172.21.15.12:0/900245694 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c0038c0 con 0x7f92800d21f0 2024-03-29T10:06:40.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.579+0000 7f9287000700 1 -- 172.21.15.12:0/900245694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92800cc380 con 0x7f92800d21f0 2024-03-29T10:06:40.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.579+0000 7f9287000700 1 -- 172.21.15.12:0/900245694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9280156d90 con 0x7f92800d21f0 2024-03-29T10:06:40.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.580+0000 7f9287000700 1 -- 172.21.15.12:0/900245694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92800cc8a0 con 0x7f92800d21f0 2024-03-29T10:06:40.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.581+0000 7f9277e00700 1 -- 172.21.15.12:0/900245694 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f926c021030 con 0x7f92800d21f0 2024-03-29T10:06:40.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.582+0000 7f9277e00700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f926006ae90 0x7f926006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:40.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.582+0000 7f9277e00700 1 -- 172.21.15.12:0/900245694 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6695+0+0 (secure 0 0 0) 0x7f926c08b020 con 0x7f92800d21f0 2024-03-29T10:06:40.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.582+0000 7f927fe00700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f926006ae90 0x7f926006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:40.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.583+0000 7f927fe00700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f926006ae90 0x7f926006d350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9280160f20 tx=0x7f9270009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:40.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.586+0000 7f9277e00700 1 -- 172.21.15.12:0/900245694 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92800cc8a0 con 0x7f92800d21f0 2024-03-29T10:06:40.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.833+0000 7f9287000700 1 -- 172.21.15.12:0/900245694 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f92800cbf30 con 0x7f926006ae90 2024-03-29T10:06:40.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.936+0000 7f9277e00700 1 -- 172.21.15.12:0/900245694 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f92800cbf30 con 0x7f926006ae90 2024-03-29T10:06:40.936 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/dir6 2024-03-29T10:06:40.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 -- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f926006ae90 msgr2=0x7f926006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:40.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f926006ae90 0x7f926006d350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9280160f20 tx=0x7f9270009040 comp rx=0 tx=0).stop 2024-03-29T10:06:40.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 -- 172.21.15.12:0/900245694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 msgr2=0x7f92800b85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:40.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 0x7f92800b85a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f926c000c00 tx=0x7f926c004750 comp rx=0 tx=0).stop 2024-03-29T10:06:40.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 -- 172.21.15.12:0/900245694 shutdown_connections 2024-03-29T10:06:40.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f926006ae90 0x7f926006d350 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:40.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92801611b0 0x7f92800b8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:40.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92800cec40 0x7f92801e6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:40.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 --2- 172.21.15.12:0/900245694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92800d21f0 0x7f92800b85a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:40.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 -- 172.21.15.12:0/900245694 >> 172.21.15.12:0/900245694 conn(0x7f928008aff0 msgr2=0x7f928015f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:40.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 -- 172.21.15.12:0/900245694 shutdown_connections 2024-03-29T10:06:40.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:40.939+0000 7f9287000700 1 -- 172.21.15.12:0/900245694 wait complete. 2024-03-29T10:06:40.957 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2024-03-29T10:06:40.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:40.957 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-03-29T10:06:41.002 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:41.002 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:41.002 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:41.002 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:41.003 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:41.003 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:41.003 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:41.003 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:41.003 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:41.003 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:41.003 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:41.004 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:41.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.004 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-09_56_00_UTC) 2024-03-29T10:06:41.057 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:41.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.057 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-09_57_00_UTC) 2024-03-29T10:06:41.133 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:41.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.134 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-09_58_00_UTC) 2024-03-29T10:06:41.188 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:41.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.188 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-09_59_00_UTC) 2024-03-29T10:06:41.244 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:41.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.244 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_00_00_UTC) 2024-03-29T10:06:41.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:41.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.296 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_01_00_UTC) 2024-03-29T10:06:41.348 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:41.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.348 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_02_00_UTC) 2024-03-29T10:06:41.403 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:41.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.404 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_03_00_UTC) 2024-03-29T10:06:41.455 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:41.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.456 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_04_00_UTC) 2024-03-29T10:06:41.507 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:41.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.507 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_05_00_UTC) 2024-03-29T10:06:41.573 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:41.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.573 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-29-10_06_00_UTC) 2024-03-29T10:06:41.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:41.625 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2024-03-29T10:06:41.672 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2024-03-29T10:06:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.874+0000 7f84f1c00700 1 Processor -- start 2024-03-29T10:06:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.875+0000 7f84f1c00700 1 -- start start 2024-03-29T10:06:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.876+0000 7f84f1c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f84ec0cbb30 0x7f84ec0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.876+0000 7f84f1c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f84ec0cd020 0x7f84ec0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.876+0000 7f84f1c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0b2070 0x7f84ec0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.876+0000 7f84f1c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84ec0cc450 con 0x7f84ec0cd020 2024-03-29T10:06:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.876+0000 7f84f1c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84ec0cdae0 con 0x7f84ec0cbb30 2024-03-29T10:06:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.876+0000 7f84f1c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec0cddc0 con 0x7f84ec0b2070 2024-03-29T10:06:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.877+0000 7f84eb400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0b2070 0x7f84ec0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.877+0000 7f84eb400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0b2070 0x7f84ec0b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:39856/0 (socket says 172.21.15.12:39856) 2024-03-29T10:06:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.877+0000 7f84eb400700 1 -- 172.21.15.12:0/39367699 learned_addr learned my addr 172.21.15.12:0/39367699 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.877+0000 7f84eaa00700 1 --2- 172.21.15.12:0/39367699 >> v2:172.21.15.159:3300/0 conn(0x7f84ec0cbb30 0x7f84ec0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.877+0000 7f84ea000700 1 --2- 172.21.15.12:0/39367699 >> v2:172.21.15.12:3300/0 conn(0x7f84ec0cd020 0x7f84ec0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.877+0000 7f84eb400700 1 -- 172.21.15.12:0/39367699 >> v2:172.21.15.159:3300/0 conn(0x7f84ec0cbb30 msgr2=0x7f84ec0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.877+0000 7f84eb400700 1 --2- 172.21.15.12:0/39367699 >> v2:172.21.15.159:3300/0 conn(0x7f84ec0cbb30 0x7f84ec0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.878+0000 7f84eb400700 1 -- 172.21.15.12:0/39367699 >> v2:172.21.15.12:3300/0 conn(0x7f84ec0cd020 msgr2=0x7f84ec0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.878+0000 7f84eb400700 1 --2- 172.21.15.12:0/39367699 >> v2:172.21.15.12:3300/0 conn(0x7f84ec0cd020 0x7f84ec0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.878+0000 7f84eb400700 1 -- 172.21.15.12:0/39367699 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec0ce0a0 con 0x7f84ec0b2070 2024-03-29T10:06:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.878+0000 7f84eb400700 1 --2- 172.21.15.12:0/39367699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0b2070 0x7f84ec0b2500 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f84ec06fd00 tx=0x7f84e400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=374817e15f1007b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.878+0000 7f84e9600700 1 -- 172.21.15.12:0/39367699 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e400e070 con 0x7f84ec0b2070 2024-03-29T10:06:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.878+0000 7f84e9600700 1 -- 172.21.15.12:0/39367699 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e400b910 con 0x7f84ec0b2070 2024-03-29T10:06:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.878+0000 7f84e9600700 1 -- 172.21.15.12:0/39367699 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e400bcc0 con 0x7f84ec0b2070 2024-03-29T10:06:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.879+0000 7f84f1c00700 1 -- 172.21.15.12:0/39367699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0b2070 msgr2=0x7f84ec0b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.879+0000 7f84f1c00700 1 --2- 172.21.15.12:0/39367699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0b2070 0x7f84ec0b2500 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f84ec06fd00 tx=0x7f84e400b330 comp rx=0 tx=0).stop 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.879+0000 7f84f1c00700 1 -- 172.21.15.12:0/39367699 shutdown_connections 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.879+0000 7f84f1c00700 1 --2- 172.21.15.12:0/39367699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0b2070 0x7f84ec0b2500 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.879+0000 7f84f1c00700 1 --2- 172.21.15.12:0/39367699 >> v2:172.21.15.159:3300/0 conn(0x7f84ec0cbb30 0x7f84ec0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.879+0000 7f84f1c00700 1 --2- 172.21.15.12:0/39367699 >> v2:172.21.15.12:3300/0 conn(0x7f84ec0cd020 0x7f84ec0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.879+0000 7f84f1c00700 1 -- 172.21.15.12:0/39367699 >> 172.21.15.12:0/39367699 conn(0x7f84ec0b0d70 msgr2=0x7f84ec0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.879+0000 7f84f1c00700 1 -- 172.21.15.12:0/39367699 shutdown_connections 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.879+0000 7f84f1c00700 1 -- 172.21.15.12:0/39367699 wait complete. 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.882+0000 7f84f1c00700 1 Processor -- start 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.882+0000 7f84f1c00700 1 -- start start 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84f1c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec0b2070 0x7f84ec098b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84f1c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0cbb30 0x7f84ec09a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84f1c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84ec0cd020 0x7f84ec15a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84f1c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec09a380 con 0x7f84ec0b2070 2024-03-29T10:06:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84f1c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec099770 con 0x7f84ec0cd020 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84f1c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec099a50 con 0x7f84ec0cbb30 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec0b2070 0x7f84ec098b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec0b2070 0x7f84ec098b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51682/0 (socket says 172.21.15.12:51682) 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84eaa00700 1 -- 172.21.15.12:0/282107986 learned_addr learned my addr 172.21.15.12:0/282107986 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84ea000700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0cbb30 0x7f84ec09a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84eb400700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84ec0cd020 0x7f84ec15a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84eaa00700 1 -- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0cbb30 msgr2=0x7f84ec09a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84eaa00700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0cbb30 0x7f84ec09a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84eaa00700 1 -- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84ec0cd020 msgr2=0x7f84ec15a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84eaa00700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84ec0cd020 0x7f84ec15a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.883+0000 7f84eaa00700 1 -- 172.21.15.12:0/282107986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e4009b30 con 0x7f84ec0b2070 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.884+0000 7f84eaa00700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec0b2070 0x7f84ec098b70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00e870 tx=0x7f84dc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.884+0000 7f84e2a00700 1 -- 172.21.15.12:0/282107986 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc009590 con 0x7f84ec0b2070 2024-03-29T10:06:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.884+0000 7f84e2a00700 1 -- 172.21.15.12:0/282107986 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc013680 con 0x7f84ec0b2070 2024-03-29T10:06:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.884+0000 7f84f1c00700 1 -- 172.21.15.12:0/282107986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ec099d30 con 0x7f84ec0b2070 2024-03-29T10:06:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.884+0000 7f84e2a00700 1 -- 172.21.15.12:0/282107986 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0139f0 con 0x7f84ec0b2070 2024-03-29T10:06:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.884+0000 7f84f1c00700 1 -- 172.21.15.12:0/282107986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ec09d080 con 0x7f84ec0b2070 2024-03-29T10:06:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.885+0000 7f84f1c00700 1 -- 172.21.15.12:0/282107986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84ec067720 con 0x7f84ec0b2070 2024-03-29T10:06:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.886+0000 7f84e2a00700 1 -- 172.21.15.12:0/282107986 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f84dc013b50 con 0x7f84ec0b2070 2024-03-29T10:06:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.887+0000 7f84e2a00700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f84cc06ae90 0x7f84cc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.887+0000 7f84e2a00700 1 -- 172.21.15.12:0/282107986 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6615+0+0 (secure 0 0 0) 0x7f84dc089860 con 0x7f84ec0b2070 2024-03-29T10:06:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.888+0000 7f84ea000700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f84cc06ae90 0x7f84cc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.888+0000 7f84ea000700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f84cc06ae90 0x7f84cc06d350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f84ec16a530 tx=0x7f84d8009210 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:41.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:41.892+0000 7f84e2a00700 1 -- 172.21.15.12:0/282107986 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f84dc054430 con 0x7f84ec0b2070 2024-03-29T10:06:42.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.146+0000 7f84f1c00700 1 -- 172.21.15.12:0/282107986 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f84ec1594a0 con 0x7f84cc06ae90 2024-03-29T10:06:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.304+0000 7f84e2a00700 1 -- 172.21.15.12:0/282107986 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f84d80098d0 con 0x7f84cc06ae90 2024-03-29T10:06:42.303 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/dir7 2024-03-29T10:06:42.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 -- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f84cc06ae90 msgr2=0x7f84cc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:42.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f84cc06ae90 0x7f84cc06d350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f84ec16a530 tx=0x7f84d8009210 comp rx=0 tx=0).stop 2024-03-29T10:06:42.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 -- 172.21.15.12:0/282107986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec0b2070 msgr2=0x7f84ec098b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:42.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec0b2070 0x7f84ec098b70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00e870 tx=0x7f84dc00eb80 comp rx=0 tx=0).stop 2024-03-29T10:06:42.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 -- 172.21.15.12:0/282107986 shutdown_connections 2024-03-29T10:06:42.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f84cc06ae90 0x7f84cc06d350 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:42.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84ec0cd020 0x7f84ec15a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:42.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec0b2070 0x7f84ec098b70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:42.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 --2- 172.21.15.12:0/282107986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84ec0cbb30 0x7f84ec09a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:42.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 -- 172.21.15.12:0/282107986 >> 172.21.15.12:0/282107986 conn(0x7f84ec0b0d70 msgr2=0x7f84ec0af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:42.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 -- 172.21.15.12:0/282107986 shutdown_connections 2024-03-29T10:06:42.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:42.306+0000 7f84f1c00700 1 -- 172.21.15.12:0/282107986 wait complete. 2024-03-29T10:06:42.324 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2024-03-29T10:06:42.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.324 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:42.370 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:42.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:42.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.372 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-09_56_00_UTC) 2024-03-29T10:06:42.440 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:42.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.441 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-09_57_00_UTC) 2024-03-29T10:06:42.492 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:42.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.493 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-09_58_00_UTC) 2024-03-29T10:06:42.544 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:42.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.544 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-09_59_00_UTC) 2024-03-29T10:06:42.629 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:42.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.629 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_00_00_UTC) 2024-03-29T10:06:42.681 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:42.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.681 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_01_00_UTC) 2024-03-29T10:06:42.732 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:42.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.733 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_02_00_UTC) 2024-03-29T10:06:42.784 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:42.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.784 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_03_00_UTC) 2024-03-29T10:06:42.835 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:42.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.835 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_04_00_UTC) 2024-03-29T10:06:42.887 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:42.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.887 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_05_00_UTC) 2024-03-29T10:06:42.938 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:42.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.938 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-29-10_06_00_UTC) 2024-03-29T10:06:42.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:42.997 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2024-03-29T10:06:43.046 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2024-03-29T10:06:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.255+0000 7ff836200700 1 Processor -- start 2024-03-29T10:06:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.255+0000 7ff836200700 1 -- start start 2024-03-29T10:06:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.256+0000 7ff836200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 0x7ff8300cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.256+0000 7ff836200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff8300cd020 0x7ff8300cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.256+0000 7ff836200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff8300b2070 0x7ff8300b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.256+0000 7ff836200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8300cc450 con 0x7ff8300cd020 2024-03-29T10:06:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.256+0000 7ff836200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8300cdae0 con 0x7ff8300b2070 2024-03-29T10:06:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.256+0000 7ff836200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300cddc0 con 0x7ff8300cbb30 2024-03-29T10:06:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff8300cd020 0x7ff8300cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff8300cd020 0x7ff8300cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51692/0 (socket says 172.21.15.12:51692) 2024-03-29T10:06:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82ea00700 1 -- 172.21.15.12:0/107995579 learned_addr learned my addr 172.21.15.12:0/107995579 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82fe00700 1 --2- 172.21.15.12:0/107995579 >> v2:172.21.15.159:3300/0 conn(0x7ff8300b2070 0x7ff8300b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82f400700 1 --2- 172.21.15.12:0/107995579 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 0x7ff8300cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82ea00700 1 -- 172.21.15.12:0/107995579 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 msgr2=0x7ff8300cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82ea00700 1 --2- 172.21.15.12:0/107995579 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 0x7ff8300cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82ea00700 1 -- 172.21.15.12:0/107995579 >> v2:172.21.15.159:3300/0 conn(0x7ff8300b2070 msgr2=0x7ff8300b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82ea00700 1 --2- 172.21.15.12:0/107995579 >> v2:172.21.15.159:3300/0 conn(0x7ff8300b2070 0x7ff8300b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82ea00700 1 -- 172.21.15.12:0/107995579 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8300ce0a0 con 0x7ff8300cd020 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82f400700 1 --2- 172.21.15.12:0/107995579 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 0x7ff8300cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.257+0000 7ff82ea00700 1 --2- 172.21.15.12:0/107995579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 0x7ff8300cd4a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff83006c9f0 tx=0x7ff81c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18eac807583a03f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff82e000700 1 -- 172.21.15.12:0/107995579 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c011830 con 0x7ff8300cd020 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff82e000700 1 -- 172.21.15.12:0/107995579 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81c011990 con 0x7ff8300cd020 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff82e000700 1 -- 172.21.15.12:0/107995579 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c011d20 con 0x7ff8300cd020 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff836200700 1 -- 172.21.15.12:0/107995579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 msgr2=0x7ff8300cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff836200700 1 --2- 172.21.15.12:0/107995579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 0x7ff8300cd4a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff83006c9f0 tx=0x7ff81c009bd0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff836200700 1 -- 172.21.15.12:0/107995579 shutdown_connections 2024-03-29T10:06:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff836200700 1 --2- 172.21.15.12:0/107995579 >> v2:172.21.15.159:3300/0 conn(0x7ff8300b2070 0x7ff8300b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff836200700 1 --2- 172.21.15.12:0/107995579 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 0x7ff8300cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff836200700 1 --2- 172.21.15.12:0/107995579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 0x7ff8300cd4a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff836200700 1 -- 172.21.15.12:0/107995579 >> 172.21.15.12:0/107995579 conn(0x7ff8300b0d70 msgr2=0x7ff8300b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff836200700 1 -- 172.21.15.12:0/107995579 shutdown_connections 2024-03-29T10:06:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.258+0000 7ff836200700 1 -- 172.21.15.12:0/107995579 wait complete. 2024-03-29T10:06:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.260+0000 7ff836200700 1 Processor -- start 2024-03-29T10:06:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.260+0000 7ff836200700 1 -- start start 2024-03-29T10:06:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.261+0000 7ff836200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8300b2070 0x7ff83014a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.261+0000 7ff836200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 0x7ff83014e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.261+0000 7ff836200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 0x7ff8300c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.261+0000 7ff836200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300c83f0 con 0x7ff8300cd020 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.261+0000 7ff836200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300c8c90 con 0x7ff8300b2070 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.261+0000 7ff836200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300ca730 con 0x7ff8300cbb30 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.261+0000 7ff82fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 0x7ff8300c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.261+0000 7ff82ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 0x7ff83014e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82f400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8300b2070 0x7ff83014a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 0x7ff8300c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51704/0 (socket says 172.21.15.12:51704) 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82fe00700 1 -- 172.21.15.12:0/1572607072 learned_addr learned my addr 172.21.15.12:0/1572607072 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82fe00700 1 -- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 msgr2=0x7ff83014e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82fe00700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 0x7ff83014e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82fe00700 1 -- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8300b2070 msgr2=0x7ff83014a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82fe00700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8300b2070 0x7ff83014a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82fe00700 1 -- 172.21.15.12:0/1572607072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81c009550 con 0x7ff8300cd020 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82ea00700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 0x7ff83014e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82f400700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8300b2070 0x7ff83014a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff82fe00700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 0x7ff8300c9d80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff82800efc0 tx=0x7ff82800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff827400700 1 -- 172.21.15.12:0/1572607072 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82800cad0 con 0x7ff8300cd020 2024-03-29T10:06:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff827400700 1 -- 172.21.15.12:0/1572607072 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82800f040 con 0x7ff8300cd020 2024-03-29T10:06:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.262+0000 7ff827400700 1 -- 172.21.15.12:0/1572607072 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82800ce70 con 0x7ff8300cd020 2024-03-29T10:06:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.263+0000 7ff836200700 1 -- 172.21.15.12:0/1572607072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8300c8640 con 0x7ff8300cd020 2024-03-29T10:06:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.263+0000 7ff836200700 1 -- 172.21.15.12:0/1572607072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83014a980 con 0x7ff8300cd020 2024-03-29T10:06:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.264+0000 7ff836200700 1 -- 172.21.15.12:0/1572607072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8300c8e00 con 0x7ff8300cd020 2024-03-29T10:06:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.265+0000 7ff827400700 1 -- 172.21.15.12:0/1572607072 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7ff828014480 con 0x7ff8300cd020 2024-03-29T10:06:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.265+0000 7ff827400700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81006ae90 0x7ff81006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.266+0000 7ff827400700 1 -- 172.21.15.12:0/1572607072 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6899+0+0 (secure 0 0 0) 0x7ff82808a3a0 con 0x7ff8300cd020 2024-03-29T10:06:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.266+0000 7ff82f400700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81006ae90 0x7ff81006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.267+0000 7ff82f400700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81006ae90 0x7ff81006d350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff820006f10 tx=0x7ff820008040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.270+0000 7ff827400700 1 -- 172.21.15.12:0/1572607072 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff828054d30 con 0x7ff8300cd020 2024-03-29T10:06:43.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.515+0000 7ff836200700 1 -- 172.21.15.12:0/1572607072 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8301594a0 con 0x7ff81006ae90 2024-03-29T10:06:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.614+0000 7ff827400700 1 -- 172.21.15.12:0/1572607072 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff820009de0 con 0x7ff81006ae90 2024-03-29T10:06:43.614 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/dir8 2024-03-29T10:06:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 -- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81006ae90 msgr2=0x7ff81006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81006ae90 0x7ff81006d350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff820006f10 tx=0x7ff820008040 comp rx=0 tx=0).stop 2024-03-29T10:06:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 -- 172.21.15.12:0/1572607072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 msgr2=0x7ff8300c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 0x7ff8300c9d80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff82800efc0 tx=0x7ff82800c0c0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 -- 172.21.15.12:0/1572607072 shutdown_connections 2024-03-29T10:06:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81006ae90 0x7ff81006d350 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8300cd020 0x7ff8300c9d80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8300b2070 0x7ff83014a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 --2- 172.21.15.12:0/1572607072 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8300cbb30 0x7ff83014e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:43.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 -- 172.21.15.12:0/1572607072 >> 172.21.15.12:0/1572607072 conn(0x7ff8300b0d70 msgr2=0x7ff83006e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:43.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 -- 172.21.15.12:0/1572607072 shutdown_connections 2024-03-29T10:06:43.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:43.617+0000 7ff836200700 1 -- 172.21.15.12:0/1572607072 wait complete. 2024-03-29T10:06:43.634 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2024-03-29T10:06:43.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:43.635 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-03-29T10:06:43.680 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:43.680 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:43.680 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:43.680 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:43.680 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:43.680 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:43.680 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:43.680 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:43.681 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:43.681 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:43.681 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:43.681 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:43.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:43.682 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-09_56_00_UTC) 2024-03-29T10:06:43.776 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:43.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:43.777 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-09_57_00_UTC) 2024-03-29T10:06:43.829 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:43.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:43.829 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-09_58_00_UTC) 2024-03-29T10:06:43.881 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:43.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:43.882 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-09_59_00_UTC) 2024-03-29T10:06:43.935 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:43.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:43.936 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_00_00_UTC) 2024-03-29T10:06:43.991 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:43.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:43.992 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_01_00_UTC) 2024-03-29T10:06:44.061 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:44.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:44.061 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_02_00_UTC) 2024-03-29T10:06:44.119 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:44.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:44.119 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_03_00_UTC) 2024-03-29T10:06:44.173 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:44.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:44.173 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_04_00_UTC) 2024-03-29T10:06:44.225 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:44.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:44.225 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_05_00_UTC) 2024-03-29T10:06:44.279 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:44.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:44.279 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-29-10_06_00_UTC) 2024-03-29T10:06:44.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:44.337 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2024-03-29T10:06:44.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2024-03-29T10:06:44.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.606+0000 7f8169000700 1 Processor -- start 2024-03-29T10:06:44.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.606+0000 7f8169000700 1 -- start start 2024-03-29T10:06:44.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.607+0000 7f8169000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81640d3f20 0x7f81640cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:44.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.607+0000 7f8169000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f81640b3f80 0x7f81640ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:44.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.607+0000 7f8169000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8164161230 con 0x7f8164160e50 2024-03-29T10:06:44.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.607+0000 7f8169000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81640d2a80 con 0x7f81640b3f80 2024-03-29T10:06:44.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.607+0000 7f8169000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81640d43b0 con 0x7f81640d3f20 2024-03-29T10:06:44.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.608+0000 7f8162800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f81640b3f80 0x7f81640ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:44.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.608+0000 7f8161400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81640d3f20 0x7f81640cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:44.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.608+0000 7f8161e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f8164160e50 0x7f81640b4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:48030/0 (socket says 172.21.15.12:48030) 2024-03-29T10:06:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.608+0000 7f8161e00700 1 -- 172.21.15.12:0/2179586762 learned_addr learned my addr 172.21.15.12:0/2179586762 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.608+0000 7f8162800700 1 -- 172.21.15.12:0/2179586762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81640d3f20 msgr2=0x7f81640cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.608+0000 7f8162800700 1 --2- 172.21.15.12:0/2179586762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81640d3f20 0x7f81640cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.608+0000 7f8162800700 1 -- 172.21.15.12:0/2179586762 >> v1:172.21.15.159:6789/0 conn(0x7f8164160e50 legacy=0x7f81640b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.608+0000 7f8162800700 1 -- 172.21.15.12:0/2179586762 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81640d21e0 con 0x7f81640b3f80 2024-03-29T10:06:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8162800700 1 --2- 172.21.15.12:0/2179586762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81640b3f80 0x7f81640ce430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f815c009d90 tx=0x7f815c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=117214a36b89f54f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8160a00700 1 -- 172.21.15.12:0/2179586762 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815c00e070 con 0x7f81640b3f80 2024-03-29T10:06:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8160a00700 1 -- 172.21.15.12:0/2179586762 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815c00bb20 con 0x7f81640b3f80 2024-03-29T10:06:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8160a00700 1 -- 172.21.15.12:0/2179586762 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815c012430 con 0x7f81640b3f80 2024-03-29T10:06:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8169000700 1 -- 172.21.15.12:0/2179586762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81640b3f80 msgr2=0x7f81640ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8169000700 1 --2- 172.21.15.12:0/2179586762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81640b3f80 0x7f81640ce430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f815c009d90 tx=0x7f815c00b330 comp rx=0 tx=0).stop 2024-03-29T10:06:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8169000700 1 -- 172.21.15.12:0/2179586762 shutdown_connections 2024-03-29T10:06:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8169000700 1 --2- 172.21.15.12:0/2179586762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81640b3f80 0x7f81640ce430 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8169000700 1 --2- 172.21.15.12:0/2179586762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81640d3f20 0x7f81640cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8169000700 1 -- 172.21.15.12:0/2179586762 >> 172.21.15.12:0/2179586762 conn(0x7f816408aff0 msgr2=0x7f81640d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.609+0000 7f8169000700 1 -- 172.21.15.12:0/2179586762 shutdown_connections 2024-03-29T10:06:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.610+0000 7f8169000700 1 -- 172.21.15.12:0/2179586762 wait complete. 2024-03-29T10:06:44.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.611+0000 7f8169000700 1 Processor -- start 2024-03-29T10:06:44.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.611+0000 7f8169000700 1 -- start start 2024-03-29T10:06:44.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.612+0000 7f8169000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81640b3f80 0x7f81641f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.612+0000 7f8169000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8164160e50 0x7f816415c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.612+0000 7f8169000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f816415cdb0 0x7f81640cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.612+0000 7f8169000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81640cc100 con 0x7f8164160e50 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.612+0000 7f8169000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816415d5d0 con 0x7f81640b3f80 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.612+0000 7f8169000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81640cb760 con 0x7f816415cdb0 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.612+0000 7f8161400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8164160e50 0x7f816415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f8161e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81640b3f80 0x7f81641f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f8161400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8164160e50 0x7f816415c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51710/0 (socket says 172.21.15.12:51710) 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f8162800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f816415cdb0 0x7f81640cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f8161400700 1 -- 172.21.15.12:0/3638420879 learned_addr learned my addr 172.21.15.12:0/3638420879 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f8161400700 1 -- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f816415cdb0 msgr2=0x7f81640cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f8161400700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f816415cdb0 0x7f81640cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:44.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f8161400700 1 -- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81640b3f80 msgr2=0x7f81641f4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f8161400700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81640b3f80 0x7f81641f4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f8161400700 1 -- 172.21.15.12:0/3638420879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815c009a70 con 0x7f8164160e50 2024-03-29T10:06:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f8161400700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8164160e50 0x7f816415c870 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f815000e9e0 tx=0x7f815000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.613+0000 7f815a000700 1 -- 172.21.15.12:0/3638420879 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8150009710 con 0x7f8164160e50 2024-03-29T10:06:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.614+0000 7f815a000700 1 -- 172.21.15.12:0/3638420879 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815000ee80 con 0x7f8164160e50 2024-03-29T10:06:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.614+0000 7f8169000700 1 -- 172.21.15.12:0/3638420879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81640cba40 con 0x7f8164160e50 2024-03-29T10:06:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.614+0000 7f815a000700 1 -- 172.21.15.12:0/3638420879 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815000ca50 con 0x7f8164160e50 2024-03-29T10:06:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.614+0000 7f8169000700 1 -- 172.21.15.12:0/3638420879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81641f46b0 con 0x7f8164160e50 2024-03-29T10:06:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.615+0000 7f8169000700 1 -- 172.21.15.12:0/3638420879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81640cc240 con 0x7f8164160e50 2024-03-29T10:06:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.616+0000 7f815a000700 1 -- 172.21.15.12:0/3638420879 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f815000cbb0 con 0x7f8164160e50 2024-03-29T10:06:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.617+0000 7f815a000700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f814406ae90 0x7f814406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:44.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.617+0000 7f815a000700 1 -- 172.21.15.12:0/3638420879 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f8150089e00 con 0x7f8164160e50 2024-03-29T10:06:44.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.617+0000 7f8161e00700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f814406ae90 0x7f814406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:44.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.618+0000 7f8161e00700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f814406ae90 0x7f814406d350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8154005910 tx=0x7f8154009040 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.621+0000 7f815a000700 1 -- 172.21.15.12:0/3638420879 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81640cc240 con 0x7f8164160e50 2024-03-29T10:06:44.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.880+0000 7f8169000700 1 -- 172.21.15.12:0/3638420879 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f81640b0100 con 0x7f814406ae90 2024-03-29T10:06:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.981+0000 7f815a000700 1 -- 172.21.15.12:0/3638420879 <== mgr.4934 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f81540014e0 con 0x7f814406ae90 2024-03-29T10:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/dir9 2024-03-29T10:06:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 -- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f814406ae90 msgr2=0x7f814406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f814406ae90 0x7f814406d350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8154005910 tx=0x7f8154009040 comp rx=0 tx=0).stop 2024-03-29T10:06:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 -- 172.21.15.12:0/3638420879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8164160e50 msgr2=0x7f816415c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8164160e50 0x7f816415c870 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f815000e9e0 tx=0x7f815000ecf0 comp rx=0 tx=0).stop 2024-03-29T10:06:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 -- 172.21.15.12:0/3638420879 shutdown_connections 2024-03-29T10:06:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f814406ae90 0x7f814406d350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f816415cdb0 0x7f81640cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81640b3f80 0x7f81641f4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 --2- 172.21.15.12:0/3638420879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8164160e50 0x7f816415c870 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 -- 172.21.15.12:0/3638420879 >> 172.21.15.12:0/3638420879 conn(0x7f816408aff0 msgr2=0x7f81640b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 -- 172.21.15.12:0/3638420879 shutdown_connections 2024-03-29T10:06:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:44.983+0000 7f8169000700 1 -- 172.21.15.12:0/3638420879 wait complete. 2024-03-29T10:06:45.009 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2024-03-29T10:06:45.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.009 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:45.057 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-09_56_00_UTC 2024-03-29T10:06:45.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.057 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-09_56_00_UTC) 2024-03-29T10:06:45.123 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-09_57_00_UTC 2024-03-29T10:06:45.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.124 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-09_57_00_UTC) 2024-03-29T10:06:45.176 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-09_58_00_UTC 2024-03-29T10:06:45.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.176 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-09_58_00_UTC) 2024-03-29T10:06:45.228 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-09_59_00_UTC 2024-03-29T10:06:45.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.228 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-09_59_00_UTC) 2024-03-29T10:06:45.283 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_00_00_UTC 2024-03-29T10:06:45.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.283 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_00_00_UTC) 2024-03-29T10:06:45.351 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_01_00_UTC 2024-03-29T10:06:45.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.351 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_01_00_UTC) 2024-03-29T10:06:45.402 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_02_00_UTC 2024-03-29T10:06:45.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.402 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_02_00_UTC) 2024-03-29T10:06:45.454 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_03_00_UTC 2024-03-29T10:06:45.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.455 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_03_00_UTC) 2024-03-29T10:06:45.506 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_04_00_UTC 2024-03-29T10:06:45.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.507 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_04_00_UTC) 2024-03-29T10:06:45.558 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_05_00_UTC 2024-03-29T10:06:45.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.558 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_05_00_UTC) 2024-03-29T10:06:45.610 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_06_00_UTC 2024-03-29T10:06:45.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.611 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-29-10_06_00_UTC) 2024-03-29T10:06:45.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:45.662 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2024-03-29T10:06:45.710 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T10:06:45.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.910+0000 7f4007600700 1 Processor -- start 2024-03-29T10:06:45.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.910+0000 7f4007600700 1 -- start start 2024-03-29T10:06:45.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.911+0000 7f4007600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f400015dad0 0x7f40000ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.911+0000 7f4007600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f40000d21d0 0x7f400015fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.911+0000 7f4007600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400015f820 con 0x7f40000d1630 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.911+0000 7f4007600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40000cfcc0 con 0x7f400015dad0 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.911+0000 7f4007600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40000cd9f0 con 0x7f40000d21d0 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.912+0000 7f4004c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f400015dad0 0x7f40000ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.912+0000 7f4004c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f400015dad0 0x7f40000ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51720/0 (socket says 172.21.15.12:51720) 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.912+0000 7f4004c00700 1 -- 172.21.15.12:0/538629292 learned_addr learned my addr 172.21.15.12:0/538629292 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.912+0000 7f4005600700 1 --2- 172.21.15.12:0/538629292 >> v2:172.21.15.159:3300/0 conn(0x7f40000d21d0 0x7f400015fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.912+0000 7f4004c00700 1 -- 172.21.15.12:0/538629292 >> v2:172.21.15.159:3300/0 conn(0x7f40000d21d0 msgr2=0x7f400015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.912+0000 7f4004c00700 1 --2- 172.21.15.12:0/538629292 >> v2:172.21.15.159:3300/0 conn(0x7f40000d21d0 0x7f400015fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.912+0000 7f4004c00700 1 -- 172.21.15.12:0/538629292 >> v1:172.21.15.159:6789/0 conn(0x7f40000d1630 legacy=0x7f40000d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.912+0000 7f4004c00700 1 -- 172.21.15.12:0/538629292 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40000cf420 con 0x7f400015dad0 2024-03-29T10:06:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.912+0000 7f4004c00700 1 --2- 172.21.15.12:0/538629292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f400015dad0 0x7f40000ced80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f400008ccb0 tx=0x7f3ff4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbff1565dfbb6ae server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:45.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.912+0000 7f3fff400700 1 -- 172.21.15.12:0/538629292 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff400b870 con 0x7f400015dad0 2024-03-29T10:06:45.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.913+0000 7f3fff400700 1 -- 172.21.15.12:0/538629292 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff400b9d0 con 0x7f400015dad0 2024-03-29T10:06:45.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.913+0000 7f3fff400700 1 -- 172.21.15.12:0/538629292 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff400bd60 con 0x7f400015dad0 2024-03-29T10:06:45.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.913+0000 7f4007600700 1 -- 172.21.15.12:0/538629292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f400015dad0 msgr2=0x7f40000ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:45.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.913+0000 7f4007600700 1 --2- 172.21.15.12:0/538629292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f400015dad0 0x7f40000ced80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f400008ccb0 tx=0x7f3ff4009bd0 comp rx=0 tx=0).stop 2024-03-29T10:06:45.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.913+0000 7f4007600700 1 -- 172.21.15.12:0/538629292 shutdown_connections 2024-03-29T10:06:45.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.913+0000 7f4007600700 1 --2- 172.21.15.12:0/538629292 >> v2:172.21.15.159:3300/0 conn(0x7f40000d21d0 0x7f400015fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:45.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.913+0000 7f4007600700 1 --2- 172.21.15.12:0/538629292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f400015dad0 0x7f40000ced80 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:45.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.913+0000 7f4007600700 1 -- 172.21.15.12:0/538629292 >> 172.21.15.12:0/538629292 conn(0x7f400008aff0 msgr2=0x7f40000d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:45.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.913+0000 7f4007600700 1 -- 172.21.15.12:0/538629292 shutdown_connections 2024-03-29T10:06:45.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.913+0000 7f4007600700 1 -- 172.21.15.12:0/538629292 wait complete. 2024-03-29T10:06:45.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.915+0000 7f4007600700 1 Processor -- start 2024-03-29T10:06:45.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.915+0000 7f4007600700 1 -- start start 2024-03-29T10:06:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f4007600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40000d1630 0x7f400014dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f4007600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40000d21d0 0x7f40000c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f4007600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f400015dad0 0x7f40000c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f4007600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40000c8810 con 0x7f40000d21d0 2024-03-29T10:06:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f4007600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40000cd3f0 con 0x7f400015dad0 2024-03-29T10:06:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f4007600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40000c7b90 con 0x7f40000d1630 2024-03-29T10:06:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f3fffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40000d21d0 0x7f40000c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f3fffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40000d21d0 0x7f40000c7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51736/0 (socket says 172.21.15.12:51736) 2024-03-29T10:06:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f3fffe00700 1 -- 172.21.15.12:0/459844954 learned_addr learned my addr 172.21.15.12:0/459844954 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f4005600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f400015dad0 0x7f40000c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.916+0000 7f4004c00700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40000d1630 0x7f400014dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f3fffe00700 1 -- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40000d1630 msgr2=0x7f400014dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f3fffe00700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40000d1630 0x7f400014dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f3fffe00700 1 -- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f400015dad0 msgr2=0x7f40000c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f3fffe00700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f400015dad0 0x7f40000c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f3fffe00700 1 -- 172.21.15.12:0/459844954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff4009550 con 0x7f40000d21d0 2024-03-29T10:06:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f3fffe00700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40000d21d0 0x7f40000c7020 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00c4e0 tx=0x7f3fec00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f3ffcc00700 1 -- 172.21.15.12:0/459844954 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec0052d0 con 0x7f40000d21d0 2024-03-29T10:06:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f3ffcc00700 1 -- 172.21.15.12:0/459844954 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fec005430 con 0x7f40000d21d0 2024-03-29T10:06:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f4007600700 1 -- 172.21.15.12:0/459844954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40000c7e70 con 0x7f40000d21d0 2024-03-29T10:06:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f3ffcc00700 1 -- 172.21.15.12:0/459844954 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec0038c0 con 0x7f40000d21d0 2024-03-29T10:06:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.917+0000 7f4007600700 1 -- 172.21.15.12:0/459844954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40000c8430 con 0x7f40000d21d0 2024-03-29T10:06:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.918+0000 7f4007600700 1 -- 172.21.15.12:0/459844954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40000c8950 con 0x7f40000d21d0 2024-03-29T10:06:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.919+0000 7f3ffcc00700 1 -- 172.21.15.12:0/459844954 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f3fec003a20 con 0x7f40000d21d0 2024-03-29T10:06:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.920+0000 7f3ffcc00700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fe006ae90 0x7f3fe006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.920+0000 7f3ffcc00700 1 -- 172.21.15.12:0/459844954 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f3fec088c90 con 0x7f40000d21d0 2024-03-29T10:06:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.921+0000 7f4004c00700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fe006ae90 0x7f3fe006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.921+0000 7f4004c00700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fe006ae90 0x7f3fe006d350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4014040 tx=0x7f3ff4008030 comp rx=0 tx=0).ready entity=mgr.4934 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:45.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:45.925+0000 7f3ffcc00700 1 -- 172.21.15.12:0/459844954 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40000c8950 con 0x7f40000d21d0 2024-03-29T10:06:46.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:46.242+0000 7f4007600700 1 -- 172.21.15.12:0/459844954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f40000c8950 con 0x7f40000d21d0 2024-03-29T10:06:47.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.118+0000 7f3ffcc00700 1 -- 172.21.15.12:0/459844954 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f3fec053d70 con 0x7f40000d21d0 2024-03-29T10:06:47.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.120+0000 7f3ffcc00700 1 -- 172.21.15.12:0/459844954 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) v1 ==== 93+0+0 (secure 0 0 0) 0x7f3fec0726f0 con 0x7f40000d21d0 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.123+0000 7f4007600700 1 -- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fe006ae90 msgr2=0x7f3fe006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.123+0000 7f4007600700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fe006ae90 0x7f3fe006d350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4014040 tx=0x7f3ff4008030 comp rx=0 tx=0).stop 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.124+0000 7f4007600700 1 -- 172.21.15.12:0/459844954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40000d21d0 msgr2=0x7f40000c7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.124+0000 7f4007600700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40000d21d0 0x7f40000c7020 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00c4e0 tx=0x7f3fec00c7f0 comp rx=0 tx=0).stop 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.124+0000 7f4007600700 1 -- 172.21.15.12:0/459844954 shutdown_connections 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.124+0000 7f4007600700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fe006ae90 0x7f3fe006d350 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.124+0000 7f4007600700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f400015dad0 0x7f40000c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.124+0000 7f4007600700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40000d1630 0x7f400014dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.124+0000 7f4007600700 1 --2- 172.21.15.12:0/459844954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40000d21d0 0x7f40000c7020 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.124+0000 7f4007600700 1 -- 172.21.15.12:0/459844954 >> 172.21.15.12:0/459844954 conn(0x7f400008aff0 msgr2=0x7f4000150b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:47.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.124+0000 7f4007600700 1 -- 172.21.15.12:0/459844954 shutdown_connections 2024-03-29T10:06:47.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.124+0000 7f4007600700 1 -- 172.21.15.12:0/459844954 wait complete. 2024-03-29T10:06:47.149 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:06:47.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:06:47.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:06:47.358 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:06:47.358 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:06:47.358 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:06:47.358 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:47.358+0000 7f9625f86240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:06:47.377 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:47.378+0000 7ff4587aa240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:06:47.406 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:06:47.407 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:06:47.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:06:47.549 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:47.550+0000 7f9625f86240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:06:47.618 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:47.618+0000 7ff4587aa240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:06:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.759+0000 7f1c94600700 1 Processor -- start 2024-03-29T10:06:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.759+0000 7f1c94600700 1 -- start start 2024-03-29T10:06:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.761+0000 7f1c94600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1c8c0b3af0 0x7f1c8c15a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.761+0000 7f1c94600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c8c0709c0 0x7f1c8c070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.761+0000 7f1c94600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c0b2180 con 0x7f1c8c072580 2024-03-29T10:06:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.761+0000 7f1c94600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c15b100 con 0x7f1c8c0b3af0 2024-03-29T10:06:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.761+0000 7f1c94600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c15b3e0 con 0x7f1c8c0709c0 2024-03-29T10:06:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.761+0000 7f1c91c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1c8c0b3af0 0x7f1c8c15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.761+0000 7f1c92600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c8c0709c0 0x7f1c8c070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.761+0000 7f1c91c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1c8c0b3af0 0x7f1c8c15a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:47094/0 (socket says 172.21.15.12:47094) 2024-03-29T10:06:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.761+0000 7f1c91c00700 1 -- 172.21.15.12:0/3553668805 learned_addr learned my addr 172.21.15.12:0/3553668805 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.762+0000 7f1c91c00700 1 -- 172.21.15.12:0/3553668805 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c8c0709c0 msgr2=0x7f1c8c070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.762+0000 7f1c91c00700 1 --2- 172.21.15.12:0/3553668805 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c8c0709c0 0x7f1c8c070e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.762+0000 7f1c91c00700 1 -- 172.21.15.12:0/3553668805 >> v1:172.21.15.159:6789/0 conn(0x7f1c8c072580 legacy=0x7f1c8c0b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.762+0000 7f1c91c00700 1 -- 172.21.15.12:0/3553668805 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c0b23f0 con 0x7f1c8c0b3af0 2024-03-29T10:06:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.762+0000 7f1c91c00700 1 --2- 172.21.15.12:0/3553668805 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c8c0b3af0 0x7f1c8c15a860 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1c800098f0 tx=0x7f1c80009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94e067e05e084ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.762+0000 7f1c8be00700 1 -- 172.21.15.12:0/3553668805 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80011830 con 0x7f1c8c0b3af0 2024-03-29T10:06:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.762+0000 7f1c8be00700 1 -- 172.21.15.12:0/3553668805 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c80011990 con 0x7f1c8c0b3af0 2024-03-29T10:06:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.762+0000 7f1c8be00700 1 -- 172.21.15.12:0/3553668805 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80011d20 con 0x7f1c8c0b3af0 2024-03-29T10:06:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.763+0000 7f1c94600700 1 -- 172.21.15.12:0/3553668805 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c8c0b3af0 msgr2=0x7f1c8c15a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.763+0000 7f1c94600700 1 --2- 172.21.15.12:0/3553668805 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c8c0b3af0 0x7f1c8c15a860 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1c800098f0 tx=0x7f1c80009c00 comp rx=0 tx=0).stop 2024-03-29T10:06:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.763+0000 7f1c94600700 1 -- 172.21.15.12:0/3553668805 shutdown_connections 2024-03-29T10:06:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.763+0000 7f1c94600700 1 --2- 172.21.15.12:0/3553668805 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c8c0709c0 0x7f1c8c070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.763+0000 7f1c94600700 1 --2- 172.21.15.12:0/3553668805 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c8c0b3af0 0x7f1c8c15a860 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.763+0000 7f1c94600700 1 -- 172.21.15.12:0/3553668805 >> 172.21.15.12:0/3553668805 conn(0x7f1c8c0cf190 msgr2=0x7f1c8c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.763+0000 7f1c94600700 1 -- 172.21.15.12:0/3553668805 shutdown_connections 2024-03-29T10:06:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.763+0000 7f1c94600700 1 -- 172.21.15.12:0/3553668805 wait complete. 2024-03-29T10:06:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.765+0000 7f1c94600700 1 Processor -- start 2024-03-29T10:06:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.765+0000 7f1c94600700 1 -- start start 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c94600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c8c0709c0 0x7f1c8c0c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c94600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c8c072580 0x7f1c8c14a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c94600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c8c0b3af0 0x7f1c8c159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c94600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0c9af0 con 0x7f1c8c072580 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c94600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c14abb0 con 0x7f1c8c0b3af0 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c94600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0c8e70 con 0x7f1c8c0709c0 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c91200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c8c072580 0x7f1c8c14a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c91200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c8c072580 0x7f1c8c14a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51746/0 (socket says 172.21.15.12:51746) 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c91200700 1 -- 172.21.15.12:0/2334801025 learned_addr learned my addr 172.21.15.12:0/2334801025 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c91c00700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c8c0709c0 0x7f1c8c0c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.766+0000 7f1c92600700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c8c0b3af0 0x7f1c8c159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c91200700 1 -- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c8c0709c0 msgr2=0x7f1c8c0c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c91200700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c8c0709c0 0x7f1c8c0c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c91200700 1 -- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c8c0b3af0 msgr2=0x7f1c8c159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c91200700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c8c0b3af0 0x7f1c8c159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c91200700 1 -- 172.21.15.12:0/2334801025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c80009550 con 0x7f1c8c072580 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c91200700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c8c072580 0x7f1c8c14a670 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1c7800c4b0 tx=0x7f1c7800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c89600700 1 -- 172.21.15.12:0/2334801025 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c780052d0 con 0x7f1c8c072580 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c89600700 1 -- 172.21.15.12:0/2334801025 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c78005430 con 0x7f1c8c072580 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c89600700 1 -- 172.21.15.12:0/2334801025 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c780038c0 con 0x7f1c8c072580 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c94600700 1 -- 172.21.15.12:0/2334801025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c0c9150 con 0x7f1c8c072580 2024-03-29T10:06:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.767+0000 7f1c94600700 1 -- 172.21.15.12:0/2334801025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c0c9710 con 0x7f1c8c072580 2024-03-29T10:06:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.769+0000 7f1c94600700 1 -- 172.21.15.12:0/2334801025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8c0c9c30 con 0x7f1c8c072580 2024-03-29T10:06:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.769+0000 7f1c89600700 1 -- 172.21.15.12:0/2334801025 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f1c78003a20 con 0x7f1c8c072580 2024-03-29T10:06:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.770+0000 7f1c89600700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c6c06ae40 0x7f1c6c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.770+0000 7f1c89600700 1 -- 172.21.15.12:0/2334801025 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6679+0+0 (secure 0 0 0) 0x7f1c78088d50 con 0x7f1c8c072580 2024-03-29T10:06:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.770+0000 7f1c91c00700 1 -- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c6c06ae40 msgr2=0x7f1c6c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.770+0000 7f1c91c00700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c6c06ae40 0x7f1c6c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.775+0000 7f1c89600700 1 -- 172.21.15.12:0/2334801025 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1c8c0c9c30 con 0x7f1c8c072580 2024-03-29T10:06:47.865 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:47.866+0000 7f9625f86240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:06:47.967 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:47.968+0000 7ff4587aa240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:06:47.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.971+0000 7f1c91c00700 1 -- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c6c06ae40 msgr2=0x7f1c6c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:47.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:47.971+0000 7f1c91c00700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c6c06ae40 0x7f1c6c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:48.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.009+0000 7f1c94600700 1 -- 172.21.15.12:0/2334801025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c8c0c9c30 con 0x7f1c8c072580 2024-03-29T10:06:48.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.010+0000 7f1c89600700 1 -- 172.21.15.12:0/2334801025 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+17906 (secure 0 0 0) 0x7f1c8c0c9c30 con 0x7f1c8c072580 2024-03-29T10:06:48.010 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:48.011 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":47,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:06:47.985612+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:55:33.508935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T09:55:34.418367+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":219,"snap_epoch":47,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6832/43444":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/3395907487":"2024-03-30T09:55:52.597859+0000","172.21.15.159:6833/43444":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/249193272":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/795330822":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/1756693282":"2024-03-30T09:55:52.597859+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":6,"length":1},{"begin":9,"length":1},{"begin":12,"length":1},{"begin":16,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":27,"length":1},{"begin":32,"length":1},{"begin":34,"length":1},{"begin":38,"length":1},{"begin":43,"length":1},{"begin":45,"length":1},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":55,"length":1},{"begin":59,"length":1},{"begin":64,"length":1},{"begin":66,"length":1},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":76,"length":1},{"begin":80,"length":2},{"begin":85,"length":1},{"begin":87,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":109,"length":1},{"begin":187,"length":33}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":109,"length":1},{"begin":219,"length":1}]}],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":4,"length":1},{"begin":7,"length":1},{"begin":10,"length":1},{"begin":14,"length":2},{"begin":19,"length":1},{"begin":25,"length":2},{"begin":30,"length":2},{"begin":36,"length":2},{"begin":41,"length":2},{"begin":47,"length":2},{"begin":51,"length":2},{"begin":57,"length":2},{"begin":62,"length":2},{"begin":68,"length":2},{"begin":72,"length":2},{"begin":78,"length":2},{"begin":83,"length":2},{"begin":89,"length":3},{"begin":94,"length":2},{"begin":100,"length":3},{"begin":105,"length":2},{"begin":147,"length":40}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:06:48.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 -- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c6c06ae40 msgr2=0x7f1c6c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:48.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c6c06ae40 0x7f1c6c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 -- 172.21.15.12:0/2334801025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c8c072580 msgr2=0x7f1c8c14a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c8c072580 0x7f1c8c14a670 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1c7800c4b0 tx=0x7f1c7800c7c0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 -- 172.21.15.12:0/2334801025 shutdown_connections 2024-03-29T10:06:48.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c6c06ae40 0x7f1c6c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c8c0b3af0 0x7f1c8c159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c8c0709c0 0x7f1c8c0c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 --2- 172.21.15.12:0/2334801025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c8c072580 0x7f1c8c14a670 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 -- 172.21.15.12:0/2334801025 >> 172.21.15.12:0/2334801025 conn(0x7f1c8c0cf190 msgr2=0x7f1c8c071420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:48.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 -- 172.21.15.12:0/2334801025 shutdown_connections 2024-03-29T10:06:48.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.013+0000 7f1c94600700 1 -- 172.21.15.12:0/2334801025 wait complete. 2024-03-29T10:06:48.031 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:55:33.508935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:55:34.418367+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 219, 'snap_epoch': 47, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:06:48.032 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:06:48.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.236+0000 7fc497200700 1 Processor -- start 2024-03-29T10:06:48.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.236+0000 7fc497200700 1 -- start start 2024-03-29T10:06:48.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.237+0000 7fc497200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc490157f90 0x7fc4900b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.237+0000 7fc497200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4901613e0 0x7fc490161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.237+0000 7fc497200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4900b3070 con 0x7fc4900b3420 2024-03-29T10:06:48.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.237+0000 7fc497200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc490158370 con 0x7fc490157f90 2024-03-29T10:06:48.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.237+0000 7fc497200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4900b2710 con 0x7fc4901613e0 2024-03-29T10:06:48.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.237+0000 7fc48fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc490157f90 0x7fc4900b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:48.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.237+0000 7fc495200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc4900b3420 0x7fc4900cd020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36626/0 (socket says 172.21.15.12:36626) 2024-03-29T10:06:48.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.238+0000 7fc495200700 1 -- 172.21.15.12:0/3994207650 learned_addr learned my addr 172.21.15.12:0/3994207650 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:48.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.238+0000 7fc487e00700 1 --2- 172.21.15.12:0/3994207650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4901613e0 0x7fc490161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:48.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.238+0000 7fc48fe00700 1 -- 172.21.15.12:0/3994207650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4901613e0 msgr2=0x7fc490161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:48.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.238+0000 7fc48fe00700 1 --2- 172.21.15.12:0/3994207650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4901613e0 0x7fc490161860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.238+0000 7fc48fe00700 1 -- 172.21.15.12:0/3994207650 >> v1:172.21.15.12:6789/0 conn(0x7fc4900b3420 legacy=0x7fc4900cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.238+0000 7fc48fe00700 1 -- 172.21.15.12:0/3994207650 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4900b2070 con 0x7fc490157f90 2024-03-29T10:06:48.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.238+0000 7fc48fe00700 1 --2- 172.21.15.12:0/3994207650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc490157f90 0x7fc4900b2b30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc480009840 tx=0x7fc480009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf445ba8e4ad5ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:48.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.238+0000 7fc48f400700 1 -- 172.21.15.12:0/3994207650 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480011830 con 0x7fc490157f90 2024-03-29T10:06:48.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.239+0000 7fc48f400700 1 -- 172.21.15.12:0/3994207650 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc480011990 con 0x7fc490157f90 2024-03-29T10:06:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.239+0000 7fc48f400700 1 -- 172.21.15.12:0/3994207650 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480011d20 con 0x7fc490157f90 2024-03-29T10:06:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.239+0000 7fc497200700 1 -- 172.21.15.12:0/3994207650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc490157f90 msgr2=0x7fc4900b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.239+0000 7fc497200700 1 --2- 172.21.15.12:0/3994207650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc490157f90 0x7fc4900b2b30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc480009840 tx=0x7fc480009b50 comp rx=0 tx=0).stop 2024-03-29T10:06:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.239+0000 7fc497200700 1 -- 172.21.15.12:0/3994207650 shutdown_connections 2024-03-29T10:06:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.239+0000 7fc497200700 1 --2- 172.21.15.12:0/3994207650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc490157f90 0x7fc4900b2b30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.239+0000 7fc497200700 1 --2- 172.21.15.12:0/3994207650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4901613e0 0x7fc490161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.239+0000 7fc497200700 1 -- 172.21.15.12:0/3994207650 >> 172.21.15.12:0/3994207650 conn(0x7fc4900b0d70 msgr2=0x7fc4900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.239+0000 7fc497200700 1 -- 172.21.15.12:0/3994207650 shutdown_connections 2024-03-29T10:06:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.239+0000 7fc497200700 1 -- 172.21.15.12:0/3994207650 wait complete. 2024-03-29T10:06:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.242+0000 7fc497200700 1 Processor -- start 2024-03-29T10:06:48.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.242+0000 7fc497200700 1 -- start start 2024-03-29T10:06:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc497200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4900b3420 0x7fc49009a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc497200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc490157f90 0x7fc4900991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc497200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4901613e0 0x7fc49006d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc497200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4901f8180 con 0x7fc4901613e0 2024-03-29T10:06:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc497200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc490149bc0 con 0x7fc4900b3420 2024-03-29T10:06:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc497200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4901f7220 con 0x7fc490157f90 2024-03-29T10:06:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc495200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4901613e0 0x7fc49006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc495200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4901613e0 0x7fc49006d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51760/0 (socket says 172.21.15.12:51760) 2024-03-29T10:06:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc495200700 1 -- 172.21.15.12:0/482867384 learned_addr learned my addr 172.21.15.12:0/482867384 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:48.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc48fe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4900b3420 0x7fc49009a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.244+0000 7fc487e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc490157f90 0x7fc4900991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc495200700 1 -- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc490157f90 msgr2=0x7fc4900991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc495200700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc490157f90 0x7fc4900991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc495200700 1 -- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4900b3420 msgr2=0x7fc49009a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc495200700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4900b3420 0x7fc49009a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc495200700 1 -- 172.21.15.12:0/482867384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4800094a0 con 0x7fc4901613e0 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc495200700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4901613e0 0x7fc49006d250 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc48800ca80 tx=0x7fc48800cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc48cc00700 1 -- 172.21.15.12:0/482867384 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48800a9e0 con 0x7fc4901613e0 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc48cc00700 1 -- 172.21.15.12:0/482867384 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48800ab40 con 0x7fc4901613e0 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc48cc00700 1 -- 172.21.15.12:0/482867384 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc488012400 con 0x7fc4901613e0 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc497200700 1 -- 172.21.15.12:0/482867384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4901f7500 con 0x7fc4901613e0 2024-03-29T10:06:48.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.245+0000 7fc497200700 1 -- 172.21.15.12:0/482867384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4901f7ac0 con 0x7fc4901613e0 2024-03-29T10:06:48.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.246+0000 7fc497200700 1 -- 172.21.15.12:0/482867384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49006f2d0 con 0x7fc4901613e0 2024-03-29T10:06:48.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.247+0000 7fc48cc00700 1 -- 172.21.15.12:0/482867384 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fc48801b3b0 con 0x7fc4901613e0 2024-03-29T10:06:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.248+0000 7fc48cc00700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc46c06ae40 0x7fc46c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.248+0000 7fc48cc00700 1 -- 172.21.15.12:0/482867384 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7fc4880892c0 con 0x7fc4901613e0 2024-03-29T10:06:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.248+0000 7fc48fe00700 1 -- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc46c06ae40 msgr2=0x7fc46c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:48.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.248+0000 7fc48fe00700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc46c06ae40 0x7fc46c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.253+0000 7fc48cc00700 1 -- 172.21.15.12:0/482867384 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc488053eb0 con 0x7fc4901613e0 2024-03-29T10:06:48.424 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:48.425+0000 7f9625f86240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:06:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.449+0000 7fc48fe00700 1 -- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc46c06ae40 msgr2=0x7fc46c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:48.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.449+0000 7fc48fe00700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc46c06ae40 0x7fc46c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:48.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.475+0000 7fc497200700 1 -- 172.21.15.12:0/482867384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4900b3070 con 0x7fc4901613e0 2024-03-29T10:06:48.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.476+0000 7fc48cc00700 1 -- 172.21.15.12:0/482867384 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4900b3070 con 0x7fc4901613e0 2024-03-29T10:06:48.476 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:06:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.478+0000 7fc497200700 1 -- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc46c06ae40 msgr2=0x7fc46c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.478+0000 7fc497200700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc46c06ae40 0x7fc46c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.479+0000 7fc497200700 1 -- 172.21.15.12:0/482867384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4901613e0 msgr2=0x7fc49006d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.479+0000 7fc497200700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4901613e0 0x7fc49006d250 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc48800ca80 tx=0x7fc48800cd90 comp rx=0 tx=0).stop 2024-03-29T10:06:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.479+0000 7fc497200700 1 -- 172.21.15.12:0/482867384 shutdown_connections 2024-03-29T10:06:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.479+0000 7fc497200700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc46c06ae40 0x7fc46c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.479+0000 7fc497200700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4901613e0 0x7fc49006d250 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.479+0000 7fc497200700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4900b3420 0x7fc49009a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.479+0000 7fc497200700 1 --2- 172.21.15.12:0/482867384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc490157f90 0x7fc4900991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.479+0000 7fc497200700 1 -- 172.21.15.12:0/482867384 >> 172.21.15.12:0/482867384 conn(0x7fc4900b0d70 msgr2=0x7fc4900b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.479+0000 7fc497200700 1 -- 172.21.15.12:0/482867384 shutdown_connections 2024-03-29T10:06:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.479+0000 7fc497200700 1 -- 172.21.15.12:0/482867384 wait complete. 2024-03-29T10:06:48.499 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:06:48.559 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:48.560+0000 7ff4587aa240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:06:48.688 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:48.689+0000 7f9625f86240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:06:48.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.706+0000 7f36f4000700 1 Processor -- start 2024-03-29T10:06:48.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.706+0000 7f36f4000700 1 -- start start 2024-03-29T10:06:48.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.707+0000 7f36f4000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f36ec0d21f0 0x7f36ec0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.707+0000 7f36f4000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f36ec0cec40 0x7f36ec161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.707+0000 7f36f4000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec0ce270 con 0x7f36ec1611b0 2024-03-29T10:06:48.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.707+0000 7f36f4000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36ec15b600 con 0x7f36ec0cec40 2024-03-29T10:06:48.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.707+0000 7f36f4000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36ec0d2760 con 0x7f36ec0d21f0 2024-03-29T10:06:48.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.708+0000 7f36f1600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f36ec1611b0 0x7f36ec15ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36630/0 (socket says 172.21.15.12:36630) 2024-03-29T10:06:48.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.708+0000 7f36f1600700 1 -- 172.21.15.12:0/1586131323 learned_addr learned my addr 172.21.15.12:0/1586131323 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:48.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.708+0000 7f36f2000700 1 --2- 172.21.15.12:0/1586131323 >> v2:172.21.15.12:3300/0 conn(0x7f36ec0cec40 0x7f36ec161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:48.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.708+0000 7f36f0c00700 1 --2- 172.21.15.12:0/1586131323 >> v2:172.21.15.159:3300/0 conn(0x7f36ec0d21f0 0x7f36ec0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.708+0000 7f36ebe00700 1 -- 172.21.15.12:0/1586131323 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576208075 0 0) 0x7f36ec0ce270 con 0x7f36ec1611b0 2024-03-29T10:06:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.708+0000 7f36ebe00700 1 -- 172.21.15.12:0/1586131323 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d40035b0 con 0x7f36ec1611b0 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.708+0000 7f36f0c00700 1 -- 172.21.15.12:0/1586131323 >> v2:172.21.15.12:3300/0 conn(0x7f36ec0cec40 msgr2=0x7f36ec161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.708+0000 7f36f0c00700 1 --2- 172.21.15.12:0/1586131323 >> v2:172.21.15.12:3300/0 conn(0x7f36ec0cec40 0x7f36ec161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.708+0000 7f36f0c00700 1 -- 172.21.15.12:0/1586131323 >> v1:172.21.15.12:6789/0 conn(0x7f36ec1611b0 legacy=0x7f36ec15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.708+0000 7f36f0c00700 1 -- 172.21.15.12:0/1586131323 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec0d2a40 con 0x7f36ec0d21f0 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.709+0000 7f36f0c00700 1 --2- 172.21.15.12:0/1586131323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ec0d21f0 0x7f36ec0ce700 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f36ec08c030 tx=0x7f36d8009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a734b44650b550 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.709+0000 7f36ebe00700 1 -- 172.21.15.12:0/1586131323 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d8011830 con 0x7f36ec0d21f0 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.709+0000 7f36ebe00700 1 -- 172.21.15.12:0/1586131323 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d8011990 con 0x7f36ec0d21f0 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.709+0000 7f36ebe00700 1 -- 172.21.15.12:0/1586131323 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d8011d20 con 0x7f36ec0d21f0 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.710+0000 7f36f4000700 1 -- 172.21.15.12:0/1586131323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ec0d21f0 msgr2=0x7f36ec0ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.710+0000 7f36f4000700 1 --2- 172.21.15.12:0/1586131323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ec0d21f0 0x7f36ec0ce700 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f36ec08c030 tx=0x7f36d8009b20 comp rx=0 tx=0).stop 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.710+0000 7f36f4000700 1 -- 172.21.15.12:0/1586131323 shutdown_connections 2024-03-29T10:06:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.710+0000 7f36f4000700 1 --2- 172.21.15.12:0/1586131323 >> v2:172.21.15.12:3300/0 conn(0x7f36ec0cec40 0x7f36ec161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.710+0000 7f36f4000700 1 --2- 172.21.15.12:0/1586131323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ec0d21f0 0x7f36ec0ce700 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.710+0000 7f36f4000700 1 -- 172.21.15.12:0/1586131323 >> 172.21.15.12:0/1586131323 conn(0x7f36ec08aff0 msgr2=0x7f36ec0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:48.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.710+0000 7f36f4000700 1 -- 172.21.15.12:0/1586131323 shutdown_connections 2024-03-29T10:06:48.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.710+0000 7f36f4000700 1 -- 172.21.15.12:0/1586131323 wait complete. 2024-03-29T10:06:48.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.711+0000 7f36f4000700 1 Processor -- start 2024-03-29T10:06:48.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.712+0000 7f36f4000700 1 -- start start 2024-03-29T10:06:48.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f4000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36ec0cec40 0x7f36ec1f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f4000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ec0d21f0 0x7f36ec0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f4000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ec1611b0 0x7f36ec0c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f4000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec0cbc50 con 0x7f36ec0cec40 2024-03-29T10:06:48.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f4000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec0cb0b0 con 0x7f36ec0d21f0 2024-03-29T10:06:48.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f4000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec0cc0e0 con 0x7f36ec1611b0 2024-03-29T10:06:48.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36ec0cec40 0x7f36ec1f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36ec0cec40 0x7f36ec1f2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51778/0 (socket says 172.21.15.12:51778) 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f1600700 1 -- 172.21.15.12:0/198069624 learned_addr learned my addr 172.21.15.12:0/198069624 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f0c00700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ec0d21f0 0x7f36ec0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f2000700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ec1611b0 0x7f36ec0c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f1600700 1 -- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ec1611b0 msgr2=0x7f36ec0c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f1600700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ec1611b0 0x7f36ec0c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f1600700 1 -- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ec0d21f0 msgr2=0x7f36ec0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f1600700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ec0d21f0 0x7f36ec0cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.713+0000 7f36f1600700 1 -- 172.21.15.12:0/198069624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d80094a0 con 0x7f36ec0cec40 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.714+0000 7f36f0c00700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ec0d21f0 0x7f36ec0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:48.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.714+0000 7f36f1600700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36ec0cec40 0x7f36ec1f2770 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f36e0001eb0 tx=0x7f36e000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.714+0000 7f36e9600700 1 -- 172.21.15.12:0/198069624 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0005d70 con 0x7f36ec0cec40 2024-03-29T10:06:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.714+0000 7f36e9600700 1 -- 172.21.15.12:0/198069624 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e00070d0 con 0x7f36ec0cec40 2024-03-29T10:06:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.714+0000 7f36e9600700 1 -- 172.21.15.12:0/198069624 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0007460 con 0x7f36ec0cec40 2024-03-29T10:06:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.714+0000 7f36f4000700 1 -- 172.21.15.12:0/198069624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec0cb320 con 0x7f36ec0cec40 2024-03-29T10:06:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.714+0000 7f36f4000700 1 -- 172.21.15.12:0/198069624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec0cb8e0 con 0x7f36ec0cec40 2024-03-29T10:06:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.715+0000 7f36f4000700 1 -- 172.21.15.12:0/198069624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ec0cbf30 con 0x7f36ec0cec40 2024-03-29T10:06:48.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.716+0000 7f36e9600700 1 -- 172.21.15.12:0/198069624 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f36e00075c0 con 0x7f36ec0cec40 2024-03-29T10:06:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.717+0000 7f36e9600700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f36cc06ae40 0x7f36cc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.717+0000 7f36e9600700 1 -- 172.21.15.12:0/198069624 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f36e0088f90 con 0x7f36ec0cec40 2024-03-29T10:06:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.717+0000 7f36f0c00700 1 -- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f36cc06ae40 msgr2=0x7f36cc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.717+0000 7f36f0c00700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f36cc06ae40 0x7f36cc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:48.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.722+0000 7f36e9600700 1 -- 172.21.15.12:0/198069624 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f36e00539a0 con 0x7f36ec0cec40 2024-03-29T10:06:48.831 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:48.831+0000 7ff4587aa240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:06:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.918+0000 7f36f0c00700 1 -- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f36cc06ae40 msgr2=0x7f36cc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.918+0000 7f36f0c00700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f36cc06ae40 0x7f36cc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:48.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.936+0000 7f36f4000700 1 -- 172.21.15.12:0/198069624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f36ec067720 con 0x7f36ec0cec40 2024-03-29T10:06:48.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.937+0000 7f36e9600700 1 -- 172.21.15.12:0/198069624 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v47) v1 ==== 104+0+11 (secure 0 0 0) 0x7f36e0017020 con 0x7f36ec0cec40 2024-03-29T10:06:48.941 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:06:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.939+0000 7f36f4000700 1 -- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f36cc06ae40 msgr2=0x7f36cc06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.939+0000 7f36f4000700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f36cc06ae40 0x7f36cc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.939+0000 7f36f4000700 1 -- 172.21.15.12:0/198069624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36ec0cec40 msgr2=0x7f36ec1f2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.939+0000 7f36f4000700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36ec0cec40 0x7f36ec1f2770 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f36e0001eb0 tx=0x7f36e000c7b0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.939+0000 7f36f4000700 1 -- 172.21.15.12:0/198069624 shutdown_connections 2024-03-29T10:06:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.939+0000 7f36f4000700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f36cc06ae40 0x7f36cc06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.939+0000 7f36f4000700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ec1611b0 0x7f36ec0c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.939+0000 7f36f4000700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36ec0cec40 0x7f36ec1f2770 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.939+0000 7f36f4000700 1 --2- 172.21.15.12:0/198069624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ec0d21f0 0x7f36ec0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.939+0000 7f36f4000700 1 -- 172.21.15.12:0/198069624 >> 172.21.15.12:0/198069624 conn(0x7f36ec08aff0 msgr2=0x7f36ec15a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.940+0000 7f36f4000700 1 -- 172.21.15.12:0/198069624 shutdown_connections 2024-03-29T10:06:48.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:48.940+0000 7f36f4000700 1 -- 172.21.15.12:0/198069624 wait complete. 2024-03-29T10:06:48.958 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:06:49.146 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:49.146+0000 7f9625f86240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:06:49.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.163+0000 7fc1a3600700 1 Processor -- start 2024-03-29T10:06:49.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.163+0000 7fc1a3600700 1 -- start start 2024-03-29T10:06:49.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.164+0000 7fc1a3600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc19c157f90 0x7fc19c0b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.164+0000 7fc1a3600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc1a3600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0b3070 con 0x7fc19c0b3420 2024-03-29T10:06:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc1a3600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19c158370 con 0x7fc19c157f90 2024-03-29T10:06:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc1a3600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0b2710 con 0x7fc19c1613e0 2024-03-29T10:06:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc1a0c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc19c157f90 0x7fc19c0b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc19be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc1a0c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc19c157f90 0x7fc19c0b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:45638/0 (socket says 172.21.15.12:45638) 2024-03-29T10:06:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc1a0c00700 1 -- 172.21.15.12:0/2235491384 learned_addr learned my addr 172.21.15.12:0/2235491384 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc19be00700 1 --2- 172.21.15.12:0/2235491384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c161860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:43390/0 (socket says 172.21.15.12:43390) 2024-03-29T10:06:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc19be00700 1 -- 172.21.15.12:0/2235491384 >> v2:172.21.15.159:3300/0 conn(0x7fc19c157f90 msgr2=0x7fc19c0b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc19be00700 1 --2- 172.21.15.12:0/2235491384 >> v2:172.21.15.159:3300/0 conn(0x7fc19c157f90 0x7fc19c0b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc19be00700 1 -- 172.21.15.12:0/2235491384 >> v1:172.21.15.159:6789/0 conn(0x7fc19c0b3420 legacy=0x7fc19c0cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc19be00700 1 -- 172.21.15.12:0/2235491384 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0b2070 con 0x7fc19c1613e0 2024-03-29T10:06:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.165+0000 7fc1a0c00700 1 --2- 172.21.15.12:0/2235491384 >> v2:172.21.15.159:3300/0 conn(0x7fc19c157f90 0x7fc19c0b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.166+0000 7fc19be00700 1 --2- 172.21.15.12:0/2235491384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c161860 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc188009840 tx=0x7fc188009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98c14dff12cc70f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.166+0000 7fc19b400700 1 -- 172.21.15.12:0/2235491384 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc188011830 con 0x7fc19c1613e0 2024-03-29T10:06:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.166+0000 7fc19b400700 1 -- 172.21.15.12:0/2235491384 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc188011990 con 0x7fc19c1613e0 2024-03-29T10:06:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.166+0000 7fc19b400700 1 -- 172.21.15.12:0/2235491384 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc188011d20 con 0x7fc19c1613e0 2024-03-29T10:06:49.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.167+0000 7fc1a3600700 1 -- 172.21.15.12:0/2235491384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 msgr2=0x7fc19c161860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:49.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.167+0000 7fc1a3600700 1 --2- 172.21.15.12:0/2235491384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c161860 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc188009840 tx=0x7fc188009b50 comp rx=0 tx=0).stop 2024-03-29T10:06:49.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.167+0000 7fc1a3600700 1 -- 172.21.15.12:0/2235491384 shutdown_connections 2024-03-29T10:06:49.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.167+0000 7fc1a3600700 1 --2- 172.21.15.12:0/2235491384 >> v2:172.21.15.159:3300/0 conn(0x7fc19c157f90 0x7fc19c0b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.167+0000 7fc1a3600700 1 --2- 172.21.15.12:0/2235491384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c161860 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.167+0000 7fc1a3600700 1 -- 172.21.15.12:0/2235491384 >> 172.21.15.12:0/2235491384 conn(0x7fc19c0b0d70 msgr2=0x7fc19c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:49.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.167+0000 7fc1a3600700 1 -- 172.21.15.12:0/2235491384 shutdown_connections 2024-03-29T10:06:49.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.167+0000 7fc1a3600700 1 -- 172.21.15.12:0/2235491384 wait complete. 2024-03-29T10:06:49.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.169+0000 7fc1a3600700 1 Processor -- start 2024-03-29T10:06:49.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.169+0000 7fc1a3600700 1 -- start start 2024-03-29T10:06:49.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc1a3600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc19c0b3420 0x7fc19c0c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc1a3600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc19c157f90 0x7fc19c099c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc1a3600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c1f6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc1a3600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0c9630 con 0x7fc19c157f90 2024-03-29T10:06:49.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc1a3600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0c9910 con 0x7fc19c0b3420 2024-03-29T10:06:49.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc1a3600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0c9bf0 con 0x7fc19c1613e0 2024-03-29T10:06:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc19be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc19c157f90 0x7fc19c099c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc19be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc19c157f90 0x7fc19c099c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36130/0 (socket says 172.21.15.12:36130) 2024-03-29T10:06:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc19be00700 1 -- 172.21.15.12:0/1917845153 learned_addr learned my addr 172.21.15.12:0/1917845153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc1a1600700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c1f6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.170+0000 7fc1a0c00700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc19c0b3420 0x7fc19c0c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc1a0c00700 1 -- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 msgr2=0x7fc19c1f6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc1a0c00700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c1f6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc1a0c00700 1 -- 172.21.15.12:0/1917845153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc19c157f90 msgr2=0x7fc19c099c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc1a0c00700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc19c157f90 0x7fc19c099c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc1a0c00700 1 -- 172.21.15.12:0/1917845153 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc190009550 con 0x7fc19c0b3420 2024-03-29T10:06:49.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc19be00700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc19c157f90 0x7fc19c099c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:49.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc1a1600700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c1f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:49.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc1a0c00700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc19c0b3420 0x7fc19c0c8e00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc19000e9a0 tx=0x7fc19000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:49.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc198c00700 1 -- 172.21.15.12:0/1917845153 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190004030 con 0x7fc19c0b3420 2024-03-29T10:06:49.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc198c00700 1 -- 172.21.15.12:0/1917845153 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc190004190 con 0x7fc19c0b3420 2024-03-29T10:06:49.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc198c00700 1 -- 172.21.15.12:0/1917845153 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190004520 con 0x7fc19c0b3420 2024-03-29T10:06:49.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc1a3600700 1 -- 172.21.15.12:0/1917845153 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1880094a0 con 0x7fc19c0b3420 2024-03-29T10:06:49.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.171+0000 7fc1a3600700 1 -- 172.21.15.12:0/1917845153 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c0c9340 con 0x7fc19c0b3420 2024-03-29T10:06:49.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.172+0000 7fc1a3600700 1 -- 172.21.15.12:0/1917845153 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c0c9e10 con 0x7fc19c0b3420 2024-03-29T10:06:49.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.173+0000 7fc198c00700 1 -- 172.21.15.12:0/1917845153 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fc1900036a0 con 0x7fc19c0b3420 2024-03-29T10:06:49.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.174+0000 7fc198c00700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17c06ae40 0x7fc17c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.174+0000 7fc198c00700 1 -- 172.21.15.12:0/1917845153 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7fc190089970 con 0x7fc19c0b3420 2024-03-29T10:06:49.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.174+0000 7fc19be00700 1 -- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17c06ae40 msgr2=0x7fc17c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:49.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.174+0000 7fc19be00700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17c06ae40 0x7fc17c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:49.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.179+0000 7fc198c00700 1 -- 172.21.15.12:0/1917845153 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc190054380 con 0x7fc19c0b3420 2024-03-29T10:06:49.321 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:49.322+0000 7ff4587aa240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:06:49.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.375+0000 7fc19be00700 1 -- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17c06ae40 msgr2=0x7fc17c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:49.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.375+0000 7fc19be00700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17c06ae40 0x7fc17c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:49.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.393+0000 7fc1a3600700 1 -- 172.21.15.12:0/1917845153 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc19c0c9a50 con 0x7fc19c0b3420 2024-03-29T10:06:49.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.393+0000 7fc198c00700 1 -- 172.21.15.12:0/1917845153 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v47) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc19c0c9a50 con 0x7fc19c0b3420 2024-03-29T10:06:49.393 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:06:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 -- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17c06ae40 msgr2=0x7fc17c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17c06ae40 0x7fc17c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 -- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc19c0b3420 msgr2=0x7fc19c0c8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc19c0b3420 0x7fc19c0c8e00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc19000e9a0 tx=0x7fc19000ecb0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 -- 172.21.15.12:0/1917845153 shutdown_connections 2024-03-29T10:06:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17c06ae40 0x7fc17c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19c1613e0 0x7fc19c1f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc19c0b3420 0x7fc19c0c8e00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 --2- 172.21.15.12:0/1917845153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc19c157f90 0x7fc19c099c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 -- 172.21.15.12:0/1917845153 >> 172.21.15.12:0/1917845153 conn(0x7fc19c0b0d70 msgr2=0x7fc19c156630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 -- 172.21.15.12:0/1917845153 shutdown_connections 2024-03-29T10:06:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.396+0000 7fc1a3600700 1 -- 172.21.15.12:0/1917845153 wait complete. 2024-03-29T10:06:49.416 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T10:06:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.617+0000 7ff379200700 1 Processor -- start 2024-03-29T10:06:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.617+0000 7ff379200700 1 -- start start 2024-03-29T10:06:49.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.618+0000 7ff379200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff374157f90 0x7ff3740b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.618+0000 7ff379200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff3740b3b30 0x7ff374161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.618+0000 7ff379200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3740b3070 con 0x7ff3741613e0 2024-03-29T10:06:49.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.618+0000 7ff379200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff374158370 con 0x7ff374157f90 2024-03-29T10:06:49.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.618+0000 7ff379200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740b2710 con 0x7ff3740b3b30 2024-03-29T10:06:49.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.618+0000 7ff371e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff374157f90 0x7ff3740b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.619+0000 7ff371e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff374157f90 0x7ff3740b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36146/0 (socket says 172.21.15.12:36146) 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.619+0000 7ff371e00700 1 -- 172.21.15.12:0/914503463 learned_addr learned my addr 172.21.15.12:0/914503463 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.619+0000 7ff372800700 1 --2- 172.21.15.12:0/914503463 >> v2:172.21.15.159:3300/0 conn(0x7ff3740b3b30 0x7ff374161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.619+0000 7ff371e00700 1 -- 172.21.15.12:0/914503463 >> v2:172.21.15.159:3300/0 conn(0x7ff3740b3b30 msgr2=0x7ff374161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.619+0000 7ff371e00700 1 --2- 172.21.15.12:0/914503463 >> v2:172.21.15.159:3300/0 conn(0x7ff3740b3b30 0x7ff374161890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.619+0000 7ff371e00700 1 -- 172.21.15.12:0/914503463 >> v1:172.21.15.159:6789/0 conn(0x7ff3741613e0 legacy=0x7ff3740b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.619+0000 7ff371e00700 1 -- 172.21.15.12:0/914503463 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740b2070 con 0x7ff374157f90 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.619+0000 7ff371e00700 1 --2- 172.21.15.12:0/914503463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 0x7ff3740b2b30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff37406eeb0 tx=0x7ff364009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ade0dc00539e8c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.619+0000 7ff370a00700 1 -- 172.21.15.12:0/914503463 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364011830 con 0x7ff374157f90 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.620+0000 7ff370a00700 1 -- 172.21.15.12:0/914503463 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff364011990 con 0x7ff374157f90 2024-03-29T10:06:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.620+0000 7ff370a00700 1 -- 172.21.15.12:0/914503463 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364011d20 con 0x7ff374157f90 2024-03-29T10:06:49.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.620+0000 7ff379200700 1 -- 172.21.15.12:0/914503463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 msgr2=0x7ff3740b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:49.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.620+0000 7ff379200700 1 --2- 172.21.15.12:0/914503463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 0x7ff3740b2b30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff37406eeb0 tx=0x7ff364009bd0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.620+0000 7ff379200700 1 -- 172.21.15.12:0/914503463 shutdown_connections 2024-03-29T10:06:49.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.620+0000 7ff379200700 1 --2- 172.21.15.12:0/914503463 >> v2:172.21.15.159:3300/0 conn(0x7ff3740b3b30 0x7ff374161890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.620+0000 7ff379200700 1 --2- 172.21.15.12:0/914503463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 0x7ff3740b2b30 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.620+0000 7ff379200700 1 -- 172.21.15.12:0/914503463 >> 172.21.15.12:0/914503463 conn(0x7ff3740b0d70 msgr2=0x7ff3740b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.620+0000 7ff379200700 1 -- 172.21.15.12:0/914503463 shutdown_connections 2024-03-29T10:06:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.620+0000 7ff379200700 1 -- 172.21.15.12:0/914503463 wait complete. 2024-03-29T10:06:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.622+0000 7ff379200700 1 Processor -- start 2024-03-29T10:06:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.622+0000 7ff379200700 1 -- start start 2024-03-29T10:06:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.623+0000 7ff379200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3740b3b30 0x7ff374153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.623+0000 7ff379200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 0x7ff37415c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.623+0000 7ff379200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3741613e0 0x7ff37415c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.623+0000 7ff379200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff374155db0 con 0x7ff374157f90 2024-03-29T10:06:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.623+0000 7ff379200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3741551a0 con 0x7ff3740b3b30 2024-03-29T10:06:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.623+0000 7ff379200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff374155480 con 0x7ff3741613e0 2024-03-29T10:06:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.623+0000 7ff371400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 0x7ff37415c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.623+0000 7ff371400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 0x7ff37415c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36162/0 (socket says 172.21.15.12:36162) 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.623+0000 7ff371400700 1 -- 172.21.15.12:0/3310931631 learned_addr learned my addr 172.21.15.12:0/3310931631 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.623+0000 7ff371e00700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3740b3b30 0x7ff374153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff372800700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3741613e0 0x7ff37415c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff371400700 1 -- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3741613e0 msgr2=0x7ff37415c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff371400700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3741613e0 0x7ff37415c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff371400700 1 -- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3740b3b30 msgr2=0x7ff374153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff371400700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3740b3b30 0x7ff374153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff371400700 1 -- 172.21.15.12:0/3310931631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff364009550 con 0x7ff374157f90 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff371400700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 0x7ff37415c400 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff36000c490 tx=0x7ff36000c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff36a000700 1 -- 172.21.15.12:0/3310931631 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36000cea0 con 0x7ff374157f90 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff36a000700 1 -- 172.21.15.12:0/3310931631 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3600052f0 con 0x7ff374157f90 2024-03-29T10:06:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff36a000700 1 -- 172.21.15.12:0/3310931631 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360006860 con 0x7ff374157f90 2024-03-29T10:06:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.624+0000 7ff379200700 1 -- 172.21.15.12:0/3310931631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff374155760 con 0x7ff374157f90 2024-03-29T10:06:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.625+0000 7ff379200700 1 -- 172.21.15.12:0/3310931631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff374153640 con 0x7ff374157f90 2024-03-29T10:06:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.626+0000 7ff379200700 1 -- 172.21.15.12:0/3310931631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff374155f60 con 0x7ff374157f90 2024-03-29T10:06:49.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.627+0000 7ff36a000700 1 -- 172.21.15.12:0/3310931631 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7ff360003680 con 0x7ff374157f90 2024-03-29T10:06:49.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.627+0000 7ff36a000700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35406ae40 0x7ff35406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:49.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.628+0000 7ff36a000700 1 -- 172.21.15.12:0/3310931631 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7ff360088a60 con 0x7ff374157f90 2024-03-29T10:06:49.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.628+0000 7ff371e00700 1 -- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35406ae40 msgr2=0x7ff35406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:49.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.628+0000 7ff371e00700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35406ae40 0x7ff35406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.632+0000 7ff36a000700 1 -- 172.21.15.12:0/3310931631 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff374155f60 con 0x7ff374157f90 2024-03-29T10:06:49.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.829+0000 7ff371e00700 1 -- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35406ae40 msgr2=0x7ff35406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:49.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.829+0000 7ff371e00700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35406ae40 0x7ff35406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:49.847 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:49.848+0000 7f9625f86240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:06:49.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.861+0000 7ff379200700 1 -- 172.21.15.12:0/3310931631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff3740b3070 con 0x7ff374157f90 2024-03-29T10:06:49.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.862+0000 7ff36a000700 1 -- 172.21.15.12:0/3310931631 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v47) v1 ==== 80+0+348 (secure 0 0 0) 0x7ff3740b3070 con 0x7ff374157f90 2024-03-29T10:06:49.862 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/43444 2024-03-30T09:55:52.597859+0000 2024-03-29T10:06:49.862 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3395907487 2024-03-30T09:55:52.597859+0000 2024-03-29T10:06:49.862 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/43444 2024-03-30T09:55:52.597859+0000 2024-03-29T10:06:49.862 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/249193272 2024-03-30T09:55:52.597859+0000 2024-03-29T10:06:49.863 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/795330822 2024-03-30T09:55:52.597859+0000 2024-03-29T10:06:49.863 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1756693282 2024-03-30T09:55:52.597859+0000 2024-03-29T10:06:49.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.864+0000 7ff379200700 1 -- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35406ae40 msgr2=0x7ff35406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:49.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.864+0000 7ff379200700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35406ae40 0x7ff35406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.864+0000 7ff379200700 1 -- 172.21.15.12:0/3310931631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 msgr2=0x7ff37415c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:49.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.864+0000 7ff379200700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 0x7ff37415c400 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff36000c490 tx=0x7ff36000c7a0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.865+0000 7ff379200700 1 -- 172.21.15.12:0/3310931631 shutdown_connections 2024-03-29T10:06:49.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.865+0000 7ff379200700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35406ae40 0x7ff35406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.865+0000 7ff379200700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3741613e0 0x7ff37415c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.865+0000 7ff379200700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3740b3b30 0x7ff374153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.865+0000 7ff379200700 1 --2- 172.21.15.12:0/3310931631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff374157f90 0x7ff37415c400 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:49.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.865+0000 7ff379200700 1 -- 172.21.15.12:0/3310931631 >> 172.21.15.12:0/3310931631 conn(0x7ff3740b0d70 msgr2=0x7ff3740ae280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:49.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.865+0000 7ff379200700 1 -- 172.21.15.12:0/3310931631 shutdown_connections 2024-03-29T10:06:49.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:49.865+0000 7ff379200700 1 -- 172.21.15.12:0/3310931631 wait complete. 2024-03-29T10:06:49.865 INFO:teuthology.orchestra.run.smithi012.stderr:listed 6 entries 2024-03-29T10:06:49.883 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T10:06:49.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:49.884 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T10:06:49.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:49.941+0000 7f9625f86240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:06:49.964 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T10:06:49.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:06:49.964 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T10:06:50.016 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:06:50.016 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:06:50.061 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:06:50.061 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:06:50.062 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:06:50.100 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:50.100+0000 7ff4587aa240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:06:50.103 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:50.104+0000 7f9625f86240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:06:50.107 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:06:50.107 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:06:50.107 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:06:50.153 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:06:50.153 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:06:50.203 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:50.203+0000 7ff4587aa240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:06:50.322 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:50.323+0000 7f9625f86240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:06:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.357+0000 7f6b0a400700 1 Processor -- start 2024-03-29T10:06:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.358+0000 7f6b0a400700 1 -- start start 2024-03-29T10:06:50.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.359+0000 7f6b0a400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6b040cb3f0 0x7f6b040ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.359+0000 7f6b0a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b040ac600 0x7f6b0406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.359+0000 7f6b0a400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0415fd50 con 0x7f6b0415b5d0 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.359+0000 7f6b0a400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b0415b9b0 con 0x7f6b040cb3f0 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.359+0000 7f6b0a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040aca50 con 0x7f6b040ac600 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.359+0000 7f6b03400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6b0415b5d0 0x7f6b040ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57510/0 (socket says 172.21.15.12:57510) 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.359+0000 7f6b03400700 1 -- 172.21.15.12:0/204779338 learned_addr learned my addr 172.21.15.12:0/204779338 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.359+0000 7f6b02a00700 1 --2- 172.21.15.12:0/204779338 >> v2:172.21.15.159:3300/0 conn(0x7f6b040cb3f0 0x7f6b040ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.359+0000 7f6b03e00700 1 --2- 172.21.15.12:0/204779338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b040ac600 0x7f6b0406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.360+0000 7f6b02000700 1 -- 172.21.15.12:0/204779338 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556160806 0 0) 0x7f6b0415fd50 con 0x7f6b0415b5d0 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.360+0000 7f6b02000700 1 -- 172.21.15.12:0/204779338 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae80035b0 con 0x7f6b0415b5d0 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.360+0000 7f6b02a00700 1 -- 172.21.15.12:0/204779338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b040ac600 msgr2=0x7f6b0406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.360+0000 7f6b02a00700 1 --2- 172.21.15.12:0/204779338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b040ac600 0x7f6b0406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.360+0000 7f6b02a00700 1 -- 172.21.15.12:0/204779338 >> v1:172.21.15.12:6789/0 conn(0x7f6b0415b5d0 legacy=0x7f6b040ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.360+0000 7f6b02a00700 1 -- 172.21.15.12:0/204779338 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0415f140 con 0x7f6b040cb3f0 2024-03-29T10:06:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.360+0000 7f6b02a00700 1 --2- 172.21.15.12:0/204779338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040cb3f0 0x7f6b040ac0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009840 tx=0x7f6af0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b471137685a80a08 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.360+0000 7f6b02000700 1 -- 172.21.15.12:0/204779338 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0011830 con 0x7f6b040cb3f0 2024-03-29T10:06:50.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.360+0000 7f6b02000700 1 -- 172.21.15.12:0/204779338 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af0011990 con 0x7f6b040cb3f0 2024-03-29T10:06:50.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.360+0000 7f6b02000700 1 -- 172.21.15.12:0/204779338 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0011d20 con 0x7f6b040cb3f0 2024-03-29T10:06:50.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.361+0000 7f6b0a400700 1 -- 172.21.15.12:0/204779338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040cb3f0 msgr2=0x7f6b040ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:50.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.361+0000 7f6b0a400700 1 --2- 172.21.15.12:0/204779338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040cb3f0 0x7f6b040ac0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009840 tx=0x7f6af0009b50 comp rx=0 tx=0).stop 2024-03-29T10:06:50.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.361+0000 7f6b0a400700 1 -- 172.21.15.12:0/204779338 shutdown_connections 2024-03-29T10:06:50.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.361+0000 7f6b0a400700 1 --2- 172.21.15.12:0/204779338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b040ac600 0x7f6b0406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.361+0000 7f6b0a400700 1 --2- 172.21.15.12:0/204779338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040cb3f0 0x7f6b040ac0c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.361+0000 7f6b0a400700 1 -- 172.21.15.12:0/204779338 >> 172.21.15.12:0/204779338 conn(0x7f6b040cf190 msgr2=0x7f6b040cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:50.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.361+0000 7f6b0a400700 1 -- 172.21.15.12:0/204779338 shutdown_connections 2024-03-29T10:06:50.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.361+0000 7f6b0a400700 1 -- 172.21.15.12:0/204779338 wait complete. 2024-03-29T10:06:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.363+0000 7f6b0a400700 1 Processor -- start 2024-03-29T10:06:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.363+0000 7f6b0a400700 1 -- start start 2024-03-29T10:06:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.364+0000 7f6b0a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040ac600 0x7f6b04070e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.364+0000 7f6b0a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b0415b5d0 0x7f6b040713b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.364+0000 7f6b0a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b040afa00 0x7f6b040ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.364+0000 7f6b0a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b04070d00 con 0x7f6b040afa00 2024-03-29T10:06:50.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.364+0000 7f6b0a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b04071d10 con 0x7f6b040ac600 2024-03-29T10:06:50.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.364+0000 7f6b0a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b04070a60 con 0x7f6b0415b5d0 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.364+0000 7f6b03400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040ac600 0x7f6b04070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.364+0000 7f6b03e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b040afa00 0x7f6b040ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.364+0000 7f6b03400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040ac600 0x7f6b04070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:45672/0 (socket says 172.21.15.12:45672) 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.364+0000 7f6b03400700 1 -- 172.21.15.12:0/193061157 learned_addr learned my addr 172.21.15.12:0/193061157 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.365+0000 7f6b02a00700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b0415b5d0 0x7f6b040713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.365+0000 7f6b03400700 1 -- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b0415b5d0 msgr2=0x7f6b040713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.365+0000 7f6b03400700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b0415b5d0 0x7f6b040713b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.365+0000 7f6b03400700 1 -- 172.21.15.12:0/193061157 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b040afa00 msgr2=0x7f6b040ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.365+0000 7f6b03400700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b040afa00 0x7f6b040ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.365+0000 7f6b03400700 1 -- 172.21.15.12:0/193061157 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af00094a0 con 0x7f6b040ac600 2024-03-29T10:06:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.365+0000 7f6b03400700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040ac600 0x7f6b04070e70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6af4001eb0 tx=0x7f6af400c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.365+0000 7f6afb400700 1 -- 172.21.15.12:0/193061157 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4005d70 con 0x7f6b040ac600 2024-03-29T10:06:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.366+0000 7f6afb400700 1 -- 172.21.15.12:0/193061157 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af40070d0 con 0x7f6b040ac600 2024-03-29T10:06:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.366+0000 7f6afb400700 1 -- 172.21.15.12:0/193061157 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4007460 con 0x7f6b040ac600 2024-03-29T10:06:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.366+0000 7f6b0a400700 1 -- 172.21.15.12:0/193061157 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b04144f40 con 0x7f6b040ac600 2024-03-29T10:06:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.366+0000 7f6b0a400700 1 -- 172.21.15.12:0/193061157 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b040ce8f0 con 0x7f6b040ac600 2024-03-29T10:06:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.367+0000 7f6b0a400700 1 -- 172.21.15.12:0/193061157 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b04144da0 con 0x7f6b040ac600 2024-03-29T10:06:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.368+0000 7f6afb400700 1 -- 172.21.15.12:0/193061157 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f6af40075c0 con 0x7f6b040ac600 2024-03-29T10:06:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.369+0000 7f6afb400700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ae406ae40 0x7f6ae406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.369+0000 7f6afb400700 1 -- 172.21.15.12:0/193061157 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f6af4088f40 con 0x7f6b040ac600 2024-03-29T10:06:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.369+0000 7f6b02a00700 1 -- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ae406ae40 msgr2=0x7f6ae406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.369+0000 7f6b02a00700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ae406ae40 0x7f6ae406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:50.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.376+0000 7f6afb400700 1 -- 172.21.15.12:0/193061157 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6af40539d0 con 0x7f6b040ac600 2024-03-29T10:06:50.378 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:50.378+0000 7ff4587aa240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:06:50.417 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:50.418+0000 7f9625f86240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:06:50.514 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:50.515+0000 7f9625f86240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:06:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.570+0000 7f6b02a00700 1 -- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ae406ae40 msgr2=0x7f6ae406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:50.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.570+0000 7f6b02a00700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ae406ae40 0x7f6ae406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:50.598 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:50.598+0000 7ff4587aa240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:06:50.611 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:50.612+0000 7f9625f86240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:06:50.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.637+0000 7f6b0a400700 1 -- 172.21.15.12:0/193061157 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b04144da0 con 0x7f6b040ac600 2024-03-29T10:06:50.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.638+0000 7f6afb400700 1 -- 172.21.15.12:0/193061157 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f6b04144da0 con 0x7f6b040ac600 2024-03-29T10:06:50.637 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:50.637 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2845880355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2845880355},{"type":"v1","addr":"172.21.15.159:6837","nonce":2845880355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2836227468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2836227468},{"type":"v1","addr":"172.21.15.159:6836","nonce":2836227468}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1863550973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1863550973},{"type":"v1","addr":"172.21.15.12:6834","nonce":1863550973}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:55:35.438233+0000","modified":"2024-03-29T09:55:38.926199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/2013939805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2013939805},{"type":"v1","addr":"172.21.15.12:6835","nonce":2013939805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-29T10:06:50.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 -- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ae406ae40 msgr2=0x7f6ae406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:50.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ae406ae40 0x7f6ae406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 -- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040ac600 msgr2=0x7f6b04070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:50.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040ac600 0x7f6b04070e70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6af4001eb0 tx=0x7f6af400c7b0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 -- 172.21.15.12:0/193061157 shutdown_connections 2024-03-29T10:06:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ae406ae40 0x7f6ae406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b040afa00 0x7f6b040ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b040ac600 0x7f6b04070e70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 --2- 172.21.15.12:0/193061157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b0415b5d0 0x7f6b040713b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 -- 172.21.15.12:0/193061157 >> 172.21.15.12:0/193061157 conn(0x7f6b040cf190 msgr2=0x7f6b0406dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 -- 172.21.15.12:0/193061157 shutdown_connections 2024-03-29T10:06:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.640+0000 7f6b0a400700 1 -- 172.21.15.12:0/193061157 wait complete. 2024-03-29T10:06:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-29T10:06:50.658 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:06:50.701 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:50.701+0000 7ff4587aa240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:06:50.709 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:50.710+0000 7f9625f86240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:06:50.807 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:50.807+0000 7ff4587aa240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:06:50.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.858+0000 7f5699600700 1 Processor -- start 2024-03-29T10:06:50.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.858+0000 7f5699600700 1 -- start start 2024-03-29T10:06:50.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.860+0000 7f5699600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f569415d640 0x7f56940d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.860+0000 7f5699600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f56940ceec0 0x7f56941612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.860+0000 7f5699600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56940d30d0 con 0x7f5694161800 2024-03-29T10:06:50.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.860+0000 7f5699600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5694161c90 con 0x7f56940ceec0 2024-03-29T10:06:50.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.860+0000 7f5699600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56940b6840 con 0x7f569415d640 2024-03-29T10:06:50.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.860+0000 7f5692000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f56940ceec0 0x7f56941612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:50.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.860+0000 7f5693400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5694161800 0x7f56940cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57518/0 (socket says 172.21.15.12:57518) 2024-03-29T10:06:50.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.860+0000 7f5693400700 1 -- 172.21.15.12:0/923298648 learned_addr learned my addr 172.21.15.12:0/923298648 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:50.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.860+0000 7f5692a00700 1 --2- 172.21.15.12:0/923298648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f569415d640 0x7f56940d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:50.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.861+0000 7f5692000700 1 -- 172.21.15.12:0/923298648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f569415d640 msgr2=0x7f56940d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:50.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.861+0000 7f5692000700 1 --2- 172.21.15.12:0/923298648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f569415d640 0x7f56940d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.861+0000 7f5692000700 1 -- 172.21.15.12:0/923298648 >> v1:172.21.15.12:6789/0 conn(0x7f5694161800 legacy=0x7f56940cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:50.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.861+0000 7f5692000700 1 -- 172.21.15.12:0/923298648 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56940d2880 con 0x7f56940ceec0 2024-03-29T10:06:50.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.861+0000 7f5691600700 1 -- 172.21.15.12:0/923298648 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481751326 0 0) 0x7f56940d30d0 con 0x7f5694161800 2024-03-29T10:06:50.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.861+0000 7f5692000700 1 --2- 172.21.15.12:0/923298648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56940ceec0 0x7f56941612c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5680009840 tx=0x7f5680009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8b23fc3fa4b47f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.861+0000 7f5691600700 1 -- 172.21.15.12:0/923298648 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5680011830 con 0x7f56940ceec0 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.861+0000 7f5691600700 1 -- 172.21.15.12:0/923298648 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5680011990 con 0x7f56940ceec0 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.861+0000 7f5691600700 1 -- 172.21.15.12:0/923298648 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5680011d20 con 0x7f56940ceec0 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.862+0000 7f5699600700 1 -- 172.21.15.12:0/923298648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56940ceec0 msgr2=0x7f56941612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.862+0000 7f5699600700 1 --2- 172.21.15.12:0/923298648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56940ceec0 0x7f56941612c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5680009840 tx=0x7f5680009b50 comp rx=0 tx=0).stop 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.862+0000 7f5699600700 1 -- 172.21.15.12:0/923298648 shutdown_connections 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.862+0000 7f5699600700 1 --2- 172.21.15.12:0/923298648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f569415d640 0x7f56940d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.862+0000 7f5699600700 1 --2- 172.21.15.12:0/923298648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56940ceec0 0x7f56941612c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.862+0000 7f5699600700 1 -- 172.21.15.12:0/923298648 >> 172.21.15.12:0/923298648 conn(0x7f569408aff0 msgr2=0x7f56940d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.862+0000 7f5699600700 1 -- 172.21.15.12:0/923298648 shutdown_connections 2024-03-29T10:06:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.862+0000 7f5699600700 1 -- 172.21.15.12:0/923298648 wait complete. 2024-03-29T10:06:50.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.864+0000 7f5699600700 1 Processor -- start 2024-03-29T10:06:50.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.864+0000 7f5699600700 1 -- start start 2024-03-29T10:06:50.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5699600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56940ceec0 0x7f5694143fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5699600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f569415d640 0x7f5694143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5699600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5694161800 0x7f5694154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5699600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56941437c0 con 0x7f569415d640 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5699600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569415f6a0 con 0x7f56940ceec0 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5699600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56941554d0 con 0x7f5694161800 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5692000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f569415d640 0x7f5694143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5693400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5694161800 0x7f5694154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5693400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5694161800 0x7f5694154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:43456/0 (socket says 172.21.15.12:43456) 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5692000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f569415d640 0x7f5694143280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36192/0 (socket says 172.21.15.12:36192) 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5693400700 1 -- 172.21.15.12:0/4017183067 learned_addr learned my addr 172.21.15.12:0/4017183067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.865+0000 7f5692a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56940ceec0 0x7f5694143fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.866+0000 7f5693400700 1 -- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56940ceec0 msgr2=0x7f5694143fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:50.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.866+0000 7f5693400700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56940ceec0 0x7f5694143fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.866+0000 7f5693400700 1 -- 172.21.15.12:0/4017183067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f569415d640 msgr2=0x7f5694143280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:50.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.866+0000 7f5693400700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f569415d640 0x7f5694143280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:50.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.866+0000 7f5693400700 1 -- 172.21.15.12:0/4017183067 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56800094a0 con 0x7f5694161800 2024-03-29T10:06:50.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.866+0000 7f5692000700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f569415d640 0x7f5694143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:50.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.866+0000 7f5693400700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5694161800 0x7f5694154f90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f568c002230 tx=0x7f568c00cf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.866+0000 7f568aa00700 1 -- 172.21.15.12:0/4017183067 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c00aac0 con 0x7f5694161800 2024-03-29T10:06:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.866+0000 7f568aa00700 1 -- 172.21.15.12:0/4017183067 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568c00ac40 con 0x7f5694161800 2024-03-29T10:06:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.866+0000 7f5699600700 1 -- 172.21.15.12:0/4017183067 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569415ed90 con 0x7f5694161800 2024-03-29T10:06:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.867+0000 7f568aa00700 1 -- 172.21.15.12:0/4017183067 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c0070d0 con 0x7f5694161800 2024-03-29T10:06:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.867+0000 7f5699600700 1 -- 172.21.15.12:0/4017183067 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569415f050 con 0x7f5694161800 2024-03-29T10:06:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.868+0000 7f5699600700 1 -- 172.21.15.12:0/4017183067 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569415f810 con 0x7f5694161800 2024-03-29T10:06:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.869+0000 7f568aa00700 1 -- 172.21.15.12:0/4017183067 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f568c007250 con 0x7f5694161800 2024-03-29T10:06:50.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.869+0000 7f568aa00700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f567406ae40 0x7f567406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:50.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.869+0000 7f568aa00700 1 -- 172.21.15.12:0/4017183067 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f568c0895f0 con 0x7f5694161800 2024-03-29T10:06:50.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.870+0000 7f5692a00700 1 -- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f567406ae40 msgr2=0x7f567406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:50.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.870+0000 7f5692a00700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f567406ae40 0x7f567406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:50.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:50.875+0000 7f568aa00700 1 -- 172.21.15.12:0/4017183067 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f568c054000 con 0x7f5694161800 2024-03-29T10:06:50.915 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:50.915+0000 7ff4587aa240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:06:50.925 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:50.924+0000 7f9625f86240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:06:51.019 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:51.020+0000 7ff4587aa240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:06:51.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.071+0000 7f5692a00700 1 -- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f567406ae40 msgr2=0x7f567406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:51.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.071+0000 7f5692a00700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f567406ae40 0x7f567406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:51.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.093+0000 7f5699600700 1 -- 172.21.15.12:0/4017183067 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5694067720 con 0x7f5694161800 2024-03-29T10:06:51.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.095+0000 7f568aa00700 1 -- 172.21.15.12:0/4017183067 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+17906 (secure 0 0 0) 0x7f568c057620 con 0x7f5694161800 2024-03-29T10:06:51.095 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:51.095 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":47,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:06:47.985612+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:55:33.508935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T09:55:34.418367+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":219,"snap_epoch":47,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6832/43444":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/3395907487":"2024-03-30T09:55:52.597859+0000","172.21.15.159:6833/43444":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/249193272":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/795330822":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/1756693282":"2024-03-30T09:55:52.597859+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":6,"length":1},{"begin":9,"length":1},{"begin":12,"length":1},{"begin":16,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":27,"length":1},{"begin":32,"length":1},{"begin":34,"length":1},{"begin":38,"length":1},{"begin":43,"length":1},{"begin":45,"length":1},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":55,"length":1},{"begin":59,"length":1},{"begin":64,"length":1},{"begin":66,"length":1},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":76,"length":1},{"begin":80,"length":2},{"begin":85,"length":1},{"begin":87,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":109,"length":1},{"begin":187,"length":33}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":109,"length":1},{"begin":219,"length":1}]}],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":4,"length":1},{"begin":7,"length":1},{"begin":10,"length":1},{"begin":14,"length":2},{"begin":19,"length":1},{"begin":25,"length":2},{"begin":30,"length":2},{"begin":36,"length":2},{"begin":41,"length":2},{"begin":47,"length":2},{"begin":51,"length":2},{"begin":57,"length":2},{"begin":62,"length":2},{"begin":68,"length":2},{"begin":72,"length":2},{"begin":78,"length":2},{"begin":83,"length":2},{"begin":89,"length":3},{"begin":94,"length":2},{"begin":100,"length":3},{"begin":105,"length":2},{"begin":147,"length":40}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:06:51.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 -- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f567406ae40 msgr2=0x7f567406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:51.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f567406ae40 0x7f567406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 -- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5694161800 msgr2=0x7f5694154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5694161800 0x7f5694154f90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f568c002230 tx=0x7f568c00cf50 comp rx=0 tx=0).stop 2024-03-29T10:06:51.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 -- 172.21.15.12:0/4017183067 shutdown_connections 2024-03-29T10:06:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f567406ae40 0x7f567406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5694161800 0x7f5694154f90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56940ceec0 0x7f5694143fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 --2- 172.21.15.12:0/4017183067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f569415d640 0x7f5694143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 -- 172.21.15.12:0/4017183067 >> 172.21.15.12:0/4017183067 conn(0x7f569408aff0 msgr2=0x7f569406d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 -- 172.21.15.12:0/4017183067 shutdown_connections 2024-03-29T10:06:51.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.097+0000 7f5699600700 1 -- 172.21.15.12:0/4017183067 wait complete. 2024-03-29T10:06:51.116 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T09:55:33.508935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T09:55:34.418367+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 219, 'snap_epoch': 47, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:06:51.116 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:06:51.185 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:51.186+0000 7f9625f86240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:06:51.250 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:51.250+0000 7ff4587aa240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:06:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.363+0000 7f7488000700 1 Processor -- start 2024-03-29T10:06:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.363+0000 7f7488000700 1 -- start start 2024-03-29T10:06:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.364+0000 7f7488000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cd020 0x7f74800cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.364+0000 7f7488000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74800cdae0 con 0x7f74800cbb30 2024-03-29T10:06:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.364+0000 7f7488000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74800cddc0 con 0x7f74800b2070 2024-03-29T10:06:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.364+0000 7f7488000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74800ce0a0 con 0x7f74800cd020 2024-03-29T10:06:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.364+0000 7f7485600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f74800cbb30 0x7f74800cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57520/0 (socket says 172.21.15.12:57520) 2024-03-29T10:06:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.365+0000 7f7485600700 1 -- 172.21.15.12:0/2580934385 learned_addr learned my addr 172.21.15.12:0/2580934385 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.365+0000 7f7484c00700 1 --2- 172.21.15.12:0/2580934385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cd020 0x7f74800cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:51.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.365+0000 7f747fe00700 1 -- 172.21.15.12:0/2580934385 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658524006 0 0) 0x7f74800cdae0 con 0x7f74800cbb30 2024-03-29T10:06:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.365+0000 7f747fe00700 1 -- 172.21.15.12:0/2580934385 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74680035b0 con 0x7f74800cbb30 2024-03-29T10:06:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.365+0000 7f747fe00700 1 -- 172.21.15.12:0/2580934385 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808397184 0 0) 0x7f74800cddc0 con 0x7f74800b2070 2024-03-29T10:06:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.365+0000 7f7484c00700 1 -- 172.21.15.12:0/2580934385 >> v1:172.21.15.159:6789/0 conn(0x7f74800b2070 legacy=0x7f748006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.365+0000 7f7484c00700 1 -- 172.21.15.12:0/2580934385 >> v1:172.21.15.12:6789/0 conn(0x7f74800cbb30 legacy=0x7f74800cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.365+0000 7f7484c00700 1 -- 172.21.15.12:0/2580934385 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748006dc20 con 0x7f74800cd020 2024-03-29T10:06:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.365+0000 7f7484c00700 1 --2- 172.21.15.12:0/2580934385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cd020 0x7f74800cd480 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f746c009870 tx=0x7f746c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44dbe41fed4d9e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.366+0000 7f747fe00700 1 -- 172.21.15.12:0/2580934385 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c011830 con 0x7f74800cd020 2024-03-29T10:06:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.366+0000 7f747fe00700 1 -- 172.21.15.12:0/2580934385 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c011990 con 0x7f74800cd020 2024-03-29T10:06:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.366+0000 7f747fe00700 1 -- 172.21.15.12:0/2580934385 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c011d20 con 0x7f74800cd020 2024-03-29T10:06:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.366+0000 7f7488000700 1 -- 172.21.15.12:0/2580934385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cd020 msgr2=0x7f74800cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.366+0000 7f7488000700 1 --2- 172.21.15.12:0/2580934385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cd020 0x7f74800cd480 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f746c009870 tx=0x7f746c009b80 comp rx=0 tx=0).stop 2024-03-29T10:06:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.367+0000 7f7488000700 1 -- 172.21.15.12:0/2580934385 shutdown_connections 2024-03-29T10:06:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.367+0000 7f7488000700 1 --2- 172.21.15.12:0/2580934385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cd020 0x7f74800cd480 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.367+0000 7f7488000700 1 -- 172.21.15.12:0/2580934385 >> 172.21.15.12:0/2580934385 conn(0x7f74800b0d70 msgr2=0x7f74800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.367+0000 7f7488000700 1 -- 172.21.15.12:0/2580934385 shutdown_connections 2024-03-29T10:06:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.367+0000 7f7488000700 1 -- 172.21.15.12:0/2580934385 wait complete. 2024-03-29T10:06:51.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.369+0000 7f7488000700 1 Processor -- start 2024-03-29T10:06:51.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.369+0000 7f7488000700 1 -- start start 2024-03-29T10:06:51.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7488000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74800b2070 0x7f74800c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7488000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cbb30 0x7f74801f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7488000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74800cd020 0x7f74800ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7488000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74800ad970 con 0x7f74800b2070 2024-03-29T10:06:51.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7488000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748014fdc0 con 0x7f74800cd020 2024-03-29T10:06:51.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7488000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74800adbc0 con 0x7f74800cbb30 2024-03-29T10:06:51.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7485600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74800b2070 0x7f74800c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:51.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7484c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cbb30 0x7f74801f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:51.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7485600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74800b2070 0x7f74800c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36204/0 (socket says 172.21.15.12:36204) 2024-03-29T10:06:51.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7485600700 1 -- 172.21.15.12:0/2213957481 learned_addr learned my addr 172.21.15.12:0/2213957481 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:51.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7486000700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74800cd020 0x7f74800ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:51.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7484c00700 1 -- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74800cd020 msgr2=0x7f74800ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7484c00700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74800cd020 0x7f74800ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7484c00700 1 -- 172.21.15.12:0/2213957481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74800b2070 msgr2=0x7f74800c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7484c00700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74800b2070 0x7f74800c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.370+0000 7f7484c00700 1 -- 172.21.15.12:0/2213957481 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746c0094a0 con 0x7f74800cbb30 2024-03-29T10:06:51.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.371+0000 7f7484c00700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cbb30 0x7f74801f29d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f746c016000 tx=0x7f746c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:51.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.371+0000 7f747d600700 1 -- 172.21.15.12:0/2213957481 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c01a7e0 con 0x7f74800cbb30 2024-03-29T10:06:51.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.371+0000 7f747d600700 1 -- 172.21.15.12:0/2213957481 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c01a960 con 0x7f74800cbb30 2024-03-29T10:06:51.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.371+0000 7f7488000700 1 -- 172.21.15.12:0/2213957481 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74800adea0 con 0x7f74800cbb30 2024-03-29T10:06:51.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.371+0000 7f747d600700 1 -- 172.21.15.12:0/2213957481 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c01ad10 con 0x7f74800cbb30 2024-03-29T10:06:51.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.371+0000 7f7488000700 1 -- 172.21.15.12:0/2213957481 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74801f2f10 con 0x7f74800cbb30 2024-03-29T10:06:51.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.372+0000 7f7488000700 1 -- 172.21.15.12:0/2213957481 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74800709c0 con 0x7f74800cbb30 2024-03-29T10:06:51.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.373+0000 7f747d600700 1 -- 172.21.15.12:0/2213957481 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f746c022070 con 0x7f74800cbb30 2024-03-29T10:06:51.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.374+0000 7f747d600700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f746006ae40 0x7f746006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.374+0000 7f747d600700 1 -- 172.21.15.12:0/2213957481 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f746c08aba0 con 0x7f74800cbb30 2024-03-29T10:06:51.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.374+0000 7f7485600700 1 -- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f746006ae40 msgr2=0x7f746006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:51.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.374+0000 7f7485600700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f746006ae40 0x7f746006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:51.377 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:51.378+0000 7f9625f86240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:06:51.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.379+0000 7f747d600700 1 -- 172.21.15.12:0/2213957481 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f74800709c0 con 0x7f74800cbb30 2024-03-29T10:06:51.469 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:51.470+0000 7f9625f86240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:06:51.517 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:51.517+0000 7ff4587aa240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:06:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.575+0000 7f7485600700 1 -- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f746006ae40 msgr2=0x7f746006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.575+0000 7f7485600700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f746006ae40 0x7f746006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:51.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.594+0000 7f7488000700 1 -- 172.21.15.12:0/2213957481 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7480070da0 con 0x7f74800cbb30 2024-03-29T10:06:51.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.594+0000 7f747d600700 1 -- 172.21.15.12:0/2213957481 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7480070da0 con 0x7f74800cbb30 2024-03-29T10:06:51.594 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:06:51.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 -- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f746006ae40 msgr2=0x7f746006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:51.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f746006ae40 0x7f746006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 -- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cbb30 msgr2=0x7f74801f29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cbb30 0x7f74801f29d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f746c016000 tx=0x7f746c004750 comp rx=0 tx=0).stop 2024-03-29T10:06:51.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 -- 172.21.15.12:0/2213957481 shutdown_connections 2024-03-29T10:06:51.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f746006ae40 0x7f746006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74800cd020 0x7f74800ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74800b2070 0x7f74800c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 --2- 172.21.15.12:0/2213957481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74800cbb30 0x7f74801f29d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 -- 172.21.15.12:0/2213957481 >> 172.21.15.12:0/2213957481 conn(0x7f74800b0d70 msgr2=0x7f7480073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:51.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 -- 172.21.15.12:0/2213957481 shutdown_connections 2024-03-29T10:06:51.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.597+0000 7f7488000700 1 -- 172.21.15.12:0/2213957481 wait complete. 2024-03-29T10:06:51.617 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:06:51.721 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:51.722+0000 7ff4587aa240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:06:51.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.820+0000 7f1c0f000700 1 Processor -- start 2024-03-29T10:06:51.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.820+0000 7f1c0f000700 1 -- start start 2024-03-29T10:06:51.820 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:51.821+0000 7ff4587aa240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:06:51.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.821+0000 7f1c0f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c081611b0 0x7f1c0815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.821+0000 7f1c0f000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c080cee10 0x7f1c08161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.821+0000 7f1c0f000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c080ce270 con 0x7f1c0815f4d0 2024-03-29T10:06:51.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.821+0000 7f1c0f000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0815b600 con 0x7f1c080cee10 2024-03-29T10:06:51.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.821+0000 7f1c0f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080d2760 con 0x7f1c081611b0 2024-03-29T10:06:51.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c0d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c080cee10 0x7f1c08161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c07e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c081611b0 0x7f1c0815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c0d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c080cee10 0x7f1c08161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36208/0 (socket says 172.21.15.12:36208) 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c0d000700 1 -- 172.21.15.12:0/3686683117 learned_addr learned my addr 172.21.15.12:0/3686683117 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c07400700 1 --1- 172.21.15.12:0/3686683117 >> v1:172.21.15.159:6789/0 conn(0x7f1c0815f4d0 0x7f1c080ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:53150/0 (socket says 172.21.15.12:53150) 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c0d000700 1 -- 172.21.15.12:0/3686683117 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c081611b0 msgr2=0x7f1c0815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c0d000700 1 --2- 172.21.15.12:0/3686683117 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c081611b0 0x7f1c0815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c0d000700 1 -- 172.21.15.12:0/3686683117 >> v1:172.21.15.159:6789/0 conn(0x7f1c0815f4d0 legacy=0x7f1c080ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c06a00700 1 -- 172.21.15.12:0/3686683117 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215952088 0 0) 0x7f1c080ce270 con 0x7f1c0815f4d0 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c0d000700 1 -- 172.21.15.12:0/3686683117 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c080d2a40 con 0x7f1c080cee10 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.822+0000 7f1c0d000700 1 --2- 172.21.15.12:0/3686683117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c080cee10 0x7f1c08161590 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009e50 tx=0x7f1c0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7689514a884dead server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c06a00700 1 -- 172.21.15.12:0/3686683117 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000e070 con 0x7f1c080cee10 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c06a00700 1 -- 172.21.15.12:0/3686683117 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0000b910 con 0x7f1c080cee10 2024-03-29T10:06:51.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c06a00700 1 -- 172.21.15.12:0/3686683117 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000bcc0 con 0x7f1c080cee10 2024-03-29T10:06:51.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c0f000700 1 -- 172.21.15.12:0/3686683117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c080cee10 msgr2=0x7f1c08161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c0f000700 1 --2- 172.21.15.12:0/3686683117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c080cee10 0x7f1c08161590 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009e50 tx=0x7f1c0000b330 comp rx=0 tx=0).stop 2024-03-29T10:06:51.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c0f000700 1 -- 172.21.15.12:0/3686683117 shutdown_connections 2024-03-29T10:06:51.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c0f000700 1 --2- 172.21.15.12:0/3686683117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c080cee10 0x7f1c08161590 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c0f000700 1 --2- 172.21.15.12:0/3686683117 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c081611b0 0x7f1c0815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c0f000700 1 -- 172.21.15.12:0/3686683117 >> 172.21.15.12:0/3686683117 conn(0x7f1c0808aff0 msgr2=0x7f1c080d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:51.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c0f000700 1 -- 172.21.15.12:0/3686683117 shutdown_connections 2024-03-29T10:06:51.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.823+0000 7f1c0f000700 1 -- 172.21.15.12:0/3686683117 wait complete. 2024-03-29T10:06:51.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.825+0000 7f1c0f000700 1 Processor -- start 2024-03-29T10:06:51.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.825+0000 7f1c0f000700 1 -- start start 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.826+0000 7f1c0f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c080cee10 0x7f1c081f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.826+0000 7f1c0f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c0815f4d0 0x7f1c080ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.826+0000 7f1c0f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c081611b0 0x7f1c080cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.826+0000 7f1c0f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081f7520 con 0x7f1c0815f4d0 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.826+0000 7f1c0f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080cd700 con 0x7f1c081611b0 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.826+0000 7f1c0f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080b7230 con 0x7f1c080cee10 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c07400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c0815f4d0 0x7f1c080ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c07e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c080cee10 0x7f1c081f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c07400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c0815f4d0 0x7f1c080ccc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36210/0 (socket says 172.21.15.12:36210) 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c07400700 1 -- 172.21.15.12:0/777580699 learned_addr learned my addr 172.21.15.12:0/777580699 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c0d000700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c081611b0 0x7f1c080cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c07400700 1 -- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c080cee10 msgr2=0x7f1c081f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c07400700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c080cee10 0x7f1c081f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c07400700 1 -- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c081611b0 msgr2=0x7f1c080cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:51.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c07400700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c081611b0 0x7f1c080cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:51.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c07400700 1 -- 172.21.15.12:0/777580699 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c00009b30 con 0x7f1c0815f4d0 2024-03-29T10:06:51.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.827+0000 7f1c07400700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c0815f4d0 0x7f1c080ccc80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400c540 tx=0x7f1bf400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:51.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.828+0000 7f1bffe00700 1 -- 172.21.15.12:0/777580699 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4004c80 con 0x7f1c0815f4d0 2024-03-29T10:06:51.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.828+0000 7f1bffe00700 1 -- 172.21.15.12:0/777580699 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf4004de0 con 0x7f1c0815f4d0 2024-03-29T10:06:51.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.828+0000 7f1bffe00700 1 -- 172.21.15.12:0/777580699 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4007310 con 0x7f1c0815f4d0 2024-03-29T10:06:51.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.828+0000 7f1c0f000700 1 -- 172.21.15.12:0/777580699 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c080b7510 con 0x7f1c0815f4d0 2024-03-29T10:06:51.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.828+0000 7f1c0f000700 1 -- 172.21.15.12:0/777580699 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c080cb770 con 0x7f1c0815f4d0 2024-03-29T10:06:51.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.829+0000 7f1c0f000700 1 -- 172.21.15.12:0/777580699 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c080cb490 con 0x7f1c0815f4d0 2024-03-29T10:06:51.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.830+0000 7f1bffe00700 1 -- 172.21.15.12:0/777580699 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f1bf4007470 con 0x7f1c0815f4d0 2024-03-29T10:06:51.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.831+0000 7f1bffe00700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be806ae40 0x7f1be806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:51.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.831+0000 7f1bffe00700 1 -- 172.21.15.12:0/777580699 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f1bf4089800 con 0x7f1c0815f4d0 2024-03-29T10:06:51.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.831+0000 7f1c07e00700 1 -- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be806ae40 msgr2=0x7f1be806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:51.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.831+0000 7f1c07e00700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be806ae40 0x7f1be806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:51.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:51.835+0000 7f1bffe00700 1 -- 172.21.15.12:0/777580699 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1c080cb490 con 0x7f1c0815f4d0 2024-03-29T10:06:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.032+0000 7f1c07e00700 1 -- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be806ae40 msgr2=0x7f1be806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.032+0000 7f1c07e00700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be806ae40 0x7f1be806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:52.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.051+0000 7f1c0f000700 1 -- 172.21.15.12:0/777580699 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1c080cb490 con 0x7f1c0815f4d0 2024-03-29T10:06:52.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.051+0000 7f1bffe00700 1 -- 172.21.15.12:0/777580699 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v47) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1c080cb490 con 0x7f1c0815f4d0 2024-03-29T10:06:52.051 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.053+0000 7f1c0f000700 1 -- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be806ae40 msgr2=0x7f1be806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.053+0000 7f1c0f000700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be806ae40 0x7f1be806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.054+0000 7f1c0f000700 1 -- 172.21.15.12:0/777580699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c0815f4d0 msgr2=0x7f1c080ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.054+0000 7f1c0f000700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c0815f4d0 0x7f1c080ccc80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400c540 tx=0x7f1bf400c850 comp rx=0 tx=0).stop 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.054+0000 7f1c0f000700 1 -- 172.21.15.12:0/777580699 shutdown_connections 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.054+0000 7f1c0f000700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be806ae40 0x7f1be806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.054+0000 7f1c0f000700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c081611b0 0x7f1c080cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.054+0000 7f1c0f000700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c080cee10 0x7f1c081f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.054+0000 7f1c0f000700 1 --2- 172.21.15.12:0/777580699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c0815f4d0 0x7f1c080ccc80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.054+0000 7f1c0f000700 1 -- 172.21.15.12:0/777580699 >> 172.21.15.12:0/777580699 conn(0x7f1c0808aff0 msgr2=0x7f1c0806bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.054+0000 7f1c0f000700 1 -- 172.21.15.12:0/777580699 shutdown_connections 2024-03-29T10:06:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.054+0000 7f1c0f000700 1 -- 172.21.15.12:0/777580699 wait complete. 2024-03-29T10:06:52.073 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:06:52.196 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:52.197+0000 7f9625f86240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:06:52.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.279+0000 7eff88400700 1 Processor -- start 2024-03-29T10:06:52.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.279+0000 7eff88400700 1 -- start start 2024-03-29T10:06:52.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.281+0000 7eff88400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff800cd480 0x7eff8006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:52.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.281+0000 7eff88400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff800cdae0 con 0x7eff800cbb30 2024-03-29T10:06:52.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.281+0000 7eff88400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff800cddc0 con 0x7eff800cd020 2024-03-29T10:06:52.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.281+0000 7eff88400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff800ce0a0 con 0x7eff800cd480 2024-03-29T10:06:52.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.281+0000 7eff85a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7eff800cbb30 0x7eff800cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57524/0 (socket says 172.21.15.12:57524) 2024-03-29T10:06:52.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.281+0000 7eff85a00700 1 -- 172.21.15.12:0/4261497283 learned_addr learned my addr 172.21.15.12:0/4261497283 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:52.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.281+0000 7eff86400700 1 --2- 172.21.15.12:0/4261497283 >> v2:172.21.15.12:3300/0 conn(0x7eff800cd480 0x7eff8006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.281+0000 7eff7fe00700 1 -- 172.21.15.12:0/4261497283 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360924996 0 0) 0x7eff800cdae0 con 0x7eff800cbb30 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.282+0000 7eff7fe00700 1 -- 172.21.15.12:0/4261497283 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff680035b0 con 0x7eff800cbb30 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.282+0000 7eff86400700 1 -- 172.21.15.12:0/4261497283 >> v1:172.21.15.159:6789/0 conn(0x7eff800cd020 legacy=0x7eff800b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.282+0000 7eff7fe00700 1 -- 172.21.15.12:0/4261497283 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230919257 0 0) 0x7eff800cddc0 con 0x7eff800cd020 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.282+0000 7eff86400700 1 -- 172.21.15.12:0/4261497283 >> v1:172.21.15.12:6789/0 conn(0x7eff800cbb30 legacy=0x7eff800cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.282+0000 7eff86400700 1 -- 172.21.15.12:0/4261497283 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8006dc20 con 0x7eff800cd480 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.282+0000 7eff86400700 1 --2- 172.21.15.12:0/4261497283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd480 0x7eff8006e420 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff78009d90 tx=0x7eff7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9fd9f7a8c102ce server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.282+0000 7eff7fe00700 1 -- 172.21.15.12:0/4261497283 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7800e070 con 0x7eff800cd480 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.282+0000 7eff7fe00700 1 -- 172.21.15.12:0/4261497283 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7800bb20 con 0x7eff800cd480 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.282+0000 7eff7fe00700 1 -- 172.21.15.12:0/4261497283 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7800beb0 con 0x7eff800cd480 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.283+0000 7eff88400700 1 -- 172.21.15.12:0/4261497283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd480 msgr2=0x7eff8006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.283+0000 7eff88400700 1 --2- 172.21.15.12:0/4261497283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd480 0x7eff8006e420 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff78009d90 tx=0x7eff7800b330 comp rx=0 tx=0).stop 2024-03-29T10:06:52.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.283+0000 7eff88400700 1 -- 172.21.15.12:0/4261497283 shutdown_connections 2024-03-29T10:06:52.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.283+0000 7eff88400700 1 --2- 172.21.15.12:0/4261497283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd480 0x7eff8006e420 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.283+0000 7eff88400700 1 -- 172.21.15.12:0/4261497283 >> 172.21.15.12:0/4261497283 conn(0x7eff800b0d70 msgr2=0x7eff800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:52.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.283+0000 7eff88400700 1 -- 172.21.15.12:0/4261497283 shutdown_connections 2024-03-29T10:06:52.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.283+0000 7eff88400700 1 -- 172.21.15.12:0/4261497283 wait complete. 2024-03-29T10:06:52.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.285+0000 7eff88400700 1 Processor -- start 2024-03-29T10:06:52.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.285+0000 7eff88400700 1 -- start start 2024-03-29T10:06:52.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.286+0000 7eff88400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff800cbb30 0x7eff800adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:52.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.286+0000 7eff88400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd020 0x7eff800ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:52.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.286+0000 7eff88400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800cd480 0x7eff801f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:52.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.286+0000 7eff88400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff800ac590 con 0x7eff800cd020 2024-03-29T10:06:52.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.286+0000 7eff88400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff801f36e0 con 0x7eff800cbb30 2024-03-29T10:06:52.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.286+0000 7eff88400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff801f39c0 con 0x7eff800cd480 2024-03-29T10:06:52.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.286+0000 7eff85000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd020 0x7eff800ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:52.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.286+0000 7eff85000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd020 0x7eff800ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36232/0 (socket says 172.21.15.12:36232) 2024-03-29T10:06:52.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.286+0000 7eff85000700 1 -- 172.21.15.12:0/617740274 learned_addr learned my addr 172.21.15.12:0/617740274 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:52.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff85000700 1 -- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800cd480 msgr2=0x7eff801f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff85000700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800cd480 0x7eff801f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff85000700 1 -- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff800cbb30 msgr2=0x7eff800adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff85000700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff800cbb30 0x7eff800adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff85000700 1 -- 172.21.15.12:0/617740274 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff78009a70 con 0x7eff800cd020 2024-03-29T10:06:52.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff85000700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd020 0x7eff800ac050 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00c550 tx=0x7eff6c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:52.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff7d600700 1 -- 172.21.15.12:0/617740274 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0054e0 con 0x7eff800cd020 2024-03-29T10:06:52.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff7d600700 1 -- 172.21.15.12:0/617740274 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c003680 con 0x7eff800cd020 2024-03-29T10:06:52.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff88400700 1 -- 172.21.15.12:0/617740274 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff801f3ca0 con 0x7eff800cd020 2024-03-29T10:06:52.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff7d600700 1 -- 172.21.15.12:0/617740274 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c003a30 con 0x7eff800cd020 2024-03-29T10:06:52.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.287+0000 7eff88400700 1 -- 172.21.15.12:0/617740274 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff801f33f0 con 0x7eff800cd020 2024-03-29T10:06:52.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.289+0000 7eff88400700 1 -- 172.21.15.12:0/617740274 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff800ca190 con 0x7eff800cd020 2024-03-29T10:06:52.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.289+0000 7eff7d600700 1 -- 172.21.15.12:0/617740274 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7eff6c003b90 con 0x7eff800cd020 2024-03-29T10:06:52.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.290+0000 7eff7d600700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae40 0x7eff6006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:52.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.290+0000 7eff7d600700 1 -- 172.21.15.12:0/617740274 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7eff6c0890a0 con 0x7eff800cd020 2024-03-29T10:06:52.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.290+0000 7eff85a00700 1 -- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae40 msgr2=0x7eff6006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:52.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.290+0000 7eff85a00700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae40 0x7eff6006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:52.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.294+0000 7eff7d600700 1 -- 172.21.15.12:0/617740274 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff800ca190 con 0x7eff800cd020 2024-03-29T10:06:52.404 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:52.405+0000 7f9625f86240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:06:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.491+0000 7eff85a00700 1 -- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae40 msgr2=0x7eff6006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.491+0000 7eff85a00700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae40 0x7eff6006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.510+0000 7eff88400700 1 -- 172.21.15.12:0/617740274 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7eff800ca190 con 0x7eff800cd020 2024-03-29T10:06:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.511+0000 7eff7d600700 1 -- 172.21.15.12:0/617740274 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v47) v1 ==== 100+0+11 (secure 0 0 0) 0x7eff6c017020 con 0x7eff800cd020 2024-03-29T10:06:52.511 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:06:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.513+0000 7eff88400700 1 -- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae40 msgr2=0x7eff6006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.513+0000 7eff88400700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae40 0x7eff6006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.514+0000 7eff88400700 1 -- 172.21.15.12:0/617740274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd020 msgr2=0x7eff800ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.514+0000 7eff88400700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd020 0x7eff800ac050 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00c550 tx=0x7eff6c00c910 comp rx=0 tx=0).stop 2024-03-29T10:06:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.514+0000 7eff88400700 1 -- 172.21.15.12:0/617740274 shutdown_connections 2024-03-29T10:06:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.514+0000 7eff88400700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae40 0x7eff6006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.514+0000 7eff88400700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800cd480 0x7eff801f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.514+0000 7eff88400700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff800cbb30 0x7eff800adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.514+0000 7eff88400700 1 --2- 172.21.15.12:0/617740274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff800cd020 0x7eff800ac050 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.514+0000 7eff88400700 1 -- 172.21.15.12:0/617740274 >> 172.21.15.12:0/617740274 conn(0x7eff800b0d70 msgr2=0x7eff800ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.514+0000 7eff88400700 1 -- 172.21.15.12:0/617740274 shutdown_connections 2024-03-29T10:06:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.514+0000 7eff88400700 1 -- 172.21.15.12:0/617740274 wait complete. 2024-03-29T10:06:52.533 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:06:52.604 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:52.605+0000 7f9625f86240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:06:52.607 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:52.608+0000 7ff4587aa240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:06:52.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.733+0000 7f6f3dc00700 1 Processor -- start 2024-03-29T10:06:52.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.734+0000 7f6f3dc00700 1 -- start start 2024-03-29T10:06:52.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.735+0000 7f6f3dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f380d18b0 0x7f6f380cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:52.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.735+0000 7f6f3dc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380ceda0 con 0x7f6f38161590 2024-03-29T10:06:52.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.735+0000 7f6f3dc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f381612f0 con 0x7f6f38160f10 2024-03-29T10:06:52.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.735+0000 7f6f3dc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380d1d30 con 0x7f6f380d18b0 2024-03-29T10:06:52.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.735+0000 7f6f37400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6f38161590 0x7f6f380b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57526/0 (socket says 172.21.15.12:57526) 2024-03-29T10:06:52.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.735+0000 7f6f37400700 1 -- 172.21.15.12:0/2878672238 learned_addr learned my addr 172.21.15.12:0/2878672238 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:52.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.735+0000 7f6f36000700 1 --2- 172.21.15.12:0/2878672238 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f380d18b0 0x7f6f380cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:52.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.736+0000 7f6f35600700 1 -- 172.21.15.12:0/2878672238 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383805498 0 0) 0x7f6f380ceda0 con 0x7f6f38161590 2024-03-29T10:06:52.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.736+0000 7f6f36000700 1 -- 172.21.15.12:0/2878672238 >> v1:172.21.15.159:6789/0 conn(0x7f6f38160f10 legacy=0x7f6f380d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.736+0000 7f6f36000700 1 -- 172.21.15.12:0/2878672238 >> v1:172.21.15.12:6789/0 conn(0x7f6f38161590 legacy=0x7f6f380b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.736+0000 7f6f36000700 1 -- 172.21.15.12:0/2878672238 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380ce470 con 0x7f6f380d18b0 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.736+0000 7f6f36000700 1 --2- 172.21.15.12:0/2878672238 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f380d18b0 0x7f6f380cf230 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009820 tx=0x7f6f24009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4794a19dbfaba00c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.736+0000 7f6f35600700 1 -- 172.21.15.12:0/2878672238 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f24004230 con 0x7f6f380d18b0 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.736+0000 7f6f35600700 1 -- 172.21.15.12:0/2878672238 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f24004390 con 0x7f6f380d18b0 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.736+0000 7f6f35600700 1 -- 172.21.15.12:0/2878672238 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f240036a0 con 0x7f6f380d18b0 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.737+0000 7f6f3dc00700 1 -- 172.21.15.12:0/2878672238 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f380d18b0 msgr2=0x7f6f380cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.737+0000 7f6f3dc00700 1 --2- 172.21.15.12:0/2878672238 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f380d18b0 0x7f6f380cf230 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009820 tx=0x7f6f24009b30 comp rx=0 tx=0).stop 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.737+0000 7f6f3dc00700 1 -- 172.21.15.12:0/2878672238 shutdown_connections 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.737+0000 7f6f3dc00700 1 --2- 172.21.15.12:0/2878672238 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f380d18b0 0x7f6f380cf230 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.737+0000 7f6f3dc00700 1 -- 172.21.15.12:0/2878672238 >> 172.21.15.12:0/2878672238 conn(0x7f6f3808aff0 msgr2=0x7f6f380d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.737+0000 7f6f3dc00700 1 -- 172.21.15.12:0/2878672238 shutdown_connections 2024-03-29T10:06:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.737+0000 7f6f3dc00700 1 -- 172.21.15.12:0/2878672238 wait complete. 2024-03-29T10:06:52.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.740+0000 7f6f3dc00700 1 Processor -- start 2024-03-29T10:06:52.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.740+0000 7f6f3dc00700 1 -- start start 2024-03-29T10:06:52.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.742+0000 7f6f3dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f380d18b0 0x7f6f38152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:52.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.742+0000 7f6f3dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f38160f10 0x7f6f381559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:52.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.742+0000 7f6f3dc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f38161590 0x7f6f38155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:52.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.742+0000 7f6f3dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f38154430 con 0x7f6f380d18b0 2024-03-29T10:06:52.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.742+0000 7f6f3dc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f38154710 con 0x7f6f38161590 2024-03-29T10:06:52.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.742+0000 7f6f3dc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f381549f0 con 0x7f6f38160f10 2024-03-29T10:06:52.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.742+0000 7f6f36a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f380d18b0 0x7f6f38152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.742+0000 7f6f36000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f38160f10 0x7f6f381559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.742+0000 7f6f36a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f380d18b0 0x7f6f38152a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36236/0 (socket says 172.21.15.12:36236) 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.742+0000 7f6f36a00700 1 -- 172.21.15.12:0/664459068 learned_addr learned my addr 172.21.15.12:0/664459068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.743+0000 7f6f37400700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f38161590 0x7f6f38155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.743+0000 7f6f36000700 1 -- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f38161590 msgr2=0x7f6f38155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.743+0000 7f6f36000700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f38161590 0x7f6f38155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.743+0000 7f6f36000700 1 -- 172.21.15.12:0/664459068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f380d18b0 msgr2=0x7f6f38152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.743+0000 7f6f36000700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f380d18b0 0x7f6f38152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.743+0000 7f6f36000700 1 -- 172.21.15.12:0/664459068 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f240094a0 con 0x7f6f38160f10 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.743+0000 7f6f36000700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f38160f10 0x7f6f381559b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6f24015000 tx=0x7f6f240047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.743+0000 7f6f2ea00700 1 -- 172.21.15.12:0/664459068 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f24011630 con 0x7f6f38160f10 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.744+0000 7f6f2ea00700 1 -- 172.21.15.12:0/664459068 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f240117b0 con 0x7f6f38160f10 2024-03-29T10:06:52.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.744+0000 7f6f2ea00700 1 -- 172.21.15.12:0/664459068 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f24011b60 con 0x7f6f38160f10 2024-03-29T10:06:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.744+0000 7f6f3dc00700 1 -- 172.21.15.12:0/664459068 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f38154cd0 con 0x7f6f38160f10 2024-03-29T10:06:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.744+0000 7f6f3dc00700 1 -- 172.21.15.12:0/664459068 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f38154100 con 0x7f6f38160f10 2024-03-29T10:06:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.745+0000 7f6f3dc00700 1 -- 172.21.15.12:0/664459068 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f38067720 con 0x7f6f38160f10 2024-03-29T10:06:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.746+0000 7f6f2ea00700 1 -- 172.21.15.12:0/664459068 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f6f24011cc0 con 0x7f6f38160f10 2024-03-29T10:06:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.747+0000 7f6f2ea00700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f1806ae40 0x7f6f1806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.747+0000 7f6f2ea00700 1 -- 172.21.15.12:0/664459068 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f6f2408b1e0 con 0x7f6f38160f10 2024-03-29T10:06:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.747+0000 7f6f36a00700 1 -- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f1806ae40 msgr2=0x7f6f1806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.747+0000 7f6f36a00700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f1806ae40 0x7f6f1806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:52.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.751+0000 7f6f2ea00700 1 -- 172.21.15.12:0/664459068 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f24055da0 con 0x7f6f38160f10 2024-03-29T10:06:52.838 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:52.838+0000 7ff4587aa240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:06:52.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.948+0000 7f6f36a00700 1 -- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f1806ae40 msgr2=0x7f6f1806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:52.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:52.948+0000 7f6f36a00700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f1806ae40 0x7f6f1806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:53.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.013+0000 7f6f3dc00700 1 -- 172.21.15.12:0/664459068 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f38154b30 con 0x7f6f38160f10 2024-03-29T10:06:53.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.013+0000 7f6f2ea00700 1 -- 172.21.15.12:0/664459068 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f6f38154b30 con 0x7f6f38160f10 2024-03-29T10:06:53.013 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:53.013 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2845880355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2845880355},{"type":"v1","addr":"172.21.15.159:6837","nonce":2845880355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2836227468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2836227468},{"type":"v1","addr":"172.21.15.159:6836","nonce":2836227468}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1863550973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1863550973},{"type":"v1","addr":"172.21.15.12:6834","nonce":1863550973}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:55:35.438233+0000","modified":"2024-03-29T09:55:38.926199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/2013939805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2013939805},{"type":"v1","addr":"172.21.15.12:6835","nonce":2013939805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-29T10:06:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 -- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f1806ae40 msgr2=0x7f6f1806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f1806ae40 0x7f6f1806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 -- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f38160f10 msgr2=0x7f6f381559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f38160f10 0x7f6f381559b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6f24015000 tx=0x7f6f240047c0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 -- 172.21.15.12:0/664459068 shutdown_connections 2024-03-29T10:06:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f1806ae40 0x7f6f1806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f38161590 0x7f6f38155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f380d18b0 0x7f6f38152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 --2- 172.21.15.12:0/664459068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f38160f10 0x7f6f381559b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 -- 172.21.15.12:0/664459068 >> 172.21.15.12:0/664459068 conn(0x7f6f3808aff0 msgr2=0x7f6f38160230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:53.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 -- 172.21.15.12:0/664459068 shutdown_connections 2024-03-29T10:06:53.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.016+0000 7f6f3dc00700 1 -- 172.21.15.12:0/664459068 wait complete. 2024-03-29T10:06:53.017 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-29T10:06:53.035 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:06:53.060 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:53.061+0000 7ff4587aa240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:06:53.074 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.075+0000 7f9625f86240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:06:53.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.241+0000 7f8c46000700 1 Processor -- start 2024-03-29T10:06:53.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.241+0000 7f8c46000700 1 -- start start 2024-03-29T10:06:53.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.243+0000 7f8c46000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400d1ed0 0x7f8c400cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:53.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.243+0000 7f8c46000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c400d2330 con 0x7f8c400cf9a0 2024-03-29T10:06:53.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.243+0000 7f8c46000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c400cce30 con 0x7f8c4015b660 2024-03-29T10:06:53.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.243+0000 7f8c46000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c400cfe30 con 0x7f8c400d1ed0 2024-03-29T10:06:53.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.243+0000 7f8c3fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8c400cf9a0 0x7f8c4009a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57534/0 (socket says 172.21.15.12:57534) 2024-03-29T10:06:53.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.243+0000 7f8c3fe00700 1 -- 172.21.15.12:0/1789340923 learned_addr learned my addr 172.21.15.12:0/1789340923 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:53.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.243+0000 7f8c3ea00700 1 --2- 172.21.15.12:0/1789340923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400d1ed0 0x7f8c400cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:53.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.243+0000 7f8c3e000700 1 -- 172.21.15.12:0/1789340923 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069419903 0 0) 0x7f8c400d2330 con 0x7f8c400cf9a0 2024-03-29T10:06:53.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.244+0000 7f8c3e000700 1 -- 172.21.15.12:0/1789340923 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c240035b0 con 0x7f8c400cf9a0 2024-03-29T10:06:53.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.244+0000 7f8c3e000700 1 -- 172.21.15.12:0/1789340923 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740763888 0 0) 0x7f8c400cce30 con 0x7f8c4015b660 2024-03-29T10:06:53.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.244+0000 7f8c3e000700 1 -- 172.21.15.12:0/1789340923 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c400d2330 con 0x7f8c4015b660 2024-03-29T10:06:53.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.244+0000 7f8c3ea00700 1 -- 172.21.15.12:0/1789340923 >> v1:172.21.15.159:6789/0 conn(0x7f8c4015b660 legacy=0x7f8c400c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.244+0000 7f8c3ea00700 1 -- 172.21.15.12:0/1789340923 >> v1:172.21.15.12:6789/0 conn(0x7f8c400cf9a0 legacy=0x7f8c4009a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.244+0000 7f8c3ea00700 1 -- 172.21.15.12:0/1789340923 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c400cd110 con 0x7f8c400d1ed0 2024-03-29T10:06:53.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.244+0000 7f8c3ea00700 1 --2- 172.21.15.12:0/1789340923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400d1ed0 0x7f8c400cf460 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c009870 tx=0x7f8c2c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b653bb6d8d39e4e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:53.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.244+0000 7f8c3e000700 1 -- 172.21.15.12:0/1789340923 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c011830 con 0x7f8c400d1ed0 2024-03-29T10:06:53.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.244+0000 7f8c3e000700 1 -- 172.21.15.12:0/1789340923 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2c011990 con 0x7f8c400d1ed0 2024-03-29T10:06:53.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.244+0000 7f8c3e000700 1 -- 172.21.15.12:0/1789340923 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c011d20 con 0x7f8c400d1ed0 2024-03-29T10:06:53.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.245+0000 7f8c46000700 1 -- 172.21.15.12:0/1789340923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400d1ed0 msgr2=0x7f8c400cf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.245+0000 7f8c46000700 1 --2- 172.21.15.12:0/1789340923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400d1ed0 0x7f8c400cf460 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c009870 tx=0x7f8c2c009b80 comp rx=0 tx=0).stop 2024-03-29T10:06:53.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.245+0000 7f8c46000700 1 -- 172.21.15.12:0/1789340923 shutdown_connections 2024-03-29T10:06:53.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.245+0000 7f8c46000700 1 --2- 172.21.15.12:0/1789340923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400d1ed0 0x7f8c400cf460 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.245+0000 7f8c46000700 1 -- 172.21.15.12:0/1789340923 >> 172.21.15.12:0/1789340923 conn(0x7f8c400d13e0 msgr2=0x7f8c400d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:53.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.245+0000 7f8c46000700 1 -- 172.21.15.12:0/1789340923 shutdown_connections 2024-03-29T10:06:53.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.245+0000 7f8c46000700 1 -- 172.21.15.12:0/1789340923 wait complete. 2024-03-29T10:06:53.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.247+0000 7f8c46000700 1 Processor -- start 2024-03-29T10:06:53.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.247+0000 7f8c46000700 1 -- start start 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.248+0000 7f8c46000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400cf9a0 0x7f8c4015f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.248+0000 7f8c46000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c400d1ed0 0x7f8c400ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.248+0000 7f8c46000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c4015b660 0x7f8c400cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.248+0000 7f8c46000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c40070930 con 0x7f8c400d1ed0 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.248+0000 7f8c46000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4015bd00 con 0x7f8c4015b660 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.248+0000 7f8c46000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c400b6550 con 0x7f8c400cf9a0 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.248+0000 7f8c3ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c400d1ed0 0x7f8c400ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.248+0000 7f8c3ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c400d1ed0 0x7f8c400ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36242/0 (socket says 172.21.15.12:36242) 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.248+0000 7f8c3ea00700 1 -- 172.21.15.12:0/2726695490 learned_addr learned my addr 172.21.15.12:0/2726695490 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.249+0000 7f8c3f400700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400cf9a0 0x7f8c4015f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.249+0000 7f8c3fe00700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c4015b660 0x7f8c400cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.249+0000 7f8c3ea00700 1 -- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400cf9a0 msgr2=0x7f8c4015f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.249+0000 7f8c3ea00700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400cf9a0 0x7f8c4015f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.249+0000 7f8c3ea00700 1 -- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c4015b660 msgr2=0x7f8c400cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.249+0000 7f8c3ea00700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c4015b660 0x7f8c400cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.249+0000 7f8c3ea00700 1 -- 172.21.15.12:0/2726695490 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2c0094a0 con 0x7f8c400d1ed0 2024-03-29T10:06:53.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.249+0000 7f8c3ea00700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c400d1ed0 0x7f8c400ca680 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c0090b0 tx=0x7f8c2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:53.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.250+0000 7f8c37400700 1 -- 172.21.15.12:0/2726695490 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c01a760 con 0x7f8c400d1ed0 2024-03-29T10:06:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.250+0000 7f8c37400700 1 -- 172.21.15.12:0/2726695490 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2c01a8c0 con 0x7f8c400d1ed0 2024-03-29T10:06:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.250+0000 7f8c37400700 1 -- 172.21.15.12:0/2726695490 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c01ac70 con 0x7f8c400d1ed0 2024-03-29T10:06:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.250+0000 7f8c46000700 1 -- 172.21.15.12:0/2726695490 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c400b6830 con 0x7f8c400d1ed0 2024-03-29T10:06:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.250+0000 7f8c46000700 1 -- 172.21.15.12:0/2726695490 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c400b6df0 con 0x7f8c400d1ed0 2024-03-29T10:06:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.251+0000 7f8c46000700 1 -- 172.21.15.12:0/2726695490 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4006e470 con 0x7f8c400d1ed0 2024-03-29T10:06:53.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.253+0000 7f8c37400700 1 -- 172.21.15.12:0/2726695490 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f8c2c022070 con 0x7f8c400d1ed0 2024-03-29T10:06:53.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.253+0000 7f8c37400700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c2006ae40 0x7f8c2006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:53.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.253+0000 7f8c37400700 1 -- 172.21.15.12:0/2726695490 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f8c2c08a710 con 0x7f8c400d1ed0 2024-03-29T10:06:53.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.254+0000 7f8c3f400700 1 -- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c2006ae40 msgr2=0x7f8c2006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:53.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.254+0000 7f8c3f400700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c2006ae40 0x7f8c2006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.262+0000 7f8c37400700 1 -- 172.21.15.12:0/2726695490 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8c2c0551d0 con 0x7f8c400d1ed0 2024-03-29T10:06:53.278 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.279+0000 7f9625f86240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:06:53.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.455+0000 7f8c3f400700 1 -- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c2006ae40 msgr2=0x7f8c2006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:53.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.455+0000 7f8c3f400700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c2006ae40 0x7f8c2006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:06:53.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.481+0000 7f8c46000700 1 -- 172.21.15.12:0/2726695490 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c4015be70 con 0x7f8c400d1ed0 2024-03-29T10:06:53.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.482+0000 7f8c37400700 1 -- 172.21.15.12:0/2726695490 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+17906 (secure 0 0 0) 0x7f8c2c0587f0 con 0x7f8c400d1ed0 2024-03-29T10:06:53.482 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:53.482 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":47,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:06:47.985612+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:55:33.508935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T09:55:34.418367+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":219,"snap_epoch":47,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6832/43444":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/3395907487":"2024-03-30T09:55:52.597859+0000","172.21.15.159:6833/43444":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/249193272":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/795330822":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/1756693282":"2024-03-30T09:55:52.597859+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":6,"length":1},{"begin":9,"length":1},{"begin":12,"length":1},{"begin":16,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":27,"length":1},{"begin":32,"length":1},{"begin":34,"length":1},{"begin":38,"length":1},{"begin":43,"length":1},{"begin":45,"length":1},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":55,"length":1},{"begin":59,"length":1},{"begin":64,"length":1},{"begin":66,"length":1},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":76,"length":1},{"begin":80,"length":2},{"begin":85,"length":1},{"begin":87,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":109,"length":1},{"begin":187,"length":33}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":109,"length":1},{"begin":219,"length":1}]}],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":4,"length":1},{"begin":7,"length":1},{"begin":10,"length":1},{"begin":14,"length":2},{"begin":19,"length":1},{"begin":25,"length":2},{"begin":30,"length":2},{"begin":36,"length":2},{"begin":41,"length":2},{"begin":47,"length":2},{"begin":51,"length":2},{"begin":57,"length":2},{"begin":62,"length":2},{"begin":68,"length":2},{"begin":72,"length":2},{"begin":78,"length":2},{"begin":83,"length":2},{"begin":89,"length":3},{"begin":94,"length":2},{"begin":100,"length":3},{"begin":105,"length":2},{"begin":147,"length":40}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:06:53.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.486+0000 7f8c46000700 1 -- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c2006ae40 msgr2=0x7f8c2006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:53.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.486+0000 7f8c46000700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c2006ae40 0x7f8c2006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.486+0000 7f8c46000700 1 -- 172.21.15.12:0/2726695490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c400d1ed0 msgr2=0x7f8c400ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.486+0000 7f8c46000700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c400d1ed0 0x7f8c400ca680 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c0090b0 tx=0x7f8c2c004750 comp rx=0 tx=0).stop 2024-03-29T10:06:53.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.486+0000 7f8c46000700 1 -- 172.21.15.12:0/2726695490 shutdown_connections 2024-03-29T10:06:53.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.486+0000 7f8c46000700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c2006ae40 0x7f8c2006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.486+0000 7f8c46000700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c4015b660 0x7f8c400cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.486+0000 7f8c46000700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c400cf9a0 0x7f8c4015f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.486+0000 7f8c46000700 1 --2- 172.21.15.12:0/2726695490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c400d1ed0 0x7f8c400ca680 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.486+0000 7f8c46000700 1 -- 172.21.15.12:0/2726695490 >> 172.21.15.12:0/2726695490 conn(0x7f8c400d13e0 msgr2=0x7f8c400d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:53.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.487+0000 7f8c46000700 1 -- 172.21.15.12:0/2726695490 shutdown_connections 2024-03-29T10:06:53.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.487+0000 7f8c46000700 1 -- 172.21.15.12:0/2726695490 wait complete. 2024-03-29T10:06:53.515 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T10:06:53.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:06:53.591 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:53.591+0000 7ff4587aa240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:06:53.718 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.719+0000 7f9625f86240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:06:53.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.755+0000 7efc1d400700 1 Processor -- start 2024-03-29T10:06:53.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.755+0000 7efc1d400700 1 -- start start 2024-03-29T10:06:53.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.756+0000 7efc1d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc180cebd0 0x7efc180cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:53.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.756+0000 7efc1d400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1806f850 con 0x7efc1806a790 2024-03-29T10:06:53.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.756+0000 7efc1d400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1806e3f0 con 0x7efc180afbe0 2024-03-29T10:06:53.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.756+0000 7efc1d400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1806fad0 con 0x7efc180cebd0 2024-03-29T10:06:53.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.757+0000 7efc16800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7efc1806a790 0x7efc1806ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57548/0 (socket says 172.21.15.12:57548) 2024-03-29T10:06:53.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.757+0000 7efc16800700 1 -- 172.21.15.12:0/3669491991 learned_addr learned my addr 172.21.15.12:0/3669491991 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:53.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.757+0000 7efc17200700 1 --2- 172.21.15.12:0/3669491991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc180cebd0 0x7efc180cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.757+0000 7efc15400700 1 -- 172.21.15.12:0/3669491991 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092547084 0 0) 0x7efc1806f850 con 0x7efc1806a790 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.757+0000 7efc15400700 1 -- 172.21.15.12:0/3669491991 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbfc0035b0 con 0x7efc1806a790 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.757+0000 7efc15400700 1 -- 172.21.15.12:0/3669491991 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51634538 0 0) 0x7efc1806e3f0 con 0x7efc180afbe0 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.757+0000 7efc15400700 1 -- 172.21.15.12:0/3669491991 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1806f850 con 0x7efc180afbe0 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.757+0000 7efc17200700 1 -- 172.21.15.12:0/3669491991 >> v1:172.21.15.159:6789/0 conn(0x7efc180afbe0 legacy=0x7efc1806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.757+0000 7efc17200700 1 -- 172.21.15.12:0/3669491991 >> v1:172.21.15.12:6789/0 conn(0x7efc1806a790 legacy=0x7efc1806ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.757+0000 7efc17200700 1 -- 172.21.15.12:0/3669491991 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc180af970 con 0x7efc180cebd0 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.758+0000 7efc17200700 1 --2- 172.21.15.12:0/3669491991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc180cebd0 0x7efc180cf040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efc10009dc0 tx=0x7efc1000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5aa1ce30ff850be server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.758+0000 7efc15400700 1 -- 172.21.15.12:0/3669491991 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1000e070 con 0x7efc180cebd0 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.758+0000 7efc15400700 1 -- 172.21.15.12:0/3669491991 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1000bb20 con 0x7efc180cebd0 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.758+0000 7efc15400700 1 -- 172.21.15.12:0/3669491991 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1000be90 con 0x7efc180cebd0 2024-03-29T10:06:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.758+0000 7efc1d400700 1 -- 172.21.15.12:0/3669491991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc180cebd0 msgr2=0x7efc180cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.759+0000 7efc1d400700 1 --2- 172.21.15.12:0/3669491991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc180cebd0 0x7efc180cf040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efc10009dc0 tx=0x7efc1000b330 comp rx=0 tx=0).stop 2024-03-29T10:06:53.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.759+0000 7efc1d400700 1 -- 172.21.15.12:0/3669491991 shutdown_connections 2024-03-29T10:06:53.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.759+0000 7efc1d400700 1 --2- 172.21.15.12:0/3669491991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc180cebd0 0x7efc180cf040 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.759+0000 7efc1d400700 1 -- 172.21.15.12:0/3669491991 >> 172.21.15.12:0/3669491991 conn(0x7efc180b5ab0 msgr2=0x7efc180b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:53.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.759+0000 7efc1d400700 1 -- 172.21.15.12:0/3669491991 shutdown_connections 2024-03-29T10:06:53.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.759+0000 7efc1d400700 1 -- 172.21.15.12:0/3669491991 wait complete. 2024-03-29T10:06:53.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.761+0000 7efc1d400700 1 Processor -- start 2024-03-29T10:06:53.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.761+0000 7efc1d400700 1 -- start start 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.762+0000 7efc1d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc1806a790 0x7efc181f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.762+0000 7efc1d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc180afbe0 0x7efc18158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.762+0000 7efc1d400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efc180cebd0 0x7efc18155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.762+0000 7efc1d400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc181588b0 con 0x7efc180afbe0 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.762+0000 7efc1d400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc18156450 con 0x7efc180cebd0 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.762+0000 7efc1d400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc18158ac0 con 0x7efc1806a790 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc15e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc180afbe0 0x7efc18158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc15e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc180afbe0 0x7efc18158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36248/0 (socket says 172.21.15.12:36248) 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc15e00700 1 -- 172.21.15.12:0/328240323 learned_addr learned my addr 172.21.15.12:0/328240323 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc16800700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc1806a790 0x7efc181f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc17200700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efc180cebd0 0x7efc18155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc15e00700 1 -- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc1806a790 msgr2=0x7efc181f2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc15e00700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc1806a790 0x7efc181f2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc15e00700 1 -- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efc180cebd0 msgr2=0x7efc18155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:53.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc15e00700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efc180cebd0 0x7efc18155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc15e00700 1 -- 172.21.15.12:0/328240323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc10009a70 con 0x7efc180afbe0 2024-03-29T10:06:53.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.763+0000 7efc15e00700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc180afbe0 0x7efc18158370 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efc0400c520 tx=0x7efc0400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:53.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.764+0000 7efc0ea00700 1 -- 172.21.15.12:0/328240323 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc040052d0 con 0x7efc180afbe0 2024-03-29T10:06:53.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.764+0000 7efc0ea00700 1 -- 172.21.15.12:0/328240323 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc04005430 con 0x7efc180afbe0 2024-03-29T10:06:53.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.764+0000 7efc1d400700 1 -- 172.21.15.12:0/328240323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc181523e0 con 0x7efc180afbe0 2024-03-29T10:06:53.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.764+0000 7efc0ea00700 1 -- 172.21.15.12:0/328240323 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc040038c0 con 0x7efc180afbe0 2024-03-29T10:06:53.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.764+0000 7efc1d400700 1 -- 172.21.15.12:0/328240323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc181529a0 con 0x7efc180afbe0 2024-03-29T10:06:53.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.765+0000 7efc1d400700 1 -- 172.21.15.12:0/328240323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1815d0a0 con 0x7efc180afbe0 2024-03-29T10:06:53.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.767+0000 7efc0ea00700 1 -- 172.21.15.12:0/328240323 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7efc04003a20 con 0x7efc180afbe0 2024-03-29T10:06:53.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.768+0000 7efc0ea00700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efbf806ae40 0x7efbf806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:53.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.768+0000 7efc0ea00700 1 -- 172.21.15.12:0/328240323 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6899+0+0 (secure 0 0 0) 0x7efc04088f20 con 0x7efc180afbe0 2024-03-29T10:06:53.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.768+0000 7efc16800700 1 -- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efbf806ae40 msgr2=0x7efbf806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:06:53.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.768+0000 7efc16800700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efbf806ae40 0x7efbf806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:06:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.773+0000 7efc0ea00700 1 -- 172.21.15.12:0/328240323 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efc040539c0 con 0x7efc180afbe0 2024-03-29T10:06:53.812 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:53.813+0000 7ff4587aa240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:06:53.827 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.829+0000 7f9625f86240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:06:53.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.843+0000 7efc0ea00700 1 -- 172.21.15.12:0/328240323 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 15) v1 ==== 88848+0+0 (secure 0 0 0) 0x7efc0408c030 con 0x7efc180afbe0 2024-03-29T10:06:53.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.843+0000 7efc0ea00700 1 -- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efbf806ae40 msgr2=0x7efbf806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:06:53.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:53.843+0000 7efc0ea00700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efbf806ae40 0x7efbf806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:53.909 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.910+0000 7f9594a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:06:53.910 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.910+0000 7f9594a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:06:53.910 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.910+0000 7f9594a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:06:53.910 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.910+0000 7f9594a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:06:53.910 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.910+0000 7f9594a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:06:53.910 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.911+0000 7f9599000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:06:53.910 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.911+0000 7f9599000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:06:53.910 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.911+0000 7f9599000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:06:53.911 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.911+0000 7f9599000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:06:53.911 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:06:53.911+0000 7f9599000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:06:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.042+0000 7efc1d400700 1 -- 172.21.15.12:0/328240323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc18067720 con 0x7efc180afbe0 2024-03-29T10:06:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.042+0000 7efc0ea00700 1 -- 172.21.15.12:0/328240323 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4740 (secure 0 0 0) 0x7efc04071ce0 con 0x7efc180afbe0 2024-03-29T10:06:54.042 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:54.042 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2845880355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2845880355},{"type":"v1","addr":"172.21.15.159:6837","nonce":2845880355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2836227468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2836227468},{"type":"v1","addr":"172.21.15.159:6836","nonce":2836227468}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1863550973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1863550973},{"type":"v1","addr":"172.21.15.12:6834","nonce":1863550973}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:55:35.438233+0000","modified":"2024-03-29T09:55:38.926199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/2013939805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2013939805},{"type":"v1","addr":"172.21.15.12:6835","nonce":2013939805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-29T10:06:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.045+0000 7efc1d400700 1 -- 172.21.15.12:0/328240323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc180afbe0 msgr2=0x7efc18158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.045+0000 7efc1d400700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc180afbe0 0x7efc18158370 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efc0400c520 tx=0x7efc0400c830 comp rx=0 tx=0).stop 2024-03-29T10:06:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.045+0000 7efc1d400700 1 -- 172.21.15.12:0/328240323 shutdown_connections 2024-03-29T10:06:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.045+0000 7efc1d400700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efbf806ae40 0x7efbf806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.045+0000 7efc1d400700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efc180cebd0 0x7efc18155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.045+0000 7efc1d400700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc1806a790 0x7efc181f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.045+0000 7efc1d400700 1 --2- 172.21.15.12:0/328240323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc180afbe0 0x7efc18158370 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.045+0000 7efc1d400700 1 -- 172.21.15.12:0/328240323 >> 172.21.15.12:0/328240323 conn(0x7efc180b5ab0 msgr2=0x7efc180736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.045+0000 7efc1d400700 1 -- 172.21.15.12:0/328240323 shutdown_connections 2024-03-29T10:06:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.045+0000 7efc1d400700 1 -- 172.21.15.12:0/328240323 wait complete. 2024-03-29T10:06:54.046 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-29T10:06:54.064 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:06:54.280 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:54.280+0000 7ff4587aa240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:06:54.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.306+0000 7f70aea00700 1 Processor -- start 2024-03-29T10:06:54.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.306+0000 7f70aea00700 1 -- start start 2024-03-29T10:06:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.308+0000 7f70aea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80d05e0 0x7f70a80d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.308+0000 7f70aea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f70a80b6990 0x7f70a80b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.308+0000 7f70aea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a80d0f80 con 0x7f70a815da50 2024-03-29T10:06:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.308+0000 7f70aea00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a80b95d0 con 0x7f70a80b6990 2024-03-29T10:06:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.308+0000 7f70aea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a815de30 con 0x7f70a80d05e0 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.308+0000 7f70a7e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f70a815da50 0x7f70a80ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57564/0 (socket says 172.21.15.12:57564) 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.308+0000 7f70a7e00700 1 -- 172.21.15.12:0/1247475003 learned_addr learned my addr 172.21.15.12:0/1247475003 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.308+0000 7f70aca00700 1 --2- 172.21.15.12:0/1247475003 >> v2:172.21.15.12:3300/0 conn(0x7f70a80b6990 0x7f70a80b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.308+0000 7f70a7400700 1 --2- 172.21.15.12:0/1247475003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80d05e0 0x7f70a80d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.309+0000 7f70a6a00700 1 -- 172.21.15.12:0/1247475003 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862613409 0 0) 0x7f70a80d0f80 con 0x7f70a815da50 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.309+0000 7f70a6a00700 1 -- 172.21.15.12:0/1247475003 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f708c0035b0 con 0x7f70a815da50 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.309+0000 7f70a7400700 1 -- 172.21.15.12:0/1247475003 >> v2:172.21.15.12:3300/0 conn(0x7f70a80b6990 msgr2=0x7f70a80b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.309+0000 7f70a7400700 1 --2- 172.21.15.12:0/1247475003 >> v2:172.21.15.12:3300/0 conn(0x7f70a80b6990 0x7f70a80b6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.309+0000 7f70a7400700 1 -- 172.21.15.12:0/1247475003 >> v1:172.21.15.12:6789/0 conn(0x7f70a815da50 legacy=0x7f70a80ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.309+0000 7f70a7400700 1 -- 172.21.15.12:0/1247475003 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a80d24e0 con 0x7f70a80d05e0 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.309+0000 7f70aca00700 1 --2- 172.21.15.12:0/1247475003 >> v2:172.21.15.12:3300/0 conn(0x7f70a80b6990 0x7f70a80b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.309+0000 7f70a7400700 1 --2- 172.21.15.12:0/1247475003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80d05e0 0x7f70a80d0a40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7094009840 tx=0x7f7094009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80cab738f65d2f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.309+0000 7f70a6a00700 1 -- 172.21.15.12:0/1247475003 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7094011830 con 0x7f70a80d05e0 2024-03-29T10:06:54.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.309+0000 7f70a6a00700 1 -- 172.21.15.12:0/1247475003 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7094011990 con 0x7f70a80d05e0 2024-03-29T10:06:54.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.310+0000 7f70a6a00700 1 -- 172.21.15.12:0/1247475003 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7094011d20 con 0x7f70a80d05e0 2024-03-29T10:06:54.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.310+0000 7f70aea00700 1 -- 172.21.15.12:0/1247475003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80d05e0 msgr2=0x7f70a80d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:54.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.310+0000 7f70aea00700 1 --2- 172.21.15.12:0/1247475003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80d05e0 0x7f70a80d0a40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7094009840 tx=0x7f7094009b50 comp rx=0 tx=0).stop 2024-03-29T10:06:54.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.310+0000 7f70aea00700 1 -- 172.21.15.12:0/1247475003 shutdown_connections 2024-03-29T10:06:54.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.310+0000 7f70aea00700 1 --2- 172.21.15.12:0/1247475003 >> v2:172.21.15.12:3300/0 conn(0x7f70a80b6990 0x7f70a80b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.310+0000 7f70aea00700 1 --2- 172.21.15.12:0/1247475003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80d05e0 0x7f70a80d0a40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.310+0000 7f70aea00700 1 -- 172.21.15.12:0/1247475003 >> 172.21.15.12:0/1247475003 conn(0x7f70a80666a0 msgr2=0x7f70a8066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:54.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.310+0000 7f70aea00700 1 -- 172.21.15.12:0/1247475003 shutdown_connections 2024-03-29T10:06:54.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.310+0000 7f70aea00700 1 -- 172.21.15.12:0/1247475003 wait complete. 2024-03-29T10:06:54.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.313+0000 7f70aea00700 1 Processor -- start 2024-03-29T10:06:54.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.313+0000 7f70aea00700 1 -- start start 2024-03-29T10:06:54.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70aea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80b6990 0x7f70a8148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70aea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a80d05e0 0x7f70a80b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70aea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a815da50 0x7f70a80b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70aea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a80b5790 con 0x7f70a815da50 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70aea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a80b5a70 con 0x7f70a80d05e0 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70aea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a814d420 con 0x7f70a80b6990 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70aca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a815da50 0x7f70a80b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70aca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a815da50 0x7f70a80b4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36260/0 (socket says 172.21.15.12:36260) 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70aca00700 1 -- 172.21.15.12:0/34385127 learned_addr learned my addr 172.21.15.12:0/34385127 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70a7e00700 1 --2- 172.21.15.12:0/34385127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80b6990 0x7f70a8148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.314+0000 7f70a7400700 1 --2- 172.21.15.12:0/34385127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a80d05e0 0x7f70a80b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.315+0000 7f70aca00700 1 -- 172.21.15.12:0/34385127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80b6990 msgr2=0x7f70a8148ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.315+0000 7f70aca00700 1 --2- 172.21.15.12:0/34385127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80b6990 0x7f70a8148ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.315+0000 7f70aca00700 1 -- 172.21.15.12:0/34385127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a80d05e0 msgr2=0x7f70a80b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.315+0000 7f70aca00700 1 --2- 172.21.15.12:0/34385127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a80d05e0 0x7f70a80b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.315+0000 7f70aca00700 1 -- 172.21.15.12:0/34385127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a0009a70 con 0x7f70a815da50 2024-03-29T10:06:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.315+0000 7f70aca00700 1 --2- 172.21.15.12:0/34385127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a815da50 0x7f70a80b4df0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f70a000efc0 tx=0x7f70a000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.315+0000 7f709fe00700 1 -- 172.21.15.12:0/34385127 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a000ca90 con 0x7f70a815da50 2024-03-29T10:06:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.315+0000 7f709fe00700 1 -- 172.21.15.12:0/34385127 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a000f040 con 0x7f70a815da50 2024-03-29T10:06:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.315+0000 7f709fe00700 1 -- 172.21.15.12:0/34385127 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a000ce10 con 0x7f70a815da50 2024-03-29T10:06:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.315+0000 7f70aea00700 1 -- 172.21.15.12:0/34385127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70940094a0 con 0x7f70a815da50 2024-03-29T10:06:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.316+0000 7f70aea00700 1 -- 172.21.15.12:0/34385127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a814d9e0 con 0x7f70a815da50 2024-03-29T10:06:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.317+0000 7f70aea00700 1 -- 172.21.15.12:0/34385127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a814dbd0 con 0x7f70a815da50 2024-03-29T10:06:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.317+0000 7f709fe00700 1 -- 172.21.15.12:0/34385127 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 15) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f70a0014480 con 0x7f70a815da50 2024-03-29T10:06:54.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.318+0000 7f709fe00700 1 -- 172.21.15.12:0/34385127 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6792+0+0 (secure 0 0 0) 0x7f70a0089820 con 0x7f70a815da50 2024-03-29T10:06:54.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.323+0000 7f709fe00700 1 -- 172.21.15.12:0/34385127 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f70a0054920 con 0x7f70a815da50 2024-03-29T10:06:54.400 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:06:54.401+0000 7ff4587aa240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:06:54.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.583+0000 7f70aea00700 1 -- 172.21.15.12:0/34385127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70a8067720 con 0x7f70a815da50 2024-03-29T10:06:54.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.583+0000 7f709fe00700 1 -- 172.21.15.12:0/34385127 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f70a0057f40 con 0x7f70a815da50 2024-03-29T10:06:54.583 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:54.586 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2845880355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2845880355},{"type":"v1","addr":"172.21.15.159:6837","nonce":2845880355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2836227468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2836227468},{"type":"v1","addr":"172.21.15.159:6836","nonce":2836227468}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1863550973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1863550973},{"type":"v1","addr":"172.21.15.12:6834","nonce":1863550973}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T09:55:35.438233+0000","modified":"2024-03-29T09:55:38.926199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/2013939805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2013939805},{"type":"v1","addr":"172.21.15.12:6835","nonce":2013939805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-29T10:06:54.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.585+0000 7f70aea00700 1 -- 172.21.15.12:0/34385127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a815da50 msgr2=0x7f70a80b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:54.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.585+0000 7f70aea00700 1 --2- 172.21.15.12:0/34385127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a815da50 0x7f70a80b4df0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f70a000efc0 tx=0x7f70a000c3a0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.586+0000 7f70aea00700 1 -- 172.21.15.12:0/34385127 shutdown_connections 2024-03-29T10:06:54.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.586+0000 7f70aea00700 1 --2- 172.21.15.12:0/34385127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a815da50 0x7f70a80b4df0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.586+0000 7f70aea00700 1 --2- 172.21.15.12:0/34385127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80b6990 0x7f70a8148ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.586+0000 7f70aea00700 1 --2- 172.21.15.12:0/34385127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a80d05e0 0x7f70a80b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.586+0000 7f70aea00700 1 -- 172.21.15.12:0/34385127 >> 172.21.15.12:0/34385127 conn(0x7f70a80666a0 msgr2=0x7f70a8154a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:54.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.586+0000 7f70aea00700 1 -- 172.21.15.12:0/34385127 shutdown_connections 2024-03-29T10:06:54.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.586+0000 7f70aea00700 1 -- 172.21.15.12:0/34385127 wait complete. 2024-03-29T10:06:54.588 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 15 2024-03-29T10:06:54.604 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:06:54.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.804+0000 7f0756c00700 1 Processor -- start 2024-03-29T10:06:54.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.804+0000 7f0756c00700 1 -- start start 2024-03-29T10:06:54.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.805+0000 7f0756c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f075015d640 0x7f07500d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.805+0000 7f0756c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 0x7f07501612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.805+0000 7f0756c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0750161800 0x7f07500cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.805+0000 7f0756c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0750161c90 con 0x7f0750161800 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.805+0000 7f0756c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07500d30d0 con 0x7f075015d640 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.805+0000 7f0756c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07500b6840 con 0x7f07500ceec0 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.805+0000 7f0754c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0750161800 0x7f07500cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.805+0000 7f0754c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0750161800 0x7f07500cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36272/0 (socket says 172.21.15.12:36272) 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.805+0000 7f0754c00700 1 -- 172.21.15.12:0/1389683055 learned_addr learned my addr 172.21.15.12:0/1389683055 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f074f400700 1 --2- 172.21.15.12:0/1389683055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 0x7f07501612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f074fe00700 1 --2- 172.21.15.12:0/1389683055 >> v2:172.21.15.159:3300/0 conn(0x7f075015d640 0x7f07500d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f0754c00700 1 -- 172.21.15.12:0/1389683055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 msgr2=0x7f07501612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f0754c00700 1 --2- 172.21.15.12:0/1389683055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 0x7f07501612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f0754c00700 1 -- 172.21.15.12:0/1389683055 >> v2:172.21.15.159:3300/0 conn(0x7f075015d640 msgr2=0x7f07500d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:54.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f0754c00700 1 --2- 172.21.15.12:0/1389683055 >> v2:172.21.15.159:3300/0 conn(0x7f075015d640 0x7f07500d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f0754c00700 1 -- 172.21.15.12:0/1389683055 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07500d2880 con 0x7f0750161800 2024-03-29T10:06:54.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f074f400700 1 --2- 172.21.15.12:0/1389683055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 0x7f07501612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:54.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f074fe00700 1 --2- 172.21.15.12:0/1389683055 >> v2:172.21.15.159:3300/0 conn(0x7f075015d640 0x7f07500d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:06:54.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f0754c00700 1 --2- 172.21.15.12:0/1389683055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 0x7f07500cdba0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f07500b5c20 tx=0x7f074800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4379e18edfa86a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:54.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f074ea00700 1 -- 172.21.15.12:0/1389683055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074800e070 con 0x7f0750161800 2024-03-29T10:06:54.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f074ea00700 1 -- 172.21.15.12:0/1389683055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074800b9c0 con 0x7f0750161800 2024-03-29T10:06:54.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.806+0000 7f074ea00700 1 -- 172.21.15.12:0/1389683055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074800bd70 con 0x7f0750161800 2024-03-29T10:06:54.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.807+0000 7f0756c00700 1 -- 172.21.15.12:0/1389683055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 msgr2=0x7f07500cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:54.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.807+0000 7f0756c00700 1 --2- 172.21.15.12:0/1389683055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 0x7f07500cdba0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f07500b5c20 tx=0x7f074800b330 comp rx=0 tx=0).stop 2024-03-29T10:06:54.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.807+0000 7f0756c00700 1 -- 172.21.15.12:0/1389683055 shutdown_connections 2024-03-29T10:06:54.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.807+0000 7f0756c00700 1 --2- 172.21.15.12:0/1389683055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 0x7f07500cdba0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.807+0000 7f0756c00700 1 --2- 172.21.15.12:0/1389683055 >> v2:172.21.15.159:3300/0 conn(0x7f075015d640 0x7f07500d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.807+0000 7f0756c00700 1 --2- 172.21.15.12:0/1389683055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 0x7f07501612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.807+0000 7f0756c00700 1 -- 172.21.15.12:0/1389683055 >> 172.21.15.12:0/1389683055 conn(0x7f075008aff0 msgr2=0x7f07500d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.807+0000 7f0756c00700 1 -- 172.21.15.12:0/1389683055 shutdown_connections 2024-03-29T10:06:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.807+0000 7f0756c00700 1 -- 172.21.15.12:0/1389683055 wait complete. 2024-03-29T10:06:54.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.809+0000 7f0756c00700 1 Processor -- start 2024-03-29T10:06:54.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.809+0000 7f0756c00700 1 -- start start 2024-03-29T10:06:54.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0756c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 0x7f075015f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0756c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075015d640 0x7f07501587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0756c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 0x7f0750154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0756c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075015f720 con 0x7f0750161800 2024-03-29T10:06:54.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0756c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075015f9f0 con 0x7f075015d640 2024-03-29T10:06:54.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0756c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075015fcd0 con 0x7f07500ceec0 2024-03-29T10:06:54.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0754c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 0x7f0750154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0754c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 0x7f0750154af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36284/0 (socket says 172.21.15.12:36284) 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0754c00700 1 -- 172.21.15.12:0/23713462 learned_addr learned my addr 172.21.15.12:0/23713462 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f074fe00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 0x7f075015f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f074f400700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075015d640 0x7f07501587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0754c00700 1 -- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 msgr2=0x7f075015f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.810+0000 7f0754c00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 0x7f075015f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.811+0000 7f0754c00700 1 -- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075015d640 msgr2=0x7f07501587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.811+0000 7f0754c00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075015d640 0x7f07501587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.811+0000 7f0754c00700 1 -- 172.21.15.12:0/23713462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0748009b30 con 0x7f0750161800 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.811+0000 7f0754c00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 0x7f0750154af0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f075008c490 tx=0x7f0748004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.811+0000 7f0747e00700 1 -- 172.21.15.12:0/23713462 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07480092e0 con 0x7f0750161800 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.811+0000 7f0747e00700 1 -- 172.21.15.12:0/23713462 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074801c440 con 0x7f0750161800 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.811+0000 7f0756c00700 1 -- 172.21.15.12:0/23713462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075015ffb0 con 0x7f0750161800 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.811+0000 7f0747e00700 1 -- 172.21.15.12:0/23713462 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074801c7f0 con 0x7f0750161800 2024-03-29T10:06:54.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.811+0000 7f0756c00700 1 -- 172.21.15.12:0/23713462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0750160570 con 0x7f0750161800 2024-03-29T10:06:54.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.812+0000 7f0756c00700 1 -- 172.21.15.12:0/23713462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0750160760 con 0x7f0750161800 2024-03-29T10:06:54.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.813+0000 7f0747e00700 1 -- 172.21.15.12:0/23713462 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 15) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f074801c950 con 0x7f0750161800 2024-03-29T10:06:54.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.814+0000 7f0747e00700 1 -- 172.21.15.12:0/23713462 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6792+0+0 (secure 0 0 0) 0x7f074808ac30 con 0x7f0750161800 2024-03-29T10:06:54.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.818+0000 7f0747e00700 1 -- 172.21.15.12:0/23713462 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0748055d30 con 0x7f0750161800 2024-03-29T10:06:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.844+0000 7f0747e00700 1 -- 172.21.15.12:0/23713462 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 16) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f07480557e0 con 0x7f0750161800 2024-03-29T10:06:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.845+0000 7f0747e00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0730036150 0x7f0730072810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.846+0000 7f074fe00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0730036150 0x7f0730072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:54.853+0000 7f074fe00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0730036150 0x7f0730072810 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0740009990 tx=0x7f0740008040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:55.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.041+0000 7f0756c00700 1 -- 172.21.15.12:0/23713462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0750067720 con 0x7f0750161800 2024-03-29T10:06:55.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.042+0000 7f0747e00700 1 -- 172.21.15.12:0/23713462 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+17726 (secure 0 0 0) 0x7f074806f9e0 con 0x7f0750161800 2024-03-29T10:06:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:06:55.042 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":48,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:06:53.831914+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-29T09:55:33.508935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-29T09:55:34.418367+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":219,"snap_epoch":47,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3041526185":"2024-03-30T10:06:53.831833+0000","172.21.15.159:0/2869621893":"2024-03-30T10:06:53.831833+0000","172.21.15.159:0/2327439438":"2024-03-30T10:06:53.831833+0000","172.21.15.159:0/2937383753":"2024-03-30T10:06:53.831833+0000","172.21.15.159:0/679146532":"2024-03-30T10:06:53.831833+0000","172.21.15.159:0/2142375154":"2024-03-30T10:06:53.831833+0000","172.21.15.159:0/2161019748":"2024-03-30T10:06:53.831833+0000","172.21.15.159:0/1756693282":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/795330822":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/249193272":"2024-03-30T09:55:52.597859+0000","172.21.15.159:6833/43444":"2024-03-30T09:55:52.597859+0000","172.21.15.159:0/3395907487":"2024-03-30T09:55:52.597859+0000","172.21.15.159:6832/43444":"2024-03-30T09:55:52.597859+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":6,"length":1},{"begin":9,"length":1},{"begin":12,"length":1},{"begin":16,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":27,"length":1},{"begin":32,"length":1},{"begin":34,"length":1},{"begin":38,"length":1},{"begin":43,"length":1},{"begin":45,"length":1},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":55,"length":1},{"begin":59,"length":1},{"begin":64,"length":1},{"begin":66,"length":1},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":76,"length":1},{"begin":80,"length":2},{"begin":85,"length":1},{"begin":87,"length":1},{"begin":92,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":103,"length":1},{"begin":107,"length":1},{"begin":109,"length":1},{"begin":187,"length":33}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.044+0000 7f0756c00700 1 -- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0730036150 msgr2=0x7f0730072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0730036150 0x7f0730072810 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0740009990 tx=0x7f0740008040 comp rx=0 tx=0).stop 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 -- 172.21.15.12:0/23713462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 msgr2=0x7f0750154af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 0x7f0750154af0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f075008c490 tx=0x7f0748004ad0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 -- 172.21.15.12:0/23713462 shutdown_connections 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0730036150 0x7f0730072810 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0750161800 0x7f0750154af0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07500ceec0 0x7f075015f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 --2- 172.21.15.12:0/23713462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075015d640 0x7f07501587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 -- 172.21.15.12:0/23713462 >> 172.21.15.12:0/23713462 conn(0x7f075008aff0 msgr2=0x7f07500cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:55.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 -- 172.21.15.12:0/23713462 shutdown_connections 2024-03-29T10:06:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.045+0000 7f0756c00700 1 -- 172.21.15.12:0/23713462 wait complete. 2024-03-29T10:06:55.063 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T10:06:55.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.263+0000 7fe3f2800700 1 Processor -- start 2024-03-29T10:06:55.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.263+0000 7fe3f2800700 1 -- start start 2024-03-29T10:06:55.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.264+0000 7fe3f2800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe3ec15d9d0 0x7fe3ec0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.264+0000 7fe3f2800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 0x7fe3ec0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.264+0000 7fe3f2800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ec161420 con 0x7fe3ec0d2370 2024-03-29T10:06:55.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.264+0000 7fe3f2800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec0d2820 con 0x7fe3ec15d9d0 2024-03-29T10:06:55.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.264+0000 7fe3f2800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec161700 con 0x7fe3ec0d16a0 2024-03-29T10:06:55.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.265+0000 7fe3eb400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe3ec15d9d0 0x7fe3ec0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.265+0000 7fe3eaa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 0x7fe3ec0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.265+0000 7fe3eb400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe3ec15d9d0 0x7fe3ec0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:45784/0 (socket says 172.21.15.12:45784) 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.265+0000 7fe3eaa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 0x7fe3ec0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:43606/0 (socket says 172.21.15.12:43606) 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.265+0000 7fe3eb400700 1 -- 172.21.15.12:0/672306563 learned_addr learned my addr 172.21.15.12:0/672306563 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.265+0000 7fe3ebe00700 1 --1- 172.21.15.12:0/672306563 >> v1:172.21.15.159:6789/0 conn(0x7fe3ec0d2370 0x7fe3ec15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:53206/0 (socket says 172.21.15.12:53206) 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.265+0000 7fe3eaa00700 1 -- 172.21.15.12:0/672306563 >> v2:172.21.15.159:3300/0 conn(0x7fe3ec15d9d0 msgr2=0x7fe3ec0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.265+0000 7fe3eaa00700 1 --2- 172.21.15.12:0/672306563 >> v2:172.21.15.159:3300/0 conn(0x7fe3ec15d9d0 0x7fe3ec0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.265+0000 7fe3eaa00700 1 -- 172.21.15.12:0/672306563 >> v1:172.21.15.159:6789/0 conn(0x7fe3ec0d2370 legacy=0x7fe3ec15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.265+0000 7fe3ea000700 1 -- 172.21.15.12:0/672306563 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988140319 0 0) 0x7fe3ec161420 con 0x7fe3ec0d2370 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.266+0000 7fe3eaa00700 1 -- 172.21.15.12:0/672306563 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec1619e0 con 0x7fe3ec0d16a0 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.266+0000 7fe3eaa00700 1 --2- 172.21.15.12:0/672306563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 0x7fe3ec0d1e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009820 tx=0x7fe3d8009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=629fe0a0522a7ba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:55.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.266+0000 7fe3ea000700 1 -- 172.21.15.12:0/672306563 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8005e30 con 0x7fe3ec0d16a0 2024-03-29T10:06:55.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.266+0000 7fe3ea000700 1 -- 172.21.15.12:0/672306563 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d8006030 con 0x7fe3ec0d16a0 2024-03-29T10:06:55.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.266+0000 7fe3ea000700 1 -- 172.21.15.12:0/672306563 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d80063c0 con 0x7fe3ec0d16a0 2024-03-29T10:06:55.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.267+0000 7fe3f2800700 1 -- 172.21.15.12:0/672306563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 msgr2=0x7fe3ec0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.267+0000 7fe3f2800700 1 --2- 172.21.15.12:0/672306563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 0x7fe3ec0d1e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009820 tx=0x7fe3d8009b30 comp rx=0 tx=0).stop 2024-03-29T10:06:55.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.267+0000 7fe3f2800700 1 -- 172.21.15.12:0/672306563 shutdown_connections 2024-03-29T10:06:55.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.267+0000 7fe3f2800700 1 --2- 172.21.15.12:0/672306563 >> v2:172.21.15.159:3300/0 conn(0x7fe3ec15d9d0 0x7fe3ec0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.267+0000 7fe3f2800700 1 --2- 172.21.15.12:0/672306563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 0x7fe3ec0d1e30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.267+0000 7fe3f2800700 1 -- 172.21.15.12:0/672306563 >> 172.21.15.12:0/672306563 conn(0x7fe3ec08aff0 msgr2=0x7fe3ec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:55.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.267+0000 7fe3f2800700 1 -- 172.21.15.12:0/672306563 shutdown_connections 2024-03-29T10:06:55.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.267+0000 7fe3f2800700 1 -- 172.21.15.12:0/672306563 wait complete. 2024-03-29T10:06:55.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.269+0000 7fe3f2800700 1 Processor -- start 2024-03-29T10:06:55.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.270+0000 7fe3f2800700 1 -- start start 2024-03-29T10:06:55.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3f2800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 0x7fe3ec0b6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3f2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ec0d2370 0x7fe3ec1f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3f2800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ec15d9d0 0x7fe3ec0b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3f2800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec15fed0 con 0x7fe3ec0d2370 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3f2800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec15f250 con 0x7fe3ec15d9d0 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3f2800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec15f530 con 0x7fe3ec0d16a0 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ec0d2370 0x7fe3ec1f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ec0d2370 0x7fe3ec1f3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36288/0 (socket says 172.21.15.12:36288) 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3eaa00700 1 -- 172.21.15.12:0/143832182 learned_addr learned my addr 172.21.15.12:0/143832182 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3ebe00700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ec15d9d0 0x7fe3ec0b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.271+0000 7fe3eb400700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 0x7fe3ec0b6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3eaa00700 1 -- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 msgr2=0x7fe3ec0b6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3eaa00700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 0x7fe3ec0b6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3eaa00700 1 -- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ec15d9d0 msgr2=0x7fe3ec0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3eaa00700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ec15d9d0 0x7fe3ec0b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3eaa00700 1 -- 172.21.15.12:0/143832182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d8009430 con 0x7fe3ec0d2370 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3ebe00700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ec15d9d0 0x7fe3ec0b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:55.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3eaa00700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ec0d2370 0x7fe3ec1f3920 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8000c00 tx=0x7fe3d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:55.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3e3400700 1 -- 172.21.15.12:0/143832182 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d80061a0 con 0x7fe3ec0d2370 2024-03-29T10:06:55.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3e3400700 1 -- 172.21.15.12:0/143832182 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d8020070 con 0x7fe3ec0d2370 2024-03-29T10:06:55.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3e3400700 1 -- 172.21.15.12:0/143832182 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8006540 con 0x7fe3ec0d2370 2024-03-29T10:06:55.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3f2800700 1 -- 172.21.15.12:0/143832182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec15f810 con 0x7fe3ec0d2370 2024-03-29T10:06:55.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.272+0000 7fe3f2800700 1 -- 172.21.15.12:0/143832182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec0b7160 con 0x7fe3ec0d2370 2024-03-29T10:06:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.273+0000 7fe3f2800700 1 -- 172.21.15.12:0/143832182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ec0d1ca0 con 0x7fe3ec0d2370 2024-03-29T10:06:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.274+0000 7fe3e3400700 1 -- 172.21.15.12:0/143832182 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fe3d8022030 con 0x7fe3ec0d2370 2024-03-29T10:06:55.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.276+0000 7fe3e3400700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3cc06aae0 0x7fe3cc06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.276+0000 7fe3e3400700 1 -- 172.21.15.12:0/143832182 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6792+0+0 (secure 0 0 0) 0x7fe3d808a900 con 0x7fe3ec0d2370 2024-03-29T10:06:55.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.276+0000 7fe3eb400700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3cc06aae0 0x7fe3cc06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.277+0000 7fe3eb400700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3cc06aae0 0x7fe3cc06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0b6600 tx=0x7fe3dc0092c0 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:55.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.281+0000 7fe3e3400700 1 -- 172.21.15.12:0/143832182 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe3d8055910 con 0x7fe3ec0d2370 2024-03-29T10:06:55.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.545+0000 7fe3f2800700 1 -- 172.21.15.12:0/143832182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe3ec160010 con 0x7fe3ec0d2370 2024-03-29T10:06:55.545 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T10:06:55.546+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T10:06:55.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.556+0000 7fe3e3400700 1 -- 172.21.15.12:0/143832182 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe3ec160010 con 0x7fe3ec0d2370 2024-03-29T10:06:55.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 -- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3cc06aae0 msgr2=0x7fe3cc06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3cc06aae0 0x7fe3cc06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0b6600 tx=0x7fe3dc0092c0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 -- 172.21.15.12:0/143832182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ec0d2370 msgr2=0x7fe3ec1f3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ec0d2370 0x7fe3ec1f3920 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8000c00 tx=0x7fe3d8004750 comp rx=0 tx=0).stop 2024-03-29T10:06:55.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 -- 172.21.15.12:0/143832182 shutdown_connections 2024-03-29T10:06:55.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe3cc06aae0 0x7fe3cc06cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ec15d9d0 0x7fe3ec0b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ec0d16a0 0x7fe3ec0b6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 --2- 172.21.15.12:0/143832182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ec0d2370 0x7fe3ec1f3920 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 -- 172.21.15.12:0/143832182 >> 172.21.15.12:0/143832182 conn(0x7fe3ec08aff0 msgr2=0x7fe3ec1523c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:55.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 -- 172.21.15.12:0/143832182 shutdown_connections 2024-03-29T10:06:55.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.559+0000 7fe3f2800700 1 -- 172.21.15.12:0/143832182 wait complete. 2024-03-29T10:06:55.560 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T10:06:55.578 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T10:06:55.622 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:06:55.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.783+0000 7f1a3ba00700 1 Processor -- start 2024-03-29T10:06:55.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.784+0000 7f1a3ba00700 1 -- start start 2024-03-29T10:06:55.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.785+0000 7f1a3ba00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1a340cbb30 0x7f1a340cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.785+0000 7f1a3ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd480 0x7f1a3406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.785+0000 7f1a3ba00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a340cc450 con 0x7f1a340cd020 2024-03-29T10:06:55.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.785+0000 7f1a3ba00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a340cdae0 con 0x7f1a340cbb30 2024-03-29T10:06:55.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.785+0000 7f1a3ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a340cddc0 con 0x7f1a340cd480 2024-03-29T10:06:55.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.785+0000 7f1a33e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1a340cd020 0x7f1a340b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57588/0 (socket says 172.21.15.12:57588) 2024-03-29T10:06:55.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.785+0000 7f1a33e00700 1 -- 172.21.15.12:0/244160264 learned_addr learned my addr 172.21.15.12:0/244160264 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:55.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.785+0000 7f1a39000700 1 --2- 172.21.15.12:0/244160264 >> v2:172.21.15.159:3300/0 conn(0x7f1a340cbb30 0x7f1a340cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.785+0000 7f1a39a00700 1 --2- 172.21.15.12:0/244160264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd480 0x7f1a3406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.786+0000 7f1a33400700 1 -- 172.21.15.12:0/244160264 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687605444 0 0) 0x7f1a340cc450 con 0x7f1a340cd020 2024-03-29T10:06:55.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.786+0000 7f1a33400700 1 -- 172.21.15.12:0/244160264 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a1c0035b0 con 0x7f1a340cd020 2024-03-29T10:06:55.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.786+0000 7f1a39a00700 1 -- 172.21.15.12:0/244160264 >> v2:172.21.15.159:3300/0 conn(0x7f1a340cbb30 msgr2=0x7f1a340cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:55.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.786+0000 7f1a39a00700 1 --2- 172.21.15.12:0/244160264 >> v2:172.21.15.159:3300/0 conn(0x7f1a340cbb30 0x7f1a340cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.786+0000 7f1a39a00700 1 -- 172.21.15.12:0/244160264 >> v1:172.21.15.12:6789/0 conn(0x7f1a340cd020 legacy=0x7f1a340b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.786+0000 7f1a39a00700 1 -- 172.21.15.12:0/244160264 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a340ce0a0 con 0x7f1a340cd480 2024-03-29T10:06:55.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.786+0000 7f1a39a00700 1 --2- 172.21.15.12:0/244160264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd480 0x7f1a3406e420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009d90 tx=0x7f1a2c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fad231aa90306e33 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:55.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.786+0000 7f1a33400700 1 -- 172.21.15.12:0/244160264 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c00e070 con 0x7f1a340cd480 2024-03-29T10:06:55.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.786+0000 7f1a33400700 1 -- 172.21.15.12:0/244160264 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c00bb20 con 0x7f1a340cd480 2024-03-29T10:06:55.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.786+0000 7f1a33400700 1 -- 172.21.15.12:0/244160264 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c00be90 con 0x7f1a340cd480 2024-03-29T10:06:55.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.787+0000 7f1a3ba00700 1 -- 172.21.15.12:0/244160264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd480 msgr2=0x7f1a3406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.787+0000 7f1a3ba00700 1 --2- 172.21.15.12:0/244160264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd480 0x7f1a3406e420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009d90 tx=0x7f1a2c00b330 comp rx=0 tx=0).stop 2024-03-29T10:06:55.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.787+0000 7f1a3ba00700 1 -- 172.21.15.12:0/244160264 shutdown_connections 2024-03-29T10:06:55.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.787+0000 7f1a3ba00700 1 --2- 172.21.15.12:0/244160264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd480 0x7f1a3406e420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.787+0000 7f1a3ba00700 1 --2- 172.21.15.12:0/244160264 >> v2:172.21.15.159:3300/0 conn(0x7f1a340cbb30 0x7f1a340cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.787+0000 7f1a3ba00700 1 -- 172.21.15.12:0/244160264 >> 172.21.15.12:0/244160264 conn(0x7f1a340b0d70 msgr2=0x7f1a340b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:55.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.787+0000 7f1a3ba00700 1 -- 172.21.15.12:0/244160264 shutdown_connections 2024-03-29T10:06:55.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.787+0000 7f1a3ba00700 1 -- 172.21.15.12:0/244160264 wait complete. 2024-03-29T10:06:55.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.789+0000 7f1a3ba00700 1 Processor -- start 2024-03-29T10:06:55.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.789+0000 7f1a3ba00700 1 -- start start 2024-03-29T10:06:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a3ba00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a340cbb30 0x7f1a3414d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a3ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd020 0x7f1a340c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a3ba00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340cd480 0x7f1a3414e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a3ba00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a340c83f0 con 0x7f1a340cd480 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a3ba00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a34071840 con 0x7f1a340cbb30 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a3ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a340c86b0 con 0x7f1a340cd020 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a39a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340cd480 0x7f1a3414e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a39a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340cd480 0x7f1a3414e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36306/0 (socket says 172.21.15.12:36306) 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a39a00700 1 -- 172.21.15.12:0/2510231780 learned_addr learned my addr 172.21.15.12:0/2510231780 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a39000700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a340cbb30 0x7f1a3414d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.790+0000 7f1a33e00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd020 0x7f1a340c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a39a00700 1 -- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd020 msgr2=0x7f1a340c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a39a00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd020 0x7f1a340c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a39a00700 1 -- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a340cbb30 msgr2=0x7f1a3414d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a39a00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a340cbb30 0x7f1a3414d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a39a00700 1 -- 172.21.15.12:0/2510231780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c009a70 con 0x7f1a340cd480 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a39000700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a340cbb30 0x7f1a3414d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:06:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a39a00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340cd480 0x7f1a3414e2b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c014040 tx=0x7f1a2c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a30c00700 1 -- 172.21.15.12:0/2510231780 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c007000 con 0x7f1a340cd480 2024-03-29T10:06:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a30c00700 1 -- 172.21.15.12:0/2510231780 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c012650 con 0x7f1a340cd480 2024-03-29T10:06:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a3ba00700 1 -- 172.21.15.12:0/2510231780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a340c8990 con 0x7f1a340cd480 2024-03-29T10:06:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a30c00700 1 -- 172.21.15.12:0/2510231780 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c012a00 con 0x7f1a340cd480 2024-03-29T10:06:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.791+0000 7f1a3ba00700 1 -- 172.21.15.12:0/2510231780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3414dd00 con 0x7f1a340cd480 2024-03-29T10:06:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.793+0000 7f1a3ba00700 1 -- 172.21.15.12:0/2510231780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a340c8eb0 con 0x7f1a340cd480 2024-03-29T10:06:55.793 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:06:55.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.793+0000 7f1a30c00700 1 -- 172.21.15.12:0/2510231780 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f1a2c012b60 con 0x7f1a340cd480 2024-03-29T10:06:55.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.794+0000 7f1a30c00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1406aae0 0x7f1a1406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:55.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.794+0000 7f1a30c00700 1 -- 172.21.15.12:0/2510231780 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6878+0+0 (secure 0 0 0) 0x7f1a2c08b0a0 con 0x7f1a340cd480 2024-03-29T10:06:55.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.795+0000 7f1a39000700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1406aae0 0x7f1a1406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:55.799 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:06:55.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.795+0000 7f1a39000700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1406aae0 0x7f1a1406cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1a28009f30 tx=0x7f1a28009210 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:55.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:55.801+0000 7f1a30c00700 1 -- 172.21.15.12:0/2510231780 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1a340c8eb0 con 0x7f1a340cd480 2024-03-29T10:06:56.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.071+0000 7f1a3ba00700 1 -- 172.21.15.12:0/2510231780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1a340c8eb0 con 0x7f1a340cd480 2024-03-29T10:06:56.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.559+0000 7f1a30c00700 1 -- 172.21.15.12:0/2510231780 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1a340c8eb0 con 0x7f1a340cd480 2024-03-29T10:06:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.562+0000 7f1a3ba00700 1 -- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1406aae0 msgr2=0x7f1a1406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.562+0000 7f1a3ba00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1406aae0 0x7f1a1406cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1a28009f30 tx=0x7f1a28009210 comp rx=0 tx=0).stop 2024-03-29T10:06:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.562+0000 7f1a3ba00700 1 -- 172.21.15.12:0/2510231780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340cd480 msgr2=0x7f1a3414e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:56.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.562+0000 7f1a3ba00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340cd480 0x7f1a3414e2b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c014040 tx=0x7f1a2c004ad0 comp rx=0 tx=0).stop 2024-03-29T10:06:56.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.562+0000 7f1a3ba00700 1 -- 172.21.15.12:0/2510231780 shutdown_connections 2024-03-29T10:06:56.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.562+0000 7f1a3ba00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1406aae0 0x7f1a1406cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:56.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.562+0000 7f1a3ba00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340cd480 0x7f1a3414e2b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:56.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.562+0000 7f1a3ba00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a340cbb30 0x7f1a3414d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:56.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.562+0000 7f1a3ba00700 1 --2- 172.21.15.12:0/2510231780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a340cd020 0x7f1a340c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:56.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.562+0000 7f1a3ba00700 1 -- 172.21.15.12:0/2510231780 >> 172.21.15.12:0/2510231780 conn(0x7f1a340b0d70 msgr2=0x7f1a3406e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:56.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.563+0000 7f1a3ba00700 1 -- 172.21.15.12:0/2510231780 shutdown_connections 2024-03-29T10:06:56.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.563+0000 7f1a3ba00700 1 -- 172.21.15.12:0/2510231780 wait complete. 2024-03-29T10:06:56.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T10:06:56.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.790+0000 7fa3b2200700 1 Processor -- start 2024-03-29T10:06:56.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.790+0000 7fa3b2200700 1 -- start start 2024-03-29T10:06:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.791+0000 7fa3b2200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa3ac15d930 0x7fa3ac0d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.791+0000 7fa3b2200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac0d22a0 0x7fa3ac0ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3b2200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac15dd10 con 0x7fa3ac162030 2024-03-29T10:06:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3b2200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ac15ec40 con 0x7fa3ac15d930 2024-03-29T10:06:56.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3b2200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac15dfc0 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3aaa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa3ac162030 0x7fa3ac0d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57592/0 (socket says 172.21.15.12:57592) 2024-03-29T10:06:56.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3ab400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa3ac15d930 0x7fa3ac0d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:56.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3aaa00700 1 -- 172.21.15.12:0/3504955779 learned_addr learned my addr 172.21.15.12:0/3504955779 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3abe00700 1 --2- 172.21.15.12:0/3504955779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac0d22a0 0x7fa3ac0ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3ab400700 1 -- 172.21.15.12:0/3504955779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac0d22a0 msgr2=0x7fa3ac0ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3ab400700 1 --2- 172.21.15.12:0/3504955779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac0d22a0 0x7fa3ac0ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3ab400700 1 -- 172.21.15.12:0/3504955779 >> v1:172.21.15.12:6789/0 conn(0x7fa3ac162030 legacy=0x7fa3ac0d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3ab400700 1 -- 172.21.15.12:0/3504955779 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac15e2a0 con 0x7fa3ac15d930 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.792+0000 7fa3aa000700 1 -- 172.21.15.12:0/3504955779 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696878110 0 0) 0x7fa3ac15dd10 con 0x7fa3ac162030 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.793+0000 7fa3ab400700 1 --2- 172.21.15.12:0/3504955779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac15d930 0x7fa3ac0d3ed0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa39c009840 tx=0x7fa39c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec23ffa5ae3d5ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.793+0000 7fa3aa000700 1 -- 172.21.15.12:0/3504955779 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c011830 con 0x7fa3ac15d930 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.793+0000 7fa3aa000700 1 -- 172.21.15.12:0/3504955779 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39c011990 con 0x7fa3ac15d930 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.793+0000 7fa3aa000700 1 -- 172.21.15.12:0/3504955779 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c011d20 con 0x7fa3ac15d930 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.793+0000 7fa3b2200700 1 -- 172.21.15.12:0/3504955779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac15d930 msgr2=0x7fa3ac0d3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.793+0000 7fa3b2200700 1 --2- 172.21.15.12:0/3504955779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac15d930 0x7fa3ac0d3ed0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa39c009840 tx=0x7fa39c009b50 comp rx=0 tx=0).stop 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.793+0000 7fa3b2200700 1 -- 172.21.15.12:0/3504955779 shutdown_connections 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.793+0000 7fa3b2200700 1 --2- 172.21.15.12:0/3504955779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac0d22a0 0x7fa3ac0ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.793+0000 7fa3b2200700 1 --2- 172.21.15.12:0/3504955779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac15d930 0x7fa3ac0d3ed0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:56.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.793+0000 7fa3b2200700 1 -- 172.21.15.12:0/3504955779 >> 172.21.15.12:0/3504955779 conn(0x7fa3ac08aff0 msgr2=0x7fa3ac0d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:56.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.794+0000 7fa3b2200700 1 -- 172.21.15.12:0/3504955779 shutdown_connections 2024-03-29T10:06:56.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.794+0000 7fa3b2200700 1 -- 172.21.15.12:0/3504955779 wait complete. 2024-03-29T10:06:56.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.795+0000 7fa3b2200700 1 Processor -- start 2024-03-29T10:06:56.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.795+0000 7fa3b2200700 1 -- start start 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.796+0000 7fa3b2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac0d22a0 0x7fa3ac15aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.796+0000 7fa3b2200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac15d930 0x7fa3ac0b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.796+0000 7fa3b2200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac162030 0x7fa3ac0ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.796+0000 7fa3b2200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac0c70e0 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.796+0000 7fa3b2200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac0c7330 con 0x7fa3ac15d930 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.796+0000 7fa3b2200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac0c7610 con 0x7fa3ac162030 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3ab400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac0d22a0 0x7fa3ac15aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3ab400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac0d22a0 0x7fa3ac15aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36328/0 (socket says 172.21.15.12:36328) 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3ab400700 1 -- 172.21.15.12:0/819053317 learned_addr learned my addr 172.21.15.12:0/819053317 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3abe00700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac162030 0x7fa3ac0ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3aaa00700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac15d930 0x7fa3ac0b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:56.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3ab400700 1 -- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac162030 msgr2=0x7fa3ac0ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3ab400700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac162030 0x7fa3ac0ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3ab400700 1 -- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac15d930 msgr2=0x7fa3ac0b7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3ab400700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac15d930 0x7fa3ac0b7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3ab400700 1 -- 172.21.15.12:0/819053317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c0094a0 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3aaa00700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac15d930 0x7fa3ac0b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.797+0000 7fa3ab400700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac0d22a0 0x7fa3ac15aa30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0090b0 tx=0x7fa39c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.798+0000 7fa3a3400700 1 -- 172.21.15.12:0/819053317 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c0038a0 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.798+0000 7fa3a3400700 1 -- 172.21.15.12:0/819053317 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39c01f070 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.798+0000 7fa3b2200700 1 -- 172.21.15.12:0/819053317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac0c78f0 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.798+0000 7fa3a3400700 1 -- 172.21.15.12:0/819053317 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c003c00 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.798+0000 7fa3b2200700 1 -- 172.21.15.12:0/819053317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac15af70 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.799+0000 7fa3b2200700 1 -- 172.21.15.12:0/819053317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac067720 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.800+0000 7fa3a3400700 1 -- 172.21.15.12:0/819053317 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fa39c01a430 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.800+0000 7fa3a3400700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa38c06ad20 0x7fa38c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.801+0000 7fa3a3400700 1 -- 172.21.15.12:0/819053317 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6878+0+0 (secure 0 0 0) 0x7fa39c021070 con 0x7fa3ac0d22a0 2024-03-29T10:06:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.801+0000 7fa3aaa00700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa38c06ad20 0x7fa38c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.802+0000 7fa3aaa00700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa38c06ad20 0x7fa38c06d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa398007fd0 tx=0x7fa398009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:56.805+0000 7fa3a3400700 1 -- 172.21.15.12:0/819053317 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa39c056010 con 0x7fa3ac0d22a0 2024-03-29T10:06:57.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.017+0000 7fa3b2200700 1 -- 172.21.15.12:0/819053317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa3ac0b9530 con 0x7fa3ac0d22a0 2024-03-29T10:06:57.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.848+0000 7fa3a3400700 1 -- 172.21.15.12:0/819053317 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v50) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa39c015020 con 0x7fa3ac0d22a0 2024-03-29T10:06:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.854+0000 7fa3b2200700 1 -- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa38c06ad20 msgr2=0x7fa38c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.854+0000 7fa3b2200700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa38c06ad20 0x7fa38c06d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa398007fd0 tx=0x7fa398009040 comp rx=0 tx=0).stop 2024-03-29T10:06:57.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.854+0000 7fa3b2200700 1 -- 172.21.15.12:0/819053317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac0d22a0 msgr2=0x7fa3ac15aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.854+0000 7fa3b2200700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac0d22a0 0x7fa3ac15aa30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0090b0 tx=0x7fa39c004750 comp rx=0 tx=0).stop 2024-03-29T10:06:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.854+0000 7fa3b2200700 1 -- 172.21.15.12:0/819053317 shutdown_connections 2024-03-29T10:06:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.854+0000 7fa3b2200700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa38c06ad20 0x7fa38c06d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.854+0000 7fa3b2200700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac162030 0x7fa3ac0ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.854+0000 7fa3b2200700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3ac0d22a0 0x7fa3ac15aa30 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.854+0000 7fa3b2200700 1 --2- 172.21.15.12:0/819053317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac15d930 0x7fa3ac0b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.854+0000 7fa3b2200700 1 -- 172.21.15.12:0/819053317 >> 172.21.15.12:0/819053317 conn(0x7fa3ac08aff0 msgr2=0x7fa3ac161120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:57.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.855+0000 7fa3b2200700 1 -- 172.21.15.12:0/819053317 shutdown_connections 2024-03-29T10:06:57.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:57.855+0000 7fa3b2200700 1 -- 172.21.15.12:0/819053317 wait complete. 2024-03-29T10:06:57.856 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T10:06:57.877 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T10:06:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.087+0000 7f393be00700 1 Processor -- start 2024-03-29T10:06:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.088+0000 7f393be00700 1 -- start start 2024-03-29T10:06:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.089+0000 7f393be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39340ced30 0x7f39340d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.089+0000 7f393be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39340d2710 0x7f393415f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.089+0000 7f393be00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39340ce8a0 con 0x7f393415ddf0 2024-03-29T10:06:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.089+0000 7f393be00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393415ec60 con 0x7f39340d2710 2024-03-29T10:06:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.089+0000 7f393be00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39340cdc90 con 0x7f39340ced30 2024-03-29T10:06:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.089+0000 7f3939e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39340d2710 0x7f393415f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.089+0000 7f3939e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39340d2710 0x7f393415f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36336/0 (socket says 172.21.15.12:36336) 2024-03-29T10:06:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.089+0000 7f3939e00700 1 -- 172.21.15.12:0/3901587310 learned_addr learned my addr 172.21.15.12:0/3901587310 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.089+0000 7f3938a00700 1 --2- 172.21.15.12:0/3901587310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39340ced30 0x7f39340d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.090+0000 7f3939e00700 1 -- 172.21.15.12:0/3901587310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39340ced30 msgr2=0x7f39340d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.090+0000 7f3939e00700 1 --2- 172.21.15.12:0/3901587310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39340ced30 0x7f39340d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.090+0000 7f3939e00700 1 -- 172.21.15.12:0/3901587310 >> v1:172.21.15.159:6789/0 conn(0x7f393415ddf0 legacy=0x7f393415e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.090+0000 7f3939e00700 1 -- 172.21.15.12:0/3901587310 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39340cdf70 con 0x7f39340d2710 2024-03-29T10:06:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.090+0000 7f3939e00700 1 --2- 172.21.15.12:0/3901587310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 0x7f393415f0f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f392c009e50 tx=0x7f392c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d4cb507d90639f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.090+0000 7f3933e00700 1 -- 172.21.15.12:0/3901587310 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c00e070 con 0x7f39340d2710 2024-03-29T10:06:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.090+0000 7f3933e00700 1 -- 172.21.15.12:0/3901587310 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c00b910 con 0x7f39340d2710 2024-03-29T10:06:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.090+0000 7f3933e00700 1 -- 172.21.15.12:0/3901587310 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c00bcc0 con 0x7f39340d2710 2024-03-29T10:06:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.091+0000 7f393be00700 1 -- 172.21.15.12:0/3901587310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 msgr2=0x7f393415f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.091+0000 7f393be00700 1 --2- 172.21.15.12:0/3901587310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 0x7f393415f0f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f392c009e50 tx=0x7f392c00b330 comp rx=0 tx=0).stop 2024-03-29T10:06:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.091+0000 7f393be00700 1 -- 172.21.15.12:0/3901587310 shutdown_connections 2024-03-29T10:06:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.091+0000 7f393be00700 1 --2- 172.21.15.12:0/3901587310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 0x7f393415f0f0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.091+0000 7f393be00700 1 --2- 172.21.15.12:0/3901587310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39340ced30 0x7f39340d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.091+0000 7f393be00700 1 -- 172.21.15.12:0/3901587310 >> 172.21.15.12:0/3901587310 conn(0x7f393408aff0 msgr2=0x7f39340d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.091+0000 7f393be00700 1 -- 172.21.15.12:0/3901587310 shutdown_connections 2024-03-29T10:06:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.091+0000 7f393be00700 1 -- 172.21.15.12:0/3901587310 wait complete. 2024-03-29T10:06:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.093+0000 7f393be00700 1 Processor -- start 2024-03-29T10:06:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.093+0000 7f393be00700 1 -- start start 2024-03-29T10:06:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f393be00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39340ced30 0x7f39340c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f393be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 0x7f393409d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f393be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f393415ddf0 0x7f39340ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f393be00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39340a0460 con 0x7f39340d2710 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f393be00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39340a0740 con 0x7f39340ced30 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f393be00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39340a0a20 con 0x7f393415ddf0 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f3938a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 0x7f393409d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f3938a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 0x7f393409d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36338/0 (socket says 172.21.15.12:36338) 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f3938a00700 1 -- 172.21.15.12:0/3005962595 learned_addr learned my addr 172.21.15.12:0/3005962595 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f3939e00700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f393415ddf0 0x7f39340ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f3938a00700 1 -- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f393415ddf0 msgr2=0x7f39340ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f3938a00700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f393415ddf0 0x7f39340ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f3938a00700 1 -- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39340ced30 msgr2=0x7f39340c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f3938a00700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39340ced30 0x7f39340c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.094+0000 7f3938a00700 1 -- 172.21.15.12:0/3005962595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c009b30 con 0x7f39340d2710 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.095+0000 7f3938a00700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 0x7f393409d960 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f392000e870 tx=0x7f392000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.095+0000 7f3931600700 1 -- 172.21.15.12:0/3005962595 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920009590 con 0x7f39340d2710 2024-03-29T10:06:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.095+0000 7f3931600700 1 -- 172.21.15.12:0/3005962595 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3920013680 con 0x7f39340d2710 2024-03-29T10:06:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.095+0000 7f393be00700 1 -- 172.21.15.12:0/3005962595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39340a0d00 con 0x7f39340d2710 2024-03-29T10:06:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.095+0000 7f3931600700 1 -- 172.21.15.12:0/3005962595 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200139f0 con 0x7f39340d2710 2024-03-29T10:06:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.095+0000 7f393be00700 1 -- 172.21.15.12:0/3005962595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39340c97e0 con 0x7f39340d2710 2024-03-29T10:06:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.096+0000 7f393be00700 1 -- 172.21.15.12:0/3005962595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39340d4370 con 0x7f39340d2710 2024-03-29T10:06:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.097+0000 7f3931600700 1 -- 172.21.15.12:0/3005962595 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f3920013b50 con 0x7f39340d2710 2024-03-29T10:06:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.098+0000 7f3931600700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f391406ad20 0x7f391406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.098+0000 7f3931600700 1 -- 172.21.15.12:0/3005962595 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f39200894b0 con 0x7f39340d2710 2024-03-29T10:06:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.098+0000 7f3939400700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f391406ad20 0x7f391406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.099+0000 7f3939400700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f391406ad20 0x7f391406d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3928000f60 tx=0x7f3928007a40 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.103+0000 7f3931600700 1 -- 172.21.15.12:0/3005962595 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3920054450 con 0x7f39340d2710 2024-03-29T10:06:58.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.138+0000 7f3931600700 1 -- 172.21.15.12:0/3005962595 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3920054820 con 0x7f39340d2710 2024-03-29T10:06:58.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:58.321+0000 7f393be00700 1 -- 172.21.15.12:0/3005962595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f39340cf110 con 0x7f39340d2710 2024-03-29T10:06:59.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.138+0000 7f3931600700 1 -- 172.21.15.12:0/3005962595 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v52) v1 ==== 169+0+0 (secure 0 0 0) 0x7f392005ce60 con 0x7f39340d2710 2024-03-29T10:06:59.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 -- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f391406ad20 msgr2=0x7f391406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f391406ad20 0x7f391406d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3928000f60 tx=0x7f3928007a40 comp rx=0 tx=0).stop 2024-03-29T10:06:59.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 -- 172.21.15.12:0/3005962595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 msgr2=0x7f393409d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 0x7f393409d960 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f392000e870 tx=0x7f392000eb80 comp rx=0 tx=0).stop 2024-03-29T10:06:59.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 -- 172.21.15.12:0/3005962595 shutdown_connections 2024-03-29T10:06:59.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f391406ad20 0x7f391406d1e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f393415ddf0 0x7f39340ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39340ced30 0x7f39340c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 --2- 172.21.15.12:0/3005962595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39340d2710 0x7f393409d960 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 -- 172.21.15.12:0/3005962595 >> 172.21.15.12:0/3005962595 conn(0x7f393408aff0 msgr2=0x7f393415e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:59.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 -- 172.21.15.12:0/3005962595 shutdown_connections 2024-03-29T10:06:59.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.141+0000 7f393be00700 1 -- 172.21.15.12:0/3005962595 wait complete. 2024-03-29T10:06:59.142 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T10:06:59.161 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T10:06:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.366+0000 7ff000600700 1 Processor -- start 2024-03-29T10:06:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.366+0000 7ff000600700 1 -- start start 2024-03-29T10:06:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.367+0000 7ff000600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7feff80b4a50 0x7feff815b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.367+0000 7ff000600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff806eae0 con 0x7feff806da80 2024-03-29T10:06:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7ff000600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff806f660 con 0x7feff815af30 2024-03-29T10:06:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7ff000600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff806faf0 con 0x7feff80b4a50 2024-03-29T10:06:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feffd200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7feff806da80 0x7feff80b2c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41428/0 (socket says 172.21.15.12:41428) 2024-03-29T10:06:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feffd200700 1 -- 172.21.15.12:0/2553600001 learned_addr learned my addr 172.21.15.12:0/2553600001 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feffe600700 1 --2- 172.21.15.12:0/2553600001 >> v2:172.21.15.159:3300/0 conn(0x7feff80b4a50 0x7feff815b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708739867 0 0) 0x7feff806eae0 con 0x7feff806da80 2024-03-29T10:06:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe00035b0 con 0x7feff806da80 2024-03-29T10:06:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481585100 0 0) 0x7fefe00035b0 con 0x7feff806da80 2024-03-29T10:06:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 >> v2:172.21.15.159:3300/0 conn(0x7feff80b4a50 msgr2=0x7feff815b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feff7e00700 1 --2- 172.21.15.12:0/2553600001 >> v2:172.21.15.159:3300/0 conn(0x7feff80b4a50 0x7feff815b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 >> v1:172.21.15.159:6789/0 conn(0x7feff815af30 legacy=0x7feff806d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff806e150 con 0x7feff806da80 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.368+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fefe4001670 con 0x7feff806da80 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 >> v1:172.21.15.12:6789/0 conn(0x7feff806da80 legacy=0x7feff80b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feff7e00700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefe0003a30 0x7fefe0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feff7e00700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefe0004850 0x7fefe0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feff7e00700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7fefe000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff806faf0 con 0x7fefe0005210 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe000c110 con 0x7fefe0003a30 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe000c2d0 con 0x7fefe0004850 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feffe600700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7fefe000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feffdc00700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefe0003a30 0x7fefe0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feffd200700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefe0004850 0x7fefe0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feffe600700 1 -- 172.21.15.12:0/2553600001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefe0004850 msgr2=0x7fefe0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feffe600700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefe0004850 0x7fefe0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feffe600700 1 -- 172.21.15.12:0/2553600001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefe0003a30 msgr2=0x7fefe0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feffe600700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefe0003a30 0x7fefe0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feffe600700 1 -- 172.21.15.12:0/2553600001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe00035b0 con 0x7fefe0005210 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.369+0000 7feffe600700 1 -- 172.21.15.12:0/2553600001 reap_dead start 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7feffe600700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7fefe000ba10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7feff0009fc0 tx=0x7fefe0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7126f11ca2a5040c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0003a20 con 0x7fefe0005210 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff000f040 con 0x7fefe0005210 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7feff7e00700 1 -- 172.21.15.12:0/2553600001 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0003de0 con 0x7fefe0005210 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7ff000600700 1 -- 172.21.15.12:0/2553600001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 msgr2=0x7fefe000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7ff000600700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7fefe000ba10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7feff0009fc0 tx=0x7fefe0005880 comp rx=0 tx=0).stop 2024-03-29T10:06:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7ff000600700 1 -- 172.21.15.12:0/2553600001 shutdown_connections 2024-03-29T10:06:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7ff000600700 1 --2- 172.21.15.12:0/2553600001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7fefe000ba10 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7ff000600700 1 -- 172.21.15.12:0/2553600001 >> 172.21.15.12:0/2553600001 conn(0x7feff8071fb0 msgr2=0x7feff806a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7ff000600700 1 -- 172.21.15.12:0/2553600001 shutdown_connections 2024-03-29T10:06:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.370+0000 7ff000600700 1 -- 172.21.15.12:0/2553600001 wait complete. 2024-03-29T10:06:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.372+0000 7ff000600700 1 Processor -- start 2024-03-29T10:06:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.372+0000 7ff000600700 1 -- start start 2024-03-29T10:06:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.373+0000 7ff000600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7feff81577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.373+0000 7ff000600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff806b7d0 0x7feff806a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.373+0000 7ff000600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff806bc50 0x7feff806acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.373+0000 7ff000600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff80cba10 con 0x7fefe0005210 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.373+0000 7ff000600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff80cce60 con 0x7feff806b7d0 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.373+0000 7ff000600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff8157d10 con 0x7feff806bc50 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feffdc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7feff81577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feffdc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7feff81577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45042/0 (socket says 172.21.15.12:45042) 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feffdc00700 1 -- 172.21.15.12:0/4154022921 learned_addr learned my addr 172.21.15.12:0/4154022921 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feffdc00700 1 -- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff806bc50 msgr2=0x7feff806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feffdc00700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff806bc50 0x7feff806acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feffdc00700 1 -- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff806b7d0 msgr2=0x7feff806a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feffdc00700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff806b7d0 0x7feff806a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feffdc00700 1 -- 172.21.15.12:0/4154022921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff80cd0b0 con 0x7fefe0005210 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feffdc00700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7feff81577d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fefec00f780 tx=0x7fefec00fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feff5600700 1 -- 172.21.15.12:0/4154022921 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec0052e0 con 0x7fefe0005210 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feff5600700 1 -- 172.21.15.12:0/4154022921 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec005460 con 0x7fefe0005210 2024-03-29T10:06:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7feff5600700 1 -- 172.21.15.12:0/4154022921 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec0038c0 con 0x7fefe0005210 2024-03-29T10:06:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.374+0000 7ff000600700 1 -- 172.21.15.12:0/4154022921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff80cd390 con 0x7fefe0005210 2024-03-29T10:06:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.375+0000 7ff000600700 1 -- 172.21.15.12:0/4154022921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff80cd950 con 0x7fefe0005210 2024-03-29T10:06:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.376+0000 7ff000600700 1 -- 172.21.15.12:0/4154022921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff806fc30 con 0x7fefe0005210 2024-03-29T10:06:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.377+0000 7feff5600700 1 -- 172.21.15.12:0/4154022921 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fefec003a20 con 0x7fefe0005210 2024-03-29T10:06:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.377+0000 7feff5600700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefd806adb0 0x7fefd806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.378+0000 7feff5600700 1 -- 172.21.15.12:0/4154022921 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fefec089280 con 0x7fefe0005210 2024-03-29T10:06:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.378+0000 7feffd200700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefd806adb0 0x7fefd806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.379+0000 7feffd200700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefd806adb0 0x7fefd806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7feff815ee80 tx=0x7fefe4001d10 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.382+0000 7feff5600700 1 -- 172.21.15.12:0/4154022921 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fefec054430 con 0x7fefe0005210 2024-03-29T10:06:59.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.602+0000 7ff000600700 1 -- 172.21.15.12:0/4154022921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7feff8067720 con 0x7fefe0005210 2024-03-29T10:06:59.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.606+0000 7feff5600700 1 -- 172.21.15.12:0/4154022921 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v13) v1 ==== 115+0+0 (secure 0 0 0) 0x7fefec057a50 con 0x7fefe0005210 2024-03-29T10:06:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 -- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefd806adb0 msgr2=0x7fefd806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefd806adb0 0x7fefd806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7feff815ee80 tx=0x7fefe4001d10 comp rx=0 tx=0).stop 2024-03-29T10:06:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 -- 172.21.15.12:0/4154022921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 msgr2=0x7feff81577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7feff81577d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fefec00f780 tx=0x7fefec00fb40 comp rx=0 tx=0).stop 2024-03-29T10:06:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 -- 172.21.15.12:0/4154022921 shutdown_connections 2024-03-29T10:06:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefd806adb0 0x7fefd806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff806bc50 0x7feff806acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe0005210 0x7feff81577d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 --2- 172.21.15.12:0/4154022921 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff806b7d0 0x7feff806a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 -- 172.21.15.12:0/4154022921 >> 172.21.15.12:0/4154022921 conn(0x7feff8071fb0 msgr2=0x7feff80b36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:59.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 -- 172.21.15.12:0/4154022921 shutdown_connections 2024-03-29T10:06:59.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.609+0000 7ff000600700 1 -- 172.21.15.12:0/4154022921 wait complete. 2024-03-29T10:06:59.626 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T10:06:59.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.827+0000 7f1c0c200700 1 Processor -- start 2024-03-29T10:06:59.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.828+0000 7f1c0c200700 1 -- start start 2024-03-29T10:06:59.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.829+0000 7f1c0c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c040cbb30 0x7f1c040cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.829+0000 7f1c0c200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1c040cd4a0 0x7f1c0406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.829+0000 7f1c0c200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c040cc450 con 0x7f1c040cd020 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.829+0000 7f1c0c200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c040cdae0 con 0x7f1c040cd4a0 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.829+0000 7f1c0c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040cddc0 con 0x7f1c040cbb30 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.829+0000 7f1c09800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c040cbb30 0x7f1c040cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.829+0000 7f1c0a200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1c040cd4a0 0x7f1c0406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.829+0000 7f1c08e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f1c040cd020 0x7f1c040b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:40960/0 (socket says 172.21.15.12:40960) 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.829+0000 7f1c08e00700 1 -- 172.21.15.12:0/3781516397 learned_addr learned my addr 172.21.15.12:0/3781516397 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.830+0000 7f1c0a200700 1 -- 172.21.15.12:0/3781516397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c040cbb30 msgr2=0x7f1c040cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.830+0000 7f1c0a200700 1 --2- 172.21.15.12:0/3781516397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c040cbb30 0x7f1c040cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.830+0000 7f1c0a200700 1 -- 172.21.15.12:0/3781516397 >> v1:172.21.15.159:6789/0 conn(0x7f1c040cd020 legacy=0x7f1c040b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.830+0000 7f1c0a200700 1 -- 172.21.15.12:0/3781516397 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c040ce0a0 con 0x7f1c040cd4a0 2024-03-29T10:06:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.830+0000 7f1c0a200700 1 --2- 172.21.15.12:0/3781516397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c040cd4a0 0x7f1c0406e420 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009d90 tx=0x7f1bfc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b42c37ceb559d4d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.830+0000 7f1c03e00700 1 -- 172.21.15.12:0/3781516397 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc00e070 con 0x7f1c040cd4a0 2024-03-29T10:06:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.831+0000 7f1c03e00700 1 -- 172.21.15.12:0/3781516397 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc00bb20 con 0x7f1c040cd4a0 2024-03-29T10:06:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.831+0000 7f1c03e00700 1 -- 172.21.15.12:0/3781516397 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc012430 con 0x7f1c040cd4a0 2024-03-29T10:06:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.831+0000 7f1c0c200700 1 -- 172.21.15.12:0/3781516397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c040cd4a0 msgr2=0x7f1c0406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.831+0000 7f1c0c200700 1 --2- 172.21.15.12:0/3781516397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c040cd4a0 0x7f1c0406e420 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009d90 tx=0x7f1bfc00b330 comp rx=0 tx=0).stop 2024-03-29T10:06:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.831+0000 7f1c0c200700 1 -- 172.21.15.12:0/3781516397 shutdown_connections 2024-03-29T10:06:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.831+0000 7f1c0c200700 1 --2- 172.21.15.12:0/3781516397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c040cd4a0 0x7f1c0406e420 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.831+0000 7f1c0c200700 1 --2- 172.21.15.12:0/3781516397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c040cbb30 0x7f1c040cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.831+0000 7f1c0c200700 1 -- 172.21.15.12:0/3781516397 >> 172.21.15.12:0/3781516397 conn(0x7f1c040b0d70 msgr2=0x7f1c040b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:06:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.831+0000 7f1c0c200700 1 -- 172.21.15.12:0/3781516397 shutdown_connections 2024-03-29T10:06:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.831+0000 7f1c0c200700 1 -- 172.21.15.12:0/3781516397 wait complete. 2024-03-29T10:06:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.833+0000 7f1c0c200700 1 Processor -- start 2024-03-29T10:06:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.833+0000 7f1c0c200700 1 -- start start 2024-03-29T10:06:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.834+0000 7f1c0c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c040cbb30 0x7f1c041f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.834+0000 7f1c0c200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c040cd020 0x7f1c040709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.834+0000 7f1c0c200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c040cd4a0 0x7f1c040ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.834+0000 7f1c0c200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04071480 con 0x7f1c040cd020 2024-03-29T10:06:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.834+0000 7f1c0c200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04071760 con 0x7f1c040cd4a0 2024-03-29T10:06:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.834+0000 7f1c0c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04071a40 con 0x7f1c040cbb30 2024-03-29T10:06:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.834+0000 7f1c08e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c040cd020 0x7f1c040709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.834+0000 7f1c08e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c040cd020 0x7f1c040709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45044/0 (socket says 172.21.15.12:45044) 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.834+0000 7f1c08e00700 1 -- 172.21.15.12:0/3116645235 learned_addr learned my addr 172.21.15.12:0/3116645235 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c0a200700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c040cd4a0 0x7f1c040ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c08e00700 1 -- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c040cbb30 msgr2=0x7f1c041f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c08e00700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c040cbb30 0x7f1c041f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c08e00700 1 -- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c040cd4a0 msgr2=0x7f1c040ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c08e00700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c040cd4a0 0x7f1c040ad0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c08e00700 1 -- 172.21.15.12:0/3116645235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc009a70 con 0x7f1c040cd020 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c0a200700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c040cd4a0 0x7f1c040ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c08e00700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c040cd020 0x7f1c040709c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000c480 tx=0x7f1bf000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c01600700 1 -- 172.21.15.12:0/3116645235 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0005d70 con 0x7f1c040cd020 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c01600700 1 -- 172.21.15.12:0/3116645235 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf00070d0 con 0x7f1c040cd020 2024-03-29T10:06:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c01600700 1 -- 172.21.15.12:0/3116645235 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0007440 con 0x7f1c040cd020 2024-03-29T10:06:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c0c200700 1 -- 172.21.15.12:0/3116645235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c04071d20 con 0x7f1c040cd020 2024-03-29T10:06:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.835+0000 7f1c0c200700 1 -- 172.21.15.12:0/3116645235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c041f3330 con 0x7f1c040cd020 2024-03-29T10:06:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.836+0000 7f1c0c200700 1 -- 172.21.15.12:0/3116645235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c04067720 con 0x7f1c040cd020 2024-03-29T10:06:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.837+0000 7f1c01600700 1 -- 172.21.15.12:0/3116645235 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1bf00075a0 con 0x7f1c040cd020 2024-03-29T10:06:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.838+0000 7f1c01600700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be406adb0 0x7f1be406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:06:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.838+0000 7f1c01600700 1 -- 172.21.15.12:0/3116645235 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f1bf0088810 con 0x7f1c040cd020 2024-03-29T10:06:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.839+0000 7f1c09800700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be406adb0 0x7f1be406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:06:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.839+0000 7f1c09800700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be406adb0 0x7f1be406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8006b20 tx=0x7f1bf8006ab0 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:06:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:06:59.843+0000 7f1c01600700 1 -- 172.21.15.12:0/3116645235 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1bf0053a50 con 0x7f1c040cd020 2024-03-29T10:07:00.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.058+0000 7f1c0c200700 1 -- 172.21.15.12:0/3116645235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1c0415bda0 con 0x7f1c040cd020 2024-03-29T10:07:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.063+0000 7f1c01600700 1 -- 172.21.15.12:0/3116645235 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1bf0057070 con 0x7f1c040cd020 2024-03-29T10:07:00.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 -- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be406adb0 msgr2=0x7f1be406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be406adb0 0x7f1be406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8006b20 tx=0x7f1bf8006ab0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 -- 172.21.15.12:0/3116645235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c040cd020 msgr2=0x7f1c040709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c040cd020 0x7f1c040709c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000c480 tx=0x7f1bf000c840 comp rx=0 tx=0).stop 2024-03-29T10:07:00.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 -- 172.21.15.12:0/3116645235 shutdown_connections 2024-03-29T10:07:00.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1be406adb0 0x7f1be406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c040cd4a0 0x7f1c040ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c040cbb30 0x7f1c041f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 --2- 172.21.15.12:0/3116645235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c040cd020 0x7f1c040709c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 -- 172.21.15.12:0/3116645235 >> 172.21.15.12:0/3116645235 conn(0x7f1c040b0d70 msgr2=0x7f1c0406e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:00.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 -- 172.21.15.12:0/3116645235 shutdown_connections 2024-03-29T10:07:00.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.065+0000 7f1c0c200700 1 -- 172.21.15.12:0/3116645235 wait complete. 2024-03-29T10:07:00.083 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T10:07:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.283+0000 7fed01800700 1 Processor -- start 2024-03-29T10:07:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.284+0000 7fed01800700 1 -- start start 2024-03-29T10:07:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.285+0000 7fed01800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fecfc0d2540 0x7fecfc1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.285+0000 7fed01800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecfc161420 con 0x7fecfc0d16a0 2024-03-29T10:07:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.285+0000 7fed01800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecfc161700 con 0x7fecfc15d9d0 2024-03-29T10:07:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.285+0000 7fed01800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecfc0d4340 con 0x7fecfc0d2540 2024-03-29T10:07:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.285+0000 7fecfa000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fecfc0d16a0 0x7fecfc0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41438/0 (socket says 172.21.15.12:41438) 2024-03-29T10:07:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.285+0000 7fecfa000700 1 -- 172.21.15.12:0/387301839 learned_addr learned my addr 172.21.15.12:0/387301839 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.285+0000 7fecfb400700 1 --2- 172.21.15.12:0/387301839 >> v2:172.21.15.159:3300/0 conn(0x7fecfc0d2540 0x7fecfc1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804117308 0 0) 0x7fecfc161420 con 0x7fecfc0d16a0 2024-03-29T10:07:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece00035b0 con 0x7fecfc0d16a0 2024-03-29T10:07:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1898400025 0 0) 0x7fece00035b0 con 0x7fecfc0d16a0 2024-03-29T10:07:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 >> v2:172.21.15.159:3300/0 conn(0x7fecfc0d2540 msgr2=0x7fecfc1687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 --2- 172.21.15.12:0/387301839 >> v2:172.21.15.159:3300/0 conn(0x7fecfc0d2540 0x7fecfc1687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 >> v1:172.21.15.159:6789/0 conn(0x7fecfc15d9d0 legacy=0x7fecfc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc1619e0 con 0x7fecfc0d16a0 2024-03-29T10:07:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fece8005120 con 0x7fecfc0d16a0 2024-03-29T10:07:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 >> v1:172.21.15.12:6789/0 conn(0x7fecfc0d16a0 legacy=0x7fecfc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fece0003a00 0x7fece0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fece0004400 0x7fece00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fece000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece000c080 con 0x7fece0005170 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece000c1f0 con 0x7fece0003a00 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece000c3b0 con 0x7fece0004400 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecfb400700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fece000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecfaa00700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fece0003a00 0x7fece0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.286+0000 7fecfa000700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fece0004400 0x7fece00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.287+0000 7fecfb400700 1 -- 172.21.15.12:0/387301839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fece0004400 msgr2=0x7fece00048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.287+0000 7fecfb400700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fece0004400 0x7fece00048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.287+0000 7fecfb400700 1 -- 172.21.15.12:0/387301839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fece0003a00 msgr2=0x7fece0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.287+0000 7fecfb400700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fece0003a00 0x7fece0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.287+0000 7fecfb400700 1 -- 172.21.15.12:0/387301839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece00035b0 con 0x7fece0005170 2024-03-29T10:07:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.287+0000 7fecfb400700 1 -- 172.21.15.12:0/387301839 reap_dead start 2024-03-29T10:07:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.287+0000 7fecfb400700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fece000ba10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009f90 tx=0x7fece0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7ef77e555c0432 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.287+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4003a20 con 0x7fece0005170 2024-03-29T10:07:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.287+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf400f040 con 0x7fece0005170 2024-03-29T10:07:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.287+0000 7fecf9600700 1 -- 172.21.15.12:0/387301839 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4003de0 con 0x7fece0005170 2024-03-29T10:07:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.288+0000 7fed01800700 1 -- 172.21.15.12:0/387301839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 msgr2=0x7fece000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.288+0000 7fed01800700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fece000ba10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009f90 tx=0x7fece0005880 comp rx=0 tx=0).stop 2024-03-29T10:07:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.288+0000 7fed01800700 1 -- 172.21.15.12:0/387301839 shutdown_connections 2024-03-29T10:07:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.288+0000 7fed01800700 1 --2- 172.21.15.12:0/387301839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fece000ba10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.288+0000 7fed01800700 1 -- 172.21.15.12:0/387301839 >> 172.21.15.12:0/387301839 conn(0x7fecfc08aff0 msgr2=0x7fecfc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.288+0000 7fed01800700 1 -- 172.21.15.12:0/387301839 shutdown_connections 2024-03-29T10:07:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.288+0000 7fed01800700 1 -- 172.21.15.12:0/387301839 wait complete. 2024-03-29T10:07:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.290+0000 7fed01800700 1 Processor -- start 2024-03-29T10:07:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.291+0000 7fed01800700 1 -- start start 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fed01800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fecfc06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fed01800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecfc1504f0 0x7fecfc150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fed01800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecfc0ca6a0 0x7fecfc0cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fed01800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0b7440 con 0x7fece0005170 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fed01800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc150eb0 con 0x7fecfc0ca6a0 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fed01800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0cb420 con 0x7fecfc1504f0 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fecfc06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fecfc06cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45058/0 (socket says 172.21.15.12:45058) 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfaa00700 1 -- 172.21.15.12:0/619056547 learned_addr learned my addr 172.21.15.12:0/619056547 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfa000700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecfc1504f0 0x7fecfc150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfb400700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecfc0ca6a0 0x7fecfc0cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfaa00700 1 -- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecfc1504f0 msgr2=0x7fecfc150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfaa00700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecfc1504f0 0x7fecfc150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfaa00700 1 -- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecfc0ca6a0 msgr2=0x7fecfc0cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfaa00700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecfc0ca6a0 0x7fecfc0cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfaa00700 1 -- 172.21.15.12:0/619056547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0cb700 con 0x7fece0005170 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfb400700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecfc0ca6a0 0x7fecfc0cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.292+0000 7fecfa000700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecfc1504f0 0x7fecfc150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.293+0000 7fecfaa00700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fecfc06cf90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fecec00f780 tx=0x7fecec00fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.293+0000 7fecf2a00700 1 -- 172.21.15.12:0/619056547 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0052e0 con 0x7fece0005170 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.293+0000 7fecf2a00700 1 -- 172.21.15.12:0/619056547 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec005460 con 0x7fece0005170 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.293+0000 7fed01800700 1 -- 172.21.15.12:0/619056547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc0cb9e0 con 0x7fece0005170 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.293+0000 7fecf2a00700 1 -- 172.21.15.12:0/619056547 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0038c0 con 0x7fece0005170 2024-03-29T10:07:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.293+0000 7fed01800700 1 -- 172.21.15.12:0/619056547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc0cb060 con 0x7fece0005170 2024-03-29T10:07:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.294+0000 7fed01800700 1 -- 172.21.15.12:0/619056547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecfc0cb1f0 con 0x7fece0005170 2024-03-29T10:07:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.295+0000 7fecf2a00700 1 -- 172.21.15.12:0/619056547 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fecec003a20 con 0x7fece0005170 2024-03-29T10:07:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.296+0000 7fecf2a00700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecdc06adb0 0x7fecdc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.296+0000 7fecf2a00700 1 -- 172.21.15.12:0/619056547 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fecec089370 con 0x7fece0005170 2024-03-29T10:07:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.297+0000 7fecfa000700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecdc06adb0 0x7fecdc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.297+0000 7fecfa000700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecdc06adb0 0x7fecdc06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fece8005250 tx=0x7fece8009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.301+0000 7fecf2a00700 1 -- 172.21.15.12:0/619056547 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fecec054480 con 0x7fece0005170 2024-03-29T10:07:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.524+0000 7fed01800700 1 -- 172.21.15.12:0/619056547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fecfc067720 con 0x7fece0005170 2024-03-29T10:07:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.528+0000 7fecf2a00700 1 -- 172.21.15.12:0/619056547 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7fecec057aa0 con 0x7fece0005170 2024-03-29T10:07:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.530+0000 7fed01800700 1 -- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecdc06adb0 msgr2=0x7fecdc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.530+0000 7fed01800700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecdc06adb0 0x7fecdc06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fece8005250 tx=0x7fece8009040 comp rx=0 tx=0).stop 2024-03-29T10:07:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.530+0000 7fed01800700 1 -- 172.21.15.12:0/619056547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 msgr2=0x7fecfc06cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.530+0000 7fed01800700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fecfc06cf90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fecec00f780 tx=0x7fecec00fb40 comp rx=0 tx=0).stop 2024-03-29T10:07:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.530+0000 7fed01800700 1 -- 172.21.15.12:0/619056547 shutdown_connections 2024-03-29T10:07:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.530+0000 7fed01800700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecdc06adb0 0x7fecdc06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.530+0000 7fed01800700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecfc0ca6a0 0x7fecfc0cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.530+0000 7fed01800700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fece0005170 0x7fecfc06cf90 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.530+0000 7fed01800700 1 --2- 172.21.15.12:0/619056547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecfc1504f0 0x7fecfc150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.530+0000 7fed01800700 1 -- 172.21.15.12:0/619056547 >> 172.21.15.12:0/619056547 conn(0x7fecfc08aff0 msgr2=0x7fecfc0b5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.531+0000 7fed01800700 1 -- 172.21.15.12:0/619056547 shutdown_connections 2024-03-29T10:07:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.531+0000 7fed01800700 1 -- 172.21.15.12:0/619056547 wait complete. 2024-03-29T10:07:00.548 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-03-29T10:07:00.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.747+0000 7f2505200700 1 Processor -- start 2024-03-29T10:07:00.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.747+0000 7f2505200700 1 -- start start 2024-03-29T10:07:00.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.748+0000 7f2505200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f250015d640 0x7f25000d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.748+0000 7f2505200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f25000ceec0 0x7f25001612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.748+0000 7f2505200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2500161800 0x7f25000cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.748+0000 7f2505200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2500161c90 con 0x7f2500161800 2024-03-29T10:07:00.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.748+0000 7f2505200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25000d30d0 con 0x7f25000ceec0 2024-03-29T10:07:00.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.748+0000 7f2505200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25000b6840 con 0x7f250015d640 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.748+0000 7f24fe800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2500161800 0x7f25000cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.749+0000 7f24fe800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2500161800 0x7f25000cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45070/0 (socket says 172.21.15.12:45070) 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.749+0000 7f24fe800700 1 -- 172.21.15.12:0/2218819765 learned_addr learned my addr 172.21.15.12:0/2218819765 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.749+0000 7f24fde00700 1 --2- 172.21.15.12:0/2218819765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f250015d640 0x7f25000d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.749+0000 7f24fd400700 1 --2- 172.21.15.12:0/2218819765 >> v2:172.21.15.159:3300/0 conn(0x7f25000ceec0 0x7f25001612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.749+0000 7f24fe800700 1 -- 172.21.15.12:0/2218819765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f250015d640 msgr2=0x7f25000d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.749+0000 7f24fe800700 1 --2- 172.21.15.12:0/2218819765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f250015d640 0x7f25000d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.749+0000 7f24fe800700 1 -- 172.21.15.12:0/2218819765 >> v2:172.21.15.159:3300/0 conn(0x7f25000ceec0 msgr2=0x7f25001612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.749+0000 7f24fe800700 1 --2- 172.21.15.12:0/2218819765 >> v2:172.21.15.159:3300/0 conn(0x7f25000ceec0 0x7f25001612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.749+0000 7f24fe800700 1 -- 172.21.15.12:0/2218819765 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25000d2880 con 0x7f2500161800 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.749+0000 7f24fe800700 1 --2- 172.21.15.12:0/2218819765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2500161800 0x7f25000cdba0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f250008c1c0 tx=0x7f24f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5263820ebca741 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f24fca00700 1 -- 172.21.15.12:0/2218819765 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f800e070 con 0x7f2500161800 2024-03-29T10:07:00.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f24fca00700 1 -- 172.21.15.12:0/2218819765 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f800b9c0 con 0x7f2500161800 2024-03-29T10:07:00.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f24fca00700 1 -- 172.21.15.12:0/2218819765 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f800bd70 con 0x7f2500161800 2024-03-29T10:07:00.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f2505200700 1 -- 172.21.15.12:0/2218819765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2500161800 msgr2=0x7f25000cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f2505200700 1 --2- 172.21.15.12:0/2218819765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2500161800 0x7f25000cdba0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f250008c1c0 tx=0x7f24f800b330 comp rx=0 tx=0).stop 2024-03-29T10:07:00.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f2505200700 1 -- 172.21.15.12:0/2218819765 shutdown_connections 2024-03-29T10:07:00.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f2505200700 1 --2- 172.21.15.12:0/2218819765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2500161800 0x7f25000cdba0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f2505200700 1 --2- 172.21.15.12:0/2218819765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f250015d640 0x7f25000d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f2505200700 1 --2- 172.21.15.12:0/2218819765 >> v2:172.21.15.159:3300/0 conn(0x7f25000ceec0 0x7f25001612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f2505200700 1 -- 172.21.15.12:0/2218819765 >> 172.21.15.12:0/2218819765 conn(0x7f250008aff0 msgr2=0x7f25000d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:00.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f2505200700 1 -- 172.21.15.12:0/2218819765 shutdown_connections 2024-03-29T10:07:00.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.750+0000 7f2505200700 1 -- 172.21.15.12:0/2218819765 wait complete. 2024-03-29T10:07:00.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.752+0000 7f2505200700 1 Processor -- start 2024-03-29T10:07:00.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.752+0000 7f2505200700 1 -- start start 2024-03-29T10:07:00.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f2505200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25000ceec0 0x7f2500157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f2505200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f250015d640 0x7f25000c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f2505200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2500161800 0x7f250015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f2505200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25000c8df0 con 0x7f25000ceec0 2024-03-29T10:07:00.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f2505200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2500157fe0 con 0x7f250015d640 2024-03-29T10:07:00.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f2505200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25000c81e0 con 0x7f2500161800 2024-03-29T10:07:00.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f24fde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25000ceec0 0x7f2500157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f24fde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25000ceec0 0x7f2500157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45084/0 (socket says 172.21.15.12:45084) 2024-03-29T10:07:00.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f24fde00700 1 -- 172.21.15.12:0/2392026998 learned_addr learned my addr 172.21.15.12:0/2392026998 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:00.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f24fe800700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2500161800 0x7f250015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.753+0000 7f24fd400700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f250015d640 0x7f25000c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24fde00700 1 -- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2500161800 msgr2=0x7f250015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24fde00700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2500161800 0x7f250015f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24fde00700 1 -- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f250015d640 msgr2=0x7f25000c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24fde00700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f250015d640 0x7f25000c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24fde00700 1 -- 172.21.15.12:0/2392026998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f8009b30 con 0x7f25000ceec0 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24fe800700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2500161800 0x7f250015f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24fd400700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f250015d640 0x7f25000c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24fde00700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25000ceec0 0x7f2500157aa0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f24f000e810 tx=0x7f24f000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24f6000700 1 -- 172.21.15.12:0/2392026998 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0013610 con 0x7f25000ceec0 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24f6000700 1 -- 172.21.15.12:0/2392026998 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f0013770 con 0x7f25000ceec0 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f2505200700 1 -- 172.21.15.12:0/2392026998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25000c84c0 con 0x7f25000ceec0 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f24f6000700 1 -- 172.21.15.12:0/2392026998 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0013b20 con 0x7f25000ceec0 2024-03-29T10:07:00.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.754+0000 7f2505200700 1 -- 172.21.15.12:0/2392026998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25000c8a80 con 0x7f25000ceec0 2024-03-29T10:07:00.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.755+0000 7f2505200700 1 -- 172.21.15.12:0/2392026998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2500067720 con 0x7f25000ceec0 2024-03-29T10:07:00.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.756+0000 7f24f6000700 1 -- 172.21.15.12:0/2392026998 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f24f0013c80 con 0x7f25000ceec0 2024-03-29T10:07:00.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.758+0000 7f24f6000700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24e006adb0 0x7f24e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:00.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.759+0000 7f24f6000700 1 -- 172.21.15.12:0/2392026998 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f24f0089210 con 0x7f25000ceec0 2024-03-29T10:07:00.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.759+0000 7f24fd400700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24e006adb0 0x7f24e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:00.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.760+0000 7f24fd400700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24e006adb0 0x7f24e006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f25000cd900 tx=0x7f24ec008040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:00.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:00.763+0000 7f24f6000700 1 -- 172.21.15.12:0/2392026998 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f24f0054210 con 0x7f25000ceec0 2024-03-29T10:07:01.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.060+0000 7f2505200700 1 -- 172.21.15.12:0/2392026998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f25000c90d0 con 0x7f25000ceec0 2024-03-29T10:07:01.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.141+0000 7f24f6000700 1 -- 172.21.15.12:0/2392026998 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v451) v1 ==== 158+0+0 (secure 0 0 0) 0x7f24f0057830 con 0x7f25000ceec0 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.143+0000 7f2505200700 1 -- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24e006adb0 msgr2=0x7f24e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.143+0000 7f2505200700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24e006adb0 0x7f24e006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f25000cd900 tx=0x7f24ec008040 comp rx=0 tx=0).stop 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.143+0000 7f2505200700 1 -- 172.21.15.12:0/2392026998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25000ceec0 msgr2=0x7f2500157aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.143+0000 7f2505200700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25000ceec0 0x7f2500157aa0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f24f000e810 tx=0x7f24f000ebd0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.144+0000 7f2505200700 1 -- 172.21.15.12:0/2392026998 shutdown_connections 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.144+0000 7f2505200700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f24e006adb0 0x7f24e006d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.144+0000 7f2505200700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2500161800 0x7f250015f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.144+0000 7f2505200700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25000ceec0 0x7f2500157aa0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.144+0000 7f2505200700 1 --2- 172.21.15.12:0/2392026998 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f250015d640 0x7f25000c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.144+0000 7f2505200700 1 -- 172.21.15.12:0/2392026998 >> 172.21.15.12:0/2392026998 conn(0x7f250008aff0 msgr2=0x7f250015aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.144+0000 7f2505200700 1 -- 172.21.15.12:0/2392026998 shutdown_connections 2024-03-29T10:07:01.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.144+0000 7f2505200700 1 -- 172.21.15.12:0/2392026998 wait complete. 2024-03-29T10:07:01.161 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2024-03-29T10:07:01.162 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2024-03-29T10:07:01.162 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T10:07:01.162 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-03-29T10:07:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.361+0000 7f166aa00700 1 Processor -- start 2024-03-29T10:07:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.361+0000 7f166aa00700 1 -- start start 2024-03-29T10:07:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.362+0000 7f166aa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1664160f10 0x7f16640d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.362+0000 7f166aa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640ceda0 con 0x7f1664161590 2024-03-29T10:07:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.362+0000 7f166aa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16641612f0 con 0x7f16640cf230 2024-03-29T10:07:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.362+0000 7f166aa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16640ce470 con 0x7f1664160f10 2024-03-29T10:07:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.362+0000 7f1668a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1664161590 0x7f16640b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41450/0 (socket says 172.21.15.12:41450) 2024-03-29T10:07:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.362+0000 7f1668a00700 1 -- 172.21.15.12:0/1126281161 learned_addr learned my addr 172.21.15.12:0/1126281161 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1663e00700 1 --2- 172.21.15.12:0/1126281161 >> v2:172.21.15.159:3300/0 conn(0x7f1664160f10 0x7f16640d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488882950 0 0) 0x7f16640ceda0 con 0x7f1664161590 2024-03-29T10:07:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16480035b0 con 0x7f1664161590 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284738957 0 0) 0x7f16641612f0 con 0x7f16640cf230 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16640ceda0 con 0x7f16640cf230 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 344274857 0 0) 0x7f16480035b0 con 0x7f1664161590 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 >> v2:172.21.15.159:3300/0 conn(0x7f1664160f10 msgr2=0x7f16640d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 --2- 172.21.15.12:0/1126281161 >> v2:172.21.15.159:3300/0 conn(0x7f1664160f10 0x7f16640d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 >> v1:172.21.15.159:6789/0 conn(0x7f16640cf230 legacy=0x7f16640d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640ce750 con 0x7f1664161590 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f165c0019f0 con 0x7f1664161590 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 >> v1:172.21.15.12:6789/0 conn(0x7f1664161590 legacy=0x7f16640b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1648003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1648004850 0x7f1648004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1648005260 0x7f164800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640ce470 con 0x7f1648003be0 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164800c190 con 0x7f1648004850 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164800c350 con 0x7f1648005260 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.363+0000 7f1663e00700 1 --2- 172.21.15.12:0/1126281161 >> v2:172.21.15.159:3300/0 conn(0x7f1664160f10 0x7f16640d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.364+0000 7f1663e00700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1648003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.364+0000 7f1663400700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1648004850 0x7f1648004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.364+0000 7f1668a00700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1648005260 0x7f164800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.364+0000 7f1663e00700 1 -- 172.21.15.12:0/1126281161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1648005260 msgr2=0x7f164800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.364+0000 7f1663e00700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1648005260 0x7f164800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.364+0000 7f1663e00700 1 -- 172.21.15.12:0/1126281161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1648004850 msgr2=0x7f1648004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.364+0000 7f1663e00700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1648004850 0x7f1648004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.364+0000 7f1663e00700 1 -- 172.21.15.12:0/1126281161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16480035b0 con 0x7f1648003be0 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.364+0000 7f1668a00700 1 -- 172.21.15.12:0/1126281161 reap_dead start 2024-03-29T10:07:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.364+0000 7f1663e00700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1648003fe0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1654009d10 tx=0x7f1648005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfef8a210f278a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.365+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654009e60 con 0x7f1648003be0 2024-03-29T10:07:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.365+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165400e9a0 con 0x7f1648003be0 2024-03-29T10:07:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.365+0000 7f1662a00700 1 -- 172.21.15.12:0/1126281161 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165400ed30 con 0x7f1648003be0 2024-03-29T10:07:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.365+0000 7f166aa00700 1 -- 172.21.15.12:0/1126281161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 msgr2=0x7f1648003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.365+0000 7f166aa00700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1648003fe0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1654009d10 tx=0x7f1648005880 comp rx=0 tx=0).stop 2024-03-29T10:07:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.365+0000 7f166aa00700 1 -- 172.21.15.12:0/1126281161 shutdown_connections 2024-03-29T10:07:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.365+0000 7f166aa00700 1 --2- 172.21.15.12:0/1126281161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1648003fe0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.365+0000 7f166aa00700 1 -- 172.21.15.12:0/1126281161 >> 172.21.15.12:0/1126281161 conn(0x7f166408aff0 msgr2=0x7f16640d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.365+0000 7f166aa00700 1 -- 172.21.15.12:0/1126281161 shutdown_connections 2024-03-29T10:07:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.365+0000 7f166aa00700 1 -- 172.21.15.12:0/1126281161 wait complete. 2024-03-29T10:07:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.367+0000 7f166aa00700 1 Processor -- start 2024-03-29T10:07:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.367+0000 7f166aa00700 1 -- start start 2024-03-29T10:07:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.368+0000 7f166aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1664159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.368+0000 7f166aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1664153bb0 0x7f1664154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.368+0000 7f166aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16641574d0 0x7f1664157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.368+0000 7f166aa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640b6d40 con 0x7f1648003be0 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.368+0000 7f166aa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640b6150 con 0x7f16641574d0 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.368+0000 7f166aa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16640b63a0 con 0x7f1664153bb0 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.368+0000 7f1663e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1664159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.368+0000 7f1663e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1664159610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45098/0 (socket says 172.21.15.12:45098) 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.368+0000 7f1663e00700 1 -- 172.21.15.12:0/2215138841 learned_addr learned my addr 172.21.15.12:0/2215138841 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f1663400700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1664153bb0 0x7f1664154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f1668a00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16641574d0 0x7f1664157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f1663e00700 1 -- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1664153bb0 msgr2=0x7f1664154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f1663e00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1664153bb0 0x7f1664154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f1663e00700 1 -- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16641574d0 msgr2=0x7f1664157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f1663e00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16641574d0 0x7f1664157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f1663e00700 1 -- 172.21.15.12:0/2215138841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640b6680 con 0x7f1648003be0 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f1668a00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16641574d0 0x7f1664157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f1663e00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1664159610 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f165400e790 tx=0x7f16540047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f165be00700 1 -- 172.21.15.12:0/2215138841 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654003ab0 con 0x7f1648003be0 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f165be00700 1 -- 172.21.15.12:0/2215138841 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1654020070 con 0x7f1648003be0 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.370+0000 7f166aa00700 1 -- 172.21.15.12:0/2215138841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16640b6960 con 0x7f1648003be0 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.369+0000 7f165be00700 1 -- 172.21.15.12:0/2215138841 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654003e70 con 0x7f1648003be0 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.370+0000 7f166aa00700 1 -- 172.21.15.12:0/2215138841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16640b7160 con 0x7f1648003be0 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.371+0000 7f166aa00700 1 -- 172.21.15.12:0/2215138841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16640b6fb0 con 0x7f1648003be0 2024-03-29T10:07:01.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.372+0000 7f165be00700 1 -- 172.21.15.12:0/2215138841 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1654013470 con 0x7f1648003be0 2024-03-29T10:07:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.372+0000 7f165be00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f164406adb0 0x7f164406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.372+0000 7f165be00700 1 -- 172.21.15.12:0/2215138841 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f1654022070 con 0x7f1648003be0 2024-03-29T10:07:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.373+0000 7f1663400700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f164406adb0 0x7f164406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.373+0000 7f1663400700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f164406adb0 0x7f164406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1650000c00 tx=0x7f1650009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:01.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.377+0000 7f165be00700 1 -- 172.21.15.12:0/2215138841 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f16540598a0 con 0x7f1648003be0 2024-03-29T10:07:01.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:01.677+0000 7f166aa00700 1 -- 172.21.15.12:0/2215138841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f16640d4520 con 0x7f1648003be0 2024-03-29T10:07:02.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.147+0000 7f165be00700 1 -- 172.21.15.12:0/2215138841 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v452) v1 ==== 157+0+0 (secure 0 0 0) 0x7f1654009e60 con 0x7f1648003be0 2024-03-29T10:07:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 -- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f164406adb0 msgr2=0x7f164406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f164406adb0 0x7f164406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1650000c00 tx=0x7f1650009040 comp rx=0 tx=0).stop 2024-03-29T10:07:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 -- 172.21.15.12:0/2215138841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 msgr2=0x7f1664159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1664159610 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f165400e790 tx=0x7f16540047c0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 -- 172.21.15.12:0/2215138841 shutdown_connections 2024-03-29T10:07:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f164406adb0 0x7f164406d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16641574d0 0x7f1664157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1648003be0 0x7f1664159610 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 --2- 172.21.15.12:0/2215138841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1664153bb0 0x7f1664154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 -- 172.21.15.12:0/2215138841 >> 172.21.15.12:0/2215138841 conn(0x7f166408aff0 msgr2=0x7f16640cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:02.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 -- 172.21.15.12:0/2215138841 shutdown_connections 2024-03-29T10:07:02.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.150+0000 7f166aa00700 1 -- 172.21.15.12:0/2215138841 wait complete. 2024-03-29T10:07:02.168 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T10:07:02.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.370+0000 7f79fc600700 1 Processor -- start 2024-03-29T10:07:02.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.370+0000 7f79fc600700 1 -- start start 2024-03-29T10:07:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.371+0000 7f79fc600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79f40d16a0 0x7f79f40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.371+0000 7f79fc600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f79f40d2370 0x7f79f40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.371+0000 7f79fc600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f4161420 con 0x7f79f415d9d0 2024-03-29T10:07:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.371+0000 7f79fc600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f40d2820 con 0x7f79f40d2370 2024-03-29T10:07:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.371+0000 7f79fc600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f40d4340 con 0x7f79f40d16a0 2024-03-29T10:07:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.371+0000 7f79f9c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f79f415d9d0 0x7f79f40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41452/0 (socket says 172.21.15.12:41452) 2024-03-29T10:07:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.371+0000 7f79f9c00700 1 -- 172.21.15.12:0/3095061357 learned_addr learned my addr 172.21.15.12:0/3095061357 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.371+0000 7f79fa600700 1 --2- 172.21.15.12:0/3095061357 >> v2:172.21.15.12:3300/0 conn(0x7f79f40d2370 0x7f79f40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.371+0000 7f79f9200700 1 --2- 172.21.15.12:0/3095061357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79f40d16a0 0x7f79f40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:02.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.372+0000 7f79f3e00700 1 -- 172.21.15.12:0/3095061357 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262622233 0 0) 0x7f79f4161420 con 0x7f79f415d9d0 2024-03-29T10:07:02.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.372+0000 7f79f3e00700 1 -- 172.21.15.12:0/3095061357 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79dc0035b0 con 0x7f79f415d9d0 2024-03-29T10:07:02.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.372+0000 7f79fa600700 1 -- 172.21.15.12:0/3095061357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79f40d16a0 msgr2=0x7f79f40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:02.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.372+0000 7f79fa600700 1 --2- 172.21.15.12:0/3095061357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79f40d16a0 0x7f79f40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.372+0000 7f79fa600700 1 -- 172.21.15.12:0/3095061357 >> v1:172.21.15.12:6789/0 conn(0x7f79f415d9d0 legacy=0x7f79f40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:02.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.372+0000 7f79fa600700 1 -- 172.21.15.12:0/3095061357 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f4161700 con 0x7f79f40d2370 2024-03-29T10:07:02.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.372+0000 7f79fa600700 1 --2- 172.21.15.12:0/3095061357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d2370 0x7f79f40ce360 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009dc0 tx=0x7f79ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9680141d0b8f50b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:02.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.372+0000 7f79f3e00700 1 -- 172.21.15.12:0/3095061357 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00e070 con 0x7f79f40d2370 2024-03-29T10:07:02.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.372+0000 7f79f3e00700 1 -- 172.21.15.12:0/3095061357 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec00bae0 con 0x7f79f40d2370 2024-03-29T10:07:02.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.373+0000 7f79f3e00700 1 -- 172.21.15.12:0/3095061357 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00be90 con 0x7f79f40d2370 2024-03-29T10:07:02.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.373+0000 7f79fc600700 1 -- 172.21.15.12:0/3095061357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d2370 msgr2=0x7f79f40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:02.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.373+0000 7f79fc600700 1 --2- 172.21.15.12:0/3095061357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d2370 0x7f79f40ce360 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009dc0 tx=0x7f79ec00b330 comp rx=0 tx=0).stop 2024-03-29T10:07:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.373+0000 7f79fc600700 1 -- 172.21.15.12:0/3095061357 shutdown_connections 2024-03-29T10:07:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.373+0000 7f79fc600700 1 --2- 172.21.15.12:0/3095061357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d2370 0x7f79f40ce360 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.373+0000 7f79fc600700 1 --2- 172.21.15.12:0/3095061357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79f40d16a0 0x7f79f40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.373+0000 7f79fc600700 1 -- 172.21.15.12:0/3095061357 >> 172.21.15.12:0/3095061357 conn(0x7f79f408aff0 msgr2=0x7f79f40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.373+0000 7f79fc600700 1 -- 172.21.15.12:0/3095061357 shutdown_connections 2024-03-29T10:07:02.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.373+0000 7f79fc600700 1 -- 172.21.15.12:0/3095061357 wait complete. 2024-03-29T10:07:02.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.375+0000 7f79fc600700 1 Processor -- start 2024-03-29T10:07:02.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.375+0000 7f79fc600700 1 -- start start 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79fc600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d16a0 0x7f79f414ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79fc600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79f40d2370 0x7f79f40b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79fc600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79f415d9d0 0x7f79f40b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79fc600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f415fbb0 con 0x7f79f40d16a0 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79fc600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f41f37c0 con 0x7f79f415d9d0 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79fc600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f40b7640 con 0x7f79f40d2370 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79f9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d16a0 0x7f79f414ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79f9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d16a0 0x7f79f414ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45120/0 (socket says 172.21.15.12:45120) 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79f9c00700 1 -- 172.21.15.12:0/1294000199 learned_addr learned my addr 172.21.15.12:0/1294000199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79f9200700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79f40d2370 0x7f79f40b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.376+0000 7f79fa600700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79f415d9d0 0x7f79f40b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79f9c00700 1 -- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79f40d2370 msgr2=0x7f79f40b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79f9c00700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79f40d2370 0x7f79f40b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79f9c00700 1 -- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79f415d9d0 msgr2=0x7f79f40b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79f9c00700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79f415d9d0 0x7f79f40b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79f9c00700 1 -- 172.21.15.12:0/1294000199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ec009a70 con 0x7f79f40d16a0 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79fa600700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79f415d9d0 0x7f79f40b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79f9c00700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d16a0 0x7f79f414ea90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f79e8001eb0 tx=0x7f79e800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79f1600700 1 -- 172.21.15.12:0/1294000199 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e8005d70 con 0x7f79f40d16a0 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79f1600700 1 -- 172.21.15.12:0/1294000199 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e80070d0 con 0x7f79f40d16a0 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79f1600700 1 -- 172.21.15.12:0/1294000199 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e8007460 con 0x7f79f40d16a0 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79fc600700 1 -- 172.21.15.12:0/1294000199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f40b7920 con 0x7f79f40d16a0 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.377+0000 7f79fc600700 1 -- 172.21.15.12:0/1294000199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f40b7ee0 con 0x7f79f40d16a0 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.378+0000 7f79fc600700 1 -- 172.21.15.12:0/1294000199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f40d1ca0 con 0x7f79f40d16a0 2024-03-29T10:07:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.379+0000 7f79f1600700 1 -- 172.21.15.12:0/1294000199 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f79e80075c0 con 0x7f79f40d16a0 2024-03-29T10:07:02.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.380+0000 7f79f1600700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f79d406adb0 0x7f79d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:02.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.380+0000 7f79f1600700 1 -- 172.21.15.12:0/1294000199 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f79e8088970 con 0x7f79f40d16a0 2024-03-29T10:07:02.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.381+0000 7f79f9200700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f79d406adb0 0x7f79d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:02.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.381+0000 7f79f9200700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f79d406adb0 0x7f79d406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f79f406b630 tx=0x7f79e00091c0 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.385+0000 7f79f1600700 1 -- 172.21.15.12:0/1294000199 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f79e80539e0 con 0x7f79f40d16a0 2024-03-29T10:07:02.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.596+0000 7f79fc600700 1 -- 172.21.15.12:0/1294000199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f79f40d4080 con 0x7f79f40d16a0 2024-03-29T10:07:02.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.601+0000 7f79f1600700 1 -- 172.21.15.12:0/1294000199 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f79e8017020 con 0x7f79f40d16a0 2024-03-29T10:07:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 -- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f79d406adb0 msgr2=0x7f79d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f79d406adb0 0x7f79d406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f79f406b630 tx=0x7f79e00091c0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 -- 172.21.15.12:0/1294000199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d16a0 msgr2=0x7f79f414ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d16a0 0x7f79f414ea90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f79e8001eb0 tx=0x7f79e800c7b0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 -- 172.21.15.12:0/1294000199 shutdown_connections 2024-03-29T10:07:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f79d406adb0 0x7f79d406d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79f415d9d0 0x7f79f40b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79f40d16a0 0x7f79f414ea90 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 --2- 172.21.15.12:0/1294000199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79f40d2370 0x7f79f40b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 -- 172.21.15.12:0/1294000199 >> 172.21.15.12:0/1294000199 conn(0x7f79f408aff0 msgr2=0x7f79f415ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.603+0000 7f79fc600700 1 -- 172.21.15.12:0/1294000199 shutdown_connections 2024-03-29T10:07:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:02.604+0000 7f79fc600700 1 -- 172.21.15.12:0/1294000199 wait complete. 2024-03-29T10:07:02.621 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:07:02.622 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:07:02.706 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:07:02.799 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:07:02.799 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:07:02.799 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:07:02.844 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:07:02.844 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:07:02.844 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:07:02.891 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:07:02.891 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:07:02.891 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:07:02.935 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:07:02.935 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:07:02.936 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:07:02.981 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:07:02.981 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T10:07:02.981 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:07:03.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.181+0000 7f026a800700 1 Processor -- start 2024-03-29T10:07:03.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.181+0000 7f026a800700 1 -- start start 2024-03-29T10:07:03.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f026a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0264160f10 0x7f02640d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:03.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f026a800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02640ceda0 con 0x7f02640cf230 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f026a800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02641612f0 con 0x7f0264161590 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f026a800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02640ce470 con 0x7f0264160f10 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f0262a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f02640cf230 0x7f02640d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41458/0 (socket says 172.21.15.12:41458) 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f0262a00700 1 -- 172.21.15.12:0/3974492150 learned_addr learned my addr 172.21.15.12:0/3974492150 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f0263400700 1 --2- 172.21.15.12:0/3974492150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0264160f10 0x7f02640d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f0262000700 1 -- 172.21.15.12:0/3974492150 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352434383 0 0) 0x7f02640ceda0 con 0x7f02640cf230 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f0262000700 1 -- 172.21.15.12:0/3974492150 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02480035b0 con 0x7f02640cf230 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f0262000700 1 -- 172.21.15.12:0/3974492150 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758859563 0 0) 0x7f02641612f0 con 0x7f0264161590 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.183+0000 7f0262000700 1 -- 172.21.15.12:0/3974492150 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02640ceda0 con 0x7f0264161590 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.184+0000 7f0263400700 1 -- 172.21.15.12:0/3974492150 >> v1:172.21.15.159:6789/0 conn(0x7f0264161590 legacy=0x7f02640b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.184+0000 7f0263400700 1 -- 172.21.15.12:0/3974492150 >> v1:172.21.15.12:6789/0 conn(0x7f02640cf230 legacy=0x7f02640d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.184+0000 7f0263400700 1 -- 172.21.15.12:0/3974492150 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02640ce750 con 0x7f0264160f10 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.184+0000 7f0263400700 1 --2- 172.21.15.12:0/3974492150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0264160f10 0x7f02640d18b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f02540098a0 tx=0x7f0254009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40679f0d2458e96b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:03.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.184+0000 7f0262000700 1 -- 172.21.15.12:0/3974492150 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254011830 con 0x7f0264160f10 2024-03-29T10:07:03.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.184+0000 7f0262000700 1 -- 172.21.15.12:0/3974492150 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0254011990 con 0x7f0264160f10 2024-03-29T10:07:03.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.184+0000 7f0262000700 1 -- 172.21.15.12:0/3974492150 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254011d00 con 0x7f0264160f10 2024-03-29T10:07:03.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.185+0000 7f026a800700 1 -- 172.21.15.12:0/3974492150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0264160f10 msgr2=0x7f02640d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.185+0000 7f026a800700 1 --2- 172.21.15.12:0/3974492150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0264160f10 0x7f02640d18b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f02540098a0 tx=0x7f0254009bb0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.185+0000 7f026a800700 1 -- 172.21.15.12:0/3974492150 shutdown_connections 2024-03-29T10:07:03.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.185+0000 7f026a800700 1 --2- 172.21.15.12:0/3974492150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0264160f10 0x7f02640d18b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.185+0000 7f026a800700 1 -- 172.21.15.12:0/3974492150 >> 172.21.15.12:0/3974492150 conn(0x7f026408aff0 msgr2=0x7f02640d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:03.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.185+0000 7f026a800700 1 -- 172.21.15.12:0/3974492150 shutdown_connections 2024-03-29T10:07:03.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.185+0000 7f026a800700 1 -- 172.21.15.12:0/3974492150 wait complete. 2024-03-29T10:07:03.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.186+0000 7f026a800700 1 Processor -- start 2024-03-29T10:07:03.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.187+0000 7f026a800700 1 -- start start 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f026a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02640cf230 0x7f02640b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f026a800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0264160f10 0x7f02641553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f026a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0264161590 0x7f02640b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f026a800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02640b8d40 con 0x7f0264161590 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f026a800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0264155900 con 0x7f0264160f10 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f026a800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02641f6670 con 0x7f02640cf230 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0263400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02640cf230 0x7f02640b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0263e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0264161590 0x7f02640b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0263400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02640cf230 0x7f02640b7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:57620/0 (socket says 172.21.15.12:57620) 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0263400700 1 -- 172.21.15.12:0/490271599 learned_addr learned my addr 172.21.15.12:0/490271599 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0262a00700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0264160f10 0x7f02641553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0263400700 1 -- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0264160f10 msgr2=0x7f02641553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0263400700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0264160f10 0x7f02641553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0263400700 1 -- 172.21.15.12:0/490271599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0264161590 msgr2=0x7f02640b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0263400700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0264161590 0x7f02640b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0263400700 1 -- 172.21.15.12:0/490271599 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02540094a0 con 0x7f02640cf230 2024-03-29T10:07:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.188+0000 7f0263e00700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0264161590 0x7f02640b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:07:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.189+0000 7f0263400700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02640cf230 0x7f02640b7a50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0254009870 tx=0x7f0254004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.189+0000 7f025b400700 1 -- 172.21.15.12:0/490271599 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254004550 con 0x7f02640cf230 2024-03-29T10:07:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.189+0000 7f025b400700 1 -- 172.21.15.12:0/490271599 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025401f070 con 0x7f02640cf230 2024-03-29T10:07:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.189+0000 7f025b400700 1 -- 172.21.15.12:0/490271599 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02540038e0 con 0x7f02640cf230 2024-03-29T10:07:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.189+0000 7f026a800700 1 -- 172.21.15.12:0/490271599 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02640b80c0 con 0x7f02640cf230 2024-03-29T10:07:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.189+0000 7f026a800700 1 -- 172.21.15.12:0/490271599 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02640b8680 con 0x7f02640cf230 2024-03-29T10:07:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.190+0000 7f026a800700 1 -- 172.21.15.12:0/490271599 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02640b8e80 con 0x7f02640cf230 2024-03-29T10:07:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.191+0000 7f025b400700 1 -- 172.21.15.12:0/490271599 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0254003a40 con 0x7f02640cf230 2024-03-29T10:07:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.192+0000 7f025b400700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f024406adb0 0x7f024406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:03.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.192+0000 7f025b400700 1 -- 172.21.15.12:0/490271599 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f0254021070 con 0x7f02640cf230 2024-03-29T10:07:03.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.192+0000 7f0262a00700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f024406adb0 0x7f024406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:03.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.193+0000 7f0262a00700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f024406adb0 0x7f024406d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f02641e86a0 tx=0x7f0250007de0 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:03.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.198+0000 7f025b400700 1 -- 172.21.15.12:0/490271599 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f02640b8e80 con 0x7f02640cf230 2024-03-29T10:07:03.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.459+0000 7f026a800700 1 -- 172.21.15.12:0/490271599 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02640b8e80 con 0x7f02640cf230 2024-03-29T10:07:03.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.460+0000 7f025b400700 1 -- 172.21.15.12:0/490271599 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f02640b8e80 con 0x7f02640cf230 2024-03-29T10:07:03.460 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:03.460 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2845880355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2845880355},{"type":"v1","addr":"172.21.15.159:6837","nonce":2845880355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2836227468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2836227468},{"type":"v1","addr":"172.21.15.159:6836","nonce":2836227468}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1863550973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1863550973},{"type":"v1","addr":"172.21.15.12:6834","nonce":1863550973}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2337943317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2337943317},{"type":"v1","addr":"172.21.15.12:6835","nonce":2337943317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[]} 2024-03-29T10:07:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.462+0000 7f026a800700 1 -- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f024406adb0 msgr2=0x7f024406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f024406adb0 0x7f024406d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f02641e86a0 tx=0x7f0250007de0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 -- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02640cf230 msgr2=0x7f02640b7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02640cf230 0x7f02640b7a50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0254009870 tx=0x7f0254004750 comp rx=0 tx=0).stop 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 -- 172.21.15.12:0/490271599 shutdown_connections 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f024406adb0 0x7f024406d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0264161590 0x7f02640b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02640cf230 0x7f02640b7a50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 --2- 172.21.15.12:0/490271599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0264160f10 0x7f02641553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 -- 172.21.15.12:0/490271599 >> 172.21.15.12:0/490271599 conn(0x7f026408aff0 msgr2=0x7f02640cacf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 -- 172.21.15.12:0/490271599 shutdown_connections 2024-03-29T10:07:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.463+0000 7f026a800700 1 -- 172.21.15.12:0/490271599 wait complete. 2024-03-29T10:07:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 17 2024-03-29T10:07:03.481 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T10:07:03.481 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T10:07:03.482 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T10:07:03.482 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:07:03.482 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T10:07:03.482 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T10:07:03.482 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T10:07:03.482 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:07:03.483 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T10:07:03.483 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T10:07:03.483 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T10:07:03.483 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:07:03.483 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T10:07:03.483 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T10:07:03.483 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T10:07:03.484 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:07:03.497 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T10:07:03.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T10:07:03.499 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T10:07:03.499 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T10:07:03.500 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T10:07:03.500 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T10:07:03.500 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T10:07:03.500 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T10:07:03.543 INFO:tasks.ceph.mds.a:Started 2024-03-29T10:07:03.543 DEBUG:teuthology.parallel:result is None 2024-03-29T10:07:03.544 INFO:tasks.ceph.mds.b:Started 2024-03-29T10:07:03.544 DEBUG:teuthology.parallel:result is None 2024-03-29T10:07:03.545 INFO:tasks.ceph.mds.d:Started 2024-03-29T10:07:03.546 DEBUG:teuthology.parallel:result is None 2024-03-29T10:07:03.546 INFO:tasks.ceph.mds.c:Started 2024-03-29T10:07:03.546 DEBUG:teuthology.parallel:result is None 2024-03-29T10:07:03.546 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T10:07:03.687 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T10:07:03.687 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T10:07:03.693 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:07:03.693 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T10:07:03.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.759+0000 7fbce8a00700 1 Processor -- start 2024-03-29T10:07:03.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.759+0000 7fbce8a00700 1 -- start start 2024-03-29T10:07:03.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.760+0000 7fbce8a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbce40ce840 0x7fbce4165370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:03.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.760+0000 7fbce8a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce40cb310 con 0x7fbce415a5c0 2024-03-29T10:07:03.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.760+0000 7fbce8a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce40cca90 con 0x7fbce415dfe0 2024-03-29T10:07:03.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.760+0000 7fbce8a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbce40711a0 con 0x7fbce40ce840 2024-03-29T10:07:03.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.760+0000 7fbce1e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbce415a5c0 0x7fbce40709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41492/0 (socket says 172.21.15.12:41492) 2024-03-29T10:07:03.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.760+0000 7fbce1e00700 1 -- 172.21.15.12:0/4223900045 learned_addr learned my addr 172.21.15.12:0/4223900045 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:03.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.761+0000 7fbce2800700 1 --2- 172.21.15.12:0/4223900045 >> v2:172.21.15.159:3300/0 conn(0x7fbce40ce840 0x7fbce4165370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.761+0000 7fbce0a00700 1 -- 172.21.15.12:0/4223900045 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577322777 0 0) 0x7fbce40cb310 con 0x7fbce415a5c0 2024-03-29T10:07:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.761+0000 7fbce0a00700 1 -- 172.21.15.12:0/4223900045 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcc80057c0 con 0x7fbce415a5c0 2024-03-29T10:07:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.761+0000 7fbce0a00700 1 -- 172.21.15.12:0/4223900045 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520534671 0 0) 0x7fbce40cca90 con 0x7fbce415dfe0 2024-03-29T10:07:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.761+0000 7fbce0a00700 1 -- 172.21.15.12:0/4223900045 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce40cb310 con 0x7fbce415dfe0 2024-03-29T10:07:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.761+0000 7fbce2800700 1 -- 172.21.15.12:0/4223900045 >> v1:172.21.15.159:6789/0 conn(0x7fbce415dfe0 legacy=0x7fbce40ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.761+0000 7fbce2800700 1 -- 172.21.15.12:0/4223900045 >> v1:172.21.15.12:6789/0 conn(0x7fbce415a5c0 legacy=0x7fbce40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.761+0000 7fbce2800700 1 -- 172.21.15.12:0/4223900045 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce4071480 con 0x7fbce40ce840 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.761+0000 7fbce2800700 1 --2- 172.21.15.12:0/4223900045 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbce40ce840 0x7fbce4165370 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009d90 tx=0x7fbcdc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0105268dee6a134 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.762+0000 7fbce0a00700 1 -- 172.21.15.12:0/4223900045 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc00e070 con 0x7fbce40ce840 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.762+0000 7fbce0a00700 1 -- 172.21.15.12:0/4223900045 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcdc00bb20 con 0x7fbce40ce840 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.762+0000 7fbce0a00700 1 -- 172.21.15.12:0/4223900045 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc012430 con 0x7fbce40ce840 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.762+0000 7fbce8a00700 1 -- 172.21.15.12:0/4223900045 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbce40ce840 msgr2=0x7fbce4165370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.762+0000 7fbce8a00700 1 --2- 172.21.15.12:0/4223900045 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbce40ce840 0x7fbce4165370 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009d90 tx=0x7fbcdc00b330 comp rx=0 tx=0).stop 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.762+0000 7fbce8a00700 1 -- 172.21.15.12:0/4223900045 shutdown_connections 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.762+0000 7fbce8a00700 1 --2- 172.21.15.12:0/4223900045 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbce40ce840 0x7fbce4165370 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.762+0000 7fbce8a00700 1 -- 172.21.15.12:0/4223900045 >> 172.21.15.12:0/4223900045 conn(0x7fbce40b0d70 msgr2=0x7fbce40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.762+0000 7fbce8a00700 1 -- 172.21.15.12:0/4223900045 shutdown_connections 2024-03-29T10:07:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.762+0000 7fbce8a00700 1 -- 172.21.15.12:0/4223900045 wait complete. 2024-03-29T10:07:03.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.764+0000 7fbce8a00700 1 Processor -- start 2024-03-29T10:07:03.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.764+0000 7fbce8a00700 1 -- start start 2024-03-29T10:07:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.765+0000 7fbce8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbce40ce840 0x7fbce4154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.765+0000 7fbce8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbce415a5c0 0x7fbce40ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.765+0000 7fbce8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbce415dfe0 0x7fbce40ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.765+0000 7fbce8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce4157bb0 con 0x7fbce40ce840 2024-03-29T10:07:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.765+0000 7fbce8a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce4157e00 con 0x7fbce415dfe0 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.765+0000 7fbce8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce41580e0 con 0x7fbce415a5c0 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.765+0000 7fbce1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbce40ce840 0x7fbce4154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.765+0000 7fbce1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbce40ce840 0x7fbce4154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45182/0 (socket says 172.21.15.12:45182) 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.765+0000 7fbce1e00700 1 -- 172.21.15.12:0/188464276 learned_addr learned my addr 172.21.15.12:0/188464276 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.766+0000 7fbce2800700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbce415dfe0 0x7fbce40ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.766+0000 7fbce1400700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbce415a5c0 0x7fbce40ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.766+0000 7fbce1e00700 1 -- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbce415a5c0 msgr2=0x7fbce40ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.766+0000 7fbce1e00700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbce415a5c0 0x7fbce40ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.766+0000 7fbce1e00700 1 -- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbce415dfe0 msgr2=0x7fbce40ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.766+0000 7fbce1e00700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbce415dfe0 0x7fbce40ac590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.766+0000 7fbce1e00700 1 -- 172.21.15.12:0/188464276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc009a70 con 0x7fbce40ce840 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.766+0000 7fbce1e00700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbce40ce840 0x7fbce4154630 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400c700 tx=0x7fbcd400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.766+0000 7fbcda000700 1 -- 172.21.15.12:0/188464276 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4005d70 con 0x7fbce40ce840 2024-03-29T10:07:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.767+0000 7fbcda000700 1 -- 172.21.15.12:0/188464276 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd40070d0 con 0x7fbce40ce840 2024-03-29T10:07:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.767+0000 7fbce8a00700 1 -- 172.21.15.12:0/188464276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce41583c0 con 0x7fbce40ce840 2024-03-29T10:07:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.767+0000 7fbcda000700 1 -- 172.21.15.12:0/188464276 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4007440 con 0x7fbce40ce840 2024-03-29T10:07:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.767+0000 7fbce8a00700 1 -- 172.21.15.12:0/188464276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce4158980 con 0x7fbce40ce840 2024-03-29T10:07:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.768+0000 7fbce8a00700 1 -- 172.21.15.12:0/188464276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce4158ba0 con 0x7fbce40ce840 2024-03-29T10:07:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.769+0000 7fbcda000700 1 -- 172.21.15.12:0/188464276 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbcd40075a0 con 0x7fbce40ce840 2024-03-29T10:07:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.771+0000 7fbcda000700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbcc406adb0 0x7fbcc406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.771+0000 7fbcda000700 1 -- 172.21.15.12:0/188464276 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fbcd40886a0 con 0x7fbce40ce840 2024-03-29T10:07:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.771+0000 7fbce1400700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbcc406adb0 0x7fbcc406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:03.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.772+0000 7fbce1400700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbcc406adb0 0x7fbcc406d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbce414de00 tx=0x7fbcd0009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:03.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:03.775+0000 7fbcda000700 1 -- 172.21.15.12:0/188464276 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbcd40538e0 con 0x7fbce40ce840 2024-03-29T10:07:04.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.007+0000 7fbce8a00700 1 -- 172.21.15.12:0/188464276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbce4067720 con 0x7fbce40ce840 2024-03-29T10:07:04.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.722+0000 7fbcda000700 1 -- 172.21.15.12:0/188464276 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v57) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbcd4017080 con 0x7fbce40ce840 2024-03-29T10:07:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 -- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbcc406adb0 msgr2=0x7fbcc406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbcc406adb0 0x7fbcc406d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbce414de00 tx=0x7fbcd0009040 comp rx=0 tx=0).stop 2024-03-29T10:07:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 -- 172.21.15.12:0/188464276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbce40ce840 msgr2=0x7fbce4154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbce40ce840 0x7fbce4154630 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400c700 tx=0x7fbcd400ca10 comp rx=0 tx=0).stop 2024-03-29T10:07:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 -- 172.21.15.12:0/188464276 shutdown_connections 2024-03-29T10:07:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbcc406adb0 0x7fbcc406d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbce415dfe0 0x7fbce40ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbce40ce840 0x7fbce4154630 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:04.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 --2- 172.21.15.12:0/188464276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbce415a5c0 0x7fbce40ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:04.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 -- 172.21.15.12:0/188464276 >> 172.21.15.12:0/188464276 conn(0x7fbce40b0d70 msgr2=0x7fbce40cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:04.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 -- 172.21.15.12:0/188464276 shutdown_connections 2024-03-29T10:07:04.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.726+0000 7fbce8a00700 1 -- 172.21.15.12:0/188464276 wait complete. 2024-03-29T10:07:04.729 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T10:07:04.750 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T10:07:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.954+0000 7f8542200700 1 Processor -- start 2024-03-29T10:07:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.955+0000 7f8542200700 1 -- start start 2024-03-29T10:07:04.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.956+0000 7f8542200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f853c15b5d0 0x7f853c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:04.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.956+0000 7f8542200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c15fd50 con 0x7f853c0cb3f0 2024-03-29T10:07:04.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.956+0000 7f8542200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c15f140 con 0x7f853c0ac7d0 2024-03-29T10:07:04.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.956+0000 7f8542200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853c15b9b0 con 0x7f853c15b5d0 2024-03-29T10:07:04.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.956+0000 7f853b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f853c15b5d0 0x7f853c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:04.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.956+0000 7f853b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f853c15b5d0 0x7f853c0ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45186/0 (socket says 172.21.15.12:45186) 2024-03-29T10:07:04.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.956+0000 7f853b400700 1 -- 172.21.15.12:0/2855611944 learned_addr learned my addr 172.21.15.12:0/2855611944 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:04.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.957+0000 7f853b400700 1 -- 172.21.15.12:0/2855611944 >> v1:172.21.15.159:6789/0 conn(0x7f853c0ac7d0 legacy=0x7f853c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:04.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.957+0000 7f853b400700 1 -- 172.21.15.12:0/2855611944 >> v1:172.21.15.12:6789/0 conn(0x7f853c0cb3f0 legacy=0x7f853c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:04.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.957+0000 7f853b400700 1 -- 172.21.15.12:0/2855611944 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c15f420 con 0x7f853c15b5d0 2024-03-29T10:07:04.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.957+0000 7f853b400700 1 --2- 172.21.15.12:0/2855611944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 0x7f853c0ae3f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f852c0098f0 tx=0x7f852c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a9c1989d231bf4f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:04.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.957+0000 7f853a000700 1 -- 172.21.15.12:0/2855611944 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00b8a0 con 0x7f853c15b5d0 2024-03-29T10:07:04.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.957+0000 7f853a000700 1 -- 172.21.15.12:0/2855611944 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c00ba00 con 0x7f853c15b5d0 2024-03-29T10:07:04.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.957+0000 7f853a000700 1 -- 172.21.15.12:0/2855611944 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00bd90 con 0x7f853c15b5d0 2024-03-29T10:07:04.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.958+0000 7f8542200700 1 -- 172.21.15.12:0/2855611944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 msgr2=0x7f853c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:04.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.958+0000 7f8542200700 1 --2- 172.21.15.12:0/2855611944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 0x7f853c0ae3f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f852c0098f0 tx=0x7f852c009c00 comp rx=0 tx=0).stop 2024-03-29T10:07:04.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.958+0000 7f8542200700 1 -- 172.21.15.12:0/2855611944 shutdown_connections 2024-03-29T10:07:04.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.958+0000 7f8542200700 1 --2- 172.21.15.12:0/2855611944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 0x7f853c0ae3f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:04.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.958+0000 7f8542200700 1 -- 172.21.15.12:0/2855611944 >> 172.21.15.12:0/2855611944 conn(0x7f853c0cf190 msgr2=0x7f853c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:04.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.958+0000 7f8542200700 1 -- 172.21.15.12:0/2855611944 shutdown_connections 2024-03-29T10:07:04.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.958+0000 7f8542200700 1 -- 172.21.15.12:0/2855611944 wait complete. 2024-03-29T10:07:04.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.960+0000 7f8542200700 1 Processor -- start 2024-03-29T10:07:04.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.960+0000 7f8542200700 1 -- start start 2024-03-29T10:07:04.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f8542200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f853c0ac7d0 0x7f853c157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:04.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f8542200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 0x7f853c159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f8542200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f853c1f2de0 0x7f853c0cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f8542200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c15ae70 con 0x7f853c15b5d0 2024-03-29T10:07:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f8542200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c15a320 con 0x7f853c1f2de0 2024-03-29T10:07:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f8542200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853c15a5d0 con 0x7f853c0ac7d0 2024-03-29T10:07:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f853aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 0x7f853c159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f853aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 0x7f853c159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45196/0 (socket says 172.21.15.12:45196) 2024-03-29T10:07:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f853aa00700 1 -- 172.21.15.12:0/3743685096 learned_addr learned my addr 172.21.15.12:0/3743685096 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f853b400700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f853c0ac7d0 0x7f853c157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f853be00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f853c1f2de0 0x7f853c0cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f853aa00700 1 -- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f853c0ac7d0 msgr2=0x7f853c157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f853aa00700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f853c0ac7d0 0x7f853c157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f853aa00700 1 -- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f853c1f2de0 msgr2=0x7f853c0cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f853aa00700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f853c1f2de0 0x7f853c0cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.961+0000 7f853aa00700 1 -- 172.21.15.12:0/3743685096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c009550 con 0x7f853c15b5d0 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.962+0000 7f853aa00700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 0x7f853c159de0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f852800c4e0 tx=0x7f852800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.962+0000 7f8533400700 1 -- 172.21.15.12:0/3743685096 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85280052d0 con 0x7f853c15b5d0 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.962+0000 7f8533400700 1 -- 172.21.15.12:0/3743685096 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8528005430 con 0x7f853c15b5d0 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.962+0000 7f8542200700 1 -- 172.21.15.12:0/3743685096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853c15a8b0 con 0x7f853c15b5d0 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.962+0000 7f8533400700 1 -- 172.21.15.12:0/3743685096 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85280038c0 con 0x7f853c15b5d0 2024-03-29T10:07:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.962+0000 7f8542200700 1 -- 172.21.15.12:0/3743685096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853c0cbed0 con 0x7f853c15b5d0 2024-03-29T10:07:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.963+0000 7f8542200700 1 -- 172.21.15.12:0/3743685096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853c0cbd30 con 0x7f853c15b5d0 2024-03-29T10:07:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.964+0000 7f8533400700 1 -- 172.21.15.12:0/3743685096 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8528003a20 con 0x7f853c15b5d0 2024-03-29T10:07:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.965+0000 7f8533400700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f851c06adb0 0x7f851c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.965+0000 7f8533400700 1 -- 172.21.15.12:0/3743685096 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8528088610 con 0x7f853c15b5d0 2024-03-29T10:07:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.965+0000 7f853b400700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f851c06adb0 0x7f851c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.966+0000 7f853b400700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f851c06adb0 0x7f851c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f852c014040 tx=0x7f852c008030 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:04.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:04.969+0000 7f8533400700 1 -- 172.21.15.12:0/3743685096 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f853c0cbd30 con 0x7f853c15b5d0 2024-03-29T10:07:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.263+0000 7f8542200700 1 -- 172.21.15.12:0/3743685096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f853c0cbd30 con 0x7f853c15b5d0 2024-03-29T10:07:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.264+0000 7f8533400700 1 -- 172.21.15.12:0/3743685096 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f853c0cbd30 con 0x7f853c15b5d0 2024-03-29T10:07:05.263 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:05.263 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T10:07:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:07:05.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:07:05.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.268 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:07:05.269 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:07:05.270 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.270 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.270 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.270 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T10:07:05.270 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T10:07:05.270 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.270 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:07:05.270 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:07:05.270 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T10:07:05.271 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T10:07:05.272 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T10:07:05.273 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:07:05.274 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T10:07:05.276 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T10:07:05.277 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:07:05.277 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:07:05.277 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:07:05.277 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:07:05.277 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.277 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:05.277 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T10:07:05.277 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:07:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 -- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f851c06adb0 msgr2=0x7f851c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f851c06adb0 0x7f851c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f852c014040 tx=0x7f852c008030 comp rx=0 tx=0).stop 2024-03-29T10:07:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 -- 172.21.15.12:0/3743685096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 msgr2=0x7f853c159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 0x7f853c159de0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f852800c4e0 tx=0x7f852800c7f0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 -- 172.21.15.12:0/3743685096 shutdown_connections 2024-03-29T10:07:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f851c06adb0 0x7f851c06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f853c1f2de0 0x7f853c0cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f853c0ac7d0 0x7f853c157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 --2- 172.21.15.12:0/3743685096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f853c15b5d0 0x7f853c159de0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 -- 172.21.15.12:0/3743685096 >> 172.21.15.12:0/3743685096 conn(0x7f853c0cf190 msgr2=0x7f853c15e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 -- 172.21.15.12:0/3743685096 shutdown_connections 2024-03-29T10:07:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.266+0000 7f8542200700 1 -- 172.21.15.12:0/3743685096 wait complete. 2024-03-29T10:07:05.284 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:07:05.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.481+0000 7fd6ae600700 1 Processor -- start 2024-03-29T10:07:05.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.481+0000 7fd6ae600700 1 -- start start 2024-03-29T10:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.482+0000 7fd6ae600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd6a80d2870 0x7fd6a80d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.482+0000 7fd6ae600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a80d2dc0 con 0x7fd6a80d0a00 2024-03-29T10:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.482+0000 7fd6ae600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a80d30a0 con 0x7fd6a815d5d0 2024-03-29T10:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.482+0000 7fd6ae600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a815d9b0 con 0x7fd6a80d2870 2024-03-29T10:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.482+0000 7fd6a7e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd6a80d0a00 0x7fd6a80d0f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41518/0 (socket says 172.21.15.12:41518) 2024-03-29T10:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.482+0000 7fd6a7e00700 1 -- 172.21.15.12:0/2924617218 learned_addr learned my addr 172.21.15.12:0/2924617218 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.483+0000 7fd6adc00700 1 --2- 172.21.15.12:0/2924617218 >> v2:172.21.15.159:3300/0 conn(0x7fd6a80d2870 0x7fd6a80d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.483+0000 7fd6a7400700 1 -- 172.21.15.12:0/2924617218 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035705150 0 0) 0x7fd6a80d2dc0 con 0x7fd6a80d0a00 2024-03-29T10:07:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.483+0000 7fd6a7400700 1 -- 172.21.15.12:0/2924617218 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6900035b0 con 0x7fd6a80d0a00 2024-03-29T10:07:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.483+0000 7fd6a7400700 1 -- 172.21.15.12:0/2924617218 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536070714 0 0) 0x7fd6a80d30a0 con 0x7fd6a815d5d0 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.483+0000 7fd6a7400700 1 -- 172.21.15.12:0/2924617218 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a80d2dc0 con 0x7fd6a815d5d0 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.483+0000 7fd6adc00700 1 -- 172.21.15.12:0/2924617218 >> v1:172.21.15.159:6789/0 conn(0x7fd6a815d5d0 legacy=0x7fd6a80d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.483+0000 7fd6adc00700 1 -- 172.21.15.12:0/2924617218 >> v1:172.21.15.12:6789/0 conn(0x7fd6a80d0a00 legacy=0x7fd6a80d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.483+0000 7fd6adc00700 1 -- 172.21.15.12:0/2924617218 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a80d3380 con 0x7fd6a80d2870 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.483+0000 7fd6adc00700 1 --2- 172.21.15.12:0/2924617218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6a80d2870 0x7fd6a80d3e60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009d90 tx=0x7fd6a000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a41616b3bc031fe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.484+0000 7fd6a7400700 1 -- 172.21.15.12:0/2924617218 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a000e070 con 0x7fd6a80d2870 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.484+0000 7fd6a7400700 1 -- 172.21.15.12:0/2924617218 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a000bb20 con 0x7fd6a80d2870 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.484+0000 7fd6a7400700 1 -- 172.21.15.12:0/2924617218 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0012430 con 0x7fd6a80d2870 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.484+0000 7fd6ae600700 1 -- 172.21.15.12:0/2924617218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6a80d2870 msgr2=0x7fd6a80d3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.484+0000 7fd6ae600700 1 --2- 172.21.15.12:0/2924617218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6a80d2870 0x7fd6a80d3e60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009d90 tx=0x7fd6a000b330 comp rx=0 tx=0).stop 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.484+0000 7fd6ae600700 1 -- 172.21.15.12:0/2924617218 shutdown_connections 2024-03-29T10:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.484+0000 7fd6ae600700 1 --2- 172.21.15.12:0/2924617218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6a80d2870 0x7fd6a80d3e60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.484+0000 7fd6ae600700 1 -- 172.21.15.12:0/2924617218 >> 172.21.15.12:0/2924617218 conn(0x7fd6a80668e0 msgr2=0x7fd6a8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:05.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.484+0000 7fd6ae600700 1 -- 172.21.15.12:0/2924617218 shutdown_connections 2024-03-29T10:07:05.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.484+0000 7fd6ae600700 1 -- 172.21.15.12:0/2924617218 wait complete. 2024-03-29T10:07:05.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.486+0000 7fd6ae600700 1 Processor -- start 2024-03-29T10:07:05.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.486+0000 7fd6ae600700 1 -- start start 2024-03-29T10:07:05.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.487+0000 7fd6ae600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6a80d0a00 0x7fd6a8151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.487+0000 7fd6ae600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6a80d2870 0x7fd6a815b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.487+0000 7fd6ae600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6a815d5d0 0x7fd6a815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.487+0000 7fd6ae600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80b3ad0 con 0x7fd6a80d0a00 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.487+0000 7fd6ae600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8152360 con 0x7fd6a815d5d0 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.487+0000 7fd6ae600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8156b80 con 0x7fd6a80d2870 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.487+0000 7fd6ad200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6a80d0a00 0x7fd6a8151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.487+0000 7fd6ad200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6a80d0a00 0x7fd6a8151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45208/0 (socket says 172.21.15.12:45208) 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.487+0000 7fd6ad200700 1 -- 172.21.15.12:0/4278588848 learned_addr learned my addr 172.21.15.12:0/4278588848 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.487+0000 7fd6a7e00700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6a80d2870 0x7fd6a815b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6adc00700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6a815d5d0 0x7fd6a815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6ad200700 1 -- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6a80d2870 msgr2=0x7fd6a815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6ad200700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6a80d2870 0x7fd6a815b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6ad200700 1 -- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6a815d5d0 msgr2=0x7fd6a815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6ad200700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6a815d5d0 0x7fd6a815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6ad200700 1 -- 172.21.15.12:0/4278588848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a0009a70 con 0x7fd6a80d0a00 2024-03-29T10:07:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6a7e00700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6a80d2870 0x7fd6a815b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6adc00700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6a815d5d0 0x7fd6a815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6ad200700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6a80d0a00 0x7fd6a8151e20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00c520 tx=0x7fd69c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6a4c00700 1 -- 172.21.15.12:0/4278588848 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0052d0 con 0x7fd6a80d0a00 2024-03-29T10:07:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6a4c00700 1 -- 172.21.15.12:0/4278588848 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c005430 con 0x7fd6a80d0a00 2024-03-29T10:07:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6ae600700 1 -- 172.21.15.12:0/4278588848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a8156e60 con 0x7fd6a80d0a00 2024-03-29T10:07:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6a4c00700 1 -- 172.21.15.12:0/4278588848 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0038c0 con 0x7fd6a80d0a00 2024-03-29T10:07:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.488+0000 7fd6ae600700 1 -- 172.21.15.12:0/4278588848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a8157420 con 0x7fd6a80d0a00 2024-03-29T10:07:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.489+0000 7fd6ae600700 1 -- 172.21.15.12:0/4278588848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a80b85f0 con 0x7fd6a80d0a00 2024-03-29T10:07:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.490+0000 7fd6a4c00700 1 -- 172.21.15.12:0/4278588848 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd69c003a20 con 0x7fd6a80d0a00 2024-03-29T10:07:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.491+0000 7fd6a4c00700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd68806ade0 0x7fd68806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.491+0000 7fd6a4c00700 1 -- 172.21.15.12:0/4278588848 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd69c088780 con 0x7fd6a80d0a00 2024-03-29T10:07:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.492+0000 7fd6a7e00700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd68806ade0 0x7fd68806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.493+0000 7fd6a7e00700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd68806ade0 0x7fd68806d2a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80b7f30 tx=0x7fd694009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:05.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.496+0000 7fd6a4c00700 1 -- 172.21.15.12:0/4278588848 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd69c053a70 con 0x7fd6a80d0a00 2024-03-29T10:07:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.711+0000 7fd6ae600700 1 -- 172.21.15.12:0/4278588848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6a8168020 con 0x7fd6a80d0a00 2024-03-29T10:07:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.712+0000 7fd6a4c00700 1 -- 172.21.15.12:0/4278588848 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+12389 (secure 0 0 0) 0x7fd69c017090 con 0x7fd6a80d0a00 2024-03-29T10:07:05.711 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:05.712 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":57,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:07:04.718404+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:07:05.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 -- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd68806ade0 msgr2=0x7fd68806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd68806ade0 0x7fd68806d2a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80b7f30 tx=0x7fd694009040 comp rx=0 tx=0).stop 2024-03-29T10:07:05.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 -- 172.21.15.12:0/4278588848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6a80d0a00 msgr2=0x7fd6a8151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6a80d0a00 0x7fd6a8151e20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00c520 tx=0x7fd69c00c8e0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 -- 172.21.15.12:0/4278588848 shutdown_connections 2024-03-29T10:07:05.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd68806ade0 0x7fd68806d2a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6a815d5d0 0x7fd6a815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6a80d0a00 0x7fd6a8151e20 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 --2- 172.21.15.12:0/4278588848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6a80d2870 0x7fd6a815b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 -- 172.21.15.12:0/4278588848 >> 172.21.15.12:0/4278588848 conn(0x7fd6a80668e0 msgr2=0x7fd6a80b7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:05.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 -- 172.21.15.12:0/4278588848 shutdown_connections 2024-03-29T10:07:05.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.714+0000 7fd6ae600700 1 -- 172.21.15.12:0/4278588848 wait complete. 2024-03-29T10:07:05.732 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:07:05.733 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:07:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.934+0000 7f95ebc00700 1 Processor -- start 2024-03-29T10:07:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.934+0000 7f95ebc00700 1 -- start start 2024-03-29T10:07:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.935+0000 7f95ebc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95e415dad0 0x7f95e40ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.935+0000 7f95ebc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f95e40d1630 0x7f95e40d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.935+0000 7f95ebc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e40d2000 0x7f95e415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.935+0000 7f95ebc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e40cfcc0 con 0x7f95e415dad0 2024-03-29T10:07:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.935+0000 7f95ebc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e40cd9f0 con 0x7f95e40d1630 2024-03-29T10:07:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.935+0000 7f95ebc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e415f820 con 0x7f95e40d2000 2024-03-29T10:07:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e9200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95e415dad0 0x7f95e40ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e9200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95e415dad0 0x7f95e40ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45212/0 (socket says 172.21.15.12:45212) 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e9200700 1 -- 172.21.15.12:0/970782120 learned_addr learned my addr 172.21.15.12:0/970782120 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e3e00700 1 --2- 172.21.15.12:0/970782120 >> v2:172.21.15.159:3300/0 conn(0x7f95e40d1630 0x7f95e40d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e9c00700 1 --2- 172.21.15.12:0/970782120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e40d2000 0x7f95e415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e9200700 1 -- 172.21.15.12:0/970782120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e40d2000 msgr2=0x7f95e415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e9200700 1 --2- 172.21.15.12:0/970782120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e40d2000 0x7f95e415deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e9200700 1 -- 172.21.15.12:0/970782120 >> v2:172.21.15.159:3300/0 conn(0x7f95e40d1630 msgr2=0x7f95e40d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e9200700 1 --2- 172.21.15.12:0/970782120 >> v2:172.21.15.159:3300/0 conn(0x7f95e40d1630 0x7f95e40d1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e9200700 1 -- 172.21.15.12:0/970782120 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e40d2470 con 0x7f95e415dad0 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e3e00700 1 --2- 172.21.15.12:0/970782120 >> v2:172.21.15.159:3300/0 conn(0x7f95e40d1630 0x7f95e40d1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.936+0000 7f95e9200700 1 --2- 172.21.15.12:0/970782120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e415dad0 0x7f95e40ced80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f95e408b730 tx=0x7f95d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b840580e304e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:05.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95e3400700 1 -- 172.21.15.12:0/970782120 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d800b870 con 0x7f95e415dad0 2024-03-29T10:07:05.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95e3400700 1 -- 172.21.15.12:0/970782120 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d800b9d0 con 0x7f95e415dad0 2024-03-29T10:07:05.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95e3400700 1 -- 172.21.15.12:0/970782120 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d800bd60 con 0x7f95e415dad0 2024-03-29T10:07:05.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95ebc00700 1 -- 172.21.15.12:0/970782120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e415dad0 msgr2=0x7f95e40ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95ebc00700 1 --2- 172.21.15.12:0/970782120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e415dad0 0x7f95e40ced80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f95e408b730 tx=0x7f95d8009bd0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95ebc00700 1 -- 172.21.15.12:0/970782120 shutdown_connections 2024-03-29T10:07:05.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95ebc00700 1 --2- 172.21.15.12:0/970782120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e40d2000 0x7f95e415deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95ebc00700 1 --2- 172.21.15.12:0/970782120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e415dad0 0x7f95e40ced80 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95ebc00700 1 --2- 172.21.15.12:0/970782120 >> v2:172.21.15.159:3300/0 conn(0x7f95e40d1630 0x7f95e40d1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95ebc00700 1 -- 172.21.15.12:0/970782120 >> 172.21.15.12:0/970782120 conn(0x7f95e408aff0 msgr2=0x7f95e40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95ebc00700 1 -- 172.21.15.12:0/970782120 shutdown_connections 2024-03-29T10:07:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.937+0000 7f95ebc00700 1 -- 172.21.15.12:0/970782120 wait complete. 2024-03-29T10:07:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.939+0000 7f95ebc00700 1 Processor -- start 2024-03-29T10:07:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.939+0000 7f95ebc00700 1 -- start start 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95ebc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95e40d1630 0x7f95e409fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95ebc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e40d2000 0x7f95e409d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95ebc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e415dad0 0x7f95e40ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95ebc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e40b6370 con 0x7f95e40d2000 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95ebc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e414ca90 con 0x7f95e40d1630 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95ebc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e414cd70 con 0x7f95e415dad0 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95e3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e40d2000 0x7f95e409d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95e3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e40d2000 0x7f95e409d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45222/0 (socket says 172.21.15.12:45222) 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95e3e00700 1 -- 172.21.15.12:0/3749065017 learned_addr learned my addr 172.21.15.12:0/3749065017 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95e9c00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e415dad0 0x7f95e40ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.940+0000 7f95e9200700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95e40d1630 0x7f95e409fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e3e00700 1 -- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e415dad0 msgr2=0x7f95e40ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e3e00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e415dad0 0x7f95e40ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e3e00700 1 -- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95e40d1630 msgr2=0x7f95e409fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e3e00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95e40d1630 0x7f95e409fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e3e00700 1 -- 172.21.15.12:0/3749065017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d8009550 con 0x7f95e40d2000 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e9200700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95e40d1630 0x7f95e409fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e9c00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e415dad0 0x7f95e40ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e3e00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e40d2000 0x7f95e409d390 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f95d000d800 tx=0x7f95d000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e0c00700 1 -- 172.21.15.12:0/3749065017 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0004030 con 0x7f95e40d2000 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e0c00700 1 -- 172.21.15.12:0/3749065017 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d0004190 con 0x7f95e40d2000 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95e0c00700 1 -- 172.21.15.12:0/3749065017 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0004520 con 0x7f95e40d2000 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95ebc00700 1 -- 172.21.15.12:0/3749065017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e414d050 con 0x7f95e40d2000 2024-03-29T10:07:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.941+0000 7f95ebc00700 1 -- 172.21.15.12:0/3749065017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e40a0400 con 0x7f95e40d2000 2024-03-29T10:07:05.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.943+0000 7f95ebc00700 1 -- 172.21.15.12:0/3749065017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e4067720 con 0x7f95e40d2000 2024-03-29T10:07:05.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.943+0000 7f95e0c00700 1 -- 172.21.15.12:0/3749065017 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f95d00036a0 con 0x7f95e40d2000 2024-03-29T10:07:05.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.944+0000 7f95e0c00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95c406adb0 0x7f95c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:05.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.944+0000 7f95e0c00700 1 -- 172.21.15.12:0/3749065017 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f95d0089620 con 0x7f95e40d2000 2024-03-29T10:07:05.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.945+0000 7f95e9200700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95c406adb0 0x7f95c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:05.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.946+0000 7f95e9200700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95c406adb0 0x7f95c406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f95d8014040 tx=0x7f95d8008030 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:05.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:05.949+0000 7f95e0c00700 1 -- 172.21.15.12:0/3749065017 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f95d0054ad0 con 0x7f95e40d2000 2024-03-29T10:07:06.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.165+0000 7f95ebc00700 1 -- 172.21.15.12:0/3749065017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f95e40b6130 con 0x7f95e40d2000 2024-03-29T10:07:06.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.165+0000 7f95e0c00700 1 -- 172.21.15.12:0/3749065017 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f95d00580f0 con 0x7f95e40d2000 2024-03-29T10:07:06.165 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:07:06.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.167+0000 7f95ebc00700 1 -- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95c406adb0 msgr2=0x7f95c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.167+0000 7f95ebc00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95c406adb0 0x7f95c406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f95d8014040 tx=0x7f95d8008030 comp rx=0 tx=0).stop 2024-03-29T10:07:06.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.167+0000 7f95ebc00700 1 -- 172.21.15.12:0/3749065017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e40d2000 msgr2=0x7f95e409d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.167+0000 7f95ebc00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e40d2000 0x7f95e409d390 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f95d000d800 tx=0x7f95d000dbc0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.167+0000 7f95ebc00700 1 -- 172.21.15.12:0/3749065017 shutdown_connections 2024-03-29T10:07:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.167+0000 7f95ebc00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95c406adb0 0x7f95c406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.167+0000 7f95ebc00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95e415dad0 0x7f95e40ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.167+0000 7f95ebc00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95e40d1630 0x7f95e409fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.167+0000 7f95ebc00700 1 --2- 172.21.15.12:0/3749065017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e40d2000 0x7f95e409d390 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.167+0000 7f95ebc00700 1 -- 172.21.15.12:0/3749065017 >> 172.21.15.12:0/3749065017 conn(0x7f95e408aff0 msgr2=0x7f95e40cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.168+0000 7f95ebc00700 1 -- 172.21.15.12:0/3749065017 shutdown_connections 2024-03-29T10:07:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.168+0000 7f95ebc00700 1 -- 172.21.15.12:0/3749065017 wait complete. 2024-03-29T10:07:06.186 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T10:07:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.384+0000 7ff7d8e00700 1 Processor -- start 2024-03-29T10:07:06.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.384+0000 7ff7d8e00700 1 -- start start 2024-03-29T10:07:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.385+0000 7ff7d8e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7d406a790 0x7ff7d406ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.386+0000 7ff7d8e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d406f850 con 0x7ff7d40cebd0 2024-03-29T10:07:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.386+0000 7ff7d8e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d406e3f0 con 0x7ff7d40afbe0 2024-03-29T10:07:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.386+0000 7ff7d8e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d40cf6f0 con 0x7ff7d406a790 2024-03-29T10:07:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.386+0000 7ff7d1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7d406a790 0x7ff7d406ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.386+0000 7ff7d1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7d406a790 0x7ff7d406ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45236/0 (socket says 172.21.15.12:45236) 2024-03-29T10:07:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.386+0000 7ff7d1e00700 1 -- 172.21.15.12:0/1710450510 learned_addr learned my addr 172.21.15.12:0/1710450510 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.386+0000 7ff7d1e00700 1 -- 172.21.15.12:0/1710450510 >> v1:172.21.15.159:6789/0 conn(0x7ff7d40afbe0 legacy=0x7ff7d406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.386+0000 7ff7d1e00700 1 -- 172.21.15.12:0/1710450510 >> v1:172.21.15.12:6789/0 conn(0x7ff7d40cebd0 legacy=0x7ff7d40cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.386+0000 7ff7d1e00700 1 -- 172.21.15.12:0/1710450510 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d406b0b0 con 0x7ff7d406a790 2024-03-29T10:07:06.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.387+0000 7ff7d1e00700 1 --2- 172.21.15.12:0/1710450510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d406a790 0x7ff7d406ab70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40098f0 tx=0x7ff7c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62f746d524f2cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.387+0000 7ff7d0a00700 1 -- 172.21.15.12:0/1710450510 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4004230 con 0x7ff7d406a790 2024-03-29T10:07:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.387+0000 7ff7d0a00700 1 -- 172.21.15.12:0/1710450510 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c4004390 con 0x7ff7d406a790 2024-03-29T10:07:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.387+0000 7ff7d0a00700 1 -- 172.21.15.12:0/1710450510 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c40036a0 con 0x7ff7d406a790 2024-03-29T10:07:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.387+0000 7ff7d8e00700 1 -- 172.21.15.12:0/1710450510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d406a790 msgr2=0x7ff7d406ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.387+0000 7ff7d8e00700 1 --2- 172.21.15.12:0/1710450510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d406a790 0x7ff7d406ab70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40098f0 tx=0x7ff7c4009c00 comp rx=0 tx=0).stop 2024-03-29T10:07:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.387+0000 7ff7d8e00700 1 -- 172.21.15.12:0/1710450510 shutdown_connections 2024-03-29T10:07:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.387+0000 7ff7d8e00700 1 --2- 172.21.15.12:0/1710450510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d406a790 0x7ff7d406ab70 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.387+0000 7ff7d8e00700 1 -- 172.21.15.12:0/1710450510 >> 172.21.15.12:0/1710450510 conn(0x7ff7d40b5ab0 msgr2=0x7ff7d40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.387+0000 7ff7d8e00700 1 -- 172.21.15.12:0/1710450510 shutdown_connections 2024-03-29T10:07:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.388+0000 7ff7d8e00700 1 -- 172.21.15.12:0/1710450510 wait complete. 2024-03-29T10:07:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.389+0000 7ff7d8e00700 1 Processor -- start 2024-03-29T10:07:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.389+0000 7ff7d8e00700 1 -- start start 2024-03-29T10:07:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.390+0000 7ff7d8e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7d406a790 0x7ff7d409b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.390+0000 7ff7d8e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d40afbe0 0x7ff7d40caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.390+0000 7ff7d8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d40cebd0 0x7ff7d40c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.390+0000 7ff7d8e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40cb0e0 con 0x7ff7d40cebd0 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.390+0000 7ff7d8e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40cc4b0 con 0x7ff7d406a790 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.390+0000 7ff7d8e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d41f3370 con 0x7ff7d40afbe0 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.390+0000 7ff7d2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d40cebd0 0x7ff7d40c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d40cebd0 0x7ff7d40c9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45244/0 (socket says 172.21.15.12:45244) 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d2800700 1 -- 172.21.15.12:0/2520237353 learned_addr learned my addr 172.21.15.12:0/2520237353 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d1e00700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7d406a790 0x7ff7d409b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d1400700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d40afbe0 0x7ff7d40caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d2800700 1 -- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d40afbe0 msgr2=0x7ff7d40caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d2800700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d40afbe0 0x7ff7d40caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d2800700 1 -- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7d406a790 msgr2=0x7ff7d409b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d2800700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7d406a790 0x7ff7d409b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d2800700 1 -- 172.21.15.12:0/2520237353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c4009550 con 0x7ff7d40cebd0 2024-03-29T10:07:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d1400700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d40afbe0 0x7ff7d40caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d2800700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d40cebd0 0x7ff7d40c9aa0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00cc50 tx=0x7ff7cc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7ca000700 1 -- 172.21.15.12:0/2520237353 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc003c00 con 0x7ff7d40cebd0 2024-03-29T10:07:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7ca000700 1 -- 172.21.15.12:0/2520237353 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc003d60 con 0x7ff7d40cebd0 2024-03-29T10:07:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7d8e00700 1 -- 172.21.15.12:0/2520237353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d41f3650 con 0x7ff7d40cebd0 2024-03-29T10:07:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.391+0000 7ff7ca000700 1 -- 172.21.15.12:0/2520237353 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc00ab60 con 0x7ff7d40cebd0 2024-03-29T10:07:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.392+0000 7ff7d8e00700 1 -- 172.21.15.12:0/2520237353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d41f3c10 con 0x7ff7d40cebd0 2024-03-29T10:07:06.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.392+0000 7ff7d8e00700 1 -- 172.21.15.12:0/2520237353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d4067720 con 0x7ff7d40cebd0 2024-03-29T10:07:06.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.393+0000 7ff7ca000700 1 -- 172.21.15.12:0/2520237353 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff7cc00acc0 con 0x7ff7d40cebd0 2024-03-29T10:07:06.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.395+0000 7ff7ca000700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7b406adb0 0x7ff7b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.395+0000 7ff7ca000700 1 -- 172.21.15.12:0/2520237353 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff7cc089240 con 0x7ff7d40cebd0 2024-03-29T10:07:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.395+0000 7ff7d1e00700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7b406adb0 0x7ff7b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.396+0000 7ff7d1e00700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7b406adb0 0x7ff7b406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009130 tx=0x7ff7c4008030 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.399+0000 7ff7ca000700 1 -- 172.21.15.12:0/2520237353 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff7cc0545c0 con 0x7ff7d40cebd0 2024-03-29T10:07:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.613+0000 7ff7d8e00700 1 -- 172.21.15.12:0/2520237353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff7d41f3da0 con 0x7ff7d40cebd0 2024-03-29T10:07:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.614+0000 7ff7ca000700 1 -- 172.21.15.12:0/2520237353 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v57) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff7cc057be0 con 0x7ff7d40cebd0 2024-03-29T10:07:06.614 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:06.614 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:07:06.614 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:06.614 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T10:07:06.614 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T10:07:06.614 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:06.614 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:07:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 -- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7b406adb0 msgr2=0x7ff7b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7b406adb0 0x7ff7b406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009130 tx=0x7ff7c4008030 comp rx=0 tx=0).stop 2024-03-29T10:07:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 -- 172.21.15.12:0/2520237353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d40cebd0 msgr2=0x7ff7d40c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d40cebd0 0x7ff7d40c9aa0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00cc50 tx=0x7ff7cc00cf60 comp rx=0 tx=0).stop 2024-03-29T10:07:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 -- 172.21.15.12:0/2520237353 shutdown_connections 2024-03-29T10:07:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7b406adb0 0x7ff7b406d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7d40cebd0 0x7ff7d40c9aa0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7d406a790 0x7ff7d409b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 --2- 172.21.15.12:0/2520237353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d40afbe0 0x7ff7d40caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 -- 172.21.15.12:0/2520237353 >> 172.21.15.12:0/2520237353 conn(0x7ff7d40b5ab0 msgr2=0x7ff7d415c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 -- 172.21.15.12:0/2520237353 shutdown_connections 2024-03-29T10:07:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.616+0000 7ff7d8e00700 1 -- 172.21.15.12:0/2520237353 wait complete. 2024-03-29T10:07:06.634 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T10:07:06.634 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T10:07:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.837+0000 7fcbca600700 1 Processor -- start 2024-03-29T10:07:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.837+0000 7fcbca600700 1 -- start start 2024-03-29T10:07:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.838+0000 7fcbca600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcbc415ddf0 0x7fcbc40ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.838+0000 7fcbca600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcbc40d28e0 0x7fcbc415f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.838+0000 7fcbca600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc40ce8a0 con 0x7fcbc415e1d0 2024-03-29T10:07:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.838+0000 7fcbca600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc415ec60 con 0x7fcbc415ddf0 2024-03-29T10:07:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.838+0000 7fcbca600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc40cdc90 con 0x7fcbc40d28e0 2024-03-29T10:07:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.838+0000 7fcbc3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcbc415ddf0 0x7fcbc40ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.838+0000 7fcbc3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcbc415ddf0 0x7fcbc40ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45252/0 (socket says 172.21.15.12:45252) 2024-03-29T10:07:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.838+0000 7fcbc3400700 1 -- 172.21.15.12:0/3413044098 learned_addr learned my addr 172.21.15.12:0/3413044098 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.838+0000 7fcbc3e00700 1 --2- 172.21.15.12:0/3413044098 >> v2:172.21.15.159:3300/0 conn(0x7fcbc40d28e0 0x7fcbc415f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.839+0000 7fcbc3400700 1 -- 172.21.15.12:0/3413044098 >> v2:172.21.15.159:3300/0 conn(0x7fcbc40d28e0 msgr2=0x7fcbc415f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.839+0000 7fcbc3400700 1 --2- 172.21.15.12:0/3413044098 >> v2:172.21.15.159:3300/0 conn(0x7fcbc40d28e0 0x7fcbc415f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.839+0000 7fcbc3400700 1 -- 172.21.15.12:0/3413044098 >> v1:172.21.15.159:6789/0 conn(0x7fcbc415e1d0 legacy=0x7fcbc40d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.839+0000 7fcbc3400700 1 -- 172.21.15.12:0/3413044098 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc40cdf70 con 0x7fcbc415ddf0 2024-03-29T10:07:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.839+0000 7fcbc2000700 1 -- 172.21.15.12:0/3413044098 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976337808 0 0) 0x7fcbc40ce8a0 con 0x7fcbc415e1d0 2024-03-29T10:07:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.839+0000 7fcbc3400700 1 --2- 172.21.15.12:0/3413044098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 0x7fcbc40ced30 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40ca570 tx=0x7fcbb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568004a1036118dd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.839+0000 7fcbc2000700 1 -- 172.21.15.12:0/3413044098 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4011830 con 0x7fcbc415ddf0 2024-03-29T10:07:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.839+0000 7fcbc2000700 1 -- 172.21.15.12:0/3413044098 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb4011990 con 0x7fcbc415ddf0 2024-03-29T10:07:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.839+0000 7fcbc2000700 1 -- 172.21.15.12:0/3413044098 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4011d20 con 0x7fcbc415ddf0 2024-03-29T10:07:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.840+0000 7fcbca600700 1 -- 172.21.15.12:0/3413044098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 msgr2=0x7fcbc40ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.840+0000 7fcbca600700 1 --2- 172.21.15.12:0/3413044098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 0x7fcbc40ced30 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40ca570 tx=0x7fcbb4009bd0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.840+0000 7fcbca600700 1 -- 172.21.15.12:0/3413044098 shutdown_connections 2024-03-29T10:07:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.840+0000 7fcbca600700 1 --2- 172.21.15.12:0/3413044098 >> v2:172.21.15.159:3300/0 conn(0x7fcbc40d28e0 0x7fcbc415f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.840+0000 7fcbca600700 1 --2- 172.21.15.12:0/3413044098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 0x7fcbc40ced30 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.840+0000 7fcbca600700 1 -- 172.21.15.12:0/3413044098 >> 172.21.15.12:0/3413044098 conn(0x7fcbc408aff0 msgr2=0x7fcbc40d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.840+0000 7fcbca600700 1 -- 172.21.15.12:0/3413044098 shutdown_connections 2024-03-29T10:07:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.840+0000 7fcbca600700 1 -- 172.21.15.12:0/3413044098 wait complete. 2024-03-29T10:07:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.842+0000 7fcbca600700 1 Processor -- start 2024-03-29T10:07:06.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.842+0000 7fcbca600700 1 -- start start 2024-03-29T10:07:06.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbca600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 0x7fcbc41f44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbca600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcbc415e1d0 0x7fcbc415c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbca600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcbc40c9600 0x7fcbc40c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbca600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40c7020 con 0x7fcbc415ddf0 2024-03-29T10:07:06.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbca600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40c7b60 con 0x7fcbc40c9600 2024-03-29T10:07:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbca600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40c9f80 con 0x7fcbc415e1d0 2024-03-29T10:07:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbc3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 0x7fcbc41f44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbc3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 0x7fcbc41f44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45262/0 (socket says 172.21.15.12:45262) 2024-03-29T10:07:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbc3400700 1 -- 172.21.15.12:0/146738843 learned_addr learned my addr 172.21.15.12:0/146738843 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbc3e00700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcbc40c9600 0x7fcbc40c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.843+0000 7fcbc2a00700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcbc415e1d0 0x7fcbc415c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbc3400700 1 -- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcbc415e1d0 msgr2=0x7fcbc415c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbc3400700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcbc415e1d0 0x7fcbc415c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbc3400700 1 -- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcbc40c9600 msgr2=0x7fcbc40c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbc3400700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcbc40c9600 0x7fcbc40c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:06.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbc3400700 1 -- 172.21.15.12:0/146738843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb4009550 con 0x7fcbc415ddf0 2024-03-29T10:07:06.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbc2a00700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcbc415e1d0 0x7fcbc415c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:06.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbc3400700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 0x7fcbc41f44b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009c10 tx=0x7fcbb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:06.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbbb400700 1 -- 172.21.15.12:0/146738843 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb40036a0 con 0x7fcbc415ddf0 2024-03-29T10:07:06.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbbb400700 1 -- 172.21.15.12:0/146738843 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb4003800 con 0x7fcbc415ddf0 2024-03-29T10:07:06.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbbb400700 1 -- 172.21.15.12:0/146738843 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4003b70 con 0x7fcbc415ddf0 2024-03-29T10:07:06.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbca600700 1 -- 172.21.15.12:0/146738843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc40c7230 con 0x7fcbc415ddf0 2024-03-29T10:07:06.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.844+0000 7fcbca600700 1 -- 172.21.15.12:0/146738843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc40c77f0 con 0x7fcbc415ddf0 2024-03-29T10:07:06.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.845+0000 7fcbca600700 1 -- 172.21.15.12:0/146738843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc4067720 con 0x7fcbc415ddf0 2024-03-29T10:07:06.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.846+0000 7fcbbb400700 1 -- 172.21.15.12:0/146738843 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcbb4003cd0 con 0x7fcbc415ddf0 2024-03-29T10:07:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.847+0000 7fcbbb400700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcba406adb0 0x7fcba406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.847+0000 7fcbbb400700 1 -- 172.21.15.12:0/146738843 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcbb4021030 con 0x7fcbc415ddf0 2024-03-29T10:07:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.847+0000 7fcbc2a00700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcba406adb0 0x7fcba406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.848+0000 7fcbc2a00700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcba406adb0 0x7fcba406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcbc406ccb0 tx=0x7fcbb0009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:06.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:06.852+0000 7fcbbb400700 1 -- 172.21.15.12:0/146738843 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcbb4055c70 con 0x7fcbc415ddf0 2024-03-29T10:07:07.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.064+0000 7fcbca600700 1 -- 172.21.15.12:0/146738843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcbc40d3050 con 0x7fcbc415ddf0 2024-03-29T10:07:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.172+0000 7fcbbb400700 1 -- 172.21.15.12:0/146738843 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v58) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcbb402b0b0 con 0x7fcbc415ddf0 2024-03-29T10:07:07.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.175+0000 7fcbca600700 1 -- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcba406adb0 msgr2=0x7fcba406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:07.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.175+0000 7fcbca600700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcba406adb0 0x7fcba406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcbc406ccb0 tx=0x7fcbb0009040 comp rx=0 tx=0).stop 2024-03-29T10:07:07.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.175+0000 7fcbca600700 1 -- 172.21.15.12:0/146738843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 msgr2=0x7fcbc41f44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:07.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.175+0000 7fcbca600700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 0x7fcbc41f44b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009c10 tx=0x7fcbb4004750 comp rx=0 tx=0).stop 2024-03-29T10:07:07.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.176+0000 7fcbca600700 1 -- 172.21.15.12:0/146738843 shutdown_connections 2024-03-29T10:07:07.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.176+0000 7fcbca600700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcba406adb0 0x7fcba406d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:07.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.176+0000 7fcbca600700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcbc40c9600 0x7fcbc40c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:07.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.176+0000 7fcbca600700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbc415ddf0 0x7fcbc41f44b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:07.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.176+0000 7fcbca600700 1 --2- 172.21.15.12:0/146738843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcbc415e1d0 0x7fcbc415c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:07.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.176+0000 7fcbca600700 1 -- 172.21.15.12:0/146738843 >> 172.21.15.12:0/146738843 conn(0x7fcbc408aff0 msgr2=0x7fcbc40d2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:07.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.176+0000 7fcbca600700 1 -- 172.21.15.12:0/146738843 shutdown_connections 2024-03-29T10:07:07.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.176+0000 7fcbca600700 1 -- 172.21.15.12:0/146738843 wait complete. 2024-03-29T10:07:07.178 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T10:07:07.197 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T10:07:07.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.402+0000 7fd431000700 1 Processor -- start 2024-03-29T10:07:07.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.402+0000 7fd431000700 1 -- start start 2024-03-29T10:07:07.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.403+0000 7fd431000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd42c15d9d0 0x7fd42c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.403+0000 7fd431000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd42c0d16a0 0x7fd42c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.403+0000 7fd431000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42c161420 con 0x7fd42c0d2370 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.403+0000 7fd431000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd42c0d2820 con 0x7fd42c15d9d0 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.403+0000 7fd431000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c161700 con 0x7fd42c0d16a0 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.404+0000 7fd429e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd42c15d9d0 0x7fd42c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.404+0000 7fd429e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd42c15d9d0 0x7fd42c0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45270/0 (socket says 172.21.15.12:45270) 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.404+0000 7fd429e00700 1 -- 172.21.15.12:0/4210005314 learned_addr learned my addr 172.21.15.12:0/4210005314 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.404+0000 7fd429400700 1 --2- 172.21.15.12:0/4210005314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd42c0d16a0 0x7fd42c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.404+0000 7fd429e00700 1 -- 172.21.15.12:0/4210005314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd42c0d16a0 msgr2=0x7fd42c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.404+0000 7fd429e00700 1 --2- 172.21.15.12:0/4210005314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd42c0d16a0 0x7fd42c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.404+0000 7fd429e00700 1 -- 172.21.15.12:0/4210005314 >> v1:172.21.15.12:6789/0 conn(0x7fd42c0d2370 legacy=0x7fd42c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.404+0000 7fd429e00700 1 -- 172.21.15.12:0/4210005314 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c1619e0 con 0x7fd42c15d9d0 2024-03-29T10:07:07.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.404+0000 7fd429e00700 1 --2- 172.21.15.12:0/4210005314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c15d9d0 0x7fd42c0d3c30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0098f0 tx=0x7fd41c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74dc2ba6720028e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:07.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd428a00700 1 -- 172.21.15.12:0/4210005314 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41c011830 con 0x7fd42c15d9d0 2024-03-29T10:07:07.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd428a00700 1 -- 172.21.15.12:0/4210005314 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41c011990 con 0x7fd42c15d9d0 2024-03-29T10:07:07.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd428a00700 1 -- 172.21.15.12:0/4210005314 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41c011d20 con 0x7fd42c15d9d0 2024-03-29T10:07:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd431000700 1 -- 172.21.15.12:0/4210005314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c15d9d0 msgr2=0x7fd42c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd431000700 1 --2- 172.21.15.12:0/4210005314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c15d9d0 0x7fd42c0d3c30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0098f0 tx=0x7fd41c009c00 comp rx=0 tx=0).stop 2024-03-29T10:07:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd431000700 1 -- 172.21.15.12:0/4210005314 shutdown_connections 2024-03-29T10:07:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd431000700 1 --2- 172.21.15.12:0/4210005314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c15d9d0 0x7fd42c0d3c30 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd431000700 1 --2- 172.21.15.12:0/4210005314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd42c0d16a0 0x7fd42c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd431000700 1 -- 172.21.15.12:0/4210005314 >> 172.21.15.12:0/4210005314 conn(0x7fd42c08aff0 msgr2=0x7fd42c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd431000700 1 -- 172.21.15.12:0/4210005314 shutdown_connections 2024-03-29T10:07:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.405+0000 7fd431000700 1 -- 172.21.15.12:0/4210005314 wait complete. 2024-03-29T10:07:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.407+0000 7fd431000700 1 Processor -- start 2024-03-29T10:07:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.407+0000 7fd431000700 1 -- start start 2024-03-29T10:07:07.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.408+0000 7fd431000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd42c0d16a0 0x7fd42c155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:07.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.408+0000 7fd431000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c0d2370 0x7fd42c0b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:07.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.408+0000 7fd431000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd42c15d9d0 0x7fd42c159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:07.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.408+0000 7fd431000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c15e810 con 0x7fd42c0d2370 2024-03-29T10:07:07.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.408+0000 7fd431000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c0b8df0 con 0x7fd42c15d9d0 2024-03-29T10:07:07.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.408+0000 7fd431000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c15dee0 con 0x7fd42c0d16a0 2024-03-29T10:07:07.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.408+0000 7fd429400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c0d2370 0x7fd42c0b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:07.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.408+0000 7fd429400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c0d2370 0x7fd42c0b8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45274/0 (socket says 172.21.15.12:45274) 2024-03-29T10:07:07.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.408+0000 7fd429400700 1 -- 172.21.15.12:0/2375237666 learned_addr learned my addr 172.21.15.12:0/2375237666 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:07.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd429e00700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd42c0d16a0 0x7fd42c155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:07.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd42a800700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd42c15d9d0 0x7fd42c159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd429400700 1 -- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd42c0d16a0 msgr2=0x7fd42c155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd429400700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd42c0d16a0 0x7fd42c155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd429400700 1 -- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd42c15d9d0 msgr2=0x7fd42c159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd429400700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd42c15d9d0 0x7fd42c159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd429400700 1 -- 172.21.15.12:0/2375237666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41c009550 con 0x7fd42c0d2370 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd42a800700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd42c15d9d0 0x7fd42c159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd429400700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c0d2370 0x7fd42c0b8480 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd41800e960 tx=0x7fd41800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd422000700 1 -- 172.21.15.12:0/2375237666 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4180096f0 con 0x7fd42c0d2370 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd422000700 1 -- 172.21.15.12:0/2375237666 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41800eeb0 con 0x7fd42c0d2370 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd431000700 1 -- 172.21.15.12:0/2375237666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd42c15e1c0 con 0x7fd42c0d2370 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.409+0000 7fd422000700 1 -- 172.21.15.12:0/2375237666 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41800ca70 con 0x7fd42c0d2370 2024-03-29T10:07:07.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.410+0000 7fd431000700 1 -- 172.21.15.12:0/2375237666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd42c155560 con 0x7fd42c0d2370 2024-03-29T10:07:07.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.410+0000 7fd431000700 1 -- 172.21.15.12:0/2375237666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd42c0d1ca0 con 0x7fd42c0d2370 2024-03-29T10:07:07.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.412+0000 7fd422000700 1 -- 172.21.15.12:0/2375237666 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd41800cbd0 con 0x7fd42c0d2370 2024-03-29T10:07:07.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.413+0000 7fd422000700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd40c06adb0 0x7fd40c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:07.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.413+0000 7fd422000700 1 -- 172.21.15.12:0/2375237666 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd418089a10 con 0x7fd42c0d2370 2024-03-29T10:07:07.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.414+0000 7fd429e00700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd40c06adb0 0x7fd40c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:07.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.414+0000 7fd429e00700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd40c06adb0 0x7fd40c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd41c014040 tx=0x7fd41c008030 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:07.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.417+0000 7fd422000700 1 -- 172.21.15.12:0/2375237666 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd418054e30 con 0x7fd42c0d2370 2024-03-29T10:07:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:07.628+0000 7fd431000700 1 -- 172.21.15.12:0/2375237666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd42c067720 con 0x7fd42c0d2370 2024-03-29T10:07:08.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.187+0000 7fd422000700 1 -- 172.21.15.12:0/2375237666 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v59) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd4180210d0 con 0x7fd42c0d2370 2024-03-29T10:07:08.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.194+0000 7fd431000700 1 -- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd40c06adb0 msgr2=0x7fd40c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.194+0000 7fd431000700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd40c06adb0 0x7fd40c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd41c014040 tx=0x7fd41c008030 comp rx=0 tx=0).stop 2024-03-29T10:07:08.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.194+0000 7fd431000700 1 -- 172.21.15.12:0/2375237666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c0d2370 msgr2=0x7fd42c0b8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.194+0000 7fd431000700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c0d2370 0x7fd42c0b8480 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd41800e960 tx=0x7fd41800ec70 comp rx=0 tx=0).stop 2024-03-29T10:07:08.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.195+0000 7fd431000700 1 -- 172.21.15.12:0/2375237666 shutdown_connections 2024-03-29T10:07:08.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.195+0000 7fd431000700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd40c06adb0 0x7fd40c06d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.195+0000 7fd431000700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd42c15d9d0 0x7fd42c159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.195+0000 7fd431000700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd42c0d16a0 0x7fd42c155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.195+0000 7fd431000700 1 --2- 172.21.15.12:0/2375237666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd42c0d2370 0x7fd42c0b8480 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.195+0000 7fd431000700 1 -- 172.21.15.12:0/2375237666 >> 172.21.15.12:0/2375237666 conn(0x7fd42c08aff0 msgr2=0x7fd42c0b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:08.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.197+0000 7fd431000700 1 -- 172.21.15.12:0/2375237666 shutdown_connections 2024-03-29T10:07:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.197+0000 7fd431000700 1 -- 172.21.15.12:0/2375237666 wait complete. 2024-03-29T10:07:08.207 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T10:07:08.225 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T10:07:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.428+0000 7f07ccc00700 1 Processor -- start 2024-03-29T10:07:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.428+0000 7f07ccc00700 1 -- start start 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07ccc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07c815d9d0 0x7f07c80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07ccc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f07c80d16a0 0x7f07c80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07ccc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07c80d2370 0x7f07c815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07ccc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c80d2820 con 0x7f07c80d2370 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07ccc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c8161420 con 0x7f07c80d16a0 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07ccc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8161700 con 0x7f07c815d9d0 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07c6800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07c80d2370 0x7f07c815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07c6800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07c80d2370 0x7f07c815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45288/0 (socket says 172.21.15.12:45288) 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07c6800700 1 -- 172.21.15.12:0/4216801809 learned_addr learned my addr 172.21.15.12:0/4216801809 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07c5e00700 1 --2- 172.21.15.12:0/4216801809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07c815d9d0 0x7f07c80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.429+0000 7f07c5400700 1 --2- 172.21.15.12:0/4216801809 >> v2:172.21.15.159:3300/0 conn(0x7f07c80d16a0 0x7f07c80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.430+0000 7f07c6800700 1 -- 172.21.15.12:0/4216801809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07c815d9d0 msgr2=0x7f07c80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.430+0000 7f07c6800700 1 --2- 172.21.15.12:0/4216801809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07c815d9d0 0x7f07c80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.430+0000 7f07c6800700 1 -- 172.21.15.12:0/4216801809 >> v2:172.21.15.159:3300/0 conn(0x7f07c80d16a0 msgr2=0x7f07c80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.430+0000 7f07c6800700 1 --2- 172.21.15.12:0/4216801809 >> v2:172.21.15.159:3300/0 conn(0x7f07c80d16a0 0x7f07c80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.430+0000 7f07c6800700 1 -- 172.21.15.12:0/4216801809 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c81619e0 con 0x7f07c80d2370 2024-03-29T10:07:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.430+0000 7f07c6800700 1 --2- 172.21.15.12:0/4216801809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 0x7f07c815ddb0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f07c806b5e0 tx=0x7f07c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba091ae26f8d31f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.430+0000 7f07c4a00700 1 -- 172.21.15.12:0/4216801809 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c000e070 con 0x7f07c80d2370 2024-03-29T10:07:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.430+0000 7f07c4a00700 1 -- 172.21.15.12:0/4216801809 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c000b910 con 0x7f07c80d2370 2024-03-29T10:07:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.430+0000 7f07c4a00700 1 -- 172.21.15.12:0/4216801809 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c000bcc0 con 0x7f07c80d2370 2024-03-29T10:07:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.431+0000 7f07ccc00700 1 -- 172.21.15.12:0/4216801809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 msgr2=0x7f07c815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.431+0000 7f07ccc00700 1 --2- 172.21.15.12:0/4216801809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 0x7f07c815ddb0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f07c806b5e0 tx=0x7f07c000b330 comp rx=0 tx=0).stop 2024-03-29T10:07:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.431+0000 7f07ccc00700 1 -- 172.21.15.12:0/4216801809 shutdown_connections 2024-03-29T10:07:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.431+0000 7f07ccc00700 1 --2- 172.21.15.12:0/4216801809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 0x7f07c815ddb0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.431+0000 7f07ccc00700 1 --2- 172.21.15.12:0/4216801809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07c815d9d0 0x7f07c80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.431+0000 7f07ccc00700 1 --2- 172.21.15.12:0/4216801809 >> v2:172.21.15.159:3300/0 conn(0x7f07c80d16a0 0x7f07c80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.431+0000 7f07ccc00700 1 -- 172.21.15.12:0/4216801809 >> 172.21.15.12:0/4216801809 conn(0x7f07c808aff0 msgr2=0x7f07c80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.431+0000 7f07ccc00700 1 -- 172.21.15.12:0/4216801809 shutdown_connections 2024-03-29T10:07:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.431+0000 7f07ccc00700 1 -- 172.21.15.12:0/4216801809 wait complete. 2024-03-29T10:07:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.433+0000 7f07ccc00700 1 Processor -- start 2024-03-29T10:07:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.433+0000 7f07ccc00700 1 -- start start 2024-03-29T10:07:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07ccc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07c80d16a0 0x7f07c80ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07ccc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 0x7f07c81e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07ccc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07c815d9d0 0x7f07c81e77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07ccc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80cc0e0 con 0x7f07c80d2370 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07ccc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80cb040 con 0x7f07c815d9d0 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07ccc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80cb320 con 0x7f07c80d16a0 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07c5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 0x7f07c81e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07c5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 0x7f07c81e7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45296/0 (socket says 172.21.15.12:45296) 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07c5400700 1 -- 172.21.15.12:0/3220415173 learned_addr learned my addr 172.21.15.12:0/3220415173 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07c5e00700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07c80d16a0 0x7f07c80ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07c6800700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07c815d9d0 0x7f07c81e77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07c5400700 1 -- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07c80d16a0 msgr2=0x7f07c80ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07c5400700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07c80d16a0 0x7f07c80ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07c5400700 1 -- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07c815d9d0 msgr2=0x7f07c81e77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07c5400700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07c815d9d0 0x7f07c81e77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.435+0000 7f07c5400700 1 -- 172.21.15.12:0/3220415173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c0009b30 con 0x7f07c80d2370 2024-03-29T10:07:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.436+0000 7f07c5400700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 0x7f07c81e7280 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f07b400e810 tx=0x7f07b400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.436+0000 7f07be000700 1 -- 172.21.15.12:0/3220415173 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b4013610 con 0x7f07c80d2370 2024-03-29T10:07:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.436+0000 7f07be000700 1 -- 172.21.15.12:0/3220415173 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b4013770 con 0x7f07c80d2370 2024-03-29T10:07:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.436+0000 7f07be000700 1 -- 172.21.15.12:0/3220415173 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b4013b00 con 0x7f07c80d2370 2024-03-29T10:07:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.436+0000 7f07ccc00700 1 -- 172.21.15.12:0/3220415173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c80cb600 con 0x7f07c80d2370 2024-03-29T10:07:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.436+0000 7f07ccc00700 1 -- 172.21.15.12:0/3220415173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c80cbbc0 con 0x7f07c80d2370 2024-03-29T10:07:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.437+0000 7f07ccc00700 1 -- 172.21.15.12:0/3220415173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c80cbd80 con 0x7f07c80d2370 2024-03-29T10:07:08.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.438+0000 7f07be000700 1 -- 172.21.15.12:0/3220415173 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f07b4013c60 con 0x7f07c80d2370 2024-03-29T10:07:08.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.439+0000 7f07be000700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07a806adb0 0x7f07a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.439+0000 7f07be000700 1 -- 172.21.15.12:0/3220415173 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f07b40891f0 con 0x7f07c80d2370 2024-03-29T10:07:08.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.440+0000 7f07c5e00700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07a806adb0 0x7f07a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.440+0000 7f07c5e00700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07a806adb0 0x7f07a806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f07c80cd5f0 tx=0x7f07b8008040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:08.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.444+0000 7f07be000700 1 -- 172.21.15.12:0/3220415173 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f07b4054180 con 0x7f07c80d2370 2024-03-29T10:07:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.704+0000 7f07ccc00700 1 -- 172.21.15.12:0/3220415173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f07c80d1ca0 con 0x7f07c80d2370 2024-03-29T10:07:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.711+0000 7f07be000700 1 -- 172.21.15.12:0/3220415173 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v19) v1 ==== 172+0+0 (secure 0 0 0) 0x7f07b40577a0 con 0x7f07c80d2370 2024-03-29T10:07:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.718+0000 7f07ccc00700 1 -- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07a806adb0 msgr2=0x7f07a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.718+0000 7f07ccc00700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07a806adb0 0x7f07a806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f07c80cd5f0 tx=0x7f07b8008040 comp rx=0 tx=0).stop 2024-03-29T10:07:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.718+0000 7f07ccc00700 1 -- 172.21.15.12:0/3220415173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 msgr2=0x7f07c81e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.718+0000 7f07ccc00700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 0x7f07c81e7280 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f07b400e810 tx=0x7f07b400ebd0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.720+0000 7f07ccc00700 1 -- 172.21.15.12:0/3220415173 shutdown_connections 2024-03-29T10:07:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.722+0000 7f07ccc00700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07a806adb0 0x7f07a806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.722+0000 7f07ccc00700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07c815d9d0 0x7f07c81e77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.722+0000 7f07ccc00700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07c80d16a0 0x7f07c80ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.722+0000 7f07ccc00700 1 --2- 172.21.15.12:0/3220415173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07c80d2370 0x7f07c81e7280 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.722+0000 7f07ccc00700 1 -- 172.21.15.12:0/3220415173 >> 172.21.15.12:0/3220415173 conn(0x7f07c808aff0 msgr2=0x7f07c80b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.723+0000 7f07ccc00700 1 -- 172.21.15.12:0/3220415173 shutdown_connections 2024-03-29T10:07:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.723+0000 7f07ccc00700 1 -- 172.21.15.12:0/3220415173 wait complete. 2024-03-29T10:07:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 6 and data pool 7 2024-03-29T10:07:08.756 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:07:08.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.954+0000 7f9a8f800700 1 Processor -- start 2024-03-29T10:07:08.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.954+0000 7f9a8f800700 1 -- start start 2024-03-29T10:07:08.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.955+0000 7f9a8f800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9a88160be0 0x7f9a880cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.955+0000 7f9a8f800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a88160fc0 con 0x7f9a880ceca0 2024-03-29T10:07:08.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.955+0000 7f9a8f800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a880d21e0 con 0x7f9a880ce0e0 2024-03-29T10:07:08.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.955+0000 7f9a8f800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a880b65d0 con 0x7f9a88160be0 2024-03-29T10:07:08.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.956+0000 7f9a8d800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9a880ceca0 0x7f9a880b6a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35854/0 (socket says 172.21.15.12:35854) 2024-03-29T10:07:08.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.956+0000 7f9a8d800700 1 -- 172.21.15.12:0/195182323 learned_addr learned my addr 172.21.15.12:0/195182323 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:08.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.956+0000 7f9a8ce00700 1 --2- 172.21.15.12:0/195182323 >> v2:172.21.15.159:3300/0 conn(0x7f9a88160be0 0x7f9a880cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.956+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452125334 0 0) 0x7f9a88160fc0 con 0x7f9a880ceca0 2024-03-29T10:07:08.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.956+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a700035b0 con 0x7f9a880ceca0 2024-03-29T10:07:08.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.956+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91861407 0 0) 0x7f9a880d21e0 con 0x7f9a880ce0e0 2024-03-29T10:07:08.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.956+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a88160fc0 con 0x7f9a880ce0e0 2024-03-29T10:07:08.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.956+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3601120521 0 0) 0x7f9a700035b0 con 0x7f9a880ceca0 2024-03-29T10:07:08.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 >> v2:172.21.15.159:3300/0 conn(0x7f9a88160be0 msgr2=0x7f9a880cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:08.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 --2- 172.21.15.12:0/195182323 >> v2:172.21.15.159:3300/0 conn(0x7f9a88160be0 0x7f9a880cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 >> v1:172.21.15.159:6789/0 conn(0x7f9a880ce0e0 legacy=0x7f9a880ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a880d24c0 con 0x7f9a880ceca0 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f9a800019f0 con 0x7f9a880ceca0 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a8ce00700 1 --2- 172.21.15.12:0/195182323 >> v2:172.21.15.159:3300/0 conn(0x7f9a88160be0 0x7f9a880cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 >> v1:172.21.15.12:6789/0 conn(0x7f9a880ceca0 legacy=0x7f9a880b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a70003b90 0x7f9a70003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a70004850 0x7f9a70004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a70005210 0x7f9a7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880b65d0 con 0x7f9a70004850 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7000c090 con 0x7f9a70003b90 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7000c250 con 0x7f9a70005210 2024-03-29T10:07:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a87e00700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a70004850 0x7f9a70004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a8ce00700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a70003b90 0x7f9a70003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.957+0000 7f9a8d800700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a70005210 0x7f9a7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.958+0000 7f9a87e00700 1 -- 172.21.15.12:0/195182323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a70005210 msgr2=0x7f9a7000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.958+0000 7f9a87e00700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a70005210 0x7f9a7000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.958+0000 7f9a87e00700 1 -- 172.21.15.12:0/195182323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a70003b90 msgr2=0x7f9a70003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.958+0000 7f9a87e00700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a70003b90 0x7f9a70003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.958+0000 7f9a87e00700 1 -- 172.21.15.12:0/195182323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a700035b0 con 0x7f9a70004850 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.958+0000 7f9a8d800700 1 -- 172.21.15.12:0/195182323 reap_dead start 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.958+0000 7f9a87e00700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a70004850 0x7f9a70004cd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9a74001940 tx=0x7f9a70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3994c438456a6032 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.958+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74001430 con 0x7f9a70004850 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.958+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a74001d70 con 0x7f9a70004850 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.958+0000 7f9a87400700 1 -- 172.21.15.12:0/195182323 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74005510 con 0x7f9a70004850 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.959+0000 7f9a8f800700 1 -- 172.21.15.12:0/195182323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a70004850 msgr2=0x7f9a70004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.959+0000 7f9a8f800700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a70004850 0x7f9a70004cd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9a74001940 tx=0x7f9a70005880 comp rx=0 tx=0).stop 2024-03-29T10:07:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.959+0000 7f9a8f800700 1 -- 172.21.15.12:0/195182323 shutdown_connections 2024-03-29T10:07:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.959+0000 7f9a8f800700 1 --2- 172.21.15.12:0/195182323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a70004850 0x7f9a70004cd0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.959+0000 7f9a8f800700 1 -- 172.21.15.12:0/195182323 >> 172.21.15.12:0/195182323 conn(0x7f9a8808aff0 msgr2=0x7f9a880d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.959+0000 7f9a8f800700 1 -- 172.21.15.12:0/195182323 shutdown_connections 2024-03-29T10:07:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.959+0000 7f9a8f800700 1 -- 172.21.15.12:0/195182323 wait complete. 2024-03-29T10:07:08.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.960+0000 7f9a8f800700 1 Processor -- start 2024-03-29T10:07:08.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.961+0000 7f9a8f800700 1 -- start start 2024-03-29T10:07:08.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.961+0000 7f9a8f800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a70004850 0x7f9a8815c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a8f800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a8815c6a0 0x7f9a880c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a8f800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a880c8610 0x7f9a880c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a8f800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880cc460 con 0x7f9a8815c6a0 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a8f800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a881f6430 con 0x7f9a880c8610 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a8f800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8815cb20 con 0x7f9a70004850 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a87e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a8815c6a0 0x7f9a880c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a87e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a8815c6a0 0x7f9a880c80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47378/0 (socket says 172.21.15.12:47378) 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a87e00700 1 -- 172.21.15.12:0/1479030676 learned_addr learned my addr 172.21.15.12:0/1479030676 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a8ce00700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a70004850 0x7f9a8815c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a8d800700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a880c8610 0x7f9a880c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a87e00700 1 -- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a70004850 msgr2=0x7f9a8815c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a87e00700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a70004850 0x7f9a8815c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a87e00700 1 -- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a880c8610 msgr2=0x7f9a880c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a87e00700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a880c8610 0x7f9a880c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a87e00700 1 -- 172.21.15.12:0/1479030676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8814e460 con 0x7f9a8815c6a0 2024-03-29T10:07:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.962+0000 7f9a87e00700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a8815c6a0 0x7f9a880c80d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9a74005040 tx=0x7f9a740012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.963+0000 7f9a84c00700 1 -- 172.21.15.12:0/1479030676 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74006a70 con 0x7f9a8815c6a0 2024-03-29T10:07:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.963+0000 7f9a84c00700 1 -- 172.21.15.12:0/1479030676 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a74006bd0 con 0x7f9a8815c6a0 2024-03-29T10:07:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.963+0000 7f9a8f800700 1 -- 172.21.15.12:0/1479030676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a881485c0 con 0x7f9a8815c6a0 2024-03-29T10:07:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.963+0000 7f9a84c00700 1 -- 172.21.15.12:0/1479030676 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74007040 con 0x7f9a8815c6a0 2024-03-29T10:07:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.963+0000 7f9a8f800700 1 -- 172.21.15.12:0/1479030676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8814b510 con 0x7f9a8815c6a0 2024-03-29T10:07:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.964+0000 7f9a8f800700 1 -- 172.21.15.12:0/1479030676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a880d3c70 con 0x7f9a8815c6a0 2024-03-29T10:07:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.965+0000 7f9a84c00700 1 -- 172.21.15.12:0/1479030676 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9a740071a0 con 0x7f9a8815c6a0 2024-03-29T10:07:08.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.966+0000 7f9a84c00700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a6806adb0 0x7f9a6806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:08.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.966+0000 7f9a84c00700 1 -- 172.21.15.12:0/1479030676 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a74089ea0 con 0x7f9a8815c6a0 2024-03-29T10:07:08.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.966+0000 7f9a8ce00700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a6806adb0 0x7f9a6806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:08.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.967+0000 7f9a8ce00700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a6806adb0 0x7f9a6806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009cd0 tx=0x7f9a7c008040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:08.970+0000 7f9a84c00700 1 -- 172.21.15.12:0/1479030676 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9a74055080 con 0x7f9a8815c6a0 2024-03-29T10:07:09.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.189+0000 7f9a8f800700 1 -- 172.21.15.12:0/1479030676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a88067720 con 0x7f9a8815c6a0 2024-03-29T10:07:09.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.191+0000 7f9a84c00700 1 -- 172.21.15.12:0/1479030676 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16058 (secure 0 0 0) 0x7f9a7402b090 con 0x7f9a8815c6a0 2024-03-29T10:07:09.190 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:09.191 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":60,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:07:08.705446+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:07:07.106799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-29T10:07:07.671044+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 -- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a6806adb0 msgr2=0x7f9a6806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a6806adb0 0x7f9a6806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009cd0 tx=0x7f9a7c008040 comp rx=0 tx=0).stop 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 -- 172.21.15.12:0/1479030676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a8815c6a0 msgr2=0x7f9a880c80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a8815c6a0 0x7f9a880c80d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9a74005040 tx=0x7f9a740012d0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 -- 172.21.15.12:0/1479030676 shutdown_connections 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a6806adb0 0x7f9a6806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a880c8610 0x7f9a880c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a70004850 0x7f9a8815c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 --2- 172.21.15.12:0/1479030676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a8815c6a0 0x7f9a880c80d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 -- 172.21.15.12:0/1479030676 >> 172.21.15.12:0/1479030676 conn(0x7f9a8808aff0 msgr2=0x7f9a8815b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 -- 172.21.15.12:0/1479030676 shutdown_connections 2024-03-29T10:07:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.193+0000 7f9a8f800700 1 -- 172.21.15.12:0/1479030676 wait complete. 2024-03-29T10:07:09.211 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:07:09.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.409+0000 7f8929a00700 1 Processor -- start 2024-03-29T10:07:09.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.409+0000 7f8929a00700 1 -- start start 2024-03-29T10:07:09.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.410+0000 7f8929a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f892415b560 0x7f892409aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.410+0000 7f8929a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8924099a60 0x7f8924099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.410+0000 7f8929a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892409a3f0 con 0x7f89240cc900 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.410+0000 7f8929a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89240cefc0 con 0x7f892415b560 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.410+0000 7f8929a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89240cf230 con 0x7f8924099a60 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.410+0000 7f8922000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f89240cc900 0x7f89240cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35858/0 (socket says 172.21.15.12:35858) 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.410+0000 7f8922000700 1 -- 172.21.15.12:0/416720597 learned_addr learned my addr 172.21.15.12:0/416720597 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.410+0000 7f8922a00700 1 --2- 172.21.15.12:0/416720597 >> v2:172.21.15.159:3300/0 conn(0x7f892415b560 0x7f892409aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.410+0000 7f8923400700 1 --2- 172.21.15.12:0/416720597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8924099a60 0x7f8924099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.411+0000 7f8921600700 1 -- 172.21.15.12:0/416720597 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66695013 0 0) 0x7f8910001110 con 0x7f89240cc900 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.411+0000 7f8921600700 1 -- 172.21.15.12:0/416720597 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89080035b0 con 0x7f89240cc900 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.411+0000 7f8923400700 1 -- 172.21.15.12:0/416720597 >> v2:172.21.15.159:3300/0 conn(0x7f892415b560 msgr2=0x7f892409aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.411+0000 7f8923400700 1 --2- 172.21.15.12:0/416720597 >> v2:172.21.15.159:3300/0 conn(0x7f892415b560 0x7f892409aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.411+0000 7f8923400700 1 -- 172.21.15.12:0/416720597 >> v1:172.21.15.12:6789/0 conn(0x7f89240cc900 legacy=0x7f89240cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.411+0000 7f8923400700 1 -- 172.21.15.12:0/416720597 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89240cf510 con 0x7f8924099a60 2024-03-29T10:07:09.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.411+0000 7f8922a00700 1 --2- 172.21.15.12:0/416720597 >> v2:172.21.15.159:3300/0 conn(0x7f892415b560 0x7f892409aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:07:09.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.411+0000 7f8923400700 1 --2- 172.21.15.12:0/416720597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8924099a60 0x7f8924099eb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f891c009d90 tx=0x7f891c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e05aaf5dc1f741cb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.411+0000 7f8921600700 1 -- 172.21.15.12:0/416720597 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c00e070 con 0x7f8924099a60 2024-03-29T10:07:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.412+0000 7f8921600700 1 -- 172.21.15.12:0/416720597 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c00bb20 con 0x7f8924099a60 2024-03-29T10:07:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.412+0000 7f8921600700 1 -- 172.21.15.12:0/416720597 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c012430 con 0x7f8924099a60 2024-03-29T10:07:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.412+0000 7f8929a00700 1 -- 172.21.15.12:0/416720597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8924099a60 msgr2=0x7f8924099eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.412+0000 7f8929a00700 1 --2- 172.21.15.12:0/416720597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8924099a60 0x7f8924099eb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f891c009d90 tx=0x7f891c00b330 comp rx=0 tx=0).stop 2024-03-29T10:07:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.412+0000 7f8929a00700 1 -- 172.21.15.12:0/416720597 shutdown_connections 2024-03-29T10:07:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.412+0000 7f8929a00700 1 --2- 172.21.15.12:0/416720597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8924099a60 0x7f8924099eb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.412+0000 7f8929a00700 1 --2- 172.21.15.12:0/416720597 >> v2:172.21.15.159:3300/0 conn(0x7f892415b560 0x7f892409aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.412+0000 7f8929a00700 1 -- 172.21.15.12:0/416720597 >> 172.21.15.12:0/416720597 conn(0x7f89240d1350 msgr2=0x7f89240d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.412+0000 7f8929a00700 1 -- 172.21.15.12:0/416720597 shutdown_connections 2024-03-29T10:07:09.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.412+0000 7f8929a00700 1 -- 172.21.15.12:0/416720597 wait complete. 2024-03-29T10:07:09.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.414+0000 7f8929a00700 1 Processor -- start 2024-03-29T10:07:09.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.414+0000 7f8929a00700 1 -- start start 2024-03-29T10:07:09.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8929a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8924099a60 0x7f89240b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8929a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89240cc900 0x7f89240b64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8929a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f892415b560 0x7f8924158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8929a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240b5b90 con 0x7f8924099a60 2024-03-29T10:07:09.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8929a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240b4af0 con 0x7f89240cc900 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8929a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89240b4dd0 con 0x7f892415b560 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8922a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8924099a60 0x7f89240b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8922a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8924099a60 0x7f89240b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47390/0 (socket says 172.21.15.12:47390) 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8922a00700 1 -- 172.21.15.12:0/2445173014 learned_addr learned my addr 172.21.15.12:0/2445173014 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8922000700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89240cc900 0x7f89240b64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.415+0000 7f8923400700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f892415b560 0x7f8924158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f8922a00700 1 -- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f892415b560 msgr2=0x7f8924158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f8922a00700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f892415b560 0x7f8924158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f8922a00700 1 -- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89240cc900 msgr2=0x7f89240b64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f8922a00700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89240cc900 0x7f89240b64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f8922a00700 1 -- 172.21.15.12:0/2445173014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89140094a0 con 0x7f8924099a60 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f8922000700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89240cc900 0x7f89240b64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f8922a00700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8924099a60 0x7f89240b41a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f891400ebe0 tx=0x7f891400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f891aa00700 1 -- 172.21.15.12:0/2445173014 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914009800 con 0x7f8924099a60 2024-03-29T10:07:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f891aa00700 1 -- 172.21.15.12:0/2445173014 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891400c910 con 0x7f8924099a60 2024-03-29T10:07:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f891aa00700 1 -- 172.21.15.12:0/2445173014 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891400cc80 con 0x7f8924099a60 2024-03-29T10:07:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f8929a00700 1 -- 172.21.15.12:0/2445173014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891c009a70 con 0x7f8924099a60 2024-03-29T10:07:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.416+0000 7f8929a00700 1 -- 172.21.15.12:0/2445173014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89240b5390 con 0x7f8924099a60 2024-03-29T10:07:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.417+0000 7f8929a00700 1 -- 172.21.15.12:0/2445173014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89240b59e0 con 0x7f8924099a60 2024-03-29T10:07:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.418+0000 7f891aa00700 1 -- 172.21.15.12:0/2445173014 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8914013470 con 0x7f8924099a60 2024-03-29T10:07:09.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.420+0000 7f891aa00700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f890406adb0 0x7f890406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.420+0000 7f891aa00700 1 -- 172.21.15.12:0/2445173014 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8914089610 con 0x7f8924099a60 2024-03-29T10:07:09.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.420+0000 7f8922000700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f890406adb0 0x7f890406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.421+0000 7f8922000700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f890406adb0 0x7f890406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f892406cb50 tx=0x7f8910009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:09.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.424+0000 7f891aa00700 1 -- 172.21.15.12:0/2445173014 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f89140546e0 con 0x7f8924099a60 2024-03-29T10:07:09.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.637+0000 7f8929a00700 1 -- 172.21.15.12:0/2445173014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8924067720 con 0x7f8924099a60 2024-03-29T10:07:09.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.638+0000 7f891aa00700 1 -- 172.21.15.12:0/2445173014 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16058 (secure 0 0 0) 0x7f8914057d00 con 0x7f8924099a60 2024-03-29T10:07:09.638 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:09.638 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":60,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:07:08.705446+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:07:07.106799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-29T10:07:07.671044+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:07:09.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 -- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f890406adb0 msgr2=0x7f890406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f890406adb0 0x7f890406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f892406cb50 tx=0x7f8910009040 comp rx=0 tx=0).stop 2024-03-29T10:07:09.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 -- 172.21.15.12:0/2445173014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8924099a60 msgr2=0x7f89240b41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8924099a60 0x7f89240b41a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f891400ebe0 tx=0x7f891400eef0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 -- 172.21.15.12:0/2445173014 shutdown_connections 2024-03-29T10:07:09.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f890406adb0 0x7f890406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f892415b560 0x7f8924158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8924099a60 0x7f89240b41a0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 --2- 172.21.15.12:0/2445173014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89240cc900 0x7f89240b64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 -- 172.21.15.12:0/2445173014 >> 172.21.15.12:0/2445173014 conn(0x7f89240d1350 msgr2=0x7f892406c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:09.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 -- 172.21.15.12:0/2445173014 shutdown_connections 2024-03-29T10:07:09.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.640+0000 7f8929a00700 1 -- 172.21.15.12:0/2445173014 wait complete. 2024-03-29T10:07:09.658 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T10:07:09.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.863+0000 7fccc9000700 1 Processor -- start 2024-03-29T10:07:09.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.863+0000 7fccc9000700 1 -- start start 2024-03-29T10:07:09.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.864+0000 7fccc9000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fccc415dad0 0x7fccc40ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.864+0000 7fccc9000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc415f820 con 0x7fccc40d21d0 2024-03-29T10:07:09.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.865+0000 7fccc9000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc40cf420 con 0x7fccc40d1630 2024-03-29T10:07:09.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.865+0000 7fccc9000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc40cfcc0 con 0x7fccc415dad0 2024-03-29T10:07:09.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.865+0000 7fccc2800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fccc40d21d0 0x7fccc415fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35862/0 (socket says 172.21.15.12:35862) 2024-03-29T10:07:09.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.865+0000 7fccc2800700 1 -- 172.21.15.12:0/861931912 learned_addr learned my addr 172.21.15.12:0/861931912 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:09.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.865+0000 7fccc1e00700 1 --2- 172.21.15.12:0/861931912 >> v2:172.21.15.159:3300/0 conn(0x7fccc415dad0 0x7fccc40ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.865+0000 7fccc0a00700 1 -- 172.21.15.12:0/861931912 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199947838 0 0) 0x7fccc415f820 con 0x7fccc40d21d0 2024-03-29T10:07:09.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.865+0000 7fccc0a00700 1 -- 172.21.15.12:0/861931912 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca80035b0 con 0x7fccc40d21d0 2024-03-29T10:07:09.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.865+0000 7fccc1e00700 1 -- 172.21.15.12:0/861931912 >> v1:172.21.15.159:6789/0 conn(0x7fccc40d1630 legacy=0x7fccc40d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.865+0000 7fccc1e00700 1 -- 172.21.15.12:0/861931912 >> v1:172.21.15.12:6789/0 conn(0x7fccc40d21d0 legacy=0x7fccc415fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.865+0000 7fccc1e00700 1 -- 172.21.15.12:0/861931912 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc40cf700 con 0x7fccc415dad0 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.866+0000 7fccc1e00700 1 --2- 172.21.15.12:0/861931912 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccc415dad0 0x7fccc40ced80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009840 tx=0x7fccb4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d7ac25336edfb10 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.866+0000 7fccc0a00700 1 -- 172.21.15.12:0/861931912 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4011830 con 0x7fccc415dad0 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.866+0000 7fccc0a00700 1 -- 172.21.15.12:0/861931912 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb4011990 con 0x7fccc415dad0 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.866+0000 7fccc0a00700 1 -- 172.21.15.12:0/861931912 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4011d00 con 0x7fccc415dad0 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.867+0000 7fccc9000700 1 -- 172.21.15.12:0/861931912 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccc415dad0 msgr2=0x7fccc40ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.867+0000 7fccc9000700 1 --2- 172.21.15.12:0/861931912 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccc415dad0 0x7fccc40ced80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009840 tx=0x7fccb4009b50 comp rx=0 tx=0).stop 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.867+0000 7fccc9000700 1 -- 172.21.15.12:0/861931912 shutdown_connections 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.867+0000 7fccc9000700 1 --2- 172.21.15.12:0/861931912 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccc415dad0 0x7fccc40ced80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.867+0000 7fccc9000700 1 -- 172.21.15.12:0/861931912 >> 172.21.15.12:0/861931912 conn(0x7fccc408aff0 msgr2=0x7fccc40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.867+0000 7fccc9000700 1 -- 172.21.15.12:0/861931912 shutdown_connections 2024-03-29T10:07:09.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.867+0000 7fccc9000700 1 -- 172.21.15.12:0/861931912 wait complete. 2024-03-29T10:07:09.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.868+0000 7fccc9000700 1 Processor -- start 2024-03-29T10:07:09.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.869+0000 7fccc9000700 1 -- start start 2024-03-29T10:07:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.869+0000 7fccc9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc40d1630 0x7fccc40c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc9000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccc40d21d0 0x7fccc41582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc9000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccc415dad0 0x7fccc4158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc9000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc409de90 con 0x7fccc40d1630 2024-03-29T10:07:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc9000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc4158d40 con 0x7fccc415dad0 2024-03-29T10:07:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc9000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc40c8710 con 0x7fccc40d21d0 2024-03-29T10:07:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc40d1630 0x7fccc40c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc40d1630 0x7fccc40c97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47406/0 (socket says 172.21.15.12:47406) 2024-03-29T10:07:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc1e00700 1 -- 172.21.15.12:0/1643431617 learned_addr learned my addr 172.21.15.12:0/1643431617 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc2800700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccc415dad0 0x7fccc4158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc1400700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccc40d21d0 0x7fccc41582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc1e00700 1 -- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccc40d21d0 msgr2=0x7fccc41582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc1e00700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccc40d21d0 0x7fccc41582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc1e00700 1 -- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccc415dad0 msgr2=0x7fccc4158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc1e00700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccc415dad0 0x7fccc4158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc1e00700 1 -- 172.21.15.12:0/1643431617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40094a0 con 0x7fccc40d1630 2024-03-29T10:07:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.870+0000 7fccc1400700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccc40d21d0 0x7fccc41582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:07:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.871+0000 7fccc1e00700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc40d1630 0x7fccc40c97a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009fd0 tx=0x7fccb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:09.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.871+0000 7fccba000700 1 -- 172.21.15.12:0/1643431617 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb40036a0 con 0x7fccc40d1630 2024-03-29T10:07:09.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.871+0000 7fccba000700 1 -- 172.21.15.12:0/1643431617 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb401e070 con 0x7fccc40d1630 2024-03-29T10:07:09.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.871+0000 7fccc9000700 1 -- 172.21.15.12:0/1643431617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc40c7b00 con 0x7fccc40d1630 2024-03-29T10:07:09.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.871+0000 7fccba000700 1 -- 172.21.15.12:0/1643431617 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4003a40 con 0x7fccc40d1630 2024-03-29T10:07:09.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.871+0000 7fccc9000700 1 -- 172.21.15.12:0/1643431617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc40c83a0 con 0x7fccc40d1630 2024-03-29T10:07:09.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.872+0000 7fccc9000700 1 -- 172.21.15.12:0/1643431617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc40c7ef0 con 0x7fccc40d1630 2024-03-29T10:07:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.873+0000 7fccba000700 1 -- 172.21.15.12:0/1643431617 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fccb4003ba0 con 0x7fccc40d1630 2024-03-29T10:07:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.875+0000 7fccba000700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcca406adb0 0x7fcca406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.875+0000 7fccba000700 1 -- 172.21.15.12:0/1643431617 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccb408a460 con 0x7fccc40d1630 2024-03-29T10:07:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.875+0000 7fccc1400700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcca406adb0 0x7fcca406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.876+0000 7fccc1400700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcca406adb0 0x7fcca406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fccc40d2dc0 tx=0x7fccb0009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:09.879+0000 7fccba000700 1 -- 172.21.15.12:0/1643431617 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fccb4055520 con 0x7fccc40d1630 2024-03-29T10:07:10.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.147+0000 7fccc9000700 1 -- 172.21.15.12:0/1643431617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fccc4067720 con 0x7fccc40d1630 2024-03-29T10:07:10.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.720+0000 7fccba000700 1 -- 172.21.15.12:0/1643431617 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v22) v1 ==== 118+0+0 (secure 0 0 0) 0x7fccb4058b40 con 0x7fccc40d1630 2024-03-29T10:07:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 -- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcca406adb0 msgr2=0x7fcca406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcca406adb0 0x7fcca406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fccc40d2dc0 tx=0x7fccb0009040 comp rx=0 tx=0).stop 2024-03-29T10:07:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 -- 172.21.15.12:0/1643431617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc40d1630 msgr2=0x7fccc40c97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc40d1630 0x7fccc40c97a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009fd0 tx=0x7fccb4004750 comp rx=0 tx=0).stop 2024-03-29T10:07:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 -- 172.21.15.12:0/1643431617 shutdown_connections 2024-03-29T10:07:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcca406adb0 0x7fcca406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccc415dad0 0x7fccc4158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc40d1630 0x7fccc40c97a0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 --2- 172.21.15.12:0/1643431617 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccc40d21d0 0x7fccc41582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 -- 172.21.15.12:0/1643431617 >> 172.21.15.12:0/1643431617 conn(0x7fccc408aff0 msgr2=0x7fccc4159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:10.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 -- 172.21.15.12:0/1643431617 shutdown_connections 2024-03-29T10:07:10.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.724+0000 7fccc9000700 1 -- 172.21.15.12:0/1643431617 wait complete. 2024-03-29T10:07:10.744 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T10:07:10.744 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T10:07:10.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.949+0000 7f0ec8200700 1 Processor -- start 2024-03-29T10:07:10.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.949+0000 7f0ec8200700 1 -- start start 2024-03-29T10:07:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.950+0000 7f0ec8200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ec015d9d0 0x7f0ec00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.950+0000 7f0ec8200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0ec00d16a0 0x7f0ec00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.950+0000 7f0ec8200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d2370 0x7f0ec015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.950+0000 7f0ec8200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec00d2820 con 0x7f0ec015d9d0 2024-03-29T10:07:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.950+0000 7f0ec8200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec0161420 con 0x7f0ec00d16a0 2024-03-29T10:07:10.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.950+0000 7f0ec8200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec0161700 con 0x7f0ec00d2370 2024-03-29T10:07:10.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec5800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ec015d9d0 0x7f0ec00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:10.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec5800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ec015d9d0 0x7f0ec00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47422/0 (socket says 172.21.15.12:47422) 2024-03-29T10:07:10.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec5800700 1 -- 172.21.15.12:0/1845450909 learned_addr learned my addr 172.21.15.12:0/1845450909 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:10.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec4e00700 1 --2- 172.21.15.12:0/1845450909 >> v2:172.21.15.159:3300/0 conn(0x7f0ec00d16a0 0x7f0ec00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:10.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec6200700 1 --2- 172.21.15.12:0/1845450909 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d2370 0x7f0ec015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:10.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec5800700 1 -- 172.21.15.12:0/1845450909 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d2370 msgr2=0x7f0ec015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec5800700 1 --2- 172.21.15.12:0/1845450909 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d2370 0x7f0ec015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec5800700 1 -- 172.21.15.12:0/1845450909 >> v2:172.21.15.159:3300/0 conn(0x7f0ec00d16a0 msgr2=0x7f0ec00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec5800700 1 --2- 172.21.15.12:0/1845450909 >> v2:172.21.15.159:3300/0 conn(0x7f0ec00d16a0 0x7f0ec00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec5800700 1 -- 172.21.15.12:0/1845450909 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec01619e0 con 0x7f0ec015d9d0 2024-03-29T10:07:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.951+0000 7f0ec4e00700 1 --2- 172.21.15.12:0/1845450909 >> v2:172.21.15.159:3300/0 conn(0x7f0ec00d16a0 0x7f0ec00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ec5800700 1 --2- 172.21.15.12:0/1845450909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 0x7f0ec00d3c30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00d12f0 tx=0x7f0eb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56be1c465ba3bb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ebfe00700 1 -- 172.21.15.12:0/1845450909 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4011830 con 0x7f0ec015d9d0 2024-03-29T10:07:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ebfe00700 1 -- 172.21.15.12:0/1845450909 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb4011990 con 0x7f0ec015d9d0 2024-03-29T10:07:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ebfe00700 1 -- 172.21.15.12:0/1845450909 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4011d20 con 0x7f0ec015d9d0 2024-03-29T10:07:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ec8200700 1 -- 172.21.15.12:0/1845450909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 msgr2=0x7f0ec00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ec8200700 1 --2- 172.21.15.12:0/1845450909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 0x7f0ec00d3c30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00d12f0 tx=0x7f0eb4009bd0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ec8200700 1 -- 172.21.15.12:0/1845450909 shutdown_connections 2024-03-29T10:07:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ec8200700 1 --2- 172.21.15.12:0/1845450909 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d2370 0x7f0ec015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ec8200700 1 --2- 172.21.15.12:0/1845450909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 0x7f0ec00d3c30 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ec8200700 1 --2- 172.21.15.12:0/1845450909 >> v2:172.21.15.159:3300/0 conn(0x7f0ec00d16a0 0x7f0ec00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.952+0000 7f0ec8200700 1 -- 172.21.15.12:0/1845450909 >> 172.21.15.12:0/1845450909 conn(0x7f0ec008aff0 msgr2=0x7f0ec00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.953+0000 7f0ec8200700 1 -- 172.21.15.12:0/1845450909 shutdown_connections 2024-03-29T10:07:10.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.953+0000 7f0ec8200700 1 -- 172.21.15.12:0/1845450909 wait complete. 2024-03-29T10:07:10.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.954+0000 7f0ec8200700 1 Processor -- start 2024-03-29T10:07:10.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.954+0000 7f0ec8200700 1 -- start start 2024-03-29T10:07:10.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.955+0000 7f0ec8200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d16a0 0x7f0ec01f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:10.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.955+0000 7f0ec8200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ec00d2370 0x7f0ec014d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:10.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.955+0000 7f0ec8200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 0x7f0ec00cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:10.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.955+0000 7f0ec8200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec01f6d80 con 0x7f0ec015d9d0 2024-03-29T10:07:10.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.955+0000 7f0ec8200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec01f7340 con 0x7f0ec00d2370 2024-03-29T10:07:10.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.955+0000 7f0ec8200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec01f7ca0 con 0x7f0ec00d16a0 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 0x7f0ec00cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 0x7f0ec00cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47438/0 (socket says 172.21.15.12:47438) 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec6200700 1 -- 172.21.15.12:0/3056397471 learned_addr learned my addr 172.21.15.12:0/3056397471 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec4e00700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ec00d2370 0x7f0ec014d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec5800700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d16a0 0x7f0ec01f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec6200700 1 -- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d16a0 msgr2=0x7f0ec01f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec6200700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d16a0 0x7f0ec01f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec6200700 1 -- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ec00d2370 msgr2=0x7f0ec014d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec6200700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ec00d2370 0x7f0ec014d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec6200700 1 -- 172.21.15.12:0/3056397471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb4009550 con 0x7f0ec015d9d0 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec5800700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d16a0 0x7f0ec01f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec6200700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 0x7f0ec00cd440 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800efc0 tx=0x7f0eb800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ebd600700 1 -- 172.21.15.12:0/3056397471 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb800cd40 con 0x7f0ec015d9d0 2024-03-29T10:07:10.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ebd600700 1 -- 172.21.15.12:0/3056397471 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb800f040 con 0x7f0ec015d9d0 2024-03-29T10:07:10.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.956+0000 7f0ec8200700 1 -- 172.21.15.12:0/3056397471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec014d800 con 0x7f0ec015d9d0 2024-03-29T10:07:10.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.957+0000 7f0ebd600700 1 -- 172.21.15.12:0/3056397471 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb80146c0 con 0x7f0ec015d9d0 2024-03-29T10:07:10.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.957+0000 7f0ec8200700 1 -- 172.21.15.12:0/3056397471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec0161030 con 0x7f0ec015d9d0 2024-03-29T10:07:10.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.958+0000 7f0ec8200700 1 -- 172.21.15.12:0/3056397471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec00d1ca0 con 0x7f0ec015d9d0 2024-03-29T10:07:10.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.959+0000 7f0ebd600700 1 -- 172.21.15.12:0/3056397471 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0eb8014820 con 0x7f0ec015d9d0 2024-03-29T10:07:10.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.959+0000 7f0ebd600700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ea006adb0 0x7f0ea006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:10.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.959+0000 7f0ebd600700 1 -- 172.21.15.12:0/3056397471 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0eb808a1e0 con 0x7f0ec015d9d0 2024-03-29T10:07:10.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.960+0000 7f0ec5800700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ea006adb0 0x7f0ea006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:10.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.961+0000 7f0ec5800700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ea006adb0 0x7f0ea006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4014040 tx=0x7f0eb4008030 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:10.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:10.964+0000 7f0ebd600700 1 -- 172.21.15.12:0/3056397471 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0eb8055100 con 0x7f0ec015d9d0 2024-03-29T10:07:11.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:11.230+0000 7f0ec8200700 1 -- 172.21.15.12:0/3056397471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0ec01f7480 con 0x7f0ec015d9d0 2024-03-29T10:07:12.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.198+0000 7f0ebd600700 1 -- 172.21.15.12:0/3056397471 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v23) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0ec01f7480 con 0x7f0ec015d9d0 2024-03-29T10:07:12.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.201+0000 7f0ec8200700 1 -- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ea006adb0 msgr2=0x7f0ea006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.201+0000 7f0ec8200700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ea006adb0 0x7f0ea006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4014040 tx=0x7f0eb4008030 comp rx=0 tx=0).stop 2024-03-29T10:07:12.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.201+0000 7f0ec8200700 1 -- 172.21.15.12:0/3056397471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 msgr2=0x7f0ec00cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.201+0000 7f0ec8200700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 0x7f0ec00cd440 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800efc0 tx=0x7f0eb800c3a0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.201+0000 7f0ec8200700 1 -- 172.21.15.12:0/3056397471 shutdown_connections 2024-03-29T10:07:12.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.201+0000 7f0ec8200700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0ea006adb0 0x7f0ea006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.201+0000 7f0ec8200700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ec015d9d0 0x7f0ec00cd440 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.201+0000 7f0ec8200700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ec00d16a0 0x7f0ec01f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.201+0000 7f0ec8200700 1 --2- 172.21.15.12:0/3056397471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ec00d2370 0x7f0ec014d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.201+0000 7f0ec8200700 1 -- 172.21.15.12:0/3056397471 >> 172.21.15.12:0/3056397471 conn(0x7f0ec008aff0 msgr2=0x7f0ec00ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:12.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.202+0000 7f0ec8200700 1 -- 172.21.15.12:0/3056397471 shutdown_connections 2024-03-29T10:07:12.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.202+0000 7f0ec8200700 1 -- 172.21.15.12:0/3056397471 wait complete. 2024-03-29T10:07:12.222 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T10:07:12.223 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:07:12.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.425+0000 7fd1e0a00700 1 Processor -- start 2024-03-29T10:07:12.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.425+0000 7fd1e0a00700 1 -- start start 2024-03-29T10:07:12.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.426+0000 7fd1e0a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd1dc0ce590 0x7fd1dc0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.427+0000 7fd1e0a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1dc0ceef0 con 0x7fd1dc160be0 2024-03-29T10:07:12.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.427+0000 7fd1e0a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1dc160fc0 con 0x7fd1dc0b6a20 2024-03-29T10:07:12.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.427+0000 7fd1e0a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1dc0b65d0 con 0x7fd1dc0ce590 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.427+0000 7fd1d9e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd1dc160be0 0x7fd1dc0cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35876/0 (socket says 172.21.15.12:35876) 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.427+0000 7fd1d9e00700 1 -- 172.21.15.12:0/3266646747 learned_addr learned my addr 172.21.15.12:0/3266646747 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.427+0000 7fd1d9400700 1 --2- 172.21.15.12:0/3266646747 >> v2:172.21.15.159:3300/0 conn(0x7fd1dc0ce590 0x7fd1dc0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.427+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921011338 0 0) 0x7fd1dc0ceef0 con 0x7fd1dc160be0 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.427+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c00035b0 con 0x7fd1dc160be0 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.427+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 293549298 0 0) 0x7fd1c00035b0 con 0x7fd1dc160be0 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 >> v2:172.21.15.159:3300/0 conn(0x7fd1dc0ce590 msgr2=0x7fd1dc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 --2- 172.21.15.12:0/3266646747 >> v2:172.21.15.159:3300/0 conn(0x7fd1dc0ce590 0x7fd1dc0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 >> v1:172.21.15.159:6789/0 conn(0x7fd1dc0b6a20 legacy=0x7fd1dc0b6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc0d21e0 con 0x7fd1dc160be0 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fd1cc001670 con 0x7fd1dc160be0 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 >> v1:172.21.15.12:6789/0 conn(0x7fd1dc160be0 legacy=0x7fd1dc0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1c0003a00 0x7fd1c0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c0004850 0x7fd1c0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1c0005210 0x7fd1c000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc0b65d0 con 0x7fd1c0003a00 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c000c0b0 con 0x7fd1c0004850 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c000c270 con 0x7fd1c0005210 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d9e00700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1c0003a00 0x7fd1c0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1da800700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1c0005210 0x7fd1c000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d9400700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c0004850 0x7fd1c0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d9e00700 1 -- 172.21.15.12:0/3266646747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1c0005210 msgr2=0x7fd1c000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d9e00700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1c0005210 0x7fd1c000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d9e00700 1 -- 172.21.15.12:0/3266646747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c0004850 msgr2=0x7fd1c0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d9e00700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c0004850 0x7fd1c0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1d9e00700 1 -- 172.21.15.12:0/3266646747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c00035b0 con 0x7fd1c0003a00 2024-03-29T10:07:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.428+0000 7fd1da800700 1 -- 172.21.15.12:0/3266646747 reap_dead start 2024-03-29T10:07:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.429+0000 7fd1d9e00700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1c0003a00 0x7fd1c0003e20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00c240 tx=0x7fd1c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea72a20a6e1d41b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.429+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc007c10 con 0x7fd1c0003a00 2024-03-29T10:07:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.429+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc00ce00 con 0x7fd1c0003a00 2024-03-29T10:07:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.429+0000 7fd1d8a00700 1 -- 172.21.15.12:0/3266646747 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc005ba0 con 0x7fd1c0003a00 2024-03-29T10:07:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.429+0000 7fd1e0a00700 1 -- 172.21.15.12:0/3266646747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1c0003a00 msgr2=0x7fd1c0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.429+0000 7fd1e0a00700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1c0003a00 0x7fd1c0003e20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00c240 tx=0x7fd1c0005880 comp rx=0 tx=0).stop 2024-03-29T10:07:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.429+0000 7fd1e0a00700 1 -- 172.21.15.12:0/3266646747 shutdown_connections 2024-03-29T10:07:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.429+0000 7fd1e0a00700 1 --2- 172.21.15.12:0/3266646747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1c0003a00 0x7fd1c0003e20 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.429+0000 7fd1e0a00700 1 -- 172.21.15.12:0/3266646747 >> 172.21.15.12:0/3266646747 conn(0x7fd1dc08aff0 msgr2=0x7fd1dc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.430+0000 7fd1e0a00700 1 -- 172.21.15.12:0/3266646747 shutdown_connections 2024-03-29T10:07:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.430+0000 7fd1e0a00700 1 -- 172.21.15.12:0/3266646747 wait complete. 2024-03-29T10:07:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.432+0000 7fd1e0a00700 1 Processor -- start 2024-03-29T10:07:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.432+0000 7fd1e0a00700 1 -- start start 2024-03-29T10:07:12.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.433+0000 7fd1e0a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c0003a00 0x7fd1dc1f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.433+0000 7fd1e0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1dc153830 0x7fd1dc0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.433+0000 7fd1e0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc0ccc20 0x7fd1dc09e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.433+0000 7fd1e0a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc0cafc0 con 0x7fd1dc153830 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.433+0000 7fd1e0a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc09f7d0 con 0x7fd1c0003a00 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.433+0000 7fd1e0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc1f3900 con 0x7fd1dc0ccc20 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.434+0000 7fd1d9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1dc153830 0x7fd1dc0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.434+0000 7fd1d9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1dc153830 0x7fd1dc0cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47456/0 (socket says 172.21.15.12:47456) 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.434+0000 7fd1d9e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c0003a00 0x7fd1dc1f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.434+0000 7fd1d9400700 1 -- 172.21.15.12:0/399040431 learned_addr learned my addr 172.21.15.12:0/399040431 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.434+0000 7fd1da800700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc0ccc20 0x7fd1dc09e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.434+0000 7fd1d9400700 1 -- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc0ccc20 msgr2=0x7fd1dc09e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.434+0000 7fd1d9400700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc0ccc20 0x7fd1dc09e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.434+0000 7fd1d9400700 1 -- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c0003a00 msgr2=0x7fd1dc1f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.434+0000 7fd1d9400700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c0003a00 0x7fd1dc1f33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.434+0000 7fd1d9400700 1 -- 172.21.15.12:0/399040431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc0a1130 con 0x7fd1dc153830 2024-03-29T10:07:12.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.435+0000 7fd1d9400700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1dc153830 0x7fd1dc0cc6e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8011a00 tx=0x7fd1c8011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:12.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.435+0000 7fd1d2000700 1 -- 172.21.15.12:0/399040431 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8011e40 con 0x7fd1dc153830 2024-03-29T10:07:12.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.435+0000 7fd1d2000700 1 -- 172.21.15.12:0/399040431 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c8004240 con 0x7fd1dc153830 2024-03-29T10:07:12.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.435+0000 7fd1d2000700 1 -- 172.21.15.12:0/399040431 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80036a0 con 0x7fd1dc153830 2024-03-29T10:07:12.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.435+0000 7fd1e0a00700 1 -- 172.21.15.12:0/399040431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1dc0cd230 con 0x7fd1dc153830 2024-03-29T10:07:12.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.435+0000 7fd1e0a00700 1 -- 172.21.15.12:0/399040431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1dc0ca900 con 0x7fd1dc153830 2024-03-29T10:07:12.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.436+0000 7fd1e0a00700 1 -- 172.21.15.12:0/399040431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1dc0cd000 con 0x7fd1dc153830 2024-03-29T10:07:12.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.437+0000 7fd1d2000700 1 -- 172.21.15.12:0/399040431 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd1c8003800 con 0x7fd1dc153830 2024-03-29T10:07:12.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.438+0000 7fd1d2000700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1bc06adb0 0x7fd1bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.438+0000 7fd1d2000700 1 -- 172.21.15.12:0/399040431 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1c8089740 con 0x7fd1dc153830 2024-03-29T10:07:12.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.438+0000 7fd1d9e00700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1bc06adb0 0x7fd1bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.439+0000 7fd1d9e00700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1bc06adb0 0x7fd1bc06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc000c00 tx=0x7fd1cc007000 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:12.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.442+0000 7fd1d2000700 1 -- 172.21.15.12:0/399040431 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1c8054800 con 0x7fd1dc153830 2024-03-29T10:07:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.704+0000 7fd1e0a00700 1 -- 172.21.15.12:0/399040431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1dc067720 con 0x7fd1dc153830 2024-03-29T10:07:12.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.704+0000 7fd1d2000700 1 -- 172.21.15.12:0/399040431 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fd1c8019090 con 0x7fd1dc153830 2024-03-29T10:07:12.704 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:12.704 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5746,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/981039791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":981039791},{"type":"v1","addr":"172.21.15.159:6836","nonce":981039791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1682878615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1682878615},{"type":"v1","addr":"172.21.15.159:6837","nonce":1682878615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5752,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1663013205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":1663013205},{"type":"v1","addr":"172.21.15.12:6834","nonce":1663013205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:07:08.705772+0000","modified":"2024-03-29T10:07:12.186244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5877},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5877":{"gid":5877,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3970376731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3970376731},{"type":"v1","addr":"172.21.15.12:6835","nonce":3970376731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-29T10:07:12.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 -- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1bc06adb0 msgr2=0x7fd1bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1bc06adb0 0x7fd1bc06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc000c00 tx=0x7fd1cc007000 comp rx=0 tx=0).stop 2024-03-29T10:07:12.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 -- 172.21.15.12:0/399040431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1dc153830 msgr2=0x7fd1dc0cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1dc153830 0x7fd1dc0cc6e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8011a00 tx=0x7fd1c8011d10 comp rx=0 tx=0).stop 2024-03-29T10:07:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 -- 172.21.15.12:0/399040431 shutdown_connections 2024-03-29T10:07:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1bc06adb0 0x7fd1bc06d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc0ccc20 0x7fd1dc09e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c0003a00 0x7fd1dc1f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 --2- 172.21.15.12:0/399040431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1dc153830 0x7fd1dc0cc6e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 -- 172.21.15.12:0/399040431 >> 172.21.15.12:0/399040431 conn(0x7fd1dc08aff0 msgr2=0x7fd1dc06b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 -- 172.21.15.12:0/399040431 shutdown_connections 2024-03-29T10:07:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.707+0000 7fd1e0a00700 1 -- 172.21.15.12:0/399040431 wait complete. 2024-03-29T10:07:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-29T10:07:12.725 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:07:12.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.923+0000 7f3731c00700 1 Processor -- start 2024-03-29T10:07:12.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.923+0000 7f3731c00700 1 -- start start 2024-03-29T10:07:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.924+0000 7f3731c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f372c15d760 0x7f372c0d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.925+0000 7f3731c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f372c0d1330 con 0x7f372c15db40 2024-03-29T10:07:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.925+0000 7f3731c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f372c0d1610 con 0x7f372c0cf3c0 2024-03-29T10:07:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.925+0000 7f3731c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372c0d1fb0 con 0x7f372c15d760 2024-03-29T10:07:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.925+0000 7f372aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f372c15d760 0x7f372c0d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.925+0000 7f372aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f372c15d760 0x7f372c0d23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47458/0 (socket says 172.21.15.12:47458) 2024-03-29T10:07:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.925+0000 7f372aa00700 1 -- 172.21.15.12:0/977826772 learned_addr learned my addr 172.21.15.12:0/977826772 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.925+0000 7f372aa00700 1 -- 172.21.15.12:0/977826772 >> v1:172.21.15.159:6789/0 conn(0x7f372c0cf3c0 legacy=0x7f372c0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.925+0000 7f372aa00700 1 -- 172.21.15.12:0/977826772 >> v1:172.21.15.12:6789/0 conn(0x7f372c15db40 legacy=0x7f372c0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.925+0000 7f372aa00700 1 -- 172.21.15.12:0/977826772 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c0d18f0 con 0x7f372c15d760 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.926+0000 7f372aa00700 1 --2- 172.21.15.12:0/977826772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15d760 0x7f372c0d23d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f371c0098f0 tx=0x7f371c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3186d7ca5190ff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.926+0000 7f3729600700 1 -- 172.21.15.12:0/977826772 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c004230 con 0x7f372c15d760 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.926+0000 7f3729600700 1 -- 172.21.15.12:0/977826772 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c004390 con 0x7f372c15d760 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.926+0000 7f3729600700 1 -- 172.21.15.12:0/977826772 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c0036a0 con 0x7f372c15d760 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.926+0000 7f3731c00700 1 -- 172.21.15.12:0/977826772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15d760 msgr2=0x7f372c0d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.926+0000 7f3731c00700 1 --2- 172.21.15.12:0/977826772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15d760 0x7f372c0d23d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f371c0098f0 tx=0x7f371c009c00 comp rx=0 tx=0).stop 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.927+0000 7f3731c00700 1 -- 172.21.15.12:0/977826772 shutdown_connections 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.927+0000 7f3731c00700 1 --2- 172.21.15.12:0/977826772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15d760 0x7f372c0d23d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.927+0000 7f3731c00700 1 -- 172.21.15.12:0/977826772 >> 172.21.15.12:0/977826772 conn(0x7f372c08aff0 msgr2=0x7f372c0d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.927+0000 7f3731c00700 1 -- 172.21.15.12:0/977826772 shutdown_connections 2024-03-29T10:07:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.927+0000 7f3731c00700 1 -- 172.21.15.12:0/977826772 wait complete. 2024-03-29T10:07:12.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.928+0000 7f3731c00700 1 Processor -- start 2024-03-29T10:07:12.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.928+0000 7f3731c00700 1 -- start start 2024-03-29T10:07:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.929+0000 7f3731c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f372c0cf3c0 0x7f372c0b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.929+0000 7f3731c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f372c15d760 0x7f372c0c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.929+0000 7f3731c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15db40 0x7f372c0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.929+0000 7f3731c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0c9150 con 0x7f372c15db40 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.929+0000 7f3731c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0c93c0 con 0x7f372c0cf3c0 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.929+0000 7f3731c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0c6f00 con 0x7f372c15d760 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f372b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15db40 0x7f372c0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f372b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15db40 0x7f372c0c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47462/0 (socket says 172.21.15.12:47462) 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f372b400700 1 -- 172.21.15.12:0/1132492126 learned_addr learned my addr 172.21.15.12:0/1132492126 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f372aa00700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f372c0cf3c0 0x7f372c0b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f372b400700 1 -- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f372c15d760 msgr2=0x7f372c0c86d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f372b400700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f372c15d760 0x7f372c0c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f372b400700 1 -- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f372c0cf3c0 msgr2=0x7f372c0b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f372b400700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f372c0cf3c0 0x7f372c0b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f372b400700 1 -- 172.21.15.12:0/1132492126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c009550 con 0x7f372c15db40 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f372b400700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15db40 0x7f372c0c8c10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f372400cba0 tx=0x7f372400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.930+0000 7f3722a00700 1 -- 172.21.15.12:0/1132492126 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372400ab80 con 0x7f372c15db40 2024-03-29T10:07:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.931+0000 7f3722a00700 1 -- 172.21.15.12:0/1132492126 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372400ace0 con 0x7f372c15db40 2024-03-29T10:07:12.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.931+0000 7f3722a00700 1 -- 172.21.15.12:0/1132492126 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724012660 con 0x7f372c15db40 2024-03-29T10:07:12.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.931+0000 7f3731c00700 1 -- 172.21.15.12:0/1132492126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372c0c71e0 con 0x7f372c15db40 2024-03-29T10:07:12.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.931+0000 7f3731c00700 1 -- 172.21.15.12:0/1132492126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c0c77a0 con 0x7f372c15db40 2024-03-29T10:07:12.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.932+0000 7f3731c00700 1 -- 172.21.15.12:0/1132492126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372c0c79c0 con 0x7f372c15db40 2024-03-29T10:07:12.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.933+0000 7f3722a00700 1 -- 172.21.15.12:0/1132492126 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f37240127c0 con 0x7f372c15db40 2024-03-29T10:07:12.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.933+0000 7f3722a00700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f370c06adb0 0x7f370c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:12.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.934+0000 7f3722a00700 1 -- 172.21.15.12:0/1132492126 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3724089cc0 con 0x7f372c15db40 2024-03-29T10:07:12.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.934+0000 7f372aa00700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f370c06adb0 0x7f370c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:12.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.935+0000 7f372aa00700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f370c06adb0 0x7f370c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f371c009130 tx=0x7f371c008030 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:12.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:12.938+0000 7f3722a00700 1 -- 172.21.15.12:0/1132492126 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3724090080 con 0x7f372c15db40 2024-03-29T10:07:13.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.152+0000 7f3731c00700 1 -- 172.21.15.12:0/1132492126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f372c067720 con 0x7f372c15db40 2024-03-29T10:07:13.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.153+0000 7f3722a00700 1 -- 172.21.15.12:0/1132492126 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16045 (secure 0 0 0) 0x7f3724016070 con 0x7f372c15db40 2024-03-29T10:07:13.153 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":61,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:07:09.708540+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:07:07.106799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-29T10:07:07.671044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:07:13.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 -- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f370c06adb0 msgr2=0x7f370c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f370c06adb0 0x7f370c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f371c009130 tx=0x7f371c008030 comp rx=0 tx=0).stop 2024-03-29T10:07:13.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 -- 172.21.15.12:0/1132492126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15db40 msgr2=0x7f372c0c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15db40 0x7f372c0c8c10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f372400cba0 tx=0x7f372400cf60 comp rx=0 tx=0).stop 2024-03-29T10:07:13.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 -- 172.21.15.12:0/1132492126 shutdown_connections 2024-03-29T10:07:13.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f370c06adb0 0x7f370c06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f372c15db40 0x7f372c0c8c10 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f372c0cf3c0 0x7f372c0b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 --2- 172.21.15.12:0/1132492126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f372c15d760 0x7f372c0c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 -- 172.21.15.12:0/1132492126 >> 172.21.15.12:0/1132492126 conn(0x7f372c08aff0 msgr2=0x7f372c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:13.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 -- 172.21.15.12:0/1132492126 shutdown_connections 2024-03-29T10:07:13.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.156+0000 7f3731c00700 1 -- 172.21.15.12:0/1132492126 wait complete. 2024-03-29T10:07:13.174 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T10:07:13.174 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:07:13.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.373+0000 7f7316e00700 1 Processor -- start 2024-03-29T10:07:13.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.373+0000 7f7316e00700 1 -- start start 2024-03-29T10:07:13.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.374+0000 7f7316e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f731015b5d0 0x7f73100ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.374+0000 7f7316e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f73100cb3f0 0x7f73100ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.374+0000 7f7316e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731015fd50 con 0x7f73100ae930 2024-03-29T10:07:13.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.374+0000 7f7316e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731015b9b0 con 0x7f73100cb3f0 2024-03-29T10:07:13.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.374+0000 7f7316e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731015f140 con 0x7f731015b5d0 2024-03-29T10:07:13.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.374+0000 7f730f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f73100cb3f0 0x7f73100ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.374+0000 7f7314e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f73100ae930 0x7f7310140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35900/0 (socket says 172.21.15.12:35900) 2024-03-29T10:07:13.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.374+0000 7f7314e00700 1 -- 172.21.15.12:0/34975100 learned_addr learned my addr 172.21.15.12:0/34975100 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:13.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.375+0000 7f730fe00700 1 --2- 172.21.15.12:0/34975100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f731015b5d0 0x7f73100ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.375+0000 7f730f400700 1 -- 172.21.15.12:0/34975100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f731015b5d0 msgr2=0x7f73100ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:13.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.375+0000 7f730f400700 1 --2- 172.21.15.12:0/34975100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f731015b5d0 0x7f73100ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.375+0000 7f730f400700 1 -- 172.21.15.12:0/34975100 >> v1:172.21.15.12:6789/0 conn(0x7f73100ae930 legacy=0x7f7310140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.375+0000 7f730f400700 1 -- 172.21.15.12:0/34975100 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731015f420 con 0x7f73100cb3f0 2024-03-29T10:07:13.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.375+0000 7f730ea00700 1 -- 172.21.15.12:0/34975100 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187120600 0 0) 0x7f731015fd50 con 0x7f73100ae930 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.375+0000 7f730f400700 1 --2- 172.21.15.12:0/34975100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73100cb3f0 0x7f73100ac0c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009840 tx=0x7f72fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f4b6793a1f5516 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.375+0000 7f730ea00700 1 -- 172.21.15.12:0/34975100 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc011830 con 0x7f73100cb3f0 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.375+0000 7f730ea00700 1 -- 172.21.15.12:0/34975100 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc011990 con 0x7f73100cb3f0 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.376+0000 7f730ea00700 1 -- 172.21.15.12:0/34975100 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc011d20 con 0x7f73100cb3f0 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.376+0000 7f7316e00700 1 -- 172.21.15.12:0/34975100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73100cb3f0 msgr2=0x7f73100ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.376+0000 7f7316e00700 1 --2- 172.21.15.12:0/34975100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73100cb3f0 0x7f73100ac0c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009840 tx=0x7f72fc009b50 comp rx=0 tx=0).stop 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.376+0000 7f7316e00700 1 -- 172.21.15.12:0/34975100 shutdown_connections 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.376+0000 7f7316e00700 1 --2- 172.21.15.12:0/34975100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f731015b5d0 0x7f73100ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.376+0000 7f7316e00700 1 --2- 172.21.15.12:0/34975100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73100cb3f0 0x7f73100ac0c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.376+0000 7f7316e00700 1 -- 172.21.15.12:0/34975100 >> 172.21.15.12:0/34975100 conn(0x7f73100cf190 msgr2=0x7f73100cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.376+0000 7f7316e00700 1 -- 172.21.15.12:0/34975100 shutdown_connections 2024-03-29T10:07:13.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.376+0000 7f7316e00700 1 -- 172.21.15.12:0/34975100 wait complete. 2024-03-29T10:07:13.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.378+0000 7f7316e00700 1 Processor -- start 2024-03-29T10:07:13.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.378+0000 7f7316e00700 1 -- start start 2024-03-29T10:07:13.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.379+0000 7f7316e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73100ae930 0x7f73101f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.379+0000 7f7316e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f731015b5d0 0x7f731015d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.379+0000 7f7316e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f731008e450 0x7f731014f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.379+0000 7f7316e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73101f5180 con 0x7f731008e450 2024-03-29T10:07:13.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.379+0000 7f7316e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731014f030 con 0x7f731015b5d0 2024-03-29T10:07:13.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.379+0000 7f7316e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731008e8e0 con 0x7f73100ae930 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.379+0000 7f7314e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f731008e450 0x7f731014f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.379+0000 7f7314e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f731008e450 0x7f731014f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47476/0 (socket says 172.21.15.12:47476) 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.379+0000 7f7314e00700 1 -- 172.21.15.12:0/366514415 learned_addr learned my addr 172.21.15.12:0/366514415 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f730fe00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73100ae930 0x7f73101f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f730f400700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f731015b5d0 0x7f731015d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f7314e00700 1 -- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73100ae930 msgr2=0x7f73101f4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f7314e00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73100ae930 0x7f73101f4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f7314e00700 1 -- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f731015b5d0 msgr2=0x7f731015d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f7314e00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f731015b5d0 0x7f731015d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f7314e00700 1 -- 172.21.15.12:0/366514415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc0094a0 con 0x7f731008e450 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f730f400700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f731015b5d0 0x7f731015d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f7314e00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f731008e450 0x7f731014f4b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f730800cba0 tx=0x7f730800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f7307e00700 1 -- 172.21.15.12:0/366514415 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730800aad0 con 0x7f731008e450 2024-03-29T10:07:13.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f7307e00700 1 -- 172.21.15.12:0/366514415 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730800ac30 con 0x7f731008e450 2024-03-29T10:07:13.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f7307e00700 1 -- 172.21.15.12:0/366514415 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73080070d0 con 0x7f731008e450 2024-03-29T10:07:13.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.380+0000 7f7316e00700 1 -- 172.21.15.12:0/366514415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731008ebc0 con 0x7f731008e450 2024-03-29T10:07:13.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.381+0000 7f7316e00700 1 -- 172.21.15.12:0/366514415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731008f180 con 0x7f731008e450 2024-03-29T10:07:13.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.381+0000 7f7316e00700 1 -- 172.21.15.12:0/366514415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731008f3a0 con 0x7f731008e450 2024-03-29T10:07:13.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.383+0000 7f7307e00700 1 -- 172.21.15.12:0/366514415 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7308007230 con 0x7f731008e450 2024-03-29T10:07:13.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.383+0000 7f7307e00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f72f006adb0 0x7f72f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.384+0000 7f7307e00700 1 -- 172.21.15.12:0/366514415 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7308088fc0 con 0x7f731008e450 2024-03-29T10:07:13.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.384+0000 7f730fe00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f72f006adb0 0x7f72f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.385+0000 7f730fe00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f72f006adb0 0x7f72f006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7300007a30 tx=0x7f7300008040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:13.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.388+0000 7f7307e00700 1 -- 172.21.15.12:0/366514415 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7308054080 con 0x7f731008e450 2024-03-29T10:07:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.599+0000 7f7316e00700 1 -- 172.21.15.12:0/366514415 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f731015f9e0 con 0x7f72f006adb0 2024-03-29T10:07:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.610+0000 7f7307e00700 1 -- 172.21.15.12:0/366514415 <== mgr.5210 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289005 (secure 0 0 0) 0x7f7300010740 con 0x7f72f006adb0 2024-03-29T10:07:13.615 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 -- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f72f006adb0 msgr2=0x7f72f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f72f006adb0 0x7f72f006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7300007a30 tx=0x7f7300008040 comp rx=0 tx=0).stop 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 -- 172.21.15.12:0/366514415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f731008e450 msgr2=0x7f731014f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f731008e450 0x7f731014f4b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f730800cba0 tx=0x7f730800ceb0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 -- 172.21.15.12:0/366514415 shutdown_connections 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f72f006adb0 0x7f72f006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f731008e450 0x7f731014f4b0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73100ae930 0x7f73101f4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 --2- 172.21.15.12:0/366514415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f731015b5d0 0x7f731015d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 -- 172.21.15.12:0/366514415 >> 172.21.15.12:0/366514415 conn(0x7f73100cf190 msgr2=0x7f731014c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 -- 172.21.15.12:0/366514415 shutdown_connections 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.615+0000 7f7316e00700 1 -- 172.21.15.12:0/366514415 wait complete. 2024-03-29T10:07:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:07:13.633 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-29T10:07:11.857473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":300052,"kb_used_data":6788,"kb_used_omap":12,"kb_used_meta":293107,"kb_avail":754674668,"statfs":{"total":773094113280,"available":772786860032,"internally_reserved":0,"allocated":6950912,"data_stored":3830073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":300142137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.681137"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822629+0000","last_change":"2024-03-29T10:07:08.734768+0000","last_active":"2024-03-29T10:07:09.822629+0000","last_peered":"2024-03-29T10:07:09.822629+0000","last_clean":"2024-03-29T10:07:09.822629+0000","last_became_active":"2024-03-29T10:07:08.734123+0000","last_became_peered":"2024-03-29T10:07:08.734123+0000","last_unstale":"2024-03-29T10:07:09.822629+0000","last_undegraded":"2024-03-29T10:07:09.822629+0000","last_fullsized":"2024-03-29T10:07:09.822629+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:41.760118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717945+0000","last_change":"2024-03-29T10:07:08.725603+0000","last_active":"2024-03-29T10:07:09.717945+0000","last_peered":"2024-03-29T10:07:09.717945+0000","last_clean":"2024-03-29T10:07:09.717945+0000","last_became_active":"2024-03-29T10:07:08.724365+0000","last_became_peered":"2024-03-29T10:07:08.724365+0000","last_unstale":"2024-03-29T10:07:09.717945+0000","last_undegraded":"2024-03-29T10:07:09.717945+0000","last_fullsized":"2024-03-29T10:07:09.717945+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:05.814187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823120+0000","last_change":"2024-03-29T10:07:08.729857+0000","last_active":"2024-03-29T10:07:09.823120+0000","last_peered":"2024-03-29T10:07:09.823120+0000","last_clean":"2024-03-29T10:07:09.823120+0000","last_became_active":"2024-03-29T10:07:08.729539+0000","last_became_peered":"2024-03-29T10:07:08.729539+0000","last_unstale":"2024-03-29T10:07:09.823120+0000","last_undegraded":"2024-03-29T10:07:09.823120+0000","last_fullsized":"2024-03-29T10:07:09.823120+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:41.082469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716672+0000","last_change":"2024-03-29T10:07:08.727599+0000","last_active":"2024-03-29T10:07:09.716672+0000","last_peered":"2024-03-29T10:07:09.716672+0000","last_clean":"2024-03-29T10:07:09.716672+0000","last_became_active":"2024-03-29T10:07:08.726751+0000","last_became_peered":"2024-03-29T10:07:08.726751+0000","last_unstale":"2024-03-29T10:07:09.716672+0000","last_undegraded":"2024-03-29T10:07:09.716672+0000","last_fullsized":"2024-03-29T10:07:09.716672+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:27:01.767573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717705+0000","last_change":"2024-03-29T10:07:08.724700+0000","last_active":"2024-03-29T10:07:09.717705+0000","last_peered":"2024-03-29T10:07:09.717705+0000","last_clean":"2024-03-29T10:07:09.717705+0000","last_became_active":"2024-03-29T10:07:08.722834+0000","last_became_peered":"2024-03-29T10:07:08.722834+0000","last_unstale":"2024-03-29T10:07:09.717705+0000","last_undegraded":"2024-03-29T10:07:09.717705+0000","last_fullsized":"2024-03-29T10:07:09.717705+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:27:01.767573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822442+0000","last_change":"2024-03-29T10:07:08.734816+0000","last_active":"2024-03-29T10:07:09.822442+0000","last_peered":"2024-03-29T10:07:09.822442+0000","last_clean":"2024-03-29T10:07:09.822442+0000","last_became_active":"2024-03-29T10:07:08.734505+0000","last_became_peered":"2024-03-29T10:07:08.734505+0000","last_unstale":"2024-03-29T10:07:09.822442+0000","last_undegraded":"2024-03-29T10:07:09.822442+0000","last_fullsized":"2024-03-29T10:07:09.822442+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:29.277567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822176+0000","last_change":"2024-03-29T10:07:08.738355+0000","last_active":"2024-03-29T10:07:09.822176+0000","last_peered":"2024-03-29T10:07:09.822176+0000","last_clean":"2024-03-29T10:07:09.822176+0000","last_became_active":"2024-03-29T10:07:08.737863+0000","last_became_peered":"2024-03-29T10:07:08.737863+0000","last_unstale":"2024-03-29T10:07:09.822176+0000","last_undegraded":"2024-03-29T10:07:09.822176+0000","last_fullsized":"2024-03-29T10:07:09.822176+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:04:27.581246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.715987+0000","last_change":"2024-03-29T10:07:08.728815+0000","last_active":"2024-03-29T10:07:09.715987+0000","last_peered":"2024-03-29T10:07:09.715987+0000","last_clean":"2024-03-29T10:07:09.715987+0000","last_became_active":"2024-03-29T10:07:08.728550+0000","last_became_peered":"2024-03-29T10:07:08.728550+0000","last_unstale":"2024-03-29T10:07:09.715987+0000","last_undegraded":"2024-03-29T10:07:09.715987+0000","last_fullsized":"2024-03-29T10:07:09.715987+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:41.082469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716882+0000","last_change":"2024-03-29T10:07:08.739595+0000","last_active":"2024-03-29T10:07:09.716882+0000","last_peered":"2024-03-29T10:07:09.716882+0000","last_clean":"2024-03-29T10:07:09.716882+0000","last_became_active":"2024-03-29T10:07:08.739313+0000","last_became_peered":"2024-03-29T10:07:08.739313+0000","last_unstale":"2024-03-29T10:07:09.716882+0000","last_undegraded":"2024-03-29T10:07:09.716882+0000","last_fullsized":"2024-03-29T10:07:09.716882+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:49:52.665440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716870+0000","last_change":"2024-03-29T10:07:08.726816+0000","last_active":"2024-03-29T10:07:09.716870+0000","last_peered":"2024-03-29T10:07:09.716870+0000","last_clean":"2024-03-29T10:07:09.716870+0000","last_became_active":"2024-03-29T10:07:08.726456+0000","last_became_peered":"2024-03-29T10:07:08.726456+0000","last_unstale":"2024-03-29T10:07:09.716870+0000","last_undegraded":"2024-03-29T10:07:09.716870+0000","last_fullsized":"2024-03-29T10:07:09.716870+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:24.701904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716747+0000","last_change":"2024-03-29T10:07:08.738365+0000","last_active":"2024-03-29T10:07:09.716747+0000","last_peered":"2024-03-29T10:07:09.716747+0000","last_clean":"2024-03-29T10:07:09.716747+0000","last_became_active":"2024-03-29T10:07:08.738101+0000","last_became_peered":"2024-03-29T10:07:08.738101+0000","last_unstale":"2024-03-29T10:07:09.716747+0000","last_undegraded":"2024-03-29T10:07:09.716747+0000","last_fullsized":"2024-03-29T10:07:09.716747+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:22:33.174657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717832+0000","last_change":"2024-03-29T10:07:08.728958+0000","last_active":"2024-03-29T10:07:09.717832+0000","last_peered":"2024-03-29T10:07:09.717832+0000","last_clean":"2024-03-29T10:07:09.717832+0000","last_became_active":"2024-03-29T10:07:08.728384+0000","last_became_peered":"2024-03-29T10:07:08.728384+0000","last_unstale":"2024-03-29T10:07:09.717832+0000","last_undegraded":"2024-03-29T10:07:09.717832+0000","last_fullsized":"2024-03-29T10:07:09.717832+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:24.701904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823166+0000","last_change":"2024-03-29T10:07:08.743187+0000","last_active":"2024-03-29T10:07:09.823166+0000","last_peered":"2024-03-29T10:07:09.823166+0000","last_clean":"2024-03-29T10:07:09.823166+0000","last_became_active":"2024-03-29T10:07:08.742950+0000","last_became_peered":"2024-03-29T10:07:08.742950+0000","last_unstale":"2024-03-29T10:07:09.823166+0000","last_undegraded":"2024-03-29T10:07:09.823166+0000","last_fullsized":"2024-03-29T10:07:09.823166+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:26.899688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716462+0000","last_change":"2024-03-29T10:07:08.736913+0000","last_active":"2024-03-29T10:07:09.716462+0000","last_peered":"2024-03-29T10:07:09.716462+0000","last_clean":"2024-03-29T10:07:09.716462+0000","last_became_active":"2024-03-29T10:07:08.736686+0000","last_became_peered":"2024-03-29T10:07:08.736686+0000","last_unstale":"2024-03-29T10:07:09.716462+0000","last_undegraded":"2024-03-29T10:07:09.716462+0000","last_fullsized":"2024-03-29T10:07:09.716462+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:55.283504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822426+0000","last_change":"2024-03-29T10:07:08.736055+0000","last_active":"2024-03-29T10:07:09.822426+0000","last_peered":"2024-03-29T10:07:09.822426+0000","last_clean":"2024-03-29T10:07:09.822426+0000","last_became_active":"2024-03-29T10:07:08.735466+0000","last_became_peered":"2024-03-29T10:07:08.735466+0000","last_unstale":"2024-03-29T10:07:09.822426+0000","last_undegraded":"2024-03-29T10:07:09.822426+0000","last_fullsized":"2024-03-29T10:07:09.822426+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:49.393397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718321+0000","last_change":"2024-03-29T10:07:08.728724+0000","last_active":"2024-03-29T10:07:09.718321+0000","last_peered":"2024-03-29T10:07:09.718321+0000","last_clean":"2024-03-29T10:07:09.718321+0000","last_became_active":"2024-03-29T10:07:08.728206+0000","last_became_peered":"2024-03-29T10:07:08.728206+0000","last_unstale":"2024-03-29T10:07:09.718321+0000","last_undegraded":"2024-03-29T10:07:09.718321+0000","last_fullsized":"2024-03-29T10:07:09.718321+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:30:11.674318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823412+0000","last_change":"2024-03-29T10:07:08.725265+0000","last_active":"2024-03-29T10:07:09.823412+0000","last_peered":"2024-03-29T10:07:09.823412+0000","last_clean":"2024-03-29T10:07:09.823412+0000","last_became_active":"2024-03-29T10:07:08.724966+0000","last_became_peered":"2024-03-29T10:07:08.724966+0000","last_unstale":"2024-03-29T10:07:09.823412+0000","last_undegraded":"2024-03-29T10:07:09.823412+0000","last_fullsized":"2024-03-29T10:07:09.823412+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:44.154060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823137+0000","last_change":"2024-03-29T10:07:08.725417+0000","last_active":"2024-03-29T10:07:09.823137+0000","last_peered":"2024-03-29T10:07:09.823137+0000","last_clean":"2024-03-29T10:07:09.823137+0000","last_became_active":"2024-03-29T10:07:08.725144+0000","last_became_peered":"2024-03-29T10:07:08.725144+0000","last_unstale":"2024-03-29T10:07:09.823137+0000","last_undegraded":"2024-03-29T10:07:09.823137+0000","last_fullsized":"2024-03-29T10:07:09.823137+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:24.701904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718566+0000","last_change":"2024-03-29T10:07:08.742693+0000","last_active":"2024-03-29T10:07:09.718566+0000","last_peered":"2024-03-29T10:07:09.718566+0000","last_clean":"2024-03-29T10:07:09.718566+0000","last_became_active":"2024-03-29T10:07:08.742212+0000","last_became_peered":"2024-03-29T10:07:08.742212+0000","last_unstale":"2024-03-29T10:07:09.718566+0000","last_undegraded":"2024-03-29T10:07:09.718566+0000","last_fullsized":"2024-03-29T10:07:09.718566+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:17:30.334734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823037+0000","last_change":"2024-03-29T10:07:08.741339+0000","last_active":"2024-03-29T10:07:09.823037+0000","last_peered":"2024-03-29T10:07:09.823037+0000","last_clean":"2024-03-29T10:07:09.823037+0000","last_became_active":"2024-03-29T10:07:08.741073+0000","last_became_peered":"2024-03-29T10:07:08.741073+0000","last_unstale":"2024-03-29T10:07:09.823037+0000","last_undegraded":"2024-03-29T10:07:09.823037+0000","last_fullsized":"2024-03-29T10:07:09.823037+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:58.536135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822905+0000","last_change":"2024-03-29T10:07:08.742136+0000","last_active":"2024-03-29T10:07:09.822905+0000","last_peered":"2024-03-29T10:07:09.822905+0000","last_clean":"2024-03-29T10:07:09.822905+0000","last_became_active":"2024-03-29T10:07:08.741898+0000","last_became_peered":"2024-03-29T10:07:08.741898+0000","last_unstale":"2024-03-29T10:07:09.822905+0000","last_undegraded":"2024-03-29T10:07:09.822905+0000","last_fullsized":"2024-03-29T10:07:09.822905+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:35:45.448688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716551+0000","last_change":"2024-03-29T10:07:08.736431+0000","last_active":"2024-03-29T10:07:09.716551+0000","last_peered":"2024-03-29T10:07:09.716551+0000","last_clean":"2024-03-29T10:07:09.716551+0000","last_became_active":"2024-03-29T10:07:08.728993+0000","last_became_peered":"2024-03-29T10:07:08.728993+0000","last_unstale":"2024-03-29T10:07:09.716551+0000","last_undegraded":"2024-03-29T10:07:09.716551+0000","last_fullsized":"2024-03-29T10:07:09.716551+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:04.547033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717987+0000","last_change":"2024-03-29T10:07:08.734843+0000","last_active":"2024-03-29T10:07:09.717987+0000","last_peered":"2024-03-29T10:07:09.717987+0000","last_clean":"2024-03-29T10:07:09.717987+0000","last_became_active":"2024-03-29T10:07:08.734541+0000","last_became_peered":"2024-03-29T10:07:08.734541+0000","last_unstale":"2024-03-29T10:07:09.717987+0000","last_undegraded":"2024-03-29T10:07:09.717987+0000","last_fullsized":"2024-03-29T10:07:09.717987+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:52:16.286951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717672+0000","last_change":"2024-03-29T10:07:08.726256+0000","last_active":"2024-03-29T10:07:09.717672+0000","last_peered":"2024-03-29T10:07:09.717672+0000","last_clean":"2024-03-29T10:07:09.717672+0000","last_became_active":"2024-03-29T10:07:08.725908+0000","last_became_peered":"2024-03-29T10:07:08.725908+0000","last_unstale":"2024-03-29T10:07:09.717672+0000","last_undegraded":"2024-03-29T10:07:09.717672+0000","last_fullsized":"2024-03-29T10:07:09.717672+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:54.786449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718705+0000","last_change":"2024-03-29T10:07:08.732612+0000","last_active":"2024-03-29T10:07:09.718705+0000","last_peered":"2024-03-29T10:07:09.718705+0000","last_clean":"2024-03-29T10:07:09.718705+0000","last_became_active":"2024-03-29T10:07:08.730428+0000","last_became_peered":"2024-03-29T10:07:08.730428+0000","last_unstale":"2024-03-29T10:07:09.718705+0000","last_undegraded":"2024-03-29T10:07:09.718705+0000","last_fullsized":"2024-03-29T10:07:09.718705+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:46.657982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823711+0000","last_change":"2024-03-29T10:07:08.747522+0000","last_active":"2024-03-29T10:07:09.823711+0000","last_peered":"2024-03-29T10:07:09.823711+0000","last_clean":"2024-03-29T10:07:09.823711+0000","last_became_active":"2024-03-29T10:07:08.747234+0000","last_became_peered":"2024-03-29T10:07:08.747234+0000","last_unstale":"2024-03-29T10:07:09.823711+0000","last_undegraded":"2024-03-29T10:07:09.823711+0000","last_fullsized":"2024-03-29T10:07:09.823711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:52:16.286951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823288+0000","last_change":"2024-03-29T10:07:08.730041+0000","last_active":"2024-03-29T10:07:09.823288+0000","last_peered":"2024-03-29T10:07:09.823288+0000","last_clean":"2024-03-29T10:07:09.823288+0000","last_became_active":"2024-03-29T10:07:08.724864+0000","last_became_peered":"2024-03-29T10:07:08.724864+0000","last_unstale":"2024-03-29T10:07:09.823288+0000","last_undegraded":"2024-03-29T10:07:09.823288+0000","last_fullsized":"2024-03-29T10:07:09.823288+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:57.250306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717577+0000","last_change":"2024-03-29T10:07:08.729118+0000","last_active":"2024-03-29T10:07:09.717577+0000","last_peered":"2024-03-29T10:07:09.717577+0000","last_clean":"2024-03-29T10:07:09.717577+0000","last_became_active":"2024-03-29T10:07:08.728706+0000","last_became_peered":"2024-03-29T10:07:08.728706+0000","last_unstale":"2024-03-29T10:07:09.717577+0000","last_undegraded":"2024-03-29T10:07:09.717577+0000","last_fullsized":"2024-03-29T10:07:09.717577+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:41.082469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718395+0000","last_change":"2024-03-29T10:07:08.736372+0000","last_active":"2024-03-29T10:07:09.718395+0000","last_peered":"2024-03-29T10:07:09.718395+0000","last_clean":"2024-03-29T10:07:09.718395+0000","last_became_active":"2024-03-29T10:07:08.733791+0000","last_became_peered":"2024-03-29T10:07:08.733791+0000","last_unstale":"2024-03-29T10:07:09.718395+0000","last_undegraded":"2024-03-29T10:07:09.718395+0000","last_fullsized":"2024-03-29T10:07:09.718395+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:56:53.198672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823130+0000","last_change":"2024-03-29T10:07:08.735460+0000","last_active":"2024-03-29T10:07:09.823130+0000","last_peered":"2024-03-29T10:07:09.823130+0000","last_clean":"2024-03-29T10:07:09.823130+0000","last_became_active":"2024-03-29T10:07:08.735172+0000","last_became_peered":"2024-03-29T10:07:08.735172+0000","last_unstale":"2024-03-29T10:07:09.823130+0000","last_undegraded":"2024-03-29T10:07:09.823130+0000","last_fullsized":"2024-03-29T10:07:09.823130+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:04.547033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822970+0000","last_change":"2024-03-29T10:07:08.736284+0000","last_active":"2024-03-29T10:07:09.822970+0000","last_peered":"2024-03-29T10:07:09.822970+0000","last_clean":"2024-03-29T10:07:09.822970+0000","last_became_active":"2024-03-29T10:07:08.735769+0000","last_became_peered":"2024-03-29T10:07:08.735769+0000","last_unstale":"2024-03-29T10:07:09.822970+0000","last_undegraded":"2024-03-29T10:07:09.822970+0000","last_fullsized":"2024-03-29T10:07:09.822970+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:32.063403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822317+0000","last_change":"2024-03-29T10:07:08.728963+0000","last_active":"2024-03-29T10:07:09.822317+0000","last_peered":"2024-03-29T10:07:09.822317+0000","last_clean":"2024-03-29T10:07:09.822317+0000","last_became_active":"2024-03-29T10:07:08.728699+0000","last_became_peered":"2024-03-29T10:07:08.728699+0000","last_unstale":"2024-03-29T10:07:09.822317+0000","last_undegraded":"2024-03-29T10:07:09.822317+0000","last_fullsized":"2024-03-29T10:07:09.822317+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:56:53.198672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822792+0000","last_change":"2024-03-29T10:07:08.745181+0000","last_active":"2024-03-29T10:07:09.822792+0000","last_peered":"2024-03-29T10:07:09.822792+0000","last_clean":"2024-03-29T10:07:09.822792+0000","last_became_active":"2024-03-29T10:07:08.744863+0000","last_became_peered":"2024-03-29T10:07:08.744863+0000","last_unstale":"2024-03-29T10:07:09.822792+0000","last_undegraded":"2024-03-29T10:07:09.822792+0000","last_fullsized":"2024-03-29T10:07:09.822792+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:54.786449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"60'4","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822819+0000","last_change":"2024-03-29T10:07:08.205299+0000","last_active":"2024-03-29T10:07:09.822819+0000","last_peered":"2024-03-29T10:07:09.822819+0000","last_clean":"2024-03-29T10:07:09.822819+0000","last_became_active":"2024-03-29T10:07:08.200084+0000","last_became_peered":"2024-03-29T10:07:08.200084+0000","last_unstale":"2024-03-29T10:07:09.822819+0000","last_undegraded":"2024-03-29T10:07:09.822819+0000","last_fullsized":"2024-03-29T10:07:09.822819+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:27:00.758718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718582+0000","last_change":"2024-03-29T10:07:08.723099+0000","last_active":"2024-03-29T10:07:09.718582+0000","last_peered":"2024-03-29T10:07:09.718582+0000","last_clean":"2024-03-29T10:07:09.718582+0000","last_became_active":"2024-03-29T10:07:08.722735+0000","last_became_peered":"2024-03-29T10:07:08.722735+0000","last_unstale":"2024-03-29T10:07:09.718582+0000","last_undegraded":"2024-03-29T10:07:09.718582+0000","last_fullsized":"2024-03-29T10:07:09.718582+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:04.724057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823396+0000","last_change":"2024-03-29T10:07:08.199352+0000","last_active":"2024-03-29T10:07:09.823396+0000","last_peered":"2024-03-29T10:07:09.823396+0000","last_clean":"2024-03-29T10:07:09.823396+0000","last_became_active":"2024-03-29T10:07:08.199016+0000","last_became_peered":"2024-03-29T10:07:08.199016+0000","last_unstale":"2024-03-29T10:07:09.823396+0000","last_undegraded":"2024-03-29T10:07:09.823396+0000","last_fullsized":"2024-03-29T10:07:09.823396+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:54:33.541274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717679+0000","last_change":"2024-03-29T10:07:08.742877+0000","last_active":"2024-03-29T10:07:09.717679+0000","last_peered":"2024-03-29T10:07:09.717679+0000","last_clean":"2024-03-29T10:07:09.717679+0000","last_became_active":"2024-03-29T10:07:08.742497+0000","last_became_peered":"2024-03-29T10:07:08.742497+0000","last_unstale":"2024-03-29T10:07:09.717679+0000","last_undegraded":"2024-03-29T10:07:09.717679+0000","last_fullsized":"2024-03-29T10:07:09.717679+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:44.154060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823623+0000","last_change":"2024-03-29T10:07:08.203370+0000","last_active":"2024-03-29T10:07:09.823623+0000","last_peered":"2024-03-29T10:07:09.823623+0000","last_clean":"2024-03-29T10:07:09.823623+0000","last_became_active":"2024-03-29T10:07:08.203123+0000","last_became_peered":"2024-03-29T10:07:08.203123+0000","last_unstale":"2024-03-29T10:07:09.823623+0000","last_undegraded":"2024-03-29T10:07:09.823623+0000","last_fullsized":"2024-03-29T10:07:09.823623+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:30:10.665464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823407+0000","last_change":"2024-03-29T10:07:08.728395+0000","last_active":"2024-03-29T10:07:09.823407+0000","last_peered":"2024-03-29T10:07:09.823407+0000","last_clean":"2024-03-29T10:07:09.823407+0000","last_became_active":"2024-03-29T10:07:08.727415+0000","last_became_peered":"2024-03-29T10:07:08.727415+0000","last_unstale":"2024-03-29T10:07:09.823407+0000","last_undegraded":"2024-03-29T10:07:09.823407+0000","last_fullsized":"2024-03-29T10:07:09.823407+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:44.957025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822454+0000","last_change":"2024-03-29T10:07:08.205826+0000","last_active":"2024-03-29T10:07:09.822454+0000","last_peered":"2024-03-29T10:07:09.822454+0000","last_clean":"2024-03-29T10:07:09.822454+0000","last_became_active":"2024-03-29T10:07:08.205510+0000","last_became_peered":"2024-03-29T10:07:08.205510+0000","last_unstale":"2024-03-29T10:07:09.822454+0000","last_undegraded":"2024-03-29T10:07:09.822454+0000","last_fullsized":"2024-03-29T10:07:09.822454+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:27:00.758718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823291+0000","last_change":"2024-03-29T10:07:08.719507+0000","last_active":"2024-03-29T10:07:09.823291+0000","last_peered":"2024-03-29T10:07:09.823291+0000","last_clean":"2024-03-29T10:07:09.823291+0000","last_became_active":"2024-03-29T10:07:08.718752+0000","last_became_peered":"2024-03-29T10:07:08.718752+0000","last_unstale":"2024-03-29T10:07:09.823291+0000","last_undegraded":"2024-03-29T10:07:09.823291+0000","last_fullsized":"2024-03-29T10:07:09.823291+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:05.814187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822641+0000","last_change":"2024-03-29T10:07:08.205487+0000","last_active":"2024-03-29T10:07:09.822641+0000","last_peered":"2024-03-29T10:07:09.822641+0000","last_clean":"2024-03-29T10:07:09.822641+0000","last_became_active":"2024-03-29T10:07:08.201262+0000","last_became_peered":"2024-03-29T10:07:08.201262+0000","last_unstale":"2024-03-29T10:07:09.822641+0000","last_undegraded":"2024-03-29T10:07:09.822641+0000","last_fullsized":"2024-03-29T10:07:09.822641+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:58.258755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716382+0000","last_change":"2024-03-29T10:07:08.732681+0000","last_active":"2024-03-29T10:07:09.716382+0000","last_peered":"2024-03-29T10:07:09.716382+0000","last_clean":"2024-03-29T10:07:09.716382+0000","last_became_active":"2024-03-29T10:07:08.732422+0000","last_became_peered":"2024-03-29T10:07:08.732422+0000","last_unstale":"2024-03-29T10:07:09.716382+0000","last_undegraded":"2024-03-29T10:07:09.716382+0000","last_fullsized":"2024-03-29T10:07:09.716382+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:55:25.859062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823449+0000","last_change":"2024-03-29T10:07:08.192550+0000","last_active":"2024-03-29T10:07:09.823449+0000","last_peered":"2024-03-29T10:07:09.823449+0000","last_clean":"2024-03-29T10:07:09.823449+0000","last_became_active":"2024-03-29T10:07:08.192278+0000","last_became_peered":"2024-03-29T10:07:08.192278+0000","last_unstale":"2024-03-29T10:07:09.823449+0000","last_undegraded":"2024-03-29T10:07:09.823449+0000","last_fullsized":"2024-03-29T10:07:09.823449+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:43.948171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716198+0000","last_change":"2024-03-29T10:07:08.724451+0000","last_active":"2024-03-29T10:07:09.716198+0000","last_peered":"2024-03-29T10:07:09.716198+0000","last_clean":"2024-03-29T10:07:09.716198+0000","last_became_active":"2024-03-29T10:07:08.724175+0000","last_became_peered":"2024-03-29T10:07:08.724175+0000","last_unstale":"2024-03-29T10:07:09.716198+0000","last_undegraded":"2024-03-29T10:07:09.716198+0000","last_fullsized":"2024-03-29T10:07:09.716198+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:05.814187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717202+0000","last_change":"2024-03-29T10:07:08.197270+0000","last_active":"2024-03-29T10:07:09.717202+0000","last_peered":"2024-03-29T10:07:09.717202+0000","last_clean":"2024-03-29T10:07:09.717202+0000","last_became_active":"2024-03-29T10:07:08.196938+0000","last_became_peered":"2024-03-29T10:07:08.196938+0000","last_unstale":"2024-03-29T10:07:09.717202+0000","last_undegraded":"2024-03-29T10:07:09.717202+0000","last_fullsized":"2024-03-29T10:07:09.717202+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:53.777594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718104+0000","last_change":"2024-03-29T10:07:08.735588+0000","last_active":"2024-03-29T10:07:09.718104+0000","last_peered":"2024-03-29T10:07:09.718104+0000","last_clean":"2024-03-29T10:07:09.718104+0000","last_became_active":"2024-03-29T10:07:08.735285+0000","last_became_peered":"2024-03-29T10:07:08.735285+0000","last_unstale":"2024-03-29T10:07:09.718104+0000","last_undegraded":"2024-03-29T10:07:09.718104+0000","last_fullsized":"2024-03-29T10:07:09.718104+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:06:31.440735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716184+0000","last_change":"2024-03-29T10:07:08.206236+0000","last_active":"2024-03-29T10:07:09.716184+0000","last_peered":"2024-03-29T10:07:09.716184+0000","last_clean":"2024-03-29T10:07:09.716184+0000","last_became_active":"2024-03-29T10:07:08.205977+0000","last_became_peered":"2024-03-29T10:07:08.205977+0000","last_unstale":"2024-03-29T10:07:09.716184+0000","last_undegraded":"2024-03-29T10:07:09.716184+0000","last_fullsized":"2024-03-29T10:07:09.716184+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:39:59.847323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823281+0000","last_change":"2024-03-29T10:07:08.738498+0000","last_active":"2024-03-29T10:07:09.823281+0000","last_peered":"2024-03-29T10:07:09.823281+0000","last_clean":"2024-03-29T10:07:09.823281+0000","last_became_active":"2024-03-29T10:07:08.738191+0000","last_became_peered":"2024-03-29T10:07:08.738191+0000","last_unstale":"2024-03-29T10:07:09.823281+0000","last_undegraded":"2024-03-29T10:07:09.823281+0000","last_fullsized":"2024-03-29T10:07:09.823281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:38:26.106725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823517+0000","last_change":"2024-03-29T10:07:08.196381+0000","last_active":"2024-03-29T10:07:09.823517+0000","last_peered":"2024-03-29T10:07:09.823517+0000","last_clean":"2024-03-29T10:07:09.823517+0000","last_became_active":"2024-03-29T10:07:08.195195+0000","last_became_peered":"2024-03-29T10:07:08.195195+0000","last_unstale":"2024-03-29T10:07:09.823517+0000","last_undegraded":"2024-03-29T10:07:09.823517+0000","last_fullsized":"2024-03-29T10:07:09.823517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:39.653088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822699+0000","last_change":"2024-03-29T10:07:08.734816+0000","last_active":"2024-03-29T10:07:09.822699+0000","last_peered":"2024-03-29T10:07:09.822699+0000","last_clean":"2024-03-29T10:07:09.822699+0000","last_became_active":"2024-03-29T10:07:08.734548+0000","last_became_peered":"2024-03-29T10:07:08.734548+0000","last_unstale":"2024-03-29T10:07:09.822699+0000","last_undegraded":"2024-03-29T10:07:09.822699+0000","last_fullsized":"2024-03-29T10:07:09.822699+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:35:45.448688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716946+0000","last_change":"2024-03-29T10:07:08.207064+0000","last_active":"2024-03-29T10:07:09.716946+0000","last_peered":"2024-03-29T10:07:09.716946+0000","last_clean":"2024-03-29T10:07:09.716946+0000","last_became_active":"2024-03-29T10:07:08.206773+0000","last_became_peered":"2024-03-29T10:07:08.206773+0000","last_unstale":"2024-03-29T10:07:09.716946+0000","last_undegraded":"2024-03-29T10:07:09.716946+0000","last_fullsized":"2024-03-29T10:07:09.716946+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:19:00.560729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822588+0000","last_change":"2024-03-29T10:07:08.741377+0000","last_active":"2024-03-29T10:07:09.822588+0000","last_peered":"2024-03-29T10:07:09.822588+0000","last_clean":"2024-03-29T10:07:09.822588+0000","last_became_active":"2024-03-29T10:07:08.741053+0000","last_became_peered":"2024-03-29T10:07:08.741053+0000","last_unstale":"2024-03-29T10:07:09.822588+0000","last_undegraded":"2024-03-29T10:07:09.822588+0000","last_fullsized":"2024-03-29T10:07:09.822588+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:04.366498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"60'3","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822656+0000","last_change":"2024-03-29T10:07:08.206197+0000","last_active":"2024-03-29T10:07:09.822656+0000","last_peered":"2024-03-29T10:07:09.822656+0000","last_clean":"2024-03-29T10:07:09.822656+0000","last_became_active":"2024-03-29T10:07:08.205913+0000","last_became_peered":"2024-03-29T10:07:08.205913+0000","last_unstale":"2024-03-29T10:07:09.822656+0000","last_undegraded":"2024-03-29T10:07:09.822656+0000","last_fullsized":"2024-03-29T10:07:09.822656+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:03.538179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822528+0000","last_change":"2024-03-29T10:07:08.737195+0000","last_active":"2024-03-29T10:07:09.822528+0000","last_peered":"2024-03-29T10:07:09.822528+0000","last_clean":"2024-03-29T10:07:09.822528+0000","last_became_active":"2024-03-29T10:07:08.736915+0000","last_became_peered":"2024-03-29T10:07:08.736915+0000","last_unstale":"2024-03-29T10:07:09.822528+0000","last_undegraded":"2024-03-29T10:07:09.822528+0000","last_fullsized":"2024-03-29T10:07:09.822528+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:32:09.269929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"60'6","reported_seq":17,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823186+0000","last_change":"2024-03-29T10:07:08.199198+0000","last_active":"2024-03-29T10:07:09.823186+0000","last_peered":"2024-03-29T10:07:09.823186+0000","last_clean":"2024-03-29T10:07:09.823186+0000","last_became_active":"2024-03-29T10:07:08.198885+0000","last_became_peered":"2024-03-29T10:07:08.198885+0000","last_unstale":"2024-03-29T10:07:09.823186+0000","last_undegraded":"2024-03-29T10:07:09.823186+0000","last_fullsized":"2024-03-29T10:07:09.823186+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:28:42.506688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718207+0000","last_change":"2024-03-29T10:07:08.733310+0000","last_active":"2024-03-29T10:07:09.718207+0000","last_peered":"2024-03-29T10:07:09.718207+0000","last_clean":"2024-03-29T10:07:09.718207+0000","last_became_active":"2024-03-29T10:07:08.732961+0000","last_became_peered":"2024-03-29T10:07:08.732961+0000","last_unstale":"2024-03-29T10:07:09.718207+0000","last_undegraded":"2024-03-29T10:07:09.718207+0000","last_fullsized":"2024-03-29T10:07:09.718207+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:40.661943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822735+0000","last_change":"2024-03-29T10:07:08.207886+0000","last_active":"2024-03-29T10:07:09.822735+0000","last_peered":"2024-03-29T10:07:09.822735+0000","last_clean":"2024-03-29T10:07:09.822735+0000","last_became_active":"2024-03-29T10:07:08.200426+0000","last_became_peered":"2024-03-29T10:07:08.200426+0000","last_unstale":"2024-03-29T10:07:09.822735+0000","last_undegraded":"2024-03-29T10:07:09.822735+0000","last_fullsized":"2024-03-29T10:07:09.822735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:04.805332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716466+0000","last_change":"2024-03-29T10:07:08.731290+0000","last_active":"2024-03-29T10:07:09.716466+0000","last_peered":"2024-03-29T10:07:09.716466+0000","last_clean":"2024-03-29T10:07:09.716466+0000","last_became_active":"2024-03-29T10:07:08.731000+0000","last_became_peered":"2024-03-29T10:07:08.731000+0000","last_unstale":"2024-03-29T10:07:09.716466+0000","last_undegraded":"2024-03-29T10:07:09.716466+0000","last_fullsized":"2024-03-29T10:07:09.716466+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:52:16.286951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822311+0000","last_change":"2024-03-29T10:07:08.209619+0000","last_active":"2024-03-29T10:07:09.822311+0000","last_peered":"2024-03-29T10:07:09.822311+0000","last_clean":"2024-03-29T10:07:09.822311+0000","last_became_active":"2024-03-29T10:07:08.209349+0000","last_became_peered":"2024-03-29T10:07:08.209349+0000","last_unstale":"2024-03-29T10:07:09.822311+0000","last_undegraded":"2024-03-29T10:07:09.822311+0000","last_fullsized":"2024-03-29T10:07:09.822311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:40.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822464+0000","last_change":"2024-03-29T10:07:08.733800+0000","last_active":"2024-03-29T10:07:09.822464+0000","last_peered":"2024-03-29T10:07:09.822464+0000","last_clean":"2024-03-29T10:07:09.822464+0000","last_became_active":"2024-03-29T10:07:08.733545+0000","last_became_peered":"2024-03-29T10:07:08.733545+0000","last_unstale":"2024-03-29T10:07:09.822464+0000","last_undegraded":"2024-03-29T10:07:09.822464+0000","last_fullsized":"2024-03-29T10:07:09.822464+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:58:10.549612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822883+0000","last_change":"2024-03-29T10:07:08.208058+0000","last_active":"2024-03-29T10:07:09.822883+0000","last_peered":"2024-03-29T10:07:09.822883+0000","last_clean":"2024-03-29T10:07:09.822883+0000","last_became_active":"2024-03-29T10:07:08.206370+0000","last_became_peered":"2024-03-29T10:07:08.206370+0000","last_unstale":"2024-03-29T10:07:09.822883+0000","last_undegraded":"2024-03-29T10:07:09.822883+0000","last_fullsized":"2024-03-29T10:07:09.822883+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:03.538179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822301+0000","last_change":"2024-03-29T10:07:08.738620+0000","last_active":"2024-03-29T10:07:09.822301+0000","last_peered":"2024-03-29T10:07:09.822301+0000","last_clean":"2024-03-29T10:07:09.822301+0000","last_became_active":"2024-03-29T10:07:08.737378+0000","last_became_peered":"2024-03-29T10:07:08.737378+0000","last_unstale":"2024-03-29T10:07:09.822301+0000","last_undegraded":"2024-03-29T10:07:09.822301+0000","last_fullsized":"2024-03-29T10:07:09.822301+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:17:01.202851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"60'8","reported_seq":19,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716087+0000","last_change":"2024-03-29T10:07:08.201721+0000","last_active":"2024-03-29T10:07:09.716087+0000","last_peered":"2024-03-29T10:07:09.716087+0000","last_clean":"2024-03-29T10:07:09.716087+0000","last_became_active":"2024-03-29T10:07:08.201362+0000","last_became_peered":"2024-03-29T10:07:08.201362+0000","last_unstale":"2024-03-29T10:07:09.716087+0000","last_undegraded":"2024-03-29T10:07:09.716087+0000","last_fullsized":"2024-03-29T10:07:09.716087+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:57.273274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823759+0000","last_change":"2024-03-29T10:07:08.725051+0000","last_active":"2024-03-29T10:07:09.823759+0000","last_peered":"2024-03-29T10:07:09.823759+0000","last_clean":"2024-03-29T10:07:09.823759+0000","last_became_active":"2024-03-29T10:07:08.724617+0000","last_became_peered":"2024-03-29T10:07:08.724617+0000","last_unstale":"2024-03-29T10:07:09.823759+0000","last_undegraded":"2024-03-29T10:07:09.823759+0000","last_fullsized":"2024-03-29T10:07:09.823759+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:46.657982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823910+0000","last_change":"2024-03-29T10:07:08.204845+0000","last_active":"2024-03-29T10:07:09.823910+0000","last_peered":"2024-03-29T10:07:09.823910+0000","last_clean":"2024-03-29T10:07:09.823910+0000","last_became_active":"2024-03-29T10:07:08.204400+0000","last_became_peered":"2024-03-29T10:07:08.204400+0000","last_unstale":"2024-03-29T10:07:09.823910+0000","last_undegraded":"2024-03-29T10:07:09.823910+0000","last_fullsized":"2024-03-29T10:07:09.823910+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:30:10.665464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716947+0000","last_change":"2024-03-29T10:07:08.744649+0000","last_active":"2024-03-29T10:07:09.716947+0000","last_peered":"2024-03-29T10:07:09.716947+0000","last_clean":"2024-03-29T10:07:09.716947+0000","last_became_active":"2024-03-29T10:07:08.744048+0000","last_became_peered":"2024-03-29T10:07:08.744048+0000","last_unstale":"2024-03-29T10:07:09.716947+0000","last_undegraded":"2024-03-29T10:07:09.716947+0000","last_fullsized":"2024-03-29T10:07:09.716947+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:59.267610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823264+0000","last_change":"2024-03-29T10:07:08.200629+0000","last_active":"2024-03-29T10:07:09.823264+0000","last_peered":"2024-03-29T10:07:09.823264+0000","last_clean":"2024-03-29T10:07:09.823264+0000","last_became_active":"2024-03-29T10:07:08.199307+0000","last_became_peered":"2024-03-29T10:07:08.199307+0000","last_unstale":"2024-03-29T10:07:09.823264+0000","last_undegraded":"2024-03-29T10:07:09.823264+0000","last_fullsized":"2024-03-29T10:07:09.823264+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:17:29.325879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718528+0000","last_change":"2024-03-29T10:07:08.745407+0000","last_active":"2024-03-29T10:07:09.718528+0000","last_peered":"2024-03-29T10:07:09.718528+0000","last_clean":"2024-03-29T10:07:09.718528+0000","last_became_active":"2024-03-29T10:07:08.745057+0000","last_became_peered":"2024-03-29T10:07:08.745057+0000","last_unstale":"2024-03-29T10:07:09.718528+0000","last_undegraded":"2024-03-29T10:07:09.718528+0000","last_fullsized":"2024-03-29T10:07:09.718528+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:44.957025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717500+0000","last_change":"2024-03-29T10:07:08.197172+0000","last_active":"2024-03-29T10:07:09.717500+0000","last_peered":"2024-03-29T10:07:09.717500+0000","last_clean":"2024-03-29T10:07:09.717500+0000","last_became_active":"2024-03-29T10:07:08.196895+0000","last_became_peered":"2024-03-29T10:07:08.196895+0000","last_unstale":"2024-03-29T10:07:09.717500+0000","last_undegraded":"2024-03-29T10:07:09.717500+0000","last_fullsized":"2024-03-29T10:07:09.717500+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:53.898093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718660+0000","last_change":"2024-03-29T10:07:08.734869+0000","last_active":"2024-03-29T10:07:09.718660+0000","last_peered":"2024-03-29T10:07:09.718660+0000","last_clean":"2024-03-29T10:07:09.718660+0000","last_became_active":"2024-03-29T10:07:08.734370+0000","last_became_peered":"2024-03-29T10:07:08.734370+0000","last_unstale":"2024-03-29T10:07:09.718660+0000","last_undegraded":"2024-03-29T10:07:09.718660+0000","last_fullsized":"2024-03-29T10:07:09.718660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:54.906947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822816+0000","last_change":"2024-03-29T10:07:08.200837+0000","last_active":"2024-03-29T10:07:09.822816+0000","last_peered":"2024-03-29T10:07:09.822816+0000","last_clean":"2024-03-29T10:07:09.822816+0000","last_became_active":"2024-03-29T10:07:08.200087+0000","last_became_peered":"2024-03-29T10:07:08.200087+0000","last_unstale":"2024-03-29T10:07:09.822816+0000","last_undegraded":"2024-03-29T10:07:09.822816+0000","last_fullsized":"2024-03-29T10:07:09.822816+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:40.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822174+0000","last_change":"2024-03-29T10:07:08.733860+0000","last_active":"2024-03-29T10:07:09.822174+0000","last_peered":"2024-03-29T10:07:09.822174+0000","last_clean":"2024-03-29T10:07:09.822174+0000","last_became_active":"2024-03-29T10:07:08.733585+0000","last_became_peered":"2024-03-29T10:07:08.733585+0000","last_unstale":"2024-03-29T10:07:09.822174+0000","last_undegraded":"2024-03-29T10:07:09.822174+0000","last_fullsized":"2024-03-29T10:07:09.822174+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:09:57.508574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822822+0000","last_change":"2024-03-29T10:07:08.201523+0000","last_active":"2024-03-29T10:07:09.822822+0000","last_peered":"2024-03-29T10:07:09.822822+0000","last_clean":"2024-03-29T10:07:09.822822+0000","last_became_active":"2024-03-29T10:07:08.200965+0000","last_became_peered":"2024-03-29T10:07:08.200965+0000","last_unstale":"2024-03-29T10:07:09.822822+0000","last_undegraded":"2024-03-29T10:07:09.822822+0000","last_fullsized":"2024-03-29T10:07:09.822822+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:23.693049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822939+0000","last_change":"2024-03-29T10:07:08.735590+0000","last_active":"2024-03-29T10:07:09.822939+0000","last_peered":"2024-03-29T10:07:09.822939+0000","last_clean":"2024-03-29T10:07:09.822939+0000","last_became_active":"2024-03-29T10:07:08.735343+0000","last_became_peered":"2024-03-29T10:07:08.735343+0000","last_unstale":"2024-03-29T10:07:09.822939+0000","last_undegraded":"2024-03-29T10:07:09.822939+0000","last_fullsized":"2024-03-29T10:07:09.822939+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:54.906947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717318+0000","last_change":"2024-03-29T10:07:08.194559+0000","last_active":"2024-03-29T10:07:09.717318+0000","last_peered":"2024-03-29T10:07:09.717318+0000","last_clean":"2024-03-29T10:07:09.717318+0000","last_became_active":"2024-03-29T10:07:08.194019+0000","last_became_peered":"2024-03-29T10:07:08.194019+0000","last_unstale":"2024-03-29T10:07:09.717318+0000","last_undegraded":"2024-03-29T10:07:09.717318+0000","last_fullsized":"2024-03-29T10:07:09.717318+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:56:52.189817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822375+0000","last_change":"2024-03-29T10:07:08.734737+0000","last_active":"2024-03-29T10:07:09.822375+0000","last_peered":"2024-03-29T10:07:09.822375+0000","last_clean":"2024-03-29T10:07:09.822375+0000","last_became_active":"2024-03-29T10:07:08.734466+0000","last_became_peered":"2024-03-29T10:07:08.734466+0000","last_unstale":"2024-03-29T10:07:09.822375+0000","last_undegraded":"2024-03-29T10:07:09.822375+0000","last_fullsized":"2024-03-29T10:07:09.822375+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:06:31.440735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716226+0000","last_change":"2024-03-29T10:07:08.205793+0000","last_active":"2024-03-29T10:07:09.716226+0000","last_peered":"2024-03-29T10:07:09.716226+0000","last_clean":"2024-03-29T10:07:09.716226+0000","last_became_active":"2024-03-29T10:07:08.205554+0000","last_became_peered":"2024-03-29T10:07:08.205554+0000","last_unstale":"2024-03-29T10:07:09.716226+0000","last_undegraded":"2024-03-29T10:07:09.716226+0000","last_fullsized":"2024-03-29T10:07:09.716226+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:15:59.255088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716018+0000","last_change":"2024-03-29T10:07:08.732734+0000","last_active":"2024-03-29T10:07:09.716018+0000","last_peered":"2024-03-29T10:07:09.716018+0000","last_clean":"2024-03-29T10:07:09.716018+0000","last_became_active":"2024-03-29T10:07:08.732343+0000","last_became_peered":"2024-03-29T10:07:08.732343+0000","last_unstale":"2024-03-29T10:07:09.716018+0000","last_undegraded":"2024-03-29T10:07:09.716018+0000","last_fullsized":"2024-03-29T10:07:09.716018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:04.724057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822171+0000","last_change":"2024-03-29T10:07:08.203965+0000","last_active":"2024-03-29T10:07:09.822171+0000","last_peered":"2024-03-29T10:07:09.822171+0000","last_clean":"2024-03-29T10:07:09.822171+0000","last_became_active":"2024-03-29T10:07:08.203702+0000","last_became_peered":"2024-03-29T10:07:08.203702+0000","last_unstale":"2024-03-29T10:07:09.822171+0000","last_undegraded":"2024-03-29T10:07:09.822171+0000","last_fullsized":"2024-03-29T10:07:09.822171+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:23.693049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718644+0000","last_change":"2024-03-29T10:07:08.740198+0000","last_active":"2024-03-29T10:07:09.718644+0000","last_peered":"2024-03-29T10:07:09.718644+0000","last_clean":"2024-03-29T10:07:09.718644+0000","last_became_active":"2024-03-29T10:07:08.739927+0000","last_became_peered":"2024-03-29T10:07:08.739927+0000","last_unstale":"2024-03-29T10:07:09.718644+0000","last_undegraded":"2024-03-29T10:07:09.718644+0000","last_fullsized":"2024-03-29T10:07:09.718644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:54:34.550128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822485+0000","last_change":"2024-03-29T10:07:08.198415+0000","last_active":"2024-03-29T10:07:09.822485+0000","last_peered":"2024-03-29T10:07:09.822485+0000","last_clean":"2024-03-29T10:07:09.822485+0000","last_became_active":"2024-03-29T10:07:08.197508+0000","last_became_peered":"2024-03-29T10:07:08.197508+0000","last_unstale":"2024-03-29T10:07:09.822485+0000","last_undegraded":"2024-03-29T10:07:09.822485+0000","last_fullsized":"2024-03-29T10:07:09.822485+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:22:32.165803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716278+0000","last_change":"2024-03-29T10:07:08.721781+0000","last_active":"2024-03-29T10:07:09.716278+0000","last_peered":"2024-03-29T10:07:09.716278+0000","last_clean":"2024-03-29T10:07:09.716278+0000","last_became_active":"2024-03-29T10:07:08.721481+0000","last_became_peered":"2024-03-29T10:07:08.721481+0000","last_unstale":"2024-03-29T10:07:09.716278+0000","last_undegraded":"2024-03-29T10:07:09.716278+0000","last_fullsized":"2024-03-29T10:07:09.716278+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:06:31.440735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822677+0000","last_change":"2024-03-29T10:07:08.208437+0000","last_active":"2024-03-29T10:07:09.822677+0000","last_peered":"2024-03-29T10:07:09.822677+0000","last_clean":"2024-03-29T10:07:09.822677+0000","last_became_active":"2024-03-29T10:07:08.208183+0000","last_became_peered":"2024-03-29T10:07:08.208183+0000","last_unstale":"2024-03-29T10:07:09.822677+0000","last_undegraded":"2024-03-29T10:07:09.822677+0000","last_fullsized":"2024-03-29T10:07:09.822677+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:43.145205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822587+0000","last_change":"2024-03-29T10:07:08.738937+0000","last_active":"2024-03-29T10:07:09.822587+0000","last_peered":"2024-03-29T10:07:09.822587+0000","last_clean":"2024-03-29T10:07:09.822587+0000","last_became_active":"2024-03-29T10:07:08.738655+0000","last_became_peered":"2024-03-29T10:07:08.738655+0000","last_unstale":"2024-03-29T10:07:09.822587+0000","last_undegraded":"2024-03-29T10:07:09.822587+0000","last_fullsized":"2024-03-29T10:07:09.822587+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:48.896404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.824027+0000","last_change":"2024-03-29T10:07:08.199454+0000","last_active":"2024-03-29T10:07:09.824027+0000","last_peered":"2024-03-29T10:07:09.824027+0000","last_clean":"2024-03-29T10:07:09.824027+0000","last_became_active":"2024-03-29T10:07:08.198502+0000","last_became_peered":"2024-03-29T10:07:08.198502+0000","last_unstale":"2024-03-29T10:07:09.824027+0000","last_undegraded":"2024-03-29T10:07:09.824027+0000","last_fullsized":"2024-03-29T10:07:09.824027+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:28:42.506688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822280+0000","last_change":"2024-03-29T10:07:08.733276+0000","last_active":"2024-03-29T10:07:09.822280+0000","last_peered":"2024-03-29T10:07:09.822280+0000","last_clean":"2024-03-29T10:07:09.822280+0000","last_became_active":"2024-03-29T10:07:08.733018+0000","last_became_peered":"2024-03-29T10:07:08.733018+0000","last_unstale":"2024-03-29T10:07:09.822280+0000","last_undegraded":"2024-03-29T10:07:09.822280+0000","last_fullsized":"2024-03-29T10:07:09.822280+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:58:51.764293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822376+0000","last_change":"2024-03-29T10:07:08.207012+0000","last_active":"2024-03-29T10:07:09.822376+0000","last_peered":"2024-03-29T10:07:09.822376+0000","last_clean":"2024-03-29T10:07:09.822376+0000","last_became_active":"2024-03-29T10:07:08.206780+0000","last_became_peered":"2024-03-29T10:07:08.206780+0000","last_unstale":"2024-03-29T10:07:09.822376+0000","last_undegraded":"2024-03-29T10:07:09.822376+0000","last_fullsized":"2024-03-29T10:07:09.822376+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:04.805332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822550+0000","last_change":"2024-03-29T10:07:08.740768+0000","last_active":"2024-03-29T10:07:09.822550+0000","last_peered":"2024-03-29T10:07:09.822550+0000","last_clean":"2024-03-29T10:07:09.822550+0000","last_became_active":"2024-03-29T10:07:08.740462+0000","last_became_peered":"2024-03-29T10:07:08.740462+0000","last_unstale":"2024-03-29T10:07:09.822550+0000","last_undegraded":"2024-03-29T10:07:09.822550+0000","last_fullsized":"2024-03-29T10:07:09.822550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:59:28.495332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823411+0000","last_change":"2024-03-29T10:07:08.206790+0000","last_active":"2024-03-29T10:07:09.823411+0000","last_peered":"2024-03-29T10:07:09.823411+0000","last_clean":"2024-03-29T10:07:09.823411+0000","last_became_active":"2024-03-29T10:07:08.206510+0000","last_became_peered":"2024-03-29T10:07:08.206510+0000","last_unstale":"2024-03-29T10:07:09.823411+0000","last_undegraded":"2024-03-29T10:07:09.823411+0000","last_fullsized":"2024-03-29T10:07:09.823411+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:03.715203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717429+0000","last_change":"2024-03-29T10:07:08.727909+0000","last_active":"2024-03-29T10:07:09.717429+0000","last_peered":"2024-03-29T10:07:09.717429+0000","last_clean":"2024-03-29T10:07:09.717429+0000","last_became_active":"2024-03-29T10:07:08.727031+0000","last_became_peered":"2024-03-29T10:07:08.727031+0000","last_unstale":"2024-03-29T10:07:09.717429+0000","last_undegraded":"2024-03-29T10:07:09.717429+0000","last_fullsized":"2024-03-29T10:07:09.717429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:04.547033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822248+0000","last_change":"2024-03-29T10:07:08.202910+0000","last_active":"2024-03-29T10:07:09.822248+0000","last_peered":"2024-03-29T10:07:09.822248+0000","last_clean":"2024-03-29T10:07:09.822248+0000","last_became_active":"2024-03-29T10:07:08.202459+0000","last_became_peered":"2024-03-29T10:07:08.202459+0000","last_unstale":"2024-03-29T10:07:09.822248+0000","last_undegraded":"2024-03-29T10:07:09.822248+0000","last_fullsized":"2024-03-29T10:07:09.822248+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:57.273274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823101+0000","last_change":"2024-03-29T10:07:08.745893+0000","last_active":"2024-03-29T10:07:09.823101+0000","last_peered":"2024-03-29T10:07:09.823101+0000","last_clean":"2024-03-29T10:07:09.823101+0000","last_became_active":"2024-03-29T10:07:08.745612+0000","last_became_peered":"2024-03-29T10:07:08.745612+0000","last_unstale":"2024-03-29T10:07:09.823101+0000","last_undegraded":"2024-03-29T10:07:09.823101+0000","last_fullsized":"2024-03-29T10:07:09.823101+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:27:01.767573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716085+0000","last_change":"2024-03-29T10:07:08.207099+0000","last_active":"2024-03-29T10:07:09.716085+0000","last_peered":"2024-03-29T10:07:09.716085+0000","last_clean":"2024-03-29T10:07:09.716085+0000","last_became_active":"2024-03-29T10:07:08.206706+0000","last_became_peered":"2024-03-29T10:07:08.206706+0000","last_unstale":"2024-03-29T10:07:09.716085+0000","last_undegraded":"2024-03-29T10:07:09.716085+0000","last_fullsized":"2024-03-29T10:07:09.716085+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:00.678829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716975+0000","last_change":"2024-03-29T10:07:08.739624+0000","last_active":"2024-03-29T10:07:09.716975+0000","last_peered":"2024-03-29T10:07:09.716975+0000","last_clean":"2024-03-29T10:07:09.716975+0000","last_became_active":"2024-03-29T10:07:08.739063+0000","last_became_peered":"2024-03-29T10:07:08.739063+0000","last_unstale":"2024-03-29T10:07:09.716975+0000","last_undegraded":"2024-03-29T10:07:09.716975+0000","last_fullsized":"2024-03-29T10:07:09.716975+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:46.657982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718126+0000","last_change":"2024-03-29T10:07:08.201370+0000","last_active":"2024-03-29T10:07:09.718126+0000","last_peered":"2024-03-29T10:07:09.718126+0000","last_clean":"2024-03-29T10:07:09.718126+0000","last_became_active":"2024-03-29T10:07:08.201044+0000","last_became_peered":"2024-03-29T10:07:08.201044+0000","last_unstale":"2024-03-29T10:07:09.718126+0000","last_undegraded":"2024-03-29T10:07:09.718126+0000","last_fullsized":"2024-03-29T10:07:09.718126+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:45.649127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"49'139","reported_seq":283,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-29T10:07:08.726454+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:07:08.726454+0000","last_peered":"2024-03-29T10:07:08.726454+0000","last_clean":"2024-03-29T10:07:08.726454+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:07:08.726454+0000","last_undegraded":"2024-03-29T10:07:08.726454+0000","last_fullsized":"2024-03-29T10:07:08.726454+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575003,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32400,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339440,"statfs":{"total":96636764160,"available":96603586560,"internally_reserved":0,"allocated":1310720,"data_stored":921162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.71899999999999997,"15min":0.70899999999999996},"min":{"1min":0.48299999999999998,"5min":0.44500000000000001,"15min":0.32500000000000001},"max":{"1min":0.97899999999999998,"5min":1.0029999999999999,"15min":1.0109999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70999999999999996,"15min":0.69699999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.046},"last":0.63800000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73199999999999998,"15min":0.71299999999999997},"min":{"1min":0.48899999999999999,"5min":0.48799999999999999,"15min":0.44500000000000001},"max":{"1min":0.91900000000000004,"5min":0.99099999999999999,"15min":1.1910000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.73599999999999999,"15min":0.72499999999999998},"min":{"1min":0.42199999999999999,"5min":0.41199999999999998,"15min":0.372},"max":{"1min":0.94799999999999995,"5min":0.96299999999999997,"15min":1},"last":0.61899999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.72599999999999998,"15min":0.73599999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":0.89900000000000002,"5min":0.97699999999999998,"15min":1.046},"last":0.75},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.76100000000000001,"15min":0.747},"min":{"1min":0.55200000000000005,"5min":0.51100000000000001,"15min":0.48999999999999999},"max":{"1min":0.93999999999999995,"5min":1.014,"15min":1.079},"last":0.79400000000000004}]},{"osd":3,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73699999999999999,"15min":0.73199999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.95499999999999996,"5min":1.0149999999999999,"15min":1.081},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78500000000000003,"15min":0.78100000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":0.98099999999999998,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.78300000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.60699999999999998},"min":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.29199999999999998},"max":{"1min":0.72299999999999998,"5min":0.87,"15min":1.111},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.621,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.746,"5min":0.88100000000000001,"15min":0.95599999999999996},"last":0.76900000000000002}]},{"osd":5,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.66700000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.83299999999999996,"5min":0.94599999999999995,"15min":1.0089999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66700000000000004,"15min":0.69599999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83299999999999996,"5min":0.92800000000000005,"15min":1.141},"last":0.89600000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.73799999999999999},"min":{"1min":0.57199999999999995,"5min":0.51200000000000001,"15min":0.49199999999999999},"max":{"1min":0.79800000000000004,"5min":0.93500000000000005,"15min":1.1679999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.435},"max":{"1min":0.81799999999999995,"5min":0.95599999999999996,"15min":1.121},"last":0.873}]}]},{"osd":6,"up_from":13,"seq":55834575004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":720896,"data_stored":331376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.70099999999999996,"15min":0.77800000000000002},"min":{"1min":0.52400000000000002,"5min":0.38,"15min":0.34699999999999998},"max":{"1min":0.84099999999999997,"5min":0.95599999999999996,"15min":7.4080000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.75,"15min":0.80300000000000005},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.42499999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":6.7610000000000001},"last":0.497}]},{"osd":1,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72699999999999998,"15min":0.71999999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.92100000000000004,"5min":0.97299999999999998,"15min":1.2549999999999999},"last":0.627},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72899999999999998,"15min":0.71299999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98099999999999998,"5min":0.995,"15min":1.0209999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.73199999999999998,"15min":0.72299999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96499999999999997,"5min":1.014,"15min":1.3169999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.74099999999999999,"15min":0.75600000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":0.97099999999999997,"5min":0.99199999999999999,"15min":1.415},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.75600000000000001,"15min":0.78300000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":1.02,"15min":3.1960000000000002},"last":0.747},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.77100000000000002,"15min":0.82199999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.014,"5min":1.0229999999999999,"15min":4.3760000000000003},"last":0.73199999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":0.69699999999999995,"5min":0.90900000000000003,"15min":1.109},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.69099999999999995,"15min":0.72099999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83399999999999996,"5min":0.95599999999999996,"15min":3.1219999999999999},"last":0.67800000000000005}]},{"osd":5,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65800000000000003,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.42899999999999999},"max":{"1min":0.874,"5min":0.88900000000000001,"15min":2.8279999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.48199999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.86799999999999999,"5min":0.96899999999999997,"15min":1},"last":0.77400000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.754,"15min":0.76800000000000002},"min":{"1min":0.56599999999999995,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.98699999999999999,"5min":1.048,"15min":1.2769999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.72599999999999998,"15min":0.746},"min":{"1min":0.56599999999999995,"5min":0.49399999999999999,"15min":0.40000000000000002},"max":{"1min":0.998,"5min":0.998,"15min":1.7869999999999999},"last":0.78600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49868,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321972,"statfs":{"total":96636764160,"available":96585699328,"internally_reserved":0,"allocated":716800,"data_stored":330774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.435,"15min":0.45400000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55600000000000005,"5min":0.72799999999999998,"15min":1.252},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52400000000000002,"15min":0.51400000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32700000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":1.2989999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52700000000000002,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.33700000000000002},"max":{"1min":0.60099999999999998,"5min":0.83399999999999996,"15min":4.282},"last":0.433}]},{"osd":3,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39000000000000001},"max":{"1min":0.745,"5min":0.78000000000000003,"15min":4.3200000000000003},"last":0.377},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59199999999999997,"15min":0.57699999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.373},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":1.006},"last":0.59399999999999997}]},{"osd":4,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0580000000000001,"5min":1.179,"15min":1.365},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.77300000000000002,"15min":0.82899999999999996},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.51000000000000001},"max":{"1min":0.79300000000000004,"5min":1.155,"15min":4.3559999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80900000000000005,"15min":0.82199999999999995},"min":{"1min":0.64200000000000002,"5min":0.55500000000000005,"15min":0.51800000000000002},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.9119999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.79800000000000004,"15min":0.82399999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.01,"5min":1.345,"15min":1.6100000000000001},"last":0.83899999999999997}]},{"osd":6,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.80400000000000005,"15min":0.85799999999999998},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.51900000000000002},"max":{"1min":0.86399999999999999,"5min":1.236,"15min":4.3899999999999997},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.81499999999999995,"15min":0.83599999999999997},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.55400000000000005},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.73999999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84799999999999998,"15min":0.85299999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.56200000000000006},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.82799999999999996,"15min":0.85099999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.55700000000000005},"max":{"1min":0.95799999999999996,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.66600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575004,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1306624,"data_stored":920087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":0.67400000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.33800000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.66100000000000003}]},{"osd":2,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57799999999999996,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.35499999999999998},"max":{"1min":0.80700000000000005,"5min":0.96899999999999997,"15min":1.0069999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.51800000000000002,"15min":0.497},"min":{"1min":0.33400000000000002,"5min":0.32200000000000001,"15min":0.28599999999999998},"max":{"1min":0.56000000000000005,"5min":0.83899999999999997,"15min":0.97199999999999998},"last":0.41699999999999998}]},{"osd":3,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57499999999999996,"15min":0.54500000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.34699999999999998},"max":{"1min":0.73599999999999999,"5min":1.077,"15min":1.077},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.63200000000000001,"15min":0.60499999999999998},"min":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.41699999999999998},"max":{"1min":0.78900000000000003,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.49299999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84099999999999997,"15min":0.80200000000000005},"min":{"1min":0.51600000000000001,"5min":0.46400000000000002,"15min":0.39200000000000002},"max":{"1min":1.083,"5min":1.214,"15min":1.226},"last":1.625},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.85299999999999998,"15min":0.82599999999999996},"min":{"1min":0.61399999999999999,"5min":0.50800000000000001,"15min":0.45800000000000002},"max":{"1min":0.97699999999999998,"5min":1.1419999999999999,"15min":1.167},"last":0.88400000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.79400000000000004,"15min":0.76500000000000001},"min":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.51600000000000001},"max":{"1min":0.99399999999999999,"5min":1.137,"15min":1.137},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.83099999999999996,"15min":0.80100000000000005},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.41099999999999998},"max":{"1min":0.93400000000000005,"5min":1.151,"15min":1.2390000000000001},"last":1.5309999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.83799999999999997,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.56299999999999994,"15min":0.53400000000000003},"max":{"1min":0.98199999999999998,"5min":1.1659999999999999,"15min":1.228},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.88200000000000001,"15min":0.84499999999999997},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.46899999999999997},"max":{"1min":1.0209999999999999,"5min":1.1950000000000001,"15min":1.2450000000000001},"last":1.601}]},{"osd":7,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.83799999999999997,"15min":0.80300000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.90600000000000003,"5min":1.0529999999999999,"15min":1.1890000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.87,"15min":0.84099999999999997},"min":{"1min":0.624,"5min":0.59199999999999997,"15min":0.49299999999999999},"max":{"1min":1,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":1.5669999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35932,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335908,"statfs":{"total":96636764160,"available":96599969792,"internally_reserved":0,"allocated":729088,"data_stored":332290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.59699999999999998,"5min":0.69599999999999995,"15min":0.86599999999999999},"last":0.373},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66200000000000003,"5min":0.78800000000000003,"15min":1.3140000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46400000000000002,"15min":0.46700000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.63500000000000001,"5min":0.751,"15min":0.94499999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56399999999999995,"15min":0.55400000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.32700000000000001},"max":{"1min":0.76600000000000001,"5min":1.032,"15min":1.032},"last":0.53300000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57399999999999995,"15min":0.57899999999999996},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.377},"max":{"1min":0.749,"5min":0.80000000000000004,"15min":1.127},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.54100000000000004,"15min":0.55100000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69999999999999996,"5min":0.82399999999999995,"15min":1.1479999999999999},"last":0.41899999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.78100000000000003,"15min":0.78800000000000003},"min":{"1min":0.44800000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.02,"5min":1.3520000000000001,"15min":1.554},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.81000000000000005,"15min":0.81599999999999995},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.51600000000000001},"max":{"1min":0.89400000000000002,"5min":1.236,"15min":1.577},"last":0.67000000000000004}]},{"osd":5,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.82499999999999996,"15min":0.81699999999999995},"min":{"1min":0.58299999999999996,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.034,"5min":1.2589999999999999,"15min":1.5209999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.81299999999999994,"15min":0.81599999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.51800000000000002},"max":{"1min":0.96899999999999997,"5min":1.2250000000000001,"15min":1.3300000000000001},"last":0.65300000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81899999999999995,"15min":0.83999999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54100000000000004},"max":{"1min":0.98599999999999999,"5min":1.1539999999999999,"15min":1.758},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.57799999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.071,"5min":1.3560000000000001,"15min":1.4750000000000001},"last":0.76200000000000001}]},{"osd":7,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.879,"15min":0.871},"min":{"1min":0.628,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0549999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.627},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.88300000000000001,"15min":0.88700000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":0.996,"5min":1.4199999999999999,"15min":1.4390000000000001},"last":0.80300000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":724992,"data_stored":331342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.46200000000000002,"15min":0.47999999999999998},"min":{"1min":0.30299999999999999,"5min":0.26500000000000001,"15min":0.26000000000000001},"max":{"1min":0.57199999999999995,"5min":0.81299999999999994,"15min":4.3799999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52100000000000002,"15min":0.54600000000000004},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":4.6379999999999999},"last":0.47299999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.499,"15min":0.51900000000000002},"min":{"1min":0.36099999999999999,"5min":0.35799999999999998,"15min":0.27600000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":4.1840000000000002},"last":0.503},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58199999999999996,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.314},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.048},"last":0.56499999999999995}]},{"osd":2,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57199999999999995,"15min":0.55100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.33700000000000002},"max":{"1min":0.63100000000000001,"5min":0.82499999999999996,"15min":1.369},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.64500000000000002,"15min":0.66200000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":4.5970000000000004},"last":0.68200000000000005}]},{"osd":4,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.002,"5min":1.288,"15min":4.3319999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.79400000000000004,"15min":0.77300000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.39400000000000002},"max":{"1min":1.0640000000000001,"5min":1.46,"15min":1.46},"last":0.86599999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.78300000000000003,"15min":0.79300000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.435},"max":{"1min":0.873,"5min":1.3580000000000001,"15min":1.3740000000000001},"last":0.995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.82499999999999996,"15min":0.82299999999999995},"min":{"1min":0.5,"5min":0.5,"15min":0.44500000000000001},"max":{"1min":1.0629999999999999,"5min":1.429,"15min":4.4180000000000001},"last":0.88300000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.83399999999999996,"15min":0.81100000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.101,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.82799999999999996,"15min":0.81899999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.503},"max":{"1min":0.94199999999999995,"5min":1.3999999999999999,"15min":1.417},"last":0.96099999999999997}]},{"osd":7,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.85299999999999998,"15min":0.86099999999999999},"min":{"1min":0.57499999999999996,"5min":0.56799999999999995,"15min":0.54300000000000004},"max":{"1min":1.123,"5min":1.5029999999999999,"15min":4.4550000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.84899999999999998,"15min":0.84199999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.56200000000000006},"max":{"1min":1.0660000000000001,"5min":1.345,"15min":1.476},"last":0.84999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53976,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317864,"statfs":{"total":96636764160,"available":96581492736,"internally_reserved":0,"allocated":724992,"data_stored":332268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.69399999999999995,"15min":0.66900000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79600000000000004,"5min":1.417,"15min":1.417},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.71399999999999997,"15min":0.68799999999999994},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81599999999999995,"5min":1.165,"15min":1.165},"last":0.60399999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71899999999999997,"15min":0.69699999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.91300000000000003,"5min":0.98399999999999999,"15min":1.139},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.72399999999999998,"15min":0.70299999999999996},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.44800000000000001},"max":{"1min":0.82399999999999995,"5min":1.333,"15min":1.333},"last":0.62}]},{"osd":2,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78200000000000003,"15min":0.75800000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.41299999999999998},"max":{"1min":0.86799999999999999,"5min":1.03,"15min":1.863},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.76600000000000001,"15min":0.75600000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.84899999999999998,"5min":1.387,"15min":1.387},"last":0.70999999999999996}]},{"osd":3,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.73499999999999999,"15min":0.71499999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46999999999999997},"max":{"1min":0.93700000000000006,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.76000000000000001,"15min":0.73499999999999999},"min":{"1min":0.58599999999999997,"5min":0.52100000000000002,"15min":0.46600000000000003},"max":{"1min":0.876,"5min":1.3180000000000001,"15min":1.905},"last":0.76900000000000002}]},{"osd":5,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.57599999999999996,"15min":0.60299999999999998},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77200000000000002,"5min":0.85599999999999998,"15min":0.92000000000000004},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.63300000000000001},"min":{"1min":0.46000000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89000000000000001,"5min":1.294,"15min":1.294},"last":0.50900000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61699999999999999,"15min":0.64300000000000002},"min":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.41399999999999998},"max":{"1min":0.79800000000000004,"5min":1.0129999999999999,"15min":1.9339999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67000000000000004,"15min":0.68200000000000005},"min":{"1min":0.53000000000000003,"5min":0.45600000000000002,"15min":0.433},"max":{"1min":0.78400000000000003,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.74099999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.70499999999999996,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.442,"15min":0.41799999999999998},"max":{"1min":0.97799999999999998,"5min":1.345,"15min":1.345},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76900000000000002,"15min":0.77800000000000002},"min":{"1min":0.61699999999999999,"5min":0.51300000000000001,"15min":0.50700000000000001},"max":{"1min":0.90400000000000003,"5min":1.214,"15min":1.97},"last":0.79900000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":716800,"data_stored":330774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.71699999999999997,"15min":0.67400000000000004},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":0.871,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.70199999999999996},"min":{"1min":0.55300000000000005,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0529999999999999,"5min":1.2649999999999999,"15min":1.339},"last":0.79900000000000004}]},{"osd":1,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73199999999999998,"15min":0.71199999999999997},"min":{"1min":0.501,"5min":0.377,"15min":0.377},"max":{"1min":0.93500000000000005,"5min":0.95199999999999996,"15min":1.0169999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.71399999999999997,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.6040000000000001},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.747,"15min":0.72499999999999998},"min":{"1min":0.59999999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.476},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72299999999999998,"15min":0.70699999999999996},"min":{"1min":0.55300000000000005,"5min":0.441,"15min":0.441},"max":{"1min":1.095,"5min":1.095,"15min":1.502},"last":0.39700000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.745,"15min":0.72899999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.41799999999999998},"max":{"1min":1.113,"5min":1.113,"15min":1.1619999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76700000000000002,"15min":0.751},"min":{"1min":0.57599999999999996,"5min":0.52500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95399999999999996,"5min":0.98399999999999999,"15min":1.744},"last":0.77800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61799999999999999,"15min":0.59999999999999998},"min":{"1min":0.46400000000000002,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.84699999999999998,"5min":0.91400000000000003,"15min":0.95899999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63900000000000001,"15min":0.65000000000000002},"min":{"1min":0.48199999999999998,"5min":0.433,"15min":0.35299999999999998},"max":{"1min":0.83899999999999997,"5min":0.99099999999999999,"15min":1.214},"last":0.48399999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.46999999999999997,"5min":0.435,"15min":0.36799999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":1.151},"last":0.501},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68799999999999994,"15min":0.69399999999999995},"min":{"1min":0.53200000000000003,"5min":0.40899999999999997,"15min":0.40300000000000002},"max":{"1min":0.85399999999999998,"5min":0.94599999999999995,"15min":1.5489999999999999},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71099999999999997,"15min":0.72199999999999998},"min":{"1min":0.51400000000000001,"5min":0.44500000000000001,"15min":0.40200000000000002},"max":{"1min":0.92200000000000004,"5min":1.0249999999999999,"15min":1.325},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72999999999999998,"15min":0.73799999999999999},"min":{"1min":0.54300000000000004,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.1990000000000001},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:07:13.638 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:07:13.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.873+0000 7eff0a400700 1 Processor -- start 2024-03-29T10:07:13.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.873+0000 7eff0a400700 1 -- start start 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.874+0000 7eff0a400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff0409aba0 0x7eff0406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.874+0000 7eff0a400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7eff0406c480 0x7eff040ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.874+0000 7eff0a400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0406c8d0 con 0x7eff0415d710 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.874+0000 7eff0a400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0406bb40 con 0x7eff0409aba0 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.874+0000 7eff0a400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff04099b00 con 0x7eff0406c480 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.875+0000 7eff02a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff0409aba0 0x7eff0406bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.875+0000 7eff02a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff0409aba0 0x7eff0406bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47484/0 (socket says 172.21.15.12:47484) 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.875+0000 7eff02a00700 1 -- 172.21.15.12:0/374755048 learned_addr learned my addr 172.21.15.12:0/374755048 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.875+0000 7eff03e00700 1 --2- 172.21.15.12:0/374755048 >> v2:172.21.15.159:3300/0 conn(0x7eff0406c480 0x7eff040ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.875+0000 7eff02a00700 1 -- 172.21.15.12:0/374755048 >> v2:172.21.15.159:3300/0 conn(0x7eff0406c480 msgr2=0x7eff040ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:13.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.875+0000 7eff02a00700 1 --2- 172.21.15.12:0/374755048 >> v2:172.21.15.159:3300/0 conn(0x7eff0406c480 0x7eff040ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.875+0000 7eff02a00700 1 -- 172.21.15.12:0/374755048 >> v1:172.21.15.159:6789/0 conn(0x7eff0415d710 legacy=0x7eff040d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.875+0000 7eff02a00700 1 -- 172.21.15.12:0/374755048 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff04099de0 con 0x7eff0409aba0 2024-03-29T10:07:13.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.875+0000 7eff02a00700 1 --2- 172.21.15.12:0/374755048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 0x7eff0406bf40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7eff040722c0 tx=0x7efef0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5064ed8ae2bc243e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff02000700 1 -- 172.21.15.12:0/374755048 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef0011830 con 0x7eff0409aba0 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff02000700 1 -- 172.21.15.12:0/374755048 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef0011990 con 0x7eff0409aba0 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff02000700 1 -- 172.21.15.12:0/374755048 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef0011d20 con 0x7eff0409aba0 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff0a400700 1 -- 172.21.15.12:0/374755048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 msgr2=0x7eff0406bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff0a400700 1 --2- 172.21.15.12:0/374755048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 0x7eff0406bf40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7eff040722c0 tx=0x7efef0009bd0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff0a400700 1 -- 172.21.15.12:0/374755048 shutdown_connections 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff0a400700 1 --2- 172.21.15.12:0/374755048 >> v2:172.21.15.159:3300/0 conn(0x7eff0406c480 0x7eff040ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff0a400700 1 --2- 172.21.15.12:0/374755048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 0x7eff0406bf40 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff0a400700 1 -- 172.21.15.12:0/374755048 >> 172.21.15.12:0/374755048 conn(0x7eff040d13e0 msgr2=0x7eff040d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff0a400700 1 -- 172.21.15.12:0/374755048 shutdown_connections 2024-03-29T10:07:13.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.876+0000 7eff0a400700 1 -- 172.21.15.12:0/374755048 wait complete. 2024-03-29T10:07:13.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.878+0000 7eff0a400700 1 Processor -- start 2024-03-29T10:07:13.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.878+0000 7eff0a400700 1 -- start start 2024-03-29T10:07:13.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.879+0000 7eff0a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff0406c480 0x7eff0415a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.879+0000 7eff0a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 0x7eff0406f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.879+0000 7eff0a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff0415d710 0x7eff0406fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.879+0000 7eff0a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0409e110 con 0x7eff0409aba0 2024-03-29T10:07:13.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.879+0000 7eff0a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0409ec30 con 0x7eff0406c480 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.879+0000 7eff0a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0409e340 con 0x7eff0415d710 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.879+0000 7eff02a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 0x7eff0406f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff02a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 0x7eff0406f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47492/0 (socket says 172.21.15.12:47492) 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff02a00700 1 -- 172.21.15.12:0/2646133478 learned_addr learned my addr 172.21.15.12:0/2646133478 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff03400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff0406c480 0x7eff0415a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff03e00700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff0415d710 0x7eff0406fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff02a00700 1 -- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff0415d710 msgr2=0x7eff0406fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff02a00700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff0415d710 0x7eff0406fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff02a00700 1 -- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff0406c480 msgr2=0x7eff0415a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff02a00700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff0406c480 0x7eff0415a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff02a00700 1 -- 172.21.15.12:0/2646133478 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef0009550 con 0x7eff0409aba0 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff03400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff0406c480 0x7eff0415a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff02a00700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 0x7eff0406f5d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efef0000c00 tx=0x7efef0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7efefb400700 1 -- 172.21.15.12:0/2646133478 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef0004570 con 0x7eff0409aba0 2024-03-29T10:07:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7efefb400700 1 -- 172.21.15.12:0/2646133478 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef001f070 con 0x7eff0409aba0 2024-03-29T10:07:13.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.880+0000 7eff0a400700 1 -- 172.21.15.12:0/2646133478 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0409e5e0 con 0x7eff0409aba0 2024-03-29T10:07:13.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.881+0000 7efefb400700 1 -- 172.21.15.12:0/2646133478 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef00038c0 con 0x7eff0409aba0 2024-03-29T10:07:13.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.881+0000 7eff0a400700 1 -- 172.21.15.12:0/2646133478 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0409cc00 con 0x7eff0409aba0 2024-03-29T10:07:13.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.882+0000 7eff0a400700 1 -- 172.21.15.12:0/2646133478 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0409eda0 con 0x7eff0409aba0 2024-03-29T10:07:13.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.882+0000 7efefb400700 1 -- 172.21.15.12:0/2646133478 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efef0021030 con 0x7eff0409aba0 2024-03-29T10:07:13.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.883+0000 7efefb400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efee406adb0 0x7efee406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:13.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.883+0000 7efefb400700 1 -- 172.21.15.12:0/2646133478 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efef008aaa0 con 0x7eff0409aba0 2024-03-29T10:07:13.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.884+0000 7eff03400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efee406adb0 0x7efee406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:13.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.885+0000 7eff03400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efee406adb0 0x7efee406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7eff040b2d20 tx=0x7efef4009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:13.888+0000 7efefb400700 1 -- 172.21.15.12:0/2646133478 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efef0055c80 con 0x7eff0409aba0 2024-03-29T10:07:14.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.099+0000 7eff0a400700 1 -- 172.21.15.12:0/2646133478 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7eff0409a680 con 0x7efee406adb0 2024-03-29T10:07:14.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.108+0000 7efefb400700 1 -- 172.21.15.12:0/2646133478 <== mgr.5210 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289005 (secure 0 0 0) 0x7efef40050d0 con 0x7efee406adb0 2024-03-29T10:07:14.111 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 -- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efee406adb0 msgr2=0x7efee406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efee406adb0 0x7efee406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7eff040b2d20 tx=0x7efef4009040 comp rx=0 tx=0).stop 2024-03-29T10:07:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 -- 172.21.15.12:0/2646133478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 msgr2=0x7eff0406f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 0x7eff0406f5d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efef0000c00 tx=0x7efef0004750 comp rx=0 tx=0).stop 2024-03-29T10:07:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 -- 172.21.15.12:0/2646133478 shutdown_connections 2024-03-29T10:07:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efee406adb0 0x7efee406d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff0415d710 0x7eff0406fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff0406c480 0x7eff0415a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 --2- 172.21.15.12:0/2646133478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff0409aba0 0x7eff0406f5d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 -- 172.21.15.12:0/2646133478 >> 172.21.15.12:0/2646133478 conn(0x7eff040d13e0 msgr2=0x7eff040ae160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:14.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 -- 172.21.15.12:0/2646133478 shutdown_connections 2024-03-29T10:07:14.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.113+0000 7eff0a400700 1 -- 172.21.15.12:0/2646133478 wait complete. 2024-03-29T10:07:14.115 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:07:14.130 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-29T10:07:13.858039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":300052,"kb_used_data":6788,"kb_used_omap":12,"kb_used_meta":293107,"kb_avail":754674668,"statfs":{"total":773094113280,"available":772786860032,"internally_reserved":0,"allocated":6950912,"data_stored":3830073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":300142137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001469"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822629+0000","last_change":"2024-03-29T10:07:08.734768+0000","last_active":"2024-03-29T10:07:09.822629+0000","last_peered":"2024-03-29T10:07:09.822629+0000","last_clean":"2024-03-29T10:07:09.822629+0000","last_became_active":"2024-03-29T10:07:08.734123+0000","last_became_peered":"2024-03-29T10:07:08.734123+0000","last_unstale":"2024-03-29T10:07:09.822629+0000","last_undegraded":"2024-03-29T10:07:09.822629+0000","last_fullsized":"2024-03-29T10:07:09.822629+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:41.760118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717945+0000","last_change":"2024-03-29T10:07:08.725603+0000","last_active":"2024-03-29T10:07:09.717945+0000","last_peered":"2024-03-29T10:07:09.717945+0000","last_clean":"2024-03-29T10:07:09.717945+0000","last_became_active":"2024-03-29T10:07:08.724365+0000","last_became_peered":"2024-03-29T10:07:08.724365+0000","last_unstale":"2024-03-29T10:07:09.717945+0000","last_undegraded":"2024-03-29T10:07:09.717945+0000","last_fullsized":"2024-03-29T10:07:09.717945+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:05.814187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823120+0000","last_change":"2024-03-29T10:07:08.729857+0000","last_active":"2024-03-29T10:07:09.823120+0000","last_peered":"2024-03-29T10:07:09.823120+0000","last_clean":"2024-03-29T10:07:09.823120+0000","last_became_active":"2024-03-29T10:07:08.729539+0000","last_became_peered":"2024-03-29T10:07:08.729539+0000","last_unstale":"2024-03-29T10:07:09.823120+0000","last_undegraded":"2024-03-29T10:07:09.823120+0000","last_fullsized":"2024-03-29T10:07:09.823120+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:41.082469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716672+0000","last_change":"2024-03-29T10:07:08.727599+0000","last_active":"2024-03-29T10:07:09.716672+0000","last_peered":"2024-03-29T10:07:09.716672+0000","last_clean":"2024-03-29T10:07:09.716672+0000","last_became_active":"2024-03-29T10:07:08.726751+0000","last_became_peered":"2024-03-29T10:07:08.726751+0000","last_unstale":"2024-03-29T10:07:09.716672+0000","last_undegraded":"2024-03-29T10:07:09.716672+0000","last_fullsized":"2024-03-29T10:07:09.716672+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:27:01.767573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717705+0000","last_change":"2024-03-29T10:07:08.724700+0000","last_active":"2024-03-29T10:07:09.717705+0000","last_peered":"2024-03-29T10:07:09.717705+0000","last_clean":"2024-03-29T10:07:09.717705+0000","last_became_active":"2024-03-29T10:07:08.722834+0000","last_became_peered":"2024-03-29T10:07:08.722834+0000","last_unstale":"2024-03-29T10:07:09.717705+0000","last_undegraded":"2024-03-29T10:07:09.717705+0000","last_fullsized":"2024-03-29T10:07:09.717705+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:27:01.767573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822442+0000","last_change":"2024-03-29T10:07:08.734816+0000","last_active":"2024-03-29T10:07:09.822442+0000","last_peered":"2024-03-29T10:07:09.822442+0000","last_clean":"2024-03-29T10:07:09.822442+0000","last_became_active":"2024-03-29T10:07:08.734505+0000","last_became_peered":"2024-03-29T10:07:08.734505+0000","last_unstale":"2024-03-29T10:07:09.822442+0000","last_undegraded":"2024-03-29T10:07:09.822442+0000","last_fullsized":"2024-03-29T10:07:09.822442+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:29.277567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822176+0000","last_change":"2024-03-29T10:07:08.738355+0000","last_active":"2024-03-29T10:07:09.822176+0000","last_peered":"2024-03-29T10:07:09.822176+0000","last_clean":"2024-03-29T10:07:09.822176+0000","last_became_active":"2024-03-29T10:07:08.737863+0000","last_became_peered":"2024-03-29T10:07:08.737863+0000","last_unstale":"2024-03-29T10:07:09.822176+0000","last_undegraded":"2024-03-29T10:07:09.822176+0000","last_fullsized":"2024-03-29T10:07:09.822176+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:04:27.581246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.715987+0000","last_change":"2024-03-29T10:07:08.728815+0000","last_active":"2024-03-29T10:07:09.715987+0000","last_peered":"2024-03-29T10:07:09.715987+0000","last_clean":"2024-03-29T10:07:09.715987+0000","last_became_active":"2024-03-29T10:07:08.728550+0000","last_became_peered":"2024-03-29T10:07:08.728550+0000","last_unstale":"2024-03-29T10:07:09.715987+0000","last_undegraded":"2024-03-29T10:07:09.715987+0000","last_fullsized":"2024-03-29T10:07:09.715987+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:41.082469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716882+0000","last_change":"2024-03-29T10:07:08.739595+0000","last_active":"2024-03-29T10:07:09.716882+0000","last_peered":"2024-03-29T10:07:09.716882+0000","last_clean":"2024-03-29T10:07:09.716882+0000","last_became_active":"2024-03-29T10:07:08.739313+0000","last_became_peered":"2024-03-29T10:07:08.739313+0000","last_unstale":"2024-03-29T10:07:09.716882+0000","last_undegraded":"2024-03-29T10:07:09.716882+0000","last_fullsized":"2024-03-29T10:07:09.716882+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:49:52.665440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716870+0000","last_change":"2024-03-29T10:07:08.726816+0000","last_active":"2024-03-29T10:07:09.716870+0000","last_peered":"2024-03-29T10:07:09.716870+0000","last_clean":"2024-03-29T10:07:09.716870+0000","last_became_active":"2024-03-29T10:07:08.726456+0000","last_became_peered":"2024-03-29T10:07:08.726456+0000","last_unstale":"2024-03-29T10:07:09.716870+0000","last_undegraded":"2024-03-29T10:07:09.716870+0000","last_fullsized":"2024-03-29T10:07:09.716870+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:24.701904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716747+0000","last_change":"2024-03-29T10:07:08.738365+0000","last_active":"2024-03-29T10:07:09.716747+0000","last_peered":"2024-03-29T10:07:09.716747+0000","last_clean":"2024-03-29T10:07:09.716747+0000","last_became_active":"2024-03-29T10:07:08.738101+0000","last_became_peered":"2024-03-29T10:07:08.738101+0000","last_unstale":"2024-03-29T10:07:09.716747+0000","last_undegraded":"2024-03-29T10:07:09.716747+0000","last_fullsized":"2024-03-29T10:07:09.716747+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:22:33.174657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717832+0000","last_change":"2024-03-29T10:07:08.728958+0000","last_active":"2024-03-29T10:07:09.717832+0000","last_peered":"2024-03-29T10:07:09.717832+0000","last_clean":"2024-03-29T10:07:09.717832+0000","last_became_active":"2024-03-29T10:07:08.728384+0000","last_became_peered":"2024-03-29T10:07:08.728384+0000","last_unstale":"2024-03-29T10:07:09.717832+0000","last_undegraded":"2024-03-29T10:07:09.717832+0000","last_fullsized":"2024-03-29T10:07:09.717832+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:24.701904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823166+0000","last_change":"2024-03-29T10:07:08.743187+0000","last_active":"2024-03-29T10:07:09.823166+0000","last_peered":"2024-03-29T10:07:09.823166+0000","last_clean":"2024-03-29T10:07:09.823166+0000","last_became_active":"2024-03-29T10:07:08.742950+0000","last_became_peered":"2024-03-29T10:07:08.742950+0000","last_unstale":"2024-03-29T10:07:09.823166+0000","last_undegraded":"2024-03-29T10:07:09.823166+0000","last_fullsized":"2024-03-29T10:07:09.823166+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:26.899688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716462+0000","last_change":"2024-03-29T10:07:08.736913+0000","last_active":"2024-03-29T10:07:09.716462+0000","last_peered":"2024-03-29T10:07:09.716462+0000","last_clean":"2024-03-29T10:07:09.716462+0000","last_became_active":"2024-03-29T10:07:08.736686+0000","last_became_peered":"2024-03-29T10:07:08.736686+0000","last_unstale":"2024-03-29T10:07:09.716462+0000","last_undegraded":"2024-03-29T10:07:09.716462+0000","last_fullsized":"2024-03-29T10:07:09.716462+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:55.283504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822426+0000","last_change":"2024-03-29T10:07:08.736055+0000","last_active":"2024-03-29T10:07:09.822426+0000","last_peered":"2024-03-29T10:07:09.822426+0000","last_clean":"2024-03-29T10:07:09.822426+0000","last_became_active":"2024-03-29T10:07:08.735466+0000","last_became_peered":"2024-03-29T10:07:08.735466+0000","last_unstale":"2024-03-29T10:07:09.822426+0000","last_undegraded":"2024-03-29T10:07:09.822426+0000","last_fullsized":"2024-03-29T10:07:09.822426+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:49.393397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718321+0000","last_change":"2024-03-29T10:07:08.728724+0000","last_active":"2024-03-29T10:07:09.718321+0000","last_peered":"2024-03-29T10:07:09.718321+0000","last_clean":"2024-03-29T10:07:09.718321+0000","last_became_active":"2024-03-29T10:07:08.728206+0000","last_became_peered":"2024-03-29T10:07:08.728206+0000","last_unstale":"2024-03-29T10:07:09.718321+0000","last_undegraded":"2024-03-29T10:07:09.718321+0000","last_fullsized":"2024-03-29T10:07:09.718321+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:30:11.674318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823412+0000","last_change":"2024-03-29T10:07:08.725265+0000","last_active":"2024-03-29T10:07:09.823412+0000","last_peered":"2024-03-29T10:07:09.823412+0000","last_clean":"2024-03-29T10:07:09.823412+0000","last_became_active":"2024-03-29T10:07:08.724966+0000","last_became_peered":"2024-03-29T10:07:08.724966+0000","last_unstale":"2024-03-29T10:07:09.823412+0000","last_undegraded":"2024-03-29T10:07:09.823412+0000","last_fullsized":"2024-03-29T10:07:09.823412+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:44.154060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823137+0000","last_change":"2024-03-29T10:07:08.725417+0000","last_active":"2024-03-29T10:07:09.823137+0000","last_peered":"2024-03-29T10:07:09.823137+0000","last_clean":"2024-03-29T10:07:09.823137+0000","last_became_active":"2024-03-29T10:07:08.725144+0000","last_became_peered":"2024-03-29T10:07:08.725144+0000","last_unstale":"2024-03-29T10:07:09.823137+0000","last_undegraded":"2024-03-29T10:07:09.823137+0000","last_fullsized":"2024-03-29T10:07:09.823137+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:24.701904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718566+0000","last_change":"2024-03-29T10:07:08.742693+0000","last_active":"2024-03-29T10:07:09.718566+0000","last_peered":"2024-03-29T10:07:09.718566+0000","last_clean":"2024-03-29T10:07:09.718566+0000","last_became_active":"2024-03-29T10:07:08.742212+0000","last_became_peered":"2024-03-29T10:07:08.742212+0000","last_unstale":"2024-03-29T10:07:09.718566+0000","last_undegraded":"2024-03-29T10:07:09.718566+0000","last_fullsized":"2024-03-29T10:07:09.718566+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:17:30.334734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823037+0000","last_change":"2024-03-29T10:07:08.741339+0000","last_active":"2024-03-29T10:07:09.823037+0000","last_peered":"2024-03-29T10:07:09.823037+0000","last_clean":"2024-03-29T10:07:09.823037+0000","last_became_active":"2024-03-29T10:07:08.741073+0000","last_became_peered":"2024-03-29T10:07:08.741073+0000","last_unstale":"2024-03-29T10:07:09.823037+0000","last_undegraded":"2024-03-29T10:07:09.823037+0000","last_fullsized":"2024-03-29T10:07:09.823037+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:58.536135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822905+0000","last_change":"2024-03-29T10:07:08.742136+0000","last_active":"2024-03-29T10:07:09.822905+0000","last_peered":"2024-03-29T10:07:09.822905+0000","last_clean":"2024-03-29T10:07:09.822905+0000","last_became_active":"2024-03-29T10:07:08.741898+0000","last_became_peered":"2024-03-29T10:07:08.741898+0000","last_unstale":"2024-03-29T10:07:09.822905+0000","last_undegraded":"2024-03-29T10:07:09.822905+0000","last_fullsized":"2024-03-29T10:07:09.822905+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:35:45.448688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716551+0000","last_change":"2024-03-29T10:07:08.736431+0000","last_active":"2024-03-29T10:07:09.716551+0000","last_peered":"2024-03-29T10:07:09.716551+0000","last_clean":"2024-03-29T10:07:09.716551+0000","last_became_active":"2024-03-29T10:07:08.728993+0000","last_became_peered":"2024-03-29T10:07:08.728993+0000","last_unstale":"2024-03-29T10:07:09.716551+0000","last_undegraded":"2024-03-29T10:07:09.716551+0000","last_fullsized":"2024-03-29T10:07:09.716551+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:04.547033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717987+0000","last_change":"2024-03-29T10:07:08.734843+0000","last_active":"2024-03-29T10:07:09.717987+0000","last_peered":"2024-03-29T10:07:09.717987+0000","last_clean":"2024-03-29T10:07:09.717987+0000","last_became_active":"2024-03-29T10:07:08.734541+0000","last_became_peered":"2024-03-29T10:07:08.734541+0000","last_unstale":"2024-03-29T10:07:09.717987+0000","last_undegraded":"2024-03-29T10:07:09.717987+0000","last_fullsized":"2024-03-29T10:07:09.717987+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:52:16.286951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717672+0000","last_change":"2024-03-29T10:07:08.726256+0000","last_active":"2024-03-29T10:07:09.717672+0000","last_peered":"2024-03-29T10:07:09.717672+0000","last_clean":"2024-03-29T10:07:09.717672+0000","last_became_active":"2024-03-29T10:07:08.725908+0000","last_became_peered":"2024-03-29T10:07:08.725908+0000","last_unstale":"2024-03-29T10:07:09.717672+0000","last_undegraded":"2024-03-29T10:07:09.717672+0000","last_fullsized":"2024-03-29T10:07:09.717672+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:54.786449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718705+0000","last_change":"2024-03-29T10:07:08.732612+0000","last_active":"2024-03-29T10:07:09.718705+0000","last_peered":"2024-03-29T10:07:09.718705+0000","last_clean":"2024-03-29T10:07:09.718705+0000","last_became_active":"2024-03-29T10:07:08.730428+0000","last_became_peered":"2024-03-29T10:07:08.730428+0000","last_unstale":"2024-03-29T10:07:09.718705+0000","last_undegraded":"2024-03-29T10:07:09.718705+0000","last_fullsized":"2024-03-29T10:07:09.718705+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:46.657982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823711+0000","last_change":"2024-03-29T10:07:08.747522+0000","last_active":"2024-03-29T10:07:09.823711+0000","last_peered":"2024-03-29T10:07:09.823711+0000","last_clean":"2024-03-29T10:07:09.823711+0000","last_became_active":"2024-03-29T10:07:08.747234+0000","last_became_peered":"2024-03-29T10:07:08.747234+0000","last_unstale":"2024-03-29T10:07:09.823711+0000","last_undegraded":"2024-03-29T10:07:09.823711+0000","last_fullsized":"2024-03-29T10:07:09.823711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:52:16.286951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823288+0000","last_change":"2024-03-29T10:07:08.730041+0000","last_active":"2024-03-29T10:07:09.823288+0000","last_peered":"2024-03-29T10:07:09.823288+0000","last_clean":"2024-03-29T10:07:09.823288+0000","last_became_active":"2024-03-29T10:07:08.724864+0000","last_became_peered":"2024-03-29T10:07:08.724864+0000","last_unstale":"2024-03-29T10:07:09.823288+0000","last_undegraded":"2024-03-29T10:07:09.823288+0000","last_fullsized":"2024-03-29T10:07:09.823288+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:57.250306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717577+0000","last_change":"2024-03-29T10:07:08.729118+0000","last_active":"2024-03-29T10:07:09.717577+0000","last_peered":"2024-03-29T10:07:09.717577+0000","last_clean":"2024-03-29T10:07:09.717577+0000","last_became_active":"2024-03-29T10:07:08.728706+0000","last_became_peered":"2024-03-29T10:07:08.728706+0000","last_unstale":"2024-03-29T10:07:09.717577+0000","last_undegraded":"2024-03-29T10:07:09.717577+0000","last_fullsized":"2024-03-29T10:07:09.717577+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:41.082469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718395+0000","last_change":"2024-03-29T10:07:08.736372+0000","last_active":"2024-03-29T10:07:09.718395+0000","last_peered":"2024-03-29T10:07:09.718395+0000","last_clean":"2024-03-29T10:07:09.718395+0000","last_became_active":"2024-03-29T10:07:08.733791+0000","last_became_peered":"2024-03-29T10:07:08.733791+0000","last_unstale":"2024-03-29T10:07:09.718395+0000","last_undegraded":"2024-03-29T10:07:09.718395+0000","last_fullsized":"2024-03-29T10:07:09.718395+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:56:53.198672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823130+0000","last_change":"2024-03-29T10:07:08.735460+0000","last_active":"2024-03-29T10:07:09.823130+0000","last_peered":"2024-03-29T10:07:09.823130+0000","last_clean":"2024-03-29T10:07:09.823130+0000","last_became_active":"2024-03-29T10:07:08.735172+0000","last_became_peered":"2024-03-29T10:07:08.735172+0000","last_unstale":"2024-03-29T10:07:09.823130+0000","last_undegraded":"2024-03-29T10:07:09.823130+0000","last_fullsized":"2024-03-29T10:07:09.823130+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:04.547033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822970+0000","last_change":"2024-03-29T10:07:08.736284+0000","last_active":"2024-03-29T10:07:09.822970+0000","last_peered":"2024-03-29T10:07:09.822970+0000","last_clean":"2024-03-29T10:07:09.822970+0000","last_became_active":"2024-03-29T10:07:08.735769+0000","last_became_peered":"2024-03-29T10:07:08.735769+0000","last_unstale":"2024-03-29T10:07:09.822970+0000","last_undegraded":"2024-03-29T10:07:09.822970+0000","last_fullsized":"2024-03-29T10:07:09.822970+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:32.063403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822317+0000","last_change":"2024-03-29T10:07:08.728963+0000","last_active":"2024-03-29T10:07:09.822317+0000","last_peered":"2024-03-29T10:07:09.822317+0000","last_clean":"2024-03-29T10:07:09.822317+0000","last_became_active":"2024-03-29T10:07:08.728699+0000","last_became_peered":"2024-03-29T10:07:08.728699+0000","last_unstale":"2024-03-29T10:07:09.822317+0000","last_undegraded":"2024-03-29T10:07:09.822317+0000","last_fullsized":"2024-03-29T10:07:09.822317+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:56:53.198672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822792+0000","last_change":"2024-03-29T10:07:08.745181+0000","last_active":"2024-03-29T10:07:09.822792+0000","last_peered":"2024-03-29T10:07:09.822792+0000","last_clean":"2024-03-29T10:07:09.822792+0000","last_became_active":"2024-03-29T10:07:08.744863+0000","last_became_peered":"2024-03-29T10:07:08.744863+0000","last_unstale":"2024-03-29T10:07:09.822792+0000","last_undegraded":"2024-03-29T10:07:09.822792+0000","last_fullsized":"2024-03-29T10:07:09.822792+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:54.786449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"60'4","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822819+0000","last_change":"2024-03-29T10:07:08.205299+0000","last_active":"2024-03-29T10:07:09.822819+0000","last_peered":"2024-03-29T10:07:09.822819+0000","last_clean":"2024-03-29T10:07:09.822819+0000","last_became_active":"2024-03-29T10:07:08.200084+0000","last_became_peered":"2024-03-29T10:07:08.200084+0000","last_unstale":"2024-03-29T10:07:09.822819+0000","last_undegraded":"2024-03-29T10:07:09.822819+0000","last_fullsized":"2024-03-29T10:07:09.822819+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:27:00.758718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718582+0000","last_change":"2024-03-29T10:07:08.723099+0000","last_active":"2024-03-29T10:07:09.718582+0000","last_peered":"2024-03-29T10:07:09.718582+0000","last_clean":"2024-03-29T10:07:09.718582+0000","last_became_active":"2024-03-29T10:07:08.722735+0000","last_became_peered":"2024-03-29T10:07:08.722735+0000","last_unstale":"2024-03-29T10:07:09.718582+0000","last_undegraded":"2024-03-29T10:07:09.718582+0000","last_fullsized":"2024-03-29T10:07:09.718582+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:04.724057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823396+0000","last_change":"2024-03-29T10:07:08.199352+0000","last_active":"2024-03-29T10:07:09.823396+0000","last_peered":"2024-03-29T10:07:09.823396+0000","last_clean":"2024-03-29T10:07:09.823396+0000","last_became_active":"2024-03-29T10:07:08.199016+0000","last_became_peered":"2024-03-29T10:07:08.199016+0000","last_unstale":"2024-03-29T10:07:09.823396+0000","last_undegraded":"2024-03-29T10:07:09.823396+0000","last_fullsized":"2024-03-29T10:07:09.823396+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:54:33.541274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717679+0000","last_change":"2024-03-29T10:07:08.742877+0000","last_active":"2024-03-29T10:07:09.717679+0000","last_peered":"2024-03-29T10:07:09.717679+0000","last_clean":"2024-03-29T10:07:09.717679+0000","last_became_active":"2024-03-29T10:07:08.742497+0000","last_became_peered":"2024-03-29T10:07:08.742497+0000","last_unstale":"2024-03-29T10:07:09.717679+0000","last_undegraded":"2024-03-29T10:07:09.717679+0000","last_fullsized":"2024-03-29T10:07:09.717679+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:44.154060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823623+0000","last_change":"2024-03-29T10:07:08.203370+0000","last_active":"2024-03-29T10:07:09.823623+0000","last_peered":"2024-03-29T10:07:09.823623+0000","last_clean":"2024-03-29T10:07:09.823623+0000","last_became_active":"2024-03-29T10:07:08.203123+0000","last_became_peered":"2024-03-29T10:07:08.203123+0000","last_unstale":"2024-03-29T10:07:09.823623+0000","last_undegraded":"2024-03-29T10:07:09.823623+0000","last_fullsized":"2024-03-29T10:07:09.823623+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:30:10.665464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823407+0000","last_change":"2024-03-29T10:07:08.728395+0000","last_active":"2024-03-29T10:07:09.823407+0000","last_peered":"2024-03-29T10:07:09.823407+0000","last_clean":"2024-03-29T10:07:09.823407+0000","last_became_active":"2024-03-29T10:07:08.727415+0000","last_became_peered":"2024-03-29T10:07:08.727415+0000","last_unstale":"2024-03-29T10:07:09.823407+0000","last_undegraded":"2024-03-29T10:07:09.823407+0000","last_fullsized":"2024-03-29T10:07:09.823407+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:44.957025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822454+0000","last_change":"2024-03-29T10:07:08.205826+0000","last_active":"2024-03-29T10:07:09.822454+0000","last_peered":"2024-03-29T10:07:09.822454+0000","last_clean":"2024-03-29T10:07:09.822454+0000","last_became_active":"2024-03-29T10:07:08.205510+0000","last_became_peered":"2024-03-29T10:07:08.205510+0000","last_unstale":"2024-03-29T10:07:09.822454+0000","last_undegraded":"2024-03-29T10:07:09.822454+0000","last_fullsized":"2024-03-29T10:07:09.822454+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:27:00.758718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823291+0000","last_change":"2024-03-29T10:07:08.719507+0000","last_active":"2024-03-29T10:07:09.823291+0000","last_peered":"2024-03-29T10:07:09.823291+0000","last_clean":"2024-03-29T10:07:09.823291+0000","last_became_active":"2024-03-29T10:07:08.718752+0000","last_became_peered":"2024-03-29T10:07:08.718752+0000","last_unstale":"2024-03-29T10:07:09.823291+0000","last_undegraded":"2024-03-29T10:07:09.823291+0000","last_fullsized":"2024-03-29T10:07:09.823291+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:05.814187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822641+0000","last_change":"2024-03-29T10:07:08.205487+0000","last_active":"2024-03-29T10:07:09.822641+0000","last_peered":"2024-03-29T10:07:09.822641+0000","last_clean":"2024-03-29T10:07:09.822641+0000","last_became_active":"2024-03-29T10:07:08.201262+0000","last_became_peered":"2024-03-29T10:07:08.201262+0000","last_unstale":"2024-03-29T10:07:09.822641+0000","last_undegraded":"2024-03-29T10:07:09.822641+0000","last_fullsized":"2024-03-29T10:07:09.822641+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:58.258755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716382+0000","last_change":"2024-03-29T10:07:08.732681+0000","last_active":"2024-03-29T10:07:09.716382+0000","last_peered":"2024-03-29T10:07:09.716382+0000","last_clean":"2024-03-29T10:07:09.716382+0000","last_became_active":"2024-03-29T10:07:08.732422+0000","last_became_peered":"2024-03-29T10:07:08.732422+0000","last_unstale":"2024-03-29T10:07:09.716382+0000","last_undegraded":"2024-03-29T10:07:09.716382+0000","last_fullsized":"2024-03-29T10:07:09.716382+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:55:25.859062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823449+0000","last_change":"2024-03-29T10:07:08.192550+0000","last_active":"2024-03-29T10:07:09.823449+0000","last_peered":"2024-03-29T10:07:09.823449+0000","last_clean":"2024-03-29T10:07:09.823449+0000","last_became_active":"2024-03-29T10:07:08.192278+0000","last_became_peered":"2024-03-29T10:07:08.192278+0000","last_unstale":"2024-03-29T10:07:09.823449+0000","last_undegraded":"2024-03-29T10:07:09.823449+0000","last_fullsized":"2024-03-29T10:07:09.823449+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:43.948171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716198+0000","last_change":"2024-03-29T10:07:08.724451+0000","last_active":"2024-03-29T10:07:09.716198+0000","last_peered":"2024-03-29T10:07:09.716198+0000","last_clean":"2024-03-29T10:07:09.716198+0000","last_became_active":"2024-03-29T10:07:08.724175+0000","last_became_peered":"2024-03-29T10:07:08.724175+0000","last_unstale":"2024-03-29T10:07:09.716198+0000","last_undegraded":"2024-03-29T10:07:09.716198+0000","last_fullsized":"2024-03-29T10:07:09.716198+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:05.814187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717202+0000","last_change":"2024-03-29T10:07:08.197270+0000","last_active":"2024-03-29T10:07:09.717202+0000","last_peered":"2024-03-29T10:07:09.717202+0000","last_clean":"2024-03-29T10:07:09.717202+0000","last_became_active":"2024-03-29T10:07:08.196938+0000","last_became_peered":"2024-03-29T10:07:08.196938+0000","last_unstale":"2024-03-29T10:07:09.717202+0000","last_undegraded":"2024-03-29T10:07:09.717202+0000","last_fullsized":"2024-03-29T10:07:09.717202+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:53.777594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718104+0000","last_change":"2024-03-29T10:07:08.735588+0000","last_active":"2024-03-29T10:07:09.718104+0000","last_peered":"2024-03-29T10:07:09.718104+0000","last_clean":"2024-03-29T10:07:09.718104+0000","last_became_active":"2024-03-29T10:07:08.735285+0000","last_became_peered":"2024-03-29T10:07:08.735285+0000","last_unstale":"2024-03-29T10:07:09.718104+0000","last_undegraded":"2024-03-29T10:07:09.718104+0000","last_fullsized":"2024-03-29T10:07:09.718104+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:06:31.440735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716184+0000","last_change":"2024-03-29T10:07:08.206236+0000","last_active":"2024-03-29T10:07:09.716184+0000","last_peered":"2024-03-29T10:07:09.716184+0000","last_clean":"2024-03-29T10:07:09.716184+0000","last_became_active":"2024-03-29T10:07:08.205977+0000","last_became_peered":"2024-03-29T10:07:08.205977+0000","last_unstale":"2024-03-29T10:07:09.716184+0000","last_undegraded":"2024-03-29T10:07:09.716184+0000","last_fullsized":"2024-03-29T10:07:09.716184+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:39:59.847323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823281+0000","last_change":"2024-03-29T10:07:08.738498+0000","last_active":"2024-03-29T10:07:09.823281+0000","last_peered":"2024-03-29T10:07:09.823281+0000","last_clean":"2024-03-29T10:07:09.823281+0000","last_became_active":"2024-03-29T10:07:08.738191+0000","last_became_peered":"2024-03-29T10:07:08.738191+0000","last_unstale":"2024-03-29T10:07:09.823281+0000","last_undegraded":"2024-03-29T10:07:09.823281+0000","last_fullsized":"2024-03-29T10:07:09.823281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:38:26.106725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823517+0000","last_change":"2024-03-29T10:07:08.196381+0000","last_active":"2024-03-29T10:07:09.823517+0000","last_peered":"2024-03-29T10:07:09.823517+0000","last_clean":"2024-03-29T10:07:09.823517+0000","last_became_active":"2024-03-29T10:07:08.195195+0000","last_became_peered":"2024-03-29T10:07:08.195195+0000","last_unstale":"2024-03-29T10:07:09.823517+0000","last_undegraded":"2024-03-29T10:07:09.823517+0000","last_fullsized":"2024-03-29T10:07:09.823517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:39.653088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822699+0000","last_change":"2024-03-29T10:07:08.734816+0000","last_active":"2024-03-29T10:07:09.822699+0000","last_peered":"2024-03-29T10:07:09.822699+0000","last_clean":"2024-03-29T10:07:09.822699+0000","last_became_active":"2024-03-29T10:07:08.734548+0000","last_became_peered":"2024-03-29T10:07:08.734548+0000","last_unstale":"2024-03-29T10:07:09.822699+0000","last_undegraded":"2024-03-29T10:07:09.822699+0000","last_fullsized":"2024-03-29T10:07:09.822699+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:35:45.448688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716946+0000","last_change":"2024-03-29T10:07:08.207064+0000","last_active":"2024-03-29T10:07:09.716946+0000","last_peered":"2024-03-29T10:07:09.716946+0000","last_clean":"2024-03-29T10:07:09.716946+0000","last_became_active":"2024-03-29T10:07:08.206773+0000","last_became_peered":"2024-03-29T10:07:08.206773+0000","last_unstale":"2024-03-29T10:07:09.716946+0000","last_undegraded":"2024-03-29T10:07:09.716946+0000","last_fullsized":"2024-03-29T10:07:09.716946+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:19:00.560729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822588+0000","last_change":"2024-03-29T10:07:08.741377+0000","last_active":"2024-03-29T10:07:09.822588+0000","last_peered":"2024-03-29T10:07:09.822588+0000","last_clean":"2024-03-29T10:07:09.822588+0000","last_became_active":"2024-03-29T10:07:08.741053+0000","last_became_peered":"2024-03-29T10:07:08.741053+0000","last_unstale":"2024-03-29T10:07:09.822588+0000","last_undegraded":"2024-03-29T10:07:09.822588+0000","last_fullsized":"2024-03-29T10:07:09.822588+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:04.366498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"60'3","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822656+0000","last_change":"2024-03-29T10:07:08.206197+0000","last_active":"2024-03-29T10:07:09.822656+0000","last_peered":"2024-03-29T10:07:09.822656+0000","last_clean":"2024-03-29T10:07:09.822656+0000","last_became_active":"2024-03-29T10:07:08.205913+0000","last_became_peered":"2024-03-29T10:07:08.205913+0000","last_unstale":"2024-03-29T10:07:09.822656+0000","last_undegraded":"2024-03-29T10:07:09.822656+0000","last_fullsized":"2024-03-29T10:07:09.822656+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:03.538179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822528+0000","last_change":"2024-03-29T10:07:08.737195+0000","last_active":"2024-03-29T10:07:09.822528+0000","last_peered":"2024-03-29T10:07:09.822528+0000","last_clean":"2024-03-29T10:07:09.822528+0000","last_became_active":"2024-03-29T10:07:08.736915+0000","last_became_peered":"2024-03-29T10:07:08.736915+0000","last_unstale":"2024-03-29T10:07:09.822528+0000","last_undegraded":"2024-03-29T10:07:09.822528+0000","last_fullsized":"2024-03-29T10:07:09.822528+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:32:09.269929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"60'6","reported_seq":17,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823186+0000","last_change":"2024-03-29T10:07:08.199198+0000","last_active":"2024-03-29T10:07:09.823186+0000","last_peered":"2024-03-29T10:07:09.823186+0000","last_clean":"2024-03-29T10:07:09.823186+0000","last_became_active":"2024-03-29T10:07:08.198885+0000","last_became_peered":"2024-03-29T10:07:08.198885+0000","last_unstale":"2024-03-29T10:07:09.823186+0000","last_undegraded":"2024-03-29T10:07:09.823186+0000","last_fullsized":"2024-03-29T10:07:09.823186+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:28:42.506688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718207+0000","last_change":"2024-03-29T10:07:08.733310+0000","last_active":"2024-03-29T10:07:09.718207+0000","last_peered":"2024-03-29T10:07:09.718207+0000","last_clean":"2024-03-29T10:07:09.718207+0000","last_became_active":"2024-03-29T10:07:08.732961+0000","last_became_peered":"2024-03-29T10:07:08.732961+0000","last_unstale":"2024-03-29T10:07:09.718207+0000","last_undegraded":"2024-03-29T10:07:09.718207+0000","last_fullsized":"2024-03-29T10:07:09.718207+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:40.661943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822735+0000","last_change":"2024-03-29T10:07:08.207886+0000","last_active":"2024-03-29T10:07:09.822735+0000","last_peered":"2024-03-29T10:07:09.822735+0000","last_clean":"2024-03-29T10:07:09.822735+0000","last_became_active":"2024-03-29T10:07:08.200426+0000","last_became_peered":"2024-03-29T10:07:08.200426+0000","last_unstale":"2024-03-29T10:07:09.822735+0000","last_undegraded":"2024-03-29T10:07:09.822735+0000","last_fullsized":"2024-03-29T10:07:09.822735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:04.805332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716466+0000","last_change":"2024-03-29T10:07:08.731290+0000","last_active":"2024-03-29T10:07:09.716466+0000","last_peered":"2024-03-29T10:07:09.716466+0000","last_clean":"2024-03-29T10:07:09.716466+0000","last_became_active":"2024-03-29T10:07:08.731000+0000","last_became_peered":"2024-03-29T10:07:08.731000+0000","last_unstale":"2024-03-29T10:07:09.716466+0000","last_undegraded":"2024-03-29T10:07:09.716466+0000","last_fullsized":"2024-03-29T10:07:09.716466+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:52:16.286951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822311+0000","last_change":"2024-03-29T10:07:08.209619+0000","last_active":"2024-03-29T10:07:09.822311+0000","last_peered":"2024-03-29T10:07:09.822311+0000","last_clean":"2024-03-29T10:07:09.822311+0000","last_became_active":"2024-03-29T10:07:08.209349+0000","last_became_peered":"2024-03-29T10:07:08.209349+0000","last_unstale":"2024-03-29T10:07:09.822311+0000","last_undegraded":"2024-03-29T10:07:09.822311+0000","last_fullsized":"2024-03-29T10:07:09.822311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:40.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822464+0000","last_change":"2024-03-29T10:07:08.733800+0000","last_active":"2024-03-29T10:07:09.822464+0000","last_peered":"2024-03-29T10:07:09.822464+0000","last_clean":"2024-03-29T10:07:09.822464+0000","last_became_active":"2024-03-29T10:07:08.733545+0000","last_became_peered":"2024-03-29T10:07:08.733545+0000","last_unstale":"2024-03-29T10:07:09.822464+0000","last_undegraded":"2024-03-29T10:07:09.822464+0000","last_fullsized":"2024-03-29T10:07:09.822464+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:58:10.549612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822883+0000","last_change":"2024-03-29T10:07:08.208058+0000","last_active":"2024-03-29T10:07:09.822883+0000","last_peered":"2024-03-29T10:07:09.822883+0000","last_clean":"2024-03-29T10:07:09.822883+0000","last_became_active":"2024-03-29T10:07:08.206370+0000","last_became_peered":"2024-03-29T10:07:08.206370+0000","last_unstale":"2024-03-29T10:07:09.822883+0000","last_undegraded":"2024-03-29T10:07:09.822883+0000","last_fullsized":"2024-03-29T10:07:09.822883+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:03.538179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822301+0000","last_change":"2024-03-29T10:07:08.738620+0000","last_active":"2024-03-29T10:07:09.822301+0000","last_peered":"2024-03-29T10:07:09.822301+0000","last_clean":"2024-03-29T10:07:09.822301+0000","last_became_active":"2024-03-29T10:07:08.737378+0000","last_became_peered":"2024-03-29T10:07:08.737378+0000","last_unstale":"2024-03-29T10:07:09.822301+0000","last_undegraded":"2024-03-29T10:07:09.822301+0000","last_fullsized":"2024-03-29T10:07:09.822301+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:17:01.202851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"60'8","reported_seq":19,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716087+0000","last_change":"2024-03-29T10:07:08.201721+0000","last_active":"2024-03-29T10:07:09.716087+0000","last_peered":"2024-03-29T10:07:09.716087+0000","last_clean":"2024-03-29T10:07:09.716087+0000","last_became_active":"2024-03-29T10:07:08.201362+0000","last_became_peered":"2024-03-29T10:07:08.201362+0000","last_unstale":"2024-03-29T10:07:09.716087+0000","last_undegraded":"2024-03-29T10:07:09.716087+0000","last_fullsized":"2024-03-29T10:07:09.716087+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:57.273274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823759+0000","last_change":"2024-03-29T10:07:08.725051+0000","last_active":"2024-03-29T10:07:09.823759+0000","last_peered":"2024-03-29T10:07:09.823759+0000","last_clean":"2024-03-29T10:07:09.823759+0000","last_became_active":"2024-03-29T10:07:08.724617+0000","last_became_peered":"2024-03-29T10:07:08.724617+0000","last_unstale":"2024-03-29T10:07:09.823759+0000","last_undegraded":"2024-03-29T10:07:09.823759+0000","last_fullsized":"2024-03-29T10:07:09.823759+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:46.657982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823910+0000","last_change":"2024-03-29T10:07:08.204845+0000","last_active":"2024-03-29T10:07:09.823910+0000","last_peered":"2024-03-29T10:07:09.823910+0000","last_clean":"2024-03-29T10:07:09.823910+0000","last_became_active":"2024-03-29T10:07:08.204400+0000","last_became_peered":"2024-03-29T10:07:08.204400+0000","last_unstale":"2024-03-29T10:07:09.823910+0000","last_undegraded":"2024-03-29T10:07:09.823910+0000","last_fullsized":"2024-03-29T10:07:09.823910+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:30:10.665464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716947+0000","last_change":"2024-03-29T10:07:08.744649+0000","last_active":"2024-03-29T10:07:09.716947+0000","last_peered":"2024-03-29T10:07:09.716947+0000","last_clean":"2024-03-29T10:07:09.716947+0000","last_became_active":"2024-03-29T10:07:08.744048+0000","last_became_peered":"2024-03-29T10:07:08.744048+0000","last_unstale":"2024-03-29T10:07:09.716947+0000","last_undegraded":"2024-03-29T10:07:09.716947+0000","last_fullsized":"2024-03-29T10:07:09.716947+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:59.267610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823264+0000","last_change":"2024-03-29T10:07:08.200629+0000","last_active":"2024-03-29T10:07:09.823264+0000","last_peered":"2024-03-29T10:07:09.823264+0000","last_clean":"2024-03-29T10:07:09.823264+0000","last_became_active":"2024-03-29T10:07:08.199307+0000","last_became_peered":"2024-03-29T10:07:08.199307+0000","last_unstale":"2024-03-29T10:07:09.823264+0000","last_undegraded":"2024-03-29T10:07:09.823264+0000","last_fullsized":"2024-03-29T10:07:09.823264+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:17:29.325879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718528+0000","last_change":"2024-03-29T10:07:08.745407+0000","last_active":"2024-03-29T10:07:09.718528+0000","last_peered":"2024-03-29T10:07:09.718528+0000","last_clean":"2024-03-29T10:07:09.718528+0000","last_became_active":"2024-03-29T10:07:08.745057+0000","last_became_peered":"2024-03-29T10:07:08.745057+0000","last_unstale":"2024-03-29T10:07:09.718528+0000","last_undegraded":"2024-03-29T10:07:09.718528+0000","last_fullsized":"2024-03-29T10:07:09.718528+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:44.957025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717500+0000","last_change":"2024-03-29T10:07:08.197172+0000","last_active":"2024-03-29T10:07:09.717500+0000","last_peered":"2024-03-29T10:07:09.717500+0000","last_clean":"2024-03-29T10:07:09.717500+0000","last_became_active":"2024-03-29T10:07:08.196895+0000","last_became_peered":"2024-03-29T10:07:08.196895+0000","last_unstale":"2024-03-29T10:07:09.717500+0000","last_undegraded":"2024-03-29T10:07:09.717500+0000","last_fullsized":"2024-03-29T10:07:09.717500+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:53.898093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718660+0000","last_change":"2024-03-29T10:07:08.734869+0000","last_active":"2024-03-29T10:07:09.718660+0000","last_peered":"2024-03-29T10:07:09.718660+0000","last_clean":"2024-03-29T10:07:09.718660+0000","last_became_active":"2024-03-29T10:07:08.734370+0000","last_became_peered":"2024-03-29T10:07:08.734370+0000","last_unstale":"2024-03-29T10:07:09.718660+0000","last_undegraded":"2024-03-29T10:07:09.718660+0000","last_fullsized":"2024-03-29T10:07:09.718660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:54.906947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822816+0000","last_change":"2024-03-29T10:07:08.200837+0000","last_active":"2024-03-29T10:07:09.822816+0000","last_peered":"2024-03-29T10:07:09.822816+0000","last_clean":"2024-03-29T10:07:09.822816+0000","last_became_active":"2024-03-29T10:07:08.200087+0000","last_became_peered":"2024-03-29T10:07:08.200087+0000","last_unstale":"2024-03-29T10:07:09.822816+0000","last_undegraded":"2024-03-29T10:07:09.822816+0000","last_fullsized":"2024-03-29T10:07:09.822816+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:40.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822174+0000","last_change":"2024-03-29T10:07:08.733860+0000","last_active":"2024-03-29T10:07:09.822174+0000","last_peered":"2024-03-29T10:07:09.822174+0000","last_clean":"2024-03-29T10:07:09.822174+0000","last_became_active":"2024-03-29T10:07:08.733585+0000","last_became_peered":"2024-03-29T10:07:08.733585+0000","last_unstale":"2024-03-29T10:07:09.822174+0000","last_undegraded":"2024-03-29T10:07:09.822174+0000","last_fullsized":"2024-03-29T10:07:09.822174+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:09:57.508574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822822+0000","last_change":"2024-03-29T10:07:08.201523+0000","last_active":"2024-03-29T10:07:09.822822+0000","last_peered":"2024-03-29T10:07:09.822822+0000","last_clean":"2024-03-29T10:07:09.822822+0000","last_became_active":"2024-03-29T10:07:08.200965+0000","last_became_peered":"2024-03-29T10:07:08.200965+0000","last_unstale":"2024-03-29T10:07:09.822822+0000","last_undegraded":"2024-03-29T10:07:09.822822+0000","last_fullsized":"2024-03-29T10:07:09.822822+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:23.693049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822939+0000","last_change":"2024-03-29T10:07:08.735590+0000","last_active":"2024-03-29T10:07:09.822939+0000","last_peered":"2024-03-29T10:07:09.822939+0000","last_clean":"2024-03-29T10:07:09.822939+0000","last_became_active":"2024-03-29T10:07:08.735343+0000","last_became_peered":"2024-03-29T10:07:08.735343+0000","last_unstale":"2024-03-29T10:07:09.822939+0000","last_undegraded":"2024-03-29T10:07:09.822939+0000","last_fullsized":"2024-03-29T10:07:09.822939+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:54.906947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717318+0000","last_change":"2024-03-29T10:07:08.194559+0000","last_active":"2024-03-29T10:07:09.717318+0000","last_peered":"2024-03-29T10:07:09.717318+0000","last_clean":"2024-03-29T10:07:09.717318+0000","last_became_active":"2024-03-29T10:07:08.194019+0000","last_became_peered":"2024-03-29T10:07:08.194019+0000","last_unstale":"2024-03-29T10:07:09.717318+0000","last_undegraded":"2024-03-29T10:07:09.717318+0000","last_fullsized":"2024-03-29T10:07:09.717318+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:56:52.189817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822375+0000","last_change":"2024-03-29T10:07:08.734737+0000","last_active":"2024-03-29T10:07:09.822375+0000","last_peered":"2024-03-29T10:07:09.822375+0000","last_clean":"2024-03-29T10:07:09.822375+0000","last_became_active":"2024-03-29T10:07:08.734466+0000","last_became_peered":"2024-03-29T10:07:08.734466+0000","last_unstale":"2024-03-29T10:07:09.822375+0000","last_undegraded":"2024-03-29T10:07:09.822375+0000","last_fullsized":"2024-03-29T10:07:09.822375+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:06:31.440735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716226+0000","last_change":"2024-03-29T10:07:08.205793+0000","last_active":"2024-03-29T10:07:09.716226+0000","last_peered":"2024-03-29T10:07:09.716226+0000","last_clean":"2024-03-29T10:07:09.716226+0000","last_became_active":"2024-03-29T10:07:08.205554+0000","last_became_peered":"2024-03-29T10:07:08.205554+0000","last_unstale":"2024-03-29T10:07:09.716226+0000","last_undegraded":"2024-03-29T10:07:09.716226+0000","last_fullsized":"2024-03-29T10:07:09.716226+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:15:59.255088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716018+0000","last_change":"2024-03-29T10:07:08.732734+0000","last_active":"2024-03-29T10:07:09.716018+0000","last_peered":"2024-03-29T10:07:09.716018+0000","last_clean":"2024-03-29T10:07:09.716018+0000","last_became_active":"2024-03-29T10:07:08.732343+0000","last_became_peered":"2024-03-29T10:07:08.732343+0000","last_unstale":"2024-03-29T10:07:09.716018+0000","last_undegraded":"2024-03-29T10:07:09.716018+0000","last_fullsized":"2024-03-29T10:07:09.716018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:04.724057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822171+0000","last_change":"2024-03-29T10:07:08.203965+0000","last_active":"2024-03-29T10:07:09.822171+0000","last_peered":"2024-03-29T10:07:09.822171+0000","last_clean":"2024-03-29T10:07:09.822171+0000","last_became_active":"2024-03-29T10:07:08.203702+0000","last_became_peered":"2024-03-29T10:07:08.203702+0000","last_unstale":"2024-03-29T10:07:09.822171+0000","last_undegraded":"2024-03-29T10:07:09.822171+0000","last_fullsized":"2024-03-29T10:07:09.822171+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:23.693049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718644+0000","last_change":"2024-03-29T10:07:08.740198+0000","last_active":"2024-03-29T10:07:09.718644+0000","last_peered":"2024-03-29T10:07:09.718644+0000","last_clean":"2024-03-29T10:07:09.718644+0000","last_became_active":"2024-03-29T10:07:08.739927+0000","last_became_peered":"2024-03-29T10:07:08.739927+0000","last_unstale":"2024-03-29T10:07:09.718644+0000","last_undegraded":"2024-03-29T10:07:09.718644+0000","last_fullsized":"2024-03-29T10:07:09.718644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:54:34.550128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822485+0000","last_change":"2024-03-29T10:07:08.198415+0000","last_active":"2024-03-29T10:07:09.822485+0000","last_peered":"2024-03-29T10:07:09.822485+0000","last_clean":"2024-03-29T10:07:09.822485+0000","last_became_active":"2024-03-29T10:07:08.197508+0000","last_became_peered":"2024-03-29T10:07:08.197508+0000","last_unstale":"2024-03-29T10:07:09.822485+0000","last_undegraded":"2024-03-29T10:07:09.822485+0000","last_fullsized":"2024-03-29T10:07:09.822485+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:22:32.165803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716278+0000","last_change":"2024-03-29T10:07:08.721781+0000","last_active":"2024-03-29T10:07:09.716278+0000","last_peered":"2024-03-29T10:07:09.716278+0000","last_clean":"2024-03-29T10:07:09.716278+0000","last_became_active":"2024-03-29T10:07:08.721481+0000","last_became_peered":"2024-03-29T10:07:08.721481+0000","last_unstale":"2024-03-29T10:07:09.716278+0000","last_undegraded":"2024-03-29T10:07:09.716278+0000","last_fullsized":"2024-03-29T10:07:09.716278+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:06:31.440735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822677+0000","last_change":"2024-03-29T10:07:08.208437+0000","last_active":"2024-03-29T10:07:09.822677+0000","last_peered":"2024-03-29T10:07:09.822677+0000","last_clean":"2024-03-29T10:07:09.822677+0000","last_became_active":"2024-03-29T10:07:08.208183+0000","last_became_peered":"2024-03-29T10:07:08.208183+0000","last_unstale":"2024-03-29T10:07:09.822677+0000","last_undegraded":"2024-03-29T10:07:09.822677+0000","last_fullsized":"2024-03-29T10:07:09.822677+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:43.145205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822587+0000","last_change":"2024-03-29T10:07:08.738937+0000","last_active":"2024-03-29T10:07:09.822587+0000","last_peered":"2024-03-29T10:07:09.822587+0000","last_clean":"2024-03-29T10:07:09.822587+0000","last_became_active":"2024-03-29T10:07:08.738655+0000","last_became_peered":"2024-03-29T10:07:08.738655+0000","last_unstale":"2024-03-29T10:07:09.822587+0000","last_undegraded":"2024-03-29T10:07:09.822587+0000","last_fullsized":"2024-03-29T10:07:09.822587+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:48.896404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.824027+0000","last_change":"2024-03-29T10:07:08.199454+0000","last_active":"2024-03-29T10:07:09.824027+0000","last_peered":"2024-03-29T10:07:09.824027+0000","last_clean":"2024-03-29T10:07:09.824027+0000","last_became_active":"2024-03-29T10:07:08.198502+0000","last_became_peered":"2024-03-29T10:07:08.198502+0000","last_unstale":"2024-03-29T10:07:09.824027+0000","last_undegraded":"2024-03-29T10:07:09.824027+0000","last_fullsized":"2024-03-29T10:07:09.824027+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:28:42.506688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822280+0000","last_change":"2024-03-29T10:07:08.733276+0000","last_active":"2024-03-29T10:07:09.822280+0000","last_peered":"2024-03-29T10:07:09.822280+0000","last_clean":"2024-03-29T10:07:09.822280+0000","last_became_active":"2024-03-29T10:07:08.733018+0000","last_became_peered":"2024-03-29T10:07:08.733018+0000","last_unstale":"2024-03-29T10:07:09.822280+0000","last_undegraded":"2024-03-29T10:07:09.822280+0000","last_fullsized":"2024-03-29T10:07:09.822280+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:58:51.764293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822376+0000","last_change":"2024-03-29T10:07:08.207012+0000","last_active":"2024-03-29T10:07:09.822376+0000","last_peered":"2024-03-29T10:07:09.822376+0000","last_clean":"2024-03-29T10:07:09.822376+0000","last_became_active":"2024-03-29T10:07:08.206780+0000","last_became_peered":"2024-03-29T10:07:08.206780+0000","last_unstale":"2024-03-29T10:07:09.822376+0000","last_undegraded":"2024-03-29T10:07:09.822376+0000","last_fullsized":"2024-03-29T10:07:09.822376+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:04.805332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822550+0000","last_change":"2024-03-29T10:07:08.740768+0000","last_active":"2024-03-29T10:07:09.822550+0000","last_peered":"2024-03-29T10:07:09.822550+0000","last_clean":"2024-03-29T10:07:09.822550+0000","last_became_active":"2024-03-29T10:07:08.740462+0000","last_became_peered":"2024-03-29T10:07:08.740462+0000","last_unstale":"2024-03-29T10:07:09.822550+0000","last_undegraded":"2024-03-29T10:07:09.822550+0000","last_fullsized":"2024-03-29T10:07:09.822550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:59:28.495332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823411+0000","last_change":"2024-03-29T10:07:08.206790+0000","last_active":"2024-03-29T10:07:09.823411+0000","last_peered":"2024-03-29T10:07:09.823411+0000","last_clean":"2024-03-29T10:07:09.823411+0000","last_became_active":"2024-03-29T10:07:08.206510+0000","last_became_peered":"2024-03-29T10:07:08.206510+0000","last_unstale":"2024-03-29T10:07:09.823411+0000","last_undegraded":"2024-03-29T10:07:09.823411+0000","last_fullsized":"2024-03-29T10:07:09.823411+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:03.715203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.717429+0000","last_change":"2024-03-29T10:07:08.727909+0000","last_active":"2024-03-29T10:07:09.717429+0000","last_peered":"2024-03-29T10:07:09.717429+0000","last_clean":"2024-03-29T10:07:09.717429+0000","last_became_active":"2024-03-29T10:07:08.727031+0000","last_became_peered":"2024-03-29T10:07:08.727031+0000","last_unstale":"2024-03-29T10:07:09.717429+0000","last_undegraded":"2024-03-29T10:07:09.717429+0000","last_fullsized":"2024-03-29T10:07:09.717429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:04.547033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.822248+0000","last_change":"2024-03-29T10:07:08.202910+0000","last_active":"2024-03-29T10:07:09.822248+0000","last_peered":"2024-03-29T10:07:09.822248+0000","last_clean":"2024-03-29T10:07:09.822248+0000","last_became_active":"2024-03-29T10:07:08.202459+0000","last_became_peered":"2024-03-29T10:07:08.202459+0000","last_unstale":"2024-03-29T10:07:09.822248+0000","last_undegraded":"2024-03-29T10:07:09.822248+0000","last_fullsized":"2024-03-29T10:07:09.822248+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:57.273274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.823101+0000","last_change":"2024-03-29T10:07:08.745893+0000","last_active":"2024-03-29T10:07:09.823101+0000","last_peered":"2024-03-29T10:07:09.823101+0000","last_clean":"2024-03-29T10:07:09.823101+0000","last_became_active":"2024-03-29T10:07:08.745612+0000","last_became_peered":"2024-03-29T10:07:08.745612+0000","last_unstale":"2024-03-29T10:07:09.823101+0000","last_undegraded":"2024-03-29T10:07:09.823101+0000","last_fullsized":"2024-03-29T10:07:09.823101+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:27:01.767573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716085+0000","last_change":"2024-03-29T10:07:08.207099+0000","last_active":"2024-03-29T10:07:09.716085+0000","last_peered":"2024-03-29T10:07:09.716085+0000","last_clean":"2024-03-29T10:07:09.716085+0000","last_became_active":"2024-03-29T10:07:08.206706+0000","last_became_peered":"2024-03-29T10:07:08.206706+0000","last_unstale":"2024-03-29T10:07:09.716085+0000","last_undegraded":"2024-03-29T10:07:09.716085+0000","last_fullsized":"2024-03-29T10:07:09.716085+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:00.678829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.716975+0000","last_change":"2024-03-29T10:07:08.739624+0000","last_active":"2024-03-29T10:07:09.716975+0000","last_peered":"2024-03-29T10:07:09.716975+0000","last_clean":"2024-03-29T10:07:09.716975+0000","last_became_active":"2024-03-29T10:07:08.739063+0000","last_became_peered":"2024-03-29T10:07:08.739063+0000","last_unstale":"2024-03-29T10:07:09.716975+0000","last_undegraded":"2024-03-29T10:07:09.716975+0000","last_fullsized":"2024-03-29T10:07:09.716975+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:08.171753+0000","last_clean_scrub_stamp":"2024-03-29T10:07:08.171753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:46.657982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-29T10:07:09.718126+0000","last_change":"2024-03-29T10:07:08.201370+0000","last_active":"2024-03-29T10:07:09.718126+0000","last_peered":"2024-03-29T10:07:09.718126+0000","last_clean":"2024-03-29T10:07:09.718126+0000","last_became_active":"2024-03-29T10:07:08.201044+0000","last_became_peered":"2024-03-29T10:07:08.201044+0000","last_unstale":"2024-03-29T10:07:09.718126+0000","last_undegraded":"2024-03-29T10:07:09.718126+0000","last_fullsized":"2024-03-29T10:07:09.718126+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:07:07.162899+0000","last_clean_scrub_stamp":"2024-03-29T10:07:07.162899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:45.649127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"49'139","reported_seq":283,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-29T10:07:08.726454+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:07:08.726454+0000","last_peered":"2024-03-29T10:07:08.726454+0000","last_clean":"2024-03-29T10:07:08.726454+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:07:08.726454+0000","last_undegraded":"2024-03-29T10:07:08.726454+0000","last_fullsized":"2024-03-29T10:07:08.726454+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575003,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32400,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339440,"statfs":{"total":96636764160,"available":96603586560,"internally_reserved":0,"allocated":1310720,"data_stored":921162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.71899999999999997,"15min":0.70899999999999996},"min":{"1min":0.48299999999999998,"5min":0.44500000000000001,"15min":0.32500000000000001},"max":{"1min":0.97899999999999998,"5min":1.0029999999999999,"15min":1.0109999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70999999999999996,"15min":0.69699999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.046},"last":0.63800000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73199999999999998,"15min":0.71299999999999997},"min":{"1min":0.48899999999999999,"5min":0.48799999999999999,"15min":0.44500000000000001},"max":{"1min":0.91900000000000004,"5min":0.99099999999999999,"15min":1.1910000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.73599999999999999,"15min":0.72499999999999998},"min":{"1min":0.42199999999999999,"5min":0.41199999999999998,"15min":0.372},"max":{"1min":0.94799999999999995,"5min":0.96299999999999997,"15min":1},"last":0.61899999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.72599999999999998,"15min":0.73599999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":0.89900000000000002,"5min":0.97699999999999998,"15min":1.046},"last":0.75},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.76100000000000001,"15min":0.747},"min":{"1min":0.55200000000000005,"5min":0.51100000000000001,"15min":0.48999999999999999},"max":{"1min":0.93999999999999995,"5min":1.014,"15min":1.079},"last":0.79400000000000004}]},{"osd":3,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73699999999999999,"15min":0.73199999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.95499999999999996,"5min":1.0149999999999999,"15min":1.081},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78500000000000003,"15min":0.78100000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":0.98099999999999998,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.78300000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.60699999999999998},"min":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.29199999999999998},"max":{"1min":0.72299999999999998,"5min":0.87,"15min":1.111},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.621,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.746,"5min":0.88100000000000001,"15min":0.95599999999999996},"last":0.76900000000000002}]},{"osd":5,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.66700000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.83299999999999996,"5min":0.94599999999999995,"15min":1.0089999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66700000000000004,"15min":0.69599999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83299999999999996,"5min":0.92800000000000005,"15min":1.141},"last":0.89600000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.73799999999999999},"min":{"1min":0.57199999999999995,"5min":0.51200000000000001,"15min":0.49199999999999999},"max":{"1min":0.79800000000000004,"5min":0.93500000000000005,"15min":1.1679999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.435},"max":{"1min":0.81799999999999995,"5min":0.95599999999999996,"15min":1.121},"last":0.873}]}]},{"osd":6,"up_from":13,"seq":55834575004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":720896,"data_stored":331376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.70099999999999996,"15min":0.77800000000000002},"min":{"1min":0.52400000000000002,"5min":0.38,"15min":0.34699999999999998},"max":{"1min":0.84099999999999997,"5min":0.95599999999999996,"15min":7.4080000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.75,"15min":0.80300000000000005},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.42499999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":6.7610000000000001},"last":0.497}]},{"osd":1,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72699999999999998,"15min":0.71999999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.92100000000000004,"5min":0.97299999999999998,"15min":1.2549999999999999},"last":0.627},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72899999999999998,"15min":0.71299999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98099999999999998,"5min":0.995,"15min":1.0209999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.73199999999999998,"15min":0.72299999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96499999999999997,"5min":1.014,"15min":1.3169999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.74099999999999999,"15min":0.75600000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":0.97099999999999997,"5min":0.99199999999999999,"15min":1.415},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.75600000000000001,"15min":0.78300000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":1.02,"15min":3.1960000000000002},"last":0.747},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.77100000000000002,"15min":0.82199999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.014,"5min":1.0229999999999999,"15min":4.3760000000000003},"last":0.73199999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":0.69699999999999995,"5min":0.90900000000000003,"15min":1.109},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.69099999999999995,"15min":0.72099999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83399999999999996,"5min":0.95599999999999996,"15min":3.1219999999999999},"last":0.67800000000000005}]},{"osd":5,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65800000000000003,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.42899999999999999},"max":{"1min":0.874,"5min":0.88900000000000001,"15min":2.8279999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.48199999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.86799999999999999,"5min":0.96899999999999997,"15min":1},"last":0.77400000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.754,"15min":0.76800000000000002},"min":{"1min":0.56599999999999995,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.98699999999999999,"5min":1.048,"15min":1.2769999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.72599999999999998,"15min":0.746},"min":{"1min":0.56599999999999995,"5min":0.49399999999999999,"15min":0.40000000000000002},"max":{"1min":0.998,"5min":0.998,"15min":1.7869999999999999},"last":0.78600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49868,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321972,"statfs":{"total":96636764160,"available":96585699328,"internally_reserved":0,"allocated":716800,"data_stored":330774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.435,"15min":0.45400000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55600000000000005,"5min":0.72799999999999998,"15min":1.252},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52400000000000002,"15min":0.51400000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32700000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":1.2989999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52700000000000002,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.33700000000000002},"max":{"1min":0.60099999999999998,"5min":0.83399999999999996,"15min":4.282},"last":0.433}]},{"osd":3,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39000000000000001},"max":{"1min":0.745,"5min":0.78000000000000003,"15min":4.3200000000000003},"last":0.377},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59199999999999997,"15min":0.57699999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.373},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":1.006},"last":0.59399999999999997}]},{"osd":4,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0580000000000001,"5min":1.179,"15min":1.365},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.77300000000000002,"15min":0.82899999999999996},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.51000000000000001},"max":{"1min":0.79300000000000004,"5min":1.155,"15min":4.3559999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80900000000000005,"15min":0.82199999999999995},"min":{"1min":0.64200000000000002,"5min":0.55500000000000005,"15min":0.51800000000000002},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.9119999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.79800000000000004,"15min":0.82399999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.01,"5min":1.345,"15min":1.6100000000000001},"last":0.83899999999999997}]},{"osd":6,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.80400000000000005,"15min":0.85799999999999998},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.51900000000000002},"max":{"1min":0.86399999999999999,"5min":1.236,"15min":4.3899999999999997},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.81499999999999995,"15min":0.83599999999999997},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.55400000000000005},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.73999999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84799999999999998,"15min":0.85299999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.56200000000000006},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.82799999999999996,"15min":0.85099999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.55700000000000005},"max":{"1min":0.95799999999999996,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.66600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575004,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1306624,"data_stored":920087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":0.67400000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.33800000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.66100000000000003}]},{"osd":2,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57799999999999996,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.35499999999999998},"max":{"1min":0.80700000000000005,"5min":0.96899999999999997,"15min":1.0069999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.51800000000000002,"15min":0.497},"min":{"1min":0.33400000000000002,"5min":0.32200000000000001,"15min":0.28599999999999998},"max":{"1min":0.56000000000000005,"5min":0.83899999999999997,"15min":0.97199999999999998},"last":0.41699999999999998}]},{"osd":3,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57499999999999996,"15min":0.54500000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.34699999999999998},"max":{"1min":0.73599999999999999,"5min":1.077,"15min":1.077},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.63200000000000001,"15min":0.60499999999999998},"min":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.41699999999999998},"max":{"1min":0.78900000000000003,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.49299999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84099999999999997,"15min":0.80200000000000005},"min":{"1min":0.51600000000000001,"5min":0.46400000000000002,"15min":0.39200000000000002},"max":{"1min":1.083,"5min":1.214,"15min":1.226},"last":1.625},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.85299999999999998,"15min":0.82599999999999996},"min":{"1min":0.61399999999999999,"5min":0.50800000000000001,"15min":0.45800000000000002},"max":{"1min":0.97699999999999998,"5min":1.1419999999999999,"15min":1.167},"last":0.88400000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.79400000000000004,"15min":0.76500000000000001},"min":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.51600000000000001},"max":{"1min":0.99399999999999999,"5min":1.137,"15min":1.137},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.83099999999999996,"15min":0.80100000000000005},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.41099999999999998},"max":{"1min":0.93400000000000005,"5min":1.151,"15min":1.2390000000000001},"last":1.5309999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.83799999999999997,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.56299999999999994,"15min":0.53400000000000003},"max":{"1min":0.98199999999999998,"5min":1.1659999999999999,"15min":1.228},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.88200000000000001,"15min":0.84499999999999997},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.46899999999999997},"max":{"1min":1.0209999999999999,"5min":1.1950000000000001,"15min":1.2450000000000001},"last":1.601}]},{"osd":7,"last update":"Fri Mar 29 10:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.83799999999999997,"15min":0.80300000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.90600000000000003,"5min":1.0529999999999999,"15min":1.1890000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.87,"15min":0.84099999999999997},"min":{"1min":0.624,"5min":0.59199999999999997,"15min":0.49299999999999999},"max":{"1min":1,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":1.5669999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35932,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335908,"statfs":{"total":96636764160,"available":96599969792,"internally_reserved":0,"allocated":729088,"data_stored":332290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.59699999999999998,"5min":0.69599999999999995,"15min":0.86599999999999999},"last":0.373},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66200000000000003,"5min":0.78800000000000003,"15min":1.3140000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46400000000000002,"15min":0.46700000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.63500000000000001,"5min":0.751,"15min":0.94499999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56399999999999995,"15min":0.55400000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.32700000000000001},"max":{"1min":0.76600000000000001,"5min":1.032,"15min":1.032},"last":0.53300000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57399999999999995,"15min":0.57899999999999996},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.377},"max":{"1min":0.749,"5min":0.80000000000000004,"15min":1.127},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.54100000000000004,"15min":0.55100000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69999999999999996,"5min":0.82399999999999995,"15min":1.1479999999999999},"last":0.41899999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.78100000000000003,"15min":0.78800000000000003},"min":{"1min":0.44800000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.02,"5min":1.3520000000000001,"15min":1.554},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.81000000000000005,"15min":0.81599999999999995},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.51600000000000001},"max":{"1min":0.89400000000000002,"5min":1.236,"15min":1.577},"last":0.67000000000000004}]},{"osd":5,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.82499999999999996,"15min":0.81699999999999995},"min":{"1min":0.58299999999999996,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.034,"5min":1.2589999999999999,"15min":1.5209999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.81299999999999994,"15min":0.81599999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.51800000000000002},"max":{"1min":0.96899999999999997,"5min":1.2250000000000001,"15min":1.3300000000000001},"last":0.65300000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81899999999999995,"15min":0.83999999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54100000000000004},"max":{"1min":0.98599999999999999,"5min":1.1539999999999999,"15min":1.758},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.57799999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.071,"5min":1.3560000000000001,"15min":1.4750000000000001},"last":0.76200000000000001}]},{"osd":7,"last update":"Fri Mar 29 10:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.879,"15min":0.871},"min":{"1min":0.628,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0549999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.627},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.88300000000000001,"15min":0.88700000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":0.996,"5min":1.4199999999999999,"15min":1.4390000000000001},"last":0.80300000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":724992,"data_stored":331342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.46200000000000002,"15min":0.47999999999999998},"min":{"1min":0.30299999999999999,"5min":0.26500000000000001,"15min":0.26000000000000001},"max":{"1min":0.57199999999999995,"5min":0.81299999999999994,"15min":4.3799999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52100000000000002,"15min":0.54600000000000004},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":4.6379999999999999},"last":0.47299999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.499,"15min":0.51900000000000002},"min":{"1min":0.36099999999999999,"5min":0.35799999999999998,"15min":0.27600000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":4.1840000000000002},"last":0.503},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58199999999999996,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.314},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.048},"last":0.56499999999999995}]},{"osd":2,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57199999999999995,"15min":0.55100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.33700000000000002},"max":{"1min":0.63100000000000001,"5min":0.82499999999999996,"15min":1.369},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.64500000000000002,"15min":0.66200000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":4.5970000000000004},"last":0.68200000000000005}]},{"osd":4,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.002,"5min":1.288,"15min":4.3319999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.79400000000000004,"15min":0.77300000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.39400000000000002},"max":{"1min":1.0640000000000001,"5min":1.46,"15min":1.46},"last":0.86599999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.78300000000000003,"15min":0.79300000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.435},"max":{"1min":0.873,"5min":1.3580000000000001,"15min":1.3740000000000001},"last":0.995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.82499999999999996,"15min":0.82299999999999995},"min":{"1min":0.5,"5min":0.5,"15min":0.44500000000000001},"max":{"1min":1.0629999999999999,"5min":1.429,"15min":4.4180000000000001},"last":0.88300000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.83399999999999996,"15min":0.81100000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.101,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.82799999999999996,"15min":0.81899999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.503},"max":{"1min":0.94199999999999995,"5min":1.3999999999999999,"15min":1.417},"last":0.96099999999999997}]},{"osd":7,"last update":"Fri Mar 29 10:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.85299999999999998,"15min":0.86099999999999999},"min":{"1min":0.57499999999999996,"5min":0.56799999999999995,"15min":0.54300000000000004},"max":{"1min":1.123,"5min":1.5029999999999999,"15min":4.4550000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.84899999999999998,"15min":0.84199999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.56200000000000006},"max":{"1min":1.0660000000000001,"5min":1.345,"15min":1.476},"last":0.84999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53976,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317864,"statfs":{"total":96636764160,"available":96581492736,"internally_reserved":0,"allocated":724992,"data_stored":332268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.69399999999999995,"15min":0.66900000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79600000000000004,"5min":1.417,"15min":1.417},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.71399999999999997,"15min":0.68799999999999994},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81599999999999995,"5min":1.165,"15min":1.165},"last":0.60399999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71899999999999997,"15min":0.69699999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.91300000000000003,"5min":0.98399999999999999,"15min":1.139},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.72399999999999998,"15min":0.70299999999999996},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.44800000000000001},"max":{"1min":0.82399999999999995,"5min":1.333,"15min":1.333},"last":0.62}]},{"osd":2,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78200000000000003,"15min":0.75800000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.41299999999999998},"max":{"1min":0.86799999999999999,"5min":1.03,"15min":1.863},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.76600000000000001,"15min":0.75600000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.84899999999999998,"5min":1.387,"15min":1.387},"last":0.70999999999999996}]},{"osd":3,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.73499999999999999,"15min":0.71499999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46999999999999997},"max":{"1min":0.93700000000000006,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.76000000000000001,"15min":0.73499999999999999},"min":{"1min":0.58599999999999997,"5min":0.52100000000000002,"15min":0.46600000000000003},"max":{"1min":0.876,"5min":1.3180000000000001,"15min":1.905},"last":0.76900000000000002}]},{"osd":5,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.57599999999999996,"15min":0.60299999999999998},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77200000000000002,"5min":0.85599999999999998,"15min":0.92000000000000004},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.63300000000000001},"min":{"1min":0.46000000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89000000000000001,"5min":1.294,"15min":1.294},"last":0.50900000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61699999999999999,"15min":0.64300000000000002},"min":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.41399999999999998},"max":{"1min":0.79800000000000004,"5min":1.0129999999999999,"15min":1.9339999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67000000000000004,"15min":0.68200000000000005},"min":{"1min":0.53000000000000003,"5min":0.45600000000000002,"15min":0.433},"max":{"1min":0.78400000000000003,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.74099999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.70499999999999996,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.442,"15min":0.41799999999999998},"max":{"1min":0.97799999999999998,"5min":1.345,"15min":1.345},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76900000000000002,"15min":0.77800000000000002},"min":{"1min":0.61699999999999999,"5min":0.51300000000000001,"15min":0.50700000000000001},"max":{"1min":0.90400000000000003,"5min":1.214,"15min":1.97},"last":0.79900000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":716800,"data_stored":330774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.71699999999999997,"15min":0.67400000000000004},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":0.871,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.70199999999999996},"min":{"1min":0.55300000000000005,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0529999999999999,"5min":1.2649999999999999,"15min":1.339},"last":0.79900000000000004}]},{"osd":1,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73199999999999998,"15min":0.71199999999999997},"min":{"1min":0.501,"5min":0.377,"15min":0.377},"max":{"1min":0.93500000000000005,"5min":0.95199999999999996,"15min":1.0169999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.71399999999999997,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.6040000000000001},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.747,"15min":0.72499999999999998},"min":{"1min":0.59999999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.476},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72299999999999998,"15min":0.70699999999999996},"min":{"1min":0.55300000000000005,"5min":0.441,"15min":0.441},"max":{"1min":1.095,"5min":1.095,"15min":1.502},"last":0.39700000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.745,"15min":0.72899999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.41799999999999998},"max":{"1min":1.113,"5min":1.113,"15min":1.1619999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76700000000000002,"15min":0.751},"min":{"1min":0.57599999999999996,"5min":0.52500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95399999999999996,"5min":0.98399999999999999,"15min":1.744},"last":0.77800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61799999999999999,"15min":0.59999999999999998},"min":{"1min":0.46400000000000002,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.84699999999999998,"5min":0.91400000000000003,"15min":0.95899999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63900000000000001,"15min":0.65000000000000002},"min":{"1min":0.48199999999999998,"5min":0.433,"15min":0.35299999999999998},"max":{"1min":0.83899999999999997,"5min":0.99099999999999999,"15min":1.214},"last":0.48399999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.46999999999999997,"5min":0.435,"15min":0.36799999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":1.151},"last":0.501},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68799999999999994,"15min":0.69399999999999995},"min":{"1min":0.53200000000000003,"5min":0.40899999999999997,"15min":0.40300000000000002},"max":{"1min":0.85399999999999998,"5min":0.94599999999999995,"15min":1.5489999999999999},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Mar 29 10:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71099999999999997,"15min":0.72199999999999998},"min":{"1min":0.51400000000000001,"5min":0.44500000000000001,"15min":0.40200000000000002},"max":{"1min":0.92200000000000004,"5min":1.0249999999999999,"15min":1.325},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72999999999999998,"15min":0.73799999999999999},"min":{"1min":0.54300000000000004,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.1990000000000001},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:07:14.135 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T10:07:14.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T10:07:14.159 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-29T10:07:14.160+0000 7f0c3c400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:07:14.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.366+0000 7f06d9c00700 1 Processor -- start 2024-03-29T10:07:14.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.366+0000 7f06d9c00700 1 -- start start 2024-03-29T10:07:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.367+0000 7f06d9c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 0x7f06d40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.367+0000 7f06d9c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d4161420 con 0x7f06d40d2370 2024-03-29T10:07:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.367+0000 7f06d9c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d4161700 con 0x7f06d415d9d0 2024-03-29T10:07:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.367+0000 7f06d9c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40d2820 con 0x7f06d40d16a0 2024-03-29T10:07:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.367+0000 7f06d3400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f06d40d2370 0x7f06d40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35914/0 (socket says 172.21.15.12:35914) 2024-03-29T10:07:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.367+0000 7f06d3400700 1 -- 172.21.15.12:0/1836505408 learned_addr learned my addr 172.21.15.12:0/1836505408 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.367+0000 7f06d2000700 1 --2- 172.21.15.12:0/1836505408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 0x7f06d40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:14.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.368+0000 7f06d1600700 1 -- 172.21.15.12:0/1836505408 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211745756 0 0) 0x7f06d4161420 con 0x7f06d40d2370 2024-03-29T10:07:14.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.368+0000 7f06d1600700 1 -- 172.21.15.12:0/1836505408 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b80035b0 con 0x7f06d40d2370 2024-03-29T10:07:14.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.368+0000 7f06d1600700 1 -- 172.21.15.12:0/1836505408 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453283946 0 0) 0x7f06d4161700 con 0x7f06d415d9d0 2024-03-29T10:07:14.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.368+0000 7f06d1600700 1 -- 172.21.15.12:0/1836505408 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d4161420 con 0x7f06d415d9d0 2024-03-29T10:07:14.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.368+0000 7f06d2000700 1 -- 172.21.15.12:0/1836505408 >> v1:172.21.15.159:6789/0 conn(0x7f06d415d9d0 legacy=0x7f06d40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.368+0000 7f06d2000700 1 -- 172.21.15.12:0/1836505408 >> v1:172.21.15.12:6789/0 conn(0x7f06d40d2370 legacy=0x7f06d40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.368+0000 7f06d1600700 1 -- 172.21.15.12:0/1836505408 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3477690068 0 0) 0x7f06b80035b0 con 0x7f06d40d2370 2024-03-29T10:07:14.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.368+0000 7f06d2000700 1 -- 172.21.15.12:0/1836505408 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d41619e0 con 0x7f06d40d16a0 2024-03-29T10:07:14.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.368+0000 7f06d2000700 1 --2- 172.21.15.12:0/1836505408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 0x7f06d40d1e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009850 tx=0x7f06c0009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e307dee62759db7f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:14.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.369+0000 7f06d1600700 1 -- 172.21.15.12:0/1836505408 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0019f0 con 0x7f06d40d16a0 2024-03-29T10:07:14.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.369+0000 7f06d1600700 1 -- 172.21.15.12:0/1836505408 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c0005c30 con 0x7f06d40d16a0 2024-03-29T10:07:14.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.369+0000 7f06d1600700 1 -- 172.21.15.12:0/1836505408 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0004030 con 0x7f06d40d16a0 2024-03-29T10:07:14.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.369+0000 7f06d9c00700 1 -- 172.21.15.12:0/1836505408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 msgr2=0x7f06d40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.369+0000 7f06d9c00700 1 --2- 172.21.15.12:0/1836505408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 0x7f06d40d1e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009850 tx=0x7f06c0009b60 comp rx=0 tx=0).stop 2024-03-29T10:07:14.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.369+0000 7f06d9c00700 1 -- 172.21.15.12:0/1836505408 shutdown_connections 2024-03-29T10:07:14.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.369+0000 7f06d9c00700 1 --2- 172.21.15.12:0/1836505408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 0x7f06d40d1e30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.369+0000 7f06d9c00700 1 -- 172.21.15.12:0/1836505408 >> 172.21.15.12:0/1836505408 conn(0x7f06d408aff0 msgr2=0x7f06d40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:14.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.369+0000 7f06d9c00700 1 -- 172.21.15.12:0/1836505408 shutdown_connections 2024-03-29T10:07:14.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.369+0000 7f06d9c00700 1 -- 172.21.15.12:0/1836505408 wait complete. 2024-03-29T10:07:14.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.371+0000 7f06d9c00700 1 Processor -- start 2024-03-29T10:07:14.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.371+0000 7f06d9c00700 1 -- start start 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d9c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 0x7f06d41569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d9c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06d40d2370 0x7f06d415fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d9d0 0x7f06d41602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d9c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d415eaa0 con 0x7f06d415d9d0 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d9c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d415ed80 con 0x7f06d40d2370 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d9c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d415f060 con 0x7f06d40d16a0 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d9d0 0x7f06d41602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d9d0 0x7f06d41602d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47506/0 (socket says 172.21.15.12:47506) 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d3400700 1 -- 172.21.15.12:0/2176582723 learned_addr learned my addr 172.21.15.12:0/2176582723 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d2a00700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 0x7f06d41569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.372+0000 7f06d2000700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06d40d2370 0x7f06d415fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.373+0000 7f06d3400700 1 -- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 msgr2=0x7f06d41569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.373+0000 7f06d3400700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 0x7f06d41569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.373+0000 7f06d3400700 1 -- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06d40d2370 msgr2=0x7f06d415fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.373+0000 7f06d3400700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06d40d2370 0x7f06d415fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.373+0000 7f06d3400700 1 -- 172.21.15.12:0/2176582723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c0009430 con 0x7f06d415d9d0 2024-03-29T10:07:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.373+0000 7f06d3400700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d9d0 0x7f06d41602d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0017f0 tx=0x7f06cc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.373+0000 7f06caa00700 1 -- 172.21.15.12:0/2176582723 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc00ab70 con 0x7f06d415d9d0 2024-03-29T10:07:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.373+0000 7f06caa00700 1 -- 172.21.15.12:0/2176582723 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc00acd0 con 0x7f06d415d9d0 2024-03-29T10:07:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.373+0000 7f06caa00700 1 -- 172.21.15.12:0/2176582723 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0072d0 con 0x7f06d415d9d0 2024-03-29T10:07:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.373+0000 7f06d9c00700 1 -- 172.21.15.12:0/2176582723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d415f340 con 0x7f06d415d9d0 2024-03-29T10:07:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.374+0000 7f06d9c00700 1 -- 172.21.15.12:0/2176582723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d415ddb0 con 0x7f06d415d9d0 2024-03-29T10:07:14.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.374+0000 7f06d9c00700 1 -- 172.21.15.12:0/2176582723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d40d1ca0 con 0x7f06d415d9d0 2024-03-29T10:07:14.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.375+0000 7f06caa00700 1 -- 172.21.15.12:0/2176582723 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f06cc007430 con 0x7f06d415d9d0 2024-03-29T10:07:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.376+0000 7f06caa00700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f06b406adb0 0x7f06b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.376+0000 7f06caa00700 1 -- 172.21.15.12:0/2176582723 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06cc088ff0 con 0x7f06d415d9d0 2024-03-29T10:07:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.377+0000 7f06d2a00700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f06b406adb0 0x7f06b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:14.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.378+0000 7f06d2a00700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f06b406adb0 0x7f06b406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06c40011e0 tx=0x7f06c4009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:14.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.381+0000 7f06caa00700 1 -- 172.21.15.12:0/2176582723 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06cc0bc580 con 0x7f06d415d9d0 2024-03-29T10:07:14.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.678+0000 7f06d9c00700 1 -- 172.21.15.12:0/2176582723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f06d4067720 con 0x7f06d415d9d0 2024-03-29T10:07:14.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.683+0000 7f06caa00700 1 -- 172.21.15.12:0/2176582723 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+0 (secure 0 0 0) 0x7f06cc0180e0 con 0x7f06d415d9d0 2024-03-29T10:07:14.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 -- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f06b406adb0 msgr2=0x7f06b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f06b406adb0 0x7f06b406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06c40011e0 tx=0x7f06c4009040 comp rx=0 tx=0).stop 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 -- 172.21.15.12:0/2176582723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d9d0 msgr2=0x7f06d41602d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d9d0 0x7f06d41602d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0017f0 tx=0x7f06cc00a060 comp rx=0 tx=0).stop 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 -- 172.21.15.12:0/2176582723 shutdown_connections 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f06b406adb0 0x7f06b406d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d9d0 0x7f06d41602d0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d40d16a0 0x7f06d41569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 --2- 172.21.15.12:0/2176582723 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06d40d2370 0x7f06d415fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 -- 172.21.15.12:0/2176582723 >> 172.21.15.12:0/2176582723 conn(0x7f06d408aff0 msgr2=0x7f06d40d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.685+0000 7f06d9c00700 1 -- 172.21.15.12:0/2176582723 shutdown_connections 2024-03-29T10:07:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.686+0000 7f06d9c00700 1 -- 172.21.15.12:0/2176582723 wait complete. 2024-03-29T10:07:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T10:07:14.704 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:07:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.902+0000 7f76fbe00700 1 Processor -- start 2024-03-29T10:07:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.902+0000 7f76fbe00700 1 -- start start 2024-03-29T10:07:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.903+0000 7f76fbe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76f40cf3c0 0x7f76f40d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.903+0000 7f76fbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f76f415db40 0x7f76f40cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.903+0000 7f76fbe00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f40d1330 con 0x7f76f415d760 2024-03-29T10:07:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.903+0000 7f76fbe00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f40d1fb0 con 0x7f76f415db40 2024-03-29T10:07:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.903+0000 7f76fbe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f40d1610 con 0x7f76f40cf3c0 2024-03-29T10:07:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.903+0000 7f76f9400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f76f415d760 0x7f76f40cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35926/0 (socket says 172.21.15.12:35926) 2024-03-29T10:07:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.903+0000 7f76f9400700 1 -- 172.21.15.12:0/1270975915 learned_addr learned my addr 172.21.15.12:0/1270975915 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.903+0000 7f76f9e00700 1 --2- 172.21.15.12:0/1270975915 >> v2:172.21.15.12:3300/0 conn(0x7f76f415db40 0x7f76f40cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.904+0000 7f76f9e00700 1 -- 172.21.15.12:0/1270975915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76f40cf3c0 msgr2=0x7f76f40d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.904+0000 7f76f9e00700 1 --2- 172.21.15.12:0/1270975915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76f40cf3c0 0x7f76f40d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.904+0000 7f76f9e00700 1 -- 172.21.15.12:0/1270975915 >> v1:172.21.15.12:6789/0 conn(0x7f76f415d760 legacy=0x7f76f40cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.904+0000 7f76f9e00700 1 -- 172.21.15.12:0/1270975915 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f40d18f0 con 0x7f76f415db40 2024-03-29T10:07:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.904+0000 7f76f9e00700 1 --2- 172.21.15.12:0/1270975915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f415db40 0x7f76f40cffd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009d90 tx=0x7f76ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2908fad3f4f3aaac server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.904+0000 7f76f3e00700 1 -- 172.21.15.12:0/1270975915 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec00e070 con 0x7f76f415db40 2024-03-29T10:07:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.904+0000 7f76f3e00700 1 -- 172.21.15.12:0/1270975915 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ec00bbd0 con 0x7f76f415db40 2024-03-29T10:07:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.904+0000 7f76f3e00700 1 -- 172.21.15.12:0/1270975915 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec012430 con 0x7f76f415db40 2024-03-29T10:07:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.905+0000 7f76fbe00700 1 -- 172.21.15.12:0/1270975915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f415db40 msgr2=0x7f76f40cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.905+0000 7f76fbe00700 1 --2- 172.21.15.12:0/1270975915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f415db40 0x7f76f40cffd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009d90 tx=0x7f76ec00b330 comp rx=0 tx=0).stop 2024-03-29T10:07:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.905+0000 7f76fbe00700 1 -- 172.21.15.12:0/1270975915 shutdown_connections 2024-03-29T10:07:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.905+0000 7f76fbe00700 1 --2- 172.21.15.12:0/1270975915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f415db40 0x7f76f40cffd0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.905+0000 7f76fbe00700 1 --2- 172.21.15.12:0/1270975915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76f40cf3c0 0x7f76f40d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.905+0000 7f76fbe00700 1 -- 172.21.15.12:0/1270975915 >> 172.21.15.12:0/1270975915 conn(0x7f76f408aff0 msgr2=0x7f76f40d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.905+0000 7f76fbe00700 1 -- 172.21.15.12:0/1270975915 shutdown_connections 2024-03-29T10:07:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.905+0000 7f76fbe00700 1 -- 172.21.15.12:0/1270975915 wait complete. 2024-03-29T10:07:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.907+0000 7f76fbe00700 1 Processor -- start 2024-03-29T10:07:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.908+0000 7f76fbe00700 1 -- start start 2024-03-29T10:07:14.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f40cf3c0 0x7f76f40b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76fbe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76f415d760 0x7f76f40b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76fbe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76f415db40 0x7f76f4157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76fbe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f41f3980 con 0x7f76f40cf3c0 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76fbe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f41f30b0 con 0x7f76f415db40 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76fbe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f40b67a0 con 0x7f76f415d760 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76f9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f40cf3c0 0x7f76f40b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76f9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f40cf3c0 0x7f76f40b6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47522/0 (socket says 172.21.15.12:47522) 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76f9400700 1 -- 172.21.15.12:0/3197358770 learned_addr learned my addr 172.21.15.12:0/3197358770 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76f8a00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76f415d760 0x7f76f40b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.909+0000 7f76f9e00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76f415db40 0x7f76f4157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:14.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76f9400700 1 -- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76f415d760 msgr2=0x7f76f40b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76f9400700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76f415d760 0x7f76f40b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76f9400700 1 -- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76f415db40 msgr2=0x7f76f4157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76f9400700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76f415db40 0x7f76f4157470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76f9400700 1 -- 172.21.15.12:0/3197358770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec009a70 con 0x7f76f40cf3c0 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76f8a00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76f415d760 0x7f76f40b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76f9400700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f40cf3c0 0x7f76f40b6ba0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f76e800c520 tx=0x7f76e800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76f1600700 1 -- 172.21.15.12:0/3197358770 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8005d70 con 0x7f76f40cf3c0 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76f1600700 1 -- 172.21.15.12:0/3197358770 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e80070d0 con 0x7f76f40cf3c0 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76fbe00700 1 -- 172.21.15.12:0/3197358770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f41f32c0 con 0x7f76f40cf3c0 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76f1600700 1 -- 172.21.15.12:0/3197358770 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8007440 con 0x7f76f40cf3c0 2024-03-29T10:07:14.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.910+0000 7f76fbe00700 1 -- 172.21.15.12:0/3197358770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f40b70e0 con 0x7f76f40cf3c0 2024-03-29T10:07:14.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.911+0000 7f76fbe00700 1 -- 172.21.15.12:0/3197358770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f41f3ac0 con 0x7f76f40cf3c0 2024-03-29T10:07:14.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.912+0000 7f76f1600700 1 -- 172.21.15.12:0/3197358770 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f76e80075a0 con 0x7f76f40cf3c0 2024-03-29T10:07:14.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.913+0000 7f76f1600700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76d406adb0 0x7f76d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:14.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.913+0000 7f76f1600700 1 -- 172.21.15.12:0/3197358770 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76e80888d0 con 0x7f76f40cf3c0 2024-03-29T10:07:14.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.913+0000 7f76f8a00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76d406adb0 0x7f76d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:14.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.914+0000 7f76f8a00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76d406adb0 0x7f76d406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f76e0006fd0 tx=0x7f76e0008040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:14.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:14.917+0000 7f76f1600700 1 -- 172.21.15.12:0/3197358770 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76f41f3ac0 con 0x7f76f40cf3c0 2024-03-29T10:07:15.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.173+0000 7f76fbe00700 1 -- 172.21.15.12:0/3197358770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76f4067720 con 0x7f76f40cf3c0 2024-03-29T10:07:15.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.173+0000 7f76f1600700 1 -- 172.21.15.12:0/3197358770 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f76e8017090 con 0x7f76f40cf3c0 2024-03-29T10:07:15.173 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:15.173 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5746,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/981039791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":981039791},{"type":"v1","addr":"172.21.15.159:6836","nonce":981039791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1682878615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1682878615},{"type":"v1","addr":"172.21.15.159:6837","nonce":1682878615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5752,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1663013205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":1663013205},{"type":"v1","addr":"172.21.15.12:6834","nonce":1663013205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:07:08.705772+0000","modified":"2024-03-29T10:07:12.186244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5877},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5877":{"gid":5877,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3970376731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3970376731},{"type":"v1","addr":"172.21.15.12:6835","nonce":3970376731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-29T10:07:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 -- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76d406adb0 msgr2=0x7f76d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76d406adb0 0x7f76d406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f76e0006fd0 tx=0x7f76e0008040 comp rx=0 tx=0).stop 2024-03-29T10:07:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 -- 172.21.15.12:0/3197358770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f40cf3c0 msgr2=0x7f76f40b6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f40cf3c0 0x7f76f40b6ba0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f76e800c520 tx=0x7f76e800c8e0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 -- 172.21.15.12:0/3197358770 shutdown_connections 2024-03-29T10:07:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76d406adb0 0x7f76d406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76f415db40 0x7f76f4157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f40cf3c0 0x7f76f40b6ba0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 --2- 172.21.15.12:0/3197358770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76f415d760 0x7f76f40b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 -- 172.21.15.12:0/3197358770 >> 172.21.15.12:0/3197358770 conn(0x7f76f408aff0 msgr2=0x7f76f40b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 -- 172.21.15.12:0/3197358770 shutdown_connections 2024-03-29T10:07:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.176+0000 7f76fbe00700 1 -- 172.21.15.12:0/3197358770 wait complete. 2024-03-29T10:07:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-29T10:07:15.598 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T10:07:08.705772+0000', 'modified': '2024-03-29T10:07:12.186244+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5877}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5877': {'gid': 5877, 'name': 'a', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6835/3970376731', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6832', 'nonce': 3970376731}, {'type': 'v1', 'addr': '172.21.15.12:6835', 'nonce': 3970376731}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T10:07:15.598 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T10:07:15.598 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-29T10:07:15.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.805+0000 7f0469e00700 1 Processor -- start 2024-03-29T10:07:15.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.806+0000 7f0469e00700 1 -- start start 2024-03-29T10:07:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.807+0000 7f0469e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 0x7f04640d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.807+0000 7f0469e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640d0970 con 0x7f046415d990 2024-03-29T10:07:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.807+0000 7f0469e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046406c020 con 0x7f046409a750 2024-03-29T10:07:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.807+0000 7f0469e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04640c81a0 con 0x7f0464099a60 2024-03-29T10:07:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.807+0000 7f0462000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f046415d990 0x7f0464099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35928/0 (socket says 172.21.15.12:35928) 2024-03-29T10:07:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.807+0000 7f0462000700 1 -- 172.21.15.12:0/115017275 learned_addr learned my addr 172.21.15.12:0/115017275 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.807+0000 7f0462a00700 1 --2- 172.21.15.12:0/115017275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 0x7f04640d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.808+0000 7f0461600700 1 -- 172.21.15.12:0/115017275 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360353525 0 0) 0x7f04640d0970 con 0x7f046415d990 2024-03-29T10:07:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.808+0000 7f0461600700 1 -- 172.21.15.12:0/115017275 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04480035b0 con 0x7f046415d990 2024-03-29T10:07:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.808+0000 7f0461600700 1 -- 172.21.15.12:0/115017275 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887510791 0 0) 0x7f046406c020 con 0x7f046409a750 2024-03-29T10:07:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.808+0000 7f0461600700 1 -- 172.21.15.12:0/115017275 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04640d0970 con 0x7f046409a750 2024-03-29T10:07:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.808+0000 7f0462a00700 1 -- 172.21.15.12:0/115017275 >> v1:172.21.15.159:6789/0 conn(0x7f046409a750 legacy=0x7f046406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.808+0000 7f0462a00700 1 -- 172.21.15.12:0/115017275 >> v1:172.21.15.12:6789/0 conn(0x7f046415d990 legacy=0x7f0464099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.808+0000 7f0462a00700 1 -- 172.21.15.12:0/115017275 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04640d0690 con 0x7f0464099a60 2024-03-29T10:07:15.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.808+0000 7f0462a00700 1 --2- 172.21.15.12:0/115017275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 0x7f04640d0060 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0454009870 tx=0x7f0454009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adff72868374ae0c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:15.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.808+0000 7f0461600700 1 -- 172.21.15.12:0/115017275 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454011830 con 0x7f0464099a60 2024-03-29T10:07:15.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.809+0000 7f0461600700 1 -- 172.21.15.12:0/115017275 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0454011990 con 0x7f0464099a60 2024-03-29T10:07:15.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.809+0000 7f0461600700 1 -- 172.21.15.12:0/115017275 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454011d20 con 0x7f0464099a60 2024-03-29T10:07:15.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.809+0000 7f0469e00700 1 -- 172.21.15.12:0/115017275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 msgr2=0x7f04640d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.809+0000 7f0469e00700 1 --2- 172.21.15.12:0/115017275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 0x7f04640d0060 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0454009870 tx=0x7f0454009b80 comp rx=0 tx=0).stop 2024-03-29T10:07:15.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.809+0000 7f0469e00700 1 -- 172.21.15.12:0/115017275 shutdown_connections 2024-03-29T10:07:15.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.809+0000 7f0469e00700 1 --2- 172.21.15.12:0/115017275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 0x7f04640d0060 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.809+0000 7f0469e00700 1 -- 172.21.15.12:0/115017275 >> 172.21.15.12:0/115017275 conn(0x7f04640b2e20 msgr2=0x7f04640b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:15.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.809+0000 7f0469e00700 1 -- 172.21.15.12:0/115017275 shutdown_connections 2024-03-29T10:07:15.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.809+0000 7f0469e00700 1 -- 172.21.15.12:0/115017275 wait complete. 2024-03-29T10:07:15.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.811+0000 7f0469e00700 1 Processor -- start 2024-03-29T10:07:15.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.811+0000 7f0469e00700 1 -- start start 2024-03-29T10:07:15.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0469e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 0x7f04640c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0469e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046409a750 0x7f04640caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0469e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415d990 0x7f046415dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0469e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640c74a0 con 0x7f046415d990 2024-03-29T10:07:15.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0469e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640c89c0 con 0x7f046409a750 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0469e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04640ca120 con 0x7f0464099a60 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0463400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415d990 0x7f046415dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0462000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046409a750 0x7f04640caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0462000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046409a750 0x7f04640caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:34800/0 (socket says 172.21.15.12:34800) 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0462000700 1 -- 172.21.15.12:0/717091693 learned_addr learned my addr 172.21.15.12:0/717091693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0463400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415d990 0x7f046415dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47536/0 (socket says 172.21.15.12:47536) 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.812+0000 7f0462a00700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 0x7f04640c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.813+0000 7f0463400700 1 -- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 msgr2=0x7f04640c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.813+0000 7f0463400700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 0x7f04640c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.813+0000 7f0463400700 1 -- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046409a750 msgr2=0x7f04640caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.813+0000 7f0463400700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046409a750 0x7f04640caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.813+0000 7f0463400700 1 -- 172.21.15.12:0/717091693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04540094a0 con 0x7f046415d990 2024-03-29T10:07:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.813+0000 7f0462000700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046409a750 0x7f04640caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:07:15.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.813+0000 7f0463400700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415d990 0x7f046415dda0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f045c00cbc0 tx=0x7f045c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:15.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.813+0000 7f045b400700 1 -- 172.21.15.12:0/717091693 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c003c00 con 0x7f046415d990 2024-03-29T10:07:15.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.813+0000 7f045b400700 1 -- 172.21.15.12:0/717091693 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c003d60 con 0x7f046415d990 2024-03-29T10:07:15.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.813+0000 7f045b400700 1 -- 172.21.15.12:0/717091693 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c00aa80 con 0x7f046415d990 2024-03-29T10:07:15.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.814+0000 7f0469e00700 1 -- 172.21.15.12:0/717091693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04640ca400 con 0x7f046415d990 2024-03-29T10:07:15.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.814+0000 7f0469e00700 1 -- 172.21.15.12:0/717091693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04640ca9c0 con 0x7f046415d990 2024-03-29T10:07:15.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.816+0000 7f045b400700 1 -- 172.21.15.12:0/717091693 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f045c00abe0 con 0x7f046415d990 2024-03-29T10:07:15.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.816+0000 7f045b400700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f044406adb0 0x7f044406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.816+0000 7f045b400700 1 -- 172.21.15.12:0/717091693 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f045c0894c0 con 0x7f046415d990 2024-03-29T10:07:15.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.817+0000 7f0462a00700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f044406adb0 0x7f044406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.818+0000 7f0462a00700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f044406adb0 0x7f044406d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f04540097c0 tx=0x7f045401c040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:15.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.819+0000 7f047da07b80 1 Processor -- start 2024-03-29T10:07:15.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.819+0000 7f047da07b80 1 -- start start 2024-03-29T10:07:15.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f047da07b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d9719830 0x5570d9719c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f047da07b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d96efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f047da07b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d96eab60 0x5570d9716a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f047da07b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d9664650 con 0x5570d96ecbe0 2024-03-29T10:07:15.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f047da07b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d9645c30 con 0x5570d9719830 2024-03-29T10:07:15.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f047da07b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570d9643430 con 0x5570d96eab60 2024-03-29T10:07:15.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f0462000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d96efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f0462000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d96efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47550/0 (socket says 172.21.15.12:47550) 2024-03-29T10:07:15.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f0462000700 1 -- 172.21.15.12:0/1482422926 learned_addr learned my addr 172.21.15.12:0/1482422926 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:15.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f0462000700 1 -- 172.21.15.12:0/1482422926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d96eab60 msgr2=0x5570d9716a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:15.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f0462000700 1 --2- 172.21.15.12:0/1482422926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d96eab60 0x5570d9716a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f0462000700 1 -- 172.21.15.12:0/1482422926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d9719830 msgr2=0x5570d9719c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:15.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f0462000700 1 --2- 172.21.15.12:0/1482422926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d9719830 0x5570d9719c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.820+0000 7f0462000700 1 -- 172.21.15.12:0/1482422926 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570d9432790 con 0x5570d96ecbe0 2024-03-29T10:07:15.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.821+0000 7f0462000700 1 --2- 172.21.15.12:0/1482422926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d96efa10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0450001ba0 tx=0x7f0450007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a91b5f45e3b6524 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:15.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.821+0000 7f044f400700 1 -- 172.21.15.12:0/1482422926 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450005b50 con 0x5570d96ecbe0 2024-03-29T10:07:15.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.821+0000 7f044f400700 1 -- 172.21.15.12:0/1482422926 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0450005cb0 con 0x5570d96ecbe0 2024-03-29T10:07:15.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.821+0000 7f044f400700 1 -- 172.21.15.12:0/1482422926 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045000fa80 con 0x5570d96ecbe0 2024-03-29T10:07:15.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.821+0000 7f047da07b80 1 -- 172.21.15.12:0/1482422926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 msgr2=0x5570d96efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.821+0000 7f047da07b80 1 --2- 172.21.15.12:0/1482422926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d96efa10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0450001ba0 tx=0x7f0450007ea0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.822+0000 7f047da07b80 1 -- 172.21.15.12:0/1482422926 shutdown_connections 2024-03-29T10:07:15.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.822+0000 7f047da07b80 1 --2- 172.21.15.12:0/1482422926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d96eab60 0x5570d9716a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.822+0000 7f047da07b80 1 --2- 172.21.15.12:0/1482422926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d9719830 0x5570d9719c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.822+0000 7f047da07b80 1 --2- 172.21.15.12:0/1482422926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d96efa10 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.822+0000 7f047da07b80 1 -- 172.21.15.12:0/1482422926 >> 172.21.15.12:0/1482422926 conn(0x5570d964d6a0 msgr2=0x5570d9734640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:15.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.822+0000 7f047da07b80 1 -- 172.21.15.12:0/1482422926 shutdown_connections 2024-03-29T10:07:15.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.822+0000 7f047da07b80 1 -- 172.21.15.12:0/1482422926 wait complete. 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.822+0000 7f047da07b80 1 -- start start 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.823+0000 7f047da07b80 1 Processor -- start 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f047da07b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96eab60 0x5570d971c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f047da07b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d97138d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f047da07b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d9719830 0x5570d9713e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f047da07b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d963ed20 con 0x5570d96ecbe0 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f047da07b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d95da770 con 0x5570d96eab60 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f047da07b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570d95fd6f0 con 0x5570d9719830 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f047da07b80 10 client.0 _ll_register_callbacks cb 0x5570d95e45e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f0462000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d97138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f0462000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d97138d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47566/0 (socket says 172.21.15.12:47566) 2024-03-29T10:07:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f0462000700 1 -- 172.21.15.12:0/3227116549 learned_addr learned my addr 172.21.15.12:0/3227116549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f0462a00700 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96eab60 0x5570d971c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f0463400700 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d9719830 0x5570d9713e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f0462000700 1 -- 172.21.15.12:0/3227116549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d9719830 msgr2=0x5570d9713e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f0462000700 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d9719830 0x5570d9713e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f0462000700 1 -- 172.21.15.12:0/3227116549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96eab60 msgr2=0x5570d971c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f0462000700 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96eab60 0x5570d971c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.824+0000 7f0462000700 1 -- 172.21.15.12:0/3227116549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04480035b0 con 0x5570d96ecbe0 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f0462000700 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d97138d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0450000f60 tx=0x7f0450000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fddc5e2925e64b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f0443e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f0443e00700 1 -- 172.21.15.12:0/3227116549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04500072d0 con 0x5570d96ecbe0 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f0443e00700 1 -- 172.21.15.12:0/3227116549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0450027ae0 con 0x5570d96ecbe0 2024-03-29T10:07:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f047da07b80 1 -- 172.21.15.12:0/3227116549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570d96e1920 con 0x5570d96ecbe0 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f0443e00700 1 -- 172.21.15.12:0/3227116549 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0450027ca0 con 0x5570d96ecbe0 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f047da07b80 10 client.6012 fetch_fsmap learned FSMap version 23 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f047da07b80 1 -- 172.21.15.12:0/3227116549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x5570d951f0a0 con 0x5570d96ecbe0 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f0443e00700 1 -- 172.21.15.12:0/3227116549 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5570d96e1920 con 0x5570d96ecbe0 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f0443e00700 1 -- 172.21.15.12:0/3227116549 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0450012070 con 0x5570d96ecbe0 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.825+0000 7f047da07b80 10 client.6012 fetch_fsmap finished waiting for FSMap version 23 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.826+0000 7f047da07b80 10 client.6012 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.826+0000 7f047da07b80 20 client.6012 populate_metadata read hostname 'smithi012' 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.826+0000 7f047da07b80 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.12:6832/3970376731,v1:172.21.15.12:6835/3970376731] conn(0x5570d9717000 0x5570d97513d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.826+0000 7f047da07b80 4 client.6012 mds_command: new command op to 5877 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.826+0000 7f047da07b80 1 -- 172.21.15.12:0/3227116549 --> [v2:172.21.15.12:6832/3970376731,v1:172.21.15.12:6835/3970376731] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570d96be6f0 con 0x5570d9717000 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.826+0000 7f0462a00700 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.12:6832/3970376731,v1:172.21.15.12:6835/3970376731] conn(0x5570d9717000 0x5570d97513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.826+0000 7f0462a00700 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.12:6832/3970376731,v1:172.21.15.12:6835/3970376731] conn(0x5570d9717000 0x5570d97513d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32739fa1401c803c server_cookie=220a60f5269f12b in_seq=0 out_seq=0 2024-03-29T10:07:15.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.826+0000 7f0443e00700 10 client.6012 ms_handle_connect on v2:172.21.15.12:6832/3970376731 2024-03-29T10:07:15.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f0443e00700 1 -- 172.21.15.12:0/3227116549 <== mds.0 v2:172.21.15.12:6832/3970376731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5570d96be6f0 con 0x5570d9717000 2024-03-29T10:07:15.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f0443e00700 10 client.6012 handle_command_reply: tid=0 2024-03-29T10:07:15.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f047da07b80 1 client.6012 shutdown 2024-03-29T10:07:15.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f047da07b80 1 -- 172.21.15.12:0/3227116549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 msgr2=0x5570d97138d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f047da07b80 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d97138d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0450000f60 tx=0x7f0450000ed0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f047da07b80 1 -- 172.21.15.12:0/3227116549 shutdown_connections 2024-03-29T10:07:15.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f047da07b80 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d9719830 0x5570d9713e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f047da07b80 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.12:6832/3970376731,v1:172.21.15.12:6835/3970376731] conn(0x5570d9717000 0x5570d97513d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f047da07b80 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96eab60 0x5570d971c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f047da07b80 1 --2- 172.21.15.12:0/3227116549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96ecbe0 0x5570d97138d0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f0443e00700 0 client.6012 ms_handle_reset on v2:172.21.15.12:6832/3970376731 2024-03-29T10:07:15.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.827+0000 7f047da07b80 1 -- 172.21.15.12:0/3227116549 >> 172.21.15.12:0/3227116549 conn(0x5570d964d6a0 msgr2=0x5570d9737a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:15.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.828+0000 7f047da07b80 1 -- 172.21.15.12:0/3227116549 shutdown_connections 2024-03-29T10:07:15.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.828+0000 7f047da07b80 1 -- 172.21.15.12:0/3227116549 wait complete. 2024-03-29T10:07:15.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.828+0000 7f047da07b80 20 client.6012 trim_cache size 0 max 16384 2024-03-29T10:07:15.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.846+0000 7f047da07b80 1 Processor -- start 2024-03-29T10:07:15.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.846+0000 7f047da07b80 1 -- start start 2024-03-29T10:07:15.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f047da07b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96eab60 0x5570d96eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f047da07b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d96ecbe0 0x5570d9714410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f047da07b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d9719830 0x5570d9714950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f047da07b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d9433390 con 0x5570d9719830 2024-03-29T10:07:15.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f047da07b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d9746fc0 con 0x5570d96eab60 2024-03-29T10:07:15.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f047da07b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570d971a650 con 0x5570d96ecbe0 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f0463400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d9719830 0x5570d9714950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f0463400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d9719830 0x5570d9714950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47582/0 (socket says 172.21.15.12:47582) 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f0463400700 1 -- 172.21.15.12:0/931984080 learned_addr learned my addr 172.21.15.12:0/931984080 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f0463400700 1 -- 172.21.15.12:0/931984080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d96ecbe0 msgr2=0x5570d9714410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.847+0000 7f0463400700 1 --2- 172.21.15.12:0/931984080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d96ecbe0 0x5570d9714410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.848+0000 7f0463400700 1 -- 172.21.15.12:0/931984080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96eab60 msgr2=0x5570d96eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.848+0000 7f0463400700 1 --2- 172.21.15.12:0/931984080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96eab60 0x5570d96eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.848+0000 7f0463400700 1 -- 172.21.15.12:0/931984080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570d9605010 con 0x5570d9719830 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.848+0000 7f0463400700 1 --2- 172.21.15.12:0/931984080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d9719830 0x5570d9714950 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f045c058150 tx=0x7f045c057d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=650f780da0e94c4f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.848+0000 7f044fe00700 1 -- 172.21.15.12:0/931984080 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c0581c0 con 0x5570d9719830 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.848+0000 7f044fe00700 1 -- 172.21.15.12:0/931984080 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c05bff0 con 0x5570d9719830 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.848+0000 7f044fe00700 1 -- 172.21.15.12:0/931984080 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c0605b0 con 0x5570d9719830 2024-03-29T10:07:15.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.848+0000 7f047da07b80 1 -- 172.21.15.12:0/931984080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d9719830 msgr2=0x5570d9714950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.848+0000 7f047da07b80 1 --2- 172.21.15.12:0/931984080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d9719830 0x5570d9714950 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f045c058150 tx=0x7f045c057d80 comp rx=0 tx=0).stop 2024-03-29T10:07:15.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.849+0000 7f047da07b80 1 -- 172.21.15.12:0/931984080 shutdown_connections 2024-03-29T10:07:15.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.849+0000 7f047da07b80 1 --2- 172.21.15.12:0/931984080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d9719830 0x5570d9714950 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.849+0000 7f047da07b80 1 --2- 172.21.15.12:0/931984080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96eab60 0x5570d96eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.849+0000 7f047da07b80 1 --2- 172.21.15.12:0/931984080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d96ecbe0 0x5570d9714410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.849+0000 7f047da07b80 1 -- 172.21.15.12:0/931984080 >> 172.21.15.12:0/931984080 conn(0x5570d964d6a0 msgr2=0x5570d971c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:15.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.849+0000 7f047da07b80 1 -- 172.21.15.12:0/931984080 shutdown_connections 2024-03-29T10:07:15.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.849+0000 7f047da07b80 1 -- 172.21.15.12:0/931984080 wait complete. 2024-03-29T10:07:15.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.849+0000 7f047da07b80 1 -- start start 2024-03-29T10:07:15.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.849+0000 7f047da07b80 1 Processor -- start 2024-03-29T10:07:15.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.850+0000 7f047da07b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96eab60 0x5570d97176c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.850+0000 7f047da07b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96ecbe0 0x5570d9717c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.850+0000 7f047da07b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d9719830 0x5570d979b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.850+0000 7f047da07b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d9731990 con 0x5570d96eab60 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.850+0000 7f047da07b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d97313f0 con 0x5570d96ecbe0 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.850+0000 7f047da07b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570d9731be0 con 0x5570d9719830 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f047da07b80 10 client.0 _ll_register_callbacks cb 0x7f0450001a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96eab60 0x5570d97176c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96eab60 0x5570d97176c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47586/0 (socket says 172.21.15.12:47586) 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462a00700 1 -- 172.21.15.12:0/332818713 learned_addr learned my addr 172.21.15.12:0/332818713 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0463400700 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d9719830 0x5570d979b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462000700 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96ecbe0 0x5570d9717c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462a00700 1 -- 172.21.15.12:0/332818713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d9719830 msgr2=0x5570d979b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462a00700 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d9719830 0x5570d979b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462a00700 1 -- 172.21.15.12:0/332818713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96ecbe0 msgr2=0x5570d9717c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:15.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462a00700 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96ecbe0 0x5570d9717c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462a00700 1 -- 172.21.15.12:0/332818713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04640ca9c0 con 0x5570d96eab60 2024-03-29T10:07:15.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462000700 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96ecbe0 0x5570d9717c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:07:15.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0462a00700 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96eab60 0x5570d97176c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0454009470 tx=0x7f04540047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2016838acd86be server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:15.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.851+0000 7f0443e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:07:15.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f0443e00700 1 -- 172.21.15.12:0/332818713 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04540313e0 con 0x5570d96eab60 2024-03-29T10:07:15.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f047da07b80 1 -- 172.21.15.12:0/332818713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0450005cb0 con 0x5570d96eab60 2024-03-29T10:07:15.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f0443e00700 1 -- 172.21.15.12:0/332818713 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f045401f040 con 0x5570d96eab60 2024-03-29T10:07:15.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f0443e00700 1 -- 172.21.15.12:0/332818713 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0454031540 con 0x5570d96eab60 2024-03-29T10:07:15.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f047da07b80 10 client.6018 fetch_fsmap learned FSMap version 23 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f047da07b80 1 -- 172.21.15.12:0/332818713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x5570d94931d0 con 0x5570d96eab60 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f0443e00700 1 -- 172.21.15.12:0/332818713 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0450005cb0 con 0x5570d96eab60 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f0443e00700 1 -- 172.21.15.12:0/332818713 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0454025470 con 0x5570d96eab60 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f047da07b80 10 client.6018 fetch_fsmap finished waiting for FSMap version 23 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f047da07b80 10 client.6018 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f047da07b80 20 client.6018 populate_metadata read hostname 'smithi012' 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f047da07b80 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.12:6832/3970376731,v1:172.21.15.12:6835/3970376731] conn(0x5570d9738030 0x5570d9738410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f047da07b80 4 client.6018 mds_command: new command op to 5877 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.852+0000 7f047da07b80 1 -- 172.21.15.12:0/332818713 --> [v2:172.21.15.12:6832/3970376731,v1:172.21.15.12:6835/3970376731] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5570d9478040 con 0x5570d9738030 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.853+0000 7f0462000700 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.12:6832/3970376731,v1:172.21.15.12:6835/3970376731] conn(0x5570d9738030 0x5570d9738410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.853+0000 7f0462000700 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.12:6832/3970376731,v1:172.21.15.12:6835/3970376731] conn(0x5570d9738030 0x5570d9738410 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb7a192f2c326a52 server_cookie=c3ef37358c69af6f in_seq=0 out_seq=0 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.853+0000 7f0443e00700 10 client.6018 ms_handle_connect on v2:172.21.15.12:6832/3970376731 2024-03-29T10:07:15.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.853+0000 7f0443e00700 1 -- 172.21.15.12:0/332818713 <== mds.0 v2:172.21.15.12:6832/3970376731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5570d9478040 con 0x5570d9738030 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.853+0000 7f0443e00700 10 client.6018 handle_command_reply: tid=0 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.853+0000 7f047da07b80 1 client.6018 shutdown 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.853+0000 7f047da07b80 1 -- 172.21.15.12:0/332818713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96eab60 msgr2=0x5570d97176c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.853+0000 7f047da07b80 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96eab60 0x5570d97176c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0454009470 tx=0x7f04540047c0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.853+0000 7f047da07b80 1 -- 172.21.15.12:0/332818713 shutdown_connections 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f047da07b80 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5570d9719830 0x5570d979b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f047da07b80 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5570d96eab60 0x5570d97176c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f047da07b80 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.12:6832/3970376731,v1:172.21.15.12:6835/3970376731] conn(0x5570d9738030 0x5570d9738410 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f047da07b80 1 --2- 172.21.15.12:0/332818713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5570d96ecbe0 0x5570d9717c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f047da07b80 1 -- 172.21.15.12:0/332818713 >> 172.21.15.12:0/332818713 conn(0x5570d964d6a0 msgr2=0x5570d9714320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f0443e00700 0 client.6018 ms_handle_reset on v2:172.21.15.12:6832/3970376731 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f047da07b80 1 -- 172.21.15.12:0/332818713 shutdown_connections 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f047da07b80 1 -- 172.21.15.12:0/332818713 wait complete. 2024-03-29T10:07:15.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f047da07b80 20 client.6018 trim_cache size 0 max 16384 2024-03-29T10:07:15.860 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:07:15.860 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T10:07:15.860 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T10:07:15.860 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 5877, 2024-03-29T10:07:15.860 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T10:07:15.860 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T10:07:15.860 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T10:07:15.860 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1458300760000002, 2024-03-29T10:07:15.860 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 23, 2024-03-29T10:07:15.860 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 61, 2024-03-29T10:07:15.861 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 61, 2024-03-29T10:07:15.861 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12.155130206000001 2024-03-29T10:07:15.861 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:07:15.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f0469e00700 1 -- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f044406adb0 msgr2=0x7f044406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f0469e00700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f044406adb0 0x7f044406d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f04540097c0 tx=0x7f045401c040 comp rx=0 tx=0).stop 2024-03-29T10:07:15.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f0469e00700 1 -- 172.21.15.12:0/717091693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415d990 msgr2=0x7f046415dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:15.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.854+0000 7f0469e00700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415d990 0x7f046415dda0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f045c00cbc0 tx=0x7f045c00ced0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.855+0000 7f0469e00700 1 -- 172.21.15.12:0/717091693 shutdown_connections 2024-03-29T10:07:15.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.855+0000 7f0469e00700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f044406adb0 0x7f044406d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.855+0000 7f0469e00700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415d990 0x7f046415dda0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.855+0000 7f0469e00700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0464099a60 0x7f04640c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.855+0000 7f0469e00700 1 --2- 172.21.15.12:0/717091693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046409a750 0x7f04640caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:15.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.855+0000 7f0469e00700 1 -- 172.21.15.12:0/717091693 >> 172.21.15.12:0/717091693 conn(0x7f04640b2e20 msgr2=0x7f046406ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:15.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.855+0000 7f0469e00700 1 -- 172.21.15.12:0/717091693 shutdown_connections 2024-03-29T10:07:15.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:15.855+0000 7f0469e00700 1 -- 172.21.15.12:0/717091693 wait complete. 2024-03-29T10:07:15.872 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T10:07:16.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.071+0000 7fec45c00700 1 Processor -- start 2024-03-29T10:07:16.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.071+0000 7fec45c00700 1 -- start start 2024-03-29T10:07:16.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.072+0000 7fec45c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fec40160f10 0x7fec400d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:16.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.072+0000 7fec45c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec400cf230 0x7fec400d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:16.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.072+0000 7fec45c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec40161590 0x7fec40161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:16.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.072+0000 7fec45c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec40161f60 con 0x7fec40161590 2024-03-29T10:07:16.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.072+0000 7fec45c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec400ceda0 con 0x7fec40160f10 2024-03-29T10:07:16.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.072+0000 7fec45c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec400d23d0 con 0x7fec400cf230 2024-03-29T10:07:16.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.072+0000 7fec3f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec40161590 0x7fec40161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:16.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.072+0000 7fec3f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec40161590 0x7fec40161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47600/0 (socket says 172.21.15.12:47600) 2024-03-29T10:07:16.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.072+0000 7fec3f400700 1 -- 172.21.15.12:0/2349111046 learned_addr learned my addr 172.21.15.12:0/2349111046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:16.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.073+0000 7fec3ea00700 1 --2- 172.21.15.12:0/2349111046 >> v2:172.21.15.159:3300/0 conn(0x7fec40160f10 0x7fec400d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:16.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.073+0000 7fec3e000700 1 --2- 172.21.15.12:0/2349111046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec400cf230 0x7fec400d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:16.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.073+0000 7fec3f400700 1 -- 172.21.15.12:0/2349111046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec400cf230 msgr2=0x7fec400d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:16.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.073+0000 7fec3f400700 1 --2- 172.21.15.12:0/2349111046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec400cf230 0x7fec400d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.073+0000 7fec3f400700 1 -- 172.21.15.12:0/2349111046 >> v2:172.21.15.159:3300/0 conn(0x7fec40160f10 msgr2=0x7fec400d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:16.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.073+0000 7fec3f400700 1 --2- 172.21.15.12:0/2349111046 >> v2:172.21.15.159:3300/0 conn(0x7fec40160f10 0x7fec400d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.073+0000 7fec3f400700 1 -- 172.21.15.12:0/2349111046 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec401612f0 con 0x7fec40161590 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.073+0000 7fec3f400700 1 --2- 172.21.15.12:0/2349111046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40161590 0x7fec40161a20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fec400d1080 tx=0x7fec3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805a2d42bf3c6bec server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.073+0000 7fec3d600700 1 -- 172.21.15.12:0/2349111046 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3800e070 con 0x7fec40161590 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec3d600700 1 -- 172.21.15.12:0/2349111046 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3800b910 con 0x7fec40161590 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec3d600700 1 -- 172.21.15.12:0/2349111046 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3800bcc0 con 0x7fec40161590 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec45c00700 1 -- 172.21.15.12:0/2349111046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40161590 msgr2=0x7fec40161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec45c00700 1 --2- 172.21.15.12:0/2349111046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40161590 0x7fec40161a20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fec400d1080 tx=0x7fec3800b330 comp rx=0 tx=0).stop 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec45c00700 1 -- 172.21.15.12:0/2349111046 shutdown_connections 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec45c00700 1 --2- 172.21.15.12:0/2349111046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40161590 0x7fec40161a20 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec45c00700 1 --2- 172.21.15.12:0/2349111046 >> v2:172.21.15.159:3300/0 conn(0x7fec40160f10 0x7fec400d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec45c00700 1 --2- 172.21.15.12:0/2349111046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec400cf230 0x7fec400d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec45c00700 1 -- 172.21.15.12:0/2349111046 >> 172.21.15.12:0/2349111046 conn(0x7fec4008aff0 msgr2=0x7fec400d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec45c00700 1 -- 172.21.15.12:0/2349111046 shutdown_connections 2024-03-29T10:07:16.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.074+0000 7fec45c00700 1 -- 172.21.15.12:0/2349111046 wait complete. 2024-03-29T10:07:16.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.077+0000 7fec45c00700 1 Processor -- start 2024-03-29T10:07:16.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.077+0000 7fec45c00700 1 -- start start 2024-03-29T10:07:16.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec45c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec400cf230 0x7fec400b6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec45c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40160f10 0x7fec401f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec45c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec40161590 0x7fec4015f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec45c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400b7670 con 0x7fec40160f10 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec45c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400b7b00 con 0x7fec400cf230 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec45c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec400b6a60 con 0x7fec40161590 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec3e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40160f10 0x7fec401f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec3e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40160f10 0x7fec401f65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47616/0 (socket says 172.21.15.12:47616) 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec3e000700 1 -- 172.21.15.12:0/4065379493 learned_addr learned my addr 172.21.15.12:0/4065379493 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec3ea00700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec400cf230 0x7fec400b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.078+0000 7fec3f400700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec40161590 0x7fec4015f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.079+0000 7fec3e000700 1 -- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec40161590 msgr2=0x7fec4015f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.079+0000 7fec3e000700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec40161590 0x7fec4015f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.079+0000 7fec3e000700 1 -- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec400cf230 msgr2=0x7fec400b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:16.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.079+0000 7fec3e000700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec400cf230 0x7fec400b6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.079+0000 7fec3e000700 1 -- 172.21.15.12:0/4065379493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec38009b30 con 0x7fec40160f10 2024-03-29T10:07:16.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.079+0000 7fec3e000700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40160f10 0x7fec401f65e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00e870 tx=0x7fec2c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:16.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.079+0000 7fec36a00700 1 -- 172.21.15.12:0/4065379493 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c009590 con 0x7fec40160f10 2024-03-29T10:07:16.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.079+0000 7fec36a00700 1 -- 172.21.15.12:0/4065379493 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c013680 con 0x7fec40160f10 2024-03-29T10:07:16.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.079+0000 7fec36a00700 1 -- 172.21.15.12:0/4065379493 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c0139f0 con 0x7fec40160f10 2024-03-29T10:07:16.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.079+0000 7fec45c00700 1 -- 172.21.15.12:0/4065379493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec400b6d40 con 0x7fec40160f10 2024-03-29T10:07:16.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.080+0000 7fec45c00700 1 -- 172.21.15.12:0/4065379493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec400b7300 con 0x7fec40160f10 2024-03-29T10:07:16.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.080+0000 7fec45c00700 1 -- 172.21.15.12:0/4065379493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec400b7820 con 0x7fec40160f10 2024-03-29T10:07:16.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.081+0000 7fec36a00700 1 -- 172.21.15.12:0/4065379493 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fec2c013b50 con 0x7fec40160f10 2024-03-29T10:07:16.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.082+0000 7fec36a00700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fec2006adb0 0x7fec2006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:16.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.082+0000 7fec36a00700 1 -- 172.21.15.12:0/4065379493 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec2c0892f0 con 0x7fec40160f10 2024-03-29T10:07:16.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.083+0000 7fec3ea00700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fec2006adb0 0x7fec2006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:16.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.083+0000 7fec3ea00700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fec2006adb0 0x7fec2006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fec30009ac0 tx=0x7fec30009210 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:16.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.087+0000 7fec36a00700 1 -- 172.21.15.12:0/4065379493 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec400b7820 con 0x7fec40160f10 2024-03-29T10:07:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.350+0000 7fec45c00700 1 -- 172.21.15.12:0/4065379493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fec400b7820 con 0x7fec40160f10 2024-03-29T10:07:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.351+0000 7fec36a00700 1 -- 172.21.15.12:0/4065379493 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+73 (secure 0 0 0) 0x7fec400b7820 con 0x7fec40160f10 2024-03-29T10:07:16.351 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.353+0000 7fec45c00700 1 -- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fec2006adb0 msgr2=0x7fec2006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.353+0000 7fec45c00700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fec2006adb0 0x7fec2006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fec30009ac0 tx=0x7fec30009210 comp rx=0 tx=0).stop 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.353+0000 7fec45c00700 1 -- 172.21.15.12:0/4065379493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40160f10 msgr2=0x7fec401f65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.353+0000 7fec45c00700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40160f10 0x7fec401f65e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00e870 tx=0x7fec2c00eb80 comp rx=0 tx=0).stop 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.353+0000 7fec45c00700 1 -- 172.21.15.12:0/4065379493 shutdown_connections 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.353+0000 7fec45c00700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fec2006adb0 0x7fec2006d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.353+0000 7fec45c00700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec40161590 0x7fec4015f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.353+0000 7fec45c00700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec400cf230 0x7fec400b6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.353+0000 7fec45c00700 1 --2- 172.21.15.12:0/4065379493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec40160f10 0x7fec401f65e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.353+0000 7fec45c00700 1 -- 172.21.15.12:0/4065379493 >> 172.21.15.12:0/4065379493 conn(0x7fec4008aff0 msgr2=0x7fec400cc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.354+0000 7fec45c00700 1 -- 172.21.15.12:0/4065379493 shutdown_connections 2024-03-29T10:07:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:16.354+0000 7fec45c00700 1 -- 172.21.15.12:0/4065379493 wait complete. 2024-03-29T10:07:16.371 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T10:07:16.371 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T10:07:16.371 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T10:07:16.371 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T10:07:16.371 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T10:07:16.372 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T10:07:16.372 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T10:07:16.372 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T10:07:16.372 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:07:16.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:16.372 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T10:07:16.419 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T10:07:16.419 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T10:07:16.419 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T10:07:16.419 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:07:16.419 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T10:07:16.420 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:07:16.420 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:07:16.420 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:07:16.420 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T10:07:16.420 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:07:16.420 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:07:16.420 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T10:07:16.420 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T10:07:16.421 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T10:07:16.421 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T10:07:16.421 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T10:07:16.421 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 40681sec preferred_lft 40681sec 2024-03-29T10:07:16.421 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T10:07:16.421 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:07:16.421 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:07:16.421 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T10:07:16.422 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T10:07:16.422 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T10:07:16.422 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:07:16.422 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T10:07:16.422 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:07:16.422 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T10:07:16.422 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:07:16.422 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:07:16.423 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T10:07:16.423 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T10:07:16.423 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:07:16.423 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:07:16.423 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T10:07:16.423 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T10:07:16.423 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:07:16.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:16.424 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T10:07:16.525 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T10:07:16.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:16.528 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T10:07:16.576 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T10:07:16.576 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T10:07:16.576 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T10:07:16.576 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T10:07:16.576 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T10:07:16.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:16.578 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T10:07:16.578 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T10:07:16.578 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T10:07:16.578 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T10:07:16.578 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T10:07:16.578 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T10:07:16.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:16.745 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T10:07:16.789 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T10:07:16.789 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T10:07:16.790 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:07:16.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:16.790 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T10:07:16.898 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:07:16.898 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T10:07:16.942 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:07:16.942 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:07:16.942 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:07:17.026 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T10:07:17.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:17.026 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:07:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:07:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:07:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:07:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:07:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T10:07:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T10:07:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T10:07:17.201 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T10:07:17.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:17.201 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:07:17.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:17.269 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T10:07:17.374 INFO:teuthology.orchestra.run.smithi012.stdout:client5483 2024-03-29T10:07:17.374 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:07:17.374 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:07:17.419 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T10:07:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.657+0000 7f1ccdc00700 1 Processor -- start 2024-03-29T10:07:17.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.658+0000 7f1ccdc00700 1 -- start start 2024-03-29T10:07:17.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.659+0000 7f1ccdc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cbb30 0x7f1cc80cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.659+0000 7f1ccdc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc80cc450 con 0x7f1cc80cd4a0 2024-03-29T10:07:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.659+0000 7f1ccdc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc80cdae0 con 0x7f1cc80cd020 2024-03-29T10:07:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.659+0000 7f1ccdc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80cddc0 con 0x7f1cc80cbb30 2024-03-29T10:07:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.659+0000 7f1cc7400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1cc80cd4a0 0x7f1cc806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35934/0 (socket says 172.21.15.12:35934) 2024-03-29T10:07:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.659+0000 7f1cc7400700 1 -- 172.21.15.12:0/2631223413 learned_addr learned my addr 172.21.15.12:0/2631223413 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.659+0000 7f1cc6a00700 1 --2- 172.21.15.12:0/2631223413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cbb30 0x7f1cc80cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.660+0000 7f1cc5600700 1 -- 172.21.15.12:0/2631223413 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454547961 0 0) 0x7f1cc80cc450 con 0x7f1cc80cd4a0 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.660+0000 7f1cc5600700 1 -- 172.21.15.12:0/2631223413 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cac0035b0 con 0x7f1cc80cd4a0 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.660+0000 7f1cc5600700 1 -- 172.21.15.12:0/2631223413 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696323572 0 0) 0x7f1cc80cdae0 con 0x7f1cc80cd020 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.660+0000 7f1cc5600700 1 -- 172.21.15.12:0/2631223413 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc80cc450 con 0x7f1cc80cd020 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.660+0000 7f1cc5600700 1 -- 172.21.15.12:0/2631223413 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 976746916 0 0) 0x7f1cac0035b0 con 0x7f1cc80cd4a0 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.660+0000 7f1cc6a00700 1 -- 172.21.15.12:0/2631223413 >> v1:172.21.15.159:6789/0 conn(0x7f1cc80cd020 legacy=0x7f1cc80b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.660+0000 7f1cc6a00700 1 -- 172.21.15.12:0/2631223413 >> v1:172.21.15.12:6789/0 conn(0x7f1cc80cd4a0 legacy=0x7f1cc806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.660+0000 7f1cc6a00700 1 -- 172.21.15.12:0/2631223413 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc80ce0a0 con 0x7f1cc80cbb30 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.660+0000 7f1cc6a00700 1 --2- 172.21.15.12:0/2631223413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cbb30 0x7f1cc80cbf10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009870 tx=0x7f1cb8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e9759b32aa7ddf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.661+0000 7f1cc5600700 1 -- 172.21.15.12:0/2631223413 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc00019f0 con 0x7f1cc80cbb30 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.661+0000 7f1cc5600700 1 -- 172.21.15.12:0/2631223413 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb8004030 con 0x7f1cc80cbb30 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.661+0000 7f1cc5600700 1 -- 172.21.15.12:0/2631223413 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb80043a0 con 0x7f1cc80cbb30 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.661+0000 7f1ccdc00700 1 -- 172.21.15.12:0/2631223413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cbb30 msgr2=0x7f1cc80cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.661+0000 7f1ccdc00700 1 --2- 172.21.15.12:0/2631223413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cbb30 0x7f1cc80cbf10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009870 tx=0x7f1cb8009b80 comp rx=0 tx=0).stop 2024-03-29T10:07:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.661+0000 7f1ccdc00700 1 -- 172.21.15.12:0/2631223413 shutdown_connections 2024-03-29T10:07:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.661+0000 7f1ccdc00700 1 --2- 172.21.15.12:0/2631223413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cbb30 0x7f1cc80cbf10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.661+0000 7f1ccdc00700 1 -- 172.21.15.12:0/2631223413 >> 172.21.15.12:0/2631223413 conn(0x7f1cc80b0d70 msgr2=0x7f1cc80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.661+0000 7f1ccdc00700 1 -- 172.21.15.12:0/2631223413 shutdown_connections 2024-03-29T10:07:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.661+0000 7f1ccdc00700 1 -- 172.21.15.12:0/2631223413 wait complete. 2024-03-29T10:07:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.663+0000 7f1ccdc00700 1 Processor -- start 2024-03-29T10:07:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.663+0000 7f1ccdc00700 1 -- start start 2024-03-29T10:07:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.664+0000 7f1ccdc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc80cbb30 0x7f1cc8072510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.664+0000 7f1ccdc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc80cd020 0x7f1cc8155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.664+0000 7f1ccdc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cd4a0 0x7f1cc80c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.664+0000 7f1ccdc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc8155dd0 con 0x7f1cc80cd020 2024-03-29T10:07:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.664+0000 7f1ccdc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc8155ab0 con 0x7f1cc80cbb30 2024-03-29T10:07:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.664+0000 7f1ccdc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80ca730 con 0x7f1cc80cd4a0 2024-03-29T10:07:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.664+0000 7f1cc6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc80cd020 0x7f1cc8155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.664+0000 7f1cc6a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc80cbb30 0x7f1cc8072510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.664+0000 7f1cc6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc80cd020 0x7f1cc8155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47628/0 (socket says 172.21.15.12:47628) 2024-03-29T10:07:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.664+0000 7f1cc6000700 1 -- 172.21.15.12:0/3353565789 learned_addr learned my addr 172.21.15.12:0/3353565789 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:17.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.665+0000 7f1cc7400700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cd4a0 0x7f1cc80c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:17.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.665+0000 7f1cc6a00700 1 -- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cd4a0 msgr2=0x7f1cc80c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:17.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.665+0000 7f1cc6a00700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cd4a0 0x7f1cc80c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:17.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.665+0000 7f1cc6a00700 1 -- 172.21.15.12:0/3353565789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc80cd020 msgr2=0x7f1cc8155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:17.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.665+0000 7f1cc6a00700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc80cd020 0x7f1cc8155570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:17.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.665+0000 7f1cc6a00700 1 -- 172.21.15.12:0/3353565789 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb80094a0 con 0x7f1cc80cbb30 2024-03-29T10:07:17.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.665+0000 7f1cc6a00700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc80cbb30 0x7f1cc8072510 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8015000 tx=0x7f1cb8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:17.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.665+0000 7f1cbea00700 1 -- 172.21.15.12:0/3353565789 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8003c50 con 0x7f1cc80cbb30 2024-03-29T10:07:17.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.665+0000 7f1cbea00700 1 -- 172.21.15.12:0/3353565789 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb801f070 con 0x7f1cc80cbb30 2024-03-29T10:07:17.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.666+0000 7f1ccdc00700 1 -- 172.21.15.12:0/3353565789 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc80caa10 con 0x7f1cc80cbb30 2024-03-29T10:07:17.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.666+0000 7f1cbea00700 1 -- 172.21.15.12:0/3353565789 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb80045b0 con 0x7f1cc80cbb30 2024-03-29T10:07:17.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.666+0000 7f1ccdc00700 1 -- 172.21.15.12:0/3353565789 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc8156020 con 0x7f1cc80cbb30 2024-03-29T10:07:17.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.667+0000 7f1ccdc00700 1 -- 172.21.15.12:0/3353565789 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc8067720 con 0x7f1cc80cbb30 2024-03-29T10:07:17.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.668+0000 7f1cbea00700 1 -- 172.21.15.12:0/3353565789 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1cb8021030 con 0x7f1cc80cbb30 2024-03-29T10:07:17.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.668+0000 7f1cbea00700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca806ada0 0x7f1ca806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:17.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.669+0000 7f1cbea00700 1 -- 172.21.15.12:0/3353565789 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1cb808a670 con 0x7f1cc80cbb30 2024-03-29T10:07:17.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.669+0000 7f1cc6000700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca806ada0 0x7f1ca806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:17.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.669+0000 7f1cc6000700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca806ada0 0x7f1ca806d260 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1cc81f75e0 tx=0x7f1cb4009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:17.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.674+0000 7f1cbea00700 1 -- 172.21.15.12:0/3353565789 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1cb8055730 con 0x7f1cc80cbb30 2024-03-29T10:07:17.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.918+0000 7f1ccdc00700 1 -- 172.21.15.12:0/3353565789 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cc81594a0 con 0x7f1ca806ada0 2024-03-29T10:07:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.920+0000 7f1cbea00700 1 -- 172.21.15.12:0/3353565789 <== mgr.5210 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f1cb40015d0 con 0x7f1ca806ada0 2024-03-29T10:07:17.919 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:07:17.920 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:07:17.920 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T10:07:17.920 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:07:17.920 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:07:17.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.922+0000 7f1ccdc00700 1 -- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca806ada0 msgr2=0x7f1ca806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:17.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.922+0000 7f1ccdc00700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca806ada0 0x7f1ca806d260 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1cc81f75e0 tx=0x7f1cb4009040 comp rx=0 tx=0).stop 2024-03-29T10:07:17.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.922+0000 7f1ccdc00700 1 -- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc80cbb30 msgr2=0x7f1cc8072510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.922+0000 7f1ccdc00700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc80cbb30 0x7f1cc8072510 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8015000 tx=0x7f1cb8004750 comp rx=0 tx=0).stop 2024-03-29T10:07:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.922+0000 7f1ccdc00700 1 -- 172.21.15.12:0/3353565789 shutdown_connections 2024-03-29T10:07:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.922+0000 7f1ccdc00700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca806ada0 0x7f1ca806d260 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.922+0000 7f1ccdc00700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc80cd4a0 0x7f1cc80c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.922+0000 7f1ccdc00700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc80cbb30 0x7f1cc8072510 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.922+0000 7f1ccdc00700 1 --2- 172.21.15.12:0/3353565789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc80cd020 0x7f1cc8155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.922+0000 7f1ccdc00700 1 -- 172.21.15.12:0/3353565789 >> 172.21.15.12:0/3353565789 conn(0x7f1cc80b0d70 msgr2=0x7f1cc80b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.923+0000 7f1ccdc00700 1 -- 172.21.15.12:0/3353565789 shutdown_connections 2024-03-29T10:07:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:17.923+0000 7f1ccdc00700 1 -- 172.21.15.12:0/3353565789 wait complete. 2024-03-29T10:07:17.941 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T10:07:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.142+0000 7f31a6200700 1 Processor -- start 2024-03-29T10:07:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.142+0000 7f31a6200700 1 -- start start 2024-03-29T10:07:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.143+0000 7f31a6200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f31a00d2540 0x7f31a01687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.143+0000 7f31a6200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a0161420 con 0x7f31a00d16a0 2024-03-29T10:07:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.143+0000 7f31a6200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a0161700 con 0x7f31a015d9d0 2024-03-29T10:07:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.143+0000 7f31a6200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a00d4340 con 0x7f31a00d2540 2024-03-29T10:07:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.144+0000 7f319ea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f31a00d16a0 0x7f31a00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35944/0 (socket says 172.21.15.12:35944) 2024-03-29T10:07:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.144+0000 7f319ea00700 1 -- 172.21.15.12:0/2657715773 learned_addr learned my addr 172.21.15.12:0/2657715773 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.144+0000 7f319fe00700 1 --2- 172.21.15.12:0/2657715773 >> v2:172.21.15.159:3300/0 conn(0x7f31a00d2540 0x7f31a01687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.144+0000 7f319e000700 1 -- 172.21.15.12:0/2657715773 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573771213 0 0) 0x7f31a0161420 con 0x7f31a00d16a0 2024-03-29T10:07:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.144+0000 7f319e000700 1 -- 172.21.15.12:0/2657715773 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31840035b0 con 0x7f31a00d16a0 2024-03-29T10:07:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.144+0000 7f319fe00700 1 -- 172.21.15.12:0/2657715773 >> v1:172.21.15.159:6789/0 conn(0x7f31a015d9d0 legacy=0x7f31a00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.144+0000 7f319fe00700 1 -- 172.21.15.12:0/2657715773 >> v1:172.21.15.12:6789/0 conn(0x7f31a00d16a0 legacy=0x7f31a00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.144+0000 7f319fe00700 1 -- 172.21.15.12:0/2657715773 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a01619e0 con 0x7f31a00d2540 2024-03-29T10:07:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.145+0000 7f319fe00700 1 --2- 172.21.15.12:0/2657715773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31a00d2540 0x7f31a01687b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3198009d90 tx=0x7f319800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f5ab7c5dc44c93b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.145+0000 7f319e000700 1 -- 172.21.15.12:0/2657715773 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319800e070 con 0x7f31a00d2540 2024-03-29T10:07:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.145+0000 7f319e000700 1 -- 172.21.15.12:0/2657715773 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319800bb20 con 0x7f31a00d2540 2024-03-29T10:07:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.145+0000 7f319e000700 1 -- 172.21.15.12:0/2657715773 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319800be90 con 0x7f31a00d2540 2024-03-29T10:07:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.145+0000 7f31a6200700 1 -- 172.21.15.12:0/2657715773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31a00d2540 msgr2=0x7f31a01687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.145+0000 7f31a6200700 1 --2- 172.21.15.12:0/2657715773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31a00d2540 0x7f31a01687b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3198009d90 tx=0x7f319800b330 comp rx=0 tx=0).stop 2024-03-29T10:07:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.146+0000 7f31a6200700 1 -- 172.21.15.12:0/2657715773 shutdown_connections 2024-03-29T10:07:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.146+0000 7f31a6200700 1 --2- 172.21.15.12:0/2657715773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31a00d2540 0x7f31a01687b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.146+0000 7f31a6200700 1 -- 172.21.15.12:0/2657715773 >> 172.21.15.12:0/2657715773 conn(0x7f31a008aff0 msgr2=0x7f31a00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.146+0000 7f31a6200700 1 -- 172.21.15.12:0/2657715773 shutdown_connections 2024-03-29T10:07:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.146+0000 7f31a6200700 1 -- 172.21.15.12:0/2657715773 wait complete. 2024-03-29T10:07:18.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.147+0000 7f31a6200700 1 Processor -- start 2024-03-29T10:07:18.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.148+0000 7f31a6200700 1 -- start start 2024-03-29T10:07:18.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.148+0000 7f31a6200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31a00d16a0 0x7f31a015c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:18.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.148+0000 7f31a6200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31a00d2540 0x7f31a006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:18.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.148+0000 7f31a6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31a015d9d0 0x7f31a006c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:18.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.148+0000 7f31a6200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a01e93d0 con 0x7f31a015d9d0 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f31a6200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a01e8370 con 0x7f31a00d16a0 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f31a6200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a01e8610 con 0x7f31a00d2540 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31a015d9d0 0x7f31a006c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31a015d9d0 0x7f31a006c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47640/0 (socket says 172.21.15.12:47640) 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319fe00700 1 -- 172.21.15.12:0/4065665284 learned_addr learned my addr 172.21.15.12:0/4065665284 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319f400700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31a00d16a0 0x7f31a015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319ea00700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31a00d2540 0x7f31a006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319fe00700 1 -- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31a00d2540 msgr2=0x7f31a006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319fe00700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31a00d2540 0x7f31a006bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319fe00700 1 -- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31a00d16a0 msgr2=0x7f31a015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319fe00700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31a00d16a0 0x7f31a015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319fe00700 1 -- 172.21.15.12:0/4065665284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3198009a70 con 0x7f31a015d9d0 2024-03-29T10:07:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319ea00700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31a00d2540 0x7f31a006bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:07:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.149+0000 7f319fe00700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31a015d9d0 0x7f31a006c120 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3198014040 tx=0x7f3198004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.150+0000 7f3197400700 1 -- 172.21.15.12:0/4065665284 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31980098f0 con 0x7f31a015d9d0 2024-03-29T10:07:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.150+0000 7f3197400700 1 -- 172.21.15.12:0/4065665284 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3198020070 con 0x7f31a015d9d0 2024-03-29T10:07:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.150+0000 7f31a6200700 1 -- 172.21.15.12:0/4065665284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a01e88f0 con 0x7f31a015d9d0 2024-03-29T10:07:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.150+0000 7f3197400700 1 -- 172.21.15.12:0/4065665284 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31980276a0 con 0x7f31a015d9d0 2024-03-29T10:07:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.150+0000 7f31a6200700 1 -- 172.21.15.12:0/4065665284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a01e8eb0 con 0x7f31a015d9d0 2024-03-29T10:07:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.151+0000 7f31a6200700 1 -- 172.21.15.12:0/4065665284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a00d1ca0 con 0x7f31a015d9d0 2024-03-29T10:07:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.152+0000 7f3197400700 1 -- 172.21.15.12:0/4065665284 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3198012430 con 0x7f31a015d9d0 2024-03-29T10:07:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.153+0000 7f3197400700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f318006adb0 0x7f318006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.153+0000 7f3197400700 1 -- 172.21.15.12:0/4065665284 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3198022070 con 0x7f31a015d9d0 2024-03-29T10:07:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.153+0000 7f319f400700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f318006adb0 0x7f318006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.154+0000 7f319f400700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f318006adb0 0x7f318006d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f31a015d090 tx=0x7f3190009040 comp rx=0 tx=0).ready entity=mgr.5210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:18.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.157+0000 7f3197400700 1 -- 172.21.15.12:0/4065665284 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3198056af0 con 0x7f31a015d9d0 2024-03-29T10:07:18.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:18.437+0000 7f31a6200700 1 -- 172.21.15.12:0/4065665284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f31a0067720 con 0x7f31a015d9d0 2024-03-29T10:07:19.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.221+0000 7f3197400700 1 -- 172.21.15.12:0/4065665284 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 19) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f3198056ec0 con 0x7f31a015d9d0 2024-03-29T10:07:19.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.222+0000 7f3197400700 1 -- 172.21.15.12:0/4065665284 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f319806ec30 con 0x7f31a015d9d0 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 -- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f318006adb0 msgr2=0x7f318006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f318006adb0 0x7f318006d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f31a015d090 tx=0x7f3190009040 comp rx=0 tx=0).stop 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 -- 172.21.15.12:0/4065665284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31a015d9d0 msgr2=0x7f31a006c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31a015d9d0 0x7f31a006c120 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3198014040 tx=0x7f3198004ad0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 -- 172.21.15.12:0/4065665284 shutdown_connections 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f318006adb0 0x7f318006d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31a015d9d0 0x7f31a006c120 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31a00d16a0 0x7f31a015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 --2- 172.21.15.12:0/4065665284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31a00d2540 0x7f31a006bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 -- 172.21.15.12:0/4065665284 >> 172.21.15.12:0/4065665284 conn(0x7f31a008aff0 msgr2=0x7f31a015e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 -- 172.21.15.12:0/4065665284 shutdown_connections 2024-03-29T10:07:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.226+0000 7f31a6200700 1 -- 172.21.15.12:0/4065665284 wait complete. 2024-03-29T10:07:19.248 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T10:07:19.445 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:19.445+0000 7ff553f9f240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:07:19.461 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:19.462+0000 7f8f19a4f240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:07:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.467+0000 7f0ba9600700 1 Processor -- start 2024-03-29T10:07:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.467+0000 7f0ba9600700 1 -- start start 2024-03-29T10:07:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.468+0000 7f0ba9600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ba40cbb30 0x7f0ba40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.468+0000 7f0ba9600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba40cc450 con 0x7f0ba40cd020 2024-03-29T10:07:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba9600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba40cdae0 con 0x7f0ba40cd480 2024-03-29T10:07:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba9600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba40cd8f0 con 0x7f0ba40cbb30 2024-03-29T10:07:19.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba2000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0ba40cd020 0x7f0ba40b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37936/0 (socket says 172.21.15.12:37936) 2024-03-29T10:07:19.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ba40cbb30 0x7f0ba40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:19.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba2000700 1 -- 172.21.15.12:0/3957518291 learned_addr learned my addr 172.21.15.12:0/3957518291 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba1600700 1 -- 172.21.15.12:0/3957518291 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792062869 0 0) 0x7f0ba40cc450 con 0x7f0ba40cd020 2024-03-29T10:07:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba1600700 1 -- 172.21.15.12:0/3957518291 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b880035b0 con 0x7f0ba40cd020 2024-03-29T10:07:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba2a00700 1 -- 172.21.15.12:0/3957518291 >> v1:172.21.15.159:6789/0 conn(0x7f0ba40cd480 legacy=0x7f0ba406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba1600700 1 -- 172.21.15.12:0/3957518291 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195606250 0 0) 0x7f0ba40cdae0 con 0x7f0ba40cd480 2024-03-29T10:07:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba2a00700 1 -- 172.21.15.12:0/3957518291 >> v1:172.21.15.12:6789/0 conn(0x7f0ba40cd020 legacy=0x7f0ba40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.469+0000 7f0ba2a00700 1 -- 172.21.15.12:0/3957518291 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba40cddc0 con 0x7f0ba40cbb30 2024-03-29T10:07:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.470+0000 7f0ba2a00700 1 --2- 172.21.15.12:0/3957518291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cbb30 0x7f0ba40cbf10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009840 tx=0x7f0b94009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8010122b3e2070d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.470+0000 7f0ba1600700 1 -- 172.21.15.12:0/3957518291 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94011830 con 0x7f0ba40cbb30 2024-03-29T10:07:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.470+0000 7f0ba1600700 1 -- 172.21.15.12:0/3957518291 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b94011990 con 0x7f0ba40cbb30 2024-03-29T10:07:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.470+0000 7f0ba1600700 1 -- 172.21.15.12:0/3957518291 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94011d20 con 0x7f0ba40cbb30 2024-03-29T10:07:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.470+0000 7f0ba9600700 1 -- 172.21.15.12:0/3957518291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cbb30 msgr2=0x7f0ba40cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.471+0000 7f0ba9600700 1 --2- 172.21.15.12:0/3957518291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cbb30 0x7f0ba40cbf10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009840 tx=0x7f0b94009b50 comp rx=0 tx=0).stop 2024-03-29T10:07:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.471+0000 7f0ba9600700 1 -- 172.21.15.12:0/3957518291 shutdown_connections 2024-03-29T10:07:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.471+0000 7f0ba9600700 1 --2- 172.21.15.12:0/3957518291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cbb30 0x7f0ba40cbf10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.471+0000 7f0ba9600700 1 -- 172.21.15.12:0/3957518291 >> 172.21.15.12:0/3957518291 conn(0x7f0ba40b0d70 msgr2=0x7f0ba40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.471+0000 7f0ba9600700 1 -- 172.21.15.12:0/3957518291 shutdown_connections 2024-03-29T10:07:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.471+0000 7f0ba9600700 1 -- 172.21.15.12:0/3957518291 wait complete. 2024-03-29T10:07:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.473+0000 7f0ba9600700 1 Processor -- start 2024-03-29T10:07:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.473+0000 7f0ba9600700 1 -- start start 2024-03-29T10:07:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba9600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba40cbb30 0x7f0ba4071c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba9600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ba40cd020 0x7f0ba40ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cd480 0x7f0ba41f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba9600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba415ce10 con 0x7f0ba40cd480 2024-03-29T10:07:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba9600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40ca2d0 con 0x7f0ba40cd020 2024-03-29T10:07:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba9600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40ca5b0 con 0x7f0ba40cbb30 2024-03-29T10:07:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cd480 0x7f0ba41f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba2000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ba40cd020 0x7f0ba40ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cd480 0x7f0ba41f5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48460/0 (socket says 172.21.15.12:48460) 2024-03-29T10:07:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba3400700 1 -- 172.21.15.12:0/956613503 learned_addr learned my addr 172.21.15.12:0/956613503 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba2a00700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba40cbb30 0x7f0ba4071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba3400700 1 -- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba40cbb30 msgr2=0x7f0ba4071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba3400700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba40cbb30 0x7f0ba4071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba3400700 1 -- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ba40cd020 msgr2=0x7f0ba40ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.474+0000 7f0ba3400700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ba40cd020 0x7f0ba40ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.475+0000 7f0ba3400700 1 -- 172.21.15.12:0/956613503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b940094a0 con 0x7f0ba40cd480 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.475+0000 7f0ba3400700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cd480 0x7f0ba41f5d30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c001560 tx=0x7f0b9c00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.475+0000 7f0b9aa00700 1 -- 172.21.15.12:0/956613503 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c003e30 con 0x7f0ba40cd480 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.475+0000 7f0b9aa00700 1 -- 172.21.15.12:0/956613503 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9c005070 con 0x7f0ba40cd480 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.475+0000 7f0ba9600700 1 -- 172.21.15.12:0/956613503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba40ca890 con 0x7f0ba40cd480 2024-03-29T10:07:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.475+0000 7f0b9aa00700 1 -- 172.21.15.12:0/956613503 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c005420 con 0x7f0ba40cd480 2024-03-29T10:07:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.475+0000 7f0ba9600700 1 -- 172.21.15.12:0/956613503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba4072150 con 0x7f0ba40cd480 2024-03-29T10:07:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.476+0000 7f0ba9600700 1 -- 172.21.15.12:0/956613503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba4067720 con 0x7f0ba40cd480 2024-03-29T10:07:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.477+0000 7f0b9aa00700 1 -- 172.21.15.12:0/956613503 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f0b9c005580 con 0x7f0ba40cd480 2024-03-29T10:07:19.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.478+0000 7f0b9aa00700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406ae00 0x7f0b8406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.478+0000 7f0b9aa00700 1 -- 172.21.15.12:0/956613503 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b9c088b30 con 0x7f0ba40cd480 2024-03-29T10:07:19.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.478+0000 7f0ba2a00700 1 -- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406ae00 msgr2=0x7f0b8406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:07:19.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.478+0000 7f0ba2a00700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406ae00 0x7f0b8406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:07:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.482+0000 7f0b9aa00700 1 -- 172.21.15.12:0/956613503 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b9c053bf0 con 0x7f0ba40cd480 2024-03-29T10:07:19.631 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:19.631+0000 7ff553f9f240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:07:19.669 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:19.669+0000 7f8f19a4f240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:07:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.679+0000 7f0ba2a00700 1 -- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406ae00 msgr2=0x7f0b8406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:07:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.679+0000 7f0ba2a00700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406ae00 0x7f0b8406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:07:19.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.716+0000 7f0ba9600700 1 -- 172.21.15.12:0/956613503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f0ba40ce2e0 con 0x7f0ba40cd480 2024-03-29T10:07:19.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.722+0000 7f0b9aa00700 1 -- 172.21.15.12:0/956613503 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0b9c017070 con 0x7f0ba40cd480 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 -- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406ae00 msgr2=0x7f0b8406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406ae00 0x7f0b8406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 -- 172.21.15.12:0/956613503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cd480 msgr2=0x7f0ba41f5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cd480 0x7f0ba41f5d30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c001560 tx=0x7f0b9c00cb50 comp rx=0 tx=0).stop 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 -- 172.21.15.12:0/956613503 shutdown_connections 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406ae00 0x7f0b8406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba40cd480 0x7f0ba41f5d30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba40cbb30 0x7f0ba4071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 --2- 172.21.15.12:0/956613503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ba40cd020 0x7f0ba40ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 -- 172.21.15.12:0/956613503 >> 172.21.15.12:0/956613503 conn(0x7f0ba40b0d70 msgr2=0x7f0ba415c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:19.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 -- 172.21.15.12:0/956613503 shutdown_connections 2024-03-29T10:07:19.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.725+0000 7f0ba9600700 1 -- 172.21.15.12:0/956613503 wait complete. 2024-03-29T10:07:19.743 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T10:07:19.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.947+0000 7ff6e3200700 1 Processor -- start 2024-03-29T10:07:19.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.947+0000 7ff6e3200700 1 -- start start 2024-03-29T10:07:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.948+0000 7ff6e3200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6dc1611b0 0x7ff6dc15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.948+0000 7ff6e3200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff6dc15f4d0 0x7ff6dc0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.948+0000 7ff6e3200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6dc0d21f0 0x7ff6dc0cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.948+0000 7ff6e3200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6dc0cf180 con 0x7ff6dc1611b0 2024-03-29T10:07:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.948+0000 7ff6e3200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6dc0ce270 con 0x7ff6dc15f4d0 2024-03-29T10:07:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.948+0000 7ff6e3200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc15b600 con 0x7ff6dc0d21f0 2024-03-29T10:07:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6dbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6dc1611b0 0x7ff6dc15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6db400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff6dc15f4d0 0x7ff6dc0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6dbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6dc1611b0 0x7ff6dc15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48476/0 (socket says 172.21.15.12:48476) 2024-03-29T10:07:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6dbe00700 1 -- 172.21.15.12:0/3488008414 learned_addr learned my addr 172.21.15.12:0/3488008414 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6e1200700 1 --2- 172.21.15.12:0/3488008414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6dc0d21f0 0x7ff6dc0cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6dbe00700 1 -- 172.21.15.12:0/3488008414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6dc0d21f0 msgr2=0x7ff6dc0cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6dbe00700 1 --2- 172.21.15.12:0/3488008414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6dc0d21f0 0x7ff6dc0cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6dbe00700 1 -- 172.21.15.12:0/3488008414 >> v2:172.21.15.159:3300/0 conn(0x7ff6dc15f4d0 msgr2=0x7ff6dc0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6dbe00700 1 --2- 172.21.15.12:0/3488008414 >> v2:172.21.15.159:3300/0 conn(0x7ff6dc15f4d0 0x7ff6dc0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6dbe00700 1 -- 172.21.15.12:0/3488008414 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc0d2760 con 0x7ff6dc1611b0 2024-03-29T10:07:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6db400700 1 --2- 172.21.15.12:0/3488008414 >> v2:172.21.15.159:3300/0 conn(0x7ff6dc15f4d0 0x7ff6dc0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:07:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.949+0000 7ff6dbe00700 1 --2- 172.21.15.12:0/3488008414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 0x7ff6dc15ef90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc08c030 tx=0x7ff6cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd42b66658ab37d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:19.951 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:19.948+0000 7ff553f9f240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:07:19.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6daa00700 1 -- 172.21.15.12:0/3488008414 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc011830 con 0x7ff6dc1611b0 2024-03-29T10:07:19.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6daa00700 1 -- 172.21.15.12:0/3488008414 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc011990 con 0x7ff6dc1611b0 2024-03-29T10:07:19.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6daa00700 1 -- 172.21.15.12:0/3488008414 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc011d20 con 0x7ff6dc1611b0 2024-03-29T10:07:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6e3200700 1 -- 172.21.15.12:0/3488008414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 msgr2=0x7ff6dc15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6e3200700 1 --2- 172.21.15.12:0/3488008414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 0x7ff6dc15ef90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc08c030 tx=0x7ff6cc009bd0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6e3200700 1 -- 172.21.15.12:0/3488008414 shutdown_connections 2024-03-29T10:07:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6e3200700 1 --2- 172.21.15.12:0/3488008414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6dc0d21f0 0x7ff6dc0cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6e3200700 1 --2- 172.21.15.12:0/3488008414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 0x7ff6dc15ef90 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6e3200700 1 --2- 172.21.15.12:0/3488008414 >> v2:172.21.15.159:3300/0 conn(0x7ff6dc15f4d0 0x7ff6dc0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6e3200700 1 -- 172.21.15.12:0/3488008414 >> 172.21.15.12:0/3488008414 conn(0x7ff6dc08aff0 msgr2=0x7ff6dc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.950+0000 7ff6e3200700 1 -- 172.21.15.12:0/3488008414 shutdown_connections 2024-03-29T10:07:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.951+0000 7ff6e3200700 1 -- 172.21.15.12:0/3488008414 wait complete. 2024-03-29T10:07:19.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.952+0000 7ff6e3200700 1 Processor -- start 2024-03-29T10:07:19.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.952+0000 7ff6e3200700 1 -- start start 2024-03-29T10:07:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.953+0000 7ff6e3200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6dc0d21f0 0x7ff6dc1f7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.953+0000 7ff6e3200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6dc15f4d0 0x7ff6dc0a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.953+0000 7ff6e3200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 0x7ff6dc09ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.953+0000 7ff6e3200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc1572b0 con 0x7ff6dc1611b0 2024-03-29T10:07:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.953+0000 7ff6e3200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc09d220 con 0x7ff6dc15f4d0 2024-03-29T10:07:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.953+0000 7ff6e3200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc09d500 con 0x7ff6dc0d21f0 2024-03-29T10:07:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6e1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 0x7ff6dc09ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6e1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 0x7ff6dc09ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48492/0 (socket says 172.21.15.12:48492) 2024-03-29T10:07:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6e1200700 1 -- 172.21.15.12:0/2507827762 learned_addr learned my addr 172.21.15.12:0/2507827762 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6dbe00700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6dc0d21f0 0x7ff6dc1f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6db400700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6dc15f4d0 0x7ff6dc0a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6e1200700 1 -- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6dc0d21f0 msgr2=0x7ff6dc1f7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6e1200700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6dc0d21f0 0x7ff6dc1f7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6e1200700 1 -- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6dc15f4d0 msgr2=0x7ff6dc0a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6e1200700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6dc15f4d0 0x7ff6dc0a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6e1200700 1 -- 172.21.15.12:0/2507827762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c80094a0 con 0x7ff6dc1611b0 2024-03-29T10:07:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.954+0000 7ff6e1200700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 0x7ff6dc09ebd0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400eed0 tx=0x7ff6d400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:19.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.955+0000 7ff6d3e00700 1 -- 172.21.15.12:0/2507827762 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4010040 con 0x7ff6dc1611b0 2024-03-29T10:07:19.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.955+0000 7ff6d3e00700 1 -- 172.21.15.12:0/2507827762 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d400bc80 con 0x7ff6dc1611b0 2024-03-29T10:07:19.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.955+0000 7ff6e3200700 1 -- 172.21.15.12:0/2507827762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6cc009550 con 0x7ff6dc1611b0 2024-03-29T10:07:19.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.955+0000 7ff6d3e00700 1 -- 172.21.15.12:0/2507827762 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4009e40 con 0x7ff6dc1611b0 2024-03-29T10:07:19.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.955+0000 7ff6e3200700 1 -- 172.21.15.12:0/2507827762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc09dac0 con 0x7ff6dc1611b0 2024-03-29T10:07:19.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.956+0000 7ff6e3200700 1 -- 172.21.15.12:0/2507827762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc0a1270 con 0x7ff6dc1611b0 2024-03-29T10:07:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.957+0000 7ff6d3e00700 1 -- 172.21.15.12:0/2507827762 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) v1 ==== 89256+0+0 (secure 0 0 0) 0x7ff6d4014400 con 0x7ff6dc1611b0 2024-03-29T10:07:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.958+0000 7ff6d3e00700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6bc06ae00 0x7ff6bc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.958+0000 7ff6d3e00700 1 -- 172.21.15.12:0/2507827762 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6d40899e0 con 0x7ff6dc1611b0 2024-03-29T10:07:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.958+0000 7ff6dbe00700 1 -- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6bc06ae00 msgr2=0x7ff6bc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:07:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.958+0000 7ff6dbe00700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6bc06ae00 0x7ff6bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:07:19.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:19.962+0000 7ff6d3e00700 1 -- 172.21.15.12:0/2507827762 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6d4054900 con 0x7ff6dc1611b0 2024-03-29T10:07:20.017 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:20.018+0000 7f8f19a4f240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:07:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.159+0000 7ff6dbe00700 1 -- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6bc06ae00 msgr2=0x7ff6bc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:07:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.159+0000 7ff6dbe00700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6bc06ae00 0x7ff6bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:07:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.178+0000 7ff6e3200700 1 -- 172.21.15.12:0/2507827762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7ff6dc067720 con 0x7ff6dc1611b0 2024-03-29T10:07:20.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.184+0000 7ff6d3e00700 1 -- 172.21.15.12:0/2507827762 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff6d40180c0 con 0x7ff6dc1611b0 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 -- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6bc06ae00 msgr2=0x7ff6bc06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6bc06ae00 0x7ff6bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 -- 172.21.15.12:0/2507827762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 msgr2=0x7ff6dc09ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 0x7ff6dc09ebd0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400eed0 tx=0x7ff6d400b390 comp rx=0 tx=0).stop 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 -- 172.21.15.12:0/2507827762 shutdown_connections 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6bc06ae00 0x7ff6bc06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6dc1611b0 0x7ff6dc09ebd0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6dc0d21f0 0x7ff6dc1f7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 --2- 172.21.15.12:0/2507827762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6dc15f4d0 0x7ff6dc0a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 -- 172.21.15.12:0/2507827762 >> 172.21.15.12:0/2507827762 conn(0x7ff6dc08aff0 msgr2=0x7ff6dc161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 -- 172.21.15.12:0/2507827762 shutdown_connections 2024-03-29T10:07:20.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.187+0000 7ff6e3200700 1 -- 172.21.15.12:0/2507827762 wait complete. 2024-03-29T10:07:20.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:20.205 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-29T10:07:20.260 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-03-29T10:07:20.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.466+0000 7f1bb0a00700 1 Processor -- start 2024-03-29T10:07:20.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.466+0000 7f1bb0a00700 1 -- start start 2024-03-29T10:07:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.468+0000 7f1bb0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac15dad0 0x7f1bac0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.468+0000 7f1bb0a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bac15f820 con 0x7f1bac0d1630 2024-03-29T10:07:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.468+0000 7f1bb0a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bac0cf420 con 0x7f1bac0d21a0 2024-03-29T10:07:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.468+0000 7f1bb0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac0cfcc0 con 0x7f1bac15dad0 2024-03-29T10:07:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.468+0000 7f1ba9400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1bac0d1630 0x7f1bac0d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37950/0 (socket says 172.21.15.12:37950) 2024-03-29T10:07:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.468+0000 7f1ba9400700 1 -- 172.21.15.12:0/3390956438 learned_addr learned my addr 172.21.15.12:0/3390956438 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.469+0000 7f1ba9e00700 1 --2- 172.21.15.12:0/3390956438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac15dad0 0x7f1bac0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.469+0000 7f1ba8a00700 1 -- 172.21.15.12:0/3390956438 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647322659 0 0) 0x7f1bac15f820 con 0x7f1bac0d1630 2024-03-29T10:07:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.469+0000 7f1ba9e00700 1 -- 172.21.15.12:0/3390956438 >> v1:172.21.15.159:6789/0 conn(0x7f1bac0d21a0 legacy=0x7f1bac15fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.469+0000 7f1ba9e00700 1 -- 172.21.15.12:0/3390956438 >> v1:172.21.15.12:6789/0 conn(0x7f1bac0d1630 legacy=0x7f1bac0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.469+0000 7f1ba9e00700 1 -- 172.21.15.12:0/3390956438 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac0cf700 con 0x7f1bac15dad0 2024-03-29T10:07:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.469+0000 7f1ba9e00700 1 --2- 172.21.15.12:0/3390956438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac15dad0 0x7f1bac0ced80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0097f0 tx=0x7f1b9c009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebd247034522d3e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.470+0000 7f1ba8a00700 1 -- 172.21.15.12:0/3390956438 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c004230 con 0x7f1bac15dad0 2024-03-29T10:07:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.470+0000 7f1ba8a00700 1 -- 172.21.15.12:0/3390956438 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9c004390 con 0x7f1bac15dad0 2024-03-29T10:07:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.470+0000 7f1ba8a00700 1 -- 172.21.15.12:0/3390956438 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c0036a0 con 0x7f1bac15dad0 2024-03-29T10:07:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.470+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3390956438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac15dad0 msgr2=0x7f1bac0ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.470+0000 7f1bb0a00700 1 --2- 172.21.15.12:0/3390956438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac15dad0 0x7f1bac0ced80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0097f0 tx=0x7f1b9c009b00 comp rx=0 tx=0).stop 2024-03-29T10:07:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.470+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3390956438 shutdown_connections 2024-03-29T10:07:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.470+0000 7f1bb0a00700 1 --2- 172.21.15.12:0/3390956438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac15dad0 0x7f1bac0ced80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.470+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3390956438 >> 172.21.15.12:0/3390956438 conn(0x7f1bac08aff0 msgr2=0x7f1bac0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.470+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3390956438 shutdown_connections 2024-03-29T10:07:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.471+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3390956438 wait complete. 2024-03-29T10:07:20.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.472+0000 7f1bb0a00700 1 Processor -- start 2024-03-29T10:07:20.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.472+0000 7f1bb0a00700 1 -- start start 2024-03-29T10:07:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.473+0000 7f1bb0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac0d1630 0x7f1bac0b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.473+0000 7f1bb0a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1bac0d21a0 0x7f1bac06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.473+0000 7f1bb0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac15dad0 0x7f1bac06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.473+0000 7f1bb0a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac0c97a0 con 0x7f1bac15dad0 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.473+0000 7f1bb0a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac0b8320 con 0x7f1bac0d21a0 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.473+0000 7f1bb0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac0b7560 con 0x7f1bac0d1630 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1baa800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac15dad0 0x7f1bac06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1baa800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac15dad0 0x7f1bac06d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48494/0 (socket says 172.21.15.12:48494) 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1baa800700 1 -- 172.21.15.12:0/3950230966 learned_addr learned my addr 172.21.15.12:0/3950230966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1ba9e00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac0d1630 0x7f1bac0b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1ba9400700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1bac0d21a0 0x7f1bac06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1baa800700 1 -- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac0d1630 msgr2=0x7f1bac0b8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1baa800700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac0d1630 0x7f1bac0b8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1baa800700 1 -- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1bac0d21a0 msgr2=0x7f1bac06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1baa800700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1bac0d21a0 0x7f1bac06cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1baa800700 1 -- 172.21.15.12:0/3950230966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c0094a0 con 0x7f1bac15dad0 2024-03-29T10:07:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.474+0000 7f1baa800700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac15dad0 0x7f1bac06d1d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400cb20 tx=0x7f1ba400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.475+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4003e30 con 0x7f1bac15dad0 2024-03-29T10:07:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.475+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3950230966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bac0b7840 con 0x7f1bac15dad0 2024-03-29T10:07:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.475+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba40020a0 con 0x7f1bac15dad0 2024-03-29T10:07:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.475+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba40052b0 con 0x7f1bac15dad0 2024-03-29T10:07:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.475+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3950230966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bac0b7e00 con 0x7f1bac15dad0 2024-03-29T10:07:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.476+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3950230966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bac067720 con 0x7f1bac15dad0 2024-03-29T10:07:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.477+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f1ba4005410 con 0x7f1bac15dad0 2024-03-29T10:07:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.478+0000 7f1ba2000700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.478+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ba4088720 con 0x7f1bac15dad0 2024-03-29T10:07:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.478+0000 7f1ba9e00700 1 -- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 msgr2=0x7f1b8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:07:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.478+0000 7f1ba9e00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:07:20.479 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:20.480+0000 7ff553f9f240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:07:20.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.482+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ba4053760 con 0x7f1bac15dad0 2024-03-29T10:07:20.586 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:20.586+0000 7f8f19a4f240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:07:20.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.679+0000 7f1ba9e00700 1 -- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 msgr2=0x7f1b8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:07:20.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.679+0000 7f1ba9e00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:07:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:20.726+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3950230966 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bac1603c0 con 0x7f1b8c06ae00 2024-03-29T10:07:20.734 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:20.734+0000 7ff553f9f240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:07:20.850 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:20.850+0000 7f8f19a4f240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:07:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:21.080+0000 7f1ba9e00700 1 -- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 msgr2=0x7f1b8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:07:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:21.080+0000 7f1ba9e00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T10:07:21.189 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:21.189+0000 7ff553f9f240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:07:21.324 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:21.324+0000 7f8f19a4f240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:07:21.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:21.881+0000 7f1ba9e00700 1 -- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 msgr2=0x7f1b8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:07:21.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:21.881+0000 7f1ba9e00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T10:07:21.894 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:21.894+0000 7ff553f9f240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:07:21.986 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:21.986+0000 7ff553f9f240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:07:22.062 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:22.062+0000 7f8f19a4f240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:07:22.145 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:22.145+0000 7ff553f9f240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:07:22.158 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:22.159+0000 7f8f19a4f240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:07:22.325 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:22.326+0000 7f8f19a4f240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:07:22.359 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:22.359+0000 7ff553f9f240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:07:22.449 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:22.449+0000 7ff553f9f240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:07:22.542 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:22.542+0000 7f8f19a4f240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:07:22.546 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:22.546+0000 7ff553f9f240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:07:22.637 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:22.637+0000 7f8f19a4f240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:07:22.641 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:22.641+0000 7ff553f9f240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:07:22.739 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:22.739+0000 7f8f19a4f240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:07:22.739 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:22.739+0000 7ff553f9f240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:07:22.840 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:22.841+0000 7f8f19a4f240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:07:22.942 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:22.943+0000 7f8f19a4f240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:07:22.950 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:22.951+0000 7ff553f9f240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:07:23.165 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:23.165+0000 7f8f19a4f240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:07:23.207 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:23.207+0000 7ff553f9f240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:07:23.394 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:23.394+0000 7ff553f9f240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:07:23.426 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:23.426+0000 7f8f19a4f240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:07:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:23.483+0000 7f1ba9e00700 1 -- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 msgr2=0x7f1b8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:07:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:23.483+0000 7f1ba9e00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T10:07:23.485 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:23.485+0000 7ff553f9f240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:07:23.622 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:23.622+0000 7f8f19a4f240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:07:23.718 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:23.718+0000 7f8f19a4f240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:07:24.217 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:24.217+0000 7ff553f9f240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:07:24.427 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:24.427+0000 7ff553f9f240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:07:24.476 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:24.477+0000 7f8f19a4f240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:07:24.631 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:24.631+0000 7ff553f9f240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:07:24.695 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:24.695+0000 7f8f19a4f240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:07:24.911 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:24.912+0000 7f8f19a4f240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:07:25.109 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:25.109+0000 7ff553f9f240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:07:25.310 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:25.310+0000 7ff553f9f240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:07:25.407 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:25.407+0000 7f8f19a4f240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:07:25.621 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:25.621+0000 7f8f19a4f240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:07:25.747 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:25.747+0000 7ff553f9f240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:07:25.856 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:25.856+0000 7ff553f9f240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:07:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:25.868+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 20) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f1ba4053b30 con 0x7f1bac15dad0 2024-03-29T10:07:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:25.869+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 msgr2=0x7f1b8c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:07:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:25.869+0000 7f1ba2000700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:25.925 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:07:25.929 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:07:26.046 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:26.046+0000 7ff4c0200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:07:26.046 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:26.046+0000 7ff4c0200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:07:26.046 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:26.046+0000 7ff4c0200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:07:26.046 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:26.046+0000 7ff4c0200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:07:26.046 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:26.046+0000 7ff4c0200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:07:26.046 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:26.046+0000 7ff4bc600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:07:26.046 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:26.046+0000 7ff4bc600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:07:26.047 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:26.046+0000 7ff4bc600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:07:26.047 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:26.046+0000 7ff4bc600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:07:26.047 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:07:26.046+0000 7ff4bc600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:07:26.093 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:26.093+0000 7f8f19a4f240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:07:26.207 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:07:26.207+0000 7f8f19a4f240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:07:26.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.873+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 21) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f1ba4033d00 con 0x7f1bac15dad0 2024-03-29T10:07:26.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.874+0000 7f1ba2000700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:26.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.874+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b8c035990 con 0x7f1b8c06ae00 2024-03-29T10:07:26.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.876+0000 7f1ba9e00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:26.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.878+0000 7f1ba9e00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0097c0 tx=0x7f1b9c008030 comp rx=0 tx=0).ready entity=mgr.5914 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:26.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.953+0000 7f1ba2000700 1 -- 172.21.15.12:0/3950230966 <== mgr.5914 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1b8c035990 con 0x7f1b8c06ae00 2024-03-29T10:07:26.953 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1 2024-03-29T10:07:26.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.955+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 msgr2=0x7f1b8c06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.955+0000 7f1bb0a00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0097c0 tx=0x7f1b9c008030 comp rx=0 tx=0).stop 2024-03-29T10:07:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.955+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3950230966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac15dad0 msgr2=0x7f1bac06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.955+0000 7f1bb0a00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac15dad0 0x7f1bac06d1d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400cb20 tx=0x7f1ba400ce30 comp rx=0 tx=0).stop 2024-03-29T10:07:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.956+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3950230966 shutdown_connections 2024-03-29T10:07:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.956+0000 7f1bb0a00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b8c06ae00 0x7f1b8c06d240 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.956+0000 7f1bb0a00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac15dad0 0x7f1bac06d1d0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.956+0000 7f1bb0a00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1bac0d1630 0x7f1bac0b8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.956+0000 7f1bb0a00700 1 --2- 172.21.15.12:0/3950230966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1bac0d21a0 0x7f1bac06cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.956+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3950230966 >> 172.21.15.12:0/3950230966 conn(0x7f1bac08aff0 msgr2=0x7f1bac06b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.956+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3950230966 shutdown_connections 2024-03-29T10:07:26.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:26.956+0000 7f1bb0a00700 1 -- 172.21.15.12:0/3950230966 wait complete. 2024-03-29T10:07:26.974 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-29T10:07:26.975 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2024-03-29T10:07:27.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.177+0000 7f81e6a00700 1 Processor -- start 2024-03-29T10:07:27.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.177+0000 7f81e6a00700 1 -- start start 2024-03-29T10:07:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.178+0000 7f81e6a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f81e0099a60 0x7f81e00d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.178+0000 7f81e6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 0x7f81e006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.178+0000 7f81e6a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00d0970 con 0x7f81e015d990 2024-03-29T10:07:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.178+0000 7f81e6a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e006c020 con 0x7f81e0099a60 2024-03-29T10:07:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.178+0000 7f81e6a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e00c81a0 con 0x7f81e009a750 2024-03-29T10:07:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.178+0000 7f81df400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f81e015d990 0x7f81e0099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37954/0 (socket says 172.21.15.12:37954) 2024-03-29T10:07:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.178+0000 7f81df400700 1 -- 172.21.15.12:0/1195229105 learned_addr learned my addr 172.21.15.12:0/1195229105 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.179+0000 7f81dfe00700 1 --2- 172.21.15.12:0/1195229105 >> v2:172.21.15.159:3300/0 conn(0x7f81e0099a60 0x7f81e00d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.179+0000 7f81e4a00700 1 --2- 172.21.15.12:0/1195229105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 0x7f81e006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.179+0000 7f81dea00700 1 -- 172.21.15.12:0/1195229105 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719302900 0 0) 0x7f81e00d0970 con 0x7f81e015d990 2024-03-29T10:07:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.179+0000 7f81dea00700 1 -- 172.21.15.12:0/1195229105 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c40035b0 con 0x7f81e015d990 2024-03-29T10:07:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.179+0000 7f81dfe00700 1 -- 172.21.15.12:0/1195229105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 msgr2=0x7f81e006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.179+0000 7f81dfe00700 1 --2- 172.21.15.12:0/1195229105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 0x7f81e006c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.179+0000 7f81dfe00700 1 -- 172.21.15.12:0/1195229105 >> v1:172.21.15.12:6789/0 conn(0x7f81e015d990 legacy=0x7f81e0099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.179+0000 7f81dfe00700 1 -- 172.21.15.12:0/1195229105 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e00d0690 con 0x7f81e0099a60 2024-03-29T10:07:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.179+0000 7f81e4a00700 1 --2- 172.21.15.12:0/1195229105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 0x7f81e006c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:07:27.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.179+0000 7f81dfe00700 1 --2- 172.21.15.12:0/1195229105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e0099a60 0x7f81e00d0060 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009840 tx=0x7f81d0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d4b207b6956f5e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81dea00700 1 -- 172.21.15.12:0/1195229105 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d0011830 con 0x7f81e0099a60 2024-03-29T10:07:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81dea00700 1 -- 172.21.15.12:0/1195229105 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d0011990 con 0x7f81e0099a60 2024-03-29T10:07:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81dea00700 1 -- 172.21.15.12:0/1195229105 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d0011d20 con 0x7f81e0099a60 2024-03-29T10:07:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81e6a00700 1 -- 172.21.15.12:0/1195229105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e0099a60 msgr2=0x7f81e00d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81e6a00700 1 --2- 172.21.15.12:0/1195229105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e0099a60 0x7f81e00d0060 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009840 tx=0x7f81d0009b50 comp rx=0 tx=0).stop 2024-03-29T10:07:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81e6a00700 1 -- 172.21.15.12:0/1195229105 shutdown_connections 2024-03-29T10:07:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81e6a00700 1 --2- 172.21.15.12:0/1195229105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 0x7f81e006c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81e6a00700 1 --2- 172.21.15.12:0/1195229105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e0099a60 0x7f81e00d0060 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81e6a00700 1 -- 172.21.15.12:0/1195229105 >> 172.21.15.12:0/1195229105 conn(0x7f81e00b2e20 msgr2=0x7f81e00b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81e6a00700 1 -- 172.21.15.12:0/1195229105 shutdown_connections 2024-03-29T10:07:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.180+0000 7f81e6a00700 1 -- 172.21.15.12:0/1195229105 wait complete. 2024-03-29T10:07:27.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.182+0000 7f81e6a00700 1 Processor -- start 2024-03-29T10:07:27.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.182+0000 7f81e6a00700 1 -- start start 2024-03-29T10:07:27.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81e6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0099a60 0x7f81e0150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81e6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 0x7f81e00c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81e6a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e015d990 0x7f81e00c7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81e6a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e00c8e30 con 0x7f81e0099a60 2024-03-29T10:07:27.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81e6a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e00c9110 con 0x7f81e015d990 2024-03-29T10:07:27.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81e6a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e00c93f0 con 0x7f81e009a750 2024-03-29T10:07:27.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81dfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0099a60 0x7f81e0150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81dfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0099a60 0x7f81e0150500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48500/0 (socket says 172.21.15.12:48500) 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81dfe00700 1 -- 172.21.15.12:0/2185702758 learned_addr learned my addr 172.21.15.12:0/2185702758 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81df400700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 0x7f81e00c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.183+0000 7f81e4a00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e015d990 0x7f81e00c7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.184+0000 7f81dfe00700 1 -- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 msgr2=0x7f81e00c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.184+0000 7f81dfe00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 0x7f81e00c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.184+0000 7f81dfe00700 1 -- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e015d990 msgr2=0x7f81e00c7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.184+0000 7f81dfe00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e015d990 0x7f81e00c7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.184+0000 7f81dfe00700 1 -- 172.21.15.12:0/2185702758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d8009a70 con 0x7f81e0099a60 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.184+0000 7f81dfe00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0099a60 0x7f81e0150500 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009fd0 tx=0x7f81d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.184+0000 7f81d7e00700 1 -- 172.21.15.12:0/2185702758 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d0004550 con 0x7f81e0099a60 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.184+0000 7f81d7e00700 1 -- 172.21.15.12:0/2185702758 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d001e070 con 0x7f81e0099a60 2024-03-29T10:07:27.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.184+0000 7f81e6a00700 1 -- 172.21.15.12:0/2185702758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d00094a0 con 0x7f81e0099a60 2024-03-29T10:07:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.184+0000 7f81d7e00700 1 -- 172.21.15.12:0/2185702758 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d00038e0 con 0x7f81e0099a60 2024-03-29T10:07:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.185+0000 7f81e6a00700 1 -- 172.21.15.12:0/2185702758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e01f4b50 con 0x7f81e0099a60 2024-03-29T10:07:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.186+0000 7f81e6a00700 1 -- 172.21.15.12:0/2185702758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e0067720 con 0x7f81e0099a60 2024-03-29T10:07:27.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.186+0000 7f81d7e00700 1 -- 172.21.15.12:0/2185702758 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 21) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f81d0003a40 con 0x7f81e0099a60 2024-03-29T10:07:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.187+0000 7f81d7e00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81c006ab30 0x7f81c006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.187+0000 7f81d7e00700 1 -- 172.21.15.12:0/2185702758 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f81d0089ed0 con 0x7f81e0099a60 2024-03-29T10:07:27.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.188+0000 7f81df400700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81c006ab30 0x7f81c006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.188+0000 7f81df400700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81c006ab30 0x7f81c006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f81e00c9dc0 tx=0x7f81cc007dc0 comp rx=0 tx=0).ready entity=mgr.5914 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:27.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.192+0000 7f81d7e00700 1 -- 172.21.15.12:0/2185702758 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81d0055290 con 0x7f81e0099a60 2024-03-29T10:07:27.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.429+0000 7f81e6a00700 1 -- 172.21.15.12:0/2185702758 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7f81e006bcb0 con 0x7f81c006ab30 2024-03-29T10:07:27.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.494+0000 7f81d7e00700 1 -- 172.21.15.12:0/2185702758 <== mgr.5914 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f81cc006bd0 con 0x7f81c006ab30 2024-03-29T10:07:27.494 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1 2024-03-29T10:07:27.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.497+0000 7f81e6a00700 1 -- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81c006ab30 msgr2=0x7f81c006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.497+0000 7f81e6a00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81c006ab30 0x7f81c006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f81e00c9dc0 tx=0x7f81cc007dc0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.497+0000 7f81e6a00700 1 -- 172.21.15.12:0/2185702758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0099a60 msgr2=0x7f81e0150500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.497+0000 7f81e6a00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0099a60 0x7f81e0150500 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009fd0 tx=0x7f81d0004750 comp rx=0 tx=0).stop 2024-03-29T10:07:27.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.497+0000 7f81e6a00700 1 -- 172.21.15.12:0/2185702758 shutdown_connections 2024-03-29T10:07:27.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.497+0000 7f81e6a00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f81c006ab30 0x7f81c006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.497+0000 7f81e6a00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e015d990 0x7f81e00c7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.497+0000 7f81e6a00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0099a60 0x7f81e0150500 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.497+0000 7f81e6a00700 1 --2- 172.21.15.12:0/2185702758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e009a750 0x7f81e00c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.497+0000 7f81e6a00700 1 -- 172.21.15.12:0/2185702758 >> 172.21.15.12:0/2185702758 conn(0x7f81e00b2e20 msgr2=0x7f81e00b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:27.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.498+0000 7f81e6a00700 1 -- 172.21.15.12:0/2185702758 shutdown_connections 2024-03-29T10:07:27.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.498+0000 7f81e6a00700 1 -- 172.21.15.12:0/2185702758 wait complete. 2024-03-29T10:07:27.515 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-29T10:07:27.515 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-03-29-10_08_00 in ~33s... 2024-03-29T10:07:27.516 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-03-29-10_09_00 in ~93s... 2024-03-29T10:07:27.516 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2024-03-29T10:07:27.516 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-03-29T10:07:27.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.717+0000 7fd2ea200700 1 Processor -- start 2024-03-29T10:07:27.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.717+0000 7fd2ea200700 1 -- start start 2024-03-29T10:07:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.718+0000 7fd2ea200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd2e415d9d0 0x7fd2e40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.718+0000 7fd2ea200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd2e40d16a0 0x7fd2e40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.718+0000 7fd2ea200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e4161420 con 0x7fd2e40d2370 2024-03-29T10:07:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.718+0000 7fd2ea200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e40d27e0 con 0x7fd2e40d16a0 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.719+0000 7fd2ea200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e4161700 con 0x7fd2e415d9d0 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.719+0000 7fd2e3e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd2e40d2370 0x7fd2e415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37970/0 (socket says 172.21.15.12:37970) 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.719+0000 7fd2e2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd2e40d16a0 0x7fd2e40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.719+0000 7fd2e3e00700 1 -- 172.21.15.12:0/2509943955 learned_addr learned my addr 172.21.15.12:0/2509943955 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.719+0000 7fd2e3400700 1 --2- 172.21.15.12:0/2509943955 >> v2:172.21.15.159:3300/0 conn(0x7fd2e415d9d0 0x7fd2e40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.719+0000 7fd2e2a00700 1 -- 172.21.15.12:0/2509943955 >> v2:172.21.15.159:3300/0 conn(0x7fd2e415d9d0 msgr2=0x7fd2e40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.719+0000 7fd2e2a00700 1 --2- 172.21.15.12:0/2509943955 >> v2:172.21.15.159:3300/0 conn(0x7fd2e415d9d0 0x7fd2e40d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.719+0000 7fd2e2a00700 1 -- 172.21.15.12:0/2509943955 >> v1:172.21.15.12:6789/0 conn(0x7fd2e40d2370 legacy=0x7fd2e415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.719+0000 7fd2e2a00700 1 -- 172.21.15.12:0/2509943955 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e41619e0 con 0x7fd2e40d16a0 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.719+0000 7fd2e2000700 1 -- 172.21.15.12:0/2509943955 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988586906 0 0) 0x7fd2e4161420 con 0x7fd2e40d2370 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.720+0000 7fd2e2a00700 1 --2- 172.21.15.12:0/2509943955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d16a0 0x7fd2e40d1e30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd2e40d0ff0 tx=0x7fd2d0009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9275648a6a1323 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.720+0000 7fd2e2000700 1 -- 172.21.15.12:0/2509943955 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0005e30 con 0x7fd2e40d16a0 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.720+0000 7fd2e2000700 1 -- 172.21.15.12:0/2509943955 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d0006030 con 0x7fd2e40d16a0 2024-03-29T10:07:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.720+0000 7fd2e2000700 1 -- 172.21.15.12:0/2509943955 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d00063c0 con 0x7fd2e40d16a0 2024-03-29T10:07:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.720+0000 7fd2ea200700 1 -- 172.21.15.12:0/2509943955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d16a0 msgr2=0x7fd2e40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.720+0000 7fd2ea200700 1 --2- 172.21.15.12:0/2509943955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d16a0 0x7fd2e40d1e30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd2e40d0ff0 tx=0x7fd2d0009b00 comp rx=0 tx=0).stop 2024-03-29T10:07:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.720+0000 7fd2ea200700 1 -- 172.21.15.12:0/2509943955 shutdown_connections 2024-03-29T10:07:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.720+0000 7fd2ea200700 1 --2- 172.21.15.12:0/2509943955 >> v2:172.21.15.159:3300/0 conn(0x7fd2e415d9d0 0x7fd2e40d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.720+0000 7fd2ea200700 1 --2- 172.21.15.12:0/2509943955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d16a0 0x7fd2e40d1e30 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.721+0000 7fd2ea200700 1 -- 172.21.15.12:0/2509943955 >> 172.21.15.12:0/2509943955 conn(0x7fd2e408aff0 msgr2=0x7fd2e40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.721+0000 7fd2ea200700 1 -- 172.21.15.12:0/2509943955 shutdown_connections 2024-03-29T10:07:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.721+0000 7fd2ea200700 1 -- 172.21.15.12:0/2509943955 wait complete. 2024-03-29T10:07:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.722+0000 7fd2ea200700 1 Processor -- start 2024-03-29T10:07:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.722+0000 7fd2ea200700 1 -- start start 2024-03-29T10:07:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.723+0000 7fd2ea200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2e40d16a0 0x7fd2e4158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.723+0000 7fd2ea200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d2370 0x7fd2e415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.723+0000 7fd2ea200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2e415d9d0 0x7fd2e415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.723+0000 7fd2ea200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e40cc900 con 0x7fd2e40d2370 2024-03-29T10:07:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.723+0000 7fd2ea200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e415ef90 con 0x7fd2e415d9d0 2024-03-29T10:07:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.723+0000 7fd2ea200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e415e920 con 0x7fd2e40d16a0 2024-03-29T10:07:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d2370 0x7fd2e415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e3400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2e40d16a0 0x7fd2e4158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d2370 0x7fd2e415ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48526/0 (socket says 172.21.15.12:48526) 2024-03-29T10:07:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e2a00700 1 -- 172.21.15.12:0/2691221706 learned_addr learned my addr 172.21.15.12:0/2691221706 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:07:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e3e00700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2e415d9d0 0x7fd2e415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e2a00700 1 -- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2e40d16a0 msgr2=0x7fd2e4158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e2a00700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2e40d16a0 0x7fd2e4158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e2a00700 1 -- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2e415d9d0 msgr2=0x7fd2e415e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e2a00700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2e415d9d0 0x7fd2e415e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e2a00700 1 -- 172.21.15.12:0/2691221706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d0009430 con 0x7fd2e40d2370 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.724+0000 7fd2e2a00700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d2370 0x7fd2e415ddb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0000c00 tx=0x7fd2d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.725+0000 7fd2db400700 1 -- 172.21.15.12:0/2691221706 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d00068b0 con 0x7fd2e40d2370 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.725+0000 7fd2db400700 1 -- 172.21.15.12:0/2691221706 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d0020070 con 0x7fd2e40d2370 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.725+0000 7fd2ea200700 1 -- 172.21.15.12:0/2691221706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e415ec00 con 0x7fd2e40d2370 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.725+0000 7fd2db400700 1 -- 172.21.15.12:0/2691221706 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0004280 con 0x7fd2e40d2370 2024-03-29T10:07:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.725+0000 7fd2ea200700 1 -- 172.21.15.12:0/2691221706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e415f4c0 con 0x7fd2e40d2370 2024-03-29T10:07:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.726+0000 7fd2ea200700 1 -- 172.21.15.12:0/2691221706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e40d1ca0 con 0x7fd2e40d2370 2024-03-29T10:07:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.727+0000 7fd2db400700 1 -- 172.21.15.12:0/2691221706 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 21) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fd2d0022030 con 0x7fd2e40d2370 2024-03-29T10:07:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.728+0000 7fd2db400700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2c406ab30 0x7fd2c406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:07:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.728+0000 7fd2db400700 1 -- 172.21.15.12:0/2691221706 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fd2d008a610 con 0x7fd2e40d2370 2024-03-29T10:07:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.728+0000 7fd2e3400700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2c406ab30 0x7fd2c406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:07:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.729+0000 7fd2e3400700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2c406ab30 0x7fd2c406cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4009b40 tx=0x7fd2d4009210 comp rx=0 tx=0).ready entity=mgr.5914 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:07:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.733+0000 7fd2db400700 1 -- 172.21.15.12:0/2691221706 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd2d0055880 con 0x7fd2e40d2370 2024-03-29T10:07:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.924+0000 7fd2db400700 1 -- 172.21.15.12:0/2691221706 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 22) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fd2d0055c50 con 0x7fd2e40d2370 2024-03-29T10:07:27.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.983+0000 7fd2ea200700 1 -- 172.21.15.12:0/2691221706 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2e4161fa0 con 0x7fd2c406ab30 2024-03-29T10:07:27.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.991+0000 7fd2db400700 1 -- 172.21.15.12:0/2691221706 <== mgr.5914 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7fd2d4011810 con 0x7fd2c406ab30 2024-03-29T10:07:27.991 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:07:27.991 INFO:teuthology.orchestra.run.smithi012.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-03-29T10:07:27.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 -- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2c406ab30 msgr2=0x7fd2c406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2c406ab30 0x7fd2c406cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4009b40 tx=0x7fd2d4009210 comp rx=0 tx=0).stop 2024-03-29T10:07:27.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 -- 172.21.15.12:0/2691221706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d2370 msgr2=0x7fd2e415ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:07:27.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d2370 0x7fd2e415ddb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0000c00 tx=0x7fd2d0004750 comp rx=0 tx=0).stop 2024-03-29T10:07:27.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 -- 172.21.15.12:0/2691221706 shutdown_connections 2024-03-29T10:07:27.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2c406ab30 0x7fd2c406cff0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2e415d9d0 0x7fd2e415e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2e40d16a0 0x7fd2e4158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 --2- 172.21.15.12:0/2691221706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e40d2370 0x7fd2e415ddb0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:07:27.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 -- 172.21.15.12:0/2691221706 >> 172.21.15.12:0/2691221706 conn(0x7fd2e408aff0 msgr2=0x7fd2e40b13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:07:27.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 -- 172.21.15.12:0/2691221706 shutdown_connections 2024-03-29T10:07:27.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:07:27.994+0000 7fd2ea200700 1 -- 172.21.15.12:0/2691221706 wait complete. 2024-03-29T10:07:28.013 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-03-29T10:07:28.013 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2024-03-29T10:07:28.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:28.013 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:28.060 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:28.060 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:28.060 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:29.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:29.061 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:29.113 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:29.113 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:29.113 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:30.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:30.114 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:30.166 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:30.166 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:30.166 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:31.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:31.167 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:31.220 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:31.220 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:31.220 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:32.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:32.221 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:32.272 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:32.272 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:32.272 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:33.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:33.273 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:33.325 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:33.325 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:33.325 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:34.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:34.326 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:34.378 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:34.378 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:34.378 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:35.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:35.379 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:35.431 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:35.431 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:35.431 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:36.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:36.432 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:36.484 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:36.484 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:36.484 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:37.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:37.485 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:37.538 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:37.538 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:37.538 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:38.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:38.539 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:38.591 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:38.591 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:38.591 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:39.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:39.592 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:39.644 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:39.644 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:39.644 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:40.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:40.645 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:40.697 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:40.697 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:40.697 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:41.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:41.701 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:41.753 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:41.753 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:41.754 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:42.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:42.754 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:42.806 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:42.806 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:42.806 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:43.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:43.807 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:43.859 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:43.859 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:43.860 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:44.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:44.861 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:44.912 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:44.912 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:44.912 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:45.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:45.912 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:45.964 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:45.964 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:45.964 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:46.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:46.965 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:47.017 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:47.017 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:47.017 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:48.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:48.018 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:48.070 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:48.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:48.071 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:49.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:49.072 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:49.124 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:49.124 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:49.124 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:50.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:50.125 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:50.176 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:50.177 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:50.177 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:51.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:51.178 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:51.230 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:51.230 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:51.230 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:52.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:52.232 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:52.283 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:52.283 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:52.284 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:53.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:53.284 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:53.336 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:53.336 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:53.336 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:54.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:54.338 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:54.389 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:54.390 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:54.390 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:55.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:55.394 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:55.446 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:55.446 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:55.446 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:56.161 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:07:56.166 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:07:56.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:56.448 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:56.498 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:56.498 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:56.498 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:57.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:57.499 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:57.550 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:57.551 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:57.551 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:58.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:58.552 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:58.606 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:58.606 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:58.606 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:07:59.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:07:59.607 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:07:59.659 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:07:59.659 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:07:59.659 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:00.508 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:08:00.508+0000 7ff4aae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:08:00.508 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:08:00.508+0000 7ff4aae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:08:00.508 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:08:00.508+0000 7ff4aae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:08:00.508 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:08:00.508+0000 7ff4aae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:08:00.508 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:08:00.508+0000 7ff4aae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:08:00.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:00.660 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:00.708 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:00.709 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:00.709 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:00.709 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:00.709 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-03-29-10_08_00_UTC'] 2024-03-29T10:08:00.709 DEBUG:tasks.cephfs.test_snap_schedules:exec=1711706847.5158958, now = 1711706880.7098403, timo = 33 2024-03-29T10:08:01.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:01.711 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:01.761 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:01.762 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:01.762 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:01.762 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:02.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:02.763 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:02.814 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:02.815 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:02.815 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:02.815 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:03.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:03.817 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:03.868 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:03.869 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:03.869 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:03.869 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:04.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:04.870 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:04.921 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:04.922 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:04.922 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:04.922 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:05.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:05.923 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:05.974 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:05.975 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:05.975 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:05.975 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:06.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:06.976 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:07.029 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:07.030 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:07.030 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:07.030 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:08.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:08.031 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:08.082 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:08.083 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:08.083 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:08.083 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:09.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:09.084 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:09.135 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:09.136 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:09.136 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:09.136 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:10.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:10.137 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:10.186 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:10.187 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:10.188 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:10.188 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:11.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:11.189 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:11.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:11.241 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:11.241 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:11.241 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:12.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:12.242 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:12.292 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:12.294 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:12.294 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:12.294 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:13.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:13.295 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:13.346 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:13.347 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:13.347 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:13.347 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:14.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:14.348 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:14.399 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:14.399 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:14.399 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:15.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:15.400 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:15.450 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:15.451 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:15.452 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:15.452 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:16.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:16.452 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:16.503 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:16.506 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:16.506 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:16.506 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:17.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:17.507 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:17.558 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:17.559 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:17.559 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:17.559 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:18.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:18.560 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:18.610 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:18.611 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:18.612 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:18.612 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:19.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:19.613 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:19.664 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:19.665 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:19.665 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:19.665 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:20.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:20.667 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:20.718 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:20.719 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:20.719 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:20.719 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:21.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:21.720 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:21.770 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:21.771 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:21.771 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:21.772 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:22.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:22.773 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:22.824 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:22.825 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:22.825 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:22.825 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:23.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:23.826 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:23.877 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:23.878 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:23.878 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:23.878 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:24.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:24.879 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:24.930 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:24.931 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:24.931 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:24.931 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:25.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:25.932 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:25.983 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:25.984 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:25.984 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:25.984 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:26.270 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:08:26.275 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:08:26.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:26.985 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:27.036 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:27.037 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:27.037 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:27.037 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:28.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:28.039 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:28.101 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:28.102 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:28.102 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:28.102 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:29.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:29.103 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:29.154 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:29.155 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:29.155 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:29.155 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:30.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:30.156 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:30.206 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:30.208 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:30.208 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:30.208 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:31.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:31.209 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:31.259 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:31.260 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:31.260 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:31.260 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:32.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:32.261 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:32.312 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:32.313 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:32.313 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:32.313 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:33.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:33.314 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:33.365 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:33.366 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:33.366 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:33.366 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:34.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:34.367 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:34.418 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:34.419 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:34.419 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:34.419 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:35.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:35.420 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:35.471 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:35.472 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:35.472 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:35.472 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:36.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:36.473 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:36.524 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:36.525 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:36.525 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:36.525 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:37.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:37.526 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:37.576 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:37.577 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:37.577 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:37.578 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:38.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:38.579 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:38.630 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:38.631 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:38.631 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:38.631 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:39.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:39.632 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:39.682 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:39.683 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:39.683 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:39.683 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:40.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:40.685 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:40.735 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:40.736 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:40.736 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:40.736 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:41.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:41.738 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:41.788 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:41.789 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:41.789 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:41.790 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:42.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:42.791 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:42.841 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:42.843 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:42.843 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:42.843 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:43.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:43.844 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:43.896 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:43.897 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:43.897 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:43.897 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:44.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:44.898 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:44.949 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:44.950 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:44.951 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:44.951 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:45.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:45.952 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:46.002 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:46.003 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:46.003 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:46.003 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:47.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:47.005 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:47.055 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:47.057 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:47.057 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:47.057 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:48.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:48.058 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:48.109 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:48.110 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:48.110 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:48.110 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:49.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:49.111 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:49.161 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:49.162 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:49.162 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:49.162 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:50.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:50.163 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:50.214 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:50.215 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:50.216 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:50.216 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:51.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:51.217 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:51.267 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:51.269 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:51.269 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:51.269 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:52.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:52.269 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:52.320 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:52.321 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:52.321 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:52.321 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:53.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:53.322 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:53.372 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:53.373 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:53.373 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:53.373 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:54.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:54.374 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:54.424 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:54.426 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:54.426 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:54.426 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:55.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:55.427 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:55.478 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:55.479 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:55.479 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:55.479 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:56.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:56.480 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:56.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:08:56.529 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:08:56.530 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:56.531 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:56.532 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:56.532 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:57.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:57.533 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:57.584 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:57.585 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:57.585 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:57.586 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:58.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:58.586 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:58.638 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:58.639 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:58.639 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:08:59.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:08:59.639 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:08:59.690 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:08:59.691 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:08:59.691 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:08:59.691 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:09:00.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:00.692 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:09:00.742 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:09:00.742 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_09_00_UTC 2024-03-29T10:09:00.743 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_09_00_UTC', 'scheduled-2024-03-29-10_08_00_UTC'} 2024-03-29T10:09:00.743 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-03-29-10_09_00_UTC'} 2024-03-29T10:09:00.743 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:09:00.743 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-03-29-10_09_00_UTC'] 2024-03-29T10:09:00.743 DEBUG:tasks.cephfs.test_snap_schedules:exec=1711706847.5158958, now = 1711706940.7437353, timo = 93 2024-03-29T10:09:01.744 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-03-29T10:09:01.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.957+0000 7fd168600700 1 Processor -- start 2024-03-29T10:09:01.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.957+0000 7fd168600700 1 -- start start 2024-03-29T10:09:01.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.958+0000 7fd168600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd160160d70 0x7fd1600b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:01.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.958+0000 7fd168600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd16015ef70 0x7fd16015e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:01.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.959+0000 7fd168600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd160161150 con 0x7fd16006a790 2024-03-29T10:09:01.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.959+0000 7fd168600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1600b4cd0 con 0x7fd16015ef70 2024-03-29T10:09:01.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.959+0000 7fd168600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16006dba0 con 0x7fd160160d70 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.959+0000 7fd165c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd160160d70 0x7fd1600b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.959+0000 7fd165200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd16015ef70 0x7fd16015e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.959+0000 7fd165200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd16015ef70 0x7fd16015e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:50116/0 (socket says 172.21.15.12:50116) 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.959+0000 7fd165200700 1 -- 172.21.15.12:0/1339697638 learned_addr learned my addr 172.21.15.12:0/1339697638 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.960+0000 7fd165200700 1 -- 172.21.15.12:0/1339697638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd160160d70 msgr2=0x7fd1600b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.960+0000 7fd165200700 1 --2- 172.21.15.12:0/1339697638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd160160d70 0x7fd1600b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.960+0000 7fd165200700 1 -- 172.21.15.12:0/1339697638 >> v1:172.21.15.159:6789/0 conn(0x7fd16006a790 legacy=0x7fd1601613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.960+0000 7fd165200700 1 -- 172.21.15.12:0/1339697638 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16015ed30 con 0x7fd16015ef70 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.960+0000 7fd165200700 1 --2- 172.21.15.12:0/1339697638 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd16015ef70 0x7fd16015e7f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0098f0 tx=0x7fd14c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=849eb2698b2de293 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.960+0000 7fd15fe00700 1 -- 172.21.15.12:0/1339697638 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c011830 con 0x7fd16015ef70 2024-03-29T10:09:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.960+0000 7fd15fe00700 1 -- 172.21.15.12:0/1339697638 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c011990 con 0x7fd16015ef70 2024-03-29T10:09:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.960+0000 7fd15fe00700 1 -- 172.21.15.12:0/1339697638 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c011d20 con 0x7fd16015ef70 2024-03-29T10:09:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.961+0000 7fd168600700 1 -- 172.21.15.12:0/1339697638 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd16015ef70 msgr2=0x7fd16015e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.961+0000 7fd168600700 1 --2- 172.21.15.12:0/1339697638 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd16015ef70 0x7fd16015e7f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0098f0 tx=0x7fd14c009c00 comp rx=0 tx=0).stop 2024-03-29T10:09:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.961+0000 7fd168600700 1 -- 172.21.15.12:0/1339697638 shutdown_connections 2024-03-29T10:09:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.961+0000 7fd168600700 1 --2- 172.21.15.12:0/1339697638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd160160d70 0x7fd1600b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.961+0000 7fd168600700 1 --2- 172.21.15.12:0/1339697638 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd16015ef70 0x7fd16015e7f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.961+0000 7fd168600700 1 -- 172.21.15.12:0/1339697638 >> 172.21.15.12:0/1339697638 conn(0x7fd1600b5720 msgr2=0x7fd1600b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.961+0000 7fd168600700 1 -- 172.21.15.12:0/1339697638 shutdown_connections 2024-03-29T10:09:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.961+0000 7fd168600700 1 -- 172.21.15.12:0/1339697638 wait complete. 2024-03-29T10:09:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.963+0000 7fd168600700 1 Processor -- start 2024-03-29T10:09:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.963+0000 7fd168600700 1 -- start start 2024-03-29T10:09:01.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd168600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd16006a790 0x7fd1600d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:01.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd168600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16015ef70 0x7fd16009a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:01.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd168600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd160160d70 0x7fd1601455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:01.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd168600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601f2ef0 con 0x7fd16015ef70 2024-03-29T10:09:01.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd168600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16009ab40 con 0x7fd16006a790 2024-03-29T10:09:01.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd168600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16009ae20 con 0x7fd160160d70 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd165200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16015ef70 0x7fd16009a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd165200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16015ef70 0x7fd16009a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53318/0 (socket says 172.21.15.12:53318) 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd165c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd16006a790 0x7fd1600d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd165200700 1 -- 172.21.15.12:0/2299600995 learned_addr learned my addr 172.21.15.12:0/2299600995 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.964+0000 7fd166600700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd160160d70 0x7fd1601455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd165200700 1 -- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd160160d70 msgr2=0x7fd1601455c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd165200700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd160160d70 0x7fd1601455c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd165200700 1 -- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd16006a790 msgr2=0x7fd1600d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd165200700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd16006a790 0x7fd1600d01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd165200700 1 -- 172.21.15.12:0/2299600995 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c009550 con 0x7fd16015ef70 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd165c00700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd16006a790 0x7fd1600d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd165200700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16015ef70 0x7fd16009a2f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009c40 tx=0x7fd14c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd15d600700 1 -- 172.21.15.12:0/2299600995 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c004400 con 0x7fd16015ef70 2024-03-29T10:09:01.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd15d600700 1 -- 172.21.15.12:0/2299600995 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c01f070 con 0x7fd16015ef70 2024-03-29T10:09:01.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd168600700 1 -- 172.21.15.12:0/2299600995 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16009b100 con 0x7fd16015ef70 2024-03-29T10:09:01.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd15d600700 1 -- 172.21.15.12:0/2299600995 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0038e0 con 0x7fd16015ef70 2024-03-29T10:09:01.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.965+0000 7fd168600700 1 -- 172.21.15.12:0/2299600995 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16009a830 con 0x7fd16015ef70 2024-03-29T10:09:01.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.966+0000 7fd168600700 1 -- 172.21.15.12:0/2299600995 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16009b620 con 0x7fd16015ef70 2024-03-29T10:09:01.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.967+0000 7fd15d600700 1 -- 172.21.15.12:0/2299600995 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fd14c021030 con 0x7fd16015ef70 2024-03-29T10:09:01.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.968+0000 7fd15d600700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd14006ae00 0x7fd14006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:01.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.968+0000 7fd15d600700 1 -- 172.21.15.12:0/2299600995 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fd14c08aaa0 con 0x7fd16015ef70 2024-03-29T10:09:01.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.969+0000 7fd165c00700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd14006ae00 0x7fd14006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:01.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.969+0000 7fd165c00700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd14006ae00 0x7fd14006d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd154009f30 tx=0x7fd154009210 comp rx=0 tx=0).ready entity=mgr.5914 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:01.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:01.973+0000 7fd15d600700 1 -- 172.21.15.12:0/2299600995 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd16009b620 con 0x7fd16015ef70 2024-03-29T10:09:02.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.211+0000 7fd168600700 1 -- 172.21.15.12:0/2299600995 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1600b54a0 con 0x7fd14006ae00 2024-03-29T10:09:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.291+0000 7fd15d600700 1 -- 172.21.15.12:0/2299600995 <== mgr.5914 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd154011cc0 con 0x7fd14006ae00 2024-03-29T10:09:02.291 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path snap_test_dir1 2024-03-29T10:09:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 -- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd14006ae00 msgr2=0x7fd14006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd14006ae00 0x7fd14006d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd154009f30 tx=0x7fd154009210 comp rx=0 tx=0).stop 2024-03-29T10:09:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 -- 172.21.15.12:0/2299600995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16015ef70 msgr2=0x7fd16009a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16015ef70 0x7fd16009a2f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009c40 tx=0x7fd14c004750 comp rx=0 tx=0).stop 2024-03-29T10:09:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 -- 172.21.15.12:0/2299600995 shutdown_connections 2024-03-29T10:09:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd14006ae00 0x7fd14006d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd160160d70 0x7fd1601455c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd16006a790 0x7fd1600d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 --2- 172.21.15.12:0/2299600995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16015ef70 0x7fd16009a2f0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 -- 172.21.15.12:0/2299600995 >> 172.21.15.12:0/2299600995 conn(0x7fd1600b5720 msgr2=0x7fd1600af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 -- 172.21.15.12:0/2299600995 shutdown_connections 2024-03-29T10:09:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.294+0000 7fd168600700 1 -- 172.21.15.12:0/2299600995 wait complete. 2024-03-29T10:09:02.312 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-03-29T10:09:02.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:02.313 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:09:02.358 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:09:02.358 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_09_00_UTC 2024-03-29T10:09:02.358 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_08_00_UTC 2024-03-29T10:09:02.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:02.359 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_08_00_UTC) 2024-03-29T10:09:02.448 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_09_00_UTC 2024-03-29T10:09:02.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:02.449 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_09_00_UTC) 2024-03-29T10:09:02.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:02.499 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-29T10:09:02.546 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T10:09:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.746+0000 7ff76b000700 1 Processor -- start 2024-03-29T10:09:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.746+0000 7ff76b000700 1 -- start start 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.747+0000 7ff76b000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff76415d9d0 0x7ff7640d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.747+0000 7ff76b000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7640d16a0 0x7ff7640d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.747+0000 7ff76b000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff764161420 con 0x7ff7640d2370 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.747+0000 7ff76b000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7640d27e0 con 0x7ff7640d16a0 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.747+0000 7ff76b000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff764161700 con 0x7ff76415d9d0 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.748+0000 7ff763400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7640d16a0 0x7ff7640d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.748+0000 7ff763400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7640d16a0 0x7ff7640d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53322/0 (socket says 172.21.15.12:53322) 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.748+0000 7ff763400700 1 -- 172.21.15.12:0/1085892966 learned_addr learned my addr 172.21.15.12:0/1085892966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.748+0000 7ff763e00700 1 --2- 172.21.15.12:0/1085892966 >> v2:172.21.15.159:3300/0 conn(0x7ff76415d9d0 0x7ff7640d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.748+0000 7ff763400700 1 -- 172.21.15.12:0/1085892966 >> v2:172.21.15.159:3300/0 conn(0x7ff76415d9d0 msgr2=0x7ff7640d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:02.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.748+0000 7ff763400700 1 --2- 172.21.15.12:0/1085892966 >> v2:172.21.15.159:3300/0 conn(0x7ff76415d9d0 0x7ff7640d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.748+0000 7ff763400700 1 -- 172.21.15.12:0/1085892966 >> v1:172.21.15.159:6789/0 conn(0x7ff7640d2370 legacy=0x7ff76415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.748+0000 7ff763400700 1 -- 172.21.15.12:0/1085892966 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7641619e0 con 0x7ff7640d16a0 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.748+0000 7ff763400700 1 --2- 172.21.15.12:0/1085892966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7640d16a0 0x7ff7640d1e30 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff7640d0ff0 tx=0x7ff750009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d620420de723ae2d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff762a00700 1 -- 172.21.15.12:0/1085892966 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff750004230 con 0x7ff7640d16a0 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff762a00700 1 -- 172.21.15.12:0/1085892966 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff750004390 con 0x7ff7640d16a0 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff762a00700 1 -- 172.21.15.12:0/1085892966 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7500036a0 con 0x7ff7640d16a0 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff76b000700 1 -- 172.21.15.12:0/1085892966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7640d16a0 msgr2=0x7ff7640d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff76b000700 1 --2- 172.21.15.12:0/1085892966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7640d16a0 0x7ff7640d1e30 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff7640d0ff0 tx=0x7ff750009bc0 comp rx=0 tx=0).stop 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff76b000700 1 -- 172.21.15.12:0/1085892966 shutdown_connections 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff76b000700 1 --2- 172.21.15.12:0/1085892966 >> v2:172.21.15.159:3300/0 conn(0x7ff76415d9d0 0x7ff7640d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff76b000700 1 --2- 172.21.15.12:0/1085892966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7640d16a0 0x7ff7640d1e30 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff76b000700 1 -- 172.21.15.12:0/1085892966 >> 172.21.15.12:0/1085892966 conn(0x7ff76408aff0 msgr2=0x7ff7640d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff76b000700 1 -- 172.21.15.12:0/1085892966 shutdown_connections 2024-03-29T10:09:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.749+0000 7ff76b000700 1 -- 172.21.15.12:0/1085892966 wait complete. 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.751+0000 7ff76b000700 1 Processor -- start 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.751+0000 7ff76b000700 1 -- start start 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.752+0000 7ff76b000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7640d16a0 0x7ff7640c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.752+0000 7ff76b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7640d2370 0x7ff7640cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.752+0000 7ff76b000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76415d9d0 0x7ff764154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.752+0000 7ff76b000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7640c9f10 con 0x7ff76415d9d0 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.752+0000 7ff76b000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7640b8e90 con 0x7ff7640d16a0 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.752+0000 7ff76b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7640b7df0 con 0x7ff7640d2370 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.752+0000 7ff769000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76415d9d0 0x7ff764154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.753+0000 7ff763e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7640d16a0 0x7ff7640c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.753+0000 7ff769000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76415d9d0 0x7ff764154a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53330/0 (socket says 172.21.15.12:53330) 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.753+0000 7ff769000700 1 -- 172.21.15.12:0/3480815606 learned_addr learned my addr 172.21.15.12:0/3480815606 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.753+0000 7ff763400700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7640d2370 0x7ff7640cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.753+0000 7ff769000700 1 -- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7640d2370 msgr2=0x7ff7640cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.753+0000 7ff769000700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7640d2370 0x7ff7640cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.753+0000 7ff769000700 1 -- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7640d16a0 msgr2=0x7ff7640c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.753+0000 7ff769000700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7640d16a0 0x7ff7640c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.753+0000 7ff769000700 1 -- 172.21.15.12:0/3480815606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7500094f0 con 0x7ff76415d9d0 2024-03-29T10:09:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.753+0000 7ff769000700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76415d9d0 0x7ff764154a80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff75c00cc10 tx=0x7ff75c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.754+0000 7ff75be00700 1 -- 172.21.15.12:0/3480815606 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c00aa60 con 0x7ff76415d9d0 2024-03-29T10:09:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.754+0000 7ff75be00700 1 -- 172.21.15.12:0/3480815606 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c00abc0 con 0x7ff76415d9d0 2024-03-29T10:09:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.754+0000 7ff75be00700 1 -- 172.21.15.12:0/3480815606 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c012420 con 0x7ff76415d9d0 2024-03-29T10:09:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.754+0000 7ff76b000700 1 -- 172.21.15.12:0/3480815606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7640b80d0 con 0x7ff76415d9d0 2024-03-29T10:09:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.754+0000 7ff76b000700 1 -- 172.21.15.12:0/3480815606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7640b8690 con 0x7ff76415d9d0 2024-03-29T10:09:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.755+0000 7ff76b000700 1 -- 172.21.15.12:0/3480815606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7640d1ca0 con 0x7ff76415d9d0 2024-03-29T10:09:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.756+0000 7ff75be00700 1 -- 172.21.15.12:0/3480815606 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) v1 ==== 89268+0+0 (secure 0 0 0) 0x7ff75c012580 con 0x7ff76415d9d0 2024-03-29T10:09:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.756+0000 7ff75be00700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff74406ae00 0x7ff74406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.757+0000 7ff75be00700 1 -- 172.21.15.12:0/3480815606 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6111+0+0 (secure 0 0 0) 0x7ff75c089a60 con 0x7ff76415d9d0 2024-03-29T10:09:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.757+0000 7ff763e00700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff74406ae00 0x7ff74406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.758+0000 7ff763e00700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff74406ae00 0x7ff74406d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff754006b20 tx=0x7ff754008040 comp rx=0 tx=0).ready entity=mgr.5914 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:02.761+0000 7ff75be00700 1 -- 172.21.15.12:0/3480815606 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff75c054900 con 0x7ff76415d9d0 2024-03-29T10:09:03.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:03.036+0000 7ff76b000700 1 -- 172.21.15.12:0/3480815606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ff764067720 con 0x7ff76415d9d0 2024-03-29T10:09:04.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.029+0000 7ff75be00700 1 -- 172.21.15.12:0/3480815606 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7ff75c054cd0 con 0x7ff76415d9d0 2024-03-29T10:09:04.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.031+0000 7ff75be00700 1 -- 172.21.15.12:0/3480815606 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v26) v1 ==== 93+0+0 (secure 0 0 0) 0x7ff75c071ff0 con 0x7ff76415d9d0 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 -- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff74406ae00 msgr2=0x7ff74406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff74406ae00 0x7ff74406d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff754006b20 tx=0x7ff754008040 comp rx=0 tx=0).stop 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 -- 172.21.15.12:0/3480815606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76415d9d0 msgr2=0x7ff764154a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76415d9d0 0x7ff764154a80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff75c00cc10 tx=0x7ff75c00cf20 comp rx=0 tx=0).stop 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 -- 172.21.15.12:0/3480815606 shutdown_connections 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff74406ae00 0x7ff74406d2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76415d9d0 0x7ff764154a80 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7640d16a0 0x7ff7640c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 --2- 172.21.15.12:0/3480815606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7640d2370 0x7ff7640cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 -- 172.21.15.12:0/3480815606 >> 172.21.15.12:0/3480815606 conn(0x7ff76408aff0 msgr2=0x7ff76406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 -- 172.21.15.12:0/3480815606 shutdown_connections 2024-03-29T10:09:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.035+0000 7ff76b000700 1 -- 172.21.15.12:0/3480815606 wait complete. 2024-03-29T10:09:04.062 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:09:04.062 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:09:04.167 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:09:04.253 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:09:04.253 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:09:04.253 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:04.263 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:04.263+0000 7f018ebb1240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:09:04.271 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:04.271+0000 7fcbd46e4240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:09:04.305 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:09:04.305 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:04.394 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:09:04.456 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:04.456+0000 7f018ebb1240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:09:04.479 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:04.479+0000 7fcbd46e4240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:09:04.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.658+0000 7f96f3e00700 1 Processor -- start 2024-03-29T10:09:04.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.658+0000 7f96f3e00700 1 -- start start 2024-03-29T10:09:04.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.659+0000 7f96f3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96ec072580 0x7f96ec0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.659+0000 7f96f3e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f96ec0b2e50 0x7f96ec0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.659+0000 7f96f3e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec0b2180 con 0x7f96ec0b3af0 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.659+0000 7f96f3e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ec15b100 con 0x7f96ec072580 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.659+0000 7f96f3e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ec15b3e0 con 0x7f96ec0b2e50 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.659+0000 7f96f0a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96ec072580 0x7f96ec0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.659+0000 7f96f0a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96ec072580 0x7f96ec0b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53344/0 (socket says 172.21.15.12:53344) 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.659+0000 7f96f0a00700 1 -- 172.21.15.12:0/2227651254 learned_addr learned my addr 172.21.15.12:0/2227651254 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.659+0000 7f96f1e00700 1 --2- 172.21.15.12:0/2227651254 >> v2:172.21.15.159:3300/0 conn(0x7f96ec0b2e50 0x7f96ec0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.660+0000 7f96f0a00700 1 -- 172.21.15.12:0/2227651254 >> v2:172.21.15.159:3300/0 conn(0x7f96ec0b2e50 msgr2=0x7f96ec0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.660+0000 7f96f0a00700 1 --2- 172.21.15.12:0/2227651254 >> v2:172.21.15.159:3300/0 conn(0x7f96ec0b2e50 0x7f96ec0709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.660+0000 7f96f0a00700 1 -- 172.21.15.12:0/2227651254 >> v1:172.21.15.159:6789/0 conn(0x7f96ec0b3af0 legacy=0x7f96ec15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:04.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.660+0000 7f96f0a00700 1 -- 172.21.15.12:0/2227651254 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec0b23f0 con 0x7f96ec072580 2024-03-29T10:09:04.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.660+0000 7f96f0a00700 1 --2- 172.21.15.12:0/2227651254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec072580 0x7f96ec0b2910 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f96ec158390 tx=0x7f96d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5648fb8fc9d4c6bb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.660+0000 7f96ebe00700 1 -- 172.21.15.12:0/2227651254 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8011830 con 0x7f96ec072580 2024-03-29T10:09:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.661+0000 7f96ebe00700 1 -- 172.21.15.12:0/2227651254 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d8011990 con 0x7f96ec072580 2024-03-29T10:09:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.661+0000 7f96ebe00700 1 -- 172.21.15.12:0/2227651254 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8011d20 con 0x7f96ec072580 2024-03-29T10:09:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.661+0000 7f96f3e00700 1 -- 172.21.15.12:0/2227651254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec072580 msgr2=0x7f96ec0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.661+0000 7f96f3e00700 1 --2- 172.21.15.12:0/2227651254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec072580 0x7f96ec0b2910 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f96ec158390 tx=0x7f96d8009bd0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.661+0000 7f96f3e00700 1 -- 172.21.15.12:0/2227651254 shutdown_connections 2024-03-29T10:09:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.661+0000 7f96f3e00700 1 --2- 172.21.15.12:0/2227651254 >> v2:172.21.15.159:3300/0 conn(0x7f96ec0b2e50 0x7f96ec0709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.661+0000 7f96f3e00700 1 --2- 172.21.15.12:0/2227651254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec072580 0x7f96ec0b2910 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.661+0000 7f96f3e00700 1 -- 172.21.15.12:0/2227651254 >> 172.21.15.12:0/2227651254 conn(0x7f96ec0cf190 msgr2=0x7f96ec0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:04.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.661+0000 7f96f3e00700 1 -- 172.21.15.12:0/2227651254 shutdown_connections 2024-03-29T10:09:04.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.661+0000 7f96f3e00700 1 -- 172.21.15.12:0/2227651254 wait complete. 2024-03-29T10:09:04.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.664+0000 7f96f3e00700 1 Processor -- start 2024-03-29T10:09:04.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.664+0000 7f96f3e00700 1 -- start start 2024-03-29T10:09:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f3e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96ec072580 0x7f96ec158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec0b2e50 0x7f96ec158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96ec0b3af0 0x7f96ec150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec159560 con 0x7f96ec0b2e50 2024-03-29T10:09:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f3e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec159840 con 0x7f96ec072580 2024-03-29T10:09:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f3e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec159b20 con 0x7f96ec0b3af0 2024-03-29T10:09:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f1400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96ec072580 0x7f96ec158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec0b2e50 0x7f96ec158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f1400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96ec072580 0x7f96ec158a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:50176/0 (socket says 172.21.15.12:50176) 2024-03-29T10:09:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec0b2e50 0x7f96ec158f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53352/0 (socket says 172.21.15.12:53352) 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.665+0000 7f96f1400700 1 -- 172.21.15.12:0/2159572180 learned_addr learned my addr 172.21.15.12:0/2159572180 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.666+0000 7f96f1e00700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96ec0b3af0 0x7f96ec150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.666+0000 7f96f1400700 1 -- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96ec0b3af0 msgr2=0x7f96ec150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.666+0000 7f96f1400700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96ec0b3af0 0x7f96ec150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.666+0000 7f96f1400700 1 -- 172.21.15.12:0/2159572180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec0b2e50 msgr2=0x7f96ec158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.666+0000 7f96f1400700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec0b2e50 0x7f96ec158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.666+0000 7f96f1400700 1 -- 172.21.15.12:0/2159572180 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d8009550 con 0x7f96ec072580 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.666+0000 7f96f0a00700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec0b2e50 0x7f96ec158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.666+0000 7f96f1400700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96ec072580 0x7f96ec158a30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f96e000c450 tx=0x7f96e000c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.666+0000 7f96e9600700 1 -- 172.21.15.12:0/2159572180 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e00052d0 con 0x7f96ec072580 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.667+0000 7f96e9600700 1 -- 172.21.15.12:0/2159572180 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e0005430 con 0x7f96ec072580 2024-03-29T10:09:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.667+0000 7f96f3e00700 1 -- 172.21.15.12:0/2159572180 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec159e00 con 0x7f96ec072580 2024-03-29T10:09:04.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.667+0000 7f96e9600700 1 -- 172.21.15.12:0/2159572180 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e00038c0 con 0x7f96ec072580 2024-03-29T10:09:04.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.667+0000 7f96f3e00700 1 -- 172.21.15.12:0/2159572180 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec1514f0 con 0x7f96ec072580 2024-03-29T10:09:04.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.668+0000 7f96f3e00700 1 -- 172.21.15.12:0/2159572180 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ec15a320 con 0x7f96ec072580 2024-03-29T10:09:04.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.669+0000 7f96e9600700 1 -- 172.21.15.12:0/2159572180 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f96e0003a20 con 0x7f96ec072580 2024-03-29T10:09:04.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.669+0000 7f96e9600700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f96cc06adb0 0x7f96cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:04.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.670+0000 7f96e9600700 1 -- 172.21.15.12:0/2159572180 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f96e0088a50 con 0x7f96ec072580 2024-03-29T10:09:04.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.670+0000 7f96f0a00700 1 -- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f96cc06adb0 msgr2=0x7f96cc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:04.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.670+0000 7f96f0a00700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f96cc06adb0 0x7f96cc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:04.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.675+0000 7f96e9600700 1 -- 172.21.15.12:0/2159572180 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f96ec15a320 con 0x7f96ec072580 2024-03-29T10:09:04.785 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:04.785+0000 7f018ebb1240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:09:04.827 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:04.826+0000 7fcbd46e4240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:09:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.871+0000 7f96f0a00700 1 -- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f96cc06adb0 msgr2=0x7f96cc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.871+0000 7f96f0a00700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f96cc06adb0 0x7f96cc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.893+0000 7f96f3e00700 1 -- 172.21.15.12:0/2159572180 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96ec15a320 con 0x7f96ec072580 2024-03-29T10:09:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.894+0000 7f96e9600700 1 -- 172.21.15.12:0/2159572180 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f96ec15a320 con 0x7f96ec072580 2024-03-29T10:09:04.894 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:04.895 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":65,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:04.256397+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:07:07.106799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-29T10:07:07.671044+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":5}]},{"pgid":"7.12","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6833/43444":"2024-03-30T10:07:25.860642+0000","172.21.15.159:6832/43444":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3220343982":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/2775778725":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3363559414":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/1116927207":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3950908519":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/798085437":"2024-03-30T10:07:25.860642+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:09:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 -- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f96cc06adb0 msgr2=0x7f96cc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f96cc06adb0 0x7f96cc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 -- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96ec072580 msgr2=0x7f96ec158a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96ec072580 0x7f96ec158a30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f96e000c450 tx=0x7f96e000c760 comp rx=0 tx=0).stop 2024-03-29T10:09:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 -- 172.21.15.12:0/2159572180 shutdown_connections 2024-03-29T10:09:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f96cc06adb0 0x7f96cc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96ec0b3af0 0x7f96ec150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96ec072580 0x7f96ec158a30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 --2- 172.21.15.12:0/2159572180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96ec0b2e50 0x7f96ec158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 -- 172.21.15.12:0/2159572180 >> 172.21.15.12:0/2159572180 conn(0x7f96ec0cf190 msgr2=0x7f96ec152ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 -- 172.21.15.12:0/2159572180 shutdown_connections 2024-03-29T10:09:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:04.897+0000 7f96f3e00700 1 -- 172.21.15.12:0/2159572180 wait complete. 2024-03-29T10:09:04.915 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:07:07.106799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:07:07.671044+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 65, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:09:04.916 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:09:05.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.160+0000 7f233f400700 1 Processor -- start 2024-03-29T10:09:05.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.160+0000 7f233f400700 1 -- start start 2024-03-29T10:09:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.161+0000 7f233f400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f233815d8a0 0x7f23380b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.161+0000 7f233f400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23380ced20 con 0x7f23380d3eb0 2024-03-29T10:09:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.161+0000 7f233f400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23380d4340 con 0x7f23380cf1b0 2024-03-29T10:09:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.161+0000 7f233f400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23380ce6d0 con 0x7f233815d8a0 2024-03-29T10:09:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.162+0000 7f2337e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f23380d3eb0 0x7f233806b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35248/0 (socket says 172.21.15.12:35248) 2024-03-29T10:09:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.162+0000 7f2337e00700 1 -- 172.21.15.12:0/134753168 learned_addr learned my addr 172.21.15.12:0/134753168 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:05.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.162+0000 7f233ca00700 1 --2- 172.21.15.12:0/134753168 >> v2:172.21.15.159:3300/0 conn(0x7f233815d8a0 0x7f23380b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.162+0000 7f2337400700 1 -- 172.21.15.12:0/134753168 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53864722 0 0) 0x7f23380ced20 con 0x7f23380d3eb0 2024-03-29T10:09:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.162+0000 7f2337400700 1 -- 172.21.15.12:0/134753168 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23200035b0 con 0x7f23380d3eb0 2024-03-29T10:09:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.162+0000 7f2337400700 1 -- 172.21.15.12:0/134753168 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326231191 0 0) 0x7f23380d4340 con 0x7f23380cf1b0 2024-03-29T10:09:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.162+0000 7f2337400700 1 -- 172.21.15.12:0/134753168 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23380ced20 con 0x7f23380cf1b0 2024-03-29T10:09:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.162+0000 7f233ca00700 1 -- 172.21.15.12:0/134753168 >> v1:172.21.15.159:6789/0 conn(0x7f23380cf1b0 legacy=0x7f23380d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.162+0000 7f233ca00700 1 -- 172.21.15.12:0/134753168 >> v1:172.21.15.12:6789/0 conn(0x7f23380d3eb0 legacy=0x7f233806b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.162+0000 7f233ca00700 1 -- 172.21.15.12:0/134753168 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23380ce9b0 con 0x7f233815d8a0 2024-03-29T10:09:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.163+0000 7f233ca00700 1 --2- 172.21.15.12:0/134753168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f233815d8a0 0x7f23380b2990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f232c009840 tx=0x7f232c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a83a0a4c5419e04 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.163+0000 7f2337400700 1 -- 172.21.15.12:0/134753168 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c011830 con 0x7f233815d8a0 2024-03-29T10:09:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.163+0000 7f2337400700 1 -- 172.21.15.12:0/134753168 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c011990 con 0x7f233815d8a0 2024-03-29T10:09:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.163+0000 7f2337400700 1 -- 172.21.15.12:0/134753168 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c011d20 con 0x7f233815d8a0 2024-03-29T10:09:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.164+0000 7f233f400700 1 -- 172.21.15.12:0/134753168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f233815d8a0 msgr2=0x7f23380b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.164+0000 7f233f400700 1 --2- 172.21.15.12:0/134753168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f233815d8a0 0x7f23380b2990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f232c009840 tx=0x7f232c009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.164+0000 7f233f400700 1 -- 172.21.15.12:0/134753168 shutdown_connections 2024-03-29T10:09:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.164+0000 7f233f400700 1 --2- 172.21.15.12:0/134753168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f233815d8a0 0x7f23380b2990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.164+0000 7f233f400700 1 -- 172.21.15.12:0/134753168 >> 172.21.15.12:0/134753168 conn(0x7f233808aff0 msgr2=0x7f23380d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.164+0000 7f233f400700 1 -- 172.21.15.12:0/134753168 shutdown_connections 2024-03-29T10:09:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.164+0000 7f233f400700 1 -- 172.21.15.12:0/134753168 wait complete. 2024-03-29T10:09:05.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.166+0000 7f233f400700 1 Processor -- start 2024-03-29T10:09:05.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.166+0000 7f233f400700 1 -- start start 2024-03-29T10:09:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f233f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23380cf1b0 0x7f23381f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f233f400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f233815d8a0 0x7f23381f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f233f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23380c70c0 0x7f23380b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f233f400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23380c8cd0 con 0x7f23380c70c0 2024-03-29T10:09:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f233f400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23381f3f40 con 0x7f233815d8a0 2024-03-29T10:09:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f233f400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23380c80e0 con 0x7f23380cf1b0 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f233d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23380c70c0 0x7f23380b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f2337e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f233815d8a0 0x7f23381f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f233d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23380c70c0 0x7f23380b1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53360/0 (socket says 172.21.15.12:53360) 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f233ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23380cf1b0 0x7f23381f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.167+0000 7f233d400700 1 -- 172.21.15.12:0/360425330 learned_addr learned my addr 172.21.15.12:0/360425330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.168+0000 7f233ca00700 1 -- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f233815d8a0 msgr2=0x7f23381f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.168+0000 7f233ca00700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f233815d8a0 0x7f23381f3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.168+0000 7f233ca00700 1 -- 172.21.15.12:0/360425330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23380c70c0 msgr2=0x7f23380b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.168+0000 7f233ca00700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23380c70c0 0x7f23380b1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.168+0000 7f233ca00700 1 -- 172.21.15.12:0/360425330 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c0094a0 con 0x7f23380cf1b0 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.168+0000 7f233ca00700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23380cf1b0 0x7f23381f2a90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f232c016000 tx=0x7f232c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.168+0000 7f2334c00700 1 -- 172.21.15.12:0/360425330 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c01a6e0 con 0x7f23380cf1b0 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.168+0000 7f2334c00700 1 -- 172.21.15.12:0/360425330 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c01a840 con 0x7f23380cf1b0 2024-03-29T10:09:05.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.168+0000 7f2334c00700 1 -- 172.21.15.12:0/360425330 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c01abf0 con 0x7f23380cf1b0 2024-03-29T10:09:05.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.168+0000 7f233f400700 1 -- 172.21.15.12:0/360425330 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23380c7560 con 0x7f23380cf1b0 2024-03-29T10:09:05.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.169+0000 7f233f400700 1 -- 172.21.15.12:0/360425330 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23380c7b20 con 0x7f23380cf1b0 2024-03-29T10:09:05.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.170+0000 7f233f400700 1 -- 172.21.15.12:0/360425330 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2338067720 con 0x7f23380cf1b0 2024-03-29T10:09:05.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.170+0000 7f2334c00700 1 -- 172.21.15.12:0/360425330 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f232c022070 con 0x7f23380cf1b0 2024-03-29T10:09:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.171+0000 7f2334c00700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f231806adb0 0x7f231806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.171+0000 7f2334c00700 1 -- 172.21.15.12:0/360425330 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f232c08a3c0 con 0x7f23380cf1b0 2024-03-29T10:09:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.171+0000 7f2337e00700 1 -- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f231806adb0 msgr2=0x7f231806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.171+0000 7f2337e00700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f231806adb0 0x7f231806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:05.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.176+0000 7f2334c00700 1 -- 172.21.15.12:0/360425330 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f232c0553b0 con 0x7f23380cf1b0 2024-03-29T10:09:05.331 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:05.331+0000 7f018ebb1240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:09:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.372+0000 7f2337e00700 1 -- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f231806adb0 msgr2=0x7f231806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:05.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.372+0000 7f2337e00700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f231806adb0 0x7f231806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:05.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.394+0000 7f233f400700 1 -- 172.21.15.12:0/360425330 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f23380d1950 con 0x7f23380cf1b0 2024-03-29T10:09:05.394 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:05.394+0000 7fcbd46e4240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:09:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.395+0000 7f2334c00700 1 -- 172.21.15.12:0/360425330 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7f232c0589d0 con 0x7f23380cf1b0 2024-03-29T10:09:05.394 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:09:05.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 -- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f231806adb0 msgr2=0x7f231806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:05.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f231806adb0 0x7f231806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 -- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23380cf1b0 msgr2=0x7f23381f2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23380cf1b0 0x7f23381f2a90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f232c016000 tx=0x7f232c004750 comp rx=0 tx=0).stop 2024-03-29T10:09:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 -- 172.21.15.12:0/360425330 shutdown_connections 2024-03-29T10:09:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f231806adb0 0x7f231806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23380c70c0 0x7f23380b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23380cf1b0 0x7f23381f2a90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 --2- 172.21.15.12:0/360425330 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f233815d8a0 0x7f23381f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 -- 172.21.15.12:0/360425330 >> 172.21.15.12:0/360425330 conn(0x7f233808aff0 msgr2=0x7f233806d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 -- 172.21.15.12:0/360425330 shutdown_connections 2024-03-29T10:09:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.397+0000 7f233f400700 1 -- 172.21.15.12:0/360425330 wait complete. 2024-03-29T10:09:05.421 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:09:05.595 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:05.595+0000 7f018ebb1240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:09:05.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.633+0000 7f999c400700 1 Processor -- start 2024-03-29T10:09:05.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.634+0000 7f999c400700 1 -- start start 2024-03-29T10:09:05.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.635+0000 7f999c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999415d9d0 0x7f99940d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.635+0000 7f999c400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99940d16a0 0x7f99940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.635+0000 7f999c400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9994161420 con 0x7f99940d2370 2024-03-29T10:09:05.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.635+0000 7f999c400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99940d2820 con 0x7f99940d16a0 2024-03-29T10:09:05.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.635+0000 7f999c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9994161700 con 0x7f999415d9d0 2024-03-29T10:09:05.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.635+0000 7f9999000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99940d16a0 0x7f99940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:05.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.635+0000 7f999a400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f99940d2370 0x7f999415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35250/0 (socket says 172.21.15.12:35250) 2024-03-29T10:09:05.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.635+0000 7f9999000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99940d16a0 0x7f99940d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53374/0 (socket says 172.21.15.12:53374) 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.635+0000 7f999a400700 1 -- 172.21.15.12:0/265316135 learned_addr learned my addr 172.21.15.12:0/265316135 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.635+0000 7f9999a00700 1 --2- 172.21.15.12:0/265316135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999415d9d0 0x7f99940d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.636+0000 7f9999000700 1 -- 172.21.15.12:0/265316135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999415d9d0 msgr2=0x7f99940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.636+0000 7f9999000700 1 --2- 172.21.15.12:0/265316135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999415d9d0 0x7f99940d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.636+0000 7f9999000700 1 -- 172.21.15.12:0/265316135 >> v1:172.21.15.12:6789/0 conn(0x7f99940d2370 legacy=0x7f999415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.636+0000 7f9999000700 1 -- 172.21.15.12:0/265316135 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99941619e0 con 0x7f99940d16a0 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.636+0000 7f9993e00700 1 -- 172.21.15.12:0/265316135 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564368066 0 0) 0x7f9994161420 con 0x7f99940d2370 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.636+0000 7f9999000700 1 --2- 172.21.15.12:0/265316135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99940d16a0 0x7f99940d1e30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9980009820 tx=0x7f9980009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159b3ae92e04dbd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.636+0000 7f9993e00700 1 -- 172.21.15.12:0/265316135 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9980005e30 con 0x7f99940d16a0 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.636+0000 7f9993e00700 1 -- 172.21.15.12:0/265316135 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9980006030 con 0x7f99940d16a0 2024-03-29T10:09:05.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.636+0000 7f9993e00700 1 -- 172.21.15.12:0/265316135 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99800063c0 con 0x7f99940d16a0 2024-03-29T10:09:05.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.637+0000 7f999c400700 1 -- 172.21.15.12:0/265316135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99940d16a0 msgr2=0x7f99940d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.637+0000 7f999c400700 1 --2- 172.21.15.12:0/265316135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99940d16a0 0x7f99940d1e30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9980009820 tx=0x7f9980009b30 comp rx=0 tx=0).stop 2024-03-29T10:09:05.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.637+0000 7f999c400700 1 -- 172.21.15.12:0/265316135 shutdown_connections 2024-03-29T10:09:05.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.637+0000 7f999c400700 1 --2- 172.21.15.12:0/265316135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999415d9d0 0x7f99940d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.637+0000 7f999c400700 1 --2- 172.21.15.12:0/265316135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99940d16a0 0x7f99940d1e30 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.637+0000 7f999c400700 1 -- 172.21.15.12:0/265316135 >> 172.21.15.12:0/265316135 conn(0x7f999408aff0 msgr2=0x7f99940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:05.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.637+0000 7f999c400700 1 -- 172.21.15.12:0/265316135 shutdown_connections 2024-03-29T10:09:05.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.637+0000 7f999c400700 1 -- 172.21.15.12:0/265316135 wait complete. 2024-03-29T10:09:05.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.640+0000 7f999c400700 1 Processor -- start 2024-03-29T10:09:05.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.640+0000 7f999c400700 1 -- start start 2024-03-29T10:09:05.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f999c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99940d16a0 0x7f999415f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f999c400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99940d2370 0x7f99941f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f999c400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999415d9d0 0x7f99941f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f999c400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99940b62c0 con 0x7f999415d9d0 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f999c400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999415f680 con 0x7f99940d2370 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f999c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9994160a10 con 0x7f99940d16a0 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f9999a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99940d16a0 0x7f999415f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f9999000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99940d2370 0x7f99941f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f9999000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99940d2370 0x7f99941f38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:50190/0 (socket says 172.21.15.12:50190) 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f9999a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99940d16a0 0x7f999415f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:54616/0 (socket says 172.21.15.12:54616) 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f9999000700 1 -- 172.21.15.12:0/341461260 learned_addr learned my addr 172.21.15.12:0/341461260 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:05.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.641+0000 7f999a400700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999415d9d0 0x7f99941f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:05.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.642+0000 7f9999a00700 1 -- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99940d2370 msgr2=0x7f99941f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.642+0000 7f9999a00700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99940d2370 0x7f99941f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.642+0000 7f9999a00700 1 -- 172.21.15.12:0/341461260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999415d9d0 msgr2=0x7f99941f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.642+0000 7f9999a00700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999415d9d0 0x7f99941f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.642+0000 7f9999a00700 1 -- 172.21.15.12:0/341461260 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9980009430 con 0x7f99940d16a0 2024-03-29T10:09:05.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.642+0000 7f9999a00700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99940d16a0 0x7f999415f140 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f998800e870 tx=0x7f998800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.643+0000 7f9991600700 1 -- 172.21.15.12:0/341461260 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988013680 con 0x7f99940d16a0 2024-03-29T10:09:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.643+0000 7f9991600700 1 -- 172.21.15.12:0/341461260 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99880137e0 con 0x7f99940d16a0 2024-03-29T10:09:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.643+0000 7f9991600700 1 -- 172.21.15.12:0/341461260 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988013b90 con 0x7f99940d16a0 2024-03-29T10:09:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.643+0000 7f999c400700 1 -- 172.21.15.12:0/341461260 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99940b6510 con 0x7f99940d16a0 2024-03-29T10:09:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.643+0000 7f999c400700 1 -- 172.21.15.12:0/341461260 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99940b6ad0 con 0x7f99940d16a0 2024-03-29T10:09:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.645+0000 7f999c400700 1 -- 172.21.15.12:0/341461260 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99940d1ca0 con 0x7f99940d16a0 2024-03-29T10:09:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.645+0000 7f9991600700 1 -- 172.21.15.12:0/341461260 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f9988013cf0 con 0x7f99940d16a0 2024-03-29T10:09:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.646+0000 7f9991600700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f997406adb0 0x7f997406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.646+0000 7f9991600700 1 -- 172.21.15.12:0/341461260 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f99880893a0 con 0x7f99940d16a0 2024-03-29T10:09:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.646+0000 7f9999000700 1 -- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f997406adb0 msgr2=0x7f997406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.646+0000 7f9999000700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f997406adb0 0x7f997406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.654+0000 7f9991600700 1 -- 172.21.15.12:0/341461260 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9988054320 con 0x7f99940d16a0 2024-03-29T10:09:05.694 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:05.695+0000 7fcbd46e4240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:09:05.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.847+0000 7f9999000700 1 -- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f997406adb0 msgr2=0x7f997406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:05.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.847+0000 7f9999000700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f997406adb0 0x7f997406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.886+0000 7f999c400700 1 -- 172.21.15.12:0/341461260 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9994067720 con 0x7f99940d16a0 2024-03-29T10:09:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.887+0000 7f9991600700 1 -- 172.21.15.12:0/341461260 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v66) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9988057940 con 0x7f99940d16a0 2024-03-29T10:09:05.887 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:09:05.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 -- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f997406adb0 msgr2=0x7f997406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:05.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f997406adb0 0x7f997406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 -- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99940d16a0 msgr2=0x7f999415f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:05.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99940d16a0 0x7f999415f140 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f998800e870 tx=0x7f998800eb80 comp rx=0 tx=0).stop 2024-03-29T10:09:05.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 -- 172.21.15.12:0/341461260 shutdown_connections 2024-03-29T10:09:05.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f997406adb0 0x7f997406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999415d9d0 0x7f99941f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99940d16a0 0x7f999415f140 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 --2- 172.21.15.12:0/341461260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99940d2370 0x7f99941f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 -- 172.21.15.12:0/341461260 >> 172.21.15.12:0/341461260 conn(0x7f999408aff0 msgr2=0x7f999415c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 -- 172.21.15.12:0/341461260 shutdown_connections 2024-03-29T10:09:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:05.890+0000 7f999c400700 1 -- 172.21.15.12:0/341461260 wait complete. 2024-03-29T10:09:05.908 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:09:06.064 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:06.064+0000 7f018ebb1240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:09:06.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.115+0000 7f628b800700 1 Processor -- start 2024-03-29T10:09:06.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.115+0000 7f628b800700 1 -- start start 2024-03-29T10:09:06.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.116+0000 7f628b800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6284099a60 0x7f62840d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.116+0000 7f628b800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628409a750 0x7f628406c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.116+0000 7f628b800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62840d0970 con 0x7f628415d990 2024-03-29T10:09:06.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.116+0000 7f628b800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628406c020 con 0x7f6284099a60 2024-03-29T10:09:06.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.116+0000 7f628b800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62840c81a0 con 0x7f628409a750 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.116+0000 7f6288e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6284099a60 0x7f62840d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.117+0000 7f6288e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6284099a60 0x7f62840d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53402/0 (socket says 172.21.15.12:53402) 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.117+0000 7f6288e00700 1 -- 172.21.15.12:0/1688971673 learned_addr learned my addr 172.21.15.12:0/1688971673 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.117+0000 7f6283e00700 1 --1- 172.21.15.12:0/1688971673 >> v1:172.21.15.12:6789/0 conn(0x7f628415d990 0x7f6284099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35264/0 (socket says 172.21.15.12:35264) 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.117+0000 7f6289800700 1 --2- 172.21.15.12:0/1688971673 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628409a750 0x7f628406c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.117+0000 7f6288e00700 1 -- 172.21.15.12:0/1688971673 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628409a750 msgr2=0x7f628406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.117+0000 7f6288e00700 1 --2- 172.21.15.12:0/1688971673 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628409a750 0x7f628406c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.117+0000 7f6288e00700 1 -- 172.21.15.12:0/1688971673 >> v1:172.21.15.12:6789/0 conn(0x7f628415d990 legacy=0x7f6284099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.117+0000 7f6288e00700 1 -- 172.21.15.12:0/1688971673 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62840d0690 con 0x7f6284099a60 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.117+0000 7f6283400700 1 -- 172.21.15.12:0/1688971673 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936442998 0 0) 0x7f62840d0970 con 0x7f628415d990 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.117+0000 7f6288e00700 1 --2- 172.21.15.12:0/1688971673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6284099a60 0x7f62840d0060 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f62780098f0 tx=0x7f6278009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e3898d6129e90e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:06.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.118+0000 7f6283400700 1 -- 172.21.15.12:0/1688971673 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278011830 con 0x7f6284099a60 2024-03-29T10:09:06.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.118+0000 7f6283400700 1 -- 172.21.15.12:0/1688971673 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6278011990 con 0x7f6284099a60 2024-03-29T10:09:06.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.118+0000 7f6283400700 1 -- 172.21.15.12:0/1688971673 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278011d20 con 0x7f6284099a60 2024-03-29T10:09:06.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.118+0000 7f628b800700 1 -- 172.21.15.12:0/1688971673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6284099a60 msgr2=0x7f62840d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.118+0000 7f628b800700 1 --2- 172.21.15.12:0/1688971673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6284099a60 0x7f62840d0060 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f62780098f0 tx=0x7f6278009c00 comp rx=0 tx=0).stop 2024-03-29T10:09:06.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.118+0000 7f628b800700 1 -- 172.21.15.12:0/1688971673 shutdown_connections 2024-03-29T10:09:06.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.118+0000 7f628b800700 1 --2- 172.21.15.12:0/1688971673 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628409a750 0x7f628406c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.118+0000 7f628b800700 1 --2- 172.21.15.12:0/1688971673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6284099a60 0x7f62840d0060 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.118+0000 7f628b800700 1 -- 172.21.15.12:0/1688971673 >> 172.21.15.12:0/1688971673 conn(0x7f62840b2e20 msgr2=0x7f62840b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:06.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.119+0000 7f628b800700 1 -- 172.21.15.12:0/1688971673 shutdown_connections 2024-03-29T10:09:06.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.119+0000 7f628b800700 1 -- 172.21.15.12:0/1688971673 wait complete. 2024-03-29T10:09:06.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.120+0000 7f628b800700 1 Processor -- start 2024-03-29T10:09:06.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.121+0000 7f628b800700 1 -- start start 2024-03-29T10:09:06.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.121+0000 7f628b800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6284099a60 0x7f6284152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.121+0000 7f628b800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f628409a750 0x7f62840ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.121+0000 7f628b800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628415d990 0x7f62840cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.121+0000 7f628b800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62841f37b0 con 0x7f628409a750 2024-03-29T10:09:06.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.121+0000 7f628b800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6284152d90 con 0x7f6284099a60 2024-03-29T10:09:06.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f628b800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62841f2e10 con 0x7f628415d990 2024-03-29T10:09:06.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6283e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f628409a750 0x7f62840ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6283e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f628409a750 0x7f62840ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53412/0 (socket says 172.21.15.12:53412) 2024-03-29T10:09:06.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6283e00700 1 -- 172.21.15.12:0/3006142057 learned_addr learned my addr 172.21.15.12:0/3006142057 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:06.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6288e00700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6284099a60 0x7f6284152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6289800700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628415d990 0x7f62840cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6283e00700 1 -- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628415d990 msgr2=0x7f62840cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6283e00700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628415d990 0x7f62840cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6283e00700 1 -- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6284099a60 msgr2=0x7f6284152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6283e00700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6284099a60 0x7f6284152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6283e00700 1 -- 172.21.15.12:0/3006142057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278009550 con 0x7f628409a750 2024-03-29T10:09:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6289800700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628415d990 0x7f62840cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.122+0000 7f6283e00700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f628409a750 0x7f62840ca5d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f627000c550 tx=0x7f627000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.123+0000 7f6280c00700 1 -- 172.21.15.12:0/3006142057 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270004c80 con 0x7f628409a750 2024-03-29T10:09:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.123+0000 7f6280c00700 1 -- 172.21.15.12:0/3006142057 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6270004e00 con 0x7f628409a750 2024-03-29T10:09:06.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.123+0000 7f628b800700 1 -- 172.21.15.12:0/3006142057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62841f30f0 con 0x7f628409a750 2024-03-29T10:09:06.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.123+0000 7f6280c00700 1 -- 172.21.15.12:0/3006142057 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270007310 con 0x7f628409a750 2024-03-29T10:09:06.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.123+0000 7f628b800700 1 -- 172.21.15.12:0/3006142057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62841f2a80 con 0x7f628409a750 2024-03-29T10:09:06.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.124+0000 7f628b800700 1 -- 172.21.15.12:0/3006142057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62841f38f0 con 0x7f628409a750 2024-03-29T10:09:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.125+0000 7f6280c00700 1 -- 172.21.15.12:0/3006142057 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f6270007470 con 0x7f628409a750 2024-03-29T10:09:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.126+0000 7f6280c00700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f626406adb0 0x7f626406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.126+0000 7f6280c00700 1 -- 172.21.15.12:0/3006142057 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f6270089300 con 0x7f628409a750 2024-03-29T10:09:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.126+0000 7f6288e00700 1 -- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f626406adb0 msgr2=0x7f626406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.126+0000 7f6288e00700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f626406adb0 0x7f626406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.131+0000 7f6280c00700 1 -- 172.21.15.12:0/3006142057 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f62841f38f0 con 0x7f628409a750 2024-03-29T10:09:06.181 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:06.181+0000 7fcbd46e4240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:09:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.327+0000 7f6288e00700 1 -- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f626406adb0 msgr2=0x7f626406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.327+0000 7f6288e00700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f626406adb0 0x7f626406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:06.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.345+0000 7f628b800700 1 -- 172.21.15.12:0/3006142057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f62841f38f0 con 0x7f628409a750 2024-03-29T10:09:06.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.345+0000 7f6280c00700 1 -- 172.21.15.12:0/3006142057 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v66) v1 ==== 100+0+11 (secure 0 0 0) 0x7f62841f38f0 con 0x7f628409a750 2024-03-29T10:09:06.345 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:09:06.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 -- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f626406adb0 msgr2=0x7f626406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:06.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f626406adb0 0x7f626406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 -- 172.21.15.12:0/3006142057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f628409a750 msgr2=0x7f62840ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f628409a750 0x7f62840ca5d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f627000c550 tx=0x7f627000c860 comp rx=0 tx=0).stop 2024-03-29T10:09:06.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 -- 172.21.15.12:0/3006142057 shutdown_connections 2024-03-29T10:09:06.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f626406adb0 0x7f626406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f628415d990 0x7f62840cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6284099a60 0x7f6284152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 --2- 172.21.15.12:0/3006142057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f628409a750 0x7f62840ca5d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 -- 172.21.15.12:0/3006142057 >> 172.21.15.12:0/3006142057 conn(0x7f62840b2e20 msgr2=0x7f62840cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:06.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 -- 172.21.15.12:0/3006142057 shutdown_connections 2024-03-29T10:09:06.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.348+0000 7f628b800700 1 -- 172.21.15.12:0/3006142057 wait complete. 2024-03-29T10:09:06.367 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T10:09:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.583+0000 7fbb57a00700 1 Processor -- start 2024-03-29T10:09:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.583+0000 7fbb57a00700 1 -- start start 2024-03-29T10:09:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.584+0000 7fbb57a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb501687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.584+0000 7fbb57a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb50161420 con 0x7fbb500d16a0 2024-03-29T10:09:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.584+0000 7fbb57a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb50161700 con 0x7fbb5015d9d0 2024-03-29T10:09:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.584+0000 7fbb57a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb500d4340 con 0x7fbb500d2540 2024-03-29T10:09:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.584+0000 7fbb4fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbb500d16a0 0x7fbb500d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35276/0 (socket says 172.21.15.12:35276) 2024-03-29T10:09:06.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.584+0000 7fbb4fe00700 1 -- 172.21.15.12:0/1879312155 learned_addr learned my addr 172.21.15.12:0/1879312155 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:06.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb55a00700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb501687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518094335 0 0) 0x7fbb50161420 con 0x7fbb500d16a0 2024-03-29T10:09:06.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb380035b0 con 0x7fbb500d16a0 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641233135 0 0) 0x7fbb50161700 con 0x7fbb5015d9d0 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50161420 con 0x7fbb5015d9d0 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278073976 0 0) 0x7fbb380035b0 con 0x7fbb500d16a0 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 msgr2=0x7fbb501687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb501687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 >> v1:172.21.15.159:6789/0 conn(0x7fbb5015d9d0 legacy=0x7fbb500d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb501619e0 con 0x7fbb500d16a0 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fbb3c0017f0 con 0x7fbb500d16a0 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 >> v1:172.21.15.12:6789/0 conn(0x7fbb500d16a0 legacy=0x7fbb500d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb38003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb38004120 0x7fbb38004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb38004d90 0x7fbb38005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3800bac0 con 0x7fbb38004120 2024-03-29T10:09:06.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3800bc30 con 0x7fbb38004d90 2024-03-29T10:09:06.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.585+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3800bdf0 con 0x7fbb500d2540 2024-03-29T10:09:06.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.586+0000 7fbb55000700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb38003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.586+0000 7fbb4fe00700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb38004120 0x7fbb38004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.586+0000 7fbb55a00700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb38004d90 0x7fbb38005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.586+0000 7fbb55000700 1 -- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb38004d90 msgr2=0x7fbb38005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:06.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.586+0000 7fbb55000700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb38004d90 0x7fbb38005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.586+0000 7fbb55000700 1 -- 172.21.15.12:0/1879312155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb38004120 msgr2=0x7fbb38004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.586+0000 7fbb55000700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb38004120 0x7fbb38004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.586+0000 7fbb55000700 1 -- 172.21.15.12:0/1879312155 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb380035b0 con 0x7fbb500d2540 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.587+0000 7fbb55000700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb38003bc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbb44001940 tx=0x7fbb38005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=793096aee528e024 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.587+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44001430 con 0x7fbb500d2540 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.587+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb44001d70 con 0x7fbb500d2540 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.587+0000 7fbb4f400700 1 -- 172.21.15.12:0/1879312155 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44005510 con 0x7fbb500d2540 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.587+0000 7fbb57a00700 1 -- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 msgr2=0x7fbb38003bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.587+0000 7fbb57a00700 1 --2- 172.21.15.12:0/1879312155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb38003bc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbb44001940 tx=0x7fbb38005880 comp rx=0 tx=0).stop 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.587+0000 7fbb55a00700 1 -- 172.21.15.12:0/1879312155 reap_dead start 2024-03-29T10:09:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.587+0000 7fbb57a00700 1 -- 172.21.15.12:0/1879312155 shutdown_connections 2024-03-29T10:09:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.587+0000 7fbb57a00700 1 -- 172.21.15.12:0/1879312155 >> 172.21.15.12:0/1879312155 conn(0x7fbb5008aff0 msgr2=0x7fbb500d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.588+0000 7fbb57a00700 1 -- 172.21.15.12:0/1879312155 shutdown_connections 2024-03-29T10:09:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.588+0000 7fbb57a00700 1 -- 172.21.15.12:0/1879312155 wait complete. 2024-03-29T10:09:06.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.589+0000 7fbb57a00700 1 Processor -- start 2024-03-29T10:09:06.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.589+0000 7fbb57a00700 1 -- start start 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.590+0000 7fbb57a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb501e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.590+0000 7fbb57a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb5015f810 0x7fbb5015e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.590+0000 7fbb57a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb5015e8a0 0x7fbb5015ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.590+0000 7fbb57a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb501e78d0 con 0x7fbb5015e8a0 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.590+0000 7fbb57a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb50147ae0 con 0x7fbb5015f810 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.590+0000 7fbb57a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb50147dc0 con 0x7fbb500d2540 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb55a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb5015e8a0 0x7fbb5015ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb55a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb5015e8a0 0x7fbb5015ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53432/0 (socket says 172.21.15.12:53432) 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb55a00700 1 -- 172.21.15.12:0/251258291 learned_addr learned my addr 172.21.15.12:0/251258291 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb55000700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb501e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb4fe00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb5015f810 0x7fbb5015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb55000700 1 -- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb5015f810 msgr2=0x7fbb5015e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb55000700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb5015f810 0x7fbb5015e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb55000700 1 -- 172.21.15.12:0/251258291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb5015e8a0 msgr2=0x7fbb5015ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb55000700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb5015e8a0 0x7fbb5015ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb55000700 1 -- 172.21.15.12:0/251258291 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb501480a0 con 0x7fbb500d2540 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.591+0000 7fbb55a00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb5015e8a0 0x7fbb5015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.592+0000 7fbb55000700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb501e7280 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbb44014000 tx=0x7fbb440012d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.592+0000 7fbb4cc00700 1 -- 172.21.15.12:0/251258291 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44006c70 con 0x7fbb500d2540 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.592+0000 7fbb4cc00700 1 -- 172.21.15.12:0/251258291 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb44006dd0 con 0x7fbb500d2540 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.592+0000 7fbb4cc00700 1 -- 172.21.15.12:0/251258291 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44007260 con 0x7fbb500d2540 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.592+0000 7fbb57a00700 1 -- 172.21.15.12:0/251258291 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb50148380 con 0x7fbb500d2540 2024-03-29T10:09:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.592+0000 7fbb57a00700 1 -- 172.21.15.12:0/251258291 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb500ca890 con 0x7fbb500d2540 2024-03-29T10:09:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.593+0000 7fbb57a00700 1 -- 172.21.15.12:0/251258291 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb500d1ca0 con 0x7fbb500d2540 2024-03-29T10:09:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.594+0000 7fbb4cc00700 1 -- 172.21.15.12:0/251258291 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fbb440073c0 con 0x7fbb500d2540 2024-03-29T10:09:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.595+0000 7fbb4cc00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb3006adb0 0x7fbb3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.595+0000 7fbb4cc00700 1 -- 172.21.15.12:0/251258291 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7fbb4408a5b0 con 0x7fbb500d2540 2024-03-29T10:09:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.595+0000 7fbb4fe00700 1 -- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb3006adb0 msgr2=0x7fbb3006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.595+0000 7fbb4fe00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb3006adb0 0x7fbb3006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:06.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.600+0000 7fbb4cc00700 1 -- 172.21.15.12:0/251258291 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbb440554c0 con 0x7fbb500d2540 2024-03-29T10:09:06.781 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:06.781+0000 7f018ebb1240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:09:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.796+0000 7fbb4fe00700 1 -- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb3006adb0 msgr2=0x7fbb3006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.796+0000 7fbb4fe00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb3006adb0 0x7fbb3006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:06.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.830+0000 7fbb57a00700 1 -- 172.21.15.12:0/251258291 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbb50067720 con 0x7fbb500d2540 2024-03-29T10:09:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.830+0000 7fbb4cc00700 1 -- 172.21.15.12:0/251258291 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v66) v1 ==== 80+0+467 (secure 0 0 0) 0x7fbb44058ae0 con 0x7fbb500d2540 2024-03-29T10:09:06.830 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/43444 2024-03-30T10:07:25.860642+0000 2024-03-29T10:09:06.830 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/43444 2024-03-30T10:07:25.860642+0000 2024-03-29T10:09:06.830 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3220343982 2024-03-30T10:07:25.860642+0000 2024-03-29T10:09:06.830 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2775778725 2024-03-30T10:07:25.860642+0000 2024-03-29T10:09:06.830 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3363559414 2024-03-30T10:07:25.860642+0000 2024-03-29T10:09:06.831 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1116927207 2024-03-30T10:07:25.860642+0000 2024-03-29T10:09:06.831 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3950908519 2024-03-30T10:07:25.860642+0000 2024-03-29T10:09:06.831 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/798085437 2024-03-30T10:07:25.860642+0000 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 -- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb3006adb0 msgr2=0x7fbb3006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb3006adb0 0x7fbb3006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 -- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 msgr2=0x7fbb501e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb501e7280 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbb44014000 tx=0x7fbb440012d0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 -- 172.21.15.12:0/251258291 shutdown_connections 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb3006adb0 0x7fbb3006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb5015e8a0 0x7fbb5015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb500d2540 0x7fbb501e7280 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 --2- 172.21.15.12:0/251258291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb5015f810 0x7fbb5015e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 -- 172.21.15.12:0/251258291 >> 172.21.15.12:0/251258291 conn(0x7fbb5008aff0 msgr2=0x7fbb500b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 -- 172.21.15.12:0/251258291 shutdown_connections 2024-03-29T10:09:06.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:06.833+0000 7fbb57a00700 1 -- 172.21.15.12:0/251258291 wait complete. 2024-03-29T10:09:06.834 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T10:09:06.851 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T10:09:06.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:06.851 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T10:09:06.874 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:06.874+0000 7f018ebb1240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:09:06.927 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T10:09:06.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:06.927 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T10:09:06.958 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:06.959+0000 7fcbd46e4240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:09:06.972 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:09:06.972 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:07.017 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:09:07.017 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:09:07.017 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:07.039 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:07.040+0000 7f018ebb1240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:09:07.055 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:07.056+0000 7fcbd46e4240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:09:07.062 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:09:07.062 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:09:07.062 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:07.107 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:09:07.107 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:09:07.231 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:07.231+0000 7fcbd46e4240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:09:07.258 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:07.258+0000 7f018ebb1240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:09:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.314+0000 7f4c86200700 1 Processor -- start 2024-03-29T10:09:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.314+0000 7f4c86200700 1 -- start start 2024-03-29T10:09:07.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.315+0000 7f4c86200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4c800cb3f0 0x7f4c800ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.315+0000 7f4c86200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4c800ac600 0x7f4c8006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.315+0000 7f4c86200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8015fd50 con 0x7f4c8015b5d0 2024-03-29T10:09:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.315+0000 7f4c86200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8015b9b0 con 0x7f4c800ac600 2024-03-29T10:09:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.315+0000 7f4c86200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c800aca10 con 0x7f4c800cb3f0 2024-03-29T10:09:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.315+0000 7f4c7fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4c800ac600 0x7f4c8006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.315+0000 7f4c7f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4c8015b5d0 0x7f4c800ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35284/0 (socket says 172.21.15.12:35284) 2024-03-29T10:09:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.315+0000 7f4c7f400700 1 -- 172.21.15.12:0/2607483779 learned_addr learned my addr 172.21.15.12:0/2607483779 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.315+0000 7f4c7ea00700 1 --2- 172.21.15.12:0/2607483779 >> v2:172.21.15.159:3300/0 conn(0x7f4c800cb3f0 0x7f4c800ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.316+0000 7f4c7fe00700 1 -- 172.21.15.12:0/2607483779 >> v2:172.21.15.159:3300/0 conn(0x7f4c800cb3f0 msgr2=0x7f4c800ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.316+0000 7f4c7fe00700 1 --2- 172.21.15.12:0/2607483779 >> v2:172.21.15.159:3300/0 conn(0x7f4c800cb3f0 0x7f4c800ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.316+0000 7f4c7fe00700 1 -- 172.21.15.12:0/2607483779 >> v1:172.21.15.12:6789/0 conn(0x7f4c8015b5d0 legacy=0x7f4c800ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.316+0000 7f4c7fe00700 1 -- 172.21.15.12:0/2607483779 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8015f140 con 0x7f4c800ac600 2024-03-29T10:09:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.316+0000 7f4c7e000700 1 -- 172.21.15.12:0/2607483779 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746578150 0 0) 0x7f4c8015fd50 con 0x7f4c8015b5d0 2024-03-29T10:09:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.316+0000 7f4c7fe00700 1 --2- 172.21.15.12:0/2607483779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c800ac600 0x7f4c8006c9d0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4c8006c6c0 tx=0x7f4c7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ef052328b9b05f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.316+0000 7f4c7e000700 1 -- 172.21.15.12:0/2607483779 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7800e070 con 0x7f4c800ac600 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.316+0000 7f4c7e000700 1 -- 172.21.15.12:0/2607483779 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7800bb20 con 0x7f4c800ac600 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.317+0000 7f4c7e000700 1 -- 172.21.15.12:0/2607483779 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78012430 con 0x7f4c800ac600 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.317+0000 7f4c86200700 1 -- 172.21.15.12:0/2607483779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c800ac600 msgr2=0x7f4c8006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.317+0000 7f4c86200700 1 --2- 172.21.15.12:0/2607483779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c800ac600 0x7f4c8006c9d0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4c8006c6c0 tx=0x7f4c7800b330 comp rx=0 tx=0).stop 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.317+0000 7f4c86200700 1 -- 172.21.15.12:0/2607483779 shutdown_connections 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.317+0000 7f4c86200700 1 --2- 172.21.15.12:0/2607483779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c800ac600 0x7f4c8006c9d0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.317+0000 7f4c86200700 1 --2- 172.21.15.12:0/2607483779 >> v2:172.21.15.159:3300/0 conn(0x7f4c800cb3f0 0x7f4c800ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.317+0000 7f4c86200700 1 -- 172.21.15.12:0/2607483779 >> 172.21.15.12:0/2607483779 conn(0x7f4c800cf190 msgr2=0x7f4c800cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.317+0000 7f4c86200700 1 -- 172.21.15.12:0/2607483779 shutdown_connections 2024-03-29T10:09:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.317+0000 7f4c86200700 1 -- 172.21.15.12:0/2607483779 wait complete. 2024-03-29T10:09:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.319+0000 7f4c86200700 1 Processor -- start 2024-03-29T10:09:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.319+0000 7f4c86200700 1 -- start start 2024-03-29T10:09:07.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.320+0000 7f4c86200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c800ac600 0x7f4c80144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.320+0000 7f4c86200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c8015b5d0 0x7f4c800b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.320+0000 7f4c86200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c800cac80 0x7f4c800b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.320+0000 7f4c86200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c800ca5a0 con 0x7f4c8015b5d0 2024-03-29T10:09:07.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.320+0000 7f4c86200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8015eac0 con 0x7f4c800cac80 2024-03-29T10:09:07.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.320+0000 7f4c86200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c800cdad0 con 0x7f4c800ac600 2024-03-29T10:09:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c8015b5d0 0x7f4c800b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c8015b5d0 0x7f4c800b2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53460/0 (socket says 172.21.15.12:53460) 2024-03-29T10:09:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7ea00700 1 -- 172.21.15.12:0/3040806527 learned_addr learned my addr 172.21.15.12:0/3040806527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7f400700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c800ac600 0x7f4c80144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7fe00700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c800cac80 0x7f4c800b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7ea00700 1 -- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c800ac600 msgr2=0x7f4c80144710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7ea00700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c800ac600 0x7f4c80144710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7ea00700 1 -- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c800cac80 msgr2=0x7f4c800b34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7ea00700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c800cac80 0x7f4c800b34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7ea00700 1 -- 172.21.15.12:0/3040806527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c78009a70 con 0x7f4c8015b5d0 2024-03-29T10:09:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.321+0000 7f4c7ea00700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c8015b5d0 0x7f4c800b2f80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00e9e0 tx=0x7f4c6c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.322+0000 7f4c77400700 1 -- 172.21.15.12:0/3040806527 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c009710 con 0x7f4c8015b5d0 2024-03-29T10:09:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.322+0000 7f4c77400700 1 -- 172.21.15.12:0/3040806527 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6c00ee80 con 0x7f4c8015b5d0 2024-03-29T10:09:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.322+0000 7f4c77400700 1 -- 172.21.15.12:0/3040806527 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c00ca50 con 0x7f4c8015b5d0 2024-03-29T10:09:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.322+0000 7f4c86200700 1 -- 172.21.15.12:0/3040806527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c80144c50 con 0x7f4c8015b5d0 2024-03-29T10:09:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.322+0000 7f4c86200700 1 -- 172.21.15.12:0/3040806527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c800cd760 con 0x7f4c8015b5d0 2024-03-29T10:09:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.323+0000 7f4c86200700 1 -- 172.21.15.12:0/3040806527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c800cd5c0 con 0x7f4c8015b5d0 2024-03-29T10:09:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.324+0000 7f4c77400700 1 -- 172.21.15.12:0/3040806527 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f4c6c00cbb0 con 0x7f4c8015b5d0 2024-03-29T10:09:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.325+0000 7f4c77400700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c6006adb0 0x7f4c6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.325+0000 7f4c77400700 1 -- 172.21.15.12:0/3040806527 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f4c6c089730 con 0x7f4c8015b5d0 2024-03-29T10:09:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.325+0000 7f4c7f400700 1 -- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c6006adb0 msgr2=0x7f4c6006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.325+0000 7f4c7f400700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c6006adb0 0x7f4c6006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:07.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.330+0000 7f4c77400700 1 -- 172.21.15.12:0/3040806527 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c800cd5c0 con 0x7f4c8015b5d0 2024-03-29T10:09:07.353 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:07.352+0000 7f018ebb1240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:09:07.451 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:07.451+0000 7f018ebb1240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:09:07.454 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:07.454+0000 7fcbd46e4240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:09:07.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.526+0000 7f4c7f400700 1 -- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c6006adb0 msgr2=0x7f4c6006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:07.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.526+0000 7f4c7f400700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c6006adb0 0x7f4c6006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:07.549 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:07.550+0000 7f018ebb1240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:09:07.550 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:07.551+0000 7fcbd46e4240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:09:07.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.595+0000 7f4c86200700 1 -- 172.21.15.12:0/3040806527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c800cd5c0 con 0x7f4c8015b5d0 2024-03-29T10:09:07.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.596+0000 7f4c77400700 1 -- 172.21.15.12:0/3040806527 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f4c800cd5c0 con 0x7f4c8015b5d0 2024-03-29T10:09:07.596 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:07.596 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5746,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/981039791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":981039791},{"type":"v1","addr":"172.21.15.159:6836","nonce":981039791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1682878615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1682878615},{"type":"v1","addr":"172.21.15.159:6837","nonce":1682878615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5752,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1663013205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":1663013205},{"type":"v1","addr":"172.21.15.12:6834","nonce":1663013205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:07:08.705772+0000","modified":"2024-03-29T10:07:12.186244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5877},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5877":{"gid":5877,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3970376731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3970376731},{"type":"v1","addr":"172.21.15.12:6835","nonce":3970376731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-29T10:09:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 -- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c6006adb0 msgr2=0x7f4c6006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c6006adb0 0x7f4c6006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 -- 172.21.15.12:0/3040806527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c8015b5d0 msgr2=0x7f4c800b2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c8015b5d0 0x7f4c800b2f80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00e9e0 tx=0x7f4c6c00ecf0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 -- 172.21.15.12:0/3040806527 shutdown_connections 2024-03-29T10:09:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4c6006adb0 0x7f4c6006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c800cac80 0x7f4c800b34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c800ac600 0x7f4c80144710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 --2- 172.21.15.12:0/3040806527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c8015b5d0 0x7f4c800b2f80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 -- 172.21.15.12:0/3040806527 >> 172.21.15.12:0/3040806527 conn(0x7f4c800cf190 msgr2=0x7f4c800afa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:07.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 -- 172.21.15.12:0/3040806527 shutdown_connections 2024-03-29T10:09:07.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.599+0000 7f4c86200700 1 -- 172.21.15.12:0/3040806527 wait complete. 2024-03-29T10:09:07.600 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-29T10:09:07.618 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:09:07.648 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:07.648+0000 7f018ebb1240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:09:07.660 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:07.661+0000 7fcbd46e4240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:09:07.769 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:07.770+0000 7fcbd46e4240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:09:07.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.820+0000 7f0ba3800700 1 Processor -- start 2024-03-29T10:09:07.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.821+0000 7f0ba3800700 1 -- start start 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.822+0000 7f0ba3800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0b9c072580 0x7f0b9c0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.822+0000 7f0ba3800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c0b2180 con 0x7f0b9c0b2e50 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.822+0000 7f0ba3800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c15b100 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.822+0000 7f0ba3800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9c15b3e0 con 0x7f0b9c072580 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.822+0000 7f0b9be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0b9c072580 0x7f0b9c0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.822+0000 7f0ba1800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0b9c0b2e50 0x7f0b9c0709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35292/0 (socket says 172.21.15.12:35292) 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.822+0000 7f0ba1800700 1 -- 172.21.15.12:0/3472884909 learned_addr learned my addr 172.21.15.12:0/3472884909 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.822+0000 7f0b9be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0b9c072580 0x7f0b9c0b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53466/0 (socket says 172.21.15.12:53466) 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.823+0000 7f0b9be00700 1 -- 172.21.15.12:0/3472884909 >> v1:172.21.15.159:6789/0 conn(0x7f0b9c0b3af0 legacy=0x7f0b9c15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.823+0000 7f0b9be00700 1 -- 172.21.15.12:0/3472884909 >> v1:172.21.15.12:6789/0 conn(0x7f0b9c0b2e50 legacy=0x7f0b9c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:07.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.823+0000 7f0b9be00700 1 -- 172.21.15.12:0/3472884909 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c0b23f0 con 0x7f0b9c072580 2024-03-29T10:09:07.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.823+0000 7f0b9b400700 1 -- 172.21.15.12:0/3472884909 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228800612 0 0) 0x7f0b9c0b2180 con 0x7f0b9c0b2e50 2024-03-29T10:09:07.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.823+0000 7f0b9be00700 1 --2- 172.21.15.12:0/3472884909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c072580 0x7f0b9c0b2910 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0b88009840 tx=0x7f0b88009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f919faa24cc76c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:07.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.823+0000 7f0b9b400700 1 -- 172.21.15.12:0/3472884909 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88011830 con 0x7f0b9c072580 2024-03-29T10:09:07.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.823+0000 7f0b9b400700 1 -- 172.21.15.12:0/3472884909 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b88011990 con 0x7f0b9c072580 2024-03-29T10:09:07.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.823+0000 7f0b9b400700 1 -- 172.21.15.12:0/3472884909 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88011d20 con 0x7f0b9c072580 2024-03-29T10:09:07.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.824+0000 7f0ba3800700 1 -- 172.21.15.12:0/3472884909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c072580 msgr2=0x7f0b9c0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:07.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.824+0000 7f0ba3800700 1 --2- 172.21.15.12:0/3472884909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c072580 0x7f0b9c0b2910 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0b88009840 tx=0x7f0b88009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:07.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.824+0000 7f0ba3800700 1 -- 172.21.15.12:0/3472884909 shutdown_connections 2024-03-29T10:09:07.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.824+0000 7f0ba3800700 1 --2- 172.21.15.12:0/3472884909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c072580 0x7f0b9c0b2910 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.824+0000 7f0ba3800700 1 -- 172.21.15.12:0/3472884909 >> 172.21.15.12:0/3472884909 conn(0x7f0b9c0cf190 msgr2=0x7f0b9c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:07.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.824+0000 7f0ba3800700 1 -- 172.21.15.12:0/3472884909 shutdown_connections 2024-03-29T10:09:07.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.824+0000 7f0ba3800700 1 -- 172.21.15.12:0/3472884909 wait complete. 2024-03-29T10:09:07.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.826+0000 7f0ba3800700 1 Processor -- start 2024-03-29T10:09:07.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.826+0000 7f0ba3800700 1 -- start start 2024-03-29T10:09:07.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0ba3800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b9c072580 0x7f0b9c149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0ba3800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b9c0b2e50 0x7f0b9c14a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0ba3800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c0b3af0 0x7f0b9c159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0ba3800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c0c8920 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0ba3800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c14a310 con 0x7f0b9c0b2e50 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0ba3800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c14ada0 con 0x7f0b9c072580 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0ba1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c0b3af0 0x7f0b9c159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0ba1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c0b3af0 0x7f0b9c159410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53478/0 (socket says 172.21.15.12:53478) 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0ba1800700 1 -- 172.21.15.12:0/595140349 learned_addr learned my addr 172.21.15.12:0/595140349 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0b9be00700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b9c0b2e50 0x7f0b9c14a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.827+0000 7f0ba0e00700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b9c072580 0x7f0b9c149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0ba1800700 1 -- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b9c072580 msgr2=0x7f0b9c149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0ba1800700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b9c072580 0x7f0b9c149dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0ba1800700 1 -- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b9c0b2e50 msgr2=0x7f0b9c14a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:07.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0ba1800700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b9c0b2e50 0x7f0b9c14a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:07.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0ba1800700 1 -- 172.21.15.12:0/595140349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b880094a0 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0ba1800700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c0b3af0 0x7f0b9c159410 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0b94002230 tx=0x7f0b9400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:07.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0b98c00700 1 -- 172.21.15.12:0/595140349 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9400ab70 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0b98c00700 1 -- 172.21.15.12:0/595140349 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9400acf0 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0ba3800700 1 -- 172.21.15.12:0/595140349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c0c8b50 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0b98c00700 1 -- 172.21.15.12:0/595140349 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94007310 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.828+0000 7f0ba3800700 1 -- 172.21.15.12:0/595140349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c0c9110 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.829+0000 7f0ba3800700 1 -- 172.21.15.12:0/595140349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c067720 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.830+0000 7f0b98c00700 1 -- 172.21.15.12:0/595140349 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f0b94007470 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.831+0000 7f0b98c00700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b7c06adb0 0x7f0b7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:07.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.831+0000 7f0b98c00700 1 -- 172.21.15.12:0/595140349 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f0b940894c0 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.831+0000 7f0ba0e00700 1 -- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b7c06adb0 msgr2=0x7f0b7c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:07.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.831+0000 7f0ba0e00700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b7c06adb0 0x7f0b7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:07.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:07.836+0000 7f0b98c00700 1 -- 172.21.15.12:0/595140349 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b940543a0 con 0x7f0b9c0b3af0 2024-03-29T10:09:07.862 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:07.862+0000 7f018ebb1240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:09:07.878 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:07.879+0000 7fcbd46e4240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:09:08.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.032+0000 7f0ba0e00700 1 -- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b7c06adb0 msgr2=0x7f0b7c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:08.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.032+0000 7f0ba0e00700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b7c06adb0 0x7f0b7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.053+0000 7f0ba3800700 1 -- 172.21.15.12:0/595140349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b9c1ea300 con 0x7f0b9c0b3af0 2024-03-29T10:09:08.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.054+0000 7f0b98c00700 1 -- 172.21.15.12:0/595140349 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16769 (secure 0 0 0) 0x7f0b9400ae60 con 0x7f0b9c0b3af0 2024-03-29T10:09:08.054 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:08.054 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":66,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:05.263226+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:07:07.106799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-29T10:07:07.671044+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":5}]},{"pgid":"7.12","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6833/43444":"2024-03-30T10:07:25.860642+0000","172.21.15.159:6832/43444":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3220343982":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/2775778725":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3363559414":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/1116927207":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3950908519":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/798085437":"2024-03-30T10:07:25.860642+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:09:08.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.056+0000 7f0ba3800700 1 -- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b7c06adb0 msgr2=0x7f0b7c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.056+0000 7f0ba3800700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b7c06adb0 0x7f0b7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.056+0000 7f0ba3800700 1 -- 172.21.15.12:0/595140349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c0b3af0 msgr2=0x7f0b9c159410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.056+0000 7f0ba3800700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c0b3af0 0x7f0b9c159410 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0b94002230 tx=0x7f0b9400cf50 comp rx=0 tx=0).stop 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.057+0000 7f0ba3800700 1 -- 172.21.15.12:0/595140349 shutdown_connections 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.057+0000 7f0ba3800700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b7c06adb0 0x7f0b7c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.057+0000 7f0ba3800700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b9c0b3af0 0x7f0b9c159410 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.057+0000 7f0ba3800700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b9c072580 0x7f0b9c149dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.057+0000 7f0ba3800700 1 --2- 172.21.15.12:0/595140349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b9c0b2e50 0x7f0b9c14a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.057+0000 7f0ba3800700 1 -- 172.21.15.12:0/595140349 >> 172.21.15.12:0/595140349 conn(0x7f0b9c0cf190 msgr2=0x7f0b9c0713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.057+0000 7f0ba3800700 1 -- 172.21.15.12:0/595140349 shutdown_connections 2024-03-29T10:09:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.057+0000 7f0ba3800700 1 -- 172.21.15.12:0/595140349 wait complete. 2024-03-29T10:09:08.076 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:07:07.106799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:07:07.671044+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 66, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:09:08.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:09:08.105 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:08.105+0000 7fcbd46e4240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:09:08.119 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:08.119+0000 7f018ebb1240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:09:08.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.314+0000 7fc306c00700 1 Processor -- start 2024-03-29T10:09:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.314+0000 7fc306c00700 1 -- start start 2024-03-29T10:09:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.315+0000 7fc306c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc3000d21f0 0x7fc3000ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.315+0000 7fc306c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3000ce270 con 0x7fc3001611b0 2024-03-29T10:09:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.315+0000 7fc306c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30015b600 con 0x7fc3000cec40 2024-03-29T10:09:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.315+0000 7fc306c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3000d2760 con 0x7fc3000d21f0 2024-03-29T10:09:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.315+0000 7fc2ff400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc3000d21f0 0x7fc3000ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.315+0000 7fc2ff400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc3000d21f0 0x7fc3000ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53482/0 (socket says 172.21.15.12:53482) 2024-03-29T10:09:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.315+0000 7fc2ff400700 1 -- 172.21.15.12:0/346849032 learned_addr learned my addr 172.21.15.12:0/346849032 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:08.316 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:08.315+0000 7f018ebb1240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:09:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.316+0000 7fc2ff400700 1 -- 172.21.15.12:0/346849032 >> v1:172.21.15.159:6789/0 conn(0x7fc3000cec40 legacy=0x7fc300161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.316+0000 7fc2ff400700 1 -- 172.21.15.12:0/346849032 >> v1:172.21.15.12:6789/0 conn(0x7fc3001611b0 legacy=0x7fc30015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.316+0000 7fc2ff400700 1 -- 172.21.15.12:0/346849032 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3000d2a40 con 0x7fc3000d21f0 2024-03-29T10:09:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.316+0000 7fc2fea00700 1 -- 172.21.15.12:0/346849032 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522382449 0 0) 0x7fc3000ce270 con 0x7fc3001611b0 2024-03-29T10:09:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.316+0000 7fc2ff400700 1 --2- 172.21.15.12:0/346849032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000d21f0 0x7fc3000ce700 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0098f0 tx=0x7fc2ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae59cdb8da3de282 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.316+0000 7fc2fea00700 1 -- 172.21.15.12:0/346849032 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00b8a0 con 0x7fc3000d21f0 2024-03-29T10:09:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.316+0000 7fc2fea00700 1 -- 172.21.15.12:0/346849032 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ec00ba00 con 0x7fc3000d21f0 2024-03-29T10:09:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.316+0000 7fc2fea00700 1 -- 172.21.15.12:0/346849032 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00bd90 con 0x7fc3000d21f0 2024-03-29T10:09:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.317+0000 7fc306c00700 1 -- 172.21.15.12:0/346849032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000d21f0 msgr2=0x7fc3000ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.317+0000 7fc306c00700 1 --2- 172.21.15.12:0/346849032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000d21f0 0x7fc3000ce700 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0098f0 tx=0x7fc2ec009c00 comp rx=0 tx=0).stop 2024-03-29T10:09:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.317+0000 7fc306c00700 1 -- 172.21.15.12:0/346849032 shutdown_connections 2024-03-29T10:09:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.317+0000 7fc306c00700 1 --2- 172.21.15.12:0/346849032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000d21f0 0x7fc3000ce700 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.317+0000 7fc306c00700 1 -- 172.21.15.12:0/346849032 >> 172.21.15.12:0/346849032 conn(0x7fc30008aff0 msgr2=0x7fc3000d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.317+0000 7fc306c00700 1 -- 172.21.15.12:0/346849032 shutdown_connections 2024-03-29T10:09:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.317+0000 7fc306c00700 1 -- 172.21.15.12:0/346849032 wait complete. 2024-03-29T10:09:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.319+0000 7fc306c00700 1 Processor -- start 2024-03-29T10:09:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.320+0000 7fc306c00700 1 -- start start 2024-03-29T10:09:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc306c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000cec40 0x7fc3001e8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc306c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3000d21f0 0x7fc3001e6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc306c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3001611b0 0x7fc3000b7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc306c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3000cc580 con 0x7fc3000cec40 2024-03-29T10:09:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc306c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3001e9f00 con 0x7fc3000d21f0 2024-03-29T10:09:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc306c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3000cb970 con 0x7fc3001611b0 2024-03-29T10:09:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc2ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000cec40 0x7fc3001e8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc2ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000cec40 0x7fc3001e8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53492/0 (socket says 172.21.15.12:53492) 2024-03-29T10:09:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc2ffe00700 1 -- 172.21.15.12:0/142130665 learned_addr learned my addr 172.21.15.12:0/142130665 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc2ff400700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3000d21f0 0x7fc3001e6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.321+0000 7fc304c00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3001611b0 0x7fc3000b7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.322+0000 7fc2ffe00700 1 -- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3001611b0 msgr2=0x7fc3000b7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.322+0000 7fc2ffe00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3001611b0 0x7fc3000b7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.322+0000 7fc2ffe00700 1 -- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3000d21f0 msgr2=0x7fc3001e6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.322+0000 7fc2ffe00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3000d21f0 0x7fc3001e6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.322+0000 7fc2ffe00700 1 -- 172.21.15.12:0/142130665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec009550 con 0x7fc3000cec40 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.322+0000 7fc2ffe00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000cec40 0x7fc3001e8f50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0001eb0 tx=0x7fc2f000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.322+0000 7fc2f7e00700 1 -- 172.21.15.12:0/142130665 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00054e0 con 0x7fc3000cec40 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.322+0000 7fc2f7e00700 1 -- 172.21.15.12:0/142130665 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f0003680 con 0x7fc3000cec40 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.322+0000 7fc306c00700 1 -- 172.21.15.12:0/142130665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3000cbc50 con 0x7fc3000cec40 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.322+0000 7fc2f7e00700 1 -- 172.21.15.12:0/142130665 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00039f0 con 0x7fc3000cec40 2024-03-29T10:09:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.323+0000 7fc306c00700 1 -- 172.21.15.12:0/142130665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3000cc210 con 0x7fc3000cec40 2024-03-29T10:09:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.324+0000 7fc306c00700 1 -- 172.21.15.12:0/142130665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc300067720 con 0x7fc3000cec40 2024-03-29T10:09:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.325+0000 7fc2f7e00700 1 -- 172.21.15.12:0/142130665 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fc2f0003b50 con 0x7fc3000cec40 2024-03-29T10:09:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.326+0000 7fc2f7e00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e006ade0 0x7fc2e006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.326+0000 7fc2f7e00700 1 -- 172.21.15.12:0/142130665 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7fc2f0088a60 con 0x7fc3000cec40 2024-03-29T10:09:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.326+0000 7fc2ff400700 1 -- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e006ade0 msgr2=0x7fc2e006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.326+0000 7fc2ff400700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e006ade0 0x7fc2e006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:08.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.330+0000 7fc2f7e00700 1 -- 172.21.15.12:0/142130665 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2f00538d0 con 0x7fc3000cec40 2024-03-29T10:09:08.377 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:08.377+0000 7fcbd46e4240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:09:08.408 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:08.408+0000 7f018ebb1240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:09:08.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.527+0000 7fc2ff400700 1 -- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e006ade0 msgr2=0x7fc2e006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:08.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.527+0000 7fc2ff400700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e006ade0 0x7fc2e006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:08.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.544+0000 7fc306c00700 1 -- 172.21.15.12:0/142130665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc3000b5d00 con 0x7fc3000cec40 2024-03-29T10:09:08.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.544+0000 7fc2f7e00700 1 -- 172.21.15.12:0/142130665 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc2f0017070 con 0x7fc3000cec40 2024-03-29T10:09:08.544 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:09:08.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 -- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e006ade0 msgr2=0x7fc2e006d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:08.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e006ade0 0x7fc2e006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 -- 172.21.15.12:0/142130665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000cec40 msgr2=0x7fc3001e8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000cec40 0x7fc3001e8f50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0001eb0 tx=0x7fc2f000c800 comp rx=0 tx=0).stop 2024-03-29T10:09:08.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 -- 172.21.15.12:0/142130665 shutdown_connections 2024-03-29T10:09:08.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e006ade0 0x7fc2e006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3001611b0 0x7fc3000b7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3000cec40 0x7fc3001e8f50 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 --2- 172.21.15.12:0/142130665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3000d21f0 0x7fc3001e6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 -- 172.21.15.12:0/142130665 >> 172.21.15.12:0/142130665 conn(0x7fc30008aff0 msgr2=0x7fc300161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:08.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 -- 172.21.15.12:0/142130665 shutdown_connections 2024-03-29T10:09:08.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.548+0000 7fc306c00700 1 -- 172.21.15.12:0/142130665 wait complete. 2024-03-29T10:09:08.566 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:09:08.582 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:08.583+0000 7fcbd46e4240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:09:08.683 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:08.684+0000 7fcbd46e4240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:09:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.773+0000 7fb82c400700 1 Processor -- start 2024-03-29T10:09:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.773+0000 7fb82c400700 1 -- start start 2024-03-29T10:09:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.774+0000 7fb82c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb82415d9d0 0x7fb8240d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb82c400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb8240d2540 0x7fb8240ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb82c400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb824161420 con 0x7fb8240d16a0 2024-03-29T10:09:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb82c400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82415edc0 con 0x7fb8240d2540 2024-03-29T10:09:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb82c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb824161700 con 0x7fb82415d9d0 2024-03-29T10:09:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb82a400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb8240d2540 0x7fb8240ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb829a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb82415d9d0 0x7fb8240d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb82a400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb8240d2540 0x7fb8240ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53502/0 (socket says 172.21.15.12:53502) 2024-03-29T10:09:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb82a400700 1 -- 172.21.15.12:0/3540980693 learned_addr learned my addr 172.21.15.12:0/3540980693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb829000700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb8240d16a0 0x7fb8240d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:39474/0 (socket says 172.21.15.12:39474) 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb829a00700 1 -- 172.21.15.12:0/3540980693 >> v2:172.21.15.12:3300/0 conn(0x7fb8240d2540 msgr2=0x7fb8240ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb829a00700 1 --2- 172.21.15.12:0/3540980693 >> v2:172.21.15.12:3300/0 conn(0x7fb8240d2540 0x7fb8240ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb829a00700 1 -- 172.21.15.12:0/3540980693 >> v1:172.21.15.159:6789/0 conn(0x7fb8240d16a0 legacy=0x7fb8240d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.776+0000 7fb829a00700 1 -- 172.21.15.12:0/3540980693 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8241619e0 con 0x7fb82415d9d0 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.776+0000 7fb82a400700 1 --2- 172.21.15.12:0/3540980693 >> v2:172.21.15.12:3300/0 conn(0x7fb8240d2540 0x7fb8240ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.775+0000 7fb823e00700 1 -- 172.21.15.12:0/3540980693 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298856748 0 0) 0x7fb824161420 con 0x7fb8240d16a0 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.776+0000 7fb829a00700 1 --2- 172.21.15.12:0/3540980693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb82415d9d0 0x7fb8240d3c30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb818009870 tx=0x7fb818009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=155a5345e4b93bca server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.776+0000 7fb823e00700 1 -- 172.21.15.12:0/3540980693 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818011830 con 0x7fb82415d9d0 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.776+0000 7fb823e00700 1 -- 172.21.15.12:0/3540980693 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb818011990 con 0x7fb82415d9d0 2024-03-29T10:09:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.776+0000 7fb823e00700 1 -- 172.21.15.12:0/3540980693 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818011d20 con 0x7fb82415d9d0 2024-03-29T10:09:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.777+0000 7fb82c400700 1 -- 172.21.15.12:0/3540980693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb82415d9d0 msgr2=0x7fb8240d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.777+0000 7fb82c400700 1 --2- 172.21.15.12:0/3540980693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb82415d9d0 0x7fb8240d3c30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb818009870 tx=0x7fb818009b80 comp rx=0 tx=0).stop 2024-03-29T10:09:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.777+0000 7fb82c400700 1 -- 172.21.15.12:0/3540980693 shutdown_connections 2024-03-29T10:09:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.777+0000 7fb82c400700 1 --2- 172.21.15.12:0/3540980693 >> v2:172.21.15.12:3300/0 conn(0x7fb8240d2540 0x7fb8240ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.777+0000 7fb82c400700 1 --2- 172.21.15.12:0/3540980693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb82415d9d0 0x7fb8240d3c30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.777+0000 7fb82c400700 1 -- 172.21.15.12:0/3540980693 >> 172.21.15.12:0/3540980693 conn(0x7fb82408aff0 msgr2=0x7fb8240d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.777+0000 7fb82c400700 1 -- 172.21.15.12:0/3540980693 shutdown_connections 2024-03-29T10:09:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.777+0000 7fb82c400700 1 -- 172.21.15.12:0/3540980693 wait complete. 2024-03-29T10:09:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.779+0000 7fb82c400700 1 Processor -- start 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.779+0000 7fb82c400700 1 -- start start 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb82c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8240d16a0 0x7fb8240a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb82c400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8240d2540 0x7fb8240b7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb82c400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb82415d9d0 0x7fb8240b75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb82c400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241f45f0 con 0x7fb8240d2540 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb82c400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82409d3f0 con 0x7fb82415d9d0 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb82c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8240b8480 con 0x7fb8240d16a0 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb829000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8240d2540 0x7fb8240b7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb829000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8240d2540 0x7fb8240b7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53516/0 (socket says 172.21.15.12:53516) 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb829000700 1 -- 172.21.15.12:0/282221848 learned_addr learned my addr 172.21.15.12:0/282221848 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb829a00700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8240d16a0 0x7fb8240a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:08.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.781+0000 7fb82a400700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb82415d9d0 0x7fb8240b75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:08.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb829000700 1 -- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8240d16a0 msgr2=0x7fb8240a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb829000700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8240d16a0 0x7fb8240a0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb829000700 1 -- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb82415d9d0 msgr2=0x7fb8240b75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:08.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb829000700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb82415d9d0 0x7fb8240b75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:08.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb829000700 1 -- 172.21.15.12:0/282221848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c009b30 con 0x7fb8240d2540 2024-03-29T10:09:08.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb829000700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8240d2540 0x7fb8240b7070 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb81000c550 tx=0x7fb81000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:08.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb821600700 1 -- 172.21.15.12:0/282221848 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810004cd0 con 0x7fb8240d2540 2024-03-29T10:09:08.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb821600700 1 -- 172.21.15.12:0/282221848 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb810004e50 con 0x7fb8240d2540 2024-03-29T10:09:08.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb82c400700 1 -- 172.21.15.12:0/282221848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8180094a0 con 0x7fb8240d2540 2024-03-29T10:09:08.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb821600700 1 -- 172.21.15.12:0/282221848 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810007310 con 0x7fb8240d2540 2024-03-29T10:09:08.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.782+0000 7fb82c400700 1 -- 172.21.15.12:0/282221848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8240b7ec0 con 0x7fb8240d2540 2024-03-29T10:09:08.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.784+0000 7fb82c400700 1 -- 172.21.15.12:0/282221848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8240d1ca0 con 0x7fb8240d2540 2024-03-29T10:09:08.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.784+0000 7fb821600700 1 -- 172.21.15.12:0/282221848 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fb810007470 con 0x7fb8240d2540 2024-03-29T10:09:08.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.785+0000 7fb821600700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb80406adb0 0x7fb80406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:08.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.785+0000 7fb821600700 1 -- 172.21.15.12:0/282221848 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7fb810089250 con 0x7fb8240d2540 2024-03-29T10:09:08.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.785+0000 7fb829a00700 1 -- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb80406adb0 msgr2=0x7fb80406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:08.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.785+0000 7fb829a00700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb80406adb0 0x7fb80406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:08.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.790+0000 7fb821600700 1 -- 172.21.15.12:0/282221848 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8100541d0 con 0x7fb8240d2540 2024-03-29T10:09:08.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.986+0000 7fb829a00700 1 -- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb80406adb0 msgr2=0x7fb80406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:08.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:08.986+0000 7fb829a00700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb80406adb0 0x7fb80406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.005+0000 7fb82c400700 1 -- 172.21.15.12:0/282221848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb8240d14c0 con 0x7fb8240d2540 2024-03-29T10:09:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.005+0000 7fb821600700 1 -- 172.21.15.12:0/282221848 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v66) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb8100577f0 con 0x7fb8240d2540 2024-03-29T10:09:09.005 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:09:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 -- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb80406adb0 msgr2=0x7fb80406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb80406adb0 0x7fb80406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 -- 172.21.15.12:0/282221848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8240d2540 msgr2=0x7fb8240b7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8240d2540 0x7fb8240b7070 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb81000c550 tx=0x7fb81000c860 comp rx=0 tx=0).stop 2024-03-29T10:09:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 -- 172.21.15.12:0/282221848 shutdown_connections 2024-03-29T10:09:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb80406adb0 0x7fb80406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb82415d9d0 0x7fb8240b75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8240d16a0 0x7fb8240a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 --2- 172.21.15.12:0/282221848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8240d2540 0x7fb8240b7070 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 -- 172.21.15.12:0/282221848 >> 172.21.15.12:0/282221848 conn(0x7fb82408aff0 msgr2=0x7fb8240b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 -- 172.21.15.12:0/282221848 shutdown_connections 2024-03-29T10:09:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.008+0000 7fb82c400700 1 -- 172.21.15.12:0/282221848 wait complete. 2024-03-29T10:09:09.027 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:09:09.151 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:09.151+0000 7f018ebb1240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:09:09.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.274+0000 7f60abe00700 1 Processor -- start 2024-03-29T10:09:09.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.275+0000 7f60abe00700 1 -- start start 2024-03-29T10:09:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.276+0000 7f60abe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f60a415d9d0 0x7f60a40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.276+0000 7f60abe00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a4161420 con 0x7f60a40d2540 2024-03-29T10:09:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.276+0000 7f60abe00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a4161700 con 0x7f60a40d16a0 2024-03-29T10:09:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.276+0000 7f60abe00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a415edc0 con 0x7f60a415d9d0 2024-03-29T10:09:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.276+0000 7f60a9400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f60a415d9d0 0x7f60a40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.276+0000 7f60a8a00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f60a40d16a0 0x7f60a40d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:58900/0 (socket says 172.21.15.12:58900) 2024-03-29T10:09:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.276+0000 7f60a9e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f60a40d2540 0x7f60a40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40192/0 (socket says 172.21.15.12:40192) 2024-03-29T10:09:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.276+0000 7f60a8a00700 1 -- 172.21.15.12:0/602846775 learned_addr learned my addr 172.21.15.12:0/602846775 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.277+0000 7f60a3e00700 1 -- 172.21.15.12:0/602846775 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867834765 0 0) 0x7f60a4161700 con 0x7f60a40d16a0 2024-03-29T10:09:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.277+0000 7f60a3e00700 1 -- 172.21.15.12:0/602846775 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f608c0035b0 con 0x7f60a40d16a0 2024-03-29T10:09:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.277+0000 7f60a3e00700 1 -- 172.21.15.12:0/602846775 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659673743 0 0) 0x7f60a4161420 con 0x7f60a40d2540 2024-03-29T10:09:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.277+0000 7f60a9400700 1 -- 172.21.15.12:0/602846775 >> v1:172.21.15.159:6789/0 conn(0x7f60a40d16a0 legacy=0x7f60a40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.277+0000 7f60a9400700 1 -- 172.21.15.12:0/602846775 >> v1:172.21.15.12:6789/0 conn(0x7f60a40d2540 legacy=0x7f60a40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.277+0000 7f60a9400700 1 -- 172.21.15.12:0/602846775 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a41619e0 con 0x7f60a415d9d0 2024-03-29T10:09:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.277+0000 7f60a9400700 1 --2- 172.21.15.12:0/602846775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60a415d9d0 0x7f60a40d3c30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6098009840 tx=0x7f6098009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1d6150b54a9695f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.277+0000 7f60a3e00700 1 -- 172.21.15.12:0/602846775 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6098011830 con 0x7f60a415d9d0 2024-03-29T10:09:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.277+0000 7f60a3e00700 1 -- 172.21.15.12:0/602846775 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6098011990 con 0x7f60a415d9d0 2024-03-29T10:09:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.277+0000 7f60a3e00700 1 -- 172.21.15.12:0/602846775 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6098011d20 con 0x7f60a415d9d0 2024-03-29T10:09:09.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.278+0000 7f60abe00700 1 -- 172.21.15.12:0/602846775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60a415d9d0 msgr2=0x7f60a40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.278+0000 7f60abe00700 1 --2- 172.21.15.12:0/602846775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60a415d9d0 0x7f60a40d3c30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6098009840 tx=0x7f6098009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:09.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.278+0000 7f60abe00700 1 -- 172.21.15.12:0/602846775 shutdown_connections 2024-03-29T10:09:09.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.278+0000 7f60abe00700 1 --2- 172.21.15.12:0/602846775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60a415d9d0 0x7f60a40d3c30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.278+0000 7f60abe00700 1 -- 172.21.15.12:0/602846775 >> 172.21.15.12:0/602846775 conn(0x7f60a408aff0 msgr2=0x7f60a40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:09.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.278+0000 7f60abe00700 1 -- 172.21.15.12:0/602846775 shutdown_connections 2024-03-29T10:09:09.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.278+0000 7f60abe00700 1 -- 172.21.15.12:0/602846775 wait complete. 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.280+0000 7f60abe00700 1 Processor -- start 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.280+0000 7f60abe00700 1 -- start start 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60abe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60a40d16a0 0x7f60a415be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60abe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60a40d2540 0x7f60a40caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60abe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60a415d9d0 0x7f60a41e8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60abe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a415c3a0 con 0x7f60a415d9d0 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60abe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a41e7780 con 0x7f60a40d2540 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60abe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a41e7a60 con 0x7f60a40d16a0 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60a9400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60a40d16a0 0x7f60a415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60a8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60a40d2540 0x7f60a40caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60a9400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60a40d16a0 0x7f60a415be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:53304/0 (socket says 172.21.15.12:53304) 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60a9400700 1 -- 172.21.15.12:0/2987784847 learned_addr learned my addr 172.21.15.12:0/2987784847 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.281+0000 7f60a9e00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60a415d9d0 0x7f60a41e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a9400700 1 -- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60a40d2540 msgr2=0x7f60a40caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a9400700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60a40d2540 0x7f60a40caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a9400700 1 -- 172.21.15.12:0/2987784847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60a415d9d0 msgr2=0x7f60a41e8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a9400700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60a415d9d0 0x7f60a41e8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a9400700 1 -- 172.21.15.12:0/2987784847 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60980094a0 con 0x7f60a40d16a0 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a9e00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60a415d9d0 0x7f60a41e8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a8a00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60a40d2540 0x7f60a40caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a9400700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60a40d16a0 0x7f60a415be60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f60980090b0 tx=0x7f6098004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a1600700 1 -- 172.21.15.12:0/2987784847 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609801a860 con 0x7f60a40d16a0 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a1600700 1 -- 172.21.15.12:0/2987784847 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609801a9c0 con 0x7f60a40d16a0 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60a1600700 1 -- 172.21.15.12:0/2987784847 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609801ad30 con 0x7f60a40d16a0 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.282+0000 7f60abe00700 1 -- 172.21.15.12:0/2987784847 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a41e7d40 con 0x7f60a40d16a0 2024-03-29T10:09:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.283+0000 7f60abe00700 1 -- 172.21.15.12:0/2987784847 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a41e8300 con 0x7f60a40d16a0 2024-03-29T10:09:09.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.284+0000 7f60abe00700 1 -- 172.21.15.12:0/2987784847 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a40d1ca0 con 0x7f60a40d16a0 2024-03-29T10:09:09.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.284+0000 7f60a1600700 1 -- 172.21.15.12:0/2987784847 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f6098022070 con 0x7f60a40d16a0 2024-03-29T10:09:09.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.285+0000 7f60a1600700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f608406adb0 0x7f608406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.285+0000 7f60a1600700 1 -- 172.21.15.12:0/2987784847 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f609808a290 con 0x7f60a40d16a0 2024-03-29T10:09:09.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.285+0000 7f60a8a00700 1 -- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f608406adb0 msgr2=0x7f608406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.285+0000 7f60a8a00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f608406adb0 0x7f608406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.290+0000 7f60a1600700 1 -- 172.21.15.12:0/2987784847 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f60980553e0 con 0x7f60a40d16a0 2024-03-29T10:09:09.365 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:09.365+0000 7f018ebb1240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:09:09.474 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:09.474+0000 7fcbd46e4240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:09:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.486+0000 7f60a8a00700 1 -- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f608406adb0 msgr2=0x7f608406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.486+0000 7f60a8a00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f608406adb0 0x7f608406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.503+0000 7f60abe00700 1 -- 172.21.15.12:0/2987784847 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f60a40d3310 con 0x7f60a40d16a0 2024-03-29T10:09:09.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.504+0000 7f60a1600700 1 -- 172.21.15.12:0/2987784847 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v66) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6098015020 con 0x7f60a40d16a0 2024-03-29T10:09:09.503 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:09:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.506+0000 7f60abe00700 1 -- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f608406adb0 msgr2=0x7f608406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.506+0000 7f60abe00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f608406adb0 0x7f608406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.506+0000 7f60abe00700 1 -- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60a40d16a0 msgr2=0x7f60a415be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.506+0000 7f60abe00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60a40d16a0 0x7f60a415be60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f60980090b0 tx=0x7f6098004750 comp rx=0 tx=0).stop 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.507+0000 7f60abe00700 1 -- 172.21.15.12:0/2987784847 shutdown_connections 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.507+0000 7f60abe00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f608406adb0 0x7f608406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.507+0000 7f60abe00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60a415d9d0 0x7f60a41e8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.507+0000 7f60abe00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60a40d16a0 0x7f60a415be60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.507+0000 7f60abe00700 1 --2- 172.21.15.12:0/2987784847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60a40d2540 0x7f60a40caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.507+0000 7f60abe00700 1 -- 172.21.15.12:0/2987784847 >> 172.21.15.12:0/2987784847 conn(0x7f60a408aff0 msgr2=0x7f60a40b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.507+0000 7f60abe00700 1 -- 172.21.15.12:0/2987784847 shutdown_connections 2024-03-29T10:09:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.507+0000 7f60abe00700 1 -- 172.21.15.12:0/2987784847 wait complete. 2024-03-29T10:09:09.525 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:09:09.569 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:09.569+0000 7f018ebb1240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:09:09.705 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:09.705+0000 7fcbd46e4240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:09:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.730+0000 7f696d000700 1 Processor -- start 2024-03-29T10:09:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.730+0000 7f696d000700 1 -- start start 2024-03-29T10:09:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.732+0000 7f696d000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f69680acda0 0x7f696815e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.732+0000 7f696d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680b4860 0x7f69680b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.732+0000 7f696d000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696815eb80 con 0x7f69680ad180 2024-03-29T10:09:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.732+0000 7f696d000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696806eef0 con 0x7f69680acda0 2024-03-29T10:09:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.732+0000 7f696d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69680ae7b0 con 0x7f69680b4860 2024-03-29T10:09:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.732+0000 7f695fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f69680ad180 0x7f69680b4150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40198/0 (socket says 172.21.15.12:40198) 2024-03-29T10:09:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.732+0000 7f695fe00700 1 -- 172.21.15.12:0/2084891715 learned_addr learned my addr 172.21.15.12:0/2084891715 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.732+0000 7f6965e00700 1 --2- 172.21.15.12:0/2084891715 >> v2:172.21.15.159:3300/0 conn(0x7f69680acda0 0x7f696815e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.732+0000 7f6966800700 1 --2- 172.21.15.12:0/2084891715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680b4860 0x7f69680b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6965400700 1 -- 172.21.15.12:0/2084891715 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046817752 0 0) 0x7f696815eb80 con 0x7f69680ad180 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6965400700 1 -- 172.21.15.12:0/2084891715 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f694c0035b0 con 0x7f69680ad180 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6965e00700 1 -- 172.21.15.12:0/2084891715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680b4860 msgr2=0x7f69680b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6965e00700 1 --2- 172.21.15.12:0/2084891715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680b4860 0x7f69680b4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6965e00700 1 -- 172.21.15.12:0/2084891715 >> v1:172.21.15.12:6789/0 conn(0x7f69680ad180 legacy=0x7f69680b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6965e00700 1 -- 172.21.15.12:0/2084891715 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69680aea90 con 0x7f69680acda0 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6966800700 1 --2- 172.21.15.12:0/2084891715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680b4860 0x7f69680b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6965e00700 1 --2- 172.21.15.12:0/2084891715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69680acda0 0x7f696815e640 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6958009840 tx=0x7f6958009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b99afaf3f550c33 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6965400700 1 -- 172.21.15.12:0/2084891715 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6958011830 con 0x7f69680acda0 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6965400700 1 -- 172.21.15.12:0/2084891715 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6958011990 con 0x7f69680acda0 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.733+0000 7f6965400700 1 -- 172.21.15.12:0/2084891715 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6958011d20 con 0x7f69680acda0 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.734+0000 7f696d000700 1 -- 172.21.15.12:0/2084891715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69680acda0 msgr2=0x7f696815e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.734+0000 7f696d000700 1 --2- 172.21.15.12:0/2084891715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69680acda0 0x7f696815e640 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6958009840 tx=0x7f6958009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:09.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.734+0000 7f696d000700 1 -- 172.21.15.12:0/2084891715 shutdown_connections 2024-03-29T10:09:09.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.734+0000 7f696d000700 1 --2- 172.21.15.12:0/2084891715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680b4860 0x7f69680b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.734+0000 7f696d000700 1 --2- 172.21.15.12:0/2084891715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69680acda0 0x7f696815e640 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.734+0000 7f696d000700 1 -- 172.21.15.12:0/2084891715 >> 172.21.15.12:0/2084891715 conn(0x7f6968072170 msgr2=0x7f696806a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:09.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.734+0000 7f696d000700 1 -- 172.21.15.12:0/2084891715 shutdown_connections 2024-03-29T10:09:09.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.734+0000 7f696d000700 1 -- 172.21.15.12:0/2084891715 wait complete. 2024-03-29T10:09:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.736+0000 7f696d000700 1 Processor -- start 2024-03-29T10:09:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.736+0000 7f696d000700 1 -- start start 2024-03-29T10:09:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f696d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69680acda0 0x7f69680cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f696d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680ad180 0x7f6968143150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f696d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69680b4860 0x7f696806a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f696d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6968157f70 con 0x7f69680b4860 2024-03-29T10:09:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f696d000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69680cc7f0 con 0x7f69680acda0 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f696d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6968158180 con 0x7f69680ad180 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f6966800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69680b4860 0x7f696806a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f6966800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69680b4860 0x7f696806a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58136/0 (socket says 172.21.15.12:58136) 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f6966800700 1 -- 172.21.15.12:0/3838658003 learned_addr learned my addr 172.21.15.12:0/3838658003 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f6965e00700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69680acda0 0x7f69680cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.737+0000 7f695fe00700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680ad180 0x7f6968143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f6966800700 1 -- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680ad180 msgr2=0x7f6968143150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f6966800700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680ad180 0x7f6968143150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f6966800700 1 -- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69680acda0 msgr2=0x7f69680cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f6966800700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69680acda0 0x7f69680cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f6966800700 1 -- 172.21.15.12:0/3838658003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6960009a70 con 0x7f69680b4860 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f6966800700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69680b4860 0x7f696806a790 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f696000efc0 tx=0x7f696000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f695e000700 1 -- 172.21.15.12:0/3838658003 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000ca90 con 0x7f69680b4860 2024-03-29T10:09:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f695e000700 1 -- 172.21.15.12:0/3838658003 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696000f040 con 0x7f69680b4860 2024-03-29T10:09:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f696d000700 1 -- 172.21.15.12:0/3838658003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69580094a0 con 0x7f69680b4860 2024-03-29T10:09:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f695e000700 1 -- 172.21.15.12:0/3838658003 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000ce30 con 0x7f69680b4860 2024-03-29T10:09:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.738+0000 7f696d000700 1 -- 172.21.15.12:0/3838658003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69680ce790 con 0x7f69680b4860 2024-03-29T10:09:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.739+0000 7f696d000700 1 -- 172.21.15.12:0/3838658003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6968066e90 con 0x7f69680b4860 2024-03-29T10:09:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.740+0000 7f695e000700 1 -- 172.21.15.12:0/3838658003 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f6960014480 con 0x7f69680b4860 2024-03-29T10:09:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.741+0000 7f695e000700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f694406adb0 0x7f694406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.741+0000 7f695e000700 1 -- 172.21.15.12:0/3838658003 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f6960089f20 con 0x7f69680b4860 2024-03-29T10:09:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.741+0000 7f6965e00700 1 -- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f694406adb0 msgr2=0x7f694406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.741+0000 7f6965e00700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f694406adb0 0x7f694406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.746+0000 7f695e000700 1 -- 172.21.15.12:0/3838658003 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6960054c70 con 0x7f69680b4860 2024-03-29T10:09:09.926 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:09.926+0000 7fcbd46e4240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:09:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.942+0000 7f6965e00700 1 -- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f694406adb0 msgr2=0x7f694406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:09.942+0000 7f6965e00700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f694406adb0 0x7f694406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:10.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.004+0000 7f696d000700 1 -- 172.21.15.12:0/3838658003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6968067720 con 0x7f69680b4860 2024-03-29T10:09:10.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.005+0000 7f695e000700 1 -- 172.21.15.12:0/3838658003 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f6960018090 con 0x7f69680b4860 2024-03-29T10:09:10.004 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:10.004 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5746,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/981039791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":981039791},{"type":"v1","addr":"172.21.15.159:6836","nonce":981039791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1682878615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1682878615},{"type":"v1","addr":"172.21.15.159:6837","nonce":1682878615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5752,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1663013205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":1663013205},{"type":"v1","addr":"172.21.15.12:6834","nonce":1663013205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:07:08.705772+0000","modified":"2024-03-29T10:07:12.186244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5877},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5877":{"gid":5877,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3970376731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3970376731},{"type":"v1","addr":"172.21.15.12:6835","nonce":3970376731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-29T10:09:10.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.007+0000 7f696d000700 1 -- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f694406adb0 msgr2=0x7f694406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:10.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.007+0000 7f696d000700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f694406adb0 0x7f694406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.007+0000 7f696d000700 1 -- 172.21.15.12:0/3838658003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69680b4860 msgr2=0x7f696806a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.007+0000 7f696d000700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69680b4860 0x7f696806a790 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f696000efc0 tx=0x7f696000c3a0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.007+0000 7f696d000700 1 -- 172.21.15.12:0/3838658003 shutdown_connections 2024-03-29T10:09:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.007+0000 7f696d000700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f694406adb0 0x7f694406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.007+0000 7f696d000700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69680b4860 0x7f696806a790 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.007+0000 7f696d000700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69680acda0 0x7f69680cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.007+0000 7f696d000700 1 --2- 172.21.15.12:0/3838658003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680ad180 0x7f6968143150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.007+0000 7f696d000700 1 -- 172.21.15.12:0/3838658003 >> 172.21.15.12:0/3838658003 conn(0x7f6968072170 msgr2=0x7f696815cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.008+0000 7f696d000700 1 -- 172.21.15.12:0/3838658003 shutdown_connections 2024-03-29T10:09:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.008+0000 7f696d000700 1 -- 172.21.15.12:0/3838658003 wait complete. 2024-03-29T10:09:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-29T10:09:10.026 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:09:10.047 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.047+0000 7f018ebb1240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:09:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.225+0000 7f9dff000700 1 Processor -- start 2024-03-29T10:09:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.225+0000 7f9dff000700 1 -- start start 2024-03-29T10:09:10.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.226+0000 7f9dff000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9df80b3af0 0x7f9df815a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.226+0000 7f9dff000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9df8072580 0x7f9df80b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.226+0000 7f9dff000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df80b2180 con 0x7f9df80b2e50 2024-03-29T10:09:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.226+0000 7f9dff000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df815b100 con 0x7f9df80b3af0 2024-03-29T10:09:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.226+0000 7f9dff000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df815b3e0 con 0x7f9df8072580 2024-03-29T10:09:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9dfd000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9df80b2e50 0x7f9df80709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40204/0 (socket says 172.21.15.12:40204) 2024-03-29T10:09:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9dfd000700 1 -- 172.21.15.12:0/3383935076 learned_addr learned my addr 172.21.15.12:0/3383935076 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9df7e00700 1 --2- 172.21.15.12:0/3383935076 >> v2:172.21.15.159:3300/0 conn(0x7f9df80b3af0 0x7f9df815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9df7400700 1 --2- 172.21.15.12:0/3383935076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9df8072580 0x7f9df80b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9df6a00700 1 -- 172.21.15.12:0/3383935076 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689224154 0 0) 0x7f9df80b2180 con 0x7f9df80b2e50 2024-03-29T10:09:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9df6a00700 1 -- 172.21.15.12:0/3383935076 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ddc0035b0 con 0x7f9df80b2e50 2024-03-29T10:09:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9df7e00700 1 -- 172.21.15.12:0/3383935076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9df8072580 msgr2=0x7f9df80b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9df7e00700 1 --2- 172.21.15.12:0/3383935076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9df8072580 0x7f9df80b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9df7e00700 1 -- 172.21.15.12:0/3383935076 >> v1:172.21.15.12:6789/0 conn(0x7f9df80b2e50 legacy=0x7f9df80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9df7e00700 1 -- 172.21.15.12:0/3383935076 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df80b23f0 con 0x7f9df80b3af0 2024-03-29T10:09:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.227+0000 7f9df7400700 1 --2- 172.21.15.12:0/3383935076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9df8072580 0x7f9df80b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.228+0000 7f9df7e00700 1 --2- 172.21.15.12:0/3383935076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9df80b3af0 0x7f9df815a860 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009840 tx=0x7f9de8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13ab8a31f74dde48 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.228+0000 7f9df6a00700 1 -- 172.21.15.12:0/3383935076 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8011830 con 0x7f9df80b3af0 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.228+0000 7f9df6a00700 1 -- 172.21.15.12:0/3383935076 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de8011990 con 0x7f9df80b3af0 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.228+0000 7f9df6a00700 1 -- 172.21.15.12:0/3383935076 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8011d20 con 0x7f9df80b3af0 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.228+0000 7f9dff000700 1 -- 172.21.15.12:0/3383935076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9df80b3af0 msgr2=0x7f9df815a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.228+0000 7f9dff000700 1 --2- 172.21.15.12:0/3383935076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9df80b3af0 0x7f9df815a860 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009840 tx=0x7f9de8009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.228+0000 7f9dff000700 1 -- 172.21.15.12:0/3383935076 shutdown_connections 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.228+0000 7f9dff000700 1 --2- 172.21.15.12:0/3383935076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9df80b3af0 0x7f9df815a860 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.228+0000 7f9dff000700 1 --2- 172.21.15.12:0/3383935076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9df8072580 0x7f9df80b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.229+0000 7f9dff000700 1 -- 172.21.15.12:0/3383935076 >> 172.21.15.12:0/3383935076 conn(0x7f9df80cf190 msgr2=0x7f9df80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.229+0000 7f9dff000700 1 -- 172.21.15.12:0/3383935076 shutdown_connections 2024-03-29T10:09:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.229+0000 7f9dff000700 1 -- 172.21.15.12:0/3383935076 wait complete. 2024-03-29T10:09:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.230+0000 7f9dff000700 1 Processor -- start 2024-03-29T10:09:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.231+0000 7f9dff000700 1 -- start start 2024-03-29T10:09:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.231+0000 7f9dff000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df8072580 0x7f9df814dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.231+0000 7f9dff000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9df80b2e50 0x7f9df8154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9dff000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9df80b3af0 0x7f9df80ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9dff000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8152a50 con 0x7f9df8072580 2024-03-29T10:09:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9dff000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df814db60 con 0x7f9df80b3af0 2024-03-29T10:09:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9dff000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8151e40 con 0x7f9df80b2e50 2024-03-29T10:09:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9df7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df8072580 0x7f9df814dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9df7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df8072580 0x7f9df814dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58142/0 (socket says 172.21.15.12:58142) 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9df7e00700 1 -- 172.21.15.12:0/1738559262 learned_addr learned my addr 172.21.15.12:0/1738559262 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9df7e00700 1 -- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9df80b2e50 msgr2=0x7f9df8154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9df7e00700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9df80b2e50 0x7f9df8154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9df7e00700 1 -- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9df80b3af0 msgr2=0x7f9df80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9df7e00700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9df80b3af0 0x7f9df80ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9df7e00700 1 -- 172.21.15.12:0/1738559262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de40094a0 con 0x7f9df8072580 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.232+0000 7f9df7e00700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df8072580 0x7f9df814dd90 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9de80090b0 tx=0x7f9de8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.233+0000 7f9defe00700 1 -- 172.21.15.12:0/1738559262 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80036a0 con 0x7f9df8072580 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.233+0000 7f9defe00700 1 -- 172.21.15.12:0/1738559262 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de801f070 con 0x7f9df8072580 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.233+0000 7f9defe00700 1 -- 172.21.15.12:0/1738559262 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8003a40 con 0x7f9df8072580 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.233+0000 7f9dff000700 1 -- 172.21.15.12:0/1738559262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de80094a0 con 0x7f9df8072580 2024-03-29T10:09:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.233+0000 7f9dff000700 1 -- 172.21.15.12:0/1738559262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df8152400 con 0x7f9df8072580 2024-03-29T10:09:10.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.234+0000 7f9dff000700 1 -- 172.21.15.12:0/1738559262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df8067720 con 0x7f9df8072580 2024-03-29T10:09:10.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.235+0000 7f9defe00700 1 -- 172.21.15.12:0/1738559262 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f9de8003ba0 con 0x7f9df8072580 2024-03-29T10:09:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.236+0000 7f9defe00700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dd806ade0 0x7f9dd806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.236+0000 7f9defe00700 1 -- 172.21.15.12:0/1738559262 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f9de8021070 con 0x7f9df8072580 2024-03-29T10:09:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.236+0000 7f9df7400700 1 -- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dd806ade0 msgr2=0x7f9dd806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.236+0000 7f9df7400700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dd806ade0 0x7f9dd806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:10.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.240+0000 7f9defe00700 1 -- 172.21.15.12:0/1738559262 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9de8055e00 con 0x7f9df8072580 2024-03-29T10:09:10.254 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.254+0000 7f018ebb1240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:09:10.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.437+0000 7f9df7400700 1 -- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dd806ade0 msgr2=0x7f9dd806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:10.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.437+0000 7f9df7400700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dd806ade0 0x7f9dd806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:10.440 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:10.440+0000 7fcbd46e4240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:09:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.460+0000 7f9dff000700 1 -- 172.21.15.12:0/1738559262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9df80cef70 con 0x7f9df8072580 2024-03-29T10:09:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.462+0000 7f9defe00700 1 -- 172.21.15.12:0/1738559262 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16769 (secure 0 0 0) 0x7f9de802b090 con 0x7f9df8072580 2024-03-29T10:09:10.461 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:10.462 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":66,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:05.263226+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:07:07.106799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-29T10:07:07.671044+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":5}]},{"pgid":"7.12","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6833/43444":"2024-03-30T10:07:25.860642+0000","172.21.15.159:6832/43444":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3220343982":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/2775778725":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3363559414":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/1116927207":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3950908519":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/798085437":"2024-03-30T10:07:25.860642+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:09:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 -- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dd806ade0 msgr2=0x7f9dd806d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dd806ade0 0x7f9dd806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 -- 172.21.15.12:0/1738559262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df8072580 msgr2=0x7f9df814dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df8072580 0x7f9df814dd90 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9de80090b0 tx=0x7f9de8004750 comp rx=0 tx=0).stop 2024-03-29T10:09:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 -- 172.21.15.12:0/1738559262 shutdown_connections 2024-03-29T10:09:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dd806ade0 0x7f9dd806d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9df80b3af0 0x7f9df80ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df8072580 0x7f9df814dd90 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 --2- 172.21.15.12:0/1738559262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9df80b2e50 0x7f9df8154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 -- 172.21.15.12:0/1738559262 >> 172.21.15.12:0/1738559262 conn(0x7f9df80cf190 msgr2=0x7f9df80709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 -- 172.21.15.12:0/1738559262 shutdown_connections 2024-03-29T10:09:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.464+0000 7f9dff000700 1 -- 172.21.15.12:0/1738559262 wait complete. 2024-03-29T10:09:10.482 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T10:09:10.482 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:09:10.665 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:10.666+0000 7fcbd46e4240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:09:10.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.683+0000 7f7716c00700 1 Processor -- start 2024-03-29T10:09:10.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.683+0000 7f7716c00700 1 -- start start 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.684+0000 7f7716c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77100ac580 0x7f77100b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.684+0000 7f7716c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77100b3000 0x7f771006dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.684+0000 7f7716c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77100b3890 con 0x7f771015d670 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.684+0000 7f7716c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7710159f50 con 0x7f77100b3000 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.684+0000 7f7716c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77100b35b0 con 0x7f77100ac580 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.685+0000 7f7714c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77100b3000 0x7f771006dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.685+0000 7f770fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f771015d670 0x7f771015da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40214/0 (socket says 172.21.15.12:40214) 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.685+0000 7f770fe00700 1 -- 172.21.15.12:0/4239593322 learned_addr learned my addr 172.21.15.12:0/4239593322 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.685+0000 7f770f400700 1 --2- 172.21.15.12:0/4239593322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77100ac580 0x7f77100b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.685+0000 7f7714c00700 1 -- 172.21.15.12:0/4239593322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77100ac580 msgr2=0x7f77100b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.685+0000 7f7714c00700 1 --2- 172.21.15.12:0/4239593322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77100ac580 0x7f77100b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.685+0000 7f7714c00700 1 -- 172.21.15.12:0/4239593322 >> v1:172.21.15.12:6789/0 conn(0x7f771015d670 legacy=0x7f771015da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.685+0000 7f7714c00700 1 -- 172.21.15.12:0/4239593322 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771006d400 con 0x7f77100b3000 2024-03-29T10:09:10.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.685+0000 7f7714c00700 1 --2- 172.21.15.12:0/4239593322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100b3000 0x7f771006dee0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7708009d90 tx=0x7f770800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf1aec3804b4fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f770ea00700 1 -- 172.21.15.12:0/4239593322 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770800e070 con 0x7f77100b3000 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f770ea00700 1 -- 172.21.15.12:0/4239593322 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770800bb20 con 0x7f77100b3000 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f770ea00700 1 -- 172.21.15.12:0/4239593322 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708012430 con 0x7f77100b3000 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f7716c00700 1 -- 172.21.15.12:0/4239593322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100b3000 msgr2=0x7f771006dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f7716c00700 1 --2- 172.21.15.12:0/4239593322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100b3000 0x7f771006dee0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7708009d90 tx=0x7f770800b330 comp rx=0 tx=0).stop 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f7716c00700 1 -- 172.21.15.12:0/4239593322 shutdown_connections 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f7716c00700 1 --2- 172.21.15.12:0/4239593322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100b3000 0x7f771006dee0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f7716c00700 1 --2- 172.21.15.12:0/4239593322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77100ac580 0x7f77100b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f7716c00700 1 -- 172.21.15.12:0/4239593322 >> 172.21.15.12:0/4239593322 conn(0x7f77100cf190 msgr2=0x7f77100cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f7716c00700 1 -- 172.21.15.12:0/4239593322 shutdown_connections 2024-03-29T10:09:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.686+0000 7f7716c00700 1 -- 172.21.15.12:0/4239593322 wait complete. 2024-03-29T10:09:10.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.688+0000 7f7716c00700 1 Processor -- start 2024-03-29T10:09:10.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.688+0000 7f7716c00700 1 -- start start 2024-03-29T10:09:10.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.689+0000 7f7716c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100ac580 0x7f771014c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.689+0000 7f7716c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77100b3000 0x7f771014cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.689+0000 7f7716c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f771015d670 0x7f7710153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.689+0000 7f7716c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7710153160 con 0x7f77100ac580 2024-03-29T10:09:10.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.689+0000 7f7716c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7710151d10 con 0x7f771015d670 2024-03-29T10:09:10.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.689+0000 7f7716c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77101539c0 con 0x7f77100b3000 2024-03-29T10:09:10.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.689+0000 7f770fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100ac580 0x7f771014c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:10.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.689+0000 7f770fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100ac580 0x7f771014c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58152/0 (socket says 172.21.15.12:58152) 2024-03-29T10:09:10.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.689+0000 7f770fe00700 1 -- 172.21.15.12:0/112918088 learned_addr learned my addr 172.21.15.12:0/112918088 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.689+0000 7f770f400700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77100b3000 0x7f771014cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f7714c00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f771015d670 0x7f7710153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f770fe00700 1 -- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77100b3000 msgr2=0x7f771014cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f770fe00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77100b3000 0x7f771014cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f770fe00700 1 -- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f771015d670 msgr2=0x7f7710153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f770fe00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f771015d670 0x7f7710153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f770fe00700 1 -- 172.21.15.12:0/112918088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7708009a70 con 0x7f77100ac580 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f770fe00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100ac580 0x7f771014c5a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f770000c4b0 tx=0x7f770000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f7707e00700 1 -- 172.21.15.12:0/112918088 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700005d70 con 0x7f77100ac580 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f7707e00700 1 -- 172.21.15.12:0/112918088 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77000070d0 con 0x7f77100ac580 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f7716c00700 1 -- 172.21.15.12:0/112918088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77100985f0 con 0x7f77100ac580 2024-03-29T10:09:10.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.690+0000 7f7707e00700 1 -- 172.21.15.12:0/112918088 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700007440 con 0x7f77100ac580 2024-03-29T10:09:10.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.691+0000 7f7716c00700 1 -- 172.21.15.12:0/112918088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7710152010 con 0x7f77100ac580 2024-03-29T10:09:10.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.692+0000 7f7716c00700 1 -- 172.21.15.12:0/112918088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7710153ca0 con 0x7f77100ac580 2024-03-29T10:09:10.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.694+0000 7f7707e00700 1 -- 172.21.15.12:0/112918088 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f77000075a0 con 0x7f77100ac580 2024-03-29T10:09:10.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.695+0000 7f7707e00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76f006adb0 0x7f76f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:10.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.695+0000 7f7707e00700 1 -- 172.21.15.12:0/112918088 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f7700088860 con 0x7f77100ac580 2024-03-29T10:09:10.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.695+0000 7f770f400700 1 -- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76f006adb0 msgr2=0x7f76f006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:10.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.695+0000 7f770f400700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76f006adb0 0x7f76f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:10.697 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.697+0000 7f018ebb1240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:09:10.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.700+0000 7f7707e00700 1 -- 172.21.15.12:0/112918088 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7700053890 con 0x7f77100ac580 2024-03-29T10:09:10.808 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.808+0000 7f018ebb1240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:09:10.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.820+0000 7f7707e00700 1 -- 172.21.15.12:0/112918088 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 27) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f7700053c60 con 0x7f77100ac580 2024-03-29T10:09:10.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.820+0000 7f7707e00700 1 -- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76f006adb0 msgr2=0x7f76f006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:10.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.820+0000 7f7707e00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76f006adb0 0x7f76f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.885 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.885+0000 7f0101e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:10.885 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.885+0000 7f0101e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:10.885 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.885+0000 7f0101e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:10.885 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.885+0000 7f0101e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:10.885 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.885+0000 7f0101e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:10.886 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.886+0000 7f00fe200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:10.886 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.886+0000 7f00fe200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:10.886 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.886+0000 7f00fe200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:10.886 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.886+0000 7f00fe200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:10.886 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:10.886+0000 7f00fe200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:10.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.972+0000 7f7716c00700 1 -- 172.21.15.12:0/112918088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f771006d980 con 0x7f77100ac580 2024-03-29T10:09:10.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.973+0000 7f7707e00700 1 -- 172.21.15.12:0/112918088 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f7700072fc0 con 0x7f77100ac580 2024-03-29T10:09:10.973 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:10.973 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5746,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/981039791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":981039791},{"type":"v1","addr":"172.21.15.159:6836","nonce":981039791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1682878615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1682878615},{"type":"v1","addr":"172.21.15.159:6837","nonce":1682878615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5752,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1663013205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":1663013205},{"type":"v1","addr":"172.21.15.12:6834","nonce":1663013205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:07:08.705772+0000","modified":"2024-03-29T10:07:12.186244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5877},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5877":{"gid":5877,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3970376731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3970376731},{"type":"v1","addr":"172.21.15.12:6835","nonce":3970376731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-29T10:09:10.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.975+0000 7f7716c00700 1 -- 172.21.15.12:0/112918088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100ac580 msgr2=0x7f771014c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:10.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.975+0000 7f7716c00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100ac580 0x7f771014c5a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f770000c4b0 tx=0x7f770000c7c0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.975+0000 7f7716c00700 1 -- 172.21.15.12:0/112918088 shutdown_connections 2024-03-29T10:09:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.975+0000 7f7716c00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76f006adb0 0x7f76f006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.975+0000 7f7716c00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f771015d670 0x7f7710153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.976+0000 7f7716c00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77100ac580 0x7f771014c5a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.976+0000 7f7716c00700 1 --2- 172.21.15.12:0/112918088 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77100b3000 0x7f771014cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.976+0000 7f7716c00700 1 -- 172.21.15.12:0/112918088 >> 172.21.15.12:0/112918088 conn(0x7f77100cf190 msgr2=0x7f771015c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.976+0000 7f7716c00700 1 -- 172.21.15.12:0/112918088 shutdown_connections 2024-03-29T10:09:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:10.976+0000 7f7716c00700 1 -- 172.21.15.12:0/112918088 wait complete. 2024-03-29T10:09:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-29T10:09:10.993 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:09:11.135 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:11.136+0000 7fcbd46e4240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:09:11.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.195+0000 7f31e8a00700 1 Processor -- start 2024-03-29T10:09:11.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.195+0000 7f31e8a00700 1 -- start start 2024-03-29T10:09:11.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e8a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f31e40cb3f0 0x7f31e40ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:11.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e8a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e415fd50 con 0x7f31e415b5d0 2024-03-29T10:09:11.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e8a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e40aca10 con 0x7f31e40ac600 2024-03-29T10:09:11.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e8a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31e415b9b0 con 0x7f31e40cb3f0 2024-03-29T10:09:11.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e1e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f31e415b5d0 0x7f31e40ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40230/0 (socket says 172.21.15.12:40230) 2024-03-29T10:09:11.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e1e00700 1 -- 172.21.15.12:0/585456618 learned_addr learned my addr 172.21.15.12:0/585456618 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:11.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e1400700 1 --2- 172.21.15.12:0/585456618 >> v2:172.21.15.159:3300/0 conn(0x7f31e40cb3f0 0x7f31e40ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:11.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e0a00700 1 -- 172.21.15.12:0/585456618 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831952771 0 0) 0x7f31e415fd50 con 0x7f31e415b5d0 2024-03-29T10:09:11.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e0a00700 1 -- 172.21.15.12:0/585456618 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c80035b0 con 0x7f31e415b5d0 2024-03-29T10:09:11.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e1400700 1 -- 172.21.15.12:0/585456618 >> v1:172.21.15.159:6789/0 conn(0x7f31e40ac600 legacy=0x7f31e406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.197+0000 7f31e1400700 1 -- 172.21.15.12:0/585456618 >> v1:172.21.15.12:6789/0 conn(0x7f31e415b5d0 legacy=0x7f31e40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.198+0000 7f31e1400700 1 -- 172.21.15.12:0/585456618 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e415f140 con 0x7f31e40cb3f0 2024-03-29T10:09:11.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.198+0000 7f31e1400700 1 --2- 172.21.15.12:0/585456618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31e40cb3f0 0x7f31e40ac0c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009840 tx=0x7f31d0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8db4dce40352e2ce server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:11.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.198+0000 7f31e0a00700 1 -- 172.21.15.12:0/585456618 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0011830 con 0x7f31e40cb3f0 2024-03-29T10:09:11.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.198+0000 7f31e0a00700 1 -- 172.21.15.12:0/585456618 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d0011990 con 0x7f31e40cb3f0 2024-03-29T10:09:11.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.198+0000 7f31e0a00700 1 -- 172.21.15.12:0/585456618 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0011d00 con 0x7f31e40cb3f0 2024-03-29T10:09:11.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.199+0000 7f31e8a00700 1 -- 172.21.15.12:0/585456618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31e40cb3f0 msgr2=0x7f31e40ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.199+0000 7f31e8a00700 1 --2- 172.21.15.12:0/585456618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31e40cb3f0 0x7f31e40ac0c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009840 tx=0x7f31d0009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:11.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.199+0000 7f31e8a00700 1 -- 172.21.15.12:0/585456618 shutdown_connections 2024-03-29T10:09:11.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.199+0000 7f31e8a00700 1 --2- 172.21.15.12:0/585456618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31e40cb3f0 0x7f31e40ac0c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:11.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.199+0000 7f31e8a00700 1 -- 172.21.15.12:0/585456618 >> 172.21.15.12:0/585456618 conn(0x7f31e40cf190 msgr2=0x7f31e40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:11.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.199+0000 7f31e8a00700 1 -- 172.21.15.12:0/585456618 shutdown_connections 2024-03-29T10:09:11.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.199+0000 7f31e8a00700 1 -- 172.21.15.12:0/585456618 wait complete. 2024-03-29T10:09:11.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.201+0000 7f31e8a00700 1 Processor -- start 2024-03-29T10:09:11.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.202+0000 7f31e8a00700 1 -- start start 2024-03-29T10:09:11.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31e40ac600 0x7f31e40cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:11.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e415b5d0 0x7f31e4156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31e40b3d30 0x7f31e40c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e40ce140 con 0x7f31e415b5d0 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e8a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e4158340 con 0x7f31e40b3d30 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e40cd8e0 con 0x7f31e40ac600 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e415b5d0 0x7f31e4156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e415b5d0 0x7f31e4156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58156/0 (socket says 172.21.15.12:58156) 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e1400700 1 -- 172.21.15.12:0/4217429925 learned_addr learned my addr 172.21.15.12:0/4217429925 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e2800700 1 --2- 172.21.15.12:0/4217429925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31e40b3d30 0x7f31e40c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.203+0000 7f31e1e00700 1 --2- 172.21.15.12:0/4217429925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31e40ac600 0x7f31e40cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.204+0000 7f31e1400700 1 -- 172.21.15.12:0/4217429925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31e40ac600 msgr2=0x7f31e40cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.204+0000 7f31e1400700 1 --2- 172.21.15.12:0/4217429925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31e40ac600 0x7f31e40cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.204+0000 7f31e1400700 1 -- 172.21.15.12:0/4217429925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31e40b3d30 msgr2=0x7f31e40c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.204+0000 7f31e1400700 1 --2- 172.21.15.12:0/4217429925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31e40b3d30 0x7f31e40c94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:11.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.204+0000 7f31e1400700 1 -- 172.21.15.12:0/4217429925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d00094a0 con 0x7f31e415b5d0 2024-03-29T10:09:11.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.204+0000 7f31e1400700 1 --2- 172.21.15.12:0/4217429925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e415b5d0 0x7f31e4156b20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009b90 tx=0x7f31d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:11.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.204+0000 7f31da000700 1 -- 172.21.15.12:0/4217429925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d00045d0 con 0x7f31e415b5d0 2024-03-29T10:09:11.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.204+0000 7f31da000700 1 -- 172.21.15.12:0/4217429925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d001f070 con 0x7f31e415b5d0 2024-03-29T10:09:11.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.205+0000 7f31e8a00700 1 -- 172.21.15.12:0/4217429925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e4157060 con 0x7f31e415b5d0 2024-03-29T10:09:11.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.205+0000 7f31da000700 1 -- 172.21.15.12:0/4217429925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d00038c0 con 0x7f31e415b5d0 2024-03-29T10:09:11.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.205+0000 7f31e8a00700 1 -- 172.21.15.12:0/4217429925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e40cde60 con 0x7f31e415b5d0 2024-03-29T10:09:11.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.206+0000 7f31e8a00700 1 -- 172.21.15.12:0/4217429925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e4067720 con 0x7f31e415b5d0 2024-03-29T10:09:11.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.207+0000 7f31da000700 1 -- 172.21.15.12:0/4217429925 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 27) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f31d0003a20 con 0x7f31e415b5d0 2024-03-29T10:09:11.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.209+0000 7f31da000700 1 -- 172.21.15.12:0/4217429925 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f31d0021070 con 0x7f31e415b5d0 2024-03-29T10:09:11.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.213+0000 7f31da000700 1 -- 172.21.15.12:0/4217429925 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31d0055b30 con 0x7f31e415b5d0 2024-03-29T10:09:11.255 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:11.256+0000 7fcbd46e4240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:09:11.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.547+0000 7f31e8a00700 1 -- 172.21.15.12:0/4217429925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31e406ea60 con 0x7f31e415b5d0 2024-03-29T10:09:11.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.547+0000 7f31da000700 1 -- 172.21.15.12:0/4217429925 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f31d0015020 con 0x7f31e415b5d0 2024-03-29T10:09:11.547 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:11.547 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5746,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/981039791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":981039791},{"type":"v1","addr":"172.21.15.159:6836","nonce":981039791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1682878615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1682878615},{"type":"v1","addr":"172.21.15.159:6837","nonce":1682878615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5752,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1663013205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":1663013205},{"type":"v1","addr":"172.21.15.12:6834","nonce":1663013205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:07:08.705772+0000","modified":"2024-03-29T10:07:12.186244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5877},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5877":{"gid":5877,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3970376731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3970376731},{"type":"v1","addr":"172.21.15.12:6835","nonce":3970376731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-29T10:09:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.550+0000 7f31e8a00700 1 -- 172.21.15.12:0/4217429925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e415b5d0 msgr2=0x7f31e4156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.550+0000 7f31e8a00700 1 --2- 172.21.15.12:0/4217429925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e415b5d0 0x7f31e4156b20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009b90 tx=0x7f31d0004750 comp rx=0 tx=0).stop 2024-03-29T10:09:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.550+0000 7f31e8a00700 1 -- 172.21.15.12:0/4217429925 shutdown_connections 2024-03-29T10:09:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.550+0000 7f31e8a00700 1 --2- 172.21.15.12:0/4217429925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31e40b3d30 0x7f31e40c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.550+0000 7f31e8a00700 1 --2- 172.21.15.12:0/4217429925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31e40ac600 0x7f31e40cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.550+0000 7f31e8a00700 1 --2- 172.21.15.12:0/4217429925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e415b5d0 0x7f31e4156b20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.550+0000 7f31e8a00700 1 -- 172.21.15.12:0/4217429925 >> 172.21.15.12:0/4217429925 conn(0x7f31e40cf190 msgr2=0x7f31e415bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.550+0000 7f31e8a00700 1 -- 172.21.15.12:0/4217429925 shutdown_connections 2024-03-29T10:09:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.550+0000 7f31e8a00700 1 -- 172.21.15.12:0/4217429925 wait complete. 2024-03-29T10:09:11.551 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-29T10:09:11.567 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:09:11.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.765+0000 7fa5ca600700 1 Processor -- start 2024-03-29T10:09:11.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.765+0000 7fa5ca600700 1 -- start start 2024-03-29T10:09:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.766+0000 7fa5ca600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40cb3f0 0x7fa5c40ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.766+0000 7fa5ca600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c415fd50 con 0x7fa5c40ac600 2024-03-29T10:09:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.766+0000 7fa5ca600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c415f140 con 0x7fa5c415b5d0 2024-03-29T10:09:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.766+0000 7fa5ca600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c415b9b0 con 0x7fa5c40cb3f0 2024-03-29T10:09:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c3e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa5c40ac600 0x7fa5c406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40240/0 (socket says 172.21.15.12:40240) 2024-03-29T10:09:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c3e00700 1 -- 172.21.15.12:0/1731419727 learned_addr learned my addr 172.21.15.12:0/1731419727 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:11.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c2a00700 1 --2- 172.21.15.12:0/1731419727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40cb3f0 0x7fa5c40ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:11.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c2000700 1 -- 172.21.15.12:0/1731419727 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611999698 0 0) 0x7fa5c415fd50 con 0x7fa5c40ac600 2024-03-29T10:09:11.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c2000700 1 -- 172.21.15.12:0/1731419727 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a80035b0 con 0x7fa5c40ac600 2024-03-29T10:09:11.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c2000700 1 -- 172.21.15.12:0/1731419727 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919432515 0 0) 0x7fa5c415f140 con 0x7fa5c415b5d0 2024-03-29T10:09:11.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c2000700 1 -- 172.21.15.12:0/1731419727 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c415fd50 con 0x7fa5c415b5d0 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c2000700 1 -- 172.21.15.12:0/1731419727 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1421175943 0 0) 0x7fa5a80035b0 con 0x7fa5c40ac600 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c2a00700 1 -- 172.21.15.12:0/1731419727 >> v1:172.21.15.159:6789/0 conn(0x7fa5c415b5d0 legacy=0x7fa5c40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c2a00700 1 -- 172.21.15.12:0/1731419727 >> v1:172.21.15.12:6789/0 conn(0x7fa5c40ac600 legacy=0x7fa5c406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.767+0000 7fa5c2a00700 1 -- 172.21.15.12:0/1731419727 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c415f420 con 0x7fa5c40cb3f0 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.768+0000 7fa5c2a00700 1 --2- 172.21.15.12:0/1731419727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40cb3f0 0x7fa5c40ac0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0009870 tx=0x7fa5b0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eab7275892176bfa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.768+0000 7fa5c2000700 1 -- 172.21.15.12:0/1731419727 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0019f0 con 0x7fa5c40cb3f0 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.768+0000 7fa5c2000700 1 -- 172.21.15.12:0/1731419727 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b0004030 con 0x7fa5c40cb3f0 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.768+0000 7fa5c2000700 1 -- 172.21.15.12:0/1731419727 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b00043c0 con 0x7fa5c40cb3f0 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.768+0000 7fa5ca600700 1 -- 172.21.15.12:0/1731419727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40cb3f0 msgr2=0x7fa5c40ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.768+0000 7fa5ca600700 1 --2- 172.21.15.12:0/1731419727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40cb3f0 0x7fa5c40ac0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0009870 tx=0x7fa5b0009b80 comp rx=0 tx=0).stop 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.769+0000 7fa5ca600700 1 -- 172.21.15.12:0/1731419727 shutdown_connections 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.769+0000 7fa5ca600700 1 --2- 172.21.15.12:0/1731419727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40cb3f0 0x7fa5c40ac0c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.769+0000 7fa5ca600700 1 -- 172.21.15.12:0/1731419727 >> 172.21.15.12:0/1731419727 conn(0x7fa5c40cf190 msgr2=0x7fa5c40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.769+0000 7fa5ca600700 1 -- 172.21.15.12:0/1731419727 shutdown_connections 2024-03-29T10:09:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.769+0000 7fa5ca600700 1 -- 172.21.15.12:0/1731419727 wait complete. 2024-03-29T10:09:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.770+0000 7fa5ca600700 1 Processor -- start 2024-03-29T10:09:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.771+0000 7fa5ca600700 1 -- start start 2024-03-29T10:09:11.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.771+0000 7fa5ca600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c40ac600 0x7fa5c415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:11.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.771+0000 7fa5ca600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c415b5d0 0x7fa5c41f6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.771+0000 7fa5ca600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40b07c0 0x7fa5c41f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5ca600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c40ca8a0 con 0x7fa5c415b5d0 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5ca600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c40c9cd0 con 0x7fa5c40ac600 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5ca600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c40c9f70 con 0x7fa5c40b07c0 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c415b5d0 0x7fa5c41f6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c415b5d0 0x7fa5c41f6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58158/0 (socket says 172.21.15.12:58158) 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c2a00700 1 -- 172.21.15.12:0/351029461 learned_addr learned my addr 172.21.15.12:0/351029461 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c3400700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c40ac600 0x7fa5c415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c3e00700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40b07c0 0x7fa5c41f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c2a00700 1 -- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40b07c0 msgr2=0x7fa5c41f73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c2a00700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40b07c0 0x7fa5c41f73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c2a00700 1 -- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c40ac600 msgr2=0x7fa5c415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:11.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c2a00700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c40ac600 0x7fa5c415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:11.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c2a00700 1 -- 172.21.15.12:0/351029461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b00094a0 con 0x7fa5c415b5d0 2024-03-29T10:09:11.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.772+0000 7fa5c2a00700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c415b5d0 0x7fa5c41f6e70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0000c00 tx=0x7fa5b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:11.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.773+0000 7fa5bb400700 1 -- 172.21.15.12:0/351029461 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0011630 con 0x7fa5c415b5d0 2024-03-29T10:09:11.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.773+0000 7fa5bb400700 1 -- 172.21.15.12:0/351029461 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b001f070 con 0x7fa5c415b5d0 2024-03-29T10:09:11.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.773+0000 7fa5bb400700 1 -- 172.21.15.12:0/351029461 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b00119f0 con 0x7fa5c415b5d0 2024-03-29T10:09:11.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.773+0000 7fa5ca600700 1 -- 172.21.15.12:0/351029461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c40ca250 con 0x7fa5c415b5d0 2024-03-29T10:09:11.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.773+0000 7fa5ca600700 1 -- 172.21.15.12:0/351029461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c415e9c0 con 0x7fa5c415b5d0 2024-03-29T10:09:11.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.774+0000 7fa5ca600700 1 -- 172.21.15.12:0/351029461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c40caa50 con 0x7fa5c415b5d0 2024-03-29T10:09:11.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.775+0000 7fa5bb400700 1 -- 172.21.15.12:0/351029461 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 27) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fa5b0011b50 con 0x7fa5c415b5d0 2024-03-29T10:09:11.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.776+0000 7fa5bb400700 1 -- 172.21.15.12:0/351029461 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa5b0021070 con 0x7fa5c415b5d0 2024-03-29T10:09:11.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.780+0000 7fa5bb400700 1 -- 172.21.15.12:0/351029461 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa5c40caa50 con 0x7fa5c415b5d0 2024-03-29T10:09:11.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.822+0000 7fa5bb400700 1 -- 172.21.15.12:0/351029461 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 28) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fa5b0055880 con 0x7fa5c415b5d0 2024-03-29T10:09:11.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.823+0000 7fa5bb400700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa5a4036150 0x7fa5a40723e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:11.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.826+0000 7fa5c3400700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa5a4036150 0x7fa5a40723e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:11.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:11.828+0000 7fa5c3400700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa5a4036150 0x7fa5a40723e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4001220 tx=0x7fa5b4009040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.002+0000 7fa5ca600700 1 -- 172.21.15.12:0/351029461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5c40caa50 con 0x7fa5c415b5d0 2024-03-29T10:09:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.003+0000 7fa5bb400700 1 -- 172.21.15.12:0/351029461 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+17076 (secure 0 0 0) 0x7fa5c40caa50 con 0x7fa5c415b5d0 2024-03-29T10:09:12.003 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:12.004 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":67,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:10.812887+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:07:07.106799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-29T10:07:07.671044+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":5}]},{"pgid":"7.12","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/838078716":"2024-03-30T10:09:10.812838+0000","172.21.15.159:0/636368871":"2024-03-30T10:09:10.812838+0000","172.21.15.159:0/509611987":"2024-03-30T10:09:10.812838+0000","172.21.15.159:0/286166981":"2024-03-30T10:09:10.812838+0000","172.21.15.159:6833/43444":"2024-03-30T10:07:25.860642+0000","172.21.15.159:6832/43444":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/868835752":"2024-03-30T10:09:10.812838+0000","172.21.15.159:0/3220343982":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/2775778725":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3363559414":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/1116927207":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3950908519":"2024-03-30T10:07:25.860642+0000","172.21.15.159:0/3535512468":"2024-03-30T10:09:10.812838+0000","172.21.15.159:0/798085437":"2024-03-30T10:07:25.860642+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:09:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 -- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa5a4036150 msgr2=0x7fa5a40723e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa5a4036150 0x7fa5a40723e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4001220 tx=0x7fa5b4009040 comp rx=0 tx=0).stop 2024-03-29T10:09:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 -- 172.21.15.12:0/351029461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c415b5d0 msgr2=0x7fa5c41f6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c415b5d0 0x7fa5c41f6e70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0000c00 tx=0x7fa5b0004750 comp rx=0 tx=0).stop 2024-03-29T10:09:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 -- 172.21.15.12:0/351029461 shutdown_connections 2024-03-29T10:09:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa5a4036150 0x7fa5a40723e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c40b07c0 0x7fa5c41f73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c40ac600 0x7fa5c415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 --2- 172.21.15.12:0/351029461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c415b5d0 0x7fa5c41f6e70 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 -- 172.21.15.12:0/351029461 >> 172.21.15.12:0/351029461 conn(0x7fa5c40cf190 msgr2=0x7fa5c415bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 -- 172.21.15.12:0/351029461 shutdown_connections 2024-03-29T10:09:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.006+0000 7fa5ca600700 1 -- 172.21.15.12:0/351029461 wait complete. 2024-03-29T10:09:12.024 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T10:09:12.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.222+0000 7fea46e00700 1 Processor -- start 2024-03-29T10:09:12.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.222+0000 7fea46e00700 1 -- start start 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.223+0000 7fea46e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015d9d0 0x7fea400d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.223+0000 7fea46e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fea400d2540 0x7fea400ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.224+0000 7fea46e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea40161420 con 0x7fea400d16a0 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.224+0000 7fea46e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea4015edc0 con 0x7fea400d2540 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.224+0000 7fea46e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea40161700 con 0x7fea4015d9d0 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.224+0000 7fea3f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fea400d16a0 0x7fea400d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40248/0 (socket says 172.21.15.12:40248) 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.224+0000 7fea3f400700 1 -- 172.21.15.12:0/1146221571 learned_addr learned my addr 172.21.15.12:0/1146221571 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.224+0000 7fea44e00700 1 --2- 172.21.15.12:0/1146221571 >> v2:172.21.15.159:3300/0 conn(0x7fea400d2540 0x7fea400ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.224+0000 7fea3fe00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015d9d0 0x7fea400d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.224+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290495884 0 0) 0x7fea40161420 con 0x7fea400d16a0 2024-03-29T10:09:12.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.224+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea240035b0 con 0x7fea400d16a0 2024-03-29T10:09:12.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.224+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1072039875 0 0) 0x7fea240035b0 con 0x7fea400d16a0 2024-03-29T10:09:12.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015d9d0 msgr2=0x7fea400d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:12.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015d9d0 0x7fea400d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 >> v2:172.21.15.159:3300/0 conn(0x7fea400d2540 msgr2=0x7fea400ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:12.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 --2- 172.21.15.12:0/1146221571 >> v2:172.21.15.159:3300/0 conn(0x7fea400d2540 0x7fea400ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea401619e0 con 0x7fea400d16a0 2024-03-29T10:09:12.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fea2c0017f0 con 0x7fea400d16a0 2024-03-29T10:09:12.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 >> v1:172.21.15.12:6789/0 conn(0x7fea400d16a0 legacy=0x7fea400d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea24003a00 0x7fea24003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea24004850 0x7fea24004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015d9d0 0x7fea24005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea40161700 con 0x7fea24003a00 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2400ba10 con 0x7fea24004850 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea2400bb50 con 0x7fea4015d9d0 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3fe00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea24003a00 0x7fea24003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea44e00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015d9d0 0x7fea24005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3f400700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea24004850 0x7fea24004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3fe00700 1 -- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015d9d0 msgr2=0x7fea24005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3fe00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015d9d0 0x7fea24005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3fe00700 1 -- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea24004850 msgr2=0x7fea24004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:12.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3fe00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea24004850 0x7fea24004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.225+0000 7fea3fe00700 1 -- 172.21.15.12:0/1146221571 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea240035b0 con 0x7fea24003a00 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.226+0000 7fea3fe00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea24003a00 0x7fea24003e20 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fea30009b60 tx=0x7fea24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edfff479461a5c3d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.226+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3000e660 con 0x7fea24003a00 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.226+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3000e7c0 con 0x7fea24003a00 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.226+0000 7fea3ea00700 1 -- 172.21.15.12:0/1146221571 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3000eb30 con 0x7fea24003a00 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.226+0000 7fea46e00700 1 -- 172.21.15.12:0/1146221571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea24003a00 msgr2=0x7fea24003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.226+0000 7fea46e00700 1 --2- 172.21.15.12:0/1146221571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea24003a00 0x7fea24003e20 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fea30009b60 tx=0x7fea24005880 comp rx=0 tx=0).stop 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.226+0000 7fea44e00700 1 -- 172.21.15.12:0/1146221571 reap_dead start 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.227+0000 7fea46e00700 1 -- 172.21.15.12:0/1146221571 shutdown_connections 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.227+0000 7fea46e00700 1 -- 172.21.15.12:0/1146221571 >> 172.21.15.12:0/1146221571 conn(0x7fea4008aff0 msgr2=0x7fea400d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.227+0000 7fea46e00700 1 -- 172.21.15.12:0/1146221571 shutdown_connections 2024-03-29T10:09:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.227+0000 7fea46e00700 1 -- 172.21.15.12:0/1146221571 wait complete. 2024-03-29T10:09:12.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.228+0000 7fea46e00700 1 Processor -- start 2024-03-29T10:09:12.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.229+0000 7fea46e00700 1 -- start start 2024-03-29T10:09:12.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.229+0000 7fea46e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea24003a00 0x7fea400b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.229+0000 7fea46e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea400c92c0 0x7fea400b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea46e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015f840 0x7fea4015fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea46e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400b6370 con 0x7fea400c92c0 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea46e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400b6020 con 0x7fea24003a00 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea46e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea40160570 con 0x7fea4015f840 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea3f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea400c92c0 0x7fea400b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea3f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea400c92c0 0x7fea400b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58170/0 (socket says 172.21.15.12:58170) 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea3f400700 1 -- 172.21.15.12:0/1994411492 learned_addr learned my addr 172.21.15.12:0/1994411492 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea3f400700 1 -- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015f840 msgr2=0x7fea4015fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea3f400700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015f840 0x7fea4015fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea3f400700 1 -- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea24003a00 msgr2=0x7fea400b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea3f400700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea24003a00 0x7fea400b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea3f400700 1 -- 172.21.15.12:0/1994411492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea40160850 con 0x7fea400c92c0 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.230+0000 7fea3f400700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea400c92c0 0x7fea400b4a00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fea2c006f50 tx=0x7fea2c0073d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:12.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.231+0000 7fea37e00700 1 -- 172.21.15.12:0/1994411492 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c011580 con 0x7fea400c92c0 2024-03-29T10:09:12.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.231+0000 7fea37e00700 1 -- 172.21.15.12:0/1994411492 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2c011720 con 0x7fea400c92c0 2024-03-29T10:09:12.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.231+0000 7fea46e00700 1 -- 172.21.15.12:0/1994411492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea40160b30 con 0x7fea400c92c0 2024-03-29T10:09:12.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.231+0000 7fea37e00700 1 -- 172.21.15.12:0/1994411492 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c011ad0 con 0x7fea400c92c0 2024-03-29T10:09:12.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.231+0000 7fea46e00700 1 -- 172.21.15.12:0/1994411492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea40144410 con 0x7fea400c92c0 2024-03-29T10:09:12.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.232+0000 7fea46e00700 1 -- 172.21.15.12:0/1994411492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea400d1ca0 con 0x7fea400c92c0 2024-03-29T10:09:12.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.233+0000 7fea37e00700 1 -- 172.21.15.12:0/1994411492 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 28) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fea2c011c30 con 0x7fea400c92c0 2024-03-29T10:09:12.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.233+0000 7fea37e00700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea2006aae0 0x7fea2006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.234+0000 7fea37e00700 1 -- 172.21.15.12:0/1994411492 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fea2c089330 con 0x7fea400c92c0 2024-03-29T10:09:12.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.234+0000 7fea3fe00700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea2006aae0 0x7fea2006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.235+0000 7fea3fe00700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea2006aae0 0x7fea2006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fea400b54b0 tx=0x7fea30013000 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:12.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.238+0000 7fea37e00700 1 -- 172.21.15.12:0/1994411492 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea2c0544d0 con 0x7fea400c92c0 2024-03-29T10:09:12.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.505+0000 7fea46e00700 1 -- 172.21.15.12:0/1994411492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fea400b64b0 con 0x7fea400c92c0 2024-03-29T10:09:12.505 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T10:09:12.506+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T10:09:12.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.520+0000 7fea37e00700 1 -- 172.21.15.12:0/1994411492 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) v1 ==== 161+0+0 (secure 0 0 0) 0x7fea400b64b0 con 0x7fea400c92c0 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 -- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea2006aae0 msgr2=0x7fea2006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea2006aae0 0x7fea2006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fea400b54b0 tx=0x7fea30013000 comp rx=0 tx=0).stop 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 -- 172.21.15.12:0/1994411492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea400c92c0 msgr2=0x7fea400b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea400c92c0 0x7fea400b4a00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fea2c006f50 tx=0x7fea2c0073d0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 -- 172.21.15.12:0/1994411492 shutdown_connections 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea2006aae0 0x7fea2006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea4015f840 0x7fea4015fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea24003a00 0x7fea400b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 --2- 172.21.15.12:0/1994411492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea400c92c0 0x7fea400b4a00 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 -- 172.21.15.12:0/1994411492 >> 172.21.15.12:0/1994411492 conn(0x7fea4008aff0 msgr2=0x7fea400b06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 -- 172.21.15.12:0/1994411492 shutdown_connections 2024-03-29T10:09:12.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.523+0000 7fea46e00700 1 -- 172.21.15.12:0/1994411492 wait complete. 2024-03-29T10:09:12.524 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T10:09:12.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T10:09:12.588 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:09:12.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.744+0000 7ff37d400700 1 Processor -- start 2024-03-29T10:09:12.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.745+0000 7ff37d400700 1 -- start start 2024-03-29T10:09:12.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.746+0000 7ff37d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780cd020 0x7ff3780cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.746+0000 7ff37d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3780b2070 0x7ff3780b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.746+0000 7ff37d400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3780cdae0 con 0x7ff3780cbb30 2024-03-29T10:09:12.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.746+0000 7ff37d400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3780cddc0 con 0x7ff3780b2070 2024-03-29T10:09:12.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.746+0000 7ff37d400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3780ce0a0 con 0x7ff3780cd020 2024-03-29T10:09:12.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.746+0000 7ff377200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3780b2070 0x7ff3780b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.746+0000 7ff377200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3780b2070 0x7ff3780b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58198/0 (socket says 172.21.15.12:58198) 2024-03-29T10:09:12.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.746+0000 7ff377200700 1 -- 172.21.15.12:0/4051115188 learned_addr learned my addr 172.21.15.12:0/4051115188 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:12.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.746+0000 7ff375e00700 1 --2- 172.21.15.12:0/4051115188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780cd020 0x7ff3780cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.747+0000 7ff377200700 1 -- 172.21.15.12:0/4051115188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780cd020 msgr2=0x7ff3780cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:12.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.747+0000 7ff377200700 1 --2- 172.21.15.12:0/4051115188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780cd020 0x7ff3780cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.747+0000 7ff377200700 1 -- 172.21.15.12:0/4051115188 >> v1:172.21.15.159:6789/0 conn(0x7ff3780cbb30 legacy=0x7ff3780cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.747+0000 7ff377200700 1 -- 172.21.15.12:0/4051115188 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37806dc20 con 0x7ff3780b2070 2024-03-29T10:09:12.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.747+0000 7ff375e00700 1 --2- 172.21.15.12:0/4051115188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780cd020 0x7ff3780cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:12.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.747+0000 7ff377200700 1 --2- 172.21.15.12:0/4051115188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780b2070 0x7ff3780b2500 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff370009e50 tx=0x7ff37000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee23cfa5299f3b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:12.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.747+0000 7ff375400700 1 -- 172.21.15.12:0/4051115188 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37000e070 con 0x7ff3780b2070 2024-03-29T10:09:12.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.747+0000 7ff375400700 1 -- 172.21.15.12:0/4051115188 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37000b9c0 con 0x7ff3780b2070 2024-03-29T10:09:12.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.747+0000 7ff375400700 1 -- 172.21.15.12:0/4051115188 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37000bd70 con 0x7ff3780b2070 2024-03-29T10:09:12.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.748+0000 7ff37d400700 1 -- 172.21.15.12:0/4051115188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780b2070 msgr2=0x7ff3780b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.748+0000 7ff37d400700 1 --2- 172.21.15.12:0/4051115188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780b2070 0x7ff3780b2500 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff370009e50 tx=0x7ff37000b330 comp rx=0 tx=0).stop 2024-03-29T10:09:12.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.748+0000 7ff37d400700 1 -- 172.21.15.12:0/4051115188 shutdown_connections 2024-03-29T10:09:12.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.748+0000 7ff37d400700 1 --2- 172.21.15.12:0/4051115188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780b2070 0x7ff3780b2500 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.748+0000 7ff37d400700 1 --2- 172.21.15.12:0/4051115188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780cd020 0x7ff3780cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.748+0000 7ff37d400700 1 -- 172.21.15.12:0/4051115188 >> 172.21.15.12:0/4051115188 conn(0x7ff3780b0d70 msgr2=0x7ff3780b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:12.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.748+0000 7ff37d400700 1 -- 172.21.15.12:0/4051115188 shutdown_connections 2024-03-29T10:09:12.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.748+0000 7ff37d400700 1 -- 172.21.15.12:0/4051115188 wait complete. 2024-03-29T10:09:12.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.750+0000 7ff37d400700 1 Processor -- start 2024-03-29T10:09:12.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.750+0000 7ff37d400700 1 -- start start 2024-03-29T10:09:12.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff37d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780b2070 0x7ff3781f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff37d400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3780cbb30 0x7ff3781553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff37d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780cd020 0x7ff378071f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff37d400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3781f8050 con 0x7ff3780cd020 2024-03-29T10:09:12.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff37d400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3781f70c0 con 0x7ff3780cbb30 2024-03-29T10:09:12.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff37d400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3780ac050 con 0x7ff3780b2070 2024-03-29T10:09:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff377200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780cd020 0x7ff378071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff377200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780cd020 0x7ff378071f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58208/0 (socket says 172.21.15.12:58208) 2024-03-29T10:09:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff377200700 1 -- 172.21.15.12:0/1740897865 learned_addr learned my addr 172.21.15.12:0/1740897865 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff375e00700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3780cbb30 0x7ff3781553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.751+0000 7ff376800700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780b2070 0x7ff3781f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff377200700 1 -- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780b2070 msgr2=0x7ff3781f7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff377200700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780b2070 0x7ff3781f7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff377200700 1 -- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3780cbb30 msgr2=0x7ff3781553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff377200700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3780cbb30 0x7ff3781553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:12.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff377200700 1 -- 172.21.15.12:0/1740897865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff370009b30 con 0x7ff3780cd020 2024-03-29T10:09:12.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff377200700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780cd020 0x7ff378071f00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff37806c130 tx=0x7ff370004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:12.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff36ea00700 1 -- 172.21.15.12:0/1740897865 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370004800 con 0x7ff3780cd020 2024-03-29T10:09:12.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff36ea00700 1 -- 172.21.15.12:0/1740897865 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff370004960 con 0x7ff3780cd020 2024-03-29T10:09:12.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff36ea00700 1 -- 172.21.15.12:0/1740897865 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37001c680 con 0x7ff3780cd020 2024-03-29T10:09:12.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff37d400700 1 -- 172.21.15.12:0/1740897865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3780ac2f0 con 0x7ff3780cd020 2024-03-29T10:09:12.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.752+0000 7ff37d400700 1 -- 172.21.15.12:0/1740897865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3781f7310 con 0x7ff3780cd020 2024-03-29T10:09:12.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.753+0000 7ff37d400700 1 -- 172.21.15.12:0/1740897865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3780b3fa0 con 0x7ff3780cd020 2024-03-29T10:09:12.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.754+0000 7ff36ea00700 1 -- 172.21.15.12:0/1740897865 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 28) v1 ==== 88975+0+0 (secure 0 0 0) 0x7ff37001c7e0 con 0x7ff3780cd020 2024-03-29T10:09:12.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.755+0000 7ff36ea00700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35806aae0 0x7ff35806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:12.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.755+0000 7ff36ea00700 1 -- 172.21.15.12:0/1740897865 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6483+0+0 (secure 0 0 0) 0x7ff37008ae40 con 0x7ff3780cd020 2024-03-29T10:09:12.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.756+0000 7ff376800700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35806aae0 0x7ff35806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:12.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.757+0000 7ff376800700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35806aae0 0x7ff35806cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff378155e10 tx=0x7ff368009040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:12.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:12.759+0000 7ff36ea00700 1 -- 172.21.15.12:0/1740897865 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff370056050 con 0x7ff3780cd020 2024-03-29T10:09:13.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.021+0000 7ff37d400700 1 -- 172.21.15.12:0/1740897865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff378067720 con 0x7ff3780cd020 2024-03-29T10:09:13.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.522+0000 7ff36ea00700 1 -- 172.21.15.12:0/1740897865 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff370059670 con 0x7ff3780cd020 2024-03-29T10:09:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 -- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35806aae0 msgr2=0x7ff35806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35806aae0 0x7ff35806cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff378155e10 tx=0x7ff368009040 comp rx=0 tx=0).stop 2024-03-29T10:09:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 -- 172.21.15.12:0/1740897865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780cd020 msgr2=0x7ff378071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780cd020 0x7ff378071f00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff37806c130 tx=0x7ff370004ad0 comp rx=0 tx=0).stop 2024-03-29T10:09:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 -- 172.21.15.12:0/1740897865 shutdown_connections 2024-03-29T10:09:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff35806aae0 0x7ff35806cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3780cd020 0x7ff378071f00 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3780b2070 0x7ff3781f7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:13.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 --2- 172.21.15.12:0/1740897865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3780cbb30 0x7ff3781553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:13.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 -- 172.21.15.12:0/1740897865 >> 172.21.15.12:0/1740897865 conn(0x7ff3780b0d70 msgr2=0x7ff37815cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:13.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.525+0000 7ff37d400700 1 -- 172.21.15.12:0/1740897865 shutdown_connections 2024-03-29T10:09:13.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.526+0000 7ff37d400700 1 -- 172.21.15.12:0/1740897865 wait complete. 2024-03-29T10:09:13.547 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T10:09:13.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.794+0000 7f66ccc00700 1 Processor -- start 2024-03-29T10:09:13.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.794+0000 7f66ccc00700 1 -- start start 2024-03-29T10:09:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.795+0000 7f66ccc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66c80d28e0 0x7f66c8168680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.795+0000 7f66ccc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c80ce8a0 con 0x7f66c80ced30 2024-03-29T10:09:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.795+0000 7f66ccc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c80cdc90 con 0x7f66c815ddf0 2024-03-29T10:09:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.795+0000 7f66ccc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c815ec60 con 0x7f66c80d28e0 2024-03-29T10:09:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.795+0000 7f66c6800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66c80d28e0 0x7f66c8168680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.795+0000 7f66c6800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66c80d28e0 0x7f66c8168680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58222/0 (socket says 172.21.15.12:58222) 2024-03-29T10:09:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.795+0000 7f66c5400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f66c80ced30 0x7f66c80d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40266/0 (socket says 172.21.15.12:40266) 2024-03-29T10:09:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.795+0000 7f66c6800700 1 -- 172.21.15.12:0/2318345235 learned_addr learned my addr 172.21.15.12:0/2318345235 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:13.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.796+0000 7f66c6800700 1 -- 172.21.15.12:0/2318345235 >> v1:172.21.15.159:6789/0 conn(0x7f66c815ddf0 legacy=0x7f66c815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:13.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.796+0000 7f66c6800700 1 -- 172.21.15.12:0/2318345235 >> v1:172.21.15.12:6789/0 conn(0x7f66c80ced30 legacy=0x7f66c80d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:13.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.796+0000 7f66c6800700 1 -- 172.21.15.12:0/2318345235 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c80cdf70 con 0x7f66c80d28e0 2024-03-29T10:09:13.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.796+0000 7f66c6800700 1 --2- 172.21.15.12:0/2318345235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c80d28e0 0x7f66c8168680 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009e50 tx=0x7f66c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361d167286e5cf67 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:13.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.796+0000 7f66c4a00700 1 -- 172.21.15.12:0/2318345235 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c000e070 con 0x7f66c80d28e0 2024-03-29T10:09:13.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.796+0000 7f66c4a00700 1 -- 172.21.15.12:0/2318345235 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c000b9c0 con 0x7f66c80d28e0 2024-03-29T10:09:13.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.796+0000 7f66c4a00700 1 -- 172.21.15.12:0/2318345235 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c000bd70 con 0x7f66c80d28e0 2024-03-29T10:09:13.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.797+0000 7f66ccc00700 1 -- 172.21.15.12:0/2318345235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c80d28e0 msgr2=0x7f66c8168680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:13.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.797+0000 7f66ccc00700 1 --2- 172.21.15.12:0/2318345235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c80d28e0 0x7f66c8168680 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009e50 tx=0x7f66c000b330 comp rx=0 tx=0).stop 2024-03-29T10:09:13.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.797+0000 7f66ccc00700 1 -- 172.21.15.12:0/2318345235 shutdown_connections 2024-03-29T10:09:13.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.797+0000 7f66ccc00700 1 --2- 172.21.15.12:0/2318345235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c80d28e0 0x7f66c8168680 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:13.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.797+0000 7f66ccc00700 1 -- 172.21.15.12:0/2318345235 >> 172.21.15.12:0/2318345235 conn(0x7f66c808aff0 msgr2=0x7f66c80d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:13.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.797+0000 7f66ccc00700 1 -- 172.21.15.12:0/2318345235 shutdown_connections 2024-03-29T10:09:13.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.797+0000 7f66ccc00700 1 -- 172.21.15.12:0/2318345235 wait complete. 2024-03-29T10:09:13.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.799+0000 7f66ccc00700 1 Processor -- start 2024-03-29T10:09:13.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.799+0000 7f66ccc00700 1 -- start start 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66ccc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66c80ced30 0x7f66c815bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66ccc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66c815ddf0 0x7f66c81e8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66ccc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c815c3e0 0x7f66c80c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66ccc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c81e9950 con 0x7f66c815c3e0 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66ccc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c815cc30 con 0x7f66c80ced30 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66ccc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c815c190 con 0x7f66c815ddf0 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66c6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c815c3e0 0x7f66c80c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66c6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c815c3e0 0x7f66c80c89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58238/0 (socket says 172.21.15.12:58238) 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66c6800700 1 -- 172.21.15.12:0/4165767123 learned_addr learned my addr 172.21.15.12:0/4165767123 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66c5400700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66c815ddf0 0x7f66c81e8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:13.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.800+0000 7f66c5e00700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66c80ced30 0x7f66c815bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66c6800700 1 -- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66c815ddf0 msgr2=0x7f66c81e8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66c6800700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66c815ddf0 0x7f66c81e8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66c6800700 1 -- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66c80ced30 msgr2=0x7f66c815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66c6800700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66c80ced30 0x7f66c815bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66c6800700 1 -- 172.21.15.12:0/4165767123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c0009b30 con 0x7f66c815c3e0 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66c6800700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c815c3e0 0x7f66c80c89d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f66c0012000 tx=0x7f66c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66be000700 1 -- 172.21.15.12:0/4165767123 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0004960 con 0x7f66c815c3e0 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66be000700 1 -- 172.21.15.12:0/4165767123 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c00099b0 con 0x7f66c815c3e0 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66ccc00700 1 -- 172.21.15.12:0/4165767123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c81e9390 con 0x7f66c815c3e0 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66be000700 1 -- 172.21.15.12:0/4165767123 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c001c680 con 0x7f66c815c3e0 2024-03-29T10:09:13.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.801+0000 7f66ccc00700 1 -- 172.21.15.12:0/4165767123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c81e7c70 con 0x7f66c815c3e0 2024-03-29T10:09:13.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.802+0000 7f66ccc00700 1 -- 172.21.15.12:0/4165767123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c8067720 con 0x7f66c815c3e0 2024-03-29T10:09:13.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.803+0000 7f66be000700 1 -- 172.21.15.12:0/4165767123 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 29) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f66c001c7e0 con 0x7f66c815c3e0 2024-03-29T10:09:13.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.804+0000 7f66be000700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f66a806ad20 0x7f66a806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:13.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.804+0000 7f66be000700 1 -- 172.21.15.12:0/4165767123 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f66c008b320 con 0x7f66c815c3e0 2024-03-29T10:09:13.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.805+0000 7f66c5e00700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f66a806ad20 0x7f66a806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:13.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.806+0000 7f66c5e00700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f66a806ad20 0x7f66a806d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f66b8000f60 tx=0x7f66b8009040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:13.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:13.809+0000 7f66be000700 1 -- 172.21.15.12:0/4165767123 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f66c0059b10 con 0x7f66c815c3e0 2024-03-29T10:09:14.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.025+0000 7f66ccc00700 1 -- 172.21.15.12:0/4165767123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f66c80d3050 con 0x7f66c815c3e0 2024-03-29T10:09:14.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.829+0000 7f66be000700 1 -- 172.21.15.12:0/4165767123 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v70) v1 ==== 181+0+0 (secure 0 0 0) 0x7f66c00596a0 con 0x7f66c815c3e0 2024-03-29T10:09:14.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.833+0000 7f66ccc00700 1 -- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f66a806ad20 msgr2=0x7f66a806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:14.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.833+0000 7f66ccc00700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f66a806ad20 0x7f66a806d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f66b8000f60 tx=0x7f66b8009040 comp rx=0 tx=0).stop 2024-03-29T10:09:14.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.834+0000 7f66ccc00700 1 -- 172.21.15.12:0/4165767123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c815c3e0 msgr2=0x7f66c80c89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:14.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.834+0000 7f66ccc00700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c815c3e0 0x7f66c80c89d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f66c0012000 tx=0x7f66c0004ad0 comp rx=0 tx=0).stop 2024-03-29T10:09:14.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.834+0000 7f66ccc00700 1 -- 172.21.15.12:0/4165767123 shutdown_connections 2024-03-29T10:09:14.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.834+0000 7f66ccc00700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f66a806ad20 0x7f66a806d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:14.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.834+0000 7f66ccc00700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66c815c3e0 0x7f66c80c89d0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:14.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.834+0000 7f66ccc00700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66c80ced30 0x7f66c815bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:14.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.834+0000 7f66ccc00700 1 --2- 172.21.15.12:0/4165767123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66c815ddf0 0x7f66c81e8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:14.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.834+0000 7f66ccc00700 1 -- 172.21.15.12:0/4165767123 >> 172.21.15.12:0/4165767123 conn(0x7f66c808aff0 msgr2=0x7f66c80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:14.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.834+0000 7f66ccc00700 1 -- 172.21.15.12:0/4165767123 shutdown_connections 2024-03-29T10:09:14.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:14.834+0000 7f66ccc00700 1 -- 172.21.15.12:0/4165767123 wait complete. 2024-03-29T10:09:14.835 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T10:09:14.855 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T10:09:15.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.062+0000 7ff683a00700 1 Processor -- start 2024-03-29T10:09:15.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.062+0000 7ff683a00700 1 -- start start 2024-03-29T10:09:15.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.063+0000 7ff683a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff67c15b330 0x7ff67c1652f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:15.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.063+0000 7ff683a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c070980 con 0x7ff67c1580c0 2024-03-29T10:09:15.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.063+0000 7ff683a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c15b7a0 con 0x7ff67c06e9c0 2024-03-29T10:09:15.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.063+0000 7ff683a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff67c071570 con 0x7ff67c15b330 2024-03-29T10:09:15.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.063+0000 7ff681000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff67c1580c0 0x7ff67c06e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40270/0 (socket says 172.21.15.12:40270) 2024-03-29T10:09:15.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.063+0000 7ff681000700 1 -- 172.21.15.12:0/1323710698 learned_addr learned my addr 172.21.15.12:0/1323710698 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336101969 0 0) 0x7ff67c070980 con 0x7ff67c1580c0 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6640057c0 con 0x7ff67c1580c0 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff681a00700 1 --2- 172.21.15.12:0/1323710698 >> v2:172.21.15.159:3300/0 conn(0x7ff67c15b330 0x7ff67c1652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3228623963 0 0) 0x7ff6640057c0 con 0x7ff67c1580c0 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 >> v2:172.21.15.159:3300/0 conn(0x7ff67c15b330 msgr2=0x7ff67c1652f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff67b400700 1 --2- 172.21.15.12:0/1323710698 >> v2:172.21.15.159:3300/0 conn(0x7ff67c15b330 0x7ff67c1652f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 >> v1:172.21.15.159:6789/0 conn(0x7ff67c06e9c0 legacy=0x7ff67c15ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c070bd0 con 0x7ff67c1580c0 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7ff670001670 con 0x7ff67c1580c0 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff681a00700 1 --2- 172.21.15.12:0/1323710698 >> v2:172.21.15.159:3300/0 conn(0x7ff67c15b330 0x7ff67c1652f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 >> v1:172.21.15.12:6789/0 conn(0x7ff67c1580c0 legacy=0x7ff67c06e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.064+0000 7ff67b400700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff664005c60 0x7ff664006080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:15.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff67b400700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff664006a60 0x7ff664006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:15.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff67b400700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff664007540 0x7ff66400dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:15.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c071570 con 0x7ff664005c60 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66400e460 con 0x7ff664006a60 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66400e620 con 0x7ff664007540 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff681000700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff664005c60 0x7ff664006080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff67be00700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff664006a60 0x7ff664006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff681a00700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff664007540 0x7ff66400dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff681000700 1 -- 172.21.15.12:0/1323710698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff664007540 msgr2=0x7ff66400dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff681000700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff664007540 0x7ff66400dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff681000700 1 -- 172.21.15.12:0/1323710698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff664006a60 msgr2=0x7ff664006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff681000700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff664006a60 0x7ff664006ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff681000700 1 -- 172.21.15.12:0/1323710698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640057c0 con 0x7ff664005c60 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff67be00700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff664006a60 0x7ff664006ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:15.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff681a00700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff664007540 0x7ff66400dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.065+0000 7ff681a00700 1 -- 172.21.15.12:0/1323710698 reap_dead start 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff681000700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff664005c60 0x7ff664006080 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ff67000c160 tx=0x7ff664007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7297f8bae7ceab server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670007b70 con 0x7ff664005c60 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67000cd20 con 0x7ff664005c60 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff67b400700 1 -- 172.21.15.12:0/1323710698 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670005ba0 con 0x7ff664005c60 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff683a00700 1 -- 172.21.15.12:0/1323710698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff664005c60 msgr2=0x7ff664006080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff683a00700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff664005c60 0x7ff664006080 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ff67000c160 tx=0x7ff664007a90 comp rx=0 tx=0).stop 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff683a00700 1 -- 172.21.15.12:0/1323710698 shutdown_connections 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff683a00700 1 --2- 172.21.15.12:0/1323710698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff664005c60 0x7ff664006080 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff683a00700 1 -- 172.21.15.12:0/1323710698 >> 172.21.15.12:0/1323710698 conn(0x7ff67c0cf210 msgr2=0x7ff67c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff683a00700 1 -- 172.21.15.12:0/1323710698 shutdown_connections 2024-03-29T10:09:15.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.066+0000 7ff683a00700 1 -- 172.21.15.12:0/1323710698 wait complete. 2024-03-29T10:09:15.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.068+0000 7ff683a00700 1 Processor -- start 2024-03-29T10:09:15.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.068+0000 7ff683a00700 1 -- start start 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.069+0000 7ff683a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff664005c60 0x7ff67c1544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.069+0000 7ff683a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff67c1f7310 0x7ff67c0b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.069+0000 7ff683a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff67c15b9f0 0x7ff67c15bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.069+0000 7ff683a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c071990 con 0x7ff67c15b9f0 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.069+0000 7ff683a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c0c9ea0 con 0x7ff664005c60 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.069+0000 7ff683a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c1f7790 con 0x7ff67c1f7310 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.069+0000 7ff681a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff67c15b9f0 0x7ff67c15bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.069+0000 7ff681a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff67c15b9f0 0x7ff67c15bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58254/0 (socket says 172.21.15.12:58254) 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff681a00700 1 -- 172.21.15.12:0/1358139830 learned_addr learned my addr 172.21.15.12:0/1358139830 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff67be00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff67c1f7310 0x7ff67c0b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff681000700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff664005c60 0x7ff67c1544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff681a00700 1 -- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff67c1f7310 msgr2=0x7ff67c0b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:15.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff681a00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff67c1f7310 0x7ff67c0b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:15.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff681a00700 1 -- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff664005c60 msgr2=0x7ff67c1544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:15.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff681a00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff664005c60 0x7ff67c1544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:15.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff681a00700 1 -- 172.21.15.12:0/1358139830 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c154bb0 con 0x7ff67c15b9f0 2024-03-29T10:09:15.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff681a00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff67c15b9f0 0x7ff67c15bea0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0d02f0 tx=0x7ff674004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:15.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff678c00700 1 -- 172.21.15.12:0/1358139830 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674007990 con 0x7ff67c15b9f0 2024-03-29T10:09:15.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff678c00700 1 -- 172.21.15.12:0/1358139830 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67400f050 con 0x7ff67c15b9f0 2024-03-29T10:09:15.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.070+0000 7ff683a00700 1 -- 172.21.15.12:0/1358139830 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67c0c9850 con 0x7ff67c15b9f0 2024-03-29T10:09:15.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.071+0000 7ff678c00700 1 -- 172.21.15.12:0/1358139830 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674007d30 con 0x7ff67c15b9f0 2024-03-29T10:09:15.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.071+0000 7ff683a00700 1 -- 172.21.15.12:0/1358139830 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67c0c9540 con 0x7ff67c15b9f0 2024-03-29T10:09:15.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.072+0000 7ff683a00700 1 -- 172.21.15.12:0/1358139830 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67c067720 con 0x7ff67c15b9f0 2024-03-29T10:09:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.072+0000 7ff678c00700 1 -- 172.21.15.12:0/1358139830 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 29) v1 ==== 89183+0+0 (secure 0 0 0) 0x7ff67401e460 con 0x7ff67c15b9f0 2024-03-29T10:09:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.073+0000 7ff678c00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff65c06ad20 0x7ff65c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.073+0000 7ff678c00700 1 -- 172.21.15.12:0/1358139830 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6017+0+0 (secure 0 0 0) 0x7ff674089830 con 0x7ff67c15b9f0 2024-03-29T10:09:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.074+0000 7ff681000700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff65c06ad20 0x7ff65c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:15.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.074+0000 7ff681000700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff65c06ad20 0x7ff65c06d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff670015040 tx=0x7ff670007000 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:15.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.078+0000 7ff678c00700 1 -- 172.21.15.12:0/1358139830 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6740546d0 con 0x7ff67c15b9f0 2024-03-29T10:09:15.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.288+0000 7ff678c00700 1 -- 172.21.15.12:0/1358139830 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff674054aa0 con 0x7ff67c15b9f0 2024-03-29T10:09:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:15.298+0000 7ff683a00700 1 -- 172.21.15.12:0/1358139830 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff67c0cef80 con 0x7ff67c15b9f0 2024-03-29T10:09:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.294+0000 7ff678c00700 1 -- 172.21.15.12:0/1358139830 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v71) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff6740867b0 con 0x7ff67c15b9f0 2024-03-29T10:09:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.299+0000 7ff683a00700 1 -- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff65c06ad20 msgr2=0x7ff65c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.299+0000 7ff683a00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff65c06ad20 0x7ff65c06d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff670015040 tx=0x7ff670007000 comp rx=0 tx=0).stop 2024-03-29T10:09:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.299+0000 7ff683a00700 1 -- 172.21.15.12:0/1358139830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff67c15b9f0 msgr2=0x7ff67c15bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.299+0000 7ff683a00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff67c15b9f0 0x7ff67c15bea0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0d02f0 tx=0x7ff674004690 comp rx=0 tx=0).stop 2024-03-29T10:09:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.299+0000 7ff683a00700 1 -- 172.21.15.12:0/1358139830 shutdown_connections 2024-03-29T10:09:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.299+0000 7ff683a00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff65c06ad20 0x7ff65c06d1e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.299+0000 7ff683a00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff67c15b9f0 0x7ff67c15bea0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.299+0000 7ff683a00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff664005c60 0x7ff67c1544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.299+0000 7ff683a00700 1 --2- 172.21.15.12:0/1358139830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff67c1f7310 0x7ff67c0b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.299+0000 7ff683a00700 1 -- 172.21.15.12:0/1358139830 >> 172.21.15.12:0/1358139830 conn(0x7ff67c0cf210 msgr2=0x7ff67c0ad750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:16.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.301+0000 7ff683a00700 1 -- 172.21.15.12:0/1358139830 shutdown_connections 2024-03-29T10:09:16.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.301+0000 7ff683a00700 1 -- 172.21.15.12:0/1358139830 wait complete. 2024-03-29T10:09:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T10:09:16.336 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T10:09:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.542+0000 7f9278c00700 1 Processor -- start 2024-03-29T10:09:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.543+0000 7f9278c00700 1 -- start start 2024-03-29T10:09:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.544+0000 7f9278c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f927415d9d0 0x7f92740d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.544+0000 7f9278c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9274161420 con 0x7f92740d16a0 2024-03-29T10:09:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.544+0000 7f9278c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9274161700 con 0x7f92740d2540 2024-03-29T10:09:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.544+0000 7f9278c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927415edc0 con 0x7f927415d9d0 2024-03-29T10:09:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.544+0000 7f9271400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f92740d16a0 0x7f92740d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40272/0 (socket says 172.21.15.12:40272) 2024-03-29T10:09:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.544+0000 7f9271400700 1 -- 172.21.15.12:0/457474384 learned_addr learned my addr 172.21.15.12:0/457474384 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:16.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.544+0000 7f9271e00700 1 --2- 172.21.15.12:0/457474384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f927415d9d0 0x7f92740d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:16.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.545+0000 7f9270a00700 1 -- 172.21.15.12:0/457474384 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794830886 0 0) 0x7f9274161420 con 0x7f92740d16a0 2024-03-29T10:09:16.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.545+0000 7f9270a00700 1 -- 172.21.15.12:0/457474384 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92580035b0 con 0x7f92740d16a0 2024-03-29T10:09:16.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.545+0000 7f9270a00700 1 -- 172.21.15.12:0/457474384 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044047789 0 0) 0x7f9274161700 con 0x7f92740d2540 2024-03-29T10:09:16.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.545+0000 7f9270a00700 1 -- 172.21.15.12:0/457474384 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9274161420 con 0x7f92740d2540 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.545+0000 7f9271e00700 1 -- 172.21.15.12:0/457474384 >> v1:172.21.15.159:6789/0 conn(0x7f92740d2540 legacy=0x7f92740ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.545+0000 7f9271e00700 1 -- 172.21.15.12:0/457474384 >> v1:172.21.15.12:6789/0 conn(0x7f92740d16a0 legacy=0x7f92740d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.545+0000 7f9270a00700 1 -- 172.21.15.12:0/457474384 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 734835298 0 0) 0x7f92580035b0 con 0x7f92740d16a0 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.545+0000 7f9271e00700 1 -- 172.21.15.12:0/457474384 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92741619e0 con 0x7f927415d9d0 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.545+0000 7f9271e00700 1 --2- 172.21.15.12:0/457474384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f927415d9d0 0x7f92740d3c30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f92640098a0 tx=0x7f9264009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66f49ba56e5c6031 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.546+0000 7f9270a00700 1 -- 172.21.15.12:0/457474384 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264011830 con 0x7f927415d9d0 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.546+0000 7f9270a00700 1 -- 172.21.15.12:0/457474384 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9264011990 con 0x7f927415d9d0 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.546+0000 7f9270a00700 1 -- 172.21.15.12:0/457474384 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264011d20 con 0x7f927415d9d0 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.546+0000 7f9278c00700 1 -- 172.21.15.12:0/457474384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f927415d9d0 msgr2=0x7f92740d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.546+0000 7f9278c00700 1 --2- 172.21.15.12:0/457474384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f927415d9d0 0x7f92740d3c30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f92640098a0 tx=0x7f9264009bb0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.546+0000 7f9278c00700 1 -- 172.21.15.12:0/457474384 shutdown_connections 2024-03-29T10:09:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.546+0000 7f9278c00700 1 --2- 172.21.15.12:0/457474384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f927415d9d0 0x7f92740d3c30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.546+0000 7f9278c00700 1 -- 172.21.15.12:0/457474384 >> 172.21.15.12:0/457474384 conn(0x7f927408aff0 msgr2=0x7f92740d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:16.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.546+0000 7f9278c00700 1 -- 172.21.15.12:0/457474384 shutdown_connections 2024-03-29T10:09:16.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.546+0000 7f9278c00700 1 -- 172.21.15.12:0/457474384 wait complete. 2024-03-29T10:09:16.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.548+0000 7f9278c00700 1 Processor -- start 2024-03-29T10:09:16.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.548+0000 7f9278c00700 1 -- start start 2024-03-29T10:09:16.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9278c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92740d16a0 0x7f92741601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9278c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92740d2540 0x7f92740cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9278c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f927415d9d0 0x7f927415c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9278c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92740cb9c0 con 0x7f92740d16a0 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9278c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92740cbca0 con 0x7f927415d9d0 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9278c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92740cbf80 con 0x7f92740d2540 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9271e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92740d16a0 0x7f92741601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9271e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92740d16a0 0x7f92741601d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58266/0 (socket says 172.21.15.12:58266) 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9271e00700 1 -- 172.21.15.12:0/3880682725 learned_addr learned my addr 172.21.15.12:0/3880682725 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9272800700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f927415d9d0 0x7f927415c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.549+0000 7f9271400700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92740d2540 0x7f92740cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f9271e00700 1 -- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92740d2540 msgr2=0x7f92740cafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:16.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f9271e00700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92740d2540 0x7f92740cafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f9271e00700 1 -- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f927415d9d0 msgr2=0x7f927415c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f9271e00700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f927415d9d0 0x7f927415c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f9271e00700 1 -- 172.21.15.12:0/3880682725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92640094a0 con 0x7f92740d16a0 2024-03-29T10:09:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f9271400700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92740d2540 0x7f92740cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f9271e00700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92740d16a0 0x7f92741601d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f92640090b0 tx=0x7f9264004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f926a000700 1 -- 172.21.15.12:0/3880682725 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926401a720 con 0x7f92740d16a0 2024-03-29T10:09:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f926a000700 1 -- 172.21.15.12:0/3880682725 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926401a880 con 0x7f92740d16a0 2024-03-29T10:09:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f9278c00700 1 -- 172.21.15.12:0/3880682725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92740cc260 con 0x7f92740d16a0 2024-03-29T10:09:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.550+0000 7f926a000700 1 -- 172.21.15.12:0/3880682725 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926401ac10 con 0x7f92740d16a0 2024-03-29T10:09:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.551+0000 7f9278c00700 1 -- 172.21.15.12:0/3880682725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9274160710 con 0x7f92740d16a0 2024-03-29T10:09:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.551+0000 7f9278c00700 1 -- 172.21.15.12:0/3880682725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92741f3810 con 0x7f92740d16a0 2024-03-29T10:09:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.552+0000 7f926a000700 1 -- 172.21.15.12:0/3880682725 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9264022070 con 0x7f92740d16a0 2024-03-29T10:09:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.554+0000 7f926a000700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f925406adb0 0x7f925406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.554+0000 7f926a000700 1 -- 172.21.15.12:0/3880682725 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f926408a1e0 con 0x7f92740d16a0 2024-03-29T10:09:16.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.554+0000 7f9271400700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f925406adb0 0x7f925406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:16.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.555+0000 7f9271400700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f925406adb0 0x7f925406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9260007fd0 tx=0x7f9260009040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:16.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.558+0000 7f926a000700 1 -- 172.21.15.12:0/3880682725 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92640553e0 con 0x7f92740d16a0 2024-03-29T10:09:16.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.772+0000 7f9278c00700 1 -- 172.21.15.12:0/3880682725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f92740d1ca0 con 0x7f92740d16a0 2024-03-29T10:09:16.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.776+0000 7f926a000700 1 -- 172.21.15.12:0/3880682725 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f9264058a00 con 0x7f92740d16a0 2024-03-29T10:09:16.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 -- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f925406adb0 msgr2=0x7f925406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:16.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f925406adb0 0x7f925406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9260007fd0 tx=0x7f9260009040 comp rx=0 tx=0).stop 2024-03-29T10:09:16.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 -- 172.21.15.12:0/3880682725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92740d16a0 msgr2=0x7f92741601d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:16.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92740d16a0 0x7f92741601d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f92640090b0 tx=0x7f9264004750 comp rx=0 tx=0).stop 2024-03-29T10:09:16.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 -- 172.21.15.12:0/3880682725 shutdown_connections 2024-03-29T10:09:16.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f925406adb0 0x7f925406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f927415d9d0 0x7f927415c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92740d16a0 0x7f92741601d0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 --2- 172.21.15.12:0/3880682725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92740d2540 0x7f92740cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 -- 172.21.15.12:0/3880682725 >> 172.21.15.12:0/3880682725 conn(0x7f927408aff0 msgr2=0x7f92740b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:16.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 -- 172.21.15.12:0/3880682725 shutdown_connections 2024-03-29T10:09:16.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.779+0000 7f9278c00700 1 -- 172.21.15.12:0/3880682725 wait complete. 2024-03-29T10:09:16.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T10:09:16.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.995+0000 7f182be00700 1 Processor -- start 2024-03-29T10:09:16.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.996+0000 7f182be00700 1 -- start start 2024-03-29T10:09:16.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.997+0000 7f182be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18240d40f0 0x7f18240b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:16.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.997+0000 7f182be00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f182415dab0 0x7f18240b0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:16.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.997+0000 7f182be00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824162100 con 0x7f182415d6d0 2024-03-29T10:09:16.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.997+0000 7f182be00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18240b2d20 con 0x7f182415dab0 2024-03-29T10:09:16.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.997+0000 7f182be00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18241623e0 con 0x7f18240d40f0 2024-03-29T10:09:16.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.997+0000 7f1829e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f182415dab0 0x7f18240b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:16.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.997+0000 7f1828a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18240d40f0 0x7f18240b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:16.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.997+0000 7f1829e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f182415dab0 0x7f18240b0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:34676/0 (socket says 172.21.15.12:34676) 2024-03-29T10:09:16.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.997+0000 7f1829e00700 1 -- 172.21.15.12:0/3963649503 learned_addr learned my addr 172.21.15.12:0/3963649503 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:16.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.998+0000 7f1829e00700 1 -- 172.21.15.12:0/3963649503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18240d40f0 msgr2=0x7f18240b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:16.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.998+0000 7f1829e00700 1 --2- 172.21.15.12:0/3963649503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18240d40f0 0x7f18240b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:16.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.998+0000 7f1829e00700 1 -- 172.21.15.12:0/3963649503 >> v1:172.21.15.159:6789/0 conn(0x7f182415d6d0 legacy=0x7f1824161430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:16.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.998+0000 7f1829e00700 1 -- 172.21.15.12:0/3963649503 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240ce960 con 0x7f182415dab0 2024-03-29T10:09:17.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.998+0000 7f1828a00700 1 --2- 172.21.15.12:0/3963649503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18240d40f0 0x7f18240b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:17.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.998+0000 7f1829e00700 1 --2- 172.21.15.12:0/3963649503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f182415dab0 0x7f18240b0660 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f181c009e50 tx=0x7f181c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=977ba5ac8f87904a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:17.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.998+0000 7f1823e00700 1 -- 172.21.15.12:0/3963649503 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c00e070 con 0x7f182415dab0 2024-03-29T10:09:17.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.998+0000 7f1823e00700 1 -- 172.21.15.12:0/3963649503 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c00b910 con 0x7f182415dab0 2024-03-29T10:09:17.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.999+0000 7f1823e00700 1 -- 172.21.15.12:0/3963649503 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c00bcc0 con 0x7f182415dab0 2024-03-29T10:09:17.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.999+0000 7f182be00700 1 -- 172.21.15.12:0/3963649503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f182415dab0 msgr2=0x7f18240b0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.999+0000 7f182be00700 1 --2- 172.21.15.12:0/3963649503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f182415dab0 0x7f18240b0660 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f181c009e50 tx=0x7f181c00b330 comp rx=0 tx=0).stop 2024-03-29T10:09:17.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.999+0000 7f182be00700 1 -- 172.21.15.12:0/3963649503 shutdown_connections 2024-03-29T10:09:17.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.999+0000 7f182be00700 1 --2- 172.21.15.12:0/3963649503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f182415dab0 0x7f18240b0660 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.999+0000 7f182be00700 1 --2- 172.21.15.12:0/3963649503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18240d40f0 0x7f18240b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.999+0000 7f182be00700 1 -- 172.21.15.12:0/3963649503 >> 172.21.15.12:0/3963649503 conn(0x7f182408aff0 msgr2=0x7f18240d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:17.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.999+0000 7f182be00700 1 -- 172.21.15.12:0/3963649503 shutdown_connections 2024-03-29T10:09:17.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:16.999+0000 7f182be00700 1 -- 172.21.15.12:0/3963649503 wait complete. 2024-03-29T10:09:17.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.001+0000 7f182be00700 1 Processor -- start 2024-03-29T10:09:17.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.001+0000 7f182be00700 1 -- start start 2024-03-29T10:09:17.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f182be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18240d40f0 0x7f18240ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f182be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f182415d6d0 0x7f18240b1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f182be00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f182415dab0 0x7f18240b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f182be00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240cb000 con 0x7f18240d40f0 2024-03-29T10:09:17.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f182be00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240cb2e0 con 0x7f182415dab0 2024-03-29T10:09:17.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f182be00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18240cb5c0 con 0x7f182415d6d0 2024-03-29T10:09:17.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f1829400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18240d40f0 0x7f18240ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f1829400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18240d40f0 0x7f18240ca620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58268/0 (socket says 172.21.15.12:58268) 2024-03-29T10:09:17.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f1829400700 1 -- 172.21.15.12:0/1059270076 learned_addr learned my addr 172.21.15.12:0/1059270076 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:17.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f1829e00700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f182415dab0 0x7f18240b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.002+0000 7f1828a00700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f182415d6d0 0x7f18240b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f1829400700 1 -- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f182415d6d0 msgr2=0x7f18240b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f1829400700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f182415d6d0 0x7f18240b1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f1829400700 1 -- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f182415dab0 msgr2=0x7f18240b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f1829400700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f182415dab0 0x7f18240b1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f1829400700 1 -- 172.21.15.12:0/1059270076 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18100094a0 con 0x7f18240d40f0 2024-03-29T10:09:17.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f1829400700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18240d40f0 0x7f18240ca620 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f181800c4e0 tx=0x7f181800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:17.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f1821600700 1 -- 172.21.15.12:0/1059270076 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18180052d0 con 0x7f18240d40f0 2024-03-29T10:09:17.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f1821600700 1 -- 172.21.15.12:0/1059270076 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1818005430 con 0x7f18240d40f0 2024-03-29T10:09:17.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f1821600700 1 -- 172.21.15.12:0/1059270076 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18180038c0 con 0x7f18240d40f0 2024-03-29T10:09:17.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f182be00700 1 -- 172.21.15.12:0/1059270076 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c009b30 con 0x7f18240d40f0 2024-03-29T10:09:17.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.003+0000 7f182be00700 1 -- 172.21.15.12:0/1059270076 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18240cbb80 con 0x7f18240d40f0 2024-03-29T10:09:17.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.004+0000 7f182be00700 1 -- 172.21.15.12:0/1059270076 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18240cbda0 con 0x7f18240d40f0 2024-03-29T10:09:17.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.005+0000 7f1821600700 1 -- 172.21.15.12:0/1059270076 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1818003a20 con 0x7f18240d40f0 2024-03-29T10:09:17.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.006+0000 7f1821600700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f180406adb0 0x7f180406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.006+0000 7f1821600700 1 -- 172.21.15.12:0/1059270076 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f1818088790 con 0x7f18240d40f0 2024-03-29T10:09:17.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.007+0000 7f1828a00700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f180406adb0 0x7f180406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.008+0000 7f1828a00700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f180406adb0 0x7f180406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f18241f6360 tx=0x7f1810009210 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:17.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.011+0000 7f1821600700 1 -- 172.21.15.12:0/1059270076 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1818053900 con 0x7f18240d40f0 2024-03-29T10:09:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.224+0000 7f182be00700 1 -- 172.21.15.12:0/1059270076 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f182406d6a0 con 0x7f18240d40f0 2024-03-29T10:09:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.231+0000 7f1821600700 1 -- 172.21.15.12:0/1059270076 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1818056f20 con 0x7f18240d40f0 2024-03-29T10:09:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 -- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f180406adb0 msgr2=0x7f180406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f180406adb0 0x7f180406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f18241f6360 tx=0x7f1810009210 comp rx=0 tx=0).stop 2024-03-29T10:09:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 -- 172.21.15.12:0/1059270076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18240d40f0 msgr2=0x7f18240ca620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18240d40f0 0x7f18240ca620 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f181800c4e0 tx=0x7f181800c7f0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 -- 172.21.15.12:0/1059270076 shutdown_connections 2024-03-29T10:09:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f180406adb0 0x7f180406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f182415dab0 0x7f18240b1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18240d40f0 0x7f18240ca620 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 --2- 172.21.15.12:0/1059270076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f182415d6d0 0x7f18240b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 -- 172.21.15.12:0/1059270076 >> 172.21.15.12:0/1059270076 conn(0x7f182408aff0 msgr2=0x7f18240cd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 -- 172.21.15.12:0/1059270076 shutdown_connections 2024-03-29T10:09:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.233+0000 7f182be00700 1 -- 172.21.15.12:0/1059270076 wait complete. 2024-03-29T10:09:17.251 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T10:09:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.452+0000 7f31b9000700 1 Processor -- start 2024-03-29T10:09:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.452+0000 7f31b9000700 1 -- start start 2024-03-29T10:09:17.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.453+0000 7f31b9000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b40d16a0 0x7f31b40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.453+0000 7f31b9000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f31b40d2370 0x7f31b40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.453+0000 7f31b9000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b4161420 con 0x7f31b415d9d0 2024-03-29T10:09:17.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.453+0000 7f31b9000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b40d2820 con 0x7f31b40d2370 2024-03-29T10:09:17.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.453+0000 7f31b9000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b40d4340 con 0x7f31b40d16a0 2024-03-29T10:09:17.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.453+0000 7f31b2800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f31b40d2370 0x7f31b40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.453+0000 7f31b1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b40d16a0 0x7f31b40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.453+0000 7f31b2800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f31b40d2370 0x7f31b40ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:34704/0 (socket says 172.21.15.12:34704) 2024-03-29T10:09:17.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.453+0000 7f31b2800700 1 -- 172.21.15.12:0/904984151 learned_addr learned my addr 172.21.15.12:0/904984151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:17.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.453+0000 7f31b1e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f31b415d9d0 0x7f31b40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:58966/0 (socket says 172.21.15.12:58966) 2024-03-29T10:09:17.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.454+0000 7f31b2800700 1 -- 172.21.15.12:0/904984151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b40d16a0 msgr2=0x7f31b40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:17.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.454+0000 7f31b2800700 1 --2- 172.21.15.12:0/904984151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b40d16a0 0x7f31b40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.454+0000 7f31b2800700 1 -- 172.21.15.12:0/904984151 >> v1:172.21.15.159:6789/0 conn(0x7f31b415d9d0 legacy=0x7f31b40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.454+0000 7f31b2800700 1 -- 172.21.15.12:0/904984151 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b4161700 con 0x7f31b40d2370 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.454+0000 7f31b1400700 1 --2- 172.21.15.12:0/904984151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b40d16a0 0x7f31b40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.454+0000 7f31b2800700 1 --2- 172.21.15.12:0/904984151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31b40d2370 0x7f31b40ce360 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f31ac009e80 tx=0x7f31ac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2086f55e7692abd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.454+0000 7f31b0a00700 1 -- 172.21.15.12:0/904984151 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac00e070 con 0x7f31b40d2370 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.455+0000 7f31b0a00700 1 -- 172.21.15.12:0/904984151 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ac00b8d0 con 0x7f31b40d2370 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.455+0000 7f31b0a00700 1 -- 172.21.15.12:0/904984151 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac00bc80 con 0x7f31b40d2370 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.455+0000 7f31b9000700 1 -- 172.21.15.12:0/904984151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31b40d2370 msgr2=0x7f31b40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.455+0000 7f31b9000700 1 --2- 172.21.15.12:0/904984151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31b40d2370 0x7f31b40ce360 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f31ac009e80 tx=0x7f31ac00b330 comp rx=0 tx=0).stop 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.455+0000 7f31b9000700 1 -- 172.21.15.12:0/904984151 shutdown_connections 2024-03-29T10:09:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.455+0000 7f31b9000700 1 --2- 172.21.15.12:0/904984151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31b40d2370 0x7f31b40ce360 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.455+0000 7f31b9000700 1 --2- 172.21.15.12:0/904984151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b40d16a0 0x7f31b40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.455+0000 7f31b9000700 1 -- 172.21.15.12:0/904984151 >> 172.21.15.12:0/904984151 conn(0x7f31b408aff0 msgr2=0x7f31b40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.455+0000 7f31b9000700 1 -- 172.21.15.12:0/904984151 shutdown_connections 2024-03-29T10:09:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.455+0000 7f31b9000700 1 -- 172.21.15.12:0/904984151 wait complete. 2024-03-29T10:09:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.457+0000 7f31b9000700 1 Processor -- start 2024-03-29T10:09:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.457+0000 7f31b9000700 1 -- start start 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.458+0000 7f31b9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31b40d16a0 0x7f31b40cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.458+0000 7f31b9000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31b40d2370 0x7f31b4159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.458+0000 7f31b9000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b415d9d0 0x7f31b40cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.458+0000 7f31b9000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b40b6020 con 0x7f31b40d16a0 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.458+0000 7f31b9000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b415a440 con 0x7f31b40d2370 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.458+0000 7f31b9000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b40cb2e0 con 0x7f31b415d9d0 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.458+0000 7f31b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31b40d16a0 0x7f31b40cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.458+0000 7f31b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31b40d16a0 0x7f31b40cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58278/0 (socket says 172.21.15.12:58278) 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.458+0000 7f31b1e00700 1 -- 172.21.15.12:0/1508381843 learned_addr learned my addr 172.21.15.12:0/1508381843 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31b1400700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31b40d2370 0x7f31b4159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31b2800700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b415d9d0 0x7f31b40cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31b1e00700 1 -- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b415d9d0 msgr2=0x7f31b40cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31b1e00700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b415d9d0 0x7f31b40cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31b1e00700 1 -- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31b40d2370 msgr2=0x7f31b4159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31b1e00700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31b40d2370 0x7f31b4159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31b1e00700 1 -- 172.21.15.12:0/1508381843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a0009430 con 0x7f31b40d16a0 2024-03-29T10:09:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31b1e00700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31b40d16a0 0x7f31b40cbdb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f31a400c500 tx=0x7f31a400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31aa000700 1 -- 172.21.15.12:0/1508381843 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4005cb0 con 0x7f31b40d16a0 2024-03-29T10:09:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31aa000700 1 -- 172.21.15.12:0/1508381843 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a4005e10 con 0x7f31b40d16a0 2024-03-29T10:09:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31b9000700 1 -- 172.21.15.12:0/1508381843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ac009b30 con 0x7f31b40d16a0 2024-03-29T10:09:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.459+0000 7f31aa000700 1 -- 172.21.15.12:0/1508381843 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a40072d0 con 0x7f31b40d16a0 2024-03-29T10:09:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.460+0000 7f31b9000700 1 -- 172.21.15.12:0/1508381843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b40cb8a0 con 0x7f31b40d16a0 2024-03-29T10:09:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.461+0000 7f31b9000700 1 -- 172.21.15.12:0/1508381843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b40d1ca0 con 0x7f31b40d16a0 2024-03-29T10:09:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.461+0000 7f31aa000700 1 -- 172.21.15.12:0/1508381843 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f31a4007430 con 0x7f31b40d16a0 2024-03-29T10:09:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.463+0000 7f31aa000700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f319406ade0 0x7f319406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.463+0000 7f31aa000700 1 -- 172.21.15.12:0/1508381843 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f31a40886d0 con 0x7f31b40d16a0 2024-03-29T10:09:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.463+0000 7f31b1400700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f319406ade0 0x7f319406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.464+0000 7f31b1400700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f319406ade0 0x7f319406d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f31b40b8300 tx=0x7f31a00091c0 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.467+0000 7f31aa000700 1 -- 172.21.15.12:0/1508381843 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31a40537d0 con 0x7f31b40d16a0 2024-03-29T10:09:17.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.681+0000 7f31b9000700 1 -- 172.21.15.12:0/1508381843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f31b4067720 con 0x7f31b40d16a0 2024-03-29T10:09:17.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.687+0000 7f31aa000700 1 -- 172.21.15.12:0/1508381843 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v21) v1 ==== 120+0+0 (secure 0 0 0) 0x7f31a4056df0 con 0x7f31b40d16a0 2024-03-29T10:09:17.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 -- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f319406ade0 msgr2=0x7f319406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f319406ade0 0x7f319406d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f31b40b8300 tx=0x7f31a00091c0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 -- 172.21.15.12:0/1508381843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31b40d16a0 msgr2=0x7f31b40cbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31b40d16a0 0x7f31b40cbdb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f31a400c500 tx=0x7f31a400c810 comp rx=0 tx=0).stop 2024-03-29T10:09:17.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 -- 172.21.15.12:0/1508381843 shutdown_connections 2024-03-29T10:09:17.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f319406ade0 0x7f319406d2a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31b415d9d0 0x7f31b40cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31b40d16a0 0x7f31b40cbdb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 --2- 172.21.15.12:0/1508381843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31b40d2370 0x7f31b4159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 -- 172.21.15.12:0/1508381843 >> 172.21.15.12:0/1508381843 conn(0x7f31b408aff0 msgr2=0x7f31b40b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 -- 172.21.15.12:0/1508381843 shutdown_connections 2024-03-29T10:09:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.690+0000 7f31b9000700 1 -- 172.21.15.12:0/1508381843 wait complete. 2024-03-29T10:09:17.707 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-03-29T10:09:17.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.906+0000 7fb0ed800700 1 Processor -- start 2024-03-29T10:09:17.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.906+0000 7fb0ed800700 1 -- start start 2024-03-29T10:09:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.907+0000 7fb0ed800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb0e80cb3f0 0x7fb0e80ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.907+0000 7fb0ed800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb0e80ac600 0x7fb0e806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.907+0000 7fb0ed800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e815fd50 con 0x7fb0e815b5d0 2024-03-29T10:09:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.907+0000 7fb0ed800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e815b9b0 con 0x7fb0e80cb3f0 2024-03-29T10:09:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.907+0000 7fb0ed800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e80aca10 con 0x7fb0e80ac600 2024-03-29T10:09:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.907+0000 7fb0e6a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb0e815b5d0 0x7fb0e80ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40276/0 (socket says 172.21.15.12:40276) 2024-03-29T10:09:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.907+0000 7fb0e6a00700 1 -- 172.21.15.12:0/2256223866 learned_addr learned my addr 172.21.15.12:0/2256223866 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.907+0000 7fb0e6000700 1 --2- 172.21.15.12:0/2256223866 >> v2:172.21.15.12:3300/0 conn(0x7fb0e80cb3f0 0x7fb0e80ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.908+0000 7fb0e7400700 1 --2- 172.21.15.12:0/2256223866 >> v2:172.21.15.159:3300/0 conn(0x7fb0e80ac600 0x7fb0e806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.908+0000 7fb0e6000700 1 -- 172.21.15.12:0/2256223866 >> v2:172.21.15.159:3300/0 conn(0x7fb0e80ac600 msgr2=0x7fb0e806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:17.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.908+0000 7fb0e6000700 1 --2- 172.21.15.12:0/2256223866 >> v2:172.21.15.159:3300/0 conn(0x7fb0e80ac600 0x7fb0e806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.908+0000 7fb0e6000700 1 -- 172.21.15.12:0/2256223866 >> v1:172.21.15.12:6789/0 conn(0x7fb0e815b5d0 legacy=0x7fb0e80ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.908+0000 7fb0e6000700 1 -- 172.21.15.12:0/2256223866 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e815f140 con 0x7fb0e80cb3f0 2024-03-29T10:09:17.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.908+0000 7fb0e5600700 1 -- 172.21.15.12:0/2256223866 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057756318 0 0) 0x7fb0e815fd50 con 0x7fb0e815b5d0 2024-03-29T10:09:17.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.908+0000 7fb0e6000700 1 --2- 172.21.15.12:0/2256223866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80cb3f0 0x7fb0e80ac0c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80cfdc0 tx=0x7fb0d4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249a294077ac96fd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:17.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.908+0000 7fb0e5600700 1 -- 172.21.15.12:0/2256223866 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4011830 con 0x7fb0e80cb3f0 2024-03-29T10:09:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.908+0000 7fb0e5600700 1 -- 172.21.15.12:0/2256223866 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d4011990 con 0x7fb0e80cb3f0 2024-03-29T10:09:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.908+0000 7fb0e5600700 1 -- 172.21.15.12:0/2256223866 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4011d20 con 0x7fb0e80cb3f0 2024-03-29T10:09:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.909+0000 7fb0ed800700 1 -- 172.21.15.12:0/2256223866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80cb3f0 msgr2=0x7fb0e80ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.909+0000 7fb0ed800700 1 --2- 172.21.15.12:0/2256223866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80cb3f0 0x7fb0e80ac0c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80cfdc0 tx=0x7fb0d4009b20 comp rx=0 tx=0).stop 2024-03-29T10:09:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.909+0000 7fb0ed800700 1 -- 172.21.15.12:0/2256223866 shutdown_connections 2024-03-29T10:09:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.909+0000 7fb0ed800700 1 --2- 172.21.15.12:0/2256223866 >> v2:172.21.15.159:3300/0 conn(0x7fb0e80ac600 0x7fb0e806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.909+0000 7fb0ed800700 1 --2- 172.21.15.12:0/2256223866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80cb3f0 0x7fb0e80ac0c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.909+0000 7fb0ed800700 1 -- 172.21.15.12:0/2256223866 >> 172.21.15.12:0/2256223866 conn(0x7fb0e80cf190 msgr2=0x7fb0e80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.909+0000 7fb0ed800700 1 -- 172.21.15.12:0/2256223866 shutdown_connections 2024-03-29T10:09:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.909+0000 7fb0ed800700 1 -- 172.21.15.12:0/2256223866 wait complete. 2024-03-29T10:09:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.911+0000 7fb0ed800700 1 Processor -- start 2024-03-29T10:09:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.911+0000 7fb0ed800700 1 -- start start 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0ed800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80ac600 0x7fb0e8071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0ed800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0e815b5d0 0x7fb0e8071f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0ed800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0e81f6260 0x7fb0e81f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0ed800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8145290 con 0x7fb0e80ac600 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0ed800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80cd660 con 0x7fb0e81f6260 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0ed800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80cd940 con 0x7fb0e815b5d0 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0e6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80ac600 0x7fb0e8071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0e6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80ac600 0x7fb0e8071a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58294/0 (socket says 172.21.15.12:58294) 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0e6a00700 1 -- 172.21.15.12:0/2500239910 learned_addr learned my addr 172.21.15.12:0/2500239910 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0e6000700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0e815b5d0 0x7fb0e8071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.912+0000 7fb0e7400700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0e81f6260 0x7fb0e81f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0e6a00700 1 -- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0e815b5d0 msgr2=0x7fb0e8071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0e6a00700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0e815b5d0 0x7fb0e8071f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0e6a00700 1 -- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0e81f6260 msgr2=0x7fb0e81f6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0e6a00700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0e81f6260 0x7fb0e81f6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0e6a00700 1 -- 172.21.15.12:0/2500239910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d40094a0 con 0x7fb0e80ac600 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0e6a00700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80ac600 0x7fb0e8071a10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80011e0 tx=0x7fb0d800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0dea00700 1 -- 172.21.15.12:0/2500239910 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8004c80 con 0x7fb0e80ac600 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0dea00700 1 -- 172.21.15.12:0/2500239910 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d8004e00 con 0x7fb0e80ac600 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0ed800700 1 -- 172.21.15.12:0/2500239910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e8145550 con 0x7fb0e80ac600 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0dea00700 1 -- 172.21.15.12:0/2500239910 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8007310 con 0x7fb0e80ac600 2024-03-29T10:09:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.913+0000 7fb0ed800700 1 -- 172.21.15.12:0/2500239910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e81f6c50 con 0x7fb0e80ac600 2024-03-29T10:09:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.914+0000 7fb0ed800700 1 -- 172.21.15.12:0/2500239910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e8067720 con 0x7fb0e80ac600 2024-03-29T10:09:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.915+0000 7fb0dea00700 1 -- 172.21.15.12:0/2500239910 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb0d8007470 con 0x7fb0e80ac600 2024-03-29T10:09:17.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.916+0000 7fb0dea00700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb0c806adb0 0x7fb0c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:17.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.917+0000 7fb0dea00700 1 -- 172.21.15.12:0/2500239910 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5545+0+0 (secure 0 0 0) 0x7fb0d80890f0 con 0x7fb0e80ac600 2024-03-29T10:09:17.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.917+0000 7fb0e6000700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb0c806adb0 0x7fb0c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:17.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.918+0000 7fb0e6000700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb0c806adb0 0x7fb0c806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40090b0 tx=0x7fb0d4008030 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:17.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:17.921+0000 7fb0dea00700 1 -- 172.21.15.12:0/2500239910 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0d80542f0 con 0x7fb0e80ac600 2024-03-29T10:09:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.219+0000 7fb0ed800700 1 -- 172.21.15.12:0/2500239910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fb0e80cdb00 con 0x7fb0e80ac600 2024-03-29T10:09:18.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.300+0000 7fb0dea00700 1 -- 172.21.15.12:0/2500239910 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v534) v1 ==== 154+0+0 (secure 0 0 0) 0x7fb0d8057910 con 0x7fb0e80ac600 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.302+0000 7fb0ed800700 1 -- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb0c806adb0 msgr2=0x7fb0c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.302+0000 7fb0ed800700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb0c806adb0 0x7fb0c806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40090b0 tx=0x7fb0d4008030 comp rx=0 tx=0).stop 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.302+0000 7fb0ed800700 1 -- 172.21.15.12:0/2500239910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80ac600 msgr2=0x7fb0e8071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.302+0000 7fb0ed800700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80ac600 0x7fb0e8071a10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80011e0 tx=0x7fb0d800cae0 comp rx=0 tx=0).stop 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.303+0000 7fb0ed800700 1 -- 172.21.15.12:0/2500239910 shutdown_connections 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.303+0000 7fb0ed800700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb0c806adb0 0x7fb0c806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.303+0000 7fb0ed800700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0e81f6260 0x7fb0e81f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.303+0000 7fb0ed800700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e80ac600 0x7fb0e8071a10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.303+0000 7fb0ed800700 1 --2- 172.21.15.12:0/2500239910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0e815b5d0 0x7fb0e8071f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.303+0000 7fb0ed800700 1 -- 172.21.15.12:0/2500239910 >> 172.21.15.12:0/2500239910 conn(0x7fb0e80cf190 msgr2=0x7fb0e815c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.303+0000 7fb0ed800700 1 -- 172.21.15.12:0/2500239910 shutdown_connections 2024-03-29T10:09:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.303+0000 7fb0ed800700 1 -- 172.21.15.12:0/2500239910 wait complete. 2024-03-29T10:09:18.320 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2024-03-29T10:09:18.320 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2024-03-29T10:09:18.320 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T10:09:18.321 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-03-29T10:09:18.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.519+0000 7f6140400700 1 Processor -- start 2024-03-29T10:09:18.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.519+0000 7f6140400700 1 -- start start 2024-03-29T10:09:18.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.521+0000 7f6140400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6138161ca0 0x7f61380cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:18.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.521+0000 7f6140400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61380ceda0 con 0x7f6138160f10 2024-03-29T10:09:18.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.521+0000 7f6140400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61381612f0 con 0x7f61380d18b0 2024-03-29T10:09:18.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.521+0000 7f6140400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61380d1d30 con 0x7f6138161ca0 2024-03-29T10:09:18.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.521+0000 7f613da00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6138160f10 0x7f61380d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40284/0 (socket says 172.21.15.12:40284) 2024-03-29T10:09:18.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.521+0000 7f613da00700 1 -- 172.21.15.12:0/2699910209 learned_addr learned my addr 172.21.15.12:0/2699910209 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:18.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.521+0000 7f613e400700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6138161ca0 0x7f61380cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:18.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.521+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745863624 0 0) 0x7f61380ceda0 con 0x7f6138160f10 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.521+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61200035b0 con 0x7f6138160f10 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3500565145 0 0) 0x7f61200035b0 con 0x7f6138160f10 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6138161ca0 msgr2=0x7f61380cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6138161ca0 0x7f61380cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 >> v1:172.21.15.159:6789/0 conn(0x7f61380d18b0 legacy=0x7f6138161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61380ce470 con 0x7f6138160f10 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f612c001670 con 0x7f6138160f10 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 >> v1:172.21.15.12:6789/0 conn(0x7f6138160f10 legacy=0x7f61380d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6138161ca0 0x7f6120003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6120003fb0 0x7f6120004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6120004d90 0x7f6120005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61380d1d30 con 0x7f6120003fb0 2024-03-29T10:09:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612000bba0 con 0x7f6120004d90 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612000bd60 con 0x7f6138161ca0 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f613d000700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6120003fb0 0x7f6120004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.522+0000 7f613e400700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6120004d90 0x7f6120005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f613d000700 1 -- 172.21.15.12:0/2699910209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6138161ca0 msgr2=0x7f6120003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f613d000700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6138161ca0 0x7f6120003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f613d000700 1 -- 172.21.15.12:0/2699910209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6120004d90 msgr2=0x7f6120005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f613d000700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6120004d90 0x7f6120005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f613d000700 1 -- 172.21.15.12:0/2699910209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61200035b0 con 0x7f6120003fb0 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f613d000700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6120003fb0 0x7f6120004850 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6124001800 tx=0x7f6120005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9649d2d6b83d5cda server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61240012f0 con 0x7f6120003fb0 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61240052f0 con 0x7f6120003fb0 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f6137e00700 1 -- 172.21.15.12:0/2699910209 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124006860 con 0x7f6120003fb0 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f6140400700 1 -- 172.21.15.12:0/2699910209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6120003fb0 msgr2=0x7f6120004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f6140400700 1 --2- 172.21.15.12:0/2699910209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6120003fb0 0x7f6120004850 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6124001800 tx=0x7f6120005880 comp rx=0 tx=0).stop 2024-03-29T10:09:18.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.523+0000 7f613e400700 1 -- 172.21.15.12:0/2699910209 reap_dead start 2024-03-29T10:09:18.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.524+0000 7f6140400700 1 -- 172.21.15.12:0/2699910209 shutdown_connections 2024-03-29T10:09:18.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.524+0000 7f6140400700 1 -- 172.21.15.12:0/2699910209 >> 172.21.15.12:0/2699910209 conn(0x7f613808aff0 msgr2=0x7f61380d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:18.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.524+0000 7f6140400700 1 -- 172.21.15.12:0/2699910209 shutdown_connections 2024-03-29T10:09:18.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.524+0000 7f6140400700 1 -- 172.21.15.12:0/2699910209 wait complete. 2024-03-29T10:09:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.525+0000 7f6140400700 1 Processor -- start 2024-03-29T10:09:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.526+0000 7f6140400700 1 -- start start 2024-03-29T10:09:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.526+0000 7f6140400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6120003fb0 0x7f61380b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f6140400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613814e940 0x7f61380b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f6140400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f613815ef20 0x7f613815f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f6140400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61380b7220 con 0x7f613815ef20 2024-03-29T10:09:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f6140400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613815f8e0 con 0x7f613814e940 2024-03-29T10:09:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f6140400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613815e3a0 con 0x7f6120003fb0 2024-03-29T10:09:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f613815ef20 0x7f613815f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f613815ef20 0x7f613815f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58306/0 (socket says 172.21.15.12:58306) 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613e400700 1 -- 172.21.15.12:0/1842313033 learned_addr learned my addr 172.21.15.12:0/1842313033 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613da00700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6120003fb0 0x7f61380b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613d000700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613814e940 0x7f61380b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613e400700 1 -- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6120003fb0 msgr2=0x7f61380b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613e400700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6120003fb0 0x7f61380b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613e400700 1 -- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613814e940 msgr2=0x7f61380b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613e400700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613814e940 0x7f61380b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613e400700 1 -- 172.21.15.12:0/1842313033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613815e680 con 0x7f613815ef20 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.527+0000 7f613e400700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f613815ef20 0x7f613815f3a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f613808c570 tx=0x7f6130011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.528+0000 7f6135600700 1 -- 172.21.15.12:0/1842313033 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613000e040 con 0x7f613815ef20 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.528+0000 7f6135600700 1 -- 172.21.15.12:0/1842313033 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613000f040 con 0x7f613815ef20 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.528+0000 7f6135600700 1 -- 172.21.15.12:0/1842313033 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613000fd60 con 0x7f613815ef20 2024-03-29T10:09:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.528+0000 7f6140400700 1 -- 172.21.15.12:0/1842313033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61380b6b60 con 0x7f613815ef20 2024-03-29T10:09:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.528+0000 7f6140400700 1 -- 172.21.15.12:0/1842313033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613814edc0 con 0x7f613815ef20 2024-03-29T10:09:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.529+0000 7f6140400700 1 -- 172.21.15.12:0/1842313033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61380b7360 con 0x7f613815ef20 2024-03-29T10:09:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.530+0000 7f6135600700 1 -- 172.21.15.12:0/1842313033 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6130022460 con 0x7f613815ef20 2024-03-29T10:09:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.531+0000 7f6135600700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f611806adb0 0x7f611806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:18.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.531+0000 7f6135600700 1 -- 172.21.15.12:0/1842313033 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f6130089a90 con 0x7f613815ef20 2024-03-29T10:09:18.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.531+0000 7f613da00700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f611806adb0 0x7f611806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:18.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.532+0000 7f613da00700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f611806adb0 0x7f611806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f612c001eb0 tx=0x7f612c001a00 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:18.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.535+0000 7f6135600700 1 -- 172.21.15.12:0/1842313033 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f61380b7360 con 0x7f613815ef20 2024-03-29T10:09:18.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:18.835+0000 7f6140400700 1 -- 172.21.15.12:0/1842313033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f6138067720 con 0x7f613815ef20 2024-03-29T10:09:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.304+0000 7f6135600700 1 -- 172.21.15.12:0/1842313033 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v535) v1 ==== 159+0+0 (secure 0 0 0) 0x7f6130054a50 con 0x7f613815ef20 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 -- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f611806adb0 msgr2=0x7f611806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f611806adb0 0x7f611806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f612c001eb0 tx=0x7f612c001a00 comp rx=0 tx=0).stop 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 -- 172.21.15.12:0/1842313033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f613815ef20 msgr2=0x7f613815f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f613815ef20 0x7f613815f3a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f613808c570 tx=0x7f6130011eb0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 -- 172.21.15.12:0/1842313033 shutdown_connections 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f611806adb0 0x7f611806d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f613815ef20 0x7f613815f3a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6120003fb0 0x7f61380b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 --2- 172.21.15.12:0/1842313033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613814e940 0x7f61380b8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 -- 172.21.15.12:0/1842313033 >> 172.21.15.12:0/1842313033 conn(0x7f613808aff0 msgr2=0x7f61380d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 -- 172.21.15.12:0/1842313033 shutdown_connections 2024-03-29T10:09:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.307+0000 7f6140400700 1 -- 172.21.15.12:0/1842313033 wait complete. 2024-03-29T10:09:19.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T10:09:19.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.529+0000 7facdd200700 1 Processor -- start 2024-03-29T10:09:19.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.530+0000 7facdd200700 1 -- start start 2024-03-29T10:09:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.531+0000 7facdd200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 0x7facd80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.531+0000 7facdd200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7facd80d16a0 0x7facd80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.531+0000 7facdd200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd8161420 con 0x7facd80d2370 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.531+0000 7facdd200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd80d2820 con 0x7facd80d16a0 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.531+0000 7facdd200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd8161700 con 0x7facd815d9d0 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.531+0000 7facd6800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7facd80d2370 0x7facd815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55114/0 (socket says 172.21.15.12:55114) 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.531+0000 7facd6800700 1 -- 172.21.15.12:0/2236712838 learned_addr learned my addr 172.21.15.12:0/2236712838 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.531+0000 7facd5e00700 1 --2- 172.21.15.12:0/2236712838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 0x7facd80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.531+0000 7facd5400700 1 --2- 172.21.15.12:0/2236712838 >> v2:172.21.15.159:3300/0 conn(0x7facd80d16a0 0x7facd80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.532+0000 7facd4a00700 1 -- 172.21.15.12:0/2236712838 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162197490 0 0) 0x7facd8161420 con 0x7facd80d2370 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.532+0000 7facd4a00700 1 -- 172.21.15.12:0/2236712838 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facbc0035b0 con 0x7facd80d2370 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.532+0000 7facd5e00700 1 -- 172.21.15.12:0/2236712838 >> v2:172.21.15.159:3300/0 conn(0x7facd80d16a0 msgr2=0x7facd80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.532+0000 7facd5e00700 1 --2- 172.21.15.12:0/2236712838 >> v2:172.21.15.159:3300/0 conn(0x7facd80d16a0 0x7facd80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.532+0000 7facd5e00700 1 -- 172.21.15.12:0/2236712838 >> v1:172.21.15.12:6789/0 conn(0x7facd80d2370 legacy=0x7facd815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.532+0000 7facd5e00700 1 -- 172.21.15.12:0/2236712838 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd81619e0 con 0x7facd815d9d0 2024-03-29T10:09:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.532+0000 7facd5e00700 1 --2- 172.21.15.12:0/2236712838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 0x7facd80d3c30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7facc8009840 tx=0x7facc8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95becb71eb448a19 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.532+0000 7facd4a00700 1 -- 172.21.15.12:0/2236712838 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8011830 con 0x7facd815d9d0 2024-03-29T10:09:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.532+0000 7facd4a00700 1 -- 172.21.15.12:0/2236712838 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc8011990 con 0x7facd815d9d0 2024-03-29T10:09:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.532+0000 7facd4a00700 1 -- 172.21.15.12:0/2236712838 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8011d20 con 0x7facd815d9d0 2024-03-29T10:09:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.533+0000 7facdd200700 1 -- 172.21.15.12:0/2236712838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 msgr2=0x7facd80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.533+0000 7facdd200700 1 --2- 172.21.15.12:0/2236712838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 0x7facd80d3c30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7facc8009840 tx=0x7facc8009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.533+0000 7facdd200700 1 -- 172.21.15.12:0/2236712838 shutdown_connections 2024-03-29T10:09:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.533+0000 7facdd200700 1 --2- 172.21.15.12:0/2236712838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 0x7facd80d3c30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.533+0000 7facdd200700 1 --2- 172.21.15.12:0/2236712838 >> v2:172.21.15.159:3300/0 conn(0x7facd80d16a0 0x7facd80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.533+0000 7facdd200700 1 -- 172.21.15.12:0/2236712838 >> 172.21.15.12:0/2236712838 conn(0x7facd808aff0 msgr2=0x7facd80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.533+0000 7facdd200700 1 -- 172.21.15.12:0/2236712838 shutdown_connections 2024-03-29T10:09:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.533+0000 7facdd200700 1 -- 172.21.15.12:0/2236712838 wait complete. 2024-03-29T10:09:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.535+0000 7facdd200700 1 Processor -- start 2024-03-29T10:09:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.535+0000 7facdd200700 1 -- start start 2024-03-29T10:09:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.536+0000 7facdd200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facd80d16a0 0x7facd81f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.536+0000 7facdd200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd80d2370 0x7facd815e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.536+0000 7facdd200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 0x7facd81588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.536+0000 7facdd200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd815f5a0 con 0x7facd80d2370 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.536+0000 7facdd200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd80b7320 con 0x7facd80d16a0 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.536+0000 7facdd200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd815ed10 con 0x7facd815d9d0 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.536+0000 7facd5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd80d2370 0x7facd815e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.536+0000 7facd5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd80d2370 0x7facd815e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42656/0 (socket says 172.21.15.12:42656) 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.536+0000 7facd5400700 1 -- 172.21.15.12:0/3281856037 learned_addr learned my addr 172.21.15.12:0/3281856037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facd5e00700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facd80d16a0 0x7facd81f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facd6800700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 0x7facd81588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facd5400700 1 -- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 msgr2=0x7facd81588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facd5400700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 0x7facd81588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facd5400700 1 -- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facd80d16a0 msgr2=0x7facd81f4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facd5400700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facd80d16a0 0x7facd81f4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facd5400700 1 -- 172.21.15.12:0/3281856037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc80094a0 con 0x7facd80d2370 2024-03-29T10:09:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facd5400700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd80d2370 0x7facd815e7d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7facc400e960 tx=0x7facc400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facce000700 1 -- 172.21.15.12:0/3281856037 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40096f0 con 0x7facd80d2370 2024-03-29T10:09:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facce000700 1 -- 172.21.15.12:0/3281856037 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc400eeb0 con 0x7facd80d2370 2024-03-29T10:09:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facdd200700 1 -- 172.21.15.12:0/3281856037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd815efe0 con 0x7facd80d2370 2024-03-29T10:09:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.537+0000 7facce000700 1 -- 172.21.15.12:0/3281856037 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc400ca70 con 0x7facd80d2370 2024-03-29T10:09:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.538+0000 7facdd200700 1 -- 172.21.15.12:0/3281856037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd80b7530 con 0x7facd80d2370 2024-03-29T10:09:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.538+0000 7facdd200700 1 -- 172.21.15.12:0/3281856037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd80d1ca0 con 0x7facd80d2370 2024-03-29T10:09:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.539+0000 7facce000700 1 -- 172.21.15.12:0/3281856037 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7facc400cbd0 con 0x7facd80d2370 2024-03-29T10:09:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.541+0000 7facce000700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facb806adb0 0x7facb806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.541+0000 7facce000700 1 -- 172.21.15.12:0/3281856037 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5545+0+0 (secure 0 0 0) 0x7facc4089820 con 0x7facd80d2370 2024-03-29T10:09:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.541+0000 7facd5e00700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facb806adb0 0x7facb806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:19.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.542+0000 7facd5e00700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facb806adb0 0x7facb806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7facc8000c00 tx=0x7facc801c040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.545+0000 7facce000700 1 -- 172.21.15.12:0/3281856037 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7facc4054990 con 0x7facd80d2370 2024-03-29T10:09:19.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.759+0000 7facdd200700 1 -- 172.21.15.12:0/3281856037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7facd8067720 con 0x7facd80d2370 2024-03-29T10:09:19.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.764+0000 7facce000700 1 -- 172.21.15.12:0/3281856037 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7facc40210c0 con 0x7facd80d2370 2024-03-29T10:09:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 -- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facb806adb0 msgr2=0x7facb806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facb806adb0 0x7facb806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7facc8000c00 tx=0x7facc801c040 comp rx=0 tx=0).stop 2024-03-29T10:09:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 -- 172.21.15.12:0/3281856037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd80d2370 msgr2=0x7facd815e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd80d2370 0x7facd815e7d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7facc400e960 tx=0x7facc400ec70 comp rx=0 tx=0).stop 2024-03-29T10:09:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 -- 172.21.15.12:0/3281856037 shutdown_connections 2024-03-29T10:09:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facb806adb0 0x7facb806d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facd815d9d0 0x7facd81588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facd80d16a0 0x7facd81f4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 --2- 172.21.15.12:0/3281856037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd80d2370 0x7facd815e7d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 -- 172.21.15.12:0/3281856037 >> 172.21.15.12:0/3281856037 conn(0x7facd808aff0 msgr2=0x7facd80b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 -- 172.21.15.12:0/3281856037 shutdown_connections 2024-03-29T10:09:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:19.766+0000 7facdd200700 1 -- 172.21.15.12:0/3281856037 wait complete. 2024-03-29T10:09:19.784 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:09:19.784 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:09:19.867 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:09:19.961 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:09:19.961 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:09:19.961 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:20.006 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:09:20.006 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:09:20.006 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:20.053 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:09:20.053 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:09:20.053 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:20.097 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:09:20.097 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:09:20.097 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:20.142 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:09:20.143 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T10:09:20.143 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:09:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.342+0000 7fbbad800700 1 Processor -- start 2024-03-29T10:09:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.342+0000 7fbbad800700 1 -- start start 2024-03-29T10:09:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.343+0000 7fbbad800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbba8161ca0 0x7fbba80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.343+0000 7fbbad800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba80ceda0 con 0x7fbba8160f10 2024-03-29T10:09:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.343+0000 7fbbad800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba81612f0 con 0x7fbba80d18b0 2024-03-29T10:09:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.343+0000 7fbbad800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba80d1d30 con 0x7fbba8161ca0 2024-03-29T10:09:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.344+0000 7fbba7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbba8161ca0 0x7fbba80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.344+0000 7fbba6a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbba8160f10 0x7fbba80d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55130/0 (socket says 172.21.15.12:55130) 2024-03-29T10:09:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.344+0000 7fbba6a00700 1 -- 172.21.15.12:0/1181725521 learned_addr learned my addr 172.21.15.12:0/1181725521 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.344+0000 7fbba7400700 1 -- 172.21.15.12:0/1181725521 >> v1:172.21.15.159:6789/0 conn(0x7fbba80d18b0 legacy=0x7fbba8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.344+0000 7fbba7400700 1 -- 172.21.15.12:0/1181725521 >> v1:172.21.15.12:6789/0 conn(0x7fbba8160f10 legacy=0x7fbba80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.344+0000 7fbba7400700 1 -- 172.21.15.12:0/1181725521 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba80ce470 con 0x7fbba8161ca0 2024-03-29T10:09:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.344+0000 7fbba7400700 1 --2- 172.21.15.12:0/1181725521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8161ca0 0x7fbba80cf230 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbba0009d90 tx=0x7fbba000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fa3630720ece96 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.344+0000 7fbba5600700 1 -- 172.21.15.12:0/1181725521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba000e070 con 0x7fbba8161ca0 2024-03-29T10:09:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.345+0000 7fbba5600700 1 -- 172.21.15.12:0/1181725521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba000bb90 con 0x7fbba8161ca0 2024-03-29T10:09:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.345+0000 7fbba5600700 1 -- 172.21.15.12:0/1181725521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba0003a20 con 0x7fbba8161ca0 2024-03-29T10:09:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.345+0000 7fbbad800700 1 -- 172.21.15.12:0/1181725521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8161ca0 msgr2=0x7fbba80cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.345+0000 7fbbad800700 1 --2- 172.21.15.12:0/1181725521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8161ca0 0x7fbba80cf230 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbba0009d90 tx=0x7fbba000b330 comp rx=0 tx=0).stop 2024-03-29T10:09:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.345+0000 7fbbad800700 1 -- 172.21.15.12:0/1181725521 shutdown_connections 2024-03-29T10:09:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.345+0000 7fbbad800700 1 --2- 172.21.15.12:0/1181725521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8161ca0 0x7fbba80cf230 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.345+0000 7fbbad800700 1 -- 172.21.15.12:0/1181725521 >> 172.21.15.12:0/1181725521 conn(0x7fbba808aff0 msgr2=0x7fbba80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.345+0000 7fbbad800700 1 -- 172.21.15.12:0/1181725521 shutdown_connections 2024-03-29T10:09:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.345+0000 7fbbad800700 1 -- 172.21.15.12:0/1181725521 wait complete. 2024-03-29T10:09:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.347+0000 7fbbad800700 1 Processor -- start 2024-03-29T10:09:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.347+0000 7fbbad800700 1 -- start start 2024-03-29T10:09:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbbad800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbba80d18b0 0x7fbba815ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbbad800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8160f10 0x7fbba81e8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbbad800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbba8161ca0 0x7fbba81e93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbbad800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba815f570 con 0x7fbba8160f10 2024-03-29T10:09:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbbad800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba815f850 con 0x7fbba80d18b0 2024-03-29T10:09:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbbad800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba815fb30 con 0x7fbba8161ca0 2024-03-29T10:09:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbba6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8160f10 0x7fbba81e8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbba6a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbba80d18b0 0x7fbba815ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbba6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8160f10 0x7fbba81e8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42682/0 (socket says 172.21.15.12:42682) 2024-03-29T10:09:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbba6000700 1 -- 172.21.15.12:0/1138355016 learned_addr learned my addr 172.21.15.12:0/1138355016 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.348+0000 7fbba7400700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbba8161ca0 0x7fbba81e93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.349+0000 7fbba6000700 1 -- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbba8161ca0 msgr2=0x7fbba81e93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.349+0000 7fbba6000700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbba8161ca0 0x7fbba81e93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.349+0000 7fbba6000700 1 -- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbba80d18b0 msgr2=0x7fbba815ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.349+0000 7fbba6000700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbba80d18b0 0x7fbba815ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.349+0000 7fbba6000700 1 -- 172.21.15.12:0/1138355016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba0009a70 con 0x7fbba8160f10 2024-03-29T10:09:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.349+0000 7fbba6000700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8160f10 0x7fbba81e8e60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400b4a0 tx=0x7fbb9400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.349+0000 7fbb9ea00700 1 -- 172.21.15.12:0/1138355016 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94011830 con 0x7fbba8160f10 2024-03-29T10:09:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.349+0000 7fbb9ea00700 1 -- 172.21.15.12:0/1138355016 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb94011990 con 0x7fbba8160f10 2024-03-29T10:09:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.349+0000 7fbb9ea00700 1 -- 172.21.15.12:0/1138355016 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94011d40 con 0x7fbba8160f10 2024-03-29T10:09:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.349+0000 7fbbad800700 1 -- 172.21.15.12:0/1138355016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba815fe10 con 0x7fbba8160f10 2024-03-29T10:09:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.350+0000 7fbbad800700 1 -- 172.21.15.12:0/1138355016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba815f180 con 0x7fbba8160f10 2024-03-29T10:09:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.351+0000 7fbbad800700 1 -- 172.21.15.12:0/1138355016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba80b61b0 con 0x7fbba8160f10 2024-03-29T10:09:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.351+0000 7fbb9ea00700 1 -- 172.21.15.12:0/1138355016 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbb9401a400 con 0x7fbba8160f10 2024-03-29T10:09:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.352+0000 7fbb9ea00700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb8806adb0 0x7fbb8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.352+0000 7fbb9ea00700 1 -- 172.21.15.12:0/1138355016 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5545+0+0 (secure 0 0 0) 0x7fbb94088f20 con 0x7fbba8160f10 2024-03-29T10:09:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.353+0000 7fbba6a00700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb8806adb0 0x7fbb8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.353+0000 7fbba6a00700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb8806adb0 0x7fbb8806d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb98000f60 tx=0x7fbb98007f10 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:20.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.357+0000 7fbb9ea00700 1 -- 172.21.15.12:0/1138355016 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbba80b61b0 con 0x7fbba8160f10 2024-03-29T10:09:20.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.618+0000 7fbbad800700 1 -- 172.21.15.12:0/1138355016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbba80b61b0 con 0x7fbba8160f10 2024-03-29T10:09:20.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.618+0000 7fbb9ea00700 1 -- 172.21.15.12:0/1138355016 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+3532 (secure 0 0 0) 0x7fbba80b61b0 con 0x7fbba8160f10 2024-03-29T10:09:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:20.619 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5746,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/981039791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":981039791},{"type":"v1","addr":"172.21.15.159:6836","nonce":981039791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1682878615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1682878615},{"type":"v1","addr":"172.21.15.159:6837","nonce":1682878615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5752,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1663013205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":1663013205},{"type":"v1","addr":"172.21.15.12:6834","nonce":1663013205}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":6297,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2213978278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2213978278},{"type":"v1","addr":"172.21.15.12:6835","nonce":2213978278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[]} 2024-03-29T10:09:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 -- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb8806adb0 msgr2=0x7fbb8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb8806adb0 0x7fbb8806d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb98000f60 tx=0x7fbb98007f10 comp rx=0 tx=0).stop 2024-03-29T10:09:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 -- 172.21.15.12:0/1138355016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8160f10 msgr2=0x7fbba81e8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8160f10 0x7fbba81e8e60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400b4a0 tx=0x7fbb9400b7b0 comp rx=0 tx=0).stop 2024-03-29T10:09:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 -- 172.21.15.12:0/1138355016 shutdown_connections 2024-03-29T10:09:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb8806adb0 0x7fbb8806d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbba8161ca0 0x7fbba81e93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbba80d18b0 0x7fbba815ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 --2- 172.21.15.12:0/1138355016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbba8160f10 0x7fbba81e8e60 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 -- 172.21.15.12:0/1138355016 >> 172.21.15.12:0/1138355016 conn(0x7fbba808aff0 msgr2=0x7fbba80d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 -- 172.21.15.12:0/1138355016 shutdown_connections 2024-03-29T10:09:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:20.621+0000 7fbbad800700 1 -- 172.21.15.12:0/1138355016 wait complete. 2024-03-29T10:09:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 25 2024-03-29T10:09:20.640 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T10:09:20.640 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T10:09:20.640 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T10:09:20.640 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:09:20.640 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T10:09:20.640 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T10:09:20.641 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T10:09:20.641 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:09:20.641 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T10:09:20.641 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T10:09:20.641 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T10:09:20.641 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:09:20.641 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T10:09:20.642 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T10:09:20.642 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T10:09:20.642 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:09:20.656 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T10:09:20.656 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T10:09:20.657 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T10:09:20.657 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T10:09:20.658 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T10:09:20.659 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T10:09:20.701 INFO:tasks.ceph.mds.a:Started 2024-03-29T10:09:20.701 DEBUG:teuthology.parallel:result is None 2024-03-29T10:09:20.703 INFO:tasks.ceph.mds.d:Started 2024-03-29T10:09:20.703 DEBUG:teuthology.parallel:result is None 2024-03-29T10:09:20.704 INFO:tasks.ceph.mds.c:Started 2024-03-29T10:09:20.705 DEBUG:teuthology.parallel:result is None 2024-03-29T10:09:20.842 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T10:09:20.845 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T10:09:20.845 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:09:26.638 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:09:26.642 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:09:26.659 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T10:09:26.659 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T10:09:26.692 INFO:tasks.ceph.mds.b:Started 2024-03-29T10:09:26.692 DEBUG:teuthology.parallel:result is None 2024-03-29T10:09:26.692 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T10:09:26.836 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T10:09:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.901+0000 7f12bf600700 1 Processor -- start 2024-03-29T10:09:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.902+0000 7f12bf600700 1 -- start start 2024-03-29T10:09:26.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.903+0000 7f12bf600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f12b80cee10 0x7f12b8161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:26.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.903+0000 7f12bf600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b80ce270 con 0x7f12b81611b0 2024-03-29T10:09:26.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.903+0000 7f12bf600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b815b600 con 0x7f12b80d21f0 2024-03-29T10:09:26.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.903+0000 7f12bf600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b80d2760 con 0x7f12b80cee10 2024-03-29T10:09:26.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.903+0000 7f12bcc00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f12b81611b0 0x7f12b815ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55140/0 (socket says 172.21.15.12:55140) 2024-03-29T10:09:26.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.903+0000 7f12bcc00700 1 -- 172.21.15.12:0/3541735958 learned_addr learned my addr 172.21.15.12:0/3541735958 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:26.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.903+0000 7f12bd600700 1 --2- 172.21.15.12:0/3541735958 >> v2:172.21.15.159:3300/0 conn(0x7f12b80cee10 0x7f12b8161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12b7400700 1 -- 172.21.15.12:0/3541735958 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761661384 0 0) 0x7f12b80ce270 con 0x7f12b81611b0 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12b7400700 1 -- 172.21.15.12:0/3541735958 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a00035b0 con 0x7f12b81611b0 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12b7400700 1 -- 172.21.15.12:0/3541735958 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251571031 0 0) 0x7f12b815b600 con 0x7f12b80d21f0 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12b7400700 1 -- 172.21.15.12:0/3541735958 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b80ce270 con 0x7f12b80d21f0 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12bd600700 1 -- 172.21.15.12:0/3541735958 >> v1:172.21.15.159:6789/0 conn(0x7f12b80d21f0 legacy=0x7f12b80ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12bd600700 1 -- 172.21.15.12:0/3541735958 >> v1:172.21.15.12:6789/0 conn(0x7f12b81611b0 legacy=0x7f12b815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12bd600700 1 -- 172.21.15.12:0/3541735958 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b80d2a40 con 0x7f12b80cee10 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12bd600700 1 --2- 172.21.15.12:0/3541735958 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b80cee10 0x7f12b8161590 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009d90 tx=0x7f12b000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dcb99c3ec8a70ee server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12b7400700 1 -- 172.21.15.12:0/3541735958 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b000e070 con 0x7f12b80cee10 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12b7400700 1 -- 172.21.15.12:0/3541735958 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b000bb20 con 0x7f12b80cee10 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.904+0000 7f12b7400700 1 -- 172.21.15.12:0/3541735958 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0012430 con 0x7f12b80cee10 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.905+0000 7f12bf600700 1 -- 172.21.15.12:0/3541735958 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b80cee10 msgr2=0x7f12b8161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.905+0000 7f12bf600700 1 --2- 172.21.15.12:0/3541735958 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b80cee10 0x7f12b8161590 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009d90 tx=0x7f12b000b330 comp rx=0 tx=0).stop 2024-03-29T10:09:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.905+0000 7f12bf600700 1 -- 172.21.15.12:0/3541735958 shutdown_connections 2024-03-29T10:09:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.905+0000 7f12bf600700 1 --2- 172.21.15.12:0/3541735958 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b80cee10 0x7f12b8161590 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.905+0000 7f12bf600700 1 -- 172.21.15.12:0/3541735958 >> 172.21.15.12:0/3541735958 conn(0x7f12b808aff0 msgr2=0x7f12b80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.905+0000 7f12bf600700 1 -- 172.21.15.12:0/3541735958 shutdown_connections 2024-03-29T10:09:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.905+0000 7f12bf600700 1 -- 172.21.15.12:0/3541735958 wait complete. 2024-03-29T10:09:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.907+0000 7f12bf600700 1 Processor -- start 2024-03-29T10:09:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.907+0000 7f12bf600700 1 -- start start 2024-03-29T10:09:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12bf600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b80cee10 0x7f12b80caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12bf600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b80d21f0 0x7f12b814a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12bf600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b81611b0 0x7f12b80b7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12bf600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80b71d0 con 0x7f12b80d21f0 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12bf600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80cb3e0 con 0x7f12b81611b0 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12bf600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b814ae40 con 0x7f12b80cee10 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12b7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b80d21f0 0x7f12b814a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12b7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b80d21f0 0x7f12b814a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42716/0 (socket says 172.21.15.12:42716) 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12b7e00700 1 -- 172.21.15.12:0/3451136803 learned_addr learned my addr 172.21.15.12:0/3451136803 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12bcc00700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b80cee10 0x7f12b80caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12bd600700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b81611b0 0x7f12b80b7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12b7e00700 1 -- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b80cee10 msgr2=0x7f12b80caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12b7e00700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b80cee10 0x7f12b80caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12b7e00700 1 -- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b81611b0 msgr2=0x7f12b80b7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12b7e00700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b81611b0 0x7f12b80b7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.908+0000 7f12b7e00700 1 -- 172.21.15.12:0/3451136803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b0009a70 con 0x7f12b80d21f0 2024-03-29T10:09:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.909+0000 7f12b7e00700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b80d21f0 0x7f12b814a900 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c520 tx=0x7f12a400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.909+0000 7f12b4c00700 1 -- 172.21.15.12:0/3451136803 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40052d0 con 0x7f12b80d21f0 2024-03-29T10:09:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.909+0000 7f12b4c00700 1 -- 172.21.15.12:0/3451136803 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a4005430 con 0x7f12b80d21f0 2024-03-29T10:09:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.909+0000 7f12bf600700 1 -- 172.21.15.12:0/3451136803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b80c7190 con 0x7f12b80d21f0 2024-03-29T10:09:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.909+0000 7f12b4c00700 1 -- 172.21.15.12:0/3451136803 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40038c0 con 0x7f12b80d21f0 2024-03-29T10:09:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.909+0000 7f12bf600700 1 -- 172.21.15.12:0/3451136803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b80c7750 con 0x7f12b80d21f0 2024-03-29T10:09:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.910+0000 7f12bf600700 1 -- 172.21.15.12:0/3451136803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b8067720 con 0x7f12b80d21f0 2024-03-29T10:09:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.911+0000 7f12b4c00700 1 -- 172.21.15.12:0/3451136803 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f12a4003a20 con 0x7f12b80d21f0 2024-03-29T10:09:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.912+0000 7f12b4c00700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f129806adb0 0x7f129806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.912+0000 7f12b4c00700 1 -- 172.21.15.12:0/3451136803 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f12a4088880 con 0x7f12b80d21f0 2024-03-29T10:09:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.912+0000 7f12bcc00700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f129806adb0 0x7f129806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.913+0000 7f12bcc00700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f129806adb0 0x7f129806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f12ac007fd0 tx=0x7f12ac007dc0 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:26.917+0000 7f12b4c00700 1 -- 172.21.15.12:0/3451136803 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12a40539e0 con 0x7f12b80d21f0 2024-03-29T10:09:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:27.153+0000 7f12bf600700 1 -- 172.21.15.12:0/3451136803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f12b80b5d00 con 0x7f12b80d21f0 2024-03-29T10:09:28.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.045+0000 7f12b4c00700 1 -- 172.21.15.12:0/3451136803 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v76) v1 ==== 97+0+0 (secure 0 0 0) 0x7f12a4057000 con 0x7f12b80d21f0 2024-03-29T10:09:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.048+0000 7f12bf600700 1 -- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f129806adb0 msgr2=0x7f129806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.048+0000 7f12bf600700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f129806adb0 0x7f129806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f12ac007fd0 tx=0x7f12ac007dc0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.048+0000 7f12bf600700 1 -- 172.21.15.12:0/3451136803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b80d21f0 msgr2=0x7f12b814a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.048+0000 7f12bf600700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b80d21f0 0x7f12b814a900 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c520 tx=0x7f12a400c830 comp rx=0 tx=0).stop 2024-03-29T10:09:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.049+0000 7f12bf600700 1 -- 172.21.15.12:0/3451136803 shutdown_connections 2024-03-29T10:09:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.049+0000 7f12bf600700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f129806adb0 0x7f129806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.049+0000 7f12bf600700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b81611b0 0x7f12b80b7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.049+0000 7f12bf600700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b80cee10 0x7f12b80caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.049+0000 7f12bf600700 1 --2- 172.21.15.12:0/3451136803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b80d21f0 0x7f12b814a900 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.049+0000 7f12bf600700 1 -- 172.21.15.12:0/3451136803 >> 172.21.15.12:0/3451136803 conn(0x7f12b808aff0 msgr2=0x7f12b806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.049+0000 7f12bf600700 1 -- 172.21.15.12:0/3451136803 shutdown_connections 2024-03-29T10:09:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.049+0000 7f12bf600700 1 -- 172.21.15.12:0/3451136803 wait complete. 2024-03-29T10:09:28.050 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T10:09:28.070 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T10:09:28.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.273+0000 7f8117000700 1 Processor -- start 2024-03-29T10:09:28.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.273+0000 7f8117000700 1 -- start start 2024-03-29T10:09:28.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.274+0000 7f8117000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81100b3af0 0x7f811015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.274+0000 7f8117000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8110072580 0x7f81100b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.274+0000 7f8117000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81100b2180 con 0x7f81100b2e50 2024-03-29T10:09:28.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.274+0000 7f8117000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811015b100 con 0x7f8110072580 2024-03-29T10:09:28.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.274+0000 7f8117000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811015b3e0 con 0x7f81100b3af0 2024-03-29T10:09:28.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.274+0000 7f810f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8110072580 0x7f81100b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.274+0000 7f810f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8110072580 0x7f81100b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42722/0 (socket says 172.21.15.12:42722) 2024-03-29T10:09:28.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.274+0000 7f810f400700 1 -- 172.21.15.12:0/351131538 learned_addr learned my addr 172.21.15.12:0/351131538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:28.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.275+0000 7f810fe00700 1 --2- 172.21.15.12:0/351131538 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81100b3af0 0x7f811015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.275+0000 7f810f400700 1 -- 172.21.15.12:0/351131538 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81100b3af0 msgr2=0x7f811015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:28.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.275+0000 7f810f400700 1 --2- 172.21.15.12:0/351131538 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81100b3af0 0x7f811015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.275+0000 7f810f400700 1 -- 172.21.15.12:0/351131538 >> v1:172.21.15.159:6789/0 conn(0x7f81100b2e50 legacy=0x7f81100709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.275+0000 7f810f400700 1 -- 172.21.15.12:0/351131538 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81100b23f0 con 0x7f8110072580 2024-03-29T10:09:28.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.275+0000 7f810f400700 1 --2- 172.21.15.12:0/351131538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8110072580 0x7f81100b2910 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f80fc0098f0 tx=0x7f80fc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346ee63367db9e78 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:28.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.275+0000 7f810ea00700 1 -- 172.21.15.12:0/351131538 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc00b8a0 con 0x7f8110072580 2024-03-29T10:09:28.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.275+0000 7f810ea00700 1 -- 172.21.15.12:0/351131538 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc00ba00 con 0x7f8110072580 2024-03-29T10:09:28.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.275+0000 7f810ea00700 1 -- 172.21.15.12:0/351131538 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc00bd90 con 0x7f8110072580 2024-03-29T10:09:28.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.276+0000 7f8117000700 1 -- 172.21.15.12:0/351131538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8110072580 msgr2=0x7f81100b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.276+0000 7f8117000700 1 --2- 172.21.15.12:0/351131538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8110072580 0x7f81100b2910 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f80fc0098f0 tx=0x7f80fc009c00 comp rx=0 tx=0).stop 2024-03-29T10:09:28.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.276+0000 7f8117000700 1 -- 172.21.15.12:0/351131538 shutdown_connections 2024-03-29T10:09:28.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.276+0000 7f8117000700 1 --2- 172.21.15.12:0/351131538 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81100b3af0 0x7f811015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.276+0000 7f8117000700 1 --2- 172.21.15.12:0/351131538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8110072580 0x7f81100b2910 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.276+0000 7f8117000700 1 -- 172.21.15.12:0/351131538 >> 172.21.15.12:0/351131538 conn(0x7f81100cf190 msgr2=0x7f81100cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:28.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.276+0000 7f8117000700 1 -- 172.21.15.12:0/351131538 shutdown_connections 2024-03-29T10:09:28.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.276+0000 7f8117000700 1 -- 172.21.15.12:0/351131538 wait complete. 2024-03-29T10:09:28.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.278+0000 7f8117000700 1 Processor -- start 2024-03-29T10:09:28.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.278+0000 7f8117000700 1 -- start start 2024-03-29T10:09:28.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.279+0000 7f8117000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8110072580 0x7f811014e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.279+0000 7f8117000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81100b2e50 0x7f8110159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.279+0000 7f8117000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81100b3af0 0x7f8110156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.279+0000 7f8117000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8110155660 con 0x7f81100b2e50 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.279+0000 7f8117000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8110155380 con 0x7f8110072580 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.279+0000 7f8117000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81101524d0 con 0x7f81100b3af0 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.279+0000 7f810f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81100b2e50 0x7f8110159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.279+0000 7f810f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81100b2e50 0x7f8110159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42730/0 (socket says 172.21.15.12:42730) 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.279+0000 7f810f400700 1 -- 172.21.15.12:0/856776475 learned_addr learned my addr 172.21.15.12:0/856776475 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.279+0000 7f810fe00700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8110072580 0x7f811014e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f8115000700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81100b3af0 0x7f8110156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f810f400700 1 -- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81100b3af0 msgr2=0x7f8110156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f810f400700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81100b3af0 0x7f8110156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f810f400700 1 -- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8110072580 msgr2=0x7f811014e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f810f400700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8110072580 0x7f811014e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f810f400700 1 -- 172.21.15.12:0/856776475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc009550 con 0x7f81100b2e50 2024-03-29T10:09:28.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f810f400700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81100b2e50 0x7f8110159dc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009c40 tx=0x7f80fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:28.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f8107e00700 1 -- 172.21.15.12:0/856776475 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc004590 con 0x7f81100b2e50 2024-03-29T10:09:28.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f8107e00700 1 -- 172.21.15.12:0/856776475 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc01e070 con 0x7f81100b2e50 2024-03-29T10:09:28.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f8107e00700 1 -- 172.21.15.12:0/856776475 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc0038e0 con 0x7f81100b2e50 2024-03-29T10:09:28.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f8117000700 1 -- 172.21.15.12:0/856776475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81101527b0 con 0x7f81100b2e50 2024-03-29T10:09:28.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.280+0000 7f8117000700 1 -- 172.21.15.12:0/856776475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811014eb20 con 0x7f81100b2e50 2024-03-29T10:09:28.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.281+0000 7f8117000700 1 -- 172.21.15.12:0/856776475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8110152cd0 con 0x7f81100b2e50 2024-03-29T10:09:28.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.282+0000 7f8107e00700 1 -- 172.21.15.12:0/856776475 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f80fc003a40 con 0x7f81100b2e50 2024-03-29T10:09:28.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.283+0000 7f8107e00700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f80f006adb0 0x7f80f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.283+0000 7f8107e00700 1 -- 172.21.15.12:0/856776475 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f80fc020070 con 0x7f81100b2e50 2024-03-29T10:09:28.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.284+0000 7f810fe00700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f80f006adb0 0x7f80f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.285+0000 7f810fe00700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f80f006adb0 0x7f80f006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8100006b20 tx=0x7f8100008040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:28.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.288+0000 7f8107e00700 1 -- 172.21.15.12:0/856776475 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8110152cd0 con 0x7f81100b2e50 2024-03-29T10:09:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.587+0000 7f8117000700 1 -- 172.21.15.12:0/856776475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8110152cd0 con 0x7f81100b2e50 2024-03-29T10:09:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.587+0000 7f8107e00700 1 -- 172.21.15.12:0/856776475 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8110152cd0 con 0x7f81100b2e50 2024-03-29T10:09:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:28.587 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:09:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T10:09:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T10:09:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T10:09:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:09:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:09:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T10:09:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:09:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T10:09:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T10:09:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T10:09:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T10:09:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T10:09:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:09:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:09:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:09:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:09:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T10:09:28.600 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 -- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f80f006adb0 msgr2=0x7f80f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f80f006adb0 0x7f80f006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8100006b20 tx=0x7f8100008040 comp rx=0 tx=0).stop 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 -- 172.21.15.12:0/856776475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81100b2e50 msgr2=0x7f8110159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81100b2e50 0x7f8110159dc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009c40 tx=0x7f80fc004750 comp rx=0 tx=0).stop 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 -- 172.21.15.12:0/856776475 shutdown_connections 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f80f006adb0 0x7f80f006d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81100b3af0 0x7f8110156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8110072580 0x7f811014e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 --2- 172.21.15.12:0/856776475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81100b2e50 0x7f8110159dc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 -- 172.21.15.12:0/856776475 >> 172.21.15.12:0/856776475 conn(0x7f81100cf190 msgr2=0x7f811015ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 -- 172.21.15.12:0/856776475 shutdown_connections 2024-03-29T10:09:28.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.590+0000 7f8117000700 1 -- 172.21.15.12:0/856776475 wait complete. 2024-03-29T10:09:28.607 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:09:28.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.806+0000 7fe46ee00700 1 Processor -- start 2024-03-29T10:09:28.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.806+0000 7fe46ee00700 1 -- start start 2024-03-29T10:09:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.807+0000 7fe46ee00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4680d18b0 0x7fe4680cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.807+0000 7fe46ee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468161590 0x7fe468161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.807+0000 7fe46ee00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4680ceda0 con 0x7fe468160f10 2024-03-29T10:09:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.807+0000 7fe46ee00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4680d1d30 con 0x7fe4680d18b0 2024-03-29T10:09:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.807+0000 7fe46ee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe468161f60 con 0x7fe468161590 2024-03-29T10:09:28.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.807+0000 7fe45fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4680d18b0 0x7fe4680cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.807+0000 7fe467e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe468160f10 0x7fe4680d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55154/0 (socket says 172.21.15.12:55154) 2024-03-29T10:09:28.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.807+0000 7fe467e00700 1 -- 172.21.15.12:0/1891888203 learned_addr learned my addr 172.21.15.12:0/1891888203 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:28.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe46ce00700 1 --2- 172.21.15.12:0/1891888203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468161590 0x7fe468161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe45fe00700 1 -- 172.21.15.12:0/1891888203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468161590 msgr2=0x7fe468161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:28.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe45fe00700 1 --2- 172.21.15.12:0/1891888203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468161590 0x7fe468161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe45fe00700 1 -- 172.21.15.12:0/1891888203 >> v1:172.21.15.12:6789/0 conn(0x7fe468160f10 legacy=0x7fe4680d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe45fe00700 1 -- 172.21.15.12:0/1891888203 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4681612f0 con 0x7fe4680d18b0 2024-03-29T10:09:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe46ce00700 1 --2- 172.21.15.12:0/1891888203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468161590 0x7fe468161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe467400700 1 -- 172.21.15.12:0/1891888203 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130734248 0 0) 0x7fe4680ceda0 con 0x7fe468160f10 2024-03-29T10:09:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe45fe00700 1 --2- 172.21.15.12:0/1891888203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 0x7fe4680cf230 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe450009870 tx=0x7fe450009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2db8759e9e21e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe467400700 1 -- 172.21.15.12:0/1891888203 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe450011830 con 0x7fe4680d18b0 2024-03-29T10:09:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe467400700 1 -- 172.21.15.12:0/1891888203 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe450011990 con 0x7fe4680d18b0 2024-03-29T10:09:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.808+0000 7fe467400700 1 -- 172.21.15.12:0/1891888203 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe450011d20 con 0x7fe4680d18b0 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.809+0000 7fe46ee00700 1 -- 172.21.15.12:0/1891888203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 msgr2=0x7fe4680cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.809+0000 7fe46ee00700 1 --2- 172.21.15.12:0/1891888203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 0x7fe4680cf230 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe450009870 tx=0x7fe450009b80 comp rx=0 tx=0).stop 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.809+0000 7fe46ee00700 1 -- 172.21.15.12:0/1891888203 shutdown_connections 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.809+0000 7fe46ee00700 1 --2- 172.21.15.12:0/1891888203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468161590 0x7fe468161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.809+0000 7fe46ee00700 1 --2- 172.21.15.12:0/1891888203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 0x7fe4680cf230 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.809+0000 7fe46ee00700 1 -- 172.21.15.12:0/1891888203 >> 172.21.15.12:0/1891888203 conn(0x7fe46808aff0 msgr2=0x7fe4680d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.809+0000 7fe46ee00700 1 -- 172.21.15.12:0/1891888203 shutdown_connections 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.809+0000 7fe46ee00700 1 -- 172.21.15.12:0/1891888203 wait complete. 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.811+0000 7fe46ee00700 1 Processor -- start 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.811+0000 7fe46ee00700 1 -- start start 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe46ee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 0x7fe4681f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe46ee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468160f10 0x7fe46809efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe46ee00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe468161590 0x7fe46809f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe46ee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46809fd80 con 0x7fe4680d18b0 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe46ee00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4680a0060 con 0x7fe468161590 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe46ee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4680a0340 con 0x7fe468160f10 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe467e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 0x7fe4681f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe467e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 0x7fe4681f5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42744/0 (socket says 172.21.15.12:42744) 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe467e00700 1 -- 172.21.15.12:0/3545850910 learned_addr learned my addr 172.21.15.12:0/3545850910 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe46ce00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe468161590 0x7fe46809f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.812+0000 7fe45fe00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468160f10 0x7fe46809efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe467e00700 1 -- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468160f10 msgr2=0x7fe46809efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe467e00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468160f10 0x7fe46809efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe467e00700 1 -- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe468161590 msgr2=0x7fe46809f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe467e00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe468161590 0x7fe46809f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe467e00700 1 -- 172.21.15.12:0/3545850910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4500094a0 con 0x7fe4680d18b0 2024-03-29T10:09:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe46ce00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe468161590 0x7fe46809f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe467e00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 0x7fe4681f5f20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe45800c550 tx=0x7fe45800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe464c00700 1 -- 172.21.15.12:0/3545850910 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458004c80 con 0x7fe4680d18b0 2024-03-29T10:09:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe464c00700 1 -- 172.21.15.12:0/3545850910 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe458004e00 con 0x7fe4680d18b0 2024-03-29T10:09:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe464c00700 1 -- 172.21.15.12:0/3545850910 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458007310 con 0x7fe4680d18b0 2024-03-29T10:09:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe46ee00700 1 -- 172.21.15.12:0/3545850910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4680a0620 con 0x7fe4680d18b0 2024-03-29T10:09:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.813+0000 7fe46ee00700 1 -- 172.21.15.12:0/3545850910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46809fa40 con 0x7fe4680d18b0 2024-03-29T10:09:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.814+0000 7fe46ee00700 1 -- 172.21.15.12:0/3545850910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe468067720 con 0x7fe4680d18b0 2024-03-29T10:09:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.815+0000 7fe464c00700 1 -- 172.21.15.12:0/3545850910 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe458007470 con 0x7fe4680d18b0 2024-03-29T10:09:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.816+0000 7fe464c00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe44406adb0 0x7fe44406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.816+0000 7fe464c00700 1 -- 172.21.15.12:0/3545850910 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe458088ec0 con 0x7fe4680d18b0 2024-03-29T10:09:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.817+0000 7fe45fe00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe44406adb0 0x7fe44406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.818+0000 7fe45fe00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe44406adb0 0x7fe44406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe4500090b0 tx=0x7fe45001c040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:28.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:28.820+0000 7fe464c00700 1 -- 172.21.15.12:0/3545850910 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe458054240 con 0x7fe4680d18b0 2024-03-29T10:09:29.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.039+0000 7fe46ee00700 1 -- 172.21.15.12:0/3545850910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4681f32a0 con 0x7fe4680d18b0 2024-03-29T10:09:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.040+0000 7fe464c00700 1 -- 172.21.15.12:0/3545850910 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+12389 (secure 0 0 0) 0x7fe458057860 con 0x7fe4680d18b0 2024-03-29T10:09:29.040 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:29.040 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":76,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:28.040281+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:09:29.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.042+0000 7fe46ee00700 1 -- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe44406adb0 msgr2=0x7fe44406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.042+0000 7fe46ee00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe44406adb0 0x7fe44406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe4500090b0 tx=0x7fe45001c040 comp rx=0 tx=0).stop 2024-03-29T10:09:29.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.043+0000 7fe46ee00700 1 -- 172.21.15.12:0/3545850910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 msgr2=0x7fe4681f5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.043+0000 7fe46ee00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 0x7fe4681f5f20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe45800c550 tx=0x7fe45800c910 comp rx=0 tx=0).stop 2024-03-29T10:09:29.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.043+0000 7fe46ee00700 1 -- 172.21.15.12:0/3545850910 shutdown_connections 2024-03-29T10:09:29.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.043+0000 7fe46ee00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe44406adb0 0x7fe44406d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.043+0000 7fe46ee00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe468161590 0x7fe46809f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.043+0000 7fe46ee00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4680d18b0 0x7fe4681f5f20 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.043+0000 7fe46ee00700 1 --2- 172.21.15.12:0/3545850910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe468160f10 0x7fe46809efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.043+0000 7fe46ee00700 1 -- 172.21.15.12:0/3545850910 >> 172.21.15.12:0/3545850910 conn(0x7fe46808aff0 msgr2=0x7fe4680ca9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:29.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.043+0000 7fe46ee00700 1 -- 172.21.15.12:0/3545850910 shutdown_connections 2024-03-29T10:09:29.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.043+0000 7fe46ee00700 1 -- 172.21.15.12:0/3545850910 wait complete. 2024-03-29T10:09:29.061 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:09:29.062 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:09:29.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.263+0000 7fc574c00700 1 Processor -- start 2024-03-29T10:09:29.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.263+0000 7fc574c00700 1 -- start start 2024-03-29T10:09:29.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.264+0000 7fc574c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc57015dad0 0x7fc5700ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.264+0000 7fc574c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5700d21d0 0x7fc57015fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.264+0000 7fc574c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57015f820 con 0x7fc5700d1630 2024-03-29T10:09:29.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.264+0000 7fc574c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5700cfcc0 con 0x7fc57015dad0 2024-03-29T10:09:29.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.264+0000 7fc574c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700cd9f0 con 0x7fc5700d21d0 2024-03-29T10:09:29.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56d400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc5700d1630 0x7fc5700d1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33106/0 (socket says 172.21.15.12:33106) 2024-03-29T10:09:29.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56d400700 1 -- 172.21.15.12:0/2388745924 learned_addr learned my addr 172.21.15.12:0/2388745924 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:29.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56de00700 1 --2- 172.21.15.12:0/2388745924 >> v2:172.21.15.159:3300/0 conn(0x7fc57015dad0 0x7fc5700ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56e800700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5700d21d0 0x7fc57015fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545430360 0 0) 0x7fc57015f820 con 0x7fc5700d1630 2024-03-29T10:09:29.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5540035b0 con 0x7fc5700d1630 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3698677767 0 0) 0x7fc5540035b0 con 0x7fc5700d1630 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5700d21d0 msgr2=0x7fc57015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56ca00700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5700d21d0 0x7fc57015fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 >> v2:172.21.15.159:3300/0 conn(0x7fc57015dad0 msgr2=0x7fc5700ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.265+0000 7fc56ca00700 1 --2- 172.21.15.12:0/2388745924 >> v2:172.21.15.159:3300/0 conn(0x7fc57015dad0 0x7fc5700ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5700cf420 con 0x7fc5700d1630 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fc55c001670 con 0x7fc5700d1630 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 >> v1:172.21.15.12:6789/0 conn(0x7fc5700d1630 legacy=0x7fc5700d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56ca00700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554003a00 0x7fc554003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56ca00700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5700d21d0 0x7fc554004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56ca00700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc554004d90 0x7fc554005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700cfcc0 con 0x7fc554003a00 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700cd9f0 con 0x7fc554004d90 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55400ba10 con 0x7fc5700d21d0 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56de00700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554003a00 0x7fc554003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56d400700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5700d21d0 0x7fc554004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56e800700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc554004d90 0x7fc554005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56de00700 1 -- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5700d21d0 msgr2=0x7fc554004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56de00700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5700d21d0 0x7fc554004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56de00700 1 -- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc554004d90 msgr2=0x7fc554005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56de00700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc554004d90 0x7fc554005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.266+0000 7fc56de00700 1 -- 172.21.15.12:0/2388745924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5540035b0 con 0x7fc554003a00 2024-03-29T10:09:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.267+0000 7fc56de00700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554003a00 0x7fc554003e20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc560009b60 tx=0x7fc554005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89413cf60f45720 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:29.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.267+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56000e6c0 con 0x7fc554003a00 2024-03-29T10:09:29.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.267+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56000e820 con 0x7fc554003a00 2024-03-29T10:09:29.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.267+0000 7fc56ca00700 1 -- 172.21.15.12:0/2388745924 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56000eb90 con 0x7fc554003a00 2024-03-29T10:09:29.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.267+0000 7fc574c00700 1 -- 172.21.15.12:0/2388745924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554003a00 msgr2=0x7fc554003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.267+0000 7fc574c00700 1 --2- 172.21.15.12:0/2388745924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554003a00 0x7fc554003e20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc560009b60 tx=0x7fc554005880 comp rx=0 tx=0).stop 2024-03-29T10:09:29.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.267+0000 7fc56e800700 1 -- 172.21.15.12:0/2388745924 reap_dead start 2024-03-29T10:09:29.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.268+0000 7fc574c00700 1 -- 172.21.15.12:0/2388745924 shutdown_connections 2024-03-29T10:09:29.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.268+0000 7fc574c00700 1 -- 172.21.15.12:0/2388745924 >> 172.21.15.12:0/2388745924 conn(0x7fc57008aff0 msgr2=0x7fc5700d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:29.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.268+0000 7fc574c00700 1 -- 172.21.15.12:0/2388745924 shutdown_connections 2024-03-29T10:09:29.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.268+0000 7fc574c00700 1 -- 172.21.15.12:0/2388745924 wait complete. 2024-03-29T10:09:29.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.270+0000 7fc574c00700 1 Processor -- start 2024-03-29T10:09:29.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.270+0000 7fc574c00700 1 -- start start 2024-03-29T10:09:29.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.271+0000 7fc574c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc554003a00 0x7fc5701556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.271+0000 7fc574c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5700b88c0 0x7fc5700b8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.271+0000 7fc574c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5700c97d0 0x7fc5700c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.271+0000 7fc574c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700ca170 con 0x7fc5700b88c0 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.271+0000 7fc574c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5701f3160 con 0x7fc5700c97d0 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.271+0000 7fc574c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc570155c30 con 0x7fc554003a00 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5700b88c0 0x7fc5700b8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5700b88c0 0x7fc5700b8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58054/0 (socket says 172.21.15.12:58054) 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56d400700 1 -- 172.21.15.12:0/2699075778 learned_addr learned my addr 172.21.15.12:0/2699075778 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56de00700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc554003a00 0x7fc5701556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56e800700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5700c97d0 0x7fc5700c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56d400700 1 -- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc554003a00 msgr2=0x7fc5701556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56d400700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc554003a00 0x7fc5701556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56d400700 1 -- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5700c97d0 msgr2=0x7fc5700c9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56d400700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5700c97d0 0x7fc5700c9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56d400700 1 -- 172.21.15.12:0/2699075778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5701f33b0 con 0x7fc5700b88c0 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc56d400700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5700b88c0 0x7fc5700b8d40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc55c007340 tx=0x7fc55c007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.272+0000 7fc566000700 1 -- 172.21.15.12:0/2699075778 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c00f6c0 con 0x7fc5700b88c0 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.273+0000 7fc566000700 1 -- 172.21.15.12:0/2699075778 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c00f840 con 0x7fc5700b88c0 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.273+0000 7fc566000700 1 -- 172.21.15.12:0/2699075778 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c00fbf0 con 0x7fc5700b88c0 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.273+0000 7fc574c00700 1 -- 172.21.15.12:0/2699075778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5701f3690 con 0x7fc5700b88c0 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.273+0000 7fc574c00700 1 -- 172.21.15.12:0/2699075778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5701f3c50 con 0x7fc5700b88c0 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.274+0000 7fc574c00700 1 -- 172.21.15.12:0/2699075778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc570067720 con 0x7fc5700b88c0 2024-03-29T10:09:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.275+0000 7fc566000700 1 -- 172.21.15.12:0/2699075778 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc55c00fd50 con 0x7fc5700b88c0 2024-03-29T10:09:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.276+0000 7fc566000700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc55006adb0 0x7fc55006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.276+0000 7fc566000700 1 -- 172.21.15.12:0/2699075778 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc55c089110 con 0x7fc5700b88c0 2024-03-29T10:09:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.276+0000 7fc56de00700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc55006adb0 0x7fc55006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.277+0000 7fc56de00700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc55006adb0 0x7fc55006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc5600090b0 tx=0x7fc560008030 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:29.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.280+0000 7fc566000700 1 -- 172.21.15.12:0/2699075778 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc55c054530 con 0x7fc5700b88c0 2024-03-29T10:09:29.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.493+0000 7fc574c00700 1 -- 172.21.15.12:0/2699075778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc57015b600 con 0x7fc5700b88c0 2024-03-29T10:09:29.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.493+0000 7fc566000700 1 -- 172.21.15.12:0/2699075778 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc55c057b50 con 0x7fc5700b88c0 2024-03-29T10:09:29.493 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:09:29.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.495+0000 7fc574c00700 1 -- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc55006adb0 msgr2=0x7fc55006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.495+0000 7fc574c00700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc55006adb0 0x7fc55006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc5600090b0 tx=0x7fc560008030 comp rx=0 tx=0).stop 2024-03-29T10:09:29.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.495+0000 7fc574c00700 1 -- 172.21.15.12:0/2699075778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5700b88c0 msgr2=0x7fc5700b8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.495+0000 7fc574c00700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5700b88c0 0x7fc5700b8d40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc55c007340 tx=0x7fc55c007650 comp rx=0 tx=0).stop 2024-03-29T10:09:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.495+0000 7fc574c00700 1 -- 172.21.15.12:0/2699075778 shutdown_connections 2024-03-29T10:09:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.495+0000 7fc574c00700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc55006adb0 0x7fc55006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.495+0000 7fc574c00700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5700c97d0 0x7fc5700c9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.495+0000 7fc574c00700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc554003a00 0x7fc5701556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.495+0000 7fc574c00700 1 --2- 172.21.15.12:0/2699075778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5700b88c0 0x7fc5700b8d40 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.495+0000 7fc574c00700 1 -- 172.21.15.12:0/2699075778 >> 172.21.15.12:0/2699075778 conn(0x7fc57008aff0 msgr2=0x7fc570159410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.496+0000 7fc574c00700 1 -- 172.21.15.12:0/2699075778 shutdown_connections 2024-03-29T10:09:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.496+0000 7fc574c00700 1 -- 172.21.15.12:0/2699075778 wait complete. 2024-03-29T10:09:29.513 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T10:09:29.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.711+0000 7f8f70e00700 1 Processor -- start 2024-03-29T10:09:29.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.711+0000 7f8f70e00700 1 -- start start 2024-03-29T10:09:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.712+0000 7f8f70e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8f6c15b5d0 0x7f8f6c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.712+0000 7f8f70e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c15fd50 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.713+0000 7f8f70e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c15f140 con 0x7f8f6c0cb3f0 2024-03-29T10:09:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.713+0000 7f8f70e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f6c15b9b0 con 0x7f8f6c15b5d0 2024-03-29T10:09:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.713+0000 7f8f69e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8f6c15b5d0 0x7f8f6c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.713+0000 7f8f6a800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8f6c0ac7d0 0x7f8f6c06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33114/0 (socket says 172.21.15.12:33114) 2024-03-29T10:09:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.713+0000 7f8f6a800700 1 -- 172.21.15.12:0/2226116147 learned_addr learned my addr 172.21.15.12:0/2226116147 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.713+0000 7f8f69e00700 1 -- 172.21.15.12:0/2226116147 >> v1:172.21.15.159:6789/0 conn(0x7f8f6c0cb3f0 legacy=0x7f8f6c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.713+0000 7f8f69e00700 1 -- 172.21.15.12:0/2226116147 >> v1:172.21.15.12:6789/0 conn(0x7f8f6c0ac7d0 legacy=0x7f8f6c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.713+0000 7f8f69e00700 1 -- 172.21.15.12:0/2226116147 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c15f420 con 0x7f8f6c15b5d0 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.713+0000 7f8f69e00700 1 --2- 172.21.15.12:0/2226116147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c15b5d0 0x7f8f6c0ae3f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009840 tx=0x7f8f5c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1479cd4e3fbb55 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.714+0000 7f8f68a00700 1 -- 172.21.15.12:0/2226116147 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c011830 con 0x7f8f6c15b5d0 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.714+0000 7f8f68a00700 1 -- 172.21.15.12:0/2226116147 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c011990 con 0x7f8f6c15b5d0 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.714+0000 7f8f68a00700 1 -- 172.21.15.12:0/2226116147 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c011d20 con 0x7f8f6c15b5d0 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.714+0000 7f8f70e00700 1 -- 172.21.15.12:0/2226116147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c15b5d0 msgr2=0x7f8f6c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.714+0000 7f8f70e00700 1 --2- 172.21.15.12:0/2226116147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c15b5d0 0x7f8f6c0ae3f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009840 tx=0x7f8f5c009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.714+0000 7f8f70e00700 1 -- 172.21.15.12:0/2226116147 shutdown_connections 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.714+0000 7f8f70e00700 1 --2- 172.21.15.12:0/2226116147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c15b5d0 0x7f8f6c0ae3f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.714+0000 7f8f70e00700 1 -- 172.21.15.12:0/2226116147 >> 172.21.15.12:0/2226116147 conn(0x7f8f6c0cf190 msgr2=0x7f8f6c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.714+0000 7f8f70e00700 1 -- 172.21.15.12:0/2226116147 shutdown_connections 2024-03-29T10:09:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.714+0000 7f8f70e00700 1 -- 172.21.15.12:0/2226116147 wait complete. 2024-03-29T10:09:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.717+0000 7f8f70e00700 1 Processor -- start 2024-03-29T10:09:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.717+0000 7f8f70e00700 1 -- start start 2024-03-29T10:09:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.718+0000 7f8f70e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c0ac7d0 0x7f8f6c144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.718+0000 7f8f70e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f6c15b5d0 0x7f8f6c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.718+0000 7f8f70e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f6c0b3480 0x7f8f6c0985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.718+0000 7f8f70e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0cc2a0 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.718+0000 7f8f70e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0cc580 con 0x7f8f6c15b5d0 2024-03-29T10:09:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.718+0000 7f8f70e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0cc860 con 0x7f8f6c0b3480 2024-03-29T10:09:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.718+0000 7f8f69e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c0ac7d0 0x7f8f6c144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.718+0000 7f8f69e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c0ac7d0 0x7f8f6c144800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58068/0 (socket says 172.21.15.12:58068) 2024-03-29T10:09:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f69e00700 1 -- 172.21.15.12:0/2594726533 learned_addr learned my addr 172.21.15.12:0/2594726533 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f69400700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f6c15b5d0 0x7f8f6c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f6a800700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f6c0b3480 0x7f8f6c0985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f69e00700 1 -- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f6c0b3480 msgr2=0x7f8f6c0985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f69e00700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f6c0b3480 0x7f8f6c0985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f69e00700 1 -- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f6c15b5d0 msgr2=0x7f8f6c0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f69e00700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f6c15b5d0 0x7f8f6c0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f69e00700 1 -- 172.21.15.12:0/2594726533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0094a0 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f6a800700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f6c0b3480 0x7f8f6c0985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f69e00700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c0ac7d0 0x7f8f6c144800 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c000c00 tx=0x7f8f5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f62000700 1 -- 172.21.15.12:0/2594726533 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0038a0 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.719+0000 7f8f62000700 1 -- 172.21.15.12:0/2594726533 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c01e070 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.720+0000 7f8f62000700 1 -- 172.21.15.12:0/2594726533 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c003c60 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.720+0000 7f8f70e00700 1 -- 172.21.15.12:0/2594726533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c0ccb40 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.720+0000 7f8f70e00700 1 -- 172.21.15.12:0/2594726533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c0b3900 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.721+0000 7f8f70e00700 1 -- 172.21.15.12:0/2594726533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c0cba10 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.722+0000 7f8f62000700 1 -- 172.21.15.12:0/2594726533 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8f5c02d450 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.722+0000 7f8f62000700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8f4c06adb0 0x7f8f4c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.723+0000 7f8f62000700 1 -- 172.21.15.12:0/2594726533 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8f5c08a260 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.723+0000 7f8f69400700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8f4c06adb0 0x7f8f4c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.724+0000 7f8f69400700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8f4c06adb0 0x7f8f4c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0b47f0 tx=0x7f8f58009040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.727+0000 7f8f62000700 1 -- 172.21.15.12:0/2594726533 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f5c055670 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.943+0000 7f8f70e00700 1 -- 172.21.15.12:0/2594726533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8f6c067720 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.944+0000 7f8f62000700 1 -- 172.21.15.12:0/2594726533 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8f5c088e60 con 0x7f8f6c0ac7d0 2024-03-29T10:09:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:29.944 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:09:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T10:09:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T10:09:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:29.944 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:09:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.946+0000 7f8f70e00700 1 -- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8f4c06adb0 msgr2=0x7f8f4c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.946+0000 7f8f70e00700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8f4c06adb0 0x7f8f4c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0b47f0 tx=0x7f8f58009040 comp rx=0 tx=0).stop 2024-03-29T10:09:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.946+0000 7f8f70e00700 1 -- 172.21.15.12:0/2594726533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c0ac7d0 msgr2=0x7f8f6c144800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.946+0000 7f8f70e00700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c0ac7d0 0x7f8f6c144800 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c000c00 tx=0x7f8f5c004750 comp rx=0 tx=0).stop 2024-03-29T10:09:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.947+0000 7f8f70e00700 1 -- 172.21.15.12:0/2594726533 shutdown_connections 2024-03-29T10:09:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.947+0000 7f8f70e00700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8f4c06adb0 0x7f8f4c06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.947+0000 7f8f70e00700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f6c0b3480 0x7f8f6c0985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.947+0000 7f8f70e00700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f6c0ac7d0 0x7f8f6c144800 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.947+0000 7f8f70e00700 1 --2- 172.21.15.12:0/2594726533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f6c15b5d0 0x7f8f6c0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.947+0000 7f8f70e00700 1 -- 172.21.15.12:0/2594726533 >> 172.21.15.12:0/2594726533 conn(0x7f8f6c0cf190 msgr2=0x7f8f6c15e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.947+0000 7f8f70e00700 1 -- 172.21.15.12:0/2594726533 shutdown_connections 2024-03-29T10:09:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:29.947+0000 7f8f70e00700 1 -- 172.21.15.12:0/2594726533 wait complete. 2024-03-29T10:09:29.965 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T10:09:29.965 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T10:09:30.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.168+0000 7fbe40400700 1 Processor -- start 2024-03-29T10:09:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.168+0000 7fbe40400700 1 -- start start 2024-03-29T10:09:30.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.169+0000 7fbe40400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbe380d28e0 0x7fbe38168680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:30.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.169+0000 7fbe40400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380ce8a0 con 0x7fbe3815ddf0 2024-03-29T10:09:30.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.169+0000 7fbe40400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380cdc90 con 0x7fbe380ced30 2024-03-29T10:09:30.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.169+0000 7fbe40400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe3815ec60 con 0x7fbe380d28e0 2024-03-29T10:09:30.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.169+0000 7fbe3da00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbe3815ddf0 0x7fbe3815e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33122/0 (socket says 172.21.15.12:33122) 2024-03-29T10:09:30.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.169+0000 7fbe3da00700 1 -- 172.21.15.12:0/382672225 learned_addr learned my addr 172.21.15.12:0/382672225 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.170+0000 7fbe3e400700 1 --2- 172.21.15.12:0/382672225 >> v2:172.21.15.159:3300/0 conn(0x7fbe380d28e0 0x7fbe38168680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.170+0000 7fbe37e00700 1 -- 172.21.15.12:0/382672225 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611274469 0 0) 0x7fbe380ce8a0 con 0x7fbe3815ddf0 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.170+0000 7fbe37e00700 1 -- 172.21.15.12:0/382672225 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe200035b0 con 0x7fbe3815ddf0 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.170+0000 7fbe37e00700 1 -- 172.21.15.12:0/382672225 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003591780 0 0) 0x7fbe380cdc90 con 0x7fbe380ced30 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.170+0000 7fbe37e00700 1 -- 172.21.15.12:0/382672225 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe380ce8a0 con 0x7fbe380ced30 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.170+0000 7fbe3e400700 1 -- 172.21.15.12:0/382672225 >> v1:172.21.15.159:6789/0 conn(0x7fbe380ced30 legacy=0x7fbe380d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.170+0000 7fbe37e00700 1 -- 172.21.15.12:0/382672225 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1364095290 0 0) 0x7fbe200035b0 con 0x7fbe3815ddf0 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.170+0000 7fbe3e400700 1 -- 172.21.15.12:0/382672225 >> v1:172.21.15.12:6789/0 conn(0x7fbe3815ddf0 legacy=0x7fbe3815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.170+0000 7fbe3e400700 1 -- 172.21.15.12:0/382672225 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe380cdf70 con 0x7fbe380d28e0 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.171+0000 7fbe3e400700 1 --2- 172.21.15.12:0/382672225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe380d28e0 0x7fbe38168680 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009d90 tx=0x7fbe3000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be0a945ed8fe5096 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:30.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.171+0000 7fbe37e00700 1 -- 172.21.15.12:0/382672225 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000e070 con 0x7fbe380d28e0 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.171+0000 7fbe37e00700 1 -- 172.21.15.12:0/382672225 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe30003a20 con 0x7fbe380d28e0 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.171+0000 7fbe37e00700 1 -- 172.21.15.12:0/382672225 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c001670 con 0x7fbe380d28e0 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.171+0000 7fbe40400700 1 -- 172.21.15.12:0/382672225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe380d28e0 msgr2=0x7fbe38168680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.171+0000 7fbe40400700 1 --2- 172.21.15.12:0/382672225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe380d28e0 0x7fbe38168680 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009d90 tx=0x7fbe3000b330 comp rx=0 tx=0).stop 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.171+0000 7fbe40400700 1 -- 172.21.15.12:0/382672225 shutdown_connections 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.171+0000 7fbe40400700 1 --2- 172.21.15.12:0/382672225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe380d28e0 0x7fbe38168680 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.171+0000 7fbe40400700 1 -- 172.21.15.12:0/382672225 >> 172.21.15.12:0/382672225 conn(0x7fbe3808aff0 msgr2=0x7fbe380d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.172+0000 7fbe40400700 1 -- 172.21.15.12:0/382672225 shutdown_connections 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.172+0000 7fbe40400700 1 -- 172.21.15.12:0/382672225 wait complete. 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.173+0000 7fbe40400700 1 Processor -- start 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.173+0000 7fbe40400700 1 -- start start 2024-03-29T10:09:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.174+0000 7fbe40400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe380ced30 0x7fbe380c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.174+0000 7fbe40400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe3815ddf0 0x7fbe380c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.174+0000 7fbe40400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe380b3e10 0x7fbe381f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.174+0000 7fbe40400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380c8580 con 0x7fbe380b3e10 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.174+0000 7fbe40400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380c7970 con 0x7fbe3815ddf0 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.174+0000 7fbe40400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380c7c50 con 0x7fbe380ced30 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe380b3e10 0x7fbe381f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe380b3e10 0x7fbe381f24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58076/0 (socket says 172.21.15.12:58076) 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3e400700 1 -- 172.21.15.12:0/237533015 learned_addr learned my addr 172.21.15.12:0/237533015 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3d000700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe3815ddf0 0x7fbe380c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3da00700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe380ced30 0x7fbe380c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3e400700 1 -- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe380ced30 msgr2=0x7fbe380c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3e400700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe380ced30 0x7fbe380c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3e400700 1 -- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe3815ddf0 msgr2=0x7fbe380c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3e400700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe3815ddf0 0x7fbe380c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3e400700 1 -- 172.21.15.12:0/237533015 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30009a70 con 0x7fbe380b3e10 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.175+0000 7fbe3e400700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe380b3e10 0x7fbe381f24a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000b370 tx=0x7fbe30004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.176+0000 7fbe35600700 1 -- 172.21.15.12:0/237533015 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30004440 con 0x7fbe380b3e10 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.176+0000 7fbe35600700 1 -- 172.21.15.12:0/237533015 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe300045a0 con 0x7fbe380b3e10 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.176+0000 7fbe35600700 1 -- 172.21.15.12:0/237533015 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30004950 con 0x7fbe380b3e10 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.176+0000 7fbe40400700 1 -- 172.21.15.12:0/237533015 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe380c7f30 con 0x7fbe380b3e10 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.176+0000 7fbe40400700 1 -- 172.21.15.12:0/237533015 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe380c7560 con 0x7fbe380b3e10 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.177+0000 7fbe40400700 1 -- 172.21.15.12:0/237533015 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe380c76f0 con 0x7fbe380b3e10 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.178+0000 7fbe35600700 1 -- 172.21.15.12:0/237533015 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbe3001d470 con 0x7fbe380b3e10 2024-03-29T10:09:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.178+0000 7fbe35600700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbe1806adb0 0x7fbe1806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:30.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.179+0000 7fbe35600700 1 -- 172.21.15.12:0/237533015 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbe3008aef0 con 0x7fbe380b3e10 2024-03-29T10:09:30.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.179+0000 7fbe3da00700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbe1806adb0 0x7fbe1806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:30.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.180+0000 7fbe3da00700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbe1806adb0 0x7fbe1806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c000fd0 tx=0x7fbe2c001690 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:30.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.183+0000 7fbe35600700 1 -- 172.21.15.12:0/237533015 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbe30059c80 con 0x7fbe380b3e10 2024-03-29T10:09:30.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:30.399+0000 7fbe40400700 1 -- 172.21.15.12:0/237533015 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbe38067720 con 0x7fbe380b3e10 2024-03-29T10:09:31.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.069+0000 7fbe35600700 1 -- 172.21.15.12:0/237533015 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbe30059810 con 0x7fbe380b3e10 2024-03-29T10:09:31.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.074+0000 7fbe40400700 1 -- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbe1806adb0 msgr2=0x7fbe1806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:31.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.074+0000 7fbe40400700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbe1806adb0 0x7fbe1806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c000fd0 tx=0x7fbe2c001690 comp rx=0 tx=0).stop 2024-03-29T10:09:31.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.074+0000 7fbe40400700 1 -- 172.21.15.12:0/237533015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe380b3e10 msgr2=0x7fbe381f24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:31.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.074+0000 7fbe40400700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe380b3e10 0x7fbe381f24a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000b370 tx=0x7fbe30004ad0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.074+0000 7fbe40400700 1 -- 172.21.15.12:0/237533015 shutdown_connections 2024-03-29T10:09:31.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.074+0000 7fbe40400700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbe1806adb0 0x7fbe1806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.074+0000 7fbe40400700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe380b3e10 0x7fbe381f24a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.074+0000 7fbe40400700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe380ced30 0x7fbe380c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.074+0000 7fbe40400700 1 --2- 172.21.15.12:0/237533015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe3815ddf0 0x7fbe380c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.074+0000 7fbe40400700 1 -- 172.21.15.12:0/237533015 >> 172.21.15.12:0/237533015 conn(0x7fbe3808aff0 msgr2=0x7fbe380b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:31.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.076+0000 7fbe40400700 1 -- 172.21.15.12:0/237533015 shutdown_connections 2024-03-29T10:09:31.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.077+0000 7fbe40400700 1 -- 172.21.15.12:0/237533015 wait complete. 2024-03-29T10:09:31.078 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T10:09:31.098 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T10:09:31.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.308+0000 7f8e9aa00700 1 Processor -- start 2024-03-29T10:09:31.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.309+0000 7f8e9aa00700 1 -- start start 2024-03-29T10:09:31.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.310+0000 7f8e9aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8e9415d9d0 0x7f8e940d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:31.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.310+0000 7f8e9aa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8e940d16a0 0x7f8e940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:31.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.310+0000 7f8e9aa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e94161420 con 0x7f8e940d2370 2024-03-29T10:09:31.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.310+0000 7f8e9aa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e940d27e0 con 0x7f8e9415d9d0 2024-03-29T10:09:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.310+0000 7f8e9aa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e94161700 con 0x7f8e940d16a0 2024-03-29T10:09:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.310+0000 7f8e93e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8e9415d9d0 0x7f8e940d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.310+0000 7f8e93e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8e9415d9d0 0x7f8e940d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58092/0 (socket says 172.21.15.12:58092) 2024-03-29T10:09:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.310+0000 7f8e93e00700 1 -- 172.21.15.12:0/2675333018 learned_addr learned my addr 172.21.15.12:0/2675333018 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.310+0000 7f8e93400700 1 --2- 172.21.15.12:0/2675333018 >> v2:172.21.15.159:3300/0 conn(0x7f8e940d16a0 0x7f8e940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.311+0000 7f8e93e00700 1 -- 172.21.15.12:0/2675333018 >> v2:172.21.15.159:3300/0 conn(0x7f8e940d16a0 msgr2=0x7f8e940d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.311+0000 7f8e93e00700 1 --2- 172.21.15.12:0/2675333018 >> v2:172.21.15.159:3300/0 conn(0x7f8e940d16a0 0x7f8e940d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.311+0000 7f8e93e00700 1 -- 172.21.15.12:0/2675333018 >> v1:172.21.15.159:6789/0 conn(0x7f8e940d2370 legacy=0x7f8e9415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.311+0000 7f8e93e00700 1 -- 172.21.15.12:0/2675333018 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e941619e0 con 0x7f8e9415d9d0 2024-03-29T10:09:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.311+0000 7f8e93e00700 1 --2- 172.21.15.12:0/2675333018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 0x7f8e940d3c30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8e940d12f0 tx=0x7f8e84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c75983b66e4d28b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.311+0000 7f8e92a00700 1 -- 172.21.15.12:0/2675333018 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e84011830 con 0x7f8e9415d9d0 2024-03-29T10:09:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.311+0000 7f8e92a00700 1 -- 172.21.15.12:0/2675333018 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e84011990 con 0x7f8e9415d9d0 2024-03-29T10:09:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.311+0000 7f8e92a00700 1 -- 172.21.15.12:0/2675333018 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e84011d20 con 0x7f8e9415d9d0 2024-03-29T10:09:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.312+0000 7f8e9aa00700 1 -- 172.21.15.12:0/2675333018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 msgr2=0x7f8e940d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.312+0000 7f8e9aa00700 1 --2- 172.21.15.12:0/2675333018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 0x7f8e940d3c30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8e940d12f0 tx=0x7f8e84009bd0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.312+0000 7f8e9aa00700 1 -- 172.21.15.12:0/2675333018 shutdown_connections 2024-03-29T10:09:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.312+0000 7f8e9aa00700 1 --2- 172.21.15.12:0/2675333018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 0x7f8e940d3c30 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.312+0000 7f8e9aa00700 1 --2- 172.21.15.12:0/2675333018 >> v2:172.21.15.159:3300/0 conn(0x7f8e940d16a0 0x7f8e940d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.312+0000 7f8e9aa00700 1 -- 172.21.15.12:0/2675333018 >> 172.21.15.12:0/2675333018 conn(0x7f8e9408aff0 msgr2=0x7f8e940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.312+0000 7f8e9aa00700 1 -- 172.21.15.12:0/2675333018 shutdown_connections 2024-03-29T10:09:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.312+0000 7f8e9aa00700 1 -- 172.21.15.12:0/2675333018 wait complete. 2024-03-29T10:09:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.314+0000 7f8e9aa00700 1 Processor -- start 2024-03-29T10:09:31.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.314+0000 7f8e9aa00700 1 -- start start 2024-03-29T10:09:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e9aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e940d16a0 0x7f8e940b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e9aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e940d2370 0x7f8e941580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e9aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 0x7f8e940b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e9aa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940b6fd0 con 0x7f8e9415d9d0 2024-03-29T10:09:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e9aa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940b79a0 con 0x7f8e940d2370 2024-03-29T10:09:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e9aa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940b63c0 con 0x7f8e940d16a0 2024-03-29T10:09:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e98a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 0x7f8e940b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e98a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 0x7f8e940b8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58096/0 (socket says 172.21.15.12:58096) 2024-03-29T10:09:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e98a00700 1 -- 172.21.15.12:0/1574625399 learned_addr learned my addr 172.21.15.12:0/1574625399 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e93400700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e940d2370 0x7f8e941580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.316+0000 7f8e93e00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e940d16a0 0x7f8e940b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e98a00700 1 -- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e940d16a0 msgr2=0x7f8e940b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e98a00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e940d16a0 0x7f8e940b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e98a00700 1 -- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e940d2370 msgr2=0x7f8e941580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e98a00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e940d2370 0x7f8e941580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e98a00700 1 -- 172.21.15.12:0/1574625399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e84009550 con 0x7f8e9415d9d0 2024-03-29T10:09:31.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e98a00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 0x7f8e940b8250 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00cc50 tx=0x7f8e8c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:31.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e8be00700 1 -- 172.21.15.12:0/1574625399 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c003c00 con 0x7f8e9415d9d0 2024-03-29T10:09:31.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e8be00700 1 -- 172.21.15.12:0/1574625399 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8c003d60 con 0x7f8e9415d9d0 2024-03-29T10:09:31.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e9aa00700 1 -- 172.21.15.12:0/1574625399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e940b66a0 con 0x7f8e9415d9d0 2024-03-29T10:09:31.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e8be00700 1 -- 172.21.15.12:0/1574625399 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c00aab0 con 0x7f8e9415d9d0 2024-03-29T10:09:31.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.317+0000 7f8e9aa00700 1 -- 172.21.15.12:0/1574625399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e940b6c60 con 0x7f8e9415d9d0 2024-03-29T10:09:31.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.318+0000 7f8e9aa00700 1 -- 172.21.15.12:0/1574625399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e940d1ca0 con 0x7f8e9415d9d0 2024-03-29T10:09:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.320+0000 7f8e8be00700 1 -- 172.21.15.12:0/1574625399 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8e8c00ac10 con 0x7f8e9415d9d0 2024-03-29T10:09:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.320+0000 7f8e8be00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e7406adb0 0x7f8e7406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.320+0000 7f8e8be00700 1 -- 172.21.15.12:0/1574625399 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8e8c0892c0 con 0x7f8e9415d9d0 2024-03-29T10:09:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.321+0000 7f8e93e00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e7406adb0 0x7f8e7406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:31.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.322+0000 7f8e93e00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e7406adb0 0x7f8e7406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8e84014040 tx=0x7f8e84008030 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:31.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.326+0000 7f8e8be00700 1 -- 172.21.15.12:0/1574625399 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8e8c0544d0 con 0x7f8e9415d9d0 2024-03-29T10:09:31.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:31.539+0000 7f8e9aa00700 1 -- 172.21.15.12:0/1574625399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8e9406b9b0 con 0x7f8e9415d9d0 2024-03-29T10:09:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.074+0000 7f8e8be00700 1 -- 172.21.15.12:0/1574625399 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v78) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8e8c057af0 con 0x7f8e9415d9d0 2024-03-29T10:09:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.083+0000 7f8e9aa00700 1 -- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e7406adb0 msgr2=0x7f8e7406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.083+0000 7f8e9aa00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e7406adb0 0x7f8e7406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8e84014040 tx=0x7f8e84008030 comp rx=0 tx=0).stop 2024-03-29T10:09:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.083+0000 7f8e9aa00700 1 -- 172.21.15.12:0/1574625399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 msgr2=0x7f8e940b8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.083+0000 7f8e9aa00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 0x7f8e940b8250 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00cc50 tx=0x7f8e8c00cf60 comp rx=0 tx=0).stop 2024-03-29T10:09:32.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.085+0000 7f8e9aa00700 1 -- 172.21.15.12:0/1574625399 shutdown_connections 2024-03-29T10:09:32.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.085+0000 7f8e9aa00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e7406adb0 0x7f8e7406d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.085+0000 7f8e9aa00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e9415d9d0 0x7f8e940b8250 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.085+0000 7f8e9aa00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e940d16a0 0x7f8e940b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.085+0000 7f8e9aa00700 1 --2- 172.21.15.12:0/1574625399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e940d2370 0x7f8e941580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.085+0000 7f8e9aa00700 1 -- 172.21.15.12:0/1574625399 >> 172.21.15.12:0/1574625399 conn(0x7f8e9408aff0 msgr2=0x7f8e9415bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:32.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.086+0000 7f8e9aa00700 1 -- 172.21.15.12:0/1574625399 shutdown_connections 2024-03-29T10:09:32.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.087+0000 7f8e9aa00700 1 -- 172.21.15.12:0/1574625399 wait complete. 2024-03-29T10:09:32.089 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T10:09:32.115 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T10:09:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.318+0000 7f2a42e00700 1 Processor -- start 2024-03-29T10:09:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.318+0000 7f2a42e00700 1 -- start start 2024-03-29T10:09:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.319+0000 7f2a42e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2a3c0d2540 0x7f2a3c1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.319+0000 7f2a42e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a3c161420 con 0x7f2a3c0d16a0 2024-03-29T10:09:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.319+0000 7f2a42e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a3c161700 con 0x7f2a3c15d9d0 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.319+0000 7f2a42e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a3c0d4340 con 0x7f2a3c0d2540 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.320+0000 7f2a3b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2a3c0d16a0 0x7f2a3c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33130/0 (socket says 172.21.15.12:33130) 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.320+0000 7f2a3b400700 1 -- 172.21.15.12:0/1299691791 learned_addr learned my addr 172.21.15.12:0/1299691791 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.320+0000 7f2a40e00700 1 --2- 172.21.15.12:0/1299691791 >> v2:172.21.15.12:3300/0 conn(0x7f2a3c0d2540 0x7f2a3c1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.320+0000 7f2a3aa00700 1 -- 172.21.15.12:0/1299691791 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784480696 0 0) 0x7f2a3c161420 con 0x7f2a3c0d16a0 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.320+0000 7f2a3aa00700 1 -- 172.21.15.12:0/1299691791 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a200035b0 con 0x7f2a3c0d16a0 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.320+0000 7f2a40e00700 1 -- 172.21.15.12:0/1299691791 >> v1:172.21.15.159:6789/0 conn(0x7f2a3c15d9d0 legacy=0x7f2a3c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.320+0000 7f2a3aa00700 1 -- 172.21.15.12:0/1299691791 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108972510 0 0) 0x7f2a3c161700 con 0x7f2a3c15d9d0 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.320+0000 7f2a40e00700 1 -- 172.21.15.12:0/1299691791 >> v1:172.21.15.12:6789/0 conn(0x7f2a3c0d16a0 legacy=0x7f2a3c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.320+0000 7f2a40e00700 1 -- 172.21.15.12:0/1299691791 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c1619e0 con 0x7f2a3c0d2540 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a40e00700 1 --2- 172.21.15.12:0/1299691791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 0x7f2a3c1687b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2a34009d90 tx=0x7f2a3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47fb9e4690bb2450 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a3aa00700 1 -- 172.21.15.12:0/1299691791 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3400e070 con 0x7f2a3c0d2540 2024-03-29T10:09:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a3aa00700 1 -- 172.21.15.12:0/1299691791 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3400bb20 con 0x7f2a3c0d2540 2024-03-29T10:09:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a3aa00700 1 -- 172.21.15.12:0/1299691791 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3400bed0 con 0x7f2a3c0d2540 2024-03-29T10:09:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a42e00700 1 -- 172.21.15.12:0/1299691791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 msgr2=0x7f2a3c1687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a42e00700 1 --2- 172.21.15.12:0/1299691791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 0x7f2a3c1687b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2a34009d90 tx=0x7f2a3400b330 comp rx=0 tx=0).stop 2024-03-29T10:09:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a42e00700 1 -- 172.21.15.12:0/1299691791 shutdown_connections 2024-03-29T10:09:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a42e00700 1 --2- 172.21.15.12:0/1299691791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 0x7f2a3c1687b0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a42e00700 1 -- 172.21.15.12:0/1299691791 >> 172.21.15.12:0/1299691791 conn(0x7f2a3c08aff0 msgr2=0x7f2a3c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a42e00700 1 -- 172.21.15.12:0/1299691791 shutdown_connections 2024-03-29T10:09:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.321+0000 7f2a42e00700 1 -- 172.21.15.12:0/1299691791 wait complete. 2024-03-29T10:09:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.323+0000 7f2a42e00700 1 Processor -- start 2024-03-29T10:09:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.323+0000 7f2a42e00700 1 -- start start 2024-03-29T10:09:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.324+0000 7f2a42e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0d16a0 0x7f2a3c0cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.324+0000 7f2a42e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 0x7f2a3c0cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.324+0000 7f2a42e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c15d9d0 0x7f2a3c0cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.324+0000 7f2a42e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0b7210 con 0x7f2a3c0d2540 2024-03-29T10:09:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.324+0000 7f2a42e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0ca8f0 con 0x7f2a3c0d16a0 2024-03-29T10:09:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.324+0000 7f2a42e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0b6590 con 0x7f2a3c15d9d0 2024-03-29T10:09:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.324+0000 7f2a3b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 0x7f2a3c0cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a3b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 0x7f2a3c0cbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58126/0 (socket says 172.21.15.12:58126) 2024-03-29T10:09:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a3b400700 1 -- 172.21.15.12:0/3263322808 learned_addr learned my addr 172.21.15.12:0/3263322808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a3be00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0d16a0 0x7f2a3c0cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a40e00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c15d9d0 0x7f2a3c0cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a3b400700 1 -- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c15d9d0 msgr2=0x7f2a3c0cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a3b400700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c15d9d0 0x7f2a3c0cc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a3b400700 1 -- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0d16a0 msgr2=0x7f2a3c0cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a3b400700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0d16a0 0x7f2a3c0cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a3b400700 1 -- 172.21.15.12:0/3263322808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a34009a70 con 0x7f2a3c0d2540 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a3be00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0d16a0 0x7f2a3c0cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a3b400700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 0x7f2a3c0cbb40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2a2800c730 tx=0x7f2a2800caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a33e00700 1 -- 172.21.15.12:0/3263322808 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a280074e0 con 0x7f2a3c0d2540 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a33e00700 1 -- 172.21.15.12:0/3263322808 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a28007640 con 0x7f2a3c0d2540 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.325+0000 7f2a42e00700 1 -- 172.21.15.12:0/3263322808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3c0b6870 con 0x7f2a3c0d2540 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.326+0000 7f2a33e00700 1 -- 172.21.15.12:0/3263322808 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a28005a50 con 0x7f2a3c0d2540 2024-03-29T10:09:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.326+0000 7f2a42e00700 1 -- 172.21.15.12:0/3263322808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3c0b6e30 con 0x7f2a3c0d2540 2024-03-29T10:09:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.327+0000 7f2a42e00700 1 -- 172.21.15.12:0/3263322808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3c0d1ca0 con 0x7f2a3c0d2540 2024-03-29T10:09:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.328+0000 7f2a33e00700 1 -- 172.21.15.12:0/3263322808 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2a28005bb0 con 0x7f2a3c0d2540 2024-03-29T10:09:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.329+0000 7f2a33e00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2a1c06adb0 0x7f2a1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.329+0000 7f2a33e00700 1 -- 172.21.15.12:0/3263322808 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2a280888e0 con 0x7f2a3c0d2540 2024-03-29T10:09:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.329+0000 7f2a3be00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2a1c06adb0 0x7f2a1c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.330+0000 7f2a3be00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2a1c06adb0 0x7f2a1c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c000f60 tx=0x7f2a2c009040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.334+0000 7f2a33e00700 1 -- 172.21.15.12:0/3263322808 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2a28057530 con 0x7f2a3c0d2540 2024-03-29T10:09:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.595+0000 7f2a42e00700 1 -- 172.21.15.12:0/3263322808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2a3c0b7350 con 0x7f2a3c0d2540 2024-03-29T10:09:32.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.600+0000 7f2a33e00700 1 -- 172.21.15.12:0/3263322808 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v28) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2a3c0b7350 con 0x7f2a3c0d2540 2024-03-29T10:09:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.607+0000 7f2a42e00700 1 -- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2a1c06adb0 msgr2=0x7f2a1c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.607+0000 7f2a42e00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2a1c06adb0 0x7f2a1c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c000f60 tx=0x7f2a2c009040 comp rx=0 tx=0).stop 2024-03-29T10:09:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.607+0000 7f2a42e00700 1 -- 172.21.15.12:0/3263322808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 msgr2=0x7f2a3c0cbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.607+0000 7f2a42e00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 0x7f2a3c0cbb40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2a2800c730 tx=0x7f2a2800caf0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.607+0000 7f2a42e00700 1 -- 172.21.15.12:0/3263322808 shutdown_connections 2024-03-29T10:09:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.607+0000 7f2a42e00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2a1c06adb0 0x7f2a1c06d270 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c000f60 tx=0x7f2a2c009040 comp rx=0 tx=0).stop 2024-03-29T10:09:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.607+0000 7f2a42e00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c15d9d0 0x7f2a3c0cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.607+0000 7f2a42e00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0d16a0 0x7f2a3c0cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.607+0000 7f2a42e00700 1 --2- 172.21.15.12:0/3263322808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a3c0d2540 0x7f2a3c0cbb40 secure :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2a2800c730 tx=0x7f2a2800caf0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.607+0000 7f2a42e00700 1 -- 172.21.15.12:0/3263322808 >> 172.21.15.12:0/3263322808 conn(0x7f2a3c08aff0 msgr2=0x7f2a3c0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:32.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.609+0000 7f2a42e00700 1 -- 172.21.15.12:0/3263322808 shutdown_connections 2024-03-29T10:09:32.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.610+0000 7f2a42e00700 1 -- 172.21.15.12:0/3263322808 wait complete. 2024-03-29T10:09:32.612 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 8 and data pool 9 2024-03-29T10:09:32.643 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:09:32.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.841+0000 7fb4cea00700 1 Processor -- start 2024-03-29T10:09:32.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.842+0000 7fb4cea00700 1 -- start start 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.843+0000 7fb4cea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb4c8160f10 0x7fb4c80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.843+0000 7fb4cea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c80cf230 0x7fb4c80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.843+0000 7fb4cea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c80ceda0 con 0x7fb4c8161590 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.843+0000 7fb4cea00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c80d23d0 con 0x7fb4c8160f10 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.843+0000 7fb4cea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c81612f0 con 0x7fb4c80cf230 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.843+0000 7fb4c7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb4c8160f10 0x7fb4c80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.843+0000 7fb4cca00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb4c8161590 0x7fb4c80b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33136/0 (socket says 172.21.15.12:33136) 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.843+0000 7fb4cca00700 1 -- 172.21.15.12:0/4101680871 learned_addr learned my addr 172.21.15.12:0/4101680871 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.843+0000 7fb4c7400700 1 --2- 172.21.15.12:0/4101680871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c80cf230 0x7fb4c80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.843+0000 7fb4c6a00700 1 -- 172.21.15.12:0/4101680871 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46543769 0 0) 0x7fb4c80ceda0 con 0x7fb4c8161590 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.844+0000 7fb4c7e00700 1 -- 172.21.15.12:0/4101680871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c80cf230 msgr2=0x7fb4c80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:32.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.844+0000 7fb4c7e00700 1 --2- 172.21.15.12:0/4101680871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c80cf230 0x7fb4c80d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.844+0000 7fb4c7e00700 1 -- 172.21.15.12:0/4101680871 >> v1:172.21.15.12:6789/0 conn(0x7fb4c8161590 legacy=0x7fb4c80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.844+0000 7fb4c7e00700 1 -- 172.21.15.12:0/4101680871 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c80ce470 con 0x7fb4c8160f10 2024-03-29T10:09:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.844+0000 7fb4c7400700 1 --2- 172.21.15.12:0/4101680871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c80cf230 0x7fb4c80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:09:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.844+0000 7fb4c7e00700 1 --2- 172.21.15.12:0/4101680871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 0x7fb4c80d18b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009840 tx=0x7fb4b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5117873a7ea482 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.844+0000 7fb4c6a00700 1 -- 172.21.15.12:0/4101680871 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8011830 con 0x7fb4c8160f10 2024-03-29T10:09:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.844+0000 7fb4c6a00700 1 -- 172.21.15.12:0/4101680871 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b8011990 con 0x7fb4c8160f10 2024-03-29T10:09:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.844+0000 7fb4c6a00700 1 -- 172.21.15.12:0/4101680871 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8011d20 con 0x7fb4c8160f10 2024-03-29T10:09:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.845+0000 7fb4cea00700 1 -- 172.21.15.12:0/4101680871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 msgr2=0x7fb4c80d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.845+0000 7fb4cea00700 1 --2- 172.21.15.12:0/4101680871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 0x7fb4c80d18b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009840 tx=0x7fb4b8009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.845+0000 7fb4cea00700 1 -- 172.21.15.12:0/4101680871 shutdown_connections 2024-03-29T10:09:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.845+0000 7fb4cea00700 1 --2- 172.21.15.12:0/4101680871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 0x7fb4c80d18b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.845+0000 7fb4cea00700 1 --2- 172.21.15.12:0/4101680871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c80cf230 0x7fb4c80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.845+0000 7fb4cea00700 1 -- 172.21.15.12:0/4101680871 >> 172.21.15.12:0/4101680871 conn(0x7fb4c808aff0 msgr2=0x7fb4c80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.845+0000 7fb4cea00700 1 -- 172.21.15.12:0/4101680871 shutdown_connections 2024-03-29T10:09:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.845+0000 7fb4cea00700 1 -- 172.21.15.12:0/4101680871 wait complete. 2024-03-29T10:09:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.847+0000 7fb4cea00700 1 Processor -- start 2024-03-29T10:09:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.847+0000 7fb4cea00700 1 -- start start 2024-03-29T10:09:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4cea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4c80cf230 0x7fb4c81f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4cea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 0x7fb4c81f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4cea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c8161590 0x7fb4c815ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4cea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c8158820 con 0x7fb4c8160f10 2024-03-29T10:09:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4cea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c8156380 con 0x7fb4c80cf230 2024-03-29T10:09:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4cea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c8157c10 con 0x7fb4c8161590 2024-03-29T10:09:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 0x7fb4c81f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 0x7fb4c81f3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58140/0 (socket says 172.21.15.12:58140) 2024-03-29T10:09:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4c7400700 1 -- 172.21.15.12:0/2256443169 learned_addr learned my addr 172.21.15.12:0/2256443169 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4c7e00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4c80cf230 0x7fb4c81f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4cca00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c8161590 0x7fb4c815ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4c7400700 1 -- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c8161590 msgr2=0x7fb4c815ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4c7400700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c8161590 0x7fb4c815ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4c7400700 1 -- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4c80cf230 msgr2=0x7fb4c81f6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4c7400700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4c80cf230 0x7fb4c81f6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.848+0000 7fb4c7400700 1 -- 172.21.15.12:0/2256443169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b80094a0 con 0x7fb4c8160f10 2024-03-29T10:09:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.849+0000 7fb4c7400700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 0x7fb4c81f3f80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400c7f0 tx=0x7fb4b400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.849+0000 7fb4bfe00700 1 -- 172.21.15.12:0/2256443169 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4011800 con 0x7fb4c8160f10 2024-03-29T10:09:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.849+0000 7fb4bfe00700 1 -- 172.21.15.12:0/2256443169 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b4011960 con 0x7fb4c8160f10 2024-03-29T10:09:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.849+0000 7fb4bfe00700 1 -- 172.21.15.12:0/2256443169 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4011cf0 con 0x7fb4c8160f10 2024-03-29T10:09:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.849+0000 7fb4cea00700 1 -- 172.21.15.12:0/2256443169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c8157ef0 con 0x7fb4c8160f10 2024-03-29T10:09:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.849+0000 7fb4cea00700 1 -- 172.21.15.12:0/2256443169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c81584b0 con 0x7fb4c8160f10 2024-03-29T10:09:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.850+0000 7fb4cea00700 1 -- 172.21.15.12:0/2256443169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c8158990 con 0x7fb4c8160f10 2024-03-29T10:09:32.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.851+0000 7fb4bfe00700 1 -- 172.21.15.12:0/2256443169 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb4b4018360 con 0x7fb4c8160f10 2024-03-29T10:09:32.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.853+0000 7fb4bfe00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb4a806adb0 0x7fb4a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:32.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.853+0000 7fb4bfe00700 1 -- 172.21.15.12:0/2256443169 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4b4089450 con 0x7fb4c8160f10 2024-03-29T10:09:32.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.853+0000 7fb4c7e00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb4a806adb0 0x7fb4a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:32.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.854+0000 7fb4c7e00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb4a806adb0 0x7fb4a806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80090b0 tx=0x7fb4b801c040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:32.857+0000 7fb4bfe00700 1 -- 172.21.15.12:0/2256443169 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb4b4054400 con 0x7fb4c8160f10 2024-03-29T10:09:33.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.073+0000 7fb4cea00700 1 -- 172.21.15.12:0/2256443169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4c8067720 con 0x7fb4c8160f10 2024-03-29T10:09:33.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.074+0000 7fb4bfe00700 1 -- 172.21.15.12:0/2256443169 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+16110 (secure 0 0 0) 0x7fb4b4057a20 con 0x7fb4c8160f10 2024-03-29T10:09:33.074 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:33.074 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":79,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:32.596801+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:09:30.441619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-29T10:09:31.581366+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:09:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.076+0000 7fb4cea00700 1 -- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb4a806adb0 msgr2=0x7fb4a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.076+0000 7fb4cea00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb4a806adb0 0x7fb4a806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80090b0 tx=0x7fb4b801c040 comp rx=0 tx=0).stop 2024-03-29T10:09:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.077+0000 7fb4cea00700 1 -- 172.21.15.12:0/2256443169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 msgr2=0x7fb4c81f3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.077+0000 7fb4cea00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 0x7fb4c81f3f80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400c7f0 tx=0x7fb4b400cb00 comp rx=0 tx=0).stop 2024-03-29T10:09:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.077+0000 7fb4cea00700 1 -- 172.21.15.12:0/2256443169 shutdown_connections 2024-03-29T10:09:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.077+0000 7fb4cea00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb4a806adb0 0x7fb4a806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.077+0000 7fb4cea00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4c8161590 0x7fb4c815ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.077+0000 7fb4cea00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4c80cf230 0x7fb4c81f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.077+0000 7fb4cea00700 1 --2- 172.21.15.12:0/2256443169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4c8160f10 0x7fb4c81f3f80 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.077+0000 7fb4cea00700 1 -- 172.21.15.12:0/2256443169 >> 172.21.15.12:0/2256443169 conn(0x7fb4c808aff0 msgr2=0x7fb4c80cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.077+0000 7fb4cea00700 1 -- 172.21.15.12:0/2256443169 shutdown_connections 2024-03-29T10:09:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.077+0000 7fb4cea00700 1 -- 172.21.15.12:0/2256443169 wait complete. 2024-03-29T10:09:33.096 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:09:33.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.295+0000 7fbdcf200700 1 Processor -- start 2024-03-29T10:09:33.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.295+0000 7fbdcf200700 1 -- start start 2024-03-29T10:09:33.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.296+0000 7fbdcf200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbdc80b3af0 0x7fbdc815a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.296+0000 7fbdcf200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbdc8072580 0x7fbdc80b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.296+0000 7fbdcf200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc80b2180 con 0x7fbdc80b2e50 2024-03-29T10:09:33.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.296+0000 7fbdcf200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdc815b100 con 0x7fbdc80b3af0 2024-03-29T10:09:33.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.296+0000 7fbdcf200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdc815b3e0 con 0x7fbdc8072580 2024-03-29T10:09:33.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.297+0000 7fbdc7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbdc80b3af0 0x7fbdc815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.297+0000 7fbdc7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbdc80b3af0 0x7fbdc815a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58154/0 (socket says 172.21.15.12:58154) 2024-03-29T10:09:33.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.297+0000 7fbdc7e00700 1 -- 172.21.15.12:0/2312594343 learned_addr learned my addr 172.21.15.12:0/2312594343 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:33.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.297+0000 7fbdc7400700 1 --2- 172.21.15.12:0/2312594343 >> v2:172.21.15.159:3300/0 conn(0x7fbdc8072580 0x7fbdc80b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.297+0000 7fbdc7e00700 1 -- 172.21.15.12:0/2312594343 >> v2:172.21.15.159:3300/0 conn(0x7fbdc8072580 msgr2=0x7fbdc80b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:33.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.297+0000 7fbdc7e00700 1 --2- 172.21.15.12:0/2312594343 >> v2:172.21.15.159:3300/0 conn(0x7fbdc8072580 0x7fbdc80b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.297+0000 7fbdc7e00700 1 -- 172.21.15.12:0/2312594343 >> v1:172.21.15.159:6789/0 conn(0x7fbdc80b2e50 legacy=0x7fbdc80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:33.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.297+0000 7fbdc7e00700 1 -- 172.21.15.12:0/2312594343 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc80b23f0 con 0x7fbdc80b3af0 2024-03-29T10:09:33.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.297+0000 7fbdc7e00700 1 --2- 172.21.15.12:0/2312594343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80b3af0 0x7fbdc815a860 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fbdc815dd90 tx=0x7fbdb8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b672160f209f6b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:33.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdc6a00700 1 -- 172.21.15.12:0/2312594343 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8004230 con 0x7fbdc80b3af0 2024-03-29T10:09:33.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdc6a00700 1 -- 172.21.15.12:0/2312594343 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb8004390 con 0x7fbdc80b3af0 2024-03-29T10:09:33.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdc6a00700 1 -- 172.21.15.12:0/2312594343 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb80036a0 con 0x7fbdc80b3af0 2024-03-29T10:09:33.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdcf200700 1 -- 172.21.15.12:0/2312594343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80b3af0 msgr2=0x7fbdc815a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:33.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdcf200700 1 --2- 172.21.15.12:0/2312594343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80b3af0 0x7fbdc815a860 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fbdc815dd90 tx=0x7fbdb8009bd0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdcf200700 1 -- 172.21.15.12:0/2312594343 shutdown_connections 2024-03-29T10:09:33.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdcf200700 1 --2- 172.21.15.12:0/2312594343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80b3af0 0x7fbdc815a860 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdcf200700 1 --2- 172.21.15.12:0/2312594343 >> v2:172.21.15.159:3300/0 conn(0x7fbdc8072580 0x7fbdc80b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdcf200700 1 -- 172.21.15.12:0/2312594343 >> 172.21.15.12:0/2312594343 conn(0x7fbdc80cf190 msgr2=0x7fbdc80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:33.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdcf200700 1 -- 172.21.15.12:0/2312594343 shutdown_connections 2024-03-29T10:09:33.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.298+0000 7fbdcf200700 1 -- 172.21.15.12:0/2312594343 wait complete. 2024-03-29T10:09:33.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.300+0000 7fbdcf200700 1 Processor -- start 2024-03-29T10:09:33.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.301+0000 7fbdcf200700 1 -- start start 2024-03-29T10:09:33.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.301+0000 7fbdcf200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc8072580 0x7fbdc81f6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.301+0000 7fbdcf200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdc80b2e50 0x7fbdc814bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.301+0000 7fbdcf200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdc80b3af0 0x7fbdc8071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdcf200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8158450 con 0x7fbdc8072580 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdcf200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc81f6a90 con 0x7fbdc80b2e50 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdcf200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80ce010 con 0x7fbdc80b3af0 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdc7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc8072580 0x7fbdc81f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdc7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc8072580 0x7fbdc81f6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58166/0 (socket says 172.21.15.12:58166) 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdc7e00700 1 -- 172.21.15.12:0/2298777465 learned_addr learned my addr 172.21.15.12:0/2298777465 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdc7400700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdc80b2e50 0x7fbdc814bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdcd200700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdc80b3af0 0x7fbdc8071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdc7e00700 1 -- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdc80b3af0 msgr2=0x7fbdc8071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdc7e00700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdc80b3af0 0x7fbdc8071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdc7e00700 1 -- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdc80b2e50 msgr2=0x7fbdc814bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdc7e00700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdc80b2e50 0x7fbdc814bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdc7e00700 1 -- 172.21.15.12:0/2298777465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb8009550 con 0x7fbdc8072580 2024-03-29T10:09:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.302+0000 7fbdc7e00700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc8072580 0x7fbdc81f6380 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009130 tx=0x7fbdb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:33.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.303+0000 7fbdbfe00700 1 -- 172.21.15.12:0/2298777465 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8011630 con 0x7fbdc8072580 2024-03-29T10:09:33.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.303+0000 7fbdbfe00700 1 -- 172.21.15.12:0/2298777465 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb8011790 con 0x7fbdc8072580 2024-03-29T10:09:33.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.303+0000 7fbdcf200700 1 -- 172.21.15.12:0/2298777465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc80ce2f0 con 0x7fbdc8072580 2024-03-29T10:09:33.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.303+0000 7fbdbfe00700 1 -- 172.21.15.12:0/2298777465 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8011b40 con 0x7fbdc8072580 2024-03-29T10:09:33.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.303+0000 7fbdcf200700 1 -- 172.21.15.12:0/2298777465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc80ce8b0 con 0x7fbdc8072580 2024-03-29T10:09:33.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.304+0000 7fbdcf200700 1 -- 172.21.15.12:0/2298777465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc8067720 con 0x7fbdc8072580 2024-03-29T10:09:33.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.305+0000 7fbdbfe00700 1 -- 172.21.15.12:0/2298777465 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbdb8011ca0 con 0x7fbdc8072580 2024-03-29T10:09:33.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.306+0000 7fbdbfe00700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbda806adb0 0x7fbda806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.306+0000 7fbdbfe00700 1 -- 172.21.15.12:0/2298777465 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbdb808ab00 con 0x7fbdc8072580 2024-03-29T10:09:33.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.306+0000 7fbdc7400700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbda806adb0 0x7fbda806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.307+0000 7fbdc7400700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbda806adb0 0x7fbda806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8150ea0 tx=0x7fbdb4008040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:33.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.311+0000 7fbdbfe00700 1 -- 172.21.15.12:0/2298777465 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbdb808ce60 con 0x7fbdc8072580 2024-03-29T10:09:33.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.522+0000 7fbdcf200700 1 -- 172.21.15.12:0/2298777465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdc80cef70 con 0x7fbdc8072580 2024-03-29T10:09:33.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.523+0000 7fbdbfe00700 1 -- 172.21.15.12:0/2298777465 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+16110 (secure 0 0 0) 0x7fbdb8055c70 con 0x7fbdc8072580 2024-03-29T10:09:33.523 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:33.524 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":79,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:32.596801+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:09:30.441619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-29T10:09:31.581366+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:09:33.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 -- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbda806adb0 msgr2=0x7fbda806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:33.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbda806adb0 0x7fbda806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8150ea0 tx=0x7fbdb4008040 comp rx=0 tx=0).stop 2024-03-29T10:09:33.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 -- 172.21.15.12:0/2298777465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc8072580 msgr2=0x7fbdc81f6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:33.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc8072580 0x7fbdc81f6380 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009130 tx=0x7fbdb8004750 comp rx=0 tx=0).stop 2024-03-29T10:09:33.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 -- 172.21.15.12:0/2298777465 shutdown_connections 2024-03-29T10:09:33.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbda806adb0 0x7fbda806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdc80b3af0 0x7fbdc8071430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc8072580 0x7fbdc81f6380 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 --2- 172.21.15.12:0/2298777465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdc80b2e50 0x7fbdc814bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 -- 172.21.15.12:0/2298777465 >> 172.21.15.12:0/2298777465 conn(0x7fbdc80cf190 msgr2=0x7fbdc80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 -- 172.21.15.12:0/2298777465 shutdown_connections 2024-03-29T10:09:33.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.526+0000 7fbdcf200700 1 -- 172.21.15.12:0/2298777465 wait complete. 2024-03-29T10:09:33.544 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T10:09:33.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.747+0000 7ffb0d200700 1 Processor -- start 2024-03-29T10:09:33.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.747+0000 7ffb0d200700 1 -- start start 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.748+0000 7ffb0d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 0x7ffb080d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.748+0000 7ffb0d200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffb080d16a0 0x7ffb080d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.748+0000 7ffb0d200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb08161420 con 0x7ffb080d2370 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.748+0000 7ffb0d200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb080d2820 con 0x7ffb080d16a0 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.748+0000 7ffb0d200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb08161700 con 0x7ffb0815d9d0 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.749+0000 7ffb05400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffb080d16a0 0x7ffb080d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.749+0000 7ffb05400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffb080d16a0 0x7ffb080d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58176/0 (socket says 172.21.15.12:58176) 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.749+0000 7ffb05400700 1 -- 172.21.15.12:0/2358340309 learned_addr learned my addr 172.21.15.12:0/2358340309 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.749+0000 7ffb05e00700 1 --2- 172.21.15.12:0/2358340309 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 0x7ffb080d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.749+0000 7ffb05400700 1 -- 172.21.15.12:0/2358340309 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 msgr2=0x7ffb080d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.749+0000 7ffb05400700 1 --2- 172.21.15.12:0/2358340309 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 0x7ffb080d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.749+0000 7ffb05400700 1 -- 172.21.15.12:0/2358340309 >> v1:172.21.15.159:6789/0 conn(0x7ffb080d2370 legacy=0x7ffb0815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.749+0000 7ffb05400700 1 -- 172.21.15.12:0/2358340309 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb081619e0 con 0x7ffb080d16a0 2024-03-29T10:09:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.749+0000 7ffb05e00700 1 --2- 172.21.15.12:0/2358340309 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 0x7ffb080d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:09:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.749+0000 7ffb05400700 1 --2- 172.21.15.12:0/2358340309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d16a0 0x7ffb080d1e30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40098e0 tx=0x7ffaf4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91dd4d5e68c66a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb04a00700 1 -- 172.21.15.12:0/2358340309 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4004230 con 0x7ffb080d16a0 2024-03-29T10:09:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb04a00700 1 -- 172.21.15.12:0/2358340309 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf4004390 con 0x7ffb080d16a0 2024-03-29T10:09:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb04a00700 1 -- 172.21.15.12:0/2358340309 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40036a0 con 0x7ffb080d16a0 2024-03-29T10:09:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb0d200700 1 -- 172.21.15.12:0/2358340309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d16a0 msgr2=0x7ffb080d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb0d200700 1 --2- 172.21.15.12:0/2358340309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d16a0 0x7ffb080d1e30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40098e0 tx=0x7ffaf4009bf0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb0d200700 1 -- 172.21.15.12:0/2358340309 shutdown_connections 2024-03-29T10:09:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb0d200700 1 --2- 172.21.15.12:0/2358340309 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 0x7ffb080d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb0d200700 1 --2- 172.21.15.12:0/2358340309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d16a0 0x7ffb080d1e30 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb0d200700 1 -- 172.21.15.12:0/2358340309 >> 172.21.15.12:0/2358340309 conn(0x7ffb0808aff0 msgr2=0x7ffb080d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb0d200700 1 -- 172.21.15.12:0/2358340309 shutdown_connections 2024-03-29T10:09:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.750+0000 7ffb0d200700 1 -- 172.21.15.12:0/2358340309 wait complete. 2024-03-29T10:09:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.752+0000 7ffb0d200700 1 Processor -- start 2024-03-29T10:09:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.752+0000 7ffb0d200700 1 -- start start 2024-03-29T10:09:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.753+0000 7ffb0d200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb080d16a0 0x7ffb0814b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.753+0000 7ffb0d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d2370 0x7ffb0815eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.753+0000 7ffb0d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 0x7ffb0815f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.753+0000 7ffb0d200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080b81f0 con 0x7ffb080d2370 2024-03-29T10:09:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.753+0000 7ffb0d200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080b84d0 con 0x7ffb080d16a0 2024-03-29T10:09:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.753+0000 7ffb0d200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080b87b0 con 0x7ffb0815d9d0 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.753+0000 7ffb05400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d2370 0x7ffb0815eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.753+0000 7ffb05400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d2370 0x7ffb0815eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58178/0 (socket says 172.21.15.12:58178) 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.753+0000 7ffb05400700 1 -- 172.21.15.12:0/2817891453 learned_addr learned my addr 172.21.15.12:0/2817891453 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.753+0000 7ffb05e00700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb080d16a0 0x7ffb0814b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffb06800700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 0x7ffb0815f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffb05400700 1 -- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 msgr2=0x7ffb0815f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffb05400700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 0x7ffb0815f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffb05400700 1 -- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb080d16a0 msgr2=0x7ffb0814b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffb05400700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb080d16a0 0x7ffb0814b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffb05400700 1 -- 172.21.15.12:0/2817891453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40094f0 con 0x7ffb080d2370 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffb05e00700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb080d16a0 0x7ffb0814b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffb05400700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d2370 0x7ffb0815eea0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4004000 tx=0x7ffaf4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffafe000700 1 -- 172.21.15.12:0/2817891453 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf400b660 con 0x7ffb080d2370 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffafe000700 1 -- 172.21.15.12:0/2817891453 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf400b7e0 con 0x7ffb080d2370 2024-03-29T10:09:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffb0d200700 1 -- 172.21.15.12:0/2817891453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb080b8a90 con 0x7ffb080d2370 2024-03-29T10:09:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffafe000700 1 -- 172.21.15.12:0/2817891453 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf400bb90 con 0x7ffb080d2370 2024-03-29T10:09:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.754+0000 7ffb0d200700 1 -- 172.21.15.12:0/2817891453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb080b7ea0 con 0x7ffb080d2370 2024-03-29T10:09:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.756+0000 7ffb0d200700 1 -- 172.21.15.12:0/2817891453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb080d1ca0 con 0x7ffb080d2370 2024-03-29T10:09:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.756+0000 7ffafe000700 1 -- 172.21.15.12:0/2817891453 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ffaf4022070 con 0x7ffb080d2370 2024-03-29T10:09:33.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.758+0000 7ffafe000700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffae806adb0 0x7ffae806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:33.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.758+0000 7ffafe000700 1 -- 172.21.15.12:0/2817891453 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffaf408ac50 con 0x7ffb080d2370 2024-03-29T10:09:33.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.758+0000 7ffb05e00700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffae806adb0 0x7ffae806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:33.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.759+0000 7ffb05e00700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffae806adb0 0x7ffae806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800aa20 tx=0x7ffaf8008040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:33.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:33.762+0000 7ffafe000700 1 -- 172.21.15.12:0/2817891453 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffaf4055c80 con 0x7ffb080d2370 2024-03-29T10:09:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.029+0000 7ffb0d200700 1 -- 172.21.15.12:0/2817891453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ffb08067720 con 0x7ffb080d2370 2024-03-29T10:09:34.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.607+0000 7ffafe000700 1 -- 172.21.15.12:0/2817891453 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v30) v1 ==== 118+0+0 (secure 0 0 0) 0x7ffaf400bcf0 con 0x7ffb080d2370 2024-03-29T10:09:34.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.610+0000 7ffb0d200700 1 -- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffae806adb0 msgr2=0x7ffae806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:34.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.610+0000 7ffb0d200700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffae806adb0 0x7ffae806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800aa20 tx=0x7ffaf8008040 comp rx=0 tx=0).stop 2024-03-29T10:09:34.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.610+0000 7ffb0d200700 1 -- 172.21.15.12:0/2817891453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d2370 msgr2=0x7ffb0815eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:34.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.610+0000 7ffb0d200700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d2370 0x7ffb0815eea0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4004000 tx=0x7ffaf4004750 comp rx=0 tx=0).stop 2024-03-29T10:09:34.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.611+0000 7ffb0d200700 1 -- 172.21.15.12:0/2817891453 shutdown_connections 2024-03-29T10:09:34.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.611+0000 7ffb0d200700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffae806adb0 0x7ffae806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:34.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.611+0000 7ffb0d200700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb0815d9d0 0x7ffb0815f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:34.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.611+0000 7ffb0d200700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb080d16a0 0x7ffb0814b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:34.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.611+0000 7ffb0d200700 1 --2- 172.21.15.12:0/2817891453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb080d2370 0x7ffb0815eea0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:34.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.611+0000 7ffb0d200700 1 -- 172.21.15.12:0/2817891453 >> 172.21.15.12:0/2817891453 conn(0x7ffb0808aff0 msgr2=0x7ffb080cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:34.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.611+0000 7ffb0d200700 1 -- 172.21.15.12:0/2817891453 shutdown_connections 2024-03-29T10:09:34.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.611+0000 7ffb0d200700 1 -- 172.21.15.12:0/2817891453 wait complete. 2024-03-29T10:09:34.631 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T10:09:34.631 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T10:09:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.836+0000 7f0198800700 1 Processor -- start 2024-03-29T10:09:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.836+0000 7f0198800700 1 -- start start 2024-03-29T10:09:34.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.837+0000 7f0198800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01901611b0 0x7f019015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:34.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.838+0000 7f0198800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f019015f4d0 0x7f01900ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:34.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.838+0000 7f0198800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01900ce270 con 0x7f01900d21f0 2024-03-29T10:09:34.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.838+0000 7f0198800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f019015b600 con 0x7f019015f4d0 2024-03-29T10:09:34.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.838+0000 7f0198800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01900d2760 con 0x7f01901611b0 2024-03-29T10:09:34.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.838+0000 7f0195e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01901611b0 0x7f019015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:34.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.838+0000 7f0195400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f019015f4d0 0x7f01900ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:34.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.838+0000 7f0195e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01901611b0 0x7f019015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:56196/0 (socket says 172.21.15.12:56196) 2024-03-29T10:09:34.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.838+0000 7f0195e00700 1 -- 172.21.15.12:0/1013614552 learned_addr learned my addr 172.21.15.12:0/1013614552 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:34.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.838+0000 7f0196800700 1 --1- 172.21.15.12:0/1013614552 >> v1:172.21.15.159:6789/0 conn(0x7f01900d21f0 0x7f0190161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:56202/0 (socket says 172.21.15.12:56202) 2024-03-29T10:09:34.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.838+0000 7f0194a00700 1 -- 172.21.15.12:0/1013614552 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243392725 0 0) 0x7f01900ce270 con 0x7f01900d21f0 2024-03-29T10:09:34.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.839+0000 7f0194a00700 1 -- 172.21.15.12:0/1013614552 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01780035b0 con 0x7f01900d21f0 2024-03-29T10:09:34.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.839+0000 7f0195e00700 1 -- 172.21.15.12:0/1013614552 >> v2:172.21.15.159:3300/0 conn(0x7f019015f4d0 msgr2=0x7f01900ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:34.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.839+0000 7f0195e00700 1 --2- 172.21.15.12:0/1013614552 >> v2:172.21.15.159:3300/0 conn(0x7f019015f4d0 0x7f01900ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:34.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.839+0000 7f0195e00700 1 -- 172.21.15.12:0/1013614552 >> v1:172.21.15.159:6789/0 conn(0x7f01900d21f0 legacy=0x7f0190161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.839+0000 7f0195e00700 1 -- 172.21.15.12:0/1013614552 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01900d2a40 con 0x7f01901611b0 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.839+0000 7f0195400700 1 --2- 172.21.15.12:0/1013614552 >> v2:172.21.15.159:3300/0 conn(0x7f019015f4d0 0x7f01900ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.839+0000 7f0195e00700 1 --2- 172.21.15.12:0/1013614552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01901611b0 0x7f019015ef90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f01840098f0 tx=0x7f0184009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cca56defcf16f9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.839+0000 7f0194a00700 1 -- 172.21.15.12:0/1013614552 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0184011830 con 0x7f01901611b0 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.839+0000 7f0194a00700 1 -- 172.21.15.12:0/1013614552 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0184011990 con 0x7f01901611b0 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.839+0000 7f0194a00700 1 -- 172.21.15.12:0/1013614552 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0184011d20 con 0x7f01901611b0 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.840+0000 7f0198800700 1 -- 172.21.15.12:0/1013614552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01901611b0 msgr2=0x7f019015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.840+0000 7f0198800700 1 --2- 172.21.15.12:0/1013614552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01901611b0 0x7f019015ef90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f01840098f0 tx=0x7f0184009c00 comp rx=0 tx=0).stop 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.840+0000 7f0198800700 1 -- 172.21.15.12:0/1013614552 shutdown_connections 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.840+0000 7f0198800700 1 --2- 172.21.15.12:0/1013614552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01901611b0 0x7f019015ef90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.840+0000 7f0198800700 1 --2- 172.21.15.12:0/1013614552 >> v2:172.21.15.159:3300/0 conn(0x7f019015f4d0 0x7f01900ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.840+0000 7f0198800700 1 -- 172.21.15.12:0/1013614552 >> 172.21.15.12:0/1013614552 conn(0x7f019008aff0 msgr2=0x7f01900d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:34.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.840+0000 7f0198800700 1 -- 172.21.15.12:0/1013614552 shutdown_connections 2024-03-29T10:09:34.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.840+0000 7f0198800700 1 -- 172.21.15.12:0/1013614552 wait complete. 2024-03-29T10:09:34.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.842+0000 7f0198800700 1 Processor -- start 2024-03-29T10:09:34.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.842+0000 7f0198800700 1 -- start start 2024-03-29T10:09:34.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0198800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01900d21f0 0x7f01900cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:34.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0198800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f019015f4d0 0x7f01900b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:34.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0198800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01901611b0 0x7f01901f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:34.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0198800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01900b7a30 con 0x7f019015f4d0 2024-03-29T10:09:34.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0198800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01900cb940 con 0x7f01901611b0 2024-03-29T10:09:34.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0198800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01900b6b60 con 0x7f01900d21f0 2024-03-29T10:09:34.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0195400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f019015f4d0 0x7f01900b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:34.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0195400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f019015f4d0 0x7f01900b7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58182/0 (socket says 172.21.15.12:58182) 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0195400700 1 -- 172.21.15.12:0/1265900906 learned_addr learned my addr 172.21.15.12:0/1265900906 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0196800700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01901611b0 0x7f01901f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0195e00700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01900d21f0 0x7f01900cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0195400700 1 -- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01900d21f0 msgr2=0x7f01900cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0195400700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01900d21f0 0x7f01900cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0195400700 1 -- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01901611b0 msgr2=0x7f01901f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0195400700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01901611b0 0x7f01901f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0195400700 1 -- 172.21.15.12:0/1265900906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01800094a0 con 0x7f019015f4d0 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.844+0000 7f0195e00700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01900d21f0 0x7f01900cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.845+0000 7f0195400700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f019015f4d0 0x7f01900b7ed0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f018000ec10 tx=0x7f0180009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.845+0000 7f018a000700 1 -- 172.21.15.12:0/1265900906 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018000c8c0 con 0x7f019015f4d0 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.845+0000 7f018a000700 1 -- 172.21.15.12:0/1265900906 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018000ca20 con 0x7f019015f4d0 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.845+0000 7f0198800700 1 -- 172.21.15.12:0/1265900906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0184009550 con 0x7f019015f4d0 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.845+0000 7f018a000700 1 -- 172.21.15.12:0/1265900906 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018000cdd0 con 0x7f019015f4d0 2024-03-29T10:09:34.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.845+0000 7f0198800700 1 -- 172.21.15.12:0/1265900906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01900b7370 con 0x7f019015f4d0 2024-03-29T10:09:34.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.846+0000 7f0198800700 1 -- 172.21.15.12:0/1265900906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0190067720 con 0x7f019015f4d0 2024-03-29T10:09:34.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.847+0000 7f018a000700 1 -- 172.21.15.12:0/1265900906 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0180013470 con 0x7f019015f4d0 2024-03-29T10:09:34.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.849+0000 7f018a000700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f017406adb0 0x7f017406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:34.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.849+0000 7f018a000700 1 -- 172.21.15.12:0/1265900906 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0180089280 con 0x7f019015f4d0 2024-03-29T10:09:34.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.850+0000 7f0195e00700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f017406adb0 0x7f017406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:34.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.850+0000 7f0195e00700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f017406adb0 0x7f017406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0184014040 tx=0x7f0184008030 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:34.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:34.854+0000 7f018a000700 1 -- 172.21.15.12:0/1265900906 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0180054340 con 0x7f019015f4d0 2024-03-29T10:09:35.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.114+0000 7f0198800700 1 -- 172.21.15.12:0/1265900906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f01900b5d00 con 0x7f019015f4d0 2024-03-29T10:09:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.613+0000 7f018a000700 1 -- 172.21.15.12:0/1265900906 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0180057960 con 0x7f019015f4d0 2024-03-29T10:09:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 -- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f017406adb0 msgr2=0x7f017406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f017406adb0 0x7f017406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0184014040 tx=0x7f0184008030 comp rx=0 tx=0).stop 2024-03-29T10:09:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 -- 172.21.15.12:0/1265900906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f019015f4d0 msgr2=0x7f01900b7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f019015f4d0 0x7f01900b7ed0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f018000ec10 tx=0x7f0180009e60 comp rx=0 tx=0).stop 2024-03-29T10:09:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 -- 172.21.15.12:0/1265900906 shutdown_connections 2024-03-29T10:09:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f017406adb0 0x7f017406d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01901611b0 0x7f01901f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01900d21f0 0x7f01900cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 --2- 172.21.15.12:0/1265900906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f019015f4d0 0x7f01900b7ed0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 -- 172.21.15.12:0/1265900906 >> 172.21.15.12:0/1265900906 conn(0x7f019008aff0 msgr2=0x7f019015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 -- 172.21.15.12:0/1265900906 shutdown_connections 2024-03-29T10:09:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.616+0000 7f0198800700 1 -- 172.21.15.12:0/1265900906 wait complete. 2024-03-29T10:09:35.636 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T10:09:35.636 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:09:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.835+0000 7f6f87400700 1 Processor -- start 2024-03-29T10:09:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.835+0000 7f6f87400700 1 -- start start 2024-03-29T10:09:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f87400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6f800d18b0 0x7f6f800cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f87400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f80161590 0x7f6f80161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f87400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800ceda0 con 0x7f6f80160f10 2024-03-29T10:09:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f87400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f800d1d30 con 0x7f6f80161590 2024-03-29T10:09:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f87400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f80161f60 con 0x7f6f800d18b0 2024-03-29T10:09:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f85400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f80161590 0x7f6f80161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f84a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6f80160f10 0x7f6f800d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33140/0 (socket says 172.21.15.12:33140) 2024-03-29T10:09:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f84a00700 1 -- 172.21.15.12:0/146241596 learned_addr learned my addr 172.21.15.12:0/146241596 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f7fe00700 1 --2- 172.21.15.12:0/146241596 >> v2:172.21.15.159:3300/0 conn(0x7f6f800d18b0 0x7f6f800cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f85400700 1 -- 172.21.15.12:0/146241596 >> v2:172.21.15.159:3300/0 conn(0x7f6f800d18b0 msgr2=0x7f6f800cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f7f400700 1 -- 172.21.15.12:0/146241596 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049965664 0 0) 0x7f6f800ceda0 con 0x7f6f80160f10 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f85400700 1 --2- 172.21.15.12:0/146241596 >> v2:172.21.15.159:3300/0 conn(0x7f6f800d18b0 0x7f6f800cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f85400700 1 -- 172.21.15.12:0/146241596 >> v1:172.21.15.12:6789/0 conn(0x7f6f80160f10 legacy=0x7f6f800d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f85400700 1 -- 172.21.15.12:0/146241596 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f801612f0 con 0x7f6f80161590 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.837+0000 7f6f7fe00700 1 --2- 172.21.15.12:0/146241596 >> v2:172.21.15.159:3300/0 conn(0x7f6f800d18b0 0x7f6f800cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.838+0000 7f6f85400700 1 --2- 172.21.15.12:0/146241596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80161590 0x7f6f80161a20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f6f8008ca30 tx=0x7f6f7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562c44a60fee28f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.838+0000 7f6f7f400700 1 -- 172.21.15.12:0/146241596 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7800e070 con 0x7f6f80161590 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.838+0000 7f6f7f400700 1 -- 172.21.15.12:0/146241596 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7800bbd0 con 0x7f6f80161590 2024-03-29T10:09:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.838+0000 7f6f7f400700 1 -- 172.21.15.12:0/146241596 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78012430 con 0x7f6f80161590 2024-03-29T10:09:35.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.838+0000 7f6f87400700 1 -- 172.21.15.12:0/146241596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80161590 msgr2=0x7f6f80161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:35.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.838+0000 7f6f87400700 1 --2- 172.21.15.12:0/146241596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80161590 0x7f6f80161a20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f6f8008ca30 tx=0x7f6f7800b330 comp rx=0 tx=0).stop 2024-03-29T10:09:35.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.839+0000 7f6f87400700 1 -- 172.21.15.12:0/146241596 shutdown_connections 2024-03-29T10:09:35.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.839+0000 7f6f87400700 1 --2- 172.21.15.12:0/146241596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80161590 0x7f6f80161a20 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:35.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.839+0000 7f6f87400700 1 --2- 172.21.15.12:0/146241596 >> v2:172.21.15.159:3300/0 conn(0x7f6f800d18b0 0x7f6f800cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:35.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.839+0000 7f6f87400700 1 -- 172.21.15.12:0/146241596 >> 172.21.15.12:0/146241596 conn(0x7f6f8008aff0 msgr2=0x7f6f800d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:35.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.839+0000 7f6f87400700 1 -- 172.21.15.12:0/146241596 shutdown_connections 2024-03-29T10:09:35.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.839+0000 7f6f87400700 1 -- 172.21.15.12:0/146241596 wait complete. 2024-03-29T10:09:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.841+0000 7f6f87400700 1 Processor -- start 2024-03-29T10:09:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.841+0000 7f6f87400700 1 -- start start 2024-03-29T10:09:35.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f87400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f800d18b0 0x7f6f801553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f87400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80160f10 0x7f6f801e97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f87400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f80161590 0x7f6f80154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f87400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f80155cf0 con 0x7f6f80160f10 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f87400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f800b72d0 con 0x7f6f800d18b0 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f87400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f800b75b0 con 0x7f6f80161590 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f7fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80160f10 0x7f6f801e97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f7fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80160f10 0x7f6f801e97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58200/0 (socket says 172.21.15.12:58200) 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f7fe00700 1 -- 172.21.15.12:0/995246974 learned_addr learned my addr 172.21.15.12:0/995246974 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f84a00700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f800d18b0 0x7f6f801553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f85400700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f80161590 0x7f6f80154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f7fe00700 1 -- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f80161590 msgr2=0x7f6f80154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f7fe00700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f80161590 0x7f6f80154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f7fe00700 1 -- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f800d18b0 msgr2=0x7f6f801553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f7fe00700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f800d18b0 0x7f6f801553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.842+0000 7f6f7fe00700 1 -- 172.21.15.12:0/995246974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f78009a70 con 0x7f6f80160f10 2024-03-29T10:09:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.843+0000 7f6f7fe00700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80160f10 0x7f6f801e97f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c7f0 tx=0x7f6f6c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.843+0000 7f6f7cc00700 1 -- 172.21.15.12:0/995246974 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c011800 con 0x7f6f80160f10 2024-03-29T10:09:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.843+0000 7f6f7cc00700 1 -- 172.21.15.12:0/995246974 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c011960 con 0x7f6f80160f10 2024-03-29T10:09:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.843+0000 7f6f87400700 1 -- 172.21.15.12:0/995246974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f801e7960 con 0x7f6f80160f10 2024-03-29T10:09:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.843+0000 7f6f7cc00700 1 -- 172.21.15.12:0/995246974 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c011d10 con 0x7f6f80160f10 2024-03-29T10:09:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.843+0000 7f6f87400700 1 -- 172.21.15.12:0/995246974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f801e8160 con 0x7f6f80160f10 2024-03-29T10:09:35.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.844+0000 7f6f87400700 1 -- 172.21.15.12:0/995246974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f801e77c0 con 0x7f6f80160f10 2024-03-29T10:09:35.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.845+0000 7f6f7cc00700 1 -- 172.21.15.12:0/995246974 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6f6c018360 con 0x7f6f80160f10 2024-03-29T10:09:35.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.846+0000 7f6f7cc00700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f6006adb0 0x7f6f6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:35.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.846+0000 7f6f7cc00700 1 -- 172.21.15.12:0/995246974 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f6c089600 con 0x7f6f80160f10 2024-03-29T10:09:35.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.847+0000 7f6f84a00700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f6006adb0 0x7f6f6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:35.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.847+0000 7f6f84a00700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f6006adb0 0x7f6f6006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f74000f60 tx=0x7f6f74007c90 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:35.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:35.851+0000 7f6f7cc00700 1 -- 172.21.15.12:0/995246974 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f801e77c0 con 0x7f6f80160f10 2024-03-29T10:09:36.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.114+0000 7f6f87400700 1 -- 172.21.15.12:0/995246974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f801e7e80 con 0x7f6f80160f10 2024-03-29T10:09:36.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.115+0000 7f6f7cc00700 1 -- 172.21.15.12:0/995246974 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f6f801e7e80 con 0x7f6f80160f10 2024-03-29T10:09:36.115 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:36.115 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3202930550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3202930550},{"type":"v1","addr":"172.21.15.159:6835","nonce":3202930550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4189378403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4189378403},{"type":"v1","addr":"172.21.15.12:6834","nonce":4189378403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2681299760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2681299760},{"type":"v1","addr":"172.21.15.12:6835","nonce":2681299760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:09:32.597152+0000","modified":"2024-03-29T10:09:35.607262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6241":{"gid":6241,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/4289275091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":4289275091},{"type":"v1","addr":"172.21.15.159:6837","nonce":4289275091}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 -- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f6006adb0 msgr2=0x7f6f6006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f6006adb0 0x7f6f6006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f74000f60 tx=0x7f6f74007c90 comp rx=0 tx=0).stop 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 -- 172.21.15.12:0/995246974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80160f10 msgr2=0x7f6f801e97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80160f10 0x7f6f801e97f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c7f0 tx=0x7f6f6c00cbb0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 -- 172.21.15.12:0/995246974 shutdown_connections 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f6006adb0 0x7f6f6006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f80161590 0x7f6f80154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f800d18b0 0x7f6f801553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 --2- 172.21.15.12:0/995246974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f80160f10 0x7f6f801e97f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 -- 172.21.15.12:0/995246974 >> 172.21.15.12:0/995246974 conn(0x7f6f8008aff0 msgr2=0x7f6f800cbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 -- 172.21.15.12:0/995246974 shutdown_connections 2024-03-29T10:09:36.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.118+0000 7f6f87400700 1 -- 172.21.15.12:0/995246974 wait complete. 2024-03-29T10:09:36.119 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 31 2024-03-29T10:09:36.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:09:36.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.334+0000 7f5777000700 1 Processor -- start 2024-03-29T10:09:36.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.334+0000 7f5777000700 1 -- start start 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.335+0000 7f5777000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5770160f10 0x7f57700d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.335+0000 7f5777000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57700cf230 0x7f57700d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.335+0000 7f5777000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700ceda0 con 0x7f5770161590 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.335+0000 7f5777000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57700d23d0 con 0x7f5770160f10 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.335+0000 7f5777000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57701612f0 con 0x7f57700cf230 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.335+0000 7f5775000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5770161590 0x7f57700b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33148/0 (socket says 172.21.15.12:33148) 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.335+0000 7f5775000700 1 -- 172.21.15.12:0/1318646526 learned_addr learned my addr 172.21.15.12:0/1318646526 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.336+0000 7f576f400700 1 --2- 172.21.15.12:0/1318646526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57700cf230 0x7f57700d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.336+0000 7f576fe00700 1 --2- 172.21.15.12:0/1318646526 >> v2:172.21.15.159:3300/0 conn(0x7f5770160f10 0x7f57700d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.336+0000 7f576ea00700 1 -- 172.21.15.12:0/1318646526 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485883970 0 0) 0x7f57700ceda0 con 0x7f5770161590 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.336+0000 7f576ea00700 1 -- 172.21.15.12:0/1318646526 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57540035b0 con 0x7f5770161590 2024-03-29T10:09:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.336+0000 7f576f400700 1 -- 172.21.15.12:0/1318646526 >> v2:172.21.15.159:3300/0 conn(0x7f5770160f10 msgr2=0x7f57700d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.336+0000 7f576f400700 1 --2- 172.21.15.12:0/1318646526 >> v2:172.21.15.159:3300/0 conn(0x7f5770160f10 0x7f57700d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.336+0000 7f576f400700 1 -- 172.21.15.12:0/1318646526 >> v1:172.21.15.12:6789/0 conn(0x7f5770161590 legacy=0x7f57700b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.336+0000 7f576ea00700 1 -- 172.21.15.12:0/1318646526 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2673016961 0 0) 0x7f57540035b0 con 0x7f5770161590 2024-03-29T10:09:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.336+0000 7f576f400700 1 -- 172.21.15.12:0/1318646526 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700ce470 con 0x7f57700cf230 2024-03-29T10:09:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.336+0000 7f576fe00700 1 --2- 172.21.15.12:0/1318646526 >> v2:172.21.15.159:3300/0 conn(0x7f5770160f10 0x7f57700d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.337+0000 7f576f400700 1 --2- 172.21.15.12:0/1318646526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57700cf230 0x7f57700d1e90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f575c009840 tx=0x7f575c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d853c8b96b7deb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.337+0000 7f576ea00700 1 -- 172.21.15.12:0/1318646526 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c011830 con 0x7f57700cf230 2024-03-29T10:09:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.337+0000 7f576ea00700 1 -- 172.21.15.12:0/1318646526 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c011990 con 0x7f57700cf230 2024-03-29T10:09:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.337+0000 7f576ea00700 1 -- 172.21.15.12:0/1318646526 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c011d00 con 0x7f57700cf230 2024-03-29T10:09:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.337+0000 7f5777000700 1 -- 172.21.15.12:0/1318646526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57700cf230 msgr2=0x7f57700d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.337+0000 7f5777000700 1 --2- 172.21.15.12:0/1318646526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57700cf230 0x7f57700d1e90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f575c009840 tx=0x7f575c009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.337+0000 7f5777000700 1 -- 172.21.15.12:0/1318646526 shutdown_connections 2024-03-29T10:09:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.337+0000 7f5777000700 1 --2- 172.21.15.12:0/1318646526 >> v2:172.21.15.159:3300/0 conn(0x7f5770160f10 0x7f57700d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.337+0000 7f5777000700 1 --2- 172.21.15.12:0/1318646526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57700cf230 0x7f57700d1e90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.337+0000 7f5777000700 1 -- 172.21.15.12:0/1318646526 >> 172.21.15.12:0/1318646526 conn(0x7f577008aff0 msgr2=0x7f57700d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.338+0000 7f5777000700 1 -- 172.21.15.12:0/1318646526 shutdown_connections 2024-03-29T10:09:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.338+0000 7f5777000700 1 -- 172.21.15.12:0/1318646526 wait complete. 2024-03-29T10:09:36.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.339+0000 7f5777000700 1 Processor -- start 2024-03-29T10:09:36.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.339+0000 7f5777000700 1 -- start start 2024-03-29T10:09:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.340+0000 7f5777000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57700cf230 0x7f5770155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.340+0000 7f5777000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5770160f10 0x7f5770157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.340+0000 7f5777000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5770161590 0x7f577006cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.340+0000 7f5777000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5770159130 con 0x7f57700cf230 2024-03-29T10:09:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.340+0000 7f5777000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5770159550 con 0x7f5770160f10 2024-03-29T10:09:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.340+0000 7f5777000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57701584b0 con 0x7f5770161590 2024-03-29T10:09:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f576fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57700cf230 0x7f5770155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f576fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57700cf230 0x7f5770155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58210/0 (socket says 172.21.15.12:58210) 2024-03-29T10:09:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f576fe00700 1 -- 172.21.15.12:0/690271802 learned_addr learned my addr 172.21.15.12:0/690271802 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f5775000700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5770161590 0x7f577006cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f576fe00700 1 -- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5770161590 msgr2=0x7f577006cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f576fe00700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5770161590 0x7f577006cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f576fe00700 1 -- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5770160f10 msgr2=0x7f5770157ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f576fe00700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5770160f10 0x7f5770157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f576fe00700 1 -- 172.21.15.12:0/690271802 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57600094a0 con 0x7f57700cf230 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f576fe00700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57700cf230 0x7f5770155f60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f576000ebe0 tx=0x7f5760009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f5767e00700 1 -- 172.21.15.12:0/690271802 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576000c8b0 con 0x7f57700cf230 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f5767e00700 1 -- 172.21.15.12:0/690271802 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576000ca10 con 0x7f57700cf230 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.341+0000 7f5767e00700 1 -- 172.21.15.12:0/690271802 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576000cda0 con 0x7f57700cf230 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.342+0000 7f5777000700 1 -- 172.21.15.12:0/690271802 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c0094a0 con 0x7f57700cf230 2024-03-29T10:09:36.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.342+0000 7f5777000700 1 -- 172.21.15.12:0/690271802 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5770158a70 con 0x7f57700cf230 2024-03-29T10:09:36.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.343+0000 7f5777000700 1 -- 172.21.15.12:0/690271802 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5770159270 con 0x7f57700cf230 2024-03-29T10:09:36.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.344+0000 7f5767e00700 1 -- 172.21.15.12:0/690271802 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5760013470 con 0x7f57700cf230 2024-03-29T10:09:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.344+0000 7f5767e00700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f575006adb0 0x7f575006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.345+0000 7f5767e00700 1 -- 172.21.15.12:0/690271802 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5760089520 con 0x7f57700cf230 2024-03-29T10:09:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.345+0000 7f576f400700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f575006adb0 0x7f575006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.346+0000 7f576f400700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f575006adb0 0x7f575006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f577015c020 tx=0x7f575c010000 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:36.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.349+0000 7f5767e00700 1 -- 172.21.15.12:0/690271802 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5770159270 con 0x7f57700cf230 2024-03-29T10:09:36.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.564+0000 7f5777000700 1 -- 172.21.15.12:0/690271802 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5770159270 con 0x7f57700cf230 2024-03-29T10:09:36.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.565+0000 7f5767e00700 1 -- 172.21.15.12:0/690271802 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f5760054680 con 0x7f57700cf230 2024-03-29T10:09:36.565 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:36.565 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":80,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:33.599657+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:09:30.441619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-29T10:09:31.581366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:09:36.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 -- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f575006adb0 msgr2=0x7f575006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f575006adb0 0x7f575006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f577015c020 tx=0x7f575c010000 comp rx=0 tx=0).stop 2024-03-29T10:09:36.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 -- 172.21.15.12:0/690271802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57700cf230 msgr2=0x7f5770155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57700cf230 0x7f5770155f60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f576000ebe0 tx=0x7f5760009e20 comp rx=0 tx=0).stop 2024-03-29T10:09:36.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 -- 172.21.15.12:0/690271802 shutdown_connections 2024-03-29T10:09:36.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f575006adb0 0x7f575006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5770161590 0x7f577006cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57700cf230 0x7f5770155f60 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 --2- 172.21.15.12:0/690271802 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5770160f10 0x7f5770157ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 -- 172.21.15.12:0/690271802 >> 172.21.15.12:0/690271802 conn(0x7f577008aff0 msgr2=0x7f57700caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:36.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 -- 172.21.15.12:0/690271802 shutdown_connections 2024-03-29T10:09:36.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.568+0000 7f5777000700 1 -- 172.21.15.12:0/690271802 wait complete. 2024-03-29T10:09:36.586 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T10:09:36.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:09:36.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.785+0000 7f9209e00700 1 Processor -- start 2024-03-29T10:09:36.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.785+0000 7f9209e00700 1 -- start start 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.786+0000 7f9209e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9204160f10 0x7f92040d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.786+0000 7f9209e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9204161590 0x7f9204161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.786+0000 7f9209e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040ceda0 con 0x7f92040cf230 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.786+0000 7f9209e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9204161f80 con 0x7f9204161590 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.786+0000 7f9209e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92041612f0 con 0x7f9204160f10 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.787+0000 7f9203400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9204161590 0x7f9204161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.787+0000 7f9202000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f92040cf230 0x7f92040d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33156/0 (socket says 172.21.15.12:33156) 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.787+0000 7f9202000700 1 -- 172.21.15.12:0/12869522 learned_addr learned my addr 172.21.15.12:0/12869522 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.787+0000 7f9202a00700 1 --2- 172.21.15.12:0/12869522 >> v2:172.21.15.159:3300/0 conn(0x7f9204160f10 0x7f92040d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.787+0000 7f9203400700 1 -- 172.21.15.12:0/12869522 >> v2:172.21.15.159:3300/0 conn(0x7f9204160f10 msgr2=0x7f92040d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.787+0000 7f9203400700 1 --2- 172.21.15.12:0/12869522 >> v2:172.21.15.159:3300/0 conn(0x7f9204160f10 0x7f92040d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.787+0000 7f9203400700 1 -- 172.21.15.12:0/12869522 >> v1:172.21.15.12:6789/0 conn(0x7f92040cf230 legacy=0x7f92040d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.787+0000 7f9203400700 1 -- 172.21.15.12:0/12869522 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92040ce470 con 0x7f9204161590 2024-03-29T10:09:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.787+0000 7f9203400700 1 --2- 172.21.15.12:0/12869522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 0x7f9204161a40 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f92040d1080 tx=0x7f91fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da09df6e14334ff0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.787+0000 7f9201600700 1 -- 172.21.15.12:0/12869522 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc00e070 con 0x7f9204161590 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.788+0000 7f9201600700 1 -- 172.21.15.12:0/12869522 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91fc00bbd0 con 0x7f9204161590 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.788+0000 7f9201600700 1 -- 172.21.15.12:0/12869522 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc012430 con 0x7f9204161590 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.788+0000 7f9209e00700 1 -- 172.21.15.12:0/12869522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 msgr2=0x7f9204161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.788+0000 7f9209e00700 1 --2- 172.21.15.12:0/12869522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 0x7f9204161a40 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f92040d1080 tx=0x7f91fc00b330 comp rx=0 tx=0).stop 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.788+0000 7f9209e00700 1 -- 172.21.15.12:0/12869522 shutdown_connections 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.788+0000 7f9209e00700 1 --2- 172.21.15.12:0/12869522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 0x7f9204161a40 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.788+0000 7f9209e00700 1 --2- 172.21.15.12:0/12869522 >> v2:172.21.15.159:3300/0 conn(0x7f9204160f10 0x7f92040d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.788+0000 7f9209e00700 1 -- 172.21.15.12:0/12869522 >> 172.21.15.12:0/12869522 conn(0x7f920408aff0 msgr2=0x7f92040d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.788+0000 7f9209e00700 1 -- 172.21.15.12:0/12869522 shutdown_connections 2024-03-29T10:09:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.788+0000 7f9209e00700 1 -- 172.21.15.12:0/12869522 wait complete. 2024-03-29T10:09:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.790+0000 7f9209e00700 1 Processor -- start 2024-03-29T10:09:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.790+0000 7f9209e00700 1 -- start start 2024-03-29T10:09:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.791+0000 7f9209e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92040cf230 0x7f920414e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.791+0000 7f9209e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9204160f10 0x7f920415d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.791+0000 7f9209e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 0x7f920409e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.791+0000 7f9209e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920415e7f0 con 0x7f9204161590 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.791+0000 7f9209e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920415dbe0 con 0x7f92040cf230 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.791+0000 7f9209e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920415dec0 con 0x7f9204160f10 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.791+0000 7f9203400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 0x7f920409e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f9203400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 0x7f920409e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58226/0 (socket says 172.21.15.12:58226) 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f9202a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92040cf230 0x7f920414e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f9203400700 1 -- 172.21.15.12:0/3257428127 learned_addr learned my addr 172.21.15.12:0/3257428127 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f9202000700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9204160f10 0x7f920415d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f9203400700 1 -- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9204160f10 msgr2=0x7f920415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f9203400700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9204160f10 0x7f920415d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f9203400700 1 -- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92040cf230 msgr2=0x7f920414e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f9203400700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92040cf230 0x7f920414e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f9203400700 1 -- 172.21.15.12:0/3257428127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91fc009a70 con 0x7f9204161590 2024-03-29T10:09:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f9203400700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 0x7f920409e270 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00b370 tx=0x7f91fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.792+0000 7f91fb400700 1 -- 172.21.15.12:0/3257428127 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc012430 con 0x7f9204161590 2024-03-29T10:09:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.793+0000 7f91fb400700 1 -- 172.21.15.12:0/3257428127 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91fc020070 con 0x7f9204161590 2024-03-29T10:09:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.793+0000 7f9209e00700 1 -- 172.21.15.12:0/3257428127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f920415e1a0 con 0x7f9204161590 2024-03-29T10:09:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.793+0000 7f91fb400700 1 -- 172.21.15.12:0/3257428127 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc0098f0 con 0x7f9204161590 2024-03-29T10:09:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.793+0000 7f9209e00700 1 -- 172.21.15.12:0/3257428127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f920414ef10 con 0x7f9204161590 2024-03-29T10:09:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.794+0000 7f9209e00700 1 -- 172.21.15.12:0/3257428127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920415e960 con 0x7f9204161590 2024-03-29T10:09:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.795+0000 7f91fb400700 1 -- 172.21.15.12:0/3257428127 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f91fc004040 con 0x7f9204161590 2024-03-29T10:09:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.795+0000 7f91fb400700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91e406adb0 0x7f91e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.796+0000 7f91fb400700 1 -- 172.21.15.12:0/3257428127 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91fc022070 con 0x7f9204161590 2024-03-29T10:09:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.796+0000 7f9202a00700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91e406adb0 0x7f91e406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.797+0000 7f9202a00700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91e406adb0 0x7f91e406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f91f4006b20 tx=0x7f91f4008040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:36.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:36.800+0000 7f91fb400700 1 -- 172.21.15.12:0/3257428127 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91fc056660 con 0x7f9204161590 2024-03-29T10:09:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.013+0000 7f9209e00700 1 -- 172.21.15.12:0/3257428127 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92040cf6b0 con 0x7f91e406adb0 2024-03-29T10:09:37.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.024+0000 7f91fb400700 1 -- 172.21.15.12:0/3257428127 <== mgr.6013 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287655 (secure 0 0 0) 0x7f91f40076b0 con 0x7f91e406adb0 2024-03-29T10:09:37.028 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.028+0000 7f9209e00700 1 -- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91e406adb0 msgr2=0x7f91e406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.028+0000 7f9209e00700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91e406adb0 0x7f91e406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f91f4006b20 tx=0x7f91f4008040 comp rx=0 tx=0).stop 2024-03-29T10:09:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.028+0000 7f9209e00700 1 -- 172.21.15.12:0/3257428127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 msgr2=0x7f920409e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.028+0000 7f9209e00700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 0x7f920409e270 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00b370 tx=0x7f91fc004ad0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.028+0000 7f9209e00700 1 -- 172.21.15.12:0/3257428127 shutdown_connections 2024-03-29T10:09:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.028+0000 7f9209e00700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91e406adb0 0x7f91e406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.028+0000 7f9209e00700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9204161590 0x7f920409e270 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.028+0000 7f9209e00700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92040cf230 0x7f920414e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.028+0000 7f9209e00700 1 --2- 172.21.15.12:0/3257428127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9204160f10 0x7f920415d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.028+0000 7f9209e00700 1 -- 172.21.15.12:0/3257428127 >> 172.21.15.12:0/3257428127 conn(0x7f920408aff0 msgr2=0x7f920406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.029+0000 7f9209e00700 1 -- 172.21.15.12:0/3257428127 shutdown_connections 2024-03-29T10:09:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.029+0000 7f9209e00700 1 -- 172.21.15.12:0/3257428127 wait complete. 2024-03-29T10:09:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:09:37.046 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-29T10:09:36.838605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592660,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":279,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":178,"acting":178,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":306076,"kb_used_data":8728,"kb_used_omap":12,"kb_used_meta":297203,"kb_avail":754668644,"statfs":{"total":773094113280,"available":772780691456,"internally_reserved":0,"allocated":8937472,"data_stored":4820303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":304336441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2292,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.769510"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045584+0000","last_change":"2024-03-29T10:09:32.623236+0000","last_active":"2024-03-29T10:09:34.045584+0000","last_peered":"2024-03-29T10:09:34.045584+0000","last_clean":"2024-03-29T10:09:34.045584+0000","last_became_active":"2024-03-29T10:09:32.622714+0000","last_became_peered":"2024-03-29T10:09:32.622714+0000","last_unstale":"2024-03-29T10:09:34.045584+0000","last_undegraded":"2024-03-29T10:09:34.045584+0000","last_fullsized":"2024-03-29T10:09:34.045584+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:50.745197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607572+0000","last_change":"2024-03-29T10:09:32.620014+0000","last_active":"2024-03-29T10:09:33.607572+0000","last_peered":"2024-03-29T10:09:33.607572+0000","last_clean":"2024-03-29T10:09:33.607572+0000","last_became_active":"2024-03-29T10:09:32.619747+0000","last_became_peered":"2024-03-29T10:09:32.619747+0000","last_unstale":"2024-03-29T10:09:33.607572+0000","last_undegraded":"2024-03-29T10:09:33.607572+0000","last_fullsized":"2024-03-29T10:09:33.607572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:58:19.387293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045306+0000","last_change":"2024-03-29T10:09:32.622331+0000","last_active":"2024-03-29T10:09:34.045306+0000","last_peered":"2024-03-29T10:09:34.045306+0000","last_clean":"2024-03-29T10:09:34.045306+0000","last_became_active":"2024-03-29T10:09:32.622038+0000","last_became_peered":"2024-03-29T10:09:32.622038+0000","last_unstale":"2024-03-29T10:09:34.045306+0000","last_undegraded":"2024-03-29T10:09:34.045306+0000","last_fullsized":"2024-03-29T10:09:34.045306+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:54.413063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045348+0000","last_change":"2024-03-29T10:09:32.617524+0000","last_active":"2024-03-29T10:09:34.045348+0000","last_peered":"2024-03-29T10:09:34.045348+0000","last_clean":"2024-03-29T10:09:34.045348+0000","last_became_active":"2024-03-29T10:09:32.617270+0000","last_became_peered":"2024-03-29T10:09:32.617270+0000","last_unstale":"2024-03-29T10:09:34.045348+0000","last_undegraded":"2024-03-29T10:09:34.045348+0000","last_fullsized":"2024-03-29T10:09:34.045348+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:02.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045118+0000","last_change":"2024-03-29T10:09:32.616253+0000","last_active":"2024-03-29T10:09:34.045118+0000","last_peered":"2024-03-29T10:09:34.045118+0000","last_clean":"2024-03-29T10:09:34.045118+0000","last_became_active":"2024-03-29T10:09:32.616002+0000","last_became_peered":"2024-03-29T10:09:32.616002+0000","last_unstale":"2024-03-29T10:09:34.045118+0000","last_undegraded":"2024-03-29T10:09:34.045118+0000","last_fullsized":"2024-03-29T10:09:34.045118+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:23.297716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044899+0000","last_change":"2024-03-29T10:09:32.612351+0000","last_active":"2024-03-29T10:09:34.044899+0000","last_peered":"2024-03-29T10:09:34.044899+0000","last_clean":"2024-03-29T10:09:34.044899+0000","last_became_active":"2024-03-29T10:09:32.612057+0000","last_became_peered":"2024-03-29T10:09:32.612057+0000","last_unstale":"2024-03-29T10:09:34.044899+0000","last_undegraded":"2024-03-29T10:09:34.044899+0000","last_fullsized":"2024-03-29T10:09:34.044899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:41.040149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.606905+0000","last_change":"2024-03-29T10:09:32.615482+0000","last_active":"2024-03-29T10:09:33.606905+0000","last_peered":"2024-03-29T10:09:33.606905+0000","last_clean":"2024-03-29T10:09:33.606905+0000","last_became_active":"2024-03-29T10:09:32.615198+0000","last_became_peered":"2024-03-29T10:09:32.615198+0000","last_unstale":"2024-03-29T10:09:33.606905+0000","last_undegraded":"2024-03-29T10:09:33.606905+0000","last_fullsized":"2024-03-29T10:09:33.606905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:48:35.803134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045007+0000","last_change":"2024-03-29T10:09:32.611215+0000","last_active":"2024-03-29T10:09:34.045007+0000","last_peered":"2024-03-29T10:09:34.045007+0000","last_clean":"2024-03-29T10:09:34.045007+0000","last_became_active":"2024-03-29T10:09:32.610941+0000","last_became_peered":"2024-03-29T10:09:32.610941+0000","last_unstale":"2024-03-29T10:09:34.045007+0000","last_undegraded":"2024-03-29T10:09:34.045007+0000","last_fullsized":"2024-03-29T10:09:34.045007+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:28.376324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045092+0000","last_change":"2024-03-29T10:09:32.624355+0000","last_active":"2024-03-29T10:09:34.045092+0000","last_peered":"2024-03-29T10:09:34.045092+0000","last_clean":"2024-03-29T10:09:34.045092+0000","last_became_active":"2024-03-29T10:09:32.622547+0000","last_became_peered":"2024-03-29T10:09:32.622547+0000","last_unstale":"2024-03-29T10:09:34.045092+0000","last_undegraded":"2024-03-29T10:09:34.045092+0000","last_fullsized":"2024-03-29T10:09:34.045092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:02.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044868+0000","last_change":"2024-03-29T10:09:32.628082+0000","last_active":"2024-03-29T10:09:34.044868+0000","last_peered":"2024-03-29T10:09:34.044868+0000","last_clean":"2024-03-29T10:09:34.044868+0000","last_became_active":"2024-03-29T10:09:32.627765+0000","last_became_peered":"2024-03-29T10:09:32.627765+0000","last_unstale":"2024-03-29T10:09:34.044868+0000","last_undegraded":"2024-03-29T10:09:34.044868+0000","last_fullsized":"2024-03-29T10:09:34.044868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:50.472261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.612120+0000","last_change":"2024-03-29T10:09:32.619242+0000","last_active":"2024-03-29T10:09:33.612120+0000","last_peered":"2024-03-29T10:09:33.612120+0000","last_clean":"2024-03-29T10:09:33.612120+0000","last_became_active":"2024-03-29T10:09:32.618883+0000","last_became_peered":"2024-03-29T10:09:32.618883+0000","last_unstale":"2024-03-29T10:09:33.612120+0000","last_undegraded":"2024-03-29T10:09:33.612120+0000","last_fullsized":"2024-03-29T10:09:33.612120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:08.395520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044873+0000","last_change":"2024-03-29T10:09:32.617701+0000","last_active":"2024-03-29T10:09:34.044873+0000","last_peered":"2024-03-29T10:09:34.044873+0000","last_clean":"2024-03-29T10:09:34.044873+0000","last_became_active":"2024-03-29T10:09:32.616607+0000","last_became_peered":"2024-03-29T10:09:32.616607+0000","last_unstale":"2024-03-29T10:09:34.044873+0000","last_undegraded":"2024-03-29T10:09:34.044873+0000","last_fullsized":"2024-03-29T10:09:34.044873+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:39.250301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045466+0000","last_change":"2024-03-29T10:09:32.618041+0000","last_active":"2024-03-29T10:09:34.045466+0000","last_peered":"2024-03-29T10:09:34.045466+0000","last_clean":"2024-03-29T10:09:34.045466+0000","last_became_active":"2024-03-29T10:09:32.617123+0000","last_became_peered":"2024-03-29T10:09:32.617123+0000","last_unstale":"2024-03-29T10:09:34.045466+0000","last_undegraded":"2024-03-29T10:09:34.045466+0000","last_fullsized":"2024-03-29T10:09:34.045466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:47.161213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044987+0000","last_change":"2024-03-29T10:09:32.622774+0000","last_active":"2024-03-29T10:09:34.044987+0000","last_peered":"2024-03-29T10:09:34.044987+0000","last_clean":"2024-03-29T10:09:34.044987+0000","last_became_active":"2024-03-29T10:09:32.622503+0000","last_became_peered":"2024-03-29T10:09:32.622503+0000","last_unstale":"2024-03-29T10:09:34.044987+0000","last_undegraded":"2024-03-29T10:09:34.044987+0000","last_fullsized":"2024-03-29T10:09:34.044987+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:38.603035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607446+0000","last_change":"2024-03-29T10:09:32.625499+0000","last_active":"2024-03-29T10:09:33.607446+0000","last_peered":"2024-03-29T10:09:33.607446+0000","last_clean":"2024-03-29T10:09:33.607446+0000","last_became_active":"2024-03-29T10:09:32.625200+0000","last_became_peered":"2024-03-29T10:09:32.625200+0000","last_unstale":"2024-03-29T10:09:33.607446+0000","last_undegraded":"2024-03-29T10:09:33.607446+0000","last_fullsized":"2024-03-29T10:09:33.607446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:47.193288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045942+0000","last_change":"2024-03-29T10:09:32.622371+0000","last_active":"2024-03-29T10:09:34.045942+0000","last_peered":"2024-03-29T10:09:34.045942+0000","last_clean":"2024-03-29T10:09:34.045942+0000","last_became_active":"2024-03-29T10:09:32.621892+0000","last_became_peered":"2024-03-29T10:09:32.621892+0000","last_unstale":"2024-03-29T10:09:34.045942+0000","last_undegraded":"2024-03-29T10:09:34.045942+0000","last_fullsized":"2024-03-29T10:09:34.045942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:50.946405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045592+0000","last_change":"2024-03-29T10:09:32.615044+0000","last_active":"2024-03-29T10:09:34.045592+0000","last_peered":"2024-03-29T10:09:34.045592+0000","last_clean":"2024-03-29T10:09:34.045592+0000","last_became_active":"2024-03-29T10:09:32.614754+0000","last_became_peered":"2024-03-29T10:09:32.614754+0000","last_unstale":"2024-03-29T10:09:34.045592+0000","last_undegraded":"2024-03-29T10:09:34.045592+0000","last_fullsized":"2024-03-29T10:09:34.045592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:50.472261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607170+0000","last_change":"2024-03-29T10:09:32.613680+0000","last_active":"2024-03-29T10:09:33.607170+0000","last_peered":"2024-03-29T10:09:33.607170+0000","last_clean":"2024-03-29T10:09:33.607170+0000","last_became_active":"2024-03-29T10:09:32.611800+0000","last_became_peered":"2024-03-29T10:09:32.611800+0000","last_unstale":"2024-03-29T10:09:33.607170+0000","last_undegraded":"2024-03-29T10:09:33.607170+0000","last_fullsized":"2024-03-29T10:09:33.607170+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:04.582194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045761+0000","last_change":"2024-03-29T10:09:32.623271+0000","last_active":"2024-03-29T10:09:34.045761+0000","last_peered":"2024-03-29T10:09:34.045761+0000","last_clean":"2024-03-29T10:09:34.045761+0000","last_became_active":"2024-03-29T10:09:32.622928+0000","last_became_peered":"2024-03-29T10:09:32.622928+0000","last_unstale":"2024-03-29T10:09:34.045761+0000","last_undegraded":"2024-03-29T10:09:34.045761+0000","last_fullsized":"2024-03-29T10:09:34.045761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.930784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607046+0000","last_change":"2024-03-29T10:09:32.621045+0000","last_active":"2024-03-29T10:09:33.607046+0000","last_peered":"2024-03-29T10:09:33.607046+0000","last_clean":"2024-03-29T10:09:33.607046+0000","last_became_active":"2024-03-29T10:09:32.620499+0000","last_became_peered":"2024-03-29T10:09:32.620499+0000","last_unstale":"2024-03-29T10:09:33.607046+0000","last_undegraded":"2024-03-29T10:09:33.607046+0000","last_fullsized":"2024-03-29T10:09:33.607046+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:05:04.665074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045437+0000","last_change":"2024-03-29T10:09:32.637105+0000","last_active":"2024-03-29T10:09:34.045437+0000","last_peered":"2024-03-29T10:09:34.045437+0000","last_clean":"2024-03-29T10:09:34.045437+0000","last_became_active":"2024-03-29T10:09:32.636599+0000","last_became_peered":"2024-03-29T10:09:32.636599+0000","last_unstale":"2024-03-29T10:09:34.045437+0000","last_undegraded":"2024-03-29T10:09:34.045437+0000","last_fullsized":"2024-03-29T10:09:34.045437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:54.720344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.611723+0000","last_change":"2024-03-29T10:09:32.628611+0000","last_active":"2024-03-29T10:09:33.611723+0000","last_peered":"2024-03-29T10:09:33.611723+0000","last_clean":"2024-03-29T10:09:33.611723+0000","last_became_active":"2024-03-29T10:09:32.625523+0000","last_became_peered":"2024-03-29T10:09:32.625523+0000","last_unstale":"2024-03-29T10:09:33.611723+0000","last_undegraded":"2024-03-29T10:09:33.611723+0000","last_fullsized":"2024-03-29T10:09:33.611723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:56.014772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046009+0000","last_change":"2024-03-29T10:09:32.615838+0000","last_active":"2024-03-29T10:09:34.046009+0000","last_peered":"2024-03-29T10:09:34.046009+0000","last_clean":"2024-03-29T10:09:34.046009+0000","last_became_active":"2024-03-29T10:09:32.615274+0000","last_became_peered":"2024-03-29T10:09:32.615274+0000","last_unstale":"2024-03-29T10:09:34.046009+0000","last_undegraded":"2024-03-29T10:09:34.046009+0000","last_fullsized":"2024-03-29T10:09:34.046009+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:07.635360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045741+0000","last_change":"2024-03-29T10:09:32.624374+0000","last_active":"2024-03-29T10:09:34.045741+0000","last_peered":"2024-03-29T10:09:34.045741+0000","last_clean":"2024-03-29T10:09:34.045741+0000","last_became_active":"2024-03-29T10:09:32.624101+0000","last_became_peered":"2024-03-29T10:09:32.624101+0000","last_unstale":"2024-03-29T10:09:34.045741+0000","last_undegraded":"2024-03-29T10:09:34.045741+0000","last_fullsized":"2024-03-29T10:09:34.045741+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:24:39.560229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045630+0000","last_change":"2024-03-29T10:09:32.618810+0000","last_active":"2024-03-29T10:09:34.045630+0000","last_peered":"2024-03-29T10:09:34.045630+0000","last_clean":"2024-03-29T10:09:34.045630+0000","last_became_active":"2024-03-29T10:09:32.618546+0000","last_became_peered":"2024-03-29T10:09:32.618546+0000","last_unstale":"2024-03-29T10:09:34.045630+0000","last_undegraded":"2024-03-29T10:09:34.045630+0000","last_fullsized":"2024-03-29T10:09:34.045630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:49.874012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607130+0000","last_change":"2024-03-29T10:09:32.618824+0000","last_active":"2024-03-29T10:09:33.607130+0000","last_peered":"2024-03-29T10:09:33.607130+0000","last_clean":"2024-03-29T10:09:33.607130+0000","last_became_active":"2024-03-29T10:09:32.618065+0000","last_became_peered":"2024-03-29T10:09:32.618065+0000","last_unstale":"2024-03-29T10:09:33.607130+0000","last_undegraded":"2024-03-29T10:09:33.607130+0000","last_fullsized":"2024-03-29T10:09:33.607130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:29.544771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045633+0000","last_change":"2024-03-29T10:09:32.622999+0000","last_active":"2024-03-29T10:09:34.045633+0000","last_peered":"2024-03-29T10:09:34.045633+0000","last_clean":"2024-03-29T10:09:34.045633+0000","last_became_active":"2024-03-29T10:09:32.622065+0000","last_became_peered":"2024-03-29T10:09:32.622065+0000","last_unstale":"2024-03-29T10:09:34.045633+0000","last_undegraded":"2024-03-29T10:09:34.045633+0000","last_fullsized":"2024-03-29T10:09:34.045633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:23.297716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044843+0000","last_change":"2024-03-29T10:09:32.615373+0000","last_active":"2024-03-29T10:09:34.044843+0000","last_peered":"2024-03-29T10:09:34.044843+0000","last_clean":"2024-03-29T10:09:34.044843+0000","last_became_active":"2024-03-29T10:09:32.615107+0000","last_became_peered":"2024-03-29T10:09:32.615107+0000","last_unstale":"2024-03-29T10:09:34.044843+0000","last_undegraded":"2024-03-29T10:09:34.044843+0000","last_fullsized":"2024-03-29T10:09:34.044843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:23.129448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045322+0000","last_change":"2024-03-29T10:09:32.618453+0000","last_active":"2024-03-29T10:09:34.045322+0000","last_peered":"2024-03-29T10:09:34.045322+0000","last_clean":"2024-03-29T10:09:34.045322+0000","last_became_active":"2024-03-29T10:09:32.616759+0000","last_became_peered":"2024-03-29T10:09:32.616759+0000","last_unstale":"2024-03-29T10:09:34.045322+0000","last_undegraded":"2024-03-29T10:09:34.045322+0000","last_fullsized":"2024-03-29T10:09:34.045322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:50.946405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045802+0000","last_change":"2024-03-29T10:09:32.623841+0000","last_active":"2024-03-29T10:09:34.045802+0000","last_peered":"2024-03-29T10:09:34.045802+0000","last_clean":"2024-03-29T10:09:34.045802+0000","last_became_active":"2024-03-29T10:09:32.623522+0000","last_became_peered":"2024-03-29T10:09:32.623522+0000","last_unstale":"2024-03-29T10:09:34.045802+0000","last_undegraded":"2024-03-29T10:09:34.045802+0000","last_fullsized":"2024-03-29T10:09:34.045802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:40.815771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"79'4","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638507+0000","last_change":"2024-03-29T10:09:32.101864+0000","last_active":"2024-03-29T10:09:32.638507+0000","last_peered":"2024-03-29T10:09:32.638507+0000","last_clean":"2024-03-29T10:09:32.638507+0000","last_became_active":"2024-03-29T10:09:32.101456+0000","last_became_peered":"2024-03-29T10:09:32.101456+0000","last_unstale":"2024-03-29T10:09:32.638507+0000","last_undegraded":"2024-03-29T10:09:32.638507+0000","last_fullsized":"2024-03-29T10:09:32.638507+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:05.714596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045625+0000","last_change":"2024-03-29T10:09:32.630347+0000","last_active":"2024-03-29T10:09:34.045625+0000","last_peered":"2024-03-29T10:09:34.045625+0000","last_clean":"2024-03-29T10:09:34.045625+0000","last_became_active":"2024-03-29T10:09:32.630092+0000","last_became_peered":"2024-03-29T10:09:32.630092+0000","last_unstale":"2024-03-29T10:09:34.045625+0000","last_undegraded":"2024-03-29T10:09:34.045625+0000","last_fullsized":"2024-03-29T10:09:34.045625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:50.745197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.635936+0000","last_change":"2024-03-29T10:09:32.091373+0000","last_active":"2024-03-29T10:09:32.635936+0000","last_peered":"2024-03-29T10:09:32.635936+0000","last_clean":"2024-03-29T10:09:32.635936+0000","last_became_active":"2024-03-29T10:09:32.091110+0000","last_became_peered":"2024-03-29T10:09:32.091110+0000","last_unstale":"2024-03-29T10:09:32.635936+0000","last_undegraded":"2024-03-29T10:09:32.635936+0000","last_fullsized":"2024-03-29T10:09:32.635936+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:32:31.535691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046042+0000","last_change":"2024-03-29T10:09:32.614418+0000","last_active":"2024-03-29T10:09:34.046042+0000","last_peered":"2024-03-29T10:09:34.046042+0000","last_clean":"2024-03-29T10:09:34.046042+0000","last_became_active":"2024-03-29T10:09:32.614112+0000","last_became_peered":"2024-03-29T10:09:32.614112+0000","last_unstale":"2024-03-29T10:09:34.046042+0000","last_undegraded":"2024-03-29T10:09:34.046042+0000","last_fullsized":"2024-03-29T10:09:34.046042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:17.345552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.635794+0000","last_change":"2024-03-29T10:09:32.086998+0000","last_active":"2024-03-29T10:09:32.635794+0000","last_peered":"2024-03-29T10:09:32.635794+0000","last_clean":"2024-03-29T10:09:32.635794+0000","last_became_active":"2024-03-29T10:09:32.086644+0000","last_became_peered":"2024-03-29T10:09:32.086644+0000","last_unstale":"2024-03-29T10:09:32.635794+0000","last_undegraded":"2024-03-29T10:09:32.635794+0000","last_fullsized":"2024-03-29T10:09:32.635794+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:27.370099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045835+0000","last_change":"2024-03-29T10:09:32.624387+0000","last_active":"2024-03-29T10:09:34.045835+0000","last_peered":"2024-03-29T10:09:34.045835+0000","last_clean":"2024-03-29T10:09:34.045835+0000","last_became_active":"2024-03-29T10:09:32.624015+0000","last_became_peered":"2024-03-29T10:09:32.624015+0000","last_unstale":"2024-03-29T10:09:34.045835+0000","last_undegraded":"2024-03-29T10:09:34.045835+0000","last_fullsized":"2024-03-29T10:09:34.045835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:06.669665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.072330+0000","last_change":"2024-03-29T10:09:31.072330+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.072330+0000","last_undegraded":"2024-03-29T10:09:31.072330+0000","last_fullsized":"2024-03-29T10:09:31.072330+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607151+0000","last_change":"2024-03-29T10:09:32.620748+0000","last_active":"2024-03-29T10:09:33.607151+0000","last_peered":"2024-03-29T10:09:33.607151+0000","last_clean":"2024-03-29T10:09:33.607151+0000","last_became_active":"2024-03-29T10:09:32.618821+0000","last_became_peered":"2024-03-29T10:09:32.618821+0000","last_unstale":"2024-03-29T10:09:33.607151+0000","last_undegraded":"2024-03-29T10:09:33.607151+0000","last_fullsized":"2024-03-29T10:09:33.607151+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:39:35.761571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.632616+0000","last_change":"2024-03-29T10:09:32.098141+0000","last_active":"2024-03-29T10:09:32.632616+0000","last_peered":"2024-03-29T10:09:32.632616+0000","last_clean":"2024-03-29T10:09:32.632616+0000","last_became_active":"2024-03-29T10:09:32.097781+0000","last_became_peered":"2024-03-29T10:09:32.097781+0000","last_unstale":"2024-03-29T10:09:32.632616+0000","last_undegraded":"2024-03-29T10:09:32.632616+0000","last_fullsized":"2024-03-29T10:09:32.632616+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607233+0000","last_change":"2024-03-29T10:09:32.614343+0000","last_active":"2024-03-29T10:09:33.607233+0000","last_peered":"2024-03-29T10:09:33.607233+0000","last_clean":"2024-03-29T10:09:33.607233+0000","last_became_active":"2024-03-29T10:09:32.614069+0000","last_became_peered":"2024-03-29T10:09:32.614069+0000","last_unstale":"2024-03-29T10:09:33.607233+0000","last_undegraded":"2024-03-29T10:09:33.607233+0000","last_fullsized":"2024-03-29T10:09:33.607233+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:22.440684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638211+0000","last_change":"2024-03-29T10:09:32.101572+0000","last_active":"2024-03-29T10:09:32.638211+0000","last_peered":"2024-03-29T10:09:32.638211+0000","last_clean":"2024-03-29T10:09:32.638211+0000","last_became_active":"2024-03-29T10:09:32.101206+0000","last_became_peered":"2024-03-29T10:09:32.101206+0000","last_unstale":"2024-03-29T10:09:32.638211+0000","last_undegraded":"2024-03-29T10:09:32.638211+0000","last_fullsized":"2024-03-29T10:09:32.638211+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:46.154988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.611858+0000","last_change":"2024-03-29T10:09:32.624716+0000","last_active":"2024-03-29T10:09:33.611858+0000","last_peered":"2024-03-29T10:09:33.611858+0000","last_clean":"2024-03-29T10:09:33.611858+0000","last_became_active":"2024-03-29T10:09:32.622676+0000","last_became_peered":"2024-03-29T10:09:32.622676+0000","last_unstale":"2024-03-29T10:09:33.611858+0000","last_undegraded":"2024-03-29T10:09:33.611858+0000","last_fullsized":"2024-03-29T10:09:33.611858+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:20.681595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.073144+0000","last_change":"2024-03-29T10:09:31.073144+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.073144+0000","last_undegraded":"2024-03-29T10:09:31.073144+0000","last_fullsized":"2024-03-29T10:09:31.073144+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607448+0000","last_change":"2024-03-29T10:09:32.634701+0000","last_active":"2024-03-29T10:09:33.607448+0000","last_peered":"2024-03-29T10:09:33.607448+0000","last_clean":"2024-03-29T10:09:33.607448+0000","last_became_active":"2024-03-29T10:09:32.631830+0000","last_became_peered":"2024-03-29T10:09:32.631830+0000","last_unstale":"2024-03-29T10:09:33.607448+0000","last_undegraded":"2024-03-29T10:09:33.607448+0000","last_fullsized":"2024-03-29T10:09:33.607448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:29.066092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.603049+0000","last_change":"2024-03-29T10:09:32.097646+0000","last_active":"2024-03-29T10:09:32.603049+0000","last_peered":"2024-03-29T10:09:32.603049+0000","last_clean":"2024-03-29T10:09:32.603049+0000","last_became_active":"2024-03-29T10:09:32.097364+0000","last_became_peered":"2024-03-29T10:09:32.097364+0000","last_unstale":"2024-03-29T10:09:32.603049+0000","last_undegraded":"2024-03-29T10:09:32.603049+0000","last_fullsized":"2024-03-29T10:09:32.603049+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045478+0000","last_change":"2024-03-29T10:09:32.617841+0000","last_active":"2024-03-29T10:09:34.045478+0000","last_peered":"2024-03-29T10:09:34.045478+0000","last_clean":"2024-03-29T10:09:34.045478+0000","last_became_active":"2024-03-29T10:09:32.616581+0000","last_became_peered":"2024-03-29T10:09:32.616581+0000","last_unstale":"2024-03-29T10:09:34.045478+0000","last_undegraded":"2024-03-29T10:09:34.045478+0000","last_fullsized":"2024-03-29T10:09:34.045478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:15.353895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.616868+0000","last_change":"2024-03-29T10:09:32.098837+0000","last_active":"2024-03-29T10:09:32.616868+0000","last_peered":"2024-03-29T10:09:32.616868+0000","last_clean":"2024-03-29T10:09:32.616868+0000","last_became_active":"2024-03-29T10:09:32.098517+0000","last_became_peered":"2024-03-29T10:09:32.098517+0000","last_unstale":"2024-03-29T10:09:32.616868+0000","last_undegraded":"2024-03-29T10:09:32.616868+0000","last_fullsized":"2024-03-29T10:09:32.616868+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:05.714596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045923+0000","last_change":"2024-03-29T10:09:32.636148+0000","last_active":"2024-03-29T10:09:34.045923+0000","last_peered":"2024-03-29T10:09:34.045923+0000","last_clean":"2024-03-29T10:09:34.045923+0000","last_became_active":"2024-03-29T10:09:32.635753+0000","last_became_peered":"2024-03-29T10:09:32.635753+0000","last_unstale":"2024-03-29T10:09:34.045923+0000","last_undegraded":"2024-03-29T10:09:34.045923+0000","last_fullsized":"2024-03-29T10:09:34.045923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:18.667325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.608025+0000","last_change":"2024-03-29T10:09:32.099480+0000","last_active":"2024-03-29T10:09:32.608025+0000","last_peered":"2024-03-29T10:09:32.608025+0000","last_clean":"2024-03-29T10:09:32.608025+0000","last_became_active":"2024-03-29T10:09:32.099133+0000","last_became_peered":"2024-03-29T10:09:32.099133+0000","last_unstale":"2024-03-29T10:09:32.608025+0000","last_undegraded":"2024-03-29T10:09:32.608025+0000","last_fullsized":"2024-03-29T10:09:32.608025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:13:16.625121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044664+0000","last_change":"2024-03-29T10:09:32.628412+0000","last_active":"2024-03-29T10:09:34.044664+0000","last_peered":"2024-03-29T10:09:34.044664+0000","last_clean":"2024-03-29T10:09:34.044664+0000","last_became_active":"2024-03-29T10:09:32.622661+0000","last_became_peered":"2024-03-29T10:09:32.622661+0000","last_unstale":"2024-03-29T10:09:34.044664+0000","last_undegraded":"2024-03-29T10:09:34.044664+0000","last_fullsized":"2024-03-29T10:09:34.044664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:23.129448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"79'3","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636825+0000","last_change":"2024-03-29T10:09:32.089351+0000","last_active":"2024-03-29T10:09:32.636825+0000","last_peered":"2024-03-29T10:09:32.636825+0000","last_clean":"2024-03-29T10:09:32.636825+0000","last_became_active":"2024-03-29T10:09:32.087924+0000","last_became_peered":"2024-03-29T10:09:32.087924+0000","last_unstale":"2024-03-29T10:09:32.636825+0000","last_undegraded":"2024-03-29T10:09:32.636825+0000","last_fullsized":"2024-03-29T10:09:32.636825+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:46.154988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045696+0000","last_change":"2024-03-29T10:09:32.632023+0000","last_active":"2024-03-29T10:09:34.045696+0000","last_peered":"2024-03-29T10:09:34.045696+0000","last_clean":"2024-03-29T10:09:34.045696+0000","last_became_active":"2024-03-29T10:09:32.631450+0000","last_became_peered":"2024-03-29T10:09:32.631450+0000","last_unstale":"2024-03-29T10:09:34.045696+0000","last_undegraded":"2024-03-29T10:09:34.045696+0000","last_fullsized":"2024-03-29T10:09:34.045696+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:24:39.560229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"79'6","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638720+0000","last_change":"2024-03-29T10:09:32.099892+0000","last_active":"2024-03-29T10:09:32.638720+0000","last_peered":"2024-03-29T10:09:32.638720+0000","last_clean":"2024-03-29T10:09:32.638720+0000","last_became_active":"2024-03-29T10:09:32.099626+0000","last_became_peered":"2024-03-29T10:09:32.099626+0000","last_unstale":"2024-03-29T10:09:32.638720+0000","last_undegraded":"2024-03-29T10:09:32.638720+0000","last_fullsized":"2024-03-29T10:09:32.638720+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636457+0000","last_change":"2024-03-29T10:09:32.100281+0000","last_active":"2024-03-29T10:09:32.636457+0000","last_peered":"2024-03-29T10:09:32.636457+0000","last_clean":"2024-03-29T10:09:32.636457+0000","last_became_active":"2024-03-29T10:09:32.100004+0000","last_became_peered":"2024-03-29T10:09:32.100004+0000","last_unstale":"2024-03-29T10:09:32.636457+0000","last_undegraded":"2024-03-29T10:09:32.636457+0000","last_fullsized":"2024-03-29T10:09:32.636457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:49.940180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045675+0000","last_change":"2024-03-29T10:09:32.624300+0000","last_active":"2024-03-29T10:09:34.045675+0000","last_peered":"2024-03-29T10:09:34.045675+0000","last_clean":"2024-03-29T10:09:34.045675+0000","last_became_active":"2024-03-29T10:09:32.624004+0000","last_became_peered":"2024-03-29T10:09:32.624004+0000","last_unstale":"2024-03-29T10:09:34.045675+0000","last_undegraded":"2024-03-29T10:09:34.045675+0000","last_fullsized":"2024-03-29T10:09:34.045675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:38.603035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607519+0000","last_change":"2024-03-29T10:09:32.091147+0000","last_active":"2024-03-29T10:09:32.607519+0000","last_peered":"2024-03-29T10:09:32.607519+0000","last_clean":"2024-03-29T10:09:32.607519+0000","last_became_active":"2024-03-29T10:09:32.089656+0000","last_became_peered":"2024-03-29T10:09:32.089656+0000","last_unstale":"2024-03-29T10:09:32.607519+0000","last_undegraded":"2024-03-29T10:09:32.607519+0000","last_fullsized":"2024-03-29T10:09:32.607519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:22.123223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044770+0000","last_change":"2024-03-29T10:09:32.623116+0000","last_active":"2024-03-29T10:09:34.044770+0000","last_peered":"2024-03-29T10:09:34.044770+0000","last_clean":"2024-03-29T10:09:34.044770+0000","last_became_active":"2024-03-29T10:09:32.622832+0000","last_became_peered":"2024-03-29T10:09:32.622832+0000","last_unstale":"2024-03-29T10:09:34.044770+0000","last_undegraded":"2024-03-29T10:09:34.044770+0000","last_fullsized":"2024-03-29T10:09:34.044770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:36.451327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.071636+0000","last_change":"2024-03-29T10:09:31.071636+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.071636+0000","last_undegraded":"2024-03-29T10:09:31.071636+0000","last_fullsized":"2024-03-29T10:09:31.071636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045252+0000","last_change":"2024-03-29T10:09:32.634186+0000","last_active":"2024-03-29T10:09:34.045252+0000","last_peered":"2024-03-29T10:09:34.045252+0000","last_clean":"2024-03-29T10:09:34.045252+0000","last_became_active":"2024-03-29T10:09:32.633693+0000","last_became_peered":"2024-03-29T10:09:32.633693+0000","last_unstale":"2024-03-29T10:09:34.045252+0000","last_undegraded":"2024-03-29T10:09:34.045252+0000","last_fullsized":"2024-03-29T10:09:34.045252+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:40.815771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"79'8","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.639114+0000","last_change":"2024-03-29T10:09:32.092854+0000","last_active":"2024-03-29T10:09:32.639114+0000","last_peered":"2024-03-29T10:09:32.639114+0000","last_clean":"2024-03-29T10:09:32.639114+0000","last_became_active":"2024-03-29T10:09:32.092591+0000","last_became_peered":"2024-03-29T10:09:32.092591+0000","last_unstale":"2024-03-29T10:09:32.639114+0000","last_undegraded":"2024-03-29T10:09:32.639114+0000","last_fullsized":"2024-03-29T10:09:32.639114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:40.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045937+0000","last_change":"2024-03-29T10:09:32.623867+0000","last_active":"2024-03-29T10:09:34.045937+0000","last_peered":"2024-03-29T10:09:34.045937+0000","last_clean":"2024-03-29T10:09:34.045937+0000","last_became_active":"2024-03-29T10:09:32.622284+0000","last_became_peered":"2024-03-29T10:09:32.622284+0000","last_unstale":"2024-03-29T10:09:34.045937+0000","last_undegraded":"2024-03-29T10:09:34.045937+0000","last_fullsized":"2024-03-29T10:09:34.045937+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:06.669665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636096+0000","last_change":"2024-03-29T10:09:32.097805+0000","last_active":"2024-03-29T10:09:32.636096+0000","last_peered":"2024-03-29T10:09:32.636096+0000","last_clean":"2024-03-29T10:09:32.636096+0000","last_became_active":"2024-03-29T10:09:32.097537+0000","last_became_peered":"2024-03-29T10:09:32.097537+0000","last_unstale":"2024-03-29T10:09:32.636096+0000","last_undegraded":"2024-03-29T10:09:32.636096+0000","last_fullsized":"2024-03-29T10:09:32.636096+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:35.016753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.634994+0000","last_change":"2024-03-29T10:09:32.104998+0000","last_active":"2024-03-29T10:09:32.634994+0000","last_peered":"2024-03-29T10:09:32.634994+0000","last_clean":"2024-03-29T10:09:32.634994+0000","last_became_active":"2024-03-29T10:09:32.104610+0000","last_became_peered":"2024-03-29T10:09:32.104610+0000","last_unstale":"2024-03-29T10:09:32.634994+0000","last_undegraded":"2024-03-29T10:09:32.634994+0000","last_fullsized":"2024-03-29T10:09:32.634994+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:38.244076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607169+0000","last_change":"2024-03-29T10:09:32.619003+0000","last_active":"2024-03-29T10:09:33.607169+0000","last_peered":"2024-03-29T10:09:33.607169+0000","last_clean":"2024-03-29T10:09:33.607169+0000","last_became_active":"2024-03-29T10:09:32.617951+0000","last_became_peered":"2024-03-29T10:09:32.617951+0000","last_unstale":"2024-03-29T10:09:33.607169+0000","last_undegraded":"2024-03-29T10:09:33.607169+0000","last_fullsized":"2024-03-29T10:09:33.607169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:00.709669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.072860+0000","last_change":"2024-03-29T10:09:31.072860+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.072860+0000","last_undegraded":"2024-03-29T10:09:31.072860+0000","last_fullsized":"2024-03-29T10:09:31.072860+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:27:35.691958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045708+0000","last_change":"2024-03-29T10:09:32.633244+0000","last_active":"2024-03-29T10:09:34.045708+0000","last_peered":"2024-03-29T10:09:34.045708+0000","last_clean":"2024-03-29T10:09:34.045708+0000","last_became_active":"2024-03-29T10:09:32.632943+0000","last_became_peered":"2024-03-29T10:09:32.632943+0000","last_unstale":"2024-03-29T10:09:34.045708+0000","last_undegraded":"2024-03-29T10:09:34.045708+0000","last_fullsized":"2024-03-29T10:09:34.045708+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:36.022978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.606805+0000","last_change":"2024-03-29T10:09:32.089495+0000","last_active":"2024-03-29T10:09:32.606805+0000","last_peered":"2024-03-29T10:09:32.606805+0000","last_clean":"2024-03-29T10:09:32.606805+0000","last_became_active":"2024-03-29T10:09:32.089228+0000","last_became_peered":"2024-03-29T10:09:32.089228+0000","last_unstale":"2024-03-29T10:09:32.606805+0000","last_undegraded":"2024-03-29T10:09:32.606805+0000","last_fullsized":"2024-03-29T10:09:32.606805+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:14.347670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636359+0000","last_change":"2024-03-29T10:09:32.095956+0000","last_active":"2024-03-29T10:09:32.636359+0000","last_peered":"2024-03-29T10:09:32.636359+0000","last_clean":"2024-03-29T10:09:32.636359+0000","last_became_active":"2024-03-29T10:09:32.093465+0000","last_became_peered":"2024-03-29T10:09:32.093465+0000","last_unstale":"2024-03-29T10:09:32.636359+0000","last_undegraded":"2024-03-29T10:09:32.636359+0000","last_fullsized":"2024-03-29T10:09:32.636359+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.686777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045534+0000","last_change":"2024-03-29T10:09:32.622232+0000","last_active":"2024-03-29T10:09:34.045534+0000","last_peered":"2024-03-29T10:09:34.045534+0000","last_clean":"2024-03-29T10:09:34.045534+0000","last_became_active":"2024-03-29T10:09:32.621815+0000","last_became_peered":"2024-03-29T10:09:32.621815+0000","last_unstale":"2024-03-29T10:09:34.045534+0000","last_undegraded":"2024-03-29T10:09:34.045534+0000","last_fullsized":"2024-03-29T10:09:34.045534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:54.720344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607356+0000","last_change":"2024-03-29T10:09:32.099575+0000","last_active":"2024-03-29T10:09:32.607356+0000","last_peered":"2024-03-29T10:09:32.607356+0000","last_clean":"2024-03-29T10:09:32.607356+0000","last_became_active":"2024-03-29T10:09:32.098434+0000","last_became_peered":"2024-03-29T10:09:32.098434+0000","last_unstale":"2024-03-29T10:09:32.607356+0000","last_undegraded":"2024-03-29T10:09:32.607356+0000","last_fullsized":"2024-03-29T10:09:32.607356+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:39.694641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044600+0000","last_change":"2024-03-29T10:09:32.623727+0000","last_active":"2024-03-29T10:09:34.044600+0000","last_peered":"2024-03-29T10:09:34.044600+0000","last_clean":"2024-03-29T10:09:34.044600+0000","last_became_active":"2024-03-29T10:09:32.623445+0000","last_became_peered":"2024-03-29T10:09:32.623445+0000","last_unstale":"2024-03-29T10:09:34.044600+0000","last_undegraded":"2024-03-29T10:09:34.044600+0000","last_fullsized":"2024-03-29T10:09:34.044600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:18.667325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.072820+0000","last_change":"2024-03-29T10:09:31.072820+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.072820+0000","last_undegraded":"2024-03-29T10:09:31.072820+0000","last_fullsized":"2024-03-29T10:09:31.072820+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:51:58.170982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.637133+0000","last_change":"2024-03-29T10:09:32.089390+0000","last_active":"2024-03-29T10:09:32.637133+0000","last_peered":"2024-03-29T10:09:32.637133+0000","last_clean":"2024-03-29T10:09:32.637133+0000","last_became_active":"2024-03-29T10:09:32.089088+0000","last_became_peered":"2024-03-29T10:09:32.089088+0000","last_unstale":"2024-03-29T10:09:32.637133+0000","last_undegraded":"2024-03-29T10:09:32.637133+0000","last_fullsized":"2024-03-29T10:09:32.637133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:11:34.599249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.609775+0000","last_change":"2024-03-29T10:09:32.101276+0000","last_active":"2024-03-29T10:09:32.609775+0000","last_peered":"2024-03-29T10:09:32.609775+0000","last_clean":"2024-03-29T10:09:32.609775+0000","last_became_active":"2024-03-29T10:09:32.100949+0000","last_became_peered":"2024-03-29T10:09:32.100949+0000","last_unstale":"2024-03-29T10:09:32.609775+0000","last_undegraded":"2024-03-29T10:09:32.609775+0000","last_fullsized":"2024-03-29T10:09:32.609775+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:40.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044751+0000","last_change":"2024-03-29T10:09:32.611596+0000","last_active":"2024-03-29T10:09:34.044751+0000","last_peered":"2024-03-29T10:09:34.044751+0000","last_clean":"2024-03-29T10:09:34.044751+0000","last_became_active":"2024-03-29T10:09:32.610142+0000","last_became_peered":"2024-03-29T10:09:32.610142+0000","last_unstale":"2024-03-29T10:09:34.044751+0000","last_undegraded":"2024-03-29T10:09:34.044751+0000","last_fullsized":"2024-03-29T10:09:34.044751+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:07.635360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605517+0000","last_change":"2024-03-29T10:09:32.092374+0000","last_active":"2024-03-29T10:09:32.605517+0000","last_peered":"2024-03-29T10:09:32.605517+0000","last_clean":"2024-03-29T10:09:32.605517+0000","last_became_active":"2024-03-29T10:09:32.091381+0000","last_became_peered":"2024-03-29T10:09:32.091381+0000","last_unstale":"2024-03-29T10:09:32.605517+0000","last_undegraded":"2024-03-29T10:09:32.605517+0000","last_fullsized":"2024-03-29T10:09:32.605517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:06.629135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046025+0000","last_change":"2024-03-29T10:09:32.620976+0000","last_active":"2024-03-29T10:09:34.046025+0000","last_peered":"2024-03-29T10:09:34.046025+0000","last_clean":"2024-03-29T10:09:34.046025+0000","last_became_active":"2024-03-29T10:09:32.619271+0000","last_became_peered":"2024-03-29T10:09:32.619271+0000","last_unstale":"2024-03-29T10:09:34.046025+0000","last_undegraded":"2024-03-29T10:09:34.046025+0000","last_fullsized":"2024-03-29T10:09:34.046025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.930784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.071841+0000","last_change":"2024-03-29T10:09:31.071841+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.071841+0000","last_undegraded":"2024-03-29T10:09:31.071841+0000","last_fullsized":"2024-03-29T10:09:31.071841+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:56.946534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607553+0000","last_change":"2024-03-29T10:09:32.618538+0000","last_active":"2024-03-29T10:09:33.607553+0000","last_peered":"2024-03-29T10:09:33.607553+0000","last_clean":"2024-03-29T10:09:33.607553+0000","last_became_active":"2024-03-29T10:09:32.617981+0000","last_became_peered":"2024-03-29T10:09:32.617981+0000","last_unstale":"2024-03-29T10:09:33.607553+0000","last_undegraded":"2024-03-29T10:09:33.607553+0000","last_fullsized":"2024-03-29T10:09:33.607553+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:50.197997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607488+0000","last_change":"2024-03-29T10:09:32.084059+0000","last_active":"2024-03-29T10:09:32.607488+0000","last_peered":"2024-03-29T10:09:32.607488+0000","last_clean":"2024-03-29T10:09:32.607488+0000","last_became_active":"2024-03-29T10:09:32.082631+0000","last_became_peered":"2024-03-29T10:09:32.082631+0000","last_unstale":"2024-03-29T10:09:32.607488+0000","last_undegraded":"2024-03-29T10:09:32.607488+0000","last_fullsized":"2024-03-29T10:09:32.607488+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:37:31.209166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045792+0000","last_change":"2024-03-29T10:09:32.628132+0000","last_active":"2024-03-29T10:09:34.045792+0000","last_peered":"2024-03-29T10:09:34.045792+0000","last_clean":"2024-03-29T10:09:34.045792+0000","last_became_active":"2024-03-29T10:09:32.627845+0000","last_became_peered":"2024-03-29T10:09:32.627845+0000","last_unstale":"2024-03-29T10:09:34.045792+0000","last_undegraded":"2024-03-29T10:09:34.045792+0000","last_fullsized":"2024-03-29T10:09:34.045792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:52.322978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.604249+0000","last_change":"2024-03-29T10:09:32.094306+0000","last_active":"2024-03-29T10:09:32.604249+0000","last_peered":"2024-03-29T10:09:32.604249+0000","last_clean":"2024-03-29T10:09:32.604249+0000","last_became_active":"2024-03-29T10:09:32.093998+0000","last_became_peered":"2024-03-29T10:09:32.093998+0000","last_unstale":"2024-03-29T10:09:32.604249+0000","last_undegraded":"2024-03-29T10:09:32.604249+0000","last_fullsized":"2024-03-29T10:09:32.604249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:17.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044480+0000","last_change":"2024-03-29T10:09:32.620507+0000","last_active":"2024-03-29T10:09:34.044480+0000","last_peered":"2024-03-29T10:09:34.044480+0000","last_clean":"2024-03-29T10:09:34.044480+0000","last_became_active":"2024-03-29T10:09:32.620186+0000","last_became_peered":"2024-03-29T10:09:32.620186+0000","last_unstale":"2024-03-29T10:09:34.044480+0000","last_undegraded":"2024-03-29T10:09:34.044480+0000","last_fullsized":"2024-03-29T10:09:34.044480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:49.874012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605757+0000","last_change":"2024-03-29T10:09:32.086897+0000","last_active":"2024-03-29T10:09:32.605757+0000","last_peered":"2024-03-29T10:09:32.605757+0000","last_clean":"2024-03-29T10:09:32.605757+0000","last_became_active":"2024-03-29T10:09:32.086602+0000","last_became_peered":"2024-03-29T10:09:32.086602+0000","last_unstale":"2024-03-29T10:09:32.605757+0000","last_undegraded":"2024-03-29T10:09:32.605757+0000","last_fullsized":"2024-03-29T10:09:32.605757+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:23:18.397722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607519+0000","last_change":"2024-03-29T10:09:32.633167+0000","last_active":"2024-03-29T10:09:33.607519+0000","last_peered":"2024-03-29T10:09:33.607519+0000","last_clean":"2024-03-29T10:09:33.607519+0000","last_became_active":"2024-03-29T10:09:32.632682+0000","last_became_peered":"2024-03-29T10:09:32.632682+0000","last_unstale":"2024-03-29T10:09:33.607519+0000","last_undegraded":"2024-03-29T10:09:33.607519+0000","last_fullsized":"2024-03-29T10:09:33.607519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:07.268799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605003+0000","last_change":"2024-03-29T10:09:32.092768+0000","last_active":"2024-03-29T10:09:32.605003+0000","last_peered":"2024-03-29T10:09:32.605003+0000","last_clean":"2024-03-29T10:09:32.605003+0000","last_became_active":"2024-03-29T10:09:32.092488+0000","last_became_peered":"2024-03-29T10:09:32.092488+0000","last_unstale":"2024-03-29T10:09:32.605003+0000","last_undegraded":"2024-03-29T10:09:32.605003+0000","last_fullsized":"2024-03-29T10:09:32.605003+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:16.339327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.606964+0000","last_change":"2024-03-29T10:09:32.628533+0000","last_active":"2024-03-29T10:09:33.606964+0000","last_peered":"2024-03-29T10:09:33.606964+0000","last_clean":"2024-03-29T10:09:33.606964+0000","last_became_active":"2024-03-29T10:09:32.628297+0000","last_became_peered":"2024-03-29T10:09:32.628297+0000","last_unstale":"2024-03-29T10:09:33.606964+0000","last_undegraded":"2024-03-29T10:09:33.606964+0000","last_fullsized":"2024-03-29T10:09:33.606964+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:53:16.735792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.604907+0000","last_change":"2024-03-29T10:09:32.088461+0000","last_active":"2024-03-29T10:09:32.604907+0000","last_peered":"2024-03-29T10:09:32.604907+0000","last_clean":"2024-03-29T10:09:32.604907+0000","last_became_active":"2024-03-29T10:09:32.087924+0000","last_became_peered":"2024-03-29T10:09:32.087924+0000","last_unstale":"2024-03-29T10:09:32.604907+0000","last_undegraded":"2024-03-29T10:09:32.604907+0000","last_fullsized":"2024-03-29T10:09:32.604907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:01.534207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"68'161","reported_seq":379,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.613595+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:09:32.613595+0000","last_peered":"2024-03-29T10:09:32.613595+0000","last_clean":"2024-03-29T10:09:32.613595+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:09:32.613595+0000","last_undegraded":"2024-03-29T10:09:32.613595+0000","last_fullsized":"2024-03-29T10:09:32.613595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":112,"acting":112,"num_store_stats":7},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2292,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1568768,"data_stored":1047510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.57299999999999995,"5min":0.44500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95299999999999996,"5min":1.0029999999999999,"15min":1.0109999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.69999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.89800000000000002,"5min":0.93400000000000005,"15min":1.046},"last":0.72299999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71699999999999997,"15min":0.71999999999999997},"min":{"1min":0.51900000000000002,"5min":0.47399999999999998,"15min":0.44500000000000001},"max":{"1min":0.92600000000000005,"5min":0.93200000000000005,"15min":1.1910000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.73499999999999999},"min":{"1min":0.51700000000000002,"5min":0.42199999999999999,"15min":0.372},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1},"last":0.71099999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.72899999999999998,"15min":0.74099999999999999},"min":{"1min":0.47199999999999998,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.046},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.755,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":1.079},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.48099999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.93100000000000005,"5min":0.97699999999999998,"15min":1.081},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77500000000000002,"15min":0.78300000000000003},"min":{"1min":0.55800000000000005,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":0.96199999999999997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.74199999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.29199999999999998},"max":{"1min":0.86399999999999999,"5min":0.87,"15min":1.111},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.627,"15min":0.64100000000000001},"min":{"1min":0.44800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":0.95199999999999996,"15min":0.95599999999999996},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64900000000000002,"15min":0.65900000000000003},"min":{"1min":0.432,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95299999999999996,"5min":0.96099999999999997,"15min":1.0089999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67600000000000005,"15min":0.68899999999999995},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90100000000000002,"5min":0.92800000000000005,"15min":1.141},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.71899999999999997,"15min":0.73299999999999998},"min":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.49199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":1.1679999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69699999999999995,"15min":0.70499999999999996},"min":{"1min":0.54800000000000004,"5min":0.48499999999999999,"15min":0.435},"max":{"1min":0.93799999999999994,"5min":0.98399999999999999,"15min":1.121},"last":0.48499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575031,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":966656,"data_stored":456982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70299999999999996,"15min":0.745},"min":{"1min":0.57199999999999995,"5min":0.42799999999999999,"15min":0.34699999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":7.4080000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.72599999999999998,"15min":0.76900000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.42499999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":6.7610000000000001},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70999999999999996,"15min":0.71899999999999997},"min":{"1min":0.51300000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.2549999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.72499999999999998,"15min":0.71999999999999997},"min":{"1min":0.58999999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":0.995,"15min":1.0209999999999999},"last":0.745}]},{"osd":2,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.71699999999999997,"15min":0.72399999999999998},"min":{"1min":0.65900000000000003,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94199999999999995,"5min":0.96899999999999997,"15min":1.3169999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.58899999999999997,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.415},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.74099999999999999,"15min":0.76900000000000002},"min":{"1min":0.60399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.98999999999999999,"5min":1.02,"15min":3.1960000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76800000000000002,"15min":0.79800000000000004},"min":{"1min":0.71499999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0109999999999999,"5min":1.014,"15min":4.3760000000000003},"last":0.70499999999999996}]},{"osd":4,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":1.109},"last":0.753},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.47799999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85899999999999999,"5min":0.95599999999999996,"15min":3.1219999999999999},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.67300000000000004},"min":{"1min":0.50600000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90000000000000002,"5min":0.90900000000000003,"15min":2.8279999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.79500000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.74399999999999999,"15min":0.76100000000000001},"min":{"1min":0.56499999999999995,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.95499999999999996,"5min":1.105,"15min":1.2769999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.496,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94499999999999995,"5min":0.998,"15min":1.7869999999999999},"last":0.76200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50116,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321724,"statfs":{"total":96636764160,"available":96585445376,"internally_reserved":0,"allocated":970752,"data_stored":457072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.315,"5min":0.219,"15min":0.219},"max":{"1min":0.65300000000000002,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.379},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.68999999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.504}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71599999999999997,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.52900000000000003},"min":{"1min":0.36699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":4.282},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.70999999999999996,"5min":0.78500000000000003,"15min":4.3200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.36599999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.016,"5min":1.617,"15min":1.617},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76400000000000001,"15min":0.78800000000000003},"min":{"1min":0.60899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0289999999999999,"5min":1.1419999999999999,"15min":4.3559999999999999},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.51800000000000002},"max":{"1min":1.056,"5min":1.3939999999999999,"15min":1.9119999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78600000000000003,"15min":0.80800000000000005},"min":{"1min":0.56299999999999994,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.048,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77800000000000002,"15min":0.81000000000000005},"min":{"1min":0.57499999999999996,"5min":0.52600000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":1.194,"15min":4.3899999999999997},"last":0.752},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.093,"5min":1.468,"15min":1.468},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.62,"5min":0.58699999999999997,"15min":0.56200000000000006},"max":{"1min":1.127,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.83599999999999997},"min":{"1min":0.60999999999999999,"5min":0.57099999999999995,"15min":0.55700000000000005},"max":{"1min":1.0269999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.73699999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1572864,"data_stored":1048477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.33300000000000002,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.497,"15min":0.51000000000000001},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72199999999999998,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.628}]},{"osd":2,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.435,"5min":0.39200000000000002,"15min":0.35499999999999998},"max":{"1min":0.78800000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.48799999999999999,"15min":0.496},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.63400000000000001,"5min":0.83899999999999997,"15min":0.97199999999999998},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.39000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73599999999999999,"5min":0.83199999999999996,"15min":1.077},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.47199999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83499999999999996,"5min":0.876,"15min":1.0549999999999999},"last":0.5}]},{"osd":4,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.81399999999999995,"15min":0.80200000000000005},"min":{"1min":0.65100000000000002,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":1.0469999999999999,"5min":1.625,"15min":1.625},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.82399999999999995},"min":{"1min":0.66900000000000004,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":0.97099999999999997,"5min":1.1419999999999999,"15min":1.167},"last":0.73399999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.752,"15min":0.75900000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98999999999999999,"5min":1.137,"15min":1.137},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.80600000000000005,"15min":0.80300000000000005},"min":{"1min":0.60399999999999998,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":0.90200000000000002,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.752}]},{"osd":6,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.53400000000000003},"max":{"1min":1.0129999999999999,"5min":1.0640000000000001,"15min":1.228},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.85499999999999998,"15min":0.84399999999999997},"min":{"1min":0.66400000000000003,"5min":0.52600000000000002,"15min":0.46899999999999997},"max":{"1min":1.0249999999999999,"5min":1.601,"15min":1.601},"last":0.80100000000000005}]},{"osd":7,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78300000000000003,"15min":0.79600000000000004},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.97299999999999998,"5min":1.0529999999999999,"15min":1.1890000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.84699999999999998,"15min":0.84199999999999997},"min":{"1min":0.64000000000000001,"5min":0.57499999999999996,"15min":0.49299999999999999},"max":{"1min":1.046,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.84199999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36172,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335668,"statfs":{"total":96636764160,"available":96599724032,"internally_reserved":0,"allocated":974848,"data_stored":457038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.34699999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.623,"5min":0.69599999999999995,"15min":0.86599999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.39300000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":1.3140000000000001},"last":0.47999999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47299999999999998,"15min":0.46999999999999997},"min":{"1min":0.376,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.63600000000000001,"5min":0.751,"15min":0.94499999999999995},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.45800000000000002,"5min":0.36099999999999999,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":1.032,"15min":1.032},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.504,"5min":0.375,"15min":0.375},"max":{"1min":0.73799999999999999,"5min":0.80000000000000004,"15min":1.127},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.48999999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70899999999999996,"5min":0.82399999999999995,"15min":1.1479999999999999},"last":0.51800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.77300000000000002,"15min":0.78600000000000003},"min":{"1min":0.68700000000000006,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":1.0760000000000001,"5min":1.1799999999999999,"15min":1.554},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.79500000000000004,"15min":0.81000000000000005},"min":{"1min":0.60799999999999998,"5min":0.55000000000000004,"15min":0.51600000000000001},"max":{"1min":1.0309999999999999,"5min":1.236,"15min":1.577},"last":0.86099999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.81000000000000005},"min":{"1min":0.64700000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0680000000000001,"5min":1.25,"15min":1.5209999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79600000000000004,"15min":0.80800000000000005},"min":{"1min":0.58799999999999997,"5min":0.53200000000000003,"15min":0.51800000000000002},"max":{"1min":1.0049999999999999,"5min":1.2250000000000001,"15min":1.3300000000000001},"last":0.77700000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.80400000000000005,"15min":0.82799999999999996},"min":{"1min":0.65200000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1399999999999999,"5min":1.1539999999999999,"15min":1.758},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.874,"5min":0.83699999999999997,"15min":0.84199999999999997},"min":{"1min":0.65900000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1080000000000001,"5min":1.3560000000000001,"15min":1.4750000000000001},"last":0.90900000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.86399999999999999,"15min":0.86899999999999999},"min":{"1min":0.73599999999999999,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.137,"5min":1.2709999999999999,"15min":1.4490000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.69999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.107,"5min":1.3049999999999999,"15min":1.4390000000000001},"last":0.96799999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575030,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36172,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335668,"statfs":{"total":96636764160,"available":96599724032,"internally_reserved":0,"allocated":974848,"data_stored":457038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.46600000000000003},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":0.79500000000000004,"15min":4.3799999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52600000000000002,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":4.6379999999999999},"last":0.621}]},{"osd":1,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.40200000000000002,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.69799999999999995,"5min":0.98699999999999999,"15min":4.1840000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.314},"max":{"1min":0.80800000000000005,"5min":0.96099999999999997,"15min":1.048},"last":0.67600000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.47799999999999998,"5min":0.373,"15min":0.33700000000000002},"max":{"1min":0.77200000000000002,"5min":0.82499999999999996,"15min":1.369},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.63800000000000001,"15min":0.64700000000000002},"min":{"1min":0.55700000000000005,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.79700000000000004,"5min":0.94799999999999995,"15min":4.5970000000000004},"last":0.748}]},{"osd":4,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76800000000000002,"15min":0.78100000000000003},"min":{"1min":0.64400000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.0940000000000001,"5min":1.288,"15min":4.3319999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77600000000000002,"15min":0.77500000000000002},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.39400000000000002},"max":{"1min":0.92400000000000004,"5min":1.46,"15min":1.46},"last":1.0900000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78200000000000003,"15min":0.79500000000000004},"min":{"1min":0.628,"5min":0.48899999999999999,"15min":0.435},"max":{"1min":0.97599999999999998,"5min":1.3580000000000001,"15min":1.3740000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79800000000000004,"15min":0.81000000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.44500000000000001},"max":{"1min":1.133,"5min":1.397,"15min":4.4180000000000001},"last":0.876}]},{"osd":6,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81399999999999995,"15min":0.81200000000000006},"min":{"1min":0.68899999999999995,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.0009999999999999,"5min":1.2549999999999999,"15min":1.4730000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81699999999999995,"15min":0.81899999999999995},"min":{"1min":0.65000000000000002,"5min":0.52400000000000002,"15min":0.503},"max":{"1min":1.0409999999999999,"5min":1.23,"15min":1.417},"last":1.0349999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.83999999999999997,"15min":0.84799999999999998},"min":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.54300000000000004},"max":{"1min":1.135,"5min":1.5,"15min":4.4550000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.64400000000000002,"5min":0.59899999999999998,"15min":0.56200000000000006},"max":{"1min":1.056,"5min":1.345,"15min":1.476},"last":1.0489999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54224,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317616,"statfs":{"total":96636764160,"available":96581238784,"internally_reserved":0,"allocated":978944,"data_stored":458546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.68700000000000006},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.88200000000000001,"5min":0.93200000000000005,"15min":1.417},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.92100000000000004,"5min":1.165,"15min":1.165},"last":0.434}]},{"osd":1,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.93500000000000005,"5min":0.93700000000000006,"15min":1.139},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.333},"last":0.55700000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.01,"15min":1.863},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.75,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93700000000000006,"5min":0.96299999999999997,"15min":1.387},"last":0.50600000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.5,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.88700000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.495},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.51600000000000001,"5min":0.48699999999999999,"15min":0.46600000000000003},"max":{"1min":0.80500000000000005,"5min":1.1539999999999999,"15min":1.905},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.46700000000000003,"5min":0.39300000000000002,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.48099999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.78500000000000003,"5min":1.294,"15min":1.294},"last":0.629}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.48099999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.75900000000000001,"5min":1.0129999999999999,"15min":1.9339999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.66200000000000003},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.433},"max":{"1min":0.85299999999999998,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.48099999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85999999999999999,"5min":1.345,"15min":1.345},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.59099999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89700000000000002,"5min":1.214,"15min":1.97},"last":0.755}]}]},{"osd":5,"up_from":13,"seq":55834575029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":929792,"data_stored":437640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69599999999999995,"15min":0.68200000000000005},"min":{"1min":0.432,"5min":0.41599999999999998,"15min":0.313},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.746,"15min":0.71899999999999997},"min":{"1min":0.45900000000000002,"5min":0.45500000000000002,"15min":0.33500000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.339},"last":0.435}]},{"osd":1,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.55400000000000005,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.0169999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72199999999999998,"15min":0.70499999999999996},"min":{"1min":0.47999999999999998,"5min":0.46700000000000003,"15min":0.39200000000000002},"max":{"1min":0.95199999999999996,"5min":1.022,"15min":1.6040000000000001},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73099999999999998,"15min":0.72799999999999998},"min":{"1min":0.57099999999999995,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.476},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.59299999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93700000000000006,"5min":1.095,"15min":1.502},"last":0.46200000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.63200000000000001,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":0.95799999999999996,"5min":1.113,"15min":1.1619999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.754},"min":{"1min":0.56499999999999995,"5min":0.52300000000000002,"15min":0.46500000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.744},"last":0.64900000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.80300000000000005,"5min":0.94599999999999995,"15min":0.95899999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.214},"last":0.495}]},{"osd":6,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64800000000000002,"15min":0.65400000000000003},"min":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.36799999999999999},"max":{"1min":0.85999999999999999,"5min":0.91300000000000003,"15min":1.151},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.52600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":0.97099999999999997,"15min":1.5489999999999999},"last":0.70599999999999996}]},{"osd":7,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.70499999999999996,"15min":0.71099999999999997},"min":{"1min":0.51500000000000001,"5min":0.41899999999999998,"15min":0.40200000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.325},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.53800000000000003,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":0.96999999999999997,"5min":0.98199999999999998,"15min":1.1990000000000001},"last":0.79700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:09:37.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:09:37.053 INFO:tasks.ceph.mds.b.smithi159.stderr:2024-03-29T10:09:37.047+0000 7f5993a00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:09:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.284+0000 7fbb47400700 1 Processor -- start 2024-03-29T10:09:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.284+0000 7fbb47400700 1 -- start start 2024-03-29T10:09:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.285+0000 7fbb47400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 0x7fbb401612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.285+0000 7fbb47400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbb40161800 0x7fbb400b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.285+0000 7fbb47400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb400d30d0 con 0x7fbb4015d640 2024-03-29T10:09:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.285+0000 7fbb47400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb40161c90 con 0x7fbb40161800 2024-03-29T10:09:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.285+0000 7fbb47400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400b6840 con 0x7fbb400ceec0 2024-03-29T10:09:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.285+0000 7fbb45400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbb40161800 0x7fbb400b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.286+0000 7fbb45400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbb40161800 0x7fbb400b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58232/0 (socket says 172.21.15.12:58232) 2024-03-29T10:09:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.286+0000 7fbb45400700 1 -- 172.21.15.12:0/991443053 learned_addr learned my addr 172.21.15.12:0/991443053 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.286+0000 7fbb3fe00700 1 --2- 172.21.15.12:0/991443053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 0x7fbb401612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.286+0000 7fbb45400700 1 -- 172.21.15.12:0/991443053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 msgr2=0x7fbb401612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.286+0000 7fbb45400700 1 --2- 172.21.15.12:0/991443053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 0x7fbb401612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.286+0000 7fbb45400700 1 -- 172.21.15.12:0/991443053 >> v1:172.21.15.159:6789/0 conn(0x7fbb4015d640 legacy=0x7fbb400d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.286+0000 7fbb45400700 1 -- 172.21.15.12:0/991443053 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb400d2b30 con 0x7fbb40161800 2024-03-29T10:09:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.286+0000 7fbb3fe00700 1 --2- 172.21.15.12:0/991443053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 0x7fbb401612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.286+0000 7fbb45400700 1 --2- 172.21.15.12:0/991443053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 0x7fbb400b6cd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009e50 tx=0x7fbb3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d759e99143d384 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.286+0000 7fbb3f400700 1 -- 172.21.15.12:0/991443053 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3800e070 con 0x7fbb40161800 2024-03-29T10:09:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.287+0000 7fbb3f400700 1 -- 172.21.15.12:0/991443053 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3800b9c0 con 0x7fbb40161800 2024-03-29T10:09:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.287+0000 7fbb3f400700 1 -- 172.21.15.12:0/991443053 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3800bd70 con 0x7fbb40161800 2024-03-29T10:09:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.287+0000 7fbb47400700 1 -- 172.21.15.12:0/991443053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 msgr2=0x7fbb400b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.287+0000 7fbb47400700 1 --2- 172.21.15.12:0/991443053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 0x7fbb400b6cd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009e50 tx=0x7fbb3800b330 comp rx=0 tx=0).stop 2024-03-29T10:09:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.287+0000 7fbb47400700 1 -- 172.21.15.12:0/991443053 shutdown_connections 2024-03-29T10:09:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.287+0000 7fbb47400700 1 --2- 172.21.15.12:0/991443053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 0x7fbb400b6cd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.287+0000 7fbb47400700 1 --2- 172.21.15.12:0/991443053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 0x7fbb401612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.287+0000 7fbb47400700 1 -- 172.21.15.12:0/991443053 >> 172.21.15.12:0/991443053 conn(0x7fbb4008aff0 msgr2=0x7fbb400d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.287+0000 7fbb47400700 1 -- 172.21.15.12:0/991443053 shutdown_connections 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.287+0000 7fbb47400700 1 -- 172.21.15.12:0/991443053 wait complete. 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.289+0000 7fbb47400700 1 Processor -- start 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.289+0000 7fbb47400700 1 -- start start 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.290+0000 7fbb47400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 0x7fbb4015ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.290+0000 7fbb47400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb4015d640 0x7fbb400ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.290+0000 7fbb47400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 0x7fbb400cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.290+0000 7fbb47400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4015f670 con 0x7fbb40161800 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.290+0000 7fbb47400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400ce7e0 con 0x7fbb4015d640 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.290+0000 7fbb47400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400cad20 con 0x7fbb400ceec0 2024-03-29T10:09:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb45400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 0x7fbb400cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb45400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 0x7fbb400cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58240/0 (socket says 172.21.15.12:58240) 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb45400700 1 -- 172.21.15.12:0/2648440985 learned_addr learned my addr 172.21.15.12:0/2648440985 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb44a00700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 0x7fbb4015ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb3fe00700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb4015d640 0x7fbb400ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb45400700 1 -- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 msgr2=0x7fbb4015ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb45400700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 0x7fbb4015ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb45400700 1 -- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb4015d640 msgr2=0x7fbb400ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb45400700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb4015d640 0x7fbb400ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb45400700 1 -- 172.21.15.12:0/2648440985 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb38009b30 con 0x7fbb40161800 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.291+0000 7fbb45400700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 0x7fbb400cc180 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbb4008c490 tx=0x7fbb38004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.292+0000 7fbb3cc00700 1 -- 172.21.15.12:0/2648440985 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb38004840 con 0x7fbb40161800 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.292+0000 7fbb3cc00700 1 -- 172.21.15.12:0/2648440985 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb380092e0 con 0x7fbb40161800 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.292+0000 7fbb47400700 1 -- 172.21.15.12:0/2648440985 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb400cde40 con 0x7fbb40161800 2024-03-29T10:09:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.292+0000 7fbb3cc00700 1 -- 172.21.15.12:0/2648440985 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3801c660 con 0x7fbb40161800 2024-03-29T10:09:37.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.292+0000 7fbb47400700 1 -- 172.21.15.12:0/2648440985 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb400ce400 con 0x7fbb40161800 2024-03-29T10:09:37.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.293+0000 7fbb47400700 1 -- 172.21.15.12:0/2648440985 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb400ce920 con 0x7fbb40161800 2024-03-29T10:09:37.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.294+0000 7fbb3cc00700 1 -- 172.21.15.12:0/2648440985 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbb3801c7c0 con 0x7fbb40161800 2024-03-29T10:09:37.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.295+0000 7fbb3cc00700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb2006adb0 0x7fbb2006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.295+0000 7fbb3cc00700 1 -- 172.21.15.12:0/2648440985 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb3808b1f0 con 0x7fbb40161800 2024-03-29T10:09:37.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.295+0000 7fbb44a00700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb2006adb0 0x7fbb2006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.296+0000 7fbb44a00700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb2006adb0 0x7fbb2006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb34007fd0 tx=0x7fbb34009040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:37.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.300+0000 7fbb3cc00700 1 -- 172.21.15.12:0/2648440985 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbb400ce920 con 0x7fbb40161800 2024-03-29T10:09:37.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.509+0000 7fbb47400700 1 -- 172.21.15.12:0/2648440985 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb400b64d0 con 0x7fbb2006adb0 2024-03-29T10:09:37.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.518+0000 7fbb3cc00700 1 -- 172.21.15.12:0/2648440985 <== mgr.6013 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287655 (secure 0 0 0) 0x7fbb34001560 con 0x7fbb2006adb0 2024-03-29T10:09:37.521 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:37.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 -- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb2006adb0 msgr2=0x7fbb2006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb2006adb0 0x7fbb2006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb34007fd0 tx=0x7fbb34009040 comp rx=0 tx=0).stop 2024-03-29T10:09:37.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 -- 172.21.15.12:0/2648440985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 msgr2=0x7fbb400cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 0x7fbb400cc180 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbb4008c490 tx=0x7fbb38004ad0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 -- 172.21.15.12:0/2648440985 shutdown_connections 2024-03-29T10:09:37.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbb2006adb0 0x7fbb2006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb40161800 0x7fbb400cc180 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb400ceec0 0x7fbb4015ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 --2- 172.21.15.12:0/2648440985 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb4015d640 0x7fbb400ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 -- 172.21.15.12:0/2648440985 >> 172.21.15.12:0/2648440985 conn(0x7fbb4008aff0 msgr2=0x7fbb4006cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:37.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 -- 172.21.15.12:0/2648440985 shutdown_connections 2024-03-29T10:09:37.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.523+0000 7fbb47400700 1 -- 172.21.15.12:0/2648440985 wait complete. 2024-03-29T10:09:37.525 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:09:37.540 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-29T10:09:36.838605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592660,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":279,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":178,"acting":178,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":306076,"kb_used_data":8728,"kb_used_omap":12,"kb_used_meta":297203,"kb_avail":754668644,"statfs":{"total":773094113280,"available":772780691456,"internally_reserved":0,"allocated":8937472,"data_stored":4820303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":304336441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2292,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.769510"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045584+0000","last_change":"2024-03-29T10:09:32.623236+0000","last_active":"2024-03-29T10:09:34.045584+0000","last_peered":"2024-03-29T10:09:34.045584+0000","last_clean":"2024-03-29T10:09:34.045584+0000","last_became_active":"2024-03-29T10:09:32.622714+0000","last_became_peered":"2024-03-29T10:09:32.622714+0000","last_unstale":"2024-03-29T10:09:34.045584+0000","last_undegraded":"2024-03-29T10:09:34.045584+0000","last_fullsized":"2024-03-29T10:09:34.045584+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:50.745197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607572+0000","last_change":"2024-03-29T10:09:32.620014+0000","last_active":"2024-03-29T10:09:33.607572+0000","last_peered":"2024-03-29T10:09:33.607572+0000","last_clean":"2024-03-29T10:09:33.607572+0000","last_became_active":"2024-03-29T10:09:32.619747+0000","last_became_peered":"2024-03-29T10:09:32.619747+0000","last_unstale":"2024-03-29T10:09:33.607572+0000","last_undegraded":"2024-03-29T10:09:33.607572+0000","last_fullsized":"2024-03-29T10:09:33.607572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:58:19.387293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045306+0000","last_change":"2024-03-29T10:09:32.622331+0000","last_active":"2024-03-29T10:09:34.045306+0000","last_peered":"2024-03-29T10:09:34.045306+0000","last_clean":"2024-03-29T10:09:34.045306+0000","last_became_active":"2024-03-29T10:09:32.622038+0000","last_became_peered":"2024-03-29T10:09:32.622038+0000","last_unstale":"2024-03-29T10:09:34.045306+0000","last_undegraded":"2024-03-29T10:09:34.045306+0000","last_fullsized":"2024-03-29T10:09:34.045306+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:54.413063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045348+0000","last_change":"2024-03-29T10:09:32.617524+0000","last_active":"2024-03-29T10:09:34.045348+0000","last_peered":"2024-03-29T10:09:34.045348+0000","last_clean":"2024-03-29T10:09:34.045348+0000","last_became_active":"2024-03-29T10:09:32.617270+0000","last_became_peered":"2024-03-29T10:09:32.617270+0000","last_unstale":"2024-03-29T10:09:34.045348+0000","last_undegraded":"2024-03-29T10:09:34.045348+0000","last_fullsized":"2024-03-29T10:09:34.045348+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:02.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045118+0000","last_change":"2024-03-29T10:09:32.616253+0000","last_active":"2024-03-29T10:09:34.045118+0000","last_peered":"2024-03-29T10:09:34.045118+0000","last_clean":"2024-03-29T10:09:34.045118+0000","last_became_active":"2024-03-29T10:09:32.616002+0000","last_became_peered":"2024-03-29T10:09:32.616002+0000","last_unstale":"2024-03-29T10:09:34.045118+0000","last_undegraded":"2024-03-29T10:09:34.045118+0000","last_fullsized":"2024-03-29T10:09:34.045118+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:23.297716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044899+0000","last_change":"2024-03-29T10:09:32.612351+0000","last_active":"2024-03-29T10:09:34.044899+0000","last_peered":"2024-03-29T10:09:34.044899+0000","last_clean":"2024-03-29T10:09:34.044899+0000","last_became_active":"2024-03-29T10:09:32.612057+0000","last_became_peered":"2024-03-29T10:09:32.612057+0000","last_unstale":"2024-03-29T10:09:34.044899+0000","last_undegraded":"2024-03-29T10:09:34.044899+0000","last_fullsized":"2024-03-29T10:09:34.044899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:41.040149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.606905+0000","last_change":"2024-03-29T10:09:32.615482+0000","last_active":"2024-03-29T10:09:33.606905+0000","last_peered":"2024-03-29T10:09:33.606905+0000","last_clean":"2024-03-29T10:09:33.606905+0000","last_became_active":"2024-03-29T10:09:32.615198+0000","last_became_peered":"2024-03-29T10:09:32.615198+0000","last_unstale":"2024-03-29T10:09:33.606905+0000","last_undegraded":"2024-03-29T10:09:33.606905+0000","last_fullsized":"2024-03-29T10:09:33.606905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:48:35.803134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045007+0000","last_change":"2024-03-29T10:09:32.611215+0000","last_active":"2024-03-29T10:09:34.045007+0000","last_peered":"2024-03-29T10:09:34.045007+0000","last_clean":"2024-03-29T10:09:34.045007+0000","last_became_active":"2024-03-29T10:09:32.610941+0000","last_became_peered":"2024-03-29T10:09:32.610941+0000","last_unstale":"2024-03-29T10:09:34.045007+0000","last_undegraded":"2024-03-29T10:09:34.045007+0000","last_fullsized":"2024-03-29T10:09:34.045007+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:28.376324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045092+0000","last_change":"2024-03-29T10:09:32.624355+0000","last_active":"2024-03-29T10:09:34.045092+0000","last_peered":"2024-03-29T10:09:34.045092+0000","last_clean":"2024-03-29T10:09:34.045092+0000","last_became_active":"2024-03-29T10:09:32.622547+0000","last_became_peered":"2024-03-29T10:09:32.622547+0000","last_unstale":"2024-03-29T10:09:34.045092+0000","last_undegraded":"2024-03-29T10:09:34.045092+0000","last_fullsized":"2024-03-29T10:09:34.045092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:02.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044868+0000","last_change":"2024-03-29T10:09:32.628082+0000","last_active":"2024-03-29T10:09:34.044868+0000","last_peered":"2024-03-29T10:09:34.044868+0000","last_clean":"2024-03-29T10:09:34.044868+0000","last_became_active":"2024-03-29T10:09:32.627765+0000","last_became_peered":"2024-03-29T10:09:32.627765+0000","last_unstale":"2024-03-29T10:09:34.044868+0000","last_undegraded":"2024-03-29T10:09:34.044868+0000","last_fullsized":"2024-03-29T10:09:34.044868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:50.472261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.612120+0000","last_change":"2024-03-29T10:09:32.619242+0000","last_active":"2024-03-29T10:09:33.612120+0000","last_peered":"2024-03-29T10:09:33.612120+0000","last_clean":"2024-03-29T10:09:33.612120+0000","last_became_active":"2024-03-29T10:09:32.618883+0000","last_became_peered":"2024-03-29T10:09:32.618883+0000","last_unstale":"2024-03-29T10:09:33.612120+0000","last_undegraded":"2024-03-29T10:09:33.612120+0000","last_fullsized":"2024-03-29T10:09:33.612120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:08.395520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044873+0000","last_change":"2024-03-29T10:09:32.617701+0000","last_active":"2024-03-29T10:09:34.044873+0000","last_peered":"2024-03-29T10:09:34.044873+0000","last_clean":"2024-03-29T10:09:34.044873+0000","last_became_active":"2024-03-29T10:09:32.616607+0000","last_became_peered":"2024-03-29T10:09:32.616607+0000","last_unstale":"2024-03-29T10:09:34.044873+0000","last_undegraded":"2024-03-29T10:09:34.044873+0000","last_fullsized":"2024-03-29T10:09:34.044873+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:39.250301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045466+0000","last_change":"2024-03-29T10:09:32.618041+0000","last_active":"2024-03-29T10:09:34.045466+0000","last_peered":"2024-03-29T10:09:34.045466+0000","last_clean":"2024-03-29T10:09:34.045466+0000","last_became_active":"2024-03-29T10:09:32.617123+0000","last_became_peered":"2024-03-29T10:09:32.617123+0000","last_unstale":"2024-03-29T10:09:34.045466+0000","last_undegraded":"2024-03-29T10:09:34.045466+0000","last_fullsized":"2024-03-29T10:09:34.045466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:47.161213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044987+0000","last_change":"2024-03-29T10:09:32.622774+0000","last_active":"2024-03-29T10:09:34.044987+0000","last_peered":"2024-03-29T10:09:34.044987+0000","last_clean":"2024-03-29T10:09:34.044987+0000","last_became_active":"2024-03-29T10:09:32.622503+0000","last_became_peered":"2024-03-29T10:09:32.622503+0000","last_unstale":"2024-03-29T10:09:34.044987+0000","last_undegraded":"2024-03-29T10:09:34.044987+0000","last_fullsized":"2024-03-29T10:09:34.044987+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:38.603035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607446+0000","last_change":"2024-03-29T10:09:32.625499+0000","last_active":"2024-03-29T10:09:33.607446+0000","last_peered":"2024-03-29T10:09:33.607446+0000","last_clean":"2024-03-29T10:09:33.607446+0000","last_became_active":"2024-03-29T10:09:32.625200+0000","last_became_peered":"2024-03-29T10:09:32.625200+0000","last_unstale":"2024-03-29T10:09:33.607446+0000","last_undegraded":"2024-03-29T10:09:33.607446+0000","last_fullsized":"2024-03-29T10:09:33.607446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:47.193288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045942+0000","last_change":"2024-03-29T10:09:32.622371+0000","last_active":"2024-03-29T10:09:34.045942+0000","last_peered":"2024-03-29T10:09:34.045942+0000","last_clean":"2024-03-29T10:09:34.045942+0000","last_became_active":"2024-03-29T10:09:32.621892+0000","last_became_peered":"2024-03-29T10:09:32.621892+0000","last_unstale":"2024-03-29T10:09:34.045942+0000","last_undegraded":"2024-03-29T10:09:34.045942+0000","last_fullsized":"2024-03-29T10:09:34.045942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:50.946405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045592+0000","last_change":"2024-03-29T10:09:32.615044+0000","last_active":"2024-03-29T10:09:34.045592+0000","last_peered":"2024-03-29T10:09:34.045592+0000","last_clean":"2024-03-29T10:09:34.045592+0000","last_became_active":"2024-03-29T10:09:32.614754+0000","last_became_peered":"2024-03-29T10:09:32.614754+0000","last_unstale":"2024-03-29T10:09:34.045592+0000","last_undegraded":"2024-03-29T10:09:34.045592+0000","last_fullsized":"2024-03-29T10:09:34.045592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:50.472261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607170+0000","last_change":"2024-03-29T10:09:32.613680+0000","last_active":"2024-03-29T10:09:33.607170+0000","last_peered":"2024-03-29T10:09:33.607170+0000","last_clean":"2024-03-29T10:09:33.607170+0000","last_became_active":"2024-03-29T10:09:32.611800+0000","last_became_peered":"2024-03-29T10:09:32.611800+0000","last_unstale":"2024-03-29T10:09:33.607170+0000","last_undegraded":"2024-03-29T10:09:33.607170+0000","last_fullsized":"2024-03-29T10:09:33.607170+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:04.582194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045761+0000","last_change":"2024-03-29T10:09:32.623271+0000","last_active":"2024-03-29T10:09:34.045761+0000","last_peered":"2024-03-29T10:09:34.045761+0000","last_clean":"2024-03-29T10:09:34.045761+0000","last_became_active":"2024-03-29T10:09:32.622928+0000","last_became_peered":"2024-03-29T10:09:32.622928+0000","last_unstale":"2024-03-29T10:09:34.045761+0000","last_undegraded":"2024-03-29T10:09:34.045761+0000","last_fullsized":"2024-03-29T10:09:34.045761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.930784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607046+0000","last_change":"2024-03-29T10:09:32.621045+0000","last_active":"2024-03-29T10:09:33.607046+0000","last_peered":"2024-03-29T10:09:33.607046+0000","last_clean":"2024-03-29T10:09:33.607046+0000","last_became_active":"2024-03-29T10:09:32.620499+0000","last_became_peered":"2024-03-29T10:09:32.620499+0000","last_unstale":"2024-03-29T10:09:33.607046+0000","last_undegraded":"2024-03-29T10:09:33.607046+0000","last_fullsized":"2024-03-29T10:09:33.607046+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:05:04.665074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045437+0000","last_change":"2024-03-29T10:09:32.637105+0000","last_active":"2024-03-29T10:09:34.045437+0000","last_peered":"2024-03-29T10:09:34.045437+0000","last_clean":"2024-03-29T10:09:34.045437+0000","last_became_active":"2024-03-29T10:09:32.636599+0000","last_became_peered":"2024-03-29T10:09:32.636599+0000","last_unstale":"2024-03-29T10:09:34.045437+0000","last_undegraded":"2024-03-29T10:09:34.045437+0000","last_fullsized":"2024-03-29T10:09:34.045437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:54.720344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.611723+0000","last_change":"2024-03-29T10:09:32.628611+0000","last_active":"2024-03-29T10:09:33.611723+0000","last_peered":"2024-03-29T10:09:33.611723+0000","last_clean":"2024-03-29T10:09:33.611723+0000","last_became_active":"2024-03-29T10:09:32.625523+0000","last_became_peered":"2024-03-29T10:09:32.625523+0000","last_unstale":"2024-03-29T10:09:33.611723+0000","last_undegraded":"2024-03-29T10:09:33.611723+0000","last_fullsized":"2024-03-29T10:09:33.611723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:56.014772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046009+0000","last_change":"2024-03-29T10:09:32.615838+0000","last_active":"2024-03-29T10:09:34.046009+0000","last_peered":"2024-03-29T10:09:34.046009+0000","last_clean":"2024-03-29T10:09:34.046009+0000","last_became_active":"2024-03-29T10:09:32.615274+0000","last_became_peered":"2024-03-29T10:09:32.615274+0000","last_unstale":"2024-03-29T10:09:34.046009+0000","last_undegraded":"2024-03-29T10:09:34.046009+0000","last_fullsized":"2024-03-29T10:09:34.046009+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:07.635360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045741+0000","last_change":"2024-03-29T10:09:32.624374+0000","last_active":"2024-03-29T10:09:34.045741+0000","last_peered":"2024-03-29T10:09:34.045741+0000","last_clean":"2024-03-29T10:09:34.045741+0000","last_became_active":"2024-03-29T10:09:32.624101+0000","last_became_peered":"2024-03-29T10:09:32.624101+0000","last_unstale":"2024-03-29T10:09:34.045741+0000","last_undegraded":"2024-03-29T10:09:34.045741+0000","last_fullsized":"2024-03-29T10:09:34.045741+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:24:39.560229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045630+0000","last_change":"2024-03-29T10:09:32.618810+0000","last_active":"2024-03-29T10:09:34.045630+0000","last_peered":"2024-03-29T10:09:34.045630+0000","last_clean":"2024-03-29T10:09:34.045630+0000","last_became_active":"2024-03-29T10:09:32.618546+0000","last_became_peered":"2024-03-29T10:09:32.618546+0000","last_unstale":"2024-03-29T10:09:34.045630+0000","last_undegraded":"2024-03-29T10:09:34.045630+0000","last_fullsized":"2024-03-29T10:09:34.045630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:49.874012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607130+0000","last_change":"2024-03-29T10:09:32.618824+0000","last_active":"2024-03-29T10:09:33.607130+0000","last_peered":"2024-03-29T10:09:33.607130+0000","last_clean":"2024-03-29T10:09:33.607130+0000","last_became_active":"2024-03-29T10:09:32.618065+0000","last_became_peered":"2024-03-29T10:09:32.618065+0000","last_unstale":"2024-03-29T10:09:33.607130+0000","last_undegraded":"2024-03-29T10:09:33.607130+0000","last_fullsized":"2024-03-29T10:09:33.607130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:29.544771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045633+0000","last_change":"2024-03-29T10:09:32.622999+0000","last_active":"2024-03-29T10:09:34.045633+0000","last_peered":"2024-03-29T10:09:34.045633+0000","last_clean":"2024-03-29T10:09:34.045633+0000","last_became_active":"2024-03-29T10:09:32.622065+0000","last_became_peered":"2024-03-29T10:09:32.622065+0000","last_unstale":"2024-03-29T10:09:34.045633+0000","last_undegraded":"2024-03-29T10:09:34.045633+0000","last_fullsized":"2024-03-29T10:09:34.045633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:23.297716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044843+0000","last_change":"2024-03-29T10:09:32.615373+0000","last_active":"2024-03-29T10:09:34.044843+0000","last_peered":"2024-03-29T10:09:34.044843+0000","last_clean":"2024-03-29T10:09:34.044843+0000","last_became_active":"2024-03-29T10:09:32.615107+0000","last_became_peered":"2024-03-29T10:09:32.615107+0000","last_unstale":"2024-03-29T10:09:34.044843+0000","last_undegraded":"2024-03-29T10:09:34.044843+0000","last_fullsized":"2024-03-29T10:09:34.044843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:23.129448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045322+0000","last_change":"2024-03-29T10:09:32.618453+0000","last_active":"2024-03-29T10:09:34.045322+0000","last_peered":"2024-03-29T10:09:34.045322+0000","last_clean":"2024-03-29T10:09:34.045322+0000","last_became_active":"2024-03-29T10:09:32.616759+0000","last_became_peered":"2024-03-29T10:09:32.616759+0000","last_unstale":"2024-03-29T10:09:34.045322+0000","last_undegraded":"2024-03-29T10:09:34.045322+0000","last_fullsized":"2024-03-29T10:09:34.045322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:50.946405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045802+0000","last_change":"2024-03-29T10:09:32.623841+0000","last_active":"2024-03-29T10:09:34.045802+0000","last_peered":"2024-03-29T10:09:34.045802+0000","last_clean":"2024-03-29T10:09:34.045802+0000","last_became_active":"2024-03-29T10:09:32.623522+0000","last_became_peered":"2024-03-29T10:09:32.623522+0000","last_unstale":"2024-03-29T10:09:34.045802+0000","last_undegraded":"2024-03-29T10:09:34.045802+0000","last_fullsized":"2024-03-29T10:09:34.045802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:40.815771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"79'4","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638507+0000","last_change":"2024-03-29T10:09:32.101864+0000","last_active":"2024-03-29T10:09:32.638507+0000","last_peered":"2024-03-29T10:09:32.638507+0000","last_clean":"2024-03-29T10:09:32.638507+0000","last_became_active":"2024-03-29T10:09:32.101456+0000","last_became_peered":"2024-03-29T10:09:32.101456+0000","last_unstale":"2024-03-29T10:09:32.638507+0000","last_undegraded":"2024-03-29T10:09:32.638507+0000","last_fullsized":"2024-03-29T10:09:32.638507+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:05.714596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045625+0000","last_change":"2024-03-29T10:09:32.630347+0000","last_active":"2024-03-29T10:09:34.045625+0000","last_peered":"2024-03-29T10:09:34.045625+0000","last_clean":"2024-03-29T10:09:34.045625+0000","last_became_active":"2024-03-29T10:09:32.630092+0000","last_became_peered":"2024-03-29T10:09:32.630092+0000","last_unstale":"2024-03-29T10:09:34.045625+0000","last_undegraded":"2024-03-29T10:09:34.045625+0000","last_fullsized":"2024-03-29T10:09:34.045625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:50.745197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.635936+0000","last_change":"2024-03-29T10:09:32.091373+0000","last_active":"2024-03-29T10:09:32.635936+0000","last_peered":"2024-03-29T10:09:32.635936+0000","last_clean":"2024-03-29T10:09:32.635936+0000","last_became_active":"2024-03-29T10:09:32.091110+0000","last_became_peered":"2024-03-29T10:09:32.091110+0000","last_unstale":"2024-03-29T10:09:32.635936+0000","last_undegraded":"2024-03-29T10:09:32.635936+0000","last_fullsized":"2024-03-29T10:09:32.635936+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:32:31.535691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046042+0000","last_change":"2024-03-29T10:09:32.614418+0000","last_active":"2024-03-29T10:09:34.046042+0000","last_peered":"2024-03-29T10:09:34.046042+0000","last_clean":"2024-03-29T10:09:34.046042+0000","last_became_active":"2024-03-29T10:09:32.614112+0000","last_became_peered":"2024-03-29T10:09:32.614112+0000","last_unstale":"2024-03-29T10:09:34.046042+0000","last_undegraded":"2024-03-29T10:09:34.046042+0000","last_fullsized":"2024-03-29T10:09:34.046042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:17.345552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.635794+0000","last_change":"2024-03-29T10:09:32.086998+0000","last_active":"2024-03-29T10:09:32.635794+0000","last_peered":"2024-03-29T10:09:32.635794+0000","last_clean":"2024-03-29T10:09:32.635794+0000","last_became_active":"2024-03-29T10:09:32.086644+0000","last_became_peered":"2024-03-29T10:09:32.086644+0000","last_unstale":"2024-03-29T10:09:32.635794+0000","last_undegraded":"2024-03-29T10:09:32.635794+0000","last_fullsized":"2024-03-29T10:09:32.635794+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:27.370099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045835+0000","last_change":"2024-03-29T10:09:32.624387+0000","last_active":"2024-03-29T10:09:34.045835+0000","last_peered":"2024-03-29T10:09:34.045835+0000","last_clean":"2024-03-29T10:09:34.045835+0000","last_became_active":"2024-03-29T10:09:32.624015+0000","last_became_peered":"2024-03-29T10:09:32.624015+0000","last_unstale":"2024-03-29T10:09:34.045835+0000","last_undegraded":"2024-03-29T10:09:34.045835+0000","last_fullsized":"2024-03-29T10:09:34.045835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:06.669665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.072330+0000","last_change":"2024-03-29T10:09:31.072330+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.072330+0000","last_undegraded":"2024-03-29T10:09:31.072330+0000","last_fullsized":"2024-03-29T10:09:31.072330+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607151+0000","last_change":"2024-03-29T10:09:32.620748+0000","last_active":"2024-03-29T10:09:33.607151+0000","last_peered":"2024-03-29T10:09:33.607151+0000","last_clean":"2024-03-29T10:09:33.607151+0000","last_became_active":"2024-03-29T10:09:32.618821+0000","last_became_peered":"2024-03-29T10:09:32.618821+0000","last_unstale":"2024-03-29T10:09:33.607151+0000","last_undegraded":"2024-03-29T10:09:33.607151+0000","last_fullsized":"2024-03-29T10:09:33.607151+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:39:35.761571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.632616+0000","last_change":"2024-03-29T10:09:32.098141+0000","last_active":"2024-03-29T10:09:32.632616+0000","last_peered":"2024-03-29T10:09:32.632616+0000","last_clean":"2024-03-29T10:09:32.632616+0000","last_became_active":"2024-03-29T10:09:32.097781+0000","last_became_peered":"2024-03-29T10:09:32.097781+0000","last_unstale":"2024-03-29T10:09:32.632616+0000","last_undegraded":"2024-03-29T10:09:32.632616+0000","last_fullsized":"2024-03-29T10:09:32.632616+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607233+0000","last_change":"2024-03-29T10:09:32.614343+0000","last_active":"2024-03-29T10:09:33.607233+0000","last_peered":"2024-03-29T10:09:33.607233+0000","last_clean":"2024-03-29T10:09:33.607233+0000","last_became_active":"2024-03-29T10:09:32.614069+0000","last_became_peered":"2024-03-29T10:09:32.614069+0000","last_unstale":"2024-03-29T10:09:33.607233+0000","last_undegraded":"2024-03-29T10:09:33.607233+0000","last_fullsized":"2024-03-29T10:09:33.607233+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:22.440684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638211+0000","last_change":"2024-03-29T10:09:32.101572+0000","last_active":"2024-03-29T10:09:32.638211+0000","last_peered":"2024-03-29T10:09:32.638211+0000","last_clean":"2024-03-29T10:09:32.638211+0000","last_became_active":"2024-03-29T10:09:32.101206+0000","last_became_peered":"2024-03-29T10:09:32.101206+0000","last_unstale":"2024-03-29T10:09:32.638211+0000","last_undegraded":"2024-03-29T10:09:32.638211+0000","last_fullsized":"2024-03-29T10:09:32.638211+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:46.154988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.611858+0000","last_change":"2024-03-29T10:09:32.624716+0000","last_active":"2024-03-29T10:09:33.611858+0000","last_peered":"2024-03-29T10:09:33.611858+0000","last_clean":"2024-03-29T10:09:33.611858+0000","last_became_active":"2024-03-29T10:09:32.622676+0000","last_became_peered":"2024-03-29T10:09:32.622676+0000","last_unstale":"2024-03-29T10:09:33.611858+0000","last_undegraded":"2024-03-29T10:09:33.611858+0000","last_fullsized":"2024-03-29T10:09:33.611858+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:20.681595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.073144+0000","last_change":"2024-03-29T10:09:31.073144+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.073144+0000","last_undegraded":"2024-03-29T10:09:31.073144+0000","last_fullsized":"2024-03-29T10:09:31.073144+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607448+0000","last_change":"2024-03-29T10:09:32.634701+0000","last_active":"2024-03-29T10:09:33.607448+0000","last_peered":"2024-03-29T10:09:33.607448+0000","last_clean":"2024-03-29T10:09:33.607448+0000","last_became_active":"2024-03-29T10:09:32.631830+0000","last_became_peered":"2024-03-29T10:09:32.631830+0000","last_unstale":"2024-03-29T10:09:33.607448+0000","last_undegraded":"2024-03-29T10:09:33.607448+0000","last_fullsized":"2024-03-29T10:09:33.607448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:29.066092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.603049+0000","last_change":"2024-03-29T10:09:32.097646+0000","last_active":"2024-03-29T10:09:32.603049+0000","last_peered":"2024-03-29T10:09:32.603049+0000","last_clean":"2024-03-29T10:09:32.603049+0000","last_became_active":"2024-03-29T10:09:32.097364+0000","last_became_peered":"2024-03-29T10:09:32.097364+0000","last_unstale":"2024-03-29T10:09:32.603049+0000","last_undegraded":"2024-03-29T10:09:32.603049+0000","last_fullsized":"2024-03-29T10:09:32.603049+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045478+0000","last_change":"2024-03-29T10:09:32.617841+0000","last_active":"2024-03-29T10:09:34.045478+0000","last_peered":"2024-03-29T10:09:34.045478+0000","last_clean":"2024-03-29T10:09:34.045478+0000","last_became_active":"2024-03-29T10:09:32.616581+0000","last_became_peered":"2024-03-29T10:09:32.616581+0000","last_unstale":"2024-03-29T10:09:34.045478+0000","last_undegraded":"2024-03-29T10:09:34.045478+0000","last_fullsized":"2024-03-29T10:09:34.045478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:15.353895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.616868+0000","last_change":"2024-03-29T10:09:32.098837+0000","last_active":"2024-03-29T10:09:32.616868+0000","last_peered":"2024-03-29T10:09:32.616868+0000","last_clean":"2024-03-29T10:09:32.616868+0000","last_became_active":"2024-03-29T10:09:32.098517+0000","last_became_peered":"2024-03-29T10:09:32.098517+0000","last_unstale":"2024-03-29T10:09:32.616868+0000","last_undegraded":"2024-03-29T10:09:32.616868+0000","last_fullsized":"2024-03-29T10:09:32.616868+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:05.714596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045923+0000","last_change":"2024-03-29T10:09:32.636148+0000","last_active":"2024-03-29T10:09:34.045923+0000","last_peered":"2024-03-29T10:09:34.045923+0000","last_clean":"2024-03-29T10:09:34.045923+0000","last_became_active":"2024-03-29T10:09:32.635753+0000","last_became_peered":"2024-03-29T10:09:32.635753+0000","last_unstale":"2024-03-29T10:09:34.045923+0000","last_undegraded":"2024-03-29T10:09:34.045923+0000","last_fullsized":"2024-03-29T10:09:34.045923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:18.667325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.608025+0000","last_change":"2024-03-29T10:09:32.099480+0000","last_active":"2024-03-29T10:09:32.608025+0000","last_peered":"2024-03-29T10:09:32.608025+0000","last_clean":"2024-03-29T10:09:32.608025+0000","last_became_active":"2024-03-29T10:09:32.099133+0000","last_became_peered":"2024-03-29T10:09:32.099133+0000","last_unstale":"2024-03-29T10:09:32.608025+0000","last_undegraded":"2024-03-29T10:09:32.608025+0000","last_fullsized":"2024-03-29T10:09:32.608025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:13:16.625121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044664+0000","last_change":"2024-03-29T10:09:32.628412+0000","last_active":"2024-03-29T10:09:34.044664+0000","last_peered":"2024-03-29T10:09:34.044664+0000","last_clean":"2024-03-29T10:09:34.044664+0000","last_became_active":"2024-03-29T10:09:32.622661+0000","last_became_peered":"2024-03-29T10:09:32.622661+0000","last_unstale":"2024-03-29T10:09:34.044664+0000","last_undegraded":"2024-03-29T10:09:34.044664+0000","last_fullsized":"2024-03-29T10:09:34.044664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:23.129448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"79'3","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636825+0000","last_change":"2024-03-29T10:09:32.089351+0000","last_active":"2024-03-29T10:09:32.636825+0000","last_peered":"2024-03-29T10:09:32.636825+0000","last_clean":"2024-03-29T10:09:32.636825+0000","last_became_active":"2024-03-29T10:09:32.087924+0000","last_became_peered":"2024-03-29T10:09:32.087924+0000","last_unstale":"2024-03-29T10:09:32.636825+0000","last_undegraded":"2024-03-29T10:09:32.636825+0000","last_fullsized":"2024-03-29T10:09:32.636825+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:46.154988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045696+0000","last_change":"2024-03-29T10:09:32.632023+0000","last_active":"2024-03-29T10:09:34.045696+0000","last_peered":"2024-03-29T10:09:34.045696+0000","last_clean":"2024-03-29T10:09:34.045696+0000","last_became_active":"2024-03-29T10:09:32.631450+0000","last_became_peered":"2024-03-29T10:09:32.631450+0000","last_unstale":"2024-03-29T10:09:34.045696+0000","last_undegraded":"2024-03-29T10:09:34.045696+0000","last_fullsized":"2024-03-29T10:09:34.045696+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:24:39.560229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"79'6","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638720+0000","last_change":"2024-03-29T10:09:32.099892+0000","last_active":"2024-03-29T10:09:32.638720+0000","last_peered":"2024-03-29T10:09:32.638720+0000","last_clean":"2024-03-29T10:09:32.638720+0000","last_became_active":"2024-03-29T10:09:32.099626+0000","last_became_peered":"2024-03-29T10:09:32.099626+0000","last_unstale":"2024-03-29T10:09:32.638720+0000","last_undegraded":"2024-03-29T10:09:32.638720+0000","last_fullsized":"2024-03-29T10:09:32.638720+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636457+0000","last_change":"2024-03-29T10:09:32.100281+0000","last_active":"2024-03-29T10:09:32.636457+0000","last_peered":"2024-03-29T10:09:32.636457+0000","last_clean":"2024-03-29T10:09:32.636457+0000","last_became_active":"2024-03-29T10:09:32.100004+0000","last_became_peered":"2024-03-29T10:09:32.100004+0000","last_unstale":"2024-03-29T10:09:32.636457+0000","last_undegraded":"2024-03-29T10:09:32.636457+0000","last_fullsized":"2024-03-29T10:09:32.636457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:49.940180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045675+0000","last_change":"2024-03-29T10:09:32.624300+0000","last_active":"2024-03-29T10:09:34.045675+0000","last_peered":"2024-03-29T10:09:34.045675+0000","last_clean":"2024-03-29T10:09:34.045675+0000","last_became_active":"2024-03-29T10:09:32.624004+0000","last_became_peered":"2024-03-29T10:09:32.624004+0000","last_unstale":"2024-03-29T10:09:34.045675+0000","last_undegraded":"2024-03-29T10:09:34.045675+0000","last_fullsized":"2024-03-29T10:09:34.045675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:38.603035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607519+0000","last_change":"2024-03-29T10:09:32.091147+0000","last_active":"2024-03-29T10:09:32.607519+0000","last_peered":"2024-03-29T10:09:32.607519+0000","last_clean":"2024-03-29T10:09:32.607519+0000","last_became_active":"2024-03-29T10:09:32.089656+0000","last_became_peered":"2024-03-29T10:09:32.089656+0000","last_unstale":"2024-03-29T10:09:32.607519+0000","last_undegraded":"2024-03-29T10:09:32.607519+0000","last_fullsized":"2024-03-29T10:09:32.607519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:22.123223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044770+0000","last_change":"2024-03-29T10:09:32.623116+0000","last_active":"2024-03-29T10:09:34.044770+0000","last_peered":"2024-03-29T10:09:34.044770+0000","last_clean":"2024-03-29T10:09:34.044770+0000","last_became_active":"2024-03-29T10:09:32.622832+0000","last_became_peered":"2024-03-29T10:09:32.622832+0000","last_unstale":"2024-03-29T10:09:34.044770+0000","last_undegraded":"2024-03-29T10:09:34.044770+0000","last_fullsized":"2024-03-29T10:09:34.044770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:36.451327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.071636+0000","last_change":"2024-03-29T10:09:31.071636+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.071636+0000","last_undegraded":"2024-03-29T10:09:31.071636+0000","last_fullsized":"2024-03-29T10:09:31.071636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045252+0000","last_change":"2024-03-29T10:09:32.634186+0000","last_active":"2024-03-29T10:09:34.045252+0000","last_peered":"2024-03-29T10:09:34.045252+0000","last_clean":"2024-03-29T10:09:34.045252+0000","last_became_active":"2024-03-29T10:09:32.633693+0000","last_became_peered":"2024-03-29T10:09:32.633693+0000","last_unstale":"2024-03-29T10:09:34.045252+0000","last_undegraded":"2024-03-29T10:09:34.045252+0000","last_fullsized":"2024-03-29T10:09:34.045252+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:40.815771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"79'8","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.639114+0000","last_change":"2024-03-29T10:09:32.092854+0000","last_active":"2024-03-29T10:09:32.639114+0000","last_peered":"2024-03-29T10:09:32.639114+0000","last_clean":"2024-03-29T10:09:32.639114+0000","last_became_active":"2024-03-29T10:09:32.092591+0000","last_became_peered":"2024-03-29T10:09:32.092591+0000","last_unstale":"2024-03-29T10:09:32.639114+0000","last_undegraded":"2024-03-29T10:09:32.639114+0000","last_fullsized":"2024-03-29T10:09:32.639114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:40.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045937+0000","last_change":"2024-03-29T10:09:32.623867+0000","last_active":"2024-03-29T10:09:34.045937+0000","last_peered":"2024-03-29T10:09:34.045937+0000","last_clean":"2024-03-29T10:09:34.045937+0000","last_became_active":"2024-03-29T10:09:32.622284+0000","last_became_peered":"2024-03-29T10:09:32.622284+0000","last_unstale":"2024-03-29T10:09:34.045937+0000","last_undegraded":"2024-03-29T10:09:34.045937+0000","last_fullsized":"2024-03-29T10:09:34.045937+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:06.669665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636096+0000","last_change":"2024-03-29T10:09:32.097805+0000","last_active":"2024-03-29T10:09:32.636096+0000","last_peered":"2024-03-29T10:09:32.636096+0000","last_clean":"2024-03-29T10:09:32.636096+0000","last_became_active":"2024-03-29T10:09:32.097537+0000","last_became_peered":"2024-03-29T10:09:32.097537+0000","last_unstale":"2024-03-29T10:09:32.636096+0000","last_undegraded":"2024-03-29T10:09:32.636096+0000","last_fullsized":"2024-03-29T10:09:32.636096+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:35.016753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.634994+0000","last_change":"2024-03-29T10:09:32.104998+0000","last_active":"2024-03-29T10:09:32.634994+0000","last_peered":"2024-03-29T10:09:32.634994+0000","last_clean":"2024-03-29T10:09:32.634994+0000","last_became_active":"2024-03-29T10:09:32.104610+0000","last_became_peered":"2024-03-29T10:09:32.104610+0000","last_unstale":"2024-03-29T10:09:32.634994+0000","last_undegraded":"2024-03-29T10:09:32.634994+0000","last_fullsized":"2024-03-29T10:09:32.634994+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:38.244076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607169+0000","last_change":"2024-03-29T10:09:32.619003+0000","last_active":"2024-03-29T10:09:33.607169+0000","last_peered":"2024-03-29T10:09:33.607169+0000","last_clean":"2024-03-29T10:09:33.607169+0000","last_became_active":"2024-03-29T10:09:32.617951+0000","last_became_peered":"2024-03-29T10:09:32.617951+0000","last_unstale":"2024-03-29T10:09:33.607169+0000","last_undegraded":"2024-03-29T10:09:33.607169+0000","last_fullsized":"2024-03-29T10:09:33.607169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:00.709669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.072860+0000","last_change":"2024-03-29T10:09:31.072860+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.072860+0000","last_undegraded":"2024-03-29T10:09:31.072860+0000","last_fullsized":"2024-03-29T10:09:31.072860+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:27:35.691958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045708+0000","last_change":"2024-03-29T10:09:32.633244+0000","last_active":"2024-03-29T10:09:34.045708+0000","last_peered":"2024-03-29T10:09:34.045708+0000","last_clean":"2024-03-29T10:09:34.045708+0000","last_became_active":"2024-03-29T10:09:32.632943+0000","last_became_peered":"2024-03-29T10:09:32.632943+0000","last_unstale":"2024-03-29T10:09:34.045708+0000","last_undegraded":"2024-03-29T10:09:34.045708+0000","last_fullsized":"2024-03-29T10:09:34.045708+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:36.022978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.606805+0000","last_change":"2024-03-29T10:09:32.089495+0000","last_active":"2024-03-29T10:09:32.606805+0000","last_peered":"2024-03-29T10:09:32.606805+0000","last_clean":"2024-03-29T10:09:32.606805+0000","last_became_active":"2024-03-29T10:09:32.089228+0000","last_became_peered":"2024-03-29T10:09:32.089228+0000","last_unstale":"2024-03-29T10:09:32.606805+0000","last_undegraded":"2024-03-29T10:09:32.606805+0000","last_fullsized":"2024-03-29T10:09:32.606805+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:14.347670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636359+0000","last_change":"2024-03-29T10:09:32.095956+0000","last_active":"2024-03-29T10:09:32.636359+0000","last_peered":"2024-03-29T10:09:32.636359+0000","last_clean":"2024-03-29T10:09:32.636359+0000","last_became_active":"2024-03-29T10:09:32.093465+0000","last_became_peered":"2024-03-29T10:09:32.093465+0000","last_unstale":"2024-03-29T10:09:32.636359+0000","last_undegraded":"2024-03-29T10:09:32.636359+0000","last_fullsized":"2024-03-29T10:09:32.636359+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.686777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045534+0000","last_change":"2024-03-29T10:09:32.622232+0000","last_active":"2024-03-29T10:09:34.045534+0000","last_peered":"2024-03-29T10:09:34.045534+0000","last_clean":"2024-03-29T10:09:34.045534+0000","last_became_active":"2024-03-29T10:09:32.621815+0000","last_became_peered":"2024-03-29T10:09:32.621815+0000","last_unstale":"2024-03-29T10:09:34.045534+0000","last_undegraded":"2024-03-29T10:09:34.045534+0000","last_fullsized":"2024-03-29T10:09:34.045534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:54.720344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607356+0000","last_change":"2024-03-29T10:09:32.099575+0000","last_active":"2024-03-29T10:09:32.607356+0000","last_peered":"2024-03-29T10:09:32.607356+0000","last_clean":"2024-03-29T10:09:32.607356+0000","last_became_active":"2024-03-29T10:09:32.098434+0000","last_became_peered":"2024-03-29T10:09:32.098434+0000","last_unstale":"2024-03-29T10:09:32.607356+0000","last_undegraded":"2024-03-29T10:09:32.607356+0000","last_fullsized":"2024-03-29T10:09:32.607356+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:39.694641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044600+0000","last_change":"2024-03-29T10:09:32.623727+0000","last_active":"2024-03-29T10:09:34.044600+0000","last_peered":"2024-03-29T10:09:34.044600+0000","last_clean":"2024-03-29T10:09:34.044600+0000","last_became_active":"2024-03-29T10:09:32.623445+0000","last_became_peered":"2024-03-29T10:09:32.623445+0000","last_unstale":"2024-03-29T10:09:34.044600+0000","last_undegraded":"2024-03-29T10:09:34.044600+0000","last_fullsized":"2024-03-29T10:09:34.044600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:18.667325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.072820+0000","last_change":"2024-03-29T10:09:31.072820+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.072820+0000","last_undegraded":"2024-03-29T10:09:31.072820+0000","last_fullsized":"2024-03-29T10:09:31.072820+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:51:58.170982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.637133+0000","last_change":"2024-03-29T10:09:32.089390+0000","last_active":"2024-03-29T10:09:32.637133+0000","last_peered":"2024-03-29T10:09:32.637133+0000","last_clean":"2024-03-29T10:09:32.637133+0000","last_became_active":"2024-03-29T10:09:32.089088+0000","last_became_peered":"2024-03-29T10:09:32.089088+0000","last_unstale":"2024-03-29T10:09:32.637133+0000","last_undegraded":"2024-03-29T10:09:32.637133+0000","last_fullsized":"2024-03-29T10:09:32.637133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:11:34.599249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.609775+0000","last_change":"2024-03-29T10:09:32.101276+0000","last_active":"2024-03-29T10:09:32.609775+0000","last_peered":"2024-03-29T10:09:32.609775+0000","last_clean":"2024-03-29T10:09:32.609775+0000","last_became_active":"2024-03-29T10:09:32.100949+0000","last_became_peered":"2024-03-29T10:09:32.100949+0000","last_unstale":"2024-03-29T10:09:32.609775+0000","last_undegraded":"2024-03-29T10:09:32.609775+0000","last_fullsized":"2024-03-29T10:09:32.609775+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:40.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044751+0000","last_change":"2024-03-29T10:09:32.611596+0000","last_active":"2024-03-29T10:09:34.044751+0000","last_peered":"2024-03-29T10:09:34.044751+0000","last_clean":"2024-03-29T10:09:34.044751+0000","last_became_active":"2024-03-29T10:09:32.610142+0000","last_became_peered":"2024-03-29T10:09:32.610142+0000","last_unstale":"2024-03-29T10:09:34.044751+0000","last_undegraded":"2024-03-29T10:09:34.044751+0000","last_fullsized":"2024-03-29T10:09:34.044751+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:07.635360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605517+0000","last_change":"2024-03-29T10:09:32.092374+0000","last_active":"2024-03-29T10:09:32.605517+0000","last_peered":"2024-03-29T10:09:32.605517+0000","last_clean":"2024-03-29T10:09:32.605517+0000","last_became_active":"2024-03-29T10:09:32.091381+0000","last_became_peered":"2024-03-29T10:09:32.091381+0000","last_unstale":"2024-03-29T10:09:32.605517+0000","last_undegraded":"2024-03-29T10:09:32.605517+0000","last_fullsized":"2024-03-29T10:09:32.605517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:06.629135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046025+0000","last_change":"2024-03-29T10:09:32.620976+0000","last_active":"2024-03-29T10:09:34.046025+0000","last_peered":"2024-03-29T10:09:34.046025+0000","last_clean":"2024-03-29T10:09:34.046025+0000","last_became_active":"2024-03-29T10:09:32.619271+0000","last_became_peered":"2024-03-29T10:09:32.619271+0000","last_unstale":"2024-03-29T10:09:34.046025+0000","last_undegraded":"2024-03-29T10:09:34.046025+0000","last_fullsized":"2024-03-29T10:09:34.046025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.930784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.071841+0000","last_change":"2024-03-29T10:09:31.071841+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.071841+0000","last_undegraded":"2024-03-29T10:09:31.071841+0000","last_fullsized":"2024-03-29T10:09:31.071841+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:56.946534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607553+0000","last_change":"2024-03-29T10:09:32.618538+0000","last_active":"2024-03-29T10:09:33.607553+0000","last_peered":"2024-03-29T10:09:33.607553+0000","last_clean":"2024-03-29T10:09:33.607553+0000","last_became_active":"2024-03-29T10:09:32.617981+0000","last_became_peered":"2024-03-29T10:09:32.617981+0000","last_unstale":"2024-03-29T10:09:33.607553+0000","last_undegraded":"2024-03-29T10:09:33.607553+0000","last_fullsized":"2024-03-29T10:09:33.607553+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:50.197997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607488+0000","last_change":"2024-03-29T10:09:32.084059+0000","last_active":"2024-03-29T10:09:32.607488+0000","last_peered":"2024-03-29T10:09:32.607488+0000","last_clean":"2024-03-29T10:09:32.607488+0000","last_became_active":"2024-03-29T10:09:32.082631+0000","last_became_peered":"2024-03-29T10:09:32.082631+0000","last_unstale":"2024-03-29T10:09:32.607488+0000","last_undegraded":"2024-03-29T10:09:32.607488+0000","last_fullsized":"2024-03-29T10:09:32.607488+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:37:31.209166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045792+0000","last_change":"2024-03-29T10:09:32.628132+0000","last_active":"2024-03-29T10:09:34.045792+0000","last_peered":"2024-03-29T10:09:34.045792+0000","last_clean":"2024-03-29T10:09:34.045792+0000","last_became_active":"2024-03-29T10:09:32.627845+0000","last_became_peered":"2024-03-29T10:09:32.627845+0000","last_unstale":"2024-03-29T10:09:34.045792+0000","last_undegraded":"2024-03-29T10:09:34.045792+0000","last_fullsized":"2024-03-29T10:09:34.045792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:52.322978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.604249+0000","last_change":"2024-03-29T10:09:32.094306+0000","last_active":"2024-03-29T10:09:32.604249+0000","last_peered":"2024-03-29T10:09:32.604249+0000","last_clean":"2024-03-29T10:09:32.604249+0000","last_became_active":"2024-03-29T10:09:32.093998+0000","last_became_peered":"2024-03-29T10:09:32.093998+0000","last_unstale":"2024-03-29T10:09:32.604249+0000","last_undegraded":"2024-03-29T10:09:32.604249+0000","last_fullsized":"2024-03-29T10:09:32.604249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:17.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044480+0000","last_change":"2024-03-29T10:09:32.620507+0000","last_active":"2024-03-29T10:09:34.044480+0000","last_peered":"2024-03-29T10:09:34.044480+0000","last_clean":"2024-03-29T10:09:34.044480+0000","last_became_active":"2024-03-29T10:09:32.620186+0000","last_became_peered":"2024-03-29T10:09:32.620186+0000","last_unstale":"2024-03-29T10:09:34.044480+0000","last_undegraded":"2024-03-29T10:09:34.044480+0000","last_fullsized":"2024-03-29T10:09:34.044480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:49.874012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605757+0000","last_change":"2024-03-29T10:09:32.086897+0000","last_active":"2024-03-29T10:09:32.605757+0000","last_peered":"2024-03-29T10:09:32.605757+0000","last_clean":"2024-03-29T10:09:32.605757+0000","last_became_active":"2024-03-29T10:09:32.086602+0000","last_became_peered":"2024-03-29T10:09:32.086602+0000","last_unstale":"2024-03-29T10:09:32.605757+0000","last_undegraded":"2024-03-29T10:09:32.605757+0000","last_fullsized":"2024-03-29T10:09:32.605757+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:23:18.397722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607519+0000","last_change":"2024-03-29T10:09:32.633167+0000","last_active":"2024-03-29T10:09:33.607519+0000","last_peered":"2024-03-29T10:09:33.607519+0000","last_clean":"2024-03-29T10:09:33.607519+0000","last_became_active":"2024-03-29T10:09:32.632682+0000","last_became_peered":"2024-03-29T10:09:32.632682+0000","last_unstale":"2024-03-29T10:09:33.607519+0000","last_undegraded":"2024-03-29T10:09:33.607519+0000","last_fullsized":"2024-03-29T10:09:33.607519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:07.268799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605003+0000","last_change":"2024-03-29T10:09:32.092768+0000","last_active":"2024-03-29T10:09:32.605003+0000","last_peered":"2024-03-29T10:09:32.605003+0000","last_clean":"2024-03-29T10:09:32.605003+0000","last_became_active":"2024-03-29T10:09:32.092488+0000","last_became_peered":"2024-03-29T10:09:32.092488+0000","last_unstale":"2024-03-29T10:09:32.605003+0000","last_undegraded":"2024-03-29T10:09:32.605003+0000","last_fullsized":"2024-03-29T10:09:32.605003+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:16.339327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.606964+0000","last_change":"2024-03-29T10:09:32.628533+0000","last_active":"2024-03-29T10:09:33.606964+0000","last_peered":"2024-03-29T10:09:33.606964+0000","last_clean":"2024-03-29T10:09:33.606964+0000","last_became_active":"2024-03-29T10:09:32.628297+0000","last_became_peered":"2024-03-29T10:09:32.628297+0000","last_unstale":"2024-03-29T10:09:33.606964+0000","last_undegraded":"2024-03-29T10:09:33.606964+0000","last_fullsized":"2024-03-29T10:09:33.606964+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:53:16.735792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.604907+0000","last_change":"2024-03-29T10:09:32.088461+0000","last_active":"2024-03-29T10:09:32.604907+0000","last_peered":"2024-03-29T10:09:32.604907+0000","last_clean":"2024-03-29T10:09:32.604907+0000","last_became_active":"2024-03-29T10:09:32.087924+0000","last_became_peered":"2024-03-29T10:09:32.087924+0000","last_unstale":"2024-03-29T10:09:32.604907+0000","last_undegraded":"2024-03-29T10:09:32.604907+0000","last_fullsized":"2024-03-29T10:09:32.604907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:01.534207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"68'161","reported_seq":379,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.613595+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:09:32.613595+0000","last_peered":"2024-03-29T10:09:32.613595+0000","last_clean":"2024-03-29T10:09:32.613595+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:09:32.613595+0000","last_undegraded":"2024-03-29T10:09:32.613595+0000","last_fullsized":"2024-03-29T10:09:32.613595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":112,"acting":112,"num_store_stats":7},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2292,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1568768,"data_stored":1047510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.57299999999999995,"5min":0.44500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95299999999999996,"5min":1.0029999999999999,"15min":1.0109999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.69999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.89800000000000002,"5min":0.93400000000000005,"15min":1.046},"last":0.72299999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71699999999999997,"15min":0.71999999999999997},"min":{"1min":0.51900000000000002,"5min":0.47399999999999998,"15min":0.44500000000000001},"max":{"1min":0.92600000000000005,"5min":0.93200000000000005,"15min":1.1910000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.73499999999999999},"min":{"1min":0.51700000000000002,"5min":0.42199999999999999,"15min":0.372},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1},"last":0.71099999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.72899999999999998,"15min":0.74099999999999999},"min":{"1min":0.47199999999999998,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.046},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.755,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":1.079},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.48099999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.93100000000000005,"5min":0.97699999999999998,"15min":1.081},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77500000000000002,"15min":0.78300000000000003},"min":{"1min":0.55800000000000005,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":0.96199999999999997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.74199999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.29199999999999998},"max":{"1min":0.86399999999999999,"5min":0.87,"15min":1.111},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.627,"15min":0.64100000000000001},"min":{"1min":0.44800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":0.95199999999999996,"15min":0.95599999999999996},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64900000000000002,"15min":0.65900000000000003},"min":{"1min":0.432,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95299999999999996,"5min":0.96099999999999997,"15min":1.0089999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67600000000000005,"15min":0.68899999999999995},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90100000000000002,"5min":0.92800000000000005,"15min":1.141},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.71899999999999997,"15min":0.73299999999999998},"min":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.49199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":1.1679999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69699999999999995,"15min":0.70499999999999996},"min":{"1min":0.54800000000000004,"5min":0.48499999999999999,"15min":0.435},"max":{"1min":0.93799999999999994,"5min":0.98399999999999999,"15min":1.121},"last":0.48499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575031,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":966656,"data_stored":456982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70299999999999996,"15min":0.745},"min":{"1min":0.57199999999999995,"5min":0.42799999999999999,"15min":0.34699999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":7.4080000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.72599999999999998,"15min":0.76900000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.42499999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":6.7610000000000001},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70999999999999996,"15min":0.71899999999999997},"min":{"1min":0.51300000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.2549999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.72499999999999998,"15min":0.71999999999999997},"min":{"1min":0.58999999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":0.995,"15min":1.0209999999999999},"last":0.745}]},{"osd":2,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.71699999999999997,"15min":0.72399999999999998},"min":{"1min":0.65900000000000003,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94199999999999995,"5min":0.96899999999999997,"15min":1.3169999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.58899999999999997,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.415},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.74099999999999999,"15min":0.76900000000000002},"min":{"1min":0.60399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.98999999999999999,"5min":1.02,"15min":3.1960000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76800000000000002,"15min":0.79800000000000004},"min":{"1min":0.71499999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0109999999999999,"5min":1.014,"15min":4.3760000000000003},"last":0.70499999999999996}]},{"osd":4,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":1.109},"last":0.753},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.47799999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85899999999999999,"5min":0.95599999999999996,"15min":3.1219999999999999},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.67300000000000004},"min":{"1min":0.50600000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90000000000000002,"5min":0.90900000000000003,"15min":2.8279999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.79500000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.74399999999999999,"15min":0.76100000000000001},"min":{"1min":0.56499999999999995,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.95499999999999996,"5min":1.105,"15min":1.2769999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.496,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94499999999999995,"5min":0.998,"15min":1.7869999999999999},"last":0.76200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50116,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321724,"statfs":{"total":96636764160,"available":96585445376,"internally_reserved":0,"allocated":970752,"data_stored":457072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.315,"5min":0.219,"15min":0.219},"max":{"1min":0.65300000000000002,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.379},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.68999999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.504}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71599999999999997,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.52900000000000003},"min":{"1min":0.36699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":4.282},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.70999999999999996,"5min":0.78500000000000003,"15min":4.3200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.36599999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.016,"5min":1.617,"15min":1.617},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76400000000000001,"15min":0.78800000000000003},"min":{"1min":0.60899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0289999999999999,"5min":1.1419999999999999,"15min":4.3559999999999999},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.51800000000000002},"max":{"1min":1.056,"5min":1.3939999999999999,"15min":1.9119999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78600000000000003,"15min":0.80800000000000005},"min":{"1min":0.56299999999999994,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.048,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77800000000000002,"15min":0.81000000000000005},"min":{"1min":0.57499999999999996,"5min":0.52600000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":1.194,"15min":4.3899999999999997},"last":0.752},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.093,"5min":1.468,"15min":1.468},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.62,"5min":0.58699999999999997,"15min":0.56200000000000006},"max":{"1min":1.127,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.83599999999999997},"min":{"1min":0.60999999999999999,"5min":0.57099999999999995,"15min":0.55700000000000005},"max":{"1min":1.0269999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.73699999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1572864,"data_stored":1048477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.33300000000000002,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.497,"15min":0.51000000000000001},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72199999999999998,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.628}]},{"osd":2,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.435,"5min":0.39200000000000002,"15min":0.35499999999999998},"max":{"1min":0.78800000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.48799999999999999,"15min":0.496},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.63400000000000001,"5min":0.83899999999999997,"15min":0.97199999999999998},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.39000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73599999999999999,"5min":0.83199999999999996,"15min":1.077},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.47199999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83499999999999996,"5min":0.876,"15min":1.0549999999999999},"last":0.5}]},{"osd":4,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.81399999999999995,"15min":0.80200000000000005},"min":{"1min":0.65100000000000002,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":1.0469999999999999,"5min":1.625,"15min":1.625},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.82399999999999995},"min":{"1min":0.66900000000000004,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":0.97099999999999997,"5min":1.1419999999999999,"15min":1.167},"last":0.73399999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.752,"15min":0.75900000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98999999999999999,"5min":1.137,"15min":1.137},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.80600000000000005,"15min":0.80300000000000005},"min":{"1min":0.60399999999999998,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":0.90200000000000002,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.752}]},{"osd":6,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.53400000000000003},"max":{"1min":1.0129999999999999,"5min":1.0640000000000001,"15min":1.228},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.85499999999999998,"15min":0.84399999999999997},"min":{"1min":0.66400000000000003,"5min":0.52600000000000002,"15min":0.46899999999999997},"max":{"1min":1.0249999999999999,"5min":1.601,"15min":1.601},"last":0.80100000000000005}]},{"osd":7,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78300000000000003,"15min":0.79600000000000004},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.97299999999999998,"5min":1.0529999999999999,"15min":1.1890000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.84699999999999998,"15min":0.84199999999999997},"min":{"1min":0.64000000000000001,"5min":0.57499999999999996,"15min":0.49299999999999999},"max":{"1min":1.046,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.84199999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36172,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335668,"statfs":{"total":96636764160,"available":96599724032,"internally_reserved":0,"allocated":974848,"data_stored":457038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.34699999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.623,"5min":0.69599999999999995,"15min":0.86599999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.39300000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":1.3140000000000001},"last":0.47999999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47299999999999998,"15min":0.46999999999999997},"min":{"1min":0.376,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.63600000000000001,"5min":0.751,"15min":0.94499999999999995},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.45800000000000002,"5min":0.36099999999999999,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":1.032,"15min":1.032},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.504,"5min":0.375,"15min":0.375},"max":{"1min":0.73799999999999999,"5min":0.80000000000000004,"15min":1.127},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.48999999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70899999999999996,"5min":0.82399999999999995,"15min":1.1479999999999999},"last":0.51800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.77300000000000002,"15min":0.78600000000000003},"min":{"1min":0.68700000000000006,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":1.0760000000000001,"5min":1.1799999999999999,"15min":1.554},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.79500000000000004,"15min":0.81000000000000005},"min":{"1min":0.60799999999999998,"5min":0.55000000000000004,"15min":0.51600000000000001},"max":{"1min":1.0309999999999999,"5min":1.236,"15min":1.577},"last":0.86099999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.81000000000000005},"min":{"1min":0.64700000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0680000000000001,"5min":1.25,"15min":1.5209999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79600000000000004,"15min":0.80800000000000005},"min":{"1min":0.58799999999999997,"5min":0.53200000000000003,"15min":0.51800000000000002},"max":{"1min":1.0049999999999999,"5min":1.2250000000000001,"15min":1.3300000000000001},"last":0.77700000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.80400000000000005,"15min":0.82799999999999996},"min":{"1min":0.65200000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1399999999999999,"5min":1.1539999999999999,"15min":1.758},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.874,"5min":0.83699999999999997,"15min":0.84199999999999997},"min":{"1min":0.65900000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1080000000000001,"5min":1.3560000000000001,"15min":1.4750000000000001},"last":0.90900000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.86399999999999999,"15min":0.86899999999999999},"min":{"1min":0.73599999999999999,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.137,"5min":1.2709999999999999,"15min":1.4490000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.69999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.107,"5min":1.3049999999999999,"15min":1.4390000000000001},"last":0.96799999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575030,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36172,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335668,"statfs":{"total":96636764160,"available":96599724032,"internally_reserved":0,"allocated":974848,"data_stored":457038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.46600000000000003},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":0.79500000000000004,"15min":4.3799999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52600000000000002,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":4.6379999999999999},"last":0.621}]},{"osd":1,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.40200000000000002,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.69799999999999995,"5min":0.98699999999999999,"15min":4.1840000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.314},"max":{"1min":0.80800000000000005,"5min":0.96099999999999997,"15min":1.048},"last":0.67600000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.47799999999999998,"5min":0.373,"15min":0.33700000000000002},"max":{"1min":0.77200000000000002,"5min":0.82499999999999996,"15min":1.369},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.63800000000000001,"15min":0.64700000000000002},"min":{"1min":0.55700000000000005,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.79700000000000004,"5min":0.94799999999999995,"15min":4.5970000000000004},"last":0.748}]},{"osd":4,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76800000000000002,"15min":0.78100000000000003},"min":{"1min":0.64400000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.0940000000000001,"5min":1.288,"15min":4.3319999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77600000000000002,"15min":0.77500000000000002},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.39400000000000002},"max":{"1min":0.92400000000000004,"5min":1.46,"15min":1.46},"last":1.0900000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78200000000000003,"15min":0.79500000000000004},"min":{"1min":0.628,"5min":0.48899999999999999,"15min":0.435},"max":{"1min":0.97599999999999998,"5min":1.3580000000000001,"15min":1.3740000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79800000000000004,"15min":0.81000000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.44500000000000001},"max":{"1min":1.133,"5min":1.397,"15min":4.4180000000000001},"last":0.876}]},{"osd":6,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81399999999999995,"15min":0.81200000000000006},"min":{"1min":0.68899999999999995,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.0009999999999999,"5min":1.2549999999999999,"15min":1.4730000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81699999999999995,"15min":0.81899999999999995},"min":{"1min":0.65000000000000002,"5min":0.52400000000000002,"15min":0.503},"max":{"1min":1.0409999999999999,"5min":1.23,"15min":1.417},"last":1.0349999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.83999999999999997,"15min":0.84799999999999998},"min":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.54300000000000004},"max":{"1min":1.135,"5min":1.5,"15min":4.4550000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.64400000000000002,"5min":0.59899999999999998,"15min":0.56200000000000006},"max":{"1min":1.056,"5min":1.345,"15min":1.476},"last":1.0489999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54224,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317616,"statfs":{"total":96636764160,"available":96581238784,"internally_reserved":0,"allocated":978944,"data_stored":458546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.68700000000000006},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.88200000000000001,"5min":0.93200000000000005,"15min":1.417},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.92100000000000004,"5min":1.165,"15min":1.165},"last":0.434}]},{"osd":1,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.93500000000000005,"5min":0.93700000000000006,"15min":1.139},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.333},"last":0.55700000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.01,"15min":1.863},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.75,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93700000000000006,"5min":0.96299999999999997,"15min":1.387},"last":0.50600000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.5,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.88700000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.495},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.51600000000000001,"5min":0.48699999999999999,"15min":0.46600000000000003},"max":{"1min":0.80500000000000005,"5min":1.1539999999999999,"15min":1.905},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.46700000000000003,"5min":0.39300000000000002,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.48099999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.78500000000000003,"5min":1.294,"15min":1.294},"last":0.629}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.48099999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.75900000000000001,"5min":1.0129999999999999,"15min":1.9339999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.66200000000000003},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.433},"max":{"1min":0.85299999999999998,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.48099999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85999999999999999,"5min":1.345,"15min":1.345},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.59099999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89700000000000002,"5min":1.214,"15min":1.97},"last":0.755}]}]},{"osd":5,"up_from":13,"seq":55834575029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":929792,"data_stored":437640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69599999999999995,"15min":0.68200000000000005},"min":{"1min":0.432,"5min":0.41599999999999998,"15min":0.313},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.746,"15min":0.71899999999999997},"min":{"1min":0.45900000000000002,"5min":0.45500000000000002,"15min":0.33500000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.339},"last":0.435}]},{"osd":1,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.55400000000000005,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.0169999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72199999999999998,"15min":0.70499999999999996},"min":{"1min":0.47999999999999998,"5min":0.46700000000000003,"15min":0.39200000000000002},"max":{"1min":0.95199999999999996,"5min":1.022,"15min":1.6040000000000001},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73099999999999998,"15min":0.72799999999999998},"min":{"1min":0.57099999999999995,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.476},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.59299999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93700000000000006,"5min":1.095,"15min":1.502},"last":0.46200000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.63200000000000001,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":0.95799999999999996,"5min":1.113,"15min":1.1619999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.754},"min":{"1min":0.56499999999999995,"5min":0.52300000000000002,"15min":0.46500000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.744},"last":0.64900000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.80300000000000005,"5min":0.94599999999999995,"15min":0.95899999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.214},"last":0.495}]},{"osd":6,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64800000000000002,"15min":0.65400000000000003},"min":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.36799999999999999},"max":{"1min":0.85999999999999999,"5min":0.91300000000000003,"15min":1.151},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.52600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":0.97099999999999997,"15min":1.5489999999999999},"last":0.70599999999999996}]},{"osd":7,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.70499999999999996,"15min":0.71099999999999997},"min":{"1min":0.51500000000000001,"5min":0.41899999999999998,"15min":0.40200000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.325},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.53800000000000003,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":0.96999999999999997,"5min":0.98199999999999998,"15min":1.1990000000000001},"last":0.79700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:09:37.545 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:09:37.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.779+0000 7f8c8c000700 1 Processor -- start 2024-03-29T10:09:37.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.779+0000 7f8c8c000700 1 -- start start 2024-03-29T10:09:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c8c000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8c8415b5d0 0x7f8c840ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c8c000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8c840ac7d0 0x7f8c8406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c8c000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8415fd50 con 0x7f8c840cb3f0 2024-03-29T10:09:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c8c000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8415b9b0 con 0x7f8c8415b5d0 2024-03-29T10:09:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c8c000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8415f140 con 0x7f8c840ac7d0 2024-03-29T10:09:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c89600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8c8415b5d0 0x7f8c840ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c89600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8c8415b5d0 0x7f8c840ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58244/0 (socket says 172.21.15.12:58244) 2024-03-29T10:09:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c89600700 1 -- 172.21.15.12:0/3987947306 learned_addr learned my addr 172.21.15.12:0/3987947306 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c8a000700 1 --2- 172.21.15.12:0/3987947306 >> v2:172.21.15.159:3300/0 conn(0x7f8c840ac7d0 0x7f8c8406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c89600700 1 -- 172.21.15.12:0/3987947306 >> v2:172.21.15.159:3300/0 conn(0x7f8c840ac7d0 msgr2=0x7f8c8406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c89600700 1 --2- 172.21.15.12:0/3987947306 >> v2:172.21.15.159:3300/0 conn(0x7f8c840ac7d0 0x7f8c8406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c89600700 1 -- 172.21.15.12:0/3987947306 >> v1:172.21.15.159:6789/0 conn(0x7f8c840cb3f0 legacy=0x7f8c840ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.781+0000 7f8c89600700 1 -- 172.21.15.12:0/3987947306 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8415f420 con 0x7f8c8415b5d0 2024-03-29T10:09:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.782+0000 7f8c89600700 1 --2- 172.21.15.12:0/3987947306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c8415b5d0 0x7f8c840ae3f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8c840cfdc0 tx=0x7f8c78009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7615052663b7f1ff server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.782+0000 7f8c83e00700 1 -- 172.21.15.12:0/3987947306 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78004230 con 0x7f8c8415b5d0 2024-03-29T10:09:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.782+0000 7f8c83e00700 1 -- 172.21.15.12:0/3987947306 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c78004390 con 0x7f8c8415b5d0 2024-03-29T10:09:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.782+0000 7f8c83e00700 1 -- 172.21.15.12:0/3987947306 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c780036a0 con 0x7f8c8415b5d0 2024-03-29T10:09:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.782+0000 7f8c8c000700 1 -- 172.21.15.12:0/3987947306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c8415b5d0 msgr2=0x7f8c840ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.782+0000 7f8c8c000700 1 --2- 172.21.15.12:0/3987947306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c8415b5d0 0x7f8c840ae3f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8c840cfdc0 tx=0x7f8c78009bd0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.783+0000 7f8c8c000700 1 -- 172.21.15.12:0/3987947306 shutdown_connections 2024-03-29T10:09:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.783+0000 7f8c8c000700 1 --2- 172.21.15.12:0/3987947306 >> v2:172.21.15.159:3300/0 conn(0x7f8c840ac7d0 0x7f8c8406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.783+0000 7f8c8c000700 1 --2- 172.21.15.12:0/3987947306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c8415b5d0 0x7f8c840ae3f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.783+0000 7f8c8c000700 1 -- 172.21.15.12:0/3987947306 >> 172.21.15.12:0/3987947306 conn(0x7f8c840cf190 msgr2=0x7f8c840cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.783+0000 7f8c8c000700 1 -- 172.21.15.12:0/3987947306 shutdown_connections 2024-03-29T10:09:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.783+0000 7f8c8c000700 1 -- 172.21.15.12:0/3987947306 wait complete. 2024-03-29T10:09:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.784+0000 7f8c8c000700 1 Processor -- start 2024-03-29T10:09:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.785+0000 7f8c8c000700 1 -- start start 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8c000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c840ac7d0 0x7f8c841554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8c000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c8415b5d0 0x7f8c84071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8c000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c841559f0 0x7f8c84071f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8c000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c841f33d0 con 0x7f8c841559f0 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8c000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c841f36b0 con 0x7f8c8415b5d0 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8c000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c841f3990 con 0x7f8c840ac7d0 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c841559f0 0x7f8c84071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c841559f0 0x7f8c84071f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58256/0 (socket says 172.21.15.12:58256) 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8a000700 1 -- 172.21.15.12:0/2055699364 learned_addr learned my addr 172.21.15.12:0/2055699364 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c88c00700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c8415b5d0 0x7f8c84071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c89600700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c840ac7d0 0x7f8c841554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8a000700 1 -- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c840ac7d0 msgr2=0x7f8c841554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8a000700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c840ac7d0 0x7f8c841554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8a000700 1 -- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c8415b5d0 msgr2=0x7f8c84071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8a000700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c8415b5d0 0x7f8c84071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c8a000700 1 -- 172.21.15.12:0/2055699364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c78009550 con 0x7f8c841559f0 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.786+0000 7f8c89600700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c840ac7d0 0x7f8c841554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.787+0000 7f8c8a000700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c841559f0 0x7f8c84071f50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c007e40 tx=0x7f8c7c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.787+0000 7f8c81600700 1 -- 172.21.15.12:0/2055699364 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c00ede0 con 0x7f8c841559f0 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.787+0000 7f8c81600700 1 -- 172.21.15.12:0/2055699364 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c014420 con 0x7f8c841559f0 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.787+0000 7f8c8c000700 1 -- 172.21.15.12:0/2055699364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c841f3c70 con 0x7f8c841559f0 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.787+0000 7f8c81600700 1 -- 172.21.15.12:0/2055699364 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0147d0 con 0x7f8c841559f0 2024-03-29T10:09:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.787+0000 7f8c8c000700 1 -- 172.21.15.12:0/2055699364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c840ca060 con 0x7f8c841559f0 2024-03-29T10:09:37.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.788+0000 7f8c8c000700 1 -- 172.21.15.12:0/2055699364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c84067720 con 0x7f8c841559f0 2024-03-29T10:09:37.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.789+0000 7f8c81600700 1 -- 172.21.15.12:0/2055699364 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8c7c014930 con 0x7f8c841559f0 2024-03-29T10:09:37.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.790+0000 7f8c81600700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c6406adb0 0x7f8c6406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:37.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.790+0000 7f8c81600700 1 -- 172.21.15.12:0/2055699364 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c7c089bb0 con 0x7f8c841559f0 2024-03-29T10:09:37.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.790+0000 7f8c89600700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c6406adb0 0x7f8c6406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:37.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.791+0000 7f8c89600700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c6406adb0 0x7f8c6406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009130 tx=0x7f8c78008030 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:37.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:37.794+0000 7f8c81600700 1 -- 172.21.15.12:0/2055699364 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8c7c054c70 con 0x7f8c841559f0 2024-03-29T10:09:38.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.005+0000 7f8c8c000700 1 -- 172.21.15.12:0/2055699364 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c841f3820 con 0x7f8c6406adb0 2024-03-29T10:09:38.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.013+0000 7f8c81600700 1 -- 172.21.15.12:0/2055699364 <== mgr.6013 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287655 (secure 0 0 0) 0x7f8c841f3820 con 0x7f8c6406adb0 2024-03-29T10:09:38.020 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:38.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 -- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c6406adb0 msgr2=0x7f8c6406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c6406adb0 0x7f8c6406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009130 tx=0x7f8c78008030 comp rx=0 tx=0).stop 2024-03-29T10:09:38.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 -- 172.21.15.12:0/2055699364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c841559f0 msgr2=0x7f8c84071f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c841559f0 0x7f8c84071f50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c007e40 tx=0x7f8c7c00e3c0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 -- 172.21.15.12:0/2055699364 shutdown_connections 2024-03-29T10:09:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8c6406adb0 0x7f8c6406d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c841559f0 0x7f8c84071f50 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c840ac7d0 0x7f8c841554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 --2- 172.21.15.12:0/2055699364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c8415b5d0 0x7f8c84071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 -- 172.21.15.12:0/2055699364 >> 172.21.15.12:0/2055699364 conn(0x7f8c840cf190 msgr2=0x7f8c8415dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 -- 172.21.15.12:0/2055699364 shutdown_connections 2024-03-29T10:09:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.018+0000 7f8c8c000700 1 -- 172.21.15.12:0/2055699364 wait complete. 2024-03-29T10:09:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:09:38.038 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-29T10:09:36.838605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592660,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":279,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":178,"acting":178,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":306076,"kb_used_data":8728,"kb_used_omap":12,"kb_used_meta":297203,"kb_avail":754668644,"statfs":{"total":773094113280,"available":772780691456,"internally_reserved":0,"allocated":8937472,"data_stored":4820303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":304336441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2292,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.769510"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045584+0000","last_change":"2024-03-29T10:09:32.623236+0000","last_active":"2024-03-29T10:09:34.045584+0000","last_peered":"2024-03-29T10:09:34.045584+0000","last_clean":"2024-03-29T10:09:34.045584+0000","last_became_active":"2024-03-29T10:09:32.622714+0000","last_became_peered":"2024-03-29T10:09:32.622714+0000","last_unstale":"2024-03-29T10:09:34.045584+0000","last_undegraded":"2024-03-29T10:09:34.045584+0000","last_fullsized":"2024-03-29T10:09:34.045584+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:50.745197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607572+0000","last_change":"2024-03-29T10:09:32.620014+0000","last_active":"2024-03-29T10:09:33.607572+0000","last_peered":"2024-03-29T10:09:33.607572+0000","last_clean":"2024-03-29T10:09:33.607572+0000","last_became_active":"2024-03-29T10:09:32.619747+0000","last_became_peered":"2024-03-29T10:09:32.619747+0000","last_unstale":"2024-03-29T10:09:33.607572+0000","last_undegraded":"2024-03-29T10:09:33.607572+0000","last_fullsized":"2024-03-29T10:09:33.607572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:58:19.387293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045306+0000","last_change":"2024-03-29T10:09:32.622331+0000","last_active":"2024-03-29T10:09:34.045306+0000","last_peered":"2024-03-29T10:09:34.045306+0000","last_clean":"2024-03-29T10:09:34.045306+0000","last_became_active":"2024-03-29T10:09:32.622038+0000","last_became_peered":"2024-03-29T10:09:32.622038+0000","last_unstale":"2024-03-29T10:09:34.045306+0000","last_undegraded":"2024-03-29T10:09:34.045306+0000","last_fullsized":"2024-03-29T10:09:34.045306+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:54.413063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045348+0000","last_change":"2024-03-29T10:09:32.617524+0000","last_active":"2024-03-29T10:09:34.045348+0000","last_peered":"2024-03-29T10:09:34.045348+0000","last_clean":"2024-03-29T10:09:34.045348+0000","last_became_active":"2024-03-29T10:09:32.617270+0000","last_became_peered":"2024-03-29T10:09:32.617270+0000","last_unstale":"2024-03-29T10:09:34.045348+0000","last_undegraded":"2024-03-29T10:09:34.045348+0000","last_fullsized":"2024-03-29T10:09:34.045348+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:02.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045118+0000","last_change":"2024-03-29T10:09:32.616253+0000","last_active":"2024-03-29T10:09:34.045118+0000","last_peered":"2024-03-29T10:09:34.045118+0000","last_clean":"2024-03-29T10:09:34.045118+0000","last_became_active":"2024-03-29T10:09:32.616002+0000","last_became_peered":"2024-03-29T10:09:32.616002+0000","last_unstale":"2024-03-29T10:09:34.045118+0000","last_undegraded":"2024-03-29T10:09:34.045118+0000","last_fullsized":"2024-03-29T10:09:34.045118+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:23.297716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044899+0000","last_change":"2024-03-29T10:09:32.612351+0000","last_active":"2024-03-29T10:09:34.044899+0000","last_peered":"2024-03-29T10:09:34.044899+0000","last_clean":"2024-03-29T10:09:34.044899+0000","last_became_active":"2024-03-29T10:09:32.612057+0000","last_became_peered":"2024-03-29T10:09:32.612057+0000","last_unstale":"2024-03-29T10:09:34.044899+0000","last_undegraded":"2024-03-29T10:09:34.044899+0000","last_fullsized":"2024-03-29T10:09:34.044899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:41.040149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.606905+0000","last_change":"2024-03-29T10:09:32.615482+0000","last_active":"2024-03-29T10:09:33.606905+0000","last_peered":"2024-03-29T10:09:33.606905+0000","last_clean":"2024-03-29T10:09:33.606905+0000","last_became_active":"2024-03-29T10:09:32.615198+0000","last_became_peered":"2024-03-29T10:09:32.615198+0000","last_unstale":"2024-03-29T10:09:33.606905+0000","last_undegraded":"2024-03-29T10:09:33.606905+0000","last_fullsized":"2024-03-29T10:09:33.606905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:48:35.803134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045007+0000","last_change":"2024-03-29T10:09:32.611215+0000","last_active":"2024-03-29T10:09:34.045007+0000","last_peered":"2024-03-29T10:09:34.045007+0000","last_clean":"2024-03-29T10:09:34.045007+0000","last_became_active":"2024-03-29T10:09:32.610941+0000","last_became_peered":"2024-03-29T10:09:32.610941+0000","last_unstale":"2024-03-29T10:09:34.045007+0000","last_undegraded":"2024-03-29T10:09:34.045007+0000","last_fullsized":"2024-03-29T10:09:34.045007+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:28.376324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045092+0000","last_change":"2024-03-29T10:09:32.624355+0000","last_active":"2024-03-29T10:09:34.045092+0000","last_peered":"2024-03-29T10:09:34.045092+0000","last_clean":"2024-03-29T10:09:34.045092+0000","last_became_active":"2024-03-29T10:09:32.622547+0000","last_became_peered":"2024-03-29T10:09:32.622547+0000","last_unstale":"2024-03-29T10:09:34.045092+0000","last_undegraded":"2024-03-29T10:09:34.045092+0000","last_fullsized":"2024-03-29T10:09:34.045092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:02.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044868+0000","last_change":"2024-03-29T10:09:32.628082+0000","last_active":"2024-03-29T10:09:34.044868+0000","last_peered":"2024-03-29T10:09:34.044868+0000","last_clean":"2024-03-29T10:09:34.044868+0000","last_became_active":"2024-03-29T10:09:32.627765+0000","last_became_peered":"2024-03-29T10:09:32.627765+0000","last_unstale":"2024-03-29T10:09:34.044868+0000","last_undegraded":"2024-03-29T10:09:34.044868+0000","last_fullsized":"2024-03-29T10:09:34.044868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:50.472261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.612120+0000","last_change":"2024-03-29T10:09:32.619242+0000","last_active":"2024-03-29T10:09:33.612120+0000","last_peered":"2024-03-29T10:09:33.612120+0000","last_clean":"2024-03-29T10:09:33.612120+0000","last_became_active":"2024-03-29T10:09:32.618883+0000","last_became_peered":"2024-03-29T10:09:32.618883+0000","last_unstale":"2024-03-29T10:09:33.612120+0000","last_undegraded":"2024-03-29T10:09:33.612120+0000","last_fullsized":"2024-03-29T10:09:33.612120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:08.395520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044873+0000","last_change":"2024-03-29T10:09:32.617701+0000","last_active":"2024-03-29T10:09:34.044873+0000","last_peered":"2024-03-29T10:09:34.044873+0000","last_clean":"2024-03-29T10:09:34.044873+0000","last_became_active":"2024-03-29T10:09:32.616607+0000","last_became_peered":"2024-03-29T10:09:32.616607+0000","last_unstale":"2024-03-29T10:09:34.044873+0000","last_undegraded":"2024-03-29T10:09:34.044873+0000","last_fullsized":"2024-03-29T10:09:34.044873+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:39.250301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045466+0000","last_change":"2024-03-29T10:09:32.618041+0000","last_active":"2024-03-29T10:09:34.045466+0000","last_peered":"2024-03-29T10:09:34.045466+0000","last_clean":"2024-03-29T10:09:34.045466+0000","last_became_active":"2024-03-29T10:09:32.617123+0000","last_became_peered":"2024-03-29T10:09:32.617123+0000","last_unstale":"2024-03-29T10:09:34.045466+0000","last_undegraded":"2024-03-29T10:09:34.045466+0000","last_fullsized":"2024-03-29T10:09:34.045466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:47.161213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044987+0000","last_change":"2024-03-29T10:09:32.622774+0000","last_active":"2024-03-29T10:09:34.044987+0000","last_peered":"2024-03-29T10:09:34.044987+0000","last_clean":"2024-03-29T10:09:34.044987+0000","last_became_active":"2024-03-29T10:09:32.622503+0000","last_became_peered":"2024-03-29T10:09:32.622503+0000","last_unstale":"2024-03-29T10:09:34.044987+0000","last_undegraded":"2024-03-29T10:09:34.044987+0000","last_fullsized":"2024-03-29T10:09:34.044987+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:38.603035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607446+0000","last_change":"2024-03-29T10:09:32.625499+0000","last_active":"2024-03-29T10:09:33.607446+0000","last_peered":"2024-03-29T10:09:33.607446+0000","last_clean":"2024-03-29T10:09:33.607446+0000","last_became_active":"2024-03-29T10:09:32.625200+0000","last_became_peered":"2024-03-29T10:09:32.625200+0000","last_unstale":"2024-03-29T10:09:33.607446+0000","last_undegraded":"2024-03-29T10:09:33.607446+0000","last_fullsized":"2024-03-29T10:09:33.607446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:47.193288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045942+0000","last_change":"2024-03-29T10:09:32.622371+0000","last_active":"2024-03-29T10:09:34.045942+0000","last_peered":"2024-03-29T10:09:34.045942+0000","last_clean":"2024-03-29T10:09:34.045942+0000","last_became_active":"2024-03-29T10:09:32.621892+0000","last_became_peered":"2024-03-29T10:09:32.621892+0000","last_unstale":"2024-03-29T10:09:34.045942+0000","last_undegraded":"2024-03-29T10:09:34.045942+0000","last_fullsized":"2024-03-29T10:09:34.045942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:50.946405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045592+0000","last_change":"2024-03-29T10:09:32.615044+0000","last_active":"2024-03-29T10:09:34.045592+0000","last_peered":"2024-03-29T10:09:34.045592+0000","last_clean":"2024-03-29T10:09:34.045592+0000","last_became_active":"2024-03-29T10:09:32.614754+0000","last_became_peered":"2024-03-29T10:09:32.614754+0000","last_unstale":"2024-03-29T10:09:34.045592+0000","last_undegraded":"2024-03-29T10:09:34.045592+0000","last_fullsized":"2024-03-29T10:09:34.045592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:50.472261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607170+0000","last_change":"2024-03-29T10:09:32.613680+0000","last_active":"2024-03-29T10:09:33.607170+0000","last_peered":"2024-03-29T10:09:33.607170+0000","last_clean":"2024-03-29T10:09:33.607170+0000","last_became_active":"2024-03-29T10:09:32.611800+0000","last_became_peered":"2024-03-29T10:09:32.611800+0000","last_unstale":"2024-03-29T10:09:33.607170+0000","last_undegraded":"2024-03-29T10:09:33.607170+0000","last_fullsized":"2024-03-29T10:09:33.607170+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:04.582194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045761+0000","last_change":"2024-03-29T10:09:32.623271+0000","last_active":"2024-03-29T10:09:34.045761+0000","last_peered":"2024-03-29T10:09:34.045761+0000","last_clean":"2024-03-29T10:09:34.045761+0000","last_became_active":"2024-03-29T10:09:32.622928+0000","last_became_peered":"2024-03-29T10:09:32.622928+0000","last_unstale":"2024-03-29T10:09:34.045761+0000","last_undegraded":"2024-03-29T10:09:34.045761+0000","last_fullsized":"2024-03-29T10:09:34.045761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.930784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607046+0000","last_change":"2024-03-29T10:09:32.621045+0000","last_active":"2024-03-29T10:09:33.607046+0000","last_peered":"2024-03-29T10:09:33.607046+0000","last_clean":"2024-03-29T10:09:33.607046+0000","last_became_active":"2024-03-29T10:09:32.620499+0000","last_became_peered":"2024-03-29T10:09:32.620499+0000","last_unstale":"2024-03-29T10:09:33.607046+0000","last_undegraded":"2024-03-29T10:09:33.607046+0000","last_fullsized":"2024-03-29T10:09:33.607046+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:05:04.665074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045437+0000","last_change":"2024-03-29T10:09:32.637105+0000","last_active":"2024-03-29T10:09:34.045437+0000","last_peered":"2024-03-29T10:09:34.045437+0000","last_clean":"2024-03-29T10:09:34.045437+0000","last_became_active":"2024-03-29T10:09:32.636599+0000","last_became_peered":"2024-03-29T10:09:32.636599+0000","last_unstale":"2024-03-29T10:09:34.045437+0000","last_undegraded":"2024-03-29T10:09:34.045437+0000","last_fullsized":"2024-03-29T10:09:34.045437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:54.720344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.611723+0000","last_change":"2024-03-29T10:09:32.628611+0000","last_active":"2024-03-29T10:09:33.611723+0000","last_peered":"2024-03-29T10:09:33.611723+0000","last_clean":"2024-03-29T10:09:33.611723+0000","last_became_active":"2024-03-29T10:09:32.625523+0000","last_became_peered":"2024-03-29T10:09:32.625523+0000","last_unstale":"2024-03-29T10:09:33.611723+0000","last_undegraded":"2024-03-29T10:09:33.611723+0000","last_fullsized":"2024-03-29T10:09:33.611723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:56.014772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046009+0000","last_change":"2024-03-29T10:09:32.615838+0000","last_active":"2024-03-29T10:09:34.046009+0000","last_peered":"2024-03-29T10:09:34.046009+0000","last_clean":"2024-03-29T10:09:34.046009+0000","last_became_active":"2024-03-29T10:09:32.615274+0000","last_became_peered":"2024-03-29T10:09:32.615274+0000","last_unstale":"2024-03-29T10:09:34.046009+0000","last_undegraded":"2024-03-29T10:09:34.046009+0000","last_fullsized":"2024-03-29T10:09:34.046009+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:07.635360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045741+0000","last_change":"2024-03-29T10:09:32.624374+0000","last_active":"2024-03-29T10:09:34.045741+0000","last_peered":"2024-03-29T10:09:34.045741+0000","last_clean":"2024-03-29T10:09:34.045741+0000","last_became_active":"2024-03-29T10:09:32.624101+0000","last_became_peered":"2024-03-29T10:09:32.624101+0000","last_unstale":"2024-03-29T10:09:34.045741+0000","last_undegraded":"2024-03-29T10:09:34.045741+0000","last_fullsized":"2024-03-29T10:09:34.045741+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:24:39.560229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045630+0000","last_change":"2024-03-29T10:09:32.618810+0000","last_active":"2024-03-29T10:09:34.045630+0000","last_peered":"2024-03-29T10:09:34.045630+0000","last_clean":"2024-03-29T10:09:34.045630+0000","last_became_active":"2024-03-29T10:09:32.618546+0000","last_became_peered":"2024-03-29T10:09:32.618546+0000","last_unstale":"2024-03-29T10:09:34.045630+0000","last_undegraded":"2024-03-29T10:09:34.045630+0000","last_fullsized":"2024-03-29T10:09:34.045630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:49.874012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607130+0000","last_change":"2024-03-29T10:09:32.618824+0000","last_active":"2024-03-29T10:09:33.607130+0000","last_peered":"2024-03-29T10:09:33.607130+0000","last_clean":"2024-03-29T10:09:33.607130+0000","last_became_active":"2024-03-29T10:09:32.618065+0000","last_became_peered":"2024-03-29T10:09:32.618065+0000","last_unstale":"2024-03-29T10:09:33.607130+0000","last_undegraded":"2024-03-29T10:09:33.607130+0000","last_fullsized":"2024-03-29T10:09:33.607130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:29.544771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045633+0000","last_change":"2024-03-29T10:09:32.622999+0000","last_active":"2024-03-29T10:09:34.045633+0000","last_peered":"2024-03-29T10:09:34.045633+0000","last_clean":"2024-03-29T10:09:34.045633+0000","last_became_active":"2024-03-29T10:09:32.622065+0000","last_became_peered":"2024-03-29T10:09:32.622065+0000","last_unstale":"2024-03-29T10:09:34.045633+0000","last_undegraded":"2024-03-29T10:09:34.045633+0000","last_fullsized":"2024-03-29T10:09:34.045633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:23.297716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044843+0000","last_change":"2024-03-29T10:09:32.615373+0000","last_active":"2024-03-29T10:09:34.044843+0000","last_peered":"2024-03-29T10:09:34.044843+0000","last_clean":"2024-03-29T10:09:34.044843+0000","last_became_active":"2024-03-29T10:09:32.615107+0000","last_became_peered":"2024-03-29T10:09:32.615107+0000","last_unstale":"2024-03-29T10:09:34.044843+0000","last_undegraded":"2024-03-29T10:09:34.044843+0000","last_fullsized":"2024-03-29T10:09:34.044843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:23.129448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045322+0000","last_change":"2024-03-29T10:09:32.618453+0000","last_active":"2024-03-29T10:09:34.045322+0000","last_peered":"2024-03-29T10:09:34.045322+0000","last_clean":"2024-03-29T10:09:34.045322+0000","last_became_active":"2024-03-29T10:09:32.616759+0000","last_became_peered":"2024-03-29T10:09:32.616759+0000","last_unstale":"2024-03-29T10:09:34.045322+0000","last_undegraded":"2024-03-29T10:09:34.045322+0000","last_fullsized":"2024-03-29T10:09:34.045322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:50.946405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045802+0000","last_change":"2024-03-29T10:09:32.623841+0000","last_active":"2024-03-29T10:09:34.045802+0000","last_peered":"2024-03-29T10:09:34.045802+0000","last_clean":"2024-03-29T10:09:34.045802+0000","last_became_active":"2024-03-29T10:09:32.623522+0000","last_became_peered":"2024-03-29T10:09:32.623522+0000","last_unstale":"2024-03-29T10:09:34.045802+0000","last_undegraded":"2024-03-29T10:09:34.045802+0000","last_fullsized":"2024-03-29T10:09:34.045802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:40.815771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"79'4","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638507+0000","last_change":"2024-03-29T10:09:32.101864+0000","last_active":"2024-03-29T10:09:32.638507+0000","last_peered":"2024-03-29T10:09:32.638507+0000","last_clean":"2024-03-29T10:09:32.638507+0000","last_became_active":"2024-03-29T10:09:32.101456+0000","last_became_peered":"2024-03-29T10:09:32.101456+0000","last_unstale":"2024-03-29T10:09:32.638507+0000","last_undegraded":"2024-03-29T10:09:32.638507+0000","last_fullsized":"2024-03-29T10:09:32.638507+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:05.714596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045625+0000","last_change":"2024-03-29T10:09:32.630347+0000","last_active":"2024-03-29T10:09:34.045625+0000","last_peered":"2024-03-29T10:09:34.045625+0000","last_clean":"2024-03-29T10:09:34.045625+0000","last_became_active":"2024-03-29T10:09:32.630092+0000","last_became_peered":"2024-03-29T10:09:32.630092+0000","last_unstale":"2024-03-29T10:09:34.045625+0000","last_undegraded":"2024-03-29T10:09:34.045625+0000","last_fullsized":"2024-03-29T10:09:34.045625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:50.745197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.635936+0000","last_change":"2024-03-29T10:09:32.091373+0000","last_active":"2024-03-29T10:09:32.635936+0000","last_peered":"2024-03-29T10:09:32.635936+0000","last_clean":"2024-03-29T10:09:32.635936+0000","last_became_active":"2024-03-29T10:09:32.091110+0000","last_became_peered":"2024-03-29T10:09:32.091110+0000","last_unstale":"2024-03-29T10:09:32.635936+0000","last_undegraded":"2024-03-29T10:09:32.635936+0000","last_fullsized":"2024-03-29T10:09:32.635936+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:32:31.535691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046042+0000","last_change":"2024-03-29T10:09:32.614418+0000","last_active":"2024-03-29T10:09:34.046042+0000","last_peered":"2024-03-29T10:09:34.046042+0000","last_clean":"2024-03-29T10:09:34.046042+0000","last_became_active":"2024-03-29T10:09:32.614112+0000","last_became_peered":"2024-03-29T10:09:32.614112+0000","last_unstale":"2024-03-29T10:09:34.046042+0000","last_undegraded":"2024-03-29T10:09:34.046042+0000","last_fullsized":"2024-03-29T10:09:34.046042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:17.345552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.635794+0000","last_change":"2024-03-29T10:09:32.086998+0000","last_active":"2024-03-29T10:09:32.635794+0000","last_peered":"2024-03-29T10:09:32.635794+0000","last_clean":"2024-03-29T10:09:32.635794+0000","last_became_active":"2024-03-29T10:09:32.086644+0000","last_became_peered":"2024-03-29T10:09:32.086644+0000","last_unstale":"2024-03-29T10:09:32.635794+0000","last_undegraded":"2024-03-29T10:09:32.635794+0000","last_fullsized":"2024-03-29T10:09:32.635794+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:27.370099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045835+0000","last_change":"2024-03-29T10:09:32.624387+0000","last_active":"2024-03-29T10:09:34.045835+0000","last_peered":"2024-03-29T10:09:34.045835+0000","last_clean":"2024-03-29T10:09:34.045835+0000","last_became_active":"2024-03-29T10:09:32.624015+0000","last_became_peered":"2024-03-29T10:09:32.624015+0000","last_unstale":"2024-03-29T10:09:34.045835+0000","last_undegraded":"2024-03-29T10:09:34.045835+0000","last_fullsized":"2024-03-29T10:09:34.045835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:06.669665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.072330+0000","last_change":"2024-03-29T10:09:31.072330+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.072330+0000","last_undegraded":"2024-03-29T10:09:31.072330+0000","last_fullsized":"2024-03-29T10:09:31.072330+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607151+0000","last_change":"2024-03-29T10:09:32.620748+0000","last_active":"2024-03-29T10:09:33.607151+0000","last_peered":"2024-03-29T10:09:33.607151+0000","last_clean":"2024-03-29T10:09:33.607151+0000","last_became_active":"2024-03-29T10:09:32.618821+0000","last_became_peered":"2024-03-29T10:09:32.618821+0000","last_unstale":"2024-03-29T10:09:33.607151+0000","last_undegraded":"2024-03-29T10:09:33.607151+0000","last_fullsized":"2024-03-29T10:09:33.607151+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:39:35.761571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.632616+0000","last_change":"2024-03-29T10:09:32.098141+0000","last_active":"2024-03-29T10:09:32.632616+0000","last_peered":"2024-03-29T10:09:32.632616+0000","last_clean":"2024-03-29T10:09:32.632616+0000","last_became_active":"2024-03-29T10:09:32.097781+0000","last_became_peered":"2024-03-29T10:09:32.097781+0000","last_unstale":"2024-03-29T10:09:32.632616+0000","last_undegraded":"2024-03-29T10:09:32.632616+0000","last_fullsized":"2024-03-29T10:09:32.632616+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607233+0000","last_change":"2024-03-29T10:09:32.614343+0000","last_active":"2024-03-29T10:09:33.607233+0000","last_peered":"2024-03-29T10:09:33.607233+0000","last_clean":"2024-03-29T10:09:33.607233+0000","last_became_active":"2024-03-29T10:09:32.614069+0000","last_became_peered":"2024-03-29T10:09:32.614069+0000","last_unstale":"2024-03-29T10:09:33.607233+0000","last_undegraded":"2024-03-29T10:09:33.607233+0000","last_fullsized":"2024-03-29T10:09:33.607233+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:22.440684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638211+0000","last_change":"2024-03-29T10:09:32.101572+0000","last_active":"2024-03-29T10:09:32.638211+0000","last_peered":"2024-03-29T10:09:32.638211+0000","last_clean":"2024-03-29T10:09:32.638211+0000","last_became_active":"2024-03-29T10:09:32.101206+0000","last_became_peered":"2024-03-29T10:09:32.101206+0000","last_unstale":"2024-03-29T10:09:32.638211+0000","last_undegraded":"2024-03-29T10:09:32.638211+0000","last_fullsized":"2024-03-29T10:09:32.638211+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:46.154988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.611858+0000","last_change":"2024-03-29T10:09:32.624716+0000","last_active":"2024-03-29T10:09:33.611858+0000","last_peered":"2024-03-29T10:09:33.611858+0000","last_clean":"2024-03-29T10:09:33.611858+0000","last_became_active":"2024-03-29T10:09:32.622676+0000","last_became_peered":"2024-03-29T10:09:32.622676+0000","last_unstale":"2024-03-29T10:09:33.611858+0000","last_undegraded":"2024-03-29T10:09:33.611858+0000","last_fullsized":"2024-03-29T10:09:33.611858+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:20.681595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.073144+0000","last_change":"2024-03-29T10:09:31.073144+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.073144+0000","last_undegraded":"2024-03-29T10:09:31.073144+0000","last_fullsized":"2024-03-29T10:09:31.073144+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607448+0000","last_change":"2024-03-29T10:09:32.634701+0000","last_active":"2024-03-29T10:09:33.607448+0000","last_peered":"2024-03-29T10:09:33.607448+0000","last_clean":"2024-03-29T10:09:33.607448+0000","last_became_active":"2024-03-29T10:09:32.631830+0000","last_became_peered":"2024-03-29T10:09:32.631830+0000","last_unstale":"2024-03-29T10:09:33.607448+0000","last_undegraded":"2024-03-29T10:09:33.607448+0000","last_fullsized":"2024-03-29T10:09:33.607448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:29.066092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.603049+0000","last_change":"2024-03-29T10:09:32.097646+0000","last_active":"2024-03-29T10:09:32.603049+0000","last_peered":"2024-03-29T10:09:32.603049+0000","last_clean":"2024-03-29T10:09:32.603049+0000","last_became_active":"2024-03-29T10:09:32.097364+0000","last_became_peered":"2024-03-29T10:09:32.097364+0000","last_unstale":"2024-03-29T10:09:32.603049+0000","last_undegraded":"2024-03-29T10:09:32.603049+0000","last_fullsized":"2024-03-29T10:09:32.603049+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045478+0000","last_change":"2024-03-29T10:09:32.617841+0000","last_active":"2024-03-29T10:09:34.045478+0000","last_peered":"2024-03-29T10:09:34.045478+0000","last_clean":"2024-03-29T10:09:34.045478+0000","last_became_active":"2024-03-29T10:09:32.616581+0000","last_became_peered":"2024-03-29T10:09:32.616581+0000","last_unstale":"2024-03-29T10:09:34.045478+0000","last_undegraded":"2024-03-29T10:09:34.045478+0000","last_fullsized":"2024-03-29T10:09:34.045478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:15.353895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.616868+0000","last_change":"2024-03-29T10:09:32.098837+0000","last_active":"2024-03-29T10:09:32.616868+0000","last_peered":"2024-03-29T10:09:32.616868+0000","last_clean":"2024-03-29T10:09:32.616868+0000","last_became_active":"2024-03-29T10:09:32.098517+0000","last_became_peered":"2024-03-29T10:09:32.098517+0000","last_unstale":"2024-03-29T10:09:32.616868+0000","last_undegraded":"2024-03-29T10:09:32.616868+0000","last_fullsized":"2024-03-29T10:09:32.616868+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:05.714596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045923+0000","last_change":"2024-03-29T10:09:32.636148+0000","last_active":"2024-03-29T10:09:34.045923+0000","last_peered":"2024-03-29T10:09:34.045923+0000","last_clean":"2024-03-29T10:09:34.045923+0000","last_became_active":"2024-03-29T10:09:32.635753+0000","last_became_peered":"2024-03-29T10:09:32.635753+0000","last_unstale":"2024-03-29T10:09:34.045923+0000","last_undegraded":"2024-03-29T10:09:34.045923+0000","last_fullsized":"2024-03-29T10:09:34.045923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:18.667325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.608025+0000","last_change":"2024-03-29T10:09:32.099480+0000","last_active":"2024-03-29T10:09:32.608025+0000","last_peered":"2024-03-29T10:09:32.608025+0000","last_clean":"2024-03-29T10:09:32.608025+0000","last_became_active":"2024-03-29T10:09:32.099133+0000","last_became_peered":"2024-03-29T10:09:32.099133+0000","last_unstale":"2024-03-29T10:09:32.608025+0000","last_undegraded":"2024-03-29T10:09:32.608025+0000","last_fullsized":"2024-03-29T10:09:32.608025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:13:16.625121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044664+0000","last_change":"2024-03-29T10:09:32.628412+0000","last_active":"2024-03-29T10:09:34.044664+0000","last_peered":"2024-03-29T10:09:34.044664+0000","last_clean":"2024-03-29T10:09:34.044664+0000","last_became_active":"2024-03-29T10:09:32.622661+0000","last_became_peered":"2024-03-29T10:09:32.622661+0000","last_unstale":"2024-03-29T10:09:34.044664+0000","last_undegraded":"2024-03-29T10:09:34.044664+0000","last_fullsized":"2024-03-29T10:09:34.044664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:23.129448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"79'3","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636825+0000","last_change":"2024-03-29T10:09:32.089351+0000","last_active":"2024-03-29T10:09:32.636825+0000","last_peered":"2024-03-29T10:09:32.636825+0000","last_clean":"2024-03-29T10:09:32.636825+0000","last_became_active":"2024-03-29T10:09:32.087924+0000","last_became_peered":"2024-03-29T10:09:32.087924+0000","last_unstale":"2024-03-29T10:09:32.636825+0000","last_undegraded":"2024-03-29T10:09:32.636825+0000","last_fullsized":"2024-03-29T10:09:32.636825+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:46.154988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045696+0000","last_change":"2024-03-29T10:09:32.632023+0000","last_active":"2024-03-29T10:09:34.045696+0000","last_peered":"2024-03-29T10:09:34.045696+0000","last_clean":"2024-03-29T10:09:34.045696+0000","last_became_active":"2024-03-29T10:09:32.631450+0000","last_became_peered":"2024-03-29T10:09:32.631450+0000","last_unstale":"2024-03-29T10:09:34.045696+0000","last_undegraded":"2024-03-29T10:09:34.045696+0000","last_fullsized":"2024-03-29T10:09:34.045696+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:24:39.560229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"79'6","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638720+0000","last_change":"2024-03-29T10:09:32.099892+0000","last_active":"2024-03-29T10:09:32.638720+0000","last_peered":"2024-03-29T10:09:32.638720+0000","last_clean":"2024-03-29T10:09:32.638720+0000","last_became_active":"2024-03-29T10:09:32.099626+0000","last_became_peered":"2024-03-29T10:09:32.099626+0000","last_unstale":"2024-03-29T10:09:32.638720+0000","last_undegraded":"2024-03-29T10:09:32.638720+0000","last_fullsized":"2024-03-29T10:09:32.638720+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636457+0000","last_change":"2024-03-29T10:09:32.100281+0000","last_active":"2024-03-29T10:09:32.636457+0000","last_peered":"2024-03-29T10:09:32.636457+0000","last_clean":"2024-03-29T10:09:32.636457+0000","last_became_active":"2024-03-29T10:09:32.100004+0000","last_became_peered":"2024-03-29T10:09:32.100004+0000","last_unstale":"2024-03-29T10:09:32.636457+0000","last_undegraded":"2024-03-29T10:09:32.636457+0000","last_fullsized":"2024-03-29T10:09:32.636457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:49.940180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045675+0000","last_change":"2024-03-29T10:09:32.624300+0000","last_active":"2024-03-29T10:09:34.045675+0000","last_peered":"2024-03-29T10:09:34.045675+0000","last_clean":"2024-03-29T10:09:34.045675+0000","last_became_active":"2024-03-29T10:09:32.624004+0000","last_became_peered":"2024-03-29T10:09:32.624004+0000","last_unstale":"2024-03-29T10:09:34.045675+0000","last_undegraded":"2024-03-29T10:09:34.045675+0000","last_fullsized":"2024-03-29T10:09:34.045675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:38.603035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607519+0000","last_change":"2024-03-29T10:09:32.091147+0000","last_active":"2024-03-29T10:09:32.607519+0000","last_peered":"2024-03-29T10:09:32.607519+0000","last_clean":"2024-03-29T10:09:32.607519+0000","last_became_active":"2024-03-29T10:09:32.089656+0000","last_became_peered":"2024-03-29T10:09:32.089656+0000","last_unstale":"2024-03-29T10:09:32.607519+0000","last_undegraded":"2024-03-29T10:09:32.607519+0000","last_fullsized":"2024-03-29T10:09:32.607519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:22.123223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044770+0000","last_change":"2024-03-29T10:09:32.623116+0000","last_active":"2024-03-29T10:09:34.044770+0000","last_peered":"2024-03-29T10:09:34.044770+0000","last_clean":"2024-03-29T10:09:34.044770+0000","last_became_active":"2024-03-29T10:09:32.622832+0000","last_became_peered":"2024-03-29T10:09:32.622832+0000","last_unstale":"2024-03-29T10:09:34.044770+0000","last_undegraded":"2024-03-29T10:09:34.044770+0000","last_fullsized":"2024-03-29T10:09:34.044770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:36.451327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.071636+0000","last_change":"2024-03-29T10:09:31.071636+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.071636+0000","last_undegraded":"2024-03-29T10:09:31.071636+0000","last_fullsized":"2024-03-29T10:09:31.071636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045252+0000","last_change":"2024-03-29T10:09:32.634186+0000","last_active":"2024-03-29T10:09:34.045252+0000","last_peered":"2024-03-29T10:09:34.045252+0000","last_clean":"2024-03-29T10:09:34.045252+0000","last_became_active":"2024-03-29T10:09:32.633693+0000","last_became_peered":"2024-03-29T10:09:32.633693+0000","last_unstale":"2024-03-29T10:09:34.045252+0000","last_undegraded":"2024-03-29T10:09:34.045252+0000","last_fullsized":"2024-03-29T10:09:34.045252+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:40.815771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"79'8","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.639114+0000","last_change":"2024-03-29T10:09:32.092854+0000","last_active":"2024-03-29T10:09:32.639114+0000","last_peered":"2024-03-29T10:09:32.639114+0000","last_clean":"2024-03-29T10:09:32.639114+0000","last_became_active":"2024-03-29T10:09:32.092591+0000","last_became_peered":"2024-03-29T10:09:32.092591+0000","last_unstale":"2024-03-29T10:09:32.639114+0000","last_undegraded":"2024-03-29T10:09:32.639114+0000","last_fullsized":"2024-03-29T10:09:32.639114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:40.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045937+0000","last_change":"2024-03-29T10:09:32.623867+0000","last_active":"2024-03-29T10:09:34.045937+0000","last_peered":"2024-03-29T10:09:34.045937+0000","last_clean":"2024-03-29T10:09:34.045937+0000","last_became_active":"2024-03-29T10:09:32.622284+0000","last_became_peered":"2024-03-29T10:09:32.622284+0000","last_unstale":"2024-03-29T10:09:34.045937+0000","last_undegraded":"2024-03-29T10:09:34.045937+0000","last_fullsized":"2024-03-29T10:09:34.045937+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:06.669665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636096+0000","last_change":"2024-03-29T10:09:32.097805+0000","last_active":"2024-03-29T10:09:32.636096+0000","last_peered":"2024-03-29T10:09:32.636096+0000","last_clean":"2024-03-29T10:09:32.636096+0000","last_became_active":"2024-03-29T10:09:32.097537+0000","last_became_peered":"2024-03-29T10:09:32.097537+0000","last_unstale":"2024-03-29T10:09:32.636096+0000","last_undegraded":"2024-03-29T10:09:32.636096+0000","last_fullsized":"2024-03-29T10:09:32.636096+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:35.016753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.634994+0000","last_change":"2024-03-29T10:09:32.104998+0000","last_active":"2024-03-29T10:09:32.634994+0000","last_peered":"2024-03-29T10:09:32.634994+0000","last_clean":"2024-03-29T10:09:32.634994+0000","last_became_active":"2024-03-29T10:09:32.104610+0000","last_became_peered":"2024-03-29T10:09:32.104610+0000","last_unstale":"2024-03-29T10:09:32.634994+0000","last_undegraded":"2024-03-29T10:09:32.634994+0000","last_fullsized":"2024-03-29T10:09:32.634994+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:38.244076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607169+0000","last_change":"2024-03-29T10:09:32.619003+0000","last_active":"2024-03-29T10:09:33.607169+0000","last_peered":"2024-03-29T10:09:33.607169+0000","last_clean":"2024-03-29T10:09:33.607169+0000","last_became_active":"2024-03-29T10:09:32.617951+0000","last_became_peered":"2024-03-29T10:09:32.617951+0000","last_unstale":"2024-03-29T10:09:33.607169+0000","last_undegraded":"2024-03-29T10:09:33.607169+0000","last_fullsized":"2024-03-29T10:09:33.607169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:00.709669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.072860+0000","last_change":"2024-03-29T10:09:31.072860+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.072860+0000","last_undegraded":"2024-03-29T10:09:31.072860+0000","last_fullsized":"2024-03-29T10:09:31.072860+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:27:35.691958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045708+0000","last_change":"2024-03-29T10:09:32.633244+0000","last_active":"2024-03-29T10:09:34.045708+0000","last_peered":"2024-03-29T10:09:34.045708+0000","last_clean":"2024-03-29T10:09:34.045708+0000","last_became_active":"2024-03-29T10:09:32.632943+0000","last_became_peered":"2024-03-29T10:09:32.632943+0000","last_unstale":"2024-03-29T10:09:34.045708+0000","last_undegraded":"2024-03-29T10:09:34.045708+0000","last_fullsized":"2024-03-29T10:09:34.045708+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:36.022978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.606805+0000","last_change":"2024-03-29T10:09:32.089495+0000","last_active":"2024-03-29T10:09:32.606805+0000","last_peered":"2024-03-29T10:09:32.606805+0000","last_clean":"2024-03-29T10:09:32.606805+0000","last_became_active":"2024-03-29T10:09:32.089228+0000","last_became_peered":"2024-03-29T10:09:32.089228+0000","last_unstale":"2024-03-29T10:09:32.606805+0000","last_undegraded":"2024-03-29T10:09:32.606805+0000","last_fullsized":"2024-03-29T10:09:32.606805+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:14.347670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636359+0000","last_change":"2024-03-29T10:09:32.095956+0000","last_active":"2024-03-29T10:09:32.636359+0000","last_peered":"2024-03-29T10:09:32.636359+0000","last_clean":"2024-03-29T10:09:32.636359+0000","last_became_active":"2024-03-29T10:09:32.093465+0000","last_became_peered":"2024-03-29T10:09:32.093465+0000","last_unstale":"2024-03-29T10:09:32.636359+0000","last_undegraded":"2024-03-29T10:09:32.636359+0000","last_fullsized":"2024-03-29T10:09:32.636359+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.686777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045534+0000","last_change":"2024-03-29T10:09:32.622232+0000","last_active":"2024-03-29T10:09:34.045534+0000","last_peered":"2024-03-29T10:09:34.045534+0000","last_clean":"2024-03-29T10:09:34.045534+0000","last_became_active":"2024-03-29T10:09:32.621815+0000","last_became_peered":"2024-03-29T10:09:32.621815+0000","last_unstale":"2024-03-29T10:09:34.045534+0000","last_undegraded":"2024-03-29T10:09:34.045534+0000","last_fullsized":"2024-03-29T10:09:34.045534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:54.720344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607356+0000","last_change":"2024-03-29T10:09:32.099575+0000","last_active":"2024-03-29T10:09:32.607356+0000","last_peered":"2024-03-29T10:09:32.607356+0000","last_clean":"2024-03-29T10:09:32.607356+0000","last_became_active":"2024-03-29T10:09:32.098434+0000","last_became_peered":"2024-03-29T10:09:32.098434+0000","last_unstale":"2024-03-29T10:09:32.607356+0000","last_undegraded":"2024-03-29T10:09:32.607356+0000","last_fullsized":"2024-03-29T10:09:32.607356+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:39.694641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044600+0000","last_change":"2024-03-29T10:09:32.623727+0000","last_active":"2024-03-29T10:09:34.044600+0000","last_peered":"2024-03-29T10:09:34.044600+0000","last_clean":"2024-03-29T10:09:34.044600+0000","last_became_active":"2024-03-29T10:09:32.623445+0000","last_became_peered":"2024-03-29T10:09:32.623445+0000","last_unstale":"2024-03-29T10:09:34.044600+0000","last_undegraded":"2024-03-29T10:09:34.044600+0000","last_fullsized":"2024-03-29T10:09:34.044600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:18.667325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.072820+0000","last_change":"2024-03-29T10:09:31.072820+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.072820+0000","last_undegraded":"2024-03-29T10:09:31.072820+0000","last_fullsized":"2024-03-29T10:09:31.072820+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:51:58.170982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.637133+0000","last_change":"2024-03-29T10:09:32.089390+0000","last_active":"2024-03-29T10:09:32.637133+0000","last_peered":"2024-03-29T10:09:32.637133+0000","last_clean":"2024-03-29T10:09:32.637133+0000","last_became_active":"2024-03-29T10:09:32.089088+0000","last_became_peered":"2024-03-29T10:09:32.089088+0000","last_unstale":"2024-03-29T10:09:32.637133+0000","last_undegraded":"2024-03-29T10:09:32.637133+0000","last_fullsized":"2024-03-29T10:09:32.637133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:11:34.599249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T10:09:32.067771+0000","last_change":"2024-03-29T10:09:32.067771+0000","last_active":"2024-03-29T10:09:32.067771+0000","last_peered":"2024-03-29T10:09:32.067771+0000","last_clean":"2024-03-29T10:09:32.067771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:32.067771+0000","last_undegraded":"2024-03-29T10:09:32.067771+0000","last_fullsized":"2024-03-29T10:09:32.067771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.609775+0000","last_change":"2024-03-29T10:09:32.101276+0000","last_active":"2024-03-29T10:09:32.609775+0000","last_peered":"2024-03-29T10:09:32.609775+0000","last_clean":"2024-03-29T10:09:32.609775+0000","last_became_active":"2024-03-29T10:09:32.100949+0000","last_became_peered":"2024-03-29T10:09:32.100949+0000","last_unstale":"2024-03-29T10:09:32.609775+0000","last_undegraded":"2024-03-29T10:09:32.609775+0000","last_fullsized":"2024-03-29T10:09:32.609775+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:40.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044751+0000","last_change":"2024-03-29T10:09:32.611596+0000","last_active":"2024-03-29T10:09:34.044751+0000","last_peered":"2024-03-29T10:09:34.044751+0000","last_clean":"2024-03-29T10:09:34.044751+0000","last_became_active":"2024-03-29T10:09:32.610142+0000","last_became_peered":"2024-03-29T10:09:32.610142+0000","last_unstale":"2024-03-29T10:09:34.044751+0000","last_undegraded":"2024-03-29T10:09:34.044751+0000","last_fullsized":"2024-03-29T10:09:34.044751+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:07.635360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605517+0000","last_change":"2024-03-29T10:09:32.092374+0000","last_active":"2024-03-29T10:09:32.605517+0000","last_peered":"2024-03-29T10:09:32.605517+0000","last_clean":"2024-03-29T10:09:32.605517+0000","last_became_active":"2024-03-29T10:09:32.091381+0000","last_became_peered":"2024-03-29T10:09:32.091381+0000","last_unstale":"2024-03-29T10:09:32.605517+0000","last_undegraded":"2024-03-29T10:09:32.605517+0000","last_fullsized":"2024-03-29T10:09:32.605517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:06.629135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046025+0000","last_change":"2024-03-29T10:09:32.620976+0000","last_active":"2024-03-29T10:09:34.046025+0000","last_peered":"2024-03-29T10:09:34.046025+0000","last_clean":"2024-03-29T10:09:34.046025+0000","last_became_active":"2024-03-29T10:09:32.619271+0000","last_became_peered":"2024-03-29T10:09:32.619271+0000","last_unstale":"2024-03-29T10:09:34.046025+0000","last_undegraded":"2024-03-29T10:09:34.046025+0000","last_fullsized":"2024-03-29T10:09:34.046025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.930784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-03-29T10:09:31.071841+0000","last_change":"2024-03-29T10:09:31.071841+0000","last_active":"2024-03-29T10:09:31.061546+0000","last_peered":"2024-03-29T10:09:31.061546+0000","last_clean":"2024-03-29T10:09:31.061546+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T10:09:31.071841+0000","last_undegraded":"2024-03-29T10:09:31.071841+0000","last_fullsized":"2024-03-29T10:09:31.071841+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:56.946534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607553+0000","last_change":"2024-03-29T10:09:32.618538+0000","last_active":"2024-03-29T10:09:33.607553+0000","last_peered":"2024-03-29T10:09:33.607553+0000","last_clean":"2024-03-29T10:09:33.607553+0000","last_became_active":"2024-03-29T10:09:32.617981+0000","last_became_peered":"2024-03-29T10:09:32.617981+0000","last_unstale":"2024-03-29T10:09:33.607553+0000","last_undegraded":"2024-03-29T10:09:33.607553+0000","last_fullsized":"2024-03-29T10:09:33.607553+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:50.197997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607488+0000","last_change":"2024-03-29T10:09:32.084059+0000","last_active":"2024-03-29T10:09:32.607488+0000","last_peered":"2024-03-29T10:09:32.607488+0000","last_clean":"2024-03-29T10:09:32.607488+0000","last_became_active":"2024-03-29T10:09:32.082631+0000","last_became_peered":"2024-03-29T10:09:32.082631+0000","last_unstale":"2024-03-29T10:09:32.607488+0000","last_undegraded":"2024-03-29T10:09:32.607488+0000","last_fullsized":"2024-03-29T10:09:32.607488+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:37:31.209166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045792+0000","last_change":"2024-03-29T10:09:32.628132+0000","last_active":"2024-03-29T10:09:34.045792+0000","last_peered":"2024-03-29T10:09:34.045792+0000","last_clean":"2024-03-29T10:09:34.045792+0000","last_became_active":"2024-03-29T10:09:32.627845+0000","last_became_peered":"2024-03-29T10:09:32.627845+0000","last_unstale":"2024-03-29T10:09:34.045792+0000","last_undegraded":"2024-03-29T10:09:34.045792+0000","last_fullsized":"2024-03-29T10:09:34.045792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:52.322978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.604249+0000","last_change":"2024-03-29T10:09:32.094306+0000","last_active":"2024-03-29T10:09:32.604249+0000","last_peered":"2024-03-29T10:09:32.604249+0000","last_clean":"2024-03-29T10:09:32.604249+0000","last_became_active":"2024-03-29T10:09:32.093998+0000","last_became_peered":"2024-03-29T10:09:32.093998+0000","last_unstale":"2024-03-29T10:09:32.604249+0000","last_undegraded":"2024-03-29T10:09:32.604249+0000","last_fullsized":"2024-03-29T10:09:32.604249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:17.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044480+0000","last_change":"2024-03-29T10:09:32.620507+0000","last_active":"2024-03-29T10:09:34.044480+0000","last_peered":"2024-03-29T10:09:34.044480+0000","last_clean":"2024-03-29T10:09:34.044480+0000","last_became_active":"2024-03-29T10:09:32.620186+0000","last_became_peered":"2024-03-29T10:09:32.620186+0000","last_unstale":"2024-03-29T10:09:34.044480+0000","last_undegraded":"2024-03-29T10:09:34.044480+0000","last_fullsized":"2024-03-29T10:09:34.044480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:49.874012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605757+0000","last_change":"2024-03-29T10:09:32.086897+0000","last_active":"2024-03-29T10:09:32.605757+0000","last_peered":"2024-03-29T10:09:32.605757+0000","last_clean":"2024-03-29T10:09:32.605757+0000","last_became_active":"2024-03-29T10:09:32.086602+0000","last_became_peered":"2024-03-29T10:09:32.086602+0000","last_unstale":"2024-03-29T10:09:32.605757+0000","last_undegraded":"2024-03-29T10:09:32.605757+0000","last_fullsized":"2024-03-29T10:09:32.605757+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:23:18.397722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607519+0000","last_change":"2024-03-29T10:09:32.633167+0000","last_active":"2024-03-29T10:09:33.607519+0000","last_peered":"2024-03-29T10:09:33.607519+0000","last_clean":"2024-03-29T10:09:33.607519+0000","last_became_active":"2024-03-29T10:09:32.632682+0000","last_became_peered":"2024-03-29T10:09:32.632682+0000","last_unstale":"2024-03-29T10:09:33.607519+0000","last_undegraded":"2024-03-29T10:09:33.607519+0000","last_fullsized":"2024-03-29T10:09:33.607519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:07.268799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605003+0000","last_change":"2024-03-29T10:09:32.092768+0000","last_active":"2024-03-29T10:09:32.605003+0000","last_peered":"2024-03-29T10:09:32.605003+0000","last_clean":"2024-03-29T10:09:32.605003+0000","last_became_active":"2024-03-29T10:09:32.092488+0000","last_became_peered":"2024-03-29T10:09:32.092488+0000","last_unstale":"2024-03-29T10:09:32.605003+0000","last_undegraded":"2024-03-29T10:09:32.605003+0000","last_fullsized":"2024-03-29T10:09:32.605003+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:16.339327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.606964+0000","last_change":"2024-03-29T10:09:32.628533+0000","last_active":"2024-03-29T10:09:33.606964+0000","last_peered":"2024-03-29T10:09:33.606964+0000","last_clean":"2024-03-29T10:09:33.606964+0000","last_became_active":"2024-03-29T10:09:32.628297+0000","last_became_peered":"2024-03-29T10:09:32.628297+0000","last_unstale":"2024-03-29T10:09:33.606964+0000","last_undegraded":"2024-03-29T10:09:33.606964+0000","last_fullsized":"2024-03-29T10:09:33.606964+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:53:16.735792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.604907+0000","last_change":"2024-03-29T10:09:32.088461+0000","last_active":"2024-03-29T10:09:32.604907+0000","last_peered":"2024-03-29T10:09:32.604907+0000","last_clean":"2024-03-29T10:09:32.604907+0000","last_became_active":"2024-03-29T10:09:32.087924+0000","last_became_peered":"2024-03-29T10:09:32.087924+0000","last_unstale":"2024-03-29T10:09:32.604907+0000","last_undegraded":"2024-03-29T10:09:32.604907+0000","last_fullsized":"2024-03-29T10:09:32.604907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:01.534207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"68'161","reported_seq":379,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.613595+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:09:32.613595+0000","last_peered":"2024-03-29T10:09:32.613595+0000","last_clean":"2024-03-29T10:09:32.613595+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:09:32.613595+0000","last_undegraded":"2024-03-29T10:09:32.613595+0000","last_fullsized":"2024-03-29T10:09:32.613595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":112,"acting":112,"num_store_stats":7},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2292,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1568768,"data_stored":1047510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.57299999999999995,"5min":0.44500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95299999999999996,"5min":1.0029999999999999,"15min":1.0109999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.69999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.89800000000000002,"5min":0.93400000000000005,"15min":1.046},"last":0.72299999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71699999999999997,"15min":0.71999999999999997},"min":{"1min":0.51900000000000002,"5min":0.47399999999999998,"15min":0.44500000000000001},"max":{"1min":0.92600000000000005,"5min":0.93200000000000005,"15min":1.1910000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.73499999999999999},"min":{"1min":0.51700000000000002,"5min":0.42199999999999999,"15min":0.372},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1},"last":0.71099999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.72899999999999998,"15min":0.74099999999999999},"min":{"1min":0.47199999999999998,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.046},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.755,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":1.079},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.48099999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.93100000000000005,"5min":0.97699999999999998,"15min":1.081},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77500000000000002,"15min":0.78300000000000003},"min":{"1min":0.55800000000000005,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":0.96199999999999997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.74199999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.29199999999999998},"max":{"1min":0.86399999999999999,"5min":0.87,"15min":1.111},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.627,"15min":0.64100000000000001},"min":{"1min":0.44800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":0.95199999999999996,"15min":0.95599999999999996},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64900000000000002,"15min":0.65900000000000003},"min":{"1min":0.432,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95299999999999996,"5min":0.96099999999999997,"15min":1.0089999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67600000000000005,"15min":0.68899999999999995},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90100000000000002,"5min":0.92800000000000005,"15min":1.141},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.71899999999999997,"15min":0.73299999999999998},"min":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.49199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":1.1679999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69699999999999995,"15min":0.70499999999999996},"min":{"1min":0.54800000000000004,"5min":0.48499999999999999,"15min":0.435},"max":{"1min":0.93799999999999994,"5min":0.98399999999999999,"15min":1.121},"last":0.48499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575031,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":966656,"data_stored":456982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70299999999999996,"15min":0.745},"min":{"1min":0.57199999999999995,"5min":0.42799999999999999,"15min":0.34699999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":7.4080000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.72599999999999998,"15min":0.76900000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.42499999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":6.7610000000000001},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70999999999999996,"15min":0.71899999999999997},"min":{"1min":0.51300000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.2549999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.72499999999999998,"15min":0.71999999999999997},"min":{"1min":0.58999999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":0.995,"15min":1.0209999999999999},"last":0.745}]},{"osd":2,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.71699999999999997,"15min":0.72399999999999998},"min":{"1min":0.65900000000000003,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94199999999999995,"5min":0.96899999999999997,"15min":1.3169999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.58899999999999997,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.415},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.74099999999999999,"15min":0.76900000000000002},"min":{"1min":0.60399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.98999999999999999,"5min":1.02,"15min":3.1960000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76800000000000002,"15min":0.79800000000000004},"min":{"1min":0.71499999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0109999999999999,"5min":1.014,"15min":4.3760000000000003},"last":0.70499999999999996}]},{"osd":4,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":1.109},"last":0.753},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.47799999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85899999999999999,"5min":0.95599999999999996,"15min":3.1219999999999999},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.67300000000000004},"min":{"1min":0.50600000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90000000000000002,"5min":0.90900000000000003,"15min":2.8279999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.79500000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.74399999999999999,"15min":0.76100000000000001},"min":{"1min":0.56499999999999995,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.95499999999999996,"5min":1.105,"15min":1.2769999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.496,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94499999999999995,"5min":0.998,"15min":1.7869999999999999},"last":0.76200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50116,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321724,"statfs":{"total":96636764160,"available":96585445376,"internally_reserved":0,"allocated":970752,"data_stored":457072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.315,"5min":0.219,"15min":0.219},"max":{"1min":0.65300000000000002,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.379},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.68999999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.504}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71599999999999997,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.52900000000000003},"min":{"1min":0.36699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":4.282},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.70999999999999996,"5min":0.78500000000000003,"15min":4.3200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.36599999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.016,"5min":1.617,"15min":1.617},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76400000000000001,"15min":0.78800000000000003},"min":{"1min":0.60899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0289999999999999,"5min":1.1419999999999999,"15min":4.3559999999999999},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.51800000000000002},"max":{"1min":1.056,"5min":1.3939999999999999,"15min":1.9119999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78600000000000003,"15min":0.80800000000000005},"min":{"1min":0.56299999999999994,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.048,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77800000000000002,"15min":0.81000000000000005},"min":{"1min":0.57499999999999996,"5min":0.52600000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":1.194,"15min":4.3899999999999997},"last":0.752},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.093,"5min":1.468,"15min":1.468},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.62,"5min":0.58699999999999997,"15min":0.56200000000000006},"max":{"1min":1.127,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.83599999999999997},"min":{"1min":0.60999999999999999,"5min":0.57099999999999995,"15min":0.55700000000000005},"max":{"1min":1.0269999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.73699999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1572864,"data_stored":1048477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.33300000000000002,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.497,"15min":0.51000000000000001},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72199999999999998,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.628}]},{"osd":2,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.435,"5min":0.39200000000000002,"15min":0.35499999999999998},"max":{"1min":0.78800000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.48799999999999999,"15min":0.496},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.63400000000000001,"5min":0.83899999999999997,"15min":0.97199999999999998},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.39000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73599999999999999,"5min":0.83199999999999996,"15min":1.077},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.47199999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83499999999999996,"5min":0.876,"15min":1.0549999999999999},"last":0.5}]},{"osd":4,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.81399999999999995,"15min":0.80200000000000005},"min":{"1min":0.65100000000000002,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":1.0469999999999999,"5min":1.625,"15min":1.625},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.82399999999999995},"min":{"1min":0.66900000000000004,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":0.97099999999999997,"5min":1.1419999999999999,"15min":1.167},"last":0.73399999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.752,"15min":0.75900000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98999999999999999,"5min":1.137,"15min":1.137},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.80600000000000005,"15min":0.80300000000000005},"min":{"1min":0.60399999999999998,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":0.90200000000000002,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.752}]},{"osd":6,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.53400000000000003},"max":{"1min":1.0129999999999999,"5min":1.0640000000000001,"15min":1.228},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.85499999999999998,"15min":0.84399999999999997},"min":{"1min":0.66400000000000003,"5min":0.52600000000000002,"15min":0.46899999999999997},"max":{"1min":1.0249999999999999,"5min":1.601,"15min":1.601},"last":0.80100000000000005}]},{"osd":7,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78300000000000003,"15min":0.79600000000000004},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.97299999999999998,"5min":1.0529999999999999,"15min":1.1890000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.84699999999999998,"15min":0.84199999999999997},"min":{"1min":0.64000000000000001,"5min":0.57499999999999996,"15min":0.49299999999999999},"max":{"1min":1.046,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.84199999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36172,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335668,"statfs":{"total":96636764160,"available":96599724032,"internally_reserved":0,"allocated":974848,"data_stored":457038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.34699999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.623,"5min":0.69599999999999995,"15min":0.86599999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.39300000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":1.3140000000000001},"last":0.47999999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47299999999999998,"15min":0.46999999999999997},"min":{"1min":0.376,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.63600000000000001,"5min":0.751,"15min":0.94499999999999995},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.45800000000000002,"5min":0.36099999999999999,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":1.032,"15min":1.032},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.504,"5min":0.375,"15min":0.375},"max":{"1min":0.73799999999999999,"5min":0.80000000000000004,"15min":1.127},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.48999999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70899999999999996,"5min":0.82399999999999995,"15min":1.1479999999999999},"last":0.51800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.77300000000000002,"15min":0.78600000000000003},"min":{"1min":0.68700000000000006,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":1.0760000000000001,"5min":1.1799999999999999,"15min":1.554},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.79500000000000004,"15min":0.81000000000000005},"min":{"1min":0.60799999999999998,"5min":0.55000000000000004,"15min":0.51600000000000001},"max":{"1min":1.0309999999999999,"5min":1.236,"15min":1.577},"last":0.86099999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.81000000000000005},"min":{"1min":0.64700000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0680000000000001,"5min":1.25,"15min":1.5209999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79600000000000004,"15min":0.80800000000000005},"min":{"1min":0.58799999999999997,"5min":0.53200000000000003,"15min":0.51800000000000002},"max":{"1min":1.0049999999999999,"5min":1.2250000000000001,"15min":1.3300000000000001},"last":0.77700000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.80400000000000005,"15min":0.82799999999999996},"min":{"1min":0.65200000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1399999999999999,"5min":1.1539999999999999,"15min":1.758},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.874,"5min":0.83699999999999997,"15min":0.84199999999999997},"min":{"1min":0.65900000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1080000000000001,"5min":1.3560000000000001,"15min":1.4750000000000001},"last":0.90900000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.86399999999999999,"15min":0.86899999999999999},"min":{"1min":0.73599999999999999,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.137,"5min":1.2709999999999999,"15min":1.4490000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.69999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.107,"5min":1.3049999999999999,"15min":1.4390000000000001},"last":0.96799999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575030,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36172,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335668,"statfs":{"total":96636764160,"available":96599724032,"internally_reserved":0,"allocated":974848,"data_stored":457038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.46600000000000003},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":0.79500000000000004,"15min":4.3799999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52600000000000002,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":4.6379999999999999},"last":0.621}]},{"osd":1,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.40200000000000002,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.69799999999999995,"5min":0.98699999999999999,"15min":4.1840000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.314},"max":{"1min":0.80800000000000005,"5min":0.96099999999999997,"15min":1.048},"last":0.67600000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.47799999999999998,"5min":0.373,"15min":0.33700000000000002},"max":{"1min":0.77200000000000002,"5min":0.82499999999999996,"15min":1.369},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.63800000000000001,"15min":0.64700000000000002},"min":{"1min":0.55700000000000005,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.79700000000000004,"5min":0.94799999999999995,"15min":4.5970000000000004},"last":0.748}]},{"osd":4,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76800000000000002,"15min":0.78100000000000003},"min":{"1min":0.64400000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.0940000000000001,"5min":1.288,"15min":4.3319999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77600000000000002,"15min":0.77500000000000002},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.39400000000000002},"max":{"1min":0.92400000000000004,"5min":1.46,"15min":1.46},"last":1.0900000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78200000000000003,"15min":0.79500000000000004},"min":{"1min":0.628,"5min":0.48899999999999999,"15min":0.435},"max":{"1min":0.97599999999999998,"5min":1.3580000000000001,"15min":1.3740000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79800000000000004,"15min":0.81000000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.44500000000000001},"max":{"1min":1.133,"5min":1.397,"15min":4.4180000000000001},"last":0.876}]},{"osd":6,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81399999999999995,"15min":0.81200000000000006},"min":{"1min":0.68899999999999995,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.0009999999999999,"5min":1.2549999999999999,"15min":1.4730000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81699999999999995,"15min":0.81899999999999995},"min":{"1min":0.65000000000000002,"5min":0.52400000000000002,"15min":0.503},"max":{"1min":1.0409999999999999,"5min":1.23,"15min":1.417},"last":1.0349999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.83999999999999997,"15min":0.84799999999999998},"min":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.54300000000000004},"max":{"1min":1.135,"5min":1.5,"15min":4.4550000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.64400000000000002,"5min":0.59899999999999998,"15min":0.56200000000000006},"max":{"1min":1.056,"5min":1.345,"15min":1.476},"last":1.0489999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54224,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317616,"statfs":{"total":96636764160,"available":96581238784,"internally_reserved":0,"allocated":978944,"data_stored":458546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.68700000000000006},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.88200000000000001,"5min":0.93200000000000005,"15min":1.417},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.92100000000000004,"5min":1.165,"15min":1.165},"last":0.434}]},{"osd":1,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.93500000000000005,"5min":0.93700000000000006,"15min":1.139},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.333},"last":0.55700000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.01,"15min":1.863},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.75,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93700000000000006,"5min":0.96299999999999997,"15min":1.387},"last":0.50600000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.5,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.88700000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.495},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.51600000000000001,"5min":0.48699999999999999,"15min":0.46600000000000003},"max":{"1min":0.80500000000000005,"5min":1.1539999999999999,"15min":1.905},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.46700000000000003,"5min":0.39300000000000002,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.48099999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.78500000000000003,"5min":1.294,"15min":1.294},"last":0.629}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.48099999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.75900000000000001,"5min":1.0129999999999999,"15min":1.9339999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.66200000000000003},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.433},"max":{"1min":0.85299999999999998,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.48099999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85999999999999999,"5min":1.345,"15min":1.345},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.59099999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89700000000000002,"5min":1.214,"15min":1.97},"last":0.755}]}]},{"osd":5,"up_from":13,"seq":55834575029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":929792,"data_stored":437640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69599999999999995,"15min":0.68200000000000005},"min":{"1min":0.432,"5min":0.41599999999999998,"15min":0.313},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.746,"15min":0.71899999999999997},"min":{"1min":0.45900000000000002,"5min":0.45500000000000002,"15min":0.33500000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.339},"last":0.435}]},{"osd":1,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.55400000000000005,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.0169999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72199999999999998,"15min":0.70499999999999996},"min":{"1min":0.47999999999999998,"5min":0.46700000000000003,"15min":0.39200000000000002},"max":{"1min":0.95199999999999996,"5min":1.022,"15min":1.6040000000000001},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73099999999999998,"15min":0.72799999999999998},"min":{"1min":0.57099999999999995,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.476},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.59299999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93700000000000006,"5min":1.095,"15min":1.502},"last":0.46200000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.63200000000000001,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":0.95799999999999996,"5min":1.113,"15min":1.1619999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.754},"min":{"1min":0.56499999999999995,"5min":0.52300000000000002,"15min":0.46500000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.744},"last":0.64900000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.80300000000000005,"5min":0.94599999999999995,"15min":0.95899999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.214},"last":0.495}]},{"osd":6,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64800000000000002,"15min":0.65400000000000003},"min":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.36799999999999999},"max":{"1min":0.85999999999999999,"5min":0.91300000000000003,"15min":1.151},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.52600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":0.97099999999999997,"15min":1.5489999999999999},"last":0.70599999999999996}]},{"osd":7,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.70499999999999996,"15min":0.71099999999999997},"min":{"1min":0.51500000000000001,"5min":0.41899999999999998,"15min":0.40200000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.325},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.53800000000000003,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":0.96999999999999997,"5min":0.98199999999999998,"15min":1.1990000000000001},"last":0.79700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:09:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3c is not active+clean 2024-03-29T10:09:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-29T10:09:32.067771+0000', 'last_change': '2024-03-29T10:09:32.067771+0000', 'last_active': '2024-03-29T10:09:32.067771+0000', 'last_peered': '2024-03-29T10:09:32.067771+0000', 'last_clean': '2024-03-29T10:09:32.067771+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:32.067771+0000', 'last_undegraded': '2024-03-29T10:09:32.067771+0000', 'last_fullsized': '2024-03-29T10:09:32.067771+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-29T10:09:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.34 is not active+clean 2024-03-29T10:09:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-29T10:09:32.067771+0000', 'last_change': '2024-03-29T10:09:32.067771+0000', 'last_active': '2024-03-29T10:09:32.067771+0000', 'last_peered': '2024-03-29T10:09:32.067771+0000', 'last_clean': '2024-03-29T10:09:32.067771+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:32.067771+0000', 'last_undegraded': '2024-03-29T10:09:32.067771+0000', 'last_fullsized': '2024-03-29T10:09:32.067771+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-29T10:09:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.26 is not active+clean 2024-03-29T10:09:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-29T10:09:32.067771+0000', 'last_change': '2024-03-29T10:09:32.067771+0000', 'last_active': '2024-03-29T10:09:32.067771+0000', 'last_peered': '2024-03-29T10:09:32.067771+0000', 'last_clean': '2024-03-29T10:09:32.067771+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:32.067771+0000', 'last_undegraded': '2024-03-29T10:09:32.067771+0000', 'last_fullsized': '2024-03-29T10:09:32.067771+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-29T10:09:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-03-29T10:09:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-03-29T10:09:31.072330+0000', 'last_change': '2024-03-29T10:09:31.072330+0000', 'last_active': '2024-03-29T10:09:31.061546+0000', 'last_peered': '2024-03-29T10:09:31.061546+0000', 'last_clean': '2024-03-29T10:09:31.061546+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:31.072330+0000', 'last_undegraded': '2024-03-29T10:09:31.072330+0000', 'last_fullsized': '2024-03-29T10:09:31.072330+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-29T10:09:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-03-29T10:09:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-03-29T10:09:31.073144+0000', 'last_change': '2024-03-29T10:09:31.073144+0000', 'last_active': '2024-03-29T10:09:31.061546+0000', 'last_peered': '2024-03-29T10:09:31.061546+0000', 'last_clean': '2024-03-29T10:09:31.061546+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:31.073144+0000', 'last_undegraded': '2024-03-29T10:09:31.073144+0000', 'last_fullsized': '2024-03-29T10:09:31.073144+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-29T10:09:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2024-03-29T10:09:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-29T10:09:32.067771+0000', 'last_change': '2024-03-29T10:09:32.067771+0000', 'last_active': '2024-03-29T10:09:32.067771+0000', 'last_peered': '2024-03-29T10:09:32.067771+0000', 'last_clean': '2024-03-29T10:09:32.067771+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:32.067771+0000', 'last_undegraded': '2024-03-29T10:09:32.067771+0000', 'last_fullsized': '2024-03-29T10:09:32.067771+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-29T10:09:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-03-29T10:09:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-03-29T10:09:31.071636+0000', 'last_change': '2024-03-29T10:09:31.071636+0000', 'last_active': '2024-03-29T10:09:31.061546+0000', 'last_peered': '2024-03-29T10:09:31.061546+0000', 'last_clean': '2024-03-29T10:09:31.061546+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:31.071636+0000', 'last_undegraded': '2024-03-29T10:09:31.071636+0000', 'last_fullsized': '2024-03-29T10:09:31.071636+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-29T10:09:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2024-03-29T10:09:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-29T10:09:32.067771+0000', 'last_change': '2024-03-29T10:09:32.067771+0000', 'last_active': '2024-03-29T10:09:32.067771+0000', 'last_peered': '2024-03-29T10:09:32.067771+0000', 'last_clean': '2024-03-29T10:09:32.067771+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:32.067771+0000', 'last_undegraded': '2024-03-29T10:09:32.067771+0000', 'last_fullsized': '2024-03-29T10:09:32.067771+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-29T10:09:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-03-29T10:09:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-03-29T10:09:31.072860+0000', 'last_change': '2024-03-29T10:09:31.072860+0000', 'last_active': '2024-03-29T10:09:31.061546+0000', 'last_peered': '2024-03-29T10:09:31.061546+0000', 'last_clean': '2024-03-29T10:09:31.061546+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:31.072860+0000', 'last_undegraded': '2024-03-29T10:09:31.072860+0000', 'last_fullsized': '2024-03-29T10:09:31.072860+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-30T20:27:35.691958+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-29T10:09:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2024-03-29T10:09:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-29T10:09:32.067771+0000', 'last_change': '2024-03-29T10:09:32.067771+0000', 'last_active': '2024-03-29T10:09:32.067771+0000', 'last_peered': '2024-03-29T10:09:32.067771+0000', 'last_clean': '2024-03-29T10:09:32.067771+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:32.067771+0000', 'last_undegraded': '2024-03-29T10:09:32.067771+0000', 'last_fullsized': '2024-03-29T10:09:32.067771+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-29T10:09:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-03-29T10:09:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-03-29T10:09:31.072820+0000', 'last_change': '2024-03-29T10:09:31.072820+0000', 'last_active': '2024-03-29T10:09:31.061546+0000', 'last_peered': '2024-03-29T10:09:31.061546+0000', 'last_clean': '2024-03-29T10:09:31.061546+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:31.072820+0000', 'last_undegraded': '2024-03-29T10:09:31.072820+0000', 'last_fullsized': '2024-03-29T10:09:31.072820+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-30T19:51:58.170982+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-29T10:09:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2024-03-29T10:09:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-29T10:09:32.067771+0000', 'last_change': '2024-03-29T10:09:32.067771+0000', 'last_active': '2024-03-29T10:09:32.067771+0000', 'last_peered': '2024-03-29T10:09:32.067771+0000', 'last_clean': '2024-03-29T10:09:32.067771+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:32.067771+0000', 'last_undegraded': '2024-03-29T10:09:32.067771+0000', 'last_fullsized': '2024-03-29T10:09:32.067771+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-29T10:09:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2024-03-29T10:09:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-29T10:09:32.067771+0000', 'last_change': '2024-03-29T10:09:32.067771+0000', 'last_active': '2024-03-29T10:09:32.067771+0000', 'last_peered': '2024-03-29T10:09:32.067771+0000', 'last_clean': '2024-03-29T10:09:32.067771+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:32.067771+0000', 'last_undegraded': '2024-03-29T10:09:32.067771+0000', 'last_fullsized': '2024-03-29T10:09:32.067771+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:32.067771+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-29T10:09:38.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-03-29T10:09:38.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-03-29T10:09:31.071841+0000', 'last_change': '2024-03-29T10:09:31.071841+0000', 'last_active': '2024-03-29T10:09:31.061546+0000', 'last_peered': '2024-03-29T10:09:31.061546+0000', 'last_clean': '2024-03-29T10:09:31.061546+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-29T10:09:31.071841+0000', 'last_undegraded': '2024-03-29T10:09:31.071841+0000', 'last_fullsized': '2024-03-29T10:09:31.071841+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'last_clean_scrub_stamp': '2024-03-29T10:09:31.061546+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-30T10:39:56.946534+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-29T10:09:38.053 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-29T10:09:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.275+0000 7f8f13200700 1 Processor -- start 2024-03-29T10:09:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.275+0000 7f8f13200700 1 -- start start 2024-03-29T10:09:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f13200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8f0c0afbe0 0x7f8f0c06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f13200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0cebd0 0x7f8f0c0cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f13200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0c06f850 con 0x7f8f0c06a790 2024-03-29T10:09:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f13200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0c06fad0 con 0x7f8f0c0afbe0 2024-03-29T10:09:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f13200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c0af970 con 0x7f8f0c0cebd0 2024-03-29T10:09:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f0be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8f0c06a790 0x7f8f0c06ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33172/0 (socket says 172.21.15.12:33172) 2024-03-29T10:09:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f0b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8f0c0afbe0 0x7f8f0c06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f0be00700 1 -- 172.21.15.12:0/3384457711 learned_addr learned my addr 172.21.15.12:0/3384457711 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f11200700 1 --2- 172.21.15.12:0/3384457711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0cebd0 0x7f8f0c0cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f0aa00700 1 -- 172.21.15.12:0/3384457711 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139011158 0 0) 0x7f8f0c06f850 con 0x7f8f0c06a790 2024-03-29T10:09:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.277+0000 7f8f0aa00700 1 -- 172.21.15.12:0/3384457711 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef00035b0 con 0x7f8f0c06a790 2024-03-29T10:09:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.278+0000 7f8f0b400700 1 -- 172.21.15.12:0/3384457711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0cebd0 msgr2=0x7f8f0c0cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.278+0000 7f8f0b400700 1 --2- 172.21.15.12:0/3384457711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0cebd0 0x7f8f0c0cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.278+0000 7f8f0b400700 1 -- 172.21.15.12:0/3384457711 >> v1:172.21.15.12:6789/0 conn(0x7f8f0c06a790 legacy=0x7f8f0c06ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.278+0000 7f8f0b400700 1 -- 172.21.15.12:0/3384457711 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c0af690 con 0x7f8f0c0afbe0 2024-03-29T10:09:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.278+0000 7f8f0b400700 1 --2- 172.21.15.12:0/3384457711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c0afbe0 0x7f8f0c06dce0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8009840 tx=0x7f8ef8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86469ccaf2e4d2f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.278+0000 7f8f0aa00700 1 -- 172.21.15.12:0/3384457711 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8011830 con 0x7f8f0c0afbe0 2024-03-29T10:09:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.278+0000 7f8f0aa00700 1 -- 172.21.15.12:0/3384457711 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef8011990 con 0x7f8f0c0afbe0 2024-03-29T10:09:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.278+0000 7f8f0aa00700 1 -- 172.21.15.12:0/3384457711 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8011d20 con 0x7f8f0c0afbe0 2024-03-29T10:09:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.279+0000 7f8f13200700 1 -- 172.21.15.12:0/3384457711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c0afbe0 msgr2=0x7f8f0c06dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.279+0000 7f8f13200700 1 --2- 172.21.15.12:0/3384457711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c0afbe0 0x7f8f0c06dce0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8009840 tx=0x7f8ef8009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.279+0000 7f8f13200700 1 -- 172.21.15.12:0/3384457711 shutdown_connections 2024-03-29T10:09:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.279+0000 7f8f13200700 1 --2- 172.21.15.12:0/3384457711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0cebd0 0x7f8f0c0cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.279+0000 7f8f13200700 1 --2- 172.21.15.12:0/3384457711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c0afbe0 0x7f8f0c06dce0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.279+0000 7f8f13200700 1 -- 172.21.15.12:0/3384457711 >> 172.21.15.12:0/3384457711 conn(0x7f8f0c0b5ab0 msgr2=0x7f8f0c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.279+0000 7f8f13200700 1 -- 172.21.15.12:0/3384457711 shutdown_connections 2024-03-29T10:09:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.279+0000 7f8f13200700 1 -- 172.21.15.12:0/3384457711 wait complete. 2024-03-29T10:09:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.280+0000 7f8f13200700 1 Processor -- start 2024-03-29T10:09:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.281+0000 7f8f13200700 1 -- start start 2024-03-29T10:09:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.281+0000 7f8f13200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c06a790 0x7f8f0c154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.281+0000 7f8f13200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0afbe0 0x7f8f0c09c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.281+0000 7f8f13200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f0c0cebd0 0x7f8f0c0cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f13200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c155390 con 0x7f8f0c06a790 2024-03-29T10:09:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f13200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c09b220 con 0x7f8f0c0cebd0 2024-03-29T10:09:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f13200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c09b500 con 0x7f8f0c0afbe0 2024-03-29T10:09:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c06a790 0x7f8f0c154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c06a790 0x7f8f0c154e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58272/0 (socket says 172.21.15.12:58272) 2024-03-29T10:09:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0be00700 1 -- 172.21.15.12:0/280841642 learned_addr learned my addr 172.21.15.12:0/280841642 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0b400700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0afbe0 0x7f8f0c09c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0be00700 1 -- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0afbe0 msgr2=0x7f8f0c09c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0be00700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0afbe0 0x7f8f0c09c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0be00700 1 -- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f0c0cebd0 msgr2=0x7f8f0c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f11200700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f0c0cebd0 0x7f8f0c0cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0be00700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f0c0cebd0 0x7f8f0c0cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0be00700 1 -- 172.21.15.12:0/280841642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef80094a0 con 0x7f8f0c06a790 2024-03-29T10:09:38.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0b400700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0afbe0 0x7f8f0c09c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:09:38.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f11200700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f0c0cebd0 0x7f8f0c0cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:09:38.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.282+0000 7f8f0be00700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c06a790 0x7f8f0c154e50 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8efc001eb0 tx=0x7f8efc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:38.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.283+0000 7f8f03e00700 1 -- 172.21.15.12:0/280841642 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc005d70 con 0x7f8f0c06a790 2024-03-29T10:09:38.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.283+0000 7f8f03e00700 1 -- 172.21.15.12:0/280841642 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc0070d0 con 0x7f8f0c06a790 2024-03-29T10:09:38.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.283+0000 7f8f03e00700 1 -- 172.21.15.12:0/280841642 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc007460 con 0x7f8f0c06a790 2024-03-29T10:09:38.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.283+0000 7f8f13200700 1 -- 172.21.15.12:0/280841642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0c09b7e0 con 0x7f8f0c06a790 2024-03-29T10:09:38.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.283+0000 7f8f13200700 1 -- 172.21.15.12:0/280841642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0c09bda0 con 0x7f8f0c06a790 2024-03-29T10:09:38.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.284+0000 7f8f13200700 1 -- 172.21.15.12:0/280841642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0c067720 con 0x7f8f0c06a790 2024-03-29T10:09:38.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.285+0000 7f8f03e00700 1 -- 172.21.15.12:0/280841642 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8efc0075c0 con 0x7f8f0c06a790 2024-03-29T10:09:38.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.286+0000 7f8f03e00700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8eec06adb0 0x7f8eec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.286+0000 7f8f03e00700 1 -- 172.21.15.12:0/280841642 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8efc088a70 con 0x7f8f0c06a790 2024-03-29T10:09:38.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.286+0000 7f8f0b400700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8eec06adb0 0x7f8eec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.287+0000 7f8f0b400700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8eec06adb0 0x7f8eec06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c0cb950 tx=0x7f8ef8008030 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:38.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.290+0000 7f8f03e00700 1 -- 172.21.15.12:0/280841642 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8efc053ab0 con 0x7f8f0c06a790 2024-03-29T10:09:38.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.599+0000 7f8f13200700 1 -- 172.21.15.12:0/280841642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f8f0c0b21a0 con 0x7f8f0c06a790 2024-03-29T10:09:38.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.600+0000 7f8f03e00700 1 -- 172.21.15.12:0/280841642 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+3369 (secure 0 0 0) 0x7f8efc017080 con 0x7f8f0c06a790 2024-03-29T10:09:38.601 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:38.601 INFO:teuthology.orchestra.run.smithi012.stdout:{"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":935,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":80,"num_osds":8,"num_up_osds":8,"osd_up_since":1711706055,"num_in_osds":8,"osd_in_since":1711706046,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":83},{"state_name":"unknown","count":8},{"state_name":"creating+peering","count":6}],"num_pgs":97,"num_pools":3,"num_objects":21,"data_bytes":592660,"bytes_used":313421824,"bytes_avail":772780691456,"bytes_total":773094113280,"unknown_pgs_ratio":0.082474224269390106,"inactive_pgs_ratio":0.061855670064687729,"write_bytes_sec":2307,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":31,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"b","status":"up:active","gid":6241}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-03-29T10:09:12.833256+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-29T10:09:38.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.602+0000 7f8f13200700 1 -- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8eec06adb0 msgr2=0x7f8eec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.602+0000 7f8f13200700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8eec06adb0 0x7f8eec06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c0cb950 tx=0x7f8ef8008030 comp rx=0 tx=0).stop 2024-03-29T10:09:38.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.602+0000 7f8f13200700 1 -- 172.21.15.12:0/280841642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c06a790 msgr2=0x7f8f0c154e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.602+0000 7f8f13200700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c06a790 0x7f8f0c154e50 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8efc001eb0 tx=0x7f8efc00c860 comp rx=0 tx=0).stop 2024-03-29T10:09:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.602+0000 7f8f13200700 1 -- 172.21.15.12:0/280841642 shutdown_connections 2024-03-29T10:09:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.602+0000 7f8f13200700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8eec06adb0 0x7f8eec06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.602+0000 7f8f13200700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f0c0cebd0 0x7f8f0c0cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.602+0000 7f8f13200700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f0c06a790 0x7f8f0c154e50 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.602+0000 7f8f13200700 1 --2- 172.21.15.12:0/280841642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f0c0afbe0 0x7f8f0c09c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.602+0000 7f8f13200700 1 -- 172.21.15.12:0/280841642 >> 172.21.15.12:0/280841642 conn(0x7f8f0c0b5ab0 msgr2=0x7f8f0c15b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:38.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.603+0000 7f8f13200700 1 -- 172.21.15.12:0/280841642 shutdown_connections 2024-03-29T10:09:38.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.603+0000 7f8f13200700 1 -- 172.21.15.12:0/280841642 wait complete. 2024-03-29T10:09:38.621 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-29T10:09:38.622 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:09:38.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.820+0000 7fbdbae00700 1 Processor -- start 2024-03-29T10:09:38.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.820+0000 7fbdbae00700 1 -- start start 2024-03-29T10:09:38.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.822+0000 7fbdbae00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdb415d670 0x7fbdb40ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.822+0000 7fbdbae00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbdb406dee0 0x7fbdb406e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.822+0000 7fbdbae00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb40b3890 con 0x7fbdb415da50 2024-03-29T10:09:38.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.822+0000 7fbdbae00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb40acac0 con 0x7fbdb406dee0 2024-03-29T10:09:38.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.822+0000 7fbdbae00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4159f50 con 0x7fbdb415d670 2024-03-29T10:09:38.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.822+0000 7fbdb8e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbdb406dee0 0x7fbdb406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.822+0000 7fbdb3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdb415d670 0x7fbdb40ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.822+0000 7fbdb8e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbdb406dee0 0x7fbdb406e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:35418/0 (socket says 172.21.15.12:35418) 2024-03-29T10:09:38.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.822+0000 7fbdb8e00700 1 -- 172.21.15.12:0/3043984447 learned_addr learned my addr 172.21.15.12:0/3043984447 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:38.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.823+0000 7fbdb8e00700 1 -- 172.21.15.12:0/3043984447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdb415d670 msgr2=0x7fbdb40ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:38.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.823+0000 7fbdb8e00700 1 --2- 172.21.15.12:0/3043984447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdb415d670 0x7fbdb40ac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.823+0000 7fbdb8e00700 1 -- 172.21.15.12:0/3043984447 >> v1:172.21.15.159:6789/0 conn(0x7fbdb415da50 legacy=0x7fbdb40b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.823+0000 7fbdb8e00700 1 -- 172.21.15.12:0/3043984447 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb40b35b0 con 0x7fbdb406dee0 2024-03-29T10:09:38.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.823+0000 7fbdb8e00700 1 --2- 172.21.15.12:0/3043984447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdb406dee0 0x7fbdb406e370 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00d150 tx=0x7fbdac00d460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a598ea171e2490d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:38.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.823+0000 7fbdb2a00700 1 -- 172.21.15.12:0/3043984447 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac010070 con 0x7fbdb406dee0 2024-03-29T10:09:38.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.823+0000 7fbdb2a00700 1 -- 172.21.15.12:0/3043984447 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdac00db00 con 0x7fbdb406dee0 2024-03-29T10:09:38.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.823+0000 7fbdb2a00700 1 -- 172.21.15.12:0/3043984447 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac016400 con 0x7fbdb406dee0 2024-03-29T10:09:38.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.824+0000 7fbdbae00700 1 -- 172.21.15.12:0/3043984447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdb406dee0 msgr2=0x7fbdb406e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.824+0000 7fbdbae00700 1 --2- 172.21.15.12:0/3043984447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdb406dee0 0x7fbdb406e370 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00d150 tx=0x7fbdac00d460 comp rx=0 tx=0).stop 2024-03-29T10:09:38.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.824+0000 7fbdbae00700 1 -- 172.21.15.12:0/3043984447 shutdown_connections 2024-03-29T10:09:38.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.824+0000 7fbdbae00700 1 --2- 172.21.15.12:0/3043984447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdb406dee0 0x7fbdb406e370 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.824+0000 7fbdbae00700 1 --2- 172.21.15.12:0/3043984447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdb415d670 0x7fbdb40ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.824+0000 7fbdbae00700 1 -- 172.21.15.12:0/3043984447 >> 172.21.15.12:0/3043984447 conn(0x7fbdb40cf190 msgr2=0x7fbdb40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:38.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.824+0000 7fbdbae00700 1 -- 172.21.15.12:0/3043984447 shutdown_connections 2024-03-29T10:09:38.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.824+0000 7fbdbae00700 1 -- 172.21.15.12:0/3043984447 wait complete. 2024-03-29T10:09:38.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.826+0000 7fbdbae00700 1 Processor -- start 2024-03-29T10:09:38.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.826+0000 7fbdbae00700 1 -- start start 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdbae00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdb406dee0 0x7fbdb41f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdbae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb415d670 0x7fbdb40b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdbae00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdb415da50 0x7fbdb40b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdbae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb41f5da0 con 0x7fbdb415d670 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdbae00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb41f5fb0 con 0x7fbdb406dee0 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdbae00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb40cc520 con 0x7fbdb415da50 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdb3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb415d670 0x7fbdb40b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdb3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb415d670 0x7fbdb40b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58284/0 (socket says 172.21.15.12:58284) 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdb3400700 1 -- 172.21.15.12:0/2740300431 learned_addr learned my addr 172.21.15.12:0/2740300431 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdb3e00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdb406dee0 0x7fbdb41f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.827+0000 7fbdb8e00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdb415da50 0x7fbdb40b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdb3400700 1 -- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdb415da50 msgr2=0x7fbdb40b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdb3400700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdb415da50 0x7fbdb40b2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdb3400700 1 -- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdb406dee0 msgr2=0x7fbdb41f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdb3400700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdb406dee0 0x7fbdb41f5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdb3400700 1 -- 172.21.15.12:0/2740300431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac00bd40 con 0x7fbdb415d670 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdb3e00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdb406dee0 0x7fbdb41f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdb3400700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb415d670 0x7fbdb40b1ed0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fbda000c450 tx=0x7fbda000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdabe00700 1 -- 172.21.15.12:0/2740300431 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda00052d0 con 0x7fbdb415d670 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdabe00700 1 -- 172.21.15.12:0/2740300431 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda0005430 con 0x7fbdb415d670 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdabe00700 1 -- 172.21.15.12:0/2740300431 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda00038c0 con 0x7fbdb415d670 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdbae00700 1 -- 172.21.15.12:0/2740300431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb41f6200 con 0x7fbdb415d670 2024-03-29T10:09:38.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.828+0000 7fbdbae00700 1 -- 172.21.15.12:0/2740300431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb41f67c0 con 0x7fbdb415d670 2024-03-29T10:09:38.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.829+0000 7fbdbae00700 1 -- 172.21.15.12:0/2740300431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb41f69b0 con 0x7fbdb415d670 2024-03-29T10:09:38.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.830+0000 7fbdabe00700 1 -- 172.21.15.12:0/2740300431 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbda0003a20 con 0x7fbdb415d670 2024-03-29T10:09:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.832+0000 7fbdabe00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbd9406adb0 0x7fbd9406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.832+0000 7fbdabe00700 1 -- 172.21.15.12:0/2740300431 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbda00888e0 con 0x7fbdb415d670 2024-03-29T10:09:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.832+0000 7fbdb3e00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbd9406adb0 0x7fbd9406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.833+0000 7fbdb3e00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbd9406adb0 0x7fbd9406d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009c30 tx=0x7fbda4008040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:38.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:38.836+0000 7fbdabe00700 1 -- 172.21.15.12:0/2740300431 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbda0053910 con 0x7fbdb415d670 2024-03-29T10:09:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.046+0000 7fbdbae00700 1 -- 172.21.15.12:0/2740300431 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbdb406d9c0 con 0x7fbd9406adb0 2024-03-29T10:09:39.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.056+0000 7fbdabe00700 1 -- 172.21.15.12:0/2740300431 <== mgr.6013 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289074 (secure 0 0 0) 0x7fbda4004030 con 0x7fbd9406adb0 2024-03-29T10:09:39.059 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:39.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 -- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbd9406adb0 msgr2=0x7fbd9406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:39.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbd9406adb0 0x7fbd9406d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009c30 tx=0x7fbda4008040 comp rx=0 tx=0).stop 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 -- 172.21.15.12:0/2740300431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb415d670 msgr2=0x7fbdb40b1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb415d670 0x7fbdb40b1ed0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fbda000c450 tx=0x7fbda000c810 comp rx=0 tx=0).stop 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 -- 172.21.15.12:0/2740300431 shutdown_connections 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbd9406adb0 0x7fbd9406d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdb415da50 0x7fbdb40b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdb406dee0 0x7fbdb41f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 --2- 172.21.15.12:0/2740300431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb415d670 0x7fbdb40b1ed0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 -- 172.21.15.12:0/2740300431 >> 172.21.15.12:0/2740300431 conn(0x7fbdb40cf190 msgr2=0x7fbdb406e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 -- 172.21.15.12:0/2740300431 shutdown_connections 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:39.060+0000 7fbdbae00700 1 -- 172.21.15.12:0/2740300431 wait complete. 2024-03-29T10:09:39.062 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:09:39.078 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-03-29T10:09:38.839062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":306120,"kb_used_data":8772,"kb_used_omap":12,"kb_used_meta":297203,"kb_avail":754668600,"statfs":{"total":773094113280,"available":772780646400,"internally_reserved":0,"allocated":8982528,"data_stored":4840297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":304336441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.766442"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045584+0000","last_change":"2024-03-29T10:09:32.623236+0000","last_active":"2024-03-29T10:09:34.045584+0000","last_peered":"2024-03-29T10:09:34.045584+0000","last_clean":"2024-03-29T10:09:34.045584+0000","last_became_active":"2024-03-29T10:09:32.622714+0000","last_became_peered":"2024-03-29T10:09:32.622714+0000","last_unstale":"2024-03-29T10:09:34.045584+0000","last_undegraded":"2024-03-29T10:09:34.045584+0000","last_fullsized":"2024-03-29T10:09:34.045584+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:50.745197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607572+0000","last_change":"2024-03-29T10:09:32.620014+0000","last_active":"2024-03-29T10:09:33.607572+0000","last_peered":"2024-03-29T10:09:33.607572+0000","last_clean":"2024-03-29T10:09:33.607572+0000","last_became_active":"2024-03-29T10:09:32.619747+0000","last_became_peered":"2024-03-29T10:09:32.619747+0000","last_unstale":"2024-03-29T10:09:33.607572+0000","last_undegraded":"2024-03-29T10:09:33.607572+0000","last_fullsized":"2024-03-29T10:09:33.607572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:58:19.387293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045306+0000","last_change":"2024-03-29T10:09:32.622331+0000","last_active":"2024-03-29T10:09:34.045306+0000","last_peered":"2024-03-29T10:09:34.045306+0000","last_clean":"2024-03-29T10:09:34.045306+0000","last_became_active":"2024-03-29T10:09:32.622038+0000","last_became_peered":"2024-03-29T10:09:32.622038+0000","last_unstale":"2024-03-29T10:09:34.045306+0000","last_undegraded":"2024-03-29T10:09:34.045306+0000","last_fullsized":"2024-03-29T10:09:34.045306+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:54.413063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613492+0000","last_change":"2024-03-29T10:09:32.622514+0000","last_active":"2024-03-29T10:09:33.613492+0000","last_peered":"2024-03-29T10:09:33.613492+0000","last_clean":"2024-03-29T10:09:33.613492+0000","last_became_active":"2024-03-29T10:09:32.621840+0000","last_became_peered":"2024-03-29T10:09:32.621840+0000","last_unstale":"2024-03-29T10:09:33.613492+0000","last_undegraded":"2024-03-29T10:09:33.613492+0000","last_fullsized":"2024-03-29T10:09:33.613492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:46.872891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045348+0000","last_change":"2024-03-29T10:09:32.617524+0000","last_active":"2024-03-29T10:09:34.045348+0000","last_peered":"2024-03-29T10:09:34.045348+0000","last_clean":"2024-03-29T10:09:34.045348+0000","last_became_active":"2024-03-29T10:09:32.617270+0000","last_became_peered":"2024-03-29T10:09:32.617270+0000","last_unstale":"2024-03-29T10:09:34.045348+0000","last_undegraded":"2024-03-29T10:09:34.045348+0000","last_fullsized":"2024-03-29T10:09:34.045348+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:02.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045118+0000","last_change":"2024-03-29T10:09:32.616253+0000","last_active":"2024-03-29T10:09:34.045118+0000","last_peered":"2024-03-29T10:09:34.045118+0000","last_clean":"2024-03-29T10:09:34.045118+0000","last_became_active":"2024-03-29T10:09:32.616002+0000","last_became_peered":"2024-03-29T10:09:32.616002+0000","last_unstale":"2024-03-29T10:09:34.045118+0000","last_undegraded":"2024-03-29T10:09:34.045118+0000","last_fullsized":"2024-03-29T10:09:34.045118+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:23.297716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044899+0000","last_change":"2024-03-29T10:09:32.612351+0000","last_active":"2024-03-29T10:09:34.044899+0000","last_peered":"2024-03-29T10:09:34.044899+0000","last_clean":"2024-03-29T10:09:34.044899+0000","last_became_active":"2024-03-29T10:09:32.612057+0000","last_became_peered":"2024-03-29T10:09:32.612057+0000","last_unstale":"2024-03-29T10:09:34.044899+0000","last_undegraded":"2024-03-29T10:09:34.044899+0000","last_fullsized":"2024-03-29T10:09:34.044899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:41.040149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.606905+0000","last_change":"2024-03-29T10:09:32.615482+0000","last_active":"2024-03-29T10:09:33.606905+0000","last_peered":"2024-03-29T10:09:33.606905+0000","last_clean":"2024-03-29T10:09:33.606905+0000","last_became_active":"2024-03-29T10:09:32.615198+0000","last_became_peered":"2024-03-29T10:09:32.615198+0000","last_unstale":"2024-03-29T10:09:33.606905+0000","last_undegraded":"2024-03-29T10:09:33.606905+0000","last_fullsized":"2024-03-29T10:09:33.606905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:48:35.803134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045007+0000","last_change":"2024-03-29T10:09:32.611215+0000","last_active":"2024-03-29T10:09:34.045007+0000","last_peered":"2024-03-29T10:09:34.045007+0000","last_clean":"2024-03-29T10:09:34.045007+0000","last_became_active":"2024-03-29T10:09:32.610941+0000","last_became_peered":"2024-03-29T10:09:32.610941+0000","last_unstale":"2024-03-29T10:09:34.045007+0000","last_undegraded":"2024-03-29T10:09:34.045007+0000","last_fullsized":"2024-03-29T10:09:34.045007+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:28.376324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045092+0000","last_change":"2024-03-29T10:09:32.624355+0000","last_active":"2024-03-29T10:09:34.045092+0000","last_peered":"2024-03-29T10:09:34.045092+0000","last_clean":"2024-03-29T10:09:34.045092+0000","last_became_active":"2024-03-29T10:09:32.622547+0000","last_became_peered":"2024-03-29T10:09:32.622547+0000","last_unstale":"2024-03-29T10:09:34.045092+0000","last_undegraded":"2024-03-29T10:09:34.045092+0000","last_fullsized":"2024-03-29T10:09:34.045092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:02.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044868+0000","last_change":"2024-03-29T10:09:32.628082+0000","last_active":"2024-03-29T10:09:34.044868+0000","last_peered":"2024-03-29T10:09:34.044868+0000","last_clean":"2024-03-29T10:09:34.044868+0000","last_became_active":"2024-03-29T10:09:32.627765+0000","last_became_peered":"2024-03-29T10:09:32.627765+0000","last_unstale":"2024-03-29T10:09:34.044868+0000","last_undegraded":"2024-03-29T10:09:34.044868+0000","last_fullsized":"2024-03-29T10:09:34.044868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:50.472261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613633+0000","last_change":"2024-03-29T10:09:32.622558+0000","last_active":"2024-03-29T10:09:33.613633+0000","last_peered":"2024-03-29T10:09:33.613633+0000","last_clean":"2024-03-29T10:09:33.613633+0000","last_became_active":"2024-03-29T10:09:32.622241+0000","last_became_peered":"2024-03-29T10:09:32.622241+0000","last_unstale":"2024-03-29T10:09:33.613633+0000","last_undegraded":"2024-03-29T10:09:33.613633+0000","last_fullsized":"2024-03-29T10:09:33.613633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:59.825105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.612120+0000","last_change":"2024-03-29T10:09:32.619242+0000","last_active":"2024-03-29T10:09:33.612120+0000","last_peered":"2024-03-29T10:09:33.612120+0000","last_clean":"2024-03-29T10:09:33.612120+0000","last_became_active":"2024-03-29T10:09:32.618883+0000","last_became_peered":"2024-03-29T10:09:32.618883+0000","last_unstale":"2024-03-29T10:09:33.612120+0000","last_undegraded":"2024-03-29T10:09:33.612120+0000","last_fullsized":"2024-03-29T10:09:33.612120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:08.395520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044873+0000","last_change":"2024-03-29T10:09:32.617701+0000","last_active":"2024-03-29T10:09:34.044873+0000","last_peered":"2024-03-29T10:09:34.044873+0000","last_clean":"2024-03-29T10:09:34.044873+0000","last_became_active":"2024-03-29T10:09:32.616607+0000","last_became_peered":"2024-03-29T10:09:32.616607+0000","last_unstale":"2024-03-29T10:09:34.044873+0000","last_undegraded":"2024-03-29T10:09:34.044873+0000","last_fullsized":"2024-03-29T10:09:34.044873+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:39.250301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045466+0000","last_change":"2024-03-29T10:09:32.618041+0000","last_active":"2024-03-29T10:09:34.045466+0000","last_peered":"2024-03-29T10:09:34.045466+0000","last_clean":"2024-03-29T10:09:34.045466+0000","last_became_active":"2024-03-29T10:09:32.617123+0000","last_became_peered":"2024-03-29T10:09:32.617123+0000","last_unstale":"2024-03-29T10:09:34.045466+0000","last_undegraded":"2024-03-29T10:09:34.045466+0000","last_fullsized":"2024-03-29T10:09:34.045466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:47.161213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044987+0000","last_change":"2024-03-29T10:09:32.622774+0000","last_active":"2024-03-29T10:09:34.044987+0000","last_peered":"2024-03-29T10:09:34.044987+0000","last_clean":"2024-03-29T10:09:34.044987+0000","last_became_active":"2024-03-29T10:09:32.622503+0000","last_became_peered":"2024-03-29T10:09:32.622503+0000","last_unstale":"2024-03-29T10:09:34.044987+0000","last_undegraded":"2024-03-29T10:09:34.044987+0000","last_fullsized":"2024-03-29T10:09:34.044987+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:38.603035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607446+0000","last_change":"2024-03-29T10:09:32.625499+0000","last_active":"2024-03-29T10:09:33.607446+0000","last_peered":"2024-03-29T10:09:33.607446+0000","last_clean":"2024-03-29T10:09:33.607446+0000","last_became_active":"2024-03-29T10:09:32.625200+0000","last_became_peered":"2024-03-29T10:09:32.625200+0000","last_unstale":"2024-03-29T10:09:33.607446+0000","last_undegraded":"2024-03-29T10:09:33.607446+0000","last_fullsized":"2024-03-29T10:09:33.607446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:47.193288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045942+0000","last_change":"2024-03-29T10:09:32.622371+0000","last_active":"2024-03-29T10:09:34.045942+0000","last_peered":"2024-03-29T10:09:34.045942+0000","last_clean":"2024-03-29T10:09:34.045942+0000","last_became_active":"2024-03-29T10:09:32.621892+0000","last_became_peered":"2024-03-29T10:09:32.621892+0000","last_unstale":"2024-03-29T10:09:34.045942+0000","last_undegraded":"2024-03-29T10:09:34.045942+0000","last_fullsized":"2024-03-29T10:09:34.045942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:50.946405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045592+0000","last_change":"2024-03-29T10:09:32.615044+0000","last_active":"2024-03-29T10:09:34.045592+0000","last_peered":"2024-03-29T10:09:34.045592+0000","last_clean":"2024-03-29T10:09:34.045592+0000","last_became_active":"2024-03-29T10:09:32.614754+0000","last_became_peered":"2024-03-29T10:09:32.614754+0000","last_unstale":"2024-03-29T10:09:34.045592+0000","last_undegraded":"2024-03-29T10:09:34.045592+0000","last_fullsized":"2024-03-29T10:09:34.045592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:50.472261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607170+0000","last_change":"2024-03-29T10:09:32.613680+0000","last_active":"2024-03-29T10:09:33.607170+0000","last_peered":"2024-03-29T10:09:33.607170+0000","last_clean":"2024-03-29T10:09:33.607170+0000","last_became_active":"2024-03-29T10:09:32.611800+0000","last_became_peered":"2024-03-29T10:09:32.611800+0000","last_unstale":"2024-03-29T10:09:33.607170+0000","last_undegraded":"2024-03-29T10:09:33.607170+0000","last_fullsized":"2024-03-29T10:09:33.607170+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:04.582194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045761+0000","last_change":"2024-03-29T10:09:32.623271+0000","last_active":"2024-03-29T10:09:34.045761+0000","last_peered":"2024-03-29T10:09:34.045761+0000","last_clean":"2024-03-29T10:09:34.045761+0000","last_became_active":"2024-03-29T10:09:32.622928+0000","last_became_peered":"2024-03-29T10:09:32.622928+0000","last_unstale":"2024-03-29T10:09:34.045761+0000","last_undegraded":"2024-03-29T10:09:34.045761+0000","last_fullsized":"2024-03-29T10:09:34.045761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.930784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607046+0000","last_change":"2024-03-29T10:09:32.621045+0000","last_active":"2024-03-29T10:09:33.607046+0000","last_peered":"2024-03-29T10:09:33.607046+0000","last_clean":"2024-03-29T10:09:33.607046+0000","last_became_active":"2024-03-29T10:09:32.620499+0000","last_became_peered":"2024-03-29T10:09:32.620499+0000","last_unstale":"2024-03-29T10:09:33.607046+0000","last_undegraded":"2024-03-29T10:09:33.607046+0000","last_fullsized":"2024-03-29T10:09:33.607046+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:05:04.665074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045437+0000","last_change":"2024-03-29T10:09:32.637105+0000","last_active":"2024-03-29T10:09:34.045437+0000","last_peered":"2024-03-29T10:09:34.045437+0000","last_clean":"2024-03-29T10:09:34.045437+0000","last_became_active":"2024-03-29T10:09:32.636599+0000","last_became_peered":"2024-03-29T10:09:32.636599+0000","last_unstale":"2024-03-29T10:09:34.045437+0000","last_undegraded":"2024-03-29T10:09:34.045437+0000","last_fullsized":"2024-03-29T10:09:34.045437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:54.720344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.611723+0000","last_change":"2024-03-29T10:09:32.628611+0000","last_active":"2024-03-29T10:09:33.611723+0000","last_peered":"2024-03-29T10:09:33.611723+0000","last_clean":"2024-03-29T10:09:33.611723+0000","last_became_active":"2024-03-29T10:09:32.625523+0000","last_became_peered":"2024-03-29T10:09:32.625523+0000","last_unstale":"2024-03-29T10:09:33.611723+0000","last_undegraded":"2024-03-29T10:09:33.611723+0000","last_fullsized":"2024-03-29T10:09:33.611723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:56.014772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046009+0000","last_change":"2024-03-29T10:09:32.615838+0000","last_active":"2024-03-29T10:09:34.046009+0000","last_peered":"2024-03-29T10:09:34.046009+0000","last_clean":"2024-03-29T10:09:34.046009+0000","last_became_active":"2024-03-29T10:09:32.615274+0000","last_became_peered":"2024-03-29T10:09:32.615274+0000","last_unstale":"2024-03-29T10:09:34.046009+0000","last_undegraded":"2024-03-29T10:09:34.046009+0000","last_fullsized":"2024-03-29T10:09:34.046009+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:07.635360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613549+0000","last_change":"2024-03-29T10:09:32.626084+0000","last_active":"2024-03-29T10:09:33.613549+0000","last_peered":"2024-03-29T10:09:33.613549+0000","last_clean":"2024-03-29T10:09:33.613549+0000","last_became_active":"2024-03-29T10:09:32.625773+0000","last_became_peered":"2024-03-29T10:09:32.625773+0000","last_unstale":"2024-03-29T10:09:33.613549+0000","last_undegraded":"2024-03-29T10:09:33.613549+0000","last_fullsized":"2024-03-29T10:09:33.613549+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:39.250301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045741+0000","last_change":"2024-03-29T10:09:32.624374+0000","last_active":"2024-03-29T10:09:34.045741+0000","last_peered":"2024-03-29T10:09:34.045741+0000","last_clean":"2024-03-29T10:09:34.045741+0000","last_became_active":"2024-03-29T10:09:32.624101+0000","last_became_peered":"2024-03-29T10:09:32.624101+0000","last_unstale":"2024-03-29T10:09:34.045741+0000","last_undegraded":"2024-03-29T10:09:34.045741+0000","last_fullsized":"2024-03-29T10:09:34.045741+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:24:39.560229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045630+0000","last_change":"2024-03-29T10:09:32.618810+0000","last_active":"2024-03-29T10:09:34.045630+0000","last_peered":"2024-03-29T10:09:34.045630+0000","last_clean":"2024-03-29T10:09:34.045630+0000","last_became_active":"2024-03-29T10:09:32.618546+0000","last_became_peered":"2024-03-29T10:09:32.618546+0000","last_unstale":"2024-03-29T10:09:34.045630+0000","last_undegraded":"2024-03-29T10:09:34.045630+0000","last_fullsized":"2024-03-29T10:09:34.045630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:49.874012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607130+0000","last_change":"2024-03-29T10:09:32.618824+0000","last_active":"2024-03-29T10:09:33.607130+0000","last_peered":"2024-03-29T10:09:33.607130+0000","last_clean":"2024-03-29T10:09:33.607130+0000","last_became_active":"2024-03-29T10:09:32.618065+0000","last_became_peered":"2024-03-29T10:09:32.618065+0000","last_unstale":"2024-03-29T10:09:33.607130+0000","last_undegraded":"2024-03-29T10:09:33.607130+0000","last_fullsized":"2024-03-29T10:09:33.607130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:29.544771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045633+0000","last_change":"2024-03-29T10:09:32.622999+0000","last_active":"2024-03-29T10:09:34.045633+0000","last_peered":"2024-03-29T10:09:34.045633+0000","last_clean":"2024-03-29T10:09:34.045633+0000","last_became_active":"2024-03-29T10:09:32.622065+0000","last_became_peered":"2024-03-29T10:09:32.622065+0000","last_unstale":"2024-03-29T10:09:34.045633+0000","last_undegraded":"2024-03-29T10:09:34.045633+0000","last_fullsized":"2024-03-29T10:09:34.045633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:23.297716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044843+0000","last_change":"2024-03-29T10:09:32.615373+0000","last_active":"2024-03-29T10:09:34.044843+0000","last_peered":"2024-03-29T10:09:34.044843+0000","last_clean":"2024-03-29T10:09:34.044843+0000","last_became_active":"2024-03-29T10:09:32.615107+0000","last_became_peered":"2024-03-29T10:09:32.615107+0000","last_unstale":"2024-03-29T10:09:34.044843+0000","last_undegraded":"2024-03-29T10:09:34.044843+0000","last_fullsized":"2024-03-29T10:09:34.044843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:23.129448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045322+0000","last_change":"2024-03-29T10:09:32.618453+0000","last_active":"2024-03-29T10:09:34.045322+0000","last_peered":"2024-03-29T10:09:34.045322+0000","last_clean":"2024-03-29T10:09:34.045322+0000","last_became_active":"2024-03-29T10:09:32.616759+0000","last_became_peered":"2024-03-29T10:09:32.616759+0000","last_unstale":"2024-03-29T10:09:34.045322+0000","last_undegraded":"2024-03-29T10:09:34.045322+0000","last_fullsized":"2024-03-29T10:09:34.045322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:50.946405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045802+0000","last_change":"2024-03-29T10:09:32.623841+0000","last_active":"2024-03-29T10:09:34.045802+0000","last_peered":"2024-03-29T10:09:34.045802+0000","last_clean":"2024-03-29T10:09:34.045802+0000","last_became_active":"2024-03-29T10:09:32.623522+0000","last_became_peered":"2024-03-29T10:09:32.623522+0000","last_unstale":"2024-03-29T10:09:34.045802+0000","last_undegraded":"2024-03-29T10:09:34.045802+0000","last_fullsized":"2024-03-29T10:09:34.045802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:40.815771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"79'4","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638507+0000","last_change":"2024-03-29T10:09:32.101864+0000","last_active":"2024-03-29T10:09:32.638507+0000","last_peered":"2024-03-29T10:09:32.638507+0000","last_clean":"2024-03-29T10:09:32.638507+0000","last_became_active":"2024-03-29T10:09:32.101456+0000","last_became_peered":"2024-03-29T10:09:32.101456+0000","last_unstale":"2024-03-29T10:09:32.638507+0000","last_undegraded":"2024-03-29T10:09:32.638507+0000","last_fullsized":"2024-03-29T10:09:32.638507+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:05.714596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045625+0000","last_change":"2024-03-29T10:09:32.630347+0000","last_active":"2024-03-29T10:09:34.045625+0000","last_peered":"2024-03-29T10:09:34.045625+0000","last_clean":"2024-03-29T10:09:34.045625+0000","last_became_active":"2024-03-29T10:09:32.630092+0000","last_became_peered":"2024-03-29T10:09:32.630092+0000","last_unstale":"2024-03-29T10:09:34.045625+0000","last_undegraded":"2024-03-29T10:09:34.045625+0000","last_fullsized":"2024-03-29T10:09:34.045625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:50.745197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.635936+0000","last_change":"2024-03-29T10:09:32.091373+0000","last_active":"2024-03-29T10:09:32.635936+0000","last_peered":"2024-03-29T10:09:32.635936+0000","last_clean":"2024-03-29T10:09:32.635936+0000","last_became_active":"2024-03-29T10:09:32.091110+0000","last_became_peered":"2024-03-29T10:09:32.091110+0000","last_unstale":"2024-03-29T10:09:32.635936+0000","last_undegraded":"2024-03-29T10:09:32.635936+0000","last_fullsized":"2024-03-29T10:09:32.635936+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:32:31.535691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046042+0000","last_change":"2024-03-29T10:09:32.614418+0000","last_active":"2024-03-29T10:09:34.046042+0000","last_peered":"2024-03-29T10:09:34.046042+0000","last_clean":"2024-03-29T10:09:34.046042+0000","last_became_active":"2024-03-29T10:09:32.614112+0000","last_became_peered":"2024-03-29T10:09:32.614112+0000","last_unstale":"2024-03-29T10:09:34.046042+0000","last_undegraded":"2024-03-29T10:09:34.046042+0000","last_fullsized":"2024-03-29T10:09:34.046042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:17.345552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.635794+0000","last_change":"2024-03-29T10:09:32.086998+0000","last_active":"2024-03-29T10:09:32.635794+0000","last_peered":"2024-03-29T10:09:32.635794+0000","last_clean":"2024-03-29T10:09:32.635794+0000","last_became_active":"2024-03-29T10:09:32.086644+0000","last_became_peered":"2024-03-29T10:09:32.086644+0000","last_unstale":"2024-03-29T10:09:32.635794+0000","last_undegraded":"2024-03-29T10:09:32.635794+0000","last_fullsized":"2024-03-29T10:09:32.635794+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:27.370099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045835+0000","last_change":"2024-03-29T10:09:32.624387+0000","last_active":"2024-03-29T10:09:34.045835+0000","last_peered":"2024-03-29T10:09:34.045835+0000","last_clean":"2024-03-29T10:09:34.045835+0000","last_became_active":"2024-03-29T10:09:32.624015+0000","last_became_peered":"2024-03-29T10:09:32.624015+0000","last_unstale":"2024-03-29T10:09:34.045835+0000","last_undegraded":"2024-03-29T10:09:34.045835+0000","last_fullsized":"2024-03-29T10:09:34.045835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:06.669665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.637825+0000","last_change":"2024-03-29T10:09:32.101441+0000","last_active":"2024-03-29T10:09:32.637825+0000","last_peered":"2024-03-29T10:09:32.637825+0000","last_clean":"2024-03-29T10:09:32.637825+0000","last_became_active":"2024-03-29T10:09:32.101114+0000","last_became_peered":"2024-03-29T10:09:32.101114+0000","last_unstale":"2024-03-29T10:09:32.637825+0000","last_undegraded":"2024-03-29T10:09:32.637825+0000","last_fullsized":"2024-03-29T10:09:32.637825+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607151+0000","last_change":"2024-03-29T10:09:32.620748+0000","last_active":"2024-03-29T10:09:33.607151+0000","last_peered":"2024-03-29T10:09:33.607151+0000","last_clean":"2024-03-29T10:09:33.607151+0000","last_became_active":"2024-03-29T10:09:32.618821+0000","last_became_peered":"2024-03-29T10:09:32.618821+0000","last_unstale":"2024-03-29T10:09:33.607151+0000","last_undegraded":"2024-03-29T10:09:33.607151+0000","last_fullsized":"2024-03-29T10:09:33.607151+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:39:35.761571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.632616+0000","last_change":"2024-03-29T10:09:32.098141+0000","last_active":"2024-03-29T10:09:32.632616+0000","last_peered":"2024-03-29T10:09:32.632616+0000","last_clean":"2024-03-29T10:09:32.632616+0000","last_became_active":"2024-03-29T10:09:32.097781+0000","last_became_peered":"2024-03-29T10:09:32.097781+0000","last_unstale":"2024-03-29T10:09:32.632616+0000","last_undegraded":"2024-03-29T10:09:32.632616+0000","last_fullsized":"2024-03-29T10:09:32.632616+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607233+0000","last_change":"2024-03-29T10:09:32.614343+0000","last_active":"2024-03-29T10:09:33.607233+0000","last_peered":"2024-03-29T10:09:33.607233+0000","last_clean":"2024-03-29T10:09:33.607233+0000","last_became_active":"2024-03-29T10:09:32.614069+0000","last_became_peered":"2024-03-29T10:09:32.614069+0000","last_unstale":"2024-03-29T10:09:33.607233+0000","last_undegraded":"2024-03-29T10:09:33.607233+0000","last_fullsized":"2024-03-29T10:09:33.607233+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:22.440684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638211+0000","last_change":"2024-03-29T10:09:32.101572+0000","last_active":"2024-03-29T10:09:32.638211+0000","last_peered":"2024-03-29T10:09:32.638211+0000","last_clean":"2024-03-29T10:09:32.638211+0000","last_became_active":"2024-03-29T10:09:32.101206+0000","last_became_peered":"2024-03-29T10:09:32.101206+0000","last_unstale":"2024-03-29T10:09:32.638211+0000","last_undegraded":"2024-03-29T10:09:32.638211+0000","last_fullsized":"2024-03-29T10:09:32.638211+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:46.154988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.611858+0000","last_change":"2024-03-29T10:09:32.624716+0000","last_active":"2024-03-29T10:09:33.611858+0000","last_peered":"2024-03-29T10:09:33.611858+0000","last_clean":"2024-03-29T10:09:33.611858+0000","last_became_active":"2024-03-29T10:09:32.622676+0000","last_became_peered":"2024-03-29T10:09:32.622676+0000","last_unstale":"2024-03-29T10:09:33.611858+0000","last_undegraded":"2024-03-29T10:09:33.611858+0000","last_fullsized":"2024-03-29T10:09:33.611858+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:20.681595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.610991+0000","last_change":"2024-03-29T10:09:32.091482+0000","last_active":"2024-03-29T10:09:32.610991+0000","last_peered":"2024-03-29T10:09:32.610991+0000","last_clean":"2024-03-29T10:09:32.610991+0000","last_became_active":"2024-03-29T10:09:32.091189+0000","last_became_peered":"2024-03-29T10:09:32.091189+0000","last_unstale":"2024-03-29T10:09:32.610991+0000","last_undegraded":"2024-03-29T10:09:32.610991+0000","last_fullsized":"2024-03-29T10:09:32.610991+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607448+0000","last_change":"2024-03-29T10:09:32.634701+0000","last_active":"2024-03-29T10:09:33.607448+0000","last_peered":"2024-03-29T10:09:33.607448+0000","last_clean":"2024-03-29T10:09:33.607448+0000","last_became_active":"2024-03-29T10:09:32.631830+0000","last_became_peered":"2024-03-29T10:09:32.631830+0000","last_unstale":"2024-03-29T10:09:33.607448+0000","last_undegraded":"2024-03-29T10:09:33.607448+0000","last_fullsized":"2024-03-29T10:09:33.607448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:29.066092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.603049+0000","last_change":"2024-03-29T10:09:32.097646+0000","last_active":"2024-03-29T10:09:32.603049+0000","last_peered":"2024-03-29T10:09:32.603049+0000","last_clean":"2024-03-29T10:09:32.603049+0000","last_became_active":"2024-03-29T10:09:32.097364+0000","last_became_peered":"2024-03-29T10:09:32.097364+0000","last_unstale":"2024-03-29T10:09:32.603049+0000","last_undegraded":"2024-03-29T10:09:32.603049+0000","last_fullsized":"2024-03-29T10:09:32.603049+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045478+0000","last_change":"2024-03-29T10:09:32.617841+0000","last_active":"2024-03-29T10:09:34.045478+0000","last_peered":"2024-03-29T10:09:34.045478+0000","last_clean":"2024-03-29T10:09:34.045478+0000","last_became_active":"2024-03-29T10:09:32.616581+0000","last_became_peered":"2024-03-29T10:09:32.616581+0000","last_unstale":"2024-03-29T10:09:34.045478+0000","last_undegraded":"2024-03-29T10:09:34.045478+0000","last_fullsized":"2024-03-29T10:09:34.045478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:15.353895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.616868+0000","last_change":"2024-03-29T10:09:32.098837+0000","last_active":"2024-03-29T10:09:32.616868+0000","last_peered":"2024-03-29T10:09:32.616868+0000","last_clean":"2024-03-29T10:09:32.616868+0000","last_became_active":"2024-03-29T10:09:32.098517+0000","last_became_peered":"2024-03-29T10:09:32.098517+0000","last_unstale":"2024-03-29T10:09:32.616868+0000","last_undegraded":"2024-03-29T10:09:32.616868+0000","last_fullsized":"2024-03-29T10:09:32.616868+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:05.714596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045923+0000","last_change":"2024-03-29T10:09:32.636148+0000","last_active":"2024-03-29T10:09:34.045923+0000","last_peered":"2024-03-29T10:09:34.045923+0000","last_clean":"2024-03-29T10:09:34.045923+0000","last_became_active":"2024-03-29T10:09:32.635753+0000","last_became_peered":"2024-03-29T10:09:32.635753+0000","last_unstale":"2024-03-29T10:09:34.045923+0000","last_undegraded":"2024-03-29T10:09:34.045923+0000","last_fullsized":"2024-03-29T10:09:34.045923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:18.667325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.608025+0000","last_change":"2024-03-29T10:09:32.099480+0000","last_active":"2024-03-29T10:09:32.608025+0000","last_peered":"2024-03-29T10:09:32.608025+0000","last_clean":"2024-03-29T10:09:32.608025+0000","last_became_active":"2024-03-29T10:09:32.099133+0000","last_became_peered":"2024-03-29T10:09:32.099133+0000","last_unstale":"2024-03-29T10:09:32.608025+0000","last_undegraded":"2024-03-29T10:09:32.608025+0000","last_fullsized":"2024-03-29T10:09:32.608025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:13:16.625121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044664+0000","last_change":"2024-03-29T10:09:32.628412+0000","last_active":"2024-03-29T10:09:34.044664+0000","last_peered":"2024-03-29T10:09:34.044664+0000","last_clean":"2024-03-29T10:09:34.044664+0000","last_became_active":"2024-03-29T10:09:32.622661+0000","last_became_peered":"2024-03-29T10:09:32.622661+0000","last_unstale":"2024-03-29T10:09:34.044664+0000","last_undegraded":"2024-03-29T10:09:34.044664+0000","last_fullsized":"2024-03-29T10:09:34.044664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:23.129448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"79'3","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636825+0000","last_change":"2024-03-29T10:09:32.089351+0000","last_active":"2024-03-29T10:09:32.636825+0000","last_peered":"2024-03-29T10:09:32.636825+0000","last_clean":"2024-03-29T10:09:32.636825+0000","last_became_active":"2024-03-29T10:09:32.087924+0000","last_became_peered":"2024-03-29T10:09:32.087924+0000","last_unstale":"2024-03-29T10:09:32.636825+0000","last_undegraded":"2024-03-29T10:09:32.636825+0000","last_fullsized":"2024-03-29T10:09:32.636825+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:46.154988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045696+0000","last_change":"2024-03-29T10:09:32.632023+0000","last_active":"2024-03-29T10:09:34.045696+0000","last_peered":"2024-03-29T10:09:34.045696+0000","last_clean":"2024-03-29T10:09:34.045696+0000","last_became_active":"2024-03-29T10:09:32.631450+0000","last_became_peered":"2024-03-29T10:09:32.631450+0000","last_unstale":"2024-03-29T10:09:34.045696+0000","last_undegraded":"2024-03-29T10:09:34.045696+0000","last_fullsized":"2024-03-29T10:09:34.045696+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:24:39.560229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"79'6","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638720+0000","last_change":"2024-03-29T10:09:32.099892+0000","last_active":"2024-03-29T10:09:32.638720+0000","last_peered":"2024-03-29T10:09:32.638720+0000","last_clean":"2024-03-29T10:09:32.638720+0000","last_became_active":"2024-03-29T10:09:32.099626+0000","last_became_peered":"2024-03-29T10:09:32.099626+0000","last_unstale":"2024-03-29T10:09:32.638720+0000","last_undegraded":"2024-03-29T10:09:32.638720+0000","last_fullsized":"2024-03-29T10:09:32.638720+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613351+0000","last_change":"2024-03-29T10:09:32.626554+0000","last_active":"2024-03-29T10:09:33.613351+0000","last_peered":"2024-03-29T10:09:33.613351+0000","last_clean":"2024-03-29T10:09:33.613351+0000","last_became_active":"2024-03-29T10:09:32.625988+0000","last_became_peered":"2024-03-29T10:09:32.625988+0000","last_unstale":"2024-03-29T10:09:33.613351+0000","last_undegraded":"2024-03-29T10:09:33.613351+0000","last_fullsized":"2024-03-29T10:09:33.613351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:10:07.105194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636457+0000","last_change":"2024-03-29T10:09:32.100281+0000","last_active":"2024-03-29T10:09:32.636457+0000","last_peered":"2024-03-29T10:09:32.636457+0000","last_clean":"2024-03-29T10:09:32.636457+0000","last_became_active":"2024-03-29T10:09:32.100004+0000","last_became_peered":"2024-03-29T10:09:32.100004+0000","last_unstale":"2024-03-29T10:09:32.636457+0000","last_undegraded":"2024-03-29T10:09:32.636457+0000","last_fullsized":"2024-03-29T10:09:32.636457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:49.940180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045675+0000","last_change":"2024-03-29T10:09:32.624300+0000","last_active":"2024-03-29T10:09:34.045675+0000","last_peered":"2024-03-29T10:09:34.045675+0000","last_clean":"2024-03-29T10:09:34.045675+0000","last_became_active":"2024-03-29T10:09:32.624004+0000","last_became_peered":"2024-03-29T10:09:32.624004+0000","last_unstale":"2024-03-29T10:09:34.045675+0000","last_undegraded":"2024-03-29T10:09:34.045675+0000","last_fullsized":"2024-03-29T10:09:34.045675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:38.603035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607519+0000","last_change":"2024-03-29T10:09:32.091147+0000","last_active":"2024-03-29T10:09:32.607519+0000","last_peered":"2024-03-29T10:09:32.607519+0000","last_clean":"2024-03-29T10:09:32.607519+0000","last_became_active":"2024-03-29T10:09:32.089656+0000","last_became_peered":"2024-03-29T10:09:32.089656+0000","last_unstale":"2024-03-29T10:09:32.607519+0000","last_undegraded":"2024-03-29T10:09:32.607519+0000","last_fullsized":"2024-03-29T10:09:32.607519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:22.123223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044770+0000","last_change":"2024-03-29T10:09:32.623116+0000","last_active":"2024-03-29T10:09:34.044770+0000","last_peered":"2024-03-29T10:09:34.044770+0000","last_clean":"2024-03-29T10:09:34.044770+0000","last_became_active":"2024-03-29T10:09:32.622832+0000","last_became_peered":"2024-03-29T10:09:32.622832+0000","last_unstale":"2024-03-29T10:09:34.044770+0000","last_undegraded":"2024-03-29T10:09:34.044770+0000","last_fullsized":"2024-03-29T10:09:34.044770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:36.451327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.611108+0000","last_change":"2024-03-29T10:09:32.093865+0000","last_active":"2024-03-29T10:09:32.611108+0000","last_peered":"2024-03-29T10:09:32.611108+0000","last_clean":"2024-03-29T10:09:32.611108+0000","last_became_active":"2024-03-29T10:09:32.092709+0000","last_became_peered":"2024-03-29T10:09:32.092709+0000","last_unstale":"2024-03-29T10:09:32.611108+0000","last_undegraded":"2024-03-29T10:09:32.611108+0000","last_fullsized":"2024-03-29T10:09:32.611108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045252+0000","last_change":"2024-03-29T10:09:32.634186+0000","last_active":"2024-03-29T10:09:34.045252+0000","last_peered":"2024-03-29T10:09:34.045252+0000","last_clean":"2024-03-29T10:09:34.045252+0000","last_became_active":"2024-03-29T10:09:32.633693+0000","last_became_peered":"2024-03-29T10:09:32.633693+0000","last_unstale":"2024-03-29T10:09:34.045252+0000","last_undegraded":"2024-03-29T10:09:34.045252+0000","last_fullsized":"2024-03-29T10:09:34.045252+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:40.815771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"79'8","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.639114+0000","last_change":"2024-03-29T10:09:32.092854+0000","last_active":"2024-03-29T10:09:32.639114+0000","last_peered":"2024-03-29T10:09:32.639114+0000","last_clean":"2024-03-29T10:09:32.639114+0000","last_became_active":"2024-03-29T10:09:32.092591+0000","last_became_peered":"2024-03-29T10:09:32.092591+0000","last_unstale":"2024-03-29T10:09:32.639114+0000","last_undegraded":"2024-03-29T10:09:32.639114+0000","last_fullsized":"2024-03-29T10:09:32.639114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:40.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045937+0000","last_change":"2024-03-29T10:09:32.623867+0000","last_active":"2024-03-29T10:09:34.045937+0000","last_peered":"2024-03-29T10:09:34.045937+0000","last_clean":"2024-03-29T10:09:34.045937+0000","last_became_active":"2024-03-29T10:09:32.622284+0000","last_became_peered":"2024-03-29T10:09:32.622284+0000","last_unstale":"2024-03-29T10:09:34.045937+0000","last_undegraded":"2024-03-29T10:09:34.045937+0000","last_fullsized":"2024-03-29T10:09:34.045937+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:06.669665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636096+0000","last_change":"2024-03-29T10:09:32.097805+0000","last_active":"2024-03-29T10:09:32.636096+0000","last_peered":"2024-03-29T10:09:32.636096+0000","last_clean":"2024-03-29T10:09:32.636096+0000","last_became_active":"2024-03-29T10:09:32.097537+0000","last_became_peered":"2024-03-29T10:09:32.097537+0000","last_unstale":"2024-03-29T10:09:32.636096+0000","last_undegraded":"2024-03-29T10:09:32.636096+0000","last_fullsized":"2024-03-29T10:09:32.636096+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:35.016753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613614+0000","last_change":"2024-03-29T10:09:32.612197+0000","last_active":"2024-03-29T10:09:33.613614+0000","last_peered":"2024-03-29T10:09:33.613614+0000","last_clean":"2024-03-29T10:09:33.613614+0000","last_became_active":"2024-03-29T10:09:32.611918+0000","last_became_peered":"2024-03-29T10:09:32.611918+0000","last_unstale":"2024-03-29T10:09:33.613614+0000","last_undegraded":"2024-03-29T10:09:33.613614+0000","last_fullsized":"2024-03-29T10:09:33.613614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:32:07.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.634994+0000","last_change":"2024-03-29T10:09:32.104998+0000","last_active":"2024-03-29T10:09:32.634994+0000","last_peered":"2024-03-29T10:09:32.634994+0000","last_clean":"2024-03-29T10:09:32.634994+0000","last_became_active":"2024-03-29T10:09:32.104610+0000","last_became_peered":"2024-03-29T10:09:32.104610+0000","last_unstale":"2024-03-29T10:09:32.634994+0000","last_undegraded":"2024-03-29T10:09:32.634994+0000","last_fullsized":"2024-03-29T10:09:32.634994+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:38.244076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607169+0000","last_change":"2024-03-29T10:09:32.619003+0000","last_active":"2024-03-29T10:09:33.607169+0000","last_peered":"2024-03-29T10:09:33.607169+0000","last_clean":"2024-03-29T10:09:33.607169+0000","last_became_active":"2024-03-29T10:09:32.617951+0000","last_became_peered":"2024-03-29T10:09:32.617951+0000","last_unstale":"2024-03-29T10:09:33.607169+0000","last_undegraded":"2024-03-29T10:09:33.607169+0000","last_fullsized":"2024-03-29T10:09:33.607169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:00.709669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.637733+0000","last_change":"2024-03-29T10:09:32.095098+0000","last_active":"2024-03-29T10:09:32.637733+0000","last_peered":"2024-03-29T10:09:32.637733+0000","last_clean":"2024-03-29T10:09:32.637733+0000","last_became_active":"2024-03-29T10:09:32.093937+0000","last_became_peered":"2024-03-29T10:09:32.093937+0000","last_unstale":"2024-03-29T10:09:32.637733+0000","last_undegraded":"2024-03-29T10:09:32.637733+0000","last_fullsized":"2024-03-29T10:09:32.637733+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:27:35.691958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045708+0000","last_change":"2024-03-29T10:09:32.633244+0000","last_active":"2024-03-29T10:09:34.045708+0000","last_peered":"2024-03-29T10:09:34.045708+0000","last_clean":"2024-03-29T10:09:34.045708+0000","last_became_active":"2024-03-29T10:09:32.632943+0000","last_became_peered":"2024-03-29T10:09:32.632943+0000","last_unstale":"2024-03-29T10:09:34.045708+0000","last_undegraded":"2024-03-29T10:09:34.045708+0000","last_fullsized":"2024-03-29T10:09:34.045708+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:36.022978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.606805+0000","last_change":"2024-03-29T10:09:32.089495+0000","last_active":"2024-03-29T10:09:32.606805+0000","last_peered":"2024-03-29T10:09:32.606805+0000","last_clean":"2024-03-29T10:09:32.606805+0000","last_became_active":"2024-03-29T10:09:32.089228+0000","last_became_peered":"2024-03-29T10:09:32.089228+0000","last_unstale":"2024-03-29T10:09:32.606805+0000","last_undegraded":"2024-03-29T10:09:32.606805+0000","last_fullsized":"2024-03-29T10:09:32.606805+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:14.347670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613336+0000","last_change":"2024-03-29T10:09:32.614565+0000","last_active":"2024-03-29T10:09:33.613336+0000","last_peered":"2024-03-29T10:09:33.613336+0000","last_clean":"2024-03-29T10:09:33.613336+0000","last_became_active":"2024-03-29T10:09:32.614312+0000","last_became_peered":"2024-03-29T10:09:32.614312+0000","last_unstale":"2024-03-29T10:09:33.613336+0000","last_undegraded":"2024-03-29T10:09:33.613336+0000","last_fullsized":"2024-03-29T10:09:33.613336+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:36.451327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636359+0000","last_change":"2024-03-29T10:09:32.095956+0000","last_active":"2024-03-29T10:09:32.636359+0000","last_peered":"2024-03-29T10:09:32.636359+0000","last_clean":"2024-03-29T10:09:32.636359+0000","last_became_active":"2024-03-29T10:09:32.093465+0000","last_became_peered":"2024-03-29T10:09:32.093465+0000","last_unstale":"2024-03-29T10:09:32.636359+0000","last_undegraded":"2024-03-29T10:09:32.636359+0000","last_fullsized":"2024-03-29T10:09:32.636359+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.686777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045534+0000","last_change":"2024-03-29T10:09:32.622232+0000","last_active":"2024-03-29T10:09:34.045534+0000","last_peered":"2024-03-29T10:09:34.045534+0000","last_clean":"2024-03-29T10:09:34.045534+0000","last_became_active":"2024-03-29T10:09:32.621815+0000","last_became_peered":"2024-03-29T10:09:32.621815+0000","last_unstale":"2024-03-29T10:09:34.045534+0000","last_undegraded":"2024-03-29T10:09:34.045534+0000","last_fullsized":"2024-03-29T10:09:34.045534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:54.720344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607356+0000","last_change":"2024-03-29T10:09:32.099575+0000","last_active":"2024-03-29T10:09:32.607356+0000","last_peered":"2024-03-29T10:09:32.607356+0000","last_clean":"2024-03-29T10:09:32.607356+0000","last_became_active":"2024-03-29T10:09:32.098434+0000","last_became_peered":"2024-03-29T10:09:32.098434+0000","last_unstale":"2024-03-29T10:09:32.607356+0000","last_undegraded":"2024-03-29T10:09:32.607356+0000","last_fullsized":"2024-03-29T10:09:32.607356+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:39.694641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044600+0000","last_change":"2024-03-29T10:09:32.623727+0000","last_active":"2024-03-29T10:09:34.044600+0000","last_peered":"2024-03-29T10:09:34.044600+0000","last_clean":"2024-03-29T10:09:34.044600+0000","last_became_active":"2024-03-29T10:09:32.623445+0000","last_became_peered":"2024-03-29T10:09:32.623445+0000","last_unstale":"2024-03-29T10:09:34.044600+0000","last_undegraded":"2024-03-29T10:09:34.044600+0000","last_fullsized":"2024-03-29T10:09:34.044600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:18.667325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.608745+0000","last_change":"2024-03-29T10:09:32.098093+0000","last_active":"2024-03-29T10:09:32.608745+0000","last_peered":"2024-03-29T10:09:32.608745+0000","last_clean":"2024-03-29T10:09:32.608745+0000","last_became_active":"2024-03-29T10:09:32.097686+0000","last_became_peered":"2024-03-29T10:09:32.097686+0000","last_unstale":"2024-03-29T10:09:32.608745+0000","last_undegraded":"2024-03-29T10:09:32.608745+0000","last_fullsized":"2024-03-29T10:09:32.608745+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:51:58.170982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613642+0000","last_change":"2024-03-29T10:09:32.623803+0000","last_active":"2024-03-29T10:09:33.613642+0000","last_peered":"2024-03-29T10:09:33.613642+0000","last_clean":"2024-03-29T10:09:33.613642+0000","last_became_active":"2024-03-29T10:09:32.621829+0000","last_became_peered":"2024-03-29T10:09:32.621829+0000","last_unstale":"2024-03-29T10:09:33.613642+0000","last_undegraded":"2024-03-29T10:09:33.613642+0000","last_fullsized":"2024-03-29T10:09:33.613642+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:56.974307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.637133+0000","last_change":"2024-03-29T10:09:32.089390+0000","last_active":"2024-03-29T10:09:32.637133+0000","last_peered":"2024-03-29T10:09:32.637133+0000","last_clean":"2024-03-29T10:09:32.637133+0000","last_became_active":"2024-03-29T10:09:32.089088+0000","last_became_peered":"2024-03-29T10:09:32.089088+0000","last_unstale":"2024-03-29T10:09:32.637133+0000","last_undegraded":"2024-03-29T10:09:32.637133+0000","last_fullsized":"2024-03-29T10:09:32.637133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:11:34.599249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613753+0000","last_change":"2024-03-29T10:09:32.622358+0000","last_active":"2024-03-29T10:09:33.613753+0000","last_peered":"2024-03-29T10:09:33.613753+0000","last_clean":"2024-03-29T10:09:33.613753+0000","last_became_active":"2024-03-29T10:09:32.622059+0000","last_became_peered":"2024-03-29T10:09:32.622059+0000","last_unstale":"2024-03-29T10:09:33.613753+0000","last_undegraded":"2024-03-29T10:09:33.613753+0000","last_fullsized":"2024-03-29T10:09:33.613753+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:51.135113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.609775+0000","last_change":"2024-03-29T10:09:32.101276+0000","last_active":"2024-03-29T10:09:32.609775+0000","last_peered":"2024-03-29T10:09:32.609775+0000","last_clean":"2024-03-29T10:09:32.609775+0000","last_became_active":"2024-03-29T10:09:32.100949+0000","last_became_peered":"2024-03-29T10:09:32.100949+0000","last_unstale":"2024-03-29T10:09:32.609775+0000","last_undegraded":"2024-03-29T10:09:32.609775+0000","last_fullsized":"2024-03-29T10:09:32.609775+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:40.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044751+0000","last_change":"2024-03-29T10:09:32.611596+0000","last_active":"2024-03-29T10:09:34.044751+0000","last_peered":"2024-03-29T10:09:34.044751+0000","last_clean":"2024-03-29T10:09:34.044751+0000","last_became_active":"2024-03-29T10:09:32.610142+0000","last_became_peered":"2024-03-29T10:09:32.610142+0000","last_unstale":"2024-03-29T10:09:34.044751+0000","last_undegraded":"2024-03-29T10:09:34.044751+0000","last_fullsized":"2024-03-29T10:09:34.044751+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:07.635360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605517+0000","last_change":"2024-03-29T10:09:32.092374+0000","last_active":"2024-03-29T10:09:32.605517+0000","last_peered":"2024-03-29T10:09:32.605517+0000","last_clean":"2024-03-29T10:09:32.605517+0000","last_became_active":"2024-03-29T10:09:32.091381+0000","last_became_peered":"2024-03-29T10:09:32.091381+0000","last_unstale":"2024-03-29T10:09:32.605517+0000","last_undegraded":"2024-03-29T10:09:32.605517+0000","last_fullsized":"2024-03-29T10:09:32.605517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:06.629135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046025+0000","last_change":"2024-03-29T10:09:32.620976+0000","last_active":"2024-03-29T10:09:34.046025+0000","last_peered":"2024-03-29T10:09:34.046025+0000","last_clean":"2024-03-29T10:09:34.046025+0000","last_became_active":"2024-03-29T10:09:32.619271+0000","last_became_peered":"2024-03-29T10:09:32.619271+0000","last_unstale":"2024-03-29T10:09:34.046025+0000","last_undegraded":"2024-03-29T10:09:34.046025+0000","last_fullsized":"2024-03-29T10:09:34.046025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.930784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.611252+0000","last_change":"2024-03-29T10:09:32.088757+0000","last_active":"2024-03-29T10:09:32.611252+0000","last_peered":"2024-03-29T10:09:32.611252+0000","last_clean":"2024-03-29T10:09:32.611252+0000","last_became_active":"2024-03-29T10:09:32.088508+0000","last_became_peered":"2024-03-29T10:09:32.088508+0000","last_unstale":"2024-03-29T10:09:32.611252+0000","last_undegraded":"2024-03-29T10:09:32.611252+0000","last_fullsized":"2024-03-29T10:09:32.611252+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:56.946534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607553+0000","last_change":"2024-03-29T10:09:32.618538+0000","last_active":"2024-03-29T10:09:33.607553+0000","last_peered":"2024-03-29T10:09:33.607553+0000","last_clean":"2024-03-29T10:09:33.607553+0000","last_became_active":"2024-03-29T10:09:32.617981+0000","last_became_peered":"2024-03-29T10:09:32.617981+0000","last_unstale":"2024-03-29T10:09:33.607553+0000","last_undegraded":"2024-03-29T10:09:33.607553+0000","last_fullsized":"2024-03-29T10:09:33.607553+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:50.197997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607488+0000","last_change":"2024-03-29T10:09:32.084059+0000","last_active":"2024-03-29T10:09:32.607488+0000","last_peered":"2024-03-29T10:09:32.607488+0000","last_clean":"2024-03-29T10:09:32.607488+0000","last_became_active":"2024-03-29T10:09:32.082631+0000","last_became_peered":"2024-03-29T10:09:32.082631+0000","last_unstale":"2024-03-29T10:09:32.607488+0000","last_undegraded":"2024-03-29T10:09:32.607488+0000","last_fullsized":"2024-03-29T10:09:32.607488+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:37:31.209166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045792+0000","last_change":"2024-03-29T10:09:32.628132+0000","last_active":"2024-03-29T10:09:34.045792+0000","last_peered":"2024-03-29T10:09:34.045792+0000","last_clean":"2024-03-29T10:09:34.045792+0000","last_became_active":"2024-03-29T10:09:32.627845+0000","last_became_peered":"2024-03-29T10:09:32.627845+0000","last_unstale":"2024-03-29T10:09:34.045792+0000","last_undegraded":"2024-03-29T10:09:34.045792+0000","last_fullsized":"2024-03-29T10:09:34.045792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:52.322978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.604249+0000","last_change":"2024-03-29T10:09:32.094306+0000","last_active":"2024-03-29T10:09:32.604249+0000","last_peered":"2024-03-29T10:09:32.604249+0000","last_clean":"2024-03-29T10:09:32.604249+0000","last_became_active":"2024-03-29T10:09:32.093998+0000","last_became_peered":"2024-03-29T10:09:32.093998+0000","last_unstale":"2024-03-29T10:09:32.604249+0000","last_undegraded":"2024-03-29T10:09:32.604249+0000","last_fullsized":"2024-03-29T10:09:32.604249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:17.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044480+0000","last_change":"2024-03-29T10:09:32.620507+0000","last_active":"2024-03-29T10:09:34.044480+0000","last_peered":"2024-03-29T10:09:34.044480+0000","last_clean":"2024-03-29T10:09:34.044480+0000","last_became_active":"2024-03-29T10:09:32.620186+0000","last_became_peered":"2024-03-29T10:09:32.620186+0000","last_unstale":"2024-03-29T10:09:34.044480+0000","last_undegraded":"2024-03-29T10:09:34.044480+0000","last_fullsized":"2024-03-29T10:09:34.044480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:49.874012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605757+0000","last_change":"2024-03-29T10:09:32.086897+0000","last_active":"2024-03-29T10:09:32.605757+0000","last_peered":"2024-03-29T10:09:32.605757+0000","last_clean":"2024-03-29T10:09:32.605757+0000","last_became_active":"2024-03-29T10:09:32.086602+0000","last_became_peered":"2024-03-29T10:09:32.086602+0000","last_unstale":"2024-03-29T10:09:32.605757+0000","last_undegraded":"2024-03-29T10:09:32.605757+0000","last_fullsized":"2024-03-29T10:09:32.605757+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:23:18.397722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607519+0000","last_change":"2024-03-29T10:09:32.633167+0000","last_active":"2024-03-29T10:09:33.607519+0000","last_peered":"2024-03-29T10:09:33.607519+0000","last_clean":"2024-03-29T10:09:33.607519+0000","last_became_active":"2024-03-29T10:09:32.632682+0000","last_became_peered":"2024-03-29T10:09:32.632682+0000","last_unstale":"2024-03-29T10:09:33.607519+0000","last_undegraded":"2024-03-29T10:09:33.607519+0000","last_fullsized":"2024-03-29T10:09:33.607519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:07.268799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605003+0000","last_change":"2024-03-29T10:09:32.092768+0000","last_active":"2024-03-29T10:09:32.605003+0000","last_peered":"2024-03-29T10:09:32.605003+0000","last_clean":"2024-03-29T10:09:32.605003+0000","last_became_active":"2024-03-29T10:09:32.092488+0000","last_became_peered":"2024-03-29T10:09:32.092488+0000","last_unstale":"2024-03-29T10:09:32.605003+0000","last_undegraded":"2024-03-29T10:09:32.605003+0000","last_fullsized":"2024-03-29T10:09:32.605003+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:16.339327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.606964+0000","last_change":"2024-03-29T10:09:32.628533+0000","last_active":"2024-03-29T10:09:33.606964+0000","last_peered":"2024-03-29T10:09:33.606964+0000","last_clean":"2024-03-29T10:09:33.606964+0000","last_became_active":"2024-03-29T10:09:32.628297+0000","last_became_peered":"2024-03-29T10:09:32.628297+0000","last_unstale":"2024-03-29T10:09:33.606964+0000","last_undegraded":"2024-03-29T10:09:33.606964+0000","last_fullsized":"2024-03-29T10:09:33.606964+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:53:16.735792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.604907+0000","last_change":"2024-03-29T10:09:32.088461+0000","last_active":"2024-03-29T10:09:32.604907+0000","last_peered":"2024-03-29T10:09:32.604907+0000","last_clean":"2024-03-29T10:09:32.604907+0000","last_became_active":"2024-03-29T10:09:32.087924+0000","last_became_peered":"2024-03-29T10:09:32.087924+0000","last_unstale":"2024-03-29T10:09:32.604907+0000","last_undegraded":"2024-03-29T10:09:32.604907+0000","last_fullsized":"2024-03-29T10:09:32.604907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:01.534207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"68'161","reported_seq":379,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.613595+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:09:32.613595+0000","last_peered":"2024-03-29T10:09:32.613595+0000","last_clean":"2024-03-29T10:09:32.613595+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:09:32.613595+0000","last_undegraded":"2024-03-29T10:09:32.613595+0000","last_fullsized":"2024-03-29T10:09:32.613595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1568768,"data_stored":1047510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.57299999999999995,"5min":0.44500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95299999999999996,"5min":1.0029999999999999,"15min":1.0109999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.69999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.89800000000000002,"5min":0.93400000000000005,"15min":1.046},"last":0.72299999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71699999999999997,"15min":0.71999999999999997},"min":{"1min":0.51900000000000002,"5min":0.47399999999999998,"15min":0.44500000000000001},"max":{"1min":0.92600000000000005,"5min":0.93200000000000005,"15min":1.1910000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.73499999999999999},"min":{"1min":0.51700000000000002,"5min":0.42199999999999999,"15min":0.372},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1},"last":0.71099999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.72899999999999998,"15min":0.74099999999999999},"min":{"1min":0.47199999999999998,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.046},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.755,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":1.079},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.48099999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.93100000000000005,"5min":0.97699999999999998,"15min":1.081},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77500000000000002,"15min":0.78300000000000003},"min":{"1min":0.55800000000000005,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":0.96199999999999997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.74199999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.29199999999999998},"max":{"1min":0.86399999999999999,"5min":0.87,"15min":1.111},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.627,"15min":0.64100000000000001},"min":{"1min":0.44800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":0.95199999999999996,"15min":0.95599999999999996},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64900000000000002,"15min":0.65900000000000003},"min":{"1min":0.432,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95299999999999996,"5min":0.96099999999999997,"15min":1.0089999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67600000000000005,"15min":0.68899999999999995},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90100000000000002,"5min":0.92800000000000005,"15min":1.141},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.71899999999999997,"15min":0.73299999999999998},"min":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.49199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":1.1679999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69699999999999995,"15min":0.70499999999999996},"min":{"1min":0.54800000000000004,"5min":0.48499999999999999,"15min":0.435},"max":{"1min":0.93799999999999994,"5min":0.98399999999999999,"15min":1.121},"last":0.48499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575031,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":966656,"data_stored":456982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70299999999999996,"15min":0.745},"min":{"1min":0.57199999999999995,"5min":0.42799999999999999,"15min":0.34699999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":7.4080000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.72599999999999998,"15min":0.76900000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.42499999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":6.7610000000000001},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70999999999999996,"15min":0.71899999999999997},"min":{"1min":0.51300000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.2549999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.72499999999999998,"15min":0.71999999999999997},"min":{"1min":0.58999999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":0.995,"15min":1.0209999999999999},"last":0.745}]},{"osd":2,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.71699999999999997,"15min":0.72399999999999998},"min":{"1min":0.65900000000000003,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94199999999999995,"5min":0.96899999999999997,"15min":1.3169999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.58899999999999997,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.415},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.74099999999999999,"15min":0.76900000000000002},"min":{"1min":0.60399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.98999999999999999,"5min":1.02,"15min":3.1960000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76800000000000002,"15min":0.79800000000000004},"min":{"1min":0.71499999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0109999999999999,"5min":1.014,"15min":4.3760000000000003},"last":0.70499999999999996}]},{"osd":4,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":1.109},"last":0.753},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.47799999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85899999999999999,"5min":0.95599999999999996,"15min":3.1219999999999999},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.67300000000000004},"min":{"1min":0.50600000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90000000000000002,"5min":0.90900000000000003,"15min":2.8279999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.79500000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.74399999999999999,"15min":0.76100000000000001},"min":{"1min":0.56499999999999995,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.95499999999999996,"5min":1.105,"15min":1.2769999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.496,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94499999999999995,"5min":0.998,"15min":1.7869999999999999},"last":0.76200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50116,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321724,"statfs":{"total":96636764160,"available":96585445376,"internally_reserved":0,"allocated":970752,"data_stored":457072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.315,"5min":0.219,"15min":0.219},"max":{"1min":0.65300000000000002,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.379},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.68999999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.504}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71599999999999997,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.52900000000000003},"min":{"1min":0.36699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":4.282},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.70999999999999996,"5min":0.78500000000000003,"15min":4.3200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.36599999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.016,"5min":1.617,"15min":1.617},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76400000000000001,"15min":0.78800000000000003},"min":{"1min":0.60899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0289999999999999,"5min":1.1419999999999999,"15min":4.3559999999999999},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.51800000000000002},"max":{"1min":1.056,"5min":1.3939999999999999,"15min":1.9119999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78600000000000003,"15min":0.80800000000000005},"min":{"1min":0.56299999999999994,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.048,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77800000000000002,"15min":0.81000000000000005},"min":{"1min":0.57499999999999996,"5min":0.52600000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":1.194,"15min":4.3899999999999997},"last":0.752},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.093,"5min":1.468,"15min":1.468},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.62,"5min":0.58699999999999997,"15min":0.56200000000000006},"max":{"1min":1.127,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.83599999999999997},"min":{"1min":0.60999999999999999,"5min":0.57099999999999995,"15min":0.55700000000000005},"max":{"1min":1.0269999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.73699999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1572864,"data_stored":1048477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.33300000000000002,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.497,"15min":0.51000000000000001},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72199999999999998,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.628}]},{"osd":2,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.435,"5min":0.39200000000000002,"15min":0.35499999999999998},"max":{"1min":0.78800000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.48799999999999999,"15min":0.496},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.63400000000000001,"5min":0.83899999999999997,"15min":0.97199999999999998},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.39000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73599999999999999,"5min":0.83199999999999996,"15min":1.077},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.47199999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83499999999999996,"5min":0.876,"15min":1.0549999999999999},"last":0.5}]},{"osd":4,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.81399999999999995,"15min":0.80200000000000005},"min":{"1min":0.65100000000000002,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":1.0469999999999999,"5min":1.625,"15min":1.625},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.82399999999999995},"min":{"1min":0.66900000000000004,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":0.97099999999999997,"5min":1.1419999999999999,"15min":1.167},"last":0.73399999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.752,"15min":0.75900000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98999999999999999,"5min":1.137,"15min":1.137},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.80600000000000005,"15min":0.80300000000000005},"min":{"1min":0.60399999999999998,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":0.90200000000000002,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.752}]},{"osd":6,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.53400000000000003},"max":{"1min":1.0129999999999999,"5min":1.0640000000000001,"15min":1.228},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.85499999999999998,"15min":0.84399999999999997},"min":{"1min":0.66400000000000003,"5min":0.52600000000000002,"15min":0.46899999999999997},"max":{"1min":1.0249999999999999,"5min":1.601,"15min":1.601},"last":0.80100000000000005}]},{"osd":7,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78300000000000003,"15min":0.79600000000000004},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.97299999999999998,"5min":1.0529999999999999,"15min":1.1890000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.84699999999999998,"15min":0.84199999999999997},"min":{"1min":0.64000000000000001,"5min":0.57499999999999996,"15min":0.49299999999999999},"max":{"1min":1.046,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.84199999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36172,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335668,"statfs":{"total":96636764160,"available":96599724032,"internally_reserved":0,"allocated":974848,"data_stored":457038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.34699999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.623,"5min":0.69599999999999995,"15min":0.86599999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.39300000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":1.3140000000000001},"last":0.47999999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47299999999999998,"15min":0.46999999999999997},"min":{"1min":0.376,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.63600000000000001,"5min":0.751,"15min":0.94499999999999995},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.45800000000000002,"5min":0.36099999999999999,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":1.032,"15min":1.032},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.504,"5min":0.375,"15min":0.375},"max":{"1min":0.73799999999999999,"5min":0.80000000000000004,"15min":1.127},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.48999999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70899999999999996,"5min":0.82399999999999995,"15min":1.1479999999999999},"last":0.51800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.77300000000000002,"15min":0.78600000000000003},"min":{"1min":0.68700000000000006,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":1.0760000000000001,"5min":1.1799999999999999,"15min":1.554},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.79500000000000004,"15min":0.81000000000000005},"min":{"1min":0.60799999999999998,"5min":0.55000000000000004,"15min":0.51600000000000001},"max":{"1min":1.0309999999999999,"5min":1.236,"15min":1.577},"last":0.86099999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.81000000000000005},"min":{"1min":0.64700000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0680000000000001,"5min":1.25,"15min":1.5209999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79600000000000004,"15min":0.80800000000000005},"min":{"1min":0.58799999999999997,"5min":0.53200000000000003,"15min":0.51800000000000002},"max":{"1min":1.0049999999999999,"5min":1.2250000000000001,"15min":1.3300000000000001},"last":0.77700000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.80400000000000005,"15min":0.82799999999999996},"min":{"1min":0.65200000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1399999999999999,"5min":1.1539999999999999,"15min":1.758},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.874,"5min":0.83699999999999997,"15min":0.84199999999999997},"min":{"1min":0.65900000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1080000000000001,"5min":1.3560000000000001,"15min":1.4750000000000001},"last":0.90900000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.86399999999999999,"15min":0.86899999999999999},"min":{"1min":0.73599999999999999,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.137,"5min":1.2709999999999999,"15min":1.4490000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.69999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.107,"5min":1.3049999999999999,"15min":1.4390000000000001},"last":0.96799999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575030,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36172,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335668,"statfs":{"total":96636764160,"available":96599724032,"internally_reserved":0,"allocated":974848,"data_stored":457038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.46600000000000003},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":0.79500000000000004,"15min":4.3799999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52600000000000002,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":4.6379999999999999},"last":0.621}]},{"osd":1,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.40200000000000002,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.69799999999999995,"5min":0.98699999999999999,"15min":4.1840000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.314},"max":{"1min":0.80800000000000005,"5min":0.96099999999999997,"15min":1.048},"last":0.67600000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.47799999999999998,"5min":0.373,"15min":0.33700000000000002},"max":{"1min":0.77200000000000002,"5min":0.82499999999999996,"15min":1.369},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.63800000000000001,"15min":0.64700000000000002},"min":{"1min":0.55700000000000005,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.79700000000000004,"5min":0.94799999999999995,"15min":4.5970000000000004},"last":0.748}]},{"osd":4,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76800000000000002,"15min":0.78100000000000003},"min":{"1min":0.64400000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.0940000000000001,"5min":1.288,"15min":4.3319999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77600000000000002,"15min":0.77500000000000002},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.39400000000000002},"max":{"1min":0.92400000000000004,"5min":1.46,"15min":1.46},"last":1.0900000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78200000000000003,"15min":0.79500000000000004},"min":{"1min":0.628,"5min":0.48899999999999999,"15min":0.435},"max":{"1min":0.97599999999999998,"5min":1.3580000000000001,"15min":1.3740000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79800000000000004,"15min":0.81000000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.44500000000000001},"max":{"1min":1.133,"5min":1.397,"15min":4.4180000000000001},"last":0.876}]},{"osd":6,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81399999999999995,"15min":0.81200000000000006},"min":{"1min":0.68899999999999995,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.0009999999999999,"5min":1.2549999999999999,"15min":1.4730000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81699999999999995,"15min":0.81899999999999995},"min":{"1min":0.65000000000000002,"5min":0.52400000000000002,"15min":0.503},"max":{"1min":1.0409999999999999,"5min":1.23,"15min":1.417},"last":1.0349999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.83999999999999997,"15min":0.84799999999999998},"min":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.54300000000000004},"max":{"1min":1.135,"5min":1.5,"15min":4.4550000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.64400000000000002,"5min":0.59899999999999998,"15min":0.56200000000000006},"max":{"1min":1.056,"5min":1.345,"15min":1.476},"last":1.0489999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54224,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317616,"statfs":{"total":96636764160,"available":96581238784,"internally_reserved":0,"allocated":978944,"data_stored":458546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.68700000000000006},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.88200000000000001,"5min":0.93200000000000005,"15min":1.417},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.92100000000000004,"5min":1.165,"15min":1.165},"last":0.434}]},{"osd":1,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.93500000000000005,"5min":0.93700000000000006,"15min":1.139},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.333},"last":0.55700000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.01,"15min":1.863},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.75,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93700000000000006,"5min":0.96299999999999997,"15min":1.387},"last":0.50600000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.5,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.88700000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.495},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.51600000000000001,"5min":0.48699999999999999,"15min":0.46600000000000003},"max":{"1min":0.80500000000000005,"5min":1.1539999999999999,"15min":1.905},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.46700000000000003,"5min":0.39300000000000002,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.48099999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.78500000000000003,"5min":1.294,"15min":1.294},"last":0.629}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.48099999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.75900000000000001,"5min":1.0129999999999999,"15min":1.9339999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.66200000000000003},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.433},"max":{"1min":0.85299999999999998,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.48099999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85999999999999999,"5min":1.345,"15min":1.345},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.59099999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89700000000000002,"5min":1.214,"15min":1.97},"last":0.755}]}]},{"osd":5,"up_from":13,"seq":55834575030,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":974848,"data_stored":457634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69599999999999995,"15min":0.68200000000000005},"min":{"1min":0.432,"5min":0.41599999999999998,"15min":0.313},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.746,"15min":0.71899999999999997},"min":{"1min":0.45900000000000002,"5min":0.45500000000000002,"15min":0.33500000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.339},"last":0.80100000000000005}]},{"osd":1,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.55400000000000005,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.0169999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72199999999999998,"15min":0.70499999999999996},"min":{"1min":0.47999999999999998,"5min":0.46700000000000003,"15min":0.39200000000000002},"max":{"1min":0.95199999999999996,"5min":1.022,"15min":1.6040000000000001},"last":1.0669999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73099999999999998,"15min":0.72799999999999998},"min":{"1min":0.57099999999999995,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.476},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.59299999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93700000000000006,"5min":1.095,"15min":1.502},"last":0.81899999999999995}]},{"osd":3,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.63200000000000001,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":0.95799999999999996,"5min":1.113,"15min":1.1619999999999999},"last":1.042},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.754},"min":{"1min":0.56499999999999995,"5min":0.52300000000000002,"15min":0.46500000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.744},"last":0.75900000000000001}]},{"osd":4,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.80300000000000005,"5min":0.94599999999999995,"15min":0.95899999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.214},"last":1.097}]},{"osd":6,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64800000000000002,"15min":0.65400000000000003},"min":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.36799999999999999},"max":{"1min":0.85999999999999999,"5min":0.91300000000000003,"15min":1.151},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.52600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":0.97099999999999997,"15min":1.5489999999999999},"last":0.73699999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.70499999999999996,"15min":0.71099999999999997},"min":{"1min":0.51500000000000001,"5min":0.41899999999999998,"15min":0.40200000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.325},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.53800000000000003,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":0.96999999999999997,"5min":0.98199999999999998,"15min":1.1990000000000001},"last":0.90300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:09:42.084 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:09:42.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.294+0000 7fb64b000700 1 Processor -- start 2024-03-29T10:09:42.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.294+0000 7fb64b000700 1 -- start start 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.295+0000 7fb64b000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6440afbe0 0x7fb64406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.295+0000 7fb64b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6440cebd0 0x7fb6440cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.295+0000 7fb64b000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64406f850 con 0x7fb64406a790 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.295+0000 7fb64b000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64406fad0 con 0x7fb6440afbe0 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.295+0000 7fb64b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440af970 con 0x7fb6440cebd0 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.296+0000 7fb643400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6440afbe0 0x7fb64406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.296+0000 7fb643400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6440afbe0 0x7fb64406dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48040/0 (socket says 172.21.15.12:48040) 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.296+0000 7fb643400700 1 -- 172.21.15.12:0/2795981825 learned_addr learned my addr 172.21.15.12:0/2795981825 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.296+0000 7fb649000700 1 --2- 172.21.15.12:0/2795981825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6440cebd0 0x7fb6440cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.296+0000 7fb643400700 1 -- 172.21.15.12:0/2795981825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6440cebd0 msgr2=0x7fb6440cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.296+0000 7fb643400700 1 --2- 172.21.15.12:0/2795981825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6440cebd0 0x7fb6440cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.296+0000 7fb643400700 1 -- 172.21.15.12:0/2795981825 >> v1:172.21.15.159:6789/0 conn(0x7fb64406a790 legacy=0x7fb64406ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.296+0000 7fb643400700 1 -- 172.21.15.12:0/2795981825 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6440af690 con 0x7fb6440afbe0 2024-03-29T10:09:42.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.296+0000 7fb643400700 1 --2- 172.21.15.12:0/2795981825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6440afbe0 0x7fb64406dce0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb6300098f0 tx=0x7fb630009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68d136b55ba15f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.297+0000 7fb642a00700 1 -- 172.21.15.12:0/2795981825 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63000b8a0 con 0x7fb6440afbe0 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.297+0000 7fb642a00700 1 -- 172.21.15.12:0/2795981825 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63000ba00 con 0x7fb6440afbe0 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.297+0000 7fb642a00700 1 -- 172.21.15.12:0/2795981825 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63000bd90 con 0x7fb6440afbe0 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.297+0000 7fb64b000700 1 -- 172.21.15.12:0/2795981825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6440afbe0 msgr2=0x7fb64406dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.297+0000 7fb64b000700 1 --2- 172.21.15.12:0/2795981825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6440afbe0 0x7fb64406dce0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb6300098f0 tx=0x7fb630009c00 comp rx=0 tx=0).stop 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.297+0000 7fb64b000700 1 -- 172.21.15.12:0/2795981825 shutdown_connections 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.297+0000 7fb64b000700 1 --2- 172.21.15.12:0/2795981825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6440cebd0 0x7fb6440cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.297+0000 7fb64b000700 1 --2- 172.21.15.12:0/2795981825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6440afbe0 0x7fb64406dce0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.297+0000 7fb64b000700 1 -- 172.21.15.12:0/2795981825 >> 172.21.15.12:0/2795981825 conn(0x7fb6440b5ab0 msgr2=0x7fb6440b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.298+0000 7fb64b000700 1 -- 172.21.15.12:0/2795981825 shutdown_connections 2024-03-29T10:09:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.298+0000 7fb64b000700 1 -- 172.21.15.12:0/2795981825 wait complete. 2024-03-29T10:09:42.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.299+0000 7fb64b000700 1 Processor -- start 2024-03-29T10:09:42.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.299+0000 7fb64b000700 1 -- start start 2024-03-29T10:09:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.300+0000 7fb64b000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb64406a790 0x7fb64409aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.300+0000 7fb64b000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6440afbe0 0x7fb644156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.300+0000 7fb64b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6440cebd0 0x7fb6441f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.300+0000 7fb64b000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb644156f90 con 0x7fb64406a790 2024-03-29T10:09:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.300+0000 7fb64b000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6441f7950 con 0x7fb6440afbe0 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.300+0000 7fb64b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64409cb30 con 0x7fb6440cebd0 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb643e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb64406a790 0x7fb64409aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb643e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb64406a790 0x7fb64409aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48054/0 (socket says 172.21.15.12:48054) 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb643e00700 1 -- 172.21.15.12:0/2494481493 learned_addr learned my addr 172.21.15.12:0/2494481493 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb643400700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6440afbe0 0x7fb644156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb649000700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6440cebd0 0x7fb6441f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb643e00700 1 -- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6440cebd0 msgr2=0x7fb6441f7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb643e00700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6440cebd0 0x7fb6441f7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb643e00700 1 -- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6440afbe0 msgr2=0x7fb644156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb643e00700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6440afbe0 0x7fb644156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb643e00700 1 -- 172.21.15.12:0/2494481493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb630009550 con 0x7fb64406a790 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.301+0000 7fb643e00700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb64406a790 0x7fb64409aa50 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb63400c450 tx=0x7fb63400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.302+0000 7fb63be00700 1 -- 172.21.15.12:0/2494481493 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6340052d0 con 0x7fb64406a790 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.302+0000 7fb63be00700 1 -- 172.21.15.12:0/2494481493 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb634005430 con 0x7fb64406a790 2024-03-29T10:09:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.302+0000 7fb63be00700 1 -- 172.21.15.12:0/2494481493 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6340038c0 con 0x7fb64406a790 2024-03-29T10:09:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.302+0000 7fb64b000700 1 -- 172.21.15.12:0/2494481493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64409b0f0 con 0x7fb64406a790 2024-03-29T10:09:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.302+0000 7fb64b000700 1 -- 172.21.15.12:0/2494481493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64409cda0 con 0x7fb64406a790 2024-03-29T10:09:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.303+0000 7fb64b000700 1 -- 172.21.15.12:0/2494481493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64409b610 con 0x7fb64406a790 2024-03-29T10:09:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.304+0000 7fb63be00700 1 -- 172.21.15.12:0/2494481493 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb634003a20 con 0x7fb64406a790 2024-03-29T10:09:42.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.305+0000 7fb63be00700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb62406adb0 0x7fb62406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.305+0000 7fb63be00700 1 -- 172.21.15.12:0/2494481493 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6340887c0 con 0x7fb64406a790 2024-03-29T10:09:42.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.305+0000 7fb643400700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb62406adb0 0x7fb62406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.306+0000 7fb643400700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb62406adb0 0x7fb62406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb630009c40 tx=0x7fb630008030 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:42.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.309+0000 7fb63be00700 1 -- 172.21.15.12:0/2494481493 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb634053880 con 0x7fb64406a790 2024-03-29T10:09:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.517+0000 7fb64b000700 1 -- 172.21.15.12:0/2494481493 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6440cf580 con 0x7fb62406adb0 2024-03-29T10:09:42.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.526+0000 7fb63be00700 1 -- 172.21.15.12:0/2494481493 <== mgr.6013 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289074 (secure 0 0 0) 0x7fb63000bb70 con 0x7fb62406adb0 2024-03-29T10:09:42.530 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 -- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb62406adb0 msgr2=0x7fb62406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb62406adb0 0x7fb62406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb630009c40 tx=0x7fb630008030 comp rx=0 tx=0).stop 2024-03-29T10:09:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 -- 172.21.15.12:0/2494481493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb64406a790 msgr2=0x7fb64409aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb64406a790 0x7fb64409aa50 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb63400c450 tx=0x7fb63400c760 comp rx=0 tx=0).stop 2024-03-29T10:09:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 -- 172.21.15.12:0/2494481493 shutdown_connections 2024-03-29T10:09:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb62406adb0 0x7fb62406d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6440cebd0 0x7fb6441f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb64406a790 0x7fb64409aa50 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 --2- 172.21.15.12:0/2494481493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6440afbe0 0x7fb644156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 -- 172.21.15.12:0/2494481493 >> 172.21.15.12:0/2494481493 conn(0x7fb6440b5ab0 msgr2=0x7fb644159740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 -- 172.21.15.12:0/2494481493 shutdown_connections 2024-03-29T10:09:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.531+0000 7fb64b000700 1 -- 172.21.15.12:0/2494481493 wait complete. 2024-03-29T10:09:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:09:42.549 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-29T10:09:40.839508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":306120,"kb_used_data":8772,"kb_used_omap":12,"kb_used_meta":297203,"kb_avail":754668600,"statfs":{"total":773094113280,"available":772780646400,"internally_reserved":0,"allocated":8982528,"data_stored":4840297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":304336441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.238503"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045584+0000","last_change":"2024-03-29T10:09:32.623236+0000","last_active":"2024-03-29T10:09:34.045584+0000","last_peered":"2024-03-29T10:09:34.045584+0000","last_clean":"2024-03-29T10:09:34.045584+0000","last_became_active":"2024-03-29T10:09:32.622714+0000","last_became_peered":"2024-03-29T10:09:32.622714+0000","last_unstale":"2024-03-29T10:09:34.045584+0000","last_undegraded":"2024-03-29T10:09:34.045584+0000","last_fullsized":"2024-03-29T10:09:34.045584+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:50.745197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607572+0000","last_change":"2024-03-29T10:09:32.620014+0000","last_active":"2024-03-29T10:09:33.607572+0000","last_peered":"2024-03-29T10:09:33.607572+0000","last_clean":"2024-03-29T10:09:33.607572+0000","last_became_active":"2024-03-29T10:09:32.619747+0000","last_became_peered":"2024-03-29T10:09:32.619747+0000","last_unstale":"2024-03-29T10:09:33.607572+0000","last_undegraded":"2024-03-29T10:09:33.607572+0000","last_fullsized":"2024-03-29T10:09:33.607572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:58:19.387293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045306+0000","last_change":"2024-03-29T10:09:32.622331+0000","last_active":"2024-03-29T10:09:34.045306+0000","last_peered":"2024-03-29T10:09:34.045306+0000","last_clean":"2024-03-29T10:09:34.045306+0000","last_became_active":"2024-03-29T10:09:32.622038+0000","last_became_peered":"2024-03-29T10:09:32.622038+0000","last_unstale":"2024-03-29T10:09:34.045306+0000","last_undegraded":"2024-03-29T10:09:34.045306+0000","last_fullsized":"2024-03-29T10:09:34.045306+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:54.413063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613492+0000","last_change":"2024-03-29T10:09:32.622514+0000","last_active":"2024-03-29T10:09:33.613492+0000","last_peered":"2024-03-29T10:09:33.613492+0000","last_clean":"2024-03-29T10:09:33.613492+0000","last_became_active":"2024-03-29T10:09:32.621840+0000","last_became_peered":"2024-03-29T10:09:32.621840+0000","last_unstale":"2024-03-29T10:09:33.613492+0000","last_undegraded":"2024-03-29T10:09:33.613492+0000","last_fullsized":"2024-03-29T10:09:33.613492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:46.872891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045348+0000","last_change":"2024-03-29T10:09:32.617524+0000","last_active":"2024-03-29T10:09:34.045348+0000","last_peered":"2024-03-29T10:09:34.045348+0000","last_clean":"2024-03-29T10:09:34.045348+0000","last_became_active":"2024-03-29T10:09:32.617270+0000","last_became_peered":"2024-03-29T10:09:32.617270+0000","last_unstale":"2024-03-29T10:09:34.045348+0000","last_undegraded":"2024-03-29T10:09:34.045348+0000","last_fullsized":"2024-03-29T10:09:34.045348+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:02.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045118+0000","last_change":"2024-03-29T10:09:32.616253+0000","last_active":"2024-03-29T10:09:34.045118+0000","last_peered":"2024-03-29T10:09:34.045118+0000","last_clean":"2024-03-29T10:09:34.045118+0000","last_became_active":"2024-03-29T10:09:32.616002+0000","last_became_peered":"2024-03-29T10:09:32.616002+0000","last_unstale":"2024-03-29T10:09:34.045118+0000","last_undegraded":"2024-03-29T10:09:34.045118+0000","last_fullsized":"2024-03-29T10:09:34.045118+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:23.297716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044899+0000","last_change":"2024-03-29T10:09:32.612351+0000","last_active":"2024-03-29T10:09:34.044899+0000","last_peered":"2024-03-29T10:09:34.044899+0000","last_clean":"2024-03-29T10:09:34.044899+0000","last_became_active":"2024-03-29T10:09:32.612057+0000","last_became_peered":"2024-03-29T10:09:32.612057+0000","last_unstale":"2024-03-29T10:09:34.044899+0000","last_undegraded":"2024-03-29T10:09:34.044899+0000","last_fullsized":"2024-03-29T10:09:34.044899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:41.040149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.606905+0000","last_change":"2024-03-29T10:09:32.615482+0000","last_active":"2024-03-29T10:09:33.606905+0000","last_peered":"2024-03-29T10:09:33.606905+0000","last_clean":"2024-03-29T10:09:33.606905+0000","last_became_active":"2024-03-29T10:09:32.615198+0000","last_became_peered":"2024-03-29T10:09:32.615198+0000","last_unstale":"2024-03-29T10:09:33.606905+0000","last_undegraded":"2024-03-29T10:09:33.606905+0000","last_fullsized":"2024-03-29T10:09:33.606905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:48:35.803134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045007+0000","last_change":"2024-03-29T10:09:32.611215+0000","last_active":"2024-03-29T10:09:34.045007+0000","last_peered":"2024-03-29T10:09:34.045007+0000","last_clean":"2024-03-29T10:09:34.045007+0000","last_became_active":"2024-03-29T10:09:32.610941+0000","last_became_peered":"2024-03-29T10:09:32.610941+0000","last_unstale":"2024-03-29T10:09:34.045007+0000","last_undegraded":"2024-03-29T10:09:34.045007+0000","last_fullsized":"2024-03-29T10:09:34.045007+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:28.376324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045092+0000","last_change":"2024-03-29T10:09:32.624355+0000","last_active":"2024-03-29T10:09:34.045092+0000","last_peered":"2024-03-29T10:09:34.045092+0000","last_clean":"2024-03-29T10:09:34.045092+0000","last_became_active":"2024-03-29T10:09:32.622547+0000","last_became_peered":"2024-03-29T10:09:32.622547+0000","last_unstale":"2024-03-29T10:09:34.045092+0000","last_undegraded":"2024-03-29T10:09:34.045092+0000","last_fullsized":"2024-03-29T10:09:34.045092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:02.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044868+0000","last_change":"2024-03-29T10:09:32.628082+0000","last_active":"2024-03-29T10:09:34.044868+0000","last_peered":"2024-03-29T10:09:34.044868+0000","last_clean":"2024-03-29T10:09:34.044868+0000","last_became_active":"2024-03-29T10:09:32.627765+0000","last_became_peered":"2024-03-29T10:09:32.627765+0000","last_unstale":"2024-03-29T10:09:34.044868+0000","last_undegraded":"2024-03-29T10:09:34.044868+0000","last_fullsized":"2024-03-29T10:09:34.044868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:50.472261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613633+0000","last_change":"2024-03-29T10:09:32.622558+0000","last_active":"2024-03-29T10:09:33.613633+0000","last_peered":"2024-03-29T10:09:33.613633+0000","last_clean":"2024-03-29T10:09:33.613633+0000","last_became_active":"2024-03-29T10:09:32.622241+0000","last_became_peered":"2024-03-29T10:09:32.622241+0000","last_unstale":"2024-03-29T10:09:33.613633+0000","last_undegraded":"2024-03-29T10:09:33.613633+0000","last_fullsized":"2024-03-29T10:09:33.613633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:59.825105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.612120+0000","last_change":"2024-03-29T10:09:32.619242+0000","last_active":"2024-03-29T10:09:33.612120+0000","last_peered":"2024-03-29T10:09:33.612120+0000","last_clean":"2024-03-29T10:09:33.612120+0000","last_became_active":"2024-03-29T10:09:32.618883+0000","last_became_peered":"2024-03-29T10:09:32.618883+0000","last_unstale":"2024-03-29T10:09:33.612120+0000","last_undegraded":"2024-03-29T10:09:33.612120+0000","last_fullsized":"2024-03-29T10:09:33.612120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:08.395520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044873+0000","last_change":"2024-03-29T10:09:32.617701+0000","last_active":"2024-03-29T10:09:34.044873+0000","last_peered":"2024-03-29T10:09:34.044873+0000","last_clean":"2024-03-29T10:09:34.044873+0000","last_became_active":"2024-03-29T10:09:32.616607+0000","last_became_peered":"2024-03-29T10:09:32.616607+0000","last_unstale":"2024-03-29T10:09:34.044873+0000","last_undegraded":"2024-03-29T10:09:34.044873+0000","last_fullsized":"2024-03-29T10:09:34.044873+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:39.250301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045466+0000","last_change":"2024-03-29T10:09:32.618041+0000","last_active":"2024-03-29T10:09:34.045466+0000","last_peered":"2024-03-29T10:09:34.045466+0000","last_clean":"2024-03-29T10:09:34.045466+0000","last_became_active":"2024-03-29T10:09:32.617123+0000","last_became_peered":"2024-03-29T10:09:32.617123+0000","last_unstale":"2024-03-29T10:09:34.045466+0000","last_undegraded":"2024-03-29T10:09:34.045466+0000","last_fullsized":"2024-03-29T10:09:34.045466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:47.161213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044987+0000","last_change":"2024-03-29T10:09:32.622774+0000","last_active":"2024-03-29T10:09:34.044987+0000","last_peered":"2024-03-29T10:09:34.044987+0000","last_clean":"2024-03-29T10:09:34.044987+0000","last_became_active":"2024-03-29T10:09:32.622503+0000","last_became_peered":"2024-03-29T10:09:32.622503+0000","last_unstale":"2024-03-29T10:09:34.044987+0000","last_undegraded":"2024-03-29T10:09:34.044987+0000","last_fullsized":"2024-03-29T10:09:34.044987+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:38.603035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607446+0000","last_change":"2024-03-29T10:09:32.625499+0000","last_active":"2024-03-29T10:09:33.607446+0000","last_peered":"2024-03-29T10:09:33.607446+0000","last_clean":"2024-03-29T10:09:33.607446+0000","last_became_active":"2024-03-29T10:09:32.625200+0000","last_became_peered":"2024-03-29T10:09:32.625200+0000","last_unstale":"2024-03-29T10:09:33.607446+0000","last_undegraded":"2024-03-29T10:09:33.607446+0000","last_fullsized":"2024-03-29T10:09:33.607446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:08:47.193288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045942+0000","last_change":"2024-03-29T10:09:32.622371+0000","last_active":"2024-03-29T10:09:34.045942+0000","last_peered":"2024-03-29T10:09:34.045942+0000","last_clean":"2024-03-29T10:09:34.045942+0000","last_became_active":"2024-03-29T10:09:32.621892+0000","last_became_peered":"2024-03-29T10:09:32.621892+0000","last_unstale":"2024-03-29T10:09:34.045942+0000","last_undegraded":"2024-03-29T10:09:34.045942+0000","last_fullsized":"2024-03-29T10:09:34.045942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:50.946405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045592+0000","last_change":"2024-03-29T10:09:32.615044+0000","last_active":"2024-03-29T10:09:34.045592+0000","last_peered":"2024-03-29T10:09:34.045592+0000","last_clean":"2024-03-29T10:09:34.045592+0000","last_became_active":"2024-03-29T10:09:32.614754+0000","last_became_peered":"2024-03-29T10:09:32.614754+0000","last_unstale":"2024-03-29T10:09:34.045592+0000","last_undegraded":"2024-03-29T10:09:34.045592+0000","last_fullsized":"2024-03-29T10:09:34.045592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:50.472261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607170+0000","last_change":"2024-03-29T10:09:32.613680+0000","last_active":"2024-03-29T10:09:33.607170+0000","last_peered":"2024-03-29T10:09:33.607170+0000","last_clean":"2024-03-29T10:09:33.607170+0000","last_became_active":"2024-03-29T10:09:32.611800+0000","last_became_peered":"2024-03-29T10:09:32.611800+0000","last_unstale":"2024-03-29T10:09:33.607170+0000","last_undegraded":"2024-03-29T10:09:33.607170+0000","last_fullsized":"2024-03-29T10:09:33.607170+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:04.582194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045761+0000","last_change":"2024-03-29T10:09:32.623271+0000","last_active":"2024-03-29T10:09:34.045761+0000","last_peered":"2024-03-29T10:09:34.045761+0000","last_clean":"2024-03-29T10:09:34.045761+0000","last_became_active":"2024-03-29T10:09:32.622928+0000","last_became_peered":"2024-03-29T10:09:32.622928+0000","last_unstale":"2024-03-29T10:09:34.045761+0000","last_undegraded":"2024-03-29T10:09:34.045761+0000","last_fullsized":"2024-03-29T10:09:34.045761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.930784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607046+0000","last_change":"2024-03-29T10:09:32.621045+0000","last_active":"2024-03-29T10:09:33.607046+0000","last_peered":"2024-03-29T10:09:33.607046+0000","last_clean":"2024-03-29T10:09:33.607046+0000","last_became_active":"2024-03-29T10:09:32.620499+0000","last_became_peered":"2024-03-29T10:09:32.620499+0000","last_unstale":"2024-03-29T10:09:33.607046+0000","last_undegraded":"2024-03-29T10:09:33.607046+0000","last_fullsized":"2024-03-29T10:09:33.607046+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:05:04.665074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045437+0000","last_change":"2024-03-29T10:09:32.637105+0000","last_active":"2024-03-29T10:09:34.045437+0000","last_peered":"2024-03-29T10:09:34.045437+0000","last_clean":"2024-03-29T10:09:34.045437+0000","last_became_active":"2024-03-29T10:09:32.636599+0000","last_became_peered":"2024-03-29T10:09:32.636599+0000","last_unstale":"2024-03-29T10:09:34.045437+0000","last_undegraded":"2024-03-29T10:09:34.045437+0000","last_fullsized":"2024-03-29T10:09:34.045437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:54.720344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.611723+0000","last_change":"2024-03-29T10:09:32.628611+0000","last_active":"2024-03-29T10:09:33.611723+0000","last_peered":"2024-03-29T10:09:33.611723+0000","last_clean":"2024-03-29T10:09:33.611723+0000","last_became_active":"2024-03-29T10:09:32.625523+0000","last_became_peered":"2024-03-29T10:09:32.625523+0000","last_unstale":"2024-03-29T10:09:33.611723+0000","last_undegraded":"2024-03-29T10:09:33.611723+0000","last_fullsized":"2024-03-29T10:09:33.611723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:56.014772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046009+0000","last_change":"2024-03-29T10:09:32.615838+0000","last_active":"2024-03-29T10:09:34.046009+0000","last_peered":"2024-03-29T10:09:34.046009+0000","last_clean":"2024-03-29T10:09:34.046009+0000","last_became_active":"2024-03-29T10:09:32.615274+0000","last_became_peered":"2024-03-29T10:09:32.615274+0000","last_unstale":"2024-03-29T10:09:34.046009+0000","last_undegraded":"2024-03-29T10:09:34.046009+0000","last_fullsized":"2024-03-29T10:09:34.046009+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:07.635360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613549+0000","last_change":"2024-03-29T10:09:32.626084+0000","last_active":"2024-03-29T10:09:33.613549+0000","last_peered":"2024-03-29T10:09:33.613549+0000","last_clean":"2024-03-29T10:09:33.613549+0000","last_became_active":"2024-03-29T10:09:32.625773+0000","last_became_peered":"2024-03-29T10:09:32.625773+0000","last_unstale":"2024-03-29T10:09:33.613549+0000","last_undegraded":"2024-03-29T10:09:33.613549+0000","last_fullsized":"2024-03-29T10:09:33.613549+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:39.250301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045741+0000","last_change":"2024-03-29T10:09:32.624374+0000","last_active":"2024-03-29T10:09:34.045741+0000","last_peered":"2024-03-29T10:09:34.045741+0000","last_clean":"2024-03-29T10:09:34.045741+0000","last_became_active":"2024-03-29T10:09:32.624101+0000","last_became_peered":"2024-03-29T10:09:32.624101+0000","last_unstale":"2024-03-29T10:09:34.045741+0000","last_undegraded":"2024-03-29T10:09:34.045741+0000","last_fullsized":"2024-03-29T10:09:34.045741+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:24:39.560229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045630+0000","last_change":"2024-03-29T10:09:32.618810+0000","last_active":"2024-03-29T10:09:34.045630+0000","last_peered":"2024-03-29T10:09:34.045630+0000","last_clean":"2024-03-29T10:09:34.045630+0000","last_became_active":"2024-03-29T10:09:32.618546+0000","last_became_peered":"2024-03-29T10:09:32.618546+0000","last_unstale":"2024-03-29T10:09:34.045630+0000","last_undegraded":"2024-03-29T10:09:34.045630+0000","last_fullsized":"2024-03-29T10:09:34.045630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:49.874012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607130+0000","last_change":"2024-03-29T10:09:32.618824+0000","last_active":"2024-03-29T10:09:33.607130+0000","last_peered":"2024-03-29T10:09:33.607130+0000","last_clean":"2024-03-29T10:09:33.607130+0000","last_became_active":"2024-03-29T10:09:32.618065+0000","last_became_peered":"2024-03-29T10:09:32.618065+0000","last_unstale":"2024-03-29T10:09:33.607130+0000","last_undegraded":"2024-03-29T10:09:33.607130+0000","last_fullsized":"2024-03-29T10:09:33.607130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:29.544771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045633+0000","last_change":"2024-03-29T10:09:32.622999+0000","last_active":"2024-03-29T10:09:34.045633+0000","last_peered":"2024-03-29T10:09:34.045633+0000","last_clean":"2024-03-29T10:09:34.045633+0000","last_became_active":"2024-03-29T10:09:32.622065+0000","last_became_peered":"2024-03-29T10:09:32.622065+0000","last_unstale":"2024-03-29T10:09:34.045633+0000","last_undegraded":"2024-03-29T10:09:34.045633+0000","last_fullsized":"2024-03-29T10:09:34.045633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:23.297716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044843+0000","last_change":"2024-03-29T10:09:32.615373+0000","last_active":"2024-03-29T10:09:34.044843+0000","last_peered":"2024-03-29T10:09:34.044843+0000","last_clean":"2024-03-29T10:09:34.044843+0000","last_became_active":"2024-03-29T10:09:32.615107+0000","last_became_peered":"2024-03-29T10:09:32.615107+0000","last_unstale":"2024-03-29T10:09:34.044843+0000","last_undegraded":"2024-03-29T10:09:34.044843+0000","last_fullsized":"2024-03-29T10:09:34.044843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:23.129448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045322+0000","last_change":"2024-03-29T10:09:32.618453+0000","last_active":"2024-03-29T10:09:34.045322+0000","last_peered":"2024-03-29T10:09:34.045322+0000","last_clean":"2024-03-29T10:09:34.045322+0000","last_became_active":"2024-03-29T10:09:32.616759+0000","last_became_peered":"2024-03-29T10:09:32.616759+0000","last_unstale":"2024-03-29T10:09:34.045322+0000","last_undegraded":"2024-03-29T10:09:34.045322+0000","last_fullsized":"2024-03-29T10:09:34.045322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:50.946405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045802+0000","last_change":"2024-03-29T10:09:32.623841+0000","last_active":"2024-03-29T10:09:34.045802+0000","last_peered":"2024-03-29T10:09:34.045802+0000","last_clean":"2024-03-29T10:09:34.045802+0000","last_became_active":"2024-03-29T10:09:32.623522+0000","last_became_peered":"2024-03-29T10:09:32.623522+0000","last_unstale":"2024-03-29T10:09:34.045802+0000","last_undegraded":"2024-03-29T10:09:34.045802+0000","last_fullsized":"2024-03-29T10:09:34.045802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:40.815771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"79'4","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638507+0000","last_change":"2024-03-29T10:09:32.101864+0000","last_active":"2024-03-29T10:09:32.638507+0000","last_peered":"2024-03-29T10:09:32.638507+0000","last_clean":"2024-03-29T10:09:32.638507+0000","last_became_active":"2024-03-29T10:09:32.101456+0000","last_became_peered":"2024-03-29T10:09:32.101456+0000","last_unstale":"2024-03-29T10:09:32.638507+0000","last_undegraded":"2024-03-29T10:09:32.638507+0000","last_fullsized":"2024-03-29T10:09:32.638507+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:05.714596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045625+0000","last_change":"2024-03-29T10:09:32.630347+0000","last_active":"2024-03-29T10:09:34.045625+0000","last_peered":"2024-03-29T10:09:34.045625+0000","last_clean":"2024-03-29T10:09:34.045625+0000","last_became_active":"2024-03-29T10:09:32.630092+0000","last_became_peered":"2024-03-29T10:09:32.630092+0000","last_unstale":"2024-03-29T10:09:34.045625+0000","last_undegraded":"2024-03-29T10:09:34.045625+0000","last_fullsized":"2024-03-29T10:09:34.045625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:50.745197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.635936+0000","last_change":"2024-03-29T10:09:32.091373+0000","last_active":"2024-03-29T10:09:32.635936+0000","last_peered":"2024-03-29T10:09:32.635936+0000","last_clean":"2024-03-29T10:09:32.635936+0000","last_became_active":"2024-03-29T10:09:32.091110+0000","last_became_peered":"2024-03-29T10:09:32.091110+0000","last_unstale":"2024-03-29T10:09:32.635936+0000","last_undegraded":"2024-03-29T10:09:32.635936+0000","last_fullsized":"2024-03-29T10:09:32.635936+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:32:31.535691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046042+0000","last_change":"2024-03-29T10:09:32.614418+0000","last_active":"2024-03-29T10:09:34.046042+0000","last_peered":"2024-03-29T10:09:34.046042+0000","last_clean":"2024-03-29T10:09:34.046042+0000","last_became_active":"2024-03-29T10:09:32.614112+0000","last_became_peered":"2024-03-29T10:09:32.614112+0000","last_unstale":"2024-03-29T10:09:34.046042+0000","last_undegraded":"2024-03-29T10:09:34.046042+0000","last_fullsized":"2024-03-29T10:09:34.046042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:17.345552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.635794+0000","last_change":"2024-03-29T10:09:32.086998+0000","last_active":"2024-03-29T10:09:32.635794+0000","last_peered":"2024-03-29T10:09:32.635794+0000","last_clean":"2024-03-29T10:09:32.635794+0000","last_became_active":"2024-03-29T10:09:32.086644+0000","last_became_peered":"2024-03-29T10:09:32.086644+0000","last_unstale":"2024-03-29T10:09:32.635794+0000","last_undegraded":"2024-03-29T10:09:32.635794+0000","last_fullsized":"2024-03-29T10:09:32.635794+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:27.370099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045835+0000","last_change":"2024-03-29T10:09:32.624387+0000","last_active":"2024-03-29T10:09:34.045835+0000","last_peered":"2024-03-29T10:09:34.045835+0000","last_clean":"2024-03-29T10:09:34.045835+0000","last_became_active":"2024-03-29T10:09:32.624015+0000","last_became_peered":"2024-03-29T10:09:32.624015+0000","last_unstale":"2024-03-29T10:09:34.045835+0000","last_undegraded":"2024-03-29T10:09:34.045835+0000","last_fullsized":"2024-03-29T10:09:34.045835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:06.669665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.637825+0000","last_change":"2024-03-29T10:09:32.101441+0000","last_active":"2024-03-29T10:09:32.637825+0000","last_peered":"2024-03-29T10:09:32.637825+0000","last_clean":"2024-03-29T10:09:32.637825+0000","last_became_active":"2024-03-29T10:09:32.101114+0000","last_became_peered":"2024-03-29T10:09:32.101114+0000","last_unstale":"2024-03-29T10:09:32.637825+0000","last_undegraded":"2024-03-29T10:09:32.637825+0000","last_fullsized":"2024-03-29T10:09:32.637825+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607151+0000","last_change":"2024-03-29T10:09:32.620748+0000","last_active":"2024-03-29T10:09:33.607151+0000","last_peered":"2024-03-29T10:09:33.607151+0000","last_clean":"2024-03-29T10:09:33.607151+0000","last_became_active":"2024-03-29T10:09:32.618821+0000","last_became_peered":"2024-03-29T10:09:32.618821+0000","last_unstale":"2024-03-29T10:09:33.607151+0000","last_undegraded":"2024-03-29T10:09:33.607151+0000","last_fullsized":"2024-03-29T10:09:33.607151+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:39:35.761571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.632616+0000","last_change":"2024-03-29T10:09:32.098141+0000","last_active":"2024-03-29T10:09:32.632616+0000","last_peered":"2024-03-29T10:09:32.632616+0000","last_clean":"2024-03-29T10:09:32.632616+0000","last_became_active":"2024-03-29T10:09:32.097781+0000","last_became_peered":"2024-03-29T10:09:32.097781+0000","last_unstale":"2024-03-29T10:09:32.632616+0000","last_undegraded":"2024-03-29T10:09:32.632616+0000","last_fullsized":"2024-03-29T10:09:32.632616+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:07.389295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607233+0000","last_change":"2024-03-29T10:09:32.614343+0000","last_active":"2024-03-29T10:09:33.607233+0000","last_peered":"2024-03-29T10:09:33.607233+0000","last_clean":"2024-03-29T10:09:33.607233+0000","last_became_active":"2024-03-29T10:09:32.614069+0000","last_became_peered":"2024-03-29T10:09:32.614069+0000","last_unstale":"2024-03-29T10:09:33.607233+0000","last_undegraded":"2024-03-29T10:09:33.607233+0000","last_fullsized":"2024-03-29T10:09:33.607233+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:22.440684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638211+0000","last_change":"2024-03-29T10:09:32.101572+0000","last_active":"2024-03-29T10:09:32.638211+0000","last_peered":"2024-03-29T10:09:32.638211+0000","last_clean":"2024-03-29T10:09:32.638211+0000","last_became_active":"2024-03-29T10:09:32.101206+0000","last_became_peered":"2024-03-29T10:09:32.101206+0000","last_unstale":"2024-03-29T10:09:32.638211+0000","last_undegraded":"2024-03-29T10:09:32.638211+0000","last_fullsized":"2024-03-29T10:09:32.638211+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:46.154988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.611858+0000","last_change":"2024-03-29T10:09:32.624716+0000","last_active":"2024-03-29T10:09:33.611858+0000","last_peered":"2024-03-29T10:09:33.611858+0000","last_clean":"2024-03-29T10:09:33.611858+0000","last_became_active":"2024-03-29T10:09:32.622676+0000","last_became_peered":"2024-03-29T10:09:32.622676+0000","last_unstale":"2024-03-29T10:09:33.611858+0000","last_undegraded":"2024-03-29T10:09:33.611858+0000","last_fullsized":"2024-03-29T10:09:33.611858+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:20.681595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.610991+0000","last_change":"2024-03-29T10:09:32.091482+0000","last_active":"2024-03-29T10:09:32.610991+0000","last_peered":"2024-03-29T10:09:32.610991+0000","last_clean":"2024-03-29T10:09:32.610991+0000","last_became_active":"2024-03-29T10:09:32.091189+0000","last_became_peered":"2024-03-29T10:09:32.091189+0000","last_unstale":"2024-03-29T10:09:32.610991+0000","last_undegraded":"2024-03-29T10:09:32.610991+0000","last_fullsized":"2024-03-29T10:09:32.610991+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607448+0000","last_change":"2024-03-29T10:09:32.634701+0000","last_active":"2024-03-29T10:09:33.607448+0000","last_peered":"2024-03-29T10:09:33.607448+0000","last_clean":"2024-03-29T10:09:33.607448+0000","last_became_active":"2024-03-29T10:09:32.631830+0000","last_became_peered":"2024-03-29T10:09:32.631830+0000","last_unstale":"2024-03-29T10:09:33.607448+0000","last_undegraded":"2024-03-29T10:09:33.607448+0000","last_fullsized":"2024-03-29T10:09:33.607448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:29.066092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.603049+0000","last_change":"2024-03-29T10:09:32.097646+0000","last_active":"2024-03-29T10:09:32.603049+0000","last_peered":"2024-03-29T10:09:32.603049+0000","last_clean":"2024-03-29T10:09:32.603049+0000","last_became_active":"2024-03-29T10:09:32.097364+0000","last_became_peered":"2024-03-29T10:09:32.097364+0000","last_unstale":"2024-03-29T10:09:32.603049+0000","last_undegraded":"2024-03-29T10:09:32.603049+0000","last_fullsized":"2024-03-29T10:09:32.603049+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:14:55.008547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045478+0000","last_change":"2024-03-29T10:09:32.617841+0000","last_active":"2024-03-29T10:09:34.045478+0000","last_peered":"2024-03-29T10:09:34.045478+0000","last_clean":"2024-03-29T10:09:34.045478+0000","last_became_active":"2024-03-29T10:09:32.616581+0000","last_became_peered":"2024-03-29T10:09:32.616581+0000","last_unstale":"2024-03-29T10:09:34.045478+0000","last_undegraded":"2024-03-29T10:09:34.045478+0000","last_fullsized":"2024-03-29T10:09:34.045478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:15.353895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.616868+0000","last_change":"2024-03-29T10:09:32.098837+0000","last_active":"2024-03-29T10:09:32.616868+0000","last_peered":"2024-03-29T10:09:32.616868+0000","last_clean":"2024-03-29T10:09:32.616868+0000","last_became_active":"2024-03-29T10:09:32.098517+0000","last_became_peered":"2024-03-29T10:09:32.098517+0000","last_unstale":"2024-03-29T10:09:32.616868+0000","last_undegraded":"2024-03-29T10:09:32.616868+0000","last_fullsized":"2024-03-29T10:09:32.616868+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:05.714596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045923+0000","last_change":"2024-03-29T10:09:32.636148+0000","last_active":"2024-03-29T10:09:34.045923+0000","last_peered":"2024-03-29T10:09:34.045923+0000","last_clean":"2024-03-29T10:09:34.045923+0000","last_became_active":"2024-03-29T10:09:32.635753+0000","last_became_peered":"2024-03-29T10:09:32.635753+0000","last_unstale":"2024-03-29T10:09:34.045923+0000","last_undegraded":"2024-03-29T10:09:34.045923+0000","last_fullsized":"2024-03-29T10:09:34.045923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:18.667325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.608025+0000","last_change":"2024-03-29T10:09:32.099480+0000","last_active":"2024-03-29T10:09:32.608025+0000","last_peered":"2024-03-29T10:09:32.608025+0000","last_clean":"2024-03-29T10:09:32.608025+0000","last_became_active":"2024-03-29T10:09:32.099133+0000","last_became_peered":"2024-03-29T10:09:32.099133+0000","last_unstale":"2024-03-29T10:09:32.608025+0000","last_undegraded":"2024-03-29T10:09:32.608025+0000","last_fullsized":"2024-03-29T10:09:32.608025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:13:16.625121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044664+0000","last_change":"2024-03-29T10:09:32.628412+0000","last_active":"2024-03-29T10:09:34.044664+0000","last_peered":"2024-03-29T10:09:34.044664+0000","last_clean":"2024-03-29T10:09:34.044664+0000","last_became_active":"2024-03-29T10:09:32.622661+0000","last_became_peered":"2024-03-29T10:09:32.622661+0000","last_unstale":"2024-03-29T10:09:34.044664+0000","last_undegraded":"2024-03-29T10:09:34.044664+0000","last_fullsized":"2024-03-29T10:09:34.044664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:23.129448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"79'3","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636825+0000","last_change":"2024-03-29T10:09:32.089351+0000","last_active":"2024-03-29T10:09:32.636825+0000","last_peered":"2024-03-29T10:09:32.636825+0000","last_clean":"2024-03-29T10:09:32.636825+0000","last_became_active":"2024-03-29T10:09:32.087924+0000","last_became_peered":"2024-03-29T10:09:32.087924+0000","last_unstale":"2024-03-29T10:09:32.636825+0000","last_undegraded":"2024-03-29T10:09:32.636825+0000","last_fullsized":"2024-03-29T10:09:32.636825+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:39:46.154988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045696+0000","last_change":"2024-03-29T10:09:32.632023+0000","last_active":"2024-03-29T10:09:34.045696+0000","last_peered":"2024-03-29T10:09:34.045696+0000","last_clean":"2024-03-29T10:09:34.045696+0000","last_became_active":"2024-03-29T10:09:32.631450+0000","last_became_peered":"2024-03-29T10:09:32.631450+0000","last_unstale":"2024-03-29T10:09:34.045696+0000","last_undegraded":"2024-03-29T10:09:34.045696+0000","last_fullsized":"2024-03-29T10:09:34.045696+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:24:39.560229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"79'6","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.638720+0000","last_change":"2024-03-29T10:09:32.099892+0000","last_active":"2024-03-29T10:09:32.638720+0000","last_peered":"2024-03-29T10:09:32.638720+0000","last_clean":"2024-03-29T10:09:32.638720+0000","last_became_active":"2024-03-29T10:09:32.099626+0000","last_became_peered":"2024-03-29T10:09:32.099626+0000","last_unstale":"2024-03-29T10:09:32.638720+0000","last_undegraded":"2024-03-29T10:09:32.638720+0000","last_fullsized":"2024-03-29T10:09:32.638720+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613351+0000","last_change":"2024-03-29T10:09:32.626554+0000","last_active":"2024-03-29T10:09:33.613351+0000","last_peered":"2024-03-29T10:09:33.613351+0000","last_clean":"2024-03-29T10:09:33.613351+0000","last_became_active":"2024-03-29T10:09:32.625988+0000","last_became_peered":"2024-03-29T10:09:32.625988+0000","last_unstale":"2024-03-29T10:09:33.613351+0000","last_undegraded":"2024-03-29T10:09:33.613351+0000","last_fullsized":"2024-03-29T10:09:33.613351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:10:07.105194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636457+0000","last_change":"2024-03-29T10:09:32.100281+0000","last_active":"2024-03-29T10:09:32.636457+0000","last_peered":"2024-03-29T10:09:32.636457+0000","last_clean":"2024-03-29T10:09:32.636457+0000","last_became_active":"2024-03-29T10:09:32.100004+0000","last_became_peered":"2024-03-29T10:09:32.100004+0000","last_unstale":"2024-03-29T10:09:32.636457+0000","last_undegraded":"2024-03-29T10:09:32.636457+0000","last_fullsized":"2024-03-29T10:09:32.636457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:49.940180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045675+0000","last_change":"2024-03-29T10:09:32.624300+0000","last_active":"2024-03-29T10:09:34.045675+0000","last_peered":"2024-03-29T10:09:34.045675+0000","last_clean":"2024-03-29T10:09:34.045675+0000","last_became_active":"2024-03-29T10:09:32.624004+0000","last_became_peered":"2024-03-29T10:09:32.624004+0000","last_unstale":"2024-03-29T10:09:34.045675+0000","last_undegraded":"2024-03-29T10:09:34.045675+0000","last_fullsized":"2024-03-29T10:09:34.045675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:38.603035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607519+0000","last_change":"2024-03-29T10:09:32.091147+0000","last_active":"2024-03-29T10:09:32.607519+0000","last_peered":"2024-03-29T10:09:32.607519+0000","last_clean":"2024-03-29T10:09:32.607519+0000","last_became_active":"2024-03-29T10:09:32.089656+0000","last_became_peered":"2024-03-29T10:09:32.089656+0000","last_unstale":"2024-03-29T10:09:32.607519+0000","last_undegraded":"2024-03-29T10:09:32.607519+0000","last_fullsized":"2024-03-29T10:09:32.607519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:41:22.123223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044770+0000","last_change":"2024-03-29T10:09:32.623116+0000","last_active":"2024-03-29T10:09:34.044770+0000","last_peered":"2024-03-29T10:09:34.044770+0000","last_clean":"2024-03-29T10:09:34.044770+0000","last_became_active":"2024-03-29T10:09:32.622832+0000","last_became_peered":"2024-03-29T10:09:32.622832+0000","last_unstale":"2024-03-29T10:09:34.044770+0000","last_undegraded":"2024-03-29T10:09:34.044770+0000","last_fullsized":"2024-03-29T10:09:34.044770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:36.451327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.611108+0000","last_change":"2024-03-29T10:09:32.093865+0000","last_active":"2024-03-29T10:09:32.611108+0000","last_peered":"2024-03-29T10:09:32.611108+0000","last_clean":"2024-03-29T10:09:32.611108+0000","last_became_active":"2024-03-29T10:09:32.092709+0000","last_became_peered":"2024-03-29T10:09:32.092709+0000","last_unstale":"2024-03-29T10:09:32.611108+0000","last_undegraded":"2024-03-29T10:09:32.611108+0000","last_fullsized":"2024-03-29T10:09:32.611108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:19.675370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045252+0000","last_change":"2024-03-29T10:09:32.634186+0000","last_active":"2024-03-29T10:09:34.045252+0000","last_peered":"2024-03-29T10:09:34.045252+0000","last_clean":"2024-03-29T10:09:34.045252+0000","last_became_active":"2024-03-29T10:09:32.633693+0000","last_became_peered":"2024-03-29T10:09:32.633693+0000","last_unstale":"2024-03-29T10:09:34.045252+0000","last_undegraded":"2024-03-29T10:09:34.045252+0000","last_fullsized":"2024-03-29T10:09:34.045252+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:40.815771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"79'8","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.639114+0000","last_change":"2024-03-29T10:09:32.092854+0000","last_active":"2024-03-29T10:09:32.639114+0000","last_peered":"2024-03-29T10:09:32.639114+0000","last_clean":"2024-03-29T10:09:32.639114+0000","last_became_active":"2024-03-29T10:09:32.092591+0000","last_became_peered":"2024-03-29T10:09:32.092591+0000","last_unstale":"2024-03-29T10:09:32.639114+0000","last_undegraded":"2024-03-29T10:09:32.639114+0000","last_fullsized":"2024-03-29T10:09:32.639114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:40.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045937+0000","last_change":"2024-03-29T10:09:32.623867+0000","last_active":"2024-03-29T10:09:34.045937+0000","last_peered":"2024-03-29T10:09:34.045937+0000","last_clean":"2024-03-29T10:09:34.045937+0000","last_became_active":"2024-03-29T10:09:32.622284+0000","last_became_peered":"2024-03-29T10:09:32.622284+0000","last_unstale":"2024-03-29T10:09:34.045937+0000","last_undegraded":"2024-03-29T10:09:34.045937+0000","last_fullsized":"2024-03-29T10:09:34.045937+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:07:06.669665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636096+0000","last_change":"2024-03-29T10:09:32.097805+0000","last_active":"2024-03-29T10:09:32.636096+0000","last_peered":"2024-03-29T10:09:32.636096+0000","last_clean":"2024-03-29T10:09:32.636096+0000","last_became_active":"2024-03-29T10:09:32.097537+0000","last_became_peered":"2024-03-29T10:09:32.097537+0000","last_unstale":"2024-03-29T10:09:32.636096+0000","last_undegraded":"2024-03-29T10:09:32.636096+0000","last_fullsized":"2024-03-29T10:09:32.636096+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:35.016753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613614+0000","last_change":"2024-03-29T10:09:32.612197+0000","last_active":"2024-03-29T10:09:33.613614+0000","last_peered":"2024-03-29T10:09:33.613614+0000","last_clean":"2024-03-29T10:09:33.613614+0000","last_became_active":"2024-03-29T10:09:32.611918+0000","last_became_peered":"2024-03-29T10:09:32.611918+0000","last_unstale":"2024-03-29T10:09:33.613614+0000","last_undegraded":"2024-03-29T10:09:33.613614+0000","last_fullsized":"2024-03-29T10:09:33.613614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:32:07.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.634994+0000","last_change":"2024-03-29T10:09:32.104998+0000","last_active":"2024-03-29T10:09:32.634994+0000","last_peered":"2024-03-29T10:09:32.634994+0000","last_clean":"2024-03-29T10:09:32.634994+0000","last_became_active":"2024-03-29T10:09:32.104610+0000","last_became_peered":"2024-03-29T10:09:32.104610+0000","last_unstale":"2024-03-29T10:09:32.634994+0000","last_undegraded":"2024-03-29T10:09:32.634994+0000","last_fullsized":"2024-03-29T10:09:32.634994+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:38.244076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607169+0000","last_change":"2024-03-29T10:09:32.619003+0000","last_active":"2024-03-29T10:09:33.607169+0000","last_peered":"2024-03-29T10:09:33.607169+0000","last_clean":"2024-03-29T10:09:33.607169+0000","last_became_active":"2024-03-29T10:09:32.617951+0000","last_became_peered":"2024-03-29T10:09:32.617951+0000","last_unstale":"2024-03-29T10:09:33.607169+0000","last_undegraded":"2024-03-29T10:09:33.607169+0000","last_fullsized":"2024-03-29T10:09:33.607169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:00.709669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.637733+0000","last_change":"2024-03-29T10:09:32.095098+0000","last_active":"2024-03-29T10:09:32.637733+0000","last_peered":"2024-03-29T10:09:32.637733+0000","last_clean":"2024-03-29T10:09:32.637733+0000","last_became_active":"2024-03-29T10:09:32.093937+0000","last_became_peered":"2024-03-29T10:09:32.093937+0000","last_unstale":"2024-03-29T10:09:32.637733+0000","last_undegraded":"2024-03-29T10:09:32.637733+0000","last_fullsized":"2024-03-29T10:09:32.637733+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:27:35.691958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045708+0000","last_change":"2024-03-29T10:09:32.633244+0000","last_active":"2024-03-29T10:09:34.045708+0000","last_peered":"2024-03-29T10:09:34.045708+0000","last_clean":"2024-03-29T10:09:34.045708+0000","last_became_active":"2024-03-29T10:09:32.632943+0000","last_became_peered":"2024-03-29T10:09:32.632943+0000","last_unstale":"2024-03-29T10:09:34.045708+0000","last_undegraded":"2024-03-29T10:09:34.045708+0000","last_fullsized":"2024-03-29T10:09:34.045708+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:36.022978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.606805+0000","last_change":"2024-03-29T10:09:32.089495+0000","last_active":"2024-03-29T10:09:32.606805+0000","last_peered":"2024-03-29T10:09:32.606805+0000","last_clean":"2024-03-29T10:09:32.606805+0000","last_became_active":"2024-03-29T10:09:32.089228+0000","last_became_peered":"2024-03-29T10:09:32.089228+0000","last_unstale":"2024-03-29T10:09:32.606805+0000","last_undegraded":"2024-03-29T10:09:32.606805+0000","last_fullsized":"2024-03-29T10:09:32.606805+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:14.347670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613336+0000","last_change":"2024-03-29T10:09:32.614565+0000","last_active":"2024-03-29T10:09:33.613336+0000","last_peered":"2024-03-29T10:09:33.613336+0000","last_clean":"2024-03-29T10:09:33.613336+0000","last_became_active":"2024-03-29T10:09:32.614312+0000","last_became_peered":"2024-03-29T10:09:32.614312+0000","last_unstale":"2024-03-29T10:09:33.613336+0000","last_undegraded":"2024-03-29T10:09:33.613336+0000","last_fullsized":"2024-03-29T10:09:33.613336+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:27:36.451327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.636359+0000","last_change":"2024-03-29T10:09:32.095956+0000","last_active":"2024-03-29T10:09:32.636359+0000","last_peered":"2024-03-29T10:09:32.636359+0000","last_clean":"2024-03-29T10:09:32.636359+0000","last_became_active":"2024-03-29T10:09:32.093465+0000","last_became_peered":"2024-03-29T10:09:32.093465+0000","last_unstale":"2024-03-29T10:09:32.636359+0000","last_undegraded":"2024-03-29T10:09:32.636359+0000","last_fullsized":"2024-03-29T10:09:32.636359+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.686777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045534+0000","last_change":"2024-03-29T10:09:32.622232+0000","last_active":"2024-03-29T10:09:34.045534+0000","last_peered":"2024-03-29T10:09:34.045534+0000","last_clean":"2024-03-29T10:09:34.045534+0000","last_became_active":"2024-03-29T10:09:32.621815+0000","last_became_peered":"2024-03-29T10:09:32.621815+0000","last_unstale":"2024-03-29T10:09:34.045534+0000","last_undegraded":"2024-03-29T10:09:34.045534+0000","last_fullsized":"2024-03-29T10:09:34.045534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:54.720344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607356+0000","last_change":"2024-03-29T10:09:32.099575+0000","last_active":"2024-03-29T10:09:32.607356+0000","last_peered":"2024-03-29T10:09:32.607356+0000","last_clean":"2024-03-29T10:09:32.607356+0000","last_became_active":"2024-03-29T10:09:32.098434+0000","last_became_peered":"2024-03-29T10:09:32.098434+0000","last_unstale":"2024-03-29T10:09:32.607356+0000","last_undegraded":"2024-03-29T10:09:32.607356+0000","last_fullsized":"2024-03-29T10:09:32.607356+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:39.694641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044600+0000","last_change":"2024-03-29T10:09:32.623727+0000","last_active":"2024-03-29T10:09:34.044600+0000","last_peered":"2024-03-29T10:09:34.044600+0000","last_clean":"2024-03-29T10:09:34.044600+0000","last_became_active":"2024-03-29T10:09:32.623445+0000","last_became_peered":"2024-03-29T10:09:32.623445+0000","last_unstale":"2024-03-29T10:09:34.044600+0000","last_undegraded":"2024-03-29T10:09:34.044600+0000","last_fullsized":"2024-03-29T10:09:34.044600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:18.667325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.608745+0000","last_change":"2024-03-29T10:09:32.098093+0000","last_active":"2024-03-29T10:09:32.608745+0000","last_peered":"2024-03-29T10:09:32.608745+0000","last_clean":"2024-03-29T10:09:32.608745+0000","last_became_active":"2024-03-29T10:09:32.097686+0000","last_became_peered":"2024-03-29T10:09:32.097686+0000","last_unstale":"2024-03-29T10:09:32.608745+0000","last_undegraded":"2024-03-29T10:09:32.608745+0000","last_fullsized":"2024-03-29T10:09:32.608745+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:51:58.170982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613642+0000","last_change":"2024-03-29T10:09:32.623803+0000","last_active":"2024-03-29T10:09:33.613642+0000","last_peered":"2024-03-29T10:09:33.613642+0000","last_clean":"2024-03-29T10:09:33.613642+0000","last_became_active":"2024-03-29T10:09:32.621829+0000","last_became_peered":"2024-03-29T10:09:32.621829+0000","last_unstale":"2024-03-29T10:09:33.613642+0000","last_undegraded":"2024-03-29T10:09:33.613642+0000","last_fullsized":"2024-03-29T10:09:33.613642+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:56.974307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.637133+0000","last_change":"2024-03-29T10:09:32.089390+0000","last_active":"2024-03-29T10:09:32.637133+0000","last_peered":"2024-03-29T10:09:32.637133+0000","last_clean":"2024-03-29T10:09:32.637133+0000","last_became_active":"2024-03-29T10:09:32.089088+0000","last_became_peered":"2024-03-29T10:09:32.089088+0000","last_unstale":"2024-03-29T10:09:32.637133+0000","last_undegraded":"2024-03-29T10:09:32.637133+0000","last_fullsized":"2024-03-29T10:09:32.637133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:11:34.599249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.613753+0000","last_change":"2024-03-29T10:09:32.622358+0000","last_active":"2024-03-29T10:09:33.613753+0000","last_peered":"2024-03-29T10:09:33.613753+0000","last_clean":"2024-03-29T10:09:33.613753+0000","last_became_active":"2024-03-29T10:09:32.622059+0000","last_became_peered":"2024-03-29T10:09:32.622059+0000","last_unstale":"2024-03-29T10:09:33.613753+0000","last_undegraded":"2024-03-29T10:09:33.613753+0000","last_fullsized":"2024-03-29T10:09:33.613753+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:51.135113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.609775+0000","last_change":"2024-03-29T10:09:32.101276+0000","last_active":"2024-03-29T10:09:32.609775+0000","last_peered":"2024-03-29T10:09:32.609775+0000","last_clean":"2024-03-29T10:09:32.609775+0000","last_became_active":"2024-03-29T10:09:32.100949+0000","last_became_peered":"2024-03-29T10:09:32.100949+0000","last_unstale":"2024-03-29T10:09:32.609775+0000","last_undegraded":"2024-03-29T10:09:32.609775+0000","last_fullsized":"2024-03-29T10:09:32.609775+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:40.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044751+0000","last_change":"2024-03-29T10:09:32.611596+0000","last_active":"2024-03-29T10:09:34.044751+0000","last_peered":"2024-03-29T10:09:34.044751+0000","last_clean":"2024-03-29T10:09:34.044751+0000","last_became_active":"2024-03-29T10:09:32.610142+0000","last_became_peered":"2024-03-29T10:09:32.610142+0000","last_unstale":"2024-03-29T10:09:34.044751+0000","last_undegraded":"2024-03-29T10:09:34.044751+0000","last_fullsized":"2024-03-29T10:09:34.044751+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:07.635360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605517+0000","last_change":"2024-03-29T10:09:32.092374+0000","last_active":"2024-03-29T10:09:32.605517+0000","last_peered":"2024-03-29T10:09:32.605517+0000","last_clean":"2024-03-29T10:09:32.605517+0000","last_became_active":"2024-03-29T10:09:32.091381+0000","last_became_peered":"2024-03-29T10:09:32.091381+0000","last_unstale":"2024-03-29T10:09:32.605517+0000","last_undegraded":"2024-03-29T10:09:32.605517+0000","last_fullsized":"2024-03-29T10:09:32.605517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:34:06.629135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.046025+0000","last_change":"2024-03-29T10:09:32.620976+0000","last_active":"2024-03-29T10:09:34.046025+0000","last_peered":"2024-03-29T10:09:34.046025+0000","last_clean":"2024-03-29T10:09:34.046025+0000","last_became_active":"2024-03-29T10:09:32.619271+0000","last_became_peered":"2024-03-29T10:09:32.619271+0000","last_unstale":"2024-03-29T10:09:34.046025+0000","last_undegraded":"2024-03-29T10:09:34.046025+0000","last_fullsized":"2024-03-29T10:09:34.046025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.930784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.611252+0000","last_change":"2024-03-29T10:09:32.088757+0000","last_active":"2024-03-29T10:09:32.611252+0000","last_peered":"2024-03-29T10:09:32.611252+0000","last_clean":"2024-03-29T10:09:32.611252+0000","last_became_active":"2024-03-29T10:09:32.088508+0000","last_became_peered":"2024-03-29T10:09:32.088508+0000","last_unstale":"2024-03-29T10:09:32.611252+0000","last_undegraded":"2024-03-29T10:09:32.611252+0000","last_fullsized":"2024-03-29T10:09:32.611252+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:56.946534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607553+0000","last_change":"2024-03-29T10:09:32.618538+0000","last_active":"2024-03-29T10:09:33.607553+0000","last_peered":"2024-03-29T10:09:33.607553+0000","last_clean":"2024-03-29T10:09:33.607553+0000","last_became_active":"2024-03-29T10:09:32.617981+0000","last_became_peered":"2024-03-29T10:09:32.617981+0000","last_unstale":"2024-03-29T10:09:33.607553+0000","last_undegraded":"2024-03-29T10:09:33.607553+0000","last_fullsized":"2024-03-29T10:09:33.607553+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:50.197997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.607488+0000","last_change":"2024-03-29T10:09:32.084059+0000","last_active":"2024-03-29T10:09:32.607488+0000","last_peered":"2024-03-29T10:09:32.607488+0000","last_clean":"2024-03-29T10:09:32.607488+0000","last_became_active":"2024-03-29T10:09:32.082631+0000","last_became_peered":"2024-03-29T10:09:32.082631+0000","last_unstale":"2024-03-29T10:09:32.607488+0000","last_undegraded":"2024-03-29T10:09:32.607488+0000","last_fullsized":"2024-03-29T10:09:32.607488+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:37:31.209166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.045792+0000","last_change":"2024-03-29T10:09:32.628132+0000","last_active":"2024-03-29T10:09:34.045792+0000","last_peered":"2024-03-29T10:09:34.045792+0000","last_clean":"2024-03-29T10:09:34.045792+0000","last_became_active":"2024-03-29T10:09:32.627845+0000","last_became_peered":"2024-03-29T10:09:32.627845+0000","last_unstale":"2024-03-29T10:09:34.045792+0000","last_undegraded":"2024-03-29T10:09:34.045792+0000","last_fullsized":"2024-03-29T10:09:34.045792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:52.322978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.604249+0000","last_change":"2024-03-29T10:09:32.094306+0000","last_active":"2024-03-29T10:09:32.604249+0000","last_peered":"2024-03-29T10:09:32.604249+0000","last_clean":"2024-03-29T10:09:32.604249+0000","last_became_active":"2024-03-29T10:09:32.093998+0000","last_became_peered":"2024-03-29T10:09:32.093998+0000","last_unstale":"2024-03-29T10:09:32.604249+0000","last_undegraded":"2024-03-29T10:09:32.604249+0000","last_fullsized":"2024-03-29T10:09:32.604249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:00:17.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:34.044480+0000","last_change":"2024-03-29T10:09:32.620507+0000","last_active":"2024-03-29T10:09:34.044480+0000","last_peered":"2024-03-29T10:09:34.044480+0000","last_clean":"2024-03-29T10:09:34.044480+0000","last_became_active":"2024-03-29T10:09:32.620186+0000","last_became_peered":"2024-03-29T10:09:32.620186+0000","last_unstale":"2024-03-29T10:09:34.044480+0000","last_undegraded":"2024-03-29T10:09:34.044480+0000","last_fullsized":"2024-03-29T10:09:34.044480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:49.874012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605757+0000","last_change":"2024-03-29T10:09:32.086897+0000","last_active":"2024-03-29T10:09:32.605757+0000","last_peered":"2024-03-29T10:09:32.605757+0000","last_clean":"2024-03-29T10:09:32.605757+0000","last_became_active":"2024-03-29T10:09:32.086602+0000","last_became_peered":"2024-03-29T10:09:32.086602+0000","last_unstale":"2024-03-29T10:09:32.605757+0000","last_undegraded":"2024-03-29T10:09:32.605757+0000","last_fullsized":"2024-03-29T10:09:32.605757+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:23:18.397722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.607519+0000","last_change":"2024-03-29T10:09:32.633167+0000","last_active":"2024-03-29T10:09:33.607519+0000","last_peered":"2024-03-29T10:09:33.607519+0000","last_clean":"2024-03-29T10:09:33.607519+0000","last_became_active":"2024-03-29T10:09:32.632682+0000","last_became_peered":"2024-03-29T10:09:32.632682+0000","last_unstale":"2024-03-29T10:09:33.607519+0000","last_undegraded":"2024-03-29T10:09:33.607519+0000","last_fullsized":"2024-03-29T10:09:33.607519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:07.268799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.605003+0000","last_change":"2024-03-29T10:09:32.092768+0000","last_active":"2024-03-29T10:09:32.605003+0000","last_peered":"2024-03-29T10:09:32.605003+0000","last_clean":"2024-03-29T10:09:32.605003+0000","last_became_active":"2024-03-29T10:09:32.092488+0000","last_became_peered":"2024-03-29T10:09:32.092488+0000","last_unstale":"2024-03-29T10:09:32.605003+0000","last_undegraded":"2024-03-29T10:09:32.605003+0000","last_fullsized":"2024-03-29T10:09:32.605003+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:16:16.339327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-29T10:09:33.606964+0000","last_change":"2024-03-29T10:09:32.628533+0000","last_active":"2024-03-29T10:09:33.606964+0000","last_peered":"2024-03-29T10:09:33.606964+0000","last_clean":"2024-03-29T10:09:33.606964+0000","last_became_active":"2024-03-29T10:09:32.628297+0000","last_became_peered":"2024-03-29T10:09:32.628297+0000","last_unstale":"2024-03-29T10:09:33.606964+0000","last_undegraded":"2024-03-29T10:09:33.606964+0000","last_fullsized":"2024-03-29T10:09:33.606964+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:32.067771+0000","last_clean_scrub_stamp":"2024-03-29T10:09:32.067771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:53:16.735792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.604907+0000","last_change":"2024-03-29T10:09:32.088461+0000","last_active":"2024-03-29T10:09:32.604907+0000","last_peered":"2024-03-29T10:09:32.604907+0000","last_clean":"2024-03-29T10:09:32.604907+0000","last_became_active":"2024-03-29T10:09:32.087924+0000","last_became_peered":"2024-03-29T10:09:32.087924+0000","last_unstale":"2024-03-29T10:09:32.604907+0000","last_undegraded":"2024-03-29T10:09:32.604907+0000","last_fullsized":"2024-03-29T10:09:32.604907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:09:31.061546+0000","last_clean_scrub_stamp":"2024-03-29T10:09:31.061546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:24:01.534207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"68'161","reported_seq":379,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-29T10:09:32.613595+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:09:32.613595+0000","last_peered":"2024-03-29T10:09:32.613595+0000","last_clean":"2024-03-29T10:09:32.613595+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:09:32.613595+0000","last_undegraded":"2024-03-29T10:09:32.613595+0000","last_fullsized":"2024-03-29T10:09:32.613595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1568768,"data_stored":1047510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.57299999999999995,"5min":0.44500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95299999999999996,"5min":1.0029999999999999,"15min":1.0109999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.69999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.89800000000000002,"5min":0.93400000000000005,"15min":1.046},"last":0.72299999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71699999999999997,"15min":0.71999999999999997},"min":{"1min":0.51900000000000002,"5min":0.47399999999999998,"15min":0.44500000000000001},"max":{"1min":0.92600000000000005,"5min":0.93200000000000005,"15min":1.1910000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.73499999999999999},"min":{"1min":0.51700000000000002,"5min":0.42199999999999999,"15min":0.372},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1},"last":0.71099999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.72899999999999998,"15min":0.74099999999999999},"min":{"1min":0.47199999999999998,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.046},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.755,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":1.079},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.48099999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.93100000000000005,"5min":0.97699999999999998,"15min":1.081},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77500000000000002,"15min":0.78300000000000003},"min":{"1min":0.55800000000000005,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":0.96199999999999997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.74199999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.29199999999999998},"max":{"1min":0.86399999999999999,"5min":0.87,"15min":1.111},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.627,"15min":0.64100000000000001},"min":{"1min":0.44800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":0.95199999999999996,"15min":0.95599999999999996},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64900000000000002,"15min":0.65900000000000003},"min":{"1min":0.432,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95299999999999996,"5min":0.96099999999999997,"15min":1.0089999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67600000000000005,"15min":0.68899999999999995},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90100000000000002,"5min":0.92800000000000005,"15min":1.141},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.71899999999999997,"15min":0.73299999999999998},"min":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.49199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":1.1679999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69699999999999995,"15min":0.70499999999999996},"min":{"1min":0.54800000000000004,"5min":0.48499999999999999,"15min":0.435},"max":{"1min":0.93799999999999994,"5min":0.98399999999999999,"15min":1.121},"last":0.48499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575031,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":966656,"data_stored":456982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70299999999999996,"15min":0.745},"min":{"1min":0.57199999999999995,"5min":0.42799999999999999,"15min":0.34699999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":7.4080000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.72599999999999998,"15min":0.76900000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.42499999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":6.7610000000000001},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70999999999999996,"15min":0.71899999999999997},"min":{"1min":0.51300000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.2549999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.72499999999999998,"15min":0.71999999999999997},"min":{"1min":0.58999999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":0.995,"15min":1.0209999999999999},"last":0.745}]},{"osd":2,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.71699999999999997,"15min":0.72399999999999998},"min":{"1min":0.65900000000000003,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94199999999999995,"5min":0.96899999999999997,"15min":1.3169999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.58899999999999997,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.415},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.74099999999999999,"15min":0.76900000000000002},"min":{"1min":0.60399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.98999999999999999,"5min":1.02,"15min":3.1960000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76800000000000002,"15min":0.79800000000000004},"min":{"1min":0.71499999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0109999999999999,"5min":1.014,"15min":4.3760000000000003},"last":0.70499999999999996}]},{"osd":4,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":1.109},"last":0.753},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.47799999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85899999999999999,"5min":0.95599999999999996,"15min":3.1219999999999999},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.67300000000000004},"min":{"1min":0.50600000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90000000000000002,"5min":0.90900000000000003,"15min":2.8279999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.79500000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.74399999999999999,"15min":0.76100000000000001},"min":{"1min":0.56499999999999995,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.95499999999999996,"5min":1.105,"15min":1.2769999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.496,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94499999999999995,"5min":0.998,"15min":1.7869999999999999},"last":0.76200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50116,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321724,"statfs":{"total":96636764160,"available":96585445376,"internally_reserved":0,"allocated":970752,"data_stored":457072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.315,"5min":0.219,"15min":0.219},"max":{"1min":0.65300000000000002,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.379},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.68999999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.504}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71599999999999997,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.52900000000000003},"min":{"1min":0.36699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":4.282},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.70999999999999996,"5min":0.78500000000000003,"15min":4.3200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.36599999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.016,"5min":1.617,"15min":1.617},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76400000000000001,"15min":0.78800000000000003},"min":{"1min":0.60899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0289999999999999,"5min":1.1419999999999999,"15min":4.3559999999999999},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.51800000000000002},"max":{"1min":1.056,"5min":1.3939999999999999,"15min":1.9119999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78600000000000003,"15min":0.80800000000000005},"min":{"1min":0.56299999999999994,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.048,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77800000000000002,"15min":0.81000000000000005},"min":{"1min":0.57499999999999996,"5min":0.52600000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":1.194,"15min":4.3899999999999997},"last":0.752},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.093,"5min":1.468,"15min":1.468},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.62,"5min":0.58699999999999997,"15min":0.56200000000000006},"max":{"1min":1.127,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.83599999999999997},"min":{"1min":0.60999999999999999,"5min":0.57099999999999995,"15min":0.55700000000000005},"max":{"1min":1.0269999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.73699999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1572864,"data_stored":1048477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.33300000000000002,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.497,"15min":0.51000000000000001},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72199999999999998,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.628}]},{"osd":2,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.435,"5min":0.39200000000000002,"15min":0.35499999999999998},"max":{"1min":0.78800000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.48799999999999999,"15min":0.496},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.63400000000000001,"5min":0.83899999999999997,"15min":0.97199999999999998},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.39000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73599999999999999,"5min":0.83199999999999996,"15min":1.077},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.47199999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83499999999999996,"5min":0.876,"15min":1.0549999999999999},"last":0.5}]},{"osd":4,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.81399999999999995,"15min":0.80200000000000005},"min":{"1min":0.65100000000000002,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":1.0469999999999999,"5min":1.625,"15min":1.625},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.82399999999999995},"min":{"1min":0.66900000000000004,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":0.97099999999999997,"5min":1.1419999999999999,"15min":1.167},"last":0.73399999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.752,"15min":0.75900000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98999999999999999,"5min":1.137,"15min":1.137},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.80600000000000005,"15min":0.80300000000000005},"min":{"1min":0.60399999999999998,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":0.90200000000000002,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.752}]},{"osd":6,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.53400000000000003},"max":{"1min":1.0129999999999999,"5min":1.0640000000000001,"15min":1.228},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.85499999999999998,"15min":0.84399999999999997},"min":{"1min":0.66400000000000003,"5min":0.52600000000000002,"15min":0.46899999999999997},"max":{"1min":1.0249999999999999,"5min":1.601,"15min":1.601},"last":0.80100000000000005}]},{"osd":7,"last update":"Fri Mar 29 10:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78300000000000003,"15min":0.79600000000000004},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.97299999999999998,"5min":1.0529999999999999,"15min":1.1890000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.84699999999999998,"15min":0.84199999999999997},"min":{"1min":0.64000000000000001,"5min":0.57499999999999996,"15min":0.49299999999999999},"max":{"1min":1.046,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.84199999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36172,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335668,"statfs":{"total":96636764160,"available":96599724032,"internally_reserved":0,"allocated":974848,"data_stored":457038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.34699999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.623,"5min":0.69599999999999995,"15min":0.86599999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.39300000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":1.3140000000000001},"last":0.47999999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47299999999999998,"15min":0.46999999999999997},"min":{"1min":0.376,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.63600000000000001,"5min":0.751,"15min":0.94499999999999995},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.45800000000000002,"5min":0.36099999999999999,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":1.032,"15min":1.032},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.504,"5min":0.375,"15min":0.375},"max":{"1min":0.73799999999999999,"5min":0.80000000000000004,"15min":1.127},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.48999999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70899999999999996,"5min":0.82399999999999995,"15min":1.1479999999999999},"last":0.51800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.77300000000000002,"15min":0.78600000000000003},"min":{"1min":0.68700000000000006,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":1.0760000000000001,"5min":1.1799999999999999,"15min":1.554},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.79500000000000004,"15min":0.81000000000000005},"min":{"1min":0.60799999999999998,"5min":0.55000000000000004,"15min":0.51600000000000001},"max":{"1min":1.0309999999999999,"5min":1.236,"15min":1.577},"last":0.86099999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.81000000000000005},"min":{"1min":0.64700000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0680000000000001,"5min":1.25,"15min":1.5209999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79600000000000004,"15min":0.80800000000000005},"min":{"1min":0.58799999999999997,"5min":0.53200000000000003,"15min":0.51800000000000002},"max":{"1min":1.0049999999999999,"5min":1.2250000000000001,"15min":1.3300000000000001},"last":0.77700000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.80400000000000005,"15min":0.82799999999999996},"min":{"1min":0.65200000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1399999999999999,"5min":1.1539999999999999,"15min":1.758},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.874,"5min":0.83699999999999997,"15min":0.84199999999999997},"min":{"1min":0.65900000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1080000000000001,"5min":1.3560000000000001,"15min":1.4750000000000001},"last":0.90900000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:08:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.86399999999999999,"15min":0.86899999999999999},"min":{"1min":0.73599999999999999,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.137,"5min":1.2709999999999999,"15min":1.4490000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.69999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.107,"5min":1.3049999999999999,"15min":1.4390000000000001},"last":0.96799999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575030,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36172,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335668,"statfs":{"total":96636764160,"available":96599724032,"internally_reserved":0,"allocated":974848,"data_stored":457038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.46600000000000003},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":0.79500000000000004,"15min":4.3799999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52600000000000002,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":4.6379999999999999},"last":0.621}]},{"osd":1,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.40200000000000002,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.69799999999999995,"5min":0.98699999999999999,"15min":4.1840000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.314},"max":{"1min":0.80800000000000005,"5min":0.96099999999999997,"15min":1.048},"last":0.67600000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.47799999999999998,"5min":0.373,"15min":0.33700000000000002},"max":{"1min":0.77200000000000002,"5min":0.82499999999999996,"15min":1.369},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.63800000000000001,"15min":0.64700000000000002},"min":{"1min":0.55700000000000005,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.79700000000000004,"5min":0.94799999999999995,"15min":4.5970000000000004},"last":0.748}]},{"osd":4,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76800000000000002,"15min":0.78100000000000003},"min":{"1min":0.64400000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.0940000000000001,"5min":1.288,"15min":4.3319999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77600000000000002,"15min":0.77500000000000002},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.39400000000000002},"max":{"1min":0.92400000000000004,"5min":1.46,"15min":1.46},"last":1.0900000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78200000000000003,"15min":0.79500000000000004},"min":{"1min":0.628,"5min":0.48899999999999999,"15min":0.435},"max":{"1min":0.97599999999999998,"5min":1.3580000000000001,"15min":1.3740000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79800000000000004,"15min":0.81000000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.44500000000000001},"max":{"1min":1.133,"5min":1.397,"15min":4.4180000000000001},"last":0.876}]},{"osd":6,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81399999999999995,"15min":0.81200000000000006},"min":{"1min":0.68899999999999995,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.0009999999999999,"5min":1.2549999999999999,"15min":1.4730000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81699999999999995,"15min":0.81899999999999995},"min":{"1min":0.65000000000000002,"5min":0.52400000000000002,"15min":0.503},"max":{"1min":1.0409999999999999,"5min":1.23,"15min":1.417},"last":1.0349999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.83999999999999997,"15min":0.84799999999999998},"min":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.54300000000000004},"max":{"1min":1.135,"5min":1.5,"15min":4.4550000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.64400000000000002,"5min":0.59899999999999998,"15min":0.56200000000000006},"max":{"1min":1.056,"5min":1.345,"15min":1.476},"last":1.0489999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54224,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317616,"statfs":{"total":96636764160,"available":96581238784,"internally_reserved":0,"allocated":978944,"data_stored":458546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.68700000000000006},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.88200000000000001,"5min":0.93200000000000005,"15min":1.417},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.92100000000000004,"5min":1.165,"15min":1.165},"last":0.434}]},{"osd":1,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.93500000000000005,"5min":0.93700000000000006,"15min":1.139},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.333},"last":0.55700000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.01,"15min":1.863},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.75,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93700000000000006,"5min":0.96299999999999997,"15min":1.387},"last":0.50600000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.5,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.88700000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.495},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.51600000000000001,"5min":0.48699999999999999,"15min":0.46600000000000003},"max":{"1min":0.80500000000000005,"5min":1.1539999999999999,"15min":1.905},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.46700000000000003,"5min":0.39300000000000002,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.48099999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.78500000000000003,"5min":1.294,"15min":1.294},"last":0.629}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.48099999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.75900000000000001,"5min":1.0129999999999999,"15min":1.9339999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.66200000000000003},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.433},"max":{"1min":0.85299999999999998,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.48099999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85999999999999999,"5min":1.345,"15min":1.345},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.59099999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89700000000000002,"5min":1.214,"15min":1.97},"last":0.755}]}]},{"osd":5,"up_from":13,"seq":55834575030,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":974848,"data_stored":457634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69599999999999995,"15min":0.68200000000000005},"min":{"1min":0.432,"5min":0.41599999999999998,"15min":0.313},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.746,"15min":0.71899999999999997},"min":{"1min":0.45900000000000002,"5min":0.45500000000000002,"15min":0.33500000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.339},"last":0.80100000000000005}]},{"osd":1,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.55400000000000005,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.0169999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72199999999999998,"15min":0.70499999999999996},"min":{"1min":0.47999999999999998,"5min":0.46700000000000003,"15min":0.39200000000000002},"max":{"1min":0.95199999999999996,"5min":1.022,"15min":1.6040000000000001},"last":1.0669999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73099999999999998,"15min":0.72799999999999998},"min":{"1min":0.57099999999999995,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.476},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.59299999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93700000000000006,"5min":1.095,"15min":1.502},"last":0.81899999999999995}]},{"osd":3,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.63200000000000001,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":0.95799999999999996,"5min":1.113,"15min":1.1619999999999999},"last":1.042},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.754},"min":{"1min":0.56499999999999995,"5min":0.52300000000000002,"15min":0.46500000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.744},"last":0.75900000000000001}]},{"osd":4,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.80300000000000005,"5min":0.94599999999999995,"15min":0.95899999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.214},"last":1.097}]},{"osd":6,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64800000000000002,"15min":0.65400000000000003},"min":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.36799999999999999},"max":{"1min":0.85999999999999999,"5min":0.91300000000000003,"15min":1.151},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.52600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":0.97099999999999997,"15min":1.5489999999999999},"last":0.73699999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.70499999999999996,"15min":0.71099999999999997},"min":{"1min":0.51500000000000001,"5min":0.41899999999999998,"15min":0.40200000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.325},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.53800000000000003,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":0.96999999999999997,"5min":0.98199999999999998,"15min":1.1990000000000001},"last":0.90300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:09:42.553 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T10:09:42.554 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T10:09:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.796+0000 7fa9e3a00700 1 Processor -- start 2024-03-29T10:09:42.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.796+0000 7fa9e3a00700 1 -- start start 2024-03-29T10:09:42.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9e3a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9dc0d21f0 0x7fa9dc0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9e3a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9dc0cec40 0x7fa9dc161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9e3a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc0ce270 con 0x7fa9dc1611b0 2024-03-29T10:09:42.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9e3a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc15b600 con 0x7fa9dc0d21f0 2024-03-29T10:09:42.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9e3a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0d2760 con 0x7fa9dc0cec40 2024-03-29T10:09:42.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9dbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9dc0d21f0 0x7fa9dc0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9dbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9dc0d21f0 0x7fa9dc0ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48070/0 (socket says 172.21.15.12:48070) 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9dbe00700 1 -- 172.21.15.12:0/2488160768 learned_addr learned my addr 172.21.15.12:0/2488160768 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9e1a00700 1 --2- 172.21.15.12:0/2488160768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9dc0cec40 0x7fa9dc161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9dbe00700 1 -- 172.21.15.12:0/2488160768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9dc0cec40 msgr2=0x7fa9dc161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9dbe00700 1 --2- 172.21.15.12:0/2488160768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9dc0cec40 0x7fa9dc161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.798+0000 7fa9dbe00700 1 -- 172.21.15.12:0/2488160768 >> v1:172.21.15.159:6789/0 conn(0x7fa9dc1611b0 legacy=0x7fa9dc15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.799+0000 7fa9dbe00700 1 -- 172.21.15.12:0/2488160768 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc0d2a40 con 0x7fa9dc0d21f0 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.799+0000 7fa9dbe00700 1 --2- 172.21.15.12:0/2488160768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0d21f0 0x7fa9dc0ce700 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80098f0 tx=0x7fa9c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4c45117f6551db server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.799+0000 7fa9db400700 1 -- 172.21.15.12:0/2488160768 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c800b8a0 con 0x7fa9dc0d21f0 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.799+0000 7fa9db400700 1 -- 172.21.15.12:0/2488160768 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c800ba00 con 0x7fa9dc0d21f0 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.799+0000 7fa9db400700 1 -- 172.21.15.12:0/2488160768 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c800bd90 con 0x7fa9dc0d21f0 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.799+0000 7fa9e3a00700 1 -- 172.21.15.12:0/2488160768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0d21f0 msgr2=0x7fa9dc0ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:42.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.799+0000 7fa9e3a00700 1 --2- 172.21.15.12:0/2488160768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0d21f0 0x7fa9dc0ce700 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80098f0 tx=0x7fa9c8009c00 comp rx=0 tx=0).stop 2024-03-29T10:09:42.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.800+0000 7fa9e3a00700 1 -- 172.21.15.12:0/2488160768 shutdown_connections 2024-03-29T10:09:42.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.800+0000 7fa9e3a00700 1 --2- 172.21.15.12:0/2488160768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9dc0cec40 0x7fa9dc161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.800+0000 7fa9e3a00700 1 --2- 172.21.15.12:0/2488160768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0d21f0 0x7fa9dc0ce700 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.800+0000 7fa9e3a00700 1 -- 172.21.15.12:0/2488160768 >> 172.21.15.12:0/2488160768 conn(0x7fa9dc08aff0 msgr2=0x7fa9dc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:42.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.800+0000 7fa9e3a00700 1 -- 172.21.15.12:0/2488160768 shutdown_connections 2024-03-29T10:09:42.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.800+0000 7fa9e3a00700 1 -- 172.21.15.12:0/2488160768 wait complete. 2024-03-29T10:09:42.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.801+0000 7fa9e3a00700 1 Processor -- start 2024-03-29T10:09:42.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.802+0000 7fa9e3a00700 1 -- start start 2024-03-29T10:09:42.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.802+0000 7fa9e3a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0cec40 0x7fa9dc1e6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.802+0000 7fa9e3a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9dc0d21f0 0x7fa9dc1567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.802+0000 7fa9e3a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9dc1611b0 0x7fa9dc0b6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.802+0000 7fa9e3a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0b71b0 con 0x7fa9dc0cec40 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.802+0000 7fa9e3a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0cbe70 con 0x7fa9dc1611b0 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.802+0000 7fa9e3a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc160e40 con 0x7fa9dc0d21f0 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.803+0000 7fa9e1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0cec40 0x7fa9dc1e6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.803+0000 7fa9e1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0cec40 0x7fa9dc1e6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48084/0 (socket says 172.21.15.12:48084) 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.803+0000 7fa9e1000700 1 -- 172.21.15.12:0/3100110594 learned_addr learned my addr 172.21.15.12:0/3100110594 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.803+0000 7fa9e1a00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9dc1611b0 0x7fa9dc0b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.803+0000 7fa9dbe00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9dc0d21f0 0x7fa9dc1567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.803+0000 7fa9e1000700 1 -- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9dc0d21f0 msgr2=0x7fa9dc1567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.803+0000 7fa9e1000700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9dc0d21f0 0x7fa9dc1567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.803+0000 7fa9e1000700 1 -- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9dc1611b0 msgr2=0x7fa9dc0b6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.803+0000 7fa9e1000700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9dc1611b0 0x7fa9dc0b6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.803+0000 7fa9e1000700 1 -- 172.21.15.12:0/3100110594 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8009550 con 0x7fa9dc0cec40 2024-03-29T10:09:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.804+0000 7fa9e1000700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0cec40 0x7fa9dc1e6ff0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000c450 tx=0x7fa9d000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:42.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.804+0000 7fa9d8c00700 1 -- 172.21.15.12:0/3100110594 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d00052d0 con 0x7fa9dc0cec40 2024-03-29T10:09:42.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.804+0000 7fa9d8c00700 1 -- 172.21.15.12:0/3100110594 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d0005430 con 0x7fa9dc0cec40 2024-03-29T10:09:42.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.804+0000 7fa9d8c00700 1 -- 172.21.15.12:0/3100110594 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d00038c0 con 0x7fa9dc0cec40 2024-03-29T10:09:42.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.804+0000 7fa9e3a00700 1 -- 172.21.15.12:0/3100110594 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc0b6ed0 con 0x7fa9dc0cec40 2024-03-29T10:09:42.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.804+0000 7fa9e3a00700 1 -- 172.21.15.12:0/3100110594 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc0cbb00 con 0x7fa9dc0cec40 2024-03-29T10:09:42.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.805+0000 7fa9e3a00700 1 -- 172.21.15.12:0/3100110594 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc067720 con 0x7fa9dc0cec40 2024-03-29T10:09:42.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.806+0000 7fa9d8c00700 1 -- 172.21.15.12:0/3100110594 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa9d0003a20 con 0x7fa9dc0cec40 2024-03-29T10:09:42.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.807+0000 7fa9d8c00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9bc06adb0 0x7fa9bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:42.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.807+0000 7fa9d8c00700 1 -- 172.21.15.12:0/3100110594 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9d0088850 con 0x7fa9dc0cec40 2024-03-29T10:09:42.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.807+0000 7fa9dbe00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9bc06adb0 0x7fa9bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:42.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.808+0000 7fa9dbe00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9bc06adb0 0x7fa9bc06d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009c40 tx=0x7fa9c8008030 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:42.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:42.811+0000 7fa9d8c00700 1 -- 172.21.15.12:0/3100110594 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa9d0053880 con 0x7fa9dc0cec40 2024-03-29T10:09:43.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.108+0000 7fa9e3a00700 1 -- 172.21.15.12:0/3100110594 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa9dc0b5d00 con 0x7fa9dc0cec40 2024-03-29T10:09:43.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.113+0000 7fa9d8c00700 1 -- 172.21.15.12:0/3100110594 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa9d000ce60 con 0x7fa9dc0cec40 2024-03-29T10:09:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 -- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9bc06adb0 msgr2=0x7fa9bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9bc06adb0 0x7fa9bc06d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009c40 tx=0x7fa9c8008030 comp rx=0 tx=0).stop 2024-03-29T10:09:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 -- 172.21.15.12:0/3100110594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0cec40 msgr2=0x7fa9dc1e6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0cec40 0x7fa9dc1e6ff0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000c450 tx=0x7fa9d000c760 comp rx=0 tx=0).stop 2024-03-29T10:09:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 -- 172.21.15.12:0/3100110594 shutdown_connections 2024-03-29T10:09:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9bc06adb0 0x7fa9bc06d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9dc1611b0 0x7fa9dc0b6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9dc0cec40 0x7fa9dc1e6ff0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 --2- 172.21.15.12:0/3100110594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9dc0d21f0 0x7fa9dc1567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 -- 172.21.15.12:0/3100110594 >> 172.21.15.12:0/3100110594 conn(0x7fa9dc08aff0 msgr2=0x7fa9dc15f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:43.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 -- 172.21.15.12:0/3100110594 shutdown_connections 2024-03-29T10:09:43.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.116+0000 7fa9e3a00700 1 -- 172.21.15.12:0/3100110594 wait complete. 2024-03-29T10:09:43.117 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T10:09:43.134 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:09:43.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.332+0000 7fc88f200700 1 Processor -- start 2024-03-29T10:09:43.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.332+0000 7fc88f200700 1 -- start start 2024-03-29T10:09:43.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.333+0000 7fc88f200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc8880d18b0 0x7fc8880cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:43.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.333+0000 7fc88f200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc888161590 0x7fc888161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.333+0000 7fc88f200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8880ceda0 con 0x7fc888160f10 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.333+0000 7fc88f200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8880d1d30 con 0x7fc8880d18b0 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.333+0000 7fc88f200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc888161f60 con 0x7fc888161590 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.333+0000 7fc887e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc888160f10 0x7fc8880d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60804/0 (socket says 172.21.15.12:60804) 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.333+0000 7fc887e00700 1 -- 172.21.15.12:0/2444093537 learned_addr learned my addr 172.21.15.12:0/2444093537 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.333+0000 7fc887400700 1 --2- 172.21.15.12:0/2444093537 >> v2:172.21.15.12:3300/0 conn(0x7fc8880d18b0 0x7fc8880cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc88d200700 1 --2- 172.21.15.12:0/2444093537 >> v2:172.21.15.159:3300/0 conn(0x7fc888161590 0x7fc888161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc886a00700 1 -- 172.21.15.12:0/2444093537 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363997069 0 0) 0x7fc8880ceda0 con 0x7fc888160f10 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc886a00700 1 -- 172.21.15.12:0/2444093537 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c0035b0 con 0x7fc888160f10 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc887400700 1 -- 172.21.15.12:0/2444093537 >> v2:172.21.15.159:3300/0 conn(0x7fc888161590 msgr2=0x7fc888161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc887400700 1 --2- 172.21.15.12:0/2444093537 >> v2:172.21.15.159:3300/0 conn(0x7fc888161590 0x7fc888161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc887400700 1 -- 172.21.15.12:0/2444093537 >> v1:172.21.15.12:6789/0 conn(0x7fc888160f10 legacy=0x7fc8880d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:43.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc887400700 1 -- 172.21.15.12:0/2444093537 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8881612f0 con 0x7fc8880d18b0 2024-03-29T10:09:43.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc88d200700 1 --2- 172.21.15.12:0/2444093537 >> v2:172.21.15.159:3300/0 conn(0x7fc888161590 0x7fc888161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:43.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc887400700 1 --2- 172.21.15.12:0/2444093537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8880d18b0 0x7fc8880cf230 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc8880d1030 tx=0x7fc874009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479bc57e6918221e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:43.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc886a00700 1 -- 172.21.15.12:0/2444093537 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc874011830 con 0x7fc8880d18b0 2024-03-29T10:09:43.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc886a00700 1 -- 172.21.15.12:0/2444093537 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc874011990 con 0x7fc8880d18b0 2024-03-29T10:09:43.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.334+0000 7fc886a00700 1 -- 172.21.15.12:0/2444093537 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc874011d20 con 0x7fc8880d18b0 2024-03-29T10:09:43.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.335+0000 7fc88f200700 1 -- 172.21.15.12:0/2444093537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8880d18b0 msgr2=0x7fc8880cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:43.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.335+0000 7fc88f200700 1 --2- 172.21.15.12:0/2444093537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8880d18b0 0x7fc8880cf230 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc8880d1030 tx=0x7fc874009b20 comp rx=0 tx=0).stop 2024-03-29T10:09:43.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.335+0000 7fc88f200700 1 -- 172.21.15.12:0/2444093537 shutdown_connections 2024-03-29T10:09:43.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.335+0000 7fc88f200700 1 --2- 172.21.15.12:0/2444093537 >> v2:172.21.15.159:3300/0 conn(0x7fc888161590 0x7fc888161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.335+0000 7fc88f200700 1 --2- 172.21.15.12:0/2444093537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8880d18b0 0x7fc8880cf230 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.335+0000 7fc88f200700 1 -- 172.21.15.12:0/2444093537 >> 172.21.15.12:0/2444093537 conn(0x7fc88808aff0 msgr2=0x7fc8880d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:43.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.335+0000 7fc88f200700 1 -- 172.21.15.12:0/2444093537 shutdown_connections 2024-03-29T10:09:43.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.335+0000 7fc88f200700 1 -- 172.21.15.12:0/2444093537 wait complete. 2024-03-29T10:09:43.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.337+0000 7fc88f200700 1 Processor -- start 2024-03-29T10:09:43.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.337+0000 7fc88f200700 1 -- start start 2024-03-29T10:09:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc88f200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8880d18b0 0x7fc88814b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc88f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc888160f10 0x7fc88815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc88f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc888161590 0x7fc888156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc88f200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc888156930 con 0x7fc888161590 2024-03-29T10:09:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc88f200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc888160130 con 0x7fc8880d18b0 2024-03-29T10:09:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc88f200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88815fb70 con 0x7fc888160f10 2024-03-29T10:09:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc88d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc888161590 0x7fc888156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc88d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc888161590 0x7fc888156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48106/0 (socket says 172.21.15.12:48106) 2024-03-29T10:09:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc88d200700 1 -- 172.21.15.12:0/3635515531 learned_addr learned my addr 172.21.15.12:0/3635515531 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc887e00700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8880d18b0 0x7fc88814b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.338+0000 7fc887400700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc888160f10 0x7fc88815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc88d200700 1 -- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc888160f10 msgr2=0x7fc88815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc88d200700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc888160f10 0x7fc88815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc88d200700 1 -- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8880d18b0 msgr2=0x7fc88814b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc88d200700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8880d18b0 0x7fc88814b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc88d200700 1 -- 172.21.15.12:0/3635515531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8740094a0 con 0x7fc888161590 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc88d200700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc888161590 0x7fc888156dc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc880007e70 tx=0x7fc88000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc87fe00700 1 -- 172.21.15.12:0/3635515531 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88000dd20 con 0x7fc888161590 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc87fe00700 1 -- 172.21.15.12:0/3635515531 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88000f040 con 0x7fc888161590 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc88f200700 1 -- 172.21.15.12:0/3635515531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88815fe50 con 0x7fc888161590 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc87fe00700 1 -- 172.21.15.12:0/3635515531 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88000dea0 con 0x7fc888161590 2024-03-29T10:09:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.339+0000 7fc88f200700 1 -- 172.21.15.12:0/3635515531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88814be60 con 0x7fc888161590 2024-03-29T10:09:43.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.340+0000 7fc88f200700 1 -- 172.21.15.12:0/3635515531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc888156aa0 con 0x7fc888161590 2024-03-29T10:09:43.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.341+0000 7fc87fe00700 1 -- 172.21.15.12:0/3635515531 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc880003c60 con 0x7fc888161590 2024-03-29T10:09:43.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.342+0000 7fc87fe00700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc86806ade0 0x7fc86806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:43.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.342+0000 7fc87fe00700 1 -- 172.21.15.12:0/3635515531 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8800895f0 con 0x7fc888161590 2024-03-29T10:09:43.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.343+0000 7fc887e00700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc86806ade0 0x7fc86806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.344+0000 7fc887e00700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc86806ade0 0x7fc86806d2a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc878007fd0 tx=0x7fc878007dc0 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:43.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.347+0000 7fc87fe00700 1 -- 172.21.15.12:0/3635515531 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc880054510 con 0x7fc888161590 2024-03-29T10:09:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.611+0000 7fc88f200700 1 -- 172.21.15.12:0/3635515531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc888067720 con 0x7fc888161590 2024-03-29T10:09:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.612+0000 7fc87fe00700 1 -- 172.21.15.12:0/3635515531 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fc880057b30 con 0x7fc888161590 2024-03-29T10:09:43.611 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:43.612 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3202930550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3202930550},{"type":"v1","addr":"172.21.15.159:6835","nonce":3202930550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4189378403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4189378403},{"type":"v1","addr":"172.21.15.12:6834","nonce":4189378403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2681299760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2681299760},{"type":"v1","addr":"172.21.15.12:6835","nonce":2681299760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:09:32.597152+0000","modified":"2024-03-29T10:09:35.607262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6241":{"gid":6241,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/4289275091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":4289275091},{"type":"v1","addr":"172.21.15.159:6837","nonce":4289275091}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-29T10:09:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 -- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc86806ade0 msgr2=0x7fc86806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc86806ade0 0x7fc86806d2a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc878007fd0 tx=0x7fc878007dc0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 -- 172.21.15.12:0/3635515531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc888161590 msgr2=0x7fc888156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc888161590 0x7fc888156dc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc880007e70 tx=0x7fc88000d3b0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 -- 172.21.15.12:0/3635515531 shutdown_connections 2024-03-29T10:09:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc86806ade0 0x7fc86806d2a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc888161590 0x7fc888156dc0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8880d18b0 0x7fc88814b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 --2- 172.21.15.12:0/3635515531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc888160f10 0x7fc88815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 -- 172.21.15.12:0/3635515531 >> 172.21.15.12:0/3635515531 conn(0x7fc88808aff0 msgr2=0x7fc88806cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 -- 172.21.15.12:0/3635515531 shutdown_connections 2024-03-29T10:09:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:43.614+0000 7fc88f200700 1 -- 172.21.15.12:0/3635515531 wait complete. 2024-03-29T10:09:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 31 2024-03-29T10:09:44.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 31, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T10:09:32.597152+0000', 'modified': '2024-03-29T10:09:35.607262+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6241': {'gid': 6241, 'name': 'b', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.159:6837/4289275091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6836', 'nonce': 4289275091}, {'type': 'v1', 'addr': '172.21.15.159:6837', 'nonce': 4289275091}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T10:09:44.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T10:09:44.035 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-29T10:09:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.240+0000 7fc9b0600700 1 Processor -- start 2024-03-29T10:09:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.241+0000 7fc9b0600700 1 -- start start 2024-03-29T10:09:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9b0600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 0x7fc9a80ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9b0600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc9a80d21a0 0x7fc9a815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9b0600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a815f820 con 0x7fc9a80d1630 2024-03-29T10:09:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9b0600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a80cfcc0 con 0x7fc9a80d21a0 2024-03-29T10:09:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9b0600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a80cd9f0 con 0x7fc9a815dad0 2024-03-29T10:09:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9ae600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc9a80d21a0 0x7fc9a815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9ae600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc9a80d21a0 0x7fc9a815fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48114/0 (socket says 172.21.15.12:48114) 2024-03-29T10:09:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9ae600700 1 -- 172.21.15.12:0/3176787707 learned_addr learned my addr 172.21.15.12:0/3176787707 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9adc00700 1 --2- 172.21.15.12:0/3176787707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 0x7fc9a80ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9ae600700 1 -- 172.21.15.12:0/3176787707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 msgr2=0x7fc9a80ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9ae600700 1 --2- 172.21.15.12:0/3176787707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 0x7fc9a80ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.242+0000 7fc9ae600700 1 -- 172.21.15.12:0/3176787707 >> v1:172.21.15.159:6789/0 conn(0x7fc9a80d1630 legacy=0x7fc9a80d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.243+0000 7fc9ae600700 1 -- 172.21.15.12:0/3176787707 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a80cf420 con 0x7fc9a80d21a0 2024-03-29T10:09:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.243+0000 7fc9ae600700 1 --2- 172.21.15.12:0/3176787707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d21a0 0x7fc9a815fcb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009e50 tx=0x7fc9a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215b4dfc5d4d245b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.243+0000 7fc9a7e00700 1 -- 172.21.15.12:0/3176787707 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a000e070 con 0x7fc9a80d21a0 2024-03-29T10:09:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.243+0000 7fc9a7e00700 1 -- 172.21.15.12:0/3176787707 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a000b9c0 con 0x7fc9a80d21a0 2024-03-29T10:09:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.243+0000 7fc9a7e00700 1 -- 172.21.15.12:0/3176787707 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a000bd70 con 0x7fc9a80d21a0 2024-03-29T10:09:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.243+0000 7fc9b0600700 1 -- 172.21.15.12:0/3176787707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d21a0 msgr2=0x7fc9a815fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.243+0000 7fc9b0600700 1 --2- 172.21.15.12:0/3176787707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d21a0 0x7fc9a815fcb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009e50 tx=0x7fc9a000b330 comp rx=0 tx=0).stop 2024-03-29T10:09:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.244+0000 7fc9b0600700 1 -- 172.21.15.12:0/3176787707 shutdown_connections 2024-03-29T10:09:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.244+0000 7fc9b0600700 1 --2- 172.21.15.12:0/3176787707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d21a0 0x7fc9a815fcb0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.244+0000 7fc9b0600700 1 --2- 172.21.15.12:0/3176787707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 0x7fc9a80ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.244+0000 7fc9b0600700 1 -- 172.21.15.12:0/3176787707 >> 172.21.15.12:0/3176787707 conn(0x7fc9a808aff0 msgr2=0x7fc9a80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.244+0000 7fc9b0600700 1 -- 172.21.15.12:0/3176787707 shutdown_connections 2024-03-29T10:09:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.244+0000 7fc9b0600700 1 -- 172.21.15.12:0/3176787707 wait complete. 2024-03-29T10:09:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.245+0000 7fc9b0600700 1 Processor -- start 2024-03-29T10:09:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.246+0000 7fc9b0600700 1 -- start start 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.246+0000 7fc9b0600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d1630 0x7fc9a80b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.246+0000 7fc9b0600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9a80d21a0 0x7fc9a814d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.246+0000 7fc9b0600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 0x7fc9a806ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.246+0000 7fc9b0600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a80b8f30 con 0x7fc9a80d1630 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.246+0000 7fc9b0600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8157950 con 0x7fc9a80d21a0 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.246+0000 7fc9b0600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8157c30 con 0x7fc9a815dad0 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9adc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d1630 0x7fc9a80b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9adc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d1630 0x7fc9a80b89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48118/0 (socket says 172.21.15.12:48118) 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9adc00700 1 -- 172.21.15.12:0/3809547971 learned_addr learned my addr 172.21.15.12:0/3809547971 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9ae600700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 0x7fc9a806ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9ad200700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9a80d21a0 0x7fc9a814d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9adc00700 1 -- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 msgr2=0x7fc9a806ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9adc00700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 0x7fc9a806ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9adc00700 1 -- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9a80d21a0 msgr2=0x7fc9a814d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9adc00700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9a80d21a0 0x7fc9a814d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9adc00700 1 -- 172.21.15.12:0/3809547971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a0009b30 con 0x7fc9a80d1630 2024-03-29T10:09:44.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9ae600700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 0x7fc9a806ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:44.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.247+0000 7fc9adc00700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d1630 0x7fc9a80b89f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00d800 tx=0x7fc99c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:44.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.248+0000 7fc9a5600700 1 -- 172.21.15.12:0/3809547971 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0125f0 con 0x7fc9a80d1630 2024-03-29T10:09:44.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.248+0000 7fc9a5600700 1 -- 172.21.15.12:0/3809547971 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c012750 con 0x7fc9a80d1630 2024-03-29T10:09:44.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.248+0000 7fc9b0600700 1 -- 172.21.15.12:0/3809547971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a8157f10 con 0x7fc9a80d1630 2024-03-29T10:09:44.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.248+0000 7fc9a5600700 1 -- 172.21.15.12:0/3809547971 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c012b00 con 0x7fc9a80d1630 2024-03-29T10:09:44.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.248+0000 7fc9b0600700 1 -- 172.21.15.12:0/3809547971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a81584d0 con 0x7fc9a80d1630 2024-03-29T10:09:44.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.250+0000 7fc9a5600700 1 -- 172.21.15.12:0/3809547971 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc99c012c60 con 0x7fc9a80d1630 2024-03-29T10:09:44.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.251+0000 7fc9a5600700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc98806adb0 0x7fc98806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.251+0000 7fc9a5600700 1 -- 172.21.15.12:0/3809547971 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc99c089800 con 0x7fc9a80d1630 2024-03-29T10:09:44.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.251+0000 7fc9ad200700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc98806adb0 0x7fc98806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.252+0000 7fc9ad200700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc98806adb0 0x7fc98806d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc994007fd0 tx=0x7fc994009040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.253+0000 7fc9c4155b80 1 Processor -- start 2024-03-29T10:09:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.253+0000 7fc9c4155b80 1 -- start start 2024-03-29T10:09:44.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c31be0 0x55bcd2c34a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c5ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2ba9650 con 0x55bcd2c5e830 2024-03-29T10:09:44.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2b8ac30 con 0x55bcd2c2fb60 2024-03-29T10:09:44.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2b88430 con 0x55bcd2c31be0 2024-03-29T10:09:44.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9adc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9adc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48124/0 (socket says 172.21.15.12:48124) 2024-03-29T10:09:44.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9adc00700 1 -- 172.21.15.12:0/212207547 learned_addr learned my addr 172.21.15.12:0/212207547 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:44.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9ad200700 1 --2- 172.21.15.12:0/212207547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c31be0 0x55bcd2c34a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9ae600700 1 --2- 172.21.15.12:0/212207547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c5ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9adc00700 1 -- 172.21.15.12:0/212207547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c31be0 msgr2=0x55bcd2c34a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9adc00700 1 --2- 172.21.15.12:0/212207547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c31be0 0x55bcd2c34a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9adc00700 1 -- 172.21.15.12:0/212207547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 msgr2=0x55bcd2c5ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9adc00700 1 --2- 172.21.15.12:0/212207547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c5ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9adc00700 1 -- 172.21.15.12:0/212207547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcd2977790 con 0x55bcd2c5e830 2024-03-29T10:09:44.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.254+0000 7fc9ae600700 1 --2- 172.21.15.12:0/212207547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c5ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:44.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.255+0000 7fc9adc00700 1 --2- 172.21.15.12:0/212207547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5ec10 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0549f0 tx=0x7fc99c057f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a0f43f6b83ce09 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:44.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.255+0000 7fc99a000700 1 -- 172.21.15.12:0/212207547 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c05c320 con 0x55bcd2c5e830 2024-03-29T10:09:44.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.255+0000 7fc99a000700 1 -- 172.21.15.12:0/212207547 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c0608e0 con 0x55bcd2c5e830 2024-03-29T10:09:44.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.255+0000 7fc99a000700 1 -- 172.21.15.12:0/212207547 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00f540 con 0x55bcd2c5e830 2024-03-29T10:09:44.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.255+0000 7fc9c4155b80 1 -- 172.21.15.12:0/212207547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 msgr2=0x55bcd2c5ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.255+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/212207547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5ec10 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0549f0 tx=0x7fc99c057f20 comp rx=0 tx=0).stop 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.256+0000 7fc9c4155b80 1 -- 172.21.15.12:0/212207547 shutdown_connections 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.256+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/212207547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c5ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.256+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/212207547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5ec10 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.256+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/212207547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c31be0 0x55bcd2c34a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.256+0000 7fc9c4155b80 1 -- 172.21.15.12:0/212207547 >> 172.21.15.12:0/212207547 conn(0x55bcd2b926a0 msgr2=0x55bcd2c79640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.256+0000 7fc9c4155b80 1 -- 172.21.15.12:0/212207547 shutdown_connections 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.256+0000 7fc9c4155b80 1 -- 172.21.15.12:0/212207547 wait complete. 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.256+0000 7fc9c4155b80 1 -- start start 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.257+0000 7fc9c4155b80 1 Processor -- start 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.258+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c621b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2c5caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c5e830 0x55bcd2c8c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2b83d20 con 0x55bcd2c31be0 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2b1f770 con 0x55bcd2c2fb60 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2b426f0 con 0x55bcd2c5e830 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9c4155b80 10 client.0 _ll_register_callbacks cb 0x55bcd2b295e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9ad200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2c5caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9ad200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2c5caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48132/0 (socket says 172.21.15.12:48132) 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9ad200700 1 -- 172.21.15.12:0/2935823683 learned_addr learned my addr 172.21.15.12:0/2935823683 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9adc00700 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c621b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9ae600700 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c5e830 0x55bcd2c8c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9ad200700 1 -- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c5e830 msgr2=0x55bcd2c8c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9ad200700 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c5e830 0x55bcd2c8c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9ad200700 1 -- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 msgr2=0x55bcd2c621b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9ad200700 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c621b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9ad200700 1 -- 172.21.15.12:0/2935823683 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a81584d0 con 0x55bcd2c31be0 2024-03-29T10:09:44.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.259+0000 7fc9ae600700 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c5e830 0x55bcd2c8c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.260+0000 7fc9ad200700 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2c5caf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc994010010 tx=0x7fc994001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4c6a63868a09cb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.260+0000 7fc98ea00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.260+0000 7fc98ea00700 1 -- 172.21.15.12:0/2935823683 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994014ca0 con 0x55bcd2c31be0 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.260+0000 7fc9c4155b80 1 -- 172.21.15.12:0/2935823683 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bcd2c26920 con 0x55bcd2c31be0 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.260+0000 7fc98ea00700 1 -- 172.21.15.12:0/2935823683 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc99401c040 con 0x55bcd2c31be0 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.260+0000 7fc98ea00700 1 -- 172.21.15.12:0/2935823683 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc994014e00 con 0x55bcd2c31be0 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.260+0000 7fc9c4155b80 10 client.6552 fetch_fsmap learned FSMap version 31 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.260+0000 7fc9c4155b80 1 -- 172.21.15.12:0/2935823683 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x55bcd2a640a0 con 0x55bcd2c31be0 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.260+0000 7fc98ea00700 1 -- 172.21.15.12:0/2935823683 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bcd2c26920 con 0x55bcd2c31be0 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.261+0000 7fc98ea00700 1 -- 172.21.15.12:0/2935823683 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 31) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc994021400 con 0x55bcd2c31be0 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.261+0000 7fc9c4155b80 10 client.6552 fetch_fsmap finished waiting for FSMap version 31 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.261+0000 7fc9c4155b80 10 client.6552 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.261+0000 7fc9c4155b80 20 client.6552 populate_metadata read hostname 'smithi012' 2024-03-29T10:09:44.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.261+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:6836/4289275091,v1:172.21.15.159:6837/4289275091] conn(0x55bcd2c89d70 0x55bcd2c8a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.261+0000 7fc9c4155b80 4 client.6552 mds_command: new command op to 6241 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:09:44.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.261+0000 7fc9c4155b80 1 -- 172.21.15.12:0/2935823683 --> [v2:172.21.15.159:6836/4289275091,v1:172.21.15.159:6837/4289275091] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bcd2c036f0 con 0x55bcd2c89d70 2024-03-29T10:09:44.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.261+0000 7fc9adc00700 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:6836/4289275091,v1:172.21.15.159:6837/4289275091] conn(0x55bcd2c89d70 0x55bcd2c8a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.262+0000 7fc9adc00700 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:6836/4289275091,v1:172.21.15.159:6837/4289275091] conn(0x55bcd2c89d70 0x55bcd2c8a150 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc970950cd70f9a5 server_cookie=41ee713f92e18515 in_seq=0 out_seq=0 2024-03-29T10:09:44.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.262+0000 7fc98ea00700 10 client.6552 ms_handle_connect on v2:172.21.15.159:6836/4289275091 2024-03-29T10:09:44.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc98ea00700 1 -- 172.21.15.12:0/2935823683 <== mds.0 v2:172.21.15.159:6836/4289275091 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bcd2c036f0 con 0x55bcd2c89d70 2024-03-29T10:09:44.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc98ea00700 10 client.6552 handle_command_reply: tid=0 2024-03-29T10:09:44.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 client.6552 shutdown 2024-03-29T10:09:44.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 -- 172.21.15.12:0/2935823683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 msgr2=0x55bcd2c5caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2c5caf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc994010010 tx=0x7fc994001120 comp rx=0 tx=0).stop 2024-03-29T10:09:44.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 -- 172.21.15.12:0/2935823683 shutdown_connections 2024-03-29T10:09:44.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:6836/4289275091,v1:172.21.15.159:6837/4289275091] conn(0x55bcd2c89d70 0x55bcd2c8a150 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c5e830 0x55bcd2c8c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c621b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/2935823683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2c5caf0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 -- 172.21.15.12:0/2935823683 >> 172.21.15.12:0/2935823683 conn(0x55bcd2b926a0 msgr2=0x55bcd2c7c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:44.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc98ea00700 0 client.6552 ms_handle_reset on v2:172.21.15.159:6836/4289275091 2024-03-29T10:09:44.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 -- 172.21.15.12:0/2935823683 shutdown_connections 2024-03-29T10:09:44.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.263+0000 7fc9c4155b80 1 -- 172.21.15.12:0/2935823683 wait complete. 2024-03-29T10:09:44.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.264+0000 7fc9c4155b80 20 client.6552 trim_cache size 0 max 16384 2024-03-29T10:09:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.282+0000 7fc9c4155b80 1 Processor -- start 2024-03-29T10:09:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.282+0000 7fc9c4155b80 1 -- start start 2024-03-29T10:09:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c2d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c31be0 0x55bcd2c2dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2978390 con 0x55bcd2c5e830 2024-03-29T10:09:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2c89090 con 0x55bcd2c2fb60 2024-03-29T10:09:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2c5b530 con 0x55bcd2c31be0 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9ae600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9ae600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48146/0 (socket says 172.21.15.12:48146) 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9ae600700 1 -- 172.21.15.12:0/3270248925 learned_addr learned my addr 172.21.15.12:0/3270248925 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9adc00700 1 --2- 172.21.15.12:0/3270248925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c2d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9ae600700 1 -- 172.21.15.12:0/3270248925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c31be0 msgr2=0x55bcd2c2dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9ae600700 1 --2- 172.21.15.12:0/3270248925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c31be0 0x55bcd2c2dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.283+0000 7fc9ae600700 1 -- 172.21.15.12:0/3270248925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 msgr2=0x55bcd2c2d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.284+0000 7fc9ae600700 1 --2- 172.21.15.12:0/3270248925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c2d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.284+0000 7fc9ae600700 1 -- 172.21.15.12:0/3270248925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcd2b4a010 con 0x55bcd2c5e830 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.284+0000 7fc9adc00700 1 --2- 172.21.15.12:0/3270248925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c2d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:09:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.284+0000 7fc9ae600700 1 --2- 172.21.15.12:0/3270248925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5c8d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0012000 tx=0x7fc9a0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6fa21fc23b533d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:44.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.284+0000 7fc99aa00700 1 -- 172.21.15.12:0/3270248925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a000bd70 con 0x55bcd2c5e830 2024-03-29T10:09:44.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.284+0000 7fc99aa00700 1 -- 172.21.15.12:0/3270248925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a0021070 con 0x55bcd2c5e830 2024-03-29T10:09:44.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.284+0000 7fc99aa00700 1 -- 172.21.15.12:0/3270248925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0003e60 con 0x55bcd2c5e830 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.285+0000 7fc9c4155b80 1 -- 172.21.15.12:0/3270248925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 msgr2=0x55bcd2c5c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.285+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/3270248925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5c8d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0012000 tx=0x7fc9a0010000 comp rx=0 tx=0).stop 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.285+0000 7fc9c4155b80 1 -- 172.21.15.12:0/3270248925 shutdown_connections 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.285+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/3270248925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c5e830 0x55bcd2c5c8d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.285+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/3270248925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c2fb60 0x55bcd2c2d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.285+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/3270248925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c31be0 0x55bcd2c2dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.285+0000 7fc9c4155b80 1 -- 172.21.15.12:0/3270248925 >> 172.21.15.12:0/3270248925 conn(0x55bcd2b926a0 msgr2=0x55bcd2c8c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.285+0000 7fc9c4155b80 1 -- 172.21.15.12:0/3270248925 shutdown_connections 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.285+0000 7fc9c4155b80 1 -- 172.21.15.12:0/3270248925 wait complete. 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.285+0000 7fc9c4155b80 1 -- start start 2024-03-29T10:09:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.286+0000 7fc9c4155b80 1 Processor -- start 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c2fb60 0x55bcd2c7ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2cdfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9c4155b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c5e830 0x55bcd2ce0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2c89b80 con 0x55bcd2c31be0 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2ce0b80 con 0x55bcd2c5e830 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9c4155b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcd2c89cf0 con 0x55bcd2c2fb60 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9c4155b80 10 client.0 _ll_register_callbacks cb 0x7fc99400fd10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9ad200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2cdfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9ad200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2cdfdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48152/0 (socket says 172.21.15.12:48152) 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9ad200700 1 -- 172.21.15.12:0/925259687 learned_addr learned my addr 172.21.15.12:0/925259687 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9ae600700 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c5e830 0x55bcd2ce0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9adc00700 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c2fb60 0x55bcd2c7ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9ad200700 1 -- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c2fb60 msgr2=0x55bcd2c7ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9ad200700 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c2fb60 0x55bcd2c7ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9ad200700 1 -- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c5e830 msgr2=0x55bcd2ce0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9ad200700 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c5e830 0x55bcd2ce0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9ad200700 1 -- 172.21.15.12:0/925259687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a80cf420 con 0x55bcd2c31be0 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.287+0000 7fc9adc00700 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c2fb60 0x55bcd2c7ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc9ad200700 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2cdfdf0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc9940015a0 tx=0x7fc994007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8193a3a801a849f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc98ea00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc98ea00700 1 -- 172.21.15.12:0/925259687 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994021da0 con 0x55bcd2c31be0 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc98ea00700 1 -- 172.21.15.12:0/925259687 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc994021960 con 0x55bcd2c31be0 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc9c4155b80 1 -- 172.21.15.12:0/925259687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc99c0608e0 con 0x55bcd2c31be0 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc98ea00700 1 -- 172.21.15.12:0/925259687 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9940148e0 con 0x55bcd2c31be0 2024-03-29T10:09:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc9c4155b80 10 client.6558 fetch_fsmap learned FSMap version 31 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc9c4155b80 1 -- 172.21.15.12:0/925259687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x55bcd29d81d0 con 0x55bcd2c31be0 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc98ea00700 1 -- 172.21.15.12:0/925259687 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c0608e0 con 0x55bcd2c31be0 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc98ea00700 1 -- 172.21.15.12:0/925259687 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 31) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc99403a070 con 0x55bcd2c31be0 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc9c4155b80 10 client.6558 fetch_fsmap finished waiting for FSMap version 31 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc9c4155b80 10 client.6558 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc9c4155b80 20 client.6558 populate_metadata read hostname 'smithi012' 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:6836/4289275091,v1:172.21.15.159:6837/4289275091] conn(0x55bcd2c60cf0 0x55bcd2c610d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc9c4155b80 4 client.6558 mds_command: new command op to 6241 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.288+0000 7fc9c4155b80 1 -- 172.21.15.12:0/925259687 --> [v2:172.21.15.159:6836/4289275091,v1:172.21.15.159:6837/4289275091] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bcd29bd040 con 0x55bcd2c60cf0 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.289+0000 7fc9adc00700 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:6836/4289275091,v1:172.21.15.159:6837/4289275091] conn(0x55bcd2c60cf0 0x55bcd2c610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.289+0000 7fc98ea00700 10 client.6558 ms_handle_connect on v2:172.21.15.159:6836/4289275091 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.289+0000 7fc9adc00700 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:6836/4289275091,v1:172.21.15.159:6837/4289275091] conn(0x55bcd2c60cf0 0x55bcd2c610d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1fe4456107a86a0 server_cookie=8ed9f360528d1978 in_seq=0 out_seq=0 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc98ea00700 1 -- 172.21.15.12:0/925259687 <== mds.0 v2:172.21.15.159:6836/4289275091 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55bcd29bd040 con 0x55bcd2c60cf0 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc98ea00700 10 client.6558 handle_command_reply: tid=0 2024-03-29T10:09:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 client.6558 shutdown 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 -- 172.21.15.12:0/925259687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 msgr2=0x55bcd2cdfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2cdfdf0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc9940015a0 tx=0x7fc994007ec0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 -- 172.21.15.12:0/925259687 shutdown_connections 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55bcd2c5e830 0x55bcd2ce0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:6836/4289275091,v1:172.21.15.159:6837/4289275091] conn(0x55bcd2c60cf0 0x55bcd2c610d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55bcd2c2fb60 0x55bcd2c7ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 --2- 172.21.15.12:0/925259687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bcd2c31be0 0x55bcd2cdfdf0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc98ea00700 0 client.6558 ms_handle_reset on v2:172.21.15.159:6836/4289275091 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 -- 172.21.15.12:0/925259687 >> 172.21.15.12:0/925259687 conn(0x55bcd2b926a0 msgr2=0x55bcd2c8a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 -- 172.21.15.12:0/925259687 shutdown_connections 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 1 -- 172.21.15.12:0/925259687 wait complete. 2024-03-29T10:09:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.290+0000 7fc9c4155b80 20 client.6558 trim_cache size 0 max 16384 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 6241, 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 11.689734816, 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 31, 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 80, 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 80, 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 17.448868730000001 2024-03-29T10:09:44.294 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:09:44.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 -- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc98806adb0 msgr2=0x7fc98806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc98806adb0 0x7fc98806d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc994007fd0 tx=0x7fc994009040 comp rx=0 tx=0).stop 2024-03-29T10:09:44.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 -- 172.21.15.12:0/3809547971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d1630 msgr2=0x7fc9a80b89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d1630 0x7fc9a80b89f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00d800 tx=0x7fc99c00db10 comp rx=0 tx=0).stop 2024-03-29T10:09:44.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 -- 172.21.15.12:0/3809547971 shutdown_connections 2024-03-29T10:09:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc98806adb0 0x7fc98806d270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a815dad0 0x7fc9a806ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9a80d1630 0x7fc9a80b89f0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 --2- 172.21.15.12:0/3809547971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9a80d21a0 0x7fc9a814d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 -- 172.21.15.12:0/3809547971 >> 172.21.15.12:0/3809547971 conn(0x7fc9a808aff0 msgr2=0x7fc9a815baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 -- 172.21.15.12:0/3809547971 shutdown_connections 2024-03-29T10:09:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.291+0000 7fc9b0600700 1 -- 172.21.15.12:0/3809547971 wait complete. 2024-03-29T10:09:44.308 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T10:09:44.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.507+0000 7f8ae4a00700 1 Processor -- start 2024-03-29T10:09:44.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.507+0000 7f8ae4a00700 1 -- start start 2024-03-29T10:09:44.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.508+0000 7f8ae4a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8ae00b7a30 0x7f8ae01685d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.508+0000 7f8ae4a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae0160fc0 con 0x7f8ae0160be0 2024-03-29T10:09:44.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.508+0000 7f8ae4a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae00d21e0 con 0x7f8ae00ce590 2024-03-29T10:09:44.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.508+0000 7f8ae4a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae00b65d0 con 0x7f8ae00b7a30 2024-03-29T10:09:44.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.508+0000 7f8adde00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8ae0160be0 0x7f8ae00cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60814/0 (socket says 172.21.15.12:60814) 2024-03-29T10:09:44.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.508+0000 7f8adde00700 1 -- 172.21.15.12:0/1257877885 learned_addr learned my addr 172.21.15.12:0/1257877885 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:44.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8ade800700 1 --2- 172.21.15.12:0/1257877885 >> v2:172.21.15.159:3300/0 conn(0x7f8ae00b7a30 0x7f8ae01685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710210901 0 0) 0x7f8ae0160fc0 con 0x7f8ae0160be0 2024-03-29T10:09:44.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac40035b0 con 0x7f8ae0160be0 2024-03-29T10:09:44.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2952444473 0 0) 0x7f8ac40035b0 con 0x7f8ae0160be0 2024-03-29T10:09:44.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 >> v2:172.21.15.159:3300/0 conn(0x7f8ae00b7a30 msgr2=0x7f8ae01685d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 --2- 172.21.15.12:0/1257877885 >> v2:172.21.15.159:3300/0 conn(0x7f8ae00b7a30 0x7f8ae01685d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 >> v1:172.21.15.159:6789/0 conn(0x7f8ae00ce590 legacy=0x7f8ae00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae00d24c0 con 0x7f8ae0160be0 2024-03-29T10:09:44.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f8ad0001670 con 0x7f8ae0160be0 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 >> v1:172.21.15.12:6789/0 conn(0x7f8ae0160be0 legacy=0x7f8ae00cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 0x7f8ac4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac4004850 0x7f8ac4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ac4005210 0x7f8ac400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00b65d0 con 0x7f8ac4004850 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac400c0b0 con 0x7f8ac4003a00 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.509+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac400c270 con 0x7f8ac4005210 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8adde00700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 0x7f8ac4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8add400700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac4004850 0x7f8ac4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8ade800700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ac4005210 0x7f8ac400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8adde00700 1 -- 172.21.15.12:0/1257877885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ac4005210 msgr2=0x7f8ac400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8adde00700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ac4005210 0x7f8ac400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8adde00700 1 -- 172.21.15.12:0/1257877885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac4004850 msgr2=0x7f8ac4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8adde00700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac4004850 0x7f8ac4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8adde00700 1 -- 172.21.15.12:0/1257877885 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac40035b0 con 0x7f8ac4003a00 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8add400700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac4004850 0x7f8ac4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8ade800700 1 -- 172.21.15.12:0/1257877885 reap_dead start 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.510+0000 7f8adde00700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 0x7f8ac4003e20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000c240 tx=0x7f8ac4005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=178bbeadd0f8821a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.511+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0007c10 con 0x7f8ac4003a00 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.511+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad000ce00 con 0x7f8ac4003a00 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.511+0000 7f8adca00700 1 -- 172.21.15.12:0/1257877885 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0005ba0 con 0x7f8ac4003a00 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.511+0000 7f8ae4a00700 1 -- 172.21.15.12:0/1257877885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 msgr2=0x7f8ac4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.511+0000 7f8ae4a00700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 0x7f8ac4003e20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000c240 tx=0x7f8ac4005880 comp rx=0 tx=0).stop 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.511+0000 7f8ae4a00700 1 -- 172.21.15.12:0/1257877885 shutdown_connections 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.511+0000 7f8ae4a00700 1 --2- 172.21.15.12:0/1257877885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 0x7f8ac4003e20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.511+0000 7f8ae4a00700 1 -- 172.21.15.12:0/1257877885 >> 172.21.15.12:0/1257877885 conn(0x7f8ae008aff0 msgr2=0x7f8ae00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.511+0000 7f8ae4a00700 1 -- 172.21.15.12:0/1257877885 shutdown_connections 2024-03-29T10:09:44.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.511+0000 7f8ae4a00700 1 -- 172.21.15.12:0/1257877885 wait complete. 2024-03-29T10:09:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.514+0000 7f8ae4a00700 1 Processor -- start 2024-03-29T10:09:44.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.514+0000 7f8ae4a00700 1 -- start start 2024-03-29T10:09:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.515+0000 7f8ae4a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 0x7f8ae00a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.515+0000 7f8ae4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae01f6f80 0x7f8ae00b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.515+0000 7f8ae4a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae01f26d0 0x7f8ae00cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.515+0000 7f8ae4a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae01f7400 con 0x7f8ae01f6f80 2024-03-29T10:09:44.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.515+0000 7f8ae4a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00b0660 con 0x7f8ac4003a00 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.515+0000 7f8ae4a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae0152cc0 con 0x7f8ae01f26d0 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.515+0000 7f8add400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae01f6f80 0x7f8ae00b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8add400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae01f6f80 0x7f8ae00b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48180/0 (socket says 172.21.15.12:48180) 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8add400700 1 -- 172.21.15.12:0/2670078360 learned_addr learned my addr 172.21.15.12:0/2670078360 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8adde00700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 0x7f8ae00a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8ade800700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae01f26d0 0x7f8ae00cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8add400700 1 -- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae01f26d0 msgr2=0x7f8ae00cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8add400700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae01f26d0 0x7f8ae00cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8add400700 1 -- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 msgr2=0x7f8ae00a0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8add400700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 0x7f8ae00a0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8add400700 1 -- 172.21.15.12:0/2670078360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0152fa0 con 0x7f8ae01f6f80 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8adde00700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 0x7f8ae00a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8add400700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae01f6f80 0x7f8ae00b0120 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00f7e0 tx=0x7f8acc00fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8ad6000700 1 -- 172.21.15.12:0/2670078360 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc0052e0 con 0x7f8ae01f6f80 2024-03-29T10:09:44.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8ad6000700 1 -- 172.21.15.12:0/2670078360 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc005440 con 0x7f8ae01f6f80 2024-03-29T10:09:44.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8ae4a00700 1 -- 172.21.15.12:0/2670078360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae01f64a0 con 0x7f8ae01f6f80 2024-03-29T10:09:44.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.516+0000 7f8ad6000700 1 -- 172.21.15.12:0/2670078360 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc0038c0 con 0x7f8ae01f6f80 2024-03-29T10:09:44.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.517+0000 7f8ae4a00700 1 -- 172.21.15.12:0/2670078360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae01f6a60 con 0x7f8ae01f6f80 2024-03-29T10:09:44.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.518+0000 7f8ae4a00700 1 -- 172.21.15.12:0/2670078360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae01f6bf0 con 0x7f8ae01f6f80 2024-03-29T10:09:44.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.519+0000 7f8ad6000700 1 -- 172.21.15.12:0/2670078360 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8acc003a20 con 0x7f8ae01f6f80 2024-03-29T10:09:44.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.520+0000 7f8ad6000700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac006adb0 0x7f8ac006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.520+0000 7f8ad6000700 1 -- 172.21.15.12:0/2670078360 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8acc089210 con 0x7f8ae01f6f80 2024-03-29T10:09:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.520+0000 7f8adde00700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac006adb0 0x7f8ac006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:44.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.521+0000 7f8adde00700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac006adb0 0x7f8ac006d270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0015040 tx=0x7f8ad0007000 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:44.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.524+0000 7f8ad6000700 1 -- 172.21.15.12:0/2670078360 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8acc0542e0 con 0x7f8ae01f6f80 2024-03-29T10:09:44.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.784+0000 7f8ae4a00700 1 -- 172.21.15.12:0/2670078360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8ae0067720 con 0x7f8ae01f6f80 2024-03-29T10:09:44.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.784+0000 7f8ad6000700 1 -- 172.21.15.12:0/2670078360 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v31) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8acc057900 con 0x7f8ae01f6f80 2024-03-29T10:09:44.784 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 -- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac006adb0 msgr2=0x7f8ac006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac006adb0 0x7f8ac006d270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0015040 tx=0x7f8ad0007000 comp rx=0 tx=0).stop 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 -- 172.21.15.12:0/2670078360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae01f6f80 msgr2=0x7f8ae00b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae01f6f80 0x7f8ae00b0120 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00f7e0 tx=0x7f8acc00fba0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 -- 172.21.15.12:0/2670078360 shutdown_connections 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac006adb0 0x7f8ac006d270 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae01f26d0 0x7f8ae00cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ac4003a00 0x7f8ae00a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 --2- 172.21.15.12:0/2670078360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae01f6f80 0x7f8ae00b0120 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 -- 172.21.15.12:0/2670078360 >> 172.21.15.12:0/2670078360 conn(0x7f8ae008aff0 msgr2=0x7f8ae00c7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 -- 172.21.15.12:0/2670078360 shutdown_connections 2024-03-29T10:09:44.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:44.787+0000 7f8ae4a00700 1 -- 172.21.15.12:0/2670078360 wait complete. 2024-03-29T10:09:44.805 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T10:09:44.805 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T10:09:44.805 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T10:09:44.805 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T10:09:44.805 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T10:09:44.805 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T10:09:44.806 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T10:09:44.806 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T10:09:44.806 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:09:44.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:44.806 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:09:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 40533sec preferred_lft 40533sec 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T10:09:44.853 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T10:09:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:09:44.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:44.857 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T10:09:44.957 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T10:09:44.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:44.960 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T10:09:45.007 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T10:09:45.007 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T10:09:45.007 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T10:09:45.007 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T10:09:45.007 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T10:09:45.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:45.008 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T10:09:45.008 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T10:09:45.008 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T10:09:45.008 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T10:09:45.008 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T10:09:45.008 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T10:09:45.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:45.212 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T10:09:45.256 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T10:09:45.256 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T10:09:45.257 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:09:45.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:45.257 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T10:09:45.365 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:09:45.365 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T10:09:45.409 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:09:45.409 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:09:45.409 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:09:45.493 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T10:09:45.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:45.493 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:09:45.635 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:09:45.635 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:09:45.663 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:09:45.663 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:09:45.663 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T10:09:45.663 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T10:09:45.663 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T10:09:45.669 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T10:09:45.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:45.669 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:09:45.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:45.736 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T10:09:45.846 INFO:teuthology.orchestra.run.smithi012.stdout:client6567 2024-03-29T10:09:45.847 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:09:45.847 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:45.892 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T10:09:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.130+0000 7fa20d600700 1 Processor -- start 2024-03-29T10:09:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.130+0000 7fa20d600700 1 -- start start 2024-03-29T10:09:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa20d600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa20815d710 0x7fa2080d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa20d600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2080d1b70 0x7fa2080d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa20d600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2080cff10 0x7fa2080d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa20d600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2080cf220 con 0x7fa2080cff10 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa20d600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2080d2530 con 0x7fa2080d1b70 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa20d600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080cf640 con 0x7fa20815d710 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa207400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2080cff10 0x7fa2080d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa207400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2080cff10 0x7fa2080d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48194/0 (socket says 172.21.15.12:48194) 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa207400700 1 -- 172.21.15.12:0/2238160785 learned_addr learned my addr 172.21.15.12:0/2238160785 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa206a00700 1 --2- 172.21.15.12:0/2238160785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa20815d710 0x7fa2080d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa206000700 1 --2- 172.21.15.12:0/2238160785 >> v2:172.21.15.159:3300/0 conn(0x7fa2080d1b70 0x7fa2080d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa207400700 1 -- 172.21.15.12:0/2238160785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa20815d710 msgr2=0x7fa2080d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa207400700 1 --2- 172.21.15.12:0/2238160785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa20815d710 0x7fa2080d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa207400700 1 -- 172.21.15.12:0/2238160785 >> v2:172.21.15.159:3300/0 conn(0x7fa2080d1b70 msgr2=0x7fa2080d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.132+0000 7fa207400700 1 --2- 172.21.15.12:0/2238160785 >> v2:172.21.15.159:3300/0 conn(0x7fa2080d1b70 0x7fa2080d1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.133+0000 7fa207400700 1 -- 172.21.15.12:0/2238160785 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2080ce5a0 con 0x7fa2080cff10 2024-03-29T10:09:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.133+0000 7fa207400700 1 --2- 172.21.15.12:0/2238160785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 0x7fa2080d0380 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa20808c1c0 tx=0x7fa20000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8293393794d84206 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.133+0000 7fa205600700 1 -- 172.21.15.12:0/2238160785 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20000e070 con 0x7fa2080cff10 2024-03-29T10:09:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.133+0000 7fa205600700 1 -- 172.21.15.12:0/2238160785 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20000b910 con 0x7fa2080cff10 2024-03-29T10:09:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.133+0000 7fa205600700 1 -- 172.21.15.12:0/2238160785 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20000bcc0 con 0x7fa2080cff10 2024-03-29T10:09:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.133+0000 7fa20d600700 1 -- 172.21.15.12:0/2238160785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 msgr2=0x7fa2080d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.133+0000 7fa20d600700 1 --2- 172.21.15.12:0/2238160785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 0x7fa2080d0380 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa20808c1c0 tx=0x7fa20000b330 comp rx=0 tx=0).stop 2024-03-29T10:09:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.134+0000 7fa20d600700 1 -- 172.21.15.12:0/2238160785 shutdown_connections 2024-03-29T10:09:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.134+0000 7fa20d600700 1 --2- 172.21.15.12:0/2238160785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 0x7fa2080d0380 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.134+0000 7fa20d600700 1 --2- 172.21.15.12:0/2238160785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa20815d710 0x7fa2080d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.134+0000 7fa20d600700 1 --2- 172.21.15.12:0/2238160785 >> v2:172.21.15.159:3300/0 conn(0x7fa2080d1b70 0x7fa2080d1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.134+0000 7fa20d600700 1 -- 172.21.15.12:0/2238160785 >> 172.21.15.12:0/2238160785 conn(0x7fa20808aff0 msgr2=0x7fa2080d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.134+0000 7fa20d600700 1 -- 172.21.15.12:0/2238160785 shutdown_connections 2024-03-29T10:09:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.134+0000 7fa20d600700 1 -- 172.21.15.12:0/2238160785 wait complete. 2024-03-29T10:09:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.135+0000 7fa20d600700 1 Processor -- start 2024-03-29T10:09:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.136+0000 7fa20d600700 1 -- start start 2024-03-29T10:09:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.136+0000 7fa20d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 0x7fa2080b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.136+0000 7fa20d600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2080d1b70 0x7fa2081f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa20d600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20815d710 0x7fa20815ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa20d600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2081f3e10 con 0x7fa2080cff10 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa20d600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080b6c10 con 0x7fa20815d710 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa20d600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080b6ef0 con 0x7fa2080d1b70 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa206a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 0x7fa2080b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa206a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 0x7fa2080b6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48204/0 (socket says 172.21.15.12:48204) 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa206a00700 1 -- 172.21.15.12:0/3260224355 learned_addr learned my addr 172.21.15.12:0/3260224355 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa206000700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2080d1b70 0x7fa2081f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa207400700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20815d710 0x7fa20815ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa206a00700 1 -- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2080d1b70 msgr2=0x7fa2081f38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa206a00700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2080d1b70 0x7fa2081f38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa206a00700 1 -- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20815d710 msgr2=0x7fa20815ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa206a00700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20815d710 0x7fa20815ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa206a00700 1 -- 172.21.15.12:0/3260224355 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa200009b30 con 0x7fa2080cff10 2024-03-29T10:09:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa207400700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20815d710 0x7fa20815ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.137+0000 7fa206a00700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 0x7fa2080b6550 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800e9b0 tx=0x7fa1f800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.138+0000 7fa1fea00700 1 -- 172.21.15.12:0/3260224355 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8009710 con 0x7fa2080cff10 2024-03-29T10:09:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.138+0000 7fa1fea00700 1 -- 172.21.15.12:0/3260224355 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f800c850 con 0x7fa2080cff10 2024-03-29T10:09:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.138+0000 7fa1fea00700 1 -- 172.21.15.12:0/3260224355 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f800cbc0 con 0x7fa2080cff10 2024-03-29T10:09:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.138+0000 7fa20d600700 1 -- 172.21.15.12:0/3260224355 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2080b71d0 con 0x7fa2080cff10 2024-03-29T10:09:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.138+0000 7fa20d600700 1 -- 172.21.15.12:0/3260224355 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2081604f0 con 0x7fa2080cff10 2024-03-29T10:09:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.139+0000 7fa20d600700 1 -- 172.21.15.12:0/3260224355 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa208067e30 con 0x7fa2080cff10 2024-03-29T10:09:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.140+0000 7fa1fea00700 1 -- 172.21.15.12:0/3260224355 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa1f800cd20 con 0x7fa2080cff10 2024-03-29T10:09:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.141+0000 7fa1fea00700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 0x7fa1e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.141+0000 7fa1fea00700 1 -- 172.21.15.12:0/3260224355 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1f80895b0 con 0x7fa2080cff10 2024-03-29T10:09:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.141+0000 7fa206000700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 0x7fa1e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.142+0000 7fa206000700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 0x7fa1e806d270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa2080c6a70 tx=0x7fa1f4008040 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.145+0000 7fa1fea00700 1 -- 172.21.15.12:0/3260224355 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa1f8054670 con 0x7fa2080cff10 2024-03-29T10:09:46.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.387+0000 7fa20d600700 1 -- 172.21.15.12:0/3260224355 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2080cee40 con 0x7fa1e806adb0 2024-03-29T10:09:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.390+0000 7fa1fea00700 1 -- 172.21.15.12:0/3260224355 <== mgr.6013 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa1f4009760 con 0x7fa1e806adb0 2024-03-29T10:09:46.389 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:09:46.389 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:09:46.389 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T10:09:46.389 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:09:46.389 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 -- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 msgr2=0x7fa1e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 0x7fa1e806d270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa2080c6a70 tx=0x7fa1f4008040 comp rx=0 tx=0).stop 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 -- 172.21.15.12:0/3260224355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 msgr2=0x7fa2080b6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 0x7fa2080b6550 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800e9b0 tx=0x7fa1f800ed70 comp rx=0 tx=0).stop 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 -- 172.21.15.12:0/3260224355 shutdown_connections 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 0x7fa1e806d270 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20815d710 0x7fa20815ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cff10 0x7fa2080b6550 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 --2- 172.21.15.12:0/3260224355 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2080d1b70 0x7fa2081f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 -- 172.21.15.12:0/3260224355 >> 172.21.15.12:0/3260224355 conn(0x7fa20808aff0 msgr2=0x7fa2080b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 -- 172.21.15.12:0/3260224355 shutdown_connections 2024-03-29T10:09:46.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.392+0000 7fa20d600700 1 -- 172.21.15.12:0/3260224355 wait complete. 2024-03-29T10:09:46.409 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T10:09:46.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.609+0000 7f7851600700 1 Processor -- start 2024-03-29T10:09:46.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.610+0000 7f7851600700 1 -- start start 2024-03-29T10:09:46.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.611+0000 7f7851600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 0x7f784c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.611+0000 7f7851600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f784c0d2370 0x7f784c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.611+0000 7f7851600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784c161420 con 0x7f784c15d9d0 2024-03-29T10:09:46.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.611+0000 7f7851600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784c0d2820 con 0x7f784c0d2370 2024-03-29T10:09:46.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.611+0000 7f7851600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0d4340 con 0x7f784c0d16a0 2024-03-29T10:09:46.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.612+0000 7f784a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 0x7f784c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.612+0000 7f784b400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f784c0d2370 0x7f784c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.612+0000 7f784aa00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f784c15d9d0 0x7f784c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:37902/0 (socket says 172.21.15.12:37902) 2024-03-29T10:09:46.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.612+0000 7f784a000700 1 --2- 172.21.15.12:0/3126562695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 0x7f784c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:43702/0 (socket says 172.21.15.12:43702) 2024-03-29T10:09:46.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.612+0000 7f784aa00700 1 -- 172.21.15.12:0/3126562695 learned_addr learned my addr 172.21.15.12:0/3126562695 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:46.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.613+0000 7f784a000700 1 -- 172.21.15.12:0/3126562695 >> v2:172.21.15.159:3300/0 conn(0x7f784c0d2370 msgr2=0x7f784c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.613+0000 7f784a000700 1 --2- 172.21.15.12:0/3126562695 >> v2:172.21.15.159:3300/0 conn(0x7f784c0d2370 0x7f784c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.613+0000 7f784a000700 1 -- 172.21.15.12:0/3126562695 >> v1:172.21.15.159:6789/0 conn(0x7f784c15d9d0 legacy=0x7f784c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.613+0000 7f784a000700 1 -- 172.21.15.12:0/3126562695 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c161700 con 0x7f784c0d16a0 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.613+0000 7f7849600700 1 -- 172.21.15.12:0/3126562695 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185930796 0 0) 0x7f784c161420 con 0x7f784c15d9d0 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.613+0000 7f784a000700 1 --2- 172.21.15.12:0/3126562695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 0x7f784c0d1e30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7838009820 tx=0x7f7838009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ef0fe3003bdd454 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.613+0000 7f7849600700 1 -- 172.21.15.12:0/3126562695 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838005e30 con 0x7f784c0d16a0 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.613+0000 7f7849600700 1 -- 172.21.15.12:0/3126562695 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7838006030 con 0x7f784c0d16a0 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.613+0000 7f7849600700 1 -- 172.21.15.12:0/3126562695 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380063c0 con 0x7f784c0d16a0 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.614+0000 7f7851600700 1 -- 172.21.15.12:0/3126562695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 msgr2=0x7f784c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.614+0000 7f7851600700 1 --2- 172.21.15.12:0/3126562695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 0x7f784c0d1e30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7838009820 tx=0x7f7838009b30 comp rx=0 tx=0).stop 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.614+0000 7f7851600700 1 -- 172.21.15.12:0/3126562695 shutdown_connections 2024-03-29T10:09:46.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.614+0000 7f7851600700 1 --2- 172.21.15.12:0/3126562695 >> v2:172.21.15.159:3300/0 conn(0x7f784c0d2370 0x7f784c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.614+0000 7f7851600700 1 --2- 172.21.15.12:0/3126562695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 0x7f784c0d1e30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.614+0000 7f7851600700 1 -- 172.21.15.12:0/3126562695 >> 172.21.15.12:0/3126562695 conn(0x7f784c08aff0 msgr2=0x7f784c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:46.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.614+0000 7f7851600700 1 -- 172.21.15.12:0/3126562695 shutdown_connections 2024-03-29T10:09:46.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.614+0000 7f7851600700 1 -- 172.21.15.12:0/3126562695 wait complete. 2024-03-29T10:09:46.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.616+0000 7f7851600700 1 Processor -- start 2024-03-29T10:09:46.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.616+0000 7f7851600700 1 -- start start 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f7851600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 0x7f784c158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f7851600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784c0d2370 0x7f784c0cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f7851600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f784c15d9d0 0x7f784c0b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f7851600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0cd280 con 0x7f784c0d2370 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f7851600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0cc670 con 0x7f784c15d9d0 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f7851600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0cc950 con 0x7f784c0d16a0 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f784a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784c0d2370 0x7f784c0cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f784a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784c0d2370 0x7f784c0cbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48214/0 (socket says 172.21.15.12:48214) 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f784a000700 1 -- 172.21.15.12:0/395924203 learned_addr learned my addr 172.21.15.12:0/395924203 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f784aa00700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 0x7f784c158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f784b400700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f784c15d9d0 0x7f784c0b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f784a000700 1 -- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 msgr2=0x7f784c158ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:46.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f784a000700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 0x7f784c158ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.617+0000 7f784a000700 1 -- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f784c15d9d0 msgr2=0x7f784c0b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:46.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.618+0000 7f784a000700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f784c15d9d0 0x7f784c0b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:46.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.618+0000 7f784a000700 1 -- 172.21.15.12:0/395924203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838009430 con 0x7f784c0d2370 2024-03-29T10:09:46.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.618+0000 7f784a000700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784c0d2370 0x7f784c0cbbf0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7838000c00 tx=0x7f7838004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:46.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.618+0000 7f7842a00700 1 -- 172.21.15.12:0/395924203 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380036a0 con 0x7f784c0d2370 2024-03-29T10:09:46.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.618+0000 7f7842a00700 1 -- 172.21.15.12:0/395924203 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7838020070 con 0x7f784c0d2370 2024-03-29T10:09:46.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.618+0000 7f7851600700 1 -- 172.21.15.12:0/395924203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784c0ccc30 con 0x7f784c0d2370 2024-03-29T10:09:46.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.618+0000 7f7842a00700 1 -- 172.21.15.12:0/395924203 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838003a40 con 0x7f784c0d2370 2024-03-29T10:09:46.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.618+0000 7f7851600700 1 -- 172.21.15.12:0/395924203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784c1590e0 con 0x7f784c0d2370 2024-03-29T10:09:46.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.619+0000 7f7851600700 1 -- 172.21.15.12:0/395924203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f784c0d1ca0 con 0x7f784c0d2370 2024-03-29T10:09:46.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.620+0000 7f7842a00700 1 -- 172.21.15.12:0/395924203 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7838022030 con 0x7f784c0d2370 2024-03-29T10:09:46.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.621+0000 7f7842a00700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f782c06adb0 0x7f782c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:46.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.621+0000 7f7842a00700 1 -- 172.21.15.12:0/395924203 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f783808a780 con 0x7f784c0d2370 2024-03-29T10:09:46.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.621+0000 7f784aa00700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f782c06adb0 0x7f782c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:46.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.622+0000 7f784aa00700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f782c06adb0 0x7f782c06d270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f783c000f60 tx=0x7f783c004a50 comp rx=0 tx=0).ready entity=mgr.6013 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:46.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.625+0000 7f7842a00700 1 -- 172.21.15.12:0/395924203 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7838055840 con 0x7f784c0d2370 2024-03-29T10:09:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:46.905+0000 7f7851600700 1 -- 172.21.15.12:0/395924203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f784c067720 con 0x7f784c0d2370 2024-03-29T10:09:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.066+0000 7f7842a00700 1 -- 172.21.15.12:0/395924203 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 31) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f7838055c10 con 0x7f784c0d2370 2024-03-29T10:09:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.068+0000 7f7842a00700 1 -- 172.21.15.12:0/395924203 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v31) v1 ==== 92+0+0 (secure 0 0 0) 0x7f783805ce00 con 0x7f784c0d2370 2024-03-29T10:09:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 -- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f782c06adb0 msgr2=0x7f782c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f782c06adb0 0x7f782c06d270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f783c000f60 tx=0x7f783c004a50 comp rx=0 tx=0).stop 2024-03-29T10:09:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 -- 172.21.15.12:0/395924203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784c0d2370 msgr2=0x7f784c0cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784c0d2370 0x7f784c0cbbf0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7838000c00 tx=0x7f7838004750 comp rx=0 tx=0).stop 2024-03-29T10:09:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 -- 172.21.15.12:0/395924203 shutdown_connections 2024-03-29T10:09:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f782c06adb0 0x7f782c06d270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f784c15d9d0 0x7f784c0b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f784c0d16a0 0x7f784c158ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 --2- 172.21.15.12:0/395924203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784c0d2370 0x7f784c0cbbf0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 -- 172.21.15.12:0/395924203 >> 172.21.15.12:0/395924203 conn(0x7f784c08aff0 msgr2=0x7f784c0b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 -- 172.21.15.12:0/395924203 shutdown_connections 2024-03-29T10:09:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.071+0000 7f7851600700 1 -- 172.21.15.12:0/395924203 wait complete. 2024-03-29T10:09:47.097 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T10:09:47.290 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:47.290+0000 7fedf0b8b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:09:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.311+0000 7fda0c200700 1 Processor -- start 2024-03-29T10:09:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.312+0000 7fda0c200700 1 -- start start 2024-03-29T10:09:47.312 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:47.312+0000 7ff901837240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:09:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.313+0000 7fda0c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda040cd020 0x7fda040cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.313+0000 7fda0c200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fda040b2070 0x7fda040b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.313+0000 7fda0c200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040cdae0 con 0x7fda040cbb30 2024-03-29T10:09:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.313+0000 7fda0c200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda040cddc0 con 0x7fda040b2070 2024-03-29T10:09:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.313+0000 7fda0c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda040ce0a0 con 0x7fda040cd020 2024-03-29T10:09:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.313+0000 7fda09800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fda040cbb30 0x7fda040cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60832/0 (socket says 172.21.15.12:60832) 2024-03-29T10:09:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.313+0000 7fda09800700 1 -- 172.21.15.12:0/3897212053 learned_addr learned my addr 172.21.15.12:0/3897212053 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.313+0000 7fda08e00700 1 --2- 172.21.15.12:0/3897212053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda040cd020 0x7fda040cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.313+0000 7fda0a200700 1 --2- 172.21.15.12:0/3897212053 >> v2:172.21.15.159:3300/0 conn(0x7fda040b2070 0x7fda040b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.314+0000 7fda03e00700 1 -- 172.21.15.12:0/3897212053 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039739182 0 0) 0x7fda040cdae0 con 0x7fda040cbb30 2024-03-29T10:09:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.314+0000 7fda03e00700 1 -- 172.21.15.12:0/3897212053 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9ec0035b0 con 0x7fda040cbb30 2024-03-29T10:09:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.314+0000 7fda0a200700 1 -- 172.21.15.12:0/3897212053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda040cd020 msgr2=0x7fda040cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.314+0000 7fda0a200700 1 --2- 172.21.15.12:0/3897212053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda040cd020 0x7fda040cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.314+0000 7fda0a200700 1 -- 172.21.15.12:0/3897212053 >> v1:172.21.15.12:6789/0 conn(0x7fda040cbb30 legacy=0x7fda040cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.314+0000 7fda0a200700 1 -- 172.21.15.12:0/3897212053 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0406dc20 con 0x7fda040b2070 2024-03-29T10:09:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.314+0000 7fda0a200700 1 --2- 172.21.15.12:0/3897212053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 0x7fda040b2500 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009d90 tx=0x7fd9fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c8e487ad039ac90 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.314+0000 7fda03e00700 1 -- 172.21.15.12:0/3897212053 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc00e070 con 0x7fda040b2070 2024-03-29T10:09:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.314+0000 7fda03e00700 1 -- 172.21.15.12:0/3897212053 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9fc00bb20 con 0x7fda040b2070 2024-03-29T10:09:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.314+0000 7fda03e00700 1 -- 172.21.15.12:0/3897212053 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc012430 con 0x7fda040b2070 2024-03-29T10:09:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.315+0000 7fda0c200700 1 -- 172.21.15.12:0/3897212053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 msgr2=0x7fda040b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.315+0000 7fda0c200700 1 --2- 172.21.15.12:0/3897212053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 0x7fda040b2500 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009d90 tx=0x7fd9fc00b330 comp rx=0 tx=0).stop 2024-03-29T10:09:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.315+0000 7fda0c200700 1 -- 172.21.15.12:0/3897212053 shutdown_connections 2024-03-29T10:09:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.315+0000 7fda0c200700 1 --2- 172.21.15.12:0/3897212053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 0x7fda040b2500 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.315+0000 7fda0c200700 1 --2- 172.21.15.12:0/3897212053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda040cd020 0x7fda040cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.315+0000 7fda0c200700 1 -- 172.21.15.12:0/3897212053 >> 172.21.15.12:0/3897212053 conn(0x7fda040b0d70 msgr2=0x7fda040b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.315+0000 7fda0c200700 1 -- 172.21.15.12:0/3897212053 shutdown_connections 2024-03-29T10:09:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.315+0000 7fda0c200700 1 -- 172.21.15.12:0/3897212053 wait complete. 2024-03-29T10:09:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.317+0000 7fda0c200700 1 Processor -- start 2024-03-29T10:09:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.318+0000 7fda0c200700 1 -- start start 2024-03-29T10:09:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda0c200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 0x7fda04150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda0c200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda040cbb30 0x7fda04147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda0c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda040cd020 0x7fda04152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda0c200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda04154360 con 0x7fda040cbb30 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda0c200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda040ae5e0 con 0x7fda040b2070 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda0c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda041514e0 con 0x7fda040cd020 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda09800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 0x7fda04150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda08e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda040cbb30 0x7fda04147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda09800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 0x7fda04150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:59826/0 (socket says 172.21.15.12:59826) 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda08e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda040cbb30 0x7fda04147410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48234/0 (socket says 172.21.15.12:48234) 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda09800700 1 -- 172.21.15.12:0/3608190558 learned_addr learned my addr 172.21.15.12:0/3608190558 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.319+0000 7fda0a200700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda040cd020 0x7fda04152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda09800700 1 -- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda040cd020 msgr2=0x7fda04152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda09800700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda040cd020 0x7fda04152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda09800700 1 -- 172.21.15.12:0/3608190558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda040cbb30 msgr2=0x7fda04147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda09800700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda040cbb30 0x7fda04147410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda09800700 1 -- 172.21.15.12:0/3608190558 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9fc009a70 con 0x7fda040b2070 2024-03-29T10:09:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda08e00700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda040cbb30 0x7fda04147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda09800700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 0x7fda04150fa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800c700 tx=0x7fd9f800ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda01600700 1 -- 172.21.15.12:0/3608190558 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8005d70 con 0x7fda040b2070 2024-03-29T10:09:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda01600700 1 -- 172.21.15.12:0/3608190558 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f80070d0 con 0x7fda040b2070 2024-03-29T10:09:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda0c200700 1 -- 172.21.15.12:0/3608190558 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda04147950 con 0x7fda040b2070 2024-03-29T10:09:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.320+0000 7fda01600700 1 -- 172.21.15.12:0/3608190558 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8007460 con 0x7fda040b2070 2024-03-29T10:09:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.321+0000 7fda0c200700 1 -- 172.21.15.12:0/3608190558 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda04153880 con 0x7fda040b2070 2024-03-29T10:09:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.322+0000 7fda0c200700 1 -- 172.21.15.12:0/3608190558 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda04066ff0 con 0x7fda040b2070 2024-03-29T10:09:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.323+0000 7fda01600700 1 -- 172.21.15.12:0/3608190558 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 31) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fd9f80075c0 con 0x7fda040b2070 2024-03-29T10:09:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.323+0000 7fda01600700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd9e406ae00 0x7fd9e406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.324+0000 7fda01600700 1 -- 172.21.15.12:0/3608190558 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9f8088950 con 0x7fda040b2070 2024-03-29T10:09:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.324+0000 7fda08e00700 1 -- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd9e406ae00 msgr2=0x7fd9e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.324+0000 7fda08e00700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd9e406ae00 0x7fd9e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.334+0000 7fda01600700 1 -- 172.21.15.12:0/3608190558 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9f8053a00 con 0x7fda040b2070 2024-03-29T10:09:47.481 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:47.481+0000 7fedf0b8b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:09:47.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.525+0000 7fda08e00700 1 -- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd9e406ae00 msgr2=0x7fd9e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:47.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.525+0000 7fda08e00700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd9e406ae00 0x7fd9e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:47.533 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:47.534+0000 7ff901837240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:09:47.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.603+0000 7fda0c200700 1 -- 172.21.15.12:0/3608190558 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fda04067720 con 0x7fda040b2070 2024-03-29T10:09:47.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.609+0000 7fda01600700 1 -- 172.21.15.12:0/3608190558 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd9f8017020 con 0x7fda040b2070 2024-03-29T10:09:47.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 -- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd9e406ae00 msgr2=0x7fd9e406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:47.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd9e406ae00 0x7fd9e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 -- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 msgr2=0x7fda04150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 0x7fda04150fa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800c700 tx=0x7fd9f800ca10 comp rx=0 tx=0).stop 2024-03-29T10:09:47.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 -- 172.21.15.12:0/3608190558 shutdown_connections 2024-03-29T10:09:47.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd9e406ae00 0x7fd9e406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda040cd020 0x7fda04152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda040b2070 0x7fda04150fa0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 --2- 172.21.15.12:0/3608190558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda040cbb30 0x7fda04147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 -- 172.21.15.12:0/3608190558 >> 172.21.15.12:0/3608190558 conn(0x7fda040b0d70 msgr2=0x7fda0415cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.611+0000 7fda0c200700 1 -- 172.21.15.12:0/3608190558 shutdown_connections 2024-03-29T10:09:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.612+0000 7fda0c200700 1 -- 172.21.15.12:0/3608190558 wait complete. 2024-03-29T10:09:47.629 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T10:09:47.798 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:47.798+0000 7fedf0b8b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:09:47.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.834+0000 7fd77ce00700 1 Processor -- start 2024-03-29T10:09:47.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.834+0000 7fd77ce00700 1 -- start start 2024-03-29T10:09:47.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.835+0000 7fd77ce00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd7780cbb30 0x7fd7780cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.835+0000 7fd77ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd480 0x7fd77806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.835+0000 7fd77ce00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7780cc450 con 0x7fd7780cd020 2024-03-29T10:09:47.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.835+0000 7fd77ce00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7780cdae0 con 0x7fd7780cbb30 2024-03-29T10:09:47.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.835+0000 7fd77ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780cddc0 con 0x7fd7780cd480 2024-03-29T10:09:47.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.835+0000 7fd775400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd7780cd020 0x7fd7780b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60836/0 (socket says 172.21.15.12:60836) 2024-03-29T10:09:47.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.835+0000 7fd775400700 1 -- 172.21.15.12:0/2904232073 learned_addr learned my addr 172.21.15.12:0/2904232073 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:47.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.836+0000 7fd775e00700 1 --2- 172.21.15.12:0/2904232073 >> v2:172.21.15.159:3300/0 conn(0x7fd7780cbb30 0x7fd7780cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:47.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.836+0000 7fd776800700 1 --2- 172.21.15.12:0/2904232073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd480 0x7fd77806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:47.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.836+0000 7fd774a00700 1 -- 172.21.15.12:0/2904232073 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195441039 0 0) 0x7fd7780cc450 con 0x7fd7780cd020 2024-03-29T10:09:47.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.836+0000 7fd774a00700 1 -- 172.21.15.12:0/2904232073 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd75c0035b0 con 0x7fd7780cd020 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.836+0000 7fd776800700 1 -- 172.21.15.12:0/2904232073 >> v2:172.21.15.159:3300/0 conn(0x7fd7780cbb30 msgr2=0x7fd7780cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.836+0000 7fd776800700 1 --2- 172.21.15.12:0/2904232073 >> v2:172.21.15.159:3300/0 conn(0x7fd7780cbb30 0x7fd7780cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.836+0000 7fd776800700 1 -- 172.21.15.12:0/2904232073 >> v1:172.21.15.12:6789/0 conn(0x7fd7780cd020 legacy=0x7fd7780b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.836+0000 7fd776800700 1 -- 172.21.15.12:0/2904232073 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7780ce0a0 con 0x7fd7780cd480 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.836+0000 7fd775e00700 1 --2- 172.21.15.12:0/2904232073 >> v2:172.21.15.159:3300/0 conn(0x7fd7780cbb30 0x7fd7780cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.836+0000 7fd776800700 1 --2- 172.21.15.12:0/2904232073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd480 0x7fd77806e420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd770009d90 tx=0x7fd77000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b070ef7f4b363b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.837+0000 7fd774a00700 1 -- 172.21.15.12:0/2904232073 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77000e070 con 0x7fd7780cd480 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.837+0000 7fd774a00700 1 -- 172.21.15.12:0/2904232073 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77000bb20 con 0x7fd7780cd480 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.837+0000 7fd774a00700 1 -- 172.21.15.12:0/2904232073 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770012430 con 0x7fd7780cd480 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.837+0000 7fd77ce00700 1 -- 172.21.15.12:0/2904232073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd480 msgr2=0x7fd77806e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.837+0000 7fd77ce00700 1 --2- 172.21.15.12:0/2904232073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd480 0x7fd77806e420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd770009d90 tx=0x7fd77000b330 comp rx=0 tx=0).stop 2024-03-29T10:09:47.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.837+0000 7fd77ce00700 1 -- 172.21.15.12:0/2904232073 shutdown_connections 2024-03-29T10:09:47.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.837+0000 7fd77ce00700 1 --2- 172.21.15.12:0/2904232073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd480 0x7fd77806e420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.837+0000 7fd77ce00700 1 --2- 172.21.15.12:0/2904232073 >> v2:172.21.15.159:3300/0 conn(0x7fd7780cbb30 0x7fd7780cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.837+0000 7fd77ce00700 1 -- 172.21.15.12:0/2904232073 >> 172.21.15.12:0/2904232073 conn(0x7fd7780b0d70 msgr2=0x7fd7780b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:47.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.838+0000 7fd77ce00700 1 -- 172.21.15.12:0/2904232073 shutdown_connections 2024-03-29T10:09:47.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.838+0000 7fd77ce00700 1 -- 172.21.15.12:0/2904232073 wait complete. 2024-03-29T10:09:47.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.839+0000 7fd77ce00700 1 Processor -- start 2024-03-29T10:09:47.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.839+0000 7fd77ce00700 1 -- start start 2024-03-29T10:09:47.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.840+0000 7fd77ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cbb30 0x7fd77814eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.840+0000 7fd77ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd020 0x7fd7780b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.840+0000 7fd77ce00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7780cd480 0x7fd778071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.840+0000 7fd77ce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77814f040 con 0x7fd7780cbb30 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.840+0000 7fd77ce00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780ac4e0 con 0x7fd7780cd480 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.840+0000 7fd77ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780ac7c0 con 0x7fd7780cd020 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd775e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cbb30 0x7fd77814eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd775e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cbb30 0x7fd77814eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48248/0 (socket says 172.21.15.12:48248) 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd775e00700 1 -- 172.21.15.12:0/4255235184 learned_addr learned my addr 172.21.15.12:0/4255235184 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd775400700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd020 0x7fd7780b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd776800700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7780cd480 0x7fd778071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd775e00700 1 -- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd020 msgr2=0x7fd7780b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd775e00700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd020 0x7fd7780b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd775e00700 1 -- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7780cd480 msgr2=0x7fd778071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd775e00700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7780cd480 0x7fd778071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd775e00700 1 -- 172.21.15.12:0/4255235184 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7680094a0 con 0x7fd7780cbb30 2024-03-29T10:09:47.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.841+0000 7fd775e00700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cbb30 0x7fd77814eb00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd76800ebe0 tx=0x7fd76800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:47.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.842+0000 7fd76e000700 1 -- 172.21.15.12:0/4255235184 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768009760 con 0x7fd7780cbb30 2024-03-29T10:09:47.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.842+0000 7fd76e000700 1 -- 172.21.15.12:0/4255235184 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76800c850 con 0x7fd7780cbb30 2024-03-29T10:09:47.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.842+0000 7fd77ce00700 1 -- 172.21.15.12:0/4255235184 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd770009a70 con 0x7fd7780cbb30 2024-03-29T10:09:47.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.842+0000 7fd76e000700 1 -- 172.21.15.12:0/4255235184 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76800cbe0 con 0x7fd7780cbb30 2024-03-29T10:09:47.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.842+0000 7fd77ce00700 1 -- 172.21.15.12:0/4255235184 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7780acd80 con 0x7fd7780cbb30 2024-03-29T10:09:47.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.843+0000 7fd77ce00700 1 -- 172.21.15.12:0/4255235184 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd778067720 con 0x7fd7780cbb30 2024-03-29T10:09:47.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.844+0000 7fd76e000700 1 -- 172.21.15.12:0/4255235184 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 31) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fd76800cd40 con 0x7fd7780cbb30 2024-03-29T10:09:47.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.845+0000 7fd76e000700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806ae00 0x7fd75806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:47.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.845+0000 7fd76e000700 1 -- 172.21.15.12:0/4255235184 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd768089580 con 0x7fd7780cbb30 2024-03-29T10:09:47.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.845+0000 7fd775400700 1 -- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806ae00 msgr2=0x7fd75806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:47.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.846+0000 7fd775400700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806ae00 0x7fd75806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:47.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:47.849+0000 7fd76e000700 1 -- 172.21.15.12:0/4255235184 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd7680545c0 con 0x7fd7780cbb30 2024-03-29T10:09:47.877 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:47.878+0000 7ff901837240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:09:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.047+0000 7fd775400700 1 -- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806ae00 msgr2=0x7fd75806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.047+0000 7fd775400700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806ae00 0x7fd75806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:48.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.070+0000 7fd77ce00700 1 -- 172.21.15.12:0/4255235184 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fd77815bda0 con 0x7fd7780cbb30 2024-03-29T10:09:48.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.074+0000 7fd76e000700 1 -- 172.21.15.12:0/4255235184 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd768057be0 con 0x7fd7780cbb30 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 -- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806ae00 msgr2=0x7fd75806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806ae00 0x7fd75806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 -- 172.21.15.12:0/4255235184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cbb30 msgr2=0x7fd77814eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cbb30 0x7fd77814eb00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd76800ebe0 tx=0x7fd76800eef0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 -- 172.21.15.12:0/4255235184 shutdown_connections 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806ae00 0x7fd75806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7780cd480 0x7fd778071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cbb30 0x7fd77814eb00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 --2- 172.21.15.12:0/4255235184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780cd020 0x7fd7780b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 -- 172.21.15.12:0/4255235184 >> 172.21.15.12:0/4255235184 conn(0x7fd7780b0d70 msgr2=0x7fd77806e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:48.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 -- 172.21.15.12:0/4255235184 shutdown_connections 2024-03-29T10:09:48.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.077+0000 7fd77ce00700 1 -- 172.21.15.12:0/4255235184 wait complete. 2024-03-29T10:09:48.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:48.096 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-29T10:09:48.146 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-03-29T10:09:48.331 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:48.331+0000 7fedf0b8b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:09:48.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.349+0000 7fcf15c00700 1 Processor -- start 2024-03-29T10:09:48.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.350+0000 7fcf15c00700 1 -- start start 2024-03-29T10:09:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf15c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcf100d21f0 0x7fcf100ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf15c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf100cec40 0x7fcf10161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf15c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf100ce270 con 0x7fcf101611b0 2024-03-29T10:09:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf15c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1015b600 con 0x7fcf100d21f0 2024-03-29T10:09:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf15c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf100d2760 con 0x7fcf100cec40 2024-03-29T10:09:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf0e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcf100d21f0 0x7fcf100ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf0ea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcf101611b0 0x7fcf1015ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60852/0 (socket says 172.21.15.12:60852) 2024-03-29T10:09:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf0ea00700 1 -- 172.21.15.12:0/1023294392 learned_addr learned my addr 172.21.15.12:0/1023294392 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf0f400700 1 --2- 172.21.15.12:0/1023294392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf100cec40 0x7fcf10161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf0e000700 1 -- 172.21.15.12:0/1023294392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf100cec40 msgr2=0x7fcf10161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.351+0000 7fcf0e000700 1 --2- 172.21.15.12:0/1023294392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf100cec40 0x7fcf10161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.352+0000 7fcf0e000700 1 -- 172.21.15.12:0/1023294392 >> v1:172.21.15.12:6789/0 conn(0x7fcf101611b0 legacy=0x7fcf1015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.352+0000 7fcf0e000700 1 -- 172.21.15.12:0/1023294392 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf100d2a40 con 0x7fcf100d21f0 2024-03-29T10:09:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.352+0000 7fcf0f400700 1 --2- 172.21.15.12:0/1023294392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf100cec40 0x7fcf10161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.352+0000 7fcf0e000700 1 --2- 172.21.15.12:0/1023294392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf100d21f0 0x7fcf100ce700 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009840 tx=0x7fcefc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf052622fa0453f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.352+0000 7fcf0d600700 1 -- 172.21.15.12:0/1023294392 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc011830 con 0x7fcf100d21f0 2024-03-29T10:09:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.352+0000 7fcf0d600700 1 -- 172.21.15.12:0/1023294392 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc011990 con 0x7fcf100d21f0 2024-03-29T10:09:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.352+0000 7fcf0d600700 1 -- 172.21.15.12:0/1023294392 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc011d20 con 0x7fcf100d21f0 2024-03-29T10:09:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.353+0000 7fcf15c00700 1 -- 172.21.15.12:0/1023294392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf100d21f0 msgr2=0x7fcf100ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.353+0000 7fcf15c00700 1 --2- 172.21.15.12:0/1023294392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf100d21f0 0x7fcf100ce700 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009840 tx=0x7fcefc009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:48.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.353+0000 7fcf15c00700 1 -- 172.21.15.12:0/1023294392 shutdown_connections 2024-03-29T10:09:48.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.353+0000 7fcf15c00700 1 --2- 172.21.15.12:0/1023294392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf100cec40 0x7fcf10161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.353+0000 7fcf15c00700 1 --2- 172.21.15.12:0/1023294392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf100d21f0 0x7fcf100ce700 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.353+0000 7fcf15c00700 1 -- 172.21.15.12:0/1023294392 >> 172.21.15.12:0/1023294392 conn(0x7fcf1008aff0 msgr2=0x7fcf100d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:48.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.353+0000 7fcf15c00700 1 -- 172.21.15.12:0/1023294392 shutdown_connections 2024-03-29T10:09:48.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.353+0000 7fcf15c00700 1 -- 172.21.15.12:0/1023294392 wait complete. 2024-03-29T10:09:48.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.355+0000 7fcf15c00700 1 Processor -- start 2024-03-29T10:09:48.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.355+0000 7fcf15c00700 1 -- start start 2024-03-29T10:09:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf15c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf100cec40 0x7fcf1015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf15c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf100d21f0 0x7fcf1009f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf15c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf101611b0 0x7fcf100b8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf15c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf101f66e0 con 0x7fcf101611b0 2024-03-29T10:09:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf15c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1015aef0 con 0x7fcf100cec40 2024-03-29T10:09:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf15c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf101f3a10 con 0x7fcf100d21f0 2024-03-29T10:09:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf0f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf101611b0 0x7fcf100b8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf0f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf101611b0 0x7fcf100b8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48260/0 (socket says 172.21.15.12:48260) 2024-03-29T10:09:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf0f400700 1 -- 172.21.15.12:0/978977858 learned_addr learned my addr 172.21.15.12:0/978977858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf0e000700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf100d21f0 0x7fcf1009f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:48.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.356+0000 7fcf0ea00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf100cec40 0x7fcf1015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:48.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf0f400700 1 -- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf100d21f0 msgr2=0x7fcf1009f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf0f400700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf100d21f0 0x7fcf1009f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf0f400700 1 -- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf100cec40 msgr2=0x7fcf1015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf0f400700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf100cec40 0x7fcf1015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf0f400700 1 -- 172.21.15.12:0/978977858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0094a0 con 0x7fcf101611b0 2024-03-29T10:09:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf0f400700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf101611b0 0x7fcf100b8900 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fcf08007ef0 tx=0x7fcf0800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0800dd80 con 0x7fcf101611b0 2024-03-29T10:09:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0800f040 con 0x7fcf101611b0 2024-03-29T10:09:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf08003c60 con 0x7fcf101611b0 2024-03-29T10:09:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf15c00700 1 -- 172.21.15.12:0/978977858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf101f3cf0 con 0x7fcf101611b0 2024-03-29T10:09:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.357+0000 7fcf15c00700 1 -- 172.21.15.12:0/978977858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1009eb70 con 0x7fcf101611b0 2024-03-29T10:09:48.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.359+0000 7fcf15c00700 1 -- 172.21.15.12:0/978977858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf10067720 con 0x7fcf101611b0 2024-03-29T10:09:48.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.359+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 31) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fcf08004030 con 0x7fcf101611b0 2024-03-29T10:09:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.360+0000 7fcf06a00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 0x7fcef006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.360+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf080895d0 con 0x7fcf101611b0 2024-03-29T10:09:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.361+0000 7fcf0ea00700 1 -- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 msgr2=0x7fcef006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.361+0000 7fcf0ea00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 0x7fcef006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:48.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.365+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf08054620 con 0x7fcf101611b0 2024-03-29T10:09:48.457 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:48.458+0000 7ff901837240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:09:48.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.562+0000 7fcf0ea00700 1 -- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 msgr2=0x7fcef006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:48.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.562+0000 7fcf0ea00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 0x7fcef006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:48.586 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:48.586+0000 7fedf0b8b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:09:48.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.606+0000 7fcf15c00700 1 -- 172.21.15.12:0/978977858 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf1015f6a0 con 0x7fcef006ae30 2024-03-29T10:09:48.727 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:48.728+0000 7ff901837240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:09:48.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.963+0000 7fcf0ea00700 1 -- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 msgr2=0x7fcef006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:48.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:48.963+0000 7fcf0ea00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 0x7fcef006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T10:09:49.048 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:49.048+0000 7fedf0b8b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:09:49.207 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:49.207+0000 7ff901837240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:09:49.755 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:49.755+0000 7fedf0b8b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:09:49.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:49.764+0000 7fcf0ea00700 1 -- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 msgr2=0x7fcef006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:49.764+0000 7fcf0ea00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 0x7fcef006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T10:09:49.847 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:49.847+0000 7fedf0b8b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:09:49.953 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:49.952+0000 7ff901837240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:09:50.005 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:50.005+0000 7fedf0b8b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:09:50.049 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:50.049+0000 7ff901837240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:09:50.219 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:50.219+0000 7fedf0b8b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:09:50.219 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:50.218+0000 7ff901837240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:09:50.311 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:50.311+0000 7fedf0b8b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:09:50.409 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:50.409+0000 7fedf0b8b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:09:50.437 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:50.437+0000 7ff901837240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:09:50.506 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:50.506+0000 7fedf0b8b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:09:50.533 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:50.533+0000 7ff901837240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:09:50.604 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:50.604+0000 7fedf0b8b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:09:50.636 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:50.636+0000 7ff901837240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:09:50.740 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:50.739+0000 7ff901837240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:09:50.815 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:50.815+0000 7fedf0b8b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:09:50.843 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:50.842+0000 7ff901837240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:09:51.066 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:51.065+0000 7ff901837240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:09:51.072 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:51.073+0000 7fedf0b8b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:09:51.263 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:51.263+0000 7fedf0b8b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:09:51.329 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:51.329+0000 7ff901837240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:09:51.356 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:51.356+0000 7fedf0b8b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:09:51.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:51.365+0000 7fcf0ea00700 1 -- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 msgr2=0x7fcef006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:51.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:51.365+0000 7fcf0ea00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 0x7fcef006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T10:09:51.529 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:51.529+0000 7ff901837240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:09:51.626 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:51.626+0000 7ff901837240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:09:52.089 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:52.089+0000 7fedf0b8b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:09:52.301 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:52.301+0000 7fedf0b8b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:09:52.394 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:52.394+0000 7ff901837240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:09:52.507 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:52.507+0000 7fedf0b8b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:09:52.615 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:52.615+0000 7ff901837240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:09:52.832 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:52.831+0000 7ff901837240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:09:52.985 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:52.985+0000 7fedf0b8b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:09:53.191 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.191+0000 7fedf0b8b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:09:53.334 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:53.334+0000 7ff901837240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:09:53.553 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:53.552+0000 7ff901837240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:09:53.631 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.631+0000 7fedf0b8b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:09:53.742 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.742+0000 7fedf0b8b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:09:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:53.752+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 32) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fcf08087af0 con 0x7fcf101611b0 2024-03-29T10:09:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:53.753+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 msgr2=0x7fcef006d2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:53.753+0000 7fcf06a00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 0x7fcef006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:53.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.939+0000 7fed5ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:53.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.939+0000 7fed5ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:53.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.939+0000 7fed5ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:53.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.939+0000 7fed5ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:53.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.939+0000 7fed5ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:53.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.939+0000 7fed59200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:53.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.939+0000 7fed59200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:53.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.939+0000 7fed59200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:53.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.939+0000 7fed59200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:53.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:53.939+0000 7fed59200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:09:54.020 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:54.019+0000 7ff901837240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:09:54.138 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:54.137+0000 7ff901837240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:09:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:54.759+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 33) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fcf08014dc0 con 0x7fcf101611b0 2024-03-29T10:09:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:54.759+0000 7fcf06a00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 0x7fcef006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:54.759+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcef0036900 con 0x7fcef006ae30 2024-03-29T10:09:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:54.761+0000 7fcf0ea00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 0x7fcef006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:54.762+0000 7fcf0ea00700 1 --2- 172.21.15.12:0/978977858 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcef006ae30 0x7fcef006d2f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcf00005910 tx=0x7fcf00007f50 comp rx=0 tx=0).ready entity=mgr.5879 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:54.776 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:54.776+0000 7fed65a00700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-03-29T10:09:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:54.775+0000 7fcf06a00700 1 -- 172.21.15.12:0/978977858 <== mgr.5879 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fcef0036900 con 0x7fcef006ae30 2024-03-29T10:09:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-03-29T10:09:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-29T10:09:54.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:54.789 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-29T10:09:54.838 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T10:09:55.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.040+0000 7fefde800700 1 Processor -- start 2024-03-29T10:09:55.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.041+0000 7fefde800700 1 -- start start 2024-03-29T10:09:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.042+0000 7fefde800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fefd8099a60 0x7fefd80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.042+0000 7fefde800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd80d0970 con 0x7fefd815d990 2024-03-29T10:09:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.042+0000 7fefde800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd806c020 con 0x7fefd809a750 2024-03-29T10:09:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.042+0000 7fefde800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd80c81a0 con 0x7fefd8099a60 2024-03-29T10:09:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.042+0000 7fefcfe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fefd815d990 0x7fefd8099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35922/0 (socket says 172.21.15.12:35922) 2024-03-29T10:09:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.042+0000 7fefcfe00700 1 -- 172.21.15.12:0/2584849649 learned_addr learned my addr 172.21.15.12:0/2584849649 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.042+0000 7fefd7400700 1 --2- 172.21.15.12:0/2584849649 >> v2:172.21.15.159:3300/0 conn(0x7fefd8099a60 0x7fefd80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.042+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100244344 0 0) 0x7fefd80d0970 con 0x7fefd815d990 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.042+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefbc0035b0 con 0x7fefd815d990 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903479899 0 0) 0x7fefd806c020 con 0x7fefd809a750 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd80d0970 con 0x7fefd809a750 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1658844581 0 0) 0x7fefbc0035b0 con 0x7fefd815d990 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 >> v2:172.21.15.159:3300/0 conn(0x7fefd8099a60 msgr2=0x7fefd80d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 --2- 172.21.15.12:0/2584849649 >> v2:172.21.15.159:3300/0 conn(0x7fefd8099a60 0x7fefd80d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 >> v1:172.21.15.159:6789/0 conn(0x7fefd809a750 legacy=0x7fefd806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd80d0690 con 0x7fefd815d990 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fefc0001670 con 0x7fefd815d990 2024-03-29T10:09:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 >> v1:172.21.15.12:6789/0 conn(0x7fefd815d990 legacy=0x7fefd8099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefbc003be0 0x7fefbc003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefbc004850 0x7fefbc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefbc0052d0 0x7fefbc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd80c81a0 con 0x7fefbc004850 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc00c210 con 0x7fefbc003be0 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc00c3d0 con 0x7fefbc0052d0 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd7400700 1 --2- 172.21.15.12:0/2584849649 >> v2:172.21.15.159:3300/0 conn(0x7fefd8099a60 0x7fefd80d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefcfe00700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefbc004850 0x7fefbc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.043+0000 7fefd7e00700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefbc0052d0 0x7fefbc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefd7400700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefbc003be0 0x7fefbc003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefcfe00700 1 -- 172.21.15.12:0/2584849649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefbc0052d0 msgr2=0x7fefbc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefcfe00700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefbc0052d0 0x7fefbc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefcfe00700 1 -- 172.21.15.12:0/2584849649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefbc003be0 msgr2=0x7fefbc003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefcfe00700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefbc003be0 0x7fefbc003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefcfe00700 1 -- 172.21.15.12:0/2584849649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc0035b0 con 0x7fefbc004850 2024-03-29T10:09:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefd7400700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefbc003be0 0x7fefbc003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefd7e00700 1 -- 172.21.15.12:0/2584849649 reap_dead start 2024-03-29T10:09:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefcfe00700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefbc004850 0x7fefbc004cd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fefc000c160 tx=0x7fefbc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200ce3e6dc59784d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc000cca0 con 0x7fefbc004850 2024-03-29T10:09:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc000ce20 con 0x7fefbc004850 2024-03-29T10:09:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.044+0000 7fefd6a00700 1 -- 172.21.15.12:0/2584849649 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc0005ba0 con 0x7fefbc004850 2024-03-29T10:09:55.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.045+0000 7fefde800700 1 -- 172.21.15.12:0/2584849649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefbc004850 msgr2=0x7fefbc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:55.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.045+0000 7fefde800700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefbc004850 0x7fefbc004cd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fefc000c160 tx=0x7fefbc005880 comp rx=0 tx=0).stop 2024-03-29T10:09:55.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.045+0000 7fefde800700 1 -- 172.21.15.12:0/2584849649 shutdown_connections 2024-03-29T10:09:55.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.045+0000 7fefde800700 1 --2- 172.21.15.12:0/2584849649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefbc004850 0x7fefbc004cd0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:55.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.045+0000 7fefde800700 1 -- 172.21.15.12:0/2584849649 >> 172.21.15.12:0/2584849649 conn(0x7fefd80b2e20 msgr2=0x7fefd80b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:55.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.045+0000 7fefde800700 1 -- 172.21.15.12:0/2584849649 shutdown_connections 2024-03-29T10:09:55.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.045+0000 7fefde800700 1 -- 172.21.15.12:0/2584849649 wait complete. 2024-03-29T10:09:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.047+0000 7fefde800700 1 Processor -- start 2024-03-29T10:09:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.047+0000 7fefde800700 1 -- start start 2024-03-29T10:09:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefde800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefbc004850 0x7fefd8159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefde800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefd815b2f0 0x7fefd815b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefde800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefd8160040 0x7fefd81604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefde800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd8160a30 con 0x7fefd815b2f0 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefde800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd815bcb0 con 0x7fefd8160040 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefde800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd80c92c0 con 0x7fefbc004850 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefcfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefd815b2f0 0x7fefd815b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefcfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefd815b2f0 0x7fefd815b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36882/0 (socket says 172.21.15.12:36882) 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefcfe00700 1 -- 172.21.15.12:0/2892988843 learned_addr learned my addr 172.21.15.12:0/2892988843 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefd7400700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefbc004850 0x7fefd8159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.048+0000 7fefd7e00700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefd8160040 0x7fefd81604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefcfe00700 1 -- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefbc004850 msgr2=0x7fefd8159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefcfe00700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefbc004850 0x7fefd8159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefcfe00700 1 -- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefd8160040 msgr2=0x7fefd81604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefcfe00700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefd8160040 0x7fefd81604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefcfe00700 1 -- 172.21.15.12:0/2892988843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd8160c20 con 0x7fefd815b2f0 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefcfe00700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefd815b2f0 0x7fefd815b770 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fefc000c750 tx=0x7fefc0001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefcf400700 1 -- 172.21.15.12:0/2892988843 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc0004c80 con 0x7fefd815b2f0 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefcf400700 1 -- 172.21.15.12:0/2892988843 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc0004e00 con 0x7fefd815b2f0 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefde800700 1 -- 172.21.15.12:0/2892988843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd80c9d30 con 0x7fefd815b2f0 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefcf400700 1 -- 172.21.15.12:0/2892988843 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc0007310 con 0x7fefd815b2f0 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.049+0000 7fefde800700 1 -- 172.21.15.12:0/2892988843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd80ca2f0 con 0x7fefd815b2f0 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.050+0000 7fefde800700 1 -- 172.21.15.12:0/2892988843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd8067720 con 0x7fefd815b2f0 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.051+0000 7fefcf400700 1 -- 172.21.15.12:0/2892988843 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 33) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fefc0007470 con 0x7fefd815b2f0 2024-03-29T10:09:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.052+0000 7fefcf400700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefb406ab30 0x7fefb406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.052+0000 7fefcf400700 1 -- 172.21.15.12:0/2892988843 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fefc0089a30 con 0x7fefd815b2f0 2024-03-29T10:09:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.053+0000 7fefd7400700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefb406ab30 0x7fefb406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.053+0000 7fefd7400700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefb406ab30 0x7fefb406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fefd80684a0 tx=0x7fefc8008040 comp rx=0 tx=0).ready entity=mgr.5879 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.057+0000 7fefcf400700 1 -- 172.21.15.12:0/2892988843 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fefc0054de0 con 0x7fefd815b2f0 2024-03-29T10:09:55.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:55.333+0000 7fefde800700 1 -- 172.21.15.12:0/2892988843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fefd806fc30 con 0x7fefd815b2f0 2024-03-29T10:09:56.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.047+0000 7fefcf400700 1 -- 172.21.15.12:0/2892988843 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fefc00551b0 con 0x7fefd815b2f0 2024-03-29T10:09:56.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.048+0000 7fefcf400700 1 -- 172.21.15.12:0/2892988843 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v34) v1 ==== 93+0+0 (secure 0 0 0) 0x7fefc0084bb0 con 0x7fefd815b2f0 2024-03-29T10:09:56.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.051+0000 7fefde800700 1 -- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefb406ab30 msgr2=0x7fefb406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:56.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.051+0000 7fefde800700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefb406ab30 0x7fefb406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fefd80684a0 tx=0x7fefc8008040 comp rx=0 tx=0).stop 2024-03-29T10:09:56.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.051+0000 7fefde800700 1 -- 172.21.15.12:0/2892988843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefd815b2f0 msgr2=0x7fefd815b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:56.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.051+0000 7fefde800700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefd815b2f0 0x7fefd815b770 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fefc000c750 tx=0x7fefc0001930 comp rx=0 tx=0).stop 2024-03-29T10:09:56.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.051+0000 7fefde800700 1 -- 172.21.15.12:0/2892988843 shutdown_connections 2024-03-29T10:09:56.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.052+0000 7fefde800700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefb406ab30 0x7fefb406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.052+0000 7fefde800700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefd8160040 0x7fefd81604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.052+0000 7fefde800700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefbc004850 0x7fefd8159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.052+0000 7fefde800700 1 --2- 172.21.15.12:0/2892988843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefd815b2f0 0x7fefd815b770 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.052+0000 7fefde800700 1 -- 172.21.15.12:0/2892988843 >> 172.21.15.12:0/2892988843 conn(0x7fefd80b2e20 msgr2=0x7fefd80b6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:56.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.052+0000 7fefde800700 1 -- 172.21.15.12:0/2892988843 shutdown_connections 2024-03-29T10:09:56.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.052+0000 7fefde800700 1 -- 172.21.15.12:0/2892988843 wait complete. 2024-03-29T10:09:56.073 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:09:56.073 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:09:56.176 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:09:56.263 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:09:56.264 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:09:56.264 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:56.283 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:56.283+0000 7f1caccb1240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:09:56.285 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:56.284+0000 7fd1471b9240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:09:56.315 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:09:56.318 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:56.402 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:09:56.469 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:56.469+0000 7f1caccb1240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:09:56.496 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:56.496+0000 7fd1471b9240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:09:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.671+0000 7fa210400700 1 Processor -- start 2024-03-29T10:09:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.671+0000 7fa210400700 1 -- start start 2024-03-29T10:09:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.673+0000 7fa210400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa20815d670 0x7fa2080ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.673+0000 7fa210400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa20806dee0 0x7fa20806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.673+0000 7fa210400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2080b3890 con 0x7fa20815da50 2024-03-29T10:09:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.673+0000 7fa210400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2080acac0 con 0x7fa20815d670 2024-03-29T10:09:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.673+0000 7fa210400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa208159f50 con 0x7fa20806dee0 2024-03-29T10:09:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.673+0000 7fa20da00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa20815d670 0x7fa2080ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.673+0000 7fa20da00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa20815d670 0x7fa2080ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36908/0 (socket says 172.21.15.12:36908) 2024-03-29T10:09:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.673+0000 7fa20da00700 1 -- 172.21.15.12:0/2324334293 learned_addr learned my addr 172.21.15.12:0/2324334293 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.673+0000 7fa20e400700 1 --2- 172.21.15.12:0/2324334293 >> v2:172.21.15.159:3300/0 conn(0x7fa20806dee0 0x7fa20806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.674+0000 7fa20da00700 1 -- 172.21.15.12:0/2324334293 >> v2:172.21.15.159:3300/0 conn(0x7fa20806dee0 msgr2=0x7fa20806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.674+0000 7fa20da00700 1 --2- 172.21.15.12:0/2324334293 >> v2:172.21.15.159:3300/0 conn(0x7fa20806dee0 0x7fa20806e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.674+0000 7fa20da00700 1 -- 172.21.15.12:0/2324334293 >> v1:172.21.15.159:6789/0 conn(0x7fa20815da50 legacy=0x7fa2080b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.674+0000 7fa20da00700 1 -- 172.21.15.12:0/2324334293 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2080b35b0 con 0x7fa20815d670 2024-03-29T10:09:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.674+0000 7fa20da00700 1 --2- 172.21.15.12:0/2324334293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa20815d670 0x7fa2080ac580 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa2080cfe30 tx=0x7fa1fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd672d430c3c77d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.674+0000 7fa207e00700 1 -- 172.21.15.12:0/2324334293 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc00f8a0 con 0x7fa20815d670 2024-03-29T10:09:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.674+0000 7fa207e00700 1 -- 172.21.15.12:0/2324334293 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc00fa00 con 0x7fa20815d670 2024-03-29T10:09:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.674+0000 7fa207e00700 1 -- 172.21.15.12:0/2324334293 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc00fd90 con 0x7fa20815d670 2024-03-29T10:09:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.675+0000 7fa210400700 1 -- 172.21.15.12:0/2324334293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa20815d670 msgr2=0x7fa2080ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.675+0000 7fa210400700 1 --2- 172.21.15.12:0/2324334293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa20815d670 0x7fa2080ac580 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa2080cfe30 tx=0x7fa1fc009bd0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.675+0000 7fa210400700 1 -- 172.21.15.12:0/2324334293 shutdown_connections 2024-03-29T10:09:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.675+0000 7fa210400700 1 --2- 172.21.15.12:0/2324334293 >> v2:172.21.15.159:3300/0 conn(0x7fa20806dee0 0x7fa20806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.675+0000 7fa210400700 1 --2- 172.21.15.12:0/2324334293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa20815d670 0x7fa2080ac580 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.675+0000 7fa210400700 1 -- 172.21.15.12:0/2324334293 >> 172.21.15.12:0/2324334293 conn(0x7fa2080cf190 msgr2=0x7fa2080cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.675+0000 7fa210400700 1 -- 172.21.15.12:0/2324334293 shutdown_connections 2024-03-29T10:09:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.675+0000 7fa210400700 1 -- 172.21.15.12:0/2324334293 wait complete. 2024-03-29T10:09:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.677+0000 7fa210400700 1 Processor -- start 2024-03-29T10:09:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.677+0000 7fa210400700 1 -- start start 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa210400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20806dee0 0x7fa20809c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa210400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa20815d670 0x7fa2080cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa210400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa20815da50 0x7fa2080af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa210400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080c90e0 con 0x7fa20815d670 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa210400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080c8460 con 0x7fa20806dee0 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa210400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080c8740 con 0x7fa20815da50 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa20d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa20815d670 0x7fa2080cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa20da00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20806dee0 0x7fa20809c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa20da00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20806dee0 0x7fa20809c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:57166/0 (socket says 172.21.15.12:57166) 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa20da00700 1 -- 172.21.15.12:0/1085930475 learned_addr learned my addr 172.21.15.12:0/1085930475 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.678+0000 7fa20e400700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa20815da50 0x7fa2080af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa20da00700 1 -- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa20815da50 msgr2=0x7fa2080af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa20da00700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa20815da50 0x7fa2080af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa20da00700 1 -- 172.21.15.12:0/1085930475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa20815d670 msgr2=0x7fa2080cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa20da00700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa20815d670 0x7fa2080cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa20da00700 1 -- 172.21.15.12:0/1085930475 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc009550 con 0x7fa20806dee0 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa20d000700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa20815d670 0x7fa2080cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa20da00700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20806dee0 0x7fa20809c0a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc000c00 tx=0x7fa1fc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa205600700 1 -- 172.21.15.12:0/1085930475 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc0038a0 con 0x7fa20806dee0 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa205600700 1 -- 172.21.15.12:0/1085930475 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc003a20 con 0x7fa20806dee0 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa210400700 1 -- 172.21.15.12:0/1085930475 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2080c8a20 con 0x7fa20806dee0 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa205600700 1 -- 172.21.15.12:0/1085930475 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc003dd0 con 0x7fa20806dee0 2024-03-29T10:09:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.679+0000 7fa210400700 1 -- 172.21.15.12:0/1085930475 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20809cb30 con 0x7fa20806dee0 2024-03-29T10:09:56.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.680+0000 7fa210400700 1 -- 172.21.15.12:0/1085930475 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2080c9220 con 0x7fa20806dee0 2024-03-29T10:09:56.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.681+0000 7fa205600700 1 -- 172.21.15.12:0/1085930475 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fa1fc02a430 con 0x7fa20806dee0 2024-03-29T10:09:56.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.682+0000 7fa205600700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 0x7fa1e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:56.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.682+0000 7fa205600700 1 -- 172.21.15.12:0/1085930475 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fa1fc08a260 con 0x7fa20806dee0 2024-03-29T10:09:56.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.682+0000 7fa20d000700 1 -- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 msgr2=0x7fa1e806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:56.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.682+0000 7fa20d000700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 0x7fa1e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.687+0000 7fa205600700 1 -- 172.21.15.12:0/1085930475 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2080c9220 con 0x7fa20806dee0 2024-03-29T10:09:56.798 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:56.798+0000 7f1caccb1240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:09:56.835 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:09:56.839 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:09:56.841 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:56.841+0000 7fd1471b9240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:09:56.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.883+0000 7fa20d000700 1 -- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 msgr2=0x7fa1e806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:56.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.883+0000 7fa20d000700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 0x7fa1e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:56.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.904+0000 7fa210400700 1 -- 172.21.15.12:0/1085930475 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2080c9220 con 0x7fa20806dee0 2024-03-29T10:09:56.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.906+0000 7fa205600700 1 -- 172.21.15.12:0/1085930475 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16595 (secure 0 0 0) 0x7fa2080c9220 con 0x7fa20806dee0 2024-03-29T10:09:56.907 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:56.907 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:53.746428+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:09:30.441619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-29T10:09:31.581366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3367842275":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/3723093447":"2024-03-30T10:09:53.746392+0000","172.21.15.159:6833/43444":"2024-03-30T10:09:53.746392+0000","172.21.15.159:6832/43444":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/1578988741":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/1180347815":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/667128976":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/3208721490":"2024-03-30T10:09:53.746392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 -- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 msgr2=0x7fa1e806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 0x7fa1e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 -- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20806dee0 msgr2=0x7fa20809c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20806dee0 0x7fa20809c0a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc000c00 tx=0x7fa1fc004750 comp rx=0 tx=0).stop 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 -- 172.21.15.12:0/1085930475 shutdown_connections 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806adb0 0x7fa1e806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa20815da50 0x7fa2080af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa20806dee0 0x7fa20809c0a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 --2- 172.21.15.12:0/1085930475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa20815d670 0x7fa2080cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 -- 172.21.15.12:0/1085930475 >> 172.21.15.12:0/1085930475 conn(0x7fa2080cf190 msgr2=0x7fa20806cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.910+0000 7fa210400700 1 -- 172.21.15.12:0/1085930475 shutdown_connections 2024-03-29T10:09:56.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:56.911+0000 7fa210400700 1 -- 172.21.15.12:0/1085930475 wait complete. 2024-03-29T10:09:56.937 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:09:30.441619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:09:31.581366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:09:56.937 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:09:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.180+0000 7f8aee800700 1 Processor -- start 2024-03-29T10:09:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.180+0000 7f8aee800700 1 -- start start 2024-03-29T10:09:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.181+0000 7f8aee800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ae8157f90 0x7f8ae80b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.181+0000 7f8aee800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8ae81613e0 0x7f8ae8161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.181+0000 7f8aee800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae80b3070 con 0x7f8ae80b3420 2024-03-29T10:09:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.181+0000 7f8aee800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae8158370 con 0x7f8ae8157f90 2024-03-29T10:09:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.181+0000 7f8aee800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae80b2710 con 0x7f8ae81613e0 2024-03-29T10:09:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.181+0000 7f8ae7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ae8157f90 0x7f8ae80b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.182+0000 7f8ae7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ae8157f90 0x7f8ae80b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36920/0 (socket says 172.21.15.12:36920) 2024-03-29T10:09:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.182+0000 7f8ae7400700 1 -- 172.21.15.12:0/2972399856 learned_addr learned my addr 172.21.15.12:0/2972399856 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.182+0000 7f8ae6a00700 1 --2- 172.21.15.12:0/2972399856 >> v2:172.21.15.159:3300/0 conn(0x7f8ae81613e0 0x7f8ae8161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.182+0000 7f8ae7400700 1 -- 172.21.15.12:0/2972399856 >> v2:172.21.15.159:3300/0 conn(0x7f8ae81613e0 msgr2=0x7f8ae8161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.182+0000 7f8ae7400700 1 --2- 172.21.15.12:0/2972399856 >> v2:172.21.15.159:3300/0 conn(0x7f8ae81613e0 0x7f8ae8161890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.182+0000 7f8ae7400700 1 -- 172.21.15.12:0/2972399856 >> v1:172.21.15.12:6789/0 conn(0x7f8ae80b3420 legacy=0x7f8ae80b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.182+0000 7f8ae7400700 1 -- 172.21.15.12:0/2972399856 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae80b2070 con 0x7f8ae8157f90 2024-03-29T10:09:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.182+0000 7f8ae7400700 1 --2- 172.21.15.12:0/2972399856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae8157f90 0x7f8ae80b2b30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8ae80b4210 tx=0x7f8ad8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b61c0316f9c4724 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.182+0000 7f8ae6000700 1 -- 172.21.15.12:0/2972399856 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8011830 con 0x7f8ae8157f90 2024-03-29T10:09:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.183+0000 7f8ae6000700 1 -- 172.21.15.12:0/2972399856 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad8011990 con 0x7f8ae8157f90 2024-03-29T10:09:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.183+0000 7f8ae6000700 1 -- 172.21.15.12:0/2972399856 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8011d20 con 0x7f8ae8157f90 2024-03-29T10:09:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.183+0000 7f8aee800700 1 -- 172.21.15.12:0/2972399856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae8157f90 msgr2=0x7f8ae80b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.183+0000 7f8aee800700 1 --2- 172.21.15.12:0/2972399856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae8157f90 0x7f8ae80b2b30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8ae80b4210 tx=0x7f8ad8009bd0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.183+0000 7f8aee800700 1 -- 172.21.15.12:0/2972399856 shutdown_connections 2024-03-29T10:09:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.183+0000 7f8aee800700 1 --2- 172.21.15.12:0/2972399856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae8157f90 0x7f8ae80b2b30 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.183+0000 7f8aee800700 1 --2- 172.21.15.12:0/2972399856 >> v2:172.21.15.159:3300/0 conn(0x7f8ae81613e0 0x7f8ae8161890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.183+0000 7f8aee800700 1 -- 172.21.15.12:0/2972399856 >> 172.21.15.12:0/2972399856 conn(0x7f8ae80b0d70 msgr2=0x7f8ae80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.183+0000 7f8aee800700 1 -- 172.21.15.12:0/2972399856 shutdown_connections 2024-03-29T10:09:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.184+0000 7f8aee800700 1 -- 172.21.15.12:0/2972399856 wait complete. 2024-03-29T10:09:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.185+0000 7f8aee800700 1 Processor -- start 2024-03-29T10:09:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.185+0000 7f8aee800700 1 -- start start 2024-03-29T10:09:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.186+0000 7f8aee800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae80b3420 0x7f8ae814b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.186+0000 7f8aee800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae8157f90 0x7f8ae8158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.186+0000 7f8aee800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ae81613e0 0x7f8ae8159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.186+0000 7f8aee800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae81f37d0 con 0x7f8ae80b3420 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.186+0000 7f8aee800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae815bf40 con 0x7f8ae81613e0 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.186+0000 7f8aee800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae80ad7d0 con 0x7f8ae8157f90 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae80b3420 0x7f8ae814b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae8157f90 0x7f8ae8158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae80b3420 0x7f8ae814b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36924/0 (socket says 172.21.15.12:36924) 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae7400700 1 -- 172.21.15.12:0/3475776167 learned_addr learned my addr 172.21.15.12:0/3475776167 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae7e00700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ae81613e0 0x7f8ae8159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae7400700 1 -- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae8157f90 msgr2=0x7f8ae8158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae7400700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae8157f90 0x7f8ae8158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae7400700 1 -- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ae81613e0 msgr2=0x7f8ae8159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:57.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae7400700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ae81613e0 0x7f8ae8159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae7400700 1 -- 172.21.15.12:0/3475776167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad8009550 con 0x7f8ae80b3420 2024-03-29T10:09:57.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.187+0000 7f8ae6a00700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae8157f90 0x7f8ae8158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:57.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.188+0000 7f8ae7400700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae80b3420 0x7f8ae814b310 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009c10 tx=0x7f8ad8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:57.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.188+0000 7f8adf400700 1 -- 172.21.15.12:0/3475776167 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8004510 con 0x7f8ae80b3420 2024-03-29T10:09:57.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.188+0000 7f8adf400700 1 -- 172.21.15.12:0/3475776167 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad801e070 con 0x7f8ae80b3420 2024-03-29T10:09:57.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.188+0000 7f8adf400700 1 -- 172.21.15.12:0/3475776167 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80038e0 con 0x7f8ae80b3420 2024-03-29T10:09:57.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.188+0000 7f8aee800700 1 -- 172.21.15.12:0/3475776167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae815c190 con 0x7f8ae80b3420 2024-03-29T10:09:57.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.188+0000 7f8aee800700 1 -- 172.21.15.12:0/3475776167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae815c750 con 0x7f8ae80b3420 2024-03-29T10:09:57.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.189+0000 7f8aee800700 1 -- 172.21.15.12:0/3475776167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae8067720 con 0x7f8ae80b3420 2024-03-29T10:09:57.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.190+0000 7f8adf400700 1 -- 172.21.15.12:0/3475776167 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f8ad8003a40 con 0x7f8ae80b3420 2024-03-29T10:09:57.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.191+0000 7f8adf400700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac806adb0 0x7f8ac806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.191+0000 7f8adf400700 1 -- 172.21.15.12:0/3475776167 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f8ad808a400 con 0x7f8ae80b3420 2024-03-29T10:09:57.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.191+0000 7f8ae6a00700 1 -- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac806adb0 msgr2=0x7f8ac806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:57.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.191+0000 7f8ae6a00700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac806adb0 0x7f8ac806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:57.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.196+0000 7f8adf400700 1 -- 172.21.15.12:0/3475776167 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ad8055360 con 0x7f8ae80b3420 2024-03-29T10:09:57.341 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:57.341+0000 7f1caccb1240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:09:57.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.392+0000 7f8ae6a00700 1 -- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac806adb0 msgr2=0x7f8ac806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:57.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.392+0000 7f8ae6a00700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac806adb0 0x7f8ac806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:57.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.411+0000 7f8aee800700 1 -- 172.21.15.12:0/3475776167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ae80ce2e0 con 0x7f8ae80b3420 2024-03-29T10:09:57.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.411+0000 7f8adf400700 1 -- 172.21.15.12:0/3475776167 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ad8058980 con 0x7f8ae80b3420 2024-03-29T10:09:57.412 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:09:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 -- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac806adb0 msgr2=0x7f8ac806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac806adb0 0x7f8ac806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 -- 172.21.15.12:0/3475776167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae80b3420 msgr2=0x7f8ae814b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae80b3420 0x7f8ae814b310 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009c10 tx=0x7f8ad8004750 comp rx=0 tx=0).stop 2024-03-29T10:09:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 -- 172.21.15.12:0/3475776167 shutdown_connections 2024-03-29T10:09:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ac806adb0 0x7f8ac806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ae81613e0 0x7f8ae8159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ae80b3420 0x7f8ae814b310 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 --2- 172.21.15.12:0/3475776167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ae8157f90 0x7f8ae8158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 -- 172.21.15.12:0/3475776167 >> 172.21.15.12:0/3475776167 conn(0x7f8ae80b0d70 msgr2=0x7f8ae80b0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:57.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 -- 172.21.15.12:0/3475776167 shutdown_connections 2024-03-29T10:09:57.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.414+0000 7f8aee800700 1 -- 172.21.15.12:0/3475776167 wait complete. 2024-03-29T10:09:57.420 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:57.420+0000 7fd1471b9240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:09:57.433 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:09:57.596 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:57.596+0000 7f1caccb1240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:09:57.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.638+0000 7fbc43600700 1 Processor -- start 2024-03-29T10:09:57.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.638+0000 7fbc43600700 1 -- start start 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.639+0000 7fbc43600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c0ced80 0x7fbc3c0d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.639+0000 7fbc43600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc3c15fcb0 0x7fbc3c160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.639+0000 7fbc43600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3c0cf1e0 con 0x7fbc3c15dad0 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.639+0000 7fbc43600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc3c0cfcc0 con 0x7fbc3c15fcb0 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.639+0000 7fbc43600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c0cd9f0 con 0x7fbc3c0ced80 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.640+0000 7fbc41600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc3c15fcb0 0x7fbc3c160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.640+0000 7fbc40c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbc3c15dad0 0x7fbc3c0d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35946/0 (socket says 172.21.15.12:35946) 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.640+0000 7fbc40c00700 1 -- 172.21.15.12:0/178786657 learned_addr learned my addr 172.21.15.12:0/178786657 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.640+0000 7fbc3be00700 1 --2- 172.21.15.12:0/178786657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c0ced80 0x7fbc3c0d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.640+0000 7fbc41600700 1 -- 172.21.15.12:0/178786657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c0ced80 msgr2=0x7fbc3c0d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.640+0000 7fbc41600700 1 --2- 172.21.15.12:0/178786657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c0ced80 0x7fbc3c0d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.640+0000 7fbc41600700 1 -- 172.21.15.12:0/178786657 >> v1:172.21.15.12:6789/0 conn(0x7fbc3c15dad0 legacy=0x7fbc3c0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.640+0000 7fbc41600700 1 -- 172.21.15.12:0/178786657 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c15f820 con 0x7fbc3c15fcb0 2024-03-29T10:09:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.640+0000 7fbc3b400700 1 -- 172.21.15.12:0/178786657 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013652377 0 0) 0x7fbc30001110 con 0x7fbc3c15dad0 2024-03-29T10:09:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.640+0000 7fbc41600700 1 --2- 172.21.15.12:0/178786657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc3c15fcb0 0x7fbc3c160140 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbc34009d90 tx=0x7fbc3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5abe8182b758dc8c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.641+0000 7fbc3b400700 1 -- 172.21.15.12:0/178786657 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3400e070 con 0x7fbc3c15fcb0 2024-03-29T10:09:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.641+0000 7fbc3b400700 1 -- 172.21.15.12:0/178786657 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3400bb20 con 0x7fbc3c15fcb0 2024-03-29T10:09:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.641+0000 7fbc3b400700 1 -- 172.21.15.12:0/178786657 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34012430 con 0x7fbc3c15fcb0 2024-03-29T10:09:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.641+0000 7fbc43600700 1 -- 172.21.15.12:0/178786657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc3c15fcb0 msgr2=0x7fbc3c160140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.641+0000 7fbc43600700 1 --2- 172.21.15.12:0/178786657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc3c15fcb0 0x7fbc3c160140 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbc34009d90 tx=0x7fbc3400b330 comp rx=0 tx=0).stop 2024-03-29T10:09:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.642+0000 7fbc43600700 1 -- 172.21.15.12:0/178786657 shutdown_connections 2024-03-29T10:09:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.642+0000 7fbc43600700 1 --2- 172.21.15.12:0/178786657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc3c15fcb0 0x7fbc3c160140 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.642+0000 7fbc43600700 1 --2- 172.21.15.12:0/178786657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c0ced80 0x7fbc3c0d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.642+0000 7fbc43600700 1 -- 172.21.15.12:0/178786657 >> 172.21.15.12:0/178786657 conn(0x7fbc3c08aff0 msgr2=0x7fbc3c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.642+0000 7fbc43600700 1 -- 172.21.15.12:0/178786657 shutdown_connections 2024-03-29T10:09:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.642+0000 7fbc43600700 1 -- 172.21.15.12:0/178786657 wait complete. 2024-03-29T10:09:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.643+0000 7fbc43600700 1 Processor -- start 2024-03-29T10:09:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.644+0000 7fbc43600700 1 -- start start 2024-03-29T10:09:57.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.644+0000 7fbc43600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc3c0ced80 0x7fbc3c15cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.644+0000 7fbc43600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c15dad0 0x7fbc3c151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc43600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc3c15fcb0 0x7fbc3c0b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc43600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c0b8fd0 con 0x7fbc3c15fcb0 2024-03-29T10:09:57.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc43600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c0b87e0 con 0x7fbc3c0ced80 2024-03-29T10:09:57.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc43600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c0b8a10 con 0x7fbc3c15dad0 2024-03-29T10:09:57.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc41600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc3c15fcb0 0x7fbc3c0b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:57.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc3be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c15dad0 0x7fbc3c151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:57.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc3be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c15dad0 0x7fbc3c151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:34734/0 (socket says 172.21.15.12:34734) 2024-03-29T10:09:57.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc40c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc3c0ced80 0x7fbc3c15cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:57.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc3be00700 1 -- 172.21.15.12:0/2218406358 learned_addr learned my addr 172.21.15.12:0/2218406358 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:57.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc3be00700 1 -- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc3c0ced80 msgr2=0x7fbc3c15cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc3be00700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc3c0ced80 0x7fbc3c15cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc3be00700 1 -- 172.21.15.12:0/2218406358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc3c15fcb0 msgr2=0x7fbc3c0b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc3be00700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc3c15fcb0 0x7fbc3c0b82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.645+0000 7fbc3be00700 1 -- 172.21.15.12:0/2218406358 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc34009a70 con 0x7fbc3c15dad0 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.646+0000 7fbc3be00700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c15dad0 0x7fbc3c151190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800e9e0 tx=0x7fbc2800ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.646+0000 7fbc38c00700 1 -- 172.21.15.12:0/2218406358 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28009710 con 0x7fbc3c15dad0 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.646+0000 7fbc38c00700 1 -- 172.21.15.12:0/2218406358 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2800ee80 con 0x7fbc3c15dad0 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.646+0000 7fbc38c00700 1 -- 172.21.15.12:0/2218406358 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2800ca50 con 0x7fbc3c15dad0 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.646+0000 7fbc43600700 1 -- 172.21.15.12:0/2218406358 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc3c0b8cf0 con 0x7fbc3c15dad0 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.646+0000 7fbc43600700 1 -- 172.21.15.12:0/2218406358 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc3c0caa70 con 0x7fbc3c15dad0 2024-03-29T10:09:57.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.647+0000 7fbc43600700 1 -- 172.21.15.12:0/2218406358 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3c0ca5f0 con 0x7fbc3c15dad0 2024-03-29T10:09:57.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.648+0000 7fbc38c00700 1 -- 172.21.15.12:0/2218406358 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fbc2800cbb0 con 0x7fbc3c15dad0 2024-03-29T10:09:57.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.649+0000 7fbc38c00700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc1c06adb0 0x7fbc1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:57.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.649+0000 7fbc38c00700 1 -- 172.21.15.12:0/2218406358 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fbc28089650 con 0x7fbc3c15dad0 2024-03-29T10:09:57.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.649+0000 7fbc40c00700 1 -- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc1c06adb0 msgr2=0x7fbc1c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:57.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.649+0000 7fbc40c00700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc1c06adb0 0x7fbc1c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:57.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.654+0000 7fbc38c00700 1 -- 172.21.15.12:0/2218406358 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc3c0ca5f0 con 0x7fbc3c15dad0 2024-03-29T10:09:57.698 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:57.698+0000 7fd1471b9240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:09:57.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.850+0000 7fbc40c00700 1 -- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc1c06adb0 msgr2=0x7fbc1c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:57.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.850+0000 7fbc40c00700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc1c06adb0 0x7fbc1c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:57.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.870+0000 7fbc43600700 1 -- 172.21.15.12:0/2218406358 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbc3c0ca5f0 con 0x7fbc3c15dad0 2024-03-29T10:09:57.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.870+0000 7fbc38c00700 1 -- 172.21.15.12:0/2218406358 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbc3c0ca5f0 con 0x7fbc3c15dad0 2024-03-29T10:09:57.871 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:09:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 -- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc1c06adb0 msgr2=0x7fbc1c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc1c06adb0 0x7fbc1c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 -- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c15dad0 msgr2=0x7fbc3c151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c15dad0 0x7fbc3c151190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800e9e0 tx=0x7fbc2800ecf0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 -- 172.21.15.12:0/2218406358 shutdown_connections 2024-03-29T10:09:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbc1c06adb0 0x7fbc1c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc3c15fcb0 0x7fbc3c0b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc3c0ced80 0x7fbc3c15cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 --2- 172.21.15.12:0/2218406358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc3c15dad0 0x7fbc3c151190 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:57.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 -- 172.21.15.12:0/2218406358 >> 172.21.15.12:0/2218406358 conn(0x7fbc3c08aff0 msgr2=0x7fbc3c1536b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:57.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 -- 172.21.15.12:0/2218406358 shutdown_connections 2024-03-29T10:09:57.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:57.873+0000 7fbc43600700 1 -- 172.21.15.12:0/2218406358 wait complete. 2024-03-29T10:09:57.893 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:09:58.061 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:58.061+0000 7f1caccb1240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:09:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.097+0000 7fc29a400700 1 Processor -- start 2024-03-29T10:09:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.097+0000 7fc29a400700 1 -- start start 2024-03-29T10:09:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.098+0000 7fc29a400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc29415d9d0 0x7fc2940d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.098+0000 7fc29a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2940d16a0 0x7fc2940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc29a400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc294161420 con 0x7fc2940d2370 2024-03-29T10:09:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc29a400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2940d2820 con 0x7fc29415d9d0 2024-03-29T10:09:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc29a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc294161700 con 0x7fc2940d16a0 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc293400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc29415d9d0 0x7fc2940d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc292a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2940d16a0 0x7fc2940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc292a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2940d16a0 0x7fc2940d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:34740/0 (socket says 172.21.15.12:34740) 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc293400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc29415d9d0 0x7fc2940d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:57190/0 (socket says 172.21.15.12:57190) 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc292a00700 1 -- 172.21.15.12:0/3685758333 learned_addr learned my addr 172.21.15.12:0/3685758333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc293e00700 1 --1- 172.21.15.12:0/3685758333 >> v1:172.21.15.159:6789/0 conn(0x7fc2940d2370 0x7fc29415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:57974/0 (socket says 172.21.15.12:57974) 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc292a00700 1 -- 172.21.15.12:0/3685758333 >> v2:172.21.15.159:3300/0 conn(0x7fc29415d9d0 msgr2=0x7fc2940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.099+0000 7fc292a00700 1 --2- 172.21.15.12:0/3685758333 >> v2:172.21.15.159:3300/0 conn(0x7fc29415d9d0 0x7fc2940d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.100+0000 7fc292a00700 1 -- 172.21.15.12:0/3685758333 >> v1:172.21.15.159:6789/0 conn(0x7fc2940d2370 legacy=0x7fc29415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.100+0000 7fc292a00700 1 -- 172.21.15.12:0/3685758333 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2941619e0 con 0x7fc2940d16a0 2024-03-29T10:09:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.100+0000 7fc292a00700 1 --2- 172.21.15.12:0/3685758333 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2940d16a0 0x7fc2940d1e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc280006fd0 tx=0x7fc280009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ed9df536842a9ef server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.100+0000 7fc292000700 1 -- 172.21.15.12:0/3685758333 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280005e30 con 0x7fc2940d16a0 2024-03-29T10:09:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.100+0000 7fc292000700 1 -- 172.21.15.12:0/3685758333 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc280004030 con 0x7fc2940d16a0 2024-03-29T10:09:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.100+0000 7fc292000700 1 -- 172.21.15.12:0/3685758333 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2800043e0 con 0x7fc2940d16a0 2024-03-29T10:09:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.101+0000 7fc29a400700 1 -- 172.21.15.12:0/3685758333 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2940d16a0 msgr2=0x7fc2940d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.101+0000 7fc29a400700 1 --2- 172.21.15.12:0/3685758333 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2940d16a0 0x7fc2940d1e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc280006fd0 tx=0x7fc280009bd0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.101+0000 7fc29a400700 1 -- 172.21.15.12:0/3685758333 shutdown_connections 2024-03-29T10:09:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.101+0000 7fc29a400700 1 --2- 172.21.15.12:0/3685758333 >> v2:172.21.15.159:3300/0 conn(0x7fc29415d9d0 0x7fc2940d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.101+0000 7fc29a400700 1 --2- 172.21.15.12:0/3685758333 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2940d16a0 0x7fc2940d1e30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.101+0000 7fc29a400700 1 -- 172.21.15.12:0/3685758333 >> 172.21.15.12:0/3685758333 conn(0x7fc29408aff0 msgr2=0x7fc2940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.101+0000 7fc29a400700 1 -- 172.21.15.12:0/3685758333 shutdown_connections 2024-03-29T10:09:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.101+0000 7fc29a400700 1 -- 172.21.15.12:0/3685758333 wait complete. 2024-03-29T10:09:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.103+0000 7fc29a400700 1 Processor -- start 2024-03-29T10:09:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.103+0000 7fc29a400700 1 -- start start 2024-03-29T10:09:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc29a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2940d16a0 0x7fc29415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc29a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2940d2370 0x7fc2940cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc29a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc29415d9d0 0x7fc29415c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc29a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940cd5e0 con 0x7fc2940d2370 2024-03-29T10:09:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc29a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940b8cb0 con 0x7fc2940d16a0 2024-03-29T10:09:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc29a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940b70d0 con 0x7fc29415d9d0 2024-03-29T10:09:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc293e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc29415d9d0 0x7fc29415c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc292a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2940d2370 0x7fc2940cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc293400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2940d16a0 0x7fc29415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc293400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2940d16a0 0x7fc29415cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:57194/0 (socket says 172.21.15.12:57194) 2024-03-29T10:09:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc293e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc29415d9d0 0x7fc29415c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:34752/0 (socket says 172.21.15.12:34752) 2024-03-29T10:09:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc293400700 1 -- 172.21.15.12:0/3162551075 learned_addr learned my addr 172.21.15.12:0/3162551075 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.104+0000 7fc292a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2940d2370 0x7fc2940cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36946/0 (socket says 172.21.15.12:36946) 2024-03-29T10:09:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc293400700 1 -- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc29415d9d0 msgr2=0x7fc29415c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc293400700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc29415d9d0 0x7fc29415c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc293400700 1 -- 172.21.15.12:0/3162551075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2940d2370 msgr2=0x7fc2940cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc293400700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2940d2370 0x7fc2940cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc293400700 1 -- 172.21.15.12:0/3162551075 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc280009500 con 0x7fc2940d16a0 2024-03-29T10:09:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc293e00700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc29415d9d0 0x7fc29415c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc292a00700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2940d2370 0x7fc2940cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc293400700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2940d16a0 0x7fc29415cb00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc28400e870 tx=0x7fc28400eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc28b400700 1 -- 172.21.15.12:0/3162551075 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284009590 con 0x7fc2940d16a0 2024-03-29T10:09:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc28b400700 1 -- 172.21.15.12:0/3162551075 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc284013680 con 0x7fc2940d16a0 2024-03-29T10:09:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.105+0000 7fc28b400700 1 -- 172.21.15.12:0/3162551075 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2840139f0 con 0x7fc2940d16a0 2024-03-29T10:09:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.106+0000 7fc29a400700 1 -- 172.21.15.12:0/3162551075 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2940b73b0 con 0x7fc2940d16a0 2024-03-29T10:09:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.106+0000 7fc29a400700 1 -- 172.21.15.12:0/3162551075 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2940ca690 con 0x7fc2940d16a0 2024-03-29T10:09:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.107+0000 7fc29a400700 1 -- 172.21.15.12:0/3162551075 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2940d1ca0 con 0x7fc2940d16a0 2024-03-29T10:09:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.108+0000 7fc28b400700 1 -- 172.21.15.12:0/3162551075 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fc284013b50 con 0x7fc2940d16a0 2024-03-29T10:09:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.108+0000 7fc28b400700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc27406adb0 0x7fc27406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.109+0000 7fc28b400700 1 -- 172.21.15.12:0/3162551075 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc2840894f0 con 0x7fc2940d16a0 2024-03-29T10:09:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.109+0000 7fc292a00700 1 -- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc27406adb0 msgr2=0x7fc27406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.109+0000 7fc292a00700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc27406adb0 0x7fc27406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.113+0000 7fc28b400700 1 -- 172.21.15.12:0/3162551075 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2840543c0 con 0x7fc2940d16a0 2024-03-29T10:09:58.193 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:58.192+0000 7fd1471b9240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:09:58.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.310+0000 7fc292a00700 1 -- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc27406adb0 msgr2=0x7fc27406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:58.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.310+0000 7fc292a00700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc27406adb0 0x7fc27406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:58.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.329+0000 7fc29a400700 1 -- 172.21.15.12:0/3162551075 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc294067720 con 0x7fc2940d16a0 2024-03-29T10:09:58.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.330+0000 7fc28b400700 1 -- 172.21.15.12:0/3162551075 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc2840579e0 con 0x7fc2940d16a0 2024-03-29T10:09:58.330 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:09:58.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.332+0000 7fc29a400700 1 -- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc27406adb0 msgr2=0x7fc27406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.332+0000 7fc29a400700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc27406adb0 0x7fc27406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.332+0000 7fc29a400700 1 -- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2940d16a0 msgr2=0x7fc29415cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.332+0000 7fc29a400700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2940d16a0 0x7fc29415cb00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc28400e870 tx=0x7fc28400eb80 comp rx=0 tx=0).stop 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.332+0000 7fc29a400700 1 -- 172.21.15.12:0/3162551075 shutdown_connections 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.332+0000 7fc29a400700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc27406adb0 0x7fc27406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.332+0000 7fc29a400700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc29415d9d0 0x7fc29415c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.332+0000 7fc29a400700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2940d16a0 0x7fc29415cb00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.332+0000 7fc29a400700 1 --2- 172.21.15.12:0/3162551075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2940d2370 0x7fc2940cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.332+0000 7fc29a400700 1 -- 172.21.15.12:0/3162551075 >> 172.21.15.12:0/3162551075 conn(0x7fc29408aff0 msgr2=0x7fc2940b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.333+0000 7fc29a400700 1 -- 172.21.15.12:0/3162551075 shutdown_connections 2024-03-29T10:09:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.333+0000 7fc29a400700 1 -- 172.21.15.12:0/3162551075 wait complete. 2024-03-29T10:09:58.351 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T10:09:58.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.552+0000 7facf7600700 1 Processor -- start 2024-03-29T10:09:58.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.552+0000 7facf7600700 1 -- start start 2024-03-29T10:09:58.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facf7600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7facf015dad0 0x7facf00ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facf7600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7facf00d1630 0x7facf00d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facf7600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf00d2000 0x7facf015deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facf7600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf00cfcc0 con 0x7facf00d1630 2024-03-29T10:09:58.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facf7600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf00cd9f0 con 0x7facf015dad0 2024-03-29T10:09:58.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facf7600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf015f820 con 0x7facf00d2000 2024-03-29T10:09:58.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facefe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7facf00d1630 0x7facf00d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facefe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7facf00d1630 0x7facf00d1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36956/0 (socket says 172.21.15.12:36956) 2024-03-29T10:09:58.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facefe00700 1 -- 172.21.15.12:0/2627971469 learned_addr learned my addr 172.21.15.12:0/2627971469 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facf5600700 1 --2- 172.21.15.12:0/2627971469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf00d2000 0x7facf015deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.553+0000 7facf4c00700 1 --2- 172.21.15.12:0/2627971469 >> v2:172.21.15.159:3300/0 conn(0x7facf015dad0 0x7facf00ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facefe00700 1 -- 172.21.15.12:0/2627971469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf00d2000 msgr2=0x7facf015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facefe00700 1 --2- 172.21.15.12:0/2627971469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf00d2000 0x7facf015deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facefe00700 1 -- 172.21.15.12:0/2627971469 >> v2:172.21.15.159:3300/0 conn(0x7facf015dad0 msgr2=0x7facf00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facefe00700 1 --2- 172.21.15.12:0/2627971469 >> v2:172.21.15.159:3300/0 conn(0x7facf015dad0 0x7facf00ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facefe00700 1 -- 172.21.15.12:0/2627971469 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf00d2470 con 0x7facf00d1630 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facf5600700 1 --2- 172.21.15.12:0/2627971469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf00d2000 0x7facf015deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facf4c00700 1 --2- 172.21.15.12:0/2627971469 >> v2:172.21.15.159:3300/0 conn(0x7facf015dad0 0x7facf00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facefe00700 1 --2- 172.21.15.12:0/2627971469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d1630 0x7facf00d1ac0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7facf008d380 tx=0x7facdc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f1c888858427f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facef400700 1 -- 172.21.15.12:0/2627971469 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc00b870 con 0x7facf00d1630 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facef400700 1 -- 172.21.15.12:0/2627971469 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc00b9d0 con 0x7facf00d1630 2024-03-29T10:09:58.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.554+0000 7facef400700 1 -- 172.21.15.12:0/2627971469 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc00bd60 con 0x7facf00d1630 2024-03-29T10:09:58.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.555+0000 7facf7600700 1 -- 172.21.15.12:0/2627971469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d1630 msgr2=0x7facf00d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:58.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.555+0000 7facf7600700 1 --2- 172.21.15.12:0/2627971469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d1630 0x7facf00d1ac0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7facf008d380 tx=0x7facdc009bd0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.555+0000 7facf7600700 1 -- 172.21.15.12:0/2627971469 shutdown_connections 2024-03-29T10:09:58.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.555+0000 7facf7600700 1 --2- 172.21.15.12:0/2627971469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf00d2000 0x7facf015deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.555+0000 7facf7600700 1 --2- 172.21.15.12:0/2627971469 >> v2:172.21.15.159:3300/0 conn(0x7facf015dad0 0x7facf00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.555+0000 7facf7600700 1 --2- 172.21.15.12:0/2627971469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d1630 0x7facf00d1ac0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.555+0000 7facf7600700 1 -- 172.21.15.12:0/2627971469 >> 172.21.15.12:0/2627971469 conn(0x7facf008aff0 msgr2=0x7facf00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:58.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.555+0000 7facf7600700 1 -- 172.21.15.12:0/2627971469 shutdown_connections 2024-03-29T10:09:58.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.555+0000 7facf7600700 1 -- 172.21.15.12:0/2627971469 wait complete. 2024-03-29T10:09:58.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.557+0000 7facf7600700 1 Processor -- start 2024-03-29T10:09:58.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.557+0000 7facf7600700 1 -- start start 2024-03-29T10:09:58.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facf7600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facf00d1630 0x7facf01528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facf7600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d2000 0x7facf01f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facf7600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf015dad0 0x7facf00c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facf7600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf0144ed0 con 0x7facf00d2000 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facf7600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf0152e00 con 0x7facf00d1630 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facf7600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf01558f0 con 0x7facf015dad0 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facefe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d2000 0x7facf01f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facefe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d2000 0x7facf01f2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36958/0 (socket says 172.21.15.12:36958) 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facefe00700 1 -- 172.21.15.12:0/4240475475 learned_addr learned my addr 172.21.15.12:0/4240475475 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facf4c00700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facf00d1630 0x7facf01528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.558+0000 7facf5600700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf015dad0 0x7facf00c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facefe00700 1 -- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf015dad0 msgr2=0x7facf00c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facefe00700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf015dad0 0x7facf00c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facefe00700 1 -- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facf00d1630 msgr2=0x7facf01528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facefe00700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facf00d1630 0x7facf01528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facefe00700 1 -- 172.21.15.12:0/4240475475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc009550 con 0x7facf00d2000 2024-03-29T10:09:58.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facefe00700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d2000 0x7facf01f2f40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7facdc000c00 tx=0x7facdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facecc00700 1 -- 172.21.15.12:0/4240475475 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc004480 con 0x7facf00d2000 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facecc00700 1 -- 172.21.15.12:0/4240475475 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc0045e0 con 0x7facf00d2000 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facecc00700 1 -- 172.21.15.12:0/4240475475 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0038e0 con 0x7facf00d2000 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facf7600700 1 -- 172.21.15.12:0/4240475475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf01f3480 con 0x7facf00d2000 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.559+0000 7facf7600700 1 -- 172.21.15.12:0/4240475475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf00b90b0 con 0x7facf00d2000 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.560+0000 7facf7600700 1 -- 172.21.15.12:0/4240475475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf0155aa0 con 0x7facf00d2000 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.561+0000 7facecc00700 1 -- 172.21.15.12:0/4240475475 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7facdc021070 con 0x7facf00d2000 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.562+0000 7facecc00700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facd006adb0 0x7facd006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.562+0000 7facecc00700 1 -- 172.21.15.12:0/4240475475 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7facdc08ab00 con 0x7facf00d2000 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.562+0000 7facf4c00700 1 -- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facd006adb0 msgr2=0x7facd006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.562+0000 7facf4c00700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facd006adb0 0x7facd006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:58.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.567+0000 7facecc00700 1 -- 172.21.15.12:0/4240475475 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7facf0155aa0 con 0x7facf00d2000 2024-03-29T10:09:58.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.763+0000 7facf4c00700 1 -- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facd006adb0 msgr2=0x7facd006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:58.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.763+0000 7facf4c00700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facd006adb0 0x7facd006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:58.778 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:58.779+0000 7f1caccb1240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:09:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.799+0000 7facf7600700 1 -- 172.21.15.12:0/4240475475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7facf0067e30 con 0x7facf00d2000 2024-03-29T10:09:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.800+0000 7facecc00700 1 -- 172.21.15.12:0/4240475475 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v81) v1 ==== 80+0+467 (secure 0 0 0) 0x7facdc029080 con 0x7facf00d2000 2024-03-29T10:09:58.801 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3367842275 2024-03-30T10:09:53.746392+0000 2024-03-29T10:09:58.801 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3723093447 2024-03-30T10:09:53.746392+0000 2024-03-29T10:09:58.801 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/43444 2024-03-30T10:09:53.746392+0000 2024-03-29T10:09:58.801 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/43444 2024-03-30T10:09:53.746392+0000 2024-03-29T10:09:58.801 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1578988741 2024-03-30T10:09:53.746392+0000 2024-03-29T10:09:58.801 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1180347815 2024-03-30T10:09:53.746392+0000 2024-03-29T10:09:58.801 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/667128976 2024-03-30T10:09:53.746392+0000 2024-03-29T10:09:58.801 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3208721490 2024-03-30T10:09:53.746392+0000 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.802+0000 7facf7600700 1 -- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facd006adb0 msgr2=0x7facd006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.802+0000 7facf7600700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facd006adb0 0x7facd006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.802+0000 7facf7600700 1 -- 172.21.15.12:0/4240475475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d2000 msgr2=0x7facf01f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.802+0000 7facf7600700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d2000 0x7facf01f2f40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7facdc000c00 tx=0x7facdc004750 comp rx=0 tx=0).stop 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.803+0000 7facf7600700 1 -- 172.21.15.12:0/4240475475 shutdown_connections 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.803+0000 7facf7600700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7facd006adb0 0x7facd006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.803+0000 7facf7600700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facf015dad0 0x7facf00c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.803+0000 7facf7600700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facf00d1630 0x7facf01528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.803+0000 7facf7600700 1 --2- 172.21.15.12:0/4240475475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facf00d2000 0x7facf01f2f40 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.803+0000 7facf7600700 1 -- 172.21.15.12:0/4240475475 >> 172.21.15.12:0/4240475475 conn(0x7facf008aff0 msgr2=0x7facf00ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.803+0000 7facf7600700 1 -- 172.21.15.12:0/4240475475 shutdown_connections 2024-03-29T10:09:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:58.803+0000 7facf7600700 1 -- 172.21.15.12:0/4240475475 wait complete. 2024-03-29T10:09:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T10:09:58.821 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T10:09:58.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:58.822 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T10:09:58.871 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:58.871+0000 7f1caccb1240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:09:58.896 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T10:09:58.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:09:58.896 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T10:09:58.941 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:09:58.942 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:58.965 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:58.964+0000 7fd1471b9240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:09:58.986 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:09:58.986 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:09:58.986 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:59.031 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:09:59.031 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:09:59.031 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:09:59.037 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:59.037+0000 7f1caccb1240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:09:59.064 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:59.064+0000 7fd1471b9240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:09:59.076 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:09:59.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:09:59.241 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:59.240+0000 7fd1471b9240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:09:59.257 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:59.257+0000 7f1caccb1240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:09:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.276+0000 7f2446c00700 1 Processor -- start 2024-03-29T10:09:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.276+0000 7f2446c00700 1 -- start start 2024-03-29T10:09:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.277+0000 7f2446c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f24400ceec0 0x7f24401612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.277+0000 7f2446c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2440161c50 con 0x7f2440161800 2024-03-29T10:09:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.277+0000 7f2446c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24400d30d0 con 0x7f244015d640 2024-03-29T10:09:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.277+0000 7f2446c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24400b6840 con 0x7f24400ceec0 2024-03-29T10:09:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.278+0000 7f243f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f24400ceec0 0x7f24401612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.278+0000 7f2444c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2440161800 0x7f24400b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36264/0 (socket says 172.21.15.12:36264) 2024-03-29T10:09:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.278+0000 7f2444c00700 1 -- 172.21.15.12:0/2164198216 learned_addr learned my addr 172.21.15.12:0/2164198216 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.278+0000 7f243f400700 1 -- 172.21.15.12:0/2164198216 >> v1:172.21.15.159:6789/0 conn(0x7f244015d640 legacy=0x7f24400d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.278+0000 7f243f400700 1 -- 172.21.15.12:0/2164198216 >> v1:172.21.15.12:6789/0 conn(0x7f2440161800 legacy=0x7f24400b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.278+0000 7f243ea00700 1 -- 172.21.15.12:0/2164198216 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928759262 0 0) 0x7f2438001490 con 0x7f2440161800 2024-03-29T10:09:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.278+0000 7f243f400700 1 -- 172.21.15.12:0/2164198216 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24400d2b30 con 0x7f24400ceec0 2024-03-29T10:09:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.279+0000 7f243f400700 1 --2- 172.21.15.12:0/2164198216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24400ceec0 0x7f24401612c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f242c009840 tx=0x7f242c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0ea2db979426b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.279+0000 7f243ea00700 1 -- 172.21.15.12:0/2164198216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c011830 con 0x7f24400ceec0 2024-03-29T10:09:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.279+0000 7f243ea00700 1 -- 172.21.15.12:0/2164198216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242c011990 con 0x7f24400ceec0 2024-03-29T10:09:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.279+0000 7f243ea00700 1 -- 172.21.15.12:0/2164198216 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c011d20 con 0x7f24400ceec0 2024-03-29T10:09:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.279+0000 7f2446c00700 1 -- 172.21.15.12:0/2164198216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24400ceec0 msgr2=0x7f24401612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.279+0000 7f2446c00700 1 --2- 172.21.15.12:0/2164198216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24400ceec0 0x7f24401612c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f242c009840 tx=0x7f242c009b50 comp rx=0 tx=0).stop 2024-03-29T10:09:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.280+0000 7f2446c00700 1 -- 172.21.15.12:0/2164198216 shutdown_connections 2024-03-29T10:09:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.280+0000 7f2446c00700 1 --2- 172.21.15.12:0/2164198216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24400ceec0 0x7f24401612c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.280+0000 7f2446c00700 1 -- 172.21.15.12:0/2164198216 >> 172.21.15.12:0/2164198216 conn(0x7f244008aff0 msgr2=0x7f24400d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.280+0000 7f2446c00700 1 -- 172.21.15.12:0/2164198216 shutdown_connections 2024-03-29T10:09:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.280+0000 7f2446c00700 1 -- 172.21.15.12:0/2164198216 wait complete. 2024-03-29T10:09:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.281+0000 7f2446c00700 1 Processor -- start 2024-03-29T10:09:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.282+0000 7f2446c00700 1 -- start start 2024-03-29T10:09:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.282+0000 7f2446c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24400ceec0 0x7f24400c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f2446c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244015d640 0x7f244015f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f2446c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2440161800 0x7f244015fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f2446c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2440153320 con 0x7f244015d640 2024-03-29T10:09:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f2446c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24400c9d80 con 0x7f2440161800 2024-03-29T10:09:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f2446c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24400c8090 con 0x7f24400ceec0 2024-03-29T10:09:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f243f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244015d640 0x7f244015f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f243f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244015d640 0x7f244015f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55394/0 (socket says 172.21.15.12:55394) 2024-03-29T10:09:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f243f400700 1 -- 172.21.15.12:0/1682307492 learned_addr learned my addr 172.21.15.12:0/1682307492 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f2444c00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2440161800 0x7f244015fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f243fe00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24400ceec0 0x7f24400c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f243f400700 1 -- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24400ceec0 msgr2=0x7f24400c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f243f400700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24400ceec0 0x7f24400c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f243f400700 1 -- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2440161800 msgr2=0x7f244015fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f243f400700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2440161800 0x7f244015fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.283+0000 7f243f400700 1 -- 172.21.15.12:0/1682307492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c0094a0 con 0x7f244015d640 2024-03-29T10:09:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.284+0000 7f243f400700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244015d640 0x7f244015f670 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f242c014040 tx=0x7f242c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.284+0000 7f2437e00700 1 -- 172.21.15.12:0/1682307492 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c01a7e0 con 0x7f244015d640 2024-03-29T10:09:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.284+0000 7f2437e00700 1 -- 172.21.15.12:0/1682307492 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242c01a940 con 0x7f244015d640 2024-03-29T10:09:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.284+0000 7f2446c00700 1 -- 172.21.15.12:0/1682307492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24400c8370 con 0x7f244015d640 2024-03-29T10:09:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.284+0000 7f2437e00700 1 -- 172.21.15.12:0/1682307492 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c01acb0 con 0x7f244015d640 2024-03-29T10:09:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.284+0000 7f2446c00700 1 -- 172.21.15.12:0/1682307492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24400c8930 con 0x7f244015d640 2024-03-29T10:09:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.285+0000 7f2446c00700 1 -- 172.21.15.12:0/1682307492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24400c8b20 con 0x7f244015d640 2024-03-29T10:09:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.287+0000 7f2437e00700 1 -- 172.21.15.12:0/1682307492 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f242c022070 con 0x7f244015d640 2024-03-29T10:09:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.288+0000 7f2437e00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f242006adb0 0x7f242006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.288+0000 7f2437e00700 1 -- 172.21.15.12:0/1682307492 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f242c08a7f0 con 0x7f244015d640 2024-03-29T10:09:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.288+0000 7f243fe00700 1 -- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f242006adb0 msgr2=0x7f242006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.288+0000 7f243fe00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f242006adb0 0x7f242006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.293+0000 7f2437e00700 1 -- 172.21.15.12:0/1682307492 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f242c055750 con 0x7f244015d640 2024-03-29T10:09:59.349 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:59.349+0000 7f1caccb1240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:09:59.448 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:59.448+0000 7f1caccb1240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:09:59.464 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:59.464+0000 7fd1471b9240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:09:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.489+0000 7f243fe00700 1 -- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f242006adb0 msgr2=0x7f242006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.489+0000 7f243fe00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f242006adb0 0x7f242006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:09:59.546 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:59.546+0000 7f1caccb1240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:09:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.552+0000 7f2446c00700 1 -- 172.21.15.12:0/1682307492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f244009f980 con 0x7f244015d640 2024-03-29T10:09:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.553+0000 7f2437e00700 1 -- 172.21.15.12:0/1682307492 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f242c058d70 con 0x7f244015d640 2024-03-29T10:09:59.554 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:09:59.554 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3202930550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3202930550},{"type":"v1","addr":"172.21.15.159:6835","nonce":3202930550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4189378403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4189378403},{"type":"v1","addr":"172.21.15.12:6834","nonce":4189378403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2681299760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2681299760},{"type":"v1","addr":"172.21.15.12:6835","nonce":2681299760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:09:32.597152+0000","modified":"2024-03-29T10:09:35.607262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6241":{"gid":6241,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/4289275091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":4289275091},{"type":"v1","addr":"172.21.15.159:6837","nonce":4289275091}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.555+0000 7f2446c00700 1 -- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f242006adb0 msgr2=0x7f242006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.555+0000 7f2446c00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f242006adb0 0x7f242006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.556+0000 7f2446c00700 1 -- 172.21.15.12:0/1682307492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244015d640 msgr2=0x7f244015f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.556+0000 7f2446c00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244015d640 0x7f244015f670 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f242c014040 tx=0x7f242c004750 comp rx=0 tx=0).stop 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.556+0000 7f2446c00700 1 -- 172.21.15.12:0/1682307492 shutdown_connections 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.556+0000 7f2446c00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f242006adb0 0x7f242006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.556+0000 7f2446c00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2440161800 0x7f244015fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.556+0000 7f2446c00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24400ceec0 0x7f24400c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.556+0000 7f2446c00700 1 --2- 172.21.15.12:0/1682307492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244015d640 0x7f244015f670 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.556+0000 7f2446c00700 1 -- 172.21.15.12:0/1682307492 >> 172.21.15.12:0/1682307492 conn(0x7f244008aff0 msgr2=0x7f24400d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.556+0000 7f2446c00700 1 -- 172.21.15.12:0/1682307492 shutdown_connections 2024-03-29T10:09:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.556+0000 7f2446c00700 1 -- 172.21.15.12:0/1682307492 wait complete. 2024-03-29T10:09:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 31 2024-03-29T10:09:59.562 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:59.561+0000 7fd1471b9240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:09:59.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:09:59.645 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:59.645+0000 7f1caccb1240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:09:59.670 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:59.669+0000 7fd1471b9240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:09:59.776 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:59.776+0000 7fd1471b9240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:09:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.776+0000 7f6c50000700 1 Processor -- start 2024-03-29T10:09:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.776+0000 7f6c50000700 1 -- start start 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.777+0000 7f6c50000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6c480d05e0 0x7f6c480d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.777+0000 7f6c50000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6c480b6990 0x7f6c480b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.777+0000 7f6c50000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c480d0f80 con 0x7f6c4815da50 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.777+0000 7f6c50000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c480b95d0 con 0x7f6c480b6990 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.777+0000 7f6c50000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c4815de30 con 0x7f6c480d05e0 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.778+0000 7f6c4d600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6c4815da50 0x7f6c480ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36276/0 (socket says 172.21.15.12:36276) 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.778+0000 7f6c4d600700 1 -- 172.21.15.12:0/3567047061 learned_addr learned my addr 172.21.15.12:0/3567047061 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.778+0000 7f6c4e000700 1 --2- 172.21.15.12:0/3567047061 >> v2:172.21.15.12:3300/0 conn(0x7f6c480b6990 0x7f6c480b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.778+0000 7f6c4cc00700 1 --2- 172.21.15.12:0/3567047061 >> v2:172.21.15.159:3300/0 conn(0x7f6c480d05e0 0x7f6c480d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.778+0000 7f6c47e00700 1 -- 172.21.15.12:0/3567047061 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304354678 0 0) 0x7f6c480d0f80 con 0x7f6c4815da50 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.778+0000 7f6c4e000700 1 -- 172.21.15.12:0/3567047061 >> v2:172.21.15.159:3300/0 conn(0x7f6c480d05e0 msgr2=0x7f6c480d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:09:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.778+0000 7f6c4e000700 1 --2- 172.21.15.12:0/3567047061 >> v2:172.21.15.159:3300/0 conn(0x7f6c480d05e0 0x7f6c480d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.778+0000 7f6c4e000700 1 -- 172.21.15.12:0/3567047061 >> v1:172.21.15.12:6789/0 conn(0x7f6c4815da50 legacy=0x7f6c480ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.778+0000 7f6c4e000700 1 -- 172.21.15.12:0/3567047061 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c480d24e0 con 0x7f6c480b6990 2024-03-29T10:09:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.779+0000 7f6c4e000700 1 --2- 172.21.15.12:0/3567047061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480b6990 0x7f6c480b6e40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6c480740a0 tx=0x7f6c4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8fd3f3e50cf6c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.779+0000 7f6c47e00700 1 -- 172.21.15.12:0/3567047061 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4000e070 con 0x7f6c480b6990 2024-03-29T10:09:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.779+0000 7f6c47e00700 1 -- 172.21.15.12:0/3567047061 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4000bb20 con 0x7f6c480b6990 2024-03-29T10:09:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.779+0000 7f6c47e00700 1 -- 172.21.15.12:0/3567047061 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40012430 con 0x7f6c480b6990 2024-03-29T10:09:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.779+0000 7f6c50000700 1 -- 172.21.15.12:0/3567047061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480b6990 msgr2=0x7f6c480b6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.779+0000 7f6c50000700 1 --2- 172.21.15.12:0/3567047061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480b6990 0x7f6c480b6e40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6c480740a0 tx=0x7f6c4000b330 comp rx=0 tx=0).stop 2024-03-29T10:09:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.780+0000 7f6c50000700 1 -- 172.21.15.12:0/3567047061 shutdown_connections 2024-03-29T10:09:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.780+0000 7f6c50000700 1 --2- 172.21.15.12:0/3567047061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480b6990 0x7f6c480b6e40 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.780+0000 7f6c50000700 1 --2- 172.21.15.12:0/3567047061 >> v2:172.21.15.159:3300/0 conn(0x7f6c480d05e0 0x7f6c480d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.780+0000 7f6c50000700 1 -- 172.21.15.12:0/3567047061 >> 172.21.15.12:0/3567047061 conn(0x7f6c480666a0 msgr2=0x7f6c48066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:09:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.780+0000 7f6c50000700 1 -- 172.21.15.12:0/3567047061 shutdown_connections 2024-03-29T10:09:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.780+0000 7f6c50000700 1 -- 172.21.15.12:0/3567047061 wait complete. 2024-03-29T10:09:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.781+0000 7f6c50000700 1 Processor -- start 2024-03-29T10:09:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.782+0000 7f6c50000700 1 -- start start 2024-03-29T10:09:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.782+0000 7f6c50000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c480b6990 0x7f6c480c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c50000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480d05e0 0x7f6c480b7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c50000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c4815da50 0x7f6c480b7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c50000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480cd890 con 0x7f6c480d05e0 2024-03-29T10:09:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c50000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480ccff0 con 0x7f6c4815da50 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c50000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480cd2d0 con 0x7f6c480b6990 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480d05e0 0x7f6c480b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480d05e0 0x7f6c480b7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55404/0 (socket says 172.21.15.12:55404) 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4cc00700 1 -- 172.21.15.12:0/4100977482 learned_addr learned my addr 172.21.15.12:0/4100977482 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4e000700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c4815da50 0x7f6c480b7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4d600700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c480b6990 0x7f6c480c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4cc00700 1 -- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c480b6990 msgr2=0x7f6c480c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4cc00700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c480b6990 0x7f6c480c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4cc00700 1 -- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c4815da50 msgr2=0x7f6c480b7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4cc00700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c4815da50 0x7f6c480b7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4cc00700 1 -- 172.21.15.12:0/4100977482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c40009a70 con 0x7f6c480d05e0 2024-03-29T10:09:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.783+0000 7f6c4e000700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c4815da50 0x7f6c480b7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:09:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.784+0000 7f6c4cc00700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480d05e0 0x7f6c480b7830 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400e980 tx=0x7f6c3400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:09:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.784+0000 7f6c45600700 1 -- 172.21.15.12:0/4100977482 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c34009710 con 0x7f6c480d05e0 2024-03-29T10:09:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.784+0000 7f6c45600700 1 -- 172.21.15.12:0/4100977482 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3400c690 con 0x7f6c480d05e0 2024-03-29T10:09:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.784+0000 7f6c45600700 1 -- 172.21.15.12:0/4100977482 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3400ca20 con 0x7f6c480d05e0 2024-03-29T10:09:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.784+0000 7f6c50000700 1 -- 172.21.15.12:0/4100977482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c480cd5b0 con 0x7f6c480d05e0 2024-03-29T10:09:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.784+0000 7f6c50000700 1 -- 172.21.15.12:0/4100977482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c480ca220 con 0x7f6c480d05e0 2024-03-29T10:09:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.785+0000 7f6c50000700 1 -- 172.21.15.12:0/4100977482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c48067720 con 0x7f6c480d05e0 2024-03-29T10:09:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.787+0000 7f6c45600700 1 -- 172.21.15.12:0/4100977482 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f6c3400cb80 con 0x7f6c480d05e0 2024-03-29T10:09:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.788+0000 7f6c45600700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c2806adb0 0x7f6c2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:09:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.789+0000 7f6c45600700 1 -- 172.21.15.12:0/4100977482 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6c34089600 con 0x7f6c480d05e0 2024-03-29T10:09:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.789+0000 7f6c4d600700 1 -- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c2806adb0 msgr2=0x7f6c2806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:59.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.789+0000 7f6c4d600700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c2806adb0 0x7f6c2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:09:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.793+0000 7f6c45600700 1 -- 172.21.15.12:0/4100977482 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c340544d0 con 0x7f6c480d05e0 2024-03-29T10:09:59.858 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:09:59.859+0000 7f1caccb1240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:09:59.885 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:09:59.884+0000 7fd1471b9240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:09:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.990+0000 7f6c4d600700 1 -- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c2806adb0 msgr2=0x7f6c2806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:09:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:09:59.990+0000 7f6c4d600700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c2806adb0 0x7f6c2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.009+0000 7f6c50000700 1 -- 172.21.15.12:0/4100977482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c480b5850 con 0x7f6c480d05e0 2024-03-29T10:10:00.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.010+0000 7f6c45600700 1 -- 172.21.15.12:0/4100977482 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16595 (secure 0 0 0) 0x7f6c34057af0 con 0x7f6c480d05e0 2024-03-29T10:10:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:00.011 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:53.746428+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:09:30.441619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-29T10:09:31.581366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3367842275":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/3723093447":"2024-03-30T10:09:53.746392+0000","172.21.15.159:6833/43444":"2024-03-30T10:09:53.746392+0000","172.21.15.159:6832/43444":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/1578988741":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/1180347815":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/667128976":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/3208721490":"2024-03-30T10:09:53.746392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 -- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c2806adb0 msgr2=0x7f6c2806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c2806adb0 0x7f6c2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 -- 172.21.15.12:0/4100977482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480d05e0 msgr2=0x7f6c480b7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480d05e0 0x7f6c480b7830 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400e980 tx=0x7f6c3400ec90 comp rx=0 tx=0).stop 2024-03-29T10:10:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 -- 172.21.15.12:0/4100977482 shutdown_connections 2024-03-29T10:10:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c2806adb0 0x7f6c2806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c4815da50 0x7f6c480b7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c480b6990 0x7f6c480c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 --2- 172.21.15.12:0/4100977482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c480d05e0 0x7f6c480b7830 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 -- 172.21.15.12:0/4100977482 >> 172.21.15.12:0/4100977482 conn(0x7f6c480666a0 msgr2=0x7f6c48154d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:00.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 -- 172.21.15.12:0/4100977482 shutdown_connections 2024-03-29T10:10:00.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.013+0000 7f6c50000700 1 -- 172.21.15.12:0/4100977482 wait complete. 2024-03-29T10:10:00.032 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:09:30.441619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:09:31.581366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:10:00.033 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:10:00.111 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:00.110+0000 7fd1471b9240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:10:00.116 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:00.116+0000 7f1caccb1240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:10:00.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.239+0000 7f9979800700 1 Processor -- start 2024-03-29T10:10:00.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.240+0000 7f9979800700 1 -- start start 2024-03-29T10:10:00.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.241+0000 7f9979800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f99741611b0 0x7f997415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.241+0000 7f9979800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99740cee10 0x7f9974161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.241+0000 7f9979800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99740ce270 con 0x7f997415f4d0 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.241+0000 7f9979800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f997415b600 con 0x7f99740cee10 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.241+0000 7f9979800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99740d2760 con 0x7f99741611b0 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.241+0000 7f9973400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99740cee10 0x7f9974161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.241+0000 7f9972000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f997415f4d0 0x7f99740ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36280/0 (socket says 172.21.15.12:36280) 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.241+0000 7f9972000700 1 -- 172.21.15.12:0/151884888 learned_addr learned my addr 172.21.15.12:0/151884888 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.241+0000 7f9972a00700 1 --2- 172.21.15.12:0/151884888 >> v2:172.21.15.159:3300/0 conn(0x7f99741611b0 0x7f997415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.241+0000 7f9971600700 1 -- 172.21.15.12:0/151884888 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533041087 0 0) 0x7f99740ce270 con 0x7f997415f4d0 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.242+0000 7f9973400700 1 -- 172.21.15.12:0/151884888 >> v2:172.21.15.159:3300/0 conn(0x7f99741611b0 msgr2=0x7f997415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.242+0000 7f9973400700 1 --2- 172.21.15.12:0/151884888 >> v2:172.21.15.159:3300/0 conn(0x7f99741611b0 0x7f997415ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.242+0000 7f9973400700 1 -- 172.21.15.12:0/151884888 >> v1:172.21.15.12:6789/0 conn(0x7f997415f4d0 legacy=0x7f99740ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.242+0000 7f9973400700 1 -- 172.21.15.12:0/151884888 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99740d2a40 con 0x7f99740cee10 2024-03-29T10:10:00.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.242+0000 7f9973400700 1 --2- 172.21.15.12:0/151884888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 0x7f9974161590 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f997408b520 tx=0x7f996c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d70ed6dcc5bafc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:00.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.242+0000 7f9971600700 1 -- 172.21.15.12:0/151884888 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c00e070 con 0x7f99740cee10 2024-03-29T10:10:00.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.242+0000 7f9971600700 1 -- 172.21.15.12:0/151884888 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c00bb20 con 0x7f99740cee10 2024-03-29T10:10:00.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.242+0000 7f9971600700 1 -- 172.21.15.12:0/151884888 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c012430 con 0x7f99740cee10 2024-03-29T10:10:00.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.243+0000 7f9979800700 1 -- 172.21.15.12:0/151884888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 msgr2=0x7f9974161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.243+0000 7f9979800700 1 --2- 172.21.15.12:0/151884888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 0x7f9974161590 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f997408b520 tx=0x7f996c00b330 comp rx=0 tx=0).stop 2024-03-29T10:10:00.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.243+0000 7f9979800700 1 -- 172.21.15.12:0/151884888 shutdown_connections 2024-03-29T10:10:00.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.243+0000 7f9979800700 1 --2- 172.21.15.12:0/151884888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 0x7f9974161590 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.243+0000 7f9979800700 1 --2- 172.21.15.12:0/151884888 >> v2:172.21.15.159:3300/0 conn(0x7f99741611b0 0x7f997415ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.243+0000 7f9979800700 1 -- 172.21.15.12:0/151884888 >> 172.21.15.12:0/151884888 conn(0x7f997408aff0 msgr2=0x7f99740d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:00.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.243+0000 7f9979800700 1 -- 172.21.15.12:0/151884888 shutdown_connections 2024-03-29T10:10:00.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.243+0000 7f9979800700 1 -- 172.21.15.12:0/151884888 wait complete. 2024-03-29T10:10:00.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.246+0000 7f9979800700 1 Processor -- start 2024-03-29T10:10:00.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.246+0000 7f9979800700 1 -- start start 2024-03-29T10:10:00.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.247+0000 7f9979800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 0x7f99740cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.247+0000 7f9979800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f997415f4d0 0x7f99740cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.247+0000 7f9979800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99741611b0 0x7f99740b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.247+0000 7f9979800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99740b78f0 con 0x7f99740cee10 2024-03-29T10:10:00.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.247+0000 7f9979800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99740b7bd0 con 0x7f99741611b0 2024-03-29T10:10:00.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.247+0000 7f9979800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740b7eb0 con 0x7f997415f4d0 2024-03-29T10:10:00.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.247+0000 7f9972a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 0x7f99740cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:00.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.248+0000 7f9972000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f997415f4d0 0x7f99740cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:00.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.248+0000 7f9972a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 0x7f99740cd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55420/0 (socket says 172.21.15.12:55420) 2024-03-29T10:10:00.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.248+0000 7f9972a00700 1 -- 172.21.15.12:0/4199983935 learned_addr learned my addr 172.21.15.12:0/4199983935 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:00.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.248+0000 7f9973400700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99741611b0 0x7f99740b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:00.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.248+0000 7f9972a00700 1 -- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f997415f4d0 msgr2=0x7f99740cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.248+0000 7f9972a00700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f997415f4d0 0x7f99740cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.248+0000 7f9972a00700 1 -- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99741611b0 msgr2=0x7f99740b8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.248+0000 7f9972a00700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99741611b0 0x7f99740b8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.248+0000 7f9972a00700 1 -- 172.21.15.12:0/4199983935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c009a70 con 0x7f99740cee10 2024-03-29T10:10:00.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.248+0000 7f9972a00700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 0x7f99740cd0e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f996400e870 tx=0x7f996400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:00.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.249+0000 7f996aa00700 1 -- 172.21.15.12:0/4199983935 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964009590 con 0x7f99740cee10 2024-03-29T10:10:00.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.249+0000 7f9979800700 1 -- 172.21.15.12:0/4199983935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99740b8190 con 0x7f99740cee10 2024-03-29T10:10:00.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.249+0000 7f9979800700 1 -- 172.21.15.12:0/4199983935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9974156e80 con 0x7f99740cee10 2024-03-29T10:10:00.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.249+0000 7f996aa00700 1 -- 172.21.15.12:0/4199983935 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9964013680 con 0x7f99740cee10 2024-03-29T10:10:00.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.249+0000 7f996aa00700 1 -- 172.21.15.12:0/4199983935 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99640139f0 con 0x7f99740cee10 2024-03-29T10:10:00.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.250+0000 7f9979800700 1 -- 172.21.15.12:0/4199983935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c0052f0 con 0x7f99740cee10 2024-03-29T10:10:00.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.251+0000 7f996aa00700 1 -- 172.21.15.12:0/4199983935 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f9964013b90 con 0x7f99740cee10 2024-03-29T10:10:00.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.253+0000 7f996aa00700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995406adb0 0x7f995406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.253+0000 7f996aa00700 1 -- 172.21.15.12:0/4199983935 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f9964089500 con 0x7f99740cee10 2024-03-29T10:10:00.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.253+0000 7f9972000700 1 -- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995406adb0 msgr2=0x7f995406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:00.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.253+0000 7f9972000700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995406adb0 0x7f995406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:00.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.257+0000 7f996aa00700 1 -- 172.21.15.12:0/4199983935 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f99640543d0 con 0x7f99740cee10 2024-03-29T10:10:00.307 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:00.307+0000 7f1caccb1240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:10:00.378 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:00.378+0000 7fd1471b9240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:10:00.400 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:00.400+0000 7f1caccb1240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:00.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.454+0000 7f9972000700 1 -- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995406adb0 msgr2=0x7f995406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:00.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.454+0000 7f9972000700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995406adb0 0x7f995406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:00.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.473+0000 7f9979800700 1 -- 172.21.15.12:0/4199983935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f994c005b20 con 0x7f99740cee10 2024-03-29T10:10:00.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.474+0000 7f996aa00700 1 -- 172.21.15.12:0/4199983935 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f99640579f0 con 0x7f99740cee10 2024-03-29T10:10:00.475 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:10:00.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.476+0000 7f9979800700 1 -- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995406adb0 msgr2=0x7f995406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:00.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.476+0000 7f9979800700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995406adb0 0x7f995406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.476+0000 7f9979800700 1 -- 172.21.15.12:0/4199983935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 msgr2=0x7f99740cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.476+0000 7f9979800700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 0x7f99740cd0e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f996400e870 tx=0x7f996400eb80 comp rx=0 tx=0).stop 2024-03-29T10:10:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.477+0000 7f9979800700 1 -- 172.21.15.12:0/4199983935 shutdown_connections 2024-03-29T10:10:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.477+0000 7f9979800700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995406adb0 0x7f995406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.477+0000 7f9979800700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99741611b0 0x7f99740b8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.477+0000 7f9979800700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99740cee10 0x7f99740cd0e0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.477+0000 7f9979800700 1 --2- 172.21.15.12:0/4199983935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f997415f4d0 0x7f99740cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.477+0000 7f9979800700 1 -- 172.21.15.12:0/4199983935 >> 172.21.15.12:0/4199983935 conn(0x7f997408aff0 msgr2=0x7f997415da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.477+0000 7f9979800700 1 -- 172.21.15.12:0/4199983935 shutdown_connections 2024-03-29T10:10:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.477+0000 7f9979800700 1 -- 172.21.15.12:0/4199983935 wait complete. 2024-03-29T10:10:00.496 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:10:00.584 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:00.583+0000 7fd1471b9240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:10:00.684 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:00.684+0000 7fd1471b9240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:00.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.701+0000 7f6389c00700 1 Processor -- start 2024-03-29T10:10:00.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.701+0000 7f6389c00700 1 -- start start 2024-03-29T10:10:00.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6389c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f638415f1e0 0x7f638415f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6389c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f63840cef40 0x7f638415bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6389c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63840ccdd0 con 0x7f63840cf3f0 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6389c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63840cd260 con 0x7f638415f1e0 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6389c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63841600c0 con 0x7f63840cef40 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6382a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f638415f1e0 0x7f638415f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6382a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f638415f1e0 0x7f638415f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55434/0 (socket says 172.21.15.12:55434) 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6382a00700 1 -- 172.21.15.12:0/3743976051 learned_addr learned my addr 172.21.15.12:0/3743976051 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6382000700 1 --2- 172.21.15.12:0/3743976051 >> v2:172.21.15.159:3300/0 conn(0x7f63840cef40 0x7f638415bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6382a00700 1 -- 172.21.15.12:0/3743976051 >> v2:172.21.15.159:3300/0 conn(0x7f63840cef40 msgr2=0x7f638415bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6382a00700 1 --2- 172.21.15.12:0/3743976051 >> v2:172.21.15.159:3300/0 conn(0x7f63840cef40 0x7f638415bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6382a00700 1 -- 172.21.15.12:0/3743976051 >> v1:172.21.15.159:6789/0 conn(0x7f63840cf3f0 legacy=0x7f638415c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6382a00700 1 -- 172.21.15.12:0/3743976051 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638415fb00 con 0x7f638415f1e0 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.703+0000 7f6382000700 1 --2- 172.21.15.12:0/3743976051 >> v2:172.21.15.159:3300/0 conn(0x7f63840cef40 0x7f638415bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.704+0000 7f6382a00700 1 --2- 172.21.15.12:0/3743976051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638415f1e0 0x7f638415f5c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f63740098f0 tx=0x7f6374009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1a09519c4dc256 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.704+0000 7f6381600700 1 -- 172.21.15.12:0/3743976051 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637400b860 con 0x7f638415f1e0 2024-03-29T10:10:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.704+0000 7f6381600700 1 -- 172.21.15.12:0/3743976051 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637400b9c0 con 0x7f638415f1e0 2024-03-29T10:10:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.704+0000 7f6381600700 1 -- 172.21.15.12:0/3743976051 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637400bd50 con 0x7f638415f1e0 2024-03-29T10:10:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.705+0000 7f6389c00700 1 -- 172.21.15.12:0/3743976051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638415f1e0 msgr2=0x7f638415f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.705+0000 7f6389c00700 1 --2- 172.21.15.12:0/3743976051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638415f1e0 0x7f638415f5c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f63740098f0 tx=0x7f6374009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.705+0000 7f6389c00700 1 -- 172.21.15.12:0/3743976051 shutdown_connections 2024-03-29T10:10:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.705+0000 7f6389c00700 1 --2- 172.21.15.12:0/3743976051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638415f1e0 0x7f638415f5c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.705+0000 7f6389c00700 1 --2- 172.21.15.12:0/3743976051 >> v2:172.21.15.159:3300/0 conn(0x7f63840cef40 0x7f638415bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.705+0000 7f6389c00700 1 -- 172.21.15.12:0/3743976051 >> 172.21.15.12:0/3743976051 conn(0x7f63840d1370 msgr2=0x7f63840d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.705+0000 7f6389c00700 1 -- 172.21.15.12:0/3743976051 shutdown_connections 2024-03-29T10:10:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.705+0000 7f6389c00700 1 -- 172.21.15.12:0/3743976051 wait complete. 2024-03-29T10:10:00.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.707+0000 7f6389c00700 1 Processor -- start 2024-03-29T10:10:00.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.707+0000 7f6389c00700 1 -- start start 2024-03-29T10:10:00.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6389c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63840cef40 0x7f63840b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6389c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63840cf3f0 0x7f63840b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6389c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f638415f1e0 0x7f63840b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6389c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63840c8e00 con 0x7f63840cf3f0 2024-03-29T10:10:00.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6389c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638415cf40 con 0x7f638415f1e0 2024-03-29T10:10:00.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6389c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63840c9050 con 0x7f63840cef40 2024-03-29T10:10:00.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6382000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63840cf3f0 0x7f63840b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:00.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6382000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63840cf3f0 0x7f63840b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55440/0 (socket says 172.21.15.12:55440) 2024-03-29T10:10:00.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6382000700 1 -- 172.21.15.12:0/2757201216 learned_addr learned my addr 172.21.15.12:0/2757201216 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:00.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6383400700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f638415f1e0 0x7f63840b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6382a00700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63840cef40 0x7f63840b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6382000700 1 -- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63840cef40 msgr2=0x7f63840b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6382000700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63840cef40 0x7f63840b6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6382000700 1 -- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f638415f1e0 msgr2=0x7f63840b59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6382000700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f638415f1e0 0x7f63840b59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.708+0000 7f6382000700 1 -- 172.21.15.12:0/2757201216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6374009550 con 0x7f63840cf3f0 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.709+0000 7f6382000700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63840cf3f0 0x7f63840b5470 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f637000e810 tx=0x7f637000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.709+0000 7f637aa00700 1 -- 172.21.15.12:0/2757201216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637000c070 con 0x7f63840cf3f0 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.709+0000 7f637aa00700 1 -- 172.21.15.12:0/2757201216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6370004230 con 0x7f63840cf3f0 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.709+0000 7f637aa00700 1 -- 172.21.15.12:0/2757201216 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63700036a0 con 0x7f63840cf3f0 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.709+0000 7f6389c00700 1 -- 172.21.15.12:0/2757201216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63840c9330 con 0x7f63840cf3f0 2024-03-29T10:10:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.709+0000 7f6389c00700 1 -- 172.21.15.12:0/2757201216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638415cbd0 con 0x7f63840cf3f0 2024-03-29T10:10:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.710+0000 7f6389c00700 1 -- 172.21.15.12:0/2757201216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63840cbde0 con 0x7f63840cf3f0 2024-03-29T10:10:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.711+0000 7f637aa00700 1 -- 172.21.15.12:0/2757201216 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f6370003800 con 0x7f63840cf3f0 2024-03-29T10:10:00.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.712+0000 7f637aa00700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f636406adb0 0x7f636406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:00.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.712+0000 7f637aa00700 1 -- 172.21.15.12:0/2757201216 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6370089420 con 0x7f63840cf3f0 2024-03-29T10:10:00.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.712+0000 7f6382a00700 1 -- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f636406adb0 msgr2=0x7f636406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:00.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.712+0000 7f6382a00700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f636406adb0 0x7f636406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:00.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.716+0000 7f637aa00700 1 -- 172.21.15.12:0/2757201216 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f63700541e0 con 0x7f63840cf3f0 2024-03-29T10:10:00.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.913+0000 7f6382a00700 1 -- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f636406adb0 msgr2=0x7f636406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:00.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.913+0000 7f6382a00700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f636406adb0 0x7f636406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:00.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.932+0000 7f6389c00700 1 -- 172.21.15.12:0/2757201216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6384067720 con 0x7f63840cf3f0 2024-03-29T10:10:00.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.932+0000 7f637aa00700 1 -- 172.21.15.12:0/2757201216 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f63700094c0 con 0x7f63840cf3f0 2024-03-29T10:10:00.933 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:10:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 -- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f636406adb0 msgr2=0x7f636406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f636406adb0 0x7f636406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 -- 172.21.15.12:0/2757201216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63840cf3f0 msgr2=0x7f63840b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63840cf3f0 0x7f63840b5470 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f637000e810 tx=0x7f637000eb20 comp rx=0 tx=0).stop 2024-03-29T10:10:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 -- 172.21.15.12:0/2757201216 shutdown_connections 2024-03-29T10:10:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f636406adb0 0x7f636406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f638415f1e0 0x7f63840b59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63840cef40 0x7f63840b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 --2- 172.21.15.12:0/2757201216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63840cf3f0 0x7f63840b5470 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 -- 172.21.15.12:0/2757201216 >> 172.21.15.12:0/2757201216 conn(0x7f63840d1370 msgr2=0x7f63840b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 -- 172.21.15.12:0/2757201216 shutdown_connections 2024-03-29T10:10:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:00.935+0000 7f6389c00700 1 -- 172.21.15.12:0/2757201216 wait complete. 2024-03-29T10:10:00.954 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:10:01.134 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:01.134+0000 7f1caccb1240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:10:01.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.156+0000 7f09b5c00700 1 Processor -- start 2024-03-29T10:10:01.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.156+0000 7f09b5c00700 1 -- start start 2024-03-29T10:10:01.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.157+0000 7f09b5c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f09b015d9d0 0x7f09b00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.157+0000 7f09b5c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09b00d2540 0x7f09b00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.157+0000 7f09b5c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b0161420 con 0x7f09b00d16a0 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.157+0000 7f09b5c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b015edc0 con 0x7f09b00d2540 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.157+0000 7f09b5c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b0161700 con 0x7f09b015d9d0 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.158+0000 7f09af400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09b00d2540 0x7f09b00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.158+0000 7f09ae000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f09b00d16a0 0x7f09b00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36292/0 (socket says 172.21.15.12:36292) 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.158+0000 7f09ae000700 1 -- 172.21.15.12:0/2575777045 learned_addr learned my addr 172.21.15.12:0/2575777045 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.158+0000 7f09aea00700 1 --2- 172.21.15.12:0/2575777045 >> v2:172.21.15.159:3300/0 conn(0x7f09b015d9d0 0x7f09b00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.158+0000 7f09af400700 1 -- 172.21.15.12:0/2575777045 >> v2:172.21.15.159:3300/0 conn(0x7f09b015d9d0 msgr2=0x7f09b00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.158+0000 7f09af400700 1 --2- 172.21.15.12:0/2575777045 >> v2:172.21.15.159:3300/0 conn(0x7f09b015d9d0 0x7f09b00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.158+0000 7f09af400700 1 -- 172.21.15.12:0/2575777045 >> v1:172.21.15.12:6789/0 conn(0x7f09b00d16a0 legacy=0x7f09b00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.158+0000 7f09af400700 1 -- 172.21.15.12:0/2575777045 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b01619e0 con 0x7f09b00d2540 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.158+0000 7f09ad600700 1 -- 172.21.15.12:0/2575777045 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274412806 0 0) 0x7f09b0161420 con 0x7f09b00d16a0 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.159+0000 7f09af400700 1 --2- 172.21.15.12:0/2575777045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d2540 0x7f09b00ce360 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f09b00d0ff0 tx=0x7f09a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db51dd2ec9dc6e0b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.159+0000 7f09ad600700 1 -- 172.21.15.12:0/2575777045 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a800e070 con 0x7f09b00d2540 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.159+0000 7f09ad600700 1 -- 172.21.15.12:0/2575777045 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a800bb20 con 0x7f09b00d2540 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.159+0000 7f09ad600700 1 -- 172.21.15.12:0/2575777045 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8012430 con 0x7f09b00d2540 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.160+0000 7f09b5c00700 1 -- 172.21.15.12:0/2575777045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d2540 msgr2=0x7f09b00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.160+0000 7f09b5c00700 1 --2- 172.21.15.12:0/2575777045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d2540 0x7f09b00ce360 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f09b00d0ff0 tx=0x7f09a800b330 comp rx=0 tx=0).stop 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.160+0000 7f09b5c00700 1 -- 172.21.15.12:0/2575777045 shutdown_connections 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.160+0000 7f09b5c00700 1 --2- 172.21.15.12:0/2575777045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d2540 0x7f09b00ce360 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.160+0000 7f09b5c00700 1 --2- 172.21.15.12:0/2575777045 >> v2:172.21.15.159:3300/0 conn(0x7f09b015d9d0 0x7f09b00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.160+0000 7f09b5c00700 1 -- 172.21.15.12:0/2575777045 >> 172.21.15.12:0/2575777045 conn(0x7f09b008aff0 msgr2=0x7f09b00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:01.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.160+0000 7f09b5c00700 1 -- 172.21.15.12:0/2575777045 shutdown_connections 2024-03-29T10:10:01.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.160+0000 7f09b5c00700 1 -- 172.21.15.12:0/2575777045 wait complete. 2024-03-29T10:10:01.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.163+0000 7f09b5c00700 1 Processor -- start 2024-03-29T10:10:01.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.163+0000 7f09b5c00700 1 -- start start 2024-03-29T10:10:01.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.164+0000 7f09b5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d16a0 0x7f09b01f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09b5c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b00d2540 0x7f09b01f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09b5c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b015d9d0 0x7f09b00cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09b5c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b01f7580 con 0x7f09b00d16a0 2024-03-29T10:10:01.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09b5c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b01f2f50 con 0x7f09b00d2540 2024-03-29T10:10:01.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09b5c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00b6bf0 con 0x7f09b015d9d0 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09aea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d16a0 0x7f09b01f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09aea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d16a0 0x7f09b01f7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55452/0 (socket says 172.21.15.12:55452) 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09aea00700 1 -- 172.21.15.12:0/2077420519 learned_addr learned my addr 172.21.15.12:0/2077420519 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09ae000700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b00d2540 0x7f09b01f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09af400700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b015d9d0 0x7f09b00cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09aea00700 1 -- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b015d9d0 msgr2=0x7f09b00cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09aea00700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b015d9d0 0x7f09b00cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09aea00700 1 -- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b00d2540 msgr2=0x7f09b01f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09aea00700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b00d2540 0x7f09b01f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09aea00700 1 -- 172.21.15.12:0/2077420519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a8009a70 con 0x7f09b00d16a0 2024-03-29T10:10:01.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.165+0000 7f09ae000700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b00d2540 0x7f09b01f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:01.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.166+0000 7f09aea00700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d16a0 0x7f09b01f7040 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e870 tx=0x7f09a000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:01.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.166+0000 7f09a6a00700 1 -- 172.21.15.12:0/2077420519 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0009590 con 0x7f09b00d16a0 2024-03-29T10:10:01.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.166+0000 7f09a6a00700 1 -- 172.21.15.12:0/2077420519 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a0013680 con 0x7f09b00d16a0 2024-03-29T10:10:01.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.166+0000 7f09a6a00700 1 -- 172.21.15.12:0/2077420519 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a00139f0 con 0x7f09b00d16a0 2024-03-29T10:10:01.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.166+0000 7f09b5c00700 1 -- 172.21.15.12:0/2077420519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b00b6ed0 con 0x7f09b00d16a0 2024-03-29T10:10:01.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.166+0000 7f09b5c00700 1 -- 172.21.15.12:0/2077420519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b00b7490 con 0x7f09b00d16a0 2024-03-29T10:10:01.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.167+0000 7f09b5c00700 1 -- 172.21.15.12:0/2077420519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b00d1ca0 con 0x7f09b00d16a0 2024-03-29T10:10:01.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.168+0000 7f09a6a00700 1 -- 172.21.15.12:0/2077420519 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f09a0013b50 con 0x7f09b00d16a0 2024-03-29T10:10:01.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.169+0000 7f09a6a00700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099006adb0 0x7f099006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.169+0000 7f09a6a00700 1 -- 172.21.15.12:0/2077420519 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f09a00894f0 con 0x7f09b00d16a0 2024-03-29T10:10:01.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.169+0000 7f09ae000700 1 -- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099006adb0 msgr2=0x7f099006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:01.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.169+0000 7f09ae000700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099006adb0 0x7f099006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:01.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.173+0000 7f09a6a00700 1 -- 172.21.15.12:0/2077420519 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09a0054450 con 0x7f09b00d16a0 2024-03-29T10:10:01.349 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:01.349+0000 7f1caccb1240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:10:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.370+0000 7f09ae000700 1 -- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099006adb0 msgr2=0x7f099006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.370+0000 7f09ae000700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099006adb0 0x7f099006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:01.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.385+0000 7f09b5c00700 1 -- 172.21.15.12:0/2077420519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09b0067720 con 0x7f09b00d16a0 2024-03-29T10:10:01.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.386+0000 7f09a6a00700 1 -- 172.21.15.12:0/2077420519 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09a005ce60 con 0x7f09b00d16a0 2024-03-29T10:10:01.386 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:10:01.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.388+0000 7f09b5c00700 1 -- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099006adb0 msgr2=0x7f099006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.388+0000 7f09b5c00700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099006adb0 0x7f099006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.388+0000 7f09b5c00700 1 -- 172.21.15.12:0/2077420519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d16a0 msgr2=0x7f09b01f7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.388+0000 7f09b5c00700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d16a0 0x7f09b01f7040 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e870 tx=0x7f09a000eb80 comp rx=0 tx=0).stop 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.388+0000 7f09b5c00700 1 -- 172.21.15.12:0/2077420519 shutdown_connections 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.388+0000 7f09b5c00700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099006adb0 0x7f099006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.388+0000 7f09b5c00700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b015d9d0 0x7f09b00cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.388+0000 7f09b5c00700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b00d16a0 0x7f09b01f7040 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.388+0000 7f09b5c00700 1 --2- 172.21.15.12:0/2077420519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b00d2540 0x7f09b01f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.388+0000 7f09b5c00700 1 -- 172.21.15.12:0/2077420519 >> 172.21.15.12:0/2077420519 conn(0x7f09b008aff0 msgr2=0x7f09b015f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.389+0000 7f09b5c00700 1 -- 172.21.15.12:0/2077420519 shutdown_connections 2024-03-29T10:10:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.389+0000 7f09b5c00700 1 -- 172.21.15.12:0/2077420519 wait complete. 2024-03-29T10:10:01.408 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:01.472 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:01.471+0000 7fd1471b9240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:10:01.554 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:01.554+0000 7f1caccb1240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.609+0000 7ff78c800700 1 Processor -- start 2024-03-29T10:10:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.610+0000 7ff78c800700 1 -- start start 2024-03-29T10:10:01.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.611+0000 7ff78c800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7840cb3f0 0x7ff7840ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.611+0000 7ff78c800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78415fd50 con 0x7ff7840ac600 2024-03-29T10:10:01.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.611+0000 7ff78c800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7840aca10 con 0x7ff78415b5d0 2024-03-29T10:10:01.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.611+0000 7ff78c800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff78415b9b0 con 0x7ff7840cb3f0 2024-03-29T10:10:01.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.611+0000 7ff789400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7840cb3f0 0x7ff7840ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:01.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.611+0000 7ff78a800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff7840ac600 0x7ff78406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36308/0 (socket says 172.21.15.12:36308) 2024-03-29T10:10:01.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.611+0000 7ff78a800700 1 -- 172.21.15.12:0/1641977645 learned_addr learned my addr 172.21.15.12:0/1641977645 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:01.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.612+0000 7ff789400700 1 -- 172.21.15.12:0/1641977645 >> v1:172.21.15.159:6789/0 conn(0x7ff78415b5d0 legacy=0x7ff7840ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.612+0000 7ff789400700 1 -- 172.21.15.12:0/1641977645 >> v1:172.21.15.12:6789/0 conn(0x7ff7840ac600 legacy=0x7ff78406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.612+0000 7ff789400700 1 -- 172.21.15.12:0/1641977645 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78415f140 con 0x7ff7840cb3f0 2024-03-29T10:10:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.612+0000 7ff789400700 1 --2- 172.21.15.12:0/1641977645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840cb3f0 0x7ff7840ac0c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff774009840 tx=0x7ff774009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b54d6603055171 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.612+0000 7ff788a00700 1 -- 172.21.15.12:0/1641977645 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774011830 con 0x7ff7840cb3f0 2024-03-29T10:10:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.612+0000 7ff788a00700 1 -- 172.21.15.12:0/1641977645 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff774011990 con 0x7ff7840cb3f0 2024-03-29T10:10:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.612+0000 7ff788a00700 1 -- 172.21.15.12:0/1641977645 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774011d20 con 0x7ff7840cb3f0 2024-03-29T10:10:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.613+0000 7ff78c800700 1 -- 172.21.15.12:0/1641977645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840cb3f0 msgr2=0x7ff7840ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.613+0000 7ff78c800700 1 --2- 172.21.15.12:0/1641977645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840cb3f0 0x7ff7840ac0c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff774009840 tx=0x7ff774009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.613+0000 7ff78c800700 1 -- 172.21.15.12:0/1641977645 shutdown_connections 2024-03-29T10:10:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.613+0000 7ff78c800700 1 --2- 172.21.15.12:0/1641977645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840cb3f0 0x7ff7840ac0c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.613+0000 7ff78c800700 1 -- 172.21.15.12:0/1641977645 >> 172.21.15.12:0/1641977645 conn(0x7ff7840cf190 msgr2=0x7ff7840cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.613+0000 7ff78c800700 1 -- 172.21.15.12:0/1641977645 shutdown_connections 2024-03-29T10:10:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.613+0000 7ff78c800700 1 -- 172.21.15.12:0/1641977645 wait complete. 2024-03-29T10:10:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.616+0000 7ff78c800700 1 Processor -- start 2024-03-29T10:10:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.616+0000 7ff78c800700 1 -- start start 2024-03-29T10:10:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff78c800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840ac600 0x7ff784151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff78c800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff78415b5d0 0x7ff784098a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff78c800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff784144230 0x7ff784152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff78c800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff784098f80 con 0x7ff7840ac600 2024-03-29T10:10:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff78c800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff784144ab0 con 0x7ff78415b5d0 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff78c800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff784153470 con 0x7ff784144230 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff789e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840ac600 0x7ff784151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff789e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840ac600 0x7ff784151790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55456/0 (socket says 172.21.15.12:55456) 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff789e00700 1 -- 172.21.15.12:0/4243548482 learned_addr learned my addr 172.21.15.12:0/4243548482 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff78a800700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff784144230 0x7ff784152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff789400700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff78415b5d0 0x7ff784098a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff789e00700 1 -- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff784144230 msgr2=0x7ff784152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff789e00700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff784144230 0x7ff784152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff789e00700 1 -- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff78415b5d0 msgr2=0x7ff784098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.618+0000 7ff789e00700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff78415b5d0 0x7ff784098a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.619+0000 7ff789e00700 1 -- 172.21.15.12:0/4243548482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7740094a0 con 0x7ff7840ac600 2024-03-29T10:10:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.619+0000 7ff789e00700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840ac600 0x7ff784151790 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff77800c730 tx=0x7ff77800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.619+0000 7ff77e000700 1 -- 172.21.15.12:0/4243548482 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7780052d0 con 0x7ff7840ac600 2024-03-29T10:10:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.619+0000 7ff77e000700 1 -- 172.21.15.12:0/4243548482 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff778005430 con 0x7ff7840ac600 2024-03-29T10:10:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.619+0000 7ff77e000700 1 -- 172.21.15.12:0/4243548482 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7780038a0 con 0x7ff7840ac600 2024-03-29T10:10:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.619+0000 7ff78c800700 1 -- 172.21.15.12:0/4243548482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff784151e90 con 0x7ff7840ac600 2024-03-29T10:10:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.619+0000 7ff78c800700 1 -- 172.21.15.12:0/4243548482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff784152450 con 0x7ff7840ac600 2024-03-29T10:10:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.620+0000 7ff78c800700 1 -- 172.21.15.12:0/4243548482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff784067720 con 0x7ff7840ac600 2024-03-29T10:10:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.622+0000 7ff77e000700 1 -- 172.21.15.12:0/4243548482 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7ff778003a00 con 0x7ff7840ac600 2024-03-29T10:10:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.623+0000 7ff77e000700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff76806adb0 0x7ff76806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.623+0000 7ff77e000700 1 -- 172.21.15.12:0/4243548482 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff778088a60 con 0x7ff7840ac600 2024-03-29T10:10:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.623+0000 7ff789400700 1 -- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff76806adb0 msgr2=0x7ff76806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.623+0000 7ff789400700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff76806adb0 0x7ff76806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.627+0000 7ff77e000700 1 -- 172.21.15.12:0/4243548482 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff778053930 con 0x7ff7840ac600 2024-03-29T10:10:01.717 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:01.716+0000 7fd1471b9240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:10:01.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.824+0000 7ff789400700 1 -- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff76806adb0 msgr2=0x7ff76806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:01.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.824+0000 7ff789400700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff76806adb0 0x7ff76806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.909+0000 7ff78c800700 1 -- 172.21.15.12:0/4243548482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff78406ea60 con 0x7ff7840ac600 2024-03-29T10:10:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.910+0000 7ff77e000700 1 -- 172.21.15.12:0/4243548482 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4740 (secure 0 0 0) 0x7ff778017090 con 0x7ff7840ac600 2024-03-29T10:10:01.911 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:01.911 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3202930550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3202930550},{"type":"v1","addr":"172.21.15.159:6835","nonce":3202930550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4189378403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4189378403},{"type":"v1","addr":"172.21.15.12:6834","nonce":4189378403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2681299760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2681299760},{"type":"v1","addr":"172.21.15.12:6835","nonce":2681299760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:09:32.597152+0000","modified":"2024-03-29T10:09:35.607262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6241":{"gid":6241,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/4289275091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":4289275091},{"type":"v1","addr":"172.21.15.159:6837","nonce":4289275091}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-29T10:10:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.912+0000 7ff78c800700 1 -- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff76806adb0 msgr2=0x7ff76806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.912+0000 7ff78c800700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff76806adb0 0x7ff76806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.912+0000 7ff78c800700 1 -- 172.21.15.12:0/4243548482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840ac600 msgr2=0x7ff784151790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.912+0000 7ff78c800700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840ac600 0x7ff784151790 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff77800c730 tx=0x7ff77800ca40 comp rx=0 tx=0).stop 2024-03-29T10:10:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.912+0000 7ff78c800700 1 -- 172.21.15.12:0/4243548482 shutdown_connections 2024-03-29T10:10:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.912+0000 7ff78c800700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff76806adb0 0x7ff76806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.912+0000 7ff78c800700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff784144230 0x7ff784152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.912+0000 7ff78c800700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7840ac600 0x7ff784151790 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.912+0000 7ff78c800700 1 --2- 172.21.15.12:0/4243548482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff78415b5d0 0x7ff784098a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.912+0000 7ff78c800700 1 -- 172.21.15.12:0/4243548482 >> 172.21.15.12:0/4243548482 conn(0x7ff7840cf190 msgr2=0x7ff78415e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.913+0000 7ff78c800700 1 -- 172.21.15.12:0/4243548482 shutdown_connections 2024-03-29T10:10:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:01.913+0000 7ff78c800700 1 -- 172.21.15.12:0/4243548482 wait complete. 2024-03-29T10:10:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 31 2024-03-29T10:10:01.933 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:01.941 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:01.941+0000 7fd1471b9240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:02.030 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.030+0000 7f1caccb1240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:10:02.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.133+0000 7f0763c00700 1 Processor -- start 2024-03-29T10:10:02.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.134+0000 7f0763c00700 1 -- start start 2024-03-29T10:10:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.135+0000 7f0763c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c0d18b0 0x7f075c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.135+0000 7f0763c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075c0ceda0 con 0x7f075c160f10 2024-03-29T10:10:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.135+0000 7f0763c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075c1612f0 con 0x7f075c161590 2024-03-29T10:10:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.135+0000 7f0763c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0d1d30 con 0x7f075c0d18b0 2024-03-29T10:10:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.135+0000 7f0761200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f075c160f10 0x7f075c0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36324/0 (socket says 172.21.15.12:36324) 2024-03-29T10:10:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.135+0000 7f0761200700 1 -- 172.21.15.12:0/3276013354 learned_addr learned my addr 172.21.15.12:0/3276013354 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.135+0000 7f075be00700 1 --2- 172.21.15.12:0/3276013354 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c0d18b0 0x7f075c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:02.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075b400700 1 -- 172.21.15.12:0/3276013354 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352861340 0 0) 0x7f075c0ceda0 con 0x7f075c160f10 2024-03-29T10:10:02.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075b400700 1 -- 172.21.15.12:0/3276013354 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07440035b0 con 0x7f075c160f10 2024-03-29T10:10:02.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075b400700 1 -- 172.21.15.12:0/3276013354 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925012179 0 0) 0x7f075c1612f0 con 0x7f075c161590 2024-03-29T10:10:02.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075b400700 1 -- 172.21.15.12:0/3276013354 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f075c0ceda0 con 0x7f075c161590 2024-03-29T10:10:02.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075be00700 1 -- 172.21.15.12:0/3276013354 >> v1:172.21.15.159:6789/0 conn(0x7f075c161590 legacy=0x7f075c0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075be00700 1 -- 172.21.15.12:0/3276013354 >> v1:172.21.15.12:6789/0 conn(0x7f075c160f10 legacy=0x7f075c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075be00700 1 -- 172.21.15.12:0/3276013354 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c0ce470 con 0x7f075c0d18b0 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075be00700 1 --2- 172.21.15.12:0/3276013354 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c0d18b0 0x7f075c0cf230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f07480098a0 tx=0x7f0748009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=942437c2db034ade server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075b400700 1 -- 172.21.15.12:0/3276013354 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0748011830 con 0x7f075c0d18b0 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075b400700 1 -- 172.21.15.12:0/3276013354 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0748011990 con 0x7f075c0d18b0 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.136+0000 7f075b400700 1 -- 172.21.15.12:0/3276013354 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0748011d20 con 0x7f075c0d18b0 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.137+0000 7f0763c00700 1 -- 172.21.15.12:0/3276013354 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c0d18b0 msgr2=0x7f075c0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.137+0000 7f0763c00700 1 --2- 172.21.15.12:0/3276013354 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c0d18b0 0x7f075c0cf230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f07480098a0 tx=0x7f0748009bb0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.137+0000 7f0763c00700 1 -- 172.21.15.12:0/3276013354 shutdown_connections 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.137+0000 7f0763c00700 1 --2- 172.21.15.12:0/3276013354 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c0d18b0 0x7f075c0cf230 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.137+0000 7f0763c00700 1 -- 172.21.15.12:0/3276013354 >> 172.21.15.12:0/3276013354 conn(0x7f075c08aff0 msgr2=0x7f075c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.137+0000 7f0763c00700 1 -- 172.21.15.12:0/3276013354 shutdown_connections 2024-03-29T10:10:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.137+0000 7f0763c00700 1 -- 172.21.15.12:0/3276013354 wait complete. 2024-03-29T10:10:02.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.139+0000 7f0763c00700 1 Processor -- start 2024-03-29T10:10:02.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.139+0000 7f0763c00700 1 -- start start 2024-03-29T10:10:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f0763c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075c0d18b0 0x7f075c1f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f0763c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c160f10 0x7f075c1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f0763c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f075c161590 0x7f075c0b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f0763c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c1f3360 con 0x7f075c161590 2024-03-29T10:10:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f0763c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c156df0 con 0x7f075c0d18b0 2024-03-29T10:10:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f0763c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c159540 con 0x7f075c160f10 2024-03-29T10:10:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f0761200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075c0d18b0 0x7f075c1f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f075be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c160f10 0x7f075c1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f0761200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075c0d18b0 0x7f075c1f2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:33350/0 (socket says 172.21.15.12:33350) 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f0761200700 1 -- 172.21.15.12:0/3382639349 learned_addr learned my addr 172.21.15.12:0/3382639349 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.140+0000 7f0761c00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f075c161590 0x7f075c0b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f0761200700 1 -- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c160f10 msgr2=0x7f075c1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f0761200700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c160f10 0x7f075c1568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f0761200700 1 -- 172.21.15.12:0/3382639349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f075c161590 msgr2=0x7f075c0b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f0761200700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f075c161590 0x7f075c0b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f0761200700 1 -- 172.21.15.12:0/3382639349 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07480094a0 con 0x7f075c0d18b0 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f075be00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c160f10 0x7f075c1568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f0761200700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075c0d18b0 0x7f075c1f2e20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0750001eb0 tx=0x7f075000c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f0758c00700 1 -- 172.21.15.12:0/3382639349 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750005d70 con 0x7f075c0d18b0 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f0758c00700 1 -- 172.21.15.12:0/3382639349 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07500070d0 con 0x7f075c0d18b0 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f0758c00700 1 -- 172.21.15.12:0/3382639349 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750007440 con 0x7f075c0d18b0 2024-03-29T10:10:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.141+0000 7f0763c00700 1 -- 172.21.15.12:0/3382639349 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c159820 con 0x7f075c0d18b0 2024-03-29T10:10:02.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.142+0000 7f0763c00700 1 -- 172.21.15.12:0/3382639349 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c153cc0 con 0x7f075c0d18b0 2024-03-29T10:10:02.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.142+0000 7f0763c00700 1 -- 172.21.15.12:0/3382639349 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075c067720 con 0x7f075c0d18b0 2024-03-29T10:10:02.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.143+0000 7f0758c00700 1 -- 172.21.15.12:0/3382639349 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f07500075a0 con 0x7f075c0d18b0 2024-03-29T10:10:02.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.144+0000 7f0758c00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f073c06ade0 0x7f073c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.144+0000 7f0758c00700 1 -- 172.21.15.12:0/3382639349 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f07500888e0 con 0x7f075c0d18b0 2024-03-29T10:10:02.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.144+0000 7f075be00700 1 -- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f073c06ade0 msgr2=0x7f073c06d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:02.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.144+0000 7f075be00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f073c06ade0 0x7f073c06d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.149+0000 7f0758c00700 1 -- 172.21.15.12:0/3382639349 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0750053860 con 0x7f075c0d18b0 2024-03-29T10:10:02.239 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.239+0000 7f1caccb1240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.345+0000 7f075be00700 1 -- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f073c06ade0 msgr2=0x7f073c06d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.345+0000 7f075be00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f073c06ade0 0x7f073c06d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:02.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.365+0000 7f0763c00700 1 -- 172.21.15.12:0/3382639349 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f075c08b760 con 0x7f075c0d18b0 2024-03-29T10:10:02.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.366+0000 7f0758c00700 1 -- 172.21.15.12:0/3382639349 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16595 (secure 0 0 0) 0x7f0750017090 con 0x7f075c0d18b0 2024-03-29T10:10:02.367 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:02.368 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:09:53.746428+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:09:30.441619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-29T10:09:31.581366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3367842275":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/3723093447":"2024-03-30T10:09:53.746392+0000","172.21.15.159:6833/43444":"2024-03-30T10:09:53.746392+0000","172.21.15.159:6832/43444":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/1578988741":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/1180347815":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/667128976":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/3208721490":"2024-03-30T10:09:53.746392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:02.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 -- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f073c06ade0 msgr2=0x7f073c06d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:02.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f073c06ade0 0x7f073c06d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 -- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075c0d18b0 msgr2=0x7f075c1f2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075c0d18b0 0x7f075c1f2e20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0750001eb0 tx=0x7f075000c7b0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 -- 172.21.15.12:0/3382639349 shutdown_connections 2024-03-29T10:10:02.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f073c06ade0 0x7f073c06d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f075c161590 0x7f075c0b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f075c0d18b0 0x7f075c1f2e20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 --2- 172.21.15.12:0/3382639349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f075c160f10 0x7f075c1568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 -- 172.21.15.12:0/3382639349 >> 172.21.15.12:0/3382639349 conn(0x7f075c08aff0 msgr2=0x7f075c06ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 -- 172.21.15.12:0/3382639349 shutdown_connections 2024-03-29T10:10:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.369+0000 7f0763c00700 1 -- 172.21.15.12:0/3382639349 wait complete. 2024-03-29T10:10:02.388 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T10:10:02.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:02.456 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:02.455+0000 7fd1471b9240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:10:02.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.627+0000 7f40d0800700 1 Processor -- start 2024-03-29T10:10:02.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.627+0000 7f40d0800700 1 -- start start 2024-03-29T10:10:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40d0800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f40c80d05e0 0x7f40c80d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40d0800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c80b6990 0x7f40c80b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40d0800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c80d0f80 con 0x7f40c815da50 2024-03-29T10:10:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40d0800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c80b95d0 con 0x7f40c80d05e0 2024-03-29T10:10:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40d0800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c815de30 con 0x7f40c80b6990 2024-03-29T10:10:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40cde00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f40c815da50 0x7f40c80ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36332/0 (socket says 172.21.15.12:36332) 2024-03-29T10:10:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40cde00700 1 -- 172.21.15.12:0/1769284201 learned_addr learned my addr 172.21.15.12:0/1769284201 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40ce800700 1 --2- 172.21.15.12:0/1769284201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c80b6990 0x7f40c80b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40cd400700 1 --2- 172.21.15.12:0/1769284201 >> v2:172.21.15.159:3300/0 conn(0x7f40c80d05e0 0x7f40c80d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40cca00700 1 -- 172.21.15.12:0/1769284201 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528113468 0 0) 0x7f40c80d0f80 con 0x7f40c815da50 2024-03-29T10:10:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.629+0000 7f40cca00700 1 -- 172.21.15.12:0/1769284201 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b00035b0 con 0x7f40c815da50 2024-03-29T10:10:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.630+0000 7f40ce800700 1 -- 172.21.15.12:0/1769284201 >> v2:172.21.15.159:3300/0 conn(0x7f40c80d05e0 msgr2=0x7f40c80d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.630+0000 7f40ce800700 1 --2- 172.21.15.12:0/1769284201 >> v2:172.21.15.159:3300/0 conn(0x7f40c80d05e0 0x7f40c80d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.630+0000 7f40ce800700 1 -- 172.21.15.12:0/1769284201 >> v1:172.21.15.12:6789/0 conn(0x7f40c815da50 legacy=0x7f40c80ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.630+0000 7f40ce800700 1 -- 172.21.15.12:0/1769284201 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c80d24e0 con 0x7f40c80b6990 2024-03-29T10:10:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.630+0000 7f40ce800700 1 --2- 172.21.15.12:0/1769284201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c80b6990 0x7f40c80b6e40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009d90 tx=0x7f40c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e3775f3bd66313 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.630+0000 7f40cca00700 1 -- 172.21.15.12:0/1769284201 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c400e070 con 0x7f40c80b6990 2024-03-29T10:10:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.630+0000 7f40cca00700 1 -- 172.21.15.12:0/1769284201 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c400bb20 con 0x7f40c80b6990 2024-03-29T10:10:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.630+0000 7f40cca00700 1 -- 172.21.15.12:0/1769284201 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c4012430 con 0x7f40c80b6990 2024-03-29T10:10:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.631+0000 7f40d0800700 1 -- 172.21.15.12:0/1769284201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c80b6990 msgr2=0x7f40c80b6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.631+0000 7f40d0800700 1 --2- 172.21.15.12:0/1769284201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c80b6990 0x7f40c80b6e40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009d90 tx=0x7f40c400b330 comp rx=0 tx=0).stop 2024-03-29T10:10:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.631+0000 7f40d0800700 1 -- 172.21.15.12:0/1769284201 shutdown_connections 2024-03-29T10:10:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.631+0000 7f40d0800700 1 --2- 172.21.15.12:0/1769284201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c80b6990 0x7f40c80b6e40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.631+0000 7f40d0800700 1 --2- 172.21.15.12:0/1769284201 >> v2:172.21.15.159:3300/0 conn(0x7f40c80d05e0 0x7f40c80d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.631+0000 7f40d0800700 1 -- 172.21.15.12:0/1769284201 >> 172.21.15.12:0/1769284201 conn(0x7f40c80666a0 msgr2=0x7f40c8066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.631+0000 7f40d0800700 1 -- 172.21.15.12:0/1769284201 shutdown_connections 2024-03-29T10:10:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.631+0000 7f40d0800700 1 -- 172.21.15.12:0/1769284201 wait complete. 2024-03-29T10:10:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.633+0000 7f40d0800700 1 Processor -- start 2024-03-29T10:10:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.633+0000 7f40d0800700 1 -- start start 2024-03-29T10:10:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40d0800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c80b6990 0x7f40c80b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40d0800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40c80d05e0 0x7f40c80ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40d0800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c815da50 0x7f40c814d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40d0800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80cd450 con 0x7f40c80d05e0 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40d0800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80b6040 con 0x7f40c80b6990 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40d0800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80cb340 con 0x7f40c815da50 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40cde00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c80b6990 0x7f40c80b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40cd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40c80d05e0 0x7f40c80ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40cde00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c80b6990 0x7f40c80b59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:33356/0 (socket says 172.21.15.12:33356) 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40cd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40c80d05e0 0x7f40c80ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55466/0 (socket says 172.21.15.12:55466) 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40cde00700 1 -- 172.21.15.12:0/2518380745 learned_addr learned my addr 172.21.15.12:0/2518380745 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.634+0000 7f40ce800700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c815da50 0x7f40c814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40cde00700 1 -- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c815da50 msgr2=0x7f40c814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40cde00700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c815da50 0x7f40c814d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40cde00700 1 -- 172.21.15.12:0/2518380745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40c80d05e0 msgr2=0x7f40c80ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40cde00700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40c80d05e0 0x7f40c80ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40cde00700 1 -- 172.21.15.12:0/2518380745 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c4009a70 con 0x7f40c80b6990 2024-03-29T10:10:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40cd400700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40c80d05e0 0x7f40c80ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40cde00700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c80b6990 0x7f40c80b59b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f40bc001eb0 tx=0x7f40bc00c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40c2000700 1 -- 172.21.15.12:0/2518380745 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc005d70 con 0x7f40c80b6990 2024-03-29T10:10:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40c2000700 1 -- 172.21.15.12:0/2518380745 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40bc0070d0 con 0x7f40c80b6990 2024-03-29T10:10:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40d0800700 1 -- 172.21.15.12:0/2518380745 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c80cb620 con 0x7f40c80b6990 2024-03-29T10:10:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.635+0000 7f40c2000700 1 -- 172.21.15.12:0/2518380745 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc007460 con 0x7f40c80b6990 2024-03-29T10:10:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.636+0000 7f40d0800700 1 -- 172.21.15.12:0/2518380745 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c80cbbe0 con 0x7f40c80b6990 2024-03-29T10:10:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.637+0000 7f40d0800700 1 -- 172.21.15.12:0/2518380745 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c8067720 con 0x7f40c80b6990 2024-03-29T10:10:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.637+0000 7f40c2000700 1 -- 172.21.15.12:0/2518380745 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 34) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f40bc0075c0 con 0x7f40c80b6990 2024-03-29T10:10:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.638+0000 7f40c2000700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f40ac06adb0 0x7f40ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.638+0000 7f40c2000700 1 -- 172.21.15.12:0/2518380745 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f40bc088a60 con 0x7f40c80b6990 2024-03-29T10:10:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.638+0000 7f40cd400700 1 -- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f40ac06adb0 msgr2=0x7f40ac06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.639+0000 7f40cd400700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f40ac06adb0 0x7f40ac06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.643+0000 7f40c2000700 1 -- 172.21.15.12:0/2518380745 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40bc053930 con 0x7f40c80b6990 2024-03-29T10:10:02.680 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.680+0000 7f1caccb1240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:10:02.681 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:02.680+0000 7fd1471b9240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:02.790 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.790+0000 7f1caccb1240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.804+0000 7f40c2000700 1 -- 172.21.15.12:0/2518380745 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mgrmap(e 35) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f40bc053d00 con 0x7f40c80b6990 2024-03-29T10:10:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.805+0000 7f40c2000700 1 -- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f40ac06adb0 msgr2=0x7f40ac06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.805+0000 7f40c2000700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f40ac06adb0 0x7f40ac06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.873 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.872+0000 7f1c1fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:02.873 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.872+0000 7f1c1fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:02.873 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.872+0000 7f1c1fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:02.873 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.872+0000 7f1c1fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:02.873 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.872+0000 7f1c1fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:02.873 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.874+0000 7f1c1c200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:02.873 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.874+0000 7f1c1c200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:02.874 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.874+0000 7f1c1c200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:02.874 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.874+0000 7f1c1c200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:02.874 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:02.874+0000 7f1c1c200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:02.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.914+0000 7f40d0800700 1 -- 172.21.15.12:0/2518380745 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40c81610f0 con 0x7f40c80b6990 2024-03-29T10:10:02.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.915+0000 7f40c2000700 1 -- 172.21.15.12:0/2518380745 <== mon.1 v2:172.21.15.159:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f40bc073110 con 0x7f40c80b6990 2024-03-29T10:10:02.915 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:02.915 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3202930550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3202930550},{"type":"v1","addr":"172.21.15.159:6835","nonce":3202930550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4189378403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4189378403},{"type":"v1","addr":"172.21.15.12:6834","nonce":4189378403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2681299760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2681299760},{"type":"v1","addr":"172.21.15.12:6835","nonce":2681299760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:09:32.597152+0000","modified":"2024-03-29T10:09:35.607262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6241":{"gid":6241,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/4289275091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":4289275091},{"type":"v1","addr":"172.21.15.159:6837","nonce":4289275091}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-29T10:10:02.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.917+0000 7f40d0800700 1 -- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c80b6990 msgr2=0x7f40c80b59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:02.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.917+0000 7f40d0800700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c80b6990 0x7f40c80b59b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f40bc001eb0 tx=0x7f40bc00c7b0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.917+0000 7f40d0800700 1 -- 172.21.15.12:0/2518380745 shutdown_connections 2024-03-29T10:10:02.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.917+0000 7f40d0800700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f40ac06adb0 0x7f40ac06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.917+0000 7f40d0800700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c815da50 0x7f40c814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.917+0000 7f40d0800700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c80b6990 0x7f40c80b59b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.917+0000 7f40d0800700 1 --2- 172.21.15.12:0/2518380745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40c80d05e0 0x7f40c80ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:02.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.917+0000 7f40d0800700 1 -- 172.21.15.12:0/2518380745 >> 172.21.15.12:0/2518380745 conn(0x7f40c80666a0 msgr2=0x7f40c80d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:02.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.917+0000 7f40d0800700 1 -- 172.21.15.12:0/2518380745 shutdown_connections 2024-03-29T10:10:02.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:02.917+0000 7f40d0800700 1 -- 172.21.15.12:0/2518380745 wait complete. 2024-03-29T10:10:02.919 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 31 2024-03-29T10:10:02.939 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:03.151 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:03.150+0000 7fd1471b9240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:10:03.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.182+0000 7f03fb800700 1 Processor -- start 2024-03-29T10:10:03.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.182+0000 7f03fb800700 1 -- start start 2024-03-29T10:10:03.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.183+0000 7f03fb800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f03f4160c90 0x7f03f40b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.183+0000 7f03fb800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03f40718e0 0x7f03f40b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.183+0000 7f03fb800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f40ac4d0 con 0x7f03f40709c0 2024-03-29T10:10:03.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.183+0000 7f03fb800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f4071e50 con 0x7f03f4160c90 2024-03-29T10:10:03.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.183+0000 7f03fb800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f4072130 con 0x7f03f40718e0 2024-03-29T10:10:03.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.183+0000 7f03f8e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f03f4160c90 0x7f03f40b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.184+0000 7f03f9800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f03f40709c0 0x7f03f406cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36348/0 (socket says 172.21.15.12:36348) 2024-03-29T10:10:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.184+0000 7f03f9800700 1 -- 172.21.15.12:0/2579556526 learned_addr learned my addr 172.21.15.12:0/2579556526 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.184+0000 7f03f3e00700 1 --2- 172.21.15.12:0/2579556526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03f40718e0 0x7f03f40b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.184+0000 7f03f8e00700 1 -- 172.21.15.12:0/2579556526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03f40718e0 msgr2=0x7f03f40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.184+0000 7f03f8e00700 1 --2- 172.21.15.12:0/2579556526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03f40718e0 0x7f03f40b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.184+0000 7f03f8e00700 1 -- 172.21.15.12:0/2579556526 >> v1:172.21.15.12:6789/0 conn(0x7f03f40709c0 legacy=0x7f03f406cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.184+0000 7f03f3400700 1 -- 172.21.15.12:0/2579556526 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001595002 0 0) 0x7f03f40ac4d0 con 0x7f03f40709c0 2024-03-29T10:10:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.184+0000 7f03f8e00700 1 -- 172.21.15.12:0/2579556526 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f4072410 con 0x7f03f4160c90 2024-03-29T10:10:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.184+0000 7f03f8e00700 1 --2- 172.21.15.12:0/2579556526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f4160c90 0x7f03f40b20e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009840 tx=0x7f03e8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8e634391f59330 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03f3400700 1 -- 172.21.15.12:0/2579556526 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8011830 con 0x7f03f4160c90 2024-03-29T10:10:03.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03f3400700 1 -- 172.21.15.12:0/2579556526 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e8011990 con 0x7f03f4160c90 2024-03-29T10:10:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03f3400700 1 -- 172.21.15.12:0/2579556526 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8011d20 con 0x7f03f4160c90 2024-03-29T10:10:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03fb800700 1 -- 172.21.15.12:0/2579556526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f4160c90 msgr2=0x7f03f40b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03fb800700 1 --2- 172.21.15.12:0/2579556526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f4160c90 0x7f03f40b20e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009840 tx=0x7f03e8009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03fb800700 1 -- 172.21.15.12:0/2579556526 shutdown_connections 2024-03-29T10:10:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03fb800700 1 --2- 172.21.15.12:0/2579556526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f4160c90 0x7f03f40b20e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03fb800700 1 --2- 172.21.15.12:0/2579556526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03f40718e0 0x7f03f40b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03fb800700 1 -- 172.21.15.12:0/2579556526 >> 172.21.15.12:0/2579556526 conn(0x7f03f40cf190 msgr2=0x7f03f40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03fb800700 1 -- 172.21.15.12:0/2579556526 shutdown_connections 2024-03-29T10:10:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.185+0000 7f03fb800700 1 -- 172.21.15.12:0/2579556526 wait complete. 2024-03-29T10:10:03.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.188+0000 7f03fb800700 1 Processor -- start 2024-03-29T10:10:03.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.188+0000 7f03fb800700 1 -- start start 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.189+0000 7f03fb800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f40709c0 0x7f03f40b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.189+0000 7f03fb800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03f40718e0 0x7f03f40cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.189+0000 7f03fb800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03f4160c90 0x7f03f40c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.189+0000 7f03fb800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40b0830 con 0x7f03f40709c0 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.189+0000 7f03fb800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40ccd10 con 0x7f03f4160c90 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.189+0000 7f03fb800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40ccf60 con 0x7f03f40718e0 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f40709c0 0x7f03f40b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f40709c0 0x7f03f40b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55482/0 (socket says 172.21.15.12:55482) 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f8e00700 1 -- 172.21.15.12:0/2990322422 learned_addr learned my addr 172.21.15.12:0/2990322422 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f3e00700 1 --2- 172.21.15.12:0/2990322422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03f40718e0 0x7f03f40cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f9800700 1 --2- 172.21.15.12:0/2990322422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03f4160c90 0x7f03f40c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f8e00700 1 -- 172.21.15.12:0/2990322422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03f40718e0 msgr2=0x7f03f40cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f8e00700 1 --2- 172.21.15.12:0/2990322422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03f40718e0 0x7f03f40cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f8e00700 1 -- 172.21.15.12:0/2990322422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03f4160c90 msgr2=0x7f03f40c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f8e00700 1 --2- 172.21.15.12:0/2990322422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03f4160c90 0x7f03f40c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f8e00700 1 -- 172.21.15.12:0/2990322422 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e80094a0 con 0x7f03f40709c0 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.190+0000 7f03f8e00700 1 --2- 172.21.15.12:0/2990322422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f40709c0 0x7f03f40b02f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009b90 tx=0x7f03e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.191+0000 7f03f0c00700 1 -- 172.21.15.12:0/2990322422 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e80036a0 con 0x7f03f40709c0 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.191+0000 7f03f0c00700 1 -- 172.21.15.12:0/2990322422 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e801e070 con 0x7f03f40709c0 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.191+0000 7f03fb800700 1 -- 172.21.15.12:0/2990322422 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f40cd240 con 0x7f03f40709c0 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.191+0000 7f03f0c00700 1 -- 172.21.15.12:0/2990322422 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8003a60 con 0x7f03f40709c0 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.191+0000 7f03fb800700 1 -- 172.21.15.12:0/2990322422 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f40cd800 con 0x7f03f40709c0 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.192+0000 7f03fb800700 1 -- 172.21.15.12:0/2990322422 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f4067720 con 0x7f03f40709c0 2024-03-29T10:10:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.193+0000 7f03f0c00700 1 -- 172.21.15.12:0/2990322422 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 35) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f03e8003bc0 con 0x7f03f40709c0 2024-03-29T10:10:03.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.194+0000 7f03f0c00700 1 -- 172.21.15.12:0/2990322422 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f03e808a060 con 0x7f03f40709c0 2024-03-29T10:10:03.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.199+0000 7f03f0c00700 1 -- 172.21.15.12:0/2990322422 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f03e80bd580 con 0x7f03f40709c0 2024-03-29T10:10:03.273 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:03.272+0000 7fd1471b9240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:03.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.458+0000 7f03fb800700 1 -- 172.21.15.12:0/2990322422 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03f40b39a0 con 0x7f03f40709c0 2024-03-29T10:10:03.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.458+0000 7f03f0c00700 1 -- 172.21.15.12:0/2990322422 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f03e80552c0 con 0x7f03f40709c0 2024-03-29T10:10:03.459 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:03.459 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3202930550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3202930550},{"type":"v1","addr":"172.21.15.159:6835","nonce":3202930550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4189378403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4189378403},{"type":"v1","addr":"172.21.15.12:6834","nonce":4189378403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2681299760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2681299760},{"type":"v1","addr":"172.21.15.12:6835","nonce":2681299760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:09:32.597152+0000","modified":"2024-03-29T10:09:35.607262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6241":{"gid":6241,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/4289275091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":4289275091},{"type":"v1","addr":"172.21.15.159:6837","nonce":4289275091}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-29T10:10:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.461+0000 7f03fb800700 1 -- 172.21.15.12:0/2990322422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f40709c0 msgr2=0x7f03f40b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.461+0000 7f03fb800700 1 --2- 172.21.15.12:0/2990322422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f40709c0 0x7f03f40b02f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009b90 tx=0x7f03e8004750 comp rx=0 tx=0).stop 2024-03-29T10:10:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.461+0000 7f03fb800700 1 -- 172.21.15.12:0/2990322422 shutdown_connections 2024-03-29T10:10:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.461+0000 7f03fb800700 1 --2- 172.21.15.12:0/2990322422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03f4160c90 0x7f03f40c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.461+0000 7f03fb800700 1 --2- 172.21.15.12:0/2990322422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03f40709c0 0x7f03f40b02f0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.461+0000 7f03fb800700 1 --2- 172.21.15.12:0/2990322422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03f40718e0 0x7f03f40cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.461+0000 7f03fb800700 1 -- 172.21.15.12:0/2990322422 >> 172.21.15.12:0/2990322422 conn(0x7f03f40cf190 msgr2=0x7f03f40ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.461+0000 7f03fb800700 1 -- 172.21.15.12:0/2990322422 shutdown_connections 2024-03-29T10:10:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.461+0000 7f03fb800700 1 -- 172.21.15.12:0/2990322422 wait complete. 2024-03-29T10:10:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 31 2024-03-29T10:10:03.479 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:03.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.676+0000 7f315d400700 1 Processor -- start 2024-03-29T10:10:03.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.677+0000 7f315d400700 1 -- start start 2024-03-29T10:10:03.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f315d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f315815d920 0x7f315806c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f315d400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f315809a910 0x7f31580cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f315d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3158099a60 0x7f3158099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f315d400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315809a3f0 con 0x7f3158099a60 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f315d400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31580d08d0 con 0x7f315809a910 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f315d400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315806c300 con 0x7f315815d920 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3157200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3158099a60 0x7f3158099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3157200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3158099a60 0x7f3158099eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55496/0 (socket says 172.21.15.12:55496) 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3157200700 1 -- 172.21.15.12:0/802302720 learned_addr learned my addr 172.21.15.12:0/802302720 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3156800700 1 --2- 172.21.15.12:0/802302720 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f315815d920 0x7f315806c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3155e00700 1 --2- 172.21.15.12:0/802302720 >> v2:172.21.15.159:3300/0 conn(0x7f315809a910 0x7f31580cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3157200700 1 -- 172.21.15.12:0/802302720 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f315815d920 msgr2=0x7f315806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3157200700 1 --2- 172.21.15.12:0/802302720 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f315815d920 0x7f315806c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3157200700 1 -- 172.21.15.12:0/802302720 >> v2:172.21.15.159:3300/0 conn(0x7f315809a910 msgr2=0x7f31580cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3157200700 1 --2- 172.21.15.12:0/802302720 >> v2:172.21.15.159:3300/0 conn(0x7f315809a910 0x7f31580cfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3157200700 1 -- 172.21.15.12:0/802302720 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315806b9d0 con 0x7f3158099a60 2024-03-29T10:10:03.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3156800700 1 --2- 172.21.15.12:0/802302720 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f315815d920 0x7f315806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:03.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.678+0000 7f3155e00700 1 --2- 172.21.15.12:0/802302720 >> v2:172.21.15.159:3300/0 conn(0x7f315809a910 0x7f31580cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:03.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.679+0000 7f3157200700 1 --2- 172.21.15.12:0/802302720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3158099a60 0x7f3158099eb0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f3158072220 tx=0x7f315000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540040a9e92b0be7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:03.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.679+0000 7f3155400700 1 -- 172.21.15.12:0/802302720 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315000e070 con 0x7f3158099a60 2024-03-29T10:10:03.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.679+0000 7f3155400700 1 -- 172.21.15.12:0/802302720 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315000b9c0 con 0x7f3158099a60 2024-03-29T10:10:03.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.679+0000 7f3155400700 1 -- 172.21.15.12:0/802302720 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315000bd70 con 0x7f3158099a60 2024-03-29T10:10:03.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.679+0000 7f315d400700 1 -- 172.21.15.12:0/802302720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3158099a60 msgr2=0x7f3158099eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:03.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.679+0000 7f315d400700 1 --2- 172.21.15.12:0/802302720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3158099a60 0x7f3158099eb0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f3158072220 tx=0x7f315000b330 comp rx=0 tx=0).stop 2024-03-29T10:10:03.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.680+0000 7f315d400700 1 -- 172.21.15.12:0/802302720 shutdown_connections 2024-03-29T10:10:03.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.680+0000 7f315d400700 1 --2- 172.21.15.12:0/802302720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3158099a60 0x7f3158099eb0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.680+0000 7f315d400700 1 --2- 172.21.15.12:0/802302720 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f315815d920 0x7f315806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.680+0000 7f315d400700 1 --2- 172.21.15.12:0/802302720 >> v2:172.21.15.159:3300/0 conn(0x7f315809a910 0x7f31580cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.680+0000 7f315d400700 1 -- 172.21.15.12:0/802302720 >> 172.21.15.12:0/802302720 conn(0x7f31580d1350 msgr2=0x7f31580d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.680+0000 7f315d400700 1 -- 172.21.15.12:0/802302720 shutdown_connections 2024-03-29T10:10:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.680+0000 7f315d400700 1 -- 172.21.15.12:0/802302720 wait complete. 2024-03-29T10:10:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.681+0000 7f315d400700 1 Processor -- start 2024-03-29T10:10:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.681+0000 7f315d400700 1 -- start start 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.682+0000 7f315d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3158099a60 0x7f31581f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.682+0000 7f315d400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f315809a910 0x7f3158156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.682+0000 7f315d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f315815d920 0x7f315815fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.682+0000 7f315d400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31581f2f70 con 0x7f315815d920 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.682+0000 7f315d400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31580c9680 con 0x7f315809a910 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.682+0000 7f315d400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31580c9960 con 0x7f3158099a60 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3157200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f315815d920 0x7f315815fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3157200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f315815d920 0x7f315815fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55502/0 (socket says 172.21.15.12:55502) 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3157200700 1 -- 172.21.15.12:0/3623875033 learned_addr learned my addr 172.21.15.12:0/3623875033 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3156800700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3158099a60 0x7f31581f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3155e00700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f315809a910 0x7f3158156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3157200700 1 -- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3158099a60 msgr2=0x7f31581f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3157200700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3158099a60 0x7f31581f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3157200700 1 -- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f315809a910 msgr2=0x7f3158156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:03.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3157200700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f315809a910 0x7f3158156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3157200700 1 -- 172.21.15.12:0/3623875033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3150009b30 con 0x7f315815d920 2024-03-29T10:10:03.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.683+0000 7f3157200700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f315815d920 0x7f315815fed0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f31580727d0 tx=0x7f3150004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:03.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.684+0000 7f314ea00700 1 -- 172.21.15.12:0/3623875033 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150004250 con 0x7f315815d920 2024-03-29T10:10:03.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.684+0000 7f314ea00700 1 -- 172.21.15.12:0/3623875033 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31500043d0 con 0x7f315815d920 2024-03-29T10:10:03.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.684+0000 7f315d400700 1 -- 172.21.15.12:0/3623875033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31580c9c40 con 0x7f315815d920 2024-03-29T10:10:03.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.684+0000 7f314ea00700 1 -- 172.21.15.12:0/3623875033 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150004780 con 0x7f315815d920 2024-03-29T10:10:03.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.684+0000 7f315d400700 1 -- 172.21.15.12:0/3623875033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31580ca200 con 0x7f315815d920 2024-03-29T10:10:03.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.685+0000 7f315d400700 1 -- 172.21.15.12:0/3623875033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3158067720 con 0x7f315815d920 2024-03-29T10:10:03.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.686+0000 7f314ea00700 1 -- 172.21.15.12:0/3623875033 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 35) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f315001b430 con 0x7f315815d920 2024-03-29T10:10:03.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.687+0000 7f314ea00700 1 -- 172.21.15.12:0/3623875033 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f315008a260 con 0x7f315815d920 2024-03-29T10:10:03.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.691+0000 7f314ea00700 1 -- 172.21.15.12:0/3623875033 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3150055540 con 0x7f315815d920 2024-03-29T10:10:03.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.806+0000 7f314ea00700 1 -- 172.21.15.12:0/3623875033 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 36) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f3150054ff0 con 0x7f315815d920 2024-03-29T10:10:03.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.806+0000 7f314ea00700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3138036150 0x7f31380722c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:03.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.809+0000 7f3156800700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3138036150 0x7f31380722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:03.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.812+0000 7f3156800700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3138036150 0x7f31380722c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f314800c900 tx=0x7f3148008040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:03.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.909+0000 7f315d400700 1 -- 172.21.15.12:0/3623875033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31580708d0 con 0x7f315815d920 2024-03-29T10:10:03.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.911+0000 7f314ea00700 1 -- 172.21.15.12:0/3623875033 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16973 (secure 0 0 0) 0x7f3150070100 con 0x7f315815d920 2024-03-29T10:10:03.912 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:03.912 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":82,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:02.794666+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:09:30.441619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-29T10:09:31.581366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1229127693":"2024-03-30T10:10:02.794611+0000","172.21.15.159:0/3367842275":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/2513478349":"2024-03-30T10:10:02.794611+0000","172.21.15.159:0/3331610524":"2024-03-30T10:10:02.794611+0000","172.21.15.159:0/3723093447":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/4096971436":"2024-03-30T10:10:02.794611+0000","172.21.15.159:6833/43444":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/3389068513":"2024-03-30T10:10:02.794611+0000","172.21.15.159:6832/43444":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/1578988741":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/1180347815":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/2971357001":"2024-03-30T10:10:02.794611+0000","172.21.15.159:0/667128976":"2024-03-30T10:09:53.746392+0000","172.21.15.159:0/3208721490":"2024-03-30T10:09:53.746392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:03.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 -- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3138036150 msgr2=0x7f31380722c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:03.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3138036150 0x7f31380722c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f314800c900 tx=0x7f3148008040 comp rx=0 tx=0).stop 2024-03-29T10:10:03.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 -- 172.21.15.12:0/3623875033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f315815d920 msgr2=0x7f315815fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:03.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f315815d920 0x7f315815fed0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f31580727d0 tx=0x7f3150004ad0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 -- 172.21.15.12:0/3623875033 shutdown_connections 2024-03-29T10:10:03.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3138036150 0x7f31380722c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f315815d920 0x7f315815fed0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3158099a60 0x7f31581f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 --2- 172.21.15.12:0/3623875033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f315809a910 0x7f3158156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:03.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 -- 172.21.15.12:0/3623875033 >> 172.21.15.12:0/3623875033 conn(0x7f31580d1350 msgr2=0x7f31580ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:03.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 -- 172.21.15.12:0/3623875033 shutdown_connections 2024-03-29T10:10:03.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:03.913+0000 7f315d400700 1 -- 172.21.15.12:0/3623875033 wait complete. 2024-03-29T10:10:03.932 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T10:10:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.132+0000 7f5312200700 1 Processor -- start 2024-03-29T10:10:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.132+0000 7f5312200700 1 -- start start 2024-03-29T10:10:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.133+0000 7f5312200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f530c0cbb30 0x7f530c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.133+0000 7f5312200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0cc450 con 0x7f530c0cd480 2024-03-29T10:10:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.133+0000 7f5312200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0cdae0 con 0x7f530c0cd020 2024-03-29T10:10:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.134+0000 7f5312200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f530c0cd8f0 con 0x7f530c0cbb30 2024-03-29T10:10:04.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.134+0000 7f530b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f530c0cbb30 0x7f530c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.134+0000 7f530be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f530c0cd480 0x7f530c06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36360/0 (socket says 172.21.15.12:36360) 2024-03-29T10:10:04.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.134+0000 7f530be00700 1 -- 172.21.15.12:0/2264278265 learned_addr learned my addr 172.21.15.12:0/2264278265 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.134+0000 7f530b400700 1 -- 172.21.15.12:0/2264278265 >> v1:172.21.15.159:6789/0 conn(0x7f530c0cd020 legacy=0x7f530c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.134+0000 7f530b400700 1 -- 172.21.15.12:0/2264278265 >> v1:172.21.15.12:6789/0 conn(0x7f530c0cd480 legacy=0x7f530c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.134+0000 7f530b400700 1 -- 172.21.15.12:0/2264278265 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c0cddc0 con 0x7f530c0cbb30 2024-03-29T10:10:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.134+0000 7f530b400700 1 --2- 172.21.15.12:0/2264278265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cbb30 0x7f530c0cbf10 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009840 tx=0x7f52fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f87e2db0697a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.135+0000 7f530a000700 1 -- 172.21.15.12:0/2264278265 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc004230 con 0x7f530c0cbb30 2024-03-29T10:10:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.135+0000 7f530a000700 1 -- 172.21.15.12:0/2264278265 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52fc004390 con 0x7f530c0cbb30 2024-03-29T10:10:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.135+0000 7f530a000700 1 -- 172.21.15.12:0/2264278265 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc0036a0 con 0x7f530c0cbb30 2024-03-29T10:10:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.135+0000 7f5312200700 1 -- 172.21.15.12:0/2264278265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cbb30 msgr2=0x7f530c0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.135+0000 7f5312200700 1 --2- 172.21.15.12:0/2264278265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cbb30 0x7f530c0cbf10 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009840 tx=0x7f52fc009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:04.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.135+0000 7f5312200700 1 -- 172.21.15.12:0/2264278265 shutdown_connections 2024-03-29T10:10:04.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.135+0000 7f5312200700 1 --2- 172.21.15.12:0/2264278265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cbb30 0x7f530c0cbf10 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.135+0000 7f5312200700 1 -- 172.21.15.12:0/2264278265 >> 172.21.15.12:0/2264278265 conn(0x7f530c0b0d70 msgr2=0x7f530c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:04.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.135+0000 7f5312200700 1 -- 172.21.15.12:0/2264278265 shutdown_connections 2024-03-29T10:10:04.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.135+0000 7f5312200700 1 -- 172.21.15.12:0/2264278265 wait complete. 2024-03-29T10:10:04.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.137+0000 7f5312200700 1 Processor -- start 2024-03-29T10:10:04.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.137+0000 7f5312200700 1 -- start start 2024-03-29T10:10:04.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.138+0000 7f5312200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f530c0cbb30 0x7f530c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.138+0000 7f5312200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cd020 0x7f530c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.138+0000 7f5312200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f530c0cd480 0x7f530c0cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.138+0000 7f5312200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c0712c0 con 0x7f530c0cd020 2024-03-29T10:10:04.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.138+0000 7f5312200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c0715a0 con 0x7f530c0cbb30 2024-03-29T10:10:04.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.138+0000 7f5312200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530c071880 con 0x7f530c0cd480 2024-03-29T10:10:04.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cd020 0x7f530c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cd020 0x7f530c0709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55520/0 (socket says 172.21.15.12:55520) 2024-03-29T10:10:04.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530aa00700 1 -- 172.21.15.12:0/829060373 learned_addr learned my addr 172.21.15.12:0/829060373 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:04.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530b400700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f530c0cbb30 0x7f530c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530be00700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f530c0cd480 0x7f530c0cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530aa00700 1 -- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f530c0cd480 msgr2=0x7f530c0cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530aa00700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f530c0cd480 0x7f530c0cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530aa00700 1 -- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f530c0cbb30 msgr2=0x7f530c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530aa00700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f530c0cbb30 0x7f530c0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530aa00700 1 -- 172.21.15.12:0/829060373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc0094a0 con 0x7f530c0cd020 2024-03-29T10:10:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.139+0000 7f530aa00700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cd020 0x7f530c0709c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f52f800b4a0 tx=0x7f52f800b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.140+0000 7f5303400700 1 -- 172.21.15.12:0/829060373 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f8011830 con 0x7f530c0cd020 2024-03-29T10:10:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.140+0000 7f5303400700 1 -- 172.21.15.12:0/829060373 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f8011990 con 0x7f530c0cd020 2024-03-29T10:10:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.140+0000 7f5303400700 1 -- 172.21.15.12:0/829060373 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f8011d40 con 0x7f530c0cd020 2024-03-29T10:10:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.140+0000 7f5312200700 1 -- 172.21.15.12:0/829060373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530c071b60 con 0x7f530c0cd020 2024-03-29T10:10:04.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.140+0000 7f5312200700 1 -- 172.21.15.12:0/829060373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c072340 con 0x7f530c0cd020 2024-03-29T10:10:04.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.141+0000 7f5312200700 1 -- 172.21.15.12:0/829060373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530c067720 con 0x7f530c0cd020 2024-03-29T10:10:04.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.142+0000 7f5303400700 1 -- 172.21.15.12:0/829060373 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 36) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f52f801a400 con 0x7f530c0cd020 2024-03-29T10:10:04.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.143+0000 7f5303400700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f52ec06aae0 0x7f52ec06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.143+0000 7f5303400700 1 -- 172.21.15.12:0/829060373 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f52f8088e30 con 0x7f530c0cd020 2024-03-29T10:10:04.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.143+0000 7f530b400700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f52ec06aae0 0x7f52ec06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.144+0000 7f530b400700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f52ec06aae0 0x7f52ec06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0090b0 tx=0x7f52fc008030 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:04.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.147+0000 7f5303400700 1 -- 172.21.15.12:0/829060373 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f52f8054020 con 0x7f530c0cd020 2024-03-29T10:10:04.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.406+0000 7f5312200700 1 -- 172.21.15.12:0/829060373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f530c0cbfa0 con 0x7f530c0cd020 2024-03-29T10:10:04.408 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T10:10:04.408+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T10:10:04.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.420+0000 7f5303400700 1 -- 172.21.15.12:0/829060373 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) v1 ==== 161+0+0 (secure 0 0 0) 0x7f52f8015090 con 0x7f530c0cd020 2024-03-29T10:10:04.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 -- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f52ec06aae0 msgr2=0x7f52ec06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f52ec06aae0 0x7f52ec06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0090b0 tx=0x7f52fc008030 comp rx=0 tx=0).stop 2024-03-29T10:10:04.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 -- 172.21.15.12:0/829060373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cd020 msgr2=0x7f530c0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cd020 0x7f530c0709c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f52f800b4a0 tx=0x7f52f800b7b0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 -- 172.21.15.12:0/829060373 shutdown_connections 2024-03-29T10:10:04.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f52ec06aae0 0x7f52ec06cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f530c0cd480 0x7f530c0cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f530c0cbb30 0x7f530c0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 --2- 172.21.15.12:0/829060373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0cd020 0x7f530c0709c0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 -- 172.21.15.12:0/829060373 >> 172.21.15.12:0/829060373 conn(0x7f530c0b0d70 msgr2=0x7f530c15d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:04.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 -- 172.21.15.12:0/829060373 shutdown_connections 2024-03-29T10:10:04.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.423+0000 7f5312200700 1 -- 172.21.15.12:0/829060373 wait complete. 2024-03-29T10:10:04.425 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T10:10:04.432 INFO:tasks.ceph.mds.b.smithi159.stderr: -5113> 2024-03-29T10:09:37.047+0000 7f5993a00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:10:04.441 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T10:10:04.483 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T10:10:04.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.644+0000 7f888c800700 1 Processor -- start 2024-03-29T10:10:04.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.644+0000 7f888c800700 1 -- start start 2024-03-29T10:10:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.645+0000 7f888c800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f88840d2540 0x7f88841687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.645+0000 7f888c800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8884161420 con 0x7f88840d16a0 2024-03-29T10:10:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.645+0000 7f888c800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8884161700 con 0x7f888415d9d0 2024-03-29T10:10:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.645+0000 7f888c800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88840d4340 con 0x7f88840d2540 2024-03-29T10:10:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.645+0000 7f8889400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f88840d16a0 0x7f88840d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36368/0 (socket says 172.21.15.12:36368) 2024-03-29T10:10:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.645+0000 7f8889400700 1 -- 172.21.15.12:0/3451396466 learned_addr learned my addr 172.21.15.12:0/3451396466 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584817904 0 0) 0x7f8884161420 con 0x7f88840d16a0 2024-03-29T10:10:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886c0035b0 con 0x7f88840d16a0 2024-03-29T10:10:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f888a800700 1 --2- 172.21.15.12:0/3451396466 >> v2:172.21.15.159:3300/0 conn(0x7f88840d2540 0x7f88841687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1857812816 0 0) 0x7f886c0035b0 con 0x7f88840d16a0 2024-03-29T10:10:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 >> v2:172.21.15.159:3300/0 conn(0x7f88840d2540 msgr2=0x7f88841687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 --2- 172.21.15.12:0/3451396466 >> v2:172.21.15.159:3300/0 conn(0x7f88840d2540 0x7f88841687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 >> v1:172.21.15.159:6789/0 conn(0x7f888415d9d0 legacy=0x7f88840d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88841619e0 con 0x7f88840d16a0 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f88740017f0 con 0x7f88840d16a0 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 >> v1:172.21.15.12:6789/0 conn(0x7f88840d16a0 legacy=0x7f88840d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f886c003a00 0x7f886c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f886c004850 0x7f886c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c005210 0x7f886c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c00c0e0 con 0x7f886c005210 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c00c250 con 0x7f886c003a00 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.646+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c00c410 con 0x7f886c004850 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.647+0000 7f8889e00700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f886c003a00 0x7f886c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.647+0000 7f888a800700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c005210 0x7f886c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.647+0000 7f8889400700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f886c004850 0x7f886c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.647+0000 7f888a800700 1 -- 172.21.15.12:0/3451396466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f886c004850 msgr2=0x7f886c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.647+0000 7f888a800700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f886c004850 0x7f886c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.647+0000 7f888a800700 1 -- 172.21.15.12:0/3451396466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f886c003a00 msgr2=0x7f886c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.647+0000 7f888a800700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f886c003a00 0x7f886c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.647+0000 7f888a800700 1 -- 172.21.15.12:0/3451396466 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c0035b0 con 0x7f886c005210 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.647+0000 7f888a800700 1 -- 172.21.15.12:0/3451396466 reap_dead start 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.647+0000 7f888a800700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c005210 0x7f886c00ba10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8880009f90 tx=0x7f886c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06c3923c0886081 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.648+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880003a20 con 0x7f886c005210 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.648+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888000f040 con 0x7f886c005210 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.648+0000 7f8888a00700 1 -- 172.21.15.12:0/3451396466 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880003de0 con 0x7f886c005210 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.648+0000 7f888c800700 1 -- 172.21.15.12:0/3451396466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c005210 msgr2=0x7f886c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.648+0000 7f888c800700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c005210 0x7f886c00ba10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8880009f90 tx=0x7f886c005880 comp rx=0 tx=0).stop 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.648+0000 7f888c800700 1 -- 172.21.15.12:0/3451396466 shutdown_connections 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.648+0000 7f888c800700 1 --2- 172.21.15.12:0/3451396466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c005210 0x7f886c00ba10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.648+0000 7f888c800700 1 -- 172.21.15.12:0/3451396466 >> 172.21.15.12:0/3451396466 conn(0x7f888408aff0 msgr2=0x7f88840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.648+0000 7f888c800700 1 -- 172.21.15.12:0/3451396466 shutdown_connections 2024-03-29T10:10:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.648+0000 7f888c800700 1 -- 172.21.15.12:0/3451396466 wait complete. 2024-03-29T10:10:04.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.650+0000 7f888c800700 1 Processor -- start 2024-03-29T10:10:04.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.650+0000 7f888c800700 1 -- start start 2024-03-29T10:10:04.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.651+0000 7f888c800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f886c005210 0x7f88840ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.651+0000 7f888c800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88840cafe0 0x7f88841550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.651+0000 7f888c800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8884160450 0x7f88840b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.651+0000 7f888c800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888415f770 con 0x7f88840cafe0 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.651+0000 7f888c800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88841f31e0 con 0x7f8884160450 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.651+0000 7f888c800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88840cb810 con 0x7f886c005210 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.651+0000 7f8889400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88840cafe0 0x7f88841550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.651+0000 7f8889400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88840cafe0 0x7f88841550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55526/0 (socket says 172.21.15.12:55526) 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.651+0000 7f8889400700 1 -- 172.21.15.12:0/3109369666 learned_addr learned my addr 172.21.15.12:0/3109369666 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f888a800700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8884160450 0x7f88840b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f8889e00700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f886c005210 0x7f88840ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f8889400700 1 -- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f886c005210 msgr2=0x7f88840ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f8889400700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f886c005210 0x7f88840ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f8889400700 1 -- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8884160450 msgr2=0x7f88840b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:04.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f8889400700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8884160450 0x7f88840b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:04.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f8889400700 1 -- 172.21.15.12:0/3109369666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88840cb530 con 0x7f88840cafe0 2024-03-29T10:10:04.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f8889400700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88840cafe0 0x7f88841550f0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8874006f20 tx=0x7f8874007320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:04.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f887e000700 1 -- 172.21.15.12:0/3109369666 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874011580 con 0x7f88840cafe0 2024-03-29T10:10:04.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f887e000700 1 -- 172.21.15.12:0/3109369666 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8874011700 con 0x7f88840cafe0 2024-03-29T10:10:04.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f888c800700 1 -- 172.21.15.12:0/3109369666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888415f970 con 0x7f88840cafe0 2024-03-29T10:10:04.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.652+0000 7f887e000700 1 -- 172.21.15.12:0/3109369666 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874011ab0 con 0x7f88840cafe0 2024-03-29T10:10:04.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.653+0000 7f888c800700 1 -- 172.21.15.12:0/3109369666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888415ff30 con 0x7f88840cafe0 2024-03-29T10:10:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.654+0000 7f888c800700 1 -- 172.21.15.12:0/3109369666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8884067720 con 0x7f88840cafe0 2024-03-29T10:10:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.655+0000 7f887e000700 1 -- 172.21.15.12:0/3109369666 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 36) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f8874011c10 con 0x7f88840cafe0 2024-03-29T10:10:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.656+0000 7f887e000700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f886806aae0 0x7f886806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.656+0000 7f887e000700 1 -- 172.21.15.12:0/3109369666 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f88740890e0 con 0x7f88840cafe0 2024-03-29T10:10:04.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.656+0000 7f8889e00700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f886806aae0 0x7f886806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:04.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.657+0000 7f8889e00700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f886806aae0 0x7f886806cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8878000c00 tx=0x7f8878009040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:04.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.660+0000 7f887e000700 1 -- 172.21.15.12:0/3109369666 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8874054350 con 0x7f88840cafe0 2024-03-29T10:10:04.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:04.925+0000 7f888c800700 1 -- 172.21.15.12:0/3109369666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8884067eb0 con 0x7f88840cafe0 2024-03-29T10:10:05.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.423+0000 7f887e000700 1 -- 172.21.15.12:0/3109369666 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8874057970 con 0x7f88840cafe0 2024-03-29T10:10:05.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.425+0000 7f888c800700 1 -- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f886806aae0 msgr2=0x7f886806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.425+0000 7f888c800700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f886806aae0 0x7f886806cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8878000c00 tx=0x7f8878009040 comp rx=0 tx=0).stop 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.425+0000 7f888c800700 1 -- 172.21.15.12:0/3109369666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88840cafe0 msgr2=0x7f88841550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.425+0000 7f888c800700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88840cafe0 0x7f88841550f0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8874006f20 tx=0x7f8874007320 comp rx=0 tx=0).stop 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.426+0000 7f888c800700 1 -- 172.21.15.12:0/3109369666 shutdown_connections 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.426+0000 7f888c800700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f886806aae0 0x7f886806cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.426+0000 7f888c800700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8884160450 0x7f88840b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.426+0000 7f888c800700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f886c005210 0x7f88840ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.426+0000 7f888c800700 1 --2- 172.21.15.12:0/3109369666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88840cafe0 0x7f88841550f0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.426+0000 7f888c800700 1 -- 172.21.15.12:0/3109369666 >> 172.21.15.12:0/3109369666 conn(0x7f888408aff0 msgr2=0x7f888415c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.426+0000 7f888c800700 1 -- 172.21.15.12:0/3109369666 shutdown_connections 2024-03-29T10:10:05.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.426+0000 7f888c800700 1 -- 172.21.15.12:0/3109369666 wait complete. 2024-03-29T10:10:05.448 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T10:10:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.652+0000 7fde7d200700 1 Processor -- start 2024-03-29T10:10:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.652+0000 7fde7d200700 1 -- start start 2024-03-29T10:10:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde7d200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fde780ced30 0x7fde780d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde7d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde780d2710 0x7fde7815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde7d200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde780ce8a0 con 0x7fde7815ddf0 2024-03-29T10:10:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde7d200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7815ec60 con 0x7fde780ced30 2024-03-29T10:10:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde7d200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde780cdc90 con 0x7fde780d2710 2024-03-29T10:10:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde75e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fde7815ddf0 0x7fde7815e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36372/0 (socket says 172.21.15.12:36372) 2024-03-29T10:10:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde75e00700 1 -- 172.21.15.12:0/2473666632 learned_addr learned my addr 172.21.15.12:0/2473666632 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde75400700 1 --2- 172.21.15.12:0/2473666632 >> v2:172.21.15.159:3300/0 conn(0x7fde780ced30 0x7fde780d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde76800700 1 --2- 172.21.15.12:0/2473666632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde780d2710 0x7fde7815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde74a00700 1 -- 172.21.15.12:0/2473666632 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862850676 0 0) 0x7fde780ce8a0 con 0x7fde7815ddf0 2024-03-29T10:10:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.654+0000 7fde74a00700 1 -- 172.21.15.12:0/2473666632 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde5c0035b0 con 0x7fde7815ddf0 2024-03-29T10:10:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.655+0000 7fde75400700 1 -- 172.21.15.12:0/2473666632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde780d2710 msgr2=0x7fde7815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.655+0000 7fde75400700 1 --2- 172.21.15.12:0/2473666632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde780d2710 0x7fde7815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.655+0000 7fde75400700 1 -- 172.21.15.12:0/2473666632 >> v1:172.21.15.12:6789/0 conn(0x7fde7815ddf0 legacy=0x7fde7815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.655+0000 7fde75400700 1 -- 172.21.15.12:0/2473666632 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde780cdf70 con 0x7fde780ced30 2024-03-29T10:10:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.655+0000 7fde75400700 1 --2- 172.21.15.12:0/2473666632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde780ced30 0x7fde780d21d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fde64009840 tx=0x7fde64009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16d5f2e11521b11d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.655+0000 7fde74a00700 1 -- 172.21.15.12:0/2473666632 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64011830 con 0x7fde780ced30 2024-03-29T10:10:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.655+0000 7fde74a00700 1 -- 172.21.15.12:0/2473666632 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde64011990 con 0x7fde780ced30 2024-03-29T10:10:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.655+0000 7fde74a00700 1 -- 172.21.15.12:0/2473666632 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64011d20 con 0x7fde780ced30 2024-03-29T10:10:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.656+0000 7fde7d200700 1 -- 172.21.15.12:0/2473666632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde780ced30 msgr2=0x7fde780d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.656+0000 7fde7d200700 1 --2- 172.21.15.12:0/2473666632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde780ced30 0x7fde780d21d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fde64009840 tx=0x7fde64009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.656+0000 7fde7d200700 1 -- 172.21.15.12:0/2473666632 shutdown_connections 2024-03-29T10:10:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.656+0000 7fde7d200700 1 --2- 172.21.15.12:0/2473666632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde780d2710 0x7fde7815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.656+0000 7fde7d200700 1 --2- 172.21.15.12:0/2473666632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde780ced30 0x7fde780d21d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.656+0000 7fde7d200700 1 -- 172.21.15.12:0/2473666632 >> 172.21.15.12:0/2473666632 conn(0x7fde7808aff0 msgr2=0x7fde780d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.656+0000 7fde7d200700 1 -- 172.21.15.12:0/2473666632 shutdown_connections 2024-03-29T10:10:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.656+0000 7fde7d200700 1 -- 172.21.15.12:0/2473666632 wait complete. 2024-03-29T10:10:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.658+0000 7fde7d200700 1 Processor -- start 2024-03-29T10:10:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.658+0000 7fde7d200700 1 -- start start 2024-03-29T10:10:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde7d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde780ced30 0x7fde7815be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde7d200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde780d2710 0x7fde780ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde7d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde7815ddf0 0x7fde7815b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde7d200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7815bd00 con 0x7fde780ced30 2024-03-29T10:10:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde7d200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde780c84a0 con 0x7fde780d2710 2024-03-29T10:10:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde7d200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde780cac70 con 0x7fde7815ddf0 2024-03-29T10:10:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde75e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde780ced30 0x7fde7815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde75e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde780ced30 0x7fde7815be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55532/0 (socket says 172.21.15.12:55532) 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde75e00700 1 -- 172.21.15.12:0/2319141411 learned_addr learned my addr 172.21.15.12:0/2319141411 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde75400700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde780d2710 0x7fde780ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde76800700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde7815ddf0 0x7fde7815b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde75e00700 1 -- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde7815ddf0 msgr2=0x7fde7815b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde75e00700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde7815ddf0 0x7fde7815b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde75e00700 1 -- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde780d2710 msgr2=0x7fde780ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde75e00700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde780d2710 0x7fde780ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde75e00700 1 -- 172.21.15.12:0/2319141411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde640094a0 con 0x7fde780ced30 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.659+0000 7fde76800700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde7815ddf0 0x7fde7815b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.660+0000 7fde75e00700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde780ced30 0x7fde7815be70 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fde68001eb0 tx=0x7fde6800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.660+0000 7fde6e000700 1 -- 172.21.15.12:0/2319141411 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68005d70 con 0x7fde780ced30 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.660+0000 7fde6e000700 1 -- 172.21.15.12:0/2319141411 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde680070d0 con 0x7fde780ced30 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.660+0000 7fde7d200700 1 -- 172.21.15.12:0/2319141411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde78161640 con 0x7fde780ced30 2024-03-29T10:10:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.660+0000 7fde6e000700 1 -- 172.21.15.12:0/2319141411 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68007460 con 0x7fde780ced30 2024-03-29T10:10:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.660+0000 7fde7d200700 1 -- 172.21.15.12:0/2319141411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7815c660 con 0x7fde780ced30 2024-03-29T10:10:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.661+0000 7fde7d200700 1 -- 172.21.15.12:0/2319141411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde780c8670 con 0x7fde780ced30 2024-03-29T10:10:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.663+0000 7fde6e000700 1 -- 172.21.15.12:0/2319141411 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 37) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fde680075c0 con 0x7fde780ced30 2024-03-29T10:10:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.663+0000 7fde6e000700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde5806ad20 0x7fde5806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.663+0000 7fde6e000700 1 -- 172.21.15.12:0/2319141411 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fde68088b50 con 0x7fde780ced30 2024-03-29T10:10:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.664+0000 7fde75400700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde5806ad20 0x7fde5806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.664+0000 7fde75400700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde5806ad20 0x7fde5806d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde64000c00 tx=0x7fde64008030 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.667+0000 7fde6e000700 1 -- 172.21.15.12:0/2319141411 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fde68053a20 con 0x7fde780ced30 2024-03-29T10:10:05.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:05.882+0000 7fde7d200700 1 -- 172.21.15.12:0/2319141411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fde78067720 con 0x7fde780ced30 2024-03-29T10:10:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.814+0000 7fde6e000700 1 -- 172.21.15.12:0/2319141411 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v84) v1 ==== 181+0+0 (secure 0 0 0) 0x7fde68057040 con 0x7fde780ced30 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 -- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde5806ad20 msgr2=0x7fde5806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde5806ad20 0x7fde5806d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde64000c00 tx=0x7fde64008030 comp rx=0 tx=0).stop 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 -- 172.21.15.12:0/2319141411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde780ced30 msgr2=0x7fde7815be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde780ced30 0x7fde7815be70 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fde68001eb0 tx=0x7fde6800c7b0 comp rx=0 tx=0).stop 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 -- 172.21.15.12:0/2319141411 shutdown_connections 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fde5806ad20 0x7fde5806d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde7815ddf0 0x7fde7815b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde780ced30 0x7fde7815be70 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 --2- 172.21.15.12:0/2319141411 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde780d2710 0x7fde780ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 -- 172.21.15.12:0/2319141411 >> 172.21.15.12:0/2319141411 conn(0x7fde7808aff0 msgr2=0x7fde780b84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 -- 172.21.15.12:0/2319141411 shutdown_connections 2024-03-29T10:10:06.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:06.818+0000 7fde7d200700 1 -- 172.21.15.12:0/2319141411 wait complete. 2024-03-29T10:10:06.820 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T10:10:06.840 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T10:10:07.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.045+0000 7fe1efe00700 1 Processor -- start 2024-03-29T10:10:07.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.045+0000 7fe1efe00700 1 -- start start 2024-03-29T10:10:07.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.046+0000 7fe1efe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe1e815db10 0x7fe1e8099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:07.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1efe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe1e80cc280 0x7fe1e80cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:07.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1efe00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e80cbdf0 con 0x7fe1e80b3930 2024-03-29T10:10:07.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1efe00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e806bec0 con 0x7fe1e80cc280 2024-03-29T10:10:07.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1efe00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e80b27a0 con 0x7fe1e815db10 2024-03-29T10:10:07.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1ede00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe1e80cc280 0x7fe1e80cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:07.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1ede00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe1e80cc280 0x7fe1e80cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55536/0 (socket says 172.21.15.12:55536) 2024-03-29T10:10:07.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1eca00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe1e80b3930 0x7fe1e8099fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36378/0 (socket says 172.21.15.12:36378) 2024-03-29T10:10:07.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1ede00700 1 -- 172.21.15.12:0/1390549554 learned_addr learned my addr 172.21.15.12:0/1390549554 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:07.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1ed400700 1 --2- 172.21.15.12:0/1390549554 >> v2:172.21.15.159:3300/0 conn(0x7fe1e815db10 0x7fe1e8099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:07.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1ede00700 1 -- 172.21.15.12:0/1390549554 >> v2:172.21.15.159:3300/0 conn(0x7fe1e815db10 msgr2=0x7fe1e8099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:07.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1ede00700 1 --2- 172.21.15.12:0/1390549554 >> v2:172.21.15.159:3300/0 conn(0x7fe1e815db10 0x7fe1e8099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:07.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1ede00700 1 -- 172.21.15.12:0/1390549554 >> v1:172.21.15.12:6789/0 conn(0x7fe1e80b3930 legacy=0x7fe1e8099fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:07.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1ede00700 1 -- 172.21.15.12:0/1390549554 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e806bbe0 con 0x7fe1e80cc280 2024-03-29T10:10:07.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.047+0000 7fe1e7e00700 1 -- 172.21.15.12:0/1390549554 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153906467 0 0) 0x7fe1e80cbdf0 con 0x7fe1e80b3930 2024-03-29T10:10:07.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.048+0000 7fe1ede00700 1 --2- 172.21.15.12:0/1390549554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80cc280 0x7fe1e80cc710 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8072480 tx=0x7fe1e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb452c83a7c98981 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:07.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.048+0000 7fe1e7e00700 1 -- 172.21.15.12:0/1390549554 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e000e070 con 0x7fe1e80cc280 2024-03-29T10:10:07.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.048+0000 7fe1e7e00700 1 -- 172.21.15.12:0/1390549554 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e000b910 con 0x7fe1e80cc280 2024-03-29T10:10:07.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.048+0000 7fe1e7e00700 1 -- 172.21.15.12:0/1390549554 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e000bcc0 con 0x7fe1e80cc280 2024-03-29T10:10:07.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.050+0000 7fe1efe00700 1 -- 172.21.15.12:0/1390549554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80cc280 msgr2=0x7fe1e80cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:07.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.050+0000 7fe1efe00700 1 --2- 172.21.15.12:0/1390549554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80cc280 0x7fe1e80cc710 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8072480 tx=0x7fe1e000b330 comp rx=0 tx=0).stop 2024-03-29T10:10:07.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.050+0000 7fe1efe00700 1 -- 172.21.15.12:0/1390549554 shutdown_connections 2024-03-29T10:10:07.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.050+0000 7fe1efe00700 1 --2- 172.21.15.12:0/1390549554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80cc280 0x7fe1e80cc710 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:07.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.050+0000 7fe1efe00700 1 --2- 172.21.15.12:0/1390549554 >> v2:172.21.15.159:3300/0 conn(0x7fe1e815db10 0x7fe1e8099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:07.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.050+0000 7fe1efe00700 1 -- 172.21.15.12:0/1390549554 >> 172.21.15.12:0/1390549554 conn(0x7fe1e80b2e10 msgr2=0x7fe1e80b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:07.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.050+0000 7fe1efe00700 1 -- 172.21.15.12:0/1390549554 shutdown_connections 2024-03-29T10:10:07.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.050+0000 7fe1efe00700 1 -- 172.21.15.12:0/1390549554 wait complete. 2024-03-29T10:10:07.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.050+0000 7fe1efe00700 1 Processor -- start 2024-03-29T10:10:07.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.051+0000 7fe1efe00700 1 -- start start 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1efe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80b3930 0x7fe1e80cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1efe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1e80cc280 0x7fe1e814d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1efe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1e815db10 0x7fe1e814d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1efe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e80cb680 con 0x7fe1e80b3930 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1efe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e80c6a30 con 0x7fe1e80cc280 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1efe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e80c6d10 con 0x7fe1e815db10 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1ed400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80b3930 0x7fe1e80cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1ed400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80b3930 0x7fe1e80cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55548/0 (socket says 172.21.15.12:55548) 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1ed400700 1 -- 172.21.15.12:0/4199319601 learned_addr learned my addr 172.21.15.12:0/4199319601 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1eca00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1e80cc280 0x7fe1e814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1ed400700 1 -- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1e815db10 msgr2=0x7fe1e814d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:07.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1ede00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1e815db10 0x7fe1e814d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1ed400700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1e815db10 0x7fe1e814d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1ed400700 1 -- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1e80cc280 msgr2=0x7fe1e814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1ed400700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1e80cc280 0x7fe1e814d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1ed400700 1 -- 172.21.15.12:0/4199319601 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e0009b30 con 0x7fe1e80b3930 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1eca00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1e80cc280 0x7fe1e814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1ed400700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80b3930 0x7fe1e80cb140 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00e870 tx=0x7fe1dc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1e5600700 1 -- 172.21.15.12:0/4199319601 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc009590 con 0x7fe1e80b3930 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1e5600700 1 -- 172.21.15.12:0/4199319601 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1dc00ee70 con 0x7fe1e80b3930 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1e5600700 1 -- 172.21.15.12:0/4199319601 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc00ca50 con 0x7fe1e80b3930 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1efe00700 1 -- 172.21.15.12:0/4199319601 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e80c6ff0 con 0x7fe1e80b3930 2024-03-29T10:10:07.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.053+0000 7fe1efe00700 1 -- 172.21.15.12:0/4199319601 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e80c6670 con 0x7fe1e80b3930 2024-03-29T10:10:07.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.054+0000 7fe1efe00700 1 -- 172.21.15.12:0/4199319601 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e80c7640 con 0x7fe1e80b3930 2024-03-29T10:10:07.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.056+0000 7fe1e5600700 1 -- 172.21.15.12:0/4199319601 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 37) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fe1dc00cbb0 con 0x7fe1e80b3930 2024-03-29T10:10:07.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.056+0000 7fe1e5600700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe1c806ad20 0x7fe1c806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:07.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.056+0000 7fe1e5600700 1 -- 172.21.15.12:0/4199319601 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5959+0+0 (secure 0 0 0) 0x7fe1dc0893f0 con 0x7fe1e80b3930 2024-03-29T10:10:07.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.056+0000 7fe1eca00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe1c806ad20 0x7fe1c806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:07.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.057+0000 7fe1eca00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe1c806ad20 0x7fe1c806d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4007fd0 tx=0x7fe1d4009040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:07.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.061+0000 7fe1e5600700 1 -- 172.21.15.12:0/4199319601 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1dc0544e0 con 0x7fe1e80b3930 2024-03-29T10:10:07.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.066+0000 7fe1e5600700 1 -- 172.21.15.12:0/4199319601 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe1dc0548b0 con 0x7fe1e80b3930 2024-03-29T10:10:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:07.280+0000 7fe1efe00700 1 -- 172.21.15.12:0/4199319601 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe1e80cef70 con 0x7fe1e80b3930 2024-03-29T10:10:08.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.070+0000 7fe1e5600700 1 -- 172.21.15.12:0/4199319601 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe1dc087480 con 0x7fe1e80b3930 2024-03-29T10:10:08.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.074+0000 7fe1efe00700 1 -- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe1c806ad20 msgr2=0x7fe1c806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.074+0000 7fe1efe00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe1c806ad20 0x7fe1c806d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4007fd0 tx=0x7fe1d4009040 comp rx=0 tx=0).stop 2024-03-29T10:10:08.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.074+0000 7fe1efe00700 1 -- 172.21.15.12:0/4199319601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80b3930 msgr2=0x7fe1e80cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.074+0000 7fe1efe00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80b3930 0x7fe1e80cb140 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00e870 tx=0x7fe1dc00ec30 comp rx=0 tx=0).stop 2024-03-29T10:10:08.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.075+0000 7fe1efe00700 1 -- 172.21.15.12:0/4199319601 shutdown_connections 2024-03-29T10:10:08.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.075+0000 7fe1efe00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe1c806ad20 0x7fe1c806d1e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.075+0000 7fe1efe00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1e815db10 0x7fe1e814d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.075+0000 7fe1efe00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e80b3930 0x7fe1e80cb140 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.075+0000 7fe1efe00700 1 --2- 172.21.15.12:0/4199319601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe1e80cc280 0x7fe1e814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.075+0000 7fe1efe00700 1 -- 172.21.15.12:0/4199319601 >> 172.21.15.12:0/4199319601 conn(0x7fe1e80b2e10 msgr2=0x7fe1e815c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:08.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.075+0000 7fe1efe00700 1 -- 172.21.15.12:0/4199319601 shutdown_connections 2024-03-29T10:10:08.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.075+0000 7fe1efe00700 1 -- 172.21.15.12:0/4199319601 wait complete. 2024-03-29T10:10:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T10:10:08.096 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T10:10:08.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.298+0000 7fcf27400700 1 Processor -- start 2024-03-29T10:10:08.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.298+0000 7fcf27400700 1 -- start start 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.299+0000 7fcf27400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcf2015d9d0 0x7fcf200d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.299+0000 7fcf27400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf200d2540 0x7fcf200ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.299+0000 7fcf27400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf20161420 con 0x7fcf200d16a0 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.299+0000 7fcf27400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2015edc0 con 0x7fcf2015d9d0 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.299+0000 7fcf27400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf20161700 con 0x7fcf200d2540 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.300+0000 7fcf24a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcf2015d9d0 0x7fcf200d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.300+0000 7fcf24a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcf2015d9d0 0x7fcf200d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55562/0 (socket says 172.21.15.12:55562) 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.300+0000 7fcf24a00700 1 -- 172.21.15.12:0/946193170 learned_addr learned my addr 172.21.15.12:0/946193170 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.300+0000 7fcf24a00700 1 -- 172.21.15.12:0/946193170 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf200d2540 msgr2=0x7fcf200ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.300+0000 7fcf24a00700 1 --2- 172.21.15.12:0/946193170 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf200d2540 0x7fcf200ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.300+0000 7fcf24a00700 1 -- 172.21.15.12:0/946193170 >> v1:172.21.15.12:6789/0 conn(0x7fcf200d16a0 legacy=0x7fcf200d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.300+0000 7fcf24a00700 1 -- 172.21.15.12:0/946193170 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf201619e0 con 0x7fcf2015d9d0 2024-03-29T10:10:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.300+0000 7fcf24a00700 1 --2- 172.21.15.12:0/946193170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 0x7fcf200d3c30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fcf140098f0 tx=0x7fcf14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530d7a46e7790950 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf1f400700 1 -- 172.21.15.12:0/946193170 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1400b8a0 con 0x7fcf2015d9d0 2024-03-29T10:10:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf1f400700 1 -- 172.21.15.12:0/946193170 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1400ba00 con 0x7fcf2015d9d0 2024-03-29T10:10:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf1f400700 1 -- 172.21.15.12:0/946193170 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1400bd90 con 0x7fcf2015d9d0 2024-03-29T10:10:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf27400700 1 -- 172.21.15.12:0/946193170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 msgr2=0x7fcf200d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf27400700 1 --2- 172.21.15.12:0/946193170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 0x7fcf200d3c30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fcf140098f0 tx=0x7fcf14009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf27400700 1 -- 172.21.15.12:0/946193170 shutdown_connections 2024-03-29T10:10:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf27400700 1 --2- 172.21.15.12:0/946193170 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf200d2540 0x7fcf200ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf27400700 1 --2- 172.21.15.12:0/946193170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 0x7fcf200d3c30 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf27400700 1 -- 172.21.15.12:0/946193170 >> 172.21.15.12:0/946193170 conn(0x7fcf2008aff0 msgr2=0x7fcf200d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf27400700 1 -- 172.21.15.12:0/946193170 shutdown_connections 2024-03-29T10:10:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.301+0000 7fcf27400700 1 -- 172.21.15.12:0/946193170 wait complete. 2024-03-29T10:10:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.303+0000 7fcf27400700 1 Processor -- start 2024-03-29T10:10:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.303+0000 7fcf27400700 1 -- start start 2024-03-29T10:10:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.304+0000 7fcf27400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf200d16a0 0x7fcf201f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.304+0000 7fcf27400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf200d2540 0x7fcf20158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.304+0000 7fcf27400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 0x7fcf200b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.304+0000 7fcf27400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf20156ed0 con 0x7fcf2015d9d0 2024-03-29T10:10:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.304+0000 7fcf27400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf201562c0 con 0x7fcf200d16a0 2024-03-29T10:10:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.304+0000 7fcf27400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf201565a0 con 0x7fcf200d2540 2024-03-29T10:10:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.304+0000 7fcf25400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 0x7fcf200b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.304+0000 7fcf25400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 0x7fcf200b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55568/0 (socket says 172.21.15.12:55568) 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.304+0000 7fcf25400700 1 -- 172.21.15.12:0/165295831 learned_addr learned my addr 172.21.15.12:0/165295831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf24a00700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf200d16a0 0x7fcf201f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf1fe00700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf200d2540 0x7fcf20158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf25400700 1 -- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf200d2540 msgr2=0x7fcf20158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf25400700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf200d2540 0x7fcf20158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf25400700 1 -- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf200d16a0 msgr2=0x7fcf201f3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf25400700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf200d16a0 0x7fcf201f3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf25400700 1 -- 172.21.15.12:0/165295831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf14009550 con 0x7fcf2015d9d0 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf25400700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 0x7fcf200b00d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800be50 tx=0x7fcf18009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf1cc00700 1 -- 172.21.15.12:0/165295831 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1800cc90 con 0x7fcf2015d9d0 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf1cc00700 1 -- 172.21.15.12:0/165295831 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1800cdf0 con 0x7fcf2015d9d0 2024-03-29T10:10:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf27400700 1 -- 172.21.15.12:0/165295831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf20156880 con 0x7fcf2015d9d0 2024-03-29T10:10:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf1cc00700 1 -- 172.21.15.12:0/165295831 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18014680 con 0x7fcf2015d9d0 2024-03-29T10:10:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.305+0000 7fcf27400700 1 -- 172.21.15.12:0/165295831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf20158c60 con 0x7fcf2015d9d0 2024-03-29T10:10:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.306+0000 7fcf27400700 1 -- 172.21.15.12:0/165295831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf200d1ca0 con 0x7fcf2015d9d0 2024-03-29T10:10:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.307+0000 7fcf1cc00700 1 -- 172.21.15.12:0/165295831 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcf1801d3b0 con 0x7fcf2015d9d0 2024-03-29T10:10:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.308+0000 7fcf1cc00700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcf0006adb0 0x7fcf0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.308+0000 7fcf1cc00700 1 -- 172.21.15.12:0/165295831 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5545+0+0 (secure 0 0 0) 0x7fcf18006e40 con 0x7fcf2015d9d0 2024-03-29T10:10:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.309+0000 7fcf24a00700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcf0006adb0 0x7fcf0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.309+0000 7fcf24a00700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcf0006adb0 0x7fcf0006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcf14014040 tx=0x7fcf14008030 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:08.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.313+0000 7fcf1cc00700 1 -- 172.21.15.12:0/165295831 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf18054580 con 0x7fcf2015d9d0 2024-03-29T10:10:08.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.540+0000 7fcf27400700 1 -- 172.21.15.12:0/165295831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fcf20158e50 con 0x7fcf2015d9d0 2024-03-29T10:10:08.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.544+0000 7fcf1cc00700 1 -- 172.21.15.12:0/165295831 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcf18057ba0 con 0x7fcf2015d9d0 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 -- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcf0006adb0 msgr2=0x7fcf0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcf0006adb0 0x7fcf0006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcf14014040 tx=0x7fcf14008030 comp rx=0 tx=0).stop 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 -- 172.21.15.12:0/165295831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 msgr2=0x7fcf200b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 0x7fcf200b00d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800be50 tx=0x7fcf18009f00 comp rx=0 tx=0).stop 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 -- 172.21.15.12:0/165295831 shutdown_connections 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcf0006adb0 0x7fcf0006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf2015d9d0 0x7fcf200b00d0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf200d16a0 0x7fcf201f3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 --2- 172.21.15.12:0/165295831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf200d2540 0x7fcf20158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 -- 172.21.15.12:0/165295831 >> 172.21.15.12:0/165295831 conn(0x7fcf2008aff0 msgr2=0x7fcf200cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 -- 172.21.15.12:0/165295831 shutdown_connections 2024-03-29T10:10:08.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.547+0000 7fcf27400700 1 -- 172.21.15.12:0/165295831 wait complete. 2024-03-29T10:10:08.564 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T10:10:08.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.764+0000 7fa951000700 1 Processor -- start 2024-03-29T10:10:08.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.765+0000 7fa951000700 1 -- start start 2024-03-29T10:10:08.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa951000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa94c0ce230 0x7fa94c0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa951000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa94c0b2070 0x7fa94c0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa951000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94c0709c0 con 0x7fa94c15d800 2024-03-29T10:10:08.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa951000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa94c0b40f0 con 0x7fa94c0ce230 2024-03-29T10:10:08.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa951000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c070c80 con 0x7fa94c0b2070 2024-03-29T10:10:08.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa949e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa94c15d800 0x7fa94c0725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36404/0 (socket says 172.21.15.12:36404) 2024-03-29T10:10:08.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa949e00700 1 -- 172.21.15.12:0/3344380002 learned_addr learned my addr 172.21.15.12:0/3344380002 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:08.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa94a800700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa94c0b2070 0x7fa94c0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264516091 0 0) 0x7fa94c0709c0 con 0x7fa94c15d800 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa949400700 1 --2- 172.21.15.12:0/3344380002 >> v2:172.21.15.159:3300/0 conn(0x7fa94c0ce230 0x7fa94c0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.766+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9300035b0 con 0x7fa94c15d800 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1442762827 0 0) 0x7fa9300035b0 con 0x7fa94c15d800 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa94c0b2070 msgr2=0x7fa94c0b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa94c0b2070 0x7fa94c0b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 >> v2:172.21.15.159:3300/0 conn(0x7fa94c0ce230 msgr2=0x7fa94c0ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 --2- 172.21.15.12:0/3344380002 >> v2:172.21.15.159:3300/0 conn(0x7fa94c0ce230 0x7fa94c0ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c070f60 con 0x7fa94c15d800 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fa93c001670 con 0x7fa94c15d800 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 >> v1:172.21.15.12:6789/0 conn(0x7fa94c15d800 legacy=0x7fa94c0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa930003a00 0x7fa930003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa94c0b2070 0x7fa930004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa930004d90 0x7fa930005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c0b40f0 con 0x7fa930003a00 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c070c80 con 0x7fa930004d90 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93000ba10 con 0x7fa94c0b2070 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa949e00700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa930003a00 0x7fa930003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa949400700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa94c0b2070 0x7fa930004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.767+0000 7fa94a800700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa930004d90 0x7fa930005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.768+0000 7fa949e00700 1 -- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa94c0b2070 msgr2=0x7fa930004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.768+0000 7fa949e00700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa94c0b2070 0x7fa930004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.768+0000 7fa949e00700 1 -- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa930004d90 msgr2=0x7fa930005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.768+0000 7fa949e00700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa930004d90 0x7fa930005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.768+0000 7fa949e00700 1 -- 172.21.15.12:0/3344380002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9300035b0 con 0x7fa930003a00 2024-03-29T10:10:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.768+0000 7fa949e00700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa930003a00 0x7fa930003e20 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00c190 tx=0x7fa930005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4071461899edb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.768+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c007b70 con 0x7fa930003a00 2024-03-29T10:10:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.768+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c00cd30 con 0x7fa930003a00 2024-03-29T10:10:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.768+0000 7fa948a00700 1 -- 172.21.15.12:0/3344380002 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c00cea0 con 0x7fa930003a00 2024-03-29T10:10:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.769+0000 7fa951000700 1 -- 172.21.15.12:0/3344380002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa930003a00 msgr2=0x7fa930003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.769+0000 7fa951000700 1 --2- 172.21.15.12:0/3344380002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa930003a00 0x7fa930003e20 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00c190 tx=0x7fa930005880 comp rx=0 tx=0).stop 2024-03-29T10:10:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.769+0000 7fa94a800700 1 -- 172.21.15.12:0/3344380002 reap_dead start 2024-03-29T10:10:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.769+0000 7fa951000700 1 -- 172.21.15.12:0/3344380002 shutdown_connections 2024-03-29T10:10:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.769+0000 7fa951000700 1 -- 172.21.15.12:0/3344380002 >> 172.21.15.12:0/3344380002 conn(0x7fa94c0b0d70 msgr2=0x7fa94c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.769+0000 7fa951000700 1 -- 172.21.15.12:0/3344380002 shutdown_connections 2024-03-29T10:10:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.769+0000 7fa951000700 1 -- 172.21.15.12:0/3344380002 wait complete. 2024-03-29T10:10:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.771+0000 7fa951000700 1 Processor -- start 2024-03-29T10:10:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.771+0000 7fa951000700 1 -- start start 2024-03-29T10:10:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa951000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa930003a00 0x7fa94c14b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa951000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa94c1547a0 0x7fa94c154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa951000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa94c06dac0 0x7fa94c06df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa951000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c06e480 con 0x7fa94c1547a0 2024-03-29T10:10:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa951000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c15c1a0 con 0x7fa94c06dac0 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa951000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c152f60 con 0x7fa930003a00 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa949400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa94c1547a0 0x7fa94c154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa949400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa94c1547a0 0x7fa94c154c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55586/0 (socket says 172.21.15.12:55586) 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa949400700 1 -- 172.21.15.12:0/1952048235 learned_addr learned my addr 172.21.15.12:0/1952048235 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa949400700 1 -- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa930003a00 msgr2=0x7fa94c14b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa949400700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa930003a00 0x7fa94c14b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa949400700 1 -- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa94c06dac0 msgr2=0x7fa94c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa949400700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa94c06dac0 0x7fa94c06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa949400700 1 -- 172.21.15.12:0/1952048235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c153240 con 0x7fa94c1547a0 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.772+0000 7fa949400700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa94c1547a0 0x7fa94c154c20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa938011e60 tx=0x7fa938010720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.773+0000 7fa942000700 1 -- 172.21.15.12:0/1952048235 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93801a640 con 0x7fa94c1547a0 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.773+0000 7fa942000700 1 -- 172.21.15.12:0/1952048235 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93800be90 con 0x7fa94c1547a0 2024-03-29T10:10:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.773+0000 7fa942000700 1 -- 172.21.15.12:0/1952048235 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93801aa00 con 0x7fa94c1547a0 2024-03-29T10:10:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.773+0000 7fa951000700 1 -- 172.21.15.12:0/1952048235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94c153520 con 0x7fa94c1547a0 2024-03-29T10:10:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.773+0000 7fa951000700 1 -- 172.21.15.12:0/1952048235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94c153ae0 con 0x7fa94c1547a0 2024-03-29T10:10:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.774+0000 7fa951000700 1 -- 172.21.15.12:0/1952048235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94c153ca0 con 0x7fa94c1547a0 2024-03-29T10:10:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.775+0000 7fa942000700 1 -- 172.21.15.12:0/1952048235 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa93801ab60 con 0x7fa94c1547a0 2024-03-29T10:10:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.776+0000 7fa942000700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa92c06adb0 0x7fa92c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.776+0000 7fa942000700 1 -- 172.21.15.12:0/1952048235 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5545+0+0 (secure 0 0 0) 0x7fa93808a0c0 con 0x7fa94c1547a0 2024-03-29T10:10:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.776+0000 7fa949e00700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa92c06adb0 0x7fa92c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.777+0000 7fa949e00700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa92c06adb0 0x7fa92c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa94c06e7f0 tx=0x7fa93c00f000 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.781+0000 7fa942000700 1 -- 172.21.15.12:0/1952048235 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa938055230 con 0x7fa94c1547a0 2024-03-29T10:10:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:08.996+0000 7fa951000700 1 -- 172.21.15.12:0/1952048235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa94c153e50 con 0x7fa94c1547a0 2024-03-29T10:10:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.002+0000 7fa942000700 1 -- 172.21.15.12:0/1952048235 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa938058850 con 0x7fa94c1547a0 2024-03-29T10:10:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 -- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa92c06adb0 msgr2=0x7fa92c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa92c06adb0 0x7fa92c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa94c06e7f0 tx=0x7fa93c00f000 comp rx=0 tx=0).stop 2024-03-29T10:10:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 -- 172.21.15.12:0/1952048235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa94c1547a0 msgr2=0x7fa94c154c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa94c1547a0 0x7fa94c154c20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa938011e60 tx=0x7fa938010720 comp rx=0 tx=0).stop 2024-03-29T10:10:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 -- 172.21.15.12:0/1952048235 shutdown_connections 2024-03-29T10:10:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa92c06adb0 0x7fa92c06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa94c06dac0 0x7fa94c06df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa930003a00 0x7fa94c14b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 --2- 172.21.15.12:0/1952048235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa94c1547a0 0x7fa94c154c20 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 -- 172.21.15.12:0/1952048235 >> 172.21.15.12:0/1952048235 conn(0x7fa94c0b0d70 msgr2=0x7fa94c156990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 -- 172.21.15.12:0/1952048235 shutdown_connections 2024-03-29T10:10:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.005+0000 7fa951000700 1 -- 172.21.15.12:0/1952048235 wait complete. 2024-03-29T10:10:09.023 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T10:10:09.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.223+0000 7f85b9200700 1 Processor -- start 2024-03-29T10:10:09.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.223+0000 7f85b9200700 1 -- start start 2024-03-29T10:10:09.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.224+0000 7f85b9200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b40d16a0 0x7f85b40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.224+0000 7f85b9200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b4161420 con 0x7f85b40d2370 2024-03-29T10:10:09.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.224+0000 7f85b9200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b4161700 con 0x7f85b415d9d0 2024-03-29T10:10:09.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.224+0000 7f85b9200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40d2820 con 0x7f85b40d16a0 2024-03-29T10:10:09.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.224+0000 7f85b2800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f85b40d2370 0x7f85b40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51822/0 (socket says 172.21.15.12:51822) 2024-03-29T10:10:09.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.224+0000 7f85b2800700 1 -- 172.21.15.12:0/1959544710 learned_addr learned my addr 172.21.15.12:0/1959544710 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:09.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.225+0000 7f85b1400700 1 --2- 172.21.15.12:0/1959544710 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b40d16a0 0x7f85b40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.225+0000 7f85b0a00700 1 -- 172.21.15.12:0/1959544710 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803193720 0 0) 0x7f85b4161420 con 0x7f85b40d2370 2024-03-29T10:10:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.225+0000 7f85b0a00700 1 -- 172.21.15.12:0/1959544710 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85980035b0 con 0x7f85b40d2370 2024-03-29T10:10:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.225+0000 7f85b1400700 1 -- 172.21.15.12:0/1959544710 >> v1:172.21.15.159:6789/0 conn(0x7f85b415d9d0 legacy=0x7f85b40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.225+0000 7f85b0a00700 1 -- 172.21.15.12:0/1959544710 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707523690 0 0) 0x7f85b4161700 con 0x7f85b415d9d0 2024-03-29T10:10:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.225+0000 7f85b1400700 1 -- 172.21.15.12:0/1959544710 >> v1:172.21.15.12:6789/0 conn(0x7f85b40d2370 legacy=0x7f85b40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.225+0000 7f85b1400700 1 -- 172.21.15.12:0/1959544710 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b41619e0 con 0x7f85b40d16a0 2024-03-29T10:10:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.225+0000 7f85b1400700 1 --2- 172.21.15.12:0/1959544710 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b40d16a0 0x7f85b40d1e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009850 tx=0x7f85a0009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f82bf3c03a79ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:09.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.226+0000 7f85b0a00700 1 -- 172.21.15.12:0/1959544710 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0005e30 con 0x7f85b40d16a0 2024-03-29T10:10:09.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.226+0000 7f85b0a00700 1 -- 172.21.15.12:0/1959544710 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a0006030 con 0x7f85b40d16a0 2024-03-29T10:10:09.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.226+0000 7f85b0a00700 1 -- 172.21.15.12:0/1959544710 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a00063e0 con 0x7f85b40d16a0 2024-03-29T10:10:09.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.226+0000 7f85b9200700 1 -- 172.21.15.12:0/1959544710 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b40d16a0 msgr2=0x7f85b40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.226+0000 7f85b9200700 1 --2- 172.21.15.12:0/1959544710 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b40d16a0 0x7f85b40d1e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009850 tx=0x7f85a0009b60 comp rx=0 tx=0).stop 2024-03-29T10:10:09.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.226+0000 7f85b9200700 1 -- 172.21.15.12:0/1959544710 shutdown_connections 2024-03-29T10:10:09.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.226+0000 7f85b9200700 1 --2- 172.21.15.12:0/1959544710 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b40d16a0 0x7f85b40d1e30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.226+0000 7f85b9200700 1 -- 172.21.15.12:0/1959544710 >> 172.21.15.12:0/1959544710 conn(0x7f85b408aff0 msgr2=0x7f85b40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:09.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.226+0000 7f85b9200700 1 -- 172.21.15.12:0/1959544710 shutdown_connections 2024-03-29T10:10:09.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.226+0000 7f85b9200700 1 -- 172.21.15.12:0/1959544710 wait complete. 2024-03-29T10:10:09.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.228+0000 7f85b9200700 1 Processor -- start 2024-03-29T10:10:09.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.228+0000 7f85b9200700 1 -- start start 2024-03-29T10:10:09.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.229+0000 7f85b9200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85b40d16a0 0x7f85b414f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.229+0000 7f85b9200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85b40d2370 0x7f85b41605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.229+0000 7f85b9200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b415d9d0 0x7f85b415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.229+0000 7f85b9200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40ca5f0 con 0x7f85b40d16a0 2024-03-29T10:10:09.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.229+0000 7f85b9200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40cb450 con 0x7f85b40d2370 2024-03-29T10:10:09.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.229+0000 7f85b9200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40ca840 con 0x7f85b415d9d0 2024-03-29T10:10:09.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.229+0000 7f85b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85b40d16a0 0x7f85b414f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.229+0000 7f85b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85b40d16a0 0x7f85b414f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45474/0 (socket says 172.21.15.12:45474) 2024-03-29T10:10:09.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.229+0000 7f85b1e00700 1 -- 172.21.15.12:0/808846362 learned_addr learned my addr 172.21.15.12:0/808846362 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:09.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.229+0000 7f85b1400700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85b40d2370 0x7f85b41605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b2800700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b415d9d0 0x7f85b415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b1e00700 1 -- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b415d9d0 msgr2=0x7f85b415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b1e00700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b415d9d0 0x7f85b415bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b1e00700 1 -- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85b40d2370 msgr2=0x7f85b41605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b1e00700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85b40d2370 0x7f85b41605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b1e00700 1 -- 172.21.15.12:0/808846362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a0009430 con 0x7f85b40d16a0 2024-03-29T10:10:09.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b2800700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b415d9d0 0x7f85b415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:09.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b1400700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85b40d2370 0x7f85b41605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:09.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b1e00700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85b40d16a0 0x7f85b414f8f0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f85a400c580 tx=0x7f85a400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:09.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85aa000700 1 -- 172.21.15.12:0/808846362 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a40054e0 con 0x7f85b40d16a0 2024-03-29T10:10:09.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85aa000700 1 -- 172.21.15.12:0/808846362 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a4003680 con 0x7f85b40d16a0 2024-03-29T10:10:09.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85aa000700 1 -- 172.21.15.12:0/808846362 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a4003a30 con 0x7f85b40d16a0 2024-03-29T10:10:09.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b9200700 1 -- 172.21.15.12:0/808846362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b40cab20 con 0x7f85b40d16a0 2024-03-29T10:10:09.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.230+0000 7f85b9200700 1 -- 172.21.15.12:0/808846362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b40cb0e0 con 0x7f85b40d16a0 2024-03-29T10:10:09.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.231+0000 7f85b9200700 1 -- 172.21.15.12:0/808846362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b40d1ca0 con 0x7f85b40d16a0 2024-03-29T10:10:09.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.232+0000 7f85aa000700 1 -- 172.21.15.12:0/808846362 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f85a4003b90 con 0x7f85b40d16a0 2024-03-29T10:10:09.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.233+0000 7f85aa000700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f859406adb0 0x7f859406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.233+0000 7f85aa000700 1 -- 172.21.15.12:0/808846362 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f85a4088920 con 0x7f85b40d16a0 2024-03-29T10:10:09.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.234+0000 7f85b1400700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f859406adb0 0x7f859406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.235+0000 7f85b1400700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f859406adb0 0x7f859406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f85b40d1ae0 tx=0x7f85a0008070 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:09.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.238+0000 7f85aa000700 1 -- 172.21.15.12:0/808846362 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85a40539f0 con 0x7f85b40d16a0 2024-03-29T10:10:09.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.452+0000 7f85b9200700 1 -- 172.21.15.12:0/808846362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f85b4067720 con 0x7f85b40d16a0 2024-03-29T10:10:09.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.458+0000 7f85aa000700 1 -- 172.21.15.12:0/808846362 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v27) v1 ==== 120+0+0 (secure 0 0 0) 0x7f85a40170b0 con 0x7f85b40d16a0 2024-03-29T10:10:09.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 -- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f859406adb0 msgr2=0x7f859406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f859406adb0 0x7f859406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f85b40d1ae0 tx=0x7f85a0008070 comp rx=0 tx=0).stop 2024-03-29T10:10:09.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 -- 172.21.15.12:0/808846362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85b40d16a0 msgr2=0x7f85b414f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85b40d16a0 0x7f85b414f8f0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f85a400c580 tx=0x7f85a400c940 comp rx=0 tx=0).stop 2024-03-29T10:10:09.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 -- 172.21.15.12:0/808846362 shutdown_connections 2024-03-29T10:10:09.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f859406adb0 0x7f859406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85b415d9d0 0x7f85b415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85b40d16a0 0x7f85b414f8f0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 --2- 172.21.15.12:0/808846362 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85b40d2370 0x7f85b41605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 -- 172.21.15.12:0/808846362 >> 172.21.15.12:0/808846362 conn(0x7f85b408aff0 msgr2=0x7f85b406c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:09.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 -- 172.21.15.12:0/808846362 shutdown_connections 2024-03-29T10:10:09.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.461+0000 7f85b9200700 1 -- 172.21.15.12:0/808846362 wait complete. 2024-03-29T10:10:09.479 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-03-29T10:10:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.679+0000 7f3b66800700 1 Processor -- start 2024-03-29T10:10:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.679+0000 7f3b66800700 1 -- start start 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.681+0000 7f3b66800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 0x7f3b601612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.681+0000 7f3b66800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b60161800 0x7f3b600b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.681+0000 7f3b66800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b600d30d0 con 0x7f3b6015d640 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.681+0000 7f3b66800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b60161c90 con 0x7f3b60161800 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.681+0000 7f3b66800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b600b6840 con 0x7f3b600ceec0 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.681+0000 7f3b5f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3b6015d640 0x7f3b600d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51826/0 (socket says 172.21.15.12:51826) 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.681+0000 7f3b5f400700 1 -- 172.21.15.12:0/1122103293 learned_addr learned my addr 172.21.15.12:0/1122103293 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.681+0000 7f3b5fe00700 1 --2- 172.21.15.12:0/1122103293 >> v2:172.21.15.12:3300/0 conn(0x7f3b60161800 0x7f3b600b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.682+0000 7f3b5ea00700 1 --2- 172.21.15.12:0/1122103293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 0x7f3b601612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.682+0000 7f3b5fe00700 1 -- 172.21.15.12:0/1122103293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 msgr2=0x7f3b601612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.682+0000 7f3b5fe00700 1 --2- 172.21.15.12:0/1122103293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 0x7f3b601612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.682+0000 7f3b5fe00700 1 -- 172.21.15.12:0/1122103293 >> v1:172.21.15.12:6789/0 conn(0x7f3b6015d640 legacy=0x7f3b600d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.682+0000 7f3b5fe00700 1 -- 172.21.15.12:0/1122103293 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b600d2b30 con 0x7f3b60161800 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.682+0000 7f3b5ea00700 1 --2- 172.21.15.12:0/1122103293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 0x7f3b601612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.682+0000 7f3b5fe00700 1 --2- 172.21.15.12:0/1122103293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b60161800 0x7f3b600b6cd0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009d90 tx=0x7f3b5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a464aa770aac63 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.682+0000 7f3b5e000700 1 -- 172.21.15.12:0/1122103293 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5800e070 con 0x7f3b60161800 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.682+0000 7f3b5e000700 1 -- 172.21.15.12:0/1122103293 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5800bbd0 con 0x7f3b60161800 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.682+0000 7f3b5e000700 1 -- 172.21.15.12:0/1122103293 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b58012430 con 0x7f3b60161800 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.683+0000 7f3b66800700 1 -- 172.21.15.12:0/1122103293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b60161800 msgr2=0x7f3b600b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.683+0000 7f3b66800700 1 --2- 172.21.15.12:0/1122103293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b60161800 0x7f3b600b6cd0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009d90 tx=0x7f3b5800b330 comp rx=0 tx=0).stop 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.683+0000 7f3b66800700 1 -- 172.21.15.12:0/1122103293 shutdown_connections 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.683+0000 7f3b66800700 1 --2- 172.21.15.12:0/1122103293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b60161800 0x7f3b600b6cd0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.683+0000 7f3b66800700 1 --2- 172.21.15.12:0/1122103293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 0x7f3b601612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.683+0000 7f3b66800700 1 -- 172.21.15.12:0/1122103293 >> 172.21.15.12:0/1122103293 conn(0x7f3b6008aff0 msgr2=0x7f3b600d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.683+0000 7f3b66800700 1 -- 172.21.15.12:0/1122103293 shutdown_connections 2024-03-29T10:10:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.683+0000 7f3b66800700 1 -- 172.21.15.12:0/1122103293 wait complete. 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.685+0000 7f3b66800700 1 Processor -- start 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.685+0000 7f3b66800700 1 -- start start 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b66800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 0x7f3b6015a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b66800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b6015d640 0x7f3b600a02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b66800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b60161800 0x7f3b6006cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b66800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6009e680 con 0x7f3b6015d640 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b66800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6015ab50 con 0x7f3b60161800 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b66800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6009da00 con 0x7f3b600ceec0 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b6015d640 0x7f3b600a02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b6015d640 0x7f3b600a02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45496/0 (socket says 172.21.15.12:45496) 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5ea00700 1 -- 172.21.15.12:0/4100175381 learned_addr learned my addr 172.21.15.12:0/4100175381 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5fe00700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b60161800 0x7f3b6006cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5f400700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 0x7f3b6015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5ea00700 1 -- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 msgr2=0x7f3b6015a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5ea00700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 0x7f3b6015a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5ea00700 1 -- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b60161800 msgr2=0x7f3b6006cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5ea00700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b60161800 0x7f3b6006cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5ea00700 1 -- 172.21.15.12:0/4100175381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b58009a70 con 0x7f3b6015d640 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.686+0000 7f3b5f400700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 0x7f3b6015a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.687+0000 7f3b5ea00700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b6015d640 0x7f3b600a02d0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00c7f0 tx=0x7f3b4c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.687+0000 7f3b57400700 1 -- 172.21.15.12:0/4100175381 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c011800 con 0x7f3b6015d640 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.687+0000 7f3b57400700 1 -- 172.21.15.12:0/4100175381 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c011960 con 0x7f3b6015d640 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.687+0000 7f3b57400700 1 -- 172.21.15.12:0/4100175381 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c011cf0 con 0x7f3b6015d640 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.687+0000 7f3b66800700 1 -- 172.21.15.12:0/4100175381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6009dce0 con 0x7f3b6015d640 2024-03-29T10:10:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.687+0000 7f3b66800700 1 -- 172.21.15.12:0/4100175381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6009e2a0 con 0x7f3b6015d640 2024-03-29T10:10:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.688+0000 7f3b66800700 1 -- 172.21.15.12:0/4100175381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b60067720 con 0x7f3b6015d640 2024-03-29T10:10:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.689+0000 7f3b57400700 1 -- 172.21.15.12:0/4100175381 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3b4c018360 con 0x7f3b6015d640 2024-03-29T10:10:09.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.690+0000 7f3b57400700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b4006adb0 0x7f3b4006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:09.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.690+0000 7f3b57400700 1 -- 172.21.15.12:0/4100175381 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f3b4c0894e0 con 0x7f3b6015d640 2024-03-29T10:10:09.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.691+0000 7f3b5f400700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b4006adb0 0x7f3b4006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:09.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.691+0000 7f3b5f400700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b4006adb0 0x7f3b4006d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3b50007fd0 tx=0x7f3b50009040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:09.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.694+0000 7f3b57400700 1 -- 172.21.15.12:0/4100175381 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3b4c054550 con 0x7f3b6015d640 2024-03-29T10:10:09.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:09.995+0000 7f3b66800700 1 -- 172.21.15.12:0/4100175381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f3b600b5c70 con 0x7f3b6015d640 2024-03-29T10:10:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.075+0000 7f3b57400700 1 -- 172.21.15.12:0/4100175381 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v576) v1 ==== 156+0+0 (secure 0 0 0) 0x7f3b4c057b70 con 0x7f3b6015d640 2024-03-29T10:10:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 -- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b4006adb0 msgr2=0x7f3b4006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b4006adb0 0x7f3b4006d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3b50007fd0 tx=0x7f3b50009040 comp rx=0 tx=0).stop 2024-03-29T10:10:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 -- 172.21.15.12:0/4100175381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b6015d640 msgr2=0x7f3b600a02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b6015d640 0x7f3b600a02d0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00c7f0 tx=0x7f3b4c00cb00 comp rx=0 tx=0).stop 2024-03-29T10:10:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 -- 172.21.15.12:0/4100175381 shutdown_connections 2024-03-29T10:10:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b4006adb0 0x7f3b4006d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b60161800 0x7f3b6006cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b600ceec0 0x7f3b6015a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 --2- 172.21.15.12:0/4100175381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b6015d640 0x7f3b600a02d0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 -- 172.21.15.12:0/4100175381 >> 172.21.15.12:0/4100175381 conn(0x7f3b6008aff0 msgr2=0x7f3b600b7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 -- 172.21.15.12:0/4100175381 shutdown_connections 2024-03-29T10:10:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.078+0000 7f3b66800700 1 -- 172.21.15.12:0/4100175381 wait complete. 2024-03-29T10:10:10.095 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2024-03-29T10:10:10.096 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2024-03-29T10:10:10.096 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T10:10:10.096 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-03-29T10:10:10.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.293+0000 7fd2cc200700 1 Processor -- start 2024-03-29T10:10:10.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.294+0000 7fd2cc200700 1 -- start start 2024-03-29T10:10:10.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.295+0000 7fd2cc200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd2c4160f10 0x7fd2c40d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:10.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.295+0000 7fd2cc200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd2c40cf230 0x7fd2c40d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:10.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.295+0000 7fd2cc200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40ceda0 con 0x7fd2c4161590 2024-03-29T10:10:10.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.295+0000 7fd2cc200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c40d23d0 con 0x7fd2c40cf230 2024-03-29T10:10:10.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.295+0000 7fd2cc200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c41612f0 con 0x7fd2c4160f10 2024-03-29T10:10:10.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.295+0000 7fd2c8e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd2c40cf230 0x7fd2c40d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:10.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.295+0000 7fd2c8e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd2c40cf230 0x7fd2c40d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45502/0 (socket says 172.21.15.12:45502) 2024-03-29T10:10:10.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.295+0000 7fd2c8e00700 1 -- 172.21.15.12:0/4253660939 learned_addr learned my addr 172.21.15.12:0/4253660939 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:10.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.295+0000 7fd2c9800700 1 --2- 172.21.15.12:0/4253660939 >> v2:172.21.15.159:3300/0 conn(0x7fd2c4160f10 0x7fd2c40d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:10.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.296+0000 7fd2c8e00700 1 -- 172.21.15.12:0/4253660939 >> v2:172.21.15.159:3300/0 conn(0x7fd2c4160f10 msgr2=0x7fd2c40d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:10.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.296+0000 7fd2c8e00700 1 --2- 172.21.15.12:0/4253660939 >> v2:172.21.15.159:3300/0 conn(0x7fd2c4160f10 0x7fd2c40d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:10.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.296+0000 7fd2c8e00700 1 -- 172.21.15.12:0/4253660939 >> v1:172.21.15.159:6789/0 conn(0x7fd2c4161590 legacy=0x7fd2c40b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:10.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.296+0000 7fd2c8e00700 1 -- 172.21.15.12:0/4253660939 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40ce470 con 0x7fd2c40cf230 2024-03-29T10:10:10.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.296+0000 7fd2c9800700 1 --2- 172.21.15.12:0/4253660939 >> v2:172.21.15.159:3300/0 conn(0x7fd2c4160f10 0x7fd2c40d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:10.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.296+0000 7fd2c8e00700 1 --2- 172.21.15.12:0/4253660939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 0x7fd2c40d1e90 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40d1080 tx=0x7fd2b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e337eb27cb82eb1f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:10.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.296+0000 7fd2c3e00700 1 -- 172.21.15.12:0/4253660939 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b000b870 con 0x7fd2c40cf230 2024-03-29T10:10:10.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.296+0000 7fd2c3e00700 1 -- 172.21.15.12:0/4253660939 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b000b9d0 con 0x7fd2c40cf230 2024-03-29T10:10:10.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.296+0000 7fd2c3e00700 1 -- 172.21.15.12:0/4253660939 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b000bd60 con 0x7fd2c40cf230 2024-03-29T10:10:10.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.297+0000 7fd2cc200700 1 -- 172.21.15.12:0/4253660939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 msgr2=0x7fd2c40d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:10.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.297+0000 7fd2cc200700 1 --2- 172.21.15.12:0/4253660939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 0x7fd2c40d1e90 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40d1080 tx=0x7fd2b0009bd0 comp rx=0 tx=0).stop 2024-03-29T10:10:10.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.297+0000 7fd2cc200700 1 -- 172.21.15.12:0/4253660939 shutdown_connections 2024-03-29T10:10:10.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.297+0000 7fd2cc200700 1 --2- 172.21.15.12:0/4253660939 >> v2:172.21.15.159:3300/0 conn(0x7fd2c4160f10 0x7fd2c40d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:10.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.297+0000 7fd2cc200700 1 --2- 172.21.15.12:0/4253660939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 0x7fd2c40d1e90 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:10.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.297+0000 7fd2cc200700 1 -- 172.21.15.12:0/4253660939 >> 172.21.15.12:0/4253660939 conn(0x7fd2c408aff0 msgr2=0x7fd2c40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:10.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.297+0000 7fd2cc200700 1 -- 172.21.15.12:0/4253660939 shutdown_connections 2024-03-29T10:10:10.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.297+0000 7fd2cc200700 1 -- 172.21.15.12:0/4253660939 wait complete. 2024-03-29T10:10:10.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.299+0000 7fd2cc200700 1 Processor -- start 2024-03-29T10:10:10.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.299+0000 7fd2cc200700 1 -- start start 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2cc200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 0x7fd2c4156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2cc200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2c4160f10 0x7fd2c41484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2cc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2c4161590 0x7fd2c41583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2cc200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c4159200 con 0x7fd2c40cf230 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2cc200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41594e0 con 0x7fd2c4160f10 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2cc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41597c0 con 0x7fd2c4161590 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2c9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 0x7fd2c4156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2c9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 0x7fd2c4156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45514/0 (socket says 172.21.15.12:45514) 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2c9800700 1 -- 172.21.15.12:0/4268000811 learned_addr learned my addr 172.21.15.12:0/4268000811 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2c8e00700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2c4160f10 0x7fd2c41484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.300+0000 7fd2ca200700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2c4161590 0x7fd2c41583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2c9800700 1 -- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2c4161590 msgr2=0x7fd2c41583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2c9800700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2c4161590 0x7fd2c41583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2c9800700 1 -- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2c4160f10 msgr2=0x7fd2c41484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2c9800700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2c4160f10 0x7fd2c41484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2c9800700 1 -- 172.21.15.12:0/4268000811 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b0009550 con 0x7fd2c40cf230 2024-03-29T10:10:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2c9800700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 0x7fd2c4156940 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800e760 tx=0x7fd2b800ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2c1600700 1 -- 172.21.15.12:0/4268000811 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b800c070 con 0x7fd2c40cf230 2024-03-29T10:10:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2c1600700 1 -- 172.21.15.12:0/4268000811 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b8013610 con 0x7fd2c40cf230 2024-03-29T10:10:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2c1600700 1 -- 172.21.15.12:0/4268000811 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b80139c0 con 0x7fd2c40cf230 2024-03-29T10:10:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2cc200700 1 -- 172.21.15.12:0/4268000811 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c4159aa0 con 0x7fd2c40cf230 2024-03-29T10:10:10.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.301+0000 7fd2cc200700 1 -- 172.21.15.12:0/4268000811 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c4148a00 con 0x7fd2c40cf230 2024-03-29T10:10:10.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.302+0000 7fd2cc200700 1 -- 172.21.15.12:0/4268000811 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c4148c50 con 0x7fd2c40cf230 2024-03-29T10:10:10.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.303+0000 7fd2c1600700 1 -- 172.21.15.12:0/4268000811 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd2b8013b20 con 0x7fd2c40cf230 2024-03-29T10:10:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.304+0000 7fd2c1600700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2a406adb0 0x7fd2a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.304+0000 7fd2c1600700 1 -- 172.21.15.12:0/4268000811 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5545+0+0 (secure 0 0 0) 0x7fd2b8089240 con 0x7fd2c40cf230 2024-03-29T10:10:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.305+0000 7fd2c8e00700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2a406adb0 0x7fd2a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.306+0000 7fd2c8e00700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2a406adb0 0x7fd2a406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0000c00 tx=0x7fd2b0008030 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:10.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.309+0000 7fd2c1600700 1 -- 172.21.15.12:0/4268000811 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd2b8054290 con 0x7fd2c40cf230 2024-03-29T10:10:10.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:10.605+0000 7fd2cc200700 1 -- 172.21.15.12:0/4268000811 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fd2c4159c90 con 0x7fd2c40cf230 2024-03-29T10:10:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.082+0000 7fd2c1600700 1 -- 172.21.15.12:0/4268000811 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v577) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd2b80578b0 con 0x7fd2c40cf230 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 -- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2a406adb0 msgr2=0x7fd2a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2a406adb0 0x7fd2a406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0000c00 tx=0x7fd2b0008030 comp rx=0 tx=0).stop 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 -- 172.21.15.12:0/4268000811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 msgr2=0x7fd2c4156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 0x7fd2c4156940 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800e760 tx=0x7fd2b800ea70 comp rx=0 tx=0).stop 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 -- 172.21.15.12:0/4268000811 shutdown_connections 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd2a406adb0 0x7fd2a406d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2c4161590 0x7fd2c41583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2c40cf230 0x7fd2c4156940 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 --2- 172.21.15.12:0/4268000811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2c4160f10 0x7fd2c41484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 -- 172.21.15.12:0/4268000811 >> 172.21.15.12:0/4268000811 conn(0x7fd2c408aff0 msgr2=0x7fd2c40cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 -- 172.21.15.12:0/4268000811 shutdown_connections 2024-03-29T10:10:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.085+0000 7fd2cc200700 1 -- 172.21.15.12:0/4268000811 wait complete. 2024-03-29T10:10:11.104 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T10:10:11.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.306+0000 7f195a400700 1 Processor -- start 2024-03-29T10:10:11.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.307+0000 7f195a400700 1 -- start start 2024-03-29T10:10:11.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f195a400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f195415d9d0 0x7f19540d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:11.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f195a400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f19540d2540 0x7f19540ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:11.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f195a400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1954161420 con 0x7f19540d16a0 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f195a400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195415edc0 con 0x7f195415d9d0 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f195a400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1954161700 con 0x7f19540d2540 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f1953400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f195415d9d0 0x7f19540d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f1953400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f195415d9d0 0x7f19540d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45520/0 (socket says 172.21.15.12:45520) 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f1953400700 1 -- 172.21.15.12:0/3723674663 learned_addr learned my addr 172.21.15.12:0/3723674663 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f1953400700 1 -- 172.21.15.12:0/3723674663 >> v2:172.21.15.159:3300/0 conn(0x7f19540d2540 msgr2=0x7f19540ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f1953400700 1 --2- 172.21.15.12:0/3723674663 >> v2:172.21.15.159:3300/0 conn(0x7f19540d2540 0x7f19540ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f1953400700 1 -- 172.21.15.12:0/3723674663 >> v1:172.21.15.159:6789/0 conn(0x7f19540d16a0 legacy=0x7f19540d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.308+0000 7f1953400700 1 -- 172.21.15.12:0/3723674663 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19541619e0 con 0x7f195415d9d0 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.309+0000 7f1953400700 1 --2- 172.21.15.12:0/3723674663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 0x7f19540d3c30 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f195408ca10 tx=0x7f1944009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967be3c0e55dfeb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.309+0000 7f1952000700 1 -- 172.21.15.12:0/3723674663 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1944004230 con 0x7f195415d9d0 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.309+0000 7f1952000700 1 -- 172.21.15.12:0/3723674663 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1944004390 con 0x7f195415d9d0 2024-03-29T10:10:11.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.309+0000 7f1952000700 1 -- 172.21.15.12:0/3723674663 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19440036a0 con 0x7f195415d9d0 2024-03-29T10:10:11.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.309+0000 7f195a400700 1 -- 172.21.15.12:0/3723674663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 msgr2=0x7f19540d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:11.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.309+0000 7f195a400700 1 --2- 172.21.15.12:0/3723674663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 0x7f19540d3c30 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f195408ca10 tx=0x7f1944009bd0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.309+0000 7f195a400700 1 -- 172.21.15.12:0/3723674663 shutdown_connections 2024-03-29T10:10:11.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.309+0000 7f195a400700 1 --2- 172.21.15.12:0/3723674663 >> v2:172.21.15.159:3300/0 conn(0x7f19540d2540 0x7f19540ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.309+0000 7f195a400700 1 --2- 172.21.15.12:0/3723674663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 0x7f19540d3c30 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.310+0000 7f195a400700 1 -- 172.21.15.12:0/3723674663 >> 172.21.15.12:0/3723674663 conn(0x7f195408aff0 msgr2=0x7f19540d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:11.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.310+0000 7f195a400700 1 -- 172.21.15.12:0/3723674663 shutdown_connections 2024-03-29T10:10:11.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.310+0000 7f195a400700 1 -- 172.21.15.12:0/3723674663 wait complete. 2024-03-29T10:10:11.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.311+0000 7f195a400700 1 Processor -- start 2024-03-29T10:10:11.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.312+0000 7f195a400700 1 -- start start 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.312+0000 7f195a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19540d16a0 0x7f19541581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.312+0000 7f195a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19540d2540 0x7f19540b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.312+0000 7f195a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 0x7f1954155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f195a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1954159a80 con 0x7f195415d9d0 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f195a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19540b82b0 con 0x7f19540d2540 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f195a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19540b8590 con 0x7f19540d16a0 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f1953e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 0x7f1954155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f1953e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 0x7f1954155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45526/0 (socket says 172.21.15.12:45526) 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f1953e00700 1 -- 172.21.15.12:0/994661784 learned_addr learned my addr 172.21.15.12:0/994661784 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f1953400700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19540d16a0 0x7f19541581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f1952a00700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19540d2540 0x7f19540b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f1953e00700 1 -- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19540d16a0 msgr2=0x7f19541581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f1953e00700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19540d16a0 0x7f19541581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f1953e00700 1 -- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19540d2540 msgr2=0x7f19540b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:11.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f1953e00700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19540d2540 0x7f19540b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.313+0000 7f1953e00700 1 -- 172.21.15.12:0/994661784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1944009550 con 0x7f195415d9d0 2024-03-29T10:10:11.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.314+0000 7f1953e00700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 0x7f1954155820 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f194c00b9e0 tx=0x7f194c00bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.314+0000 7f194b400700 1 -- 172.21.15.12:0/994661784 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c00d8e0 con 0x7f195415d9d0 2024-03-29T10:10:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.314+0000 7f194b400700 1 -- 172.21.15.12:0/994661784 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c00da60 con 0x7f195415d9d0 2024-03-29T10:10:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.314+0000 7f194b400700 1 -- 172.21.15.12:0/994661784 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c00de10 con 0x7f195415d9d0 2024-03-29T10:10:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.314+0000 7f195a400700 1 -- 172.21.15.12:0/994661784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19540b8870 con 0x7f195415d9d0 2024-03-29T10:10:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.314+0000 7f195a400700 1 -- 172.21.15.12:0/994661784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19540b7fc0 con 0x7f195415d9d0 2024-03-29T10:10:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.315+0000 7f195a400700 1 -- 172.21.15.12:0/994661784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19540d1ca0 con 0x7f195415d9d0 2024-03-29T10:10:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.316+0000 7f194b400700 1 -- 172.21.15.12:0/994661784 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f194c015420 con 0x7f195415d9d0 2024-03-29T10:10:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.317+0000 7f194b400700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f193406adb0 0x7f193406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.317+0000 7f194b400700 1 -- 172.21.15.12:0/994661784 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f194c011070 con 0x7f195415d9d0 2024-03-29T10:10:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.317+0000 7f1953400700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f193406adb0 0x7f193406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.318+0000 7f1953400700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f193406adb0 0x7f193406d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1944009130 tx=0x7f1944008030 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:11.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.321+0000 7f194b400700 1 -- 172.21.15.12:0/994661784 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f194c04f580 con 0x7f195415d9d0 2024-03-29T10:10:11.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.533+0000 7f195a400700 1 -- 172.21.15.12:0/994661784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1954159bc0 con 0x7f195415d9d0 2024-03-29T10:10:11.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.539+0000 7f194b400700 1 -- 172.21.15.12:0/994661784 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f194c054440 con 0x7f195415d9d0 2024-03-29T10:10:11.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.541+0000 7f195a400700 1 -- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f193406adb0 msgr2=0x7f193406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.541+0000 7f195a400700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f193406adb0 0x7f193406d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1944009130 tx=0x7f1944008030 comp rx=0 tx=0).stop 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.541+0000 7f195a400700 1 -- 172.21.15.12:0/994661784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 msgr2=0x7f1954155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.541+0000 7f195a400700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 0x7f1954155820 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f194c00b9e0 tx=0x7f194c00bcf0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.542+0000 7f195a400700 1 -- 172.21.15.12:0/994661784 shutdown_connections 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.542+0000 7f195a400700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f193406adb0 0x7f193406d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.542+0000 7f195a400700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195415d9d0 0x7f1954155820 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.542+0000 7f195a400700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19540d16a0 0x7f19541581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.542+0000 7f195a400700 1 --2- 172.21.15.12:0/994661784 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19540d2540 0x7f19540b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.542+0000 7f195a400700 1 -- 172.21.15.12:0/994661784 >> 172.21.15.12:0/994661784 conn(0x7f195408aff0 msgr2=0x7f19540b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.542+0000 7f195a400700 1 -- 172.21.15.12:0/994661784 shutdown_connections 2024-03-29T10:10:11.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:11.542+0000 7f195a400700 1 -- 172.21.15.12:0/994661784 wait complete. 2024-03-29T10:10:11.559 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:10:11.559 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:10:11.642 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:10:11.733 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:10:11.733 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:10:11.733 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:11.778 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:10:11.778 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:10:11.778 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:11.824 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:10:11.824 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:10:11.824 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:11.870 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:10:11.870 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:10:11.870 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:11.916 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:10:11.916 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T10:10:11.916 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:12.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.115+0000 7fc477000700 1 Processor -- start 2024-03-29T10:10:12.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.115+0000 7fc477000700 1 -- start start 2024-03-29T10:10:12.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.116+0000 7fc477000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc4700afbe0 0x7fc47006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.116+0000 7fc477000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4700cebd0 0x7fc4700cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.116+0000 7fc477000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47006f850 con 0x7fc47006a790 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.116+0000 7fc477000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc47006fad0 con 0x7fc4700afbe0 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.116+0000 7fc477000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4700af970 con 0x7fc4700cebd0 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.117+0000 7fc46fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc47006a790 0x7fc47006ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51838/0 (socket says 172.21.15.12:51838) 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.117+0000 7fc46fe00700 1 -- 172.21.15.12:0/38479750 learned_addr learned my addr 172.21.15.12:0/38479750 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.117+0000 7fc46f400700 1 --2- 172.21.15.12:0/38479750 >> v2:172.21.15.12:3300/0 conn(0x7fc4700afbe0 0x7fc47006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.117+0000 7fc475000700 1 --2- 172.21.15.12:0/38479750 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4700cebd0 0x7fc4700cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.117+0000 7fc46f400700 1 -- 172.21.15.12:0/38479750 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4700cebd0 msgr2=0x7fc4700cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.117+0000 7fc46f400700 1 --2- 172.21.15.12:0/38479750 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4700cebd0 0x7fc4700cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.117+0000 7fc46f400700 1 -- 172.21.15.12:0/38479750 >> v1:172.21.15.12:6789/0 conn(0x7fc47006a790 legacy=0x7fc47006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.117+0000 7fc46f400700 1 -- 172.21.15.12:0/38479750 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4700af690 con 0x7fc4700afbe0 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.117+0000 7fc46ea00700 1 -- 172.21.15.12:0/38479750 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672997116 0 0) 0x7fc47006f850 con 0x7fc47006a790 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.118+0000 7fc46f400700 1 --2- 172.21.15.12:0/38479750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 0x7fc47006dce0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009840 tx=0x7fc45c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecab07e26e7cf3df server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:12.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.118+0000 7fc46ea00700 1 -- 172.21.15.12:0/38479750 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c011830 con 0x7fc4700afbe0 2024-03-29T10:10:12.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.118+0000 7fc46ea00700 1 -- 172.21.15.12:0/38479750 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45c011990 con 0x7fc4700afbe0 2024-03-29T10:10:12.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.118+0000 7fc46ea00700 1 -- 172.21.15.12:0/38479750 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c011d20 con 0x7fc4700afbe0 2024-03-29T10:10:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.118+0000 7fc477000700 1 -- 172.21.15.12:0/38479750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 msgr2=0x7fc47006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.118+0000 7fc477000700 1 --2- 172.21.15.12:0/38479750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 0x7fc47006dce0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009840 tx=0x7fc45c009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.118+0000 7fc477000700 1 -- 172.21.15.12:0/38479750 shutdown_connections 2024-03-29T10:10:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.118+0000 7fc477000700 1 --2- 172.21.15.12:0/38479750 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc4700cebd0 0x7fc4700cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.118+0000 7fc477000700 1 --2- 172.21.15.12:0/38479750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 0x7fc47006dce0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.118+0000 7fc477000700 1 -- 172.21.15.12:0/38479750 >> 172.21.15.12:0/38479750 conn(0x7fc4700b5ab0 msgr2=0x7fc4700b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.119+0000 7fc477000700 1 -- 172.21.15.12:0/38479750 shutdown_connections 2024-03-29T10:10:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.119+0000 7fc477000700 1 -- 172.21.15.12:0/38479750 wait complete. 2024-03-29T10:10:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.120+0000 7fc477000700 1 Processor -- start 2024-03-29T10:10:12.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.120+0000 7fc477000700 1 -- start start 2024-03-29T10:10:12.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.121+0000 7fc477000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc47006a790 0x7fc47014cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.121+0000 7fc477000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 0x7fc4700ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.121+0000 7fc477000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4700cebd0 0x7fc470156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.121+0000 7fc477000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4700ae860 con 0x7fc4700afbe0 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.121+0000 7fc477000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc470156d10 con 0x7fc4700cebd0 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.121+0000 7fc477000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4700b2ec0 con 0x7fc47006a790 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 0x7fc4700ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 0x7fc4700ca660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45548/0 (socket says 172.21.15.12:45548) 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46fe00700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc47006a790 0x7fc47014cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46f400700 1 -- 172.21.15.12:0/206773548 learned_addr learned my addr 172.21.15.12:0/206773548 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc475000700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4700cebd0 0x7fc470156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46fe00700 1 -- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4700cebd0 msgr2=0x7fc470156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46fe00700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4700cebd0 0x7fc470156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46fe00700 1 -- 172.21.15.12:0/206773548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 msgr2=0x7fc4700ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46fe00700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 0x7fc4700ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46fe00700 1 -- 172.21.15.12:0/206773548 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c0094a0 con 0x7fc47006a790 2024-03-29T10:10:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46f400700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 0x7fc4700ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.122+0000 7fc46fe00700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc47006a790 0x7fc47014cc40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc46000c410 tx=0x7fc46000c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.123+0000 7fc467e00700 1 -- 172.21.15.12:0/206773548 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc460004c80 con 0x7fc47006a790 2024-03-29T10:10:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.123+0000 7fc467e00700 1 -- 172.21.15.12:0/206773548 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc460004de0 con 0x7fc47006a790 2024-03-29T10:10:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.123+0000 7fc477000700 1 -- 172.21.15.12:0/206773548 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47014d180 con 0x7fc47006a790 2024-03-29T10:10:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.123+0000 7fc467e00700 1 -- 172.21.15.12:0/206773548 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc460007310 con 0x7fc47006a790 2024-03-29T10:10:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.123+0000 7fc477000700 1 -- 172.21.15.12:0/206773548 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4700b3440 con 0x7fc47006a790 2024-03-29T10:10:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.124+0000 7fc477000700 1 -- 172.21.15.12:0/206773548 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47006ee70 con 0x7fc47006a790 2024-03-29T10:10:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.125+0000 7fc467e00700 1 -- 172.21.15.12:0/206773548 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc460007470 con 0x7fc47006a790 2024-03-29T10:10:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.125+0000 7fc467e00700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc45006adb0 0x7fc45006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.126+0000 7fc467e00700 1 -- 172.21.15.12:0/206773548 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5545+0+0 (secure 0 0 0) 0x7fc460088ff0 con 0x7fc47006a790 2024-03-29T10:10:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.126+0000 7fc46f400700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc45006adb0 0x7fc45006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.127+0000 7fc46f400700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc45006adb0 0x7fc45006d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc4700ae4c0 tx=0x7fc45c008030 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:12.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.131+0000 7fc467e00700 1 -- 172.21.15.12:0/206773548 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4600540c0 con 0x7fc47006a790 2024-03-29T10:10:12.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.392+0000 7fc477000700 1 -- 172.21.15.12:0/206773548 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc470066ff0 con 0x7fc47006a790 2024-03-29T10:10:12.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.392+0000 7fc467e00700 1 -- 172.21.15.12:0/206773548 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fc4600576e0 con 0x7fc47006a790 2024-03-29T10:10:12.392 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:12.392 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":33,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3202930550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3202930550},{"type":"v1","addr":"172.21.15.159:6835","nonce":3202930550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4189378403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4189378403},{"type":"v1","addr":"172.21.15.12:6834","nonce":4189378403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2681299760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2681299760},{"type":"v1","addr":"172.21.15.12:6835","nonce":2681299760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":6628,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2736755418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":2736755418},{"type":"v1","addr":"172.21.15.159:6837","nonce":2736755418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[]} 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 -- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc45006adb0 msgr2=0x7fc45006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc45006adb0 0x7fc45006d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc4700ae4c0 tx=0x7fc45c008030 comp rx=0 tx=0).stop 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 -- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc47006a790 msgr2=0x7fc47014cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc47006a790 0x7fc47014cc40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc46000c410 tx=0x7fc46000c720 comp rx=0 tx=0).stop 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 -- 172.21.15.12:0/206773548 shutdown_connections 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc45006adb0 0x7fc45006d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4700cebd0 0x7fc470156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc47006a790 0x7fc47014cc40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 --2- 172.21.15.12:0/206773548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4700afbe0 0x7fc4700ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 -- 172.21.15.12:0/206773548 >> 172.21.15.12:0/206773548 conn(0x7fc4700b5ab0 msgr2=0x7fc47015dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:12.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 -- 172.21.15.12:0/206773548 shutdown_connections 2024-03-29T10:10:12.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.395+0000 7fc477000700 1 -- 172.21.15.12:0/206773548 wait complete. 2024-03-29T10:10:12.396 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 33 2024-03-29T10:10:12.413 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T10:10:12.413 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T10:10:12.413 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T10:10:12.414 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:10:12.414 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T10:10:12.414 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T10:10:12.414 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T10:10:12.414 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:10:12.414 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T10:10:12.414 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T10:10:12.415 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T10:10:12.415 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:10:12.415 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T10:10:12.415 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T10:10:12.415 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T10:10:12.415 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:10:12.427 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T10:10:12.427 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T10:10:12.429 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T10:10:12.429 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T10:10:12.432 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T10:10:12.432 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T10:10:12.433 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T10:10:12.433 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T10:10:12.472 INFO:tasks.ceph.mds.a:Started 2024-03-29T10:10:12.472 DEBUG:teuthology.parallel:result is None 2024-03-29T10:10:12.475 INFO:tasks.ceph.mds.c:Started 2024-03-29T10:10:12.475 DEBUG:teuthology.parallel:result is None 2024-03-29T10:10:12.477 INFO:tasks.ceph.mds.b:Started 2024-03-29T10:10:12.477 DEBUG:teuthology.parallel:result is None 2024-03-29T10:10:12.479 INFO:tasks.ceph.mds.d:Started 2024-03-29T10:10:12.479 DEBUG:teuthology.parallel:result is None 2024-03-29T10:10:12.479 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T10:10:12.622 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T10:10:12.622 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:10:12.623 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T10:10:12.623 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T10:10:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.693+0000 7f5935a00700 1 Processor -- start 2024-03-29T10:10:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.693+0000 7f5935a00700 1 -- start start 2024-03-29T10:10:12.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.694+0000 7f5935a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59300b2060 0x7f59300d3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.694+0000 7f5935a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f59300b2440 0x7f5930161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.694+0000 7f5935a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59300affd0 con 0x7f5930161980 2024-03-29T10:10:12.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.694+0000 7f5935a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593006d5f0 con 0x7f59300b2440 2024-03-29T10:10:12.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.694+0000 7f5935a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593006caa0 con 0x7f59300b2060 2024-03-29T10:10:12.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.694+0000 7f592f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f59300b2440 0x7f5930161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.694+0000 7f5935000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5930161980 0x7f593006b7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51860/0 (socket says 172.21.15.12:51860) 2024-03-29T10:10:12.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.694+0000 7f5935000700 1 -- 172.21.15.12:0/745290802 learned_addr learned my addr 172.21.15.12:0/745290802 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:12.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.695+0000 7f592fe00700 1 --2- 172.21.15.12:0/745290802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59300b2060 0x7f59300d3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.695+0000 7f592f400700 1 -- 172.21.15.12:0/745290802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59300b2060 msgr2=0x7f59300d3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:12.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.695+0000 7f592f400700 1 --2- 172.21.15.12:0/745290802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59300b2060 0x7f59300d3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.695+0000 7f592f400700 1 -- 172.21.15.12:0/745290802 >> v1:172.21.15.12:6789/0 conn(0x7f5930161980 legacy=0x7f593006b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:12.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.695+0000 7f592f400700 1 -- 172.21.15.12:0/745290802 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593006cd80 con 0x7f59300b2440 2024-03-29T10:10:12.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.695+0000 7f592ea00700 1 -- 172.21.15.12:0/745290802 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750734256 0 0) 0x7f59300affd0 con 0x7f5930161980 2024-03-29T10:10:12.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.695+0000 7f592f400700 1 --2- 172.21.15.12:0/745290802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 0x7f5930161440 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f591c009840 tx=0x7f591c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e846c4925b0f4898 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:12.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.695+0000 7f592ea00700 1 -- 172.21.15.12:0/745290802 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c011830 con 0x7f59300b2440 2024-03-29T10:10:12.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.695+0000 7f592ea00700 1 -- 172.21.15.12:0/745290802 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c011990 con 0x7f59300b2440 2024-03-29T10:10:12.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.695+0000 7f592ea00700 1 -- 172.21.15.12:0/745290802 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c011d20 con 0x7f59300b2440 2024-03-29T10:10:12.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.696+0000 7f5935a00700 1 -- 172.21.15.12:0/745290802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 msgr2=0x7f5930161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:12.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.696+0000 7f5935a00700 1 --2- 172.21.15.12:0/745290802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 0x7f5930161440 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f591c009840 tx=0x7f591c009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:12.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.696+0000 7f5935a00700 1 -- 172.21.15.12:0/745290802 shutdown_connections 2024-03-29T10:10:12.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.696+0000 7f5935a00700 1 --2- 172.21.15.12:0/745290802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59300b2060 0x7f59300d3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.696+0000 7f5935a00700 1 --2- 172.21.15.12:0/745290802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 0x7f5930161440 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.696+0000 7f5935a00700 1 -- 172.21.15.12:0/745290802 >> 172.21.15.12:0/745290802 conn(0x7f593009c6e0 msgr2=0x7f59300d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:12.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.696+0000 7f5935a00700 1 -- 172.21.15.12:0/745290802 shutdown_connections 2024-03-29T10:10:12.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.696+0000 7f5935a00700 1 -- 172.21.15.12:0/745290802 wait complete. 2024-03-29T10:10:12.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.698+0000 7f5935a00700 1 Processor -- start 2024-03-29T10:10:12.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.698+0000 7f5935a00700 1 -- start start 2024-03-29T10:10:12.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.699+0000 7f5935a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59300b2060 0x7f59300ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.699+0000 7f5935a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 0x7f59300cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.699+0000 7f5935a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5930161980 0x7f593006ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.699+0000 7f5935a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59300caae0 con 0x7f59300b2440 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.699+0000 7f5935a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59300cadc0 con 0x7f5930161980 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.699+0000 7f5935a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59300cb0a0 con 0x7f59300b2060 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.699+0000 7f592f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 0x7f59300cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.699+0000 7f592f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 0x7f59300cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45592/0 (socket says 172.21.15.12:45592) 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.699+0000 7f592f400700 1 -- 172.21.15.12:0/4285797988 learned_addr learned my addr 172.21.15.12:0/4285797988 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f5935000700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5930161980 0x7f593006ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f592fe00700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59300b2060 0x7f59300ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f592f400700 1 -- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59300b2060 msgr2=0x7f59300ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f592f400700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59300b2060 0x7f59300ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f592f400700 1 -- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5930161980 msgr2=0x7f593006ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f592f400700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5930161980 0x7f593006ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f592f400700 1 -- 172.21.15.12:0/4285797988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c0094a0 con 0x7f59300b2440 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f5935000700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5930161980 0x7f593006ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f592f400700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 0x7f59300cc0b0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f591c009b90 tx=0x7f591c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f5927e00700 1 -- 172.21.15.12:0/4285797988 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c0038a0 con 0x7f59300b2440 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f5927e00700 1 -- 172.21.15.12:0/4285797988 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c01f070 con 0x7f59300b2440 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f5935a00700 1 -- 172.21.15.12:0/4285797988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59300cb380 con 0x7f59300b2440 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f5927e00700 1 -- 172.21.15.12:0/4285797988 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c003c20 con 0x7f59300b2440 2024-03-29T10:10:12.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.700+0000 7f5935a00700 1 -- 172.21.15.12:0/4285797988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59301f2f90 con 0x7f59300b2440 2024-03-29T10:10:12.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.701+0000 7f5935a00700 1 -- 172.21.15.12:0/4285797988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593004ea80 con 0x7f59300b2440 2024-03-29T10:10:12.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.702+0000 7f5927e00700 1 -- 172.21.15.12:0/4285797988 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f591c026480 con 0x7f59300b2440 2024-03-29T10:10:12.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.703+0000 7f5927e00700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f591006adb0 0x7f591006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:12.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.703+0000 7f5927e00700 1 -- 172.21.15.12:0/4285797988 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f591c021070 con 0x7f59300b2440 2024-03-29T10:10:12.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.703+0000 7f592fe00700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f591006adb0 0x7f591006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:12.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.704+0000 7f592fe00700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f591006adb0 0x7f591006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5920009b70 tx=0x7f5920008040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.707+0000 7f5927e00700 1 -- 172.21.15.12:0/4285797988 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f591c055f10 con 0x7f59300b2440 2024-03-29T10:10:12.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:12.938+0000 7f5935a00700 1 -- 172.21.15.12:0/4285797988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f59300d0dc0 con 0x7f59300b2440 2024-03-29T10:10:13.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.658+0000 7f5927e00700 1 -- 172.21.15.12:0/4285797988 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v90) v1 ==== 97+0+0 (secure 0 0 0) 0x7f591c059530 con 0x7f59300b2440 2024-03-29T10:10:13.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 -- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f591006adb0 msgr2=0x7f591006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:13.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f591006adb0 0x7f591006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5920009b70 tx=0x7f5920008040 comp rx=0 tx=0).stop 2024-03-29T10:10:13.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 -- 172.21.15.12:0/4285797988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 msgr2=0x7f59300cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:13.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 0x7f59300cc0b0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f591c009b90 tx=0x7f591c004750 comp rx=0 tx=0).stop 2024-03-29T10:10:13.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 -- 172.21.15.12:0/4285797988 shutdown_connections 2024-03-29T10:10:13.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f591006adb0 0x7f591006d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:13.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5930161980 0x7f593006ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:13.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59300b2060 0x7f59300ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:13.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 --2- 172.21.15.12:0/4285797988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59300b2440 0x7f59300cc0b0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:13.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 -- 172.21.15.12:0/4285797988 >> 172.21.15.12:0/4285797988 conn(0x7f593009c6e0 msgr2=0x7f59300cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:13.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 -- 172.21.15.12:0/4285797988 shutdown_connections 2024-03-29T10:10:13.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.661+0000 7f5935a00700 1 -- 172.21.15.12:0/4285797988 wait complete. 2024-03-29T10:10:13.662 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T10:10:13.681 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T10:10:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.881+0000 7f2cc2a00700 1 Processor -- start 2024-03-29T10:10:13.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.881+0000 7f2cc2a00700 1 -- start start 2024-03-29T10:10:13.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.882+0000 7f2cc2a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2cbc0ceec0 0x7f2cbc1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:13.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.882+0000 7f2cc2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cbc161800 0x7f2cbc0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.882+0000 7f2cc2a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cbc0d30d0 con 0x7f2cbc15d640 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.882+0000 7f2cc2a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cbc161c90 con 0x7f2cbc0ceec0 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.882+0000 7f2cc2a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc0b6840 con 0x7f2cbc161800 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2cbc15d640 0x7f2cbc0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51876/0 (socket says 172.21.15.12:51876) 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbbe00700 1 -- 172.21.15.12:0/886049785 learned_addr learned my addr 172.21.15.12:0/886049785 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbb400700 1 --2- 172.21.15.12:0/886049785 >> v2:172.21.15.159:3300/0 conn(0x7f2cbc0ceec0 0x7f2cbc1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965026522 0 0) 0x7f2cbc0d30d0 con 0x7f2cbc15d640 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca00035b0 con 0x7f2cbc15d640 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cc0a00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cbc161800 0x7f2cbc0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1437866926 0 0) 0x7f2ca00035b0 con 0x7f2cbc15d640 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cbc161800 msgr2=0x7f2cbc0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbaa00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cbc161800 0x7f2cbc0b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:13.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 >> v2:172.21.15.159:3300/0 conn(0x7f2cbc0ceec0 msgr2=0x7f2cbc1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbaa00700 1 --2- 172.21.15.12:0/886049785 >> v2:172.21.15.159:3300/0 conn(0x7f2cbc0ceec0 0x7f2cbc1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc0d2b30 con 0x7f2cbc15d640 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.883+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f2cac001670 con 0x7f2cbc15d640 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 >> v1:172.21.15.12:6789/0 conn(0x7f2cbc15d640 legacy=0x7f2cbc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cbaa00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ca0003a00 0x7f2ca0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cbaa00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ca00043f0 0x7f2ca00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cbaa00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca0005090 0x7f2ca000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc161c90 con 0x7f2ca0005090 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc0b6840 con 0x7f2ca00043f0 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca000c030 con 0x7f2ca0003a00 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cc0a00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cbc161800 0x7f2cbc0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cbbe00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ca0003a00 0x7f2ca0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:13.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cc0a00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca0005090 0x7f2ca000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cbb400700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ca00043f0 0x7f2ca00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cc0a00700 1 -- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ca0003a00 msgr2=0x7f2ca0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cc0a00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ca0003a00 0x7f2ca0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cc0a00700 1 -- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ca00043f0 msgr2=0x7f2ca00048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cc0a00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ca00043f0 0x7f2ca00048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.884+0000 7f2cc0a00700 1 -- 172.21.15.12:0/886049785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca00035b0 con 0x7f2ca0005090 2024-03-29T10:10:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.885+0000 7f2cc0a00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca0005090 0x7f2ca000ba10 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0cffa0 tx=0x7f2ca0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcb042fdfd622dd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.885+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4007d90 con 0x7f2ca0005090 2024-03-29T10:10:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.885+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb400f040 con 0x7f2ca0005090 2024-03-29T10:10:13.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.885+0000 7f2cbaa00700 1 -- 172.21.15.12:0/886049785 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4003c40 con 0x7f2ca0005090 2024-03-29T10:10:13.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.885+0000 7f2cc2a00700 1 -- 172.21.15.12:0/886049785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca0005090 msgr2=0x7f2ca000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:13.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.885+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/886049785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca0005090 0x7f2ca000ba10 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0cffa0 tx=0x7f2ca0005880 comp rx=0 tx=0).stop 2024-03-29T10:10:13.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.885+0000 7f2cc0a00700 1 -- 172.21.15.12:0/886049785 reap_dead start 2024-03-29T10:10:13.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.885+0000 7f2cc2a00700 1 -- 172.21.15.12:0/886049785 shutdown_connections 2024-03-29T10:10:13.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.885+0000 7f2cc2a00700 1 -- 172.21.15.12:0/886049785 >> 172.21.15.12:0/886049785 conn(0x7f2cbc08aff0 msgr2=0x7f2cbc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:13.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.886+0000 7f2cc2a00700 1 -- 172.21.15.12:0/886049785 shutdown_connections 2024-03-29T10:10:13.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.886+0000 7f2cc2a00700 1 -- 172.21.15.12:0/886049785 wait complete. 2024-03-29T10:10:13.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.887+0000 7f2cc2a00700 1 Processor -- start 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.887+0000 7f2cc2a00700 1 -- start start 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.888+0000 7f2cc2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ca0005090 0x7f2cbc158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.888+0000 7f2cc2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cbc09e0f0 0x7f2cbc0c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.888+0000 7f2cc2a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cbc15edd0 0x7f2cbc15f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.888+0000 7f2cc2a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc15f790 con 0x7f2cbc09e0f0 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.888+0000 7f2cc2a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc09f9d0 con 0x7f2cbc15edd0 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.888+0000 7f2cc2a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc0c9b30 con 0x7f2ca0005090 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cbc09e0f0 0x7f2cbc0c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cbc09e0f0 0x7f2cbc0c87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45600/0 (socket says 172.21.15.12:45600) 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbb400700 1 -- 172.21.15.12:0/972586182 learned_addr learned my addr 172.21.15.12:0/972586182 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbbe00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ca0005090 0x7f2cbc158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cc0a00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cbc15edd0 0x7f2cbc15f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbb400700 1 -- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ca0005090 msgr2=0x7f2cbc158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:13.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbb400700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ca0005090 0x7f2cbc158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbb400700 1 -- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cbc15edd0 msgr2=0x7f2cbc15f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbb400700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cbc15edd0 0x7f2cbc15f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbb400700 1 -- 172.21.15.12:0/972586182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc09e570 con 0x7f2cbc09e0f0 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbbe00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ca0005090 0x7f2cbc158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cc0a00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cbc15edd0 0x7f2cbc15f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cbb400700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cbc09e0f0 0x7f2cbc0c87d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8011a60 tx=0x7f2ca8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.889+0000 7f2cb3e00700 1 -- 172.21.15.12:0/972586182 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8004240 con 0x7f2cbc09e0f0 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.890+0000 7f2cb3e00700 1 -- 172.21.15.12:0/972586182 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca80043a0 con 0x7f2cbc09e0f0 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.890+0000 7f2cc2a00700 1 -- 172.21.15.12:0/972586182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cbc0c8d10 con 0x7f2cbc09e0f0 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.890+0000 7f2cb3e00700 1 -- 172.21.15.12:0/972586182 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca800f7f0 con 0x7f2cbc09e0f0 2024-03-29T10:10:13.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.890+0000 7f2cc2a00700 1 -- 172.21.15.12:0/972586182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cbc0c9290 con 0x7f2cbc09e0f0 2024-03-29T10:10:13.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.891+0000 7f2cc2a00700 1 -- 172.21.15.12:0/972586182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cbc067720 con 0x7f2cbc09e0f0 2024-03-29T10:10:13.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.892+0000 7f2cb3e00700 1 -- 172.21.15.12:0/972586182 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2ca800f950 con 0x7f2cbc09e0f0 2024-03-29T10:10:13.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.893+0000 7f2cb3e00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c9c06adb0 0x7f2c9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:13.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.893+0000 7f2cb3e00700 1 -- 172.21.15.12:0/972586182 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2ca8089870 con 0x7f2cbc09e0f0 2024-03-29T10:10:13.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.893+0000 7f2cbbe00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c9c06adb0 0x7f2c9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:13.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.894+0000 7f2cbbe00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c9c06adb0 0x7f2c9c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc068000 tx=0x7f2cac009040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:13.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:13.897+0000 7f2cb3e00700 1 -- 172.21.15.12:0/972586182 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ca8054bf0 con 0x7f2cbc09e0f0 2024-03-29T10:10:14.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.197+0000 7f2cc2a00700 1 -- 172.21.15.12:0/972586182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2cbc15d040 con 0x7f2cbc09e0f0 2024-03-29T10:10:14.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.197+0000 7f2cb3e00700 1 -- 172.21.15.12:0/972586182 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2ca8019090 con 0x7f2cbc09e0f0 2024-03-29T10:10:14.197 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:14.197 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:10:14.197 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.198 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.199 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.200 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T10:10:14.201 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T10:10:14.203 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:14.205 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.206 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:10:14.207 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.208 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:10:14.209 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T10:10:14.210 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:10:14.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 -- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c9c06adb0 msgr2=0x7f2c9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c9c06adb0 0x7f2c9c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc068000 tx=0x7f2cac009040 comp rx=0 tx=0).stop 2024-03-29T10:10:14.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 -- 172.21.15.12:0/972586182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cbc09e0f0 msgr2=0x7f2cbc0c87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cbc09e0f0 0x7f2cbc0c87d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8011a60 tx=0x7f2ca8011e20 comp rx=0 tx=0).stop 2024-03-29T10:10:14.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 -- 172.21.15.12:0/972586182 shutdown_connections 2024-03-29T10:10:14.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c9c06adb0 0x7f2c9c06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cbc15edd0 0x7f2cbc15f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ca0005090 0x7f2cbc158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 --2- 172.21.15.12:0/972586182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cbc09e0f0 0x7f2cbc0c87d0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 -- 172.21.15.12:0/972586182 >> 172.21.15.12:0/972586182 conn(0x7f2cbc08aff0 msgr2=0x7f2cbc0b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:14.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 -- 172.21.15.12:0/972586182 shutdown_connections 2024-03-29T10:10:14.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.200+0000 7f2cc2a00700 1 -- 172.21.15.12:0/972586182 wait complete. 2024-03-29T10:10:14.218 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:14.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.416+0000 7f76d7400700 1 Processor -- start 2024-03-29T10:10:14.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.417+0000 7f76d7400700 1 -- start start 2024-03-29T10:10:14.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.418+0000 7f76d7400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f76d00afbe0 0x7f76d006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.418+0000 7f76d7400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d006f850 con 0x7f76d006a790 2024-03-29T10:10:14.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.418+0000 7f76d7400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d006fad0 con 0x7f76d00cebd0 2024-03-29T10:10:14.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.418+0000 7f76d7400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d00cf710 con 0x7f76d00afbe0 2024-03-29T10:10:14.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.418+0000 7f76d4a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f76d006a790 0x7f76d006ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51884/0 (socket says 172.21.15.12:51884) 2024-03-29T10:10:14.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.418+0000 7f76d4a00700 1 -- 172.21.15.12:0/1698483221 learned_addr learned my addr 172.21.15.12:0/1698483221 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755662522 0 0) 0x7f76d006f850 con 0x7f76d006a790 2024-03-29T10:10:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b80035b0 con 0x7f76d006a790 2024-03-29T10:10:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cfe00700 1 --2- 172.21.15.12:0/1698483221 >> v2:172.21.15.159:3300/0 conn(0x7f76d00afbe0 0x7f76d006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 211909543 0 0) 0x7f76b80035b0 con 0x7f76d006a790 2024-03-29T10:10:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 >> v2:172.21.15.159:3300/0 conn(0x7f76d00afbe0 msgr2=0x7f76d006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 --2- 172.21.15.12:0/1698483221 >> v2:172.21.15.159:3300/0 conn(0x7f76d00afbe0 0x7f76d006dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 >> v1:172.21.15.159:6789/0 conn(0x7f76d00cebd0 legacy=0x7f76d00cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d00af970 con 0x7f76d006a790 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f76c4001670 con 0x7f76d006a790 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 >> v1:172.21.15.12:6789/0 conn(0x7f76d006a790 legacy=0x7f76d006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8003a00 0x7f76b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8004850 0x7f76b8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8005410 0x7f76b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00cf710 con 0x7f76b8005410 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b800c2a0 con 0x7f76b8004850 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b800c460 con 0x7f76b8003a00 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76d5400700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8005410 0x7f76b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.419+0000 7f76cfe00700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8004850 0x7f76b8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76d4a00700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8003a00 0x7f76b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76d5400700 1 -- 172.21.15.12:0/1698483221 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8003a00 msgr2=0x7f76b8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76d5400700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8003a00 0x7f76b8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76d5400700 1 -- 172.21.15.12:0/1698483221 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8004850 msgr2=0x7f76b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76d5400700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8004850 0x7f76b8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76d5400700 1 -- 172.21.15.12:0/1698483221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b80035b0 con 0x7f76b8005410 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76d5400700 1 -- 172.21.15.12:0/1698483221 reap_dead start 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76d5400700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8005410 0x7f76b800ba10 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f76c800c190 tx=0x7f76b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1aa3279f309c8b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8001db0 con 0x7f76b8005410 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c800ccd0 con 0x7f76b8005410 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.420+0000 7f76cf400700 1 -- 172.21.15.12:0/1698483221 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8003c40 con 0x7f76b8005410 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.421+0000 7f76d7400700 1 -- 172.21.15.12:0/1698483221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8005410 msgr2=0x7f76b800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.421+0000 7f76d7400700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8005410 0x7f76b800ba10 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f76c800c190 tx=0x7f76b8005880 comp rx=0 tx=0).stop 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.421+0000 7f76d7400700 1 -- 172.21.15.12:0/1698483221 shutdown_connections 2024-03-29T10:10:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.421+0000 7f76d7400700 1 --2- 172.21.15.12:0/1698483221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8005410 0x7f76b800ba10 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.421+0000 7f76d7400700 1 -- 172.21.15.12:0/1698483221 >> 172.21.15.12:0/1698483221 conn(0x7f76d00b5ab0 msgr2=0x7f76d00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.421+0000 7f76d7400700 1 -- 172.21.15.12:0/1698483221 shutdown_connections 2024-03-29T10:10:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.421+0000 7f76d7400700 1 -- 172.21.15.12:0/1698483221 wait complete. 2024-03-29T10:10:14.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.423+0000 7f76d7400700 1 Processor -- start 2024-03-29T10:10:14.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.423+0000 7f76d7400700 1 -- start start 2024-03-29T10:10:14.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76d7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8005410 0x7f76d0153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76d7400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76d00ca100 0x7f76d00cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76d7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d00ca580 0x7f76d00cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76d7400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00cda10 con 0x7f76d00ca580 2024-03-29T10:10:14.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76d7400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d01f3200 con 0x7f76d00ca100 2024-03-29T10:10:14.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76d7400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0154380 con 0x7f76b8005410 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76d5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d00ca580 0x7f76d00cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76d5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d00ca580 0x7f76d00cb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45622/0 (socket says 172.21.15.12:45622) 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76d5400700 1 -- 172.21.15.12:0/1483819949 learned_addr learned my addr 172.21.15.12:0/1483819949 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76d4a00700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8005410 0x7f76d0153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.424+0000 7f76cfe00700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76d00ca100 0x7f76d00cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.425+0000 7f76d5400700 1 -- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8005410 msgr2=0x7f76d0153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.425+0000 7f76d5400700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8005410 0x7f76d0153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.425+0000 7f76d5400700 1 -- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76d00ca100 msgr2=0x7f76d00cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.425+0000 7f76d5400700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76d00ca100 0x7f76d00cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.425+0000 7f76d5400700 1 -- 172.21.15.12:0/1483819949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d009dfe0 con 0x7f76d00ca580 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.425+0000 7f76d5400700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d00ca580 0x7f76d00cb8f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f76c800c780 tx=0x7f76c80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.425+0000 7f76ccc00700 1 -- 172.21.15.12:0/1483819949 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8003de0 con 0x7f76d00ca580 2024-03-29T10:10:14.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.425+0000 7f76ccc00700 1 -- 172.21.15.12:0/1483819949 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c8005070 con 0x7f76d00ca580 2024-03-29T10:10:14.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.425+0000 7f76ccc00700 1 -- 172.21.15.12:0/1483819949 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8005220 con 0x7f76d00ca580 2024-03-29T10:10:14.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.425+0000 7f76d7400700 1 -- 172.21.15.12:0/1483819949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d009e2c0 con 0x7f76d00ca580 2024-03-29T10:10:14.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.426+0000 7f76d7400700 1 -- 172.21.15.12:0/1483819949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d009b690 con 0x7f76d00ca580 2024-03-29T10:10:14.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.427+0000 7f76d7400700 1 -- 172.21.15.12:0/1483819949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d0067720 con 0x7f76d00ca580 2024-03-29T10:10:14.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.427+0000 7f76ccc00700 1 -- 172.21.15.12:0/1483819949 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f76c8005380 con 0x7f76d00ca580 2024-03-29T10:10:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.428+0000 7f76ccc00700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b006adb0 0x7f76b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.428+0000 7f76ccc00700 1 -- 172.21.15.12:0/1483819949 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f76c8089a70 con 0x7f76d00ca580 2024-03-29T10:10:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.429+0000 7f76d4a00700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b006adb0 0x7f76b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.430+0000 7f76d4a00700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b006adb0 0x7f76b006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76c4007db0 tx=0x7f76c4000bc0 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:14.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.433+0000 7f76ccc00700 1 -- 172.21.15.12:0/1483819949 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76c8054e80 con 0x7f76d00ca580 2024-03-29T10:10:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.645+0000 7f76d7400700 1 -- 172.21.15.12:0/1483819949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76d006b080 con 0x7f76d00ca580 2024-03-29T10:10:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.646+0000 7f76ccc00700 1 -- 172.21.15.12:0/1483819949 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+12389 (secure 0 0 0) 0x7f76c80584a0 con 0x7f76d00ca580 2024-03-29T10:10:14.646 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:14.646 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:13.651671+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 -- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b006adb0 msgr2=0x7f76b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b006adb0 0x7f76b006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76c4007db0 tx=0x7f76c4000bc0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 -- 172.21.15.12:0/1483819949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d00ca580 msgr2=0x7f76d00cb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d00ca580 0x7f76d00cb8f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f76c800c780 tx=0x7f76c80014a0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 -- 172.21.15.12:0/1483819949 shutdown_connections 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b006adb0 0x7f76b006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d00ca580 0x7f76d00cb8f0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8005410 0x7f76d0153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 --2- 172.21.15.12:0/1483819949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76d00ca100 0x7f76d00cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 -- 172.21.15.12:0/1483819949 >> 172.21.15.12:0/1483819949 conn(0x7f76d00b5ab0 msgr2=0x7f76d00b3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 -- 172.21.15.12:0/1483819949 shutdown_connections 2024-03-29T10:10:14.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.649+0000 7f76d7400700 1 -- 172.21.15.12:0/1483819949 wait complete. 2024-03-29T10:10:14.667 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:10:14.667 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:10:14.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.868+0000 7f7787a00700 1 Processor -- start 2024-03-29T10:10:14.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.868+0000 7f7787a00700 1 -- start start 2024-03-29T10:10:14.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f7787a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77800cd020 0x7f77800cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f7787a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f77800b2070 0x7f77800b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f7787a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77800cdae0 con 0x7f77800cbb30 2024-03-29T10:10:14.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f7787a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77800cddc0 con 0x7f77800cd020 2024-03-29T10:10:14.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f7787a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77800ce0a0 con 0x7f77800b2070 2024-03-29T10:10:14.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f777fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77800cd020 0x7f77800cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f777fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77800cd020 0x7f77800cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45636/0 (socket says 172.21.15.12:45636) 2024-03-29T10:10:14.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f777fe00700 1 -- 172.21.15.12:0/2650221542 learned_addr learned my addr 172.21.15.12:0/2650221542 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:14.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f7785a00700 1 --2- 172.21.15.12:0/2650221542 >> v2:172.21.15.159:3300/0 conn(0x7f77800b2070 0x7f77800b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f777fe00700 1 -- 172.21.15.12:0/2650221542 >> v2:172.21.15.159:3300/0 conn(0x7f77800b2070 msgr2=0x7f77800b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:14.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f777fe00700 1 --2- 172.21.15.12:0/2650221542 >> v2:172.21.15.159:3300/0 conn(0x7f77800b2070 0x7f77800b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f777fe00700 1 -- 172.21.15.12:0/2650221542 >> v1:172.21.15.159:6789/0 conn(0x7f77800cbb30 legacy=0x7f77800cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.870+0000 7f777fe00700 1 -- 172.21.15.12:0/2650221542 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778006dc20 con 0x7f77800cd020 2024-03-29T10:10:14.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.871+0000 7f777fe00700 1 --2- 172.21.15.12:0/2650221542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800cd020 0x7f77800cd480 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f77800b1d60 tx=0x7f776c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39b231bcdd23f84 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:14.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.871+0000 7f777f400700 1 -- 172.21.15.12:0/2650221542 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c011830 con 0x7f77800cd020 2024-03-29T10:10:14.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.871+0000 7f777f400700 1 -- 172.21.15.12:0/2650221542 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c011990 con 0x7f77800cd020 2024-03-29T10:10:14.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.871+0000 7f777f400700 1 -- 172.21.15.12:0/2650221542 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c011d20 con 0x7f77800cd020 2024-03-29T10:10:14.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.872+0000 7f7787a00700 1 -- 172.21.15.12:0/2650221542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800cd020 msgr2=0x7f77800cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.872+0000 7f7787a00700 1 --2- 172.21.15.12:0/2650221542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800cd020 0x7f77800cd480 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f77800b1d60 tx=0x7f776c009bd0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.872+0000 7f7787a00700 1 -- 172.21.15.12:0/2650221542 shutdown_connections 2024-03-29T10:10:14.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.872+0000 7f7787a00700 1 --2- 172.21.15.12:0/2650221542 >> v2:172.21.15.159:3300/0 conn(0x7f77800b2070 0x7f77800b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.872+0000 7f7787a00700 1 --2- 172.21.15.12:0/2650221542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800cd020 0x7f77800cd480 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.872+0000 7f7787a00700 1 -- 172.21.15.12:0/2650221542 >> 172.21.15.12:0/2650221542 conn(0x7f77800b0d70 msgr2=0x7f77800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:14.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.872+0000 7f7787a00700 1 -- 172.21.15.12:0/2650221542 shutdown_connections 2024-03-29T10:10:14.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.872+0000 7f7787a00700 1 -- 172.21.15.12:0/2650221542 wait complete. 2024-03-29T10:10:14.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.873+0000 7f7787a00700 1 Processor -- start 2024-03-29T10:10:14.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.874+0000 7f7787a00700 1 -- start start 2024-03-29T10:10:14.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.874+0000 7f7787a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800b2070 0x7f77800991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7787a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77800cbb30 0x7f77801f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7787a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77800cd020 0x7f778015e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7787a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7780099710 con 0x7f77800b2070 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7787a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778009c350 con 0x7f77800cd020 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7787a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77801f0b70 con 0x7f77800cbb30 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7785000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800b2070 0x7f77800991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7785000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800b2070 0x7f77800991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45644/0 (socket says 172.21.15.12:45644) 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7785000700 1 -- 172.21.15.12:0/3907280442 learned_addr learned my addr 172.21.15.12:0/3907280442 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f777fe00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77800cbb30 0x7f77801f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7785a00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77800cd020 0x7f778015e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7785000700 1 -- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77800cbb30 msgr2=0x7f77801f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7785000700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77800cbb30 0x7f77801f0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7785000700 1 -- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77800cd020 msgr2=0x7f778015e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7785000700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77800cd020 0x7f778015e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7785000700 1 -- 172.21.15.12:0/3907280442 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c009550 con 0x7f77800b2070 2024-03-29T10:10:14.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.875+0000 7f7785a00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77800cd020 0x7f778015e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:14.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.876+0000 7f7785000700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800b2070 0x7f77800991d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f777400c4e0 tx=0x7f777400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:14.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.876+0000 7f777cc00700 1 -- 172.21.15.12:0/3907280442 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77740052d0 con 0x7f77800b2070 2024-03-29T10:10:14.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.876+0000 7f777cc00700 1 -- 172.21.15.12:0/3907280442 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774005430 con 0x7f77800b2070 2024-03-29T10:10:14.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.876+0000 7f777cc00700 1 -- 172.21.15.12:0/3907280442 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77740038c0 con 0x7f77800b2070 2024-03-29T10:10:14.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.876+0000 7f7787a00700 1 -- 172.21.15.12:0/3907280442 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778009c5f0 con 0x7f77800b2070 2024-03-29T10:10:14.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.876+0000 7f7787a00700 1 -- 172.21.15.12:0/3907280442 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778009cbb0 con 0x7f77800b2070 2024-03-29T10:10:14.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.877+0000 7f7787a00700 1 -- 172.21.15.12:0/3907280442 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7780067720 con 0x7f77800b2070 2024-03-29T10:10:14.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.878+0000 7f777cc00700 1 -- 172.21.15.12:0/3907280442 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7774003a20 con 0x7f77800b2070 2024-03-29T10:10:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.879+0000 7f777cc00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f776006adb0 0x7f776006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:14.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.879+0000 7f777cc00700 1 -- 172.21.15.12:0/3907280442 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7774088620 con 0x7f77800b2070 2024-03-29T10:10:14.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.879+0000 7f777fe00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f776006adb0 0x7f776006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:14.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.880+0000 7f777fe00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f776006adb0 0x7f776006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f776c014040 tx=0x7f776c008030 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:14.883+0000 7f777cc00700 1 -- 172.21.15.12:0/3907280442 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f77740539b0 con 0x7f77800b2070 2024-03-29T10:10:15.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.096+0000 7f7787a00700 1 -- 172.21.15.12:0/3907280442 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f77800ce2e0 con 0x7f77800b2070 2024-03-29T10:10:15.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.096+0000 7f777cc00700 1 -- 172.21.15.12:0/3907280442 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7774056fd0 con 0x7f77800b2070 2024-03-29T10:10:15.096 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:10:15.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.098+0000 7f7787a00700 1 -- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f776006adb0 msgr2=0x7f776006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.098+0000 7f7787a00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f776006adb0 0x7f776006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f776c014040 tx=0x7f776c008030 comp rx=0 tx=0).stop 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.098+0000 7f7787a00700 1 -- 172.21.15.12:0/3907280442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800b2070 msgr2=0x7f77800991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.098+0000 7f7787a00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800b2070 0x7f77800991d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f777400c4e0 tx=0x7f777400c7f0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.099+0000 7f7787a00700 1 -- 172.21.15.12:0/3907280442 shutdown_connections 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.099+0000 7f7787a00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f776006adb0 0x7f776006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.099+0000 7f7787a00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77800cd020 0x7f778015e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.099+0000 7f7787a00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77800b2070 0x7f77800991d0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.099+0000 7f7787a00700 1 --2- 172.21.15.12:0/3907280442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77800cbb30 0x7f77801f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.099+0000 7f7787a00700 1 -- 172.21.15.12:0/3907280442 >> 172.21.15.12:0/3907280442 conn(0x7f77800b0d70 msgr2=0x7f77800aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.099+0000 7f7787a00700 1 -- 172.21.15.12:0/3907280442 shutdown_connections 2024-03-29T10:10:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.099+0000 7f7787a00700 1 -- 172.21.15.12:0/3907280442 wait complete. 2024-03-29T10:10:15.117 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T10:10:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.314+0000 7fc225600700 1 Processor -- start 2024-03-29T10:10:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.315+0000 7fc225600700 1 -- start start 2024-03-29T10:10:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.316+0000 7fc225600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc220160f10 0x7fc2200d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.316+0000 7fc225600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2200ceda0 con 0x7fc220161590 2024-03-29T10:10:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.316+0000 7fc225600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2201612f0 con 0x7fc2200cf230 2024-03-29T10:10:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.316+0000 7fc225600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2200ce470 con 0x7fc220160f10 2024-03-29T10:10:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.316+0000 7fc21f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc220161590 0x7fc2200b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51896/0 (socket says 172.21.15.12:51896) 2024-03-29T10:10:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.316+0000 7fc21f400700 1 -- 172.21.15.12:0/1072879778 learned_addr learned my addr 172.21.15.12:0/1072879778 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.316+0000 7fc21ea00700 1 --2- 172.21.15.12:0/1072879778 >> v2:172.21.15.159:3300/0 conn(0x7fc220160f10 0x7fc2200d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.316+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358764457 0 0) 0x7fc2200ceda0 con 0x7fc220161590 2024-03-29T10:10:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2040035b0 con 0x7fc220161590 2024-03-29T10:10:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1922021624 0 0) 0x7fc2040035b0 con 0x7fc220161590 2024-03-29T10:10:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 >> v2:172.21.15.159:3300/0 conn(0x7fc220160f10 msgr2=0x7fc2200d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 --2- 172.21.15.12:0/1072879778 >> v2:172.21.15.159:3300/0 conn(0x7fc220160f10 0x7fc2200d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 >> v1:172.21.15.159:6789/0 conn(0x7fc2200cf230 legacy=0x7fc2200d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2200ce750 con 0x7fc220161590 2024-03-29T10:10:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fc2180019f0 con 0x7fc220161590 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 >> v1:172.21.15.12:6789/0 conn(0x7fc220161590 legacy=0x7fc2200b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc204003a00 0x7fc204003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc204004850 0x7fc204004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc204005210 0x7fc20400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2200ce470 con 0x7fc204005210 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20400c1b0 con 0x7fc204003a00 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20400c370 con 0x7fc204004850 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21f400700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc204005210 0x7fc20400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21ea00700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc204003a00 0x7fc204003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.317+0000 7fc21e000700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc204004850 0x7fc204004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.318+0000 7fc21f400700 1 -- 172.21.15.12:0/1072879778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc204004850 msgr2=0x7fc204004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.318+0000 7fc21f400700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc204004850 0x7fc204004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.318+0000 7fc21f400700 1 -- 172.21.15.12:0/1072879778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc204003a00 msgr2=0x7fc204003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.318+0000 7fc21f400700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc204003a00 0x7fc204003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.318+0000 7fc21f400700 1 -- 172.21.15.12:0/1072879778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2040035b0 con 0x7fc204005210 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.318+0000 7fc21f400700 1 -- 172.21.15.12:0/1072879778 reap_dead start 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.318+0000 7fc21f400700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc204005210 0x7fc20400ba10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc21800a830 tx=0x7fc204005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a1280b0e14e3a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.318+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21800a2f0 con 0x7fc204005210 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.318+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc218012620 con 0x7fc204005210 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.318+0000 7fc21d600700 1 -- 172.21.15.12:0/1072879778 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2180129d0 con 0x7fc204005210 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.319+0000 7fc225600700 1 -- 172.21.15.12:0/1072879778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc204005210 msgr2=0x7fc20400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.319+0000 7fc225600700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc204005210 0x7fc20400ba10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc21800a830 tx=0x7fc204005880 comp rx=0 tx=0).stop 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.319+0000 7fc225600700 1 -- 172.21.15.12:0/1072879778 shutdown_connections 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.319+0000 7fc225600700 1 --2- 172.21.15.12:0/1072879778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc204005210 0x7fc20400ba10 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.319+0000 7fc225600700 1 -- 172.21.15.12:0/1072879778 >> 172.21.15.12:0/1072879778 conn(0x7fc22008aff0 msgr2=0x7fc2200d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.319+0000 7fc225600700 1 -- 172.21.15.12:0/1072879778 shutdown_connections 2024-03-29T10:10:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.319+0000 7fc225600700 1 -- 172.21.15.12:0/1072879778 wait complete. 2024-03-29T10:10:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.321+0000 7fc225600700 1 Processor -- start 2024-03-29T10:10:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.321+0000 7fc225600700 1 -- start start 2024-03-29T10:10:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc225600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc204005210 0x7fc220151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc225600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc22015c0f0 0x7fc22015c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc225600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2200b8710 0x7fc2200b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc225600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22015cab0 con 0x7fc22015c0f0 2024-03-29T10:10:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc225600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2200b7240 con 0x7fc2200b8710 2024-03-29T10:10:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc225600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2200b7520 con 0x7fc204005210 2024-03-29T10:10:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc21e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc22015c0f0 0x7fc22015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc21e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc22015c0f0 0x7fc22015c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45660/0 (socket says 172.21.15.12:45660) 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc21e000700 1 -- 172.21.15.12:0/1618668406 learned_addr learned my addr 172.21.15.12:0/1618668406 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc21ea00700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc204005210 0x7fc220151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc21f400700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2200b8710 0x7fc2200b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc21e000700 1 -- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc204005210 msgr2=0x7fc220151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc21e000700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc204005210 0x7fc220151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc21e000700 1 -- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2200b8710 msgr2=0x7fc2200b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc21e000700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2200b8710 0x7fc2200b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.322+0000 7fc21e000700 1 -- 172.21.15.12:0/1618668406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2200b7800 con 0x7fc22015c0f0 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.323+0000 7fc21e000700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc22015c0f0 0x7fc22015c570 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00f610 tx=0x7fc20c00f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.323+0000 7fc216a00700 1 -- 172.21.15.12:0/1618668406 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c0052e0 con 0x7fc22015c0f0 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.323+0000 7fc216a00700 1 -- 172.21.15.12:0/1618668406 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20c005440 con 0x7fc22015c0f0 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.323+0000 7fc216a00700 1 -- 172.21.15.12:0/1618668406 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c0038a0 con 0x7fc22015c0f0 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.323+0000 7fc225600700 1 -- 172.21.15.12:0/1618668406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2200b7ae0 con 0x7fc22015c0f0 2024-03-29T10:10:15.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.323+0000 7fc225600700 1 -- 172.21.15.12:0/1618668406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc220152160 con 0x7fc22015c0f0 2024-03-29T10:10:15.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.324+0000 7fc225600700 1 -- 172.21.15.12:0/1618668406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2200d4520 con 0x7fc22015c0f0 2024-03-29T10:10:15.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.325+0000 7fc216a00700 1 -- 172.21.15.12:0/1618668406 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc20c003a00 con 0x7fc22015c0f0 2024-03-29T10:10:15.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.326+0000 7fc216a00700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc20006adb0 0x7fc20006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.326+0000 7fc216a00700 1 -- 172.21.15.12:0/1618668406 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc20c088ef0 con 0x7fc22015c0f0 2024-03-29T10:10:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.326+0000 7fc21ea00700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc20006adb0 0x7fc20006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.327+0000 7fc21ea00700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc20006adb0 0x7fc20006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc210009b20 tx=0x7fc210008040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:15.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.330+0000 7fc216a00700 1 -- 172.21.15.12:0/1618668406 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc20c054270 con 0x7fc22015c0f0 2024-03-29T10:10:15.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.546+0000 7fc225600700 1 -- 172.21.15.12:0/1618668406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc220156de0 con 0x7fc22015c0f0 2024-03-29T10:10:15.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.546+0000 7fc216a00700 1 -- 172.21.15.12:0/1618668406 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v90) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc20c003cb0 con 0x7fc22015c0f0 2024-03-29T10:10:15.546 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:15.546 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:10:15.546 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:15.546 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T10:10:15.546 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T10:10:15.546 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:15.546 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:10:15.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.548+0000 7fc225600700 1 -- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc20006adb0 msgr2=0x7fc20006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.548+0000 7fc225600700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc20006adb0 0x7fc20006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc210009b20 tx=0x7fc210008040 comp rx=0 tx=0).stop 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.548+0000 7fc225600700 1 -- 172.21.15.12:0/1618668406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc22015c0f0 msgr2=0x7fc22015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.548+0000 7fc225600700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc22015c0f0 0x7fc22015c570 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00f610 tx=0x7fc20c00f9d0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.549+0000 7fc225600700 1 -- 172.21.15.12:0/1618668406 shutdown_connections 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.549+0000 7fc225600700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc20006adb0 0x7fc20006d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.549+0000 7fc225600700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2200b8710 0x7fc2200b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.549+0000 7fc225600700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc204005210 0x7fc220151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.549+0000 7fc225600700 1 --2- 172.21.15.12:0/1618668406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc22015c0f0 0x7fc22015c570 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.549+0000 7fc225600700 1 -- 172.21.15.12:0/1618668406 >> 172.21.15.12:0/1618668406 conn(0x7fc22008aff0 msgr2=0x7fc2200cc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.549+0000 7fc225600700 1 -- 172.21.15.12:0/1618668406 shutdown_connections 2024-03-29T10:10:15.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.549+0000 7fc225600700 1 -- 172.21.15.12:0/1618668406 wait complete. 2024-03-29T10:10:15.566 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T10:10:15.567 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T10:10:15.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.768+0000 7f00a5000700 1 Processor -- start 2024-03-29T10:10:15.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.769+0000 7f00a5000700 1 -- start start 2024-03-29T10:10:15.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f00a5000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f00a00ced30 0x7f00a00d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f00a5000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00d2710 0x7f00a015f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f00a5000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a00ce8a0 con 0x7f00a015ddf0 2024-03-29T10:10:15.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f00a5000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a015ec60 con 0x7f00a00ced30 2024-03-29T10:10:15.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f00a5000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00cdc90 con 0x7f00a00d2710 2024-03-29T10:10:15.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f009de00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f00a015ddf0 0x7f00a015e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51902/0 (socket says 172.21.15.12:51902) 2024-03-29T10:10:15.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f009de00700 1 -- 172.21.15.12:0/95709005 learned_addr learned my addr 172.21.15.12:0/95709005 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:15.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f009e800700 1 --2- 172.21.15.12:0/95709005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00d2710 0x7f00a015f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f009d400700 1 --2- 172.21.15.12:0/95709005 >> v2:172.21.15.159:3300/0 conn(0x7f00a00ced30 0x7f00a00d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f009ca00700 1 -- 172.21.15.12:0/95709005 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930956159 0 0) 0x7f00a00ce8a0 con 0x7f00a015ddf0 2024-03-29T10:10:15.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.770+0000 7f009ca00700 1 -- 172.21.15.12:0/95709005 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00840035b0 con 0x7f00a015ddf0 2024-03-29T10:10:15.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.771+0000 7f009d400700 1 -- 172.21.15.12:0/95709005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00d2710 msgr2=0x7f00a015f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:15.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.771+0000 7f009d400700 1 --2- 172.21.15.12:0/95709005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00d2710 0x7f00a015f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.771+0000 7f009d400700 1 -- 172.21.15.12:0/95709005 >> v1:172.21.15.12:6789/0 conn(0x7f00a015ddf0 legacy=0x7f00a015e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.771+0000 7f009d400700 1 -- 172.21.15.12:0/95709005 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a00cdf70 con 0x7f00a00ced30 2024-03-29T10:10:15.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.771+0000 7f009d400700 1 --2- 172.21.15.12:0/95709005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a00ced30 0x7f00a00d21d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f008c009840 tx=0x7f008c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20e5715eeb6772 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:15.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.771+0000 7f009ca00700 1 -- 172.21.15.12:0/95709005 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c011830 con 0x7f00a00ced30 2024-03-29T10:10:15.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.771+0000 7f009ca00700 1 -- 172.21.15.12:0/95709005 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c011990 con 0x7f00a00ced30 2024-03-29T10:10:15.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.771+0000 7f009ca00700 1 -- 172.21.15.12:0/95709005 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c011d00 con 0x7f00a00ced30 2024-03-29T10:10:15.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.772+0000 7f00a5000700 1 -- 172.21.15.12:0/95709005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a00ced30 msgr2=0x7f00a00d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.772+0000 7f00a5000700 1 --2- 172.21.15.12:0/95709005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a00ced30 0x7f00a00d21d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f008c009840 tx=0x7f008c009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:15.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.772+0000 7f00a5000700 1 -- 172.21.15.12:0/95709005 shutdown_connections 2024-03-29T10:10:15.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.772+0000 7f00a5000700 1 --2- 172.21.15.12:0/95709005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00d2710 0x7f00a015f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.772+0000 7f00a5000700 1 --2- 172.21.15.12:0/95709005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a00ced30 0x7f00a00d21d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.772+0000 7f00a5000700 1 -- 172.21.15.12:0/95709005 >> 172.21.15.12:0/95709005 conn(0x7f00a008aff0 msgr2=0x7f00a00d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:15.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.772+0000 7f00a5000700 1 -- 172.21.15.12:0/95709005 shutdown_connections 2024-03-29T10:10:15.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.772+0000 7f00a5000700 1 -- 172.21.15.12:0/95709005 wait complete. 2024-03-29T10:10:15.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.774+0000 7f00a5000700 1 Processor -- start 2024-03-29T10:10:15.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.774+0000 7f00a5000700 1 -- start start 2024-03-29T10:10:15.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f00a5000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00a00ced30 0x7f00a01562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f00a5000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a00d2710 0x7f00a00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f00a5000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a015ddf0 0x7f00a00b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f00a5000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00c8120 con 0x7f00a00ced30 2024-03-29T10:10:15.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f00a5000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00c83a0 con 0x7f00a00d2710 2024-03-29T10:10:15.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f00a5000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00c8680 con 0x7f00a015ddf0 2024-03-29T10:10:15.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f009de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00a00ced30 0x7f00a01562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f009de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00a00ced30 0x7f00a01562b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45666/0 (socket says 172.21.15.12:45666) 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f009de00700 1 -- 172.21.15.12:0/8456418 learned_addr learned my addr 172.21.15.12:0/8456418 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f009d400700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a00d2710 0x7f00a00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f009e800700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a015ddf0 0x7f00a00b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f009de00700 1 -- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a015ddf0 msgr2=0x7f00a00b7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f009de00700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a015ddf0 0x7f00a00b7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f009de00700 1 -- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a00d2710 msgr2=0x7f00a00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f009de00700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a00d2710 0x7f00a00ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.775+0000 7f009de00700 1 -- 172.21.15.12:0/8456418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c0094a0 con 0x7f00a00ced30 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.776+0000 7f009de00700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00a00ced30 0x7f00a01562b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f009000c420 tx=0x7f009000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.776+0000 7f0096000700 1 -- 172.21.15.12:0/8456418 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090005d70 con 0x7f00a00ced30 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.776+0000 7f0096000700 1 -- 172.21.15.12:0/8456418 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00900070d0 con 0x7f00a00ced30 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.776+0000 7f00a5000700 1 -- 172.21.15.12:0/8456418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a00c8960 con 0x7f00a00ced30 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.776+0000 7f0096000700 1 -- 172.21.15.12:0/8456418 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090007440 con 0x7f00a00ced30 2024-03-29T10:10:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.776+0000 7f00a5000700 1 -- 172.21.15.12:0/8456418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a00c8f20 con 0x7f00a00ced30 2024-03-29T10:10:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.777+0000 7f00a5000700 1 -- 172.21.15.12:0/8456418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a0067720 con 0x7f00a00ced30 2024-03-29T10:10:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.778+0000 7f0096000700 1 -- 172.21.15.12:0/8456418 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f00900075a0 con 0x7f00a00ced30 2024-03-29T10:10:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.780+0000 7f0096000700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f008006adb0 0x7f008006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.780+0000 7f0096000700 1 -- 172.21.15.12:0/8456418 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0090088420 con 0x7f00a00ced30 2024-03-29T10:10:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.781+0000 7f009d400700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f008006adb0 0x7f008006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.781+0000 7f009d400700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f008006adb0 0x7f008006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f008c009fd0 tx=0x7f008c008030 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.785+0000 7f0096000700 1 -- 172.21.15.12:0/8456418 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f00900538e0 con 0x7f00a00ced30 2024-03-29T10:10:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:15.997+0000 7f00a5000700 1 -- 172.21.15.12:0/8456418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f00a00b0540 con 0x7f00a00ced30 2024-03-29T10:10:16.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.099+0000 7f0096000700 1 -- 172.21.15.12:0/8456418 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v91) v1 ==== 138+0+0 (secure 0 0 0) 0x7f00900170b0 con 0x7f00a00ced30 2024-03-29T10:10:16.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 -- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f008006adb0 msgr2=0x7f008006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:16.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f008006adb0 0x7f008006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f008c009fd0 tx=0x7f008c008030 comp rx=0 tx=0).stop 2024-03-29T10:10:16.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 -- 172.21.15.12:0/8456418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00a00ced30 msgr2=0x7f00a01562b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:16.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00a00ced30 0x7f00a01562b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f009000c420 tx=0x7f009000c730 comp rx=0 tx=0).stop 2024-03-29T10:10:16.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 -- 172.21.15.12:0/8456418 shutdown_connections 2024-03-29T10:10:16.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f008006adb0 0x7f008006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:16.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a015ddf0 0x7f00a00b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:16.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00a00ced30 0x7f00a01562b0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:16.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 --2- 172.21.15.12:0/8456418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a00d2710 0x7f00a00ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 -- 172.21.15.12:0/8456418 >> 172.21.15.12:0/8456418 conn(0x7f00a008aff0 msgr2=0x7f00a00b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.104+0000 7f00a5000700 1 -- 172.21.15.12:0/8456418 shutdown_connections 2024-03-29T10:10:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.105+0000 7f00a5000700 1 -- 172.21.15.12:0/8456418 wait complete. 2024-03-29T10:10:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T10:10:16.125 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T10:10:16.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.330+0000 7f5c70800700 1 Processor -- start 2024-03-29T10:10:16.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.330+0000 7f5c70800700 1 -- start start 2024-03-29T10:10:16.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.331+0000 7f5c70800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5c680cd020 0x7f5c680cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:16.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.331+0000 7f5c70800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5c680b2070 0x7f5c680b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:16.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.331+0000 7f5c70800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c680cdae0 con 0x7f5c680cbb30 2024-03-29T10:10:16.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.331+0000 7f5c70800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c680cddc0 con 0x7f5c680cd020 2024-03-29T10:10:16.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.331+0000 7f5c70800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c680ce0a0 con 0x7f5c680b2070 2024-03-29T10:10:16.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.331+0000 7f5c6d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5c680cd020 0x7f5c680cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:16.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.331+0000 7f5c6de00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5c680cbb30 0x7f5c680cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51918/0 (socket says 172.21.15.12:51918) 2024-03-29T10:10:16.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.331+0000 7f5c6de00700 1 -- 172.21.15.12:0/2541381740 learned_addr learned my addr 172.21.15.12:0/2541381740 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:16.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.332+0000 7f5c6e800700 1 --2- 172.21.15.12:0/2541381740 >> v2:172.21.15.159:3300/0 conn(0x7f5c680b2070 0x7f5c680b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:16.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.332+0000 7f5c6d400700 1 -- 172.21.15.12:0/2541381740 >> v2:172.21.15.159:3300/0 conn(0x7f5c680b2070 msgr2=0x7f5c680b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:16.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.332+0000 7f5c6d400700 1 --2- 172.21.15.12:0/2541381740 >> v2:172.21.15.159:3300/0 conn(0x7f5c680b2070 0x7f5c680b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:16.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.332+0000 7f5c6ca00700 1 -- 172.21.15.12:0/2541381740 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370628213 0 0) 0x7f5c680cdae0 con 0x7f5c680cbb30 2024-03-29T10:10:16.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.332+0000 7f5c6d400700 1 -- 172.21.15.12:0/2541381740 >> v1:172.21.15.12:6789/0 conn(0x7f5c680cbb30 legacy=0x7f5c680cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:16.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.332+0000 7f5c6d400700 1 -- 172.21.15.12:0/2541381740 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6806dc20 con 0x7f5c680cd020 2024-03-29T10:10:16.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.332+0000 7f5c6d400700 1 --2- 172.21.15.12:0/2541381740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 0x7f5c680cd480 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5c680b1d60 tx=0x7f5c58009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55088a5e7347b8a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c6ca00700 1 -- 172.21.15.12:0/2541381740 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58011830 con 0x7f5c680cd020 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c6ca00700 1 -- 172.21.15.12:0/2541381740 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c58011990 con 0x7f5c680cd020 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c6ca00700 1 -- 172.21.15.12:0/2541381740 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58011d20 con 0x7f5c680cd020 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c70800700 1 -- 172.21.15.12:0/2541381740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 msgr2=0x7f5c680cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c70800700 1 --2- 172.21.15.12:0/2541381740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 0x7f5c680cd480 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5c680b1d60 tx=0x7f5c58009b20 comp rx=0 tx=0).stop 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c70800700 1 -- 172.21.15.12:0/2541381740 shutdown_connections 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c70800700 1 --2- 172.21.15.12:0/2541381740 >> v2:172.21.15.159:3300/0 conn(0x7f5c680b2070 0x7f5c680b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c70800700 1 --2- 172.21.15.12:0/2541381740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 0x7f5c680cd480 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c70800700 1 -- 172.21.15.12:0/2541381740 >> 172.21.15.12:0/2541381740 conn(0x7f5c680b0d70 msgr2=0x7f5c680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c70800700 1 -- 172.21.15.12:0/2541381740 shutdown_connections 2024-03-29T10:10:16.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.333+0000 7f5c70800700 1 -- 172.21.15.12:0/2541381740 wait complete. 2024-03-29T10:10:16.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.335+0000 7f5c70800700 1 Processor -- start 2024-03-29T10:10:16.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.335+0000 7f5c70800700 1 -- start start 2024-03-29T10:10:16.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c70800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c680b2070 0x7f5c680af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:16.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c70800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c680cbb30 0x7f5c68147fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:16.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c70800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 0x7f5c681484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c70800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c680adc50 con 0x7f5c680cd020 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c70800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c680adf30 con 0x7f5c680cbb30 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c70800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c680ae210 con 0x7f5c680b2070 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c6e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 0x7f5c681484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c6e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 0x7f5c681484f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45694/0 (socket says 172.21.15.12:45694) 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c6e800700 1 -- 172.21.15.12:0/2017866323 learned_addr learned my addr 172.21.15.12:0/2017866323 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c6d400700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c680cbb30 0x7f5c68147fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.336+0000 7f5c6de00700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c680b2070 0x7f5c680af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c6e800700 1 -- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c680b2070 msgr2=0x7f5c680af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c6e800700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c680b2070 0x7f5c680af3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c6e800700 1 -- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c680cbb30 msgr2=0x7f5c68147fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c6e800700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c680cbb30 0x7f5c68147fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c6e800700 1 -- 172.21.15.12:0/2017866323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c580094a0 con 0x7f5c680cd020 2024-03-29T10:10:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c6de00700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c680b2070 0x7f5c680af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:16.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c6e800700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 0x7f5c681484f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400efc0 tx=0x7f5c6400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:16.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c62000700 1 -- 172.21.15.12:0/2017866323 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64009ea0 con 0x7f5c680cd020 2024-03-29T10:10:16.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c62000700 1 -- 172.21.15.12:0/2017866323 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6400f040 con 0x7f5c680cd020 2024-03-29T10:10:16.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c62000700 1 -- 172.21.15.12:0/2017866323 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64014480 con 0x7f5c680cd020 2024-03-29T10:10:16.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c70800700 1 -- 172.21.15.12:0/2017866323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c680ae4f0 con 0x7f5c680cd020 2024-03-29T10:10:16.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.337+0000 7f5c70800700 1 -- 172.21.15.12:0/2017866323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c680afad0 con 0x7f5c680cd020 2024-03-29T10:10:16.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.338+0000 7f5c70800700 1 -- 172.21.15.12:0/2017866323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c680ae680 con 0x7f5c680cd020 2024-03-29T10:10:16.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.340+0000 7f5c62000700 1 -- 172.21.15.12:0/2017866323 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5c640145e0 con 0x7f5c680cd020 2024-03-29T10:10:16.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.341+0000 7f5c62000700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c4c06ade0 0x7f5c4c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:16.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.341+0000 7f5c62000700 1 -- 172.21.15.12:0/2017866323 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f5c6408a290 con 0x7f5c680cd020 2024-03-29T10:10:16.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.341+0000 7f5c6de00700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c4c06ade0 0x7f5c4c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:16.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.342+0000 7f5c6de00700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c4c06ade0 0x7f5c4c06d2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c000f60 tx=0x7f5c5c007ca0 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:16.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.346+0000 7f5c62000700 1 -- 172.21.15.12:0/2017866323 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c64055480 con 0x7f5c680cd020 2024-03-29T10:10:16.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:16.562+0000 7f5c70800700 1 -- 172.21.15.12:0/2017866323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5c680afad0 con 0x7f5c680cd020 2024-03-29T10:10:17.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.103+0000 7f5c62000700 1 -- 172.21.15.12:0/2017866323 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v92) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5c680afad0 con 0x7f5c680cd020 2024-03-29T10:10:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.111+0000 7f5c70800700 1 -- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c4c06ade0 msgr2=0x7f5c4c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.111+0000 7f5c70800700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c4c06ade0 0x7f5c4c06d2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c000f60 tx=0x7f5c5c007ca0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.111+0000 7f5c70800700 1 -- 172.21.15.12:0/2017866323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 msgr2=0x7f5c681484f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.111+0000 7f5c70800700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 0x7f5c681484f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400efc0 tx=0x7f5c6400c3a0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.111+0000 7f5c70800700 1 -- 172.21.15.12:0/2017866323 shutdown_connections 2024-03-29T10:10:17.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.111+0000 7f5c70800700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c4c06ade0 0x7f5c4c06d2a0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c000f60 tx=0x7f5c5c007ca0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.111+0000 7f5c70800700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c680cd020 0x7f5c681484f0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.111+0000 7f5c70800700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c680b2070 0x7f5c680af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.111+0000 7f5c70800700 1 --2- 172.21.15.12:0/2017866323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c680cbb30 0x7f5c68147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.111+0000 7f5c70800700 1 -- 172.21.15.12:0/2017866323 >> 172.21.15.12:0/2017866323 conn(0x7f5c680b0d70 msgr2=0x7f5c680b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:17.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.113+0000 7f5c70800700 1 -- 172.21.15.12:0/2017866323 shutdown_connections 2024-03-29T10:10:17.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.113+0000 7f5c70800700 1 -- 172.21.15.12:0/2017866323 wait complete. 2024-03-29T10:10:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T10:10:17.141 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T10:10:17.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.343+0000 7fd049000700 1 Processor -- start 2024-03-29T10:10:17.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.343+0000 7fd049000700 1 -- start start 2024-03-29T10:10:17.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.344+0000 7fd049000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd04415d9d0 0x7fd0440d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.344+0000 7fd049000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd0440d2540 0x7fd0440ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.344+0000 7fd049000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd044161420 con 0x7fd0440d16a0 2024-03-29T10:10:17.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.344+0000 7fd049000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04415edc0 con 0x7fd0440d2540 2024-03-29T10:10:17.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.344+0000 7fd049000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd044161700 con 0x7fd04415d9d0 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.344+0000 7fd042800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd0440d2540 0x7fd0440ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.344+0000 7fd041400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd0440d16a0 0x7fd0440d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51920/0 (socket says 172.21.15.12:51920) 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.344+0000 7fd041400700 1 -- 172.21.15.12:0/383245253 learned_addr learned my addr 172.21.15.12:0/383245253 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd041e00700 1 --2- 172.21.15.12:0/383245253 >> v2:172.21.15.159:3300/0 conn(0x7fd04415d9d0 0x7fd0440d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd042800700 1 -- 172.21.15.12:0/383245253 >> v2:172.21.15.159:3300/0 conn(0x7fd04415d9d0 msgr2=0x7fd0440d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd042800700 1 --2- 172.21.15.12:0/383245253 >> v2:172.21.15.159:3300/0 conn(0x7fd04415d9d0 0x7fd0440d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd042800700 1 -- 172.21.15.12:0/383245253 >> v1:172.21.15.12:6789/0 conn(0x7fd0440d16a0 legacy=0x7fd0440d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd042800700 1 -- 172.21.15.12:0/383245253 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0441619e0 con 0x7fd0440d2540 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd041e00700 1 --2- 172.21.15.12:0/383245253 >> v2:172.21.15.159:3300/0 conn(0x7fd04415d9d0 0x7fd0440d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd040a00700 1 -- 172.21.15.12:0/383245253 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121491938 0 0) 0x7fd044161420 con 0x7fd0440d16a0 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd042800700 1 --2- 172.21.15.12:0/383245253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0440d2540 0x7fd0440ce360 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fd0440d0ff0 tx=0x7fd03c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24d75f1adc17064 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd040a00700 1 -- 172.21.15.12:0/383245253 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c00e070 con 0x7fd0440d2540 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd040a00700 1 -- 172.21.15.12:0/383245253 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c00bbd0 con 0x7fd0440d2540 2024-03-29T10:10:17.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.345+0000 7fd040a00700 1 -- 172.21.15.12:0/383245253 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c012430 con 0x7fd0440d2540 2024-03-29T10:10:17.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.346+0000 7fd049000700 1 -- 172.21.15.12:0/383245253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0440d2540 msgr2=0x7fd0440ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.346+0000 7fd049000700 1 --2- 172.21.15.12:0/383245253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0440d2540 0x7fd0440ce360 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fd0440d0ff0 tx=0x7fd03c00b330 comp rx=0 tx=0).stop 2024-03-29T10:10:17.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.346+0000 7fd049000700 1 -- 172.21.15.12:0/383245253 shutdown_connections 2024-03-29T10:10:17.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.346+0000 7fd049000700 1 --2- 172.21.15.12:0/383245253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0440d2540 0x7fd0440ce360 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.346+0000 7fd049000700 1 --2- 172.21.15.12:0/383245253 >> v2:172.21.15.159:3300/0 conn(0x7fd04415d9d0 0x7fd0440d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.346+0000 7fd049000700 1 -- 172.21.15.12:0/383245253 >> 172.21.15.12:0/383245253 conn(0x7fd04408aff0 msgr2=0x7fd0440d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:17.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.346+0000 7fd049000700 1 -- 172.21.15.12:0/383245253 shutdown_connections 2024-03-29T10:10:17.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.346+0000 7fd049000700 1 -- 172.21.15.12:0/383245253 wait complete. 2024-03-29T10:10:17.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.348+0000 7fd049000700 1 Processor -- start 2024-03-29T10:10:17.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.348+0000 7fd049000700 1 -- start start 2024-03-29T10:10:17.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.349+0000 7fd049000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0440d16a0 0x7fd0440cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.349+0000 7fd049000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0440d2540 0x7fd04415fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.349+0000 7fd049000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd04415d9d0 0x7fd04414e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.349+0000 7fd049000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0441605c0 con 0x7fd04415d9d0 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.349+0000 7fd049000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0441608a0 con 0x7fd0440d16a0 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.349+0000 7fd049000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd044160b80 con 0x7fd0440d2540 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.349+0000 7fd042800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd04415d9d0 0x7fd04414e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.349+0000 7fd042800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd04415d9d0 0x7fd04414e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45714/0 (socket says 172.21.15.12:45714) 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.349+0000 7fd042800700 1 -- 172.21.15.12:0/3493104169 learned_addr learned my addr 172.21.15.12:0/3493104169 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.349+0000 7fd041e00700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0440d16a0 0x7fd0440cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd041400700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0440d2540 0x7fd04415fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd042800700 1 -- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0440d2540 msgr2=0x7fd04415fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd042800700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0440d2540 0x7fd04415fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd042800700 1 -- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0440d16a0 msgr2=0x7fd0440cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd042800700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0440d16a0 0x7fd0440cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd042800700 1 -- 172.21.15.12:0/3493104169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c009a70 con 0x7fd04415d9d0 2024-03-29T10:10:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd041400700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0440d2540 0x7fd04415fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd042800700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd04415d9d0 0x7fd04414e8e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd03c014040 tx=0x7fd03c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd03a000700 1 -- 172.21.15.12:0/3493104169 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c012430 con 0x7fd04415d9d0 2024-03-29T10:10:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd03a000700 1 -- 172.21.15.12:0/3493104169 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c020070 con 0x7fd04415d9d0 2024-03-29T10:10:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd03a000700 1 -- 172.21.15.12:0/3493104169 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c004280 con 0x7fd04415d9d0 2024-03-29T10:10:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd049000700 1 -- 172.21.15.12:0/3493104169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd044160e60 con 0x7fd04415d9d0 2024-03-29T10:10:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.350+0000 7fd049000700 1 -- 172.21.15.12:0/3493104169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04414ee20 con 0x7fd04415d9d0 2024-03-29T10:10:17.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.351+0000 7fd049000700 1 -- 172.21.15.12:0/3493104169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0440d1ca0 con 0x7fd04415d9d0 2024-03-29T10:10:17.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.353+0000 7fd03a000700 1 -- 172.21.15.12:0/3493104169 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd03c0043e0 con 0x7fd04415d9d0 2024-03-29T10:10:17.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.353+0000 7fd03a000700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd02406ade0 0x7fd02406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.354+0000 7fd03a000700 1 -- 172.21.15.12:0/3493104169 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fd03c022070 con 0x7fd04415d9d0 2024-03-29T10:10:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.354+0000 7fd041e00700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd02406ade0 0x7fd02406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.355+0000 7fd041e00700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd02406ade0 0x7fd02406d2a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd03400c900 tx=0x7fd034008040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:17.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.357+0000 7fd03a000700 1 -- 172.21.15.12:0/3493104169 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd03c056a10 con 0x7fd04415d9d0 2024-03-29T10:10:17.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.623+0000 7fd049000700 1 -- 172.21.15.12:0/3493104169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd044161380 con 0x7fd04415d9d0 2024-03-29T10:10:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.631+0000 7fd03a000700 1 -- 172.21.15.12:0/3493104169 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v35) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd03c02c0c0 con 0x7fd04415d9d0 2024-03-29T10:10:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.637+0000 7fd049000700 1 -- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd02406ade0 msgr2=0x7fd02406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.637+0000 7fd049000700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd02406ade0 0x7fd02406d2a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd03400c900 tx=0x7fd034008040 comp rx=0 tx=0).stop 2024-03-29T10:10:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.637+0000 7fd049000700 1 -- 172.21.15.12:0/3493104169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd04415d9d0 msgr2=0x7fd04414e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.637+0000 7fd049000700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd04415d9d0 0x7fd04414e8e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd03c014040 tx=0x7fd03c004ad0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.639+0000 7fd049000700 1 -- 172.21.15.12:0/3493104169 shutdown_connections 2024-03-29T10:10:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.639+0000 7fd049000700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd02406ade0 0x7fd02406d2a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.639+0000 7fd049000700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd04415d9d0 0x7fd04414e8e0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.639+0000 7fd049000700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0440d16a0 0x7fd0440cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.639+0000 7fd049000700 1 --2- 172.21.15.12:0/3493104169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0440d2540 0x7fd04415fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.639+0000 7fd049000700 1 -- 172.21.15.12:0/3493104169 >> 172.21.15.12:0/3493104169 conn(0x7fd04408aff0 msgr2=0x7fd04406c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.639+0000 7fd049000700 1 -- 172.21.15.12:0/3493104169 shutdown_connections 2024-03-29T10:10:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.639+0000 7fd049000700 1 -- 172.21.15.12:0/3493104169 wait complete. 2024-03-29T10:10:17.642 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 10 and data pool 11 2024-03-29T10:10:17.669 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:17.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.868+0000 7fdd66800700 1 Processor -- start 2024-03-29T10:10:17.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.868+0000 7fdd66800700 1 -- start start 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.869+0000 7fdd66800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdd600acda0 0x7fdd6015e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.869+0000 7fdd66800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdd600ad180 0x7fdd600b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.869+0000 7fdd66800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6015eb80 con 0x7fdd600b4690 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.869+0000 7fdd66800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd6006eef0 con 0x7fdd600acda0 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.869+0000 7fdd66800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd600ae7b0 con 0x7fdd600ad180 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.870+0000 7fdd5f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdd600acda0 0x7fdd6015e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.870+0000 7fdd5fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdd600b4690 0x7fdd600b4b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51932/0 (socket says 172.21.15.12:51932) 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.870+0000 7fdd5fe00700 1 -- 172.21.15.12:0/3525313317 learned_addr learned my addr 172.21.15.12:0/3525313317 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.870+0000 7fdd5ea00700 1 --2- 172.21.15.12:0/3525313317 >> v2:172.21.15.159:3300/0 conn(0x7fdd600ad180 0x7fdd600b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.870+0000 7fdd5f400700 1 -- 172.21.15.12:0/3525313317 >> v2:172.21.15.159:3300/0 conn(0x7fdd600ad180 msgr2=0x7fdd600b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.870+0000 7fdd5f400700 1 --2- 172.21.15.12:0/3525313317 >> v2:172.21.15.159:3300/0 conn(0x7fdd600ad180 0x7fdd600b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.870+0000 7fdd5f400700 1 -- 172.21.15.12:0/3525313317 >> v1:172.21.15.12:6789/0 conn(0x7fdd600b4690 legacy=0x7fdd600b4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.870+0000 7fdd5f400700 1 -- 172.21.15.12:0/3525313317 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd600aea90 con 0x7fdd600acda0 2024-03-29T10:10:17.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.870+0000 7fdd5e000700 1 -- 172.21.15.12:0/3525313317 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311819789 0 0) 0x7fdd6015eb80 con 0x7fdd600b4690 2024-03-29T10:10:17.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.870+0000 7fdd5f400700 1 --2- 172.21.15.12:0/3525313317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600acda0 0x7fdd6015e640 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fdd60066620 tx=0x7fdd50009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40d434135436fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:17.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd5e000700 1 -- 172.21.15.12:0/3525313317 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50011830 con 0x7fdd600acda0 2024-03-29T10:10:17.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd5e000700 1 -- 172.21.15.12:0/3525313317 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd50011990 con 0x7fdd600acda0 2024-03-29T10:10:17.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd5e000700 1 -- 172.21.15.12:0/3525313317 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50011d20 con 0x7fdd600acda0 2024-03-29T10:10:17.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd66800700 1 -- 172.21.15.12:0/3525313317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600acda0 msgr2=0x7fdd6015e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd66800700 1 --2- 172.21.15.12:0/3525313317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600acda0 0x7fdd6015e640 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fdd60066620 tx=0x7fdd50009b20 comp rx=0 tx=0).stop 2024-03-29T10:10:17.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd66800700 1 -- 172.21.15.12:0/3525313317 shutdown_connections 2024-03-29T10:10:17.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd66800700 1 --2- 172.21.15.12:0/3525313317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600acda0 0x7fdd6015e640 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd66800700 1 --2- 172.21.15.12:0/3525313317 >> v2:172.21.15.159:3300/0 conn(0x7fdd600ad180 0x7fdd600b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd66800700 1 -- 172.21.15.12:0/3525313317 >> 172.21.15.12:0/3525313317 conn(0x7fdd60072170 msgr2=0x7fdd6006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:17.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd66800700 1 -- 172.21.15.12:0/3525313317 shutdown_connections 2024-03-29T10:10:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.871+0000 7fdd66800700 1 -- 172.21.15.12:0/3525313317 wait complete. 2024-03-29T10:10:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.873+0000 7fdd66800700 1 Processor -- start 2024-03-29T10:10:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.873+0000 7fdd66800700 1 -- start start 2024-03-29T10:10:17.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.874+0000 7fdd66800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd600acda0 0x7fdd601f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.874+0000 7fdd66800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600ad180 0x7fdd60143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.874+0000 7fdd66800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd600b4690 0x7fdd601f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.874+0000 7fdd66800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6015a590 con 0x7fdd600ad180 2024-03-29T10:10:17.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.874+0000 7fdd66800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd60144490 con 0x7fdd600b4690 2024-03-29T10:10:17.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.874+0000 7fdd66800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd60151be0 con 0x7fdd600acda0 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.874+0000 7fdd5ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600ad180 0x7fdd60143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.874+0000 7fdd5ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600ad180 0x7fdd60143f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45738/0 (socket says 172.21.15.12:45738) 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.874+0000 7fdd5ea00700 1 -- 172.21.15.12:0/3094500434 learned_addr learned my addr 172.21.15.12:0/3094500434 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd5f400700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd600acda0 0x7fdd601f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd5fe00700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd600b4690 0x7fdd601f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd5ea00700 1 -- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd600acda0 msgr2=0x7fdd601f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd5ea00700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd600acda0 0x7fdd601f2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd5ea00700 1 -- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd600b4690 msgr2=0x7fdd601f6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd5ea00700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd600b4690 0x7fdd601f6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd5ea00700 1 -- 172.21.15.12:0/3094500434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500094a0 con 0x7fdd600ad180 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd5ea00700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600ad180 0x7fdd60143f50 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00e9e0 tx=0x7fdd4c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd57400700 1 -- 172.21.15.12:0/3094500434 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c009710 con 0x7fdd600ad180 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd57400700 1 -- 172.21.15.12:0/3094500434 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c00c850 con 0x7fdd600ad180 2024-03-29T10:10:17.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd66800700 1 -- 172.21.15.12:0/3094500434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6015a810 con 0x7fdd600ad180 2024-03-29T10:10:17.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.875+0000 7fdd57400700 1 -- 172.21.15.12:0/3094500434 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c00cbc0 con 0x7fdd600ad180 2024-03-29T10:10:17.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.876+0000 7fdd66800700 1 -- 172.21.15.12:0/3094500434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd601580a0 con 0x7fdd600ad180 2024-03-29T10:10:17.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.876+0000 7fdd66800700 1 -- 172.21.15.12:0/3094500434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6015a9a0 con 0x7fdd600ad180 2024-03-29T10:10:17.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.878+0000 7fdd57400700 1 -- 172.21.15.12:0/3094500434 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdd4c00cd20 con 0x7fdd600ad180 2024-03-29T10:10:17.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.878+0000 7fdd57400700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd4006adb0 0x7fdd4006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:17.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.879+0000 7fdd57400700 1 -- 172.21.15.12:0/3094500434 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd4c0895a0 con 0x7fdd600ad180 2024-03-29T10:10:17.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.879+0000 7fdd5f400700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd4006adb0 0x7fdd4006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:17.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.880+0000 7fdd5f400700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd4006adb0 0x7fdd4006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdd500090b0 tx=0x7fdd50008030 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:17.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:17.883+0000 7fdd57400700 1 -- 172.21.15.12:0/3094500434 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd4c054660 con 0x7fdd600ad180 2024-03-29T10:10:18.093 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-29T10:10:18.093+0000 7fe908200700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:10:18.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.096+0000 7fdd66800700 1 -- 172.21.15.12:0/3094500434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd60066e90 con 0x7fdd600ad180 2024-03-29T10:10:18.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.097+0000 7fdd57400700 1 -- 172.21.15.12:0/3094500434 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16113 (secure 0 0 0) 0x7fdd4c057c80 con 0x7fdd600ad180 2024-03-29T10:10:18.097 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:18.097 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":93,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:17.624640+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:16.040572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:16.605686+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 -- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd4006adb0 msgr2=0x7fdd4006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd4006adb0 0x7fdd4006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdd500090b0 tx=0x7fdd50008030 comp rx=0 tx=0).stop 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 -- 172.21.15.12:0/3094500434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600ad180 msgr2=0x7fdd60143f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600ad180 0x7fdd60143f50 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00e9e0 tx=0x7fdd4c00eda0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 -- 172.21.15.12:0/3094500434 shutdown_connections 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd4006adb0 0x7fdd4006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd600b4690 0x7fdd601f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd600acda0 0x7fdd601f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 --2- 172.21.15.12:0/3094500434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd600ad180 0x7fdd60143f50 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 -- 172.21.15.12:0/3094500434 >> 172.21.15.12:0/3094500434 conn(0x7fdd60072170 msgr2=0x7fdd60157540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:18.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 -- 172.21.15.12:0/3094500434 shutdown_connections 2024-03-29T10:10:18.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.100+0000 7fdd66800700 1 -- 172.21.15.12:0/3094500434 wait complete. 2024-03-29T10:10:18.119 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:18.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.318+0000 7f0923400700 1 Processor -- start 2024-03-29T10:10:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.318+0000 7f0923400700 1 -- start start 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.319+0000 7f0923400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f091c161ca0 0x7f091c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.319+0000 7f0923400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c0ceda0 con 0x7f091c160f10 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.319+0000 7f0923400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c1612f0 con 0x7f091c0d18b0 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.319+0000 7f0923400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c0d1d30 con 0x7f091c161ca0 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.319+0000 7f0920a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f091c160f10 0x7f091c0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51938/0 (socket says 172.21.15.12:51938) 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.319+0000 7f0920a00700 1 -- 172.21.15.12:0/976284790 learned_addr learned my addr 172.21.15.12:0/976284790 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f0921400700 1 --2- 172.21.15.12:0/976284790 >> v2:172.21.15.159:3300/0 conn(0x7f091c161ca0 0x7f091c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107868280 0 0) 0x7f091c0ceda0 con 0x7f091c160f10 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09040035b0 con 0x7f091c160f10 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482909756 0 0) 0x7f091c1612f0 con 0x7f091c0d18b0 2024-03-29T10:10:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f091c0ceda0 con 0x7f091c0d18b0 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3460242812 0 0) 0x7f09040035b0 con 0x7f091c160f10 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 >> v2:172.21.15.159:3300/0 conn(0x7f091c161ca0 msgr2=0x7f091c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 --2- 172.21.15.12:0/976284790 >> v2:172.21.15.159:3300/0 conn(0x7f091c161ca0 0x7f091c0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 >> v1:172.21.15.159:6789/0 conn(0x7f091c0d18b0 legacy=0x7f091c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c0ce470 con 0x7f091c160f10 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0910001670 con 0x7f091c160f10 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 >> v1:172.21.15.12:6789/0 conn(0x7f091c160f10 legacy=0x7f091c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904003be0 0x7f0904003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0904004850 0x7f0904004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0904005260 0x7f090400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c0d1d30 con 0x7f0904003be0 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090400c190 con 0x7f0904005260 2024-03-29T10:10:18.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090400c350 con 0x7f0904004850 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.320+0000 7f0920a00700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904003be0 0x7f0904003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f091be00700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0904004850 0x7f0904004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f0921400700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0904005260 0x7f090400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f0920a00700 1 -- 172.21.15.12:0/976284790 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0904004850 msgr2=0x7f0904004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f0920a00700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0904004850 0x7f0904004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f0920a00700 1 -- 172.21.15.12:0/976284790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0904005260 msgr2=0x7f090400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f0920a00700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0904005260 0x7f090400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f0920a00700 1 -- 172.21.15.12:0/976284790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09040035b0 con 0x7f0904003be0 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f0921400700 1 -- 172.21.15.12:0/976284790 reap_dead start 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f0920a00700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904003be0 0x7f0904003fe0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f091000c110 tx=0x7f0904005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42db052566f2b6de server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091000ccb0 con 0x7f0904003be0 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091000ce10 con 0x7f0904003be0 2024-03-29T10:10:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.321+0000 7f091b400700 1 -- 172.21.15.12:0/976284790 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910005ba0 con 0x7f0904003be0 2024-03-29T10:10:18.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.322+0000 7f0923400700 1 -- 172.21.15.12:0/976284790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904003be0 msgr2=0x7f0904003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.322+0000 7f0923400700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904003be0 0x7f0904003fe0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f091000c110 tx=0x7f0904005880 comp rx=0 tx=0).stop 2024-03-29T10:10:18.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.322+0000 7f0923400700 1 -- 172.21.15.12:0/976284790 shutdown_connections 2024-03-29T10:10:18.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.322+0000 7f0923400700 1 --2- 172.21.15.12:0/976284790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904003be0 0x7f0904003fe0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.322+0000 7f0923400700 1 -- 172.21.15.12:0/976284790 >> 172.21.15.12:0/976284790 conn(0x7f091c08aff0 msgr2=0x7f091c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:18.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.322+0000 7f0923400700 1 -- 172.21.15.12:0/976284790 shutdown_connections 2024-03-29T10:10:18.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.322+0000 7f0923400700 1 -- 172.21.15.12:0/976284790 wait complete. 2024-03-29T10:10:18.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.324+0000 7f0923400700 1 Processor -- start 2024-03-29T10:10:18.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.324+0000 7f0923400700 1 -- start start 2024-03-29T10:10:18.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.325+0000 7f0923400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0904003be0 0x7f091c1553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.325+0000 7f0923400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c15bc50 0x7f091c15c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.325+0000 7f0923400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f091c157750 0x7f091c157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.325+0000 7f0923400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c15c610 con 0x7f091c15bc50 2024-03-29T10:10:18.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.325+0000 7f0923400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c158a90 con 0x7f0904003be0 2024-03-29T10:10:18.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.325+0000 7f0923400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c1584d0 con 0x7f091c157750 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.325+0000 7f091be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c15bc50 0x7f091c15c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.325+0000 7f091be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c15bc50 0x7f091c15c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45754/0 (socket says 172.21.15.12:45754) 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.325+0000 7f091be00700 1 -- 172.21.15.12:0/1698336823 learned_addr learned my addr 172.21.15.12:0/1698336823 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f0921400700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f091c157750 0x7f091c157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f0920a00700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0904003be0 0x7f091c1553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f091be00700 1 -- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f091c157750 msgr2=0x7f091c157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f091be00700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f091c157750 0x7f091c157bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f091be00700 1 -- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0904003be0 msgr2=0x7f091c1553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f091be00700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0904003be0 0x7f091c1553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f091be00700 1 -- 172.21.15.12:0/1698336823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c1587b0 con 0x7f091c15bc50 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f091be00700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c15bc50 0x7f091c15c0d0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f090800f7e0 tx=0x7f090800fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f0918c00700 1 -- 172.21.15.12:0/1698336823 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09080052e0 con 0x7f091c15bc50 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f0918c00700 1 -- 172.21.15.12:0/1698336823 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908005440 con 0x7f091c15bc50 2024-03-29T10:10:18.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f0918c00700 1 -- 172.21.15.12:0/1698336823 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09080038c0 con 0x7f091c15bc50 2024-03-29T10:10:18.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f0923400700 1 -- 172.21.15.12:0/1698336823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091c0b6280 con 0x7f091c15bc50 2024-03-29T10:10:18.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.326+0000 7f0923400700 1 -- 172.21.15.12:0/1698336823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c155900 con 0x7f091c15bc50 2024-03-29T10:10:18.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.327+0000 7f0923400700 1 -- 172.21.15.12:0/1698336823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091c0b67a0 con 0x7f091c15bc50 2024-03-29T10:10:18.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.328+0000 7f0918c00700 1 -- 172.21.15.12:0/1698336823 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0908003a20 con 0x7f091c15bc50 2024-03-29T10:10:18.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.329+0000 7f0918c00700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08fc06adb0 0x7f08fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.329+0000 7f0918c00700 1 -- 172.21.15.12:0/1698336823 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0908089420 con 0x7f091c15bc50 2024-03-29T10:10:18.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.330+0000 7f0920a00700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08fc06adb0 0x7f08fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.331+0000 7f0920a00700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08fc06adb0 0x7f08fc06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f091000c110 tx=0x7f0910007000 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:18.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.334+0000 7f0918c00700 1 -- 172.21.15.12:0/1698336823 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f091c0b67a0 con 0x7f091c15bc50 2024-03-29T10:10:18.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.547+0000 7f0923400700 1 -- 172.21.15.12:0/1698336823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f091c0b67a0 con 0x7f091c15bc50 2024-03-29T10:10:18.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.549+0000 7f0918c00700 1 -- 172.21.15.12:0/1698336823 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16113 (secure 0 0 0) 0x7f091c0b67a0 con 0x7f091c15bc50 2024-03-29T10:10:18.548 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:18.549 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":93,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:17.624640+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:16.040572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:16.605686+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:18.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 -- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08fc06adb0 msgr2=0x7f08fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08fc06adb0 0x7f08fc06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f091000c110 tx=0x7f0910007000 comp rx=0 tx=0).stop 2024-03-29T10:10:18.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 -- 172.21.15.12:0/1698336823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c15bc50 msgr2=0x7f091c15c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c15bc50 0x7f091c15c0d0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f090800f7e0 tx=0x7f090800fba0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 -- 172.21.15.12:0/1698336823 shutdown_connections 2024-03-29T10:10:18.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08fc06adb0 0x7f08fc06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f091c157750 0x7f091c157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0904003be0 0x7f091c1553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 --2- 172.21.15.12:0/1698336823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c15bc50 0x7f091c15c0d0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 -- 172.21.15.12:0/1698336823 >> 172.21.15.12:0/1698336823 conn(0x7f091c08aff0 msgr2=0x7f091c0caa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:18.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 -- 172.21.15.12:0/1698336823 shutdown_connections 2024-03-29T10:10:18.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.551+0000 7f0923400700 1 -- 172.21.15.12:0/1698336823 wait complete. 2024-03-29T10:10:18.570 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T10:10:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.773+0000 7fae4ac00700 1 Processor -- start 2024-03-29T10:10:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.773+0000 7fae4ac00700 1 -- start start 2024-03-29T10:10:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.774+0000 7fae4ac00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fae441611b0 0x7fae4415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.774+0000 7fae4ac00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae4415f4d0 0x7fae440ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.774+0000 7fae4ac00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae440ce270 con 0x7fae440d21f0 2024-03-29T10:10:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.774+0000 7fae4ac00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4415b600 con 0x7fae441611b0 2024-03-29T10:10:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.774+0000 7fae4ac00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae440d2760 con 0x7fae4415f4d0 2024-03-29T10:10:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.775+0000 7fae43e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fae441611b0 0x7fae4415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.775+0000 7fae48c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fae440d21f0 0x7fae44161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51942/0 (socket says 172.21.15.12:51942) 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.775+0000 7fae48c00700 1 -- 172.21.15.12:0/2578317390 learned_addr learned my addr 172.21.15.12:0/2578317390 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.775+0000 7fae43400700 1 --2- 172.21.15.12:0/2578317390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae4415f4d0 0x7fae440ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.775+0000 7fae43e00700 1 -- 172.21.15.12:0/2578317390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae4415f4d0 msgr2=0x7fae440ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.775+0000 7fae43e00700 1 --2- 172.21.15.12:0/2578317390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae4415f4d0 0x7fae440ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.775+0000 7fae43e00700 1 -- 172.21.15.12:0/2578317390 >> v1:172.21.15.12:6789/0 conn(0x7fae440d21f0 legacy=0x7fae44161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.775+0000 7fae43e00700 1 -- 172.21.15.12:0/2578317390 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae440d2a40 con 0x7fae441611b0 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.775+0000 7fae42a00700 1 -- 172.21.15.12:0/2578317390 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981717104 0 0) 0x7fae440ce270 con 0x7fae440d21f0 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae43e00700 1 --2- 172.21.15.12:0/2578317390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae441611b0 0x7fae4415ef90 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fae34009840 tx=0x7fae34009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82531e2234746d0d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae42a00700 1 -- 172.21.15.12:0/2578317390 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae34011830 con 0x7fae441611b0 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae42a00700 1 -- 172.21.15.12:0/2578317390 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae34011990 con 0x7fae441611b0 2024-03-29T10:10:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae42a00700 1 -- 172.21.15.12:0/2578317390 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae34011d20 con 0x7fae441611b0 2024-03-29T10:10:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae4ac00700 1 -- 172.21.15.12:0/2578317390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae441611b0 msgr2=0x7fae4415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae4ac00700 1 --2- 172.21.15.12:0/2578317390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae441611b0 0x7fae4415ef90 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fae34009840 tx=0x7fae34009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae4ac00700 1 -- 172.21.15.12:0/2578317390 shutdown_connections 2024-03-29T10:10:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae4ac00700 1 --2- 172.21.15.12:0/2578317390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae441611b0 0x7fae4415ef90 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae4ac00700 1 --2- 172.21.15.12:0/2578317390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae4415f4d0 0x7fae440ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae4ac00700 1 -- 172.21.15.12:0/2578317390 >> 172.21.15.12:0/2578317390 conn(0x7fae4408aff0 msgr2=0x7fae440d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae4ac00700 1 -- 172.21.15.12:0/2578317390 shutdown_connections 2024-03-29T10:10:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.776+0000 7fae4ac00700 1 -- 172.21.15.12:0/2578317390 wait complete. 2024-03-29T10:10:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.779+0000 7fae4ac00700 1 Processor -- start 2024-03-29T10:10:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.779+0000 7fae4ac00700 1 -- start start 2024-03-29T10:10:18.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.780+0000 7fae4ac00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae440d21f0 0x7fae441f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.780+0000 7fae4ac00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae4415f4d0 0x7fae440b67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.780+0000 7fae4ac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae441611b0 0x7fae440b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.780+0000 7fae4ac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae440c9c20 con 0x7fae441611b0 2024-03-29T10:10:18.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.780+0000 7fae4ac00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae440b64e0 con 0x7fae4415f4d0 2024-03-29T10:10:18.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.780+0000 7fae4ac00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae441f63c0 con 0x7fae440d21f0 2024-03-29T10:10:18.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae48c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae441611b0 0x7fae440b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae43e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae440d21f0 0x7fae441f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae43400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae4415f4d0 0x7fae440b67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae43e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae440d21f0 0x7fae441f2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:53246/0 (socket says 172.21.15.12:53246) 2024-03-29T10:10:18.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae43e00700 1 -- 172.21.15.12:0/4134316581 learned_addr learned my addr 172.21.15.12:0/4134316581 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:18.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae48c00700 1 -- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae440d21f0 msgr2=0x7fae441f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae48c00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae440d21f0 0x7fae441f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae48c00700 1 -- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae4415f4d0 msgr2=0x7fae440b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:18.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae48c00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae4415f4d0 0x7fae440b67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:18.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae48c00700 1 -- 172.21.15.12:0/4134316581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae340094a0 con 0x7fae441611b0 2024-03-29T10:10:18.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.781+0000 7fae48c00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae441611b0 0x7fae440b8970 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00cc10 tx=0x7fae3c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:18.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.782+0000 7fae3be00700 1 -- 172.21.15.12:0/4134316581 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c00aa00 con 0x7fae441611b0 2024-03-29T10:10:18.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.782+0000 7fae3be00700 1 -- 172.21.15.12:0/4134316581 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c00ab80 con 0x7fae441611b0 2024-03-29T10:10:18.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.782+0000 7fae3be00700 1 -- 172.21.15.12:0/4134316581 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c0070d0 con 0x7fae441611b0 2024-03-29T10:10:18.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.782+0000 7fae4ac00700 1 -- 172.21.15.12:0/4134316581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae440b6ce0 con 0x7fae441611b0 2024-03-29T10:10:18.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.782+0000 7fae4ac00700 1 -- 172.21.15.12:0/4134316581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae441f35d0 con 0x7fae441611b0 2024-03-29T10:10:18.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.783+0000 7fae4ac00700 1 -- 172.21.15.12:0/4134316581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae441f6550 con 0x7fae441611b0 2024-03-29T10:10:18.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.784+0000 7fae3be00700 1 -- 172.21.15.12:0/4134316581 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fae3c007230 con 0x7fae441611b0 2024-03-29T10:10:18.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.785+0000 7fae3be00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae2406adb0 0x7fae2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:18.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.785+0000 7fae3be00700 1 -- 172.21.15.12:0/4134316581 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae3c088e20 con 0x7fae441611b0 2024-03-29T10:10:18.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.785+0000 7fae43e00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae2406adb0 0x7fae2406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:18.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.786+0000 7fae43e00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae2406adb0 0x7fae2406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae34009b90 tx=0x7fae3401c040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:18.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:18.789+0000 7fae3be00700 1 -- 172.21.15.12:0/4134316581 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae3c053ee0 con 0x7fae441611b0 2024-03-29T10:10:19.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.053+0000 7fae4ac00700 1 -- 172.21.15.12:0/4134316581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fae44067720 con 0x7fae441611b0 2024-03-29T10:10:19.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.636+0000 7fae3be00700 1 -- 172.21.15.12:0/4134316581 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v37) v1 ==== 118+0+0 (secure 0 0 0) 0x7fae3c018020 con 0x7fae441611b0 2024-03-29T10:10:19.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 -- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae2406adb0 msgr2=0x7fae2406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:19.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae2406adb0 0x7fae2406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae34009b90 tx=0x7fae3401c040 comp rx=0 tx=0).stop 2024-03-29T10:10:19.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 -- 172.21.15.12:0/4134316581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae441611b0 msgr2=0x7fae440b8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:19.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae441611b0 0x7fae440b8970 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00cc10 tx=0x7fae3c00cf20 comp rx=0 tx=0).stop 2024-03-29T10:10:19.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 -- 172.21.15.12:0/4134316581 shutdown_connections 2024-03-29T10:10:19.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fae2406adb0 0x7fae2406d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:19.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae441611b0 0x7fae440b8970 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:19.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae440d21f0 0x7fae441f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:19.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 --2- 172.21.15.12:0/4134316581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae4415f4d0 0x7fae440b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:19.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 -- 172.21.15.12:0/4134316581 >> 172.21.15.12:0/4134316581 conn(0x7fae4408aff0 msgr2=0x7fae4415f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:19.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 -- 172.21.15.12:0/4134316581 shutdown_connections 2024-03-29T10:10:19.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.639+0000 7fae4ac00700 1 -- 172.21.15.12:0/4134316581 wait complete. 2024-03-29T10:10:19.662 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T10:10:19.662 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T10:10:19.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.864+0000 7f4da9800700 1 Processor -- start 2024-03-29T10:10:19.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.864+0000 7f4da9800700 1 -- start start 2024-03-29T10:10:19.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.865+0000 7f4da9800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 0x7f4da40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:19.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.865+0000 7f4da9800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da40cc450 con 0x7f4da40cd020 2024-03-29T10:10:19.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.865+0000 7f4da9800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da40cdae0 con 0x7f4da40cd4a0 2024-03-29T10:10:19.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.865+0000 7f4da9800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da40cddc0 con 0x7f4da40cbb30 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.866+0000 7f4da2000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4da40cd020 0x7f4da40b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44192/0 (socket says 172.21.15.12:44192) 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.866+0000 7f4da2000700 1 -- 172.21.15.12:0/311546125 learned_addr learned my addr 172.21.15.12:0/311546125 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.866+0000 7f4da2a00700 1 --2- 172.21.15.12:0/311546125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 0x7f4da40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.866+0000 7f4da1600700 1 -- 172.21.15.12:0/311546125 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763964424 0 0) 0x7f4da40cc450 con 0x7f4da40cd020 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.866+0000 7f4da1600700 1 -- 172.21.15.12:0/311546125 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d880035b0 con 0x7f4da40cd020 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.866+0000 7f4da1600700 1 -- 172.21.15.12:0/311546125 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390377610 0 0) 0x7f4da40cdae0 con 0x7f4da40cd4a0 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.866+0000 7f4da1600700 1 -- 172.21.15.12:0/311546125 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da40cc450 con 0x7f4da40cd4a0 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.867+0000 7f4da1600700 1 -- 172.21.15.12:0/311546125 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1792508591 0 0) 0x7f4d880035b0 con 0x7f4da40cd020 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.867+0000 7f4da2a00700 1 -- 172.21.15.12:0/311546125 >> v1:172.21.15.159:6789/0 conn(0x7f4da40cd4a0 legacy=0x7f4da406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.867+0000 7f4da2a00700 1 -- 172.21.15.12:0/311546125 >> v1:172.21.15.12:6789/0 conn(0x7f4da40cd020 legacy=0x7f4da40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.867+0000 7f4da2a00700 1 -- 172.21.15.12:0/311546125 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da40ce0a0 con 0x7f4da40cbb30 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.867+0000 7f4da2a00700 1 --2- 172.21.15.12:0/311546125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 0x7f4da40cbf10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009870 tx=0x7f4d94009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9af97235718ece79 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.867+0000 7f4da1600700 1 -- 172.21.15.12:0/311546125 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d90001670 con 0x7f4da40cbb30 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.867+0000 7f4da1600700 1 -- 172.21.15.12:0/311546125 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d94004030 con 0x7f4da40cbb30 2024-03-29T10:10:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.867+0000 7f4da1600700 1 -- 172.21.15.12:0/311546125 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d940043c0 con 0x7f4da40cbb30 2024-03-29T10:10:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.868+0000 7f4da9800700 1 -- 172.21.15.12:0/311546125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 msgr2=0x7f4da40cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.868+0000 7f4da9800700 1 --2- 172.21.15.12:0/311546125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 0x7f4da40cbf10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009870 tx=0x7f4d94009b80 comp rx=0 tx=0).stop 2024-03-29T10:10:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.868+0000 7f4da9800700 1 -- 172.21.15.12:0/311546125 shutdown_connections 2024-03-29T10:10:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.868+0000 7f4da9800700 1 --2- 172.21.15.12:0/311546125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 0x7f4da40cbf10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.868+0000 7f4da9800700 1 -- 172.21.15.12:0/311546125 >> 172.21.15.12:0/311546125 conn(0x7f4da40b0d70 msgr2=0x7f4da40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.868+0000 7f4da9800700 1 -- 172.21.15.12:0/311546125 shutdown_connections 2024-03-29T10:10:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.868+0000 7f4da9800700 1 -- 172.21.15.12:0/311546125 wait complete. 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.870+0000 7f4da9800700 1 Processor -- start 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.870+0000 7f4da9800700 1 -- start start 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da9800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 0x7f4da4155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4da40cd020 0x7f4da41f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da9800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4da40cd4a0 0x7f4da41ed280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da9800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da4157b40 con 0x7f4da40cd020 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da9800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da41f4330 con 0x7f4da40cd4a0 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da9800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da41565a0 con 0x7f4da40cbb30 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4da40cd020 0x7f4da41f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4da40cd020 0x7f4da41f3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37902/0 (socket says 172.21.15.12:37902) 2024-03-29T10:10:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 0x7f4da4155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da2000700 1 -- 172.21.15.12:0/2966131241 learned_addr learned my addr 172.21.15.12:0/2966131241 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da3400700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4da40cd4a0 0x7f4da41ed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da2a00700 1 -- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4da40cd4a0 msgr2=0x7f4da41ed280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da2a00700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4da40cd4a0 0x7f4da41ed280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da2a00700 1 -- 172.21.15.12:0/2966131241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4da40cd020 msgr2=0x7f4da41f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da2a00700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4da40cd020 0x7f4da41f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.871+0000 7f4da2a00700 1 -- 172.21.15.12:0/2966131241 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d940094a0 con 0x7f4da40cbb30 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.872+0000 7f4da2a00700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 0x7f4da4155f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4d94015000 tx=0x7f4d94004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.872+0000 7f4d9aa00700 1 -- 172.21.15.12:0/2966131241 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94003cd0 con 0x7f4da40cbb30 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.872+0000 7f4d9aa00700 1 -- 172.21.15.12:0/2966131241 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9401f070 con 0x7f4da40cbb30 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.872+0000 7f4da9800700 1 -- 172.21.15.12:0/2966131241 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da4156880 con 0x7f4da40cbb30 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.872+0000 7f4d9aa00700 1 -- 172.21.15.12:0/2966131241 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94011870 con 0x7f4da40cbb30 2024-03-29T10:10:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.872+0000 7f4da9800700 1 -- 172.21.15.12:0/2966131241 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da4156e40 con 0x7f4da40cbb30 2024-03-29T10:10:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.873+0000 7f4da9800700 1 -- 172.21.15.12:0/2966131241 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da4067720 con 0x7f4da40cbb30 2024-03-29T10:10:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.874+0000 7f4d9aa00700 1 -- 172.21.15.12:0/2966131241 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4d94021030 con 0x7f4da40cbb30 2024-03-29T10:10:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.875+0000 7f4d9aa00700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4d8406adb0 0x7f4d8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.875+0000 7f4d9aa00700 1 -- 172.21.15.12:0/2966131241 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d9408a980 con 0x7f4da40cbb30 2024-03-29T10:10:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.875+0000 7f4da2000700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4d8406adb0 0x7f4d8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.876+0000 7f4da2000700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4d8406adb0 0x7f4d8406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d90001ab0 tx=0x7f4d90001a40 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:19.880+0000 7f4d9aa00700 1 -- 172.21.15.12:0/2966131241 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d94055be0 con 0x7f4da40cbb30 2024-03-29T10:10:20.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:20.140+0000 7f4da9800700 1 -- 172.21.15.12:0/2966131241 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4da415bda0 con 0x7f4da40cbb30 2024-03-29T10:10:21.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.135+0000 7f4d9aa00700 1 -- 172.21.15.12:0/2966131241 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v38) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4d94059200 con 0x7f4da40cbb30 2024-03-29T10:10:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 -- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4d8406adb0 msgr2=0x7f4d8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4d8406adb0 0x7f4d8406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d90001ab0 tx=0x7f4d90001a40 comp rx=0 tx=0).stop 2024-03-29T10:10:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 -- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 msgr2=0x7f4da4155f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 0x7f4da4155f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4d94015000 tx=0x7f4d94004750 comp rx=0 tx=0).stop 2024-03-29T10:10:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 -- 172.21.15.12:0/2966131241 shutdown_connections 2024-03-29T10:10:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4d8406adb0 0x7f4d8406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4da40cd4a0 0x7f4da41ed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da40cbb30 0x7f4da4155f70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 --2- 172.21.15.12:0/2966131241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4da40cd020 0x7f4da41f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 -- 172.21.15.12:0/2966131241 >> 172.21.15.12:0/2966131241 conn(0x7f4da40b0d70 msgr2=0x7f4da415bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 -- 172.21.15.12:0/2966131241 shutdown_connections 2024-03-29T10:10:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.138+0000 7f4da9800700 1 -- 172.21.15.12:0/2966131241 wait complete. 2024-03-29T10:10:21.158 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T10:10:21.159 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:21.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.360+0000 7f0dc9200700 1 Processor -- start 2024-03-29T10:10:21.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.361+0000 7f0dc9200700 1 -- start start 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.362+0000 7f0dc9200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0dc415b5d0 0x7f0dc40ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.362+0000 7f0dc9200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0dc40cb3f0 0x7f0dc40ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.362+0000 7f0dc9200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc40ae930 0x7f0dc40ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.362+0000 7f0dc9200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc415b9b0 con 0x7f0dc40cb3f0 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.362+0000 7f0dc9200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc415fd50 con 0x7f0dc415b5d0 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.362+0000 7f0dc9200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc415f140 con 0x7f0dc40ae930 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.362+0000 7f0dc1400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0dc40cb3f0 0x7f0dc40ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.362+0000 7f0dc1400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0dc40cb3f0 0x7f0dc40ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37912/0 (socket says 172.21.15.12:37912) 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.362+0000 7f0dc1400700 1 -- 172.21.15.12:0/2989360803 learned_addr learned my addr 172.21.15.12:0/2989360803 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc1e00700 1 --2- 172.21.15.12:0/2989360803 >> v2:172.21.15.159:3300/0 conn(0x7f0dc415b5d0 0x7f0dc40ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc2800700 1 --2- 172.21.15.12:0/2989360803 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc40ae930 0x7f0dc40ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:21.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc1400700 1 -- 172.21.15.12:0/2989360803 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc40ae930 msgr2=0x7f0dc40ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:21.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc1400700 1 --2- 172.21.15.12:0/2989360803 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc40ae930 0x7f0dc40ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc1400700 1 -- 172.21.15.12:0/2989360803 >> v2:172.21.15.159:3300/0 conn(0x7f0dc415b5d0 msgr2=0x7f0dc40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:21.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc1400700 1 --2- 172.21.15.12:0/2989360803 >> v2:172.21.15.159:3300/0 conn(0x7f0dc415b5d0 0x7f0dc40ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc1400700 1 -- 172.21.15.12:0/2989360803 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc415f420 con 0x7f0dc40cb3f0 2024-03-29T10:10:21.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc1400700 1 --2- 172.21.15.12:0/2989360803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40cb3f0 0x7f0dc40ac0c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0dc406b8b0 tx=0x7f0db0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e03b89d948d9eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:21.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc0a00700 1 -- 172.21.15.12:0/2989360803 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db000b870 con 0x7f0dc40cb3f0 2024-03-29T10:10:21.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc0a00700 1 -- 172.21.15.12:0/2989360803 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db000b9d0 con 0x7f0dc40cb3f0 2024-03-29T10:10:21.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.363+0000 7f0dc0a00700 1 -- 172.21.15.12:0/2989360803 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db000bd60 con 0x7f0dc40cb3f0 2024-03-29T10:10:21.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.364+0000 7f0dc9200700 1 -- 172.21.15.12:0/2989360803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40cb3f0 msgr2=0x7f0dc40ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.364+0000 7f0dc9200700 1 --2- 172.21.15.12:0/2989360803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40cb3f0 0x7f0dc40ac0c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0dc406b8b0 tx=0x7f0db0009bd0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.364+0000 7f0dc9200700 1 -- 172.21.15.12:0/2989360803 shutdown_connections 2024-03-29T10:10:21.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.364+0000 7f0dc9200700 1 --2- 172.21.15.12:0/2989360803 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc40ae930 0x7f0dc40ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.364+0000 7f0dc9200700 1 --2- 172.21.15.12:0/2989360803 >> v2:172.21.15.159:3300/0 conn(0x7f0dc415b5d0 0x7f0dc40ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.364+0000 7f0dc9200700 1 --2- 172.21.15.12:0/2989360803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40cb3f0 0x7f0dc40ac0c0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.364+0000 7f0dc9200700 1 -- 172.21.15.12:0/2989360803 >> 172.21.15.12:0/2989360803 conn(0x7f0dc40cf190 msgr2=0x7f0dc40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:21.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.364+0000 7f0dc9200700 1 -- 172.21.15.12:0/2989360803 shutdown_connections 2024-03-29T10:10:21.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.364+0000 7f0dc9200700 1 -- 172.21.15.12:0/2989360803 wait complete. 2024-03-29T10:10:21.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.366+0000 7f0dc9200700 1 Processor -- start 2024-03-29T10:10:21.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.366+0000 7f0dc9200700 1 -- start start 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.367+0000 7f0dc9200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dc40ae930 0x7f0dc4157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.367+0000 7f0dc9200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc415b5d0 0x7f0dc41f1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.367+0000 7f0dc9200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40b06e0 0x7f0dc4143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.367+0000 7f0dc9200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4157f60 con 0x7f0dc40b06e0 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.367+0000 7f0dc9200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4158200 con 0x7f0dc40ae930 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.367+0000 7f0dc9200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc41584e0 con 0x7f0dc415b5d0 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.367+0000 7f0dc2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40b06e0 0x7f0dc4143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.367+0000 7f0dc2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40b06e0 0x7f0dc4143660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37922/0 (socket says 172.21.15.12:37922) 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.367+0000 7f0dc2800700 1 -- 172.21.15.12:0/2950510299 learned_addr learned my addr 172.21.15.12:0/2950510299 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dc1e00700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dc40ae930 0x7f0dc4157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:21.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dc1400700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc415b5d0 0x7f0dc41f1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dc2800700 1 -- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc415b5d0 msgr2=0x7f0dc41f1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dc2800700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc415b5d0 0x7f0dc41f1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dc2800700 1 -- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dc40ae930 msgr2=0x7f0dc4157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dc2800700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dc40ae930 0x7f0dc4157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dc2800700 1 -- 172.21.15.12:0/2950510299 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0009550 con 0x7f0dc40b06e0 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dc2800700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40b06e0 0x7f0dc4143660 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc007e30 tx=0x7f0dbc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dba000700 1 -- 172.21.15.12:0/2950510299 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc00ede0 con 0x7f0dc40b06e0 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dba000700 1 -- 172.21.15.12:0/2950510299 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc014420 con 0x7f0dc40b06e0 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dba000700 1 -- 172.21.15.12:0/2950510299 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0147d0 con 0x7f0dc40b06e0 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dc9200700 1 -- 172.21.15.12:0/2950510299 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc40cce00 con 0x7f0dc40b06e0 2024-03-29T10:10:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.368+0000 7f0dc9200700 1 -- 172.21.15.12:0/2950510299 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc41f1b40 con 0x7f0dc40b06e0 2024-03-29T10:10:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.369+0000 7f0dc9200700 1 -- 172.21.15.12:0/2950510299 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc4066ff0 con 0x7f0dc40b06e0 2024-03-29T10:10:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.370+0000 7f0dba000700 1 -- 172.21.15.12:0/2950510299 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0dbc014930 con 0x7f0dc40b06e0 2024-03-29T10:10:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.371+0000 7f0dba000700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0da406adb0 0x7f0da406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.371+0000 7f0dba000700 1 -- 172.21.15.12:0/2950510299 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0dbc089cb0 con 0x7f0dc40b06e0 2024-03-29T10:10:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.372+0000 7f0dc1e00700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0da406adb0 0x7f0da406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:21.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.372+0000 7f0dc1e00700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0da406adb0 0x7f0da406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0db400b9b0 tx=0x7f0db4008040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:21.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.376+0000 7f0dba000700 1 -- 172.21.15.12:0/2950510299 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0dbc054ce0 con 0x7f0dc40b06e0 2024-03-29T10:10:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.635+0000 7f0dc9200700 1 -- 172.21.15.12:0/2950510299 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dc4067720 con 0x7f0dc40b06e0 2024-03-29T10:10:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.636+0000 7f0dba000700 1 -- 172.21.15.12:0/2950510299 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f0dbc009960 con 0x7f0dc40b06e0 2024-03-29T10:10:21.636 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:21.636 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":38,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/4026715411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":4026715411},{"type":"v1","addr":"172.21.15.159:6836","nonce":4026715411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/225205526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":225205526},{"type":"v1","addr":"172.21.15.12:6834","nonce":225205526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6918,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2358504162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2358504162},{"type":"v1","addr":"172.21.15.159:6837","nonce":2358504162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:17.624968+0000","modified":"2024-03-29T10:10:21.121687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6924},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6924":{"gid":6924,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3359577869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3359577869},{"type":"v1","addr":"172.21.15.12:6835","nonce":3359577869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-29T10:10:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 -- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0da406adb0 msgr2=0x7f0da406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0da406adb0 0x7f0da406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0db400b9b0 tx=0x7f0db4008040 comp rx=0 tx=0).stop 2024-03-29T10:10:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 -- 172.21.15.12:0/2950510299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40b06e0 msgr2=0x7f0dc4143660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40b06e0 0x7f0dc4143660 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc007e30 tx=0x7f0dbc00e3c0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 -- 172.21.15.12:0/2950510299 shutdown_connections 2024-03-29T10:10:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0da406adb0 0x7f0da406d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc40b06e0 0x7f0dc4143660 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dc40ae930 0x7f0dc4157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 --2- 172.21.15.12:0/2950510299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc415b5d0 0x7f0dc41f1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 -- 172.21.15.12:0/2950510299 >> 172.21.15.12:0/2950510299 conn(0x7f0dc40cf190 msgr2=0x7f0dc406c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 -- 172.21.15.12:0/2950510299 shutdown_connections 2024-03-29T10:10:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.638+0000 7f0dc9200700 1 -- 172.21.15.12:0/2950510299 wait complete. 2024-03-29T10:10:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 38 2024-03-29T10:10:21.657 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:21.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.855+0000 7feae7000700 1 Processor -- start 2024-03-29T10:10:21.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.855+0000 7feae7000700 1 -- start start 2024-03-29T10:10:21.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.856+0000 7feae7000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feae00d18b0 0x7feae00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.856+0000 7feae7000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae00ceda0 con 0x7feae0160f10 2024-03-29T10:10:21.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.856+0000 7feae7000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae01612f0 con 0x7feae0161590 2024-03-29T10:10:21.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.856+0000 7feae7000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae00d1d30 con 0x7feae00d18b0 2024-03-29T10:10:21.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.857+0000 7feadfe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7feae0160f10 0x7feae00d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44202/0 (socket says 172.21.15.12:44202) 2024-03-29T10:10:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.857+0000 7feadfe00700 1 -- 172.21.15.12:0/738070495 learned_addr learned my addr 172.21.15.12:0/738070495 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:21.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.857+0000 7feadf400700 1 --2- 172.21.15.12:0/738070495 >> v2:172.21.15.12:3300/0 conn(0x7feae00d18b0 0x7feae00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.857+0000 7feadf400700 1 -- 172.21.15.12:0/738070495 >> v1:172.21.15.159:6789/0 conn(0x7feae0161590 legacy=0x7feae00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.857+0000 7feadf400700 1 -- 172.21.15.12:0/738070495 >> v1:172.21.15.12:6789/0 conn(0x7feae0160f10 legacy=0x7feae00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.857+0000 7feadf400700 1 -- 172.21.15.12:0/738070495 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae00ce470 con 0x7feae00d18b0 2024-03-29T10:10:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.858+0000 7feadf400700 1 --2- 172.21.15.12:0/738070495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 0x7feae00cf230 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7feacc009840 tx=0x7feacc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649c8aad30384c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.858+0000 7feadea00700 1 -- 172.21.15.12:0/738070495 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc011830 con 0x7feae00d18b0 2024-03-29T10:10:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.858+0000 7feadea00700 1 -- 172.21.15.12:0/738070495 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc011990 con 0x7feae00d18b0 2024-03-29T10:10:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.858+0000 7feadea00700 1 -- 172.21.15.12:0/738070495 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc011d20 con 0x7feae00d18b0 2024-03-29T10:10:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.858+0000 7feae7000700 1 -- 172.21.15.12:0/738070495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 msgr2=0x7feae00cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.858+0000 7feae7000700 1 --2- 172.21.15.12:0/738070495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 0x7feae00cf230 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7feacc009840 tx=0x7feacc009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.858+0000 7feae7000700 1 -- 172.21.15.12:0/738070495 shutdown_connections 2024-03-29T10:10:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.858+0000 7feae7000700 1 --2- 172.21.15.12:0/738070495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 0x7feae00cf230 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.858+0000 7feae7000700 1 -- 172.21.15.12:0/738070495 >> 172.21.15.12:0/738070495 conn(0x7feae008aff0 msgr2=0x7feae00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.859+0000 7feae7000700 1 -- 172.21.15.12:0/738070495 shutdown_connections 2024-03-29T10:10:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.859+0000 7feae7000700 1 -- 172.21.15.12:0/738070495 wait complete. 2024-03-29T10:10:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.860+0000 7feae7000700 1 Processor -- start 2024-03-29T10:10:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.860+0000 7feae7000700 1 -- start start 2024-03-29T10:10:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.861+0000 7feae7000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 0x7feae0151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.861+0000 7feae7000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feae0160f10 0x7feae0151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.861+0000 7feae7000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae0161590 0x7feae015ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.861+0000 7feae7000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae0154960 con 0x7feae00d18b0 2024-03-29T10:10:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.861+0000 7feae7000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae0151ad0 con 0x7feae0160f10 2024-03-29T10:10:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.861+0000 7feae7000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae01526c0 con 0x7feae0161590 2024-03-29T10:10:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.861+0000 7feadfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 0x7feae0151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.861+0000 7feadfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 0x7feae0151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37940/0 (socket says 172.21.15.12:37940) 2024-03-29T10:10:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.861+0000 7feadfe00700 1 -- 172.21.15.12:0/3957904344 learned_addr learned my addr 172.21.15.12:0/3957904344 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7feadfe00700 1 -- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae0161590 msgr2=0x7feae015ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7feadfe00700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae0161590 0x7feae015ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7feadfe00700 1 -- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feae0160f10 msgr2=0x7feae0151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7feadfe00700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feae0160f10 0x7feae0151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7feadfe00700 1 -- 172.21.15.12:0/3957904344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc0094a0 con 0x7feae00d18b0 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7feadfe00700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 0x7feae0151590 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fead000c480 tx=0x7fead000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7fead7e00700 1 -- 172.21.15.12:0/3957904344 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0005d70 con 0x7feae00d18b0 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7fead7e00700 1 -- 172.21.15.12:0/3957904344 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead00070d0 con 0x7feae00d18b0 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7feae7000700 1 -- 172.21.15.12:0/3957904344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae01529a0 con 0x7feae00d18b0 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7fead7e00700 1 -- 172.21.15.12:0/3957904344 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0007440 con 0x7feae00d18b0 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.862+0000 7feae7000700 1 -- 172.21.15.12:0/3957904344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae0152f60 con 0x7feae00d18b0 2024-03-29T10:10:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.864+0000 7feae7000700 1 -- 172.21.15.12:0/3957904344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae0067720 con 0x7feae00d18b0 2024-03-29T10:10:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.865+0000 7fead7e00700 1 -- 172.21.15.12:0/3957904344 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fead00075a0 con 0x7feae00d18b0 2024-03-29T10:10:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.865+0000 7fead7e00700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac006ae00 0x7feac006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.865+0000 7fead7e00700 1 -- 172.21.15.12:0/3957904344 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fead0088830 con 0x7feae00d18b0 2024-03-29T10:10:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.866+0000 7feadf400700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac006ae00 0x7feac006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.866+0000 7feadf400700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac006ae00 0x7feac006d2c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feae00b95d0 tx=0x7feacc010000 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:21.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:21.870+0000 7fead7e00700 1 -- 172.21.15.12:0/3957904344 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fead0053a90 con 0x7feae00d18b0 2024-03-29T10:10:22.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.083+0000 7feae7000700 1 -- 172.21.15.12:0/3957904344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feae008b760 con 0x7feae00d18b0 2024-03-29T10:10:22.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.084+0000 7fead7e00700 1 -- 172.21.15.12:0/3957904344 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fead00570b0 con 0x7feae00d18b0 2024-03-29T10:10:22.084 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:22.084 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":94,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:18.627118+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:16.040572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:16.605686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 -- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac006ae00 msgr2=0x7feac006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac006ae00 0x7feac006d2c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feae00b95d0 tx=0x7feacc010000 comp rx=0 tx=0).stop 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 -- 172.21.15.12:0/3957904344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 msgr2=0x7feae0151590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 0x7feae0151590 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fead000c480 tx=0x7fead000c840 comp rx=0 tx=0).stop 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 -- 172.21.15.12:0/3957904344 shutdown_connections 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac006ae00 0x7feac006d2c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae0161590 0x7feae015ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae00d18b0 0x7feae0151590 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 --2- 172.21.15.12:0/3957904344 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feae0160f10 0x7feae0151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 -- 172.21.15.12:0/3957904344 >> 172.21.15.12:0/3957904344 conn(0x7feae008aff0 msgr2=0x7feae006d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 -- 172.21.15.12:0/3957904344 shutdown_connections 2024-03-29T10:10:22.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.087+0000 7feae7000700 1 -- 172.21.15.12:0/3957904344 wait complete. 2024-03-29T10:10:22.104 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T10:10:22.105 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:10:22.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.303+0000 7f3274e00700 1 Processor -- start 2024-03-29T10:10:22.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.303+0000 7f3274e00700 1 -- start start 2024-03-29T10:10:22.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.304+0000 7f3274e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270160f10 0x7f32700d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.305+0000 7f3274e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f32700cf230 0x7f32700d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.305+0000 7f3274e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32700ceda0 con 0x7f3270161590 2024-03-29T10:10:22.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.305+0000 7f3274e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32700d23d0 con 0x7f32700cf230 2024-03-29T10:10:22.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.305+0000 7f3274e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32701612f0 con 0x7f3270160f10 2024-03-29T10:10:22.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.305+0000 7f326de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270160f10 0x7f32700d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.305+0000 7f326d400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f32700cf230 0x7f32700d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.305+0000 7f326e800700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f3270161590 0x7f32700b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:60532/0 (socket says 172.21.15.12:60532) 2024-03-29T10:10:22.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.305+0000 7f326e800700 1 -- 172.21.15.12:0/475829862 learned_addr learned my addr 172.21.15.12:0/475829862 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.306+0000 7f326de00700 1 -- 172.21.15.12:0/475829862 >> v2:172.21.15.159:3300/0 conn(0x7f32700cf230 msgr2=0x7f32700d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.306+0000 7f326de00700 1 --2- 172.21.15.12:0/475829862 >> v2:172.21.15.159:3300/0 conn(0x7f32700cf230 0x7f32700d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.306+0000 7f326de00700 1 -- 172.21.15.12:0/475829862 >> v1:172.21.15.159:6789/0 conn(0x7f3270161590 legacy=0x7f32700b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.306+0000 7f326de00700 1 -- 172.21.15.12:0/475829862 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32700ce470 con 0x7f3270160f10 2024-03-29T10:10:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.306+0000 7f326ca00700 1 -- 172.21.15.12:0/475829862 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753336097 0 0) 0x7f32700ceda0 con 0x7f3270161590 2024-03-29T10:10:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.306+0000 7f326d400700 1 --2- 172.21.15.12:0/475829862 >> v2:172.21.15.159:3300/0 conn(0x7f32700cf230 0x7f32700d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.306+0000 7f326de00700 1 --2- 172.21.15.12:0/475829862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270160f10 0x7f32700d18b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3260009840 tx=0x7f3260009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50926a6636c64cba server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.306+0000 7f326ca00700 1 -- 172.21.15.12:0/475829862 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260011830 con 0x7f3270160f10 2024-03-29T10:10:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.307+0000 7f326ca00700 1 -- 172.21.15.12:0/475829862 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3260011990 con 0x7f3270160f10 2024-03-29T10:10:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.307+0000 7f326ca00700 1 -- 172.21.15.12:0/475829862 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260011d20 con 0x7f3270160f10 2024-03-29T10:10:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.307+0000 7f3274e00700 1 -- 172.21.15.12:0/475829862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270160f10 msgr2=0x7f32700d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.307+0000 7f3274e00700 1 --2- 172.21.15.12:0/475829862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270160f10 0x7f32700d18b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3260009840 tx=0x7f3260009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.307+0000 7f3274e00700 1 -- 172.21.15.12:0/475829862 shutdown_connections 2024-03-29T10:10:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.307+0000 7f3274e00700 1 --2- 172.21.15.12:0/475829862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270160f10 0x7f32700d18b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.307+0000 7f3274e00700 1 --2- 172.21.15.12:0/475829862 >> v2:172.21.15.159:3300/0 conn(0x7f32700cf230 0x7f32700d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.307+0000 7f3274e00700 1 -- 172.21.15.12:0/475829862 >> 172.21.15.12:0/475829862 conn(0x7f327008aff0 msgr2=0x7f32700d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.307+0000 7f3274e00700 1 -- 172.21.15.12:0/475829862 shutdown_connections 2024-03-29T10:10:22.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.307+0000 7f3274e00700 1 -- 172.21.15.12:0/475829862 wait complete. 2024-03-29T10:10:22.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.309+0000 7f3274e00700 1 Processor -- start 2024-03-29T10:10:22.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.309+0000 7f3274e00700 1 -- start start 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.310+0000 7f3274e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32700cf230 0x7f3270156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.310+0000 7f3274e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3270160f10 0x7f32700c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.310+0000 7f3274e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270161590 0x7f3270151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.310+0000 7f3274e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327015bb30 con 0x7f32700cf230 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.310+0000 7f3274e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327015bd40 con 0x7f3270160f10 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.310+0000 7f3274e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327015c020 con 0x7f3270161590 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.310+0000 7f326de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32700cf230 0x7f3270156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32700cf230 0x7f3270156ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37942/0 (socket says 172.21.15.12:37942) 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326de00700 1 -- 172.21.15.12:0/2708081216 learned_addr learned my addr 172.21.15.12:0/2708081216 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326e800700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270161590 0x7f3270151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326d400700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3270160f10 0x7f32700c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326de00700 1 -- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270161590 msgr2=0x7f3270151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326de00700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270161590 0x7f3270151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326de00700 1 -- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3270160f10 msgr2=0x7f32700c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326de00700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3270160f10 0x7f32700c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326de00700 1 -- 172.21.15.12:0/2708081216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c0094a0 con 0x7f32700cf230 2024-03-29T10:10:22.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326d400700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3270160f10 0x7f32700c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:22.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.311+0000 7f326de00700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32700cf230 0x7f3270156ad0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f3260014040 tx=0x7f3260004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:22.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.312+0000 7f3266000700 1 -- 172.21.15.12:0/2708081216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32600038a0 con 0x7f32700cf230 2024-03-29T10:10:22.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.312+0000 7f3266000700 1 -- 172.21.15.12:0/2708081216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326001e070 con 0x7f32700cf230 2024-03-29T10:10:22.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.312+0000 7f3266000700 1 -- 172.21.15.12:0/2708081216 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260003c20 con 0x7f32700cf230 2024-03-29T10:10:22.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.312+0000 7f3274e00700 1 -- 172.21.15.12:0/2708081216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32600094a0 con 0x7f32700cf230 2024-03-29T10:10:22.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.312+0000 7f3274e00700 1 -- 172.21.15.12:0/2708081216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327015c5e0 con 0x7f32700cf230 2024-03-29T10:10:22.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.313+0000 7f3274e00700 1 -- 172.21.15.12:0/2708081216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3270067720 con 0x7f32700cf230 2024-03-29T10:10:22.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.314+0000 7f3266000700 1 -- 172.21.15.12:0/2708081216 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f326002d450 con 0x7f32700cf230 2024-03-29T10:10:22.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.315+0000 7f3266000700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f325006adb0 0x7f325006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.315+0000 7f3266000700 1 -- 172.21.15.12:0/2708081216 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f326008a490 con 0x7f32700cf230 2024-03-29T10:10:22.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.315+0000 7f326d400700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f325006adb0 0x7f325006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.316+0000 7f326d400700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f325006adb0 0x7f325006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f325c009d20 tx=0x7f325c008040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.319+0000 7f3266000700 1 -- 172.21.15.12:0/2708081216 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32600555f0 con 0x7f32700cf230 2024-03-29T10:10:22.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.530+0000 7f3274e00700 1 -- 172.21.15.12:0/2708081216 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32700cf6b0 con 0x7f325006adb0 2024-03-29T10:10:22.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.541+0000 7f3266000700 1 -- 172.21.15.12:0/2708081216 <== mgr.5954 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288876 (secure 0 0 0) 0x7f325c009970 con 0x7f325006adb0 2024-03-29T10:10:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.545+0000 7f3274e00700 1 -- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f325006adb0 msgr2=0x7f325006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.545+0000 7f3274e00700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f325006adb0 0x7f325006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f325c009d20 tx=0x7f325c008040 comp rx=0 tx=0).stop 2024-03-29T10:10:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.545+0000 7f3274e00700 1 -- 172.21.15.12:0/2708081216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32700cf230 msgr2=0x7f3270156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.545+0000 7f3274e00700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32700cf230 0x7f3270156ad0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f3260014040 tx=0x7f3260004750 comp rx=0 tx=0).stop 2024-03-29T10:10:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.546+0000 7f3274e00700 1 -- 172.21.15.12:0/2708081216 shutdown_connections 2024-03-29T10:10:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.546+0000 7f3274e00700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f325006adb0 0x7f325006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.546+0000 7f3274e00700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3270161590 0x7f3270151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.546+0000 7f3274e00700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32700cf230 0x7f3270156ad0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.546+0000 7f3274e00700 1 --2- 172.21.15.12:0/2708081216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3270160f10 0x7f32700c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.546+0000 7f3274e00700 1 -- 172.21.15.12:0/2708081216 >> 172.21.15.12:0/2708081216 conn(0x7f327008aff0 msgr2=0x7f32700cad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.546+0000 7f3274e00700 1 -- 172.21.15.12:0/2708081216 shutdown_connections 2024-03-29T10:10:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.546+0000 7f3274e00700 1 -- 172.21.15.12:0/2708081216 wait complete. 2024-03-29T10:10:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:10:22.563 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-29T10:10:20.820482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":311532,"kb_used_data":10084,"kb_used_omap":12,"kb_used_meta":301299,"kb_avail":754663188,"statfs":{"total":773094113280,"available":772775104512,"internally_reserved":0,"allocated":10326016,"data_stored":5483265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":308530745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.722760"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.667006+0000","last_change":"2024-03-29T10:10:17.663972+0000","last_active":"2024-03-29T10:10:17.667006+0000","last_peered":"2024-03-29T10:10:17.667006+0000","last_clean":"2024-03-29T10:10:17.667006+0000","last_became_active":"2024-03-29T10:10:17.663691+0000","last_became_peered":"2024-03-29T10:10:17.663691+0000","last_unstale":"2024-03-29T10:10:17.667006+0000","last_undegraded":"2024-03-29T10:10:17.667006+0000","last_fullsized":"2024-03-29T10:10:17.667006+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:32:49.333388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637719+0000","last_change":"2024-03-29T10:10:17.654165+0000","last_active":"2024-03-29T10:10:18.637719+0000","last_peered":"2024-03-29T10:10:18.637719+0000","last_clean":"2024-03-29T10:10:18.637719+0000","last_became_active":"2024-03-29T10:10:17.653207+0000","last_became_peered":"2024-03-29T10:10:17.653207+0000","last_unstale":"2024-03-29T10:10:18.637719+0000","last_undegraded":"2024-03-29T10:10:18.637719+0000","last_fullsized":"2024-03-29T10:10:18.637719+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:47.601889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.648762+0000","last_change":"2024-03-29T10:10:17.645506+0000","last_active":"2024-03-29T10:10:17.648762+0000","last_peered":"2024-03-29T10:10:17.648762+0000","last_clean":"2024-03-29T10:10:17.648762+0000","last_became_active":"2024-03-29T10:10:17.645244+0000","last_became_peered":"2024-03-29T10:10:17.645244+0000","last_unstale":"2024-03-29T10:10:17.648762+0000","last_undegraded":"2024-03-29T10:10:17.648762+0000","last_fullsized":"2024-03-29T10:10:17.648762+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:07.975443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631810+0000","last_change":"2024-03-29T10:10:17.654242+0000","last_active":"2024-03-29T10:10:18.631810+0000","last_peered":"2024-03-29T10:10:18.631810+0000","last_clean":"2024-03-29T10:10:18.631810+0000","last_became_active":"2024-03-29T10:10:17.653740+0000","last_became_peered":"2024-03-29T10:10:17.653740+0000","last_unstale":"2024-03-29T10:10:18.631810+0000","last_undegraded":"2024-03-29T10:10:18.631810+0000","last_fullsized":"2024-03-29T10:10:18.631810+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:42.641698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663602+0000","last_change":"2024-03-29T10:10:17.656879+0000","last_active":"2024-03-29T10:10:17.663602+0000","last_peered":"2024-03-29T10:10:17.663602+0000","last_clean":"2024-03-29T10:10:17.663602+0000","last_became_active":"2024-03-29T10:10:17.656560+0000","last_became_peered":"2024-03-29T10:10:17.656560+0000","last_unstale":"2024-03-29T10:10:17.663602+0000","last_undegraded":"2024-03-29T10:10:17.663602+0000","last_fullsized":"2024-03-29T10:10:17.663602+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:35.582327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637017+0000","last_change":"2024-03-29T10:10:17.653703+0000","last_active":"2024-03-29T10:10:18.637017+0000","last_peered":"2024-03-29T10:10:18.637017+0000","last_clean":"2024-03-29T10:10:18.637017+0000","last_became_active":"2024-03-29T10:10:17.653398+0000","last_became_peered":"2024-03-29T10:10:17.653398+0000","last_unstale":"2024-03-29T10:10:18.637017+0000","last_undegraded":"2024-03-29T10:10:18.637017+0000","last_fullsized":"2024-03-29T10:10:18.637017+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:04.588448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636518+0000","last_change":"2024-03-29T10:10:17.642071+0000","last_active":"2024-03-29T10:10:18.636518+0000","last_peered":"2024-03-29T10:10:18.636518+0000","last_clean":"2024-03-29T10:10:18.636518+0000","last_became_active":"2024-03-29T10:10:17.638378+0000","last_became_peered":"2024-03-29T10:10:17.638378+0000","last_unstale":"2024-03-29T10:10:18.636518+0000","last_undegraded":"2024-03-29T10:10:18.636518+0000","last_fullsized":"2024-03-29T10:10:18.636518+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:08.435381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637107+0000","last_change":"2024-03-29T10:10:17.640398+0000","last_active":"2024-03-29T10:10:18.637107+0000","last_peered":"2024-03-29T10:10:18.637107+0000","last_clean":"2024-03-29T10:10:18.637107+0000","last_became_active":"2024-03-29T10:10:17.640044+0000","last_became_peered":"2024-03-29T10:10:17.640044+0000","last_unstale":"2024-03-29T10:10:18.637107+0000","last_undegraded":"2024-03-29T10:10:18.637107+0000","last_fullsized":"2024-03-29T10:10:18.637107+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:41:43.134146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664433+0000","last_change":"2024-03-29T10:10:17.659145+0000","last_active":"2024-03-29T10:10:17.664433+0000","last_peered":"2024-03-29T10:10:17.664433+0000","last_clean":"2024-03-29T10:10:17.664433+0000","last_became_active":"2024-03-29T10:10:17.658026+0000","last_became_peered":"2024-03-29T10:10:17.658026+0000","last_unstale":"2024-03-29T10:10:17.664433+0000","last_undegraded":"2024-03-29T10:10:17.664433+0000","last_fullsized":"2024-03-29T10:10:17.664433+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:50.247091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.662851+0000","last_change":"2024-03-29T10:10:17.659927+0000","last_active":"2024-03-29T10:10:17.662851+0000","last_peered":"2024-03-29T10:10:17.662851+0000","last_clean":"2024-03-29T10:10:17.662851+0000","last_became_active":"2024-03-29T10:10:17.659572+0000","last_became_peered":"2024-03-29T10:10:17.659572+0000","last_unstale":"2024-03-29T10:10:17.662851+0000","last_undegraded":"2024-03-29T10:10:17.662851+0000","last_fullsized":"2024-03-29T10:10:17.662851+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:13.147091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.662492+0000","last_change":"2024-03-29T10:10:17.657259+0000","last_active":"2024-03-29T10:10:17.662492+0000","last_peered":"2024-03-29T10:10:17.662492+0000","last_clean":"2024-03-29T10:10:17.662492+0000","last_became_active":"2024-03-29T10:10:17.656924+0000","last_became_peered":"2024-03-29T10:10:17.656924+0000","last_unstale":"2024-03-29T10:10:17.662492+0000","last_undegraded":"2024-03-29T10:10:17.662492+0000","last_fullsized":"2024-03-29T10:10:17.662492+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:33:07.032980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631834+0000","last_change":"2024-03-29T10:10:17.659739+0000","last_active":"2024-03-29T10:10:18.631834+0000","last_peered":"2024-03-29T10:10:18.631834+0000","last_clean":"2024-03-29T10:10:18.631834+0000","last_became_active":"2024-03-29T10:10:17.659488+0000","last_became_peered":"2024-03-29T10:10:17.659488+0000","last_unstale":"2024-03-29T10:10:18.631834+0000","last_undegraded":"2024-03-29T10:10:18.631834+0000","last_fullsized":"2024-03-29T10:10:18.631834+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:45:01.639827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.657931+0000","last_change":"2024-03-29T10:10:17.650043+0000","last_active":"2024-03-29T10:10:17.657931+0000","last_peered":"2024-03-29T10:10:17.657931+0000","last_clean":"2024-03-29T10:10:17.657931+0000","last_became_active":"2024-03-29T10:10:17.649701+0000","last_became_peered":"2024-03-29T10:10:17.649701+0000","last_unstale":"2024-03-29T10:10:17.657931+0000","last_undegraded":"2024-03-29T10:10:17.657931+0000","last_fullsized":"2024-03-29T10:10:17.657931+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:26:39.092036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664479+0000","last_change":"2024-03-29T10:10:17.659628+0000","last_active":"2024-03-29T10:10:17.664479+0000","last_peered":"2024-03-29T10:10:17.664479+0000","last_clean":"2024-03-29T10:10:17.664479+0000","last_became_active":"2024-03-29T10:10:17.656796+0000","last_became_peered":"2024-03-29T10:10:17.656796+0000","last_unstale":"2024-03-29T10:10:17.664479+0000","last_undegraded":"2024-03-29T10:10:17.664479+0000","last_fullsized":"2024-03-29T10:10:17.664479+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:13:14.570934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664640+0000","last_change":"2024-03-29T10:10:17.657742+0000","last_active":"2024-03-29T10:10:17.664640+0000","last_peered":"2024-03-29T10:10:17.664640+0000","last_clean":"2024-03-29T10:10:17.664640+0000","last_became_active":"2024-03-29T10:10:17.657470+0000","last_became_peered":"2024-03-29T10:10:17.657470+0000","last_unstale":"2024-03-29T10:10:17.664640+0000","last_undegraded":"2024-03-29T10:10:17.664640+0000","last_fullsized":"2024-03-29T10:10:17.664640+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:58.382328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.656812+0000","last_change":"2024-03-29T10:10:17.651786+0000","last_active":"2024-03-29T10:10:17.656812+0000","last_peered":"2024-03-29T10:10:17.656812+0000","last_clean":"2024-03-29T10:10:17.656812+0000","last_became_active":"2024-03-29T10:10:17.651524+0000","last_became_peered":"2024-03-29T10:10:17.651524+0000","last_unstale":"2024-03-29T10:10:17.656812+0000","last_undegraded":"2024-03-29T10:10:17.656812+0000","last_fullsized":"2024-03-29T10:10:17.656812+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:27.677477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663718+0000","last_change":"2024-03-29T10:10:17.657712+0000","last_active":"2024-03-29T10:10:17.663718+0000","last_peered":"2024-03-29T10:10:17.663718+0000","last_clean":"2024-03-29T10:10:17.663718+0000","last_became_active":"2024-03-29T10:10:17.656954+0000","last_became_peered":"2024-03-29T10:10:17.656954+0000","last_unstale":"2024-03-29T10:10:17.663718+0000","last_undegraded":"2024-03-29T10:10:17.663718+0000","last_fullsized":"2024-03-29T10:10:17.663718+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:44.085465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.632229+0000","last_change":"2024-03-29T10:10:17.643735+0000","last_active":"2024-03-29T10:10:18.632229+0000","last_peered":"2024-03-29T10:10:18.632229+0000","last_clean":"2024-03-29T10:10:18.632229+0000","last_became_active":"2024-03-29T10:10:17.643486+0000","last_became_peered":"2024-03-29T10:10:17.643486+0000","last_unstale":"2024-03-29T10:10:18.632229+0000","last_undegraded":"2024-03-29T10:10:18.632229+0000","last_fullsized":"2024-03-29T10:10:18.632229+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:47:17.241189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637462+0000","last_change":"2024-03-29T10:10:17.651977+0000","last_active":"2024-03-29T10:10:18.637462+0000","last_peered":"2024-03-29T10:10:18.637462+0000","last_clean":"2024-03-29T10:10:18.637462+0000","last_became_active":"2024-03-29T10:10:17.649831+0000","last_became_peered":"2024-03-29T10:10:17.649831+0000","last_unstale":"2024-03-29T10:10:18.637462+0000","last_undegraded":"2024-03-29T10:10:18.637462+0000","last_fullsized":"2024-03-29T10:10:18.637462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:11:01.442096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.649479+0000","last_change":"2024-03-29T10:10:17.647936+0000","last_active":"2024-03-29T10:10:17.649479+0000","last_peered":"2024-03-29T10:10:17.649479+0000","last_clean":"2024-03-29T10:10:17.649479+0000","last_became_active":"2024-03-29T10:10:17.646965+0000","last_became_peered":"2024-03-29T10:10:17.646965+0000","last_unstale":"2024-03-29T10:10:17.649479+0000","last_undegraded":"2024-03-29T10:10:17.649479+0000","last_fullsized":"2024-03-29T10:10:17.649479+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:42.303434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.655702+0000","last_change":"2024-03-29T10:10:17.648416+0000","last_active":"2024-03-29T10:10:17.655702+0000","last_peered":"2024-03-29T10:10:17.655702+0000","last_clean":"2024-03-29T10:10:17.655702+0000","last_became_active":"2024-03-29T10:10:17.647554+0000","last_became_peered":"2024-03-29T10:10:17.647554+0000","last_unstale":"2024-03-29T10:10:17.655702+0000","last_undegraded":"2024-03-29T10:10:17.655702+0000","last_fullsized":"2024-03-29T10:10:17.655702+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:57.257695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.665035+0000","last_change":"2024-03-29T10:10:17.663232+0000","last_active":"2024-03-29T10:10:17.665035+0000","last_peered":"2024-03-29T10:10:17.665035+0000","last_clean":"2024-03-29T10:10:17.665035+0000","last_became_active":"2024-03-29T10:10:17.662119+0000","last_became_peered":"2024-03-29T10:10:17.662119+0000","last_unstale":"2024-03-29T10:10:17.665035+0000","last_undegraded":"2024-03-29T10:10:17.665035+0000","last_fullsized":"2024-03-29T10:10:17.665035+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:41:43.134146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.658980+0000","last_change":"2024-03-29T10:10:17.653979+0000","last_active":"2024-03-29T10:10:17.658980+0000","last_peered":"2024-03-29T10:10:17.658980+0000","last_clean":"2024-03-29T10:10:17.658980+0000","last_became_active":"2024-03-29T10:10:17.653716+0000","last_became_peered":"2024-03-29T10:10:17.653716+0000","last_unstale":"2024-03-29T10:10:17.658980+0000","last_undegraded":"2024-03-29T10:10:17.658980+0000","last_fullsized":"2024-03-29T10:10:17.658980+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:21:56.054067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663470+0000","last_change":"2024-03-29T10:10:17.652657+0000","last_active":"2024-03-29T10:10:17.663470+0000","last_peered":"2024-03-29T10:10:17.663470+0000","last_clean":"2024-03-29T10:10:17.663470+0000","last_became_active":"2024-03-29T10:10:17.652395+0000","last_became_peered":"2024-03-29T10:10:17.652395+0000","last_unstale":"2024-03-29T10:10:17.663470+0000","last_undegraded":"2024-03-29T10:10:17.663470+0000","last_fullsized":"2024-03-29T10:10:17.663470+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:21:28.986763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.658629+0000","last_change":"2024-03-29T10:10:17.655786+0000","last_active":"2024-03-29T10:10:17.658629+0000","last_peered":"2024-03-29T10:10:17.658629+0000","last_clean":"2024-03-29T10:10:17.658629+0000","last_became_active":"2024-03-29T10:10:17.654162+0000","last_became_peered":"2024-03-29T10:10:17.654162+0000","last_unstale":"2024-03-29T10:10:17.658629+0000","last_undegraded":"2024-03-29T10:10:17.658629+0000","last_fullsized":"2024-03-29T10:10:17.658629+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:04.588448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637349+0000","last_change":"2024-03-29T10:10:17.641516+0000","last_active":"2024-03-29T10:10:18.637349+0000","last_peered":"2024-03-29T10:10:18.637349+0000","last_clean":"2024-03-29T10:10:18.637349+0000","last_became_active":"2024-03-29T10:10:17.641173+0000","last_became_peered":"2024-03-29T10:10:17.641173+0000","last_unstale":"2024-03-29T10:10:18.637349+0000","last_undegraded":"2024-03-29T10:10:18.637349+0000","last_fullsized":"2024-03-29T10:10:18.637349+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:04.588448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664790+0000","last_change":"2024-03-29T10:10:17.655838+0000","last_active":"2024-03-29T10:10:17.664790+0000","last_peered":"2024-03-29T10:10:17.664790+0000","last_clean":"2024-03-29T10:10:17.664790+0000","last_became_active":"2024-03-29T10:10:17.655584+0000","last_became_peered":"2024-03-29T10:10:17.655584+0000","last_unstale":"2024-03-29T10:10:17.664790+0000","last_undegraded":"2024-03-29T10:10:17.664790+0000","last_fullsized":"2024-03-29T10:10:17.664790+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:59:32.098044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.662876+0000","last_change":"2024-03-29T10:10:17.659801+0000","last_active":"2024-03-29T10:10:17.662876+0000","last_peered":"2024-03-29T10:10:17.662876+0000","last_clean":"2024-03-29T10:10:17.662876+0000","last_became_active":"2024-03-29T10:10:17.659451+0000","last_became_peered":"2024-03-29T10:10:17.659451+0000","last_unstale":"2024-03-29T10:10:17.662876+0000","last_undegraded":"2024-03-29T10:10:17.662876+0000","last_fullsized":"2024-03-29T10:10:17.662876+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:11:01.442096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.646602+0000","last_change":"2024-03-29T10:10:17.637556+0000","last_active":"2024-03-29T10:10:17.646602+0000","last_peered":"2024-03-29T10:10:17.646602+0000","last_clean":"2024-03-29T10:10:17.646602+0000","last_became_active":"2024-03-29T10:10:17.637299+0000","last_became_peered":"2024-03-29T10:10:17.637299+0000","last_unstale":"2024-03-29T10:10:17.646602+0000","last_undegraded":"2024-03-29T10:10:17.646602+0000","last_fullsized":"2024-03-29T10:10:17.646602+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:38.480817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636640+0000","last_change":"2024-03-29T10:10:17.637608+0000","last_active":"2024-03-29T10:10:18.636640+0000","last_peered":"2024-03-29T10:10:18.636640+0000","last_clean":"2024-03-29T10:10:18.636640+0000","last_became_active":"2024-03-29T10:10:17.637278+0000","last_became_peered":"2024-03-29T10:10:17.637278+0000","last_unstale":"2024-03-29T10:10:18.636640+0000","last_undegraded":"2024-03-29T10:10:18.636640+0000","last_fullsized":"2024-03-29T10:10:18.636640+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:47.601889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636499+0000","last_change":"2024-03-29T10:10:17.659753+0000","last_active":"2024-03-29T10:10:18.636499+0000","last_peered":"2024-03-29T10:10:18.636499+0000","last_clean":"2024-03-29T10:10:18.636499+0000","last_became_active":"2024-03-29T10:10:17.659482+0000","last_became_peered":"2024-03-29T10:10:17.659482+0000","last_unstale":"2024-03-29T10:10:18.636499+0000","last_undegraded":"2024-03-29T10:10:18.636499+0000","last_fullsized":"2024-03-29T10:10:18.636499+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:27.677477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637006+0000","last_change":"2024-03-29T10:10:17.646361+0000","last_active":"2024-03-29T10:10:18.637006+0000","last_peered":"2024-03-29T10:10:18.637006+0000","last_clean":"2024-03-29T10:10:18.637006+0000","last_became_active":"2024-03-29T10:10:17.646082+0000","last_became_peered":"2024-03-29T10:10:17.646082+0000","last_unstale":"2024-03-29T10:10:18.637006+0000","last_undegraded":"2024-03-29T10:10:18.637006+0000","last_fullsized":"2024-03-29T10:10:18.637006+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:27.168707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.669756+0000","last_change":"2024-03-29T10:10:17.664714+0000","last_active":"2024-03-29T10:10:17.669756+0000","last_peered":"2024-03-29T10:10:17.669756+0000","last_clean":"2024-03-29T10:10:17.669756+0000","last_became_active":"2024-03-29T10:10:17.664060+0000","last_became_peered":"2024-03-29T10:10:17.664060+0000","last_unstale":"2024-03-29T10:10:17.669756+0000","last_undegraded":"2024-03-29T10:10:17.669756+0000","last_fullsized":"2024-03-29T10:10:17.669756+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:36.566454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"93'4","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.662617+0000","last_change":"2024-03-29T10:10:17.126210+0000","last_active":"2024-03-29T10:10:17.662617+0000","last_peered":"2024-03-29T10:10:17.662617+0000","last_clean":"2024-03-29T10:10:17.662617+0000","last_became_active":"2024-03-29T10:10:17.125927+0000","last_became_peered":"2024-03-29T10:10:17.125927+0000","last_unstale":"2024-03-29T10:10:17.662617+0000","last_undegraded":"2024-03-29T10:10:17.662617+0000","last_fullsized":"2024-03-29T10:10:17.662617+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:11:00.439534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663693+0000","last_change":"2024-03-29T10:10:17.657004+0000","last_active":"2024-03-29T10:10:17.663693+0000","last_peered":"2024-03-29T10:10:17.663693+0000","last_clean":"2024-03-29T10:10:17.663693+0000","last_became_active":"2024-03-29T10:10:17.654677+0000","last_became_peered":"2024-03-29T10:10:17.654677+0000","last_unstale":"2024-03-29T10:10:17.663693+0000","last_undegraded":"2024-03-29T10:10:17.663693+0000","last_fullsized":"2024-03-29T10:10:17.663693+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:45.929036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"93'1","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.657252+0000","last_change":"2024-03-29T10:10:17.123538+0000","last_active":"2024-03-29T10:10:17.657252+0000","last_peered":"2024-03-29T10:10:17.657252+0000","last_clean":"2024-03-29T10:10:17.657252+0000","last_became_active":"2024-03-29T10:10:17.123274+0000","last_became_peered":"2024-03-29T10:10:17.123274+0000","last_unstale":"2024-03-29T10:10:17.657252+0000","last_undegraded":"2024-03-29T10:10:17.657252+0000","last_fullsized":"2024-03-29T10:10:17.657252+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:41:42.131584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637486+0000","last_change":"2024-03-29T10:10:17.653648+0000","last_active":"2024-03-29T10:10:18.637486+0000","last_peered":"2024-03-29T10:10:18.637486+0000","last_clean":"2024-03-29T10:10:18.637486+0000","last_became_active":"2024-03-29T10:10:17.653378+0000","last_became_peered":"2024-03-29T10:10:17.653378+0000","last_unstale":"2024-03-29T10:10:18.637486+0000","last_undegraded":"2024-03-29T10:10:18.637486+0000","last_fullsized":"2024-03-29T10:10:18.637486+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:57.257695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"93'2","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664182+0000","last_change":"2024-03-29T10:10:17.131303+0000","last_active":"2024-03-29T10:10:17.664182+0000","last_peered":"2024-03-29T10:10:17.664182+0000","last_clean":"2024-03-29T10:10:17.664182+0000","last_became_active":"2024-03-29T10:10:17.130984+0000","last_became_peered":"2024-03-29T10:10:17.130984+0000","last_unstale":"2024-03-29T10:10:17.664182+0000","last_undegraded":"2024-03-29T10:10:17.664182+0000","last_fullsized":"2024-03-29T10:10:17.664182+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.082904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.665033+0000","last_change":"2024-03-29T10:10:17.662407+0000","last_active":"2024-03-29T10:10:17.665033+0000","last_peered":"2024-03-29T10:10:17.665033+0000","last_clean":"2024-03-29T10:10:17.665033+0000","last_became_active":"2024-03-29T10:10:17.660494+0000","last_became_peered":"2024-03-29T10:10:17.660494+0000","last_unstale":"2024-03-29T10:10:17.665033+0000","last_undegraded":"2024-03-29T10:10:17.665033+0000","last_fullsized":"2024-03-29T10:10:17.665033+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:26:39.092036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"93'2","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637693+0000","last_change":"2024-03-29T10:10:17.123003+0000","last_active":"2024-03-29T10:10:18.637693+0000","last_peered":"2024-03-29T10:10:18.637693+0000","last_clean":"2024-03-29T10:10:18.637693+0000","last_became_active":"2024-03-29T10:10:17.120876+0000","last_became_peered":"2024-03-29T10:10:17.120876+0000","last_unstale":"2024-03-29T10:10:18.637693+0000","last_undegraded":"2024-03-29T10:10:18.637693+0000","last_fullsized":"2024-03-29T10:10:18.637693+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:34.579766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636904+0000","last_change":"2024-03-29T10:10:17.647351+0000","last_active":"2024-03-29T10:10:18.636904+0000","last_peered":"2024-03-29T10:10:18.636904+0000","last_clean":"2024-03-29T10:10:18.636904+0000","last_became_active":"2024-03-29T10:10:17.646462+0000","last_became_peered":"2024-03-29T10:10:17.646462+0000","last_unstale":"2024-03-29T10:10:18.636904+0000","last_undegraded":"2024-03-29T10:10:18.636904+0000","last_fullsized":"2024-03-29T10:10:18.636904+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:11:01.442096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"93'1","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.656001+0000","last_change":"2024-03-29T10:10:17.126464+0000","last_active":"2024-03-29T10:10:17.656001+0000","last_peered":"2024-03-29T10:10:17.656001+0000","last_clean":"2024-03-29T10:10:17.656001+0000","last_became_active":"2024-03-29T10:10:17.126214+0000","last_became_peered":"2024-03-29T10:10:17.126214+0000","last_unstale":"2024-03-29T10:10:17.656001+0000","last_undegraded":"2024-03-29T10:10:17.656001+0000","last_fullsized":"2024-03-29T10:10:17.656001+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:02.154374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.655749+0000","last_change":"2024-03-29T10:10:17.648596+0000","last_active":"2024-03-29T10:10:17.655749+0000","last_peered":"2024-03-29T10:10:17.655749+0000","last_clean":"2024-03-29T10:10:17.655749+0000","last_became_active":"2024-03-29T10:10:17.648126+0000","last_became_peered":"2024-03-29T10:10:17.648126+0000","last_unstale":"2024-03-29T10:10:17.655749+0000","last_undegraded":"2024-03-29T10:10:17.655749+0000","last_fullsized":"2024-03-29T10:10:17.655749+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:47.601889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"93'1","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636820+0000","last_change":"2024-03-29T10:10:17.120935+0000","last_active":"2024-03-29T10:10:18.636820+0000","last_peered":"2024-03-29T10:10:18.636820+0000","last_clean":"2024-03-29T10:10:18.636820+0000","last_became_active":"2024-03-29T10:10:17.120686+0000","last_became_peered":"2024-03-29T10:10:17.120686+0000","last_unstale":"2024-03-29T10:10:18.636820+0000","last_undegraded":"2024-03-29T10:10:18.636820+0000","last_fullsized":"2024-03-29T10:10:18.636820+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:12:56.523092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663770+0000","last_change":"2024-03-29T10:10:17.656894+0000","last_active":"2024-03-29T10:10:17.663770+0000","last_peered":"2024-03-29T10:10:17.663770+0000","last_clean":"2024-03-29T10:10:17.663770+0000","last_became_active":"2024-03-29T10:10:17.656577+0000","last_became_peered":"2024-03-29T10:10:17.656577+0000","last_unstale":"2024-03-29T10:10:17.663770+0000","last_undegraded":"2024-03-29T10:10:17.663770+0000","last_fullsized":"2024-03-29T10:10:17.663770+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:35:47.779265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631447+0000","last_change":"2024-03-29T10:10:17.125986+0000","last_active":"2024-03-29T10:10:18.631447+0000","last_peered":"2024-03-29T10:10:18.631447+0000","last_clean":"2024-03-29T10:10:18.631447+0000","last_became_active":"2024-03-29T10:10:17.125718+0000","last_became_peered":"2024-03-29T10:10:17.125718+0000","last_unstale":"2024-03-29T10:10:18.631447+0000","last_undegraded":"2024-03-29T10:10:18.631447+0000","last_fullsized":"2024-03-29T10:10:18.631447+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:42:19.065517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663168+0000","last_change":"2024-03-29T10:10:17.652224+0000","last_active":"2024-03-29T10:10:17.663168+0000","last_peered":"2024-03-29T10:10:17.663168+0000","last_clean":"2024-03-29T10:10:17.663168+0000","last_became_active":"2024-03-29T10:10:17.648943+0000","last_became_peered":"2024-03-29T10:10:17.648943+0000","last_unstale":"2024-03-29T10:10:17.663168+0000","last_undegraded":"2024-03-29T10:10:17.663168+0000","last_fullsized":"2024-03-29T10:10:17.663168+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:42:28.134507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.654598+0000","last_change":"2024-03-29T10:10:17.117316+0000","last_active":"2024-03-29T10:10:17.654598+0000","last_peered":"2024-03-29T10:10:17.654598+0000","last_clean":"2024-03-29T10:10:17.654598+0000","last_became_active":"2024-03-29T10:10:17.117051+0000","last_became_peered":"2024-03-29T10:10:17.117051+0000","last_unstale":"2024-03-29T10:10:17.654598+0000","last_undegraded":"2024-03-29T10:10:17.654598+0000","last_fullsized":"2024-03-29T10:10:17.654598+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:11.581029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637454+0000","last_change":"2024-03-29T10:10:17.654321+0000","last_active":"2024-03-29T10:10:18.637454+0000","last_peered":"2024-03-29T10:10:18.637454+0000","last_clean":"2024-03-29T10:10:18.637454+0000","last_became_active":"2024-03-29T10:10:17.654030+0000","last_became_peered":"2024-03-29T10:10:17.654030+0000","last_unstale":"2024-03-29T10:10:18.637454+0000","last_undegraded":"2024-03-29T10:10:18.637454+0000","last_fullsized":"2024-03-29T10:10:18.637454+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:20.641015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.643289+0000","last_change":"2024-03-29T10:10:17.124975+0000","last_active":"2024-03-29T10:10:17.643289+0000","last_peered":"2024-03-29T10:10:17.643289+0000","last_clean":"2024-03-29T10:10:17.643289+0000","last_became_active":"2024-03-29T10:10:17.124599+0000","last_became_peered":"2024-03-29T10:10:17.124599+0000","last_unstale":"2024-03-29T10:10:17.643289+0000","last_undegraded":"2024-03-29T10:10:17.643289+0000","last_fullsized":"2024-03-29T10:10:17.643289+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:13:13.568372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.667029+0000","last_change":"2024-03-29T10:10:17.664343+0000","last_active":"2024-03-29T10:10:17.667029+0000","last_peered":"2024-03-29T10:10:17.667029+0000","last_clean":"2024-03-29T10:10:17.667029+0000","last_became_active":"2024-03-29T10:10:17.663984+0000","last_became_peered":"2024-03-29T10:10:17.663984+0000","last_unstale":"2024-03-29T10:10:17.667029+0000","last_undegraded":"2024-03-29T10:10:17.667029+0000","last_fullsized":"2024-03-29T10:10:17.667029+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:07.055847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637805+0000","last_change":"2024-03-29T10:10:17.118934+0000","last_active":"2024-03-29T10:10:18.637805+0000","last_peered":"2024-03-29T10:10:18.637805+0000","last_clean":"2024-03-29T10:10:18.637805+0000","last_became_active":"2024-03-29T10:10:17.118294+0000","last_became_peered":"2024-03-29T10:10:17.118294+0000","last_unstale":"2024-03-29T10:10:18.637805+0000","last_undegraded":"2024-03-29T10:10:18.637805+0000","last_fullsized":"2024-03-29T10:10:18.637805+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:49.244529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637617+0000","last_change":"2024-03-29T10:10:17.653094+0000","last_active":"2024-03-29T10:10:18.637617+0000","last_peered":"2024-03-29T10:10:18.637617+0000","last_clean":"2024-03-29T10:10:18.637617+0000","last_became_active":"2024-03-29T10:10:17.652194+0000","last_became_peered":"2024-03-29T10:10:17.652194+0000","last_unstale":"2024-03-29T10:10:18.637617+0000","last_undegraded":"2024-03-29T10:10:18.637617+0000","last_fullsized":"2024-03-29T10:10:18.637617+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:27.677477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"93'3","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.651819+0000","last_change":"2024-03-29T10:10:17.119279+0000","last_active":"2024-03-29T10:10:17.651819+0000","last_peered":"2024-03-29T10:10:17.651819+0000","last_clean":"2024-03-29T10:10:17.651819+0000","last_became_active":"2024-03-29T10:10:17.119006+0000","last_became_peered":"2024-03-29T10:10:17.119006+0000","last_unstale":"2024-03-29T10:10:17.651819+0000","last_undegraded":"2024-03-29T10:10:17.651819+0000","last_fullsized":"2024-03-29T10:10:17.651819+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:33:06.030419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.660297+0000","last_change":"2024-03-29T10:10:17.655328+0000","last_active":"2024-03-29T10:10:17.660297+0000","last_peered":"2024-03-29T10:10:17.660297+0000","last_clean":"2024-03-29T10:10:17.660297+0000","last_became_active":"2024-03-29T10:10:17.655085+0000","last_became_peered":"2024-03-29T10:10:17.655085+0000","last_unstale":"2024-03-29T10:10:17.660297+0000","last_undegraded":"2024-03-29T10:10:17.660297+0000","last_fullsized":"2024-03-29T10:10:17.660297+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:30.413167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"93'6","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631952+0000","last_change":"2024-03-29T10:10:17.118649+0000","last_active":"2024-03-29T10:10:18.631952+0000","last_peered":"2024-03-29T10:10:18.631952+0000","last_clean":"2024-03-29T10:10:18.631952+0000","last_became_active":"2024-03-29T10:10:17.118214+0000","last_became_peered":"2024-03-29T10:10:17.118214+0000","last_unstale":"2024-03-29T10:10:18.631952+0000","last_undegraded":"2024-03-29T10:10:18.631952+0000","last_fullsized":"2024-03-29T10:10:18.631952+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:47:04.088591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.665022+0000","last_change":"2024-03-29T10:10:17.662283+0000","last_active":"2024-03-29T10:10:17.665022+0000","last_peered":"2024-03-29T10:10:17.665022+0000","last_clean":"2024-03-29T10:10:17.665022+0000","last_became_active":"2024-03-29T10:10:17.661322+0000","last_became_peered":"2024-03-29T10:10:17.661322+0000","last_unstale":"2024-03-29T10:10:17.665022+0000","last_undegraded":"2024-03-29T10:10:17.665022+0000","last_fullsized":"2024-03-29T10:10:17.665022+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:03.156935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"93'2","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631628+0000","last_change":"2024-03-29T10:10:17.119319+0000","last_active":"2024-03-29T10:10:18.631628+0000","last_peered":"2024-03-29T10:10:18.631628+0000","last_clean":"2024-03-29T10:10:18.631628+0000","last_became_active":"2024-03-29T10:10:17.119069+0000","last_became_peered":"2024-03-29T10:10:17.119069+0000","last_unstale":"2024-03-29T10:10:18.631628+0000","last_undegraded":"2024-03-29T10:10:18.631628+0000","last_fullsized":"2024-03-29T10:10:18.631628+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:16:52.646444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637131+0000","last_change":"2024-03-29T10:10:17.648356+0000","last_active":"2024-03-29T10:10:18.637131+0000","last_peered":"2024-03-29T10:10:18.637131+0000","last_clean":"2024-03-29T10:10:18.637131+0000","last_became_active":"2024-03-29T10:10:17.648024+0000","last_became_peered":"2024-03-29T10:10:17.648024+0000","last_unstale":"2024-03-29T10:10:18.637131+0000","last_undegraded":"2024-03-29T10:10:18.637131+0000","last_fullsized":"2024-03-29T10:10:18.637131+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:41:43.134146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636919+0000","last_change":"2024-03-29T10:10:17.119202+0000","last_active":"2024-03-29T10:10:18.636919+0000","last_peered":"2024-03-29T10:10:18.636919+0000","last_clean":"2024-03-29T10:10:18.636919+0000","last_became_active":"2024-03-29T10:10:17.118930+0000","last_became_peered":"2024-03-29T10:10:17.118930+0000","last_unstale":"2024-03-29T10:10:18.636919+0000","last_undegraded":"2024-03-29T10:10:18.636919+0000","last_fullsized":"2024-03-29T10:10:18.636919+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:56:41.191132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.651415+0000","last_change":"2024-03-29T10:10:17.645985+0000","last_active":"2024-03-29T10:10:17.651415+0000","last_peered":"2024-03-29T10:10:17.651415+0000","last_clean":"2024-03-29T10:10:17.651415+0000","last_became_active":"2024-03-29T10:10:17.645684+0000","last_became_peered":"2024-03-29T10:10:17.645684+0000","last_unstale":"2024-03-29T10:10:17.651415+0000","last_undegraded":"2024-03-29T10:10:17.651415+0000","last_fullsized":"2024-03-29T10:10:17.651415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:56:25.371298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636655+0000","last_change":"2024-03-29T10:10:17.117927+0000","last_active":"2024-03-29T10:10:18.636655+0000","last_peered":"2024-03-29T10:10:18.636655+0000","last_clean":"2024-03-29T10:10:18.636655+0000","last_became_active":"2024-03-29T10:10:17.117598+0000","last_became_peered":"2024-03-29T10:10:17.117598+0000","last_unstale":"2024-03-29T10:10:18.636655+0000","last_undegraded":"2024-03-29T10:10:18.636655+0000","last_fullsized":"2024-03-29T10:10:18.636655+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:13:13.568372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663919+0000","last_change":"2024-03-29T10:10:17.656554+0000","last_active":"2024-03-29T10:10:17.663919+0000","last_peered":"2024-03-29T10:10:17.663919+0000","last_clean":"2024-03-29T10:10:17.663919+0000","last_became_active":"2024-03-29T10:10:17.650748+0000","last_became_peered":"2024-03-29T10:10:17.650748+0000","last_unstale":"2024-03-29T10:10:17.663919+0000","last_undegraded":"2024-03-29T10:10:17.663919+0000","last_fullsized":"2024-03-29T10:10:17.663919+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:25.955220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"93'8","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663418+0000","last_change":"2024-03-29T10:10:17.129015+0000","last_active":"2024-03-29T10:10:17.663418+0000","last_peered":"2024-03-29T10:10:17.663418+0000","last_clean":"2024-03-29T10:10:17.663418+0000","last_became_active":"2024-03-29T10:10:17.128081+0000","last_became_peered":"2024-03-29T10:10:17.128081+0000","last_unstale":"2024-03-29T10:10:17.663418+0000","last_undegraded":"2024-03-29T10:10:17.663418+0000","last_fullsized":"2024-03-29T10:10:17.663418+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:49.244529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.655893+0000","last_change":"2024-03-29T10:10:17.645764+0000","last_active":"2024-03-29T10:10:17.655893+0000","last_peered":"2024-03-29T10:10:17.655893+0000","last_clean":"2024-03-29T10:10:17.655893+0000","last_became_active":"2024-03-29T10:10:17.645449+0000","last_became_peered":"2024-03-29T10:10:17.645449+0000","last_unstale":"2024-03-29T10:10:17.655893+0000","last_undegraded":"2024-03-29T10:10:17.655893+0000","last_fullsized":"2024-03-29T10:10:17.655893+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:47:42.484636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"93'1","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637919+0000","last_change":"2024-03-29T10:10:17.124268+0000","last_active":"2024-03-29T10:10:18.637919+0000","last_peered":"2024-03-29T10:10:18.637919+0000","last_clean":"2024-03-29T10:10:18.637919+0000","last_became_active":"2024-03-29T10:10:17.123698+0000","last_became_peered":"2024-03-29T10:10:17.123698+0000","last_unstale":"2024-03-29T10:10:18.637919+0000","last_undegraded":"2024-03-29T10:10:18.637919+0000","last_fullsized":"2024-03-29T10:10:18.637919+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.082904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663744+0000","last_change":"2024-03-29T10:10:17.659876+0000","last_active":"2024-03-29T10:10:17.663744+0000","last_peered":"2024-03-29T10:10:17.663744+0000","last_clean":"2024-03-29T10:10:17.663744+0000","last_became_active":"2024-03-29T10:10:17.658841+0000","last_became_peered":"2024-03-29T10:10:17.658841+0000","last_unstale":"2024-03-29T10:10:17.663744+0000","last_undegraded":"2024-03-29T10:10:17.663744+0000","last_fullsized":"2024-03-29T10:10:17.663744+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:56:25.371298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"93'2","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664138+0000","last_change":"2024-03-29T10:10:17.126435+0000","last_active":"2024-03-29T10:10:17.664138+0000","last_peered":"2024-03-29T10:10:17.664138+0000","last_clean":"2024-03-29T10:10:17.664138+0000","last_became_active":"2024-03-29T10:10:17.126123+0000","last_became_peered":"2024-03-29T10:10:17.126123+0000","last_unstale":"2024-03-29T10:10:17.664138+0000","last_undegraded":"2024-03-29T10:10:17.664138+0000","last_fullsized":"2024-03-29T10:10:17.664138+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:44.926474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631925+0000","last_change":"2024-03-29T10:10:17.661191+0000","last_active":"2024-03-29T10:10:18.631925+0000","last_peered":"2024-03-29T10:10:18.631925+0000","last_clean":"2024-03-29T10:10:18.631925+0000","last_became_active":"2024-03-29T10:10:17.660557+0000","last_became_peered":"2024-03-29T10:10:17.660557+0000","last_unstale":"2024-03-29T10:10:18.631925+0000","last_undegraded":"2024-03-29T10:10:18.631925+0000","last_fullsized":"2024-03-29T10:10:18.631925+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:34:20.242868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"93'1","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637484+0000","last_change":"2024-03-29T10:10:17.126064+0000","last_active":"2024-03-29T10:10:18.637484+0000","last_peered":"2024-03-29T10:10:18.637484+0000","last_clean":"2024-03-29T10:10:18.637484+0000","last_became_active":"2024-03-29T10:10:17.125734+0000","last_became_peered":"2024-03-29T10:10:17.125734+0000","last_unstale":"2024-03-29T10:10:18.637484+0000","last_undegraded":"2024-03-29T10:10:18.637484+0000","last_fullsized":"2024-03-29T10:10:18.637484+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:01.925007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.660005+0000","last_change":"2024-03-29T10:10:17.655303+0000","last_active":"2024-03-29T10:10:17.660005+0000","last_peered":"2024-03-29T10:10:17.660005+0000","last_clean":"2024-03-29T10:10:17.660005+0000","last_became_active":"2024-03-29T10:10:17.654222+0000","last_became_peered":"2024-03-29T10:10:17.654222+0000","last_unstale":"2024-03-29T10:10:17.660005+0000","last_undegraded":"2024-03-29T10:10:17.660005+0000","last_fullsized":"2024-03-29T10:10:17.660005+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:56:42.193693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.644585+0000","last_change":"2024-03-29T10:10:17.126454+0000","last_active":"2024-03-29T10:10:17.644585+0000","last_peered":"2024-03-29T10:10:17.644585+0000","last_clean":"2024-03-29T10:10:17.644585+0000","last_became_active":"2024-03-29T10:10:17.126095+0000","last_became_peered":"2024-03-29T10:10:17.126095+0000","last_unstale":"2024-03-29T10:10:17.644585+0000","last_undegraded":"2024-03-29T10:10:17.644585+0000","last_fullsized":"2024-03-29T10:10:17.644585+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:56.255133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.646626+0000","last_change":"2024-03-29T10:10:17.640780+0000","last_active":"2024-03-29T10:10:17.646626+0000","last_peered":"2024-03-29T10:10:17.646626+0000","last_clean":"2024-03-29T10:10:17.646626+0000","last_became_active":"2024-03-29T10:10:17.640514+0000","last_became_peered":"2024-03-29T10:10:17.640514+0000","last_unstale":"2024-03-29T10:10:17.646626+0000","last_undegraded":"2024-03-29T10:10:17.646626+0000","last_fullsized":"2024-03-29T10:10:17.646626+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:08.919330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637095+0000","last_change":"2024-03-29T10:10:17.120504+0000","last_active":"2024-03-29T10:10:18.637095+0000","last_peered":"2024-03-29T10:10:18.637095+0000","last_clean":"2024-03-29T10:10:18.637095+0000","last_became_active":"2024-03-29T10:10:17.120202+0000","last_became_peered":"2024-03-29T10:10:17.120202+0000","last_unstale":"2024-03-29T10:10:18.637095+0000","last_undegraded":"2024-03-29T10:10:18.637095+0000","last_fullsized":"2024-03-29T10:10:18.637095+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:26:31.933626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.646903+0000","last_change":"2024-03-29T10:10:17.637795+0000","last_active":"2024-03-29T10:10:17.646903+0000","last_peered":"2024-03-29T10:10:17.646903+0000","last_clean":"2024-03-29T10:10:17.646903+0000","last_became_active":"2024-03-29T10:10:17.637538+0000","last_became_peered":"2024-03-29T10:10:17.637538+0000","last_unstale":"2024-03-29T10:10:17.646903+0000","last_undegraded":"2024-03-29T10:10:17.646903+0000","last_fullsized":"2024-03-29T10:10:17.646903+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:42.679046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637027+0000","last_change":"2024-03-29T10:10:17.125657+0000","last_active":"2024-03-29T10:10:18.637027+0000","last_peered":"2024-03-29T10:10:18.637027+0000","last_clean":"2024-03-29T10:10:18.637027+0000","last_became_active":"2024-03-29T10:10:17.125358+0000","last_became_peered":"2024-03-29T10:10:17.125358+0000","last_unstale":"2024-03-29T10:10:18.637027+0000","last_undegraded":"2024-03-29T10:10:18.637027+0000","last_fullsized":"2024-03-29T10:10:18.637027+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.082904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.646669+0000","last_change":"2024-03-29T10:10:17.637766+0000","last_active":"2024-03-29T10:10:17.646669+0000","last_peered":"2024-03-29T10:10:17.646669+0000","last_clean":"2024-03-29T10:10:17.646669+0000","last_became_active":"2024-03-29T10:10:17.637469+0000","last_became_peered":"2024-03-29T10:10:17.637469+0000","last_unstale":"2024-03-29T10:10:17.646669+0000","last_undegraded":"2024-03-29T10:10:17.646669+0000","last_fullsized":"2024-03-29T10:10:17.646669+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:56.411283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.637172+0000","last_change":"2024-03-29T10:10:17.119727+0000","last_active":"2024-03-29T10:10:17.637172+0000","last_peered":"2024-03-29T10:10:17.637172+0000","last_clean":"2024-03-29T10:10:17.637172+0000","last_became_active":"2024-03-29T10:10:17.119480+0000","last_became_peered":"2024-03-29T10:10:17.119480+0000","last_unstale":"2024-03-29T10:10:17.637172+0000","last_undegraded":"2024-03-29T10:10:17.637172+0000","last_fullsized":"2024-03-29T10:10:17.637172+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:46.599328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664598+0000","last_change":"2024-03-29T10:10:17.658110+0000","last_active":"2024-03-29T10:10:17.664598+0000","last_peered":"2024-03-29T10:10:17.664598+0000","last_clean":"2024-03-29T10:10:17.664598+0000","last_became_active":"2024-03-29T10:10:17.657445+0000","last_became_peered":"2024-03-29T10:10:17.657445+0000","last_unstale":"2024-03-29T10:10:17.664598+0000","last_undegraded":"2024-03-29T10:10:17.664598+0000","last_fullsized":"2024-03-29T10:10:17.664598+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:01:10.259496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"93'2","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637918+0000","last_change":"2024-03-29T10:10:17.124032+0000","last_active":"2024-03-29T10:10:18.637918+0000","last_peered":"2024-03-29T10:10:18.637918+0000","last_clean":"2024-03-29T10:10:18.637918+0000","last_became_active":"2024-03-29T10:10:17.123333+0000","last_became_peered":"2024-03-29T10:10:17.123333+0000","last_unstale":"2024-03-29T10:10:18.637918+0000","last_undegraded":"2024-03-29T10:10:18.637918+0000","last_fullsized":"2024-03-29T10:10:18.637918+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:44.926474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636912+0000","last_change":"2024-03-29T10:10:17.644480+0000","last_active":"2024-03-29T10:10:18.636912+0000","last_peered":"2024-03-29T10:10:18.636912+0000","last_clean":"2024-03-29T10:10:18.636912+0000","last_became_active":"2024-03-29T10:10:17.644181+0000","last_became_peered":"2024-03-29T10:10:17.644181+0000","last_unstale":"2024-03-29T10:10:18.636912+0000","last_undegraded":"2024-03-29T10:10:18.636912+0000","last_fullsized":"2024-03-29T10:10:18.636912+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:57.257695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637545+0000","last_change":"2024-03-29T10:10:17.116018+0000","last_active":"2024-03-29T10:10:18.637545+0000","last_peered":"2024-03-29T10:10:18.637545+0000","last_clean":"2024-03-29T10:10:18.637545+0000","last_became_active":"2024-03-29T10:10:17.115598+0000","last_became_peered":"2024-03-29T10:10:17.115598+0000","last_unstale":"2024-03-29T10:10:18.637545+0000","last_undegraded":"2024-03-29T10:10:18.637545+0000","last_fullsized":"2024-03-29T10:10:18.637545+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:49.244529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.667077+0000","last_change":"2024-03-29T10:10:17.664430+0000","last_active":"2024-03-29T10:10:17.667077+0000","last_peered":"2024-03-29T10:10:17.667077+0000","last_clean":"2024-03-29T10:10:17.667077+0000","last_became_active":"2024-03-29T10:10:17.663576+0000","last_became_peered":"2024-03-29T10:10:17.663576+0000","last_unstale":"2024-03-29T10:10:17.667077+0000","last_undegraded":"2024-03-29T10:10:17.667077+0000","last_fullsized":"2024-03-29T10:10:17.667077+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:40:01.285102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.649991+0000","last_change":"2024-03-29T10:10:17.116045+0000","last_active":"2024-03-29T10:10:17.649991+0000","last_peered":"2024-03-29T10:10:17.649991+0000","last_clean":"2024-03-29T10:10:17.649991+0000","last_became_active":"2024-03-29T10:10:17.115770+0000","last_became_peered":"2024-03-29T10:10:17.115770+0000","last_unstale":"2024-03-29T10:10:17.649991+0000","last_undegraded":"2024-03-29T10:10:17.649991+0000","last_fullsized":"2024-03-29T10:10:17.649991+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:05.665071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663899+0000","last_change":"2024-03-29T10:10:17.658780+0000","last_active":"2024-03-29T10:10:17.663899+0000","last_peered":"2024-03-29T10:10:17.663899+0000","last_clean":"2024-03-29T10:10:17.663899+0000","last_became_active":"2024-03-29T10:10:17.657250+0000","last_became_peered":"2024-03-29T10:10:17.657250+0000","last_unstale":"2024-03-29T10:10:17.663899+0000","last_undegraded":"2024-03-29T10:10:17.663899+0000","last_fullsized":"2024-03-29T10:10:17.663899+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:42:28.134507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.640735+0000","last_change":"2024-03-29T10:10:17.117782+0000","last_active":"2024-03-29T10:10:17.640735+0000","last_peered":"2024-03-29T10:10:17.640735+0000","last_clean":"2024-03-29T10:10:17.640735+0000","last_became_active":"2024-03-29T10:10:17.117506+0000","last_became_peered":"2024-03-29T10:10:17.117506+0000","last_unstale":"2024-03-29T10:10:17.640735+0000","last_undegraded":"2024-03-29T10:10:17.640735+0000","last_fullsized":"2024-03-29T10:10:17.640735+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:03.585887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664460+0000","last_change":"2024-03-29T10:10:17.659435+0000","last_active":"2024-03-29T10:10:17.664460+0000","last_peered":"2024-03-29T10:10:17.664460+0000","last_clean":"2024-03-29T10:10:17.664460+0000","last_became_active":"2024-03-29T10:10:17.650714+0000","last_became_peered":"2024-03-29T10:10:17.650714+0000","last_unstale":"2024-03-29T10:10:17.664460+0000","last_undegraded":"2024-03-29T10:10:17.664460+0000","last_fullsized":"2024-03-29T10:10:17.664460+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:02.927569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.649946+0000","last_change":"2024-03-29T10:10:17.122709+0000","last_active":"2024-03-29T10:10:17.649946+0000","last_peered":"2024-03-29T10:10:17.649946+0000","last_clean":"2024-03-29T10:10:17.649946+0000","last_became_active":"2024-03-29T10:10:17.122289+0000","last_became_peered":"2024-03-29T10:10:17.122289+0000","last_unstale":"2024-03-29T10:10:17.649946+0000","last_undegraded":"2024-03-29T10:10:17.649946+0000","last_fullsized":"2024-03-29T10:10:17.649946+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:50:42.044699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636845+0000","last_change":"2024-03-29T10:10:17.646330+0000","last_active":"2024-03-29T10:10:18.636845+0000","last_peered":"2024-03-29T10:10:18.636845+0000","last_clean":"2024-03-29T10:10:18.636845+0000","last_became_active":"2024-03-29T10:10:17.645988+0000","last_became_peered":"2024-03-29T10:10:17.645988+0000","last_unstale":"2024-03-29T10:10:18.636845+0000","last_undegraded":"2024-03-29T10:10:18.636845+0000","last_fullsized":"2024-03-29T10:10:18.636845+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:08.435381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631552+0000","last_change":"2024-03-29T10:10:17.117333+0000","last_active":"2024-03-29T10:10:18.631552+0000","last_peered":"2024-03-29T10:10:18.631552+0000","last_clean":"2024-03-29T10:10:18.631552+0000","last_became_active":"2024-03-29T10:10:17.117063+0000","last_became_peered":"2024-03-29T10:10:17.117063+0000","last_unstale":"2024-03-29T10:10:18.631552+0000","last_undegraded":"2024-03-29T10:10:18.631552+0000","last_fullsized":"2024-03-29T10:10:18.631552+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:15:56.412964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631444+0000","last_change":"2024-03-29T10:10:17.648826+0000","last_active":"2024-03-29T10:10:18.631444+0000","last_peered":"2024-03-29T10:10:18.631444+0000","last_clean":"2024-03-29T10:10:18.631444+0000","last_became_active":"2024-03-29T10:10:17.648520+0000","last_became_peered":"2024-03-29T10:10:17.648520+0000","last_unstale":"2024-03-29T10:10:18.631444+0000","last_undegraded":"2024-03-29T10:10:18.631444+0000","last_fullsized":"2024-03-29T10:10:18.631444+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:07.440690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.637131+0000","last_change":"2024-03-29T10:10:17.128866+0000","last_active":"2024-03-29T10:10:17.637131+0000","last_peered":"2024-03-29T10:10:17.637131+0000","last_clean":"2024-03-29T10:10:17.637131+0000","last_became_active":"2024-03-29T10:10:17.128576+0000","last_became_peered":"2024-03-29T10:10:17.128576+0000","last_unstale":"2024-03-29T10:10:17.637131+0000","last_undegraded":"2024-03-29T10:10:17.637131+0000","last_fullsized":"2024-03-29T10:10:17.637131+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:01.925007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637048+0000","last_change":"2024-03-29T10:10:17.650715+0000","last_active":"2024-03-29T10:10:18.637048+0000","last_peered":"2024-03-29T10:10:18.637048+0000","last_clean":"2024-03-29T10:10:18.637048+0000","last_became_active":"2024-03-29T10:10:17.649935+0000","last_became_peered":"2024-03-29T10:10:17.649935+0000","last_unstale":"2024-03-29T10:10:18.637048+0000","last_undegraded":"2024-03-29T10:10:18.637048+0000","last_fullsized":"2024-03-29T10:10:18.637048+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:13.147091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637293+0000","last_change":"2024-03-29T10:10:17.122272+0000","last_active":"2024-03-29T10:10:18.637293+0000","last_peered":"2024-03-29T10:10:18.637293+0000","last_clean":"2024-03-29T10:10:18.637293+0000","last_became_active":"2024-03-29T10:10:17.121968+0000","last_became_peered":"2024-03-29T10:10:17.121968+0000","last_unstale":"2024-03-29T10:10:18.637293+0000","last_undegraded":"2024-03-29T10:10:18.637293+0000","last_fullsized":"2024-03-29T10:10:18.637293+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:29.410605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.644945+0000","last_change":"2024-03-29T10:10:17.637281+0000","last_active":"2024-03-29T10:10:17.644945+0000","last_peered":"2024-03-29T10:10:17.644945+0000","last_clean":"2024-03-29T10:10:17.644945+0000","last_became_active":"2024-03-29T10:10:17.637009+0000","last_became_peered":"2024-03-29T10:10:17.637009+0000","last_unstale":"2024-03-29T10:10:17.644945+0000","last_undegraded":"2024-03-29T10:10:17.644945+0000","last_fullsized":"2024-03-29T10:10:17.644945+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:34.531392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"93'2","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636958+0000","last_change":"2024-03-29T10:10:17.121650+0000","last_active":"2024-03-29T10:10:18.636958+0000","last_peered":"2024-03-29T10:10:18.636958+0000","last_clean":"2024-03-29T10:10:18.636958+0000","last_became_active":"2024-03-29T10:10:17.121308+0000","last_became_peered":"2024-03-29T10:10:17.121308+0000","last_unstale":"2024-03-29T10:10:18.636958+0000","last_undegraded":"2024-03-29T10:10:18.636958+0000","last_fullsized":"2024-03-29T10:10:18.636958+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:24.952658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'183","reported_seq":468,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.114449+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:10:17.114449+0000","last_peered":"2024-03-29T10:10:17.114449+0000","last_clean":"2024-03-29T10:10:17.114449+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:10:17.114449+0000","last_undegraded":"2024-03-29T10:10:17.114449+0000","last_fullsized":"2024-03-29T10:10:17.114449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575037,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32820,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339020,"statfs":{"total":96636764160,"available":96603156480,"internally_reserved":0,"allocated":1736704,"data_stored":1128337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.69799999999999995,"15min":0.71699999999999997},"min":{"1min":0.44600000000000001,"5min":0.44500000000000001,"15min":0.32500000000000001},"max":{"1min":0.91600000000000004,"5min":0.996,"15min":1.0109999999999999},"last":0.629},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68400000000000005,"15min":0.70999999999999996},"min":{"1min":0.50800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.81299999999999994,"5min":0.93400000000000005,"15min":1.046},"last":0.75600000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71099999999999997,"15min":0.72299999999999998},"min":{"1min":0.51800000000000002,"5min":0.47399999999999998,"15min":0.44500000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.1910000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71899999999999997,"15min":0.73599999999999999},"min":{"1min":0.52500000000000002,"5min":0.42199999999999999,"15min":0.372},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.57499999999999996}]},{"osd":2,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.72799999999999998,"15min":0.74299999999999999},"min":{"1min":0.47799999999999998,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.046},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.73799999999999999,"15min":0.752},"min":{"1min":0.54900000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.079},"last":0.70399999999999996}]},{"osd":3,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.70799999999999996,"15min":0.73599999999999999},"min":{"1min":0.51700000000000002,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.80700000000000005,"5min":0.97699999999999998,"15min":1.081},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.76400000000000001,"15min":0.78200000000000003},"min":{"1min":0.54500000000000004,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.71899999999999997}]},{"osd":4,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.29199999999999998},"max":{"1min":0.80100000000000005,"5min":0.86399999999999999,"15min":1.111},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63400000000000001,"15min":0.63700000000000001},"min":{"1min":0.39200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.85199999999999998,"5min":0.95199999999999996,"15min":0.95599999999999996},"last":0.84399999999999997}]},{"osd":5,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.46300000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95699999999999996,"5min":0.96099999999999997,"15min":1.0089999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.68500000000000005},"min":{"1min":0.49199999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82699999999999996,"5min":0.92800000000000005,"15min":1.141},"last":0.81499999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72499999999999998,"15min":0.72999999999999998},"min":{"1min":0.59599999999999997,"5min":0.50600000000000001,"15min":0.49199999999999999},"max":{"1min":0.92400000000000004,"5min":0.99099999999999999,"15min":1.1679999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70299999999999996,"15min":0.70099999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.435},"max":{"1min":0.94299999999999995,"5min":0.98399999999999999,"15min":1.121},"last":0.80600000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1142784,"data_stored":537445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.72199999999999998,"15min":0.73099999999999998},"min":{"1min":0.46200000000000002,"5min":0.42799999999999999,"15min":0.34699999999999998},"max":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":7.4080000000000004},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71399999999999997,"15min":0.75},"min":{"1min":0.47599999999999998,"5min":0.435,"15min":0.42499999999999999},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":6.7610000000000001},"last":0.70699999999999996}]},{"osd":1,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.46100000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.624},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73599999999999999,"15min":0.72699999999999998},"min":{"1min":0.45200000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.6709999999999998,"5min":2.6709999999999998,"15min":2.6709999999999998},"last":0.495}]},{"osd":2,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.3169999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.75},"min":{"1min":0.54900000000000004,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.64900000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.72599999999999998,"15min":0.75800000000000001},"min":{"1min":0.54400000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.131,"5min":1.131,"15min":3.1960000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.78000000000000003,"15min":0.78700000000000003},"min":{"1min":0.44500000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.718,"5min":2.718,"15min":4.3760000000000003},"last":0.78200000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.62,"15min":0.61399999999999999},"min":{"1min":0.437,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.48999999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.544,"5min":2.544,"15min":3.1219999999999999},"last":0.89100000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.68700000000000006,"15min":0.67100000000000004},"min":{"1min":0.36499999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.2280000000000002,"5min":3.2280000000000002,"15min":3.2280000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.72299999999999998,"15min":0.69399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2.835,"5min":2.835,"15min":2.835},"last":0.73299999999999998}]},{"osd":7,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.54200000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.74199999999999999,"15min":0.73099999999999998},"min":{"1min":0.503,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.7170000000000001,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.752}]}]},{"osd":1,"up_from":13,"seq":55834575036,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50292,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321548,"statfs":{"total":96636764160,"available":96585265152,"internally_reserved":0,"allocated":1155072,"data_stored":539533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.315,"5min":0.219,"15min":0.219},"max":{"1min":0.65300000000000002,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.68999999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71599999999999997,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.52900000000000003},"min":{"1min":0.36699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":4.282},"last":0.39500000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.70999999999999996,"5min":0.78500000000000003,"15min":4.3200000000000003},"last":0.501},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.48799999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.016,"5min":1.617,"15min":1.617},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76400000000000001,"15min":0.78800000000000003},"min":{"1min":0.60899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0289999999999999,"5min":1.1419999999999999,"15min":4.3559999999999999},"last":0.94899999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.51800000000000002},"max":{"1min":1.056,"5min":1.3939999999999999,"15min":1.9119999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78600000000000003,"15min":0.80800000000000005},"min":{"1min":0.56299999999999994,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.048,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.91600000000000004}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77800000000000002,"15min":0.81000000000000005},"min":{"1min":0.57499999999999996,"5min":0.52600000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":1.194,"15min":4.3899999999999997},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.093,"5min":1.468,"15min":1.468},"last":0.82599999999999996}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.62,"5min":0.58699999999999997,"15min":0.56200000000000006},"max":{"1min":1.127,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.83599999999999997},"min":{"1min":0.60999999999999999,"5min":0.57099999999999995,"15min":0.55700000000000005},"max":{"1min":1.0269999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.871}]}]},{"osd":0,"up_from":13,"seq":55834575038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1724416,"data_stored":1126702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.44900000000000001,"15min":0.45300000000000001},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.23999999999999999},"max":{"1min":0.628,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.48099999999999998,"15min":0.50700000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.628,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.497}]},{"osd":2,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.35499999999999998},"max":{"1min":0.70299999999999996,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.46700000000000003,"15min":0.49399999999999999},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.68500000000000005,"5min":0.83899999999999997,"15min":0.97199999999999998},"last":0.36399999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70199999999999996,"5min":0.80500000000000005,"15min":1.077},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58699999999999997,"15min":0.60799999999999998},"min":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.71099999999999997,"5min":0.876,"15min":1.0549999999999999},"last":0.623}]},{"osd":4,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.78600000000000003,"15min":0.79500000000000004},"min":{"1min":0.499,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":0.88200000000000001,"5min":1.625,"15min":1.625},"last":0.877},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.78900000000000003,"15min":0.81799999999999995},"min":{"1min":0.5,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":0.97399999999999998,"5min":1.1419999999999999,"15min":1.167},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.72799999999999998,"15min":0.752},"min":{"1min":0.438,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.79200000000000004,"5min":0.99399999999999999,"15min":1.137},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.79800000000000004},"min":{"1min":0.54000000000000004,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":0.90900000000000003,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.94299999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.78800000000000003,"15min":0.81399999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53400000000000003},"max":{"1min":0.92200000000000004,"5min":1.0640000000000001,"15min":1.228},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.749,"5min":0.82299999999999995,"15min":0.83799999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94199999999999995,"5min":1.601,"15min":1.601},"last":0.79100000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76100000000000001,"15min":0.79200000000000004},"min":{"1min":0.48599999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90300000000000002,"5min":1.03,"15min":1.1890000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.81799999999999995,"15min":0.83599999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.96399999999999997,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.91900000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40432,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331408,"statfs":{"total":96636764160,"available":96595361792,"internally_reserved":0,"allocated":1146880,"data_stored":538917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.28999999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.746,"5min":0.746,"15min":0.86599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.505,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.377,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.67900000000000005,"5min":0.78800000000000003,"15min":1.3140000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.48399999999999999,"15min":0.47499999999999998},"min":{"1min":0.35699999999999998,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.94499999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":1.032,"15min":1.032},"last":0.61099999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.46300000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.74099999999999999,"5min":0.80000000000000004,"15min":1.127},"last":0.623},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.36699999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.64800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.76800000000000002,"15min":0.78300000000000003},"min":{"1min":0.54300000000000004,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":1.0209999999999999,"5min":1.1799999999999999,"15min":1.554},"last":0.871},{"interface":"front","average":{"1min":0.755,"5min":0.77700000000000002,"15min":0.80500000000000005},"min":{"1min":0.59399999999999997,"5min":0.55000000000000004,"15min":0.51600000000000001},"max":{"1min":0.95099999999999996,"5min":1.236,"15min":1.577},"last":1.0029999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78600000000000003,"15min":0.80700000000000005},"min":{"1min":0.55200000000000005,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96699999999999997,"5min":1.25,"15min":1.5209999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.77500000000000002,"15min":0.80300000000000005},"min":{"1min":0.55000000000000004,"5min":0.53200000000000003,"15min":0.51800000000000002},"max":{"1min":0.98399999999999999,"5min":1.2250000000000001,"15min":1.3300000000000001},"last":1.115}]},{"osd":6,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.79400000000000004,"15min":0.82099999999999995},"min":{"1min":0.58399999999999996,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1220000000000001,"5min":1.1539999999999999,"15min":1.758},"last":1.107},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81999999999999995,"15min":0.83899999999999997},"min":{"1min":0.63900000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0940000000000001,"5min":1.3560000000000001,"15min":1.4750000000000001},"last":1.085}]},{"osd":7,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.84999999999999998,"15min":0.86499999999999999},"min":{"1min":0.58799999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.071,"5min":1.2709999999999999,"15min":1.4490000000000001},"last":1.1339999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.88},"min":{"1min":0.68200000000000005,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.056,"5min":1.3049999999999999,"15min":1.4390000000000001},"last":1.056}]}]},{"osd":3,"up_from":13,"seq":55834575037,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36332,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335508,"statfs":{"total":96636764160,"available":96599560192,"internally_reserved":0,"allocated":1142784,"data_stored":537465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.45600000000000002,"15min":0.46200000000000002},"min":{"1min":0.309,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.75600000000000001,"5min":0.79500000000000004,"15min":4.3799999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73699999999999999,"5min":0.86899999999999999,"15min":4.6379999999999999},"last":0.41299999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.34200000000000003,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.59599999999999997,"5min":0.98699999999999999,"15min":4.1840000000000002},"last":0.441},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.314},"max":{"1min":0.71299999999999997,"5min":0.96099999999999997,"15min":1.048},"last":0.42899999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.33700000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":1.369},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.63600000000000001},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.86199999999999999,"5min":0.94799999999999995,"15min":4.5970000000000004},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.755,"15min":0.77200000000000002},"min":{"1min":0.497,"5min":0.439,"15min":0.439},"max":{"1min":0.98599999999999999,"5min":1.288,"15min":4.3319999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.755,"15min":0.77300000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.39400000000000002},"max":{"1min":1.0900000000000001,"5min":1.46,"15min":1.46},"last":0.78100000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.435},"max":{"1min":0.96199999999999997,"5min":1.3580000000000001,"15min":1.3740000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.78000000000000003,"15min":0.80100000000000005},"min":{"1min":0.57899999999999996,"5min":0.46899999999999997,"15min":0.44500000000000001},"max":{"1min":0.92700000000000005,"5min":1.397,"15min":4.4180000000000001},"last":0.96099999999999997}]},{"osd":6,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.79400000000000004,"15min":0.80800000000000005},"min":{"1min":0.54500000000000004,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.0189999999999999,"5min":1.2549999999999999,"15min":1.4730000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.80500000000000005,"15min":0.81499999999999995},"min":{"1min":0.59199999999999997,"5min":0.52400000000000002,"15min":0.503},"max":{"1min":1.0349999999999999,"5min":1.23,"15min":1.417},"last":0.93999999999999995}]},{"osd":7,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.82799999999999996,"15min":0.83899999999999997},"min":{"1min":0.60699999999999998,"5min":0.57299999999999995,"15min":0.54300000000000004},"max":{"1min":1.0049999999999999,"5min":1.5,"15min":4.4550000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.82199999999999995,"15min":0.83599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.56200000000000006},"max":{"1min":1.0489999999999999,"5min":1.345,"15min":1.476},"last":0.79900000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575038,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54376,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317464,"statfs":{"total":96636764160,"available":96581083136,"internally_reserved":0,"allocated":1138688,"data_stored":537423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.68700000000000006},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.88200000000000001,"5min":0.93200000000000005,"15min":1.417},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.92100000000000004,"5min":1.165,"15min":1.165},"last":0.52800000000000002}]},{"osd":1,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.93500000000000005,"5min":0.93700000000000006,"15min":1.139},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.333},"last":0.498}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.01,"15min":1.863},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.75,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93700000000000006,"5min":0.96299999999999997,"15min":1.387},"last":0.67200000000000004}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.5,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.88700000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.51600000000000001,"5min":0.48699999999999999,"15min":0.46600000000000003},"max":{"1min":0.80500000000000005,"5min":1.1539999999999999,"15min":1.905},"last":0.68400000000000005}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.46700000000000003,"5min":0.39300000000000002,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.48099999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.78500000000000003,"5min":1.294,"15min":1.294},"last":0.81599999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.48099999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.75900000000000001,"5min":1.0129999999999999,"15min":1.9339999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.66200000000000003},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.433},"max":{"1min":0.85299999999999998,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.78200000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.48099999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85999999999999999,"5min":1.345,"15min":1.345},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.59099999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89700000000000002,"5min":1.214,"15min":1.97},"last":0.88800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575037,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1138688,"data_stored":537443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.68300000000000005,"15min":0.68400000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.313},"max":{"1min":0.84099999999999997,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.73399999999999999,"15min":0.72399999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.33500000000000002},"max":{"1min":0.90400000000000003,"5min":1.21,"15min":1.339},"last":0.79400000000000004}]},{"osd":1,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71399999999999997,"15min":0.71899999999999997},"min":{"1min":0.53100000000000003,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.96499999999999997,"5min":0.96699999999999997,"15min":1.0169999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.70999999999999996},"min":{"1min":0.50700000000000001,"5min":0.46700000000000003,"15min":0.39200000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.6040000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.73199999999999998,"15min":0.73299999999999998},"min":{"1min":0.51700000000000002,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.95799999999999996,"5min":0.98899999999999999,"15min":1.476},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70999999999999996,"15min":0.71199999999999997},"min":{"1min":0.46200000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83899999999999997,"5min":1.095,"15min":1.502},"last":0.68999999999999995}]},{"osd":3,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.72499999999999998,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":1.042,"5min":1.113,"15min":1.1619999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.74199999999999999,"15min":0.754},"min":{"1min":0.60999999999999999,"5min":0.52300000000000002,"15min":0.46500000000000002},"max":{"1min":0.84499999999999997,"5min":1.0249999999999999,"15min":1.744},"last":0.65300000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.59999999999999998},"min":{"1min":0.42699999999999999,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.84899999999999998,"5min":0.94599999999999995,"15min":0.95899999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65100000000000002,"15min":0.64100000000000001},"min":{"1min":0.47399999999999998,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":1.097,"5min":1.097,"15min":1.214},"last":0.61099999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.504,"5min":0.44600000000000001,"15min":0.36799999999999999},"max":{"1min":0.874,"5min":0.91300000000000003,"15min":1.151},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.67700000000000005,"15min":0.67900000000000005},"min":{"1min":0.45900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95899999999999996,"5min":0.97099999999999997,"15min":1.5489999999999999},"last":0.72399999999999998}]},{"osd":7,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.69999999999999996,"15min":0.70599999999999996},"min":{"1min":0.56399999999999995,"5min":0.41899999999999998,"15min":0.40200000000000002},"max":{"1min":0.90200000000000002,"5min":0.95299999999999996,"15min":1.325},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.73399999999999999,"15min":0.73499999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91800000000000004,"5min":0.98199999999999998,"15min":1.1990000000000001},"last":0.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:10:22.569 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:10:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.802+0000 7f35c2200700 1 Processor -- start 2024-03-29T10:10:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.802+0000 7f35c2200700 1 -- start start 2024-03-29T10:10:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.803+0000 7f35c2200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f35bc0b3af0 0x7f35bc15a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.803+0000 7f35c2200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35bc072580 0x7f35bc0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35c2200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc0b2180 con 0x7f35bc0b2e50 2024-03-29T10:10:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35c2200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc15b100 con 0x7f35bc072580 2024-03-29T10:10:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35c2200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc15b3e0 con 0x7f35bc0b3af0 2024-03-29T10:10:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35baa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35bc072580 0x7f35bc0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35baa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35bc072580 0x7f35bc0b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37954/0 (socket says 172.21.15.12:37954) 2024-03-29T10:10:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35baa00700 1 -- 172.21.15.12:0/2264167028 learned_addr learned my addr 172.21.15.12:0/2264167028 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35bb400700 1 --2- 172.21.15.12:0/2264167028 >> v2:172.21.15.159:3300/0 conn(0x7f35bc0b3af0 0x7f35bc15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35baa00700 1 -- 172.21.15.12:0/2264167028 >> v2:172.21.15.159:3300/0 conn(0x7f35bc0b3af0 msgr2=0x7f35bc15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35baa00700 1 --2- 172.21.15.12:0/2264167028 >> v2:172.21.15.159:3300/0 conn(0x7f35bc0b3af0 0x7f35bc15a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35baa00700 1 -- 172.21.15.12:0/2264167028 >> v1:172.21.15.159:6789/0 conn(0x7f35bc0b2e50 legacy=0x7f35bc0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.804+0000 7f35baa00700 1 -- 172.21.15.12:0/2264167028 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc0b23f0 con 0x7f35bc072580 2024-03-29T10:10:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35baa00700 1 --2- 172.21.15.12:0/2264167028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc072580 0x7f35bc0b2910 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f35bc06b860 tx=0x7f35a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d0c9d685a676e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35ba000700 1 -- 172.21.15.12:0/2264167028 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8011830 con 0x7f35bc072580 2024-03-29T10:10:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35ba000700 1 -- 172.21.15.12:0/2264167028 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8011990 con 0x7f35bc072580 2024-03-29T10:10:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35ba000700 1 -- 172.21.15.12:0/2264167028 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8011d20 con 0x7f35bc072580 2024-03-29T10:10:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35c2200700 1 -- 172.21.15.12:0/2264167028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc072580 msgr2=0x7f35bc0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35c2200700 1 --2- 172.21.15.12:0/2264167028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc072580 0x7f35bc0b2910 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f35bc06b860 tx=0x7f35a8009bd0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35c2200700 1 -- 172.21.15.12:0/2264167028 shutdown_connections 2024-03-29T10:10:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35c2200700 1 --2- 172.21.15.12:0/2264167028 >> v2:172.21.15.159:3300/0 conn(0x7f35bc0b3af0 0x7f35bc15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35c2200700 1 --2- 172.21.15.12:0/2264167028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc072580 0x7f35bc0b2910 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35c2200700 1 -- 172.21.15.12:0/2264167028 >> 172.21.15.12:0/2264167028 conn(0x7f35bc0cf190 msgr2=0x7f35bc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.805+0000 7f35c2200700 1 -- 172.21.15.12:0/2264167028 shutdown_connections 2024-03-29T10:10:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.806+0000 7f35c2200700 1 -- 172.21.15.12:0/2264167028 wait complete. 2024-03-29T10:10:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.807+0000 7f35c2200700 1 Processor -- start 2024-03-29T10:10:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.807+0000 7f35c2200700 1 -- start start 2024-03-29T10:10:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.808+0000 7f35c2200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35bc072580 0x7f35bc14ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.808+0000 7f35c2200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35bc0b2e50 0x7f35bc1540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.808+0000 7f35c2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc0b3af0 0x7f35bc06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.808+0000 7f35c2200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc155020 con 0x7f35bc0b3af0 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.808+0000 7f35c2200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc155300 con 0x7f35bc0b2e50 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.808+0000 7f35c2200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc1555e0 con 0x7f35bc072580 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc0b3af0 0x7f35bc06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35bb400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35bc072580 0x7f35bc14ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc0b3af0 0x7f35bc06d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37966/0 (socket says 172.21.15.12:37966) 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35bbe00700 1 -- 172.21.15.12:0/190138798 learned_addr learned my addr 172.21.15.12:0/190138798 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35baa00700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35bc0b2e50 0x7f35bc1540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35bbe00700 1 -- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35bc072580 msgr2=0x7f35bc14ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35bbe00700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35bc072580 0x7f35bc14ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35bbe00700 1 -- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35bc0b2e50 msgr2=0x7f35bc1540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35bbe00700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35bc0b2e50 0x7f35bc1540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35bbe00700 1 -- 172.21.15.12:0/190138798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a8009550 con 0x7f35bc0b3af0 2024-03-29T10:10:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.809+0000 7f35bbe00700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc0b3af0 0x7f35bc06d260 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f35b400cc50 tx=0x7f35b400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.810+0000 7f35b3400700 1 -- 172.21.15.12:0/190138798 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b4003c00 con 0x7f35bc0b3af0 2024-03-29T10:10:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.810+0000 7f35b3400700 1 -- 172.21.15.12:0/190138798 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b4003d60 con 0x7f35bc0b3af0 2024-03-29T10:10:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.810+0000 7f35b3400700 1 -- 172.21.15.12:0/190138798 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b400aab0 con 0x7f35bc0b3af0 2024-03-29T10:10:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.810+0000 7f35c2200700 1 -- 172.21.15.12:0/190138798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc1535f0 con 0x7f35bc0b3af0 2024-03-29T10:10:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.810+0000 7f35c2200700 1 -- 172.21.15.12:0/190138798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc153bb0 con 0x7f35bc0b3af0 2024-03-29T10:10:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.811+0000 7f35c2200700 1 -- 172.21.15.12:0/190138798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc067720 con 0x7f35bc0b3af0 2024-03-29T10:10:22.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.812+0000 7f35b3400700 1 -- 172.21.15.12:0/190138798 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f35b400ac10 con 0x7f35bc0b3af0 2024-03-29T10:10:22.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.813+0000 7f35b3400700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f359c06adb0 0x7f359c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:22.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.813+0000 7f35b3400700 1 -- 172.21.15.12:0/190138798 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35b4089510 con 0x7f35bc0b3af0 2024-03-29T10:10:22.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.813+0000 7f35bb400700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f359c06adb0 0x7f359c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:22.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.814+0000 7f35bb400700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f359c06adb0 0x7f359c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f35bc159610 tx=0x7f35ac009210 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:22.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:22.817+0000 7f35b3400700 1 -- 172.21.15.12:0/190138798 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f35b4054540 con 0x7f35bc0b3af0 2024-03-29T10:10:23.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.027+0000 7f35c2200700 1 -- 172.21.15.12:0/190138798 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35bc15b6c0 con 0x7f359c06adb0 2024-03-29T10:10:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.037+0000 7f35b3400700 1 -- 172.21.15.12:0/190138798 <== mgr.5954 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288876 (secure 0 0 0) 0x7f35ac011840 con 0x7f359c06adb0 2024-03-29T10:10:23.040 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 -- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f359c06adb0 msgr2=0x7f359c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f359c06adb0 0x7f359c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f35bc159610 tx=0x7f35ac009210 comp rx=0 tx=0).stop 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 -- 172.21.15.12:0/190138798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc0b3af0 msgr2=0x7f35bc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc0b3af0 0x7f35bc06d260 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f35b400cc50 tx=0x7f35b400cf60 comp rx=0 tx=0).stop 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 -- 172.21.15.12:0/190138798 shutdown_connections 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f359c06adb0 0x7f359c06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35bc0b3af0 0x7f35bc06d260 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35bc072580 0x7f35bc14ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 --2- 172.21.15.12:0/190138798 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35bc0b2e50 0x7f35bc1540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 -- 172.21.15.12:0/190138798 >> 172.21.15.12:0/190138798 conn(0x7f35bc0cf190 msgr2=0x7f35bc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:23.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 -- 172.21.15.12:0/190138798 shutdown_connections 2024-03-29T10:10:23.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.041+0000 7f35c2200700 1 -- 172.21.15.12:0/190138798 wait complete. 2024-03-29T10:10:23.044 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:10:23.059 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-29T10:10:22.820976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":311532,"kb_used_data":10084,"kb_used_omap":12,"kb_used_meta":301299,"kb_avail":754663188,"statfs":{"total":773094113280,"available":772775104512,"internally_reserved":0,"allocated":10326016,"data_stored":5483265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":308530745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001585"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.667006+0000","last_change":"2024-03-29T10:10:17.663972+0000","last_active":"2024-03-29T10:10:17.667006+0000","last_peered":"2024-03-29T10:10:17.667006+0000","last_clean":"2024-03-29T10:10:17.667006+0000","last_became_active":"2024-03-29T10:10:17.663691+0000","last_became_peered":"2024-03-29T10:10:17.663691+0000","last_unstale":"2024-03-29T10:10:17.667006+0000","last_undegraded":"2024-03-29T10:10:17.667006+0000","last_fullsized":"2024-03-29T10:10:17.667006+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:32:49.333388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637719+0000","last_change":"2024-03-29T10:10:17.654165+0000","last_active":"2024-03-29T10:10:18.637719+0000","last_peered":"2024-03-29T10:10:18.637719+0000","last_clean":"2024-03-29T10:10:18.637719+0000","last_became_active":"2024-03-29T10:10:17.653207+0000","last_became_peered":"2024-03-29T10:10:17.653207+0000","last_unstale":"2024-03-29T10:10:18.637719+0000","last_undegraded":"2024-03-29T10:10:18.637719+0000","last_fullsized":"2024-03-29T10:10:18.637719+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:47.601889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.648762+0000","last_change":"2024-03-29T10:10:17.645506+0000","last_active":"2024-03-29T10:10:17.648762+0000","last_peered":"2024-03-29T10:10:17.648762+0000","last_clean":"2024-03-29T10:10:17.648762+0000","last_became_active":"2024-03-29T10:10:17.645244+0000","last_became_peered":"2024-03-29T10:10:17.645244+0000","last_unstale":"2024-03-29T10:10:17.648762+0000","last_undegraded":"2024-03-29T10:10:17.648762+0000","last_fullsized":"2024-03-29T10:10:17.648762+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:07.975443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631810+0000","last_change":"2024-03-29T10:10:17.654242+0000","last_active":"2024-03-29T10:10:18.631810+0000","last_peered":"2024-03-29T10:10:18.631810+0000","last_clean":"2024-03-29T10:10:18.631810+0000","last_became_active":"2024-03-29T10:10:17.653740+0000","last_became_peered":"2024-03-29T10:10:17.653740+0000","last_unstale":"2024-03-29T10:10:18.631810+0000","last_undegraded":"2024-03-29T10:10:18.631810+0000","last_fullsized":"2024-03-29T10:10:18.631810+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:42.641698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663602+0000","last_change":"2024-03-29T10:10:17.656879+0000","last_active":"2024-03-29T10:10:17.663602+0000","last_peered":"2024-03-29T10:10:17.663602+0000","last_clean":"2024-03-29T10:10:17.663602+0000","last_became_active":"2024-03-29T10:10:17.656560+0000","last_became_peered":"2024-03-29T10:10:17.656560+0000","last_unstale":"2024-03-29T10:10:17.663602+0000","last_undegraded":"2024-03-29T10:10:17.663602+0000","last_fullsized":"2024-03-29T10:10:17.663602+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:35.582327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637017+0000","last_change":"2024-03-29T10:10:17.653703+0000","last_active":"2024-03-29T10:10:18.637017+0000","last_peered":"2024-03-29T10:10:18.637017+0000","last_clean":"2024-03-29T10:10:18.637017+0000","last_became_active":"2024-03-29T10:10:17.653398+0000","last_became_peered":"2024-03-29T10:10:17.653398+0000","last_unstale":"2024-03-29T10:10:18.637017+0000","last_undegraded":"2024-03-29T10:10:18.637017+0000","last_fullsized":"2024-03-29T10:10:18.637017+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:04.588448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636518+0000","last_change":"2024-03-29T10:10:17.642071+0000","last_active":"2024-03-29T10:10:18.636518+0000","last_peered":"2024-03-29T10:10:18.636518+0000","last_clean":"2024-03-29T10:10:18.636518+0000","last_became_active":"2024-03-29T10:10:17.638378+0000","last_became_peered":"2024-03-29T10:10:17.638378+0000","last_unstale":"2024-03-29T10:10:18.636518+0000","last_undegraded":"2024-03-29T10:10:18.636518+0000","last_fullsized":"2024-03-29T10:10:18.636518+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:08.435381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637107+0000","last_change":"2024-03-29T10:10:17.640398+0000","last_active":"2024-03-29T10:10:18.637107+0000","last_peered":"2024-03-29T10:10:18.637107+0000","last_clean":"2024-03-29T10:10:18.637107+0000","last_became_active":"2024-03-29T10:10:17.640044+0000","last_became_peered":"2024-03-29T10:10:17.640044+0000","last_unstale":"2024-03-29T10:10:18.637107+0000","last_undegraded":"2024-03-29T10:10:18.637107+0000","last_fullsized":"2024-03-29T10:10:18.637107+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:41:43.134146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664433+0000","last_change":"2024-03-29T10:10:17.659145+0000","last_active":"2024-03-29T10:10:17.664433+0000","last_peered":"2024-03-29T10:10:17.664433+0000","last_clean":"2024-03-29T10:10:17.664433+0000","last_became_active":"2024-03-29T10:10:17.658026+0000","last_became_peered":"2024-03-29T10:10:17.658026+0000","last_unstale":"2024-03-29T10:10:17.664433+0000","last_undegraded":"2024-03-29T10:10:17.664433+0000","last_fullsized":"2024-03-29T10:10:17.664433+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:50.247091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.662851+0000","last_change":"2024-03-29T10:10:17.659927+0000","last_active":"2024-03-29T10:10:17.662851+0000","last_peered":"2024-03-29T10:10:17.662851+0000","last_clean":"2024-03-29T10:10:17.662851+0000","last_became_active":"2024-03-29T10:10:17.659572+0000","last_became_peered":"2024-03-29T10:10:17.659572+0000","last_unstale":"2024-03-29T10:10:17.662851+0000","last_undegraded":"2024-03-29T10:10:17.662851+0000","last_fullsized":"2024-03-29T10:10:17.662851+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:13.147091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.662492+0000","last_change":"2024-03-29T10:10:17.657259+0000","last_active":"2024-03-29T10:10:17.662492+0000","last_peered":"2024-03-29T10:10:17.662492+0000","last_clean":"2024-03-29T10:10:17.662492+0000","last_became_active":"2024-03-29T10:10:17.656924+0000","last_became_peered":"2024-03-29T10:10:17.656924+0000","last_unstale":"2024-03-29T10:10:17.662492+0000","last_undegraded":"2024-03-29T10:10:17.662492+0000","last_fullsized":"2024-03-29T10:10:17.662492+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:33:07.032980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631834+0000","last_change":"2024-03-29T10:10:17.659739+0000","last_active":"2024-03-29T10:10:18.631834+0000","last_peered":"2024-03-29T10:10:18.631834+0000","last_clean":"2024-03-29T10:10:18.631834+0000","last_became_active":"2024-03-29T10:10:17.659488+0000","last_became_peered":"2024-03-29T10:10:17.659488+0000","last_unstale":"2024-03-29T10:10:18.631834+0000","last_undegraded":"2024-03-29T10:10:18.631834+0000","last_fullsized":"2024-03-29T10:10:18.631834+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:45:01.639827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.657931+0000","last_change":"2024-03-29T10:10:17.650043+0000","last_active":"2024-03-29T10:10:17.657931+0000","last_peered":"2024-03-29T10:10:17.657931+0000","last_clean":"2024-03-29T10:10:17.657931+0000","last_became_active":"2024-03-29T10:10:17.649701+0000","last_became_peered":"2024-03-29T10:10:17.649701+0000","last_unstale":"2024-03-29T10:10:17.657931+0000","last_undegraded":"2024-03-29T10:10:17.657931+0000","last_fullsized":"2024-03-29T10:10:17.657931+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:26:39.092036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664479+0000","last_change":"2024-03-29T10:10:17.659628+0000","last_active":"2024-03-29T10:10:17.664479+0000","last_peered":"2024-03-29T10:10:17.664479+0000","last_clean":"2024-03-29T10:10:17.664479+0000","last_became_active":"2024-03-29T10:10:17.656796+0000","last_became_peered":"2024-03-29T10:10:17.656796+0000","last_unstale":"2024-03-29T10:10:17.664479+0000","last_undegraded":"2024-03-29T10:10:17.664479+0000","last_fullsized":"2024-03-29T10:10:17.664479+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:13:14.570934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664640+0000","last_change":"2024-03-29T10:10:17.657742+0000","last_active":"2024-03-29T10:10:17.664640+0000","last_peered":"2024-03-29T10:10:17.664640+0000","last_clean":"2024-03-29T10:10:17.664640+0000","last_became_active":"2024-03-29T10:10:17.657470+0000","last_became_peered":"2024-03-29T10:10:17.657470+0000","last_unstale":"2024-03-29T10:10:17.664640+0000","last_undegraded":"2024-03-29T10:10:17.664640+0000","last_fullsized":"2024-03-29T10:10:17.664640+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:58.382328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.656812+0000","last_change":"2024-03-29T10:10:17.651786+0000","last_active":"2024-03-29T10:10:17.656812+0000","last_peered":"2024-03-29T10:10:17.656812+0000","last_clean":"2024-03-29T10:10:17.656812+0000","last_became_active":"2024-03-29T10:10:17.651524+0000","last_became_peered":"2024-03-29T10:10:17.651524+0000","last_unstale":"2024-03-29T10:10:17.656812+0000","last_undegraded":"2024-03-29T10:10:17.656812+0000","last_fullsized":"2024-03-29T10:10:17.656812+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:27.677477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663718+0000","last_change":"2024-03-29T10:10:17.657712+0000","last_active":"2024-03-29T10:10:17.663718+0000","last_peered":"2024-03-29T10:10:17.663718+0000","last_clean":"2024-03-29T10:10:17.663718+0000","last_became_active":"2024-03-29T10:10:17.656954+0000","last_became_peered":"2024-03-29T10:10:17.656954+0000","last_unstale":"2024-03-29T10:10:17.663718+0000","last_undegraded":"2024-03-29T10:10:17.663718+0000","last_fullsized":"2024-03-29T10:10:17.663718+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:44.085465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.632229+0000","last_change":"2024-03-29T10:10:17.643735+0000","last_active":"2024-03-29T10:10:18.632229+0000","last_peered":"2024-03-29T10:10:18.632229+0000","last_clean":"2024-03-29T10:10:18.632229+0000","last_became_active":"2024-03-29T10:10:17.643486+0000","last_became_peered":"2024-03-29T10:10:17.643486+0000","last_unstale":"2024-03-29T10:10:18.632229+0000","last_undegraded":"2024-03-29T10:10:18.632229+0000","last_fullsized":"2024-03-29T10:10:18.632229+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:47:17.241189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637462+0000","last_change":"2024-03-29T10:10:17.651977+0000","last_active":"2024-03-29T10:10:18.637462+0000","last_peered":"2024-03-29T10:10:18.637462+0000","last_clean":"2024-03-29T10:10:18.637462+0000","last_became_active":"2024-03-29T10:10:17.649831+0000","last_became_peered":"2024-03-29T10:10:17.649831+0000","last_unstale":"2024-03-29T10:10:18.637462+0000","last_undegraded":"2024-03-29T10:10:18.637462+0000","last_fullsized":"2024-03-29T10:10:18.637462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:11:01.442096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.649479+0000","last_change":"2024-03-29T10:10:17.647936+0000","last_active":"2024-03-29T10:10:17.649479+0000","last_peered":"2024-03-29T10:10:17.649479+0000","last_clean":"2024-03-29T10:10:17.649479+0000","last_became_active":"2024-03-29T10:10:17.646965+0000","last_became_peered":"2024-03-29T10:10:17.646965+0000","last_unstale":"2024-03-29T10:10:17.649479+0000","last_undegraded":"2024-03-29T10:10:17.649479+0000","last_fullsized":"2024-03-29T10:10:17.649479+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:42.303434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.655702+0000","last_change":"2024-03-29T10:10:17.648416+0000","last_active":"2024-03-29T10:10:17.655702+0000","last_peered":"2024-03-29T10:10:17.655702+0000","last_clean":"2024-03-29T10:10:17.655702+0000","last_became_active":"2024-03-29T10:10:17.647554+0000","last_became_peered":"2024-03-29T10:10:17.647554+0000","last_unstale":"2024-03-29T10:10:17.655702+0000","last_undegraded":"2024-03-29T10:10:17.655702+0000","last_fullsized":"2024-03-29T10:10:17.655702+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:57.257695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.665035+0000","last_change":"2024-03-29T10:10:17.663232+0000","last_active":"2024-03-29T10:10:17.665035+0000","last_peered":"2024-03-29T10:10:17.665035+0000","last_clean":"2024-03-29T10:10:17.665035+0000","last_became_active":"2024-03-29T10:10:17.662119+0000","last_became_peered":"2024-03-29T10:10:17.662119+0000","last_unstale":"2024-03-29T10:10:17.665035+0000","last_undegraded":"2024-03-29T10:10:17.665035+0000","last_fullsized":"2024-03-29T10:10:17.665035+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:41:43.134146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.658980+0000","last_change":"2024-03-29T10:10:17.653979+0000","last_active":"2024-03-29T10:10:17.658980+0000","last_peered":"2024-03-29T10:10:17.658980+0000","last_clean":"2024-03-29T10:10:17.658980+0000","last_became_active":"2024-03-29T10:10:17.653716+0000","last_became_peered":"2024-03-29T10:10:17.653716+0000","last_unstale":"2024-03-29T10:10:17.658980+0000","last_undegraded":"2024-03-29T10:10:17.658980+0000","last_fullsized":"2024-03-29T10:10:17.658980+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:21:56.054067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663470+0000","last_change":"2024-03-29T10:10:17.652657+0000","last_active":"2024-03-29T10:10:17.663470+0000","last_peered":"2024-03-29T10:10:17.663470+0000","last_clean":"2024-03-29T10:10:17.663470+0000","last_became_active":"2024-03-29T10:10:17.652395+0000","last_became_peered":"2024-03-29T10:10:17.652395+0000","last_unstale":"2024-03-29T10:10:17.663470+0000","last_undegraded":"2024-03-29T10:10:17.663470+0000","last_fullsized":"2024-03-29T10:10:17.663470+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:21:28.986763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.658629+0000","last_change":"2024-03-29T10:10:17.655786+0000","last_active":"2024-03-29T10:10:17.658629+0000","last_peered":"2024-03-29T10:10:17.658629+0000","last_clean":"2024-03-29T10:10:17.658629+0000","last_became_active":"2024-03-29T10:10:17.654162+0000","last_became_peered":"2024-03-29T10:10:17.654162+0000","last_unstale":"2024-03-29T10:10:17.658629+0000","last_undegraded":"2024-03-29T10:10:17.658629+0000","last_fullsized":"2024-03-29T10:10:17.658629+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:04.588448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637349+0000","last_change":"2024-03-29T10:10:17.641516+0000","last_active":"2024-03-29T10:10:18.637349+0000","last_peered":"2024-03-29T10:10:18.637349+0000","last_clean":"2024-03-29T10:10:18.637349+0000","last_became_active":"2024-03-29T10:10:17.641173+0000","last_became_peered":"2024-03-29T10:10:17.641173+0000","last_unstale":"2024-03-29T10:10:18.637349+0000","last_undegraded":"2024-03-29T10:10:18.637349+0000","last_fullsized":"2024-03-29T10:10:18.637349+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:04.588448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664790+0000","last_change":"2024-03-29T10:10:17.655838+0000","last_active":"2024-03-29T10:10:17.664790+0000","last_peered":"2024-03-29T10:10:17.664790+0000","last_clean":"2024-03-29T10:10:17.664790+0000","last_became_active":"2024-03-29T10:10:17.655584+0000","last_became_peered":"2024-03-29T10:10:17.655584+0000","last_unstale":"2024-03-29T10:10:17.664790+0000","last_undegraded":"2024-03-29T10:10:17.664790+0000","last_fullsized":"2024-03-29T10:10:17.664790+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:59:32.098044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.662876+0000","last_change":"2024-03-29T10:10:17.659801+0000","last_active":"2024-03-29T10:10:17.662876+0000","last_peered":"2024-03-29T10:10:17.662876+0000","last_clean":"2024-03-29T10:10:17.662876+0000","last_became_active":"2024-03-29T10:10:17.659451+0000","last_became_peered":"2024-03-29T10:10:17.659451+0000","last_unstale":"2024-03-29T10:10:17.662876+0000","last_undegraded":"2024-03-29T10:10:17.662876+0000","last_fullsized":"2024-03-29T10:10:17.662876+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:11:01.442096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.646602+0000","last_change":"2024-03-29T10:10:17.637556+0000","last_active":"2024-03-29T10:10:17.646602+0000","last_peered":"2024-03-29T10:10:17.646602+0000","last_clean":"2024-03-29T10:10:17.646602+0000","last_became_active":"2024-03-29T10:10:17.637299+0000","last_became_peered":"2024-03-29T10:10:17.637299+0000","last_unstale":"2024-03-29T10:10:17.646602+0000","last_undegraded":"2024-03-29T10:10:17.646602+0000","last_fullsized":"2024-03-29T10:10:17.646602+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:38.480817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636640+0000","last_change":"2024-03-29T10:10:17.637608+0000","last_active":"2024-03-29T10:10:18.636640+0000","last_peered":"2024-03-29T10:10:18.636640+0000","last_clean":"2024-03-29T10:10:18.636640+0000","last_became_active":"2024-03-29T10:10:17.637278+0000","last_became_peered":"2024-03-29T10:10:17.637278+0000","last_unstale":"2024-03-29T10:10:18.636640+0000","last_undegraded":"2024-03-29T10:10:18.636640+0000","last_fullsized":"2024-03-29T10:10:18.636640+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:47.601889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636499+0000","last_change":"2024-03-29T10:10:17.659753+0000","last_active":"2024-03-29T10:10:18.636499+0000","last_peered":"2024-03-29T10:10:18.636499+0000","last_clean":"2024-03-29T10:10:18.636499+0000","last_became_active":"2024-03-29T10:10:17.659482+0000","last_became_peered":"2024-03-29T10:10:17.659482+0000","last_unstale":"2024-03-29T10:10:18.636499+0000","last_undegraded":"2024-03-29T10:10:18.636499+0000","last_fullsized":"2024-03-29T10:10:18.636499+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:27.677477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637006+0000","last_change":"2024-03-29T10:10:17.646361+0000","last_active":"2024-03-29T10:10:18.637006+0000","last_peered":"2024-03-29T10:10:18.637006+0000","last_clean":"2024-03-29T10:10:18.637006+0000","last_became_active":"2024-03-29T10:10:17.646082+0000","last_became_peered":"2024-03-29T10:10:17.646082+0000","last_unstale":"2024-03-29T10:10:18.637006+0000","last_undegraded":"2024-03-29T10:10:18.637006+0000","last_fullsized":"2024-03-29T10:10:18.637006+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:42:27.168707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.669756+0000","last_change":"2024-03-29T10:10:17.664714+0000","last_active":"2024-03-29T10:10:17.669756+0000","last_peered":"2024-03-29T10:10:17.669756+0000","last_clean":"2024-03-29T10:10:17.669756+0000","last_became_active":"2024-03-29T10:10:17.664060+0000","last_became_peered":"2024-03-29T10:10:17.664060+0000","last_unstale":"2024-03-29T10:10:17.669756+0000","last_undegraded":"2024-03-29T10:10:17.669756+0000","last_fullsized":"2024-03-29T10:10:17.669756+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:36.566454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"93'4","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.662617+0000","last_change":"2024-03-29T10:10:17.126210+0000","last_active":"2024-03-29T10:10:17.662617+0000","last_peered":"2024-03-29T10:10:17.662617+0000","last_clean":"2024-03-29T10:10:17.662617+0000","last_became_active":"2024-03-29T10:10:17.125927+0000","last_became_peered":"2024-03-29T10:10:17.125927+0000","last_unstale":"2024-03-29T10:10:17.662617+0000","last_undegraded":"2024-03-29T10:10:17.662617+0000","last_fullsized":"2024-03-29T10:10:17.662617+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:11:00.439534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663693+0000","last_change":"2024-03-29T10:10:17.657004+0000","last_active":"2024-03-29T10:10:17.663693+0000","last_peered":"2024-03-29T10:10:17.663693+0000","last_clean":"2024-03-29T10:10:17.663693+0000","last_became_active":"2024-03-29T10:10:17.654677+0000","last_became_peered":"2024-03-29T10:10:17.654677+0000","last_unstale":"2024-03-29T10:10:17.663693+0000","last_undegraded":"2024-03-29T10:10:17.663693+0000","last_fullsized":"2024-03-29T10:10:17.663693+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:45.929036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"93'1","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.657252+0000","last_change":"2024-03-29T10:10:17.123538+0000","last_active":"2024-03-29T10:10:17.657252+0000","last_peered":"2024-03-29T10:10:17.657252+0000","last_clean":"2024-03-29T10:10:17.657252+0000","last_became_active":"2024-03-29T10:10:17.123274+0000","last_became_peered":"2024-03-29T10:10:17.123274+0000","last_unstale":"2024-03-29T10:10:17.657252+0000","last_undegraded":"2024-03-29T10:10:17.657252+0000","last_fullsized":"2024-03-29T10:10:17.657252+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:41:42.131584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637486+0000","last_change":"2024-03-29T10:10:17.653648+0000","last_active":"2024-03-29T10:10:18.637486+0000","last_peered":"2024-03-29T10:10:18.637486+0000","last_clean":"2024-03-29T10:10:18.637486+0000","last_became_active":"2024-03-29T10:10:17.653378+0000","last_became_peered":"2024-03-29T10:10:17.653378+0000","last_unstale":"2024-03-29T10:10:18.637486+0000","last_undegraded":"2024-03-29T10:10:18.637486+0000","last_fullsized":"2024-03-29T10:10:18.637486+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:57.257695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"93'2","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664182+0000","last_change":"2024-03-29T10:10:17.131303+0000","last_active":"2024-03-29T10:10:17.664182+0000","last_peered":"2024-03-29T10:10:17.664182+0000","last_clean":"2024-03-29T10:10:17.664182+0000","last_became_active":"2024-03-29T10:10:17.130984+0000","last_became_peered":"2024-03-29T10:10:17.130984+0000","last_unstale":"2024-03-29T10:10:17.664182+0000","last_undegraded":"2024-03-29T10:10:17.664182+0000","last_fullsized":"2024-03-29T10:10:17.664182+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.082904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.665033+0000","last_change":"2024-03-29T10:10:17.662407+0000","last_active":"2024-03-29T10:10:17.665033+0000","last_peered":"2024-03-29T10:10:17.665033+0000","last_clean":"2024-03-29T10:10:17.665033+0000","last_became_active":"2024-03-29T10:10:17.660494+0000","last_became_peered":"2024-03-29T10:10:17.660494+0000","last_unstale":"2024-03-29T10:10:17.665033+0000","last_undegraded":"2024-03-29T10:10:17.665033+0000","last_fullsized":"2024-03-29T10:10:17.665033+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:26:39.092036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"93'2","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637693+0000","last_change":"2024-03-29T10:10:17.123003+0000","last_active":"2024-03-29T10:10:18.637693+0000","last_peered":"2024-03-29T10:10:18.637693+0000","last_clean":"2024-03-29T10:10:18.637693+0000","last_became_active":"2024-03-29T10:10:17.120876+0000","last_became_peered":"2024-03-29T10:10:17.120876+0000","last_unstale":"2024-03-29T10:10:18.637693+0000","last_undegraded":"2024-03-29T10:10:18.637693+0000","last_fullsized":"2024-03-29T10:10:18.637693+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:34.579766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636904+0000","last_change":"2024-03-29T10:10:17.647351+0000","last_active":"2024-03-29T10:10:18.636904+0000","last_peered":"2024-03-29T10:10:18.636904+0000","last_clean":"2024-03-29T10:10:18.636904+0000","last_became_active":"2024-03-29T10:10:17.646462+0000","last_became_peered":"2024-03-29T10:10:17.646462+0000","last_unstale":"2024-03-29T10:10:18.636904+0000","last_undegraded":"2024-03-29T10:10:18.636904+0000","last_fullsized":"2024-03-29T10:10:18.636904+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:11:01.442096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"93'1","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.656001+0000","last_change":"2024-03-29T10:10:17.126464+0000","last_active":"2024-03-29T10:10:17.656001+0000","last_peered":"2024-03-29T10:10:17.656001+0000","last_clean":"2024-03-29T10:10:17.656001+0000","last_became_active":"2024-03-29T10:10:17.126214+0000","last_became_peered":"2024-03-29T10:10:17.126214+0000","last_unstale":"2024-03-29T10:10:17.656001+0000","last_undegraded":"2024-03-29T10:10:17.656001+0000","last_fullsized":"2024-03-29T10:10:17.656001+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:02.154374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.655749+0000","last_change":"2024-03-29T10:10:17.648596+0000","last_active":"2024-03-29T10:10:17.655749+0000","last_peered":"2024-03-29T10:10:17.655749+0000","last_clean":"2024-03-29T10:10:17.655749+0000","last_became_active":"2024-03-29T10:10:17.648126+0000","last_became_peered":"2024-03-29T10:10:17.648126+0000","last_unstale":"2024-03-29T10:10:17.655749+0000","last_undegraded":"2024-03-29T10:10:17.655749+0000","last_fullsized":"2024-03-29T10:10:17.655749+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:47.601889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"93'1","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636820+0000","last_change":"2024-03-29T10:10:17.120935+0000","last_active":"2024-03-29T10:10:18.636820+0000","last_peered":"2024-03-29T10:10:18.636820+0000","last_clean":"2024-03-29T10:10:18.636820+0000","last_became_active":"2024-03-29T10:10:17.120686+0000","last_became_peered":"2024-03-29T10:10:17.120686+0000","last_unstale":"2024-03-29T10:10:18.636820+0000","last_undegraded":"2024-03-29T10:10:18.636820+0000","last_fullsized":"2024-03-29T10:10:18.636820+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:12:56.523092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663770+0000","last_change":"2024-03-29T10:10:17.656894+0000","last_active":"2024-03-29T10:10:17.663770+0000","last_peered":"2024-03-29T10:10:17.663770+0000","last_clean":"2024-03-29T10:10:17.663770+0000","last_became_active":"2024-03-29T10:10:17.656577+0000","last_became_peered":"2024-03-29T10:10:17.656577+0000","last_unstale":"2024-03-29T10:10:17.663770+0000","last_undegraded":"2024-03-29T10:10:17.663770+0000","last_fullsized":"2024-03-29T10:10:17.663770+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:35:47.779265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631447+0000","last_change":"2024-03-29T10:10:17.125986+0000","last_active":"2024-03-29T10:10:18.631447+0000","last_peered":"2024-03-29T10:10:18.631447+0000","last_clean":"2024-03-29T10:10:18.631447+0000","last_became_active":"2024-03-29T10:10:17.125718+0000","last_became_peered":"2024-03-29T10:10:17.125718+0000","last_unstale":"2024-03-29T10:10:18.631447+0000","last_undegraded":"2024-03-29T10:10:18.631447+0000","last_fullsized":"2024-03-29T10:10:18.631447+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:42:19.065517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663168+0000","last_change":"2024-03-29T10:10:17.652224+0000","last_active":"2024-03-29T10:10:17.663168+0000","last_peered":"2024-03-29T10:10:17.663168+0000","last_clean":"2024-03-29T10:10:17.663168+0000","last_became_active":"2024-03-29T10:10:17.648943+0000","last_became_peered":"2024-03-29T10:10:17.648943+0000","last_unstale":"2024-03-29T10:10:17.663168+0000","last_undegraded":"2024-03-29T10:10:17.663168+0000","last_fullsized":"2024-03-29T10:10:17.663168+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:42:28.134507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.654598+0000","last_change":"2024-03-29T10:10:17.117316+0000","last_active":"2024-03-29T10:10:17.654598+0000","last_peered":"2024-03-29T10:10:17.654598+0000","last_clean":"2024-03-29T10:10:17.654598+0000","last_became_active":"2024-03-29T10:10:17.117051+0000","last_became_peered":"2024-03-29T10:10:17.117051+0000","last_unstale":"2024-03-29T10:10:17.654598+0000","last_undegraded":"2024-03-29T10:10:17.654598+0000","last_fullsized":"2024-03-29T10:10:17.654598+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:11.581029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637454+0000","last_change":"2024-03-29T10:10:17.654321+0000","last_active":"2024-03-29T10:10:18.637454+0000","last_peered":"2024-03-29T10:10:18.637454+0000","last_clean":"2024-03-29T10:10:18.637454+0000","last_became_active":"2024-03-29T10:10:17.654030+0000","last_became_peered":"2024-03-29T10:10:17.654030+0000","last_unstale":"2024-03-29T10:10:18.637454+0000","last_undegraded":"2024-03-29T10:10:18.637454+0000","last_fullsized":"2024-03-29T10:10:18.637454+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:20.641015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.643289+0000","last_change":"2024-03-29T10:10:17.124975+0000","last_active":"2024-03-29T10:10:17.643289+0000","last_peered":"2024-03-29T10:10:17.643289+0000","last_clean":"2024-03-29T10:10:17.643289+0000","last_became_active":"2024-03-29T10:10:17.124599+0000","last_became_peered":"2024-03-29T10:10:17.124599+0000","last_unstale":"2024-03-29T10:10:17.643289+0000","last_undegraded":"2024-03-29T10:10:17.643289+0000","last_fullsized":"2024-03-29T10:10:17.643289+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:13:13.568372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.667029+0000","last_change":"2024-03-29T10:10:17.664343+0000","last_active":"2024-03-29T10:10:17.667029+0000","last_peered":"2024-03-29T10:10:17.667029+0000","last_clean":"2024-03-29T10:10:17.667029+0000","last_became_active":"2024-03-29T10:10:17.663984+0000","last_became_peered":"2024-03-29T10:10:17.663984+0000","last_unstale":"2024-03-29T10:10:17.667029+0000","last_undegraded":"2024-03-29T10:10:17.667029+0000","last_fullsized":"2024-03-29T10:10:17.667029+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:07.055847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637805+0000","last_change":"2024-03-29T10:10:17.118934+0000","last_active":"2024-03-29T10:10:18.637805+0000","last_peered":"2024-03-29T10:10:18.637805+0000","last_clean":"2024-03-29T10:10:18.637805+0000","last_became_active":"2024-03-29T10:10:17.118294+0000","last_became_peered":"2024-03-29T10:10:17.118294+0000","last_unstale":"2024-03-29T10:10:18.637805+0000","last_undegraded":"2024-03-29T10:10:18.637805+0000","last_fullsized":"2024-03-29T10:10:18.637805+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:49.244529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637617+0000","last_change":"2024-03-29T10:10:17.653094+0000","last_active":"2024-03-29T10:10:18.637617+0000","last_peered":"2024-03-29T10:10:18.637617+0000","last_clean":"2024-03-29T10:10:18.637617+0000","last_became_active":"2024-03-29T10:10:17.652194+0000","last_became_peered":"2024-03-29T10:10:17.652194+0000","last_unstale":"2024-03-29T10:10:18.637617+0000","last_undegraded":"2024-03-29T10:10:18.637617+0000","last_fullsized":"2024-03-29T10:10:18.637617+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:27.677477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"93'3","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.651819+0000","last_change":"2024-03-29T10:10:17.119279+0000","last_active":"2024-03-29T10:10:17.651819+0000","last_peered":"2024-03-29T10:10:17.651819+0000","last_clean":"2024-03-29T10:10:17.651819+0000","last_became_active":"2024-03-29T10:10:17.119006+0000","last_became_peered":"2024-03-29T10:10:17.119006+0000","last_unstale":"2024-03-29T10:10:17.651819+0000","last_undegraded":"2024-03-29T10:10:17.651819+0000","last_fullsized":"2024-03-29T10:10:17.651819+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:33:06.030419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.660297+0000","last_change":"2024-03-29T10:10:17.655328+0000","last_active":"2024-03-29T10:10:17.660297+0000","last_peered":"2024-03-29T10:10:17.660297+0000","last_clean":"2024-03-29T10:10:17.660297+0000","last_became_active":"2024-03-29T10:10:17.655085+0000","last_became_peered":"2024-03-29T10:10:17.655085+0000","last_unstale":"2024-03-29T10:10:17.660297+0000","last_undegraded":"2024-03-29T10:10:17.660297+0000","last_fullsized":"2024-03-29T10:10:17.660297+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:30.413167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"93'6","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631952+0000","last_change":"2024-03-29T10:10:17.118649+0000","last_active":"2024-03-29T10:10:18.631952+0000","last_peered":"2024-03-29T10:10:18.631952+0000","last_clean":"2024-03-29T10:10:18.631952+0000","last_became_active":"2024-03-29T10:10:17.118214+0000","last_became_peered":"2024-03-29T10:10:17.118214+0000","last_unstale":"2024-03-29T10:10:18.631952+0000","last_undegraded":"2024-03-29T10:10:18.631952+0000","last_fullsized":"2024-03-29T10:10:18.631952+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:47:04.088591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.665022+0000","last_change":"2024-03-29T10:10:17.662283+0000","last_active":"2024-03-29T10:10:17.665022+0000","last_peered":"2024-03-29T10:10:17.665022+0000","last_clean":"2024-03-29T10:10:17.665022+0000","last_became_active":"2024-03-29T10:10:17.661322+0000","last_became_peered":"2024-03-29T10:10:17.661322+0000","last_unstale":"2024-03-29T10:10:17.665022+0000","last_undegraded":"2024-03-29T10:10:17.665022+0000","last_fullsized":"2024-03-29T10:10:17.665022+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:03.156935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"93'2","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631628+0000","last_change":"2024-03-29T10:10:17.119319+0000","last_active":"2024-03-29T10:10:18.631628+0000","last_peered":"2024-03-29T10:10:18.631628+0000","last_clean":"2024-03-29T10:10:18.631628+0000","last_became_active":"2024-03-29T10:10:17.119069+0000","last_became_peered":"2024-03-29T10:10:17.119069+0000","last_unstale":"2024-03-29T10:10:18.631628+0000","last_undegraded":"2024-03-29T10:10:18.631628+0000","last_fullsized":"2024-03-29T10:10:18.631628+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:16:52.646444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637131+0000","last_change":"2024-03-29T10:10:17.648356+0000","last_active":"2024-03-29T10:10:18.637131+0000","last_peered":"2024-03-29T10:10:18.637131+0000","last_clean":"2024-03-29T10:10:18.637131+0000","last_became_active":"2024-03-29T10:10:17.648024+0000","last_became_peered":"2024-03-29T10:10:17.648024+0000","last_unstale":"2024-03-29T10:10:18.637131+0000","last_undegraded":"2024-03-29T10:10:18.637131+0000","last_fullsized":"2024-03-29T10:10:18.637131+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:41:43.134146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636919+0000","last_change":"2024-03-29T10:10:17.119202+0000","last_active":"2024-03-29T10:10:18.636919+0000","last_peered":"2024-03-29T10:10:18.636919+0000","last_clean":"2024-03-29T10:10:18.636919+0000","last_became_active":"2024-03-29T10:10:17.118930+0000","last_became_peered":"2024-03-29T10:10:17.118930+0000","last_unstale":"2024-03-29T10:10:18.636919+0000","last_undegraded":"2024-03-29T10:10:18.636919+0000","last_fullsized":"2024-03-29T10:10:18.636919+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:56:41.191132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.651415+0000","last_change":"2024-03-29T10:10:17.645985+0000","last_active":"2024-03-29T10:10:17.651415+0000","last_peered":"2024-03-29T10:10:17.651415+0000","last_clean":"2024-03-29T10:10:17.651415+0000","last_became_active":"2024-03-29T10:10:17.645684+0000","last_became_peered":"2024-03-29T10:10:17.645684+0000","last_unstale":"2024-03-29T10:10:17.651415+0000","last_undegraded":"2024-03-29T10:10:17.651415+0000","last_fullsized":"2024-03-29T10:10:17.651415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:56:25.371298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636655+0000","last_change":"2024-03-29T10:10:17.117927+0000","last_active":"2024-03-29T10:10:18.636655+0000","last_peered":"2024-03-29T10:10:18.636655+0000","last_clean":"2024-03-29T10:10:18.636655+0000","last_became_active":"2024-03-29T10:10:17.117598+0000","last_became_peered":"2024-03-29T10:10:17.117598+0000","last_unstale":"2024-03-29T10:10:18.636655+0000","last_undegraded":"2024-03-29T10:10:18.636655+0000","last_fullsized":"2024-03-29T10:10:18.636655+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:13:13.568372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663919+0000","last_change":"2024-03-29T10:10:17.656554+0000","last_active":"2024-03-29T10:10:17.663919+0000","last_peered":"2024-03-29T10:10:17.663919+0000","last_clean":"2024-03-29T10:10:17.663919+0000","last_became_active":"2024-03-29T10:10:17.650748+0000","last_became_peered":"2024-03-29T10:10:17.650748+0000","last_unstale":"2024-03-29T10:10:17.663919+0000","last_undegraded":"2024-03-29T10:10:17.663919+0000","last_fullsized":"2024-03-29T10:10:17.663919+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:25.955220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"93'8","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663418+0000","last_change":"2024-03-29T10:10:17.129015+0000","last_active":"2024-03-29T10:10:17.663418+0000","last_peered":"2024-03-29T10:10:17.663418+0000","last_clean":"2024-03-29T10:10:17.663418+0000","last_became_active":"2024-03-29T10:10:17.128081+0000","last_became_peered":"2024-03-29T10:10:17.128081+0000","last_unstale":"2024-03-29T10:10:17.663418+0000","last_undegraded":"2024-03-29T10:10:17.663418+0000","last_fullsized":"2024-03-29T10:10:17.663418+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:49.244529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.655893+0000","last_change":"2024-03-29T10:10:17.645764+0000","last_active":"2024-03-29T10:10:17.655893+0000","last_peered":"2024-03-29T10:10:17.655893+0000","last_clean":"2024-03-29T10:10:17.655893+0000","last_became_active":"2024-03-29T10:10:17.645449+0000","last_became_peered":"2024-03-29T10:10:17.645449+0000","last_unstale":"2024-03-29T10:10:17.655893+0000","last_undegraded":"2024-03-29T10:10:17.655893+0000","last_fullsized":"2024-03-29T10:10:17.655893+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:47:42.484636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"93'1","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637919+0000","last_change":"2024-03-29T10:10:17.124268+0000","last_active":"2024-03-29T10:10:18.637919+0000","last_peered":"2024-03-29T10:10:18.637919+0000","last_clean":"2024-03-29T10:10:18.637919+0000","last_became_active":"2024-03-29T10:10:17.123698+0000","last_became_peered":"2024-03-29T10:10:17.123698+0000","last_unstale":"2024-03-29T10:10:18.637919+0000","last_undegraded":"2024-03-29T10:10:18.637919+0000","last_fullsized":"2024-03-29T10:10:18.637919+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.082904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663744+0000","last_change":"2024-03-29T10:10:17.659876+0000","last_active":"2024-03-29T10:10:17.663744+0000","last_peered":"2024-03-29T10:10:17.663744+0000","last_clean":"2024-03-29T10:10:17.663744+0000","last_became_active":"2024-03-29T10:10:17.658841+0000","last_became_peered":"2024-03-29T10:10:17.658841+0000","last_unstale":"2024-03-29T10:10:17.663744+0000","last_undegraded":"2024-03-29T10:10:17.663744+0000","last_fullsized":"2024-03-29T10:10:17.663744+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:56:25.371298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"93'2","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664138+0000","last_change":"2024-03-29T10:10:17.126435+0000","last_active":"2024-03-29T10:10:17.664138+0000","last_peered":"2024-03-29T10:10:17.664138+0000","last_clean":"2024-03-29T10:10:17.664138+0000","last_became_active":"2024-03-29T10:10:17.126123+0000","last_became_peered":"2024-03-29T10:10:17.126123+0000","last_unstale":"2024-03-29T10:10:17.664138+0000","last_undegraded":"2024-03-29T10:10:17.664138+0000","last_fullsized":"2024-03-29T10:10:17.664138+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:44.926474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631925+0000","last_change":"2024-03-29T10:10:17.661191+0000","last_active":"2024-03-29T10:10:18.631925+0000","last_peered":"2024-03-29T10:10:18.631925+0000","last_clean":"2024-03-29T10:10:18.631925+0000","last_became_active":"2024-03-29T10:10:17.660557+0000","last_became_peered":"2024-03-29T10:10:17.660557+0000","last_unstale":"2024-03-29T10:10:18.631925+0000","last_undegraded":"2024-03-29T10:10:18.631925+0000","last_fullsized":"2024-03-29T10:10:18.631925+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:34:20.242868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"93'1","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637484+0000","last_change":"2024-03-29T10:10:17.126064+0000","last_active":"2024-03-29T10:10:18.637484+0000","last_peered":"2024-03-29T10:10:18.637484+0000","last_clean":"2024-03-29T10:10:18.637484+0000","last_became_active":"2024-03-29T10:10:17.125734+0000","last_became_peered":"2024-03-29T10:10:17.125734+0000","last_unstale":"2024-03-29T10:10:18.637484+0000","last_undegraded":"2024-03-29T10:10:18.637484+0000","last_fullsized":"2024-03-29T10:10:18.637484+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:01.925007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.660005+0000","last_change":"2024-03-29T10:10:17.655303+0000","last_active":"2024-03-29T10:10:17.660005+0000","last_peered":"2024-03-29T10:10:17.660005+0000","last_clean":"2024-03-29T10:10:17.660005+0000","last_became_active":"2024-03-29T10:10:17.654222+0000","last_became_peered":"2024-03-29T10:10:17.654222+0000","last_unstale":"2024-03-29T10:10:17.660005+0000","last_undegraded":"2024-03-29T10:10:17.660005+0000","last_fullsized":"2024-03-29T10:10:17.660005+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:56:42.193693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.644585+0000","last_change":"2024-03-29T10:10:17.126454+0000","last_active":"2024-03-29T10:10:17.644585+0000","last_peered":"2024-03-29T10:10:17.644585+0000","last_clean":"2024-03-29T10:10:17.644585+0000","last_became_active":"2024-03-29T10:10:17.126095+0000","last_became_peered":"2024-03-29T10:10:17.126095+0000","last_unstale":"2024-03-29T10:10:17.644585+0000","last_undegraded":"2024-03-29T10:10:17.644585+0000","last_fullsized":"2024-03-29T10:10:17.644585+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:56.255133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.646626+0000","last_change":"2024-03-29T10:10:17.640780+0000","last_active":"2024-03-29T10:10:17.646626+0000","last_peered":"2024-03-29T10:10:17.646626+0000","last_clean":"2024-03-29T10:10:17.646626+0000","last_became_active":"2024-03-29T10:10:17.640514+0000","last_became_peered":"2024-03-29T10:10:17.640514+0000","last_unstale":"2024-03-29T10:10:17.646626+0000","last_undegraded":"2024-03-29T10:10:17.646626+0000","last_fullsized":"2024-03-29T10:10:17.646626+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:08.919330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637095+0000","last_change":"2024-03-29T10:10:17.120504+0000","last_active":"2024-03-29T10:10:18.637095+0000","last_peered":"2024-03-29T10:10:18.637095+0000","last_clean":"2024-03-29T10:10:18.637095+0000","last_became_active":"2024-03-29T10:10:17.120202+0000","last_became_peered":"2024-03-29T10:10:17.120202+0000","last_unstale":"2024-03-29T10:10:18.637095+0000","last_undegraded":"2024-03-29T10:10:18.637095+0000","last_fullsized":"2024-03-29T10:10:18.637095+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:26:31.933626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.646903+0000","last_change":"2024-03-29T10:10:17.637795+0000","last_active":"2024-03-29T10:10:17.646903+0000","last_peered":"2024-03-29T10:10:17.646903+0000","last_clean":"2024-03-29T10:10:17.646903+0000","last_became_active":"2024-03-29T10:10:17.637538+0000","last_became_peered":"2024-03-29T10:10:17.637538+0000","last_unstale":"2024-03-29T10:10:17.646903+0000","last_undegraded":"2024-03-29T10:10:17.646903+0000","last_fullsized":"2024-03-29T10:10:17.646903+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:42.679046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637027+0000","last_change":"2024-03-29T10:10:17.125657+0000","last_active":"2024-03-29T10:10:18.637027+0000","last_peered":"2024-03-29T10:10:18.637027+0000","last_clean":"2024-03-29T10:10:18.637027+0000","last_became_active":"2024-03-29T10:10:17.125358+0000","last_became_peered":"2024-03-29T10:10:17.125358+0000","last_unstale":"2024-03-29T10:10:18.637027+0000","last_undegraded":"2024-03-29T10:10:18.637027+0000","last_fullsized":"2024-03-29T10:10:18.637027+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.082904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.646669+0000","last_change":"2024-03-29T10:10:17.637766+0000","last_active":"2024-03-29T10:10:17.646669+0000","last_peered":"2024-03-29T10:10:17.646669+0000","last_clean":"2024-03-29T10:10:17.646669+0000","last_became_active":"2024-03-29T10:10:17.637469+0000","last_became_peered":"2024-03-29T10:10:17.637469+0000","last_unstale":"2024-03-29T10:10:17.646669+0000","last_undegraded":"2024-03-29T10:10:17.646669+0000","last_fullsized":"2024-03-29T10:10:17.646669+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:56.411283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.637172+0000","last_change":"2024-03-29T10:10:17.119727+0000","last_active":"2024-03-29T10:10:17.637172+0000","last_peered":"2024-03-29T10:10:17.637172+0000","last_clean":"2024-03-29T10:10:17.637172+0000","last_became_active":"2024-03-29T10:10:17.119480+0000","last_became_peered":"2024-03-29T10:10:17.119480+0000","last_unstale":"2024-03-29T10:10:17.637172+0000","last_undegraded":"2024-03-29T10:10:17.637172+0000","last_fullsized":"2024-03-29T10:10:17.637172+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:46.599328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664598+0000","last_change":"2024-03-29T10:10:17.658110+0000","last_active":"2024-03-29T10:10:17.664598+0000","last_peered":"2024-03-29T10:10:17.664598+0000","last_clean":"2024-03-29T10:10:17.664598+0000","last_became_active":"2024-03-29T10:10:17.657445+0000","last_became_peered":"2024-03-29T10:10:17.657445+0000","last_unstale":"2024-03-29T10:10:17.664598+0000","last_undegraded":"2024-03-29T10:10:17.664598+0000","last_fullsized":"2024-03-29T10:10:17.664598+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:01:10.259496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"93'2","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637918+0000","last_change":"2024-03-29T10:10:17.124032+0000","last_active":"2024-03-29T10:10:18.637918+0000","last_peered":"2024-03-29T10:10:18.637918+0000","last_clean":"2024-03-29T10:10:18.637918+0000","last_became_active":"2024-03-29T10:10:17.123333+0000","last_became_peered":"2024-03-29T10:10:17.123333+0000","last_unstale":"2024-03-29T10:10:18.637918+0000","last_undegraded":"2024-03-29T10:10:18.637918+0000","last_fullsized":"2024-03-29T10:10:18.637918+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:44.926474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636912+0000","last_change":"2024-03-29T10:10:17.644480+0000","last_active":"2024-03-29T10:10:18.636912+0000","last_peered":"2024-03-29T10:10:18.636912+0000","last_clean":"2024-03-29T10:10:18.636912+0000","last_became_active":"2024-03-29T10:10:17.644181+0000","last_became_peered":"2024-03-29T10:10:17.644181+0000","last_unstale":"2024-03-29T10:10:18.636912+0000","last_undegraded":"2024-03-29T10:10:18.636912+0000","last_fullsized":"2024-03-29T10:10:18.636912+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:57.257695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637545+0000","last_change":"2024-03-29T10:10:17.116018+0000","last_active":"2024-03-29T10:10:18.637545+0000","last_peered":"2024-03-29T10:10:18.637545+0000","last_clean":"2024-03-29T10:10:18.637545+0000","last_became_active":"2024-03-29T10:10:17.115598+0000","last_became_peered":"2024-03-29T10:10:17.115598+0000","last_unstale":"2024-03-29T10:10:18.637545+0000","last_undegraded":"2024-03-29T10:10:18.637545+0000","last_fullsized":"2024-03-29T10:10:18.637545+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:49.244529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.667077+0000","last_change":"2024-03-29T10:10:17.664430+0000","last_active":"2024-03-29T10:10:17.667077+0000","last_peered":"2024-03-29T10:10:17.667077+0000","last_clean":"2024-03-29T10:10:17.667077+0000","last_became_active":"2024-03-29T10:10:17.663576+0000","last_became_peered":"2024-03-29T10:10:17.663576+0000","last_unstale":"2024-03-29T10:10:17.667077+0000","last_undegraded":"2024-03-29T10:10:17.667077+0000","last_fullsized":"2024-03-29T10:10:17.667077+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:40:01.285102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.649991+0000","last_change":"2024-03-29T10:10:17.116045+0000","last_active":"2024-03-29T10:10:17.649991+0000","last_peered":"2024-03-29T10:10:17.649991+0000","last_clean":"2024-03-29T10:10:17.649991+0000","last_became_active":"2024-03-29T10:10:17.115770+0000","last_became_peered":"2024-03-29T10:10:17.115770+0000","last_unstale":"2024-03-29T10:10:17.649991+0000","last_undegraded":"2024-03-29T10:10:17.649991+0000","last_fullsized":"2024-03-29T10:10:17.649991+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:05.665071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.663899+0000","last_change":"2024-03-29T10:10:17.658780+0000","last_active":"2024-03-29T10:10:17.663899+0000","last_peered":"2024-03-29T10:10:17.663899+0000","last_clean":"2024-03-29T10:10:17.663899+0000","last_became_active":"2024-03-29T10:10:17.657250+0000","last_became_peered":"2024-03-29T10:10:17.657250+0000","last_unstale":"2024-03-29T10:10:17.663899+0000","last_undegraded":"2024-03-29T10:10:17.663899+0000","last_fullsized":"2024-03-29T10:10:17.663899+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:42:28.134507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.640735+0000","last_change":"2024-03-29T10:10:17.117782+0000","last_active":"2024-03-29T10:10:17.640735+0000","last_peered":"2024-03-29T10:10:17.640735+0000","last_clean":"2024-03-29T10:10:17.640735+0000","last_became_active":"2024-03-29T10:10:17.117506+0000","last_became_peered":"2024-03-29T10:10:17.117506+0000","last_unstale":"2024-03-29T10:10:17.640735+0000","last_undegraded":"2024-03-29T10:10:17.640735+0000","last_fullsized":"2024-03-29T10:10:17.640735+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:03.585887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.664460+0000","last_change":"2024-03-29T10:10:17.659435+0000","last_active":"2024-03-29T10:10:17.664460+0000","last_peered":"2024-03-29T10:10:17.664460+0000","last_clean":"2024-03-29T10:10:17.664460+0000","last_became_active":"2024-03-29T10:10:17.650714+0000","last_became_peered":"2024-03-29T10:10:17.650714+0000","last_unstale":"2024-03-29T10:10:17.664460+0000","last_undegraded":"2024-03-29T10:10:17.664460+0000","last_fullsized":"2024-03-29T10:10:17.664460+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:02.927569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.649946+0000","last_change":"2024-03-29T10:10:17.122709+0000","last_active":"2024-03-29T10:10:17.649946+0000","last_peered":"2024-03-29T10:10:17.649946+0000","last_clean":"2024-03-29T10:10:17.649946+0000","last_became_active":"2024-03-29T10:10:17.122289+0000","last_became_peered":"2024-03-29T10:10:17.122289+0000","last_unstale":"2024-03-29T10:10:17.649946+0000","last_undegraded":"2024-03-29T10:10:17.649946+0000","last_fullsized":"2024-03-29T10:10:17.649946+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:50:42.044699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636845+0000","last_change":"2024-03-29T10:10:17.646330+0000","last_active":"2024-03-29T10:10:18.636845+0000","last_peered":"2024-03-29T10:10:18.636845+0000","last_clean":"2024-03-29T10:10:18.636845+0000","last_became_active":"2024-03-29T10:10:17.645988+0000","last_became_peered":"2024-03-29T10:10:17.645988+0000","last_unstale":"2024-03-29T10:10:18.636845+0000","last_undegraded":"2024-03-29T10:10:18.636845+0000","last_fullsized":"2024-03-29T10:10:18.636845+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:08.435381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631552+0000","last_change":"2024-03-29T10:10:17.117333+0000","last_active":"2024-03-29T10:10:18.631552+0000","last_peered":"2024-03-29T10:10:18.631552+0000","last_clean":"2024-03-29T10:10:18.631552+0000","last_became_active":"2024-03-29T10:10:17.117063+0000","last_became_peered":"2024-03-29T10:10:17.117063+0000","last_unstale":"2024-03-29T10:10:18.631552+0000","last_undegraded":"2024-03-29T10:10:18.631552+0000","last_fullsized":"2024-03-29T10:10:18.631552+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:15:56.412964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.631444+0000","last_change":"2024-03-29T10:10:17.648826+0000","last_active":"2024-03-29T10:10:18.631444+0000","last_peered":"2024-03-29T10:10:18.631444+0000","last_clean":"2024-03-29T10:10:18.631444+0000","last_became_active":"2024-03-29T10:10:17.648520+0000","last_became_peered":"2024-03-29T10:10:17.648520+0000","last_unstale":"2024-03-29T10:10:18.631444+0000","last_undegraded":"2024-03-29T10:10:18.631444+0000","last_fullsized":"2024-03-29T10:10:18.631444+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:07.440690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.637131+0000","last_change":"2024-03-29T10:10:17.128866+0000","last_active":"2024-03-29T10:10:17.637131+0000","last_peered":"2024-03-29T10:10:17.637131+0000","last_clean":"2024-03-29T10:10:17.637131+0000","last_became_active":"2024-03-29T10:10:17.128576+0000","last_became_peered":"2024-03-29T10:10:17.128576+0000","last_unstale":"2024-03-29T10:10:17.637131+0000","last_undegraded":"2024-03-29T10:10:17.637131+0000","last_fullsized":"2024-03-29T10:10:17.637131+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:01.925007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637048+0000","last_change":"2024-03-29T10:10:17.650715+0000","last_active":"2024-03-29T10:10:18.637048+0000","last_peered":"2024-03-29T10:10:18.637048+0000","last_clean":"2024-03-29T10:10:18.637048+0000","last_became_active":"2024-03-29T10:10:17.649935+0000","last_became_peered":"2024-03-29T10:10:17.649935+0000","last_unstale":"2024-03-29T10:10:18.637048+0000","last_undegraded":"2024-03-29T10:10:18.637048+0000","last_fullsized":"2024-03-29T10:10:18.637048+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:13.147091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.637293+0000","last_change":"2024-03-29T10:10:17.122272+0000","last_active":"2024-03-29T10:10:18.637293+0000","last_peered":"2024-03-29T10:10:18.637293+0000","last_clean":"2024-03-29T10:10:18.637293+0000","last_became_active":"2024-03-29T10:10:17.121968+0000","last_became_peered":"2024-03-29T10:10:17.121968+0000","last_unstale":"2024-03-29T10:10:18.637293+0000","last_undegraded":"2024-03-29T10:10:18.637293+0000","last_fullsized":"2024-03-29T10:10:18.637293+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:22:29.410605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.644945+0000","last_change":"2024-03-29T10:10:17.637281+0000","last_active":"2024-03-29T10:10:17.644945+0000","last_peered":"2024-03-29T10:10:17.644945+0000","last_clean":"2024-03-29T10:10:17.644945+0000","last_became_active":"2024-03-29T10:10:17.637009+0000","last_became_peered":"2024-03-29T10:10:17.637009+0000","last_unstale":"2024-03-29T10:10:17.644945+0000","last_undegraded":"2024-03-29T10:10:17.644945+0000","last_fullsized":"2024-03-29T10:10:17.644945+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:17.096496+0000","last_clean_scrub_stamp":"2024-03-29T10:10:17.096496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:34.531392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"93'2","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-29T10:10:18.636958+0000","last_change":"2024-03-29T10:10:17.121650+0000","last_active":"2024-03-29T10:10:18.636958+0000","last_peered":"2024-03-29T10:10:18.636958+0000","last_clean":"2024-03-29T10:10:18.636958+0000","last_became_active":"2024-03-29T10:10:17.121308+0000","last_became_peered":"2024-03-29T10:10:17.121308+0000","last_unstale":"2024-03-29T10:10:18.636958+0000","last_undegraded":"2024-03-29T10:10:18.636958+0000","last_fullsized":"2024-03-29T10:10:18.636958+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:16.093934+0000","last_clean_scrub_stamp":"2024-03-29T10:10:16.093934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:24.952658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"83'183","reported_seq":468,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-29T10:10:17.114449+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:10:17.114449+0000","last_peered":"2024-03-29T10:10:17.114449+0000","last_clean":"2024-03-29T10:10:17.114449+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:10:17.114449+0000","last_undegraded":"2024-03-29T10:10:17.114449+0000","last_fullsized":"2024-03-29T10:10:17.114449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575037,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32820,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339020,"statfs":{"total":96636764160,"available":96603156480,"internally_reserved":0,"allocated":1736704,"data_stored":1128337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.69799999999999995,"15min":0.71699999999999997},"min":{"1min":0.44600000000000001,"5min":0.44500000000000001,"15min":0.32500000000000001},"max":{"1min":0.91600000000000004,"5min":0.996,"15min":1.0109999999999999},"last":0.629},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68400000000000005,"15min":0.70999999999999996},"min":{"1min":0.50800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.81299999999999994,"5min":0.93400000000000005,"15min":1.046},"last":0.75600000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71099999999999997,"15min":0.72299999999999998},"min":{"1min":0.51800000000000002,"5min":0.47399999999999998,"15min":0.44500000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.1910000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71899999999999997,"15min":0.73599999999999999},"min":{"1min":0.52500000000000002,"5min":0.42199999999999999,"15min":0.372},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.57499999999999996}]},{"osd":2,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.72799999999999998,"15min":0.74299999999999999},"min":{"1min":0.47799999999999998,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.046},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.73799999999999999,"15min":0.752},"min":{"1min":0.54900000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.079},"last":0.70399999999999996}]},{"osd":3,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.70799999999999996,"15min":0.73599999999999999},"min":{"1min":0.51700000000000002,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.80700000000000005,"5min":0.97699999999999998,"15min":1.081},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.76400000000000001,"15min":0.78200000000000003},"min":{"1min":0.54500000000000004,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.71899999999999997}]},{"osd":4,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.29199999999999998},"max":{"1min":0.80100000000000005,"5min":0.86399999999999999,"15min":1.111},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63400000000000001,"15min":0.63700000000000001},"min":{"1min":0.39200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.85199999999999998,"5min":0.95199999999999996,"15min":0.95599999999999996},"last":0.84399999999999997}]},{"osd":5,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.46300000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95699999999999996,"5min":0.96099999999999997,"15min":1.0089999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.68500000000000005},"min":{"1min":0.49199999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82699999999999996,"5min":0.92800000000000005,"15min":1.141},"last":0.81499999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72499999999999998,"15min":0.72999999999999998},"min":{"1min":0.59599999999999997,"5min":0.50600000000000001,"15min":0.49199999999999999},"max":{"1min":0.92400000000000004,"5min":0.99099999999999999,"15min":1.1679999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70299999999999996,"15min":0.70099999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.435},"max":{"1min":0.94299999999999995,"5min":0.98399999999999999,"15min":1.121},"last":0.80600000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1142784,"data_stored":537445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.72199999999999998,"15min":0.73099999999999998},"min":{"1min":0.46200000000000002,"5min":0.42799999999999999,"15min":0.34699999999999998},"max":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":7.4080000000000004},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71399999999999997,"15min":0.75},"min":{"1min":0.47599999999999998,"5min":0.435,"15min":0.42499999999999999},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":6.7610000000000001},"last":0.70699999999999996}]},{"osd":1,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.46100000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.624},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73599999999999999,"15min":0.72699999999999998},"min":{"1min":0.45200000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.6709999999999998,"5min":2.6709999999999998,"15min":2.6709999999999998},"last":0.495}]},{"osd":2,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.3169999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.75},"min":{"1min":0.54900000000000004,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.64900000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.72599999999999998,"15min":0.75800000000000001},"min":{"1min":0.54400000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.131,"5min":1.131,"15min":3.1960000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.78000000000000003,"15min":0.78700000000000003},"min":{"1min":0.44500000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.718,"5min":2.718,"15min":4.3760000000000003},"last":0.78200000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.62,"15min":0.61399999999999999},"min":{"1min":0.437,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.48999999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.544,"5min":2.544,"15min":3.1219999999999999},"last":0.89100000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.68700000000000006,"15min":0.67100000000000004},"min":{"1min":0.36499999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.2280000000000002,"5min":3.2280000000000002,"15min":3.2280000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.72299999999999998,"15min":0.69399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2.835,"5min":2.835,"15min":2.835},"last":0.73299999999999998}]},{"osd":7,"last update":"Fri Mar 29 10:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.54200000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.74199999999999999,"15min":0.73099999999999998},"min":{"1min":0.503,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.7170000000000001,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.752}]}]},{"osd":1,"up_from":13,"seq":55834575036,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50292,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321548,"statfs":{"total":96636764160,"available":96585265152,"internally_reserved":0,"allocated":1155072,"data_stored":539533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.315,"5min":0.219,"15min":0.219},"max":{"1min":0.65300000000000002,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.68999999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71599999999999997,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.52900000000000003},"min":{"1min":0.36699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":4.282},"last":0.39500000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.70999999999999996,"5min":0.78500000000000003,"15min":4.3200000000000003},"last":0.501},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.48799999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.016,"5min":1.617,"15min":1.617},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76400000000000001,"15min":0.78800000000000003},"min":{"1min":0.60899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0289999999999999,"5min":1.1419999999999999,"15min":4.3559999999999999},"last":0.94899999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.51800000000000002},"max":{"1min":1.056,"5min":1.3939999999999999,"15min":1.9119999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78600000000000003,"15min":0.80800000000000005},"min":{"1min":0.56299999999999994,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.048,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.91600000000000004}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77800000000000002,"15min":0.81000000000000005},"min":{"1min":0.57499999999999996,"5min":0.52600000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":1.194,"15min":4.3899999999999997},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.093,"5min":1.468,"15min":1.468},"last":0.82599999999999996}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.62,"5min":0.58699999999999997,"15min":0.56200000000000006},"max":{"1min":1.127,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.83599999999999997},"min":{"1min":0.60999999999999999,"5min":0.57099999999999995,"15min":0.55700000000000005},"max":{"1min":1.0269999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.871}]}]},{"osd":0,"up_from":13,"seq":55834575038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1724416,"data_stored":1126702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.44900000000000001,"15min":0.45300000000000001},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.23999999999999999},"max":{"1min":0.628,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.48099999999999998,"15min":0.50700000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.628,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.497}]},{"osd":2,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.35499999999999998},"max":{"1min":0.70299999999999996,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.46700000000000003,"15min":0.49399999999999999},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":0.68500000000000005,"5min":0.83899999999999997,"15min":0.97199999999999998},"last":0.36399999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70199999999999996,"5min":0.80500000000000005,"15min":1.077},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58699999999999997,"15min":0.60799999999999998},"min":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.71099999999999997,"5min":0.876,"15min":1.0549999999999999},"last":0.623}]},{"osd":4,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.78600000000000003,"15min":0.79500000000000004},"min":{"1min":0.499,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":0.88200000000000001,"5min":1.625,"15min":1.625},"last":0.877},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.78900000000000003,"15min":0.81799999999999995},"min":{"1min":0.5,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":0.97399999999999998,"5min":1.1419999999999999,"15min":1.167},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.72799999999999998,"15min":0.752},"min":{"1min":0.438,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.79200000000000004,"5min":0.99399999999999999,"15min":1.137},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.79800000000000004},"min":{"1min":0.54000000000000004,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":0.90900000000000003,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.94299999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.78800000000000003,"15min":0.81399999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53400000000000003},"max":{"1min":0.92200000000000004,"5min":1.0640000000000001,"15min":1.228},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.749,"5min":0.82299999999999995,"15min":0.83799999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94199999999999995,"5min":1.601,"15min":1.601},"last":0.79100000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76100000000000001,"15min":0.79200000000000004},"min":{"1min":0.48599999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90300000000000002,"5min":1.03,"15min":1.1890000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.81799999999999995,"15min":0.83599999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.96399999999999997,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.91900000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40432,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331408,"statfs":{"total":96636764160,"available":96595361792,"internally_reserved":0,"allocated":1146880,"data_stored":538917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.28999999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.746,"5min":0.746,"15min":0.86599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.505,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.377,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.67900000000000005,"5min":0.78800000000000003,"15min":1.3140000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.48399999999999999,"15min":0.47499999999999998},"min":{"1min":0.35699999999999998,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.94499999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":1.032,"15min":1.032},"last":0.61099999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.46300000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.74099999999999999,"5min":0.80000000000000004,"15min":1.127},"last":0.623},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.36699999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.64800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.76800000000000002,"15min":0.78300000000000003},"min":{"1min":0.54300000000000004,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":1.0209999999999999,"5min":1.1799999999999999,"15min":1.554},"last":0.871},{"interface":"front","average":{"1min":0.755,"5min":0.77700000000000002,"15min":0.80500000000000005},"min":{"1min":0.59399999999999997,"5min":0.55000000000000004,"15min":0.51600000000000001},"max":{"1min":0.95099999999999996,"5min":1.236,"15min":1.577},"last":1.0029999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78600000000000003,"15min":0.80700000000000005},"min":{"1min":0.55200000000000005,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96699999999999997,"5min":1.25,"15min":1.5209999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.77500000000000002,"15min":0.80300000000000005},"min":{"1min":0.55000000000000004,"5min":0.53200000000000003,"15min":0.51800000000000002},"max":{"1min":0.98399999999999999,"5min":1.2250000000000001,"15min":1.3300000000000001},"last":1.115}]},{"osd":6,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.79400000000000004,"15min":0.82099999999999995},"min":{"1min":0.58399999999999996,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1220000000000001,"5min":1.1539999999999999,"15min":1.758},"last":1.107},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81999999999999995,"15min":0.83899999999999997},"min":{"1min":0.63900000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0940000000000001,"5min":1.3560000000000001,"15min":1.4750000000000001},"last":1.085}]},{"osd":7,"last update":"Fri Mar 29 10:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.84999999999999998,"15min":0.86499999999999999},"min":{"1min":0.58799999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.071,"5min":1.2709999999999999,"15min":1.4490000000000001},"last":1.1339999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.88},"min":{"1min":0.68200000000000005,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.056,"5min":1.3049999999999999,"15min":1.4390000000000001},"last":1.056}]}]},{"osd":3,"up_from":13,"seq":55834575037,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36332,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335508,"statfs":{"total":96636764160,"available":96599560192,"internally_reserved":0,"allocated":1142784,"data_stored":537465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.45600000000000002,"15min":0.46200000000000002},"min":{"1min":0.309,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.75600000000000001,"5min":0.79500000000000004,"15min":4.3799999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73699999999999999,"5min":0.86899999999999999,"15min":4.6379999999999999},"last":0.41299999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.34200000000000003,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.59599999999999997,"5min":0.98699999999999999,"15min":4.1840000000000002},"last":0.441},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.314},"max":{"1min":0.71299999999999997,"5min":0.96099999999999997,"15min":1.048},"last":0.42899999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.33700000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":1.369},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.63600000000000001},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.86199999999999999,"5min":0.94799999999999995,"15min":4.5970000000000004},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.755,"15min":0.77200000000000002},"min":{"1min":0.497,"5min":0.439,"15min":0.439},"max":{"1min":0.98599999999999999,"5min":1.288,"15min":4.3319999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.755,"15min":0.77300000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.39400000000000002},"max":{"1min":1.0900000000000001,"5min":1.46,"15min":1.46},"last":0.78100000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.435},"max":{"1min":0.96199999999999997,"5min":1.3580000000000001,"15min":1.3740000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.78000000000000003,"15min":0.80100000000000005},"min":{"1min":0.57899999999999996,"5min":0.46899999999999997,"15min":0.44500000000000001},"max":{"1min":0.92700000000000005,"5min":1.397,"15min":4.4180000000000001},"last":0.96099999999999997}]},{"osd":6,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.79400000000000004,"15min":0.80800000000000005},"min":{"1min":0.54500000000000004,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.0189999999999999,"5min":1.2549999999999999,"15min":1.4730000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.80500000000000005,"15min":0.81499999999999995},"min":{"1min":0.59199999999999997,"5min":0.52400000000000002,"15min":0.503},"max":{"1min":1.0349999999999999,"5min":1.23,"15min":1.417},"last":0.93999999999999995}]},{"osd":7,"last update":"Fri Mar 29 10:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.82799999999999996,"15min":0.83899999999999997},"min":{"1min":0.60699999999999998,"5min":0.57299999999999995,"15min":0.54300000000000004},"max":{"1min":1.0049999999999999,"5min":1.5,"15min":4.4550000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.82199999999999995,"15min":0.83599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.56200000000000006},"max":{"1min":1.0489999999999999,"5min":1.345,"15min":1.476},"last":0.79900000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575038,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54376,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317464,"statfs":{"total":96636764160,"available":96581083136,"internally_reserved":0,"allocated":1138688,"data_stored":537423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.68700000000000006},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.88200000000000001,"5min":0.93200000000000005,"15min":1.417},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.92100000000000004,"5min":1.165,"15min":1.165},"last":0.52800000000000002}]},{"osd":1,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.93500000000000005,"5min":0.93700000000000006,"15min":1.139},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.333},"last":0.498}]},{"osd":2,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.01,"15min":1.863},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.75,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93700000000000006,"5min":0.96299999999999997,"15min":1.387},"last":0.67200000000000004}]},{"osd":3,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.5,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.88700000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.51600000000000001,"5min":0.48699999999999999,"15min":0.46600000000000003},"max":{"1min":0.80500000000000005,"5min":1.1539999999999999,"15min":1.905},"last":0.68400000000000005}]},{"osd":5,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.46700000000000003,"5min":0.39300000000000002,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.48099999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.78500000000000003,"5min":1.294,"15min":1.294},"last":0.81599999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.48099999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.75900000000000001,"5min":1.0129999999999999,"15min":1.9339999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.66100000000000003,"15min":0.66200000000000003},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.433},"max":{"1min":0.85299999999999998,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.78200000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.48099999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85999999999999999,"5min":1.345,"15min":1.345},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.59099999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89700000000000002,"5min":1.214,"15min":1.97},"last":0.88800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575037,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1138688,"data_stored":537443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.68300000000000005,"15min":0.68400000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.313},"max":{"1min":0.84099999999999997,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.73399999999999999,"15min":0.72399999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.33500000000000002},"max":{"1min":0.90400000000000003,"5min":1.21,"15min":1.339},"last":0.79400000000000004}]},{"osd":1,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71399999999999997,"15min":0.71899999999999997},"min":{"1min":0.53100000000000003,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.96499999999999997,"5min":0.96699999999999997,"15min":1.0169999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.70999999999999996},"min":{"1min":0.50700000000000001,"5min":0.46700000000000003,"15min":0.39200000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.6040000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.73199999999999998,"15min":0.73299999999999998},"min":{"1min":0.51700000000000002,"5min":0.44800000000000001,"15min":0.435},"max":{"1min":0.95799999999999996,"5min":0.98899999999999999,"15min":1.476},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70999999999999996,"15min":0.71199999999999997},"min":{"1min":0.46200000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83899999999999997,"5min":1.095,"15min":1.502},"last":0.68999999999999995}]},{"osd":3,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.72499999999999998,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":1.042,"5min":1.113,"15min":1.1619999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.74199999999999999,"15min":0.754},"min":{"1min":0.60999999999999999,"5min":0.52300000000000002,"15min":0.46500000000000002},"max":{"1min":0.84499999999999997,"5min":1.0249999999999999,"15min":1.744},"last":0.65300000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.59999999999999998},"min":{"1min":0.42699999999999999,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.84899999999999998,"5min":0.94599999999999995,"15min":0.95899999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65100000000000002,"15min":0.64100000000000001},"min":{"1min":0.47399999999999998,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":1.097,"5min":1.097,"15min":1.214},"last":0.61099999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.504,"5min":0.44600000000000001,"15min":0.36799999999999999},"max":{"1min":0.874,"5min":0.91300000000000003,"15min":1.151},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.67700000000000005,"15min":0.67900000000000005},"min":{"1min":0.45900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95899999999999996,"5min":0.97099999999999997,"15min":1.5489999999999999},"last":0.72399999999999998}]},{"osd":7,"last update":"Fri Mar 29 10:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.69999999999999996,"15min":0.70599999999999996},"min":{"1min":0.56399999999999995,"5min":0.41899999999999998,"15min":0.40200000000000002},"max":{"1min":0.90200000000000002,"5min":0.95299999999999996,"15min":1.325},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.73399999999999999,"15min":0.73499999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91800000000000004,"5min":0.98199999999999998,"15min":1.1990000000000001},"last":0.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:10:23.064 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T10:10:23.064 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T10:10:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.305+0000 7f0baa000700 1 Processor -- start 2024-03-29T10:10:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.306+0000 7f0baa000700 1 -- start start 2024-03-29T10:10:23.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0baa000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ba41611b0 0x7f0ba415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0baa000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba415f4d0 0x7f0ba40ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0baa000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba40ce270 con 0x7f0ba40d21f0 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0baa000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba415b600 con 0x7f0ba41611b0 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0baa000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40d2760 con 0x7f0ba415f4d0 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0ba3e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0ba40d21f0 0x7f0ba4161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44212/0 (socket says 172.21.15.12:44212) 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0ba3e00700 1 -- 172.21.15.12:0/230253874 learned_addr learned my addr 172.21.15.12:0/230253874 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0ba3400700 1 --2- 172.21.15.12:0/230253874 >> v2:172.21.15.12:3300/0 conn(0x7f0ba41611b0 0x7f0ba415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0ba2a00700 1 --2- 172.21.15.12:0/230253874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba415f4d0 0x7f0ba40ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0ba2000700 1 -- 172.21.15.12:0/230253874 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354071789 0 0) 0x7f0ba40ce270 con 0x7f0ba40d21f0 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0ba2000700 1 -- 172.21.15.12:0/230253874 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b880035b0 con 0x7f0ba40d21f0 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0ba3400700 1 -- 172.21.15.12:0/230253874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba415f4d0 msgr2=0x7f0ba40ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.307+0000 7f0ba3400700 1 --2- 172.21.15.12:0/230253874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba415f4d0 0x7f0ba40ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.308+0000 7f0ba3400700 1 -- 172.21.15.12:0/230253874 >> v1:172.21.15.12:6789/0 conn(0x7f0ba40d21f0 legacy=0x7f0ba4161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.308+0000 7f0ba3400700 1 -- 172.21.15.12:0/230253874 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba40d2a40 con 0x7f0ba41611b0 2024-03-29T10:10:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.308+0000 7f0ba3400700 1 --2- 172.21.15.12:0/230253874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 0x7f0ba415ef90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009840 tx=0x7f0b94009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce009ceddc811b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.308+0000 7f0ba2000700 1 -- 172.21.15.12:0/230253874 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94011830 con 0x7f0ba41611b0 2024-03-29T10:10:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.308+0000 7f0ba2000700 1 -- 172.21.15.12:0/230253874 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b94011990 con 0x7f0ba41611b0 2024-03-29T10:10:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.308+0000 7f0ba2000700 1 -- 172.21.15.12:0/230253874 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94011d20 con 0x7f0ba41611b0 2024-03-29T10:10:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.309+0000 7f0baa000700 1 -- 172.21.15.12:0/230253874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 msgr2=0x7f0ba415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.309+0000 7f0baa000700 1 --2- 172.21.15.12:0/230253874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 0x7f0ba415ef90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009840 tx=0x7f0b94009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.309+0000 7f0baa000700 1 -- 172.21.15.12:0/230253874 shutdown_connections 2024-03-29T10:10:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.309+0000 7f0baa000700 1 --2- 172.21.15.12:0/230253874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 0x7f0ba415ef90 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.309+0000 7f0baa000700 1 --2- 172.21.15.12:0/230253874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba415f4d0 0x7f0ba40ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.309+0000 7f0baa000700 1 -- 172.21.15.12:0/230253874 >> 172.21.15.12:0/230253874 conn(0x7f0ba408aff0 msgr2=0x7f0ba40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.309+0000 7f0baa000700 1 -- 172.21.15.12:0/230253874 shutdown_connections 2024-03-29T10:10:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.309+0000 7f0baa000700 1 -- 172.21.15.12:0/230253874 wait complete. 2024-03-29T10:10:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.310+0000 7f0baa000700 1 Processor -- start 2024-03-29T10:10:23.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.311+0000 7f0baa000700 1 -- start start 2024-03-29T10:10:23.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.311+0000 7f0baa000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba40d21f0 0x7f0ba41568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.311+0000 7f0baa000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ba415f4d0 0x7f0ba40ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0baa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 0x7f0ba415c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0baa000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40cb660 con 0x7f0ba41611b0 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0baa000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40cb0a0 con 0x7f0ba415f4d0 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0baa000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40cb380 con 0x7f0ba40d21f0 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 0x7f0ba415c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 0x7f0ba415c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37992/0 (socket says 172.21.15.12:37992) 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba3e00700 1 -- 172.21.15.12:0/2446443313 learned_addr learned my addr 172.21.15.12:0/2446443313 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba3400700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba40d21f0 0x7f0ba41568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba2a00700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ba415f4d0 0x7f0ba40ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba3e00700 1 -- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba40d21f0 msgr2=0x7f0ba41568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba3e00700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba40d21f0 0x7f0ba41568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba3e00700 1 -- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ba415f4d0 msgr2=0x7f0ba40ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba3e00700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ba415f4d0 0x7f0ba40ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba3e00700 1 -- 172.21.15.12:0/2446443313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b940094a0 con 0x7f0ba41611b0 2024-03-29T10:10:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.312+0000 7f0ba3e00700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 0x7f0ba415c960 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00cb40 tx=0x7f0b9c00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.313+0000 7f0b9b400700 1 -- 172.21.15.12:0/2446443313 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c00ab50 con 0x7f0ba41611b0 2024-03-29T10:10:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.313+0000 7f0b9b400700 1 -- 172.21.15.12:0/2446443313 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9c00acb0 con 0x7f0ba41611b0 2024-03-29T10:10:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.313+0000 7f0baa000700 1 -- 172.21.15.12:0/2446443313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba40cb960 con 0x7f0ba41611b0 2024-03-29T10:10:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.313+0000 7f0b9b400700 1 -- 172.21.15.12:0/2446443313 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c012620 con 0x7f0ba41611b0 2024-03-29T10:10:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.313+0000 7f0baa000700 1 -- 172.21.15.12:0/2446443313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba4156df0 con 0x7f0ba41611b0 2024-03-29T10:10:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.314+0000 7f0baa000700 1 -- 172.21.15.12:0/2446443313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba40cbe80 con 0x7f0ba41611b0 2024-03-29T10:10:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.315+0000 7f0b9b400700 1 -- 172.21.15.12:0/2446443313 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0b9c01b3b0 con 0x7f0ba41611b0 2024-03-29T10:10:23.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.316+0000 7f0b9b400700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406adb0 0x7f0b8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.316+0000 7f0b9b400700 1 -- 172.21.15.12:0/2446443313 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b9c0892b0 con 0x7f0ba41611b0 2024-03-29T10:10:23.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.317+0000 7f0ba3400700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406adb0 0x7f0b8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.317+0000 7f0ba3400700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406adb0 0x7f0b8406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0b94000c00 tx=0x7f0b9401c040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:23.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.321+0000 7f0b9b400700 1 -- 172.21.15.12:0/2446443313 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ba40cbe80 con 0x7f0ba41611b0 2024-03-29T10:10:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.621+0000 7f0baa000700 1 -- 172.21.15.12:0/2446443313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0ba40cbe80 con 0x7f0ba41611b0 2024-03-29T10:10:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.628+0000 7f0b9b400700 1 -- 172.21.15.12:0/2446443313 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0ba40cbe80 con 0x7f0ba41611b0 2024-03-29T10:10:23.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 -- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406adb0 msgr2=0x7f0b8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406adb0 0x7f0b8406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0b94000c00 tx=0x7f0b9401c040 comp rx=0 tx=0).stop 2024-03-29T10:10:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 -- 172.21.15.12:0/2446443313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 msgr2=0x7f0ba415c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 0x7f0ba415c960 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00cb40 tx=0x7f0b9c00cf00 comp rx=0 tx=0).stop 2024-03-29T10:10:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 -- 172.21.15.12:0/2446443313 shutdown_connections 2024-03-29T10:10:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b8406adb0 0x7f0b8406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ba41611b0 0x7f0ba415c960 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ba40d21f0 0x7f0ba41568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 --2- 172.21.15.12:0/2446443313 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ba415f4d0 0x7f0ba40ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 -- 172.21.15.12:0/2446443313 >> 172.21.15.12:0/2446443313 conn(0x7f0ba408aff0 msgr2=0x7f0ba406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 -- 172.21.15.12:0/2446443313 shutdown_connections 2024-03-29T10:10:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.630+0000 7f0baa000700 1 -- 172.21.15.12:0/2446443313 wait complete. 2024-03-29T10:10:23.633 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T10:10:23.650 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.849+0000 7f3595400700 1 Processor -- start 2024-03-29T10:10:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.850+0000 7f3595400700 1 -- start start 2024-03-29T10:10:23.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f3595400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35900d18b0 0x7f35900cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f3595400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900ceda0 con 0x7f3590161590 2024-03-29T10:10:23.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f3595400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35901612f0 con 0x7f3590160f10 2024-03-29T10:10:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f3595400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35900d1d30 con 0x7f35900d18b0 2024-03-29T10:10:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f358de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35900d18b0 0x7f35900cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f358f200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3590161590 0x7f35900b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44226/0 (socket says 172.21.15.12:44226) 2024-03-29T10:10:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f358f200700 1 -- 172.21.15.12:0/4231097105 learned_addr learned my addr 172.21.15.12:0/4231097105 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f358de00700 1 -- 172.21.15.12:0/4231097105 >> v1:172.21.15.159:6789/0 conn(0x7f3590160f10 legacy=0x7f35900d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f358de00700 1 -- 172.21.15.12:0/4231097105 >> v1:172.21.15.12:6789/0 conn(0x7f3590161590 legacy=0x7f35900b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f358de00700 1 -- 172.21.15.12:0/4231097105 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900ce470 con 0x7f35900d18b0 2024-03-29T10:10:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.851+0000 7f358d400700 1 -- 172.21.15.12:0/4231097105 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696573299 0 0) 0x7f35900ceda0 con 0x7f3590161590 2024-03-29T10:10:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.852+0000 7f358de00700 1 --2- 172.21.15.12:0/4231097105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35900d18b0 0x7f35900cf230 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f357c009840 tx=0x7f357c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f08df0e82b64dc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.852+0000 7f358d400700 1 -- 172.21.15.12:0/4231097105 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c011830 con 0x7f35900d18b0 2024-03-29T10:10:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.852+0000 7f358d400700 1 -- 172.21.15.12:0/4231097105 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c011990 con 0x7f35900d18b0 2024-03-29T10:10:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.852+0000 7f358d400700 1 -- 172.21.15.12:0/4231097105 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c011d20 con 0x7f35900d18b0 2024-03-29T10:10:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.852+0000 7f3595400700 1 -- 172.21.15.12:0/4231097105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35900d18b0 msgr2=0x7f35900cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.853+0000 7f3595400700 1 --2- 172.21.15.12:0/4231097105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35900d18b0 0x7f35900cf230 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f357c009840 tx=0x7f357c009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.853+0000 7f3595400700 1 -- 172.21.15.12:0/4231097105 shutdown_connections 2024-03-29T10:10:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.853+0000 7f3595400700 1 --2- 172.21.15.12:0/4231097105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35900d18b0 0x7f35900cf230 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.853+0000 7f3595400700 1 -- 172.21.15.12:0/4231097105 >> 172.21.15.12:0/4231097105 conn(0x7f359008aff0 msgr2=0x7f35900d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.853+0000 7f3595400700 1 -- 172.21.15.12:0/4231097105 shutdown_connections 2024-03-29T10:10:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.853+0000 7f3595400700 1 -- 172.21.15.12:0/4231097105 wait complete. 2024-03-29T10:10:23.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.855+0000 7f3595400700 1 Processor -- start 2024-03-29T10:10:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.855+0000 7f3595400700 1 -- start start 2024-03-29T10:10:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f3595400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35900d18b0 0x7f35901e90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f3595400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3590160f10 0x7f35900cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f3595400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3590161590 0x7f35900ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f3595400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35901e66b0 con 0x7f3590160f10 2024-03-29T10:10:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f3595400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35901e6970 con 0x7f35900d18b0 2024-03-29T10:10:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f3595400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35901e6c50 con 0x7f3590161590 2024-03-29T10:10:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f358de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3590160f10 0x7f35900cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f358de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3590160f10 0x7f35900cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38014/0 (socket says 172.21.15.12:38014) 2024-03-29T10:10:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f358de00700 1 -- 172.21.15.12:0/1564152786 learned_addr learned my addr 172.21.15.12:0/1564152786 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f358e800700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35900d18b0 0x7f35901e90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.856+0000 7f358f200700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3590161590 0x7f35900ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f358de00700 1 -- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3590161590 msgr2=0x7f35900ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f358de00700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3590161590 0x7f35900ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f358de00700 1 -- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35900d18b0 msgr2=0x7f35901e90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f358de00700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35900d18b0 0x7f35901e90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f358de00700 1 -- 172.21.15.12:0/1564152786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357c0094a0 con 0x7f3590160f10 2024-03-29T10:10:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f358de00700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3590160f10 0x7f35900cc960 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f357c014040 tx=0x7f357c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f3586a00700 1 -- 172.21.15.12:0/1564152786 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c01a860 con 0x7f3590160f10 2024-03-29T10:10:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f3586a00700 1 -- 172.21.15.12:0/1564152786 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c01a9c0 con 0x7f3590160f10 2024-03-29T10:10:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f3595400700 1 -- 172.21.15.12:0/1564152786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35901e6f30 con 0x7f3590160f10 2024-03-29T10:10:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f3586a00700 1 -- 172.21.15.12:0/1564152786 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c01ad50 con 0x7f3590160f10 2024-03-29T10:10:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.857+0000 7f3595400700 1 -- 172.21.15.12:0/1564152786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35901e95e0 con 0x7f3590160f10 2024-03-29T10:10:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.858+0000 7f3595400700 1 -- 172.21.15.12:0/1564152786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35901e7450 con 0x7f3590160f10 2024-03-29T10:10:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.859+0000 7f3586a00700 1 -- 172.21.15.12:0/1564152786 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f357c022070 con 0x7f3590160f10 2024-03-29T10:10:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.860+0000 7f3586a00700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f357006adb0 0x7f357006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.860+0000 7f3586a00700 1 -- 172.21.15.12:0/1564152786 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f357c08a590 con 0x7f3590160f10 2024-03-29T10:10:23.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.861+0000 7f358e800700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f357006adb0 0x7f357006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:23.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.861+0000 7f358e800700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f357006adb0 0x7f357006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3580007fd0 tx=0x7f3580009040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:23.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:23.865+0000 7f3586a00700 1 -- 172.21.15.12:0/1564152786 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f35901e7450 con 0x7f3590160f10 2024-03-29T10:10:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.122+0000 7f3595400700 1 -- 172.21.15.12:0/1564152786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35901e7450 con 0x7f3590160f10 2024-03-29T10:10:24.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.123+0000 7f3586a00700 1 -- 172.21.15.12:0/1564152786 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f35901e7450 con 0x7f3590160f10 2024-03-29T10:10:24.123 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:24.124 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":38,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/4026715411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":4026715411},{"type":"v1","addr":"172.21.15.159:6836","nonce":4026715411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/225205526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":225205526},{"type":"v1","addr":"172.21.15.12:6834","nonce":225205526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6918,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2358504162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2358504162},{"type":"v1","addr":"172.21.15.159:6837","nonce":2358504162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:17.624968+0000","modified":"2024-03-29T10:10:21.121687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6924},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6924":{"gid":6924,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3359577869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3359577869},{"type":"v1","addr":"172.21.15.12:6835","nonce":3359577869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-29T10:10:24.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 -- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f357006adb0 msgr2=0x7f357006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f357006adb0 0x7f357006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3580007fd0 tx=0x7f3580009040 comp rx=0 tx=0).stop 2024-03-29T10:10:24.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 -- 172.21.15.12:0/1564152786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3590160f10 msgr2=0x7f35900cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3590160f10 0x7f35900cc960 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f357c014040 tx=0x7f357c004750 comp rx=0 tx=0).stop 2024-03-29T10:10:24.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 -- 172.21.15.12:0/1564152786 shutdown_connections 2024-03-29T10:10:24.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f357006adb0 0x7f357006d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3590161590 0x7f35900ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35900d18b0 0x7f35901e90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 --2- 172.21.15.12:0/1564152786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3590160f10 0x7f35900cc960 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 -- 172.21.15.12:0/1564152786 >> 172.21.15.12:0/1564152786 conn(0x7f359008aff0 msgr2=0x7f3590155950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 -- 172.21.15.12:0/1564152786 shutdown_connections 2024-03-29T10:10:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.126+0000 7f3595400700 1 -- 172.21.15.12:0/1564152786 wait complete. 2024-03-29T10:10:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 38 2024-03-29T10:10:24.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 38, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T10:10:17.624968+0000', 'modified': '2024-03-29T10:10:21.121687+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6924}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6924': {'gid': 6924, 'name': 'a', 'rank': 0, 'incarnation': 35, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6835/3359577869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6833', 'nonce': 3359577869}, {'type': 'v1', 'addr': '172.21.15.12:6835', 'nonce': 3359577869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T10:10:24.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T10:10:24.549 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-29T10:10:24.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.752+0000 7fd89b600700 1 Processor -- start 2024-03-29T10:10:24.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.752+0000 7fd89b600700 1 -- start start 2024-03-29T10:10:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.753+0000 7fd89b600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd8941611b0 0x7fd89415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.753+0000 7fd89b600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 0x7fd894161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.753+0000 7fd89b600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8940ce270 con 0x7fd89415f4d0 2024-03-29T10:10:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.753+0000 7fd89b600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89415b600 con 0x7fd8941611b0 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.753+0000 7fd89b600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940d2760 con 0x7fd8940cee10 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd898c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd8941611b0 0x7fd89415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd893e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd89415f4d0 0x7fd8940ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44232/0 (socket says 172.21.15.12:44232) 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd893e00700 1 -- 172.21.15.12:0/831279028 learned_addr learned my addr 172.21.15.12:0/831279028 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd899600700 1 --2- 172.21.15.12:0/831279028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 0x7fd894161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd893400700 1 -- 172.21.15.12:0/831279028 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228766807 0 0) 0x7fd8940ce270 con 0x7fd89415f4d0 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd893400700 1 -- 172.21.15.12:0/831279028 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd87c0035b0 con 0x7fd89415f4d0 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd898c00700 1 -- 172.21.15.12:0/831279028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 msgr2=0x7fd894161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd898c00700 1 --2- 172.21.15.12:0/831279028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 0x7fd894161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd898c00700 1 -- 172.21.15.12:0/831279028 >> v1:172.21.15.12:6789/0 conn(0x7fd89415f4d0 legacy=0x7fd8940ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd898c00700 1 -- 172.21.15.12:0/831279028 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8940d2a40 con 0x7fd8941611b0 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.754+0000 7fd898c00700 1 --2- 172.21.15.12:0/831279028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 0x7fd89415ef90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd888009840 tx=0x7fd888009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc63124a670944d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd893400700 1 -- 172.21.15.12:0/831279028 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd888011830 con 0x7fd8941611b0 2024-03-29T10:10:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd893400700 1 -- 172.21.15.12:0/831279028 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd888011990 con 0x7fd8941611b0 2024-03-29T10:10:24.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd893400700 1 -- 172.21.15.12:0/831279028 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd888011d20 con 0x7fd8941611b0 2024-03-29T10:10:24.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd89b600700 1 -- 172.21.15.12:0/831279028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 msgr2=0x7fd89415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd89b600700 1 --2- 172.21.15.12:0/831279028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 0x7fd89415ef90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd888009840 tx=0x7fd888009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:24.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd89b600700 1 -- 172.21.15.12:0/831279028 shutdown_connections 2024-03-29T10:10:24.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd89b600700 1 --2- 172.21.15.12:0/831279028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 0x7fd894161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd89b600700 1 --2- 172.21.15.12:0/831279028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 0x7fd89415ef90 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd89b600700 1 -- 172.21.15.12:0/831279028 >> 172.21.15.12:0/831279028 conn(0x7fd89408aff0 msgr2=0x7fd8940d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:24.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd89b600700 1 -- 172.21.15.12:0/831279028 shutdown_connections 2024-03-29T10:10:24.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.755+0000 7fd89b600700 1 -- 172.21.15.12:0/831279028 wait complete. 2024-03-29T10:10:24.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.757+0000 7fd89b600700 1 Processor -- start 2024-03-29T10:10:24.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.757+0000 7fd89b600700 1 -- start start 2024-03-29T10:10:24.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.758+0000 7fd89b600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 0x7fd8941564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.758+0000 7fd89b600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd89415f4d0 0x7fd894155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.758+0000 7fd89b600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 0x7fd8940cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.758+0000 7fd89b600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8941e8770 con 0x7fd8941611b0 2024-03-29T10:10:24.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.758+0000 7fd89b600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940ca600 con 0x7fd89415f4d0 2024-03-29T10:10:24.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.758+0000 7fd89b600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940cd540 con 0x7fd8940cee10 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd899600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 0x7fd8940cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd898c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 0x7fd8941564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd899600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 0x7fd8940cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38030/0 (socket says 172.21.15.12:38030) 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd899600700 1 -- 172.21.15.12:0/2297012605 learned_addr learned my addr 172.21.15.12:0/2297012605 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd898c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 0x7fd8941564a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51248/0 (socket says 172.21.15.12:51248) 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd893e00700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd89415f4d0 0x7fd894155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd899600700 1 -- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 msgr2=0x7fd8941564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd899600700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 0x7fd8941564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd899600700 1 -- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd89415f4d0 msgr2=0x7fd894155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd899600700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd89415f4d0 0x7fd894155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd899600700 1 -- 172.21.15.12:0/2297012605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8880094a0 con 0x7fd8941611b0 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.759+0000 7fd899600700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 0x7fd8940cd000 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00efc0 tx=0x7fd88c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:24.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.760+0000 7fd890c00700 1 -- 172.21.15.12:0/2297012605 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c009ea0 con 0x7fd8941611b0 2024-03-29T10:10:24.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.760+0000 7fd890c00700 1 -- 172.21.15.12:0/2297012605 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c00f040 con 0x7fd8941611b0 2024-03-29T10:10:24.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.760+0000 7fd89b600700 1 -- 172.21.15.12:0/2297012605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8940ca850 con 0x7fd8941611b0 2024-03-29T10:10:24.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.760+0000 7fd890c00700 1 -- 172.21.15.12:0/2297012605 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c00ced0 con 0x7fd8941611b0 2024-03-29T10:10:24.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.760+0000 7fd89b600700 1 -- 172.21.15.12:0/2297012605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8941e80b0 con 0x7fd8941611b0 2024-03-29T10:10:24.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.762+0000 7fd890c00700 1 -- 172.21.15.12:0/2297012605 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd88c014480 con 0x7fd8941611b0 2024-03-29T10:10:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.763+0000 7fd890c00700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd87406adb0 0x7fd87406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.763+0000 7fd890c00700 1 -- 172.21.15.12:0/2297012605 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd88c089e30 con 0x7fd8941611b0 2024-03-29T10:10:24.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.763+0000 7fd898c00700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd87406adb0 0x7fd87406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.764+0000 7fd898c00700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd87406adb0 0x7fd87406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd888000c00 tx=0x7fd88801c040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.765+0000 7fd8af1cdb80 1 Processor -- start 2024-03-29T10:10:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.765+0000 7fd8af1cdb80 1 -- start start 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x560009357c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56000932abe0 0x56000932da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560009328b60 0x560009354a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600092a2650 con 0x56000932abe0 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x560009283c30 con 0x560009328b60 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x560009281430 con 0x560009357830 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd893e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56000932abe0 0x56000932da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd893e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56000932abe0 0x56000932da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38038/0 (socket says 172.21.15.12:38038) 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd893e00700 1 -- 172.21.15.12:0/2815585512 learned_addr learned my addr 172.21.15.12:0/2815585512 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd898c00700 1 --2- 172.21.15.12:0/2815585512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x560009357c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd899600700 1 --2- 172.21.15.12:0/2815585512 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560009328b60 0x560009354a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd893e00700 1 -- 172.21.15.12:0/2815585512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 msgr2=0x560009357c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:24.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd893e00700 1 --2- 172.21.15.12:0/2815585512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x560009357c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd893e00700 1 -- 172.21.15.12:0/2815585512 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560009328b60 msgr2=0x560009354a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:24.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd893e00700 1 --2- 172.21.15.12:0/2815585512 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560009328b60 0x560009354a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.766+0000 7fd893e00700 1 -- 172.21.15.12:0/2815585512 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560009070790 con 0x56000932abe0 2024-03-29T10:10:24.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.767+0000 7fd893e00700 1 --2- 172.21.15.12:0/2815585512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56000932abe0 0x56000932da10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd89408cca0 tx=0x7fd880007320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ec217d896541fe server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:24.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.767+0000 7fd885200700 1 -- 172.21.15.12:0/2815585512 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880006ea0 con 0x56000932abe0 2024-03-29T10:10:24.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.767+0000 7fd885200700 1 -- 172.21.15.12:0/2815585512 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd880001580 con 0x56000932abe0 2024-03-29T10:10:24.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.767+0000 7fd885200700 1 -- 172.21.15.12:0/2815585512 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880001910 con 0x56000932abe0 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.768+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2815585512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56000932abe0 msgr2=0x56000932da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.768+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2815585512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56000932abe0 0x56000932da10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd89408cca0 tx=0x7fd880007320 comp rx=0 tx=0).stop 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.768+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2815585512 shutdown_connections 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.768+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2815585512 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560009328b60 0x560009354a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.768+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2815585512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x560009357c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.768+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2815585512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56000932abe0 0x56000932da10 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.768+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2815585512 >> 172.21.15.12:0/2815585512 conn(0x56000928b6a0 msgr2=0x560009372640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.768+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2815585512 shutdown_connections 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.768+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2815585512 wait complete. 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.769+0000 7fd8af1cdb80 1 -- start start 2024-03-29T10:10:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.769+0000 7fd8af1cdb80 1 Processor -- start 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009328b60 0x5600093744d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x5600093701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009357830 0x56000935a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56000927cd20 con 0x560009357830 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x560009218770 con 0x56000932abe0 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56000923b6f0 con 0x560009328b60 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd8af1cdb80 10 client.0 _ll_register_callbacks cb 0x5600092225e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd899600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009357830 0x56000935a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd899600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009357830 0x56000935a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38040/0 (socket says 172.21.15.12:38040) 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd899600700 1 -- 172.21.15.12:0/2288204870 learned_addr learned my addr 172.21.15.12:0/2288204870 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd893e00700 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x5600093701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.770+0000 7fd898c00700 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009328b60 0x5600093744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd899600700 1 -- 172.21.15.12:0/2288204870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009328b60 msgr2=0x5600093744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd899600700 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009328b60 0x5600093744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd899600700 1 -- 172.21.15.12:0/2288204870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 msgr2=0x5600093701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd899600700 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x5600093701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd899600700 1 -- 172.21.15.12:0/2288204870 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c0035b0 con 0x560009357830 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd899600700 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009357830 0x56000935a030 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00c6f0 tx=0x7fd88c05c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558dbab141822a2a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd879600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd879600700 1 -- 172.21.15.12:0/2288204870 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c057f90 con 0x560009357830 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd879600700 1 -- 172.21.15.12:0/2288204870 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd88c0580f0 con 0x560009357830 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2288204870 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56000931f920 con 0x560009357830 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd879600700 1 -- 172.21.15.12:0/2288204870 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd88c060a00 con 0x560009357830 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd8af1cdb80 10 client.7068 fetch_fsmap learned FSMap version 38 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2288204870 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=2+}) v3 -- 0x56000915d0a0 con 0x560009357830 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.771+0000 7fd879600700 1 -- 172.21.15.12:0/2288204870 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56000931f920 con 0x560009357830 2024-03-29T10:10:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.772+0000 7fd879600700 1 -- 172.21.15.12:0/2288204870 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 38) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd88c070180 con 0x560009357830 2024-03-29T10:10:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.772+0000 7fd8af1cdb80 10 client.7068 fetch_fsmap finished waiting for FSMap version 38 2024-03-29T10:10:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.772+0000 7fd8af1cdb80 10 client.7068 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T10:10:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.772+0000 7fd8af1cdb80 20 client.7068 populate_metadata read hostname 'smithi012' 2024-03-29T10:10:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.772+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.12:6833/3359577869,v1:172.21.15.12:6835/3359577869] conn(0x560009376050 0x5600093510c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.772+0000 7fd8af1cdb80 4 client.7068 mds_command: new command op to 6924 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:10:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.772+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2288204870 --> [v2:172.21.15.12:6833/3359577869,v1:172.21.15.12:6835/3359577869] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600092fc6f0 con 0x560009376050 2024-03-29T10:10:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.772+0000 7fd898c00700 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.12:6833/3359577869,v1:172.21.15.12:6835/3359577869] conn(0x560009376050 0x5600093510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.772+0000 7fd879600700 10 client.7068 ms_handle_connect on v2:172.21.15.12:6833/3359577869 2024-03-29T10:10:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.772+0000 7fd898c00700 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.12:6833/3359577869,v1:172.21.15.12:6835/3359577869] conn(0x560009376050 0x5600093510c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9397228f00618d45 server_cookie=489acbc5b30648a9 in_seq=0 out_seq=0 2024-03-29T10:10:24.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.773+0000 7fd879600700 1 -- 172.21.15.12:0/2288204870 <== mds.0 v2:172.21.15.12:6833/3359577869 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5600092fc6f0 con 0x560009376050 2024-03-29T10:10:24.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.773+0000 7fd879600700 10 client.7068 handle_command_reply: tid=0 2024-03-29T10:10:24.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.773+0000 7fd8af1cdb80 1 client.7068 shutdown 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.773+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2288204870 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009357830 msgr2=0x56000935a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.773+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009357830 0x56000935a030 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00c6f0 tx=0x7fd88c05c8d0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.773+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2288204870 shutdown_connections 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.773+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.12:6833/3359577869,v1:172.21.15.12:6835/3359577869] conn(0x560009376050 0x5600093510c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.773+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009357830 0x56000935a030 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.773+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009328b60 0x5600093744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.773+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2288204870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x5600093701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.774+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2288204870 >> 172.21.15.12:0/2288204870 conn(0x56000928b6a0 msgr2=0x560009353f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.774+0000 7fd879600700 0 client.7068 ms_handle_reset on v2:172.21.15.12:6833/3359577869 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.774+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2288204870 shutdown_connections 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.774+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2288204870 wait complete. 2024-03-29T10:10:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.774+0000 7fd8af1cdb80 20 client.7068 trim_cache size 0 max 16384 2024-03-29T10:10:24.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.792+0000 7fd8af1cdb80 1 Processor -- start 2024-03-29T10:10:24.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.792+0000 7fd8af1cdb80 1 -- start start 2024-03-29T10:10:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x560009376490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x56000932e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x56000932e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560009071390 con 0x560009328b60 2024-03-29T10:10:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x560009328a20 con 0x56000932abe0 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x560009376230 con 0x560009357830 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd898c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x560009376490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd898c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x560009376490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38052/0 (socket says 172.21.15.12:38052) 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd898c00700 1 -- 172.21.15.12:0/2050651397 learned_addr learned my addr 172.21.15.12:0/2050651397 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd898c00700 1 -- 172.21.15.12:0/2050651397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 msgr2=0x56000932e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd898c00700 1 --2- 172.21.15.12:0/2050651397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x56000932e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd898c00700 1 -- 172.21.15.12:0/2050651397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 msgr2=0x56000932e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd898c00700 1 --2- 172.21.15.12:0/2050651397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x56000932e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.793+0000 7fd898c00700 1 -- 172.21.15.12:0/2050651397 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560009243010 con 0x560009328b60 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd898c00700 1 --2- 172.21.15.12:0/2050651397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x560009376490 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd8880090b0 tx=0x7fd888010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7743ba5aa1f1f03d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd885c00700 1 -- 172.21.15.12:0/2050651397 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88802a070 con 0x560009328b60 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd885c00700 1 -- 172.21.15.12:0/2050651397 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88802b040 con 0x560009328b60 2024-03-29T10:10:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd885c00700 1 -- 172.21.15.12:0/2050651397 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd888031980 con 0x560009328b60 2024-03-29T10:10:24.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2050651397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 msgr2=0x560009376490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2050651397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x560009376490 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd8880090b0 tx=0x7fd888010000 comp rx=0 tx=0).stop 2024-03-29T10:10:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2050651397 shutdown_connections 2024-03-29T10:10:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2050651397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x56000932e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2050651397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x560009376490 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2050651397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x56000932e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.794+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2050651397 >> 172.21.15.12:0/2050651397 conn(0x56000928b6a0 msgr2=0x560009359df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.795+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2050651397 shutdown_connections 2024-03-29T10:10:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.795+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2050651397 wait complete. 2024-03-29T10:10:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.795+0000 7fd8af1cdb80 1 -- start start 2024-03-29T10:10:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.795+0000 7fd8af1cdb80 1 Processor -- start 2024-03-29T10:10:24.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.796+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x56000932be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.796+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x560009383770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.796+0000 7fd8af1cdb80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x560009327640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.796+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600093cf9b0 con 0x560009328b60 2024-03-29T10:10:24.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.796+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600093d9880 con 0x56000932abe0 2024-03-29T10:10:24.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.796+0000 7fd8af1cdb80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56000932b340 con 0x560009357830 2024-03-29T10:10:24.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.796+0000 7fd8af1cdb80 10 client.0 _ll_register_callbacks cb 0x5600092225e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:10:24.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd898c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x56000932be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd898c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x56000932be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38068/0 (socket says 172.21.15.12:38068) 2024-03-29T10:10:24.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd898c00700 1 -- 172.21.15.12:0/2893569082 learned_addr learned my addr 172.21.15.12:0/2893569082 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd893e00700 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x560009383770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd899600700 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x560009327640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd898c00700 1 -- 172.21.15.12:0/2893569082 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 msgr2=0x560009327640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd898c00700 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x560009327640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd898c00700 1 -- 172.21.15.12:0/2893569082 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 msgr2=0x560009383770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd898c00700 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x560009383770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd898c00700 1 -- 172.21.15.12:0/2893569082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8940d2a40 con 0x560009328b60 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd899600700 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x560009327640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd898c00700 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x56000932be20 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd888009ea0 tx=0x7fd8880047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde7fe6d1c446424 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.797+0000 7fd879600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd879600700 1 -- 172.21.15.12:0/2893569082 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88802a070 con 0x560009328b60 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd879600700 1 -- 172.21.15.12:0/2893569082 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd88801f070 con 0x560009328b60 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2893569082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560009358cc0 con 0x560009328b60 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd879600700 1 -- 172.21.15.12:0/2893569082 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd888031b90 con 0x560009328b60 2024-03-29T10:10:24.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd8af1cdb80 10 client.7074 fetch_fsmap learned FSMap version 38 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2893569082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=2+}) v3 -- 0x5600090d11d0 con 0x560009328b60 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd879600700 1 -- 172.21.15.12:0/2893569082 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560009358cc0 con 0x560009328b60 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd879600700 1 -- 172.21.15.12:0/2893569082 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 38) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd888040020 con 0x560009328b60 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd8af1cdb80 10 client.7074 fetch_fsmap finished waiting for FSMap version 38 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd8af1cdb80 10 client.7074 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd8af1cdb80 20 client.7074 populate_metadata read hostname 'smithi012' 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.12:6833/3359577869,v1:172.21.15.12:6835/3359577869] conn(0x56000936e9d0 0x56000936edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd8af1cdb80 4 client.7074 mds_command: new command op to 6924 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.798+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2893569082 --> [v2:172.21.15.12:6833/3359577869,v1:172.21.15.12:6835/3359577869] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fd880001580 con 0x56000936e9d0 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.799+0000 7fd893e00700 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.12:6833/3359577869,v1:172.21.15.12:6835/3359577869] conn(0x56000936e9d0 0x56000936edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.799+0000 7fd893e00700 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.12:6833/3359577869,v1:172.21.15.12:6835/3359577869] conn(0x56000936e9d0 0x56000936edb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a8fe829b167ab93 server_cookie=3e998e4a2c34048c in_seq=0 out_seq=0 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.799+0000 7fd879600700 10 client.7074 ms_handle_connect on v2:172.21.15.12:6833/3359577869 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.799+0000 7fd879600700 1 -- 172.21.15.12:0/2893569082 <== mds.0 v2:172.21.15.12:6833/3359577869 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x7fd880001580 con 0x56000936e9d0 2024-03-29T10:10:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.799+0000 7fd879600700 10 client.7074 handle_command_reply: tid=0 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.799+0000 7fd8af1cdb80 1 client.7074 shutdown 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.799+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2893569082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 msgr2=0x56000932be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x56000932be20 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd888009ea0 tx=0x7fd8880047c0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2893569082 shutdown_connections 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.12:6833/3359577869,v1:172.21.15.12:6835/3359577869] conn(0x56000936e9d0 0x56000936edb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560009357830 0x560009327640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560009328b60 0x56000932be20 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd879600700 0 client.7074 ms_handle_reset on v2:172.21.15.12:6833/3359577869 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd8af1cdb80 1 --2- 172.21.15.12:0/2893569082 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56000932abe0 0x560009383770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2893569082 >> 172.21.15.12:0/2893569082 conn(0x56000928b6a0 msgr2=0x560009376370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2893569082 shutdown_connections 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd8af1cdb80 1 -- 172.21.15.12:0/2893569082 wait complete. 2024-03-29T10:10:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd8af1cdb80 20 client.7074 trim_cache size 0 max 16384 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 6924, 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.172766899, 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 38, 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 94, 2024-03-29T10:10:24.804 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12.170365065 2024-03-29T10:10:24.805 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:10:24.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd89b600700 1 -- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd87406adb0 msgr2=0x7fd87406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd89b600700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd87406adb0 0x7fd87406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd888000c00 tx=0x7fd88801c040 comp rx=0 tx=0).stop 2024-03-29T10:10:24.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd89b600700 1 -- 172.21.15.12:0/2297012605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 msgr2=0x7fd8940cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:24.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.800+0000 7fd89b600700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 0x7fd8940cd000 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00efc0 tx=0x7fd88c00c3a0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.801+0000 7fd89b600700 1 -- 172.21.15.12:0/2297012605 shutdown_connections 2024-03-29T10:10:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.801+0000 7fd89b600700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd87406adb0 0x7fd87406d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.801+0000 7fd89b600700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8941611b0 0x7fd8940cd000 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.801+0000 7fd89b600700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8940cee10 0x7fd8941564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.801+0000 7fd89b600700 1 --2- 172.21.15.12:0/2297012605 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd89415f4d0 0x7fd894155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.801+0000 7fd89b600700 1 -- 172.21.15.12:0/2297012605 >> 172.21.15.12:0/2297012605 conn(0x7fd89408aff0 msgr2=0x7fd89415f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.801+0000 7fd89b600700 1 -- 172.21.15.12:0/2297012605 shutdown_connections 2024-03-29T10:10:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:24.801+0000 7fd89b600700 1 -- 172.21.15.12:0/2297012605 wait complete. 2024-03-29T10:10:24.819 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T10:10:25.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.016+0000 7fb24dc00700 1 Processor -- start 2024-03-29T10:10:25.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.016+0000 7fb24dc00700 1 -- start start 2024-03-29T10:10:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.017+0000 7fb24dc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb2480b3af0 0x7fb24815a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.017+0000 7fb24dc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2480b2180 con 0x7fb2480709c0 2024-03-29T10:10:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.017+0000 7fb24dc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24815b100 con 0x7fb248072580 2024-03-29T10:10:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.017+0000 7fb24dc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24815b3e0 con 0x7fb2480b3af0 2024-03-29T10:10:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb247400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb2480709c0 0x7fb24806e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44246/0 (socket says 172.21.15.12:44246) 2024-03-29T10:10:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb247400700 1 -- 172.21.15.12:0/2623922164 learned_addr learned my addr 172.21.15.12:0/2623922164 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:25.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb246a00700 1 --2- 172.21.15.12:0/2623922164 >> v2:172.21.15.159:3300/0 conn(0x7fb2480b3af0 0x7fb24815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:25.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401561762 0 0) 0x7fb2480b2180 con 0x7fb2480709c0 2024-03-29T10:10:25.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb22c0035b0 con 0x7fb2480709c0 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4116549884 0 0) 0x7fb22c0035b0 con 0x7fb2480709c0 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 >> v2:172.21.15.159:3300/0 conn(0x7fb2480b3af0 msgr2=0x7fb24815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb245600700 1 --2- 172.21.15.12:0/2623922164 >> v2:172.21.15.159:3300/0 conn(0x7fb2480b3af0 0x7fb24815a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 >> v1:172.21.15.159:6789/0 conn(0x7fb248072580 legacy=0x7fb2480b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2480b23f0 con 0x7fb2480709c0 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fb2400019f0 con 0x7fb2480709c0 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 >> v1:172.21.15.12:6789/0 conn(0x7fb2480709c0 legacy=0x7fb24806e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.018+0000 7fb245600700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c003a00 0x7fb22c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb245600700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb22c004400 0x7fb22c0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb245600700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb22c0050d0 0x7fb22c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24815b3e0 con 0x7fb22c003a00 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c00c030 con 0x7fb22c0050d0 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c00c1f0 con 0x7fb22c004400 2024-03-29T10:10:25.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb246a00700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c003a00 0x7fb22c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb247400700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb22c0050d0 0x7fb22c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb246000700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb22c004400 0x7fb22c0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb246a00700 1 -- 172.21.15.12:0/2623922164 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb22c004400 msgr2=0x7fb22c0048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb246a00700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb22c004400 0x7fb22c0048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb246a00700 1 -- 172.21.15.12:0/2623922164 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb22c0050d0 msgr2=0x7fb22c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb246a00700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb22c0050d0 0x7fb22c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb246a00700 1 -- 172.21.15.12:0/2623922164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c0035b0 con 0x7fb22c003a00 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb247400700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb22c0050d0 0x7fb22c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb247400700 1 -- 172.21.15.12:0/2623922164 reap_dead start 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.019+0000 7fb246a00700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c003a00 0x7fb22c003de0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb238009b60 tx=0x7fb22c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a71f5d735d66d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.020+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23800e6c0 con 0x7fb22c003a00 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.020+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23800e820 con 0x7fb22c003a00 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.020+0000 7fb245600700 1 -- 172.21.15.12:0/2623922164 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23800eb90 con 0x7fb22c003a00 2024-03-29T10:10:25.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.020+0000 7fb24dc00700 1 -- 172.21.15.12:0/2623922164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c003a00 msgr2=0x7fb22c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.020+0000 7fb24dc00700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c003a00 0x7fb22c003de0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb238009b60 tx=0x7fb22c005880 comp rx=0 tx=0).stop 2024-03-29T10:10:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.020+0000 7fb24dc00700 1 -- 172.21.15.12:0/2623922164 shutdown_connections 2024-03-29T10:10:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.020+0000 7fb24dc00700 1 --2- 172.21.15.12:0/2623922164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c003a00 0x7fb22c003de0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.020+0000 7fb24dc00700 1 -- 172.21.15.12:0/2623922164 >> 172.21.15.12:0/2623922164 conn(0x7fb2480cf190 msgr2=0x7fb2480cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.020+0000 7fb24dc00700 1 -- 172.21.15.12:0/2623922164 shutdown_connections 2024-03-29T10:10:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.020+0000 7fb24dc00700 1 -- 172.21.15.12:0/2623922164 wait complete. 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.022+0000 7fb24dc00700 1 Processor -- start 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.022+0000 7fb24dc00700 1 -- start start 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.023+0000 7fb24dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb22c003a00 0x7fb2481f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.023+0000 7fb24dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb248159260 0x7fb2481596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.023+0000 7fb24dc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb24815e380 0x7fb24815e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.023+0000 7fb24dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24815ed40 con 0x7fb248159260 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.023+0000 7fb24dc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb248151db0 con 0x7fb24815e380 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.023+0000 7fb24dc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481511a0 con 0x7fb22c003a00 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.023+0000 7fb246000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb248159260 0x7fb2481596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.024+0000 7fb246000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb248159260 0x7fb2481596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38096/0 (socket says 172.21.15.12:38096) 2024-03-29T10:10:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.024+0000 7fb246a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb22c003a00 0x7fb2481f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.024+0000 7fb246000700 1 -- 172.21.15.12:0/1787243855 learned_addr learned my addr 172.21.15.12:0/1787243855 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.024+0000 7fb247400700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb24815e380 0x7fb24815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.024+0000 7fb246000700 1 -- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb22c003a00 msgr2=0x7fb2481f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.024+0000 7fb246000700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb22c003a00 0x7fb2481f2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.024+0000 7fb246000700 1 -- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb24815e380 msgr2=0x7fb24815e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.024+0000 7fb246000700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb24815e380 0x7fb24815e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.024+0000 7fb246000700 1 -- 172.21.15.12:0/1787243855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb248151480 con 0x7fb248159260 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.024+0000 7fb246000700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb248159260 0x7fb2481596e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fb23400e800 tx=0x7fb23400eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.025+0000 7fb23ea00700 1 -- 172.21.15.12:0/1787243855 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2340052e0 con 0x7fb248159260 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.025+0000 7fb23ea00700 1 -- 172.21.15.12:0/1787243855 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb234005460 con 0x7fb248159260 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.025+0000 7fb23ea00700 1 -- 172.21.15.12:0/1787243855 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2340038c0 con 0x7fb248159260 2024-03-29T10:10:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.025+0000 7fb24dc00700 1 -- 172.21.15.12:0/1787243855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb248151760 con 0x7fb248159260 2024-03-29T10:10:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.025+0000 7fb24dc00700 1 -- 172.21.15.12:0/1787243855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2481f3140 con 0x7fb248159260 2024-03-29T10:10:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.026+0000 7fb24dc00700 1 -- 172.21.15.12:0/1787243855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb248067720 con 0x7fb248159260 2024-03-29T10:10:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.027+0000 7fb23ea00700 1 -- 172.21.15.12:0/1787243855 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb234003a20 con 0x7fb248159260 2024-03-29T10:10:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.027+0000 7fb23ea00700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb22806adb0 0x7fb22806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.027+0000 7fb23ea00700 1 -- 172.21.15.12:0/1787243855 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb234088ca0 con 0x7fb248159260 2024-03-29T10:10:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.028+0000 7fb246a00700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb22806adb0 0x7fb22806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.029+0000 7fb246a00700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb22806adb0 0x7fb22806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb2380090b0 tx=0x7fb23801e040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.032+0000 7fb23ea00700 1 -- 172.21.15.12:0/1787243855 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb234053cd0 con 0x7fb248159260 2024-03-29T10:10:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.302+0000 7fb24dc00700 1 -- 172.21.15.12:0/1787243855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb2480cb980 con 0x7fb248159260 2024-03-29T10:10:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.302+0000 7fb23ea00700 1 -- 172.21.15.12:0/1787243855 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v38) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb234019020 con 0x7fb248159260 2024-03-29T10:10:25.302 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T10:10:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 -- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb22806adb0 msgr2=0x7fb22806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb22806adb0 0x7fb22806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb2380090b0 tx=0x7fb23801e040 comp rx=0 tx=0).stop 2024-03-29T10:10:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 -- 172.21.15.12:0/1787243855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb248159260 msgr2=0x7fb2481596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb248159260 0x7fb2481596e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fb23400e800 tx=0x7fb23400eb10 comp rx=0 tx=0).stop 2024-03-29T10:10:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 -- 172.21.15.12:0/1787243855 shutdown_connections 2024-03-29T10:10:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb22806adb0 0x7fb22806d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb24815e380 0x7fb24815e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb22c003a00 0x7fb2481f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 --2- 172.21.15.12:0/1787243855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb248159260 0x7fb2481596e0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 -- 172.21.15.12:0/1787243855 >> 172.21.15.12:0/1787243855 conn(0x7fb2480cf190 msgr2=0x7fb24806d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 -- 172.21.15.12:0/1787243855 shutdown_connections 2024-03-29T10:10:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:25.305+0000 7fb24dc00700 1 -- 172.21.15.12:0/1787243855 wait complete. 2024-03-29T10:10:25.323 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T10:10:25.323 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T10:10:25.323 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T10:10:25.323 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T10:10:25.323 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T10:10:25.323 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T10:10:25.323 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T10:10:25.323 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T10:10:25.324 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:10:25.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:25.324 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 40492sec preferred_lft 40492sec 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:10:25.371 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T10:10:25.372 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:10:25.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:25.373 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T10:10:25.475 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T10:10:25.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:25.479 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T10:10:25.525 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T10:10:25.525 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T10:10:25.525 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T10:10:25.525 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T10:10:25.526 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T10:10:25.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:25.527 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T10:10:25.527 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T10:10:25.527 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T10:10:25.527 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T10:10:25.527 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T10:10:25.527 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T10:10:25.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:25.693 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T10:10:25.737 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T10:10:25.737 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T10:10:25.738 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:10:25.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:25.739 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T10:10:25.847 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:10:25.847 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T10:10:25.891 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:10:25.892 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:10:25.892 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:10:25.975 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T10:10:25.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:25.975 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:10:26.117 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:10:26.117 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:10:26.143 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:10:26.144 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:10:26.144 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T10:10:26.144 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T10:10:26.144 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T10:10:26.149 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T10:10:26.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:26.149 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:10:26.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:26.219 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T10:10:26.318 INFO:teuthology.orchestra.run.smithi012.stdout:client6200 2024-03-29T10:10:26.319 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:10:26.319 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:26.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T10:10:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.602+0000 7f0a9e400700 1 Processor -- start 2024-03-29T10:10:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.602+0000 7f0a9e400700 1 -- start start 2024-03-29T10:10:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.604+0000 7f0a9e400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a980cbb30 0x7f0a980cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.604+0000 7f0a9e400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a980cc450 con 0x7f0a980cd020 2024-03-29T10:10:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.604+0000 7f0a9e400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a980cdae0 con 0x7f0a980cd480 2024-03-29T10:10:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.604+0000 7f0a9e400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980cd8f0 con 0x7f0a980cbb30 2024-03-29T10:10:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.604+0000 7f0a97400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a980cbb30 0x7f0a980cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.604+0000 7f0a96a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0a980cd020 0x7f0a980b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44252/0 (socket says 172.21.15.12:44252) 2024-03-29T10:10:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.604+0000 7f0a96a00700 1 -- 172.21.15.12:0/996824646 learned_addr learned my addr 172.21.15.12:0/996824646 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.604+0000 7f0a97400700 1 -- 172.21.15.12:0/996824646 >> v1:172.21.15.159:6789/0 conn(0x7f0a980cd480 legacy=0x7f0a9806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.604+0000 7f0a97400700 1 -- 172.21.15.12:0/996824646 >> v1:172.21.15.12:6789/0 conn(0x7f0a980cd020 legacy=0x7f0a980b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.604+0000 7f0a97400700 1 -- 172.21.15.12:0/996824646 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a980cddc0 con 0x7f0a980cbb30 2024-03-29T10:10:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.605+0000 7f0a97400700 1 --2- 172.21.15.12:0/996824646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 0x7f0a980cbf10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009840 tx=0x7f0a88009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb4265079aef900 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.605+0000 7f0a96000700 1 -- 172.21.15.12:0/996824646 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88011830 con 0x7f0a980cbb30 2024-03-29T10:10:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.605+0000 7f0a96000700 1 -- 172.21.15.12:0/996824646 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a88011990 con 0x7f0a980cbb30 2024-03-29T10:10:26.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.605+0000 7f0a96000700 1 -- 172.21.15.12:0/996824646 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88011d20 con 0x7f0a980cbb30 2024-03-29T10:10:26.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.605+0000 7f0a9e400700 1 -- 172.21.15.12:0/996824646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 msgr2=0x7f0a980cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:26.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.605+0000 7f0a9e400700 1 --2- 172.21.15.12:0/996824646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 0x7f0a980cbf10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009840 tx=0x7f0a88009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:26.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.606+0000 7f0a9e400700 1 -- 172.21.15.12:0/996824646 shutdown_connections 2024-03-29T10:10:26.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.606+0000 7f0a9e400700 1 --2- 172.21.15.12:0/996824646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 0x7f0a980cbf10 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:26.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.606+0000 7f0a9e400700 1 -- 172.21.15.12:0/996824646 >> 172.21.15.12:0/996824646 conn(0x7f0a980b0d70 msgr2=0x7f0a980b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:26.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.606+0000 7f0a9e400700 1 -- 172.21.15.12:0/996824646 shutdown_connections 2024-03-29T10:10:26.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.606+0000 7f0a9e400700 1 -- 172.21.15.12:0/996824646 wait complete. 2024-03-29T10:10:26.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.607+0000 7f0a9e400700 1 Processor -- start 2024-03-29T10:10:26.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.608+0000 7f0a9e400700 1 -- start start 2024-03-29T10:10:26.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a9e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 0x7f0a981521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:26.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a9e400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a980cd020 0x7f0a98154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a9e400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a980cd480 0x7f0a98152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a9e400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98099240 con 0x7f0a980cbb30 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a9e400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98153860 con 0x7f0a980cd020 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a9e400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98153b40 con 0x7f0a980cd480 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a97400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 0x7f0a981521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a97400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 0x7f0a981521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38118/0 (socket says 172.21.15.12:38118) 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a97400700 1 -- 172.21.15.12:0/505693379 learned_addr learned my addr 172.21.15.12:0/505693379 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a96a00700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a980cd020 0x7f0a98154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a97400700 1 -- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a980cd480 msgr2=0x7f0a98152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a97e00700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a980cd480 0x7f0a98152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a97400700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a980cd480 0x7f0a98152e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a97400700 1 -- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a980cd020 msgr2=0x7f0a98154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a97400700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a980cd020 0x7f0a98154900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a97400700 1 -- 172.21.15.12:0/505693379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a880094a0 con 0x7f0a980cbb30 2024-03-29T10:10:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a96a00700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a980cd020 0x7f0a98154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.609+0000 7f0a97e00700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a980cd480 0x7f0a98152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.610+0000 7f0a97400700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 0x7f0a981521a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f0a88000c00 tx=0x7f0a88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.610+0000 7f0a8f400700 1 -- 172.21.15.12:0/505693379 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a880038a0 con 0x7f0a980cbb30 2024-03-29T10:10:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.610+0000 7f0a8f400700 1 -- 172.21.15.12:0/505693379 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8801e070 con 0x7f0a980cbb30 2024-03-29T10:10:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.610+0000 7f0a8f400700 1 -- 172.21.15.12:0/505693379 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88003c60 con 0x7f0a980cbb30 2024-03-29T10:10:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.610+0000 7f0a9e400700 1 -- 172.21.15.12:0/505693379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a98153e20 con 0x7f0a980cbb30 2024-03-29T10:10:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.610+0000 7f0a9e400700 1 -- 172.21.15.12:0/505693379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a981543e0 con 0x7f0a980cbb30 2024-03-29T10:10:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.611+0000 7f0a9e400700 1 -- 172.21.15.12:0/505693379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a981545d0 con 0x7f0a980cbb30 2024-03-29T10:10:26.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.612+0000 7f0a8f400700 1 -- 172.21.15.12:0/505693379 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0a8802d450 con 0x7f0a980cbb30 2024-03-29T10:10:26.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.613+0000 7f0a8f400700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a7806adb0 0x7f0a7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:26.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.613+0000 7f0a8f400700 1 -- 172.21.15.12:0/505693379 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a8808a660 con 0x7f0a980cbb30 2024-03-29T10:10:26.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.613+0000 7f0a96a00700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a7806adb0 0x7f0a7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:26.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.614+0000 7f0a96a00700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a7806adb0 0x7f0a7806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0a98156090 tx=0x7f0a84009040 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:26.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.617+0000 7f0a8f400700 1 -- 172.21.15.12:0/505693379 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0a88055730 con 0x7f0a980cbb30 2024-03-29T10:10:26.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.859+0000 7f0a9e400700 1 -- 172.21.15.12:0/505693379 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a9806df00 con 0x7f0a7806adb0 2024-03-29T10:10:26.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.861+0000 7f0a8f400700 1 -- 172.21.15.12:0/505693379 <== mgr.5954 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0a84005960 con 0x7f0a7806adb0 2024-03-29T10:10:26.861 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:10:26.861 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:26.861 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T10:10:26.861 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:26.861 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:10:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 -- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a7806adb0 msgr2=0x7f0a7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a7806adb0 0x7f0a7806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0a98156090 tx=0x7f0a84009040 comp rx=0 tx=0).stop 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 -- 172.21.15.12:0/505693379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 msgr2=0x7f0a981521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 0x7f0a981521a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f0a88000c00 tx=0x7f0a88004750 comp rx=0 tx=0).stop 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 -- 172.21.15.12:0/505693379 shutdown_connections 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0a7806adb0 0x7f0a7806d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a980cd480 0x7f0a98152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a980cbb30 0x7f0a981521a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 --2- 172.21.15.12:0/505693379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a980cd020 0x7f0a98154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 -- 172.21.15.12:0/505693379 >> 172.21.15.12:0/505693379 conn(0x7f0a980b0d70 msgr2=0x7f0a9806e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 -- 172.21.15.12:0/505693379 shutdown_connections 2024-03-29T10:10:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:26.864+0000 7f0a9e400700 1 -- 172.21.15.12:0/505693379 wait complete. 2024-03-29T10:10:26.882 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T10:10:26.933 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:10:26.936 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:10:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.087+0000 7f7841a00700 1 Processor -- start 2024-03-29T10:10:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.088+0000 7f7841a00700 1 -- start start 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.089+0000 7f7841a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1611b0 0x7f783c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.089+0000 7f7841a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f783c0cee10 0x7f783c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.089+0000 7f7841a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783c0ce270 con 0x7f783c15f4d0 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.089+0000 7f7841a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f783c15b600 con 0x7f783c0cee10 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.089+0000 7f7841a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783c0d2760 con 0x7f783c1611b0 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.089+0000 7f783a000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f783c15f4d0 0x7f783c0ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44262/0 (socket says 172.21.15.12:44262) 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.089+0000 7f783a000700 1 -- 172.21.15.12:0/1958747560 learned_addr learned my addr 172.21.15.12:0/1958747560 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.089+0000 7f783b400700 1 --2- 172.21.15.12:0/1958747560 >> v2:172.21.15.12:3300/0 conn(0x7f783c0cee10 0x7f783c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.089+0000 7f783aa00700 1 --2- 172.21.15.12:0/1958747560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1611b0 0x7f783c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.089+0000 7f7839600700 1 -- 172.21.15.12:0/1958747560 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498118653 0 0) 0x7f783c0ce270 con 0x7f783c15f4d0 2024-03-29T10:10:27.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.090+0000 7f7839600700 1 -- 172.21.15.12:0/1958747560 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78200035b0 con 0x7f783c15f4d0 2024-03-29T10:10:27.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.090+0000 7f783b400700 1 -- 172.21.15.12:0/1958747560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1611b0 msgr2=0x7f783c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:27.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.090+0000 7f783b400700 1 --2- 172.21.15.12:0/1958747560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1611b0 0x7f783c15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:27.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.090+0000 7f783b400700 1 -- 172.21.15.12:0/1958747560 >> v1:172.21.15.12:6789/0 conn(0x7f783c15f4d0 legacy=0x7f783c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:27.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.090+0000 7f783b400700 1 -- 172.21.15.12:0/1958747560 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c0d2a40 con 0x7f783c0cee10 2024-03-29T10:10:27.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.090+0000 7f783b400700 1 --2- 172.21.15.12:0/1958747560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 0x7f783c161590 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7834009d90 tx=0x7f783400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e9bddebdbc10ab server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:27.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.090+0000 7f7839600700 1 -- 172.21.15.12:0/1958747560 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783400e070 con 0x7f783c0cee10 2024-03-29T10:10:27.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.090+0000 7f7839600700 1 -- 172.21.15.12:0/1958747560 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783400bb20 con 0x7f783c0cee10 2024-03-29T10:10:27.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.090+0000 7f7839600700 1 -- 172.21.15.12:0/1958747560 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7834012430 con 0x7f783c0cee10 2024-03-29T10:10:27.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.091+0000 7f7841a00700 1 -- 172.21.15.12:0/1958747560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 msgr2=0x7f783c161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:27.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.091+0000 7f7841a00700 1 --2- 172.21.15.12:0/1958747560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 0x7f783c161590 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7834009d90 tx=0x7f783400b330 comp rx=0 tx=0).stop 2024-03-29T10:10:27.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.091+0000 7f7841a00700 1 -- 172.21.15.12:0/1958747560 shutdown_connections 2024-03-29T10:10:27.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.091+0000 7f7841a00700 1 --2- 172.21.15.12:0/1958747560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 0x7f783c161590 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:27.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.091+0000 7f7841a00700 1 --2- 172.21.15.12:0/1958747560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1611b0 0x7f783c15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:27.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.091+0000 7f7841a00700 1 -- 172.21.15.12:0/1958747560 >> 172.21.15.12:0/1958747560 conn(0x7f783c08aff0 msgr2=0x7f783c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:27.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.091+0000 7f7841a00700 1 -- 172.21.15.12:0/1958747560 shutdown_connections 2024-03-29T10:10:27.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.091+0000 7f7841a00700 1 -- 172.21.15.12:0/1958747560 wait complete. 2024-03-29T10:10:27.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.093+0000 7f7841a00700 1 Processor -- start 2024-03-29T10:10:27.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.093+0000 7f7841a00700 1 -- start start 2024-03-29T10:10:27.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f7841a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 0x7f783c0cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f7841a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c15f4d0 0x7f783c0c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f7841a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1611b0 0x7f783c0c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f7841a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783c0c9240 con 0x7f783c0cee10 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f7841a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783c0b6570 con 0x7f783c15f4d0 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f7841a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783c152fd0 con 0x7f783c1611b0 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f783aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 0x7f783c0cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f783aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 0x7f783c0cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38140/0 (socket says 172.21.15.12:38140) 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f783aa00700 1 -- 172.21.15.12:0/4044351507 learned_addr learned my addr 172.21.15.12:0/4044351507 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f783aa00700 1 -- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1611b0 msgr2=0x7f783c0c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f783aa00700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1611b0 0x7f783c0c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f783aa00700 1 -- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c15f4d0 msgr2=0x7f783c0c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f783aa00700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c15f4d0 0x7f783c0c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.094+0000 7f783aa00700 1 -- 172.21.15.12:0/4044351507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7834009a70 con 0x7f783c0cee10 2024-03-29T10:10:27.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.095+0000 7f783aa00700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 0x7f783c0cc770 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f782c00e870 tx=0x7f782c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:27.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.095+0000 7f7832a00700 1 -- 172.21.15.12:0/4044351507 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c009590 con 0x7f783c0cee10 2024-03-29T10:10:27.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.095+0000 7f7832a00700 1 -- 172.21.15.12:0/4044351507 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782c00ee70 con 0x7f783c0cee10 2024-03-29T10:10:27.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.095+0000 7f7841a00700 1 -- 172.21.15.12:0/4044351507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783c0b6810 con 0x7f783c0cee10 2024-03-29T10:10:27.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.095+0000 7f7832a00700 1 -- 172.21.15.12:0/4044351507 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c00ca50 con 0x7f783c0cee10 2024-03-29T10:10:27.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.095+0000 7f7841a00700 1 -- 172.21.15.12:0/4044351507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f783c0b6dd0 con 0x7f783c0cee10 2024-03-29T10:10:27.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.096+0000 7f7841a00700 1 -- 172.21.15.12:0/4044351507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f783c067720 con 0x7f783c0cee10 2024-03-29T10:10:27.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.097+0000 7f7832a00700 1 -- 172.21.15.12:0/4044351507 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 38) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f782c00cbb0 con 0x7f783c0cee10 2024-03-29T10:10:27.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.098+0000 7f7832a00700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f781c06ade0 0x7f781c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:27.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.098+0000 7f7832a00700 1 -- 172.21.15.12:0/4044351507 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f782c089470 con 0x7f783c0cee10 2024-03-29T10:10:27.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.098+0000 7f783a000700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f781c06ade0 0x7f781c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:27.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.099+0000 7f783a000700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f781c06ade0 0x7f781c06d2a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f783c162210 tx=0x7f78280079c0 comp rx=0 tx=0).ready entity=mgr.5954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:27.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.102+0000 7f7832a00700 1 -- 172.21.15.12:0/4044351507 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f782c054530 con 0x7f783c0cee10 2024-03-29T10:10:27.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:27.380+0000 7f7841a00700 1 -- 172.21.15.12:0/4044351507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f783c0b5210 con 0x7f783c0cee10 2024-03-29T10:10:28.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.156+0000 7f7832a00700 1 -- 172.21.15.12:0/4044351507 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 39) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f782c054900 con 0x7f783c0cee10 2024-03-29T10:10:28.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.158+0000 7f7832a00700 1 -- 172.21.15.12:0/4044351507 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v39) v1 ==== 92+0+0 (secure 0 0 0) 0x7f782c086be0 con 0x7f783c0cee10 2024-03-29T10:10:28.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 -- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f781c06ade0 msgr2=0x7f781c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f781c06ade0 0x7f781c06d2a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f783c162210 tx=0x7f78280079c0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 -- 172.21.15.12:0/4044351507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 msgr2=0x7f783c0cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 0x7f783c0cc770 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f782c00e870 tx=0x7f782c00ec30 comp rx=0 tx=0).stop 2024-03-29T10:10:28.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 -- 172.21.15.12:0/4044351507 shutdown_connections 2024-03-29T10:10:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f781c06ade0 0x7f781c06d2a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1611b0 0x7f783c0c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f783c0cee10 0x7f783c0cc770 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 --2- 172.21.15.12:0/4044351507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c15f4d0 0x7f783c0c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 -- 172.21.15.12:0/4044351507 >> 172.21.15.12:0/4044351507 conn(0x7f783c08aff0 msgr2=0x7f783c15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 -- 172.21.15.12:0/4044351507 shutdown_connections 2024-03-29T10:10:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.161+0000 7f7841a00700 1 -- 172.21.15.12:0/4044351507 wait complete. 2024-03-29T10:10:28.183 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T10:10:28.377 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:28.377+0000 7fd5b3b59240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:10:28.392 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:28.392+0000 7fac74ec0240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:10:28.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.392+0000 7f23b3200700 1 Processor -- start 2024-03-29T10:10:28.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.393+0000 7f23b3200700 1 -- start start 2024-03-29T10:10:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.394+0000 7f23b3200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b2b30 0x7f23ac0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.394+0000 7f23b3200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f23ac0b3960 0x7f23ac0cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.394+0000 7f23b3200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac0b3db0 con 0x7f23ac157f90 2024-03-29T10:10:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.394+0000 7f23b3200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ac158370 con 0x7f23ac0b3960 2024-03-29T10:10:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.394+0000 7f23b3200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0b2710 con 0x7f23ac0b2b30 2024-03-29T10:10:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.394+0000 7f23b1200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f23ac0b3960 0x7f23ac0cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.394+0000 7f23b1200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f23ac0b3960 0x7f23ac0cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38174/0 (socket says 172.21.15.12:38174) 2024-03-29T10:10:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.394+0000 7f23b1200700 1 -- 172.21.15.12:0/4294935455 learned_addr learned my addr 172.21.15.12:0/4294935455 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.394+0000 7f23ab400700 1 --2- 172.21.15.12:0/4294935455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b2b30 0x7f23ac0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.395+0000 7f23b1200700 1 -- 172.21.15.12:0/4294935455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b2b30 msgr2=0x7f23ac0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.395+0000 7f23b1200700 1 --2- 172.21.15.12:0/4294935455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b2b30 0x7f23ac0b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.395+0000 7f23b1200700 1 -- 172.21.15.12:0/4294935455 >> v1:172.21.15.159:6789/0 conn(0x7f23ac157f90 legacy=0x7f23ac1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.395+0000 7f23b1200700 1 -- 172.21.15.12:0/4294935455 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0b2070 con 0x7f23ac0b3960 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.394+0000 7f23aaa00700 1 -- 172.21.15.12:0/4294935455 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932025517 0 0) 0x7f239c001110 con 0x7f23ac157f90 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.395+0000 7f23b1200700 1 --2- 172.21.15.12:0/4294935455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23ac0b3960 0x7f23ac0cd020 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009e50 tx=0x7f23a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f385273f8a1a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.395+0000 7f23aaa00700 1 -- 172.21.15.12:0/4294935455 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a400e070 con 0x7f23ac0b3960 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.395+0000 7f23aaa00700 1 -- 172.21.15.12:0/4294935455 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a400b910 con 0x7f23ac0b3960 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.395+0000 7f23aaa00700 1 -- 172.21.15.12:0/4294935455 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a400bcc0 con 0x7f23ac0b3960 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.396+0000 7f23b3200700 1 -- 172.21.15.12:0/4294935455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23ac0b3960 msgr2=0x7f23ac0cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.396+0000 7f23b3200700 1 --2- 172.21.15.12:0/4294935455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23ac0b3960 0x7f23ac0cd020 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009e50 tx=0x7f23a400b330 comp rx=0 tx=0).stop 2024-03-29T10:10:28.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.396+0000 7f23b3200700 1 -- 172.21.15.12:0/4294935455 shutdown_connections 2024-03-29T10:10:28.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.396+0000 7f23b3200700 1 --2- 172.21.15.12:0/4294935455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23ac0b3960 0x7f23ac0cd020 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.396+0000 7f23b3200700 1 --2- 172.21.15.12:0/4294935455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b2b30 0x7f23ac0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.396+0000 7f23b3200700 1 -- 172.21.15.12:0/4294935455 >> 172.21.15.12:0/4294935455 conn(0x7f23ac0b0d70 msgr2=0x7f23ac0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:28.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.396+0000 7f23b3200700 1 -- 172.21.15.12:0/4294935455 shutdown_connections 2024-03-29T10:10:28.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.396+0000 7f23b3200700 1 -- 172.21.15.12:0/4294935455 wait complete. 2024-03-29T10:10:28.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.398+0000 7f23b3200700 1 Processor -- start 2024-03-29T10:10:28.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.398+0000 7f23b3200700 1 -- start start 2024-03-29T10:10:28.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b3200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23ac0b2b30 0x7f23ac09a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b3200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b3960 0x7f23ac153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b3200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23ac157f90 0x7f23ac158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b3200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac159b40 con 0x7f23ac157f90 2024-03-29T10:10:28.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b3200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac15f710 con 0x7f23ac0b2b30 2024-03-29T10:10:28.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b3200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac09a7e0 con 0x7f23ac0b3960 2024-03-29T10:10:28.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23ac157f90 0x7f23ac158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:28.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23ab400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b3960 0x7f23ac153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:28.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23ab400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b3960 0x7f23ac153790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51348/0 (socket says 172.21.15.12:51348) 2024-03-29T10:10:28.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23ab400700 1 -- 172.21.15.12:0/2928069105 learned_addr learned my addr 172.21.15.12:0/2928069105 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:28.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23abe00700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23ac0b2b30 0x7f23ac09a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:28.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b1200700 1 -- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b3960 msgr2=0x7f23ac153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b1200700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b3960 0x7f23ac153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b1200700 1 -- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23ac0b2b30 msgr2=0x7f23ac09a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b1200700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23ac0b2b30 0x7f23ac09a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.399+0000 7f23b1200700 1 -- 172.21.15.12:0/2928069105 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a4009b30 con 0x7f23ac157f90 2024-03-29T10:10:28.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.400+0000 7f23b1200700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23ac157f90 0x7f23ac158ec0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f23ac06fd80 tx=0x7f23a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:28.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.400+0000 7f23a3e00700 1 -- 172.21.15.12:0/2928069105 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4003ea0 con 0x7f23ac157f90 2024-03-29T10:10:28.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.400+0000 7f23a3e00700 1 -- 172.21.15.12:0/2928069105 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a40048a0 con 0x7f23ac157f90 2024-03-29T10:10:28.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.400+0000 7f23b3200700 1 -- 172.21.15.12:0/2928069105 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ac159860 con 0x7f23ac157f90 2024-03-29T10:10:28.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.400+0000 7f23a3e00700 1 -- 172.21.15.12:0/2928069105 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a401c680 con 0x7f23ac157f90 2024-03-29T10:10:28.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.400+0000 7f23b3200700 1 -- 172.21.15.12:0/2928069105 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ac15f3a0 con 0x7f23ac157f90 2024-03-29T10:10:28.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.401+0000 7f23b3200700 1 -- 172.21.15.12:0/2928069105 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ac15f8a0 con 0x7f23ac157f90 2024-03-29T10:10:28.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.402+0000 7f23a3e00700 1 -- 172.21.15.12:0/2928069105 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 39) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f23a401c7e0 con 0x7f23ac157f90 2024-03-29T10:10:28.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.403+0000 7f23a3e00700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f238c06ae00 0x7f238c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.403+0000 7f23a3e00700 1 -- 172.21.15.12:0/2928069105 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23a408ae80 con 0x7f23ac157f90 2024-03-29T10:10:28.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.403+0000 7f23abe00700 1 -- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f238c06ae00 msgr2=0x7f238c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:28.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.403+0000 7f23abe00700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f238c06ae00 0x7f238c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:28.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.408+0000 7f23a3e00700 1 -- 172.21.15.12:0/2928069105 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23a4055e30 con 0x7f23ac157f90 2024-03-29T10:10:28.564 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:28.564+0000 7fd5b3b59240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:10:28.601 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:28.601+0000 7fac74ec0240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:10:28.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.604+0000 7f23abe00700 1 -- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f238c06ae00 msgr2=0x7f238c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:28.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.604+0000 7f23abe00700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f238c06ae00 0x7f238c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:28.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.662+0000 7f23b3200700 1 -- 172.21.15.12:0/2928069105 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f23ac067720 con 0x7f23ac157f90 2024-03-29T10:10:28.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.666+0000 7f23a3e00700 1 -- 172.21.15.12:0/2928069105 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f23a4059450 con 0x7f23ac157f90 2024-03-29T10:10:28.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 -- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f238c06ae00 msgr2=0x7f238c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:28.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f238c06ae00 0x7f238c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 -- 172.21.15.12:0/2928069105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23ac157f90 msgr2=0x7f23ac158ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23ac157f90 0x7f23ac158ec0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f23ac06fd80 tx=0x7f23a4004ad0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 -- 172.21.15.12:0/2928069105 shutdown_connections 2024-03-29T10:10:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f238c06ae00 0x7f238c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23ac157f90 0x7f23ac158ec0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23ac0b2b30 0x7f23ac09a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 --2- 172.21.15.12:0/2928069105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23ac0b3960 0x7f23ac153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 -- 172.21.15.12:0/2928069105 >> 172.21.15.12:0/2928069105 conn(0x7f23ac0b0d70 msgr2=0x7f23ac15d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 -- 172.21.15.12:0/2928069105 shutdown_connections 2024-03-29T10:10:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.669+0000 7f23b3200700 1 -- 172.21.15.12:0/2928069105 wait complete. 2024-03-29T10:10:28.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T10:10:28.886 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:28.886+0000 7fd5b3b59240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:10:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.893+0000 7f9220a00700 1 Processor -- start 2024-03-29T10:10:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.893+0000 7f9220a00700 1 -- start start 2024-03-29T10:10:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.894+0000 7f9220a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f921c0cd480 0x7f921c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f9220a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921c0cdae0 con 0x7f921c0cd020 2024-03-29T10:10:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f9220a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921c0cddc0 con 0x7f921c0cbb30 2024-03-29T10:10:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f9220a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f921c0ce0a0 con 0x7f921c0cd480 2024-03-29T10:10:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f9219400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f921c0cd020 0x7f921c0b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39890/0 (socket says 172.21.15.12:39890) 2024-03-29T10:10:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f9219400700 1 -- 172.21.15.12:0/140992738 learned_addr learned my addr 172.21.15.12:0/140992738 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f921a800700 1 --2- 172.21.15.12:0/140992738 >> v2:172.21.15.159:3300/0 conn(0x7f921c0cd480 0x7f921c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f9218a00700 1 -- 172.21.15.12:0/140992738 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957459709 0 0) 0x7f921c0cdae0 con 0x7f921c0cd020 2024-03-29T10:10:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f9218a00700 1 -- 172.21.15.12:0/140992738 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92000035b0 con 0x7f921c0cd020 2024-03-29T10:10:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f9218a00700 1 -- 172.21.15.12:0/140992738 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966045703 0 0) 0x7f921c0cddc0 con 0x7f921c0cbb30 2024-03-29T10:10:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f9218a00700 1 -- 172.21.15.12:0/140992738 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f921c0cdae0 con 0x7f921c0cbb30 2024-03-29T10:10:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f921a800700 1 -- 172.21.15.12:0/140992738 >> v1:172.21.15.159:6789/0 conn(0x7f921c0cbb30 legacy=0x7f921c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f921a800700 1 -- 172.21.15.12:0/140992738 >> v1:172.21.15.12:6789/0 conn(0x7f921c0cd020 legacy=0x7f921c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.895+0000 7f921a800700 1 -- 172.21.15.12:0/140992738 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c06dc20 con 0x7f921c0cd480 2024-03-29T10:10:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.896+0000 7f921a800700 1 --2- 172.21.15.12:0/140992738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f921c0cd480 0x7f921c06e420 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9214009d90 tx=0x7f921400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=283ed94bcb5d1ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.896+0000 7f9218a00700 1 -- 172.21.15.12:0/140992738 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921400e070 con 0x7f921c0cd480 2024-03-29T10:10:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.896+0000 7f9218a00700 1 -- 172.21.15.12:0/140992738 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921400bb20 con 0x7f921c0cd480 2024-03-29T10:10:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.896+0000 7f9218a00700 1 -- 172.21.15.12:0/140992738 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9214012430 con 0x7f921c0cd480 2024-03-29T10:10:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.897+0000 7f9220a00700 1 -- 172.21.15.12:0/140992738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f921c0cd480 msgr2=0x7f921c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.897+0000 7f9220a00700 1 --2- 172.21.15.12:0/140992738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f921c0cd480 0x7f921c06e420 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9214009d90 tx=0x7f921400b330 comp rx=0 tx=0).stop 2024-03-29T10:10:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.897+0000 7f9220a00700 1 -- 172.21.15.12:0/140992738 shutdown_connections 2024-03-29T10:10:28.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.897+0000 7f9220a00700 1 --2- 172.21.15.12:0/140992738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f921c0cd480 0x7f921c06e420 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.897+0000 7f9220a00700 1 -- 172.21.15.12:0/140992738 >> 172.21.15.12:0/140992738 conn(0x7f921c0b0d70 msgr2=0x7f921c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:28.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.897+0000 7f9220a00700 1 -- 172.21.15.12:0/140992738 shutdown_connections 2024-03-29T10:10:28.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.897+0000 7f9220a00700 1 -- 172.21.15.12:0/140992738 wait complete. 2024-03-29T10:10:28.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.899+0000 7f9220a00700 1 Processor -- start 2024-03-29T10:10:28.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.899+0000 7f9220a00700 1 -- start start 2024-03-29T10:10:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f9220a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f921c0cbb30 0x7f921c157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f9220a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921c0cd020 0x7f921c0afe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f9220a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f921c0cd480 0x7f921c0b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f9220a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c0ae860 con 0x7f921c0cbb30 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f9220a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c0cabb0 con 0x7f921c0cd480 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f9220a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921c0aeb00 con 0x7f921c0cd020 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f9219e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f921c0cbb30 0x7f921c157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f9219e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f921c0cbb30 0x7f921c157cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36284/0 (socket says 172.21.15.12:36284) 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f9219e00700 1 -- 172.21.15.12:0/3629285968 learned_addr learned my addr 172.21.15.12:0/3629285968 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f921a800700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f921c0cd480 0x7f921c0b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.900+0000 7f9219400700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921c0cd020 0x7f921c0afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.901+0000 7f921a800700 1 -- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921c0cd020 msgr2=0x7f921c0afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.901+0000 7f921a800700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921c0cd020 0x7f921c0afe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.901+0000 7f921a800700 1 -- 172.21.15.12:0/3629285968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f921c0cbb30 msgr2=0x7f921c157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:28.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.901+0000 7f921a800700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f921c0cbb30 0x7f921c157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:28.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.901+0000 7f921a800700 1 -- 172.21.15.12:0/3629285968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9214009a70 con 0x7f921c0cd480 2024-03-29T10:10:28.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.901+0000 7f9219e00700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f921c0cbb30 0x7f921c157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:28.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.901+0000 7f9219400700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921c0cd020 0x7f921c0afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:28.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.901+0000 7f921a800700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f921c0cd480 0x7f921c0b03a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9214016000 tx=0x7f9214004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:28.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.901+0000 7f9212000700 1 -- 172.21.15.12:0/3629285968 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9214012430 con 0x7f921c0cd480 2024-03-29T10:10:28.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.902+0000 7f9212000700 1 -- 172.21.15.12:0/3629285968 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9214020070 con 0x7f921c0cd480 2024-03-29T10:10:28.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.902+0000 7f9220a00700 1 -- 172.21.15.12:0/3629285968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921c0aede0 con 0x7f921c0cd480 2024-03-29T10:10:28.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.902+0000 7f9212000700 1 -- 172.21.15.12:0/3629285968 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921401ba90 con 0x7f921c0cd480 2024-03-29T10:10:28.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.902+0000 7f9220a00700 1 -- 172.21.15.12:0/3629285968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921c158200 con 0x7f921c0cd480 2024-03-29T10:10:28.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.903+0000 7f9220a00700 1 -- 172.21.15.12:0/3629285968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921c158390 con 0x7f921c0cd480 2024-03-29T10:10:28.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.904+0000 7f9212000700 1 -- 172.21.15.12:0/3629285968 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 39) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f921401bbf0 con 0x7f921c0cd480 2024-03-29T10:10:28.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.905+0000 7f9212000700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91fc06ae00 0x7f91fc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:28.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.905+0000 7f9212000700 1 -- 172.21.15.12:0/3629285968 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9214022070 con 0x7f921c0cd480 2024-03-29T10:10:28.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.905+0000 7f9219e00700 1 -- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91fc06ae00 msgr2=0x7f91fc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:28.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.905+0000 7f9219e00700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91fc06ae00 0x7f91fc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:28.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:28.910+0000 7f9212000700 1 -- 172.21.15.12:0/3629285968 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9214056820 con 0x7f921c0cd480 2024-03-29T10:10:28.944 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:28.945+0000 7fac74ec0240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:10:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.106+0000 7f9219e00700 1 -- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91fc06ae00 msgr2=0x7f91fc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:29.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.106+0000 7f9219e00700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91fc06ae00 0x7f91fc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:29.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.127+0000 7f9220a00700 1 -- 172.21.15.12:0/3629285968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f921c0af300 con 0x7f921c0cd480 2024-03-29T10:10:29.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.133+0000 7f9212000700 1 -- 172.21.15.12:0/3629285968 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f921c0af300 con 0x7f921c0cd480 2024-03-29T10:10:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 -- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91fc06ae00 msgr2=0x7f91fc06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91fc06ae00 0x7f91fc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 -- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f921c0cd480 msgr2=0x7f921c0b03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f921c0cd480 0x7f921c0b03a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9214016000 tx=0x7f9214004ad0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 -- 172.21.15.12:0/3629285968 shutdown_connections 2024-03-29T10:10:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f91fc06ae00 0x7f91fc06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f921c0cd480 0x7f921c0b03a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f921c0cbb30 0x7f921c157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 --2- 172.21.15.12:0/3629285968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921c0cd020 0x7f921c0afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 -- 172.21.15.12:0/3629285968 >> 172.21.15.12:0/3629285968 conn(0x7f921c0b0d70 msgr2=0x7f921c0ce540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 -- 172.21.15.12:0/3629285968 shutdown_connections 2024-03-29T10:10:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.135+0000 7f9220a00700 1 -- 172.21.15.12:0/3629285968 wait complete. 2024-03-29T10:10:29.153 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-03-29T10:10:29.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.358+0000 7efe79a00700 1 Processor -- start 2024-03-29T10:10:29.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.359+0000 7efe79a00700 1 -- start start 2024-03-29T10:10:29.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.360+0000 7efe79a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efe740ced80 0x7efe740d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:29.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.360+0000 7efe79a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7415fcb0 0x7efe74160120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:29.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.360+0000 7efe79a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7415f820 con 0x7efe7415dad0 2024-03-29T10:10:29.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.360+0000 7efe79a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe740cfcc0 con 0x7efe740ced80 2024-03-29T10:10:29.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.360+0000 7efe79a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe740cd9f0 con 0x7efe7415fcb0 2024-03-29T10:10:29.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.361+0000 7efe72000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efe740ced80 0x7efe740d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:29.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.361+0000 7efe73400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7415fcb0 0x7efe74160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:29.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.361+0000 7efe72000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efe740ced80 0x7efe740d1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:48132/0 (socket says 172.21.15.12:48132) 2024-03-29T10:10:29.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.361+0000 7efe72000700 1 -- 172.21.15.12:0/438961718 learned_addr learned my addr 172.21.15.12:0/438961718 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:29.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.361+0000 7efe72000700 1 -- 172.21.15.12:0/438961718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7415fcb0 msgr2=0x7efe74160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:29.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.361+0000 7efe72000700 1 --2- 172.21.15.12:0/438961718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7415fcb0 0x7efe74160120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.361+0000 7efe72000700 1 -- 172.21.15.12:0/438961718 >> v1:172.21.15.159:6789/0 conn(0x7efe7415dad0 legacy=0x7efe740d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:29.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.361+0000 7efe72000700 1 -- 172.21.15.12:0/438961718 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe740cf420 con 0x7efe740ced80 2024-03-29T10:10:29.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.362+0000 7efe72000700 1 --2- 172.21.15.12:0/438961718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe740ced80 0x7efe740d1d40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efe600098f0 tx=0x7efe60009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=829fe8d0965e4f4d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:29.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.362+0000 7efe71600700 1 -- 172.21.15.12:0/438961718 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60011830 con 0x7efe740ced80 2024-03-29T10:10:29.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.362+0000 7efe71600700 1 -- 172.21.15.12:0/438961718 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60011990 con 0x7efe740ced80 2024-03-29T10:10:29.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.362+0000 7efe71600700 1 -- 172.21.15.12:0/438961718 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60011d20 con 0x7efe740ced80 2024-03-29T10:10:29.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.362+0000 7efe79a00700 1 -- 172.21.15.12:0/438961718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe740ced80 msgr2=0x7efe740d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:29.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.362+0000 7efe79a00700 1 --2- 172.21.15.12:0/438961718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe740ced80 0x7efe740d1d40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efe600098f0 tx=0x7efe60009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:29.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.363+0000 7efe79a00700 1 -- 172.21.15.12:0/438961718 shutdown_connections 2024-03-29T10:10:29.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.363+0000 7efe79a00700 1 --2- 172.21.15.12:0/438961718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7415fcb0 0x7efe74160120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.363+0000 7efe79a00700 1 --2- 172.21.15.12:0/438961718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe740ced80 0x7efe740d1d40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.363+0000 7efe79a00700 1 -- 172.21.15.12:0/438961718 >> 172.21.15.12:0/438961718 conn(0x7efe7408aff0 msgr2=0x7efe740d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:29.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.363+0000 7efe79a00700 1 -- 172.21.15.12:0/438961718 shutdown_connections 2024-03-29T10:10:29.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.363+0000 7efe79a00700 1 -- 172.21.15.12:0/438961718 wait complete. 2024-03-29T10:10:29.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.365+0000 7efe79a00700 1 Processor -- start 2024-03-29T10:10:29.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.365+0000 7efe79a00700 1 -- start start 2024-03-29T10:10:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.366+0000 7efe79a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe740ced80 0x7efe74150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.366+0000 7efe79a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7415dad0 0x7efe740a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.366+0000 7efe79a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe7415fcb0 0x7efe7406ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe79a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740c9950 con 0x7efe7415fcb0 2024-03-29T10:10:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe79a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740c8ff0 con 0x7efe740ced80 2024-03-29T10:10:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe79a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe740c9290 con 0x7efe7415dad0 2024-03-29T10:10:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe73400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe7415fcb0 0x7efe7406ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe72000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7415dad0 0x7efe740a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe72a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe740ced80 0x7efe74150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe72000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7415dad0 0x7efe740a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:34456/0 (socket says 172.21.15.12:34456) 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe72000700 1 -- 172.21.15.12:0/3302413519 learned_addr learned my addr 172.21.15.12:0/3302413519 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe72000700 1 -- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe740ced80 msgr2=0x7efe74150990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe72000700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe740ced80 0x7efe74150990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe72000700 1 -- 172.21.15.12:0/3302413519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe7415fcb0 msgr2=0x7efe7406ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe72000700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe7415fcb0 0x7efe7406ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe72000700 1 -- 172.21.15.12:0/3302413519 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe60009550 con 0x7efe7415dad0 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe72a00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe740ced80 0x7efe74150990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.367+0000 7efe73400700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe7415fcb0 0x7efe7406ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:29.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.368+0000 7efe72000700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7415dad0 0x7efe740a0850 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efe60000c00 tx=0x7efe60004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:29.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.368+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60004400 con 0x7efe7415dad0 2024-03-29T10:10:29.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.368+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6001f070 con 0x7efe7415dad0 2024-03-29T10:10:29.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.368+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe600038e0 con 0x7efe7415dad0 2024-03-29T10:10:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.368+0000 7efe79a00700 1 -- 172.21.15.12:0/3302413519 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe740c9570 con 0x7efe7415dad0 2024-03-29T10:10:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.368+0000 7efe79a00700 1 -- 172.21.15.12:0/3302413519 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe74150ed0 con 0x7efe7415dad0 2024-03-29T10:10:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.369+0000 7efe79a00700 1 -- 172.21.15.12:0/3302413519 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe740c9a90 con 0x7efe7415dad0 2024-03-29T10:10:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.370+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 39) v1 ==== 89256+0+0 (secure 0 0 0) 0x7efe60021030 con 0x7efe7415dad0 2024-03-29T10:10:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.371+0000 7efe6aa00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 0x7efe5406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.371+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe6008a890 con 0x7efe7415dad0 2024-03-29T10:10:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.371+0000 7efe72a00700 1 -- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 msgr2=0x7efe5406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.371+0000 7efe72a00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 0x7efe5406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.376+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efe740c9a90 con 0x7efe7415dad0 2024-03-29T10:10:29.422 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:29.422+0000 7fd5b3b59240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:10:29.510 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:29.510+0000 7fac74ec0240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:10:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.572+0000 7efe72a00700 1 -- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 msgr2=0x7efe5406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.572+0000 7efe72a00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 0x7efe5406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:29.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.624+0000 7efe79a00700 1 -- 172.21.15.12:0/3302413519 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7efe740cf9e0 con 0x7efe5406ae00 2024-03-29T10:10:29.678 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:29.678+0000 7fd5b3b59240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:10:29.775 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:29.775+0000 7fac74ec0240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:10:29.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.973+0000 7efe72a00700 1 -- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 msgr2=0x7efe5406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:29.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:29.973+0000 7efe72a00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 0x7efe5406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T10:10:30.137 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:30.137+0000 7fd5b3b59240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:10:30.250 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:30.251+0000 7fac74ec0240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:10:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:30.774+0000 7efe72a00700 1 -- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 msgr2=0x7efe5406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:30.774+0000 7efe72a00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 0x7efe5406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T10:10:30.842 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:30.842+0000 7fd5b3b59240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:10:30.932 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:30.932+0000 7fd5b3b59240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:10:30.991 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:30.991+0000 7fac74ec0240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:10:31.087 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:31.087+0000 7fac74ec0240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:10:31.089 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:31.089+0000 7fd5b3b59240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:10:31.254 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:31.255+0000 7fac74ec0240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:10:31.302 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:31.302+0000 7fd5b3b59240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:10:31.392 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:31.392+0000 7fd5b3b59240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:10:31.472 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:31.472+0000 7fac74ec0240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:10:31.489 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:31.489+0000 7fd5b3b59240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:10:31.567 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:31.567+0000 7fac74ec0240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:10:31.587 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:31.587+0000 7fd5b3b59240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:10:31.670 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:31.671+0000 7fac74ec0240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:10:31.686 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:31.686+0000 7fd5b3b59240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:10:31.772 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:31.773+0000 7fac74ec0240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:10:31.875 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:31.875+0000 7fac74ec0240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:10:31.897 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:31.897+0000 7fd5b3b59240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:10:32.096 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:32.096+0000 7fac74ec0240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:10:32.157 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:32.154+0000 7fd5b3b59240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:10:32.343 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:32.343+0000 7fd5b3b59240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:10:32.360 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:32.359+0000 7fac74ec0240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:10:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:32.376+0000 7efe72a00700 1 -- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 msgr2=0x7efe5406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:32.376+0000 7efe72a00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 0x7efe5406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T10:10:32.437 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:32.436+0000 7fd5b3b59240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:32.562 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:32.557+0000 7fac74ec0240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:10:32.655 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:32.655+0000 7fac74ec0240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:33.169 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:33.169+0000 7fd5b3b59240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:10:33.379 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:33.379+0000 7fd5b3b59240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:10:33.419 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:33.419+0000 7fac74ec0240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:10:33.586 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:33.586+0000 7fd5b3b59240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:33.642 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:33.643+0000 7fac74ec0240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:10:33.861 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:33.861+0000 7fac74ec0240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:34.063 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:34.063+0000 7fd5b3b59240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:10:34.269 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:34.269+0000 7fd5b3b59240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:34.362 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:34.362+0000 7fac74ec0240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:10:34.577 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:34.578+0000 7fac74ec0240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:34.711 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:34.711+0000 7fd5b3b59240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:10:34.821 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:34.821+0000 7fd5b3b59240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:34.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:34.834+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mgrmap(e 40) v1 ==== 88865+0+0 (secure 0 0 0) 0x7efe60055e60 con 0x7efe7415dad0 2024-03-29T10:10:34.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:34.834+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 msgr2=0x7efe5406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:34.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:34.834+0000 7efe6aa00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 0x7efe5406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:35.014 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.013+0000 7fd51c200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:35.014 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.013+0000 7fd51c200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:35.014 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.013+0000 7fd51c200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:35.014 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.013+0000 7fd51c200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:35.014 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.013+0000 7fd51c200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:35.014 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.013+0000 7fd51fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:35.014 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.013+0000 7fd51fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:35.015 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.013+0000 7fd51fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:35.015 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.013+0000 7fd51fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:35.015 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.013+0000 7fd51fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:35.040 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:35.040+0000 7fac74ec0240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:10:35.157 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:35.158+0000 7fac74ec0240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:35.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:35.839+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 <== mon.2 v2:172.21.15.159:3301/0 8 ==== mgrmap(e 41) v1 ==== 88992+0+0 (secure 0 0 0) 0x7efe60088d80 con 0x7efe7415dad0 2024-03-29T10:10:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:35.839+0000 7efe6aa00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 0x7efe5406d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:35.839+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7efe540038f0 con 0x7efe5406ae00 2024-03-29T10:10:35.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:35.841+0000 7efe72a00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 0x7efe5406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:35.843+0000 7efe72a00700 1 --2- 172.21.15.12:0/3302413519 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe5406ae00 0x7efe5406d240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efe64007fd0 tx=0x7efe64007ec0 comp rx=0 tx=0).ready entity=mgr.6850 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:35.854 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:35.854+0000 7fd528a00700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-03-29T10:10:35.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:35.855+0000 7efe6aa00700 1 -- 172.21.15.12:0/3302413519 <== mgr.6850 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7efe64001b20 con 0x7efe5406ae00 2024-03-29T10:10:35.857 INFO:teuthology.orchestra.run.smithi012.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-03-29T10:10:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-29T10:10:35.866 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T10:10:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.071+0000 7f92aac00700 1 Processor -- start 2024-03-29T10:10:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.072+0000 7f92aac00700 1 -- start start 2024-03-29T10:10:36.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.073+0000 7f92aac00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f92a40cd020 0x7f92a40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:36.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.073+0000 7f92aac00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a40cdae0 con 0x7f92a40b2070 2024-03-29T10:10:36.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.073+0000 7f92aac00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a40cddc0 con 0x7f92a40cbb30 2024-03-29T10:10:36.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.073+0000 7f92aac00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a40ce0a0 con 0x7f92a40cd020 2024-03-29T10:10:36.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.073+0000 7f92a8c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f92a40b2070 0x7f92a406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39902/0 (socket says 172.21.15.12:39902) 2024-03-29T10:10:36.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.073+0000 7f92a8c00700 1 -- 172.21.15.12:0/807920731 learned_addr learned my addr 172.21.15.12:0/807920731 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:36.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.073+0000 7f92a3400700 1 --2- 172.21.15.12:0/807920731 >> v2:172.21.15.159:3300/0 conn(0x7f92a40cd020 0x7f92a40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:36.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.074+0000 7f92a2a00700 1 -- 172.21.15.12:0/807920731 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785676279 0 0) 0x7f92a40cdae0 con 0x7f92a40b2070 2024-03-29T10:10:36.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.074+0000 7f92a2a00700 1 -- 172.21.15.12:0/807920731 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92880035b0 con 0x7f92a40b2070 2024-03-29T10:10:36.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.074+0000 7f92a2a00700 1 -- 172.21.15.12:0/807920731 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297009285 0 0) 0x7f92a40cddc0 con 0x7f92a40cbb30 2024-03-29T10:10:36.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.074+0000 7f92a2a00700 1 -- 172.21.15.12:0/807920731 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a40cdae0 con 0x7f92a40cbb30 2024-03-29T10:10:36.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.074+0000 7f92a3400700 1 -- 172.21.15.12:0/807920731 >> v1:172.21.15.159:6789/0 conn(0x7f92a40cbb30 legacy=0x7f92a40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:36.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.074+0000 7f92a3400700 1 -- 172.21.15.12:0/807920731 >> v1:172.21.15.12:6789/0 conn(0x7f92a40b2070 legacy=0x7f92a406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:36.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.074+0000 7f92a2a00700 1 -- 172.21.15.12:0/807920731 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2228103153 0 0) 0x7f92880035b0 con 0x7f92a40b2070 2024-03-29T10:10:36.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.074+0000 7f92a3400700 1 -- 172.21.15.12:0/807920731 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a406dc20 con 0x7f92a40cd020 2024-03-29T10:10:36.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.074+0000 7f92a3400700 1 --2- 172.21.15.12:0/807920731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92a40cd020 0x7f92a40cd480 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9290009840 tx=0x7f9290009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91f1d753711a59e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.074+0000 7f92a2a00700 1 -- 172.21.15.12:0/807920731 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9290011830 con 0x7f92a40cd020 2024-03-29T10:10:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.075+0000 7f92a2a00700 1 -- 172.21.15.12:0/807920731 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9290011990 con 0x7f92a40cd020 2024-03-29T10:10:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.075+0000 7f92a2a00700 1 -- 172.21.15.12:0/807920731 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9290011d00 con 0x7f92a40cd020 2024-03-29T10:10:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.075+0000 7f92aac00700 1 -- 172.21.15.12:0/807920731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92a40cd020 msgr2=0x7f92a40cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.075+0000 7f92aac00700 1 --2- 172.21.15.12:0/807920731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92a40cd020 0x7f92a40cd480 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9290009840 tx=0x7f9290009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.075+0000 7f92aac00700 1 -- 172.21.15.12:0/807920731 shutdown_connections 2024-03-29T10:10:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.075+0000 7f92aac00700 1 --2- 172.21.15.12:0/807920731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92a40cd020 0x7f92a40cd480 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.075+0000 7f92aac00700 1 -- 172.21.15.12:0/807920731 >> 172.21.15.12:0/807920731 conn(0x7f92a40b0d70 msgr2=0x7f92a40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.075+0000 7f92aac00700 1 -- 172.21.15.12:0/807920731 shutdown_connections 2024-03-29T10:10:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.075+0000 7f92aac00700 1 -- 172.21.15.12:0/807920731 wait complete. 2024-03-29T10:10:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.077+0000 7f92aac00700 1 Processor -- start 2024-03-29T10:10:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.077+0000 7f92aac00700 1 -- start start 2024-03-29T10:10:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92aac00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92a40b2070 0x7f92a4151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92aac00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92a40cbb30 0x7f92a4157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92aac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92a40cd020 0x7f92a40ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92aac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a41f2f20 con 0x7f92a40cd020 2024-03-29T10:10:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92aac00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a41f31e0 con 0x7f92a40cbb30 2024-03-29T10:10:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92aac00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a41f34c0 con 0x7f92a40b2070 2024-03-29T10:10:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92a8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92a40cd020 0x7f92a40ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92a8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92a40cd020 0x7f92a40ad0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36308/0 (socket says 172.21.15.12:36308) 2024-03-29T10:10:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92a8c00700 1 -- 172.21.15.12:0/4173942046 learned_addr learned my addr 172.21.15.12:0/4173942046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92a3400700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92a40cbb30 0x7f92a4157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.078+0000 7f92a3e00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92a40b2070 0x7f92a4151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f92a8c00700 1 -- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92a40b2070 msgr2=0x7f92a4151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f92a8c00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92a40b2070 0x7f92a4151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f92a8c00700 1 -- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92a40cbb30 msgr2=0x7f92a4157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f92a8c00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92a40cbb30 0x7f92a4157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f92a8c00700 1 -- 172.21.15.12:0/4173942046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92900094a0 con 0x7f92a40cd020 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f92a8c00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92a40cd020 0x7f92a40ad0a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f929c00cb10 tx=0x7f929c00ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f929be00700 1 -- 172.21.15.12:0/4173942046 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929c00a9e0 con 0x7f92a40cd020 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f929be00700 1 -- 172.21.15.12:0/4173942046 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929c00ab40 con 0x7f92a40cd020 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f92aac00700 1 -- 172.21.15.12:0/4173942046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a41f37a0 con 0x7f92a40cd020 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f929be00700 1 -- 172.21.15.12:0/4173942046 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929c012400 con 0x7f92a40cd020 2024-03-29T10:10:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.079+0000 7f92aac00700 1 -- 172.21.15.12:0/4173942046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a4151f40 con 0x7f92a40cd020 2024-03-29T10:10:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.080+0000 7f92aac00700 1 -- 172.21.15.12:0/4173942046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a41f3cc0 con 0x7f92a40cd020 2024-03-29T10:10:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.081+0000 7f929be00700 1 -- 172.21.15.12:0/4173942046 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 41) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f929c012560 con 0x7f92a40cd020 2024-03-29T10:10:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.083+0000 7f929be00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f928406ab30 0x7f928406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.083+0000 7f929be00700 1 -- 172.21.15.12:0/4173942046 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f929c089380 con 0x7f92a40cd020 2024-03-29T10:10:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.083+0000 7f92a3e00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f928406ab30 0x7f928406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.084+0000 7f92a3e00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f928406ab30 0x7f928406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f92940011e0 tx=0x7f9294009040 comp rx=0 tx=0).ready entity=mgr.6850 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:36.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.088+0000 7f929be00700 1 -- 172.21.15.12:0/4173942046 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f929c054740 con 0x7f92a40cd020 2024-03-29T10:10:36.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.366+0000 7f92aac00700 1 -- 172.21.15.12:0/4173942046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f92a4067720 con 0x7f92a40cd020 2024-03-29T10:10:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.887+0000 7f929be00700 1 -- 172.21.15.12:0/4173942046 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f929c0890d0 con 0x7f92a40cd020 2024-03-29T10:10:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.888+0000 7f929be00700 1 -- 172.21.15.12:0/4173942046 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v42) v1 ==== 93+0+0 (secure 0 0 0) 0x7f929c0723a0 con 0x7f92a40cd020 2024-03-29T10:10:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 -- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f928406ab30 msgr2=0x7f928406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f928406ab30 0x7f928406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f92940011e0 tx=0x7f9294009040 comp rx=0 tx=0).stop 2024-03-29T10:10:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 -- 172.21.15.12:0/4173942046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92a40cd020 msgr2=0x7f92a40ad0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92a40cd020 0x7f92a40ad0a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f929c00cb10 tx=0x7f929c00ce20 comp rx=0 tx=0).stop 2024-03-29T10:10:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 -- 172.21.15.12:0/4173942046 shutdown_connections 2024-03-29T10:10:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f928406ab30 0x7f928406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92a40cd020 0x7f92a40ad0a0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92a40b2070 0x7f92a4151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 --2- 172.21.15.12:0/4173942046 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92a40cbb30 0x7f92a4157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 -- 172.21.15.12:0/4173942046 >> 172.21.15.12:0/4173942046 conn(0x7f92a40b0d70 msgr2=0x7f92a40af560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 -- 172.21.15.12:0/4173942046 shutdown_connections 2024-03-29T10:10:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:36.891+0000 7f92aac00700 1 -- 172.21.15.12:0/4173942046 wait complete. 2024-03-29T10:10:36.913 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:10:36.913 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:10:37.001 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:10:37.084 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:10:37.085 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:10:37.085 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:37.112 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:37.113+0000 7f8ecce46240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:10:37.114 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:37.114+0000 7f3fb2c40240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:10:37.130 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:10:37.130 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:37.216 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:37.298 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:37.299+0000 7f3fb2c40240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:10:37.321 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:37.322+0000 7f8ecce46240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:10:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.492+0000 7feaeb200700 1 Processor -- start 2024-03-29T10:10:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.492+0000 7feaeb200700 1 -- start start 2024-03-29T10:10:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.494+0000 7feaeb200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feae409a910 0x7feae406c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.494+0000 7feaeb200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 0x7feae4099e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.494+0000 7feaeb200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae40d08d0 con 0x7feae415d920 2024-03-29T10:10:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.494+0000 7feaeb200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae406c300 con 0x7feae409a910 2024-03-29T10:10:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.494+0000 7feaeb200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae409a3d0 con 0x7feae4099a60 2024-03-29T10:10:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.494+0000 7feae3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feae409a910 0x7feae406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.494+0000 7feae3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feae409a910 0x7feae406c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36338/0 (socket says 172.21.15.12:36338) 2024-03-29T10:10:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.494+0000 7feae9200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 0x7feae4099e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.494+0000 7feae3400700 1 -- 172.21.15.12:0/2202383327 learned_addr learned my addr 172.21.15.12:0/2202383327 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.495+0000 7feae3400700 1 -- 172.21.15.12:0/2202383327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 msgr2=0x7feae4099e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.495+0000 7feae3400700 1 --2- 172.21.15.12:0/2202383327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 0x7feae4099e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.495+0000 7feae3400700 1 -- 172.21.15.12:0/2202383327 >> v1:172.21.15.159:6789/0 conn(0x7feae415d920 legacy=0x7feae40cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.495+0000 7feae2a00700 1 -- 172.21.15.12:0/2202383327 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53156624 0 0) 0x7feae40d08d0 con 0x7feae415d920 2024-03-29T10:10:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.495+0000 7feae3400700 1 -- 172.21.15.12:0/2202383327 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae406b9d0 con 0x7feae409a910 2024-03-29T10:10:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.495+0000 7feae3400700 1 --2- 172.21.15.12:0/2202383327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae409a910 0x7feae406c790 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fead00098f0 tx=0x7fead0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1bbcb147941fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.495+0000 7feae2a00700 1 -- 172.21.15.12:0/2202383327 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0011830 con 0x7feae409a910 2024-03-29T10:10:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.495+0000 7feae2a00700 1 -- 172.21.15.12:0/2202383327 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead0011990 con 0x7feae409a910 2024-03-29T10:10:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.496+0000 7feae2a00700 1 -- 172.21.15.12:0/2202383327 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0011d20 con 0x7feae409a910 2024-03-29T10:10:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.496+0000 7feaeb200700 1 -- 172.21.15.12:0/2202383327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae409a910 msgr2=0x7feae406c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.496+0000 7feaeb200700 1 --2- 172.21.15.12:0/2202383327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae409a910 0x7feae406c790 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fead00098f0 tx=0x7fead0009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.496+0000 7feaeb200700 1 -- 172.21.15.12:0/2202383327 shutdown_connections 2024-03-29T10:10:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.496+0000 7feaeb200700 1 --2- 172.21.15.12:0/2202383327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 0x7feae4099e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.496+0000 7feaeb200700 1 --2- 172.21.15.12:0/2202383327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae409a910 0x7feae406c790 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.496+0000 7feaeb200700 1 -- 172.21.15.12:0/2202383327 >> 172.21.15.12:0/2202383327 conn(0x7feae40d1350 msgr2=0x7feae40d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.496+0000 7feaeb200700 1 -- 172.21.15.12:0/2202383327 shutdown_connections 2024-03-29T10:10:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.496+0000 7feaeb200700 1 -- 172.21.15.12:0/2202383327 wait complete. 2024-03-29T10:10:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.498+0000 7feaeb200700 1 Processor -- start 2024-03-29T10:10:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.498+0000 7feaeb200700 1 -- start start 2024-03-29T10:10:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feaeb200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 0x7feae41f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feaeb200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feae409a910 0x7feae40b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feaeb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae415d920 0x7feae40b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feaeb200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae415f6e0 con 0x7feae415d920 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feaeb200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae40cb3a0 con 0x7feae409a910 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feaeb200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae415ea60 con 0x7feae4099a60 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feae3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 0x7feae41f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feae9200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae415d920 0x7feae40b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feae3400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feae409a910 0x7feae40b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feae3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 0x7feae41f5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:34498/0 (socket says 172.21.15.12:34498) 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.499+0000 7feae3e00700 1 -- 172.21.15.12:0/1801204063 learned_addr learned my addr 172.21.15.12:0/1801204063 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feae3e00700 1 -- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feae409a910 msgr2=0x7feae40b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feae3e00700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feae409a910 0x7feae40b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feae3e00700 1 -- 172.21.15.12:0/1801204063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae415d920 msgr2=0x7feae40b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feae3e00700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae415d920 0x7feae40b5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feae3e00700 1 -- 172.21.15.12:0/1801204063 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead0009550 con 0x7feae4099a60 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feae9200700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae415d920 0x7feae40b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feae3e00700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 0x7feae41f5df0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fead400c580 tx=0x7fead400c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feadbe00700 1 -- 172.21.15.12:0/1801204063 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead40054e0 con 0x7feae4099a60 2024-03-29T10:10:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feadbe00700 1 -- 172.21.15.12:0/1801204063 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead4003680 con 0x7feae4099a60 2024-03-29T10:10:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feaeb200700 1 -- 172.21.15.12:0/1801204063 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae415ed40 con 0x7feae4099a60 2024-03-29T10:10:37.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.500+0000 7feadbe00700 1 -- 172.21.15.12:0/1801204063 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead4003a30 con 0x7feae4099a60 2024-03-29T10:10:37.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.501+0000 7feaeb200700 1 -- 172.21.15.12:0/1801204063 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae415f300 con 0x7feae4099a60 2024-03-29T10:10:37.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.501+0000 7feaeb200700 1 -- 172.21.15.12:0/1801204063 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae4067720 con 0x7feae4099a60 2024-03-29T10:10:37.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.502+0000 7feadbe00700 1 -- 172.21.15.12:0/1801204063 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fead4003b90 con 0x7feae4099a60 2024-03-29T10:10:37.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.503+0000 7feadbe00700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac406adb0 0x7feac406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.503+0000 7feadbe00700 1 -- 172.21.15.12:0/1801204063 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fead40889c0 con 0x7feae4099a60 2024-03-29T10:10:37.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.503+0000 7feae3400700 1 -- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac406adb0 msgr2=0x7feac406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:37.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.503+0000 7feae3400700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac406adb0 0x7feac406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:37.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.508+0000 7feadbe00700 1 -- 172.21.15.12:0/1801204063 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fead4053890 con 0x7feae4099a60 2024-03-29T10:10:37.619 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:37.619+0000 7f3fb2c40240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:10:37.670 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:37.670+0000 7f8ecce46240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:10:37.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.704+0000 7feae3400700 1 -- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac406adb0 msgr2=0x7feac406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:37.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.704+0000 7feae3400700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac406adb0 0x7feac406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.722+0000 7feaeb200700 1 -- 172.21.15.12:0/1801204063 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feae40d0de0 con 0x7feae4099a60 2024-03-29T10:10:37.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.724+0000 7feadbe00700 1 -- 172.21.15.12:0/1801204063 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fead4017070 con 0x7feae4099a60 2024-03-29T10:10:37.724 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:37.724 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":95,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:34.825237+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:16.040572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:16.605686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/706246125":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/2287631103":"2024-03-30T10:10:34.825193+0000","172.21.15.159:6833/43444":"2024-03-30T10:10:34.825193+0000","172.21.15.159:6832/43444":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/356119888":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/3972721365":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/1459270455":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/2421061145":"2024-03-30T10:10:34.825193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 -- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac406adb0 msgr2=0x7feac406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac406adb0 0x7feac406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 -- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 msgr2=0x7feae41f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 0x7feae41f5df0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fead400c580 tx=0x7fead400c890 comp rx=0 tx=0).stop 2024-03-29T10:10:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 -- 172.21.15.12:0/1801204063 shutdown_connections 2024-03-29T10:10:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feac406adb0 0x7feac406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feae415d920 0x7feae40b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feae4099a60 0x7feae41f5df0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 --2- 172.21.15.12:0/1801204063 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feae409a910 0x7feae40b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 -- 172.21.15.12:0/1801204063 >> 172.21.15.12:0/1801204063 conn(0x7feae40d1350 msgr2=0x7feae406ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 -- 172.21.15.12:0/1801204063 shutdown_connections 2024-03-29T10:10:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.726+0000 7feaeb200700 1 -- 172.21.15.12:0/1801204063 wait complete. 2024-03-29T10:10:37.745 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:10:16.040572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:10:16.605686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:10:37.745 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:10:37.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.953+0000 7f916f000700 1 Processor -- start 2024-03-29T10:10:37.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.953+0000 7f916f000700 1 -- start start 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f916f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b3b30 0x7f91680cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f916f000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91680b2070 con 0x7f91680b2b30 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f916f000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916815e0f0 con 0x7f9168157f90 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f916f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9168158370 con 0x7f91680b3b30 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f9167400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f91680b2b30 0x7f91680b3420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39908/0 (socket says 172.21.15.12:39908) 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f9167400700 1 -- 172.21.15.12:0/3646664324 learned_addr learned my addr 172.21.15.12:0/3646664324 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f916d000700 1 --2- 172.21.15.12:0/3646664324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b3b30 0x7f91680cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f9166a00700 1 -- 172.21.15.12:0/3646664324 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718219415 0 0) 0x7f91680b2070 con 0x7f91680b2b30 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f9166a00700 1 -- 172.21.15.12:0/3646664324 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f914c0035b0 con 0x7f91680b2b30 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f9166a00700 1 -- 172.21.15.12:0/3646664324 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821074195 0 0) 0x7f916815e0f0 con 0x7f9168157f90 2024-03-29T10:10:37.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.955+0000 7f9166a00700 1 -- 172.21.15.12:0/3646664324 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91680b2070 con 0x7f9168157f90 2024-03-29T10:10:37.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.956+0000 7f916d000700 1 -- 172.21.15.12:0/3646664324 >> v1:172.21.15.159:6789/0 conn(0x7f9168157f90 legacy=0x7f91681613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:37.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.956+0000 7f916d000700 1 -- 172.21.15.12:0/3646664324 >> v1:172.21.15.12:6789/0 conn(0x7f91680b2b30 legacy=0x7f91680b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:37.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.956+0000 7f916d000700 1 -- 172.21.15.12:0/3646664324 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91680b2fe0 con 0x7f91680b3b30 2024-03-29T10:10:37.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.956+0000 7f916d000700 1 --2- 172.21.15.12:0/3646664324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b3b30 0x7f91680cd020 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9160009dc0 tx=0x7f916000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b38865b0b182759 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:37.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.956+0000 7f9166a00700 1 -- 172.21.15.12:0/3646664324 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000e070 con 0x7f91680b3b30 2024-03-29T10:10:37.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.956+0000 7f9166a00700 1 -- 172.21.15.12:0/3646664324 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916000bb20 con 0x7f91680b3b30 2024-03-29T10:10:37.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.956+0000 7f9166a00700 1 -- 172.21.15.12:0/3646664324 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160012430 con 0x7f91680b3b30 2024-03-29T10:10:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.957+0000 7f916f000700 1 -- 172.21.15.12:0/3646664324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b3b30 msgr2=0x7f91680cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.957+0000 7f916f000700 1 --2- 172.21.15.12:0/3646664324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b3b30 0x7f91680cd020 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9160009dc0 tx=0x7f916000b330 comp rx=0 tx=0).stop 2024-03-29T10:10:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.957+0000 7f916f000700 1 -- 172.21.15.12:0/3646664324 shutdown_connections 2024-03-29T10:10:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.957+0000 7f916f000700 1 --2- 172.21.15.12:0/3646664324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b3b30 0x7f91680cd020 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.957+0000 7f916f000700 1 -- 172.21.15.12:0/3646664324 >> 172.21.15.12:0/3646664324 conn(0x7f91680b0d70 msgr2=0x7f91680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.957+0000 7f916f000700 1 -- 172.21.15.12:0/3646664324 shutdown_connections 2024-03-29T10:10:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.957+0000 7f916f000700 1 -- 172.21.15.12:0/3646664324 wait complete. 2024-03-29T10:10:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.959+0000 7f916f000700 1 Processor -- start 2024-03-29T10:10:37.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.959+0000 7f916f000700 1 -- start start 2024-03-29T10:10:37.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f916f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b2b30 0x7f91681595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f916f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91680b3b30 0x7f9168149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f916f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9168157f90 0x7f91680b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f916f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91680b2710 con 0x7f91680b3b30 2024-03-29T10:10:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f916f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91681591d0 con 0x7f9168157f90 2024-03-29T10:10:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f916f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9168158550 con 0x7f91680b2b30 2024-03-29T10:10:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f9167400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91680b3b30 0x7f9168149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f9167400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91680b3b30 0x7f9168149430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36344/0 (socket says 172.21.15.12:36344) 2024-03-29T10:10:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f9167400700 1 -- 172.21.15.12:0/1732065758 learned_addr learned my addr 172.21.15.12:0/1732065758 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f9167e00700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b2b30 0x7f91681595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.960+0000 7f916d000700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9168157f90 0x7f91680b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f9167400700 1 -- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b2b30 msgr2=0x7f91681595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f9167400700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b2b30 0x7f91681595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f9167400700 1 -- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9168157f90 msgr2=0x7f91680b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f9167400700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9168157f90 0x7f91680b1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f9167400700 1 -- 172.21.15.12:0/1732065758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9160009a70 con 0x7f91680b3b30 2024-03-29T10:10:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f916d000700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9168157f90 0x7f91680b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f9167400700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91680b3b30 0x7f9168149430 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f9154001eb0 tx=0x7f915400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f915fe00700 1 -- 172.21.15.12:0/1732065758 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154005d70 con 0x7f91680b3b30 2024-03-29T10:10:37.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f915fe00700 1 -- 172.21.15.12:0/1732065758 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91540070d0 con 0x7f91680b3b30 2024-03-29T10:10:37.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f916f000700 1 -- 172.21.15.12:0/1732065758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9168158830 con 0x7f91680b3b30 2024-03-29T10:10:37.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f915fe00700 1 -- 172.21.15.12:0/1732065758 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154007440 con 0x7f91680b3b30 2024-03-29T10:10:37.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.961+0000 7f916f000700 1 -- 172.21.15.12:0/1732065758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9168158df0 con 0x7f91680b3b30 2024-03-29T10:10:37.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.962+0000 7f916f000700 1 -- 172.21.15.12:0/1732065758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9168159310 con 0x7f91680b3b30 2024-03-29T10:10:37.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.963+0000 7f915fe00700 1 -- 172.21.15.12:0/1732065758 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f91540075a0 con 0x7f91680b3b30 2024-03-29T10:10:37.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.964+0000 7f915fe00700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f914806adb0 0x7f914806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:37.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.964+0000 7f915fe00700 1 -- 172.21.15.12:0/1732065758 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f9154088920 con 0x7f91680b3b30 2024-03-29T10:10:37.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.964+0000 7f9167e00700 1 -- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f914806adb0 msgr2=0x7f914806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:37.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.964+0000 7f9167e00700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f914806adb0 0x7f914806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:37.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:37.969+0000 7f915fe00700 1 -- 172.21.15.12:0/1732065758 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9168159310 con 0x7f91680b3b30 2024-03-29T10:10:38.155 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:38.155+0000 7f3fb2c40240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:10:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.165+0000 7f9167e00700 1 -- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f914806adb0 msgr2=0x7f914806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.165+0000 7f9167e00700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f914806adb0 0x7f914806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.182+0000 7f916f000700 1 -- 172.21.15.12:0/1732065758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9168159310 con 0x7f91680b3b30 2024-03-29T10:10:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.183+0000 7f915fe00700 1 -- 172.21.15.12:0/1732065758 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9168159310 con 0x7f91680b3b30 2024-03-29T10:10:38.183 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:10:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 -- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f914806adb0 msgr2=0x7f914806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f914806adb0 0x7f914806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 -- 172.21.15.12:0/1732065758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91680b3b30 msgr2=0x7f9168149430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91680b3b30 0x7f9168149430 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f9154001eb0 tx=0x7f915400c7b0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 -- 172.21.15.12:0/1732065758 shutdown_connections 2024-03-29T10:10:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f914806adb0 0x7f914806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9168157f90 0x7f91680b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91680b2b30 0x7f91681595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 --2- 172.21.15.12:0/1732065758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91680b3b30 0x7f9168149430 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 -- 172.21.15.12:0/1732065758 >> 172.21.15.12:0/1732065758 conn(0x7f91680b0d70 msgr2=0x7f916815ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.185+0000 7f916f000700 1 -- 172.21.15.12:0/1732065758 shutdown_connections 2024-03-29T10:10:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.186+0000 7f916f000700 1 -- 172.21.15.12:0/1732065758 wait complete. 2024-03-29T10:10:38.204 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:10:38.247 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:38.247+0000 7f8ecce46240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:10:38.409 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:38.409+0000 7f3fb2c40240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:10:38.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.413+0000 7f1d7a000700 1 Processor -- start 2024-03-29T10:10:38.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.413+0000 7f1d7a000700 1 -- start start 2024-03-29T10:10:38.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.414+0000 7f1d7a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d740b4420 0x7f1d740ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d7a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1d740af600 0x7f1d7406dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d7a000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d740ade10 con 0x7f1d7415a680 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d7a000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d740ae040 con 0x7f1d740af600 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d7a000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d740ae320 con 0x7f1d740b4420 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d73e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1d740af600 0x7f1d7406dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d73e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1d740af600 0x7f1d7406dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36348/0 (socket says 172.21.15.12:36348) 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d73e00700 1 -- 172.21.15.12:0/3420723699 learned_addr learned my addr 172.21.15.12:0/3420723699 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d72a00700 1 --2- 172.21.15.12:0/3420723699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d740b4420 0x7f1d740ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d73e00700 1 -- 172.21.15.12:0/3420723699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d740b4420 msgr2=0x7f1d740ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d73e00700 1 --2- 172.21.15.12:0/3420723699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d740b4420 0x7f1d740ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d73e00700 1 -- 172.21.15.12:0/3420723699 >> v1:172.21.15.159:6789/0 conn(0x7f1d7415a680 legacy=0x7f1d740b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:38.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d73e00700 1 -- 172.21.15.12:0/3420723699 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d740ae600 con 0x7f1d740af600 2024-03-29T10:10:38.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.415+0000 7f1d72000700 1 -- 172.21.15.12:0/3420723699 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186411536 0 0) 0x7f1d740ade10 con 0x7f1d7415a680 2024-03-29T10:10:38.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.416+0000 7f1d73e00700 1 --2- 172.21.15.12:0/3420723699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740af600 0x7f1d7406dd40 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009e50 tx=0x7f1d6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5de447ecf4f9174 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:38.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.416+0000 7f1d72000700 1 -- 172.21.15.12:0/3420723699 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00e070 con 0x7f1d740af600 2024-03-29T10:10:38.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.416+0000 7f1d72000700 1 -- 172.21.15.12:0/3420723699 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6c00b910 con 0x7f1d740af600 2024-03-29T10:10:38.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.416+0000 7f1d72000700 1 -- 172.21.15.12:0/3420723699 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00bcc0 con 0x7f1d740af600 2024-03-29T10:10:38.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.416+0000 7f1d7a000700 1 -- 172.21.15.12:0/3420723699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740af600 msgr2=0x7f1d7406dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:38.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.416+0000 7f1d7a000700 1 --2- 172.21.15.12:0/3420723699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740af600 0x7f1d7406dd40 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009e50 tx=0x7f1d6c00b330 comp rx=0 tx=0).stop 2024-03-29T10:10:38.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.417+0000 7f1d7a000700 1 -- 172.21.15.12:0/3420723699 shutdown_connections 2024-03-29T10:10:38.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.417+0000 7f1d7a000700 1 --2- 172.21.15.12:0/3420723699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740af600 0x7f1d7406dd40 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.417+0000 7f1d7a000700 1 --2- 172.21.15.12:0/3420723699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d740b4420 0x7f1d740ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.417+0000 7f1d7a000700 1 -- 172.21.15.12:0/3420723699 >> 172.21.15.12:0/3420723699 conn(0x7f1d740b5720 msgr2=0x7f1d740b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:38.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.417+0000 7f1d7a000700 1 -- 172.21.15.12:0/3420723699 shutdown_connections 2024-03-29T10:10:38.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.417+0000 7f1d7a000700 1 -- 172.21.15.12:0/3420723699 wait complete. 2024-03-29T10:10:38.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.418+0000 7f1d7a000700 1 Processor -- start 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.419+0000 7f1d7a000700 1 -- start start 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d7a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d740af600 0x7f1d741527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d7a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740b4420 0x7f1d74150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d7a000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d7415a680 0x7f1d740afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d7a000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74151db0 con 0x7f1d740b4420 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d7a000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74152e40 con 0x7f1d7415a680 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d7a000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d740b09a0 con 0x7f1d740af600 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d72a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740b4420 0x7f1d74150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d72a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740b4420 0x7f1d74150ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36362/0 (socket says 172.21.15.12:36362) 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d72a00700 1 -- 172.21.15.12:0/1504654975 learned_addr learned my addr 172.21.15.12:0/1504654975 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d73400700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d740af600 0x7f1d741527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d73e00700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d7415a680 0x7f1d740afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:38.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d72a00700 1 -- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d740af600 msgr2=0x7f1d741527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:38.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d72a00700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d740af600 0x7f1d741527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d72a00700 1 -- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d7415a680 msgr2=0x7f1d740afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d72a00700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d7415a680 0x7f1d740afd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.420+0000 7f1d72a00700 1 -- 172.21.15.12:0/1504654975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c009b30 con 0x7f1d740b4420 2024-03-29T10:10:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.421+0000 7f1d73e00700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d7415a680 0x7f1d740afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.421+0000 7f1d72a00700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740b4420 0x7f1d74150ca0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000e9b0 tx=0x7f1d6000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.421+0000 7f1d6b400700 1 -- 172.21.15.12:0/1504654975 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d60009710 con 0x7f1d740b4420 2024-03-29T10:10:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.421+0000 7f1d6b400700 1 -- 172.21.15.12:0/1504654975 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6000ee50 con 0x7f1d740b4420 2024-03-29T10:10:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.421+0000 7f1d7a000700 1 -- 172.21.15.12:0/1504654975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d740b0c80 con 0x7f1d740b4420 2024-03-29T10:10:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.421+0000 7f1d6b400700 1 -- 172.21.15.12:0/1504654975 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6000ca50 con 0x7f1d740b4420 2024-03-29T10:10:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.421+0000 7f1d7a000700 1 -- 172.21.15.12:0/1504654975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d740b1240 con 0x7f1d740b4420 2024-03-29T10:10:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.422+0000 7f1d7a000700 1 -- 172.21.15.12:0/1504654975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d740b3a30 con 0x7f1d740b4420 2024-03-29T10:10:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.423+0000 7f1d6b400700 1 -- 172.21.15.12:0/1504654975 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f1d6000cbb0 con 0x7f1d740b4420 2024-03-29T10:10:38.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.424+0000 7f1d6b400700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d5406adb0 0x7f1d5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.424+0000 7f1d6b400700 1 -- 172.21.15.12:0/1504654975 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f1d600895b0 con 0x7f1d740b4420 2024-03-29T10:10:38.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.425+0000 7f1d73400700 1 -- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d5406adb0 msgr2=0x7f1d5406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:38.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.425+0000 7f1d73400700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d5406adb0 0x7f1d5406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:38.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.429+0000 7f1d6b400700 1 -- 172.21.15.12:0/1504654975 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1d60054510 con 0x7f1d740b4420 2024-03-29T10:10:38.521 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:38.521+0000 7f8ecce46240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:10:38.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.626+0000 7f1d73400700 1 -- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d5406adb0 msgr2=0x7f1d5406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:38.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.626+0000 7f1d73400700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d5406adb0 0x7f1d5406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:38.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.644+0000 7f1d7a000700 1 -- 172.21.15.12:0/1504654975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1d74067720 con 0x7f1d740b4420 2024-03-29T10:10:38.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.644+0000 7f1d6b400700 1 -- 172.21.15.12:0/1504654975 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1d6000ce60 con 0x7f1d740b4420 2024-03-29T10:10:38.644 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:10:38.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.646+0000 7f1d7a000700 1 -- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d5406adb0 msgr2=0x7f1d5406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:38.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.646+0000 7f1d7a000700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d5406adb0 0x7f1d5406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.647+0000 7f1d7a000700 1 -- 172.21.15.12:0/1504654975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740b4420 msgr2=0x7f1d74150ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:38.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.647+0000 7f1d7a000700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740b4420 0x7f1d74150ca0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000e9b0 tx=0x7f1d6000ecc0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.647+0000 7f1d7a000700 1 -- 172.21.15.12:0/1504654975 shutdown_connections 2024-03-29T10:10:38.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.647+0000 7f1d7a000700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d5406adb0 0x7f1d5406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.647+0000 7f1d7a000700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d7415a680 0x7f1d740afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.647+0000 7f1d7a000700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d740af600 0x7f1d741527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.647+0000 7f1d7a000700 1 --2- 172.21.15.12:0/1504654975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d740b4420 0x7f1d74150ca0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.647+0000 7f1d7a000700 1 -- 172.21.15.12:0/1504654975 >> 172.21.15.12:0/1504654975 conn(0x7f1d740b5720 msgr2=0x7f1d740b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:38.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.647+0000 7f1d7a000700 1 -- 172.21.15.12:0/1504654975 shutdown_connections 2024-03-29T10:10:38.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.647+0000 7f1d7a000700 1 -- 172.21.15.12:0/1504654975 wait complete. 2024-03-29T10:10:38.665 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:10:38.869 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:38.869+0000 7f3fb2c40240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:10:38.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.910+0000 7f4ad8a00700 1 Processor -- start 2024-03-29T10:10:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.910+0000 7f4ad8a00700 1 -- start start 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.911+0000 7f4ad8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ad415d9d0 0x7f4ad40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.911+0000 7f4ad8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 0x7f4ad40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.911+0000 7f4ad8a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad4161420 con 0x7f4ad40d16a0 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.911+0000 7f4ad8a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad415edc0 con 0x7f4ad415d9d0 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.911+0000 7f4ad8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad4161700 con 0x7f4ad40d2540 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ad415d9d0 0x7f4ad40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ad415d9d0 0x7f4ad40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35590/0 (socket says 172.21.15.12:35590) 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad1e00700 1 -- 172.21.15.12:0/2688026777 learned_addr learned my addr 172.21.15.12:0/2688026777 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad1400700 1 --1- 172.21.15.12:0/2688026777 >> v1:172.21.15.12:6789/0 conn(0x7f4ad40d16a0 0x7f4ad40d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48996/0 (socket says 172.21.15.12:48996) 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad2800700 1 --2- 172.21.15.12:0/2688026777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 0x7f4ad40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad1e00700 1 -- 172.21.15.12:0/2688026777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 msgr2=0x7f4ad40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:38.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad1e00700 1 --2- 172.21.15.12:0/2688026777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 0x7f4ad40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad1e00700 1 -- 172.21.15.12:0/2688026777 >> v1:172.21.15.12:6789/0 conn(0x7f4ad40d16a0 legacy=0x7f4ad40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad1e00700 1 -- 172.21.15.12:0/2688026777 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad41619e0 con 0x7f4ad415d9d0 2024-03-29T10:10:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad0a00700 1 -- 172.21.15.12:0/2688026777 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497417647 0 0) 0x7f4ad4161420 con 0x7f4ad40d16a0 2024-03-29T10:10:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.912+0000 7f4ad1e00700 1 --2- 172.21.15.12:0/2688026777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad415d9d0 0x7f4ad40d3c30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40098f0 tx=0x7f4ac4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcdc6c6363310e2d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.913+0000 7f4ad0a00700 1 -- 172.21.15.12:0/2688026777 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4011830 con 0x7f4ad415d9d0 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.913+0000 7f4ad0a00700 1 -- 172.21.15.12:0/2688026777 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac4011990 con 0x7f4ad415d9d0 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.913+0000 7f4ad0a00700 1 -- 172.21.15.12:0/2688026777 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4011d20 con 0x7f4ad415d9d0 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.913+0000 7f4ad8a00700 1 -- 172.21.15.12:0/2688026777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad415d9d0 msgr2=0x7f4ad40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.913+0000 7f4ad8a00700 1 --2- 172.21.15.12:0/2688026777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad415d9d0 0x7f4ad40d3c30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40098f0 tx=0x7f4ac4009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.914+0000 7f4ad8a00700 1 -- 172.21.15.12:0/2688026777 shutdown_connections 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.914+0000 7f4ad8a00700 1 --2- 172.21.15.12:0/2688026777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 0x7f4ad40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.914+0000 7f4ad8a00700 1 --2- 172.21.15.12:0/2688026777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad415d9d0 0x7f4ad40d3c30 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.914+0000 7f4ad8a00700 1 -- 172.21.15.12:0/2688026777 >> 172.21.15.12:0/2688026777 conn(0x7f4ad408aff0 msgr2=0x7f4ad40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.914+0000 7f4ad8a00700 1 -- 172.21.15.12:0/2688026777 shutdown_connections 2024-03-29T10:10:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.914+0000 7f4ad8a00700 1 -- 172.21.15.12:0/2688026777 wait complete. 2024-03-29T10:10:38.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.916+0000 7f4ad8a00700 1 Processor -- start 2024-03-29T10:10:38.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.916+0000 7f4ad8a00700 1 -- start start 2024-03-29T10:10:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad40d16a0 0x7f4ad41529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 0x7f4ad40cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad415d9d0 0x7f4ad40b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40cb840 con 0x7f4ad40d16a0 2024-03-29T10:10:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad8a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40cc1a0 con 0x7f4ad415d9d0 2024-03-29T10:10:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40ccc00 con 0x7f4ad40d2540 2024-03-29T10:10:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad40d16a0 0x7f4ad41529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad40d16a0 0x7f4ad41529b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35592/0 (socket says 172.21.15.12:35592) 2024-03-29T10:10:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad1e00700 1 -- 172.21.15.12:0/795277196 learned_addr learned my addr 172.21.15.12:0/795277196 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad1400700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 0x7f4ad40cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:38.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.917+0000 7f4ad2800700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad415d9d0 0x7f4ad40b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4ad1400700 1 -- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad415d9d0 msgr2=0x7f4ad40b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4ad1400700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad415d9d0 0x7f4ad40b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4ad1400700 1 -- 172.21.15.12:0/795277196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad40d16a0 msgr2=0x7f4ad41529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4ad1400700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad40d16a0 0x7f4ad41529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4ad1400700 1 -- 172.21.15.12:0/795277196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac4009550 con 0x7f4ad40d2540 2024-03-29T10:10:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4ad1e00700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad40d16a0 0x7f4ad41529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4ad2800700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad415d9d0 0x7f4ad40b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4ad1400700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 0x7f4ad40cbc60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c4e0 tx=0x7f4ac000c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4aca000700 1 -- 172.21.15.12:0/795277196 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00074e0 con 0x7f4ad40d2540 2024-03-29T10:10:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4aca000700 1 -- 172.21.15.12:0/795277196 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac0007660 con 0x7f4ad40d2540 2024-03-29T10:10:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4ad8a00700 1 -- 172.21.15.12:0/795277196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad40cb180 con 0x7f4ad40d2540 2024-03-29T10:10:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4aca000700 1 -- 172.21.15.12:0/795277196 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0005a70 con 0x7f4ad40d2540 2024-03-29T10:10:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.918+0000 7f4ad8a00700 1 -- 172.21.15.12:0/795277196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad40cc790 con 0x7f4ad40d2540 2024-03-29T10:10:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.919+0000 7f4ad8a00700 1 -- 172.21.15.12:0/795277196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad40d1ca0 con 0x7f4ad40d2540 2024-03-29T10:10:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.920+0000 7f4aca000700 1 -- 172.21.15.12:0/795277196 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f4ac0005bd0 con 0x7f4ad40d2540 2024-03-29T10:10:38.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.922+0000 7f4aca000700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ab406adb0 0x7f4ab406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:38.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.922+0000 7f4aca000700 1 -- 172.21.15.12:0/795277196 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f4ac0088a00 con 0x7f4ad40d2540 2024-03-29T10:10:38.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.922+0000 7f4ad1e00700 1 -- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ab406adb0 msgr2=0x7f4ab406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.922+0000 7f4ad1e00700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ab406adb0 0x7f4ab406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:38.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:38.926+0000 7f4aca000700 1 -- 172.21.15.12:0/795277196 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ac00538d0 con 0x7f4ad40d2540 2024-03-29T10:10:39.010 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:39.010+0000 7f8ecce46240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:10:39.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.123+0000 7f4ad1e00700 1 -- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ab406adb0 msgr2=0x7f4ab406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:39.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.123+0000 7f4ad1e00700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ab406adb0 0x7f4ab406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:39.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.141+0000 7f4ad8a00700 1 -- 172.21.15.12:0/795277196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4ad4067720 con 0x7f4ad40d2540 2024-03-29T10:10:39.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.141+0000 7f4aca000700 1 -- 172.21.15.12:0/795277196 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4ac0017020 con 0x7f4ad40d2540 2024-03-29T10:10:39.141 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:10:39.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 -- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ab406adb0 msgr2=0x7f4ab406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:39.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ab406adb0 0x7f4ab406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 -- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 msgr2=0x7f4ad40cbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:39.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 0x7f4ad40cbc60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c4e0 tx=0x7f4ac000c7f0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 -- 172.21.15.12:0/795277196 shutdown_connections 2024-03-29T10:10:39.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ab406adb0 0x7f4ab406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad415d9d0 0x7f4ad40b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ad40d16a0 0x7f4ad41529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 --2- 172.21.15.12:0/795277196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad40d2540 0x7f4ad40cbc60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 -- 172.21.15.12:0/795277196 >> 172.21.15.12:0/795277196 conn(0x7f4ad408aff0 msgr2=0x7f4ad40b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:39.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 -- 172.21.15.12:0/795277196 shutdown_connections 2024-03-29T10:10:39.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.144+0000 7f4ad8a00700 1 -- 172.21.15.12:0/795277196 wait complete. 2024-03-29T10:10:39.163 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T10:10:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.365+0000 7facb3600700 1 Processor -- start 2024-03-29T10:10:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.365+0000 7facb3600700 1 -- start start 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.366+0000 7facb3600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facac0d3f20 0x7facac0cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.366+0000 7facb3600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facac161230 con 0x7facac0b3f80 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.366+0000 7facb3600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facac0d2a80 con 0x7facac160e50 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.366+0000 7facb3600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facac0d43b0 con 0x7facac0d3f20 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facb1600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7facac0b3f80 0x7facac0ce430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49002/0 (socket says 172.21.15.12:49002) 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facb1600700 1 -- 172.21.15.12:0/2500867860 learned_addr learned my addr 172.21.15.12:0/2500867860 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facabe00700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facac0d3f20 0x7facac0cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422326636 0 0) 0x7facac161230 con 0x7facac0b3f80 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac940035b0 con 0x7facac0b3f80 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957016592 0 0) 0x7facac0d2a80 con 0x7facac160e50 2024-03-29T10:10:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facac161230 con 0x7facac160e50 2024-03-29T10:10:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 255944997 0 0) 0x7fac940035b0 con 0x7facac0b3f80 2024-03-29T10:10:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facac0d3f20 msgr2=0x7facac0cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facab400700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7facac0d3f20 0x7facac0cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 >> v1:172.21.15.159:6789/0 conn(0x7facac160e50 legacy=0x7facac0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.367+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac0d21e0 con 0x7facac0b3f80 2024-03-29T10:10:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7faca40019f0 con 0x7facac0b3f80 2024-03-29T10:10:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 >> v1:172.21.15.12:6789/0 conn(0x7facac0b3f80 legacy=0x7facac0ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facab400700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac94003bc0 0x7fac94003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facab400700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac94004850 0x7fac94004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facab400700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac94005210 0x7fac9400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac0d43b0 con 0x7fac94004850 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9400bf50 con 0x7fac94003bc0 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9400c0c0 con 0x7fac94005210 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facabe00700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac94004850 0x7fac94004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facb1600700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac94005210 0x7fac9400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facb0c00700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac94003bc0 0x7fac94003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facabe00700 1 -- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac94005210 msgr2=0x7fac9400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facabe00700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac94005210 0x7fac9400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facabe00700 1 -- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac94003bc0 msgr2=0x7fac94003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facabe00700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac94003bc0 0x7fac94003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facabe00700 1 -- 172.21.15.12:0/2500867860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940035b0 con 0x7fac94004850 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facb0c00700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac94003bc0 0x7fac94003fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.368+0000 7facb1600700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac94005210 0x7fac9400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.369+0000 7facabe00700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac94004850 0x7fac94004cd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fac98009b60 tx=0x7fac94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c141cae487daea46 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.369+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9800e660 con 0x7fac94004850 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.369+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9800e7c0 con 0x7fac94004850 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.369+0000 7facab400700 1 -- 172.21.15.12:0/2500867860 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9800eb30 con 0x7fac94004850 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.369+0000 7facb3600700 1 -- 172.21.15.12:0/2500867860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac94004850 msgr2=0x7fac94004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.369+0000 7facb3600700 1 --2- 172.21.15.12:0/2500867860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac94004850 0x7fac94004cd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fac98009b60 tx=0x7fac94005880 comp rx=0 tx=0).stop 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.369+0000 7facb1600700 1 -- 172.21.15.12:0/2500867860 reap_dead start 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.370+0000 7facb3600700 1 -- 172.21.15.12:0/2500867860 shutdown_connections 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.370+0000 7facb3600700 1 -- 172.21.15.12:0/2500867860 >> 172.21.15.12:0/2500867860 conn(0x7facac08aff0 msgr2=0x7facac0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.370+0000 7facb3600700 1 -- 172.21.15.12:0/2500867860 shutdown_connections 2024-03-29T10:10:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.370+0000 7facb3600700 1 -- 172.21.15.12:0/2500867860 wait complete. 2024-03-29T10:10:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.372+0000 7facb3600700 1 Processor -- start 2024-03-29T10:10:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.372+0000 7facb3600700 1 -- start start 2024-03-29T10:10:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facb3600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac94004850 0x7facac15c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facb3600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facac0ccee0 0x7facac0ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facb3600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac0cea60 0x7facac1f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facb3600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac0ceef0 con 0x7facac0cea60 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facb3600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac0ca120 con 0x7facac0ccee0 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facb3600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facac0c8010 con 0x7fac94004850 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facb1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac0cea60 0x7facac1f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facb1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac0cea60 0x7facac1f6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35612/0 (socket says 172.21.15.12:35612) 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facb1600700 1 -- 172.21.15.12:0/1837574454 learned_addr learned my addr 172.21.15.12:0/1837574454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facabe00700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facac0ccee0 0x7facac0ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.373+0000 7facb0c00700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac94004850 0x7facac15c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7facb1600700 1 -- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac94004850 msgr2=0x7facac15c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7facb1600700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac94004850 0x7facac15c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7facb1600700 1 -- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facac0ccee0 msgr2=0x7facac0ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7facb1600700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facac0ccee0 0x7facac0ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7facb1600700 1 -- 172.21.15.12:0/1837574454 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac1f3de0 con 0x7facac0cea60 2024-03-29T10:10:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7facb1600700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac0cea60 0x7facac1f6420 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7faca4002230 tx=0x7faca400e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7faca8c00700 1 -- 172.21.15.12:0/1837574454 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca400ebc0 con 0x7facac0cea60 2024-03-29T10:10:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7faca8c00700 1 -- 172.21.15.12:0/1837574454 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca400ed20 con 0x7facac0cea60 2024-03-29T10:10:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7faca8c00700 1 -- 172.21.15.12:0/1837574454 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca4013620 con 0x7facac0cea60 2024-03-29T10:10:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7facb3600700 1 -- 172.21.15.12:0/1837574454 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac15ce50 con 0x7facac0cea60 2024-03-29T10:10:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.374+0000 7facb3600700 1 -- 172.21.15.12:0/1837574454 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facac1f3540 con 0x7facac0cea60 2024-03-29T10:10:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.375+0000 7facb3600700 1 -- 172.21.15.12:0/1837574454 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac067720 con 0x7facac0cea60 2024-03-29T10:10:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.376+0000 7faca8c00700 1 -- 172.21.15.12:0/1837574454 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7faca4013780 con 0x7facac0cea60 2024-03-29T10:10:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.377+0000 7faca8c00700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac8c06adb0 0x7fac8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.377+0000 7faca8c00700 1 -- 172.21.15.12:0/1837574454 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7faca4089a10 con 0x7facac0cea60 2024-03-29T10:10:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.377+0000 7facb0c00700 1 -- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac8c06adb0 msgr2=0x7fac8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.377+0000 7facb0c00700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac8c06adb0 0x7fac8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.382+0000 7faca8c00700 1 -- 172.21.15.12:0/1837574454 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faca4054970 con 0x7facac0cea60 2024-03-29T10:10:39.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.578+0000 7facb0c00700 1 -- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac8c06adb0 msgr2=0x7fac8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:39.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.578+0000 7facb0c00700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac8c06adb0 0x7fac8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:39.582 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:39.583+0000 7f3fb2c40240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:10:39.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.614+0000 7facb3600700 1 -- 172.21.15.12:0/1837574454 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7facac0d2fc0 con 0x7facac0cea60 2024-03-29T10:10:39.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.615+0000 7faca8c00700 1 -- 172.21.15.12:0/1837574454 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v95) v1 ==== 80+0+466 (secure 0 0 0) 0x7faca4057f90 con 0x7facac0cea60 2024-03-29T10:10:39.615 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/706246125 2024-03-30T10:10:34.825193+0000 2024-03-29T10:10:39.615 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2287631103 2024-03-30T10:10:34.825193+0000 2024-03-29T10:10:39.615 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/43444 2024-03-30T10:10:34.825193+0000 2024-03-29T10:10:39.615 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/43444 2024-03-30T10:10:34.825193+0000 2024-03-29T10:10:39.615 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/356119888 2024-03-30T10:10:34.825193+0000 2024-03-29T10:10:39.615 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3972721365 2024-03-30T10:10:34.825193+0000 2024-03-29T10:10:39.615 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1459270455 2024-03-30T10:10:34.825193+0000 2024-03-29T10:10:39.615 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2421061145 2024-03-30T10:10:34.825193+0000 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.617+0000 7facb3600700 1 -- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac8c06adb0 msgr2=0x7fac8c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.617+0000 7facb3600700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac8c06adb0 0x7fac8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.618+0000 7facb3600700 1 -- 172.21.15.12:0/1837574454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac0cea60 msgr2=0x7facac1f6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.618+0000 7facb3600700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac0cea60 0x7facac1f6420 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7faca4002230 tx=0x7faca400e440 comp rx=0 tx=0).stop 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.618+0000 7facb3600700 1 -- 172.21.15.12:0/1837574454 shutdown_connections 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.618+0000 7facb3600700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fac8c06adb0 0x7fac8c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.618+0000 7facb3600700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac0cea60 0x7facac1f6420 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.618+0000 7facb3600700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac94004850 0x7facac15c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.618+0000 7facb3600700 1 --2- 172.21.15.12:0/1837574454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7facac0ccee0 0x7facac0ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.618+0000 7facb3600700 1 -- 172.21.15.12:0/1837574454 >> 172.21.15.12:0/1837574454 conn(0x7facac08aff0 msgr2=0x7facac0d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.618+0000 7facb3600700 1 -- 172.21.15.12:0/1837574454 shutdown_connections 2024-03-29T10:10:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:39.618+0000 7facb3600700 1 -- 172.21.15.12:0/1837574454 wait complete. 2024-03-29T10:10:39.619 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T10:10:39.636 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T10:10:39.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:39.637 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T10:10:39.675 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:39.675+0000 7f3fb2c40240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:10:39.711 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T10:10:39.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:10:39.711 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T10:10:39.758 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:10:39.758 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:39.785 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:39.786+0000 7f8ecce46240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:10:39.808 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:10:39.808 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:10:39.808 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:39.845 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:39.845+0000 7f3fb2c40240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:10:39.853 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:10:39.853 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:10:39.853 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:39.884 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:39.884+0000 7f8ecce46240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:10:39.903 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:10:39.904 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:40.057 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:40.058+0000 7f8ecce46240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:10:40.068 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:40.069+0000 7f3fb2c40240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:10:40.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.105+0000 7fdd28800700 1 Processor -- start 2024-03-29T10:10:40.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.106+0000 7fdd28800700 1 -- start start 2024-03-29T10:10:40.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd28800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdd2015b560 0x7fdd2009aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd28800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdd200cc900 0x7fdd200cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd28800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20099a60 0x7fdd20099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd28800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2009a3f0 con 0x7fdd2015b560 2024-03-29T10:10:40.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd28800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd200cefc0 con 0x7fdd200cc900 2024-03-29T10:10:40.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd28800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200cf230 con 0x7fdd20099a60 2024-03-29T10:10:40.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd25e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdd2015b560 0x7fdd2009aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:40.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd25e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdd2015b560 0x7fdd2009aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35626/0 (socket says 172.21.15.12:35626) 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd25e00700 1 -- 172.21.15.12:0/3623187173 learned_addr learned my addr 172.21.15.12:0/3623187173 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd25400700 1 --2- 172.21.15.12:0/3623187173 >> v2:172.21.15.159:3300/0 conn(0x7fdd200cc900 0x7fdd200cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd26800700 1 --2- 172.21.15.12:0/3623187173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20099a60 0x7fdd20099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd25e00700 1 -- 172.21.15.12:0/3623187173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20099a60 msgr2=0x7fdd20099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd25e00700 1 --2- 172.21.15.12:0/3623187173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20099a60 0x7fdd20099eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd25e00700 1 -- 172.21.15.12:0/3623187173 >> v2:172.21.15.159:3300/0 conn(0x7fdd200cc900 msgr2=0x7fdd200cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd25e00700 1 --2- 172.21.15.12:0/3623187173 >> v2:172.21.15.159:3300/0 conn(0x7fdd200cc900 0x7fdd200cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.107+0000 7fdd25e00700 1 -- 172.21.15.12:0/3623187173 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd200cf510 con 0x7fdd2015b560 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.108+0000 7fdd25e00700 1 --2- 172.21.15.12:0/3623187173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd2015b560 0x7fdd2009aa70 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fdd200d0ec0 tx=0x7fdd14009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d96cae5f78c715 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.108+0000 7fdd24a00700 1 -- 172.21.15.12:0/3623187173 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14011830 con 0x7fdd2015b560 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.108+0000 7fdd24a00700 1 -- 172.21.15.12:0/3623187173 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd14011990 con 0x7fdd2015b560 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.108+0000 7fdd24a00700 1 -- 172.21.15.12:0/3623187173 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14011d20 con 0x7fdd2015b560 2024-03-29T10:10:40.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.109+0000 7fdd28800700 1 -- 172.21.15.12:0/3623187173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd2015b560 msgr2=0x7fdd2009aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:40.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.109+0000 7fdd28800700 1 --2- 172.21.15.12:0/3623187173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd2015b560 0x7fdd2009aa70 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fdd200d0ec0 tx=0x7fdd14009bd0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.109+0000 7fdd28800700 1 -- 172.21.15.12:0/3623187173 shutdown_connections 2024-03-29T10:10:40.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.109+0000 7fdd28800700 1 --2- 172.21.15.12:0/3623187173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20099a60 0x7fdd20099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.109+0000 7fdd28800700 1 --2- 172.21.15.12:0/3623187173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd2015b560 0x7fdd2009aa70 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.109+0000 7fdd28800700 1 --2- 172.21.15.12:0/3623187173 >> v2:172.21.15.159:3300/0 conn(0x7fdd200cc900 0x7fdd200cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.109+0000 7fdd28800700 1 -- 172.21.15.12:0/3623187173 >> 172.21.15.12:0/3623187173 conn(0x7fdd200d1350 msgr2=0x7fdd200d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:40.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.109+0000 7fdd28800700 1 -- 172.21.15.12:0/3623187173 shutdown_connections 2024-03-29T10:10:40.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.109+0000 7fdd28800700 1 -- 172.21.15.12:0/3623187173 wait complete. 2024-03-29T10:10:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.111+0000 7fdd28800700 1 Processor -- start 2024-03-29T10:10:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.111+0000 7fdd28800700 1 -- start start 2024-03-29T10:10:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd28800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20099a60 0x7fdd2009c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd28800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd200cc900 0x7fdd2009b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd28800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015b560 0x7fdd200c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd28800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd201f6e50 con 0x7fdd200cc900 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd28800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd201f7130 con 0x7fdd2015b560 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd28800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd201f7410 con 0x7fdd20099a60 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd25400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd200cc900 0x7fdd2009b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd26800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015b560 0x7fdd200c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd25400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd200cc900 0x7fdd2009b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35630/0 (socket says 172.21.15.12:35630) 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd25400700 1 -- 172.21.15.12:0/2063217736 learned_addr learned my addr 172.21.15.12:0/2063217736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd25e00700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20099a60 0x7fdd2009c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd25400700 1 -- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20099a60 msgr2=0x7fdd2009c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd25400700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20099a60 0x7fdd2009c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd25400700 1 -- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015b560 msgr2=0x7fdd200c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.112+0000 7fdd25400700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015b560 0x7fdd200c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.113+0000 7fdd25400700 1 -- 172.21.15.12:0/2063217736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd14009550 con 0x7fdd200cc900 2024-03-29T10:10:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.113+0000 7fdd26800700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015b560 0x7fdd200c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.113+0000 7fdd25400700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd200cc900 0x7fdd2009b250 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000e870 tx=0x7fdd1000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.113+0000 7fdd1a000700 1 -- 172.21.15.12:0/2063217736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10009590 con 0x7fdd200cc900 2024-03-29T10:10:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.113+0000 7fdd1a000700 1 -- 172.21.15.12:0/2063217736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd10013680 con 0x7fdd200cc900 2024-03-29T10:10:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.113+0000 7fdd28800700 1 -- 172.21.15.12:0/2063217736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd201f76f0 con 0x7fdd200cc900 2024-03-29T10:10:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.113+0000 7fdd1a000700 1 -- 172.21.15.12:0/2063217736 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd100139f0 con 0x7fdd200cc900 2024-03-29T10:10:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.113+0000 7fdd28800700 1 -- 172.21.15.12:0/2063217736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2009b790 con 0x7fdd200cc900 2024-03-29T10:10:40.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.114+0000 7fdd28800700 1 -- 172.21.15.12:0/2063217736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2009e0c0 con 0x7fdd200cc900 2024-03-29T10:10:40.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.115+0000 7fdd1a000700 1 -- 172.21.15.12:0/2063217736 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fdd10013b50 con 0x7fdd200cc900 2024-03-29T10:10:40.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.117+0000 7fdd1a000700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd0406adb0 0x7fdd0406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.117+0000 7fdd1a000700 1 -- 172.21.15.12:0/2063217736 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fdd10089390 con 0x7fdd200cc900 2024-03-29T10:10:40.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.117+0000 7fdd25e00700 1 -- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd0406adb0 msgr2=0x7fdd0406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:40.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.117+0000 7fdd25e00700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd0406adb0 0x7fdd0406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.121+0000 7fdd1a000700 1 -- 172.21.15.12:0/2063217736 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd10054390 con 0x7fdd200cc900 2024-03-29T10:10:40.161 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:40.161+0000 7f3fb2c40240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:10:40.259 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:40.259+0000 7f3fb2c40240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:10:40.280 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:40.281+0000 7f8ecce46240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:10:40.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.318+0000 7fdd25e00700 1 -- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd0406adb0 msgr2=0x7fdd0406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.318+0000 7fdd25e00700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd0406adb0 0x7fdd0406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:40.357 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:40.357+0000 7f3fb2c40240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:10:40.377 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:40.377+0000 7f8ecce46240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:10:40.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.383+0000 7fdd28800700 1 -- 172.21.15.12:0/2063217736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd201f7880 con 0x7fdd200cc900 2024-03-29T10:10:40.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.383+0000 7fdd1a000700 1 -- 172.21.15.12:0/2063217736 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fdd100579b0 con 0x7fdd200cc900 2024-03-29T10:10:40.383 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:40.383 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":38,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/4026715411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":4026715411},{"type":"v1","addr":"172.21.15.159:6836","nonce":4026715411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/225205526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":225205526},{"type":"v1","addr":"172.21.15.12:6834","nonce":225205526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6918,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2358504162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2358504162},{"type":"v1","addr":"172.21.15.159:6837","nonce":2358504162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:17.624968+0000","modified":"2024-03-29T10:10:21.121687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6924},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6924":{"gid":6924,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3359577869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3359577869},{"type":"v1","addr":"172.21.15.12:6835","nonce":3359577869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-29T10:10:40.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 -- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd0406adb0 msgr2=0x7fdd0406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:40.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd0406adb0 0x7fdd0406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 -- 172.21.15.12:0/2063217736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd200cc900 msgr2=0x7fdd2009b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:40.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd200cc900 0x7fdd2009b250 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000e870 tx=0x7fdd1000eb80 comp rx=0 tx=0).stop 2024-03-29T10:10:40.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 -- 172.21.15.12:0/2063217736 shutdown_connections 2024-03-29T10:10:40.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdd0406adb0 0x7fdd0406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015b560 0x7fdd200c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20099a60 0x7fdd2009c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 --2- 172.21.15.12:0/2063217736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd200cc900 0x7fdd2009b250 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 -- 172.21.15.12:0/2063217736 >> 172.21.15.12:0/2063217736 conn(0x7fdd200d1350 msgr2=0x7fdd2015a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:40.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 -- 172.21.15.12:0/2063217736 shutdown_connections 2024-03-29T10:10:40.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.386+0000 7fdd28800700 1 -- 172.21.15.12:0/2063217736 wait complete. 2024-03-29T10:10:40.387 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 38 2024-03-29T10:10:40.406 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:40.454 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:40.454+0000 7f3fb2c40240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:10:40.492 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:40.492+0000 7f8ecce46240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:10:40.598 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:40.598+0000 7f8ecce46240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:10:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.614+0000 7f4940e00700 1 Processor -- start 2024-03-29T10:10:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.615+0000 7f4940e00700 1 -- start start 2024-03-29T10:10:40.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.616+0000 7f4940e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f493c0709c0 0x7f493c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.616+0000 7f4940e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c0b2180 con 0x7f493c0b3af0 2024-03-29T10:10:40.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.616+0000 7f4940e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c15b100 con 0x7f493c072580 2024-03-29T10:10:40.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.616+0000 7f4940e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493c15b3e0 con 0x7f493c0709c0 2024-03-29T10:10:40.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.616+0000 7f4939e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f493c0b3af0 0x7f493c15a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49006/0 (socket says 172.21.15.12:49006) 2024-03-29T10:10:40.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.616+0000 7f4939e00700 1 -- 172.21.15.12:0/3270654231 learned_addr learned my addr 172.21.15.12:0/3270654231 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:40.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.616+0000 7f493a800700 1 --2- 172.21.15.12:0/3270654231 >> v2:172.21.15.159:3300/0 conn(0x7f493c0709c0 0x7f493c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:40.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.617+0000 7f4938a00700 1 -- 172.21.15.12:0/3270654231 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126832884 0 0) 0x7f493c0b2180 con 0x7f493c0b3af0 2024-03-29T10:10:40.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.617+0000 7f4938a00700 1 -- 172.21.15.12:0/3270654231 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49200035b0 con 0x7f493c0b3af0 2024-03-29T10:10:40.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.617+0000 7f493a800700 1 -- 172.21.15.12:0/3270654231 >> v1:172.21.15.159:6789/0 conn(0x7f493c072580 legacy=0x7f493c0b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:40.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.617+0000 7f493a800700 1 -- 172.21.15.12:0/3270654231 >> v1:172.21.15.12:6789/0 conn(0x7f493c0b3af0 legacy=0x7f493c15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:40.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.617+0000 7f4938a00700 1 -- 172.21.15.12:0/3270654231 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419123592 0 0) 0x7f493c15b100 con 0x7f493c072580 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.617+0000 7f493a800700 1 -- 172.21.15.12:0/3270654231 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c0b23f0 con 0x7f493c0709c0 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.617+0000 7f493a800700 1 --2- 172.21.15.12:0/3270654231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f493c0709c0 0x7f493c070e00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4934009d90 tx=0x7f493400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=907bb1d00fc135df server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.617+0000 7f4938a00700 1 -- 172.21.15.12:0/3270654231 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493400e070 con 0x7f493c0709c0 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.618+0000 7f4938a00700 1 -- 172.21.15.12:0/3270654231 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493400bb20 con 0x7f493c0709c0 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.618+0000 7f4938a00700 1 -- 172.21.15.12:0/3270654231 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493400bed0 con 0x7f493c0709c0 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.618+0000 7f4940e00700 1 -- 172.21.15.12:0/3270654231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f493c0709c0 msgr2=0x7f493c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.618+0000 7f4940e00700 1 --2- 172.21.15.12:0/3270654231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f493c0709c0 0x7f493c070e00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4934009d90 tx=0x7f493400b330 comp rx=0 tx=0).stop 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.618+0000 7f4940e00700 1 -- 172.21.15.12:0/3270654231 shutdown_connections 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.618+0000 7f4940e00700 1 --2- 172.21.15.12:0/3270654231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f493c0709c0 0x7f493c070e00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.618+0000 7f4940e00700 1 -- 172.21.15.12:0/3270654231 >> 172.21.15.12:0/3270654231 conn(0x7f493c0cf190 msgr2=0x7f493c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.618+0000 7f4940e00700 1 -- 172.21.15.12:0/3270654231 shutdown_connections 2024-03-29T10:10:40.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.618+0000 7f4940e00700 1 -- 172.21.15.12:0/3270654231 wait complete. 2024-03-29T10:10:40.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.620+0000 7f4940e00700 1 Processor -- start 2024-03-29T10:10:40.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.620+0000 7f4940e00700 1 -- start start 2024-03-29T10:10:40.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.621+0000 7f4940e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f493c0709c0 0x7f493c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.621+0000 7f4940e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f493c072580 0x7f493c153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.621+0000 7f4940e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f493c0b3af0 0x7f493c155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.621+0000 7f4940e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c1566c0 con 0x7f493c072580 2024-03-29T10:10:40.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.621+0000 7f4940e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0c9710 con 0x7f493c0709c0 2024-03-29T10:10:40.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.621+0000 7f4940e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0ca290 con 0x7f493c0b3af0 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.621+0000 7f4939400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f493c072580 0x7f493c153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.621+0000 7f4939400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f493c072580 0x7f493c153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35634/0 (socket says 172.21.15.12:35634) 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.621+0000 7f4939400700 1 -- 172.21.15.12:0/1437147171 learned_addr learned my addr 172.21.15.12:0/1437147171 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.621+0000 7f4939e00700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f493c0709c0 0x7f493c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f493a800700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f493c0b3af0 0x7f493c155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4939400700 1 -- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f493c0b3af0 msgr2=0x7f493c155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4939400700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f493c0b3af0 0x7f493c155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4939400700 1 -- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f493c0709c0 msgr2=0x7f493c0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4939400700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f493c0709c0 0x7f493c0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4939400700 1 -- 172.21.15.12:0/1437147171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4934009a70 con 0x7f493c072580 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4939e00700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f493c0709c0 0x7f493c0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:40.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4939400700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f493c072580 0x7f493c153d50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f492800c520 tx=0x7f492800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:40.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4932000700 1 -- 172.21.15.12:0/1437147171 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49280054e0 con 0x7f493c072580 2024-03-29T10:10:40.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4932000700 1 -- 172.21.15.12:0/1437147171 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4928003680 con 0x7f493c072580 2024-03-29T10:10:40.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4940e00700 1 -- 172.21.15.12:0/1437147171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493c0cc0e0 con 0x7f493c072580 2024-03-29T10:10:40.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.622+0000 7f4932000700 1 -- 172.21.15.12:0/1437147171 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928003a30 con 0x7f493c072580 2024-03-29T10:10:40.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.623+0000 7f4940e00700 1 -- 172.21.15.12:0/1437147171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493c0c9c40 con 0x7f493c072580 2024-03-29T10:10:40.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.624+0000 7f4940e00700 1 -- 172.21.15.12:0/1437147171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493c066ff0 con 0x7f493c072580 2024-03-29T10:10:40.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.625+0000 7f4932000700 1 -- 172.21.15.12:0/1437147171 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f4928003b90 con 0x7f493c072580 2024-03-29T10:10:40.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.626+0000 7f4932000700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f491c06adb0 0x7f491c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:40.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.626+0000 7f4932000700 1 -- 172.21.15.12:0/1437147171 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f4928088a50 con 0x7f493c072580 2024-03-29T10:10:40.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.626+0000 7f4939e00700 1 -- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f491c06adb0 msgr2=0x7f491c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:40.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.626+0000 7f4939e00700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f491c06adb0 0x7f491c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:40.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.630+0000 7f4932000700 1 -- 172.21.15.12:0/1437147171 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f49280539c0 con 0x7f493c072580 2024-03-29T10:10:40.668 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:40.668+0000 7f3fb2c40240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:10:40.705 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:40.705+0000 7f8ecce46240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:10:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.827+0000 7f4939e00700 1 -- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f491c06adb0 msgr2=0x7f491c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.827+0000 7f4939e00700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f491c06adb0 0x7f491c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:40.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.846+0000 7f4940e00700 1 -- 172.21.15.12:0/1437147171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f493c06cef0 con 0x7f493c072580 2024-03-29T10:10:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.847+0000 7f4932000700 1 -- 172.21.15.12:0/1437147171 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f4928017020 con 0x7f493c072580 2024-03-29T10:10:40.847 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:40.848 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":95,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:34.825237+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:16.040572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:16.605686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/706246125":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/2287631103":"2024-03-30T10:10:34.825193+0000","172.21.15.159:6833/43444":"2024-03-30T10:10:34.825193+0000","172.21.15.159:6832/43444":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/356119888":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/3972721365":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/1459270455":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/2421061145":"2024-03-30T10:10:34.825193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 -- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f491c06adb0 msgr2=0x7f491c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f491c06adb0 0x7f491c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 -- 172.21.15.12:0/1437147171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f493c072580 msgr2=0x7f493c153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f493c072580 0x7f493c153d50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f492800c520 tx=0x7f492800c830 comp rx=0 tx=0).stop 2024-03-29T10:10:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 -- 172.21.15.12:0/1437147171 shutdown_connections 2024-03-29T10:10:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f491c06adb0 0x7f491c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f493c0b3af0 0x7f493c155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f493c0709c0 0x7f493c0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 --2- 172.21.15.12:0/1437147171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f493c072580 0x7f493c153d50 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 -- 172.21.15.12:0/1437147171 >> 172.21.15.12:0/1437147171 conn(0x7f493c0cf190 msgr2=0x7f493c070da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 -- 172.21.15.12:0/1437147171 shutdown_connections 2024-03-29T10:10:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:40.850+0000 7f4940e00700 1 -- 172.21.15.12:0/1437147171 wait complete. 2024-03-29T10:10:40.871 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:10:16.040572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:10:16.605686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:10:40.871 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:10:40.925 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:40.925+0000 7f3fb2c40240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:10:40.937 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:40.938+0000 7f8ecce46240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:10:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.083+0000 7f2196800700 1 Processor -- start 2024-03-29T10:10:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.083+0000 7f2196800700 1 -- start start 2024-03-29T10:10:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.084+0000 7f2196800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2190160e50 0x7f21900b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f2196800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21900d3f20 0x7f21900b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f2196800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21900cef00 con 0x7f21900b3f80 2024-03-29T10:10:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f2196800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2190161230 con 0x7f21900d3f20 2024-03-29T10:10:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f2196800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21900d2a80 con 0x7f2190160e50 2024-03-29T10:10:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f218ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21900d3f20 0x7f21900b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f218f400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2190160e50 0x7f21900b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f218ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21900d3f20 0x7f21900b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35638/0 (socket says 172.21.15.12:35638) 2024-03-29T10:10:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f218ea00700 1 -- 172.21.15.12:0/196558318 learned_addr learned my addr 172.21.15.12:0/196558318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f218ea00700 1 -- 172.21.15.12:0/196558318 >> v2:172.21.15.159:3300/0 conn(0x7f2190160e50 msgr2=0x7f21900b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f218ea00700 1 --2- 172.21.15.12:0/196558318 >> v2:172.21.15.159:3300/0 conn(0x7f2190160e50 0x7f21900b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f218ea00700 1 -- 172.21.15.12:0/196558318 >> v1:172.21.15.159:6789/0 conn(0x7f21900b3f80 legacy=0x7f21900b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.085+0000 7f218ea00700 1 -- 172.21.15.12:0/196558318 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21900d43b0 con 0x7f21900d3f20 2024-03-29T10:10:41.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.086+0000 7f218ea00700 1 --2- 172.21.15.12:0/196558318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21900d3f20 0x7f21900b53d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f21900cff10 tx=0x7f217c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99bd750663e40f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:41.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.086+0000 7f218e000700 1 -- 172.21.15.12:0/196558318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c011830 con 0x7f21900d3f20 2024-03-29T10:10:41.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.086+0000 7f218e000700 1 -- 172.21.15.12:0/196558318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c011990 con 0x7f21900d3f20 2024-03-29T10:10:41.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.086+0000 7f218e000700 1 -- 172.21.15.12:0/196558318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c011d20 con 0x7f21900d3f20 2024-03-29T10:10:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.087+0000 7f2196800700 1 -- 172.21.15.12:0/196558318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21900d3f20 msgr2=0x7f21900b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.087+0000 7f2196800700 1 --2- 172.21.15.12:0/196558318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21900d3f20 0x7f21900b53d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f21900cff10 tx=0x7f217c009bd0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.087+0000 7f2196800700 1 -- 172.21.15.12:0/196558318 shutdown_connections 2024-03-29T10:10:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.087+0000 7f2196800700 1 --2- 172.21.15.12:0/196558318 >> v2:172.21.15.159:3300/0 conn(0x7f2190160e50 0x7f21900b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.087+0000 7f2196800700 1 --2- 172.21.15.12:0/196558318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21900d3f20 0x7f21900b53d0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.087+0000 7f2196800700 1 -- 172.21.15.12:0/196558318 >> 172.21.15.12:0/196558318 conn(0x7f219008aff0 msgr2=0x7f21900d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.087+0000 7f2196800700 1 -- 172.21.15.12:0/196558318 shutdown_connections 2024-03-29T10:10:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.087+0000 7f2196800700 1 -- 172.21.15.12:0/196558318 wait complete. 2024-03-29T10:10:41.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.089+0000 7f2196800700 1 Processor -- start 2024-03-29T10:10:41.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.089+0000 7f2196800700 1 -- start start 2024-03-29T10:10:41.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f2196800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21900b3f80 0x7f21900b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f2196800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2190160e50 0x7f21901583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f2196800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21900c6e00 0x7f21901588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f2196800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21900b9140 con 0x7f21900c6e00 2024-03-29T10:10:41.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f2196800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21900c7290 con 0x7f2190160e50 2024-03-29T10:10:41.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f2196800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21900b8b80 con 0x7f21900b3f80 2024-03-29T10:10:41.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f218fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21900c6e00 0x7f21901588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:41.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f218ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2190160e50 0x7f21901583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:41.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f218ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2190160e50 0x7f21901583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:56502/0 (socket says 172.21.15.12:56502) 2024-03-29T10:10:41.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f218f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21900b3f80 0x7f21900b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:41.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.090+0000 7f218ea00700 1 -- 172.21.15.12:0/1388242050 learned_addr learned my addr 172.21.15.12:0/1388242050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:41.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.091+0000 7f218ea00700 1 -- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21900b3f80 msgr2=0x7f21900b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:41.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.091+0000 7f218ea00700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21900b3f80 0x7f21900b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.091+0000 7f218ea00700 1 -- 172.21.15.12:0/1388242050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21900c6e00 msgr2=0x7f21901588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.091+0000 7f218ea00700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21900c6e00 0x7f21901588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.091+0000 7f218ea00700 1 -- 172.21.15.12:0/1388242050 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217c009550 con 0x7f2190160e50 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.091+0000 7f218f400700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21900b3f80 0x7f21900b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.091+0000 7f218fe00700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21900c6e00 0x7f21901588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.091+0000 7f218ea00700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2190160e50 0x7f21901583a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f217c000c00 tx=0x7f217c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.091+0000 7f2187400700 1 -- 172.21.15.12:0/1388242050 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c004380 con 0x7f2190160e50 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.092+0000 7f2187400700 1 -- 172.21.15.12:0/1388242050 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c01f070 con 0x7f2190160e50 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.092+0000 7f2196800700 1 -- 172.21.15.12:0/1388242050 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21900b8e60 con 0x7f2190160e50 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.092+0000 7f2187400700 1 -- 172.21.15.12:0/1388242050 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c0036a0 con 0x7f2190160e50 2024-03-29T10:10:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.092+0000 7f2196800700 1 -- 172.21.15.12:0/1388242050 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f219015a3e0 con 0x7f2190160e50 2024-03-29T10:10:41.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.093+0000 7f2196800700 1 -- 172.21.15.12:0/1388242050 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21900b89e0 con 0x7f2190160e50 2024-03-29T10:10:41.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.094+0000 7f2187400700 1 -- 172.21.15.12:0/1388242050 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f217c021030 con 0x7f2190160e50 2024-03-29T10:10:41.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.095+0000 7f2187400700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f217006adb0 0x7f217006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.095+0000 7f2187400700 1 -- 172.21.15.12:0/1388242050 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f217c08aaa0 con 0x7f2190160e50 2024-03-29T10:10:41.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.095+0000 7f218f400700 1 -- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f217006adb0 msgr2=0x7f217006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:41.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.095+0000 7f218f400700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f217006adb0 0x7f217006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:41.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.099+0000 7f2187400700 1 -- 172.21.15.12:0/1388242050 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21900b89e0 con 0x7f2190160e50 2024-03-29T10:10:41.117 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:41.118+0000 7f3fb2c40240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:10:41.209 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:41.210+0000 7f8ecce46240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:10:41.212 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:41.212+0000 7f3fb2c40240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.296+0000 7f218f400700 1 -- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f217006adb0 msgr2=0x7f217006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.296+0000 7f218f400700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f217006adb0 0x7f217006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.315+0000 7f2196800700 1 -- 172.21.15.12:0/1388242050 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21900b18b0 con 0x7f2190160e50 2024-03-29T10:10:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.315+0000 7f2187400700 1 -- 172.21.15.12:0/1388242050 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f21900b18b0 con 0x7f2190160e50 2024-03-29T10:10:41.315 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:10:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 -- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f217006adb0 msgr2=0x7f217006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f217006adb0 0x7f217006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 -- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2190160e50 msgr2=0x7f21901583a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2190160e50 0x7f21901583a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f217c000c00 tx=0x7f217c004750 comp rx=0 tx=0).stop 2024-03-29T10:10:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 -- 172.21.15.12:0/1388242050 shutdown_connections 2024-03-29T10:10:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f217006adb0 0x7f217006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21900c6e00 0x7f21901588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21900b3f80 0x7f21900b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 --2- 172.21.15.12:0/1388242050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2190160e50 0x7f21901583a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 -- 172.21.15.12:0/1388242050 >> 172.21.15.12:0/1388242050 conn(0x7f219008aff0 msgr2=0x7f219015dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 -- 172.21.15.12:0/1388242050 shutdown_connections 2024-03-29T10:10:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.318+0000 7f2196800700 1 -- 172.21.15.12:0/1388242050 wait complete. 2024-03-29T10:10:41.337 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:10:41.416 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:41.416+0000 7f8ecce46240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:10:41.516 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:41.517+0000 7f8ecce46240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:41.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.540+0000 7f2847200700 1 Processor -- start 2024-03-29T10:10:41.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.541+0000 7f2847200700 1 -- start start 2024-03-29T10:10:41.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.542+0000 7f2847200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f28400cbb30 0x7f28400cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.542+0000 7f2847200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28400cd480 0x7f284006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.542+0000 7f2847200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400cc450 con 0x7f28400cd020 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.542+0000 7f2847200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28400cdae0 con 0x7f28400cbb30 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.542+0000 7f2847200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28400cddc0 con 0x7f28400cd480 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.542+0000 7f283fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f28400cbb30 0x7f28400cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.542+0000 7f283fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f28400cbb30 0x7f28400cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35654/0 (socket says 172.21.15.12:35654) 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.542+0000 7f283fe00700 1 -- 172.21.15.12:0/2040236222 learned_addr learned my addr 172.21.15.12:0/2040236222 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.542+0000 7f2845200700 1 --2- 172.21.15.12:0/2040236222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28400cd480 0x7f284006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.543+0000 7f283fe00700 1 -- 172.21.15.12:0/2040236222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28400cd480 msgr2=0x7f284006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.543+0000 7f283fe00700 1 --2- 172.21.15.12:0/2040236222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28400cd480 0x7f284006e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.543+0000 7f283fe00700 1 -- 172.21.15.12:0/2040236222 >> v1:172.21.15.159:6789/0 conn(0x7f28400cd020 legacy=0x7f28400b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.543+0000 7f283fe00700 1 -- 172.21.15.12:0/2040236222 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28400ce0a0 con 0x7f28400cbb30 2024-03-29T10:10:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.543+0000 7f283fe00700 1 --2- 172.21.15.12:0/2040236222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cbb30 0x7f28400cbf10 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f28300098f0 tx=0x7f2830009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7274a28cc51d809 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.543+0000 7f283ea00700 1 -- 172.21.15.12:0/2040236222 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830011830 con 0x7f28400cbb30 2024-03-29T10:10:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.543+0000 7f283ea00700 1 -- 172.21.15.12:0/2040236222 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2830011990 con 0x7f28400cbb30 2024-03-29T10:10:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.543+0000 7f283ea00700 1 -- 172.21.15.12:0/2040236222 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830011d20 con 0x7f28400cbb30 2024-03-29T10:10:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.544+0000 7f2847200700 1 -- 172.21.15.12:0/2040236222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cbb30 msgr2=0x7f28400cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.544+0000 7f2847200700 1 --2- 172.21.15.12:0/2040236222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cbb30 0x7f28400cbf10 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f28300098f0 tx=0x7f2830009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.544+0000 7f2847200700 1 -- 172.21.15.12:0/2040236222 shutdown_connections 2024-03-29T10:10:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.544+0000 7f2847200700 1 --2- 172.21.15.12:0/2040236222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28400cd480 0x7f284006e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.544+0000 7f2847200700 1 --2- 172.21.15.12:0/2040236222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cbb30 0x7f28400cbf10 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.544+0000 7f2847200700 1 -- 172.21.15.12:0/2040236222 >> 172.21.15.12:0/2040236222 conn(0x7f28400b0d70 msgr2=0x7f28400b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.544+0000 7f2847200700 1 -- 172.21.15.12:0/2040236222 shutdown_connections 2024-03-29T10:10:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.544+0000 7f2847200700 1 -- 172.21.15.12:0/2040236222 wait complete. 2024-03-29T10:10:41.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.546+0000 7f2847200700 1 Processor -- start 2024-03-29T10:10:41.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.546+0000 7f2847200700 1 -- start start 2024-03-29T10:10:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f2847200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28400cbb30 0x7f28401f7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f2847200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cd020 0x7f2840156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f2847200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28400cd480 0x7f28401f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f2847200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284009c4a0 con 0x7f28400cd020 2024-03-29T10:10:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f2847200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28401f6fa0 con 0x7f28400cbb30 2024-03-29T10:10:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f2847200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28401f6970 con 0x7f28400cd480 2024-03-29T10:10:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f283f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cd020 0x7f2840156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f283f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cd020 0x7f2840156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35668/0 (socket says 172.21.15.12:35668) 2024-03-29T10:10:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f283f400700 1 -- 172.21.15.12:0/1313124777 learned_addr learned my addr 172.21.15.12:0/1313124777 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f283fe00700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28400cbb30 0x7f28401f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f2845200700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28400cd480 0x7f28401f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f283f400700 1 -- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28400cd480 msgr2=0x7f28401f33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f283f400700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28400cd480 0x7f28401f33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f283f400700 1 -- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28400cbb30 msgr2=0x7f28401f7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f283f400700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28400cbb30 0x7f28401f7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.547+0000 7f283f400700 1 -- 172.21.15.12:0/1313124777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2830009550 con 0x7f28400cd020 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.548+0000 7f283f400700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cd020 0x7f2840156130 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f282c00c4e0 tx=0x7f282c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.548+0000 7f2837e00700 1 -- 172.21.15.12:0/1313124777 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c0052d0 con 0x7f28400cd020 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.548+0000 7f2837e00700 1 -- 172.21.15.12:0/1313124777 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282c005430 con 0x7f28400cd020 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.548+0000 7f2837e00700 1 -- 172.21.15.12:0/1313124777 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c0038c0 con 0x7f28400cd020 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.548+0000 7f2847200700 1 -- 172.21.15.12:0/1313124777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28401f3eb0 con 0x7f28400cd020 2024-03-29T10:10:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.548+0000 7f2847200700 1 -- 172.21.15.12:0/1313124777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28401f6c30 con 0x7f28400cd020 2024-03-29T10:10:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.549+0000 7f2847200700 1 -- 172.21.15.12:0/1313124777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28401f7130 con 0x7f28400cd020 2024-03-29T10:10:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.550+0000 7f2837e00700 1 -- 172.21.15.12:0/1313124777 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f282c003a20 con 0x7f28400cd020 2024-03-29T10:10:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.551+0000 7f2837e00700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f282006adb0 0x7f282006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.551+0000 7f2837e00700 1 -- 172.21.15.12:0/1313124777 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f282c088990 con 0x7f28400cd020 2024-03-29T10:10:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.551+0000 7f283fe00700 1 -- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f282006adb0 msgr2=0x7f282006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.551+0000 7f283fe00700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f282006adb0 0x7f282006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.555+0000 7f2837e00700 1 -- 172.21.15.12:0/1313124777 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f282c0538f0 con 0x7f28400cd020 2024-03-29T10:10:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.752+0000 7f283fe00700 1 -- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f282006adb0 msgr2=0x7f282006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.752+0000 7f283fe00700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f282006adb0 0x7f282006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.775+0000 7f2847200700 1 -- 172.21.15.12:0/1313124777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f28401f6dc0 con 0x7f28400cd020 2024-03-29T10:10:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.775+0000 7f2837e00700 1 -- 172.21.15.12:0/1313124777 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7f282c017020 con 0x7f28400cd020 2024-03-29T10:10:41.775 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:10:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 -- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f282006adb0 msgr2=0x7f282006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f282006adb0 0x7f282006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 -- 172.21.15.12:0/1313124777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cd020 msgr2=0x7f2840156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cd020 0x7f2840156130 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f282c00c4e0 tx=0x7f282c00c7f0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 -- 172.21.15.12:0/1313124777 shutdown_connections 2024-03-29T10:10:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f282006adb0 0x7f282006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28400cd480 0x7f28401f33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28400cbb30 0x7f28401f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 --2- 172.21.15.12:0/1313124777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28400cd020 0x7f2840156130 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 -- 172.21.15.12:0/1313124777 >> 172.21.15.12:0/1313124777 conn(0x7f28400b0d70 msgr2=0x7f2840159950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 -- 172.21.15.12:0/1313124777 shutdown_connections 2024-03-29T10:10:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:41.778+0000 7f2847200700 1 -- 172.21.15.12:0/1313124777 wait complete. 2024-03-29T10:10:41.797 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:10:41.948 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:41.948+0000 7f3fb2c40240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:10:42.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.000+0000 7f8a49800700 1 Processor -- start 2024-03-29T10:10:42.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.000+0000 7f8a49800700 1 -- start start 2024-03-29T10:10:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a49800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8a4415a5c0 0x7f8a440709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a49800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4415dfe0 0x7f8a440ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a49800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8a440ce670 0x7f8a440b2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a49800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a440ceb00 con 0x7f8a440ce670 2024-03-29T10:10:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a49800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a440cb310 con 0x7f8a4415a5c0 2024-03-29T10:10:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a49800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a44070f00 con 0x7f8a4415dfe0 2024-03-29T10:10:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a43400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8a440ce670 0x7f8a440b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a42a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8a4415a5c0 0x7f8a440709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a43400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8a440ce670 0x7f8a440b2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35682/0 (socket says 172.21.15.12:35682) 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a43400700 1 -- 172.21.15.12:0/2804147095 learned_addr learned my addr 172.21.15.12:0/2804147095 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.002+0000 7f8a42000700 1 --2- 172.21.15.12:0/2804147095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4415dfe0 0x7f8a440ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.003+0000 7f8a43400700 1 -- 172.21.15.12:0/2804147095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4415dfe0 msgr2=0x7f8a440ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.003+0000 7f8a43400700 1 --2- 172.21.15.12:0/2804147095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4415dfe0 0x7f8a440ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.003+0000 7f8a43400700 1 -- 172.21.15.12:0/2804147095 >> v2:172.21.15.159:3300/0 conn(0x7f8a4415a5c0 msgr2=0x7f8a440709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.003+0000 7f8a43400700 1 --2- 172.21.15.12:0/2804147095 >> v2:172.21.15.159:3300/0 conn(0x7f8a4415a5c0 0x7f8a440709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.003+0000 7f8a43400700 1 -- 172.21.15.12:0/2804147095 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a440711a0 con 0x7f8a440ce670 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.003+0000 7f8a43400700 1 --2- 172.21.15.12:0/2804147095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 0x7f8a440b2640 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8a44072e70 tx=0x7f8a3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d14673a78d257e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.003+0000 7f8a41600700 1 -- 172.21.15.12:0/2804147095 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00e070 con 0x7f8a440ce670 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.003+0000 7f8a41600700 1 -- 172.21.15.12:0/2804147095 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c00b910 con 0x7f8a440ce670 2024-03-29T10:10:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.003+0000 7f8a41600700 1 -- 172.21.15.12:0/2804147095 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00bcc0 con 0x7f8a440ce670 2024-03-29T10:10:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.004+0000 7f8a49800700 1 -- 172.21.15.12:0/2804147095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 msgr2=0x7f8a440b2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.004+0000 7f8a49800700 1 --2- 172.21.15.12:0/2804147095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 0x7f8a440b2640 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8a44072e70 tx=0x7f8a3c00b330 comp rx=0 tx=0).stop 2024-03-29T10:10:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.004+0000 7f8a49800700 1 -- 172.21.15.12:0/2804147095 shutdown_connections 2024-03-29T10:10:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.004+0000 7f8a49800700 1 --2- 172.21.15.12:0/2804147095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 0x7f8a440b2640 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.004+0000 7f8a49800700 1 --2- 172.21.15.12:0/2804147095 >> v2:172.21.15.159:3300/0 conn(0x7f8a4415a5c0 0x7f8a440709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.004+0000 7f8a49800700 1 --2- 172.21.15.12:0/2804147095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4415dfe0 0x7f8a440ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.004+0000 7f8a49800700 1 -- 172.21.15.12:0/2804147095 >> 172.21.15.12:0/2804147095 conn(0x7f8a440b0d70 msgr2=0x7f8a440b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.004+0000 7f8a49800700 1 -- 172.21.15.12:0/2804147095 shutdown_connections 2024-03-29T10:10:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.004+0000 7f8a49800700 1 -- 172.21.15.12:0/2804147095 wait complete. 2024-03-29T10:10:42.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.007+0000 7f8a49800700 1 Processor -- start 2024-03-29T10:10:42.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.007+0000 7f8a49800700 1 -- start start 2024-03-29T10:10:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a49800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 0x7f8a4414fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a49800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4415a5c0 0x7f8a440afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a49800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a4415dfe0 0x7f8a441591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a49800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a440ad710 con 0x7f8a440ce670 2024-03-29T10:10:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a49800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a440b0ab0 con 0x7f8a4415dfe0 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a49800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a440b07d0 con 0x7f8a4415a5c0 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a42a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 0x7f8a4414fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a42a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 0x7f8a4414fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35692/0 (socket says 172.21.15.12:35692) 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a42a00700 1 -- 172.21.15.12:0/3577942940 learned_addr learned my addr 172.21.15.12:0/3577942940 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a42000700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4415a5c0 0x7f8a440afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a43400700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a4415dfe0 0x7f8a441591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.009+0000 7f8a42a00700 1 -- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4415a5c0 msgr2=0x7f8a440afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.010+0000 7f8a42a00700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4415a5c0 0x7f8a440afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.010+0000 7f8a42a00700 1 -- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a4415dfe0 msgr2=0x7f8a441591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.010+0000 7f8a42a00700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a4415dfe0 0x7f8a441591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.010+0000 7f8a42a00700 1 -- 172.21.15.12:0/3577942940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c009b30 con 0x7f8a440ce670 2024-03-29T10:10:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.010+0000 7f8a42a00700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 0x7f8a4414fa80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400ed40 tx=0x7f8a3400d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.010+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3577942940 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3400dd90 con 0x7f8a440ce670 2024-03-29T10:10:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.010+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3577942940 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a34010440 con 0x7f8a440ce670 2024-03-29T10:10:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.010+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3577942940 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a340107b0 con 0x7f8a440ce670 2024-03-29T10:10:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.010+0000 7f8a49800700 1 -- 172.21.15.12:0/3577942940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a440ad940 con 0x7f8a440ce670 2024-03-29T10:10:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.010+0000 7f8a49800700 1 -- 172.21.15.12:0/3577942940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a440adf00 con 0x7f8a440ce670 2024-03-29T10:10:42.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.012+0000 7f8a49800700 1 -- 172.21.15.12:0/3577942940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a440ae270 con 0x7f8a440ce670 2024-03-29T10:10:42.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.012+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3577942940 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f8a34010910 con 0x7f8a440ce670 2024-03-29T10:10:42.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.013+0000 7f8a3aa00700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406adb0 0x7f8a2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.013+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3577942940 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f8a34089990 con 0x7f8a440ce670 2024-03-29T10:10:42.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.013+0000 7f8a42000700 1 -- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406adb0 msgr2=0x7f8a2406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:42.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.013+0000 7f8a42000700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406adb0 0x7f8a2406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:42.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.017+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3577942940 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a340547f0 con 0x7f8a440ce670 2024-03-29T10:10:42.161 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:42.161+0000 7f3fb2c40240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:10:42.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.214+0000 7f8a42000700 1 -- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406adb0 msgr2=0x7f8a2406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:42.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.214+0000 7f8a42000700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406adb0 0x7f8a2406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.233+0000 7f8a49800700 1 -- 172.21.15.12:0/3577942940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8a44067720 con 0x7f8a440ce670 2024-03-29T10:10:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.233+0000 7f8a3aa00700 1 -- 172.21.15.12:0/3577942940 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8a3401d0a0 con 0x7f8a440ce670 2024-03-29T10:10:42.233 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:10:42.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 -- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406adb0 msgr2=0x7f8a2406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:42.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406adb0 0x7f8a2406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 -- 172.21.15.12:0/3577942940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 msgr2=0x7f8a4414fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 0x7f8a4414fa80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400ed40 tx=0x7f8a3400d630 comp rx=0 tx=0).stop 2024-03-29T10:10:42.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 -- 172.21.15.12:0/3577942940 shutdown_connections 2024-03-29T10:10:42.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8a2406adb0 0x7f8a2406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a4415dfe0 0x7f8a441591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a440ce670 0x7f8a4414fa80 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 --2- 172.21.15.12:0/3577942940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4415a5c0 0x7f8a440afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 -- 172.21.15.12:0/3577942940 >> 172.21.15.12:0/3577942940 conn(0x7f8a440b0d70 msgr2=0x7f8a440cd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:42.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 -- 172.21.15.12:0/3577942940 shutdown_connections 2024-03-29T10:10:42.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.236+0000 7f8a49800700 1 -- 172.21.15.12:0/3577942940 wait complete. 2024-03-29T10:10:42.254 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:42.306 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:42.307+0000 7f8ecce46240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:10:42.365 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:42.365+0000 7f3fb2c40240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:42.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.454+0000 7fd3efc00700 1 Processor -- start 2024-03-29T10:10:42.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.455+0000 7fd3efc00700 1 -- start start 2024-03-29T10:10:42.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.456+0000 7fd3efc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd3e815b5d0 0x7fd3e80ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.456+0000 7fd3efc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e80cb3f0 0x7fd3e80ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.456+0000 7fd3efc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e815fd50 con 0x7fd3e80ae930 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.456+0000 7fd3efc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e815b9b0 con 0x7fd3e815b5d0 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.456+0000 7fd3efc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e815f140 con 0x7fd3e80cb3f0 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.456+0000 7fd3ed200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd3e815b5d0 0x7fd3e80ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.456+0000 7fd3e7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e80cb3f0 0x7fd3e80ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.456+0000 7fd3ed200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd3e815b5d0 0x7fd3e80ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:56548/0 (socket says 172.21.15.12:56548) 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.456+0000 7fd3ed200700 1 -- 172.21.15.12:0/230136900 learned_addr learned my addr 172.21.15.12:0/230136900 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.457+0000 7fd3ed200700 1 -- 172.21.15.12:0/230136900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e80cb3f0 msgr2=0x7fd3e80ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.457+0000 7fd3ed200700 1 --2- 172.21.15.12:0/230136900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e80cb3f0 0x7fd3e80ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.457+0000 7fd3ed200700 1 -- 172.21.15.12:0/230136900 >> v1:172.21.15.159:6789/0 conn(0x7fd3e80ae930 legacy=0x7fd3e8140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.457+0000 7fd3ed200700 1 -- 172.21.15.12:0/230136900 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e815f420 con 0x7fd3e815b5d0 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.457+0000 7fd3e7e00700 1 --2- 172.21.15.12:0/230136900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e80cb3f0 0x7fd3e80ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:42.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.457+0000 7fd3ed200700 1 --2- 172.21.15.12:0/230136900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e815b5d0 0x7fd3e80ae3f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0098f0 tx=0x7fd3dc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=436a389057a7555f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.457+0000 7fd3e7400700 1 -- 172.21.15.12:0/230136900 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc011830 con 0x7fd3e815b5d0 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.457+0000 7fd3e7400700 1 -- 172.21.15.12:0/230136900 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3dc011990 con 0x7fd3e815b5d0 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.457+0000 7fd3e7400700 1 -- 172.21.15.12:0/230136900 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc011d20 con 0x7fd3e815b5d0 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.458+0000 7fd3efc00700 1 -- 172.21.15.12:0/230136900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e815b5d0 msgr2=0x7fd3e80ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.458+0000 7fd3efc00700 1 --2- 172.21.15.12:0/230136900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e815b5d0 0x7fd3e80ae3f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0098f0 tx=0x7fd3dc009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.458+0000 7fd3efc00700 1 -- 172.21.15.12:0/230136900 shutdown_connections 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.458+0000 7fd3efc00700 1 --2- 172.21.15.12:0/230136900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e815b5d0 0x7fd3e80ae3f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.458+0000 7fd3efc00700 1 --2- 172.21.15.12:0/230136900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e80cb3f0 0x7fd3e80ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.458+0000 7fd3efc00700 1 -- 172.21.15.12:0/230136900 >> 172.21.15.12:0/230136900 conn(0x7fd3e80cf190 msgr2=0x7fd3e80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.458+0000 7fd3efc00700 1 -- 172.21.15.12:0/230136900 shutdown_connections 2024-03-29T10:10:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.458+0000 7fd3efc00700 1 -- 172.21.15.12:0/230136900 wait complete. 2024-03-29T10:10:42.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.460+0000 7fd3efc00700 1 Processor -- start 2024-03-29T10:10:42.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.460+0000 7fd3efc00700 1 -- start start 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3efc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e80ae930 0x7fd3e814dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3efc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e815b5d0 0x7fd3e80ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3efc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3e8155c90 0x7fd3e8156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3efc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e8156650 con 0x7fd3e8155c90 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3efc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e81f65d0 con 0x7fd3e80ae930 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3efc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e81f68b0 con 0x7fd3e815b5d0 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3ed200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e80ae930 0x7fd3e814dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3e7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e815b5d0 0x7fd3e80ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3edc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3e8155c90 0x7fd3e8156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3ed200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e80ae930 0x7fd3e814dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:56560/0 (socket says 172.21.15.12:56560) 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3ed200700 1 -- 172.21.15.12:0/1947409931 learned_addr learned my addr 172.21.15.12:0/1947409931 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.461+0000 7fd3e7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e815b5d0 0x7fd3e80ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:38284/0 (socket says 172.21.15.12:38284) 2024-03-29T10:10:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.462+0000 7fd3ed200700 1 -- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e815b5d0 msgr2=0x7fd3e80ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.462+0000 7fd3ed200700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e815b5d0 0x7fd3e80ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.462+0000 7fd3ed200700 1 -- 172.21.15.12:0/1947409931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3e8155c90 msgr2=0x7fd3e8156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.462+0000 7fd3ed200700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3e8155c90 0x7fd3e8156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.462+0000 7fd3ed200700 1 -- 172.21.15.12:0/1947409931 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d40094a0 con 0x7fd3e80ae930 2024-03-29T10:10:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.462+0000 7fd3edc00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3e8155c90 0x7fd3e8156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.462+0000 7fd3e7e00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e815b5d0 0x7fd3e80ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.462+0000 7fd3ed200700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e80ae930 0x7fd3e814dfa0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc000c00 tx=0x7fd3dc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.463+0000 7fd3e4c00700 1 -- 172.21.15.12:0/1947409931 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc004400 con 0x7fd3e80ae930 2024-03-29T10:10:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.463+0000 7fd3e4c00700 1 -- 172.21.15.12:0/1947409931 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3dc01f070 con 0x7fd3e80ae930 2024-03-29T10:10:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.463+0000 7fd3e4c00700 1 -- 172.21.15.12:0/1947409931 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0038e0 con 0x7fd3e80ae930 2024-03-29T10:10:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.463+0000 7fd3efc00700 1 -- 172.21.15.12:0/1947409931 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3dc009550 con 0x7fd3e80ae930 2024-03-29T10:10:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.463+0000 7fd3efc00700 1 -- 172.21.15.12:0/1947409931 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e81f6e70 con 0x7fd3e80ae930 2024-03-29T10:10:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.464+0000 7fd3efc00700 1 -- 172.21.15.12:0/1947409931 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e80ca690 con 0x7fd3e80ae930 2024-03-29T10:10:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.465+0000 7fd3e4c00700 1 -- 172.21.15.12:0/1947409931 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fd3dc021030 con 0x7fd3e80ae930 2024-03-29T10:10:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.466+0000 7fd3e4c00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd3c806adb0 0x7fd3c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.466+0000 7fd3e4c00700 1 -- 172.21.15.12:0/1947409931 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fd3dc08a990 con 0x7fd3e80ae930 2024-03-29T10:10:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.466+0000 7fd3e7e00700 1 -- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd3c806adb0 msgr2=0x7fd3c806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.466+0000 7fd3e7e00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd3c806adb0 0x7fd3c806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.470+0000 7fd3e4c00700 1 -- 172.21.15.12:0/1947409931 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd3dc055920 con 0x7fd3e80ae930 2024-03-29T10:10:42.542 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:42.543+0000 7f8ecce46240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:10:42.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.667+0000 7fd3e7e00700 1 -- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd3c806adb0 msgr2=0x7fd3c806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:42.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.667+0000 7fd3e7e00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd3c806adb0 0x7fd3c806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:42.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.739+0000 7fd3efc00700 1 -- 172.21.15.12:0/1947409931 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3e8067720 con 0x7fd3e80ae930 2024-03-29T10:10:42.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.739+0000 7fd3e4c00700 1 -- 172.21.15.12:0/1947409931 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fd3dc058f40 con 0x7fd3e80ae930 2024-03-29T10:10:42.739 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:42.739 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":38,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/4026715411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":4026715411},{"type":"v1","addr":"172.21.15.159:6836","nonce":4026715411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/225205526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":225205526},{"type":"v1","addr":"172.21.15.12:6834","nonce":225205526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6918,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2358504162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2358504162},{"type":"v1","addr":"172.21.15.159:6837","nonce":2358504162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:17.624968+0000","modified":"2024-03-29T10:10:21.121687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6924},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6924":{"gid":6924,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3359577869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3359577869},{"type":"v1","addr":"172.21.15.12:6835","nonce":3359577869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-29T10:10:42.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 -- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd3c806adb0 msgr2=0x7fd3c806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:42.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd3c806adb0 0x7fd3c806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 -- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e80ae930 msgr2=0x7fd3e814dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e80ae930 0x7fd3e814dfa0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc000c00 tx=0x7fd3dc004750 comp rx=0 tx=0).stop 2024-03-29T10:10:42.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 -- 172.21.15.12:0/1947409931 shutdown_connections 2024-03-29T10:10:42.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd3c806adb0 0x7fd3c806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3e8155c90 0x7fd3e8156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3e80ae930 0x7fd3e814dfa0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 --2- 172.21.15.12:0/1947409931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3e815b5d0 0x7fd3e80ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 -- 172.21.15.12:0/1947409931 >> 172.21.15.12:0/1947409931 conn(0x7fd3e80cf190 msgr2=0x7fd3e815c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 -- 172.21.15.12:0/1947409931 shutdown_connections 2024-03-29T10:10:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.742+0000 7fd3efc00700 1 -- 172.21.15.12:0/1947409931 wait complete. 2024-03-29T10:10:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 38 2024-03-29T10:10:42.760 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:42.764 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:42.762+0000 7f8ecce46240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:42.837 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:42.837+0000 7f3fb2c40240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:10:42.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.962+0000 7f5c32a00700 1 Processor -- start 2024-03-29T10:10:42.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.962+0000 7f5c32a00700 1 -- start start 2024-03-29T10:10:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.963+0000 7f5c32a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 0x7f5c2c166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.963+0000 7f5c32a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c15fd50 con 0x7f5c2c15b5d0 2024-03-29T10:10:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.963+0000 7f5c32a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c15f140 con 0x7f5c2c0cb3f0 2024-03-29T10:10:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.963+0000 7f5c32a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c15b9b0 con 0x7f5c2c0ac7d0 2024-03-29T10:10:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.963+0000 7f5c2be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5c2c15b5d0 0x7f5c2c0ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49014/0 (socket says 172.21.15.12:49014) 2024-03-29T10:10:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.963+0000 7f5c2be00700 1 -- 172.21.15.12:0/2267402108 learned_addr learned my addr 172.21.15.12:0/2267402108 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:42.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.963+0000 7f5c30a00700 1 --2- 172.21.15.12:0/2267402108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 0x7f5c2c166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.964+0000 7f5c2aa00700 1 -- 172.21.15.12:0/2267402108 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467925181 0 0) 0x7f5c2c15fd50 con 0x7f5c2c15b5d0 2024-03-29T10:10:42.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.964+0000 7f5c2aa00700 1 -- 172.21.15.12:0/2267402108 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c100035b0 con 0x7f5c2c15b5d0 2024-03-29T10:10:42.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.964+0000 7f5c2aa00700 1 -- 172.21.15.12:0/2267402108 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749417554 0 0) 0x7f5c2c15f140 con 0x7f5c2c0cb3f0 2024-03-29T10:10:42.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.964+0000 7f5c2aa00700 1 -- 172.21.15.12:0/2267402108 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c15fd50 con 0x7f5c2c0cb3f0 2024-03-29T10:10:42.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.964+0000 7f5c30a00700 1 -- 172.21.15.12:0/2267402108 >> v1:172.21.15.159:6789/0 conn(0x7f5c2c0cb3f0 legacy=0x7f5c2c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.964+0000 7f5c30a00700 1 -- 172.21.15.12:0/2267402108 >> v1:172.21.15.12:6789/0 conn(0x7f5c2c15b5d0 legacy=0x7f5c2c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.964+0000 7f5c30a00700 1 -- 172.21.15.12:0/2267402108 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c15f420 con 0x7f5c2c0ac7d0 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.964+0000 7f5c30a00700 1 --2- 172.21.15.12:0/2267402108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 0x7f5c2c166430 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009dc0 tx=0x7f5c2400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6ad9ba884d9f6f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.965+0000 7f5c2aa00700 1 -- 172.21.15.12:0/2267402108 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2400e070 con 0x7f5c2c0ac7d0 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.965+0000 7f5c2aa00700 1 -- 172.21.15.12:0/2267402108 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2400bb20 con 0x7f5c2c0ac7d0 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.965+0000 7f5c2aa00700 1 -- 172.21.15.12:0/2267402108 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c24012430 con 0x7f5c2c0ac7d0 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.965+0000 7f5c32a00700 1 -- 172.21.15.12:0/2267402108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 msgr2=0x7f5c2c166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.965+0000 7f5c32a00700 1 --2- 172.21.15.12:0/2267402108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 0x7f5c2c166430 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009dc0 tx=0x7f5c2400b330 comp rx=0 tx=0).stop 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.965+0000 7f5c32a00700 1 -- 172.21.15.12:0/2267402108 shutdown_connections 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.965+0000 7f5c32a00700 1 --2- 172.21.15.12:0/2267402108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 0x7f5c2c166430 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.965+0000 7f5c32a00700 1 -- 172.21.15.12:0/2267402108 >> 172.21.15.12:0/2267402108 conn(0x7f5c2c0cf190 msgr2=0x7f5c2c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.965+0000 7f5c32a00700 1 -- 172.21.15.12:0/2267402108 shutdown_connections 2024-03-29T10:10:42.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.965+0000 7f5c32a00700 1 -- 172.21.15.12:0/2267402108 wait complete. 2024-03-29T10:10:42.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.967+0000 7f5c32a00700 1 Processor -- start 2024-03-29T10:10:42.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.967+0000 7f5c32a00700 1 -- start start 2024-03-29T10:10:42.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.968+0000 7f5c32a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 0x7f5c2c06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.968+0000 7f5c32a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2c15b5d0 0x7f5c2c155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.968+0000 7f5c32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c2c0cd600 0x7f5c2c1529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.968+0000 7f5c32a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c144be0 con 0x7f5c2c0cd600 2024-03-29T10:10:42.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.968+0000 7f5c32a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c0ce6a0 con 0x7f5c2c15b5d0 2024-03-29T10:10:42.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.968+0000 7f5c32a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c0cda90 con 0x7f5c2c0ac7d0 2024-03-29T10:10:42.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.968+0000 7f5c30a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c2c0cd600 0x7f5c2c1529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.969+0000 7f5c30a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c2c0cd600 0x7f5c2c1529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35702/0 (socket says 172.21.15.12:35702) 2024-03-29T10:10:42.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.969+0000 7f5c30a00700 1 -- 172.21.15.12:0/1487152035 learned_addr learned my addr 172.21.15.12:0/1487152035 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:42.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.969+0000 7f5c2be00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 0x7f5c2c06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.969+0000 7f5c2b400700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2c15b5d0 0x7f5c2c155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.969+0000 7f5c30a00700 1 -- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 msgr2=0x7f5c2c06cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.969+0000 7f5c30a00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 0x7f5c2c06cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.969+0000 7f5c30a00700 1 -- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2c15b5d0 msgr2=0x7f5c2c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.969+0000 7f5c30a00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2c15b5d0 0x7f5c2c155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.969+0000 7f5c30a00700 1 -- 172.21.15.12:0/1487152035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c24009a70 con 0x7f5c2c0cd600 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.969+0000 7f5c30a00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c2c0cd600 0x7f5c2c1529e0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400b370 tx=0x7f5c24004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.970+0000 7f5c23e00700 1 -- 172.21.15.12:0/1487152035 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2401b810 con 0x7f5c2c0cd600 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.970+0000 7f5c23e00700 1 -- 172.21.15.12:0/1487152035 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c24020070 con 0x7f5c2c0cd600 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.970+0000 7f5c32a00700 1 -- 172.21.15.12:0/1487152035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2c0cdd70 con 0x7f5c2c0cd600 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.970+0000 7f5c23e00700 1 -- 172.21.15.12:0/1487152035 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2401bbf0 con 0x7f5c2c0cd600 2024-03-29T10:10:42.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.970+0000 7f5c32a00700 1 -- 172.21.15.12:0/1487152035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2c0ce330 con 0x7f5c2c0cd600 2024-03-29T10:10:42.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.971+0000 7f5c32a00700 1 -- 172.21.15.12:0/1487152035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2c15f140 con 0x7f5c2c0cd600 2024-03-29T10:10:42.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.972+0000 7f5c23e00700 1 -- 172.21.15.12:0/1487152035 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f5c2401bd50 con 0x7f5c2c0cd600 2024-03-29T10:10:42.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.973+0000 7f5c23e00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c0c06adb0 0x7f5c0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:42.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.973+0000 7f5c2be00700 1 -- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c0c06adb0 msgr2=0x7f5c0c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:42.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.973+0000 7f5c23e00700 1 -- 172.21.15.12:0/1487152035 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f5c24022070 con 0x7f5c2c0cd600 2024-03-29T10:10:42.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.973+0000 7f5c2be00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c0c06adb0 0x7f5c0c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:42.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:42.977+0000 7f5c23e00700 1 -- 172.21.15.12:0/1487152035 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c24056ba0 con 0x7f5c2c0cd600 2024-03-29T10:10:43.043 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.043+0000 7f3fb2c40240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.174+0000 7f5c2be00700 1 -- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c0c06adb0 msgr2=0x7f5c0c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.174+0000 7f5c2be00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c0c06adb0 0x7f5c0c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:10:43.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.193+0000 7f5c32a00700 1 -- 172.21.15.12:0/1487152035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c2c0ac5f0 con 0x7f5c2c0cd600 2024-03-29T10:10:43.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.194+0000 7f5c23e00700 1 -- 172.21.15.12:0/1487152035 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f5c2405a1c0 con 0x7f5c2c0cd600 2024-03-29T10:10:43.194 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:43.194 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":95,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:34.825237+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:16.040572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:16.605686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/706246125":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/2287631103":"2024-03-30T10:10:34.825193+0000","172.21.15.159:6833/43444":"2024-03-30T10:10:34.825193+0000","172.21.15.159:6832/43444":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/356119888":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/3972721365":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/1459270455":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/2421061145":"2024-03-30T10:10:34.825193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:43.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 -- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c0c06adb0 msgr2=0x7f5c0c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:43.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c0c06adb0 0x7f5c0c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 -- 172.21.15.12:0/1487152035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c2c0cd600 msgr2=0x7f5c2c1529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c2c0cd600 0x7f5c2c1529e0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400b370 tx=0x7f5c24004ad0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 -- 172.21.15.12:0/1487152035 shutdown_connections 2024-03-29T10:10:43.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5c0c06adb0 0x7f5c0c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c2c0cd600 0x7f5c2c1529e0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c2c0ac7d0 0x7f5c2c06cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 --2- 172.21.15.12:0/1487152035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2c15b5d0 0x7f5c2c155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 -- 172.21.15.12:0/1487152035 >> 172.21.15.12:0/1487152035 conn(0x7f5c2c0cf190 msgr2=0x7f5c2c0aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:43.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 -- 172.21.15.12:0/1487152035 shutdown_connections 2024-03-29T10:10:43.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.197+0000 7f5c32a00700 1 -- 172.21.15.12:0/1487152035 wait complete. 2024-03-29T10:10:43.216 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T10:10:43.216 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:43.287 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:43.287+0000 7f8ecce46240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:10:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.415+0000 7fbadc800700 1 Processor -- start 2024-03-29T10:10:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.415+0000 7fbadc800700 1 -- start start 2024-03-29T10:10:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.416+0000 7fbadc800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad406c650 0x7fbad40ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.416+0000 7fbadc800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad406bb40 con 0x7fbad415d710 2024-03-29T10:10:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.416+0000 7fbadc800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad4099b00 con 0x7fbad409aba0 2024-03-29T10:10:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.416+0000 7fbadc800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad4099de0 con 0x7fbad406c650 2024-03-29T10:10:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.416+0000 7fbad9e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbad415d710 0x7fbad40d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49028/0 (socket says 172.21.15.12:49028) 2024-03-29T10:10:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.417+0000 7fbad9e00700 1 -- 172.21.15.12:0/1740573399 learned_addr learned my addr 172.21.15.12:0/1740573399 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.417+0000 7fbada800700 1 --2- 172.21.15.12:0/1740573399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad406c650 0x7fbad40ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.417+0000 7fbad8a00700 1 -- 172.21.15.12:0/1740573399 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275171556 0 0) 0x7fbad406bb40 con 0x7fbad415d710 2024-03-29T10:10:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.417+0000 7fbad8a00700 1 -- 172.21.15.12:0/1740573399 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbabc0035b0 con 0x7fbad415d710 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.417+0000 7fbada800700 1 -- 172.21.15.12:0/1740573399 >> v1:172.21.15.159:6789/0 conn(0x7fbad409aba0 legacy=0x7fbad406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.417+0000 7fbada800700 1 -- 172.21.15.12:0/1740573399 >> v1:172.21.15.12:6789/0 conn(0x7fbad415d710 legacy=0x7fbad40d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.417+0000 7fbada800700 1 -- 172.21.15.12:0/1740573399 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad409a0c0 con 0x7fbad406c650 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.417+0000 7fbada800700 1 --2- 172.21.15.12:0/1740573399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad406c650 0x7fbad40ccbe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009dc0 tx=0x7fbad000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=438afbe92516583b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.418+0000 7fbad8a00700 1 -- 172.21.15.12:0/1740573399 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad000e070 con 0x7fbad406c650 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.418+0000 7fbad8a00700 1 -- 172.21.15.12:0/1740573399 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad000bb20 con 0x7fbad406c650 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.418+0000 7fbad8a00700 1 -- 172.21.15.12:0/1740573399 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0012430 con 0x7fbad406c650 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.418+0000 7fbadc800700 1 -- 172.21.15.12:0/1740573399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad406c650 msgr2=0x7fbad40ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.418+0000 7fbadc800700 1 --2- 172.21.15.12:0/1740573399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad406c650 0x7fbad40ccbe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009dc0 tx=0x7fbad000b330 comp rx=0 tx=0).stop 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.418+0000 7fbadc800700 1 -- 172.21.15.12:0/1740573399 shutdown_connections 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.418+0000 7fbadc800700 1 --2- 172.21.15.12:0/1740573399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad406c650 0x7fbad40ccbe0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.418+0000 7fbadc800700 1 -- 172.21.15.12:0/1740573399 >> 172.21.15.12:0/1740573399 conn(0x7fbad40d13e0 msgr2=0x7fbad40d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.418+0000 7fbadc800700 1 -- 172.21.15.12:0/1740573399 shutdown_connections 2024-03-29T10:10:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.418+0000 7fbadc800700 1 -- 172.21.15.12:0/1740573399 wait complete. 2024-03-29T10:10:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.421+0000 7fbadc800700 1 Processor -- start 2024-03-29T10:10:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.421+0000 7fbadc800700 1 -- start start 2024-03-29T10:10:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.422+0000 7fbadc800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbad406c650 0x7fbad40c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.422+0000 7fbadc800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad409aba0 0x7fbad415dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.422+0000 7fbadc800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbad415d710 0x7fbad41467e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.422+0000 7fbadc800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad41f35f0 con 0x7fbad415d710 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.422+0000 7fbadc800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad40c7900 con 0x7fbad406c650 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.422+0000 7fbadc800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad40c6860 con 0x7fbad409aba0 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.422+0000 7fbad9e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbad406c650 0x7fbad40c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.422+0000 7fbada800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbad415d710 0x7fbad41467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.423+0000 7fbad9e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbad406c650 0x7fbad40c5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:56572/0 (socket says 172.21.15.12:56572) 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.423+0000 7fbad9e00700 1 -- 172.21.15.12:0/2806397738 learned_addr learned my addr 172.21.15.12:0/2806397738 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.423+0000 7fbad9400700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad409aba0 0x7fbad415dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.423+0000 7fbad9400700 1 -- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbad406c650 msgr2=0x7fbad40c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.423+0000 7fbad9400700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbad406c650 0x7fbad40c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.423+0000 7fbad9400700 1 -- 172.21.15.12:0/2806397738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbad415d710 msgr2=0x7fbad41467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.423+0000 7fbad9400700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbad415d710 0x7fbad41467e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.423+0000 7fbad9400700 1 -- 172.21.15.12:0/2806397738 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad0009a70 con 0x7fbad409aba0 2024-03-29T10:10:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.423+0000 7fbad9400700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad409aba0 0x7fbad415dd60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbac400c4d0 tx=0x7fbac400c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.423+0000 7fbace000700 1 -- 172.21.15.12:0/2806397738 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac40052d0 con 0x7fbad409aba0 2024-03-29T10:10:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.424+0000 7fbace000700 1 -- 172.21.15.12:0/2806397738 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac4005430 con 0x7fbad409aba0 2024-03-29T10:10:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.424+0000 7fbace000700 1 -- 172.21.15.12:0/2806397738 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4003880 con 0x7fbad409aba0 2024-03-29T10:10:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.424+0000 7fbadc800700 1 -- 172.21.15.12:0/2806397738 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad40c6b40 con 0x7fbad409aba0 2024-03-29T10:10:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.424+0000 7fbadc800700 1 -- 172.21.15.12:0/2806397738 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad40c6ed0 con 0x7fbad409aba0 2024-03-29T10:10:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.425+0000 7fbadc800700 1 -- 172.21.15.12:0/2806397738 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad4067720 con 0x7fbad409aba0 2024-03-29T10:10:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.426+0000 7fbace000700 1 -- 172.21.15.12:0/2806397738 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 42) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fbac4003bf0 con 0x7fbad409aba0 2024-03-29T10:10:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.426+0000 7fbace000700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbab806adb0 0x7fbab806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.427+0000 7fbad9e00700 1 -- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbab806adb0 msgr2=0x7fbab806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:10:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.427+0000 7fbace000700 1 -- 172.21.15.12:0/2806397738 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fbac4088a40 con 0x7fbad409aba0 2024-03-29T10:10:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.427+0000 7fbad9e00700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbab806adb0 0x7fbab806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:10:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.431+0000 7fbace000700 1 -- 172.21.15.12:0/2806397738 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbac4053930 con 0x7fbad409aba0 2024-03-29T10:10:43.486 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.486+0000 7f3fb2c40240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:10:43.535 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:43.536+0000 7f8ecce46240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:10:43.594 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.594+0000 7f3fb2c40240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.606+0000 7fbace000700 1 -- 172.21.15.12:0/2806397738 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mgrmap(e 43) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fbac4053d00 con 0x7fbad409aba0 2024-03-29T10:10:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.607+0000 7fbace000700 1 -- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbab806adb0 msgr2=0x7fbab806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:10:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.607+0000 7fbace000700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbab806adb0 0x7fbab806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.674 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.673+0000 7f3f20e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:43.674 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.673+0000 7f3f20e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:43.675 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.673+0000 7f3f20e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:43.675 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.673+0000 7f3f20e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:43.675 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.673+0000 7f3f20e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:43.675 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.674+0000 7f3f25e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:43.675 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.674+0000 7f3f25e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:43.675 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.674+0000 7f3f25e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:43.675 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.674+0000 7f3f25e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:43.675 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:10:43.674+0000 7f3f25e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:10:43.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.740+0000 7fbadc800700 1 -- 172.21.15.12:0/2806397738 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbad40d0f20 con 0x7fbad409aba0 2024-03-29T10:10:43.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.741+0000 7fbace000700 1 -- 172.21.15.12:0/2806397738 <== mon.2 v2:172.21.15.159:3301/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fbac4073640 con 0x7fbad409aba0 2024-03-29T10:10:43.740 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:43.740 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":38,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/4026715411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":4026715411},{"type":"v1","addr":"172.21.15.159:6836","nonce":4026715411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/225205526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":225205526},{"type":"v1","addr":"172.21.15.12:6834","nonce":225205526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6918,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2358504162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2358504162},{"type":"v1","addr":"172.21.15.159:6837","nonce":2358504162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:17.624968+0000","modified":"2024-03-29T10:10:21.121687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6924},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6924":{"gid":6924,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3359577869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3359577869},{"type":"v1","addr":"172.21.15.12:6835","nonce":3359577869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-29T10:10:43.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.743+0000 7fbadc800700 1 -- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad409aba0 msgr2=0x7fbad415dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.743+0000 7fbadc800700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad409aba0 0x7fbad415dd60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbac400c4d0 tx=0x7fbac400c7e0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.743+0000 7fbadc800700 1 -- 172.21.15.12:0/2806397738 shutdown_connections 2024-03-29T10:10:43.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.743+0000 7fbadc800700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbab806adb0 0x7fbab806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.743+0000 7fbadc800700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbad415d710 0x7fbad41467e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.743+0000 7fbadc800700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbad406c650 0x7fbad40c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.743+0000 7fbadc800700 1 --2- 172.21.15.12:0/2806397738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbad409aba0 0x7fbad415dd60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.743+0000 7fbadc800700 1 -- 172.21.15.12:0/2806397738 >> 172.21.15.12:0/2806397738 conn(0x7fbad40d13e0 msgr2=0x7fbad40cde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:43.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.743+0000 7fbadc800700 1 -- 172.21.15.12:0/2806397738 shutdown_connections 2024-03-29T10:10:43.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.744+0000 7fbadc800700 1 -- 172.21.15.12:0/2806397738 wait complete. 2024-03-29T10:10:43.744 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 38 2024-03-29T10:10:43.765 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:43.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.970+0000 7f3c69600700 1 Processor -- start 2024-03-29T10:10:43.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.970+0000 7f3c69600700 1 -- start start 2024-03-29T10:10:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.972+0000 7f3c69600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 0x7f3c640ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.972+0000 7f3c69600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3c6406dee0 0x7f3c6406e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.972+0000 7f3c69600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640b3890 con 0x7f3c6415da50 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.972+0000 7f3c69600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c640acac0 con 0x7f3c6406dee0 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.972+0000 7f3c69600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c64159f50 con 0x7f3c6415d670 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.972+0000 7f3c63400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3c6406dee0 0x7f3c6406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.973+0000 7f3c63400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3c6406dee0 0x7f3c6406e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35710/0 (socket says 172.21.15.12:35710) 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.973+0000 7f3c63400700 1 -- 172.21.15.12:0/1269476496 learned_addr learned my addr 172.21.15.12:0/1269476496 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.973+0000 7f3c62a00700 1 --2- 172.21.15.12:0/1269476496 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 0x7f3c640ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.973+0000 7f3c63400700 1 -- 172.21.15.12:0/1269476496 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 msgr2=0x7f3c640ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.973+0000 7f3c63400700 1 --2- 172.21.15.12:0/1269476496 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 0x7f3c640ac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.973+0000 7f3c63400700 1 -- 172.21.15.12:0/1269476496 >> v1:172.21.15.159:6789/0 conn(0x7f3c6415da50 legacy=0x7f3c640b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.973+0000 7f3c63400700 1 -- 172.21.15.12:0/1269476496 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c640b35b0 con 0x7f3c6406dee0 2024-03-29T10:10:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.973+0000 7f3c63400700 1 --2- 172.21.15.12:0/1269476496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6406dee0 0x7f3c6406e370 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00d150 tx=0x7f3c5c00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661401433e6f5751 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.973+0000 7f3c61600700 1 -- 172.21.15.12:0/1269476496 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c010070 con 0x7f3c6406dee0 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.974+0000 7f3c61600700 1 -- 172.21.15.12:0/1269476496 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c00db00 con 0x7f3c6406dee0 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.974+0000 7f3c61600700 1 -- 172.21.15.12:0/1269476496 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c016400 con 0x7f3c6406dee0 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.974+0000 7f3c69600700 1 -- 172.21.15.12:0/1269476496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6406dee0 msgr2=0x7f3c6406e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.974+0000 7f3c69600700 1 --2- 172.21.15.12:0/1269476496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6406dee0 0x7f3c6406e370 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00d150 tx=0x7f3c5c00d460 comp rx=0 tx=0).stop 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.974+0000 7f3c69600700 1 -- 172.21.15.12:0/1269476496 shutdown_connections 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.974+0000 7f3c69600700 1 --2- 172.21.15.12:0/1269476496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6406dee0 0x7f3c6406e370 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.974+0000 7f3c69600700 1 --2- 172.21.15.12:0/1269476496 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 0x7f3c640ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.974+0000 7f3c69600700 1 -- 172.21.15.12:0/1269476496 >> 172.21.15.12:0/1269476496 conn(0x7f3c640cf190 msgr2=0x7f3c640cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.974+0000 7f3c69600700 1 -- 172.21.15.12:0/1269476496 shutdown_connections 2024-03-29T10:10:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.974+0000 7f3c69600700 1 -- 172.21.15.12:0/1269476496 wait complete. 2024-03-29T10:10:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.976+0000 7f3c69600700 1 Processor -- start 2024-03-29T10:10:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.976+0000 7f3c69600700 1 -- start start 2024-03-29T10:10:43.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.977+0000 7f3c69600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c6406dee0 0x7f3c641f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:43.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.977+0000 7f3c69600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 0x7f3c640b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:43.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.977+0000 7f3c69600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6415da50 0x7f3c640b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:43.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.977+0000 7f3c69600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c641f5da0 con 0x7f3c6415da50 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.977+0000 7f3c69600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c641f5fb0 con 0x7f3c6406dee0 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.977+0000 7f3c69600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c640cc520 con 0x7f3c6415d670 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c63400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6415da50 0x7f3c640b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c63400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6415da50 0x7f3c640b2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35722/0 (socket says 172.21.15.12:35722) 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c63400700 1 -- 172.21.15.12:0/3247609125 learned_addr learned my addr 172.21.15.12:0/3247609125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c62a00700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c6406dee0 0x7f3c641f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c62000700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 0x7f3c640b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c63400700 1 -- 172.21.15.12:0/3247609125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 msgr2=0x7f3c640b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c63400700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 0x7f3c640b1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c63400700 1 -- 172.21.15.12:0/3247609125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c6406dee0 msgr2=0x7f3c641f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c63400700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c6406dee0 0x7f3c641f5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c63400700 1 -- 172.21.15.12:0/3247609125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c00bd40 con 0x7f3c6415da50 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c62a00700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c6406dee0 0x7f3c641f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c62000700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 0x7f3c640b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.978+0000 7f3c63400700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6415da50 0x7f3c640b2410 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3c6406b930 tx=0x7f3c5c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.979+0000 7f3c5aa00700 1 -- 172.21.15.12:0/3247609125 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c017740 con 0x7f3c6415da50 2024-03-29T10:10:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.979+0000 7f3c5aa00700 1 -- 172.21.15.12:0/3247609125 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c0178c0 con 0x7f3c6415da50 2024-03-29T10:10:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.979+0000 7f3c5aa00700 1 -- 172.21.15.12:0/3247609125 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c017c70 con 0x7f3c6415da50 2024-03-29T10:10:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.979+0000 7f3c69600700 1 -- 172.21.15.12:0/3247609125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c641f6200 con 0x7f3c6415da50 2024-03-29T10:10:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.979+0000 7f3c69600700 1 -- 172.21.15.12:0/3247609125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c641f67c0 con 0x7f3c6415da50 2024-03-29T10:10:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.980+0000 7f3c69600700 1 -- 172.21.15.12:0/3247609125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c64067720 con 0x7f3c6415da50 2024-03-29T10:10:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.981+0000 7f3c5aa00700 1 -- 172.21.15.12:0/3247609125 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 43) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f3c5c004040 con 0x7f3c6415da50 2024-03-29T10:10:43.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.982+0000 7f3c5aa00700 1 -- 172.21.15.12:0/3247609125 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f3c5c08aab0 con 0x7f3c6415da50 2024-03-29T10:10:43.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:43.987+0000 7f3c5aa00700 1 -- 172.21.15.12:0/3247609125 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3c5c055e80 con 0x7f3c6415da50 2024-03-29T10:10:44.019 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:44.019+0000 7f8ecce46240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:10:44.137 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:10:44.138+0000 7f8ecce46240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:10:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.243+0000 7f3c69600700 1 -- 172.21.15.12:0/3247609125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c64071080 con 0x7f3c6415da50 2024-03-29T10:10:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.244+0000 7f3c5aa00700 1 -- 172.21.15.12:0/3247609125 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f3c5c0594a0 con 0x7f3c6415da50 2024-03-29T10:10:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:44.244 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":38,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/4026715411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":4026715411},{"type":"v1","addr":"172.21.15.159:6836","nonce":4026715411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/225205526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":225205526},{"type":"v1","addr":"172.21.15.12:6834","nonce":225205526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6918,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2358504162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2358504162},{"type":"v1","addr":"172.21.15.159:6837","nonce":2358504162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:17.624968+0000","modified":"2024-03-29T10:10:21.121687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6924},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6924":{"gid":6924,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3359577869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3359577869},{"type":"v1","addr":"172.21.15.12:6835","nonce":3359577869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-29T10:10:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.246+0000 7f3c69600700 1 -- 172.21.15.12:0/3247609125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6415da50 msgr2=0x7f3c640b2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.246+0000 7f3c69600700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6415da50 0x7f3c640b2410 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3c6406b930 tx=0x7f3c5c004ad0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.246+0000 7f3c69600700 1 -- 172.21.15.12:0/3247609125 shutdown_connections 2024-03-29T10:10:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.246+0000 7f3c69600700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c6415da50 0x7f3c640b2410 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.246+0000 7f3c69600700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c6406dee0 0x7f3c641f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.246+0000 7f3c69600700 1 --2- 172.21.15.12:0/3247609125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c6415d670 0x7f3c640b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.246+0000 7f3c69600700 1 -- 172.21.15.12:0/3247609125 >> 172.21.15.12:0/3247609125 conn(0x7f3c640cf190 msgr2=0x7f3c6406e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.246+0000 7f3c69600700 1 -- 172.21.15.12:0/3247609125 shutdown_connections 2024-03-29T10:10:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.246+0000 7f3c69600700 1 -- 172.21.15.12:0/3247609125 wait complete. 2024-03-29T10:10:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 38 2024-03-29T10:10:44.264 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:44.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.461+0000 7fe5bd600700 1 Processor -- start 2024-03-29T10:10:44.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.462+0000 7fe5bd600700 1 -- start start 2024-03-29T10:10:44.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5bd600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5b8160f10 0x7fe5b80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5bd600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5b8161590 0x7fe5b8161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5bd600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b80ceda0 con 0x7fe5b80cf230 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5bd600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b8161f80 con 0x7fe5b8160f10 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5bd600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b81612f0 con 0x7fe5b8161590 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5b6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5b8160f10 0x7fe5b80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5b6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5b8160f10 0x7fe5b80d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35732/0 (socket says 172.21.15.12:35732) 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5b6a00700 1 -- 172.21.15.12:0/335509795 learned_addr learned my addr 172.21.15.12:0/335509795 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5b7400700 1 --2- 172.21.15.12:0/335509795 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5b8161590 0x7fe5b8161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5b6a00700 1 -- 172.21.15.12:0/335509795 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5b8161590 msgr2=0x7fe5b8161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5b6a00700 1 --2- 172.21.15.12:0/335509795 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5b8161590 0x7fe5b8161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.463+0000 7fe5b6a00700 1 -- 172.21.15.12:0/335509795 >> v1:172.21.15.12:6789/0 conn(0x7fe5b80cf230 legacy=0x7fe5b80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.464+0000 7fe5b6a00700 1 -- 172.21.15.12:0/335509795 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b80ce470 con 0x7fe5b8160f10 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.464+0000 7fe5b6a00700 1 --2- 172.21.15.12:0/335509795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b8160f10 0x7fe5b80d18b0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80098f0 tx=0x7fe5a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b46e95d2bebb1d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:44.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.464+0000 7fe5b5600700 1 -- 172.21.15.12:0/335509795 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a800b8a0 con 0x7fe5b8160f10 2024-03-29T10:10:44.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.464+0000 7fe5b5600700 1 -- 172.21.15.12:0/335509795 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a800ba00 con 0x7fe5b8160f10 2024-03-29T10:10:44.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.464+0000 7fe5b5600700 1 -- 172.21.15.12:0/335509795 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a800bd90 con 0x7fe5b8160f10 2024-03-29T10:10:44.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.465+0000 7fe5bd600700 1 -- 172.21.15.12:0/335509795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b8160f10 msgr2=0x7fe5b80d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.465+0000 7fe5bd600700 1 --2- 172.21.15.12:0/335509795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b8160f10 0x7fe5b80d18b0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80098f0 tx=0x7fe5a8009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:44.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.465+0000 7fe5bd600700 1 -- 172.21.15.12:0/335509795 shutdown_connections 2024-03-29T10:10:44.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.465+0000 7fe5bd600700 1 --2- 172.21.15.12:0/335509795 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5b8161590 0x7fe5b8161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.465+0000 7fe5bd600700 1 --2- 172.21.15.12:0/335509795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b8160f10 0x7fe5b80d18b0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.465+0000 7fe5bd600700 1 -- 172.21.15.12:0/335509795 >> 172.21.15.12:0/335509795 conn(0x7fe5b808aff0 msgr2=0x7fe5b80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:44.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.465+0000 7fe5bd600700 1 -- 172.21.15.12:0/335509795 shutdown_connections 2024-03-29T10:10:44.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.465+0000 7fe5bd600700 1 -- 172.21.15.12:0/335509795 wait complete. 2024-03-29T10:10:44.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.467+0000 7fe5bd600700 1 Processor -- start 2024-03-29T10:10:44.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.467+0000 7fe5bd600700 1 -- start start 2024-03-29T10:10:44.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.468+0000 7fe5bd600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b80cf230 0x7fe5b81f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5bd600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5b8160f10 0x7fe5b80b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5bd600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5b8161590 0x7fe5b80b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5bd600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b81e7770 con 0x7fe5b80cf230 2024-03-29T10:10:44.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5bd600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b8160080 con 0x7fe5b8161590 2024-03-29T10:10:44.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5bd600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b81e6dd0 con 0x7fe5b8160f10 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b80cf230 0x7fe5b81f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b80cf230 0x7fe5b81f2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35740/0 (socket says 172.21.15.12:35740) 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5b6a00700 1 -- 172.21.15.12:0/2613342905 learned_addr learned my addr 172.21.15.12:0/2613342905 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5b7400700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5b8161590 0x7fe5b80b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5b6000700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5b8160f10 0x7fe5b80b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5b6a00700 1 -- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5b8160f10 msgr2=0x7fe5b80b65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5b6a00700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5b8160f10 0x7fe5b80b65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5b6a00700 1 -- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5b8161590 msgr2=0x7fe5b80b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.469+0000 7fe5b6a00700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5b8161590 0x7fe5b80b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.470+0000 7fe5b6a00700 1 -- 172.21.15.12:0/2613342905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a8009550 con 0x7fe5b80cf230 2024-03-29T10:10:44.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.470+0000 7fe5b6a00700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b80cf230 0x7fe5b81f2e90 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8009c40 tx=0x7fe5a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:44.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.470+0000 7fe5aea00700 1 -- 172.21.15.12:0/2613342905 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a80036a0 con 0x7fe5b80cf230 2024-03-29T10:10:44.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.470+0000 7fe5aea00700 1 -- 172.21.15.12:0/2613342905 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a801e070 con 0x7fe5b80cf230 2024-03-29T10:10:44.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.470+0000 7fe5aea00700 1 -- 172.21.15.12:0/2613342905 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8003a60 con 0x7fe5b80cf230 2024-03-29T10:10:44.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.470+0000 7fe5bd600700 1 -- 172.21.15.12:0/2613342905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b81e70b0 con 0x7fe5b80cf230 2024-03-29T10:10:44.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.470+0000 7fe5bd600700 1 -- 172.21.15.12:0/2613342905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b8158e20 con 0x7fe5b80cf230 2024-03-29T10:10:44.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.471+0000 7fe5bd600700 1 -- 172.21.15.12:0/2613342905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b8067720 con 0x7fe5b80cf230 2024-03-29T10:10:44.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.472+0000 7fe5aea00700 1 -- 172.21.15.12:0/2613342905 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 43) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fe5a8003bc0 con 0x7fe5b80cf230 2024-03-29T10:10:44.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.473+0000 7fe5aea00700 1 -- 172.21.15.12:0/2613342905 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6311+0+0 (secure 0 0 0) 0x7fe5a8020070 con 0x7fe5b80cf230 2024-03-29T10:10:44.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.478+0000 7fe5aea00700 1 -- 172.21.15.12:0/2613342905 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5a8055890 con 0x7fe5b80cf230 2024-03-29T10:10:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.610+0000 7fe5aea00700 1 -- 172.21.15.12:0/2613342905 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 44) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fe5a8055340 con 0x7fe5b80cf230 2024-03-29T10:10:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.610+0000 7fe5aea00700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe598036100 0x7fe5980722c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.612+0000 7fe5b6000700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe598036100 0x7fe5980722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.616+0000 7fe5b6000700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe598036100 0x7fe5980722c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80cfc90 tx=0x7fe5a4009040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.696+0000 7fe5bd600700 1 -- 172.21.15.12:0/2613342905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5b815caf0 con 0x7fe5b80cf230 2024-03-29T10:10:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.697+0000 7fe5aea00700 1 -- 172.21.15.12:0/2613342905 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16974 (secure 0 0 0) 0x7fe5a805c8f0 con 0x7fe5b80cf230 2024-03-29T10:10:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:44.697 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":96,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:43.598739+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:16.040572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:16.605686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/4005962473":"2024-03-30T10:10:43.598680+0000","172.21.15.159:0/2424534213":"2024-03-30T10:10:43.598680+0000","172.21.15.159:0/1430041293":"2024-03-30T10:10:43.598680+0000","172.21.15.159:0/4231364391":"2024-03-30T10:10:43.598680+0000","172.21.15.159:0/706246125":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/699588224":"2024-03-30T10:10:43.598680+0000","172.21.15.159:0/2287631103":"2024-03-30T10:10:34.825193+0000","172.21.15.159:6833/43444":"2024-03-30T10:10:34.825193+0000","172.21.15.159:6832/43444":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/356119888":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/3972721365":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/1459270455":"2024-03-30T10:10:34.825193+0000","172.21.15.159:0/3398664705":"2024-03-30T10:10:43.598680+0000","172.21.15.159:0/2421061145":"2024-03-30T10:10:34.825193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 -- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe598036100 msgr2=0x7fe5980722c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe598036100 0x7fe5980722c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80cfc90 tx=0x7fe5a4009040 comp rx=0 tx=0).stop 2024-03-29T10:10:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 -- 172.21.15.12:0/2613342905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b80cf230 msgr2=0x7fe5b81f2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b80cf230 0x7fe5b81f2e90 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8009c40 tx=0x7fe5a8004750 comp rx=0 tx=0).stop 2024-03-29T10:10:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 -- 172.21.15.12:0/2613342905 shutdown_connections 2024-03-29T10:10:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe598036100 0x7fe5980722c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5b8161590 0x7fe5b80b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5b80cf230 0x7fe5b81f2e90 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 --2- 172.21.15.12:0/2613342905 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5b8160f10 0x7fe5b80b65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 -- 172.21.15.12:0/2613342905 >> 172.21.15.12:0/2613342905 conn(0x7fe5b808aff0 msgr2=0x7fe5b80b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 -- 172.21.15.12:0/2613342905 shutdown_connections 2024-03-29T10:10:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.700+0000 7fe5bd600700 1 -- 172.21.15.12:0/2613342905 wait complete. 2024-03-29T10:10:44.718 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T10:10:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.917+0000 7f15c7000700 1 Processor -- start 2024-03-29T10:10:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.917+0000 7f15c7000700 1 -- start start 2024-03-29T10:10:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15c7000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15c015a5c0 0x7f15c00709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15c7000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015dfe0 0x7f15c00ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15c7000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c00cb310 con 0x7f15c00ce670 2024-03-29T10:10:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15c7000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c00ceb00 con 0x7f15c015a5c0 2024-03-29T10:10:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15c7000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c0070f00 con 0x7f15c015dfe0 2024-03-29T10:10:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15c5000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f15c00ce670 0x7f15c00b2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49050/0 (socket says 172.21.15.12:49050) 2024-03-29T10:10:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15bfe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15c015a5c0 0x7f15c00709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15c5000700 1 -- 172.21.15.12:0/2870478321 learned_addr learned my addr 172.21.15.12:0/2870478321 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15bf400700 1 --2- 172.21.15.12:0/2870478321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015dfe0 0x7f15c00ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15bea00700 1 -- 172.21.15.12:0/2870478321 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56716520 0 0) 0x7f15c00cb310 con 0x7f15c00ce670 2024-03-29T10:10:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15bfe00700 1 -- 172.21.15.12:0/2870478321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015dfe0 msgr2=0x7f15c00ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15bfe00700 1 --2- 172.21.15.12:0/2870478321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015dfe0 0x7f15c00ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15bfe00700 1 -- 172.21.15.12:0/2870478321 >> v1:172.21.15.12:6789/0 conn(0x7f15c00ce670 legacy=0x7f15c00b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15bfe00700 1 -- 172.21.15.12:0/2870478321 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c00cca90 con 0x7f15c015a5c0 2024-03-29T10:10:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.919+0000 7f15bf400700 1 --2- 172.21.15.12:0/2870478321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015dfe0 0x7f15c00ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.920+0000 7f15bfe00700 1 --2- 172.21.15.12:0/2870478321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015a5c0 0x7f15c00709c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009840 tx=0x7f15b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21307f0ce908c474 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.920+0000 7f15bea00700 1 -- 172.21.15.12:0/2870478321 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b0011830 con 0x7f15c015a5c0 2024-03-29T10:10:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.920+0000 7f15bea00700 1 -- 172.21.15.12:0/2870478321 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b0011990 con 0x7f15c015a5c0 2024-03-29T10:10:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.920+0000 7f15bea00700 1 -- 172.21.15.12:0/2870478321 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b0011d20 con 0x7f15c015a5c0 2024-03-29T10:10:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.920+0000 7f15c7000700 1 -- 172.21.15.12:0/2870478321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015a5c0 msgr2=0x7f15c00709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.920+0000 7f15c7000700 1 --2- 172.21.15.12:0/2870478321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015a5c0 0x7f15c00709c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009840 tx=0x7f15b0009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.921+0000 7f15c7000700 1 -- 172.21.15.12:0/2870478321 shutdown_connections 2024-03-29T10:10:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.921+0000 7f15c7000700 1 --2- 172.21.15.12:0/2870478321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015a5c0 0x7f15c00709c0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.921+0000 7f15c7000700 1 --2- 172.21.15.12:0/2870478321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015dfe0 0x7f15c00ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.921+0000 7f15c7000700 1 -- 172.21.15.12:0/2870478321 >> 172.21.15.12:0/2870478321 conn(0x7f15c00b0d70 msgr2=0x7f15c00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.921+0000 7f15c7000700 1 -- 172.21.15.12:0/2870478321 shutdown_connections 2024-03-29T10:10:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.921+0000 7f15c7000700 1 -- 172.21.15.12:0/2870478321 wait complete. 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.922+0000 7f15c7000700 1 Processor -- start 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.923+0000 7f15c7000700 1 -- start start 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.923+0000 7f15c7000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15c00ce670 0x7f15c00adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.923+0000 7f15c7000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015a5c0 0x7f15c00ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c7000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015dfe0 0x7f15c014d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c7000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c0157270 con 0x7f15c015dfe0 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c7000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00ac590 con 0x7f15c00ce670 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c7000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c014dac0 con 0x7f15c015a5c0 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c5000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015dfe0 0x7f15c014d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c5000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015dfe0 0x7f15c014d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35760/0 (socket says 172.21.15.12:35760) 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c5000700 1 -- 172.21.15.12:0/700610197 learned_addr learned my addr 172.21.15.12:0/700610197 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15bf400700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015a5c0 0x7f15c00ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15bfe00700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15c00ce670 0x7f15c00adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c5000700 1 -- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015a5c0 msgr2=0x7f15c00ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c5000700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015a5c0 0x7f15c00ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c5000700 1 -- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15c00ce670 msgr2=0x7f15c00adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c5000700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15c00ce670 0x7f15c00adc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c5000700 1 -- 172.21.15.12:0/700610197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b00094a0 con 0x7f15c015dfe0 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15bf400700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015a5c0 0x7f15c00ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.924+0000 7f15c5000700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015dfe0 0x7f15c014d580 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f15b8002230 tx=0x7f15b800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.925+0000 7f15b7e00700 1 -- 172.21.15.12:0/700610197 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b800ac20 con 0x7f15c015dfe0 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.925+0000 7f15b7e00700 1 -- 172.21.15.12:0/700610197 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b800ada0 con 0x7f15c015dfe0 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.925+0000 7f15b7e00700 1 -- 172.21.15.12:0/700610197 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8007310 con 0x7f15c015dfe0 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.925+0000 7f15c7000700 1 -- 172.21.15.12:0/700610197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c00ac7a0 con 0x7f15c015dfe0 2024-03-29T10:10:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.925+0000 7f15c7000700 1 -- 172.21.15.12:0/700610197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c00acd60 con 0x7f15c015dfe0 2024-03-29T10:10:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.926+0000 7f15c7000700 1 -- 172.21.15.12:0/700610197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c0156fd0 con 0x7f15c015dfe0 2024-03-29T10:10:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.927+0000 7f15b7e00700 1 -- 172.21.15.12:0/700610197 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 44) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f15b8007470 con 0x7f15c015dfe0 2024-03-29T10:10:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.928+0000 7f15b7e00700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15a006aae0 0x7f15a006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.928+0000 7f15b7e00700 1 -- 172.21.15.12:0/700610197 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f15b8088df0 con 0x7f15c015dfe0 2024-03-29T10:10:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.928+0000 7f15bfe00700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15a006aae0 0x7f15a006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.929+0000 7f15bfe00700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15a006aae0 0x7f15a006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009b90 tx=0x7f15b0008030 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:44.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:44.932+0000 7f15b7e00700 1 -- 172.21.15.12:0/700610197 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15c0156fd0 con 0x7f15c015dfe0 2024-03-29T10:10:45.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.194+0000 7f15c7000700 1 -- 172.21.15.12:0/700610197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f15c0156fd0 con 0x7f15c015dfe0 2024-03-29T10:10:45.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T10:10:45.195+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T10:10:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.207+0000 7f15b7e00700 1 -- 172.21.15.12:0/700610197 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v39) v1 ==== 161+0+0 (secure 0 0 0) 0x7f15c0156fd0 con 0x7f15c015dfe0 2024-03-29T10:10:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 -- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15a006aae0 msgr2=0x7f15a006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15a006aae0 0x7f15a006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009b90 tx=0x7f15b0008030 comp rx=0 tx=0).stop 2024-03-29T10:10:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 -- 172.21.15.12:0/700610197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015dfe0 msgr2=0x7f15c014d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015dfe0 0x7f15c014d580 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f15b8002230 tx=0x7f15b800a060 comp rx=0 tx=0).stop 2024-03-29T10:10:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 -- 172.21.15.12:0/700610197 shutdown_connections 2024-03-29T10:10:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f15a006aae0 0x7f15a006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c015dfe0 0x7f15c014d580 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15c00ce670 0x7f15c00adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 --2- 172.21.15.12:0/700610197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15c015a5c0 0x7f15c00ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 -- 172.21.15.12:0/700610197 >> 172.21.15.12:0/700610197 conn(0x7f15c00b0d70 msgr2=0x7f15c00aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:45.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 -- 172.21.15.12:0/700610197 shutdown_connections 2024-03-29T10:10:45.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.210+0000 7f15c7000700 1 -- 172.21.15.12:0/700610197 wait complete. 2024-03-29T10:10:45.211 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T10:10:45.215 INFO:tasks.ceph.mds.a.smithi012.stderr: -4079> 2024-03-29T10:10:18.093+0000 7fe908200700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:10:45.228 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T10:10:45.266 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:10:45.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.429+0000 7f87a7200700 1 Processor -- start 2024-03-29T10:10:45.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.429+0000 7f87a7200700 1 -- start start 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.430+0000 7f87a7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a015d9d0 0x7f87a00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.430+0000 7f87a7200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f87a00d16a0 0x7f87a00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.430+0000 7f87a7200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f87a00d2370 0x7f87a015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.430+0000 7f87a7200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a00d2820 con 0x7f87a00d16a0 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.430+0000 7f87a7200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a0161420 con 0x7f87a00d2370 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.430+0000 7f87a7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a0161700 con 0x7f87a015d9d0 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.430+0000 7f879f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f87a00d16a0 0x7f87a00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.430+0000 7f879f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f87a00d16a0 0x7f87a00d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35786/0 (socket says 172.21.15.12:35786) 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.430+0000 7f879f400700 1 -- 172.21.15.12:0/4016218153 learned_addr learned my addr 172.21.15.12:0/4016218153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879fe00700 1 --2- 172.21.15.12:0/4016218153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a015d9d0 0x7f87a00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f87a5200700 1 --2- 172.21.15.12:0/4016218153 >> v2:172.21.15.159:3300/0 conn(0x7f87a00d2370 0x7f87a015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:45.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879f400700 1 -- 172.21.15.12:0/4016218153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a015d9d0 msgr2=0x7f87a00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879f400700 1 --2- 172.21.15.12:0/4016218153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a015d9d0 0x7f87a00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879f400700 1 -- 172.21.15.12:0/4016218153 >> v2:172.21.15.159:3300/0 conn(0x7f87a00d2370 msgr2=0x7f87a015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879f400700 1 --2- 172.21.15.12:0/4016218153 >> v2:172.21.15.159:3300/0 conn(0x7f87a00d2370 0x7f87a015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879f400700 1 -- 172.21.15.12:0/4016218153 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a01619e0 con 0x7f87a00d16a0 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879fe00700 1 --2- 172.21.15.12:0/4016218153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a015d9d0 0x7f87a00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f87a5200700 1 --2- 172.21.15.12:0/4016218153 >> v2:172.21.15.159:3300/0 conn(0x7f87a00d2370 0x7f87a015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879f400700 1 --2- 172.21.15.12:0/4016218153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 0x7f87a00d1e30 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f87a00cdc00 tx=0x7f878c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b51a4a0ace2dad server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879ea00700 1 -- 172.21.15.12:0/4016218153 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c005e30 con 0x7f87a00d16a0 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879ea00700 1 -- 172.21.15.12:0/4016218153 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878c004030 con 0x7f87a00d16a0 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.431+0000 7f879ea00700 1 -- 172.21.15.12:0/4016218153 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c0043e0 con 0x7f87a00d16a0 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.432+0000 7f87a7200700 1 -- 172.21.15.12:0/4016218153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 msgr2=0x7f87a00d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.432+0000 7f87a7200700 1 --2- 172.21.15.12:0/4016218153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 0x7f87a00d1e30 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f87a00cdc00 tx=0x7f878c009bd0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.432+0000 7f87a7200700 1 -- 172.21.15.12:0/4016218153 shutdown_connections 2024-03-29T10:10:45.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.432+0000 7f87a7200700 1 --2- 172.21.15.12:0/4016218153 >> v2:172.21.15.159:3300/0 conn(0x7f87a00d2370 0x7f87a015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.432+0000 7f87a7200700 1 --2- 172.21.15.12:0/4016218153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a015d9d0 0x7f87a00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.432+0000 7f87a7200700 1 --2- 172.21.15.12:0/4016218153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 0x7f87a00d1e30 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.432+0000 7f87a7200700 1 -- 172.21.15.12:0/4016218153 >> 172.21.15.12:0/4016218153 conn(0x7f87a008aff0 msgr2=0x7f87a00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:45.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.432+0000 7f87a7200700 1 -- 172.21.15.12:0/4016218153 shutdown_connections 2024-03-29T10:10:45.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.432+0000 7f87a7200700 1 -- 172.21.15.12:0/4016218153 wait complete. 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.434+0000 7f87a7200700 1 Processor -- start 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.434+0000 7f87a7200700 1 -- start start 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f87a7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 0x7f87a00cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f87a7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00d2370 0x7f87a01529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f87a7200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a015d9d0 0x7f87a00b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f87a7200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a00b71a0 con 0x7f87a00d16a0 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f87a7200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a015ce90 con 0x7f87a015d9d0 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f87a7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a00b6520 con 0x7f87a00d2370 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f879fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 0x7f87a00cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f879fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 0x7f87a00cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35798/0 (socket says 172.21.15.12:35798) 2024-03-29T10:10:45.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f879fe00700 1 -- 172.21.15.12:0/187573724 learned_addr learned my addr 172.21.15.12:0/187573724 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f879f400700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00d2370 0x7f87a01529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.435+0000 7f87a5200700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a015d9d0 0x7f87a00b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f879fe00700 1 -- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00d2370 msgr2=0x7f87a01529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f879fe00700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00d2370 0x7f87a01529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f879fe00700 1 -- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a015d9d0 msgr2=0x7f87a00b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f879fe00700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a015d9d0 0x7f87a00b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f879fe00700 1 -- 172.21.15.12:0/187573724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c009500 con 0x7f87a00d16a0 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f879fe00700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 0x7f87a00cd090 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f879000c7f0 tx=0x7f879000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f8797e00700 1 -- 172.21.15.12:0/187573724 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790011800 con 0x7f87a00d16a0 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f8797e00700 1 -- 172.21.15.12:0/187573724 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8790011960 con 0x7f87a00d16a0 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f8797e00700 1 -- 172.21.15.12:0/187573724 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790011d10 con 0x7f87a00d16a0 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f87a7200700 1 -- 172.21.15.12:0/187573724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a00b6800 con 0x7f87a00d16a0 2024-03-29T10:10:45.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.436+0000 7f87a7200700 1 -- 172.21.15.12:0/187573724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a00b6dc0 con 0x7f87a00d16a0 2024-03-29T10:10:45.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.437+0000 7f87a7200700 1 -- 172.21.15.12:0/187573724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a00d1ca0 con 0x7f87a00d16a0 2024-03-29T10:10:45.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.438+0000 7f8797e00700 1 -- 172.21.15.12:0/187573724 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 44) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f8790018360 con 0x7f87a00d16a0 2024-03-29T10:10:45.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.439+0000 7f8797e00700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006ab10 0x7f878006cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:45.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.439+0000 7f8797e00700 1 -- 172.21.15.12:0/187573724 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8790089260 con 0x7f87a00d16a0 2024-03-29T10:10:45.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.439+0000 7f879f400700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006ab10 0x7f878006cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:45.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.440+0000 7f879f400700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006ab10 0x7f878006cfd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f87a00cdc00 tx=0x7f878c008070 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:45.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.444+0000 7f8797e00700 1 -- 172.21.15.12:0/187573724 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8790054400 con 0x7f87a00d16a0 2024-03-29T10:10:45.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:45.710+0000 7f87a7200700 1 -- 172.21.15.12:0/187573724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f87a00b72e0 con 0x7f87a00d16a0 2024-03-29T10:10:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.210+0000 7f8797e00700 1 -- 172.21.15.12:0/187573724 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v40) v1 ==== 104+0+0 (secure 0 0 0) 0x7f87a00b72e0 con 0x7f87a00d16a0 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.212+0000 7f87a7200700 1 -- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006ab10 msgr2=0x7f878006cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.212+0000 7f87a7200700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006ab10 0x7f878006cfd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f87a00cdc00 tx=0x7f878c008070 comp rx=0 tx=0).stop 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.212+0000 7f87a7200700 1 -- 172.21.15.12:0/187573724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 msgr2=0x7f87a00cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.212+0000 7f87a7200700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 0x7f87a00cd090 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f879000c7f0 tx=0x7f879000cbb0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.213+0000 7f87a7200700 1 -- 172.21.15.12:0/187573724 shutdown_connections 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.213+0000 7f87a7200700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006ab10 0x7f878006cfd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.213+0000 7f87a7200700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a015d9d0 0x7f87a00b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.213+0000 7f87a7200700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00d16a0 0x7f87a00cd090 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.213+0000 7f87a7200700 1 --2- 172.21.15.12:0/187573724 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00d2370 0x7f87a01529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.213+0000 7f87a7200700 1 -- 172.21.15.12:0/187573724 >> 172.21.15.12:0/187573724 conn(0x7f87a008aff0 msgr2=0x7f87a00d39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:46.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.213+0000 7f87a7200700 1 -- 172.21.15.12:0/187573724 shutdown_connections 2024-03-29T10:10:46.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.213+0000 7f87a7200700 1 -- 172.21.15.12:0/187573724 wait complete. 2024-03-29T10:10:46.235 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T10:10:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.480+0000 7f6675000700 1 Processor -- start 2024-03-29T10:10:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.480+0000 7f6675000700 1 -- start start 2024-03-29T10:10:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.481+0000 7f6675000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f667015ddf0 0x7f66700ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.481+0000 7f6675000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 0x7f66700d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.481+0000 7f6675000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66700d2710 0x7f66701612f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.481+0000 7f6675000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66700d2b80 con 0x7f66700d2710 2024-03-29T10:10:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.481+0000 7f6675000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667015ec60 con 0x7f667015ddf0 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.481+0000 7f6675000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66700ce8a0 con 0x7f667015e1d0 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66700d2710 0x7f66701612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66700d2710 0x7f66701612f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35812/0 (socket says 172.21.15.12:35812) 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666e800700 1 -- 172.21.15.12:0/359997006 learned_addr learned my addr 172.21.15.12:0/359997006 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666d400700 1 --2- 172.21.15.12:0/359997006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 0x7f66700d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666de00700 1 --2- 172.21.15.12:0/359997006 >> v2:172.21.15.159:3300/0 conn(0x7f667015ddf0 0x7f66700ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666e800700 1 -- 172.21.15.12:0/359997006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 msgr2=0x7f66700d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666e800700 1 --2- 172.21.15.12:0/359997006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 0x7f66700d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666e800700 1 -- 172.21.15.12:0/359997006 >> v2:172.21.15.159:3300/0 conn(0x7f667015ddf0 msgr2=0x7f66700ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666e800700 1 --2- 172.21.15.12:0/359997006 >> v2:172.21.15.159:3300/0 conn(0x7f667015ddf0 0x7f66700ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666e800700 1 -- 172.21.15.12:0/359997006 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66700cdc90 con 0x7f66700d2710 2024-03-29T10:10:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.482+0000 7f666e800700 1 --2- 172.21.15.12:0/359997006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66700d2710 0x7f66701612f0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f667006b3e0 tx=0x7f666800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e555b6c327505abe server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f666ca00700 1 -- 172.21.15.12:0/359997006 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666800e070 con 0x7f66700d2710 2024-03-29T10:10:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f666ca00700 1 -- 172.21.15.12:0/359997006 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666800b9c0 con 0x7f66700d2710 2024-03-29T10:10:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f666ca00700 1 -- 172.21.15.12:0/359997006 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666800bd70 con 0x7f66700d2710 2024-03-29T10:10:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f6675000700 1 -- 172.21.15.12:0/359997006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66700d2710 msgr2=0x7f66701612f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f6675000700 1 --2- 172.21.15.12:0/359997006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66700d2710 0x7f66701612f0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f667006b3e0 tx=0x7f666800b330 comp rx=0 tx=0).stop 2024-03-29T10:10:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f6675000700 1 -- 172.21.15.12:0/359997006 shutdown_connections 2024-03-29T10:10:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f6675000700 1 --2- 172.21.15.12:0/359997006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66700d2710 0x7f66701612f0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f6675000700 1 --2- 172.21.15.12:0/359997006 >> v2:172.21.15.159:3300/0 conn(0x7f667015ddf0 0x7f66700ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f6675000700 1 --2- 172.21.15.12:0/359997006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 0x7f66700d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f6675000700 1 -- 172.21.15.12:0/359997006 >> 172.21.15.12:0/359997006 conn(0x7f667008aff0 msgr2=0x7f66700d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f6675000700 1 -- 172.21.15.12:0/359997006 shutdown_connections 2024-03-29T10:10:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.483+0000 7f6675000700 1 -- 172.21.15.12:0/359997006 wait complete. 2024-03-29T10:10:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.485+0000 7f6675000700 1 Processor -- start 2024-03-29T10:10:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.485+0000 7f6675000700 1 -- start start 2024-03-29T10:10:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.486+0000 7f6675000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66700d2710 0x7f66700cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.486+0000 7f6675000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f667015ddf0 0x7f66700c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.486+0000 7f6675000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 0x7f6670156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.486+0000 7f6675000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66701f7720 con 0x7f667015ddf0 2024-03-29T10:10:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.486+0000 7f6675000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66700ca5e0 con 0x7f66700d2710 2024-03-29T10:10:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.486+0000 7f6675000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66700cb8d0 con 0x7f667015e1d0 2024-03-29T10:10:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f667015ddf0 0x7f66700c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f667015ddf0 0x7f66700c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35824/0 (socket says 172.21.15.12:35824) 2024-03-29T10:10:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666d400700 1 -- 172.21.15.12:0/1996839185 learned_addr learned my addr 172.21.15.12:0/1996839185 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666de00700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66700d2710 0x7f66700cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666e800700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 0x7f6670156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666d400700 1 -- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 msgr2=0x7f6670156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666d400700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 0x7f6670156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666d400700 1 -- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66700d2710 msgr2=0x7f66700cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666d400700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66700d2710 0x7f66700cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666d400700 1 -- 172.21.15.12:0/1996839185 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6668009b30 con 0x7f667015ddf0 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666e800700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 0x7f6670156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f666d400700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f667015ddf0 0x7f66700c8d90 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f665c00d800 tx=0x7f665c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f6666000700 1 -- 172.21.15.12:0/1996839185 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c0125f0 con 0x7f667015ddf0 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f6666000700 1 -- 172.21.15.12:0/1996839185 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665c012750 con 0x7f667015ddf0 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f6675000700 1 -- 172.21.15.12:0/1996839185 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66700ca830 con 0x7f667015ddf0 2024-03-29T10:10:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.487+0000 7f6666000700 1 -- 172.21.15.12:0/1996839185 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c012b00 con 0x7f667015ddf0 2024-03-29T10:10:46.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.488+0000 7f6675000700 1 -- 172.21.15.12:0/1996839185 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66700cb0d0 con 0x7f667015ddf0 2024-03-29T10:10:46.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.489+0000 7f6675000700 1 -- 172.21.15.12:0/1996839185 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66700cac20 con 0x7f667015ddf0 2024-03-29T10:10:46.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.489+0000 7f6666000700 1 -- 172.21.15.12:0/1996839185 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 45) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f665c012c60 con 0x7f667015ddf0 2024-03-29T10:10:46.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.491+0000 7f6666000700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f665006ad20 0x7f665006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:46.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.491+0000 7f6666000700 1 -- 172.21.15.12:0/1996839185 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f665c089930 con 0x7f667015ddf0 2024-03-29T10:10:46.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.491+0000 7f666de00700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f665006ad20 0x7f665006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:46.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.492+0000 7f666de00700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f665006ad20 0x7f665006d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6660006b20 tx=0x7f6660008040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.495+0000 7f6666000700 1 -- 172.21.15.12:0/1996839185 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f665c054770 con 0x7f667015ddf0 2024-03-29T10:10:46.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:46.709+0000 7f6675000700 1 -- 172.21.15.12:0/1996839185 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6670067720 con 0x7f667015ddf0 2024-03-29T10:10:47.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.614+0000 7f6666000700 1 -- 172.21.15.12:0/1996839185 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) v1 ==== 181+0+0 (secure 0 0 0) 0x7f665c057d90 con 0x7f667015ddf0 2024-03-29T10:10:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.617+0000 7f6675000700 1 -- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f665006ad20 msgr2=0x7f665006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.617+0000 7f6675000700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f665006ad20 0x7f665006d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6660006b20 tx=0x7f6660008040 comp rx=0 tx=0).stop 2024-03-29T10:10:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.617+0000 7f6675000700 1 -- 172.21.15.12:0/1996839185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f667015ddf0 msgr2=0x7f66700c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.617+0000 7f6675000700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f667015ddf0 0x7f66700c8d90 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f665c00d800 tx=0x7f665c00db10 comp rx=0 tx=0).stop 2024-03-29T10:10:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.617+0000 7f6675000700 1 -- 172.21.15.12:0/1996839185 shutdown_connections 2024-03-29T10:10:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.617+0000 7f6675000700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f665006ad20 0x7f665006d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.617+0000 7f6675000700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667015e1d0 0x7f6670156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.617+0000 7f6675000700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66700d2710 0x7f66700cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.617+0000 7f6675000700 1 --2- 172.21.15.12:0/1996839185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f667015ddf0 0x7f66700c8d90 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.617+0000 7f6675000700 1 -- 172.21.15.12:0/1996839185 >> 172.21.15.12:0/1996839185 conn(0x7f667008aff0 msgr2=0x7f667006cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.618+0000 7f6675000700 1 -- 172.21.15.12:0/1996839185 shutdown_connections 2024-03-29T10:10:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.618+0000 7f6675000700 1 -- 172.21.15.12:0/1996839185 wait complete. 2024-03-29T10:10:47.619 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T10:10:47.639 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T10:10:47.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.848+0000 7f2ffea00700 1 Processor -- start 2024-03-29T10:10:47.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.848+0000 7f2ffea00700 1 -- start start 2024-03-29T10:10:47.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.849+0000 7f2ffea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2ff80b56e0 0x7f2ff80d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:47.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.849+0000 7f2ffea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff80ceb70 con 0x7f2ff815d7f0 2024-03-29T10:10:47.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.849+0000 7f2ffea00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff80cf000 con 0x7f2ff80d1770 2024-03-29T10:10:47.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.849+0000 7f2ffea00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff815dbd0 con 0x7f2ff80b56e0 2024-03-29T10:10:47.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.849+0000 7f2ff7e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2ff815d7f0 0x7f2ff80b4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49064/0 (socket says 172.21.15.12:49064) 2024-03-29T10:10:47.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.849+0000 7f2ff7e00700 1 -- 172.21.15.12:0/74558208 learned_addr learned my addr 172.21.15.12:0/74558208 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276241614 0 0) 0x7f2ff80ceb70 con 0x7f2ff815d7f0 2024-03-29T10:10:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fdc0035b0 con 0x7f2ff815d7f0 2024-03-29T10:10:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff7400700 1 --2- 172.21.15.12:0/74558208 >> v2:172.21.15.159:3300/0 conn(0x7f2ff80b56e0 0x7f2ff80d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2732711813 0 0) 0x7f2fdc0035b0 con 0x7f2ff815d7f0 2024-03-29T10:10:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 >> v2:172.21.15.159:3300/0 conn(0x7f2ff80b56e0 msgr2=0x7f2ff80d25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 --2- 172.21.15.12:0/74558208 >> v2:172.21.15.159:3300/0 conn(0x7f2ff80b56e0 0x7f2ff80d25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 >> v1:172.21.15.159:6789/0 conn(0x7f2ff80d1770 legacy=0x7f2ff806ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff80cdf60 con 0x7f2ff815d7f0 2024-03-29T10:10:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f2fe8001670 con 0x7f2ff815d7f0 2024-03-29T10:10:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 >> v1:172.21.15.12:6789/0 conn(0x7f2ff815d7f0 legacy=0x7f2ff80b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:47.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fdc003a50 0x7f2fdc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:47.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fdc004850 0x7f2fdc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:47.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fdc005210 0x7f2fdc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:47.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc00c0e0 con 0x7f2fdc003a50 2024-03-29T10:10:47.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc00c250 con 0x7f2fdc005210 2024-03-29T10:10:47.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.850+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc00c410 con 0x7f2fdc004850 2024-03-29T10:10:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.851+0000 7f2ff7e00700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fdc003a50 0x7f2fdc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.851+0000 7f2ff7400700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fdc004850 0x7f2fdc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.851+0000 7f2ffca00700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fdc005210 0x7f2fdc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.851+0000 7f2ff7e00700 1 -- 172.21.15.12:0/74558208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fdc004850 msgr2=0x7f2fdc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.851+0000 7f2ff7e00700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fdc004850 0x7f2fdc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.851+0000 7f2ff7e00700 1 -- 172.21.15.12:0/74558208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fdc005210 msgr2=0x7f2fdc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.851+0000 7f2ff7e00700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fdc005210 0x7f2fdc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.851+0000 7f2ffca00700 1 -- 172.21.15.12:0/74558208 reap_dead start 2024-03-29T10:10:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.851+0000 7f2ff7e00700 1 -- 172.21.15.12:0/74558208 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc0035b0 con 0x7f2fdc003a50 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.851+0000 7f2ff7e00700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fdc003a50 0x7f2fdc003e70 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800c160 tx=0x7f2fdc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d3b8159c67f6cb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.852+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8007b70 con 0x7f2fdc003a50 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.852+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe800cd20 con 0x7f2fdc003a50 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.852+0000 7f2ff6a00700 1 -- 172.21.15.12:0/74558208 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8005ba0 con 0x7f2fdc003a50 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.852+0000 7f2ffea00700 1 -- 172.21.15.12:0/74558208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fdc003a50 msgr2=0x7f2fdc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.852+0000 7f2ffea00700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fdc003a50 0x7f2fdc003e70 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800c160 tx=0x7f2fdc005880 comp rx=0 tx=0).stop 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.852+0000 7f2ffea00700 1 -- 172.21.15.12:0/74558208 shutdown_connections 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.852+0000 7f2ffea00700 1 --2- 172.21.15.12:0/74558208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fdc003a50 0x7f2fdc003e70 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.852+0000 7f2ffea00700 1 -- 172.21.15.12:0/74558208 >> 172.21.15.12:0/74558208 conn(0x7f2ff808aff0 msgr2=0x7f2ff80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.852+0000 7f2ffea00700 1 -- 172.21.15.12:0/74558208 shutdown_connections 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.852+0000 7f2ffea00700 1 -- 172.21.15.12:0/74558208 wait complete. 2024-03-29T10:10:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.854+0000 7f2ffea00700 1 Processor -- start 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.854+0000 7f2ffea00700 1 -- start start 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.855+0000 7f2ffea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fdc003a50 0x7f2ff80ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.855+0000 7f2ffea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff80cbf60 0x7f2ff80cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.855+0000 7f2ffea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ff80c9360 0x7f2ff80c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.855+0000 7f2ffea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff80c71c0 con 0x7f2ff80cbf60 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.855+0000 7f2ffea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff80cab60 con 0x7f2fdc003a50 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.855+0000 7f2ffea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff815fcd0 con 0x7f2ff80c9360 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.855+0000 7f2ff7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff80cbf60 0x7f2ff80cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.855+0000 7f2ff7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff80cbf60 0x7f2ff80cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35844/0 (socket says 172.21.15.12:35844) 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.855+0000 7f2ff7400700 1 -- 172.21.15.12:0/2099544100 learned_addr learned my addr 172.21.15.12:0/2099544100 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2ff7e00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fdc003a50 0x7f2ff80ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2ffca00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ff80c9360 0x7f2ff80c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2ff7400700 1 -- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ff80c9360 msgr2=0x7f2ff80c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2ff7400700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ff80c9360 0x7f2ff80c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2ff7400700 1 -- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fdc003a50 msgr2=0x7f2ff80ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2ff7400700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fdc003a50 0x7f2ff80ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2ff7400700 1 -- 172.21.15.12:0/2099544100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff815ffb0 con 0x7f2ff80cbf60 2024-03-29T10:10:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2ff7400700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff80cbf60 0x7f2ff80cc3e0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4011aa0 tx=0x7f2fe4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2fefe00700 1 -- 172.21.15.12:0/2099544100 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe400f9e0 con 0x7f2ff80cbf60 2024-03-29T10:10:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2fefe00700 1 -- 172.21.15.12:0/2099544100 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe400fb40 con 0x7f2ff80cbf60 2024-03-29T10:10:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2ffea00700 1 -- 172.21.15.12:0/2099544100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff8160290 con 0x7f2ff80cbf60 2024-03-29T10:10:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.856+0000 7f2fefe00700 1 -- 172.21.15.12:0/2099544100 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4014440 con 0x7f2ff80cbf60 2024-03-29T10:10:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.857+0000 7f2ffea00700 1 -- 172.21.15.12:0/2099544100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff80cc920 con 0x7f2ff80cbf60 2024-03-29T10:10:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.858+0000 7f2ffea00700 1 -- 172.21.15.12:0/2099544100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff8067720 con 0x7f2ff80cbf60 2024-03-29T10:10:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.858+0000 7f2fefe00700 1 -- 172.21.15.12:0/2099544100 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 45) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f2fe40145a0 con 0x7f2ff80cbf60 2024-03-29T10:10:47.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.859+0000 7f2fefe00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fd806ad20 0x7f2fd806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:47.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.859+0000 7f2fefe00700 1 -- 172.21.15.12:0/2099544100 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5959+0+0 (secure 0 0 0) 0x7f2fe4089bb0 con 0x7f2ff80cbf60 2024-03-29T10:10:47.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.860+0000 7f2ff7e00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fd806ad20 0x7f2fd806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:47.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.861+0000 7f2ff7e00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fd806ad20 0x7f2fd806d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8000c00 tx=0x7f2fe8007000 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:47.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:47.864+0000 7f2fefe00700 1 -- 172.21.15.12:0/2099544100 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2fe4054c30 con 0x7f2ff80cbf60 2024-03-29T10:10:48.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:48.070+0000 7f2fefe00700 1 -- 172.21.15.12:0/2099544100 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2fe4055000 con 0x7f2ff80cbf60 2024-03-29T10:10:48.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:48.081+0000 7f2ffea00700 1 -- 172.21.15.12:0/2099544100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2ff80c7430 con 0x7f2ff80cbf60 2024-03-29T10:10:49.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.071+0000 7f2fefe00700 1 -- 172.21.15.12:0/2099544100 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2fe4086160 con 0x7f2ff80cbf60 2024-03-29T10:10:49.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.076+0000 7f2ffea00700 1 -- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fd806ad20 msgr2=0x7f2fd806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.076+0000 7f2ffea00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fd806ad20 0x7f2fd806d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8000c00 tx=0x7f2fe8007000 comp rx=0 tx=0).stop 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.076+0000 7f2ffea00700 1 -- 172.21.15.12:0/2099544100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff80cbf60 msgr2=0x7f2ff80cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.076+0000 7f2ffea00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff80cbf60 0x7f2ff80cc3e0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4011aa0 tx=0x7f2fe4011e60 comp rx=0 tx=0).stop 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.077+0000 7f2ffea00700 1 -- 172.21.15.12:0/2099544100 shutdown_connections 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.077+0000 7f2ffea00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fd806ad20 0x7f2fd806d1e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.077+0000 7f2ffea00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ff80c9360 0x7f2ff80c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.077+0000 7f2ffea00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fdc003a50 0x7f2ff80ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.077+0000 7f2ffea00700 1 --2- 172.21.15.12:0/2099544100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff80cbf60 0x7f2ff80cc3e0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.077+0000 7f2ffea00700 1 -- 172.21.15.12:0/2099544100 >> 172.21.15.12:0/2099544100 conn(0x7f2ff808aff0 msgr2=0x7f2ff80d2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.077+0000 7f2ffea00700 1 -- 172.21.15.12:0/2099544100 shutdown_connections 2024-03-29T10:10:49.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.077+0000 7f2ffea00700 1 -- 172.21.15.12:0/2099544100 wait complete. 2024-03-29T10:10:49.078 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T10:10:49.097 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T10:10:49.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.302+0000 7f89be000700 1 Processor -- start 2024-03-29T10:10:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.302+0000 7f89be000700 1 -- start start 2024-03-29T10:10:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.303+0000 7f89be000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f89b815d9d0 0x7f89b80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.303+0000 7f89be000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b80d2540 0x7f89b80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.303+0000 7f89be000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8161420 con 0x7f89b80d16a0 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.303+0000 7f89be000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b815edc0 con 0x7f89b815d9d0 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.303+0000 7f89be000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b8161700 con 0x7f89b80d2540 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.303+0000 7f89b7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f89b815d9d0 0x7f89b80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.303+0000 7f89b6a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f89b80d16a0 0x7f89b80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60592/0 (socket says 172.21.15.12:60592) 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.303+0000 7f89b6a00700 1 -- 172.21.15.12:0/3132311025 learned_addr learned my addr 172.21.15.12:0/3132311025 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.304+0000 7f89b7e00700 1 --2- 172.21.15.12:0/3132311025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b80d2540 0x7f89b80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.304+0000 7f89b6000700 1 -- 172.21.15.12:0/3132311025 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113239908 0 0) 0x7f89b8161420 con 0x7f89b80d16a0 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.304+0000 7f89b7400700 1 -- 172.21.15.12:0/3132311025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b80d2540 msgr2=0x7f89b80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.304+0000 7f89b7400700 1 --2- 172.21.15.12:0/3132311025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b80d2540 0x7f89b80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.304+0000 7f89b7400700 1 -- 172.21.15.12:0/3132311025 >> v1:172.21.15.12:6789/0 conn(0x7f89b80d16a0 legacy=0x7f89b80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.304+0000 7f89b7400700 1 -- 172.21.15.12:0/3132311025 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b81619e0 con 0x7f89b815d9d0 2024-03-29T10:10:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.304+0000 7f89b7400700 1 --2- 172.21.15.12:0/3132311025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b815d9d0 0x7f89b80d3c30 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009840 tx=0x7f89a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb9830999ae6116 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.304+0000 7f89b6000700 1 -- 172.21.15.12:0/3132311025 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8011830 con 0x7f89b815d9d0 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.304+0000 7f89b6000700 1 -- 172.21.15.12:0/3132311025 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a8011990 con 0x7f89b815d9d0 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.304+0000 7f89b6000700 1 -- 172.21.15.12:0/3132311025 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8011d20 con 0x7f89b815d9d0 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.305+0000 7f89be000700 1 -- 172.21.15.12:0/3132311025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b815d9d0 msgr2=0x7f89b80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.305+0000 7f89be000700 1 --2- 172.21.15.12:0/3132311025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b815d9d0 0x7f89b80d3c30 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009840 tx=0x7f89a8009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.305+0000 7f89be000700 1 -- 172.21.15.12:0/3132311025 shutdown_connections 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.305+0000 7f89be000700 1 --2- 172.21.15.12:0/3132311025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b80d2540 0x7f89b80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.305+0000 7f89be000700 1 --2- 172.21.15.12:0/3132311025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b815d9d0 0x7f89b80d3c30 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.305+0000 7f89be000700 1 -- 172.21.15.12:0/3132311025 >> 172.21.15.12:0/3132311025 conn(0x7f89b808aff0 msgr2=0x7f89b80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.305+0000 7f89be000700 1 -- 172.21.15.12:0/3132311025 shutdown_connections 2024-03-29T10:10:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.305+0000 7f89be000700 1 -- 172.21.15.12:0/3132311025 wait complete. 2024-03-29T10:10:49.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.307+0000 7f89be000700 1 Processor -- start 2024-03-29T10:10:49.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.307+0000 7f89be000700 1 -- start start 2024-03-29T10:10:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89be000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b80d16a0 0x7f89b81601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89be000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b80d2540 0x7f89b81e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89be000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b815d9d0 0x7f89b81e8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89be000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b81e7bf0 con 0x7f89b80d2540 2024-03-29T10:10:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89be000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b81e7ed0 con 0x7f89b80d16a0 2024-03-29T10:10:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89be000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b81e81b0 con 0x7f89b815d9d0 2024-03-29T10:10:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b80d2540 0x7f89b81e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b80d2540 0x7f89b81e7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36792/0 (socket says 172.21.15.12:36792) 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89b6a00700 1 -- 172.21.15.12:0/4136268142 learned_addr learned my addr 172.21.15.12:0/4136268142 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89b7400700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b80d16a0 0x7f89b81601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.308+0000 7f89b7e00700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b815d9d0 0x7f89b81e8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89b6a00700 1 -- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b815d9d0 msgr2=0x7f89b81e8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89b6a00700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b815d9d0 0x7f89b81e8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89b6a00700 1 -- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b80d16a0 msgr2=0x7f89b81601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89b6a00700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b80d16a0 0x7f89b81601d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89b6a00700 1 -- 172.21.15.12:0/4136268142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a80094a0 con 0x7f89b80d2540 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89b6a00700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b80d2540 0x7f89b81e7280 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f89a400c6b0 tx=0x7f89a400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89af400700 1 -- 172.21.15.12:0/4136268142 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a40074e0 con 0x7f89b80d2540 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89af400700 1 -- 172.21.15.12:0/4136268142 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a4007640 con 0x7f89b80d2540 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89be000700 1 -- 172.21.15.12:0/4136268142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b81e8490 con 0x7f89b80d2540 2024-03-29T10:10:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89af400700 1 -- 172.21.15.12:0/4136268142 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4005a30 con 0x7f89b80d2540 2024-03-29T10:10:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.309+0000 7f89be000700 1 -- 172.21.15.12:0/4136268142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b8160710 con 0x7f89b80d2540 2024-03-29T10:10:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.311+0000 7f89be000700 1 -- 172.21.15.12:0/4136268142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b80d1ca0 con 0x7f89b80d2540 2024-03-29T10:10:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.311+0000 7f89af400700 1 -- 172.21.15.12:0/4136268142 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f89a4005b90 con 0x7f89b80d2540 2024-03-29T10:10:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.312+0000 7f89af400700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f899806adb0 0x7f899806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.312+0000 7f89af400700 1 -- 172.21.15.12:0/4136268142 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f89a4088600 con 0x7f89b80d2540 2024-03-29T10:10:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.313+0000 7f89b7400700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f899806adb0 0x7f899806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.314+0000 7f89b7400700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f899806adb0 0x7f899806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f89a8000c00 tx=0x7f89a801c040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.317+0000 7f89af400700 1 -- 172.21.15.12:0/4136268142 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f89a40538b0 con 0x7f89b80d2540 2024-03-29T10:10:49.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.529+0000 7f89be000700 1 -- 172.21.15.12:0/4136268142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f89b8067720 con 0x7f89b80d2540 2024-03-29T10:10:49.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.535+0000 7f89af400700 1 -- 172.21.15.12:0/4136268142 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v31) v1 ==== 115+0+0 (secure 0 0 0) 0x7f89a40170b0 con 0x7f89b80d2540 2024-03-29T10:10:49.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.538+0000 7f89be000700 1 -- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f899806adb0 msgr2=0x7f899806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.538+0000 7f89be000700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f899806adb0 0x7f899806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f89a8000c00 tx=0x7f89a801c040 comp rx=0 tx=0).stop 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.538+0000 7f89be000700 1 -- 172.21.15.12:0/4136268142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b80d2540 msgr2=0x7f89b81e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.538+0000 7f89be000700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b80d2540 0x7f89b81e7280 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f89a400c6b0 tx=0x7f89a400c9c0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.539+0000 7f89be000700 1 -- 172.21.15.12:0/4136268142 shutdown_connections 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.539+0000 7f89be000700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f899806adb0 0x7f899806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.539+0000 7f89be000700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b815d9d0 0x7f89b81e8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.539+0000 7f89be000700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b80d16a0 0x7f89b81601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.539+0000 7f89be000700 1 --2- 172.21.15.12:0/4136268142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b80d2540 0x7f89b81e7280 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.539+0000 7f89be000700 1 -- 172.21.15.12:0/4136268142 >> 172.21.15.12:0/4136268142 conn(0x7f89b808aff0 msgr2=0x7f89b80b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.539+0000 7f89be000700 1 -- 172.21.15.12:0/4136268142 shutdown_connections 2024-03-29T10:10:49.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.539+0000 7f89be000700 1 -- 172.21.15.12:0/4136268142 wait complete. 2024-03-29T10:10:49.556 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T10:10:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.759+0000 7ffad5200700 1 Processor -- start 2024-03-29T10:10:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.759+0000 7ffad5200700 1 -- start start 2024-03-29T10:10:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.760+0000 7ffad5200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffad00d16a0 0x7ffad00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.760+0000 7ffad5200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad0161420 con 0x7ffad00d2370 2024-03-29T10:10:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.760+0000 7ffad5200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad0161700 con 0x7ffad015d9d0 2024-03-29T10:10:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.760+0000 7ffad5200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00d4340 con 0x7ffad00d16a0 2024-03-29T10:10:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.760+0000 7ffacd400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffad00d16a0 0x7ffad00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.760+0000 7fface800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ffad00d2370 0x7ffad00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60602/0 (socket says 172.21.15.12:60602) 2024-03-29T10:10:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.760+0000 7fface800700 1 -- 172.21.15.12:0/716697098 learned_addr learned my addr 172.21.15.12:0/716697098 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.761+0000 7ffacd400700 1 -- 172.21.15.12:0/716697098 >> v1:172.21.15.159:6789/0 conn(0x7ffad015d9d0 legacy=0x7ffad00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.761+0000 7ffacd400700 1 -- 172.21.15.12:0/716697098 >> v1:172.21.15.12:6789/0 conn(0x7ffad00d2370 legacy=0x7ffad00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.761+0000 7ffacd400700 1 -- 172.21.15.12:0/716697098 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad01619e0 con 0x7ffad00d16a0 2024-03-29T10:10:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.761+0000 7ffacca00700 1 -- 172.21.15.12:0/716697098 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861496964 0 0) 0x7ffad0161420 con 0x7ffad00d2370 2024-03-29T10:10:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.761+0000 7ffacd400700 1 --2- 172.21.15.12:0/716697098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 0x7ffad00d1e30 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009820 tx=0x7ffabc009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fe353284b69f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.761+0000 7ffacca00700 1 -- 172.21.15.12:0/716697098 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc005e30 con 0x7ffad00d16a0 2024-03-29T10:10:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.761+0000 7ffacca00700 1 -- 172.21.15.12:0/716697098 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffabc006030 con 0x7ffad00d16a0 2024-03-29T10:10:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.761+0000 7ffacca00700 1 -- 172.21.15.12:0/716697098 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc0063e0 con 0x7ffad00d16a0 2024-03-29T10:10:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.762+0000 7ffad5200700 1 -- 172.21.15.12:0/716697098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 msgr2=0x7ffad00d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.762+0000 7ffad5200700 1 --2- 172.21.15.12:0/716697098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 0x7ffad00d1e30 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009820 tx=0x7ffabc009b30 comp rx=0 tx=0).stop 2024-03-29T10:10:49.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.762+0000 7ffad5200700 1 -- 172.21.15.12:0/716697098 shutdown_connections 2024-03-29T10:10:49.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.762+0000 7ffad5200700 1 --2- 172.21.15.12:0/716697098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 0x7ffad00d1e30 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.762+0000 7ffad5200700 1 -- 172.21.15.12:0/716697098 >> 172.21.15.12:0/716697098 conn(0x7ffad008aff0 msgr2=0x7ffad00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:49.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.762+0000 7ffad5200700 1 -- 172.21.15.12:0/716697098 shutdown_connections 2024-03-29T10:10:49.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.762+0000 7ffad5200700 1 -- 172.21.15.12:0/716697098 wait complete. 2024-03-29T10:10:49.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.764+0000 7ffad5200700 1 Processor -- start 2024-03-29T10:10:49.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.764+0000 7ffad5200700 1 -- start start 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffad5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 0x7ffad014e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffad5200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffad00d2370 0x7ffad00cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffad5200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffad015d9d0 0x7ffad01606d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffad5200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad014ef20 con 0x7ffad00d16a0 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffad5200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00cc2f0 con 0x7ffad00d2370 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffad5200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00cba70 con 0x7ffad015d9d0 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffacde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 0x7ffad014e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffacde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 0x7ffad014e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36802/0 (socket says 172.21.15.12:36802) 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffacde00700 1 -- 172.21.15.12:0/202898736 learned_addr learned my addr 172.21.15.12:0/202898736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffacde00700 1 -- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffad015d9d0 msgr2=0x7ffad01606d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffacde00700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffad015d9d0 0x7ffad01606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffacde00700 1 -- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffad00d2370 msgr2=0x7ffad00cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.765+0000 7ffacde00700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffad00d2370 0x7ffad00cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.766+0000 7ffacde00700 1 -- 172.21.15.12:0/202898736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc009430 con 0x7ffad00d16a0 2024-03-29T10:10:49.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.766+0000 7ffacde00700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 0x7ffad014e9e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ffac000c450 tx=0x7ffac000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:49.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.766+0000 7ffac6000700 1 -- 172.21.15.12:0/202898736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac00052d0 con 0x7ffad00d16a0 2024-03-29T10:10:49.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.766+0000 7ffac6000700 1 -- 172.21.15.12:0/202898736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac0005430 con 0x7ffad00d16a0 2024-03-29T10:10:49.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.766+0000 7ffac6000700 1 -- 172.21.15.12:0/202898736 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac00038c0 con 0x7ffad00d16a0 2024-03-29T10:10:49.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.766+0000 7ffad5200700 1 -- 172.21.15.12:0/202898736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad01f39d0 con 0x7ffad00d16a0 2024-03-29T10:10:49.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.766+0000 7ffad5200700 1 -- 172.21.15.12:0/202898736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad01f3f90 con 0x7ffad00d16a0 2024-03-29T10:10:49.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.767+0000 7ffad5200700 1 -- 172.21.15.12:0/202898736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad00d1ca0 con 0x7ffad00d16a0 2024-03-29T10:10:49.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.768+0000 7ffac6000700 1 -- 172.21.15.12:0/202898736 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ffac0003a20 con 0x7ffad00d16a0 2024-03-29T10:10:49.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.769+0000 7ffac6000700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffab006adb0 0x7ffab006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:49.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.769+0000 7ffac6000700 1 -- 172.21.15.12:0/202898736 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5545+0+0 (secure 0 0 0) 0x7ffac0088840 con 0x7ffad00d16a0 2024-03-29T10:10:49.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.769+0000 7ffacd400700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffab006adb0 0x7ffab006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:49.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.770+0000 7ffacd400700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffab006adb0 0x7ffab006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffad00d1ae0 tx=0x7ffabc010000 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:49.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.773+0000 7ffac6000700 1 -- 172.21.15.12:0/202898736 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffac0053910 con 0x7ffad00d16a0 2024-03-29T10:10:49.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.991+0000 7ffad5200700 1 -- 172.21.15.12:0/202898736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ffad0067720 con 0x7ffad00d16a0 2024-03-29T10:10:49.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.996+0000 7ffac6000700 1 -- 172.21.15.12:0/202898736 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffac0056f30 con 0x7ffad00d16a0 2024-03-29T10:10:49.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.998+0000 7ffad5200700 1 -- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffab006adb0 msgr2=0x7ffab006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.998+0000 7ffad5200700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffab006adb0 0x7ffab006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffad00d1ae0 tx=0x7ffabc010000 comp rx=0 tx=0).stop 2024-03-29T10:10:49.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.998+0000 7ffad5200700 1 -- 172.21.15.12:0/202898736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 msgr2=0x7ffad014e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:49.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.998+0000 7ffad5200700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 0x7ffad014e9e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ffac000c450 tx=0x7ffac000c810 comp rx=0 tx=0).stop 2024-03-29T10:10:49.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.998+0000 7ffad5200700 1 -- 172.21.15.12:0/202898736 shutdown_connections 2024-03-29T10:10:49.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.998+0000 7ffad5200700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffab006adb0 0x7ffab006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.998+0000 7ffad5200700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffad015d9d0 0x7ffad01606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.998+0000 7ffad5200700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffad00d16a0 0x7ffad014e9e0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.998+0000 7ffad5200700 1 --2- 172.21.15.12:0/202898736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffad00d2370 0x7ffad00cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:49.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.998+0000 7ffad5200700 1 -- 172.21.15.12:0/202898736 >> 172.21.15.12:0/202898736 conn(0x7ffad008aff0 msgr2=0x7ffad00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:49.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.999+0000 7ffad5200700 1 -- 172.21.15.12:0/202898736 shutdown_connections 2024-03-29T10:10:49.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:49.999+0000 7ffad5200700 1 -- 172.21.15.12:0/202898736 wait complete. 2024-03-29T10:10:50.016 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T10:10:50.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.217+0000 7f1ccb800700 1 Processor -- start 2024-03-29T10:10:50.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.218+0000 7f1ccb800700 1 -- start start 2024-03-29T10:10:50.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.219+0000 7f1ccb800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1cc415d9d0 0x7f1cc40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.219+0000 7f1ccb800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 0x7f1cc40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.219+0000 7f1ccb800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc4161420 con 0x7f1cc40d2370 2024-03-29T10:10:50.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.219+0000 7f1ccb800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc40d2820 con 0x7f1cc415d9d0 2024-03-29T10:10:50.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.219+0000 7f1ccb800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc4161700 con 0x7f1cc40d16a0 2024-03-29T10:10:50.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.219+0000 7f1cc8e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1cc415d9d0 0x7f1cc40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.219+0000 7f1cc3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 0x7f1cc40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.219+0000 7f1cc8e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1cc415d9d0 0x7f1cc40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46322/0 (socket says 172.21.15.12:46322) 2024-03-29T10:10:50.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.219+0000 7f1cc8e00700 1 -- 172.21.15.12:0/2478756887 learned_addr learned my addr 172.21.15.12:0/2478756887 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:50.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.220+0000 7f1cc3e00700 1 -- 172.21.15.12:0/2478756887 >> v2:172.21.15.159:3300/0 conn(0x7f1cc415d9d0 msgr2=0x7f1cc40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:50.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.220+0000 7f1cc3e00700 1 --2- 172.21.15.12:0/2478756887 >> v2:172.21.15.159:3300/0 conn(0x7f1cc415d9d0 0x7f1cc40d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.220+0000 7f1cc3e00700 1 -- 172.21.15.12:0/2478756887 >> v1:172.21.15.159:6789/0 conn(0x7f1cc40d2370 legacy=0x7f1cc415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:50.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.220+0000 7f1cc3e00700 1 -- 172.21.15.12:0/2478756887 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc41619e0 con 0x7f1cc40d16a0 2024-03-29T10:10:50.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.220+0000 7f1cc8e00700 1 --2- 172.21.15.12:0/2478756887 >> v2:172.21.15.159:3300/0 conn(0x7f1cc415d9d0 0x7f1cc40d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:50.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.220+0000 7f1cc3e00700 1 --2- 172.21.15.12:0/2478756887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 0x7f1cc40d1e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009820 tx=0x7f1cb0009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd97909b1cd2d33d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:50.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.220+0000 7f1cc3400700 1 -- 172.21.15.12:0/2478756887 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0005e30 con 0x7f1cc40d16a0 2024-03-29T10:10:50.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.221+0000 7f1cc3400700 1 -- 172.21.15.12:0/2478756887 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb0006030 con 0x7f1cc40d16a0 2024-03-29T10:10:50.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.221+0000 7f1cc3400700 1 -- 172.21.15.12:0/2478756887 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb00063c0 con 0x7f1cc40d16a0 2024-03-29T10:10:50.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.221+0000 7f1ccb800700 1 -- 172.21.15.12:0/2478756887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 msgr2=0x7f1cc40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:50.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.221+0000 7f1ccb800700 1 --2- 172.21.15.12:0/2478756887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 0x7f1cc40d1e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009820 tx=0x7f1cb0009b30 comp rx=0 tx=0).stop 2024-03-29T10:10:50.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.221+0000 7f1ccb800700 1 -- 172.21.15.12:0/2478756887 shutdown_connections 2024-03-29T10:10:50.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.221+0000 7f1ccb800700 1 --2- 172.21.15.12:0/2478756887 >> v2:172.21.15.159:3300/0 conn(0x7f1cc415d9d0 0x7f1cc40d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.221+0000 7f1ccb800700 1 --2- 172.21.15.12:0/2478756887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 0x7f1cc40d1e30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.221+0000 7f1ccb800700 1 -- 172.21.15.12:0/2478756887 >> 172.21.15.12:0/2478756887 conn(0x7f1cc408aff0 msgr2=0x7f1cc40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:50.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.221+0000 7f1ccb800700 1 -- 172.21.15.12:0/2478756887 shutdown_connections 2024-03-29T10:10:50.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.221+0000 7f1ccb800700 1 -- 172.21.15.12:0/2478756887 wait complete. 2024-03-29T10:10:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.223+0000 7f1ccb800700 1 Processor -- start 2024-03-29T10:10:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.223+0000 7f1ccb800700 1 -- start start 2024-03-29T10:10:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1ccb800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 0x7f1cc414ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1ccb800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc40d2370 0x7f1cc40ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1ccb800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc415d9d0 0x7f1cc40cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1ccb800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc40b70c0 con 0x7f1cc40d2370 2024-03-29T10:10:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1ccb800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc414b090 con 0x7f1cc415d9d0 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1ccb800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc40b6790 con 0x7f1cc40d16a0 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1cc3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc40d2370 0x7f1cc40ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1cc3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc40d2370 0x7f1cc40ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36812/0 (socket says 172.21.15.12:36812) 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1cc3e00700 1 -- 172.21.15.12:0/3722838837 learned_addr learned my addr 172.21.15.12:0/3722838837 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1cc8e00700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 0x7f1cc414ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.224+0000 7f1cc9800700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc415d9d0 0x7f1cc40cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1cc3e00700 1 -- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 msgr2=0x7f1cc414ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1cc3e00700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 0x7f1cc414ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1cc3e00700 1 -- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc415d9d0 msgr2=0x7f1cc40cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1cc3e00700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc415d9d0 0x7f1cc40cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1cc3e00700 1 -- 172.21.15.12:0/3722838837 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb8009550 con 0x7f1cc40d2370 2024-03-29T10:10:50.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1cc3e00700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc40d2370 0x7f1cc40ca5f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0000c00 tx=0x7f1cb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:50.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1cc0c00700 1 -- 172.21.15.12:0/3722838837 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb00061a0 con 0x7f1cc40d2370 2024-03-29T10:10:50.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1cc0c00700 1 -- 172.21.15.12:0/3722838837 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb0020070 con 0x7f1cc40d2370 2024-03-29T10:10:50.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1cc0c00700 1 -- 172.21.15.12:0/3722838837 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0006540 con 0x7f1cc40d2370 2024-03-29T10:10:50.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1ccb800700 1 -- 172.21.15.12:0/3722838837 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb0009430 con 0x7f1cc40d2370 2024-03-29T10:10:50.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.225+0000 7f1ccb800700 1 -- 172.21.15.12:0/3722838837 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc40b6d50 con 0x7f1cc40d2370 2024-03-29T10:10:50.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.226+0000 7f1ccb800700 1 -- 172.21.15.12:0/3722838837 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc40b6f10 con 0x7f1cc40d2370 2024-03-29T10:10:50.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.227+0000 7f1cc0c00700 1 -- 172.21.15.12:0/3722838837 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1cb0022030 con 0x7f1cc40d2370 2024-03-29T10:10:50.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.228+0000 7f1cc0c00700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca406adb0 0x7f1ca406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.228+0000 7f1cc0c00700 1 -- 172.21.15.12:0/3722838837 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f1cb008a7e0 con 0x7f1cc40d2370 2024-03-29T10:10:50.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.229+0000 7f1cc8e00700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca406adb0 0x7f1ca406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.229+0000 7f1cc8e00700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca406adb0 0x7f1ca406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8006d30 tx=0x7f1cb80092c0 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:50.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.232+0000 7f1cc0c00700 1 -- 172.21.15.12:0/3722838837 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1cb00559e0 con 0x7f1cc40d2370 2024-03-29T10:10:50.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.450+0000 7f1ccb800700 1 -- 172.21.15.12:0/3722838837 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f1cc40d1ca0 con 0x7f1cc40d2370 2024-03-29T10:10:50.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.454+0000 7f1cc0c00700 1 -- 172.21.15.12:0/3722838837 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v33) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1cb0059000 con 0x7f1cc40d2370 2024-03-29T10:10:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 -- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca406adb0 msgr2=0x7f1ca406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca406adb0 0x7f1ca406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8006d30 tx=0x7f1cb80092c0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 -- 172.21.15.12:0/3722838837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc40d2370 msgr2=0x7f1cc40ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc40d2370 0x7f1cc40ca5f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0000c00 tx=0x7f1cb0004750 comp rx=0 tx=0).stop 2024-03-29T10:10:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 -- 172.21.15.12:0/3722838837 shutdown_connections 2024-03-29T10:10:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ca406adb0 0x7f1ca406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1cc415d9d0 0x7f1cc40cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1cc40d16a0 0x7f1cc414ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 --2- 172.21.15.12:0/3722838837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cc40d2370 0x7f1cc40ca5f0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 -- 172.21.15.12:0/3722838837 >> 172.21.15.12:0/3722838837 conn(0x7f1cc408aff0 msgr2=0x7f1cc40b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 -- 172.21.15.12:0/3722838837 shutdown_connections 2024-03-29T10:10:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.457+0000 7f1ccb800700 1 -- 172.21.15.12:0/3722838837 wait complete. 2024-03-29T10:10:50.475 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-03-29T10:10:50.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.679+0000 7f8336400700 1 Processor -- start 2024-03-29T10:10:50.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.679+0000 7f8336400700 1 -- start start 2024-03-29T10:10:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.680+0000 7f8336400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83300d18b0 0x7f83300cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.680+0000 7f8336400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8330161590 0x7f8330161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.680+0000 7f8336400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83300ceda0 con 0x7f8330160f10 2024-03-29T10:10:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.680+0000 7f8336400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83300d1d30 con 0x7f8330161590 2024-03-29T10:10:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.680+0000 7f8336400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8330161f60 con 0x7f83300d18b0 2024-03-29T10:10:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.680+0000 7f832f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8330160f10 0x7f83300d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60606/0 (socket says 172.21.15.12:60606) 2024-03-29T10:10:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.680+0000 7f832f400700 1 -- 172.21.15.12:0/988414663 learned_addr learned my addr 172.21.15.12:0/988414663 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.680+0000 7f832fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8330161590 0x7f8330161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.680+0000 7f832ea00700 1 --2- 172.21.15.12:0/988414663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83300d18b0 0x7f83300cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.681+0000 7f832fe00700 1 -- 172.21.15.12:0/988414663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83300d18b0 msgr2=0x7f83300cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.681+0000 7f832fe00700 1 --2- 172.21.15.12:0/988414663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83300d18b0 0x7f83300cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.681+0000 7f832fe00700 1 -- 172.21.15.12:0/988414663 >> v1:172.21.15.12:6789/0 conn(0x7f8330160f10 legacy=0x7f83300d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.681+0000 7f832fe00700 1 -- 172.21.15.12:0/988414663 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83301612f0 con 0x7f8330161590 2024-03-29T10:10:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.681+0000 7f832fe00700 1 --2- 172.21.15.12:0/988414663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 0x7f8330161a20 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8328009dc0 tx=0x7f832800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689f86235fc2a453 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.681+0000 7f832e000700 1 -- 172.21.15.12:0/988414663 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832800e070 con 0x7f8330161590 2024-03-29T10:10:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.681+0000 7f832e000700 1 -- 172.21.15.12:0/988414663 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832800bb90 con 0x7f8330161590 2024-03-29T10:10:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.681+0000 7f832e000700 1 -- 172.21.15.12:0/988414663 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328012430 con 0x7f8330161590 2024-03-29T10:10:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.682+0000 7f8336400700 1 -- 172.21.15.12:0/988414663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 msgr2=0x7f8330161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.682+0000 7f8336400700 1 --2- 172.21.15.12:0/988414663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 0x7f8330161a20 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8328009dc0 tx=0x7f832800b330 comp rx=0 tx=0).stop 2024-03-29T10:10:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.682+0000 7f8336400700 1 -- 172.21.15.12:0/988414663 shutdown_connections 2024-03-29T10:10:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.682+0000 7f8336400700 1 --2- 172.21.15.12:0/988414663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 0x7f8330161a20 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.682+0000 7f8336400700 1 --2- 172.21.15.12:0/988414663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83300d18b0 0x7f83300cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.682+0000 7f8336400700 1 -- 172.21.15.12:0/988414663 >> 172.21.15.12:0/988414663 conn(0x7f833008aff0 msgr2=0x7f83300d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.682+0000 7f8336400700 1 -- 172.21.15.12:0/988414663 shutdown_connections 2024-03-29T10:10:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.682+0000 7f8336400700 1 -- 172.21.15.12:0/988414663 wait complete. 2024-03-29T10:10:50.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.684+0000 7f8336400700 1 Processor -- start 2024-03-29T10:10:50.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.684+0000 7f8336400700 1 -- start start 2024-03-29T10:10:50.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f8336400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83300d18b0 0x7f833014c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f8336400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8330160f10 0x7f833014f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f8336400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 0x7f833014ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f8336400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833009d540 con 0x7f8330161590 2024-03-29T10:10:50.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f8336400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833009d820 con 0x7f8330160f10 2024-03-29T10:10:50.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f8336400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8330149700 con 0x7f83300d18b0 2024-03-29T10:10:50.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f832fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 0x7f833014ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f832fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 0x7f833014ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36824/0 (socket says 172.21.15.12:36824) 2024-03-29T10:10:50.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f832fe00700 1 -- 172.21.15.12:0/3455672563 learned_addr learned my addr 172.21.15.12:0/3455672563 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f832f400700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83300d18b0 0x7f833014c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f832ea00700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8330160f10 0x7f833014f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f832fe00700 1 -- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83300d18b0 msgr2=0x7f833014c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f832fe00700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83300d18b0 0x7f833014c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f832fe00700 1 -- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8330160f10 msgr2=0x7f833014f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f832fe00700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8330160f10 0x7f833014f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.685+0000 7f832fe00700 1 -- 172.21.15.12:0/3455672563 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328009a70 con 0x7f8330161590 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.686+0000 7f832fe00700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 0x7f833014ba00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f832800b370 tx=0x7f8328004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.686+0000 7f8327400700 1 -- 172.21.15.12:0/3455672563 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328012430 con 0x7f8330161590 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.686+0000 7f8327400700 1 -- 172.21.15.12:0/3455672563 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8328020070 con 0x7f8330161590 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.686+0000 7f8327400700 1 -- 172.21.15.12:0/3455672563 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328004280 con 0x7f8330161590 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.686+0000 7f8336400700 1 -- 172.21.15.12:0/3455672563 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83301499e0 con 0x7f8330161590 2024-03-29T10:10:50.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.686+0000 7f8336400700 1 -- 172.21.15.12:0/3455672563 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833014bf40 con 0x7f8330161590 2024-03-29T10:10:50.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.687+0000 7f8336400700 1 -- 172.21.15.12:0/3455672563 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8330149f00 con 0x7f8330161590 2024-03-29T10:10:50.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.688+0000 7f8327400700 1 -- 172.21.15.12:0/3455672563 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f83280043e0 con 0x7f8330161590 2024-03-29T10:10:50.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.689+0000 7f8327400700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f831006adb0 0x7f831006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:50.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.689+0000 7f8327400700 1 -- 172.21.15.12:0/3455672563 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f8328022070 con 0x7f8330161590 2024-03-29T10:10:50.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.689+0000 7f832f400700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f831006adb0 0x7f831006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:50.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.690+0000 7f832f400700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f831006adb0 0x7f831006d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8320005910 tx=0x7f8320009040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:50.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.693+0000 7f8327400700 1 -- 172.21.15.12:0/3455672563 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8330149f00 con 0x7f8330161590 2024-03-29T10:10:50.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:50.992+0000 7f8336400700 1 -- 172.21.15.12:0/3455672563 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f8330149f00 con 0x7f8330161590 2024-03-29T10:10:51.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.073+0000 7f8327400700 1 -- 172.21.15.12:0/3455672563 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v608) v1 ==== 166+0+0 (secure 0 0 0) 0x7f8330149f00 con 0x7f8330161590 2024-03-29T10:10:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 -- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f831006adb0 msgr2=0x7f831006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f831006adb0 0x7f831006d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8320005910 tx=0x7f8320009040 comp rx=0 tx=0).stop 2024-03-29T10:10:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 -- 172.21.15.12:0/3455672563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 msgr2=0x7f833014ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 0x7f833014ba00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f832800b370 tx=0x7f8328004ad0 comp rx=0 tx=0).stop 2024-03-29T10:10:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 -- 172.21.15.12:0/3455672563 shutdown_connections 2024-03-29T10:10:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f831006adb0 0x7f831006d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8330161590 0x7f833014ba00 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83300d18b0 0x7f833014c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 --2- 172.21.15.12:0/3455672563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8330160f10 0x7f833014f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 -- 172.21.15.12:0/3455672563 >> 172.21.15.12:0/3455672563 conn(0x7f833008aff0 msgr2=0x7f83300b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 -- 172.21.15.12:0/3455672563 shutdown_connections 2024-03-29T10:10:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.076+0000 7f8336400700 1 -- 172.21.15.12:0/3455672563 wait complete. 2024-03-29T10:10:51.094 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2024-03-29T10:10:51.094 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2024-03-29T10:10:51.094 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T10:10:51.094 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-03-29T10:10:51.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.292+0000 7f4a3fe00700 1 Processor -- start 2024-03-29T10:10:51.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.292+0000 7f4a3fe00700 1 -- start start 2024-03-29T10:10:51.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.293+0000 7f4a3fe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a38161ca0 0x7f4a380cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:51.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.293+0000 7f4a3fe00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a380ceda0 con 0x7f4a38160f10 2024-03-29T10:10:51.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.293+0000 7f4a3fe00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a381612f0 con 0x7f4a380d18b0 2024-03-29T10:10:51.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.293+0000 7f4a3fe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380d1d30 con 0x7f4a38161ca0 2024-03-29T10:10:51.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.293+0000 7f4a3d400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4a38160f10 0x7f4a380d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60610/0 (socket says 172.21.15.12:60610) 2024-03-29T10:10:51.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.293+0000 7f4a3d400700 1 -- 172.21.15.12:0/1493849055 learned_addr learned my addr 172.21.15.12:0/1493849055 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a3de00700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a38161ca0 0x7f4a380cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200047706 0 0) 0x7f4a380ceda0 con 0x7f4a38160f10 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a200035b0 con 0x7f4a38160f10 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658773605 0 0) 0x7f4a381612f0 con 0x7f4a380d18b0 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a380ceda0 con 0x7f4a380d18b0 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1318233549 0 0) 0x7f4a200035b0 con 0x7f4a38160f10 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a38161ca0 msgr2=0x7f4a380cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a38161ca0 0x7f4a380cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 >> v1:172.21.15.159:6789/0 conn(0x7f4a380d18b0 legacy=0x7f4a38161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a380ce470 con 0x7f4a38160f10 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f4a2c001670 con 0x7f4a38160f10 2024-03-29T10:10:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 >> v1:172.21.15.12:6789/0 conn(0x7f4a38160f10 legacy=0x7f4a380d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a20003bc0 0x7f4a20003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a20004850 0x7f4a20004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a38161ca0 0x7f4a20005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380d1d30 con 0x7f4a20003bc0 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2000bba0 con 0x7f4a20004850 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.294+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2000bd60 con 0x7f4a38161ca0 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a3d400700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a20003bc0 0x7f4a20003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a3ca00700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a20004850 0x7f4a20004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a3de00700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a38161ca0 0x7f4a20005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a3d400700 1 -- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a38161ca0 msgr2=0x7f4a20005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a3d400700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a38161ca0 0x7f4a20005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a3d400700 1 -- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a20004850 msgr2=0x7f4a20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a3d400700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a20004850 0x7f4a20004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a3d400700 1 -- 172.21.15.12:0/1493849055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a200035b0 con 0x7f4a20003bc0 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a3de00700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a38161ca0 0x7f4a20005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a3d400700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a20003bc0 0x7f4a20003fc0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c210 tx=0x7f4a20005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0f53e9b45fc08c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00cdb0 con 0x7f4a20003bc0 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c001af0 con 0x7f4a20003bc0 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.295+0000 7f4a37e00700 1 -- 172.21.15.12:0/1493849055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c005ba0 con 0x7f4a20003bc0 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.296+0000 7f4a3fe00700 1 -- 172.21.15.12:0/1493849055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a20003bc0 msgr2=0x7f4a20003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.296+0000 7f4a3fe00700 1 --2- 172.21.15.12:0/1493849055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a20003bc0 0x7f4a20003fc0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c210 tx=0x7f4a20005880 comp rx=0 tx=0).stop 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.296+0000 7f4a3de00700 1 -- 172.21.15.12:0/1493849055 reap_dead start 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.296+0000 7f4a3fe00700 1 -- 172.21.15.12:0/1493849055 shutdown_connections 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.296+0000 7f4a3fe00700 1 -- 172.21.15.12:0/1493849055 >> 172.21.15.12:0/1493849055 conn(0x7f4a3808aff0 msgr2=0x7f4a380d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.296+0000 7f4a3fe00700 1 -- 172.21.15.12:0/1493849055 shutdown_connections 2024-03-29T10:10:51.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.296+0000 7f4a3fe00700 1 -- 172.21.15.12:0/1493849055 wait complete. 2024-03-29T10:10:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.298+0000 7f4a3fe00700 1 Processor -- start 2024-03-29T10:10:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.298+0000 7f4a3fe00700 1 -- start start 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3fe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a20003bc0 0x7f4a380a0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3814e830 0x7f4a3809ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3fe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a3814ecb0 0x7f4a3815d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3fe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3809f8f0 con 0x7f4a3814e830 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3fe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3809fbd0 con 0x7f4a20003bc0 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3fe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3809feb0 con 0x7f4a3814ecb0 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3814e830 0x7f4a3809ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3814e830 0x7f4a3809ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36854/0 (socket says 172.21.15.12:36854) 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3ca00700 1 -- 172.21.15.12:0/2678352227 learned_addr learned my addr 172.21.15.12:0/2678352227 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3de00700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a3814ecb0 0x7f4a3815d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.299+0000 7f4a3d400700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a20003bc0 0x7f4a380a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a3ca00700 1 -- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a3814ecb0 msgr2=0x7f4a3815d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a3ca00700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a3814ecb0 0x7f4a3815d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a3ca00700 1 -- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a20003bc0 msgr2=0x7f4a380a0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a3ca00700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a20003bc0 0x7f4a380a0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a3ca00700 1 -- 172.21.15.12:0/2678352227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a380a0190 con 0x7f4a3814e830 2024-03-29T10:10:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a3ca00700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3814e830 0x7f4a3809ee50 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400e7b0 tx=0x7f4a2400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a35600700 1 -- 172.21.15.12:0/2678352227 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a240052e0 con 0x7f4a3814e830 2024-03-29T10:10:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a35600700 1 -- 172.21.15.12:0/2678352227 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a24005440 con 0x7f4a3814e830 2024-03-29T10:10:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a35600700 1 -- 172.21.15.12:0/2678352227 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a240038c0 con 0x7f4a3814e830 2024-03-29T10:10:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a3fe00700 1 -- 172.21.15.12:0/2678352227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a380a0470 con 0x7f4a3814e830 2024-03-29T10:10:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.300+0000 7f4a3fe00700 1 -- 172.21.15.12:0/2678352227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3809f390 con 0x7f4a3814e830 2024-03-29T10:10:51.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.301+0000 7f4a3fe00700 1 -- 172.21.15.12:0/2678352227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a38067720 con 0x7f4a3814e830 2024-03-29T10:10:51.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.302+0000 7f4a35600700 1 -- 172.21.15.12:0/2678352227 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4a24003a20 con 0x7f4a3814e830 2024-03-29T10:10:51.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.303+0000 7f4a35600700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a1806adb0 0x7f4a1806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:51.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.303+0000 7f4a35600700 1 -- 172.21.15.12:0/2678352227 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f4a24088bc0 con 0x7f4a3814e830 2024-03-29T10:10:51.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.304+0000 7f4a3d400700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a1806adb0 0x7f4a1806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:51.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.304+0000 7f4a3d400700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a1806adb0 0x7f4a1806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c015040 tx=0x7f4a2c00a000 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:51.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.308+0000 7f4a35600700 1 -- 172.21.15.12:0/2678352227 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4a24053d20 con 0x7f4a3814e830 2024-03-29T10:10:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:51.606+0000 7f4a3fe00700 1 -- 172.21.15.12:0/2678352227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f4a3808b760 con 0x7f4a3814e830 2024-03-29T10:10:52.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.079+0000 7f4a35600700 1 -- 172.21.15.12:0/2678352227 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v609) v1 ==== 186+0+0 (secure 0 0 0) 0x7f4a24019070 con 0x7f4a3814e830 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 -- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a1806adb0 msgr2=0x7f4a1806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a1806adb0 0x7f4a1806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c015040 tx=0x7f4a2c00a000 comp rx=0 tx=0).stop 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 -- 172.21.15.12:0/2678352227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3814e830 msgr2=0x7f4a3809ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3814e830 0x7f4a3809ee50 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400e7b0 tx=0x7f4a2400eac0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 -- 172.21.15.12:0/2678352227 shutdown_connections 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4a1806adb0 0x7f4a1806d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a3814ecb0 0x7f4a3815d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a20003bc0 0x7f4a380a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 --2- 172.21.15.12:0/2678352227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3814e830 0x7f4a3809ee50 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 -- 172.21.15.12:0/2678352227 >> 172.21.15.12:0/2678352227 conn(0x7f4a3808aff0 msgr2=0x7f4a380d03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:52.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 -- 172.21.15.12:0/2678352227 shutdown_connections 2024-03-29T10:10:52.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.082+0000 7f4a3fe00700 1 -- 172.21.15.12:0/2678352227 wait complete. 2024-03-29T10:10:52.101 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T10:10:52.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.305+0000 7f7587000700 1 Processor -- start 2024-03-29T10:10:52.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.305+0000 7f7587000700 1 -- start start 2024-03-29T10:10:52.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.307+0000 7f7587000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f758015d9d0 0x7f75800d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:52.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.307+0000 7f7587000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f75800d2540 0x7f75800ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:52.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.307+0000 7f7587000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7580161420 con 0x7f75800d16a0 2024-03-29T10:10:52.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.307+0000 7f7587000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f758015edc0 con 0x7f75800d2540 2024-03-29T10:10:52.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.307+0000 7f7587000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7580161700 con 0x7f758015d9d0 2024-03-29T10:10:52.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.307+0000 7f7585000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f75800d2540 0x7f75800ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:52.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.307+0000 7f7585000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f75800d2540 0x7f75800ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36868/0 (socket says 172.21.15.12:36868) 2024-03-29T10:10:52.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.307+0000 7f7585000700 1 -- 172.21.15.12:0/4282087151 learned_addr learned my addr 172.21.15.12:0/4282087151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:52.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.307+0000 7f757fe00700 1 --2- 172.21.15.12:0/4282087151 >> v2:172.21.15.159:3300/0 conn(0x7f758015d9d0 0x7f75800d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:52.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.308+0000 7f7585000700 1 -- 172.21.15.12:0/4282087151 >> v2:172.21.15.159:3300/0 conn(0x7f758015d9d0 msgr2=0x7f75800d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:52.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.308+0000 7f7585000700 1 --2- 172.21.15.12:0/4282087151 >> v2:172.21.15.159:3300/0 conn(0x7f758015d9d0 0x7f75800d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.308+0000 7f7585000700 1 -- 172.21.15.12:0/4282087151 >> v1:172.21.15.159:6789/0 conn(0x7f75800d16a0 legacy=0x7f75800d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:52.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.308+0000 7f7585000700 1 -- 172.21.15.12:0/4282087151 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75801619e0 con 0x7f75800d2540 2024-03-29T10:10:52.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.308+0000 7f7585000700 1 --2- 172.21.15.12:0/4282087151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 0x7f75800ce360 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f75800d0ff0 tx=0x7f757800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402f07ba1e8db233 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:52.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.308+0000 7f757ea00700 1 -- 172.21.15.12:0/4282087151 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757800e070 con 0x7f75800d2540 2024-03-29T10:10:52.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.308+0000 7f757ea00700 1 -- 172.21.15.12:0/4282087151 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757800b9c0 con 0x7f75800d2540 2024-03-29T10:10:52.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.308+0000 7f757ea00700 1 -- 172.21.15.12:0/4282087151 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757800bd70 con 0x7f75800d2540 2024-03-29T10:10:52.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.309+0000 7f7587000700 1 -- 172.21.15.12:0/4282087151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 msgr2=0x7f75800ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:52.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.309+0000 7f7587000700 1 --2- 172.21.15.12:0/4282087151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 0x7f75800ce360 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f75800d0ff0 tx=0x7f757800b330 comp rx=0 tx=0).stop 2024-03-29T10:10:52.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.309+0000 7f7587000700 1 -- 172.21.15.12:0/4282087151 shutdown_connections 2024-03-29T10:10:52.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.309+0000 7f7587000700 1 --2- 172.21.15.12:0/4282087151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 0x7f75800ce360 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.309+0000 7f7587000700 1 --2- 172.21.15.12:0/4282087151 >> v2:172.21.15.159:3300/0 conn(0x7f758015d9d0 0x7f75800d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.309+0000 7f7587000700 1 -- 172.21.15.12:0/4282087151 >> 172.21.15.12:0/4282087151 conn(0x7f758008aff0 msgr2=0x7f75800d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:52.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.309+0000 7f7587000700 1 -- 172.21.15.12:0/4282087151 shutdown_connections 2024-03-29T10:10:52.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.309+0000 7f7587000700 1 -- 172.21.15.12:0/4282087151 wait complete. 2024-03-29T10:10:52.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.311+0000 7f7587000700 1 Processor -- start 2024-03-29T10:10:52.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.312+0000 7f7587000700 1 -- start start 2024-03-29T10:10:52.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f7587000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75800d16a0 0x7f758015bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:52.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f7587000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 0x7f75800b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:52.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f7587000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f758015d9d0 0x7f758015ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:52.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f7587000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758015c440 con 0x7f75800d2540 2024-03-29T10:10:52.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f7587000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75800b6990 con 0x7f75800d16a0 2024-03-29T10:10:52.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f7587000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75800b6c70 con 0x7f758015d9d0 2024-03-29T10:10:52.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f757f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 0x7f75800b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:52.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f757f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 0x7f75800b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36878/0 (socket says 172.21.15.12:36878) 2024-03-29T10:10:52.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f757f400700 1 -- 172.21.15.12:0/759617196 learned_addr learned my addr 172.21.15.12:0/759617196 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f757fe00700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75800d16a0 0x7f758015bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f7585000700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f758015d9d0 0x7f758015ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f757f400700 1 -- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f758015d9d0 msgr2=0x7f758015ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f757f400700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f758015d9d0 0x7f758015ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f757f400700 1 -- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75800d16a0 msgr2=0x7f758015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f757f400700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75800d16a0 0x7f758015bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f757f400700 1 -- 172.21.15.12:0/759617196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7578009b30 con 0x7f75800d2540 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f7585000700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f758015d9d0 0x7f758015ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.313+0000 7f757fe00700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75800d16a0 0x7f758015bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.314+0000 7f757f400700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 0x7f75800b6020 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f756c00c500 tx=0x7f756c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.314+0000 7f7577e00700 1 -- 172.21.15.12:0/759617196 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c0052d0 con 0x7f75800d2540 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.314+0000 7f7577e00700 1 -- 172.21.15.12:0/759617196 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756c005450 con 0x7f75800d2540 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.314+0000 7f7587000700 1 -- 172.21.15.12:0/759617196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75800b6f50 con 0x7f75800d2540 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.314+0000 7f7577e00700 1 -- 172.21.15.12:0/759617196 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c003910 con 0x7f75800d2540 2024-03-29T10:10:52.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.314+0000 7f7587000700 1 -- 172.21.15.12:0/759617196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75800cb220 con 0x7f75800d2540 2024-03-29T10:10:52.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.315+0000 7f7587000700 1 -- 172.21.15.12:0/759617196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75800d1ca0 con 0x7f75800d2540 2024-03-29T10:10:52.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.316+0000 7f7577e00700 1 -- 172.21.15.12:0/759617196 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f756c003a70 con 0x7f75800d2540 2024-03-29T10:10:52.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.317+0000 7f7577e00700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f756006adb0 0x7f756006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:52.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.318+0000 7f7577e00700 1 -- 172.21.15.12:0/759617196 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f756c0888a0 con 0x7f75800d2540 2024-03-29T10:10:52.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.318+0000 7f757fe00700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f756006adb0 0x7f756006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:52.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.319+0000 7f757fe00700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f756006adb0 0x7f756006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7570007a50 tx=0x7f7570008040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:52.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.322+0000 7f7577e00700 1 -- 172.21.15.12:0/759617196 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f756c053a10 con 0x7f75800d2540 2024-03-29T10:10:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.538+0000 7f7587000700 1 -- 172.21.15.12:0/759617196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f75800cb840 con 0x7f75800d2540 2024-03-29T10:10:52.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.542+0000 7f7577e00700 1 -- 172.21.15.12:0/759617196 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f75800cb840 con 0x7f75800d2540 2024-03-29T10:10:52.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.544+0000 7f7587000700 1 -- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f756006adb0 msgr2=0x7f756006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:52.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.544+0000 7f7587000700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f756006adb0 0x7f756006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7570007a50 tx=0x7f7570008040 comp rx=0 tx=0).stop 2024-03-29T10:10:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.544+0000 7f7587000700 1 -- 172.21.15.12:0/759617196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 msgr2=0x7f75800b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.544+0000 7f7587000700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 0x7f75800b6020 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f756c00c500 tx=0x7f756c00c8c0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.544+0000 7f7587000700 1 -- 172.21.15.12:0/759617196 shutdown_connections 2024-03-29T10:10:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.544+0000 7f7587000700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f756006adb0 0x7f756006d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.544+0000 7f7587000700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f758015d9d0 0x7f758015ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.544+0000 7f7587000700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75800d16a0 0x7f758015bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.544+0000 7f7587000700 1 --2- 172.21.15.12:0/759617196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75800d2540 0x7f75800b6020 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.544+0000 7f7587000700 1 -- 172.21.15.12:0/759617196 >> 172.21.15.12:0/759617196 conn(0x7f758008aff0 msgr2=0x7f7580160780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.545+0000 7f7587000700 1 -- 172.21.15.12:0/759617196 shutdown_connections 2024-03-29T10:10:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:52.545+0000 7f7587000700 1 -- 172.21.15.12:0/759617196 wait complete. 2024-03-29T10:10:52.562 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:10:52.562 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:10:52.647 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:10:52.742 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:10:52.743 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:10:52.743 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:52.787 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:10:52.787 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:10:52.787 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:52.833 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:10:52.834 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:10:52.834 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:52.877 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:10:52.877 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:10:52.878 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:10:52.923 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:10:52.923 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T10:10:52.923 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:10:53.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.122+0000 7ff72d600700 1 Processor -- start 2024-03-29T10:10:53.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.122+0000 7ff72d600700 1 -- start start 2024-03-29T10:10:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.123+0000 7ff72d600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff72815d640 0x7ff7280d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.123+0000 7ff72d600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7280ceec0 0x7ff7281612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.123+0000 7ff72d600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7280d30d0 con 0x7ff728161800 2024-03-29T10:10:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.123+0000 7ff72d600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff728161c90 con 0x7ff72815d640 2024-03-29T10:10:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.123+0000 7ff72d600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7280b6840 con 0x7ff7280ceec0 2024-03-29T10:10:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.123+0000 7ff726a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff72815d640 0x7ff7280d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.123+0000 7ff726a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff72815d640 0x7ff7280d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36890/0 (socket says 172.21.15.12:36890) 2024-03-29T10:10:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.123+0000 7ff726a00700 1 -- 172.21.15.12:0/4141447527 learned_addr learned my addr 172.21.15.12:0/4141447527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.123+0000 7ff727400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff728161800 0x7ff7280cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60614/0 (socket says 172.21.15.12:60614) 2024-03-29T10:10:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.124+0000 7ff726000700 1 --2- 172.21.15.12:0/4141447527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7280ceec0 0x7ff7281612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.124+0000 7ff726a00700 1 -- 172.21.15.12:0/4141447527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7280ceec0 msgr2=0x7ff7281612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:53.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.124+0000 7ff726a00700 1 --2- 172.21.15.12:0/4141447527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7280ceec0 0x7ff7281612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.124+0000 7ff726a00700 1 -- 172.21.15.12:0/4141447527 >> v1:172.21.15.12:6789/0 conn(0x7ff728161800 legacy=0x7ff7280cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:53.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.124+0000 7ff726a00700 1 -- 172.21.15.12:0/4141447527 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7280d2880 con 0x7ff72815d640 2024-03-29T10:10:53.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.124+0000 7ff726a00700 1 --2- 172.21.15.12:0/4141447527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff72815d640 0x7ff7280d2340 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff7180098f0 tx=0x7ff718009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdb3ddcbe095db4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:53.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.124+0000 7ff725600700 1 -- 172.21.15.12:0/4141447527 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71800b8a0 con 0x7ff72815d640 2024-03-29T10:10:53.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.124+0000 7ff725600700 1 -- 172.21.15.12:0/4141447527 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71800ba00 con 0x7ff72815d640 2024-03-29T10:10:53.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.124+0000 7ff725600700 1 -- 172.21.15.12:0/4141447527 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71800bd90 con 0x7ff72815d640 2024-03-29T10:10:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.125+0000 7ff72d600700 1 -- 172.21.15.12:0/4141447527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff72815d640 msgr2=0x7ff7280d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.125+0000 7ff72d600700 1 --2- 172.21.15.12:0/4141447527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff72815d640 0x7ff7280d2340 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff7180098f0 tx=0x7ff718009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.125+0000 7ff72d600700 1 -- 172.21.15.12:0/4141447527 shutdown_connections 2024-03-29T10:10:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.125+0000 7ff72d600700 1 --2- 172.21.15.12:0/4141447527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff72815d640 0x7ff7280d2340 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.125+0000 7ff72d600700 1 --2- 172.21.15.12:0/4141447527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7280ceec0 0x7ff7281612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.125+0000 7ff72d600700 1 -- 172.21.15.12:0/4141447527 >> 172.21.15.12:0/4141447527 conn(0x7ff72808aff0 msgr2=0x7ff7280d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.125+0000 7ff72d600700 1 -- 172.21.15.12:0/4141447527 shutdown_connections 2024-03-29T10:10:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.125+0000 7ff72d600700 1 -- 172.21.15.12:0/4141447527 wait complete. 2024-03-29T10:10:53.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.127+0000 7ff72d600700 1 Processor -- start 2024-03-29T10:10:53.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.127+0000 7ff72d600700 1 -- start start 2024-03-29T10:10:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff72d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7280ceec0 0x7ff7281410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff72d600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72815d640 0x7ff728156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff72d600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff728161800 0x7ff728146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff72d600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7281566c0 con 0x7ff7280ceec0 2024-03-29T10:10:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff72d600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff728146fd0 con 0x7ff72815d640 2024-03-29T10:10:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff72d600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff728159050 con 0x7ff728161800 2024-03-29T10:10:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff726a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7280ceec0 0x7ff7281410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff726a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7280ceec0 0x7ff7281410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36896/0 (socket says 172.21.15.12:36896) 2024-03-29T10:10:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff726a00700 1 -- 172.21.15.12:0/2841065677 learned_addr learned my addr 172.21.15.12:0/2841065677 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff726000700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72815d640 0x7ff728156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff727400700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff728161800 0x7ff728146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff726a00700 1 -- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff728161800 msgr2=0x7ff728146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff726a00700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff728161800 0x7ff728146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff726a00700 1 -- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72815d640 msgr2=0x7ff728156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff726a00700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72815d640 0x7ff728156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.128+0000 7ff726a00700 1 -- 172.21.15.12:0/2841065677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff718009550 con 0x7ff7280ceec0 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.129+0000 7ff726a00700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7280ceec0 0x7ff7281410e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff72808c490 tx=0x7ff718004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.129+0000 7ff71ea00700 1 -- 172.21.15.12:0/2841065677 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718004480 con 0x7ff7280ceec0 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.129+0000 7ff71ea00700 1 -- 172.21.15.12:0/2841065677 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71801f070 con 0x7ff7280ceec0 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.129+0000 7ff72d600700 1 -- 172.21.15.12:0/2841065677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff728158d70 con 0x7ff7280ceec0 2024-03-29T10:10:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.129+0000 7ff71ea00700 1 -- 172.21.15.12:0/2841065677 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7180038c0 con 0x7ff7280ceec0 2024-03-29T10:10:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.129+0000 7ff72d600700 1 -- 172.21.15.12:0/2841065677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72815f180 con 0x7ff7280ceec0 2024-03-29T10:10:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.130+0000 7ff72d600700 1 -- 172.21.15.12:0/2841065677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff728067720 con 0x7ff7280ceec0 2024-03-29T10:10:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.131+0000 7ff71ea00700 1 -- 172.21.15.12:0/2841065677 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff718021030 con 0x7ff7280ceec0 2024-03-29T10:10:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.132+0000 7ff71ea00700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff70806adb0 0x7ff70806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.132+0000 7ff71ea00700 1 -- 172.21.15.12:0/2841065677 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5545+0+0 (secure 0 0 0) 0x7ff71808a770 con 0x7ff7280ceec0 2024-03-29T10:10:53.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.132+0000 7ff726000700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff70806adb0 0x7ff70806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.133+0000 7ff726000700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff70806adb0 0x7ff70806d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff728160e00 tx=0x7ff714008040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:53.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.137+0000 7ff71ea00700 1 -- 172.21.15.12:0/2841065677 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff7180559f0 con 0x7ff7280ceec0 2024-03-29T10:10:53.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.397+0000 7ff72d600700 1 -- 172.21.15.12:0/2841065677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7280b5c70 con 0x7ff7280ceec0 2024-03-29T10:10:53.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.397+0000 7ff71ea00700 1 -- 172.21.15.12:0/2841065677 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+3532 (secure 0 0 0) 0x7ff718015090 con 0x7ff7280ceec0 2024-03-29T10:10:53.397 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:53.397 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/4026715411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":4026715411},{"type":"v1","addr":"172.21.15.159:6836","nonce":4026715411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/225205526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":225205526},{"type":"v1","addr":"172.21.15.12:6834","nonce":225205526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6918,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2358504162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2358504162},{"type":"v1","addr":"172.21.15.159:6837","nonce":2358504162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":7072,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1573914733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":1573914733},{"type":"v1","addr":"172.21.15.12:6835","nonce":1573914733}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[]} 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 -- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff70806adb0 msgr2=0x7ff70806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff70806adb0 0x7ff70806d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff728160e00 tx=0x7ff714008040 comp rx=0 tx=0).stop 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 -- 172.21.15.12:0/2841065677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7280ceec0 msgr2=0x7ff7281410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7280ceec0 0x7ff7281410e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff72808c490 tx=0x7ff718004750 comp rx=0 tx=0).stop 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 -- 172.21.15.12:0/2841065677 shutdown_connections 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff70806adb0 0x7ff70806d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff728161800 0x7ff728146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7280ceec0 0x7ff7281410e0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 --2- 172.21.15.12:0/2841065677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72815d640 0x7ff728156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 -- 172.21.15.12:0/2841065677 >> 172.21.15.12:0/2841065677 conn(0x7ff72808aff0 msgr2=0x7ff7280d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 -- 172.21.15.12:0/2841065677 shutdown_connections 2024-03-29T10:10:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.400+0000 7ff72d600700 1 -- 172.21.15.12:0/2841065677 wait complete. 2024-03-29T10:10:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 40 2024-03-29T10:10:53.421 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T10:10:53.421 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T10:10:53.422 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T10:10:53.422 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:10:53.422 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T10:10:53.422 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T10:10:53.422 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T10:10:53.422 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:10:53.422 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T10:10:53.422 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T10:10:53.423 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T10:10:53.423 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:10:53.423 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T10:10:53.423 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T10:10:53.423 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T10:10:53.423 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:10:53.436 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T10:10:53.436 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T10:10:53.437 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T10:10:53.437 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T10:10:53.439 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T10:10:53.439 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T10:10:53.440 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T10:10:53.440 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T10:10:53.480 INFO:tasks.ceph.mds.c:Started 2024-03-29T10:10:53.480 DEBUG:teuthology.parallel:result is None 2024-03-29T10:10:53.483 INFO:tasks.ceph.mds.d:Started 2024-03-29T10:10:53.483 INFO:tasks.ceph.mds.a:Started 2024-03-29T10:10:53.483 DEBUG:teuthology.parallel:result is None 2024-03-29T10:10:53.483 DEBUG:teuthology.parallel:result is None 2024-03-29T10:10:53.485 INFO:tasks.ceph.mds.b:Started 2024-03-29T10:10:53.485 DEBUG:teuthology.parallel:result is None 2024-03-29T10:10:53.485 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T10:10:53.630 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T10:10:53.631 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T10:10:53.631 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T10:10:53.632 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:10:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.702+0000 7f1738800700 1 Processor -- start 2024-03-29T10:10:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.702+0000 7f1738800700 1 -- start start 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.703+0000 7f1738800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f173015d9d0 0x7f17300d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.703+0000 7f1738800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f17300d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.703+0000 7f1738800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1730161420 con 0x7f17300d2370 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.703+0000 7f1738800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17300d2820 con 0x7f173015d9d0 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.703+0000 7f1738800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1730161700 con 0x7f17300d16a0 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f17300d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f173015d9d0 0x7f17300d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f173015d9d0 0x7f17300d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:46438/0 (socket says 172.21.15.12:46438) 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f17300d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:43602/0 (socket says 172.21.15.12:43602) 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735e00700 1 -- 172.21.15.12:0/207059323 learned_addr learned my addr 172.21.15.12:0/207059323 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1736800700 1 --1- 172.21.15.12:0/207059323 >> v1:172.21.15.159:6789/0 conn(0x7f17300d2370 0x7f173015ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:43398/0 (socket says 172.21.15.12:43398) 2024-03-29T10:10:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735e00700 1 -- 172.21.15.12:0/207059323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 msgr2=0x7f17300d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735e00700 1 --2- 172.21.15.12:0/207059323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f17300d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735e00700 1 -- 172.21.15.12:0/207059323 >> v1:172.21.15.159:6789/0 conn(0x7f17300d2370 legacy=0x7f173015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735e00700 1 -- 172.21.15.12:0/207059323 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17301619e0 con 0x7f173015d9d0 2024-03-29T10:10:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735400700 1 --2- 172.21.15.12:0/207059323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f17300d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.704+0000 7f1735e00700 1 --2- 172.21.15.12:0/207059323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f173015d9d0 0x7f17300d3c30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f17240098f0 tx=0x7f1724009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95a0133daa20aef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1734a00700 1 -- 172.21.15.12:0/207059323 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724011830 con 0x7f173015d9d0 2024-03-29T10:10:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1734a00700 1 -- 172.21.15.12:0/207059323 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1724011990 con 0x7f173015d9d0 2024-03-29T10:10:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1734a00700 1 -- 172.21.15.12:0/207059323 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724011d20 con 0x7f173015d9d0 2024-03-29T10:10:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1738800700 1 -- 172.21.15.12:0/207059323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f173015d9d0 msgr2=0x7f17300d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1738800700 1 --2- 172.21.15.12:0/207059323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f173015d9d0 0x7f17300d3c30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f17240098f0 tx=0x7f1724009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1738800700 1 -- 172.21.15.12:0/207059323 shutdown_connections 2024-03-29T10:10:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1738800700 1 --2- 172.21.15.12:0/207059323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f173015d9d0 0x7f17300d3c30 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1738800700 1 --2- 172.21.15.12:0/207059323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f17300d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1738800700 1 -- 172.21.15.12:0/207059323 >> 172.21.15.12:0/207059323 conn(0x7f173008aff0 msgr2=0x7f17300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1738800700 1 -- 172.21.15.12:0/207059323 shutdown_connections 2024-03-29T10:10:53.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.705+0000 7f1738800700 1 -- 172.21.15.12:0/207059323 wait complete. 2024-03-29T10:10:53.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.707+0000 7f1738800700 1 Processor -- start 2024-03-29T10:10:53.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.707+0000 7f1738800700 1 -- start start 2024-03-29T10:10:53.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.708+0000 7f1738800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f1730155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.708+0000 7f1738800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d2370 0x7f17300b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.708+0000 7f1738800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f173015d9d0 0x7f1730159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.708+0000 7f1738800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173015e810 con 0x7f17300d2370 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.708+0000 7f1738800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17300b8df0 con 0x7f173015d9d0 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.708+0000 7f1738800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173015dee0 con 0x7f17300d16a0 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.708+0000 7f1735400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d2370 0x7f17300b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.708+0000 7f1735400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d2370 0x7f17300b8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36930/0 (socket says 172.21.15.12:36930) 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.708+0000 7f1735400700 1 -- 172.21.15.12:0/4204106863 learned_addr learned my addr 172.21.15.12:0/4204106863 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.709+0000 7f1735e00700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f1730155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.709+0000 7f1736800700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f173015d9d0 0x7f1730159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.709+0000 7f1735400700 1 -- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 msgr2=0x7f1730155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.709+0000 7f1735400700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f1730155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.709+0000 7f1735400700 1 -- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f173015d9d0 msgr2=0x7f1730159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.709+0000 7f1735400700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f173015d9d0 0x7f1730159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.709+0000 7f1735400700 1 -- 172.21.15.12:0/4204106863 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1720009430 con 0x7f17300d2370 2024-03-29T10:10:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.709+0000 7f1735e00700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f1730155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:10:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.709+0000 7f1735400700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d2370 0x7f17300b8480 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f172000ebe0 tx=0x7f172000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.709+0000 7f172a000700 1 -- 172.21.15.12:0/4204106863 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720009780 con 0x7f17300d2370 2024-03-29T10:10:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.710+0000 7f172a000700 1 -- 172.21.15.12:0/4204106863 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1720005a20 con 0x7f17300d2370 2024-03-29T10:10:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.710+0000 7f172a000700 1 -- 172.21.15.12:0/4204106863 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720005dd0 con 0x7f17300d2370 2024-03-29T10:10:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.710+0000 7f1738800700 1 -- 172.21.15.12:0/4204106863 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1724009550 con 0x7f17300d2370 2024-03-29T10:10:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.710+0000 7f1738800700 1 -- 172.21.15.12:0/4204106863 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173015e4a0 con 0x7f17300d2370 2024-03-29T10:10:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.711+0000 7f1738800700 1 -- 172.21.15.12:0/4204106863 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17300d1ca0 con 0x7f17300d2370 2024-03-29T10:10:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.712+0000 7f172a000700 1 -- 172.21.15.12:0/4204106863 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f172000c850 con 0x7f17300d2370 2024-03-29T10:10:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.713+0000 7f172a000700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f171406adb0 0x7f171406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.713+0000 7f172a000700 1 -- 172.21.15.12:0/4204106863 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5545+0+0 (secure 0 0 0) 0x7f1720089550 con 0x7f17300d2370 2024-03-29T10:10:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.713+0000 7f1735e00700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f171406adb0 0x7f171406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:53.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.714+0000 7f1735e00700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f171406adb0 0x7f171406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1724014040 tx=0x7f1724008030 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:53.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.717+0000 7f172a000700 1 -- 172.21.15.12:0/4204106863 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1720054750 con 0x7f17300d2370 2024-03-29T10:10:53.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:53.947+0000 7f1738800700 1 -- 172.21.15.12:0/4204106863 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1730067720 con 0x7f17300d2370 2024-03-29T10:10:54.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.673+0000 7f172a000700 1 -- 172.21.15.12:0/4204106863 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1720057d70 con 0x7f17300d2370 2024-03-29T10:10:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 -- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f171406adb0 msgr2=0x7f171406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f171406adb0 0x7f171406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1724014040 tx=0x7f1724008030 comp rx=0 tx=0).stop 2024-03-29T10:10:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 -- 172.21.15.12:0/4204106863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d2370 msgr2=0x7f17300b8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d2370 0x7f17300b8480 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f172000ebe0 tx=0x7f172000eef0 comp rx=0 tx=0).stop 2024-03-29T10:10:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 -- 172.21.15.12:0/4204106863 shutdown_connections 2024-03-29T10:10:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f171406adb0 0x7f171406d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f173015d9d0 0x7f1730159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17300d16a0 0x7f1730155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 --2- 172.21.15.12:0/4204106863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d2370 0x7f17300b8480 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 -- 172.21.15.12:0/4204106863 >> 172.21.15.12:0/4204106863 conn(0x7f173008aff0 msgr2=0x7f17300b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:54.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 -- 172.21.15.12:0/4204106863 shutdown_connections 2024-03-29T10:10:54.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.676+0000 7f1738800700 1 -- 172.21.15.12:0/4204106863 wait complete. 2024-03-29T10:10:54.677 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T10:10:54.698 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T10:10:54.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.901+0000 7f714b400700 1 Processor -- start 2024-03-29T10:10:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.901+0000 7f714b400700 1 -- start start 2024-03-29T10:10:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.902+0000 7f714b400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f71440d18b0 0x7f71440cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.902+0000 7f714b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7144161590 0x7f7144161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.902+0000 7f714b400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71440ceda0 con 0x7f7144160f10 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.902+0000 7f714b400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71440d1d30 con 0x7f7144161590 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.902+0000 7f714b400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7144161f60 con 0x7f71440d18b0 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.902+0000 7f7149400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7144161590 0x7f7144161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.902+0000 7f7149400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7144161590 0x7f7144161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36934/0 (socket says 172.21.15.12:36934) 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.902+0000 7f7149400700 1 -- 172.21.15.12:0/3178703577 learned_addr learned my addr 172.21.15.12:0/3178703577 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.903+0000 7f7143e00700 1 --2- 172.21.15.12:0/3178703577 >> v2:172.21.15.159:3300/0 conn(0x7f71440d18b0 0x7f71440cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.903+0000 7f7149400700 1 -- 172.21.15.12:0/3178703577 >> v2:172.21.15.159:3300/0 conn(0x7f71440d18b0 msgr2=0x7f71440cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.903+0000 7f7149400700 1 --2- 172.21.15.12:0/3178703577 >> v2:172.21.15.159:3300/0 conn(0x7f71440d18b0 0x7f71440cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.903+0000 7f7149400700 1 -- 172.21.15.12:0/3178703577 >> v1:172.21.15.159:6789/0 conn(0x7f7144160f10 legacy=0x7f71440d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.903+0000 7f7149400700 1 -- 172.21.15.12:0/3178703577 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71441612f0 con 0x7f7144161590 2024-03-29T10:10:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.903+0000 7f7149400700 1 --2- 172.21.15.12:0/3178703577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 0x7f7144161a20 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f714408ca30 tx=0x7f713c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ae11f38d843588 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.903+0000 7f7143400700 1 -- 172.21.15.12:0/3178703577 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c00e070 con 0x7f7144161590 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.903+0000 7f7143400700 1 -- 172.21.15.12:0/3178703577 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c00b910 con 0x7f7144161590 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.903+0000 7f7143400700 1 -- 172.21.15.12:0/3178703577 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c00bcc0 con 0x7f7144161590 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.904+0000 7f714b400700 1 -- 172.21.15.12:0/3178703577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 msgr2=0x7f7144161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.904+0000 7f714b400700 1 --2- 172.21.15.12:0/3178703577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 0x7f7144161a20 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f714408ca30 tx=0x7f713c00b330 comp rx=0 tx=0).stop 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.904+0000 7f714b400700 1 -- 172.21.15.12:0/3178703577 shutdown_connections 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.904+0000 7f714b400700 1 --2- 172.21.15.12:0/3178703577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 0x7f7144161a20 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.904+0000 7f714b400700 1 --2- 172.21.15.12:0/3178703577 >> v2:172.21.15.159:3300/0 conn(0x7f71440d18b0 0x7f71440cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.904+0000 7f714b400700 1 -- 172.21.15.12:0/3178703577 >> 172.21.15.12:0/3178703577 conn(0x7f714408aff0 msgr2=0x7f71440d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.904+0000 7f714b400700 1 -- 172.21.15.12:0/3178703577 shutdown_connections 2024-03-29T10:10:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.904+0000 7f714b400700 1 -- 172.21.15.12:0/3178703577 wait complete. 2024-03-29T10:10:54.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.906+0000 7f714b400700 1 Processor -- start 2024-03-29T10:10:54.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.906+0000 7f714b400700 1 -- start start 2024-03-29T10:10:54.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f714b400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71440d18b0 0x7f71441e74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:54.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f714b400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7144160f10 0x7f71441e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:54.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f714b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 0x7f714415aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:54.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f714b400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714415a350 con 0x7f7144161590 2024-03-29T10:10:54.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f714b400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714415cbb0 con 0x7f71440d18b0 2024-03-29T10:10:54.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f714b400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714415bab0 con 0x7f7144160f10 2024-03-29T10:10:54.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f7149400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 0x7f714415aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f7149400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 0x7f714415aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36950/0 (socket says 172.21.15.12:36950) 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f7149400700 1 -- 172.21.15.12:0/846808089 learned_addr learned my addr 172.21.15.12:0/846808089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f7148a00700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71440d18b0 0x7f71441e74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.907+0000 7f7143e00700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7144160f10 0x7f71441e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f7149400700 1 -- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7144160f10 msgr2=0x7f71441e7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f7149400700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7144160f10 0x7f71441e7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f7149400700 1 -- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71440d18b0 msgr2=0x7f71441e74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f7149400700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71440d18b0 0x7f71441e74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f7149400700 1 -- 172.21.15.12:0/846808089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c009b30 con 0x7f7144161590 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f7149400700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 0x7f714415aed0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f713c00b370 tx=0x7f713c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f7140c00700 1 -- 172.21.15.12:0/846808089 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c004880 con 0x7f7144161590 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f7140c00700 1 -- 172.21.15.12:0/846808089 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c0099b0 con 0x7f7144161590 2024-03-29T10:10:54.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f7140c00700 1 -- 172.21.15.12:0/846808089 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c01c680 con 0x7f7144161590 2024-03-29T10:10:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f714b400700 1 -- 172.21.15.12:0/846808089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714409f0d0 con 0x7f7144161590 2024-03-29T10:10:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.908+0000 7f714b400700 1 -- 172.21.15.12:0/846808089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714409f690 con 0x7f7144161590 2024-03-29T10:10:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.909+0000 7f714b400700 1 -- 172.21.15.12:0/846808089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7144067720 con 0x7f7144161590 2024-03-29T10:10:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.910+0000 7f7140c00700 1 -- 172.21.15.12:0/846808089 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f713c01c7e0 con 0x7f7144161590 2024-03-29T10:10:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.911+0000 7f7140c00700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f712406adb0 0x7f712406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.911+0000 7f7140c00700 1 -- 172.21.15.12:0/846808089 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f713c08aab0 con 0x7f7144161590 2024-03-29T10:10:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.912+0000 7f7148a00700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f712406adb0 0x7f712406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.912+0000 7f7148a00700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f712406adb0 0x7f712406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7138007fd0 tx=0x7f7138009040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:54.916+0000 7f7140c00700 1 -- 172.21.15.12:0/846808089 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f713c055db0 con 0x7f7144161590 2024-03-29T10:10:55.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.216+0000 7f714b400700 1 -- 172.21.15.12:0/846808089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f71440ce1b0 con 0x7f7144161590 2024-03-29T10:10:55.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.217+0000 7f7140c00700 1 -- 172.21.15.12:0/846808089 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f713c0593d0 con 0x7f7144161590 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T10:10:55.218 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:10:55.219 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T10:10:55.220 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.222 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T10:10:55.223 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:10:55.225 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.226 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.227 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:10:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:10:55.229 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:10:55.229 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:10:55.229 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.229 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:55.229 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T10:10:55.229 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 -- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f712406adb0 msgr2=0x7f712406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f712406adb0 0x7f712406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7138007fd0 tx=0x7f7138009040 comp rx=0 tx=0).stop 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 -- 172.21.15.12:0/846808089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 msgr2=0x7f714415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 0x7f714415aed0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f713c00b370 tx=0x7f713c004ad0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 -- 172.21.15.12:0/846808089 shutdown_connections 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f712406adb0 0x7f712406d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7144161590 0x7f714415aed0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71440d18b0 0x7f71441e74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 --2- 172.21.15.12:0/846808089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7144160f10 0x7f71441e7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 -- 172.21.15.12:0/846808089 >> 172.21.15.12:0/846808089 conn(0x7f714408aff0 msgr2=0x7f714406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:55.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 -- 172.21.15.12:0/846808089 shutdown_connections 2024-03-29T10:10:55.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.219+0000 7f714b400700 1 -- 172.21.15.12:0/846808089 wait complete. 2024-03-29T10:10:55.237 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:55.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.435+0000 7f7648a00700 1 Processor -- start 2024-03-29T10:10:55.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.435+0000 7f7648a00700 1 -- start start 2024-03-29T10:10:55.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.436+0000 7f7648a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f764415d640 0x7f76440d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:55.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.436+0000 7f7648a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76440d30d0 con 0x7f76440ceec0 2024-03-29T10:10:55.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.436+0000 7f7648a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76440b6840 con 0x7f76441619d0 2024-03-29T10:10:55.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.436+0000 7f7648a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76440d2880 con 0x7f764415d640 2024-03-29T10:10:55.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.437+0000 7f7641400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f76440ceec0 0x7f76441612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60630/0 (socket says 172.21.15.12:60630) 2024-03-29T10:10:55.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.437+0000 7f7641400700 1 -- 172.21.15.12:0/1297153410 learned_addr learned my addr 172.21.15.12:0/1297153410 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:55.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.437+0000 7f7641e00700 1 --2- 172.21.15.12:0/1297153410 >> v2:172.21.15.159:3300/0 conn(0x7f764415d640 0x7f76440d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:55.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.437+0000 7f7640a00700 1 -- 172.21.15.12:0/1297153410 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121226355 0 0) 0x7f76440d30d0 con 0x7f76440ceec0 2024-03-29T10:10:55.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.437+0000 7f7640a00700 1 -- 172.21.15.12:0/1297153410 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76280035b0 con 0x7f76440ceec0 2024-03-29T10:10:55.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.437+0000 7f7641e00700 1 -- 172.21.15.12:0/1297153410 >> v1:172.21.15.159:6789/0 conn(0x7f76441619d0 legacy=0x7f76440b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.437+0000 7f7641e00700 1 -- 172.21.15.12:0/1297153410 >> v1:172.21.15.12:6789/0 conn(0x7f76440ceec0 legacy=0x7f76441612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.437+0000 7f7641e00700 1 -- 172.21.15.12:0/1297153410 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440d2b30 con 0x7f764415d640 2024-03-29T10:10:55.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.438+0000 7f7641e00700 1 --2- 172.21.15.12:0/1297153410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f764415d640 0x7f76440d2340 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7634009840 tx=0x7f7634009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48dafbcf9e2949cf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:55.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.438+0000 7f7640a00700 1 -- 172.21.15.12:0/1297153410 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634011830 con 0x7f764415d640 2024-03-29T10:10:55.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.438+0000 7f7640a00700 1 -- 172.21.15.12:0/1297153410 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7634011990 con 0x7f764415d640 2024-03-29T10:10:55.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.438+0000 7f7640a00700 1 -- 172.21.15.12:0/1297153410 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634011d20 con 0x7f764415d640 2024-03-29T10:10:55.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.438+0000 7f7648a00700 1 -- 172.21.15.12:0/1297153410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f764415d640 msgr2=0x7f76440d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.438+0000 7f7648a00700 1 --2- 172.21.15.12:0/1297153410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f764415d640 0x7f76440d2340 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7634009840 tx=0x7f7634009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:55.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.438+0000 7f7648a00700 1 -- 172.21.15.12:0/1297153410 shutdown_connections 2024-03-29T10:10:55.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.439+0000 7f7648a00700 1 --2- 172.21.15.12:0/1297153410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f764415d640 0x7f76440d2340 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.439+0000 7f7648a00700 1 -- 172.21.15.12:0/1297153410 >> 172.21.15.12:0/1297153410 conn(0x7f764408aff0 msgr2=0x7f76440d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:55.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.439+0000 7f7648a00700 1 -- 172.21.15.12:0/1297153410 shutdown_connections 2024-03-29T10:10:55.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.439+0000 7f7648a00700 1 -- 172.21.15.12:0/1297153410 wait complete. 2024-03-29T10:10:55.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.440+0000 7f7648a00700 1 Processor -- start 2024-03-29T10:10:55.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.440+0000 7f7648a00700 1 -- start start 2024-03-29T10:10:55.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.441+0000 7f7648a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76440ceec0 0x7f7644151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.441+0000 7f7648a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f764415d640 0x7f76441566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.441+0000 7f7648a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76441619d0 0x7f7644156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.441+0000 7f7648a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76441590e0 con 0x7f76440ceec0 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.441+0000 7f7648a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764415f330 con 0x7f76441619d0 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.441+0000 7f7648a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7644158840 con 0x7f764415d640 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7641e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76440ceec0 0x7f7644151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7641e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76440ceec0 0x7f7644151b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36954/0 (socket says 172.21.15.12:36954) 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7641e00700 1 -- 172.21.15.12:0/4268719665 learned_addr learned my addr 172.21.15.12:0/4268719665 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7641400700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f764415d640 0x7f76441566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7642800700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76441619d0 0x7f7644156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7641e00700 1 -- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f764415d640 msgr2=0x7f76441566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7641e00700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f764415d640 0x7f76441566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7641e00700 1 -- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76441619d0 msgr2=0x7f7644156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7641e00700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76441619d0 0x7f7644156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7641e00700 1 -- 172.21.15.12:0/4268719665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76340094a0 con 0x7f76440ceec0 2024-03-29T10:10:55.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.442+0000 7f7641e00700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76440ceec0 0x7f7644151b60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7634009fd0 tx=0x7f7634004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:55.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.443+0000 7f763a000700 1 -- 172.21.15.12:0/4268719665 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76340036a0 con 0x7f76440ceec0 2024-03-29T10:10:55.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.443+0000 7f763a000700 1 -- 172.21.15.12:0/4268719665 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763401f070 con 0x7f76440ceec0 2024-03-29T10:10:55.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.443+0000 7f7648a00700 1 -- 172.21.15.12:0/4268719665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7644158b20 con 0x7f76440ceec0 2024-03-29T10:10:55.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.443+0000 7f763a000700 1 -- 172.21.15.12:0/4268719665 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634003a60 con 0x7f76440ceec0 2024-03-29T10:10:55.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.443+0000 7f7648a00700 1 -- 172.21.15.12:0/4268719665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7644158460 con 0x7f76440ceec0 2024-03-29T10:10:55.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.444+0000 7f7648a00700 1 -- 172.21.15.12:0/4268719665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76441585f0 con 0x7f76440ceec0 2024-03-29T10:10:55.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.445+0000 7f763a000700 1 -- 172.21.15.12:0/4268719665 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7634003bc0 con 0x7f76440ceec0 2024-03-29T10:10:55.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.446+0000 7f763a000700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f762406adb0 0x7f762406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:55.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.446+0000 7f763a000700 1 -- 172.21.15.12:0/4268719665 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7634021070 con 0x7f76440ceec0 2024-03-29T10:10:55.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.447+0000 7f7641400700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f762406adb0 0x7f762406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:55.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.447+0000 7f7641400700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f762406adb0 0x7f762406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76440b9450 tx=0x7f7630007dc0 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:55.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.450+0000 7f763a000700 1 -- 172.21.15.12:0/4268719665 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7634055dd0 con 0x7f76440ceec0 2024-03-29T10:10:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.667+0000 7f7648a00700 1 -- 172.21.15.12:0/4268719665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7644067720 con 0x7f76440ceec0 2024-03-29T10:10:55.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.668+0000 7f763a000700 1 -- 172.21.15.12:0/4268719665 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+12391 (secure 0 0 0) 0x7f7634015090 con 0x7f76440ceec0 2024-03-29T10:10:55.668 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:55.668 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":104,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:54.668966+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:55.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.670+0000 7f7648a00700 1 -- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f762406adb0 msgr2=0x7f762406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.670+0000 7f7648a00700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f762406adb0 0x7f762406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76440b9450 tx=0x7f7630007dc0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.670+0000 7f7648a00700 1 -- 172.21.15.12:0/4268719665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76440ceec0 msgr2=0x7f7644151b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.670+0000 7f7648a00700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76440ceec0 0x7f7644151b60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7634009fd0 tx=0x7f7634004750 comp rx=0 tx=0).stop 2024-03-29T10:10:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.670+0000 7f7648a00700 1 -- 172.21.15.12:0/4268719665 shutdown_connections 2024-03-29T10:10:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.670+0000 7f7648a00700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f762406adb0 0x7f762406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.670+0000 7f7648a00700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76441619d0 0x7f7644156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.670+0000 7f7648a00700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76440ceec0 0x7f7644151b60 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.670+0000 7f7648a00700 1 --2- 172.21.15.12:0/4268719665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f764415d640 0x7f76441566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.670+0000 7f7648a00700 1 -- 172.21.15.12:0/4268719665 >> 172.21.15.12:0/4268719665 conn(0x7f764408aff0 msgr2=0x7f76440cffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.671+0000 7f7648a00700 1 -- 172.21.15.12:0/4268719665 shutdown_connections 2024-03-29T10:10:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.671+0000 7f7648a00700 1 -- 172.21.15.12:0/4268719665 wait complete. 2024-03-29T10:10:55.688 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:10:55.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:10:55.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.891+0000 7fd1ee200700 1 Processor -- start 2024-03-29T10:10:55.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.891+0000 7fd1ee200700 1 -- start start 2024-03-29T10:10:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.892+0000 7fd1ee200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd1e815d990 0x7fd1e80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.892+0000 7fd1ee200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80d0970 con 0x7fd1e809a750 2024-03-29T10:10:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.892+0000 7fd1ee200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e806c020 con 0x7fd1e8099a60 2024-03-29T10:10:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.892+0000 7fd1ee200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e80c81a0 con 0x7fd1e815d990 2024-03-29T10:10:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.892+0000 7fd1e6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd1e815d990 0x7fd1e80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.892+0000 7fd1e7e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd1e809a750 0x7fd1e806c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60636/0 (socket says 172.21.15.12:60636) 2024-03-29T10:10:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.892+0000 7fd1e7e00700 1 -- 172.21.15.12:0/3383488373 learned_addr learned my addr 172.21.15.12:0/3383488373 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.893+0000 7fd1e6a00700 1 -- 172.21.15.12:0/3383488373 >> v1:172.21.15.159:6789/0 conn(0x7fd1e8099a60 legacy=0x7fd1e8099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.893+0000 7fd1e6a00700 1 -- 172.21.15.12:0/3383488373 >> v1:172.21.15.12:6789/0 conn(0x7fd1e809a750 legacy=0x7fd1e806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.893+0000 7fd1e6a00700 1 -- 172.21.15.12:0/3383488373 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80d0690 con 0x7fd1e815d990 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.893+0000 7fd1e6a00700 1 --2- 172.21.15.12:0/3383488373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 0x7fd1e80d0060 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4009840 tx=0x7fd1d4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c6331d9f75ec6c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.893+0000 7fd1e6000700 1 -- 172.21.15.12:0/3383488373 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4011830 con 0x7fd1e815d990 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.893+0000 7fd1e6000700 1 -- 172.21.15.12:0/3383488373 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d4011990 con 0x7fd1e815d990 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.893+0000 7fd1e6000700 1 -- 172.21.15.12:0/3383488373 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4011d20 con 0x7fd1e815d990 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.894+0000 7fd1ee200700 1 -- 172.21.15.12:0/3383488373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 msgr2=0x7fd1e80d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.894+0000 7fd1ee200700 1 --2- 172.21.15.12:0/3383488373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 0x7fd1e80d0060 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4009840 tx=0x7fd1d4009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.894+0000 7fd1ee200700 1 -- 172.21.15.12:0/3383488373 shutdown_connections 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.894+0000 7fd1ee200700 1 --2- 172.21.15.12:0/3383488373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 0x7fd1e80d0060 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.894+0000 7fd1ee200700 1 -- 172.21.15.12:0/3383488373 >> 172.21.15.12:0/3383488373 conn(0x7fd1e80b2e20 msgr2=0x7fd1e80b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.894+0000 7fd1ee200700 1 -- 172.21.15.12:0/3383488373 shutdown_connections 2024-03-29T10:10:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.894+0000 7fd1ee200700 1 -- 172.21.15.12:0/3383488373 wait complete. 2024-03-29T10:10:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.896+0000 7fd1ee200700 1 Processor -- start 2024-03-29T10:10:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.896+0000 7fd1ee200700 1 -- start start 2024-03-29T10:10:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1ee200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1e8099a60 0x7fd1e8141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1ee200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1e809a750 0x7fd1e80c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1ee200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 0x7fd1e80c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1ee200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e8141990 con 0x7fd1e815d990 2024-03-29T10:10:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1ee200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e8154df0 con 0x7fd1e8099a60 2024-03-29T10:10:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1ee200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e8160a30 con 0x7fd1e809a750 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 0x7fd1e80c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 0x7fd1e80c6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36968/0 (socket says 172.21.15.12:36968) 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1e7e00700 1 -- 172.21.15.12:0/1910689075 learned_addr learned my addr 172.21.15.12:0/1910689075 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1e6a00700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1e809a750 0x7fd1e80c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.897+0000 7fd1e7400700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1e8099a60 0x7fd1e8141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.898+0000 7fd1e7e00700 1 -- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1e809a750 msgr2=0x7fd1e80c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.898+0000 7fd1e7e00700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1e809a750 0x7fd1e80c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.898+0000 7fd1e7e00700 1 -- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1e8099a60 msgr2=0x7fd1e8141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.898+0000 7fd1e7e00700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1e8099a60 0x7fd1e8141450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.898+0000 7fd1e7e00700 1 -- 172.21.15.12:0/1910689075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d40094a0 con 0x7fd1e815d990 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.898+0000 7fd1e7e00700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 0x7fd1e80c6bb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000cbf0 tx=0x7fd1e000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.898+0000 7fd1df400700 1 -- 172.21.15.12:0/1910689075 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e000aa60 con 0x7fd1e815d990 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.898+0000 7fd1df400700 1 -- 172.21.15.12:0/1910689075 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e000abc0 con 0x7fd1e815d990 2024-03-29T10:10:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.898+0000 7fd1df400700 1 -- 172.21.15.12:0/1910689075 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0012400 con 0x7fd1e815d990 2024-03-29T10:10:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.898+0000 7fd1ee200700 1 -- 172.21.15.12:0/1910689075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e80ce870 con 0x7fd1e815d990 2024-03-29T10:10:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.899+0000 7fd1ee200700 1 -- 172.21.15.12:0/1910689075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e80c6400 con 0x7fd1e815d990 2024-03-29T10:10:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.900+0000 7fd1ee200700 1 -- 172.21.15.12:0/1910689075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e8067720 con 0x7fd1e815d990 2024-03-29T10:10:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.900+0000 7fd1df400700 1 -- 172.21.15.12:0/1910689075 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd1e001b3b0 con 0x7fd1e815d990 2024-03-29T10:10:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.901+0000 7fd1df400700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1c806adb0 0x7fd1c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.901+0000 7fd1df400700 1 -- 172.21.15.12:0/1910689075 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd1e0088c60 con 0x7fd1e815d990 2024-03-29T10:10:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.902+0000 7fd1e7400700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1c806adb0 0x7fd1c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.903+0000 7fd1e7400700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1c806adb0 0x7fd1c806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8000f60 tx=0x7fd1d8009040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:55.906+0000 7fd1df400700 1 -- 172.21.15.12:0/1910689075 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1e0054130 con 0x7fd1e815d990 2024-03-29T10:10:56.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.119+0000 7fd1ee200700 1 -- 172.21.15.12:0/1910689075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1e80b2a40 con 0x7fd1e815d990 2024-03-29T10:10:56.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.119+0000 7fd1df400700 1 -- 172.21.15.12:0/1910689075 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd1e00170a0 con 0x7fd1e815d990 2024-03-29T10:10:56.119 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.121+0000 7fd1ee200700 1 -- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1c806adb0 msgr2=0x7fd1c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.121+0000 7fd1ee200700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1c806adb0 0x7fd1c806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8000f60 tx=0x7fd1d8009040 comp rx=0 tx=0).stop 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.122+0000 7fd1ee200700 1 -- 172.21.15.12:0/1910689075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 msgr2=0x7fd1e80c6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.122+0000 7fd1ee200700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 0x7fd1e80c6bb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000cbf0 tx=0x7fd1e000cf00 comp rx=0 tx=0).stop 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.122+0000 7fd1ee200700 1 -- 172.21.15.12:0/1910689075 shutdown_connections 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.122+0000 7fd1ee200700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd1c806adb0 0x7fd1c806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.122+0000 7fd1ee200700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1e815d990 0x7fd1e80c6bb0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.122+0000 7fd1ee200700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1e8099a60 0x7fd1e8141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.122+0000 7fd1ee200700 1 --2- 172.21.15.12:0/1910689075 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1e809a750 0x7fd1e80c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.122+0000 7fd1ee200700 1 -- 172.21.15.12:0/1910689075 >> 172.21.15.12:0/1910689075 conn(0x7fd1e80b2e20 msgr2=0x7fd1e80b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.122+0000 7fd1ee200700 1 -- 172.21.15.12:0/1910689075 shutdown_connections 2024-03-29T10:10:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.122+0000 7fd1ee200700 1 -- 172.21.15.12:0/1910689075 wait complete. 2024-03-29T10:10:56.139 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T10:10:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.336+0000 7fb169e00700 1 Processor -- start 2024-03-29T10:10:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.336+0000 7fb169e00700 1 -- start start 2024-03-29T10:10:56.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.337+0000 7fb169e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb16415b5d0 0x7fb1640ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:56.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.337+0000 7fb169e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16415fd50 con 0x7fb1640ac7d0 2024-03-29T10:10:56.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.337+0000 7fb169e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16415f140 con 0x7fb1640cb3f0 2024-03-29T10:10:56.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.337+0000 7fb169e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb16415b9b0 con 0x7fb16415b5d0 2024-03-29T10:10:56.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.338+0000 7fb163400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb1640ac7d0 0x7fb16406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60652/0 (socket says 172.21.15.12:60652) 2024-03-29T10:10:56.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.338+0000 7fb163400700 1 -- 172.21.15.12:0/2919260253 learned_addr learned my addr 172.21.15.12:0/2919260253 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:56.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.338+0000 7fb162a00700 1 --2- 172.21.15.12:0/2919260253 >> v2:172.21.15.159:3300/0 conn(0x7fb16415b5d0 0x7fb1640ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:56.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.338+0000 7fb161600700 1 -- 172.21.15.12:0/2919260253 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702496435 0 0) 0x7fb16415fd50 con 0x7fb1640ac7d0 2024-03-29T10:10:56.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.338+0000 7fb161600700 1 -- 172.21.15.12:0/2919260253 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1480035b0 con 0x7fb1640ac7d0 2024-03-29T10:10:56.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.338+0000 7fb162a00700 1 -- 172.21.15.12:0/2919260253 >> v1:172.21.15.159:6789/0 conn(0x7fb1640cb3f0 legacy=0x7fb1640ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.338+0000 7fb162a00700 1 -- 172.21.15.12:0/2919260253 >> v1:172.21.15.12:6789/0 conn(0x7fb1640ac7d0 legacy=0x7fb16406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.338+0000 7fb162a00700 1 -- 172.21.15.12:0/2919260253 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16415f420 con 0x7fb16415b5d0 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.339+0000 7fb162a00700 1 --2- 172.21.15.12:0/2919260253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb16415b5d0 0x7fb1640ae3f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb154009840 tx=0x7fb154009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f3fd41cf1feeb8f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.339+0000 7fb161600700 1 -- 172.21.15.12:0/2919260253 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154011830 con 0x7fb16415b5d0 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.339+0000 7fb161600700 1 -- 172.21.15.12:0/2919260253 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb154011990 con 0x7fb16415b5d0 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.339+0000 7fb161600700 1 -- 172.21.15.12:0/2919260253 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154011d20 con 0x7fb16415b5d0 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.339+0000 7fb169e00700 1 -- 172.21.15.12:0/2919260253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb16415b5d0 msgr2=0x7fb1640ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.339+0000 7fb169e00700 1 --2- 172.21.15.12:0/2919260253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb16415b5d0 0x7fb1640ae3f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb154009840 tx=0x7fb154009b50 comp rx=0 tx=0).stop 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.340+0000 7fb169e00700 1 -- 172.21.15.12:0/2919260253 shutdown_connections 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.340+0000 7fb169e00700 1 --2- 172.21.15.12:0/2919260253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb16415b5d0 0x7fb1640ae3f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.340+0000 7fb169e00700 1 -- 172.21.15.12:0/2919260253 >> 172.21.15.12:0/2919260253 conn(0x7fb1640cf190 msgr2=0x7fb1640cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.340+0000 7fb169e00700 1 -- 172.21.15.12:0/2919260253 shutdown_connections 2024-03-29T10:10:56.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.340+0000 7fb169e00700 1 -- 172.21.15.12:0/2919260253 wait complete. 2024-03-29T10:10:56.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.341+0000 7fb169e00700 1 Processor -- start 2024-03-29T10:10:56.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.341+0000 7fb169e00700 1 -- start start 2024-03-29T10:10:56.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.342+0000 7fb169e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1640ac7d0 0x7fb1640725f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:56.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.342+0000 7fb169e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb16415b5d0 0x7fb164153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:56.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.342+0000 7fb169e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb164156370 0x7fb1640c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:56.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.342+0000 7fb169e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1641537a0 con 0x7fb16415b5d0 2024-03-29T10:10:56.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.342+0000 7fb169e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb164071550 con 0x7fb1640ac7d0 2024-03-29T10:10:56.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.342+0000 7fb169e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb164071830 con 0x7fb164156370 2024-03-29T10:10:56.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb162000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb16415b5d0 0x7fb164153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:56.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb162000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb16415b5d0 0x7fb164153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36980/0 (socket says 172.21.15.12:36980) 2024-03-29T10:10:56.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb162000700 1 -- 172.21.15.12:0/3806207001 learned_addr learned my addr 172.21.15.12:0/3806207001 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:56.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb162a00700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1640ac7d0 0x7fb1640725f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:56.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb163400700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb164156370 0x7fb1640c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:56.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb162000700 1 -- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb164156370 msgr2=0x7fb1640c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb162000700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb164156370 0x7fb1640c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb162000700 1 -- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1640ac7d0 msgr2=0x7fb1640725f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb162000700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1640ac7d0 0x7fb1640725f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb162000700 1 -- 172.21.15.12:0/3806207001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1540094a0 con 0x7fb16415b5d0 2024-03-29T10:10:56.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.343+0000 7fb162000700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb16415b5d0 0x7fb164153260 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb15000c4e0 tx=0x7fb15000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:56.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.344+0000 7fb15b400700 1 -- 172.21.15.12:0/3806207001 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1500052d0 con 0x7fb16415b5d0 2024-03-29T10:10:56.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.344+0000 7fb15b400700 1 -- 172.21.15.12:0/3806207001 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb150005430 con 0x7fb16415b5d0 2024-03-29T10:10:56.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.344+0000 7fb15b400700 1 -- 172.21.15.12:0/3806207001 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1500038c0 con 0x7fb16415b5d0 2024-03-29T10:10:56.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.344+0000 7fb169e00700 1 -- 172.21.15.12:0/3806207001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb164071b10 con 0x7fb16415b5d0 2024-03-29T10:10:56.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.344+0000 7fb169e00700 1 -- 172.21.15.12:0/3806207001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1640720d0 con 0x7fb16415b5d0 2024-03-29T10:10:56.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.345+0000 7fb169e00700 1 -- 172.21.15.12:0/3806207001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb164067720 con 0x7fb16415b5d0 2024-03-29T10:10:56.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.346+0000 7fb15b400700 1 -- 172.21.15.12:0/3806207001 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb150003a20 con 0x7fb16415b5d0 2024-03-29T10:10:56.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.347+0000 7fb15b400700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb14406adb0 0x7fb14406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:56.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.347+0000 7fb15b400700 1 -- 172.21.15.12:0/3806207001 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb150088570 con 0x7fb16415b5d0 2024-03-29T10:10:56.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.348+0000 7fb162a00700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb14406adb0 0x7fb14406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:56.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.348+0000 7fb162a00700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb14406adb0 0x7fb14406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb1540090b0 tx=0x7fb15401c040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:56.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.352+0000 7fb15b400700 1 -- 172.21.15.12:0/3806207001 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb1500538f0 con 0x7fb16415b5d0 2024-03-29T10:10:56.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.567+0000 7fb169e00700 1 -- 172.21.15.12:0/3806207001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb16406ea60 con 0x7fb16415b5d0 2024-03-29T10:10:56.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.567+0000 7fb15b400700 1 -- 172.21.15.12:0/3806207001 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb150017060 con 0x7fb16415b5d0 2024-03-29T10:10:56.567 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:56.567 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:10:56.567 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:10:56.567 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T10:10:56.567 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T10:10:56.567 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:10:56.567 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:10:56.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 -- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb14406adb0 msgr2=0x7fb14406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb14406adb0 0x7fb14406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb1540090b0 tx=0x7fb15401c040 comp rx=0 tx=0).stop 2024-03-29T10:10:56.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 -- 172.21.15.12:0/3806207001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb16415b5d0 msgr2=0x7fb164153260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb16415b5d0 0x7fb164153260 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb15000c4e0 tx=0x7fb15000c7f0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 -- 172.21.15.12:0/3806207001 shutdown_connections 2024-03-29T10:10:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb14406adb0 0x7fb14406d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb164156370 0x7fb1640c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1640ac7d0 0x7fb1640725f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 --2- 172.21.15.12:0/3806207001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb16415b5d0 0x7fb164153260 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 -- 172.21.15.12:0/3806207001 >> 172.21.15.12:0/3806207001 conn(0x7fb1640cf190 msgr2=0x7fb16415e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 -- 172.21.15.12:0/3806207001 shutdown_connections 2024-03-29T10:10:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.570+0000 7fb169e00700 1 -- 172.21.15.12:0/3806207001 wait complete. 2024-03-29T10:10:56.588 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T10:10:56.588 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T10:10:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.790+0000 7fc0b5e00700 1 Processor -- start 2024-03-29T10:10:56.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.790+0000 7fc0b5e00700 1 -- start start 2024-03-29T10:10:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.791+0000 7fc0b5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc0b00b3320 0x7fc0b006f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.791+0000 7fc0b5e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b006fc50 con 0x7fc0b006db70 2024-03-29T10:10:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.791+0000 7fc0b5e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b015e860 con 0x7fc0b00d0190 2024-03-29T10:10:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.791+0000 7fc0b5e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b015dd20 con 0x7fc0b00b3320 2024-03-29T10:10:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.791+0000 7fc0ae000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc0b00b3320 0x7fc0b006f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.791+0000 7fc0ae000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc0b00b3320 0x7fc0b006f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36994/0 (socket says 172.21.15.12:36994) 2024-03-29T10:10:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.791+0000 7fc0ae000700 1 -- 172.21.15.12:0/1421687520 learned_addr learned my addr 172.21.15.12:0/1421687520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.791+0000 7fc0af400700 1 --1- 172.21.15.12:0/1421687520 >> v1:172.21.15.12:6789/0 conn(0x7fc0b006db70 0x7fc0b006e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60656/0 (socket says 172.21.15.12:60656) 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.792+0000 7fc0ae000700 1 -- 172.21.15.12:0/1421687520 >> v1:172.21.15.159:6789/0 conn(0x7fc0b00d0190 legacy=0x7fc0b00d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.792+0000 7fc0ae000700 1 -- 172.21.15.12:0/1421687520 >> v1:172.21.15.12:6789/0 conn(0x7fc0b006db70 legacy=0x7fc0b006e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.792+0000 7fc0ae000700 1 -- 172.21.15.12:0/1421687520 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b015ecf0 con 0x7fc0b00b3320 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.792+0000 7fc0ad600700 1 -- 172.21.15.12:0/1421687520 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459205412 0 0) 0x7fc0a8001490 con 0x7fc0b006db70 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.792+0000 7fc0ae000700 1 --2- 172.21.15.12:0/1421687520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00b3320 0x7fc0b006f400 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0098f0 tx=0x7fc09c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce55bf5585be6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.792+0000 7fc0ad600700 1 -- 172.21.15.12:0/1421687520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c004230 con 0x7fc0b00b3320 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.792+0000 7fc0ad600700 1 -- 172.21.15.12:0/1421687520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c004390 con 0x7fc0b00b3320 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.792+0000 7fc0ad600700 1 -- 172.21.15.12:0/1421687520 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c0036a0 con 0x7fc0b00b3320 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.793+0000 7fc0b5e00700 1 -- 172.21.15.12:0/1421687520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00b3320 msgr2=0x7fc0b006f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.793+0000 7fc0b5e00700 1 --2- 172.21.15.12:0/1421687520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00b3320 0x7fc0b006f400 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0098f0 tx=0x7fc09c009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:56.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.793+0000 7fc0b5e00700 1 -- 172.21.15.12:0/1421687520 shutdown_connections 2024-03-29T10:10:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.793+0000 7fc0b5e00700 1 --2- 172.21.15.12:0/1421687520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00b3320 0x7fc0b006f400 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.793+0000 7fc0b5e00700 1 -- 172.21.15.12:0/1421687520 >> 172.21.15.12:0/1421687520 conn(0x7fc0b00b57b0 msgr2=0x7fc0b00b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.793+0000 7fc0b5e00700 1 -- 172.21.15.12:0/1421687520 shutdown_connections 2024-03-29T10:10:56.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.793+0000 7fc0b5e00700 1 -- 172.21.15.12:0/1421687520 wait complete. 2024-03-29T10:10:56.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.795+0000 7fc0b5e00700 1 Processor -- start 2024-03-29T10:10:56.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.796+0000 7fc0b5e00700 1 -- start start 2024-03-29T10:10:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.797+0000 7fc0b5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0b006db70 0x7fc0b0146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.797+0000 7fc0b5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b00b3320 0x7fc0b00b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.797+0000 7fc0b5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00d0190 0x7fc0b00b1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.797+0000 7fc0b5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b012fb50 con 0x7fc0b00d0190 2024-03-29T10:10:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.797+0000 7fc0b5e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b009a780 con 0x7fc0b006db70 2024-03-29T10:10:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.797+0000 7fc0b5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b00af160 con 0x7fc0b00b3320 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.797+0000 7fc0af400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00d0190 0x7fc0b00b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.797+0000 7fc0af400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00d0190 0x7fc0b00b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37010/0 (socket says 172.21.15.12:37010) 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.797+0000 7fc0af400700 1 -- 172.21.15.12:0/2263886989 learned_addr learned my addr 172.21.15.12:0/2263886989 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.797+0000 7fc0aea00700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0b006db70 0x7fc0b0146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0ae000700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b00b3320 0x7fc0b00b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0af400700 1 -- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b00b3320 msgr2=0x7fc0b00b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0af400700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b00b3320 0x7fc0b00b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0af400700 1 -- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0b006db70 msgr2=0x7fc0b0146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0af400700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0b006db70 0x7fc0b0146850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0af400700 1 -- 172.21.15.12:0/2263886989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c009550 con 0x7fc0b00d0190 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0af400700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00d0190 0x7fc0b00b1470 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800cca0 tx=0x7fc0a800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0a7400700 1 -- 172.21.15.12:0/2263886989 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a800cea0 con 0x7fc0b00d0190 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0a7400700 1 -- 172.21.15.12:0/2263886989 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a800ade0 con 0x7fc0b00d0190 2024-03-29T10:10:56.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0b5e00700 1 -- 172.21.15.12:0/2263886989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b012fda0 con 0x7fc0b00d0190 2024-03-29T10:10:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0a7400700 1 -- 172.21.15.12:0/2263886989 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8007310 con 0x7fc0b00d0190 2024-03-29T10:10:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.798+0000 7fc0b5e00700 1 -- 172.21.15.12:0/2263886989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b0130360 con 0x7fc0b00d0190 2024-03-29T10:10:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.799+0000 7fc0b5e00700 1 -- 172.21.15.12:0/2263886989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b0130580 con 0x7fc0b00d0190 2024-03-29T10:10:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.800+0000 7fc0a7400700 1 -- 172.21.15.12:0/2263886989 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc0a8007470 con 0x7fc0b00d0190 2024-03-29T10:10:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.801+0000 7fc0a7400700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc09006adb0 0x7fc09006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.801+0000 7fc0a7400700 1 -- 172.21.15.12:0/2263886989 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc0a8088e20 con 0x7fc0b00d0190 2024-03-29T10:10:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.802+0000 7fc0aea00700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc09006adb0 0x7fc09006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:56.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.802+0000 7fc0aea00700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc09006adb0 0x7fc09006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc0b01f6a20 tx=0x7fc0a0008040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:56.806+0000 7fc0a7400700 1 -- 172.21.15.12:0/2263886989 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0a80542d0 con 0x7fc0b00d0190 2024-03-29T10:10:57.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.021+0000 7fc0b5e00700 1 -- 172.21.15.12:0/2263886989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc0b006fc50 con 0x7fc0b00d0190 2024-03-29T10:10:57.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.102+0000 7fc0a7400700 1 -- 172.21.15.12:0/2263886989 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v105) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc0b006fc50 con 0x7fc0b00d0190 2024-03-29T10:10:57.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.106+0000 7fc0b5e00700 1 -- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc09006adb0 msgr2=0x7fc09006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:57.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.106+0000 7fc0b5e00700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc09006adb0 0x7fc09006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc0b01f6a20 tx=0x7fc0a0008040 comp rx=0 tx=0).stop 2024-03-29T10:10:57.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.106+0000 7fc0b5e00700 1 -- 172.21.15.12:0/2263886989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00d0190 msgr2=0x7fc0b00b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:57.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.106+0000 7fc0b5e00700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00d0190 0x7fc0b00b1470 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800cca0 tx=0x7fc0a800a340 comp rx=0 tx=0).stop 2024-03-29T10:10:57.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.107+0000 7fc0b5e00700 1 -- 172.21.15.12:0/2263886989 shutdown_connections 2024-03-29T10:10:57.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.107+0000 7fc0b5e00700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc09006adb0 0x7fc09006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:57.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.107+0000 7fc0b5e00700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b00d0190 0x7fc0b00b1470 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:57.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.107+0000 7fc0b5e00700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0b006db70 0x7fc0b0146850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:57.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.107+0000 7fc0b5e00700 1 --2- 172.21.15.12:0/2263886989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b00b3320 0x7fc0b00b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:57.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.107+0000 7fc0b5e00700 1 -- 172.21.15.12:0/2263886989 >> 172.21.15.12:0/2263886989 conn(0x7fc0b00b57b0 msgr2=0x7fc0b006e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:57.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.107+0000 7fc0b5e00700 1 -- 172.21.15.12:0/2263886989 shutdown_connections 2024-03-29T10:10:57.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.107+0000 7fc0b5e00700 1 -- 172.21.15.12:0/2263886989 wait complete. 2024-03-29T10:10:57.110 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T10:10:57.127 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T10:10:57.273 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:10:57.276 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:10:57.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.335+0000 7fe75a000700 1 Processor -- start 2024-03-29T10:10:57.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.335+0000 7fe75a000700 1 -- start start 2024-03-29T10:10:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.336+0000 7fe75a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7540d2540 0x7fe7541687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.336+0000 7fe75a000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe754161420 con 0x7fe7540d16a0 2024-03-29T10:10:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.337+0000 7fe75a000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe754161700 con 0x7fe75415d9d0 2024-03-29T10:10:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.337+0000 7fe75a000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7540d4340 con 0x7fe7540d2540 2024-03-29T10:10:57.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.337+0000 7fe752a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe7540d16a0 0x7fe7540d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60658/0 (socket says 172.21.15.12:60658) 2024-03-29T10:10:57.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.337+0000 7fe752a00700 1 -- 172.21.15.12:0/2724150785 learned_addr learned my addr 172.21.15.12:0/2724150785 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:57.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.337+0000 7fe753e00700 1 --2- 172.21.15.12:0/2724150785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7540d2540 0x7fe7541687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.337+0000 7fe752000700 1 -- 172.21.15.12:0/2724150785 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678930187 0 0) 0x7fe754161420 con 0x7fe7540d16a0 2024-03-29T10:10:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.337+0000 7fe752000700 1 -- 172.21.15.12:0/2724150785 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7380035b0 con 0x7fe7540d16a0 2024-03-29T10:10:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.337+0000 7fe752000700 1 -- 172.21.15.12:0/2724150785 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570787732 0 0) 0x7fe754161700 con 0x7fe75415d9d0 2024-03-29T10:10:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.337+0000 7fe752000700 1 -- 172.21.15.12:0/2724150785 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe754161420 con 0x7fe75415d9d0 2024-03-29T10:10:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.338+0000 7fe753e00700 1 -- 172.21.15.12:0/2724150785 >> v1:172.21.15.159:6789/0 conn(0x7fe75415d9d0 legacy=0x7fe7540d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.338+0000 7fe753e00700 1 -- 172.21.15.12:0/2724150785 >> v1:172.21.15.12:6789/0 conn(0x7fe7540d16a0 legacy=0x7fe7540d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.338+0000 7fe753e00700 1 -- 172.21.15.12:0/2724150785 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7541619e0 con 0x7fe7540d2540 2024-03-29T10:10:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.338+0000 7fe753e00700 1 --2- 172.21.15.12:0/2724150785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7540d2540 0x7fe7541687b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009dc0 tx=0x7fe74c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16601bcc2601c592 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.338+0000 7fe752000700 1 -- 172.21.15.12:0/2724150785 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c00e070 con 0x7fe7540d2540 2024-03-29T10:10:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.338+0000 7fe752000700 1 -- 172.21.15.12:0/2724150785 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c00bb20 con 0x7fe7540d2540 2024-03-29T10:10:57.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.338+0000 7fe752000700 1 -- 172.21.15.12:0/2724150785 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c012430 con 0x7fe7540d2540 2024-03-29T10:10:57.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.339+0000 7fe75a000700 1 -- 172.21.15.12:0/2724150785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7540d2540 msgr2=0x7fe7541687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:57.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.339+0000 7fe75a000700 1 --2- 172.21.15.12:0/2724150785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7540d2540 0x7fe7541687b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009dc0 tx=0x7fe74c00b330 comp rx=0 tx=0).stop 2024-03-29T10:10:57.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.339+0000 7fe75a000700 1 -- 172.21.15.12:0/2724150785 shutdown_connections 2024-03-29T10:10:57.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.339+0000 7fe75a000700 1 --2- 172.21.15.12:0/2724150785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7540d2540 0x7fe7541687b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.339+0000 7fe75a000700 1 -- 172.21.15.12:0/2724150785 >> 172.21.15.12:0/2724150785 conn(0x7fe75408aff0 msgr2=0x7fe7540d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.339+0000 7fe75a000700 1 -- 172.21.15.12:0/2724150785 shutdown_connections 2024-03-29T10:10:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.339+0000 7fe75a000700 1 -- 172.21.15.12:0/2724150785 wait complete. 2024-03-29T10:10:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.341+0000 7fe75a000700 1 Processor -- start 2024-03-29T10:10:57.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.341+0000 7fe75a000700 1 -- start start 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe75a000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7540d16a0 0x7fe75415c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe75a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7540d2540 0x7fe75415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe75a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe75415d9d0 0x7fe75415f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe75a000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75415c560 con 0x7fe7540d2540 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe75a000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75415e370 con 0x7fe7540d16a0 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe75a000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75415e650 con 0x7fe75415d9d0 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe752a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7540d2540 0x7fe75415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe753400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7540d16a0 0x7fe75415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe752a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7540d2540 0x7fe75415c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37024/0 (socket says 172.21.15.12:37024) 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe752a00700 1 -- 172.21.15.12:0/159226799 learned_addr learned my addr 172.21.15.12:0/159226799 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.343+0000 7fe753e00700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe75415d9d0 0x7fe75415f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe752a00700 1 -- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe75415d9d0 msgr2=0x7fe75415f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:57.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe752a00700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe75415d9d0 0x7fe75415f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:57.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe752a00700 1 -- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7540d16a0 msgr2=0x7fe75415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:57.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe752a00700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7540d16a0 0x7fe75415c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:57.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe752a00700 1 -- 172.21.15.12:0/159226799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c009a70 con 0x7fe7540d2540 2024-03-29T10:10:57.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe752a00700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7540d2540 0x7fe75415c020 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe740001260 tx=0x7fe74000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:57.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe74b400700 1 -- 172.21.15.12:0/159226799 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7400074e0 con 0x7fe7540d2540 2024-03-29T10:10:57.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe74b400700 1 -- 172.21.15.12:0/159226799 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe740007640 con 0x7fe7540d2540 2024-03-29T10:10:57.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe74b400700 1 -- 172.21.15.12:0/159226799 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740005a70 con 0x7fe7540d2540 2024-03-29T10:10:57.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe75a000700 1 -- 172.21.15.12:0/159226799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75415e930 con 0x7fe7540d2540 2024-03-29T10:10:57.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.344+0000 7fe75a000700 1 -- 172.21.15.12:0/159226799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75415eef0 con 0x7fe7540d2540 2024-03-29T10:10:57.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.345+0000 7fe75a000700 1 -- 172.21.15.12:0/159226799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7540d1ca0 con 0x7fe7540d2540 2024-03-29T10:10:57.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.347+0000 7fe74b400700 1 -- 172.21.15.12:0/159226799 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe740005bd0 con 0x7fe7540d2540 2024-03-29T10:10:57.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.348+0000 7fe74b400700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe73406adb0 0x7fe73406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:57.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.348+0000 7fe74b400700 1 -- 172.21.15.12:0/159226799 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe740088b10 con 0x7fe7540d2540 2024-03-29T10:10:57.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.348+0000 7fe753400700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe73406adb0 0x7fe73406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:57.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.349+0000 7fe753400700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe73406adb0 0x7fe73406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe7440011e0 tx=0x7fe744009040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:57.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.352+0000 7fe74b400700 1 -- 172.21.15.12:0/159226799 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe740053e30 con 0x7fe7540d2540 2024-03-29T10:10:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:57.564+0000 7fe75a000700 1 -- 172.21.15.12:0/159226799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe754067720 con 0x7fe7540d2540 2024-03-29T10:10:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.109+0000 7fe74b400700 1 -- 172.21.15.12:0/159226799 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v106) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe740017020 con 0x7fe7540d2540 2024-03-29T10:10:58.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.116+0000 7fe75a000700 1 -- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe73406adb0 msgr2=0x7fe73406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.116+0000 7fe75a000700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe73406adb0 0x7fe73406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe7440011e0 tx=0x7fe744009040 comp rx=0 tx=0).stop 2024-03-29T10:10:58.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.116+0000 7fe75a000700 1 -- 172.21.15.12:0/159226799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7540d2540 msgr2=0x7fe75415c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.116+0000 7fe75a000700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7540d2540 0x7fe75415c020 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe740001260 tx=0x7fe74000c9e0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.116+0000 7fe75a000700 1 -- 172.21.15.12:0/159226799 shutdown_connections 2024-03-29T10:10:58.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.116+0000 7fe75a000700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe73406adb0 0x7fe73406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.116+0000 7fe75a000700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe75415d9d0 0x7fe75415f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.116+0000 7fe75a000700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7540d16a0 0x7fe75415c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.116+0000 7fe75a000700 1 --2- 172.21.15.12:0/159226799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7540d2540 0x7fe75415c020 secure :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe740001260 tx=0x7fe74000c9e0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.116+0000 7fe75a000700 1 -- 172.21.15.12:0/159226799 >> 172.21.15.12:0/159226799 conn(0x7fe75408aff0 msgr2=0x7fe7540b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:58.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.117+0000 7fe75a000700 1 -- 172.21.15.12:0/159226799 shutdown_connections 2024-03-29T10:10:58.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.117+0000 7fe75a000700 1 -- 172.21.15.12:0/159226799 wait complete. 2024-03-29T10:10:58.125 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T10:10:58.145 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T10:10:58.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.353+0000 7f07dd600700 1 Processor -- start 2024-03-29T10:10:58.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.353+0000 7f07dd600700 1 -- start start 2024-03-29T10:10:58.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07dd600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07d80cd020 0x7f07d80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07dd600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07d80b2070 0x7f07d80b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07dd600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d80cdae0 con 0x7f07d80cbb30 2024-03-29T10:10:58.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07dd600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d80cddc0 con 0x7f07d80cd020 2024-03-29T10:10:58.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07dd600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80ce0a0 con 0x7f07d80b2070 2024-03-29T10:10:58.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07d6000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07d80cd020 0x7f07d80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07d6000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07d80cd020 0x7f07d80cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37028/0 (socket says 172.21.15.12:37028) 2024-03-29T10:10:58.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07d6000700 1 -- 172.21.15.12:0/2173865090 learned_addr learned my addr 172.21.15.12:0/2173865090 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:58.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07d7400700 1 --2- 172.21.15.12:0/2173865090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07d80b2070 0x7f07d80b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07d6000700 1 -- 172.21.15.12:0/2173865090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07d80b2070 msgr2=0x7f07d80b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:58.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07d6000700 1 --2- 172.21.15.12:0/2173865090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07d80b2070 0x7f07d80b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07d6000700 1 -- 172.21.15.12:0/2173865090 >> v1:172.21.15.159:6789/0 conn(0x7f07d80cbb30 legacy=0x7f07d80cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.355+0000 7f07d6000700 1 -- 172.21.15.12:0/2173865090 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d806dc20 con 0x7f07d80cd020 2024-03-29T10:10:58.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.356+0000 7f07d6000700 1 --2- 172.21.15.12:0/2173865090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cd020 0x7f07d80cd480 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f07c40098f0 tx=0x7f07c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f65c2f2ce2cd78d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:58.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.356+0000 7f07d5600700 1 -- 172.21.15.12:0/2173865090 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4011830 con 0x7f07d80cd020 2024-03-29T10:10:58.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.356+0000 7f07d5600700 1 -- 172.21.15.12:0/2173865090 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c4011990 con 0x7f07d80cd020 2024-03-29T10:10:58.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.356+0000 7f07d5600700 1 -- 172.21.15.12:0/2173865090 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4011d20 con 0x7f07d80cd020 2024-03-29T10:10:58.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.356+0000 7f07dd600700 1 -- 172.21.15.12:0/2173865090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cd020 msgr2=0x7f07d80cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.356+0000 7f07dd600700 1 --2- 172.21.15.12:0/2173865090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cd020 0x7f07d80cd480 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f07c40098f0 tx=0x7f07c4009c00 comp rx=0 tx=0).stop 2024-03-29T10:10:58.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.357+0000 7f07dd600700 1 -- 172.21.15.12:0/2173865090 shutdown_connections 2024-03-29T10:10:58.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.357+0000 7f07dd600700 1 --2- 172.21.15.12:0/2173865090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07d80b2070 0x7f07d80b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.357+0000 7f07dd600700 1 --2- 172.21.15.12:0/2173865090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cd020 0x7f07d80cd480 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.357+0000 7f07dd600700 1 -- 172.21.15.12:0/2173865090 >> 172.21.15.12:0/2173865090 conn(0x7f07d80b0d70 msgr2=0x7f07d80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:58.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.357+0000 7f07dd600700 1 -- 172.21.15.12:0/2173865090 shutdown_connections 2024-03-29T10:10:58.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.357+0000 7f07dd600700 1 -- 172.21.15.12:0/2173865090 wait complete. 2024-03-29T10:10:58.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.358+0000 7f07dd600700 1 Processor -- start 2024-03-29T10:10:58.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.359+0000 7f07dd600700 1 -- start start 2024-03-29T10:10:58.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.359+0000 7f07dd600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07d80b2070 0x7f07d8150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.359+0000 7f07dd600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cbb30 0x7f07d81f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.359+0000 7f07dd600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07d80cd020 0x7f07d8151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.359+0000 7f07dd600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8151160 con 0x7f07d80cbb30 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.359+0000 7f07dd600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8159c90 con 0x7f07d80cd020 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.359+0000 7f07dd600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8159f30 con 0x7f07d80b2070 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cbb30 0x7f07d81f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cbb30 0x7f07d81f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37038/0 (socket says 172.21.15.12:37038) 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d6000700 1 -- 172.21.15.12:0/1386451953 learned_addr learned my addr 172.21.15.12:0/1386451953 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d7400700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07d80cd020 0x7f07d8151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d6a00700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07d80b2070 0x7f07d8150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d6000700 1 -- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07d80b2070 msgr2=0x7f07d8150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d6000700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07d80b2070 0x7f07d8150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d6000700 1 -- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07d80cd020 msgr2=0x7f07d8151360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d6000700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07d80cd020 0x7f07d8151360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d6000700 1 -- 172.21.15.12:0/1386451953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4009550 con 0x7f07d80cbb30 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.360+0000 7f07d6000700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cbb30 0x7f07d81f3200 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f07c4000c00 tx=0x7f07c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:58.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.361+0000 7f07cea00700 1 -- 172.21.15.12:0/1386451953 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4004530 con 0x7f07d80cbb30 2024-03-29T10:10:58.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.361+0000 7f07cea00700 1 -- 172.21.15.12:0/1386451953 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c401f070 con 0x7f07d80cbb30 2024-03-29T10:10:58.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.361+0000 7f07cea00700 1 -- 172.21.15.12:0/1386451953 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40038e0 con 0x7f07d80cbb30 2024-03-29T10:10:58.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.361+0000 7f07dd600700 1 -- 172.21.15.12:0/1386451953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d815a210 con 0x7f07d80cbb30 2024-03-29T10:10:58.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.361+0000 7f07dd600700 1 -- 172.21.15.12:0/1386451953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d815a7d0 con 0x7f07d80cbb30 2024-03-29T10:10:58.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.362+0000 7f07dd600700 1 -- 172.21.15.12:0/1386451953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d8067720 con 0x7f07d80cbb30 2024-03-29T10:10:58.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.363+0000 7f07cea00700 1 -- 172.21.15.12:0/1386451953 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f07c4021030 con 0x7f07d80cbb30 2024-03-29T10:10:58.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.364+0000 7f07cea00700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07b806adb0 0x7f07b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.364+0000 7f07cea00700 1 -- 172.21.15.12:0/1386451953 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f07c408a8d0 con 0x7f07d80cbb30 2024-03-29T10:10:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.364+0000 7f07d6a00700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07b806adb0 0x7f07b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.365+0000 7f07d6a00700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07b806adb0 0x7f07b806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f07c8000f60 tx=0x7f07c8009040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:58.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.369+0000 7f07cea00700 1 -- 172.21.15.12:0/1386451953 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f07c40559f0 con 0x7f07d80cbb30 2024-03-29T10:10:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.632+0000 7f07dd600700 1 -- 172.21.15.12:0/1386451953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f07d80ce2e0 con 0x7f07d80cbb30 2024-03-29T10:10:58.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.641+0000 7f07cea00700 1 -- 172.21.15.12:0/1386451953 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v42) v1 ==== 174+0+0 (secure 0 0 0) 0x7f07c40150c0 con 0x7f07d80cbb30 2024-03-29T10:10:58.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.649+0000 7f07dd600700 1 -- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07b806adb0 msgr2=0x7f07b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.649+0000 7f07dd600700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07b806adb0 0x7f07b806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f07c8000f60 tx=0x7f07c8009040 comp rx=0 tx=0).stop 2024-03-29T10:10:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.649+0000 7f07dd600700 1 -- 172.21.15.12:0/1386451953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cbb30 msgr2=0x7f07d81f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.649+0000 7f07dd600700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cbb30 0x7f07d81f3200 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f07c4000c00 tx=0x7f07c4004750 comp rx=0 tx=0).stop 2024-03-29T10:10:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.649+0000 7f07dd600700 1 -- 172.21.15.12:0/1386451953 shutdown_connections 2024-03-29T10:10:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.649+0000 7f07dd600700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07b806adb0 0x7f07b806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.649+0000 7f07dd600700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07d80cd020 0x7f07d8151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.649+0000 7f07dd600700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07d80b2070 0x7f07d8150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.649+0000 7f07dd600700 1 --2- 172.21.15.12:0/1386451953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07d80cbb30 0x7f07d81f3200 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.649+0000 7f07dd600700 1 -- 172.21.15.12:0/1386451953 >> 172.21.15.12:0/1386451953 conn(0x7f07d80b0d70 msgr2=0x7f07d80cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.650+0000 7f07dd600700 1 -- 172.21.15.12:0/1386451953 shutdown_connections 2024-03-29T10:10:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.650+0000 7f07dd600700 1 -- 172.21.15.12:0/1386451953 wait complete. 2024-03-29T10:10:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 12 and data pool 13 2024-03-29T10:10:58.683 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.881+0000 7f6aa5e00700 1 Processor -- start 2024-03-29T10:10:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.882+0000 7f6aa5e00700 1 -- start start 2024-03-29T10:10:58.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6aa5e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6aa0160f10 0x7f6aa00d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6aa5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6aa0161590 0x7f6aa0161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6aa5e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa00ceda0 con 0x7f6aa00cf230 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6aa5e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa0161f80 con 0x7f6aa0161590 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6aa5e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa01612f0 con 0x7f6aa0160f10 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6a9f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6aa0161590 0x7f6aa0161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6a9e000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6aa00cf230 0x7f6aa00d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60764/0 (socket says 172.21.15.12:60764) 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6a9f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6aa0161590 0x7f6aa0161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39366/0 (socket says 172.21.15.12:39366) 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6a9e000700 1 -- 172.21.15.12:0/2142533842 learned_addr learned my addr 172.21.15.12:0/2142533842 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6a9ea00700 1 --2- 172.21.15.12:0/2142533842 >> v2:172.21.15.159:3300/0 conn(0x7f6aa0160f10 0x7f6aa00d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.883+0000 7f6a9f400700 1 -- 172.21.15.12:0/2142533842 >> v2:172.21.15.159:3300/0 conn(0x7f6aa0160f10 msgr2=0x7f6aa00d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.884+0000 7f6a9f400700 1 --2- 172.21.15.12:0/2142533842 >> v2:172.21.15.159:3300/0 conn(0x7f6aa0160f10 0x7f6aa00d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.884+0000 7f6a9f400700 1 -- 172.21.15.12:0/2142533842 >> v1:172.21.15.12:6789/0 conn(0x7f6aa00cf230 legacy=0x7f6aa00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.884+0000 7f6a9f400700 1 -- 172.21.15.12:0/2142533842 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa00ce470 con 0x7f6aa0161590 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.884+0000 7f6a9d600700 1 -- 172.21.15.12:0/2142533842 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406484852 0 0) 0x7f6aa00ceda0 con 0x7f6aa00cf230 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.884+0000 7f6a9f400700 1 --2- 172.21.15.12:0/2142533842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 0x7f6aa0161a40 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00d1080 tx=0x7f6a9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3e34ab6af05f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.884+0000 7f6a9d600700 1 -- 172.21.15.12:0/2142533842 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9800e070 con 0x7f6aa0161590 2024-03-29T10:10:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.884+0000 7f6a9d600700 1 -- 172.21.15.12:0/2142533842 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9800bbd0 con 0x7f6aa0161590 2024-03-29T10:10:58.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.884+0000 7f6a9d600700 1 -- 172.21.15.12:0/2142533842 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a98012430 con 0x7f6aa0161590 2024-03-29T10:10:58.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.885+0000 7f6aa5e00700 1 -- 172.21.15.12:0/2142533842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 msgr2=0x7f6aa0161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.885+0000 7f6aa5e00700 1 --2- 172.21.15.12:0/2142533842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 0x7f6aa0161a40 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00d1080 tx=0x7f6a9800b330 comp rx=0 tx=0).stop 2024-03-29T10:10:58.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.885+0000 7f6aa5e00700 1 -- 172.21.15.12:0/2142533842 shutdown_connections 2024-03-29T10:10:58.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.885+0000 7f6aa5e00700 1 --2- 172.21.15.12:0/2142533842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 0x7f6aa0161a40 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.885+0000 7f6aa5e00700 1 --2- 172.21.15.12:0/2142533842 >> v2:172.21.15.159:3300/0 conn(0x7f6aa0160f10 0x7f6aa00d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.885+0000 7f6aa5e00700 1 -- 172.21.15.12:0/2142533842 >> 172.21.15.12:0/2142533842 conn(0x7f6aa008aff0 msgr2=0x7f6aa00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:58.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.885+0000 7f6aa5e00700 1 -- 172.21.15.12:0/2142533842 shutdown_connections 2024-03-29T10:10:58.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.885+0000 7f6aa5e00700 1 -- 172.21.15.12:0/2142533842 wait complete. 2024-03-29T10:10:58.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.887+0000 7f6aa5e00700 1 Processor -- start 2024-03-29T10:10:58.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.887+0000 7f6aa5e00700 1 -- start start 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6aa5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6aa00cf230 0x7f6aa015f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6aa5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6aa0160f10 0x7f6aa01e7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6aa5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 0x7f6aa01e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6aa5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa01f3440 con 0x7f6aa0161590 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6aa5e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa01593c0 con 0x7f6aa0160f10 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6aa5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa00b6120 con 0x7f6aa00cf230 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6a9f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 0x7f6aa01e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6a9f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 0x7f6aa01e7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39376/0 (socket says 172.21.15.12:39376) 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6a9f400700 1 -- 172.21.15.12:0/3145076734 learned_addr learned my addr 172.21.15.12:0/3145076734 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6a9e000700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6aa0160f10 0x7f6aa01e7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6a9ea00700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6aa00cf230 0x7f6aa015f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6a9f400700 1 -- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6aa00cf230 msgr2=0x7f6aa015f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6a9f400700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6aa00cf230 0x7f6aa015f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6a9f400700 1 -- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6aa0160f10 msgr2=0x7f6aa01e7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6a9f400700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6aa0160f10 0x7f6aa01e7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.888+0000 7f6a9f400700 1 -- 172.21.15.12:0/3145076734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a98009a70 con 0x7f6aa0161590 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.889+0000 7f6a9f400700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 0x7f6aa01e7a40 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f6aa008c570 tx=0x7f6a98004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.889+0000 7f6a97400700 1 -- 172.21.15.12:0/3145076734 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a98012430 con 0x7f6aa0161590 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.889+0000 7f6a97400700 1 -- 172.21.15.12:0/3145076734 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a98020070 con 0x7f6aa0161590 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.889+0000 7f6a97400700 1 -- 172.21.15.12:0/3145076734 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a98004280 con 0x7f6aa0161590 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.889+0000 7f6aa5e00700 1 -- 172.21.15.12:0/3145076734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa015f9c0 con 0x7f6aa0161590 2024-03-29T10:10:58.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.889+0000 7f6aa5e00700 1 -- 172.21.15.12:0/3145076734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa015ff80 con 0x7f6aa0161590 2024-03-29T10:10:58.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.890+0000 7f6aa5e00700 1 -- 172.21.15.12:0/3145076734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa00b4e20 con 0x7f6aa0161590 2024-03-29T10:10:58.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.891+0000 7f6a97400700 1 -- 172.21.15.12:0/3145076734 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6a980043e0 con 0x7f6aa0161590 2024-03-29T10:10:58.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.892+0000 7f6a97400700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a8006adb0 0x7f6a8006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:58.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.892+0000 7f6a97400700 1 -- 172.21.15.12:0/3145076734 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a98022070 con 0x7f6aa0161590 2024-03-29T10:10:58.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.893+0000 7f6a9ea00700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a8006adb0 0x7f6a8006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:58.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.893+0000 7f6a9ea00700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a8006adb0 0x7f6a8006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a90006b20 tx=0x7f6a90008040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:58.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:58.897+0000 7f6a97400700 1 -- 172.21.15.12:0/3145076734 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a980565d0 con 0x7f6aa0161590 2024-03-29T10:10:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.115+0000 7f6aa5e00700 1 -- 172.21.15.12:0/3145076734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6aa0067720 con 0x7f6aa0161590 2024-03-29T10:10:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.116+0000 7f6a97400700 1 -- 172.21.15.12:0/3145076734 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16116 (secure 0 0 0) 0x7f6a98059bf0 con 0x7f6aa0161590 2024-03-29T10:10:59.116 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:59.116 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":107,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:58.633870+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:57.064406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:57.607254+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 -- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a8006adb0 msgr2=0x7f6a8006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a8006adb0 0x7f6a8006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a90006b20 tx=0x7f6a90008040 comp rx=0 tx=0).stop 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 -- 172.21.15.12:0/3145076734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 msgr2=0x7f6aa01e7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 0x7f6aa01e7a40 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f6aa008c570 tx=0x7f6a98004ad0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 -- 172.21.15.12:0/3145076734 shutdown_connections 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a8006adb0 0x7f6a8006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aa0161590 0x7f6aa01e7a40 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6aa00cf230 0x7f6aa015f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 --2- 172.21.15.12:0/3145076734 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6aa0160f10 0x7f6aa01e7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 -- 172.21.15.12:0/3145076734 >> 172.21.15.12:0/3145076734 conn(0x7f6aa008aff0 msgr2=0x7f6aa00ca930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 -- 172.21.15.12:0/3145076734 shutdown_connections 2024-03-29T10:10:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.119+0000 7f6aa5e00700 1 -- 172.21.15.12:0/3145076734 wait complete. 2024-03-29T10:10:59.138 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:10:59.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.337+0000 7f41d2c00700 1 Processor -- start 2024-03-29T10:10:59.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.337+0000 7f41d2c00700 1 -- start start 2024-03-29T10:10:59.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.338+0000 7f41d2c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41cc161ca0 0x7f41cc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.338+0000 7f41d2c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41cc0ceda0 con 0x7f41cc160f10 2024-03-29T10:10:59.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.338+0000 7f41d2c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41cc1612f0 con 0x7f41cc0d18b0 2024-03-29T10:10:59.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.338+0000 7f41d2c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41cc0d1d30 con 0x7f41cc161ca0 2024-03-29T10:10:59.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.339+0000 7f41d0c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41cc161ca0 0x7f41cc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.339+0000 7f41d0c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41cc161ca0 0x7f41cc0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39386/0 (socket says 172.21.15.12:39386) 2024-03-29T10:10:59.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.339+0000 7f41cbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f41cc160f10 0x7f41cc0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60772/0 (socket says 172.21.15.12:60772) 2024-03-29T10:10:59.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.339+0000 7f41d0c00700 1 -- 172.21.15.12:0/760088549 learned_addr learned my addr 172.21.15.12:0/760088549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:59.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.339+0000 7f41d0c00700 1 -- 172.21.15.12:0/760088549 >> v1:172.21.15.159:6789/0 conn(0x7f41cc0d18b0 legacy=0x7f41cc161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.339+0000 7f41d0c00700 1 -- 172.21.15.12:0/760088549 >> v1:172.21.15.12:6789/0 conn(0x7f41cc160f10 legacy=0x7f41cc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.339+0000 7f41d0c00700 1 -- 172.21.15.12:0/760088549 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc0ce470 con 0x7f41cc161ca0 2024-03-29T10:10:59.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.339+0000 7f41d0c00700 1 --2- 172.21.15.12:0/760088549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 0x7f41cc0cf230 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009e50 tx=0x7f41c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410d4cc59c228aea server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:59.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.340+0000 7f41caa00700 1 -- 172.21.15.12:0/760088549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c400e070 con 0x7f41cc161ca0 2024-03-29T10:10:59.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.340+0000 7f41caa00700 1 -- 172.21.15.12:0/760088549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c400b9c0 con 0x7f41cc161ca0 2024-03-29T10:10:59.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.340+0000 7f41caa00700 1 -- 172.21.15.12:0/760088549 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c400bd70 con 0x7f41cc161ca0 2024-03-29T10:10:59.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.340+0000 7f41d2c00700 1 -- 172.21.15.12:0/760088549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 msgr2=0x7f41cc0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.340+0000 7f41d2c00700 1 --2- 172.21.15.12:0/760088549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 0x7f41cc0cf230 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009e50 tx=0x7f41c400b330 comp rx=0 tx=0).stop 2024-03-29T10:10:59.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.340+0000 7f41d2c00700 1 -- 172.21.15.12:0/760088549 shutdown_connections 2024-03-29T10:10:59.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.340+0000 7f41d2c00700 1 --2- 172.21.15.12:0/760088549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 0x7f41cc0cf230 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.340+0000 7f41d2c00700 1 -- 172.21.15.12:0/760088549 >> 172.21.15.12:0/760088549 conn(0x7f41cc08aff0 msgr2=0x7f41cc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:59.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.340+0000 7f41d2c00700 1 -- 172.21.15.12:0/760088549 shutdown_connections 2024-03-29T10:10:59.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.340+0000 7f41d2c00700 1 -- 172.21.15.12:0/760088549 wait complete. 2024-03-29T10:10:59.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.342+0000 7f41d2c00700 1 Processor -- start 2024-03-29T10:10:59.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.342+0000 7f41d2c00700 1 -- start start 2024-03-29T10:10:59.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.343+0000 7f41d2c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41cc0d18b0 0x7f41cc15c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.343+0000 7f41d2c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41cc160f10 0x7f41cc1e7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.343+0000 7f41d2c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 0x7f41cc1e8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.343+0000 7f41d2c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc1f4060 con 0x7f41cc161ca0 2024-03-29T10:10:59.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.343+0000 7f41d2c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc1f3450 con 0x7f41cc160f10 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.343+0000 7f41d2c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc1f3730 con 0x7f41cc0d18b0 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.343+0000 7f41d0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 0x7f41cc1e8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.343+0000 7f41d0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 0x7f41cc1e8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39400/0 (socket says 172.21.15.12:39400) 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.343+0000 7f41d0c00700 1 -- 172.21.15.12:0/972143644 learned_addr learned my addr 172.21.15.12:0/972143644 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41cbe00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41cc0d18b0 0x7f41cc15c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41cb400700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41cc160f10 0x7f41cc1e7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41d0c00700 1 -- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41cc0d18b0 msgr2=0x7f41cc15c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41d0c00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41cc0d18b0 0x7f41cc15c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41d0c00700 1 -- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41cc160f10 msgr2=0x7f41cc1e7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41d0c00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41cc160f10 0x7f41cc1e7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41d0c00700 1 -- 172.21.15.12:0/972143644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c4009b30 con 0x7f41cc161ca0 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41cbe00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41cc0d18b0 0x7f41cc15c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41cb400700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41cc160f10 0x7f41cc1e7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41d0c00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 0x7f41cc1e8320 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f41cc08c570 tx=0x7f41c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41c3e00700 1 -- 172.21.15.12:0/972143644 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4004960 con 0x7f41cc161ca0 2024-03-29T10:10:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41c3e00700 1 -- 172.21.15.12:0/972143644 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c40099b0 con 0x7f41cc161ca0 2024-03-29T10:10:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41c3e00700 1 -- 172.21.15.12:0/972143644 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c401b670 con 0x7f41cc161ca0 2024-03-29T10:10:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.344+0000 7f41d2c00700 1 -- 172.21.15.12:0/972143644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41cc1f3a10 con 0x7f41cc161ca0 2024-03-29T10:10:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.345+0000 7f41d2c00700 1 -- 172.21.15.12:0/972143644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41cc15cc70 con 0x7f41cc161ca0 2024-03-29T10:10:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.346+0000 7f41d2c00700 1 -- 172.21.15.12:0/972143644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41cc15ce30 con 0x7f41cc161ca0 2024-03-29T10:10:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.347+0000 7f41c3e00700 1 -- 172.21.15.12:0/972143644 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f41c401b7d0 con 0x7f41cc161ca0 2024-03-29T10:10:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.347+0000 7f41c3e00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41ac06adb0 0x7f41ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.347+0000 7f41c3e00700 1 -- 172.21.15.12:0/972143644 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41c408b060 con 0x7f41cc161ca0 2024-03-29T10:10:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.348+0000 7f41cbe00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41ac06adb0 0x7f41ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.349+0000 7f41cbe00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41ac06adb0 0x7f41ac06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f41bc000f60 tx=0x7f41bc007df0 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.352+0000 7f41c3e00700 1 -- 172.21.15.12:0/972143644 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f41c40be580 con 0x7f41cc161ca0 2024-03-29T10:10:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.564+0000 7f41d2c00700 1 -- 172.21.15.12:0/972143644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41cc067720 con 0x7f41cc161ca0 2024-03-29T10:10:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.566+0000 7f41c3e00700 1 -- 172.21.15.12:0/972143644 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16116 (secure 0 0 0) 0x7f41c4059aa0 con 0x7f41cc161ca0 2024-03-29T10:10:59.566 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:10:59.566 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":107,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:58.633870+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:57.064406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:57.607254+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:10:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 -- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41ac06adb0 msgr2=0x7f41ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41ac06adb0 0x7f41ac06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f41bc000f60 tx=0x7f41bc007df0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 -- 172.21.15.12:0/972143644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 msgr2=0x7f41cc1e8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 0x7f41cc1e8320 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f41cc08c570 tx=0x7f41c4004ad0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 -- 172.21.15.12:0/972143644 shutdown_connections 2024-03-29T10:10:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41ac06adb0 0x7f41ac06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41cc161ca0 0x7f41cc1e8320 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41cc0d18b0 0x7f41cc15c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 --2- 172.21.15.12:0/972143644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41cc160f10 0x7f41cc1e7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 -- 172.21.15.12:0/972143644 >> 172.21.15.12:0/972143644 conn(0x7f41cc08aff0 msgr2=0x7f41cc0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 -- 172.21.15.12:0/972143644 shutdown_connections 2024-03-29T10:10:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.568+0000 7f41d2c00700 1 -- 172.21.15.12:0/972143644 wait complete. 2024-03-29T10:10:59.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T10:10:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.789+0000 7f603f400700 1 Processor -- start 2024-03-29T10:10:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.789+0000 7f603f400700 1 -- start start 2024-03-29T10:10:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.790+0000 7f603f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60380ce840 0x7f6038165370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.790+0000 7f603f400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60380cb310 con 0x7f603815a5c0 2024-03-29T10:10:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.790+0000 7f603f400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60380cca90 con 0x7f603815dfe0 2024-03-29T10:10:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.790+0000 7f603f400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60380711a0 con 0x7f60380ce840 2024-03-29T10:10:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.790+0000 7f603ca00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f603815a5c0 0x7f60380709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60776/0 (socket says 172.21.15.12:60776) 2024-03-29T10:10:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f603ca00700 1 -- 172.21.15.12:0/1475043991 learned_addr learned my addr 172.21.15.12:0/1475043991 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:10:59.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f603d400700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60380ce840 0x7f6038165370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366230883 0 0) 0x7f60380cb310 con 0x7f603815a5c0 2024-03-29T10:10:59.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60200035b0 con 0x7f603815a5c0 2024-03-29T10:10:59.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116113067 0 0) 0x7f60200035b0 con 0x7f603815a5c0 2024-03-29T10:10:59.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60380ce840 msgr2=0x7f6038165370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:59.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60380ce840 0x7f6038165370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 >> v1:172.21.15.159:6789/0 conn(0x7f603815dfe0 legacy=0x7f60380ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6038071480 con 0x7f603815a5c0 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f602c001670 con 0x7f603815a5c0 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 >> v1:172.21.15.12:6789/0 conn(0x7f603815a5c0 legacy=0x7f60380709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60380ce840 0x7f6020003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6020003f70 0x7f6020004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 0x7f6020005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380711a0 con 0x7f6020004d90 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602000dc80 con 0x7f6020003f70 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.791+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602000de40 con 0x7f60380ce840 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.792+0000 7f603d400700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 0x7f6020005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.792+0000 7f603ca00700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60380ce840 0x7f6020003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.792+0000 7f6037e00700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6020003f70 0x7f6020004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.792+0000 7f603d400700 1 -- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60380ce840 msgr2=0x7f6020003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.792+0000 7f603d400700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60380ce840 0x7f6020003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.792+0000 7f603d400700 1 -- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6020003f70 msgr2=0x7f6020004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.792+0000 7f603d400700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6020003f70 0x7f6020004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.792+0000 7f603d400700 1 -- 172.21.15.12:0/1475043991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60200035b0 con 0x7f6020004d90 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.792+0000 7f6037e00700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6020003f70 0x7f6020004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.792+0000 7f603d400700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 0x7f6020005210 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f6030009fc0 tx=0x7f6020005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409cfed883cbbcf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.793+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603000ec30 con 0x7f6020004d90 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.793+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603000f040 con 0x7f6020004d90 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.793+0000 7f6037400700 1 -- 172.21.15.12:0/1475043991 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030014480 con 0x7f6020004d90 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.793+0000 7f603f400700 1 -- 172.21.15.12:0/1475043991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 msgr2=0x7f6020005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.793+0000 7f603f400700 1 --2- 172.21.15.12:0/1475043991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 0x7f6020005210 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f6030009fc0 tx=0x7f6020005880 comp rx=0 tx=0).stop 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.793+0000 7f603d400700 1 -- 172.21.15.12:0/1475043991 reap_dead start 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.793+0000 7f603f400700 1 -- 172.21.15.12:0/1475043991 shutdown_connections 2024-03-29T10:10:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.793+0000 7f603f400700 1 -- 172.21.15.12:0/1475043991 >> 172.21.15.12:0/1475043991 conn(0x7f60380b0d70 msgr2=0x7f60380b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:10:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.793+0000 7f603f400700 1 -- 172.21.15.12:0/1475043991 shutdown_connections 2024-03-29T10:10:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.793+0000 7f603f400700 1 -- 172.21.15.12:0/1475043991 wait complete. 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.795+0000 7f603f400700 1 Processor -- start 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.795+0000 7f603f400700 1 -- start start 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.796+0000 7f603f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 0x7f60381f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.796+0000 7f603f400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60380acba0 0x7f60380ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.796+0000 7f603f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60381554c0 0x7f6038155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.796+0000 7f603f400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6038155e80 con 0x7f6020004d90 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.796+0000 7f603f400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380aee10 con 0x7f60380acba0 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.796+0000 7f603f400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60380ad660 con 0x7f60381554c0 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.796+0000 7f603ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 0x7f60381f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.796+0000 7f603ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 0x7f60381f2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39418/0 (socket says 172.21.15.12:39418) 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f603ca00700 1 -- 172.21.15.12:0/3465444537 learned_addr learned my addr 172.21.15.12:0/3465444537 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f6037e00700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60380acba0 0x7f60380ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f603d400700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60381554c0 0x7f6038155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f603ca00700 1 -- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60381554c0 msgr2=0x7f6038155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f603ca00700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60381554c0 0x7f6038155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f603ca00700 1 -- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60380acba0 msgr2=0x7f60380ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f603ca00700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60380acba0 0x7f60380ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f603ca00700 1 -- 172.21.15.12:0/3465444537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60380af0b0 con 0x7f6020004d90 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f603ca00700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 0x7f60381f2f40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f602c0072c0 tx=0x7f602c0075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f6034c00700 1 -- 172.21.15.12:0/3465444537 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c0014e0 con 0x7f6020004d90 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f6034c00700 1 -- 172.21.15.12:0/3465444537 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c00f680 con 0x7f6020004d90 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.797+0000 7f603f400700 1 -- 172.21.15.12:0/3465444537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60380af390 con 0x7f6020004d90 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.798+0000 7f6034c00700 1 -- 172.21.15.12:0/3465444537 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c00fa50 con 0x7f6020004d90 2024-03-29T10:10:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.798+0000 7f603f400700 1 -- 172.21.15.12:0/3465444537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60380ac2b0 con 0x7f6020004d90 2024-03-29T10:10:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.799+0000 7f603f400700 1 -- 172.21.15.12:0/3465444537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6038067720 con 0x7f6020004d90 2024-03-29T10:10:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.800+0000 7f6034c00700 1 -- 172.21.15.12:0/3465444537 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f602c00fbb0 con 0x7f6020004d90 2024-03-29T10:10:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.800+0000 7f6034c00700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f601806adb0 0x7f601806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:10:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.801+0000 7f6034c00700 1 -- 172.21.15.12:0/3465444537 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f602c089430 con 0x7f6020004d90 2024-03-29T10:10:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.801+0000 7f6037e00700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f601806adb0 0x7f601806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:10:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.801+0000 7f6037e00700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f601806adb0 0x7f601806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f60380ce3f0 tx=0x7f6024009040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:10:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:10:59.805+0000 7f6034c00700 1 -- 172.21.15.12:0/3465444537 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f602c057f80 con 0x7f6020004d90 2024-03-29T10:11:00.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.066+0000 7f603f400700 1 -- 172.21.15.12:0/3465444537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f603806c460 con 0x7f6020004d90 2024-03-29T10:11:00.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.645+0000 7f6034c00700 1 -- 172.21.15.12:0/3465444537 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v45) v1 ==== 118+0+0 (secure 0 0 0) 0x7f602c057b10 con 0x7f6020004d90 2024-03-29T10:11:00.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 -- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f601806adb0 msgr2=0x7f601806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:00.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f601806adb0 0x7f601806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f60380ce3f0 tx=0x7f6024009040 comp rx=0 tx=0).stop 2024-03-29T10:11:00.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 -- 172.21.15.12:0/3465444537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 msgr2=0x7f60381f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:00.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 0x7f60381f2f40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f602c0072c0 tx=0x7f602c0075d0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 -- 172.21.15.12:0/3465444537 shutdown_connections 2024-03-29T10:11:00.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f601806adb0 0x7f601806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60381554c0 0x7f6038155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6020004d90 0x7f60381f2f40 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 --2- 172.21.15.12:0/3465444537 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60380acba0 0x7f60380ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 -- 172.21.15.12:0/3465444537 >> 172.21.15.12:0/3465444537 conn(0x7f60380b0d70 msgr2=0x7f60380cd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:00.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.648+0000 7f603f400700 1 -- 172.21.15.12:0/3465444537 shutdown_connections 2024-03-29T10:11:00.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.649+0000 7f603f400700 1 -- 172.21.15.12:0/3465444537 wait complete. 2024-03-29T10:11:00.670 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T10:11:00.670 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T10:11:00.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.875+0000 7f15baa00700 1 Processor -- start 2024-03-29T10:11:00.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.875+0000 7f15baa00700 1 -- start start 2024-03-29T10:11:00.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.876+0000 7f15baa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f15b4160d70 0x7f15b40b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:00.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.876+0000 7f15baa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 0x7f15b415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:00.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.876+0000 7f15baa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15b406a790 0x7f15b406ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:00.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.876+0000 7f15baa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b406b180 con 0x7f15b406a790 2024-03-29T10:11:00.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.876+0000 7f15baa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b40b4cd0 con 0x7f15b4160d70 2024-03-29T10:11:00.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.876+0000 7f15baa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b4161150 con 0x7f15b415ef70 2024-03-29T10:11:00.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15b406a790 0x7f15b406ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:00.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15b406a790 0x7f15b406ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39424/0 (socket says 172.21.15.12:39424) 2024-03-29T10:11:00.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b8a00700 1 -- 172.21.15.12:0/2795927345 learned_addr learned my addr 172.21.15.12:0/2795927345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b3e00700 1 --2- 172.21.15.12:0/2795927345 >> v2:172.21.15.159:3300/0 conn(0x7f15b4160d70 0x7f15b40b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b3400700 1 --2- 172.21.15.12:0/2795927345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 0x7f15b415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b8a00700 1 -- 172.21.15.12:0/2795927345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 msgr2=0x7f15b415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b8a00700 1 --2- 172.21.15.12:0/2795927345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 0x7f15b415e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b8a00700 1 -- 172.21.15.12:0/2795927345 >> v2:172.21.15.159:3300/0 conn(0x7f15b4160d70 msgr2=0x7f15b40b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b8a00700 1 --2- 172.21.15.12:0/2795927345 >> v2:172.21.15.159:3300/0 conn(0x7f15b4160d70 0x7f15b40b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b8a00700 1 -- 172.21.15.12:0/2795927345 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b406dba0 con 0x7f15b406a790 2024-03-29T10:11:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.877+0000 7f15b3400700 1 --2- 172.21.15.12:0/2795927345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 0x7f15b415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:11:00.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15b8a00700 1 --2- 172.21.15.12:0/2795927345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 0x7f15b406ac40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f15b406c880 tx=0x7f15ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d656b7805642b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:00.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15b2a00700 1 -- 172.21.15.12:0/2795927345 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac00e070 con 0x7f15b406a790 2024-03-29T10:11:00.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15b2a00700 1 -- 172.21.15.12:0/2795927345 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac00b910 con 0x7f15b406a790 2024-03-29T10:11:00.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15b2a00700 1 -- 172.21.15.12:0/2795927345 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac00bcc0 con 0x7f15b406a790 2024-03-29T10:11:00.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15baa00700 1 -- 172.21.15.12:0/2795927345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 msgr2=0x7f15b406ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:00.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15baa00700 1 --2- 172.21.15.12:0/2795927345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 0x7f15b406ac40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f15b406c880 tx=0x7f15ac00b330 comp rx=0 tx=0).stop 2024-03-29T10:11:00.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15baa00700 1 -- 172.21.15.12:0/2795927345 shutdown_connections 2024-03-29T10:11:00.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15baa00700 1 --2- 172.21.15.12:0/2795927345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 0x7f15b406ac40 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15baa00700 1 --2- 172.21.15.12:0/2795927345 >> v2:172.21.15.159:3300/0 conn(0x7f15b4160d70 0x7f15b40b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15baa00700 1 --2- 172.21.15.12:0/2795927345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 0x7f15b415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.878+0000 7f15baa00700 1 -- 172.21.15.12:0/2795927345 >> 172.21.15.12:0/2795927345 conn(0x7f15b40b5720 msgr2=0x7f15b40b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.879+0000 7f15baa00700 1 -- 172.21.15.12:0/2795927345 shutdown_connections 2024-03-29T10:11:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.879+0000 7f15baa00700 1 -- 172.21.15.12:0/2795927345 wait complete. 2024-03-29T10:11:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.880+0000 7f15baa00700 1 Processor -- start 2024-03-29T10:11:00.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.880+0000 7f15baa00700 1 -- start start 2024-03-29T10:11:00.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.881+0000 7f15baa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 0x7f15b4157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:00.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.881+0000 7f15baa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 0x7f15b41f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.881+0000 7f15baa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b4160d70 0x7f15b41546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.881+0000 7f15baa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b40b0890 con 0x7f15b406a790 2024-03-29T10:11:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.881+0000 7f15baa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b40b0b70 con 0x7f15b4160d70 2024-03-29T10:11:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.881+0000 7f15baa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b40b0e50 con 0x7f15b415ef70 2024-03-29T10:11:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 0x7f15b4157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 0x7f15b4157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39438/0 (socket says 172.21.15.12:39438) 2024-03-29T10:11:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b3e00700 1 -- 172.21.15.12:0/2235466658 learned_addr learned my addr 172.21.15.12:0/2235466658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b3400700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 0x7f15b41f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b8a00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b4160d70 0x7f15b41546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:00.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b3e00700 1 -- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 msgr2=0x7f15b41f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b3e00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 0x7f15b41f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b3e00700 1 -- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b4160d70 msgr2=0x7f15b41546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b3e00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b4160d70 0x7f15b41546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b3e00700 1 -- 172.21.15.12:0/2235466658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac009b30 con 0x7f15b406a790 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b8a00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b4160d70 0x7f15b41546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.882+0000 7f15b3e00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 0x7f15b4157240 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f15a400e9b0 tx=0x7f15a400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.883+0000 7f15abe00700 1 -- 172.21.15.12:0/2235466658 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4009710 con 0x7f15b406a790 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.883+0000 7f15abe00700 1 -- 172.21.15.12:0/2235466658 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a400c850 con 0x7f15b406a790 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.883+0000 7f15abe00700 1 -- 172.21.15.12:0/2235466658 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a400cbc0 con 0x7f15b406a790 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.883+0000 7f15baa00700 1 -- 172.21.15.12:0/2235466658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b40b1130 con 0x7f15b406a790 2024-03-29T10:11:00.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.883+0000 7f15baa00700 1 -- 172.21.15.12:0/2235466658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b4157780 con 0x7f15b406a790 2024-03-29T10:11:00.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.884+0000 7f15baa00700 1 -- 172.21.15.12:0/2235466658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b4067720 con 0x7f15b406a790 2024-03-29T10:11:00.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.885+0000 7f15abe00700 1 -- 172.21.15.12:0/2235466658 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f15a400cd20 con 0x7f15b406a790 2024-03-29T10:11:00.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.886+0000 7f15abe00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159406adb0 0x7f159406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:00.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.886+0000 7f15abe00700 1 -- 172.21.15.12:0/2235466658 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15a40895a0 con 0x7f15b406a790 2024-03-29T10:11:00.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.886+0000 7f15b3400700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159406adb0 0x7f159406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:00.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.887+0000 7f15b3400700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159406adb0 0x7f159406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f15b40d0dd0 tx=0x7f15a0008040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:00.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:00.890+0000 7f15abe00700 1 -- 172.21.15.12:0/2235466658 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15a4054670 con 0x7f15b406a790 2024-03-29T10:11:01.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:01.154+0000 7f15baa00700 1 -- 172.21.15.12:0/2235466658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f15b406eda0 con 0x7f15b406a790 2024-03-29T10:11:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.119+0000 7f15abe00700 1 -- 172.21.15.12:0/2235466658 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f15a4057c90 con 0x7f15b406a790 2024-03-29T10:11:02.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 -- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159406adb0 msgr2=0x7f159406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159406adb0 0x7f159406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f15b40d0dd0 tx=0x7f15a0008040 comp rx=0 tx=0).stop 2024-03-29T10:11:02.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 -- 172.21.15.12:0/2235466658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 msgr2=0x7f15b4157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 0x7f15b4157240 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f15a400e9b0 tx=0x7f15a400ed70 comp rx=0 tx=0).stop 2024-03-29T10:11:02.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 -- 172.21.15.12:0/2235466658 shutdown_connections 2024-03-29T10:11:02.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f159406adb0 0x7f159406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b4160d70 0x7f15b41546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b406a790 0x7f15b4157240 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 --2- 172.21.15.12:0/2235466658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b415ef70 0x7f15b41f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 -- 172.21.15.12:0/2235466658 >> 172.21.15.12:0/2235466658 conn(0x7f15b40b5720 msgr2=0x7f15b4156110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:02.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 -- 172.21.15.12:0/2235466658 shutdown_connections 2024-03-29T10:11:02.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.122+0000 7f15baa00700 1 -- 172.21.15.12:0/2235466658 wait complete. 2024-03-29T10:11:02.143 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T10:11:02.143 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:11:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.345+0000 7f0c77200700 1 Processor -- start 2024-03-29T10:11:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.345+0000 7f0c77200700 1 -- start start 2024-03-29T10:11:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.346+0000 7f0c77200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700cb3f0 0x7f0c700ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.346+0000 7f0c77200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0c700ac600 0x7f0c7006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.346+0000 7f0c77200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7015fd50 con 0x7f0c7015b5d0 2024-03-29T10:11:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.346+0000 7f0c77200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c7015b9b0 con 0x7f0c700ac600 2024-03-29T10:11:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.346+0000 7f0c77200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700aca70 con 0x7f0c700cb3f0 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.347+0000 7f0c75200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0c700ac600 0x7f0c7006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.347+0000 7f0c6f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700cb3f0 0x7f0c700ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.347+0000 7f0c75200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0c700ac600 0x7f0c7006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:49452/0 (socket says 172.21.15.12:49452) 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.347+0000 7f0c75200700 1 -- 172.21.15.12:0/929848042 learned_addr learned my addr 172.21.15.12:0/929848042 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.347+0000 7f0c6fe00700 1 --1- 172.21.15.12:0/929848042 >> v1:172.21.15.159:6789/0 conn(0x7f0c7015b5d0 0x7f0c700ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:35182/0 (socket says 172.21.15.12:35182) 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.347+0000 7f0c6f400700 1 -- 172.21.15.12:0/929848042 >> v2:172.21.15.159:3300/0 conn(0x7f0c700ac600 msgr2=0x7f0c7006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.347+0000 7f0c6f400700 1 --2- 172.21.15.12:0/929848042 >> v2:172.21.15.159:3300/0 conn(0x7f0c700ac600 0x7f0c7006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.347+0000 7f0c6f400700 1 -- 172.21.15.12:0/929848042 >> v1:172.21.15.159:6789/0 conn(0x7f0c7015b5d0 legacy=0x7f0c700ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.347+0000 7f0c6f400700 1 -- 172.21.15.12:0/929848042 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7015f140 con 0x7f0c700cb3f0 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.347+0000 7f0c75200700 1 --2- 172.21.15.12:0/929848042 >> v2:172.21.15.159:3300/0 conn(0x7f0c700ac600 0x7f0c7006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.348+0000 7f0c6f400700 1 --2- 172.21.15.12:0/929848042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700cb3f0 0x7f0c700ac0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009840 tx=0x7f0c5c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37f4ef3803378ae2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.348+0000 7f0c6ea00700 1 -- 172.21.15.12:0/929848042 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c011830 con 0x7f0c700cb3f0 2024-03-29T10:11:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.348+0000 7f0c6ea00700 1 -- 172.21.15.12:0/929848042 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c011990 con 0x7f0c700cb3f0 2024-03-29T10:11:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.348+0000 7f0c6ea00700 1 -- 172.21.15.12:0/929848042 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c011d20 con 0x7f0c700cb3f0 2024-03-29T10:11:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.349+0000 7f0c77200700 1 -- 172.21.15.12:0/929848042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700cb3f0 msgr2=0x7f0c700ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.349+0000 7f0c77200700 1 --2- 172.21.15.12:0/929848042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700cb3f0 0x7f0c700ac0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009840 tx=0x7f0c5c009b50 comp rx=0 tx=0).stop 2024-03-29T10:11:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.349+0000 7f0c77200700 1 -- 172.21.15.12:0/929848042 shutdown_connections 2024-03-29T10:11:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.349+0000 7f0c77200700 1 --2- 172.21.15.12:0/929848042 >> v2:172.21.15.159:3300/0 conn(0x7f0c700ac600 0x7f0c7006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.349+0000 7f0c77200700 1 --2- 172.21.15.12:0/929848042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700cb3f0 0x7f0c700ac0c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.349+0000 7f0c77200700 1 -- 172.21.15.12:0/929848042 >> 172.21.15.12:0/929848042 conn(0x7f0c700cf190 msgr2=0x7f0c700cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:02.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.349+0000 7f0c77200700 1 -- 172.21.15.12:0/929848042 shutdown_connections 2024-03-29T10:11:02.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.349+0000 7f0c77200700 1 -- 172.21.15.12:0/929848042 wait complete. 2024-03-29T10:11:02.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.350+0000 7f0c77200700 1 Processor -- start 2024-03-29T10:11:02.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.351+0000 7f0c77200700 1 -- start start 2024-03-29T10:11:02.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.351+0000 7f0c77200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700ac600 0x7f0c700c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.351+0000 7f0c77200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c7015b5d0 0x7f0c700cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.351+0000 7f0c77200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c700ce460 0x7f0c70071640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c77200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c70072130 con 0x7f0c700ce460 2024-03-29T10:11:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c77200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700c8590 con 0x7f0c7015b5d0 2024-03-29T10:11:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c77200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700c8870 con 0x7f0c700ac600 2024-03-29T10:11:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c75200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c700ce460 0x7f0c70071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c75200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c700ce460 0x7f0c70071640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39442/0 (socket says 172.21.15.12:39442) 2024-03-29T10:11:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c75200700 1 -- 172.21.15.12:0/1266594840 learned_addr learned my addr 172.21.15.12:0/1266594840 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c6fe00700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700ac600 0x7f0c700c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c6f400700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c7015b5d0 0x7f0c700cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c75200700 1 -- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700ac600 msgr2=0x7f0c700c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c75200700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700ac600 0x7f0c700c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c75200700 1 -- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c7015b5d0 msgr2=0x7f0c700cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c75200700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c7015b5d0 0x7f0c700cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.352+0000 7f0c75200700 1 -- 172.21.15.12:0/1266594840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c68009b30 con 0x7f0c700ce460 2024-03-29T10:11:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.353+0000 7f0c75200700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c700ce460 0x7f0c70071640 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800efc0 tx=0x7f0c6800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.353+0000 7f0c67e00700 1 -- 172.21.15.12:0/1266594840 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68018070 con 0x7f0c700ce460 2024-03-29T10:11:02.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.353+0000 7f0c67e00700 1 -- 172.21.15.12:0/1266594840 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6800f040 con 0x7f0c700ce460 2024-03-29T10:11:02.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.353+0000 7f0c77200700 1 -- 172.21.15.12:0/1266594840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c5c0094a0 con 0x7f0c700ce460 2024-03-29T10:11:02.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.353+0000 7f0c67e00700 1 -- 172.21.15.12:0/1266594840 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68003c20 con 0x7f0c700ce460 2024-03-29T10:11:02.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.353+0000 7f0c77200700 1 -- 172.21.15.12:0/1266594840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c700c8e30 con 0x7f0c700ce460 2024-03-29T10:11:02.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.354+0000 7f0c77200700 1 -- 172.21.15.12:0/1266594840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c700caf10 con 0x7f0c700ce460 2024-03-29T10:11:02.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.355+0000 7f0c67e00700 1 -- 172.21.15.12:0/1266594840 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0c6800c880 con 0x7f0c700ce460 2024-03-29T10:11:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.356+0000 7f0c67e00700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0c5006adb0 0x7f0c5006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.356+0000 7f0c67e00700 1 -- 172.21.15.12:0/1266594840 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c68089750 con 0x7f0c700ce460 2024-03-29T10:11:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.356+0000 7f0c6fe00700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0c5006adb0 0x7f0c5006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.357+0000 7f0c6fe00700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0c5006adb0 0x7f0c5006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0c60000f60 tx=0x7f0c60007ed0 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:02.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.360+0000 7f0c67e00700 1 -- 172.21.15.12:0/1266594840 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c68054700 con 0x7f0c700ce460 2024-03-29T10:11:02.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.618+0000 7f0c77200700 1 -- 172.21.15.12:0/1266594840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c70067720 con 0x7f0c700ce460 2024-03-29T10:11:02.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.619+0000 7f0c67e00700 1 -- 172.21.15.12:0/1266594840 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f0c68057d20 con 0x7f0c700ce460 2024-03-29T10:11:02.618 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:11:02.619 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6452,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2137480356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2137480356},{"type":"v1","addr":"172.21.15.159:6836","nonce":2137480356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":6455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2356039533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2356039533},{"type":"v1","addr":"172.21.15.159:6837","nonce":2356039533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":7153,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2916743006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2916743006},{"type":"v1","addr":"172.21.15.12:6835","nonce":2916743006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:58.634187+0000","modified":"2024-03-29T10:11:02.111942+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7410":{"gid":7410,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3023331068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3023331068},{"type":"v1","addr":"172.21.15.12:6834","nonce":3023331068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-29T10:11:02.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 -- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0c5006adb0 msgr2=0x7f0c5006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0c5006adb0 0x7f0c5006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0c60000f60 tx=0x7f0c60007ed0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 -- 172.21.15.12:0/1266594840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c700ce460 msgr2=0x7f0c70071640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c700ce460 0x7f0c70071640 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800efc0 tx=0x7f0c6800c3a0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 -- 172.21.15.12:0/1266594840 shutdown_connections 2024-03-29T10:11:02.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0c5006adb0 0x7f0c5006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c700ce460 0x7f0c70071640 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c700ac600 0x7f0c700c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 --2- 172.21.15.12:0/1266594840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c7015b5d0 0x7f0c700cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 -- 172.21.15.12:0/1266594840 >> 172.21.15.12:0/1266594840 conn(0x7f0c700cf190 msgr2=0x7f0c70156c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 -- 172.21.15.12:0/1266594840 shutdown_connections 2024-03-29T10:11:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.621+0000 7f0c77200700 1 -- 172.21.15.12:0/1266594840 wait complete. 2024-03-29T10:11:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 46 2024-03-29T10:11:02.639 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:11:02.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.838+0000 7f1e37600700 1 Processor -- start 2024-03-29T10:11:02.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.838+0000 7f1e37600700 1 -- start start 2024-03-29T10:11:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.839+0000 7f1e37600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e30160f10 0x7f1e300d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.839+0000 7f1e37600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1e300cf230 0x7f1e300d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.839+0000 7f1e37600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e300ceda0 con 0x7f1e30161590 2024-03-29T10:11:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.839+0000 7f1e37600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e300d23d0 con 0x7f1e30160f10 2024-03-29T10:11:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.839+0000 7f1e37600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e301612f0 con 0x7f1e300cf230 2024-03-29T10:11:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.839+0000 7f1e34c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e30160f10 0x7f1e300d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.839+0000 7f1e34c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e30160f10 0x7f1e300d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39450/0 (socket says 172.21.15.12:39450) 2024-03-29T10:11:02.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.839+0000 7f1e34c00700 1 -- 172.21.15.12:0/526054429 learned_addr learned my addr 172.21.15.12:0/526054429 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:02.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.839+0000 7f1e2fe00700 1 --2- 172.21.15.12:0/526054429 >> v2:172.21.15.159:3300/0 conn(0x7f1e300cf230 0x7f1e300d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.840+0000 7f1e34c00700 1 -- 172.21.15.12:0/526054429 >> v2:172.21.15.159:3300/0 conn(0x7f1e300cf230 msgr2=0x7f1e300d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.840+0000 7f1e34c00700 1 --2- 172.21.15.12:0/526054429 >> v2:172.21.15.159:3300/0 conn(0x7f1e300cf230 0x7f1e300d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.840+0000 7f1e34c00700 1 -- 172.21.15.12:0/526054429 >> v1:172.21.15.159:6789/0 conn(0x7f1e30161590 legacy=0x7f1e300b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.840+0000 7f1e34c00700 1 -- 172.21.15.12:0/526054429 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e300ce470 con 0x7f1e30160f10 2024-03-29T10:11:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.840+0000 7f1e34c00700 1 --2- 172.21.15.12:0/526054429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e30160f10 0x7f1e300d18b0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f1e3008cdb0 tx=0x7f1e24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c26a8e9cbd757a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.840+0000 7f1e2f400700 1 -- 172.21.15.12:0/526054429 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2400b870 con 0x7f1e30160f10 2024-03-29T10:11:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.840+0000 7f1e2f400700 1 -- 172.21.15.12:0/526054429 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2400b9d0 con 0x7f1e30160f10 2024-03-29T10:11:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.840+0000 7f1e2f400700 1 -- 172.21.15.12:0/526054429 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2400bd60 con 0x7f1e30160f10 2024-03-29T10:11:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.841+0000 7f1e37600700 1 -- 172.21.15.12:0/526054429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e30160f10 msgr2=0x7f1e300d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.841+0000 7f1e37600700 1 --2- 172.21.15.12:0/526054429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e30160f10 0x7f1e300d18b0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f1e3008cdb0 tx=0x7f1e24009bd0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.841+0000 7f1e37600700 1 -- 172.21.15.12:0/526054429 shutdown_connections 2024-03-29T10:11:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.841+0000 7f1e37600700 1 --2- 172.21.15.12:0/526054429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e30160f10 0x7f1e300d18b0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.841+0000 7f1e37600700 1 --2- 172.21.15.12:0/526054429 >> v2:172.21.15.159:3300/0 conn(0x7f1e300cf230 0x7f1e300d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.841+0000 7f1e37600700 1 -- 172.21.15.12:0/526054429 >> 172.21.15.12:0/526054429 conn(0x7f1e3008aff0 msgr2=0x7f1e300d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.841+0000 7f1e37600700 1 -- 172.21.15.12:0/526054429 shutdown_connections 2024-03-29T10:11:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.841+0000 7f1e37600700 1 -- 172.21.15.12:0/526054429 wait complete. 2024-03-29T10:11:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.843+0000 7f1e37600700 1 Processor -- start 2024-03-29T10:11:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.843+0000 7f1e37600700 1 -- start start 2024-03-29T10:11:02.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e37600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e300cf230 0x7f1e301525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e37600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e30160f10 0x7f1e30158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e37600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e30161590 0x7f1e30155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e37600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e301f74f0 con 0x7f1e300cf230 2024-03-29T10:11:02.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e37600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30157f00 con 0x7f1e30161590 2024-03-29T10:11:02.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e37600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e301581e0 con 0x7f1e30160f10 2024-03-29T10:11:02.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e34c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e300cf230 0x7f1e301525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e34c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e300cf230 0x7f1e301525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39456/0 (socket says 172.21.15.12:39456) 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e34c00700 1 -- 172.21.15.12:0/10760373 learned_addr learned my addr 172.21.15.12:0/10760373 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e2fe00700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e30160f10 0x7f1e30158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e35600700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e30161590 0x7f1e30155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e34c00700 1 -- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e30160f10 msgr2=0x7f1e30158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e34c00700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e30160f10 0x7f1e30158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e34c00700 1 -- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e30161590 msgr2=0x7f1e30155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e34c00700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e30161590 0x7f1e30155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e34c00700 1 -- 172.21.15.12:0/10760373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e24009550 con 0x7f1e300cf230 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e2fe00700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e30160f10 0x7f1e30158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.844+0000 7f1e35600700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e30161590 0x7f1e30155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:11:02.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.845+0000 7f1e34c00700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e300cf230 0x7f1e301525c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f1e24000c00 tx=0x7f1e24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:02.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.845+0000 7f1e2cc00700 1 -- 172.21.15.12:0/10760373 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24004520 con 0x7f1e300cf230 2024-03-29T10:11:02.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.845+0000 7f1e2cc00700 1 -- 172.21.15.12:0/10760373 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2401e070 con 0x7f1e300cf230 2024-03-29T10:11:02.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.845+0000 7f1e2cc00700 1 -- 172.21.15.12:0/10760373 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e240038a0 con 0x7f1e300cf230 2024-03-29T10:11:02.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.845+0000 7f1e37600700 1 -- 172.21.15.12:0/10760373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e301584c0 con 0x7f1e300cf230 2024-03-29T10:11:02.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.845+0000 7f1e37600700 1 -- 172.21.15.12:0/10760373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e30158a80 con 0x7f1e300cf230 2024-03-29T10:11:02.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.846+0000 7f1e37600700 1 -- 172.21.15.12:0/10760373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e30158cc0 con 0x7f1e300cf230 2024-03-29T10:11:02.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.847+0000 7f1e2cc00700 1 -- 172.21.15.12:0/10760373 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1e24003a00 con 0x7f1e300cf230 2024-03-29T10:11:02.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.848+0000 7f1e2cc00700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e1006adb0 0x7f1e1006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:02.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.848+0000 7f1e2cc00700 1 -- 172.21.15.12:0/10760373 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e24020070 con 0x7f1e300cf230 2024-03-29T10:11:02.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.848+0000 7f1e2fe00700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e1006adb0 0x7f1e1006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:02.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.849+0000 7f1e2fe00700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e1006adb0 0x7f1e1006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c007a50 tx=0x7f1e1c008040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:02.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:02.852+0000 7f1e2cc00700 1 -- 172.21.15.12:0/10760373 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e30158cc0 con 0x7f1e300cf230 2024-03-29T10:11:03.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.073+0000 7f1e37600700 1 -- 172.21.15.12:0/10760373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e30158cc0 con 0x7f1e300cf230 2024-03-29T10:11:03.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.074+0000 7f1e2cc00700 1 -- 172.21.15.12:0/10760373 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16103 (secure 0 0 0) 0x7f1e30158cc0 con 0x7f1e300cf230 2024-03-29T10:11:03.074 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:11:03.074 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":108,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:10:59.637023+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:57.064406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:57.607254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:11:03.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.076+0000 7f1e37600700 1 -- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e1006adb0 msgr2=0x7f1e1006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.076+0000 7f1e37600700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e1006adb0 0x7f1e1006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c007a50 tx=0x7f1e1c008040 comp rx=0 tx=0).stop 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.076+0000 7f1e37600700 1 -- 172.21.15.12:0/10760373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e300cf230 msgr2=0x7f1e301525c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.076+0000 7f1e37600700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e300cf230 0x7f1e301525c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f1e24000c00 tx=0x7f1e24004750 comp rx=0 tx=0).stop 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.077+0000 7f1e37600700 1 -- 172.21.15.12:0/10760373 shutdown_connections 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.077+0000 7f1e37600700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1e1006adb0 0x7f1e1006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.077+0000 7f1e37600700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e30161590 0x7f1e30155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.077+0000 7f1e37600700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e300cf230 0x7f1e301525c0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.077+0000 7f1e37600700 1 --2- 172.21.15.12:0/10760373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e30160f10 0x7f1e30158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.077+0000 7f1e37600700 1 -- 172.21.15.12:0/10760373 >> 172.21.15.12:0/10760373 conn(0x7f1e3008aff0 msgr2=0x7f1e300cad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.077+0000 7f1e37600700 1 -- 172.21.15.12:0/10760373 shutdown_connections 2024-03-29T10:11:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.077+0000 7f1e37600700 1 -- 172.21.15.12:0/10760373 wait complete. 2024-03-29T10:11:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T10:11:03.095 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:11:03.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.293+0000 7fd633e00700 1 Processor -- start 2024-03-29T10:11:03.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.293+0000 7fd633e00700 1 -- start start 2024-03-29T10:11:03.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.294+0000 7fd633e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c160f10 0x7fd62c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.294+0000 7fd633e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd62c161590 0x7fd62c161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.294+0000 7fd633e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62c0ceda0 con 0x7fd62c0cf230 2024-03-29T10:11:03.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.294+0000 7fd633e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd62c161f80 con 0x7fd62c161590 2024-03-29T10:11:03.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.294+0000 7fd633e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c1612f0 con 0x7fd62c160f10 2024-03-29T10:11:03.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd631e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd62c161590 0x7fd62c161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd630a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd62c0cf230 0x7fd62c0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60778/0 (socket says 172.21.15.12:60778) 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd631e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd62c161590 0x7fd62c161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39464/0 (socket says 172.21.15.12:39464) 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd630a00700 1 -- 172.21.15.12:0/2049575856 learned_addr learned my addr 172.21.15.12:0/2049575856 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd631400700 1 --2- 172.21.15.12:0/2049575856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c160f10 0x7fd62c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd631e00700 1 -- 172.21.15.12:0/2049575856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c160f10 msgr2=0x7fd62c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd631e00700 1 --2- 172.21.15.12:0/2049575856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c160f10 0x7fd62c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd631e00700 1 -- 172.21.15.12:0/2049575856 >> v1:172.21.15.12:6789/0 conn(0x7fd62c0cf230 legacy=0x7fd62c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd631e00700 1 -- 172.21.15.12:0/2049575856 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c0ce470 con 0x7fd62c161590 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd62be00700 1 -- 172.21.15.12:0/2049575856 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43080918 0 0) 0x7fd62c0ceda0 con 0x7fd62c0cf230 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.295+0000 7fd631e00700 1 --2- 172.21.15.12:0/2049575856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c161590 0x7fd62c161a40 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd624009dc0 tx=0x7fd62400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67b7703cd9a6cd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd62be00700 1 -- 172.21.15.12:0/2049575856 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62400e070 con 0x7fd62c161590 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd62be00700 1 -- 172.21.15.12:0/2049575856 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62400bb90 con 0x7fd62c161590 2024-03-29T10:11:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd62be00700 1 -- 172.21.15.12:0/2049575856 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624012430 con 0x7fd62c161590 2024-03-29T10:11:03.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd633e00700 1 -- 172.21.15.12:0/2049575856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c161590 msgr2=0x7fd62c161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd633e00700 1 --2- 172.21.15.12:0/2049575856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c161590 0x7fd62c161a40 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd624009dc0 tx=0x7fd62400b330 comp rx=0 tx=0).stop 2024-03-29T10:11:03.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd633e00700 1 -- 172.21.15.12:0/2049575856 shutdown_connections 2024-03-29T10:11:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd633e00700 1 --2- 172.21.15.12:0/2049575856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c161590 0x7fd62c161a40 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd633e00700 1 --2- 172.21.15.12:0/2049575856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c160f10 0x7fd62c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd633e00700 1 -- 172.21.15.12:0/2049575856 >> 172.21.15.12:0/2049575856 conn(0x7fd62c08aff0 msgr2=0x7fd62c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd633e00700 1 -- 172.21.15.12:0/2049575856 shutdown_connections 2024-03-29T10:11:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.296+0000 7fd633e00700 1 -- 172.21.15.12:0/2049575856 wait complete. 2024-03-29T10:11:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.298+0000 7fd633e00700 1 Processor -- start 2024-03-29T10:11:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.298+0000 7fd633e00700 1 -- start start 2024-03-29T10:11:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.299+0000 7fd633e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd62c0cf230 0x7fd62c14eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.299+0000 7fd633e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c160f10 0x7fd62c15dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.299+0000 7fd633e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c161590 0x7fd62c09e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.299+0000 7fd633e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c14f080 con 0x7fd62c160f10 2024-03-29T10:11:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.299+0000 7fd633e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c15e5f0 con 0x7fd62c0cf230 2024-03-29T10:11:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.299+0000 7fd633e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c15e8d0 con 0x7fd62c161590 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.299+0000 7fd630a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c160f10 0x7fd62c15dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.299+0000 7fd630a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c160f10 0x7fd62c15dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39472/0 (socket says 172.21.15.12:39472) 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.299+0000 7fd630a00700 1 -- 172.21.15.12:0/1555599535 learned_addr learned my addr 172.21.15.12:0/1555599535 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd631400700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd62c0cf230 0x7fd62c14eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd631e00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c161590 0x7fd62c09e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd630a00700 1 -- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c161590 msgr2=0x7fd62c09e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd630a00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c161590 0x7fd62c09e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd630a00700 1 -- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd62c0cf230 msgr2=0x7fd62c14eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd630a00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd62c0cf230 0x7fd62c14eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd630a00700 1 -- 172.21.15.12:0/1555599535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd624009a70 con 0x7fd62c160f10 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd631400700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd62c0cf230 0x7fd62c14eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:11:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd631e00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c161590 0x7fd62c09e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:11:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd630a00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c160f10 0x7fd62c15dd30 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd61800c520 tx=0x7fd61800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd629600700 1 -- 172.21.15.12:0/1555599535 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd618004c80 con 0x7fd62c160f10 2024-03-29T10:11:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd629600700 1 -- 172.21.15.12:0/1555599535 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd618004de0 con 0x7fd62c160f10 2024-03-29T10:11:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd629600700 1 -- 172.21.15.12:0/1555599535 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180072d0 con 0x7fd62c160f10 2024-03-29T10:11:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd633e00700 1 -- 172.21.15.12:0/1555599535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62c15ebb0 con 0x7fd62c160f10 2024-03-29T10:11:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.300+0000 7fd633e00700 1 -- 172.21.15.12:0/1555599535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62c15f170 con 0x7fd62c160f10 2024-03-29T10:11:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.302+0000 7fd633e00700 1 -- 172.21.15.12:0/1555599535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62c15f390 con 0x7fd62c160f10 2024-03-29T10:11:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.302+0000 7fd629600700 1 -- 172.21.15.12:0/1555599535 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd618007430 con 0x7fd62c160f10 2024-03-29T10:11:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.303+0000 7fd629600700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd60c06adb0 0x7fd60c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.303+0000 7fd629600700 1 -- 172.21.15.12:0/1555599535 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6180891b0 con 0x7fd62c160f10 2024-03-29T10:11:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.304+0000 7fd631400700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd60c06adb0 0x7fd60c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.305+0000 7fd631400700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd60c06adb0 0x7fd60c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0c9a20 tx=0x7fd620008040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.308+0000 7fd629600700 1 -- 172.21.15.12:0/1555599535 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd618054270 con 0x7fd62c160f10 2024-03-29T10:11:03.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.522+0000 7fd633e00700 1 -- 172.21.15.12:0/1555599535 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd62c0cea30 con 0x7fd60c06adb0 2024-03-29T10:11:03.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.533+0000 7fd629600700 1 -- 172.21.15.12:0/1555599535 <== mgr.6940 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289215 (secure 0 0 0) 0x7fd6200133f0 con 0x7fd60c06adb0 2024-03-29T10:11:03.538 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:11:03.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 -- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd60c06adb0 msgr2=0x7fd60c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd60c06adb0 0x7fd60c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0c9a20 tx=0x7fd620008040 comp rx=0 tx=0).stop 2024-03-29T10:11:03.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 -- 172.21.15.12:0/1555599535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c160f10 msgr2=0x7fd62c15dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c160f10 0x7fd62c15dd30 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd61800c520 tx=0x7fd61800c8e0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 -- 172.21.15.12:0/1555599535 shutdown_connections 2024-03-29T10:11:03.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd60c06adb0 0x7fd60c06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd62c161590 0x7fd62c09e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd62c0cf230 0x7fd62c14eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 --2- 172.21.15.12:0/1555599535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62c160f10 0x7fd62c15dd30 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 -- 172.21.15.12:0/1555599535 >> 172.21.15.12:0/1555599535 conn(0x7fd62c08aff0 msgr2=0x7fd62c0cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:03.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 -- 172.21.15.12:0/1555599535 shutdown_connections 2024-03-29T10:11:03.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.538+0000 7fd633e00700 1 -- 172.21.15.12:0/1555599535 wait complete. 2024-03-29T10:11:03.542 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:11:03.556 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-29T10:11:01.621879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":355,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":321112,"kb_used_data":11464,"kb_used_omap":12,"kb_used_meta":309491,"kb_avail":754653608,"statfs":{"total":773094113280,"available":772765294592,"internally_reserved":0,"allocated":11739136,"data_stored":6168204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":316919353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.519127"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.657359+0000","last_change":"2024-03-29T10:10:58.652413+0000","last_active":"2024-03-29T10:10:58.657359+0000","last_peered":"2024-03-29T10:10:58.657359+0000","last_clean":"2024-03-29T10:10:58.657359+0000","last_became_active":"2024-03-29T10:10:58.650620+0000","last_became_peered":"2024-03-29T10:10:58.650620+0000","last_unstale":"2024-03-29T10:10:58.657359+0000","last_undegraded":"2024-03-29T10:10:58.657359+0000","last_fullsized":"2024-03-29T10:10:58.657359+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:43:18.124198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675685+0000","last_change":"2024-03-29T10:10:58.665186+0000","last_active":"2024-03-29T10:10:58.675685+0000","last_peered":"2024-03-29T10:10:58.675685+0000","last_clean":"2024-03-29T10:10:58.675685+0000","last_became_active":"2024-03-29T10:10:58.664316+0000","last_became_peered":"2024-03-29T10:10:58.664316+0000","last_unstale":"2024-03-29T10:10:58.675685+0000","last_undegraded":"2024-03-29T10:10:58.675685+0000","last_fullsized":"2024-03-29T10:10:58.675685+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:16.050232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669060+0000","last_change":"2024-03-29T10:10:58.667945+0000","last_active":"2024-03-29T10:10:58.669060+0000","last_peered":"2024-03-29T10:10:58.669060+0000","last_clean":"2024-03-29T10:10:58.669060+0000","last_became_active":"2024-03-29T10:10:58.667633+0000","last_became_peered":"2024-03-29T10:10:58.667633+0000","last_unstale":"2024-03-29T10:10:58.669060+0000","last_undegraded":"2024-03-29T10:10:58.669060+0000","last_fullsized":"2024-03-29T10:10:58.669060+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:13.760911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.667174+0000","last_change":"2024-03-29T10:10:58.655940+0000","last_active":"2024-03-29T10:10:58.667174+0000","last_peered":"2024-03-29T10:10:58.667174+0000","last_clean":"2024-03-29T10:10:58.667174+0000","last_became_active":"2024-03-29T10:10:58.654685+0000","last_became_peered":"2024-03-29T10:10:58.654685+0000","last_unstale":"2024-03-29T10:10:58.667174+0000","last_undegraded":"2024-03-29T10:10:58.667174+0000","last_fullsized":"2024-03-29T10:10:58.667174+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:37.434553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.664086+0000","last_change":"2024-03-29T10:10:58.662439+0000","last_active":"2024-03-29T10:10:58.664086+0000","last_peered":"2024-03-29T10:10:58.664086+0000","last_clean":"2024-03-29T10:10:58.664086+0000","last_became_active":"2024-03-29T10:10:58.662105+0000","last_became_peered":"2024-03-29T10:10:58.662105+0000","last_unstale":"2024-03-29T10:10:58.664086+0000","last_undegraded":"2024-03-29T10:10:58.664086+0000","last_fullsized":"2024-03-29T10:10:58.664086+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:57:08.688729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669723+0000","last_change":"2024-03-29T10:10:58.667532+0000","last_active":"2024-03-29T10:10:58.669723+0000","last_peered":"2024-03-29T10:10:58.669723+0000","last_clean":"2024-03-29T10:10:58.669723+0000","last_became_active":"2024-03-29T10:10:58.665389+0000","last_became_peered":"2024-03-29T10:10:58.665389+0000","last_unstale":"2024-03-29T10:10:58.669723+0000","last_undegraded":"2024-03-29T10:10:58.669723+0000","last_fullsized":"2024-03-29T10:10:58.669723+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:24.296829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.666669+0000","last_change":"2024-03-29T10:10:58.656245+0000","last_active":"2024-03-29T10:10:58.666669+0000","last_peered":"2024-03-29T10:10:58.666669+0000","last_clean":"2024-03-29T10:10:58.666669+0000","last_became_active":"2024-03-29T10:10:58.655953+0000","last_became_peered":"2024-03-29T10:10:58.655953+0000","last_unstale":"2024-03-29T10:10:58.666669+0000","last_undegraded":"2024-03-29T10:10:58.666669+0000","last_fullsized":"2024-03-29T10:10:58.666669+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:50:56.622833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.657364+0000","last_change":"2024-03-29T10:10:58.653695+0000","last_active":"2024-03-29T10:10:58.657364+0000","last_peered":"2024-03-29T10:10:58.657364+0000","last_clean":"2024-03-29T10:10:58.657364+0000","last_became_active":"2024-03-29T10:10:58.653412+0000","last_became_peered":"2024-03-29T10:10:58.653412+0000","last_unstale":"2024-03-29T10:10:58.657364+0000","last_undegraded":"2024-03-29T10:10:58.657364+0000","last_fullsized":"2024-03-29T10:10:58.657364+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:17:24.194613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.668675+0000","last_change":"2024-03-29T10:10:58.665371+0000","last_active":"2024-03-29T10:10:58.668675+0000","last_peered":"2024-03-29T10:10:58.668675+0000","last_clean":"2024-03-29T10:10:58.668675+0000","last_became_active":"2024-03-29T10:10:58.665082+0000","last_became_peered":"2024-03-29T10:10:58.665082+0000","last_unstale":"2024-03-29T10:10:58.668675+0000","last_undegraded":"2024-03-29T10:10:58.668675+0000","last_fullsized":"2024-03-29T10:10:58.668675+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:21.918440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.666270+0000","last_change":"2024-03-29T10:10:58.660152+0000","last_active":"2024-03-29T10:10:58.666270+0000","last_peered":"2024-03-29T10:10:58.666270+0000","last_clean":"2024-03-29T10:10:58.666270+0000","last_became_active":"2024-03-29T10:10:58.659014+0000","last_became_peered":"2024-03-29T10:10:58.659014+0000","last_unstale":"2024-03-29T10:10:58.666270+0000","last_undegraded":"2024-03-29T10:10:58.666270+0000","last_fullsized":"2024-03-29T10:10:58.666270+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:29:27.283488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.665330+0000","last_change":"2024-03-29T10:10:58.662886+0000","last_active":"2024-03-29T10:10:58.665330+0000","last_peered":"2024-03-29T10:10:58.665330+0000","last_clean":"2024-03-29T10:10:58.665330+0000","last_became_active":"2024-03-29T10:10:58.662182+0000","last_became_peered":"2024-03-29T10:10:58.662182+0000","last_unstale":"2024-03-29T10:10:58.665330+0000","last_undegraded":"2024-03-29T10:10:58.665330+0000","last_fullsized":"2024-03-29T10:10:58.665330+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:52:39.940245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.664040+0000","last_change":"2024-03-29T10:10:58.661018+0000","last_active":"2024-03-29T10:10:58.664040+0000","last_peered":"2024-03-29T10:10:58.664040+0000","last_clean":"2024-03-29T10:10:58.664040+0000","last_became_active":"2024-03-29T10:10:58.660752+0000","last_became_peered":"2024-03-29T10:10:58.660752+0000","last_unstale":"2024-03-29T10:10:58.664040+0000","last_undegraded":"2024-03-29T10:10:58.664040+0000","last_fullsized":"2024-03-29T10:10:58.664040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:24.188405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662745+0000","last_change":"2024-03-29T10:10:58.658393+0000","last_active":"2024-03-29T10:10:58.662745+0000","last_peered":"2024-03-29T10:10:58.662745+0000","last_clean":"2024-03-29T10:10:58.662745+0000","last_became_active":"2024-03-29T10:10:58.657921+0000","last_became_peered":"2024-03-29T10:10:58.657921+0000","last_unstale":"2024-03-29T10:10:58.662745+0000","last_undegraded":"2024-03-29T10:10:58.662745+0000","last_fullsized":"2024-03-29T10:10:58.662745+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:37:57.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.665493+0000","last_change":"2024-03-29T10:10:58.654494+0000","last_active":"2024-03-29T10:10:58.665493+0000","last_peered":"2024-03-29T10:10:58.665493+0000","last_clean":"2024-03-29T10:10:58.665493+0000","last_became_active":"2024-03-29T10:10:58.654124+0000","last_became_peered":"2024-03-29T10:10:58.654124+0000","last_unstale":"2024-03-29T10:10:58.665493+0000","last_undegraded":"2024-03-29T10:10:58.665493+0000","last_fullsized":"2024-03-29T10:10:58.665493+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:25:38.029319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662667+0000","last_change":"2024-03-29T10:10:58.658270+0000","last_active":"2024-03-29T10:10:58.662667+0000","last_peered":"2024-03-29T10:10:58.662667+0000","last_clean":"2024-03-29T10:10:58.662667+0000","last_became_active":"2024-03-29T10:10:58.658002+0000","last_became_peered":"2024-03-29T10:10:58.658002+0000","last_unstale":"2024-03-29T10:10:58.662667+0000","last_undegraded":"2024-03-29T10:10:58.662667+0000","last_fullsized":"2024-03-29T10:10:58.662667+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:06.388556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.659117+0000","last_change":"2024-03-29T10:10:58.656125+0000","last_active":"2024-03-29T10:10:58.659117+0000","last_peered":"2024-03-29T10:10:58.659117+0000","last_clean":"2024-03-29T10:10:58.659117+0000","last_became_active":"2024-03-29T10:10:58.655492+0000","last_became_peered":"2024-03-29T10:10:58.655492+0000","last_unstale":"2024-03-29T10:10:58.659117+0000","last_undegraded":"2024-03-29T10:10:58.659117+0000","last_fullsized":"2024-03-29T10:10:58.659117+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:54:00.175790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.664153+0000","last_change":"2024-03-29T10:10:58.663049+0000","last_active":"2024-03-29T10:10:58.664153+0000","last_peered":"2024-03-29T10:10:58.664153+0000","last_clean":"2024-03-29T10:10:58.664153+0000","last_became_active":"2024-03-29T10:10:58.662733+0000","last_became_peered":"2024-03-29T10:10:58.662733+0000","last_unstale":"2024-03-29T10:10:58.664153+0000","last_undegraded":"2024-03-29T10:10:58.664153+0000","last_fullsized":"2024-03-29T10:10:58.664153+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:16.754618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.661556+0000","last_change":"2024-03-29T10:10:58.647492+0000","last_active":"2024-03-29T10:10:58.661556+0000","last_peered":"2024-03-29T10:10:58.661556+0000","last_clean":"2024-03-29T10:10:58.661556+0000","last_became_active":"2024-03-29T10:10:58.647235+0000","last_became_peered":"2024-03-29T10:10:58.647235+0000","last_unstale":"2024-03-29T10:10:58.661556+0000","last_undegraded":"2024-03-29T10:10:58.661556+0000","last_fullsized":"2024-03-29T10:10:58.661556+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:05:54.869449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.672766+0000","last_change":"2024-03-29T10:10:58.664720+0000","last_active":"2024-03-29T10:10:58.672766+0000","last_peered":"2024-03-29T10:10:58.672766+0000","last_clean":"2024-03-29T10:10:58.672766+0000","last_became_active":"2024-03-29T10:10:58.664450+0000","last_became_peered":"2024-03-29T10:10:58.664450+0000","last_unstale":"2024-03-29T10:10:58.672766+0000","last_undegraded":"2024-03-29T10:10:58.672766+0000","last_fullsized":"2024-03-29T10:10:58.672766+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:50.527149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.671890+0000","last_change":"2024-03-29T10:10:58.667073+0000","last_active":"2024-03-29T10:10:58.671890+0000","last_peered":"2024-03-29T10:10:58.671890+0000","last_clean":"2024-03-29T10:10:58.671890+0000","last_became_active":"2024-03-29T10:10:58.666735+0000","last_became_peered":"2024-03-29T10:10:58.666735+0000","last_unstale":"2024-03-29T10:10:58.671890+0000","last_undegraded":"2024-03-29T10:10:58.671890+0000","last_fullsized":"2024-03-29T10:10:58.671890+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:16.050232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.666209+0000","last_change":"2024-03-29T10:10:58.654617+0000","last_active":"2024-03-29T10:10:58.666209+0000","last_peered":"2024-03-29T10:10:58.666209+0000","last_clean":"2024-03-29T10:10:58.666209+0000","last_became_active":"2024-03-29T10:10:58.654337+0000","last_became_peered":"2024-03-29T10:10:58.654337+0000","last_unstale":"2024-03-29T10:10:58.666209+0000","last_undegraded":"2024-03-29T10:10:58.666209+0000","last_fullsized":"2024-03-29T10:10:58.666209+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:31.763079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.665070+0000","last_change":"2024-03-29T10:10:58.658771+0000","last_active":"2024-03-29T10:10:58.665070+0000","last_peered":"2024-03-29T10:10:58.665070+0000","last_clean":"2024-03-29T10:10:58.665070+0000","last_became_active":"2024-03-29T10:10:58.658509+0000","last_became_peered":"2024-03-29T10:10:58.658509+0000","last_unstale":"2024-03-29T10:10:58.665070+0000","last_undegraded":"2024-03-29T10:10:58.665070+0000","last_fullsized":"2024-03-29T10:10:58.665070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:28:14.453536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669940+0000","last_change":"2024-03-29T10:10:58.664754+0000","last_active":"2024-03-29T10:10:58.669940+0000","last_peered":"2024-03-29T10:10:58.669940+0000","last_clean":"2024-03-29T10:10:58.669940+0000","last_became_active":"2024-03-29T10:10:58.663924+0000","last_became_peered":"2024-03-29T10:10:58.663924+0000","last_unstale":"2024-03-29T10:10:58.669940+0000","last_undegraded":"2024-03-29T10:10:58.669940+0000","last_fullsized":"2024-03-29T10:10:58.669940+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:50.527149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.666585+0000","last_change":"2024-03-29T10:10:58.654776+0000","last_active":"2024-03-29T10:10:58.666585+0000","last_peered":"2024-03-29T10:10:58.666585+0000","last_clean":"2024-03-29T10:10:58.666585+0000","last_became_active":"2024-03-29T10:10:58.654357+0000","last_became_peered":"2024-03-29T10:10:58.654357+0000","last_unstale":"2024-03-29T10:10:58.666585+0000","last_undegraded":"2024-03-29T10:10:58.666585+0000","last_fullsized":"2024-03-29T10:10:58.666585+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:40:11.624381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.678742+0000","last_change":"2024-03-29T10:10:58.673760+0000","last_active":"2024-03-29T10:10:58.678742+0000","last_peered":"2024-03-29T10:10:58.678742+0000","last_clean":"2024-03-29T10:10:58.678742+0000","last_became_active":"2024-03-29T10:10:58.667258+0000","last_became_peered":"2024-03-29T10:10:58.667258+0000","last_unstale":"2024-03-29T10:10:58.678742+0000","last_undegraded":"2024-03-29T10:10:58.678742+0000","last_fullsized":"2024-03-29T10:10:58.678742+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:33:07.658914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675138+0000","last_change":"2024-03-29T10:10:58.671196+0000","last_active":"2024-03-29T10:10:58.675138+0000","last_peered":"2024-03-29T10:10:58.675138+0000","last_clean":"2024-03-29T10:10:58.675138+0000","last_became_active":"2024-03-29T10:10:58.670860+0000","last_became_peered":"2024-03-29T10:10:58.670860+0000","last_unstale":"2024-03-29T10:10:58.675138+0000","last_undegraded":"2024-03-29T10:10:58.675138+0000","last_fullsized":"2024-03-29T10:10:58.675138+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:29:27.283488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662271+0000","last_change":"2024-03-29T10:10:58.660830+0000","last_active":"2024-03-29T10:10:58.662271+0000","last_peered":"2024-03-29T10:10:58.662271+0000","last_clean":"2024-03-29T10:10:58.662271+0000","last_became_active":"2024-03-29T10:10:58.660515+0000","last_became_peered":"2024-03-29T10:10:58.660515+0000","last_unstale":"2024-03-29T10:10:58.662271+0000","last_undegraded":"2024-03-29T10:10:58.662271+0000","last_fullsized":"2024-03-29T10:10:58.662271+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:30:01.048443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.661495+0000","last_change":"2024-03-29T10:10:58.654639+0000","last_active":"2024-03-29T10:10:58.661495+0000","last_peered":"2024-03-29T10:10:58.661495+0000","last_clean":"2024-03-29T10:10:58.661495+0000","last_became_active":"2024-03-29T10:10:58.654253+0000","last_became_peered":"2024-03-29T10:10:58.654253+0000","last_unstale":"2024-03-29T10:10:58.661495+0000","last_undegraded":"2024-03-29T10:10:58.661495+0000","last_fullsized":"2024-03-29T10:10:58.661495+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:13.760911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.676230+0000","last_change":"2024-03-29T10:10:58.666480+0000","last_active":"2024-03-29T10:10:58.676230+0000","last_peered":"2024-03-29T10:10:58.676230+0000","last_clean":"2024-03-29T10:10:58.676230+0000","last_became_active":"2024-03-29T10:10:58.665590+0000","last_became_peered":"2024-03-29T10:10:58.665590+0000","last_unstale":"2024-03-29T10:10:58.676230+0000","last_undegraded":"2024-03-29T10:10:58.676230+0000","last_fullsized":"2024-03-29T10:10:58.676230+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:32.463011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.678691+0000","last_change":"2024-03-29T10:10:58.676114+0000","last_active":"2024-03-29T10:10:58.678691+0000","last_peered":"2024-03-29T10:10:58.678691+0000","last_clean":"2024-03-29T10:10:58.678691+0000","last_became_active":"2024-03-29T10:10:58.675795+0000","last_became_peered":"2024-03-29T10:10:58.675795+0000","last_unstale":"2024-03-29T10:10:58.678691+0000","last_undegraded":"2024-03-29T10:10:58.678691+0000","last_fullsized":"2024-03-29T10:10:58.678691+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:50:56.622833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662068+0000","last_change":"2024-03-29T10:10:58.658334+0000","last_active":"2024-03-29T10:10:58.662068+0000","last_peered":"2024-03-29T10:10:58.662068+0000","last_clean":"2024-03-29T10:10:58.662068+0000","last_became_active":"2024-03-29T10:10:58.657792+0000","last_became_peered":"2024-03-29T10:10:58.657792+0000","last_unstale":"2024-03-29T10:10:58.662068+0000","last_undegraded":"2024-03-29T10:10:58.662068+0000","last_fullsized":"2024-03-29T10:10:58.662068+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:05.682812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.661696+0000","last_change":"2024-03-29T10:10:58.656325+0000","last_active":"2024-03-29T10:10:58.661696+0000","last_peered":"2024-03-29T10:10:58.661696+0000","last_clean":"2024-03-29T10:10:58.661696+0000","last_became_active":"2024-03-29T10:10:58.655838+0000","last_became_peered":"2024-03-29T10:10:58.655838+0000","last_unstale":"2024-03-29T10:10:58.661696+0000","last_undegraded":"2024-03-29T10:10:58.661696+0000","last_fullsized":"2024-03-29T10:10:58.661696+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:53.750734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.654258+0000","last_change":"2024-03-29T10:10:58.652425+0000","last_active":"2024-03-29T10:10:58.654258+0000","last_peered":"2024-03-29T10:10:58.654258+0000","last_clean":"2024-03-29T10:10:58.654258+0000","last_became_active":"2024-03-29T10:10:58.651963+0000","last_became_peered":"2024-03-29T10:10:58.651963+0000","last_unstale":"2024-03-29T10:10:58.654258+0000","last_undegraded":"2024-03-29T10:10:58.654258+0000","last_fullsized":"2024-03-29T10:10:58.654258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:50.527149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"107'4","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.673802+0000","last_change":"2024-03-29T10:10:58.130267+0000","last_active":"2024-03-29T10:10:58.673802+0000","last_peered":"2024-03-29T10:10:58.673802+0000","last_clean":"2024-03-29T10:10:58.673802+0000","last_became_active":"2024-03-29T10:10:58.129956+0000","last_became_peered":"2024-03-29T10:10:58.129956+0000","last_unstale":"2024-03-29T10:10:58.673802+0000","last_undegraded":"2024-03-29T10:10:58.673802+0000","last_fullsized":"2024-03-29T10:10:58.673802+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:52:17.719442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.668600+0000","last_change":"2024-03-29T10:10:58.664102+0000","last_active":"2024-03-29T10:10:58.668600+0000","last_peered":"2024-03-29T10:10:58.668600+0000","last_clean":"2024-03-29T10:10:58.668600+0000","last_became_active":"2024-03-29T10:10:58.663841+0000","last_became_peered":"2024-03-29T10:10:58.663841+0000","last_unstale":"2024-03-29T10:10:58.668600+0000","last_undegraded":"2024-03-29T10:10:58.668600+0000","last_fullsized":"2024-03-29T10:10:58.668600+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:08:37.164467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"107'1","reported_seq":11,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675465+0000","last_change":"2024-03-29T10:10:58.122547+0000","last_active":"2024-03-29T10:10:58.675465+0000","last_peered":"2024-03-29T10:10:58.675465+0000","last_clean":"2024-03-29T10:10:58.675465+0000","last_became_active":"2024-03-29T10:10:58.120320+0000","last_became_peered":"2024-03-29T10:10:58.120320+0000","last_unstale":"2024-03-29T10:10:58.675465+0000","last_undegraded":"2024-03-29T10:10:58.675465+0000","last_fullsized":"2024-03-29T10:10:58.675465+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:10.070572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.661721+0000","last_change":"2024-03-29T10:10:58.656445+0000","last_active":"2024-03-29T10:10:58.661721+0000","last_peered":"2024-03-29T10:10:58.661721+0000","last_clean":"2024-03-29T10:10:58.661721+0000","last_became_active":"2024-03-29T10:10:58.655845+0000","last_became_peered":"2024-03-29T10:10:58.655845+0000","last_unstale":"2024-03-29T10:10:58.661721+0000","last_undegraded":"2024-03-29T10:10:58.661721+0000","last_fullsized":"2024-03-29T10:10:58.661721+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:06.388556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.680817+0000","last_change":"2024-03-29T10:10:58.130227+0000","last_active":"2024-03-29T10:10:58.680817+0000","last_peered":"2024-03-29T10:10:58.680817+0000","last_clean":"2024-03-29T10:10:58.680817+0000","last_became_active":"2024-03-29T10:10:58.129940+0000","last_became_peered":"2024-03-29T10:10:58.129940+0000","last_unstale":"2024-03-29T10:10:58.680817+0000","last_undegraded":"2024-03-29T10:10:58.680817+0000","last_fullsized":"2024-03-29T10:10:58.680817+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:15.747324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669904+0000","last_change":"2024-03-29T10:10:58.668625+0000","last_active":"2024-03-29T10:10:58.669904+0000","last_peered":"2024-03-29T10:10:58.669904+0000","last_clean":"2024-03-29T10:10:58.669904+0000","last_became_active":"2024-03-29T10:10:58.668121+0000","last_became_peered":"2024-03-29T10:10:58.668121+0000","last_unstale":"2024-03-29T10:10:58.669904+0000","last_undegraded":"2024-03-29T10:10:58.669904+0000","last_fullsized":"2024-03-29T10:10:58.669904+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:50.527149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.680947+0000","last_change":"2024-03-29T10:10:58.130656+0000","last_active":"2024-03-29T10:10:58.680947+0000","last_peered":"2024-03-29T10:10:58.680947+0000","last_clean":"2024-03-29T10:10:58.680947+0000","last_became_active":"2024-03-29T10:10:58.129435+0000","last_became_peered":"2024-03-29T10:10:58.129435+0000","last_unstale":"2024-03-29T10:10:58.680947+0000","last_undegraded":"2024-03-29T10:10:58.680947+0000","last_fullsized":"2024-03-29T10:10:58.680947+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:30:00.041149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.678711+0000","last_change":"2024-03-29T10:10:58.676273+0000","last_active":"2024-03-29T10:10:58.678711+0000","last_peered":"2024-03-29T10:10:58.678711+0000","last_clean":"2024-03-29T10:10:58.678711+0000","last_became_active":"2024-03-29T10:10:58.675815+0000","last_became_peered":"2024-03-29T10:10:58.675815+0000","last_unstale":"2024-03-29T10:10:58.678711+0000","last_undegraded":"2024-03-29T10:10:58.678711+0000","last_fullsized":"2024-03-29T10:10:58.678711+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:13:28.530296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"107'1","reported_seq":11,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.677191+0000","last_change":"2024-03-29T10:10:58.119864+0000","last_active":"2024-03-29T10:10:58.677191+0000","last_peered":"2024-03-29T10:10:58.677191+0000","last_clean":"2024-03-29T10:10:58.677191+0000","last_became_active":"2024-03-29T10:10:58.119486+0000","last_became_peered":"2024-03-29T10:10:58.119486+0000","last_unstale":"2024-03-29T10:10:58.677191+0000","last_undegraded":"2024-03-29T10:10:58.677191+0000","last_fullsized":"2024-03-29T10:10:58.677191+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:40:10.617087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.664413+0000","last_change":"2024-03-29T10:10:58.663372+0000","last_active":"2024-03-29T10:10:58.664413+0000","last_peered":"2024-03-29T10:10:58.664413+0000","last_clean":"2024-03-29T10:10:58.664413+0000","last_became_active":"2024-03-29T10:10:58.663056+0000","last_became_peered":"2024-03-29T10:10:58.663056+0000","last_unstale":"2024-03-29T10:10:58.664413+0000","last_undegraded":"2024-03-29T10:10:58.664413+0000","last_fullsized":"2024-03-29T10:10:58.664413+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:15.239902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"107'1","reported_seq":11,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675423+0000","last_change":"2024-03-29T10:10:58.119337+0000","last_active":"2024-03-29T10:10:58.675423+0000","last_peered":"2024-03-29T10:10:58.675423+0000","last_clean":"2024-03-29T10:10:58.675423+0000","last_became_active":"2024-03-29T10:10:58.119003+0000","last_became_peered":"2024-03-29T10:10:58.119003+0000","last_unstale":"2024-03-29T10:10:58.675423+0000","last_undegraded":"2024-03-29T10:10:58.675423+0000","last_fullsized":"2024-03-29T10:10:58.675423+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:36.199617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.663613+0000","last_change":"2024-03-29T10:10:58.658725+0000","last_active":"2024-03-29T10:10:58.663613+0000","last_peered":"2024-03-29T10:10:58.663613+0000","last_clean":"2024-03-29T10:10:58.663613+0000","last_became_active":"2024-03-29T10:10:58.658456+0000","last_became_peered":"2024-03-29T10:10:58.658456+0000","last_unstale":"2024-03-29T10:10:58.663613+0000","last_undegraded":"2024-03-29T10:10:58.663613+0000","last_fullsized":"2024-03-29T10:10:58.663613+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:57:08.688729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.654301+0000","last_change":"2024-03-29T10:10:58.125683+0000","last_active":"2024-03-29T10:10:58.654301+0000","last_peered":"2024-03-29T10:10:58.654301+0000","last_clean":"2024-03-29T10:10:58.654301+0000","last_became_active":"2024-03-29T10:10:58.125421+0000","last_became_peered":"2024-03-29T10:10:58.125421+0000","last_unstale":"2024-03-29T10:10:58.654301+0000","last_undegraded":"2024-03-29T10:10:58.654301+0000","last_fullsized":"2024-03-29T10:10:58.654301+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:36:13.048236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662646+0000","last_change":"2024-03-29T10:10:58.657777+0000","last_active":"2024-03-29T10:10:58.662646+0000","last_peered":"2024-03-29T10:10:58.662646+0000","last_clean":"2024-03-29T10:10:58.662646+0000","last_became_active":"2024-03-29T10:10:58.657445+0000","last_became_peered":"2024-03-29T10:10:58.657445+0000","last_unstale":"2024-03-29T10:10:58.662646+0000","last_undegraded":"2024-03-29T10:10:58.662646+0000","last_fullsized":"2024-03-29T10:10:58.662646+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:08:37.164467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.656417+0000","last_change":"2024-03-29T10:10:58.127497+0000","last_active":"2024-03-29T10:10:58.656417+0000","last_peered":"2024-03-29T10:10:58.656417+0000","last_clean":"2024-03-29T10:10:58.656417+0000","last_became_active":"2024-03-29T10:10:58.127154+0000","last_became_peered":"2024-03-29T10:10:58.127154+0000","last_unstale":"2024-03-29T10:10:58.656417+0000","last_undegraded":"2024-03-29T10:10:58.656417+0000","last_fullsized":"2024-03-29T10:10:58.656417+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:20:36.618357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675093+0000","last_change":"2024-03-29T10:10:58.670879+0000","last_active":"2024-03-29T10:10:58.675093+0000","last_peered":"2024-03-29T10:10:58.675093+0000","last_clean":"2024-03-29T10:10:58.675093+0000","last_became_active":"2024-03-29T10:10:58.670477+0000","last_became_peered":"2024-03-29T10:10:58.670477+0000","last_unstale":"2024-03-29T10:10:58.675093+0000","last_undegraded":"2024-03-29T10:10:58.675093+0000","last_fullsized":"2024-03-29T10:10:58.675093+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:32:54.224552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.642166+0000","last_change":"2024-03-29T10:10:58.127665+0000","last_active":"2024-03-29T10:10:58.642166+0000","last_peered":"2024-03-29T10:10:58.642166+0000","last_clean":"2024-03-29T10:10:58.642166+0000","last_became_active":"2024-03-29T10:10:58.127373+0000","last_became_peered":"2024-03-29T10:10:58.127373+0000","last_unstale":"2024-03-29T10:10:58.642166+0000","last_undegraded":"2024-03-29T10:10:58.642166+0000","last_fullsized":"2024-03-29T10:10:58.642166+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:17:23.187320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.650758+0000","last_change":"2024-03-29T10:10:58.648192+0000","last_active":"2024-03-29T10:10:58.650758+0000","last_peered":"2024-03-29T10:10:58.650758+0000","last_clean":"2024-03-29T10:10:58.650758+0000","last_became_active":"2024-03-29T10:10:58.647907+0000","last_became_peered":"2024-03-29T10:10:58.647907+0000","last_unstale":"2024-03-29T10:10:58.650758+0000","last_undegraded":"2024-03-29T10:10:58.650758+0000","last_fullsized":"2024-03-29T10:10:58.650758+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:31.402667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.643364+0000","last_change":"2024-03-29T10:10:58.129139+0000","last_active":"2024-03-29T10:10:58.643364+0000","last_peered":"2024-03-29T10:10:58.643364+0000","last_clean":"2024-03-29T10:10:58.643364+0000","last_became_active":"2024-03-29T10:10:58.127784+0000","last_became_peered":"2024-03-29T10:10:58.127784+0000","last_unstale":"2024-03-29T10:10:58.643364+0000","last_undegraded":"2024-03-29T10:10:58.643364+0000","last_fullsized":"2024-03-29T10:10:58.643364+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:20.219556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.677769+0000","last_change":"2024-03-29T10:10:58.670325+0000","last_active":"2024-03-29T10:10:58.677769+0000","last_peered":"2024-03-29T10:10:58.677769+0000","last_clean":"2024-03-29T10:10:58.677769+0000","last_became_active":"2024-03-29T10:10:58.670027+0000","last_became_peered":"2024-03-29T10:10:58.670027+0000","last_unstale":"2024-03-29T10:10:58.677769+0000","last_undegraded":"2024-03-29T10:10:58.677769+0000","last_fullsized":"2024-03-29T10:10:58.677769+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:43:18.124198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"107'3","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.678251+0000","last_change":"2024-03-29T10:10:58.125628+0000","last_active":"2024-03-29T10:10:58.678251+0000","last_peered":"2024-03-29T10:10:58.678251+0000","last_clean":"2024-03-29T10:10:58.678251+0000","last_became_active":"2024-03-29T10:10:58.125319+0000","last_became_peered":"2024-03-29T10:10:58.125319+0000","last_unstale":"2024-03-29T10:10:58.678251+0000","last_undegraded":"2024-03-29T10:10:58.678251+0000","last_fullsized":"2024-03-29T10:10:58.678251+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:32.039300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.668083+0000","last_change":"2024-03-29T10:10:58.666333+0000","last_active":"2024-03-29T10:10:58.668083+0000","last_peered":"2024-03-29T10:10:58.668083+0000","last_clean":"2024-03-29T10:10:58.668083+0000","last_became_active":"2024-03-29T10:10:58.666007+0000","last_became_peered":"2024-03-29T10:10:58.666007+0000","last_unstale":"2024-03-29T10:10:58.668083+0000","last_undegraded":"2024-03-29T10:10:58.668083+0000","last_fullsized":"2024-03-29T10:10:58.668083+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:34.327562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"107'6","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.680427+0000","last_change":"2024-03-29T10:10:58.130489+0000","last_active":"2024-03-29T10:10:58.680427+0000","last_peered":"2024-03-29T10:10:58.680427+0000","last_clean":"2024-03-29T10:10:58.680427+0000","last_became_active":"2024-03-29T10:10:58.130184+0000","last_became_peered":"2024-03-29T10:10:58.130184+0000","last_unstale":"2024-03-29T10:10:58.680427+0000","last_undegraded":"2024-03-29T10:10:58.680427+0000","last_fullsized":"2024-03-29T10:10:58.680427+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:15.042938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.671946+0000","last_change":"2024-03-29T10:10:58.664647+0000","last_active":"2024-03-29T10:10:58.671946+0000","last_peered":"2024-03-29T10:10:58.671946+0000","last_clean":"2024-03-29T10:10:58.671946+0000","last_became_active":"2024-03-29T10:10:58.664040+0000","last_became_peered":"2024-03-29T10:10:58.664040+0000","last_unstale":"2024-03-29T10:10:58.671946+0000","last_undegraded":"2024-03-29T10:10:58.671946+0000","last_fullsized":"2024-03-29T10:10:58.671946+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:24.188405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.679343+0000","last_change":"2024-03-29T10:10:58.131540+0000","last_active":"2024-03-29T10:10:58.679343+0000","last_peered":"2024-03-29T10:10:58.679343+0000","last_clean":"2024-03-29T10:10:58.679343+0000","last_became_active":"2024-03-29T10:10:58.129783+0000","last_became_peered":"2024-03-29T10:10:58.129783+0000","last_unstale":"2024-03-29T10:10:58.679343+0000","last_undegraded":"2024-03-29T10:10:58.679343+0000","last_fullsized":"2024-03-29T10:10:58.679343+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:35:52.111805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.668160+0000","last_change":"2024-03-29T10:10:58.665133+0000","last_active":"2024-03-29T10:10:58.668160+0000","last_peered":"2024-03-29T10:10:58.668160+0000","last_clean":"2024-03-29T10:10:58.668160+0000","last_became_active":"2024-03-29T10:10:58.664862+0000","last_became_peered":"2024-03-29T10:10:58.664862+0000","last_unstale":"2024-03-29T10:10:58.668160+0000","last_undegraded":"2024-03-29T10:10:58.668160+0000","last_fullsized":"2024-03-29T10:10:58.668160+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:08:37.164467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.653293+0000","last_change":"2024-03-29T10:10:58.125376+0000","last_active":"2024-03-29T10:10:58.653293+0000","last_peered":"2024-03-29T10:10:58.653293+0000","last_clean":"2024-03-29T10:10:58.653293+0000","last_became_active":"2024-03-29T10:10:58.125081+0000","last_became_peered":"2024-03-29T10:10:58.125081+0000","last_unstale":"2024-03-29T10:10:58.653293+0000","last_undegraded":"2024-03-29T10:10:58.653293+0000","last_fullsized":"2024-03-29T10:10:58.653293+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:53:59.168496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.677936+0000","last_change":"2024-03-29T10:10:58.672606+0000","last_active":"2024-03-29T10:10:58.677936+0000","last_peered":"2024-03-29T10:10:58.677936+0000","last_clean":"2024-03-29T10:10:58.677936+0000","last_became_active":"2024-03-29T10:10:58.672142+0000","last_became_peered":"2024-03-29T10:10:58.672142+0000","last_unstale":"2024-03-29T10:10:58.677936+0000","last_undegraded":"2024-03-29T10:10:58.677936+0000","last_fullsized":"2024-03-29T10:10:58.677936+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:33:10.033901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.640960+0000","last_change":"2024-03-29T10:10:58.127472+0000","last_active":"2024-03-29T10:10:58.640960+0000","last_peered":"2024-03-29T10:10:58.640960+0000","last_clean":"2024-03-29T10:10:58.640960+0000","last_became_active":"2024-03-29T10:10:58.126945+0000","last_became_peered":"2024-03-29T10:10:58.126945+0000","last_unstale":"2024-03-29T10:10:58.640960+0000","last_undegraded":"2024-03-29T10:10:58.640960+0000","last_fullsized":"2024-03-29T10:10:58.640960+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:30.755785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.676181+0000","last_change":"2024-03-29T10:10:58.667181+0000","last_active":"2024-03-29T10:10:58.676181+0000","last_peered":"2024-03-29T10:10:58.676181+0000","last_clean":"2024-03-29T10:10:58.676181+0000","last_became_active":"2024-03-29T10:10:58.663135+0000","last_became_peered":"2024-03-29T10:10:58.663135+0000","last_unstale":"2024-03-29T10:10:58.676181+0000","last_undegraded":"2024-03-29T10:10:58.676181+0000","last_fullsized":"2024-03-29T10:10:58.676181+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:27.747600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"107'8","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.680605+0000","last_change":"2024-03-29T10:10:58.121985+0000","last_active":"2024-03-29T10:10:58.680605+0000","last_peered":"2024-03-29T10:10:58.680605+0000","last_clean":"2024-03-29T10:10:58.680605+0000","last_became_active":"2024-03-29T10:10:58.121702+0000","last_became_peered":"2024-03-29T10:10:58.121702+0000","last_unstale":"2024-03-29T10:10:58.680605+0000","last_undegraded":"2024-03-29T10:10:58.680605+0000","last_fullsized":"2024-03-29T10:10:58.680605+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:52.743441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662351+0000","last_change":"2024-03-29T10:10:58.658104+0000","last_active":"2024-03-29T10:10:58.662351+0000","last_peered":"2024-03-29T10:10:58.662351+0000","last_clean":"2024-03-29T10:10:58.662351+0000","last_became_active":"2024-03-29T10:10:58.657810+0000","last_became_peered":"2024-03-29T10:10:58.657810+0000","last_unstale":"2024-03-29T10:10:58.662351+0000","last_undegraded":"2024-03-29T10:10:58.662351+0000","last_fullsized":"2024-03-29T10:10:58.662351+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:24.188405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"107'1","reported_seq":11,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669993+0000","last_change":"2024-03-29T10:10:58.130426+0000","last_active":"2024-03-29T10:10:58.669993+0000","last_peered":"2024-03-29T10:10:58.669993+0000","last_clean":"2024-03-29T10:10:58.669993+0000","last_became_active":"2024-03-29T10:10:58.129526+0000","last_became_peered":"2024-03-29T10:10:58.129526+0000","last_unstale":"2024-03-29T10:10:58.669993+0000","last_undegraded":"2024-03-29T10:10:58.669993+0000","last_fullsized":"2024-03-29T10:10:58.669993+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:15.723444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.667839+0000","last_change":"2024-03-29T10:10:58.656983+0000","last_active":"2024-03-29T10:10:58.667839+0000","last_peered":"2024-03-29T10:10:58.667839+0000","last_clean":"2024-03-29T10:10:58.667839+0000","last_became_active":"2024-03-29T10:10:58.656698+0000","last_became_peered":"2024-03-29T10:10:58.656698+0000","last_unstale":"2024-03-29T10:10:58.667839+0000","last_undegraded":"2024-03-29T10:10:58.667839+0000","last_fullsized":"2024-03-29T10:10:58.667839+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:54:00.175790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.679181+0000","last_change":"2024-03-29T10:10:58.129375+0000","last_active":"2024-03-29T10:10:58.679181+0000","last_peered":"2024-03-29T10:10:58.679181+0000","last_clean":"2024-03-29T10:10:58.679181+0000","last_became_active":"2024-03-29T10:10:58.129087+0000","last_became_peered":"2024-03-29T10:10:58.129087+0000","last_unstale":"2024-03-29T10:10:58.679181+0000","last_undegraded":"2024-03-29T10:10:58.679181+0000","last_fullsized":"2024-03-29T10:10:58.679181+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:30.395373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.668105+0000","last_change":"2024-03-29T10:10:58.666916+0000","last_active":"2024-03-29T10:10:58.668105+0000","last_peered":"2024-03-29T10:10:58.668105+0000","last_clean":"2024-03-29T10:10:58.668105+0000","last_became_active":"2024-03-29T10:10:58.666616+0000","last_became_peered":"2024-03-29T10:10:58.666616+0000","last_unstale":"2024-03-29T10:10:58.668105+0000","last_undegraded":"2024-03-29T10:10:58.668105+0000","last_fullsized":"2024-03-29T10:10:58.668105+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:06:03.899109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.656883+0000","last_change":"2024-03-29T10:10:58.125217+0000","last_active":"2024-03-29T10:10:58.656883+0000","last_peered":"2024-03-29T10:10:58.656883+0000","last_clean":"2024-03-29T10:10:58.656883+0000","last_became_active":"2024-03-29T10:10:58.124956+0000","last_became_peered":"2024-03-29T10:10:58.124956+0000","last_unstale":"2024-03-29T10:10:58.656883+0000","last_undegraded":"2024-03-29T10:10:58.656883+0000","last_fullsized":"2024-03-29T10:10:58.656883+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:22.422695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.666880+0000","last_change":"2024-03-29T10:10:58.664513+0000","last_active":"2024-03-29T10:10:58.666880+0000","last_peered":"2024-03-29T10:10:58.666880+0000","last_clean":"2024-03-29T10:10:58.666880+0000","last_became_active":"2024-03-29T10:10:58.664258+0000","last_became_peered":"2024-03-29T10:10:58.664258+0000","last_unstale":"2024-03-29T10:10:58.666880+0000","last_undegraded":"2024-03-29T10:10:58.666880+0000","last_fullsized":"2024-03-29T10:10:58.666880+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:11.077866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.674607+0000","last_change":"2024-03-29T10:10:58.133969+0000","last_active":"2024-03-29T10:10:58.674607+0000","last_peered":"2024-03-29T10:10:58.674607+0000","last_clean":"2024-03-29T10:10:58.674607+0000","last_became_active":"2024-03-29T10:10:58.132913+0000","last_became_peered":"2024-03-29T10:10:58.132913+0000","last_unstale":"2024-03-29T10:10:58.674607+0000","last_undegraded":"2024-03-29T10:10:58.674607+0000","last_fullsized":"2024-03-29T10:10:58.674607+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:49.519856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.657046+0000","last_change":"2024-03-29T10:10:58.651563+0000","last_active":"2024-03-29T10:10:58.657046+0000","last_peered":"2024-03-29T10:10:58.657046+0000","last_clean":"2024-03-29T10:10:58.657046+0000","last_became_active":"2024-03-29T10:10:58.651272+0000","last_became_peered":"2024-03-29T10:10:58.651272+0000","last_unstale":"2024-03-29T10:10:58.657046+0000","last_undegraded":"2024-03-29T10:10:58.657046+0000","last_fullsized":"2024-03-29T10:10:58.657046+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:52:18.726736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.641950+0000","last_change":"2024-03-29T10:10:58.131438+0000","last_active":"2024-03-29T10:10:58.641950+0000","last_peered":"2024-03-29T10:10:58.641950+0000","last_clean":"2024-03-29T10:10:58.641950+0000","last_became_active":"2024-03-29T10:10:58.131085+0000","last_became_peered":"2024-03-29T10:10:58.131085+0000","last_unstale":"2024-03-29T10:10:58.641950+0000","last_undegraded":"2024-03-29T10:10:58.641950+0000","last_fullsized":"2024-03-29T10:10:58.641950+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:26.740306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.660474+0000","last_change":"2024-03-29T10:10:58.646342+0000","last_active":"2024-03-29T10:10:58.660474+0000","last_peered":"2024-03-29T10:10:58.660474+0000","last_clean":"2024-03-29T10:10:58.660474+0000","last_became_active":"2024-03-29T10:10:58.646123+0000","last_became_peered":"2024-03-29T10:10:58.646123+0000","last_unstale":"2024-03-29T10:10:58.660474+0000","last_undegraded":"2024-03-29T10:10:58.660474+0000","last_fullsized":"2024-03-29T10:10:58.660474+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:37.206911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.641418+0000","last_change":"2024-03-29T10:10:58.127054+0000","last_active":"2024-03-29T10:10:58.641418+0000","last_peered":"2024-03-29T10:10:58.641418+0000","last_clean":"2024-03-29T10:10:58.641418+0000","last_became_active":"2024-03-29T10:10:58.125738+0000","last_became_peered":"2024-03-29T10:10:58.125738+0000","last_unstale":"2024-03-29T10:10:58.641418+0000","last_undegraded":"2024-03-29T10:10:58.641418+0000","last_fullsized":"2024-03-29T10:10:58.641418+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:47:46.853250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.658909+0000","last_change":"2024-03-29T10:10:58.655398+0000","last_active":"2024-03-29T10:10:58.658909+0000","last_peered":"2024-03-29T10:10:58.658909+0000","last_clean":"2024-03-29T10:10:58.658909+0000","last_became_active":"2024-03-29T10:10:58.655074+0000","last_became_peered":"2024-03-29T10:10:58.655074+0000","last_unstale":"2024-03-29T10:10:58.658909+0000","last_undegraded":"2024-03-29T10:10:58.658909+0000","last_fullsized":"2024-03-29T10:10:58.658909+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:28.681653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.653234+0000","last_change":"2024-03-29T10:10:58.132443+0000","last_active":"2024-03-29T10:10:58.653234+0000","last_peered":"2024-03-29T10:10:58.653234+0000","last_clean":"2024-03-29T10:10:58.653234+0000","last_became_active":"2024-03-29T10:10:58.131885+0000","last_became_peered":"2024-03-29T10:10:58.131885+0000","last_unstale":"2024-03-29T10:10:58.653234+0000","last_undegraded":"2024-03-29T10:10:58.653234+0000","last_fullsized":"2024-03-29T10:10:58.653234+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:14.232608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.652160+0000","last_change":"2024-03-29T10:10:58.650455+0000","last_active":"2024-03-29T10:10:58.652160+0000","last_peered":"2024-03-29T10:10:58.652160+0000","last_clean":"2024-03-29T10:10:58.652160+0000","last_became_active":"2024-03-29T10:10:58.649269+0000","last_became_peered":"2024-03-29T10:10:58.649269+0000","last_unstale":"2024-03-29T10:10:58.652160+0000","last_undegraded":"2024-03-29T10:10:58.652160+0000","last_fullsized":"2024-03-29T10:10:58.652160+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:33.106330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.678806+0000","last_change":"2024-03-29T10:10:58.126002+0000","last_active":"2024-03-29T10:10:58.678806+0000","last_peered":"2024-03-29T10:10:58.678806+0000","last_clean":"2024-03-29T10:10:58.678806+0000","last_became_active":"2024-03-29T10:10:58.125619+0000","last_became_peered":"2024-03-29T10:10:58.125619+0000","last_unstale":"2024-03-29T10:10:58.678806+0000","last_undegraded":"2024-03-29T10:10:58.678806+0000","last_fullsized":"2024-03-29T10:10:58.678806+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:31:24.045982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669770+0000","last_change":"2024-03-29T10:10:58.667794+0000","last_active":"2024-03-29T10:10:58.669770+0000","last_peered":"2024-03-29T10:10:58.669770+0000","last_clean":"2024-03-29T10:10:58.669770+0000","last_became_active":"2024-03-29T10:10:58.667461+0000","last_became_peered":"2024-03-29T10:10:58.667461+0000","last_unstale":"2024-03-29T10:10:58.669770+0000","last_undegraded":"2024-03-29T10:10:58.669770+0000","last_fullsized":"2024-03-29T10:10:58.669770+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:43:18.124198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.660040+0000","last_change":"2024-03-29T10:10:58.132739+0000","last_active":"2024-03-29T10:10:58.660040+0000","last_peered":"2024-03-29T10:10:58.660040+0000","last_clean":"2024-03-29T10:10:58.660040+0000","last_became_active":"2024-03-29T10:10:58.130371+0000","last_became_peered":"2024-03-29T10:10:58.130371+0000","last_unstale":"2024-03-29T10:10:58.660040+0000","last_undegraded":"2024-03-29T10:10:58.660040+0000","last_fullsized":"2024-03-29T10:10:58.660040+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:22:26.419714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.665157+0000","last_change":"2024-03-29T10:10:58.662756+0000","last_active":"2024-03-29T10:10:58.665157+0000","last_peered":"2024-03-29T10:10:58.665157+0000","last_clean":"2024-03-29T10:10:58.665157+0000","last_became_active":"2024-03-29T10:10:58.662372+0000","last_became_peered":"2024-03-29T10:10:58.662372+0000","last_unstale":"2024-03-29T10:10:58.665157+0000","last_undegraded":"2024-03-29T10:10:58.665157+0000","last_fullsized":"2024-03-29T10:10:58.665157+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:37:57.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.641268+0000","last_change":"2024-03-29T10:10:58.131798+0000","last_active":"2024-03-29T10:10:58.641268+0000","last_peered":"2024-03-29T10:10:58.641268+0000","last_clean":"2024-03-29T10:10:58.641268+0000","last_became_active":"2024-03-29T10:10:58.131481+0000","last_became_peered":"2024-03-29T10:10:58.131481+0000","last_unstale":"2024-03-29T10:10:58.641268+0000","last_undegraded":"2024-03-29T10:10:58.641268+0000","last_fullsized":"2024-03-29T10:10:58.641268+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:27.674359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662251+0000","last_change":"2024-03-29T10:10:58.659207+0000","last_active":"2024-03-29T10:10:58.662251+0000","last_peered":"2024-03-29T10:10:58.662251+0000","last_clean":"2024-03-29T10:10:58.662251+0000","last_became_active":"2024-03-29T10:10:58.658931+0000","last_became_peered":"2024-03-29T10:10:58.658931+0000","last_unstale":"2024-03-29T10:10:58.662251+0000","last_undegraded":"2024-03-29T10:10:58.662251+0000","last_fullsized":"2024-03-29T10:10:58.662251+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:53.750734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.643154+0000","last_change":"2024-03-29T10:10:58.130295+0000","last_active":"2024-03-29T10:10:58.643154+0000","last_peered":"2024-03-29T10:10:58.643154+0000","last_clean":"2024-03-29T10:10:58.643154+0000","last_became_active":"2024-03-29T10:10:58.129993+0000","last_became_peered":"2024-03-29T10:10:58.129993+0000","last_unstale":"2024-03-29T10:10:58.643154+0000","last_undegraded":"2024-03-29T10:10:58.643154+0000","last_fullsized":"2024-03-29T10:10:58.643154+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:05:53.862156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.657418+0000","last_change":"2024-03-29T10:10:58.653696+0000","last_active":"2024-03-29T10:10:58.657418+0000","last_peered":"2024-03-29T10:10:58.657418+0000","last_clean":"2024-03-29T10:10:58.657418+0000","last_became_active":"2024-03-29T10:10:58.652283+0000","last_became_peered":"2024-03-29T10:10:58.652283+0000","last_unstale":"2024-03-29T10:10:58.657418+0000","last_undegraded":"2024-03-29T10:10:58.657418+0000","last_fullsized":"2024-03-29T10:10:58.657418+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:21.226850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.655621+0000","last_change":"2024-03-29T10:10:58.124487+0000","last_active":"2024-03-29T10:10:58.655621+0000","last_peered":"2024-03-29T10:10:58.655621+0000","last_clean":"2024-03-29T10:10:58.655621+0000","last_became_active":"2024-03-29T10:10:58.124222+0000","last_became_peered":"2024-03-29T10:10:58.124222+0000","last_unstale":"2024-03-29T10:10:58.655621+0000","last_undegraded":"2024-03-29T10:10:58.655621+0000","last_fullsized":"2024-03-29T10:10:58.655621+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:06:02.891816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669295+0000","last_change":"2024-03-29T10:10:58.659988+0000","last_active":"2024-03-29T10:10:58.669295+0000","last_peered":"2024-03-29T10:10:58.669295+0000","last_clean":"2024-03-29T10:10:58.669295+0000","last_became_active":"2024-03-29T10:10:58.658255+0000","last_became_peered":"2024-03-29T10:10:58.658255+0000","last_unstale":"2024-03-29T10:10:58.669295+0000","last_undegraded":"2024-03-29T10:10:58.669295+0000","last_fullsized":"2024-03-29T10:10:58.669295+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:13.760911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.647426+0000","last_change":"2024-03-29T10:10:58.125726+0000","last_active":"2024-03-29T10:10:58.647426+0000","last_peered":"2024-03-29T10:10:58.647426+0000","last_clean":"2024-03-29T10:10:58.647426+0000","last_became_active":"2024-03-29T10:10:58.125464+0000","last_became_peered":"2024-03-29T10:10:58.125464+0000","last_unstale":"2024-03-29T10:10:58.647426+0000","last_undegraded":"2024-03-29T10:10:58.647426+0000","last_fullsized":"2024-03-29T10:10:58.647426+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:36.427259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675106+0000","last_change":"2024-03-29T10:10:58.671053+0000","last_active":"2024-03-29T10:10:58.675106+0000","last_peered":"2024-03-29T10:10:58.675106+0000","last_clean":"2024-03-29T10:10:58.675106+0000","last_became_active":"2024-03-29T10:10:58.670765+0000","last_became_peered":"2024-03-29T10:10:58.670765+0000","last_unstale":"2024-03-29T10:10:58.675106+0000","last_undegraded":"2024-03-29T10:10:58.675106+0000","last_fullsized":"2024-03-29T10:10:58.675106+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:06.388556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.641944+0000","last_change":"2024-03-29T10:10:58.126774+0000","last_active":"2024-03-29T10:10:58.641944+0000","last_peered":"2024-03-29T10:10:58.641944+0000","last_clean":"2024-03-29T10:10:58.641944+0000","last_became_active":"2024-03-29T10:10:58.126502+0000","last_became_peered":"2024-03-29T10:10:58.126502+0000","last_unstale":"2024-03-29T10:10:58.641944+0000","last_undegraded":"2024-03-29T10:10:58.641944+0000","last_fullsized":"2024-03-29T10:10:58.641944+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:36.199617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.653369+0000","last_change":"2024-03-29T10:10:58.648665+0000","last_active":"2024-03-29T10:10:58.653369+0000","last_peered":"2024-03-29T10:10:58.653369+0000","last_clean":"2024-03-29T10:10:58.653369+0000","last_became_active":"2024-03-29T10:10:58.648405+0000","last_became_peered":"2024-03-29T10:10:58.648405+0000","last_unstale":"2024-03-29T10:10:58.653369+0000","last_undegraded":"2024-03-29T10:10:58.653369+0000","last_fullsized":"2024-03-29T10:10:58.653369+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:15.239902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.647197+0000","last_change":"2024-03-29T10:10:58.123574+0000","last_active":"2024-03-29T10:10:58.647197+0000","last_peered":"2024-03-29T10:10:58.647197+0000","last_clean":"2024-03-29T10:10:58.647197+0000","last_became_active":"2024-03-29T10:10:58.123296+0000","last_became_peered":"2024-03-29T10:10:58.123296+0000","last_unstale":"2024-03-29T10:10:58.647197+0000","last_undegraded":"2024-03-29T10:10:58.647197+0000","last_fullsized":"2024-03-29T10:10:58.647197+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:30.755785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662708+0000","last_change":"2024-03-29T10:10:58.658470+0000","last_active":"2024-03-29T10:10:58.662708+0000","last_peered":"2024-03-29T10:10:58.662708+0000","last_clean":"2024-03-29T10:10:58.662708+0000","last_became_active":"2024-03-29T10:10:58.658237+0000","last_became_peered":"2024-03-29T10:10:58.658237+0000","last_unstale":"2024-03-29T10:10:58.662708+0000","last_undegraded":"2024-03-29T10:10:58.662708+0000","last_fullsized":"2024-03-29T10:10:58.662708+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:03:25.171304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"107'1","reported_seq":11,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.679216+0000","last_change":"2024-03-29T10:10:58.130206+0000","last_active":"2024-03-29T10:10:58.679216+0000","last_peered":"2024-03-29T10:10:58.679216+0000","last_clean":"2024-03-29T10:10:58.679216+0000","last_became_active":"2024-03-29T10:10:58.129948+0000","last_became_peered":"2024-03-29T10:10:58.129948+0000","last_unstale":"2024-03-29T10:10:58.679216+0000","last_undegraded":"2024-03-29T10:10:58.679216+0000","last_fullsized":"2024-03-29T10:10:58.679216+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:30.395373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"97'201","reported_seq":544,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.121556+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:10:58.121556+0000","last_peered":"2024-03-29T10:10:58.121556+0000","last_clean":"2024-03-29T10:10:58.121556+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:10:58.121556+0000","last_undegraded":"2024-03-29T10:10:58.121556+0000","last_fullsized":"2024-03-29T10:10:58.121556+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":311,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":311,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":1912832,"data_stored":1214462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.68100000000000005,"15min":0.71799999999999997},"min":{"1min":0.47899999999999998,"5min":0.44600000000000001,"15min":0.32500000000000001},"max":{"1min":0.82199999999999995,"5min":0.98699999999999999,"15min":1.0109999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68300000000000005,"15min":0.71599999999999997},"min":{"1min":0.52700000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.60699999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.70599999999999996,"15min":0.72599999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.372},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.746}]},{"osd":2,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.72099999999999997,"15min":0.74099999999999999},"min":{"1min":0.501,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73299999999999998,"15min":0.754},"min":{"1min":0.49299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.70099999999999996,"15min":0.73799999999999999},"min":{"1min":0.48499999999999999,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.745,"15min":0.77800000000000002},"min":{"1min":0.52300000000000002,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":0.91200000000000003,"5min":1.129,"15min":1.129},"last":0.57099999999999995}]},{"osd":4,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.40500000000000003,"15min":0.29199999999999998},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.441},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.63800000000000001,"15min":0.63500000000000001},"min":{"1min":0.42899999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.505}]},{"osd":5,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.43099999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.50800000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.72199999999999998}]},{"osd":6,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.49199999999999999},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.71199999999999997,"15min":0.70299999999999996},"min":{"1min":0.51100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.64000000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1323008,"data_stored":624628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70499999999999996,"15min":0.70699999999999996},"min":{"1min":0.436,"5min":0.42899999999999999,"15min":0.34699999999999998},"max":{"1min":0.92000000000000004,"5min":2.7770000000000001,"15min":2.7770000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.72399999999999998},"min":{"1min":0.46400000000000002,"5min":0.435,"15min":0.42499999999999999},"max":{"1min":0.83799999999999997,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.65400000000000003}]},{"osd":1,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":0.70699999999999996},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.81799999999999995,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.49199999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.80600000000000005,"5min":2.6709999999999998,"15min":2.6709999999999998},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.67200000000000004,"15min":0.71099999999999997},"min":{"1min":0.45100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":1.256,"15min":1.256},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.71799999999999997,"15min":0.74299999999999999},"min":{"1min":0.48199999999999998,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":0.88100000000000001,"5min":1.454,"15min":1.454},"last":0.69999999999999996}]},{"osd":3,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.69499999999999995,"15min":0.74199999999999999},"min":{"1min":0.496,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81699999999999995,"5min":1.131,"15min":1.145},"last":0.496},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.76700000000000002,"15min":0.76800000000000002},"min":{"1min":0.60999999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85099999999999998,"5min":2.718,"15min":2.718},"last":0.624}]},{"osd":4,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.41799999999999998,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":0.88200000000000001,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.46999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93799999999999994,"5min":2.544,"15min":2.544},"last":0.58799999999999997}]},{"osd":5,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67300000000000004,"15min":0.65800000000000003},"min":{"1min":0.40300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.91800000000000004,"5min":3.2280000000000002,"15min":3.2280000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72299999999999998,"15min":0.68899999999999995},"min":{"1min":0.45600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89100000000000001,"5min":2.835,"15min":2.835},"last":0.67000000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.755,"15min":0.753},"min":{"1min":0.499,"5min":0.48899999999999999,"15min":0.45100000000000001},"max":{"1min":0.98799999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72799999999999998,"15min":0.72199999999999998},"min":{"1min":0.55200000000000005,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.96099999999999997,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.64100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575042,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50468,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321372,"statfs":{"total":96636764160,"available":96585084928,"internally_reserved":0,"allocated":1327104,"data_stored":624218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.33700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.79100000000000004,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65600000000000003,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.34399999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.40000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.222,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.438},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.39100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56499999999999995,"15min":0.58399999999999996},"min":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.90800000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.026,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.747,"15min":0.76200000000000001},"min":{"1min":0.57499999999999996,"5min":0.497,"15min":0.497},"max":{"1min":1.2709999999999999,"5min":1.617,"15min":1.617},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.75,"15min":0.77100000000000002},"min":{"1min":0.54900000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0109999999999999,"5min":1.1419999999999999,"15min":1.155},"last":0.77900000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76700000000000002,"15min":0.78600000000000003},"min":{"1min":0.56799999999999995,"5min":0.56699999999999995,"15min":0.51800000000000002},"max":{"1min":0.99299999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.55800000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3140000000000001,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.69199999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.76100000000000001,"15min":0.79000000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.99399999999999999,"5min":1.0840000000000001,"15min":1.236},"last":0.746},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.78400000000000003,"15min":0.81000000000000005},"min":{"1min":0.61299999999999999,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1579999999999999,"5min":1.468,"15min":1.468},"last":0.65100000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.81100000000000005,"15min":0.83399999999999996},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.1319999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80200000000000005,"15min":0.82999999999999996},"min":{"1min":0.61699999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.4530000000000001,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.66000000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575044,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":1908736,"data_stored":1213407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.433,"15min":0.44900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.23999999999999999},"max":{"1min":0.58299999999999996,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.251},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.50700000000000001},"min":{"1min":0.31,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.86699999999999999,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.39300000000000002}]},{"osd":2,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52700000000000002,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73399999999999999,"5min":0.80700000000000005,"15min":1.0069999999999999},"last":0.439},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.45800000000000002,"15min":0.495},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.97199999999999998},"last":0.41499999999999998}]},{"osd":3,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.54400000000000004},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":1.077},"last":0.373},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57999999999999996,"15min":0.60899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73699999999999999,"5min":0.83499999999999996,"15min":1.0549999999999999},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.747,"15min":0.79000000000000004},"min":{"1min":0.54600000000000004,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":1.0409999999999999,"5min":1.625,"15min":1.625},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.76300000000000001,"15min":0.81499999999999995},"min":{"1min":0.627,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":0.95799999999999996,"5min":0.97699999999999998,"15min":1.167},"last":0.69999999999999996}]},{"osd":5,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69999999999999996,"15min":0.748},"min":{"1min":0.48799999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.137},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.751,"15min":0.79500000000000004},"min":{"1min":0.57599999999999996,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.64400000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.81200000000000006},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.999,"5min":1.0640000000000001,"15min":1.228},"last":0.749},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.79100000000000004,"15min":0.83199999999999996},"min":{"1min":0.53400000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.077,"5min":1.601,"15min":1.601},"last":0.57199999999999995}]},{"osd":7,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74199999999999999,"15min":0.79000000000000004},"min":{"1min":0.59799999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96199999999999997,"5min":1.004,"15min":1.1890000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.78800000000000003,"15min":0.83099999999999996},"min":{"1min":0.59799999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.62}]}]},{"osd":2,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40596,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331244,"statfs":{"total":96636764160,"available":96595193856,"internally_reserved":0,"allocated":1318912,"data_stored":624606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.43099999999999999,"15min":0.44400000000000001},"min":{"1min":0.30199999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.59499999999999997,"5min":0.746,"15min":0.86099999999999999},"last":0.374},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.50900000000000001},"min":{"1min":0.33200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69299999999999995,"5min":0.73699999999999999,"15min":0.82399999999999995},"last":0.39700000000000002}]},{"osd":1,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48199999999999998,"15min":0.47399999999999998},"min":{"1min":0.315,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.63600000000000001,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54200000000000004,"15min":0.55200000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32700000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":1.032},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70599999999999996,"5min":0.749,"15min":0.99099999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53800000000000003,"15min":0.54500000000000004},"min":{"1min":0.40200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82699999999999996,"5min":1.516,"15min":1.516},"last":0.42099999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.755,"15min":0.77900000000000003},"min":{"1min":0.51100000000000001,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":1.125,"5min":1.125,"15min":1.3520000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.75800000000000001,"15min":0.80000000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53100000000000003},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.85399999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76900000000000002,"15min":0.80100000000000005},"min":{"1min":0.58299999999999996,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.2589999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.752,"15min":0.79700000000000004},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.115,"5min":1.115,"15min":1.2250000000000001},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77900000000000003,"15min":0.81200000000000006},"min":{"1min":0.56100000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.2350000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.79600000000000004,"15min":0.83299999999999996},"min":{"1min":0.58599999999999997,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0860000000000001,"5min":1.1120000000000001,"15min":1.3560000000000001},"last":0.83899999999999997}]},{"osd":7,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.82999999999999996,"15min":0.85899999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.53400000000000003},"max":{"1min":1.258,"5min":1.258,"15min":1.4490000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.83199999999999996,"15min":0.874},"min":{"1min":0.60899999999999999,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.226,"5min":1.226,"15min":1.4199999999999999},"last":0.79300000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575043,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":1302528,"data_stored":617249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.441,"15min":0.45100000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26000000000000001},"max":{"1min":0.68200000000000005,"5min":0.75600000000000001,"15min":0.81299999999999994},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.373,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69699999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.40600000000000003}]},{"osd":1,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.46500000000000002,"15min":0.48399999999999999},"min":{"1min":0.313,"5min":0.30499999999999999,"15min":0.29799999999999999},"max":{"1min":0.51800000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.372},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.314},"max":{"1min":0.82699999999999996,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48899999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54200000000000004,"15min":0.55700000000000005},"min":{"1min":0.40600000000000003,"5min":0.373,"15min":0.34200000000000003},"max":{"1min":0.75800000000000001,"5min":0.83999999999999997,"15min":0.98299999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60199999999999998,"15min":0.625},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.86899999999999999,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.46700000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.76200000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.034,"5min":1.0940000000000001,"15min":1.288},"last":0.998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.77200000000000002},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.46},"last":1.093}]},{"osd":5,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.748,"15min":0.78900000000000003},"min":{"1min":0.53800000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95199999999999996,"5min":1.075,"15min":1.3580000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.753,"15min":0.79300000000000004},"min":{"1min":0.50700000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.429},"last":1.1379999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77300000000000002,"15min":0.80600000000000005},"min":{"1min":0.57799999999999996,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.085,"5min":1.101,"15min":1.4730000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77700000000000002,"15min":0.81299999999999994},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.056,"5min":1.056,"15min":1.3999999999999999},"last":1.0309999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79700000000000004,"15min":0.82999999999999996},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.54300000000000004},"max":{"1min":1.103,"5min":1.135,"15min":1.5029999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":0.83199999999999996},"min":{"1min":0.63700000000000001,"5min":0.58899999999999997,"15min":0.56200000000000006},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.345},"last":0.80800000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575044,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58648,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313192,"statfs":{"total":96636764160,"available":96576708608,"internally_reserved":0,"allocated":1314816,"data_stored":624024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65700000000000003,"15min":0.69199999999999995},"min":{"1min":0.35399999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.84399999999999997,"5min":0.91400000000000003,"15min":1.417},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.67300000000000004,"15min":0.69899999999999995},"min":{"1min":0.434,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.93200000000000005,"5min":0.93899999999999995,"15min":1.165},"last":0.59399999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.66200000000000003,"15min":0.69999999999999996},"min":{"1min":0.48099999999999998,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.89300000000000002,"5min":0.93700000000000006,"15min":1.014},"last":0.628},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67000000000000004,"15min":0.70699999999999996},"min":{"1min":0.498,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83599999999999997,"5min":0.98199999999999998,"15min":1.333},"last":0.76500000000000001}]},{"osd":2,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.747,"15min":0.76100000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.41299999999999998},"max":{"1min":0.91700000000000004,"5min":0.99399999999999999,"15min":1.03},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73599999999999999,"15min":0.76700000000000002},"min":{"1min":0.50600000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91600000000000004,"5min":0.96299999999999997,"15min":1.387},"last":0.80300000000000005}]},{"osd":3,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.70099999999999996,"15min":0.71999999999999997},"min":{"1min":0.495,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95999999999999996,"5min":0.98299999999999998,"15min":1.2350000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70599999999999996,"15min":0.73299999999999998},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.95199999999999996,"5min":0.95599999999999996,"15min":1.3180000000000001},"last":0.70199999999999996}]},{"osd":5,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59799999999999998,"15min":0.58599999999999997},"min":{"1min":0.436,"5min":0.39300000000000002,"15min":0.38100000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.60299999999999998,"15min":0.60399999999999998},"min":{"1min":0.48999999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.89500000000000002,"5min":1.0169999999999999,"15min":1.294},"last":0.77700000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.629,"15min":0.61599999999999999},"min":{"1min":0.45600000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.874,"5min":0.96999999999999997,"15min":1.0129999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.1359999999999999},"last":0.72799999999999998}]},{"osd":7,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67700000000000005,"15min":0.68300000000000005},"min":{"1min":0.54000000000000004,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.97199999999999998,"5min":1.0489999999999999,"15min":1.345},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.754,"15min":0.755},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.998,"5min":0.998,"15min":1.214},"last":0.83099999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36516,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335324,"statfs":{"total":96636764160,"available":96599371776,"internally_reserved":0,"allocated":1331200,"data_stored":625610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.64500000000000002,"15min":0.68100000000000005},"min":{"1min":0.44400000000000001,"5min":0.39200000000000002,"15min":0.313},"max":{"1min":0.754,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.71499999999999997,"15min":0.72499999999999998},"min":{"1min":0.47899999999999998,"5min":0.435,"15min":0.33500000000000002},"max":{"1min":0.83399999999999996,"5min":1.21,"15min":1.339},"last":0.56599999999999995}]},{"osd":1,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.69199999999999995,"15min":0.71699999999999997},"min":{"1min":0.47599999999999998,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.83799999999999997,"5min":0.96699999999999997,"15min":1.0169999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.68400000000000005,"15min":0.70599999999999996},"min":{"1min":0.441,"5min":0.441,"15min":0.39200000000000002},"max":{"1min":0.76500000000000001,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.70899999999999996,"15min":0.72799999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.77400000000000002,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68600000000000005,"15min":0.70599999999999996},"min":{"1min":0.46999999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72099999999999997,"5min":1.095,"15min":1.095},"last":0.54000000000000004}]},{"osd":3,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.70299999999999996,"15min":0.73199999999999998},"min":{"1min":0.5,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":0.83499999999999996,"5min":1.113,"15min":1.1619999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.72099999999999997,"15min":0.751},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.91100000000000003,"5min":1.0249999999999999,"15min":1.4850000000000001},"last":0.60299999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59599999999999997,"15min":0.59799999999999998},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.79500000000000004,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64800000000000002,"15min":0.64100000000000001},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":0.86799999999999999,"5min":1.097,"15min":1.214},"last":0.70199999999999996}]},{"osd":6,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.64700000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.36799999999999999},"max":{"1min":0.81499999999999995,"5min":0.91300000000000003,"15min":1.151},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82099999999999995,"5min":0.97099999999999997,"15min":1.095},"last":0.66400000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.69999999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89300000000000002,"5min":0.95299999999999996,"15min":1.325},"last":0.749},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.72699999999999998,"15min":0.73099999999999998},"min":{"1min":0.45600000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.1990000000000001},"last":0.76700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:11:03.561 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:11:03.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.797+0000 7f91b4c00700 1 Processor -- start 2024-03-29T10:11:03.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.797+0000 7f91b4c00700 1 -- start start 2024-03-29T10:11:03.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.798+0000 7f91b4c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b00d18b0 0x7f91b00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.798+0000 7f91b4c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91b0161590 0x7f91b0161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.798+0000 7f91b4c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b00ceda0 con 0x7f91b0160f10 2024-03-29T10:11:03.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.798+0000 7f91b4c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b00d1d30 con 0x7f91b0161590 2024-03-29T10:11:03.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.798+0000 7f91b4c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0161f60 con 0x7f91b00d18b0 2024-03-29T10:11:03.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.798+0000 7f91ae800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91b0161590 0x7f91b0161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.798+0000 7f91ae800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91b0161590 0x7f91b0161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39474/0 (socket says 172.21.15.12:39474) 2024-03-29T10:11:03.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.798+0000 7f91ae800700 1 -- 172.21.15.12:0/1576127163 learned_addr learned my addr 172.21.15.12:0/1576127163 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:03.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.798+0000 7f91ad400700 1 --2- 172.21.15.12:0/1576127163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b00d18b0 0x7f91b00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.799+0000 7f91ae800700 1 -- 172.21.15.12:0/1576127163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b00d18b0 msgr2=0x7f91b00cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:03.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.799+0000 7f91ae800700 1 --2- 172.21.15.12:0/1576127163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b00d18b0 0x7f91b00cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.799+0000 7f91ae800700 1 -- 172.21.15.12:0/1576127163 >> v1:172.21.15.159:6789/0 conn(0x7f91b0160f10 legacy=0x7f91b00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.799+0000 7f91ae800700 1 -- 172.21.15.12:0/1576127163 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b01612f0 con 0x7f91b0161590 2024-03-29T10:11:03.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.799+0000 7f91ae800700 1 --2- 172.21.15.12:0/1576127163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 0x7f91b0161a20 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f91a8009e80 tx=0x7f91a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7184a5edb17ec0e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:03.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.799+0000 7f91aca00700 1 -- 172.21.15.12:0/1576127163 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a800e070 con 0x7f91b0161590 2024-03-29T10:11:03.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.799+0000 7f91aca00700 1 -- 172.21.15.12:0/1576127163 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a800b8d0 con 0x7f91b0161590 2024-03-29T10:11:03.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.799+0000 7f91aca00700 1 -- 172.21.15.12:0/1576127163 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a800bc80 con 0x7f91b0161590 2024-03-29T10:11:03.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.800+0000 7f91b4c00700 1 -- 172.21.15.12:0/1576127163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 msgr2=0x7f91b0161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.800+0000 7f91b4c00700 1 --2- 172.21.15.12:0/1576127163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 0x7f91b0161a20 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f91a8009e80 tx=0x7f91a800b330 comp rx=0 tx=0).stop 2024-03-29T10:11:03.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.800+0000 7f91b4c00700 1 -- 172.21.15.12:0/1576127163 shutdown_connections 2024-03-29T10:11:03.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.800+0000 7f91b4c00700 1 --2- 172.21.15.12:0/1576127163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 0x7f91b0161a20 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.800+0000 7f91b4c00700 1 --2- 172.21.15.12:0/1576127163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b00d18b0 0x7f91b00cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.800+0000 7f91b4c00700 1 -- 172.21.15.12:0/1576127163 >> 172.21.15.12:0/1576127163 conn(0x7f91b008aff0 msgr2=0x7f91b00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:03.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.800+0000 7f91b4c00700 1 -- 172.21.15.12:0/1576127163 shutdown_connections 2024-03-29T10:11:03.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.800+0000 7f91b4c00700 1 -- 172.21.15.12:0/1576127163 wait complete. 2024-03-29T10:11:03.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.802+0000 7f91b4c00700 1 Processor -- start 2024-03-29T10:11:03.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.802+0000 7f91b4c00700 1 -- start start 2024-03-29T10:11:03.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91b4c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91b00d18b0 0x7f91b00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91b4c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b0160f10 0x7f91b0157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91b4c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 0x7f91b0154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91b4c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0158950 con 0x7f91b0161590 2024-03-29T10:11:03.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91b4c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0155200 con 0x7f91b00d18b0 2024-03-29T10:11:03.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91b4c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0158670 con 0x7f91b0160f10 2024-03-29T10:11:03.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ae800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 0x7f91b0154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ae800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 0x7f91b0154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39478/0 (socket says 172.21.15.12:39478) 2024-03-29T10:11:03.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ae800700 1 -- 172.21.15.12:0/2760218119 learned_addr learned my addr 172.21.15.12:0/2760218119 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ad400700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b0160f10 0x7f91b0157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ade00700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91b00d18b0 0x7f91b00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ae800700 1 -- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b0160f10 msgr2=0x7f91b0157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ae800700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b0160f10 0x7f91b0157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ae800700 1 -- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91b00d18b0 msgr2=0x7f91b00b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ae800700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91b00d18b0 0x7f91b00b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ae800700 1 -- 172.21.15.12:0/2760218119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a8009b30 con 0x7f91b0161590 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.803+0000 7f91ad400700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b0160f10 0x7f91b0157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.804+0000 7f91ae800700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 0x7f91b0154cc0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f91b008c570 tx=0x7f91a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.804+0000 7f91a6000700 1 -- 172.21.15.12:0/2760218119 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8004960 con 0x7f91b0161590 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.804+0000 7f91a6000700 1 -- 172.21.15.12:0/2760218119 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a80099b0 con 0x7f91b0161590 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.804+0000 7f91a6000700 1 -- 172.21.15.12:0/2760218119 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a801c680 con 0x7f91b0161590 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.804+0000 7f91b4c00700 1 -- 172.21.15.12:0/2760218119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b01e71f0 con 0x7f91b0161590 2024-03-29T10:11:03.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.804+0000 7f91b4c00700 1 -- 172.21.15.12:0/2760218119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b01e77b0 con 0x7f91b0161590 2024-03-29T10:11:03.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.805+0000 7f91b4c00700 1 -- 172.21.15.12:0/2760218119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b01e7050 con 0x7f91b0161590 2024-03-29T10:11:03.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.806+0000 7f91a6000700 1 -- 172.21.15.12:0/2760218119 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f91a801c7e0 con 0x7f91b0161590 2024-03-29T10:11:03.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.807+0000 7f91a6000700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f919006adb0 0x7f919006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:03.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.807+0000 7f91a6000700 1 -- 172.21.15.12:0/2760218119 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91a808b020 con 0x7f91b0161590 2024-03-29T10:11:03.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.807+0000 7f91ade00700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f919006adb0 0x7f919006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:03.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.808+0000 7f91ade00700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f919006adb0 0x7f919006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f91a0000f60 tx=0x7f91a0009040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:03.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:03.811+0000 7f91a6000700 1 -- 172.21.15.12:0/2760218119 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91b01e7050 con 0x7f91b0161590 2024-03-29T10:11:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.021+0000 7f91b4c00700 1 -- 172.21.15.12:0/2760218119 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f91b00ce750 con 0x7f919006adb0 2024-03-29T10:11:04.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.030+0000 7f91a6000700 1 -- 172.21.15.12:0/2760218119 <== mgr.6940 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289215 (secure 0 0 0) 0x7f91a00013f0 con 0x7f919006adb0 2024-03-29T10:11:04.034 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:11:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 -- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f919006adb0 msgr2=0x7f919006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f919006adb0 0x7f919006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f91a0000f60 tx=0x7f91a0009040 comp rx=0 tx=0).stop 2024-03-29T10:11:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 -- 172.21.15.12:0/2760218119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 msgr2=0x7f91b0154cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 0x7f91b0154cc0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f91b008c570 tx=0x7f91a8004ad0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 -- 172.21.15.12:0/2760218119 shutdown_connections 2024-03-29T10:11:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f919006adb0 0x7f919006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91b0161590 0x7f91b0154cc0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91b00d18b0 0x7f91b00b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 --2- 172.21.15.12:0/2760218119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91b0160f10 0x7f91b0157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 -- 172.21.15.12:0/2760218119 >> 172.21.15.12:0/2760218119 conn(0x7f91b008aff0 msgr2=0x7f91b00cae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 -- 172.21.15.12:0/2760218119 shutdown_connections 2024-03-29T10:11:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.035+0000 7f91b4c00700 1 -- 172.21.15.12:0/2760218119 wait complete. 2024-03-29T10:11:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:11:04.053 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-29T10:11:03.622298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":355,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":321112,"kb_used_data":11464,"kb_used_omap":12,"kb_used_meta":309491,"kb_avail":754653608,"statfs":{"total":773094113280,"available":772765294592,"internally_reserved":0,"allocated":11739136,"data_stored":6168204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":316919353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001506"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.657359+0000","last_change":"2024-03-29T10:10:58.652413+0000","last_active":"2024-03-29T10:10:58.657359+0000","last_peered":"2024-03-29T10:10:58.657359+0000","last_clean":"2024-03-29T10:10:58.657359+0000","last_became_active":"2024-03-29T10:10:58.650620+0000","last_became_peered":"2024-03-29T10:10:58.650620+0000","last_unstale":"2024-03-29T10:10:58.657359+0000","last_undegraded":"2024-03-29T10:10:58.657359+0000","last_fullsized":"2024-03-29T10:10:58.657359+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:43:18.124198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675685+0000","last_change":"2024-03-29T10:10:58.665186+0000","last_active":"2024-03-29T10:10:58.675685+0000","last_peered":"2024-03-29T10:10:58.675685+0000","last_clean":"2024-03-29T10:10:58.675685+0000","last_became_active":"2024-03-29T10:10:58.664316+0000","last_became_peered":"2024-03-29T10:10:58.664316+0000","last_unstale":"2024-03-29T10:10:58.675685+0000","last_undegraded":"2024-03-29T10:10:58.675685+0000","last_fullsized":"2024-03-29T10:10:58.675685+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:16.050232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669060+0000","last_change":"2024-03-29T10:10:58.667945+0000","last_active":"2024-03-29T10:10:58.669060+0000","last_peered":"2024-03-29T10:10:58.669060+0000","last_clean":"2024-03-29T10:10:58.669060+0000","last_became_active":"2024-03-29T10:10:58.667633+0000","last_became_peered":"2024-03-29T10:10:58.667633+0000","last_unstale":"2024-03-29T10:10:58.669060+0000","last_undegraded":"2024-03-29T10:10:58.669060+0000","last_fullsized":"2024-03-29T10:10:58.669060+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:13.760911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.667174+0000","last_change":"2024-03-29T10:10:58.655940+0000","last_active":"2024-03-29T10:10:58.667174+0000","last_peered":"2024-03-29T10:10:58.667174+0000","last_clean":"2024-03-29T10:10:58.667174+0000","last_became_active":"2024-03-29T10:10:58.654685+0000","last_became_peered":"2024-03-29T10:10:58.654685+0000","last_unstale":"2024-03-29T10:10:58.667174+0000","last_undegraded":"2024-03-29T10:10:58.667174+0000","last_fullsized":"2024-03-29T10:10:58.667174+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:37.434553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.664086+0000","last_change":"2024-03-29T10:10:58.662439+0000","last_active":"2024-03-29T10:10:58.664086+0000","last_peered":"2024-03-29T10:10:58.664086+0000","last_clean":"2024-03-29T10:10:58.664086+0000","last_became_active":"2024-03-29T10:10:58.662105+0000","last_became_peered":"2024-03-29T10:10:58.662105+0000","last_unstale":"2024-03-29T10:10:58.664086+0000","last_undegraded":"2024-03-29T10:10:58.664086+0000","last_fullsized":"2024-03-29T10:10:58.664086+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:57:08.688729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669723+0000","last_change":"2024-03-29T10:10:58.667532+0000","last_active":"2024-03-29T10:10:58.669723+0000","last_peered":"2024-03-29T10:10:58.669723+0000","last_clean":"2024-03-29T10:10:58.669723+0000","last_became_active":"2024-03-29T10:10:58.665389+0000","last_became_peered":"2024-03-29T10:10:58.665389+0000","last_unstale":"2024-03-29T10:10:58.669723+0000","last_undegraded":"2024-03-29T10:10:58.669723+0000","last_fullsized":"2024-03-29T10:10:58.669723+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:24.296829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.666669+0000","last_change":"2024-03-29T10:10:58.656245+0000","last_active":"2024-03-29T10:10:58.666669+0000","last_peered":"2024-03-29T10:10:58.666669+0000","last_clean":"2024-03-29T10:10:58.666669+0000","last_became_active":"2024-03-29T10:10:58.655953+0000","last_became_peered":"2024-03-29T10:10:58.655953+0000","last_unstale":"2024-03-29T10:10:58.666669+0000","last_undegraded":"2024-03-29T10:10:58.666669+0000","last_fullsized":"2024-03-29T10:10:58.666669+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:50:56.622833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.657364+0000","last_change":"2024-03-29T10:10:58.653695+0000","last_active":"2024-03-29T10:10:58.657364+0000","last_peered":"2024-03-29T10:10:58.657364+0000","last_clean":"2024-03-29T10:10:58.657364+0000","last_became_active":"2024-03-29T10:10:58.653412+0000","last_became_peered":"2024-03-29T10:10:58.653412+0000","last_unstale":"2024-03-29T10:10:58.657364+0000","last_undegraded":"2024-03-29T10:10:58.657364+0000","last_fullsized":"2024-03-29T10:10:58.657364+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:17:24.194613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.668675+0000","last_change":"2024-03-29T10:10:58.665371+0000","last_active":"2024-03-29T10:10:58.668675+0000","last_peered":"2024-03-29T10:10:58.668675+0000","last_clean":"2024-03-29T10:10:58.668675+0000","last_became_active":"2024-03-29T10:10:58.665082+0000","last_became_peered":"2024-03-29T10:10:58.665082+0000","last_unstale":"2024-03-29T10:10:58.668675+0000","last_undegraded":"2024-03-29T10:10:58.668675+0000","last_fullsized":"2024-03-29T10:10:58.668675+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:21.918440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.666270+0000","last_change":"2024-03-29T10:10:58.660152+0000","last_active":"2024-03-29T10:10:58.666270+0000","last_peered":"2024-03-29T10:10:58.666270+0000","last_clean":"2024-03-29T10:10:58.666270+0000","last_became_active":"2024-03-29T10:10:58.659014+0000","last_became_peered":"2024-03-29T10:10:58.659014+0000","last_unstale":"2024-03-29T10:10:58.666270+0000","last_undegraded":"2024-03-29T10:10:58.666270+0000","last_fullsized":"2024-03-29T10:10:58.666270+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:29:27.283488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.665330+0000","last_change":"2024-03-29T10:10:58.662886+0000","last_active":"2024-03-29T10:10:58.665330+0000","last_peered":"2024-03-29T10:10:58.665330+0000","last_clean":"2024-03-29T10:10:58.665330+0000","last_became_active":"2024-03-29T10:10:58.662182+0000","last_became_peered":"2024-03-29T10:10:58.662182+0000","last_unstale":"2024-03-29T10:10:58.665330+0000","last_undegraded":"2024-03-29T10:10:58.665330+0000","last_fullsized":"2024-03-29T10:10:58.665330+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:52:39.940245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.664040+0000","last_change":"2024-03-29T10:10:58.661018+0000","last_active":"2024-03-29T10:10:58.664040+0000","last_peered":"2024-03-29T10:10:58.664040+0000","last_clean":"2024-03-29T10:10:58.664040+0000","last_became_active":"2024-03-29T10:10:58.660752+0000","last_became_peered":"2024-03-29T10:10:58.660752+0000","last_unstale":"2024-03-29T10:10:58.664040+0000","last_undegraded":"2024-03-29T10:10:58.664040+0000","last_fullsized":"2024-03-29T10:10:58.664040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:24.188405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662745+0000","last_change":"2024-03-29T10:10:58.658393+0000","last_active":"2024-03-29T10:10:58.662745+0000","last_peered":"2024-03-29T10:10:58.662745+0000","last_clean":"2024-03-29T10:10:58.662745+0000","last_became_active":"2024-03-29T10:10:58.657921+0000","last_became_peered":"2024-03-29T10:10:58.657921+0000","last_unstale":"2024-03-29T10:10:58.662745+0000","last_undegraded":"2024-03-29T10:10:58.662745+0000","last_fullsized":"2024-03-29T10:10:58.662745+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:37:57.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.665493+0000","last_change":"2024-03-29T10:10:58.654494+0000","last_active":"2024-03-29T10:10:58.665493+0000","last_peered":"2024-03-29T10:10:58.665493+0000","last_clean":"2024-03-29T10:10:58.665493+0000","last_became_active":"2024-03-29T10:10:58.654124+0000","last_became_peered":"2024-03-29T10:10:58.654124+0000","last_unstale":"2024-03-29T10:10:58.665493+0000","last_undegraded":"2024-03-29T10:10:58.665493+0000","last_fullsized":"2024-03-29T10:10:58.665493+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:25:38.029319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662667+0000","last_change":"2024-03-29T10:10:58.658270+0000","last_active":"2024-03-29T10:10:58.662667+0000","last_peered":"2024-03-29T10:10:58.662667+0000","last_clean":"2024-03-29T10:10:58.662667+0000","last_became_active":"2024-03-29T10:10:58.658002+0000","last_became_peered":"2024-03-29T10:10:58.658002+0000","last_unstale":"2024-03-29T10:10:58.662667+0000","last_undegraded":"2024-03-29T10:10:58.662667+0000","last_fullsized":"2024-03-29T10:10:58.662667+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:06.388556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.659117+0000","last_change":"2024-03-29T10:10:58.656125+0000","last_active":"2024-03-29T10:10:58.659117+0000","last_peered":"2024-03-29T10:10:58.659117+0000","last_clean":"2024-03-29T10:10:58.659117+0000","last_became_active":"2024-03-29T10:10:58.655492+0000","last_became_peered":"2024-03-29T10:10:58.655492+0000","last_unstale":"2024-03-29T10:10:58.659117+0000","last_undegraded":"2024-03-29T10:10:58.659117+0000","last_fullsized":"2024-03-29T10:10:58.659117+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:54:00.175790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.664153+0000","last_change":"2024-03-29T10:10:58.663049+0000","last_active":"2024-03-29T10:10:58.664153+0000","last_peered":"2024-03-29T10:10:58.664153+0000","last_clean":"2024-03-29T10:10:58.664153+0000","last_became_active":"2024-03-29T10:10:58.662733+0000","last_became_peered":"2024-03-29T10:10:58.662733+0000","last_unstale":"2024-03-29T10:10:58.664153+0000","last_undegraded":"2024-03-29T10:10:58.664153+0000","last_fullsized":"2024-03-29T10:10:58.664153+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:16.754618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.661556+0000","last_change":"2024-03-29T10:10:58.647492+0000","last_active":"2024-03-29T10:10:58.661556+0000","last_peered":"2024-03-29T10:10:58.661556+0000","last_clean":"2024-03-29T10:10:58.661556+0000","last_became_active":"2024-03-29T10:10:58.647235+0000","last_became_peered":"2024-03-29T10:10:58.647235+0000","last_unstale":"2024-03-29T10:10:58.661556+0000","last_undegraded":"2024-03-29T10:10:58.661556+0000","last_fullsized":"2024-03-29T10:10:58.661556+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:05:54.869449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.672766+0000","last_change":"2024-03-29T10:10:58.664720+0000","last_active":"2024-03-29T10:10:58.672766+0000","last_peered":"2024-03-29T10:10:58.672766+0000","last_clean":"2024-03-29T10:10:58.672766+0000","last_became_active":"2024-03-29T10:10:58.664450+0000","last_became_peered":"2024-03-29T10:10:58.664450+0000","last_unstale":"2024-03-29T10:10:58.672766+0000","last_undegraded":"2024-03-29T10:10:58.672766+0000","last_fullsized":"2024-03-29T10:10:58.672766+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:50.527149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.671890+0000","last_change":"2024-03-29T10:10:58.667073+0000","last_active":"2024-03-29T10:10:58.671890+0000","last_peered":"2024-03-29T10:10:58.671890+0000","last_clean":"2024-03-29T10:10:58.671890+0000","last_became_active":"2024-03-29T10:10:58.666735+0000","last_became_peered":"2024-03-29T10:10:58.666735+0000","last_unstale":"2024-03-29T10:10:58.671890+0000","last_undegraded":"2024-03-29T10:10:58.671890+0000","last_fullsized":"2024-03-29T10:10:58.671890+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:16.050232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.666209+0000","last_change":"2024-03-29T10:10:58.654617+0000","last_active":"2024-03-29T10:10:58.666209+0000","last_peered":"2024-03-29T10:10:58.666209+0000","last_clean":"2024-03-29T10:10:58.666209+0000","last_became_active":"2024-03-29T10:10:58.654337+0000","last_became_peered":"2024-03-29T10:10:58.654337+0000","last_unstale":"2024-03-29T10:10:58.666209+0000","last_undegraded":"2024-03-29T10:10:58.666209+0000","last_fullsized":"2024-03-29T10:10:58.666209+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:31.763079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.665070+0000","last_change":"2024-03-29T10:10:58.658771+0000","last_active":"2024-03-29T10:10:58.665070+0000","last_peered":"2024-03-29T10:10:58.665070+0000","last_clean":"2024-03-29T10:10:58.665070+0000","last_became_active":"2024-03-29T10:10:58.658509+0000","last_became_peered":"2024-03-29T10:10:58.658509+0000","last_unstale":"2024-03-29T10:10:58.665070+0000","last_undegraded":"2024-03-29T10:10:58.665070+0000","last_fullsized":"2024-03-29T10:10:58.665070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:28:14.453536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669940+0000","last_change":"2024-03-29T10:10:58.664754+0000","last_active":"2024-03-29T10:10:58.669940+0000","last_peered":"2024-03-29T10:10:58.669940+0000","last_clean":"2024-03-29T10:10:58.669940+0000","last_became_active":"2024-03-29T10:10:58.663924+0000","last_became_peered":"2024-03-29T10:10:58.663924+0000","last_unstale":"2024-03-29T10:10:58.669940+0000","last_undegraded":"2024-03-29T10:10:58.669940+0000","last_fullsized":"2024-03-29T10:10:58.669940+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:50.527149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.666585+0000","last_change":"2024-03-29T10:10:58.654776+0000","last_active":"2024-03-29T10:10:58.666585+0000","last_peered":"2024-03-29T10:10:58.666585+0000","last_clean":"2024-03-29T10:10:58.666585+0000","last_became_active":"2024-03-29T10:10:58.654357+0000","last_became_peered":"2024-03-29T10:10:58.654357+0000","last_unstale":"2024-03-29T10:10:58.666585+0000","last_undegraded":"2024-03-29T10:10:58.666585+0000","last_fullsized":"2024-03-29T10:10:58.666585+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:40:11.624381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.678742+0000","last_change":"2024-03-29T10:10:58.673760+0000","last_active":"2024-03-29T10:10:58.678742+0000","last_peered":"2024-03-29T10:10:58.678742+0000","last_clean":"2024-03-29T10:10:58.678742+0000","last_became_active":"2024-03-29T10:10:58.667258+0000","last_became_peered":"2024-03-29T10:10:58.667258+0000","last_unstale":"2024-03-29T10:10:58.678742+0000","last_undegraded":"2024-03-29T10:10:58.678742+0000","last_fullsized":"2024-03-29T10:10:58.678742+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:33:07.658914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675138+0000","last_change":"2024-03-29T10:10:58.671196+0000","last_active":"2024-03-29T10:10:58.675138+0000","last_peered":"2024-03-29T10:10:58.675138+0000","last_clean":"2024-03-29T10:10:58.675138+0000","last_became_active":"2024-03-29T10:10:58.670860+0000","last_became_peered":"2024-03-29T10:10:58.670860+0000","last_unstale":"2024-03-29T10:10:58.675138+0000","last_undegraded":"2024-03-29T10:10:58.675138+0000","last_fullsized":"2024-03-29T10:10:58.675138+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:29:27.283488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662271+0000","last_change":"2024-03-29T10:10:58.660830+0000","last_active":"2024-03-29T10:10:58.662271+0000","last_peered":"2024-03-29T10:10:58.662271+0000","last_clean":"2024-03-29T10:10:58.662271+0000","last_became_active":"2024-03-29T10:10:58.660515+0000","last_became_peered":"2024-03-29T10:10:58.660515+0000","last_unstale":"2024-03-29T10:10:58.662271+0000","last_undegraded":"2024-03-29T10:10:58.662271+0000","last_fullsized":"2024-03-29T10:10:58.662271+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:30:01.048443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.661495+0000","last_change":"2024-03-29T10:10:58.654639+0000","last_active":"2024-03-29T10:10:58.661495+0000","last_peered":"2024-03-29T10:10:58.661495+0000","last_clean":"2024-03-29T10:10:58.661495+0000","last_became_active":"2024-03-29T10:10:58.654253+0000","last_became_peered":"2024-03-29T10:10:58.654253+0000","last_unstale":"2024-03-29T10:10:58.661495+0000","last_undegraded":"2024-03-29T10:10:58.661495+0000","last_fullsized":"2024-03-29T10:10:58.661495+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:13.760911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.676230+0000","last_change":"2024-03-29T10:10:58.666480+0000","last_active":"2024-03-29T10:10:58.676230+0000","last_peered":"2024-03-29T10:10:58.676230+0000","last_clean":"2024-03-29T10:10:58.676230+0000","last_became_active":"2024-03-29T10:10:58.665590+0000","last_became_peered":"2024-03-29T10:10:58.665590+0000","last_unstale":"2024-03-29T10:10:58.676230+0000","last_undegraded":"2024-03-29T10:10:58.676230+0000","last_fullsized":"2024-03-29T10:10:58.676230+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:32.463011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.678691+0000","last_change":"2024-03-29T10:10:58.676114+0000","last_active":"2024-03-29T10:10:58.678691+0000","last_peered":"2024-03-29T10:10:58.678691+0000","last_clean":"2024-03-29T10:10:58.678691+0000","last_became_active":"2024-03-29T10:10:58.675795+0000","last_became_peered":"2024-03-29T10:10:58.675795+0000","last_unstale":"2024-03-29T10:10:58.678691+0000","last_undegraded":"2024-03-29T10:10:58.678691+0000","last_fullsized":"2024-03-29T10:10:58.678691+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:50:56.622833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662068+0000","last_change":"2024-03-29T10:10:58.658334+0000","last_active":"2024-03-29T10:10:58.662068+0000","last_peered":"2024-03-29T10:10:58.662068+0000","last_clean":"2024-03-29T10:10:58.662068+0000","last_became_active":"2024-03-29T10:10:58.657792+0000","last_became_peered":"2024-03-29T10:10:58.657792+0000","last_unstale":"2024-03-29T10:10:58.662068+0000","last_undegraded":"2024-03-29T10:10:58.662068+0000","last_fullsized":"2024-03-29T10:10:58.662068+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:05.682812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.661696+0000","last_change":"2024-03-29T10:10:58.656325+0000","last_active":"2024-03-29T10:10:58.661696+0000","last_peered":"2024-03-29T10:10:58.661696+0000","last_clean":"2024-03-29T10:10:58.661696+0000","last_became_active":"2024-03-29T10:10:58.655838+0000","last_became_peered":"2024-03-29T10:10:58.655838+0000","last_unstale":"2024-03-29T10:10:58.661696+0000","last_undegraded":"2024-03-29T10:10:58.661696+0000","last_fullsized":"2024-03-29T10:10:58.661696+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:53.750734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.654258+0000","last_change":"2024-03-29T10:10:58.652425+0000","last_active":"2024-03-29T10:10:58.654258+0000","last_peered":"2024-03-29T10:10:58.654258+0000","last_clean":"2024-03-29T10:10:58.654258+0000","last_became_active":"2024-03-29T10:10:58.651963+0000","last_became_peered":"2024-03-29T10:10:58.651963+0000","last_unstale":"2024-03-29T10:10:58.654258+0000","last_undegraded":"2024-03-29T10:10:58.654258+0000","last_fullsized":"2024-03-29T10:10:58.654258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:50.527149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"107'4","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.673802+0000","last_change":"2024-03-29T10:10:58.130267+0000","last_active":"2024-03-29T10:10:58.673802+0000","last_peered":"2024-03-29T10:10:58.673802+0000","last_clean":"2024-03-29T10:10:58.673802+0000","last_became_active":"2024-03-29T10:10:58.129956+0000","last_became_peered":"2024-03-29T10:10:58.129956+0000","last_unstale":"2024-03-29T10:10:58.673802+0000","last_undegraded":"2024-03-29T10:10:58.673802+0000","last_fullsized":"2024-03-29T10:10:58.673802+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:52:17.719442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.668600+0000","last_change":"2024-03-29T10:10:58.664102+0000","last_active":"2024-03-29T10:10:58.668600+0000","last_peered":"2024-03-29T10:10:58.668600+0000","last_clean":"2024-03-29T10:10:58.668600+0000","last_became_active":"2024-03-29T10:10:58.663841+0000","last_became_peered":"2024-03-29T10:10:58.663841+0000","last_unstale":"2024-03-29T10:10:58.668600+0000","last_undegraded":"2024-03-29T10:10:58.668600+0000","last_fullsized":"2024-03-29T10:10:58.668600+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:08:37.164467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"107'1","reported_seq":11,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675465+0000","last_change":"2024-03-29T10:10:58.122547+0000","last_active":"2024-03-29T10:10:58.675465+0000","last_peered":"2024-03-29T10:10:58.675465+0000","last_clean":"2024-03-29T10:10:58.675465+0000","last_became_active":"2024-03-29T10:10:58.120320+0000","last_became_peered":"2024-03-29T10:10:58.120320+0000","last_unstale":"2024-03-29T10:10:58.675465+0000","last_undegraded":"2024-03-29T10:10:58.675465+0000","last_fullsized":"2024-03-29T10:10:58.675465+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:10.070572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.661721+0000","last_change":"2024-03-29T10:10:58.656445+0000","last_active":"2024-03-29T10:10:58.661721+0000","last_peered":"2024-03-29T10:10:58.661721+0000","last_clean":"2024-03-29T10:10:58.661721+0000","last_became_active":"2024-03-29T10:10:58.655845+0000","last_became_peered":"2024-03-29T10:10:58.655845+0000","last_unstale":"2024-03-29T10:10:58.661721+0000","last_undegraded":"2024-03-29T10:10:58.661721+0000","last_fullsized":"2024-03-29T10:10:58.661721+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:06.388556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.680817+0000","last_change":"2024-03-29T10:10:58.130227+0000","last_active":"2024-03-29T10:10:58.680817+0000","last_peered":"2024-03-29T10:10:58.680817+0000","last_clean":"2024-03-29T10:10:58.680817+0000","last_became_active":"2024-03-29T10:10:58.129940+0000","last_became_peered":"2024-03-29T10:10:58.129940+0000","last_unstale":"2024-03-29T10:10:58.680817+0000","last_undegraded":"2024-03-29T10:10:58.680817+0000","last_fullsized":"2024-03-29T10:10:58.680817+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:15.747324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669904+0000","last_change":"2024-03-29T10:10:58.668625+0000","last_active":"2024-03-29T10:10:58.669904+0000","last_peered":"2024-03-29T10:10:58.669904+0000","last_clean":"2024-03-29T10:10:58.669904+0000","last_became_active":"2024-03-29T10:10:58.668121+0000","last_became_peered":"2024-03-29T10:10:58.668121+0000","last_unstale":"2024-03-29T10:10:58.669904+0000","last_undegraded":"2024-03-29T10:10:58.669904+0000","last_fullsized":"2024-03-29T10:10:58.669904+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:50.527149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.680947+0000","last_change":"2024-03-29T10:10:58.130656+0000","last_active":"2024-03-29T10:10:58.680947+0000","last_peered":"2024-03-29T10:10:58.680947+0000","last_clean":"2024-03-29T10:10:58.680947+0000","last_became_active":"2024-03-29T10:10:58.129435+0000","last_became_peered":"2024-03-29T10:10:58.129435+0000","last_unstale":"2024-03-29T10:10:58.680947+0000","last_undegraded":"2024-03-29T10:10:58.680947+0000","last_fullsized":"2024-03-29T10:10:58.680947+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:30:00.041149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.678711+0000","last_change":"2024-03-29T10:10:58.676273+0000","last_active":"2024-03-29T10:10:58.678711+0000","last_peered":"2024-03-29T10:10:58.678711+0000","last_clean":"2024-03-29T10:10:58.678711+0000","last_became_active":"2024-03-29T10:10:58.675815+0000","last_became_peered":"2024-03-29T10:10:58.675815+0000","last_unstale":"2024-03-29T10:10:58.678711+0000","last_undegraded":"2024-03-29T10:10:58.678711+0000","last_fullsized":"2024-03-29T10:10:58.678711+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:13:28.530296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"107'1","reported_seq":11,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.677191+0000","last_change":"2024-03-29T10:10:58.119864+0000","last_active":"2024-03-29T10:10:58.677191+0000","last_peered":"2024-03-29T10:10:58.677191+0000","last_clean":"2024-03-29T10:10:58.677191+0000","last_became_active":"2024-03-29T10:10:58.119486+0000","last_became_peered":"2024-03-29T10:10:58.119486+0000","last_unstale":"2024-03-29T10:10:58.677191+0000","last_undegraded":"2024-03-29T10:10:58.677191+0000","last_fullsized":"2024-03-29T10:10:58.677191+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:40:10.617087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.664413+0000","last_change":"2024-03-29T10:10:58.663372+0000","last_active":"2024-03-29T10:10:58.664413+0000","last_peered":"2024-03-29T10:10:58.664413+0000","last_clean":"2024-03-29T10:10:58.664413+0000","last_became_active":"2024-03-29T10:10:58.663056+0000","last_became_peered":"2024-03-29T10:10:58.663056+0000","last_unstale":"2024-03-29T10:10:58.664413+0000","last_undegraded":"2024-03-29T10:10:58.664413+0000","last_fullsized":"2024-03-29T10:10:58.664413+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:15.239902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"107'1","reported_seq":11,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675423+0000","last_change":"2024-03-29T10:10:58.119337+0000","last_active":"2024-03-29T10:10:58.675423+0000","last_peered":"2024-03-29T10:10:58.675423+0000","last_clean":"2024-03-29T10:10:58.675423+0000","last_became_active":"2024-03-29T10:10:58.119003+0000","last_became_peered":"2024-03-29T10:10:58.119003+0000","last_unstale":"2024-03-29T10:10:58.675423+0000","last_undegraded":"2024-03-29T10:10:58.675423+0000","last_fullsized":"2024-03-29T10:10:58.675423+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:36.199617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.663613+0000","last_change":"2024-03-29T10:10:58.658725+0000","last_active":"2024-03-29T10:10:58.663613+0000","last_peered":"2024-03-29T10:10:58.663613+0000","last_clean":"2024-03-29T10:10:58.663613+0000","last_became_active":"2024-03-29T10:10:58.658456+0000","last_became_peered":"2024-03-29T10:10:58.658456+0000","last_unstale":"2024-03-29T10:10:58.663613+0000","last_undegraded":"2024-03-29T10:10:58.663613+0000","last_fullsized":"2024-03-29T10:10:58.663613+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:57:08.688729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.654301+0000","last_change":"2024-03-29T10:10:58.125683+0000","last_active":"2024-03-29T10:10:58.654301+0000","last_peered":"2024-03-29T10:10:58.654301+0000","last_clean":"2024-03-29T10:10:58.654301+0000","last_became_active":"2024-03-29T10:10:58.125421+0000","last_became_peered":"2024-03-29T10:10:58.125421+0000","last_unstale":"2024-03-29T10:10:58.654301+0000","last_undegraded":"2024-03-29T10:10:58.654301+0000","last_fullsized":"2024-03-29T10:10:58.654301+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:36:13.048236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662646+0000","last_change":"2024-03-29T10:10:58.657777+0000","last_active":"2024-03-29T10:10:58.662646+0000","last_peered":"2024-03-29T10:10:58.662646+0000","last_clean":"2024-03-29T10:10:58.662646+0000","last_became_active":"2024-03-29T10:10:58.657445+0000","last_became_peered":"2024-03-29T10:10:58.657445+0000","last_unstale":"2024-03-29T10:10:58.662646+0000","last_undegraded":"2024-03-29T10:10:58.662646+0000","last_fullsized":"2024-03-29T10:10:58.662646+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:08:37.164467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.656417+0000","last_change":"2024-03-29T10:10:58.127497+0000","last_active":"2024-03-29T10:10:58.656417+0000","last_peered":"2024-03-29T10:10:58.656417+0000","last_clean":"2024-03-29T10:10:58.656417+0000","last_became_active":"2024-03-29T10:10:58.127154+0000","last_became_peered":"2024-03-29T10:10:58.127154+0000","last_unstale":"2024-03-29T10:10:58.656417+0000","last_undegraded":"2024-03-29T10:10:58.656417+0000","last_fullsized":"2024-03-29T10:10:58.656417+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:20:36.618357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675093+0000","last_change":"2024-03-29T10:10:58.670879+0000","last_active":"2024-03-29T10:10:58.675093+0000","last_peered":"2024-03-29T10:10:58.675093+0000","last_clean":"2024-03-29T10:10:58.675093+0000","last_became_active":"2024-03-29T10:10:58.670477+0000","last_became_peered":"2024-03-29T10:10:58.670477+0000","last_unstale":"2024-03-29T10:10:58.675093+0000","last_undegraded":"2024-03-29T10:10:58.675093+0000","last_fullsized":"2024-03-29T10:10:58.675093+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:32:54.224552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.642166+0000","last_change":"2024-03-29T10:10:58.127665+0000","last_active":"2024-03-29T10:10:58.642166+0000","last_peered":"2024-03-29T10:10:58.642166+0000","last_clean":"2024-03-29T10:10:58.642166+0000","last_became_active":"2024-03-29T10:10:58.127373+0000","last_became_peered":"2024-03-29T10:10:58.127373+0000","last_unstale":"2024-03-29T10:10:58.642166+0000","last_undegraded":"2024-03-29T10:10:58.642166+0000","last_fullsized":"2024-03-29T10:10:58.642166+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:17:23.187320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.650758+0000","last_change":"2024-03-29T10:10:58.648192+0000","last_active":"2024-03-29T10:10:58.650758+0000","last_peered":"2024-03-29T10:10:58.650758+0000","last_clean":"2024-03-29T10:10:58.650758+0000","last_became_active":"2024-03-29T10:10:58.647907+0000","last_became_peered":"2024-03-29T10:10:58.647907+0000","last_unstale":"2024-03-29T10:10:58.650758+0000","last_undegraded":"2024-03-29T10:10:58.650758+0000","last_fullsized":"2024-03-29T10:10:58.650758+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:31.402667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.643364+0000","last_change":"2024-03-29T10:10:58.129139+0000","last_active":"2024-03-29T10:10:58.643364+0000","last_peered":"2024-03-29T10:10:58.643364+0000","last_clean":"2024-03-29T10:10:58.643364+0000","last_became_active":"2024-03-29T10:10:58.127784+0000","last_became_peered":"2024-03-29T10:10:58.127784+0000","last_unstale":"2024-03-29T10:10:58.643364+0000","last_undegraded":"2024-03-29T10:10:58.643364+0000","last_fullsized":"2024-03-29T10:10:58.643364+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:20.219556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.677769+0000","last_change":"2024-03-29T10:10:58.670325+0000","last_active":"2024-03-29T10:10:58.677769+0000","last_peered":"2024-03-29T10:10:58.677769+0000","last_clean":"2024-03-29T10:10:58.677769+0000","last_became_active":"2024-03-29T10:10:58.670027+0000","last_became_peered":"2024-03-29T10:10:58.670027+0000","last_unstale":"2024-03-29T10:10:58.677769+0000","last_undegraded":"2024-03-29T10:10:58.677769+0000","last_fullsized":"2024-03-29T10:10:58.677769+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:43:18.124198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"107'3","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.678251+0000","last_change":"2024-03-29T10:10:58.125628+0000","last_active":"2024-03-29T10:10:58.678251+0000","last_peered":"2024-03-29T10:10:58.678251+0000","last_clean":"2024-03-29T10:10:58.678251+0000","last_became_active":"2024-03-29T10:10:58.125319+0000","last_became_peered":"2024-03-29T10:10:58.125319+0000","last_unstale":"2024-03-29T10:10:58.678251+0000","last_undegraded":"2024-03-29T10:10:58.678251+0000","last_fullsized":"2024-03-29T10:10:58.678251+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:32.039300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.668083+0000","last_change":"2024-03-29T10:10:58.666333+0000","last_active":"2024-03-29T10:10:58.668083+0000","last_peered":"2024-03-29T10:10:58.668083+0000","last_clean":"2024-03-29T10:10:58.668083+0000","last_became_active":"2024-03-29T10:10:58.666007+0000","last_became_peered":"2024-03-29T10:10:58.666007+0000","last_unstale":"2024-03-29T10:10:58.668083+0000","last_undegraded":"2024-03-29T10:10:58.668083+0000","last_fullsized":"2024-03-29T10:10:58.668083+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:34.327562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"107'6","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.680427+0000","last_change":"2024-03-29T10:10:58.130489+0000","last_active":"2024-03-29T10:10:58.680427+0000","last_peered":"2024-03-29T10:10:58.680427+0000","last_clean":"2024-03-29T10:10:58.680427+0000","last_became_active":"2024-03-29T10:10:58.130184+0000","last_became_peered":"2024-03-29T10:10:58.130184+0000","last_unstale":"2024-03-29T10:10:58.680427+0000","last_undegraded":"2024-03-29T10:10:58.680427+0000","last_fullsized":"2024-03-29T10:10:58.680427+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:15.042938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.671946+0000","last_change":"2024-03-29T10:10:58.664647+0000","last_active":"2024-03-29T10:10:58.671946+0000","last_peered":"2024-03-29T10:10:58.671946+0000","last_clean":"2024-03-29T10:10:58.671946+0000","last_became_active":"2024-03-29T10:10:58.664040+0000","last_became_peered":"2024-03-29T10:10:58.664040+0000","last_unstale":"2024-03-29T10:10:58.671946+0000","last_undegraded":"2024-03-29T10:10:58.671946+0000","last_fullsized":"2024-03-29T10:10:58.671946+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:24.188405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.679343+0000","last_change":"2024-03-29T10:10:58.131540+0000","last_active":"2024-03-29T10:10:58.679343+0000","last_peered":"2024-03-29T10:10:58.679343+0000","last_clean":"2024-03-29T10:10:58.679343+0000","last_became_active":"2024-03-29T10:10:58.129783+0000","last_became_peered":"2024-03-29T10:10:58.129783+0000","last_unstale":"2024-03-29T10:10:58.679343+0000","last_undegraded":"2024-03-29T10:10:58.679343+0000","last_fullsized":"2024-03-29T10:10:58.679343+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:35:52.111805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.668160+0000","last_change":"2024-03-29T10:10:58.665133+0000","last_active":"2024-03-29T10:10:58.668160+0000","last_peered":"2024-03-29T10:10:58.668160+0000","last_clean":"2024-03-29T10:10:58.668160+0000","last_became_active":"2024-03-29T10:10:58.664862+0000","last_became_peered":"2024-03-29T10:10:58.664862+0000","last_unstale":"2024-03-29T10:10:58.668160+0000","last_undegraded":"2024-03-29T10:10:58.668160+0000","last_fullsized":"2024-03-29T10:10:58.668160+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:08:37.164467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.653293+0000","last_change":"2024-03-29T10:10:58.125376+0000","last_active":"2024-03-29T10:10:58.653293+0000","last_peered":"2024-03-29T10:10:58.653293+0000","last_clean":"2024-03-29T10:10:58.653293+0000","last_became_active":"2024-03-29T10:10:58.125081+0000","last_became_peered":"2024-03-29T10:10:58.125081+0000","last_unstale":"2024-03-29T10:10:58.653293+0000","last_undegraded":"2024-03-29T10:10:58.653293+0000","last_fullsized":"2024-03-29T10:10:58.653293+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:53:59.168496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.677936+0000","last_change":"2024-03-29T10:10:58.672606+0000","last_active":"2024-03-29T10:10:58.677936+0000","last_peered":"2024-03-29T10:10:58.677936+0000","last_clean":"2024-03-29T10:10:58.677936+0000","last_became_active":"2024-03-29T10:10:58.672142+0000","last_became_peered":"2024-03-29T10:10:58.672142+0000","last_unstale":"2024-03-29T10:10:58.677936+0000","last_undegraded":"2024-03-29T10:10:58.677936+0000","last_fullsized":"2024-03-29T10:10:58.677936+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:33:10.033901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.640960+0000","last_change":"2024-03-29T10:10:58.127472+0000","last_active":"2024-03-29T10:10:58.640960+0000","last_peered":"2024-03-29T10:10:58.640960+0000","last_clean":"2024-03-29T10:10:58.640960+0000","last_became_active":"2024-03-29T10:10:58.126945+0000","last_became_peered":"2024-03-29T10:10:58.126945+0000","last_unstale":"2024-03-29T10:10:58.640960+0000","last_undegraded":"2024-03-29T10:10:58.640960+0000","last_fullsized":"2024-03-29T10:10:58.640960+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:30.755785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.676181+0000","last_change":"2024-03-29T10:10:58.667181+0000","last_active":"2024-03-29T10:10:58.676181+0000","last_peered":"2024-03-29T10:10:58.676181+0000","last_clean":"2024-03-29T10:10:58.676181+0000","last_became_active":"2024-03-29T10:10:58.663135+0000","last_became_peered":"2024-03-29T10:10:58.663135+0000","last_unstale":"2024-03-29T10:10:58.676181+0000","last_undegraded":"2024-03-29T10:10:58.676181+0000","last_fullsized":"2024-03-29T10:10:58.676181+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:27.747600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"107'8","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.680605+0000","last_change":"2024-03-29T10:10:58.121985+0000","last_active":"2024-03-29T10:10:58.680605+0000","last_peered":"2024-03-29T10:10:58.680605+0000","last_clean":"2024-03-29T10:10:58.680605+0000","last_became_active":"2024-03-29T10:10:58.121702+0000","last_became_peered":"2024-03-29T10:10:58.121702+0000","last_unstale":"2024-03-29T10:10:58.680605+0000","last_undegraded":"2024-03-29T10:10:58.680605+0000","last_fullsized":"2024-03-29T10:10:58.680605+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:52.743441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662351+0000","last_change":"2024-03-29T10:10:58.658104+0000","last_active":"2024-03-29T10:10:58.662351+0000","last_peered":"2024-03-29T10:10:58.662351+0000","last_clean":"2024-03-29T10:10:58.662351+0000","last_became_active":"2024-03-29T10:10:58.657810+0000","last_became_peered":"2024-03-29T10:10:58.657810+0000","last_unstale":"2024-03-29T10:10:58.662351+0000","last_undegraded":"2024-03-29T10:10:58.662351+0000","last_fullsized":"2024-03-29T10:10:58.662351+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:24.188405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"107'1","reported_seq":11,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669993+0000","last_change":"2024-03-29T10:10:58.130426+0000","last_active":"2024-03-29T10:10:58.669993+0000","last_peered":"2024-03-29T10:10:58.669993+0000","last_clean":"2024-03-29T10:10:58.669993+0000","last_became_active":"2024-03-29T10:10:58.129526+0000","last_became_peered":"2024-03-29T10:10:58.129526+0000","last_unstale":"2024-03-29T10:10:58.669993+0000","last_undegraded":"2024-03-29T10:10:58.669993+0000","last_fullsized":"2024-03-29T10:10:58.669993+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:15.723444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.667839+0000","last_change":"2024-03-29T10:10:58.656983+0000","last_active":"2024-03-29T10:10:58.667839+0000","last_peered":"2024-03-29T10:10:58.667839+0000","last_clean":"2024-03-29T10:10:58.667839+0000","last_became_active":"2024-03-29T10:10:58.656698+0000","last_became_peered":"2024-03-29T10:10:58.656698+0000","last_unstale":"2024-03-29T10:10:58.667839+0000","last_undegraded":"2024-03-29T10:10:58.667839+0000","last_fullsized":"2024-03-29T10:10:58.667839+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:54:00.175790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.679181+0000","last_change":"2024-03-29T10:10:58.129375+0000","last_active":"2024-03-29T10:10:58.679181+0000","last_peered":"2024-03-29T10:10:58.679181+0000","last_clean":"2024-03-29T10:10:58.679181+0000","last_became_active":"2024-03-29T10:10:58.129087+0000","last_became_peered":"2024-03-29T10:10:58.129087+0000","last_unstale":"2024-03-29T10:10:58.679181+0000","last_undegraded":"2024-03-29T10:10:58.679181+0000","last_fullsized":"2024-03-29T10:10:58.679181+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:30.395373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.668105+0000","last_change":"2024-03-29T10:10:58.666916+0000","last_active":"2024-03-29T10:10:58.668105+0000","last_peered":"2024-03-29T10:10:58.668105+0000","last_clean":"2024-03-29T10:10:58.668105+0000","last_became_active":"2024-03-29T10:10:58.666616+0000","last_became_peered":"2024-03-29T10:10:58.666616+0000","last_unstale":"2024-03-29T10:10:58.668105+0000","last_undegraded":"2024-03-29T10:10:58.668105+0000","last_fullsized":"2024-03-29T10:10:58.668105+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:06:03.899109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.656883+0000","last_change":"2024-03-29T10:10:58.125217+0000","last_active":"2024-03-29T10:10:58.656883+0000","last_peered":"2024-03-29T10:10:58.656883+0000","last_clean":"2024-03-29T10:10:58.656883+0000","last_became_active":"2024-03-29T10:10:58.124956+0000","last_became_peered":"2024-03-29T10:10:58.124956+0000","last_unstale":"2024-03-29T10:10:58.656883+0000","last_undegraded":"2024-03-29T10:10:58.656883+0000","last_fullsized":"2024-03-29T10:10:58.656883+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:46:22.422695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.666880+0000","last_change":"2024-03-29T10:10:58.664513+0000","last_active":"2024-03-29T10:10:58.666880+0000","last_peered":"2024-03-29T10:10:58.666880+0000","last_clean":"2024-03-29T10:10:58.666880+0000","last_became_active":"2024-03-29T10:10:58.664258+0000","last_became_peered":"2024-03-29T10:10:58.664258+0000","last_unstale":"2024-03-29T10:10:58.666880+0000","last_undegraded":"2024-03-29T10:10:58.666880+0000","last_fullsized":"2024-03-29T10:10:58.666880+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:11.077866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.674607+0000","last_change":"2024-03-29T10:10:58.133969+0000","last_active":"2024-03-29T10:10:58.674607+0000","last_peered":"2024-03-29T10:10:58.674607+0000","last_clean":"2024-03-29T10:10:58.674607+0000","last_became_active":"2024-03-29T10:10:58.132913+0000","last_became_peered":"2024-03-29T10:10:58.132913+0000","last_unstale":"2024-03-29T10:10:58.674607+0000","last_undegraded":"2024-03-29T10:10:58.674607+0000","last_fullsized":"2024-03-29T10:10:58.674607+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:49.519856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.657046+0000","last_change":"2024-03-29T10:10:58.651563+0000","last_active":"2024-03-29T10:10:58.657046+0000","last_peered":"2024-03-29T10:10:58.657046+0000","last_clean":"2024-03-29T10:10:58.657046+0000","last_became_active":"2024-03-29T10:10:58.651272+0000","last_became_peered":"2024-03-29T10:10:58.651272+0000","last_unstale":"2024-03-29T10:10:58.657046+0000","last_undegraded":"2024-03-29T10:10:58.657046+0000","last_fullsized":"2024-03-29T10:10:58.657046+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:52:18.726736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.641950+0000","last_change":"2024-03-29T10:10:58.131438+0000","last_active":"2024-03-29T10:10:58.641950+0000","last_peered":"2024-03-29T10:10:58.641950+0000","last_clean":"2024-03-29T10:10:58.641950+0000","last_became_active":"2024-03-29T10:10:58.131085+0000","last_became_peered":"2024-03-29T10:10:58.131085+0000","last_unstale":"2024-03-29T10:10:58.641950+0000","last_undegraded":"2024-03-29T10:10:58.641950+0000","last_fullsized":"2024-03-29T10:10:58.641950+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:26.740306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.660474+0000","last_change":"2024-03-29T10:10:58.646342+0000","last_active":"2024-03-29T10:10:58.660474+0000","last_peered":"2024-03-29T10:10:58.660474+0000","last_clean":"2024-03-29T10:10:58.660474+0000","last_became_active":"2024-03-29T10:10:58.646123+0000","last_became_peered":"2024-03-29T10:10:58.646123+0000","last_unstale":"2024-03-29T10:10:58.660474+0000","last_undegraded":"2024-03-29T10:10:58.660474+0000","last_fullsized":"2024-03-29T10:10:58.660474+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:37.206911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.641418+0000","last_change":"2024-03-29T10:10:58.127054+0000","last_active":"2024-03-29T10:10:58.641418+0000","last_peered":"2024-03-29T10:10:58.641418+0000","last_clean":"2024-03-29T10:10:58.641418+0000","last_became_active":"2024-03-29T10:10:58.125738+0000","last_became_peered":"2024-03-29T10:10:58.125738+0000","last_unstale":"2024-03-29T10:10:58.641418+0000","last_undegraded":"2024-03-29T10:10:58.641418+0000","last_fullsized":"2024-03-29T10:10:58.641418+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:47:46.853250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.658909+0000","last_change":"2024-03-29T10:10:58.655398+0000","last_active":"2024-03-29T10:10:58.658909+0000","last_peered":"2024-03-29T10:10:58.658909+0000","last_clean":"2024-03-29T10:10:58.658909+0000","last_became_active":"2024-03-29T10:10:58.655074+0000","last_became_peered":"2024-03-29T10:10:58.655074+0000","last_unstale":"2024-03-29T10:10:58.658909+0000","last_undegraded":"2024-03-29T10:10:58.658909+0000","last_fullsized":"2024-03-29T10:10:58.658909+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:28.681653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.653234+0000","last_change":"2024-03-29T10:10:58.132443+0000","last_active":"2024-03-29T10:10:58.653234+0000","last_peered":"2024-03-29T10:10:58.653234+0000","last_clean":"2024-03-29T10:10:58.653234+0000","last_became_active":"2024-03-29T10:10:58.131885+0000","last_became_peered":"2024-03-29T10:10:58.131885+0000","last_unstale":"2024-03-29T10:10:58.653234+0000","last_undegraded":"2024-03-29T10:10:58.653234+0000","last_fullsized":"2024-03-29T10:10:58.653234+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:14.232608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.652160+0000","last_change":"2024-03-29T10:10:58.650455+0000","last_active":"2024-03-29T10:10:58.652160+0000","last_peered":"2024-03-29T10:10:58.652160+0000","last_clean":"2024-03-29T10:10:58.652160+0000","last_became_active":"2024-03-29T10:10:58.649269+0000","last_became_peered":"2024-03-29T10:10:58.649269+0000","last_unstale":"2024-03-29T10:10:58.652160+0000","last_undegraded":"2024-03-29T10:10:58.652160+0000","last_fullsized":"2024-03-29T10:10:58.652160+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:33.106330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"107'2","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.678806+0000","last_change":"2024-03-29T10:10:58.126002+0000","last_active":"2024-03-29T10:10:58.678806+0000","last_peered":"2024-03-29T10:10:58.678806+0000","last_clean":"2024-03-29T10:10:58.678806+0000","last_became_active":"2024-03-29T10:10:58.125619+0000","last_became_peered":"2024-03-29T10:10:58.125619+0000","last_unstale":"2024-03-29T10:10:58.678806+0000","last_undegraded":"2024-03-29T10:10:58.678806+0000","last_fullsized":"2024-03-29T10:10:58.678806+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:31:24.045982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669770+0000","last_change":"2024-03-29T10:10:58.667794+0000","last_active":"2024-03-29T10:10:58.669770+0000","last_peered":"2024-03-29T10:10:58.669770+0000","last_clean":"2024-03-29T10:10:58.669770+0000","last_became_active":"2024-03-29T10:10:58.667461+0000","last_became_peered":"2024-03-29T10:10:58.667461+0000","last_unstale":"2024-03-29T10:10:58.669770+0000","last_undegraded":"2024-03-29T10:10:58.669770+0000","last_fullsized":"2024-03-29T10:10:58.669770+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:43:18.124198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.660040+0000","last_change":"2024-03-29T10:10:58.132739+0000","last_active":"2024-03-29T10:10:58.660040+0000","last_peered":"2024-03-29T10:10:58.660040+0000","last_clean":"2024-03-29T10:10:58.660040+0000","last_became_active":"2024-03-29T10:10:58.130371+0000","last_became_peered":"2024-03-29T10:10:58.130371+0000","last_unstale":"2024-03-29T10:10:58.660040+0000","last_undegraded":"2024-03-29T10:10:58.660040+0000","last_fullsized":"2024-03-29T10:10:58.660040+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:22:26.419714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.665157+0000","last_change":"2024-03-29T10:10:58.662756+0000","last_active":"2024-03-29T10:10:58.665157+0000","last_peered":"2024-03-29T10:10:58.665157+0000","last_clean":"2024-03-29T10:10:58.665157+0000","last_became_active":"2024-03-29T10:10:58.662372+0000","last_became_peered":"2024-03-29T10:10:58.662372+0000","last_unstale":"2024-03-29T10:10:58.665157+0000","last_undegraded":"2024-03-29T10:10:58.665157+0000","last_fullsized":"2024-03-29T10:10:58.665157+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:37:57.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.641268+0000","last_change":"2024-03-29T10:10:58.131798+0000","last_active":"2024-03-29T10:10:58.641268+0000","last_peered":"2024-03-29T10:10:58.641268+0000","last_clean":"2024-03-29T10:10:58.641268+0000","last_became_active":"2024-03-29T10:10:58.131481+0000","last_became_peered":"2024-03-29T10:10:58.131481+0000","last_unstale":"2024-03-29T10:10:58.641268+0000","last_undegraded":"2024-03-29T10:10:58.641268+0000","last_fullsized":"2024-03-29T10:10:58.641268+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:27.674359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662251+0000","last_change":"2024-03-29T10:10:58.659207+0000","last_active":"2024-03-29T10:10:58.662251+0000","last_peered":"2024-03-29T10:10:58.662251+0000","last_clean":"2024-03-29T10:10:58.662251+0000","last_became_active":"2024-03-29T10:10:58.658931+0000","last_became_peered":"2024-03-29T10:10:58.658931+0000","last_unstale":"2024-03-29T10:10:58.662251+0000","last_undegraded":"2024-03-29T10:10:58.662251+0000","last_fullsized":"2024-03-29T10:10:58.662251+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:53.750734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.643154+0000","last_change":"2024-03-29T10:10:58.130295+0000","last_active":"2024-03-29T10:10:58.643154+0000","last_peered":"2024-03-29T10:10:58.643154+0000","last_clean":"2024-03-29T10:10:58.643154+0000","last_became_active":"2024-03-29T10:10:58.129993+0000","last_became_peered":"2024-03-29T10:10:58.129993+0000","last_unstale":"2024-03-29T10:10:58.643154+0000","last_undegraded":"2024-03-29T10:10:58.643154+0000","last_fullsized":"2024-03-29T10:10:58.643154+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:05:53.862156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.657418+0000","last_change":"2024-03-29T10:10:58.653696+0000","last_active":"2024-03-29T10:10:58.657418+0000","last_peered":"2024-03-29T10:10:58.657418+0000","last_clean":"2024-03-29T10:10:58.657418+0000","last_became_active":"2024-03-29T10:10:58.652283+0000","last_became_peered":"2024-03-29T10:10:58.652283+0000","last_unstale":"2024-03-29T10:10:58.657418+0000","last_undegraded":"2024-03-29T10:10:58.657418+0000","last_fullsized":"2024-03-29T10:10:58.657418+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:21.226850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.655621+0000","last_change":"2024-03-29T10:10:58.124487+0000","last_active":"2024-03-29T10:10:58.655621+0000","last_peered":"2024-03-29T10:10:58.655621+0000","last_clean":"2024-03-29T10:10:58.655621+0000","last_became_active":"2024-03-29T10:10:58.124222+0000","last_became_peered":"2024-03-29T10:10:58.124222+0000","last_unstale":"2024-03-29T10:10:58.655621+0000","last_undegraded":"2024-03-29T10:10:58.655621+0000","last_fullsized":"2024-03-29T10:10:58.655621+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:06:02.891816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.669295+0000","last_change":"2024-03-29T10:10:58.659988+0000","last_active":"2024-03-29T10:10:58.669295+0000","last_peered":"2024-03-29T10:10:58.669295+0000","last_clean":"2024-03-29T10:10:58.669295+0000","last_became_active":"2024-03-29T10:10:58.658255+0000","last_became_peered":"2024-03-29T10:10:58.658255+0000","last_unstale":"2024-03-29T10:10:58.669295+0000","last_undegraded":"2024-03-29T10:10:58.669295+0000","last_fullsized":"2024-03-29T10:10:58.669295+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:13.760911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.647426+0000","last_change":"2024-03-29T10:10:58.125726+0000","last_active":"2024-03-29T10:10:58.647426+0000","last_peered":"2024-03-29T10:10:58.647426+0000","last_clean":"2024-03-29T10:10:58.647426+0000","last_became_active":"2024-03-29T10:10:58.125464+0000","last_became_peered":"2024-03-29T10:10:58.125464+0000","last_unstale":"2024-03-29T10:10:58.647426+0000","last_undegraded":"2024-03-29T10:10:58.647426+0000","last_fullsized":"2024-03-29T10:10:58.647426+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:36.427259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.675106+0000","last_change":"2024-03-29T10:10:58.671053+0000","last_active":"2024-03-29T10:10:58.675106+0000","last_peered":"2024-03-29T10:10:58.675106+0000","last_clean":"2024-03-29T10:10:58.675106+0000","last_became_active":"2024-03-29T10:10:58.670765+0000","last_became_peered":"2024-03-29T10:10:58.670765+0000","last_unstale":"2024-03-29T10:10:58.675106+0000","last_undegraded":"2024-03-29T10:10:58.675106+0000","last_fullsized":"2024-03-29T10:10:58.675106+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:06.388556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.641944+0000","last_change":"2024-03-29T10:10:58.126774+0000","last_active":"2024-03-29T10:10:58.641944+0000","last_peered":"2024-03-29T10:10:58.641944+0000","last_clean":"2024-03-29T10:10:58.641944+0000","last_became_active":"2024-03-29T10:10:58.126502+0000","last_became_peered":"2024-03-29T10:10:58.126502+0000","last_unstale":"2024-03-29T10:10:58.641944+0000","last_undegraded":"2024-03-29T10:10:58.641944+0000","last_fullsized":"2024-03-29T10:10:58.641944+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:35:36.199617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.653369+0000","last_change":"2024-03-29T10:10:58.648665+0000","last_active":"2024-03-29T10:10:58.653369+0000","last_peered":"2024-03-29T10:10:58.653369+0000","last_clean":"2024-03-29T10:10:58.653369+0000","last_became_active":"2024-03-29T10:10:58.648405+0000","last_became_peered":"2024-03-29T10:10:58.648405+0000","last_unstale":"2024-03-29T10:10:58.653369+0000","last_undegraded":"2024-03-29T10:10:58.653369+0000","last_fullsized":"2024-03-29T10:10:58.653369+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:15.239902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.647197+0000","last_change":"2024-03-29T10:10:58.123574+0000","last_active":"2024-03-29T10:10:58.647197+0000","last_peered":"2024-03-29T10:10:58.647197+0000","last_clean":"2024-03-29T10:10:58.647197+0000","last_became_active":"2024-03-29T10:10:58.123296+0000","last_became_peered":"2024-03-29T10:10:58.123296+0000","last_unstale":"2024-03-29T10:10:58.647197+0000","last_undegraded":"2024-03-29T10:10:58.647197+0000","last_fullsized":"2024-03-29T10:10:58.647197+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:30.755785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.662708+0000","last_change":"2024-03-29T10:10:58.658470+0000","last_active":"2024-03-29T10:10:58.662708+0000","last_peered":"2024-03-29T10:10:58.662708+0000","last_clean":"2024-03-29T10:10:58.662708+0000","last_became_active":"2024-03-29T10:10:58.658237+0000","last_became_peered":"2024-03-29T10:10:58.658237+0000","last_unstale":"2024-03-29T10:10:58.662708+0000","last_undegraded":"2024-03-29T10:10:58.662708+0000","last_fullsized":"2024-03-29T10:10:58.662708+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:58.101824+0000","last_clean_scrub_stamp":"2024-03-29T10:10:58.101824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:03:25.171304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"107'1","reported_seq":11,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.679216+0000","last_change":"2024-03-29T10:10:58.130206+0000","last_active":"2024-03-29T10:10:58.679216+0000","last_peered":"2024-03-29T10:10:58.679216+0000","last_clean":"2024-03-29T10:10:58.679216+0000","last_became_active":"2024-03-29T10:10:58.129948+0000","last_became_peered":"2024-03-29T10:10:58.129948+0000","last_unstale":"2024-03-29T10:10:58.679216+0000","last_undegraded":"2024-03-29T10:10:58.679216+0000","last_fullsized":"2024-03-29T10:10:58.679216+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:10:57.094530+0000","last_clean_scrub_stamp":"2024-03-29T10:10:57.094530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:30.395373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"97'201","reported_seq":544,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-29T10:10:58.121556+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:10:58.121556+0000","last_peered":"2024-03-29T10:10:58.121556+0000","last_clean":"2024-03-29T10:10:58.121556+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:10:58.121556+0000","last_undegraded":"2024-03-29T10:10:58.121556+0000","last_fullsized":"2024-03-29T10:10:58.121556+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":311,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":311,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":1912832,"data_stored":1214462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":31848908},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.68100000000000005,"15min":0.71799999999999997},"min":{"1min":0.47899999999999998,"5min":0.44600000000000001,"15min":0.32500000000000001},"max":{"1min":0.82199999999999995,"5min":0.98699999999999999,"15min":1.0109999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68300000000000005,"15min":0.71599999999999997},"min":{"1min":0.52700000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.60699999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.70599999999999996,"15min":0.72599999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.372},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.746}]},{"osd":2,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.72099999999999997,"15min":0.74099999999999999},"min":{"1min":0.501,"5min":0.45200000000000001,"15min":0.40999999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73299999999999998,"15min":0.754},"min":{"1min":0.49299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.70099999999999996,"15min":0.73799999999999999},"min":{"1min":0.48499999999999999,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.745,"15min":0.77800000000000002},"min":{"1min":0.52300000000000002,"5min":0.50700000000000001,"15min":0.44500000000000001},"max":{"1min":0.91200000000000003,"5min":1.129,"15min":1.129},"last":0.57099999999999995}]},{"osd":4,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.40500000000000003,"15min":0.29199999999999998},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.441},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.63800000000000001,"15min":0.63500000000000001},"min":{"1min":0.42899999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.505}]},{"osd":5,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.43099999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.50800000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.72199999999999998}]},{"osd":6,"last update":"Fri Mar 29 10:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.49199999999999999},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.71199999999999997,"15min":0.70299999999999996},"min":{"1min":0.51100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.64000000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1323008,"data_stored":624628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70499999999999996,"15min":0.70699999999999996},"min":{"1min":0.436,"5min":0.42899999999999999,"15min":0.34699999999999998},"max":{"1min":0.92000000000000004,"5min":2.7770000000000001,"15min":2.7770000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.72399999999999998},"min":{"1min":0.46400000000000002,"5min":0.435,"15min":0.42499999999999999},"max":{"1min":0.83799999999999997,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.65400000000000003}]},{"osd":1,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":0.70699999999999996},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.81799999999999995,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.49199999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.80600000000000005,"5min":2.6709999999999998,"15min":2.6709999999999998},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.67200000000000004,"15min":0.71099999999999997},"min":{"1min":0.45100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":1.256,"15min":1.256},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.71799999999999997,"15min":0.74299999999999999},"min":{"1min":0.48199999999999998,"5min":0.45700000000000002,"15min":0.40400000000000003},"max":{"1min":0.88100000000000001,"5min":1.454,"15min":1.454},"last":0.69999999999999996}]},{"osd":3,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.69499999999999995,"15min":0.74199999999999999},"min":{"1min":0.496,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81699999999999995,"5min":1.131,"15min":1.145},"last":0.496},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.76700000000000002,"15min":0.76800000000000002},"min":{"1min":0.60999999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85099999999999998,"5min":2.718,"15min":2.718},"last":0.624}]},{"osd":4,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.41799999999999998,"5min":0.35899999999999999,"15min":0.35199999999999998},"max":{"1min":0.88200000000000001,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.46999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93799999999999994,"5min":2.544,"15min":2.544},"last":0.58799999999999997}]},{"osd":5,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67300000000000004,"15min":0.65800000000000003},"min":{"1min":0.40300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.91800000000000004,"5min":3.2280000000000002,"15min":3.2280000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72299999999999998,"15min":0.68899999999999995},"min":{"1min":0.45600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89100000000000001,"5min":2.835,"15min":2.835},"last":0.67000000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.755,"15min":0.753},"min":{"1min":0.499,"5min":0.48899999999999999,"15min":0.45100000000000001},"max":{"1min":0.98799999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72799999999999998,"15min":0.72199999999999998},"min":{"1min":0.55200000000000005,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.96099999999999997,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.64100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575042,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50468,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321372,"statfs":{"total":96636764160,"available":96585084928,"internally_reserved":0,"allocated":1327104,"data_stored":624218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.33700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.79100000000000004,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65600000000000003,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.34399999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.40000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.222,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.438},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.39100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56499999999999995,"15min":0.58399999999999996},"min":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.90800000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.026,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.747,"15min":0.76200000000000001},"min":{"1min":0.57499999999999996,"5min":0.497,"15min":0.497},"max":{"1min":1.2709999999999999,"5min":1.617,"15min":1.617},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.75,"15min":0.77100000000000002},"min":{"1min":0.54900000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0109999999999999,"5min":1.1419999999999999,"15min":1.155},"last":0.77900000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76700000000000002,"15min":0.78600000000000003},"min":{"1min":0.56799999999999995,"5min":0.56699999999999995,"15min":0.51800000000000002},"max":{"1min":0.99299999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.55800000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3140000000000001,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.69199999999999995}]},{"osd":6,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.76100000000000001,"15min":0.79000000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.99399999999999999,"5min":1.0840000000000001,"15min":1.236},"last":0.746},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.78400000000000003,"15min":0.81000000000000005},"min":{"1min":0.61299999999999999,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1579999999999999,"5min":1.468,"15min":1.468},"last":0.65100000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.81100000000000005,"15min":0.83399999999999996},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.1319999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80200000000000005,"15min":0.82999999999999996},"min":{"1min":0.61699999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.4530000000000001,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.66000000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575044,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":1908736,"data_stored":1213407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.433,"15min":0.44900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.23999999999999999},"max":{"1min":0.58299999999999996,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.251},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.50700000000000001},"min":{"1min":0.31,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.86699999999999999,"5min":0.94899999999999995,"15min":1.0529999999999999},"last":0.39300000000000002}]},{"osd":2,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52700000000000002,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73399999999999999,"5min":0.80700000000000005,"15min":1.0069999999999999},"last":0.439},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.45800000000000002,"15min":0.495},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.97199999999999998},"last":0.41499999999999998}]},{"osd":3,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.54400000000000004},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":1.077},"last":0.373},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57999999999999996,"15min":0.60899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73699999999999999,"5min":0.83499999999999996,"15min":1.0549999999999999},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.747,"15min":0.79000000000000004},"min":{"1min":0.54600000000000004,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":1.0409999999999999,"5min":1.625,"15min":1.625},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.76300000000000001,"15min":0.81499999999999995},"min":{"1min":0.627,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":0.95799999999999996,"5min":0.97699999999999998,"15min":1.167},"last":0.69999999999999996}]},{"osd":5,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69999999999999996,"15min":0.748},"min":{"1min":0.48799999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.137},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.751,"15min":0.79500000000000004},"min":{"1min":0.57599999999999996,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.64400000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.81200000000000006},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.999,"5min":1.0640000000000001,"15min":1.228},"last":0.749},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.79100000000000004,"15min":0.83199999999999996},"min":{"1min":0.53400000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.077,"5min":1.601,"15min":1.601},"last":0.57199999999999995}]},{"osd":7,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74199999999999999,"15min":0.79000000000000004},"min":{"1min":0.59799999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96199999999999997,"5min":1.004,"15min":1.1890000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.78800000000000003,"15min":0.83099999999999996},"min":{"1min":0.59799999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.62}]}]},{"osd":2,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40596,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331244,"statfs":{"total":96636764160,"available":96595193856,"internally_reserved":0,"allocated":1318912,"data_stored":624606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.43099999999999999,"15min":0.44400000000000001},"min":{"1min":0.30199999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.59499999999999997,"5min":0.746,"15min":0.86099999999999999},"last":0.374},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.50900000000000001},"min":{"1min":0.33200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69299999999999995,"5min":0.73699999999999999,"15min":0.82399999999999995},"last":0.39700000000000002}]},{"osd":1,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48199999999999998,"15min":0.47399999999999998},"min":{"1min":0.315,"5min":0.29199999999999998,"15min":0.26200000000000001},"max":{"1min":0.63600000000000001,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54200000000000004,"15min":0.55200000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32700000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":1.032},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70599999999999996,"5min":0.749,"15min":0.99099999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53800000000000003,"15min":0.54500000000000004},"min":{"1min":0.40200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82699999999999996,"5min":1.516,"15min":1.516},"last":0.42099999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.755,"15min":0.77900000000000003},"min":{"1min":0.51100000000000001,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":1.125,"5min":1.125,"15min":1.3520000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.75800000000000001,"15min":0.80000000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53100000000000003},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.85399999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76900000000000002,"15min":0.80100000000000005},"min":{"1min":0.58299999999999996,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.2589999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.752,"15min":0.79700000000000004},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.115,"5min":1.115,"15min":1.2250000000000001},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77900000000000003,"15min":0.81200000000000006},"min":{"1min":0.56100000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.2350000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.79600000000000004,"15min":0.83299999999999996},"min":{"1min":0.58599999999999997,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0860000000000001,"5min":1.1120000000000001,"15min":1.3560000000000001},"last":0.83899999999999997}]},{"osd":7,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.82999999999999996,"15min":0.85899999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.53400000000000003},"max":{"1min":1.258,"5min":1.258,"15min":1.4490000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.83199999999999996,"15min":0.874},"min":{"1min":0.60899999999999999,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.226,"5min":1.226,"15min":1.4199999999999999},"last":0.79300000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575043,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":1302528,"data_stored":617249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.441,"15min":0.45100000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26000000000000001},"max":{"1min":0.68200000000000005,"5min":0.75600000000000001,"15min":0.81299999999999994},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.373,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69699999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.40600000000000003}]},{"osd":1,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.46500000000000002,"15min":0.48399999999999999},"min":{"1min":0.313,"5min":0.30499999999999999,"15min":0.29799999999999999},"max":{"1min":0.51800000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.372},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.314},"max":{"1min":0.82699999999999996,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48899999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54200000000000004,"15min":0.55700000000000005},"min":{"1min":0.40600000000000003,"5min":0.373,"15min":0.34200000000000003},"max":{"1min":0.75800000000000001,"5min":0.83999999999999997,"15min":0.98299999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60199999999999998,"15min":0.625},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.86899999999999999,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.46700000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.76200000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.034,"5min":1.0940000000000001,"15min":1.288},"last":0.998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.77200000000000002},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.46},"last":1.093}]},{"osd":5,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.748,"15min":0.78900000000000003},"min":{"1min":0.53800000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95199999999999996,"5min":1.075,"15min":1.3580000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.753,"15min":0.79300000000000004},"min":{"1min":0.50700000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.429},"last":1.1379999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77300000000000002,"15min":0.80600000000000005},"min":{"1min":0.57799999999999996,"5min":0.54200000000000004,"15min":0.53400000000000003},"max":{"1min":1.085,"5min":1.101,"15min":1.4730000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77700000000000002,"15min":0.81299999999999994},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.056,"5min":1.056,"15min":1.3999999999999999},"last":1.0309999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79700000000000004,"15min":0.82999999999999996},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.54300000000000004},"max":{"1min":1.103,"5min":1.135,"15min":1.5029999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":0.83199999999999996},"min":{"1min":0.63700000000000001,"5min":0.58899999999999997,"15min":0.56200000000000006},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.345},"last":0.80800000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575044,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58648,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313192,"statfs":{"total":96636764160,"available":96576708608,"internally_reserved":0,"allocated":1314816,"data_stored":624024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65700000000000003,"15min":0.69199999999999995},"min":{"1min":0.35399999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.84399999999999997,"5min":0.91400000000000003,"15min":1.417},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.67300000000000004,"15min":0.69899999999999995},"min":{"1min":0.434,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.93200000000000005,"5min":0.93899999999999995,"15min":1.165},"last":0.59399999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.66200000000000003,"15min":0.69999999999999996},"min":{"1min":0.48099999999999998,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.89300000000000002,"5min":0.93700000000000006,"15min":1.014},"last":0.628},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67000000000000004,"15min":0.70699999999999996},"min":{"1min":0.498,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83599999999999997,"5min":0.98199999999999998,"15min":1.333},"last":0.76500000000000001}]},{"osd":2,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.747,"15min":0.76100000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.41299999999999998},"max":{"1min":0.91700000000000004,"5min":0.99399999999999999,"15min":1.03},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73599999999999999,"15min":0.76700000000000002},"min":{"1min":0.50600000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91600000000000004,"5min":0.96299999999999997,"15min":1.387},"last":0.80300000000000005}]},{"osd":3,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.70099999999999996,"15min":0.71999999999999997},"min":{"1min":0.495,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95999999999999996,"5min":0.98299999999999998,"15min":1.2350000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70599999999999996,"15min":0.73299999999999998},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.95199999999999996,"5min":0.95599999999999996,"15min":1.3180000000000001},"last":0.70199999999999996}]},{"osd":5,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59799999999999998,"15min":0.58599999999999997},"min":{"1min":0.436,"5min":0.39300000000000002,"15min":0.38100000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.60299999999999998,"15min":0.60399999999999998},"min":{"1min":0.48999999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.89500000000000002,"5min":1.0169999999999999,"15min":1.294},"last":0.77700000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.629,"15min":0.61599999999999999},"min":{"1min":0.45600000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.874,"5min":0.96999999999999997,"15min":1.0129999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.1359999999999999},"last":0.72799999999999998}]},{"osd":7,"last update":"Fri Mar 29 10:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67700000000000005,"15min":0.68300000000000005},"min":{"1min":0.54000000000000004,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.97199999999999998,"5min":1.0489999999999999,"15min":1.345},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.754,"15min":0.755},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.998,"5min":0.998,"15min":1.214},"last":0.83099999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36516,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335324,"statfs":{"total":96636764160,"available":96599371776,"internally_reserved":0,"allocated":1331200,"data_stored":625610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.64500000000000002,"15min":0.68100000000000005},"min":{"1min":0.44400000000000001,"5min":0.39200000000000002,"15min":0.313},"max":{"1min":0.754,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.71499999999999997,"15min":0.72499999999999998},"min":{"1min":0.47899999999999998,"5min":0.435,"15min":0.33500000000000002},"max":{"1min":0.83399999999999996,"5min":1.21,"15min":1.339},"last":0.56599999999999995}]},{"osd":1,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.69199999999999995,"15min":0.71699999999999997},"min":{"1min":0.47599999999999998,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.83799999999999997,"5min":0.96699999999999997,"15min":1.0169999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.68400000000000005,"15min":0.70599999999999996},"min":{"1min":0.441,"5min":0.441,"15min":0.39200000000000002},"max":{"1min":0.76500000000000001,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.70899999999999996,"15min":0.72799999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.77400000000000002,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68600000000000005,"15min":0.70599999999999996},"min":{"1min":0.46999999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72099999999999997,"5min":1.095,"15min":1.095},"last":0.54000000000000004}]},{"osd":3,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.70299999999999996,"15min":0.73199999999999998},"min":{"1min":0.5,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":0.83499999999999996,"5min":1.113,"15min":1.1619999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.72099999999999997,"15min":0.751},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.91100000000000003,"5min":1.0249999999999999,"15min":1.4850000000000001},"last":0.60299999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59599999999999997,"15min":0.59799999999999998},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.79500000000000004,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64800000000000002,"15min":0.64100000000000001},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":0.86799999999999999,"5min":1.097,"15min":1.214},"last":0.70199999999999996}]},{"osd":6,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.64700000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.36799999999999999},"max":{"1min":0.81499999999999995,"5min":0.91300000000000003,"15min":1.151},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82099999999999995,"5min":0.97099999999999997,"15min":1.095},"last":0.66400000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.69999999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89300000000000002,"5min":0.95299999999999996,"15min":1.325},"last":0.749},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.72699999999999998,"15min":0.73099999999999998},"min":{"1min":0.45600000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.1990000000000001},"last":0.76700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:11:04.057 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T10:11:04.057 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T10:11:04.092 INFO:tasks.ceph.mds.c.smithi012.stderr:2024-03-29T10:11:04.093+0000 7f7111600700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:11:04.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.298+0000 7fc19b800700 1 Processor -- start 2024-03-29T10:11:04.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.298+0000 7fc19b800700 1 -- start start 2024-03-29T10:11:04.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.299+0000 7fc19b800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1940d16a0 0x7fc1940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.299+0000 7fc19b800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc1940d2370 0x7fc1940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc19b800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc194161420 con 0x7fc19415d9d0 2024-03-29T10:11:04.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc19b800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1940d4340 con 0x7fc1940d16a0 2024-03-29T10:11:04.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc19b800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19415edc0 con 0x7fc1940d2370 2024-03-29T10:11:04.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc193e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1940d16a0 0x7fc1940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc198e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc19415d9d0 0x7fc1940d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60782/0 (socket says 172.21.15.12:60782) 2024-03-29T10:11:04.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc198e00700 1 -- 172.21.15.12:0/2560848698 learned_addr learned my addr 172.21.15.12:0/2560848698 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:11:04.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc199800700 1 --2- 172.21.15.12:0/2560848698 >> v2:172.21.15.159:3300/0 conn(0x7fc1940d2370 0x7fc1940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc193e00700 1 -- 172.21.15.12:0/2560848698 >> v2:172.21.15.159:3300/0 conn(0x7fc1940d2370 msgr2=0x7fc1940ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc193e00700 1 --2- 172.21.15.12:0/2560848698 >> v2:172.21.15.159:3300/0 conn(0x7fc1940d2370 0x7fc1940ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc193e00700 1 -- 172.21.15.12:0/2560848698 >> v1:172.21.15.12:6789/0 conn(0x7fc19415d9d0 legacy=0x7fc1940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc193e00700 1 -- 172.21.15.12:0/2560848698 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194161700 con 0x7fc1940d16a0 2024-03-29T10:11:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.300+0000 7fc193400700 1 -- 172.21.15.12:0/2560848698 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737319797 0 0) 0x7fc194161420 con 0x7fc19415d9d0 2024-03-29T10:11:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.301+0000 7fc193e00700 1 --2- 172.21.15.12:0/2560848698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d16a0 0x7fc1940d1e30 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc19408ca10 tx=0x7fc180009b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd681c3106890f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.301+0000 7fc193400700 1 -- 172.21.15.12:0/2560848698 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180011830 con 0x7fc1940d16a0 2024-03-29T10:11:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.301+0000 7fc193400700 1 -- 172.21.15.12:0/2560848698 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc180011990 con 0x7fc1940d16a0 2024-03-29T10:11:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.301+0000 7fc193400700 1 -- 172.21.15.12:0/2560848698 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180011d40 con 0x7fc1940d16a0 2024-03-29T10:11:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.301+0000 7fc19b800700 1 -- 172.21.15.12:0/2560848698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d16a0 msgr2=0x7fc1940d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.301+0000 7fc19b800700 1 --2- 172.21.15.12:0/2560848698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d16a0 0x7fc1940d1e30 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc19408ca10 tx=0x7fc180009b10 comp rx=0 tx=0).stop 2024-03-29T10:11:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.302+0000 7fc19b800700 1 -- 172.21.15.12:0/2560848698 shutdown_connections 2024-03-29T10:11:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.302+0000 7fc19b800700 1 --2- 172.21.15.12:0/2560848698 >> v2:172.21.15.159:3300/0 conn(0x7fc1940d2370 0x7fc1940ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.302+0000 7fc19b800700 1 --2- 172.21.15.12:0/2560848698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d16a0 0x7fc1940d1e30 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.302+0000 7fc19b800700 1 -- 172.21.15.12:0/2560848698 >> 172.21.15.12:0/2560848698 conn(0x7fc19408aff0 msgr2=0x7fc1940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.302+0000 7fc19b800700 1 -- 172.21.15.12:0/2560848698 shutdown_connections 2024-03-29T10:11:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.302+0000 7fc19b800700 1 -- 172.21.15.12:0/2560848698 wait complete. 2024-03-29T10:11:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.303+0000 7fc19b800700 1 Processor -- start 2024-03-29T10:11:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.304+0000 7fc19b800700 1 -- start start 2024-03-29T10:11:04.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.304+0000 7fc19b800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1940d16a0 0x7fc1941f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc19b800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d2370 0x7fc19415e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc19b800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19415d9d0 0x7fc1941f0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc19b800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1940a1040 con 0x7fc1940d2370 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc19b800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941f42b0 con 0x7fc1940d16a0 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc19b800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941f77d0 con 0x7fc19415d9d0 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc193e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d2370 0x7fc19415e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc193e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d2370 0x7fc19415e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39496/0 (socket says 172.21.15.12:39496) 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc193e00700 1 -- 172.21.15.12:0/2075643517 learned_addr learned my addr 172.21.15.12:0/2075643517 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc198e00700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1940d16a0 0x7fc1941f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc199800700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19415d9d0 0x7fc1941f0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc193e00700 1 -- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19415d9d0 msgr2=0x7fc1941f0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc193e00700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19415d9d0 0x7fc1941f0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc193e00700 1 -- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1940d16a0 msgr2=0x7fc1941f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc193e00700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1940d16a0 0x7fc1941f7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.305+0000 7fc193e00700 1 -- 172.21.15.12:0/2075643517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc180009440 con 0x7fc1940d2370 2024-03-29T10:11:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.306+0000 7fc193e00700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d2370 0x7fc19415e020 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc19408c230 tx=0x7fc180004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.306+0000 7fc190c00700 1 -- 172.21.15.12:0/2075643517 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1800042f0 con 0x7fc1940d2370 2024-03-29T10:11:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.306+0000 7fc190c00700 1 -- 172.21.15.12:0/2075643517 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc180004450 con 0x7fc1940d2370 2024-03-29T10:11:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.306+0000 7fc19b800700 1 -- 172.21.15.12:0/2075643517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1941f3630 con 0x7fc1940d2370 2024-03-29T10:11:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.306+0000 7fc190c00700 1 -- 172.21.15.12:0/2075643517 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1800038a0 con 0x7fc1940d2370 2024-03-29T10:11:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.306+0000 7fc19b800700 1 -- 172.21.15.12:0/2075643517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1941f3bf0 con 0x7fc1940d2370 2024-03-29T10:11:04.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.307+0000 7fc19b800700 1 -- 172.21.15.12:0/2075643517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1940d1ca0 con 0x7fc1940d2370 2024-03-29T10:11:04.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.308+0000 7fc190c00700 1 -- 172.21.15.12:0/2075643517 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc180022030 con 0x7fc1940d2370 2024-03-29T10:11:04.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.309+0000 7fc190c00700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17406adb0 0x7fc17406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.309+0000 7fc190c00700 1 -- 172.21.15.12:0/2075643517 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc18008aa60 con 0x7fc1940d2370 2024-03-29T10:11:04.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.309+0000 7fc198e00700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17406adb0 0x7fc17406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.310+0000 7fc198e00700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17406adb0 0x7fc17406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc188000f60 tx=0x7fc188007e70 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.313+0000 7fc190c00700 1 -- 172.21.15.12:0/2075643517 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc180055c40 con 0x7fc1940d2370 2024-03-29T10:11:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.614+0000 7fc19b800700 1 -- 172.21.15.12:0/2075643517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc194067720 con 0x7fc1940d2370 2024-03-29T10:11:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.619+0000 7fc190c00700 1 -- 172.21.15.12:0/2075643517 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7fc1800150e0 con 0x7fc1940d2370 2024-03-29T10:11:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.621+0000 7fc19b800700 1 -- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17406adb0 msgr2=0x7fc17406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.621+0000 7fc19b800700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17406adb0 0x7fc17406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc188000f60 tx=0x7fc188007e70 comp rx=0 tx=0).stop 2024-03-29T10:11:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.621+0000 7fc19b800700 1 -- 172.21.15.12:0/2075643517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d2370 msgr2=0x7fc19415e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.621+0000 7fc19b800700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d2370 0x7fc19415e020 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc19408c230 tx=0x7fc180004750 comp rx=0 tx=0).stop 2024-03-29T10:11:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.622+0000 7fc19b800700 1 -- 172.21.15.12:0/2075643517 shutdown_connections 2024-03-29T10:11:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.622+0000 7fc19b800700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc17406adb0 0x7fc17406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.622+0000 7fc19b800700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc19415d9d0 0x7fc1941f0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.622+0000 7fc19b800700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1940d16a0 0x7fc1941f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.622+0000 7fc19b800700 1 --2- 172.21.15.12:0/2075643517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1940d2370 0x7fc19415e020 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.622+0000 7fc19b800700 1 -- 172.21.15.12:0/2075643517 >> 172.21.15.12:0/2075643517 conn(0x7fc19408aff0 msgr2=0x7fc1940b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:04.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.622+0000 7fc19b800700 1 -- 172.21.15.12:0/2075643517 shutdown_connections 2024-03-29T10:11:04.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.622+0000 7fc19b800700 1 -- 172.21.15.12:0/2075643517 wait complete. 2024-03-29T10:11:04.623 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T10:11:04.640 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:11:04.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.838+0000 7f1c7ae00700 1 Processor -- start 2024-03-29T10:11:04.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.839+0000 7f1c7ae00700 1 -- start start 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.840+0000 7f1c7ae00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1c7415d640 0x7f1c740d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.840+0000 7f1c7ae00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c740ceec0 0x7f1c741612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.840+0000 7f1c7ae00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c740d30d0 con 0x7f1c74161800 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.840+0000 7f1c7ae00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c740b6840 con 0x7f1c740ceec0 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.840+0000 7f1c7ae00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c74161c20 con 0x7f1c7415d640 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.840+0000 7f1c73400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c740ceec0 0x7f1c741612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.840+0000 7f1c73400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c740ceec0 0x7f1c741612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39506/0 (socket says 172.21.15.12:39506) 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.840+0000 7f1c73400700 1 -- 172.21.15.12:0/2002473296 learned_addr learned my addr 172.21.15.12:0/2002473296 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.840+0000 7f1c73e00700 1 --2- 172.21.15.12:0/2002473296 >> v2:172.21.15.159:3300/0 conn(0x7f1c7415d640 0x7f1c740d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.841+0000 7f1c73400700 1 -- 172.21.15.12:0/2002473296 >> v2:172.21.15.159:3300/0 conn(0x7f1c7415d640 msgr2=0x7f1c740d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:04.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.841+0000 7f1c73400700 1 --2- 172.21.15.12:0/2002473296 >> v2:172.21.15.159:3300/0 conn(0x7f1c7415d640 0x7f1c740d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.841+0000 7f1c73400700 1 -- 172.21.15.12:0/2002473296 >> v1:172.21.15.159:6789/0 conn(0x7f1c74161800 legacy=0x7f1c740cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:04.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.841+0000 7f1c73400700 1 -- 172.21.15.12:0/2002473296 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c740d2880 con 0x7f1c740ceec0 2024-03-29T10:11:04.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.841+0000 7f1c73e00700 1 --2- 172.21.15.12:0/2002473296 >> v2:172.21.15.159:3300/0 conn(0x7f1c7415d640 0x7f1c740d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:11:04.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.841+0000 7f1c73400700 1 --2- 172.21.15.12:0/2002473296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 0x7f1c741612c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f1c740b5c20 tx=0x7f1c60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2c58b3f63670a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:04.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.841+0000 7f1c72a00700 1 -- 172.21.15.12:0/2002473296 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60004230 con 0x7f1c740ceec0 2024-03-29T10:11:04.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.841+0000 7f1c72a00700 1 -- 172.21.15.12:0/2002473296 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c60004390 con 0x7f1c740ceec0 2024-03-29T10:11:04.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.841+0000 7f1c72a00700 1 -- 172.21.15.12:0/2002473296 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c600036a0 con 0x7f1c740ceec0 2024-03-29T10:11:04.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.842+0000 7f1c7ae00700 1 -- 172.21.15.12:0/2002473296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 msgr2=0x7f1c741612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.842+0000 7f1c7ae00700 1 --2- 172.21.15.12:0/2002473296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 0x7f1c741612c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f1c740b5c20 tx=0x7f1c60009bd0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.842+0000 7f1c7ae00700 1 -- 172.21.15.12:0/2002473296 shutdown_connections 2024-03-29T10:11:04.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.842+0000 7f1c7ae00700 1 --2- 172.21.15.12:0/2002473296 >> v2:172.21.15.159:3300/0 conn(0x7f1c7415d640 0x7f1c740d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.842+0000 7f1c7ae00700 1 --2- 172.21.15.12:0/2002473296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 0x7f1c741612c0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.842+0000 7f1c7ae00700 1 -- 172.21.15.12:0/2002473296 >> 172.21.15.12:0/2002473296 conn(0x7f1c7408aff0 msgr2=0x7f1c740d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:04.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.842+0000 7f1c7ae00700 1 -- 172.21.15.12:0/2002473296 shutdown_connections 2024-03-29T10:11:04.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.842+0000 7f1c7ae00700 1 -- 172.21.15.12:0/2002473296 wait complete. 2024-03-29T10:11:04.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.844+0000 7f1c7ae00700 1 Processor -- start 2024-03-29T10:11:04.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.844+0000 7f1c7ae00700 1 -- start start 2024-03-29T10:11:04.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c7ae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 0x7f1c74158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c7ae00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c7415d640 0x7f1c7415f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c7ae00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c74161800 0x7f1c7415f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c7ae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740a12a0 con 0x7f1c740ceec0 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c7ae00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740a04e0 con 0x7f1c7415d640 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c7ae00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740a07c0 con 0x7f1c74161800 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c73e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 0x7f1c74158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c73e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 0x7f1c74158a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39522/0 (socket says 172.21.15.12:39522) 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c73e00700 1 -- 172.21.15.12:0/1073354199 learned_addr learned my addr 172.21.15.12:0/1073354199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c73400700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c7415d640 0x7f1c7415f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c73e00700 1 -- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c74161800 msgr2=0x7f1c7415f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c73e00700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c74161800 0x7f1c7415f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c73e00700 1 -- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c7415d640 msgr2=0x7f1c7415f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c73e00700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c7415d640 0x7f1c7415f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:04.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.845+0000 7f1c73e00700 1 -- 172.21.15.12:0/1073354199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c60009550 con 0x7f1c740ceec0 2024-03-29T10:11:04.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.846+0000 7f1c73e00700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 0x7f1c74158a40 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400c750 tx=0x7f1c6400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:04.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.846+0000 7f1c6be00700 1 -- 172.21.15.12:0/1073354199 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64011800 con 0x7f1c740ceec0 2024-03-29T10:11:04.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.846+0000 7f1c6be00700 1 -- 172.21.15.12:0/1073354199 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c64011960 con 0x7f1c740ceec0 2024-03-29T10:11:04.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.846+0000 7f1c6be00700 1 -- 172.21.15.12:0/1073354199 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64011cd0 con 0x7f1c740ceec0 2024-03-29T10:11:04.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.846+0000 7f1c7ae00700 1 -- 172.21.15.12:0/1073354199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c740a0aa0 con 0x7f1c740ceec0 2024-03-29T10:11:04.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.846+0000 7f1c7ae00700 1 -- 172.21.15.12:0/1073354199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c740a01b0 con 0x7f1c740ceec0 2024-03-29T10:11:04.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.847+0000 7f1c7ae00700 1 -- 172.21.15.12:0/1073354199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c740a0fc0 con 0x7f1c740ceec0 2024-03-29T10:11:04.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.848+0000 7f1c6be00700 1 -- 172.21.15.12:0/1073354199 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1c64021350 con 0x7f1c740ceec0 2024-03-29T10:11:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.849+0000 7f1c6be00700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c5406adb0 0x7f1c5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.849+0000 7f1c6be00700 1 -- 172.21.15.12:0/1073354199 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c64089620 con 0x7f1c740ceec0 2024-03-29T10:11:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.850+0000 7f1c73400700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c5406adb0 0x7f1c5406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.850+0000 7f1c73400700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c5406adb0 0x7f1c5406d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1c740cfaf0 tx=0x7f1c60008030 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:04.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:04.854+0000 7f1c6be00700 1 -- 172.21.15.12:0/1073354199 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1c740a0fc0 con 0x7f1c740ceec0 2024-03-29T10:11:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.115+0000 7f1c7ae00700 1 -- 172.21.15.12:0/1073354199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c740a0fc0 con 0x7f1c740ceec0 2024-03-29T10:11:05.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.116+0000 7f1c6be00700 1 -- 172.21.15.12:0/1073354199 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f1c740a0fc0 con 0x7f1c740ceec0 2024-03-29T10:11:05.116 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:11:05.116 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6452,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2137480356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2137480356},{"type":"v1","addr":"172.21.15.159:6836","nonce":2137480356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":6455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2356039533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2356039533},{"type":"v1","addr":"172.21.15.159:6837","nonce":2356039533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":7153,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2916743006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2916743006},{"type":"v1","addr":"172.21.15.12:6835","nonce":2916743006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:58.634187+0000","modified":"2024-03-29T10:11:02.111942+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7410":{"gid":7410,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3023331068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3023331068},{"type":"v1","addr":"172.21.15.12:6834","nonce":3023331068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-29T10:11:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.118+0000 7f1c7ae00700 1 -- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c5406adb0 msgr2=0x7f1c5406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.118+0000 7f1c7ae00700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c5406adb0 0x7f1c5406d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1c740cfaf0 tx=0x7f1c60008030 comp rx=0 tx=0).stop 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.118+0000 7f1c7ae00700 1 -- 172.21.15.12:0/1073354199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 msgr2=0x7f1c74158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.118+0000 7f1c7ae00700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 0x7f1c74158a40 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400c750 tx=0x7f1c6400cb10 comp rx=0 tx=0).stop 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.118+0000 7f1c7ae00700 1 -- 172.21.15.12:0/1073354199 shutdown_connections 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.118+0000 7f1c7ae00700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1c5406adb0 0x7f1c5406d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.118+0000 7f1c7ae00700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c74161800 0x7f1c7415f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.118+0000 7f1c7ae00700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c740ceec0 0x7f1c74158a40 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.118+0000 7f1c7ae00700 1 --2- 172.21.15.12:0/1073354199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c7415d640 0x7f1c7415f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.118+0000 7f1c7ae00700 1 -- 172.21.15.12:0/1073354199 >> 172.21.15.12:0/1073354199 conn(0x7f1c7408aff0 msgr2=0x7f1c740d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.119+0000 7f1c7ae00700 1 -- 172.21.15.12:0/1073354199 shutdown_connections 2024-03-29T10:11:05.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.119+0000 7f1c7ae00700 1 -- 172.21.15.12:0/1073354199 wait complete. 2024-03-29T10:11:05.120 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 46 2024-03-29T10:11:05.539 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T10:10:58.634187+0000', 'modified': '2024-03-29T10:11:02.111942+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7410}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7410': {'gid': 7410, 'name': 'c', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6834/3023331068', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6832', 'nonce': 3023331068}, {'type': 'v1', 'addr': '172.21.15.12:6834', 'nonce': 3023331068}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T10:11:05.540 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T10:11:05.540 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-29T10:11:05.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.743+0000 7fedd6600700 1 Processor -- start 2024-03-29T10:11:05.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.743+0000 7fedd6600700 1 -- start start 2024-03-29T10:11:05.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.744+0000 7fedd6600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fedd00d16a0 0x7fedd00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.744+0000 7fedd6600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d2370 0x7fedd00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.744+0000 7fedd6600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd0161420 con 0x7fedd015d9d0 2024-03-29T10:11:05.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.744+0000 7fedd6600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd00d2820 con 0x7fedd00d16a0 2024-03-29T10:11:05.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.744+0000 7fedd6600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd00d4340 con 0x7fedd00d2370 2024-03-29T10:11:05.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.744+0000 7fedcea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fedd00d16a0 0x7fedd00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.744+0000 7fedcf400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fedd015d9d0 0x7fedd00d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60796/0 (socket says 172.21.15.12:60796) 2024-03-29T10:11:05.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.744+0000 7fedcf400700 1 -- 172.21.15.12:0/2643389631 learned_addr learned my addr 172.21.15.12:0/2643389631 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:11:05.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.744+0000 7fedcea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fedd00d16a0 0x7fedd00d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39528/0 (socket says 172.21.15.12:39528) 2024-03-29T10:11:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.745+0000 7fedcfe00700 1 --2- 172.21.15.12:0/2643389631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d2370 0x7fedd00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.745+0000 7fedcea00700 1 -- 172.21.15.12:0/2643389631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d2370 msgr2=0x7fedd00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.745+0000 7fedcea00700 1 --2- 172.21.15.12:0/2643389631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d2370 0x7fedd00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.745+0000 7fedcea00700 1 -- 172.21.15.12:0/2643389631 >> v1:172.21.15.12:6789/0 conn(0x7fedd015d9d0 legacy=0x7fedd00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.745+0000 7fedcea00700 1 -- 172.21.15.12:0/2643389631 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd0161700 con 0x7fedd00d16a0 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.745+0000 7fedcfe00700 1 --2- 172.21.15.12:0/2643389631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d2370 0x7fedd00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.745+0000 7fedcea00700 1 --2- 172.21.15.12:0/2643389631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d16a0 0x7fedd00d1e30 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fedbc009820 tx=0x7fedbc009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1bbdfc104043c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.745+0000 7fedce000700 1 -- 172.21.15.12:0/2643389631 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc005e30 con 0x7fedd00d16a0 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.745+0000 7fedce000700 1 -- 172.21.15.12:0/2643389631 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedbc006030 con 0x7fedd00d16a0 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.745+0000 7fedce000700 1 -- 172.21.15.12:0/2643389631 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc0063e0 con 0x7fedd00d16a0 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.746+0000 7fedd6600700 1 -- 172.21.15.12:0/2643389631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d16a0 msgr2=0x7fedd00d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.746+0000 7fedd6600700 1 --2- 172.21.15.12:0/2643389631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d16a0 0x7fedd00d1e30 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fedbc009820 tx=0x7fedbc009b30 comp rx=0 tx=0).stop 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.746+0000 7fedd6600700 1 -- 172.21.15.12:0/2643389631 shutdown_connections 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.746+0000 7fedd6600700 1 --2- 172.21.15.12:0/2643389631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d2370 0x7fedd00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.746+0000 7fedd6600700 1 --2- 172.21.15.12:0/2643389631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d16a0 0x7fedd00d1e30 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.746+0000 7fedd6600700 1 -- 172.21.15.12:0/2643389631 >> 172.21.15.12:0/2643389631 conn(0x7fedd008aff0 msgr2=0x7fedd00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.746+0000 7fedd6600700 1 -- 172.21.15.12:0/2643389631 shutdown_connections 2024-03-29T10:11:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.746+0000 7fedd6600700 1 -- 172.21.15.12:0/2643389631 wait complete. 2024-03-29T10:11:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.748+0000 7fedd6600700 1 Processor -- start 2024-03-29T10:11:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.748+0000 7fedd6600700 1 -- start start 2024-03-29T10:11:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedd6600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d16a0 0x7fedd00ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedd6600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d2370 0x7fedd0155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedd6600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedd015d9d0 0x7fedd00b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedd6600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd00cc950 con 0x7fedd00d2370 2024-03-29T10:11:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedd6600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd00cab30 con 0x7fedd015d9d0 2024-03-29T10:11:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedd6600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd01558a0 con 0x7fedd00d16a0 2024-03-29T10:11:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d2370 0x7fedd0155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d2370 0x7fedd0155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39536/0 (socket says 172.21.15.12:39536) 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcea00700 1 -- 172.21.15.12:0/2711652889 learned_addr learned my addr 172.21.15.12:0/2711652889 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcf400700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d16a0 0x7fedd00ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcfe00700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedd015d9d0 0x7fedd00b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcea00700 1 -- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d16a0 msgr2=0x7fedd00ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcea00700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d16a0 0x7fedd00ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcea00700 1 -- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedd015d9d0 msgr2=0x7fedd00b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcea00700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedd015d9d0 0x7fedd00b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcea00700 1 -- 172.21.15.12:0/2711652889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc009430 con 0x7fedd00d2370 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcf400700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d16a0 0x7fedd00ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.749+0000 7fedcfe00700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedd015d9d0 0x7fedd00b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.750+0000 7fedcea00700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d2370 0x7fedd0155360 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fedbc009b70 tx=0x7fedbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.750+0000 7fedc7400700 1 -- 172.21.15.12:0/2711652889 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc004250 con 0x7fedd00d2370 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.750+0000 7fedc7400700 1 -- 172.21.15.12:0/2711652889 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedbc0043b0 con 0x7fedd00d2370 2024-03-29T10:11:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.750+0000 7fedd6600700 1 -- 172.21.15.12:0/2711652889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd015df50 con 0x7fedd00d2370 2024-03-29T10:11:05.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.750+0000 7fedc7400700 1 -- 172.21.15.12:0/2711652889 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc0036a0 con 0x7fedd00d2370 2024-03-29T10:11:05.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.750+0000 7fedd6600700 1 -- 172.21.15.12:0/2711652889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd015e510 con 0x7fedd00d2370 2024-03-29T10:11:05.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.752+0000 7fedc7400700 1 -- 172.21.15.12:0/2711652889 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fedbc022070 con 0x7fedd00d2370 2024-03-29T10:11:05.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.753+0000 7fedc7400700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fedb006adb0 0x7fedb006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.753+0000 7fedc7400700 1 -- 172.21.15.12:0/2711652889 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedbc08ae20 con 0x7fedd00d2370 2024-03-29T10:11:05.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.753+0000 7fedcf400700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fedb006adb0 0x7fedb006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.754+0000 7fedcf400700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fedb006adb0 0x7fedb006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fedc0007fd0 tx=0x7fedc0009040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:05.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.755+0000 7fedea1c4b80 1 Processor -- start 2024-03-29T10:11:05.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.755+0000 7fedea1c4b80 1 -- start start 2024-03-29T10:11:05.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 0x55eee9856c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9829be0 0x55eee982ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9853a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee97a1650 con 0x55eee9827b60 2024-03-29T10:11:05.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee9782c30 con 0x55eee9856830 2024-03-29T10:11:05.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eee9780430 con 0x55eee9829be0 2024-03-29T10:11:05.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedcfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9853a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedcfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9853a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39544/0 (socket says 172.21.15.12:39544) 2024-03-29T10:11:05.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedcf400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 0x55eee9856c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedcfe00700 1 -- 172.21.15.12:0/4066020280 learned_addr learned my addr 172.21.15.12:0/4066020280 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:05.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.756+0000 7fedcea00700 1 --2- 172.21.15.12:0/4066020280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9829be0 0x55eee982ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.757+0000 7fedcfe00700 1 -- 172.21.15.12:0/4066020280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9829be0 msgr2=0x55eee982ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.757+0000 7fedcfe00700 1 --2- 172.21.15.12:0/4066020280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9829be0 0x55eee982ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.757+0000 7fedcfe00700 1 -- 172.21.15.12:0/4066020280 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 msgr2=0x55eee9856c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.757+0000 7fedcfe00700 1 --2- 172.21.15.12:0/4066020280 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 0x55eee9856c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.757+0000 7fedcfe00700 1 -- 172.21.15.12:0/4066020280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eee956f790 con 0x55eee9827b60 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.757+0000 7fedcfe00700 1 --2- 172.21.15.12:0/4066020280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9853a20 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fedc800e5f0 tx=0x7fedc800e620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a215c473fdeb6a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.757+0000 7fedbbe00700 1 -- 172.21.15.12:0/4066020280 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc800a840 con 0x55eee9827b60 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.757+0000 7fedbbe00700 1 -- 172.21.15.12:0/4066020280 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc800a9a0 con 0x55eee9827b60 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.757+0000 7fedbbe00700 1 -- 172.21.15.12:0/4066020280 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc800ad30 con 0x55eee9827b60 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.758+0000 7fedea1c4b80 1 -- 172.21.15.12:0/4066020280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 msgr2=0x55eee9853a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.758+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/4066020280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9853a20 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fedc800e5f0 tx=0x7fedc800e620 comp rx=0 tx=0).stop 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.758+0000 7fedea1c4b80 1 -- 172.21.15.12:0/4066020280 shutdown_connections 2024-03-29T10:11:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.758+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/4066020280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9853a20 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.758+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/4066020280 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 0x55eee9856c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.758+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/4066020280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9829be0 0x55eee982ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.758+0000 7fedea1c4b80 1 -- 172.21.15.12:0/4066020280 >> 172.21.15.12:0/4066020280 conn(0x55eee978a6a0 msgr2=0x55eee9871640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.758+0000 7fedea1c4b80 1 -- 172.21.15.12:0/4066020280 shutdown_connections 2024-03-29T10:11:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.758+0000 7fedea1c4b80 1 -- 172.21.15.12:0/4066020280 wait complete. 2024-03-29T10:11:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.759+0000 7fedea1c4b80 1 -- start start 2024-03-29T10:11:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.759+0000 7fedea1c4b80 1 Processor -- start 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.760+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9827b60 0x55eee9850740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.760+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9829be0 0x55eee98749a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.760+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9856830 0x55eee9851e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.760+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee977bd20 con 0x55eee9829be0 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.760+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee9717770 con 0x55eee9827b60 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.760+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eee973a6f0 con 0x55eee9856830 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.760+0000 7fedea1c4b80 10 client.0 _ll_register_callbacks cb 0x55eee97215e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.760+0000 7fedcea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9829be0 0x55eee98749a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.760+0000 7fedcea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9829be0 0x55eee98749a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39552/0 (socket says 172.21.15.12:39552) 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.760+0000 7fedcea00700 1 -- 172.21.15.12:0/878923558 learned_addr learned my addr 172.21.15.12:0/878923558 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedcf400700 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9827b60 0x55eee9850740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedcfe00700 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9856830 0x55eee9851e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedcea00700 1 -- 172.21.15.12:0/878923558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9856830 msgr2=0x55eee9851e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedcea00700 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9856830 0x55eee9851e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedcea00700 1 -- 172.21.15.12:0/878923558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9827b60 msgr2=0x55eee9850740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedcea00700 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9827b60 0x55eee9850740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedcea00700 1 -- 172.21.15.12:0/878923558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc8007dd0 con 0x55eee9829be0 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedcea00700 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9829be0 0x55eee98749a0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fedbc089220 tx=0x7fedbc05a3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acda4cb3ac684f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedb8c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedb8c00700 1 -- 172.21.15.12:0/878923558 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc084020 con 0x55eee9829be0 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedb8c00700 1 -- 172.21.15.12:0/878923558 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedbc02e5b0 con 0x55eee9829be0 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedea1c4b80 1 -- 172.21.15.12:0/878923558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eee981e920 con 0x55eee9829be0 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedb8c00700 1 -- 172.21.15.12:0/878923558 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedbc061f90 con 0x55eee9829be0 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedea1c4b80 10 client.7548 fetch_fsmap learned FSMap version 46 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.761+0000 7fedea1c4b80 1 -- 172.21.15.12:0/878923558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55eee965c0a0 con 0x55eee9829be0 2024-03-29T10:11:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.762+0000 7fedb8c00700 1 -- 172.21.15.12:0/878923558 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55eee981e920 con 0x55eee9829be0 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.762+0000 7fedb8c00700 1 -- 172.21.15.12:0/878923558 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 46) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fedbc059090 con 0x55eee9829be0 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.762+0000 7fedea1c4b80 10 client.7548 fetch_fsmap finished waiting for FSMap version 46 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.762+0000 7fedea1c4b80 10 client.7548 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.762+0000 7fedea1c4b80 20 client.7548 populate_metadata read hostname 'smithi012' 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.762+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.12:6832/3023331068,v1:172.21.15.12:6834/3023331068] conn(0x55eee9852370 0x55eee984eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.762+0000 7fedea1c4b80 4 client.7548 mds_command: new command op to 7410 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.762+0000 7fedea1c4b80 1 -- 172.21.15.12:0/878923558 --> [v2:172.21.15.12:6832/3023331068,v1:172.21.15.12:6834/3023331068] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eee97fb6f0 con 0x55eee9852370 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.762+0000 7fedcf400700 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.12:6832/3023331068,v1:172.21.15.12:6834/3023331068] conn(0x55eee9852370 0x55eee984eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.763+0000 7fedcf400700 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.12:6832/3023331068,v1:172.21.15.12:6834/3023331068] conn(0x55eee9852370 0x55eee984eed0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa2a34c84938558c server_cookie=7fedc0492420fd37 in_seq=0 out_seq=0 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.763+0000 7fedb8c00700 10 client.7548 ms_handle_connect on v2:172.21.15.12:6832/3023331068 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.763+0000 7fedb8c00700 1 -- 172.21.15.12:0/878923558 <== mds.0 v2:172.21.15.12:6832/3023331068 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55eee97fb6f0 con 0x55eee9852370 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.763+0000 7fedb8c00700 10 client.7548 handle_command_reply: tid=0 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.763+0000 7fedea1c4b80 1 client.7548 shutdown 2024-03-29T10:11:05.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 1 -- 172.21.15.12:0/878923558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9829be0 msgr2=0x55eee98749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9829be0 0x55eee98749a0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fedbc089220 tx=0x7fedbc05a3e0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 1 -- 172.21.15.12:0/878923558 shutdown_connections 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.12:6832/3023331068,v1:172.21.15.12:6834/3023331068] conn(0x55eee9852370 0x55eee984eed0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9856830 0x55eee9851e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9827b60 0x55eee9850740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/878923558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9829be0 0x55eee98749a0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 1 -- 172.21.15.12:0/878923558 >> 172.21.15.12:0/878923558 conn(0x55eee978a6a0 msgr2=0x55eee982a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedb8c00700 0 client.7548 ms_handle_reset on v2:172.21.15.12:6832/3023331068 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 1 -- 172.21.15.12:0/878923558 shutdown_connections 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 1 -- 172.21.15.12:0/878923558 wait complete. 2024-03-29T10:11:05.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.764+0000 7fedea1c4b80 20 client.7548 trim_cache size 0 max 16384 2024-03-29T10:11:05.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.782+0000 7fedea1c4b80 1 Processor -- start 2024-03-29T10:11:05.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.782+0000 7fedea1c4b80 1 -- start start 2024-03-29T10:11:05.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.783+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9856e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.783+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9829be0 0x55eee9871440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.783+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9856830 0x55eee9871980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.783+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee9570390 con 0x55eee9827b60 2024-03-29T10:11:05.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.783+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee984e7d0 con 0x55eee9829be0 2024-03-29T10:11:05.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.783+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eee982ab50 con 0x55eee9856830 2024-03-29T10:11:05.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9856e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9856e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39554/0 (socket says 172.21.15.12:39554) 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcf400700 1 -- 172.21.15.12:0/1489844787 learned_addr learned my addr 172.21.15.12:0/1489844787 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcfe00700 1 --2- 172.21.15.12:0/1489844787 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9856830 0x55eee9871980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcea00700 1 --2- 172.21.15.12:0/1489844787 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9829be0 0x55eee9871440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcf400700 1 -- 172.21.15.12:0/1489844787 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9856830 msgr2=0x55eee9871980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcf400700 1 --2- 172.21.15.12:0/1489844787 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9856830 0x55eee9871980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcf400700 1 -- 172.21.15.12:0/1489844787 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9829be0 msgr2=0x55eee9871440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcf400700 1 --2- 172.21.15.12:0/1489844787 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9829be0 0x55eee9871440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcf400700 1 -- 172.21.15.12:0/1489844787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eee9742010 con 0x55eee9827b60 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedcf400700 1 --2- 172.21.15.12:0/1489844787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9856e00 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fedc0005910 tx=0x7fedc0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d425a39695bedf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.784+0000 7fedc4c00700 1 -- 172.21.15.12:0/1489844787 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc0005d70 con 0x55eee9827b60 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedc4c00700 1 -- 172.21.15.12:0/1489844787 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc00074b0 con 0x55eee9827b60 2024-03-29T10:11:05.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedc4c00700 1 -- 172.21.15.12:0/1489844787 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc0007820 con 0x55eee9827b60 2024-03-29T10:11:05.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedea1c4b80 1 -- 172.21.15.12:0/1489844787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 msgr2=0x55eee9856e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/1489844787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9856e00 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fedc0005910 tx=0x7fedc0012000 comp rx=0 tx=0).stop 2024-03-29T10:11:05.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedea1c4b80 1 -- 172.21.15.12:0/1489844787 shutdown_connections 2024-03-29T10:11:05.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/1489844787 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9856830 0x55eee9871980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/1489844787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee9856e00 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/1489844787 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9829be0 0x55eee9871440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedea1c4b80 1 -- 172.21.15.12:0/1489844787 >> 172.21.15.12:0/1489844787 conn(0x55eee978a6a0 msgr2=0x55eee9855230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedea1c4b80 1 -- 172.21.15.12:0/1489844787 shutdown_connections 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.785+0000 7fedea1c4b80 1 -- 172.21.15.12:0/1489844787 wait complete. 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.786+0000 7fedea1c4b80 1 -- start start 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.786+0000 7fedea1c4b80 1 Processor -- start 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.787+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee98720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.787+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9829be0 0x55eee9873e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.787+0000 7fedea1c4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 0x55eee986f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.787+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee9870630 con 0x55eee9827b60 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.787+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee98747e0 con 0x55eee9856830 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.787+0000 7fedea1c4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eee9871c30 con 0x55eee9829be0 2024-03-29T10:11:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.787+0000 7fedea1c4b80 10 client.0 _ll_register_callbacks cb 0x7fedbc059520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.787+0000 7fedcf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee98720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.787+0000 7fedcf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee98720c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39570/0 (socket says 172.21.15.12:39570) 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.787+0000 7fedcf400700 1 -- 172.21.15.12:0/3845101703 learned_addr learned my addr 172.21.15.12:0/3845101703 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedcea00700 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9829be0 0x55eee9873e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedcfe00700 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 0x55eee986f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedcf400700 1 -- 172.21.15.12:0/3845101703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9829be0 msgr2=0x55eee9873e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedcf400700 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9829be0 0x55eee9873e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedcf400700 1 -- 172.21.15.12:0/3845101703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 msgr2=0x55eee986f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedcf400700 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 0x55eee986f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedcf400700 1 -- 172.21.15.12:0/3845101703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc000f040 con 0x55eee9827b60 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedcfe00700 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 0x55eee986f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedcf400700 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee98720c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009010 tx=0x7fedc0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488b614a4b144e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedb8c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:11:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedb8c00700 1 -- 172.21.15.12:0/3845101703 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc00077c0 con 0x55eee9827b60 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedb8c00700 1 -- 172.21.15.12:0/3845101703 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedc0037aa0 con 0x55eee9827b60 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.788+0000 7fedea1c4b80 1 -- 172.21.15.12:0/3845101703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eee9875b80 con 0x55eee9827b60 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedb8c00700 1 -- 172.21.15.12:0/3845101703 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedc0037c60 con 0x55eee9827b60 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedea1c4b80 10 client.7554 fetch_fsmap learned FSMap version 46 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedea1c4b80 1 -- 172.21.15.12:0/3845101703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55eee95d01d0 con 0x55eee9827b60 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedb8c00700 1 -- 172.21.15.12:0/3845101703 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc003d070 con 0x55eee9827b60 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedb8c00700 1 -- 172.21.15.12:0/3845101703 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 46) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fedc0025a80 con 0x55eee9827b60 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedea1c4b80 10 client.7554 fetch_fsmap finished waiting for FSMap version 46 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedea1c4b80 10 client.7554 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedea1c4b80 20 client.7554 populate_metadata read hostname 'smithi012' 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.12:6832/3023331068,v1:172.21.15.12:6834/3023331068] conn(0x55eee9825fe0 0x55eee98263c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedea1c4b80 4 client.7554 mds_command: new command op to 7410 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T10:11:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedea1c4b80 1 -- 172.21.15.12:0/3845101703 --> [v2:172.21.15.12:6832/3023331068,v1:172.21.15.12:6834/3023331068] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fedc800a9a0 con 0x55eee9825fe0 2024-03-29T10:11:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.789+0000 7fedcea00700 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.12:6832/3023331068,v1:172.21.15.12:6834/3023331068] conn(0x55eee9825fe0 0x55eee98263c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedcea00700 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.12:6832/3023331068,v1:172.21.15.12:6834/3023331068] conn(0x55eee9825fe0 0x55eee98263c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96aa06f20eb63e0a server_cookie=84b8f34a81f7c6b8 in_seq=0 out_seq=0 2024-03-29T10:11:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedb8c00700 10 client.7554 ms_handle_connect on v2:172.21.15.12:6832/3023331068 2024-03-29T10:11:05.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedb8c00700 1 -- 172.21.15.12:0/3845101703 <== mds.0 v2:172.21.15.12:6832/3023331068 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7fedc800a9a0 con 0x55eee9825fe0 2024-03-29T10:11:05.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedb8c00700 10 client.7554 handle_command_reply: tid=0 2024-03-29T10:11:05.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 client.7554 shutdown 2024-03-29T10:11:05.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 -- 172.21.15.12:0/3845101703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 msgr2=0x55eee98720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee98720c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009010 tx=0x7fedc0000ed0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 -- 172.21.15.12:0/3845101703 shutdown_connections 2024-03-29T10:11:05.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.12:6832/3023331068,v1:172.21.15.12:6834/3023331068] conn(0x55eee9825fe0 0x55eee98263c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eee9856830 0x55eee986f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eee9827b60 0x55eee98720c0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 --2- 172.21.15.12:0/3845101703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eee9829be0 0x55eee9873e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedb8c00700 0 client.7554 ms_handle_reset on v2:172.21.15.12:6832/3023331068 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 -- 172.21.15.12:0/3845101703 >> 172.21.15.12:0/3845101703 conn(0x55eee978a6a0 msgr2=0x55eee9855230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 -- 172.21.15.12:0/3845101703 shutdown_connections 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.790+0000 7fedea1c4b80 1 -- 172.21.15.12:0/3845101703 wait complete. 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedea1c4b80 20 client.7554 trim_cache size 0 max 16384 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 7410, 2024-03-29T10:11:05.795 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T10:11:05.796 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T10:11:05.796 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T10:11:05.796 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1539230209999998, 2024-03-29T10:11:05.796 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 46, 2024-03-29T10:11:05.796 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 108, 2024-03-29T10:11:05.796 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 108, 2024-03-29T10:11:05.796 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12.154285752 2024-03-29T10:11:05.796 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:11:05.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 -- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fedb006adb0 msgr2=0x7fedb006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fedb006adb0 0x7fedb006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fedc0007fd0 tx=0x7fedc0009040 comp rx=0 tx=0).stop 2024-03-29T10:11:05.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 -- 172.21.15.12:0/2711652889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d2370 msgr2=0x7fedd0155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:05.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d2370 0x7fedd0155360 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fedbc009b70 tx=0x7fedbc004750 comp rx=0 tx=0).stop 2024-03-29T10:11:05.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 -- 172.21.15.12:0/2711652889 shutdown_connections 2024-03-29T10:11:05.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fedb006adb0 0x7fedb006d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedd015d9d0 0x7fedd00b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedd00d16a0 0x7fedd00ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 --2- 172.21.15.12:0/2711652889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedd00d2370 0x7fedd0155360 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:05.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 -- 172.21.15.12:0/2711652889 >> 172.21.15.12:0/2711652889 conn(0x7fedd008aff0 msgr2=0x7fedd00b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:05.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 -- 172.21.15.12:0/2711652889 shutdown_connections 2024-03-29T10:11:05.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:05.791+0000 7fedd6600700 1 -- 172.21.15.12:0/2711652889 wait complete. 2024-03-29T10:11:05.809 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T10:11:06.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.006+0000 7fd77cc00700 1 Processor -- start 2024-03-29T10:11:06.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.007+0000 7fd77cc00700 1 -- start start 2024-03-29T10:11:06.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd77cc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd77815b5d0 0x7fd7780ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:06.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd77cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ac7d0 0x7fd77806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:06.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd77cc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77815fd50 con 0x7fd7780cb3f0 2024-03-29T10:11:06.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd77cc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77815b9b0 con 0x7fd77815b5d0 2024-03-29T10:11:06.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd77cc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77815f140 con 0x7fd7780ac7d0 2024-03-29T10:11:06.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd775e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd77815b5d0 0x7fd7780ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:06.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd776800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ac7d0 0x7fd77806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:06.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd776800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ac7d0 0x7fd77806c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:56140/0 (socket says 172.21.15.12:56140) 2024-03-29T10:11:06.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd776800700 1 -- 172.21.15.12:0/2930994572 learned_addr learned my addr 172.21.15.12:0/2930994572 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:06.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd775e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd77815b5d0 0x7fd7780ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:49602/0 (socket says 172.21.15.12:49602) 2024-03-29T10:11:06.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.008+0000 7fd775400700 1 --1- 172.21.15.12:0/2930994572 >> v1:172.21.15.159:6789/0 conn(0x7fd7780cb3f0 0x7fd7780ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:35210/0 (socket says 172.21.15.12:35210) 2024-03-29T10:11:06.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.009+0000 7fd775e00700 1 -- 172.21.15.12:0/2930994572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ac7d0 msgr2=0x7fd77806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:06.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.009+0000 7fd774a00700 1 -- 172.21.15.12:0/2930994572 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544078961 0 0) 0x7fd77815fd50 con 0x7fd7780cb3f0 2024-03-29T10:11:06.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.009+0000 7fd775e00700 1 --2- 172.21.15.12:0/2930994572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ac7d0 0x7fd77806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:06.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.009+0000 7fd775e00700 1 -- 172.21.15.12:0/2930994572 >> v1:172.21.15.159:6789/0 conn(0x7fd7780cb3f0 legacy=0x7fd7780ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:06.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.009+0000 7fd775e00700 1 -- 172.21.15.12:0/2930994572 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77815f420 con 0x7fd77815b5d0 2024-03-29T10:11:06.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.009+0000 7fd776800700 1 --2- 172.21.15.12:0/2930994572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ac7d0 0x7fd77806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:11:06.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.009+0000 7fd775e00700 1 --2- 172.21.15.12:0/2930994572 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77815b5d0 0x7fd7780ae3f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd768009840 tx=0x7fd768009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8538f74aa937326 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:06.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.009+0000 7fd774a00700 1 -- 172.21.15.12:0/2930994572 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768011830 con 0x7fd77815b5d0 2024-03-29T10:11:06.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.010+0000 7fd774a00700 1 -- 172.21.15.12:0/2930994572 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd768011990 con 0x7fd77815b5d0 2024-03-29T10:11:06.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.010+0000 7fd774a00700 1 -- 172.21.15.12:0/2930994572 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768011d20 con 0x7fd77815b5d0 2024-03-29T10:11:06.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.010+0000 7fd77cc00700 1 -- 172.21.15.12:0/2930994572 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77815b5d0 msgr2=0x7fd7780ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:06.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.010+0000 7fd77cc00700 1 --2- 172.21.15.12:0/2930994572 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77815b5d0 0x7fd7780ae3f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd768009840 tx=0x7fd768009b50 comp rx=0 tx=0).stop 2024-03-29T10:11:06.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.010+0000 7fd77cc00700 1 -- 172.21.15.12:0/2930994572 shutdown_connections 2024-03-29T10:11:06.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.010+0000 7fd77cc00700 1 --2- 172.21.15.12:0/2930994572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ac7d0 0x7fd77806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:06.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.010+0000 7fd77cc00700 1 --2- 172.21.15.12:0/2930994572 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77815b5d0 0x7fd7780ae3f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:06.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.010+0000 7fd77cc00700 1 -- 172.21.15.12:0/2930994572 >> 172.21.15.12:0/2930994572 conn(0x7fd7780cf190 msgr2=0x7fd7780cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:06.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.010+0000 7fd77cc00700 1 -- 172.21.15.12:0/2930994572 shutdown_connections 2024-03-29T10:11:06.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.010+0000 7fd77cc00700 1 -- 172.21.15.12:0/2930994572 wait complete. 2024-03-29T10:11:06.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.012+0000 7fd77cc00700 1 Processor -- start 2024-03-29T10:11:06.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.012+0000 7fd77cc00700 1 -- start start 2024-03-29T10:11:06.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.013+0000 7fd77cc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7780ac7d0 0x7fd77806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:06.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.013+0000 7fd77cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77815b5d0 0x7fd7780cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:06.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.013+0000 7fd77cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cc7e0 0x7fd778072190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:06.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.013+0000 7fd77cc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780c8460 con 0x7fd7780cc7e0 2024-03-29T10:11:06.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.013+0000 7fd77cc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780c8740 con 0x7fd7780ac7d0 2024-03-29T10:11:06.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.013+0000 7fd77cc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780c8a20 con 0x7fd77815b5d0 2024-03-29T10:11:06.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.013+0000 7fd776800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cc7e0 0x7fd778072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.013+0000 7fd776800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cc7e0 0x7fd778072190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39580/0 (socket says 172.21.15.12:39580) 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.013+0000 7fd776800700 1 -- 172.21.15.12:0/2994282809 learned_addr learned my addr 172.21.15.12:0/2994282809 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.013+0000 7fd775e00700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7780ac7d0 0x7fd77806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.014+0000 7fd775400700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77815b5d0 0x7fd7780cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.014+0000 7fd776800700 1 -- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77815b5d0 msgr2=0x7fd7780cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.014+0000 7fd776800700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77815b5d0 0x7fd7780cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.014+0000 7fd776800700 1 -- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7780ac7d0 msgr2=0x7fd77806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.014+0000 7fd776800700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7780ac7d0 0x7fd77806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.014+0000 7fd776800700 1 -- 172.21.15.12:0/2994282809 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd770009b30 con 0x7fd7780cc7e0 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.014+0000 7fd776800700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cc7e0 0x7fd778072190 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fd77000ef90 tx=0x7fd77000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.014+0000 7fd76e000700 1 -- 172.21.15.12:0/2994282809 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770018070 con 0x7fd7780cc7e0 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.014+0000 7fd77cc00700 1 -- 172.21.15.12:0/2994282809 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7680094a0 con 0x7fd7780cc7e0 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.014+0000 7fd76e000700 1 -- 172.21.15.12:0/2994282809 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77000f040 con 0x7fd7780cc7e0 2024-03-29T10:11:06.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.015+0000 7fd76e000700 1 -- 172.21.15.12:0/2994282809 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770003c60 con 0x7fd7780cc7e0 2024-03-29T10:11:06.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.015+0000 7fd77cc00700 1 -- 172.21.15.12:0/2994282809 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7780c8fe0 con 0x7fd7780cc7e0 2024-03-29T10:11:06.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.016+0000 7fd77cc00700 1 -- 172.21.15.12:0/2994282809 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77815c980 con 0x7fd7780cc7e0 2024-03-29T10:11:06.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.016+0000 7fd76e000700 1 -- 172.21.15.12:0/2994282809 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd77000c880 con 0x7fd7780cc7e0 2024-03-29T10:11:06.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.018+0000 7fd76e000700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:06.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.018+0000 7fd76e000700 1 -- 172.21.15.12:0/2994282809 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd770089760 con 0x7fd7780cc7e0 2024-03-29T10:11:06.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.018+0000 7fd775e00700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:06.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.019+0000 7fd775e00700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 0x7fd75806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd7680090b0 tx=0x7fd76801c040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:06.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.022+0000 7fd76e000700 1 -- 172.21.15.12:0/2994282809 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd770054690 con 0x7fd7780cc7e0 2024-03-29T10:11:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.284+0000 7fd77cc00700 1 -- 172.21.15.12:0/2994282809 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd778067720 con 0x7fd7780cc7e0 2024-03-29T10:11:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.285+0000 7fd76e000700 1 -- 172.21.15.12:0/2994282809 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd770057cb0 con 0x7fd7780cc7e0 2024-03-29T10:11:06.285 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T10:11:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 -- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 msgr2=0x7fd75806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 0x7fd75806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd7680090b0 tx=0x7fd76801c040 comp rx=0 tx=0).stop 2024-03-29T10:11:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 -- 172.21.15.12:0/2994282809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cc7e0 msgr2=0x7fd778072190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cc7e0 0x7fd778072190 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fd77000ef90 tx=0x7fd77000c3a0 comp rx=0 tx=0).stop 2024-03-29T10:11:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 -- 172.21.15.12:0/2994282809 shutdown_connections 2024-03-29T10:11:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780cc7e0 0x7fd778072190 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7780ac7d0 0x7fd77806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 --2- 172.21.15.12:0/2994282809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77815b5d0 0x7fd7780cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 -- 172.21.15.12:0/2994282809 >> 172.21.15.12:0/2994282809 conn(0x7fd7780cf190 msgr2=0x7fd77815cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 -- 172.21.15.12:0/2994282809 shutdown_connections 2024-03-29T10:11:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:06.287+0000 7fd77cc00700 1 -- 172.21.15.12:0/2994282809 wait complete. 2024-03-29T10:11:06.305 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T10:11:06.305 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T10:11:06.305 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T10:11:06.305 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T10:11:06.305 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T10:11:06.305 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T10:11:06.305 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T10:11:06.305 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T10:11:06.305 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:11:06.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:11:06.306 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T10:11:06.351 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T10:11:06.351 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T10:11:06.351 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T10:11:06.351 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:11:06.351 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 40451sec preferred_lft 40451sec 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T10:11:06.352 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:11:06.353 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T10:11:06.354 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T10:11:06.354 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:11:06.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:11:06.354 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T10:11:06.457 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T10:11:06.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:11:06.460 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T10:11:06.507 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T10:11:06.508 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T10:11:06.508 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T10:11:06.508 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T10:11:06.508 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T10:11:06.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:11:06.509 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T10:11:06.509 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T10:11:06.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T10:11:06.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T10:11:06.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T10:11:06.509 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T10:11:06.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:11:06.675 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T10:11:06.719 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T10:11:06.719 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T10:11:06.720 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:11:06.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:11:06.720 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T10:11:06.827 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:11:06.827 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T10:11:06.871 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:11:06.871 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:11:06.871 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:11:06.956 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T10:11:06.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:11:06.956 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:11:07.098 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:11:07.098 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:11:07.125 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:11:07.125 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:11:07.125 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T10:11:07.125 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T10:11:07.126 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T10:11:07.130 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T10:11:07.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:11:07.130 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:11:07.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:11:07.199 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T10:11:07.298 INFO:teuthology.orchestra.run.smithi012.stdout:client7560 2024-03-29T10:11:07.298 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:11:07.298 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:11:07.342 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T10:11:07.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.581+0000 7f4c1e800700 1 Processor -- start 2024-03-29T10:11:07.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.581+0000 7f4c1e800700 1 -- start start 2024-03-29T10:11:07.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c1e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4c180cd020 0x7f4c180cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:07.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c1e800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c180cdae0 con 0x7f4c180cbb30 2024-03-29T10:11:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c1e800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c180cddc0 con 0x7f4c180b2070 2024-03-29T10:11:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c1e800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c180ce0a0 con 0x7f4c180cd020 2024-03-29T10:11:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c16a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4c180cd020 0x7f4c180cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c17400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4c180cbb30 0x7f4c180cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60806/0 (socket says 172.21.15.12:60806) 2024-03-29T10:11:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c17400700 1 -- 172.21.15.12:0/3828725180 learned_addr learned my addr 172.21.15.12:0/3828725180 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:11:07.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c16a00700 1 -- 172.21.15.12:0/3828725180 >> v1:172.21.15.159:6789/0 conn(0x7f4c180b2070 legacy=0x7f4c1806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:07.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c16a00700 1 -- 172.21.15.12:0/3828725180 >> v1:172.21.15.12:6789/0 conn(0x7f4c180cbb30 legacy=0x7f4c180cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:07.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c16a00700 1 -- 172.21.15.12:0/3828725180 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1806dc20 con 0x7f4c180cd020 2024-03-29T10:11:07.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.583+0000 7f4c16000700 1 -- 172.21.15.12:0/3828725180 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976549835 0 0) 0x7f4c180cdae0 con 0x7f4c180cbb30 2024-03-29T10:11:07.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.584+0000 7f4c16a00700 1 --2- 172.21.15.12:0/3828725180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 0x7f4c180cd480 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009840 tx=0x7f4c04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b4d8b85d3088e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:07.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.584+0000 7f4c16000700 1 -- 172.21.15.12:0/3828725180 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04004230 con 0x7f4c180cd020 2024-03-29T10:11:07.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.584+0000 7f4c16000700 1 -- 172.21.15.12:0/3828725180 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c04004390 con 0x7f4c180cd020 2024-03-29T10:11:07.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.584+0000 7f4c16000700 1 -- 172.21.15.12:0/3828725180 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c040036a0 con 0x7f4c180cd020 2024-03-29T10:11:07.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.584+0000 7f4c1e800700 1 -- 172.21.15.12:0/3828725180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 msgr2=0x7f4c180cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:07.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.584+0000 7f4c1e800700 1 --2- 172.21.15.12:0/3828725180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 0x7f4c180cd480 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009840 tx=0x7f4c04009b50 comp rx=0 tx=0).stop 2024-03-29T10:11:07.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.585+0000 7f4c1e800700 1 -- 172.21.15.12:0/3828725180 shutdown_connections 2024-03-29T10:11:07.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.585+0000 7f4c1e800700 1 --2- 172.21.15.12:0/3828725180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 0x7f4c180cd480 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:07.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.585+0000 7f4c1e800700 1 -- 172.21.15.12:0/3828725180 >> 172.21.15.12:0/3828725180 conn(0x7f4c180b0d70 msgr2=0x7f4c180b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:07.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.585+0000 7f4c1e800700 1 -- 172.21.15.12:0/3828725180 shutdown_connections 2024-03-29T10:11:07.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.585+0000 7f4c1e800700 1 -- 172.21.15.12:0/3828725180 wait complete. 2024-03-29T10:11:07.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.586+0000 7f4c1e800700 1 Processor -- start 2024-03-29T10:11:07.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.586+0000 7f4c1e800700 1 -- start start 2024-03-29T10:11:07.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.587+0000 7f4c1e800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c180b2070 0x7f4c18154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:07.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.587+0000 7f4c1e800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c180cbb30 0x7f4c180ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:07.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.587+0000 7f4c1e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 0x7f4c180cab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:07.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.587+0000 7f4c1e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c18071da0 con 0x7f4c180cd020 2024-03-29T10:11:07.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.587+0000 7f4c1e800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1815cbb0 con 0x7f4c180cbb30 2024-03-29T10:11:07.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.587+0000 7f4c1e800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1815afb0 con 0x7f4c180b2070 2024-03-29T10:11:07.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 0x7f4c180cab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:07.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 0x7f4c180cab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39586/0 (socket says 172.21.15.12:39586) 2024-03-29T10:11:07.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17e00700 1 -- 172.21.15.12:0/2020565933 learned_addr learned my addr 172.21.15.12:0/2020565933 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:07.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c16a00700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c180cbb30 0x7f4c180ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:07.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17400700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c180b2070 0x7f4c18154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:07.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17e00700 1 -- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c180b2070 msgr2=0x7f4c18154ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:07.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17e00700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c180b2070 0x7f4c18154ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:07.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17e00700 1 -- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c180cbb30 msgr2=0x7f4c180ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:07.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17e00700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c180cbb30 0x7f4c180ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:07.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17e00700 1 -- 172.21.15.12:0/2020565933 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c040094a0 con 0x7f4c180cd020 2024-03-29T10:11:07.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17400700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c180b2070 0x7f4c18154ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:11:07.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c16a00700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c180cbb30 0x7f4c180ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:11:07.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c17e00700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 0x7f4c180cab60 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000b970 tx=0x7f4c1000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:07.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.588+0000 7f4c0f400700 1 -- 172.21.15.12:0/2020565933 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1000c9b0 con 0x7f4c180cd020 2024-03-29T10:11:07.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.589+0000 7f4c0f400700 1 -- 172.21.15.12:0/2020565933 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1000cb10 con 0x7f4c180cd020 2024-03-29T10:11:07.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.589+0000 7f4c1e800700 1 -- 172.21.15.12:0/2020565933 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1815b290 con 0x7f4c180cd020 2024-03-29T10:11:07.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.589+0000 7f4c0f400700 1 -- 172.21.15.12:0/2020565933 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c10014440 con 0x7f4c180cd020 2024-03-29T10:11:07.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.589+0000 7f4c1e800700 1 -- 172.21.15.12:0/2020565933 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1815d0e0 con 0x7f4c180cd020 2024-03-29T10:11:07.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.590+0000 7f4c1e800700 1 -- 172.21.15.12:0/2020565933 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1815d300 con 0x7f4c180cd020 2024-03-29T10:11:07.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.591+0000 7f4c0f400700 1 -- 172.21.15.12:0/2020565933 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4c1001d3b0 con 0x7f4c180cd020 2024-03-29T10:11:07.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.591+0000 7f4c0f400700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bf806adb0 0x7f4bf806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:07.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.591+0000 7f4c0f400700 1 -- 172.21.15.12:0/2020565933 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c10089030 con 0x7f4c180cd020 2024-03-29T10:11:07.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.592+0000 7f4c17400700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bf806adb0 0x7f4bf806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:07.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.593+0000 7f4c17400700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bf806adb0 0x7f4bf806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4c08000f60 tx=0x7f4c08009040 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:07.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.596+0000 7f4c0f400700 1 -- 172.21.15.12:0/2020565933 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c10054190 con 0x7f4c180cd020 2024-03-29T10:11:07.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.840+0000 7f4c1e800700 1 -- 172.21.15.12:0/2020565933 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c181594a0 con 0x7f4bf806adb0 2024-03-29T10:11:07.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.843+0000 7f4c0f400700 1 -- 172.21.15.12:0/2020565933 <== mgr.6940 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4c08004870 con 0x7f4bf806adb0 2024-03-29T10:11:07.843 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:11:07.843 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:11:07.843 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T10:11:07.843 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:11:07.843 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:11:07.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 -- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bf806adb0 msgr2=0x7f4bf806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:07.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bf806adb0 0x7f4bf806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4c08000f60 tx=0x7f4c08009040 comp rx=0 tx=0).stop 2024-03-29T10:11:07.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 -- 172.21.15.12:0/2020565933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 msgr2=0x7f4c180cab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:07.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 0x7f4c180cab60 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000b970 tx=0x7f4c1000bd30 comp rx=0 tx=0).stop 2024-03-29T10:11:07.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 -- 172.21.15.12:0/2020565933 shutdown_connections 2024-03-29T10:11:07.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bf806adb0 0x7f4bf806d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:07.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c180cd020 0x7f4c180cab60 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:07.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c180b2070 0x7f4c18154ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:07.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 --2- 172.21.15.12:0/2020565933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c180cbb30 0x7f4c180ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:07.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 -- 172.21.15.12:0/2020565933 >> 172.21.15.12:0/2020565933 conn(0x7f4c180b0d70 msgr2=0x7f4c181e6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:07.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 -- 172.21.15.12:0/2020565933 shutdown_connections 2024-03-29T10:11:07.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:07.845+0000 7f4c1e800700 1 -- 172.21.15.12:0/2020565933 wait complete. 2024-03-29T10:11:07.864 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T10:11:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.065+0000 7fbf31400700 1 Processor -- start 2024-03-29T10:11:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.065+0000 7fbf31400700 1 -- start start 2024-03-29T10:11:08.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.066+0000 7fbf31400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbf2c1611b0 0x7fbf2c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:08.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.066+0000 7fbf31400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbf2c0cee10 0x7fbf2c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:08.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.066+0000 7fbf31400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c0ce270 con 0x7fbf2c15f4d0 2024-03-29T10:11:08.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.066+0000 7fbf31400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2c15b600 con 0x7fbf2c0cee10 2024-03-29T10:11:08.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.066+0000 7fbf31400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2c0d2760 con 0x7fbf2c1611b0 2024-03-29T10:11:08.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.066+0000 7fbf2b200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbf2c0cee10 0x7fbf2c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:08.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.066+0000 7fbf2b200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbf2c0cee10 0x7fbf2c161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39600/0 (socket says 172.21.15.12:39600) 2024-03-29T10:11:08.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.066+0000 7fbf2b200700 1 -- 172.21.15.12:0/3688132839 learned_addr learned my addr 172.21.15.12:0/3688132839 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:08.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.067+0000 7fbf2a800700 1 --2- 172.21.15.12:0/3688132839 >> v2:172.21.15.159:3300/0 conn(0x7fbf2c1611b0 0x7fbf2c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:08.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.067+0000 7fbf2b200700 1 -- 172.21.15.12:0/3688132839 >> v2:172.21.15.159:3300/0 conn(0x7fbf2c1611b0 msgr2=0x7fbf2c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:08.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.067+0000 7fbf2b200700 1 --2- 172.21.15.12:0/3688132839 >> v2:172.21.15.159:3300/0 conn(0x7fbf2c1611b0 0x7fbf2c15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:08.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.067+0000 7fbf2b200700 1 -- 172.21.15.12:0/3688132839 >> v1:172.21.15.12:6789/0 conn(0x7fbf2c15f4d0 legacy=0x7fbf2c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:08.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.067+0000 7fbf29400700 1 -- 172.21.15.12:0/3688132839 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584364835 0 0) 0x7fbf2c0ce270 con 0x7fbf2c15f4d0 2024-03-29T10:11:08.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.067+0000 7fbf2b200700 1 -- 172.21.15.12:0/3688132839 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c0d2a40 con 0x7fbf2c0cee10 2024-03-29T10:11:08.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.067+0000 7fbf2b200700 1 --2- 172.21.15.12:0/3688132839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c0cee10 0x7fbf2c161590 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c08b520 tx=0x7fbf2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd36785e515a9a37 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:08.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.067+0000 7fbf29400700 1 -- 172.21.15.12:0/3688132839 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2400e070 con 0x7fbf2c0cee10 2024-03-29T10:11:08.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.067+0000 7fbf29400700 1 -- 172.21.15.12:0/3688132839 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2400b910 con 0x7fbf2c0cee10 2024-03-29T10:11:08.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.067+0000 7fbf29400700 1 -- 172.21.15.12:0/3688132839 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2400bcc0 con 0x7fbf2c0cee10 2024-03-29T10:11:08.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.068+0000 7fbf31400700 1 -- 172.21.15.12:0/3688132839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c0cee10 msgr2=0x7fbf2c161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:08.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.068+0000 7fbf31400700 1 --2- 172.21.15.12:0/3688132839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c0cee10 0x7fbf2c161590 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c08b520 tx=0x7fbf2400b330 comp rx=0 tx=0).stop 2024-03-29T10:11:08.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.068+0000 7fbf31400700 1 -- 172.21.15.12:0/3688132839 shutdown_connections 2024-03-29T10:11:08.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.068+0000 7fbf31400700 1 --2- 172.21.15.12:0/3688132839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c0cee10 0x7fbf2c161590 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:08.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.068+0000 7fbf31400700 1 --2- 172.21.15.12:0/3688132839 >> v2:172.21.15.159:3300/0 conn(0x7fbf2c1611b0 0x7fbf2c15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:08.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.068+0000 7fbf31400700 1 -- 172.21.15.12:0/3688132839 >> 172.21.15.12:0/3688132839 conn(0x7fbf2c08aff0 msgr2=0x7fbf2c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:08.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.068+0000 7fbf31400700 1 -- 172.21.15.12:0/3688132839 shutdown_connections 2024-03-29T10:11:08.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.068+0000 7fbf31400700 1 -- 172.21.15.12:0/3688132839 wait complete. 2024-03-29T10:11:08.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.070+0000 7fbf31400700 1 Processor -- start 2024-03-29T10:11:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.071+0000 7fbf31400700 1 -- start start 2024-03-29T10:11:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf31400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c0cee10 0x7fbf2c0cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf31400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c15f4d0 0x7fbf2c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf31400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c1611b0 0x7fbf2c0b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf31400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c0b78f0 con 0x7fbf2c1611b0 2024-03-29T10:11:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf31400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c0b7bd0 con 0x7fbf2c0cee10 2024-03-29T10:11:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf31400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c0b7eb0 con 0x7fbf2c15f4d0 2024-03-29T10:11:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf2b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c1611b0 0x7fbf2c0b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf2b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c1611b0 0x7fbf2c0b8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39614/0 (socket says 172.21.15.12:39614) 2024-03-29T10:11:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf2b200700 1 -- 172.21.15.12:0/1764029735 learned_addr learned my addr 172.21.15.12:0/1764029735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf2a800700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c0cee10 0x7fbf2c0cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.072+0000 7fbf29e00700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c15f4d0 0x7fbf2c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf2b200700 1 -- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c15f4d0 msgr2=0x7fbf2c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf2b200700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c15f4d0 0x7fbf2c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf2b200700 1 -- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c0cee10 msgr2=0x7fbf2c0cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf2b200700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c0cee10 0x7fbf2c0cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf2b200700 1 -- 172.21.15.12:0/1764029735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24009b30 con 0x7fbf2c1611b0 2024-03-29T10:11:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf2b200700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c1611b0 0x7fbf2c0b8990 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c08ce10 tx=0x7fbf24004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf22a00700 1 -- 172.21.15.12:0/1764029735 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24004840 con 0x7fbf2c1611b0 2024-03-29T10:11:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf22a00700 1 -- 172.21.15.12:0/1764029735 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf240099b0 con 0x7fbf2c1611b0 2024-03-29T10:11:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf31400700 1 -- 172.21.15.12:0/1764029735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2c0b8190 con 0x7fbf2c1611b0 2024-03-29T10:11:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf22a00700 1 -- 172.21.15.12:0/1764029735 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2401c680 con 0x7fbf2c1611b0 2024-03-29T10:11:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.073+0000 7fbf31400700 1 -- 172.21.15.12:0/1764029735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2c156e80 con 0x7fbf2c1611b0 2024-03-29T10:11:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.075+0000 7fbf17e00700 1 -- 172.21.15.12:0/1764029735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2c067720 con 0x7fbf2c1611b0 2024-03-29T10:11:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.075+0000 7fbf22a00700 1 -- 172.21.15.12:0/1764029735 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 46) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbf2401c7e0 con 0x7fbf2c1611b0 2024-03-29T10:11:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.076+0000 7fbf22a00700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbf0c06adb0 0x7fbf0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.076+0000 7fbf22a00700 1 -- 172.21.15.12:0/1764029735 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf2408ade0 con 0x7fbf2c1611b0 2024-03-29T10:11:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.077+0000 7fbf2a800700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbf0c06adb0 0x7fbf0c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.077+0000 7fbf2a800700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbf0c06adb0 0x7fbf0c06d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009b40 tx=0x7fbf1c009210 comp rx=0 tx=0).ready entity=mgr.6940 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.081+0000 7fbf22a00700 1 -- 172.21.15.12:0/1764029735 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf24055e30 con 0x7fbf2c1611b0 2024-03-29T10:11:08.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:08.354+0000 7fbf17e00700 1 -- 172.21.15.12:0/1764029735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fbf2c0b5d00 con 0x7fbf2c1611b0 2024-03-29T10:11:09.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.153+0000 7fbf22a00700 1 -- 172.21.15.12:0/1764029735 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 47) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fbf24089300 con 0x7fbf2c1611b0 2024-03-29T10:11:09.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.154+0000 7fbf22a00700 1 -- 172.21.15.12:0/1764029735 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v47) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbf24070d80 con 0x7fbf2c1611b0 2024-03-29T10:11:09.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.156+0000 7fbf17e00700 1 -- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbf0c06adb0 msgr2=0x7fbf0c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbf0c06adb0 0x7fbf0c06d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009b40 tx=0x7fbf1c009210 comp rx=0 tx=0).stop 2024-03-29T10:11:09.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 -- 172.21.15.12:0/1764029735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c1611b0 msgr2=0x7fbf2c0b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c1611b0 0x7fbf2c0b8990 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c08ce10 tx=0x7fbf24004ad0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 -- 172.21.15.12:0/1764029735 shutdown_connections 2024-03-29T10:11:09.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbf0c06adb0 0x7fbf0c06d270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf2c1611b0 0x7fbf2c0b8990 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c0cee10 0x7fbf2c0cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 --2- 172.21.15.12:0/1764029735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c15f4d0 0x7fbf2c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 -- 172.21.15.12:0/1764029735 >> 172.21.15.12:0/1764029735 conn(0x7fbf2c08aff0 msgr2=0x7fbf2c15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:09.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 -- 172.21.15.12:0/1764029735 shutdown_connections 2024-03-29T10:11:09.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.157+0000 7fbf17e00700 1 -- 172.21.15.12:0/1764029735 wait complete. 2024-03-29T10:11:09.179 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T10:11:09.377 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:09.378+0000 7fa3e7435240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:11:09.384 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:09.385+0000 7ff3009c1240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:11:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.387+0000 7f6a78e00700 1 Processor -- start 2024-03-29T10:11:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.388+0000 7f6a78e00700 1 -- start start 2024-03-29T10:11:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.389+0000 7f6a78e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6a740ced80 0x7f6a740d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.389+0000 7f6a78e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6a7415fcb0 0x7f6a741600e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.389+0000 7f6a78e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7415f820 con 0x7f6a7415dad0 2024-03-29T10:11:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.389+0000 7f6a78e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a740cfcc0 con 0x7f6a7415fcb0 2024-03-29T10:11:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.389+0000 7f6a78e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a740cd9f0 con 0x7f6a740ced80 2024-03-29T10:11:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.389+0000 7f6a71e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6a7415dad0 0x7f6a740d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60206/0 (socket says 172.21.15.12:60206) 2024-03-29T10:11:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.389+0000 7f6a71e00700 1 -- 172.21.15.12:0/50479852 learned_addr learned my addr 172.21.15.12:0/50479852 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:11:09.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.389+0000 7f6a72800700 1 --2- 172.21.15.12:0/50479852 >> v2:172.21.15.12:3300/0 conn(0x7f6a7415fcb0 0x7f6a741600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:09.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.389+0000 7f6a71400700 1 --2- 172.21.15.12:0/50479852 >> v2:172.21.15.159:3300/0 conn(0x7f6a740ced80 0x7f6a740d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.390+0000 7f6a71400700 1 -- 172.21.15.12:0/50479852 >> v2:172.21.15.12:3300/0 conn(0x7f6a7415fcb0 msgr2=0x7f6a741600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.390+0000 7f6a71400700 1 --2- 172.21.15.12:0/50479852 >> v2:172.21.15.12:3300/0 conn(0x7f6a7415fcb0 0x7f6a741600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.390+0000 7f6a71400700 1 -- 172.21.15.12:0/50479852 >> v1:172.21.15.12:6789/0 conn(0x7f6a7415dad0 legacy=0x7f6a740d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.390+0000 7f6a71400700 1 -- 172.21.15.12:0/50479852 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a740cf420 con 0x7f6a740ced80 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.390+0000 7f6a71400700 1 --2- 172.21.15.12:0/50479852 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a740ced80 0x7f6a740d1d40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6a7408b730 tx=0x7f6a60009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30e772d492aaa463 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.390+0000 7f6a70a00700 1 -- 172.21.15.12:0/50479852 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60011830 con 0x7f6a740ced80 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.390+0000 7f6a70a00700 1 -- 172.21.15.12:0/50479852 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a60011990 con 0x7f6a740ced80 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.390+0000 7f6a70a00700 1 -- 172.21.15.12:0/50479852 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60011d20 con 0x7f6a740ced80 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.391+0000 7f6a78e00700 1 -- 172.21.15.12:0/50479852 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a740ced80 msgr2=0x7f6a740d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.391+0000 7f6a78e00700 1 --2- 172.21.15.12:0/50479852 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a740ced80 0x7f6a740d1d40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6a7408b730 tx=0x7f6a60009b20 comp rx=0 tx=0).stop 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.391+0000 7f6a78e00700 1 -- 172.21.15.12:0/50479852 shutdown_connections 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.391+0000 7f6a78e00700 1 --2- 172.21.15.12:0/50479852 >> v2:172.21.15.12:3300/0 conn(0x7f6a7415fcb0 0x7f6a741600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.391+0000 7f6a78e00700 1 --2- 172.21.15.12:0/50479852 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a740ced80 0x7f6a740d1d40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.391+0000 7f6a78e00700 1 -- 172.21.15.12:0/50479852 >> 172.21.15.12:0/50479852 conn(0x7f6a7408aff0 msgr2=0x7f6a740d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.391+0000 7f6a78e00700 1 -- 172.21.15.12:0/50479852 shutdown_connections 2024-03-29T10:11:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.391+0000 7f6a78e00700 1 -- 172.21.15.12:0/50479852 wait complete. 2024-03-29T10:11:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.393+0000 7f6a78e00700 1 Processor -- start 2024-03-29T10:11:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.393+0000 7f6a78e00700 1 -- start start 2024-03-29T10:11:09.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a78e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a740ced80 0x7f6a740c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a78e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a7415dad0 0x7f6a74154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a78e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a7415fcb0 0x7f6a740cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a78e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740b6f40 con 0x7f6a7415dad0 2024-03-29T10:11:09.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a78e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740b6370 con 0x7f6a7415fcb0 2024-03-29T10:11:09.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a78e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740b6610 con 0x7f6a740ced80 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a71400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a7415dad0 0x7f6a74154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a71400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a7415dad0 0x7f6a74154af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44744/0 (socket says 172.21.15.12:44744) 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a71400700 1 -- 172.21.15.12:0/2317590713 learned_addr learned my addr 172.21.15.12:0/2317590713 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a71e00700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a740ced80 0x7f6a740c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.394+0000 7f6a72800700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a7415fcb0 0x7f6a740cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.395+0000 7f6a71400700 1 -- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a740ced80 msgr2=0x7f6a740c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.395+0000 7f6a71400700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a740ced80 0x7f6a740c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.395+0000 7f6a71400700 1 -- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a7415fcb0 msgr2=0x7f6a740cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.395+0000 7f6a71400700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a7415fcb0 0x7f6a740cbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.395+0000 7f6a71400700 1 -- 172.21.15.12:0/2317590713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a600094a0 con 0x7f6a7415dad0 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.395+0000 7f6a71400700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a7415dad0 0x7f6a74154af0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009b60 tx=0x7f6a60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.395+0000 7f6a6a000700 1 -- 172.21.15.12:0/2317590713 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a600036a0 con 0x7f6a7415dad0 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.395+0000 7f6a6a000700 1 -- 172.21.15.12:0/2317590713 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6001f070 con 0x7f6a7415dad0 2024-03-29T10:11:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.395+0000 7f6a6a000700 1 -- 172.21.15.12:0/2317590713 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60003a60 con 0x7f6a7415dad0 2024-03-29T10:11:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.395+0000 7f6a78e00700 1 -- 172.21.15.12:0/2317590713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a740b68f0 con 0x7f6a7415dad0 2024-03-29T10:11:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.396+0000 7f6a78e00700 1 -- 172.21.15.12:0/2317590713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a740ce8d0 con 0x7f6a7415dad0 2024-03-29T10:11:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.397+0000 7f6a78e00700 1 -- 172.21.15.12:0/2317590713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a740b70b0 con 0x7f6a7415dad0 2024-03-29T10:11:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.398+0000 7f6a6a000700 1 -- 172.21.15.12:0/2317590713 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 47) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f6a60003bc0 con 0x7f6a7415dad0 2024-03-29T10:11:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.399+0000 7f6a6a000700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a5406ae00 0x7f6a5406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.399+0000 7f6a6a000700 1 -- 172.21.15.12:0/2317590713 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a60021070 con 0x7f6a7415dad0 2024-03-29T10:11:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.399+0000 7f6a71e00700 1 -- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a5406ae00 msgr2=0x7f6a5406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:11:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.399+0000 7f6a71e00700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a5406ae00 0x7f6a5406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:11:09.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.402+0000 7f6a6a000700 1 -- 172.21.15.12:0/2317590713 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a6008ae60 con 0x7f6a7415dad0 2024-03-29T10:11:09.563 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:09.563+0000 7fa3e7435240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:11:09.593 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:09.593+0000 7ff3009c1240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:11:09.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.600+0000 7f6a71e00700 1 -- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a5406ae00 msgr2=0x7f6a5406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:11:09.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.600+0000 7f6a71e00700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a5406ae00 0x7f6a5406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:11:09.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.656+0000 7f6a78e00700 1 -- 172.21.15.12:0/2317590713 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f6a74067720 con 0x7f6a7415dad0 2024-03-29T10:11:09.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.660+0000 7f6a6a000700 1 -- 172.21.15.12:0/2317590713 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6a600150d0 con 0x7f6a7415dad0 2024-03-29T10:11:09.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 -- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a5406ae00 msgr2=0x7f6a5406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:11:09.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a5406ae00 0x7f6a5406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 -- 172.21.15.12:0/2317590713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a7415dad0 msgr2=0x7f6a74154af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a7415dad0 0x7f6a74154af0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009b60 tx=0x7f6a60004750 comp rx=0 tx=0).stop 2024-03-29T10:11:09.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 -- 172.21.15.12:0/2317590713 shutdown_connections 2024-03-29T10:11:09.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a5406ae00 0x7f6a5406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a7415fcb0 0x7f6a740cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a740ced80 0x7f6a740c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 --2- 172.21.15.12:0/2317590713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a7415dad0 0x7f6a74154af0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 -- 172.21.15.12:0/2317590713 >> 172.21.15.12:0/2317590713 conn(0x7f6a7408aff0 msgr2=0x7f6a7415c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:09.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 -- 172.21.15.12:0/2317590713 shutdown_connections 2024-03-29T10:11:09.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.663+0000 7f6a78e00700 1 -- 172.21.15.12:0/2317590713 wait complete. 2024-03-29T10:11:09.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T10:11:09.881 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:09.882+0000 7fa3e7435240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:11:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.888+0000 7f33ba400700 1 Processor -- start 2024-03-29T10:11:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.889+0000 7f33ba400700 1 -- start start 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.890+0000 7f33ba400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f33b415dfe0 0x7f33b40ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.890+0000 7f33ba400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f33b40ce670 0x7f33b40cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.890+0000 7f33ba400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b40cb310 con 0x7f33b415a5c0 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.890+0000 7f33ba400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b40ceb00 con 0x7f33b415dfe0 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.890+0000 7f33ba400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b40711a0 con 0x7f33b40ce670 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.890+0000 7f33b2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f33b415dfe0 0x7f33b40ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.890+0000 7f33b2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f33b415dfe0 0x7f33b40ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44758/0 (socket says 172.21.15.12:44758) 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.890+0000 7f33b2a00700 1 -- 172.21.15.12:0/3506956992 learned_addr learned my addr 172.21.15.12:0/3506956992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.890+0000 7f33b3e00700 1 --2- 172.21.15.12:0/3506956992 >> v2:172.21.15.159:3300/0 conn(0x7f33b40ce670 0x7f33b40cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.891+0000 7f33b2a00700 1 -- 172.21.15.12:0/3506956992 >> v2:172.21.15.159:3300/0 conn(0x7f33b40ce670 msgr2=0x7f33b40cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.891+0000 7f33b2a00700 1 --2- 172.21.15.12:0/3506956992 >> v2:172.21.15.159:3300/0 conn(0x7f33b40ce670 0x7f33b40cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.891+0000 7f33b2a00700 1 -- 172.21.15.12:0/3506956992 >> v1:172.21.15.159:6789/0 conn(0x7f33b415a5c0 legacy=0x7f33b40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.891+0000 7f33b2a00700 1 -- 172.21.15.12:0/3506956992 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b4071480 con 0x7f33b415dfe0 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.891+0000 7f33b2a00700 1 --2- 172.21.15.12:0/3506956992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 0x7f33b40ce130 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f33b4070950 tx=0x7f33a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94345a0fa164ec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.891+0000 7f33b2000700 1 -- 172.21.15.12:0/3506956992 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a000f8a0 con 0x7f33b415dfe0 2024-03-29T10:11:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.891+0000 7f33b2000700 1 -- 172.21.15.12:0/3506956992 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a000fa00 con 0x7f33b415dfe0 2024-03-29T10:11:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.891+0000 7f33b2000700 1 -- 172.21.15.12:0/3506956992 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a000fd90 con 0x7f33b415dfe0 2024-03-29T10:11:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.892+0000 7f33ba400700 1 -- 172.21.15.12:0/3506956992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 msgr2=0x7f33b40ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.892+0000 7f33ba400700 1 --2- 172.21.15.12:0/3506956992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 0x7f33b40ce130 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f33b4070950 tx=0x7f33a0009bd0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.892+0000 7f33ba400700 1 -- 172.21.15.12:0/3506956992 shutdown_connections 2024-03-29T10:11:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.892+0000 7f33ba400700 1 --2- 172.21.15.12:0/3506956992 >> v2:172.21.15.159:3300/0 conn(0x7f33b40ce670 0x7f33b40cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.892+0000 7f33ba400700 1 --2- 172.21.15.12:0/3506956992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 0x7f33b40ce130 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.892+0000 7f33ba400700 1 -- 172.21.15.12:0/3506956992 >> 172.21.15.12:0/3506956992 conn(0x7f33b40b0d70 msgr2=0x7f33b40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.892+0000 7f33ba400700 1 -- 172.21.15.12:0/3506956992 shutdown_connections 2024-03-29T10:11:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.892+0000 7f33ba400700 1 -- 172.21.15.12:0/3506956992 wait complete. 2024-03-29T10:11:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.894+0000 7f33ba400700 1 Processor -- start 2024-03-29T10:11:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.894+0000 7f33ba400700 1 -- start start 2024-03-29T10:11:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33ba400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33b40ce670 0x7f33b41f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33ba400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33b415a5c0 0x7f33b415aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33ba400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 0x7f33b40cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33ba400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b415d4f0 con 0x7f33b415dfe0 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33ba400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b415c940 con 0x7f33b415a5c0 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33ba400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b415cbc0 con 0x7f33b40ce670 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33b3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 0x7f33b40cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33b3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 0x7f33b40cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44774/0 (socket says 172.21.15.12:44774) 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33b3e00700 1 -- 172.21.15.12:0/2284234222 learned_addr learned my addr 172.21.15.12:0/2284234222 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33b2a00700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33b415a5c0 0x7f33b415aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.895+0000 7f33b3400700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33b40ce670 0x7f33b41f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33b3e00700 1 -- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33b40ce670 msgr2=0x7f33b41f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33b3e00700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33b40ce670 0x7f33b41f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33b3e00700 1 -- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33b415a5c0 msgr2=0x7f33b415aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33b3e00700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33b415a5c0 0x7f33b415aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33b3e00700 1 -- 172.21.15.12:0/2284234222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a0009550 con 0x7f33b415dfe0 2024-03-29T10:11:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33b2a00700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33b415a5c0 0x7f33b415aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:11:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33b3e00700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 0x7f33b40cd020 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009dd0 tx=0x7f33ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33ab400700 1 -- 172.21.15.12:0/2284234222 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac009c40 con 0x7f33b415dfe0 2024-03-29T10:11:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33ab400700 1 -- 172.21.15.12:0/2284234222 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ac00f040 con 0x7f33b415dfe0 2024-03-29T10:11:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33ba400700 1 -- 172.21.15.12:0/2284234222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b415cea0 con 0x7f33b415dfe0 2024-03-29T10:11:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33ab400700 1 -- 172.21.15.12:0/2284234222 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac00ceb0 con 0x7f33b415dfe0 2024-03-29T10:11:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.896+0000 7f33ba400700 1 -- 172.21.15.12:0/2284234222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b41f2fa0 con 0x7f33b415dfe0 2024-03-29T10:11:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.898+0000 7f33ba400700 1 -- 172.21.15.12:0/2284234222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b415d6a0 con 0x7f33b415dfe0 2024-03-29T10:11:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.899+0000 7f33ab400700 1 -- 172.21.15.12:0/2284234222 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 47) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f33ac014480 con 0x7f33b415dfe0 2024-03-29T10:11:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.900+0000 7f33ab400700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f339406ae00 0x7f339406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.900+0000 7f33ab400700 1 -- 172.21.15.12:0/2284234222 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33ac08a1f0 con 0x7f33b415dfe0 2024-03-29T10:11:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.900+0000 7f33b3400700 1 -- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f339406ae00 msgr2=0x7f339406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:11:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.900+0000 7f33b3400700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f339406ae00 0x7f339406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:11:09.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:09.905+0000 7f33ab400700 1 -- 172.21.15.12:0/2284234222 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f33ac055110 con 0x7f33b415dfe0 2024-03-29T10:11:09.935 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:09.935+0000 7ff3009c1240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:11:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.101+0000 7f33b3400700 1 -- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f339406ae00 msgr2=0x7f339406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:11:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.101+0000 7f33b3400700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f339406ae00 0x7f339406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:11:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.119+0000 7f33ba400700 1 -- 172.21.15.12:0/2284234222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f33b40ae9c0 con 0x7f33b415dfe0 2024-03-29T10:11:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.124+0000 7f33ab400700 1 -- 172.21.15.12:0/2284234222 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f33ac058730 con 0x7f33b415dfe0 2024-03-29T10:11:10.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 -- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f339406ae00 msgr2=0x7f339406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:11:10.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f339406ae00 0x7f339406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 -- 172.21.15.12:0/2284234222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 msgr2=0x7f33b40cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:10.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 0x7f33b40cd020 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009dd0 tx=0x7f33ac00c3a0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 -- 172.21.15.12:0/2284234222 shutdown_connections 2024-03-29T10:11:10.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f339406ae00 0x7f339406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33b415dfe0 0x7f33b40cd020 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33b40ce670 0x7f33b41f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 --2- 172.21.15.12:0/2284234222 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33b415a5c0 0x7f33b415aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 -- 172.21.15.12:0/2284234222 >> 172.21.15.12:0/2284234222 conn(0x7f33b40b0d70 msgr2=0x7f33b406d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:10.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 -- 172.21.15.12:0/2284234222 shutdown_connections 2024-03-29T10:11:10.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.126+0000 7f33ba400700 1 -- 172.21.15.12:0/2284234222 wait complete. 2024-03-29T10:11:10.145 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2024-03-29T10:11:10.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.351+0000 7f9a39c00700 1 Processor -- start 2024-03-29T10:11:10.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.351+0000 7f9a39c00700 1 -- start start 2024-03-29T10:11:10.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.353+0000 7f9a39c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a3415d800 0x7f9a340725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:10.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.353+0000 7f9a39c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9a340ce670 0x7f9a3415dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.353+0000 7f9a39c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a340709c0 con 0x7f9a340ce230 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.353+0000 7f9a39c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a340ceb00 con 0x7f9a340ce670 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.353+0000 7f9a39c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a340b40f0 con 0x7f9a3415d800 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.353+0000 7f9a32000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9a340ce230 0x7f9a340b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60220/0 (socket says 172.21.15.12:60220) 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.353+0000 7f9a32000700 1 -- 172.21.15.12:0/1839560432 learned_addr learned my addr 172.21.15.12:0/1839560432 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.353+0000 7f9a32a00700 1 --2- 172.21.15.12:0/1839560432 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a3415d800 0x7f9a340725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.353+0000 7f9a33400700 1 --2- 172.21.15.12:0/1839560432 >> v2:172.21.15.159:3300/0 conn(0x7f9a340ce670 0x7f9a3415dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.354+0000 7f9a31600700 1 -- 172.21.15.12:0/1839560432 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893074881 0 0) 0x7f9a340709c0 con 0x7f9a340ce230 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.354+0000 7f9a31600700 1 -- 172.21.15.12:0/1839560432 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a180035b0 con 0x7f9a340ce230 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.354+0000 7f9a32a00700 1 -- 172.21.15.12:0/1839560432 >> v2:172.21.15.159:3300/0 conn(0x7f9a340ce670 msgr2=0x7f9a3415dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.354+0000 7f9a32a00700 1 --2- 172.21.15.12:0/1839560432 >> v2:172.21.15.159:3300/0 conn(0x7f9a340ce670 0x7f9a3415dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.354+0000 7f9a32a00700 1 -- 172.21.15.12:0/1839560432 >> v1:172.21.15.12:6789/0 conn(0x7f9a340ce230 legacy=0x7f9a340b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:10.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.354+0000 7f9a32a00700 1 -- 172.21.15.12:0/1839560432 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a34070c80 con 0x7f9a3415d800 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.354+0000 7f9a32a00700 1 --2- 172.21.15.12:0/1839560432 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a3415d800 0x7f9a340725c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9a24009840 tx=0x7f9a24009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=371f3bb16a93d1a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.354+0000 7f9a31600700 1 -- 172.21.15.12:0/1839560432 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24011830 con 0x7f9a3415d800 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.354+0000 7f9a31600700 1 -- 172.21.15.12:0/1839560432 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a24011990 con 0x7f9a3415d800 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.354+0000 7f9a31600700 1 -- 172.21.15.12:0/1839560432 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24011d20 con 0x7f9a3415d800 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.355+0000 7f9a39c00700 1 -- 172.21.15.12:0/1839560432 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a3415d800 msgr2=0x7f9a340725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.355+0000 7f9a39c00700 1 --2- 172.21.15.12:0/1839560432 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a3415d800 0x7f9a340725c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9a24009840 tx=0x7f9a24009b50 comp rx=0 tx=0).stop 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.355+0000 7f9a39c00700 1 -- 172.21.15.12:0/1839560432 shutdown_connections 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.355+0000 7f9a39c00700 1 --2- 172.21.15.12:0/1839560432 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a3415d800 0x7f9a340725c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.355+0000 7f9a39c00700 1 --2- 172.21.15.12:0/1839560432 >> v2:172.21.15.159:3300/0 conn(0x7f9a340ce670 0x7f9a3415dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.355+0000 7f9a39c00700 1 -- 172.21.15.12:0/1839560432 >> 172.21.15.12:0/1839560432 conn(0x7f9a340b0d70 msgr2=0x7f9a340b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.355+0000 7f9a39c00700 1 -- 172.21.15.12:0/1839560432 shutdown_connections 2024-03-29T10:11:10.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.356+0000 7f9a39c00700 1 -- 172.21.15.12:0/1839560432 wait complete. 2024-03-29T10:11:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.357+0000 7f9a39c00700 1 Processor -- start 2024-03-29T10:11:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.357+0000 7f9a39c00700 1 -- start start 2024-03-29T10:11:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.358+0000 7f9a39c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a340ce230 0x7f9a340af600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.358+0000 7f9a39c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a340ce670 0x7f9a341f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.358+0000 7f9a39c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a3415d800 0x7f9a341f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.358+0000 7f9a39c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3406ea00 con 0x7f9a340ce230 2024-03-29T10:11:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.358+0000 7f9a39c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3406dd80 con 0x7f9a3415d800 2024-03-29T10:11:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.358+0000 7f9a39c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3406e060 con 0x7f9a340ce670 2024-03-29T10:11:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.358+0000 7f9a32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a340ce230 0x7f9a340af600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a340ce230 0x7f9a340af600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44778/0 (socket says 172.21.15.12:44778) 2024-03-29T10:11:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a32a00700 1 -- 172.21.15.12:0/3450868714 learned_addr learned my addr 172.21.15.12:0/3450868714 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a33400700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a3415d800 0x7f9a341f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a32000700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a340ce670 0x7f9a341f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a32a00700 1 -- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a340ce670 msgr2=0x7f9a341f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a32a00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a340ce670 0x7f9a341f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a32a00700 1 -- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a3415d800 msgr2=0x7f9a341f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a32a00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a3415d800 0x7f9a341f7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a32a00700 1 -- 172.21.15.12:0/3450868714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a240094a0 con 0x7f9a340ce230 2024-03-29T10:11:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a32000700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a340ce670 0x7f9a341f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:11:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a32a00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a340ce230 0x7f9a340af600 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f9a240090b0 tx=0x7f9a24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.359+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a240036a0 con 0x7f9a340ce230 2024-03-29T10:11:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.360+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2401f070 con 0x7f9a340ce230 2024-03-29T10:11:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.360+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24003a00 con 0x7f9a340ce230 2024-03-29T10:11:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.360+0000 7f9a39c00700 1 -- 172.21.15.12:0/3450868714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3406e340 con 0x7f9a340ce230 2024-03-29T10:11:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.360+0000 7f9a39c00700 1 -- 172.21.15.12:0/3450868714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a340afb40 con 0x7f9a340ce230 2024-03-29T10:11:10.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.361+0000 7f9a39c00700 1 -- 172.21.15.12:0/3450868714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a34148890 con 0x7f9a340ce230 2024-03-29T10:11:10.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.362+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 47) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f9a24003b60 con 0x7f9a340ce230 2024-03-29T10:11:10.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.362+0000 7f9a2aa00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:10.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.363+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a24021070 con 0x7f9a340ce230 2024-03-29T10:11:10.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.363+0000 7f9a32000700 1 -- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 msgr2=0x7f9a1406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:11:10.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.363+0000 7f9a32000700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:11:10.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.367+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9a24055da0 con 0x7f9a340ce230 2024-03-29T10:11:10.410 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:10.411+0000 7fa3e7435240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:11:10.522 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:10.522+0000 7ff3009c1240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:11:10.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.564+0000 7f9a32000700 1 -- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 msgr2=0x7f9a1406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:11:10.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.564+0000 7f9a32000700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:11:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.639+0000 7f9a39c00700 1 -- 172.21.15.12:0/3450868714 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a340cd9e0 con 0x7f9a1406ae00 2024-03-29T10:11:10.670 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:10.671+0000 7fa3e7435240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:11:10.788 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:10.789+0000 7ff3009c1240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:11:10.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.965+0000 7f9a32000700 1 -- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 msgr2=0x7f9a1406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:11:10.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:10.965+0000 7f9a32000700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T10:11:11.126 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:11.127+0000 7fa3e7435240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:11:11.266 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:11.266+0000 7ff3009c1240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:11:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:11.766+0000 7f9a32000700 1 -- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 msgr2=0x7f9a1406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:11:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:11.766+0000 7f9a32000700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T10:11:11.832 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:11.833+0000 7fa3e7435240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:11:11.924 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:11.925+0000 7fa3e7435240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:11:12.009 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:12.010+0000 7ff3009c1240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:11:12.083 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:12.084+0000 7fa3e7435240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:11:12.105 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:12.106+0000 7ff3009c1240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:11:12.273 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:12.273+0000 7ff3009c1240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:11:12.296 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:12.297+0000 7fa3e7435240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:11:12.386 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:12.387+0000 7fa3e7435240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:11:12.483 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:12.484+0000 7fa3e7435240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:11:12.491 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:12.492+0000 7ff3009c1240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:11:12.581 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:12.582+0000 7fa3e7435240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:11:12.587 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:12.588+0000 7ff3009c1240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:11:12.678 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:12.679+0000 7fa3e7435240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:11:12.690 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:12.691+0000 7ff3009c1240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:11:12.794 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:12.794+0000 7ff3009c1240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:11:12.888 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:12.889+0000 7fa3e7435240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:11:12.896 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:12.897+0000 7ff3009c1240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:11:13.118 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:13.119+0000 7ff3009c1240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:11:13.146 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:13.147+0000 7fa3e7435240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:11:13.335 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:13.336+0000 7fa3e7435240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:11:13.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:13.368+0000 7f9a32000700 1 -- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 msgr2=0x7f9a1406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:11:13.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:13.368+0000 7f9a32000700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T10:11:13.382 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:13.383+0000 7ff3009c1240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:11:13.427 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:13.428+0000 7fa3e7435240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:11:13.581 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:13.582+0000 7ff3009c1240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:11:13.679 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:13.679+0000 7ff3009c1240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:11:14.158 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:14.159+0000 7fa3e7435240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:11:14.369 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:14.370+0000 7fa3e7435240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:11:14.446 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:14.446+0000 7ff3009c1240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:11:14.573 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:14.574+0000 7fa3e7435240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:11:14.668 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:14.669+0000 7ff3009c1240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:11:14.886 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:14.886+0000 7ff3009c1240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:11:15.049 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:15.050+0000 7fa3e7435240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:11:15.254 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:15.255+0000 7fa3e7435240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:11:15.386 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:15.386+0000 7ff3009c1240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:11:15.602 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:15.603+0000 7ff3009c1240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:11:15.694 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:15.695+0000 7fa3e7435240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:11:15.804 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:15.805+0000 7fa3e7435240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:11:15.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:15.816+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 48) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f9a24056170 con 0x7f9a340ce230 2024-03-29T10:11:15.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:15.816+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 msgr2=0x7f9a1406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:11:15.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:15.816+0000 7f9a2aa00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:16.001 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:16.002+0000 7fa353800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:11:16.001 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:16.002+0000 7fa353800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:11:16.001 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:16.002+0000 7fa353800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:11:16.001 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:16.002+0000 7fa353800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:11:16.001 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:16.002+0000 7fa353800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:11:16.001 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:16.002+0000 7fa34fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:11:16.001 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:16.002+0000 7fa34fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:11:16.002 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:16.002+0000 7fa34fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:11:16.002 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:16.002+0000 7fa34fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:11:16.002 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:11:16.002+0000 7fa34fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:11:16.071 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:16.072+0000 7ff3009c1240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:11:16.188 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:11:16.188+0000 7ff3009c1240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:11:16.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.822+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 49) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f9a240891b0 con 0x7f9a340ce230 2024-03-29T10:11:16.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.822+0000 7f9a2aa00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:16.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.822+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a14035990 con 0x7f9a1406ae00 2024-03-29T10:11:16.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.823+0000 7f9a32000700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.830+0000 7f9a32000700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d240 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a20007fd0 tx=0x7f9a20009040 comp rx=0 tx=0).ready entity=mgr.7294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.910+0000 7f9a2aa00700 1 -- 172.21.15.12:0/3450868714 <== mgr.7294 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f9a14035990 con 0x7f9a1406ae00 2024-03-29T10:11:16.910 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /bad-path 2024-03-29T10:11:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 -- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 msgr2=0x7f9a1406d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d240 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a20007fd0 tx=0x7f9a20009040 comp rx=0 tx=0).stop 2024-03-29T10:11:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 -- 172.21.15.12:0/3450868714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a340ce230 msgr2=0x7f9a340af600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a340ce230 0x7f9a340af600 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f9a240090b0 tx=0x7f9a24004750 comp rx=0 tx=0).stop 2024-03-29T10:11:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 -- 172.21.15.12:0/3450868714 shutdown_connections 2024-03-29T10:11:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a1406ae00 0x7f9a1406d240 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a3415d800 0x7f9a341f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a340ce230 0x7f9a340af600 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 --2- 172.21.15.12:0/3450868714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a340ce670 0x7f9a341f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 -- 172.21.15.12:0/3450868714 >> 172.21.15.12:0/3450868714 conn(0x7f9a340b0d70 msgr2=0x7f9a340b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 -- 172.21.15.12:0/3450868714 shutdown_connections 2024-03-29T10:11:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:16.912+0000 7f9a39c00700 1 -- 172.21.15.12:0/3450868714 wait complete. 2024-03-29T10:11:16.932 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2024-03-29T10:11:16.932 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-03-29T10:11:17.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.133+0000 7f159ca00700 1 Processor -- start 2024-03-29T10:11:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.134+0000 7f159ca00700 1 -- start start 2024-03-29T10:11:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f159ca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15980cbb30 0x7f15980cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f159ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980cd020 0x7f15980cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f159ca00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15980cc450 con 0x7f15980b2070 2024-03-29T10:11:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f159ca00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15980cdae0 con 0x7f15980cbb30 2024-03-29T10:11:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f159ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15980cddc0 con 0x7f15980cd020 2024-03-29T10:11:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f1595e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15980cbb30 0x7f15980cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f1595e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f15980cbb30 0x7f15980cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44790/0 (socket says 172.21.15.12:44790) 2024-03-29T10:11:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f1595e00700 1 -- 172.21.15.12:0/966165966 learned_addr learned my addr 172.21.15.12:0/966165966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f1595400700 1 --2- 172.21.15.12:0/966165966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980cd020 0x7f15980cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f1595e00700 1 -- 172.21.15.12:0/966165966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980cd020 msgr2=0x7f15980cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:11:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f1595e00700 1 --2- 172.21.15.12:0/966165966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980cd020 0x7f15980cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f1595e00700 1 -- 172.21.15.12:0/966165966 >> v1:172.21.15.12:6789/0 conn(0x7f15980b2070 legacy=0x7f159806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.135+0000 7f1595e00700 1 -- 172.21.15.12:0/966165966 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15980ce0a0 con 0x7f15980cbb30 2024-03-29T10:11:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.136+0000 7f1595e00700 1 --2- 172.21.15.12:0/966165966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 0x7f15980cbf10 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f15880098f0 tx=0x7f1588009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790eea2c20263ac9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.136+0000 7f1594a00700 1 -- 172.21.15.12:0/966165966 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1588011830 con 0x7f15980cbb30 2024-03-29T10:11:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.136+0000 7f1594a00700 1 -- 172.21.15.12:0/966165966 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1588011990 con 0x7f15980cbb30 2024-03-29T10:11:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.136+0000 7f1594a00700 1 -- 172.21.15.12:0/966165966 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1588011d20 con 0x7f15980cbb30 2024-03-29T10:11:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.136+0000 7f159ca00700 1 -- 172.21.15.12:0/966165966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 msgr2=0x7f15980cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.136+0000 7f159ca00700 1 --2- 172.21.15.12:0/966165966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 0x7f15980cbf10 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f15880098f0 tx=0x7f1588009c00 comp rx=0 tx=0).stop 2024-03-29T10:11:17.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.137+0000 7f159ca00700 1 -- 172.21.15.12:0/966165966 shutdown_connections 2024-03-29T10:11:17.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.137+0000 7f159ca00700 1 --2- 172.21.15.12:0/966165966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 0x7f15980cbf10 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:17.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.137+0000 7f159ca00700 1 --2- 172.21.15.12:0/966165966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980cd020 0x7f15980cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:17.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.137+0000 7f159ca00700 1 -- 172.21.15.12:0/966165966 >> 172.21.15.12:0/966165966 conn(0x7f15980b0d70 msgr2=0x7f15980b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:17.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.137+0000 7f159ca00700 1 -- 172.21.15.12:0/966165966 shutdown_connections 2024-03-29T10:11:17.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.137+0000 7f159ca00700 1 -- 172.21.15.12:0/966165966 wait complete. 2024-03-29T10:11:17.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.138+0000 7f159ca00700 1 Processor -- start 2024-03-29T10:11:17.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.139+0000 7f159ca00700 1 -- start start 2024-03-29T10:11:17.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.139+0000 7f159ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b2070 0x7f1598152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:17.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.139+0000 7f159ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 0x7f15980723d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f159ca00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15980cd020 0x7f15980ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f159ca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15980ae480 con 0x7f15980cbb30 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f159ca00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15980ca3a0 con 0x7f15980cd020 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f159ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15980c9ab0 con 0x7f15980b2070 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1595400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 0x7f15980723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1595400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 0x7f15980723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44792/0 (socket says 172.21.15.12:44792) 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1595400700 1 -- 172.21.15.12:0/596356641 learned_addr learned my addr 172.21.15.12:0/596356641 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1595e00700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b2070 0x7f1598152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1596800700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15980cd020 0x7f15980ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1595400700 1 -- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b2070 msgr2=0x7f1598152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1595400700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b2070 0x7f1598152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1595400700 1 -- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15980cd020 msgr2=0x7f15980ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:17.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1595400700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15980cd020 0x7f15980ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:17.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1595400700 1 -- 172.21.15.12:0/596356641 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1588009550 con 0x7f15980cbb30 2024-03-29T10:11:17.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.140+0000 7f1595400700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 0x7f15980723d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f158400e9b0 tx=0x7f158400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:17.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.141+0000 7f158e000700 1 -- 172.21.15.12:0/596356641 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1584009710 con 0x7f15980cbb30 2024-03-29T10:11:17.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.141+0000 7f158e000700 1 -- 172.21.15.12:0/596356641 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158400c850 con 0x7f15980cbb30 2024-03-29T10:11:17.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.141+0000 7f159ca00700 1 -- 172.21.15.12:0/596356641 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15980ae1a0 con 0x7f15980cbb30 2024-03-29T10:11:17.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.141+0000 7f158e000700 1 -- 172.21.15.12:0/596356641 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158400cbc0 con 0x7f15980cbb30 2024-03-29T10:11:17.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.141+0000 7f159ca00700 1 -- 172.21.15.12:0/596356641 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15980ca030 con 0x7f15980cbb30 2024-03-29T10:11:17.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.142+0000 7f159ca00700 1 -- 172.21.15.12:0/596356641 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1598067720 con 0x7f15980cbb30 2024-03-29T10:11:17.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.143+0000 7f158e000700 1 -- 172.21.15.12:0/596356641 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 49) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f158400cd20 con 0x7f15980cbb30 2024-03-29T10:11:17.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.144+0000 7f158e000700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f157806ab30 0x7f157806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:11:17.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.144+0000 7f158e000700 1 -- 172.21.15.12:0/596356641 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f1584089200 con 0x7f15980cbb30 2024-03-29T10:11:17.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.145+0000 7f1595e00700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f157806ab30 0x7f157806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:11:17.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.145+0000 7f1595e00700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f157806ab30 0x7f157806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1588014040 tx=0x7f1588008030 comp rx=0 tx=0).ready entity=mgr.7294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:11:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.149+0000 7f158e000700 1 -- 172.21.15.12:0/596356641 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15840544f0 con 0x7f15980cbb30 2024-03-29T10:11:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.390+0000 7f159ca00700 1 -- 172.21.15.12:0/596356641 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f15981594a0 con 0x7f157806ab30 2024-03-29T10:11:17.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.402+0000 7f158e000700 1 -- 172.21.15.12:0/596356641 <== mgr.7294 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+302 (secure 0 0 0) 0x7f1588011b80 con 0x7f157806ab30 2024-03-29T10:11:17.402 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:11:17.402 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T10:11:16", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-03-29T10:11:17.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.404+0000 7f159ca00700 1 -- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f157806ab30 msgr2=0x7f157806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.404+0000 7f159ca00700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f157806ab30 0x7f157806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1588014040 tx=0x7f1588008030 comp rx=0 tx=0).stop 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.404+0000 7f159ca00700 1 -- 172.21.15.12:0/596356641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 msgr2=0x7f15980723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.404+0000 7f159ca00700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 0x7f15980723d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f158400e9b0 tx=0x7f158400ecc0 comp rx=0 tx=0).stop 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.404+0000 7f159ca00700 1 -- 172.21.15.12:0/596356641 shutdown_connections 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.404+0000 7f159ca00700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f157806ab30 0x7f157806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.404+0000 7f159ca00700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15980cd020 0x7f15980ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.404+0000 7f159ca00700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b2070 0x7f1598152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.404+0000 7f159ca00700 1 --2- 172.21.15.12:0/596356641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15980cbb30 0x7f15980723d0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.404+0000 7f159ca00700 1 -- 172.21.15.12:0/596356641 >> 172.21.15.12:0/596356641 conn(0x7f15980b0d70 msgr2=0x7f159815c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.405+0000 7f159ca00700 1 -- 172.21.15.12:0/596356641 shutdown_connections 2024-03-29T10:11:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:11:17.405+0000 7f159ca00700 1 -- 172.21.15.12:0/596356641 wait complete. 2024-03-29T10:11:17.422 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T10:11:16", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-03-29T10:11:27.690 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:11:27.695 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:11:57.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:11:57.814 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:12:00.923 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:00.924+0000 7fa33ee00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:00.923 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:00.924+0000 7fa33ee00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:00.923 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:00.924+0000 7fa33ee00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:00.923 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:00.924+0000 7fa33ee00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:00.923 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:00.924+0000 7fa33ee00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:17.423 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-03-29T10:12:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.637+0000 7f3a96200700 1 Processor -- start 2024-03-29T10:12:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.637+0000 7f3a96200700 1 -- start start 2024-03-29T10:12:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.638+0000 7f3a96200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3a900cd020 0x7f3a900cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.638+0000 7f3a96200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3a900b2070 0x7f3a900b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.638+0000 7f3a96200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a900cdae0 con 0x7f3a900cbb30 2024-03-29T10:12:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.638+0000 7f3a96200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a900cddc0 con 0x7f3a900b2070 2024-03-29T10:12:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.638+0000 7f3a96200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a900ce0a0 con 0x7f3a900cd020 2024-03-29T10:12:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.638+0000 7f3a8fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3a900b2070 0x7f3a900b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.638+0000 7f3a8fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3a900b2070 0x7f3a900b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38356/0 (socket says 172.21.15.12:38356) 2024-03-29T10:12:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.638+0000 7f3a8fe00700 1 -- 172.21.15.12:0/3816775368 learned_addr learned my addr 172.21.15.12:0/3816775368 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.639+0000 7f3a8ea00700 1 --2- 172.21.15.12:0/3816775368 >> v2:172.21.15.159:3300/0 conn(0x7f3a900cd020 0x7f3a900cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.639+0000 7f3a8fe00700 1 -- 172.21.15.12:0/3816775368 >> v2:172.21.15.159:3300/0 conn(0x7f3a900cd020 msgr2=0x7f3a900cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.639+0000 7f3a8fe00700 1 --2- 172.21.15.12:0/3816775368 >> v2:172.21.15.159:3300/0 conn(0x7f3a900cd020 0x7f3a900cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.639+0000 7f3a8fe00700 1 -- 172.21.15.12:0/3816775368 >> v1:172.21.15.159:6789/0 conn(0x7f3a900cbb30 legacy=0x7f3a900cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.639+0000 7f3a8fe00700 1 -- 172.21.15.12:0/3816775368 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9006dc20 con 0x7f3a900b2070 2024-03-29T10:12:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.639+0000 7f3a8ea00700 1 --2- 172.21.15.12:0/3816775368 >> v2:172.21.15.159:3300/0 conn(0x7f3a900cd020 0x7f3a900cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.639+0000 7f3a8fe00700 1 --2- 172.21.15.12:0/3816775368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900b2070 0x7f3a900b2500 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3a900b1e30 tx=0x7f3a8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5ca5032f3aeb7f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.639+0000 7f3a8e000700 1 -- 172.21.15.12:0/3816775368 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8800e070 con 0x7f3a900b2070 2024-03-29T10:12:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.639+0000 7f3a8e000700 1 -- 172.21.15.12:0/3816775368 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8800b9c0 con 0x7f3a900b2070 2024-03-29T10:12:17.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.639+0000 7f3a8e000700 1 -- 172.21.15.12:0/3816775368 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8800bd70 con 0x7f3a900b2070 2024-03-29T10:12:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.640+0000 7f3a96200700 1 -- 172.21.15.12:0/3816775368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900b2070 msgr2=0x7f3a900b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.640+0000 7f3a96200700 1 --2- 172.21.15.12:0/3816775368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900b2070 0x7f3a900b2500 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3a900b1e30 tx=0x7f3a8800b330 comp rx=0 tx=0).stop 2024-03-29T10:12:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.640+0000 7f3a96200700 1 -- 172.21.15.12:0/3816775368 shutdown_connections 2024-03-29T10:12:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.640+0000 7f3a96200700 1 --2- 172.21.15.12:0/3816775368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900b2070 0x7f3a900b2500 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.640+0000 7f3a96200700 1 --2- 172.21.15.12:0/3816775368 >> v2:172.21.15.159:3300/0 conn(0x7f3a900cd020 0x7f3a900cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.640+0000 7f3a96200700 1 -- 172.21.15.12:0/3816775368 >> 172.21.15.12:0/3816775368 conn(0x7f3a900b0d70 msgr2=0x7f3a900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.640+0000 7f3a96200700 1 -- 172.21.15.12:0/3816775368 shutdown_connections 2024-03-29T10:12:17.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.640+0000 7f3a96200700 1 -- 172.21.15.12:0/3816775368 wait complete. 2024-03-29T10:12:17.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.642+0000 7f3a96200700 1 Processor -- start 2024-03-29T10:12:17.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.642+0000 7f3a96200700 1 -- start start 2024-03-29T10:12:17.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.643+0000 7f3a96200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a900b2070 0x7f3a90159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.643+0000 7f3a96200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a900cbb30 0x7f3a9014c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.643+0000 7f3a96200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900cd020 0x7f3a900cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.643+0000 7f3a96200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a90153a60 con 0x7f3a900cd020 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.643+0000 7f3a96200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9015a930 con 0x7f3a900b2070 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.643+0000 7f3a96200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9014cb60 con 0x7f3a900cbb30 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.643+0000 7f3a8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900cd020 0x7f3a900cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.643+0000 7f3a8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900cd020 0x7f3a900cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38364/0 (socket says 172.21.15.12:38364) 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a8fe00700 1 -- 172.21.15.12:0/3567582856 learned_addr learned my addr 172.21.15.12:0/3567582856 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a8f400700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a900b2070 0x7f3a90159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a8ea00700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a900cbb30 0x7f3a9014c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a8fe00700 1 -- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a900cbb30 msgr2=0x7f3a9014c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a8fe00700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a900cbb30 0x7f3a9014c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a8fe00700 1 -- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a900b2070 msgr2=0x7f3a90159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:17.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a8fe00700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a900b2070 0x7f3a90159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:17.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a8fe00700 1 -- 172.21.15.12:0/3567582856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a88009b30 con 0x7f3a900cd020 2024-03-29T10:12:17.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a8fe00700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900cd020 0x7f3a900cacd0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3a9006c130 tx=0x7f3a88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:17.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a87400700 1 -- 172.21.15.12:0/3567582856 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88004840 con 0x7f3a900cd020 2024-03-29T10:12:17.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a87400700 1 -- 172.21.15.12:0/3567582856 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a880099b0 con 0x7f3a900cd020 2024-03-29T10:12:17.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a96200700 1 -- 172.21.15.12:0/3567582856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a900cb210 con 0x7f3a900cd020 2024-03-29T10:12:17.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.644+0000 7f3a87400700 1 -- 172.21.15.12:0/3567582856 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8801c680 con 0x7f3a900cd020 2024-03-29T10:12:17.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.645+0000 7f3a96200700 1 -- 172.21.15.12:0/3567582856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9015a000 con 0x7f3a900cd020 2024-03-29T10:12:17.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.646+0000 7f3a96200700 1 -- 172.21.15.12:0/3567582856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9015aaa0 con 0x7f3a900cd020 2024-03-29T10:12:17.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.647+0000 7f3a87400700 1 -- 172.21.15.12:0/3567582856 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 52) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f3a8801c7e0 con 0x7f3a900cd020 2024-03-29T10:12:17.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.648+0000 7f3a87400700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a7006af50 0x7f3a7006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:17.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.648+0000 7f3a87400700 1 -- 172.21.15.12:0/3567582856 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f3a8808b630 con 0x7f3a900cd020 2024-03-29T10:12:17.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.648+0000 7f3a8f400700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a7006af50 0x7f3a7006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:17.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.649+0000 7f3a8f400700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a7006af50 0x7f3a7006d410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3a80007fd0 tx=0x7f3a80009040 comp rx=0 tx=0).ready entity=mgr.7294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:17.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.652+0000 7f3a87400700 1 -- 172.21.15.12:0/3567582856 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3a88056210 con 0x7f3a900cd020 2024-03-29T10:12:17.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.895+0000 7f3a96200700 1 -- 172.21.15.12:0/3567582856 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a9015bdd0 con 0x7f3a7006af50 2024-03-29T10:12:17.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.904+0000 7f3a87400700 1 -- 172.21.15.12:0/3567582856 <== mgr.7294 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+303 (secure 0 0 0) 0x7f3a80001560 con 0x7f3a7006af50 2024-03-29T10:12:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:17.903 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T10:11:16", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-03-29T10:12:17.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 -- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a7006af50 msgr2=0x7f3a7006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:17.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a7006af50 0x7f3a7006d410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3a80007fd0 tx=0x7f3a80009040 comp rx=0 tx=0).stop 2024-03-29T10:12:17.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 -- 172.21.15.12:0/3567582856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900cd020 msgr2=0x7f3a900cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:17.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900cd020 0x7f3a900cacd0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3a9006c130 tx=0x7f3a88004ad0 comp rx=0 tx=0).stop 2024-03-29T10:12:17.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 -- 172.21.15.12:0/3567582856 shutdown_connections 2024-03-29T10:12:17.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a7006af50 0x7f3a7006d410 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:17.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a900cd020 0x7f3a900cacd0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:17.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a900b2070 0x7f3a90159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:17.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 --2- 172.21.15.12:0/3567582856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a900cbb30 0x7f3a9014c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 -- 172.21.15.12:0/3567582856 >> 172.21.15.12:0/3567582856 conn(0x7f3a900b0d70 msgr2=0x7f3a900afdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 -- 172.21.15.12:0/3567582856 shutdown_connections 2024-03-29T10:12:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:17.906+0000 7f3a96200700 1 -- 172.21.15.12:0/3567582856 wait complete. 2024-03-29T10:12:17.924 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T10:11:16", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-03-29T10:12:17.924 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2024-03-29T10:12:18.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.127+0000 7eff59a00700 1 Processor -- start 2024-03-29T10:12:18.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.127+0000 7eff59a00700 1 -- start start 2024-03-29T10:12:18.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.128+0000 7eff59a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff54160e50 0x7eff540b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.128+0000 7eff59a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff540d3f20 0x7eff540b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff59a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7eff540b3f80 0x7eff540b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff59a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff54161230 con 0x7eff54160e50 2024-03-29T10:12:18.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff59a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff540d2a80 con 0x7eff540b3f80 2024-03-29T10:12:18.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff59a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff540d43b0 con 0x7eff540d3f20 2024-03-29T10:12:18.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff52a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff54160e50 0x7eff540b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff52a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff54160e50 0x7eff540b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38380/0 (socket says 172.21.15.12:38380) 2024-03-29T10:12:18.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff52a00700 1 -- 172.21.15.12:0/346440874 learned_addr learned my addr 172.21.15.12:0/346440874 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:18.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff53400700 1 --2- 172.21.15.12:0/346440874 >> v2:172.21.15.159:3300/0 conn(0x7eff540b3f80 0x7eff540b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff52000700 1 --2- 172.21.15.12:0/346440874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff540d3f20 0x7eff540b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff52a00700 1 -- 172.21.15.12:0/346440874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff540d3f20 msgr2=0x7eff540b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:18.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff52a00700 1 --2- 172.21.15.12:0/346440874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff540d3f20 0x7eff540b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff52a00700 1 -- 172.21.15.12:0/346440874 >> v2:172.21.15.159:3300/0 conn(0x7eff540b3f80 msgr2=0x7eff540b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:18.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff52a00700 1 --2- 172.21.15.12:0/346440874 >> v2:172.21.15.159:3300/0 conn(0x7eff540b3f80 0x7eff540b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.129+0000 7eff52a00700 1 -- 172.21.15.12:0/346440874 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff540d21e0 con 0x7eff54160e50 2024-03-29T10:12:18.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.130+0000 7eff52a00700 1 --2- 172.21.15.12:0/346440874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 0x7eff540b4e90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7eff540b95b0 tx=0x7eff44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ea8e7d1eb9db6f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:18.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.130+0000 7eff51600700 1 -- 172.21.15.12:0/346440874 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff44011830 con 0x7eff54160e50 2024-03-29T10:12:18.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.130+0000 7eff51600700 1 -- 172.21.15.12:0/346440874 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff44011990 con 0x7eff54160e50 2024-03-29T10:12:18.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.130+0000 7eff51600700 1 -- 172.21.15.12:0/346440874 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff44011d20 con 0x7eff54160e50 2024-03-29T10:12:18.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.131+0000 7eff59a00700 1 -- 172.21.15.12:0/346440874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 msgr2=0x7eff540b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:18.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.131+0000 7eff59a00700 1 --2- 172.21.15.12:0/346440874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 0x7eff540b4e90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7eff540b95b0 tx=0x7eff44009bd0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.131+0000 7eff59a00700 1 -- 172.21.15.12:0/346440874 shutdown_connections 2024-03-29T10:12:18.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.131+0000 7eff59a00700 1 --2- 172.21.15.12:0/346440874 >> v2:172.21.15.159:3300/0 conn(0x7eff540b3f80 0x7eff540b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.131+0000 7eff59a00700 1 --2- 172.21.15.12:0/346440874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 0x7eff540b4e90 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.131+0000 7eff59a00700 1 --2- 172.21.15.12:0/346440874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff540d3f20 0x7eff540b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.131+0000 7eff59a00700 1 -- 172.21.15.12:0/346440874 >> 172.21.15.12:0/346440874 conn(0x7eff5408aff0 msgr2=0x7eff540d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:18.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.131+0000 7eff59a00700 1 -- 172.21.15.12:0/346440874 shutdown_connections 2024-03-29T10:12:18.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.131+0000 7eff59a00700 1 -- 172.21.15.12:0/346440874 wait complete. 2024-03-29T10:12:18.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.133+0000 7eff59a00700 1 Processor -- start 2024-03-29T10:12:18.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.133+0000 7eff59a00700 1 -- start start 2024-03-29T10:12:18.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.134+0000 7eff59a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff540b3f80 0x7eff540cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.134+0000 7eff59a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 0x7eff540b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.134+0000 7eff59a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff540b6150 0x7eff540b8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.134+0000 7eff59a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff540b7cd0 con 0x7eff54160e50 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.134+0000 7eff59a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff540b66f0 con 0x7eff540b6150 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.134+0000 7eff59a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff540b69d0 con 0x7eff540b3f80 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff52000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 0x7eff540b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff52000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 0x7eff540b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38396/0 (socket says 172.21.15.12:38396) 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff52000700 1 -- 172.21.15.12:0/3670239109 learned_addr learned my addr 172.21.15.12:0/3670239109 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff52a00700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff540b3f80 0x7eff540cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff53400700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff540b6150 0x7eff540b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff52000700 1 -- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff540b3f80 msgr2=0x7eff540cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff52000700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff540b3f80 0x7eff540cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff52000700 1 -- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff540b6150 msgr2=0x7eff540b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff52000700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff540b6150 0x7eff540b8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff52000700 1 -- 172.21.15.12:0/3670239109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff44009550 con 0x7eff54160e50 2024-03-29T10:12:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.135+0000 7eff52000700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 0x7eff540b7790 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7eff4000e9b0 tx=0x7eff4000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.136+0000 7eff4aa00700 1 -- 172.21.15.12:0/3670239109 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40009710 con 0x7eff54160e50 2024-03-29T10:12:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.136+0000 7eff4aa00700 1 -- 172.21.15.12:0/3670239109 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4000ee50 con 0x7eff54160e50 2024-03-29T10:12:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.136+0000 7eff4aa00700 1 -- 172.21.15.12:0/3670239109 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4000ca50 con 0x7eff54160e50 2024-03-29T10:12:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.136+0000 7eff59a00700 1 -- 172.21.15.12:0/3670239109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff540b6cb0 con 0x7eff54160e50 2024-03-29T10:12:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.136+0000 7eff59a00700 1 -- 172.21.15.12:0/3670239109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff540b7270 con 0x7eff54160e50 2024-03-29T10:12:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.137+0000 7eff59a00700 1 -- 172.21.15.12:0/3670239109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff54067720 con 0x7eff54160e50 2024-03-29T10:12:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.138+0000 7eff4aa00700 1 -- 172.21.15.12:0/3670239109 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 52) v1 ==== 89370+0+0 (secure 0 0 0) 0x7eff4000cbb0 con 0x7eff54160e50 2024-03-29T10:12:18.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.139+0000 7eff4aa00700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff3406af20 0x7eff3406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.139+0000 7eff4aa00700 1 -- 172.21.15.12:0/3670239109 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7eff40089980 con 0x7eff54160e50 2024-03-29T10:12:18.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.139+0000 7eff52a00700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff3406af20 0x7eff3406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.140+0000 7eff52a00700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff3406af20 0x7eff3406d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7eff44014040 tx=0x7eff44008030 comp rx=0 tx=0).ready entity=mgr.7294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.143+0000 7eff4aa00700 1 -- 172.21.15.12:0/3670239109 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff400545e0 con 0x7eff54160e50 2024-03-29T10:12:18.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.387+0000 7eff59a00700 1 -- 172.21.15.12:0/3670239109 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7eff540b0100 con 0x7eff3406af20 2024-03-29T10:12:18.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.469+0000 7eff4aa00700 1 -- 172.21.15.12:0/3670239109 <== mgr.7294 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7eff44011b00 con 0x7eff3406af20 2024-03-29T10:12:18.470 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /bad-path 2024-03-29T10:12:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 -- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff3406af20 msgr2=0x7eff3406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff3406af20 0x7eff3406d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7eff44014040 tx=0x7eff44008030 comp rx=0 tx=0).stop 2024-03-29T10:12:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 -- 172.21.15.12:0/3670239109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 msgr2=0x7eff540b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 0x7eff540b7790 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7eff4000e9b0 tx=0x7eff4000ecc0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 -- 172.21.15.12:0/3670239109 shutdown_connections 2024-03-29T10:12:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff3406af20 0x7eff3406d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff540b6150 0x7eff540b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff540b3f80 0x7eff540cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 --2- 172.21.15.12:0/3670239109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff54160e50 0x7eff540b7790 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 -- 172.21.15.12:0/3670239109 >> 172.21.15.12:0/3670239109 conn(0x7eff5408aff0 msgr2=0x7eff5406c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 -- 172.21.15.12:0/3670239109 shutdown_connections 2024-03-29T10:12:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.472+0000 7eff59a00700 1 -- 172.21.15.12:0/3670239109 wait complete. 2024-03-29T10:12:18.489 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2024-03-29T10:12:18.490 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T10:12:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.692+0000 7ff6ddc00700 1 Processor -- start 2024-03-29T10:12:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.692+0000 7ff6ddc00700 1 -- start start 2024-03-29T10:12:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.693+0000 7ff6ddc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 0x7ff6d80cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.693+0000 7ff6ddc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff6d80cd020 0x7ff6d80cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.693+0000 7ff6ddc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d80cc450 con 0x7ff6d80b2070 2024-03-29T10:12:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.693+0000 7ff6ddc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d80cdae0 con 0x7ff6d80cd020 2024-03-29T10:12:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.693+0000 7ff6ddc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80cddc0 con 0x7ff6d80cbb30 2024-03-29T10:12:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.694+0000 7ff6d6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 0x7ff6d80cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.694+0000 7ff6d6000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff6d80cd020 0x7ff6d80cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.694+0000 7ff6d6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 0x7ff6d80cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:34316/0 (socket says 172.21.15.12:34316) 2024-03-29T10:12:18.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.694+0000 7ff6d6a00700 1 -- 172.21.15.12:0/2341673193 learned_addr learned my addr 172.21.15.12:0/2341673193 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:18.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.694+0000 7ff6d7400700 1 --1- 172.21.15.12:0/2341673193 >> v1:172.21.15.159:6789/0 conn(0x7ff6d80b2070 0x7ff6d806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:35102/0 (socket says 172.21.15.12:35102) 2024-03-29T10:12:18.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.694+0000 7ff6d6000700 1 -- 172.21.15.12:0/2341673193 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 msgr2=0x7ff6d80cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:18.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.694+0000 7ff6d6000700 1 --2- 172.21.15.12:0/2341673193 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 0x7ff6d80cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.694+0000 7ff6d6000700 1 -- 172.21.15.12:0/2341673193 >> v1:172.21.15.159:6789/0 conn(0x7ff6d80b2070 legacy=0x7ff6d806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:18.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.694+0000 7ff6d6000700 1 -- 172.21.15.12:0/2341673193 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d80ce0a0 con 0x7ff6d80cd020 2024-03-29T10:12:18.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.694+0000 7ff6d6000700 1 --2- 172.21.15.12:0/2341673193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6d80cd020 0x7ff6d80cd4a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009840 tx=0x7ff6c4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e85c037a40642fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:18.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6d5600700 1 -- 172.21.15.12:0/2341673193 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4011830 con 0x7ff6d80cd020 2024-03-29T10:12:18.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6d5600700 1 -- 172.21.15.12:0/2341673193 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c4011990 con 0x7ff6d80cd020 2024-03-29T10:12:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6d5600700 1 -- 172.21.15.12:0/2341673193 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4011d20 con 0x7ff6d80cd020 2024-03-29T10:12:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6ddc00700 1 -- 172.21.15.12:0/2341673193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6d80cd020 msgr2=0x7ff6d80cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6ddc00700 1 --2- 172.21.15.12:0/2341673193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6d80cd020 0x7ff6d80cd4a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009840 tx=0x7ff6c4009b50 comp rx=0 tx=0).stop 2024-03-29T10:12:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6ddc00700 1 -- 172.21.15.12:0/2341673193 shutdown_connections 2024-03-29T10:12:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6ddc00700 1 --2- 172.21.15.12:0/2341673193 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 0x7ff6d80cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6ddc00700 1 --2- 172.21.15.12:0/2341673193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6d80cd020 0x7ff6d80cd4a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6ddc00700 1 -- 172.21.15.12:0/2341673193 >> 172.21.15.12:0/2341673193 conn(0x7ff6d80b0d70 msgr2=0x7ff6d80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6ddc00700 1 -- 172.21.15.12:0/2341673193 shutdown_connections 2024-03-29T10:12:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.695+0000 7ff6ddc00700 1 -- 172.21.15.12:0/2341673193 wait complete. 2024-03-29T10:12:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.697+0000 7ff6ddc00700 1 Processor -- start 2024-03-29T10:12:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.697+0000 7ff6ddc00700 1 -- start start 2024-03-29T10:12:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.698+0000 7ff6ddc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6d80b2070 0x7ff6d8146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.698+0000 7ff6ddc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 0x7ff6d80ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.698+0000 7ff6ddc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6d80cd020 0x7ff6d80c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.698+0000 7ff6ddc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d815a7a0 con 0x7ff6d80b2070 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.698+0000 7ff6ddc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8146a70 con 0x7ff6d80cd020 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.698+0000 7ff6ddc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8159b90 con 0x7ff6d80cbb30 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.698+0000 7ff6d6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6d80b2070 0x7ff6d8146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.698+0000 7ff6d6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6d80b2070 0x7ff6d8146530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38410/0 (socket says 172.21.15.12:38410) 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.698+0000 7ff6d6a00700 1 -- 172.21.15.12:0/296690747 learned_addr learned my addr 172.21.15.12:0/296690747 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6d6000700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 0x7ff6d80ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6d7400700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6d80cd020 0x7ff6d80c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6d6a00700 1 -- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 msgr2=0x7ff6d80ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6d6a00700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 0x7ff6d80ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6d6a00700 1 -- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6d80cd020 msgr2=0x7ff6d80c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6d6a00700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6d80cd020 0x7ff6d80c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6d6a00700 1 -- 172.21.15.12:0/296690747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c40094a0 con 0x7ff6d80b2070 2024-03-29T10:12:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6d6a00700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6d80b2070 0x7ff6d8146530 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800e8e0 tx=0x7ff6c800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:18.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6cea00700 1 -- 172.21.15.12:0/296690747 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8009600 con 0x7ff6d80b2070 2024-03-29T10:12:18.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6cea00700 1 -- 172.21.15.12:0/296690747 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c800c850 con 0x7ff6d80b2070 2024-03-29T10:12:18.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6ddc00700 1 -- 172.21.15.12:0/296690747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d8159e70 con 0x7ff6d80b2070 2024-03-29T10:12:18.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.699+0000 7ff6cea00700 1 -- 172.21.15.12:0/296690747 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c800eee0 con 0x7ff6d80b2070 2024-03-29T10:12:18.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.700+0000 7ff6ddc00700 1 -- 172.21.15.12:0/296690747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d815a430 con 0x7ff6d80b2070 2024-03-29T10:12:18.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.701+0000 7ff6ddc00700 1 -- 172.21.15.12:0/296690747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d8067720 con 0x7ff6d80b2070 2024-03-29T10:12:18.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.702+0000 7ff6cea00700 1 -- 172.21.15.12:0/296690747 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 52) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff6c800cc00 con 0x7ff6d80b2070 2024-03-29T10:12:18.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.702+0000 7ff6cea00700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6b806af20 0x7ff6b806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:18.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.702+0000 7ff6cea00700 1 -- 172.21.15.12:0/296690747 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7ff6c8089a00 con 0x7ff6d80b2070 2024-03-29T10:12:18.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.703+0000 7ff6d6000700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6b806af20 0x7ff6b806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:18.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.704+0000 7ff6d6000700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6b806af20 0x7ff6b806d3e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4000c00 tx=0x7ff6c401c040 comp rx=0 tx=0).ready entity=mgr.7294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.707+0000 7ff6cea00700 1 -- 172.21.15.12:0/296690747 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6c80545e0 con 0x7ff6d80b2070 2024-03-29T10:12:18.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:18.982+0000 7ff6ddc00700 1 -- 172.21.15.12:0/296690747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ff6d815bda0 con 0x7ff6d80b2070 2024-03-29T10:12:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.201+0000 7ff6cea00700 1 -- 172.21.15.12:0/296690747 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7ff6c80549b0 con 0x7ff6d80b2070 2024-03-29T10:12:19.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.202+0000 7ff6cea00700 1 -- 172.21.15.12:0/296690747 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v53) v1 ==== 93+0+0 (secure 0 0 0) 0x7ff6c8074d50 con 0x7ff6d80b2070 2024-03-29T10:12:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 -- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6b806af20 msgr2=0x7ff6b806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6b806af20 0x7ff6b806d3e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4000c00 tx=0x7ff6c401c040 comp rx=0 tx=0).stop 2024-03-29T10:12:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 -- 172.21.15.12:0/296690747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6d80b2070 msgr2=0x7ff6d8146530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6d80b2070 0x7ff6d8146530 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800e8e0 tx=0x7ff6c800eca0 comp rx=0 tx=0).stop 2024-03-29T10:12:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 -- 172.21.15.12:0/296690747 shutdown_connections 2024-03-29T10:12:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff6b806af20 0x7ff6b806d3e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6d80cd020 0x7ff6d80c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6d80b2070 0x7ff6d8146530 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 --2- 172.21.15.12:0/296690747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6d80cbb30 0x7ff6d80ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 -- 172.21.15.12:0/296690747 >> 172.21.15.12:0/296690747 conn(0x7ff6d80b0d70 msgr2=0x7ff6d806e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 -- 172.21.15.12:0/296690747 shutdown_connections 2024-03-29T10:12:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.205+0000 7ff6ddc00700 1 -- 172.21.15.12:0/296690747 wait complete. 2024-03-29T10:12:19.224 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:12:19.224 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:12:19.312 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:12:19.400 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:12:19.400 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:12:19.400 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:12:19.427 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:19.427+0000 7ff3990d7240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:12:19.434 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:19.434+0000 7f3f29e3c240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:12:19.445 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:12:19.445 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:12:19.530 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:12:19.620 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:19.620+0000 7f3f29e3c240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:12:19.633 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:19.633+0000 7ff3990d7240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:12:19.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.800+0000 7ff975400700 1 Processor -- start 2024-03-29T10:12:19.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.801+0000 7ff975400700 1 -- start start 2024-03-29T10:12:19.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.802+0000 7ff975400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701619d0 0x7ff9701685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:19.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.802+0000 7ff975400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9700d30d0 con 0x7ff97015d640 2024-03-29T10:12:19.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.802+0000 7ff975400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9700b6840 con 0x7ff9700ceec0 2024-03-29T10:12:19.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.802+0000 7ff975400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700d2b30 con 0x7ff9701619d0 2024-03-29T10:12:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.803+0000 7ff96e800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff97015d640 0x7ff9700d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54328/0 (socket says 172.21.15.12:54328) 2024-03-29T10:12:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.803+0000 7ff96e800700 1 -- 172.21.15.12:0/2383434545 learned_addr learned my addr 172.21.15.12:0/2383434545 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.803+0000 7ff96f200700 1 --2- 172.21.15.12:0/2383434545 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701619d0 0x7ff9701685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.803+0000 7ff96d400700 1 -- 172.21.15.12:0/2383434545 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424174250 0 0) 0x7ff9700d30d0 con 0x7ff97015d640 2024-03-29T10:12:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.803+0000 7ff96d400700 1 -- 172.21.15.12:0/2383434545 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9540035b0 con 0x7ff97015d640 2024-03-29T10:12:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.803+0000 7ff96f200700 1 -- 172.21.15.12:0/2383434545 >> v1:172.21.15.159:6789/0 conn(0x7ff9700ceec0 legacy=0x7ff9701612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.803+0000 7ff96f200700 1 -- 172.21.15.12:0/2383434545 >> v1:172.21.15.12:6789/0 conn(0x7ff97015d640 legacy=0x7ff9700d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.803+0000 7ff96d400700 1 -- 172.21.15.12:0/2383434545 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691557303 0 0) 0x7ff9700b6840 con 0x7ff9700ceec0 2024-03-29T10:12:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.803+0000 7ff96f200700 1 -- 172.21.15.12:0/2383434545 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9700d2df0 con 0x7ff9701619d0 2024-03-29T10:12:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.804+0000 7ff96f200700 1 --2- 172.21.15.12:0/2383434545 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701619d0 0x7ff9701685f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff968009dc0 tx=0x7ff96800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78a04d6a65f5a443 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.804+0000 7ff96d400700 1 -- 172.21.15.12:0/2383434545 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96800e070 con 0x7ff9701619d0 2024-03-29T10:12:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.804+0000 7ff96d400700 1 -- 172.21.15.12:0/2383434545 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96800bb20 con 0x7ff9701619d0 2024-03-29T10:12:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.804+0000 7ff96d400700 1 -- 172.21.15.12:0/2383434545 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96800bed0 con 0x7ff9701619d0 2024-03-29T10:12:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.805+0000 7ff975400700 1 -- 172.21.15.12:0/2383434545 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701619d0 msgr2=0x7ff9701685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.805+0000 7ff975400700 1 --2- 172.21.15.12:0/2383434545 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701619d0 0x7ff9701685f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff968009dc0 tx=0x7ff96800b330 comp rx=0 tx=0).stop 2024-03-29T10:12:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.805+0000 7ff975400700 1 -- 172.21.15.12:0/2383434545 shutdown_connections 2024-03-29T10:12:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.805+0000 7ff975400700 1 --2- 172.21.15.12:0/2383434545 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701619d0 0x7ff9701685f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.805+0000 7ff975400700 1 -- 172.21.15.12:0/2383434545 >> 172.21.15.12:0/2383434545 conn(0x7ff97008aff0 msgr2=0x7ff9700d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.805+0000 7ff975400700 1 -- 172.21.15.12:0/2383434545 shutdown_connections 2024-03-29T10:12:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.805+0000 7ff975400700 1 -- 172.21.15.12:0/2383434545 wait complete. 2024-03-29T10:12:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.807+0000 7ff975400700 1 Processor -- start 2024-03-29T10:12:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.807+0000 7ff975400700 1 -- start start 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.808+0000 7ff975400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9700ceec0 0x7ff9700a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff975400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff97015d640 0x7ff9700c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff975400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9701619d0 0x7ff97015a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff975400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700c9fa0 con 0x7ff9701619d0 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff975400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700ca620 con 0x7ff9700ceec0 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff975400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700cd810 con 0x7ff97015d640 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff96e800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9700ceec0 0x7ff9700a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff96f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9701619d0 0x7ff97015a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff96e800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9700ceec0 0x7ff9700a0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44076/0 (socket says 172.21.15.12:44076) 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff96e800700 1 -- 172.21.15.12:0/1705717875 learned_addr learned my addr 172.21.15.12:0/1705717875 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff96de00700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff97015d640 0x7ff9700c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff96e800700 1 -- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff97015d640 msgr2=0x7ff9700c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff96e800700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff97015d640 0x7ff9700c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff96e800700 1 -- 172.21.15.12:0/1705717875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9701619d0 msgr2=0x7ff97015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff96e800700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9701619d0 0x7ff97015a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.809+0000 7ff96e800700 1 -- 172.21.15.12:0/1705717875 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff968009a70 con 0x7ff9700ceec0 2024-03-29T10:12:19.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.810+0000 7ff96e800700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9700ceec0 0x7ff9700a0590 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff96000c4d0 tx=0x7ff96000c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:19.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.810+0000 7ff966a00700 1 -- 172.21.15.12:0/1705717875 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960005d70 con 0x7ff9700ceec0 2024-03-29T10:12:19.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.810+0000 7ff966a00700 1 -- 172.21.15.12:0/1705717875 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9600070d0 con 0x7ff9700ceec0 2024-03-29T10:12:19.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.810+0000 7ff966a00700 1 -- 172.21.15.12:0/1705717875 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960007440 con 0x7ff9700ceec0 2024-03-29T10:12:19.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.810+0000 7ff975400700 1 -- 172.21.15.12:0/1705717875 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9700ca850 con 0x7ff9700ceec0 2024-03-29T10:12:19.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.810+0000 7ff975400700 1 -- 172.21.15.12:0/1705717875 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9700cdd70 con 0x7ff9700ceec0 2024-03-29T10:12:19.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.811+0000 7ff975400700 1 -- 172.21.15.12:0/1705717875 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff970067720 con 0x7ff9700ceec0 2024-03-29T10:12:19.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.812+0000 7ff966a00700 1 -- 172.21.15.12:0/1705717875 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7ff9600075a0 con 0x7ff9700ceec0 2024-03-29T10:12:19.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.813+0000 7ff966a00700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff95006aed0 0x7ff95006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:19.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.813+0000 7ff966a00700 1 -- 172.21.15.12:0/1705717875 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7ff960088cb0 con 0x7ff9700ceec0 2024-03-29T10:12:19.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.813+0000 7ff96de00700 1 -- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff95006aed0 msgr2=0x7ff95006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:19.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.813+0000 7ff96de00700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff95006aed0 0x7ff95006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:19.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:19.818+0000 7ff966a00700 1 -- 172.21.15.12:0/1705717875 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff9600539c0 con 0x7ff9700ceec0 2024-03-29T10:12:19.941 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:19.941+0000 7f3f29e3c240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:12:19.974 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:19.975+0000 7ff3990d7240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:12:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.014+0000 7ff96de00700 1 -- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff95006aed0 msgr2=0x7ff95006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.014+0000 7ff96de00700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff95006aed0 0x7ff95006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.034+0000 7ff975400700 1 -- 172.21.15.12:0/1705717875 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9700ceb00 con 0x7ff9700ceec0 2024-03-29T10:12:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.036+0000 7ff966a00700 1 -- 172.21.15.12:0/1705717875 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16650 (secure 0 0 0) 0x7ff960017020 con 0x7ff9700ceec0 2024-03-29T10:12:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:20.036 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":111,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:12:17.179326+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:57.064406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:57.607254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3e","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6833/43444":"2024-03-30T10:11:15.808228+0000","172.21.15.159:6832/43444":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/430021803":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2229015445":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2018718607":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/663205968":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/1058856906":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2814107463":"2024-03-30T10:11:15.808228+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:12:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 -- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff95006aed0 msgr2=0x7ff95006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff95006aed0 0x7ff95006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 -- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9700ceec0 msgr2=0x7ff9700a0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9700ceec0 0x7ff9700a0590 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff96000c4d0 tx=0x7ff96000c7e0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 -- 172.21.15.12:0/1705717875 shutdown_connections 2024-03-29T10:12:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff95006aed0 0x7ff95006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9701619d0 0x7ff97015a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9700ceec0 0x7ff9700a0590 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 --2- 172.21.15.12:0/1705717875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff97015d640 0x7ff9700c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 -- 172.21.15.12:0/1705717875 >> 172.21.15.12:0/1705717875 conn(0x7ff97008aff0 msgr2=0x7ff97015c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 -- 172.21.15.12:0/1705717875 shutdown_connections 2024-03-29T10:12:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.038+0000 7ff975400700 1 -- 172.21.15.12:0/1705717875 wait complete. 2024-03-29T10:12:20.057 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:10:57.064406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:10:57.607254+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:12:20.058 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:12:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.260+0000 7f1a37200700 1 Processor -- start 2024-03-29T10:12:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.261+0000 7f1a37200700 1 -- start start 2024-03-29T10:12:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.262+0000 7f1a37200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a30157f90 0x7f1a300b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.262+0000 7f1a37200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a300b3b30 0x7f1a30161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.262+0000 7f1a37200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a300b3070 con 0x7f1a301613e0 2024-03-29T10:12:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.262+0000 7f1a37200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a30158370 con 0x7f1a30157f90 2024-03-29T10:12:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.262+0000 7f1a37200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300b2710 con 0x7f1a300b3b30 2024-03-29T10:12:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.262+0000 7f1a2fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a30157f90 0x7f1a300b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.262+0000 7f1a2fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a30157f90 0x7f1a300b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40204/0 (socket says 172.21.15.12:40204) 2024-03-29T10:12:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.262+0000 7f1a2fe00700 1 -- 172.21.15.12:0/1682662908 learned_addr learned my addr 172.21.15.12:0/1682662908 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.262+0000 7f1a35200700 1 --2- 172.21.15.12:0/1682662908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a300b3b30 0x7f1a30161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.263+0000 7f1a2fe00700 1 -- 172.21.15.12:0/1682662908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a300b3b30 msgr2=0x7f1a30161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.263+0000 7f1a2fe00700 1 --2- 172.21.15.12:0/1682662908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a300b3b30 0x7f1a30161890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.263+0000 7f1a2fe00700 1 -- 172.21.15.12:0/1682662908 >> v1:172.21.15.159:6789/0 conn(0x7f1a301613e0 legacy=0x7f1a300b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.263+0000 7f1a2fe00700 1 -- 172.21.15.12:0/1682662908 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a300b2070 con 0x7f1a30157f90 2024-03-29T10:12:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.263+0000 7f1a2fe00700 1 --2- 172.21.15.12:0/1682662908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a30157f90 0x7f1a300b2b30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009920 tx=0x7f1a20009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19c5841d85e0c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.263+0000 7f1a2ea00700 1 -- 172.21.15.12:0/1682662908 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20011830 con 0x7f1a30157f90 2024-03-29T10:12:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.263+0000 7f1a2ea00700 1 -- 172.21.15.12:0/1682662908 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a20011990 con 0x7f1a30157f90 2024-03-29T10:12:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.263+0000 7f1a2ea00700 1 -- 172.21.15.12:0/1682662908 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20011d20 con 0x7f1a30157f90 2024-03-29T10:12:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.264+0000 7f1a37200700 1 -- 172.21.15.12:0/1682662908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a30157f90 msgr2=0x7f1a300b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.264+0000 7f1a37200700 1 --2- 172.21.15.12:0/1682662908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a30157f90 0x7f1a300b2b30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009920 tx=0x7f1a20009c30 comp rx=0 tx=0).stop 2024-03-29T10:12:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.264+0000 7f1a37200700 1 -- 172.21.15.12:0/1682662908 shutdown_connections 2024-03-29T10:12:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.264+0000 7f1a37200700 1 --2- 172.21.15.12:0/1682662908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a300b3b30 0x7f1a30161890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.264+0000 7f1a37200700 1 --2- 172.21.15.12:0/1682662908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a30157f90 0x7f1a300b2b30 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.264+0000 7f1a37200700 1 -- 172.21.15.12:0/1682662908 >> 172.21.15.12:0/1682662908 conn(0x7f1a300b0d70 msgr2=0x7f1a300b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.264+0000 7f1a37200700 1 -- 172.21.15.12:0/1682662908 shutdown_connections 2024-03-29T10:12:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.264+0000 7f1a37200700 1 -- 172.21.15.12:0/1682662908 wait complete. 2024-03-29T10:12:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.266+0000 7f1a37200700 1 Processor -- start 2024-03-29T10:12:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.267+0000 7f1a37200700 1 -- start start 2024-03-29T10:12:20.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a37200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a300b3b30 0x7f1a301458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a37200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a30157f90 0x7f1a3015fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a37200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a301613e0 0x7f1a301602a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a37200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a30155dd0 con 0x7f1a300b3b30 2024-03-29T10:12:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a37200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3015f570 con 0x7f1a301613e0 2024-03-29T10:12:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a37200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a301551c0 con 0x7f1a30157f90 2024-03-29T10:12:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a2fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a300b3b30 0x7f1a301458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a2fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a300b3b30 0x7f1a301458b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40206/0 (socket says 172.21.15.12:40206) 2024-03-29T10:12:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a2fe00700 1 -- 172.21.15.12:0/2649481383 learned_addr learned my addr 172.21.15.12:0/2649481383 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a2f400700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a30157f90 0x7f1a3015fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.268+0000 7f1a35200700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a301613e0 0x7f1a301602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.269+0000 7f1a2fe00700 1 -- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a30157f90 msgr2=0x7f1a3015fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.269+0000 7f1a2fe00700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a30157f90 0x7f1a3015fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.269+0000 7f1a2fe00700 1 -- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a301613e0 msgr2=0x7f1a301602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.269+0000 7f1a2fe00700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a301613e0 0x7f1a301602a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.269+0000 7f1a2fe00700 1 -- 172.21.15.12:0/2649481383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a20009550 con 0x7f1a300b3b30 2024-03-29T10:12:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.269+0000 7f1a2fe00700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a300b3b30 0x7f1a301458b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009870 tx=0x7f1a20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.269+0000 7f1a27e00700 1 -- 172.21.15.12:0/2649481383 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20004300 con 0x7f1a300b3b30 2024-03-29T10:12:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.269+0000 7f1a27e00700 1 -- 172.21.15.12:0/2649481383 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2001f070 con 0x7f1a300b3b30 2024-03-29T10:12:20.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.269+0000 7f1a37200700 1 -- 172.21.15.12:0/2649481383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a301554a0 con 0x7f1a300b3b30 2024-03-29T10:12:20.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.269+0000 7f1a27e00700 1 -- 172.21.15.12:0/2649481383 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a200036a0 con 0x7f1a300b3b30 2024-03-29T10:12:20.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.270+0000 7f1a37200700 1 -- 172.21.15.12:0/2649481383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a30155a60 con 0x7f1a300b3b30 2024-03-29T10:12:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.271+0000 7f1a37200700 1 -- 172.21.15.12:0/2649481383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a30067720 con 0x7f1a300b3b30 2024-03-29T10:12:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.271+0000 7f1a27e00700 1 -- 172.21.15.12:0/2649481383 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f1a20021030 con 0x7f1a300b3b30 2024-03-29T10:12:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.272+0000 7f1a27e00700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1006aed0 0x7f1a1006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.272+0000 7f1a27e00700 1 -- 172.21.15.12:0/2649481383 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f1a2008ab10 con 0x7f1a300b3b30 2024-03-29T10:12:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.272+0000 7f1a2f400700 1 -- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1006aed0 msgr2=0x7f1a1006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.273+0000 7f1a2f400700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1006aed0 0x7f1a1006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:20.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.281+0000 7f1a27e00700 1 -- 172.21.15.12:0/2649481383 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1a20055820 con 0x7f1a300b3b30 2024-03-29T10:12:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.474+0000 7f1a2f400700 1 -- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1006aed0 msgr2=0x7f1a1006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.474+0000 7f1a2f400700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1006aed0 0x7f1a1006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:20.480 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:20.480+0000 7f3f29e3c240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:12:20.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.496+0000 7f1a37200700 1 -- 172.21.15.12:0/2649481383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1a3006ece0 con 0x7f1a300b3b30 2024-03-29T10:12:20.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.497+0000 7f1a27e00700 1 -- 172.21.15.12:0/2649481383 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1a20015020 con 0x7f1a300b3b30 2024-03-29T10:12:20.497 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:12:20.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.499+0000 7f1a37200700 1 -- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1006aed0 msgr2=0x7f1a1006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:20.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.499+0000 7f1a37200700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1006aed0 0x7f1a1006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.499+0000 7f1a37200700 1 -- 172.21.15.12:0/2649481383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a300b3b30 msgr2=0x7f1a301458b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.499+0000 7f1a37200700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a300b3b30 0x7f1a301458b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009870 tx=0x7f1a20004750 comp rx=0 tx=0).stop 2024-03-29T10:12:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.499+0000 7f1a37200700 1 -- 172.21.15.12:0/2649481383 shutdown_connections 2024-03-29T10:12:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.499+0000 7f1a37200700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a1006aed0 0x7f1a1006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.499+0000 7f1a37200700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a301613e0 0x7f1a301602a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.499+0000 7f1a37200700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a300b3b30 0x7f1a301458b0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.499+0000 7f1a37200700 1 --2- 172.21.15.12:0/2649481383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a30157f90 0x7f1a3015fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.499+0000 7f1a37200700 1 -- 172.21.15.12:0/2649481383 >> 172.21.15.12:0/2649481383 conn(0x7f1a300b0d70 msgr2=0x7f1a300af390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.500+0000 7f1a37200700 1 -- 172.21.15.12:0/2649481383 shutdown_connections 2024-03-29T10:12:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.500+0000 7f1a37200700 1 -- 172.21.15.12:0/2649481383 wait complete. 2024-03-29T10:12:20.517 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:12:20.543 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:20.544+0000 7ff3990d7240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:12:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.721+0000 7feccc800700 1 Processor -- start 2024-03-29T10:12:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.721+0000 7feccc800700 1 -- start start 2024-03-29T10:12:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.722+0000 7feccc800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecc415d9d0 0x7fecc40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.722+0000 7feccc800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fecc40d2540 0x7fecc40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.722+0000 7feccc800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc4161420 con 0x7fecc40d16a0 2024-03-29T10:12:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.722+0000 7feccc800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc415edc0 con 0x7fecc415d9d0 2024-03-29T10:12:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.722+0000 7feccc800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc4161700 con 0x7fecc40d2540 2024-03-29T10:12:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.722+0000 7fecc9400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fecc40d16a0 0x7fecc40d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54338/0 (socket says 172.21.15.12:54338) 2024-03-29T10:12:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.723+0000 7fecc9400700 1 -- 172.21.15.12:0/123598514 learned_addr learned my addr 172.21.15.12:0/123598514 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.722+0000 7fecc9e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecc415d9d0 0x7fecc40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.723+0000 7fecca800700 1 --2- 172.21.15.12:0/123598514 >> v2:172.21.15.159:3300/0 conn(0x7fecc40d2540 0x7fecc40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.723+0000 7fecc8a00700 1 -- 172.21.15.12:0/123598514 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683756369 0 0) 0x7fecc4161420 con 0x7fecc40d16a0 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.723+0000 7fecc8a00700 1 -- 172.21.15.12:0/123598514 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecac0035b0 con 0x7fecc40d16a0 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.723+0000 7fecc9e00700 1 -- 172.21.15.12:0/123598514 >> v2:172.21.15.159:3300/0 conn(0x7fecc40d2540 msgr2=0x7fecc40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.723+0000 7fecc9e00700 1 --2- 172.21.15.12:0/123598514 >> v2:172.21.15.159:3300/0 conn(0x7fecc40d2540 0x7fecc40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.723+0000 7fecc9e00700 1 -- 172.21.15.12:0/123598514 >> v1:172.21.15.12:6789/0 conn(0x7fecc40d16a0 legacy=0x7fecc40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.723+0000 7fecc9e00700 1 -- 172.21.15.12:0/123598514 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc41619e0 con 0x7fecc415d9d0 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.723+0000 7fecc9e00700 1 --2- 172.21.15.12:0/123598514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecc415d9d0 0x7fecc40d3c30 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fecc408ca10 tx=0x7fecb8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3437c321a12e5a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.724+0000 7fecc8a00700 1 -- 172.21.15.12:0/123598514 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8011830 con 0x7fecc415d9d0 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.724+0000 7fecc8a00700 1 -- 172.21.15.12:0/123598514 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb8011990 con 0x7fecc415d9d0 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.724+0000 7fecc8a00700 1 -- 172.21.15.12:0/123598514 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8011d20 con 0x7fecc415d9d0 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.724+0000 7feccc800700 1 -- 172.21.15.12:0/123598514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecc415d9d0 msgr2=0x7fecc40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.724+0000 7feccc800700 1 --2- 172.21.15.12:0/123598514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecc415d9d0 0x7fecc40d3c30 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fecc408ca10 tx=0x7fecb8009b20 comp rx=0 tx=0).stop 2024-03-29T10:12:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.724+0000 7feccc800700 1 -- 172.21.15.12:0/123598514 shutdown_connections 2024-03-29T10:12:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.724+0000 7feccc800700 1 --2- 172.21.15.12:0/123598514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecc415d9d0 0x7fecc40d3c30 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.724+0000 7feccc800700 1 --2- 172.21.15.12:0/123598514 >> v2:172.21.15.159:3300/0 conn(0x7fecc40d2540 0x7fecc40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.724+0000 7feccc800700 1 -- 172.21.15.12:0/123598514 >> 172.21.15.12:0/123598514 conn(0x7fecc408aff0 msgr2=0x7fecc40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.724+0000 7feccc800700 1 -- 172.21.15.12:0/123598514 shutdown_connections 2024-03-29T10:12:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.725+0000 7feccc800700 1 -- 172.21.15.12:0/123598514 wait complete. 2024-03-29T10:12:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.726+0000 7feccc800700 1 Processor -- start 2024-03-29T10:12:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.726+0000 7feccc800700 1 -- start start 2024-03-29T10:12:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.727+0000 7feccc800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecc40d16a0 0x7fecc40b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.727+0000 7feccc800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecc40d2540 0x7fecc40b7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.727+0000 7feccc800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecc415d9d0 0x7fecc415c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.727+0000 7feccc800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc40b8440 con 0x7fecc40d2540 2024-03-29T10:12:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.727+0000 7feccc800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc40b6700 con 0x7fecc415d9d0 2024-03-29T10:12:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.727+0000 7feccc800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc40b69e0 con 0x7fecc40d16a0 2024-03-29T10:12:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecc9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecc40d2540 0x7fecc40b7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecc9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecc40d16a0 0x7fecc40b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecca800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecc415d9d0 0x7fecc415c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecc9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecc40d16a0 0x7fecc40b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:59432/0 (socket says 172.21.15.12:59432) 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecc9e00700 1 -- 172.21.15.12:0/891898966 learned_addr learned my addr 172.21.15.12:0/891898966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecca800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecc415d9d0 0x7fecc415c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44094/0 (socket says 172.21.15.12:44094) 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecc9e00700 1 -- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecc415d9d0 msgr2=0x7fecc415c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecc9e00700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecc415d9d0 0x7fecc415c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecc9e00700 1 -- 172.21.15.12:0/891898966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecc40d2540 msgr2=0x7fecc40b7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecc9e00700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecc40d2540 0x7fecc40b7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecc9e00700 1 -- 172.21.15.12:0/891898966 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb80094a0 con 0x7fecc40d16a0 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.728+0000 7fecc9400700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecc40d2540 0x7fecc40b7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.729+0000 7fecc9e00700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecc40d16a0 0x7fecc40b6020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fecb8000c00 tx=0x7fecb8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.729+0000 7fecbe000700 1 -- 172.21.15.12:0/891898966 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8004590 con 0x7fecc40d16a0 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.729+0000 7fecbe000700 1 -- 172.21.15.12:0/891898966 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb801e070 con 0x7fecc40d16a0 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.729+0000 7fecbe000700 1 -- 172.21.15.12:0/891898966 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb80038e0 con 0x7fecc40d16a0 2024-03-29T10:12:20.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.729+0000 7feccc800700 1 -- 172.21.15.12:0/891898966 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc40b6cc0 con 0x7fecc40d16a0 2024-03-29T10:12:20.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.729+0000 7feccc800700 1 -- 172.21.15.12:0/891898966 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc40b7280 con 0x7fecc40d16a0 2024-03-29T10:12:20.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.730+0000 7feccc800700 1 -- 172.21.15.12:0/891898966 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc40d1ca0 con 0x7fecc40d16a0 2024-03-29T10:12:20.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.731+0000 7fecbe000700 1 -- 172.21.15.12:0/891898966 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fecb8003a40 con 0x7fecc40d16a0 2024-03-29T10:12:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.732+0000 7fecbe000700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feca806aed0 0x7feca806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.732+0000 7fecbe000700 1 -- 172.21.15.12:0/891898966 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fecb808aa80 con 0x7fecc40d16a0 2024-03-29T10:12:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.732+0000 7fecc9400700 1 -- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feca806aed0 msgr2=0x7feca806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.732+0000 7fecc9400700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feca806aed0 0x7feca806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.736+0000 7fecbe000700 1 -- 172.21.15.12:0/891898966 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fecb8055770 con 0x7fecc40d16a0 2024-03-29T10:12:20.741 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:20.741+0000 7f3f29e3c240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:12:20.819 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:20.820+0000 7ff3990d7240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:12:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.933+0000 7fecc9400700 1 -- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feca806aed0 msgr2=0x7feca806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.933+0000 7fecc9400700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feca806aed0 0x7feca806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.954+0000 7feccc800700 1 -- 172.21.15.12:0/891898966 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fecc4067720 con 0x7fecc40d16a0 2024-03-29T10:12:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.954+0000 7fecbe000700 1 -- 172.21.15.12:0/891898966 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+11 (secure 0 0 0) 0x7fecb8015020 con 0x7fecc40d16a0 2024-03-29T10:12:20.954 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:12:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 -- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feca806aed0 msgr2=0x7feca806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feca806aed0 0x7feca806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 -- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecc40d16a0 msgr2=0x7fecc40b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecc40d16a0 0x7fecc40b6020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fecb8000c00 tx=0x7fecb8004750 comp rx=0 tx=0).stop 2024-03-29T10:12:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 -- 172.21.15.12:0/891898966 shutdown_connections 2024-03-29T10:12:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feca806aed0 0x7feca806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecc415d9d0 0x7fecc415c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecc40d16a0 0x7fecc40b6020 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 --2- 172.21.15.12:0/891898966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecc40d2540 0x7fecc40b7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 -- 172.21.15.12:0/891898966 >> 172.21.15.12:0/891898966 conn(0x7fecc408aff0 msgr2=0x7fecc40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 -- 172.21.15.12:0/891898966 shutdown_connections 2024-03-29T10:12:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:20.957+0000 7feccc800700 1 -- 172.21.15.12:0/891898966 wait complete. 2024-03-29T10:12:20.976 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:12:21.200 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:21.200+0000 7f3f29e3c240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:12:21.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.223+0000 7f682aa00700 1 Processor -- start 2024-03-29T10:12:21.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.223+0000 7f682aa00700 1 -- start start 2024-03-29T10:12:21.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f682aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415d9d0 0x7f68240d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f682aa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f68240d2540 0x7f68240ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f682aa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6824161420 con 0x7f68240d16a0 2024-03-29T10:12:21.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f682aa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f682415edc0 con 0x7f68240d2540 2024-03-29T10:12:21.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f682aa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6824161700 con 0x7f682415d9d0 2024-03-29T10:12:21.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f6823e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415d9d0 0x7f68240d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:21.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f6828a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f68240d2540 0x7f68240ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:21.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f6828a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f68240d2540 0x7f68240ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44100/0 (socket says 172.21.15.12:44100) 2024-03-29T10:12:21.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f6823e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415d9d0 0x7f68240d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:59448/0 (socket says 172.21.15.12:59448) 2024-03-29T10:12:21.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f6828a00700 1 -- 172.21.15.12:0/2406926168 learned_addr learned my addr 172.21.15.12:0/2406926168 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:21.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.224+0000 7f6823400700 1 --1- 172.21.15.12:0/2406926168 >> v1:172.21.15.159:6789/0 conn(0x7f68240d16a0 0x7f68240d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:33234/0 (socket says 172.21.15.12:33234) 2024-03-29T10:12:21.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.225+0000 7f6823e00700 1 -- 172.21.15.12:0/2406926168 >> v2:172.21.15.159:3300/0 conn(0x7f68240d2540 msgr2=0x7f68240ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:21.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.225+0000 7f6823e00700 1 --2- 172.21.15.12:0/2406926168 >> v2:172.21.15.159:3300/0 conn(0x7f68240d2540 0x7f68240ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.225+0000 7f6823e00700 1 -- 172.21.15.12:0/2406926168 >> v1:172.21.15.159:6789/0 conn(0x7f68240d16a0 legacy=0x7f68240d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:21.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.225+0000 7f6823e00700 1 -- 172.21.15.12:0/2406926168 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68241619e0 con 0x7f682415d9d0 2024-03-29T10:12:21.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.225+0000 7f6823e00700 1 --2- 172.21.15.12:0/2406926168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415d9d0 0x7f68240d3c30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6814009870 tx=0x7f6814009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a01e0dc429bdf816 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:21.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.226+0000 7f6822a00700 1 -- 172.21.15.12:0/2406926168 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6814011830 con 0x7f682415d9d0 2024-03-29T10:12:21.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.226+0000 7f6822a00700 1 -- 172.21.15.12:0/2406926168 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6814011990 con 0x7f682415d9d0 2024-03-29T10:12:21.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.226+0000 7f6822a00700 1 -- 172.21.15.12:0/2406926168 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6814011d20 con 0x7f682415d9d0 2024-03-29T10:12:21.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.227+0000 7f682aa00700 1 -- 172.21.15.12:0/2406926168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415d9d0 msgr2=0x7f68240d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:21.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.227+0000 7f682aa00700 1 --2- 172.21.15.12:0/2406926168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415d9d0 0x7f68240d3c30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6814009870 tx=0x7f6814009b80 comp rx=0 tx=0).stop 2024-03-29T10:12:21.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.227+0000 7f682aa00700 1 -- 172.21.15.12:0/2406926168 shutdown_connections 2024-03-29T10:12:21.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.227+0000 7f682aa00700 1 --2- 172.21.15.12:0/2406926168 >> v2:172.21.15.159:3300/0 conn(0x7f68240d2540 0x7f68240ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.227+0000 7f682aa00700 1 --2- 172.21.15.12:0/2406926168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f682415d9d0 0x7f68240d3c30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.227+0000 7f682aa00700 1 -- 172.21.15.12:0/2406926168 >> 172.21.15.12:0/2406926168 conn(0x7f682408aff0 msgr2=0x7f68240d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:21.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.227+0000 7f682aa00700 1 -- 172.21.15.12:0/2406926168 shutdown_connections 2024-03-29T10:12:21.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.227+0000 7f682aa00700 1 -- 172.21.15.12:0/2406926168 wait complete. 2024-03-29T10:12:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.229+0000 7f682aa00700 1 Processor -- start 2024-03-29T10:12:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.229+0000 7f682aa00700 1 -- start start 2024-03-29T10:12:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f682aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68240d16a0 0x7f682415c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f682aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d2540 0x7f68240b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f682aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f682415d9d0 0x7f68241e9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f682aa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68240b7d90 con 0x7f68240d2540 2024-03-29T10:12:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f682aa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68240cb3e0 con 0x7f682415d9d0 2024-03-29T10:12:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f682aa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68240b74f0 con 0x7f68240d16a0 2024-03-29T10:12:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f6823400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d2540 0x7f68240b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f6823e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68240d16a0 0x7f682415c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f6828a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f682415d9d0 0x7f68241e9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f6823400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d2540 0x7f68240b6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40230/0 (socket says 172.21.15.12:40230) 2024-03-29T10:12:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f6823e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68240d16a0 0x7f682415c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:59462/0 (socket says 172.21.15.12:59462) 2024-03-29T10:12:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f6823400700 1 -- 172.21.15.12:0/2346817736 learned_addr learned my addr 172.21.15.12:0/2346817736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f6823e00700 1 -- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f682415d9d0 msgr2=0x7f68241e9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f6823e00700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f682415d9d0 0x7f68241e9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f6823e00700 1 -- 172.21.15.12:0/2346817736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d2540 msgr2=0x7f68240b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.230+0000 7f6823e00700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d2540 0x7f68240b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.231+0000 7f6823e00700 1 -- 172.21.15.12:0/2346817736 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68140094a0 con 0x7f68240d16a0 2024-03-29T10:12:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.231+0000 7f6823400700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d2540 0x7f68240b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.231+0000 7f6823e00700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68240d16a0 0x7f682415c9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6814016000 tx=0x7f6814004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.231+0000 7f681be00700 1 -- 172.21.15.12:0/2346817736 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68140036a0 con 0x7f68240d16a0 2024-03-29T10:12:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.231+0000 7f681be00700 1 -- 172.21.15.12:0/2346817736 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681401e070 con 0x7f68240d16a0 2024-03-29T10:12:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.231+0000 7f681be00700 1 -- 172.21.15.12:0/2346817736 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6814003a40 con 0x7f68240d16a0 2024-03-29T10:12:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.231+0000 7f682aa00700 1 -- 172.21.15.12:0/2346817736 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68240b77d0 con 0x7f68240d16a0 2024-03-29T10:12:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.231+0000 7f682aa00700 1 -- 172.21.15.12:0/2346817736 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68240cb680 con 0x7f68240d16a0 2024-03-29T10:12:21.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.232+0000 7f682aa00700 1 -- 172.21.15.12:0/2346817736 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68240d1ca0 con 0x7f68240d16a0 2024-03-29T10:12:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.233+0000 7f681be00700 1 -- 172.21.15.12:0/2346817736 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f6814003ba0 con 0x7f68240d16a0 2024-03-29T10:12:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.234+0000 7f681be00700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406aed0 0x7f680406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.234+0000 7f681be00700 1 -- 172.21.15.12:0/2346817736 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f681408a730 con 0x7f68240d16a0 2024-03-29T10:12:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.234+0000 7f6823400700 1 -- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406aed0 msgr2=0x7f680406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.234+0000 7f6823400700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406aed0 0x7f680406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.239+0000 7f681be00700 1 -- 172.21.15.12:0/2346817736 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6814055420 con 0x7f68240d16a0 2024-03-29T10:12:21.313 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:21.313+0000 7ff3990d7240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:12:21.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.435+0000 7f6823400700 1 -- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406aed0 msgr2=0x7f680406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.435+0000 7f6823400700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406aed0 0x7f680406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:21.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.455+0000 7f682aa00700 1 -- 172.21.15.12:0/2346817736 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6824067720 con 0x7f68240d16a0 2024-03-29T10:12:21.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.455+0000 7f681be00700 1 -- 172.21.15.12:0/2346817736 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6814088e60 con 0x7f68240d16a0 2024-03-29T10:12:21.455 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:12:21.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 -- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406aed0 msgr2=0x7f680406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:21.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406aed0 0x7f680406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 -- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68240d16a0 msgr2=0x7f682415c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:21.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68240d16a0 0x7f682415c9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6814016000 tx=0x7f6814004750 comp rx=0 tx=0).stop 2024-03-29T10:12:21.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 -- 172.21.15.12:0/2346817736 shutdown_connections 2024-03-29T10:12:21.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f680406aed0 0x7f680406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f682415d9d0 0x7f68241e9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68240d16a0 0x7f682415c9c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 --2- 172.21.15.12:0/2346817736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68240d2540 0x7f68240b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 -- 172.21.15.12:0/2346817736 >> 172.21.15.12:0/2346817736 conn(0x7f682408aff0 msgr2=0x7f68240b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:21.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 -- 172.21.15.12:0/2346817736 shutdown_connections 2024-03-29T10:12:21.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.458+0000 7f682aa00700 1 -- 172.21.15.12:0/2346817736 wait complete. 2024-03-29T10:12:21.477 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T10:12:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.678+0000 7f3697000700 1 Processor -- start 2024-03-29T10:12:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.678+0000 7f3697000700 1 -- start start 2024-03-29T10:12:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.679+0000 7f3697000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f369015dad0 0x7f36900ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.679+0000 7f3697000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f36900d1630 0x7f36900d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.679+0000 7f3697000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369015f820 con 0x7f36900d2000 2024-03-29T10:12:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.679+0000 7f3697000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36900cfcc0 con 0x7f36900d1630 2024-03-29T10:12:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.679+0000 7f3697000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36900cd9f0 con 0x7f369015dad0 2024-03-29T10:12:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.679+0000 7f368f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f36900d1630 0x7f36900d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.679+0000 7f3695000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f36900d2000 0x7f369015deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54348/0 (socket says 172.21.15.12:54348) 2024-03-29T10:12:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.679+0000 7f3695000700 1 -- 172.21.15.12:0/1381812992 learned_addr learned my addr 172.21.15.12:0/1381812992 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.680+0000 7f368fe00700 1 --2- 172.21.15.12:0/1381812992 >> v2:172.21.15.159:3300/0 conn(0x7f369015dad0 0x7f36900ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:21.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.680+0000 7f368ea00700 1 -- 172.21.15.12:0/1381812992 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230771180 0 0) 0x7f369015f820 con 0x7f36900d2000 2024-03-29T10:12:21.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.680+0000 7f368f400700 1 -- 172.21.15.12:0/1381812992 >> v2:172.21.15.159:3300/0 conn(0x7f369015dad0 msgr2=0x7f36900ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:21.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.680+0000 7f368f400700 1 --2- 172.21.15.12:0/1381812992 >> v2:172.21.15.159:3300/0 conn(0x7f369015dad0 0x7f36900ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.680+0000 7f368f400700 1 -- 172.21.15.12:0/1381812992 >> v1:172.21.15.12:6789/0 conn(0x7f36900d2000 legacy=0x7f369015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.680+0000 7f368f400700 1 -- 172.21.15.12:0/1381812992 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36900d2430 con 0x7f36900d1630 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.680+0000 7f368f400700 1 --2- 172.21.15.12:0/1381812992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36900d1630 0x7f36900d1ac0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f369008d380 tx=0x7f367c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ca057a0f36ce71 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.680+0000 7f368ea00700 1 -- 172.21.15.12:0/1381812992 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c011830 con 0x7f36900d1630 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.681+0000 7f368ea00700 1 -- 172.21.15.12:0/1381812992 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367c011990 con 0x7f36900d1630 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.681+0000 7f368ea00700 1 -- 172.21.15.12:0/1381812992 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c011d20 con 0x7f36900d1630 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.681+0000 7f3697000700 1 -- 172.21.15.12:0/1381812992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36900d1630 msgr2=0x7f36900d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.681+0000 7f3697000700 1 --2- 172.21.15.12:0/1381812992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36900d1630 0x7f36900d1ac0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f369008d380 tx=0x7f367c009b20 comp rx=0 tx=0).stop 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.681+0000 7f3697000700 1 -- 172.21.15.12:0/1381812992 shutdown_connections 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.681+0000 7f3697000700 1 --2- 172.21.15.12:0/1381812992 >> v2:172.21.15.159:3300/0 conn(0x7f369015dad0 0x7f36900ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.681+0000 7f3697000700 1 --2- 172.21.15.12:0/1381812992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36900d1630 0x7f36900d1ac0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.681+0000 7f3697000700 1 -- 172.21.15.12:0/1381812992 >> 172.21.15.12:0/1381812992 conn(0x7f369008aff0 msgr2=0x7f36900d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.681+0000 7f3697000700 1 -- 172.21.15.12:0/1381812992 shutdown_connections 2024-03-29T10:12:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.681+0000 7f3697000700 1 -- 172.21.15.12:0/1381812992 wait complete. 2024-03-29T10:12:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.683+0000 7f3697000700 1 Processor -- start 2024-03-29T10:12:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.683+0000 7f3697000700 1 -- start start 2024-03-29T10:12:21.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f3697000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36900d1630 0x7f36900b85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f3697000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36900d2000 0x7f36901f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f3697000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f369015dad0 0x7f36900c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f3697000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36901f3e90 con 0x7f369015dad0 2024-03-29T10:12:21.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f3697000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36900c7b20 con 0x7f36900d2000 2024-03-29T10:12:21.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f3697000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36900b8b10 con 0x7f36900d1630 2024-03-29T10:12:21.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f3695000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f369015dad0 0x7f36900c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f368fe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36900d1630 0x7f36900b85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f368f400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36900d2000 0x7f36901f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f3695000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f369015dad0 0x7f36900c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40248/0 (socket says 172.21.15.12:40248) 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.684+0000 7f3695000700 1 -- 172.21.15.12:0/2515735146 learned_addr learned my addr 172.21.15.12:0/2515735146 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.685+0000 7f3695000700 1 -- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36900d1630 msgr2=0x7f36900b85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.685+0000 7f3695000700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36900d1630 0x7f36900b85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.685+0000 7f3695000700 1 -- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36900d2000 msgr2=0x7f36901f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.685+0000 7f3695000700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36900d2000 0x7f36901f4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.685+0000 7f3695000700 1 -- 172.21.15.12:0/2515735146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c0094a0 con 0x7f369015dad0 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.685+0000 7f3695000700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f369015dad0 0x7f36900c9240 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f368800cca0 tx=0x7f368800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.685+0000 7f3687e00700 1 -- 172.21.15.12:0/2515735146 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368800ab70 con 0x7f369015dad0 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.686+0000 7f3687e00700 1 -- 172.21.15.12:0/2515735146 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368800acd0 con 0x7f369015dad0 2024-03-29T10:12:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.686+0000 7f3687e00700 1 -- 172.21.15.12:0/2515735146 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688007310 con 0x7f369015dad0 2024-03-29T10:12:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.686+0000 7f3697000700 1 -- 172.21.15.12:0/2515735146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36900c71f0 con 0x7f369015dad0 2024-03-29T10:12:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.686+0000 7f3697000700 1 -- 172.21.15.12:0/2515735146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36900c77b0 con 0x7f369015dad0 2024-03-29T10:12:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.687+0000 7f3697000700 1 -- 172.21.15.12:0/2515735146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3690067720 con 0x7f369015dad0 2024-03-29T10:12:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.688+0000 7f3687e00700 1 -- 172.21.15.12:0/2515735146 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f3688007470 con 0x7f369015dad0 2024-03-29T10:12:21.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.688+0000 7f3687e00700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f367006aed0 0x7f367006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:21.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.689+0000 7f3687e00700 1 -- 172.21.15.12:0/2515735146 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f3688089540 con 0x7f369015dad0 2024-03-29T10:12:21.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.689+0000 7f368fe00700 1 -- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f367006aed0 msgr2=0x7f367006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:21.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.689+0000 7f368fe00700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f367006aed0 0x7f367006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.697+0000 7f3687e00700 1 -- 172.21.15.12:0/2515735146 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3688054160 con 0x7f369015dad0 2024-03-29T10:12:21.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.890+0000 7f368fe00700 1 -- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f367006aed0 msgr2=0x7f367006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:21.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.890+0000 7f368fe00700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f367006aed0 0x7f367006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:21.916 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:21.916+0000 7f3f29e3c240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:12:21.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.930+0000 7f3697000700 1 -- 172.21.15.12:0/2515735146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f36900b6130 con 0x7f369015dad0 2024-03-29T10:12:21.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.931+0000 7f3687e00700 1 -- 172.21.15.12:0/2515735146 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v111) v1 ==== 80+0+466 (secure 0 0 0) 0x7f3688057780 con 0x7f369015dad0 2024-03-29T10:12:21.931 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/43444 2024-03-30T10:11:15.808228+0000 2024-03-29T10:12:21.931 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/43444 2024-03-30T10:11:15.808228+0000 2024-03-29T10:12:21.931 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/430021803 2024-03-30T10:11:15.808228+0000 2024-03-29T10:12:21.931 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2229015445 2024-03-30T10:11:15.808228+0000 2024-03-29T10:12:21.931 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2018718607 2024-03-30T10:11:15.808228+0000 2024-03-29T10:12:21.931 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/663205968 2024-03-30T10:11:15.808228+0000 2024-03-29T10:12:21.931 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1058856906 2024-03-30T10:11:15.808228+0000 2024-03-29T10:12:21.931 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2814107463 2024-03-30T10:11:15.808228+0000 2024-03-29T10:12:21.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.933+0000 7f3697000700 1 -- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f367006aed0 msgr2=0x7f367006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.933+0000 7f3697000700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f367006aed0 0x7f367006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.933+0000 7f3697000700 1 -- 172.21.15.12:0/2515735146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f369015dad0 msgr2=0x7f36900c9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.933+0000 7f3697000700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f369015dad0 0x7f36900c9240 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f368800cca0 tx=0x7f368800a060 comp rx=0 tx=0).stop 2024-03-29T10:12:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.933+0000 7f3697000700 1 -- 172.21.15.12:0/2515735146 shutdown_connections 2024-03-29T10:12:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.933+0000 7f3697000700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f367006aed0 0x7f367006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.933+0000 7f3697000700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f369015dad0 0x7f36900c9240 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.933+0000 7f3697000700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36900d1630 0x7f36900b85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.933+0000 7f3697000700 1 --2- 172.21.15.12:0/2515735146 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36900d2000 0x7f36901f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.933+0000 7f3697000700 1 -- 172.21.15.12:0/2515735146 >> 172.21.15.12:0/2515735146 conn(0x7f369008aff0 msgr2=0x7f369006c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.934+0000 7f3697000700 1 -- 172.21.15.12:0/2515735146 shutdown_connections 2024-03-29T10:12:21.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:21.934+0000 7f3697000700 1 -- 172.21.15.12:0/2515735146 wait complete. 2024-03-29T10:12:21.938 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T10:12:21.952 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T10:12:21.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:21.953 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T10:12:22.007 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:22.007+0000 7f3f29e3c240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:12:22.028 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T10:12:22.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:22.029 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T10:12:22.074 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:12:22.074 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:12:22.085 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:22.086+0000 7ff3990d7240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:12:22.119 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:12:22.150 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:12:22.150 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:12:22.171 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:22.171+0000 7f3f29e3c240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:12:22.182 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:22.182+0000 7ff3990d7240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:12:22.198 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:12:22.198 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:12:22.198 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:12:22.251 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:12:22.251 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:12:22.353 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:22.354+0000 7ff3990d7240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:12:22.390 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:22.390+0000 7f3f29e3c240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:12:22.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.456+0000 7f19fb600700 1 Processor -- start 2024-03-29T10:12:22.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.456+0000 7f19fb600700 1 -- start start 2024-03-29T10:12:22.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.458+0000 7f19fb600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f19f415b5d0 0x7f19f40ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.458+0000 7f19fb600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f40cb3f0 0x7f19f40ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.458+0000 7f19fb600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f415fd50 con 0x7f19f40ae930 2024-03-29T10:12:22.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.458+0000 7f19fb600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f415b9b0 con 0x7f19f415b5d0 2024-03-29T10:12:22.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.458+0000 7f19fb600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f415f140 con 0x7f19f40cb3f0 2024-03-29T10:12:22.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.458+0000 7f19f3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f40cb3f0 0x7f19f40ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:22.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.458+0000 7f19f8c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f19f415b5d0 0x7f19f40ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:22.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.458+0000 7f19f3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f40cb3f0 0x7f19f40ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:59494/0 (socket says 172.21.15.12:59494) 2024-03-29T10:12:22.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.458+0000 7f19f3e00700 1 -- 172.21.15.12:0/2251657851 learned_addr learned my addr 172.21.15.12:0/2251657851 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:22.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.459+0000 7f19f3e00700 1 -- 172.21.15.12:0/2251657851 >> v2:172.21.15.159:3300/0 conn(0x7f19f415b5d0 msgr2=0x7f19f40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:22.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.459+0000 7f19f3e00700 1 --2- 172.21.15.12:0/2251657851 >> v2:172.21.15.159:3300/0 conn(0x7f19f415b5d0 0x7f19f40ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.459+0000 7f19f3e00700 1 -- 172.21.15.12:0/2251657851 >> v1:172.21.15.159:6789/0 conn(0x7f19f40ae930 legacy=0x7f19f4140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:22.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.459+0000 7f19f3e00700 1 -- 172.21.15.12:0/2251657851 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f415f420 con 0x7f19f40cb3f0 2024-03-29T10:12:22.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.459+0000 7f19f8c00700 1 --2- 172.21.15.12:0/2251657851 >> v2:172.21.15.159:3300/0 conn(0x7f19f415b5d0 0x7f19f40ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:22.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.459+0000 7f19f3e00700 1 --2- 172.21.15.12:0/2251657851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f40cb3f0 0x7f19f40ac0c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f19e00098f0 tx=0x7f19e0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6a1aafca53dfeb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:22.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.459+0000 7f19f3400700 1 -- 172.21.15.12:0/2251657851 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0011830 con 0x7f19f40cb3f0 2024-03-29T10:12:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.459+0000 7f19f3400700 1 -- 172.21.15.12:0/2251657851 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0011990 con 0x7f19f40cb3f0 2024-03-29T10:12:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.459+0000 7f19f3400700 1 -- 172.21.15.12:0/2251657851 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0011d20 con 0x7f19f40cb3f0 2024-03-29T10:12:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.460+0000 7f19fb600700 1 -- 172.21.15.12:0/2251657851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f40cb3f0 msgr2=0x7f19f40ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.460+0000 7f19fb600700 1 --2- 172.21.15.12:0/2251657851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f40cb3f0 0x7f19f40ac0c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f19e00098f0 tx=0x7f19e0009c00 comp rx=0 tx=0).stop 2024-03-29T10:12:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.460+0000 7f19fb600700 1 -- 172.21.15.12:0/2251657851 shutdown_connections 2024-03-29T10:12:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.460+0000 7f19fb600700 1 --2- 172.21.15.12:0/2251657851 >> v2:172.21.15.159:3300/0 conn(0x7f19f415b5d0 0x7f19f40ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.460+0000 7f19fb600700 1 --2- 172.21.15.12:0/2251657851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f40cb3f0 0x7f19f40ac0c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.460+0000 7f19fb600700 1 -- 172.21.15.12:0/2251657851 >> 172.21.15.12:0/2251657851 conn(0x7f19f40cf190 msgr2=0x7f19f40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.460+0000 7f19fb600700 1 -- 172.21.15.12:0/2251657851 shutdown_connections 2024-03-29T10:12:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.460+0000 7f19fb600700 1 -- 172.21.15.12:0/2251657851 wait complete. 2024-03-29T10:12:22.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.462+0000 7f19fb600700 1 Processor -- start 2024-03-29T10:12:22.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.462+0000 7f19fb600700 1 -- start start 2024-03-29T10:12:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.463+0000 7f19fb600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f40ae930 0x7f19f415cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.463+0000 7f19fb600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19f415b5d0 0x7f19f41559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.463+0000 7f19fb600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f4156580 0x7f19f4155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.463+0000 7f19fb600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f4157b30 con 0x7f19f415b5d0 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.463+0000 7f19fb600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f4156420 con 0x7f19f40ae930 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.463+0000 7f19fb600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40b3900 con 0x7f19f4156580 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.463+0000 7f19f3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19f415b5d0 0x7f19f41559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.463+0000 7f19f8c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f40ae930 0x7f19f415cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.464+0000 7f19f3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19f415b5d0 0x7f19f41559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40256/0 (socket says 172.21.15.12:40256) 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.463+0000 7f19f9600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f4156580 0x7f19f4155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.464+0000 7f19f3e00700 1 -- 172.21.15.12:0/1896621459 learned_addr learned my addr 172.21.15.12:0/1896621459 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.464+0000 7f19f8c00700 1 -- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f4156580 msgr2=0x7f19f4155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.464+0000 7f19f8c00700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f4156580 0x7f19f4155ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.464+0000 7f19f8c00700 1 -- 172.21.15.12:0/1896621459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19f415b5d0 msgr2=0x7f19f41559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.464+0000 7f19f8c00700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19f415b5d0 0x7f19f41559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.464+0000 7f19f8c00700 1 -- 172.21.15.12:0/1896621459 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e80094a0 con 0x7f19f40ae930 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.464+0000 7f19f3e00700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19f415b5d0 0x7f19f41559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.464+0000 7f19f8c00700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f40ae930 0x7f19f415cec0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f19e800e900 tx=0x7f19e800ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.465+0000 7f19f0c00700 1 -- 172.21.15.12:0/1896621459 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8013680 con 0x7f19f40ae930 2024-03-29T10:12:22.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.465+0000 7f19f0c00700 1 -- 172.21.15.12:0/1896621459 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e80137e0 con 0x7f19f40ae930 2024-03-29T10:12:22.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.465+0000 7f19f0c00700 1 -- 172.21.15.12:0/1896621459 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8013b70 con 0x7f19f40ae930 2024-03-29T10:12:22.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.465+0000 7f19fb600700 1 -- 172.21.15.12:0/1896621459 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e0009550 con 0x7f19f40ae930 2024-03-29T10:12:22.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.465+0000 7f19fb600700 1 -- 172.21.15.12:0/1896621459 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f4157750 con 0x7f19f40ae930 2024-03-29T10:12:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.466+0000 7f19fb600700 1 -- 172.21.15.12:0/1896621459 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f40b3dc0 con 0x7f19f40ae930 2024-03-29T10:12:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.467+0000 7f19f0c00700 1 -- 172.21.15.12:0/1896621459 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f19e8013cd0 con 0x7f19f40ae930 2024-03-29T10:12:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.468+0000 7f19f0c00700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f19d406aed0 0x7f19d406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.468+0000 7f19f0c00700 1 -- 172.21.15.12:0/1896621459 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f19e80898d0 con 0x7f19f40ae930 2024-03-29T10:12:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.468+0000 7f19f3e00700 1 -- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f19d406aed0 msgr2=0x7f19d406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.468+0000 7f19f3e00700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f19d406aed0 0x7f19d406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:22.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.472+0000 7f19f0c00700 1 -- 172.21.15.12:0/1896621459 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f19e8054530 con 0x7f19f40ae930 2024-03-29T10:12:22.482 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:22.482+0000 7f3f29e3c240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:12:22.576 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:22.576+0000 7ff3990d7240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:12:22.594 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:22.594+0000 7f3f29e3c240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:12:22.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.669+0000 7f19f3e00700 1 -- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f19d406aed0 msgr2=0x7f19d406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.669+0000 7f19f3e00700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f19d406aed0 0x7f19d406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:22.672 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:22.673+0000 7ff3990d7240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:12:22.693 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:22.693+0000 7f3f29e3c240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:12:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.739+0000 7f19fb600700 1 -- 172.21.15.12:0/1896621459 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19f4067720 con 0x7f19f40ae930 2024-03-29T10:12:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.740+0000 7f19f0c00700 1 -- 172.21.15.12:0/1896621459 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f19e8057b50 con 0x7f19f40ae930 2024-03-29T10:12:22.740 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:22.740 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6452,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2137480356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2137480356},{"type":"v1","addr":"172.21.15.159:6836","nonce":2137480356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":6455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2356039533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2356039533},{"type":"v1","addr":"172.21.15.159:6837","nonce":2356039533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":7153,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2916743006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2916743006},{"type":"v1","addr":"172.21.15.12:6835","nonce":2916743006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:58.634187+0000","modified":"2024-03-29T10:11:02.111942+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7410":{"gid":7410,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3023331068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3023331068},{"type":"v1","addr":"172.21.15.12:6834","nonce":3023331068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.742+0000 7f19fb600700 1 -- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f19d406aed0 msgr2=0x7f19d406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.742+0000 7f19fb600700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f19d406aed0 0x7f19d406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.742+0000 7f19fb600700 1 -- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f40ae930 msgr2=0x7f19f415cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.742+0000 7f19fb600700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f40ae930 0x7f19f415cec0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f19e800e900 tx=0x7f19e800ec10 comp rx=0 tx=0).stop 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.743+0000 7f19fb600700 1 -- 172.21.15.12:0/1896621459 shutdown_connections 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.743+0000 7f19fb600700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f19d406aed0 0x7f19d406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.743+0000 7f19fb600700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f4156580 0x7f19f4155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.743+0000 7f19fb600700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f40ae930 0x7f19f415cec0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.743+0000 7f19fb600700 1 --2- 172.21.15.12:0/1896621459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19f415b5d0 0x7f19f41559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.743+0000 7f19fb600700 1 -- 172.21.15.12:0/1896621459 >> 172.21.15.12:0/1896621459 conn(0x7f19f40cf190 msgr2=0x7f19f40aff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.743+0000 7f19fb600700 1 -- 172.21.15.12:0/1896621459 shutdown_connections 2024-03-29T10:12:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.743+0000 7f19fb600700 1 -- 172.21.15.12:0/1896621459 wait complete. 2024-03-29T10:12:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 46 2024-03-29T10:12:22.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:12:22.779 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:22.779+0000 7ff3990d7240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:12:22.791 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:22.792+0000 7f3f29e3c240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:12:22.886 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:22.886+0000 7ff3990d7240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:12:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.961+0000 7fb4b6800700 1 Processor -- start 2024-03-29T10:12:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.961+0000 7fb4b6800700 1 -- start start 2024-03-29T10:12:22.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.962+0000 7fb4b6800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b015b5d0 0x7fb4b00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.962+0000 7fb4b6800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb4b00cb3f0 0x7fb4b00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.962+0000 7fb4b6800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b015fd50 con 0x7fb4b00ae930 2024-03-29T10:12:22.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.962+0000 7fb4b6800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b015b9b0 con 0x7fb4b00cb3f0 2024-03-29T10:12:22.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.962+0000 7fb4b6800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b015f140 con 0x7fb4b015b5d0 2024-03-29T10:12:22.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.962+0000 7fb4afe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb4b00ae930 0x7fb4b0140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54358/0 (socket says 172.21.15.12:54358) 2024-03-29T10:12:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.962+0000 7fb4afe00700 1 -- 172.21.15.12:0/2809614562 learned_addr learned my addr 172.21.15.12:0/2809614562 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.962+0000 7fb4af400700 1 --2- 172.21.15.12:0/2809614562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b015b5d0 0x7fb4b00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.962+0000 7fb4aea00700 1 --2- 172.21.15.12:0/2809614562 >> v2:172.21.15.159:3300/0 conn(0x7fb4b00cb3f0 0x7fb4b00ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:22.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.963+0000 7fb4af400700 1 -- 172.21.15.12:0/2809614562 >> v2:172.21.15.159:3300/0 conn(0x7fb4b00cb3f0 msgr2=0x7fb4b00ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:22.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.963+0000 7fb4af400700 1 --2- 172.21.15.12:0/2809614562 >> v2:172.21.15.159:3300/0 conn(0x7fb4b00cb3f0 0x7fb4b00ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.963+0000 7fb4af400700 1 -- 172.21.15.12:0/2809614562 >> v1:172.21.15.12:6789/0 conn(0x7fb4b00ae930 legacy=0x7fb4b0140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.963+0000 7fb4ae000700 1 -- 172.21.15.12:0/2809614562 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354726829 0 0) 0x7fb4b015fd50 con 0x7fb4b00ae930 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.963+0000 7fb4af400700 1 -- 172.21.15.12:0/2809614562 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b015f420 con 0x7fb4b015b5d0 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.963+0000 7fb4af400700 1 --2- 172.21.15.12:0/2809614562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b015b5d0 0x7fb4b00ae3f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009840 tx=0x7fb4a0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3251e20e72597a9b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.963+0000 7fb4ae000700 1 -- 172.21.15.12:0/2809614562 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0011830 con 0x7fb4b015b5d0 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.963+0000 7fb4ae000700 1 -- 172.21.15.12:0/2809614562 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a0011990 con 0x7fb4b015b5d0 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.964+0000 7fb4ae000700 1 -- 172.21.15.12:0/2809614562 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0011d20 con 0x7fb4b015b5d0 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.964+0000 7fb4b6800700 1 -- 172.21.15.12:0/2809614562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b015b5d0 msgr2=0x7fb4b00ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.964+0000 7fb4b6800700 1 --2- 172.21.15.12:0/2809614562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b015b5d0 0x7fb4b00ae3f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009840 tx=0x7fb4a0009b50 comp rx=0 tx=0).stop 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.964+0000 7fb4b6800700 1 -- 172.21.15.12:0/2809614562 shutdown_connections 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.964+0000 7fb4b6800700 1 --2- 172.21.15.12:0/2809614562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b015b5d0 0x7fb4b00ae3f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.964+0000 7fb4b6800700 1 --2- 172.21.15.12:0/2809614562 >> v2:172.21.15.159:3300/0 conn(0x7fb4b00cb3f0 0x7fb4b00ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.964+0000 7fb4b6800700 1 -- 172.21.15.12:0/2809614562 >> 172.21.15.12:0/2809614562 conn(0x7fb4b00cf190 msgr2=0x7fb4b00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.964+0000 7fb4b6800700 1 -- 172.21.15.12:0/2809614562 shutdown_connections 2024-03-29T10:12:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.964+0000 7fb4b6800700 1 -- 172.21.15.12:0/2809614562 wait complete. 2024-03-29T10:12:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.966+0000 7fb4b6800700 1 Processor -- start 2024-03-29T10:12:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.966+0000 7fb4b6800700 1 -- start start 2024-03-29T10:12:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4b6800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b00ae930 0x7fb4b00ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4b6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4b015b5d0 0x7fb4b0156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4b6800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4b00989b0 0x7fb4b00cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4b6800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b00cd570 con 0x7fb4b015b5d0 2024-03-29T10:12:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4b6800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b00cc0e0 con 0x7fb4b00989b0 2024-03-29T10:12:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4b6800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b00cd810 con 0x7fb4b00ae930 2024-03-29T10:12:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4aea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4b015b5d0 0x7fb4b0156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4af400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b00ae930 0x7fb4b00ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4aea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4b015b5d0 0x7fb4b0156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40260/0 (socket says 172.21.15.12:40260) 2024-03-29T10:12:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4aea00700 1 -- 172.21.15.12:0/112117152 learned_addr learned my addr 172.21.15.12:0/112117152 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.967+0000 7fb4afe00700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4b00989b0 0x7fb4b00cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.968+0000 7fb4af400700 1 -- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4b00989b0 msgr2=0x7fb4b00cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.968+0000 7fb4af400700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4b00989b0 0x7fb4b00cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.968+0000 7fb4af400700 1 -- 172.21.15.12:0/112117152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4b015b5d0 msgr2=0x7fb4b0156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.968+0000 7fb4af400700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4b015b5d0 0x7fb4b0156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.968+0000 7fb4af400700 1 -- 172.21.15.12:0/112117152 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a00094a0 con 0x7fb4b00ae930 2024-03-29T10:12:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.968+0000 7fb4af400700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b00ae930 0x7fb4b00ce5d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0016000 tx=0x7fb4a0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:22.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.968+0000 7fb4a7400700 1 -- 172.21.15.12:0/112117152 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0004550 con 0x7fb4b00ae930 2024-03-29T10:12:22.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.968+0000 7fb4a7400700 1 -- 172.21.15.12:0/112117152 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a001f070 con 0x7fb4b00ae930 2024-03-29T10:12:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.968+0000 7fb4b6800700 1 -- 172.21.15.12:0/112117152 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b00cdaf0 con 0x7fb4b00ae930 2024-03-29T10:12:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.968+0000 7fb4a7400700 1 -- 172.21.15.12:0/112117152 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a00038e0 con 0x7fb4b00ae930 2024-03-29T10:12:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.969+0000 7fb4b6800700 1 -- 172.21.15.12:0/112117152 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b00ce0b0 con 0x7fb4b00ae930 2024-03-29T10:12:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.970+0000 7fb4b6800700 1 -- 172.21.15.12:0/112117152 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b0067720 con 0x7fb4b00ae930 2024-03-29T10:12:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.970+0000 7fb4a7400700 1 -- 172.21.15.12:0/112117152 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fb4a0003a40 con 0x7fb4b00ae930 2024-03-29T10:12:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.971+0000 7fb4a7400700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb49006aed0 0x7fb49006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.971+0000 7fb4a7400700 1 -- 172.21.15.12:0/112117152 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fb4a0021070 con 0x7fb4b00ae930 2024-03-29T10:12:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.971+0000 7fb4aea00700 1 -- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb49006aed0 msgr2=0x7fb49006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.971+0000 7fb4aea00700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb49006aed0 0x7fb49006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:22.981+0000 7fb4a7400700 1 -- 172.21.15.12:0/112117152 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb4a0055d00 con 0x7fb4b00ae930 2024-03-29T10:12:22.994 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:22.995+0000 7ff3990d7240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:12:23.008 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:23.008+0000 7f3f29e3c240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:12:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.172+0000 7fb4aea00700 1 -- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb49006aed0 msgr2=0x7fb49006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.172+0000 7fb4aea00700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb49006aed0 0x7fb49006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.197+0000 7fb4b6800700 1 -- 172.21.15.12:0/112117152 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4b00cd950 con 0x7fb4b00ae930 2024-03-29T10:12:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.199+0000 7fb4a7400700 1 -- 172.21.15.12:0/112117152 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16650 (secure 0 0 0) 0x7fb4b00cd950 con 0x7fb4b00ae930 2024-03-29T10:12:23.199 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:23.199 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":111,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:12:17.179326+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:57.064406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:57.607254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3e","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6833/43444":"2024-03-30T10:11:15.808228+0000","172.21.15.159:6832/43444":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/430021803":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2229015445":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2018718607":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/663205968":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/1058856906":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2814107463":"2024-03-30T10:11:15.808228+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:12:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.201+0000 7fb4b6800700 1 -- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb49006aed0 msgr2=0x7fb49006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.201+0000 7fb4b6800700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb49006aed0 0x7fb49006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.202+0000 7fb4b6800700 1 -- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b00ae930 msgr2=0x7fb4b00ce5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.202+0000 7fb4b6800700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b00ae930 0x7fb4b00ce5d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0016000 tx=0x7fb4a0004750 comp rx=0 tx=0).stop 2024-03-29T10:12:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.202+0000 7fb4b6800700 1 -- 172.21.15.12:0/112117152 shutdown_connections 2024-03-29T10:12:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.202+0000 7fb4b6800700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb49006aed0 0x7fb49006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.202+0000 7fb4b6800700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4b00989b0 0x7fb4b00cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.202+0000 7fb4b6800700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4b00ae930 0x7fb4b00ce5d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.202+0000 7fb4b6800700 1 --2- 172.21.15.12:0/112117152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4b015b5d0 0x7fb4b0156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.202+0000 7fb4b6800700 1 -- 172.21.15.12:0/112117152 >> 172.21.15.12:0/112117152 conn(0x7fb4b00cf190 msgr2=0x7fb4b00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.202+0000 7fb4b6800700 1 -- 172.21.15.12:0/112117152 shutdown_connections 2024-03-29T10:12:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.202+0000 7fb4b6800700 1 -- 172.21.15.12:0/112117152 wait complete. 2024-03-29T10:12:23.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:10:57.064406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:10:57.607254+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:12:23.221 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:12:23.223 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:23.223+0000 7ff3990d7240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:12:23.270 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:23.270+0000 7f3f29e3c240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:12:23.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.430+0000 7fdc79800700 1 Processor -- start 2024-03-29T10:12:23.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.430+0000 7fdc79800700 1 -- start start 2024-03-29T10:12:23.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.431+0000 7fdc79800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740d21f0 0x7fdc740ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.431+0000 7fdc79800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc740ce270 con 0x7fdc740cec40 2024-03-29T10:12:23.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.431+0000 7fdc79800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7415b600 con 0x7fdc741611b0 2024-03-29T10:12:23.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.431+0000 7fdc79800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc740cf0b0 con 0x7fdc740d21f0 2024-03-29T10:12:23.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.432+0000 7fdc73400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdc740cec40 0x7fdc74161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54368/0 (socket says 172.21.15.12:54368) 2024-03-29T10:12:23.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.432+0000 7fdc73400700 1 -- 172.21.15.12:0/1520162627 learned_addr learned my addr 172.21.15.12:0/1520162627 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:23.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.432+0000 7fdc72000700 1 --2- 172.21.15.12:0/1520162627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740d21f0 0x7fdc740ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:23.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.432+0000 7fdc71600700 1 -- 172.21.15.12:0/1520162627 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802482533 0 0) 0x7fdc7415b600 con 0x7fdc741611b0 2024-03-29T10:12:23.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.432+0000 7fdc71600700 1 -- 172.21.15.12:0/1520162627 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc580035b0 con 0x7fdc741611b0 2024-03-29T10:12:23.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.432+0000 7fdc71600700 1 -- 172.21.15.12:0/1520162627 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365289821 0 0) 0x7fdc740ce270 con 0x7fdc740cec40 2024-03-29T10:12:23.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.432+0000 7fdc72000700 1 -- 172.21.15.12:0/1520162627 >> v1:172.21.15.159:6789/0 conn(0x7fdc741611b0 legacy=0x7fdc7415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.432+0000 7fdc72000700 1 -- 172.21.15.12:0/1520162627 >> v1:172.21.15.12:6789/0 conn(0x7fdc740cec40 legacy=0x7fdc74161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.432+0000 7fdc72000700 1 -- 172.21.15.12:0/1520162627 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc740d2760 con 0x7fdc740d21f0 2024-03-29T10:12:23.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.432+0000 7fdc72000700 1 --2- 172.21.15.12:0/1520162627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740d21f0 0x7fdc740ce700 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009870 tx=0x7fdc60009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=850a14bd1d8909a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:23.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.433+0000 7fdc71600700 1 -- 172.21.15.12:0/1520162627 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60011830 con 0x7fdc740d21f0 2024-03-29T10:12:23.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.433+0000 7fdc71600700 1 -- 172.21.15.12:0/1520162627 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc60011990 con 0x7fdc740d21f0 2024-03-29T10:12:23.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.433+0000 7fdc71600700 1 -- 172.21.15.12:0/1520162627 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60011d20 con 0x7fdc740d21f0 2024-03-29T10:12:23.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.433+0000 7fdc79800700 1 -- 172.21.15.12:0/1520162627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740d21f0 msgr2=0x7fdc740ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.433+0000 7fdc79800700 1 --2- 172.21.15.12:0/1520162627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740d21f0 0x7fdc740ce700 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009870 tx=0x7fdc60009b80 comp rx=0 tx=0).stop 2024-03-29T10:12:23.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.433+0000 7fdc79800700 1 -- 172.21.15.12:0/1520162627 shutdown_connections 2024-03-29T10:12:23.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.433+0000 7fdc79800700 1 --2- 172.21.15.12:0/1520162627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740d21f0 0x7fdc740ce700 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.433+0000 7fdc79800700 1 -- 172.21.15.12:0/1520162627 >> 172.21.15.12:0/1520162627 conn(0x7fdc7408aff0 msgr2=0x7fdc740d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:23.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.434+0000 7fdc79800700 1 -- 172.21.15.12:0/1520162627 shutdown_connections 2024-03-29T10:12:23.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.434+0000 7fdc79800700 1 -- 172.21.15.12:0/1520162627 wait complete. 2024-03-29T10:12:23.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.435+0000 7fdc79800700 1 Processor -- start 2024-03-29T10:12:23.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.436+0000 7fdc79800700 1 -- start start 2024-03-29T10:12:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.436+0000 7fdc79800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740cec40 0x7fdc741f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc79800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc740d21f0 0x7fdc7415f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc79800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc741611b0 0x7fdc7415fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc79800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc741f3d90 con 0x7fdc741611b0 2024-03-29T10:12:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc79800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc741f62c0 con 0x7fdc740d21f0 2024-03-29T10:12:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc79800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc741f3110 con 0x7fdc740cec40 2024-03-29T10:12:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc73400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc741611b0 0x7fdc7415fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc72000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc740d21f0 0x7fdc7415f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc73400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc741611b0 0x7fdc7415fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40276/0 (socket says 172.21.15.12:40276) 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc73400700 1 -- 172.21.15.12:0/2687628566 learned_addr learned my addr 172.21.15.12:0/2687628566 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc72a00700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740cec40 0x7fdc741f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc73400700 1 -- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740cec40 msgr2=0x7fdc741f5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc73400700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740cec40 0x7fdc741f5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc73400700 1 -- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc740d21f0 msgr2=0x7fdc7415f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc73400700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc740d21f0 0x7fdc7415f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.437+0000 7fdc73400700 1 -- 172.21.15.12:0/2687628566 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc600094a0 con 0x7fdc741611b0 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.438+0000 7fdc73400700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc741611b0 0x7fdc7415fdc0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00cbf0 tx=0x7fdc6c00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.438+0000 7fdc6aa00700 1 -- 172.21.15.12:0/2687628566 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c00aaa0 con 0x7fdc741611b0 2024-03-29T10:12:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.438+0000 7fdc6aa00700 1 -- 172.21.15.12:0/2687628566 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6c00ac00 con 0x7fdc741611b0 2024-03-29T10:12:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.438+0000 7fdc6aa00700 1 -- 172.21.15.12:0/2687628566 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c0070d0 con 0x7fdc741611b0 2024-03-29T10:12:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.438+0000 7fdc79800700 1 -- 172.21.15.12:0/2687628566 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc741f33f0 con 0x7fdc741611b0 2024-03-29T10:12:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.438+0000 7fdc79800700 1 -- 172.21.15.12:0/2687628566 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc741f39b0 con 0x7fdc741611b0 2024-03-29T10:12:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.439+0000 7fdc79800700 1 -- 172.21.15.12:0/2687628566 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc74067720 con 0x7fdc741611b0 2024-03-29T10:12:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.440+0000 7fdc6aa00700 1 -- 172.21.15.12:0/2687628566 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fdc6c007230 con 0x7fdc741611b0 2024-03-29T10:12:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.441+0000 7fdc6aa00700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5406aed0 0x7fdc5406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.441+0000 7fdc6aa00700 1 -- 172.21.15.12:0/2687628566 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fdc6c0892a0 con 0x7fdc741611b0 2024-03-29T10:12:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.441+0000 7fdc72a00700 1 -- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5406aed0 msgr2=0x7fdc5406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.441+0000 7fdc72a00700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5406aed0 0x7fdc5406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:23.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.446+0000 7fdc6aa00700 1 -- 172.21.15.12:0/2687628566 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdc6c054040 con 0x7fdc741611b0 2024-03-29T10:12:23.462 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:23.462+0000 7f3f29e3c240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:12:23.493 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:23.494+0000 7ff3990d7240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:12:23.554 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:23.554+0000 7f3f29e3c240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:12:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.642+0000 7fdc72a00700 1 -- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5406aed0 msgr2=0x7fdc5406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.642+0000 7fdc72a00700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5406aed0 0x7fdc5406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:23.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.662+0000 7fdc79800700 1 -- 172.21.15.12:0/2687628566 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdc740b5a90 con 0x7fdc741611b0 2024-03-29T10:12:23.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.662+0000 7fdc6aa00700 1 -- 172.21.15.12:0/2687628566 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdc6c0180a0 con 0x7fdc741611b0 2024-03-29T10:12:23.662 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:12:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 -- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5406aed0 msgr2=0x7fdc5406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5406aed0 0x7fdc5406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 -- 172.21.15.12:0/2687628566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc741611b0 msgr2=0x7fdc7415fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc741611b0 0x7fdc7415fdc0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00cbf0 tx=0x7fdc6c00cf00 comp rx=0 tx=0).stop 2024-03-29T10:12:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 -- 172.21.15.12:0/2687628566 shutdown_connections 2024-03-29T10:12:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc5406aed0 0x7fdc5406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc741611b0 0x7fdc7415fdc0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc740cec40 0x7fdc741f5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 --2- 172.21.15.12:0/2687628566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc740d21f0 0x7fdc7415f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 -- 172.21.15.12:0/2687628566 >> 172.21.15.12:0/2687628566 conn(0x7fdc7408aff0 msgr2=0x7fdc7406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 -- 172.21.15.12:0/2687628566 shutdown_connections 2024-03-29T10:12:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.665+0000 7fdc79800700 1 -- 172.21.15.12:0/2687628566 wait complete. 2024-03-29T10:12:23.683 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:12:23.698 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:23.698+0000 7ff3990d7240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:12:23.801 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:23.801+0000 7ff3990d7240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:12:23.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.887+0000 7f9347e00700 1 Processor -- start 2024-03-29T10:12:23.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.887+0000 7f9347e00700 1 -- start start 2024-03-29T10:12:23.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.888+0000 7f9347e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015d9d0 0x7f93400d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.888+0000 7f9347e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f93400d2540 0x7f93400ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.888+0000 7f9347e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9340161420 con 0x7f93400d16a0 2024-03-29T10:12:23.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.888+0000 7f9347e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934015edc0 con 0x7f93400d2540 2024-03-29T10:12:23.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.888+0000 7f9347e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9340161700 con 0x7f934015d9d0 2024-03-29T10:12:23.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.889+0000 7f9345400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015d9d0 0x7f93400d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:23.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.889+0000 7f9345e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f93400d2540 0x7f93400ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:23.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.889+0000 7f9345e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f93400d2540 0x7f93400ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44164/0 (socket says 172.21.15.12:44164) 2024-03-29T10:12:23.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.889+0000 7f9345e00700 1 -- 172.21.15.12:0/4043564340 learned_addr learned my addr 172.21.15.12:0/4043564340 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:23.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.889+0000 7f9345400700 1 -- 172.21.15.12:0/4043564340 >> v2:172.21.15.159:3300/0 conn(0x7f93400d2540 msgr2=0x7f93400ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:23.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.889+0000 7f9345400700 1 --2- 172.21.15.12:0/4043564340 >> v2:172.21.15.159:3300/0 conn(0x7f93400d2540 0x7f93400ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.889+0000 7f9345400700 1 -- 172.21.15.12:0/4043564340 >> v1:172.21.15.159:6789/0 conn(0x7f93400d16a0 legacy=0x7f93400d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.889+0000 7f9345400700 1 -- 172.21.15.12:0/4043564340 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93401619e0 con 0x7f934015d9d0 2024-03-29T10:12:23.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.889+0000 7f9345e00700 1 --2- 172.21.15.12:0/4043564340 >> v2:172.21.15.159:3300/0 conn(0x7f93400d2540 0x7f93400ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:23.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.889+0000 7f9345400700 1 --2- 172.21.15.12:0/4043564340 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015d9d0 0x7f93400d3c30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9334009870 tx=0x7f9334009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8966729d2471aac server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:23.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.890+0000 7f933fe00700 1 -- 172.21.15.12:0/4043564340 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334011830 con 0x7f934015d9d0 2024-03-29T10:12:23.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.890+0000 7f933fe00700 1 -- 172.21.15.12:0/4043564340 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9334011990 con 0x7f934015d9d0 2024-03-29T10:12:23.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.890+0000 7f933fe00700 1 -- 172.21.15.12:0/4043564340 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334011d20 con 0x7f934015d9d0 2024-03-29T10:12:23.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.890+0000 7f9347e00700 1 -- 172.21.15.12:0/4043564340 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015d9d0 msgr2=0x7f93400d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.890+0000 7f9347e00700 1 --2- 172.21.15.12:0/4043564340 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015d9d0 0x7f93400d3c30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9334009870 tx=0x7f9334009b80 comp rx=0 tx=0).stop 2024-03-29T10:12:23.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.890+0000 7f9347e00700 1 -- 172.21.15.12:0/4043564340 shutdown_connections 2024-03-29T10:12:23.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.890+0000 7f9347e00700 1 --2- 172.21.15.12:0/4043564340 >> v2:172.21.15.159:3300/0 conn(0x7f93400d2540 0x7f93400ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.890+0000 7f9347e00700 1 --2- 172.21.15.12:0/4043564340 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015d9d0 0x7f93400d3c30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.890+0000 7f9347e00700 1 -- 172.21.15.12:0/4043564340 >> 172.21.15.12:0/4043564340 conn(0x7f934008aff0 msgr2=0x7f93400d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:23.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.891+0000 7f9347e00700 1 -- 172.21.15.12:0/4043564340 shutdown_connections 2024-03-29T10:12:23.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.891+0000 7f9347e00700 1 -- 172.21.15.12:0/4043564340 wait complete. 2024-03-29T10:12:23.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.892+0000 7f9347e00700 1 Processor -- start 2024-03-29T10:12:23.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.892+0000 7f9347e00700 1 -- start start 2024-03-29T10:12:23.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.893+0000 7f9347e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d16a0 0x7f934015c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.893+0000 7f9347e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93400d2540 0x7f93400b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.893+0000 7f9347e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f934015d9d0 0x7f93401e9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.893+0000 7f9347e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93400b7d90 con 0x7f934015d9d0 2024-03-29T10:12:23.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.893+0000 7f9347e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93400cb3e0 con 0x7f93400d2540 2024-03-29T10:12:23.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.893+0000 7f9347e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93400b74f0 con 0x7f93400d16a0 2024-03-29T10:12:23.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f934015d9d0 0x7f93401e9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:23.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d16a0 0x7f934015c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:23.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9344a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93400d2540 0x7f93400b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:23.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f934015d9d0 0x7f93401e9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40288/0 (socket says 172.21.15.12:40288) 2024-03-29T10:12:23.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d16a0 0x7f934015c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:59568/0 (socket says 172.21.15.12:59568) 2024-03-29T10:12:23.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345e00700 1 -- 172.21.15.12:0/1564769942 learned_addr learned my addr 172.21.15.12:0/1564769942 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345400700 1 -- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93400d2540 msgr2=0x7f93400b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345400700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93400d2540 0x7f93400b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345400700 1 -- 172.21.15.12:0/1564769942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f934015d9d0 msgr2=0x7f93401e9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345400700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f934015d9d0 0x7f93401e9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345400700 1 -- 172.21.15.12:0/1564769942 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9338009b30 con 0x7f93400d16a0 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345e00700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f934015d9d0 0x7f93401e9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.894+0000 7f9345400700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d16a0 0x7f934015c9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9334016000 tx=0x7f9334004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.895+0000 7f933d600700 1 -- 172.21.15.12:0/1564769942 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340036a0 con 0x7f93400d16a0 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.895+0000 7f933d600700 1 -- 172.21.15.12:0/1564769942 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933401e070 con 0x7f93400d16a0 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.895+0000 7f933d600700 1 -- 172.21.15.12:0/1564769942 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334003a40 con 0x7f93400d16a0 2024-03-29T10:12:23.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.895+0000 7f9347e00700 1 -- 172.21.15.12:0/1564769942 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93340094a0 con 0x7f93400d16a0 2024-03-29T10:12:23.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.895+0000 7f9347e00700 1 -- 172.21.15.12:0/1564769942 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93400b7ab0 con 0x7f93400d16a0 2024-03-29T10:12:23.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.896+0000 7f9347e00700 1 -- 172.21.15.12:0/1564769942 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93400d1ca0 con 0x7f93400d16a0 2024-03-29T10:12:23.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.897+0000 7f933d600700 1 -- 172.21.15.12:0/1564769942 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f9334003ba0 con 0x7f93400d16a0 2024-03-29T10:12:23.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.898+0000 7f933d600700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006aed0 0x7f932006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:23.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.898+0000 7f933d600700 1 -- 172.21.15.12:0/1564769942 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f933408a690 con 0x7f93400d16a0 2024-03-29T10:12:23.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.898+0000 7f9344a00700 1 -- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006aed0 msgr2=0x7f932006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:23.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.898+0000 7f9344a00700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006aed0 0x7f932006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:23.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:23.903+0000 7f933d600700 1 -- 172.21.15.12:0/1564769942 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9334055420 con 0x7f93400d16a0 2024-03-29T10:12:24.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.099+0000 7f9344a00700 1 -- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006aed0 msgr2=0x7f932006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:24.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.099+0000 7f9344a00700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006aed0 0x7f932006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:24.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.119+0000 7f9347e00700 1 -- 172.21.15.12:0/1564769942 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9340067720 con 0x7f93400d16a0 2024-03-29T10:12:24.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.119+0000 7f933d600700 1 -- 172.21.15.12:0/1564769942 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9334088e60 con 0x7f93400d16a0 2024-03-29T10:12:24.119 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:12:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 -- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006aed0 msgr2=0x7f932006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006aed0 0x7f932006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 -- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d16a0 msgr2=0x7f934015c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d16a0 0x7f934015c9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9334016000 tx=0x7f9334004750 comp rx=0 tx=0).stop 2024-03-29T10:12:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 -- 172.21.15.12:0/1564769942 shutdown_connections 2024-03-29T10:12:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006aed0 0x7f932006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f934015d9d0 0x7f93401e9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d16a0 0x7f934015c9c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 --2- 172.21.15.12:0/1564769942 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93400d2540 0x7f93400b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 -- 172.21.15.12:0/1564769942 >> 172.21.15.12:0/1564769942 conn(0x7f934008aff0 msgr2=0x7f93400b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:24.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 -- 172.21.15.12:0/1564769942 shutdown_connections 2024-03-29T10:12:24.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.122+0000 7f9347e00700 1 -- 172.21.15.12:0/1564769942 wait complete. 2024-03-29T10:12:24.140 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:12:24.292 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:24.292+0000 7f3f29e3c240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:12:24.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.384+0000 7f97e4e00700 1 Processor -- start 2024-03-29T10:12:24.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.385+0000 7f97e4e00700 1 -- start start 2024-03-29T10:12:24.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.386+0000 7f97e4e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e006fa20 0x7f97e015e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.386+0000 7f97e4e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f97e015cfe0 0x7f97e015d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.386+0000 7f97e4e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e015eda0 con 0x7f97e00cd5c0 2024-03-29T10:12:24.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.386+0000 7f97e4e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e0099dd0 con 0x7f97e015cfe0 2024-03-29T10:12:24.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.386+0000 7f97e4e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e009a0b0 con 0x7f97e006fa20 2024-03-29T10:12:24.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.387+0000 7f97de800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f97e015cfe0 0x7f97e015d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:24.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.387+0000 7f97dd400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e006fa20 0x7f97e015e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:24.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.387+0000 7f97de800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f97e015cfe0 0x7f97e015d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44180/0 (socket says 172.21.15.12:44180) 2024-03-29T10:12:24.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.387+0000 7f97dd400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e006fa20 0x7f97e015e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:59580/0 (socket says 172.21.15.12:59580) 2024-03-29T10:12:24.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.387+0000 7f97de800700 1 -- 172.21.15.12:0/1034524083 learned_addr learned my addr 172.21.15.12:0/1034524083 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.387+0000 7f97dd400700 1 -- 172.21.15.12:0/1034524083 >> v2:172.21.15.159:3300/0 conn(0x7f97e015cfe0 msgr2=0x7f97e015d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.387+0000 7f97dd400700 1 --2- 172.21.15.12:0/1034524083 >> v2:172.21.15.159:3300/0 conn(0x7f97e015cfe0 0x7f97e015d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.387+0000 7f97dd400700 1 -- 172.21.15.12:0/1034524083 >> v1:172.21.15.159:6789/0 conn(0x7f97e00cd5c0 legacy=0x7f97e006f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.387+0000 7f97dd400700 1 -- 172.21.15.12:0/1034524083 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e009a390 con 0x7f97e006fa20 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.388+0000 7f97dd400700 1 --2- 172.21.15.12:0/1034524083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e006fa20 0x7f97e015e860 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f97cc009840 tx=0x7f97cc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19e92945cd851c77 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.388+0000 7f97dca00700 1 -- 172.21.15.12:0/1034524083 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc011830 con 0x7f97e006fa20 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.388+0000 7f97dca00700 1 -- 172.21.15.12:0/1034524083 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97cc011990 con 0x7f97e006fa20 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.388+0000 7f97dca00700 1 -- 172.21.15.12:0/1034524083 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc011d20 con 0x7f97e006fa20 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.389+0000 7f97e4e00700 1 -- 172.21.15.12:0/1034524083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e006fa20 msgr2=0x7f97e015e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.389+0000 7f97e4e00700 1 --2- 172.21.15.12:0/1034524083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e006fa20 0x7f97e015e860 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f97cc009840 tx=0x7f97cc009b50 comp rx=0 tx=0).stop 2024-03-29T10:12:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.389+0000 7f97e4e00700 1 -- 172.21.15.12:0/1034524083 shutdown_connections 2024-03-29T10:12:24.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.389+0000 7f97e4e00700 1 --2- 172.21.15.12:0/1034524083 >> v2:172.21.15.159:3300/0 conn(0x7f97e015cfe0 0x7f97e015d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.389+0000 7f97e4e00700 1 --2- 172.21.15.12:0/1034524083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e006fa20 0x7f97e015e860 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.389+0000 7f97e4e00700 1 -- 172.21.15.12:0/1034524083 >> 172.21.15.12:0/1034524083 conn(0x7f97e00d1370 msgr2=0x7f97e00d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:24.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.389+0000 7f97e4e00700 1 -- 172.21.15.12:0/1034524083 shutdown_connections 2024-03-29T10:12:24.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.389+0000 7f97e4e00700 1 -- 172.21.15.12:0/1034524083 wait complete. 2024-03-29T10:12:24.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.391+0000 7f97e4e00700 1 Processor -- start 2024-03-29T10:12:24.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.391+0000 7f97e4e00700 1 -- start start 2024-03-29T10:12:24.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97e4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97e006fa20 0x7f97e00b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97e4e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e00cd5c0 0x7f97e00c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97e4e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97e015cfe0 0x7f97e00b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97e4e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00b1ed0 con 0x7f97e006fa20 2024-03-29T10:12:24.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97e4e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00b22f0 con 0x7f97e015cfe0 2024-03-29T10:12:24.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97e4e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00c7f80 con 0x7f97e00cd5c0 2024-03-29T10:12:24.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97dde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97e006fa20 0x7f97e00b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:24.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97dde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97e006fa20 0x7f97e00b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40296/0 (socket says 172.21.15.12:40296) 2024-03-29T10:12:24.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97dde00700 1 -- 172.21.15.12:0/495783345 learned_addr learned my addr 172.21.15.12:0/495783345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:24.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97dd400700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e00cd5c0 0x7f97e00c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:24.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97de800700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97e015cfe0 0x7f97e00b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:24.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97dd400700 1 -- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97e015cfe0 msgr2=0x7f97e00b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:24.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97dd400700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97e015cfe0 0x7f97e00b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97dd400700 1 -- 172.21.15.12:0/495783345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97e006fa20 msgr2=0x7f97e00b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:24.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97dd400700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97e006fa20 0x7f97e00b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.393+0000 7f97dd400700 1 -- 172.21.15.12:0/495783345 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc0094a0 con 0x7f97e00cd5c0 2024-03-29T10:12:24.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.394+0000 7f97dd400700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e00cd5c0 0x7f97e00c7a40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f97cc016000 tx=0x7f97cc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:24.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.394+0000 7f97d6000700 1 -- 172.21.15.12:0/495783345 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc004510 con 0x7f97e00cd5c0 2024-03-29T10:12:24.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.394+0000 7f97d6000700 1 -- 172.21.15.12:0/495783345 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97cc01f070 con 0x7f97e00cd5c0 2024-03-29T10:12:24.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.394+0000 7f97d6000700 1 -- 172.21.15.12:0/495783345 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc0038c0 con 0x7f97e00cd5c0 2024-03-29T10:12:24.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.394+0000 7f97e4e00700 1 -- 172.21.15.12:0/495783345 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e00b49a0 con 0x7f97e00cd5c0 2024-03-29T10:12:24.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.394+0000 7f97e4e00700 1 -- 172.21.15.12:0/495783345 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e00b1530 con 0x7f97e00cd5c0 2024-03-29T10:12:24.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.395+0000 7f97e4e00700 1 -- 172.21.15.12:0/495783345 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e00b2010 con 0x7f97e00cd5c0 2024-03-29T10:12:24.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.396+0000 7f97d6000700 1 -- 172.21.15.12:0/495783345 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f97cc003a20 con 0x7f97e00cd5c0 2024-03-29T10:12:24.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.397+0000 7f97d6000700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97c006af00 0x7f97c006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.397+0000 7f97d6000700 1 -- 172.21.15.12:0/495783345 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f97cc021070 con 0x7f97e00cd5c0 2024-03-29T10:12:24.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.397+0000 7f97dde00700 1 -- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97c006af00 msgr2=0x7f97c006d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:24.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.397+0000 7f97dde00700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97c006af00 0x7f97c006d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:24.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.401+0000 7f97d6000700 1 -- 172.21.15.12:0/495783345 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f97e00b2010 con 0x7f97e00cd5c0 2024-03-29T10:12:24.507 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:24.507+0000 7f3f29e3c240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:12:24.586 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:24.586+0000 7ff3990d7240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:12:24.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.598+0000 7f97dde00700 1 -- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97c006af00 msgr2=0x7f97c006d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.598+0000 7f97dde00700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97c006af00 0x7f97c006d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.620+0000 7f97e4e00700 1 -- 172.21.15.12:0/495783345 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f97e00b2010 con 0x7f97e00cd5c0 2024-03-29T10:12:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.620+0000 7f97d6000700 1 -- 172.21.15.12:0/495783345 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+11 (secure 0 0 0) 0x7f97e00b2010 con 0x7f97e00cd5c0 2024-03-29T10:12:24.620 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:12:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 -- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97c006af00 msgr2=0x7f97c006d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97c006af00 0x7f97c006d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 -- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e00cd5c0 msgr2=0x7f97e00c7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e00cd5c0 0x7f97e00c7a40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f97cc016000 tx=0x7f97cc004750 comp rx=0 tx=0).stop 2024-03-29T10:12:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 -- 172.21.15.12:0/495783345 shutdown_connections 2024-03-29T10:12:24.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97c006af00 0x7f97c006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97e015cfe0 0x7f97e00b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97e006fa20 0x7f97e00b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 --2- 172.21.15.12:0/495783345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97e00cd5c0 0x7f97e00c7a40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 -- 172.21.15.12:0/495783345 >> 172.21.15.12:0/495783345 conn(0x7f97e00d1370 msgr2=0x7f97e006c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:24.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 -- 172.21.15.12:0/495783345 shutdown_connections 2024-03-29T10:12:24.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.623+0000 7f97e4e00700 1 -- 172.21.15.12:0/495783345 wait complete. 2024-03-29T10:12:24.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:12:24.715 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:24.715+0000 7f3f29e3c240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:12:24.815 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:24.815+0000 7ff3990d7240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:12:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.843+0000 7fdb83400700 1 Processor -- start 2024-03-29T10:12:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.843+0000 7fdb83400700 1 -- start start 2024-03-29T10:12:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb83400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdb7c0d05e0 0x7fdb7c0d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb83400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb7c0b6990 0x7fdb7c0b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb83400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7c0d0f80 con 0x7fdb7c15da50 2024-03-29T10:12:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb83400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb7c0b95d0 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb83400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c15de30 con 0x7fdb7c0b6990 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb80a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdb7c15da50 0x7fdb7c0ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54376/0 (socket says 172.21.15.12:54376) 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb80a00700 1 -- 172.21.15.12:0/3528824231 learned_addr learned my addr 172.21.15.12:0/3528824231 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb7be00700 1 --2- 172.21.15.12:0/3528824231 >> v2:172.21.15.159:3300/0 conn(0x7fdb7c0d05e0 0x7fdb7c0d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb81400700 1 --2- 172.21.15.12:0/3528824231 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb7c0b6990 0x7fdb7c0b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb7b400700 1 -- 172.21.15.12:0/3528824231 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693108626 0 0) 0x7fdb7c0d0f80 con 0x7fdb7c15da50 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb7b400700 1 -- 172.21.15.12:0/3528824231 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb640035b0 con 0x7fdb7c15da50 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb7be00700 1 -- 172.21.15.12:0/3528824231 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb7c0b6990 msgr2=0x7fdb7c0b6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.845+0000 7fdb7be00700 1 --2- 172.21.15.12:0/3528824231 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb7c0b6990 0x7fdb7c0b6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.846+0000 7fdb7be00700 1 -- 172.21.15.12:0/3528824231 >> v1:172.21.15.12:6789/0 conn(0x7fdb7c15da50 legacy=0x7fdb7c0ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.846+0000 7fdb7be00700 1 -- 172.21.15.12:0/3528824231 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7c0d24e0 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.846+0000 7fdb7be00700 1 --2- 172.21.15.12:0/3528824231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb7c0d05e0 0x7fdb7c0d0a40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009840 tx=0x7fdb68009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=716873b93b6d29f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:24.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.846+0000 7fdb7b400700 1 -- 172.21.15.12:0/3528824231 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb68011830 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.846+0000 7fdb7b400700 1 -- 172.21.15.12:0/3528824231 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb68011990 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.846+0000 7fdb7b400700 1 -- 172.21.15.12:0/3528824231 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb68011d20 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.847+0000 7fdb83400700 1 -- 172.21.15.12:0/3528824231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb7c0d05e0 msgr2=0x7fdb7c0d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:24.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.847+0000 7fdb83400700 1 --2- 172.21.15.12:0/3528824231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb7c0d05e0 0x7fdb7c0d0a40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009840 tx=0x7fdb68009b50 comp rx=0 tx=0).stop 2024-03-29T10:12:24.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.847+0000 7fdb83400700 1 -- 172.21.15.12:0/3528824231 shutdown_connections 2024-03-29T10:12:24.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.847+0000 7fdb83400700 1 --2- 172.21.15.12:0/3528824231 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb7c0b6990 0x7fdb7c0b6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.847+0000 7fdb83400700 1 --2- 172.21.15.12:0/3528824231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb7c0d05e0 0x7fdb7c0d0a40 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.847+0000 7fdb83400700 1 -- 172.21.15.12:0/3528824231 >> 172.21.15.12:0/3528824231 conn(0x7fdb7c0666a0 msgr2=0x7fdb7c066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:24.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.847+0000 7fdb83400700 1 -- 172.21.15.12:0/3528824231 shutdown_connections 2024-03-29T10:12:24.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.847+0000 7fdb83400700 1 -- 172.21.15.12:0/3528824231 wait complete. 2024-03-29T10:12:24.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.849+0000 7fdb83400700 1 Processor -- start 2024-03-29T10:12:24.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.849+0000 7fdb83400700 1 -- start start 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb83400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb7c0b6990 0x7fdb7c1575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb83400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb7c0d05e0 0x7fdb7c15df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb83400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb7c15da50 0x7fdb7c152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb83400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c15f360 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb83400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c15e4b0 con 0x7fdb7c0b6990 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb83400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c152f70 con 0x7fdb7c15da50 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb7be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb7c0d05e0 0x7fdb7c15df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb80a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb7c0b6990 0x7fdb7c1575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb7be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb7c0d05e0 0x7fdb7c15df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40298/0 (socket says 172.21.15.12:40298) 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb7be00700 1 -- 172.21.15.12:0/2076899070 learned_addr learned my addr 172.21.15.12:0/2076899070 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb81400700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb7c15da50 0x7fdb7c152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:24.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb7be00700 1 -- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb7c15da50 msgr2=0x7fdb7c152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:24.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb7be00700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb7c15da50 0x7fdb7c152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb7be00700 1 -- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb7c0b6990 msgr2=0x7fdb7c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:24.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb7be00700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb7c0b6990 0x7fdb7c1575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:24.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.851+0000 7fdb7be00700 1 -- 172.21.15.12:0/2076899070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb680094a0 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.852+0000 7fdb7be00700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb7c0d05e0 0x7fdb7c15df70 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fdb680090b0 tx=0x7fdb68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:24.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.852+0000 7fdb78c00700 1 -- 172.21.15.12:0/2076899070 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb68004590 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.852+0000 7fdb83400700 1 -- 172.21.15.12:0/2076899070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7c15e6e0 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.852+0000 7fdb78c00700 1 -- 172.21.15.12:0/2076899070 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6801f070 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.852+0000 7fdb78c00700 1 -- 172.21.15.12:0/2076899070 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb680038e0 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.852+0000 7fdb83400700 1 -- 172.21.15.12:0/2076899070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb7c15eca0 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.853+0000 7fdb83400700 1 -- 172.21.15.12:0/2076899070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb7c15f4a0 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.854+0000 7fdb78c00700 1 -- 172.21.15.12:0/2076899070 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fdb68003a40 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.855+0000 7fdb78c00700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdb5c06aed0 0x7fdb5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:24.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.855+0000 7fdb78c00700 1 -- 172.21.15.12:0/2076899070 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fdb68021070 con 0x7fdb7c0d05e0 2024-03-29T10:12:24.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.855+0000 7fdb80a00700 1 -- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdb5c06aed0 msgr2=0x7fdb5c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:24.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.855+0000 7fdb80a00700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdb5c06aed0 0x7fdb5c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:24.864+0000 7fdb78c00700 1 -- 172.21.15.12:0/2076899070 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdb7c15f4a0 con 0x7fdb7c0d05e0 2024-03-29T10:12:25.042 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:25.042+0000 7ff3990d7240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:12:25.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.056+0000 7fdb80a00700 1 -- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdb5c06aed0 msgr2=0x7fdb5c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:25.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.056+0000 7fdb80a00700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdb5c06aed0 0x7fdb5c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:25.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.125+0000 7fdb83400700 1 -- 172.21.15.12:0/2076899070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb7c15f4a0 con 0x7fdb7c0d05e0 2024-03-29T10:12:25.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.125+0000 7fdb78c00700 1 -- 172.21.15.12:0/2076899070 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fdb7c15f4a0 con 0x7fdb7c0d05e0 2024-03-29T10:12:25.125 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:25.125 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6452,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2137480356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2137480356},{"type":"v1","addr":"172.21.15.159:6836","nonce":2137480356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":6455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2356039533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2356039533},{"type":"v1","addr":"172.21.15.159:6837","nonce":2356039533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":7153,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2916743006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2916743006},{"type":"v1","addr":"172.21.15.12:6835","nonce":2916743006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:58.634187+0000","modified":"2024-03-29T10:11:02.111942+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7410":{"gid":7410,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3023331068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3023331068},{"type":"v1","addr":"172.21.15.12:6834","nonce":3023331068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-29T10:12:25.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 -- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdb5c06aed0 msgr2=0x7fdb5c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:25.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdb5c06aed0 0x7fdb5c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 -- 172.21.15.12:0/2076899070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb7c0d05e0 msgr2=0x7fdb7c15df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:25.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb7c0d05e0 0x7fdb7c15df70 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fdb680090b0 tx=0x7fdb68004750 comp rx=0 tx=0).stop 2024-03-29T10:12:25.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 -- 172.21.15.12:0/2076899070 shutdown_connections 2024-03-29T10:12:25.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdb5c06aed0 0x7fdb5c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb7c15da50 0x7fdb7c152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb7c0b6990 0x7fdb7c1575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 --2- 172.21.15.12:0/2076899070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb7c0d05e0 0x7fdb7c15df70 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 -- 172.21.15.12:0/2076899070 >> 172.21.15.12:0/2076899070 conn(0x7fdb7c0666a0 msgr2=0x7fdb7c0d36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:25.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 -- 172.21.15.12:0/2076899070 shutdown_connections 2024-03-29T10:12:25.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.128+0000 7fdb83400700 1 -- 172.21.15.12:0/2076899070 wait complete. 2024-03-29T10:12:25.130 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 46 2024-03-29T10:12:25.147 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:12:25.191 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:25.191+0000 7f3f29e3c240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:12:25.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.347+0000 7f09bb800700 1 Processor -- start 2024-03-29T10:12:25.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.347+0000 7f09bb800700 1 -- start start 2024-03-29T10:12:25.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.348+0000 7f09bb800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f09b415a5a0 0x7f09b4099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.348+0000 7f09bb800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b406e2f0 0x7f09b40ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09bb800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b406e780 con 0x7f09b40cc6f0 2024-03-29T10:12:25.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09bb800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b4072440 con 0x7f09b415a5a0 2024-03-29T10:12:25.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09bb800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b406f320 con 0x7f09b406e2f0 2024-03-29T10:12:25.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09b8e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f09b415a5a0 0x7f09b4099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:25.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09b9800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b406e2f0 0x7f09b40ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:25.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09b8e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f09b415a5a0 0x7f09b4099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44236/0 (socket says 172.21.15.12:44236) 2024-03-29T10:12:25.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09b8e00700 1 -- 172.21.15.12:0/463302492 learned_addr learned my addr 172.21.15.12:0/463302492 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:25.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09b8e00700 1 -- 172.21.15.12:0/463302492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b406e2f0 msgr2=0x7f09b40ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:25.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09b8e00700 1 --2- 172.21.15.12:0/463302492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b406e2f0 0x7f09b40ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09b8e00700 1 -- 172.21.15.12:0/463302492 >> v1:172.21.15.159:6789/0 conn(0x7f09b40cc6f0 legacy=0x7f09b406dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:25.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.350+0000 7f09b8e00700 1 -- 172.21.15.12:0/463302492 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b406e9f0 con 0x7f09b415a5a0 2024-03-29T10:12:25.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.350+0000 7f09b9800700 1 --2- 172.21.15.12:0/463302492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b406e2f0 0x7f09b40ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:25.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.349+0000 7f09b3400700 1 -- 172.21.15.12:0/463302492 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540047288 0 0) 0x7f09b406e780 con 0x7f09b40cc6f0 2024-03-29T10:12:25.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.350+0000 7f09b8e00700 1 --2- 172.21.15.12:0/463302492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b415a5a0 0x7f09b4099a60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f09a80098f0 tx=0x7f09a8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e34d3770b7b4534f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:25.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.350+0000 7f09b3400700 1 -- 172.21.15.12:0/463302492 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8011830 con 0x7f09b415a5a0 2024-03-29T10:12:25.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.350+0000 7f09b3400700 1 -- 172.21.15.12:0/463302492 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a8011990 con 0x7f09b415a5a0 2024-03-29T10:12:25.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.350+0000 7f09b3400700 1 -- 172.21.15.12:0/463302492 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8011d20 con 0x7f09b415a5a0 2024-03-29T10:12:25.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.351+0000 7f09bb800700 1 -- 172.21.15.12:0/463302492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b415a5a0 msgr2=0x7f09b4099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:25.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.351+0000 7f09bb800700 1 --2- 172.21.15.12:0/463302492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b415a5a0 0x7f09b4099a60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f09a80098f0 tx=0x7f09a8009c00 comp rx=0 tx=0).stop 2024-03-29T10:12:25.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.351+0000 7f09bb800700 1 -- 172.21.15.12:0/463302492 shutdown_connections 2024-03-29T10:12:25.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.351+0000 7f09bb800700 1 --2- 172.21.15.12:0/463302492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b406e2f0 0x7f09b40ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.351+0000 7f09bb800700 1 --2- 172.21.15.12:0/463302492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b415a5a0 0x7f09b4099a60 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.351+0000 7f09bb800700 1 -- 172.21.15.12:0/463302492 >> 172.21.15.12:0/463302492 conn(0x7f09b40b5ab0 msgr2=0x7f09b40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:25.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.351+0000 7f09bb800700 1 -- 172.21.15.12:0/463302492 shutdown_connections 2024-03-29T10:12:25.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.351+0000 7f09bb800700 1 -- 172.21.15.12:0/463302492 wait complete. 2024-03-29T10:12:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.353+0000 7f09bb800700 1 Processor -- start 2024-03-29T10:12:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.353+0000 7f09bb800700 1 -- start start 2024-03-29T10:12:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09bb800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b406e2f0 0x7f09b415d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09bb800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b40cc6f0 0x7f09b41f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09bb800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b415a5a0 0x7f09b40b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09bb800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b40b2270 con 0x7f09b415a5a0 2024-03-29T10:12:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09bb800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b415aba0 con 0x7f09b406e2f0 2024-03-29T10:12:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09bb800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b415ae80 con 0x7f09b40cc6f0 2024-03-29T10:12:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09b9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b415a5a0 0x7f09b40b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09b9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b415a5a0 0x7f09b40b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40304/0 (socket says 172.21.15.12:40304) 2024-03-29T10:12:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09b9800700 1 -- 172.21.15.12:0/4120060379 learned_addr learned my addr 172.21.15.12:0/4120060379 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09b8e00700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b406e2f0 0x7f09b415d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09b3e00700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b40cc6f0 0x7f09b41f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09b9800700 1 -- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b40cc6f0 msgr2=0x7f09b41f27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09b9800700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b40cc6f0 0x7f09b41f27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09b9800700 1 -- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b406e2f0 msgr2=0x7f09b415d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09b9800700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b406e2f0 0x7f09b415d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.354+0000 7f09b9800700 1 -- 172.21.15.12:0/4120060379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac009a70 con 0x7f09b415a5a0 2024-03-29T10:12:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.355+0000 7f09b9800700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b415a5a0 0x7f09b40b1a40 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00efc0 tx=0x7f09ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.355+0000 7f09b0c00700 1 -- 172.21.15.12:0/4120060379 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac00ca90 con 0x7f09b415a5a0 2024-03-29T10:12:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.355+0000 7f09b0c00700 1 -- 172.21.15.12:0/4120060379 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac00f040 con 0x7f09b415a5a0 2024-03-29T10:12:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.355+0000 7f09b0c00700 1 -- 172.21.15.12:0/4120060379 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac00ce10 con 0x7f09b415a5a0 2024-03-29T10:12:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.355+0000 7f09bb800700 1 -- 172.21.15.12:0/4120060379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a8009550 con 0x7f09b415a5a0 2024-03-29T10:12:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.355+0000 7f09bb800700 1 -- 172.21.15.12:0/4120060379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b40b1f80 con 0x7f09b415a5a0 2024-03-29T10:12:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.356+0000 7f09bb800700 1 -- 172.21.15.12:0/4120060379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b4067720 con 0x7f09b415a5a0 2024-03-29T10:12:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.357+0000 7f09b0c00700 1 -- 172.21.15.12:0/4120060379 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f09ac014480 con 0x7f09b415a5a0 2024-03-29T10:12:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.358+0000 7f09b0c00700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099406aed0 0x7f099406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.358+0000 7f09b0c00700 1 -- 172.21.15.12:0/4120060379 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f09ac08a130 con 0x7f09b415a5a0 2024-03-29T10:12:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.358+0000 7f09b8e00700 1 -- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099406aed0 msgr2=0x7f099406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.358+0000 7f09b8e00700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099406aed0 0x7f099406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:25.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.367+0000 7f09b0c00700 1 -- 172.21.15.12:0/4120060379 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09ac054c00 con 0x7f09b415a5a0 2024-03-29T10:12:25.402 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:25.402+0000 7f3f29e3c240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:12:25.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.559+0000 7f09b8e00700 1 -- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099406aed0 msgr2=0x7f099406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:25.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.559+0000 7f09b8e00700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099406aed0 0x7f099406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:25.565 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:25.565+0000 7ff3990d7240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:12:25.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.587+0000 7f09bb800700 1 -- 172.21.15.12:0/4120060379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09b415a980 con 0x7f09b415a5a0 2024-03-29T10:12:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.588+0000 7f09b0c00700 1 -- 172.21.15.12:0/4120060379 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16650 (secure 0 0 0) 0x7f09ac009d70 con 0x7f09b415a5a0 2024-03-29T10:12:25.588 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:25.588 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":111,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:12:17.179326+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:57.064406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:57.607254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3e","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6833/43444":"2024-03-30T10:11:15.808228+0000","172.21.15.159:6832/43444":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/430021803":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2229015445":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2018718607":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/663205968":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/1058856906":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2814107463":"2024-03-30T10:11:15.808228+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.590+0000 7f09bb800700 1 -- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099406aed0 msgr2=0x7f099406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.590+0000 7f09bb800700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099406aed0 0x7f099406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.590+0000 7f09bb800700 1 -- 172.21.15.12:0/4120060379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b415a5a0 msgr2=0x7f09b40b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.590+0000 7f09bb800700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b415a5a0 0x7f09b40b1a40 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00efc0 tx=0x7f09ac00c3a0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.591+0000 7f09bb800700 1 -- 172.21.15.12:0/4120060379 shutdown_connections 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.591+0000 7f09bb800700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f099406aed0 0x7f099406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.591+0000 7f09bb800700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09b415a5a0 0x7f09b40b1a40 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.591+0000 7f09bb800700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09b406e2f0 0x7f09b415d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.591+0000 7f09bb800700 1 --2- 172.21.15.12:0/4120060379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09b40cc6f0 0x7f09b41f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.591+0000 7f09bb800700 1 -- 172.21.15.12:0/4120060379 >> 172.21.15.12:0/4120060379 conn(0x7f09b40b5ab0 msgr2=0x7f09b41579e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.591+0000 7f09bb800700 1 -- 172.21.15.12:0/4120060379 shutdown_connections 2024-03-29T10:12:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.591+0000 7f09bb800700 1 -- 172.21.15.12:0/4120060379 wait complete. 2024-03-29T10:12:25.609 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T10:12:25.609 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:12:25.789 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:25.790+0000 7ff3990d7240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:12:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.810+0000 7f9421200700 1 Processor -- start 2024-03-29T10:12:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.810+0000 7f9421200700 1 -- start start 2024-03-29T10:12:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.811+0000 7f9421200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f941c15da50 0x7f941c0ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.811+0000 7f9421200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f941c0cec70 0x7f941c0d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.811+0000 7f9421200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f941c0d24e0 con 0x7f941c0d0b20 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.811+0000 7f9421200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941c0d0fd0 con 0x7f941c0cec70 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.811+0000 7f9421200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941c0b95d0 con 0x7f941c15da50 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.811+0000 7f9419400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f941c0cec70 0x7f941c0d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.811+0000 7f9419400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f941c0cec70 0x7f941c0d05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40318/0 (socket says 172.21.15.12:40318) 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.811+0000 7f9419400700 1 -- 172.21.15.12:0/1164407502 learned_addr learned my addr 172.21.15.12:0/1164407502 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.811+0000 7f9419e00700 1 --2- 172.21.15.12:0/1164407502 >> v2:172.21.15.159:3300/0 conn(0x7f941c15da50 0x7f941c0ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.812+0000 7f9419400700 1 -- 172.21.15.12:0/1164407502 >> v2:172.21.15.159:3300/0 conn(0x7f941c15da50 msgr2=0x7f941c0ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.812+0000 7f9419400700 1 --2- 172.21.15.12:0/1164407502 >> v2:172.21.15.159:3300/0 conn(0x7f941c15da50 0x7f941c0ce730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.812+0000 7f9419400700 1 -- 172.21.15.12:0/1164407502 >> v1:172.21.15.159:6789/0 conn(0x7f941c0d0b20 legacy=0x7f941c0b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.812+0000 7f9419400700 1 -- 172.21.15.12:0/1164407502 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941c0d27c0 con 0x7f941c0cec70 2024-03-29T10:12:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.812+0000 7f9419400700 1 --2- 172.21.15.12:0/1164407502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f941c0cec70 0x7f941c0d05e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f941c0cfbb0 tx=0x7f9408009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd6265e0b89e276 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.812+0000 7f9418a00700 1 -- 172.21.15.12:0/1164407502 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408011830 con 0x7f941c0cec70 2024-03-29T10:12:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.813+0000 7f9418a00700 1 -- 172.21.15.12:0/1164407502 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408011990 con 0x7f941c0cec70 2024-03-29T10:12:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.813+0000 7f9418a00700 1 -- 172.21.15.12:0/1164407502 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408011d20 con 0x7f941c0cec70 2024-03-29T10:12:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.813+0000 7f9421200700 1 -- 172.21.15.12:0/1164407502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f941c0cec70 msgr2=0x7f941c0d05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.813+0000 7f9421200700 1 --2- 172.21.15.12:0/1164407502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f941c0cec70 0x7f941c0d05e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f941c0cfbb0 tx=0x7f9408009bd0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.813+0000 7f9421200700 1 -- 172.21.15.12:0/1164407502 shutdown_connections 2024-03-29T10:12:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.813+0000 7f9421200700 1 --2- 172.21.15.12:0/1164407502 >> v2:172.21.15.159:3300/0 conn(0x7f941c15da50 0x7f941c0ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.813+0000 7f9421200700 1 --2- 172.21.15.12:0/1164407502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f941c0cec70 0x7f941c0d05e0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.813+0000 7f9421200700 1 -- 172.21.15.12:0/1164407502 >> 172.21.15.12:0/1164407502 conn(0x7f941c0666a0 msgr2=0x7f941c066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:25.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.813+0000 7f9421200700 1 -- 172.21.15.12:0/1164407502 shutdown_connections 2024-03-29T10:12:25.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.813+0000 7f9421200700 1 -- 172.21.15.12:0/1164407502 wait complete. 2024-03-29T10:12:25.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.815+0000 7f9421200700 1 Processor -- start 2024-03-29T10:12:25.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.815+0000 7f9421200700 1 -- start start 2024-03-29T10:12:25.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f9421200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f941c0cec70 0x7f941c1538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f9421200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f941c0d0b20 0x7f941c0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f9421200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f941c15da50 0x7f941c0c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f9421200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c0c7c80 con 0x7f941c15da50 2024-03-29T10:12:25.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f9421200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c0c7f60 con 0x7f941c0d0b20 2024-03-29T10:12:25.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f9421200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941c0c8240 con 0x7f941c0cec70 2024-03-29T10:12:25.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f941a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f941c15da50 0x7f941c0c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:25.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f9419400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f941c0d0b20 0x7f941c0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f9419e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f941c0cec70 0x7f941c1538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f9419400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f941c0d0b20 0x7f941c0c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44256/0 (socket says 172.21.15.12:44256) 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.816+0000 7f9419400700 1 -- 172.21.15.12:0/2440721009 learned_addr learned my addr 172.21.15.12:0/2440721009 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.817+0000 7f9419400700 1 -- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f941c0cec70 msgr2=0x7f941c1538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.817+0000 7f9419400700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f941c0cec70 0x7f941c1538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.817+0000 7f9419400700 1 -- 172.21.15.12:0/2440721009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f941c15da50 msgr2=0x7f941c0c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.817+0000 7f9419400700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f941c15da50 0x7f941c0c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.817+0000 7f9419400700 1 -- 172.21.15.12:0/2440721009 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9408009550 con 0x7f941c0d0b20 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.817+0000 7f9419e00700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f941c0cec70 0x7f941c1538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.817+0000 7f9419400700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f941c0d0b20 0x7f941c0c8d20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9408000c00 tx=0x7f9408004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:25.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.817+0000 7f9412000700 1 -- 172.21.15.12:0/2440721009 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408004380 con 0x7f941c0d0b20 2024-03-29T10:12:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.817+0000 7f9412000700 1 -- 172.21.15.12:0/2440721009 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940801f070 con 0x7f941c0d0b20 2024-03-29T10:12:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.817+0000 7f9412000700 1 -- 172.21.15.12:0/2440721009 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94080036a0 con 0x7f941c0d0b20 2024-03-29T10:12:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.818+0000 7f9421200700 1 -- 172.21.15.12:0/2440721009 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941c0c8520 con 0x7f941c0d0b20 2024-03-29T10:12:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.818+0000 7f9421200700 1 -- 172.21.15.12:0/2440721009 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941c0c9260 con 0x7f941c0d0b20 2024-03-29T10:12:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.819+0000 7f9421200700 1 -- 172.21.15.12:0/2440721009 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941c0c8a40 con 0x7f941c0d0b20 2024-03-29T10:12:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.820+0000 7f9412000700 1 -- 172.21.15.12:0/2440721009 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 53) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f9408021030 con 0x7f941c0d0b20 2024-03-29T10:12:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.821+0000 7f9412000700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f93fc06aed0 0x7f93fc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.821+0000 7f9412000700 1 -- 172.21.15.12:0/2440721009 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f940808ae10 con 0x7f941c0d0b20 2024-03-29T10:12:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.821+0000 7f9419e00700 1 -- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f93fc06aed0 msgr2=0x7f93fc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.821+0000 7f9419e00700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f93fc06aed0 0x7f93fc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.824+0000 7f9412000700 1 -- 172.21.15.12:0/2440721009 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9408055b80 con 0x7f941c0d0b20 2024-03-29T10:12:25.844 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:25.844+0000 7f3f29e3c240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:12:25.954 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:25.954+0000 7f3f29e3c240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:12:25.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.967+0000 7f9412000700 1 -- 172.21.15.12:0/2440721009 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mgrmap(e 54) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f9408055f50 con 0x7f941c0d0b20 2024-03-29T10:12:25.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.967+0000 7f9412000700 1 -- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f93fc06aed0 msgr2=0x7f93fc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:25.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:25.967+0000 7f9412000700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f93fc06aed0 0x7f93fc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.029 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:26.029+0000 7f3e9d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:26.030 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:26.029+0000 7f3e9d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:26.030 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:26.029+0000 7f3e9d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:26.030 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:26.029+0000 7f3e9d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:26.030 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:26.029+0000 7f3e9d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:26.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:26.029+0000 7f3e99600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:26.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:26.030+0000 7f3e99600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:26.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:26.030+0000 7f3e99600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:26.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:26.030+0000 7f3e99600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:26.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:26.030+0000 7f3e99600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:12:26.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.094+0000 7f9421200700 1 -- 172.21.15.12:0/2440721009 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f941c0c8a40 con 0x7f941c0d0b20 2024-03-29T10:12:26.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.095+0000 7f9412000700 1 -- 172.21.15.12:0/2440721009 <== mon.1 v2:172.21.15.159:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f941c0c8a40 con 0x7f941c0d0b20 2024-03-29T10:12:26.095 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:26.095 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6452,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2137480356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2137480356},{"type":"v1","addr":"172.21.15.159:6836","nonce":2137480356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":6455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2356039533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2356039533},{"type":"v1","addr":"172.21.15.159:6837","nonce":2356039533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":7153,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2916743006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2916743006},{"type":"v1","addr":"172.21.15.12:6835","nonce":2916743006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:58.634187+0000","modified":"2024-03-29T10:11:02.111942+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7410":{"gid":7410,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3023331068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3023331068},{"type":"v1","addr":"172.21.15.12:6834","nonce":3023331068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-29T10:12:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.097+0000 7f9421200700 1 -- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f941c0d0b20 msgr2=0x7f941c0c8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.097+0000 7f9421200700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f941c0d0b20 0x7f941c0c8d20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9408000c00 tx=0x7f9408004750 comp rx=0 tx=0).stop 2024-03-29T10:12:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.098+0000 7f9421200700 1 -- 172.21.15.12:0/2440721009 shutdown_connections 2024-03-29T10:12:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.098+0000 7f9421200700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f93fc06aed0 0x7f93fc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.098+0000 7f9421200700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f941c15da50 0x7f941c0c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.098+0000 7f9421200700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f941c0cec70 0x7f941c1538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.098+0000 7f9421200700 1 --2- 172.21.15.12:0/2440721009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f941c0d0b20 0x7f941c0c8d20 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.098+0000 7f9421200700 1 -- 172.21.15.12:0/2440721009 >> 172.21.15.12:0/2440721009 conn(0x7f941c0666a0 msgr2=0x7f941c0b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.098+0000 7f9421200700 1 -- 172.21.15.12:0/2440721009 shutdown_connections 2024-03-29T10:12:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.098+0000 7f9421200700 1 -- 172.21.15.12:0/2440721009 wait complete. 2024-03-29T10:12:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 46 2024-03-29T10:12:26.117 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:12:26.260 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:26.260+0000 7ff3990d7240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:12:26.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.321+0000 7f738ea00700 1 Processor -- start 2024-03-29T10:12:26.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.321+0000 7f738ea00700 1 -- start start 2024-03-29T10:12:26.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.323+0000 7f738ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73880ceec0 0x7f73881612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:26.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.323+0000 7f738ea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73880d30d0 con 0x7f7388161800 2024-03-29T10:12:26.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.323+0000 7f738ea00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73880b6840 con 0x7f738815d640 2024-03-29T10:12:26.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.323+0000 7f738ea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7388161c90 con 0x7f73880ceec0 2024-03-29T10:12:26.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.323+0000 7f738ca00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7388161800 0x7f73880b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54380/0 (socket says 172.21.15.12:54380) 2024-03-29T10:12:26.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.323+0000 7f738ca00700 1 -- 172.21.15.12:0/1198976707 learned_addr learned my addr 172.21.15.12:0/1198976707 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.323+0000 7f7387400700 1 --2- 172.21.15.12:0/1198976707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73880ceec0 0x7f73881612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.324+0000 7f7386a00700 1 -- 172.21.15.12:0/1198976707 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140886665 0 0) 0x7f73880b6840 con 0x7f738815d640 2024-03-29T10:12:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.324+0000 7f7386a00700 1 -- 172.21.15.12:0/1198976707 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f736c0035b0 con 0x7f738815d640 2024-03-29T10:12:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.324+0000 7f7386a00700 1 -- 172.21.15.12:0/1198976707 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861095362 0 0) 0x7f73880d30d0 con 0x7f7388161800 2024-03-29T10:12:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.324+0000 7f7386a00700 1 -- 172.21.15.12:0/1198976707 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73880b6840 con 0x7f7388161800 2024-03-29T10:12:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.324+0000 7f7387400700 1 -- 172.21.15.12:0/1198976707 >> v1:172.21.15.159:6789/0 conn(0x7f738815d640 legacy=0x7f73880d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.324+0000 7f7387400700 1 -- 172.21.15.12:0/1198976707 >> v1:172.21.15.12:6789/0 conn(0x7f7388161800 legacy=0x7f73880b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.324+0000 7f7387400700 1 -- 172.21.15.12:0/1198976707 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73880d2b30 con 0x7f73880ceec0 2024-03-29T10:12:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.324+0000 7f7387400700 1 --2- 172.21.15.12:0/1198976707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73880ceec0 0x7f73881612c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7374009870 tx=0x7f7374009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c945597d2ada3413 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.324+0000 7f7386a00700 1 -- 172.21.15.12:0/1198976707 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374011830 con 0x7f73880ceec0 2024-03-29T10:12:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.325+0000 7f7386a00700 1 -- 172.21.15.12:0/1198976707 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7374011990 con 0x7f73880ceec0 2024-03-29T10:12:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.325+0000 7f7386a00700 1 -- 172.21.15.12:0/1198976707 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374011d20 con 0x7f73880ceec0 2024-03-29T10:12:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.325+0000 7f738ea00700 1 -- 172.21.15.12:0/1198976707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73880ceec0 msgr2=0x7f73881612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.325+0000 7f738ea00700 1 --2- 172.21.15.12:0/1198976707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73880ceec0 0x7f73881612c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7374009870 tx=0x7f7374009b80 comp rx=0 tx=0).stop 2024-03-29T10:12:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.325+0000 7f738ea00700 1 -- 172.21.15.12:0/1198976707 shutdown_connections 2024-03-29T10:12:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.325+0000 7f738ea00700 1 --2- 172.21.15.12:0/1198976707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73880ceec0 0x7f73881612c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.325+0000 7f738ea00700 1 -- 172.21.15.12:0/1198976707 >> 172.21.15.12:0/1198976707 conn(0x7f738808aff0 msgr2=0x7f73880d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.325+0000 7f738ea00700 1 -- 172.21.15.12:0/1198976707 shutdown_connections 2024-03-29T10:12:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.325+0000 7f738ea00700 1 -- 172.21.15.12:0/1198976707 wait complete. 2024-03-29T10:12:26.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.327+0000 7f738ea00700 1 Processor -- start 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.327+0000 7f738ea00700 1 -- start start 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.328+0000 7f738ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73880ceec0 0x7f738814fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.328+0000 7f738ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f738815d640 0x7f738809d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.328+0000 7f738ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388161800 0x7f738806ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.328+0000 7f738ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738809e820 con 0x7f7388161800 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.328+0000 7f738ea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7388150380 con 0x7f73880ceec0 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.328+0000 7f738ea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738809dba0 con 0x7f738815d640 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f738ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388161800 0x7f738806ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f7387e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73880ceec0 0x7f738814fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f7387400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f738815d640 0x7f738809d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f738ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388161800 0x7f738806ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40336/0 (socket says 172.21.15.12:40336) 2024-03-29T10:12:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f7387e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73880ceec0 0x7f738814fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44264/0 (socket says 172.21.15.12:44264) 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f738ca00700 1 -- 172.21.15.12:0/947086960 learned_addr learned my addr 172.21.15.12:0/947086960 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f7387400700 1 -- 172.21.15.12:0/947086960 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73880ceec0 msgr2=0x7f738814fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f7387400700 1 --2- 172.21.15.12:0/947086960 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73880ceec0 0x7f738814fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f7387400700 1 -- 172.21.15.12:0/947086960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388161800 msgr2=0x7f738806ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f7387400700 1 --2- 172.21.15.12:0/947086960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388161800 0x7f738806ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f7387400700 1 -- 172.21.15.12:0/947086960 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73740094a0 con 0x7f738815d640 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f738ca00700 1 --2- 172.21.15.12:0/947086960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388161800 0x7f738806ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.329+0000 7f7387400700 1 --2- 172.21.15.12:0/947086960 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f738815d640 0x7f738809d230 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f73880c7050 tx=0x7f7374004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.330+0000 7f737fe00700 1 -- 172.21.15.12:0/947086960 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737401a8f0 con 0x7f738815d640 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.330+0000 7f737fe00700 1 -- 172.21.15.12:0/947086960 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737401aa70 con 0x7f738815d640 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.330+0000 7f737fe00700 1 -- 172.21.15.12:0/947086960 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737401ae20 con 0x7f738815d640 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.330+0000 7f738ea00700 1 -- 172.21.15.12:0/947086960 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738809de80 con 0x7f738815d640 2024-03-29T10:12:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.330+0000 7f738ea00700 1 -- 172.21.15.12:0/947086960 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738809e440 con 0x7f738815d640 2024-03-29T10:12:26.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.331+0000 7f738ea00700 1 -- 172.21.15.12:0/947086960 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7388067720 con 0x7f738815d640 2024-03-29T10:12:26.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.332+0000 7f737fe00700 1 -- 172.21.15.12:0/947086960 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 54) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f737402d450 con 0x7f738815d640 2024-03-29T10:12:26.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.333+0000 7f737fe00700 1 -- 172.21.15.12:0/947086960 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f737408a390 con 0x7f738815d640 2024-03-29T10:12:26.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.337+0000 7f737fe00700 1 -- 172.21.15.12:0/947086960 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7374055720 con 0x7f738815d640 2024-03-29T10:12:26.385 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:26.386+0000 7ff3990d7240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:12:26.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.607+0000 7f738ea00700 1 -- 172.21.15.12:0/947086960 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73880b5c70 con 0x7f738815d640 2024-03-29T10:12:26.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.608+0000 7f737fe00700 1 -- 172.21.15.12:0/947086960 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f7374058d40 con 0x7f738815d640 2024-03-29T10:12:26.607 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:26.608 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6452,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2137480356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2137480356},{"type":"v1","addr":"172.21.15.159:6836","nonce":2137480356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":6455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2356039533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2356039533},{"type":"v1","addr":"172.21.15.159:6837","nonce":2356039533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":7153,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2916743006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2916743006},{"type":"v1","addr":"172.21.15.12:6835","nonce":2916743006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:10:58.634187+0000","modified":"2024-03-29T10:11:02.111942+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7410":{"gid":7410,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3023331068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3023331068},{"type":"v1","addr":"172.21.15.12:6834","nonce":3023331068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-29T10:12:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.610+0000 7f738ea00700 1 -- 172.21.15.12:0/947086960 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f738815d640 msgr2=0x7f738809d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.610+0000 7f738ea00700 1 --2- 172.21.15.12:0/947086960 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f738815d640 0x7f738809d230 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f73880c7050 tx=0x7f7374004750 comp rx=0 tx=0).stop 2024-03-29T10:12:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.610+0000 7f738ea00700 1 -- 172.21.15.12:0/947086960 shutdown_connections 2024-03-29T10:12:26.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.610+0000 7f738ea00700 1 --2- 172.21.15.12:0/947086960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388161800 0x7f738806ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.610+0000 7f738ea00700 1 --2- 172.21.15.12:0/947086960 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73880ceec0 0x7f738814fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.610+0000 7f738ea00700 1 --2- 172.21.15.12:0/947086960 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f738815d640 0x7f738809d230 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.610+0000 7f738ea00700 1 -- 172.21.15.12:0/947086960 >> 172.21.15.12:0/947086960 conn(0x7f738808aff0 msgr2=0x7f73880b6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.610+0000 7f738ea00700 1 -- 172.21.15.12:0/947086960 shutdown_connections 2024-03-29T10:12:26.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.610+0000 7f738ea00700 1 -- 172.21.15.12:0/947086960 wait complete. 2024-03-29T10:12:26.612 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 46 2024-03-29T10:12:26.628 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:12:26.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.830+0000 7f3a56400700 1 Processor -- start 2024-03-29T10:12:26.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.830+0000 7f3a56400700 1 -- start start 2024-03-29T10:12:26.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.831+0000 7f3a56400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a5015b5d0 0x7f3a500ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:26.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.831+0000 7f3a56400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3a500cb3f0 0x7f3a500ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:26.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.831+0000 7f3a56400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5015fd50 con 0x7f3a500ae930 2024-03-29T10:12:26.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.831+0000 7f3a56400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5015b9b0 con 0x7f3a500cb3f0 2024-03-29T10:12:26.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.831+0000 7f3a56400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5015f140 con 0x7f3a5015b5d0 2024-03-29T10:12:26.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.832+0000 7f3a4ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3a500cb3f0 0x7f3a500ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:26.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.832+0000 7f3a4f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a5015b5d0 0x7f3a500ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.832+0000 7f3a4f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a5015b5d0 0x7f3a500ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:59654/0 (socket says 172.21.15.12:59654) 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.832+0000 7f3a4fe00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f3a500ae930 0x7f3a50140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:33318/0 (socket says 172.21.15.12:33318) 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.832+0000 7f3a4f400700 1 -- 172.21.15.12:0/453628808 learned_addr learned my addr 172.21.15.12:0/453628808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.832+0000 7f3a4ea00700 1 --2- 172.21.15.12:0/453628808 >> v2:172.21.15.159:3300/0 conn(0x7f3a500cb3f0 0x7f3a500ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44270/0 (socket says 172.21.15.12:44270) 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.832+0000 7f3a4f400700 1 -- 172.21.15.12:0/453628808 >> v2:172.21.15.159:3300/0 conn(0x7f3a500cb3f0 msgr2=0x7f3a500ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.832+0000 7f3a4f400700 1 --2- 172.21.15.12:0/453628808 >> v2:172.21.15.159:3300/0 conn(0x7f3a500cb3f0 0x7f3a500ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.832+0000 7f3a4f400700 1 -- 172.21.15.12:0/453628808 >> v1:172.21.15.159:6789/0 conn(0x7f3a500ae930 legacy=0x7f3a50140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.832+0000 7f3a4f400700 1 -- 172.21.15.12:0/453628808 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5015f420 con 0x7f3a5015b5d0 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.833+0000 7f3a4f400700 1 --2- 172.21.15.12:0/453628808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a5015b5d0 0x7f3a500ae3f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3a400098f0 tx=0x7f3a40009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8656af7530d9ec29 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.833+0000 7f3a4e000700 1 -- 172.21.15.12:0/453628808 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40011830 con 0x7f3a5015b5d0 2024-03-29T10:12:26.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.833+0000 7f3a4e000700 1 -- 172.21.15.12:0/453628808 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a40011990 con 0x7f3a5015b5d0 2024-03-29T10:12:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.833+0000 7f3a4e000700 1 -- 172.21.15.12:0/453628808 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40011d20 con 0x7f3a5015b5d0 2024-03-29T10:12:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.833+0000 7f3a56400700 1 -- 172.21.15.12:0/453628808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a5015b5d0 msgr2=0x7f3a500ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.833+0000 7f3a56400700 1 --2- 172.21.15.12:0/453628808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a5015b5d0 0x7f3a500ae3f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3a400098f0 tx=0x7f3a40009c00 comp rx=0 tx=0).stop 2024-03-29T10:12:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.833+0000 7f3a56400700 1 -- 172.21.15.12:0/453628808 shutdown_connections 2024-03-29T10:12:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.833+0000 7f3a56400700 1 --2- 172.21.15.12:0/453628808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a5015b5d0 0x7f3a500ae3f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.833+0000 7f3a56400700 1 --2- 172.21.15.12:0/453628808 >> v2:172.21.15.159:3300/0 conn(0x7f3a500cb3f0 0x7f3a500ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.833+0000 7f3a56400700 1 -- 172.21.15.12:0/453628808 >> 172.21.15.12:0/453628808 conn(0x7f3a500cf190 msgr2=0x7f3a500cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.834+0000 7f3a56400700 1 -- 172.21.15.12:0/453628808 shutdown_connections 2024-03-29T10:12:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.834+0000 7f3a56400700 1 -- 172.21.15.12:0/453628808 wait complete. 2024-03-29T10:12:26.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.835+0000 7f3a56400700 1 Processor -- start 2024-03-29T10:12:26.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.835+0000 7f3a56400700 1 -- start start 2024-03-29T10:12:26.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.836+0000 7f3a56400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a500ae930 0x7f3a5014bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:26.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.836+0000 7f3a56400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a5015b5d0 0x7f3a500b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:26.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.836+0000 7f3a56400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a500b3460 0x7f3a50071a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.836+0000 7f3a56400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a500b38d0 con 0x7f3a5015b5d0 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.836+0000 7f3a56400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5014c250 con 0x7f3a500b3460 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.836+0000 7f3a56400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a500709c0 con 0x7f3a500ae930 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.836+0000 7f3a4ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a5015b5d0 0x7f3a500b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a5015b5d0 0x7f3a500b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40350/0 (socket says 172.21.15.12:40350) 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4ea00700 1 -- 172.21.15.12:0/115599461 learned_addr learned my addr 172.21.15.12:0/115599461 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4f400700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a500ae930 0x7f3a5014bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4fe00700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a500b3460 0x7f3a50071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4ea00700 1 -- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a500ae930 msgr2=0x7f3a5014bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4ea00700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a500ae930 0x7f3a5014bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4ea00700 1 -- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a500b3460 msgr2=0x7f3a50071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4ea00700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a500b3460 0x7f3a50071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4ea00700 1 -- 172.21.15.12:0/115599461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a40009550 con 0x7f3a5015b5d0 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4fe00700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a500b3460 0x7f3a50071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a4ea00700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a5015b5d0 0x7f3a500b2f20 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00e9e0 tx=0x7f3a3c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.837+0000 7f3a47400700 1 -- 172.21.15.12:0/115599461 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c009710 con 0x7f3a5015b5d0 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.838+0000 7f3a47400700 1 -- 172.21.15.12:0/115599461 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c00c7f0 con 0x7f3a5015b5d0 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.838+0000 7f3a56400700 1 -- 172.21.15.12:0/115599461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a50070c10 con 0x7f3a5015b5d0 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.838+0000 7f3a47400700 1 -- 172.21.15.12:0/115599461 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c00cb60 con 0x7f3a5015b5d0 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.838+0000 7f3a56400700 1 -- 172.21.15.12:0/115599461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a500711d0 con 0x7f3a5015b5d0 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.839+0000 7f3a56400700 1 -- 172.21.15.12:0/115599461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a50067720 con 0x7f3a5015b5d0 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.840+0000 7f3a47400700 1 -- 172.21.15.12:0/115599461 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 54) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f3a3c00ccc0 con 0x7f3a5015b5d0 2024-03-29T10:12:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.841+0000 7f3a47400700 1 -- 172.21.15.12:0/115599461 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f3a3c0891b0 con 0x7f3a5015b5d0 2024-03-29T10:12:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.845+0000 7f3a47400700 1 -- 172.21.15.12:0/115599461 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3a3c0543a0 con 0x7f3a5015b5d0 2024-03-29T10:12:26.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.971+0000 7f3a47400700 1 -- 172.21.15.12:0/115599461 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 55) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f3a3c053e50 con 0x7f3a5015b5d0 2024-03-29T10:12:26.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.971+0000 7f3a47400700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a30036150 0x7f3a300723c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:26.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.972+0000 7f3a4f400700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a30036150 0x7f3a300723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:26.978+0000 7f3a4f400700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a30036150 0x7f3a300723c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009c40 tx=0x7f3a4001d040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.067+0000 7f3a56400700 1 -- 172.21.15.12:0/115599461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a500c9b60 con 0x7f3a5015b5d0 2024-03-29T10:12:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.068+0000 7f3a47400700 1 -- 172.21.15.12:0/115599461 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+17153 (secure 0 0 0) 0x7f3a3c0739e0 con 0x7f3a5015b5d0 2024-03-29T10:12:27.068 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:27.068 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":112,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:12:25.958666+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:10:57.064406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-29T10:10:57.607254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3e","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1679493112":"2024-03-30T10:12:25.958627+0000","172.21.15.159:0/1057979317":"2024-03-30T10:12:25.958627+0000","172.21.15.159:0/3999703123":"2024-03-30T10:12:25.958627+0000","172.21.15.159:0/2553658157":"2024-03-30T10:12:25.958627+0000","172.21.15.159:0/1876367710":"2024-03-30T10:12:25.958627+0000","172.21.15.159:6833/43444":"2024-03-30T10:11:15.808228+0000","172.21.15.159:6832/43444":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/430021803":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2229015445":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2198159023":"2024-03-30T10:12:25.958627+0000","172.21.15.159:0/253147461":"2024-03-30T10:12:25.958627+0000","172.21.15.159:0/2018718607":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/663205968":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2623916491":"2024-03-30T10:12:25.958627+0000","172.21.15.159:0/1058856906":"2024-03-30T10:11:15.808228+0000","172.21.15.159:0/2814107463":"2024-03-30T10:11:15.808228+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 -- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a30036150 msgr2=0x7f3a300723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a30036150 0x7f3a300723c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009c40 tx=0x7f3a4001d040 comp rx=0 tx=0).stop 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 -- 172.21.15.12:0/115599461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a5015b5d0 msgr2=0x7f3a500b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a5015b5d0 0x7f3a500b2f20 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00e9e0 tx=0x7f3a3c00ecf0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 -- 172.21.15.12:0/115599461 shutdown_connections 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a30036150 0x7f3a300723c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a500b3460 0x7f3a50071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a500ae930 0x7f3a5014bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 --2- 172.21.15.12:0/115599461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a5015b5d0 0x7f3a500b2f20 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 -- 172.21.15.12:0/115599461 >> 172.21.15.12:0/115599461 conn(0x7f3a500cf190 msgr2=0x7f3a5006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 -- 172.21.15.12:0/115599461 shutdown_connections 2024-03-29T10:12:27.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.071+0000 7f3a56400700 1 -- 172.21.15.12:0/115599461 wait complete. 2024-03-29T10:12:27.089 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T10:12:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.289+0000 7f64b8a00700 1 Processor -- start 2024-03-29T10:12:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.289+0000 7f64b8a00700 1 -- start start 2024-03-29T10:12:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.290+0000 7f64b8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f64b40cbb30 0x7f64b40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.290+0000 7f64b8a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f64b40cd020 0x7f64b40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.290+0000 7f64b8a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b40cc450 con 0x7f64b40b2070 2024-03-29T10:12:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.290+0000 7f64b8a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b40cdae0 con 0x7f64b40cbb30 2024-03-29T10:12:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.290+0000 7f64b8a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b40cddc0 con 0x7f64b40cd020 2024-03-29T10:12:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.290+0000 7f64b1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f64b40cbb30 0x7f64b40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.290+0000 7f64b1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f64b40cbb30 0x7f64b40cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40360/0 (socket says 172.21.15.12:40360) 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.290+0000 7f64b1e00700 1 -- 172.21.15.12:0/1829970484 learned_addr learned my addr 172.21.15.12:0/1829970484 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.291+0000 7f64b1400700 1 --2- 172.21.15.12:0/1829970484 >> v2:172.21.15.159:3300/0 conn(0x7f64b40cd020 0x7f64b40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.291+0000 7f64b1e00700 1 -- 172.21.15.12:0/1829970484 >> v2:172.21.15.159:3300/0 conn(0x7f64b40cd020 msgr2=0x7f64b40cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.291+0000 7f64b1e00700 1 --2- 172.21.15.12:0/1829970484 >> v2:172.21.15.159:3300/0 conn(0x7f64b40cd020 0x7f64b40cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.291+0000 7f64b1e00700 1 -- 172.21.15.12:0/1829970484 >> v1:172.21.15.159:6789/0 conn(0x7f64b40b2070 legacy=0x7f64b406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.291+0000 7f64b1e00700 1 -- 172.21.15.12:0/1829970484 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b40ce0a0 con 0x7f64b40cbb30 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.291+0000 7f64b1400700 1 --2- 172.21.15.12:0/1829970484 >> v2:172.21.15.159:3300/0 conn(0x7f64b40cd020 0x7f64b40cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.291+0000 7f64b1e00700 1 --2- 172.21.15.12:0/1829970484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40cbb30 0x7f64b40cbf10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f64b40b1e30 tx=0x7f64a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae53337cf7a49ae server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.291+0000 7f64b0a00700 1 -- 172.21.15.12:0/1829970484 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4004230 con 0x7f64b40cbb30 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.291+0000 7f64b0a00700 1 -- 172.21.15.12:0/1829970484 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a4004390 con 0x7f64b40cbb30 2024-03-29T10:12:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.291+0000 7f64b0a00700 1 -- 172.21.15.12:0/1829970484 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a40036a0 con 0x7f64b40cbb30 2024-03-29T10:12:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.292+0000 7f64b8a00700 1 -- 172.21.15.12:0/1829970484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40cbb30 msgr2=0x7f64b40cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.292+0000 7f64b8a00700 1 --2- 172.21.15.12:0/1829970484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40cbb30 0x7f64b40cbf10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f64b40b1e30 tx=0x7f64a4009bd0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.292+0000 7f64b8a00700 1 -- 172.21.15.12:0/1829970484 shutdown_connections 2024-03-29T10:12:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.292+0000 7f64b8a00700 1 --2- 172.21.15.12:0/1829970484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40cbb30 0x7f64b40cbf10 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.292+0000 7f64b8a00700 1 --2- 172.21.15.12:0/1829970484 >> v2:172.21.15.159:3300/0 conn(0x7f64b40cd020 0x7f64b40cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.292+0000 7f64b8a00700 1 -- 172.21.15.12:0/1829970484 >> 172.21.15.12:0/1829970484 conn(0x7f64b40b0d70 msgr2=0x7f64b40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.292+0000 7f64b8a00700 1 -- 172.21.15.12:0/1829970484 shutdown_connections 2024-03-29T10:12:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.292+0000 7f64b8a00700 1 -- 172.21.15.12:0/1829970484 wait complete. 2024-03-29T10:12:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.294+0000 7f64b8a00700 1 Processor -- start 2024-03-29T10:12:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.294+0000 7f64b8a00700 1 -- start start 2024-03-29T10:12:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40b2070 0x7f64b414b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64b40cbb30 0x7f64b40adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64b40cd020 0x7f64b4071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b40c92c0 con 0x7f64b40b2070 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b8a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b414ba30 con 0x7f64b40cd020 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b40c83f0 con 0x7f64b40cbb30 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40b2070 0x7f64b414b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40b2070 0x7f64b414b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40374/0 (socket says 172.21.15.12:40374) 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b1e00700 1 -- 172.21.15.12:0/1795599276 learned_addr learned my addr 172.21.15.12:0/1795599276 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b2800700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64b40cd020 0x7f64b4071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b1400700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64b40cbb30 0x7f64b40adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.295+0000 7f64b1e00700 1 -- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64b40cbb30 msgr2=0x7f64b40adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64b1e00700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64b40cbb30 0x7f64b40adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64b1e00700 1 -- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64b40cd020 msgr2=0x7f64b4071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64b1e00700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64b40cd020 0x7f64b4071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64b1e00700 1 -- 172.21.15.12:0/1795599276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a4009550 con 0x7f64b40b2070 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64b2800700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64b40cd020 0x7f64b4071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64b1400700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64b40cbb30 0x7f64b40adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64b1e00700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40b2070 0x7f64b414b4f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f64b406c130 tx=0x7f64a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64aa000700 1 -- 172.21.15.12:0/1795599276 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4011630 con 0x7f64b40b2070 2024-03-29T10:12:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64aa000700 1 -- 172.21.15.12:0/1795599276 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a4011790 con 0x7f64b40b2070 2024-03-29T10:12:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64b8a00700 1 -- 172.21.15.12:0/1795599276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b40c8640 con 0x7f64b40b2070 2024-03-29T10:12:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64aa000700 1 -- 172.21.15.12:0/1795599276 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4011b40 con 0x7f64b40b2070 2024-03-29T10:12:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.296+0000 7f64b8a00700 1 -- 172.21.15.12:0/1795599276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b40c8c00 con 0x7f64b40b2070 2024-03-29T10:12:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.297+0000 7f64b8a00700 1 -- 172.21.15.12:0/1795599276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b40c9400 con 0x7f64b40b2070 2024-03-29T10:12:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.298+0000 7f64aa000700 1 -- 172.21.15.12:0/1795599276 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 55) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f64a4011ca0 con 0x7f64b40b2070 2024-03-29T10:12:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.299+0000 7f64aa000700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f649406aae0 0x7f649406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.299+0000 7f64aa000700 1 -- 172.21.15.12:0/1795599276 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f64a408aa70 con 0x7f64b40b2070 2024-03-29T10:12:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.300+0000 7f64b1400700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f649406aae0 0x7f649406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.300+0000 7f64b1400700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f649406aae0 0x7f649406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f64b41583e0 tx=0x7f64a0008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.304+0000 7f64aa000700 1 -- 172.21.15.12:0/1795599276 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64b40c9400 con 0x7f64b40b2070 2024-03-29T10:12:27.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.561+0000 7f64b8a00700 1 -- 172.21.15.12:0/1795599276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f64b40c9400 con 0x7f64b40b2070 2024-03-29T10:12:27.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T10:12:27.562+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T10:12:27.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.574+0000 7f64aa000700 1 -- 172.21.15.12:0/1795599276 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v47) v1 ==== 161+0+0 (secure 0 0 0) 0x7f64b40c9400 con 0x7f64b40b2070 2024-03-29T10:12:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 -- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f649406aae0 msgr2=0x7f649406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f649406aae0 0x7f649406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f64b41583e0 tx=0x7f64a0008040 comp rx=0 tx=0).stop 2024-03-29T10:12:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 -- 172.21.15.12:0/1795599276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40b2070 msgr2=0x7f64b414b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40b2070 0x7f64b414b4f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f64b406c130 tx=0x7f64a4004750 comp rx=0 tx=0).stop 2024-03-29T10:12:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 -- 172.21.15.12:0/1795599276 shutdown_connections 2024-03-29T10:12:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f649406aae0 0x7f649406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64b40cd020 0x7f64b4071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64b40b2070 0x7f64b414b4f0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 --2- 172.21.15.12:0/1795599276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64b40cbb30 0x7f64b40adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 -- 172.21.15.12:0/1795599276 >> 172.21.15.12:0/1795599276 conn(0x7f64b40b0d70 msgr2=0x7f64b406e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:27.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 -- 172.21.15.12:0/1795599276 shutdown_connections 2024-03-29T10:12:27.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.577+0000 7f64b8a00700 1 -- 172.21.15.12:0/1795599276 wait complete. 2024-03-29T10:12:27.578 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T10:12:27.586 INFO:tasks.ceph.mds.c.smithi012.stderr: -8425> 2024-03-29T10:11:04.093+0000 7f7111600700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:12:27.595 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T10:12:27.640 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T10:12:27.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.799+0000 7feab8e00700 1 Processor -- start 2024-03-29T10:12:27.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.800+0000 7feab8e00700 1 -- start start 2024-03-29T10:12:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.801+0000 7feab8e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7feab40cbb30 0x7feab40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.801+0000 7feab8e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feab40cd480 0x7feab406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.801+0000 7feab8e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab40cc450 con 0x7feab40cd020 2024-03-29T10:12:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.801+0000 7feab8e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab40cd8f0 con 0x7feab40cd480 2024-03-29T10:12:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.801+0000 7feab8e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab40cdae0 con 0x7feab40cbb30 2024-03-29T10:12:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.801+0000 7feab2800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feab40cd480 0x7feab406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.801+0000 7feab2800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feab40cd480 0x7feab406e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40394/0 (socket says 172.21.15.12:40394) 2024-03-29T10:12:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.801+0000 7feab2800700 1 -- 172.21.15.12:0/460111944 learned_addr learned my addr 172.21.15.12:0/460111944 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:27.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.801+0000 7feab1e00700 1 --2- 172.21.15.12:0/460111944 >> v2:172.21.15.159:3300/0 conn(0x7feab40cbb30 0x7feab40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.802+0000 7feab2800700 1 -- 172.21.15.12:0/460111944 >> v2:172.21.15.159:3300/0 conn(0x7feab40cbb30 msgr2=0x7feab40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:27.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.802+0000 7feab2800700 1 --2- 172.21.15.12:0/460111944 >> v2:172.21.15.159:3300/0 conn(0x7feab40cbb30 0x7feab40cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.802+0000 7feab2800700 1 -- 172.21.15.12:0/460111944 >> v1:172.21.15.159:6789/0 conn(0x7feab40cd020 legacy=0x7feab40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:27.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.802+0000 7feab2800700 1 -- 172.21.15.12:0/460111944 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab40cddc0 con 0x7feab40cd480 2024-03-29T10:12:27.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.802+0000 7feab2800700 1 --2- 172.21.15.12:0/460111944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 0x7feab406e420 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7feab406fd00 tx=0x7feaac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805306a614692d28 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:27.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.802+0000 7feab0a00700 1 -- 172.21.15.12:0/460111944 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac00e070 con 0x7feab40cd480 2024-03-29T10:12:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.802+0000 7feab0a00700 1 -- 172.21.15.12:0/460111944 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac00b9c0 con 0x7feab40cd480 2024-03-29T10:12:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.802+0000 7feab0a00700 1 -- 172.21.15.12:0/460111944 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac00bd70 con 0x7feab40cd480 2024-03-29T10:12:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.803+0000 7feab8e00700 1 -- 172.21.15.12:0/460111944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 msgr2=0x7feab406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.803+0000 7feab8e00700 1 --2- 172.21.15.12:0/460111944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 0x7feab406e420 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7feab406fd00 tx=0x7feaac00b330 comp rx=0 tx=0).stop 2024-03-29T10:12:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.803+0000 7feab8e00700 1 -- 172.21.15.12:0/460111944 shutdown_connections 2024-03-29T10:12:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.803+0000 7feab8e00700 1 --2- 172.21.15.12:0/460111944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 0x7feab406e420 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.803+0000 7feab8e00700 1 --2- 172.21.15.12:0/460111944 >> v2:172.21.15.159:3300/0 conn(0x7feab40cbb30 0x7feab40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.803+0000 7feab8e00700 1 -- 172.21.15.12:0/460111944 >> 172.21.15.12:0/460111944 conn(0x7feab40b0d70 msgr2=0x7feab40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.803+0000 7feab8e00700 1 -- 172.21.15.12:0/460111944 shutdown_connections 2024-03-29T10:12:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.803+0000 7feab8e00700 1 -- 172.21.15.12:0/460111944 wait complete. 2024-03-29T10:12:27.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.805+0000 7feab8e00700 1 Processor -- start 2024-03-29T10:12:27.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.805+0000 7feab8e00700 1 -- start start 2024-03-29T10:12:27.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab8e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feab40cbb30 0x7feab414b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab8e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feab40cd020 0x7feab40b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 0x7feab415d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab8e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab414bb30 con 0x7feab40cd480 2024-03-29T10:12:27.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab8e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab4155c70 con 0x7feab40cbb30 2024-03-29T10:12:27.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab8e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab40ca090 con 0x7feab40cd020 2024-03-29T10:12:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 0x7feab415d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab1e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feab40cbb30 0x7feab414b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 0x7feab415d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40402/0 (socket says 172.21.15.12:40402) 2024-03-29T10:12:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab2800700 1 -- 172.21.15.12:0/2380741792 learned_addr learned my addr 172.21.15.12:0/2380741792 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab1400700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feab40cd020 0x7feab40b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab2800700 1 -- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feab40cd020 msgr2=0x7feab40b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab2800700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feab40cd020 0x7feab40b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab2800700 1 -- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feab40cbb30 msgr2=0x7feab414b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab2800700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feab40cbb30 0x7feab414b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.806+0000 7feab2800700 1 -- 172.21.15.12:0/2380741792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac009b30 con 0x7feab40cd480 2024-03-29T10:12:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.807+0000 7feab2800700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 0x7feab415d6b0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7feab406c130 tx=0x7feaac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.807+0000 7feaaa000700 1 -- 172.21.15.12:0/2380741792 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac004880 con 0x7feab40cd480 2024-03-29T10:12:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.807+0000 7feaaa000700 1 -- 172.21.15.12:0/2380741792 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac0099b0 con 0x7feab40cd480 2024-03-29T10:12:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.807+0000 7feaaa000700 1 -- 172.21.15.12:0/2380741792 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac01c680 con 0x7feab40cd480 2024-03-29T10:12:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.807+0000 7feab8e00700 1 -- 172.21.15.12:0/2380741792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab40ca370 con 0x7feab40cd480 2024-03-29T10:12:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.807+0000 7feab8e00700 1 -- 172.21.15.12:0/2380741792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab40ca930 con 0x7feab40cd480 2024-03-29T10:12:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.808+0000 7feab8e00700 1 -- 172.21.15.12:0/2380741792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab4067720 con 0x7feab40cd480 2024-03-29T10:12:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.809+0000 7feaaa000700 1 -- 172.21.15.12:0/2380741792 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 55) v1 ==== 88975+0+0 (secure 0 0 0) 0x7feaac01c7e0 con 0x7feab40cd480 2024-03-29T10:12:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.810+0000 7feaaa000700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea9406aae0 0x7fea9406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:27.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.810+0000 7feaaa000700 1 -- 172.21.15.12:0/2380741792 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6512+0+0 (secure 0 0 0) 0x7feaac08abb0 con 0x7feab40cd480 2024-03-29T10:12:27.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.810+0000 7feab1e00700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea9406aae0 0x7fea9406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:27.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.811+0000 7feab1e00700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea9406aae0 0x7fea9406cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feaa4006b20 tx=0x7feaa4008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:27.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:27.815+0000 7feaaa000700 1 -- 172.21.15.12:0/2380741792 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feaac055d00 con 0x7feab40cd480 2024-03-29T10:12:27.934 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:12:27.938 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:12:28.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.083+0000 7feab8e00700 1 -- 172.21.15.12:0/2380741792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7feab40ce2e0 con 0x7feab40cd480 2024-03-29T10:12:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.575+0000 7feaaa000700 1 -- 172.21.15.12:0/2380741792 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v48) v1 ==== 104+0+0 (secure 0 0 0) 0x7feaac059320 con 0x7feab40cd480 2024-03-29T10:12:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 -- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea9406aae0 msgr2=0x7fea9406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea9406aae0 0x7fea9406cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feaa4006b20 tx=0x7feaa4008040 comp rx=0 tx=0).stop 2024-03-29T10:12:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 -- 172.21.15.12:0/2380741792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 msgr2=0x7feab415d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 0x7feab415d6b0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7feab406c130 tx=0x7feaac004ad0 comp rx=0 tx=0).stop 2024-03-29T10:12:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 -- 172.21.15.12:0/2380741792 shutdown_connections 2024-03-29T10:12:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea9406aae0 0x7fea9406cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feab40cd480 0x7feab415d6b0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feab40cbb30 0x7feab414b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 --2- 172.21.15.12:0/2380741792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feab40cd020 0x7feab40b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 -- 172.21.15.12:0/2380741792 >> 172.21.15.12:0/2380741792 conn(0x7feab40b0d70 msgr2=0x7feab40b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 -- 172.21.15.12:0/2380741792 shutdown_connections 2024-03-29T10:12:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.579+0000 7feab8e00700 1 -- 172.21.15.12:0/2380741792 wait complete. 2024-03-29T10:12:28.603 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T10:12:28.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.813+0000 7f5444c00700 1 Processor -- start 2024-03-29T10:12:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.813+0000 7f5444c00700 1 -- start start 2024-03-29T10:12:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.814+0000 7f5444c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5440162030 0x7f54400d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.814+0000 7f5444c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54400d20d0 0x7f54400ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.814+0000 7f5444c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544015dd10 con 0x7f544015d930 2024-03-29T10:12:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.814+0000 7f5444c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f544015ec40 con 0x7f5440162030 2024-03-29T10:12:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.814+0000 7f5444c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400d2560 con 0x7f54400d20d0 2024-03-29T10:12:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.815+0000 7f543de00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f544015d930 0x7f54400d3ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54398/0 (socket says 172.21.15.12:54398) 2024-03-29T10:12:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.815+0000 7f543de00700 1 -- 172.21.15.12:0/2608909451 learned_addr learned my addr 172.21.15.12:0/2608909451 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.815+0000 7f5437e00700 1 --2- 172.21.15.12:0/2608909451 >> v2:172.21.15.12:3300/0 conn(0x7f5440162030 0x7f54400d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.815+0000 7f543e800700 1 --2- 172.21.15.12:0/2608909451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54400d20d0 0x7f54400ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.815+0000 7f5437e00700 1 -- 172.21.15.12:0/2608909451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54400d20d0 msgr2=0x7f54400ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.815+0000 7f5437e00700 1 --2- 172.21.15.12:0/2608909451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54400d20d0 0x7f54400ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.815+0000 7f5437e00700 1 -- 172.21.15.12:0/2608909451 >> v1:172.21.15.12:6789/0 conn(0x7f544015d930 legacy=0x7f54400d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.815+0000 7f543d400700 1 -- 172.21.15.12:0/2608909451 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813297696 0 0) 0x7f544015dd10 con 0x7f544015d930 2024-03-29T10:12:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.815+0000 7f5437e00700 1 -- 172.21.15.12:0/2608909451 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544015dfc0 con 0x7f5440162030 2024-03-29T10:12:28.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.816+0000 7f5437e00700 1 --2- 172.21.15.12:0/2608909451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 0x7f54400d1b90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f5428009840 tx=0x7f5428009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b97c58ae3f70233 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:28.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.816+0000 7f543d400700 1 -- 172.21.15.12:0/2608909451 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428011830 con 0x7f5440162030 2024-03-29T10:12:28.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.816+0000 7f543d400700 1 -- 172.21.15.12:0/2608909451 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5428011990 con 0x7f5440162030 2024-03-29T10:12:28.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.816+0000 7f543d400700 1 -- 172.21.15.12:0/2608909451 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428011d20 con 0x7f5440162030 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.816+0000 7f5444c00700 1 -- 172.21.15.12:0/2608909451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 msgr2=0x7f54400d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.816+0000 7f5444c00700 1 --2- 172.21.15.12:0/2608909451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 0x7f54400d1b90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f5428009840 tx=0x7f5428009b50 comp rx=0 tx=0).stop 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.816+0000 7f5444c00700 1 -- 172.21.15.12:0/2608909451 shutdown_connections 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.816+0000 7f5444c00700 1 --2- 172.21.15.12:0/2608909451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54400d20d0 0x7f54400ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.816+0000 7f5444c00700 1 --2- 172.21.15.12:0/2608909451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 0x7f54400d1b90 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.816+0000 7f5444c00700 1 -- 172.21.15.12:0/2608909451 >> 172.21.15.12:0/2608909451 conn(0x7f544008aff0 msgr2=0x7f54400d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.817+0000 7f5444c00700 1 -- 172.21.15.12:0/2608909451 shutdown_connections 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.817+0000 7f5444c00700 1 -- 172.21.15.12:0/2608909451 wait complete. 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.818+0000 7f5444c00700 1 Processor -- start 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.818+0000 7f5444c00700 1 -- start start 2024-03-29T10:12:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.819+0000 7f5444c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54400d20d0 0x7f54401f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.819+0000 7f5444c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f544015d930 0x7f54400b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f5444c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 0x7f54400ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f5444c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54401f2fd0 con 0x7f5440162030 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f5444c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400b4770 con 0x7f544015d930 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f5444c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400b4a50 con 0x7f54400d20d0 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f543e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 0x7f54400ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f543e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 0x7f54400ccdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40408/0 (socket says 172.21.15.12:40408) 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f543e800700 1 -- 172.21.15.12:0/2075419304 learned_addr learned my addr 172.21.15.12:0/2075419304 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f543de00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54400d20d0 0x7f54401f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f5437e00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f544015d930 0x7f54400b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f543e800700 1 -- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54400d20d0 msgr2=0x7f54401f2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f543e800700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54400d20d0 0x7f54401f2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f543e800700 1 -- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f544015d930 msgr2=0x7f54400b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f543e800700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f544015d930 0x7f54400b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f543e800700 1 -- 172.21.15.12:0/2075419304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54280094a0 con 0x7f5440162030 2024-03-29T10:12:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f5437e00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f544015d930 0x7f54400b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.820+0000 7f543e800700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 0x7f54400ccdd0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f5438007e40 tx=0x7f543800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.821+0000 7f5436000700 1 -- 172.21.15.12:0/2075419304 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543800ede0 con 0x7f5440162030 2024-03-29T10:12:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.821+0000 7f5436000700 1 -- 172.21.15.12:0/2075419304 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5438014420 con 0x7f5440162030 2024-03-29T10:12:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.821+0000 7f5436000700 1 -- 172.21.15.12:0/2075419304 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54380147d0 con 0x7f5440162030 2024-03-29T10:12:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.821+0000 7f5444c00700 1 -- 172.21.15.12:0/2075419304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54400b4d30 con 0x7f5440162030 2024-03-29T10:12:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.821+0000 7f5444c00700 1 -- 172.21.15.12:0/2075419304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54400b52f0 con 0x7f5440162030 2024-03-29T10:12:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.822+0000 7f5444c00700 1 -- 172.21.15.12:0/2075419304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5440073e70 con 0x7f5440162030 2024-03-29T10:12:28.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.823+0000 7f5436000700 1 -- 172.21.15.12:0/2075419304 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 56) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f5438014930 con 0x7f5440162030 2024-03-29T10:12:28.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.824+0000 7f5436000700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541c06ad20 0x7f541c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:28.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.824+0000 7f5436000700 1 -- 172.21.15.12:0/2075419304 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f5438089ea0 con 0x7f5440162030 2024-03-29T10:12:28.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.825+0000 7f543de00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541c06ad20 0x7f541c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:28.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.826+0000 7f543de00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541c06ad20 0x7f541c06d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5430007fd0 tx=0x7f5430007de0 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:28.829+0000 7f5436000700 1 -- 172.21.15.12:0/2075419304 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5438054c90 con 0x7f5440162030 2024-03-29T10:12:29.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.048+0000 7f5444c00700 1 -- 172.21.15.12:0/2075419304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5440067720 con 0x7f5440162030 2024-03-29T10:12:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.975+0000 7f5436000700 1 -- 172.21.15.12:0/2075419304 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v114) v1 ==== 181+0+0 (secure 0 0 0) 0x7f54380582b0 con 0x7f5440162030 2024-03-29T10:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 -- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541c06ad20 msgr2=0x7f541c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541c06ad20 0x7f541c06d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5430007fd0 tx=0x7f5430007de0 comp rx=0 tx=0).stop 2024-03-29T10:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 -- 172.21.15.12:0/2075419304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 msgr2=0x7f54400ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 0x7f54400ccdd0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f5438007e40 tx=0x7f543800e3c0 comp rx=0 tx=0).stop 2024-03-29T10:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 -- 172.21.15.12:0/2075419304 shutdown_connections 2024-03-29T10:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f541c06ad20 0x7f541c06d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5440162030 0x7f54400ccdd0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54400d20d0 0x7f54401f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 --2- 172.21.15.12:0/2075419304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f544015d930 0x7f54400b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 -- 172.21.15.12:0/2075419304 >> 172.21.15.12:0/2075419304 conn(0x7f544008aff0 msgr2=0x7f54400ceb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 -- 172.21.15.12:0/2075419304 shutdown_connections 2024-03-29T10:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:29.979+0000 7f5444c00700 1 -- 172.21.15.12:0/2075419304 wait complete. 2024-03-29T10:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T10:12:30.001 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T10:12:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.208+0000 7fe437c00700 1 Processor -- start 2024-03-29T10:12:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.208+0000 7fe437c00700 1 -- start start 2024-03-29T10:12:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.209+0000 7fe437c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe43015d930 0x7fe4300d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.209+0000 7fe437c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 0x7fe4300d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.209+0000 7fe437c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43015dd10 con 0x7fe4300d20d0 2024-03-29T10:12:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.209+0000 7fe437c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe43015ec40 con 0x7fe43015d930 2024-03-29T10:12:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.209+0000 7fe437c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300d2560 con 0x7fe430162030 2024-03-29T10:12:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe435200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe43015d930 0x7fe4300d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe435200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe43015d930 0x7fe4300d3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44636/0 (socket says 172.21.15.12:44636) 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe435200700 1 -- 172.21.15.12:0/3791145695 learned_addr learned my addr 172.21.15.12:0/3791145695 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe42fe00700 1 --2- 172.21.15.12:0/3791145695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 0x7fe4300d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe435200700 1 -- 172.21.15.12:0/3791145695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 msgr2=0x7fe4300d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe435200700 1 --2- 172.21.15.12:0/3791145695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 0x7fe4300d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe435200700 1 -- 172.21.15.12:0/3791145695 >> v1:172.21.15.12:6789/0 conn(0x7fe4300d20d0 legacy=0x7fe4300cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe435200700 1 -- 172.21.15.12:0/3791145695 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43015dfc0 con 0x7fe43015d930 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe42f400700 1 -- 172.21.15.12:0/3791145695 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147719410 0 0) 0x7fe43015dd10 con 0x7fe4300d20d0 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe42fe00700 1 --2- 172.21.15.12:0/3791145695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 0x7fe4300d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.210+0000 7fe435200700 1 --2- 172.21.15.12:0/3791145695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe43015d930 0x7fe4300d3ed0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe4240098f0 tx=0x7fe424009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b55b25e810db192 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe42f400700 1 -- 172.21.15.12:0/3791145695 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42400b8a0 con 0x7fe43015d930 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe42f400700 1 -- 172.21.15.12:0/3791145695 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42400ba00 con 0x7fe43015d930 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe42f400700 1 -- 172.21.15.12:0/3791145695 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42400bd90 con 0x7fe43015d930 2024-03-29T10:12:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe437c00700 1 -- 172.21.15.12:0/3791145695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe43015d930 msgr2=0x7fe4300d3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe437c00700 1 --2- 172.21.15.12:0/3791145695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe43015d930 0x7fe4300d3ed0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe4240098f0 tx=0x7fe424009c00 comp rx=0 tx=0).stop 2024-03-29T10:12:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe437c00700 1 -- 172.21.15.12:0/3791145695 shutdown_connections 2024-03-29T10:12:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe437c00700 1 --2- 172.21.15.12:0/3791145695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe43015d930 0x7fe4300d3ed0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe437c00700 1 --2- 172.21.15.12:0/3791145695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 0x7fe4300d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe437c00700 1 -- 172.21.15.12:0/3791145695 >> 172.21.15.12:0/3791145695 conn(0x7fe43008aff0 msgr2=0x7fe4300d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe437c00700 1 -- 172.21.15.12:0/3791145695 shutdown_connections 2024-03-29T10:12:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.211+0000 7fe437c00700 1 -- 172.21.15.12:0/3791145695 wait complete. 2024-03-29T10:12:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.213+0000 7fe437c00700 1 Processor -- start 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.213+0000 7fe437c00700 1 -- start start 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.214+0000 7fe437c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4300d20d0 0x7fe430157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.214+0000 7fe437c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe43015d930 0x7fe4300c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.214+0000 7fe437c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 0x7fe4300b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.214+0000 7fe437c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300ca440 con 0x7fe4300d20d0 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.214+0000 7fe437c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300cb480 con 0x7fe43015d930 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.214+0000 7fe437c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe430157e80 con 0x7fe430162030 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.214+0000 7fe435200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4300d20d0 0x7fe430157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.214+0000 7fe435200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4300d20d0 0x7fe430157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44642/0 (socket says 172.21.15.12:44642) 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.214+0000 7fe435200700 1 -- 172.21.15.12:0/197508807 learned_addr learned my addr 172.21.15.12:0/197508807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe435c00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 0x7fe4300b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe42fe00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe43015d930 0x7fe4300c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe435200700 1 -- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 msgr2=0x7fe4300b0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe435200700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 0x7fe4300b0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe435200700 1 -- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe43015d930 msgr2=0x7fe4300c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe435200700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe43015d930 0x7fe4300c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe435200700 1 -- 172.21.15.12:0/197508807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe424009550 con 0x7fe4300d20d0 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe435200700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4300d20d0 0x7fe430157940 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe43008d050 tx=0x7fe424004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe42cc00700 1 -- 172.21.15.12:0/197508807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe424004380 con 0x7fe4300d20d0 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe42cc00700 1 -- 172.21.15.12:0/197508807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42401f070 con 0x7fe4300d20d0 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe42cc00700 1 -- 172.21.15.12:0/197508807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4240036a0 con 0x7fe4300d20d0 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.215+0000 7fe437c00700 1 -- 172.21.15.12:0/197508807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4300c9e80 con 0x7fe4300d20d0 2024-03-29T10:12:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.216+0000 7fe437c00700 1 -- 172.21.15.12:0/197508807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4300cb110 con 0x7fe4300d20d0 2024-03-29T10:12:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.217+0000 7fe437c00700 1 -- 172.21.15.12:0/197508807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe430067720 con 0x7fe4300d20d0 2024-03-29T10:12:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.218+0000 7fe42cc00700 1 -- 172.21.15.12:0/197508807 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 56) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fe424021030 con 0x7fe4300d20d0 2024-03-29T10:12:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.218+0000 7fe42cc00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe41006ad20 0x7fe41006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.218+0000 7fe42cc00700 1 -- 172.21.15.12:0/197508807 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6074+0+0 (secure 0 0 0) 0x7fe42408a9f0 con 0x7fe4300d20d0 2024-03-29T10:12:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.219+0000 7fe42fe00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe41006ad20 0x7fe41006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.219+0000 7fe42fe00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe41006ad20 0x7fe41006d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe430158560 tx=0x7fe41c008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.223+0000 7fe42cc00700 1 -- 172.21.15.12:0/197508807 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe424055b20 con 0x7fe4300d20d0 2024-03-29T10:12:30.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.227+0000 7fe42cc00700 1 -- 172.21.15.12:0/197508807 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe42408cd50 con 0x7fe4300d20d0 2024-03-29T10:12:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:30.441+0000 7fe437c00700 1 -- 172.21.15.12:0/197508807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe4300b9530 con 0x7fe4300d20d0 2024-03-29T10:12:31.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.228+0000 7fe42cc00700 1 -- 172.21.15.12:0/197508807 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v115) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe42406fd30 con 0x7fe4300d20d0 2024-03-29T10:12:31.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 -- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe41006ad20 msgr2=0x7fe41006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe41006ad20 0x7fe41006d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe430158560 tx=0x7fe41c008040 comp rx=0 tx=0).stop 2024-03-29T10:12:31.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 -- 172.21.15.12:0/197508807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4300d20d0 msgr2=0x7fe430157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4300d20d0 0x7fe430157940 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe43008d050 tx=0x7fe424004750 comp rx=0 tx=0).stop 2024-03-29T10:12:31.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 -- 172.21.15.12:0/197508807 shutdown_connections 2024-03-29T10:12:31.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe41006ad20 0x7fe41006d1e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe430162030 0x7fe4300b0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4300d20d0 0x7fe430157940 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 --2- 172.21.15.12:0/197508807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe43015d930 0x7fe4300c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 -- 172.21.15.12:0/197508807 >> 172.21.15.12:0/197508807 conn(0x7fe43008aff0 msgr2=0x7fe43015cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:31.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 -- 172.21.15.12:0/197508807 shutdown_connections 2024-03-29T10:12:31.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.232+0000 7fe437c00700 1 -- 172.21.15.12:0/197508807 wait complete. 2024-03-29T10:12:31.233 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T10:12:31.255 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T10:12:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.462+0000 7fe6d3200700 1 Processor -- start 2024-03-29T10:12:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.463+0000 7fe6d3200700 1 -- start start 2024-03-29T10:12:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.464+0000 7fe6d3200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe6cc15d9d0 0x7fe6cc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.464+0000 7fe6d3200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6cc0d16a0 0x7fe6cc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.464+0000 7fe6d3200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6cc161420 con 0x7fe6cc0d2370 2024-03-29T10:12:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.464+0000 7fe6d3200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc0d27e0 con 0x7fe6cc0d16a0 2024-03-29T10:12:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.464+0000 7fe6d3200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc161700 con 0x7fe6cc15d9d0 2024-03-29T10:12:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.464+0000 7fe6cb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6cc0d16a0 0x7fe6cc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.464+0000 7fe6cb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6cc0d16a0 0x7fe6cc0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44646/0 (socket says 172.21.15.12:44646) 2024-03-29T10:12:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.464+0000 7fe6cb400700 1 -- 172.21.15.12:0/2443920378 learned_addr learned my addr 172.21.15.12:0/2443920378 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.464+0000 7fe6cbe00700 1 --2- 172.21.15.12:0/2443920378 >> v2:172.21.15.159:3300/0 conn(0x7fe6cc15d9d0 0x7fe6cc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.465+0000 7fe6cb400700 1 -- 172.21.15.12:0/2443920378 >> v2:172.21.15.159:3300/0 conn(0x7fe6cc15d9d0 msgr2=0x7fe6cc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:31.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.465+0000 7fe6cb400700 1 --2- 172.21.15.12:0/2443920378 >> v2:172.21.15.159:3300/0 conn(0x7fe6cc15d9d0 0x7fe6cc0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.465+0000 7fe6cb400700 1 -- 172.21.15.12:0/2443920378 >> v1:172.21.15.159:6789/0 conn(0x7fe6cc0d2370 legacy=0x7fe6cc15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.465+0000 7fe6cb400700 1 -- 172.21.15.12:0/2443920378 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc1619e0 con 0x7fe6cc0d16a0 2024-03-29T10:12:31.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.465+0000 7fe6cb400700 1 --2- 172.21.15.12:0/2443920378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d16a0 0x7fe6cc0d1e30 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0d0ff0 tx=0x7fe6b8009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed55c38b7df4bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:31.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.465+0000 7fe6caa00700 1 -- 172.21.15.12:0/2443920378 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8011830 con 0x7fe6cc0d16a0 2024-03-29T10:12:31.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.465+0000 7fe6caa00700 1 -- 172.21.15.12:0/2443920378 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b8011990 con 0x7fe6cc0d16a0 2024-03-29T10:12:31.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.465+0000 7fe6caa00700 1 -- 172.21.15.12:0/2443920378 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8011d40 con 0x7fe6cc0d16a0 2024-03-29T10:12:31.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.466+0000 7fe6d3200700 1 -- 172.21.15.12:0/2443920378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d16a0 msgr2=0x7fe6cc0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.466+0000 7fe6d3200700 1 --2- 172.21.15.12:0/2443920378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d16a0 0x7fe6cc0d1e30 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0d0ff0 tx=0x7fe6b8009bc0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.466+0000 7fe6d3200700 1 -- 172.21.15.12:0/2443920378 shutdown_connections 2024-03-29T10:12:31.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.466+0000 7fe6d3200700 1 --2- 172.21.15.12:0/2443920378 >> v2:172.21.15.159:3300/0 conn(0x7fe6cc15d9d0 0x7fe6cc0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.466+0000 7fe6d3200700 1 --2- 172.21.15.12:0/2443920378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d16a0 0x7fe6cc0d1e30 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.466+0000 7fe6d3200700 1 -- 172.21.15.12:0/2443920378 >> 172.21.15.12:0/2443920378 conn(0x7fe6cc08aff0 msgr2=0x7fe6cc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:31.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.466+0000 7fe6d3200700 1 -- 172.21.15.12:0/2443920378 shutdown_connections 2024-03-29T10:12:31.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.466+0000 7fe6d3200700 1 -- 172.21.15.12:0/2443920378 wait complete. 2024-03-29T10:12:31.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.468+0000 7fe6d3200700 1 Processor -- start 2024-03-29T10:12:31.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.468+0000 7fe6d3200700 1 -- start start 2024-03-29T10:12:31.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6d3200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6cc0d16a0 0x7fe6cc0cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6d3200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d2370 0x7fe6cc0cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6d3200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6cc15d9d0 0x7fe6cc0b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6d3200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1f43e0 con 0x7fe6cc0d2370 2024-03-29T10:12:31.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6d3200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc0ca790 con 0x7fe6cc0d16a0 2024-03-29T10:12:31.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6d3200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc0caa70 con 0x7fe6cc15d9d0 2024-03-29T10:12:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6cb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d2370 0x7fe6cc0cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6cb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d2370 0x7fe6cc0cc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44652/0 (socket says 172.21.15.12:44652) 2024-03-29T10:12:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6cb400700 1 -- 172.21.15.12:0/2799497410 learned_addr learned my addr 172.21.15.12:0/2799497410 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6cbe00700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6cc0d16a0 0x7fe6cc0cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6d1200700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6cc15d9d0 0x7fe6cc0b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6cb400700 1 -- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6cc15d9d0 msgr2=0x7fe6cc0b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6cb400700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6cc15d9d0 0x7fe6cc0b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6cb400700 1 -- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6cc0d16a0 msgr2=0x7fe6cc0cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6cb400700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6cc0d16a0 0x7fe6cc0cb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.469+0000 7fe6cb400700 1 -- 172.21.15.12:0/2799497410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b80094f0 con 0x7fe6cc0d2370 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.470+0000 7fe6cb400700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d2370 0x7fe6cc0cc240 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8013000 tx=0x7fe6b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.470+0000 7fe6c3e00700 1 -- 172.21.15.12:0/2799497410 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b801aa10 con 0x7fe6cc0d2370 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.470+0000 7fe6c3e00700 1 -- 172.21.15.12:0/2799497410 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b801f070 con 0x7fe6cc0d2370 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.470+0000 7fe6d3200700 1 -- 172.21.15.12:0/2799497410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc0cad50 con 0x7fe6cc0d2370 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.470+0000 7fe6c3e00700 1 -- 172.21.15.12:0/2799497410 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b801ad90 con 0x7fe6cc0d2370 2024-03-29T10:12:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.470+0000 7fe6d3200700 1 -- 172.21.15.12:0/2799497410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc0cba90 con 0x7fe6cc0d2370 2024-03-29T10:12:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.471+0000 7fe6d3200700 1 -- 172.21.15.12:0/2799497410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc0cbc80 con 0x7fe6cc0d2370 2024-03-29T10:12:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.472+0000 7fe6c3e00700 1 -- 172.21.15.12:0/2799497410 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe6b802e450 con 0x7fe6cc0d2370 2024-03-29T10:12:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.473+0000 7fe6c3e00700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe6ac06adb0 0x7fe6ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.473+0000 7fe6c3e00700 1 -- 172.21.15.12:0/2799497410 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fe6b808a580 con 0x7fe6cc0d2370 2024-03-29T10:12:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.473+0000 7fe6cbe00700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe6ac06adb0 0x7fe6ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.474+0000 7fe6cbe00700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe6ac06adb0 0x7fe6ac06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009ad0 tx=0x7fe6bc008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.477+0000 7fe6c3e00700 1 -- 172.21.15.12:0/2799497410 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6b8055690 con 0x7fe6cc0d2370 2024-03-29T10:12:31.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.699+0000 7fe6d3200700 1 -- 172.21.15.12:0/2799497410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fe6cc0d1ca0 con 0x7fe6cc0d2370 2024-03-29T10:12:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.704+0000 7fe6c3e00700 1 -- 172.21.15.12:0/2799497410 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v37) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe6b8058cb0 con 0x7fe6cc0d2370 2024-03-29T10:12:31.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 -- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe6ac06adb0 msgr2=0x7fe6ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe6ac06adb0 0x7fe6ac06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009ad0 tx=0x7fe6bc008040 comp rx=0 tx=0).stop 2024-03-29T10:12:31.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 -- 172.21.15.12:0/2799497410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d2370 msgr2=0x7fe6cc0cc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d2370 0x7fe6cc0cc240 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8013000 tx=0x7fe6b8004750 comp rx=0 tx=0).stop 2024-03-29T10:12:31.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 -- 172.21.15.12:0/2799497410 shutdown_connections 2024-03-29T10:12:31.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe6ac06adb0 0x7fe6ac06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6cc15d9d0 0x7fe6cc0b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6cc0d16a0 0x7fe6cc0cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 --2- 172.21.15.12:0/2799497410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6cc0d2370 0x7fe6cc0cc240 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 -- 172.21.15.12:0/2799497410 >> 172.21.15.12:0/2799497410 conn(0x7fe6cc08aff0 msgr2=0x7fe6cc06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:31.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 -- 172.21.15.12:0/2799497410 shutdown_connections 2024-03-29T10:12:31.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.706+0000 7fe6d3200700 1 -- 172.21.15.12:0/2799497410 wait complete. 2024-03-29T10:12:31.724 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T10:12:31.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.924+0000 7f5115400700 1 Processor -- start 2024-03-29T10:12:31.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.924+0000 7f5115400700 1 -- start start 2024-03-29T10:12:31.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.926+0000 7f5115400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f51100d16a0 0x7f51100d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.926+0000 7f5115400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51100d2370 0x7f51100ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.926+0000 7f5115400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5110161420 con 0x7f511015d9d0 2024-03-29T10:12:31.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.926+0000 7f5115400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51100d2820 con 0x7f51100d16a0 2024-03-29T10:12:31.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.926+0000 7f5115400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51100d4340 con 0x7f51100d2370 2024-03-29T10:12:31.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.926+0000 7f510e800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f511015d9d0 0x7f51100d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59720/0 (socket says 172.21.15.12:59720) 2024-03-29T10:12:31.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.926+0000 7f510e800700 1 -- 172.21.15.12:0/403966352 learned_addr learned my addr 172.21.15.12:0/403966352 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:31.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.926+0000 7f510de00700 1 --2- 172.21.15.12:0/403966352 >> v2:172.21.15.159:3300/0 conn(0x7f51100d16a0 0x7f51100d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.926+0000 7f510f200700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51100d2370 0x7f51100ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769177934 0 0) 0x7f5110161420 con 0x7f511015d9d0 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50f40035b0 con 0x7f511015d9d0 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659728916 0 0) 0x7f50f40035b0 con 0x7f511015d9d0 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51100d2370 msgr2=0x7f51100ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51100d2370 0x7f51100ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 >> v2:172.21.15.159:3300/0 conn(0x7f51100d16a0 msgr2=0x7f51100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 --2- 172.21.15.12:0/403966352 >> v2:172.21.15.159:3300/0 conn(0x7f51100d16a0 0x7f51100d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5110161700 con 0x7f511015d9d0 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f5100001670 con 0x7f511015d9d0 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 >> v1:172.21.15.12:6789/0 conn(0x7f511015d9d0 legacy=0x7f51100d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f4003a00 0x7f50f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f4004850 0x7f50f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51100d2370 0x7f50f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51100d2820 con 0x7f50f4003a00 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f400ba10 con 0x7f50f4004850 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f400bb50 con 0x7f51100d2370 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.927+0000 7f510e800700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f4003a00 0x7f50f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510de00700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f4004850 0x7f50f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510f200700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51100d2370 0x7f50f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510e800700 1 -- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51100d2370 msgr2=0x7f50f4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510e800700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51100d2370 0x7f50f4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510e800700 1 -- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f4004850 msgr2=0x7f50f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510e800700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f4004850 0x7f50f4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510e800700 1 -- 172.21.15.12:0/403966352 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f40035b0 con 0x7f50f4003a00 2024-03-29T10:12:31.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510e800700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f4003a00 0x7f50f4003e20 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f510000c210 tx=0x7f50f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39277c5b3a0d82e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:31.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510000ccf0 con 0x7f50f4003a00 2024-03-29T10:12:31.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510000ce50 con 0x7f50f4003a00 2024-03-29T10:12:31.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.928+0000 7f510d400700 1 -- 172.21.15.12:0/403966352 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5100005ba0 con 0x7f50f4003a00 2024-03-29T10:12:31.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.929+0000 7f5115400700 1 -- 172.21.15.12:0/403966352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f4003a00 msgr2=0x7f50f4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.929+0000 7f5115400700 1 --2- 172.21.15.12:0/403966352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f4003a00 0x7f50f4003e20 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f510000c210 tx=0x7f50f4005880 comp rx=0 tx=0).stop 2024-03-29T10:12:31.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.929+0000 7f510f200700 1 -- 172.21.15.12:0/403966352 reap_dead start 2024-03-29T10:12:31.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.929+0000 7f5115400700 1 -- 172.21.15.12:0/403966352 shutdown_connections 2024-03-29T10:12:31.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.929+0000 7f5115400700 1 -- 172.21.15.12:0/403966352 >> 172.21.15.12:0/403966352 conn(0x7f511008aff0 msgr2=0x7f51100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:31.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.929+0000 7f5115400700 1 -- 172.21.15.12:0/403966352 shutdown_connections 2024-03-29T10:12:31.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.929+0000 7f5115400700 1 -- 172.21.15.12:0/403966352 wait complete. 2024-03-29T10:12:31.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.931+0000 7f5115400700 1 Processor -- start 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.931+0000 7f5115400700 1 -- start start 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.932+0000 7f5115400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f4003a00 0x7f51100b8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.932+0000 7f5115400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51101f7520 0x7f51101602f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.932+0000 7f5115400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5110160830 0x7f51101f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.932+0000 7f5115400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51101f3ee0 con 0x7f51101f7520 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.932+0000 7f5115400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5110159880 con 0x7f50f4003a00 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.932+0000 7f5115400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51101f3540 con 0x7f5110160830 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.932+0000 7f510de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51101f7520 0x7f51101602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.932+0000 7f510de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51101f7520 0x7f51101602f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44672/0 (socket says 172.21.15.12:44672) 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.932+0000 7f510de00700 1 -- 172.21.15.12:0/3776273351 learned_addr learned my addr 172.21.15.12:0/3776273351 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f510e800700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f4003a00 0x7f51100b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f510f200700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5110160830 0x7f51101f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f510de00700 1 -- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5110160830 msgr2=0x7f51101f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f510de00700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5110160830 0x7f51101f2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f510de00700 1 -- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f4003a00 msgr2=0x7f51100b8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f510de00700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f4003a00 0x7f51100b8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f510de00700 1 -- 172.21.15.12:0/3776273351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51101f3820 con 0x7f51101f7520 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f510e800700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f4003a00 0x7f51100b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f510de00700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51101f7520 0x7f51101602f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f50fc011910 tx=0x7f50fc011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f5106a00700 1 -- 172.21.15.12:0/3776273351 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc005c30 con 0x7f51101f7520 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f5106a00700 1 -- 172.21.15.12:0/3776273351 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50fc005d90 con 0x7f51101f7520 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f5115400700 1 -- 172.21.15.12:0/3776273351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51101f3b00 con 0x7f51101f7520 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f5106a00700 1 -- 172.21.15.12:0/3776273351 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc00f970 con 0x7f51101f7520 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.933+0000 7f5115400700 1 -- 172.21.15.12:0/3776273351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51100b8a70 con 0x7f51101f7520 2024-03-29T10:12:31.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.934+0000 7f5115400700 1 -- 172.21.15.12:0/3776273351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51100d1ca0 con 0x7f51101f7520 2024-03-29T10:12:31.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.935+0000 7f5106a00700 1 -- 172.21.15.12:0/3776273351 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f50fc00fad0 con 0x7f51101f7520 2024-03-29T10:12:31.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.936+0000 7f5106a00700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50f006adb0 0x7f50f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:31.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.936+0000 7f5106a00700 1 -- 172.21.15.12:0/3776273351 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f50fc089960 con 0x7f51101f7520 2024-03-29T10:12:31.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.937+0000 7f510e800700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50f006adb0 0x7f50f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:31.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.937+0000 7f510e800700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50f006adb0 0x7f50f006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f510000c210 tx=0x7f510000a000 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:31.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:31.940+0000 7f5106a00700 1 -- 172.21.15.12:0/3776273351 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f50fc0549e0 con 0x7f51101f7520 2024-03-29T10:12:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.160+0000 7f5115400700 1 -- 172.21.15.12:0/3776273351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f51100b8a70 con 0x7f51101f7520 2024-03-29T10:12:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.164+0000 7f5106a00700 1 -- 172.21.15.12:0/3776273351 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7f51100b8a70 con 0x7f51101f7520 2024-03-29T10:12:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 -- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50f006adb0 msgr2=0x7f50f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50f006adb0 0x7f50f006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f510000c210 tx=0x7f510000a000 comp rx=0 tx=0).stop 2024-03-29T10:12:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 -- 172.21.15.12:0/3776273351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51101f7520 msgr2=0x7f51101602f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51101f7520 0x7f51101602f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f50fc011910 tx=0x7f50fc011cd0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 -- 172.21.15.12:0/3776273351 shutdown_connections 2024-03-29T10:12:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f50f006adb0 0x7f50f006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5110160830 0x7f51101f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f4003a00 0x7f51100b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 --2- 172.21.15.12:0/3776273351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51101f7520 0x7f51101602f0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 -- 172.21.15.12:0/3776273351 >> 172.21.15.12:0/3776273351 conn(0x7f511008aff0 msgr2=0x7f511006c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 -- 172.21.15.12:0/3776273351 shutdown_connections 2024-03-29T10:12:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.167+0000 7f5115400700 1 -- 172.21.15.12:0/3776273351 wait complete. 2024-03-29T10:12:32.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T10:12:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.384+0000 7fe393200700 1 Processor -- start 2024-03-29T10:12:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.385+0000 7fe393200700 1 -- start start 2024-03-29T10:12:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe393200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe38c099a60 0x7fe38c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe393200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe38c15d990 0x7fe38c099e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe393200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c0d0970 con 0x7fe38c09a750 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe393200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe38c06c020 con 0x7fe38c15d990 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe393200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe38c09a380 con 0x7fe38c099a60 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe38b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe38c15d990 0x7fe38c099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe38b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe38c15d990 0x7fe38c099e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44678/0 (socket says 172.21.15.12:44678) 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe38b400700 1 -- 172.21.15.12:0/4272574698 learned_addr learned my addr 172.21.15.12:0/4272574698 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe38be00700 1 --2- 172.21.15.12:0/4272574698 >> v2:172.21.15.159:3300/0 conn(0x7fe38c099a60 0x7fe38c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe38b400700 1 -- 172.21.15.12:0/4272574698 >> v2:172.21.15.159:3300/0 conn(0x7fe38c099a60 msgr2=0x7fe38c0d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe38b400700 1 --2- 172.21.15.12:0/4272574698 >> v2:172.21.15.159:3300/0 conn(0x7fe38c099a60 0x7fe38c0d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe38b400700 1 -- 172.21.15.12:0/4272574698 >> v1:172.21.15.159:6789/0 conn(0x7fe38c09a750 legacy=0x7fe38c06c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.386+0000 7fe38b400700 1 -- 172.21.15.12:0/4272574698 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c0c81a0 con 0x7fe38c15d990 2024-03-29T10:12:32.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.387+0000 7fe38b400700 1 --2- 172.21.15.12:0/4272574698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 0x7fe38c099e40 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fe38c071f40 tx=0x7fe378009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a264257a30291ea server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:32.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.387+0000 7fe38aa00700 1 -- 172.21.15.12:0/4272574698 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378004230 con 0x7fe38c15d990 2024-03-29T10:12:32.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.387+0000 7fe38aa00700 1 -- 172.21.15.12:0/4272574698 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe378004390 con 0x7fe38c15d990 2024-03-29T10:12:32.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.387+0000 7fe38aa00700 1 -- 172.21.15.12:0/4272574698 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3780036a0 con 0x7fe38c15d990 2024-03-29T10:12:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.387+0000 7fe393200700 1 -- 172.21.15.12:0/4272574698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 msgr2=0x7fe38c099e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.387+0000 7fe393200700 1 --2- 172.21.15.12:0/4272574698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 0x7fe38c099e40 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fe38c071f40 tx=0x7fe378009bd0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.388+0000 7fe393200700 1 -- 172.21.15.12:0/4272574698 shutdown_connections 2024-03-29T10:12:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.388+0000 7fe393200700 1 --2- 172.21.15.12:0/4272574698 >> v2:172.21.15.159:3300/0 conn(0x7fe38c099a60 0x7fe38c0d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.388+0000 7fe393200700 1 --2- 172.21.15.12:0/4272574698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 0x7fe38c099e40 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.388+0000 7fe393200700 1 -- 172.21.15.12:0/4272574698 >> 172.21.15.12:0/4272574698 conn(0x7fe38c0b2e20 msgr2=0x7fe38c0b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.388+0000 7fe393200700 1 -- 172.21.15.12:0/4272574698 shutdown_connections 2024-03-29T10:12:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.388+0000 7fe393200700 1 -- 172.21.15.12:0/4272574698 wait complete. 2024-03-29T10:12:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.389+0000 7fe393200700 1 Processor -- start 2024-03-29T10:12:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.390+0000 7fe393200700 1 -- start start 2024-03-29T10:12:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.390+0000 7fe393200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe38c099a60 0x7fe38c0ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.390+0000 7fe393200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe38c09a750 0x7fe38c0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.390+0000 7fe393200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 0x7fe38c0c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.390+0000 7fe393200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0cee00 con 0x7fe38c15d990 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.390+0000 7fe393200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c147e60 con 0x7fe38c09a750 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.390+0000 7fe393200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0cae00 con 0x7fe38c099a60 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe391200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 0x7fe38c0c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe391200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 0x7fe38c0c9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44692/0 (socket says 172.21.15.12:44692) 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe391200700 1 -- 172.21.15.12:0/2566139093 learned_addr learned my addr 172.21.15.12:0/2566139093 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe38be00700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe38c099a60 0x7fe38c0ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe38b400700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe38c09a750 0x7fe38c0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe391200700 1 -- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe38c099a60 msgr2=0x7fe38c0ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe391200700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe38c099a60 0x7fe38c0ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe391200700 1 -- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe38c09a750 msgr2=0x7fe38c0ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe391200700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe38c09a750 0x7fe38c0ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe391200700 1 -- 172.21.15.12:0/2566139093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe378009550 con 0x7fe38c15d990 2024-03-29T10:12:32.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe38b400700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe38c09a750 0x7fe38c0ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:32.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.391+0000 7fe391200700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 0x7fe38c0c9e60 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fe38400b970 tx=0x7fe38400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:32.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.392+0000 7fe383e00700 1 -- 172.21.15.12:0/2566139093 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38400c930 con 0x7fe38c15d990 2024-03-29T10:12:32.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.392+0000 7fe383e00700 1 -- 172.21.15.12:0/2566139093 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38400cad0 con 0x7fe38c15d990 2024-03-29T10:12:32.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.392+0000 7fe393200700 1 -- 172.21.15.12:0/2566139093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38c1f2da0 con 0x7fe38c15d990 2024-03-29T10:12:32.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.392+0000 7fe383e00700 1 -- 172.21.15.12:0/2566139093 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38400ce80 con 0x7fe38c15d990 2024-03-29T10:12:32.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.392+0000 7fe393200700 1 -- 172.21.15.12:0/2566139093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c147810 con 0x7fe38c15d990 2024-03-29T10:12:32.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.393+0000 7fe393200700 1 -- 172.21.15.12:0/2566139093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38c147fd0 con 0x7fe38c15d990 2024-03-29T10:12:32.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.394+0000 7fe383e00700 1 -- 172.21.15.12:0/2566139093 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe384014440 con 0x7fe38c15d990 2024-03-29T10:12:32.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.395+0000 7fe383e00700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe36c06adb0 0x7fe36c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.395+0000 7fe383e00700 1 -- 172.21.15.12:0/2566139093 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fe384088fe0 con 0x7fe38c15d990 2024-03-29T10:12:32.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.395+0000 7fe38be00700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe36c06adb0 0x7fe36c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.396+0000 7fe38be00700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe36c06adb0 0x7fe36c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe38c0cb960 tx=0x7fe37c008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:32.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.399+0000 7fe383e00700 1 -- 172.21.15.12:0/2566139093 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe3840541c0 con 0x7fe38c15d990 2024-03-29T10:12:32.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.613+0000 7fe393200700 1 -- 172.21.15.12:0/2566139093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe38c067720 con 0x7fe38c15d990 2024-03-29T10:12:32.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.618+0000 7fe383e00700 1 -- 172.21.15.12:0/2566139093 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v39) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe3840577e0 con 0x7fe38c15d990 2024-03-29T10:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.620+0000 7fe393200700 1 -- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe36c06adb0 msgr2=0x7fe36c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.620+0000 7fe393200700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe36c06adb0 0x7fe36c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe38c0cb960 tx=0x7fe37c008040 comp rx=0 tx=0).stop 2024-03-29T10:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.620+0000 7fe393200700 1 -- 172.21.15.12:0/2566139093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 msgr2=0x7fe38c0c9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.620+0000 7fe393200700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 0x7fe38c0c9e60 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fe38400b970 tx=0x7fe38400bc80 comp rx=0 tx=0).stop 2024-03-29T10:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.620+0000 7fe393200700 1 -- 172.21.15.12:0/2566139093 shutdown_connections 2024-03-29T10:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.620+0000 7fe393200700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe36c06adb0 0x7fe36c06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.620+0000 7fe393200700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe38c15d990 0x7fe38c0c9e60 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.620+0000 7fe393200700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe38c099a60 0x7fe38c0ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.620+0000 7fe393200700 1 --2- 172.21.15.12:0/2566139093 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe38c09a750 0x7fe38c0ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.620+0000 7fe393200700 1 -- 172.21.15.12:0/2566139093 >> 172.21.15.12:0/2566139093 conn(0x7fe38c0b2e20 msgr2=0x7fe38c0c5150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.621+0000 7fe393200700 1 -- 172.21.15.12:0/2566139093 shutdown_connections 2024-03-29T10:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.621+0000 7fe393200700 1 -- 172.21.15.12:0/2566139093 wait complete. 2024-03-29T10:12:32.639 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-03-29T10:12:32.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.837+0000 7f41de000700 1 Processor -- start 2024-03-29T10:12:32.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.837+0000 7f41de000700 1 -- start start 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.839+0000 7f41de000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41d80ceec0 0x7f41d81612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.839+0000 7f41de000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f41d8161800 0x7f41d80b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.839+0000 7f41de000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d8161c50 con 0x7f41d815d640 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.839+0000 7f41de000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d80d30d0 con 0x7f41d80ceec0 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.839+0000 7f41de000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d80b6840 con 0x7f41d8161800 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.839+0000 7f41d6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41d80ceec0 0x7f41d81612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.839+0000 7f41d6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41d80ceec0 0x7f41d81612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44706/0 (socket says 172.21.15.12:44706) 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.839+0000 7f41d6a00700 1 -- 172.21.15.12:0/3907340927 learned_addr learned my addr 172.21.15.12:0/3907340927 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.840+0000 7f41d7e00700 1 --2- 172.21.15.12:0/3907340927 >> v2:172.21.15.159:3300/0 conn(0x7f41d8161800 0x7f41d80b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.840+0000 7f41d6a00700 1 -- 172.21.15.12:0/3907340927 >> v2:172.21.15.159:3300/0 conn(0x7f41d8161800 msgr2=0x7f41d80b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.840+0000 7f41d6a00700 1 --2- 172.21.15.12:0/3907340927 >> v2:172.21.15.159:3300/0 conn(0x7f41d8161800 0x7f41d80b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.840+0000 7f41d6a00700 1 -- 172.21.15.12:0/3907340927 >> v1:172.21.15.12:6789/0 conn(0x7f41d815d640 legacy=0x7f41d80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.840+0000 7f41d6000700 1 -- 172.21.15.12:0/3907340927 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900928230 0 0) 0x7f41c8001110 con 0x7f41d815d640 2024-03-29T10:12:32.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.840+0000 7f41d6a00700 1 -- 172.21.15.12:0/3907340927 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d80d2b30 con 0x7f41d80ceec0 2024-03-29T10:12:32.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.840+0000 7f41d6a00700 1 --2- 172.21.15.12:0/3907340927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 0x7f41d81612c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f41d8162360 tx=0x7f41c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3037d43b01be3ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.840+0000 7f41d6000700 1 -- 172.21.15.12:0/3907340927 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4011830 con 0x7f41d80ceec0 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.841+0000 7f41d6000700 1 -- 172.21.15.12:0/3907340927 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c4011990 con 0x7f41d80ceec0 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.841+0000 7f41d6000700 1 -- 172.21.15.12:0/3907340927 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4011d20 con 0x7f41d80ceec0 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.841+0000 7f41de000700 1 -- 172.21.15.12:0/3907340927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 msgr2=0x7f41d81612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.841+0000 7f41de000700 1 --2- 172.21.15.12:0/3907340927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 0x7f41d81612c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f41d8162360 tx=0x7f41c4009bd0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.841+0000 7f41de000700 1 -- 172.21.15.12:0/3907340927 shutdown_connections 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.841+0000 7f41de000700 1 --2- 172.21.15.12:0/3907340927 >> v2:172.21.15.159:3300/0 conn(0x7f41d8161800 0x7f41d80b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.841+0000 7f41de000700 1 --2- 172.21.15.12:0/3907340927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 0x7f41d81612c0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.841+0000 7f41de000700 1 -- 172.21.15.12:0/3907340927 >> 172.21.15.12:0/3907340927 conn(0x7f41d808aff0 msgr2=0x7f41d80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.841+0000 7f41de000700 1 -- 172.21.15.12:0/3907340927 shutdown_connections 2024-03-29T10:12:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.841+0000 7f41de000700 1 -- 172.21.15.12:0/3907340927 wait complete. 2024-03-29T10:12:32.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.843+0000 7f41de000700 1 Processor -- start 2024-03-29T10:12:32.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.843+0000 7f41de000700 1 -- start start 2024-03-29T10:12:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.844+0000 7f41de000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 0x7f41d80ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.844+0000 7f41de000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41d815d640 0x7f41d80cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.844+0000 7f41de000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41d8161800 0x7f41d80ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.844+0000 7f41de000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80cd980 con 0x7f41d80ceec0 2024-03-29T10:12:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.844+0000 7f41de000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80cdb70 con 0x7f41d8161800 2024-03-29T10:12:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.844+0000 7f41de000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80cde50 con 0x7f41d815d640 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.844+0000 7f41d7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 0x7f41d80ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.844+0000 7f41d7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 0x7f41d80ccf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44722/0 (socket says 172.21.15.12:44722) 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.844+0000 7f41d7400700 1 -- 172.21.15.12:0/3827755504 learned_addr learned my addr 172.21.15.12:0/3827755504 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41d7e00700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41d8161800 0x7f41d80ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41d7400700 1 -- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41d815d640 msgr2=0x7f41d80cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41d6a00700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41d815d640 0x7f41d80cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41d7400700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41d815d640 0x7f41d80cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41d7400700 1 -- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41d8161800 msgr2=0x7f41d80ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41d7400700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41d8161800 0x7f41d80ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41d7400700 1 -- 172.21.15.12:0/3827755504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c4009550 con 0x7f41d80ceec0 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41d7e00700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41d8161800 0x7f41d80ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41d6a00700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41d815d640 0x7f41d80cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41d7400700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 0x7f41d80ccf00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c750 tx=0x7f41c800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41cf400700 1 -- 172.21.15.12:0/3827755504 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c80054e0 con 0x7f41d80ceec0 2024-03-29T10:12:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41cf400700 1 -- 172.21.15.12:0/3827755504 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c8003680 con 0x7f41d80ceec0 2024-03-29T10:12:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41de000700 1 -- 172.21.15.12:0/3827755504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d80ce130 con 0x7f41d80ceec0 2024-03-29T10:12:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41cf400700 1 -- 172.21.15.12:0/3827755504 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8003a30 con 0x7f41d80ceec0 2024-03-29T10:12:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.845+0000 7f41de000700 1 -- 172.21.15.12:0/3827755504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d80ce6f0 con 0x7f41d80ceec0 2024-03-29T10:12:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.846+0000 7f41de000700 1 -- 172.21.15.12:0/3827755504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d8067720 con 0x7f41d80ceec0 2024-03-29T10:12:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.847+0000 7f41cf400700 1 -- 172.21.15.12:0/3827755504 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f41c8003b90 con 0x7f41d80ceec0 2024-03-29T10:12:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.848+0000 7f41cf400700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41b806adb0 0x7f41b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.848+0000 7f41cf400700 1 -- 172.21.15.12:0/3827755504 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f41c8088680 con 0x7f41d80ceec0 2024-03-29T10:12:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.849+0000 7f41d6a00700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41b806adb0 0x7f41b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.849+0000 7f41d6a00700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41b806adb0 0x7f41b806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f41d815cc70 tx=0x7f41c4008030 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:32.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:32.853+0000 7f41cf400700 1 -- 172.21.15.12:0/3827755504 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f41c8057370 con 0x7f41d80ceec0 2024-03-29T10:12:33.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.149+0000 7f41de000700 1 -- 172.21.15.12:0/3827755504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f41d8161c50 con 0x7f41d80ceec0 2024-03-29T10:12:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.233+0000 7f41cf400700 1 -- 172.21.15.12:0/3827755504 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v676) v1 ==== 183+0+0 (secure 0 0 0) 0x7f41d8161c50 con 0x7f41d80ceec0 2024-03-29T10:12:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 -- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41b806adb0 msgr2=0x7f41b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41b806adb0 0x7f41b806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f41d815cc70 tx=0x7f41c4008030 comp rx=0 tx=0).stop 2024-03-29T10:12:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 -- 172.21.15.12:0/3827755504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 msgr2=0x7f41d80ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 0x7f41d80ccf00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c750 tx=0x7f41c800cb10 comp rx=0 tx=0).stop 2024-03-29T10:12:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 -- 172.21.15.12:0/3827755504 shutdown_connections 2024-03-29T10:12:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f41b806adb0 0x7f41b806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41d8161800 0x7f41d80ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41d80ceec0 0x7f41d80ccf00 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 --2- 172.21.15.12:0/3827755504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41d815d640 0x7f41d80cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 -- 172.21.15.12:0/3827755504 >> 172.21.15.12:0/3827755504 conn(0x7f41d808aff0 msgr2=0x7f41d80d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 -- 172.21.15.12:0/3827755504 shutdown_connections 2024-03-29T10:12:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.236+0000 7f41de000700 1 -- 172.21.15.12:0/3827755504 wait complete. 2024-03-29T10:12:33.255 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2024-03-29T10:12:33.255 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2024-03-29T10:12:33.255 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T10:12:33.255 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-03-29T10:12:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.453+0000 7f0325600700 1 Processor -- start 2024-03-29T10:12:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.453+0000 7f0325600700 1 -- start start 2024-03-29T10:12:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.454+0000 7f0325600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0320160f10 0x7f03200d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.455+0000 7f0325600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03200ceda0 con 0x7f0320161590 2024-03-29T10:12:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.455+0000 7f0325600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03201612f0 con 0x7f03200cf230 2024-03-29T10:12:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.455+0000 7f0325600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03200ce470 con 0x7f0320160f10 2024-03-29T10:12:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.455+0000 7f031f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0320161590 0x7f03200b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59734/0 (socket says 172.21.15.12:59734) 2024-03-29T10:12:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.455+0000 7f031f400700 1 -- 172.21.15.12:0/2002858712 learned_addr learned my addr 172.21.15.12:0/2002858712 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.455+0000 7f031ea00700 1 --2- 172.21.15.12:0/2002858712 >> v2:172.21.15.159:3300/0 conn(0x7f0320160f10 0x7f03200d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.455+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481194148 0 0) 0x7f03200ceda0 con 0x7f0320161590 2024-03-29T10:12:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.455+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03040035b0 con 0x7f0320161590 2024-03-29T10:12:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838333779 0 0) 0x7f03201612f0 con 0x7f03200cf230 2024-03-29T10:12:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03200ceda0 con 0x7f03200cf230 2024-03-29T10:12:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2825415323 0 0) 0x7f03040035b0 con 0x7f0320161590 2024-03-29T10:12:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 >> v2:172.21.15.159:3300/0 conn(0x7f0320160f10 msgr2=0x7f03200d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 --2- 172.21.15.12:0/2002858712 >> v2:172.21.15.159:3300/0 conn(0x7f0320160f10 0x7f03200d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 >> v1:172.21.15.159:6789/0 conn(0x7f03200cf230 legacy=0x7f03200d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03200ce750 con 0x7f0320161590 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f03180019f0 con 0x7f0320161590 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 >> v1:172.21.15.12:6789/0 conn(0x7f0320161590 legacy=0x7f03200b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0304003b90 0x7f0304003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0304004850 0x7f0304004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0304005210 0x7f030400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03200ce470 con 0x7f0304004850 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030400c140 con 0x7f0304003b90 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030400c300 con 0x7f0304005210 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031ea00700 1 --2- 172.21.15.12:0/2002858712 >> v2:172.21.15.159:3300/0 conn(0x7f0320160f10 0x7f03200d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f0317e00700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0304004850 0x7f0304004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031f400700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0304005210 0x7f030400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.456+0000 7f031ea00700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0304003b90 0x7f0304003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f0317e00700 1 -- 172.21.15.12:0/2002858712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0304005210 msgr2=0x7f030400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f0317e00700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0304005210 0x7f030400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f0317e00700 1 -- 172.21.15.12:0/2002858712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0304003b90 msgr2=0x7f0304003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f0317e00700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0304003b90 0x7f0304003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f0317e00700 1 -- 172.21.15.12:0/2002858712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040035b0 con 0x7f0304004850 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f031ea00700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0304003b90 0x7f0304003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f031f400700 1 -- 172.21.15.12:0/2002858712 reap_dead start 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f0317e00700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0304004850 0x7f0304004cd0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f03200d25a0 tx=0x7f0304005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84cb74987fb6893 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03080052d0 con 0x7f0304004850 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0308005450 con 0x7f0304004850 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.457+0000 7f031e000700 1 -- 172.21.15.12:0/2002858712 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03080038c0 con 0x7f0304004850 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.458+0000 7f0325600700 1 -- 172.21.15.12:0/2002858712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0304004850 msgr2=0x7f0304004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.458+0000 7f0325600700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0304004850 0x7f0304004cd0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f03200d25a0 tx=0x7f0304005880 comp rx=0 tx=0).stop 2024-03-29T10:12:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.458+0000 7f0325600700 1 -- 172.21.15.12:0/2002858712 shutdown_connections 2024-03-29T10:12:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.458+0000 7f0325600700 1 --2- 172.21.15.12:0/2002858712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0304004850 0x7f0304004cd0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.458+0000 7f0325600700 1 -- 172.21.15.12:0/2002858712 >> 172.21.15.12:0/2002858712 conn(0x7f032008aff0 msgr2=0x7f03200d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.458+0000 7f0325600700 1 -- 172.21.15.12:0/2002858712 shutdown_connections 2024-03-29T10:12:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.458+0000 7f0325600700 1 -- 172.21.15.12:0/2002858712 wait complete. 2024-03-29T10:12:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.459+0000 7f0325600700 1 Processor -- start 2024-03-29T10:12:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.460+0000 7f0325600700 1 -- start start 2024-03-29T10:12:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.460+0000 7f0325600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0304004850 0x7f0320155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f0325600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0320150fe0 0x7f0320153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f0325600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320153d50 0x7f032015ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f0325600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0320151510 con 0x7f0320153d50 2024-03-29T10:12:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f0325600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03201517f0 con 0x7f0304004850 2024-03-29T10:12:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f0325600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0320151ad0 con 0x7f0320150fe0 2024-03-29T10:12:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f031f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320153d50 0x7f032015ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f031f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320153d50 0x7f032015ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44748/0 (socket says 172.21.15.12:44748) 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f031f400700 1 -- 172.21.15.12:0/1992063602 learned_addr learned my addr 172.21.15.12:0/1992063602 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f031ea00700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0304004850 0x7f0320155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f0317e00700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0320150fe0 0x7f0320153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f031f400700 1 -- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0320150fe0 msgr2=0x7f0320153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f031f400700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0320150fe0 0x7f0320153810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f031f400700 1 -- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0304004850 msgr2=0x7f0320155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f031f400700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0304004850 0x7f0320155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f031f400700 1 -- 172.21.15.12:0/1992063602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0320151db0 con 0x7f0320153d50 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.461+0000 7f0317e00700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0320150fe0 0x7f0320153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.462+0000 7f031f400700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320153d50 0x7f032015ae20 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0320161970 tx=0x7f0318007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.462+0000 7f0316a00700 1 -- 172.21.15.12:0/1992063602 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031800ebc0 con 0x7f0320153d50 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.462+0000 7f0316a00700 1 -- 172.21.15.12:0/1992063602 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031800ed20 con 0x7f0320153d50 2024-03-29T10:12:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.462+0000 7f0316a00700 1 -- 172.21.15.12:0/1992063602 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318013660 con 0x7f0320153d50 2024-03-29T10:12:33.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.462+0000 7f0325600700 1 -- 172.21.15.12:0/1992063602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0320155e70 con 0x7f0320153d50 2024-03-29T10:12:33.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.462+0000 7f0325600700 1 -- 172.21.15.12:0/1992063602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032015b540 con 0x7f0320153d50 2024-03-29T10:12:33.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.463+0000 7f0325600700 1 -- 172.21.15.12:0/1992063602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0320067720 con 0x7f0320153d50 2024-03-29T10:12:33.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.464+0000 7f0316a00700 1 -- 172.21.15.12:0/1992063602 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f03180137c0 con 0x7f0320153d50 2024-03-29T10:12:33.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.465+0000 7f0316a00700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02fc06adb0 0x7f02fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:33.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.465+0000 7f0316a00700 1 -- 172.21.15.12:0/1992063602 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f0318089a30 con 0x7f0320153d50 2024-03-29T10:12:33.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.465+0000 7f031ea00700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02fc06adb0 0x7f02fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:33.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.466+0000 7f031ea00700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02fc06adb0 0x7f02fc06d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f03200684a0 tx=0x7f0310008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.470+0000 7f0316a00700 1 -- 172.21.15.12:0/1992063602 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0318054ab0 con 0x7f0320153d50 2024-03-29T10:12:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:33.772+0000 7f0325600700 1 -- 172.21.15.12:0/1992063602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f0320067720 con 0x7f0320153d50 2024-03-29T10:12:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.239+0000 7f0316a00700 1 -- 172.21.15.12:0/1992063602 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v677) v1 ==== 151+0+0 (secure 0 0 0) 0x7f03180580d0 con 0x7f0320153d50 2024-03-29T10:12:34.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 -- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02fc06adb0 msgr2=0x7f02fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:34.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02fc06adb0 0x7f02fc06d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f03200684a0 tx=0x7f0310008040 comp rx=0 tx=0).stop 2024-03-29T10:12:34.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 -- 172.21.15.12:0/1992063602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320153d50 msgr2=0x7f032015ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:34.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320153d50 0x7f032015ae20 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0320161970 tx=0x7f0318007c90 comp rx=0 tx=0).stop 2024-03-29T10:12:34.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 -- 172.21.15.12:0/1992063602 shutdown_connections 2024-03-29T10:12:34.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02fc06adb0 0x7f02fc06d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320153d50 0x7f032015ae20 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0304004850 0x7f0320155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 --2- 172.21.15.12:0/1992063602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0320150fe0 0x7f0320153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 -- 172.21.15.12:0/1992063602 >> 172.21.15.12:0/1992063602 conn(0x7f032008aff0 msgr2=0x7f0320152820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:34.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 -- 172.21.15.12:0/1992063602 shutdown_connections 2024-03-29T10:12:34.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.242+0000 7f0325600700 1 -- 172.21.15.12:0/1992063602 wait complete. 2024-03-29T10:12:34.262 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T10:12:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.465+0000 7fafd1400700 1 Processor -- start 2024-03-29T10:12:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.466+0000 7fafd1400700 1 -- start start 2024-03-29T10:12:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.467+0000 7fafd1400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fafcc0d16a0 0x7fafcc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.467+0000 7fafd1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 0x7fafcc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.467+0000 7fafd1400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc161420 con 0x7fafcc15d9d0 2024-03-29T10:12:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.467+0000 7fafd1400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafcc0d2820 con 0x7fafcc0d16a0 2024-03-29T10:12:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.467+0000 7fafd1400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0d4340 con 0x7fafcc0d2370 2024-03-29T10:12:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.467+0000 7fafc9e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fafcc0d16a0 0x7fafcc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.467+0000 7fafc9e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fafcc0d16a0 0x7fafcc0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44752/0 (socket says 172.21.15.12:44752) 2024-03-29T10:12:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.467+0000 7fafc9e00700 1 -- 172.21.15.12:0/2947534271 learned_addr learned my addr 172.21.15.12:0/2947534271 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.467+0000 7fafcb200700 1 --2- 172.21.15.12:0/2947534271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 0x7fafcc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.468+0000 7fafc9e00700 1 -- 172.21.15.12:0/2947534271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 msgr2=0x7fafcc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.468+0000 7fafc9e00700 1 --2- 172.21.15.12:0/2947534271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 0x7fafcc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.468+0000 7fafc9e00700 1 -- 172.21.15.12:0/2947534271 >> v1:172.21.15.159:6789/0 conn(0x7fafcc15d9d0 legacy=0x7fafcc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.468+0000 7fafc9e00700 1 -- 172.21.15.12:0/2947534271 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc161700 con 0x7fafcc0d16a0 2024-03-29T10:12:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.468+0000 7fafcb200700 1 --2- 172.21.15.12:0/2947534271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 0x7fafcc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.468+0000 7fafc9e00700 1 --2- 172.21.15.12:0/2947534271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc0d16a0 0x7fafcc0d1e30 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fafb8006fd0 tx=0x7fafb8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a61f018be2c8ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:34.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.468+0000 7fafc9400700 1 -- 172.21.15.12:0/2947534271 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8005e30 con 0x7fafcc0d16a0 2024-03-29T10:12:34.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.468+0000 7fafc9400700 1 -- 172.21.15.12:0/2947534271 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb8004030 con 0x7fafcc0d16a0 2024-03-29T10:12:34.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.468+0000 7fafc9400700 1 -- 172.21.15.12:0/2947534271 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80043e0 con 0x7fafcc0d16a0 2024-03-29T10:12:34.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.469+0000 7fafd1400700 1 -- 172.21.15.12:0/2947534271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc0d16a0 msgr2=0x7fafcc0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:34.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.469+0000 7fafd1400700 1 --2- 172.21.15.12:0/2947534271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc0d16a0 0x7fafcc0d1e30 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fafb8006fd0 tx=0x7fafb8009bd0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.469+0000 7fafd1400700 1 -- 172.21.15.12:0/2947534271 shutdown_connections 2024-03-29T10:12:34.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.469+0000 7fafd1400700 1 --2- 172.21.15.12:0/2947534271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 0x7fafcc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.469+0000 7fafd1400700 1 --2- 172.21.15.12:0/2947534271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc0d16a0 0x7fafcc0d1e30 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.469+0000 7fafd1400700 1 -- 172.21.15.12:0/2947534271 >> 172.21.15.12:0/2947534271 conn(0x7fafcc08aff0 msgr2=0x7fafcc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:34.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.469+0000 7fafd1400700 1 -- 172.21.15.12:0/2947534271 shutdown_connections 2024-03-29T10:12:34.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.469+0000 7fafd1400700 1 -- 172.21.15.12:0/2947534271 wait complete. 2024-03-29T10:12:34.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.470+0000 7fafd1400700 1 Processor -- start 2024-03-29T10:12:34.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.471+0000 7fafd1400700 1 -- start start 2024-03-29T10:12:34.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafd1400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc0d16a0 0x7fafcc0ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:34.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafd1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 0x7fafcc14c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:34.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafd1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc15d9d0 0x7fafcc0cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafd1400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0cbf40 con 0x7fafcc15d9d0 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafd1400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0cc220 con 0x7fafcc0d16a0 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafd1400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0cc500 con 0x7fafcc0d2370 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafcb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc15d9d0 0x7fafcc0cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafcb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc15d9d0 0x7fafcc0cb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44760/0 (socket says 172.21.15.12:44760) 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafcb200700 1 -- 172.21.15.12:0/2690885966 learned_addr learned my addr 172.21.15.12:0/2690885966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafca800700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc0d16a0 0x7fafcc0ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafc9e00700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 0x7fafcc14c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafcb200700 1 -- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 msgr2=0x7fafcc14c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafcb200700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 0x7fafcc14c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafcb200700 1 -- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc0d16a0 msgr2=0x7fafcc0ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafcb200700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc0d16a0 0x7fafcc0ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafcb200700 1 -- 172.21.15.12:0/2690885966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb8009500 con 0x7fafcc15d9d0 2024-03-29T10:12:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.472+0000 7fafca800700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc0d16a0 0x7fafcc0ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.473+0000 7fafcb200700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc15d9d0 0x7fafcc0cb640 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fafc4007ef0 tx=0x7fafc400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.473+0000 7fafc2a00700 1 -- 172.21.15.12:0/2690885966 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc400de60 con 0x7fafcc15d9d0 2024-03-29T10:12:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.473+0000 7fafc2a00700 1 -- 172.21.15.12:0/2690885966 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc400f040 con 0x7fafcc15d9d0 2024-03-29T10:12:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.473+0000 7fafd1400700 1 -- 172.21.15.12:0/2690885966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafcc0cc7e0 con 0x7fafcc15d9d0 2024-03-29T10:12:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.473+0000 7fafc2a00700 1 -- 172.21.15.12:0/2690885966 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4003c60 con 0x7fafcc15d9d0 2024-03-29T10:12:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.473+0000 7fafd1400700 1 -- 172.21.15.12:0/2690885966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafcc14ce00 con 0x7fafcc15d9d0 2024-03-29T10:12:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.474+0000 7fafd1400700 1 -- 172.21.15.12:0/2690885966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafcc0d1ca0 con 0x7fafcc15d9d0 2024-03-29T10:12:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.475+0000 7fafc2a00700 1 -- 172.21.15.12:0/2690885966 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fafc4004030 con 0x7fafcc15d9d0 2024-03-29T10:12:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.476+0000 7fafc2a00700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafac06adb0 0x7fafac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.476+0000 7fafc2a00700 1 -- 172.21.15.12:0/2690885966 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fafc40896d0 con 0x7fafcc15d9d0 2024-03-29T10:12:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.476+0000 7fafca800700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafac06adb0 0x7fafac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.477+0000 7fafca800700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafac06adb0 0x7fafac06d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fafbc000f60 tx=0x7fafbc009040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.480+0000 7fafc2a00700 1 -- 172.21.15.12:0/2690885966 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fafc4054770 con 0x7fafcc15d9d0 2024-03-29T10:12:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.694+0000 7fafd1400700 1 -- 172.21.15.12:0/2690885966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fafcc0ccd00 con 0x7fafcc15d9d0 2024-03-29T10:12:34.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.700+0000 7fafc2a00700 1 -- 172.21.15.12:0/2690885966 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7fafcc0ccd00 con 0x7fafcc15d9d0 2024-03-29T10:12:34.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.702+0000 7fafd1400700 1 -- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafac06adb0 msgr2=0x7fafac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:34.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.702+0000 7fafd1400700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafac06adb0 0x7fafac06d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fafbc000f60 tx=0x7fafbc009040 comp rx=0 tx=0).stop 2024-03-29T10:12:34.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.702+0000 7fafd1400700 1 -- 172.21.15.12:0/2690885966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc15d9d0 msgr2=0x7fafcc0cb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.702+0000 7fafd1400700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc15d9d0 0x7fafcc0cb640 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fafc4007ef0 tx=0x7fafc400d3b0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.702+0000 7fafd1400700 1 -- 172.21.15.12:0/2690885966 shutdown_connections 2024-03-29T10:12:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.702+0000 7fafd1400700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fafac06adb0 0x7fafac06d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.702+0000 7fafd1400700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafcc15d9d0 0x7fafcc0cb640 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.702+0000 7fafd1400700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc0d16a0 0x7fafcc0ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.702+0000 7fafd1400700 1 --2- 172.21.15.12:0/2690885966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0d2370 0x7fafcc14c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.702+0000 7fafd1400700 1 -- 172.21.15.12:0/2690885966 >> 172.21.15.12:0/2690885966 conn(0x7fafcc08aff0 msgr2=0x7fafcc0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.703+0000 7fafd1400700 1 -- 172.21.15.12:0/2690885966 shutdown_connections 2024-03-29T10:12:34.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:34.703+0000 7fafd1400700 1 -- 172.21.15.12:0/2690885966 wait complete. 2024-03-29T10:12:34.720 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:12:34.720 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:12:34.806 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:12:34.896 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:12:34.896 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:12:34.896 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:12:34.941 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:12:34.941 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:12:34.941 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:12:34.988 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:12:34.988 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:12:34.988 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:12:35.032 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:12:35.032 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:12:35.032 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:12:35.077 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:12:35.077 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T10:12:35.077 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:12:35.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.275+0000 7f2e6d200700 1 Processor -- start 2024-03-29T10:12:35.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.275+0000 7f2e6d200700 1 -- start start 2024-03-29T10:12:35.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.276+0000 7f2e6d200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2e68160f10 0x7f2e680d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:35.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.276+0000 7f2e6d200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2e68161590 0x7f2e68161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:35.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.276+0000 7f2e6d200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e680ceda0 con 0x7f2e680cf230 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.276+0000 7f2e6d200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e68161f80 con 0x7f2e68161590 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.276+0000 7f2e6d200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e681612f0 con 0x7f2e68160f10 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.277+0000 7f2e66800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2e68161590 0x7f2e68161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.277+0000 7f2e66800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2e68161590 0x7f2e68161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44774/0 (socket says 172.21.15.12:44774) 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.277+0000 7f2e66800700 1 -- 172.21.15.12:0/2099351812 learned_addr learned my addr 172.21.15.12:0/2099351812 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.277+0000 7f2e65e00700 1 --2- 172.21.15.12:0/2099351812 >> v2:172.21.15.159:3300/0 conn(0x7f2e68160f10 0x7f2e680d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.277+0000 7f2e66800700 1 -- 172.21.15.12:0/2099351812 >> v2:172.21.15.159:3300/0 conn(0x7f2e68160f10 msgr2=0x7f2e680d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.277+0000 7f2e66800700 1 --2- 172.21.15.12:0/2099351812 >> v2:172.21.15.159:3300/0 conn(0x7f2e68160f10 0x7f2e680d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.277+0000 7f2e66800700 1 -- 172.21.15.12:0/2099351812 >> v1:172.21.15.159:6789/0 conn(0x7f2e680cf230 legacy=0x7f2e680d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.277+0000 7f2e66800700 1 -- 172.21.15.12:0/2099351812 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e680ce470 con 0x7f2e68161590 2024-03-29T10:12:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.277+0000 7f2e66800700 1 --2- 172.21.15.12:0/2099351812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 0x7f2e68161a40 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f2e680d1080 tx=0x7f2e6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ff71fc672ef902 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:35.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e64a00700 1 -- 172.21.15.12:0/2099351812 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000e070 con 0x7f2e68161590 2024-03-29T10:12:35.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e64a00700 1 -- 172.21.15.12:0/2099351812 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6000b910 con 0x7f2e68161590 2024-03-29T10:12:35.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e64a00700 1 -- 172.21.15.12:0/2099351812 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000bcc0 con 0x7f2e68161590 2024-03-29T10:12:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e6d200700 1 -- 172.21.15.12:0/2099351812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 msgr2=0x7f2e68161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e6d200700 1 --2- 172.21.15.12:0/2099351812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 0x7f2e68161a40 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f2e680d1080 tx=0x7f2e6000b330 comp rx=0 tx=0).stop 2024-03-29T10:12:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e6d200700 1 -- 172.21.15.12:0/2099351812 shutdown_connections 2024-03-29T10:12:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e6d200700 1 --2- 172.21.15.12:0/2099351812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 0x7f2e68161a40 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e6d200700 1 --2- 172.21.15.12:0/2099351812 >> v2:172.21.15.159:3300/0 conn(0x7f2e68160f10 0x7f2e680d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e6d200700 1 -- 172.21.15.12:0/2099351812 >> 172.21.15.12:0/2099351812 conn(0x7f2e6808aff0 msgr2=0x7f2e680d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e6d200700 1 -- 172.21.15.12:0/2099351812 shutdown_connections 2024-03-29T10:12:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.278+0000 7f2e6d200700 1 -- 172.21.15.12:0/2099351812 wait complete. 2024-03-29T10:12:35.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.281+0000 7f2e6d200700 1 Processor -- start 2024-03-29T10:12:35.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.281+0000 7f2e6d200700 1 -- start start 2024-03-29T10:12:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e6d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e680cf230 0x7f2e6814e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e6d200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e68160f10 0x7f2e6809f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e6d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 0x7f2e6809fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e6d200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e681f3470 con 0x7f2e68161590 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e6d200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680a0c00 con 0x7f2e68160f10 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e6d200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680a0360 con 0x7f2e680cf230 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e66800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 0x7f2e6809fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e65e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e680cf230 0x7f2e6814e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e65e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e680cf230 0x7f2e6814e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:55298/0 (socket says 172.21.15.12:55298) 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e66800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 0x7f2e6809fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44782/0 (socket says 172.21.15.12:44782) 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e65e00700 1 -- 172.21.15.12:0/1708916336 learned_addr learned my addr 172.21.15.12:0/1708916336 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.282+0000 7f2e65400700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e68160f10 0x7f2e6809f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:35.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.283+0000 7f2e66800700 1 -- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e680cf230 msgr2=0x7f2e6814e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:35.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.283+0000 7f2e66800700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e680cf230 0x7f2e6814e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:35.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.283+0000 7f2e66800700 1 -- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e68160f10 msgr2=0x7f2e6809f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:35.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.283+0000 7f2e66800700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e68160f10 0x7f2e6809f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:35.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.283+0000 7f2e66800700 1 -- 172.21.15.12:0/1708916336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e60009b30 con 0x7f2e68161590 2024-03-29T10:12:35.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.283+0000 7f2e66800700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 0x7f2e6809fb30 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f2e60015000 tx=0x7f2e60004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:35.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.283+0000 7f2e5e000700 1 -- 172.21.15.12:0/1708916336 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60004880 con 0x7f2e68161590 2024-03-29T10:12:35.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.283+0000 7f2e5e000700 1 -- 172.21.15.12:0/1708916336 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e600099b0 con 0x7f2e68161590 2024-03-29T10:12:35.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.283+0000 7f2e5e000700 1 -- 172.21.15.12:0/1708916336 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6001c680 con 0x7f2e68161590 2024-03-29T10:12:35.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.283+0000 7f2e6d200700 1 -- 172.21.15.12:0/1708916336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e680a0640 con 0x7f2e68161590 2024-03-29T10:12:35.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.284+0000 7f2e6d200700 1 -- 172.21.15.12:0/1708916336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6814eea0 con 0x7f2e68161590 2024-03-29T10:12:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.284+0000 7f2e6d200700 1 -- 172.21.15.12:0/1708916336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e68067720 con 0x7f2e68161590 2024-03-29T10:12:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.285+0000 7f2e5e000700 1 -- 172.21.15.12:0/1708916336 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2e6001c7e0 con 0x7f2e68161590 2024-03-29T10:12:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.286+0000 7f2e5e000700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2e4806adb0 0x7f2e4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.286+0000 7f2e5e000700 1 -- 172.21.15.12:0/1708916336 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f2e6008ad40 con 0x7f2e68161590 2024-03-29T10:12:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.287+0000 7f2e65e00700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2e4806adb0 0x7f2e4806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.288+0000 7f2e65e00700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2e4806adb0 0x7f2e4806d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009c00 tx=0x7f2e58009210 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:35.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.291+0000 7f2e5e000700 1 -- 172.21.15.12:0/1708916336 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e60055d40 con 0x7f2e68161590 2024-03-29T10:12:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.551+0000 7f2e6d200700 1 -- 172.21.15.12:0/1708916336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e680ccfb0 con 0x7f2e68161590 2024-03-29T10:12:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.552+0000 7f2e5e000700 1 -- 172.21.15.12:0/1708916336 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f2e60059360 con 0x7f2e68161590 2024-03-29T10:12:35.552 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:35.552 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6452,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2137480356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2137480356},{"type":"v1","addr":"172.21.15.159:6836","nonce":2137480356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":6455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2356039533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2356039533},{"type":"v1","addr":"172.21.15.159:6837","nonce":2356039533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":7153,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2916743006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2916743006},{"type":"v1","addr":"172.21.15.12:6835","nonce":2916743006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41},{"gid":7818,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2582576288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2582576288},{"type":"v1","addr":"172.21.15.12:6834","nonce":2582576288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48}],"filesystems":[]} 2024-03-29T10:12:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.554+0000 7f2e6d200700 1 -- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2e4806adb0 msgr2=0x7f2e4806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.554+0000 7f2e6d200700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2e4806adb0 0x7f2e4806d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009c00 tx=0x7f2e58009210 comp rx=0 tx=0).stop 2024-03-29T10:12:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.554+0000 7f2e6d200700 1 -- 172.21.15.12:0/1708916336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 msgr2=0x7f2e6809fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.554+0000 7f2e6d200700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 0x7f2e6809fb30 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f2e60015000 tx=0x7f2e60004ad0 comp rx=0 tx=0).stop 2024-03-29T10:12:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.554+0000 7f2e6d200700 1 -- 172.21.15.12:0/1708916336 shutdown_connections 2024-03-29T10:12:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.554+0000 7f2e6d200700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2e4806adb0 0x7f2e4806d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.554+0000 7f2e6d200700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e68161590 0x7f2e6809fb30 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.554+0000 7f2e6d200700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e680cf230 0x7f2e6814e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.554+0000 7f2e6d200700 1 --2- 172.21.15.12:0/1708916336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e68160f10 0x7f2e6809f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.555+0000 7f2e6d200700 1 -- 172.21.15.12:0/1708916336 >> 172.21.15.12:0/1708916336 conn(0x7f2e6808aff0 msgr2=0x7f2e680cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.555+0000 7f2e6d200700 1 -- 172.21.15.12:0/1708916336 shutdown_connections 2024-03-29T10:12:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:35.555+0000 7f2e6d200700 1 -- 172.21.15.12:0/1708916336 wait complete. 2024-03-29T10:12:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 48 2024-03-29T10:12:35.573 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T10:12:35.573 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T10:12:35.573 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T10:12:35.573 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:12:35.574 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T10:12:35.574 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T10:12:35.574 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T10:12:35.574 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:12:35.574 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T10:12:35.574 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T10:12:35.574 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T10:12:35.574 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:12:35.575 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T10:12:35.575 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T10:12:35.575 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T10:12:35.575 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:12:35.590 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T10:12:35.590 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T10:12:35.591 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T10:12:35.591 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T10:12:35.592 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T10:12:35.592 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T10:12:35.634 INFO:tasks.ceph.mds.a:Started 2024-03-29T10:12:35.635 DEBUG:teuthology.parallel:result is None 2024-03-29T10:12:35.635 INFO:tasks.ceph.mds.b:Started 2024-03-29T10:12:35.636 DEBUG:teuthology.parallel:result is None 2024-03-29T10:12:35.638 INFO:tasks.ceph.mds.d:Started 2024-03-29T10:12:35.638 DEBUG:teuthology.parallel:result is None 2024-03-29T10:12:35.778 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:12:35.779 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T10:12:35.780 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T10:12:41.590 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T10:12:41.590 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T10:12:41.595 INFO:tasks.ceph.mds.c:Started 2024-03-29T10:12:41.595 DEBUG:teuthology.parallel:result is None 2024-03-29T10:12:41.596 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T10:12:41.751 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T10:12:41.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.810+0000 7fc6bfe00700 1 Processor -- start 2024-03-29T10:12:41.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.810+0000 7fc6bfe00700 1 -- start start 2024-03-29T10:12:41.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.811+0000 7fc6bfe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b81611b0 0x7fc6b815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:41.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.811+0000 7fc6bfe00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b80ce270 con 0x7fc6b815f4d0 2024-03-29T10:12:41.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.811+0000 7fc6bfe00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b815b600 con 0x7fc6b80cee10 2024-03-29T10:12:41.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.811+0000 7fc6bfe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80d2760 con 0x7fc6b81611b0 2024-03-29T10:12:41.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.811+0000 7fc6bca00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc6b815f4d0 0x7fc6b80ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44308/0 (socket says 172.21.15.12:44308) 2024-03-29T10:12:41.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.811+0000 7fc6bca00700 1 -- 172.21.15.12:0/1405030771 learned_addr learned my addr 172.21.15.12:0/1405030771 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:41.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6bd400700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b81611b0 0x7fc6b815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:41.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996066347 0 0) 0x7fc6b80ce270 con 0x7fc6b815f4d0 2024-03-29T10:12:41.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a00035b0 con 0x7fc6b815f4d0 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1708247504 0 0) 0x7fc6a00035b0 con 0x7fc6b815f4d0 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b81611b0 msgr2=0x7fc6b815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b81611b0 0x7fc6b815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 >> v1:172.21.15.159:6789/0 conn(0x7fc6b80cee10 legacy=0x7fc6b8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b80d2a40 con 0x7fc6b815f4d0 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fc6a4001670 con 0x7fc6b815f4d0 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 >> v1:172.21.15.12:6789/0 conn(0x7fc6b815f4d0 legacy=0x7fc6b80ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6a0003a30 0x7fc6a0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6a0004850 0x7fc6a0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b81611b0 0x7fc6a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80d2760 con 0x7fc6a0004850 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a000bac0 con 0x7fc6a0003a30 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.812+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a000bc80 con 0x7fc6b81611b0 2024-03-29T10:12:41.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.813+0000 7fc6bd400700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6a0003a30 0x7fc6a0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:41.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.813+0000 7fc6bca00700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6a0004850 0x7fc6a0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:41.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.813+0000 7fc6bde00700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b81611b0 0x7fc6a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:41.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.813+0000 7fc6bca00700 1 -- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b81611b0 msgr2=0x7fc6a0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:41.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.813+0000 7fc6bca00700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b81611b0 0x7fc6a0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:41.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.813+0000 7fc6bca00700 1 -- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6a0003a30 msgr2=0x7fc6a0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:41.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.813+0000 7fc6bca00700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6a0003a30 0x7fc6a0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:41.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.813+0000 7fc6bca00700 1 -- 172.21.15.12:0/1405030771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a00035b0 con 0x7fc6a0004850 2024-03-29T10:12:41.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.813+0000 7fc6bca00700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6a0004850 0x7fc6a0004cd0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc6b815f8b0 tx=0x7fc6a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9c9589e5ca3d7c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:41.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.814+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4007c10 con 0x7fc6a0004850 2024-03-29T10:12:41.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.814+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a400cda0 con 0x7fc6a0004850 2024-03-29T10:12:41.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.814+0000 7fc6b7e00700 1 -- 172.21.15.12:0/1405030771 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4005ba0 con 0x7fc6a0004850 2024-03-29T10:12:41.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.814+0000 7fc6bfe00700 1 -- 172.21.15.12:0/1405030771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6a0004850 msgr2=0x7fc6a0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:41.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.814+0000 7fc6bfe00700 1 --2- 172.21.15.12:0/1405030771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6a0004850 0x7fc6a0004cd0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc6b815f8b0 tx=0x7fc6a0005880 comp rx=0 tx=0).stop 2024-03-29T10:12:41.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.814+0000 7fc6bde00700 1 -- 172.21.15.12:0/1405030771 reap_dead start 2024-03-29T10:12:41.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.814+0000 7fc6bfe00700 1 -- 172.21.15.12:0/1405030771 shutdown_connections 2024-03-29T10:12:41.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.814+0000 7fc6bfe00700 1 -- 172.21.15.12:0/1405030771 >> 172.21.15.12:0/1405030771 conn(0x7fc6b808aff0 msgr2=0x7fc6b80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:41.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.814+0000 7fc6bfe00700 1 -- 172.21.15.12:0/1405030771 shutdown_connections 2024-03-29T10:12:41.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.814+0000 7fc6bfe00700 1 -- 172.21.15.12:0/1405030771 wait complete. 2024-03-29T10:12:41.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.816+0000 7fc6bfe00700 1 Processor -- start 2024-03-29T10:12:41.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.816+0000 7fc6bfe00700 1 -- start start 2024-03-29T10:12:41.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.817+0000 7fc6bfe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6a0004850 0x7fc6b80cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.817+0000 7fc6bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6b81f6410 0x7fc6b81f2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.817+0000 7fc6bfe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b8158750 0x7fc6b8158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.817+0000 7fc6bfe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b8159110 con 0x7fc6b81f6410 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.817+0000 7fc6bfe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80cb130 con 0x7fc6a0004850 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.817+0000 7fc6bfe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80c83d0 con 0x7fc6b8158750 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.817+0000 7fc6bca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6b81f6410 0x7fc6b81f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.817+0000 7fc6bca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6b81f6410 0x7fc6b81f2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41236/0 (socket says 172.21.15.12:41236) 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.817+0000 7fc6bca00700 1 -- 172.21.15.12:0/4089196066 learned_addr learned my addr 172.21.15.12:0/4089196066 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6bd400700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6a0004850 0x7fc6b80cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6bde00700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b8158750 0x7fc6b8158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6bca00700 1 -- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b8158750 msgr2=0x7fc6b8158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6bca00700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b8158750 0x7fc6b8158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6bca00700 1 -- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6a0004850 msgr2=0x7fc6b80cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:41.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6bca00700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6a0004850 0x7fc6b80cabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:41.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6bca00700 1 -- 172.21.15.12:0/4089196066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b80cb360 con 0x7fc6b81f6410 2024-03-29T10:12:41.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6bca00700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6b81f6410 0x7fc6b81f2780 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001040 tx=0x7fc6a4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:41.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6b5600700 1 -- 172.21.15.12:0/4089196066 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4001630 con 0x7fc6b81f6410 2024-03-29T10:12:41.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6b5600700 1 -- 172.21.15.12:0/4089196066 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a40070d0 con 0x7fc6b81f6410 2024-03-29T10:12:41.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6b5600700 1 -- 172.21.15.12:0/4089196066 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4007460 con 0x7fc6b81f6410 2024-03-29T10:12:41.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.818+0000 7fc6bfe00700 1 -- 172.21.15.12:0/4089196066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b80cb640 con 0x7fc6b81f6410 2024-03-29T10:12:41.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.819+0000 7fc6bfe00700 1 -- 172.21.15.12:0/4089196066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b81f2cc0 con 0x7fc6b81f6410 2024-03-29T10:12:41.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.819+0000 7fc6bfe00700 1 -- 172.21.15.12:0/4089196066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b8067720 con 0x7fc6b81f6410 2024-03-29T10:12:41.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.820+0000 7fc6b5600700 1 -- 172.21.15.12:0/4089196066 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc6a4019030 con 0x7fc6b81f6410 2024-03-29T10:12:41.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.821+0000 7fc6b5600700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc69806adb0 0x7fc69806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:41.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.821+0000 7fc6b5600700 1 -- 172.21.15.12:0/4089196066 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fc6a408a640 con 0x7fc6b81f6410 2024-03-29T10:12:41.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.822+0000 7fc6bd400700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc69806adb0 0x7fc69806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:41.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.822+0000 7fc6bd400700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc69806adb0 0x7fc69806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac009c60 tx=0x7fc6ac009210 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:41.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:41.826+0000 7fc6b5600700 1 -- 172.21.15.12:0/4089196066 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc6a4055750 con 0x7fc6b81f6410 2024-03-29T10:12:41.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-29T10:12:42.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.055+0000 7fc6bfe00700 1 -- 172.21.15.12:0/4089196066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc6b80c8510 con 0x7fc6b81f6410 2024-03-29T10:12:42.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.780+0000 7fc6b5600700 1 -- 172.21.15.12:0/4089196066 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v120) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc6a4058d70 con 0x7fc6b81f6410 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.783+0000 7fc6bfe00700 1 -- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc69806adb0 msgr2=0x7fc69806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.783+0000 7fc6bfe00700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc69806adb0 0x7fc69806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac009c60 tx=0x7fc6ac009210 comp rx=0 tx=0).stop 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.783+0000 7fc6bfe00700 1 -- 172.21.15.12:0/4089196066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6b81f6410 msgr2=0x7fc6b81f2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.783+0000 7fc6bfe00700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6b81f6410 0x7fc6b81f2780 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001040 tx=0x7fc6a4001a00 comp rx=0 tx=0).stop 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.784+0000 7fc6bfe00700 1 -- 172.21.15.12:0/4089196066 shutdown_connections 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.784+0000 7fc6bfe00700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc69806adb0 0x7fc69806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.784+0000 7fc6bfe00700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6b8158750 0x7fc6b8158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.784+0000 7fc6bfe00700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6a0004850 0x7fc6b80cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.784+0000 7fc6bfe00700 1 --2- 172.21.15.12:0/4089196066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6b81f6410 0x7fc6b81f2780 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.784+0000 7fc6bfe00700 1 -- 172.21.15.12:0/4089196066 >> 172.21.15.12:0/4089196066 conn(0x7fc6b808aff0 msgr2=0x7fc6b806d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:42.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.784+0000 7fc6bfe00700 1 -- 172.21.15.12:0/4089196066 shutdown_connections 2024-03-29T10:12:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:42.784+0000 7fc6bfe00700 1 -- 172.21.15.12:0/4089196066 wait complete. 2024-03-29T10:12:42.785 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T10:12:42.805 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T10:12:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.007+0000 7f6186a00700 1 Processor -- start 2024-03-29T10:12:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.007+0000 7f6186a00700 1 -- start start 2024-03-29T10:12:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.008+0000 7f6186a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6180072580 0x7f61800b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.008+0000 7f6186a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61800b2180 con 0x7f61800b3af0 2024-03-29T10:12:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.008+0000 7f6186a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618015b100 con 0x7f61800b2e50 2024-03-29T10:12:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.008+0000 7f6186a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f618015b3e0 con 0x7f6180072580 2024-03-29T10:12:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.009+0000 7f617f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6180072580 0x7f61800b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.009+0000 7f617fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f61800b3af0 0x7f618015a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44316/0 (socket says 172.21.15.12:44316) 2024-03-29T10:12:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.009+0000 7f617fe00700 1 -- 172.21.15.12:0/268623906 learned_addr learned my addr 172.21.15.12:0/268623906 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.009+0000 7f617ea00700 1 -- 172.21.15.12:0/268623906 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947581376 0 0) 0x7f61800b2180 con 0x7f61800b3af0 2024-03-29T10:12:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.009+0000 7f617ea00700 1 -- 172.21.15.12:0/268623906 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61640057c0 con 0x7f61800b3af0 2024-03-29T10:12:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.009+0000 7f617f400700 1 -- 172.21.15.12:0/268623906 >> v1:172.21.15.159:6789/0 conn(0x7f61800b2e50 legacy=0x7f61800709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.009+0000 7f617f400700 1 -- 172.21.15.12:0/268623906 >> v1:172.21.15.12:6789/0 conn(0x7f61800b3af0 legacy=0x7f618015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.009+0000 7f617f400700 1 -- 172.21.15.12:0/268623906 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61800b23f0 con 0x7f6180072580 2024-03-29T10:12:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.009+0000 7f617f400700 1 --2- 172.21.15.12:0/268623906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 0x7f61800b2910 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f616c009840 tx=0x7f616c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5b1a5426d25bd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.009+0000 7f617ea00700 1 -- 172.21.15.12:0/268623906 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c011830 con 0x7f6180072580 2024-03-29T10:12:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.010+0000 7f617ea00700 1 -- 172.21.15.12:0/268623906 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c011990 con 0x7f6180072580 2024-03-29T10:12:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.010+0000 7f617ea00700 1 -- 172.21.15.12:0/268623906 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c011d20 con 0x7f6180072580 2024-03-29T10:12:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.010+0000 7f6186a00700 1 -- 172.21.15.12:0/268623906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 msgr2=0x7f61800b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.010+0000 7f6186a00700 1 --2- 172.21.15.12:0/268623906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 0x7f61800b2910 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f616c009840 tx=0x7f616c009b50 comp rx=0 tx=0).stop 2024-03-29T10:12:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.010+0000 7f6186a00700 1 -- 172.21.15.12:0/268623906 shutdown_connections 2024-03-29T10:12:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.010+0000 7f6186a00700 1 --2- 172.21.15.12:0/268623906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 0x7f61800b2910 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.010+0000 7f6186a00700 1 -- 172.21.15.12:0/268623906 >> 172.21.15.12:0/268623906 conn(0x7f61800cf190 msgr2=0x7f61800cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.010+0000 7f6186a00700 1 -- 172.21.15.12:0/268623906 shutdown_connections 2024-03-29T10:12:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.010+0000 7f6186a00700 1 -- 172.21.15.12:0/268623906 wait complete. 2024-03-29T10:12:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.012+0000 7f6186a00700 1 Processor -- start 2024-03-29T10:12:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.012+0000 7f6186a00700 1 -- start start 2024-03-29T10:12:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.013+0000 7f6186a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 0x7f618014a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.013+0000 7f6186a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800b2e50 0x7f6180159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.013+0000 7f6186a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800b3af0 0x7f6180159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.013+0000 7f6186a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61800ccc00 con 0x7f6180072580 2024-03-29T10:12:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.013+0000 7f6186a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61800cbba0 con 0x7f61800b3af0 2024-03-29T10:12:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.013+0000 7f6186a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618014ac20 con 0x7f61800b2e50 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.013+0000 7f617fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 0x7f618014a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.013+0000 7f617fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 0x7f618014a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41260/0 (socket says 172.21.15.12:41260) 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.013+0000 7f617fe00700 1 -- 172.21.15.12:0/471478174 learned_addr learned my addr 172.21.15.12:0/471478174 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.013+0000 7f617f400700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800b2e50 0x7f6180159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f6184a00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800b3af0 0x7f6180159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f617fe00700 1 -- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800b2e50 msgr2=0x7f6180159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f617fe00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800b2e50 0x7f6180159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f617fe00700 1 -- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800b3af0 msgr2=0x7f6180159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f617fe00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800b3af0 0x7f6180159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f617fe00700 1 -- 172.21.15.12:0/471478174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0094a0 con 0x7f6180072580 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f617fe00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 0x7f618014a6e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f6170001eb0 tx=0x7f617000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f6177e00700 1 -- 172.21.15.12:0/471478174 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170005d70 con 0x7f6180072580 2024-03-29T10:12:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f6177e00700 1 -- 172.21.15.12:0/471478174 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61700070d0 con 0x7f6180072580 2024-03-29T10:12:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f6177e00700 1 -- 172.21.15.12:0/471478174 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170007460 con 0x7f6180072580 2024-03-29T10:12:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.014+0000 7f6186a00700 1 -- 172.21.15.12:0/471478174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61800cbe40 con 0x7f6180072580 2024-03-29T10:12:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.015+0000 7f6186a00700 1 -- 172.21.15.12:0/471478174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61800cc400 con 0x7f6180072580 2024-03-29T10:12:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.016+0000 7f6186a00700 1 -- 172.21.15.12:0/471478174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6180067720 con 0x7f6180072580 2024-03-29T10:12:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.017+0000 7f6177e00700 1 -- 172.21.15.12:0/471478174 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f61700075c0 con 0x7f6180072580 2024-03-29T10:12:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.017+0000 7f6177e00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f616006ade0 0x7f616006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.018+0000 7f6177e00700 1 -- 172.21.15.12:0/471478174 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f61700885d0 con 0x7f6180072580 2024-03-29T10:12:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.018+0000 7f617f400700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f616006ade0 0x7f616006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.019+0000 7f617f400700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f616006ade0 0x7f616006d2a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f616c009840 tx=0x7f616c008030 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.022+0000 7f6177e00700 1 -- 172.21.15.12:0/471478174 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6170053960 con 0x7f6180072580 2024-03-29T10:12:43.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.324+0000 7f6186a00700 1 -- 172.21.15.12:0/471478174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f61800cef70 con 0x7f6180072580 2024-03-29T10:12:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.325+0000 7f6177e00700 1 -- 172.21.15.12:0/471478174 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6170017090 con 0x7f6180072580 2024-03-29T10:12:43.325 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:43.325 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:12:43.325 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T10:12:43.325 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.325 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T10:12:43.325 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T10:12:43.325 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.325 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:12:43.325 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.326 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T10:12:43.327 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.329 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:12:43.331 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T10:12:43.332 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T10:12:43.333 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.334 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:12:43.335 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:12:43.337 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:12:43.338 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:12:43.338 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:12:43.338 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.338 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:43.338 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T10:12:43.338 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:12:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 -- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f616006ade0 msgr2=0x7f616006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f616006ade0 0x7f616006d2a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f616c009840 tx=0x7f616c008030 comp rx=0 tx=0).stop 2024-03-29T10:12:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 -- 172.21.15.12:0/471478174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 msgr2=0x7f618014a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 0x7f618014a6e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f6170001eb0 tx=0x7f617000c7b0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 -- 172.21.15.12:0/471478174 shutdown_connections 2024-03-29T10:12:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f616006ade0 0x7f616006d2a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800b3af0 0x7f6180159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6180072580 0x7f618014a6e0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 --2- 172.21.15.12:0/471478174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800b2e50 0x7f6180159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 -- 172.21.15.12:0/471478174 >> 172.21.15.12:0/471478174 conn(0x7f61800cf190 msgr2=0x7f618006e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 -- 172.21.15.12:0/471478174 shutdown_connections 2024-03-29T10:12:43.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.328+0000 7f6186a00700 1 -- 172.21.15.12:0/471478174 wait complete. 2024-03-29T10:12:43.346 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:12:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.543+0000 7f4eaf200700 1 Processor -- start 2024-03-29T10:12:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.544+0000 7f4eaf200700 1 -- start start 2024-03-29T10:12:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4eaf200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ea8160f10 0x7f4ea80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4eaf200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4ea80cf230 0x7f4ea80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4eaf200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ea8161590 0x7f4ea8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4eaf200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea8161f60 con 0x7f4ea8161590 2024-03-29T10:12:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4eaf200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea80ceda0 con 0x7f4ea80cf230 2024-03-29T10:12:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4eaf200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea80d23d0 con 0x7f4ea8160f10 2024-03-29T10:12:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4ead200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ea8161590 0x7f4ea8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4ead200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ea8161590 0x7f4ea8161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41270/0 (socket says 172.21.15.12:41270) 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4ead200700 1 -- 172.21.15.12:0/3748529286 learned_addr learned my addr 172.21.15.12:0/3748529286 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4ea7400700 1 --2- 172.21.15.12:0/3748529286 >> v2:172.21.15.159:3300/0 conn(0x7f4ea80cf230 0x7f4ea80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.545+0000 7f4ea7e00700 1 --2- 172.21.15.12:0/3748529286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ea8160f10 0x7f4ea80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.546+0000 7f4ead200700 1 -- 172.21.15.12:0/3748529286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ea8160f10 msgr2=0x7f4ea80d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.546+0000 7f4ead200700 1 --2- 172.21.15.12:0/3748529286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ea8160f10 0x7f4ea80d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.546+0000 7f4ead200700 1 -- 172.21.15.12:0/3748529286 >> v2:172.21.15.159:3300/0 conn(0x7f4ea80cf230 msgr2=0x7f4ea80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.546+0000 7f4ead200700 1 --2- 172.21.15.12:0/3748529286 >> v2:172.21.15.159:3300/0 conn(0x7f4ea80cf230 0x7f4ea80d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.546+0000 7f4ead200700 1 -- 172.21.15.12:0/3748529286 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea81612f0 con 0x7f4ea8161590 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.546+0000 7f4ea7400700 1 --2- 172.21.15.12:0/3748529286 >> v2:172.21.15.159:3300/0 conn(0x7f4ea80cf230 0x7f4ea80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.546+0000 7f4ead200700 1 --2- 172.21.15.12:0/3748529286 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8161590 0x7f4ea8161a20 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f4ea808c1d0 tx=0x7f4ea000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475453fe455c912f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.546+0000 7f4ea6a00700 1 -- 172.21.15.12:0/3748529286 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea000e070 con 0x7f4ea8161590 2024-03-29T10:12:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.546+0000 7f4ea6a00700 1 -- 172.21.15.12:0/3748529286 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea000b9c0 con 0x7f4ea8161590 2024-03-29T10:12:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.546+0000 7f4ea6a00700 1 -- 172.21.15.12:0/3748529286 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea000bd70 con 0x7f4ea8161590 2024-03-29T10:12:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.547+0000 7f4eaf200700 1 -- 172.21.15.12:0/3748529286 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8161590 msgr2=0x7f4ea8161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.547+0000 7f4eaf200700 1 --2- 172.21.15.12:0/3748529286 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8161590 0x7f4ea8161a20 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f4ea808c1d0 tx=0x7f4ea000b330 comp rx=0 tx=0).stop 2024-03-29T10:12:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.547+0000 7f4eaf200700 1 -- 172.21.15.12:0/3748529286 shutdown_connections 2024-03-29T10:12:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.547+0000 7f4eaf200700 1 --2- 172.21.15.12:0/3748529286 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8161590 0x7f4ea8161a20 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.547+0000 7f4eaf200700 1 --2- 172.21.15.12:0/3748529286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ea8160f10 0x7f4ea80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.547+0000 7f4eaf200700 1 --2- 172.21.15.12:0/3748529286 >> v2:172.21.15.159:3300/0 conn(0x7f4ea80cf230 0x7f4ea80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.547+0000 7f4eaf200700 1 -- 172.21.15.12:0/3748529286 >> 172.21.15.12:0/3748529286 conn(0x7f4ea808aff0 msgr2=0x7f4ea80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.547+0000 7f4eaf200700 1 -- 172.21.15.12:0/3748529286 shutdown_connections 2024-03-29T10:12:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.547+0000 7f4eaf200700 1 -- 172.21.15.12:0/3748529286 wait complete. 2024-03-29T10:12:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.548+0000 7f4eaf200700 1 Processor -- start 2024-03-29T10:12:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.549+0000 7f4eaf200700 1 -- start start 2024-03-29T10:12:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4eaf200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ea80cf230 0x7f4ea80b8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4eaf200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8160f10 0x7f4ea81e77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4eaf200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ea8161590 0x7f4ea81f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4eaf200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea81e8280 con 0x7f4ea8160f10 2024-03-29T10:12:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4eaf200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea81e8560 con 0x7f4ea8161590 2024-03-29T10:12:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4eaf200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea81e8840 con 0x7f4ea80cf230 2024-03-29T10:12:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4ea7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8160f10 0x7f4ea81e77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4ea7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8160f10 0x7f4ea81e77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41278/0 (socket says 172.21.15.12:41278) 2024-03-29T10:12:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4ea7400700 1 -- 172.21.15.12:0/2296352840 learned_addr learned my addr 172.21.15.12:0/2296352840 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4ead200700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ea8161590 0x7f4ea81f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4ea7e00700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ea80cf230 0x7f4ea80b8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4ea7400700 1 -- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ea80cf230 msgr2=0x7f4ea80b8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4ea7400700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ea80cf230 0x7f4ea80b8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4ea7400700 1 -- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ea8161590 msgr2=0x7f4ea81f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4ea7400700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ea8161590 0x7f4ea81f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.550+0000 7f4ea7400700 1 -- 172.21.15.12:0/2296352840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea0009b30 con 0x7f4ea8160f10 2024-03-29T10:12:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.551+0000 7f4ea7400700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8160f10 0x7f4ea81e77c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400e810 tx=0x7f4e9400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.551+0000 7f4e9fe00700 1 -- 172.21.15.12:0/2296352840 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400c020 con 0x7f4ea8160f10 2024-03-29T10:12:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.551+0000 7f4e9fe00700 1 -- 172.21.15.12:0/2296352840 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e94004250 con 0x7f4ea8160f10 2024-03-29T10:12:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.551+0000 7f4eaf200700 1 -- 172.21.15.12:0/2296352840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea815e2c0 con 0x7f4ea8160f10 2024-03-29T10:12:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.551+0000 7f4e9fe00700 1 -- 172.21.15.12:0/2296352840 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e940036a0 con 0x7f4ea8160f10 2024-03-29T10:12:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.551+0000 7f4eaf200700 1 -- 172.21.15.12:0/2296352840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea815dee0 con 0x7f4ea8160f10 2024-03-29T10:12:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.552+0000 7f4eaf200700 1 -- 172.21.15.12:0/2296352840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea8067720 con 0x7f4ea8160f10 2024-03-29T10:12:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.553+0000 7f4e9fe00700 1 -- 172.21.15.12:0/2296352840 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4e94003800 con 0x7f4ea8160f10 2024-03-29T10:12:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.554+0000 7f4e9fe00700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e8806adb0 0x7f4e8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.554+0000 7f4e9fe00700 1 -- 172.21.15.12:0/2296352840 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4e94088cc0 con 0x7f4ea8160f10 2024-03-29T10:12:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.554+0000 7f4ea7e00700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e8806adb0 0x7f4e8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.555+0000 7f4ea7e00700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e8806adb0 0x7f4e8806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4e9800b930 tx=0x7f4e98008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.559+0000 7f4e9fe00700 1 -- 172.21.15.12:0/2296352840 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e9408f080 con 0x7f4ea8160f10 2024-03-29T10:12:43.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.772+0000 7f4eaf200700 1 -- 172.21.15.12:0/2296352840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ea815d140 con 0x7f4ea8160f10 2024-03-29T10:12:43.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.773+0000 7f4e9fe00700 1 -- 172.21.15.12:0/2296352840 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+12399 (secure 0 0 0) 0x7f4e94053fc0 con 0x7f4ea8160f10 2024-03-29T10:12:43.773 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:43.773 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":120,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:12:42.771708+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:12:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.775+0000 7f4eaf200700 1 -- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e8806adb0 msgr2=0x7f4e8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.775+0000 7f4eaf200700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e8806adb0 0x7f4e8806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4e9800b930 tx=0x7f4e98008040 comp rx=0 tx=0).stop 2024-03-29T10:12:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.776+0000 7f4eaf200700 1 -- 172.21.15.12:0/2296352840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8160f10 msgr2=0x7f4ea81e77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.776+0000 7f4eaf200700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8160f10 0x7f4ea81e77c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400e810 tx=0x7f4e9400ebd0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.776+0000 7f4eaf200700 1 -- 172.21.15.12:0/2296352840 shutdown_connections 2024-03-29T10:12:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.776+0000 7f4eaf200700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4e8806adb0 0x7f4e8806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.776+0000 7f4eaf200700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ea8161590 0x7f4ea81f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.776+0000 7f4eaf200700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ea80cf230 0x7f4ea80b8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.776+0000 7f4eaf200700 1 --2- 172.21.15.12:0/2296352840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ea8160f10 0x7f4ea81e77c0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.776+0000 7f4eaf200700 1 -- 172.21.15.12:0/2296352840 >> 172.21.15.12:0/2296352840 conn(0x7f4ea808aff0 msgr2=0x7f4ea80caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:43.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.776+0000 7f4eaf200700 1 -- 172.21.15.12:0/2296352840 shutdown_connections 2024-03-29T10:12:43.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.776+0000 7f4eaf200700 1 -- 172.21.15.12:0/2296352840 wait complete. 2024-03-29T10:12:43.794 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:12:43.795 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:12:43.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.997+0000 7f7971000700 1 Processor -- start 2024-03-29T10:12:43.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.997+0000 7f7971000700 1 -- start start 2024-03-29T10:12:43.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.998+0000 7f7971000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f796c0d16a0 0x7f796c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.998+0000 7f7971000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f796c0d2370 0x7f796c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:43.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.998+0000 7f7971000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796c161420 con 0x7f796c15d9d0 2024-03-29T10:12:43.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.998+0000 7f7971000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796c0d2820 con 0x7f796c0d16a0 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.999+0000 7f7971000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0d4340 con 0x7f796c0d2370 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.999+0000 7f7963e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f796c0d16a0 0x7f796c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.999+0000 7f7969e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f796c15d9d0 0x7f796c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44330/0 (socket says 172.21.15.12:44330) 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.999+0000 7f7969e00700 1 -- 172.21.15.12:0/2100354515 learned_addr learned my addr 172.21.15.12:0/2100354515 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.999+0000 7f796a800700 1 --2- 172.21.15.12:0/2100354515 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f796c0d2370 0x7f796c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.999+0000 7f7963e00700 1 -- 172.21.15.12:0/2100354515 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f796c0d2370 msgr2=0x7f796c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.999+0000 7f7963e00700 1 --2- 172.21.15.12:0/2100354515 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f796c0d2370 0x7f796c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.999+0000 7f7963e00700 1 -- 172.21.15.12:0/2100354515 >> v1:172.21.15.12:6789/0 conn(0x7f796c15d9d0 legacy=0x7f796c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.999+0000 7f7963e00700 1 -- 172.21.15.12:0/2100354515 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c161700 con 0x7f796c0d16a0 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:43.999+0000 7f7963e00700 1 --2- 172.21.15.12:0/2100354515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 0x7f796c0d1e30 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7954009830 tx=0x7f7954009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf423c642e4bf75 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7969400700 1 -- 172.21.15.12:0/2100354515 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954011830 con 0x7f796c0d16a0 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7969400700 1 -- 172.21.15.12:0/2100354515 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7954011990 con 0x7f796c0d16a0 2024-03-29T10:12:44.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7969400700 1 -- 172.21.15.12:0/2100354515 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954011d40 con 0x7f796c0d16a0 2024-03-29T10:12:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7971000700 1 -- 172.21.15.12:0/2100354515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 msgr2=0x7f796c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7971000700 1 --2- 172.21.15.12:0/2100354515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 0x7f796c0d1e30 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7954009830 tx=0x7f7954009b40 comp rx=0 tx=0).stop 2024-03-29T10:12:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7971000700 1 -- 172.21.15.12:0/2100354515 shutdown_connections 2024-03-29T10:12:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7971000700 1 --2- 172.21.15.12:0/2100354515 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f796c0d2370 0x7f796c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7971000700 1 --2- 172.21.15.12:0/2100354515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 0x7f796c0d1e30 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7971000700 1 -- 172.21.15.12:0/2100354515 >> 172.21.15.12:0/2100354515 conn(0x7f796c08aff0 msgr2=0x7f796c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:44.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7971000700 1 -- 172.21.15.12:0/2100354515 shutdown_connections 2024-03-29T10:12:44.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.000+0000 7f7971000700 1 -- 172.21.15.12:0/2100354515 wait complete. 2024-03-29T10:12:44.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.002+0000 7f7971000700 1 Processor -- start 2024-03-29T10:12:44.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.002+0000 7f7971000700 1 -- start start 2024-03-29T10:12:44.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.003+0000 7f7971000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 0x7f796c0b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.003+0000 7f7971000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f796c0d2370 0x7f796c14c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.003+0000 7f7971000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f796c15d9d0 0x7f796c0b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.003+0000 7f7971000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c1e9eb0 con 0x7f796c0d16a0 2024-03-29T10:12:44.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.003+0000 7f7971000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0b60c0 con 0x7f796c15d9d0 2024-03-29T10:12:44.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.003+0000 7f7971000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0b63a0 con 0x7f796c0d2370 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.003+0000 7f7969e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 0x7f796c0b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.003+0000 7f7969e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 0x7f796c0b6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41292/0 (socket says 172.21.15.12:41292) 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.003+0000 7f7969e00700 1 -- 172.21.15.12:0/1591565618 learned_addr learned my addr 172.21.15.12:0/1591565618 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.003+0000 7f796a800700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f796c15d9d0 0x7f796c0b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7963e00700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f796c0d2370 0x7f796c14c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7969e00700 1 -- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f796c0d2370 msgr2=0x7f796c14c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7969e00700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f796c0d2370 0x7f796c14c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7969e00700 1 -- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f796c15d9d0 msgr2=0x7f796c0b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7969e00700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f796c15d9d0 0x7f796c0b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7969e00700 1 -- 172.21.15.12:0/1591565618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7954009440 con 0x7f796c0d16a0 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f796a800700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f796c15d9d0 0x7f796c0b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7969e00700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 0x7f796c0b6e80 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f795c00c480 tx=0x7f795c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7962000700 1 -- 172.21.15.12:0/1591565618 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c005d70 con 0x7f796c0d16a0 2024-03-29T10:12:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7962000700 1 -- 172.21.15.12:0/1591565618 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795c0070d0 con 0x7f796c0d16a0 2024-03-29T10:12:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7962000700 1 -- 172.21.15.12:0/1591565618 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c007440 con 0x7f796c0d16a0 2024-03-29T10:12:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7971000700 1 -- 172.21.15.12:0/1591565618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c0b6680 con 0x7f796c0d16a0 2024-03-29T10:12:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.004+0000 7f7971000700 1 -- 172.21.15.12:0/1591565618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c14ce00 con 0x7f796c0d16a0 2024-03-29T10:12:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.005+0000 7f7971000700 1 -- 172.21.15.12:0/1591565618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796c0d1ca0 con 0x7f796c0d16a0 2024-03-29T10:12:44.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.006+0000 7f7962000700 1 -- 172.21.15.12:0/1591565618 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f795c0075a0 con 0x7f796c0d16a0 2024-03-29T10:12:44.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.007+0000 7f7962000700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f794806adb0 0x7f794806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.007+0000 7f7962000700 1 -- 172.21.15.12:0/1591565618 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f795c088620 con 0x7f796c0d16a0 2024-03-29T10:12:44.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.008+0000 7f7963e00700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f794806adb0 0x7f794806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.008+0000 7f7963e00700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f794806adb0 0x7f794806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f796c0d1ae0 tx=0x7f795401c040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.012+0000 7f7962000700 1 -- 172.21.15.12:0/1591565618 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f795c053a50 con 0x7f796c0d16a0 2024-03-29T10:12:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.223+0000 7f7971000700 1 -- 172.21.15.12:0/1591565618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f796c0d2190 con 0x7f796c0d16a0 2024-03-29T10:12:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.223+0000 7f7962000700 1 -- 172.21.15.12:0/1591565618 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f795c057070 con 0x7f796c0d16a0 2024-03-29T10:12:44.223 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 -- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f794806adb0 msgr2=0x7f794806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f794806adb0 0x7f794806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f796c0d1ae0 tx=0x7f795401c040 comp rx=0 tx=0).stop 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 -- 172.21.15.12:0/1591565618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 msgr2=0x7f796c0b6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 0x7f796c0b6e80 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f795c00c480 tx=0x7f795c00c840 comp rx=0 tx=0).stop 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 -- 172.21.15.12:0/1591565618 shutdown_connections 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f794806adb0 0x7f794806d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f796c15d9d0 0x7f796c0b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f796c0d16a0 0x7f796c0b6e80 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 --2- 172.21.15.12:0/1591565618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f796c0d2370 0x7f796c14c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 -- 172.21.15.12:0/1591565618 >> 172.21.15.12:0/1591565618 conn(0x7f796c08aff0 msgr2=0x7f796c06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 -- 172.21.15.12:0/1591565618 shutdown_connections 2024-03-29T10:12:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.226+0000 7f7971000700 1 -- 172.21.15.12:0/1591565618 wait complete. 2024-03-29T10:12:44.244 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T10:12:44.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.443+0000 7f59dde00700 1 Processor -- start 2024-03-29T10:12:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.444+0000 7f59dde00700 1 -- start start 2024-03-29T10:12:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.445+0000 7f59dde00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80d0a40 0x7f59d80b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.445+0000 7f59dde00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80d0ef0 con 0x7f59d815da50 2024-03-29T10:12:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.445+0000 7f59dde00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80d24e0 con 0x7f59d80d05e0 2024-03-29T10:12:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.445+0000 7f59dde00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80b95d0 con 0x7f59d80d0a40 2024-03-29T10:12:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.445+0000 7f59d6a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f59d815da50 0x7f59d80ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44338/0 (socket says 172.21.15.12:44338) 2024-03-29T10:12:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.445+0000 7f59d6a00700 1 -- 172.21.15.12:0/677477843 learned_addr learned my addr 172.21.15.12:0/677477843 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:44.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.445+0000 7f59d7400700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80d0a40 0x7f59d80b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.445+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998844363 0 0) 0x7f59d80d0ef0 con 0x7f59d815da50 2024-03-29T10:12:44.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.445+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59bc0035b0 con 0x7f59d815da50 2024-03-29T10:12:44.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2635436248 0 0) 0x7f59bc0035b0 con 0x7f59d815da50 2024-03-29T10:12:44.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80d0a40 msgr2=0x7f59d80b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:44.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80d0a40 0x7f59d80b70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 >> v1:172.21.15.159:6789/0 conn(0x7f59d80d05e0 legacy=0x7f59d80b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80d27c0 con 0x7f59d815da50 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f59c8001670 con 0x7f59d815da50 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 >> v1:172.21.15.12:6789/0 conn(0x7f59d815da50 legacy=0x7f59d80ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59bc003a30 0x7f59bc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80d0a40 0x7f59bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59bc004d90 0x7f59bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc00bac0 con 0x7f59bc003a30 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc00bc30 con 0x7f59bc004d90 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc00bdf0 con 0x7f59d80d0a40 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d6a00700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59bc003a30 0x7f59bc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d7400700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59bc004d90 0x7f59bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.446+0000 7f59d6000700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80d0a40 0x7f59bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.447+0000 7f59d6a00700 1 -- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80d0a40 msgr2=0x7f59bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.447+0000 7f59d6a00700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80d0a40 0x7f59bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.447+0000 7f59d6a00700 1 -- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59bc004d90 msgr2=0x7f59bc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.447+0000 7f59d6a00700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59bc004d90 0x7f59bc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.447+0000 7f59d6a00700 1 -- 172.21.15.12:0/677477843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59bc0035b0 con 0x7f59bc003a30 2024-03-29T10:12:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.447+0000 7f59d6000700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80d0a40 0x7f59bc004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.447+0000 7f59d6a00700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59bc003a30 0x7f59bc003e50 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f59c800c210 tx=0x7f59bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625862c6a819ce00 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.447+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8007c10 con 0x7f59bc003a30 2024-03-29T10:12:44.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.447+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c800cdd0 con 0x7f59bc003a30 2024-03-29T10:12:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.447+0000 7f59d5600700 1 -- 172.21.15.12:0/677477843 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8005ba0 con 0x7f59bc003a30 2024-03-29T10:12:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.448+0000 7f59dde00700 1 -- 172.21.15.12:0/677477843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59bc003a30 msgr2=0x7f59bc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.448+0000 7f59dde00700 1 --2- 172.21.15.12:0/677477843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59bc003a30 0x7f59bc003e50 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f59c800c210 tx=0x7f59bc005880 comp rx=0 tx=0).stop 2024-03-29T10:12:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.448+0000 7f59d7400700 1 -- 172.21.15.12:0/677477843 reap_dead start 2024-03-29T10:12:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.448+0000 7f59dde00700 1 -- 172.21.15.12:0/677477843 shutdown_connections 2024-03-29T10:12:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.448+0000 7f59dde00700 1 -- 172.21.15.12:0/677477843 >> 172.21.15.12:0/677477843 conn(0x7f59d80666a0 msgr2=0x7f59d8066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.448+0000 7f59dde00700 1 -- 172.21.15.12:0/677477843 shutdown_connections 2024-03-29T10:12:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.448+0000 7f59dde00700 1 -- 172.21.15.12:0/677477843 wait complete. 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.450+0000 7f59dde00700 1 Processor -- start 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.450+0000 7f59dde00700 1 -- start start 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.451+0000 7f59dde00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59bc003a30 0x7f59d8155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59dde00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d81f3a50 0x7f59d8155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59dde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59d80cbb50 0x7f59d80cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59dde00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80cc8e0 con 0x7f59d80cbb50 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59dde00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80ccbc0 con 0x7f59d81f3a50 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59dde00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80ccea0 con 0x7f59bc003a30 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59d80cbb50 0x7f59d80cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59d80cbb50 0x7f59d80cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41318/0 (socket says 172.21.15.12:41318) 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d7400700 1 -- 172.21.15.12:0/133110760 learned_addr learned my addr 172.21.15.12:0/133110760 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d6000700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d81f3a50 0x7f59d8155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d6a00700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59bc003a30 0x7f59d8155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d7400700 1 -- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59bc003a30 msgr2=0x7f59d8155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d7400700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59bc003a30 0x7f59d8155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d7400700 1 -- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d81f3a50 msgr2=0x7f59d8155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d7400700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d81f3a50 0x7f59d8155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d7400700 1 -- 172.21.15.12:0/133110760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80cd180 con 0x7f59d80cbb50 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.452+0000 7f59d6a00700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59bc003a30 0x7f59d8155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.453+0000 7f59d7400700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59d80cbb50 0x7f59d80cbfe0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f59d80d4910 tx=0x7f59d0011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.453+0000 7f59cf400700 1 -- 172.21.15.12:0/133110760 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d000e040 con 0x7f59d80cbb50 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.453+0000 7f59cf400700 1 -- 172.21.15.12:0/133110760 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d000f040 con 0x7f59d80cbb50 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.453+0000 7f59dde00700 1 -- 172.21.15.12:0/133110760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d8142420 con 0x7f59d80cbb50 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.453+0000 7f59cf400700 1 -- 172.21.15.12:0/133110760 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d000fc40 con 0x7f59d80cbb50 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.453+0000 7f59dde00700 1 -- 172.21.15.12:0/133110760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d8144460 con 0x7f59d80cbb50 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.454+0000 7f59dde00700 1 -- 172.21.15.12:0/133110760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d80b9870 con 0x7f59d80cbb50 2024-03-29T10:12:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.455+0000 7f59cf400700 1 -- 172.21.15.12:0/133110760 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f59d0015430 con 0x7f59d80cbb50 2024-03-29T10:12:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.456+0000 7f59cf400700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f59b806ade0 0x7f59b806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.456+0000 7f59cf400700 1 -- 172.21.15.12:0/133110760 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f59d0089590 con 0x7f59d80cbb50 2024-03-29T10:12:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.456+0000 7f59d6a00700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f59b806ade0 0x7f59b806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.457+0000 7f59d6a00700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f59b806ade0 0x7f59b806d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f59c8000c00 tx=0x7f59c8007000 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:44.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.461+0000 7f59cf400700 1 -- 172.21.15.12:0/133110760 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59d00548a0 con 0x7f59d80cbb50 2024-03-29T10:12:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.676+0000 7f59dde00700 1 -- 172.21.15.12:0/133110760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f59d8142610 con 0x7f59d80cbb50 2024-03-29T10:12:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.677+0000 7f59cf400700 1 -- 172.21.15.12:0/133110760 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v120) v1 ==== 84+0+65 (secure 0 0 0) 0x7f59d0057ec0 con 0x7f59d80cbb50 2024-03-29T10:12:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:44.677 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:12:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T10:12:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T10:12:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:44.677 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:12:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.679+0000 7f59dde00700 1 -- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f59b806ade0 msgr2=0x7f59b806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.679+0000 7f59dde00700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f59b806ade0 0x7f59b806d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f59c8000c00 tx=0x7f59c8007000 comp rx=0 tx=0).stop 2024-03-29T10:12:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.679+0000 7f59dde00700 1 -- 172.21.15.12:0/133110760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59d80cbb50 msgr2=0x7f59d80cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.679+0000 7f59dde00700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59d80cbb50 0x7f59d80cbfe0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f59d80d4910 tx=0x7f59d0011e20 comp rx=0 tx=0).stop 2024-03-29T10:12:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.679+0000 7f59dde00700 1 -- 172.21.15.12:0/133110760 shutdown_connections 2024-03-29T10:12:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.679+0000 7f59dde00700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f59b806ade0 0x7f59b806d2a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.679+0000 7f59dde00700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59d80cbb50 0x7f59d80cbfe0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.679+0000 7f59dde00700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59bc003a30 0x7f59d8155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.679+0000 7f59dde00700 1 --2- 172.21.15.12:0/133110760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d81f3a50 0x7f59d8155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.679+0000 7f59dde00700 1 -- 172.21.15.12:0/133110760 >> 172.21.15.12:0/133110760 conn(0x7f59d80666a0 msgr2=0x7f59d8154d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.680+0000 7f59dde00700 1 -- 172.21.15.12:0/133110760 shutdown_connections 2024-03-29T10:12:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.680+0000 7f59dde00700 1 -- 172.21.15.12:0/133110760 wait complete. 2024-03-29T10:12:44.697 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T10:12:44.698 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T10:12:44.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.900+0000 7fa548c00700 1 Processor -- start 2024-03-29T10:12:44.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.900+0000 7fa548c00700 1 -- start start 2024-03-29T10:12:44.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.901+0000 7fa548c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa54415ddf0 0x7fa5440ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.901+0000 7fa548c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa54415e1d0 0x7fa5440d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.901+0000 7fa548c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5440ce8a0 con 0x7fa5440d2710 2024-03-29T10:12:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.901+0000 7fa548c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5440d2b80 con 0x7fa54415ddf0 2024-03-29T10:12:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.901+0000 7fa548c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54415ec60 con 0x7fa54415e1d0 2024-03-29T10:12:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa542800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa5440d2710 0x7fa5441612f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44348/0 (socket says 172.21.15.12:44348) 2024-03-29T10:12:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa542800700 1 -- 172.21.15.12:0/4032754081 learned_addr learned my addr 172.21.15.12:0/4032754081 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa541e00700 1 --2- 172.21.15.12:0/4032754081 >> v2:172.21.15.159:3300/0 conn(0x7fa54415ddf0 0x7fa5440ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa541400700 1 --2- 172.21.15.12:0/4032754081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa54415e1d0 0x7fa5440d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa540a00700 1 -- 172.21.15.12:0/4032754081 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353559700 0 0) 0x7fa5440ce8a0 con 0x7fa5440d2710 2024-03-29T10:12:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa540a00700 1 -- 172.21.15.12:0/4032754081 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5280035b0 con 0x7fa5440d2710 2024-03-29T10:12:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa541400700 1 -- 172.21.15.12:0/4032754081 >> v2:172.21.15.159:3300/0 conn(0x7fa54415ddf0 msgr2=0x7fa5440ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa541400700 1 --2- 172.21.15.12:0/4032754081 >> v2:172.21.15.159:3300/0 conn(0x7fa54415ddf0 0x7fa5440ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa541400700 1 -- 172.21.15.12:0/4032754081 >> v1:172.21.15.12:6789/0 conn(0x7fa5440d2710 legacy=0x7fa5441612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa541400700 1 -- 172.21.15.12:0/4032754081 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5440cdc90 con 0x7fa54415e1d0 2024-03-29T10:12:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.902+0000 7fa541e00700 1 --2- 172.21.15.12:0/4032754081 >> v2:172.21.15.159:3300/0 conn(0x7fa54415ddf0 0x7fa5440ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.903+0000 7fa541400700 1 --2- 172.21.15.12:0/4032754081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa54415e1d0 0x7fa5440d21d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa530009840 tx=0x7fa530009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=400e6ee6e6ed4a6e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.903+0000 7fa540a00700 1 -- 172.21.15.12:0/4032754081 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530011830 con 0x7fa54415e1d0 2024-03-29T10:12:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.903+0000 7fa540a00700 1 -- 172.21.15.12:0/4032754081 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa530011990 con 0x7fa54415e1d0 2024-03-29T10:12:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.903+0000 7fa540a00700 1 -- 172.21.15.12:0/4032754081 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530011d20 con 0x7fa54415e1d0 2024-03-29T10:12:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.904+0000 7fa548c00700 1 -- 172.21.15.12:0/4032754081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa54415e1d0 msgr2=0x7fa5440d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.904+0000 7fa548c00700 1 --2- 172.21.15.12:0/4032754081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa54415e1d0 0x7fa5440d21d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa530009840 tx=0x7fa530009b50 comp rx=0 tx=0).stop 2024-03-29T10:12:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.904+0000 7fa548c00700 1 -- 172.21.15.12:0/4032754081 shutdown_connections 2024-03-29T10:12:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.904+0000 7fa548c00700 1 --2- 172.21.15.12:0/4032754081 >> v2:172.21.15.159:3300/0 conn(0x7fa54415ddf0 0x7fa5440ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.904+0000 7fa548c00700 1 --2- 172.21.15.12:0/4032754081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa54415e1d0 0x7fa5440d21d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.904+0000 7fa548c00700 1 -- 172.21.15.12:0/4032754081 >> 172.21.15.12:0/4032754081 conn(0x7fa54408aff0 msgr2=0x7fa5440d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.904+0000 7fa548c00700 1 -- 172.21.15.12:0/4032754081 shutdown_connections 2024-03-29T10:12:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.904+0000 7fa548c00700 1 -- 172.21.15.12:0/4032754081 wait complete. 2024-03-29T10:12:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.905+0000 7fa548c00700 1 Processor -- start 2024-03-29T10:12:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.906+0000 7fa548c00700 1 -- start start 2024-03-29T10:12:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.906+0000 7fa548c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5440d2710 0x7fa5441603f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa548c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa54415ddf0 0x7fa544156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa548c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa54415e1d0 0x7fa5441e9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa548c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa544160d50 con 0x7fa54415ddf0 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa548c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa544161030 con 0x7fa54415e1d0 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa548c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa544161310 con 0x7fa5440d2710 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa54415ddf0 0x7fa544156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa54415ddf0 0x7fa544156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41324/0 (socket says 172.21.15.12:41324) 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541400700 1 -- 172.21.15.12:0/1374553312 learned_addr learned my addr 172.21.15.12:0/1374553312 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa542800700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa54415e1d0 0x7fa5441e9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541e00700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5440d2710 0x7fa5441603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541400700 1 -- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5440d2710 msgr2=0x7fa5441603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541400700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5440d2710 0x7fa5441603f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541400700 1 -- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa54415e1d0 msgr2=0x7fa5441e9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:44.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541400700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa54415e1d0 0x7fa5441e9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:44.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541400700 1 -- 172.21.15.12:0/1374553312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5340094a0 con 0x7fa54415ddf0 2024-03-29T10:12:44.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa542800700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa54415e1d0 0x7fa5441e9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:44.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541e00700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5440d2710 0x7fa5441603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:44.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.907+0000 7fa541400700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa54415ddf0 0x7fa544156ac0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fa530009fd0 tx=0x7fa530004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:44.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.908+0000 7fa53a000700 1 -- 172.21.15.12:0/1374553312 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5300036a0 con 0x7fa54415ddf0 2024-03-29T10:12:44.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.908+0000 7fa53a000700 1 -- 172.21.15.12:0/1374553312 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53001e070 con 0x7fa54415ddf0 2024-03-29T10:12:44.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.908+0000 7fa548c00700 1 -- 172.21.15.12:0/1374553312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5300094a0 con 0x7fa54415ddf0 2024-03-29T10:12:44.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.908+0000 7fa53a000700 1 -- 172.21.15.12:0/1374553312 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530003a60 con 0x7fa54415ddf0 2024-03-29T10:12:44.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.908+0000 7fa548c00700 1 -- 172.21.15.12:0/1374553312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5441618d0 con 0x7fa54415ddf0 2024-03-29T10:12:44.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.909+0000 7fa548c00700 1 -- 172.21.15.12:0/1374553312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa544067720 con 0x7fa54415ddf0 2024-03-29T10:12:44.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.910+0000 7fa53a000700 1 -- 172.21.15.12:0/1374553312 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa530003bc0 con 0x7fa54415ddf0 2024-03-29T10:12:44.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.911+0000 7fa53a000700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa52406adb0 0x7fa52406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.911+0000 7fa53a000700 1 -- 172.21.15.12:0/1374553312 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa53008a020 con 0x7fa54415ddf0 2024-03-29T10:12:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.911+0000 7fa541e00700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa52406adb0 0x7fa52406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.912+0000 7fa541e00700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa52406adb0 0x7fa52406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa534009cf0 tx=0x7fa534008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:44.915+0000 7fa53a000700 1 -- 172.21.15.12:0/1374553312 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa530055560 con 0x7fa54415ddf0 2024-03-29T10:12:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.128+0000 7fa548c00700 1 -- 172.21.15.12:0/1374553312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa5440d3050 con 0x7fa54415ddf0 2024-03-29T10:12:45.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.282+0000 7fa53a000700 1 -- 172.21.15.12:0/1374553312 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v121) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa530058b80 con 0x7fa54415ddf0 2024-03-29T10:12:45.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.287+0000 7fa548c00700 1 -- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa52406adb0 msgr2=0x7fa52406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:45.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.287+0000 7fa548c00700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa52406adb0 0x7fa52406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa534009cf0 tx=0x7fa534008040 comp rx=0 tx=0).stop 2024-03-29T10:12:45.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.287+0000 7fa548c00700 1 -- 172.21.15.12:0/1374553312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa54415ddf0 msgr2=0x7fa544156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:45.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.287+0000 7fa548c00700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa54415ddf0 0x7fa544156ac0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fa530009fd0 tx=0x7fa530004750 comp rx=0 tx=0).stop 2024-03-29T10:12:45.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.287+0000 7fa548c00700 1 -- 172.21.15.12:0/1374553312 shutdown_connections 2024-03-29T10:12:45.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.287+0000 7fa548c00700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa52406adb0 0x7fa52406d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:45.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.287+0000 7fa548c00700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa54415e1d0 0x7fa5441e9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:45.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.287+0000 7fa548c00700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5440d2710 0x7fa5441603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:45.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.287+0000 7fa548c00700 1 --2- 172.21.15.12:0/1374553312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa54415ddf0 0x7fa544156ac0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:45.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.287+0000 7fa548c00700 1 -- 172.21.15.12:0/1374553312 >> 172.21.15.12:0/1374553312 conn(0x7fa54408aff0 msgr2=0x7fa5440b8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:45.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.288+0000 7fa548c00700 1 -- 172.21.15.12:0/1374553312 shutdown_connections 2024-03-29T10:12:45.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.288+0000 7fa548c00700 1 -- 172.21.15.12:0/1374553312 wait complete. 2024-03-29T10:12:45.290 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T10:12:45.308 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T10:12:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.513+0000 7fe4f3a00700 1 Processor -- start 2024-03-29T10:12:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.513+0000 7fe4f3a00700 1 -- start start 2024-03-29T10:12:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.514+0000 7fe4f3a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4ec15d800 0x7fe4ec0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.514+0000 7fe4f3a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec0709c0 con 0x7fe4ec0ce6a0 2024-03-29T10:12:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.515+0000 7fe4f3a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec070c80 con 0x7fe4ec0ce230 2024-03-29T10:12:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.515+0000 7fe4f3a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ec0b40f0 con 0x7fe4ec15d800 2024-03-29T10:12:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.515+0000 7fe4f1000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4ec15d800 0x7fe4ec0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.515+0000 7fe4f1000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4ec15d800 0x7fe4ec0725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41326/0 (socket says 172.21.15.12:41326) 2024-03-29T10:12:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.515+0000 7fe4f1000700 1 -- 172.21.15.12:0/2759468292 learned_addr learned my addr 172.21.15.12:0/2759468292 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.515+0000 7fe4f1000700 1 -- 172.21.15.12:0/2759468292 >> v1:172.21.15.159:6789/0 conn(0x7fe4ec0ce230 legacy=0x7fe4ec0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.515+0000 7fe4f1000700 1 -- 172.21.15.12:0/2759468292 >> v1:172.21.15.12:6789/0 conn(0x7fe4ec0ce6a0 legacy=0x7fe4ec15dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.515+0000 7fe4f1000700 1 -- 172.21.15.12:0/2759468292 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec070f60 con 0x7fe4ec15d800 2024-03-29T10:12:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.516+0000 7fe4f1000700 1 --2- 172.21.15.12:0/2759468292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 0x7fe4ec0725c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00098f0 tx=0x7fe4e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eccc92ad223dabc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.516+0000 7fe4eb400700 1 -- 172.21.15.12:0/2759468292 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e000b8a0 con 0x7fe4ec15d800 2024-03-29T10:12:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.516+0000 7fe4eb400700 1 -- 172.21.15.12:0/2759468292 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e000ba00 con 0x7fe4ec15d800 2024-03-29T10:12:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.516+0000 7fe4eb400700 1 -- 172.21.15.12:0/2759468292 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e000bd90 con 0x7fe4ec15d800 2024-03-29T10:12:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.516+0000 7fe4f3a00700 1 -- 172.21.15.12:0/2759468292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 msgr2=0x7fe4ec0725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.516+0000 7fe4f3a00700 1 --2- 172.21.15.12:0/2759468292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 0x7fe4ec0725c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00098f0 tx=0x7fe4e0009c00 comp rx=0 tx=0).stop 2024-03-29T10:12:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.516+0000 7fe4f3a00700 1 -- 172.21.15.12:0/2759468292 shutdown_connections 2024-03-29T10:12:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.516+0000 7fe4f3a00700 1 --2- 172.21.15.12:0/2759468292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 0x7fe4ec0725c0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.516+0000 7fe4f3a00700 1 -- 172.21.15.12:0/2759468292 >> 172.21.15.12:0/2759468292 conn(0x7fe4ec0b0d70 msgr2=0x7fe4ec0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.517+0000 7fe4f3a00700 1 -- 172.21.15.12:0/2759468292 shutdown_connections 2024-03-29T10:12:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.517+0000 7fe4f3a00700 1 -- 172.21.15.12:0/2759468292 wait complete. 2024-03-29T10:12:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.518+0000 7fe4f3a00700 1 Processor -- start 2024-03-29T10:12:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.518+0000 7fe4f3a00700 1 -- start start 2024-03-29T10:12:45.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.519+0000 7fe4f3a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4ec0ce230 0x7fe4ec0999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:45.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.519+0000 7fe4f3a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4ec0ce6a0 0x7fe4ec09a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:45.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.519+0000 7fe4f3a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 0x7fe4ec09a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:45.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.519+0000 7fe4f3a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec099ee0 con 0x7fe4ec15d800 2024-03-29T10:12:45.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.519+0000 7fe4f3a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec1f3e00 con 0x7fe4ec0ce6a0 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.519+0000 7fe4f3a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec148bc0 con 0x7fe4ec0ce230 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 0x7fe4ec09a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 0x7fe4ec09a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41340/0 (socket says 172.21.15.12:41340) 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1a00700 1 -- 172.21.15.12:0/3332850570 learned_addr learned my addr 172.21.15.12:0/3332850570 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4ebe00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4ec0ce6a0 0x7fe4ec09a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1000700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4ec0ce230 0x7fe4ec0999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1a00700 1 -- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4ec0ce230 msgr2=0x7fe4ec0999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1a00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4ec0ce230 0x7fe4ec0999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1a00700 1 -- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4ec0ce6a0 msgr2=0x7fe4ec09a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1a00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4ec0ce6a0 0x7fe4ec09a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1a00700 1 -- 172.21.15.12:0/3332850570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0009550 con 0x7fe4ec15d800 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1000700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4ec0ce230 0x7fe4ec0999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.520+0000 7fe4f1a00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 0x7fe4ec09a6a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fe4e400cc50 tx=0x7fe4e400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.521+0000 7fe4e8c00700 1 -- 172.21.15.12:0/3332850570 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e400ad80 con 0x7fe4ec15d800 2024-03-29T10:12:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.521+0000 7fe4e8c00700 1 -- 172.21.15.12:0/3332850570 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e400ce50 con 0x7fe4ec15d800 2024-03-29T10:12:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.521+0000 7fe4e8c00700 1 -- 172.21.15.12:0/3332850570 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4003c40 con 0x7fe4ec15d800 2024-03-29T10:12:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.521+0000 7fe4f3a00700 1 -- 172.21.15.12:0/3332850570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec1f34d0 con 0x7fe4ec15d800 2024-03-29T10:12:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.521+0000 7fe4f3a00700 1 -- 172.21.15.12:0/3332850570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec1f3a90 con 0x7fe4ec15d800 2024-03-29T10:12:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.522+0000 7fe4f3a00700 1 -- 172.21.15.12:0/3332850570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ec1f3f90 con 0x7fe4ec15d800 2024-03-29T10:12:45.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.523+0000 7fe4e8c00700 1 -- 172.21.15.12:0/3332850570 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe4e40050d0 con 0x7fe4ec15d800 2024-03-29T10:12:45.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.524+0000 7fe4e8c00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe4cc06adb0 0x7fe4cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:45.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.524+0000 7fe4e8c00700 1 -- 172.21.15.12:0/3332850570 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe4e4089f60 con 0x7fe4ec15d800 2024-03-29T10:12:45.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.525+0000 7fe4f1000700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe4cc06adb0 0x7fe4cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:45.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.525+0000 7fe4f1000700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe4cc06adb0 0x7fe4cc06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00098f0 tx=0x7fe4e0008030 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:45.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.528+0000 7fe4e8c00700 1 -- 172.21.15.12:0/3332850570 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe4e4055160 con 0x7fe4ec15d800 2024-03-29T10:12:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:45.744+0000 7fe4f3a00700 1 -- 172.21.15.12:0/3332850570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe4ec067720 con 0x7fe4ec15d800 2024-03-29T10:12:46.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.290+0000 7fe4e8c00700 1 -- 172.21.15.12:0/3332850570 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v122) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe4e4058780 con 0x7fe4ec15d800 2024-03-29T10:12:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.296+0000 7fe4f3a00700 1 -- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe4cc06adb0 msgr2=0x7fe4cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.296+0000 7fe4f3a00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe4cc06adb0 0x7fe4cc06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00098f0 tx=0x7fe4e0008030 comp rx=0 tx=0).stop 2024-03-29T10:12:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.296+0000 7fe4f3a00700 1 -- 172.21.15.12:0/3332850570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 msgr2=0x7fe4ec09a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.296+0000 7fe4f3a00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 0x7fe4ec09a6a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fe4e400cc50 tx=0x7fe4e400a340 comp rx=0 tx=0).stop 2024-03-29T10:12:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.296+0000 7fe4f3a00700 1 -- 172.21.15.12:0/3332850570 shutdown_connections 2024-03-29T10:12:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.296+0000 7fe4f3a00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe4cc06adb0 0x7fe4cc06d270 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00098f0 tx=0x7fe4e0008030 comp rx=0 tx=0).stop 2024-03-29T10:12:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.296+0000 7fe4f3a00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4ec15d800 0x7fe4ec09a6a0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.296+0000 7fe4f3a00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4ec0ce230 0x7fe4ec0999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.296+0000 7fe4f3a00700 1 --2- 172.21.15.12:0/3332850570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4ec0ce6a0 0x7fe4ec09a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.296+0000 7fe4f3a00700 1 -- 172.21.15.12:0/3332850570 >> 172.21.15.12:0/3332850570 conn(0x7fe4ec0b0d70 msgr2=0x7fe4ec0cba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:46.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.297+0000 7fe4f3a00700 1 -- 172.21.15.12:0/3332850570 shutdown_connections 2024-03-29T10:12:46.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.297+0000 7fe4f3a00700 1 -- 172.21.15.12:0/3332850570 wait complete. 2024-03-29T10:12:46.301 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T10:12:46.332 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T10:12:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.535+0000 7f5bd8e00700 1 Processor -- start 2024-03-29T10:12:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.535+0000 7f5bd8e00700 1 -- start start 2024-03-29T10:12:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.536+0000 7f5bd8e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bd40b3f80 0x7f5bd40ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.536+0000 7f5bd8e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd4161230 con 0x7f5bd40d3f20 2024-03-29T10:12:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.536+0000 7f5bd8e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40d2a80 con 0x7f5bd4160e50 2024-03-29T10:12:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.536+0000 7f5bd8e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40d43b0 con 0x7f5bd40b3f80 2024-03-29T10:12:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.536+0000 7f5bd1400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5bd40d3f20 0x7f5bd40cdef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44364/0 (socket says 172.21.15.12:44364) 2024-03-29T10:12:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.536+0000 7f5bd1400700 1 -- 172.21.15.12:0/1624555476 learned_addr learned my addr 172.21.15.12:0/1624555476 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd2800700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bd40b3f80 0x7f5bd40ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036816049 0 0) 0x7f5bd4161230 con 0x7f5bd40d3f20 2024-03-29T10:12:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb80035b0 con 0x7f5bd40d3f20 2024-03-29T10:12:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1442900650 0 0) 0x7f5bb80035b0 con 0x7f5bd40d3f20 2024-03-29T10:12:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bd40b3f80 msgr2=0x7f5bd40ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bd40b3f80 0x7f5bd40ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 >> v1:172.21.15.159:6789/0 conn(0x7f5bd4160e50 legacy=0x7f5bd40b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd40d21e0 con 0x7f5bd40d3f20 2024-03-29T10:12:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f5bc0001670 con 0x7f5bd40d3f20 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 >> v1:172.21.15.12:6789/0 conn(0x7f5bd40d3f20 legacy=0x7f5bd40cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bb8003a30 0x7f5bb8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bb80043e0 0x7f5bb80048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bd40b3f80 0x7f5bb80051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40d43b0 con 0x7f5bb80043e0 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb800bac0 con 0x7f5bb8003a30 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb800bc80 con 0x7f5bd40b3f80 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.537+0000 7f5bd1400700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bb80043e0 0x7f5bb80048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.538+0000 7f5bd1e00700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bb8003a30 0x7f5bb8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.538+0000 7f5bd2800700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bd40b3f80 0x7f5bb80051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.538+0000 7f5bd1400700 1 -- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bd40b3f80 msgr2=0x7f5bb80051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.538+0000 7f5bd1400700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bd40b3f80 0x7f5bb80051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.538+0000 7f5bd1400700 1 -- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bb8003a30 msgr2=0x7f5bb8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.538+0000 7f5bd1400700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bb8003a30 0x7f5bb8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.538+0000 7f5bd1400700 1 -- 172.21.15.12:0/1624555476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb80035b0 con 0x7f5bb80043e0 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.538+0000 7f5bd1400700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bb80043e0 0x7f5bb80048b0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000c2f0 tx=0x7f5bb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ca475550fc75dd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.538+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc000ce30 con 0x7f5bb80043e0 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.538+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc0005960 con 0x7f5bb80043e0 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.539+0000 7f5bd0a00700 1 -- 172.21.15.12:0/1624555476 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0005d10 con 0x7f5bb80043e0 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.539+0000 7f5bd8e00700 1 -- 172.21.15.12:0/1624555476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bb80043e0 msgr2=0x7f5bb80048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.539+0000 7f5bd8e00700 1 --2- 172.21.15.12:0/1624555476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bb80043e0 0x7f5bb80048b0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000c2f0 tx=0x7f5bb8005880 comp rx=0 tx=0).stop 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.539+0000 7f5bd2800700 1 -- 172.21.15.12:0/1624555476 reap_dead start 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.539+0000 7f5bd8e00700 1 -- 172.21.15.12:0/1624555476 shutdown_connections 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.539+0000 7f5bd8e00700 1 -- 172.21.15.12:0/1624555476 >> 172.21.15.12:0/1624555476 conn(0x7f5bd408aff0 msgr2=0x7f5bd40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.539+0000 7f5bd8e00700 1 -- 172.21.15.12:0/1624555476 shutdown_connections 2024-03-29T10:12:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.539+0000 7f5bd8e00700 1 -- 172.21.15.12:0/1624555476 wait complete. 2024-03-29T10:12:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.541+0000 7f5bd8e00700 1 Processor -- start 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.541+0000 7f5bd8e00700 1 -- start start 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd8e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bb80043e0 0x7f5bd40b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd415bac0 0x7f5bd40c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd8e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bd40c96e0 0x7f5bd40cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd8e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40c9b60 con 0x7f5bd415bac0 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd8e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40b6b60 con 0x7f5bd40c96e0 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd8e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40b6e00 con 0x7f5bb80043e0 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd415bac0 0x7f5bd40c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd415bac0 0x7f5bd40c91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41354/0 (socket says 172.21.15.12:41354) 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd1400700 1 -- 172.21.15.12:0/3248262839 learned_addr learned my addr 172.21.15.12:0/3248262839 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd1e00700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bb80043e0 0x7f5bd40b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.542+0000 7f5bd2800700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bd40c96e0 0x7f5bd40cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bd1400700 1 -- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bb80043e0 msgr2=0x7f5bd40b7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bd1400700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bb80043e0 0x7f5bd40b7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bd1400700 1 -- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bd40c96e0 msgr2=0x7f5bd40cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bd1400700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bd40c96e0 0x7f5bd40cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bd1400700 1 -- 172.21.15.12:0/3248262839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd40b70e0 con 0x7f5bd415bac0 2024-03-29T10:12:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bd1e00700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bb80043e0 0x7f5bd40b7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bd1400700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd415bac0 0x7f5bd40c91a0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8005810 tx=0x7f5bc0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bca000700 1 -- 172.21.15.12:0/3248262839 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0005ce0 con 0x7f5bd415bac0 2024-03-29T10:12:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bca000700 1 -- 172.21.15.12:0/3248262839 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc0005e60 con 0x7f5bd415bac0 2024-03-29T10:12:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bd8e00700 1 -- 172.21.15.12:0/3248262839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd40b73c0 con 0x7f5bd415bac0 2024-03-29T10:12:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bca000700 1 -- 172.21.15.12:0/3248262839 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0004ab0 con 0x7f5bd415bac0 2024-03-29T10:12:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.543+0000 7f5bd8e00700 1 -- 172.21.15.12:0/3248262839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd40b7980 con 0x7f5bd415bac0 2024-03-29T10:12:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.544+0000 7f5bd8e00700 1 -- 172.21.15.12:0/3248262839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd40b7cf0 con 0x7f5bd415bac0 2024-03-29T10:12:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.545+0000 7f5bca000700 1 -- 172.21.15.12:0/3248262839 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5bc0004c10 con 0x7f5bd415bac0 2024-03-29T10:12:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.547+0000 7f5bca000700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5bb406adb0 0x7f5bb406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.547+0000 7f5bca000700 1 -- 172.21.15.12:0/3248262839 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f5bc008ab20 con 0x7f5bd415bac0 2024-03-29T10:12:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.547+0000 7f5bd1e00700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5bb406adb0 0x7f5bb406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.548+0000 7f5bd1e00700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5bb406adb0 0x7f5bb406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4007fd0 tx=0x7f5bc4009040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.552+0000 7f5bca000700 1 -- 172.21.15.12:0/3248262839 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5bc0055d00 con 0x7f5bd415bac0 2024-03-29T10:12:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.809+0000 7f5bd8e00700 1 -- 172.21.15.12:0/3248262839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5bd40c9ac0 con 0x7f5bd415bac0 2024-03-29T10:12:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.816+0000 7f5bca000700 1 -- 172.21.15.12:0/3248262839 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v51) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5bc0007da0 con 0x7f5bd415bac0 2024-03-29T10:12:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.821+0000 7f5bd8e00700 1 -- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5bb406adb0 msgr2=0x7f5bb406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.821+0000 7f5bd8e00700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5bb406adb0 0x7f5bb406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4007fd0 tx=0x7f5bc4009040 comp rx=0 tx=0).stop 2024-03-29T10:12:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.821+0000 7f5bd8e00700 1 -- 172.21.15.12:0/3248262839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd415bac0 msgr2=0x7f5bd40c91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.821+0000 7f5bd8e00700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd415bac0 0x7f5bd40c91a0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8005810 tx=0x7f5bc0001a00 comp rx=0 tx=0).stop 2024-03-29T10:12:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.821+0000 7f5bd8e00700 1 -- 172.21.15.12:0/3248262839 shutdown_connections 2024-03-29T10:12:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.821+0000 7f5bd8e00700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5bb406adb0 0x7f5bb406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.821+0000 7f5bd8e00700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bd40c96e0 0x7f5bd40cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.821+0000 7f5bd8e00700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bb80043e0 0x7f5bd40b7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.821+0000 7f5bd8e00700 1 --2- 172.21.15.12:0/3248262839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd415bac0 0x7f5bd40c91a0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.821+0000 7f5bd8e00700 1 -- 172.21.15.12:0/3248262839 >> 172.21.15.12:0/3248262839 conn(0x7f5bd408aff0 msgr2=0x7f5bd406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.824+0000 7f5bd8e00700 1 -- 172.21.15.12:0/3248262839 shutdown_connections 2024-03-29T10:12:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:46.825+0000 7f5bd8e00700 1 -- 172.21.15.12:0/3248262839 wait complete. 2024-03-29T10:12:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 14 and data pool 15 2024-03-29T10:12:46.857 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:12:47.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.056+0000 7f2ddc200700 1 Processor -- start 2024-03-29T10:12:47.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.056+0000 7f2ddc200700 1 -- start start 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.057+0000 7f2ddc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd4160f10 0x7f2dd40d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.058+0000 7f2ddc200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2dd4161590 0x7f2dd4161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.058+0000 7f2ddc200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd40ceda0 con 0x7f2dd40cf230 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.058+0000 7f2ddc200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd4161f80 con 0x7f2dd4161590 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.058+0000 7f2ddc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd41612f0 con 0x7f2dd4160f10 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.058+0000 7f2dd8e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2dd40cf230 0x7f2dd40d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44370/0 (socket says 172.21.15.12:44370) 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.058+0000 7f2dd8e00700 1 -- 172.21.15.12:0/72079171 learned_addr learned my addr 172.21.15.12:0/72079171 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.058+0000 7f2dda200700 1 --2- 172.21.15.12:0/72079171 >> v2:172.21.15.159:3300/0 conn(0x7f2dd4161590 0x7f2dd4161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.058+0000 7f2dd9800700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd4160f10 0x7f2dd40d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.058+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686234864 0 0) 0x7f2dd40ceda0 con 0x7f2dd40cf230 2024-03-29T10:12:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.058+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dbc0035b0 con 0x7f2dd40cf230 2024-03-29T10:12:47.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443669718 0 0) 0x7f2dbc0035b0 con 0x7f2dd40cf230 2024-03-29T10:12:47.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd4160f10 msgr2=0x7f2dd40d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:47.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd4160f10 0x7f2dd40d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 >> v2:172.21.15.159:3300/0 conn(0x7f2dd4161590 msgr2=0x7f2dd4161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 --2- 172.21.15.12:0/72079171 >> v2:172.21.15.159:3300/0 conn(0x7f2dd4161590 0x7f2dd4161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd40ce470 con 0x7f2dd40cf230 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f2dc0001670 con 0x7f2dd40cf230 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 >> v1:172.21.15.12:6789/0 conn(0x7f2dd40cf230 legacy=0x7f2dd40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd4160f10 0x7f2dbc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dbc003f80 0x7f2dbc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dbc004d90 0x7f2dbc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd4161f80 con 0x7f2dbc003f80 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd41612f0 con 0x7f2dbc004d90 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc00ba60 con 0x7f2dd4160f10 2024-03-29T10:12:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd8e00700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dbc003f80 0x7f2dbc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd9800700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd4160f10 0x7f2dbc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dda200700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dbc004d90 0x7f2dbc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd8e00700 1 -- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd4160f10 msgr2=0x7f2dbc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd8e00700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd4160f10 0x7f2dbc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd8e00700 1 -- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dbc004d90 msgr2=0x7f2dbc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd8e00700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dbc004d90 0x7f2dbc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.059+0000 7f2dd8e00700 1 -- 172.21.15.12:0/72079171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0035b0 con 0x7f2dbc003f80 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.060+0000 7f2dd8e00700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dbc003f80 0x7f2dbc004850 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c240 tx=0x7f2dbc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893e1103eea84f7a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.060+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0007c10 con 0x7f2dbc003f80 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.060+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc000cdc0 con 0x7f2dbc003f80 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.060+0000 7f2dd3e00700 1 -- 172.21.15.12:0/72079171 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0005ba0 con 0x7f2dbc003f80 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.060+0000 7f2ddc200700 1 -- 172.21.15.12:0/72079171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dbc003f80 msgr2=0x7f2dbc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.060+0000 7f2ddc200700 1 --2- 172.21.15.12:0/72079171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dbc003f80 0x7f2dbc004850 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c240 tx=0x7f2dbc005880 comp rx=0 tx=0).stop 2024-03-29T10:12:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.060+0000 7f2dda200700 1 -- 172.21.15.12:0/72079171 reap_dead start 2024-03-29T10:12:47.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.061+0000 7f2ddc200700 1 -- 172.21.15.12:0/72079171 shutdown_connections 2024-03-29T10:12:47.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.061+0000 7f2ddc200700 1 -- 172.21.15.12:0/72079171 >> 172.21.15.12:0/72079171 conn(0x7f2dd408aff0 msgr2=0x7f2dd40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:47.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.061+0000 7f2ddc200700 1 -- 172.21.15.12:0/72079171 shutdown_connections 2024-03-29T10:12:47.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.061+0000 7f2ddc200700 1 -- 172.21.15.12:0/72079171 wait complete. 2024-03-29T10:12:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.062+0000 7f2ddc200700 1 Processor -- start 2024-03-29T10:12:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.062+0000 7f2ddc200700 1 -- start start 2024-03-29T10:12:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.063+0000 7f2ddc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dbc003f80 0x7f2dd40b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.063+0000 7f2ddc200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd409f9c0 0x7f2dd409fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.063+0000 7f2ddc200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dd415e380 0x7f2dd415e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.063+0000 7f2ddc200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd40a0360 con 0x7f2dd415e380 2024-03-29T10:12:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.063+0000 7f2ddc200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd409d3a0 con 0x7f2dd409f9c0 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.063+0000 7f2ddc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd409d680 con 0x7f2dbc003f80 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dda200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dd415e380 0x7f2dd415e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dda200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dd415e380 0x7f2dd415e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41376/0 (socket says 172.21.15.12:41376) 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dda200700 1 -- 172.21.15.12:0/1489009259 learned_addr learned my addr 172.21.15.12:0/1489009259 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dd9800700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dbc003f80 0x7f2dd40b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dda200700 1 -- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dbc003f80 msgr2=0x7f2dd40b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dda200700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dbc003f80 0x7f2dd40b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dd8e00700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd409f9c0 0x7f2dd409fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dda200700 1 -- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd409f9c0 msgr2=0x7f2dd409fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dda200700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd409f9c0 0x7f2dd409fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dda200700 1 -- 172.21.15.12:0/1489009259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd409d960 con 0x7f2dd415e380 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dd9800700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dbc003f80 0x7f2dd40b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dd8e00700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd409f9c0 0x7f2dd409fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.064+0000 7f2dda200700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dd415e380 0x7f2dd415e810 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4161970 tx=0x7f2dcc0114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:47.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.065+0000 7f2dd1600700 1 -- 172.21.15.12:0/1489009259 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc00e070 con 0x7f2dd415e380 2024-03-29T10:12:47.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.065+0000 7f2dd1600700 1 -- 172.21.15.12:0/1489009259 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc003c30 con 0x7f2dd415e380 2024-03-29T10:12:47.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.065+0000 7f2ddc200700 1 -- 172.21.15.12:0/1489009259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd409dc40 con 0x7f2dd415e380 2024-03-29T10:12:47.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.065+0000 7f2dd1600700 1 -- 172.21.15.12:0/1489009259 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc003e20 con 0x7f2dd415e380 2024-03-29T10:12:47.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.065+0000 7f2ddc200700 1 -- 172.21.15.12:0/1489009259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd415ed50 con 0x7f2dd415e380 2024-03-29T10:12:47.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.066+0000 7f2ddc200700 1 -- 172.21.15.12:0/1489009259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd409e100 con 0x7f2dd415e380 2024-03-29T10:12:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.067+0000 7f2dd1600700 1 -- 172.21.15.12:0/1489009259 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2dcc011a10 con 0x7f2dd415e380 2024-03-29T10:12:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.068+0000 7f2dd1600700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2db406adb0 0x7f2db406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.068+0000 7f2dd1600700 1 -- 172.21.15.12:0/1489009259 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dcc089b00 con 0x7f2dd415e380 2024-03-29T10:12:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.068+0000 7f2dd9800700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2db406adb0 0x7f2db406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.069+0000 7f2dd9800700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2db406adb0 0x7f2db406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2dd415fcc0 tx=0x7f2dc8008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.072+0000 7f2dd1600700 1 -- 172.21.15.12:0/1489009259 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2dcc08be60 con 0x7f2dd415e380 2024-03-29T10:12:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.292+0000 7f2ddc200700 1 -- 172.21.15.12:0/1489009259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dd4067720 con 0x7f2dd415e380 2024-03-29T10:12:47.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.293+0000 7f2dd1600700 1 -- 172.21.15.12:0/1489009259 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+16064 (secure 0 0 0) 0x7f2dcc054a20 con 0x7f2dd415e380 2024-03-29T10:12:47.293 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:47.293 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":123,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:12:46.810481+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:12:45.171490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-29T10:12:45.787701+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:12:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 -- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2db406adb0 msgr2=0x7f2db406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2db406adb0 0x7f2db406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2dd415fcc0 tx=0x7f2dc8008040 comp rx=0 tx=0).stop 2024-03-29T10:12:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 -- 172.21.15.12:0/1489009259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dd415e380 msgr2=0x7f2dd415e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dd415e380 0x7f2dd415e810 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4161970 tx=0x7f2dcc0114b0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 -- 172.21.15.12:0/1489009259 shutdown_connections 2024-03-29T10:12:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2db406adb0 0x7f2db406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dd415e380 0x7f2dd415e810 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dbc003f80 0x7f2dd40b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 --2- 172.21.15.12:0/1489009259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd409f9c0 0x7f2dd409fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 -- 172.21.15.12:0/1489009259 >> 172.21.15.12:0/1489009259 conn(0x7f2dd408aff0 msgr2=0x7f2dd40cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 -- 172.21.15.12:0/1489009259 shutdown_connections 2024-03-29T10:12:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.296+0000 7f2ddc200700 1 -- 172.21.15.12:0/1489009259 wait complete. 2024-03-29T10:12:47.315 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:12:47.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.514+0000 7f7a9ca00700 1 Processor -- start 2024-03-29T10:12:47.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.514+0000 7f7a9ca00700 1 -- start start 2024-03-29T10:12:47.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.515+0000 7f7a9ca00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7a981619d0 0x7f7a981685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.515+0000 7f7a9ca00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a980d30d0 con 0x7f7a9815d640 2024-03-29T10:12:47.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.515+0000 7f7a9ca00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a980b6840 con 0x7f7a980ceec0 2024-03-29T10:12:47.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.515+0000 7f7a9ca00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a980d2b30 con 0x7f7a981619d0 2024-03-29T10:12:47.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.515+0000 7f7a95e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7a9815d640 0x7f7a980d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44380/0 (socket says 172.21.15.12:44380) 2024-03-29T10:12:47.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.515+0000 7f7a95e00700 1 -- 172.21.15.12:0/3375135301 learned_addr learned my addr 172.21.15.12:0/3375135301 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:47.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a96800700 1 --2- 172.21.15.12:0/3375135301 >> v2:172.21.15.159:3300/0 conn(0x7f7a981619d0 0x7f7a981685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467816887 0 0) 0x7f7a980d30d0 con 0x7f7a9815d640 2024-03-29T10:12:47.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a7c0035b0 con 0x7f7a9815d640 2024-03-29T10:12:47.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184716470 0 0) 0x7f7a980b6840 con 0x7f7a980ceec0 2024-03-29T10:12:47.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a980d30d0 con 0x7f7a980ceec0 2024-03-29T10:12:47.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2589534302 0 0) 0x7f7a7c0035b0 con 0x7f7a9815d640 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 >> v2:172.21.15.159:3300/0 conn(0x7f7a981619d0 msgr2=0x7f7a981685f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 --2- 172.21.15.12:0/3375135301 >> v2:172.21.15.159:3300/0 conn(0x7f7a981619d0 0x7f7a981685f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 >> v1:172.21.15.159:6789/0 conn(0x7f7a980ceec0 legacy=0x7f7a981612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a980d2df0 con 0x7f7a9815d640 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f7a88001670 con 0x7f7a9815d640 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 >> v1:172.21.15.12:6789/0 conn(0x7f7a9815d640 legacy=0x7f7a980d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c003b90 0x7f7a7c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c004850 0x7f7a7c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c005210 0x7f7a7c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a980d2b30 con 0x7f7a7c005210 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c00c140 con 0x7f7a7c004850 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.516+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c00c300 con 0x7f7a7c003b90 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a96800700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c005210 0x7f7a7c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a95e00700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c003b90 0x7f7a7c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a95400700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c004850 0x7f7a7c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a96800700 1 -- 172.21.15.12:0/3375135301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c003b90 msgr2=0x7f7a7c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a96800700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c003b90 0x7f7a7c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a96800700 1 -- 172.21.15.12:0/3375135301 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c004850 msgr2=0x7f7a7c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a96800700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c004850 0x7f7a7c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a96800700 1 -- 172.21.15.12:0/3375135301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c0035b0 con 0x7f7a7c005210 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a96800700 1 -- 172.21.15.12:0/3375135301 reap_dead start 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a96800700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c005210 0x7f7a7c00ba10 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f7a90009fc0 tx=0x7f7a7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a69025d1eadff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.517+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a90003a20 con 0x7f7a7c005210 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.518+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9000f040 con 0x7f7a7c005210 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.518+0000 7f7a94a00700 1 -- 172.21.15.12:0/3375135301 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a90003da0 con 0x7f7a7c005210 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.518+0000 7f7a9ca00700 1 -- 172.21.15.12:0/3375135301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c005210 msgr2=0x7f7a7c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.518+0000 7f7a9ca00700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c005210 0x7f7a7c00ba10 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f7a90009fc0 tx=0x7f7a7c005880 comp rx=0 tx=0).stop 2024-03-29T10:12:47.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.518+0000 7f7a9ca00700 1 -- 172.21.15.12:0/3375135301 shutdown_connections 2024-03-29T10:12:47.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.518+0000 7f7a9ca00700 1 --2- 172.21.15.12:0/3375135301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c005210 0x7f7a7c00ba10 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.518+0000 7f7a9ca00700 1 -- 172.21.15.12:0/3375135301 >> 172.21.15.12:0/3375135301 conn(0x7f7a9808aff0 msgr2=0x7f7a980d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:47.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.518+0000 7f7a9ca00700 1 -- 172.21.15.12:0/3375135301 shutdown_connections 2024-03-29T10:12:47.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.518+0000 7f7a9ca00700 1 -- 172.21.15.12:0/3375135301 wait complete. 2024-03-29T10:12:47.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.520+0000 7f7a9ca00700 1 Processor -- start 2024-03-29T10:12:47.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.520+0000 7f7a9ca00700 1 -- start start 2024-03-29T10:12:47.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a9ca00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c005210 0x7f7a9815ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a9ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a98155c30 0x7f7a981560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a9ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a981582e0 0x7f7a98158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a9ca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a98160120 con 0x7f7a981582e0 2024-03-29T10:12:47.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a9ca00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9815f2b0 con 0x7f7a7c005210 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a9ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a981565f0 con 0x7f7a98155c30 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a96800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a981582e0 0x7f7a98158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a96800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a981582e0 0x7f7a98158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41390/0 (socket says 172.21.15.12:41390) 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a96800700 1 -- 172.21.15.12:0/638059242 learned_addr learned my addr 172.21.15.12:0/638059242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a95e00700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c005210 0x7f7a9815ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.521+0000 7f7a95400700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a98155c30 0x7f7a981560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a96800700 1 -- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a98155c30 msgr2=0x7f7a981560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a96800700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a98155c30 0x7f7a981560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a96800700 1 -- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c005210 msgr2=0x7f7a9815ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a96800700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c005210 0x7f7a9815ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a96800700 1 -- 172.21.15.12:0/638059242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9815f4a0 con 0x7f7a981582e0 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a96800700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a981582e0 0x7f7a98158790 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f7a980d3520 tx=0x7f7a90009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a8e000700 1 -- 172.21.15.12:0/638059242 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a90003e10 con 0x7f7a981582e0 2024-03-29T10:12:47.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a8e000700 1 -- 172.21.15.12:0/638059242 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a90004250 con 0x7f7a981582e0 2024-03-29T10:12:47.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a8e000700 1 -- 172.21.15.12:0/638059242 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a90004600 con 0x7f7a981582e0 2024-03-29T10:12:47.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a9ca00700 1 -- 172.21.15.12:0/638059242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9815f780 con 0x7f7a981582e0 2024-03-29T10:12:47.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.522+0000 7f7a9ca00700 1 -- 172.21.15.12:0/638059242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9815fd40 con 0x7f7a981582e0 2024-03-29T10:12:47.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.523+0000 7f7a9ca00700 1 -- 172.21.15.12:0/638059242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a98151b20 con 0x7f7a981582e0 2024-03-29T10:12:47.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.524+0000 7f7a8e000700 1 -- 172.21.15.12:0/638059242 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7a90004760 con 0x7f7a981582e0 2024-03-29T10:12:47.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.525+0000 7f7a8e000700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a7806adb0 0x7f7a7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.525+0000 7f7a8e000700 1 -- 172.21.15.12:0/638059242 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a9008a660 con 0x7f7a981582e0 2024-03-29T10:12:47.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.525+0000 7f7a95e00700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a7806adb0 0x7f7a7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.526+0000 7f7a95e00700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a7806adb0 0x7f7a7806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7a88000c00 tx=0x7f7a88001e20 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:47.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.530+0000 7f7a8e000700 1 -- 172.21.15.12:0/638059242 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7a98151b20 con 0x7f7a981582e0 2024-03-29T10:12:47.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.745+0000 7f7a9ca00700 1 -- 172.21.15.12:0/638059242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a98160260 con 0x7f7a981582e0 2024-03-29T10:12:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.746+0000 7f7a8e000700 1 -- 172.21.15.12:0/638059242 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+16064 (secure 0 0 0) 0x7f7a98160260 con 0x7f7a981582e0 2024-03-29T10:12:47.746 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":123,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:12:46.810481+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:12:45.171490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-29T10:12:45.787701+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:12:47.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.748+0000 7f7a9ca00700 1 -- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a7806adb0 msgr2=0x7f7a7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.748+0000 7f7a9ca00700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a7806adb0 0x7f7a7806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7a88000c00 tx=0x7f7a88001e20 comp rx=0 tx=0).stop 2024-03-29T10:12:47.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.748+0000 7f7a9ca00700 1 -- 172.21.15.12:0/638059242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a981582e0 msgr2=0x7f7a98158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.748+0000 7f7a9ca00700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a981582e0 0x7f7a98158790 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f7a980d3520 tx=0x7f7a90009750 comp rx=0 tx=0).stop 2024-03-29T10:12:47.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.748+0000 7f7a9ca00700 1 -- 172.21.15.12:0/638059242 shutdown_connections 2024-03-29T10:12:47.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.748+0000 7f7a9ca00700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a7806adb0 0x7f7a7806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.748+0000 7f7a9ca00700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a981582e0 0x7f7a98158790 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.748+0000 7f7a9ca00700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c005210 0x7f7a9815ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.748+0000 7f7a9ca00700 1 --2- 172.21.15.12:0/638059242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a98155c30 0x7f7a981560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.748+0000 7f7a9ca00700 1 -- 172.21.15.12:0/638059242 >> 172.21.15.12:0/638059242 conn(0x7f7a9808aff0 msgr2=0x7f7a9815bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:47.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.749+0000 7f7a9ca00700 1 -- 172.21.15.12:0/638059242 shutdown_connections 2024-03-29T10:12:47.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.749+0000 7f7a9ca00700 1 -- 172.21.15.12:0/638059242 wait complete. 2024-03-29T10:12:47.767 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T10:12:47.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.973+0000 7f3ac0800700 1 Processor -- start 2024-03-29T10:12:47.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.973+0000 7f3ac0800700 1 -- start start 2024-03-29T10:12:47.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.974+0000 7f3ac0800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3ab80d21f0 0x7f3ab80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.974+0000 7f3ac0800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab80ce270 con 0x7f3ab80cec40 2024-03-29T10:12:47.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.974+0000 7f3ac0800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab815b600 con 0x7f3ab81611b0 2024-03-29T10:12:47.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.974+0000 7f3ac0800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab80d2760 con 0x7f3ab80d21f0 2024-03-29T10:12:47.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.974+0000 7f3abe800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3ab80cec40 0x7f3ab8161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44394/0 (socket says 172.21.15.12:44394) 2024-03-29T10:12:47.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.974+0000 7f3abe800700 1 -- 172.21.15.12:0/3127859673 learned_addr learned my addr 172.21.15.12:0/3127859673 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:47.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.974+0000 7f3abd400700 1 --2- 172.21.15.12:0/3127859673 >> v2:172.21.15.159:3300/0 conn(0x7f3ab80d21f0 0x7f3ab80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.975+0000 7f3abca00700 1 -- 172.21.15.12:0/3127859673 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184515848 0 0) 0x7f3ab80ce270 con 0x7f3ab80cec40 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.975+0000 7f3abca00700 1 -- 172.21.15.12:0/3127859673 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa00035b0 con 0x7f3ab80cec40 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.975+0000 7f3abd400700 1 -- 172.21.15.12:0/3127859673 >> v1:172.21.15.159:6789/0 conn(0x7f3ab81611b0 legacy=0x7f3ab815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.975+0000 7f3abd400700 1 -- 172.21.15.12:0/3127859673 >> v1:172.21.15.12:6789/0 conn(0x7f3ab80cec40 legacy=0x7f3ab8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.975+0000 7f3abd400700 1 -- 172.21.15.12:0/3127859673 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab80d2a40 con 0x7f3ab80d21f0 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.975+0000 7f3abd400700 1 --2- 172.21.15.12:0/3127859673 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ab80d21f0 0x7f3ab80ce700 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009840 tx=0x7f3aa8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca95827490fcb377 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.976+0000 7f3abca00700 1 -- 172.21.15.12:0/3127859673 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8011830 con 0x7f3ab80d21f0 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.976+0000 7f3abca00700 1 -- 172.21.15.12:0/3127859673 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa8011990 con 0x7f3ab80d21f0 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.976+0000 7f3abca00700 1 -- 172.21.15.12:0/3127859673 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8011d20 con 0x7f3ab80d21f0 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.976+0000 7f3ac0800700 1 -- 172.21.15.12:0/3127859673 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ab80d21f0 msgr2=0x7f3ab80ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.976+0000 7f3ac0800700 1 --2- 172.21.15.12:0/3127859673 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ab80d21f0 0x7f3ab80ce700 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009840 tx=0x7f3aa8009b50 comp rx=0 tx=0).stop 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.976+0000 7f3ac0800700 1 -- 172.21.15.12:0/3127859673 shutdown_connections 2024-03-29T10:12:47.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.976+0000 7f3ac0800700 1 --2- 172.21.15.12:0/3127859673 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ab80d21f0 0x7f3ab80ce700 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.976+0000 7f3ac0800700 1 -- 172.21.15.12:0/3127859673 >> 172.21.15.12:0/3127859673 conn(0x7f3ab808aff0 msgr2=0x7f3ab80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:47.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.976+0000 7f3ac0800700 1 -- 172.21.15.12:0/3127859673 shutdown_connections 2024-03-29T10:12:47.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.976+0000 7f3ac0800700 1 -- 172.21.15.12:0/3127859673 wait complete. 2024-03-29T10:12:47.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.978+0000 7f3ac0800700 1 Processor -- start 2024-03-29T10:12:47.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.978+0000 7f3ac0800700 1 -- start start 2024-03-29T10:12:47.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.979+0000 7f3ac0800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ab80cec40 0x7f3ab80b8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.979+0000 7f3ac0800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ab80d21f0 0x7f3ab815a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.979+0000 7f3ac0800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ab81611b0 0x7f3ab815ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.979+0000 7f3ac0800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab81e7170 con 0x7f3ab81611b0 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.979+0000 7f3ac0800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab81e7450 con 0x7f3ab80d21f0 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.979+0000 7f3ac0800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab81e7730 con 0x7f3ab80cec40 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.979+0000 7f3abe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ab81611b0 0x7f3ab815ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.979+0000 7f3abe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ab81611b0 0x7f3ab815ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41402/0 (socket says 172.21.15.12:41402) 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.979+0000 7f3abe800700 1 -- 172.21.15.12:0/1425792656 learned_addr learned my addr 172.21.15.12:0/1425792656 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3abde00700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ab80cec40 0x7f3ab80b8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3abd400700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ab80d21f0 0x7f3ab815a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3abe800700 1 -- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ab80cec40 msgr2=0x7f3ab80b8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3abe800700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ab80cec40 0x7f3ab80b8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3abe800700 1 -- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ab80d21f0 msgr2=0x7f3ab815a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3abe800700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ab80d21f0 0x7f3ab815a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3abe800700 1 -- 172.21.15.12:0/1425792656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa80094a0 con 0x7f3ab81611b0 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3abe800700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ab81611b0 0x7f3ab815ab50 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400caa0 tx=0x7f3ab400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:47.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3ab2000700 1 -- 172.21.15.12:0/1425792656 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab400a9e0 con 0x7f3ab81611b0 2024-03-29T10:12:47.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3ab2000700 1 -- 172.21.15.12:0/1425792656 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab400ab40 con 0x7f3ab81611b0 2024-03-29T10:12:47.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3ac0800700 1 -- 172.21.15.12:0/1425792656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab81e7a10 con 0x7f3ab81611b0 2024-03-29T10:12:47.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.980+0000 7f3ab2000700 1 -- 172.21.15.12:0/1425792656 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4012400 con 0x7f3ab81611b0 2024-03-29T10:12:47.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.981+0000 7f3ac0800700 1 -- 172.21.15.12:0/1425792656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab80b8880 con 0x7f3ab81611b0 2024-03-29T10:12:47.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.982+0000 7f3ac0800700 1 -- 172.21.15.12:0/1425792656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab81e7f30 con 0x7f3ab81611b0 2024-03-29T10:12:47.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.983+0000 7f3ab2000700 1 -- 172.21.15.12:0/1425792656 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3ab401b3b0 con 0x7f3ab81611b0 2024-03-29T10:12:47.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.983+0000 7f3ab2000700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a9c06adb0 0x7f3a9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:47.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.984+0000 7f3ab2000700 1 -- 172.21.15.12:0/1425792656 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ab4089140 con 0x7f3ab81611b0 2024-03-29T10:12:47.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.984+0000 7f3abde00700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a9c06adb0 0x7f3a9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:47.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.985+0000 7f3abde00700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a9c06adb0 0x7f3a9c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3aac000f60 tx=0x7f3aac009040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:47.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:47.988+0000 7f3ab2000700 1 -- 172.21.15.12:0/1425792656 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ab81e7f30 con 0x7f3ab81611b0 2024-03-29T10:12:48.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:48.247+0000 7f3ac0800700 1 -- 172.21.15.12:0/1425792656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3ab81e7f30 con 0x7f3ab81611b0 2024-03-29T10:12:48.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-03-29T10:12:49.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.086+0000 7f3ab2000700 1 -- 172.21.15.12:0/1425792656 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v54) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3ab81e7f30 con 0x7f3ab81611b0 2024-03-29T10:12:49.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 -- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a9c06adb0 msgr2=0x7f3a9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:49.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a9c06adb0 0x7f3a9c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3aac000f60 tx=0x7f3aac009040 comp rx=0 tx=0).stop 2024-03-29T10:12:49.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 -- 172.21.15.12:0/1425792656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ab81611b0 msgr2=0x7f3ab815ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:49.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ab81611b0 0x7f3ab815ab50 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400caa0 tx=0x7f3ab400cdb0 comp rx=0 tx=0).stop 2024-03-29T10:12:49.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 -- 172.21.15.12:0/1425792656 shutdown_connections 2024-03-29T10:12:49.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3a9c06adb0 0x7f3a9c06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ab81611b0 0x7f3ab815ab50 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ab80cec40 0x7f3ab80b8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 --2- 172.21.15.12:0/1425792656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ab80d21f0 0x7f3ab815a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 -- 172.21.15.12:0/1425792656 >> 172.21.15.12:0/1425792656 conn(0x7f3ab808aff0 msgr2=0x7f3ab8160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 -- 172.21.15.12:0/1425792656 shutdown_connections 2024-03-29T10:12:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.089+0000 7f3ac0800700 1 -- 172.21.15.12:0/1425792656 wait complete. 2024-03-29T10:12:49.112 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T10:12:49.112 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T10:12:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.321+0000 7febe9400700 1 Processor -- start 2024-03-29T10:12:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.321+0000 7febe9400700 1 -- start start 2024-03-29T10:12:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.322+0000 7febe9400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe406aea0 0x7febe40b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.322+0000 7febe9400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe4161150 con 0x7febe415ef70 2024-03-29T10:12:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.322+0000 7febe9400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe406dba0 con 0x7febe4160d70 2024-03-29T10:12:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.322+0000 7febe9400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe40b4cd0 con 0x7febe406aea0 2024-03-29T10:12:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7febe415ef70 0x7febe406a790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47418/0 (socket says 172.21.15.12:47418) 2024-03-29T10:12:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1e00700 1 -- 172.21.15.12:0/2977683961 learned_addr learned my addr 172.21.15.12:0/2977683961 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe3200700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe406aea0 0x7febe40b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548532569 0 0) 0x7febe4161150 con 0x7febe415ef70 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc80035b0 con 0x7febe415ef70 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332456729 0 0) 0x7febe406dba0 con 0x7febe4160d70 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe4161150 con 0x7febe4160d70 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1302345183 0 0) 0x7febc80035b0 con 0x7febe415ef70 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe406aea0 msgr2=0x7febe40b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1400700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe406aea0 0x7febe40b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 >> v1:172.21.15.159:6789/0 conn(0x7febe4160d70 legacy=0x7febe415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.323+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe40b4ee0 con 0x7febe415ef70 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7febd0001670 con 0x7febe415ef70 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 >> v1:172.21.15.12:6789/0 conn(0x7febe415ef70 legacy=0x7febe406a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:49.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1400700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7febc8003bc0 0x7febc8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1400700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 0x7febc8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1400700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe406aea0 0x7febc8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40b4cd0 con 0x7febc8004850 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc800bac0 con 0x7febc8003bc0 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc800bc80 con 0x7febe406aea0 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1e00700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 0x7febc8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe2800700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7febc8003bc0 0x7febc8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe3200700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe406aea0 0x7febc8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1e00700 1 -- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe406aea0 msgr2=0x7febc8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1e00700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe406aea0 0x7febc8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1e00700 1 -- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7febc8003bc0 msgr2=0x7febc8003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1e00700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7febc8003bc0 0x7febc8003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe1e00700 1 -- 172.21.15.12:0/2977683961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc80035b0 con 0x7febc8004850 2024-03-29T10:12:49.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.324+0000 7febe3200700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe406aea0 0x7febc8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.325+0000 7febe1e00700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 0x7febc8004cd0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7febd0001eb0 tx=0x7febc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a233ae73553ea4b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.325+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd000ccf0 con 0x7febc8004850 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.325+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd000ce70 con 0x7febc8004850 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.325+0000 7febe1400700 1 -- 172.21.15.12:0/2977683961 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0005b80 con 0x7febc8004850 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.325+0000 7febe9400700 1 -- 172.21.15.12:0/2977683961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 msgr2=0x7febc8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.325+0000 7febe9400700 1 --2- 172.21.15.12:0/2977683961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 0x7febc8004cd0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7febd0001eb0 tx=0x7febc8005880 comp rx=0 tx=0).stop 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.325+0000 7febe3200700 1 -- 172.21.15.12:0/2977683961 reap_dead start 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.325+0000 7febe9400700 1 -- 172.21.15.12:0/2977683961 shutdown_connections 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.325+0000 7febe9400700 1 -- 172.21.15.12:0/2977683961 >> 172.21.15.12:0/2977683961 conn(0x7febe40b5720 msgr2=0x7febe40b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.326+0000 7febe9400700 1 -- 172.21.15.12:0/2977683961 shutdown_connections 2024-03-29T10:12:49.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.326+0000 7febe9400700 1 -- 172.21.15.12:0/2977683961 wait complete. 2024-03-29T10:12:49.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.327+0000 7febe9400700 1 Processor -- start 2024-03-29T10:12:49.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.327+0000 7febe9400700 1 -- start start 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.328+0000 7febe9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 0x7febe4146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.328+0000 7febe9400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7febe40ae050 0x7febe40b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.328+0000 7febe9400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe40b0f30 0x7febe409d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.328+0000 7febe9400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe4147520 con 0x7febc8004850 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.328+0000 7febe9400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe41480a0 con 0x7febe40ae050 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.328+0000 7febe9400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe40b13b0 con 0x7febe40b0f30 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 0x7febe4146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 0x7febe4146cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52288/0 (socket says 172.21.15.12:52288) 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe2800700 1 -- 172.21.15.12:0/3032908174 learned_addr learned my addr 172.21.15.12:0/3032908174 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe1e00700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7febe40ae050 0x7febe40b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe3200700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe40b0f30 0x7febe409d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe2800700 1 -- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe40b0f30 msgr2=0x7febe409d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:49.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe2800700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe40b0f30 0x7febe409d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe2800700 1 -- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7febe40ae050 msgr2=0x7febe40b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe2800700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7febe40ae050 0x7febe40b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe2800700 1 -- 172.21.15.12:0/3032908174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe4147200 con 0x7febc8004850 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe3200700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe40b0f30 0x7febe409d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.329+0000 7febe2800700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 0x7febe4146cc0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7febd400f7a0 tx=0x7febd400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.330+0000 7febdaa00700 1 -- 172.21.15.12:0/3032908174 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4005320 con 0x7febc8004850 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.330+0000 7febdaa00700 1 -- 172.21.15.12:0/3032908174 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd40054a0 con 0x7febc8004850 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.330+0000 7febe9400700 1 -- 172.21.15.12:0/3032908174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe4147770 con 0x7febc8004850 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.330+0000 7febdaa00700 1 -- 172.21.15.12:0/3032908174 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4006aa0 con 0x7febc8004850 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.330+0000 7febe9400700 1 -- 172.21.15.12:0/3032908174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe4147d30 con 0x7febc8004850 2024-03-29T10:12:49.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.331+0000 7febe9400700 1 -- 172.21.15.12:0/3032908174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe4067720 con 0x7febc8004850 2024-03-29T10:12:49.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.332+0000 7febdaa00700 1 -- 172.21.15.12:0/3032908174 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7febd4006c00 con 0x7febc8004850 2024-03-29T10:12:49.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.332+0000 7febdaa00700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7febc406ade0 0x7febc406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:49.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.332+0000 7febdaa00700 1 -- 172.21.15.12:0/3032908174 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febd4088d40 con 0x7febc8004850 2024-03-29T10:12:49.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.333+0000 7febe1e00700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7febc406ade0 0x7febc406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:49.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.334+0000 7febe1e00700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7febc406ade0 0x7febc406d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7febd0015040 tx=0x7febd0007f90 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:49.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.337+0000 7febdaa00700 1 -- 172.21.15.12:0/3032908174 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7febd4053e10 con 0x7febc8004850 2024-03-29T10:12:49.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:49.603+0000 7febe9400700 1 -- 172.21.15.12:0/3032908174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7febe406aba0 con 0x7febc8004850 2024-03-29T10:12:50.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.301+0000 7febdaa00700 1 -- 172.21.15.12:0/3032908174 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v55) v1 ==== 122+0+0 (secure 0 0 0) 0x7febd4057430 con 0x7febc8004850 2024-03-29T10:12:50.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 -- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7febc406ade0 msgr2=0x7febc406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:50.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7febc406ade0 0x7febc406d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7febd0015040 tx=0x7febd0007f90 comp rx=0 tx=0).stop 2024-03-29T10:12:50.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 -- 172.21.15.12:0/3032908174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 msgr2=0x7febe4146cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:50.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 0x7febe4146cc0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7febd400f7a0 tx=0x7febd400fab0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 -- 172.21.15.12:0/3032908174 shutdown_connections 2024-03-29T10:12:50.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7febc406ade0 0x7febc406d2a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7febe40b0f30 0x7febe409d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febc8004850 0x7febe4146cc0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 --2- 172.21.15.12:0/3032908174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7febe40ae050 0x7febe40b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 -- 172.21.15.12:0/3032908174 >> 172.21.15.12:0/3032908174 conn(0x7febe40b5720 msgr2=0x7febe409a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 -- 172.21.15.12:0/3032908174 shutdown_connections 2024-03-29T10:12:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.304+0000 7febe9400700 1 -- 172.21.15.12:0/3032908174 wait complete. 2024-03-29T10:12:50.325 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T10:12:50.326 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:12:50.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.529+0000 7fa538400700 1 Processor -- start 2024-03-29T10:12:50.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.529+0000 7fa538400700 1 -- start start 2024-03-29T10:12:50.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.530+0000 7fa538400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa530072580 0x7fa5300b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:50.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa538400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5300b2180 con 0x7fa5300b3af0 2024-03-29T10:12:50.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa538400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53015b100 con 0x7fa5300b2e50 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa538400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53015b3e0 con 0x7fa530072580 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa535a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa5300b3af0 0x7fa53015a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47432/0 (socket says 172.21.15.12:47432) 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa535a00700 1 -- 172.21.15.12:0/1323434130 learned_addr learned my addr 172.21.15.12:0/1323434130 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa535000700 1 --2- 172.21.15.12:0/1323434130 >> v2:172.21.15.159:3300/0 conn(0x7fa530072580 0x7fa5300b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa52fe00700 1 -- 172.21.15.12:0/1323434130 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121847467 0 0) 0x7fa5300b2180 con 0x7fa5300b3af0 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa52fe00700 1 -- 172.21.15.12:0/1323434130 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5180057c0 con 0x7fa5300b3af0 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa52fe00700 1 -- 172.21.15.12:0/1323434130 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2129678951 0 0) 0x7fa5180057c0 con 0x7fa5300b3af0 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa535000700 1 -- 172.21.15.12:0/1323434130 >> v1:172.21.15.159:6789/0 conn(0x7fa5300b2e50 legacy=0x7fa5300709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.531+0000 7fa535000700 1 -- 172.21.15.12:0/1323434130 >> v1:172.21.15.12:6789/0 conn(0x7fa5300b3af0 legacy=0x7fa53015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.532+0000 7fa535000700 1 -- 172.21.15.12:0/1323434130 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5300b23f0 con 0x7fa530072580 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.532+0000 7fa535000700 1 --2- 172.21.15.12:0/1323434130 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa530072580 0x7fa5300b2910 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009840 tx=0x7fa51c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef4314187cb02024 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.532+0000 7fa52fe00700 1 -- 172.21.15.12:0/1323434130 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa524001670 con 0x7fa530072580 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.532+0000 7fa52fe00700 1 -- 172.21.15.12:0/1323434130 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c004030 con 0x7fa530072580 2024-03-29T10:12:50.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.532+0000 7fa52fe00700 1 -- 172.21.15.12:0/1323434130 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0043c0 con 0x7fa530072580 2024-03-29T10:12:50.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.533+0000 7fa538400700 1 -- 172.21.15.12:0/1323434130 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa530072580 msgr2=0x7fa5300b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:50.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.533+0000 7fa538400700 1 --2- 172.21.15.12:0/1323434130 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa530072580 0x7fa5300b2910 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009840 tx=0x7fa51c009b50 comp rx=0 tx=0).stop 2024-03-29T10:12:50.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.533+0000 7fa538400700 1 -- 172.21.15.12:0/1323434130 shutdown_connections 2024-03-29T10:12:50.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.533+0000 7fa538400700 1 --2- 172.21.15.12:0/1323434130 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa530072580 0x7fa5300b2910 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.533+0000 7fa538400700 1 -- 172.21.15.12:0/1323434130 >> 172.21.15.12:0/1323434130 conn(0x7fa5300cf190 msgr2=0x7fa5300cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:50.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.533+0000 7fa538400700 1 -- 172.21.15.12:0/1323434130 shutdown_connections 2024-03-29T10:12:50.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.533+0000 7fa538400700 1 -- 172.21.15.12:0/1323434130 wait complete. 2024-03-29T10:12:50.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.534+0000 7fa538400700 1 Processor -- start 2024-03-29T10:12:50.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.535+0000 7fa538400700 1 -- start start 2024-03-29T10:12:50.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa538400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa530072580 0x7fa53014ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:50.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa538400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5300b2e50 0x7fa530158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:50.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa538400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5300b3af0 0x7fa530153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:50.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa538400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5301580d0 con 0x7fa530072580 2024-03-29T10:12:50.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa538400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300adce0 con 0x7fa5300b3af0 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa538400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300adf30 con 0x7fa5300b2e50 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa535a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa530072580 0x7fa53014ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa535a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa530072580 0x7fa53014ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52298/0 (socket says 172.21.15.12:52298) 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa535a00700 1 -- 172.21.15.12:0/1484647460 learned_addr learned my addr 172.21.15.12:0/1484647460 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa535000700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5300b2e50 0x7fa530158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa536400700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5300b3af0 0x7fa530153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa535a00700 1 -- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5300b2e50 msgr2=0x7fa530158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa535a00700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5300b2e50 0x7fa530158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa535a00700 1 -- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5300b3af0 msgr2=0x7fa530153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa535a00700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5300b3af0 0x7fa530153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa535a00700 1 -- 172.21.15.12:0/1484647460 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c0094a0 con 0x7fa530072580 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.536+0000 7fa536400700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5300b3af0 0x7fa530153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.537+0000 7fa535a00700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa530072580 0x7fa53014ebe0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fa52400c550 tx=0x7fa52400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:50.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.537+0000 7fa52d600700 1 -- 172.21.15.12:0/1484647460 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa524004c80 con 0x7fa530072580 2024-03-29T10:12:50.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.537+0000 7fa52d600700 1 -- 172.21.15.12:0/1484647460 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa524004de0 con 0x7fa530072580 2024-03-29T10:12:50.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.537+0000 7fa538400700 1 -- 172.21.15.12:0/1484647460 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5301582e0 con 0x7fa530072580 2024-03-29T10:12:50.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.537+0000 7fa52d600700 1 -- 172.21.15.12:0/1484647460 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa524007310 con 0x7fa530072580 2024-03-29T10:12:50.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.537+0000 7fa538400700 1 -- 172.21.15.12:0/1484647460 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa530159020 con 0x7fa530072580 2024-03-29T10:12:50.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.538+0000 7fa538400700 1 -- 172.21.15.12:0/1484647460 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa530158800 con 0x7fa530072580 2024-03-29T10:12:50.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.539+0000 7fa52d600700 1 -- 172.21.15.12:0/1484647460 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa524007470 con 0x7fa530072580 2024-03-29T10:12:50.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.540+0000 7fa52d600700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa51006adb0 0x7fa51006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:50.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.540+0000 7fa52d600700 1 -- 172.21.15.12:0/1484647460 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa524089300 con 0x7fa530072580 2024-03-29T10:12:50.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.541+0000 7fa535000700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa51006adb0 0x7fa51006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:50.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.541+0000 7fa535000700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa51006adb0 0x7fa51006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009840 tx=0x7fa51c008030 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:50.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.545+0000 7fa52d600700 1 -- 172.21.15.12:0/1484647460 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa530158800 con 0x7fa530072580 2024-03-29T10:12:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.802+0000 7fa538400700 1 -- 172.21.15.12:0/1484647460 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa530158800 con 0x7fa530072580 2024-03-29T10:12:50.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.803+0000 7fa52d600700 1 -- 172.21.15.12:0/1484647460 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4736 (secure 0 0 0) 0x7fa530158800 con 0x7fa530072580 2024-03-29T10:12:50.802 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:50.802 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":55,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7645,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/553963486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":553963486},{"type":"v1","addr":"172.21.15.159:6837","nonce":553963486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3051067707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3051067707},{"type":"v1","addr":"172.21.15.159:6836","nonce":3051067707}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3629410472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3629410472},{"type":"v1","addr":"172.21.15.12:6833","nonce":3629410472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:12:46.810806+0000","modified":"2024-03-29T10:12:50.294107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7917},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7917":{"gid":7917,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/753821395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":753821395},{"type":"v1","addr":"172.21.15.12:6835","nonce":753821395}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-29T10:12:50.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 -- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa51006adb0 msgr2=0x7fa51006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:50.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa51006adb0 0x7fa51006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009840 tx=0x7fa51c008030 comp rx=0 tx=0).stop 2024-03-29T10:12:50.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 -- 172.21.15.12:0/1484647460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa530072580 msgr2=0x7fa53014ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:50.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa530072580 0x7fa53014ebe0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fa52400c550 tx=0x7fa52400c910 comp rx=0 tx=0).stop 2024-03-29T10:12:50.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 -- 172.21.15.12:0/1484647460 shutdown_connections 2024-03-29T10:12:50.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa51006adb0 0x7fa51006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5300b3af0 0x7fa530153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa530072580 0x7fa53014ebe0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 --2- 172.21.15.12:0/1484647460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5300b2e50 0x7fa530158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:50.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 -- 172.21.15.12:0/1484647460 >> 172.21.15.12:0/1484647460 conn(0x7fa5300cf190 msgr2=0x7fa5300acde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:50.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 -- 172.21.15.12:0/1484647460 shutdown_connections 2024-03-29T10:12:50.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:50.805+0000 7fa538400700 1 -- 172.21.15.12:0/1484647460 wait complete. 2024-03-29T10:12:50.806 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 55 2024-03-29T10:12:50.823 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:12:51.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.022+0000 7fdce6200700 1 Processor -- start 2024-03-29T10:12:51.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.022+0000 7fdce6200700 1 -- start start 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.023+0000 7fdce6200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdce015b5d0 0x7fdce00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.023+0000 7fdce6200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdce00cb3f0 0x7fdce00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.023+0000 7fdce6200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce015fd50 con 0x7fdce00ae930 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.023+0000 7fdce6200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce015b9b0 con 0x7fdce015b5d0 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.023+0000 7fdce6200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce015f140 con 0x7fdce00cb3f0 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.023+0000 7fdcdf400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdce015b5d0 0x7fdce00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.023+0000 7fdcdfe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdce00ae930 0x7fdce0140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47444/0 (socket says 172.21.15.12:47444) 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.023+0000 7fdcdf400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdce015b5d0 0x7fdce00ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52300/0 (socket says 172.21.15.12:52300) 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.023+0000 7fdcdfe00700 1 -- 172.21.15.12:0/3316084129 learned_addr learned my addr 172.21.15.12:0/3316084129 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.023+0000 7fdcdea00700 1 --2- 172.21.15.12:0/3316084129 >> v2:172.21.15.159:3300/0 conn(0x7fdce00cb3f0 0x7fdce00ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.024+0000 7fdcdf400700 1 -- 172.21.15.12:0/3316084129 >> v2:172.21.15.159:3300/0 conn(0x7fdce00cb3f0 msgr2=0x7fdce00ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.024+0000 7fdcdf400700 1 --2- 172.21.15.12:0/3316084129 >> v2:172.21.15.159:3300/0 conn(0x7fdce00cb3f0 0x7fdce00ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.024+0000 7fdcdf400700 1 -- 172.21.15.12:0/3316084129 >> v1:172.21.15.12:6789/0 conn(0x7fdce00ae930 legacy=0x7fdce0140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.024+0000 7fdcdf400700 1 -- 172.21.15.12:0/3316084129 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce015f420 con 0x7fdce015b5d0 2024-03-29T10:12:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.024+0000 7fdcde000700 1 -- 172.21.15.12:0/3316084129 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249642239 0 0) 0x7fdce015fd50 con 0x7fdce00ae930 2024-03-29T10:12:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.024+0000 7fdcdf400700 1 --2- 172.21.15.12:0/3316084129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce015b5d0 0x7fdce00ae3f0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fdce006c6c0 tx=0x7fdcd0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746e77202ee5d092 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.024+0000 7fdcde000700 1 -- 172.21.15.12:0/3316084129 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0011830 con 0x7fdce015b5d0 2024-03-29T10:12:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.024+0000 7fdcde000700 1 -- 172.21.15.12:0/3316084129 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd0011990 con 0x7fdce015b5d0 2024-03-29T10:12:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.024+0000 7fdcde000700 1 -- 172.21.15.12:0/3316084129 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0011d20 con 0x7fdce015b5d0 2024-03-29T10:12:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.025+0000 7fdce6200700 1 -- 172.21.15.12:0/3316084129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce015b5d0 msgr2=0x7fdce00ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.025+0000 7fdce6200700 1 --2- 172.21.15.12:0/3316084129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce015b5d0 0x7fdce00ae3f0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fdce006c6c0 tx=0x7fdcd0009b20 comp rx=0 tx=0).stop 2024-03-29T10:12:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.025+0000 7fdce6200700 1 -- 172.21.15.12:0/3316084129 shutdown_connections 2024-03-29T10:12:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.025+0000 7fdce6200700 1 --2- 172.21.15.12:0/3316084129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce015b5d0 0x7fdce00ae3f0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.025+0000 7fdce6200700 1 --2- 172.21.15.12:0/3316084129 >> v2:172.21.15.159:3300/0 conn(0x7fdce00cb3f0 0x7fdce00ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.025+0000 7fdce6200700 1 -- 172.21.15.12:0/3316084129 >> 172.21.15.12:0/3316084129 conn(0x7fdce00cf190 msgr2=0x7fdce00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.025+0000 7fdce6200700 1 -- 172.21.15.12:0/3316084129 shutdown_connections 2024-03-29T10:12:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.025+0000 7fdce6200700 1 -- 172.21.15.12:0/3316084129 wait complete. 2024-03-29T10:12:51.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.027+0000 7fdce6200700 1 Processor -- start 2024-03-29T10:12:51.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.027+0000 7fdce6200700 1 -- start start 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdce6200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdce00ae930 0x7fdce01f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdce6200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdce015b5d0 0x7fdce00cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdce6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce00ac330 0x7fdce01f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdce6200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce01f7800 con 0x7fdce00ac330 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdce6200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00cd780 con 0x7fdce015b5d0 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdce6200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00cbba0 con 0x7fdce00ae930 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdcdfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce00ac330 0x7fdce01f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdcdfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce00ac330 0x7fdce01f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52312/0 (socket says 172.21.15.12:52312) 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdcdfe00700 1 -- 172.21.15.12:0/1793938420 learned_addr learned my addr 172.21.15.12:0/1793938420 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdcdea00700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdce015b5d0 0x7fdce00cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.028+0000 7fdcdf400700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdce00ae930 0x7fdce01f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcdfe00700 1 -- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdce00ae930 msgr2=0x7fdce01f72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcdfe00700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdce00ae930 0x7fdce01f72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcdfe00700 1 -- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdce015b5d0 msgr2=0x7fdce00cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcdfe00700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdce015b5d0 0x7fdce00cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcdfe00700 1 -- 172.21.15.12:0/1793938420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd00094a0 con 0x7fdce00ac330 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcdea00700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdce015b5d0 0x7fdce00cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcdf400700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdce00ae930 0x7fdce01f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcdfe00700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce00ac330 0x7fdce01f43b0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800cba0 tx=0x7fdcd800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcd7400700 1 -- 172.21.15.12:0/1793938420 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd800ab80 con 0x7fdce00ac330 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcd7400700 1 -- 172.21.15.12:0/1793938420 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd800ad00 con 0x7fdce00ac330 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdce6200700 1 -- 172.21.15.12:0/1793938420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce00cd990 con 0x7fdce00ac330 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdcd7400700 1 -- 172.21.15.12:0/1793938420 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8007310 con 0x7fdce00ac330 2024-03-29T10:12:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.029+0000 7fdce6200700 1 -- 172.21.15.12:0/1793938420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce00cbe40 con 0x7fdce00ac330 2024-03-29T10:12:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.030+0000 7fdce6200700 1 -- 172.21.15.12:0/1793938420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce0067720 con 0x7fdce00ac330 2024-03-29T10:12:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.031+0000 7fdcd7400700 1 -- 172.21.15.12:0/1793938420 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdcd8007470 con 0x7fdce00ac330 2024-03-29T10:12:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.032+0000 7fdcd7400700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdcc006adb0 0x7fdcc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.032+0000 7fdcd7400700 1 -- 172.21.15.12:0/1793938420 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdcd80892d0 con 0x7fdce00ac330 2024-03-29T10:12:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.033+0000 7fdcdf400700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdcc006adb0 0x7fdcc006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.033+0000 7fdcdf400700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdcc006adb0 0x7fdcc006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdcd00090b0 tx=0x7fdcd0008030 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:51.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.037+0000 7fdcd7400700 1 -- 172.21.15.12:0/1793938420 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdcd8054390 con 0x7fdce00ac330 2024-03-29T10:12:51.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.252+0000 7fdce6200700 1 -- 172.21.15.12:0/1793938420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdce006ea60 con 0x7fdce00ac330 2024-03-29T10:12:51.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.253+0000 7fdcd7400700 1 -- 172.21.15.12:0/1793938420 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16051 (secure 0 0 0) 0x7fdcd80579b0 con 0x7fdce00ac330 2024-03-29T10:12:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:51.253 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":124,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:12:47.813201+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:12:45.171490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-29T10:12:45.787701+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:12:51.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 -- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdcc006adb0 msgr2=0x7fdcc006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdcc006adb0 0x7fdcc006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdcd00090b0 tx=0x7fdcd0008030 comp rx=0 tx=0).stop 2024-03-29T10:12:51.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 -- 172.21.15.12:0/1793938420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce00ac330 msgr2=0x7fdce01f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce00ac330 0x7fdce01f43b0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800cba0 tx=0x7fdcd800cf60 comp rx=0 tx=0).stop 2024-03-29T10:12:51.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 -- 172.21.15.12:0/1793938420 shutdown_connections 2024-03-29T10:12:51.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdcc006adb0 0x7fdcc006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdce00ac330 0x7fdce01f43b0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdce00ae930 0x7fdce01f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 --2- 172.21.15.12:0/1793938420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdce015b5d0 0x7fdce00cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 -- 172.21.15.12:0/1793938420 >> 172.21.15.12:0/1793938420 conn(0x7fdce00cf190 msgr2=0x7fdce00aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:51.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 -- 172.21.15.12:0/1793938420 shutdown_connections 2024-03-29T10:12:51.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.256+0000 7fdce6200700 1 -- 172.21.15.12:0/1793938420 wait complete. 2024-03-29T10:12:51.274 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T10:12:51.274 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:12:51.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.473+0000 7f081d800700 1 Processor -- start 2024-03-29T10:12:51.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.473+0000 7f081d800700 1 -- start start 2024-03-29T10:12:51.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.474+0000 7f081d800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0818161ca0 0x7f08180cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.474+0000 7f081d800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08180ceda0 con 0x7f08180d18b0 2024-03-29T10:12:51.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.474+0000 7f081d800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08181612f0 con 0x7f0818160f10 2024-03-29T10:12:51.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.474+0000 7f081d800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08180d1d30 con 0x7f0818161ca0 2024-03-29T10:12:51.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f080fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f08180d18b0 0x7f0818161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47454/0 (socket says 172.21.15.12:47454) 2024-03-29T10:12:51.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f080fe00700 1 -- 172.21.15.12:0/1026456487 learned_addr learned my addr 172.21.15.12:0/1026456487 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:51.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f0817400700 1 --2- 172.21.15.12:0/1026456487 >> v2:172.21.15.159:3300/0 conn(0x7f0818161ca0 0x7f08180cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667925596 0 0) 0x7f08180ceda0 con 0x7f08180d18b0 2024-03-29T10:12:51.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc0035b0 con 0x7f08180d18b0 2024-03-29T10:12:51.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3383736509 0 0) 0x7f07fc0035b0 con 0x7f08180d18b0 2024-03-29T10:12:51.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 >> v2:172.21.15.159:3300/0 conn(0x7f0818161ca0 msgr2=0x7f08180cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:51.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f0816000700 1 --2- 172.21.15.12:0/1026456487 >> v2:172.21.15.159:3300/0 conn(0x7f0818161ca0 0x7f08180cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 >> v1:172.21.15.159:6789/0 conn(0x7f0818160f10 legacy=0x7f08180d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08180ce470 con 0x7f08180d18b0 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0800001670 con 0x7f08180d18b0 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.475+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 >> v1:172.21.15.12:6789/0 conn(0x7f08180d18b0 legacy=0x7f0818161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0816000700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07fc003a00 0x7f07fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0816000700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f07fc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0816000700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07fc005210 0x7f07fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08180d1d30 con 0x7f07fc004850 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc00c1b0 con 0x7f07fc003a00 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc00c370 con 0x7f07fc005210 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0817400700 1 --2- 172.21.15.12:0/1026456487 >> v2:172.21.15.159:3300/0 conn(0x7f0818161ca0 0x7f08180cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f080fe00700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f07fc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0817400700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07fc005210 0x7f07fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0816a00700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07fc003a00 0x7f07fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f080fe00700 1 -- 172.21.15.12:0/1026456487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07fc005210 msgr2=0x7f07fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f080fe00700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07fc005210 0x7f07fc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f080fe00700 1 -- 172.21.15.12:0/1026456487 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07fc003a00 msgr2=0x7f07fc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f080fe00700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07fc003a00 0x7f07fc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f080fe00700 1 -- 172.21.15.12:0/1026456487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc0035b0 con 0x7f07fc004850 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0817400700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07fc005210 0x7f07fc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.476+0000 7f0817400700 1 -- 172.21.15.12:0/1026456487 reap_dead start 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.477+0000 7f080fe00700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f07fc004cd0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f080000c240 tx=0x7f07fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad86cb1b22e845fd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.477+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800007c10 con 0x7f07fc004850 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.477+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080000ce00 con 0x7f07fc004850 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.477+0000 7f0816000700 1 -- 172.21.15.12:0/1026456487 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800005ba0 con 0x7f07fc004850 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.477+0000 7f081d800700 1 -- 172.21.15.12:0/1026456487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 msgr2=0x7f07fc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.477+0000 7f081d800700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f07fc004cd0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f080000c240 tx=0x7f07fc005880 comp rx=0 tx=0).stop 2024-03-29T10:12:51.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.477+0000 7f081d800700 1 -- 172.21.15.12:0/1026456487 shutdown_connections 2024-03-29T10:12:51.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.477+0000 7f081d800700 1 --2- 172.21.15.12:0/1026456487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f07fc004cd0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.477+0000 7f081d800700 1 -- 172.21.15.12:0/1026456487 >> 172.21.15.12:0/1026456487 conn(0x7f081808aff0 msgr2=0x7f08180d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:51.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.477+0000 7f081d800700 1 -- 172.21.15.12:0/1026456487 shutdown_connections 2024-03-29T10:12:51.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.478+0000 7f081d800700 1 -- 172.21.15.12:0/1026456487 wait complete. 2024-03-29T10:12:51.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.479+0000 7f081d800700 1 Processor -- start 2024-03-29T10:12:51.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.479+0000 7f081d800700 1 -- start start 2024-03-29T10:12:51.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.480+0000 7f081d800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f0818158ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.480+0000 7f081d800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08180b8190 0x7f08180b8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.480+0000 7f081d800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0818157330 0x7f08181577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.480+0000 7f081d800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08180b8b50 con 0x7f07fc004850 2024-03-29T10:12:51.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.480+0000 7f081d800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08181e8700 con 0x7f08180b8190 2024-03-29T10:12:51.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.480+0000 7f081d800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08181e7b80 con 0x7f0818157330 2024-03-29T10:12:51.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0816a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f0818158ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0816a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f0818158ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52330/0 (socket says 172.21.15.12:52330) 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0816a00700 1 -- 172.21.15.12:0/145183406 learned_addr learned my addr 172.21.15.12:0/145183406 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f080fe00700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08180b8190 0x7f08180b8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0817400700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0818157330 0x7f08181577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0816a00700 1 -- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0818157330 msgr2=0x7f08181577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0816a00700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0818157330 0x7f08181577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0816a00700 1 -- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08180b8190 msgr2=0x7f08180b8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0816a00700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08180b8190 0x7f08180b8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0816a00700 1 -- 172.21.15.12:0/145183406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08181e7e60 con 0x7f07fc004850 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f080fe00700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08180b8190 0x7f08180b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0817400700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0818157330 0x7f08181577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.481+0000 7f0816a00700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f0818158ea0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f080800e550 tx=0x7f080800e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.482+0000 7f080ea00700 1 -- 172.21.15.12:0/145183406 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08080052e0 con 0x7f07fc004850 2024-03-29T10:12:51.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.482+0000 7f080ea00700 1 -- 172.21.15.12:0/145183406 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808005440 con 0x7f07fc004850 2024-03-29T10:12:51.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.482+0000 7f080ea00700 1 -- 172.21.15.12:0/145183406 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08080038a0 con 0x7f07fc004850 2024-03-29T10:12:51.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.482+0000 7f081d800700 1 -- 172.21.15.12:0/145183406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08181e8140 con 0x7f07fc004850 2024-03-29T10:12:51.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.482+0000 7f081d800700 1 -- 172.21.15.12:0/145183406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0818155d30 con 0x7f07fc004850 2024-03-29T10:12:51.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.483+0000 7f081d800700 1 -- 172.21.15.12:0/145183406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0818157180 con 0x7f07fc004850 2024-03-29T10:12:51.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.484+0000 7f080ea00700 1 -- 172.21.15.12:0/145183406 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0808003a00 con 0x7f07fc004850 2024-03-29T10:12:51.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.484+0000 7f080ea00700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07f406adb0 0x7f07f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.485+0000 7f080ea00700 1 -- 172.21.15.12:0/145183406 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0808088b20 con 0x7f07fc004850 2024-03-29T10:12:51.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.485+0000 7f080fe00700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07f406adb0 0x7f07f406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.486+0000 7f080fe00700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07f406adb0 0x7f07f406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0800015040 tx=0x7f0800007000 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:51.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.489+0000 7f080ea00700 1 -- 172.21.15.12:0/145183406 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0808053d60 con 0x7f07fc004850 2024-03-29T10:12:51.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.698+0000 7f081d800700 1 -- 172.21.15.12:0/145183406 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08180ce120 con 0x7f07f406adb0 2024-03-29T10:12:51.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.709+0000 7f080ea00700 1 -- 172.21.15.12:0/145183406 <== mgr.7402 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289666 (secure 0 0 0) 0x7f0800007b30 con 0x7f07f406adb0 2024-03-29T10:12:51.713 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:51.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.713+0000 7f081d800700 1 -- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07f406adb0 msgr2=0x7f07f406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.713+0000 7f081d800700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07f406adb0 0x7f07f406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0800015040 tx=0x7f0800007000 comp rx=0 tx=0).stop 2024-03-29T10:12:51.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.713+0000 7f081d800700 1 -- 172.21.15.12:0/145183406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 msgr2=0x7f0818158ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.713+0000 7f081d800700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f0818158ea0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f080800e550 tx=0x7f080800e910 comp rx=0 tx=0).stop 2024-03-29T10:12:51.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.714+0000 7f081d800700 1 -- 172.21.15.12:0/145183406 shutdown_connections 2024-03-29T10:12:51.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.714+0000 7f081d800700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f07f406adb0 0x7f07f406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.714+0000 7f081d800700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0818157330 0x7f08181577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.714+0000 7f081d800700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07fc004850 0x7f0818158ea0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.714+0000 7f081d800700 1 --2- 172.21.15.12:0/145183406 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08180b8190 0x7f08180b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.714+0000 7f081d800700 1 -- 172.21.15.12:0/145183406 >> 172.21.15.12:0/145183406 conn(0x7f081808aff0 msgr2=0x7f08180cca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:51.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.714+0000 7f081d800700 1 -- 172.21.15.12:0/145183406 shutdown_connections 2024-03-29T10:12:51.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.714+0000 7f081d800700 1 -- 172.21.15.12:0/145183406 wait complete. 2024-03-29T10:12:51.717 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:12:51.732 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-29T10:12:49.983375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":395,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261,"ondisk_log_size":261,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":334832,"kb_used_data":12912,"kb_used_omap":12,"kb_used_meta":321779,"kb_avail":754639888,"statfs":{"total":773094113280,"available":772751245312,"internally_reserved":0,"allocated":13221888,"data_stored":6907453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":329502265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.701633"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.845782+0000","last_change":"2024-03-29T10:12:46.829400+0000","last_active":"2024-03-29T10:12:46.845782+0000","last_peered":"2024-03-29T10:12:46.845782+0000","last_clean":"2024-03-29T10:12:46.845782+0000","last_became_active":"2024-03-29T10:12:46.829140+0000","last_became_peered":"2024-03-29T10:12:46.829140+0000","last_unstale":"2024-03-29T10:12:46.845782+0000","last_undegraded":"2024-03-29T10:12:46.845782+0000","last_fullsized":"2024-03-29T10:12:46.845782+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:52.200329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.840436+0000","last_change":"2024-03-29T10:12:46.831622+0000","last_active":"2024-03-29T10:12:46.840436+0000","last_peered":"2024-03-29T10:12:46.840436+0000","last_clean":"2024-03-29T10:12:46.840436+0000","last_became_active":"2024-03-29T10:12:46.830998+0000","last_became_peered":"2024-03-29T10:12:46.830998+0000","last_unstale":"2024-03-29T10:12:46.840436+0000","last_undegraded":"2024-03-29T10:12:46.840436+0000","last_fullsized":"2024-03-29T10:12:46.840436+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:42:09.177945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.845734+0000","last_change":"2024-03-29T10:12:46.837275+0000","last_active":"2024-03-29T10:12:46.845734+0000","last_peered":"2024-03-29T10:12:46.845734+0000","last_clean":"2024-03-29T10:12:46.845734+0000","last_became_active":"2024-03-29T10:12:46.831882+0000","last_became_peered":"2024-03-29T10:12:46.831882+0000","last_unstale":"2024-03-29T10:12:46.845734+0000","last_undegraded":"2024-03-29T10:12:46.845734+0000","last_fullsized":"2024-03-29T10:12:46.845734+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:19:01.629952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.848747+0000","last_change":"2024-03-29T10:12:46.846790+0000","last_active":"2024-03-29T10:12:46.848747+0000","last_peered":"2024-03-29T10:12:46.848747+0000","last_clean":"2024-03-29T10:12:46.848747+0000","last_became_active":"2024-03-29T10:12:46.845478+0000","last_became_peered":"2024-03-29T10:12:46.845478+0000","last_unstale":"2024-03-29T10:12:46.848747+0000","last_undegraded":"2024-03-29T10:12:46.848747+0000","last_fullsized":"2024-03-29T10:12:46.848747+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:13.348427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.840973+0000","last_change":"2024-03-29T10:12:46.835467+0000","last_active":"2024-03-29T10:12:46.840973+0000","last_peered":"2024-03-29T10:12:46.840973+0000","last_clean":"2024-03-29T10:12:46.840973+0000","last_became_active":"2024-03-29T10:12:46.835033+0000","last_became_peered":"2024-03-29T10:12:46.835033+0000","last_unstale":"2024-03-29T10:12:46.840973+0000","last_undegraded":"2024-03-29T10:12:46.840973+0000","last_fullsized":"2024-03-29T10:12:46.840973+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:56.220192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.843599+0000","last_change":"2024-03-29T10:12:46.840582+0000","last_active":"2024-03-29T10:12:46.843599+0000","last_peered":"2024-03-29T10:12:46.843599+0000","last_clean":"2024-03-29T10:12:46.843599+0000","last_became_active":"2024-03-29T10:12:46.840297+0000","last_became_peered":"2024-03-29T10:12:46.840297+0000","last_unstale":"2024-03-29T10:12:46.843599+0000","last_undegraded":"2024-03-29T10:12:46.843599+0000","last_fullsized":"2024-03-29T10:12:46.843599+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:13.348427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.843642+0000","last_change":"2024-03-29T10:12:46.840753+0000","last_active":"2024-03-29T10:12:46.843642+0000","last_peered":"2024-03-29T10:12:46.843642+0000","last_clean":"2024-03-29T10:12:46.843642+0000","last_became_active":"2024-03-29T10:12:46.839371+0000","last_became_peered":"2024-03-29T10:12:46.839371+0000","last_unstale":"2024-03-29T10:12:46.843642+0000","last_undegraded":"2024-03-29T10:12:46.843642+0000","last_fullsized":"2024-03-29T10:12:46.843642+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:40:47.003642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.843833+0000","last_change":"2024-03-29T10:12:46.836293+0000","last_active":"2024-03-29T10:12:46.843833+0000","last_peered":"2024-03-29T10:12:46.843833+0000","last_clean":"2024-03-29T10:12:46.843833+0000","last_became_active":"2024-03-29T10:12:46.832168+0000","last_became_peered":"2024-03-29T10:12:46.832168+0000","last_unstale":"2024-03-29T10:12:46.843833+0000","last_undegraded":"2024-03-29T10:12:46.843833+0000","last_fullsized":"2024-03-29T10:12:46.843833+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:39:54.721927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.840056+0000","last_change":"2024-03-29T10:12:46.835070+0000","last_active":"2024-03-29T10:12:46.840056+0000","last_peered":"2024-03-29T10:12:46.840056+0000","last_clean":"2024-03-29T10:12:46.840056+0000","last_became_active":"2024-03-29T10:12:46.834789+0000","last_became_peered":"2024-03-29T10:12:46.834789+0000","last_unstale":"2024-03-29T10:12:46.840056+0000","last_undegraded":"2024-03-29T10:12:46.840056+0000","last_fullsized":"2024-03-29T10:12:46.840056+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:53.536977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.849522+0000","last_change":"2024-03-29T10:12:46.846327+0000","last_active":"2024-03-29T10:12:46.849522+0000","last_peered":"2024-03-29T10:12:46.849522+0000","last_clean":"2024-03-29T10:12:46.849522+0000","last_became_active":"2024-03-29T10:12:46.845951+0000","last_became_peered":"2024-03-29T10:12:46.845951+0000","last_unstale":"2024-03-29T10:12:46.849522+0000","last_undegraded":"2024-03-29T10:12:46.849522+0000","last_fullsized":"2024-03-29T10:12:46.849522+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:43:49.789402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.849242+0000","last_change":"2024-03-29T10:12:46.842898+0000","last_active":"2024-03-29T10:12:46.849242+0000","last_peered":"2024-03-29T10:12:46.849242+0000","last_clean":"2024-03-29T10:12:46.849242+0000","last_became_active":"2024-03-29T10:12:46.842128+0000","last_became_peered":"2024-03-29T10:12:46.842128+0000","last_unstale":"2024-03-29T10:12:46.849242+0000","last_undegraded":"2024-03-29T10:12:46.849242+0000","last_fullsized":"2024-03-29T10:12:46.849242+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:45.331011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.848991+0000","last_change":"2024-03-29T10:12:46.846480+0000","last_active":"2024-03-29T10:12:46.848991+0000","last_peered":"2024-03-29T10:12:46.848991+0000","last_clean":"2024-03-29T10:12:46.848991+0000","last_became_active":"2024-03-29T10:12:46.846162+0000","last_became_peered":"2024-03-29T10:12:46.846162+0000","last_unstale":"2024-03-29T10:12:46.848991+0000","last_undegraded":"2024-03-29T10:12:46.848991+0000","last_fullsized":"2024-03-29T10:12:46.848991+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:56.332969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.841766+0000","last_change":"2024-03-29T10:12:46.835069+0000","last_active":"2024-03-29T10:12:46.841766+0000","last_peered":"2024-03-29T10:12:46.841766+0000","last_clean":"2024-03-29T10:12:46.841766+0000","last_became_active":"2024-03-29T10:12:46.834813+0000","last_became_peered":"2024-03-29T10:12:46.834813+0000","last_unstale":"2024-03-29T10:12:46.841766+0000","last_undegraded":"2024-03-29T10:12:46.841766+0000","last_fullsized":"2024-03-29T10:12:46.841766+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:43.403428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.847528+0000","last_change":"2024-03-29T10:12:46.838992+0000","last_active":"2024-03-29T10:12:46.847528+0000","last_peered":"2024-03-29T10:12:46.847528+0000","last_clean":"2024-03-29T10:12:46.847528+0000","last_became_active":"2024-03-29T10:12:46.838712+0000","last_became_peered":"2024-03-29T10:12:46.838712+0000","last_unstale":"2024-03-29T10:12:46.847528+0000","last_undegraded":"2024-03-29T10:12:46.847528+0000","last_fullsized":"2024-03-29T10:12:46.847528+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:20:09.840649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.848750+0000","last_change":"2024-03-29T10:12:46.846667+0000","last_active":"2024-03-29T10:12:46.848750+0000","last_peered":"2024-03-29T10:12:46.848750+0000","last_clean":"2024-03-29T10:12:46.848750+0000","last_became_active":"2024-03-29T10:12:46.846362+0000","last_became_peered":"2024-03-29T10:12:46.846362+0000","last_unstale":"2024-03-29T10:12:46.848750+0000","last_undegraded":"2024-03-29T10:12:46.848750+0000","last_fullsized":"2024-03-29T10:12:46.848750+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:41.550218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.843341+0000","last_change":"2024-03-29T10:12:46.839581+0000","last_active":"2024-03-29T10:12:46.843341+0000","last_peered":"2024-03-29T10:12:46.843341+0000","last_clean":"2024-03-29T10:12:46.843341+0000","last_became_active":"2024-03-29T10:12:46.837616+0000","last_became_peered":"2024-03-29T10:12:46.837616+0000","last_unstale":"2024-03-29T10:12:46.843341+0000","last_undegraded":"2024-03-29T10:12:46.843341+0000","last_fullsized":"2024-03-29T10:12:46.843341+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:41.550218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.839157+0000","last_change":"2024-03-29T10:12:46.835776+0000","last_active":"2024-03-29T10:12:46.839157+0000","last_peered":"2024-03-29T10:12:46.839157+0000","last_clean":"2024-03-29T10:12:46.839157+0000","last_became_active":"2024-03-29T10:12:46.835460+0000","last_became_peered":"2024-03-29T10:12:46.835460+0000","last_unstale":"2024-03-29T10:12:46.839157+0000","last_undegraded":"2024-03-29T10:12:46.839157+0000","last_fullsized":"2024-03-29T10:12:46.839157+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:42:38.641476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.853821+0000","last_change":"2024-03-29T10:12:46.852405+0000","last_active":"2024-03-29T10:12:46.853821+0000","last_peered":"2024-03-29T10:12:46.853821+0000","last_clean":"2024-03-29T10:12:46.853821+0000","last_became_active":"2024-03-29T10:12:46.851997+0000","last_became_peered":"2024-03-29T10:12:46.851997+0000","last_unstale":"2024-03-29T10:12:46.853821+0000","last_undegraded":"2024-03-29T10:12:46.853821+0000","last_fullsized":"2024-03-29T10:12:46.853821+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:53.536977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.848658+0000","last_change":"2024-03-29T10:12:46.840469+0000","last_active":"2024-03-29T10:12:46.848658+0000","last_peered":"2024-03-29T10:12:46.848658+0000","last_clean":"2024-03-29T10:12:46.848658+0000","last_became_active":"2024-03-29T10:12:46.840212+0000","last_became_peered":"2024-03-29T10:12:46.840212+0000","last_unstale":"2024-03-29T10:12:46.848658+0000","last_undegraded":"2024-03-29T10:12:46.848658+0000","last_fullsized":"2024-03-29T10:12:46.848658+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:38.075899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.847100+0000","last_change":"2024-03-29T10:12:46.843181+0000","last_active":"2024-03-29T10:12:46.847100+0000","last_peered":"2024-03-29T10:12:46.847100+0000","last_clean":"2024-03-29T10:12:46.847100+0000","last_became_active":"2024-03-29T10:12:46.842878+0000","last_became_peered":"2024-03-29T10:12:46.842878+0000","last_unstale":"2024-03-29T10:12:46.847100+0000","last_undegraded":"2024-03-29T10:12:46.847100+0000","last_fullsized":"2024-03-29T10:12:46.847100+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:58:37.690997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.845248+0000","last_change":"2024-03-29T10:12:46.835563+0000","last_active":"2024-03-29T10:12:46.845248+0000","last_peered":"2024-03-29T10:12:46.845248+0000","last_clean":"2024-03-29T10:12:46.845248+0000","last_became_active":"2024-03-29T10:12:46.834725+0000","last_became_peered":"2024-03-29T10:12:46.834725+0000","last_unstale":"2024-03-29T10:12:46.845248+0000","last_undegraded":"2024-03-29T10:12:46.845248+0000","last_fullsized":"2024-03-29T10:12:46.845248+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:50:42.501111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.831654+0000","last_change":"2024-03-29T10:12:46.828788+0000","last_active":"2024-03-29T10:12:46.831654+0000","last_peered":"2024-03-29T10:12:46.831654+0000","last_clean":"2024-03-29T10:12:46.831654+0000","last_became_active":"2024-03-29T10:12:46.828528+0000","last_became_peered":"2024-03-29T10:12:46.828528+0000","last_unstale":"2024-03-29T10:12:46.831654+0000","last_undegraded":"2024-03-29T10:12:46.831654+0000","last_fullsized":"2024-03-29T10:12:46.831654+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:31.163681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.829277+0000","last_change":"2024-03-29T10:12:46.828500+0000","last_active":"2024-03-29T10:12:46.829277+0000","last_peered":"2024-03-29T10:12:46.829277+0000","last_clean":"2024-03-29T10:12:46.829277+0000","last_became_active":"2024-03-29T10:12:46.828140+0000","last_became_peered":"2024-03-29T10:12:46.828140+0000","last_unstale":"2024-03-29T10:12:46.829277+0000","last_undegraded":"2024-03-29T10:12:46.829277+0000","last_fullsized":"2024-03-29T10:12:46.829277+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:42:34.437921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.855270+0000","last_change":"2024-03-29T10:12:46.851135+0000","last_active":"2024-03-29T10:12:46.855270+0000","last_peered":"2024-03-29T10:12:46.855270+0000","last_clean":"2024-03-29T10:12:46.855270+0000","last_became_active":"2024-03-29T10:12:46.850882+0000","last_became_peered":"2024-03-29T10:12:46.850882+0000","last_unstale":"2024-03-29T10:12:46.855270+0000","last_undegraded":"2024-03-29T10:12:46.855270+0000","last_fullsized":"2024-03-29T10:12:46.855270+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:54.751799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.837808+0000","last_change":"2024-03-29T10:12:46.831971+0000","last_active":"2024-03-29T10:12:46.837808+0000","last_peered":"2024-03-29T10:12:46.837808+0000","last_clean":"2024-03-29T10:12:46.837808+0000","last_became_active":"2024-03-29T10:12:46.831695+0000","last_became_peered":"2024-03-29T10:12:46.831695+0000","last_unstale":"2024-03-29T10:12:46.837808+0000","last_undegraded":"2024-03-29T10:12:46.837808+0000","last_fullsized":"2024-03-29T10:12:46.837808+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:58:37.690997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.851887+0000","last_change":"2024-03-29T10:12:46.842057+0000","last_active":"2024-03-29T10:12:46.851887+0000","last_peered":"2024-03-29T10:12:46.851887+0000","last_clean":"2024-03-29T10:12:46.851887+0000","last_became_active":"2024-03-29T10:12:46.841783+0000","last_became_peered":"2024-03-29T10:12:46.841783+0000","last_unstale":"2024-03-29T10:12:46.851887+0000","last_undegraded":"2024-03-29T10:12:46.851887+0000","last_fullsized":"2024-03-29T10:12:46.851887+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:07:59.900469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.849380+0000","last_change":"2024-03-29T10:12:46.845468+0000","last_active":"2024-03-29T10:12:46.849380+0000","last_peered":"2024-03-29T10:12:46.849380+0000","last_clean":"2024-03-29T10:12:46.849380+0000","last_became_active":"2024-03-29T10:12:46.845175+0000","last_became_peered":"2024-03-29T10:12:46.845175+0000","last_unstale":"2024-03-29T10:12:46.849380+0000","last_undegraded":"2024-03-29T10:12:46.849380+0000","last_fullsized":"2024-03-29T10:12:46.849380+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:56.220192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.842206+0000","last_change":"2024-03-29T10:12:46.839048+0000","last_active":"2024-03-29T10:12:46.842206+0000","last_peered":"2024-03-29T10:12:46.842206+0000","last_clean":"2024-03-29T10:12:46.842206+0000","last_became_active":"2024-03-29T10:12:46.838084+0000","last_became_peered":"2024-03-29T10:12:46.838084+0000","last_unstale":"2024-03-29T10:12:46.842206+0000","last_undegraded":"2024-03-29T10:12:46.842206+0000","last_fullsized":"2024-03-29T10:12:46.842206+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:43.403428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.850629+0000","last_change":"2024-03-29T10:12:46.839770+0000","last_active":"2024-03-29T10:12:46.850629+0000","last_peered":"2024-03-29T10:12:46.850629+0000","last_clean":"2024-03-29T10:12:46.850629+0000","last_became_active":"2024-03-29T10:12:46.839511+0000","last_became_peered":"2024-03-29T10:12:46.839511+0000","last_unstale":"2024-03-29T10:12:46.850629+0000","last_undegraded":"2024-03-29T10:12:46.850629+0000","last_fullsized":"2024-03-29T10:12:46.850629+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:04.158919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.847547+0000","last_change":"2024-03-29T10:12:46.841318+0000","last_active":"2024-03-29T10:12:46.847547+0000","last_peered":"2024-03-29T10:12:46.847547+0000","last_clean":"2024-03-29T10:12:46.847547+0000","last_became_active":"2024-03-29T10:12:46.841085+0000","last_became_peered":"2024-03-29T10:12:46.841085+0000","last_unstale":"2024-03-29T10:12:46.847547+0000","last_undegraded":"2024-03-29T10:12:46.847547+0000","last_fullsized":"2024-03-29T10:12:46.847547+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:05:55.909595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.846785+0000","last_change":"2024-03-29T10:12:46.842050+0000","last_active":"2024-03-29T10:12:46.846785+0000","last_peered":"2024-03-29T10:12:46.846785+0000","last_clean":"2024-03-29T10:12:46.846785+0000","last_became_active":"2024-03-29T10:12:46.835709+0000","last_became_peered":"2024-03-29T10:12:46.835709+0000","last_unstale":"2024-03-29T10:12:46.846785+0000","last_undegraded":"2024-03-29T10:12:46.846785+0000","last_fullsized":"2024-03-29T10:12:46.846785+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:45.331011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.848557+0000","last_change":"2024-03-29T10:12:46.834325+0000","last_active":"2024-03-29T10:12:46.848557+0000","last_peered":"2024-03-29T10:12:46.848557+0000","last_clean":"2024-03-29T10:12:46.848557+0000","last_became_active":"2024-03-29T10:12:46.833484+0000","last_became_peered":"2024-03-29T10:12:46.833484+0000","last_unstale":"2024-03-29T10:12:46.848557+0000","last_undegraded":"2024-03-29T10:12:46.848557+0000","last_fullsized":"2024-03-29T10:12:46.848557+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:30:12.205591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.852515+0000","last_change":"2024-03-29T10:12:46.845641+0000","last_active":"2024-03-29T10:12:46.852515+0000","last_peered":"2024-03-29T10:12:46.852515+0000","last_clean":"2024-03-29T10:12:46.852515+0000","last_became_active":"2024-03-29T10:12:46.844642+0000","last_became_peered":"2024-03-29T10:12:46.844642+0000","last_unstale":"2024-03-29T10:12:46.852515+0000","last_undegraded":"2024-03-29T10:12:46.852515+0000","last_fullsized":"2024-03-29T10:12:46.852515+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:29:44.813491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"123'4","reported_seq":14,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.845956+0000","last_change":"2024-03-29T10:12:46.312221+0000","last_active":"2024-03-29T10:12:46.845956+0000","last_peered":"2024-03-29T10:12:46.845956+0000","last_clean":"2024-03-29T10:12:46.845956+0000","last_became_active":"2024-03-29T10:12:46.311948+0000","last_became_peered":"2024-03-29T10:12:46.311948+0000","last_unstale":"2024-03-29T10:12:46.845956+0000","last_undegraded":"2024-03-29T10:12:46.845956+0000","last_fullsized":"2024-03-29T10:12:46.845956+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:12.339045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.847271+0000","last_change":"2024-03-29T10:12:46.842054+0000","last_active":"2024-03-29T10:12:46.847271+0000","last_peered":"2024-03-29T10:12:46.847271+0000","last_clean":"2024-03-29T10:12:46.847271+0000","last_became_active":"2024-03-29T10:12:46.841790+0000","last_became_peered":"2024-03-29T10:12:46.841790+0000","last_unstale":"2024-03-29T10:12:46.847271+0000","last_undegraded":"2024-03-29T10:12:46.847271+0000","last_fullsized":"2024-03-29T10:12:46.847271+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:56.220192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"123'1","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.850725+0000","last_change":"2024-03-29T10:12:46.317237+0000","last_active":"2024-03-29T10:12:46.850725+0000","last_peered":"2024-03-29T10:12:46.850725+0000","last_clean":"2024-03-29T10:12:46.850725+0000","last_became_active":"2024-03-29T10:12:46.316931+0000","last_became_peered":"2024-03-29T10:12:46.316931+0000","last_unstale":"2024-03-29T10:12:46.850725+0000","last_undegraded":"2024-03-29T10:12:46.850725+0000","last_fullsized":"2024-03-29T10:12:46.850725+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:07.770121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.839120+0000","last_change":"2024-03-29T10:12:46.826174+0000","last_active":"2024-03-29T10:12:46.839120+0000","last_peered":"2024-03-29T10:12:46.839120+0000","last_clean":"2024-03-29T10:12:46.839120+0000","last_became_active":"2024-03-29T10:12:46.825829+0000","last_became_peered":"2024-03-29T10:12:46.825829+0000","last_unstale":"2024-03-29T10:12:46.839120+0000","last_undegraded":"2024-03-29T10:12:46.839120+0000","last_fullsized":"2024-03-29T10:12:46.839120+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:29:38.657726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"123'2","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.850526+0000","last_change":"2024-03-29T10:12:46.319156+0000","last_active":"2024-03-29T10:12:46.850526+0000","last_peered":"2024-03-29T10:12:46.850526+0000","last_clean":"2024-03-29T10:12:46.850526+0000","last_became_active":"2024-03-29T10:12:46.318764+0000","last_became_peered":"2024-03-29T10:12:46.318764+0000","last_unstale":"2024-03-29T10:12:46.850526+0000","last_undegraded":"2024-03-29T10:12:46.850526+0000","last_fullsized":"2024-03-29T10:12:46.850526+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:40:45.994260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.841811+0000","last_change":"2024-03-29T10:12:46.836926+0000","last_active":"2024-03-29T10:12:46.841811+0000","last_peered":"2024-03-29T10:12:46.841811+0000","last_clean":"2024-03-29T10:12:46.841811+0000","last_became_active":"2024-03-29T10:12:46.832584+0000","last_became_peered":"2024-03-29T10:12:46.832584+0000","last_unstale":"2024-03-29T10:12:46.841811+0000","last_undegraded":"2024-03-29T10:12:46.841811+0000","last_fullsized":"2024-03-29T10:12:46.841811+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:13.348427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"123'2","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.851227+0000","last_change":"2024-03-29T10:12:46.312199+0000","last_active":"2024-03-29T10:12:46.851227+0000","last_peered":"2024-03-29T10:12:46.851227+0000","last_clean":"2024-03-29T10:12:46.851227+0000","last_became_active":"2024-03-29T10:12:46.311547+0000","last_became_peered":"2024-03-29T10:12:46.311547+0000","last_unstale":"2024-03-29T10:12:46.851227+0000","last_undegraded":"2024-03-29T10:12:46.851227+0000","last_fullsized":"2024-03-29T10:12:46.851227+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:55.323587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.836417+0000","last_change":"2024-03-29T10:12:46.831278+0000","last_active":"2024-03-29T10:12:46.836417+0000","last_peered":"2024-03-29T10:12:46.836417+0000","last_clean":"2024-03-29T10:12:46.836417+0000","last_became_active":"2024-03-29T10:12:46.830914+0000","last_became_peered":"2024-03-29T10:12:46.830914+0000","last_unstale":"2024-03-29T10:12:46.836417+0000","last_undegraded":"2024-03-29T10:12:46.836417+0000","last_fullsized":"2024-03-29T10:12:46.836417+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:29:44.813491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"123'1","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.848881+0000","last_change":"2024-03-29T10:12:46.301452+0000","last_active":"2024-03-29T10:12:46.848881+0000","last_peered":"2024-03-29T10:12:46.848881+0000","last_clean":"2024-03-29T10:12:46.848881+0000","last_became_active":"2024-03-29T10:12:46.300949+0000","last_became_peered":"2024-03-29T10:12:46.300949+0000","last_unstale":"2024-03-29T10:12:46.848881+0000","last_undegraded":"2024-03-29T10:12:46.848881+0000","last_fullsized":"2024-03-29T10:12:46.848881+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:46:54.186603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.849149+0000","last_change":"2024-03-29T10:12:46.841913+0000","last_active":"2024-03-29T10:12:46.849149+0000","last_peered":"2024-03-29T10:12:46.849149+0000","last_clean":"2024-03-29T10:12:46.849149+0000","last_became_active":"2024-03-29T10:12:46.841068+0000","last_became_peered":"2024-03-29T10:12:46.841068+0000","last_unstale":"2024-03-29T10:12:46.849149+0000","last_undegraded":"2024-03-29T10:12:46.849149+0000","last_fullsized":"2024-03-29T10:12:46.849149+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:50.693333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"123'1","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.846137+0000","last_change":"2024-03-29T10:12:46.317421+0000","last_active":"2024-03-29T10:12:46.846137+0000","last_peered":"2024-03-29T10:12:46.846137+0000","last_clean":"2024-03-29T10:12:46.846137+0000","last_became_active":"2024-03-29T10:12:46.314110+0000","last_became_peered":"2024-03-29T10:12:46.314110+0000","last_unstale":"2024-03-29T10:12:46.846137+0000","last_undegraded":"2024-03-29T10:12:46.846137+0000","last_fullsized":"2024-03-29T10:12:46.846137+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:16:08.320929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.850004+0000","last_change":"2024-03-29T10:12:46.849430+0000","last_active":"2024-03-29T10:12:46.850004+0000","last_peered":"2024-03-29T10:12:46.850004+0000","last_clean":"2024-03-29T10:12:46.850004+0000","last_became_active":"2024-03-29T10:12:46.849158+0000","last_became_peered":"2024-03-29T10:12:46.849158+0000","last_unstale":"2024-03-29T10:12:46.850004+0000","last_undegraded":"2024-03-29T10:12:46.850004+0000","last_fullsized":"2024-03-29T10:12:46.850004+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:54.751799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.820074+0000","last_change":"2024-03-29T10:12:46.320446+0000","last_active":"2024-03-29T10:12:46.820074+0000","last_peered":"2024-03-29T10:12:46.820074+0000","last_clean":"2024-03-29T10:12:46.820074+0000","last_became_active":"2024-03-29T10:12:46.320149+0000","last_became_peered":"2024-03-29T10:12:46.320149+0000","last_unstale":"2024-03-29T10:12:46.820074+0000","last_undegraded":"2024-03-29T10:12:46.820074+0000","last_fullsized":"2024-03-29T10:12:46.820074+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:18.516455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.833802+0000","last_change":"2024-03-29T10:12:46.827124+0000","last_active":"2024-03-29T10:12:46.833802+0000","last_peered":"2024-03-29T10:12:46.833802+0000","last_clean":"2024-03-29T10:12:46.833802+0000","last_became_active":"2024-03-29T10:12:46.826802+0000","last_became_peered":"2024-03-29T10:12:46.826802+0000","last_unstale":"2024-03-29T10:12:46.833802+0000","last_undegraded":"2024-03-29T10:12:46.833802+0000","last_fullsized":"2024-03-29T10:12:46.833802+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:16:09.330311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.815169+0000","last_change":"2024-03-29T10:12:46.312469+0000","last_active":"2024-03-29T10:12:46.815169+0000","last_peered":"2024-03-29T10:12:46.815169+0000","last_clean":"2024-03-29T10:12:46.815169+0000","last_became_active":"2024-03-29T10:12:46.312178+0000","last_became_peered":"2024-03-29T10:12:46.312178+0000","last_unstale":"2024-03-29T10:12:46.815169+0000","last_undegraded":"2024-03-29T10:12:46.815169+0000","last_fullsized":"2024-03-29T10:12:46.815169+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:46:28.171211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.850408+0000","last_change":"2024-03-29T10:12:46.840355+0000","last_active":"2024-03-29T10:12:46.850408+0000","last_peered":"2024-03-29T10:12:46.850408+0000","last_clean":"2024-03-29T10:12:46.850408+0000","last_became_active":"2024-03-29T10:12:46.840098+0000","last_became_peered":"2024-03-29T10:12:46.840098+0000","last_unstale":"2024-03-29T10:12:46.850408+0000","last_undegraded":"2024-03-29T10:12:46.850408+0000","last_fullsized":"2024-03-29T10:12:46.850408+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:56:37.402871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.818748+0000","last_change":"2024-03-29T10:12:46.302880+0000","last_active":"2024-03-29T10:12:46.818748+0000","last_peered":"2024-03-29T10:12:46.818748+0000","last_clean":"2024-03-29T10:12:46.818748+0000","last_became_active":"2024-03-29T10:12:46.302582+0000","last_became_peered":"2024-03-29T10:12:46.302582+0000","last_unstale":"2024-03-29T10:12:46.818748+0000","last_undegraded":"2024-03-29T10:12:46.818748+0000","last_fullsized":"2024-03-29T10:12:46.818748+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:34:35.424203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.833875+0000","last_change":"2024-03-29T10:12:46.827498+0000","last_active":"2024-03-29T10:12:46.833875+0000","last_peered":"2024-03-29T10:12:46.833875+0000","last_clean":"2024-03-29T10:12:46.833875+0000","last_became_active":"2024-03-29T10:12:46.826934+0000","last_became_peered":"2024-03-29T10:12:46.826934+0000","last_unstale":"2024-03-29T10:12:46.833875+0000","last_undegraded":"2024-03-29T10:12:46.833875+0000","last_fullsized":"2024-03-29T10:12:46.833875+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:43:49.789402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.823117+0000","last_change":"2024-03-29T10:12:46.312627+0000","last_active":"2024-03-29T10:12:46.823117+0000","last_peered":"2024-03-29T10:12:46.823117+0000","last_clean":"2024-03-29T10:12:46.823117+0000","last_became_active":"2024-03-29T10:12:46.312376+0000","last_became_peered":"2024-03-29T10:12:46.312376+0000","last_unstale":"2024-03-29T10:12:46.823117+0000","last_undegraded":"2024-03-29T10:12:46.823117+0000","last_fullsized":"2024-03-29T10:12:46.823117+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:34:30.646213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.839982+0000","last_change":"2024-03-29T10:12:46.833312+0000","last_active":"2024-03-29T10:12:46.839982+0000","last_peered":"2024-03-29T10:12:46.839982+0000","last_clean":"2024-03-29T10:12:46.839982+0000","last_became_active":"2024-03-29T10:12:46.832917+0000","last_became_peered":"2024-03-29T10:12:46.832917+0000","last_unstale":"2024-03-29T10:12:46.839982+0000","last_undegraded":"2024-03-29T10:12:46.839982+0000","last_fullsized":"2024-03-29T10:12:46.839982+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:53.536977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"123'3","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.849326+0000","last_change":"2024-03-29T10:12:46.314591+0000","last_active":"2024-03-29T10:12:46.849326+0000","last_peered":"2024-03-29T10:12:46.849326+0000","last_clean":"2024-03-29T10:12:46.849326+0000","last_became_active":"2024-03-29T10:12:46.314184+0000","last_became_peered":"2024-03-29T10:12:46.314184+0000","last_unstale":"2024-03-29T10:12:46.849326+0000","last_undegraded":"2024-03-29T10:12:46.849326+0000","last_fullsized":"2024-03-29T10:12:46.849326+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:15.808972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.849201+0000","last_change":"2024-03-29T10:12:46.842734+0000","last_active":"2024-03-29T10:12:46.849201+0000","last_peered":"2024-03-29T10:12:46.849201+0000","last_clean":"2024-03-29T10:12:46.849201+0000","last_became_active":"2024-03-29T10:12:46.841788+0000","last_became_peered":"2024-03-29T10:12:46.841788+0000","last_unstale":"2024-03-29T10:12:46.849201+0000","last_undegraded":"2024-03-29T10:12:46.849201+0000","last_fullsized":"2024-03-29T10:12:46.849201+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:06.962212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"123'6","reported_seq":16,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.852358+0000","last_change":"2024-03-29T10:12:46.311128+0000","last_active":"2024-03-29T10:12:46.852358+0000","last_peered":"2024-03-29T10:12:46.852358+0000","last_clean":"2024-03-29T10:12:46.852358+0000","last_became_active":"2024-03-29T10:12:46.310825+0000","last_became_peered":"2024-03-29T10:12:46.310825+0000","last_unstale":"2024-03-29T10:12:46.852358+0000","last_undegraded":"2024-03-29T10:12:46.852358+0000","last_fullsized":"2024-03-29T10:12:46.852358+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:10.671851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.841697+0000","last_change":"2024-03-29T10:12:46.837420+0000","last_active":"2024-03-29T10:12:46.841697+0000","last_peered":"2024-03-29T10:12:46.841697+0000","last_clean":"2024-03-29T10:12:46.841697+0000","last_became_active":"2024-03-29T10:12:46.837167+0000","last_became_peered":"2024-03-29T10:12:46.837167+0000","last_unstale":"2024-03-29T10:12:46.841697+0000","last_undegraded":"2024-03-29T10:12:46.841697+0000","last_fullsized":"2024-03-29T10:12:46.841697+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:58:37.690997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"123'2","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.852573+0000","last_change":"2024-03-29T10:12:46.305485+0000","last_active":"2024-03-29T10:12:46.852573+0000","last_peered":"2024-03-29T10:12:46.852573+0000","last_clean":"2024-03-29T10:12:46.852573+0000","last_became_active":"2024-03-29T10:12:46.305135+0000","last_became_peered":"2024-03-29T10:12:46.305135+0000","last_unstale":"2024-03-29T10:12:46.852573+0000","last_undegraded":"2024-03-29T10:12:46.852573+0000","last_fullsized":"2024-03-29T10:12:46.852573+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:51:14.288869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.845897+0000","last_change":"2024-03-29T10:12:46.834105+0000","last_active":"2024-03-29T10:12:46.845897+0000","last_peered":"2024-03-29T10:12:46.845897+0000","last_clean":"2024-03-29T10:12:46.845897+0000","last_became_active":"2024-03-29T10:12:46.833832+0000","last_became_peered":"2024-03-29T10:12:46.833832+0000","last_unstale":"2024-03-29T10:12:46.845897+0000","last_undegraded":"2024-03-29T10:12:46.845897+0000","last_fullsized":"2024-03-29T10:12:46.845897+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:13.633043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.826167+0000","last_change":"2024-03-29T10:12:46.309019+0000","last_active":"2024-03-29T10:12:46.826167+0000","last_peered":"2024-03-29T10:12:46.826167+0000","last_clean":"2024-03-29T10:12:46.826167+0000","last_became_active":"2024-03-29T10:12:46.308043+0000","last_became_peered":"2024-03-29T10:12:46.308043+0000","last_unstale":"2024-03-29T10:12:46.826167+0000","last_undegraded":"2024-03-29T10:12:46.826167+0000","last_fullsized":"2024-03-29T10:12:46.826167+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:54.107733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.840895+0000","last_change":"2024-03-29T10:12:46.836008+0000","last_active":"2024-03-29T10:12:46.840895+0000","last_peered":"2024-03-29T10:12:46.840895+0000","last_clean":"2024-03-29T10:12:46.840895+0000","last_became_active":"2024-03-29T10:12:46.835699+0000","last_became_peered":"2024-03-29T10:12:46.835699+0000","last_unstale":"2024-03-29T10:12:46.840895+0000","last_undegraded":"2024-03-29T10:12:46.840895+0000","last_fullsized":"2024-03-29T10:12:46.840895+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:16.818354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.818156+0000","last_change":"2024-03-29T10:12:46.304474+0000","last_active":"2024-03-29T10:12:46.818156+0000","last_peered":"2024-03-29T10:12:46.818156+0000","last_clean":"2024-03-29T10:12:46.818156+0000","last_became_active":"2024-03-29T10:12:46.304207+0000","last_became_peered":"2024-03-29T10:12:46.304207+0000","last_unstale":"2024-03-29T10:12:46.818156+0000","last_undegraded":"2024-03-29T10:12:46.818156+0000","last_fullsized":"2024-03-29T10:12:46.818156+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:18.516455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.848986+0000","last_change":"2024-03-29T10:12:46.840792+0000","last_active":"2024-03-29T10:12:46.848986+0000","last_peered":"2024-03-29T10:12:46.848986+0000","last_clean":"2024-03-29T10:12:46.848986+0000","last_became_active":"2024-03-29T10:12:46.840497+0000","last_became_peered":"2024-03-29T10:12:46.840497+0000","last_unstale":"2024-03-29T10:12:46.848986+0000","last_undegraded":"2024-03-29T10:12:46.848986+0000","last_fullsized":"2024-03-29T10:12:46.848986+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:51.994336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"123'8","reported_seq":18,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.855432+0000","last_change":"2024-03-29T10:12:46.313806+0000","last_active":"2024-03-29T10:12:46.855432+0000","last_peered":"2024-03-29T10:12:46.855432+0000","last_clean":"2024-03-29T10:12:46.855432+0000","last_became_active":"2024-03-29T10:12:46.313540+0000","last_became_peered":"2024-03-29T10:12:46.313540+0000","last_unstale":"2024-03-29T10:12:46.855432+0000","last_undegraded":"2024-03-29T10:12:46.855432+0000","last_fullsized":"2024-03-29T10:12:46.855432+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:10:18.737044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.832425+0000","last_change":"2024-03-29T10:12:46.825422+0000","last_active":"2024-03-29T10:12:46.832425+0000","last_peered":"2024-03-29T10:12:46.832425+0000","last_clean":"2024-03-29T10:12:46.832425+0000","last_became_active":"2024-03-29T10:12:46.825098+0000","last_became_peered":"2024-03-29T10:12:46.825098+0000","last_unstale":"2024-03-29T10:12:46.832425+0000","last_undegraded":"2024-03-29T10:12:46.832425+0000","last_fullsized":"2024-03-29T10:12:46.832425+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:56.332969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"123'2","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.851382+0000","last_change":"2024-03-29T10:12:46.307543+0000","last_active":"2024-03-29T10:12:46.851382+0000","last_peered":"2024-03-29T10:12:46.851382+0000","last_clean":"2024-03-29T10:12:46.851382+0000","last_became_active":"2024-03-29T10:12:46.307269+0000","last_became_peered":"2024-03-29T10:12:46.307269+0000","last_unstale":"2024-03-29T10:12:46.851382+0000","last_undegraded":"2024-03-29T10:12:46.851382+0000","last_fullsized":"2024-03-29T10:12:46.851382+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:34:20.459712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.839148+0000","last_change":"2024-03-29T10:12:46.834470+0000","last_active":"2024-03-29T10:12:46.839148+0000","last_peered":"2024-03-29T10:12:46.839148+0000","last_clean":"2024-03-29T10:12:46.839148+0000","last_became_active":"2024-03-29T10:12:46.834169+0000","last_became_peered":"2024-03-29T10:12:46.834169+0000","last_unstale":"2024-03-29T10:12:46.839148+0000","last_undegraded":"2024-03-29T10:12:46.839148+0000","last_fullsized":"2024-03-29T10:12:46.839148+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:51.994336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"123'1","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.841619+0000","last_change":"2024-03-29T10:12:46.304283+0000","last_active":"2024-03-29T10:12:46.841619+0000","last_peered":"2024-03-29T10:12:46.841619+0000","last_clean":"2024-03-29T10:12:46.841619+0000","last_became_active":"2024-03-29T10:12:46.302450+0000","last_became_peered":"2024-03-29T10:12:46.302450+0000","last_unstale":"2024-03-29T10:12:46.841619+0000","last_undegraded":"2024-03-29T10:12:46.841619+0000","last_fullsized":"2024-03-29T10:12:46.841619+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:12:49.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.849258+0000","last_change":"2024-03-29T10:12:46.843073+0000","last_active":"2024-03-29T10:12:46.849258+0000","last_peered":"2024-03-29T10:12:46.849258+0000","last_clean":"2024-03-29T10:12:46.849258+0000","last_became_active":"2024-03-29T10:12:46.842479+0000","last_became_peered":"2024-03-29T10:12:46.842479+0000","last_unstale":"2024-03-29T10:12:46.849258+0000","last_undegraded":"2024-03-29T10:12:46.849258+0000","last_fullsized":"2024-03-29T10:12:46.849258+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:43.403428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.824510+0000","last_change":"2024-03-29T10:12:46.312388+0000","last_active":"2024-03-29T10:12:46.824510+0000","last_peered":"2024-03-29T10:12:46.824510+0000","last_clean":"2024-03-29T10:12:46.824510+0000","last_became_active":"2024-03-29T10:12:46.311924+0000","last_became_peered":"2024-03-29T10:12:46.311924+0000","last_unstale":"2024-03-29T10:12:46.824510+0000","last_undegraded":"2024-03-29T10:12:46.824510+0000","last_fullsized":"2024-03-29T10:12:46.824510+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:42.861576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.839083+0000","last_change":"2024-03-29T10:12:46.828829+0000","last_active":"2024-03-29T10:12:46.839083+0000","last_peered":"2024-03-29T10:12:46.839083+0000","last_clean":"2024-03-29T10:12:46.839083+0000","last_became_active":"2024-03-29T10:12:46.828570+0000","last_became_peered":"2024-03-29T10:12:46.828570+0000","last_unstale":"2024-03-29T10:12:46.839083+0000","last_undegraded":"2024-03-29T10:12:46.839083+0000","last_fullsized":"2024-03-29T10:12:46.839083+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:27:46.061296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"123'2","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.850683+0000","last_change":"2024-03-29T10:12:46.320441+0000","last_active":"2024-03-29T10:12:46.850683+0000","last_peered":"2024-03-29T10:12:46.850683+0000","last_clean":"2024-03-29T10:12:46.850683+0000","last_became_active":"2024-03-29T10:12:46.319864+0000","last_became_peered":"2024-03-29T10:12:46.319864+0000","last_unstale":"2024-03-29T10:12:46.850683+0000","last_undegraded":"2024-03-29T10:12:46.850683+0000","last_fullsized":"2024-03-29T10:12:46.850683+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:15.808972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.845225+0000","last_change":"2024-03-29T10:12:46.834833+0000","last_active":"2024-03-29T10:12:46.845225+0000","last_peered":"2024-03-29T10:12:46.845225+0000","last_clean":"2024-03-29T10:12:46.845225+0000","last_became_active":"2024-03-29T10:12:46.834327+0000","last_became_peered":"2024-03-29T10:12:46.834327+0000","last_unstale":"2024-03-29T10:12:46.845225+0000","last_undegraded":"2024-03-29T10:12:46.845225+0000","last_fullsized":"2024-03-29T10:12:46.845225+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:18:02.845983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.825106+0000","last_change":"2024-03-29T10:12:46.307976+0000","last_active":"2024-03-29T10:12:46.825106+0000","last_peered":"2024-03-29T10:12:46.825106+0000","last_clean":"2024-03-29T10:12:46.825106+0000","last_became_active":"2024-03-29T10:12:46.307713+0000","last_became_peered":"2024-03-29T10:12:46.307713+0000","last_unstale":"2024-03-29T10:12:46.825106+0000","last_undegraded":"2024-03-29T10:12:46.825106+0000","last_fullsized":"2024-03-29T10:12:46.825106+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:28:50.912139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.849484+0000","last_change":"2024-03-29T10:12:46.842521+0000","last_active":"2024-03-29T10:12:46.849484+0000","last_peered":"2024-03-29T10:12:46.849484+0000","last_clean":"2024-03-29T10:12:46.849484+0000","last_became_active":"2024-03-29T10:12:46.842222+0000","last_became_peered":"2024-03-29T10:12:46.842222+0000","last_unstale":"2024-03-29T10:12:46.849484+0000","last_undegraded":"2024-03-29T10:12:46.849484+0000","last_fullsized":"2024-03-29T10:12:46.849484+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:53.931903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.825004+0000","last_change":"2024-03-29T10:12:46.312741+0000","last_active":"2024-03-29T10:12:46.825004+0000","last_peered":"2024-03-29T10:12:46.825004+0000","last_clean":"2024-03-29T10:12:46.825004+0000","last_became_active":"2024-03-29T10:12:46.312336+0000","last_became_peered":"2024-03-29T10:12:46.312336+0000","last_unstale":"2024-03-29T10:12:46.825004+0000","last_undegraded":"2024-03-29T10:12:46.825004+0000","last_fullsized":"2024-03-29T10:12:46.825004+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:50:49.290619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.841096+0000","last_change":"2024-03-29T10:12:46.835696+0000","last_active":"2024-03-29T10:12:46.841096+0000","last_peered":"2024-03-29T10:12:46.841096+0000","last_clean":"2024-03-29T10:12:46.841096+0000","last_became_active":"2024-03-29T10:12:46.835435+0000","last_became_peered":"2024-03-29T10:12:46.835435+0000","last_unstale":"2024-03-29T10:12:46.841096+0000","last_undegraded":"2024-03-29T10:12:46.841096+0000","last_fullsized":"2024-03-29T10:12:46.841096+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:13.348427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.824371+0000","last_change":"2024-03-29T10:12:46.302806+0000","last_active":"2024-03-29T10:12:46.824371+0000","last_peered":"2024-03-29T10:12:46.824371+0000","last_clean":"2024-03-29T10:12:46.824371+0000","last_became_active":"2024-03-29T10:12:46.302462+0000","last_became_peered":"2024-03-29T10:12:46.302462+0000","last_unstale":"2024-03-29T10:12:46.824371+0000","last_undegraded":"2024-03-29T10:12:46.824371+0000","last_fullsized":"2024-03-29T10:12:46.824371+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:49.683950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.840454+0000","last_change":"2024-03-29T10:12:46.831841+0000","last_active":"2024-03-29T10:12:46.840454+0000","last_peered":"2024-03-29T10:12:46.840454+0000","last_clean":"2024-03-29T10:12:46.840454+0000","last_became_active":"2024-03-29T10:12:46.830791+0000","last_became_peered":"2024-03-29T10:12:46.830791+0000","last_unstale":"2024-03-29T10:12:46.840454+0000","last_undegraded":"2024-03-29T10:12:46.840454+0000","last_fullsized":"2024-03-29T10:12:46.840454+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:59.250713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"123'2","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.847313+0000","last_change":"2024-03-29T10:12:46.304763+0000","last_active":"2024-03-29T10:12:46.847313+0000","last_peered":"2024-03-29T10:12:46.847313+0000","last_clean":"2024-03-29T10:12:46.847313+0000","last_became_active":"2024-03-29T10:12:46.304433+0000","last_became_peered":"2024-03-29T10:12:46.304433+0000","last_unstale":"2024-03-29T10:12:46.847313+0000","last_undegraded":"2024-03-29T10:12:46.847313+0000","last_fullsized":"2024-03-29T10:12:46.847313+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:51.190947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.845804+0000","last_change":"2024-03-29T10:12:46.837100+0000","last_active":"2024-03-29T10:12:46.845804+0000","last_peered":"2024-03-29T10:12:46.845804+0000","last_clean":"2024-03-29T10:12:46.845804+0000","last_became_active":"2024-03-29T10:12:46.836793+0000","last_became_peered":"2024-03-29T10:12:46.836793+0000","last_unstale":"2024-03-29T10:12:46.845804+0000","last_undegraded":"2024-03-29T10:12:46.845804+0000","last_fullsized":"2024-03-29T10:12:46.845804+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:28:51.921521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.824615+0000","last_change":"2024-03-29T10:12:46.319070+0000","last_active":"2024-03-29T10:12:46.824615+0000","last_peered":"2024-03-29T10:12:46.824615+0000","last_clean":"2024-03-29T10:12:46.824615+0000","last_became_active":"2024-03-29T10:12:46.318749+0000","last_became_peered":"2024-03-29T10:12:46.318749+0000","last_unstale":"2024-03-29T10:12:46.824615+0000","last_undegraded":"2024-03-29T10:12:46.824615+0000","last_fullsized":"2024-03-29T10:12:46.824615+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:22.332793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.841459+0000","last_change":"2024-03-29T10:12:46.830088+0000","last_active":"2024-03-29T10:12:46.841459+0000","last_peered":"2024-03-29T10:12:46.841459+0000","last_clean":"2024-03-29T10:12:46.841459+0000","last_became_active":"2024-03-29T10:12:46.829791+0000","last_became_peered":"2024-03-29T10:12:46.829791+0000","last_unstale":"2024-03-29T10:12:46.841459+0000","last_undegraded":"2024-03-29T10:12:46.841459+0000","last_fullsized":"2024-03-29T10:12:46.841459+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:51:43.149222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.824903+0000","last_change":"2024-03-29T10:12:46.308003+0000","last_active":"2024-03-29T10:12:46.824903+0000","last_peered":"2024-03-29T10:12:46.824903+0000","last_clean":"2024-03-29T10:12:46.824903+0000","last_became_active":"2024-03-29T10:12:46.306971+0000","last_became_peered":"2024-03-29T10:12:46.306971+0000","last_unstale":"2024-03-29T10:12:46.824903+0000","last_undegraded":"2024-03-29T10:12:46.824903+0000","last_fullsized":"2024-03-29T10:12:46.824903+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:15.945108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.837335+0000","last_change":"2024-03-29T10:12:46.828528+0000","last_active":"2024-03-29T10:12:46.837335+0000","last_peered":"2024-03-29T10:12:46.837335+0000","last_clean":"2024-03-29T10:12:46.837335+0000","last_became_active":"2024-03-29T10:12:46.828249+0000","last_became_peered":"2024-03-29T10:12:46.828249+0000","last_unstale":"2024-03-29T10:12:46.837335+0000","last_undegraded":"2024-03-29T10:12:46.837335+0000","last_fullsized":"2024-03-29T10:12:46.837335+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:16:09.330311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.821996+0000","last_change":"2024-03-29T10:12:46.306751+0000","last_active":"2024-03-29T10:12:46.821996+0000","last_peered":"2024-03-29T10:12:46.821996+0000","last_clean":"2024-03-29T10:12:46.821996+0000","last_became_active":"2024-03-29T10:12:46.306387+0000","last_became_peered":"2024-03-29T10:12:46.306387+0000","last_unstale":"2024-03-29T10:12:46.821996+0000","last_undegraded":"2024-03-29T10:12:46.821996+0000","last_fullsized":"2024-03-29T10:12:46.821996+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:10.671851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.841099+0000","last_change":"2024-03-29T10:12:46.834284+0000","last_active":"2024-03-29T10:12:46.841099+0000","last_peered":"2024-03-29T10:12:46.841099+0000","last_clean":"2024-03-29T10:12:46.841099+0000","last_became_active":"2024-03-29T10:12:46.834021+0000","last_became_peered":"2024-03-29T10:12:46.834021+0000","last_unstale":"2024-03-29T10:12:46.841099+0000","last_undegraded":"2024-03-29T10:12:46.841099+0000","last_fullsized":"2024-03-29T10:12:46.841099+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:37:29.153650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.825221+0000","last_change":"2024-03-29T10:12:46.312874+0000","last_active":"2024-03-29T10:12:46.825221+0000","last_peered":"2024-03-29T10:12:46.825221+0000","last_clean":"2024-03-29T10:12:46.825221+0000","last_became_active":"2024-03-29T10:12:46.312528+0000","last_became_peered":"2024-03-29T10:12:46.312528+0000","last_unstale":"2024-03-29T10:12:46.825221+0000","last_undegraded":"2024-03-29T10:12:46.825221+0000","last_fullsized":"2024-03-29T10:12:46.825221+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:43:48.780020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.833973+0000","last_change":"2024-03-29T10:12:46.827416+0000","last_active":"2024-03-29T10:12:46.833973+0000","last_peered":"2024-03-29T10:12:46.833973+0000","last_clean":"2024-03-29T10:12:46.833973+0000","last_became_active":"2024-03-29T10:12:46.827020+0000","last_became_peered":"2024-03-29T10:12:46.827020+0000","last_unstale":"2024-03-29T10:12:46.833973+0000","last_undegraded":"2024-03-29T10:12:46.833973+0000","last_fullsized":"2024-03-29T10:12:46.833973+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:43.870958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.818587+0000","last_change":"2024-03-29T10:12:46.305217+0000","last_active":"2024-03-29T10:12:46.818587+0000","last_peered":"2024-03-29T10:12:46.818587+0000","last_clean":"2024-03-29T10:12:46.818587+0000","last_became_active":"2024-03-29T10:12:46.304901+0000","last_became_peered":"2024-03-29T10:12:46.304901+0000","last_unstale":"2024-03-29T10:12:46.818587+0000","last_undegraded":"2024-03-29T10:12:46.818587+0000","last_fullsized":"2024-03-29T10:12:46.818587+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:18:01.836601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.845941+0000","last_change":"2024-03-29T10:12:46.835369+0000","last_active":"2024-03-29T10:12:46.845941+0000","last_peered":"2024-03-29T10:12:46.845941+0000","last_clean":"2024-03-29T10:12:46.845941+0000","last_became_active":"2024-03-29T10:12:46.835124+0000","last_became_peered":"2024-03-29T10:12:46.835124+0000","last_unstale":"2024-03-29T10:12:46.845941+0000","last_undegraded":"2024-03-29T10:12:46.845941+0000","last_fullsized":"2024-03-29T10:12:46.845941+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:03:27.472539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.821370+0000","last_change":"2024-03-29T10:12:46.310673+0000","last_active":"2024-03-29T10:12:46.821370+0000","last_peered":"2024-03-29T10:12:46.821370+0000","last_clean":"2024-03-29T10:12:46.821370+0000","last_became_active":"2024-03-29T10:12:46.310391+0000","last_became_peered":"2024-03-29T10:12:46.310391+0000","last_unstale":"2024-03-29T10:12:46.821370+0000","last_undegraded":"2024-03-29T10:12:46.821370+0000","last_fullsized":"2024-03-29T10:12:46.821370+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:15.808972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.837369+0000","last_change":"2024-03-29T10:12:46.830458+0000","last_active":"2024-03-29T10:12:46.837369+0000","last_peered":"2024-03-29T10:12:46.837369+0000","last_clean":"2024-03-29T10:12:46.837369+0000","last_became_active":"2024-03-29T10:12:46.830106+0000","last_became_peered":"2024-03-29T10:12:46.830106+0000","last_unstale":"2024-03-29T10:12:46.837369+0000","last_undegraded":"2024-03-29T10:12:46.837369+0000","last_fullsized":"2024-03-29T10:12:46.837369+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:45.331011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.825422+0000","last_change":"2024-03-29T10:12:46.312698+0000","last_active":"2024-03-29T10:12:46.825422+0000","last_peered":"2024-03-29T10:12:46.825422+0000","last_clean":"2024-03-29T10:12:46.825422+0000","last_became_active":"2024-03-29T10:12:46.312481+0000","last_became_peered":"2024-03-29T10:12:46.312481+0000","last_unstale":"2024-03-29T10:12:46.825422+0000","last_undegraded":"2024-03-29T10:12:46.825422+0000","last_fullsized":"2024-03-29T10:12:46.825422+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:34.526149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.853493+0000","last_change":"2024-03-29T10:12:46.850592+0000","last_active":"2024-03-29T10:12:46.853493+0000","last_peered":"2024-03-29T10:12:46.853493+0000","last_clean":"2024-03-29T10:12:46.853493+0000","last_became_active":"2024-03-29T10:12:46.850356+0000","last_became_peered":"2024-03-29T10:12:46.850356+0000","last_unstale":"2024-03-29T10:12:46.853493+0000","last_undegraded":"2024-03-29T10:12:46.853493+0000","last_fullsized":"2024-03-29T10:12:46.853493+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:00:38.570406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"123'1","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.847396+0000","last_change":"2024-03-29T10:12:46.315886+0000","last_active":"2024-03-29T10:12:46.847396+0000","last_peered":"2024-03-29T10:12:46.847396+0000","last_clean":"2024-03-29T10:12:46.847396+0000","last_became_active":"2024-03-29T10:12:46.315577+0000","last_became_peered":"2024-03-29T10:12:46.315577+0000","last_unstale":"2024-03-29T10:12:46.847396+0000","last_undegraded":"2024-03-29T10:12:46.847396+0000","last_fullsized":"2024-03-29T10:12:46.847396+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:51:08.830748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"113'223","reported_seq":637,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.310650+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:12:46.310650+0000","last_peered":"2024-03-29T10:12:46.310650+0000","last_clean":"2024-03-29T10:12:46.310650+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:12:46.310650+0000","last_undegraded":"2024-03-29T10:12:46.310650+0000","last_fullsized":"2024-03-29T10:12:46.310650+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":351,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":351,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575062,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":2040,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":2088960,"data_stored":1301562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36043212},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.67700000000000005,"15min":0.71899999999999997},"min":{"1min":0.47399999999999998,"5min":0.44600000000000001,"15min":0.39000000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69199999999999995,"15min":0.71899999999999997},"min":{"1min":0.51000000000000001,"5min":0.40100000000000002,"15min":0.377},"max":{"1min":0.82199999999999995,"5min":1.143,"15min":1.143},"last":0.69299999999999995}]},{"osd":1,"last update":"Fri Mar 29 10:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.71099999999999997,"15min":0.72899999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.44500000000000001},"max":{"1min":0.98199999999999998,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.73899999999999999},"min":{"1min":0.51600000000000001,"5min":0.46899999999999997,"15min":0.41199999999999998},"max":{"1min":0.99099999999999999,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.94899999999999995}]},{"osd":2,"last update":"Fri Mar 29 10:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.72699999999999998,"15min":0.74399999999999999},"min":{"1min":0.48799999999999999,"5min":0.47199999999999998,"15min":0.45200000000000001},"max":{"1min":0.93899999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72499999999999998,"15min":0.754},"min":{"1min":0.51800000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.89800000000000002,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.75600000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70899999999999996,"15min":0.74099999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.40699999999999997},"max":{"1min":0.91200000000000003,"5min":1.171,"15min":1.171},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73999999999999999,"15min":0.77500000000000002},"min":{"1min":0.55400000000000005,"5min":0.52300000000000002,"15min":0.50700000000000001},"max":{"1min":0.88,"5min":1.129,"15min":1.129},"last":0.88100000000000001}]},{"osd":4,"last update":"Fri Mar 29 10:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61499999999999999,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.40600000000000003,"15min":0.38200000000000001},"max":{"1min":0.86699999999999999,"5min":1.25,"15min":1.25},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.41899999999999998,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":0.878,"5min":1.117,"15min":1.117},"last":0.96999999999999997}]},{"osd":5,"last update":"Fri Mar 29 10:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66000000000000003,"15min":0.64600000000000002},"min":{"1min":0.437,"5min":0.43099999999999999,"15min":0.40699999999999997},"max":{"1min":0.874,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.68200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46000000000000002,"15min":0.39400000000000002},"max":{"1min":0.95299999999999996,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.81000000000000005}]},{"osd":6,"last update":"Fri Mar 29 10:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74299999999999999,"15min":0.72699999999999998},"min":{"1min":0.53200000000000003,"5min":0.50600000000000001,"15min":0.504},"max":{"1min":1.0189999999999999,"5min":1.222,"15min":1.222},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.72099999999999997,"15min":0.69999999999999996},"min":{"1min":0.46899999999999997,"5min":0.443,"15min":0.443},"max":{"1min":1.0529999999999999,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":1.0740000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36688,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335152,"statfs":{"total":96636764160,"available":96599195648,"internally_reserved":0,"allocated":1507328,"data_stored":717469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.70999999999999996,"15min":0.70899999999999996},"min":{"1min":0.46500000000000002,"5min":0.42899999999999999,"15min":0.34699999999999998},"max":{"1min":1.006,"5min":2.7770000000000001,"15min":2.7770000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70399999999999996,"15min":0.72599999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93300000000000005,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.432}]},{"osd":1,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67600000000000005,"15min":0.70699999999999996},"min":{"1min":0.49199999999999999,"5min":0.46000000000000002,"15min":0.443},"max":{"1min":0.94799999999999995,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71899999999999997,"15min":0.72099999999999997},"min":{"1min":0.503,"5min":0.45200000000000001,"15min":0.42399999999999999},"max":{"1min":1.0289999999999999,"5min":2.6709999999999998,"15min":2.6709999999999998},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68000000000000005,"15min":0.71099999999999997},"min":{"1min":0.56699999999999995,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.93200000000000005,"5min":1.256,"15min":1.256},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72399999999999998,"15min":0.74299999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.40400000000000003},"max":{"1min":0.876,"5min":1.454,"15min":1.454},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.73699999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":0.97899999999999998,"5min":1.131,"15min":1.131},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.754,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.56899999999999995,"5min":0.44500000000000001,"15min":0.39000000000000001},"max":{"1min":1.034,"5min":2.718,"15min":2.718},"last":0.65500000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.626,"15min":0.60799999999999998},"min":{"1min":0.44400000000000001,"5min":0.40500000000000003,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70899999999999996,"15min":0.68999999999999995},"min":{"1min":0.51600000000000001,"5min":0.46999999999999997,"15min":0.40600000000000003},"max":{"1min":0.872,"5min":2.544,"15min":2.544},"last":0.61199999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67300000000000004,"15min":0.65600000000000003},"min":{"1min":0.41899999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79100000000000004,"5min":3.2280000000000002,"15min":3.2280000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.71399999999999997,"15min":0.68500000000000005},"min":{"1min":0.46700000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91400000000000003,"5min":2.835,"15min":2.835},"last":0.63100000000000001}]},{"osd":7,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75700000000000001,"15min":0.748},"min":{"1min":0.59699999999999998,"5min":0.48899999999999999,"15min":0.45100000000000001},"max":{"1min":0.94999999999999996,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72299999999999998,"15min":0.71599999999999997},"min":{"1min":0.501,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89800000000000002,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.68300000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575061,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54724,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317116,"statfs":{"total":96636764160,"available":96580726784,"internally_reserved":0,"allocated":1495040,"data_stored":711194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45900000000000002,"15min":0.44900000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.219},"max":{"1min":0.69399999999999995,"5min":0.81699999999999995,"15min":1.5600000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.41499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.65000000000000002,"5min":0.71799999999999997,"15min":1.0860000000000001},"last":0.47499999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.41899999999999998,"5min":0.38800000000000001,"15min":0.316},"max":{"1min":0.71499999999999997,"5min":1.222,"15min":1.7869999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53200000000000003,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33400000000000002,"15min":0.33200000000000002},"max":{"1min":0.65100000000000002,"5min":1.04,"15min":1.04},"last":0.51900000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57999999999999996,"15min":0.59299999999999997},"min":{"1min":0.48099999999999998,"5min":0.38600000000000001,"15min":0.377},"max":{"1min":0.74199999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.45700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71499999999999997,"5min":1.026,"15min":1.3049999999999999},"last":0.44400000000000001}]},{"osd":4,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.749,"15min":0.76300000000000001},"min":{"1min":0.57799999999999996,"5min":0.52900000000000003,"15min":0.497},"max":{"1min":0.95799999999999996,"5min":1.2709999999999999,"15min":1.617},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.77700000000000002,"15min":0.77500000000000002},"min":{"1min":0.60999999999999999,"5min":0.54900000000000004,"15min":0.48299999999999998},"max":{"1min":1.05,"5min":1.1419999999999999,"15min":1.155},"last":0.84899999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.77600000000000002,"15min":0.79400000000000004},"min":{"1min":0.67000000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.101,"5min":1.101,"15min":1.3939999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.78700000000000003,"15min":0.79800000000000004},"min":{"1min":0.62,"5min":0.55800000000000005,"15min":0.52900000000000003},"max":{"1min":1.0309999999999999,"5min":1.3140000000000001,"15min":1.6579999999999999},"last":0.75700000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.65200000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.996,"5min":1.0840000000000001,"15min":1.236},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.79400000000000004,"15min":0.81299999999999994},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.1419999999999999,"5min":1.1579999999999999,"15min":1.468},"last":0.93500000000000005}]},{"osd":7,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.66300000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0629999999999999,"5min":1.1319999999999999,"15min":1.6000000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.82899999999999996},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.57099999999999995},"max":{"1min":1.1240000000000001,"5min":1.4530000000000001,"15min":1.6919999999999999},"last":0.90300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575063,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2097152,"data_stored":1306818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.438,"15min":0.45300000000000001},"min":{"1min":0.35899999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.67100000000000004,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.48299999999999998,"15min":0.51000000000000001},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.70299999999999996,"5min":0.86699999999999999,"15min":1.0529999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.56299999999999994},"min":{"1min":0.38600000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.0069999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.46899999999999997,"15min":0.497},"min":{"1min":0.34200000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.65900000000000003,"5min":0.77800000000000002,"15min":0.97199999999999998},"last":0.372}]},{"osd":3,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52600000000000002,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78400000000000003,"5min":0.84899999999999998,"15min":1.077},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58399999999999996,"15min":0.61099999999999999},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78800000000000003,"5min":0.83499999999999996,"15min":1.0549999999999999},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.75800000000000001,"15min":0.79900000000000004},"min":{"1min":0.63900000000000001,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":1.0429999999999999,"5min":1.625,"15min":1.625},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77800000000000002,"15min":0.82099999999999995},"min":{"1min":0.60099999999999998,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.167},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.70399999999999996,"15min":0.755},"min":{"1min":0.56399999999999995,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97199999999999998,"5min":1.02,"15min":1.137},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.75700000000000001,"15min":0.80100000000000005},"min":{"1min":0.53300000000000003,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":1.0589999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.66600000000000004}]},{"osd":6,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.78400000000000003,"15min":0.81799999999999995},"min":{"1min":0.56799999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.014,"5min":1.0640000000000001,"15min":1.228},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.80000000000000004,"15min":0.84199999999999997},"min":{"1min":0.67800000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.105,"5min":1.601,"15min":1.601},"last":0.72699999999999998}]},{"osd":7,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.745,"15min":0.79500000000000004},"min":{"1min":0.59899999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.1890000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.79700000000000004,"15min":0.83599999999999997},"min":{"1min":0.58899999999999997,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.135,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.70499999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575062,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1523712,"data_stored":724278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.248},"max":{"1min":0.71699999999999997,"5min":0.746,"15min":0.86099999999999999},"last":1.236},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.502,"15min":0.50900000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.82399999999999995},"last":0.86499999999999999}]},{"osd":1,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48699999999999999,"15min":0.47299999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26200000000000001},"max":{"1min":0.59899999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68100000000000005,"5min":0.78500000000000003,"15min":1.032},"last":0.40600000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.442,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.99099999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55000000000000004,"15min":0.54700000000000004},"min":{"1min":0.40600000000000003,"5min":0.35199999999999998,"15min":0.32100000000000001},"max":{"1min":0.75700000000000001,"5min":1.516,"15min":1.516},"last":2.3570000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.77800000000000002},"min":{"1min":0.54200000000000004,"5min":0.499,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":1.125,"15min":1.3520000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.747,"5min":0.76000000000000001,"15min":0.79700000000000004},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":1.01}]},{"osd":5,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76500000000000001,"15min":0.80000000000000004},"min":{"1min":0.52400000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0569999999999999,"5min":1.1699999999999999,"15min":1.2589999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.76100000000000001,"15min":0.79600000000000004},"min":{"1min":0.56499999999999995,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0469999999999999,"5min":1.115,"15min":1.2250000000000001},"last":1.1919999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78000000000000003,"15min":0.81200000000000006},"min":{"1min":0.55100000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0389999999999999,"5min":1.1859999999999999,"15min":1.2350000000000001},"last":1.6739999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.83199999999999996},"min":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.99099999999999999,"5min":1.1120000000000001,"15min":1.3560000000000001},"last":0.875}]},{"osd":7,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.83499999999999996,"15min":0.86199999999999999},"min":{"1min":0.63900000000000001,"5min":0.57899999999999996,"15min":0.53400000000000003},"max":{"1min":1.1339999999999999,"5min":1.258,"15min":1.4490000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83999999999999997,"15min":0.872},"min":{"1min":0.63100000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1180000000000001,"5min":1.226,"15min":1.4199999999999999},"last":0.78300000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575062,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36692,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335148,"statfs":{"total":96636764160,"available":96599191552,"internally_reserved":0,"allocated":1507328,"data_stored":717469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.26800000000000002,"15min":0.26500000000000001},"max":{"1min":0.65100000000000002,"5min":0.75600000000000001,"15min":0.81299999999999994},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.29899999999999999,"5min":0.29099999999999998,"15min":0.28399999999999997},"max":{"1min":0.73999999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.66000000000000003}]},{"osd":1,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46200000000000002,"15min":0.48599999999999999},"min":{"1min":0.36399999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.98699999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54000000000000004,"15min":0.56200000000000006},"min":{"1min":0.46600000000000003,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72599999999999998,"5min":0.82699999999999996,"15min":0.96099999999999997},"last":0.47299999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.373},"max":{"1min":0.75900000000000001,"5min":0.83999999999999997,"15min":0.98299999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59899999999999998,"15min":0.624},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80000000000000004,"5min":0.86899999999999999,"15min":0.94799999999999995},"last":0.57099999999999995}]},{"osd":4,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.73799999999999999,"15min":0.76600000000000001},"min":{"1min":0.55300000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.99299999999999999,"5min":1.0940000000000001,"15min":1.288},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.60199999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.004,"5min":1.093,"15min":1.46},"last":0.68000000000000005}]},{"osd":5,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.77000000000000002,"15min":0.79200000000000004},"min":{"1min":0.62,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.3580000000000001},"last":0.621},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.76300000000000001,"15min":0.79700000000000004},"min":{"1min":0.53800000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92900000000000005,"5min":1.1479999999999999,"15min":1.429},"last":0.74399999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.78600000000000003,"15min":0.80800000000000005},"min":{"1min":0.58099999999999996,"5min":0.54500000000000004,"15min":0.54200000000000004},"max":{"1min":1.123,"5min":1.123,"15min":1.4730000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78800000000000003,"15min":0.81599999999999995},"min":{"1min":0.66900000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.3999999999999999},"last":0.71699999999999997}]},{"osd":7,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79700000000000004,"15min":0.83299999999999996},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0489999999999999,"5min":1.135,"15min":1.5029999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.81200000000000006,"15min":0.83499999999999996},"min":{"1min":0.63300000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.073,"5min":1.1859999999999999,"15min":1.345},"last":0.78000000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58816,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313024,"statfs":{"total":96636764160,"available":96576536576,"internally_reserved":0,"allocated":1495040,"data_stored":711194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68700000000000006,"15min":0.69299999999999995},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.91200000000000003,"5min":0.98799999999999999,"15min":1.417},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68700000000000006,"15min":0.69699999999999995},"min":{"1min":0.38,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.165},"last":0.53500000000000003}]},{"osd":1,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.69099999999999995,"15min":0.70599999999999996},"min":{"1min":0.55700000000000005,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.014},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69799999999999995,"15min":0.70999999999999996},"min":{"1min":0.46300000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.876,"5min":1.075,"15min":1.333},"last":0.51100000000000001}]},{"osd":2,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.77600000000000002,"15min":0.77400000000000002},"min":{"1min":0.66400000000000003,"5min":0.433,"15min":0.433},"max":{"1min":1.014,"5min":1.014,"15min":1.03},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.77900000000000003,"15min":0.77200000000000002},"min":{"1min":0.59699999999999998,"5min":0.50600000000000001,"15min":0.46200000000000002},"max":{"1min":1.024,"5min":1.0600000000000001,"15min":1.387},"last":0.79200000000000004}]},{"osd":3,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.72499999999999998,"15min":0.72699999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.2350000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72199999999999998,"15min":0.73499999999999999},"min":{"1min":0.53800000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.90400000000000003,"5min":1.0229999999999999,"15min":1.3180000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60499999999999998,"15min":0.58499999999999996},"min":{"1min":0.45700000000000002,"5min":0.42499999999999999,"15min":0.38100000000000001},"max":{"1min":0.83799999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.746},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.629,"15min":0.60399999999999998},"min":{"1min":0.435,"5min":0.36399999999999999,"15min":0.309},"max":{"1min":0.91600000000000004,"5min":0.98599999999999999,"15min":1.294},"last":0.73699999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64400000000000002,"15min":0.61899999999999999},"min":{"1min":0.47599999999999998,"5min":0.44,"15min":0.375},"max":{"1min":0.85099999999999998,"5min":1.026,"15min":1.026},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67300000000000004,"15min":0.65800000000000003},"min":{"1min":0.503,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97599999999999998,"5min":1.093,"15min":1.1359999999999999},"last":0.82999999999999996}]},{"osd":7,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68799999999999994,"15min":0.68100000000000005},"min":{"1min":0.51100000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.345},"last":0.874},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.77500000000000002,"15min":0.75700000000000001},"min":{"1min":0.64000000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.995,"5min":0.999,"15min":1.214},"last":0.89200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575062,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36684,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335156,"statfs":{"total":96636764160,"available":96599199744,"internally_reserved":0,"allocated":1507328,"data_stored":717469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64200000000000002,"15min":0.68300000000000005},"min":{"1min":0.41699999999999998,"5min":0.39200000000000002,"15min":0.35699999999999998},"max":{"1min":0.83999999999999997,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.746,"5min":0.71599999999999997,"15min":0.72999999999999998},"min":{"1min":0.56599999999999995,"5min":0.435,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":1.21,"15min":1.339},"last":0.56399999999999995}]},{"osd":1,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68799999999999994,"15min":0.71799999999999997},"min":{"1min":0.496,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67100000000000004,"15min":0.70699999999999996},"min":{"1min":0.52500000000000002,"5min":0.441,"15min":0.39200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70399999999999996,"15min":0.73199999999999998},"min":{"1min":0.60299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.877,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67900000000000005,"15min":0.70799999999999996},"min":{"1min":0.47499999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92900000000000005,"5min":0.93700000000000006,"15min":1.095},"last":0.60599999999999998}]},{"osd":3,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.69099999999999995,"15min":0.72899999999999998},"min":{"1min":0.54400000000000004,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":0.878,"5min":1.042,"15min":1.1419999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.72199999999999998,"15min":0.752},"min":{"1min":0.60299999999999998,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.94899999999999995,"5min":1.0249999999999999,"15min":1.4850000000000001},"last":0.59699999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.40899999999999997,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.91700000000000004,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64700000000000002,"15min":0.63800000000000001},"min":{"1min":0.45700000000000002,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":0.92900000000000005,"5min":1.097,"15min":1.097},"last":0.51800000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.65300000000000002,"15min":0.65200000000000002},"min":{"1min":0.56399999999999995,"5min":0.41299999999999998,"15min":0.36799999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":1.151},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66200000000000003,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89000000000000001,"5min":0.97099999999999997,"15min":0.996},"last":0.54700000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68300000000000005,"15min":0.69599999999999995},"min":{"1min":0.48999999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89400000000000002,"5min":0.95299999999999996,"15min":1.0289999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.52200000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94799999999999995,"5min":0.98699999999999999,"15min":1.1990000000000001},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:12:51.737 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:12:51.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.969+0000 7f8d62c00700 1 Processor -- start 2024-03-29T10:12:51.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.969+0000 7f8d62c00700 1 -- start start 2024-03-29T10:12:51.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d62c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c160f10 0x7f8d5c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d62c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8d5c161590 0x7f8d5c161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d62c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c0ceda0 con 0x7f8d5c0cf230 2024-03-29T10:12:51.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d62c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c161f80 con 0x7f8d5c161590 2024-03-29T10:12:51.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d62c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c1612f0 con 0x7f8d5c160f10 2024-03-29T10:12:51.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d5b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8d5c0cf230 0x7f8d5c0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47466/0 (socket says 172.21.15.12:47466) 2024-03-29T10:12:51.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d5b400700 1 -- 172.21.15.12:0/269473693 learned_addr learned my addr 172.21.15.12:0/269473693 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:51.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d60c00700 1 --2- 172.21.15.12:0/269473693 >> v2:172.21.15.159:3300/0 conn(0x7f8d5c161590 0x7f8d5c161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d5be00700 1 --2- 172.21.15.12:0/269473693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c160f10 0x7f8d5c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d5aa00700 1 -- 172.21.15.12:0/269473693 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211125319 0 0) 0x7f8d5c0ceda0 con 0x7f8d5c0cf230 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.971+0000 7f8d5aa00700 1 -- 172.21.15.12:0/269473693 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d400035b0 con 0x7f8d5c0cf230 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.972+0000 7f8d60c00700 1 -- 172.21.15.12:0/269473693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c160f10 msgr2=0x7f8d5c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.972+0000 7f8d60c00700 1 --2- 172.21.15.12:0/269473693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c160f10 0x7f8d5c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.972+0000 7f8d60c00700 1 -- 172.21.15.12:0/269473693 >> v1:172.21.15.12:6789/0 conn(0x7f8d5c0cf230 legacy=0x7f8d5c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.972+0000 7f8d60c00700 1 -- 172.21.15.12:0/269473693 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c0ce470 con 0x7f8d5c161590 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.972+0000 7f8d60c00700 1 --2- 172.21.15.12:0/269473693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5c161590 0x7f8d5c161a40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8d54009dc0 tx=0x7f8d5400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74aee8b73ba6eb7a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.972+0000 7f8d5aa00700 1 -- 172.21.15.12:0/269473693 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5400e070 con 0x7f8d5c161590 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.972+0000 7f8d5aa00700 1 -- 172.21.15.12:0/269473693 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5400bae0 con 0x7f8d5c161590 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.972+0000 7f8d5aa00700 1 -- 172.21.15.12:0/269473693 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5400be90 con 0x7f8d5c161590 2024-03-29T10:12:51.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.973+0000 7f8d62c00700 1 -- 172.21.15.12:0/269473693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5c161590 msgr2=0x7f8d5c161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.973+0000 7f8d62c00700 1 --2- 172.21.15.12:0/269473693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5c161590 0x7f8d5c161a40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8d54009dc0 tx=0x7f8d5400b330 comp rx=0 tx=0).stop 2024-03-29T10:12:51.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.973+0000 7f8d62c00700 1 -- 172.21.15.12:0/269473693 shutdown_connections 2024-03-29T10:12:51.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.973+0000 7f8d62c00700 1 --2- 172.21.15.12:0/269473693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c160f10 0x7f8d5c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.973+0000 7f8d62c00700 1 --2- 172.21.15.12:0/269473693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5c161590 0x7f8d5c161a40 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.973+0000 7f8d62c00700 1 -- 172.21.15.12:0/269473693 >> 172.21.15.12:0/269473693 conn(0x7f8d5c08aff0 msgr2=0x7f8d5c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:51.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.973+0000 7f8d62c00700 1 -- 172.21.15.12:0/269473693 shutdown_connections 2024-03-29T10:12:51.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.973+0000 7f8d62c00700 1 -- 172.21.15.12:0/269473693 wait complete. 2024-03-29T10:12:51.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.975+0000 7f8d62c00700 1 Processor -- start 2024-03-29T10:12:51.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.975+0000 7f8d62c00700 1 -- start start 2024-03-29T10:12:51.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d62c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c0cf230 0x7f8d5c15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d62c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d5c160f10 0x7f8d5c1f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d62c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5c161590 0x7f8d5c0b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d62c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0b7410 con 0x7f8d5c160f10 2024-03-29T10:12:51.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d62c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c1f6cb0 con 0x7f8d5c161590 2024-03-29T10:12:51.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d62c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c1f6f90 con 0x7f8d5c0cf230 2024-03-29T10:12:51.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d5b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d5c160f10 0x7f8d5c1f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d5b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d5c160f10 0x7f8d5c1f6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52342/0 (socket says 172.21.15.12:52342) 2024-03-29T10:12:51.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d5b400700 1 -- 172.21.15.12:0/1765022344 learned_addr learned my addr 172.21.15.12:0/1765022344 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:51.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d5be00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c0cf230 0x7f8d5c15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d60c00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5c161590 0x7f8d5c0b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d5b400700 1 -- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c0cf230 msgr2=0x7f8d5c15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d5b400700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c0cf230 0x7f8d5c15ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d5b400700 1 -- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5c161590 msgr2=0x7f8d5c0b7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d5b400700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5c161590 0x7f8d5c0b7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d5b400700 1 -- 172.21.15.12:0/1765022344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d54009a70 con 0x7f8d5c160f10 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.977+0000 7f8d5be00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c0cf230 0x7f8d5c15ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.978+0000 7f8d5b400700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d5c160f10 0x7f8d5c1f6230 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f8d48005910 tx=0x7f8d4800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.978+0000 7f8d53e00700 1 -- 172.21.15.12:0/1765022344 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48005d70 con 0x7f8d5c160f10 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.978+0000 7f8d53e00700 1 -- 172.21.15.12:0/1765022344 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d480070d0 con 0x7f8d5c160f10 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.978+0000 7f8d62c00700 1 -- 172.21.15.12:0/1765022344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c0b7660 con 0x7f8d5c160f10 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.978+0000 7f8d53e00700 1 -- 172.21.15.12:0/1765022344 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48007440 con 0x7f8d5c160f10 2024-03-29T10:12:51.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.978+0000 7f8d62c00700 1 -- 172.21.15.12:0/1765022344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c15b060 con 0x7f8d5c160f10 2024-03-29T10:12:51.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.979+0000 7f8d62c00700 1 -- 172.21.15.12:0/1765022344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c0b7b80 con 0x7f8d5c160f10 2024-03-29T10:12:51.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.980+0000 7f8d53e00700 1 -- 172.21.15.12:0/1765022344 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8d480075a0 con 0x7f8d5c160f10 2024-03-29T10:12:51.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.981+0000 7f8d53e00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d3c06adb0 0x7f8d3c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:51.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.981+0000 7f8d53e00700 1 -- 172.21.15.12:0/1765022344 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d48088870 con 0x7f8d5c160f10 2024-03-29T10:12:51.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.982+0000 7f8d5be00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d3c06adb0 0x7f8d3c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:51.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.983+0000 7f8d5be00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d3c06adb0 0x7f8d3c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0b93a0 tx=0x7f8d4c008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:51.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:51.986+0000 7f8d53e00700 1 -- 172.21.15.12:0/1765022344 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d5c0b7b80 con 0x7f8d5c160f10 2024-03-29T10:12:52.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.199+0000 7f8d62c00700 1 -- 172.21.15.12:0/1765022344 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d5c0cf6b0 con 0x7f8d3c06adb0 2024-03-29T10:12:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.208+0000 7f8d53e00700 1 -- 172.21.15.12:0/1765022344 <== mgr.7402 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289467 (secure 0 0 0) 0x7f8d4c011780 con 0x7f8d3c06adb0 2024-03-29T10:12:52.212 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 -- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d3c06adb0 msgr2=0x7f8d3c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d3c06adb0 0x7f8d3c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0b93a0 tx=0x7f8d4c008040 comp rx=0 tx=0).stop 2024-03-29T10:12:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 -- 172.21.15.12:0/1765022344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d5c160f10 msgr2=0x7f8d5c1f6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d5c160f10 0x7f8d5c1f6230 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f8d48005910 tx=0x7f8d4800c860 comp rx=0 tx=0).stop 2024-03-29T10:12:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 -- 172.21.15.12:0/1765022344 shutdown_connections 2024-03-29T10:12:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8d3c06adb0 0x7f8d3c06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5c161590 0x7f8d5c0b7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5c0cf230 0x7f8d5c15ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 --2- 172.21.15.12:0/1765022344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d5c160f10 0x7f8d5c1f6230 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 -- 172.21.15.12:0/1765022344 >> 172.21.15.12:0/1765022344 conn(0x7f8d5c08aff0 msgr2=0x7f8d5c0ca770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:52.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 -- 172.21.15.12:0/1765022344 shutdown_connections 2024-03-29T10:12:52.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.212+0000 7f8d62c00700 1 -- 172.21.15.12:0/1765022344 wait complete. 2024-03-29T10:12:52.216 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:12:52.230 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-29T10:12:51.983781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":396,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":262,"ondisk_log_size":262,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":334992,"kb_used_data":13072,"kb_used_omap":12,"kb_used_meta":321779,"kb_avail":754639728,"statfs":{"total":773094113280,"available":772751081472,"internally_reserved":0,"allocated":13385728,"data_stored":6980883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":329502265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001475"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976814+0000","last_change":"2024-03-29T10:12:46.829400+0000","last_active":"2024-03-29T10:12:46.976814+0000","last_peered":"2024-03-29T10:12:46.976814+0000","last_clean":"2024-03-29T10:12:46.976814+0000","last_became_active":"2024-03-29T10:12:46.829140+0000","last_became_peered":"2024-03-29T10:12:46.829140+0000","last_unstale":"2024-03-29T10:12:46.976814+0000","last_undegraded":"2024-03-29T10:12:46.976814+0000","last_fullsized":"2024-03-29T10:12:46.976814+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:52.200329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981197+0000","last_change":"2024-03-29T10:12:46.831622+0000","last_active":"2024-03-29T10:12:46.981197+0000","last_peered":"2024-03-29T10:12:46.981197+0000","last_clean":"2024-03-29T10:12:46.981197+0000","last_became_active":"2024-03-29T10:12:46.830998+0000","last_became_peered":"2024-03-29T10:12:46.830998+0000","last_unstale":"2024-03-29T10:12:46.981197+0000","last_undegraded":"2024-03-29T10:12:46.981197+0000","last_fullsized":"2024-03-29T10:12:46.981197+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:42:09.177945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976803+0000","last_change":"2024-03-29T10:12:46.837275+0000","last_active":"2024-03-29T10:12:46.976803+0000","last_peered":"2024-03-29T10:12:46.976803+0000","last_clean":"2024-03-29T10:12:46.976803+0000","last_became_active":"2024-03-29T10:12:46.831882+0000","last_became_peered":"2024-03-29T10:12:46.831882+0000","last_unstale":"2024-03-29T10:12:46.976803+0000","last_undegraded":"2024-03-29T10:12:46.976803+0000","last_fullsized":"2024-03-29T10:12:46.976803+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:19:01.629952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.979780+0000","last_change":"2024-03-29T10:12:46.846790+0000","last_active":"2024-03-29T10:12:46.979780+0000","last_peered":"2024-03-29T10:12:46.979780+0000","last_clean":"2024-03-29T10:12:46.979780+0000","last_became_active":"2024-03-29T10:12:46.845478+0000","last_became_peered":"2024-03-29T10:12:46.845478+0000","last_unstale":"2024-03-29T10:12:46.979780+0000","last_undegraded":"2024-03-29T10:12:46.979780+0000","last_fullsized":"2024-03-29T10:12:46.979780+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:13.348427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.840973+0000","last_change":"2024-03-29T10:12:46.835467+0000","last_active":"2024-03-29T10:12:46.840973+0000","last_peered":"2024-03-29T10:12:46.840973+0000","last_clean":"2024-03-29T10:12:46.840973+0000","last_became_active":"2024-03-29T10:12:46.835033+0000","last_became_peered":"2024-03-29T10:12:46.835033+0000","last_unstale":"2024-03-29T10:12:46.840973+0000","last_undegraded":"2024-03-29T10:12:46.840973+0000","last_fullsized":"2024-03-29T10:12:46.840973+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:56.220192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982559+0000","last_change":"2024-03-29T10:12:46.840582+0000","last_active":"2024-03-29T10:12:46.982559+0000","last_peered":"2024-03-29T10:12:46.982559+0000","last_clean":"2024-03-29T10:12:46.982559+0000","last_became_active":"2024-03-29T10:12:46.840297+0000","last_became_peered":"2024-03-29T10:12:46.840297+0000","last_unstale":"2024-03-29T10:12:46.982559+0000","last_undegraded":"2024-03-29T10:12:46.982559+0000","last_fullsized":"2024-03-29T10:12:46.982559+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:13.348427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982541+0000","last_change":"2024-03-29T10:12:46.840753+0000","last_active":"2024-03-29T10:12:46.982541+0000","last_peered":"2024-03-29T10:12:46.982541+0000","last_clean":"2024-03-29T10:12:46.982541+0000","last_became_active":"2024-03-29T10:12:46.839371+0000","last_became_peered":"2024-03-29T10:12:46.839371+0000","last_unstale":"2024-03-29T10:12:46.982541+0000","last_undegraded":"2024-03-29T10:12:46.982541+0000","last_fullsized":"2024-03-29T10:12:46.982541+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:40:47.003642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.980957+0000","last_change":"2024-03-29T10:12:46.836293+0000","last_active":"2024-03-29T10:12:46.980957+0000","last_peered":"2024-03-29T10:12:46.980957+0000","last_clean":"2024-03-29T10:12:46.980957+0000","last_became_active":"2024-03-29T10:12:46.832168+0000","last_became_peered":"2024-03-29T10:12:46.832168+0000","last_unstale":"2024-03-29T10:12:46.980957+0000","last_undegraded":"2024-03-29T10:12:46.980957+0000","last_fullsized":"2024-03-29T10:12:46.980957+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:39:54.721927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.840056+0000","last_change":"2024-03-29T10:12:46.835070+0000","last_active":"2024-03-29T10:12:46.840056+0000","last_peered":"2024-03-29T10:12:46.840056+0000","last_clean":"2024-03-29T10:12:46.840056+0000","last_became_active":"2024-03-29T10:12:46.834789+0000","last_became_peered":"2024-03-29T10:12:46.834789+0000","last_unstale":"2024-03-29T10:12:46.840056+0000","last_undegraded":"2024-03-29T10:12:46.840056+0000","last_fullsized":"2024-03-29T10:12:46.840056+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:53.536977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.849522+0000","last_change":"2024-03-29T10:12:46.846327+0000","last_active":"2024-03-29T10:12:46.849522+0000","last_peered":"2024-03-29T10:12:46.849522+0000","last_clean":"2024-03-29T10:12:46.849522+0000","last_became_active":"2024-03-29T10:12:46.845951+0000","last_became_peered":"2024-03-29T10:12:46.845951+0000","last_unstale":"2024-03-29T10:12:46.849522+0000","last_undegraded":"2024-03-29T10:12:46.849522+0000","last_fullsized":"2024-03-29T10:12:46.849522+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:43:49.789402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982600+0000","last_change":"2024-03-29T10:12:46.842898+0000","last_active":"2024-03-29T10:12:46.982600+0000","last_peered":"2024-03-29T10:12:46.982600+0000","last_clean":"2024-03-29T10:12:46.982600+0000","last_became_active":"2024-03-29T10:12:46.842128+0000","last_became_peered":"2024-03-29T10:12:46.842128+0000","last_unstale":"2024-03-29T10:12:46.982600+0000","last_undegraded":"2024-03-29T10:12:46.982600+0000","last_fullsized":"2024-03-29T10:12:46.982600+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:45.331011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.848991+0000","last_change":"2024-03-29T10:12:46.846480+0000","last_active":"2024-03-29T10:12:46.848991+0000","last_peered":"2024-03-29T10:12:46.848991+0000","last_clean":"2024-03-29T10:12:46.848991+0000","last_became_active":"2024-03-29T10:12:46.846162+0000","last_became_peered":"2024-03-29T10:12:46.846162+0000","last_unstale":"2024-03-29T10:12:46.848991+0000","last_undegraded":"2024-03-29T10:12:46.848991+0000","last_fullsized":"2024-03-29T10:12:46.848991+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:56.332969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.979771+0000","last_change":"2024-03-29T10:12:46.835069+0000","last_active":"2024-03-29T10:12:46.979771+0000","last_peered":"2024-03-29T10:12:46.979771+0000","last_clean":"2024-03-29T10:12:46.979771+0000","last_became_active":"2024-03-29T10:12:46.834813+0000","last_became_peered":"2024-03-29T10:12:46.834813+0000","last_unstale":"2024-03-29T10:12:46.979771+0000","last_undegraded":"2024-03-29T10:12:46.979771+0000","last_fullsized":"2024-03-29T10:12:46.979771+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:43.403428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978871+0000","last_change":"2024-03-29T10:12:46.838992+0000","last_active":"2024-03-29T10:12:46.978871+0000","last_peered":"2024-03-29T10:12:46.978871+0000","last_clean":"2024-03-29T10:12:46.978871+0000","last_became_active":"2024-03-29T10:12:46.838712+0000","last_became_peered":"2024-03-29T10:12:46.838712+0000","last_unstale":"2024-03-29T10:12:46.978871+0000","last_undegraded":"2024-03-29T10:12:46.978871+0000","last_fullsized":"2024-03-29T10:12:46.978871+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:20:09.840649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.979717+0000","last_change":"2024-03-29T10:12:46.846667+0000","last_active":"2024-03-29T10:12:46.979717+0000","last_peered":"2024-03-29T10:12:46.979717+0000","last_clean":"2024-03-29T10:12:46.979717+0000","last_became_active":"2024-03-29T10:12:46.846362+0000","last_became_peered":"2024-03-29T10:12:46.846362+0000","last_unstale":"2024-03-29T10:12:46.979717+0000","last_undegraded":"2024-03-29T10:12:46.979717+0000","last_fullsized":"2024-03-29T10:12:46.979717+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:41.550218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982474+0000","last_change":"2024-03-29T10:12:46.839581+0000","last_active":"2024-03-29T10:12:46.982474+0000","last_peered":"2024-03-29T10:12:46.982474+0000","last_clean":"2024-03-29T10:12:46.982474+0000","last_became_active":"2024-03-29T10:12:46.837616+0000","last_became_peered":"2024-03-29T10:12:46.837616+0000","last_unstale":"2024-03-29T10:12:46.982474+0000","last_undegraded":"2024-03-29T10:12:46.982474+0000","last_fullsized":"2024-03-29T10:12:46.982474+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:41.550218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981285+0000","last_change":"2024-03-29T10:12:46.835776+0000","last_active":"2024-03-29T10:12:46.981285+0000","last_peered":"2024-03-29T10:12:46.981285+0000","last_clean":"2024-03-29T10:12:46.981285+0000","last_became_active":"2024-03-29T10:12:46.835460+0000","last_became_peered":"2024-03-29T10:12:46.835460+0000","last_unstale":"2024-03-29T10:12:46.981285+0000","last_undegraded":"2024-03-29T10:12:46.981285+0000","last_fullsized":"2024-03-29T10:12:46.981285+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:42:38.641476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.979789+0000","last_change":"2024-03-29T10:12:46.852405+0000","last_active":"2024-03-29T10:12:46.979789+0000","last_peered":"2024-03-29T10:12:46.979789+0000","last_clean":"2024-03-29T10:12:46.979789+0000","last_became_active":"2024-03-29T10:12:46.851997+0000","last_became_peered":"2024-03-29T10:12:46.851997+0000","last_unstale":"2024-03-29T10:12:46.979789+0000","last_undegraded":"2024-03-29T10:12:46.979789+0000","last_fullsized":"2024-03-29T10:12:46.979789+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:53.536977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978941+0000","last_change":"2024-03-29T10:12:46.840469+0000","last_active":"2024-03-29T10:12:46.978941+0000","last_peered":"2024-03-29T10:12:46.978941+0000","last_clean":"2024-03-29T10:12:46.978941+0000","last_became_active":"2024-03-29T10:12:46.840212+0000","last_became_peered":"2024-03-29T10:12:46.840212+0000","last_unstale":"2024-03-29T10:12:46.978941+0000","last_undegraded":"2024-03-29T10:12:46.978941+0000","last_fullsized":"2024-03-29T10:12:46.978941+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:38.075899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.847100+0000","last_change":"2024-03-29T10:12:46.843181+0000","last_active":"2024-03-29T10:12:46.847100+0000","last_peered":"2024-03-29T10:12:46.847100+0000","last_clean":"2024-03-29T10:12:46.847100+0000","last_became_active":"2024-03-29T10:12:46.842878+0000","last_became_peered":"2024-03-29T10:12:46.842878+0000","last_unstale":"2024-03-29T10:12:46.847100+0000","last_undegraded":"2024-03-29T10:12:46.847100+0000","last_fullsized":"2024-03-29T10:12:46.847100+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:58:37.690997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978918+0000","last_change":"2024-03-29T10:12:46.835563+0000","last_active":"2024-03-29T10:12:46.978918+0000","last_peered":"2024-03-29T10:12:46.978918+0000","last_clean":"2024-03-29T10:12:46.978918+0000","last_became_active":"2024-03-29T10:12:46.834725+0000","last_became_peered":"2024-03-29T10:12:46.834725+0000","last_unstale":"2024-03-29T10:12:46.978918+0000","last_undegraded":"2024-03-29T10:12:46.978918+0000","last_fullsized":"2024-03-29T10:12:46.978918+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:50:42.501111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981019+0000","last_change":"2024-03-29T10:12:46.828788+0000","last_active":"2024-03-29T10:12:46.981019+0000","last_peered":"2024-03-29T10:12:46.981019+0000","last_clean":"2024-03-29T10:12:46.981019+0000","last_became_active":"2024-03-29T10:12:46.828528+0000","last_became_peered":"2024-03-29T10:12:46.828528+0000","last_unstale":"2024-03-29T10:12:46.981019+0000","last_undegraded":"2024-03-29T10:12:46.981019+0000","last_fullsized":"2024-03-29T10:12:46.981019+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:31.163681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981003+0000","last_change":"2024-03-29T10:12:46.828500+0000","last_active":"2024-03-29T10:12:46.981003+0000","last_peered":"2024-03-29T10:12:46.981003+0000","last_clean":"2024-03-29T10:12:46.981003+0000","last_became_active":"2024-03-29T10:12:46.828140+0000","last_became_peered":"2024-03-29T10:12:46.828140+0000","last_unstale":"2024-03-29T10:12:46.981003+0000","last_undegraded":"2024-03-29T10:12:46.981003+0000","last_fullsized":"2024-03-29T10:12:46.981003+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:42:34.437921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982502+0000","last_change":"2024-03-29T10:12:46.851135+0000","last_active":"2024-03-29T10:12:46.982502+0000","last_peered":"2024-03-29T10:12:46.982502+0000","last_clean":"2024-03-29T10:12:46.982502+0000","last_became_active":"2024-03-29T10:12:46.850882+0000","last_became_peered":"2024-03-29T10:12:46.850882+0000","last_unstale":"2024-03-29T10:12:46.982502+0000","last_undegraded":"2024-03-29T10:12:46.982502+0000","last_fullsized":"2024-03-29T10:12:46.982502+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:54.751799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.837808+0000","last_change":"2024-03-29T10:12:46.831971+0000","last_active":"2024-03-29T10:12:46.837808+0000","last_peered":"2024-03-29T10:12:46.837808+0000","last_clean":"2024-03-29T10:12:46.837808+0000","last_became_active":"2024-03-29T10:12:46.831695+0000","last_became_peered":"2024-03-29T10:12:46.831695+0000","last_unstale":"2024-03-29T10:12:46.837808+0000","last_undegraded":"2024-03-29T10:12:46.837808+0000","last_fullsized":"2024-03-29T10:12:46.837808+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:58:37.690997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981217+0000","last_change":"2024-03-29T10:12:46.842057+0000","last_active":"2024-03-29T10:12:46.981217+0000","last_peered":"2024-03-29T10:12:46.981217+0000","last_clean":"2024-03-29T10:12:46.981217+0000","last_became_active":"2024-03-29T10:12:46.841783+0000","last_became_peered":"2024-03-29T10:12:46.841783+0000","last_unstale":"2024-03-29T10:12:46.981217+0000","last_undegraded":"2024-03-29T10:12:46.981217+0000","last_fullsized":"2024-03-29T10:12:46.981217+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:07:59.900469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982581+0000","last_change":"2024-03-29T10:12:46.845468+0000","last_active":"2024-03-29T10:12:46.982581+0000","last_peered":"2024-03-29T10:12:46.982581+0000","last_clean":"2024-03-29T10:12:46.982581+0000","last_became_active":"2024-03-29T10:12:46.845175+0000","last_became_peered":"2024-03-29T10:12:46.845175+0000","last_unstale":"2024-03-29T10:12:46.982581+0000","last_undegraded":"2024-03-29T10:12:46.982581+0000","last_fullsized":"2024-03-29T10:12:46.982581+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:56.220192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.842206+0000","last_change":"2024-03-29T10:12:46.839048+0000","last_active":"2024-03-29T10:12:46.842206+0000","last_peered":"2024-03-29T10:12:46.842206+0000","last_clean":"2024-03-29T10:12:46.842206+0000","last_became_active":"2024-03-29T10:12:46.838084+0000","last_became_peered":"2024-03-29T10:12:46.838084+0000","last_unstale":"2024-03-29T10:12:46.842206+0000","last_undegraded":"2024-03-29T10:12:46.842206+0000","last_fullsized":"2024-03-29T10:12:46.842206+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:43.403428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981056+0000","last_change":"2024-03-29T10:12:46.839770+0000","last_active":"2024-03-29T10:12:46.981056+0000","last_peered":"2024-03-29T10:12:46.981056+0000","last_clean":"2024-03-29T10:12:46.981056+0000","last_became_active":"2024-03-29T10:12:46.839511+0000","last_became_peered":"2024-03-29T10:12:46.839511+0000","last_unstale":"2024-03-29T10:12:46.981056+0000","last_undegraded":"2024-03-29T10:12:46.981056+0000","last_fullsized":"2024-03-29T10:12:46.981056+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:04.158919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978861+0000","last_change":"2024-03-29T10:12:46.841318+0000","last_active":"2024-03-29T10:12:46.978861+0000","last_peered":"2024-03-29T10:12:46.978861+0000","last_clean":"2024-03-29T10:12:46.978861+0000","last_became_active":"2024-03-29T10:12:46.841085+0000","last_became_peered":"2024-03-29T10:12:46.841085+0000","last_unstale":"2024-03-29T10:12:46.978861+0000","last_undegraded":"2024-03-29T10:12:46.978861+0000","last_fullsized":"2024-03-29T10:12:46.978861+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:05:55.909595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.979737+0000","last_change":"2024-03-29T10:12:46.842050+0000","last_active":"2024-03-29T10:12:46.979737+0000","last_peered":"2024-03-29T10:12:46.979737+0000","last_clean":"2024-03-29T10:12:46.979737+0000","last_became_active":"2024-03-29T10:12:46.835709+0000","last_became_peered":"2024-03-29T10:12:46.835709+0000","last_unstale":"2024-03-29T10:12:46.979737+0000","last_undegraded":"2024-03-29T10:12:46.979737+0000","last_fullsized":"2024-03-29T10:12:46.979737+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:45.331011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976721+0000","last_change":"2024-03-29T10:12:46.834325+0000","last_active":"2024-03-29T10:12:46.976721+0000","last_peered":"2024-03-29T10:12:46.976721+0000","last_clean":"2024-03-29T10:12:46.976721+0000","last_became_active":"2024-03-29T10:12:46.833484+0000","last_became_peered":"2024-03-29T10:12:46.833484+0000","last_unstale":"2024-03-29T10:12:46.976721+0000","last_undegraded":"2024-03-29T10:12:46.976721+0000","last_fullsized":"2024-03-29T10:12:46.976721+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:30:12.205591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978949+0000","last_change":"2024-03-29T10:12:46.845641+0000","last_active":"2024-03-29T10:12:46.978949+0000","last_peered":"2024-03-29T10:12:46.978949+0000","last_clean":"2024-03-29T10:12:46.978949+0000","last_became_active":"2024-03-29T10:12:46.844642+0000","last_became_peered":"2024-03-29T10:12:46.844642+0000","last_unstale":"2024-03-29T10:12:46.978949+0000","last_undegraded":"2024-03-29T10:12:46.978949+0000","last_fullsized":"2024-03-29T10:12:46.978949+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:29:44.813491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"123'4","reported_seq":15,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981303+0000","last_change":"2024-03-29T10:12:46.312221+0000","last_active":"2024-03-29T10:12:46.981303+0000","last_peered":"2024-03-29T10:12:46.981303+0000","last_clean":"2024-03-29T10:12:46.981303+0000","last_became_active":"2024-03-29T10:12:46.311948+0000","last_became_peered":"2024-03-29T10:12:46.311948+0000","last_unstale":"2024-03-29T10:12:46.981303+0000","last_undegraded":"2024-03-29T10:12:46.981303+0000","last_fullsized":"2024-03-29T10:12:46.981303+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:12.339045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.979808+0000","last_change":"2024-03-29T10:12:46.842054+0000","last_active":"2024-03-29T10:12:46.979808+0000","last_peered":"2024-03-29T10:12:46.979808+0000","last_clean":"2024-03-29T10:12:46.979808+0000","last_became_active":"2024-03-29T10:12:46.841790+0000","last_became_peered":"2024-03-29T10:12:46.841790+0000","last_unstale":"2024-03-29T10:12:46.979808+0000","last_undegraded":"2024-03-29T10:12:46.979808+0000","last_fullsized":"2024-03-29T10:12:46.979808+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:56.220192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"123'1","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982636+0000","last_change":"2024-03-29T10:12:46.317237+0000","last_active":"2024-03-29T10:12:46.982636+0000","last_peered":"2024-03-29T10:12:46.982636+0000","last_clean":"2024-03-29T10:12:46.982636+0000","last_became_active":"2024-03-29T10:12:46.316931+0000","last_became_peered":"2024-03-29T10:12:46.316931+0000","last_unstale":"2024-03-29T10:12:46.982636+0000","last_undegraded":"2024-03-29T10:12:46.982636+0000","last_fullsized":"2024-03-29T10:12:46.982636+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:07.770121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976783+0000","last_change":"2024-03-29T10:12:46.826174+0000","last_active":"2024-03-29T10:12:46.976783+0000","last_peered":"2024-03-29T10:12:46.976783+0000","last_clean":"2024-03-29T10:12:46.976783+0000","last_became_active":"2024-03-29T10:12:46.825829+0000","last_became_peered":"2024-03-29T10:12:46.825829+0000","last_unstale":"2024-03-29T10:12:46.976783+0000","last_undegraded":"2024-03-29T10:12:46.976783+0000","last_fullsized":"2024-03-29T10:12:46.976783+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:29:38.657726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"123'2","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981157+0000","last_change":"2024-03-29T10:12:46.319156+0000","last_active":"2024-03-29T10:12:46.981157+0000","last_peered":"2024-03-29T10:12:46.981157+0000","last_clean":"2024-03-29T10:12:46.981157+0000","last_became_active":"2024-03-29T10:12:46.318764+0000","last_became_peered":"2024-03-29T10:12:46.318764+0000","last_unstale":"2024-03-29T10:12:46.981157+0000","last_undegraded":"2024-03-29T10:12:46.981157+0000","last_fullsized":"2024-03-29T10:12:46.981157+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:40:45.994260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.841811+0000","last_change":"2024-03-29T10:12:46.836926+0000","last_active":"2024-03-29T10:12:46.841811+0000","last_peered":"2024-03-29T10:12:46.841811+0000","last_clean":"2024-03-29T10:12:46.841811+0000","last_became_active":"2024-03-29T10:12:46.832584+0000","last_became_peered":"2024-03-29T10:12:46.832584+0000","last_unstale":"2024-03-29T10:12:46.841811+0000","last_undegraded":"2024-03-29T10:12:46.841811+0000","last_fullsized":"2024-03-29T10:12:46.841811+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:13.348427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"123'2","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981114+0000","last_change":"2024-03-29T10:12:46.312199+0000","last_active":"2024-03-29T10:12:46.981114+0000","last_peered":"2024-03-29T10:12:46.981114+0000","last_clean":"2024-03-29T10:12:46.981114+0000","last_became_active":"2024-03-29T10:12:46.311547+0000","last_became_peered":"2024-03-29T10:12:46.311547+0000","last_unstale":"2024-03-29T10:12:46.981114+0000","last_undegraded":"2024-03-29T10:12:46.981114+0000","last_fullsized":"2024-03-29T10:12:46.981114+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:55.323587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.836417+0000","last_change":"2024-03-29T10:12:46.831278+0000","last_active":"2024-03-29T10:12:46.836417+0000","last_peered":"2024-03-29T10:12:46.836417+0000","last_clean":"2024-03-29T10:12:46.836417+0000","last_became_active":"2024-03-29T10:12:46.830914+0000","last_became_peered":"2024-03-29T10:12:46.830914+0000","last_unstale":"2024-03-29T10:12:46.836417+0000","last_undegraded":"2024-03-29T10:12:46.836417+0000","last_fullsized":"2024-03-29T10:12:46.836417+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:29:44.813491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"123'1","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976773+0000","last_change":"2024-03-29T10:12:46.301452+0000","last_active":"2024-03-29T10:12:46.976773+0000","last_peered":"2024-03-29T10:12:46.976773+0000","last_clean":"2024-03-29T10:12:46.976773+0000","last_became_active":"2024-03-29T10:12:46.300949+0000","last_became_peered":"2024-03-29T10:12:46.300949+0000","last_unstale":"2024-03-29T10:12:46.976773+0000","last_undegraded":"2024-03-29T10:12:46.976773+0000","last_fullsized":"2024-03-29T10:12:46.976773+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:46:54.186603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982571+0000","last_change":"2024-03-29T10:12:46.841913+0000","last_active":"2024-03-29T10:12:46.982571+0000","last_peered":"2024-03-29T10:12:46.982571+0000","last_clean":"2024-03-29T10:12:46.982571+0000","last_became_active":"2024-03-29T10:12:46.841068+0000","last_became_peered":"2024-03-29T10:12:46.841068+0000","last_unstale":"2024-03-29T10:12:46.982571+0000","last_undegraded":"2024-03-29T10:12:46.982571+0000","last_fullsized":"2024-03-29T10:12:46.982571+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:50.693333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"123'1","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981037+0000","last_change":"2024-03-29T10:12:46.317421+0000","last_active":"2024-03-29T10:12:46.981037+0000","last_peered":"2024-03-29T10:12:46.981037+0000","last_clean":"2024-03-29T10:12:46.981037+0000","last_became_active":"2024-03-29T10:12:46.314110+0000","last_became_peered":"2024-03-29T10:12:46.314110+0000","last_unstale":"2024-03-29T10:12:46.981037+0000","last_undegraded":"2024-03-29T10:12:46.981037+0000","last_fullsized":"2024-03-29T10:12:46.981037+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:16:08.320929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.979696+0000","last_change":"2024-03-29T10:12:46.849430+0000","last_active":"2024-03-29T10:12:46.979696+0000","last_peered":"2024-03-29T10:12:46.979696+0000","last_clean":"2024-03-29T10:12:46.979696+0000","last_became_active":"2024-03-29T10:12:46.849158+0000","last_became_peered":"2024-03-29T10:12:46.849158+0000","last_unstale":"2024-03-29T10:12:46.979696+0000","last_undegraded":"2024-03-29T10:12:46.979696+0000","last_fullsized":"2024-03-29T10:12:46.979696+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:54.751799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.820074+0000","last_change":"2024-03-29T10:12:46.320446+0000","last_active":"2024-03-29T10:12:46.820074+0000","last_peered":"2024-03-29T10:12:46.820074+0000","last_clean":"2024-03-29T10:12:46.820074+0000","last_became_active":"2024-03-29T10:12:46.320149+0000","last_became_peered":"2024-03-29T10:12:46.320149+0000","last_unstale":"2024-03-29T10:12:46.820074+0000","last_undegraded":"2024-03-29T10:12:46.820074+0000","last_fullsized":"2024-03-29T10:12:46.820074+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:18.516455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.833802+0000","last_change":"2024-03-29T10:12:46.827124+0000","last_active":"2024-03-29T10:12:46.833802+0000","last_peered":"2024-03-29T10:12:46.833802+0000","last_clean":"2024-03-29T10:12:46.833802+0000","last_became_active":"2024-03-29T10:12:46.826802+0000","last_became_peered":"2024-03-29T10:12:46.826802+0000","last_unstale":"2024-03-29T10:12:46.833802+0000","last_undegraded":"2024-03-29T10:12:46.833802+0000","last_fullsized":"2024-03-29T10:12:46.833802+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:16:09.330311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976735+0000","last_change":"2024-03-29T10:12:46.312469+0000","last_active":"2024-03-29T10:12:46.976735+0000","last_peered":"2024-03-29T10:12:46.976735+0000","last_clean":"2024-03-29T10:12:46.976735+0000","last_became_active":"2024-03-29T10:12:46.312178+0000","last_became_peered":"2024-03-29T10:12:46.312178+0000","last_unstale":"2024-03-29T10:12:46.976735+0000","last_undegraded":"2024-03-29T10:12:46.976735+0000","last_fullsized":"2024-03-29T10:12:46.976735+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:46:28.171211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981236+0000","last_change":"2024-03-29T10:12:46.840355+0000","last_active":"2024-03-29T10:12:46.981236+0000","last_peered":"2024-03-29T10:12:46.981236+0000","last_clean":"2024-03-29T10:12:46.981236+0000","last_became_active":"2024-03-29T10:12:46.840098+0000","last_became_peered":"2024-03-29T10:12:46.840098+0000","last_unstale":"2024-03-29T10:12:46.981236+0000","last_undegraded":"2024-03-29T10:12:46.981236+0000","last_fullsized":"2024-03-29T10:12:46.981236+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:56:37.402871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976833+0000","last_change":"2024-03-29T10:12:46.302880+0000","last_active":"2024-03-29T10:12:46.976833+0000","last_peered":"2024-03-29T10:12:46.976833+0000","last_clean":"2024-03-29T10:12:46.976833+0000","last_became_active":"2024-03-29T10:12:46.302582+0000","last_became_peered":"2024-03-29T10:12:46.302582+0000","last_unstale":"2024-03-29T10:12:46.976833+0000","last_undegraded":"2024-03-29T10:12:46.976833+0000","last_fullsized":"2024-03-29T10:12:46.976833+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:34:35.424203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.833875+0000","last_change":"2024-03-29T10:12:46.827498+0000","last_active":"2024-03-29T10:12:46.833875+0000","last_peered":"2024-03-29T10:12:46.833875+0000","last_clean":"2024-03-29T10:12:46.833875+0000","last_became_active":"2024-03-29T10:12:46.826934+0000","last_became_peered":"2024-03-29T10:12:46.826934+0000","last_unstale":"2024-03-29T10:12:46.833875+0000","last_undegraded":"2024-03-29T10:12:46.833875+0000","last_fullsized":"2024-03-29T10:12:46.833875+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:43:49.789402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.979798+0000","last_change":"2024-03-29T10:12:46.312627+0000","last_active":"2024-03-29T10:12:46.979798+0000","last_peered":"2024-03-29T10:12:46.979798+0000","last_clean":"2024-03-29T10:12:46.979798+0000","last_became_active":"2024-03-29T10:12:46.312376+0000","last_became_peered":"2024-03-29T10:12:46.312376+0000","last_unstale":"2024-03-29T10:12:46.979798+0000","last_undegraded":"2024-03-29T10:12:46.979798+0000","last_fullsized":"2024-03-29T10:12:46.979798+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:34:30.646213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.839982+0000","last_change":"2024-03-29T10:12:46.833312+0000","last_active":"2024-03-29T10:12:46.839982+0000","last_peered":"2024-03-29T10:12:46.839982+0000","last_clean":"2024-03-29T10:12:46.839982+0000","last_became_active":"2024-03-29T10:12:46.832917+0000","last_became_peered":"2024-03-29T10:12:46.832917+0000","last_unstale":"2024-03-29T10:12:46.839982+0000","last_undegraded":"2024-03-29T10:12:46.839982+0000","last_fullsized":"2024-03-29T10:12:46.839982+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:53.536977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"123'3","reported_seq":14,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982622+0000","last_change":"2024-03-29T10:12:46.314591+0000","last_active":"2024-03-29T10:12:46.982622+0000","last_peered":"2024-03-29T10:12:46.982622+0000","last_clean":"2024-03-29T10:12:46.982622+0000","last_became_active":"2024-03-29T10:12:46.314184+0000","last_became_peered":"2024-03-29T10:12:46.314184+0000","last_unstale":"2024-03-29T10:12:46.982622+0000","last_undegraded":"2024-03-29T10:12:46.982622+0000","last_fullsized":"2024-03-29T10:12:46.982622+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:15.808972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982591+0000","last_change":"2024-03-29T10:12:46.842734+0000","last_active":"2024-03-29T10:12:46.982591+0000","last_peered":"2024-03-29T10:12:46.982591+0000","last_clean":"2024-03-29T10:12:46.982591+0000","last_became_active":"2024-03-29T10:12:46.841788+0000","last_became_peered":"2024-03-29T10:12:46.841788+0000","last_unstale":"2024-03-29T10:12:46.982591+0000","last_undegraded":"2024-03-29T10:12:46.982591+0000","last_fullsized":"2024-03-29T10:12:46.982591+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:06.962212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"123'6","reported_seq":16,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.852358+0000","last_change":"2024-03-29T10:12:46.311128+0000","last_active":"2024-03-29T10:12:46.852358+0000","last_peered":"2024-03-29T10:12:46.852358+0000","last_clean":"2024-03-29T10:12:46.852358+0000","last_became_active":"2024-03-29T10:12:46.310825+0000","last_became_peered":"2024-03-29T10:12:46.310825+0000","last_unstale":"2024-03-29T10:12:46.852358+0000","last_undegraded":"2024-03-29T10:12:46.852358+0000","last_fullsized":"2024-03-29T10:12:46.852358+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:10.671851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.979761+0000","last_change":"2024-03-29T10:12:46.837420+0000","last_active":"2024-03-29T10:12:46.979761+0000","last_peered":"2024-03-29T10:12:46.979761+0000","last_clean":"2024-03-29T10:12:46.979761+0000","last_became_active":"2024-03-29T10:12:46.837167+0000","last_became_peered":"2024-03-29T10:12:46.837167+0000","last_unstale":"2024-03-29T10:12:46.979761+0000","last_undegraded":"2024-03-29T10:12:46.979761+0000","last_fullsized":"2024-03-29T10:12:46.979761+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:58:37.690997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"123'2","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978931+0000","last_change":"2024-03-29T10:12:46.305485+0000","last_active":"2024-03-29T10:12:46.978931+0000","last_peered":"2024-03-29T10:12:46.978931+0000","last_clean":"2024-03-29T10:12:46.978931+0000","last_became_active":"2024-03-29T10:12:46.305135+0000","last_became_peered":"2024-03-29T10:12:46.305135+0000","last_unstale":"2024-03-29T10:12:46.978931+0000","last_undegraded":"2024-03-29T10:12:46.978931+0000","last_fullsized":"2024-03-29T10:12:46.978931+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:51:14.288869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976763+0000","last_change":"2024-03-29T10:12:46.834105+0000","last_active":"2024-03-29T10:12:46.976763+0000","last_peered":"2024-03-29T10:12:46.976763+0000","last_clean":"2024-03-29T10:12:46.976763+0000","last_became_active":"2024-03-29T10:12:46.833832+0000","last_became_peered":"2024-03-29T10:12:46.833832+0000","last_unstale":"2024-03-29T10:12:46.976763+0000","last_undegraded":"2024-03-29T10:12:46.976763+0000","last_fullsized":"2024-03-29T10:12:46.976763+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:13.633043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978900+0000","last_change":"2024-03-29T10:12:46.309019+0000","last_active":"2024-03-29T10:12:46.978900+0000","last_peered":"2024-03-29T10:12:46.978900+0000","last_clean":"2024-03-29T10:12:46.978900+0000","last_became_active":"2024-03-29T10:12:46.308043+0000","last_became_peered":"2024-03-29T10:12:46.308043+0000","last_unstale":"2024-03-29T10:12:46.978900+0000","last_undegraded":"2024-03-29T10:12:46.978900+0000","last_fullsized":"2024-03-29T10:12:46.978900+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:54.107733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.840895+0000","last_change":"2024-03-29T10:12:46.836008+0000","last_active":"2024-03-29T10:12:46.840895+0000","last_peered":"2024-03-29T10:12:46.840895+0000","last_clean":"2024-03-29T10:12:46.840895+0000","last_became_active":"2024-03-29T10:12:46.835699+0000","last_became_peered":"2024-03-29T10:12:46.835699+0000","last_unstale":"2024-03-29T10:12:46.840895+0000","last_undegraded":"2024-03-29T10:12:46.840895+0000","last_fullsized":"2024-03-29T10:12:46.840895+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:16.818354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.818156+0000","last_change":"2024-03-29T10:12:46.304474+0000","last_active":"2024-03-29T10:12:46.818156+0000","last_peered":"2024-03-29T10:12:46.818156+0000","last_clean":"2024-03-29T10:12:46.818156+0000","last_became_active":"2024-03-29T10:12:46.304207+0000","last_became_peered":"2024-03-29T10:12:46.304207+0000","last_unstale":"2024-03-29T10:12:46.818156+0000","last_undegraded":"2024-03-29T10:12:46.818156+0000","last_fullsized":"2024-03-29T10:12:46.818156+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:18.516455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978840+0000","last_change":"2024-03-29T10:12:46.840792+0000","last_active":"2024-03-29T10:12:46.978840+0000","last_peered":"2024-03-29T10:12:46.978840+0000","last_clean":"2024-03-29T10:12:46.978840+0000","last_became_active":"2024-03-29T10:12:46.840497+0000","last_became_peered":"2024-03-29T10:12:46.840497+0000","last_unstale":"2024-03-29T10:12:46.978840+0000","last_undegraded":"2024-03-29T10:12:46.978840+0000","last_fullsized":"2024-03-29T10:12:46.978840+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:51.994336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"123'8","reported_seq":19,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976747+0000","last_change":"2024-03-29T10:12:46.313806+0000","last_active":"2024-03-29T10:12:46.976747+0000","last_peered":"2024-03-29T10:12:46.976747+0000","last_clean":"2024-03-29T10:12:46.976747+0000","last_became_active":"2024-03-29T10:12:46.313540+0000","last_became_peered":"2024-03-29T10:12:46.313540+0000","last_unstale":"2024-03-29T10:12:46.976747+0000","last_undegraded":"2024-03-29T10:12:46.976747+0000","last_fullsized":"2024-03-29T10:12:46.976747+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:10:18.737044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.832425+0000","last_change":"2024-03-29T10:12:46.825422+0000","last_active":"2024-03-29T10:12:46.832425+0000","last_peered":"2024-03-29T10:12:46.832425+0000","last_clean":"2024-03-29T10:12:46.832425+0000","last_became_active":"2024-03-29T10:12:46.825098+0000","last_became_peered":"2024-03-29T10:12:46.825098+0000","last_unstale":"2024-03-29T10:12:46.832425+0000","last_undegraded":"2024-03-29T10:12:46.832425+0000","last_fullsized":"2024-03-29T10:12:46.832425+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:56.332969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"123'2","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.851382+0000","last_change":"2024-03-29T10:12:46.307543+0000","last_active":"2024-03-29T10:12:46.851382+0000","last_peered":"2024-03-29T10:12:46.851382+0000","last_clean":"2024-03-29T10:12:46.851382+0000","last_became_active":"2024-03-29T10:12:46.307269+0000","last_became_peered":"2024-03-29T10:12:46.307269+0000","last_unstale":"2024-03-29T10:12:46.851382+0000","last_undegraded":"2024-03-29T10:12:46.851382+0000","last_fullsized":"2024-03-29T10:12:46.851382+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:34:20.459712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.839148+0000","last_change":"2024-03-29T10:12:46.834470+0000","last_active":"2024-03-29T10:12:46.839148+0000","last_peered":"2024-03-29T10:12:46.839148+0000","last_clean":"2024-03-29T10:12:46.839148+0000","last_became_active":"2024-03-29T10:12:46.834169+0000","last_became_peered":"2024-03-29T10:12:46.834169+0000","last_unstale":"2024-03-29T10:12:46.839148+0000","last_undegraded":"2024-03-29T10:12:46.839148+0000","last_fullsized":"2024-03-29T10:12:46.839148+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:51.994336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"123'1","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.841619+0000","last_change":"2024-03-29T10:12:46.304283+0000","last_active":"2024-03-29T10:12:46.841619+0000","last_peered":"2024-03-29T10:12:46.841619+0000","last_clean":"2024-03-29T10:12:46.841619+0000","last_became_active":"2024-03-29T10:12:46.302450+0000","last_became_peered":"2024-03-29T10:12:46.302450+0000","last_unstale":"2024-03-29T10:12:46.841619+0000","last_undegraded":"2024-03-29T10:12:46.841619+0000","last_fullsized":"2024-03-29T10:12:46.841619+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:12:49.476058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982609+0000","last_change":"2024-03-29T10:12:46.843073+0000","last_active":"2024-03-29T10:12:46.982609+0000","last_peered":"2024-03-29T10:12:46.982609+0000","last_clean":"2024-03-29T10:12:46.982609+0000","last_became_active":"2024-03-29T10:12:46.842479+0000","last_became_peered":"2024-03-29T10:12:46.842479+0000","last_unstale":"2024-03-29T10:12:46.982609+0000","last_undegraded":"2024-03-29T10:12:46.982609+0000","last_fullsized":"2024-03-29T10:12:46.982609+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:43.403428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981133+0000","last_change":"2024-03-29T10:12:46.312388+0000","last_active":"2024-03-29T10:12:46.981133+0000","last_peered":"2024-03-29T10:12:46.981133+0000","last_clean":"2024-03-29T10:12:46.981133+0000","last_became_active":"2024-03-29T10:12:46.311924+0000","last_became_peered":"2024-03-29T10:12:46.311924+0000","last_unstale":"2024-03-29T10:12:46.981133+0000","last_undegraded":"2024-03-29T10:12:46.981133+0000","last_fullsized":"2024-03-29T10:12:46.981133+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:42.861576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978880+0000","last_change":"2024-03-29T10:12:46.828829+0000","last_active":"2024-03-29T10:12:46.978880+0000","last_peered":"2024-03-29T10:12:46.978880+0000","last_clean":"2024-03-29T10:12:46.978880+0000","last_became_active":"2024-03-29T10:12:46.828570+0000","last_became_peered":"2024-03-29T10:12:46.828570+0000","last_unstale":"2024-03-29T10:12:46.978880+0000","last_undegraded":"2024-03-29T10:12:46.978880+0000","last_fullsized":"2024-03-29T10:12:46.978880+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:27:46.061296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"123'2","reported_seq":13,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981078+0000","last_change":"2024-03-29T10:12:46.320441+0000","last_active":"2024-03-29T10:12:46.981078+0000","last_peered":"2024-03-29T10:12:46.981078+0000","last_clean":"2024-03-29T10:12:46.981078+0000","last_became_active":"2024-03-29T10:12:46.319864+0000","last_became_peered":"2024-03-29T10:12:46.319864+0000","last_unstale":"2024-03-29T10:12:46.981078+0000","last_undegraded":"2024-03-29T10:12:46.981078+0000","last_fullsized":"2024-03-29T10:12:46.981078+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:15.808972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978909+0000","last_change":"2024-03-29T10:12:46.834833+0000","last_active":"2024-03-29T10:12:46.978909+0000","last_peered":"2024-03-29T10:12:46.978909+0000","last_clean":"2024-03-29T10:12:46.978909+0000","last_became_active":"2024-03-29T10:12:46.834327+0000","last_became_peered":"2024-03-29T10:12:46.834327+0000","last_unstale":"2024-03-29T10:12:46.978909+0000","last_undegraded":"2024-03-29T10:12:46.978909+0000","last_fullsized":"2024-03-29T10:12:46.978909+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:18:02.845983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.825106+0000","last_change":"2024-03-29T10:12:46.307976+0000","last_active":"2024-03-29T10:12:46.825106+0000","last_peered":"2024-03-29T10:12:46.825106+0000","last_clean":"2024-03-29T10:12:46.825106+0000","last_became_active":"2024-03-29T10:12:46.307713+0000","last_became_peered":"2024-03-29T10:12:46.307713+0000","last_unstale":"2024-03-29T10:12:46.825106+0000","last_undegraded":"2024-03-29T10:12:46.825106+0000","last_fullsized":"2024-03-29T10:12:46.825106+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:28:50.912139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978819+0000","last_change":"2024-03-29T10:12:46.842521+0000","last_active":"2024-03-29T10:12:46.978819+0000","last_peered":"2024-03-29T10:12:46.978819+0000","last_clean":"2024-03-29T10:12:46.978819+0000","last_became_active":"2024-03-29T10:12:46.842222+0000","last_became_peered":"2024-03-29T10:12:46.842222+0000","last_unstale":"2024-03-29T10:12:46.978819+0000","last_undegraded":"2024-03-29T10:12:46.978819+0000","last_fullsized":"2024-03-29T10:12:46.978819+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:53.931903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.825004+0000","last_change":"2024-03-29T10:12:46.312741+0000","last_active":"2024-03-29T10:12:46.825004+0000","last_peered":"2024-03-29T10:12:46.825004+0000","last_clean":"2024-03-29T10:12:46.825004+0000","last_became_active":"2024-03-29T10:12:46.312336+0000","last_became_peered":"2024-03-29T10:12:46.312336+0000","last_unstale":"2024-03-29T10:12:46.825004+0000","last_undegraded":"2024-03-29T10:12:46.825004+0000","last_fullsized":"2024-03-29T10:12:46.825004+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:50:49.290619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.841096+0000","last_change":"2024-03-29T10:12:46.835696+0000","last_active":"2024-03-29T10:12:46.841096+0000","last_peered":"2024-03-29T10:12:46.841096+0000","last_clean":"2024-03-29T10:12:46.841096+0000","last_became_active":"2024-03-29T10:12:46.835435+0000","last_became_peered":"2024-03-29T10:12:46.835435+0000","last_unstale":"2024-03-29T10:12:46.841096+0000","last_undegraded":"2024-03-29T10:12:46.841096+0000","last_fullsized":"2024-03-29T10:12:46.841096+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:13.348427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.980984+0000","last_change":"2024-03-29T10:12:46.302806+0000","last_active":"2024-03-29T10:12:46.980984+0000","last_peered":"2024-03-29T10:12:46.980984+0000","last_clean":"2024-03-29T10:12:46.980984+0000","last_became_active":"2024-03-29T10:12:46.302462+0000","last_became_peered":"2024-03-29T10:12:46.302462+0000","last_unstale":"2024-03-29T10:12:46.980984+0000","last_undegraded":"2024-03-29T10:12:46.980984+0000","last_fullsized":"2024-03-29T10:12:46.980984+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:49.683950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981252+0000","last_change":"2024-03-29T10:12:46.831841+0000","last_active":"2024-03-29T10:12:46.981252+0000","last_peered":"2024-03-29T10:12:46.981252+0000","last_clean":"2024-03-29T10:12:46.981252+0000","last_became_active":"2024-03-29T10:12:46.830791+0000","last_became_peered":"2024-03-29T10:12:46.830791+0000","last_unstale":"2024-03-29T10:12:46.981252+0000","last_undegraded":"2024-03-29T10:12:46.981252+0000","last_fullsized":"2024-03-29T10:12:46.981252+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:59.250713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"123'2","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.847313+0000","last_change":"2024-03-29T10:12:46.304763+0000","last_active":"2024-03-29T10:12:46.847313+0000","last_peered":"2024-03-29T10:12:46.847313+0000","last_clean":"2024-03-29T10:12:46.847313+0000","last_became_active":"2024-03-29T10:12:46.304433+0000","last_became_peered":"2024-03-29T10:12:46.304433+0000","last_unstale":"2024-03-29T10:12:46.847313+0000","last_undegraded":"2024-03-29T10:12:46.847313+0000","last_fullsized":"2024-03-29T10:12:46.847313+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:51.190947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976824+0000","last_change":"2024-03-29T10:12:46.837100+0000","last_active":"2024-03-29T10:12:46.976824+0000","last_peered":"2024-03-29T10:12:46.976824+0000","last_clean":"2024-03-29T10:12:46.976824+0000","last_became_active":"2024-03-29T10:12:46.836793+0000","last_became_peered":"2024-03-29T10:12:46.836793+0000","last_unstale":"2024-03-29T10:12:46.976824+0000","last_undegraded":"2024-03-29T10:12:46.976824+0000","last_fullsized":"2024-03-29T10:12:46.976824+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:28:51.921521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"124'1","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-29T10:12:47.847525+0000","last_change":"2024-03-29T10:12:46.319070+0000","last_active":"2024-03-29T10:12:47.847525+0000","last_peered":"2024-03-29T10:12:47.847525+0000","last_clean":"2024-03-29T10:12:47.847525+0000","last_became_active":"2024-03-29T10:12:46.318749+0000","last_became_peered":"2024-03-29T10:12:46.318749+0000","last_unstale":"2024-03-29T10:12:47.847525+0000","last_undegraded":"2024-03-29T10:12:47.847525+0000","last_fullsized":"2024-03-29T10:12:47.847525+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:21:22.332793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.976793+0000","last_change":"2024-03-29T10:12:46.830088+0000","last_active":"2024-03-29T10:12:46.976793+0000","last_peered":"2024-03-29T10:12:46.976793+0000","last_clean":"2024-03-29T10:12:46.976793+0000","last_became_active":"2024-03-29T10:12:46.829791+0000","last_became_peered":"2024-03-29T10:12:46.829791+0000","last_unstale":"2024-03-29T10:12:46.976793+0000","last_undegraded":"2024-03-29T10:12:46.976793+0000","last_fullsized":"2024-03-29T10:12:46.976793+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:51:43.149222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981180+0000","last_change":"2024-03-29T10:12:46.308003+0000","last_active":"2024-03-29T10:12:46.981180+0000","last_peered":"2024-03-29T10:12:46.981180+0000","last_clean":"2024-03-29T10:12:46.981180+0000","last_became_active":"2024-03-29T10:12:46.306971+0000","last_became_peered":"2024-03-29T10:12:46.306971+0000","last_unstale":"2024-03-29T10:12:46.981180+0000","last_undegraded":"2024-03-29T10:12:46.981180+0000","last_fullsized":"2024-03-29T10:12:46.981180+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:15.945108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.837335+0000","last_change":"2024-03-29T10:12:46.828528+0000","last_active":"2024-03-29T10:12:46.837335+0000","last_peered":"2024-03-29T10:12:46.837335+0000","last_clean":"2024-03-29T10:12:46.837335+0000","last_became_active":"2024-03-29T10:12:46.828249+0000","last_became_peered":"2024-03-29T10:12:46.828249+0000","last_unstale":"2024-03-29T10:12:46.837335+0000","last_undegraded":"2024-03-29T10:12:46.837335+0000","last_fullsized":"2024-03-29T10:12:46.837335+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:16:09.330311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.821996+0000","last_change":"2024-03-29T10:12:46.306751+0000","last_active":"2024-03-29T10:12:46.821996+0000","last_peered":"2024-03-29T10:12:46.821996+0000","last_clean":"2024-03-29T10:12:46.821996+0000","last_became_active":"2024-03-29T10:12:46.306387+0000","last_became_peered":"2024-03-29T10:12:46.306387+0000","last_unstale":"2024-03-29T10:12:46.821996+0000","last_undegraded":"2024-03-29T10:12:46.821996+0000","last_fullsized":"2024-03-29T10:12:46.821996+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:10.671851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978890+0000","last_change":"2024-03-29T10:12:46.834284+0000","last_active":"2024-03-29T10:12:46.978890+0000","last_peered":"2024-03-29T10:12:46.978890+0000","last_clean":"2024-03-29T10:12:46.978890+0000","last_became_active":"2024-03-29T10:12:46.834021+0000","last_became_peered":"2024-03-29T10:12:46.834021+0000","last_unstale":"2024-03-29T10:12:46.978890+0000","last_undegraded":"2024-03-29T10:12:46.978890+0000","last_fullsized":"2024-03-29T10:12:46.978890+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:37:29.153650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.981097+0000","last_change":"2024-03-29T10:12:46.312874+0000","last_active":"2024-03-29T10:12:46.981097+0000","last_peered":"2024-03-29T10:12:46.981097+0000","last_clean":"2024-03-29T10:12:46.981097+0000","last_became_active":"2024-03-29T10:12:46.312528+0000","last_became_peered":"2024-03-29T10:12:46.312528+0000","last_unstale":"2024-03-29T10:12:46.981097+0000","last_undegraded":"2024-03-29T10:12:46.981097+0000","last_fullsized":"2024-03-29T10:12:46.981097+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:43:48.780020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.833973+0000","last_change":"2024-03-29T10:12:46.827416+0000","last_active":"2024-03-29T10:12:46.833973+0000","last_peered":"2024-03-29T10:12:46.833973+0000","last_clean":"2024-03-29T10:12:46.833973+0000","last_became_active":"2024-03-29T10:12:46.827020+0000","last_became_peered":"2024-03-29T10:12:46.827020+0000","last_unstale":"2024-03-29T10:12:46.833973+0000","last_undegraded":"2024-03-29T10:12:46.833973+0000","last_fullsized":"2024-03-29T10:12:46.833973+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:43.870958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.818587+0000","last_change":"2024-03-29T10:12:46.305217+0000","last_active":"2024-03-29T10:12:46.818587+0000","last_peered":"2024-03-29T10:12:46.818587+0000","last_clean":"2024-03-29T10:12:46.818587+0000","last_became_active":"2024-03-29T10:12:46.304901+0000","last_became_peered":"2024-03-29T10:12:46.304901+0000","last_unstale":"2024-03-29T10:12:46.818587+0000","last_undegraded":"2024-03-29T10:12:46.818587+0000","last_fullsized":"2024-03-29T10:12:46.818587+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:18:01.836601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978806+0000","last_change":"2024-03-29T10:12:46.835369+0000","last_active":"2024-03-29T10:12:46.978806+0000","last_peered":"2024-03-29T10:12:46.978806+0000","last_clean":"2024-03-29T10:12:46.978806+0000","last_became_active":"2024-03-29T10:12:46.835124+0000","last_became_peered":"2024-03-29T10:12:46.835124+0000","last_unstale":"2024-03-29T10:12:46.978806+0000","last_undegraded":"2024-03-29T10:12:46.978806+0000","last_fullsized":"2024-03-29T10:12:46.978806+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:03:27.472539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.979748+0000","last_change":"2024-03-29T10:12:46.310673+0000","last_active":"2024-03-29T10:12:46.979748+0000","last_peered":"2024-03-29T10:12:46.979748+0000","last_clean":"2024-03-29T10:12:46.979748+0000","last_became_active":"2024-03-29T10:12:46.310391+0000","last_became_peered":"2024-03-29T10:12:46.310391+0000","last_unstale":"2024-03-29T10:12:46.979748+0000","last_undegraded":"2024-03-29T10:12:46.979748+0000","last_fullsized":"2024-03-29T10:12:46.979748+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:15.808972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.837369+0000","last_change":"2024-03-29T10:12:46.830458+0000","last_active":"2024-03-29T10:12:46.837369+0000","last_peered":"2024-03-29T10:12:46.837369+0000","last_clean":"2024-03-29T10:12:46.837369+0000","last_became_active":"2024-03-29T10:12:46.830106+0000","last_became_peered":"2024-03-29T10:12:46.830106+0000","last_unstale":"2024-03-29T10:12:46.837369+0000","last_undegraded":"2024-03-29T10:12:46.837369+0000","last_fullsized":"2024-03-29T10:12:46.837369+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:45.331011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978828+0000","last_change":"2024-03-29T10:12:46.312698+0000","last_active":"2024-03-29T10:12:46.978828+0000","last_peered":"2024-03-29T10:12:46.978828+0000","last_clean":"2024-03-29T10:12:46.978828+0000","last_became_active":"2024-03-29T10:12:46.312481+0000","last_became_peered":"2024-03-29T10:12:46.312481+0000","last_unstale":"2024-03-29T10:12:46.978828+0000","last_undegraded":"2024-03-29T10:12:46.978828+0000","last_fullsized":"2024-03-29T10:12:46.978828+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:11:34.526149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":10,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.978957+0000","last_change":"2024-03-29T10:12:46.850592+0000","last_active":"2024-03-29T10:12:46.978957+0000","last_peered":"2024-03-29T10:12:46.978957+0000","last_clean":"2024-03-29T10:12:46.978957+0000","last_became_active":"2024-03-29T10:12:46.850356+0000","last_became_peered":"2024-03-29T10:12:46.850356+0000","last_unstale":"2024-03-29T10:12:46.978957+0000","last_undegraded":"2024-03-29T10:12:46.978957+0000","last_fullsized":"2024-03-29T10:12:46.978957+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:46.281885+0000","last_clean_scrub_stamp":"2024-03-29T10:12:46.281885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:00:38.570406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"123'1","reported_seq":12,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.982647+0000","last_change":"2024-03-29T10:12:46.315886+0000","last_active":"2024-03-29T10:12:46.982647+0000","last_peered":"2024-03-29T10:12:46.982647+0000","last_clean":"2024-03-29T10:12:46.982647+0000","last_became_active":"2024-03-29T10:12:46.315577+0000","last_became_peered":"2024-03-29T10:12:46.315577+0000","last_unstale":"2024-03-29T10:12:46.982647+0000","last_undegraded":"2024-03-29T10:12:46.982647+0000","last_fullsized":"2024-03-29T10:12:46.982647+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:12:45.272503+0000","last_clean_scrub_stamp":"2024-03-29T10:12:45.272503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:51:08.830748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"113'223","reported_seq":638,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-29T10:12:46.830379+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:12:46.830379+0000","last_peered":"2024-03-29T10:12:46.830379+0000","last_clean":"2024-03-29T10:12:46.830379+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:12:46.830379+0000","last_undegraded":"2024-03-29T10:12:46.830379+0000","last_fullsized":"2024-03-29T10:12:46.830379+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":351,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":351,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575063,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37304,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334536,"statfs":{"total":96636764160,"available":96598564864,"internally_reserved":0,"allocated":2134016,"data_stored":1321024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36043212},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67700000000000005,"15min":0.71299999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.89600000000000002,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.68899999999999995,"15min":0.71199999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":1.143,"15min":1.143},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.71199999999999997,"15min":0.72599999999999998},"min":{"1min":0.48899999999999999,"5min":0.45700000000000002,"15min":0.44500000000000001},"max":{"1min":1.109,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.70799999999999996,"15min":0.73699999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.41199999999999998},"max":{"1min":0.94899999999999995,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.72599999999999998,"15min":0.73999999999999999},"min":{"1min":0.52100000000000002,"5min":0.47199999999999998,"15min":0.45200000000000001},"max":{"1min":0.97099999999999997,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72599999999999998,"15min":0.752},"min":{"1min":0.54800000000000004,"5min":0.49299999999999999,"15min":0.48799999999999999},"max":{"1min":0.94099999999999995,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.749}]},{"osd":3,"last update":"Fri Mar 29 10:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.72299999999999998,"15min":0.74199999999999999},"min":{"1min":0.53800000000000003,"5min":0.47199999999999998,"15min":0.40699999999999997},"max":{"1min":1.1439999999999999,"5min":1.171,"15min":1.171},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.747,"5min":0.73699999999999999,"15min":0.77200000000000002},"min":{"1min":0.58699999999999997,"5min":0.52300000000000002,"15min":0.50700000000000001},"max":{"1min":0.88400000000000001,"5min":1.129,"15min":1.129},"last":0.77900000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61699999999999999,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.38200000000000001},"max":{"1min":0.93100000000000005,"5min":1.25,"15min":1.25},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.65400000000000003,"15min":0.63700000000000001},"min":{"1min":0.49099999999999999,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":1.0720000000000001,"5min":1.117,"15min":1.117},"last":0.72299999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.66400000000000003,"15min":0.65100000000000002},"min":{"1min":0.47099999999999997,"5min":0.43099999999999999,"15min":0.40699999999999997},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.71199999999999997,"15min":0.68500000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.39400000000000002},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.88400000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.747,"15min":0.72699999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1890000000000001,"5min":1.222,"15min":1.222},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72599999999999998,"15min":0.70599999999999996},"min":{"1min":0.504,"5min":0.443,"15min":0.443},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.876}]}]},{"osd":6,"up_from":13,"seq":55834575063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36688,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335152,"statfs":{"total":96636764160,"available":96599195648,"internally_reserved":0,"allocated":1507328,"data_stored":717469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.70999999999999996,"15min":0.70899999999999996},"min":{"1min":0.46500000000000002,"5min":0.42899999999999999,"15min":0.34699999999999998},"max":{"1min":1.006,"5min":2.7770000000000001,"15min":2.7770000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70399999999999996,"15min":0.72599999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93300000000000005,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.432}]},{"osd":1,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67600000000000005,"15min":0.70699999999999996},"min":{"1min":0.49199999999999999,"5min":0.46000000000000002,"15min":0.443},"max":{"1min":0.94799999999999995,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71899999999999997,"15min":0.72099999999999997},"min":{"1min":0.503,"5min":0.45200000000000001,"15min":0.42399999999999999},"max":{"1min":1.0289999999999999,"5min":2.6709999999999998,"15min":2.6709999999999998},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68000000000000005,"15min":0.71099999999999997},"min":{"1min":0.56699999999999995,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.93200000000000005,"5min":1.256,"15min":1.256},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72399999999999998,"15min":0.74299999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.40400000000000003},"max":{"1min":0.876,"5min":1.454,"15min":1.454},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.73699999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":0.97899999999999998,"5min":1.131,"15min":1.131},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.754,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.56899999999999995,"5min":0.44500000000000001,"15min":0.39000000000000001},"max":{"1min":1.034,"5min":2.718,"15min":2.718},"last":0.65500000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.626,"15min":0.60799999999999998},"min":{"1min":0.44400000000000001,"5min":0.40500000000000003,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70899999999999996,"15min":0.68999999999999995},"min":{"1min":0.51600000000000001,"5min":0.46999999999999997,"15min":0.40600000000000003},"max":{"1min":0.872,"5min":2.544,"15min":2.544},"last":0.61199999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67300000000000004,"15min":0.65600000000000003},"min":{"1min":0.41899999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79100000000000004,"5min":3.2280000000000002,"15min":3.2280000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.71399999999999997,"15min":0.68500000000000005},"min":{"1min":0.46700000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91400000000000003,"5min":2.835,"15min":2.835},"last":0.63100000000000001}]},{"osd":7,"last update":"Fri Mar 29 10:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75700000000000001,"15min":0.748},"min":{"1min":0.59699999999999998,"5min":0.48899999999999999,"15min":0.45100000000000001},"max":{"1min":0.94999999999999996,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72299999999999998,"15min":0.71599999999999997},"min":{"1min":0.501,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89800000000000002,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.68300000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54764,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317076,"statfs":{"total":96636764160,"available":96580685824,"internally_reserved":0,"allocated":1531904,"data_stored":730544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45900000000000002,"15min":0.44900000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.219},"max":{"1min":0.69399999999999995,"5min":0.81699999999999995,"15min":1.5600000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.41499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.65000000000000002,"5min":0.71799999999999997,"15min":1.0860000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.41899999999999998,"5min":0.38800000000000001,"15min":0.316},"max":{"1min":0.71499999999999997,"5min":1.222,"15min":1.7869999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53200000000000003,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33400000000000002,"15min":0.33200000000000002},"max":{"1min":0.65100000000000002,"5min":1.04,"15min":1.04},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57999999999999996,"15min":0.59299999999999997},"min":{"1min":0.48099999999999998,"5min":0.38600000000000001,"15min":0.377},"max":{"1min":0.74199999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.45700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71499999999999997,"5min":1.026,"15min":1.3049999999999999},"last":0.52500000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.749,"15min":0.76300000000000001},"min":{"1min":0.57799999999999996,"5min":0.52900000000000003,"15min":0.497},"max":{"1min":0.95799999999999996,"5min":1.2709999999999999,"15min":1.617},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.77700000000000002,"15min":0.77500000000000002},"min":{"1min":0.60999999999999999,"5min":0.54900000000000004,"15min":0.48299999999999998},"max":{"1min":1.05,"5min":1.1419999999999999,"15min":1.155},"last":0.79800000000000004}]},{"osd":5,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.77600000000000002,"15min":0.79400000000000004},"min":{"1min":0.67000000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.101,"5min":1.101,"15min":1.3939999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.78700000000000003,"15min":0.79800000000000004},"min":{"1min":0.62,"5min":0.55800000000000005,"15min":0.52900000000000003},"max":{"1min":1.0309999999999999,"5min":1.3140000000000001,"15min":1.6579999999999999},"last":0.77500000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.65200000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.996,"5min":1.0840000000000001,"15min":1.236},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.79400000000000004,"15min":0.81299999999999994},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.1419999999999999,"5min":1.1579999999999999,"15min":1.468},"last":0.94699999999999995}]},{"osd":7,"last update":"Fri Mar 29 10:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.66300000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0629999999999999,"5min":1.1319999999999999,"15min":1.6000000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.82899999999999996},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.57099999999999995},"max":{"1min":1.1240000000000001,"5min":1.4530000000000001,"15min":1.6919999999999999},"last":0.871}]}]},{"osd":0,"up_from":13,"seq":55834575064,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2125824,"data_stored":1320475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.438,"15min":0.45300000000000001},"min":{"1min":0.35899999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.67100000000000004,"5min":0.91000000000000003,"15min":0.93300000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.48299999999999998,"15min":0.51000000000000001},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.70299999999999996,"5min":0.86699999999999999,"15min":1.0529999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.56299999999999994},"min":{"1min":0.38600000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.0069999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.46899999999999997,"15min":0.497},"min":{"1min":0.34200000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.65900000000000003,"5min":0.77800000000000002,"15min":0.97199999999999998},"last":0.59999999999999998}]},{"osd":3,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52600000000000002,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78400000000000003,"5min":0.84899999999999998,"15min":1.077},"last":0.627},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58399999999999996,"15min":0.61099999999999999},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78800000000000003,"5min":0.83499999999999996,"15min":1.0549999999999999},"last":0.72699999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.75800000000000001,"15min":0.79900000000000004},"min":{"1min":0.63900000000000001,"5min":0.436,"15min":0.39200000000000002},"max":{"1min":1.0429999999999999,"5min":1.625,"15min":1.625},"last":0.87},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77800000000000002,"15min":0.82099999999999995},"min":{"1min":0.60099999999999998,"5min":0.49099999999999999,"15min":0.45800000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.167},"last":1.0629999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.70399999999999996,"15min":0.755},"min":{"1min":0.56399999999999995,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97199999999999998,"5min":1.02,"15min":1.137},"last":1.002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.75700000000000001,"15min":0.80100000000000005},"min":{"1min":0.53300000000000003,"5min":0.436,"15min":0.41099999999999998},"max":{"1min":1.0589999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.92700000000000005}]},{"osd":6,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.78400000000000003,"15min":0.81799999999999995},"min":{"1min":0.56799999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.014,"5min":1.0640000000000001,"15min":1.228},"last":1.125},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.80000000000000004,"15min":0.84199999999999997},"min":{"1min":0.67800000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.105,"5min":1.601,"15min":1.601},"last":1.0349999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.745,"15min":0.79500000000000004},"min":{"1min":0.59899999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.1890000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.79700000000000004,"15min":0.83599999999999997},"min":{"1min":0.58899999999999997,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.135,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.98199999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575063,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1536000,"data_stored":730578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.248},"max":{"1min":0.71699999999999997,"5min":0.746,"15min":0.86099999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.502,"15min":0.50900000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.82399999999999995},"last":0.51600000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48699999999999999,"15min":0.47299999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26200000000000001},"max":{"1min":0.59899999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68100000000000005,"5min":0.78500000000000003,"15min":1.032},"last":0.46400000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.442,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.99099999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55000000000000004,"15min":0.54700000000000004},"min":{"1min":0.40600000000000003,"5min":0.35199999999999998,"15min":0.32100000000000001},"max":{"1min":0.75700000000000001,"5min":1.516,"15min":1.516},"last":0.59399999999999997}]},{"osd":4,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.77800000000000002},"min":{"1min":0.54200000000000004,"5min":0.499,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":1.125,"15min":1.3520000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.76000000000000001,"15min":0.79700000000000004},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.96099999999999997}]},{"osd":5,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76500000000000001,"15min":0.80000000000000004},"min":{"1min":0.52400000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0569999999999999,"5min":1.1699999999999999,"15min":1.2589999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.76100000000000001,"15min":0.79600000000000004},"min":{"1min":0.56499999999999995,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0469999999999999,"5min":1.115,"15min":1.2250000000000001},"last":0.98599999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78000000000000003,"15min":0.81200000000000006},"min":{"1min":0.55100000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0389999999999999,"5min":1.1859999999999999,"15min":1.2350000000000001},"last":0.999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.83199999999999996},"min":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.99099999999999999,"5min":1.1120000000000001,"15min":1.3560000000000001},"last":1.0129999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.83499999999999996,"15min":0.86199999999999999},"min":{"1min":0.63900000000000001,"5min":0.57899999999999996,"15min":0.53400000000000003},"max":{"1min":1.1339999999999999,"5min":1.258,"15min":1.4490000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83999999999999997,"15min":0.872},"min":{"1min":0.63100000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1180000000000001,"5min":1.226,"15min":1.4199999999999999},"last":0.97099999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575063,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1548288,"data_stored":732130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.26800000000000002,"15min":0.26500000000000001},"max":{"1min":0.65100000000000002,"5min":0.75600000000000001,"15min":0.81299999999999994},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.29899999999999999,"5min":0.29099999999999998,"15min":0.28399999999999997},"max":{"1min":0.73999999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.51500000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46200000000000002,"15min":0.48599999999999999},"min":{"1min":0.36399999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.98699999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54000000000000004,"15min":0.56200000000000006},"min":{"1min":0.46600000000000003,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72599999999999998,"5min":0.82699999999999996,"15min":0.96099999999999997},"last":0.499}]},{"osd":2,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.373},"max":{"1min":0.75900000000000001,"5min":0.83999999999999997,"15min":0.98299999999999998},"last":0.623},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59899999999999998,"15min":0.624},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80000000000000004,"5min":0.86899999999999999,"15min":0.94799999999999995},"last":0.57299999999999995}]},{"osd":4,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.73799999999999999,"15min":0.76600000000000001},"min":{"1min":0.55300000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.99299999999999999,"5min":1.0940000000000001,"15min":1.288},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.60199999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.004,"5min":1.093,"15min":1.46},"last":0.84199999999999997}]},{"osd":5,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.77000000000000002,"15min":0.79200000000000004},"min":{"1min":0.62,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.3580000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.76300000000000001,"15min":0.79700000000000004},"min":{"1min":0.53800000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92900000000000005,"5min":1.1479999999999999,"15min":1.429},"last":0.67800000000000005}]},{"osd":6,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.78600000000000003,"15min":0.80800000000000005},"min":{"1min":0.58099999999999996,"5min":0.54500000000000004,"15min":0.54200000000000004},"max":{"1min":1.123,"5min":1.123,"15min":1.4730000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78800000000000003,"15min":0.81599999999999995},"min":{"1min":0.66900000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.3999999999999999},"last":0.77200000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79700000000000004,"15min":0.83299999999999996},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0489999999999999,"5min":1.135,"15min":1.5029999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.81200000000000006,"15min":0.83499999999999996},"min":{"1min":0.63300000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.073,"5min":1.1859999999999999,"15min":1.345},"last":0.83399999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58816,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313024,"statfs":{"total":96636764160,"available":96576536576,"internally_reserved":0,"allocated":1495040,"data_stored":711194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68700000000000006,"15min":0.69299999999999995},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.91200000000000003,"5min":0.98799999999999999,"15min":1.417},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68700000000000006,"15min":0.69699999999999995},"min":{"1min":0.38,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.165},"last":0.53500000000000003}]},{"osd":1,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.69099999999999995,"15min":0.70599999999999996},"min":{"1min":0.55700000000000005,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.014},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69799999999999995,"15min":0.70999999999999996},"min":{"1min":0.46300000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.876,"5min":1.075,"15min":1.333},"last":0.51100000000000001}]},{"osd":2,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.77600000000000002,"15min":0.77400000000000002},"min":{"1min":0.66400000000000003,"5min":0.433,"15min":0.433},"max":{"1min":1.014,"5min":1.014,"15min":1.03},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.77900000000000003,"15min":0.77200000000000002},"min":{"1min":0.59699999999999998,"5min":0.50600000000000001,"15min":0.46200000000000002},"max":{"1min":1.024,"5min":1.0600000000000001,"15min":1.387},"last":0.79200000000000004}]},{"osd":3,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.72499999999999998,"15min":0.72699999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.2350000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72199999999999998,"15min":0.73499999999999999},"min":{"1min":0.53800000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.90400000000000003,"5min":1.0229999999999999,"15min":1.3180000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60499999999999998,"15min":0.58499999999999996},"min":{"1min":0.45700000000000002,"5min":0.42499999999999999,"15min":0.38100000000000001},"max":{"1min":0.83799999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.746},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.629,"15min":0.60399999999999998},"min":{"1min":0.435,"5min":0.36399999999999999,"15min":0.309},"max":{"1min":0.91600000000000004,"5min":0.98599999999999999,"15min":1.294},"last":0.73699999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64400000000000002,"15min":0.61899999999999999},"min":{"1min":0.47599999999999998,"5min":0.44,"15min":0.375},"max":{"1min":0.85099999999999998,"5min":1.026,"15min":1.026},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67300000000000004,"15min":0.65800000000000003},"min":{"1min":0.503,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97599999999999998,"5min":1.093,"15min":1.1359999999999999},"last":0.82999999999999996}]},{"osd":7,"last update":"Fri Mar 29 10:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68799999999999994,"15min":0.68100000000000005},"min":{"1min":0.51100000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.345},"last":0.874},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.77500000000000002,"15min":0.75700000000000001},"min":{"1min":0.64000000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.995,"5min":0.999,"15min":1.214},"last":0.89200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575062,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36684,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335156,"statfs":{"total":96636764160,"available":96599199744,"internally_reserved":0,"allocated":1507328,"data_stored":717469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64200000000000002,"15min":0.68300000000000005},"min":{"1min":0.41699999999999998,"5min":0.39200000000000002,"15min":0.35699999999999998},"max":{"1min":0.83999999999999997,"5min":1.0429999999999999,"15min":1.1899999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.746,"5min":0.71599999999999997,"15min":0.72999999999999998},"min":{"1min":0.56599999999999995,"5min":0.435,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":1.21,"15min":1.339},"last":0.56399999999999995}]},{"osd":1,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68799999999999994,"15min":0.71799999999999997},"min":{"1min":0.496,"5min":0.46500000000000002,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67100000000000004,"15min":0.70699999999999996},"min":{"1min":0.52500000000000002,"5min":0.441,"15min":0.39200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70399999999999996,"15min":0.73199999999999998},"min":{"1min":0.60299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.877,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67900000000000005,"15min":0.70799999999999996},"min":{"1min":0.47499999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92900000000000005,"5min":0.93700000000000006,"15min":1.095},"last":0.60599999999999998}]},{"osd":3,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.69099999999999995,"15min":0.72899999999999998},"min":{"1min":0.54400000000000004,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":0.878,"5min":1.042,"15min":1.1419999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.72199999999999998,"15min":0.752},"min":{"1min":0.60299999999999998,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.94899999999999995,"5min":1.0249999999999999,"15min":1.4850000000000001},"last":0.59699999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.40899999999999997,"5min":0.34100000000000003,"15min":0.318},"max":{"1min":0.91700000000000004,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64700000000000002,"15min":0.63800000000000001},"min":{"1min":0.45700000000000002,"5min":0.38900000000000001,"15min":0.35299999999999998},"max":{"1min":0.92900000000000005,"5min":1.097,"15min":1.097},"last":0.51800000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.65300000000000002,"15min":0.65200000000000002},"min":{"1min":0.56399999999999995,"5min":0.41299999999999998,"15min":0.36799999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":1.151},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66200000000000003,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89000000000000001,"5min":0.97099999999999997,"15min":0.996},"last":0.54700000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68300000000000005,"15min":0.69599999999999995},"min":{"1min":0.48999999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89400000000000002,"5min":0.95299999999999996,"15min":1.0289999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.52200000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94799999999999995,"5min":0.98699999999999999,"15min":1.1990000000000001},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:12:52.236 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T10:12:52.236 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T10:12:52.268 INFO:tasks.ceph.mds.c.smithi012.stderr:2024-03-29T10:12:52.269+0000 7f81f5000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:12:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.473+0000 7fb052800700 1 Processor -- start 2024-03-29T10:12:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.474+0000 7fb052800700 1 -- start start 2024-03-29T10:12:52.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.475+0000 7fb052800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0cee10 0x7fb04c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:52.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.475+0000 7fb052800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c0ce270 con 0x7fb04c0d21f0 2024-03-29T10:12:52.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.475+0000 7fb052800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c15b600 con 0x7fb04c1611b0 2024-03-29T10:12:52.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.475+0000 7fb052800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0d2760 con 0x7fb04c0cee10 2024-03-29T10:12:52.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.475+0000 7fb04aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb04c0d21f0 0x7fb04c0ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47468/0 (socket says 172.21.15.12:47468) 2024-03-29T10:12:52.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.475+0000 7fb04aa00700 1 -- 172.21.15.12:0/3554932295 learned_addr learned my addr 172.21.15.12:0/3554932295 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.475+0000 7fb04be00700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0cee10 0x7fb04c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.475+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797435051 0 0) 0x7fb04c0ce270 con 0x7fb04c0d21f0 2024-03-29T10:12:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.475+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0300035b0 con 0x7fb04c0d21f0 2024-03-29T10:12:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 510838411 0 0) 0x7fb0300035b0 con 0x7fb04c0d21f0 2024-03-29T10:12:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0cee10 msgr2=0x7fb04c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0cee10 0x7fb04c161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 >> v1:172.21.15.159:6789/0 conn(0x7fb04c1611b0 legacy=0x7fb04c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c0d2a40 con 0x7fb04c0d21f0 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fb038001670 con 0x7fb04c0d21f0 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 >> v1:172.21.15.12:6789/0 conn(0x7fb04c0d21f0 legacy=0x7fb04c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb030003a30 0x7fb030003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb030004850 0x7fb030004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0cee10 0x7fb030005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0d2760 con 0x7fb030003a30 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03000bac0 con 0x7fb030004850 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03000bc30 con 0x7fb04c0cee10 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04b400700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb030003a30 0x7fb030003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04aa00700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb030004850 0x7fb030004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.476+0000 7fb04be00700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0cee10 0x7fb030005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.477+0000 7fb04b400700 1 -- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0cee10 msgr2=0x7fb030005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.477+0000 7fb04b400700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0cee10 0x7fb030005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.477+0000 7fb04b400700 1 -- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb030004850 msgr2=0x7fb030004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.477+0000 7fb04b400700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb030004850 0x7fb030004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.477+0000 7fb04b400700 1 -- 172.21.15.12:0/3554932295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0300035b0 con 0x7fb030003a30 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.477+0000 7fb04b400700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb030003a30 0x7fb030003e10 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fb03c001880 tx=0x7fb030005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eceb444cdff8ea6a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.477+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c001370 con 0x7fb030003a30 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.477+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c012640 con 0x7fb030003a30 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.477+0000 7fb04a000700 1 -- 172.21.15.12:0/3554932295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c0129f0 con 0x7fb030003a30 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.478+0000 7fb052800700 1 -- 172.21.15.12:0/3554932295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb030003a30 msgr2=0x7fb030003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.478+0000 7fb052800700 1 --2- 172.21.15.12:0/3554932295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb030003a30 0x7fb030003e10 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fb03c001880 tx=0x7fb030005880 comp rx=0 tx=0).stop 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.478+0000 7fb04be00700 1 -- 172.21.15.12:0/3554932295 reap_dead start 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.478+0000 7fb052800700 1 -- 172.21.15.12:0/3554932295 shutdown_connections 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.478+0000 7fb052800700 1 -- 172.21.15.12:0/3554932295 >> 172.21.15.12:0/3554932295 conn(0x7fb04c08aff0 msgr2=0x7fb04c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.478+0000 7fb052800700 1 -- 172.21.15.12:0/3554932295 shutdown_connections 2024-03-29T10:12:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.478+0000 7fb052800700 1 -- 172.21.15.12:0/3554932295 wait complete. 2024-03-29T10:12:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.480+0000 7fb052800700 1 Processor -- start 2024-03-29T10:12:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.480+0000 7fb052800700 1 -- start start 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb052800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb030003a30 0x7fb04c0cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb052800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c0c8d90 0x7fb04c157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb052800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0b68f0 0x7fb04c0cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb052800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0c7850 con 0x7fb04c0b68f0 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb052800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0c95a0 con 0x7fb04c0c8d90 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb052800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0c92c0 con 0x7fb030003a30 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0b68f0 0x7fb04c0cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0b68f0 0x7fb04c0cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52360/0 (socket says 172.21.15.12:52360) 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04be00700 1 -- 172.21.15.12:0/118796127 learned_addr learned my addr 172.21.15.12:0/118796127 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04aa00700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c0c8d90 0x7fb04c157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04b400700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb030003a30 0x7fb04c0cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04be00700 1 -- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb030003a30 msgr2=0x7fb04c0cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04be00700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb030003a30 0x7fb04c0cb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04be00700 1 -- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c0c8d90 msgr2=0x7fb04c157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04be00700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c0c8d90 0x7fb04c157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04be00700 1 -- 172.21.15.12:0/118796127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c0c7190 con 0x7fb04c0b68f0 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.481+0000 7fb04aa00700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c0c8d90 0x7fb04c157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.482+0000 7fb04be00700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0b68f0 0x7fb04c0cbcf0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb04c08d850 tx=0x7fb04400f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.482+0000 7fb043400700 1 -- 172.21.15.12:0/118796127 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04400e040 con 0x7fb04c0b68f0 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.482+0000 7fb043400700 1 -- 172.21.15.12:0/118796127 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04400f040 con 0x7fb04c0b68f0 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.482+0000 7fb043400700 1 -- 172.21.15.12:0/118796127 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044015430 con 0x7fb04c0b68f0 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.482+0000 7fb052800700 1 -- 172.21.15.12:0/118796127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04c0c7470 con 0x7fb04c0b68f0 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.482+0000 7fb052800700 1 -- 172.21.15.12:0/118796127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c1581e0 con 0x7fb04c0b68f0 2024-03-29T10:12:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.483+0000 7fb052800700 1 -- 172.21.15.12:0/118796127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04c067720 con 0x7fb04c0b68f0 2024-03-29T10:12:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.484+0000 7fb043400700 1 -- 172.21.15.12:0/118796127 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb044015590 con 0x7fb04c0b68f0 2024-03-29T10:12:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.485+0000 7fb043400700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb02c06adb0 0x7fb02c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.485+0000 7fb043400700 1 -- 172.21.15.12:0/118796127 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb044089e20 con 0x7fb04c0b68f0 2024-03-29T10:12:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.485+0000 7fb04b400700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb02c06adb0 0x7fb02c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.486+0000 7fb04b400700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb02c06adb0 0x7fb02c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb03c001880 tx=0x7fb03c005040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.489+0000 7fb043400700 1 -- 172.21.15.12:0/118796127 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb044054e80 con 0x7fb04c0b68f0 2024-03-29T10:12:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.783+0000 7fb052800700 1 -- 172.21.15.12:0/118796127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb04c0b5d00 con 0x7fb04c0b68f0 2024-03-29T10:12:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.788+0000 7fb043400700 1 -- 172.21.15.12:0/118796127 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+0 (secure 0 0 0) 0x7fb0440584a0 con 0x7fb04c0b68f0 2024-03-29T10:12:52.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 -- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb02c06adb0 msgr2=0x7fb02c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:52.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb02c06adb0 0x7fb02c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb03c001880 tx=0x7fb03c005040 comp rx=0 tx=0).stop 2024-03-29T10:12:52.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 -- 172.21.15.12:0/118796127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0b68f0 msgr2=0x7fb04c0cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0b68f0 0x7fb04c0cbcf0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb04c08d850 tx=0x7fb04400f490 comp rx=0 tx=0).stop 2024-03-29T10:12:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 -- 172.21.15.12:0/118796127 shutdown_connections 2024-03-29T10:12:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb02c06adb0 0x7fb02c06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0b68f0 0x7fb04c0cbcf0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb030003a30 0x7fb04c0cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 --2- 172.21.15.12:0/118796127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c0c8d90 0x7fb04c157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 -- 172.21.15.12:0/118796127 >> 172.21.15.12:0/118796127 conn(0x7fb04c08aff0 msgr2=0x7fb04c0ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 -- 172.21.15.12:0/118796127 shutdown_connections 2024-03-29T10:12:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:52.790+0000 7fb052800700 1 -- 172.21.15.12:0/118796127 wait complete. 2024-03-29T10:12:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T10:12:52.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:12:53.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.007+0000 7f4e01c00700 1 Processor -- start 2024-03-29T10:12:53.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.007+0000 7f4e01c00700 1 -- start start 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.008+0000 7f4e01c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4dfc15d920 0x7f4dfc06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.008+0000 7f4e01c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4dfc099a60 0x7f4dfc099e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.008+0000 7f4e01c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc0d08d0 con 0x7f4dfc09a910 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.008+0000 7f4e01c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dfc06c300 con 0x7f4dfc099a60 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.008+0000 7f4e01c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dfc09a380 con 0x7f4dfc15d920 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.008+0000 7f4dfb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4dfc099a60 0x7f4dfc099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.008+0000 7f4dfb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4dfc099a60 0x7f4dfc099e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52374/0 (socket says 172.21.15.12:52374) 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.008+0000 7f4dfb400700 1 -- 172.21.15.12:0/2751711876 learned_addr learned my addr 172.21.15.12:0/2751711876 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.008+0000 7f4dfaa00700 1 --2- 172.21.15.12:0/2751711876 >> v2:172.21.15.159:3300/0 conn(0x7f4dfc15d920 0x7f4dfc06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.009+0000 7f4dfb400700 1 -- 172.21.15.12:0/2751711876 >> v2:172.21.15.159:3300/0 conn(0x7f4dfc15d920 msgr2=0x7f4dfc06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.009+0000 7f4dfb400700 1 --2- 172.21.15.12:0/2751711876 >> v2:172.21.15.159:3300/0 conn(0x7f4dfc15d920 0x7f4dfc06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.009+0000 7f4dfb400700 1 -- 172.21.15.12:0/2751711876 >> v1:172.21.15.159:6789/0 conn(0x7f4dfc09a910 legacy=0x7f4dfc0cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.009+0000 7f4dfb400700 1 -- 172.21.15.12:0/2751711876 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc06b9d0 con 0x7f4dfc099a60 2024-03-29T10:12:53.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.009+0000 7f4dfaa00700 1 --2- 172.21.15.12:0/2751711876 >> v2:172.21.15.159:3300/0 conn(0x7f4dfc15d920 0x7f4dfc06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:53.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.009+0000 7f4dfb400700 1 --2- 172.21.15.12:0/2751711876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 0x7f4dfc099e40 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc09a6b0 tx=0x7f4df400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6ef8f2217b909c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:53.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.009+0000 7f4df9600700 1 -- 172.21.15.12:0/2751711876 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df400e070 con 0x7f4dfc099a60 2024-03-29T10:12:53.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.009+0000 7f4df9600700 1 -- 172.21.15.12:0/2751711876 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df400b9c0 con 0x7f4dfc099a60 2024-03-29T10:12:53.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.009+0000 7f4df9600700 1 -- 172.21.15.12:0/2751711876 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df400bd70 con 0x7f4dfc099a60 2024-03-29T10:12:53.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.010+0000 7f4e01c00700 1 -- 172.21.15.12:0/2751711876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 msgr2=0x7f4dfc099e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.010+0000 7f4e01c00700 1 --2- 172.21.15.12:0/2751711876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 0x7f4dfc099e40 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc09a6b0 tx=0x7f4df400b330 comp rx=0 tx=0).stop 2024-03-29T10:12:53.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.010+0000 7f4e01c00700 1 -- 172.21.15.12:0/2751711876 shutdown_connections 2024-03-29T10:12:53.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.010+0000 7f4e01c00700 1 --2- 172.21.15.12:0/2751711876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 0x7f4dfc099e40 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.010+0000 7f4e01c00700 1 --2- 172.21.15.12:0/2751711876 >> v2:172.21.15.159:3300/0 conn(0x7f4dfc15d920 0x7f4dfc06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.010+0000 7f4e01c00700 1 -- 172.21.15.12:0/2751711876 >> 172.21.15.12:0/2751711876 conn(0x7f4dfc0d1350 msgr2=0x7f4dfc0d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:53.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.010+0000 7f4e01c00700 1 -- 172.21.15.12:0/2751711876 shutdown_connections 2024-03-29T10:12:53.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.010+0000 7f4e01c00700 1 -- 172.21.15.12:0/2751711876 wait complete. 2024-03-29T10:12:53.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.012+0000 7f4e01c00700 1 Processor -- start 2024-03-29T10:12:53.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.012+0000 7f4e01c00700 1 -- start start 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4e01c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 0x7f4dfc14ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4e01c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dfc09a910 0x7f4dfc15f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4e01c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dfc15d920 0x7f4dfc15e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4e01c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc15dd00 con 0x7f4dfc099a60 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4e01c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc14efd0 con 0x7f4dfc15d920 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4e01c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc14f270 con 0x7f4dfc09a910 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 0x7f4dfc14ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 0x7f4dfc14ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52376/0 (socket says 172.21.15.12:52376) 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfaa00700 1 -- 172.21.15.12:0/3720463549 learned_addr learned my addr 172.21.15.12:0/3720463549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfb400700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dfc15d920 0x7f4dfc15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfaa00700 1 -- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dfc09a910 msgr2=0x7f4dfc15f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfa000700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dfc09a910 0x7f4dfc15f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfaa00700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dfc09a910 0x7f4dfc15f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfaa00700 1 -- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dfc15d920 msgr2=0x7f4dfc15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfaa00700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dfc15d920 0x7f4dfc15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfaa00700 1 -- 172.21.15.12:0/3720463549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df4009b30 con 0x7f4dfc099a60 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfb400700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dfc15d920 0x7f4dfc15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.013+0000 7f4dfa000700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dfc09a910 0x7f4dfc15f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.014+0000 7f4dfaa00700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 0x7f4dfc14ea90 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00e810 tx=0x7f4dec00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.014+0000 7f4df2a00700 1 -- 172.21.15.12:0/3720463549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec00c020 con 0x7f4dfc099a60 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.014+0000 7f4df2a00700 1 -- 172.21.15.12:0/3720463549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dec004250 con 0x7f4dfc099a60 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.014+0000 7f4df2a00700 1 -- 172.21.15.12:0/3720463549 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0036a0 con 0x7f4dfc099a60 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.014+0000 7f4e01c00700 1 -- 172.21.15.12:0/3720463549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dfc14f550 con 0x7f4dfc099a60 2024-03-29T10:12:53.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.014+0000 7f4e01c00700 1 -- 172.21.15.12:0/3720463549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dfc15df10 con 0x7f4dfc099a60 2024-03-29T10:12:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.015+0000 7f4e01c00700 1 -- 172.21.15.12:0/3720463549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dfc067720 con 0x7f4dfc099a60 2024-03-29T10:12:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.016+0000 7f4df2a00700 1 -- 172.21.15.12:0/3720463549 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4dec003800 con 0x7f4dfc099a60 2024-03-29T10:12:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.017+0000 7f4df2a00700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ddc06adb0 0x7f4ddc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.017+0000 7f4df2a00700 1 -- 172.21.15.12:0/3720463549 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4dec089450 con 0x7f4dfc099a60 2024-03-29T10:12:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.017+0000 7f4dfa000700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ddc06adb0 0x7f4ddc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.018+0000 7f4dfa000700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ddc06adb0 0x7f4ddc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0d10b0 tx=0x7f4de8009040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:53.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.021+0000 7f4df2a00700 1 -- 172.21.15.12:0/3720463549 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4dec054370 con 0x7f4dfc099a60 2024-03-29T10:12:53.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.285+0000 7f4e01c00700 1 -- 172.21.15.12:0/3720463549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dfc15e280 con 0x7f4dfc099a60 2024-03-29T10:12:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.285+0000 7f4df2a00700 1 -- 172.21.15.12:0/3720463549 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f4dec057990 con 0x7f4dfc099a60 2024-03-29T10:12:53.290 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:12:53.290 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":55,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7645,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/553963486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":553963486},{"type":"v1","addr":"172.21.15.159:6837","nonce":553963486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3051067707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3051067707},{"type":"v1","addr":"172.21.15.159:6836","nonce":3051067707}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3629410472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3629410472},{"type":"v1","addr":"172.21.15.12:6833","nonce":3629410472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:12:46.810806+0000","modified":"2024-03-29T10:12:50.294107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7917},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7917":{"gid":7917,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/753821395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":753821395},{"type":"v1","addr":"172.21.15.12:6835","nonce":753821395}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-29T10:12:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 -- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ddc06adb0 msgr2=0x7f4ddc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ddc06adb0 0x7f4ddc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0d10b0 tx=0x7f4de8009040 comp rx=0 tx=0).stop 2024-03-29T10:12:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 -- 172.21.15.12:0/3720463549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 msgr2=0x7f4dfc14ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 0x7f4dfc14ea90 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00e810 tx=0x7f4dec00ebd0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 -- 172.21.15.12:0/3720463549 shutdown_connections 2024-03-29T10:12:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4ddc06adb0 0x7f4ddc06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dfc15d920 0x7f4dfc15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc099a60 0x7f4dfc14ea90 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 --2- 172.21.15.12:0/3720463549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dfc09a910 0x7f4dfc15f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 -- 172.21.15.12:0/3720463549 >> 172.21.15.12:0/3720463549 conn(0x7f4dfc0d1350 msgr2=0x7f4dfc06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 -- 172.21.15.12:0/3720463549 shutdown_connections 2024-03-29T10:12:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.288+0000 7f4e01c00700 1 -- 172.21.15.12:0/3720463549 wait complete. 2024-03-29T10:12:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 55 2024-03-29T10:12:53.710 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T10:12:46.810806+0000', 'modified': '2024-03-29T10:12:50.294107+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7917}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7917': {'gid': 7917, 'name': 'c', 'rank': 0, 'incarnation': 51, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6835/753821395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6834', 'nonce': 753821395}, {'type': 'v1', 'addr': '172.21.15.12:6835', 'nonce': 753821395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T10:12:53.710 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T10:12:53.711 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-29T10:12:53.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.912+0000 7f0ac8000700 1 Processor -- start 2024-03-29T10:12:53.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.912+0000 7f0ac8000700 1 -- start start 2024-03-29T10:12:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.913+0000 7f0ac8000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac01611b0 0x7f0ac015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac8000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ac00cee10 0x7f0ac0161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac8000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac00ce270 con 0x7f0ac015f4d0 2024-03-29T10:12:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac8000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac015b600 con 0x7f0ac00cee10 2024-03-29T10:12:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac8000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac00d2760 con 0x7f0ac01611b0 2024-03-29T10:12:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac6000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ac00cee10 0x7f0ac0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac6000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ac00cee10 0x7f0ac0161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52384/0 (socket says 172.21.15.12:52384) 2024-03-29T10:12:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac6000700 1 -- 172.21.15.12:0/2019500695 learned_addr learned my addr 172.21.15.12:0/2019500695 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac4c00700 1 --1- 172.21.15.12:0/2019500695 >> v1:172.21.15.12:6789/0 conn(0x7f0ac015f4d0 0x7f0ac00ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47482/0 (socket says 172.21.15.12:47482) 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac5600700 1 --2- 172.21.15.12:0/2019500695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac01611b0 0x7f0ac015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac6000700 1 -- 172.21.15.12:0/2019500695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac01611b0 msgr2=0x7f0ac015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac6000700 1 --2- 172.21.15.12:0/2019500695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac01611b0 0x7f0ac015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac6000700 1 -- 172.21.15.12:0/2019500695 >> v1:172.21.15.12:6789/0 conn(0x7f0ac015f4d0 legacy=0x7f0ac00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0ac6000700 1 -- 172.21.15.12:0/2019500695 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac00d2a40 con 0x7f0ac00cee10 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.914+0000 7f0abfe00700 1 -- 172.21.15.12:0/2019500695 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032008201 0 0) 0x7f0ac00ce270 con 0x7f0ac015f4d0 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.915+0000 7f0ac6000700 1 --2- 172.21.15.12:0/2019500695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 0x7f0ac0161590 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009e50 tx=0x7f0ab800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2928ca29c7309d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.915+0000 7f0abfe00700 1 -- 172.21.15.12:0/2019500695 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800e070 con 0x7f0ac00cee10 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.915+0000 7f0abfe00700 1 -- 172.21.15.12:0/2019500695 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab800b9c0 con 0x7f0ac00cee10 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.915+0000 7f0abfe00700 1 -- 172.21.15.12:0/2019500695 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800bd70 con 0x7f0ac00cee10 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.915+0000 7f0ac8000700 1 -- 172.21.15.12:0/2019500695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 msgr2=0x7f0ac0161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.915+0000 7f0ac8000700 1 --2- 172.21.15.12:0/2019500695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 0x7f0ac0161590 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009e50 tx=0x7f0ab800b330 comp rx=0 tx=0).stop 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.916+0000 7f0ac8000700 1 -- 172.21.15.12:0/2019500695 shutdown_connections 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.916+0000 7f0ac8000700 1 --2- 172.21.15.12:0/2019500695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 0x7f0ac0161590 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.916+0000 7f0ac8000700 1 --2- 172.21.15.12:0/2019500695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac01611b0 0x7f0ac015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.916+0000 7f0ac8000700 1 -- 172.21.15.12:0/2019500695 >> 172.21.15.12:0/2019500695 conn(0x7f0ac008aff0 msgr2=0x7f0ac00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.916+0000 7f0ac8000700 1 -- 172.21.15.12:0/2019500695 shutdown_connections 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.916+0000 7f0ac8000700 1 -- 172.21.15.12:0/2019500695 wait complete. 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.917+0000 7f0ac8000700 1 Processor -- start 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.917+0000 7f0ac8000700 1 -- start start 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.918+0000 7f0ac8000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 0x7f0ac01f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.918+0000 7f0ac8000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac015f4d0 0x7f0ac00ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.918+0000 7f0ac8000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ac01611b0 0x7f0ac00cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.918+0000 7f0ac8000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01f7520 con 0x7f0ac00cee10 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.918+0000 7f0ac8000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac00cd700 con 0x7f0ac01611b0 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.918+0000 7f0ac8000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac00b7230 con 0x7f0ac015f4d0 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 0x7f0ac01f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 0x7f0ac01f6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52390/0 (socket says 172.21.15.12:52390) 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac5600700 1 -- 172.21.15.12:0/3833122162 learned_addr learned my addr 172.21.15.12:0/3833122162 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac6000700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ac01611b0 0x7f0ac00cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac5600700 1 -- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac015f4d0 msgr2=0x7f0ac00ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac015f4d0 0x7f0ac00ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac5600700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac015f4d0 0x7f0ac00ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac5600700 1 -- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ac01611b0 msgr2=0x7f0ac00cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac5600700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ac01611b0 0x7f0ac00cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac5600700 1 -- 172.21.15.12:0/3833122162 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab8009b30 con 0x7f0ac00cee10 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac015f4d0 0x7f0ac00ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.919+0000 7f0ac5600700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 0x7f0ac01f6f40 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400d800 tx=0x7f0ab400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.920+0000 7f0abd600700 1 -- 172.21.15.12:0/3833122162 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab40125f0 con 0x7f0ac00cee10 2024-03-29T10:12:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.920+0000 7f0abd600700 1 -- 172.21.15.12:0/3833122162 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab4012750 con 0x7f0ac00cee10 2024-03-29T10:12:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.920+0000 7f0ac8000700 1 -- 172.21.15.12:0/3833122162 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac00b7510 con 0x7f0ac00cee10 2024-03-29T10:12:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.920+0000 7f0abd600700 1 -- 172.21.15.12:0/3833122162 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4012b00 con 0x7f0ac00cee10 2024-03-29T10:12:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.920+0000 7f0ac8000700 1 -- 172.21.15.12:0/3833122162 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac00cb770 con 0x7f0ac00cee10 2024-03-29T10:12:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.922+0000 7f0abd600700 1 -- 172.21.15.12:0/3833122162 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0ab4012c60 con 0x7f0ac00cee10 2024-03-29T10:12:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.923+0000 7f0abd600700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0aa006adb0 0x7f0aa006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.923+0000 7f0abd600700 1 -- 172.21.15.12:0/3833122162 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ab40898e0 con 0x7f0ac00cee10 2024-03-29T10:12:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.923+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0aa006adb0 0x7f0aa006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.924+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0aa006adb0 0x7f0aa006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0aac000f60 tx=0x7f0aac009040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.925+0000 7f0adbbb4b80 1 Processor -- start 2024-03-29T10:12:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.925+0000 7f0adbbb4b80 1 -- start start 2024-03-29T10:12:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125673ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561256711be0 0x561256714a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561256689650 con 0x56125670fb60 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125666ac30 con 0x56125673e830 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x561256668430 con 0x561256711be0 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0ac6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0ac5600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125673ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0ac6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52398/0 (socket says 172.21.15.12:52398) 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0ac5600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125673ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:55724/0 (socket says 172.21.15.12:55724) 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0ac6000700 1 -- 172.21.15.12:0/4221261030 learned_addr learned my addr 172.21.15.12:0/4221261030 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.926+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/4221261030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561256711be0 0x561256714a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.927+0000 7f0ac6000700 1 -- 172.21.15.12:0/4221261030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561256711be0 msgr2=0x561256714a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.927+0000 7f0ac6000700 1 --2- 172.21.15.12:0/4221261030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561256711be0 0x561256714a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.927+0000 7f0ac6000700 1 -- 172.21.15.12:0/4221261030 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 msgr2=0x56125673ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.927+0000 7f0ac6000700 1 --2- 172.21.15.12:0/4221261030 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125673ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.927+0000 7f0ac6000700 1 -- 172.21.15.12:0/4221261030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561256457790 con 0x56125670fb60 2024-03-29T10:12:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.927+0000 7f0ac6000700 1 --2- 172.21.15.12:0/4221261030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673ba20 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f0ac008ce10 tx=0x7f0ab8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9642b90c13fd0724 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.927+0000 7f0ab1c00700 1 -- 172.21.15.12:0/4221261030 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800bd30 con 0x56125670fb60 2024-03-29T10:12:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.927+0000 7f0ab1c00700 1 -- 172.21.15.12:0/4221261030 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab8003a20 con 0x56125670fb60 2024-03-29T10:12:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.927+0000 7f0ab1c00700 1 -- 172.21.15.12:0/4221261030 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8003d90 con 0x56125670fb60 2024-03-29T10:12:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.928+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/4221261030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 msgr2=0x56125673ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.928+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/4221261030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673ba20 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f0ac008ce10 tx=0x7f0ab8010000 comp rx=0 tx=0).stop 2024-03-29T10:12:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.928+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/4221261030 shutdown_connections 2024-03-29T10:12:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.928+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/4221261030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673ba20 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.928+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/4221261030 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125673ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.928+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/4221261030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561256711be0 0x561256714a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.928+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/4221261030 >> 172.21.15.12:0/4221261030 conn(0x5612566726a0 msgr2=0x561256759640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.928+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/4221261030 shutdown_connections 2024-03-29T10:12:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.928+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/4221261030 wait complete. 2024-03-29T10:12:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.929+0000 7f0adbbb4b80 1 -- start start 2024-03-29T10:12:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.929+0000 7f0adbbb4b80 1 Processor -- start 2024-03-29T10:12:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 0x561256769480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x561256741010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125673bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561256663d20 con 0x561256711be0 2024-03-29T10:12:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612565ff770 con 0x56125673e830 2024-03-29T10:12:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x5612566226f0 con 0x56125670fb60 2024-03-29T10:12:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0adbbb4b80 10 client.0 _ll_register_callbacks cb 0x5612566095e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:12:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0ac4c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x561256741010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0ac4c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x561256741010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52404/0 (socket says 172.21.15.12:52404) 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0ac4c00700 1 -- 172.21.15.12:0/1235284050 learned_addr learned my addr 172.21.15.12:0/1235284050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0ac5600700 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 0x561256769480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0ac6000700 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125673bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0ac4c00700 1 -- 172.21.15.12:0/1235284050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 msgr2=0x561256769480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 0x561256769480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0ac4c00700 1 -- 172.21.15.12:0/1235284050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 msgr2=0x56125673bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125673bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.931+0000 7f0ac4c00700 1 -- 172.21.15.12:0/1235284050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac00d2a40 con 0x561256711be0 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.932+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x561256741010 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0aac007480 tx=0x7f0aac001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab3118579bb1030 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.932+0000 7f0aa6000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.932+0000 7f0aa6000700 1 -- 172.21.15.12:0/1235284050 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac00de00 con 0x561256711be0 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.932+0000 7f0aa6000700 1 -- 172.21.15.12:0/1235284050 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0aac021aa0 con 0x561256711be0 2024-03-29T10:12:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.932+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1235284050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561256706920 con 0x561256711be0 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.932+0000 7f0aa6000700 1 -- 172.21.15.12:0/1235284050 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0aac012410 con 0x561256711be0 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.932+0000 7f0adbbb4b80 10 client.8052 fetch_fsmap learned FSMap version 55 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.932+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1235284050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x5612565440a0 con 0x561256711be0 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.932+0000 7f0aa6000700 1 -- 172.21.15.12:0/1235284050 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561256706920 con 0x561256711be0 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.933+0000 7f0aa6000700 1 -- 172.21.15.12:0/1235284050 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 55) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0aac01f070 con 0x561256711be0 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.933+0000 7f0adbbb4b80 10 client.8052 fetch_fsmap finished waiting for FSMap version 55 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.933+0000 7f0adbbb4b80 10 client.8052 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.933+0000 7f0adbbb4b80 20 client.8052 populate_metadata read hostname 'smithi012' 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.933+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.12:6834/753821395,v1:172.21.15.12:6835/753821395] conn(0x561256742400 0x561256759eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.933+0000 7f0adbbb4b80 4 client.8052 mds_command: new command op to 7917 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.933+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1235284050 --> [v2:172.21.15.12:6834/753821395,v1:172.21.15.12:6835/753821395] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5612566e36f0 con 0x561256742400 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.933+0000 7f0ac5600700 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.12:6834/753821395,v1:172.21.15.12:6835/753821395] conn(0x561256742400 0x561256759eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.933+0000 7f0ac5600700 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.12:6834/753821395,v1:172.21.15.12:6835/753821395] conn(0x561256742400 0x561256759eb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4979db24985e5804 server_cookie=f5e47cacffdc5f5e in_seq=0 out_seq=0 2024-03-29T10:12:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.933+0000 7f0aa6000700 10 client.8052 ms_handle_connect on v2:172.21.15.12:6834/753821395 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0aa6000700 1 -- 172.21.15.12:0/1235284050 <== mds.0 v2:172.21.15.12:6834/753821395 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5612566e36f0 con 0x561256742400 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0aa6000700 10 client.8052 handle_command_reply: tid=0 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0adbbb4b80 1 client.8052 shutdown 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1235284050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 msgr2=0x561256741010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x561256741010 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0aac007480 tx=0x7f0aac001080 comp rx=0 tx=0).stop 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1235284050 shutdown_connections 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.12:6834/753821395,v1:172.21.15.12:6835/753821395] conn(0x561256742400 0x561256759eb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125673bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 0x561256769480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0aa6000700 0 client.8052 ms_handle_reset on v2:172.21.15.12:6834/753821395 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1235284050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x561256741010 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.934+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1235284050 >> 172.21.15.12:0/1235284050 conn(0x5612566726a0 msgr2=0x561256713e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.935+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1235284050 shutdown_connections 2024-03-29T10:12:53.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.935+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1235284050 wait complete. 2024-03-29T10:12:53.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.935+0000 7f0adbbb4b80 20 client.8052 trim_cache size 0 max 16384 2024-03-29T10:12:53.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.953+0000 7f0adbbb4b80 1 Processor -- start 2024-03-29T10:12:53.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.953+0000 7f0adbbb4b80 1 -- start start 2024-03-29T10:12:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561256711be0 0x56125670a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125673e830 0x561256713970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561256458390 con 0x56125670fb60 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125673c090 con 0x561256711be0 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56125670f200 con 0x56125673e830 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52416/0 (socket says 172.21.15.12:52416) 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac5600700 1 -- 172.21.15.12:0/3822019127 learned_addr learned my addr 172.21.15.12:0/3822019127 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac6000700 1 --2- 172.21.15.12:0/3822019127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125673e830 0x561256713970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/3822019127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561256711be0 0x56125670a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac5600700 1 -- 172.21.15.12:0/3822019127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125673e830 msgr2=0x561256713970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac5600700 1 --2- 172.21.15.12:0/3822019127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125673e830 0x561256713970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac5600700 1 -- 172.21.15.12:0/3822019127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561256711be0 msgr2=0x56125670a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac5600700 1 --2- 172.21.15.12:0/3822019127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561256711be0 0x56125670a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac5600700 1 -- 172.21.15.12:0/3822019127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56125673d0e0 con 0x56125670fb60 2024-03-29T10:12:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.954+0000 7f0ac6000700 1 --2- 172.21.15.12:0/3822019127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125673e830 0x561256713970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0ac5600700 1 --2- 172.21.15.12:0/3822019127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673dbf0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4060490 tx=0x7f0ab4058110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19eda3ae997439d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0ab2600700 1 -- 172.21.15.12:0/3822019127 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab40609c0 con 0x56125670fb60 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0ab2600700 1 -- 172.21.15.12:0/3822019127 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab4056970 con 0x56125670fb60 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0ab2600700 1 -- 172.21.15.12:0/3822019127 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4087f80 con 0x56125670fb60 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/3822019127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 msgr2=0x56125673dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/3822019127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673dbf0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4060490 tx=0x7f0ab4058110 comp rx=0 tx=0).stop 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/3822019127 shutdown_connections 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/3822019127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125673e830 0x561256713970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/3822019127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125670fb60 0x56125673dbf0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/3822019127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561256711be0 0x56125670a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.955+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/3822019127 >> 172.21.15.12:0/3822019127 conn(0x5612566726a0 msgr2=0x56125670e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.956+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/3822019127 shutdown_connections 2024-03-29T10:12:53.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.956+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/3822019127 wait complete. 2024-03-29T10:12:53.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.956+0000 7f0adbbb4b80 1 -- start start 2024-03-29T10:12:53.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.956+0000 7f0adbbb4b80 1 Processor -- start 2024-03-29T10:12:53.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.957+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 0x5612567c1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.957+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x56125675ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.957+0000 7f0adbbb4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125675b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.957+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561256767c70 con 0x561256711be0 2024-03-29T10:12:53.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.957+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x561256714290 con 0x56125673e830 2024-03-29T10:12:53.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.957+0000 7f0adbbb4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56125673a380 con 0x56125670fb60 2024-03-29T10:12:53.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.957+0000 7f0adbbb4b80 10 client.0 _ll_register_callbacks cb 0x7f0aac005c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:12:53.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac4c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x56125675ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac4c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x56125675ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52418/0 (socket says 172.21.15.12:52418) 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac4c00700 1 -- 172.21.15.12:0/1343462885 learned_addr learned my addr 172.21.15.12:0/1343462885 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac6000700 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125675b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac5600700 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 0x5612567c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac4c00700 1 -- 172.21.15.12:0/1343462885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 msgr2=0x5612567c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 0x5612567c1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac4c00700 1 -- 172.21.15.12:0/1343462885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 msgr2=0x56125675b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125675b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac4c00700 1 -- 172.21.15.12:0/1343462885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac00cb770 con 0x561256711be0 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac5600700 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 0x5612567c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac6000700 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125675b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0ac4c00700 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x56125675ae80 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f0aac007480 tx=0x7f0aac012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8152af45b1998ac9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.958+0000 7f0aa6000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:12:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0aa6000700 1 -- 172.21.15.12:0/1343462885 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac012c00 con 0x561256711be0 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1343462885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ab8003a20 con 0x561256711be0 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0aa6000700 1 -- 172.21.15.12:0/1343462885 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0aac000f90 con 0x561256711be0 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0aa6000700 1 -- 172.21.15.12:0/1343462885 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0aac001cc0 con 0x561256711be0 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0adbbb4b80 10 client.8058 fetch_fsmap learned FSMap version 55 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1343462885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x56125662a010 con 0x561256711be0 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0aa6000700 1 -- 172.21.15.12:0/1343462885 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab8003a20 con 0x561256711be0 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0aa6000700 1 -- 172.21.15.12:0/1343462885 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 55) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0aac035440 con 0x561256711be0 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0adbbb4b80 10 client.8058 fetch_fsmap finished waiting for FSMap version 55 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0adbbb4b80 10 client.8058 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0adbbb4b80 20 client.8058 populate_metadata read hostname 'smithi012' 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.12:6834/753821395,v1:172.21.15.12:6835/753821395] conn(0x56125673a960 0x56125673ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0adbbb4b80 4 client.8058 mds_command: new command op to 7917 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T10:12:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.959+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1343462885 --> [v2:172.21.15.12:6834/753821395,v1:172.21.15.12:6835/753821395] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56125649d040 con 0x56125673a960 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.960+0000 7f0ac5600700 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.12:6834/753821395,v1:172.21.15.12:6835/753821395] conn(0x56125673a960 0x56125673ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.960+0000 7f0ac5600700 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.12:6834/753821395,v1:172.21.15.12:6835/753821395] conn(0x56125673a960 0x56125673ad40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a759948336f79c9 server_cookie=3fdb578a3333f256 in_seq=0 out_seq=0 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.960+0000 7f0aa6000700 10 client.8058 ms_handle_connect on v2:172.21.15.12:6834/753821395 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.960+0000 7f0aa6000700 1 -- 172.21.15.12:0/1343462885 <== mds.0 v2:172.21.15.12:6834/753821395 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x56125649d040 con 0x56125673a960 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.960+0000 7f0aa6000700 10 client.8058 handle_command_reply: tid=0 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.960+0000 7f0adbbb4b80 1 client.8058 shutdown 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.960+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1343462885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 msgr2=0x56125675ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.960+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x56125675ae80 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f0aac007480 tx=0x7f0aac012000 comp rx=0 tx=0).stop 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.960+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1343462885 shutdown_connections 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.960+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.12:6834/753821395,v1:172.21.15.12:6835/753821395] conn(0x56125673a960 0x56125673ad40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56125673e830 0x56125675b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0aa6000700 0 client.8058 ms_handle_reset on v2:172.21.15.12:6834/753821395 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56125670fb60 0x5612567c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0adbbb4b80 1 --2- 172.21.15.12:0/1343462885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561256711be0 0x56125675ae80 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1343462885 >> 172.21.15.12:0/1343462885 conn(0x5612566726a0 msgr2=0x561256715400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1343462885 shutdown_connections 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0adbbb4b80 1 -- 172.21.15.12:0/1343462885 wait complete. 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0adbbb4b80 20 client.8058 trim_cache size 0 max 16384 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 7917, 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1480006180000002, 2024-03-29T10:12:53.967 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 55, 2024-03-29T10:12:53.967 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 124, 2024-03-29T10:12:53.967 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 124, 2024-03-29T10:12:53.967 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12.205309894000001 2024-03-29T10:12:53.967 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0ac8000700 1 -- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0aa006adb0 msgr2=0x7f0aa006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0ac8000700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0aa006adb0 0x7f0aa006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0aac000f60 tx=0x7f0aac009040 comp rx=0 tx=0).stop 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0ac8000700 1 -- 172.21.15.12:0/3833122162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 msgr2=0x7f0ac01f6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.961+0000 7f0ac8000700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 0x7f0ac01f6f40 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400d800 tx=0x7f0ab400dbc0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.962+0000 7f0ac8000700 1 -- 172.21.15.12:0/3833122162 shutdown_connections 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.962+0000 7f0ac8000700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0aa006adb0 0x7f0aa006d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.962+0000 7f0ac8000700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ac01611b0 0x7f0ac00cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.962+0000 7f0ac8000700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ac00cee10 0x7f0ac01f6f40 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.962+0000 7f0ac8000700 1 --2- 172.21.15.12:0/3833122162 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac015f4d0 0x7f0ac00ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.962+0000 7f0ac8000700 1 -- 172.21.15.12:0/3833122162 >> 172.21.15.12:0/3833122162 conn(0x7f0ac008aff0 msgr2=0x7f0ac006bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.962+0000 7f0ac8000700 1 -- 172.21.15.12:0/3833122162 shutdown_connections 2024-03-29T10:12:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:53.962+0000 7f0ac8000700 1 -- 172.21.15.12:0/3833122162 wait complete. 2024-03-29T10:12:53.979 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T10:12:54.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.179+0000 7f959e200700 1 Processor -- start 2024-03-29T10:12:54.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.179+0000 7f959e200700 1 -- start start 2024-03-29T10:12:54.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.180+0000 7f959e200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9598160f10 0x7f95980d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:54.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.180+0000 7f959e200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95980cf230 0x7f95980d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:54.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.180+0000 7f959e200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9598161590 0x7f9598161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:54.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.180+0000 7f959e200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9598161f60 con 0x7f9598161590 2024-03-29T10:12:54.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.180+0000 7f959e200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95980ceda0 con 0x7f9598160f10 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.180+0000 7f959e200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95980d23d0 con 0x7f95980cf230 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9597e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9598161590 0x7f9598161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9597e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9598161590 0x7f9598161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52430/0 (socket says 172.21.15.12:52430) 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9597e00700 1 -- 172.21.15.12:0/2319460844 learned_addr learned my addr 172.21.15.12:0/2319460844 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9596a00700 1 --2- 172.21.15.12:0/2319460844 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95980cf230 0x7f95980d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9597400700 1 --2- 172.21.15.12:0/2319460844 >> v2:172.21.15.159:3300/0 conn(0x7f9598160f10 0x7f95980d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9597e00700 1 -- 172.21.15.12:0/2319460844 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95980cf230 msgr2=0x7f95980d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9597e00700 1 --2- 172.21.15.12:0/2319460844 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95980cf230 0x7f95980d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9597e00700 1 -- 172.21.15.12:0/2319460844 >> v2:172.21.15.159:3300/0 conn(0x7f9598160f10 msgr2=0x7f95980d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9597e00700 1 --2- 172.21.15.12:0/2319460844 >> v2:172.21.15.159:3300/0 conn(0x7f9598160f10 0x7f95980d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9597e00700 1 -- 172.21.15.12:0/2319460844 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95981612f0 con 0x7f9598161590 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9596a00700 1 --2- 172.21.15.12:0/2319460844 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95980cf230 0x7f95980d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.181+0000 7f9597e00700 1 --2- 172.21.15.12:0/2319460844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9598161590 0x7f9598161a20 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f95980d1080 tx=0x7f959000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d99a91c301213b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:54.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f9596000700 1 -- 172.21.15.12:0/2319460844 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959000e070 con 0x7f9598161590 2024-03-29T10:12:54.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f9596000700 1 -- 172.21.15.12:0/2319460844 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959000b9c0 con 0x7f9598161590 2024-03-29T10:12:54.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f9596000700 1 -- 172.21.15.12:0/2319460844 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959000bd70 con 0x7f9598161590 2024-03-29T10:12:54.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f959e200700 1 -- 172.21.15.12:0/2319460844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9598161590 msgr2=0x7f9598161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:54.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f959e200700 1 --2- 172.21.15.12:0/2319460844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9598161590 0x7f9598161a20 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f95980d1080 tx=0x7f959000b330 comp rx=0 tx=0).stop 2024-03-29T10:12:54.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f959e200700 1 -- 172.21.15.12:0/2319460844 shutdown_connections 2024-03-29T10:12:54.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f959e200700 1 --2- 172.21.15.12:0/2319460844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9598161590 0x7f9598161a20 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f959e200700 1 --2- 172.21.15.12:0/2319460844 >> v2:172.21.15.159:3300/0 conn(0x7f9598160f10 0x7f95980d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f959e200700 1 --2- 172.21.15.12:0/2319460844 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95980cf230 0x7f95980d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f959e200700 1 -- 172.21.15.12:0/2319460844 >> 172.21.15.12:0/2319460844 conn(0x7f959808aff0 msgr2=0x7f95980d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:54.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f959e200700 1 -- 172.21.15.12:0/2319460844 shutdown_connections 2024-03-29T10:12:54.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.182+0000 7f959e200700 1 -- 172.21.15.12:0/2319460844 wait complete. 2024-03-29T10:12:54.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.184+0000 7f959e200700 1 Processor -- start 2024-03-29T10:12:54.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.184+0000 7f959e200700 1 -- start start 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.185+0000 7f959e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95980cf230 0x7f95980b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.185+0000 7f959e200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9598160f10 0x7f959815fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.185+0000 7f959e200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9598161590 0x7f95981f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.185+0000 7f959e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95980b89d0 con 0x7f95980cf230 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.185+0000 7f959e200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95981f6550 con 0x7f9598161590 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.185+0000 7f959e200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9598160110 con 0x7f9598160f10 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.185+0000 7f9597400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95980cf230 0x7f95980b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.185+0000 7f9597400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95980cf230 0x7f95980b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52432/0 (socket says 172.21.15.12:52432) 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.185+0000 7f9597400700 1 -- 172.21.15.12:0/2474051138 learned_addr learned my addr 172.21.15.12:0/2474051138 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f9597e00700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9598161590 0x7f95981f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f9596a00700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9598160f10 0x7f959815fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:54.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f9597400700 1 -- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9598160f10 msgr2=0x7f959815fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f9597400700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9598160f10 0x7f959815fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f9597400700 1 -- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9598161590 msgr2=0x7f95981f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f9597400700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9598161590 0x7f95981f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f9597400700 1 -- 172.21.15.12:0/2474051138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9590009b30 con 0x7f95980cf230 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f9597e00700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9598161590 0x7f95981f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f9597400700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95980cf230 0x7f95980b8490 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f958800d800 tx=0x7f958800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f958f400700 1 -- 172.21.15.12:0/2474051138 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95880125f0 con 0x7f95980cf230 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f958f400700 1 -- 172.21.15.12:0/2474051138 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9588012750 con 0x7f95980cf230 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f958f400700 1 -- 172.21.15.12:0/2474051138 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9588012b00 con 0x7f95980cf230 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.186+0000 7f959e200700 1 -- 172.21.15.12:0/2474051138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f959815e460 con 0x7f95980cf230 2024-03-29T10:12:54.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.187+0000 7f959e200700 1 -- 172.21.15.12:0/2474051138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959815ea20 con 0x7f95980cf230 2024-03-29T10:12:54.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.187+0000 7f959e200700 1 -- 172.21.15.12:0/2474051138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9598067720 con 0x7f95980cf230 2024-03-29T10:12:54.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.188+0000 7f958f400700 1 -- 172.21.15.12:0/2474051138 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9588012c60 con 0x7f95980cf230 2024-03-29T10:12:54.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.189+0000 7f958f400700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f957806adb0 0x7f957806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:54.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.189+0000 7f958f400700 1 -- 172.21.15.12:0/2474051138 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95880897b0 con 0x7f95980cf230 2024-03-29T10:12:54.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.190+0000 7f9596a00700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f957806adb0 0x7f957806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:54.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.190+0000 7f9596a00700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f957806adb0 0x7f957806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f95980a15b0 tx=0x7f9584008040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:54.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.194+0000 7f958f400700 1 -- 172.21.15.12:0/2474051138 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9588054880 con 0x7f95980cf230 2024-03-29T10:12:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.460+0000 7f959e200700 1 -- 172.21.15.12:0/2474051138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f95980d1de0 con 0x7f95980cf230 2024-03-29T10:12:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.460+0000 7f958f400700 1 -- 172.21.15.12:0/2474051138 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v55) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9588057ea0 con 0x7f95980cf230 2024-03-29T10:12:54.461 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T10:12:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 -- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f957806adb0 msgr2=0x7f957806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f957806adb0 0x7f957806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f95980a15b0 tx=0x7f9584008040 comp rx=0 tx=0).stop 2024-03-29T10:12:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 -- 172.21.15.12:0/2474051138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95980cf230 msgr2=0x7f95980b8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95980cf230 0x7f95980b8490 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f958800d800 tx=0x7f958800dbc0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 -- 172.21.15.12:0/2474051138 shutdown_connections 2024-03-29T10:12:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f957806adb0 0x7f957806d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9598161590 0x7f95981f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95980cf230 0x7f95980b8490 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 --2- 172.21.15.12:0/2474051138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9598160f10 0x7f959815fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 -- 172.21.15.12:0/2474051138 >> 172.21.15.12:0/2474051138 conn(0x7f959808aff0 msgr2=0x7f95980cad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 -- 172.21.15.12:0/2474051138 shutdown_connections 2024-03-29T10:12:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:54.463+0000 7f959e200700 1 -- 172.21.15.12:0/2474051138 wait complete. 2024-03-29T10:12:54.482 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T10:12:54.482 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T10:12:54.482 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T10:12:54.482 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T10:12:54.482 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T10:12:54.482 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T10:12:54.482 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T10:12:54.483 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T10:12:54.483 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:12:54.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:54.483 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T10:12:54.529 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T10:12:54.529 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T10:12:54.529 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T10:12:54.529 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:12:54.529 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T10:12:54.529 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:12:54.529 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 40343sec preferred_lft 40343sec 2024-03-29T10:12:54.530 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T10:12:54.531 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:12:54.532 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:12:54.532 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T10:12:54.532 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T10:12:54.532 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:12:54.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:54.532 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T10:12:54.634 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T10:12:54.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:54.638 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T10:12:54.685 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T10:12:54.685 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T10:12:54.685 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T10:12:54.685 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T10:12:54.685 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T10:12:54.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:54.686 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T10:12:54.686 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T10:12:54.686 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T10:12:54.686 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T10:12:54.686 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T10:12:54.686 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T10:12:54.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:54.852 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T10:12:54.897 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T10:12:54.898 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T10:12:54.898 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:12:54.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:54.898 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T10:12:55.005 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:12:55.005 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T10:12:55.050 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:12:55.050 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:12:55.051 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:12:55.136 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T10:12:55.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:55.137 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:12:55.291 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:12:55.291 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:12:55.317 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:12:55.317 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:12:55.317 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T10:12:55.318 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T10:12:55.318 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T10:12:55.321 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T10:12:55.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:55.322 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:12:55.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:55.390 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T10:12:55.491 INFO:teuthology.orchestra.run.smithi012.stdout:client7777 2024-03-29T10:12:55.492 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:12:55.492 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:12:55.537 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T10:12:55.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.779+0000 7f3e0f400700 1 Processor -- start 2024-03-29T10:12:55.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.779+0000 7f3e0f400700 1 -- start start 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e0f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080cd020 0x7f3e080cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e0f400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3e080b2070 0x7f3e080b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e0f400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e080cdae0 con 0x7f3e080cbb30 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e0f400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e080cddc0 con 0x7f3e080b2070 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e0f400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080ce0a0 con 0x7f3e080cd020 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e0ca00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3e080cbb30 0x7f3e080cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47496/0 (socket says 172.21.15.12:47496) 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e0ca00700 1 -- 172.21.15.12:0/550610037 learned_addr learned my addr 172.21.15.12:0/550610037 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e07e00700 1 --2- 172.21.15.12:0/550610037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080cd020 0x7f3e080cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e0d400700 1 --2- 172.21.15.12:0/550610037 >> v2:172.21.15.159:3300/0 conn(0x7f3e080b2070 0x7f3e080b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e07400700 1 -- 172.21.15.12:0/550610037 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137338844 0 0) 0x7f3e080cdae0 con 0x7f3e080cbb30 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.781+0000 7f3e07400700 1 -- 172.21.15.12:0/550610037 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df00035b0 con 0x7f3e080cbb30 2024-03-29T10:12:55.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.782+0000 7f3e07e00700 1 -- 172.21.15.12:0/550610037 >> v2:172.21.15.159:3300/0 conn(0x7f3e080b2070 msgr2=0x7f3e080b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.782+0000 7f3e07e00700 1 --2- 172.21.15.12:0/550610037 >> v2:172.21.15.159:3300/0 conn(0x7f3e080b2070 0x7f3e080b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.782+0000 7f3e07e00700 1 -- 172.21.15.12:0/550610037 >> v1:172.21.15.12:6789/0 conn(0x7f3e080cbb30 legacy=0x7f3e080cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.782+0000 7f3e07e00700 1 -- 172.21.15.12:0/550610037 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0806dc20 con 0x7f3e080cd020 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.782+0000 7f3e0d400700 1 --2- 172.21.15.12:0/550610037 >> v2:172.21.15.159:3300/0 conn(0x7f3e080b2070 0x7f3e080b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.782+0000 7f3e07e00700 1 --2- 172.21.15.12:0/550610037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080cd020 0x7f3e080cd480 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009840 tx=0x7f3df4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d8f3f1298b0bd88 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.782+0000 7f3e07400700 1 -- 172.21.15.12:0/550610037 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4011830 con 0x7f3e080cd020 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.782+0000 7f3e07400700 1 -- 172.21.15.12:0/550610037 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df4011990 con 0x7f3e080cd020 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.782+0000 7f3e07400700 1 -- 172.21.15.12:0/550610037 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4011d20 con 0x7f3e080cd020 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.783+0000 7f3e0f400700 1 -- 172.21.15.12:0/550610037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080cd020 msgr2=0x7f3e080cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.783+0000 7f3e0f400700 1 --2- 172.21.15.12:0/550610037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080cd020 0x7f3e080cd480 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009840 tx=0x7f3df4009b50 comp rx=0 tx=0).stop 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.783+0000 7f3e0f400700 1 -- 172.21.15.12:0/550610037 shutdown_connections 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.783+0000 7f3e0f400700 1 --2- 172.21.15.12:0/550610037 >> v2:172.21.15.159:3300/0 conn(0x7f3e080b2070 0x7f3e080b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.783+0000 7f3e0f400700 1 --2- 172.21.15.12:0/550610037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080cd020 0x7f3e080cd480 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.783+0000 7f3e0f400700 1 -- 172.21.15.12:0/550610037 >> 172.21.15.12:0/550610037 conn(0x7f3e080b0d70 msgr2=0x7f3e080b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:55.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.783+0000 7f3e0f400700 1 -- 172.21.15.12:0/550610037 shutdown_connections 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.783+0000 7f3e0f400700 1 -- 172.21.15.12:0/550610037 wait complete. 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.785+0000 7f3e0f400700 1 Processor -- start 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.785+0000 7f3e0f400700 1 -- start start 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e0f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080b2070 0x7f3e08154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e0f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e080cbb30 0x7f3e0815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e0f400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e080cd020 0x7f3e0815e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e0f400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e08155480 con 0x7f3e080cbb30 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e0f400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0815a720 con 0x7f3e080cd020 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e0f400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e081552a0 con 0x7f3e080b2070 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e07e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e080cbb30 0x7f3e0815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e07e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e080cbb30 0x7f3e0815dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52446/0 (socket says 172.21.15.12:52446) 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e07e00700 1 -- 172.21.15.12:0/3507222095 learned_addr learned my addr 172.21.15.12:0/3507222095 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:55.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e0d400700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e080cd020 0x7f3e0815e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.786+0000 7f3e0ca00700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080b2070 0x7f3e08154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e07e00700 1 -- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080b2070 msgr2=0x7f3e08154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e07e00700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080b2070 0x7f3e08154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e07e00700 1 -- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e080cd020 msgr2=0x7f3e0815e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e07e00700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e080cd020 0x7f3e0815e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e07e00700 1 -- 172.21.15.12:0/3507222095 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00009a70 con 0x7f3e080cbb30 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e0d400700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e080cd020 0x7f3e0815e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e0ca00700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080b2070 0x7f3e08154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e07e00700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e080cbb30 0x7f3e0815dc00 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009fd0 tx=0x7f3df4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e04c00700 1 -- 172.21.15.12:0/3507222095 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df40036a0 con 0x7f3e080cbb30 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e04c00700 1 -- 172.21.15.12:0/3507222095 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df401e070 con 0x7f3e080cbb30 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e04c00700 1 -- 172.21.15.12:0/3507222095 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4003a60 con 0x7f3e080cbb30 2024-03-29T10:12:55.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e0f400700 1 -- 172.21.15.12:0/3507222095 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df40094a0 con 0x7f3e080cbb30 2024-03-29T10:12:55.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.787+0000 7f3e0f400700 1 -- 172.21.15.12:0/3507222095 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e081556d0 con 0x7f3e080cbb30 2024-03-29T10:12:55.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.788+0000 7f3e0f400700 1 -- 172.21.15.12:0/3507222095 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e080cafc0 con 0x7f3e080cbb30 2024-03-29T10:12:55.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.789+0000 7f3e04c00700 1 -- 172.21.15.12:0/3507222095 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3df4003bc0 con 0x7f3e080cbb30 2024-03-29T10:12:55.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.790+0000 7f3e04c00700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3de806adb0 0x7f3de806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:55.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.790+0000 7f3e04c00700 1 -- 172.21.15.12:0/3507222095 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3df408a3b0 con 0x7f3e080cbb30 2024-03-29T10:12:55.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.790+0000 7f3e0ca00700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3de806adb0 0x7f3de806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:55.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.791+0000 7f3e0ca00700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3de806adb0 0x7f3de806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0011e0 tx=0x7f3dfc009040 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:55.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:55.794+0000 7f3e04c00700 1 -- 172.21.15.12:0/3507222095 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3df4055470 con 0x7f3e080cbb30 2024-03-29T10:12:56.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.038+0000 7f3e0f400700 1 -- 172.21.15.12:0/3507222095 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e081594a0 con 0x7f3de806adb0 2024-03-29T10:12:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.040+0000 7f3e04c00700 1 -- 172.21.15.12:0/3507222095 <== mgr.7402 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3dfc001c90 con 0x7f3de806adb0 2024-03-29T10:12:56.040 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:12:56.040 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:12:56.040 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T10:12:56.040 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:12:56.040 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:12:56.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 -- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3de806adb0 msgr2=0x7f3de806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:56.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3de806adb0 0x7f3de806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0011e0 tx=0x7f3dfc009040 comp rx=0 tx=0).stop 2024-03-29T10:12:56.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 -- 172.21.15.12:0/3507222095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e080cbb30 msgr2=0x7f3e0815dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:56.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e080cbb30 0x7f3e0815dc00 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009fd0 tx=0x7f3df4004750 comp rx=0 tx=0).stop 2024-03-29T10:12:56.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 -- 172.21.15.12:0/3507222095 shutdown_connections 2024-03-29T10:12:56.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3de806adb0 0x7f3de806d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:56.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e080cd020 0x7f3e0815e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e080b2070 0x7f3e08154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 --2- 172.21.15.12:0/3507222095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e080cbb30 0x7f3e0815dc00 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 -- 172.21.15.12:0/3507222095 >> 172.21.15.12:0/3507222095 conn(0x7f3e080b0d70 msgr2=0x7f3e080cbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 -- 172.21.15.12:0/3507222095 shutdown_connections 2024-03-29T10:12:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.042+0000 7f3e0f400700 1 -- 172.21.15.12:0/3507222095 wait complete. 2024-03-29T10:12:56.060 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T10:12:56.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.260+0000 7f5b92e00700 1 Processor -- start 2024-03-29T10:12:56.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.260+0000 7f5b92e00700 1 -- start start 2024-03-29T10:12:56.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.261+0000 7f5b92e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5b8c0d21f0 0x7f5b8c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:56.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.261+0000 7f5b92e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5b8c0cec40 0x7f5b8c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:56.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.261+0000 7f5b92e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8c0ce270 con 0x7f5b8c1611b0 2024-03-29T10:12:56.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.261+0000 7f5b92e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b8c15b600 con 0x7f5b8c0cec40 2024-03-29T10:12:56.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.261+0000 7f5b92e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b8c0d2760 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.261+0000 7f5b90e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5b8c0cec40 0x7f5b8c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:56.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.262+0000 7f5b90e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5b8c0cec40 0x7f5b8c161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52452/0 (socket says 172.21.15.12:52452) 2024-03-29T10:12:56.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.262+0000 7f5b8be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5b8c1611b0 0x7f5b8c15ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47512/0 (socket says 172.21.15.12:47512) 2024-03-29T10:12:56.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.262+0000 7f5b90e00700 1 -- 172.21.15.12:0/432450744 learned_addr learned my addr 172.21.15.12:0/432450744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:56.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.262+0000 7f5b8b400700 1 --2- 172.21.15.12:0/432450744 >> v2:172.21.15.159:3300/0 conn(0x7f5b8c0d21f0 0x7f5b8c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:56.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.262+0000 7f5b90e00700 1 -- 172.21.15.12:0/432450744 >> v2:172.21.15.159:3300/0 conn(0x7f5b8c0d21f0 msgr2=0x7f5b8c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:56.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.262+0000 7f5b90e00700 1 --2- 172.21.15.12:0/432450744 >> v2:172.21.15.159:3300/0 conn(0x7f5b8c0d21f0 0x7f5b8c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:56.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.262+0000 7f5b90e00700 1 -- 172.21.15.12:0/432450744 >> v1:172.21.15.12:6789/0 conn(0x7f5b8c1611b0 legacy=0x7f5b8c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:56.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.262+0000 7f5b90e00700 1 -- 172.21.15.12:0/432450744 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8c0d2a40 con 0x7f5b8c0cec40 2024-03-29T10:12:56.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.262+0000 7f5b8aa00700 1 -- 172.21.15.12:0/432450744 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999878147 0 0) 0x7f5b8c0ce270 con 0x7f5b8c1611b0 2024-03-29T10:12:56.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.262+0000 7f5b90e00700 1 --2- 172.21.15.12:0/432450744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0cec40 0x7f5b8c161590 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c08c030 tx=0x7f5b8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0a42b4991d0a89 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:56.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b8aa00700 1 -- 172.21.15.12:0/432450744 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400e070 con 0x7f5b8c0cec40 2024-03-29T10:12:56.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b8aa00700 1 -- 172.21.15.12:0/432450744 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8400b9c0 con 0x7f5b8c0cec40 2024-03-29T10:12:56.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b8aa00700 1 -- 172.21.15.12:0/432450744 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400bd70 con 0x7f5b8c0cec40 2024-03-29T10:12:56.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b92e00700 1 -- 172.21.15.12:0/432450744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0cec40 msgr2=0x7f5b8c161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:56.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b92e00700 1 --2- 172.21.15.12:0/432450744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0cec40 0x7f5b8c161590 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c08c030 tx=0x7f5b8400b330 comp rx=0 tx=0).stop 2024-03-29T10:12:56.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b92e00700 1 -- 172.21.15.12:0/432450744 shutdown_connections 2024-03-29T10:12:56.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b92e00700 1 --2- 172.21.15.12:0/432450744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0cec40 0x7f5b8c161590 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:56.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b92e00700 1 --2- 172.21.15.12:0/432450744 >> v2:172.21.15.159:3300/0 conn(0x7f5b8c0d21f0 0x7f5b8c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:56.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b92e00700 1 -- 172.21.15.12:0/432450744 >> 172.21.15.12:0/432450744 conn(0x7f5b8c08aff0 msgr2=0x7f5b8c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:56.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b92e00700 1 -- 172.21.15.12:0/432450744 shutdown_connections 2024-03-29T10:12:56.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.263+0000 7f5b92e00700 1 -- 172.21.15.12:0/432450744 wait complete. 2024-03-29T10:12:56.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.265+0000 7f5b92e00700 1 Processor -- start 2024-03-29T10:12:56.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.265+0000 7f5b92e00700 1 -- start start 2024-03-29T10:12:56.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.266+0000 7f5b92e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b8c0cec40 0x7f5b8c1f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.266+0000 7f5b92e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0d21f0 0x7f5b8c155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.266+0000 7f5b92e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b8c1611b0 0x7f5b8c0b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.266+0000 7f5b92e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c154ed0 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.266+0000 7f5b92e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c1551b0 con 0x7f5b8c1611b0 2024-03-29T10:12:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.266+0000 7f5b92e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c155490 con 0x7f5b8c0cec40 2024-03-29T10:12:56.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.266+0000 7f5b8b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0d21f0 0x7f5b8c155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:56.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.266+0000 7f5b8b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0d21f0 0x7f5b8c155f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52464/0 (socket says 172.21.15.12:52464) 2024-03-29T10:12:56.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.266+0000 7f5b8b400700 1 -- 172.21.15.12:0/1998733138 learned_addr learned my addr 172.21.15.12:0/1998733138 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:56.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b8be00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b8c0cec40 0x7f5b8c1f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:56.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b90e00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b8c1611b0 0x7f5b8c0b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:56.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b8b400700 1 -- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b8c0cec40 msgr2=0x7f5b8c1f27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:56.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b8b400700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b8c0cec40 0x7f5b8c1f27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:56.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b8b400700 1 -- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b8c1611b0 msgr2=0x7f5b8c0b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:56.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b8b400700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b8c1611b0 0x7f5b8c0b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:56.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b8b400700 1 -- 172.21.15.12:0/1998733138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b84009b30 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b90e00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b8c1611b0 0x7f5b8c0b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:12:56.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b8b400700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0d21f0 0x7f5b8c155f70 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800e810 tx=0x7f5b7800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:56.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b83e00700 1 -- 172.21.15.12:0/1998733138 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78013610 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b83e00700 1 -- 172.21.15.12:0/1998733138 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b78013770 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.267+0000 7f5b92e00700 1 -- 172.21.15.12:0/1998733138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8c155770 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.268+0000 7f5b83e00700 1 -- 172.21.15.12:0/1998733138 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78013b20 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.268+0000 7f5b92e00700 1 -- 172.21.15.12:0/1998733138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8c154b10 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.269+0000 7f5b92e00700 1 -- 172.21.15.12:0/1998733138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8c155c90 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.270+0000 7f5b83e00700 1 -- 172.21.15.12:0/1998733138 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 57) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5b78013c80 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.271+0000 7f5b83e00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5b6c06adb0 0x7f5b6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:56.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.271+0000 7f5b83e00700 1 -- 172.21.15.12:0/1998733138 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b78089270 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.271+0000 7f5b8be00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5b6c06adb0 0x7f5b6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:56.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.272+0000 7f5b8be00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5b6c06adb0 0x7f5b6c06d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c000f60 tx=0x7f5b7c007c90 comp rx=0 tx=0).ready entity=mgr.7402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:56.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.275+0000 7f5b83e00700 1 -- 172.21.15.12:0/1998733138 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5b8c155c90 con 0x7f5b8c0d21f0 2024-03-29T10:12:56.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:56.553+0000 7f5b92e00700 1 -- 172.21.15.12:0/1998733138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5b8c155c90 con 0x7f5b8c0d21f0 2024-03-29T10:12:57.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.323+0000 7f5b83e00700 1 -- 172.21.15.12:0/1998733138 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 58) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f5b78054570 con 0x7f5b8c0d21f0 2024-03-29T10:12:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.324+0000 7f5b83e00700 1 -- 172.21.15.12:0/1998733138 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v58) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5b78057c30 con 0x7f5b8c0d21f0 2024-03-29T10:12:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 -- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5b6c06adb0 msgr2=0x7f5b6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5b6c06adb0 0x7f5b6c06d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c000f60 tx=0x7f5b7c007c90 comp rx=0 tx=0).stop 2024-03-29T10:12:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 -- 172.21.15.12:0/1998733138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0d21f0 msgr2=0x7f5b8c155f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0d21f0 0x7f5b8c155f70 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800e810 tx=0x7f5b7800ebd0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 -- 172.21.15.12:0/1998733138 shutdown_connections 2024-03-29T10:12:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5b6c06adb0 0x7f5b6c06d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b8c1611b0 0x7f5b8c0b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b8c0cec40 0x7f5b8c1f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 --2- 172.21.15.12:0/1998733138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b8c0d21f0 0x7f5b8c155f70 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 -- 172.21.15.12:0/1998733138 >> 172.21.15.12:0/1998733138 conn(0x7f5b8c08aff0 msgr2=0x7f5b8c06d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.328+0000 7f5b92e00700 1 -- 172.21.15.12:0/1998733138 shutdown_connections 2024-03-29T10:12:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.329+0000 7f5b92e00700 1 -- 172.21.15.12:0/1998733138 wait complete. 2024-03-29T10:12:57.356 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T10:12:57.546 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:57.547+0000 7fc5f54c7240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:12:57.565 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:57.565+0000 7f50e049e240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:12:57.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.568+0000 7f2c35a00700 1 Processor -- start 2024-03-29T10:12:57.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.568+0000 7f2c35a00700 1 -- start start 2024-03-29T10:12:57.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.569+0000 7f2c35a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2c300b2b30 0x7f2c300b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:57.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.569+0000 7f2c35a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c300b3960 0x7f2c300cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:57.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.569+0000 7f2c35a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c300b3d90 con 0x7f2c30157f90 2024-03-29T10:12:57.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.569+0000 7f2c35a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c30158370 con 0x7f2c300b2b30 2024-03-29T10:12:57.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.569+0000 7f2c35a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c300b2710 con 0x7f2c300b3960 2024-03-29T10:12:57.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.570+0000 7f2c2e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2c300b2b30 0x7f2c300b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:57.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.570+0000 7f2c2e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2c300b2b30 0x7f2c300b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52484/0 (socket says 172.21.15.12:52484) 2024-03-29T10:12:57.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.570+0000 7f2c2ea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2c30157f90 0x7f2c301613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47530/0 (socket says 172.21.15.12:47530) 2024-03-29T10:12:57.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.570+0000 7f2c2e000700 1 -- 172.21.15.12:0/1661954311 learned_addr learned my addr 172.21.15.12:0/1661954311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:57.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.570+0000 7f2c2f400700 1 --2- 172.21.15.12:0/1661954311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c300b3960 0x7f2c300cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.570+0000 7f2c2e000700 1 -- 172.21.15.12:0/1661954311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c300b3960 msgr2=0x7f2c300cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.570+0000 7f2c2e000700 1 --2- 172.21.15.12:0/1661954311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c300b3960 0x7f2c300cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.570+0000 7f2c2e000700 1 -- 172.21.15.12:0/1661954311 >> v1:172.21.15.12:6789/0 conn(0x7f2c30157f90 legacy=0x7f2c301613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.570+0000 7f2c2e000700 1 -- 172.21.15.12:0/1661954311 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c300b2070 con 0x7f2c300b2b30 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.570+0000 7f2c2d600700 1 -- 172.21.15.12:0/1661954311 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447807662 0 0) 0x7f2c300b3d90 con 0x7f2c30157f90 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.571+0000 7f2c2e000700 1 --2- 172.21.15.12:0/1661954311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 0x7f2c300b3420 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0098f0 tx=0x7f2c1c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993f6c3240b586d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.571+0000 7f2c2d600700 1 -- 172.21.15.12:0/1661954311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c011830 con 0x7f2c300b2b30 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.571+0000 7f2c2d600700 1 -- 172.21.15.12:0/1661954311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c011990 con 0x7f2c300b2b30 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.571+0000 7f2c2d600700 1 -- 172.21.15.12:0/1661954311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c011d20 con 0x7f2c300b2b30 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.571+0000 7f2c35a00700 1 -- 172.21.15.12:0/1661954311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 msgr2=0x7f2c300b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.571+0000 7f2c35a00700 1 --2- 172.21.15.12:0/1661954311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 0x7f2c300b3420 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0098f0 tx=0x7f2c1c009c00 comp rx=0 tx=0).stop 2024-03-29T10:12:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.571+0000 7f2c35a00700 1 -- 172.21.15.12:0/1661954311 shutdown_connections 2024-03-29T10:12:57.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.571+0000 7f2c35a00700 1 --2- 172.21.15.12:0/1661954311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c300b3960 0x7f2c300cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.571+0000 7f2c35a00700 1 --2- 172.21.15.12:0/1661954311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 0x7f2c300b3420 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.571+0000 7f2c35a00700 1 -- 172.21.15.12:0/1661954311 >> 172.21.15.12:0/1661954311 conn(0x7f2c300b0d70 msgr2=0x7f2c300b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:57.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.572+0000 7f2c35a00700 1 -- 172.21.15.12:0/1661954311 shutdown_connections 2024-03-29T10:12:57.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.572+0000 7f2c35a00700 1 -- 172.21.15.12:0/1661954311 wait complete. 2024-03-29T10:12:57.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.574+0000 7f2c35a00700 1 Processor -- start 2024-03-29T10:12:57.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.574+0000 7f2c35a00700 1 -- start start 2024-03-29T10:12:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.575+0000 7f2c35a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 0x7f2c30156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c35a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c300b3960 0x7f2c30155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c35a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c30157f90 0x7f2c301526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c35a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c301f3490 con 0x7f2c300b2b30 2024-03-29T10:12:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c35a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c301f3770 con 0x7f2c300b3960 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c35a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c301f3a50 con 0x7f2c30157f90 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 0x7f2c30156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 0x7f2c30156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52500/0 (socket says 172.21.15.12:52500) 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2ea00700 1 -- 172.21.15.12:0/2845246000 learned_addr learned my addr 172.21.15.12:0/2845246000 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2e000700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c300b3960 0x7f2c30155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2f400700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c30157f90 0x7f2c301526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2ea00700 1 -- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c30157f90 msgr2=0x7f2c301526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2ea00700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c30157f90 0x7f2c301526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2ea00700 1 -- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c300b3960 msgr2=0x7f2c30155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2ea00700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c300b3960 0x7f2c30155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2ea00700 1 -- 172.21.15.12:0/2845246000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c009550 con 0x7f2c300b2b30 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2e000700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c300b3960 0x7f2c30155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.576+0000 7f2c2f400700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c30157f90 0x7f2c301526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.577+0000 7f2c2ea00700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 0x7f2c30156c40 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f2c200011e0 tx=0x7f2c2000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.577+0000 7f2c26a00700 1 -- 172.21.15.12:0/2845246000 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20004c80 con 0x7f2c300b2b30 2024-03-29T10:12:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.577+0000 7f2c26a00700 1 -- 172.21.15.12:0/2845246000 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c20004e00 con 0x7f2c300b2b30 2024-03-29T10:12:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.577+0000 7f2c26a00700 1 -- 172.21.15.12:0/2845246000 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20007310 con 0x7f2c300b2b30 2024-03-29T10:12:57.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.577+0000 7f2c35a00700 1 -- 172.21.15.12:0/2845246000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c301f3d30 con 0x7f2c300b2b30 2024-03-29T10:12:57.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.577+0000 7f2c35a00700 1 -- 172.21.15.12:0/2845246000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c30157180 con 0x7f2c300b2b30 2024-03-29T10:12:57.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.578+0000 7f2c35a00700 1 -- 172.21.15.12:0/2845246000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c301f4250 con 0x7f2c300b2b30 2024-03-29T10:12:57.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.580+0000 7f2c26a00700 1 -- 172.21.15.12:0/2845246000 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 58) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f2c20007470 con 0x7f2c300b2b30 2024-03-29T10:12:57.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.581+0000 7f2c26a00700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c1006ae00 0x7f2c1006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:57.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.581+0000 7f2c26a00700 1 -- 172.21.15.12:0/2845246000 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c20089140 con 0x7f2c300b2b30 2024-03-29T10:12:57.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.581+0000 7f2c2e000700 1 -- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c1006ae00 msgr2=0x7f2c1006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:57.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.581+0000 7f2c2e000700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c1006ae00 0x7f2c1006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.586+0000 7f2c26a00700 1 -- 172.21.15.12:0/2845246000 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2c301f4250 con 0x7f2c300b2b30 2024-03-29T10:12:57.732 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:57.733+0000 7fc5f54c7240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:12:57.778 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:57.778+0000 7f50e049e240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:12:57.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.782+0000 7f2c2e000700 1 -- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c1006ae00 msgr2=0x7f2c1006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:57.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.782+0000 7f2c2e000700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c1006ae00 0x7f2c1006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.838+0000 7f2c35a00700 1 -- 172.21.15.12:0/2845246000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2c301f4250 con 0x7f2c300b2b30 2024-03-29T10:12:57.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.843+0000 7f2c26a00700 1 -- 172.21.15.12:0/2845246000 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2c301f4250 con 0x7f2c300b2b30 2024-03-29T10:12:57.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.845+0000 7f2c35a00700 1 -- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c1006ae00 msgr2=0x7f2c1006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:57.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.845+0000 7f2c35a00700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c1006ae00 0x7f2c1006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.845+0000 7f2c35a00700 1 -- 172.21.15.12:0/2845246000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 msgr2=0x7f2c30156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.845+0000 7f2c35a00700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 0x7f2c30156c40 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f2c200011e0 tx=0x7f2c2000ca30 comp rx=0 tx=0).stop 2024-03-29T10:12:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.845+0000 7f2c35a00700 1 -- 172.21.15.12:0/2845246000 shutdown_connections 2024-03-29T10:12:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.845+0000 7f2c35a00700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2c1006ae00 0x7f2c1006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.845+0000 7f2c35a00700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c30157f90 0x7f2c301526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.845+0000 7f2c35a00700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c300b2b30 0x7f2c30156c40 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.845+0000 7f2c35a00700 1 --2- 172.21.15.12:0/2845246000 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c300b3960 0x7f2c30155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.845+0000 7f2c35a00700 1 -- 172.21.15.12:0/2845246000 >> 172.21.15.12:0/2845246000 conn(0x7f2c300b0d70 msgr2=0x7f2c300b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.846+0000 7f2c35a00700 1 -- 172.21.15.12:0/2845246000 shutdown_connections 2024-03-29T10:12:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:57.846+0000 7f2c35a00700 1 -- 172.21.15.12:0/2845246000 wait complete. 2024-03-29T10:12:57.864 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T10:12:58.049 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:58.051+0000 7fc5f54c7240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:12:58.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:12:58.057 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:12:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.070+0000 7f808f200700 1 Processor -- start 2024-03-29T10:12:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.071+0000 7f808f200700 1 -- start start 2024-03-29T10:12:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.072+0000 7f808f200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f80880d16a0 0x7f80880d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.072+0000 7f808f200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8088161420 con 0x7f808815d9d0 2024-03-29T10:12:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.072+0000 7f808f200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8088161700 con 0x7f80880d2370 2024-03-29T10:12:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.072+0000 7f808f200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80880d4340 con 0x7f80880d16a0 2024-03-29T10:12:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.072+0000 7f8087400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f80880d16a0 0x7f80880d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.072+0000 7f8087400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f80880d16a0 0x7f80880d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52504/0 (socket says 172.21.15.12:52504) 2024-03-29T10:12:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.072+0000 7f8087400700 1 -- 172.21.15.12:0/259090320 learned_addr learned my addr 172.21.15.12:0/259090320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.072+0000 7f8087e00700 1 --1- 172.21.15.12:0/259090320 >> v1:172.21.15.12:6789/0 conn(0x7f808815d9d0 0x7f80880d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47542/0 (socket says 172.21.15.12:47542) 2024-03-29T10:12:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.073+0000 7f8087400700 1 -- 172.21.15.12:0/259090320 >> v1:172.21.15.159:6789/0 conn(0x7f80880d2370 legacy=0x7f80880ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.073+0000 7f8087400700 1 -- 172.21.15.12:0/259090320 >> v1:172.21.15.12:6789/0 conn(0x7f808815d9d0 legacy=0x7f80880d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.073+0000 7f8087400700 1 -- 172.21.15.12:0/259090320 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80881619e0 con 0x7f80880d16a0 2024-03-29T10:12:58.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.073+0000 7f8087400700 1 --2- 172.21.15.12:0/259090320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80880d16a0 0x7f80880d1e30 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f8074006fd0 tx=0x7f8074009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673ffbd7165f834d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:58.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.073+0000 7f8086a00700 1 -- 172.21.15.12:0/259090320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074005e30 con 0x7f80880d16a0 2024-03-29T10:12:58.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.074+0000 7f8086a00700 1 -- 172.21.15.12:0/259090320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8074004030 con 0x7f80880d16a0 2024-03-29T10:12:58.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.074+0000 7f8086a00700 1 -- 172.21.15.12:0/259090320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80740043e0 con 0x7f80880d16a0 2024-03-29T10:12:58.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.074+0000 7f808f200700 1 -- 172.21.15.12:0/259090320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80880d16a0 msgr2=0x7f80880d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:58.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.074+0000 7f808f200700 1 --2- 172.21.15.12:0/259090320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80880d16a0 0x7f80880d1e30 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f8074006fd0 tx=0x7f8074009bd0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.074+0000 7f808f200700 1 -- 172.21.15.12:0/259090320 shutdown_connections 2024-03-29T10:12:58.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.074+0000 7f808f200700 1 --2- 172.21.15.12:0/259090320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80880d16a0 0x7f80880d1e30 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.074+0000 7f808f200700 1 -- 172.21.15.12:0/259090320 >> 172.21.15.12:0/259090320 conn(0x7f808808aff0 msgr2=0x7f80880d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:58.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.074+0000 7f808f200700 1 -- 172.21.15.12:0/259090320 shutdown_connections 2024-03-29T10:12:58.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.075+0000 7f808f200700 1 -- 172.21.15.12:0/259090320 wait complete. 2024-03-29T10:12:58.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.076+0000 7f808f200700 1 Processor -- start 2024-03-29T10:12:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.076+0000 7f808f200700 1 -- start start 2024-03-29T10:12:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.077+0000 7f808f200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80880d16a0 0x7f80880b8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.077+0000 7f808f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80880d2370 0x7f8088155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.077+0000 7f808f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f808815d9d0 0x7f80880b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.077+0000 7f808f200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80881561f0 con 0x7f808815d9d0 2024-03-29T10:12:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.077+0000 7f808f200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80880b9300 con 0x7f80880d16a0 2024-03-29T10:12:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.077+0000 7f808f200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80880cb730 con 0x7f80880d2370 2024-03-29T10:12:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f808d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f808815d9d0 0x7f80880b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f808d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f808815d9d0 0x7f80880b6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52516/0 (socket says 172.21.15.12:52516) 2024-03-29T10:12:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f8087e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80880d16a0 0x7f80880b8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f808d200700 1 -- 172.21.15.12:0/2598462986 learned_addr learned my addr 172.21.15.12:0/2598462986 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f8087400700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80880d2370 0x7f8088155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f808d200700 1 -- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80880d2370 msgr2=0x7f8088155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f808d200700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80880d2370 0x7f8088155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f808d200700 1 -- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80880d16a0 msgr2=0x7f80880b8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f808d200700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80880d16a0 0x7f80880b8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f808d200700 1 -- 172.21.15.12:0/2598462986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8074009500 con 0x7f808815d9d0 2024-03-29T10:12:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.078+0000 7f8087e00700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80880d16a0 0x7f80880b8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:12:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.079+0000 7f808d200700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f808815d9d0 0x7f80880b6190 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f808000cc50 tx=0x7f808000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.079+0000 7f807fe00700 1 -- 172.21.15.12:0/2598462986 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080003c00 con 0x7f808815d9d0 2024-03-29T10:12:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.079+0000 7f807fe00700 1 -- 172.21.15.12:0/2598462986 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8080003d60 con 0x7f808815d9d0 2024-03-29T10:12:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.079+0000 7f807fe00700 1 -- 172.21.15.12:0/2598462986 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808000aab0 con 0x7f808815d9d0 2024-03-29T10:12:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.079+0000 7f808f200700 1 -- 172.21.15.12:0/2598462986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8088155c60 con 0x7f808815d9d0 2024-03-29T10:12:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.079+0000 7f808f200700 1 -- 172.21.15.12:0/2598462986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80880cae00 con 0x7f808815d9d0 2024-03-29T10:12:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.081+0000 7f808f200700 1 -- 172.21.15.12:0/2598462986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80880d1ca0 con 0x7f808815d9d0 2024-03-29T10:12:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.082+0000 7f807fe00700 1 -- 172.21.15.12:0/2598462986 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 58) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f808000ac10 con 0x7f808815d9d0 2024-03-29T10:12:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.083+0000 7f807fe00700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f806806ae00 0x7f806806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.083+0000 7f807fe00700 1 -- 172.21.15.12:0/2598462986 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8080089e40 con 0x7f808815d9d0 2024-03-29T10:12:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.083+0000 7f8087e00700 1 -- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f806806ae00 msgr2=0x7f806806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.083+0000 7f8087e00700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f806806ae00 0x7f806806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.089+0000 7f807fe00700 1 -- 172.21.15.12:0/2598462986 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f80800545a0 con 0x7f808815d9d0 2024-03-29T10:12:58.126 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:58.126+0000 7f50e049e240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:12:58.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.284+0000 7f8087e00700 1 -- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f806806ae00 msgr2=0x7f806806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:58.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.284+0000 7f8087e00700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f806806ae00 0x7f806806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:58.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.310+0000 7f808f200700 1 -- 172.21.15.12:0/2598462986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f8088067720 con 0x7f808815d9d0 2024-03-29T10:12:58.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.316+0000 7f807fe00700 1 -- 172.21.15.12:0/2598462986 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f8080057bc0 con 0x7f808815d9d0 2024-03-29T10:12:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 -- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f806806ae00 msgr2=0x7f806806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:12:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f806806ae00 0x7f806806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 -- 172.21.15.12:0/2598462986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f808815d9d0 msgr2=0x7f80880b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f808815d9d0 0x7f80880b6190 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f808000cc50 tx=0x7f808000cf60 comp rx=0 tx=0).stop 2024-03-29T10:12:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 -- 172.21.15.12:0/2598462986 shutdown_connections 2024-03-29T10:12:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f806806ae00 0x7f806806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f808815d9d0 0x7f80880b6190 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80880d16a0 0x7f80880b8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 --2- 172.21.15.12:0/2598462986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80880d2370 0x7f8088155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 -- 172.21.15.12:0/2598462986 >> 172.21.15.12:0/2598462986 conn(0x7f808808aff0 msgr2=0x7f80880b0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.318+0000 7f808f200700 1 -- 172.21.15.12:0/2598462986 shutdown_connections 2024-03-29T10:12:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.319+0000 7f808f200700 1 -- 172.21.15.12:0/2598462986 wait complete. 2024-03-29T10:12:58.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:12:58.337 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-29T10:12:58.386 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-03-29T10:12:58.590 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:58.591+0000 7fc5f54c7240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:12:58.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.591+0000 7f8588a00700 1 Processor -- start 2024-03-29T10:12:58.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.591+0000 7f8588a00700 1 -- start start 2024-03-29T10:12:58.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.593+0000 7f8588a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f858415dad0 0x7f85840ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.593+0000 7f8588a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85840d1630 0x7f85840d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.593+0000 7f8588a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858415f820 con 0x7f85840d1fd0 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.593+0000 7f8588a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85840cfcc0 con 0x7f858415dad0 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.593+0000 7f8588a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85840cd9f0 con 0x7f85840d1630 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.593+0000 7f8581e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f858415dad0 0x7f85840ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.593+0000 7f8581e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f858415dad0 0x7f85840ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52532/0 (socket says 172.21.15.12:52532) 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.593+0000 7f8581e00700 1 -- 172.21.15.12:0/4128032758 learned_addr learned my addr 172.21.15.12:0/4128032758 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.593+0000 7f8581400700 1 --2- 172.21.15.12:0/4128032758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85840d1630 0x7f85840d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.594+0000 7f8581e00700 1 -- 172.21.15.12:0/4128032758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85840d1630 msgr2=0x7f85840d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.594+0000 7f8581e00700 1 --2- 172.21.15.12:0/4128032758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85840d1630 0x7f85840d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.594+0000 7f8581e00700 1 -- 172.21.15.12:0/4128032758 >> v1:172.21.15.159:6789/0 conn(0x7f85840d1fd0 legacy=0x7f858415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:58.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.594+0000 7f8581e00700 1 -- 172.21.15.12:0/4128032758 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85840d2460 con 0x7f858415dad0 2024-03-29T10:12:58.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.594+0000 7f8581e00700 1 --2- 172.21.15.12:0/4128032758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 0x7f85840ced80 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f85740098f0 tx=0x7f8574009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e4f0fda582be48 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.594+0000 7f8580a00700 1 -- 172.21.15.12:0/4128032758 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8574011830 con 0x7f858415dad0 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.594+0000 7f8580a00700 1 -- 172.21.15.12:0/4128032758 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8574011990 con 0x7f858415dad0 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.594+0000 7f8580a00700 1 -- 172.21.15.12:0/4128032758 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8574011d20 con 0x7f858415dad0 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.595+0000 7f8588a00700 1 -- 172.21.15.12:0/4128032758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 msgr2=0x7f85840ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.595+0000 7f8588a00700 1 --2- 172.21.15.12:0/4128032758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 0x7f85840ced80 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f85740098f0 tx=0x7f8574009c00 comp rx=0 tx=0).stop 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.595+0000 7f8588a00700 1 -- 172.21.15.12:0/4128032758 shutdown_connections 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.595+0000 7f8588a00700 1 --2- 172.21.15.12:0/4128032758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 0x7f85840ced80 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.595+0000 7f8588a00700 1 --2- 172.21.15.12:0/4128032758 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85840d1630 0x7f85840d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.595+0000 7f8588a00700 1 -- 172.21.15.12:0/4128032758 >> 172.21.15.12:0/4128032758 conn(0x7f858408aff0 msgr2=0x7f85840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.595+0000 7f8588a00700 1 -- 172.21.15.12:0/4128032758 shutdown_connections 2024-03-29T10:12:58.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.595+0000 7f8588a00700 1 -- 172.21.15.12:0/4128032758 wait complete. 2024-03-29T10:12:58.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.597+0000 7f8588a00700 1 Processor -- start 2024-03-29T10:12:58.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.597+0000 7f8588a00700 1 -- start start 2024-03-29T10:12:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8588a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85840d1630 0x7f8584150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8588a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85840d1fd0 0x7f85840cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8588a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 0x7f85840cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8588a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8584151060 con 0x7f858415dad0 2024-03-29T10:12:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8588a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85840b6410 con 0x7f85840d1630 2024-03-29T10:12:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8588a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85840c8340 con 0x7f85840d1fd0 2024-03-29T10:12:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8582800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 0x7f85840cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8581e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85840d1630 0x7f8584150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8582800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 0x7f85840cc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52546/0 (socket says 172.21.15.12:52546) 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8582800700 1 -- 172.21.15.12:0/634643152 learned_addr learned my addr 172.21.15.12:0/634643152 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.598+0000 7f8581400700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85840d1fd0 0x7f85840cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f8582800700 1 -- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85840d1fd0 msgr2=0x7f85840cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f8582800700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85840d1fd0 0x7f85840cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f8582800700 1 -- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85840d1630 msgr2=0x7f8584150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f8582800700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85840d1630 0x7f8584150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f8582800700 1 -- 172.21.15.12:0/634643152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8574009550 con 0x7f858415dad0 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f8582800700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 0x7f85840cc120 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f857c00cc50 tx=0x7f857c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c003c00 con 0x7f858415dad0 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857c003d60 con 0x7f858415dad0 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f8588a00700 1 -- 172.21.15.12:0/634643152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85840b66b0 con 0x7f858415dad0 2024-03-29T10:12:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c00aab0 con 0x7f858415dad0 2024-03-29T10:12:58.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.599+0000 7f8588a00700 1 -- 172.21.15.12:0/634643152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85840b6c70 con 0x7f858415dad0 2024-03-29T10:12:58.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.601+0000 7f8588a00700 1 -- 172.21.15.12:0/634643152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85840b6e90 con 0x7f858415dad0 2024-03-29T10:12:58.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.601+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 58) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f857c00ac10 con 0x7f858415dad0 2024-03-29T10:12:58.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.603+0000 7f857a000700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:12:58.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.603+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f857c089510 con 0x7f858415dad0 2024-03-29T10:12:58.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.603+0000 7f8581e00700 1 -- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 msgr2=0x7f856406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:58.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.603+0000 7f8581e00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:12:58.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.607+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f857c054560 con 0x7f858415dad0 2024-03-29T10:12:58.701 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:58.701+0000 7f50e049e240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:12:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.804+0000 7f8581e00700 1 -- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 msgr2=0x7f856406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.804+0000 7f8581e00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:12:58.849 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:58.851+0000 7fc5f54c7240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:12:58.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:58.851+0000 7f8588a00700 1 -- 172.21.15.12:0/634643152 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f85840cf9e0 con 0x7f856406ae00 2024-03-29T10:12:58.968 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:58.969+0000 7f50e049e240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:12:59.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:59.205+0000 7f8581e00700 1 -- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 msgr2=0x7f856406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:12:59.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:12:59.205+0000 7f8581e00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T10:12:59.308 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:12:59.309+0000 7fc5f54c7240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:12:59.450 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:12:59.451+0000 7f50e049e240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:13:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:00.006+0000 7f8581e00700 1 -- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 msgr2=0x7f856406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:13:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:00.006+0000 7f8581e00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T10:13:00.018 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:00.019+0000 7fc5f54c7240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:13:00.110 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:00.111+0000 7fc5f54c7240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:13:00.200 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:00.200+0000 7f50e049e240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:13:00.268 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:00.269+0000 7fc5f54c7240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:13:00.297 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:00.298+0000 7f50e049e240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:13:00.468 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:00.468+0000 7f50e049e240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:13:00.482 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:00.483+0000 7fc5f54c7240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:13:00.573 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:00.573+0000 7fc5f54c7240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:13:00.670 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:00.671+0000 7fc5f54c7240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:13:00.687 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:00.687+0000 7f50e049e240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:13:00.767 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:00.768+0000 7fc5f54c7240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:13:00.784 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:00.784+0000 7f50e049e240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:13:00.866 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:00.867+0000 7fc5f54c7240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:13:00.888 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:00.889+0000 7f50e049e240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:13:00.992 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:00.993+0000 7f50e049e240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:13:01.077 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:01.078+0000 7fc5f54c7240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:13:01.097 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:01.097+0000 7f50e049e240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:13:01.321 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:01.322+0000 7f50e049e240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:13:01.333 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:01.334+0000 7fc5f54c7240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:13:01.522 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:01.523+0000 7fc5f54c7240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:13:01.586 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:01.586+0000 7f50e049e240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:13:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:01.608+0000 7f8581e00700 1 -- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 msgr2=0x7f856406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:13:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:01.608+0000 7f8581e00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T10:13:01.614 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:01.616+0000 7fc5f54c7240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:13:01.788 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:01.788+0000 7f50e049e240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:13:01.886 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:01.887+0000 7f50e049e240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:13:02.344 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:02.344+0000 7fc5f54c7240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:13:02.556 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:02.556+0000 7fc5f54c7240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:13:02.658 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:02.659+0000 7f50e049e240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:13:02.762 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:02.762+0000 7fc5f54c7240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:13:02.887 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:02.887+0000 7f50e049e240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:13:03.108 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:03.108+0000 7f50e049e240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:13:03.240 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:03.240+0000 7fc5f54c7240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:13:03.444 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:03.444+0000 7fc5f54c7240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:13:03.613 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:03.613+0000 7f50e049e240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:13:03.834 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:03.834+0000 7f50e049e240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:13:03.885 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:03.885+0000 7fc5f54c7240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:13:03.996 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:03.996+0000 7fc5f54c7240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:13:04.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:04.009+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 59) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f857c010dc0 con 0x7f858415dad0 2024-03-29T10:13:04.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:04.009+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 msgr2=0x7f856406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:13:04.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:04.009+0000 7f857a000700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:04.206 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:04.205+0000 7fc55dc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:13:04.206 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:04.205+0000 7fc55dc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:13:04.206 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:04.205+0000 7fc55dc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:13:04.206 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:04.205+0000 7fc55dc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:13:04.206 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:04.205+0000 7fc55dc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:13:04.206 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:04.206+0000 7fc560e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:13:04.207 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:04.206+0000 7fc560e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:13:04.207 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:04.206+0000 7fc560e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:13:04.207 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:04.206+0000 7fc560e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:13:04.207 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:13:04.206+0000 7fc560e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:13:04.310 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:04.310+0000 7f50e049e240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:13:04.427 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:13:04.428+0000 7f50e049e240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:13:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.012+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 60) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f857c087630 con 0x7f858415dad0 2024-03-29T10:13:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.013+0000 7f857a000700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:13:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.013+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8564035990 con 0x7f856406ae00 2024-03-29T10:13:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.013+0000 7f8581e00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:13:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.019+0000 7f8581e00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8574014040 tx=0x7f8574008030 comp rx=0 tx=0).ready entity=mgr.6929 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:13:05.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.099+0000 7f857a000700 1 -- 172.21.15.12:0/634643152 <== mgr.6929 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8564035990 con 0x7f856406ae00 2024-03-29T10:13:05.099 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1 2024-03-29T10:13:05.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 -- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 msgr2=0x7f856406d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:13:05.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8574014040 tx=0x7f8574008030 comp rx=0 tx=0).stop 2024-03-29T10:13:05.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 -- 172.21.15.12:0/634643152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 msgr2=0x7f85840cc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:13:05.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 0x7f85840cc120 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f857c00cc50 tx=0x7f857c00cf60 comp rx=0 tx=0).stop 2024-03-29T10:13:05.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 -- 172.21.15.12:0/634643152 shutdown_connections 2024-03-29T10:13:05.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f856406ae00 0x7f856406d240 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f858415dad0 0x7f85840cc120 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85840d1630 0x7f8584150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 --2- 172.21.15.12:0/634643152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85840d1fd0 0x7f85840cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 -- 172.21.15.12:0/634643152 >> 172.21.15.12:0/634643152 conn(0x7f858408aff0 msgr2=0x7f858406b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:13:05.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 -- 172.21.15.12:0/634643152 shutdown_connections 2024-03-29T10:13:05.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.102+0000 7f8588a00700 1 -- 172.21.15.12:0/634643152 wait complete. 2024-03-29T10:13:05.121 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-29T10:13:05.121 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-03-29-10_14_00 in ~55s... 2024-03-29T10:13:05.121 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-03-29T10:13:05.122 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-03-29T10:13:05.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.322+0000 7f8f01a00700 1 Processor -- start 2024-03-29T10:13:05.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.322+0000 7f8f01a00700 1 -- start start 2024-03-29T10:13:05.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.323+0000 7f8f01a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8efc0ced80 0x7f8efc0d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:13:05.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8f01a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8efc15fcb0 0x7f8efc1600e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:13:05.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8f01a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc15f820 con 0x7f8efc15dad0 2024-03-29T10:13:05.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8f01a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8efc0cfcc0 con 0x7f8efc0ced80 2024-03-29T10:13:05.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8f01a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8efc0cd9f0 con 0x7f8efc15fcb0 2024-03-29T10:13:05.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8efa000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8efc0ced80 0x7f8efc0d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:13:05.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8efaa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8efc15dad0 0x7f8efc0d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42666/0 (socket says 172.21.15.12:42666) 2024-03-29T10:13:05.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8efaa00700 1 -- 172.21.15.12:0/3817010176 learned_addr learned my addr 172.21.15.12:0/3817010176 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:13:05.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8efb400700 1 --2- 172.21.15.12:0/3817010176 >> v2:172.21.15.159:3300/0 conn(0x7f8efc15fcb0 0x7f8efc1600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8efa000700 1 -- 172.21.15.12:0/3817010176 >> v2:172.21.15.159:3300/0 conn(0x7f8efc15fcb0 msgr2=0x7f8efc1600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8efa000700 1 --2- 172.21.15.12:0/3817010176 >> v2:172.21.15.159:3300/0 conn(0x7f8efc15fcb0 0x7f8efc1600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8efa000700 1 -- 172.21.15.12:0/3817010176 >> v1:172.21.15.12:6789/0 conn(0x7f8efc15dad0 legacy=0x7f8efc0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8efa000700 1 -- 172.21.15.12:0/3817010176 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc0cf420 con 0x7f8efc0ced80 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.324+0000 7f8ef9600700 1 -- 172.21.15.12:0/3817010176 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842568887 0 0) 0x7f8efc15f820 con 0x7f8efc15dad0 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.325+0000 7f8efa000700 1 --2- 172.21.15.12:0/3817010176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc0ced80 0x7f8efc0d1d40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8efc08ccb0 tx=0x7f8ee8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8197c19820f3a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.325+0000 7f8ef9600700 1 -- 172.21.15.12:0/3817010176 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8011830 con 0x7f8efc0ced80 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.325+0000 7f8ef9600700 1 -- 172.21.15.12:0/3817010176 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee8011990 con 0x7f8efc0ced80 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.325+0000 7f8ef9600700 1 -- 172.21.15.12:0/3817010176 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8011d20 con 0x7f8efc0ced80 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.325+0000 7f8f01a00700 1 -- 172.21.15.12:0/3817010176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc0ced80 msgr2=0x7f8efc0d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.325+0000 7f8f01a00700 1 --2- 172.21.15.12:0/3817010176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc0ced80 0x7f8efc0d1d40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8efc08ccb0 tx=0x7f8ee8009b20 comp rx=0 tx=0).stop 2024-03-29T10:13:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.326+0000 7f8f01a00700 1 -- 172.21.15.12:0/3817010176 shutdown_connections 2024-03-29T10:13:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.326+0000 7f8f01a00700 1 --2- 172.21.15.12:0/3817010176 >> v2:172.21.15.159:3300/0 conn(0x7f8efc15fcb0 0x7f8efc1600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.326+0000 7f8f01a00700 1 --2- 172.21.15.12:0/3817010176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc0ced80 0x7f8efc0d1d40 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.326+0000 7f8f01a00700 1 -- 172.21.15.12:0/3817010176 >> 172.21.15.12:0/3817010176 conn(0x7f8efc08aff0 msgr2=0x7f8efc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:13:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.326+0000 7f8f01a00700 1 -- 172.21.15.12:0/3817010176 shutdown_connections 2024-03-29T10:13:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.326+0000 7f8f01a00700 1 -- 172.21.15.12:0/3817010176 wait complete. 2024-03-29T10:13:05.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.327+0000 7f8f01a00700 1 Processor -- start 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.328+0000 7f8f01a00700 1 -- start start 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.328+0000 7f8f01a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8efc0ced80 0x7f8efc149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.328+0000 7f8f01a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc15dad0 0x7f8efc0c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.328+0000 7f8f01a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8efc15fcb0 0x7f8efc0c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.328+0000 7f8f01a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc149ed0 con 0x7f8efc15dad0 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.328+0000 7f8f01a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0c8b00 con 0x7f8efc0ced80 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.328+0000 7f8f01a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0c7ef0 con 0x7f8efc15fcb0 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc15dad0 0x7f8efc0c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc15dad0 0x7f8efc0c9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58914/0 (socket says 172.21.15.12:58914) 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efa000700 1 -- 172.21.15.12:0/1650374109 learned_addr learned my addr 172.21.15.12:0/1650374109 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efaa00700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8efc0ced80 0x7f8efc149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efb400700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8efc15fcb0 0x7f8efc0c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:13:05.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efa000700 1 -- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8efc15fcb0 msgr2=0x7f8efc0c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:13:05.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efa000700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8efc15fcb0 0x7f8efc0c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efa000700 1 -- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8efc0ced80 msgr2=0x7f8efc149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:13:05.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efa000700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8efc0ced80 0x7f8efc149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efa000700 1 -- 172.21.15.12:0/1650374109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee80094a0 con 0x7f8efc15dad0 2024-03-29T10:13:05.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.329+0000 7f8efa000700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc15dad0 0x7f8efc0c9220 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8000c00 tx=0x7f8ee8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:13:05.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.330+0000 7f8ef2a00700 1 -- 172.21.15.12:0/1650374109 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8004550 con 0x7f8efc15dad0 2024-03-29T10:13:05.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.330+0000 7f8ef2a00700 1 -- 172.21.15.12:0/1650374109 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee801f070 con 0x7f8efc15dad0 2024-03-29T10:13:05.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.330+0000 7f8ef2a00700 1 -- 172.21.15.12:0/1650374109 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee80038e0 con 0x7f8efc15dad0 2024-03-29T10:13:05.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.330+0000 7f8f01a00700 1 -- 172.21.15.12:0/1650374109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc0c81d0 con 0x7f8efc15dad0 2024-03-29T10:13:05.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.330+0000 7f8f01a00700 1 -- 172.21.15.12:0/1650374109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc0c8790 con 0x7f8efc15dad0 2024-03-29T10:13:05.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.331+0000 7f8f01a00700 1 -- 172.21.15.12:0/1650374109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc067720 con 0x7f8efc15dad0 2024-03-29T10:13:05.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.332+0000 7f8ef2a00700 1 -- 172.21.15.12:0/1650374109 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 60) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f8ee8003a40 con 0x7f8efc15dad0 2024-03-29T10:13:05.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.333+0000 7f8ef2a00700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8edc06ab30 0x7f8edc06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:13:05.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.333+0000 7f8ef2a00700 1 -- 172.21.15.12:0/1650374109 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f8ee8021070 con 0x7f8efc15dad0 2024-03-29T10:13:05.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.333+0000 7f8efaa00700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8edc06ab30 0x7f8edc06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:13:05.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.334+0000 7f8efaa00700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8edc06ab30 0x7f8edc06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007fd0 tx=0x7f8eec007de0 comp rx=0 tx=0).ready entity=mgr.6929 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:13:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.337+0000 7f8ef2a00700 1 -- 172.21.15.12:0/1650374109 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ee8055d10 con 0x7f8efc15dad0 2024-03-29T10:13:05.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.578+0000 7f8f01a00700 1 -- 172.21.15.12:0/1650374109 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8efc0cf9e0 con 0x7f8edc06ab30 2024-03-29T10:13:05.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.591+0000 7f8ef2a00700 1 -- 172.21.15.12:0/1650374109 <== mgr.6929 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f8eec004b10 con 0x7f8edc06ab30 2024-03-29T10:13:05.591 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:13:05.591 INFO:teuthology.orchestra.run.smithi012.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-03-29T10:13:05.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.593+0000 7f8f01a00700 1 -- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8edc06ab30 msgr2=0x7f8edc06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:13:05.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.593+0000 7f8f01a00700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8edc06ab30 0x7f8edc06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007fd0 tx=0x7f8eec007de0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.593+0000 7f8f01a00700 1 -- 172.21.15.12:0/1650374109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc15dad0 msgr2=0x7f8efc0c9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:13:05.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.593+0000 7f8f01a00700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc15dad0 0x7f8efc0c9220 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8000c00 tx=0x7f8ee8004750 comp rx=0 tx=0).stop 2024-03-29T10:13:05.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.593+0000 7f8f01a00700 1 -- 172.21.15.12:0/1650374109 shutdown_connections 2024-03-29T10:13:05.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.593+0000 7f8f01a00700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8edc06ab30 0x7f8edc06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.593+0000 7f8f01a00700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8efc15fcb0 0x7f8efc0c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.593+0000 7f8f01a00700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8efc0ced80 0x7f8efc149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.593+0000 7f8f01a00700 1 --2- 172.21.15.12:0/1650374109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8efc15dad0 0x7f8efc0c9220 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:13:05.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.593+0000 7f8f01a00700 1 -- 172.21.15.12:0/1650374109 >> 172.21.15.12:0/1650374109 conn(0x7f8efc08aff0 msgr2=0x7f8efc159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:13:05.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.594+0000 7f8f01a00700 1 -- 172.21.15.12:0/1650374109 shutdown_connections 2024-03-29T10:13:05.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:13:05.594+0000 7f8f01a00700 1 -- 172.21.15.12:0/1650374109 wait complete. 2024-03-29T10:13:05.611 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-03-29T10:13:05.611 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2024-03-29T10:13:05.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:05.612 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:05.658 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:05.658 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:05.658 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:06.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:06.659 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:06.708 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:06.708 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:06.708 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:07.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:07.709 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:07.761 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:07.761 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:07.761 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:08.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:08.762 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:08.814 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:08.814 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:08.814 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:09.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:09.815 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:09.867 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:09.867 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:09.867 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:10.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:10.868 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:10.919 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:10.919 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:10.919 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:11.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:11.920 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:11.971 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:11.971 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:11.971 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:12.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:12.972 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:13.024 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:13.024 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:13.024 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:14.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:14.025 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:14.077 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:14.077 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:14.077 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:15.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:15.079 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:15.131 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:15.131 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:15.131 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:16.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:16.131 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:16.183 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:16.184 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:16.184 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:17.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:17.184 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:17.233 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:17.233 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:17.233 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:18.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:18.234 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:18.286 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:18.286 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:18.286 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:19.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:19.287 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:19.339 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:19.339 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:19.339 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:20.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:20.340 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:20.392 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:20.392 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:20.392 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:21.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:21.393 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:21.450 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:21.450 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:21.450 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:22.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:22.452 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:22.504 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:22.504 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:22.504 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:23.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:23.505 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:23.556 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:23.556 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:23.556 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:24.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:24.557 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:24.613 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:24.613 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:24.613 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:25.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:25.613 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:25.665 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:25.665 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:25.665 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:26.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:26.667 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:26.718 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:26.718 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:26.718 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:27.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:27.719 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:27.771 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:27.771 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:27.771 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:28.366 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:13:28.370 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:13:28.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:28.772 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:28.823 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:28.823 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:28.823 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:29.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:29.825 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:29.876 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:29.876 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:29.877 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:30.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:30.878 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:30.929 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:30.929 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:30.929 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:31.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:31.931 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:31.982 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:31.983 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:31.983 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:32.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:32.984 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:33.035 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:33.035 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:33.035 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:34.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:34.036 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:34.088 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:34.089 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:34.089 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:35.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:35.089 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:35.141 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:35.141 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:35.142 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:36.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:36.142 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:36.195 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:36.195 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:36.195 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:37.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:37.196 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:37.248 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:37.248 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:37.248 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:38.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:38.249 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:38.301 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:38.301 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:38.301 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:39.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:39.302 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:39.355 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:39.355 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:39.355 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:40.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:40.356 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:40.407 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:40.407 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:40.407 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:41.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:41.409 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:41.461 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:41.461 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:41.461 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:42.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:42.462 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:42.513 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:42.514 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:42.514 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:43.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:43.514 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:43.566 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:43.566 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:43.566 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:44.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:44.567 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:44.619 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:44.619 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:44.619 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:45.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:45.620 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:45.672 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:45.672 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:45.672 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:46.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:46.676 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:46.728 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:46.728 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:46.728 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:47.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:47.729 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:47.780 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:47.781 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:47.781 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:48.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:48.781 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:48.833 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:48.833 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:48.833 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:49.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:49.834 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:49.886 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:49.886 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:49.886 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:50.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:50.887 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:50.939 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:50.939 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:50.939 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:51.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:51.940 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:51.992 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:51.992 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:51.993 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:52.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:52.994 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:53.051 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:53.051 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:53.051 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:54.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:54.053 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:54.105 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:54.105 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:54.105 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:55.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:55.106 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:55.158 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:55.158 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:55.158 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:56.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:56.160 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:56.212 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:56.212 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:56.212 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:57.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:57.213 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:57.264 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:57.264 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:57.264 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:58.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:58.265 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:58.318 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:58.318 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:58.318 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:13:58.479 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:13:58.484 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:13:59.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:13:59.320 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:13:59.372 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T10:13:59.372 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T10:13:59.372 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:14:00.112 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:00.112+0000 7fc54ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:00.112 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:00.112+0000 7fc54ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:00.112 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:00.112+0000 7fc54ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:00.112 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:00.112+0000 7fc54ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:00.112 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:00.112+0000 7fc54ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:00.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:00.374 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:14:00.423 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_14_00_UTC 2024-03-29T10:14:00.424 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-10_14_00_UTC'} 2024-03-29T10:14:00.424 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-03-29-10_14_00_UTC'} 2024-03-29T10:14:00.424 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T10:14:00.424 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-03-29-10_14_00_UTC'] 2024-03-29T10:14:00.424 DEBUG:tasks.cephfs.test_snap_schedules:exec=1711707185.121682, now = 1711707240.4246688, timo = 55 2024-03-29T10:14:01.425 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-03-29T10:14:01.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.637+0000 7f4573800700 1 Processor -- start 2024-03-29T10:14:01.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.637+0000 7f4573800700 1 -- start start 2024-03-29T10:14:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.639+0000 7f4573800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f456c15dfe0 0x7f456c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.639+0000 7f4573800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 0x7f456c0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.639+0000 7f4573800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c0cb310 con 0x7f456c15a5c0 2024-03-29T10:14:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.639+0000 7f4573800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456c0ceb00 con 0x7f456c15dfe0 2024-03-29T10:14:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.639+0000 7f4573800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456c0711a0 con 0x7f456c0ce670 2024-03-29T10:14:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.639+0000 7f4571800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 0x7f456c0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.639+0000 7f456be00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f456c15dfe0 0x7f456c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.639+0000 7f4570e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f456c15a5c0 0x7f456c0709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:42306/0 (socket says 172.21.15.12:42306) 2024-03-29T10:14:01.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.639+0000 7f4571800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 0x7f456c0cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:48232/0 (socket says 172.21.15.12:48232) 2024-03-29T10:14:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.639+0000 7f4570e00700 1 -- 172.21.15.12:0/3953321151 learned_addr learned my addr 172.21.15.12:0/3953321151 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.640+0000 7f4571800700 1 -- 172.21.15.12:0/3953321151 >> v2:172.21.15.159:3300/0 conn(0x7f456c15dfe0 msgr2=0x7f456c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.640+0000 7f4571800700 1 --2- 172.21.15.12:0/3953321151 >> v2:172.21.15.159:3300/0 conn(0x7f456c15dfe0 0x7f456c0ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.640+0000 7f4571800700 1 -- 172.21.15.12:0/3953321151 >> v1:172.21.15.159:6789/0 conn(0x7f456c15a5c0 legacy=0x7f456c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.640+0000 7f4571800700 1 -- 172.21.15.12:0/3953321151 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c071480 con 0x7f456c0ce670 2024-03-29T10:14:01.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.640+0000 7f456b400700 1 -- 172.21.15.12:0/3953321151 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543214456 0 0) 0x7f456c0cb310 con 0x7f456c15a5c0 2024-03-29T10:14:01.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.640+0000 7f4571800700 1 --2- 172.21.15.12:0/3953321151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 0x7f456c0cba80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f456400bfa0 tx=0x7f456400d330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b4d390c888492d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:01.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.640+0000 7f456b400700 1 -- 172.21.15.12:0/3953321151 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4564010070 con 0x7f456c0ce670 2024-03-29T10:14:01.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.640+0000 7f456b400700 1 -- 172.21.15.12:0/3953321151 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456400b2e0 con 0x7f456c0ce670 2024-03-29T10:14:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.640+0000 7f456b400700 1 -- 172.21.15.12:0/3953321151 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456400dc40 con 0x7f456c0ce670 2024-03-29T10:14:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.641+0000 7f4573800700 1 -- 172.21.15.12:0/3953321151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 msgr2=0x7f456c0cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.641+0000 7f4573800700 1 --2- 172.21.15.12:0/3953321151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 0x7f456c0cba80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f456400bfa0 tx=0x7f456400d330 comp rx=0 tx=0).stop 2024-03-29T10:14:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.641+0000 7f4573800700 1 -- 172.21.15.12:0/3953321151 shutdown_connections 2024-03-29T10:14:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.641+0000 7f4573800700 1 --2- 172.21.15.12:0/3953321151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 0x7f456c0cba80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.641+0000 7f4573800700 1 --2- 172.21.15.12:0/3953321151 >> v2:172.21.15.159:3300/0 conn(0x7f456c15dfe0 0x7f456c0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.641+0000 7f4573800700 1 -- 172.21.15.12:0/3953321151 >> 172.21.15.12:0/3953321151 conn(0x7f456c0b0d70 msgr2=0x7f456c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.641+0000 7f4573800700 1 -- 172.21.15.12:0/3953321151 shutdown_connections 2024-03-29T10:14:01.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.641+0000 7f4573800700 1 -- 172.21.15.12:0/3953321151 wait complete. 2024-03-29T10:14:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.643+0000 7f4573800700 1 Processor -- start 2024-03-29T10:14:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.643+0000 7f4573800700 1 -- start start 2024-03-29T10:14:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f4573800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 0x7f456c15d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f4573800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f456c15a5c0 0x7f456c09caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f4573800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f456c15dfe0 0x7f456c1554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f4573800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c0b08b0 con 0x7f456c15dfe0 2024-03-29T10:14:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f4573800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c15c730 con 0x7f456c15a5c0 2024-03-29T10:14:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f4573800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456c15cf90 con 0x7f456c0ce670 2024-03-29T10:14:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f4571800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f456c15dfe0 0x7f456c1554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:01.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f4571800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f456c15dfe0 0x7f456c1554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54350/0 (socket says 172.21.15.12:54350) 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f4571800700 1 -- 172.21.15.12:0/2922569504 learned_addr learned my addr 172.21.15.12:0/2922569504 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f456be00700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f456c15a5c0 0x7f456c09caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.644+0000 7f4570e00700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 0x7f456c15d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4571800700 1 -- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 msgr2=0x7f456c15d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4571800700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 0x7f456c15d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4571800700 1 -- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f456c15a5c0 msgr2=0x7f456c09caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4571800700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f456c15a5c0 0x7f456c09caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4571800700 1 -- 172.21.15.12:0/2922569504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456400bc80 con 0x7f456c15dfe0 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4571800700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f456c15dfe0 0x7f456c1554c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f456400bfd0 tx=0x7f4564004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4568c00700 1 -- 172.21.15.12:0/2922569504 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4564004450 con 0x7f456c15dfe0 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4568c00700 1 -- 172.21.15.12:0/2922569504 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45640045d0 con 0x7f456c15dfe0 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4573800700 1 -- 172.21.15.12:0/2922569504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456c1f7b20 con 0x7f456c15dfe0 2024-03-29T10:14:01.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4568c00700 1 -- 172.21.15.12:0/2922569504 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456400bb00 con 0x7f456c15dfe0 2024-03-29T10:14:01.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.645+0000 7f4573800700 1 -- 172.21.15.12:0/2922569504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456c15c940 con 0x7f456c15dfe0 2024-03-29T10:14:01.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.646+0000 7f4573800700 1 -- 172.21.15.12:0/2922569504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456c06c550 con 0x7f456c15dfe0 2024-03-29T10:14:01.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.647+0000 7f4568c00700 1 -- 172.21.15.12:0/2922569504 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 62) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f4564026450 con 0x7f456c15dfe0 2024-03-29T10:14:01.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.648+0000 7f4568c00700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f454c06ae90 0x7f454c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:01.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.648+0000 7f4568c00700 1 -- 172.21.15.12:0/2922569504 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f456408b4e0 con 0x7f456c15dfe0 2024-03-29T10:14:01.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.648+0000 7f4570e00700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f454c06ae90 0x7f454c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:01.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.649+0000 7f4570e00700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f454c06ae90 0x7f454c06d350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4560007fd0 tx=0x7f4560007de0 comp rx=0 tx=0).ready entity=mgr.6929 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:01.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.653+0000 7f4568c00700 1 -- 172.21.15.12:0/2922569504 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4564056260 con 0x7f456c15dfe0 2024-03-29T10:14:01.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.894+0000 7f4573800700 1 -- 172.21.15.12:0/2922569504 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f456c0cb560 con 0x7f454c06ae90 2024-03-29T10:14:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.984+0000 7f4568c00700 1 -- 172.21.15.12:0/2922569504 <== mgr.6929 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4560004b10 con 0x7f454c06ae90 2024-03-29T10:14:01.984 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path snap_test_dir1 2024-03-29T10:14:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 -- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f454c06ae90 msgr2=0x7f454c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f454c06ae90 0x7f454c06d350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4560007fd0 tx=0x7f4560007de0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 -- 172.21.15.12:0/2922569504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f456c15dfe0 msgr2=0x7f456c1554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f456c15dfe0 0x7f456c1554c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f456400bfd0 tx=0x7f4564004ad0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 -- 172.21.15.12:0/2922569504 shutdown_connections 2024-03-29T10:14:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f454c06ae90 0x7f454c06d350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f456c15dfe0 0x7f456c1554c0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f456c0ce670 0x7f456c15d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 --2- 172.21.15.12:0/2922569504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f456c15a5c0 0x7f456c09caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 -- 172.21.15.12:0/2922569504 >> 172.21.15.12:0/2922569504 conn(0x7f456c0b0d70 msgr2=0x7f456c0ceef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 -- 172.21.15.12:0/2922569504 shutdown_connections 2024-03-29T10:14:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:01.987+0000 7f4573800700 1 -- 172.21.15.12:0/2922569504 wait complete. 2024-03-29T10:14:02.004 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-03-29T10:14:02.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:02.004 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T10:14:02.049 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_14_00_UTC 2024-03-29T10:14:02.050 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_14_00_UTC 2024-03-29T10:14:02.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:02.051 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_14_00_UTC) 2024-03-29T10:14:02.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:02.141 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-29T10:14:02.189 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T10:14:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.391+0000 7f5721200700 1 Processor -- start 2024-03-29T10:14:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.392+0000 7f5721200700 1 -- start start 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5721200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f571c1611b0 0x7f571c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5721200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f571c15f4d0 0x7f571c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5721200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c0ce270 con 0x7f571c0d21f0 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5721200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571c15b600 con 0x7f571c1611b0 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5721200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571c0d2760 con 0x7f571c15f4d0 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5719e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f571c1611b0 0x7f571c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5719e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f571c1611b0 0x7f571c15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54364/0 (socket says 172.21.15.12:54364) 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5719e00700 1 -- 172.21.15.12:0/4282745796 learned_addr learned my addr 172.21.15.12:0/4282745796 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5719400700 1 --2- 172.21.15.12:0/4282745796 >> v2:172.21.15.159:3300/0 conn(0x7f571c15f4d0 0x7f571c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5719e00700 1 -- 172.21.15.12:0/4282745796 >> v2:172.21.15.159:3300/0 conn(0x7f571c15f4d0 msgr2=0x7f571c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5719e00700 1 --2- 172.21.15.12:0/4282745796 >> v2:172.21.15.159:3300/0 conn(0x7f571c15f4d0 0x7f571c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5719e00700 1 -- 172.21.15.12:0/4282745796 >> v1:172.21.15.159:6789/0 conn(0x7f571c0d21f0 legacy=0x7f571c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.393+0000 7f5719e00700 1 -- 172.21.15.12:0/4282745796 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c0d2a40 con 0x7f571c1611b0 2024-03-29T10:14:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.394+0000 7f5719e00700 1 --2- 172.21.15.12:0/4282745796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 0x7f571c15ef90 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f571c08c030 tx=0x7f570c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf916dc0cb6a1b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.394+0000 7f5718a00700 1 -- 172.21.15.12:0/4282745796 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c00b870 con 0x7f571c1611b0 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.394+0000 7f5718a00700 1 -- 172.21.15.12:0/4282745796 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c00b9d0 con 0x7f571c1611b0 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.394+0000 7f5718a00700 1 -- 172.21.15.12:0/4282745796 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c00bd60 con 0x7f571c1611b0 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.394+0000 7f5721200700 1 -- 172.21.15.12:0/4282745796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 msgr2=0x7f571c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.394+0000 7f5721200700 1 --2- 172.21.15.12:0/4282745796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 0x7f571c15ef90 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f571c08c030 tx=0x7f570c009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.395+0000 7f5721200700 1 -- 172.21.15.12:0/4282745796 shutdown_connections 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.395+0000 7f5721200700 1 --2- 172.21.15.12:0/4282745796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 0x7f571c15ef90 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.395+0000 7f5721200700 1 --2- 172.21.15.12:0/4282745796 >> v2:172.21.15.159:3300/0 conn(0x7f571c15f4d0 0x7f571c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.395+0000 7f5721200700 1 -- 172.21.15.12:0/4282745796 >> 172.21.15.12:0/4282745796 conn(0x7f571c08aff0 msgr2=0x7f571c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.395+0000 7f5721200700 1 -- 172.21.15.12:0/4282745796 shutdown_connections 2024-03-29T10:14:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.395+0000 7f5721200700 1 -- 172.21.15.12:0/4282745796 wait complete. 2024-03-29T10:14:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.396+0000 7f5721200700 1 Processor -- start 2024-03-29T10:14:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.397+0000 7f5721200700 1 -- start start 2024-03-29T10:14:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.397+0000 7f5721200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c0d21f0 0x7f571c14d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.397+0000 7f5721200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15f4d0 0x7f571c0b8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.397+0000 7f5721200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 0x7f571c0c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.397+0000 7f5721200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c14df20 con 0x7f571c1611b0 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.397+0000 7f5721200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0cabc0 con 0x7f571c0d21f0 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.397+0000 7f5721200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0b8ed0 con 0x7f571c15f4d0 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f571a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 0x7f571c0c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f571a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 0x7f571c0c8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54370/0 (socket says 172.21.15.12:54370) 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f571a800700 1 -- 172.21.15.12:0/1608950275 learned_addr learned my addr 172.21.15.12:0/1608950275 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f5719e00700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c0d21f0 0x7f571c14d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f5719400700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15f4d0 0x7f571c0b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f571a800700 1 -- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15f4d0 msgr2=0x7f571c0b8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f571a800700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15f4d0 0x7f571c0b8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f571a800700 1 -- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c0d21f0 msgr2=0x7f571c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f571a800700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c0d21f0 0x7f571c14d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f571a800700 1 -- 172.21.15.12:0/1608950275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c009550 con 0x7f571c1611b0 2024-03-29T10:14:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.398+0000 7f571a800700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 0x7f571c0c8870 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f571400cc50 tx=0x7f571400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.399+0000 7f5712000700 1 -- 172.21.15.12:0/1608950275 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714003c00 con 0x7f571c1611b0 2024-03-29T10:14:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.399+0000 7f5712000700 1 -- 172.21.15.12:0/1608950275 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5714003d60 con 0x7f571c1611b0 2024-03-29T10:14:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.399+0000 7f5712000700 1 -- 172.21.15.12:0/1608950275 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571400aab0 con 0x7f571c1611b0 2024-03-29T10:14:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.399+0000 7f5721200700 1 -- 172.21.15.12:0/1608950275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c0b85c0 con 0x7f571c1611b0 2024-03-29T10:14:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.399+0000 7f5721200700 1 -- 172.21.15.12:0/1608950275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c0c81c0 con 0x7f571c1611b0 2024-03-29T10:14:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.400+0000 7f5721200700 1 -- 172.21.15.12:0/1608950275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571c067720 con 0x7f571c1611b0 2024-03-29T10:14:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.401+0000 7f5712000700 1 -- 172.21.15.12:0/1608950275 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 63) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f571400ac10 con 0x7f571c1611b0 2024-03-29T10:14:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.402+0000 7f5712000700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06af50 0x7f56fc06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.402+0000 7f5712000700 1 -- 172.21.15.12:0/1608950275 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f5714089890 con 0x7f571c1611b0 2024-03-29T10:14:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.402+0000 7f5719e00700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06af50 0x7f56fc06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.403+0000 7f5719e00700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06af50 0x7f56fc06d410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f570c014040 tx=0x7f570c008030 comp rx=0 tx=0).ready entity=mgr.6929 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:02.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.406+0000 7f5712000700 1 -- 172.21.15.12:0/1608950275 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f57140545a0 con 0x7f571c1611b0 2024-03-29T10:14:02.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:02.676+0000 7f5721200700 1 -- 172.21.15.12:0/1608950275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f571c0b5d00 con 0x7f571c1611b0 2024-03-29T10:14:03.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.234+0000 7f5712000700 1 -- 172.21.15.12:0/1608950275 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f5714054970 con 0x7f571c1611b0 2024-03-29T10:14:03.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.235+0000 7f5712000700 1 -- 172.21.15.12:0/1608950275 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v64) v1 ==== 93+0+0 (secure 0 0 0) 0x7f5714072200 con 0x7f571c1611b0 2024-03-29T10:14:03.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.238+0000 7f5721200700 1 -- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06af50 msgr2=0x7f56fc06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:03.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.238+0000 7f5721200700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06af50 0x7f56fc06d410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f570c014040 tx=0x7f570c008030 comp rx=0 tx=0).stop 2024-03-29T10:14:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.238+0000 7f5721200700 1 -- 172.21.15.12:0/1608950275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 msgr2=0x7f571c0c8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.238+0000 7f5721200700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 0x7f571c0c8870 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f571400cc50 tx=0x7f571400cf60 comp rx=0 tx=0).stop 2024-03-29T10:14:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.238+0000 7f5721200700 1 -- 172.21.15.12:0/1608950275 shutdown_connections 2024-03-29T10:14:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.238+0000 7f5721200700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06af50 0x7f56fc06d410 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.238+0000 7f5721200700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c1611b0 0x7f571c0c8870 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.238+0000 7f5721200700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c0d21f0 0x7f571c14d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.238+0000 7f5721200700 1 --2- 172.21.15.12:0/1608950275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15f4d0 0x7f571c0b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.238+0000 7f5721200700 1 -- 172.21.15.12:0/1608950275 >> 172.21.15.12:0/1608950275 conn(0x7f571c08aff0 msgr2=0x7f571c0ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.239+0000 7f5721200700 1 -- 172.21.15.12:0/1608950275 shutdown_connections 2024-03-29T10:14:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.239+0000 7f5721200700 1 -- 172.21.15.12:0/1608950275 wait complete. 2024-03-29T10:14:03.260 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:14:03.261 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:14:03.362 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:14:03.449 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:14:03.449 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:14:03.450 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:03.464 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:03.464+0000 7fd4361a1240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:14:03.465 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:03.466+0000 7f209ba96240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:14:03.496 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:14:03.497 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:03.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:03.651 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:03.652+0000 7f209ba96240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:14:03.668 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:03.669+0000 7fd4361a1240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:14:03.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.843+0000 7fe2c7a00700 1 Processor -- start 2024-03-29T10:14:03.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.844+0000 7fe2c7a00700 1 -- start start 2024-03-29T10:14:03.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.845+0000 7fe2c7a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c015b5d0 0x7fe2c00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:03.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.845+0000 7fe2c7a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c015fd50 con 0x7fe2c00cb3f0 2024-03-29T10:14:03.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.845+0000 7fe2c7a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c015f140 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.845+0000 7fe2c7a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c015b9b0 con 0x7fe2c015b5d0 2024-03-29T10:14:03.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.845+0000 7fe2bfe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe2c00cb3f0 0x7fe2c00ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43266/0 (socket says 172.21.15.12:43266) 2024-03-29T10:14:03.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.845+0000 7fe2bfe00700 1 -- 172.21.15.12:0/3179521819 learned_addr learned my addr 172.21.15.12:0/3179521819 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:03.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.845+0000 7fe2c5000700 1 --2- 172.21.15.12:0/3179521819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c015b5d0 0x7fe2c00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:03.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.846+0000 7fe2c5000700 1 -- 172.21.15.12:0/3179521819 >> v1:172.21.15.159:6789/0 conn(0x7fe2c00ac7d0 legacy=0x7fe2c006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:03.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.846+0000 7fe2c5000700 1 -- 172.21.15.12:0/3179521819 >> v1:172.21.15.12:6789/0 conn(0x7fe2c00cb3f0 legacy=0x7fe2c00ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.846+0000 7fe2c5000700 1 -- 172.21.15.12:0/3179521819 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c015f420 con 0x7fe2c015b5d0 2024-03-29T10:14:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.846+0000 7fe2bf400700 1 -- 172.21.15.12:0/3179521819 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575945738 0 0) 0x7fe2c015fd50 con 0x7fe2c00cb3f0 2024-03-29T10:14:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.846+0000 7fe2c5000700 1 --2- 172.21.15.12:0/3179521819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c015b5d0 0x7fe2c00ae3f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009870 tx=0x7fe2b4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3edb609c6e0d4b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.846+0000 7fe2bf400700 1 -- 172.21.15.12:0/3179521819 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4011830 con 0x7fe2c015b5d0 2024-03-29T10:14:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.846+0000 7fe2bf400700 1 -- 172.21.15.12:0/3179521819 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b4011990 con 0x7fe2c015b5d0 2024-03-29T10:14:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.846+0000 7fe2bf400700 1 -- 172.21.15.12:0/3179521819 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4011d20 con 0x7fe2c015b5d0 2024-03-29T10:14:03.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.847+0000 7fe2c7a00700 1 -- 172.21.15.12:0/3179521819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c015b5d0 msgr2=0x7fe2c00ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:03.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.847+0000 7fe2c7a00700 1 --2- 172.21.15.12:0/3179521819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c015b5d0 0x7fe2c00ae3f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009870 tx=0x7fe2b4009b80 comp rx=0 tx=0).stop 2024-03-29T10:14:03.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.847+0000 7fe2c7a00700 1 -- 172.21.15.12:0/3179521819 shutdown_connections 2024-03-29T10:14:03.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.847+0000 7fe2c7a00700 1 --2- 172.21.15.12:0/3179521819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c015b5d0 0x7fe2c00ae3f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:03.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.847+0000 7fe2c7a00700 1 -- 172.21.15.12:0/3179521819 >> 172.21.15.12:0/3179521819 conn(0x7fe2c00cf190 msgr2=0x7fe2c00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:03.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.847+0000 7fe2c7a00700 1 -- 172.21.15.12:0/3179521819 shutdown_connections 2024-03-29T10:14:03.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.847+0000 7fe2c7a00700 1 -- 172.21.15.12:0/3179521819 wait complete. 2024-03-29T10:14:03.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.849+0000 7fe2c7a00700 1 Processor -- start 2024-03-29T10:14:03.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.849+0000 7fe2c7a00700 1 -- start start 2024-03-29T10:14:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2c7a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2c00ac7d0 0x7fe2c01f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2c7a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2c015b5d0 0x7fe2c0071ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2c7a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c00b0440 0x7fe2c00c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2c7a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c0070e30 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2c7a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c0071110 con 0x7fe2c015b5d0 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2c7a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00713f0 con 0x7fe2c00b0440 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2c5000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2c00ac7d0 0x7fe2c01f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2c5000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2c00ac7d0 0x7fe2c01f28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54388/0 (socket says 172.21.15.12:54388) 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2c5000700 1 -- 172.21.15.12:0/1918043511 learned_addr learned my addr 172.21.15.12:0/1918043511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2bfe00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2c015b5d0 0x7fe2c0071ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.850+0000 7fe2c5a00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c00b0440 0x7fe2c00c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2c5000700 1 -- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c00b0440 msgr2=0x7fe2c00c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2c5000700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c00b0440 0x7fe2c00c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2c5000700 1 -- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2c015b5d0 msgr2=0x7fe2c0071ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2c5000700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2c015b5d0 0x7fe2c0071ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2c5000700 1 -- 172.21.15.12:0/1918043511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40094a0 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2c5000700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2c00ac7d0 0x7fe2c01f28b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4014040 tx=0x7fe2b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:03.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2bcc00700 1 -- 172.21.15.12:0/1918043511 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b40036a0 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2bcc00700 1 -- 172.21.15.12:0/1918043511 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b401f070 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2c7a00700 1 -- 172.21.15.12:0/1918043511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c00716d0 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2bcc00700 1 -- 172.21.15.12:0/1918043511 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4003a40 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.851+0000 7fe2c7a00700 1 -- 172.21.15.12:0/1918043511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c00709c0 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.853+0000 7fe2c7a00700 1 -- 172.21.15.12:0/1918043511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c0071bf0 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.853+0000 7fe2bcc00700 1 -- 172.21.15.12:0/1918043511 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fe2b4003ba0 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.854+0000 7fe2bcc00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2a006aed0 0x7fe2a006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:03.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.854+0000 7fe2bcc00700 1 -- 172.21.15.12:0/1918043511 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe2b4021070 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.854+0000 7fe2bfe00700 1 -- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2a006aed0 msgr2=0x7fe2a006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:03.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.854+0000 7fe2bfe00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2a006aed0 0x7fe2a006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:03.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:03.859+0000 7fe2bcc00700 1 -- 172.21.15.12:0/1918043511 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2c0071bf0 con 0x7fe2c00ac7d0 2024-03-29T10:14:03.968 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:03.969+0000 7f209ba96240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:14:04.009 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:04.010+0000 7fd4361a1240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:14:04.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.055+0000 7fe2bfe00700 1 -- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2a006aed0 msgr2=0x7fe2a006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:04.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.055+0000 7fe2bfe00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2a006aed0 0x7fe2a006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.075+0000 7fe2c7a00700 1 -- 172.21.15.12:0/1918043511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2c0071bf0 con 0x7fe2c00ac7d0 2024-03-29T10:14:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.077+0000 7fe2bcc00700 1 -- 172.21.15.12:0/1918043511 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16548 (secure 0 0 0) 0x7fe2c0071bf0 con 0x7fe2c00ac7d0 2024-03-29T10:14:04.076 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:04.076 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":125,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:13:04.000616+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:12:45.171490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-29T10:12:45.787701+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1231831807":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/3573895366":"2024-03-30T10:13:04.000570+0000","172.21.15.159:6833/43444":"2024-03-30T10:13:04.000570+0000","172.21.15.159:6832/43444":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/2082234733":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/3500442951":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/949098073":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/497911868":"2024-03-30T10:13:04.000570+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 -- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2a006aed0 msgr2=0x7fe2a006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2a006aed0 0x7fe2a006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 -- 172.21.15.12:0/1918043511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2c00ac7d0 msgr2=0x7fe2c01f28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2c00ac7d0 0x7fe2c01f28b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4014040 tx=0x7fe2b4004750 comp rx=0 tx=0).stop 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 -- 172.21.15.12:0/1918043511 shutdown_connections 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe2a006aed0 0x7fe2a006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2c00b0440 0x7fe2c00c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2c00ac7d0 0x7fe2c01f28b0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 --2- 172.21.15.12:0/1918043511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2c015b5d0 0x7fe2c0071ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 -- 172.21.15.12:0/1918043511 >> 172.21.15.12:0/1918043511 conn(0x7fe2c00cf190 msgr2=0x7fe2c00af030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 -- 172.21.15.12:0/1918043511 shutdown_connections 2024-03-29T10:14:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.079+0000 7fe2c7a00700 1 -- 172.21.15.12:0/1918043511 wait complete. 2024-03-29T10:14:04.099 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:12:45.171490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '123', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:12:45.787701+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:14:04.100 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:14:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.346+0000 7fa9ce200700 1 Processor -- start 2024-03-29T10:14:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.347+0000 7fa9ce200700 1 -- start start 2024-03-29T10:14:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.348+0000 7fa9ce200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9c815d9d0 0x7fa9c80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.348+0000 7fa9ce200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa9c80d16a0 0x7fa9c80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.348+0000 7fa9ce200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c8161420 con 0x7fa9c80d2370 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.348+0000 7fa9ce200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c80d27e0 con 0x7fa9c815d9d0 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.348+0000 7fa9ce200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c8161700 con 0x7fa9c80d16a0 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.348+0000 7fa9c7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9c815d9d0 0x7fa9c80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.348+0000 7fa9c7e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa9c80d2370 0x7fa9c815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43272/0 (socket says 172.21.15.12:43272) 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.348+0000 7fa9c7e00700 1 -- 172.21.15.12:0/2939687497 learned_addr learned my addr 172.21.15.12:0/2939687497 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.348+0000 7fa9c6a00700 1 --2- 172.21.15.12:0/2939687497 >> v2:172.21.15.159:3300/0 conn(0x7fa9c80d16a0 0x7fa9c80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c6000700 1 -- 172.21.15.12:0/2939687497 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511382788 0 0) 0x7fa9c8161420 con 0x7fa9c80d2370 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c6000700 1 -- 172.21.15.12:0/2939687497 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ac0035b0 con 0x7fa9c80d2370 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c6a00700 1 -- 172.21.15.12:0/2939687497 >> v2:172.21.15.12:3300/0 conn(0x7fa9c815d9d0 msgr2=0x7fa9c80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:04.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c6a00700 1 --2- 172.21.15.12:0/2939687497 >> v2:172.21.15.12:3300/0 conn(0x7fa9c815d9d0 0x7fa9c80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c6a00700 1 -- 172.21.15.12:0/2939687497 >> v1:172.21.15.12:6789/0 conn(0x7fa9c80d2370 legacy=0x7fa9c815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c6a00700 1 -- 172.21.15.12:0/2939687497 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c81619e0 con 0x7fa9c80d16a0 2024-03-29T10:14:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c7400700 1 --2- 172.21.15.12:0/2939687497 >> v2:172.21.15.12:3300/0 conn(0x7fa9c815d9d0 0x7fa9c80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c6a00700 1 --2- 172.21.15.12:0/2939687497 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9c80d16a0 0x7fa9c80d1e30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80d12f0 tx=0x7fa9b4009b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d56ebdef5d5b865 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c6000700 1 -- 172.21.15.12:0/2939687497 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4005e30 con 0x7fa9c80d16a0 2024-03-29T10:14:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c6000700 1 -- 172.21.15.12:0/2939687497 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b4006030 con 0x7fa9c80d16a0 2024-03-29T10:14:04.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.349+0000 7fa9c6000700 1 -- 172.21.15.12:0/2939687497 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b40063c0 con 0x7fa9c80d16a0 2024-03-29T10:14:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.350+0000 7fa9ce200700 1 -- 172.21.15.12:0/2939687497 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9c80d16a0 msgr2=0x7fa9c80d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.350+0000 7fa9ce200700 1 --2- 172.21.15.12:0/2939687497 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9c80d16a0 0x7fa9c80d1e30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80d12f0 tx=0x7fa9b4009b00 comp rx=0 tx=0).stop 2024-03-29T10:14:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.350+0000 7fa9ce200700 1 -- 172.21.15.12:0/2939687497 shutdown_connections 2024-03-29T10:14:04.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.350+0000 7fa9ce200700 1 --2- 172.21.15.12:0/2939687497 >> v2:172.21.15.12:3300/0 conn(0x7fa9c815d9d0 0x7fa9c80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.350+0000 7fa9ce200700 1 --2- 172.21.15.12:0/2939687497 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9c80d16a0 0x7fa9c80d1e30 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.350+0000 7fa9ce200700 1 -- 172.21.15.12:0/2939687497 >> 172.21.15.12:0/2939687497 conn(0x7fa9c808aff0 msgr2=0x7fa9c80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:04.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.350+0000 7fa9ce200700 1 -- 172.21.15.12:0/2939687497 shutdown_connections 2024-03-29T10:14:04.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.350+0000 7fa9ce200700 1 -- 172.21.15.12:0/2939687497 wait complete. 2024-03-29T10:14:04.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.352+0000 7fa9ce200700 1 Processor -- start 2024-03-29T10:14:04.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.352+0000 7fa9ce200700 1 -- start start 2024-03-29T10:14:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.353+0000 7fa9ce200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9c80d16a0 0x7fa9c81f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.353+0000 7fa9ce200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9c80d2370 0x7fa9c80ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.353+0000 7fa9ce200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9c815d9d0 0x7fa9c81588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.353+0000 7fa9ce200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c80b60c0 con 0x7fa9c80d2370 2024-03-29T10:14:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.353+0000 7fa9ce200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c81f3860 con 0x7fa9c80d16a0 2024-03-29T10:14:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.353+0000 7fa9ce200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c81f3b40 con 0x7fa9c815d9d0 2024-03-29T10:14:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.353+0000 7fa9c6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9c80d2370 0x7fa9c80ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.353+0000 7fa9c6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9c80d2370 0x7fa9c80ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54408/0 (socket says 172.21.15.12:54408) 2024-03-29T10:14:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.354+0000 7fa9c6a00700 1 -- 172.21.15.12:0/2697522733 learned_addr learned my addr 172.21.15.12:0/2697522733 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:04.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.354+0000 7fa9c7400700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9c80d16a0 0x7fa9c81f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.354+0000 7fa9c7e00700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9c815d9d0 0x7fa9c81588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.354+0000 7fa9c6a00700 1 -- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9c815d9d0 msgr2=0x7fa9c81588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.354+0000 7fa9c6a00700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9c815d9d0 0x7fa9c81588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.354+0000 7fa9c6a00700 1 -- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9c80d16a0 msgr2=0x7fa9c81f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.354+0000 7fa9c6a00700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9c80d16a0 0x7fa9c81f3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.354+0000 7fa9c6a00700 1 -- 172.21.15.12:0/2697522733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b80094a0 con 0x7fa9c80d2370 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.354+0000 7fa9c6a00700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9c80d2370 0x7fa9c80ccf70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4000c00 tx=0x7fa9b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.355+0000 7fa9bf400700 1 -- 172.21.15.12:0/2697522733 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b40067b0 con 0x7fa9c80d2370 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.355+0000 7fa9bf400700 1 -- 172.21.15.12:0/2697522733 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b4020070 con 0x7fa9c80d2370 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.355+0000 7fa9ce200700 1 -- 172.21.15.12:0/2697522733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b4009430 con 0x7fa9c80d2370 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.355+0000 7fa9bf400700 1 -- 172.21.15.12:0/2697522733 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4004280 con 0x7fa9c80d2370 2024-03-29T10:14:04.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.355+0000 7fa9ce200700 1 -- 172.21.15.12:0/2697522733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c80cd4b0 con 0x7fa9c80d2370 2024-03-29T10:14:04.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.356+0000 7fa9ce200700 1 -- 172.21.15.12:0/2697522733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c80b6290 con 0x7fa9c80d2370 2024-03-29T10:14:04.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.357+0000 7fa9bf400700 1 -- 172.21.15.12:0/2697522733 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fa9b4022030 con 0x7fa9c80d2370 2024-03-29T10:14:04.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.357+0000 7fa9bf400700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9a806aed0 0x7fa9a806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.358+0000 7fa9bf400700 1 -- 172.21.15.12:0/2697522733 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fa9b408acc0 con 0x7fa9c80d2370 2024-03-29T10:14:04.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.358+0000 7fa9c7400700 1 -- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9a806aed0 msgr2=0x7fa9a806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:04.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.358+0000 7fa9c7400700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9a806aed0 0x7fa9a806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:04.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.362+0000 7fa9bf400700 1 -- 172.21.15.12:0/2697522733 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa9c80b6290 con 0x7fa9c80d2370 2024-03-29T10:14:04.496 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:04.496+0000 7f209ba96240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:14:04.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.559+0000 7fa9c7400700 1 -- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9a806aed0 msgr2=0x7fa9a806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:04.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.559+0000 7fa9c7400700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9a806aed0 0x7fa9a806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:04.575 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:04.576+0000 7fd4361a1240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:14:04.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.584+0000 7fa9ce200700 1 -- 172.21.15.12:0/2697522733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9c80b6290 con 0x7fa9c80d2370 2024-03-29T10:14:04.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.584+0000 7fa9bf400700 1 -- 172.21.15.12:0/2697522733 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v125) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9c80b6290 con 0x7fa9c80d2370 2024-03-29T10:14:04.584 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:14:04.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 -- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9a806aed0 msgr2=0x7fa9a806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:04.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9a806aed0 0x7fa9a806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 -- 172.21.15.12:0/2697522733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9c80d2370 msgr2=0x7fa9c80ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9c80d2370 0x7fa9c80ccf70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4000c00 tx=0x7fa9b4004750 comp rx=0 tx=0).stop 2024-03-29T10:14:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 -- 172.21.15.12:0/2697522733 shutdown_connections 2024-03-29T10:14:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa9a806aed0 0x7fa9a806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9c815d9d0 0x7fa9c81588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9c80d16a0 0x7fa9c81f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 --2- 172.21.15.12:0/2697522733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9c80d2370 0x7fa9c80ccf70 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 -- 172.21.15.12:0/2697522733 >> 172.21.15.12:0/2697522733 conn(0x7fa9c808aff0 msgr2=0x7fa9c80b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 -- 172.21.15.12:0/2697522733 shutdown_connections 2024-03-29T10:14:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.587+0000 7fa9ce200700 1 -- 172.21.15.12:0/2697522733 wait complete. 2024-03-29T10:14:04.605 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:14:04.750 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:04.751+0000 7f209ba96240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:14:04.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.812+0000 7faaa2c00700 1 Processor -- start 2024-03-29T10:14:04.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.812+0000 7faaa2c00700 1 -- start start 2024-03-29T10:14:04.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.813+0000 7faaa2c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7faa9c1611b0 0x7faa9c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.813+0000 7faaa2c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c0cee10 0x7faa9c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.813+0000 7faaa2c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9c0ce270 con 0x7faa9c15f4d0 2024-03-29T10:14:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.813+0000 7faaa2c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa9c15b600 con 0x7faa9c1611b0 2024-03-29T10:14:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.813+0000 7faaa2c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c0d2760 con 0x7faa9c0cee10 2024-03-29T10:14:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.814+0000 7faaa0c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c0cee10 0x7faa9c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.814+0000 7faa9be00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7faa9c1611b0 0x7faa9c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.814+0000 7faaa0c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c0cee10 0x7faa9c161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:48306/0 (socket says 172.21.15.12:48306) 2024-03-29T10:14:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.814+0000 7faaa0c00700 1 -- 172.21.15.12:0/75545743 learned_addr learned my addr 172.21.15.12:0/75545743 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.814+0000 7faaa0c00700 1 -- 172.21.15.12:0/75545743 >> v2:172.21.15.159:3300/0 conn(0x7faa9c1611b0 msgr2=0x7faa9c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.814+0000 7faaa0c00700 1 --2- 172.21.15.12:0/75545743 >> v2:172.21.15.159:3300/0 conn(0x7faa9c1611b0 0x7faa9c15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.814+0000 7faaa0c00700 1 -- 172.21.15.12:0/75545743 >> v1:172.21.15.159:6789/0 conn(0x7faa9c15f4d0 legacy=0x7faa9c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.814+0000 7faaa0c00700 1 -- 172.21.15.12:0/75545743 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c0d2a40 con 0x7faa9c0cee10 2024-03-29T10:14:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.814+0000 7faa9be00700 1 --2- 172.21.15.12:0/75545743 >> v2:172.21.15.159:3300/0 conn(0x7faa9c1611b0 0x7faa9c15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.815+0000 7faaa0c00700 1 --2- 172.21.15.12:0/75545743 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c0cee10 0x7faa9c161590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faa94009e50 tx=0x7faa9400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a83192887c52eeac server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.815+0000 7faa9aa00700 1 -- 172.21.15.12:0/75545743 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9400e070 con 0x7faa9c0cee10 2024-03-29T10:14:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.815+0000 7faa9aa00700 1 -- 172.21.15.12:0/75545743 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9400b910 con 0x7faa9c0cee10 2024-03-29T10:14:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.815+0000 7faa9aa00700 1 -- 172.21.15.12:0/75545743 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9400bcc0 con 0x7faa9c0cee10 2024-03-29T10:14:04.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.816+0000 7faaa2c00700 1 -- 172.21.15.12:0/75545743 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c0cee10 msgr2=0x7faa9c161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:04.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.816+0000 7faaa2c00700 1 --2- 172.21.15.12:0/75545743 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c0cee10 0x7faa9c161590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faa94009e50 tx=0x7faa9400b330 comp rx=0 tx=0).stop 2024-03-29T10:14:04.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.816+0000 7faaa2c00700 1 -- 172.21.15.12:0/75545743 shutdown_connections 2024-03-29T10:14:04.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.816+0000 7faaa2c00700 1 --2- 172.21.15.12:0/75545743 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c0cee10 0x7faa9c161590 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.816+0000 7faaa2c00700 1 --2- 172.21.15.12:0/75545743 >> v2:172.21.15.159:3300/0 conn(0x7faa9c1611b0 0x7faa9c15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.816+0000 7faaa2c00700 1 -- 172.21.15.12:0/75545743 >> 172.21.15.12:0/75545743 conn(0x7faa9c08aff0 msgr2=0x7faa9c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:04.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.816+0000 7faaa2c00700 1 -- 172.21.15.12:0/75545743 shutdown_connections 2024-03-29T10:14:04.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.816+0000 7faaa2c00700 1 -- 172.21.15.12:0/75545743 wait complete. 2024-03-29T10:14:04.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.818+0000 7faaa2c00700 1 Processor -- start 2024-03-29T10:14:04.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.818+0000 7faaa2c00700 1 -- start start 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faaa2c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9c0cee10 0x7faa9c0a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faaa2c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c15f4d0 0x7faa9c09e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faaa2c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa9c1611b0 0x7faa9c09f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faaa2c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c0cd4a0 con 0x7faa9c0cee10 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faaa2c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c0a07b0 con 0x7faa9c1611b0 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faaa2c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c09f940 con 0x7faa9c15f4d0 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faa9be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9c0cee10 0x7faa9c0a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faa9be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9c0cee10 0x7faa9c0a0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54410/0 (socket says 172.21.15.12:54410) 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faa9be00700 1 -- 172.21.15.12:0/4179746736 learned_addr learned my addr 172.21.15.12:0/4179746736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faaa0c00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa9c1611b0 0x7faa9c09f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faa9b400700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c15f4d0 0x7faa9c09e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.819+0000 7faa9be00700 1 -- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c15f4d0 msgr2=0x7faa9c09e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faa9be00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c15f4d0 0x7faa9c09e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faa9be00700 1 -- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa9c1611b0 msgr2=0x7faa9c09f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faa9be00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa9c1611b0 0x7faa9c09f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faa9be00700 1 -- 172.21.15.12:0/4179746736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c0094a0 con 0x7faa9c0cee10 2024-03-29T10:14:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faaa0c00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa9c1611b0 0x7faa9c09f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faa9be00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9c0cee10 0x7faa9c0a0c40 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00ebe0 tx=0x7faa8c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faa93e00700 1 -- 172.21.15.12:0/4179746736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c009760 con 0x7faa9c0cee10 2024-03-29T10:14:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faa93e00700 1 -- 172.21.15.12:0/4179746736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8c00c850 con 0x7faa9c0cee10 2024-03-29T10:14:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faaa2c00700 1 -- 172.21.15.12:0/4179746736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa94009b30 con 0x7faa9c0cee10 2024-03-29T10:14:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faa93e00700 1 -- 172.21.15.12:0/4179746736 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c00cbe0 con 0x7faa9c0cee10 2024-03-29T10:14:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.820+0000 7faaa2c00700 1 -- 172.21.15.12:0/4179746736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c0a0160 con 0x7faa9c0cee10 2024-03-29T10:14:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.821+0000 7faaa2c00700 1 -- 172.21.15.12:0/4179746736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c067720 con 0x7faa9c0cee10 2024-03-29T10:14:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.822+0000 7faa93e00700 1 -- 172.21.15.12:0/4179746736 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7faa8c00cd40 con 0x7faa9c0cee10 2024-03-29T10:14:04.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.824+0000 7faa93e00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faa7c06af00 0x7faa7c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:04.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.824+0000 7faa9b400700 1 -- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faa7c06af00 msgr2=0x7faa7c06d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:04.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.824+0000 7faa93e00700 1 -- 172.21.15.12:0/4179746736 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6053+0+0 (secure 0 0 0) 0x7faa8c089a60 con 0x7faa9c0cee10 2024-03-29T10:14:04.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.824+0000 7faa9b400700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faa7c06af00 0x7faa7c06d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:04.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:04.828+0000 7faa93e00700 1 -- 172.21.15.12:0/4179746736 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faa8c0545d0 con 0x7faa9c0cee10 2024-03-29T10:14:04.848 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:04.849+0000 7fd4361a1240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:14:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.025+0000 7faa9b400700 1 -- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faa7c06af00 msgr2=0x7faa7c06d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.025+0000 7faa9b400700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faa7c06af00 0x7faa7c06d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:05.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.043+0000 7faaa2c00700 1 -- 172.21.15.12:0/4179746736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa9c0b5d00 con 0x7faa9c0cee10 2024-03-29T10:14:05.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.044+0000 7faa93e00700 1 -- 172.21.15.12:0/4179746736 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v125) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa8c057bf0 con 0x7faa9c0cee10 2024-03-29T10:14:05.043 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:14:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 -- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faa7c06af00 msgr2=0x7faa7c06d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faa7c06af00 0x7faa7c06d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 -- 172.21.15.12:0/4179746736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9c0cee10 msgr2=0x7faa9c0a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9c0cee10 0x7faa9c0a0c40 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00ebe0 tx=0x7faa8c00eef0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 -- 172.21.15.12:0/4179746736 shutdown_connections 2024-03-29T10:14:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faa7c06af00 0x7faa7c06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa9c1611b0 0x7faa9c09f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9c0cee10 0x7faa9c0a0c40 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 --2- 172.21.15.12:0/4179746736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa9c15f4d0 0x7faa9c09e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 -- 172.21.15.12:0/4179746736 >> 172.21.15.12:0/4179746736 conn(0x7faa9c08aff0 msgr2=0x7faa9c15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.046+0000 7faaa2c00700 1 -- 172.21.15.12:0/4179746736 shutdown_connections 2024-03-29T10:14:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.047+0000 7faaa2c00700 1 -- 172.21.15.12:0/4179746736 wait complete. 2024-03-29T10:14:05.064 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:14:05.206 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:05.207+0000 7f209ba96240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:14:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.272+0000 7f2ecb200700 1 Processor -- start 2024-03-29T10:14:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.273+0000 7f2ecb200700 1 -- start start 2024-03-29T10:14:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.274+0000 7f2ecb200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ec4157f90 0x7f2ec40b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.274+0000 7f2ecb200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2ec41613e0 0x7f2ec4161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.274+0000 7f2ecb200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec40b3070 con 0x7f2ec40b3420 2024-03-29T10:14:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.274+0000 7f2ecb200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec4158370 con 0x7f2ec4157f90 2024-03-29T10:14:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.274+0000 7f2ecb200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec40b2710 con 0x7f2ec41613e0 2024-03-29T10:14:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.274+0000 7f2ec3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ec4157f90 0x7f2ec40b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.274+0000 7f2ec9200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2ec40b3420 0x7f2ec40b3830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43274/0 (socket says 172.21.15.12:43274) 2024-03-29T10:14:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.274+0000 7f2ec3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ec4157f90 0x7f2ec40b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54420/0 (socket says 172.21.15.12:54420) 2024-03-29T10:14:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.274+0000 7f2ec9200700 1 -- 172.21.15.12:0/1950301106 learned_addr learned my addr 172.21.15.12:0/1950301106 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.274+0000 7f2ec3400700 1 --2- 172.21.15.12:0/1950301106 >> v2:172.21.15.159:3300/0 conn(0x7f2ec41613e0 0x7f2ec4161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.275+0000 7f2ec3e00700 1 -- 172.21.15.12:0/1950301106 >> v2:172.21.15.159:3300/0 conn(0x7f2ec41613e0 msgr2=0x7f2ec4161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.275+0000 7f2ec3e00700 1 --2- 172.21.15.12:0/1950301106 >> v2:172.21.15.159:3300/0 conn(0x7f2ec41613e0 0x7f2ec4161890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.275+0000 7f2ec3e00700 1 -- 172.21.15.12:0/1950301106 >> v1:172.21.15.12:6789/0 conn(0x7f2ec40b3420 legacy=0x7f2ec40b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.275+0000 7f2ec3e00700 1 -- 172.21.15.12:0/1950301106 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec40b2070 con 0x7f2ec4157f90 2024-03-29T10:14:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.275+0000 7f2ec2a00700 1 -- 172.21.15.12:0/1950301106 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203613541 0 0) 0x7f2ebc001490 con 0x7f2ec40b3420 2024-03-29T10:14:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.275+0000 7f2ec3e00700 1 --2- 172.21.15.12:0/1950301106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec4157f90 0x7f2ec40b2b30 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f2ec40b4210 tx=0x7f2eb4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14622cc4214b5ed server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.275+0000 7f2ec2a00700 1 -- 172.21.15.12:0/1950301106 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4011830 con 0x7f2ec4157f90 2024-03-29T10:14:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.275+0000 7f2ec2a00700 1 -- 172.21.15.12:0/1950301106 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb4011990 con 0x7f2ec4157f90 2024-03-29T10:14:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.275+0000 7f2ec2a00700 1 -- 172.21.15.12:0/1950301106 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4011d20 con 0x7f2ec4157f90 2024-03-29T10:14:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.276+0000 7f2ecb200700 1 -- 172.21.15.12:0/1950301106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec4157f90 msgr2=0x7f2ec40b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.276+0000 7f2ecb200700 1 --2- 172.21.15.12:0/1950301106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec4157f90 0x7f2ec40b2b30 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f2ec40b4210 tx=0x7f2eb4009b20 comp rx=0 tx=0).stop 2024-03-29T10:14:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.276+0000 7f2ecb200700 1 -- 172.21.15.12:0/1950301106 shutdown_connections 2024-03-29T10:14:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.276+0000 7f2ecb200700 1 --2- 172.21.15.12:0/1950301106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec4157f90 0x7f2ec40b2b30 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.276+0000 7f2ecb200700 1 --2- 172.21.15.12:0/1950301106 >> v2:172.21.15.159:3300/0 conn(0x7f2ec41613e0 0x7f2ec4161890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.276+0000 7f2ecb200700 1 -- 172.21.15.12:0/1950301106 >> 172.21.15.12:0/1950301106 conn(0x7f2ec40b0d70 msgr2=0x7f2ec40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.276+0000 7f2ecb200700 1 -- 172.21.15.12:0/1950301106 shutdown_connections 2024-03-29T10:14:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.276+0000 7f2ecb200700 1 -- 172.21.15.12:0/1950301106 wait complete. 2024-03-29T10:14:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.278+0000 7f2ecb200700 1 Processor -- start 2024-03-29T10:14:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.278+0000 7f2ecb200700 1 -- start start 2024-03-29T10:14:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.279+0000 7f2ecb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec40b3420 0x7f2ec406c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.279+0000 7f2ecb200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ec4157f90 0x7f2ec41494d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.279+0000 7f2ecb200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ec41613e0 0x7f2ec40c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.279+0000 7f2ecb200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec406d520 con 0x7f2ec40b3420 2024-03-29T10:14:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.279+0000 7f2ecb200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec406d800 con 0x7f2ec4157f90 2024-03-29T10:14:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.279+0000 7f2ecb200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec406dae0 con 0x7f2ec41613e0 2024-03-29T10:14:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.279+0000 7f2ec3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec40b3420 0x7f2ec406c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.279+0000 7f2ec3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec40b3420 0x7f2ec406c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54426/0 (socket says 172.21.15.12:54426) 2024-03-29T10:14:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.279+0000 7f2ec3e00700 1 -- 172.21.15.12:0/2287604827 learned_addr learned my addr 172.21.15.12:0/2287604827 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.280+0000 7f2ec3400700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ec4157f90 0x7f2ec41494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.280+0000 7f2ec9200700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ec41613e0 0x7f2ec40c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.280+0000 7f2ec3e00700 1 -- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ec41613e0 msgr2=0x7f2ec40c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.280+0000 7f2ec3e00700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ec41613e0 0x7f2ec40c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.280+0000 7f2ec3e00700 1 -- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ec4157f90 msgr2=0x7f2ec41494d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.280+0000 7f2ec3e00700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ec4157f90 0x7f2ec41494d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.280+0000 7f2ec3e00700 1 -- 172.21.15.12:0/2287604827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb40094a0 con 0x7f2ec40b3420 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.280+0000 7f2ec3400700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ec4157f90 0x7f2ec41494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.280+0000 7f2ec3e00700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec40b3420 0x7f2ec406c940 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4014040 tx=0x7f2eb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.281+0000 7f2ebbe00700 1 -- 172.21.15.12:0/2287604827 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb40036a0 con 0x7f2ec40b3420 2024-03-29T10:14:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.281+0000 7f2ebbe00700 1 -- 172.21.15.12:0/2287604827 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb401e070 con 0x7f2ec40b3420 2024-03-29T10:14:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.281+0000 7f2ecb200700 1 -- 172.21.15.12:0/2287604827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec406ddc0 con 0x7f2ec40b3420 2024-03-29T10:14:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.281+0000 7f2ebbe00700 1 -- 172.21.15.12:0/2287604827 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4003a20 con 0x7f2ec40b3420 2024-03-29T10:14:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.281+0000 7f2ecb200700 1 -- 172.21.15.12:0/2287604827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec4149a10 con 0x7f2ec40b3420 2024-03-29T10:14:05.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.282+0000 7f2ecb200700 1 -- 172.21.15.12:0/2287604827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec4067720 con 0x7f2ec40b3420 2024-03-29T10:14:05.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.283+0000 7f2ebbe00700 1 -- 172.21.15.12:0/2287604827 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f2eb4003b80 con 0x7f2ec40b3420 2024-03-29T10:14:05.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.283+0000 7f2ebbe00700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ea406aed0 0x7f2ea406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.284+0000 7f2ebbe00700 1 -- 172.21.15.12:0/2287604827 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f2eb408a730 con 0x7f2ec40b3420 2024-03-29T10:14:05.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.284+0000 7f2ec3400700 1 -- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ea406aed0 msgr2=0x7f2ea406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:05.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.284+0000 7f2ec3400700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ea406aed0 0x7f2ea406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:05.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.288+0000 7f2ebbe00700 1 -- 172.21.15.12:0/2287604827 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2eb4055440 con 0x7f2ec40b3420 2024-03-29T10:14:05.337 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:05.338+0000 7fd4361a1240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:14:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.485+0000 7f2ec3400700 1 -- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ea406aed0 msgr2=0x7f2ea406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.485+0000 7f2ec3400700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ea406aed0 0x7f2ea406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.501+0000 7f2ecb200700 1 -- 172.21.15.12:0/2287604827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2ec4071040 con 0x7f2ec40b3420 2024-03-29T10:14:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.502+0000 7f2ebbe00700 1 -- 172.21.15.12:0/2287604827 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v125) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2eb4058a60 con 0x7f2ec40b3420 2024-03-29T10:14:05.501 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:14:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 -- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ea406aed0 msgr2=0x7f2ea406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ea406aed0 0x7f2ea406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 -- 172.21.15.12:0/2287604827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec40b3420 msgr2=0x7f2ec406c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec40b3420 0x7f2ec406c940 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4014040 tx=0x7f2eb4004750 comp rx=0 tx=0).stop 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 -- 172.21.15.12:0/2287604827 shutdown_connections 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ea406aed0 0x7f2ea406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ec41613e0 0x7f2ec40c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec40b3420 0x7f2ec406c940 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 --2- 172.21.15.12:0/2287604827 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ec4157f90 0x7f2ec41494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 -- 172.21.15.12:0/2287604827 >> 172.21.15.12:0/2287604827 conn(0x7f2ec40b0d70 msgr2=0x7f2ec4157040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 -- 172.21.15.12:0/2287604827 shutdown_connections 2024-03-29T10:14:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.504+0000 7f2ecb200700 1 -- 172.21.15.12:0/2287604827 wait complete. 2024-03-29T10:14:05.522 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T10:14:05.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.723+0000 7f34dfa00700 1 Processor -- start 2024-03-29T10:14:05.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.723+0000 7f34dfa00700 1 -- start start 2024-03-29T10:14:05.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.724+0000 7f34dfa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f34d815ef70 0x7f34d80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.724+0000 7f34dfa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34d806a790 0x7f34d806ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.724+0000 7f34dfa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d8161150 con 0x7f34d8160d70 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.724+0000 7f34dfa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d806b180 con 0x7f34d815ef70 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.725+0000 7f34dfa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80b4cd0 con 0x7f34d806a790 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.725+0000 7f34d7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f34d815ef70 0x7f34d80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.725+0000 7f34d7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f34d815ef70 0x7f34d80b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54432/0 (socket says 172.21.15.12:54432) 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.725+0000 7f34d7e00700 1 -- 172.21.15.12:0/2050129799 learned_addr learned my addr 172.21.15.12:0/2050129799 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.725+0000 7f34dda00700 1 --2- 172.21.15.12:0/2050129799 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34d806a790 0x7f34d806ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.725+0000 7f34d7e00700 1 -- 172.21.15.12:0/2050129799 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34d806a790 msgr2=0x7f34d806ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.725+0000 7f34d7e00700 1 --2- 172.21.15.12:0/2050129799 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34d806a790 0x7f34d806ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.725+0000 7f34d7e00700 1 -- 172.21.15.12:0/2050129799 >> v1:172.21.15.159:6789/0 conn(0x7f34d8160d70 legacy=0x7f34d815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.725+0000 7f34d7e00700 1 -- 172.21.15.12:0/2050129799 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d806dba0 con 0x7f34d815ef70 2024-03-29T10:14:05.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.726+0000 7f34d7e00700 1 --2- 172.21.15.12:0/2050129799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d815ef70 0x7f34d80b3200 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f34c40098f0 tx=0x7f34c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d765ec026b3eeda8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.726+0000 7f34d7400700 1 -- 172.21.15.12:0/2050129799 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4011830 con 0x7f34d815ef70 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.726+0000 7f34d7400700 1 -- 172.21.15.12:0/2050129799 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c4011990 con 0x7f34d815ef70 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.726+0000 7f34d7400700 1 -- 172.21.15.12:0/2050129799 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4011d20 con 0x7f34d815ef70 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.727+0000 7f34dfa00700 1 -- 172.21.15.12:0/2050129799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d815ef70 msgr2=0x7f34d80b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.727+0000 7f34dfa00700 1 --2- 172.21.15.12:0/2050129799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d815ef70 0x7f34d80b3200 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f34c40098f0 tx=0x7f34c4009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.727+0000 7f34dfa00700 1 -- 172.21.15.12:0/2050129799 shutdown_connections 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.727+0000 7f34dfa00700 1 --2- 172.21.15.12:0/2050129799 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34d806a790 0x7f34d806ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.727+0000 7f34dfa00700 1 --2- 172.21.15.12:0/2050129799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d815ef70 0x7f34d80b3200 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.727+0000 7f34dfa00700 1 -- 172.21.15.12:0/2050129799 >> 172.21.15.12:0/2050129799 conn(0x7f34d80b5720 msgr2=0x7f34d80b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.727+0000 7f34dfa00700 1 -- 172.21.15.12:0/2050129799 shutdown_connections 2024-03-29T10:14:05.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.727+0000 7f34dfa00700 1 -- 172.21.15.12:0/2050129799 wait complete. 2024-03-29T10:14:05.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.729+0000 7f34dfa00700 1 Processor -- start 2024-03-29T10:14:05.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.729+0000 7f34dfa00700 1 -- start start 2024-03-29T10:14:05.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dfa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d806a790 0x7f34d8157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dfa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34d815ef70 0x7f34d80afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dfa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34d8160d70 0x7f34d80b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dfa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8157850 con 0x7f34d806a790 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dfa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8156c40 con 0x7f34d815ef70 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dfa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8156f20 con 0x7f34d8160d70 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d806a790 0x7f34d8157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d806a790 0x7f34d8157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54438/0 (socket says 172.21.15.12:54438) 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dd000700 1 -- 172.21.15.12:0/3777381835 learned_addr learned my addr 172.21.15.12:0/3777381835 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34d7e00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34d815ef70 0x7f34d80afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dda00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34d8160d70 0x7f34d80b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dd000700 1 -- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34d8160d70 msgr2=0x7f34d80b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dd000700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34d8160d70 0x7f34d80b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dd000700 1 -- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34d815ef70 msgr2=0x7f34d80afc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dd000700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34d815ef70 0x7f34d80afc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.730+0000 7f34dd000700 1 -- 172.21.15.12:0/3777381835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c4009550 con 0x7f34d806a790 2024-03-29T10:14:05.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.731+0000 7f34dd000700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d806a790 0x7f34d8157ce0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00c450 tx=0x7f34cc00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:05.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.731+0000 7f34d4c00700 1 -- 172.21.15.12:0/3777381835 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc0052d0 con 0x7f34d806a790 2024-03-29T10:14:05.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.731+0000 7f34d4c00700 1 -- 172.21.15.12:0/3777381835 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc005430 con 0x7f34d806a790 2024-03-29T10:14:05.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.731+0000 7f34d4c00700 1 -- 172.21.15.12:0/3777381835 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc0038c0 con 0x7f34d806a790 2024-03-29T10:14:05.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.731+0000 7f34dfa00700 1 -- 172.21.15.12:0/3777381835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d8157200 con 0x7f34d806a790 2024-03-29T10:14:05.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.731+0000 7f34dfa00700 1 -- 172.21.15.12:0/3777381835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d8156880 con 0x7f34d806a790 2024-03-29T10:14:05.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.732+0000 7f34dfa00700 1 -- 172.21.15.12:0/3777381835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d8067720 con 0x7f34d806a790 2024-03-29T10:14:05.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.733+0000 7f34d4c00700 1 -- 172.21.15.12:0/3777381835 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f34cc003a20 con 0x7f34d806a790 2024-03-29T10:14:05.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.734+0000 7f34d4c00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f34b806aed0 0x7f34b806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:05.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.734+0000 7f34d4c00700 1 -- 172.21.15.12:0/3777381835 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f34cc088b90 con 0x7f34d806a790 2024-03-29T10:14:05.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.734+0000 7f34d7e00700 1 -- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f34b806aed0 msgr2=0x7f34b806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:05.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.734+0000 7f34d7e00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f34b806aed0 0x7f34b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:05.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.739+0000 7f34d4c00700 1 -- 172.21.15.12:0/3777381835 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f34cc0538f0 con 0x7f34d806a790 2024-03-29T10:14:05.915 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:05.915+0000 7f209ba96240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:14:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.935+0000 7f34d7e00700 1 -- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f34b806aed0 msgr2=0x7f34b806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.935+0000 7f34d7e00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f34b806aed0 0x7f34b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:05.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.971+0000 7f34dfa00700 1 -- 172.21.15.12:0/3777381835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f34d809c530 con 0x7f34d806a790 2024-03-29T10:14:05.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.971+0000 7f34d4c00700 1 -- 172.21.15.12:0/3777381835 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v125) v1 ==== 80+0+466 (secure 0 0 0) 0x7f34cc017020 con 0x7f34d806a790 2024-03-29T10:14:05.970 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1231831807 2024-03-30T10:13:04.000570+0000 2024-03-29T10:14:05.970 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3573895366 2024-03-30T10:13:04.000570+0000 2024-03-29T10:14:05.971 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/43444 2024-03-30T10:13:04.000570+0000 2024-03-29T10:14:05.971 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/43444 2024-03-30T10:13:04.000570+0000 2024-03-29T10:14:05.971 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2082234733 2024-03-30T10:13:04.000570+0000 2024-03-29T10:14:05.971 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3500442951 2024-03-30T10:13:04.000570+0000 2024-03-29T10:14:05.971 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/949098073 2024-03-30T10:13:04.000570+0000 2024-03-29T10:14:05.971 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/497911868 2024-03-30T10:13:04.000570+0000 2024-03-29T10:14:05.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 -- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f34b806aed0 msgr2=0x7f34b806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:05.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f34b806aed0 0x7f34b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 -- 172.21.15.12:0/3777381835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d806a790 msgr2=0x7f34d8157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:05.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d806a790 0x7f34d8157ce0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00c450 tx=0x7f34cc00c760 comp rx=0 tx=0).stop 2024-03-29T10:14:05.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 -- 172.21.15.12:0/3777381835 shutdown_connections 2024-03-29T10:14:05.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f34b806aed0 0x7f34b806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34d8160d70 0x7f34d80b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d806a790 0x7f34d8157ce0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 --2- 172.21.15.12:0/3777381835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34d815ef70 0x7f34d80afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:05.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 -- 172.21.15.12:0/3777381835 >> 172.21.15.12:0/3777381835 conn(0x7f34d80b5720 msgr2=0x7f34d806df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:05.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 -- 172.21.15.12:0/3777381835 shutdown_connections 2024-03-29T10:14:05.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:05.974+0000 7f34dfa00700 1 -- 172.21.15.12:0/3777381835 wait complete. 2024-03-29T10:14:05.974 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T10:14:05.992 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T10:14:05.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:05.992 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T10:14:06.006 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:06.007+0000 7f209ba96240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:14:06.076 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T10:14:06.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:06.076 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T10:14:06.113 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:06.114+0000 7fd4361a1240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:14:06.126 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:14:06.126 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:06.170 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:14:06.170 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:14:06.171 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:06.174 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:06.175+0000 7f209ba96240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:14:06.212 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:06.213+0000 7fd4361a1240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:14:06.216 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:14:06.216 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:14:06.216 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:06.261 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:14:06.262 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:06.389 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:06.390+0000 7fd4361a1240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:14:06.396 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:06.396+0000 7f209ba96240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:14:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.466+0000 7f2e1cc00700 1 Processor -- start 2024-03-29T10:14:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.466+0000 7f2e1cc00700 1 -- start start 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.467+0000 7f2e1cc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2e180b3af0 0x7f2e1815a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.467+0000 7f2e1cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18072580 0x7f2e180b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.467+0000 7f2e1cc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2e180b2e50 0x7f2e180709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.467+0000 7f2e1cc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e180b2180 con 0x7f2e180b3af0 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.467+0000 7f2e1cc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1815b100 con 0x7f2e180b2e50 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.467+0000 7f2e1cc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1815b3e0 con 0x7f2e18072580 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e15e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2e180b3af0 0x7f2e1815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e15e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2e180b3af0 0x7f2e1815a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54452/0 (socket says 172.21.15.12:54452) 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e15e00700 1 -- 172.21.15.12:0/2399699450 learned_addr learned my addr 172.21.15.12:0/2399699450 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e16800700 1 --2- 172.21.15.12:0/2399699450 >> v2:172.21.15.159:3300/0 conn(0x7f2e180b2e50 0x7f2e180709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e15400700 1 --2- 172.21.15.12:0/2399699450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18072580 0x7f2e180b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e15e00700 1 -- 172.21.15.12:0/2399699450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18072580 msgr2=0x7f2e180b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:06.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e15e00700 1 --2- 172.21.15.12:0/2399699450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18072580 0x7f2e180b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e15e00700 1 -- 172.21.15.12:0/2399699450 >> v2:172.21.15.159:3300/0 conn(0x7f2e180b2e50 msgr2=0x7f2e180709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:06.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e15e00700 1 --2- 172.21.15.12:0/2399699450 >> v2:172.21.15.159:3300/0 conn(0x7f2e180b2e50 0x7f2e180709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e15e00700 1 -- 172.21.15.12:0/2399699450 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e180b23f0 con 0x7f2e180b3af0 2024-03-29T10:14:06.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.468+0000 7f2e15e00700 1 --2- 172.21.15.12:0/2399699450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e180b3af0 0x7f2e1815a860 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f2e1815dd90 tx=0x7f2e08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8d7783fb0fd551 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:06.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.469+0000 7f2e14a00700 1 -- 172.21.15.12:0/2399699450 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08011830 con 0x7f2e180b3af0 2024-03-29T10:14:06.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.469+0000 7f2e14a00700 1 -- 172.21.15.12:0/2399699450 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e08011990 con 0x7f2e180b3af0 2024-03-29T10:14:06.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.469+0000 7f2e14a00700 1 -- 172.21.15.12:0/2399699450 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08011d20 con 0x7f2e180b3af0 2024-03-29T10:14:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.469+0000 7f2e1cc00700 1 -- 172.21.15.12:0/2399699450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e180b3af0 msgr2=0x7f2e1815a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.469+0000 7f2e1cc00700 1 --2- 172.21.15.12:0/2399699450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e180b3af0 0x7f2e1815a860 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f2e1815dd90 tx=0x7f2e08009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.469+0000 7f2e1cc00700 1 -- 172.21.15.12:0/2399699450 shutdown_connections 2024-03-29T10:14:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.469+0000 7f2e1cc00700 1 --2- 172.21.15.12:0/2399699450 >> v2:172.21.15.159:3300/0 conn(0x7f2e180b2e50 0x7f2e180709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.469+0000 7f2e1cc00700 1 --2- 172.21.15.12:0/2399699450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e180b3af0 0x7f2e1815a860 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.469+0000 7f2e1cc00700 1 --2- 172.21.15.12:0/2399699450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18072580 0x7f2e180b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.469+0000 7f2e1cc00700 1 -- 172.21.15.12:0/2399699450 >> 172.21.15.12:0/2399699450 conn(0x7f2e180cf190 msgr2=0x7f2e180cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.470+0000 7f2e1cc00700 1 -- 172.21.15.12:0/2399699450 shutdown_connections 2024-03-29T10:14:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.470+0000 7f2e1cc00700 1 -- 172.21.15.12:0/2399699450 wait complete. 2024-03-29T10:14:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.471+0000 7f2e1cc00700 1 Processor -- start 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.471+0000 7f2e1cc00700 1 -- start start 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.472+0000 7f2e1cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e18072580 0x7f2e18149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.472+0000 7f2e1cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e180b2e50 0x7f2e18147740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.472+0000 7f2e1cc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e180b3af0 0x7f2e181641a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.472+0000 7f2e1cc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18163d10 con 0x7f2e18072580 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.472+0000 7f2e1cc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18149320 con 0x7f2e180b3af0 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.472+0000 7f2e1cc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1815a630 con 0x7f2e180b2e50 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e15e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e18072580 0x7f2e18149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e15e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e18072580 0x7f2e18149e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54466/0 (socket says 172.21.15.12:54466) 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e15e00700 1 -- 172.21.15.12:0/1239706632 learned_addr learned my addr 172.21.15.12:0/1239706632 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e15400700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e180b2e50 0x7f2e18147740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e16800700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e180b3af0 0x7f2e181641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e15e00700 1 -- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e180b2e50 msgr2=0x7f2e18147740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e15e00700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e180b2e50 0x7f2e18147740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e15e00700 1 -- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e180b3af0 msgr2=0x7f2e181641a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e15e00700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e180b3af0 0x7f2e181641a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e15e00700 1 -- 172.21.15.12:0/1239706632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e08009550 con 0x7f2e18072580 2024-03-29T10:14:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.473+0000 7f2e15e00700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e18072580 0x7f2e18149e10 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f2e08000c00 tx=0x7f2e08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.474+0000 7f2e0e000700 1 -- 172.21.15.12:0/1239706632 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08004240 con 0x7f2e18072580 2024-03-29T10:14:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.474+0000 7f2e0e000700 1 -- 172.21.15.12:0/1239706632 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0801f070 con 0x7f2e18072580 2024-03-29T10:14:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.474+0000 7f2e0e000700 1 -- 172.21.15.12:0/1239706632 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e080045e0 con 0x7f2e18072580 2024-03-29T10:14:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.474+0000 7f2e1cc00700 1 -- 172.21.15.12:0/1239706632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e18165d80 con 0x7f2e18072580 2024-03-29T10:14:06.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.474+0000 7f2e1cc00700 1 -- 172.21.15.12:0/1239706632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e18166340 con 0x7f2e18072580 2024-03-29T10:14:06.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.475+0000 7f2e1cc00700 1 -- 172.21.15.12:0/1239706632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e18067720 con 0x7f2e18072580 2024-03-29T10:14:06.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.476+0000 7f2e0e000700 1 -- 172.21.15.12:0/1239706632 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f2e08021030 con 0x7f2e18072580 2024-03-29T10:14:06.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.477+0000 7f2e0e000700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2df806aed0 0x7f2df806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.477+0000 7f2e0e000700 1 -- 172.21.15.12:0/1239706632 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f2e0808a950 con 0x7f2e18072580 2024-03-29T10:14:06.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.477+0000 7f2e15400700 1 -- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2df806aed0 msgr2=0x7f2df806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:06.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.477+0000 7f2e15400700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2df806aed0 0x7f2df806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:06.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.481+0000 7f2e0e000700 1 -- 172.21.15.12:0/1239706632 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e08055820 con 0x7f2e18072580 2024-03-29T10:14:06.487 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:06.488+0000 7f209ba96240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:14:06.585 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:06.586+0000 7f209ba96240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:14:06.613 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:06.614+0000 7fd4361a1240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:14:06.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.678+0000 7f2e15400700 1 -- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2df806aed0 msgr2=0x7f2df806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:06.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.678+0000 7f2e15400700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2df806aed0 0x7f2df806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:06.682 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:06.683+0000 7f209ba96240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:14:06.710 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:06.711+0000 7fd4361a1240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:14:06.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.740+0000 7f2e1cc00700 1 -- 172.21.15.12:0/1239706632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e180cb980 con 0x7f2e18072580 2024-03-29T10:14:06.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.741+0000 7f2e0e000700 1 -- 172.21.15.12:0/1239706632 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f2e08015020 con 0x7f2e18072580 2024-03-29T10:14:06.740 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:06.740 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":55,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7645,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/553963486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":553963486},{"type":"v1","addr":"172.21.15.159:6837","nonce":553963486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3051067707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3051067707},{"type":"v1","addr":"172.21.15.159:6836","nonce":3051067707}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3629410472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3629410472},{"type":"v1","addr":"172.21.15.12:6833","nonce":3629410472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:12:46.810806+0000","modified":"2024-03-29T10:12:50.294107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7917},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7917":{"gid":7917,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/753821395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":753821395},{"type":"v1","addr":"172.21.15.12:6835","nonce":753821395}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-29T10:14:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 -- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2df806aed0 msgr2=0x7f2df806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2df806aed0 0x7f2df806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 -- 172.21.15.12:0/1239706632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e18072580 msgr2=0x7f2e18149e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e18072580 0x7f2e18149e10 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f2e08000c00 tx=0x7f2e08004750 comp rx=0 tx=0).stop 2024-03-29T10:14:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 -- 172.21.15.12:0/1239706632 shutdown_connections 2024-03-29T10:14:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2df806aed0 0x7f2df806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e180b3af0 0x7f2e181641a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e18072580 0x7f2e18149e10 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 --2- 172.21.15.12:0/1239706632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e180b2e50 0x7f2e18147740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 -- 172.21.15.12:0/1239706632 >> 172.21.15.12:0/1239706632 conn(0x7f2e180cf190 msgr2=0x7f2e180709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:06.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 -- 172.21.15.12:0/1239706632 shutdown_connections 2024-03-29T10:14:06.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.744+0000 7f2e1cc00700 1 -- 172.21.15.12:0/1239706632 wait complete. 2024-03-29T10:14:06.744 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 55 2024-03-29T10:14:06.762 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:06.778 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:06.779+0000 7f209ba96240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:14:06.823 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:06.823+0000 7fd4361a1240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:14:06.933 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:06.934+0000 7fd4361a1240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:14:06.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.972+0000 7f598ba00700 1 Processor -- start 2024-03-29T10:14:06.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.972+0000 7f598ba00700 1 -- start start 2024-03-29T10:14:06.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.973+0000 7f598ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 0x7f59840ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.973+0000 7f598ba00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f59840d0cf0 0x7f59840b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.973+0000 7f598ba00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59840d24e0 con 0x7f59840cec70 2024-03-29T10:14:06.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.973+0000 7f598ba00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59840b95d0 con 0x7f59840d0cf0 2024-03-29T10:14:06.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.973+0000 7f598ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598415de30 con 0x7f598415da50 2024-03-29T10:14:06.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.973+0000 7f5989a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f59840d0cf0 0x7f59840b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.973+0000 7f5989000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 0x7f59840ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.974+0000 7f5989a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f59840d0cf0 0x7f59840b6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:33150/0 (socket says 172.21.15.12:33150) 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.974+0000 7f5989a00700 1 -- 172.21.15.12:0/1711842228 learned_addr learned my addr 172.21.15.12:0/1711842228 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.974+0000 7f5989000700 1 -- 172.21.15.12:0/1711842228 >> v2:172.21.15.159:3300/0 conn(0x7f59840d0cf0 msgr2=0x7f59840b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.974+0000 7f5989000700 1 --2- 172.21.15.12:0/1711842228 >> v2:172.21.15.159:3300/0 conn(0x7f59840d0cf0 0x7f59840b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.974+0000 7f5989000700 1 -- 172.21.15.12:0/1711842228 >> v1:172.21.15.159:6789/0 conn(0x7f59840cec70 legacy=0x7f59840d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.974+0000 7f5989000700 1 -- 172.21.15.12:0/1711842228 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840d27c0 con 0x7f598415da50 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.974+0000 7f5989a00700 1 --2- 172.21.15.12:0/1711842228 >> v2:172.21.15.159:3300/0 conn(0x7f59840d0cf0 0x7f59840b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.974+0000 7f5983400700 1 -- 172.21.15.12:0/1711842228 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048495244 0 0) 0x7f59840d24e0 con 0x7f59840cec70 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.974+0000 7f5989000700 1 --2- 172.21.15.12:0/1711842228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 0x7f59840ce730 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5978009840 tx=0x7f5978009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a03174f39d2d83a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f5983400700 1 -- 172.21.15.12:0/1711842228 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978011830 con 0x7f598415da50 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f5983400700 1 -- 172.21.15.12:0/1711842228 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5978011990 con 0x7f598415da50 2024-03-29T10:14:06.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f5983400700 1 -- 172.21.15.12:0/1711842228 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978011d20 con 0x7f598415da50 2024-03-29T10:14:06.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f598ba00700 1 -- 172.21.15.12:0/1711842228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 msgr2=0x7f59840ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:06.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f598ba00700 1 --2- 172.21.15.12:0/1711842228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 0x7f59840ce730 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5978009840 tx=0x7f5978009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:06.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f598ba00700 1 -- 172.21.15.12:0/1711842228 shutdown_connections 2024-03-29T10:14:06.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f598ba00700 1 --2- 172.21.15.12:0/1711842228 >> v2:172.21.15.159:3300/0 conn(0x7f59840d0cf0 0x7f59840b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f598ba00700 1 --2- 172.21.15.12:0/1711842228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 0x7f59840ce730 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f598ba00700 1 -- 172.21.15.12:0/1711842228 >> 172.21.15.12:0/1711842228 conn(0x7f59840666a0 msgr2=0x7f5984066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:06.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f598ba00700 1 -- 172.21.15.12:0/1711842228 shutdown_connections 2024-03-29T10:14:06.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.975+0000 7f598ba00700 1 -- 172.21.15.12:0/1711842228 wait complete. 2024-03-29T10:14:06.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.977+0000 7f598ba00700 1 Processor -- start 2024-03-29T10:14:06.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.977+0000 7f598ba00700 1 -- start start 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.978+0000 7f598ba00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59840cec70 0x7f598414bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.978+0000 7f598ba00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59840d0cf0 0x7f59840b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.978+0000 7f598ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 0x7f59840b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.978+0000 7f598ba00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840cc270 con 0x7f59840cec70 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.978+0000 7f598ba00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840cb660 con 0x7f59840d0cf0 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.978+0000 7f598ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59840cb940 con 0x7f598415da50 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.978+0000 7f5989000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59840cec70 0x7f598414bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.978+0000 7f5989000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59840cec70 0x7f598414bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54474/0 (socket says 172.21.15.12:54474) 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.978+0000 7f5989000700 1 -- 172.21.15.12:0/2031017620 learned_addr learned my addr 172.21.15.12:0/2031017620 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5989a00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 0x7f59840b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5983e00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59840d0cf0 0x7f59840b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:06.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5989000700 1 -- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 msgr2=0x7f59840b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:06.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5989000700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 0x7f59840b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5989000700 1 -- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59840d0cf0 msgr2=0x7f59840b52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:06.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5989000700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59840d0cf0 0x7f59840b52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:06.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5989000700 1 -- 172.21.15.12:0/2031017620 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597c009b30 con 0x7f59840cec70 2024-03-29T10:14:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5989000700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59840cec70 0x7f598414bd80 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5978014040 tx=0x7f5978004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5980c00700 1 -- 172.21.15.12:0/2031017620 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59780036a0 con 0x7f59840cec70 2024-03-29T10:14:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5980c00700 1 -- 172.21.15.12:0/2031017620 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597801e070 con 0x7f59840cec70 2024-03-29T10:14:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.979+0000 7f5980c00700 1 -- 172.21.15.12:0/2031017620 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978003a40 con 0x7f59840cec70 2024-03-29T10:14:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.980+0000 7f598ba00700 1 -- 172.21.15.12:0/2031017620 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59780094a0 con 0x7f59840cec70 2024-03-29T10:14:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.980+0000 7f598ba00700 1 -- 172.21.15.12:0/2031017620 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59840cbf00 con 0x7f59840cec70 2024-03-29T10:14:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.981+0000 7f598ba00700 1 -- 172.21.15.12:0/2031017620 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5984067720 con 0x7f59840cec70 2024-03-29T10:14:06.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.982+0000 7f5980c00700 1 -- 172.21.15.12:0/2031017620 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f5978003ba0 con 0x7f59840cec70 2024-03-29T10:14:06.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.982+0000 7f5980c00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f596406aed0 0x7f596406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:06.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.983+0000 7f5980c00700 1 -- 172.21.15.12:0/2031017620 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f597808a900 con 0x7f59840cec70 2024-03-29T10:14:06.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.983+0000 7f5983e00700 1 -- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f596406aed0 msgr2=0x7f596406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:06.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.983+0000 7f5983e00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f596406aed0 0x7f596406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:06.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:06.987+0000 7f5980c00700 1 -- 172.21.15.12:0/2031017620 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5978055590 con 0x7f59840cec70 2024-03-29T10:14:06.992 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:06.993+0000 7f209ba96240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:14:07.041 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:07.041+0000 7fd4361a1240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:14:07.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.184+0000 7f5983e00700 1 -- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f596406aed0 msgr2=0x7f596406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:07.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.184+0000 7f5983e00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f596406aed0 0x7f596406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.205+0000 7f598ba00700 1 -- 172.21.15.12:0/2031017620 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f598415ba60 con 0x7f59840cec70 2024-03-29T10:14:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.206+0000 7f5980c00700 1 -- 172.21.15.12:0/2031017620 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16548 (secure 0 0 0) 0x7f5978070e60 con 0x7f59840cec70 2024-03-29T10:14:07.206 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:07.206 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":125,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:13:04.000616+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:12:45.171490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-29T10:12:45.787701+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1231831807":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/3573895366":"2024-03-30T10:13:04.000570+0000","172.21.15.159:6833/43444":"2024-03-30T10:13:04.000570+0000","172.21.15.159:6832/43444":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/2082234733":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/3500442951":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/949098073":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/497911868":"2024-03-30T10:13:04.000570+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 -- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f596406aed0 msgr2=0x7f596406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f596406aed0 0x7f596406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 -- 172.21.15.12:0/2031017620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59840cec70 msgr2=0x7f598414bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59840cec70 0x7f598414bd80 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5978014040 tx=0x7f5978004750 comp rx=0 tx=0).stop 2024-03-29T10:14:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 -- 172.21.15.12:0/2031017620 shutdown_connections 2024-03-29T10:14:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f596406aed0 0x7f596406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f598415da50 0x7f59840b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59840cec70 0x7f598414bd80 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 --2- 172.21.15.12:0/2031017620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59840d0cf0 0x7f59840b52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 -- 172.21.15.12:0/2031017620 >> 172.21.15.12:0/2031017620 conn(0x7f59840666a0 msgr2=0x7f59840d3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:07.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 -- 172.21.15.12:0/2031017620 shutdown_connections 2024-03-29T10:14:07.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.209+0000 7f598ba00700 1 -- 172.21.15.12:0/2031017620 wait complete. 2024-03-29T10:14:07.227 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:12:45.171490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '123', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:12:45.787701+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:14:07.228 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:14:07.254 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:07.255+0000 7f209ba96240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:14:07.271 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:07.272+0000 7fd4361a1240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:14:07.447 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:07.447+0000 7f209ba96240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:14:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.449+0000 7f87a6400700 1 Processor -- start 2024-03-29T10:14:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.449+0000 7f87a6400700 1 -- start start 2024-03-29T10:14:07.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.450+0000 7f87a6400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f87a00cd020 0x7f87a00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.450+0000 7f87a6400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 0x7f87a00b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.450+0000 7f87a6400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a00cdae0 con 0x7f87a00cbb30 2024-03-29T10:14:07.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.450+0000 7f87a6400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a00cddc0 con 0x7f87a00cd020 2024-03-29T10:14:07.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.450+0000 7f87a6400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a00ce0a0 con 0x7f87a00b2070 2024-03-29T10:14:07.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.451+0000 7f879ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f87a00cd020 0x7f87a00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:07.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.451+0000 7f879ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f87a00cd020 0x7f87a00cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:33170/0 (socket says 172.21.15.12:33170) 2024-03-29T10:14:07.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.451+0000 7f879fe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 0x7f87a00b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:07.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.451+0000 7f879ea00700 1 -- 172.21.15.12:0/1659811207 learned_addr learned my addr 172.21.15.12:0/1659811207 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.451+0000 7f879f400700 1 --1- 172.21.15.12:0/1659811207 >> v1:172.21.15.159:6789/0 conn(0x7f87a00cbb30 0x7f87a00cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:42360/0 (socket says 172.21.15.12:42360) 2024-03-29T10:14:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.451+0000 7f879ea00700 1 -- 172.21.15.12:0/1659811207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 msgr2=0x7f87a00b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.451+0000 7f879ea00700 1 --2- 172.21.15.12:0/1659811207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 0x7f87a00b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.451+0000 7f879ea00700 1 -- 172.21.15.12:0/1659811207 >> v1:172.21.15.159:6789/0 conn(0x7f87a00cbb30 legacy=0x7f87a00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.451+0000 7f879ea00700 1 -- 172.21.15.12:0/1659811207 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a006dc20 con 0x7f87a00cd020 2024-03-29T10:14:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.451+0000 7f879fe00700 1 --2- 172.21.15.12:0/1659811207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 0x7f87a00b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.452+0000 7f879ea00700 1 --2- 172.21.15.12:0/1659811207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a00cd020 0x7f87a00cd480 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f878c0098f0 tx=0x7f878c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ab44a7e3a3d0395 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.452+0000 7f879e000700 1 -- 172.21.15.12:0/1659811207 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c011830 con 0x7f87a00cd020 2024-03-29T10:14:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.452+0000 7f879e000700 1 -- 172.21.15.12:0/1659811207 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878c011990 con 0x7f87a00cd020 2024-03-29T10:14:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.452+0000 7f879e000700 1 -- 172.21.15.12:0/1659811207 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c011d20 con 0x7f87a00cd020 2024-03-29T10:14:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.452+0000 7f87a6400700 1 -- 172.21.15.12:0/1659811207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a00cd020 msgr2=0x7f87a00cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.452+0000 7f87a6400700 1 --2- 172.21.15.12:0/1659811207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a00cd020 0x7f87a00cd480 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f878c0098f0 tx=0x7f878c009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.452+0000 7f87a6400700 1 -- 172.21.15.12:0/1659811207 shutdown_connections 2024-03-29T10:14:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.452+0000 7f87a6400700 1 --2- 172.21.15.12:0/1659811207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 0x7f87a00b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.452+0000 7f87a6400700 1 --2- 172.21.15.12:0/1659811207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a00cd020 0x7f87a00cd480 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.452+0000 7f87a6400700 1 -- 172.21.15.12:0/1659811207 >> 172.21.15.12:0/1659811207 conn(0x7f87a00b0d70 msgr2=0x7f87a00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.453+0000 7f87a6400700 1 -- 172.21.15.12:0/1659811207 shutdown_connections 2024-03-29T10:14:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.453+0000 7f87a6400700 1 -- 172.21.15.12:0/1659811207 wait complete. 2024-03-29T10:14:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.454+0000 7f87a6400700 1 Processor -- start 2024-03-29T10:14:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.455+0000 7f87a6400700 1 -- start start 2024-03-29T10:14:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.455+0000 7f87a6400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 0x7f87a00cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.455+0000 7f87a6400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00cbb30 0x7f87a014b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.455+0000 7f87a6400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a00cd020 0x7f87a00c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f87a6400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a009a4f0 con 0x7f87a00cbb30 2024-03-29T10:14:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f87a6400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a014b970 con 0x7f87a00cd020 2024-03-29T10:14:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f87a6400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a00c9630 con 0x7f87a00b2070 2024-03-29T10:14:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f879ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00cbb30 0x7f87a014b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f879f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 0x7f87a00cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f879ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00cbb30 0x7f87a014b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54476/0 (socket says 172.21.15.12:54476) 2024-03-29T10:14:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f879ea00700 1 -- 172.21.15.12:0/2774782454 learned_addr learned my addr 172.21.15.12:0/2774782454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:07.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f879fe00700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a00cd020 0x7f87a00c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f879f400700 1 -- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a00cd020 msgr2=0x7f87a00c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f879f400700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a00cd020 0x7f87a00c8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f879f400700 1 -- 172.21.15.12:0/2774782454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00cbb30 msgr2=0x7f87a014b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f879f400700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00cbb30 0x7f87a014b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.456+0000 7f879f400700 1 -- 172.21.15.12:0/2774782454 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8798009a70 con 0x7f87a00b2070 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.457+0000 7f879f400700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 0x7f87a00cb340 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f879000c450 tx=0x7f879000c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.457+0000 7f8797400700 1 -- 172.21.15.12:0/2774782454 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790005cb0 con 0x7f87a00b2070 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.457+0000 7f8797400700 1 -- 172.21.15.12:0/2774782454 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8790005e10 con 0x7f87a00b2070 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.457+0000 7f8797400700 1 -- 172.21.15.12:0/2774782454 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87900072d0 con 0x7f87a00b2070 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.457+0000 7f87a6400700 1 -- 172.21.15.12:0/2774782454 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c009550 con 0x7f87a00b2070 2024-03-29T10:14:07.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.457+0000 7f87a6400700 1 -- 172.21.15.12:0/2774782454 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a00c9bf0 con 0x7f87a00b2070 2024-03-29T10:14:07.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.458+0000 7f87a6400700 1 -- 172.21.15.12:0/2774782454 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a009a350 con 0x7f87a00b2070 2024-03-29T10:14:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.460+0000 7f8797400700 1 -- 172.21.15.12:0/2774782454 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f8790007430 con 0x7f87a00b2070 2024-03-29T10:14:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.462+0000 7f8797400700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006af00 0x7f878006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.462+0000 7f8797400700 1 -- 172.21.15.12:0/2774782454 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f8790088b10 con 0x7f87a00b2070 2024-03-29T10:14:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.462+0000 7f879ea00700 1 -- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006af00 msgr2=0x7f878006d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.463+0000 7f879ea00700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006af00 0x7f878006d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.469+0000 7f8797400700 1 -- 172.21.15.12:0/2774782454 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f87a009a350 con 0x7f87a00b2070 2024-03-29T10:14:07.544 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:07.544+0000 7f209ba96240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:07.550 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:07.551+0000 7fd4361a1240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:14:07.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.664+0000 7f879ea00700 1 -- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006af00 msgr2=0x7f878006d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:07.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.664+0000 7f879ea00700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006af00 0x7f878006d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.682+0000 7f87a6400700 1 -- 172.21.15.12:0/2774782454 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87a009a350 con 0x7f87a00b2070 2024-03-29T10:14:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.682+0000 7f8797400700 1 -- 172.21.15.12:0/2774782454 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7f87a009a350 con 0x7f87a00b2070 2024-03-29T10:14:07.682 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:14:07.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 -- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006af00 msgr2=0x7f878006d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006af00 0x7f878006d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 -- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 msgr2=0x7f87a00cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 0x7f87a00cb340 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f879000c450 tx=0x7f879000c760 comp rx=0 tx=0).stop 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 -- 172.21.15.12:0/2774782454 shutdown_connections 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f878006af00 0x7f878006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87a00cd020 0x7f87a00c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f87a00b2070 0x7f87a00cb340 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 --2- 172.21.15.12:0/2774782454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87a00cbb30 0x7f87a014b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 -- 172.21.15.12:0/2774782454 >> 172.21.15.12:0/2774782454 conn(0x7f87a00b0d70 msgr2=0x7f87a00ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 -- 172.21.15.12:0/2774782454 shutdown_connections 2024-03-29T10:14:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.685+0000 7f87a6400700 1 -- 172.21.15.12:0/2774782454 wait complete. 2024-03-29T10:14:07.703 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:14:07.754 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:07.755+0000 7fd4361a1240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:14:07.859 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:07.860+0000 7fd4361a1240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.952+0000 7f6783600700 1 Processor -- start 2024-03-29T10:14:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.953+0000 7f6783600700 1 -- start start 2024-03-29T10:14:07.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.954+0000 7f6783600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f677c15dad0 0x7f677c0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.954+0000 7f6783600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c0d21d0 0x7f677c15fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.954+0000 7f6783600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677c15f820 con 0x7f677c0d1630 2024-03-29T10:14:07.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.954+0000 7f6783600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677c0cfcc0 con 0x7f677c15dad0 2024-03-29T10:14:07.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.954+0000 7f6783600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677c0cd9f0 con 0x7f677c0d21d0 2024-03-29T10:14:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.954+0000 7f677be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f677c0d1630 0x7f677c0d1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43290/0 (socket says 172.21.15.12:43290) 2024-03-29T10:14:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.954+0000 7f677be00700 1 -- 172.21.15.12:0/1641134778 learned_addr learned my addr 172.21.15.12:0/1641134778 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.954+0000 7f6780c00700 1 --2- 172.21.15.12:0/1641134778 >> v2:172.21.15.159:3300/0 conn(0x7f677c15dad0 0x7f677c0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.954+0000 7f6781600700 1 --2- 172.21.15.12:0/1641134778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c0d21d0 0x7f677c15fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:07.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.955+0000 7f677b400700 1 -- 172.21.15.12:0/1641134778 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332119554 0 0) 0x7f677c15f820 con 0x7f677c0d1630 2024-03-29T10:14:07.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.955+0000 7f6780c00700 1 -- 172.21.15.12:0/1641134778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c0d21d0 msgr2=0x7f677c15fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:07.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.955+0000 7f6780c00700 1 --2- 172.21.15.12:0/1641134778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c0d21d0 0x7f677c15fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.955+0000 7f6780c00700 1 -- 172.21.15.12:0/1641134778 >> v1:172.21.15.12:6789/0 conn(0x7f677c0d1630 legacy=0x7f677c0d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:07.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.955+0000 7f6780c00700 1 -- 172.21.15.12:0/1641134778 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c0cf420 con 0x7f677c15dad0 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.955+0000 7f6780c00700 1 --2- 172.21.15.12:0/1641134778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f677c15dad0 0x7f677c0ced80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6770009840 tx=0x7f6770009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30c4b66f4238800e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.955+0000 7f677b400700 1 -- 172.21.15.12:0/1641134778 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770011830 con 0x7f677c15dad0 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.955+0000 7f677b400700 1 -- 172.21.15.12:0/1641134778 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6770011990 con 0x7f677c15dad0 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.955+0000 7f677b400700 1 -- 172.21.15.12:0/1641134778 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770011d20 con 0x7f677c15dad0 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.956+0000 7f6783600700 1 -- 172.21.15.12:0/1641134778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f677c15dad0 msgr2=0x7f677c0ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.956+0000 7f6783600700 1 --2- 172.21.15.12:0/1641134778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f677c15dad0 0x7f677c0ced80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6770009840 tx=0x7f6770009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.956+0000 7f6783600700 1 -- 172.21.15.12:0/1641134778 shutdown_connections 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.956+0000 7f6783600700 1 --2- 172.21.15.12:0/1641134778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c0d21d0 0x7f677c15fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.956+0000 7f6783600700 1 --2- 172.21.15.12:0/1641134778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f677c15dad0 0x7f677c0ced80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.956+0000 7f6783600700 1 -- 172.21.15.12:0/1641134778 >> 172.21.15.12:0/1641134778 conn(0x7f677c08aff0 msgr2=0x7f677c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.956+0000 7f6783600700 1 -- 172.21.15.12:0/1641134778 shutdown_connections 2024-03-29T10:14:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.956+0000 7f6783600700 1 -- 172.21.15.12:0/1641134778 wait complete. 2024-03-29T10:14:07.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.958+0000 7f6783600700 1 Processor -- start 2024-03-29T10:14:07.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.958+0000 7f6783600700 1 -- start start 2024-03-29T10:14:07.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f6783600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f677c0d1630 0x7f677c1f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f6783600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f677c0d21d0 0x7f677c141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f6783600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c15dad0 0x7f677c06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f6783600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c141f70 con 0x7f677c0d1630 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f6783600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c0b8e20 con 0x7f677c0d21d0 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f6783600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677c0b9100 con 0x7f677c15dad0 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f6780c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f677c0d1630 0x7f677c1f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f6780c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f677c0d1630 0x7f677c1f7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54484/0 (socket says 172.21.15.12:54484) 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f6780c00700 1 -- 172.21.15.12:0/922018595 learned_addr learned my addr 172.21.15.12:0/922018595 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f677be00700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f677c0d21d0 0x7f677c141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.959+0000 7f6781600700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c15dad0 0x7f677c06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6780c00700 1 -- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c15dad0 msgr2=0x7f677c06ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6780c00700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c15dad0 0x7f677c06ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6780c00700 1 -- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f677c0d21d0 msgr2=0x7f677c141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6780c00700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f677c0d21d0 0x7f677c141a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6780c00700 1 -- 172.21.15.12:0/922018595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67700094a0 con 0x7f677c0d1630 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6781600700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c15dad0 0x7f677c06ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:07.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6780c00700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f677c0d1630 0x7f677c1f7740 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f6770009fd0 tx=0x7f6770004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:07.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6778c00700 1 -- 172.21.15.12:0/922018595 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67700045d0 con 0x7f677c0d1630 2024-03-29T10:14:07.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6778c00700 1 -- 172.21.15.12:0/922018595 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677001e070 con 0x7f677c0d1630 2024-03-29T10:14:07.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6778c00700 1 -- 172.21.15.12:0/922018595 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67700038e0 con 0x7f677c0d1630 2024-03-29T10:14:07.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6783600700 1 -- 172.21.15.12:0/922018595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677c1f6c60 con 0x7f677c0d1630 2024-03-29T10:14:07.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.960+0000 7f6783600700 1 -- 172.21.15.12:0/922018595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677c1f7220 con 0x7f677c0d1630 2024-03-29T10:14:07.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.962+0000 7f6783600700 1 -- 172.21.15.12:0/922018595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677c1f7410 con 0x7f677c0d1630 2024-03-29T10:14:07.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.962+0000 7f6778c00700 1 -- 172.21.15.12:0/922018595 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f6770003a40 con 0x7f677c0d1630 2024-03-29T10:14:07.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.963+0000 7f6778c00700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f675c06aed0 0x7f675c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:07.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.963+0000 7f6778c00700 1 -- 172.21.15.12:0/922018595 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f677008a880 con 0x7f677c0d1630 2024-03-29T10:14:07.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.963+0000 7f677be00700 1 -- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f675c06aed0 msgr2=0x7f675c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:07.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.963+0000 7f677be00700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f675c06aed0 0x7f675c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:07.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:07.967+0000 7f6778c00700 1 -- 172.21.15.12:0/922018595 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6770055560 con 0x7f677c0d1630 2024-03-29T10:14:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.164+0000 7f677be00700 1 -- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f675c06aed0 msgr2=0x7f675c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.164+0000 7f677be00700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f675c06aed0 0x7f675c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:08.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.197+0000 7f6783600700 1 -- 172.21.15.12:0/922018595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f677c067720 con 0x7f677c0d1630 2024-03-29T10:14:08.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.197+0000 7f6778c00700 1 -- 172.21.15.12:0/922018595 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v126) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6770058b80 con 0x7f677c0d1630 2024-03-29T10:14:08.196 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:14:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.199+0000 7f6783600700 1 -- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f675c06aed0 msgr2=0x7f675c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.199+0000 7f6783600700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f675c06aed0 0x7f675c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.199+0000 7f6783600700 1 -- 172.21.15.12:0/922018595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f677c0d1630 msgr2=0x7f677c1f7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.199+0000 7f6783600700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f677c0d1630 0x7f677c1f7740 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f6770009fd0 tx=0x7f6770004750 comp rx=0 tx=0).stop 2024-03-29T10:14:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.200+0000 7f6783600700 1 -- 172.21.15.12:0/922018595 shutdown_connections 2024-03-29T10:14:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.200+0000 7f6783600700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f675c06aed0 0x7f675c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.200+0000 7f6783600700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f677c15dad0 0x7f677c06ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.200+0000 7f6783600700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f677c0d1630 0x7f677c1f7740 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.200+0000 7f6783600700 1 --2- 172.21.15.12:0/922018595 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f677c0d21d0 0x7f677c141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.200+0000 7f6783600700 1 -- 172.21.15.12:0/922018595 >> 172.21.15.12:0/922018595 conn(0x7f677c08aff0 msgr2=0x7f677c0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.200+0000 7f6783600700 1 -- 172.21.15.12:0/922018595 shutdown_connections 2024-03-29T10:14:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.200+0000 7f6783600700 1 -- 172.21.15.12:0/922018595 wait complete. 2024-03-29T10:14:08.219 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:14:08.274 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:08.275+0000 7f209ba96240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:14:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.464+0000 7f7575e00700 1 Processor -- start 2024-03-29T10:14:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.464+0000 7f7575e00700 1 -- start start 2024-03-29T10:14:08.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.465+0000 7f7575e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75701611b0 0x7f757015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.465+0000 7f7575e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f757015f4d0 0x7f75700ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.465+0000 7f7575e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75700ce270 con 0x7f75700d21f0 2024-03-29T10:14:08.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.465+0000 7f7575e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757015b600 con 0x7f757015f4d0 2024-03-29T10:14:08.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.465+0000 7f7575e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75700d2760 con 0x7f75701611b0 2024-03-29T10:14:08.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.465+0000 7f756e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f757015f4d0 0x7f75700ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:08.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.465+0000 7f756f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f75700d21f0 0x7f7570161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43298/0 (socket says 172.21.15.12:43298) 2024-03-29T10:14:08.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.465+0000 7f756f400700 1 -- 172.21.15.12:0/4186239050 learned_addr learned my addr 172.21.15.12:0/4186239050 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:08.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.466+0000 7f756ea00700 1 --2- 172.21.15.12:0/4186239050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75701611b0 0x7f757015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:08.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.466+0000 7f756e000700 1 -- 172.21.15.12:0/4186239050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75701611b0 msgr2=0x7f757015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:08.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.466+0000 7f756e000700 1 --2- 172.21.15.12:0/4186239050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75701611b0 0x7f757015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.466+0000 7f756e000700 1 -- 172.21.15.12:0/4186239050 >> v1:172.21.15.12:6789/0 conn(0x7f75700d21f0 legacy=0x7f7570161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:08.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.466+0000 7f756e000700 1 -- 172.21.15.12:0/4186239050 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75700d2a40 con 0x7f757015f4d0 2024-03-29T10:14:08.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.466+0000 7f756d600700 1 -- 172.21.15.12:0/4186239050 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229735470 0 0) 0x7f75700ce270 con 0x7f75700d21f0 2024-03-29T10:14:08.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.466+0000 7f756e000700 1 --2- 172.21.15.12:0/4186239050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 0x7f75700ce700 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f755c009840 tx=0x7f755c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5ff981466e3153 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f756d600700 1 -- 172.21.15.12:0/4186239050 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c011830 con 0x7f757015f4d0 2024-03-29T10:14:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f756d600700 1 -- 172.21.15.12:0/4186239050 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755c011990 con 0x7f757015f4d0 2024-03-29T10:14:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f756d600700 1 -- 172.21.15.12:0/4186239050 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c011d20 con 0x7f757015f4d0 2024-03-29T10:14:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f7575e00700 1 -- 172.21.15.12:0/4186239050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 msgr2=0x7f75700ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f7575e00700 1 --2- 172.21.15.12:0/4186239050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 0x7f75700ce700 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f755c009840 tx=0x7f755c009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:08.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f7575e00700 1 -- 172.21.15.12:0/4186239050 shutdown_connections 2024-03-29T10:14:08.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f7575e00700 1 --2- 172.21.15.12:0/4186239050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75701611b0 0x7f757015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f7575e00700 1 --2- 172.21.15.12:0/4186239050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 0x7f75700ce700 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f7575e00700 1 -- 172.21.15.12:0/4186239050 >> 172.21.15.12:0/4186239050 conn(0x7f757008aff0 msgr2=0x7f75700d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:08.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f7575e00700 1 -- 172.21.15.12:0/4186239050 shutdown_connections 2024-03-29T10:14:08.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.467+0000 7f7575e00700 1 -- 172.21.15.12:0/4186239050 wait complete. 2024-03-29T10:14:08.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.469+0000 7f7575e00700 1 Processor -- start 2024-03-29T10:14:08.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.469+0000 7f7575e00700 1 -- start start 2024-03-29T10:14:08.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.470+0000 7f7575e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75700d21f0 0x7f75700cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.470+0000 7f7575e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 0x7f75701587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.470+0000 7f7575e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75701611b0 0x7f7570156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.470+0000 7f7575e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75700cb960 con 0x7f757015f4d0 2024-03-29T10:14:08.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.470+0000 7f7575e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75700b62d0 con 0x7f75700d21f0 2024-03-29T10:14:08.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.470+0000 7f7575e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75700b65b0 con 0x7f75701611b0 2024-03-29T10:14:08.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.470+0000 7f756e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 0x7f75701587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:08.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.470+0000 7f756e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 0x7f75701587c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54500/0 (socket says 172.21.15.12:54500) 2024-03-29T10:14:08.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.470+0000 7f756e000700 1 -- 172.21.15.12:0/3922593588 learned_addr learned my addr 172.21.15.12:0/3922593588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:08.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.471+0000 7f756ea00700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75700d21f0 0x7f75700cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:08.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.471+0000 7f756f400700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75701611b0 0x7f7570156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:08.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.471+0000 7f756e000700 1 -- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75701611b0 msgr2=0x7f7570156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:08.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.471+0000 7f756e000700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75701611b0 0x7f7570156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.471+0000 7f756e000700 1 -- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75700d21f0 msgr2=0x7f75700cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:08.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.471+0000 7f756e000700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75700d21f0 0x7f75700cb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.471+0000 7f756e000700 1 -- 172.21.15.12:0/3922593588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c0094a0 con 0x7f757015f4d0 2024-03-29T10:14:08.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.471+0000 7f756e000700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 0x7f75701587c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f755c0090b0 tx=0x7f755c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:08.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.472+0000 7f7567400700 1 -- 172.21.15.12:0/3922593588 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c004550 con 0x7f757015f4d0 2024-03-29T10:14:08.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.472+0000 7f7567400700 1 -- 172.21.15.12:0/3922593588 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755c01f070 con 0x7f757015f4d0 2024-03-29T10:14:08.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.472+0000 7f7567400700 1 -- 172.21.15.12:0/3922593588 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c0038e0 con 0x7f757015f4d0 2024-03-29T10:14:08.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.472+0000 7f7575e00700 1 -- 172.21.15.12:0/3922593588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75700b6890 con 0x7f757015f4d0 2024-03-29T10:14:08.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.472+0000 7f7575e00700 1 -- 172.21.15.12:0/3922593588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75700b6e50 con 0x7f757015f4d0 2024-03-29T10:14:08.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.473+0000 7f7575e00700 1 -- 172.21.15.12:0/3922593588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7570067720 con 0x7f757015f4d0 2024-03-29T10:14:08.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.474+0000 7f7567400700 1 -- 172.21.15.12:0/3922593588 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f755c003a40 con 0x7f757015f4d0 2024-03-29T10:14:08.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.474+0000 7f7567400700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f755006aed0 0x7f755006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.475+0000 7f7567400700 1 -- 172.21.15.12:0/3922593588 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f755c021070 con 0x7f757015f4d0 2024-03-29T10:14:08.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.475+0000 7f756ea00700 1 -- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f755006aed0 msgr2=0x7f755006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:08.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.475+0000 7f756ea00700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f755006aed0 0x7f755006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:08.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.484+0000 7f7567400700 1 -- 172.21.15.12:0/3922593588 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f755c055d90 con 0x7f757015f4d0 2024-03-29T10:14:08.487 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:08.488+0000 7f209ba96240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:14:08.663 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:08.664+0000 7fd4361a1240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:14:08.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.676+0000 7f756ea00700 1 -- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f755006aed0 msgr2=0x7f755006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:08.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.676+0000 7f756ea00700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f755006aed0 0x7f755006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:08.692 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:08.693+0000 7f209ba96240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:08.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.699+0000 7f7575e00700 1 -- 172.21.15.12:0/3922593588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f75700b5d00 con 0x7f757015f4d0 2024-03-29T10:14:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.700+0000 7f7567400700 1 -- 172.21.15.12:0/3922593588 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v126) v1 ==== 100+0+11 (secure 0 0 0) 0x7f755c02b0a0 con 0x7f757015f4d0 2024-03-29T10:14:08.699 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:14:08.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.702+0000 7f7575e00700 1 -- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f755006aed0 msgr2=0x7f755006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:08.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.702+0000 7f7575e00700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f755006aed0 0x7f755006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.702+0000 7f7575e00700 1 -- 172.21.15.12:0/3922593588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 msgr2=0x7f75701587c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:08.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.702+0000 7f7575e00700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 0x7f75701587c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f755c0090b0 tx=0x7f755c004750 comp rx=0 tx=0).stop 2024-03-29T10:14:08.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.703+0000 7f7575e00700 1 -- 172.21.15.12:0/3922593588 shutdown_connections 2024-03-29T10:14:08.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.703+0000 7f7575e00700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f755006aed0 0x7f755006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.703+0000 7f7575e00700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75701611b0 0x7f7570156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.703+0000 7f7575e00700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75700d21f0 0x7f75700cb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.703+0000 7f7575e00700 1 --2- 172.21.15.12:0/3922593588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f757015f4d0 0x7f75701587c0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.703+0000 7f7575e00700 1 -- 172.21.15.12:0/3922593588 >> 172.21.15.12:0/3922593588 conn(0x7f757008aff0 msgr2=0x7f757006c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.703+0000 7f7575e00700 1 -- 172.21.15.12:0/3922593588 shutdown_connections 2024-03-29T10:14:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.703+0000 7f7575e00700 1 -- 172.21.15.12:0/3922593588 wait complete. 2024-03-29T10:14:08.720 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:08.895 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:08.896+0000 7fd4361a1240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:14:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.922+0000 7f76eea00700 1 Processor -- start 2024-03-29T10:14:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.922+0000 7f76eea00700 1 -- start start 2024-03-29T10:14:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.923+0000 7f76eea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f76e8160c90 0x7f76e80b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.923+0000 7f76eea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f76e80ad970 0x7f76e80b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.923+0000 7f76eea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e80ac4d0 con 0x7f76e80718e0 2024-03-29T10:14:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.923+0000 7f76eea00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e8071e50 con 0x7f76e80ad970 2024-03-29T10:14:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.923+0000 7f76eea00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e8072130 con 0x7f76e8160c90 2024-03-29T10:14:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.923+0000 7f76e7400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f76e80718e0 0x7f76e806cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42566/0 (socket says 172.21.15.12:42566) 2024-03-29T10:14:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.923+0000 7f76e7400700 1 -- 172.21.15.12:0/247052853 learned_addr learned my addr 172.21.15.12:0/247052853 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.923+0000 7f76eca00700 1 --2- 172.21.15.12:0/247052853 >> v2:172.21.15.12:3300/0 conn(0x7f76e80ad970 0x7f76e80b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.923+0000 7f76e7e00700 1 --2- 172.21.15.12:0/247052853 >> v2:172.21.15.159:3300/0 conn(0x7f76e8160c90 0x7f76e80b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.924+0000 7f76e6a00700 1 -- 172.21.15.12:0/247052853 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365789819 0 0) 0x7f76e80ac4d0 con 0x7f76e80718e0 2024-03-29T10:14:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.924+0000 7f76e6a00700 1 -- 172.21.15.12:0/247052853 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc0035b0 con 0x7f76e80718e0 2024-03-29T10:14:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.924+0000 7f76eca00700 1 -- 172.21.15.12:0/247052853 >> v2:172.21.15.159:3300/0 conn(0x7f76e8160c90 msgr2=0x7f76e80b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.924+0000 7f76eca00700 1 --2- 172.21.15.12:0/247052853 >> v2:172.21.15.159:3300/0 conn(0x7f76e8160c90 0x7f76e80b20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.924+0000 7f76eca00700 1 -- 172.21.15.12:0/247052853 >> v1:172.21.15.12:6789/0 conn(0x7f76e80718e0 legacy=0x7f76e806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.924+0000 7f76eca00700 1 -- 172.21.15.12:0/247052853 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e8072410 con 0x7f76e80ad970 2024-03-29T10:14:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.924+0000 7f76e7e00700 1 --2- 172.21.15.12:0/247052853 >> v2:172.21.15.159:3300/0 conn(0x7f76e8160c90 0x7f76e80b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.924+0000 7f76eca00700 1 --2- 172.21.15.12:0/247052853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80ad970 0x7f76e80b2620 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f76e8070090 tx=0x7f76e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29a26d96274004c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.924+0000 7f76e6a00700 1 -- 172.21.15.12:0/247052853 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e000e070 con 0x7f76e80ad970 2024-03-29T10:14:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.924+0000 7f76e6a00700 1 -- 172.21.15.12:0/247052853 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e000bb20 con 0x7f76e80ad970 2024-03-29T10:14:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.925+0000 7f76e6a00700 1 -- 172.21.15.12:0/247052853 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0012430 con 0x7f76e80ad970 2024-03-29T10:14:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.925+0000 7f76eea00700 1 -- 172.21.15.12:0/247052853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80ad970 msgr2=0x7f76e80b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.925+0000 7f76eea00700 1 --2- 172.21.15.12:0/247052853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80ad970 0x7f76e80b2620 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f76e8070090 tx=0x7f76e000b330 comp rx=0 tx=0).stop 2024-03-29T10:14:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.925+0000 7f76eea00700 1 -- 172.21.15.12:0/247052853 shutdown_connections 2024-03-29T10:14:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.925+0000 7f76eea00700 1 --2- 172.21.15.12:0/247052853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80ad970 0x7f76e80b2620 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.925+0000 7f76eea00700 1 --2- 172.21.15.12:0/247052853 >> v2:172.21.15.159:3300/0 conn(0x7f76e8160c90 0x7f76e80b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.925+0000 7f76eea00700 1 -- 172.21.15.12:0/247052853 >> 172.21.15.12:0/247052853 conn(0x7f76e80cf190 msgr2=0x7f76e80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.925+0000 7f76eea00700 1 -- 172.21.15.12:0/247052853 shutdown_connections 2024-03-29T10:14:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.925+0000 7f76eea00700 1 -- 172.21.15.12:0/247052853 wait complete. 2024-03-29T10:14:08.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.927+0000 7f76eea00700 1 Processor -- start 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.927+0000 7f76eea00700 1 -- start start 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.928+0000 7f76eea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80718e0 0x7f76e8152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.928+0000 7f76eea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76e80ad970 0x7f76e8156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.928+0000 7f76eea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76e8160c90 0x7f76e8154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.928+0000 7f76eea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e8157630 con 0x7f76e80718e0 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.928+0000 7f76eea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e8152dc0 con 0x7f76e8160c90 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.928+0000 7f76eea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e8157390 con 0x7f76e80ad970 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.928+0000 7f76e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80718e0 0x7f76e8152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.928+0000 7f76e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80718e0 0x7f76e8152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44510/0 (socket says 172.21.15.12:44510) 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.928+0000 7f76e7e00700 1 -- 172.21.15.12:0/2172808228 learned_addr learned my addr 172.21.15.12:0/2172808228 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76e7400700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76e80ad970 0x7f76e8156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76eca00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76e8160c90 0x7f76e8154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:08.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76e7e00700 1 -- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76e80ad970 msgr2=0x7f76e8156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:08.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76e7e00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76e80ad970 0x7f76e8156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76e7e00700 1 -- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76e8160c90 msgr2=0x7f76e8154950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:08.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76e7e00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76e8160c90 0x7f76e8154950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:08.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76e7e00700 1 -- 172.21.15.12:0/2172808228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d80094a0 con 0x7f76e80718e0 2024-03-29T10:14:08.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76e7e00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80718e0 0x7f76e8152880 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f76d800ebe0 tx=0x7f76d800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:08.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76dfe00700 1 -- 172.21.15.12:0/2172808228 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d8009760 con 0x7f76e80718e0 2024-03-29T10:14:08.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76dfe00700 1 -- 172.21.15.12:0/2172808228 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d800c850 con 0x7f76e80718e0 2024-03-29T10:14:08.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76dfe00700 1 -- 172.21.15.12:0/2172808228 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d800cbe0 con 0x7f76e80718e0 2024-03-29T10:14:08.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.929+0000 7f76eea00700 1 -- 172.21.15.12:0/2172808228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e0009a70 con 0x7f76e80718e0 2024-03-29T10:14:08.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.930+0000 7f76eea00700 1 -- 172.21.15.12:0/2172808228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e815e910 con 0x7f76e80718e0 2024-03-29T10:14:08.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.931+0000 7f76eea00700 1 -- 172.21.15.12:0/2172808228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e8067720 con 0x7f76e80718e0 2024-03-29T10:14:08.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.931+0000 7f76dfe00700 1 -- 172.21.15.12:0/2172808228 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f76d800cd40 con 0x7f76e80718e0 2024-03-29T10:14:08.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.932+0000 7f76dfe00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76c806aed0 0x7f76c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:08.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.932+0000 7f76dfe00700 1 -- 172.21.15.12:0/2172808228 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f76d8089980 con 0x7f76e80718e0 2024-03-29T10:14:08.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.932+0000 7f76e7400700 1 -- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76c806aed0 msgr2=0x7f76c806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:08.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.932+0000 7f76e7400700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76c806aed0 0x7f76c806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:08.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:08.941+0000 7f76dfe00700 1 -- 172.21.15.12:0/2172808228 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76d80545d0 con 0x7f76e80718e0 2024-03-29T10:14:09.123 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:09.124+0000 7fd4361a1240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.134+0000 7f76e7400700 1 -- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76c806aed0 msgr2=0x7f76c806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.134+0000 7f76e7400700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76c806aed0 0x7f76c806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:09.166 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:09.167+0000 7f209ba96240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:14:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.203+0000 7f76eea00700 1 -- 172.21.15.12:0/2172808228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76e80b3130 con 0x7f76e80718e0 2024-03-29T10:14:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.204+0000 7f76dfe00700 1 -- 172.21.15.12:0/2172808228 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f76d8057bf0 con 0x7f76e80718e0 2024-03-29T10:14:09.204 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:09.204 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":55,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7645,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/553963486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":553963486},{"type":"v1","addr":"172.21.15.159:6837","nonce":553963486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3051067707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3051067707},{"type":"v1","addr":"172.21.15.159:6836","nonce":3051067707}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3629410472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3629410472},{"type":"v1","addr":"172.21.15.12:6833","nonce":3629410472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:12:46.810806+0000","modified":"2024-03-29T10:12:50.294107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7917},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7917":{"gid":7917,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/753821395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":753821395},{"type":"v1","addr":"172.21.15.12:6835","nonce":753821395}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-29T10:14:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 -- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76c806aed0 msgr2=0x7f76c806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76c806aed0 0x7f76c806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 -- 172.21.15.12:0/2172808228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80718e0 msgr2=0x7f76e8152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80718e0 0x7f76e8152880 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f76d800ebe0 tx=0x7f76d800eef0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 -- 172.21.15.12:0/2172808228 shutdown_connections 2024-03-29T10:14:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76c806aed0 0x7f76c806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76e8160c90 0x7f76e8154950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76e80718e0 0x7f76e8152880 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 --2- 172.21.15.12:0/2172808228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76e80ad970 0x7f76e8156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 -- 172.21.15.12:0/2172808228 >> 172.21.15.12:0/2172808228 conn(0x7f76e80cf190 msgr2=0x7f76e80b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 -- 172.21.15.12:0/2172808228 shutdown_connections 2024-03-29T10:14:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.207+0000 7f76eea00700 1 -- 172.21.15.12:0/2172808228 wait complete. 2024-03-29T10:14:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 55 2024-03-29T10:14:09.225 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:09.375 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:09.375+0000 7f209ba96240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:09.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.427+0000 7fdcc0200700 1 Processor -- start 2024-03-29T10:14:09.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.427+0000 7fdcc0200700 1 -- start start 2024-03-29T10:14:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.428+0000 7fdcc0200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcb80d05e0 0x7fdcb80d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.428+0000 7fdcc0200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb80d0f80 con 0x7fdcb80b6990 2024-03-29T10:14:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.428+0000 7fdcc0200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb80d24e0 con 0x7fdcb815da50 2024-03-29T10:14:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.428+0000 7fdcc0200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb80b95d0 con 0x7fdcb80d05e0 2024-03-29T10:14:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.428+0000 7fdcbce00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcb80d05e0 0x7fdcb80d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.429+0000 7fdcbce00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcb80d05e0 0x7fdcb80d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44524/0 (socket says 172.21.15.12:44524) 2024-03-29T10:14:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.429+0000 7fdcbe200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdcb80b6990 0x7fdcb80b6e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42578/0 (socket says 172.21.15.12:42578) 2024-03-29T10:14:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.429+0000 7fdcbce00700 1 -- 172.21.15.12:0/3590644080 learned_addr learned my addr 172.21.15.12:0/3590644080 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.429+0000 7fdcbce00700 1 -- 172.21.15.12:0/3590644080 >> v1:172.21.15.159:6789/0 conn(0x7fdcb815da50 legacy=0x7fdcb80ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.429+0000 7fdcbce00700 1 -- 172.21.15.12:0/3590644080 >> v1:172.21.15.12:6789/0 conn(0x7fdcb80b6990 legacy=0x7fdcb80b6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.429+0000 7fdcbce00700 1 -- 172.21.15.12:0/3590644080 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb80d27c0 con 0x7fdcb80d05e0 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.429+0000 7fdcbce00700 1 --2- 172.21.15.12:0/3590644080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcb80d05e0 0x7fdcb80d0a40 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fdca40098f0 tx=0x7fdca4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed3d55e6f7059e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.429+0000 7fdcb7e00700 1 -- 172.21.15.12:0/3590644080 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca400b8a0 con 0x7fdcb80d05e0 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.430+0000 7fdcb7e00700 1 -- 172.21.15.12:0/3590644080 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca400ba00 con 0x7fdcb80d05e0 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.430+0000 7fdcb7e00700 1 -- 172.21.15.12:0/3590644080 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca400bd70 con 0x7fdcb80d05e0 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.430+0000 7fdcc0200700 1 -- 172.21.15.12:0/3590644080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcb80d05e0 msgr2=0x7fdcb80d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.430+0000 7fdcc0200700 1 --2- 172.21.15.12:0/3590644080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcb80d05e0 0x7fdcb80d0a40 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fdca40098f0 tx=0x7fdca4009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.430+0000 7fdcc0200700 1 -- 172.21.15.12:0/3590644080 shutdown_connections 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.430+0000 7fdcc0200700 1 --2- 172.21.15.12:0/3590644080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcb80d05e0 0x7fdcb80d0a40 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.430+0000 7fdcc0200700 1 -- 172.21.15.12:0/3590644080 >> 172.21.15.12:0/3590644080 conn(0x7fdcb80666a0 msgr2=0x7fdcb8066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.430+0000 7fdcc0200700 1 -- 172.21.15.12:0/3590644080 shutdown_connections 2024-03-29T10:14:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.430+0000 7fdcc0200700 1 -- 172.21.15.12:0/3590644080 wait complete. 2024-03-29T10:14:09.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.432+0000 7fdcc0200700 1 Processor -- start 2024-03-29T10:14:09.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.432+0000 7fdcc0200700 1 -- start start 2024-03-29T10:14:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcc0200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcb80b6990 0x7fdcb8160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcc0200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcb80d05e0 0x7fdcb815dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcc0200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcb815da50 0x7fdcb80b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcc0200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb8160e90 con 0x7fdcb815da50 2024-03-29T10:14:09.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcc0200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb8144f20 con 0x7fdcb80b6990 2024-03-29T10:14:09.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcc0200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb815e710 con 0x7fdcb80d05e0 2024-03-29T10:14:09.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcbd800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcb80b6990 0x7fdcb8160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:09.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcbe200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcb815da50 0x7fdcb80b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:09.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcbce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcb80d05e0 0x7fdcb815dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:09.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcbd800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcb80b6990 0x7fdcb8160950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:57578/0 (socket says 172.21.15.12:57578) 2024-03-29T10:14:09.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.433+0000 7fdcbd800700 1 -- 172.21.15.12:0/1383903144 learned_addr learned my addr 172.21.15.12:0/1383903144 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:09.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcbe200700 1 -- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcb80d05e0 msgr2=0x7fdcb815dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcbe200700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcb80d05e0 0x7fdcb815dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcbe200700 1 -- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcb80b6990 msgr2=0x7fdcb8160950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcbe200700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcb80b6990 0x7fdcb8160950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcbe200700 1 -- 172.21.15.12:0/1383903144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca4009550 con 0x7fdcb815da50 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcbd800700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcb80b6990 0x7fdcb8160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcbe200700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcb815da50 0x7fdcb80b4310 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000caa0 tx=0x7fdcb000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcb5600700 1 -- 172.21.15.12:0/1383903144 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb000a9e0 con 0x7fdcb815da50 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcb5600700 1 -- 172.21.15.12:0/1383903144 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb000ab40 con 0x7fdcb815da50 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcb5600700 1 -- 172.21.15.12:0/1383903144 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0007070 con 0x7fdcb815da50 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.434+0000 7fdcc0200700 1 -- 172.21.15.12:0/1383903144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb815e9f0 con 0x7fdcb815da50 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.435+0000 7fdcc0200700 1 -- 172.21.15.12:0/1383903144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb815e400 con 0x7fdcb815da50 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.436+0000 7fdcc0200700 1 -- 172.21.15.12:0/1383903144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb815ebe0 con 0x7fdcb815da50 2024-03-29T10:14:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.436+0000 7fdcb5600700 1 -- 172.21.15.12:0/1383903144 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fdcb00071d0 con 0x7fdcb815da50 2024-03-29T10:14:09.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.437+0000 7fdcb5600700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc9806aed0 0x7fdc9806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:09.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.437+0000 7fdcb5600700 1 -- 172.21.15.12:0/1383903144 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6109+0+0 (secure 0 0 0) 0x7fdcb00892c0 con 0x7fdcb815da50 2024-03-29T10:14:09.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.438+0000 7fdcbd800700 1 -- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc9806aed0 msgr2=0x7fdc9806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:09.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.438+0000 7fdcbd800700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc9806aed0 0x7fdc9806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:09.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.442+0000 7fdcb5600700 1 -- 172.21.15.12:0/1383903144 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdcb0054050 con 0x7fdcb815da50 2024-03-29T10:14:09.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.639+0000 7fdcbd800700 1 -- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc9806aed0 msgr2=0x7fdc9806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:09.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.639+0000 7fdcbd800700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc9806aed0 0x7fdc9806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:09.638 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:09.639+0000 7fd4361a1240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:14:09.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.657+0000 7fdcc0200700 1 -- 172.21.15.12:0/1383903144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcb8067720 con 0x7fdcb815da50 2024-03-29T10:14:09.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.658+0000 7fdcb5600700 1 -- 172.21.15.12:0/1383903144 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16659 (secure 0 0 0) 0x7fdcb0017090 con 0x7fdcb815da50 2024-03-29T10:14:09.658 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:09.658 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":126,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:14:07.235303+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:12:45.171490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-29T10:12:45.787701+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":126,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1231831807":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/3573895366":"2024-03-30T10:13:04.000570+0000","172.21.15.159:6833/43444":"2024-03-30T10:13:04.000570+0000","172.21.15.159:6832/43444":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/2082234733":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/3500442951":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/949098073":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/497911868":"2024-03-30T10:13:04.000570+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:09.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 -- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc9806aed0 msgr2=0x7fdc9806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:09.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc9806aed0 0x7fdc9806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 -- 172.21.15.12:0/1383903144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcb815da50 msgr2=0x7fdcb80b4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcb815da50 0x7fdcb80b4310 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000caa0 tx=0x7fdcb000cdb0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 -- 172.21.15.12:0/1383903144 shutdown_connections 2024-03-29T10:14:09.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdc9806aed0 0x7fdc9806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcb815da50 0x7fdcb80b4310 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcb80b6990 0x7fdcb8160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 --2- 172.21.15.12:0/1383903144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcb80d05e0 0x7fdcb815dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 -- 172.21.15.12:0/1383903144 >> 172.21.15.12:0/1383903144 conn(0x7fdcb80666a0 msgr2=0x7fdcb815bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:09.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 -- 172.21.15.12:0/1383903144 shutdown_connections 2024-03-29T10:14:09.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.661+0000 7fdcc0200700 1 -- 172.21.15.12:0/1383903144 wait complete. 2024-03-29T10:14:09.679 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T10:14:09.679 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:09.819 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:09.819+0000 7f209ba96240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:14:09.862 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:09.863+0000 7fd4361a1240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:09.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.881+0000 7f0770e00700 1 Processor -- start 2024-03-29T10:14:09.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.881+0000 7f0770e00700 1 -- start start 2024-03-29T10:14:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.882+0000 7f0770e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f076c0ceec0 0x7f076c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.882+0000 7f0770e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c161c50 con 0x7f076c15d640 2024-03-29T10:14:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.882+0000 7f0770e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c0d30d0 con 0x7f076c161800 2024-03-29T10:14:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.882+0000 7f0770e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076c0b6840 con 0x7f076c0ceec0 2024-03-29T10:14:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.882+0000 7f0769400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f076c0ceec0 0x7f076c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:09.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.882+0000 7f0769e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f076c15d640 0x7f076c0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42586/0 (socket says 172.21.15.12:42586) 2024-03-29T10:14:09.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.882+0000 7f0769e00700 1 -- 172.21.15.12:0/2226176779 learned_addr learned my addr 172.21.15.12:0/2226176779 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:09.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.883+0000 7f0769400700 1 -- 172.21.15.12:0/2226176779 >> v1:172.21.15.159:6789/0 conn(0x7f076c161800 legacy=0x7f076c0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.883+0000 7f0769400700 1 -- 172.21.15.12:0/2226176779 >> v1:172.21.15.12:6789/0 conn(0x7f076c15d640 legacy=0x7f076c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.883+0000 7f0769400700 1 -- 172.21.15.12:0/2226176779 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c0d2b30 con 0x7f076c0ceec0 2024-03-29T10:14:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.883+0000 7f0769400700 1 --2- 172.21.15.12:0/2226176779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c0ceec0 0x7f076c1612c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f0758009840 tx=0x7f0758009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d7f7c34038059d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.883+0000 7f0768a00700 1 -- 172.21.15.12:0/2226176779 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0758011830 con 0x7f076c0ceec0 2024-03-29T10:14:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.883+0000 7f0768a00700 1 -- 172.21.15.12:0/2226176779 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0758011990 con 0x7f076c0ceec0 2024-03-29T10:14:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.883+0000 7f0768a00700 1 -- 172.21.15.12:0/2226176779 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0758011d20 con 0x7f076c0ceec0 2024-03-29T10:14:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.884+0000 7f0770e00700 1 -- 172.21.15.12:0/2226176779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c0ceec0 msgr2=0x7f076c1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.884+0000 7f0770e00700 1 --2- 172.21.15.12:0/2226176779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c0ceec0 0x7f076c1612c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f0758009840 tx=0x7f0758009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.884+0000 7f0770e00700 1 -- 172.21.15.12:0/2226176779 shutdown_connections 2024-03-29T10:14:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.884+0000 7f0770e00700 1 --2- 172.21.15.12:0/2226176779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c0ceec0 0x7f076c1612c0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.884+0000 7f0770e00700 1 -- 172.21.15.12:0/2226176779 >> 172.21.15.12:0/2226176779 conn(0x7f076c08aff0 msgr2=0x7f076c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:09.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.884+0000 7f0770e00700 1 -- 172.21.15.12:0/2226176779 shutdown_connections 2024-03-29T10:14:09.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.884+0000 7f0770e00700 1 -- 172.21.15.12:0/2226176779 wait complete. 2024-03-29T10:14:09.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.886+0000 7f0770e00700 1 Processor -- start 2024-03-29T10:14:09.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.886+0000 7f0770e00700 1 -- start start 2024-03-29T10:14:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f0770e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f076c0ceec0 0x7f076c1580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f0770e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c15d640 0x7f076c15bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f0770e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f076c161800 0x7f076c15c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f0770e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1f3100 con 0x7f076c15d640 2024-03-29T10:14:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f0770e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1586d0 con 0x7f076c161800 2024-03-29T10:14:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f0770e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c0a0970 con 0x7f076c0ceec0 2024-03-29T10:14:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f0769400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c15d640 0x7f076c15bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f076a800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f076c161800 0x7f076c15c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f0769e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f076c0ceec0 0x7f076c1580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f0769400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c15d640 0x7f076c15bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44560/0 (socket says 172.21.15.12:44560) 2024-03-29T10:14:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.887+0000 7f0769400700 1 -- 172.21.15.12:0/2201850308 learned_addr learned my addr 172.21.15.12:0/2201850308 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.888+0000 7f0769400700 1 -- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f076c0ceec0 msgr2=0x7f076c1580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.888+0000 7f0769400700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f076c0ceec0 0x7f076c1580f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.888+0000 7f0769400700 1 -- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f076c161800 msgr2=0x7f076c15c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.888+0000 7f0769400700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f076c161800 0x7f076c15c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.888+0000 7f0769400700 1 -- 172.21.15.12:0/2201850308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07580094a0 con 0x7f076c15d640 2024-03-29T10:14:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.888+0000 7f076a800700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f076c161800 0x7f076c15c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.888+0000 7f0769e00700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f076c0ceec0 0x7f076c1580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.888+0000 7f0769400700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c15d640 0x7f076c15bb30 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f07580090b0 tx=0x7f0758004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.888+0000 7f0762000700 1 -- 172.21.15.12:0/2201850308 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07580045d0 con 0x7f076c15d640 2024-03-29T10:14:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.888+0000 7f0762000700 1 -- 172.21.15.12:0/2201850308 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075801e070 con 0x7f076c15d640 2024-03-29T10:14:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.889+0000 7f0762000700 1 -- 172.21.15.12:0/2201850308 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07580038e0 con 0x7f076c15d640 2024-03-29T10:14:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.889+0000 7f0770e00700 1 -- 172.21.15.12:0/2201850308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c0a0c10 con 0x7f076c15d640 2024-03-29T10:14:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.889+0000 7f0770e00700 1 -- 172.21.15.12:0/2201850308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c09dce0 con 0x7f076c15d640 2024-03-29T10:14:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.890+0000 7f0770e00700 1 -- 172.21.15.12:0/2201850308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076c09df00 con 0x7f076c15d640 2024-03-29T10:14:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.891+0000 7f0762000700 1 -- 172.21.15.12:0/2201850308 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 64) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f0758003a40 con 0x7f076c15d640 2024-03-29T10:14:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.891+0000 7f0762000700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074c06aed0 0x7f074c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.892+0000 7f0762000700 1 -- 172.21.15.12:0/2201850308 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f075808a960 con 0x7f076c15d640 2024-03-29T10:14:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.892+0000 7f0769e00700 1 -- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074c06aed0 msgr2=0x7f074c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.892+0000 7f0769e00700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074c06aed0 0x7f074c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.900+0000 7f0762000700 1 -- 172.21.15.12:0/2201850308 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0758055640 con 0x7f076c15d640 2024-03-29T10:14:09.930 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:09.931+0000 7f209ba96240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:09.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.946+0000 7f0762000700 1 -- 172.21.15.12:0/2201850308 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 65) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f0758003cf0 con 0x7f076c15d640 2024-03-29T10:14:09.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.947+0000 7f0762000700 1 -- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074c06aed0 msgr2=0x7f074c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:09.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:09.947+0000 7f0762000700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074c06aed0 0x7f074c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.011 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:10.011+0000 7f200ec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:10.011 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:10.011+0000 7f200ec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:10.011 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:10.011+0000 7f200ec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:10.011 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:10.011+0000 7f200ec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:10.011 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:10.011+0000 7f200ec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:10.011 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:10.011+0000 7f200b000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:10.011 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:10.011+0000 7f200b000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:10.011 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:10.011+0000 7f200b000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:10.011 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:10.011+0000 7f200b000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:10.011 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:10.011+0000 7f200b000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.174+0000 7f0770e00700 1 -- 172.21.15.12:0/2201850308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f076c067720 con 0x7f076c15d640 2024-03-29T10:14:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.174+0000 7f0762000700 1 -- 172.21.15.12:0/2201850308 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f075805b460 con 0x7f076c15d640 2024-03-29T10:14:10.173 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:10.174 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":55,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7645,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/553963486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":553963486},{"type":"v1","addr":"172.21.15.159:6837","nonce":553963486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3051067707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3051067707},{"type":"v1","addr":"172.21.15.159:6836","nonce":3051067707}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3629410472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3629410472},{"type":"v1","addr":"172.21.15.12:6833","nonce":3629410472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:12:46.810806+0000","modified":"2024-03-29T10:12:50.294107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7917},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7917":{"gid":7917,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/753821395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":753821395},{"type":"v1","addr":"172.21.15.12:6835","nonce":753821395}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-29T10:14:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.177+0000 7f0770e00700 1 -- 172.21.15.12:0/2201850308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c15d640 msgr2=0x7f076c15bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.177+0000 7f0770e00700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c15d640 0x7f076c15bb30 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f07580090b0 tx=0x7f0758004750 comp rx=0 tx=0).stop 2024-03-29T10:14:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.177+0000 7f0770e00700 1 -- 172.21.15.12:0/2201850308 shutdown_connections 2024-03-29T10:14:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.177+0000 7f0770e00700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074c06aed0 0x7f074c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.177+0000 7f0770e00700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f076c161800 0x7f076c15c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.177+0000 7f0770e00700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f076c0ceec0 0x7f076c1580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.177+0000 7f0770e00700 1 --2- 172.21.15.12:0/2201850308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f076c15d640 0x7f076c15bb30 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.177+0000 7f0770e00700 1 -- 172.21.15.12:0/2201850308 >> 172.21.15.12:0/2201850308 conn(0x7f076c08aff0 msgr2=0x7f076c0d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.177+0000 7f0770e00700 1 -- 172.21.15.12:0/2201850308 shutdown_connections 2024-03-29T10:14:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.177+0000 7f0770e00700 1 -- 172.21.15.12:0/2201850308 wait complete. 2024-03-29T10:14:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 55 2024-03-29T10:14:10.196 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:10.340 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:10.341+0000 7fd4361a1240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:14:10.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.403+0000 7f958be00700 1 Processor -- start 2024-03-29T10:14:10.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.403+0000 7f958be00700 1 -- start start 2024-03-29T10:14:10.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.404+0000 7f958be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f958406a790 0x7f958406ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.404+0000 7f958be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 0x7f95840cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.404+0000 7f958be00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958406f850 con 0x7f95840afbe0 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.405+0000 7f958be00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f958406e3f0 con 0x7f958406a790 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.405+0000 7f958be00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958406b0b0 con 0x7f95840cebd0 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.405+0000 7f9589400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f958406a790 0x7f958406ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.405+0000 7f9589400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f958406a790 0x7f958406ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44572/0 (socket says 172.21.15.12:44572) 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.405+0000 7f9589400700 1 -- 172.21.15.12:0/1870362295 learned_addr learned my addr 172.21.15.12:0/1870362295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.405+0000 7f9589e00700 1 --2- 172.21.15.12:0/1870362295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 0x7f95840cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.405+0000 7f9589400700 1 -- 172.21.15.12:0/1870362295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 msgr2=0x7f95840cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.405+0000 7f9589400700 1 --2- 172.21.15.12:0/1870362295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 0x7f95840cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.405+0000 7f9589400700 1 -- 172.21.15.12:0/1870362295 >> v1:172.21.15.159:6789/0 conn(0x7f95840afbe0 legacy=0x7f958406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.405+0000 7f9589400700 1 -- 172.21.15.12:0/1870362295 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958406fad0 con 0x7f958406a790 2024-03-29T10:14:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.406+0000 7f9589400700 1 --2- 172.21.15.12:0/1870362295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f958406a790 0x7f958406ab70 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f95780098f0 tx=0x7f9578009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0f933e14daf099 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:10.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.406+0000 7f9583e00700 1 -- 172.21.15.12:0/1870362295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578011830 con 0x7f958406a790 2024-03-29T10:14:10.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.406+0000 7f9583e00700 1 -- 172.21.15.12:0/1870362295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9578011990 con 0x7f958406a790 2024-03-29T10:14:10.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.406+0000 7f9583e00700 1 -- 172.21.15.12:0/1870362295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578011d20 con 0x7f958406a790 2024-03-29T10:14:10.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.406+0000 7f958be00700 1 -- 172.21.15.12:0/1870362295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f958406a790 msgr2=0x7f958406ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:10.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.406+0000 7f958be00700 1 --2- 172.21.15.12:0/1870362295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f958406a790 0x7f958406ab70 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f95780098f0 tx=0x7f9578009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:10.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.407+0000 7f958be00700 1 -- 172.21.15.12:0/1870362295 shutdown_connections 2024-03-29T10:14:10.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.407+0000 7f958be00700 1 --2- 172.21.15.12:0/1870362295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 0x7f95840cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.407+0000 7f958be00700 1 --2- 172.21.15.12:0/1870362295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f958406a790 0x7f958406ab70 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.407+0000 7f958be00700 1 -- 172.21.15.12:0/1870362295 >> 172.21.15.12:0/1870362295 conn(0x7f95840b5ab0 msgr2=0x7f95840b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:10.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.407+0000 7f958be00700 1 -- 172.21.15.12:0/1870362295 shutdown_connections 2024-03-29T10:14:10.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.407+0000 7f958be00700 1 -- 172.21.15.12:0/1870362295 wait complete. 2024-03-29T10:14:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.409+0000 7f958be00700 1 Processor -- start 2024-03-29T10:14:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.410+0000 7f958be00700 1 -- start start 2024-03-29T10:14:10.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.411+0000 7f958be00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958406a790 0x7f95840cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.411+0000 7f958be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95840afbe0 0x7f958415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.411+0000 7f958be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 0x7f95840ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.411+0000 7f958be00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95840cb790 con 0x7f95840afbe0 2024-03-29T10:14:10.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.411+0000 7f958be00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958409ca50 con 0x7f958406a790 2024-03-29T10:14:10.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.411+0000 7f958be00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958409cd30 con 0x7f95840cebd0 2024-03-29T10:14:10.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.411+0000 7f9588a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95840afbe0 0x7f958415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.411+0000 7f9588a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95840afbe0 0x7f958415ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44588/0 (socket says 172.21.15.12:44588) 2024-03-29T10:14:10.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.411+0000 7f9588a00700 1 -- 172.21.15.12:0/4201427796 learned_addr learned my addr 172.21.15.12:0/4201427796 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:10.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.412+0000 7f9589e00700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 0x7f95840ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.412+0000 7f9589400700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958406a790 0x7f95840cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.412+0000 7f9588a00700 1 -- 172.21.15.12:0/4201427796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 msgr2=0x7f95840ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:10.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.412+0000 7f9588a00700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 0x7f95840ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.412+0000 7f9588a00700 1 -- 172.21.15.12:0/4201427796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958406a790 msgr2=0x7f95840cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:10.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.412+0000 7f9588a00700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958406a790 0x7f95840cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.412+0000 7f9588a00700 1 -- 172.21.15.12:0/4201427796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9578009550 con 0x7f95840afbe0 2024-03-29T10:14:10.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.412+0000 7f9589400700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958406a790 0x7f95840cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:10.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.412+0000 7f9589e00700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 0x7f95840ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:10.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.412+0000 7f9588a00700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95840afbe0 0x7f958415ba40 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f957000c450 tx=0x7f957000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:10.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.413+0000 7f9581600700 1 -- 172.21.15.12:0/4201427796 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95700052d0 con 0x7f95840afbe0 2024-03-29T10:14:10.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.413+0000 7f9581600700 1 -- 172.21.15.12:0/4201427796 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9570005430 con 0x7f95840afbe0 2024-03-29T10:14:10.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.413+0000 7f958be00700 1 -- 172.21.15.12:0/4201427796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958409d010 con 0x7f95840afbe0 2024-03-29T10:14:10.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.413+0000 7f9581600700 1 -- 172.21.15.12:0/4201427796 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95700038c0 con 0x7f95840afbe0 2024-03-29T10:14:10.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.413+0000 7f958be00700 1 -- 172.21.15.12:0/4201427796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958409c700 con 0x7f95840afbe0 2024-03-29T10:14:10.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.414+0000 7f958be00700 1 -- 172.21.15.12:0/4201427796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958409d530 con 0x7f95840afbe0 2024-03-29T10:14:10.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.415+0000 7f9581600700 1 -- 172.21.15.12:0/4201427796 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 65) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f9570003a20 con 0x7f95840afbe0 2024-03-29T10:14:10.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.416+0000 7f9581600700 1 -- 172.21.15.12:0/4201427796 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f9570088420 con 0x7f95840afbe0 2024-03-29T10:14:10.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.420+0000 7f9581600700 1 -- 172.21.15.12:0/4201427796 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f958409d530 con 0x7f95840afbe0 2024-03-29T10:14:10.464 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:10.465+0000 7fd4361a1240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:10.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.684+0000 7f958be00700 1 -- 172.21.15.12:0/4201427796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f958409d530 con 0x7f95840afbe0 2024-03-29T10:14:10.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.684+0000 7f9581600700 1 -- 172.21.15.12:0/4201427796 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f958409d530 con 0x7f95840afbe0 2024-03-29T10:14:10.684 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:10.684 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":55,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7645,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/553963486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":553963486},{"type":"v1","addr":"172.21.15.159:6837","nonce":553963486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3051067707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3051067707},{"type":"v1","addr":"172.21.15.159:6836","nonce":3051067707}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3629410472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3629410472},{"type":"v1","addr":"172.21.15.12:6833","nonce":3629410472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:12:46.810806+0000","modified":"2024-03-29T10:12:50.294107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7917},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7917":{"gid":7917,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/753821395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":753821395},{"type":"v1","addr":"172.21.15.12:6835","nonce":753821395}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-29T10:14:10.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.687+0000 7f958be00700 1 -- 172.21.15.12:0/4201427796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95840afbe0 msgr2=0x7f958415ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.687+0000 7f958be00700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95840afbe0 0x7f958415ba40 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f957000c450 tx=0x7f957000c760 comp rx=0 tx=0).stop 2024-03-29T10:14:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.687+0000 7f958be00700 1 -- 172.21.15.12:0/4201427796 shutdown_connections 2024-03-29T10:14:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.687+0000 7f958be00700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95840cebd0 0x7f95840ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.687+0000 7f958be00700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958406a790 0x7f95840cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.687+0000 7f958be00700 1 --2- 172.21.15.12:0/4201427796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95840afbe0 0x7f958415ba40 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.687+0000 7f958be00700 1 -- 172.21.15.12:0/4201427796 >> 172.21.15.12:0/4201427796 conn(0x7f95840b5ab0 msgr2=0x7f95840b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.687+0000 7f958be00700 1 -- 172.21.15.12:0/4201427796 shutdown_connections 2024-03-29T10:14:10.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.687+0000 7f958be00700 1 -- 172.21.15.12:0/4201427796 wait complete. 2024-03-29T10:14:10.688 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 55 2024-03-29T10:14:10.704 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:10.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.903+0000 7fc742e00700 1 Processor -- start 2024-03-29T10:14:10.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.903+0000 7fc742e00700 1 -- start start 2024-03-29T10:14:10.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.904+0000 7fc742e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc73c15d920 0x7fc73c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.904+0000 7fc742e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc73c09a910 0x7fc73c0cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.904+0000 7fc742e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc73c099a60 0x7fc73c099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.904+0000 7fc742e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73c09a3f0 con 0x7fc73c15d920 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.904+0000 7fc742e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73c0d08d0 con 0x7fc73c099a60 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.904+0000 7fc742e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c06c300 con 0x7fc73c09a910 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.904+0000 7fc73be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc73c15d920 0x7fc73c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.905+0000 7fc73be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc73c15d920 0x7fc73c06c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44600/0 (socket says 172.21.15.12:44600) 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.905+0000 7fc73be00700 1 -- 172.21.15.12:0/2873609381 learned_addr learned my addr 172.21.15.12:0/2873609381 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.905+0000 7fc73b400700 1 --2- 172.21.15.12:0/2873609381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc73c09a910 0x7fc73c0cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.905+0000 7fc740e00700 1 --2- 172.21.15.12:0/2873609381 >> v2:172.21.15.159:3300/0 conn(0x7fc73c099a60 0x7fc73c099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.905+0000 7fc73be00700 1 -- 172.21.15.12:0/2873609381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc73c09a910 msgr2=0x7fc73c0cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.905+0000 7fc73be00700 1 --2- 172.21.15.12:0/2873609381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc73c09a910 0x7fc73c0cfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.905+0000 7fc73be00700 1 -- 172.21.15.12:0/2873609381 >> v2:172.21.15.159:3300/0 conn(0x7fc73c099a60 msgr2=0x7fc73c099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.905+0000 7fc73be00700 1 --2- 172.21.15.12:0/2873609381 >> v2:172.21.15.159:3300/0 conn(0x7fc73c099a60 0x7fc73c099eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.905+0000 7fc73be00700 1 -- 172.21.15.12:0/2873609381 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c06b9d0 con 0x7fc73c15d920 2024-03-29T10:14:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.905+0000 7fc73be00700 1 --2- 172.21.15.12:0/2873609381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 0x7fc73c06c790 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc73c074590 tx=0x7fc72c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3697a26d61b2b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:10.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc73aa00700 1 -- 172.21.15.12:0/2873609381 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c011830 con 0x7fc73c15d920 2024-03-29T10:14:10.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc73aa00700 1 -- 172.21.15.12:0/2873609381 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c011990 con 0x7fc73c15d920 2024-03-29T10:14:10.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc73aa00700 1 -- 172.21.15.12:0/2873609381 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c011d20 con 0x7fc73c15d920 2024-03-29T10:14:10.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc742e00700 1 -- 172.21.15.12:0/2873609381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 msgr2=0x7fc73c06c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:10.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc742e00700 1 --2- 172.21.15.12:0/2873609381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 0x7fc73c06c790 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc73c074590 tx=0x7fc72c009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc742e00700 1 -- 172.21.15.12:0/2873609381 shutdown_connections 2024-03-29T10:14:10.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc742e00700 1 --2- 172.21.15.12:0/2873609381 >> v2:172.21.15.159:3300/0 conn(0x7fc73c099a60 0x7fc73c099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc742e00700 1 --2- 172.21.15.12:0/2873609381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 0x7fc73c06c790 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc742e00700 1 --2- 172.21.15.12:0/2873609381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc73c09a910 0x7fc73c0cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc742e00700 1 -- 172.21.15.12:0/2873609381 >> 172.21.15.12:0/2873609381 conn(0x7fc73c0d1350 msgr2=0x7fc73c0d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:10.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc742e00700 1 -- 172.21.15.12:0/2873609381 shutdown_connections 2024-03-29T10:14:10.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.906+0000 7fc742e00700 1 -- 172.21.15.12:0/2873609381 wait complete. 2024-03-29T10:14:10.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.908+0000 7fc742e00700 1 Processor -- start 2024-03-29T10:14:10.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.908+0000 7fc742e00700 1 -- start start 2024-03-29T10:14:10.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.909+0000 7fc742e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc73c099a60 0x7fc73c0b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.909+0000 7fc742e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc73c09a910 0x7fc73c151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.909+0000 7fc742e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 0x7fc73c0afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.909+0000 7fc742e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c15e210 con 0x7fc73c15d920 2024-03-29T10:14:10.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.909+0000 7fc742e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0b5a10 con 0x7fc73c099a60 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.909+0000 7fc742e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0b6220 con 0x7fc73c09a910 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc740e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 0x7fc73c0afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc740e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 0x7fc73c0afbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44604/0 (socket says 172.21.15.12:44604) 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc740e00700 1 -- 172.21.15.12:0/1941191521 learned_addr learned my addr 172.21.15.12:0/1941191521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc73b400700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc73c09a910 0x7fc73c151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc73be00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc73c099a60 0x7fc73c0b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc740e00700 1 -- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc73c09a910 msgr2=0x7fc73c151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc740e00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc73c09a910 0x7fc73c151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc740e00700 1 -- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc73c099a60 msgr2=0x7fc73c0b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc740e00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc73c099a60 0x7fc73c0b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc740e00700 1 -- 172.21.15.12:0/1941191521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c009550 con 0x7fc73c15d920 2024-03-29T10:14:10.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc740e00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 0x7fc73c0afbe0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc73400eed0 tx=0x7fc73400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:10.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.910+0000 7fc733e00700 1 -- 172.21.15.12:0/1941191521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc734010040 con 0x7fc73c15d920 2024-03-29T10:14:10.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.911+0000 7fc733e00700 1 -- 172.21.15.12:0/1941191521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73400b9f0 con 0x7fc73c15d920 2024-03-29T10:14:10.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.911+0000 7fc733e00700 1 -- 172.21.15.12:0/1941191521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73400bda0 con 0x7fc73c15d920 2024-03-29T10:14:10.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.911+0000 7fc742e00700 1 -- 172.21.15.12:0/1941191521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c0b50f0 con 0x7fc73c15d920 2024-03-29T10:14:10.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.911+0000 7fc742e00700 1 -- 172.21.15.12:0/1941191521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c0b5f40 con 0x7fc73c15d920 2024-03-29T10:14:10.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.912+0000 7fc742e00700 1 -- 172.21.15.12:0/1941191521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c067720 con 0x7fc73c15d920 2024-03-29T10:14:10.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.913+0000 7fc733e00700 1 -- 172.21.15.12:0/1941191521 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 65) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fc73401d400 con 0x7fc73c15d920 2024-03-29T10:14:10.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.914+0000 7fc733e00700 1 -- 172.21.15.12:0/1941191521 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fc734089670 con 0x7fc73c15d920 2024-03-29T10:14:10.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.919+0000 7fc733e00700 1 -- 172.21.15.12:0/1941191521 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc7340549d0 con 0x7fc73c15d920 2024-03-29T10:14:10.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.946+0000 7fc733e00700 1 -- 172.21.15.12:0/1941191521 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 66) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fc734054480 con 0x7fc73c15d920 2024-03-29T10:14:10.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.946+0000 7fc733e00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc71c036150 0x7fc71c0723c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:10.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.947+0000 7fc73be00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc71c036150 0x7fc71c0723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:10.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:10.950+0000 7fc73be00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc71c036150 0x7fc71c0723c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc72c014040 tx=0x7fc72c008030 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:11.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.136+0000 7fc742e00700 1 -- 172.21.15.12:0/1941191521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc73c0700f0 con 0x7fc73c15d920 2024-03-29T10:14:11.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.138+0000 7fc733e00700 1 -- 172.21.15.12:0/1941191521 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+17116 (secure 0 0 0) 0x7fc734073f10 con 0x7fc73c15d920 2024-03-29T10:14:11.137 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:11.137 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":127,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:14:09.934422+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:12:45.171490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-29T10:12:45.787701+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":126,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1314694115":"2024-03-30T10:14:09.934349+0000","172.21.15.159:0/2708567970":"2024-03-30T10:14:09.934349+0000","172.21.15.159:0/3297185412":"2024-03-30T10:14:09.934349+0000","172.21.15.159:0/3234151766":"2024-03-30T10:14:09.934349+0000","172.21.15.159:0/160485444":"2024-03-30T10:14:09.934349+0000","172.21.15.159:0/291095819":"2024-03-30T10:14:09.934349+0000","172.21.15.159:0/1231831807":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/3573895366":"2024-03-30T10:13:04.000570+0000","172.21.15.159:6833/43444":"2024-03-30T10:13:04.000570+0000","172.21.15.159:6832/43444":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/844715676":"2024-03-30T10:14:09.934349+0000","172.21.15.159:0/2082234733":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/3500442951":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/3021777852":"2024-03-30T10:14:09.934349+0000","172.21.15.159:0/949098073":"2024-03-30T10:13:04.000570+0000","172.21.15.159:0/497911868":"2024-03-30T10:13:04.000570+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:11.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 -- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc71c036150 msgr2=0x7fc71c0723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc71c036150 0x7fc71c0723c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc72c014040 tx=0x7fc72c008030 comp rx=0 tx=0).stop 2024-03-29T10:14:11.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 -- 172.21.15.12:0/1941191521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 msgr2=0x7fc73c0afbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 0x7fc73c0afbe0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc73400eed0 tx=0x7fc73400ef00 comp rx=0 tx=0).stop 2024-03-29T10:14:11.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 -- 172.21.15.12:0/1941191521 shutdown_connections 2024-03-29T10:14:11.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc71c036150 0x7fc71c0723c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc73c15d920 0x7fc73c0afbe0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc73c099a60 0x7fc73c0b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 --2- 172.21.15.12:0/1941191521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc73c09a910 0x7fc73c151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 -- 172.21.15.12:0/1941191521 >> 172.21.15.12:0/1941191521 conn(0x7fc73c0d1350 msgr2=0x7fc73c0b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:11.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 -- 172.21.15.12:0/1941191521 shutdown_connections 2024-03-29T10:14:11.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.140+0000 7fc742e00700 1 -- 172.21.15.12:0/1941191521 wait complete. 2024-03-29T10:14:11.157 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T10:14:11.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.357+0000 7f064d600700 1 Processor -- start 2024-03-29T10:14:11.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.357+0000 7f064d600700 1 -- start start 2024-03-29T10:14:11.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.358+0000 7f064d600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06480d06a0 0x7f064815e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:11.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.358+0000 7f064d600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f064815eda0 con 0x7f064806fa20 2024-03-29T10:14:11.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.358+0000 7f064d600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0648099dd0 con 0x7f06480cd5c0 2024-03-29T10:14:11.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.358+0000 7f064d600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064809a0b0 con 0x7f06480d06a0 2024-03-29T10:14:11.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.358+0000 7f0647400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06480d06a0 0x7f064815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:11.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.358+0000 7f0647400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06480d06a0 0x7f064815e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44606/0 (socket says 172.21.15.12:44606) 2024-03-29T10:14:11.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.358+0000 7f0647400700 1 -- 172.21.15.12:0/1098103107 learned_addr learned my addr 172.21.15.12:0/1098103107 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:11.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.359+0000 7f0647400700 1 -- 172.21.15.12:0/1098103107 >> v1:172.21.15.159:6789/0 conn(0x7f06480cd5c0 legacy=0x7f064806f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.359+0000 7f0647400700 1 -- 172.21.15.12:0/1098103107 >> v1:172.21.15.12:6789/0 conn(0x7f064806fa20 legacy=0x7f064815cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.359+0000 7f0647400700 1 -- 172.21.15.12:0/1098103107 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064809a390 con 0x7f06480d06a0 2024-03-29T10:14:11.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.359+0000 7f0647400700 1 --2- 172.21.15.12:0/1098103107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480d06a0 0x7f064815e860 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f0640009e80 tx=0x7f064000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9498d5db299f4645 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:11.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.359+0000 7f0646000700 1 -- 172.21.15.12:0/1098103107 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064000e070 con 0x7f06480d06a0 2024-03-29T10:14:11.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.359+0000 7f0646000700 1 -- 172.21.15.12:0/1098103107 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064000b980 con 0x7f06480d06a0 2024-03-29T10:14:11.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.359+0000 7f0646000700 1 -- 172.21.15.12:0/1098103107 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064000bd30 con 0x7f06480d06a0 2024-03-29T10:14:11.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.360+0000 7f064d600700 1 -- 172.21.15.12:0/1098103107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480d06a0 msgr2=0x7f064815e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.360+0000 7f064d600700 1 --2- 172.21.15.12:0/1098103107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480d06a0 0x7f064815e860 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f0640009e80 tx=0x7f064000b330 comp rx=0 tx=0).stop 2024-03-29T10:14:11.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.360+0000 7f064d600700 1 -- 172.21.15.12:0/1098103107 shutdown_connections 2024-03-29T10:14:11.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.360+0000 7f064d600700 1 --2- 172.21.15.12:0/1098103107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480d06a0 0x7f064815e860 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.360+0000 7f064d600700 1 -- 172.21.15.12:0/1098103107 >> 172.21.15.12:0/1098103107 conn(0x7f06480d1370 msgr2=0x7f06480d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:11.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.360+0000 7f064d600700 1 -- 172.21.15.12:0/1098103107 shutdown_connections 2024-03-29T10:14:11.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.360+0000 7f064d600700 1 -- 172.21.15.12:0/1098103107 wait complete. 2024-03-29T10:14:11.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.362+0000 7f064d600700 1 Processor -- start 2024-03-29T10:14:11.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.363+0000 7f064d600700 1 -- start start 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f064d600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f064806fa20 0x7f0648150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f064d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480cd5c0 0x7f06480b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f064d600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06480d06a0 0x7f06480b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f064d600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06480b6560 con 0x7f06480cd5c0 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f064d600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06480b09f0 con 0x7f06480d06a0 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f064d600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06480b0cd0 con 0x7f064806fa20 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f063fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480cd5c0 0x7f06480b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f063fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480cd5c0 0x7f06480b1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44616/0 (socket says 172.21.15.12:44616) 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f063fe00700 1 -- 172.21.15.12:0/2246855186 learned_addr learned my addr 172.21.15.12:0/2246855186 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f0646a00700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f064806fa20 0x7f0648150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.364+0000 7f0647400700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06480d06a0 0x7f06480b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f063fe00700 1 -- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f064806fa20 msgr2=0x7f0648150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f063fe00700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f064806fa20 0x7f0648150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f063fe00700 1 -- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06480d06a0 msgr2=0x7f06480b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f063fe00700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06480d06a0 0x7f06480b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f063fe00700 1 -- 172.21.15.12:0/2246855186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0640009b30 con 0x7f06480cd5c0 2024-03-29T10:14:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f063fe00700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480cd5c0 0x7f06480b1e40 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f063000c4b0 tx=0x7f063000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f063ea00700 1 -- 172.21.15.12:0/2246855186 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06300052d0 con 0x7f06480cd5c0 2024-03-29T10:14:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f063ea00700 1 -- 172.21.15.12:0/2246855186 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0630005430 con 0x7f06480cd5c0 2024-03-29T10:14:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f064d600700 1 -- 172.21.15.12:0/2246855186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06480b0fb0 con 0x7f06480cd5c0 2024-03-29T10:14:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f063ea00700 1 -- 172.21.15.12:0/2246855186 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06300038c0 con 0x7f06480cd5c0 2024-03-29T10:14:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.365+0000 7f064d600700 1 -- 172.21.15.12:0/2246855186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0648151320 con 0x7f06480cd5c0 2024-03-29T10:14:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.367+0000 7f064d600700 1 -- 172.21.15.12:0/2246855186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06480b14d0 con 0x7f06480cd5c0 2024-03-29T10:14:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.367+0000 7f063ea00700 1 -- 172.21.15.12:0/2246855186 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 66) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f0630003a20 con 0x7f06480cd5c0 2024-03-29T10:14:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.368+0000 7f063ea00700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f062406aae0 0x7f062406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.368+0000 7f063ea00700 1 -- 172.21.15.12:0/2246855186 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f06300884c0 con 0x7f06480cd5c0 2024-03-29T10:14:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.369+0000 7f0646a00700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f062406aae0 0x7f062406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.369+0000 7f0646a00700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f062406aae0 0x7f062406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0638007fd0 tx=0x7f0638009040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.373+0000 7f063ea00700 1 -- 172.21.15.12:0/2246855186 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06480b14d0 con 0x7f06480cd5c0 2024-03-29T10:14:11.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.628+0000 7f064d600700 1 -- 172.21.15.12:0/2246855186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f064815eda0 con 0x7f06480cd5c0 2024-03-29T10:14:11.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T10:14:11.629+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T10:14:11.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.641+0000 7f063ea00700 1 -- 172.21.15.12:0/2246855186 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v56) v1 ==== 161+0+0 (secure 0 0 0) 0x7f064815eda0 con 0x7f06480cd5c0 2024-03-29T10:14:11.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 -- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f062406aae0 msgr2=0x7f062406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f062406aae0 0x7f062406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0638007fd0 tx=0x7f0638009040 comp rx=0 tx=0).stop 2024-03-29T10:14:11.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 -- 172.21.15.12:0/2246855186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480cd5c0 msgr2=0x7f06480b1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480cd5c0 0x7f06480b1e40 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f063000c4b0 tx=0x7f063000c7c0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 -- 172.21.15.12:0/2246855186 shutdown_connections 2024-03-29T10:14:11.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f062406aae0 0x7f062406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06480d06a0 0x7f06480b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f064806fa20 0x7f0648150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 --2- 172.21.15.12:0/2246855186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06480cd5c0 0x7f06480b1e40 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 -- 172.21.15.12:0/2246855186 >> 172.21.15.12:0/2246855186 conn(0x7f06480d1370 msgr2=0x7f064815e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:11.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 -- 172.21.15.12:0/2246855186 shutdown_connections 2024-03-29T10:14:11.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.644+0000 7f064d600700 1 -- 172.21.15.12:0/2246855186 wait complete. 2024-03-29T10:14:11.644 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T10:14:11.661 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T10:14:11.707 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T10:14:11.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.867+0000 7f2271200700 1 Processor -- start 2024-03-29T10:14:11.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.867+0000 7f2271200700 1 -- start start 2024-03-29T10:14:11.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.868+0000 7f2271200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f226c0ce840 0x7f226c165370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:11.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.869+0000 7f2271200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c0cb310 con 0x7f226c15a5c0 2024-03-29T10:14:11.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.869+0000 7f2271200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c0cca90 con 0x7f226c15dfe0 2024-03-29T10:14:11.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.869+0000 7f2271200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226c0711a0 con 0x7f226c0ce840 2024-03-29T10:14:11.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.869+0000 7f2269e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f226c15a5c0 0x7f226c0709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42614/0 (socket says 172.21.15.12:42614) 2024-03-29T10:14:11.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.869+0000 7f2269e00700 1 -- 172.21.15.12:0/1286962107 learned_addr learned my addr 172.21.15.12:0/1286962107 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:11.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.869+0000 7f226a800700 1 --2- 172.21.15.12:0/1286962107 >> v2:172.21.15.159:3300/0 conn(0x7f226c0ce840 0x7f226c165370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:11.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.869+0000 7f2268a00700 1 -- 172.21.15.12:0/1286962107 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432095117 0 0) 0x7f226c0cb310 con 0x7f226c15a5c0 2024-03-29T10:14:11.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.869+0000 7f2268a00700 1 -- 172.21.15.12:0/1286962107 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22500057c0 con 0x7f226c15a5c0 2024-03-29T10:14:11.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.869+0000 7f2268a00700 1 -- 172.21.15.12:0/1286962107 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740655054 0 0) 0x7f226c0cca90 con 0x7f226c15dfe0 2024-03-29T10:14:11.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.869+0000 7f2268a00700 1 -- 172.21.15.12:0/1286962107 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f226c0cb310 con 0x7f226c15dfe0 2024-03-29T10:14:11.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.870+0000 7f226a800700 1 -- 172.21.15.12:0/1286962107 >> v1:172.21.15.159:6789/0 conn(0x7f226c15dfe0 legacy=0x7f226c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.870+0000 7f226a800700 1 -- 172.21.15.12:0/1286962107 >> v1:172.21.15.12:6789/0 conn(0x7f226c15a5c0 legacy=0x7f226c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.870+0000 7f226a800700 1 -- 172.21.15.12:0/1286962107 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c071480 con 0x7f226c0ce840 2024-03-29T10:14:11.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.870+0000 7f226a800700 1 --2- 172.21.15.12:0/1286962107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f226c0ce840 0x7f226c165370 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2264009d90 tx=0x7f226400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d59723e8152058 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:11.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.870+0000 7f2268a00700 1 -- 172.21.15.12:0/1286962107 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226400e070 con 0x7f226c0ce840 2024-03-29T10:14:11.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.870+0000 7f2268a00700 1 -- 172.21.15.12:0/1286962107 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226400bb20 con 0x7f226c0ce840 2024-03-29T10:14:11.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.870+0000 7f2268a00700 1 -- 172.21.15.12:0/1286962107 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264012430 con 0x7f226c0ce840 2024-03-29T10:14:11.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.871+0000 7f2271200700 1 -- 172.21.15.12:0/1286962107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f226c0ce840 msgr2=0x7f226c165370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.871+0000 7f2271200700 1 --2- 172.21.15.12:0/1286962107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f226c0ce840 0x7f226c165370 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2264009d90 tx=0x7f226400b330 comp rx=0 tx=0).stop 2024-03-29T10:14:11.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.871+0000 7f2271200700 1 -- 172.21.15.12:0/1286962107 shutdown_connections 2024-03-29T10:14:11.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.871+0000 7f2271200700 1 --2- 172.21.15.12:0/1286962107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f226c0ce840 0x7f226c165370 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.871+0000 7f2271200700 1 -- 172.21.15.12:0/1286962107 >> 172.21.15.12:0/1286962107 conn(0x7f226c0b0d70 msgr2=0x7f226c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:11.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.871+0000 7f2271200700 1 -- 172.21.15.12:0/1286962107 shutdown_connections 2024-03-29T10:14:11.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.871+0000 7f2271200700 1 -- 172.21.15.12:0/1286962107 wait complete. 2024-03-29T10:14:11.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.873+0000 7f2271200700 1 Processor -- start 2024-03-29T10:14:11.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.873+0000 7f2271200700 1 -- start start 2024-03-29T10:14:11.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2271200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f226c0ce840 0x7f226c154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:11.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2271200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f226c15a5c0 0x7f226c0ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:11.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2271200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f226c15dfe0 0x7f226c0ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:11.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2271200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c157bb0 con 0x7f226c0ce840 2024-03-29T10:14:11.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2271200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c157e00 con 0x7f226c15dfe0 2024-03-29T10:14:11.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2271200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c1580e0 con 0x7f226c15a5c0 2024-03-29T10:14:11.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2269e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f226c0ce840 0x7f226c154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2269e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f226c0ce840 0x7f226c154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44634/0 (socket says 172.21.15.12:44634) 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2269e00700 1 -- 172.21.15.12:0/479525738 learned_addr learned my addr 172.21.15.12:0/479525738 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2269400700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f226c15a5c0 0x7f226c0ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f226a800700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f226c15dfe0 0x7f226c0ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2269e00700 1 -- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f226c15a5c0 msgr2=0x7f226c0ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2269e00700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f226c15a5c0 0x7f226c0ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2269e00700 1 -- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f226c15dfe0 msgr2=0x7f226c0ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2269e00700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f226c15dfe0 0x7f226c0ac590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.875+0000 7f2269e00700 1 -- 172.21.15.12:0/479525738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2264009a70 con 0x7f226c0ce840 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.876+0000 7f2269e00700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f226c0ce840 0x7f226c154630 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f225c00c700 tx=0x7f225c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.876+0000 7f2262000700 1 -- 172.21.15.12:0/479525738 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c005d70 con 0x7f226c0ce840 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.876+0000 7f2262000700 1 -- 172.21.15.12:0/479525738 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c0070d0 con 0x7f226c0ce840 2024-03-29T10:14:11.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.876+0000 7f2271200700 1 -- 172.21.15.12:0/479525738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c1583c0 con 0x7f226c0ce840 2024-03-29T10:14:11.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.876+0000 7f2262000700 1 -- 172.21.15.12:0/479525738 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c007460 con 0x7f226c0ce840 2024-03-29T10:14:11.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.876+0000 7f2271200700 1 -- 172.21.15.12:0/479525738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c158980 con 0x7f226c0ce840 2024-03-29T10:14:11.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.877+0000 7f2271200700 1 -- 172.21.15.12:0/479525738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c158ba0 con 0x7f226c0ce840 2024-03-29T10:14:11.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.878+0000 7f2262000700 1 -- 172.21.15.12:0/479525738 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 66) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f225c0075c0 con 0x7f226c0ce840 2024-03-29T10:14:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.879+0000 7f2262000700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f224c06aae0 0x7f224c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.879+0000 7f2262000700 1 -- 172.21.15.12:0/479525738 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f225c088760 con 0x7f226c0ce840 2024-03-29T10:14:11.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.880+0000 7f2269400700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f224c06aae0 0x7f224c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:11.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.881+0000 7f2269400700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f224c06aae0 0x7f224c06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f226c14de00 tx=0x7f2258009040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:11.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:11.883+0000 7f2262000700 1 -- 172.21.15.12:0/479525738 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f225c053890 con 0x7f226c0ce840 2024-03-29T10:14:12.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.147+0000 7f2271200700 1 -- 172.21.15.12:0/479525738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f226c067720 con 0x7f226c0ce840 2024-03-29T10:14:12.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.644+0000 7f2262000700 1 -- 172.21.15.12:0/479525738 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v57) v1 ==== 104+0+0 (secure 0 0 0) 0x7f225c056eb0 con 0x7f226c0ce840 2024-03-29T10:14:12.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 -- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f224c06aae0 msgr2=0x7f224c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:12.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f224c06aae0 0x7f224c06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f226c14de00 tx=0x7f2258009040 comp rx=0 tx=0).stop 2024-03-29T10:14:12.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 -- 172.21.15.12:0/479525738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f226c0ce840 msgr2=0x7f226c154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:12.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f226c0ce840 0x7f226c154630 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f225c00c700 tx=0x7f225c00ca10 comp rx=0 tx=0).stop 2024-03-29T10:14:12.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 -- 172.21.15.12:0/479525738 shutdown_connections 2024-03-29T10:14:12.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f224c06aae0 0x7f224c06cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:12.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f226c15dfe0 0x7f226c0ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:12.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f226c0ce840 0x7f226c154630 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:12.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 --2- 172.21.15.12:0/479525738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f226c15a5c0 0x7f226c0ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:12.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 -- 172.21.15.12:0/479525738 >> 172.21.15.12:0/479525738 conn(0x7f226c0b0d70 msgr2=0x7f226c0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:12.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.647+0000 7f2271200700 1 -- 172.21.15.12:0/479525738 shutdown_connections 2024-03-29T10:14:12.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.648+0000 7f2271200700 1 -- 172.21.15.12:0/479525738 wait complete. 2024-03-29T10:14:12.669 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T10:14:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.874+0000 7f00d0800700 1 Processor -- start 2024-03-29T10:14:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.874+0000 7f00d0800700 1 -- start start 2024-03-29T10:14:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.875+0000 7f00d0800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c80d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.875+0000 7f00d0800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f00c8162030 0x7f00c80d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.875+0000 7f00d0800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c815dd10 con 0x7f00c80d20d0 2024-03-29T10:14:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.875+0000 7f00d0800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c815ec40 con 0x7f00c8162030 2024-03-29T10:14:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.875+0000 7f00d0800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c80d2560 con 0x7f00c815d930 2024-03-29T10:14:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.875+0000 7f00cde00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c80d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cd400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f00c8162030 0x7f00c80d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cde00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c80d3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:42676/0 (socket says 172.21.15.12:42676) 2024-03-29T10:14:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cde00700 1 -- 172.21.15.12:0/983280212 learned_addr learned my addr 172.21.15.12:0/983280212 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00ce800700 1 --1- 172.21.15.12:0/983280212 >> v1:172.21.15.159:6789/0 conn(0x7f00c80d20d0 0x7f00c80cfc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:52224/0 (socket says 172.21.15.12:52224) 2024-03-29T10:14:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cca00700 1 -- 172.21.15.12:0/983280212 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683906840 0 0) 0x7f00c815dd10 con 0x7f00c80d20d0 2024-03-29T10:14:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cca00700 1 -- 172.21.15.12:0/983280212 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b00035b0 con 0x7f00c80d20d0 2024-03-29T10:14:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cde00700 1 -- 172.21.15.12:0/983280212 >> v2:172.21.15.159:3300/0 conn(0x7f00c8162030 msgr2=0x7f00c80d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cde00700 1 --2- 172.21.15.12:0/983280212 >> v2:172.21.15.159:3300/0 conn(0x7f00c8162030 0x7f00c80d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cde00700 1 -- 172.21.15.12:0/983280212 >> v1:172.21.15.159:6789/0 conn(0x7f00c80d20d0 legacy=0x7f00c80cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cde00700 1 -- 172.21.15.12:0/983280212 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c815dfc0 con 0x7f00c815d930 2024-03-29T10:14:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cd400700 1 --2- 172.21.15.12:0/983280212 >> v2:172.21.15.159:3300/0 conn(0x7f00c8162030 0x7f00c80d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.876+0000 7f00cde00700 1 --2- 172.21.15.12:0/983280212 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c80d3ed0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0098f0 tx=0x7f00bc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93c79db3dd691529 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00cca00700 1 -- 172.21.15.12:0/983280212 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc011830 con 0x7f00c815d930 2024-03-29T10:14:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00cca00700 1 -- 172.21.15.12:0/983280212 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc011990 con 0x7f00c815d930 2024-03-29T10:14:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00cca00700 1 -- 172.21.15.12:0/983280212 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc011d20 con 0x7f00c815d930 2024-03-29T10:14:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00d0800700 1 -- 172.21.15.12:0/983280212 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 msgr2=0x7f00c80d3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00d0800700 1 --2- 172.21.15.12:0/983280212 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c80d3ed0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0098f0 tx=0x7f00bc009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00d0800700 1 -- 172.21.15.12:0/983280212 shutdown_connections 2024-03-29T10:14:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00d0800700 1 --2- 172.21.15.12:0/983280212 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c80d3ed0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00d0800700 1 --2- 172.21.15.12:0/983280212 >> v2:172.21.15.159:3300/0 conn(0x7f00c8162030 0x7f00c80d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00d0800700 1 -- 172.21.15.12:0/983280212 >> 172.21.15.12:0/983280212 conn(0x7f00c808aff0 msgr2=0x7f00c80d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00d0800700 1 -- 172.21.15.12:0/983280212 shutdown_connections 2024-03-29T10:14:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.877+0000 7f00d0800700 1 -- 172.21.15.12:0/983280212 wait complete. 2024-03-29T10:14:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.879+0000 7f00d0800700 1 Processor -- start 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.879+0000 7f00d0800700 1 -- start start 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.880+0000 7f00d0800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00c80d20d0 0x7f00c8154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.880+0000 7f00d0800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c815eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.880+0000 7f00d0800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00c8162030 0x7f00c806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.880+0000 7f00d0800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c81f2f50 con 0x7f00c8162030 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.880+0000 7f00d0800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c815f530 con 0x7f00c80d20d0 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.880+0000 7f00d0800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c815f7b0 con 0x7f00c815d930 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.880+0000 7f00ce800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00c8162030 0x7f00c806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.880+0000 7f00ce800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00c8162030 0x7f00c806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44650/0 (socket says 172.21.15.12:44650) 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00ce800700 1 -- 172.21.15.12:0/3999964569 learned_addr learned my addr 172.21.15.12:0/3999964569 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00cde00700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00c80d20d0 0x7f00c8154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00cd400700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c815eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00ce800700 1 -- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 msgr2=0x7f00c815eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00ce800700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c815eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00ce800700 1 -- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00c80d20d0 msgr2=0x7f00c8154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00ce800700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00c80d20d0 0x7f00c8154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00ce800700 1 -- 172.21.15.12:0/3999964569 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b80094a0 con 0x7f00c8162030 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00cd400700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c815eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00ce800700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00c8162030 0x7f00c806ca80 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f00c400cc50 tx=0x7f00c400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.881+0000 7f00c2000700 1 -- 172.21.15.12:0/3999964569 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c400ab80 con 0x7f00c8162030 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.882+0000 7f00c2000700 1 -- 172.21.15.12:0/3999964569 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c400ace0 con 0x7f00c8162030 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.882+0000 7f00c2000700 1 -- 172.21.15.12:0/3999964569 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c4007290 con 0x7f00c8162030 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.882+0000 7f00d0800700 1 -- 172.21.15.12:0/3999964569 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00bc009550 con 0x7f00c8162030 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.882+0000 7f00d0800700 1 -- 172.21.15.12:0/3999964569 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c815fd70 con 0x7f00c8162030 2024-03-29T10:14:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.883+0000 7f00d0800700 1 -- 172.21.15.12:0/3999964569 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c80729f0 con 0x7f00c8162030 2024-03-29T10:14:12.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.884+0000 7f00c2000700 1 -- 172.21.15.12:0/3999964569 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 67) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f00c40073f0 con 0x7f00c8162030 2024-03-29T10:14:12.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.884+0000 7f00c2000700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f00ac06ad20 0x7f00ac06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:12.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.885+0000 7f00c2000700 1 -- 172.21.15.12:0/3999964569 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f00c4089650 con 0x7f00c8162030 2024-03-29T10:14:12.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.885+0000 7f00cde00700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f00ac06ad20 0x7f00ac06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:12.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.886+0000 7f00cde00700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f00ac06ad20 0x7f00ac06d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f00bc000c00 tx=0x7f00bc008030 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:12.889+0000 7f00c2000700 1 -- 172.21.15.12:0/3999964569 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f00c4054420 con 0x7f00c8162030 2024-03-29T10:14:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:13.104+0000 7f00d0800700 1 -- 172.21.15.12:0/3999964569 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f00c8067720 con 0x7f00c8162030 2024-03-29T10:14:14.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.107+0000 7f00c2000700 1 -- 172.21.15.12:0/3999964569 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v130) v1 ==== 181+0+0 (secure 0 0 0) 0x7f00c4057a40 con 0x7f00c8162030 2024-03-29T10:14:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.110+0000 7f00d0800700 1 -- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f00ac06ad20 msgr2=0x7f00ac06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.110+0000 7f00d0800700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f00ac06ad20 0x7f00ac06d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f00bc000c00 tx=0x7f00bc008030 comp rx=0 tx=0).stop 2024-03-29T10:14:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.110+0000 7f00d0800700 1 -- 172.21.15.12:0/3999964569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00c8162030 msgr2=0x7f00c806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.110+0000 7f00d0800700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00c8162030 0x7f00c806ca80 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f00c400cc50 tx=0x7f00c400cf60 comp rx=0 tx=0).stop 2024-03-29T10:14:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.111+0000 7f00d0800700 1 -- 172.21.15.12:0/3999964569 shutdown_connections 2024-03-29T10:14:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.111+0000 7f00d0800700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f00ac06ad20 0x7f00ac06d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.111+0000 7f00d0800700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00c8162030 0x7f00c806ca80 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.111+0000 7f00d0800700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00c80d20d0 0x7f00c8154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.111+0000 7f00d0800700 1 --2- 172.21.15.12:0/3999964569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00c815d930 0x7f00c815eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.111+0000 7f00d0800700 1 -- 172.21.15.12:0/3999964569 >> 172.21.15.12:0/3999964569 conn(0x7f00c808aff0 msgr2=0x7f00c80b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:14.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.111+0000 7f00d0800700 1 -- 172.21.15.12:0/3999964569 shutdown_connections 2024-03-29T10:14:14.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.111+0000 7f00d0800700 1 -- 172.21.15.12:0/3999964569 wait complete. 2024-03-29T10:14:14.113 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T10:14:14.132 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T10:14:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.346+0000 7fea7de00700 1 Processor -- start 2024-03-29T10:14:14.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.346+0000 7fea7de00700 1 -- start start 2024-03-29T10:14:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.347+0000 7fea7de00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fea7815ddf0 0x7fea780ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.347+0000 7fea7de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fea7815e1d0 0x7fea780d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea7de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea780d2710 0x7fea781612f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea7de00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea780d2b80 con 0x7fea7815e1d0 2024-03-29T10:14:14.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea7de00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea7815ec60 con 0x7fea7815ddf0 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea7de00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea780ce8a0 con 0x7fea780d2710 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea76000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fea7815e1d0 0x7fea780d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea76000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fea7815e1d0 0x7fea780d21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44660/0 (socket says 172.21.15.12:44660) 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea76000700 1 -- 172.21.15.12:0/3431934965 learned_addr learned my addr 172.21.15.12:0/3431934965 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea77400700 1 --2- 172.21.15.12:0/3431934965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea780d2710 0x7fea781612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea76a00700 1 --2- 172.21.15.12:0/3431934965 >> v2:172.21.15.159:3300/0 conn(0x7fea7815ddf0 0x7fea780ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea76000700 1 -- 172.21.15.12:0/3431934965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea780d2710 msgr2=0x7fea781612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea76000700 1 --2- 172.21.15.12:0/3431934965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea780d2710 0x7fea781612f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea76000700 1 -- 172.21.15.12:0/3431934965 >> v2:172.21.15.159:3300/0 conn(0x7fea7815ddf0 msgr2=0x7fea780ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea76000700 1 --2- 172.21.15.12:0/3431934965 >> v2:172.21.15.159:3300/0 conn(0x7fea7815ddf0 0x7fea780ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea76000700 1 -- 172.21.15.12:0/3431934965 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea780cdc90 con 0x7fea7815e1d0 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.348+0000 7fea76a00700 1 --2- 172.21.15.12:0/3431934965 >> v2:172.21.15.159:3300/0 conn(0x7fea7815ddf0 0x7fea780ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:14:14.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea76000700 1 --2- 172.21.15.12:0/3431934965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815e1d0 0x7fea780d21d0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fea7806b3e0 tx=0x7fea64009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f184533706d8fcc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea75600700 1 -- 172.21.15.12:0/3431934965 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6400b870 con 0x7fea7815e1d0 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea75600700 1 -- 172.21.15.12:0/3431934965 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6400b9d0 con 0x7fea7815e1d0 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea75600700 1 -- 172.21.15.12:0/3431934965 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6400bd60 con 0x7fea7815e1d0 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea7de00700 1 -- 172.21.15.12:0/3431934965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815e1d0 msgr2=0x7fea780d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea7de00700 1 --2- 172.21.15.12:0/3431934965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815e1d0 0x7fea780d21d0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fea7806b3e0 tx=0x7fea64009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea7de00700 1 -- 172.21.15.12:0/3431934965 shutdown_connections 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea7de00700 1 --2- 172.21.15.12:0/3431934965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea780d2710 0x7fea781612f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea7de00700 1 --2- 172.21.15.12:0/3431934965 >> v2:172.21.15.159:3300/0 conn(0x7fea7815ddf0 0x7fea780ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea7de00700 1 --2- 172.21.15.12:0/3431934965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815e1d0 0x7fea780d21d0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.349+0000 7fea7de00700 1 -- 172.21.15.12:0/3431934965 >> 172.21.15.12:0/3431934965 conn(0x7fea7808aff0 msgr2=0x7fea780d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.350+0000 7fea7de00700 1 -- 172.21.15.12:0/3431934965 shutdown_connections 2024-03-29T10:14:14.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.350+0000 7fea7de00700 1 -- 172.21.15.12:0/3431934965 wait complete. 2024-03-29T10:14:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.351+0000 7fea7de00700 1 Processor -- start 2024-03-29T10:14:14.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.351+0000 7fea7de00700 1 -- start start 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.352+0000 7fea7de00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea780d2710 0x7fea7814c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.352+0000 7fea7de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815ddf0 0x7fea780c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.352+0000 7fea7de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea7815e1d0 0x7fea780c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.352+0000 7fea7de00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea780c87a0 con 0x7fea7815ddf0 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.352+0000 7fea7de00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7814ccf0 con 0x7fea780d2710 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.352+0000 7fea7de00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea780c7b90 con 0x7fea7815e1d0 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea76000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815ddf0 0x7fea780c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea76000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815ddf0 0x7fea780c7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44664/0 (socket says 172.21.15.12:44664) 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea76000700 1 -- 172.21.15.12:0/1486032551 learned_addr learned my addr 172.21.15.12:0/1486032551 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea76a00700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea780d2710 0x7fea7814c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea77400700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea7815e1d0 0x7fea780c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea76000700 1 -- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea7815e1d0 msgr2=0x7fea780c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea76000700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea7815e1d0 0x7fea780c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea76000700 1 -- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea780d2710 msgr2=0x7fea7814c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea76000700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea780d2710 0x7fea7814c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea76000700 1 -- 172.21.15.12:0/1486032551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea64009550 con 0x7fea7815ddf0 2024-03-29T10:14:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea76000700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815ddf0 0x7fea780c7020 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fea64000c00 tx=0x7fea64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.353+0000 7fea6f400700 1 -- 172.21.15.12:0/1486032551 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64004540 con 0x7fea7815ddf0 2024-03-29T10:14:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.354+0000 7fea6f400700 1 -- 172.21.15.12:0/1486032551 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea640036a0 con 0x7fea7815ddf0 2024-03-29T10:14:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.354+0000 7fea6f400700 1 -- 172.21.15.12:0/1486032551 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64003a50 con 0x7fea7815ddf0 2024-03-29T10:14:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.354+0000 7fea7de00700 1 -- 172.21.15.12:0/1486032551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea780c7e70 con 0x7fea7815ddf0 2024-03-29T10:14:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.354+0000 7fea7de00700 1 -- 172.21.15.12:0/1486032551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea780c8430 con 0x7fea7815ddf0 2024-03-29T10:14:14.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.355+0000 7fea7de00700 1 -- 172.21.15.12:0/1486032551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea780c85f0 con 0x7fea7815ddf0 2024-03-29T10:14:14.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.356+0000 7fea6f400700 1 -- 172.21.15.12:0/1486032551 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fea64021070 con 0x7fea7815ddf0 2024-03-29T10:14:14.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.356+0000 7fea6f400700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea5806adb0 0x7fea5806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:14.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.357+0000 7fea6f400700 1 -- 172.21.15.12:0/1486032551 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6045+0+0 (secure 0 0 0) 0x7fea6408abe0 con 0x7fea7815ddf0 2024-03-29T10:14:14.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.357+0000 7fea76a00700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea5806adb0 0x7fea5806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:14.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.358+0000 7fea76a00700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea5806adb0 0x7fea5806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea68006fd0 tx=0x7fea68008040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:14.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.361+0000 7fea6f400700 1 -- 172.21.15.12:0/1486032551 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea64055c70 con 0x7fea7815ddf0 2024-03-29T10:14:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:14.578+0000 7fea7de00700 1 -- 172.21.15.12:0/1486032551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fea78067720 con 0x7fea7815ddf0 2024-03-29T10:14:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.270+0000 7fea6f400700 1 -- 172.21.15.12:0/1486032551 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v131) v1 ==== 169+0+0 (secure 0 0 0) 0x7fea640290d0 con 0x7fea7815ddf0 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.273+0000 7fea7de00700 1 -- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea5806adb0 msgr2=0x7fea5806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.273+0000 7fea7de00700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea5806adb0 0x7fea5806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea68006fd0 tx=0x7fea68008040 comp rx=0 tx=0).stop 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.273+0000 7fea7de00700 1 -- 172.21.15.12:0/1486032551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815ddf0 msgr2=0x7fea780c7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.273+0000 7fea7de00700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815ddf0 0x7fea780c7020 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fea64000c00 tx=0x7fea64004750 comp rx=0 tx=0).stop 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.273+0000 7fea7de00700 1 -- 172.21.15.12:0/1486032551 shutdown_connections 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.273+0000 7fea7de00700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fea5806adb0 0x7fea5806d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.273+0000 7fea7de00700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea7815e1d0 0x7fea780c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.273+0000 7fea7de00700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea780d2710 0x7fea7814c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.273+0000 7fea7de00700 1 --2- 172.21.15.12:0/1486032551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea7815ddf0 0x7fea780c7020 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.273+0000 7fea7de00700 1 -- 172.21.15.12:0/1486032551 >> 172.21.15.12:0/1486032551 conn(0x7fea7808aff0 msgr2=0x7fea780b1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.274+0000 7fea7de00700 1 -- 172.21.15.12:0/1486032551 shutdown_connections 2024-03-29T10:14:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.274+0000 7fea7de00700 1 -- 172.21.15.12:0/1486032551 wait complete. 2024-03-29T10:14:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T10:14:15.295 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T10:14:15.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.514+0000 7f7ac3600700 1 Processor -- start 2024-03-29T10:14:15.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.515+0000 7f7ac3600700 1 -- start start 2024-03-29T10:14:15.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7ac3600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7abc157f90 0x7f7abc0b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7ac3600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7abc0b3b30 0x7f7abc161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7ac3600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7abc0b3070 con 0x7f7abc1613e0 2024-03-29T10:14:15.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7ac3600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7abc158370 con 0x7f7abc0b3b30 2024-03-29T10:14:15.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7ac3600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7abc0b2710 con 0x7f7abc157f90 2024-03-29T10:14:15.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7abbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7abc1613e0 0x7f7abc0b3420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42622/0 (socket says 172.21.15.12:42622) 2024-03-29T10:14:15.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7abbe00700 1 -- 172.21.15.12:0/910170263 learned_addr learned my addr 172.21.15.12:0/910170263 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:15.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7ac1600700 1 --2- 172.21.15.12:0/910170263 >> v2:172.21.15.12:3300/0 conn(0x7f7abc0b3b30 0x7f7abc161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7ac0c00700 1 --2- 172.21.15.12:0/910170263 >> v2:172.21.15.159:3300/0 conn(0x7f7abc157f90 0x7f7abc0b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7abb400700 1 -- 172.21.15.12:0/910170263 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278475054 0 0) 0x7f7aa8001110 con 0x7f7abc1613e0 2024-03-29T10:14:15.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.516+0000 7f7abb400700 1 -- 172.21.15.12:0/910170263 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa40035b0 con 0x7f7abc1613e0 2024-03-29T10:14:15.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.517+0000 7f7ac1600700 1 -- 172.21.15.12:0/910170263 >> v2:172.21.15.159:3300/0 conn(0x7f7abc157f90 msgr2=0x7f7abc0b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:15.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.517+0000 7f7ac1600700 1 --2- 172.21.15.12:0/910170263 >> v2:172.21.15.159:3300/0 conn(0x7f7abc157f90 0x7f7abc0b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.517+0000 7f7ac1600700 1 -- 172.21.15.12:0/910170263 >> v1:172.21.15.12:6789/0 conn(0x7f7abc1613e0 legacy=0x7f7abc0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.517+0000 7f7ac1600700 1 -- 172.21.15.12:0/910170263 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc0b2070 con 0x7f7abc0b3b30 2024-03-29T10:14:15.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.517+0000 7f7ac1600700 1 --2- 172.21.15.12:0/910170263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 0x7f7abc161890 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f7abc06fd80 tx=0x7f7ab400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3408464d468b941 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:15.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.517+0000 7f7abb400700 1 -- 172.21.15.12:0/910170263 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab400e070 con 0x7f7abc0b3b30 2024-03-29T10:14:15.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.517+0000 7f7abb400700 1 -- 172.21.15.12:0/910170263 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab400bb20 con 0x7f7abc0b3b30 2024-03-29T10:14:15.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.517+0000 7f7abb400700 1 -- 172.21.15.12:0/910170263 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4012430 con 0x7f7abc0b3b30 2024-03-29T10:14:15.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.518+0000 7f7ac3600700 1 -- 172.21.15.12:0/910170263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 msgr2=0x7f7abc161890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.518+0000 7f7ac3600700 1 --2- 172.21.15.12:0/910170263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 0x7f7abc161890 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f7abc06fd80 tx=0x7f7ab400b330 comp rx=0 tx=0).stop 2024-03-29T10:14:15.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.518+0000 7f7ac3600700 1 -- 172.21.15.12:0/910170263 shutdown_connections 2024-03-29T10:14:15.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.518+0000 7f7ac3600700 1 --2- 172.21.15.12:0/910170263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 0x7f7abc161890 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.518+0000 7f7ac3600700 1 --2- 172.21.15.12:0/910170263 >> v2:172.21.15.159:3300/0 conn(0x7f7abc157f90 0x7f7abc0b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.518+0000 7f7ac3600700 1 -- 172.21.15.12:0/910170263 >> 172.21.15.12:0/910170263 conn(0x7f7abc0b0d70 msgr2=0x7f7abc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:15.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.518+0000 7f7ac3600700 1 -- 172.21.15.12:0/910170263 shutdown_connections 2024-03-29T10:14:15.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.518+0000 7f7ac3600700 1 -- 172.21.15.12:0/910170263 wait complete. 2024-03-29T10:14:15.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.520+0000 7f7ac3600700 1 Processor -- start 2024-03-29T10:14:15.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.520+0000 7f7ac3600700 1 -- start start 2024-03-29T10:14:15.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac3600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 0x7f7abc14dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac3600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7abc157f90 0x7f7abc143490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac3600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7abc1613e0 0x7f7abc152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac3600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc1439d0 con 0x7f7abc0b3b30 2024-03-29T10:14:15.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac3600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc151720 con 0x7f7abc157f90 2024-03-29T10:14:15.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac3600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc1519e0 con 0x7f7abc1613e0 2024-03-29T10:14:15.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 0x7f7abc14dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 0x7f7abc14dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44694/0 (socket says 172.21.15.12:44694) 2024-03-29T10:14:15.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac0c00700 1 -- 172.21.15.12:0/2155180108 learned_addr learned my addr 172.21.15.12:0/2155180108 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:15.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac1600700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7abc1613e0 0x7f7abc152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7abbe00700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7abc157f90 0x7f7abc143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac0c00700 1 -- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7abc1613e0 msgr2=0x7f7abc152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac0c00700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7abc1613e0 0x7f7abc152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac0c00700 1 -- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7abc157f90 msgr2=0x7f7abc143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac0c00700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7abc157f90 0x7f7abc143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.521+0000 7f7ac0c00700 1 -- 172.21.15.12:0/2155180108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab4009a70 con 0x7f7abc0b3b30 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.522+0000 7f7ac0c00700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 0x7f7abc14dc20 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000e870 tx=0x7f7ab000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.522+0000 7f7ab8c00700 1 -- 172.21.15.12:0/2155180108 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0009590 con 0x7f7abc0b3b30 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.522+0000 7f7ab8c00700 1 -- 172.21.15.12:0/2155180108 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab000ee70 con 0x7f7abc0b3b30 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.522+0000 7f7ac3600700 1 -- 172.21.15.12:0/2155180108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7abc151cc0 con 0x7f7abc0b3b30 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.522+0000 7f7ab8c00700 1 -- 172.21.15.12:0/2155180108 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab000ca50 con 0x7f7abc0b3b30 2024-03-29T10:14:15.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.522+0000 7f7ac3600700 1 -- 172.21.15.12:0/2155180108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7abc152280 con 0x7f7abc0b3b30 2024-03-29T10:14:15.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.523+0000 7f7ac3600700 1 -- 172.21.15.12:0/2155180108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7abc066ff0 con 0x7f7abc0b3b30 2024-03-29T10:14:15.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.524+0000 7f7ab8c00700 1 -- 172.21.15.12:0/2155180108 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7ab000cbb0 con 0x7f7abc0b3b30 2024-03-29T10:14:15.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.525+0000 7f7ab8c00700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a9c06adb0 0x7f7a9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.525+0000 7f7ab8c00700 1 -- 172.21.15.12:0/2155180108 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f7ab0089400 con 0x7f7abc0b3b30 2024-03-29T10:14:15.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.526+0000 7f7abbe00700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a9c06adb0 0x7f7a9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.527+0000 7f7abbe00700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a9c06adb0 0x7f7a9c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7abc157ad0 tx=0x7f7aa8009040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:15.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.530+0000 7f7ab8c00700 1 -- 172.21.15.12:0/2155180108 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7ab0054520 con 0x7f7abc0b3b30 2024-03-29T10:14:15.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.750+0000 7f7ac3600700 1 -- 172.21.15.12:0/2155180108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f7abc067720 con 0x7f7abc0b3b30 2024-03-29T10:14:15.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.755+0000 7f7ab8c00700 1 -- 172.21.15.12:0/2155180108 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v43) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7ab000ce60 con 0x7f7abc0b3b30 2024-03-29T10:14:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.757+0000 7f7ac3600700 1 -- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a9c06adb0 msgr2=0x7f7a9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.757+0000 7f7ac3600700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a9c06adb0 0x7f7a9c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7abc157ad0 tx=0x7f7aa8009040 comp rx=0 tx=0).stop 2024-03-29T10:14:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.757+0000 7f7ac3600700 1 -- 172.21.15.12:0/2155180108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 msgr2=0x7f7abc14dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.757+0000 7f7ac3600700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 0x7f7abc14dc20 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000e870 tx=0x7f7ab000ec30 comp rx=0 tx=0).stop 2024-03-29T10:14:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.757+0000 7f7ac3600700 1 -- 172.21.15.12:0/2155180108 shutdown_connections 2024-03-29T10:14:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.757+0000 7f7ac3600700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a9c06adb0 0x7f7a9c06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.757+0000 7f7ac3600700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7abc1613e0 0x7f7abc152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.757+0000 7f7ac3600700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7abc0b3b30 0x7f7abc14dc20 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.757+0000 7f7ac3600700 1 --2- 172.21.15.12:0/2155180108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7abc157f90 0x7f7abc143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.758+0000 7f7ac3600700 1 -- 172.21.15.12:0/2155180108 >> 172.21.15.12:0/2155180108 conn(0x7f7abc0b0d70 msgr2=0x7f7abc0b0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:15.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.758+0000 7f7ac3600700 1 -- 172.21.15.12:0/2155180108 shutdown_connections 2024-03-29T10:14:15.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.758+0000 7f7ac3600700 1 -- 172.21.15.12:0/2155180108 wait complete. 2024-03-29T10:14:15.775 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T10:14:15.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.975+0000 7fa9a2400700 1 Processor -- start 2024-03-29T10:14:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.976+0000 7fa9a2400700 1 -- start start 2024-03-29T10:14:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.977+0000 7fa9a2400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa99c1611b0 0x7fa99c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.977+0000 7fa9a2400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0ce270 con 0x7fa99c15f4d0 2024-03-29T10:14:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.977+0000 7fa9a2400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c15b600 con 0x7fa99c0cee10 2024-03-29T10:14:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.977+0000 7fa9a2400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99c0d2760 con 0x7fa99c1611b0 2024-03-29T10:14:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.977+0000 7fa99b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa99c1611b0 0x7fa99c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.977+0000 7fa99aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa99c15f4d0 0x7fa99c0ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42632/0 (socket says 172.21.15.12:42632) 2024-03-29T10:14:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.977+0000 7fa99aa00700 1 -- 172.21.15.12:0/2092080394 learned_addr learned my addr 172.21.15.12:0/2092080394 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:15.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.977+0000 7fa99b400700 1 -- 172.21.15.12:0/2092080394 >> v1:172.21.15.159:6789/0 conn(0x7fa99c0cee10 legacy=0x7fa99c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:15.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.977+0000 7fa99b400700 1 -- 172.21.15.12:0/2092080394 >> v1:172.21.15.12:6789/0 conn(0x7fa99c15f4d0 legacy=0x7fa99c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.977+0000 7fa99b400700 1 -- 172.21.15.12:0/2092080394 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c0d2a40 con 0x7fa99c1611b0 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.978+0000 7fa99b400700 1 --2- 172.21.15.12:0/2092080394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 0x7fa99c15ef90 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009840 tx=0x7fa98c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ccd0081dab929e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.978+0000 7fa99a000700 1 -- 172.21.15.12:0/2092080394 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c004230 con 0x7fa99c1611b0 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.978+0000 7fa99a000700 1 -- 172.21.15.12:0/2092080394 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c004390 con 0x7fa99c1611b0 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.978+0000 7fa99a000700 1 -- 172.21.15.12:0/2092080394 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0036a0 con 0x7fa99c1611b0 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.978+0000 7fa9a2400700 1 -- 172.21.15.12:0/2092080394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 msgr2=0x7fa99c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.978+0000 7fa9a2400700 1 --2- 172.21.15.12:0/2092080394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 0x7fa99c15ef90 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009840 tx=0x7fa98c009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.979+0000 7fa9a2400700 1 -- 172.21.15.12:0/2092080394 shutdown_connections 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.979+0000 7fa9a2400700 1 --2- 172.21.15.12:0/2092080394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 0x7fa99c15ef90 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.979+0000 7fa9a2400700 1 -- 172.21.15.12:0/2092080394 >> 172.21.15.12:0/2092080394 conn(0x7fa99c08aff0 msgr2=0x7fa99c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.979+0000 7fa9a2400700 1 -- 172.21.15.12:0/2092080394 shutdown_connections 2024-03-29T10:14:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.979+0000 7fa9a2400700 1 -- 172.21.15.12:0/2092080394 wait complete. 2024-03-29T10:14:15.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.980+0000 7fa9a2400700 1 Processor -- start 2024-03-29T10:14:15.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.981+0000 7fa9a2400700 1 -- start start 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.981+0000 7fa9a2400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa99c0cee10 0x7fa99c0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.981+0000 7fa9a2400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa99c15f4d0 0x7fa99c0cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.981+0000 7fa9a2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 0x7fa99c157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.981+0000 7fa9a2400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0cc530 con 0x7fa99c1611b0 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.981+0000 7fa9a2400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0cc810 con 0x7fa99c0cee10 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.981+0000 7fa9a2400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0c9210 con 0x7fa99c15f4d0 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 0x7fa99c157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 0x7fa99c157120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44708/0 (socket says 172.21.15.12:44708) 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99be00700 1 -- 172.21.15.12:0/1358605810 learned_addr learned my addr 172.21.15.12:0/1358605810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99b400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa99c0cee10 0x7fa99c0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99aa00700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa99c15f4d0 0x7fa99c0cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99be00700 1 -- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa99c15f4d0 msgr2=0x7fa99c0cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99be00700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa99c15f4d0 0x7fa99c0cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99be00700 1 -- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa99c0cee10 msgr2=0x7fa99c0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99be00700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa99c0cee10 0x7fa99c0c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99be00700 1 -- 172.21.15.12:0/1358605810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c0094a0 con 0x7fa99c1611b0 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99aa00700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa99c15f4d0 0x7fa99c0cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.982+0000 7fa99b400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa99c0cee10 0x7fa99c0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.983+0000 7fa99be00700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 0x7fa99c157120 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa99400b970 tx=0x7fa99400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.983+0000 7fa993400700 1 -- 172.21.15.12:0/1358605810 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99400c9b0 con 0x7fa99c1611b0 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.983+0000 7fa993400700 1 -- 172.21.15.12:0/1358605810 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99400cb10 con 0x7fa99c1611b0 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.983+0000 7fa9a2400700 1 -- 172.21.15.12:0/1358605810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c0c94f0 con 0x7fa99c1611b0 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.983+0000 7fa993400700 1 -- 172.21.15.12:0/1358605810 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994014440 con 0x7fa99c1611b0 2024-03-29T10:14:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.983+0000 7fa9a2400700 1 -- 172.21.15.12:0/1358605810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c0cc220 con 0x7fa99c1611b0 2024-03-29T10:14:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.984+0000 7fa9a2400700 1 -- 172.21.15.12:0/1358605810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c0c9070 con 0x7fa99c1611b0 2024-03-29T10:14:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.985+0000 7fa993400700 1 -- 172.21.15.12:0/1358605810 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa99401d3b0 con 0x7fa99c1611b0 2024-03-29T10:14:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.986+0000 7fa993400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa97c06adb0 0x7fa97c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.986+0000 7fa993400700 1 -- 172.21.15.12:0/1358605810 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fa994089110 con 0x7fa99c1611b0 2024-03-29T10:14:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.987+0000 7fa99b400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa97c06adb0 0x7fa97c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:15.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.987+0000 7fa99b400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa97c06adb0 0x7fa97c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000c00 tx=0x7fa98c01c040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:15.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:15.991+0000 7fa993400700 1 -- 172.21.15.12:0/1358605810 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa99c0c9070 con 0x7fa99c1611b0 2024-03-29T10:14:16.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.204+0000 7fa9a2400700 1 -- 172.21.15.12:0/1358605810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa99c0c9730 con 0x7fa99c1611b0 2024-03-29T10:14:16.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.211+0000 7fa993400700 1 -- 172.21.15.12:0/1358605810 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa99c0c9730 con 0x7fa99c1611b0 2024-03-29T10:14:16.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 -- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa97c06adb0 msgr2=0x7fa97c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa97c06adb0 0x7fa97c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000c00 tx=0x7fa98c01c040 comp rx=0 tx=0).stop 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 -- 172.21.15.12:0/1358605810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 msgr2=0x7fa99c157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 0x7fa99c157120 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa99400b970 tx=0x7fa99400bd30 comp rx=0 tx=0).stop 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 -- 172.21.15.12:0/1358605810 shutdown_connections 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa97c06adb0 0x7fa97c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa99c1611b0 0x7fa99c157120 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa99c0cee10 0x7fa99c0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 --2- 172.21.15.12:0/1358605810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa99c15f4d0 0x7fa99c0cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 -- 172.21.15.12:0/1358605810 >> 172.21.15.12:0/1358605810 conn(0x7fa99c08aff0 msgr2=0x7fa99c1608b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 -- 172.21.15.12:0/1358605810 shutdown_connections 2024-03-29T10:14:16.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.213+0000 7fa9a2400700 1 -- 172.21.15.12:0/1358605810 wait complete. 2024-03-29T10:14:16.230 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T10:14:16.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.433+0000 7f67e0400700 1 Processor -- start 2024-03-29T10:14:16.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.433+0000 7f67e0400700 1 -- start start 2024-03-29T10:14:16.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.434+0000 7f67e0400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f67d815a5c0 0x7f67d80709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.434+0000 7f67e0400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d815dfe0 0x7f67d80ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.434+0000 7f67e0400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d80cb310 con 0x7f67d80ce670 2024-03-29T10:14:16.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.434+0000 7f67e0400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d80ceb00 con 0x7f67d815a5c0 2024-03-29T10:14:16.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.434+0000 7f67e0400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8070f00 con 0x7f67d815dfe0 2024-03-29T10:14:16.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67dd000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d815dfe0 0x7f67d80ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67dd000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d815dfe0 0x7f67d80ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:42722/0 (socket says 172.21.15.12:42722) 2024-03-29T10:14:16.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67dd000700 1 -- 172.21.15.12:0/3875531646 learned_addr learned my addr 172.21.15.12:0/3875531646 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:16.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67de400700 1 --1- 172.21.15.12:0/3875531646 >> v1:172.21.15.159:6789/0 conn(0x7f67d80ce670 0x7f67d80b2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:52230/0 (socket says 172.21.15.12:52230) 2024-03-29T10:14:16.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67dda00700 1 --2- 172.21.15.12:0/3875531646 >> v2:172.21.15.159:3300/0 conn(0x7f67d815a5c0 0x7f67d80709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67dd000700 1 -- 172.21.15.12:0/3875531646 >> v2:172.21.15.159:3300/0 conn(0x7f67d815a5c0 msgr2=0x7f67d80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:16.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67dd000700 1 --2- 172.21.15.12:0/3875531646 >> v2:172.21.15.159:3300/0 conn(0x7f67d815a5c0 0x7f67d80709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67dd000700 1 -- 172.21.15.12:0/3875531646 >> v1:172.21.15.159:6789/0 conn(0x7f67d80ce670 legacy=0x7f67d80b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67d7e00700 1 -- 172.21.15.12:0/3875531646 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136277676 0 0) 0x7f67d80cb310 con 0x7f67d80ce670 2024-03-29T10:14:16.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67dd000700 1 -- 172.21.15.12:0/3875531646 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d80cca90 con 0x7f67d815dfe0 2024-03-29T10:14:16.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.435+0000 7f67dd000700 1 --2- 172.21.15.12:0/3875531646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d815dfe0 0x7f67d80ce130 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f67c40098f0 tx=0x7f67c4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12a1e6e83bf60d79 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:16.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67d7e00700 1 -- 172.21.15.12:0/3875531646 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4011830 con 0x7f67d815dfe0 2024-03-29T10:14:16.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67d7e00700 1 -- 172.21.15.12:0/3875531646 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c4011990 con 0x7f67d815dfe0 2024-03-29T10:14:16.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67d7e00700 1 -- 172.21.15.12:0/3875531646 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4011d20 con 0x7f67d815dfe0 2024-03-29T10:14:16.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67e0400700 1 -- 172.21.15.12:0/3875531646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d815dfe0 msgr2=0x7f67d80ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67e0400700 1 --2- 172.21.15.12:0/3875531646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d815dfe0 0x7f67d80ce130 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f67c40098f0 tx=0x7f67c4009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67e0400700 1 -- 172.21.15.12:0/3875531646 shutdown_connections 2024-03-29T10:14:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67e0400700 1 --2- 172.21.15.12:0/3875531646 >> v2:172.21.15.159:3300/0 conn(0x7f67d815a5c0 0x7f67d80709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67e0400700 1 --2- 172.21.15.12:0/3875531646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d815dfe0 0x7f67d80ce130 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67e0400700 1 -- 172.21.15.12:0/3875531646 >> 172.21.15.12:0/3875531646 conn(0x7f67d80b0d70 msgr2=0x7f67d80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67e0400700 1 -- 172.21.15.12:0/3875531646 shutdown_connections 2024-03-29T10:14:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.436+0000 7f67e0400700 1 -- 172.21.15.12:0/3875531646 wait complete. 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.438+0000 7f67e0400700 1 Processor -- start 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.438+0000 7f67e0400700 1 -- start start 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.439+0000 7f67e0400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d80ce670 0x7f67d806e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.439+0000 7f67e0400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d815a5c0 0x7f67d806d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.439+0000 7f67e0400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d815dfe0 0x7f67d80ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.439+0000 7f67e0400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d806ea10 con 0x7f67d815dfe0 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.439+0000 7f67e0400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d814f530 con 0x7f67d815a5c0 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.439+0000 7f67e0400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8159b40 con 0x7f67d80ce670 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.439+0000 7f67de400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d815dfe0 0x7f67d80ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.439+0000 7f67de400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d815dfe0 0x7f67d80ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44722/0 (socket says 172.21.15.12:44722) 2024-03-29T10:14:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.439+0000 7f67de400700 1 -- 172.21.15.12:0/3612963755 learned_addr learned my addr 172.21.15.12:0/3612963755 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.440+0000 7f67dd000700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d815a5c0 0x7f67d806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.440+0000 7f67dda00700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d80ce670 0x7f67d806e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.440+0000 7f67de400700 1 -- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d80ce670 msgr2=0x7f67d806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.440+0000 7f67de400700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d80ce670 0x7f67d806e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.440+0000 7f67de400700 1 -- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d815a5c0 msgr2=0x7f67d806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.440+0000 7f67de400700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d815a5c0 0x7f67d806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.440+0000 7f67de400700 1 -- 172.21.15.12:0/3612963755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c4009550 con 0x7f67d815dfe0 2024-03-29T10:14:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.440+0000 7f67dd000700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d815a5c0 0x7f67d806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.440+0000 7f67de400700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d815dfe0 0x7f67d80ac050 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f67d0002230 tx=0x7f67d000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.440+0000 7f67d5600700 1 -- 172.21.15.12:0/3612963755 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d000ac20 con 0x7f67d815dfe0 2024-03-29T10:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.441+0000 7f67e0400700 1 -- 172.21.15.12:0/3612963755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d814f780 con 0x7f67d815dfe0 2024-03-29T10:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.441+0000 7f67d5600700 1 -- 172.21.15.12:0/3612963755 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d000ada0 con 0x7f67d815dfe0 2024-03-29T10:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.441+0000 7f67d5600700 1 -- 172.21.15.12:0/3612963755 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d0007310 con 0x7f67d815dfe0 2024-03-29T10:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.441+0000 7f67e0400700 1 -- 172.21.15.12:0/3612963755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d814fd40 con 0x7f67d815dfe0 2024-03-29T10:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.442+0000 7f67e0400700 1 -- 172.21.15.12:0/3612963755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d80b3a80 con 0x7f67d815dfe0 2024-03-29T10:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.443+0000 7f67d5600700 1 -- 172.21.15.12:0/3612963755 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f67d0007470 con 0x7f67d815dfe0 2024-03-29T10:14:16.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.443+0000 7f67d5600700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b806adb0 0x7f67b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.443+0000 7f67d5600700 1 -- 172.21.15.12:0/3612963755 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f67d0088ff0 con 0x7f67d815dfe0 2024-03-29T10:14:16.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.444+0000 7f67dda00700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b806adb0 0x7f67b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.445+0000 7f67dda00700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b806adb0 0x7f67b806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009b70 tx=0x7f67cc008040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:16.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.448+0000 7f67d5600700 1 -- 172.21.15.12:0/3612963755 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67d00540e0 con 0x7f67d815dfe0 2024-03-29T10:14:16.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.666+0000 7f67e0400700 1 -- 172.21.15.12:0/3612963755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f67d814ff30 con 0x7f67d815dfe0 2024-03-29T10:14:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.672+0000 7f67d5600700 1 -- 172.21.15.12:0/3612963755 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v45) v1 ==== 120+0+0 (secure 0 0 0) 0x7f67d0057700 con 0x7f67d815dfe0 2024-03-29T10:14:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.674+0000 7f67e0400700 1 -- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b806adb0 msgr2=0x7f67b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.674+0000 7f67e0400700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b806adb0 0x7f67b806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009b70 tx=0x7f67cc008040 comp rx=0 tx=0).stop 2024-03-29T10:14:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.674+0000 7f67e0400700 1 -- 172.21.15.12:0/3612963755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d815dfe0 msgr2=0x7f67d80ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.674+0000 7f67e0400700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d815dfe0 0x7f67d80ac050 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f67d0002230 tx=0x7f67d000a060 comp rx=0 tx=0).stop 2024-03-29T10:14:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.675+0000 7f67e0400700 1 -- 172.21.15.12:0/3612963755 shutdown_connections 2024-03-29T10:14:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.675+0000 7f67e0400700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b806adb0 0x7f67b806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.675+0000 7f67e0400700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d815dfe0 0x7f67d80ac050 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.675+0000 7f67e0400700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d80ce670 0x7f67d806e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.675+0000 7f67e0400700 1 --2- 172.21.15.12:0/3612963755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d815a5c0 0x7f67d806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.675+0000 7f67e0400700 1 -- 172.21.15.12:0/3612963755 >> 172.21.15.12:0/3612963755 conn(0x7f67d80b0d70 msgr2=0x7f67d815e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.675+0000 7f67e0400700 1 -- 172.21.15.12:0/3612963755 shutdown_connections 2024-03-29T10:14:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.675+0000 7f67e0400700 1 -- 172.21.15.12:0/3612963755 wait complete. 2024-03-29T10:14:16.691 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-03-29T10:14:16.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.890+0000 7fc7a5c00700 1 Processor -- start 2024-03-29T10:14:16.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.890+0000 7fc7a5c00700 1 -- start start 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc7a5c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a00d18b0 0x7fc7a00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc7a5c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc7a0161590 0x7fc7a0161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc7a5c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a00ceda0 con 0x7fc7a0160f10 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc7a5c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a00d1d30 con 0x7fc7a0161590 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc7a5c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a0161f60 con 0x7fc7a00d18b0 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc79f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc7a0161590 0x7fc7a0161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc79ea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc7a0160f10 0x7fc7a00d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42642/0 (socket says 172.21.15.12:42642) 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc79ea00700 1 -- 172.21.15.12:0/3481638526 learned_addr learned my addr 172.21.15.12:0/3481638526 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc79e000700 1 --2- 172.21.15.12:0/3481638526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a00d18b0 0x7fc7a00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc79f400700 1 -- 172.21.15.12:0/3481638526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a00d18b0 msgr2=0x7fc7a00cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc79f400700 1 --2- 172.21.15.12:0/3481638526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a00d18b0 0x7fc7a00cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc79d600700 1 -- 172.21.15.12:0/3481638526 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528911840 0 0) 0x7fc7a00ceda0 con 0x7fc7a0160f10 2024-03-29T10:14:16.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc79f400700 1 -- 172.21.15.12:0/3481638526 >> v1:172.21.15.12:6789/0 conn(0x7fc7a0160f10 legacy=0x7fc7a00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc79f400700 1 -- 172.21.15.12:0/3481638526 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a01612f0 con 0x7fc7a0161590 2024-03-29T10:14:16.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.892+0000 7fc79e000700 1 --2- 172.21.15.12:0/3481638526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a00d18b0 0x7fc7a00cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:14:16.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.893+0000 7fc79f400700 1 --2- 172.21.15.12:0/3481638526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 0x7fc7a0161a20 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc798009dc0 tx=0x7fc79800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120122894900ac5e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:16.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.893+0000 7fc79d600700 1 -- 172.21.15.12:0/3481638526 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79800e070 con 0x7fc7a0161590 2024-03-29T10:14:16.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.893+0000 7fc79d600700 1 -- 172.21.15.12:0/3481638526 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79800bb90 con 0x7fc7a0161590 2024-03-29T10:14:16.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.893+0000 7fc79d600700 1 -- 172.21.15.12:0/3481638526 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798012430 con 0x7fc7a0161590 2024-03-29T10:14:16.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.893+0000 7fc7a5c00700 1 -- 172.21.15.12:0/3481638526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 msgr2=0x7fc7a0161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.893+0000 7fc7a5c00700 1 --2- 172.21.15.12:0/3481638526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 0x7fc7a0161a20 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc798009dc0 tx=0x7fc79800b330 comp rx=0 tx=0).stop 2024-03-29T10:14:16.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.893+0000 7fc7a5c00700 1 -- 172.21.15.12:0/3481638526 shutdown_connections 2024-03-29T10:14:16.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.893+0000 7fc7a5c00700 1 --2- 172.21.15.12:0/3481638526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 0x7fc7a0161a20 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.893+0000 7fc7a5c00700 1 --2- 172.21.15.12:0/3481638526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a00d18b0 0x7fc7a00cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.894+0000 7fc7a5c00700 1 -- 172.21.15.12:0/3481638526 >> 172.21.15.12:0/3481638526 conn(0x7fc7a008aff0 msgr2=0x7fc7a00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:16.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.894+0000 7fc7a5c00700 1 -- 172.21.15.12:0/3481638526 shutdown_connections 2024-03-29T10:14:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.894+0000 7fc7a5c00700 1 -- 172.21.15.12:0/3481638526 wait complete. 2024-03-29T10:14:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.895+0000 7fc7a5c00700 1 Processor -- start 2024-03-29T10:14:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.895+0000 7fc7a5c00700 1 -- start start 2024-03-29T10:14:16.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.896+0000 7fc7a5c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7a00d18b0 0x7fc7a01f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.896+0000 7fc7a5c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a0160f10 0x7fc7a01f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.896+0000 7fc7a5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 0x7fc7a01f3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.896+0000 7fc7a5c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a01f3c50 con 0x7fc7a0161590 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.896+0000 7fc7a5c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a01593b0 con 0x7fc7a00d18b0 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.896+0000 7fc7a5c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a00b62c0 con 0x7fc7a0160f10 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 0x7fc7a01f3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 0x7fc7a01f3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44742/0 (socket says 172.21.15.12:44742) 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79f400700 1 -- 172.21.15.12:0/2519617531 learned_addr learned my addr 172.21.15.12:0/2519617531 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79ea00700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7a00d18b0 0x7fc7a01f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79e000700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a0160f10 0x7fc7a01f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79f400700 1 -- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a0160f10 msgr2=0x7fc7a01f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79f400700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a0160f10 0x7fc7a01f26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79f400700 1 -- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7a00d18b0 msgr2=0x7fc7a01f3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:16.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79f400700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7a00d18b0 0x7fc7a01f3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79f400700 1 -- 172.21.15.12:0/2519617531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc798009a70 con 0x7fc7a0161590 2024-03-29T10:14:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79e000700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a0160f10 0x7fc7a01f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.897+0000 7fc79f400700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 0x7fc7a01f3db0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fc7a008c570 tx=0x7fc798004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.898+0000 7fc796a00700 1 -- 172.21.15.12:0/2519617531 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798012430 con 0x7fc7a0161590 2024-03-29T10:14:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.898+0000 7fc796a00700 1 -- 172.21.15.12:0/2519617531 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc798020070 con 0x7fc7a0161590 2024-03-29T10:14:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.898+0000 7fc796a00700 1 -- 172.21.15.12:0/2519617531 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798004280 con 0x7fc7a0161590 2024-03-29T10:14:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.898+0000 7fc7a5c00700 1 -- 172.21.15.12:0/2519617531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a00b65a0 con 0x7fc7a0161590 2024-03-29T10:14:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.898+0000 7fc7a5c00700 1 -- 172.21.15.12:0/2519617531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a00b6b60 con 0x7fc7a0161590 2024-03-29T10:14:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.899+0000 7fc7a5c00700 1 -- 172.21.15.12:0/2519617531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a0067720 con 0x7fc7a0161590 2024-03-29T10:14:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.900+0000 7fc796a00700 1 -- 172.21.15.12:0/2519617531 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc7980043e0 con 0x7fc7a0161590 2024-03-29T10:14:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.901+0000 7fc796a00700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc78006adb0 0x7fc78006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.901+0000 7fc796a00700 1 -- 172.21.15.12:0/2519617531 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fc798022070 con 0x7fc7a0161590 2024-03-29T10:14:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.901+0000 7fc79ea00700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc78006adb0 0x7fc78006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.902+0000 7fc79ea00700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc78006adb0 0x7fc78006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc790000f60 tx=0x7fc790007e70 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:16.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:16.905+0000 7fc796a00700 1 -- 172.21.15.12:0/2519617531 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc798056420 con 0x7fc7a0161590 2024-03-29T10:14:17.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.201+0000 7fc7a5c00700 1 -- 172.21.15.12:0/2519617531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fc7a00b5930 con 0x7fc7a0161590 2024-03-29T10:14:17.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.275+0000 7fc796a00700 1 -- 172.21.15.12:0/2519617531 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v742) v1 ==== 148+0+0 (secure 0 0 0) 0x7fc798059a40 con 0x7fc7a0161590 2024-03-29T10:14:17.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 -- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc78006adb0 msgr2=0x7fc78006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:17.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc78006adb0 0x7fc78006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc790000f60 tx=0x7fc790007e70 comp rx=0 tx=0).stop 2024-03-29T10:14:17.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 -- 172.21.15.12:0/2519617531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 msgr2=0x7fc7a01f3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:17.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 0x7fc7a01f3db0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fc7a008c570 tx=0x7fc798004ad0 comp rx=0 tx=0).stop 2024-03-29T10:14:17.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 -- 172.21.15.12:0/2519617531 shutdown_connections 2024-03-29T10:14:17.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc78006adb0 0x7fc78006d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:17.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7a0161590 0x7fc7a01f3db0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:17.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7a00d18b0 0x7fc7a01f3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:17.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 --2- 172.21.15.12:0/2519617531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a0160f10 0x7fc7a01f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:17.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 -- 172.21.15.12:0/2519617531 >> 172.21.15.12:0/2519617531 conn(0x7fc7a008aff0 msgr2=0x7fc7a006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:17.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 -- 172.21.15.12:0/2519617531 shutdown_connections 2024-03-29T10:14:17.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.278+0000 7fc7a5c00700 1 -- 172.21.15.12:0/2519617531 wait complete. 2024-03-29T10:14:17.295 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2024-03-29T10:14:17.295 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T10:14:17.295 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-03-29T10:14:17.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.495+0000 7fc91dc00700 1 Processor -- start 2024-03-29T10:14:17.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.495+0000 7fc91dc00700 1 -- start start 2024-03-29T10:14:17.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.496+0000 7fc91dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918160f10 0x7fc9180d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:17.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.496+0000 7fc91dc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc918161590 0x7fc918161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:17.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.496+0000 7fc91dc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9180ceda0 con 0x7fc9180cf230 2024-03-29T10:14:17.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.496+0000 7fc91dc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc918161f80 con 0x7fc918161590 2024-03-29T10:14:17.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.496+0000 7fc91dc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9181612f0 con 0x7fc918160f10 2024-03-29T10:14:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.496+0000 7fc916a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918160f10 0x7fc9180d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.496+0000 7fc916000700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc9180cf230 0x7fc9180d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:52242/0 (socket says 172.21.15.12:52242) 2024-03-29T10:14:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.496+0000 7fc916000700 1 -- 172.21.15.12:0/49989696 learned_addr learned my addr 172.21.15.12:0/49989696 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:17.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.496+0000 7fc917400700 1 --2- 172.21.15.12:0/49989696 >> v2:172.21.15.159:3300/0 conn(0x7fc918161590 0x7fc918161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:17.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.497+0000 7fc916a00700 1 -- 172.21.15.12:0/49989696 >> v2:172.21.15.159:3300/0 conn(0x7fc918161590 msgr2=0x7fc918161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:17.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.497+0000 7fc916a00700 1 --2- 172.21.15.12:0/49989696 >> v2:172.21.15.159:3300/0 conn(0x7fc918161590 0x7fc918161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:17.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.497+0000 7fc916a00700 1 -- 172.21.15.12:0/49989696 >> v1:172.21.15.159:6789/0 conn(0x7fc9180cf230 legacy=0x7fc9180d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:17.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.497+0000 7fc915600700 1 -- 172.21.15.12:0/49989696 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735917723 0 0) 0x7fc9180ceda0 con 0x7fc9180cf230 2024-03-29T10:14:17.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.497+0000 7fc916a00700 1 -- 172.21.15.12:0/49989696 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9180ce470 con 0x7fc918160f10 2024-03-29T10:14:17.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.497+0000 7fc916a00700 1 --2- 172.21.15.12:0/49989696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918160f10 0x7fc9180d18b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc908009870 tx=0x7fc908009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68d4ccf332654b04 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:17.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc915600700 1 -- 172.21.15.12:0/49989696 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908011830 con 0x7fc918160f10 2024-03-29T10:14:17.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc915600700 1 -- 172.21.15.12:0/49989696 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc908011990 con 0x7fc918160f10 2024-03-29T10:14:17.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc915600700 1 -- 172.21.15.12:0/49989696 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908011d20 con 0x7fc918160f10 2024-03-29T10:14:17.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc91dc00700 1 -- 172.21.15.12:0/49989696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918160f10 msgr2=0x7fc9180d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:17.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc91dc00700 1 --2- 172.21.15.12:0/49989696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918160f10 0x7fc9180d18b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc908009870 tx=0x7fc908009b80 comp rx=0 tx=0).stop 2024-03-29T10:14:17.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc91dc00700 1 -- 172.21.15.12:0/49989696 shutdown_connections 2024-03-29T10:14:17.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc91dc00700 1 --2- 172.21.15.12:0/49989696 >> v2:172.21.15.159:3300/0 conn(0x7fc918161590 0x7fc918161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:17.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc91dc00700 1 --2- 172.21.15.12:0/49989696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918160f10 0x7fc9180d18b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:17.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc91dc00700 1 -- 172.21.15.12:0/49989696 >> 172.21.15.12:0/49989696 conn(0x7fc91808aff0 msgr2=0x7fc9180d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:17.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc91dc00700 1 -- 172.21.15.12:0/49989696 shutdown_connections 2024-03-29T10:14:17.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.498+0000 7fc91dc00700 1 -- 172.21.15.12:0/49989696 wait complete. 2024-03-29T10:14:17.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.500+0000 7fc91dc00700 1 Processor -- start 2024-03-29T10:14:17.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.500+0000 7fc91dc00700 1 -- start start 2024-03-29T10:14:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc91dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9180cf230 0x7fc91809e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc91dc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc918160f10 0x7fc91815f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc91dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918161590 0x7fc91815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc91dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc918158e90 con 0x7fc9180cf230 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc91dc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9180b67e0 con 0x7fc918160f10 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc91dc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9180b6ac0 con 0x7fc918161590 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc916a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9180cf230 0x7fc91809e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc916a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9180cf230 0x7fc91809e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44750/0 (socket says 172.21.15.12:44750) 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc916a00700 1 -- 172.21.15.12:0/4228201172 learned_addr learned my addr 172.21.15.12:0/4228201172 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc916000700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc918160f10 0x7fc91815f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.501+0000 7fc917400700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918161590 0x7fc91815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc916a00700 1 -- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918161590 msgr2=0x7fc91815fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc916a00700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918161590 0x7fc91815fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc916a00700 1 -- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc918160f10 msgr2=0x7fc91815f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc916a00700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc918160f10 0x7fc91815f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc916a00700 1 -- 172.21.15.12:0/4228201172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9080094a0 con 0x7fc9180cf230 2024-03-29T10:14:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc916000700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc918160f10 0x7fc91815f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc916a00700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9180cf230 0x7fc91809e340 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fc908000c00 tx=0x7fc908004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc90ea00700 1 -- 172.21.15.12:0/4228201172 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9080036a0 con 0x7fc9180cf230 2024-03-29T10:14:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc90ea00700 1 -- 172.21.15.12:0/4228201172 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90801e070 con 0x7fc9180cf230 2024-03-29T10:14:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc91dc00700 1 -- 172.21.15.12:0/4228201172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9180b6da0 con 0x7fc9180cf230 2024-03-29T10:14:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc90ea00700 1 -- 172.21.15.12:0/4228201172 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908003a00 con 0x7fc9180cf230 2024-03-29T10:14:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.502+0000 7fc91dc00700 1 -- 172.21.15.12:0/4228201172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc918160120 con 0x7fc9180cf230 2024-03-29T10:14:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.503+0000 7fc91dc00700 1 -- 172.21.15.12:0/4228201172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9180b72c0 con 0x7fc9180cf230 2024-03-29T10:14:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.504+0000 7fc90ea00700 1 -- 172.21.15.12:0/4228201172 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc908003b60 con 0x7fc9180cf230 2024-03-29T10:14:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.505+0000 7fc90ea00700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc8f806adb0 0x7fc8f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.505+0000 7fc90ea00700 1 -- 172.21.15.12:0/4228201172 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fc90808a4f0 con 0x7fc9180cf230 2024-03-29T10:14:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.506+0000 7fc916000700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc8f806adb0 0x7fc8f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:17.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.506+0000 7fc916000700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc8f806adb0 0x7fc8f806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc904000f60 tx=0x7fc904007d30 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:17.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.510+0000 7fc90ea00700 1 -- 172.21.15.12:0/4228201172 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc9180b72c0 con 0x7fc9180cf230 2024-03-29T10:14:17.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:17.806+0000 7fc91dc00700 1 -- 172.21.15.12:0/4228201172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fc9180b72c0 con 0x7fc9180cf230 2024-03-29T10:14:18.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.283+0000 7fc90ea00700 1 -- 172.21.15.12:0/4228201172 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v743) v1 ==== 163+0+0 (secure 0 0 0) 0x7fc9180b72c0 con 0x7fc9180cf230 2024-03-29T10:14:18.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.285+0000 7fc91dc00700 1 -- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc8f806adb0 msgr2=0x7fc8f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:18.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.285+0000 7fc91dc00700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc8f806adb0 0x7fc8f806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc904000f60 tx=0x7fc904007d30 comp rx=0 tx=0).stop 2024-03-29T10:14:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.285+0000 7fc91dc00700 1 -- 172.21.15.12:0/4228201172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9180cf230 msgr2=0x7fc91809e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.285+0000 7fc91dc00700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9180cf230 0x7fc91809e340 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fc908000c00 tx=0x7fc908004750 comp rx=0 tx=0).stop 2024-03-29T10:14:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.285+0000 7fc91dc00700 1 -- 172.21.15.12:0/4228201172 shutdown_connections 2024-03-29T10:14:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.285+0000 7fc91dc00700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc8f806adb0 0x7fc8f806d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.285+0000 7fc91dc00700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc918161590 0x7fc91815fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.285+0000 7fc91dc00700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9180cf230 0x7fc91809e340 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.285+0000 7fc91dc00700 1 --2- 172.21.15.12:0/4228201172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc918160f10 0x7fc91815f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.285+0000 7fc91dc00700 1 -- 172.21.15.12:0/4228201172 >> 172.21.15.12:0/4228201172 conn(0x7fc91808aff0 msgr2=0x7fc91815ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.286+0000 7fc91dc00700 1 -- 172.21.15.12:0/4228201172 shutdown_connections 2024-03-29T10:14:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.286+0000 7fc91dc00700 1 -- 172.21.15.12:0/4228201172 wait complete. 2024-03-29T10:14:18.304 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T10:14:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.508+0000 7ff5eb200700 1 Processor -- start 2024-03-29T10:14:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.508+0000 7ff5eb200700 1 -- start start 2024-03-29T10:14:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.509+0000 7ff5eb200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e40d16a0 0x7ff5e40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.509+0000 7ff5eb200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff5e40d2370 0x7ff5e40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.509+0000 7ff5eb200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e4161420 con 0x7ff5e415d9d0 2024-03-29T10:14:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.509+0000 7ff5eb200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e40d2820 con 0x7ff5e40d2370 2024-03-29T10:14:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.509+0000 7ff5eb200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40d4340 con 0x7ff5e40d16a0 2024-03-29T10:14:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.509+0000 7ff5e3400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e40d16a0 0x7ff5e40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.509+0000 7ff5e3400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e40d16a0 0x7ff5e40d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:42772/0 (socket says 172.21.15.12:42772) 2024-03-29T10:14:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.509+0000 7ff5e3400700 1 -- 172.21.15.12:0/2343326096 learned_addr learned my addr 172.21.15.12:0/2343326096 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.509+0000 7ff5e9200700 1 --2- 172.21.15.12:0/2343326096 >> v2:172.21.15.159:3300/0 conn(0x7ff5e40d2370 0x7ff5e40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.510+0000 7ff5e2a00700 1 -- 172.21.15.12:0/2343326096 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147132369 0 0) 0x7ff5e4161420 con 0x7ff5e415d9d0 2024-03-29T10:14:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.510+0000 7ff5e3400700 1 -- 172.21.15.12:0/2343326096 >> v2:172.21.15.159:3300/0 conn(0x7ff5e40d2370 msgr2=0x7ff5e40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:18.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.510+0000 7ff5e3400700 1 --2- 172.21.15.12:0/2343326096 >> v2:172.21.15.159:3300/0 conn(0x7ff5e40d2370 0x7ff5e40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.510+0000 7ff5e3400700 1 -- 172.21.15.12:0/2343326096 >> v1:172.21.15.159:6789/0 conn(0x7ff5e415d9d0 legacy=0x7ff5e40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:18.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.510+0000 7ff5e3400700 1 -- 172.21.15.12:0/2343326096 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e4161700 con 0x7ff5e40d16a0 2024-03-29T10:14:18.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.510+0000 7ff5e3400700 1 --2- 172.21.15.12:0/2343326096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e40d16a0 0x7ff5e40d1e30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0006fd0 tx=0x7ff5d0009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fe66ff823e9f04d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:18.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5e2a00700 1 -- 172.21.15.12:0/2343326096 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0005e30 con 0x7ff5e40d16a0 2024-03-29T10:14:18.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5e2a00700 1 -- 172.21.15.12:0/2343326096 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0004030 con 0x7ff5e40d16a0 2024-03-29T10:14:18.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5e2a00700 1 -- 172.21.15.12:0/2343326096 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00043e0 con 0x7ff5e40d16a0 2024-03-29T10:14:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5eb200700 1 -- 172.21.15.12:0/2343326096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e40d16a0 msgr2=0x7ff5e40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5eb200700 1 --2- 172.21.15.12:0/2343326096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e40d16a0 0x7ff5e40d1e30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0006fd0 tx=0x7ff5d0009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5eb200700 1 -- 172.21.15.12:0/2343326096 shutdown_connections 2024-03-29T10:14:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5eb200700 1 --2- 172.21.15.12:0/2343326096 >> v2:172.21.15.159:3300/0 conn(0x7ff5e40d2370 0x7ff5e40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5eb200700 1 --2- 172.21.15.12:0/2343326096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e40d16a0 0x7ff5e40d1e30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5eb200700 1 -- 172.21.15.12:0/2343326096 >> 172.21.15.12:0/2343326096 conn(0x7ff5e408aff0 msgr2=0x7ff5e40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5eb200700 1 -- 172.21.15.12:0/2343326096 shutdown_connections 2024-03-29T10:14:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.511+0000 7ff5eb200700 1 -- 172.21.15.12:0/2343326096 wait complete. 2024-03-29T10:14:18.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.513+0000 7ff5eb200700 1 Processor -- start 2024-03-29T10:14:18.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.513+0000 7ff5eb200700 1 -- start start 2024-03-29T10:14:18.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5eb200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5e40d16a0 0x7ff5e40b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:18.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5eb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5e40d2370 0x7ff5e40c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:18.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5eb200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e415d9d0 0x7ff5e40c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:18.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5eb200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40cd090 con 0x7ff5e40d2370 2024-03-29T10:14:18.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5eb200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40cd370 con 0x7ff5e40d16a0 2024-03-29T10:14:18.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5eb200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40cd650 con 0x7ff5e415d9d0 2024-03-29T10:14:18.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5e3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5e40d2370 0x7ff5e40c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:18.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5e3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5e40d2370 0x7ff5e40c9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44754/0 (socket says 172.21.15.12:44754) 2024-03-29T10:14:18.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5e3400700 1 -- 172.21.15.12:0/568865680 learned_addr learned my addr 172.21.15.12:0/568865680 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:18.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5e3e00700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5e40d16a0 0x7ff5e40b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:18.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.514+0000 7ff5e9200700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e415d9d0 0x7ff5e40c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.515+0000 7ff5e3400700 1 -- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e415d9d0 msgr2=0x7ff5e40c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.515+0000 7ff5e3400700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e415d9d0 0x7ff5e40c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.515+0000 7ff5e3400700 1 -- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5e40d16a0 msgr2=0x7ff5e40b4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.515+0000 7ff5e3400700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5e40d16a0 0x7ff5e40b4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.515+0000 7ff5e3400700 1 -- 172.21.15.12:0/568865680 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d0009500 con 0x7ff5e40d2370 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.515+0000 7ff5e3400700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5e40d2370 0x7ff5e40c9410 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009c10 tx=0x7ff5d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.515+0000 7ff5dbe00700 1 -- 172.21.15.12:0/568865680 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0003c30 con 0x7ff5e40d2370 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.515+0000 7ff5dbe00700 1 -- 172.21.15.12:0/568865680 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0003d90 con 0x7ff5e40d2370 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.515+0000 7ff5dbe00700 1 -- 172.21.15.12:0/568865680 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d001d670 con 0x7ff5e40d2370 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.515+0000 7ff5eb200700 1 -- 172.21.15.12:0/568865680 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e40b60c0 con 0x7ff5e40d2370 2024-03-29T10:14:18.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.516+0000 7ff5eb200700 1 -- 172.21.15.12:0/568865680 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e40b53c0 con 0x7ff5e40d2370 2024-03-29T10:14:18.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.516+0000 7ff5eb200700 1 -- 172.21.15.12:0/568865680 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e40d1ca0 con 0x7ff5e40d2370 2024-03-29T10:14:18.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.517+0000 7ff5dbe00700 1 -- 172.21.15.12:0/568865680 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff5d001d7d0 con 0x7ff5e40d2370 2024-03-29T10:14:18.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.518+0000 7ff5dbe00700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5c406adb0 0x7ff5c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:18.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.518+0000 7ff5dbe00700 1 -- 172.21.15.12:0/568865680 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5631+0+0 (secure 0 0 0) 0x7ff5d008a810 con 0x7ff5e40d2370 2024-03-29T10:14:18.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.519+0000 7ff5e3e00700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5c406adb0 0x7ff5c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:18.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.519+0000 7ff5e3e00700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5c406adb0 0x7ff5c406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4000f60 tx=0x7ff5d4009040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.523+0000 7ff5dbe00700 1 -- 172.21.15.12:0/568865680 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff5d0055890 con 0x7ff5e40d2370 2024-03-29T10:14:18.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.733+0000 7ff5eb200700 1 -- 172.21.15.12:0/568865680 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff5e40b65e0 con 0x7ff5e40d2370 2024-03-29T10:14:18.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.738+0000 7ff5dbe00700 1 -- 172.21.15.12:0/568865680 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff5e40b65e0 con 0x7ff5e40d2370 2024-03-29T10:14:18.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.740+0000 7ff5eb200700 1 -- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5c406adb0 msgr2=0x7ff5c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.740+0000 7ff5eb200700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5c406adb0 0x7ff5c406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4000f60 tx=0x7ff5d4009040 comp rx=0 tx=0).stop 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.740+0000 7ff5eb200700 1 -- 172.21.15.12:0/568865680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5e40d2370 msgr2=0x7ff5e40c9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.740+0000 7ff5eb200700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5e40d2370 0x7ff5e40c9410 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009c10 tx=0x7ff5d0004750 comp rx=0 tx=0).stop 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.740+0000 7ff5eb200700 1 -- 172.21.15.12:0/568865680 shutdown_connections 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.740+0000 7ff5eb200700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff5c406adb0 0x7ff5c406d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.740+0000 7ff5eb200700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5e415d9d0 0x7ff5e40c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.740+0000 7ff5eb200700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5e40d16a0 0x7ff5e40b4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.740+0000 7ff5eb200700 1 --2- 172.21.15.12:0/568865680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5e40d2370 0x7ff5e40c9410 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.741+0000 7ff5eb200700 1 -- 172.21.15.12:0/568865680 >> 172.21.15.12:0/568865680 conn(0x7ff5e408aff0 msgr2=0x7ff5e40cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.741+0000 7ff5eb200700 1 -- 172.21.15.12:0/568865680 shutdown_connections 2024-03-29T10:14:18.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:18.741+0000 7ff5eb200700 1 -- 172.21.15.12:0/568865680 wait complete. 2024-03-29T10:14:18.757 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:14:18.757 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:14:18.854 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:14:18.949 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:14:19.351 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:14:19.351 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:19.399 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:14:19.399 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:14:19.399 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:19.451 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:14:19.451 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:14:19.451 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:19.497 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:14:19.497 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:14:19.497 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:19.547 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:14:19.547 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T10:14:19.547 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:19.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.746+0000 7f76d8c00700 1 Processor -- start 2024-03-29T10:14:19.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.747+0000 7f76d8c00700 1 -- start start 2024-03-29T10:14:19.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.748+0000 7f76d8c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f76d40b27f0 0x7f76d40ac960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:19.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.748+0000 7f76d8c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d40ac4d0 con 0x7f76d4160c90 2024-03-29T10:14:19.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.748+0000 7f76d8c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d4071e50 con 0x7f76d40718e0 2024-03-29T10:14:19.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.748+0000 7f76d8c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d4072130 con 0x7f76d40b27f0 2024-03-29T10:14:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.748+0000 7f76d1e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f76d4160c90 0x7f76d40b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49234/0 (socket says 172.21.15.12:49234) 2024-03-29T10:14:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.748+0000 7f76d1e00700 1 -- 172.21.15.12:0/2915100957 learned_addr learned my addr 172.21.15.12:0/2915100957 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772285478 0 0) 0x7f76d40ac4d0 con 0x7f76d4160c90 2024-03-29T10:14:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b80035b0 con 0x7f76d4160c90 2024-03-29T10:14:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d1400700 1 --2- 172.21.15.12:0/2915100957 >> v2:172.21.15.159:3300/0 conn(0x7f76d40b27f0 0x7f76d40ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3509393489 0 0) 0x7f76b80035b0 con 0x7f76d4160c90 2024-03-29T10:14:19.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 >> v2:172.21.15.159:3300/0 conn(0x7f76d40b27f0 msgr2=0x7f76d40ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 --2- 172.21.15.12:0/2915100957 >> v2:172.21.15.159:3300/0 conn(0x7f76d40b27f0 0x7f76d40ac960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 >> v1:172.21.15.159:6789/0 conn(0x7f76d40718e0 legacy=0x7f76d40acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4072410 con 0x7f76d4160c90 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f76c4001670 con 0x7f76d4160c90 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 >> v1:172.21.15.12:6789/0 conn(0x7f76d4160c90 legacy=0x7f76d40b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8003a50 0x7f76b8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8004850 0x7f76b8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8005260 0x7f76b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d4072130 con 0x7f76b8004850 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b800c1b0 con 0x7f76b8003a50 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b800c370 con 0x7f76b8005260 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d1400700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8004850 0x7f76b8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:19.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.749+0000 7f76d1e00700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8003a50 0x7f76b8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d2800700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8005260 0x7f76b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d1400700 1 -- 172.21.15.12:0/2915100957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8005260 msgr2=0x7f76b800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d1400700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76b8005260 0x7f76b800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d1400700 1 -- 172.21.15.12:0/2915100957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8003a50 msgr2=0x7f76b8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d1400700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8003a50 0x7f76b8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d1400700 1 -- 172.21.15.12:0/2915100957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b80035b0 con 0x7f76b8004850 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d2800700 1 -- 172.21.15.12:0/2915100957 reap_dead start 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d1400700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8004850 0x7f76b8004cd0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f76c0009890 tx=0x7f76b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70e7550f039b916 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c000c070 con 0x7f76b8004850 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c0013610 con 0x7f76b8004850 2024-03-29T10:14:19.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.750+0000 7f76d0a00700 1 -- 172.21.15.12:0/2915100957 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c00139a0 con 0x7f76b8004850 2024-03-29T10:14:19.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.751+0000 7f76d8c00700 1 -- 172.21.15.12:0/2915100957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8004850 msgr2=0x7f76b8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:19.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.751+0000 7f76d8c00700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8004850 0x7f76b8004cd0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f76c0009890 tx=0x7f76b8005880 comp rx=0 tx=0).stop 2024-03-29T10:14:19.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.751+0000 7f76d8c00700 1 -- 172.21.15.12:0/2915100957 shutdown_connections 2024-03-29T10:14:19.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.751+0000 7f76d8c00700 1 --2- 172.21.15.12:0/2915100957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76b8004850 0x7f76b8004cd0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:19.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.751+0000 7f76d8c00700 1 -- 172.21.15.12:0/2915100957 >> 172.21.15.12:0/2915100957 conn(0x7f76d40cf190 msgr2=0x7f76d40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:19.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.751+0000 7f76d8c00700 1 -- 172.21.15.12:0/2915100957 shutdown_connections 2024-03-29T10:14:19.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.751+0000 7f76d8c00700 1 -- 172.21.15.12:0/2915100957 wait complete. 2024-03-29T10:14:19.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.753+0000 7f76d8c00700 1 Processor -- start 2024-03-29T10:14:19.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.754+0000 7f76d8c00700 1 -- start start 2024-03-29T10:14:19.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d8c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8004850 0x7f76d414ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:19.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d414bd10 0x7f76d414c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:19.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d8c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76d409b4c0 0x7f76d409b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:19.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d8c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d414c6d0 con 0x7f76d414bd10 2024-03-29T10:14:19.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d8c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d409cde0 con 0x7f76b8004850 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d8c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d409c260 con 0x7f76d409b4c0 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d414bd10 0x7f76d414c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d414bd10 0x7f76d414c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50388/0 (socket says 172.21.15.12:50388) 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d1400700 1 -- 172.21.15.12:0/3331869092 learned_addr learned my addr 172.21.15.12:0/3331869092 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d1e00700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8004850 0x7f76d414ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.755+0000 7f76d2800700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76d409b4c0 0x7f76d409b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76d1400700 1 -- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76d409b4c0 msgr2=0x7f76d409b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76d1400700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76d409b4c0 0x7f76d409b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76d1400700 1 -- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8004850 msgr2=0x7f76d414ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76d1400700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8004850 0x7f76d414ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76d1400700 1 -- 172.21.15.12:0/3331869092 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d409c540 con 0x7f76d414bd10 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76d1400700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d414bd10 0x7f76d414c190 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f76c0016040 tx=0x7f76c00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76ca000700 1 -- 172.21.15.12:0/3331869092 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c0004460 con 0x7f76d414bd10 2024-03-29T10:14:19.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76ca000700 1 -- 172.21.15.12:0/3331869092 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c0013490 con 0x7f76d414bd10 2024-03-29T10:14:19.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76d8c00700 1 -- 172.21.15.12:0/3331869092 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d409c820 con 0x7f76d414bd10 2024-03-29T10:14:19.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76ca000700 1 -- 172.21.15.12:0/3331869092 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c00038c0 con 0x7f76d414bd10 2024-03-29T10:14:19.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.756+0000 7f76d8c00700 1 -- 172.21.15.12:0/3331869092 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d409be80 con 0x7f76d414bd10 2024-03-29T10:14:19.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.758+0000 7f76d8c00700 1 -- 172.21.15.12:0/3331869092 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d414c9d0 con 0x7f76d414bd10 2024-03-29T10:14:19.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.758+0000 7f76ca000700 1 -- 172.21.15.12:0/3331869092 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f76c0003a20 con 0x7f76d414bd10 2024-03-29T10:14:19.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.759+0000 7f76ca000700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b406adb0 0x7f76b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:19.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.760+0000 7f76ca000700 1 -- 172.21.15.12:0/3331869092 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f76c008a5b0 con 0x7f76d414bd10 2024-03-29T10:14:19.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.760+0000 7f76d1e00700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b406adb0 0x7f76b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:19.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.761+0000 7f76d1e00700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b406adb0 0x7f76b406d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f76c4001050 tx=0x7f76c4000fc0 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:19.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:19.764+0000 7f76ca000700 1 -- 172.21.15.12:0/3331869092 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76d414c9d0 con 0x7f76d414bd10 2024-03-29T10:14:20.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.025+0000 7f76d8c00700 1 -- 172.21.15.12:0/3331869092 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76d414c9d0 con 0x7f76d414bd10 2024-03-29T10:14:20.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.025+0000 7f76ca000700 1 -- 172.21.15.12:0/3331869092 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f76c0017de0 con 0x7f76d414bd10 2024-03-29T10:14:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:20.025 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":57,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7645,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/553963486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":553963486},{"type":"v1","addr":"172.21.15.159:6837","nonce":553963486}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3051067707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3051067707},{"type":"v1","addr":"172.21.15.159:6836","nonce":3051067707}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3629410472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3629410472},{"type":"v1","addr":"172.21.15.12:6833","nonce":3629410472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":8328,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2726220416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2726220416},{"type":"v1","addr":"172.21.15.12:6835","nonce":2726220416}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[]} 2024-03-29T10:14:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 -- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b406adb0 msgr2=0x7f76b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b406adb0 0x7f76b406d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f76c4001050 tx=0x7f76c4000fc0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 -- 172.21.15.12:0/3331869092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d414bd10 msgr2=0x7f76d414c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d414bd10 0x7f76d414c190 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f76c0016040 tx=0x7f76c00047c0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 -- 172.21.15.12:0/3331869092 shutdown_connections 2024-03-29T10:14:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76b406adb0 0x7f76b406d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76d409b4c0 0x7f76d409b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76b8004850 0x7f76d414ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 --2- 172.21.15.12:0/3331869092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76d414bd10 0x7f76d414c190 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 -- 172.21.15.12:0/3331869092 >> 172.21.15.12:0/3331869092 conn(0x7f76d40cf190 msgr2=0x7f76d40ae170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 -- 172.21.15.12:0/3331869092 shutdown_connections 2024-03-29T10:14:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.028+0000 7f76d8c00700 1 -- 172.21.15.12:0/3331869092 wait complete. 2024-03-29T10:14:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 57 2024-03-29T10:14:20.045 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T10:14:20.045 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T10:14:20.045 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T10:14:20.045 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:14:20.045 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T10:14:20.046 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T10:14:20.046 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T10:14:20.046 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:14:20.046 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T10:14:20.046 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T10:14:20.046 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T10:14:20.046 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:14:20.047 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T10:14:20.047 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T10:14:20.047 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T10:14:20.047 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:14:20.060 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T10:14:20.060 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T10:14:20.061 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T10:14:20.061 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T10:14:20.067 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T10:14:20.067 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T10:14:20.068 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T10:14:20.069 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T10:14:20.105 INFO:tasks.ceph.mds.a:Started 2024-03-29T10:14:20.105 DEBUG:teuthology.parallel:result is None 2024-03-29T10:14:20.107 INFO:tasks.ceph.mds.c:Started 2024-03-29T10:14:20.107 DEBUG:teuthology.parallel:result is None 2024-03-29T10:14:20.112 INFO:tasks.ceph.mds.b:Started 2024-03-29T10:14:20.112 DEBUG:teuthology.parallel:result is None 2024-03-29T10:14:20.115 INFO:tasks.ceph.mds.d:Started 2024-03-29T10:14:20.115 DEBUG:teuthology.parallel:result is None 2024-03-29T10:14:20.116 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T10:14:20.257 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T10:14:20.257 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:14:20.261 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T10:14:20.261 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T10:14:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.335+0000 7f053d800700 1 Processor -- start 2024-03-29T10:14:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.335+0000 7f053d800700 1 -- start start 2024-03-29T10:14:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.336+0000 7f053d800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05380d21f0 0x7f05380ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.336+0000 7f053d800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05380cec40 0x7f0538161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.336+0000 7f053d800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05380ce270 con 0x7f05381611b0 2024-03-29T10:14:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.336+0000 7f053d800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053815b600 con 0x7f05380d21f0 2024-03-29T10:14:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.336+0000 7f053d800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05380d2760 con 0x7f05380cec40 2024-03-29T10:14:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.336+0000 7f0536000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05380d21f0 0x7f05380ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.336+0000 7f0536000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05380d21f0 0x7f05380ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50438/0 (socket says 172.21.15.12:50438) 2024-03-29T10:14:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.336+0000 7f0536000700 1 -- 172.21.15.12:0/584755408 learned_addr learned my addr 172.21.15.12:0/584755408 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.336+0000 7f0536a00700 1 --1- 172.21.15.12:0/584755408 >> v1:172.21.15.12:6789/0 conn(0x7f05381611b0 0x7f053815ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49242/0 (socket says 172.21.15.12:49242) 2024-03-29T10:14:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.336+0000 7f0537400700 1 --2- 172.21.15.12:0/584755408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05380cec40 0x7f0538161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.337+0000 7f0536000700 1 -- 172.21.15.12:0/584755408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05380cec40 msgr2=0x7f0538161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.337+0000 7f0536000700 1 --2- 172.21.15.12:0/584755408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05380cec40 0x7f0538161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.337+0000 7f0536000700 1 -- 172.21.15.12:0/584755408 >> v1:172.21.15.12:6789/0 conn(0x7f05381611b0 legacy=0x7f053815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.337+0000 7f0536000700 1 -- 172.21.15.12:0/584755408 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05380d2a40 con 0x7f05380d21f0 2024-03-29T10:14:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.337+0000 7f0535600700 1 -- 172.21.15.12:0/584755408 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215466934 0 0) 0x7f05380ce270 con 0x7f05381611b0 2024-03-29T10:14:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.337+0000 7f0536000700 1 --2- 172.21.15.12:0/584755408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380d21f0 0x7f05380ce700 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f05240098f0 tx=0x7f0524009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb1ab4fd40846f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.337+0000 7f0535600700 1 -- 172.21.15.12:0/584755408 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524011830 con 0x7f05380d21f0 2024-03-29T10:14:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.337+0000 7f0535600700 1 -- 172.21.15.12:0/584755408 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0524011990 con 0x7f05380d21f0 2024-03-29T10:14:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.337+0000 7f0535600700 1 -- 172.21.15.12:0/584755408 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524011d20 con 0x7f05380d21f0 2024-03-29T10:14:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.338+0000 7f053d800700 1 -- 172.21.15.12:0/584755408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380d21f0 msgr2=0x7f05380ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.338+0000 7f053d800700 1 --2- 172.21.15.12:0/584755408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380d21f0 0x7f05380ce700 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f05240098f0 tx=0x7f0524009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.338+0000 7f053d800700 1 -- 172.21.15.12:0/584755408 shutdown_connections 2024-03-29T10:14:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.338+0000 7f053d800700 1 --2- 172.21.15.12:0/584755408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05380cec40 0x7f0538161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.338+0000 7f053d800700 1 --2- 172.21.15.12:0/584755408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380d21f0 0x7f05380ce700 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.338+0000 7f053d800700 1 -- 172.21.15.12:0/584755408 >> 172.21.15.12:0/584755408 conn(0x7f053808aff0 msgr2=0x7f05380d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.338+0000 7f053d800700 1 -- 172.21.15.12:0/584755408 shutdown_connections 2024-03-29T10:14:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.338+0000 7f053d800700 1 -- 172.21.15.12:0/584755408 wait complete. 2024-03-29T10:14:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.340+0000 7f053d800700 1 Processor -- start 2024-03-29T10:14:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.340+0000 7f053d800700 1 -- start start 2024-03-29T10:14:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f053d800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380cec40 0x7f05380b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f053d800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05380d21f0 0x7f05381f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f053d800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05381611b0 0x7f0538158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f053d800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05380cd6a0 con 0x7f05380cec40 2024-03-29T10:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f053d800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05380b70a0 con 0x7f05381611b0 2024-03-29T10:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f053d800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05380ca600 con 0x7f05380d21f0 2024-03-29T10:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f0536a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380cec40 0x7f05380b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f0536a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380cec40 0x7f05380b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50446/0 (socket says 172.21.15.12:50446) 2024-03-29T10:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f0536a00700 1 -- 172.21.15.12:0/4104002087 learned_addr learned my addr 172.21.15.12:0/4104002087 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f0536000700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05380d21f0 0x7f05381f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f0537400700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05381611b0 0x7f0538158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f0536a00700 1 -- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05380d21f0 msgr2=0x7f05381f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f0536a00700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05380d21f0 0x7f05381f34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f0536a00700 1 -- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05381611b0 msgr2=0x7f0538158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f0536a00700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05381611b0 0x7f0538158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.341+0000 7f0536a00700 1 -- 172.21.15.12:0/4104002087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0524009550 con 0x7f05380cec40 2024-03-29T10:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.342+0000 7f0536a00700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380cec40 0x7f05380b6b60 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f052800c550 tx=0x7f052800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.342+0000 7f052ea00700 1 -- 172.21.15.12:0/4104002087 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528004c80 con 0x7f05380cec40 2024-03-29T10:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.342+0000 7f052ea00700 1 -- 172.21.15.12:0/4104002087 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0528004e00 con 0x7f05380cec40 2024-03-29T10:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.342+0000 7f053d800700 1 -- 172.21.15.12:0/4104002087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05380ca850 con 0x7f05380cec40 2024-03-29T10:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.342+0000 7f052ea00700 1 -- 172.21.15.12:0/4104002087 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528007310 con 0x7f05380cec40 2024-03-29T10:14:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.342+0000 7f053d800700 1 -- 172.21.15.12:0/4104002087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05380cabe0 con 0x7f05380cec40 2024-03-29T10:14:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.343+0000 7f053d800700 1 -- 172.21.15.12:0/4104002087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05380d30b0 con 0x7f05380cec40 2024-03-29T10:14:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.344+0000 7f052ea00700 1 -- 172.21.15.12:0/4104002087 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0528007470 con 0x7f05380cec40 2024-03-29T10:14:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.345+0000 7f052ea00700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f051806adb0 0x7f051806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.345+0000 7f052ea00700 1 -- 172.21.15.12:0/4104002087 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f0528088f30 con 0x7f05380cec40 2024-03-29T10:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.346+0000 7f0536000700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f051806adb0 0x7f051806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.346+0000 7f0536000700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f051806adb0 0x7f051806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0524009840 tx=0x7f0524008030 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.350+0000 7f052ea00700 1 -- 172.21.15.12:0/4104002087 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f05280540e0 con 0x7f05380cec40 2024-03-29T10:14:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:20.578+0000 7f053d800700 1 -- 172.21.15.12:0/4104002087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f05380cf270 con 0x7f05380cec40 2024-03-29T10:14:21.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.309+0000 7f052ea00700 1 -- 172.21.15.12:0/4104002087 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v136) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0528057700 con 0x7f05380cec40 2024-03-29T10:14:21.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 -- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f051806adb0 msgr2=0x7f051806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:21.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f051806adb0 0x7f051806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0524009840 tx=0x7f0524008030 comp rx=0 tx=0).stop 2024-03-29T10:14:21.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 -- 172.21.15.12:0/4104002087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380cec40 msgr2=0x7f05380b6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:21.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380cec40 0x7f05380b6b60 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f052800c550 tx=0x7f052800c860 comp rx=0 tx=0).stop 2024-03-29T10:14:21.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 -- 172.21.15.12:0/4104002087 shutdown_connections 2024-03-29T10:14:21.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f051806adb0 0x7f051806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05381611b0 0x7f0538158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05380cec40 0x7f05380b6b60 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 --2- 172.21.15.12:0/4104002087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05380d21f0 0x7f05381f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 -- 172.21.15.12:0/4104002087 >> 172.21.15.12:0/4104002087 conn(0x7f053808aff0 msgr2=0x7f0538159ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:21.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 -- 172.21.15.12:0/4104002087 shutdown_connections 2024-03-29T10:14:21.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.312+0000 7f053d800700 1 -- 172.21.15.12:0/4104002087 wait complete. 2024-03-29T10:14:21.312 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T10:14:21.332 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T10:14:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.534+0000 7f58f5e00700 1 Processor -- start 2024-03-29T10:14:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.534+0000 7f58f5e00700 1 -- start start 2024-03-29T10:14:21.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.535+0000 7f58f5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58f0161ca0 0x7f58f00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:21.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.535+0000 7f58f5e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f00ceda0 con 0x7f58f0160f10 2024-03-29T10:14:21.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.535+0000 7f58f5e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f01612f0 con 0x7f58f00d18b0 2024-03-29T10:14:21.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.535+0000 7f58f5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00d1d30 con 0x7f58f0161ca0 2024-03-29T10:14:21.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58eea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f58f0160f10 0x7f58f00d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49258/0 (socket says 172.21.15.12:49258) 2024-03-29T10:14:21.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58eea00700 1 -- 172.21.15.12:0/1022123080 learned_addr learned my addr 172.21.15.12:0/1022123080 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:21.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ef400700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58f0161ca0 0x7f58f00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:21.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241566708 0 0) 0x7f58f00ceda0 con 0x7f58f0160f10 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d40035b0 con 0x7f58f0160f10 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 967152130 0 0) 0x7f58d40035b0 con 0x7f58f0160f10 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58f0161ca0 msgr2=0x7f58f00cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ed600700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58f0161ca0 0x7f58f00cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 >> v1:172.21.15.159:6789/0 conn(0x7f58f00d18b0 legacy=0x7f58f0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f00ce470 con 0x7f58f0160f10 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f58e0001670 con 0x7f58f0160f10 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 >> v1:172.21.15.12:6789/0 conn(0x7f58f0160f10 legacy=0x7f58f00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.536+0000 7f58ed600700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58d4003a30 0x7f58d4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ed600700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d4004850 0x7f58d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ed600700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58f0161ca0 0x7f58d40052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00d1d30 con 0x7f58d4004850 2024-03-29T10:14:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d400bc00 con 0x7f58d4003a30 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d400bdc0 con 0x7f58f0161ca0 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ee000700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d4004850 0x7f58d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ef400700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58f0161ca0 0x7f58d40052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58eea00700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58d4003a30 0x7f58d4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ee000700 1 -- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58f0161ca0 msgr2=0x7f58d40052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ee000700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58f0161ca0 0x7f58d40052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ee000700 1 -- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58d4003a30 msgr2=0x7f58d4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ee000700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58d4003a30 0x7f58d4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ee000700 1 -- 172.21.15.12:0/1022123080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d40035b0 con 0x7f58d4004850 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.537+0000 7f58ee000700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d4004850 0x7f58d4004cd0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f58dc007fd0 tx=0x7f58d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27fe8caed1b9de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.538+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc0012f0 con 0x7f58d4004850 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.538+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58dc0052f0 con 0x7f58d4004850 2024-03-29T10:14:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.538+0000 7f58ed600700 1 -- 172.21.15.12:0/1022123080 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc006860 con 0x7f58d4004850 2024-03-29T10:14:21.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.538+0000 7f58f5e00700 1 -- 172.21.15.12:0/1022123080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d4004850 msgr2=0x7f58d4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:21.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.538+0000 7f58f5e00700 1 --2- 172.21.15.12:0/1022123080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d4004850 0x7f58d4004cd0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f58dc007fd0 tx=0x7f58d4005880 comp rx=0 tx=0).stop 2024-03-29T10:14:21.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.538+0000 7f58ef400700 1 -- 172.21.15.12:0/1022123080 reap_dead start 2024-03-29T10:14:21.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.538+0000 7f58f5e00700 1 -- 172.21.15.12:0/1022123080 shutdown_connections 2024-03-29T10:14:21.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.538+0000 7f58f5e00700 1 -- 172.21.15.12:0/1022123080 >> 172.21.15.12:0/1022123080 conn(0x7f58f008aff0 msgr2=0x7f58f00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:21.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.538+0000 7f58f5e00700 1 -- 172.21.15.12:0/1022123080 shutdown_connections 2024-03-29T10:14:21.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.538+0000 7f58f5e00700 1 -- 172.21.15.12:0/1022123080 wait complete. 2024-03-29T10:14:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.540+0000 7f58f5e00700 1 Processor -- start 2024-03-29T10:14:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.540+0000 7f58f5e00700 1 -- start start 2024-03-29T10:14:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.541+0000 7f58f5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4004850 0x7f58f015f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.541+0000 7f58f5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58f00b8ea0 0x7f58f00b4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.541+0000 7f58f5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58f00b4580 0x7f58f01f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.541+0000 7f58f5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00b4a30 con 0x7f58f00b4580 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.541+0000 7f58f5e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f015fca0 con 0x7f58f00b8ea0 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.541+0000 7f58f5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00b8170 con 0x7f58d4004850 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58ef400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58f00b4580 0x7f58f01f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58ef400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58f00b4580 0x7f58f01f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50456/0 (socket says 172.21.15.12:50456) 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58ef400700 1 -- 172.21.15.12:0/642409553 learned_addr learned my addr 172.21.15.12:0/642409553 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58eea00700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4004850 0x7f58f015f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58ee000700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58f00b8ea0 0x7f58f00b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58ef400700 1 -- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4004850 msgr2=0x7f58f015f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58ef400700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4004850 0x7f58f015f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58ef400700 1 -- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58f00b8ea0 msgr2=0x7f58f00b4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58ef400700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58f00b8ea0 0x7f58f00b4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58ef400700 1 -- 172.21.15.12:0/642409553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f01f3fa0 con 0x7f58f00b4580 2024-03-29T10:14:21.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58eea00700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4004850 0x7f58f015f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:21.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.542+0000 7f58ef400700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58f00b4580 0x7f58f01f3250 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f58f008d550 tx=0x7f58e80114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:21.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.543+0000 7f58e7400700 1 -- 172.21.15.12:0/642409553 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e800e070 con 0x7f58f00b4580 2024-03-29T10:14:21.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.543+0000 7f58e7400700 1 -- 172.21.15.12:0/642409553 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e8003cf0 con 0x7f58f00b4580 2024-03-29T10:14:21.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.543+0000 7f58e7400700 1 -- 172.21.15.12:0/642409553 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8003ea0 con 0x7f58f00b4580 2024-03-29T10:14:21.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.543+0000 7f58f5e00700 1 -- 172.21.15.12:0/642409553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f00b83c0 con 0x7f58f00b4580 2024-03-29T10:14:21.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.543+0000 7f58f5e00700 1 -- 172.21.15.12:0/642409553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f00b8980 con 0x7f58f00b4580 2024-03-29T10:14:21.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.544+0000 7f58f5e00700 1 -- 172.21.15.12:0/642409553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f0067720 con 0x7f58f00b4580 2024-03-29T10:14:21.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.545+0000 7f58e7400700 1 -- 172.21.15.12:0/642409553 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f58e8011a10 con 0x7f58f00b4580 2024-03-29T10:14:21.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.546+0000 7f58e7400700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58d006adb0 0x7f58d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:21.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.546+0000 7f58e7400700 1 -- 172.21.15.12:0/642409553 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f58e80896d0 con 0x7f58f00b4580 2024-03-29T10:14:21.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.547+0000 7f58eea00700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58d006adb0 0x7f58d006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:21.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.547+0000 7f58eea00700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58d006adb0 0x7f58d006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f58e0005a80 tx=0x7f58e0009040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:21.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.551+0000 7f58e7400700 1 -- 172.21.15.12:0/642409553 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f58e80549d0 con 0x7f58f00b4580 2024-03-29T10:14:21.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.853+0000 7f58f5e00700 1 -- 172.21.15.12:0/642409553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f58f008b760 con 0x7f58f00b4580 2024-03-29T10:14:21.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.853+0000 7f58e7400700 1 -- 172.21.15.12:0/642409553 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f58e8057ff0 con 0x7f58f00b4580 2024-03-29T10:14:21.853 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:21.853 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:14:21.853 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T10:14:21.853 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.853 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T10:14:21.853 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T10:14:21.853 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:14:21.854 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.855 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:14:21.856 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T10:14:21.857 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:14:21.858 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T10:14:21.859 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:14:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.863 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.864 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T10:14:21.865 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:14:21.866 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:14:21.866 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:14:21.866 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:14:21.866 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.866 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:21.866 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T10:14:21.866 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:14:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 -- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58d006adb0 msgr2=0x7f58d006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58d006adb0 0x7f58d006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f58e0005a80 tx=0x7f58e0009040 comp rx=0 tx=0).stop 2024-03-29T10:14:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 -- 172.21.15.12:0/642409553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58f00b4580 msgr2=0x7f58f01f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58f00b4580 0x7f58f01f3250 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f58f008d550 tx=0x7f58e80114b0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 -- 172.21.15.12:0/642409553 shutdown_connections 2024-03-29T10:14:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58d006adb0 0x7f58d006d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58f00b4580 0x7f58f01f3250 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4004850 0x7f58f015f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 --2- 172.21.15.12:0/642409553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58f00b8ea0 0x7f58f00b4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 -- 172.21.15.12:0/642409553 >> 172.21.15.12:0/642409553 conn(0x7f58f008aff0 msgr2=0x7f58f00b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 -- 172.21.15.12:0/642409553 shutdown_connections 2024-03-29T10:14:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:21.856+0000 7f58f5e00700 1 -- 172.21.15.12:0/642409553 wait complete. 2024-03-29T10:14:21.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.073+0000 7fa17aa00700 1 Processor -- start 2024-03-29T10:14:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.073+0000 7fa17aa00700 1 -- start start 2024-03-29T10:14:22.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.074+0000 7fa17aa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa174072580 0x7fa1740b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.074+0000 7fa17aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1740b2e50 0x7fa1740709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa17aa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1740b2180 con 0x7fa1740b3af0 2024-03-29T10:14:22.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa17aa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17415b100 con 0x7fa1740b2e50 2024-03-29T10:14:22.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa17aa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17415b3e0 con 0x7fa174072580 2024-03-29T10:14:22.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa178a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1740b2e50 0x7fa1740709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa173e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa1740b3af0 0x7fa17415a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49260/0 (socket says 172.21.15.12:49260) 2024-03-29T10:14:22.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa173e00700 1 -- 172.21.15.12:0/2337235086 learned_addr learned my addr 172.21.15.12:0/2337235086 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:22.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa173400700 1 --2- 172.21.15.12:0/2337235086 >> v2:172.21.15.159:3300/0 conn(0x7fa174072580 0x7fa1740b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa172a00700 1 -- 172.21.15.12:0/2337235086 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725684690 0 0) 0x7fa1740b2180 con 0x7fa1740b3af0 2024-03-29T10:14:22.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa178a00700 1 -- 172.21.15.12:0/2337235086 >> v2:172.21.15.159:3300/0 conn(0x7fa174072580 msgr2=0x7fa1740b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:22.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa178a00700 1 --2- 172.21.15.12:0/2337235086 >> v2:172.21.15.159:3300/0 conn(0x7fa174072580 0x7fa1740b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa178a00700 1 -- 172.21.15.12:0/2337235086 >> v1:172.21.15.12:6789/0 conn(0x7fa1740b3af0 legacy=0x7fa17415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.075+0000 7fa178a00700 1 -- 172.21.15.12:0/2337235086 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1740b23f0 con 0x7fa1740b2e50 2024-03-29T10:14:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.076+0000 7fa178a00700 1 --2- 172.21.15.12:0/2337235086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 0x7fa1740709c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa17415dd90 tx=0x7fa16c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b98bf78e44e5b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.076+0000 7fa172a00700 1 -- 172.21.15.12:0/2337235086 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c00e070 con 0x7fa1740b2e50 2024-03-29T10:14:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.076+0000 7fa172a00700 1 -- 172.21.15.12:0/2337235086 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c00bbd0 con 0x7fa1740b2e50 2024-03-29T10:14:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.076+0000 7fa172a00700 1 -- 172.21.15.12:0/2337235086 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c012430 con 0x7fa1740b2e50 2024-03-29T10:14:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.076+0000 7fa17aa00700 1 -- 172.21.15.12:0/2337235086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 msgr2=0x7fa1740709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.076+0000 7fa17aa00700 1 --2- 172.21.15.12:0/2337235086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 0x7fa1740709c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa17415dd90 tx=0x7fa16c00b330 comp rx=0 tx=0).stop 2024-03-29T10:14:22.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.077+0000 7fa17aa00700 1 -- 172.21.15.12:0/2337235086 shutdown_connections 2024-03-29T10:14:22.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.077+0000 7fa17aa00700 1 --2- 172.21.15.12:0/2337235086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 0x7fa1740709c0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.077+0000 7fa17aa00700 1 --2- 172.21.15.12:0/2337235086 >> v2:172.21.15.159:3300/0 conn(0x7fa174072580 0x7fa1740b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.077+0000 7fa17aa00700 1 -- 172.21.15.12:0/2337235086 >> 172.21.15.12:0/2337235086 conn(0x7fa1740cf190 msgr2=0x7fa1740cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:22.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.077+0000 7fa17aa00700 1 -- 172.21.15.12:0/2337235086 shutdown_connections 2024-03-29T10:14:22.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.077+0000 7fa17aa00700 1 -- 172.21.15.12:0/2337235086 wait complete. 2024-03-29T10:14:22.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.078+0000 7fa17aa00700 1 Processor -- start 2024-03-29T10:14:22.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.078+0000 7fa17aa00700 1 -- start start 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.079+0000 7fa17aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa174072580 0x7fa1741f07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.079+0000 7fa17aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 0x7fa174155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.079+0000 7fa17aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1740b3af0 0x7fa1740ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.079+0000 7fa17aa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1740b0b60 con 0x7fa1740b2e50 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.079+0000 7fa17aa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1740b0880 con 0x7fa1740b3af0 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.079+0000 7fa17aa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa174154f60 con 0x7fa174072580 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 0x7fa174155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 0x7fa174155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50474/0 (socket says 172.21.15.12:50474) 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173400700 1 -- 172.21.15.12:0/3204630635 learned_addr learned my addr 172.21.15.12:0/3204630635 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa178a00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1740b3af0 0x7fa1740ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173e00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa174072580 0x7fa1741f07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173400700 1 -- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa174072580 msgr2=0x7fa1741f07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173400700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa174072580 0x7fa1741f07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173400700 1 -- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1740b3af0 msgr2=0x7fa1740ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173400700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1740b3af0 0x7fa1740ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173400700 1 -- 172.21.15.12:0/3204630635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c009a70 con 0x7fa1740b2e50 2024-03-29T10:14:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa178a00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1740b3af0 0x7fa1740ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:14:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173e00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa174072580 0x7fa1741f07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:14:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.080+0000 7fa173400700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 0x7fa174155a40 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fa16000e7e0 tx=0x7fa16000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.081+0000 7fa16be00700 1 -- 172.21.15.12:0/3204630635 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160013610 con 0x7fa1740b2e50 2024-03-29T10:14:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.081+0000 7fa16be00700 1 -- 172.21.15.12:0/3204630635 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa160013770 con 0x7fa1740b2e50 2024-03-29T10:14:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.081+0000 7fa16be00700 1 -- 172.21.15.12:0/3204630635 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160013ae0 con 0x7fa1740b2e50 2024-03-29T10:14:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.081+0000 7fa17aa00700 1 -- 172.21.15.12:0/3204630635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa174155240 con 0x7fa1740b2e50 2024-03-29T10:14:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.081+0000 7fa17aa00700 1 -- 172.21.15.12:0/3204630635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa174157590 con 0x7fa1740b2e50 2024-03-29T10:14:22.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.082+0000 7fa17aa00700 1 -- 172.21.15.12:0/3204630635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1741577b0 con 0x7fa1740b2e50 2024-03-29T10:14:22.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.083+0000 7fa16be00700 1 -- 172.21.15.12:0/3204630635 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa160013c40 con 0x7fa1740b2e50 2024-03-29T10:14:22.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.084+0000 7fa16be00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa15406adb0 0x7fa15406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.084+0000 7fa16be00700 1 -- 172.21.15.12:0/3204630635 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa160088f10 con 0x7fa1740b2e50 2024-03-29T10:14:22.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.084+0000 7fa173e00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa15406adb0 0x7fa15406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.085+0000 7fa173e00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa15406adb0 0x7fa15406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa164007fd0 tx=0x7fa164009040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:22.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.088+0000 7fa16be00700 1 -- 172.21.15.12:0/3204630635 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa160054210 con 0x7fa1740b2e50 2024-03-29T10:14:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.307+0000 7fa17aa00700 1 -- 172.21.15.12:0/3204630635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa174067720 con 0x7fa1740b2e50 2024-03-29T10:14:22.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.308+0000 7fa16be00700 1 -- 172.21.15.12:0/3204630635 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+12399 (secure 0 0 0) 0x7fa160057830 con 0x7fa1740b2e50 2024-03-29T10:14:22.307 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:22.307 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":136,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:14:21.301964+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:22.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 -- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa15406adb0 msgr2=0x7fa15406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa15406adb0 0x7fa15406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa164007fd0 tx=0x7fa164009040 comp rx=0 tx=0).stop 2024-03-29T10:14:22.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 -- 172.21.15.12:0/3204630635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 msgr2=0x7fa174155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 0x7fa174155a40 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fa16000e7e0 tx=0x7fa16000eba0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 -- 172.21.15.12:0/3204630635 shutdown_connections 2024-03-29T10:14:22.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa15406adb0 0x7fa15406d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1740b3af0 0x7fa1740ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa174072580 0x7fa1741f07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 --2- 172.21.15.12:0/3204630635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1740b2e50 0x7fa174155a40 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 -- 172.21.15.12:0/3204630635 >> 172.21.15.12:0/3204630635 conn(0x7fa1740cf190 msgr2=0x7fa17415ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 -- 172.21.15.12:0/3204630635 shutdown_connections 2024-03-29T10:14:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.311+0000 7fa17aa00700 1 -- 172.21.15.12:0/3204630635 wait complete. 2024-03-29T10:14:22.328 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:14:22.328 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:14:22.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.529+0000 7f6b25200700 1 Processor -- start 2024-03-29T10:14:22.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.530+0000 7f6b25200700 1 -- start start 2024-03-29T10:14:22.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b25200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6b201611b0 0x7f6b2015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b25200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6b200cee10 0x7f6b20161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b25200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b200ce270 con 0x7f6b2015f4d0 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b25200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b2015b600 con 0x7f6b200cee10 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b25200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b200d2760 con 0x7f6b201611b0 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b1e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6b200cee10 0x7f6b20161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b1e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6b200cee10 0x7f6b20161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50486/0 (socket says 172.21.15.12:50486) 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b1e800700 1 -- 172.21.15.12:0/3782094965 learned_addr learned my addr 172.21.15.12:0/3782094965 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b1de00700 1 --2- 172.21.15.12:0/3782094965 >> v2:172.21.15.159:3300/0 conn(0x7f6b201611b0 0x7f6b2015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b1e800700 1 -- 172.21.15.12:0/3782094965 >> v2:172.21.15.159:3300/0 conn(0x7f6b201611b0 msgr2=0x7f6b2015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b1e800700 1 --2- 172.21.15.12:0/3782094965 >> v2:172.21.15.159:3300/0 conn(0x7f6b201611b0 0x7f6b2015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.531+0000 7f6b1e800700 1 -- 172.21.15.12:0/3782094965 >> v1:172.21.15.159:6789/0 conn(0x7f6b2015f4d0 legacy=0x7f6b200ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.532+0000 7f6b1e800700 1 -- 172.21.15.12:0/3782094965 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b200d2a40 con 0x7f6b200cee10 2024-03-29T10:14:22.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.532+0000 7f6b1e800700 1 --2- 172.21.15.12:0/3782094965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 0x7f6b20161590 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f6b2008b520 tx=0x7f6b1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe6b2c7a62d1cb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.532+0000 7f6b1ca00700 1 -- 172.21.15.12:0/3782094965 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800e070 con 0x7f6b200cee10 2024-03-29T10:14:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.532+0000 7f6b1ca00700 1 -- 172.21.15.12:0/3782094965 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b18003a20 con 0x7f6b200cee10 2024-03-29T10:14:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.532+0000 7f6b1ca00700 1 -- 172.21.15.12:0/3782094965 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18003dd0 con 0x7f6b200cee10 2024-03-29T10:14:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.533+0000 7f6b25200700 1 -- 172.21.15.12:0/3782094965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 msgr2=0x7f6b20161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.533+0000 7f6b25200700 1 --2- 172.21.15.12:0/3782094965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 0x7f6b20161590 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f6b2008b520 tx=0x7f6b1800b330 comp rx=0 tx=0).stop 2024-03-29T10:14:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.533+0000 7f6b25200700 1 -- 172.21.15.12:0/3782094965 shutdown_connections 2024-03-29T10:14:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.533+0000 7f6b25200700 1 --2- 172.21.15.12:0/3782094965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 0x7f6b20161590 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.533+0000 7f6b25200700 1 --2- 172.21.15.12:0/3782094965 >> v2:172.21.15.159:3300/0 conn(0x7f6b201611b0 0x7f6b2015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.533+0000 7f6b25200700 1 -- 172.21.15.12:0/3782094965 >> 172.21.15.12:0/3782094965 conn(0x7f6b2008aff0 msgr2=0x7f6b200d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.533+0000 7f6b25200700 1 -- 172.21.15.12:0/3782094965 shutdown_connections 2024-03-29T10:14:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.533+0000 7f6b25200700 1 -- 172.21.15.12:0/3782094965 wait complete. 2024-03-29T10:14:22.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.534+0000 7f6b25200700 1 Processor -- start 2024-03-29T10:14:22.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.535+0000 7f6b25200700 1 -- start start 2024-03-29T10:14:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b25200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 0x7f6b2015a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b25200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b2015f4d0 0x7f6b200c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b25200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b201611b0 0x7f6b201602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b25200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2015ae60 con 0x7f6b200cee10 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b25200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200c7760 con 0x7f6b2015f4d0 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b25200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200c7a40 con 0x7f6b201611b0 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b1de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 0x7f6b2015a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b1de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 0x7f6b2015a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50498/0 (socket says 172.21.15.12:50498) 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b1de00700 1 -- 172.21.15.12:0/669586835 learned_addr learned my addr 172.21.15.12:0/669586835 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b1d400700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b2015f4d0 0x7f6b200c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b1e800700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b201611b0 0x7f6b201602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b1de00700 1 -- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b201611b0 msgr2=0x7f6b201602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b1de00700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b201611b0 0x7f6b201602c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b1de00700 1 -- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b2015f4d0 msgr2=0x7f6b200c7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b1de00700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b2015f4d0 0x7f6b200c7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.536+0000 7f6b1de00700 1 -- 172.21.15.12:0/669586835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b18009b30 con 0x7f6b200cee10 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.537+0000 7f6b1de00700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 0x7f6b2015a5e0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000d800 tx=0x7f6b1000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.537+0000 7f6b16000700 1 -- 172.21.15.12:0/669586835 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b100125f0 con 0x7f6b200cee10 2024-03-29T10:14:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.537+0000 7f6b16000700 1 -- 172.21.15.12:0/669586835 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b10012750 con 0x7f6b200cee10 2024-03-29T10:14:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.537+0000 7f6b16000700 1 -- 172.21.15.12:0/669586835 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10012b00 con 0x7f6b200cee10 2024-03-29T10:14:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.537+0000 7f6b25200700 1 -- 172.21.15.12:0/669586835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b200c7d20 con 0x7f6b200cee10 2024-03-29T10:14:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.537+0000 7f6b25200700 1 -- 172.21.15.12:0/669586835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b200c82e0 con 0x7f6b200cee10 2024-03-29T10:14:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.538+0000 7f6b25200700 1 -- 172.21.15.12:0/669586835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b200d0e40 con 0x7f6b200cee10 2024-03-29T10:14:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.539+0000 7f6b16000700 1 -- 172.21.15.12:0/669586835 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6b10012c60 con 0x7f6b200cee10 2024-03-29T10:14:22.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.540+0000 7f6b16000700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0006adb0 0x7f6b0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.540+0000 7f6b16000700 1 -- 172.21.15.12:0/669586835 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6b100894f0 con 0x7f6b200cee10 2024-03-29T10:14:22.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.540+0000 7f6b1d400700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0006adb0 0x7f6b0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.541+0000 7f6b1d400700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0006adb0 0x7f6b0006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b2015f8b0 tx=0x7f6b0c009040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:22.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.544+0000 7f6b16000700 1 -- 172.21.15.12:0/669586835 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6b10054910 con 0x7f6b200cee10 2024-03-29T10:14:22.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.757+0000 7f6b25200700 1 -- 172.21.15.12:0/669586835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b20067720 con 0x7f6b200cee10 2024-03-29T10:14:22.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.758+0000 7f6b16000700 1 -- 172.21.15.12:0/669586835 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v136) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b10057f30 con 0x7f6b200cee10 2024-03-29T10:14:22.756 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:14:22.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 -- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0006adb0 msgr2=0x7f6b0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0006adb0 0x7f6b0006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b2015f8b0 tx=0x7f6b0c009040 comp rx=0 tx=0).stop 2024-03-29T10:14:22.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 -- 172.21.15.12:0/669586835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 msgr2=0x7f6b2015a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 0x7f6b2015a5e0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000d800 tx=0x7f6b1000dbc0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 -- 172.21.15.12:0/669586835 shutdown_connections 2024-03-29T10:14:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0006adb0 0x7f6b0006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b201611b0 0x7f6b201602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b200cee10 0x7f6b2015a5e0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 --2- 172.21.15.12:0/669586835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b2015f4d0 0x7f6b200c7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 -- 172.21.15.12:0/669586835 >> 172.21.15.12:0/669586835 conn(0x7f6b2008aff0 msgr2=0x7f6b2006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 -- 172.21.15.12:0/669586835 shutdown_connections 2024-03-29T10:14:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.760+0000 7f6b25200700 1 -- 172.21.15.12:0/669586835 wait complete. 2024-03-29T10:14:22.777 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T10:14:22.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.977+0000 7f1148600700 1 Processor -- start 2024-03-29T10:14:22.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.977+0000 7f1148600700 1 -- start start 2024-03-29T10:14:22.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.978+0000 7f1148600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f114015b5d0 0x7f11400ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.978+0000 7f1148600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114015fd50 con 0x7f11400ac7d0 2024-03-29T10:14:22.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.978+0000 7f1148600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114015f140 con 0x7f11400cb3f0 2024-03-29T10:14:22.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.978+0000 7f1148600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114015b9b0 con 0x7f114015b5d0 2024-03-29T10:14:22.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.978+0000 7f1146600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f11400ac7d0 0x7f114006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49268/0 (socket says 172.21.15.12:49268) 2024-03-29T10:14:22.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.978+0000 7f1146600700 1 -- 172.21.15.12:0/364111727 learned_addr learned my addr 172.21.15.12:0/364111727 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.978+0000 7f1145c00700 1 --2- 172.21.15.12:0/364111727 >> v2:172.21.15.159:3300/0 conn(0x7f114015b5d0 0x7f11400ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119306365 0 0) 0x7f114015fd50 con 0x7f11400ac7d0 2024-03-29T10:14:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11280035b0 con 0x7f11400ac7d0 2024-03-29T10:14:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 356122165 0 0) 0x7f11280035b0 con 0x7f11400ac7d0 2024-03-29T10:14:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 >> v2:172.21.15.159:3300/0 conn(0x7f114015b5d0 msgr2=0x7f11400ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 --2- 172.21.15.12:0/364111727 >> v2:172.21.15.159:3300/0 conn(0x7f114015b5d0 0x7f11400ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 >> v1:172.21.15.159:6789/0 conn(0x7f11400cb3f0 legacy=0x7f11400ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114015f420 con 0x7f11400ac7d0 2024-03-29T10:14:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f11380019f0 con 0x7f11400ac7d0 2024-03-29T10:14:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 >> v1:172.21.15.12:6789/0 conn(0x7f11400ac7d0 legacy=0x7f114006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f1128003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1128004400 0x7f11280048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1128005170 0x7f112800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114015b9b0 con 0x7f1128003a00 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112800c0b0 con 0x7f1128005170 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f112800c270 con 0x7f1128004400 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.979+0000 7f1145c00700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f1128003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f1146600700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1128005170 0x7f112800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f1145200700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1128004400 0x7f11280048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f1145c00700 1 -- 172.21.15.12:0/364111727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1128004400 msgr2=0x7f11280048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f1145c00700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1128004400 0x7f11280048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f1145c00700 1 -- 172.21.15.12:0/364111727 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1128005170 msgr2=0x7f112800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f1145c00700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1128005170 0x7f112800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f1145c00700 1 -- 172.21.15.12:0/364111727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11280035b0 con 0x7f1128003a00 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f1146600700 1 -- 172.21.15.12:0/364111727 reap_dead start 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f1145c00700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f1128003de0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f1134009b60 tx=0x7f1128005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc4449375d4b1dd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113400e6c0 con 0x7f1128003a00 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113400e820 con 0x7f1128003a00 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.980+0000 7f113fe00700 1 -- 172.21.15.12:0/364111727 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113400eb90 con 0x7f1128003a00 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.981+0000 7f1148600700 1 -- 172.21.15.12:0/364111727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 msgr2=0x7f1128003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.981+0000 7f1148600700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f1128003de0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f1134009b60 tx=0x7f1128005880 comp rx=0 tx=0).stop 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.981+0000 7f1148600700 1 -- 172.21.15.12:0/364111727 shutdown_connections 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.981+0000 7f1148600700 1 --2- 172.21.15.12:0/364111727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f1128003de0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.981+0000 7f1148600700 1 -- 172.21.15.12:0/364111727 >> 172.21.15.12:0/364111727 conn(0x7f11400cf190 msgr2=0x7f11400cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.981+0000 7f1148600700 1 -- 172.21.15.12:0/364111727 shutdown_connections 2024-03-29T10:14:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.981+0000 7f1148600700 1 -- 172.21.15.12:0/364111727 wait complete. 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.983+0000 7f1148600700 1 Processor -- start 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.983+0000 7f1148600700 1 -- start start 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.984+0000 7f1148600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f11401fa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.984+0000 7f1148600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11400cdef0 0x7f1140159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.984+0000 7f1148600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114015a090 0x7f1140070a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.984+0000 7f1148600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11400ce380 con 0x7f1128003a00 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.984+0000 7f1148600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11400ce660 con 0x7f114015a090 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.984+0000 7f1148600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11400ce940 con 0x7f11400cdef0 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.984+0000 7f1145c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f11401fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.984+0000 7f1145c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f11401fa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50524/0 (socket says 172.21.15.12:50524) 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.984+0000 7f1145c00700 1 -- 172.21.15.12:0/4282892172 learned_addr learned my addr 172.21.15.12:0/4282892172 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.984+0000 7f1145200700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11400cdef0 0x7f1140159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.985+0000 7f1146600700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114015a090 0x7f1140070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.985+0000 7f1145c00700 1 -- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11400cdef0 msgr2=0x7f1140159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.985+0000 7f1145c00700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11400cdef0 0x7f1140159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.985+0000 7f1145c00700 1 -- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114015a090 msgr2=0x7f1140070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.985+0000 7f1145c00700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114015a090 0x7f1140070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.985+0000 7f1145c00700 1 -- 172.21.15.12:0/4282892172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11400cbc40 con 0x7f1128003a00 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.985+0000 7f1145c00700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f11401fa090 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f112800c080 tx=0x7f11340047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.985+0000 7f113d600700 1 -- 172.21.15.12:0/4282892172 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113400e6c0 con 0x7f1128003a00 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.985+0000 7f113d600700 1 -- 172.21.15.12:0/4282892172 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1134003ad0 con 0x7f1128003a00 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.985+0000 7f113d600700 1 -- 172.21.15.12:0/4282892172 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1134003e80 con 0x7f1128003a00 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.986+0000 7f1148600700 1 -- 172.21.15.12:0/4282892172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11400cbf20 con 0x7f1128003a00 2024-03-29T10:14:22.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.986+0000 7f1148600700 1 -- 172.21.15.12:0/4282892172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1140070f70 con 0x7f1128003a00 2024-03-29T10:14:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.987+0000 7f1148600700 1 -- 172.21.15.12:0/4282892172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11400cc0b0 con 0x7f1128003a00 2024-03-29T10:14:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.988+0000 7f113d600700 1 -- 172.21.15.12:0/4282892172 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1134013470 con 0x7f1128003a00 2024-03-29T10:14:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.988+0000 7f113d600700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f112006adb0 0x7f112006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.989+0000 7f113d600700 1 -- 172.21.15.12:0/4282892172 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f113408a700 con 0x7f1128003a00 2024-03-29T10:14:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.989+0000 7f1145200700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f112006adb0 0x7f112006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:22.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.990+0000 7f1145200700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f112006adb0 0x7f112006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f11400715c0 tx=0x7f112c009040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:22.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:22.993+0000 7f113d600700 1 -- 172.21.15.12:0/4282892172 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1134055bb0 con 0x7f1128003a00 2024-03-29T10:14:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.211+0000 7f1148600700 1 -- 172.21.15.12:0/4282892172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1140067720 con 0x7f1128003a00 2024-03-29T10:14:23.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.212+0000 7f113d600700 1 -- 172.21.15.12:0/4282892172 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v136) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1134017090 con 0x7f1128003a00 2024-03-29T10:14:23.211 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:23.211 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:14:23.211 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:23.211 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T10:14:23.211 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T10:14:23.211 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:23.211 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:14:23.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.214+0000 7f1148600700 1 -- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f112006adb0 msgr2=0x7f112006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:23.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f112006adb0 0x7f112006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f11400715c0 tx=0x7f112c009040 comp rx=0 tx=0).stop 2024-03-29T10:14:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 -- 172.21.15.12:0/4282892172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 msgr2=0x7f11401fa090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f11401fa090 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f112800c080 tx=0x7f11340047c0 comp rx=0 tx=0).stop 2024-03-29T10:14:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 -- 172.21.15.12:0/4282892172 shutdown_connections 2024-03-29T10:14:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f112006adb0 0x7f112006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114015a090 0x7f1140070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1128003a00 0x7f11401fa090 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 --2- 172.21.15.12:0/4282892172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11400cdef0 0x7f1140159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 -- 172.21.15.12:0/4282892172 >> 172.21.15.12:0/4282892172 conn(0x7f11400cf190 msgr2=0x7f114015e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 -- 172.21.15.12:0/4282892172 shutdown_connections 2024-03-29T10:14:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.215+0000 7f1148600700 1 -- 172.21.15.12:0/4282892172 wait complete. 2024-03-29T10:14:23.231 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T10:14:23.232 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T10:14:23.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.436+0000 7fb97cc00700 1 Processor -- start 2024-03-29T10:14:23.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.436+0000 7fb97cc00700 1 -- start start 2024-03-29T10:14:23.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.437+0000 7fb97cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780ced30 0x7fb9780d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:23.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.437+0000 7fb97cc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb9780d2710 0x7fb97815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:23.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.437+0000 7fb97cc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9780ce8a0 con 0x7fb97815ddf0 2024-03-29T10:14:23.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.437+0000 7fb97cc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97815ec60 con 0x7fb9780d2710 2024-03-29T10:14:23.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.437+0000 7fb97cc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9780cdc90 con 0x7fb9780ced30 2024-03-29T10:14:23.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.437+0000 7fb976800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb9780d2710 0x7fb97815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:23.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.437+0000 7fb975e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb97815ddf0 0x7fb97815e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49282/0 (socket says 172.21.15.12:49282) 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.437+0000 7fb975e00700 1 -- 172.21.15.12:0/247062925 learned_addr learned my addr 172.21.15.12:0/247062925 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb975400700 1 --2- 172.21.15.12:0/247062925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780ced30 0x7fb9780d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb976800700 1 -- 172.21.15.12:0/247062925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780ced30 msgr2=0x7fb9780d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb976800700 1 --2- 172.21.15.12:0/247062925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780ced30 0x7fb9780d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb976800700 1 -- 172.21.15.12:0/247062925 >> v1:172.21.15.12:6789/0 conn(0x7fb97815ddf0 legacy=0x7fb97815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb976800700 1 -- 172.21.15.12:0/247062925 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9780cdf70 con 0x7fb9780d2710 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb975400700 1 --2- 172.21.15.12:0/247062925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780ced30 0x7fb9780d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb974a00700 1 -- 172.21.15.12:0/247062925 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390803190 0 0) 0x7fb9780ce8a0 con 0x7fb97815ddf0 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb976800700 1 --2- 172.21.15.12:0/247062925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780d2710 0x7fb97815f0f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fb970009d90 tx=0x7fb97000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b845ef700dc05095 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb974a00700 1 -- 172.21.15.12:0/247062925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97000e070 con 0x7fb9780d2710 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb974a00700 1 -- 172.21.15.12:0/247062925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97000bbd0 con 0x7fb9780d2710 2024-03-29T10:14:23.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.438+0000 7fb974a00700 1 -- 172.21.15.12:0/247062925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970012430 con 0x7fb9780d2710 2024-03-29T10:14:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.439+0000 7fb97cc00700 1 -- 172.21.15.12:0/247062925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780d2710 msgr2=0x7fb97815f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.439+0000 7fb97cc00700 1 --2- 172.21.15.12:0/247062925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780d2710 0x7fb97815f0f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fb970009d90 tx=0x7fb97000b330 comp rx=0 tx=0).stop 2024-03-29T10:14:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.439+0000 7fb97cc00700 1 -- 172.21.15.12:0/247062925 shutdown_connections 2024-03-29T10:14:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.439+0000 7fb97cc00700 1 --2- 172.21.15.12:0/247062925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780d2710 0x7fb97815f0f0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.439+0000 7fb97cc00700 1 --2- 172.21.15.12:0/247062925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780ced30 0x7fb9780d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.439+0000 7fb97cc00700 1 -- 172.21.15.12:0/247062925 >> 172.21.15.12:0/247062925 conn(0x7fb97808aff0 msgr2=0x7fb9780d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.439+0000 7fb97cc00700 1 -- 172.21.15.12:0/247062925 shutdown_connections 2024-03-29T10:14:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.439+0000 7fb97cc00700 1 -- 172.21.15.12:0/247062925 wait complete. 2024-03-29T10:14:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.441+0000 7fb97cc00700 1 Processor -- start 2024-03-29T10:14:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.442+0000 7fb97cc00700 1 -- start start 2024-03-29T10:14:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb97cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780ced30 0x7fb978161760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb97cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780d2710 0x7fb97815f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb97cc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb97815ddf0 0x7fb9781e9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb97cc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb978160420 con 0x7fb9780ced30 2024-03-29T10:14:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb97cc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9781606c0 con 0x7fb97815ddf0 2024-03-29T10:14:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb97cc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9781609a0 con 0x7fb9780d2710 2024-03-29T10:14:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb975e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780ced30 0x7fb978161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb975e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780ced30 0x7fb978161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50540/0 (socket says 172.21.15.12:50540) 2024-03-29T10:14:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb975e00700 1 -- 172.21.15.12:0/1531139742 learned_addr learned my addr 172.21.15.12:0/1531139742 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb976800700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb97815ddf0 0x7fb9781e9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb975400700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780d2710 0x7fb97815f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.443+0000 7fb975e00700 1 -- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780d2710 msgr2=0x7fb97815f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb975e00700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780d2710 0x7fb97815f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb975e00700 1 -- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb97815ddf0 msgr2=0x7fb9781e9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb975e00700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb97815ddf0 0x7fb9781e9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb975e00700 1 -- 172.21.15.12:0/1531139742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb970009a70 con 0x7fb9780ced30 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb976800700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb97815ddf0 0x7fb9781e9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb975400700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780d2710 0x7fb97815f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb975e00700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780ced30 0x7fb978161760 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb96800c550 tx=0x7fb96800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb96e000700 1 -- 172.21.15.12:0/1531139742 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968004c80 con 0x7fb9780ced30 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb96e000700 1 -- 172.21.15.12:0/1531139742 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb968004e00 con 0x7fb9780ced30 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb97cc00700 1 -- 172.21.15.12:0/1531139742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb978160c80 con 0x7fb9780ced30 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb96e000700 1 -- 172.21.15.12:0/1531139742 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968007310 con 0x7fb9780ced30 2024-03-29T10:14:23.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.444+0000 7fb97cc00700 1 -- 172.21.15.12:0/1531139742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb978161240 con 0x7fb9780ced30 2024-03-29T10:14:23.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.445+0000 7fb97cc00700 1 -- 172.21.15.12:0/1531139742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb978161460 con 0x7fb9780ced30 2024-03-29T10:14:23.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.446+0000 7fb96e000700 1 -- 172.21.15.12:0/1531139742 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb968007470 con 0x7fb9780ced30 2024-03-29T10:14:23.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.447+0000 7fb96e000700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb95806adb0 0x7fb95806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:23.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.447+0000 7fb96e000700 1 -- 172.21.15.12:0/1531139742 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb968088f00 con 0x7fb9780ced30 2024-03-29T10:14:23.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.448+0000 7fb975400700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb95806adb0 0x7fb95806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:23.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.448+0000 7fb975400700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb95806adb0 0x7fb95806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb9780ca540 tx=0x7fb964008040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:23.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.452+0000 7fb96e000700 1 -- 172.21.15.12:0/1531139742 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb968054280 con 0x7fb9780ced30 2024-03-29T10:14:23.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:23.664+0000 7fb97cc00700 1 -- 172.21.15.12:0/1531139742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb978067720 con 0x7fb9780ced30 2024-03-29T10:14:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.319+0000 7fb96e000700 1 -- 172.21.15.12:0/1531139742 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v137) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb9680578a0 con 0x7fb9780ced30 2024-03-29T10:14:24.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.324+0000 7fb97cc00700 1 -- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb95806adb0 msgr2=0x7fb95806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:24.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.324+0000 7fb97cc00700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb95806adb0 0x7fb95806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb9780ca540 tx=0x7fb964008040 comp rx=0 tx=0).stop 2024-03-29T10:14:24.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.324+0000 7fb97cc00700 1 -- 172.21.15.12:0/1531139742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780ced30 msgr2=0x7fb978161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:24.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.324+0000 7fb97cc00700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780ced30 0x7fb978161760 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb96800c550 tx=0x7fb96800c910 comp rx=0 tx=0).stop 2024-03-29T10:14:24.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.325+0000 7fb97cc00700 1 -- 172.21.15.12:0/1531139742 shutdown_connections 2024-03-29T10:14:24.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.325+0000 7fb97cc00700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb95806adb0 0x7fb95806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:24.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.325+0000 7fb97cc00700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb97815ddf0 0x7fb9781e9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:24.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.325+0000 7fb97cc00700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9780ced30 0x7fb978161760 secure :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb96800c550 tx=0x7fb96800c910 comp rx=0 tx=0).stop 2024-03-29T10:14:24.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.325+0000 7fb97cc00700 1 --2- 172.21.15.12:0/1531139742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9780d2710 0x7fb97815f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:24.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.325+0000 7fb97cc00700 1 -- 172.21.15.12:0/1531139742 >> 172.21.15.12:0/1531139742 conn(0x7fb97808aff0 msgr2=0x7fb97806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.327+0000 7fb97cc00700 1 -- 172.21.15.12:0/1531139742 shutdown_connections 2024-03-29T10:14:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.327+0000 7fb97cc00700 1 -- 172.21.15.12:0/1531139742 wait complete. 2024-03-29T10:14:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T10:14:24.347 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T10:14:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.554+0000 7f097d400700 1 Processor -- start 2024-03-29T10:14:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.554+0000 7f097d400700 1 -- start start 2024-03-29T10:14:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.555+0000 7f097d400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f097815d9d0 0x7f09780d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.555+0000 7f097d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09780d16a0 0x7f09780d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.555+0000 7f097d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09780d2370 0x7f097815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.555+0000 7f097d400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09780d2820 con 0x7f09780d2370 2024-03-29T10:14:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.555+0000 7f097d400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0978161420 con 0x7f097815d9d0 2024-03-29T10:14:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.555+0000 7f097d400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0978161700 con 0x7f09780d16a0 2024-03-29T10:14:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.555+0000 7f0977200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09780d2370 0x7f097815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.555+0000 7f0977200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09780d2370 0x7f097815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50552/0 (socket says 172.21.15.12:50552) 2024-03-29T10:14:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.555+0000 7f0977200700 1 -- 172.21.15.12:0/3474162698 learned_addr learned my addr 172.21.15.12:0/3474162698 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.555+0000 7f0975e00700 1 --2- 172.21.15.12:0/3474162698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09780d16a0 0x7f09780d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.556+0000 7f0976800700 1 --2- 172.21.15.12:0/3474162698 >> v2:172.21.15.159:3300/0 conn(0x7f097815d9d0 0x7f09780d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.556+0000 7f0977200700 1 -- 172.21.15.12:0/3474162698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09780d16a0 msgr2=0x7f09780d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.556+0000 7f0977200700 1 --2- 172.21.15.12:0/3474162698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09780d16a0 0x7f09780d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.556+0000 7f0977200700 1 -- 172.21.15.12:0/3474162698 >> v2:172.21.15.159:3300/0 conn(0x7f097815d9d0 msgr2=0x7f09780d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.556+0000 7f0977200700 1 --2- 172.21.15.12:0/3474162698 >> v2:172.21.15.159:3300/0 conn(0x7f097815d9d0 0x7f09780d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.556+0000 7f0977200700 1 -- 172.21.15.12:0/3474162698 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09781619e0 con 0x7f09780d2370 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.556+0000 7f0977200700 1 --2- 172.21.15.12:0/3474162698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09780d2370 0x7f097815ddb0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f09780d0ff0 tx=0x7f097000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d789cba1d10c3068 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.556+0000 7f0975400700 1 -- 172.21.15.12:0/3474162698 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097000e070 con 0x7f09780d2370 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.556+0000 7f0975400700 1 -- 172.21.15.12:0/3474162698 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097000b910 con 0x7f09780d2370 2024-03-29T10:14:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.556+0000 7f0975400700 1 -- 172.21.15.12:0/3474162698 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097000bcc0 con 0x7f09780d2370 2024-03-29T10:14:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.557+0000 7f097d400700 1 -- 172.21.15.12:0/3474162698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09780d2370 msgr2=0x7f097815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.557+0000 7f097d400700 1 --2- 172.21.15.12:0/3474162698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09780d2370 0x7f097815ddb0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f09780d0ff0 tx=0x7f097000b330 comp rx=0 tx=0).stop 2024-03-29T10:14:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.557+0000 7f097d400700 1 -- 172.21.15.12:0/3474162698 shutdown_connections 2024-03-29T10:14:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.557+0000 7f097d400700 1 --2- 172.21.15.12:0/3474162698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09780d2370 0x7f097815ddb0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.557+0000 7f097d400700 1 --2- 172.21.15.12:0/3474162698 >> v2:172.21.15.159:3300/0 conn(0x7f097815d9d0 0x7f09780d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.557+0000 7f097d400700 1 --2- 172.21.15.12:0/3474162698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09780d16a0 0x7f09780d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.557+0000 7f097d400700 1 -- 172.21.15.12:0/3474162698 >> 172.21.15.12:0/3474162698 conn(0x7f097808aff0 msgr2=0x7f09780d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.557+0000 7f097d400700 1 -- 172.21.15.12:0/3474162698 shutdown_connections 2024-03-29T10:14:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.557+0000 7f097d400700 1 -- 172.21.15.12:0/3474162698 wait complete. 2024-03-29T10:14:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.559+0000 7f097d400700 1 Processor -- start 2024-03-29T10:14:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.559+0000 7f097d400700 1 -- start start 2024-03-29T10:14:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f097d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09780d16a0 0x7f097815ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f097d400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09780d2370 0x7f0978159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f097d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f097815d9d0 0x7f09781595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f097d400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09780cace0 con 0x7f097815d9d0 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f097d400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09780b9270 con 0x7f09780d2370 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f097d400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09780b8f90 con 0x7f09780d16a0 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f0977200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f097815d9d0 0x7f09781595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f0977200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f097815d9d0 0x7f09781595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50560/0 (socket says 172.21.15.12:50560) 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f0977200700 1 -- 172.21.15.12:0/439155462 learned_addr learned my addr 172.21.15.12:0/439155462 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f0975e00700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09780d2370 0x7f0978159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.560+0000 7f0976800700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09780d16a0 0x7f097815ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f0977200700 1 -- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09780d16a0 msgr2=0x7f097815ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f0977200700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09780d16a0 0x7f097815ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f0977200700 1 -- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09780d2370 msgr2=0x7f0978159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f0977200700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09780d2370 0x7f0978159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f0977200700 1 -- 172.21.15.12:0/439155462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970009b30 con 0x7f097815d9d0 2024-03-29T10:14:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f0977200700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f097815d9d0 0x7f09781595c0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f097808c230 tx=0x7f0970004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f096ea00700 1 -- 172.21.15.12:0/439155462 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970004880 con 0x7f097815d9d0 2024-03-29T10:14:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f096ea00700 1 -- 172.21.15.12:0/439155462 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09700099b0 con 0x7f097815d9d0 2024-03-29T10:14:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f096ea00700 1 -- 172.21.15.12:0/439155462 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097001c680 con 0x7f097815d9d0 2024-03-29T10:14:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f097d400700 1 -- 172.21.15.12:0/439155462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09780ca690 con 0x7f097815d9d0 2024-03-29T10:14:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.561+0000 7f097d400700 1 -- 172.21.15.12:0/439155462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0978160450 con 0x7f097815d9d0 2024-03-29T10:14:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.562+0000 7f097d400700 1 -- 172.21.15.12:0/439155462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09780d1ca0 con 0x7f097815d9d0 2024-03-29T10:14:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.564+0000 7f096ea00700 1 -- 172.21.15.12:0/439155462 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f097001c7e0 con 0x7f097815d9d0 2024-03-29T10:14:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.564+0000 7f096ea00700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f095806adb0 0x7f095806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.565+0000 7f096ea00700 1 -- 172.21.15.12:0/439155462 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f097008aec0 con 0x7f097815d9d0 2024-03-29T10:14:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.565+0000 7f0976800700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f095806adb0 0x7f095806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.566+0000 7f0976800700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f095806adb0 0x7f095806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0968009b40 tx=0x7f0968009210 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.568+0000 7f096ea00700 1 -- 172.21.15.12:0/439155462 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09700561d0 con 0x7f097815d9d0 2024-03-29T10:14:24.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:24.786+0000 7f097d400700 1 -- 172.21.15.12:0/439155462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f09780cfaa0 con 0x7f097815d9d0 2024-03-29T10:14:25.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.325+0000 7f096ea00700 1 -- 172.21.15.12:0/439155462 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v138) v1 ==== 170+0+0 (secure 0 0 0) 0x7f09700597f0 con 0x7f097815d9d0 2024-03-29T10:14:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.331+0000 7f097d400700 1 -- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f095806adb0 msgr2=0x7f095806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.331+0000 7f097d400700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f095806adb0 0x7f095806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0968009b40 tx=0x7f0968009210 comp rx=0 tx=0).stop 2024-03-29T10:14:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.331+0000 7f097d400700 1 -- 172.21.15.12:0/439155462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f097815d9d0 msgr2=0x7f09781595c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.331+0000 7f097d400700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f097815d9d0 0x7f09781595c0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f097808c230 tx=0x7f0970004ad0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.333+0000 7f097d400700 1 -- 172.21.15.12:0/439155462 shutdown_connections 2024-03-29T10:14:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.333+0000 7f097d400700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f095806adb0 0x7f095806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.333+0000 7f097d400700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f097815d9d0 0x7f09781595c0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.333+0000 7f097d400700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09780d16a0 0x7f097815ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.333+0000 7f097d400700 1 --2- 172.21.15.12:0/439155462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09780d2370 0x7f0978159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.333+0000 7f097d400700 1 -- 172.21.15.12:0/439155462 >> 172.21.15.12:0/439155462 conn(0x7f097808aff0 msgr2=0x7f09780b51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.334+0000 7f097d400700 1 -- 172.21.15.12:0/439155462 shutdown_connections 2024-03-29T10:14:25.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.335+0000 7f097d400700 1 -- 172.21.15.12:0/439155462 wait complete. 2024-03-29T10:14:25.341 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T10:14:25.362 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T10:14:25.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.564+0000 7f7015000700 1 Processor -- start 2024-03-29T10:14:25.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.565+0000 7f7015000700 1 -- start start 2024-03-29T10:14:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.566+0000 7f7015000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f70100d16a0 0x7f70100d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.566+0000 7f7015000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70100d2370 0x7f70100ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.566+0000 7f7015000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7010161420 con 0x7f701015d9d0 2024-03-29T10:14:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.566+0000 7f7015000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70100d2820 con 0x7f70100d16a0 2024-03-29T10:14:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.566+0000 7f7015000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100d4340 con 0x7f70100d2370 2024-03-29T10:14:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.566+0000 7f700d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f70100d16a0 0x7f70100d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.566+0000 7f700d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f70100d16a0 0x7f70100d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50562/0 (socket says 172.21.15.12:50562) 2024-03-29T10:14:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.566+0000 7f700d400700 1 -- 172.21.15.12:0/122501501 learned_addr learned my addr 172.21.15.12:0/122501501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.566+0000 7f700e800700 1 --2- 172.21.15.12:0/122501501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70100d2370 0x7f70100ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.567+0000 7f700d400700 1 -- 172.21.15.12:0/122501501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70100d2370 msgr2=0x7f70100ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.567+0000 7f700d400700 1 --2- 172.21.15.12:0/122501501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70100d2370 0x7f70100ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.567+0000 7f700d400700 1 -- 172.21.15.12:0/122501501 >> v1:172.21.15.159:6789/0 conn(0x7f701015d9d0 legacy=0x7f70100d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.567+0000 7f700d400700 1 -- 172.21.15.12:0/122501501 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7010161700 con 0x7f70100d16a0 2024-03-29T10:14:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.567+0000 7f700ca00700 1 -- 172.21.15.12:0/122501501 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977256779 0 0) 0x7f7010161420 con 0x7f701015d9d0 2024-03-29T10:14:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.567+0000 7f700d400700 1 --2- 172.21.15.12:0/122501501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 0x7f70100d1e30 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc006fd0 tx=0x7f6ffc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1b52ffe56377f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:25.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.567+0000 7f700ca00700 1 -- 172.21.15.12:0/122501501 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc005e30 con 0x7f70100d16a0 2024-03-29T10:14:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.567+0000 7f700ca00700 1 -- 172.21.15.12:0/122501501 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc004030 con 0x7f70100d16a0 2024-03-29T10:14:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.567+0000 7f700ca00700 1 -- 172.21.15.12:0/122501501 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0043e0 con 0x7f70100d16a0 2024-03-29T10:14:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.568+0000 7f7015000700 1 -- 172.21.15.12:0/122501501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 msgr2=0x7f70100d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.568+0000 7f7015000700 1 --2- 172.21.15.12:0/122501501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 0x7f70100d1e30 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc006fd0 tx=0x7f6ffc009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.568+0000 7f7015000700 1 -- 172.21.15.12:0/122501501 shutdown_connections 2024-03-29T10:14:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.568+0000 7f7015000700 1 --2- 172.21.15.12:0/122501501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70100d2370 0x7f70100ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.568+0000 7f7015000700 1 --2- 172.21.15.12:0/122501501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 0x7f70100d1e30 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.568+0000 7f7015000700 1 -- 172.21.15.12:0/122501501 >> 172.21.15.12:0/122501501 conn(0x7f701008aff0 msgr2=0x7f70100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.568+0000 7f7015000700 1 -- 172.21.15.12:0/122501501 shutdown_connections 2024-03-29T10:14:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.568+0000 7f7015000700 1 -- 172.21.15.12:0/122501501 wait complete. 2024-03-29T10:14:25.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.570+0000 7f7015000700 1 Processor -- start 2024-03-29T10:14:25.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.570+0000 7f7015000700 1 -- start start 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f7015000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 0x7f701015e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f7015000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70100d2370 0x7f701014f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f7015000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701015d9d0 0x7f70100b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f7015000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100b73a0 con 0x7f70100d16a0 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f7015000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701015ed70 con 0x7f70100d2370 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f7015000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100ca720 con 0x7f701015d9d0 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f700de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 0x7f701015e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f700de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 0x7f701015e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50570/0 (socket says 172.21.15.12:50570) 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f700de00700 1 -- 172.21.15.12:0/809332607 learned_addr learned my addr 172.21.15.12:0/809332607 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f700d400700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70100d2370 0x7f701014f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f700e800700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701015d9d0 0x7f70100b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:25.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f700de00700 1 -- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701015d9d0 msgr2=0x7f70100b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f700de00700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701015d9d0 0x7f70100b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f700de00700 1 -- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70100d2370 msgr2=0x7f701014f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f700de00700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70100d2370 0x7f701014f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.571+0000 7f700de00700 1 -- 172.21.15.12:0/809332607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc009500 con 0x7f70100d16a0 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.572+0000 7f700de00700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 0x7f701015e830 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f700000c550 tx=0x7f700000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.572+0000 7f7006000700 1 -- 172.21.15.12:0/809332607 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70000054e0 con 0x7f70100d16a0 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.572+0000 7f7006000700 1 -- 172.21.15.12:0/809332607 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7000003680 con 0x7f70100d16a0 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.572+0000 7f7006000700 1 -- 172.21.15.12:0/809332607 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000003a30 con 0x7f70100d16a0 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.572+0000 7f7015000700 1 -- 172.21.15.12:0/809332607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70100caa00 con 0x7f70100d16a0 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.572+0000 7f7015000700 1 -- 172.21.15.12:0/809332607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70100cafc0 con 0x7f70100d16a0 2024-03-29T10:14:25.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.573+0000 7f7015000700 1 -- 172.21.15.12:0/809332607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70100d1ca0 con 0x7f70100d16a0 2024-03-29T10:14:25.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.574+0000 7f7006000700 1 -- 172.21.15.12:0/809332607 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7000003b90 con 0x7f70100d16a0 2024-03-29T10:14:25.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.576+0000 7f7006000700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ff006adb0 0x7f6ff006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:25.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.576+0000 7f7006000700 1 -- 172.21.15.12:0/809332607 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f7000088b90 con 0x7f70100d16a0 2024-03-29T10:14:25.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.576+0000 7f700d400700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ff006adb0 0x7f6ff006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:25.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.577+0000 7f700d400700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ff006adb0 0x7f6ff006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc006fd0 tx=0x7f6ffc008070 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:25.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.580+0000 7f7006000700 1 -- 172.21.15.12:0/809332607 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7000053d30 con 0x7f70100d16a0 2024-03-29T10:14:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.844+0000 7f7015000700 1 -- 172.21.15.12:0/809332607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7010067720 con 0x7f70100d16a0 2024-03-29T10:14:25.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.852+0000 7f7006000700 1 -- 172.21.15.12:0/809332607 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v60) v1 ==== 174+0+0 (secure 0 0 0) 0x7f7000057350 con 0x7f70100d16a0 2024-03-29T10:14:25.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.857+0000 7f7015000700 1 -- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ff006adb0 msgr2=0x7f6ff006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:25.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.857+0000 7f7015000700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ff006adb0 0x7f6ff006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc006fd0 tx=0x7f6ffc008070 comp rx=0 tx=0).stop 2024-03-29T10:14:25.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.857+0000 7f7015000700 1 -- 172.21.15.12:0/809332607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 msgr2=0x7f701015e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.857+0000 7f7015000700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 0x7f701015e830 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f700000c550 tx=0x7f700000c860 comp rx=0 tx=0).stop 2024-03-29T10:14:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.857+0000 7f7015000700 1 -- 172.21.15.12:0/809332607 shutdown_connections 2024-03-29T10:14:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.857+0000 7f7015000700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ff006adb0 0x7f6ff006d270 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc006fd0 tx=0x7f6ffc008070 comp rx=0 tx=0).stop 2024-03-29T10:14:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.857+0000 7f7015000700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701015d9d0 0x7f70100b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.857+0000 7f7015000700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70100d16a0 0x7f701015e830 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.857+0000 7f7015000700 1 --2- 172.21.15.12:0/809332607 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70100d2370 0x7f701014f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.857+0000 7f7015000700 1 -- 172.21.15.12:0/809332607 >> 172.21.15.12:0/809332607 conn(0x7f701008aff0 msgr2=0x7f70100b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:25.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.858+0000 7f7015000700 1 -- 172.21.15.12:0/809332607 shutdown_connections 2024-03-29T10:14:25.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:25.858+0000 7f7015000700 1 -- 172.21.15.12:0/809332607 wait complete. 2024-03-29T10:14:25.860 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 16 and data pool 17 2024-03-29T10:14:25.891 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:26.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.090+0000 7fadb2400700 1 Processor -- start 2024-03-29T10:14:26.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.090+0000 7fadb2400700 1 -- start start 2024-03-29T10:14:26.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadb2400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fadac0d18b0 0x7fadac0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadb2400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac0ceda0 con 0x7fadac160f10 2024-03-29T10:14:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadb2400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac1612f0 con 0x7fadac161590 2024-03-29T10:14:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadb2400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadac0d1d30 con 0x7fadac0d18b0 2024-03-29T10:14:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadaaa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fadac0d18b0 0x7fadac0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadab400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fadac160f10 0x7fadac0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49288/0 (socket says 172.21.15.12:49288) 2024-03-29T10:14:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadab400700 1 -- 172.21.15.12:0/1309649513 learned_addr learned my addr 172.21.15.12:0/1309649513 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadaaa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fadac0d18b0 0x7fadac0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50572/0 (socket says 172.21.15.12:50572) 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadaaa00700 1 -- 172.21.15.12:0/1309649513 >> v1:172.21.15.159:6789/0 conn(0x7fadac161590 legacy=0x7fadac0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadaaa00700 1 -- 172.21.15.12:0/1309649513 >> v1:172.21.15.12:6789/0 conn(0x7fadac160f10 legacy=0x7fadac0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadaaa00700 1 -- 172.21.15.12:0/1309649513 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac0ce470 con 0x7fadac0d18b0 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.092+0000 7fadaa000700 1 -- 172.21.15.12:0/1309649513 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149567930 0 0) 0x7fadac0ceda0 con 0x7fadac160f10 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.093+0000 7fadaaa00700 1 --2- 172.21.15.12:0/1309649513 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 0x7fadac0cf230 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fad98009840 tx=0x7fad98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6108812f66ccc9e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.093+0000 7fadaa000700 1 -- 172.21.15.12:0/1309649513 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98011830 con 0x7fadac0d18b0 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.093+0000 7fadaa000700 1 -- 172.21.15.12:0/1309649513 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad98011990 con 0x7fadac0d18b0 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.093+0000 7fadaa000700 1 -- 172.21.15.12:0/1309649513 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98011d20 con 0x7fadac0d18b0 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.093+0000 7fadb2400700 1 -- 172.21.15.12:0/1309649513 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 msgr2=0x7fadac0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.093+0000 7fadb2400700 1 --2- 172.21.15.12:0/1309649513 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 0x7fadac0cf230 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fad98009840 tx=0x7fad98009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.093+0000 7fadb2400700 1 -- 172.21.15.12:0/1309649513 shutdown_connections 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.093+0000 7fadb2400700 1 --2- 172.21.15.12:0/1309649513 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 0x7fadac0cf230 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.094+0000 7fadb2400700 1 -- 172.21.15.12:0/1309649513 >> 172.21.15.12:0/1309649513 conn(0x7fadac08aff0 msgr2=0x7fadac0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:26.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.094+0000 7fadb2400700 1 -- 172.21.15.12:0/1309649513 shutdown_connections 2024-03-29T10:14:26.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.094+0000 7fadb2400700 1 -- 172.21.15.12:0/1309649513 wait complete. 2024-03-29T10:14:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.095+0000 7fadb2400700 1 Processor -- start 2024-03-29T10:14:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.095+0000 7fadb2400700 1 -- start start 2024-03-29T10:14:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.096+0000 7fadb2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 0x7fadac14b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.096+0000 7fadb2400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadac160f10 0x7fadac1f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.096+0000 7fadb2400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadac161590 0x7fadac0b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.096+0000 7fadb2400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac15d720 con 0x7fadac0d18b0 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.096+0000 7fadb2400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac1f4570 con 0x7fadac160f10 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.096+0000 7fadb2400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadac15cdf0 con 0x7fadac161590 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadab400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 0x7fadac14b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadab400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 0x7fadac14b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50586/0 (socket says 172.21.15.12:50586) 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadab400700 1 -- 172.21.15.12:0/4174055609 learned_addr learned my addr 172.21.15.12:0/4174055609 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadaaa00700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadac160f10 0x7fadac1f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadabe00700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadac161590 0x7fadac0b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadab400700 1 -- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadac161590 msgr2=0x7fadac0b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadab400700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadac161590 0x7fadac0b7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadab400700 1 -- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadac160f10 msgr2=0x7fadac1f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadab400700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadac160f10 0x7fadac1f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadab400700 1 -- 172.21.15.12:0/4174055609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad980094a0 con 0x7fadac0d18b0 2024-03-29T10:14:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadaaa00700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadac160f10 0x7fadac1f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fadab400700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 0x7fadac14b880 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00c520 tx=0x7fad9c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fada3400700 1 -- 172.21.15.12:0/4174055609 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c004c80 con 0x7fadac0d18b0 2024-03-29T10:14:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fada3400700 1 -- 172.21.15.12:0/4174055609 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c004e20 con 0x7fadac0d18b0 2024-03-29T10:14:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.097+0000 7fada3400700 1 -- 172.21.15.12:0/4174055609 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c007310 con 0x7fadac0d18b0 2024-03-29T10:14:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.098+0000 7fadb2400700 1 -- 172.21.15.12:0/4174055609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadac15d0d0 con 0x7fadac0d18b0 2024-03-29T10:14:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.098+0000 7fadb2400700 1 -- 172.21.15.12:0/4174055609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadac0b7850 con 0x7fadac0d18b0 2024-03-29T10:14:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.099+0000 7fadb2400700 1 -- 172.21.15.12:0/4174055609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadac0b7aa0 con 0x7fadac0d18b0 2024-03-29T10:14:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.100+0000 7fada3400700 1 -- 172.21.15.12:0/4174055609 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fad9c007470 con 0x7fadac0d18b0 2024-03-29T10:14:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.100+0000 7fada3400700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad8c06adb0 0x7fad8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.101+0000 7fada3400700 1 -- 172.21.15.12:0/4174055609 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad9c089150 con 0x7fadac0d18b0 2024-03-29T10:14:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.101+0000 7fadaaa00700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad8c06adb0 0x7fad8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.102+0000 7fadaaa00700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad8c06adb0 0x7fad8c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad980090b0 tx=0x7fad9801c040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:26.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.105+0000 7fada3400700 1 -- 172.21.15.12:0/4174055609 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fad9c0542b0 con 0x7fadac0d18b0 2024-03-29T10:14:26.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.316+0000 7fadb2400700 1 -- 172.21.15.12:0/4174055609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadac067720 con 0x7fadac0d18b0 2024-03-29T10:14:26.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.318+0000 7fada3400700 1 -- 172.21.15.12:0/4174055609 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16072 (secure 0 0 0) 0x7fad9c0578d0 con 0x7fadac0d18b0 2024-03-29T10:14:26.317 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:26.317 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":139,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:14:25.846049+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:14:23.708181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-29T10:14:24.829152+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:26.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 -- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad8c06adb0 msgr2=0x7fad8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad8c06adb0 0x7fad8c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad980090b0 tx=0x7fad9801c040 comp rx=0 tx=0).stop 2024-03-29T10:14:26.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 -- 172.21.15.12:0/4174055609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 msgr2=0x7fadac14b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 0x7fadac14b880 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00c520 tx=0x7fad9c00c8e0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 -- 172.21.15.12:0/4174055609 shutdown_connections 2024-03-29T10:14:26.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fad8c06adb0 0x7fad8c06d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadac161590 0x7fadac0b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadac0d18b0 0x7fadac14b880 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 --2- 172.21.15.12:0/4174055609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadac160f10 0x7fadac1f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 -- 172.21.15.12:0/4174055609 >> 172.21.15.12:0/4174055609 conn(0x7fadac08aff0 msgr2=0x7fadac0ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:26.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 -- 172.21.15.12:0/4174055609 shutdown_connections 2024-03-29T10:14:26.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.320+0000 7fadb2400700 1 -- 172.21.15.12:0/4174055609 wait complete. 2024-03-29T10:14:26.338 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.537+0000 7fb1ab600700 1 Processor -- start 2024-03-29T10:14:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.537+0000 7fb1ab600700 1 -- start start 2024-03-29T10:14:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.538+0000 7fb1ab600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb1a4072580 0x7fb1a40b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.538+0000 7fb1ab600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb1a40b2e50 0x7fb1a40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.538+0000 7fb1ab600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a40b2180 con 0x7fb1a40b3af0 2024-03-29T10:14:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.538+0000 7fb1ab600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a415b100 con 0x7fb1a4072580 2024-03-29T10:14:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.538+0000 7fb1ab600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a415b3e0 con 0x7fb1a40b2e50 2024-03-29T10:14:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.539+0000 7fb1a3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb1a4072580 0x7fb1a40b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.539+0000 7fb1a3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb1a4072580 0x7fb1a40b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50596/0 (socket says 172.21.15.12:50596) 2024-03-29T10:14:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.539+0000 7fb1a3e00700 1 -- 172.21.15.12:0/628285281 learned_addr learned my addr 172.21.15.12:0/628285281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.539+0000 7fb1a8c00700 1 --1- 172.21.15.12:0/628285281 >> v1:172.21.15.12:6789/0 conn(0x7fb1a40b3af0 0x7fb1a415a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49296/0 (socket says 172.21.15.12:49296) 2024-03-29T10:14:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.539+0000 7fb1a9600700 1 --2- 172.21.15.12:0/628285281 >> v2:172.21.15.159:3300/0 conn(0x7fb1a40b2e50 0x7fb1a40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.539+0000 7fb1a3e00700 1 -- 172.21.15.12:0/628285281 >> v2:172.21.15.159:3300/0 conn(0x7fb1a40b2e50 msgr2=0x7fb1a40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.539+0000 7fb1a3e00700 1 --2- 172.21.15.12:0/628285281 >> v2:172.21.15.159:3300/0 conn(0x7fb1a40b2e50 0x7fb1a40709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.539+0000 7fb1a3e00700 1 -- 172.21.15.12:0/628285281 >> v1:172.21.15.12:6789/0 conn(0x7fb1a40b3af0 legacy=0x7fb1a415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.539+0000 7fb1a3e00700 1 -- 172.21.15.12:0/628285281 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a40b23f0 con 0x7fb1a4072580 2024-03-29T10:14:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.539+0000 7fb1a3e00700 1 --2- 172.21.15.12:0/628285281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a4072580 0x7fb1a40b2910 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4158390 tx=0x7fb190009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb6eb6b7d5b7106 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.540+0000 7fb1a3400700 1 -- 172.21.15.12:0/628285281 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19000b870 con 0x7fb1a4072580 2024-03-29T10:14:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.540+0000 7fb1a3400700 1 -- 172.21.15.12:0/628285281 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19000b9d0 con 0x7fb1a4072580 2024-03-29T10:14:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.540+0000 7fb1a3400700 1 -- 172.21.15.12:0/628285281 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19000bd60 con 0x7fb1a4072580 2024-03-29T10:14:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.540+0000 7fb1ab600700 1 -- 172.21.15.12:0/628285281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a4072580 msgr2=0x7fb1a40b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.540+0000 7fb1ab600700 1 --2- 172.21.15.12:0/628285281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a4072580 0x7fb1a40b2910 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4158390 tx=0x7fb190009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.540+0000 7fb1ab600700 1 -- 172.21.15.12:0/628285281 shutdown_connections 2024-03-29T10:14:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.540+0000 7fb1ab600700 1 --2- 172.21.15.12:0/628285281 >> v2:172.21.15.159:3300/0 conn(0x7fb1a40b2e50 0x7fb1a40709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.540+0000 7fb1ab600700 1 --2- 172.21.15.12:0/628285281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a4072580 0x7fb1a40b2910 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.540+0000 7fb1ab600700 1 -- 172.21.15.12:0/628285281 >> 172.21.15.12:0/628285281 conn(0x7fb1a40cf190 msgr2=0x7fb1a40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.541+0000 7fb1ab600700 1 -- 172.21.15.12:0/628285281 shutdown_connections 2024-03-29T10:14:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.541+0000 7fb1ab600700 1 -- 172.21.15.12:0/628285281 wait complete. 2024-03-29T10:14:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.542+0000 7fb1ab600700 1 Processor -- start 2024-03-29T10:14:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.542+0000 7fb1ab600700 1 -- start start 2024-03-29T10:14:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.543+0000 7fb1ab600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1a4072580 0x7fb1a41465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.543+0000 7fb1ab600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1a40b2e50 0x7fb1a41e3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.543+0000 7fb1ab600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a40b3af0 0x7fb1a40cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.543+0000 7fb1ab600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a41e46f0 con 0x7fb1a40b3af0 2024-03-29T10:14:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.543+0000 7fb1ab600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a41e49d0 con 0x7fb1a40b2e50 2024-03-29T10:14:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.543+0000 7fb1ab600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a41e4cb0 con 0x7fb1a4072580 2024-03-29T10:14:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a40b3af0 0x7fb1a40cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a40b3af0 0x7fb1a40cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50610/0 (socket says 172.21.15.12:50610) 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a9600700 1 -- 172.21.15.12:0/3059727542 learned_addr learned my addr 172.21.15.12:0/3059727542 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a3e00700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1a40b2e50 0x7fb1a41e3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a8c00700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1a4072580 0x7fb1a41465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a9600700 1 -- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1a4072580 msgr2=0x7fb1a41465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a9600700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1a4072580 0x7fb1a41465a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a9600700 1 -- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1a40b2e50 msgr2=0x7fb1a41e3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a9600700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1a40b2e50 0x7fb1a41e3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a9600700 1 -- 172.21.15.12:0/3059727542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190009550 con 0x7fb1a40b3af0 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a3e00700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1a40b2e50 0x7fb1a41e3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a9600700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a40b3af0 0x7fb1a40cddb0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb19c007e40 tx=0x7fb19c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.544+0000 7fb1a0c00700 1 -- 172.21.15.12:0/3059727542 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c00eea0 con 0x7fb1a40b3af0 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.545+0000 7fb1a0c00700 1 -- 172.21.15.12:0/3059727542 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c014420 con 0x7fb1a40b3af0 2024-03-29T10:14:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.545+0000 7fb1a0c00700 1 -- 172.21.15.12:0/3059727542 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c0147b0 con 0x7fb1a40b3af0 2024-03-29T10:14:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.545+0000 7fb1ab600700 1 -- 172.21.15.12:0/3059727542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a41e4f90 con 0x7fb1a40b3af0 2024-03-29T10:14:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.545+0000 7fb1ab600700 1 -- 172.21.15.12:0/3059727542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a4152f20 con 0x7fb1a40b3af0 2024-03-29T10:14:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.546+0000 7fb1ab600700 1 -- 172.21.15.12:0/3059727542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a41e54b0 con 0x7fb1a40b3af0 2024-03-29T10:14:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.547+0000 7fb1a0c00700 1 -- 172.21.15.12:0/3059727542 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb19c014910 con 0x7fb1a40b3af0 2024-03-29T10:14:26.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.547+0000 7fb1a0c00700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb18406adb0 0x7fb18406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.548+0000 7fb1a0c00700 1 -- 172.21.15.12:0/3059727542 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb19c089ce0 con 0x7fb1a40b3af0 2024-03-29T10:14:26.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.548+0000 7fb1a8c00700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb18406adb0 0x7fb18406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.549+0000 7fb1a8c00700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb18406adb0 0x7fb18406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb198000f60 tx=0x7fb198007d30 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:26.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.552+0000 7fb1a0c00700 1 -- 172.21.15.12:0/3059727542 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb19c054d10 con 0x7fb1a40b3af0 2024-03-29T10:14:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.768+0000 7fb1ab600700 1 -- 172.21.15.12:0/3059727542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1a41530b0 con 0x7fb1a40b3af0 2024-03-29T10:14:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.769+0000 7fb1a0c00700 1 -- 172.21.15.12:0/3059727542 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16072 (secure 0 0 0) 0x7fb19c058330 con 0x7fb1a40b3af0 2024-03-29T10:14:26.768 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:26.768 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":139,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:14:25.846049+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:14:23.708181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-29T10:14:24.829152+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.771+0000 7fb1ab600700 1 -- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb18406adb0 msgr2=0x7fb18406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.771+0000 7fb1ab600700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb18406adb0 0x7fb18406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb198000f60 tx=0x7fb198007d30 comp rx=0 tx=0).stop 2024-03-29T10:14:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.771+0000 7fb1ab600700 1 -- 172.21.15.12:0/3059727542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a40b3af0 msgr2=0x7fb1a40cddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.771+0000 7fb1ab600700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a40b3af0 0x7fb1a40cddb0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb19c007e40 tx=0x7fb19c00e3c0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.772+0000 7fb1ab600700 1 -- 172.21.15.12:0/3059727542 shutdown_connections 2024-03-29T10:14:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.772+0000 7fb1ab600700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb18406adb0 0x7fb18406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.772+0000 7fb1ab600700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1a40b3af0 0x7fb1a40cddb0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.772+0000 7fb1ab600700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1a4072580 0x7fb1a41465a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.772+0000 7fb1ab600700 1 --2- 172.21.15.12:0/3059727542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1a40b2e50 0x7fb1a41e3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.772+0000 7fb1ab600700 1 -- 172.21.15.12:0/3059727542 >> 172.21.15.12:0/3059727542 conn(0x7fb1a40cf190 msgr2=0x7fb1a406e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.772+0000 7fb1ab600700 1 -- 172.21.15.12:0/3059727542 shutdown_connections 2024-03-29T10:14:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.772+0000 7fb1ab600700 1 -- 172.21.15.12:0/3059727542 wait complete. 2024-03-29T10:14:26.789 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T10:14:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.996+0000 7fe584c00700 1 Processor -- start 2024-03-29T10:14:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.997+0000 7fe584c00700 1 -- start start 2024-03-29T10:14:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe584c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5800cbb30 0x7fe5800cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe584c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5800cd020 0x7fe5800cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe584c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5800cc450 con 0x7fe5800b2070 2024-03-29T10:14:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe584c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5800cdae0 con 0x7fe5800cd020 2024-03-29T10:14:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe584c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800cddc0 con 0x7fe5800cbb30 2024-03-29T10:14:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe57d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5800cd020 0x7fe5800cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe57e800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe5800b2070 0x7fe58006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49310/0 (socket says 172.21.15.12:49310) 2024-03-29T10:14:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe57d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5800cd020 0x7fe5800cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50620/0 (socket says 172.21.15.12:50620) 2024-03-29T10:14:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe57e800700 1 -- 172.21.15.12:0/619852683 learned_addr learned my addr 172.21.15.12:0/619852683 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe57de00700 1 --2- 172.21.15.12:0/619852683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5800cbb30 0x7fe5800cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe57d400700 1 -- 172.21.15.12:0/619852683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5800cbb30 msgr2=0x7fe5800cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.998+0000 7fe57d400700 1 --2- 172.21.15.12:0/619852683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5800cbb30 0x7fe5800cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.999+0000 7fe57d400700 1 -- 172.21.15.12:0/619852683 >> v1:172.21.15.12:6789/0 conn(0x7fe5800b2070 legacy=0x7fe58006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.999+0000 7fe57d400700 1 -- 172.21.15.12:0/619852683 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5800ce0a0 con 0x7fe5800cd020 2024-03-29T10:14:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.999+0000 7fe57ca00700 1 -- 172.21.15.12:0/619852683 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177835349 0 0) 0x7fe5800cc450 con 0x7fe5800b2070 2024-03-29T10:14:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.999+0000 7fe57d400700 1 --2- 172.21.15.12:0/619852683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800cd020 0x7fe5800cd4a0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009840 tx=0x7fe56c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5263bdd6ddcff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.999+0000 7fe57ca00700 1 -- 172.21.15.12:0/619852683 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c011830 con 0x7fe5800cd020 2024-03-29T10:14:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.999+0000 7fe57ca00700 1 -- 172.21.15.12:0/619852683 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c011990 con 0x7fe5800cd020 2024-03-29T10:14:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:26.999+0000 7fe57ca00700 1 -- 172.21.15.12:0/619852683 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c011d20 con 0x7fe5800cd020 2024-03-29T10:14:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.000+0000 7fe584c00700 1 -- 172.21.15.12:0/619852683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800cd020 msgr2=0x7fe5800cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.000+0000 7fe584c00700 1 --2- 172.21.15.12:0/619852683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800cd020 0x7fe5800cd4a0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009840 tx=0x7fe56c009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.000+0000 7fe584c00700 1 -- 172.21.15.12:0/619852683 shutdown_connections 2024-03-29T10:14:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.000+0000 7fe584c00700 1 --2- 172.21.15.12:0/619852683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5800cbb30 0x7fe5800cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.000+0000 7fe584c00700 1 --2- 172.21.15.12:0/619852683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800cd020 0x7fe5800cd4a0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.000+0000 7fe584c00700 1 -- 172.21.15.12:0/619852683 >> 172.21.15.12:0/619852683 conn(0x7fe5800b0d70 msgr2=0x7fe5800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.000+0000 7fe584c00700 1 -- 172.21.15.12:0/619852683 shutdown_connections 2024-03-29T10:14:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.000+0000 7fe584c00700 1 -- 172.21.15.12:0/619852683 wait complete. 2024-03-29T10:14:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.002+0000 7fe584c00700 1 Processor -- start 2024-03-29T10:14:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.002+0000 7fe584c00700 1 -- start start 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe584c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800b2070 0x7fe58015b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe584c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5800cbb30 0x7fe5800c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe584c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5800cd020 0x7fe580146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe584c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58015b6b0 con 0x7fe5800b2070 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe584c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5801555f0 con 0x7fe5800cbb30 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe584c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800c8930 con 0x7fe5800cd020 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe57de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800b2070 0x7fe58015b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe57de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800b2070 0x7fe58015b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50636/0 (socket says 172.21.15.12:50636) 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe57de00700 1 -- 172.21.15.12:0/3737609718 learned_addr learned my addr 172.21.15.12:0/3737609718 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe57e800700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5800cd020 0x7fe580146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe57d400700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5800cbb30 0x7fe5800c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe57de00700 1 -- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5800cd020 msgr2=0x7fe580146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe57de00700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5800cd020 0x7fe580146400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe57de00700 1 -- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5800cbb30 msgr2=0x7fe5800c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe57de00700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5800cbb30 0x7fe5800c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.003+0000 7fe57de00700 1 -- 172.21.15.12:0/3737609718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c0094a0 con 0x7fe5800b2070 2024-03-29T10:14:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.004+0000 7fe57de00700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800b2070 0x7fe58015b170 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fe57000d830 tx=0x7fe57000dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.004+0000 7fe576000700 1 -- 172.21.15.12:0/3737609718 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5700125f0 con 0x7fe5800b2070 2024-03-29T10:14:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.004+0000 7fe576000700 1 -- 172.21.15.12:0/3737609718 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe570012750 con 0x7fe5800b2070 2024-03-29T10:14:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.004+0000 7fe576000700 1 -- 172.21.15.12:0/3737609718 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570012b00 con 0x7fe5800b2070 2024-03-29T10:14:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.004+0000 7fe584c00700 1 -- 172.21.15.12:0/3737609718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5801549e0 con 0x7fe5800b2070 2024-03-29T10:14:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.004+0000 7fe584c00700 1 -- 172.21.15.12:0/3737609718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe580154fa0 con 0x7fe5800b2070 2024-03-29T10:14:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.005+0000 7fe584c00700 1 -- 172.21.15.12:0/3737609718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5801558d0 con 0x7fe5800b2070 2024-03-29T10:14:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.006+0000 7fe576000700 1 -- 172.21.15.12:0/3737609718 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe570012c60 con 0x7fe5800b2070 2024-03-29T10:14:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.007+0000 7fe576000700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe56006adb0 0x7fe56006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.007+0000 7fe576000700 1 -- 172.21.15.12:0/3737609718 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe570089890 con 0x7fe5800b2070 2024-03-29T10:14:27.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.007+0000 7fe57d400700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe56006adb0 0x7fe56006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:27.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.008+0000 7fe57d400700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe56006adb0 0x7fe56006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0090b0 tx=0x7fe56c01c040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:27.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.011+0000 7fe576000700 1 -- 172.21.15.12:0/3737609718 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe570054950 con 0x7fe5800b2070 2024-03-29T10:14:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.270+0000 7fe584c00700 1 -- 172.21.15.12:0/3737609718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe58015bda0 con 0x7fe5800b2070 2024-03-29T10:14:27.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.857+0000 7fe576000700 1 -- 172.21.15.12:0/3737609718 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v63) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe570057f70 con 0x7fe5800b2070 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 -- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe56006adb0 msgr2=0x7fe56006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe56006adb0 0x7fe56006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0090b0 tx=0x7fe56c01c040 comp rx=0 tx=0).stop 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 -- 172.21.15.12:0/3737609718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800b2070 msgr2=0x7fe58015b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800b2070 0x7fe58015b170 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fe57000d830 tx=0x7fe57000dbf0 comp rx=0 tx=0).stop 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 -- 172.21.15.12:0/3737609718 shutdown_connections 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe56006adb0 0x7fe56006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5800cd020 0x7fe580146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5800b2070 0x7fe58015b170 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 --2- 172.21.15.12:0/3737609718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5800cbb30 0x7fe5800c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 -- 172.21.15.12:0/3737609718 >> 172.21.15.12:0/3737609718 conn(0x7fe5800b0d70 msgr2=0x7fe5800aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 -- 172.21.15.12:0/3737609718 shutdown_connections 2024-03-29T10:14:27.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:27.860+0000 7fe584c00700 1 -- 172.21.15.12:0/3737609718 wait complete. 2024-03-29T10:14:27.880 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T10:14:27.880 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T10:14:28.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.085+0000 7f5d71200700 1 Processor -- start 2024-03-29T10:14:28.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.085+0000 7f5d71200700 1 -- start start 2024-03-29T10:14:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.086+0000 7f5d71200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c0d1b70 0x7f5d6c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.086+0000 7f5d71200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5d6c0cff10 0x7f5d6c0d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.086+0000 7f5d71200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c0cf640 con 0x7f5d6c15d710 2024-03-29T10:14:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.086+0000 7f5d71200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d6c0cf220 con 0x7f5d6c0cff10 2024-03-29T10:14:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.087+0000 7f5d71200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c0d2560 con 0x7f5d6c0d1b70 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.087+0000 7f5d63e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c0d1b70 0x7f5d6c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.087+0000 7f5d6a800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5d6c0cff10 0x7f5d6c0d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.087+0000 7f5d63e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c0d1b70 0x7f5d6c0d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:51732/0 (socket says 172.21.15.12:51732) 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.087+0000 7f5d63e00700 1 -- 172.21.15.12:0/1181358403 learned_addr learned my addr 172.21.15.12:0/1181358403 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.087+0000 7f5d69e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f5d6c15d710 0x7f5d6c0d12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:32850/0 (socket says 172.21.15.12:32850) 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.087+0000 7f5d6a800700 1 --2- 172.21.15.12:0/1181358403 >> v2:172.21.15.159:3300/0 conn(0x7f5d6c0cff10 0x7f5d6c0d03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:43174/0 (socket says 172.21.15.12:43174) 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.088+0000 7f5d6a800700 1 -- 172.21.15.12:0/1181358403 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c0d1b70 msgr2=0x7f5d6c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.088+0000 7f5d6a800700 1 --2- 172.21.15.12:0/1181358403 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c0d1b70 0x7f5d6c0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.088+0000 7f5d6a800700 1 -- 172.21.15.12:0/1181358403 >> v1:172.21.15.159:6789/0 conn(0x7f5d6c15d710 legacy=0x7f5d6c0d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.088+0000 7f5d6a800700 1 -- 172.21.15.12:0/1181358403 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c0ce5a0 con 0x7f5d6c0cff10 2024-03-29T10:14:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.088+0000 7f5d6a800700 1 --2- 172.21.15.12:0/1181358403 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c0cff10 0x7f5d6c0d03a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5d64009d90 tx=0x7f5d6400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50c7b4e00b4a4235 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.088+0000 7f5d69400700 1 -- 172.21.15.12:0/1181358403 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6400e070 con 0x7f5d6c0cff10 2024-03-29T10:14:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.088+0000 7f5d69400700 1 -- 172.21.15.12:0/1181358403 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6400bb20 con 0x7f5d6c0cff10 2024-03-29T10:14:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.088+0000 7f5d69400700 1 -- 172.21.15.12:0/1181358403 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64012430 con 0x7f5d6c0cff10 2024-03-29T10:14:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.089+0000 7f5d71200700 1 -- 172.21.15.12:0/1181358403 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c0cff10 msgr2=0x7f5d6c0d03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.089+0000 7f5d71200700 1 --2- 172.21.15.12:0/1181358403 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c0cff10 0x7f5d6c0d03a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5d64009d90 tx=0x7f5d6400b330 comp rx=0 tx=0).stop 2024-03-29T10:14:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.089+0000 7f5d71200700 1 -- 172.21.15.12:0/1181358403 shutdown_connections 2024-03-29T10:14:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.089+0000 7f5d71200700 1 --2- 172.21.15.12:0/1181358403 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c0cff10 0x7f5d6c0d03a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.089+0000 7f5d71200700 1 --2- 172.21.15.12:0/1181358403 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c0d1b70 0x7f5d6c0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.089+0000 7f5d71200700 1 -- 172.21.15.12:0/1181358403 >> 172.21.15.12:0/1181358403 conn(0x7f5d6c08aff0 msgr2=0x7f5d6c0d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.089+0000 7f5d71200700 1 -- 172.21.15.12:0/1181358403 shutdown_connections 2024-03-29T10:14:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.089+0000 7f5d71200700 1 -- 172.21.15.12:0/1181358403 wait complete. 2024-03-29T10:14:28.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.091+0000 7f5d71200700 1 Processor -- start 2024-03-29T10:14:28.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.091+0000 7f5d71200700 1 -- start start 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d71200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c0cff10 0x7f5d6c0c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d71200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c0d1b70 0x7f5d6c0c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d71200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c15d710 0x7f5d6c15daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d71200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c0c7ea0 con 0x7f5d6c0cff10 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d71200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c0c6ff0 con 0x7f5d6c0d1b70 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d71200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c0c7290 con 0x7f5d6c15d710 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d69e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c0cff10 0x7f5d6c0c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d69e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c0cff10 0x7f5d6c0c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50646/0 (socket says 172.21.15.12:50646) 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d63e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c0d1b70 0x7f5d6c0c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d69e00700 1 -- 172.21.15.12:0/4232732344 learned_addr learned my addr 172.21.15.12:0/4232732344 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.092+0000 7f5d6a800700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c15d710 0x7f5d6c15daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d69e00700 1 -- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c15d710 msgr2=0x7f5d6c15daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d69e00700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c15d710 0x7f5d6c15daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d69e00700 1 -- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c0d1b70 msgr2=0x7f5d6c0c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d69e00700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c0d1b70 0x7f5d6c0c8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d69e00700 1 -- 172.21.15.12:0/4232732344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d64009a70 con 0x7f5d6c0cff10 2024-03-29T10:14:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d6a800700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c15d710 0x7f5d6c15daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d69e00700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c0cff10 0x7f5d6c0c8620 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00c520 tx=0x7f5d5c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d62000700 1 -- 172.21.15.12:0/4232732344 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c005cb0 con 0x7f5d6c0cff10 2024-03-29T10:14:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d62000700 1 -- 172.21.15.12:0/4232732344 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c005e10 con 0x7f5d6c0cff10 2024-03-29T10:14:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d62000700 1 -- 172.21.15.12:0/4232732344 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c0072d0 con 0x7f5d6c0cff10 2024-03-29T10:14:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d71200700 1 -- 172.21.15.12:0/4232732344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6c0c7570 con 0x7f5d6c0cff10 2024-03-29T10:14:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.093+0000 7f5d71200700 1 -- 172.21.15.12:0/4232732344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6c0c7b30 con 0x7f5d6c0cff10 2024-03-29T10:14:28.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.094+0000 7f5d71200700 1 -- 172.21.15.12:0/4232732344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6c067e30 con 0x7f5d6c0cff10 2024-03-29T10:14:28.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.095+0000 7f5d62000700 1 -- 172.21.15.12:0/4232732344 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5d5c007430 con 0x7f5d6c0cff10 2024-03-29T10:14:28.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.096+0000 7f5d62000700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d4806adb0 0x7f5d4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:28.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.096+0000 7f5d62000700 1 -- 172.21.15.12:0/4232732344 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d5c088700 con 0x7f5d6c0cff10 2024-03-29T10:14:28.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.096+0000 7f5d63e00700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d4806adb0 0x7f5d4806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:28.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.097+0000 7f5d63e00700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d4806adb0 0x7f5d4806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0cde70 tx=0x7f5d54009290 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.100+0000 7f5d62000700 1 -- 172.21.15.12:0/4232732344 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5d5c053880 con 0x7f5d6c0cff10 2024-03-29T10:14:28.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:28.362+0000 7f5d71200700 1 -- 172.21.15.12:0/4232732344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5d6c0b9510 con 0x7f5d6c0cff10 2024-03-29T10:14:28.617 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:14:28.621 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:14:29.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.118+0000 7f5d62000700 1 -- 172.21.15.12:0/4232732344 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5d5c0170c0 con 0x7f5d6c0cff10 2024-03-29T10:14:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.120+0000 7f5d71200700 1 -- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d4806adb0 msgr2=0x7f5d4806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.120+0000 7f5d71200700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d4806adb0 0x7f5d4806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0cde70 tx=0x7f5d54009290 comp rx=0 tx=0).stop 2024-03-29T10:14:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.120+0000 7f5d71200700 1 -- 172.21.15.12:0/4232732344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c0cff10 msgr2=0x7f5d6c0c8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.120+0000 7f5d71200700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c0cff10 0x7f5d6c0c8620 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00c520 tx=0x7f5d5c00c830 comp rx=0 tx=0).stop 2024-03-29T10:14:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.120+0000 7f5d71200700 1 -- 172.21.15.12:0/4232732344 shutdown_connections 2024-03-29T10:14:29.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.120+0000 7f5d71200700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5d4806adb0 0x7f5d4806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.120+0000 7f5d71200700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d6c15d710 0x7f5d6c15daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.120+0000 7f5d71200700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c0cff10 0x7f5d6c0c8620 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.120+0000 7f5d71200700 1 --2- 172.21.15.12:0/4232732344 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c0d1b70 0x7f5d6c0c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.121+0000 7f5d71200700 1 -- 172.21.15.12:0/4232732344 >> 172.21.15.12:0/4232732344 conn(0x7f5d6c08aff0 msgr2=0x7f5d6c0b1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:29.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.121+0000 7f5d71200700 1 -- 172.21.15.12:0/4232732344 shutdown_connections 2024-03-29T10:14:29.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.121+0000 7f5d71200700 1 -- 172.21.15.12:0/4232732344 wait complete. 2024-03-29T10:14:29.138 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T10:14:29.138 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:29.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.340+0000 7f6746200700 1 Processor -- start 2024-03-29T10:14:29.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.340+0000 7f6746200700 1 -- start start 2024-03-29T10:14:29.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.341+0000 7f6746200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67400b3af0 0x7f674015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.341+0000 7f6746200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6740072580 0x7f67400b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.341+0000 7f6746200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67400b2180 con 0x7f67400b2e50 2024-03-29T10:14:29.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.341+0000 7f6746200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f674015b100 con 0x7f67400b3af0 2024-03-29T10:14:29.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.341+0000 7f6746200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674015b3e0 con 0x7f6740072580 2024-03-29T10:14:29.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.341+0000 7f673f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67400b3af0 0x7f674015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f67400b2e50 0x7f67400709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40096/0 (socket says 172.21.15.12:40096) 2024-03-29T10:14:29.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67400b3af0 0x7f674015a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58980/0 (socket says 172.21.15.12:58980) 2024-03-29T10:14:29.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673fe00700 1 -- 172.21.15.12:0/438037996 learned_addr learned my addr 172.21.15.12:0/438037996 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:29.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673f400700 1 -- 172.21.15.12:0/438037996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6740072580 msgr2=0x7f67400b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:29.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673ea00700 1 --2- 172.21.15.12:0/438037996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6740072580 0x7f67400b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673f400700 1 --2- 172.21.15.12:0/438037996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6740072580 0x7f67400b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673f400700 1 -- 172.21.15.12:0/438037996 >> v1:172.21.15.12:6789/0 conn(0x7f67400b2e50 legacy=0x7f67400709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673f400700 1 -- 172.21.15.12:0/438037996 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67400b23f0 con 0x7f67400b3af0 2024-03-29T10:14:29.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673e000700 1 -- 172.21.15.12:0/438037996 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521637901 0 0) 0x7f67400b2180 con 0x7f67400b2e50 2024-03-29T10:14:29.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673ea00700 1 --2- 172.21.15.12:0/438037996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6740072580 0x7f67400b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:14:29.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.342+0000 7f673f400700 1 --2- 172.21.15.12:0/438037996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b3af0 0x7f674015a860 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f6730009840 tx=0x7f6730009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00a65db15243a40 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:29.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f673e000700 1 -- 172.21.15.12:0/438037996 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673000f8a0 con 0x7f67400b3af0 2024-03-29T10:14:29.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f673e000700 1 -- 172.21.15.12:0/438037996 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673000fa00 con 0x7f67400b3af0 2024-03-29T10:14:29.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f673e000700 1 -- 172.21.15.12:0/438037996 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673000fd90 con 0x7f67400b3af0 2024-03-29T10:14:29.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f6746200700 1 -- 172.21.15.12:0/438037996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b3af0 msgr2=0x7f674015a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f6746200700 1 --2- 172.21.15.12:0/438037996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b3af0 0x7f674015a860 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f6730009840 tx=0x7f6730009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:29.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f6746200700 1 -- 172.21.15.12:0/438037996 shutdown_connections 2024-03-29T10:14:29.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f6746200700 1 --2- 172.21.15.12:0/438037996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b3af0 0x7f674015a860 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f6746200700 1 --2- 172.21.15.12:0/438037996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6740072580 0x7f67400b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f6746200700 1 -- 172.21.15.12:0/438037996 >> 172.21.15.12:0/438037996 conn(0x7f67400cf190 msgr2=0x7f67400cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:29.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f6746200700 1 -- 172.21.15.12:0/438037996 shutdown_connections 2024-03-29T10:14:29.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.343+0000 7f6746200700 1 -- 172.21.15.12:0/438037996 wait complete. 2024-03-29T10:14:29.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.345+0000 7f6746200700 1 Processor -- start 2024-03-29T10:14:29.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.345+0000 7f6746200700 1 -- start start 2024-03-29T10:14:29.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.346+0000 7f6746200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6740072580 0x7f6740154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.346+0000 7f6746200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b2e50 0x7f674014d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.346+0000 7f6746200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67400b3af0 0x7f67401583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.346+0000 7f6746200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6740151d50 con 0x7f67400b2e50 2024-03-29T10:14:29.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.346+0000 7f6746200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67401522f0 con 0x7f6740072580 2024-03-29T10:14:29.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.346+0000 7f6746200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6740151520 con 0x7f67400b3af0 2024-03-29T10:14:29.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.346+0000 7f673ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b2e50 0x7f674014d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.346+0000 7f673ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b2e50 0x7f674014d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58994/0 (socket says 172.21.15.12:58994) 2024-03-29T10:14:29.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.346+0000 7f673ea00700 1 -- 172.21.15.12:0/3891260609 learned_addr learned my addr 172.21.15.12:0/3891260609 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f673f400700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6740072580 0x7f6740154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f673fe00700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67400b3af0 0x7f67401583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f673ea00700 1 -- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67400b3af0 msgr2=0x7f67401583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f673ea00700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67400b3af0 0x7f67401583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f673ea00700 1 -- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6740072580 msgr2=0x7f6740154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f673ea00700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6740072580 0x7f6740154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f673ea00700 1 -- 172.21.15.12:0/3891260609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67300094a0 con 0x7f67400b2e50 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f673fe00700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67400b3af0 0x7f67401583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f673ea00700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b2e50 0x7f674014d3c0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f672c00c7f0 tx=0x7f672c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f6737400700 1 -- 172.21.15.12:0/3891260609 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c011800 con 0x7f67400b2e50 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f6737400700 1 -- 172.21.15.12:0/3891260609 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c011960 con 0x7f67400b2e50 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f6737400700 1 -- 172.21.15.12:0/3891260609 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c011cf0 con 0x7f67400b2e50 2024-03-29T10:14:29.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.347+0000 7f6746200700 1 -- 172.21.15.12:0/3891260609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6740152010 con 0x7f67400b2e50 2024-03-29T10:14:29.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.348+0000 7f6746200700 1 -- 172.21.15.12:0/3891260609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67401553b0 con 0x7f67400b2e50 2024-03-29T10:14:29.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.349+0000 7f6746200700 1 -- 172.21.15.12:0/3891260609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67400af940 con 0x7f67400b2e50 2024-03-29T10:14:29.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.349+0000 7f6737400700 1 -- 172.21.15.12:0/3891260609 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f672c018360 con 0x7f67400b2e50 2024-03-29T10:14:29.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.350+0000 7f6737400700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f672006adb0 0x7f672006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.350+0000 7f6737400700 1 -- 172.21.15.12:0/3891260609 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f672c089530 con 0x7f67400b2e50 2024-03-29T10:14:29.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.351+0000 7f673f400700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f672006adb0 0x7f672006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.352+0000 7f673f400700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f672006adb0 0x7f672006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f67300090b0 tx=0x7f673001b040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:29.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.355+0000 7f6737400700 1 -- 172.21.15.12:0/3891260609 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67400af940 con 0x7f67400b2e50 2024-03-29T10:14:29.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.619+0000 7f6746200700 1 -- 172.21.15.12:0/3891260609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67400af940 con 0x7f67400b2e50 2024-03-29T10:14:29.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.619+0000 7f6737400700 1 -- 172.21.15.12:0/3891260609 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f67400af940 con 0x7f67400b2e50 2024-03-29T10:14:29.618 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:29.618 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1317631939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1317631939},{"type":"v1","addr":"172.21.15.159:6836","nonce":1317631939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1057731647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1057731647},{"type":"v1","addr":"172.21.15.159:6837","nonce":1057731647}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1567088347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1567088347},{"type":"v1","addr":"172.21.15.12:6834","nonce":1567088347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:14:25.846391+0000","modified":"2024-03-29T10:14:29.114278+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8406":{"gid":8406,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3311625077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3311625077},{"type":"v1","addr":"172.21.15.12:6835","nonce":3311625077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-29T10:14:29.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 -- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f672006adb0 msgr2=0x7f672006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f672006adb0 0x7f672006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f67300090b0 tx=0x7f673001b040 comp rx=0 tx=0).stop 2024-03-29T10:14:29.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 -- 172.21.15.12:0/3891260609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b2e50 msgr2=0x7f674014d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b2e50 0x7f674014d3c0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f672c00c7f0 tx=0x7f672c00cb00 comp rx=0 tx=0).stop 2024-03-29T10:14:29.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 -- 172.21.15.12:0/3891260609 shutdown_connections 2024-03-29T10:14:29.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f672006adb0 0x7f672006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67400b3af0 0x7f67401583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6740072580 0x7f6740154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 --2- 172.21.15.12:0/3891260609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67400b2e50 0x7f674014d3c0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 -- 172.21.15.12:0/3891260609 >> 172.21.15.12:0/3891260609 conn(0x7f67400cf190 msgr2=0x7f67400acf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:29.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 -- 172.21.15.12:0/3891260609 shutdown_connections 2024-03-29T10:14:29.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.622+0000 7f6746200700 1 -- 172.21.15.12:0/3891260609 wait complete. 2024-03-29T10:14:29.622 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 64 2024-03-29T10:14:29.639 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.838+0000 7f45e2200700 1 Processor -- start 2024-03-29T10:14:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.838+0000 7f45e2200700 1 -- start start 2024-03-29T10:14:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.839+0000 7f45e2200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 0x7f45dc15a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.839+0000 7f45e2200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f45dc072580 0x7f45dc0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.839+0000 7f45e2200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f45dc0b2e50 0x7f45dc0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.839+0000 7f45e2200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45dc0b2180 con 0x7f45dc0b2e50 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.839+0000 7f45e2200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45dc15b100 con 0x7f45dc072580 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.839+0000 7f45e2200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc15b3e0 con 0x7f45dc0b3af0 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.839+0000 7f45dbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f45dc0b2e50 0x7f45dc0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45dbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f45dc0b2e50 0x7f45dc0709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59008/0 (socket says 172.21.15.12:59008) 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45dbe00700 1 -- 172.21.15.12:0/753564711 learned_addr learned my addr 172.21.15.12:0/753564711 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45daa00700 1 --2- 172.21.15.12:0/753564711 >> v2:172.21.15.159:3300/0 conn(0x7f45dc072580 0x7f45dc0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45db400700 1 --2- 172.21.15.12:0/753564711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 0x7f45dc15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45dbe00700 1 -- 172.21.15.12:0/753564711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 msgr2=0x7f45dc15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45dbe00700 1 --2- 172.21.15.12:0/753564711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 0x7f45dc15a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45dbe00700 1 -- 172.21.15.12:0/753564711 >> v2:172.21.15.159:3300/0 conn(0x7f45dc072580 msgr2=0x7f45dc0b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45dbe00700 1 --2- 172.21.15.12:0/753564711 >> v2:172.21.15.159:3300/0 conn(0x7f45dc072580 0x7f45dc0b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45dbe00700 1 -- 172.21.15.12:0/753564711 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc0b23f0 con 0x7f45dc0b2e50 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45db400700 1 --2- 172.21.15.12:0/753564711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 0x7f45dc15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45dbe00700 1 --2- 172.21.15.12:0/753564711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc0b2e50 0x7f45dc0709c0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f45dc070030 tx=0x7f45d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b65e4c58fa289a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45da000700 1 -- 172.21.15.12:0/753564711 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d400e070 con 0x7f45dc0b2e50 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45da000700 1 -- 172.21.15.12:0/753564711 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d400b910 con 0x7f45dc0b2e50 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.840+0000 7f45da000700 1 -- 172.21.15.12:0/753564711 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d400bcc0 con 0x7f45dc0b2e50 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.841+0000 7f45e2200700 1 -- 172.21.15.12:0/753564711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc0b2e50 msgr2=0x7f45dc0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.841+0000 7f45e2200700 1 --2- 172.21.15.12:0/753564711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc0b2e50 0x7f45dc0709c0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f45dc070030 tx=0x7f45d400b330 comp rx=0 tx=0).stop 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.841+0000 7f45e2200700 1 -- 172.21.15.12:0/753564711 shutdown_connections 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.841+0000 7f45e2200700 1 --2- 172.21.15.12:0/753564711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc0b2e50 0x7f45dc0709c0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.841+0000 7f45e2200700 1 --2- 172.21.15.12:0/753564711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 0x7f45dc15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.841+0000 7f45e2200700 1 --2- 172.21.15.12:0/753564711 >> v2:172.21.15.159:3300/0 conn(0x7f45dc072580 0x7f45dc0b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.841+0000 7f45e2200700 1 -- 172.21.15.12:0/753564711 >> 172.21.15.12:0/753564711 conn(0x7f45dc0cf190 msgr2=0x7f45dc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.841+0000 7f45e2200700 1 -- 172.21.15.12:0/753564711 shutdown_connections 2024-03-29T10:14:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.841+0000 7f45e2200700 1 -- 172.21.15.12:0/753564711 wait complete. 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.843+0000 7f45e2200700 1 Processor -- start 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.843+0000 7f45e2200700 1 -- start start 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.844+0000 7f45e2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc072580 0x7f45dc0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.844+0000 7f45e2200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45dc0b2e50 0x7f45dc0c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.844+0000 7f45e2200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 0x7f45dc0cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.844+0000 7f45e2200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc0c9120 con 0x7f45dc072580 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.844+0000 7f45e2200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc0cb080 con 0x7f45dc0b2e50 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.844+0000 7f45e2200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc0cde70 con 0x7f45dc0b3af0 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.844+0000 7f45db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc072580 0x7f45dc0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.844+0000 7f45db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc072580 0x7f45dc0ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59014/0 (socket says 172.21.15.12:59014) 2024-03-29T10:14:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.844+0000 7f45db400700 1 -- 172.21.15.12:0/4258773897 learned_addr learned my addr 172.21.15.12:0/4258773897 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45daa00700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45dc0b2e50 0x7f45dc0c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45dbe00700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 0x7f45dc0cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45db400700 1 -- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 msgr2=0x7f45dc0cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45db400700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 0x7f45dc0cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45db400700 1 -- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45dc0b2e50 msgr2=0x7f45dc0c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45db400700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45dc0b2e50 0x7f45dc0c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45db400700 1 -- 172.21.15.12:0/4258773897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d4009b30 con 0x7f45dc072580 2024-03-29T10:14:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45dbe00700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 0x7f45dc0cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45db400700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc072580 0x7f45dc0ccbf0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00e760 tx=0x7f45cc00ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45d3400700 1 -- 172.21.15.12:0/4258773897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc013610 con 0x7f45dc072580 2024-03-29T10:14:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45d3400700 1 -- 172.21.15.12:0/4258773897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc013770 con 0x7f45dc072580 2024-03-29T10:14:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45d3400700 1 -- 172.21.15.12:0/4258773897 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc013b20 con 0x7f45dc072580 2024-03-29T10:14:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.845+0000 7f45e2200700 1 -- 172.21.15.12:0/4258773897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc1446b0 con 0x7f45dc072580 2024-03-29T10:14:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.846+0000 7f45e2200700 1 -- 172.21.15.12:0/4258773897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45dc0cd540 con 0x7f45dc072580 2024-03-29T10:14:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.847+0000 7f45e2200700 1 -- 172.21.15.12:0/4258773897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45dc067720 con 0x7f45dc072580 2024-03-29T10:14:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.848+0000 7f45d3400700 1 -- 172.21.15.12:0/4258773897 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f45cc013c80 con 0x7f45dc072580 2024-03-29T10:14:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.849+0000 7f45d3400700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45bc06adb0 0x7f45bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.849+0000 7f45d3400700 1 -- 172.21.15.12:0/4258773897 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45cc088ff0 con 0x7f45dc072580 2024-03-29T10:14:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.849+0000 7f45daa00700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45bc06adb0 0x7f45bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.850+0000 7f45daa00700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45bc06adb0 0x7f45bc06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45dc1532d0 tx=0x7f45c8009210 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:29.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:29.854+0000 7f45d3400700 1 -- 172.21.15.12:0/4258773897 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45cc054040 con 0x7f45dc072580 2024-03-29T10:14:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.070+0000 7f45e2200700 1 -- 172.21.15.12:0/4258773897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45dc0cef70 con 0x7f45dc072580 2024-03-29T10:14:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.071+0000 7f45d3400700 1 -- 172.21.15.12:0/4258773897 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16059 (secure 0 0 0) 0x7f45cc057660 con 0x7f45dc072580 2024-03-29T10:14:30.070 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:30.070 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":140,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:14:26.848746+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:14:23.708181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-29T10:14:24.829152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.073+0000 7f45e2200700 1 -- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45bc06adb0 msgr2=0x7f45bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.073+0000 7f45e2200700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45bc06adb0 0x7f45bc06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45dc1532d0 tx=0x7f45c8009210 comp rx=0 tx=0).stop 2024-03-29T10:14:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.074+0000 7f45e2200700 1 -- 172.21.15.12:0/4258773897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc072580 msgr2=0x7f45dc0ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.074+0000 7f45e2200700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc072580 0x7f45dc0ccbf0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00e760 tx=0x7f45cc00ea70 comp rx=0 tx=0).stop 2024-03-29T10:14:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.074+0000 7f45e2200700 1 -- 172.21.15.12:0/4258773897 shutdown_connections 2024-03-29T10:14:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.074+0000 7f45e2200700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45bc06adb0 0x7f45bc06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.074+0000 7f45e2200700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45dc0b3af0 0x7f45dc0cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.074+0000 7f45e2200700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45dc072580 0x7f45dc0ccbf0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.074+0000 7f45e2200700 1 --2- 172.21.15.12:0/4258773897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45dc0b2e50 0x7f45dc0c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.074+0000 7f45e2200700 1 -- 172.21.15.12:0/4258773897 >> 172.21.15.12:0/4258773897 conn(0x7f45dc0cf190 msgr2=0x7f45dc0709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.074+0000 7f45e2200700 1 -- 172.21.15.12:0/4258773897 shutdown_connections 2024-03-29T10:14:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.074+0000 7f45e2200700 1 -- 172.21.15.12:0/4258773897 wait complete. 2024-03-29T10:14:30.091 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T10:14:30.092 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:14:30.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.296+0000 7f6b2f600700 1 Processor -- start 2024-03-29T10:14:30.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.296+0000 7f6b2f600700 1 -- start start 2024-03-29T10:14:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.297+0000 7f6b2f600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 0x7f6b280ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.297+0000 7f6b2f600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b280b3890 con 0x7f6b2806dee0 2024-03-29T10:14:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.297+0000 7f6b2f600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b280acac0 con 0x7f6b2815da50 2024-03-29T10:14:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.297+0000 7f6b2f600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28159f50 con 0x7f6b2815d670 2024-03-29T10:14:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.297+0000 7f6b2d600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6b2806dee0 0x7f6b2806e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40112/0 (socket says 172.21.15.12:40112) 2024-03-29T10:14:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.297+0000 7f6b2d600700 1 -- 172.21.15.12:0/3047316022 learned_addr learned my addr 172.21.15.12:0/3047316022 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.297+0000 7f6b2cc00700 1 --2- 172.21.15.12:0/3047316022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 0x7f6b280ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.298+0000 7f6b27400700 1 -- 172.21.15.12:0/3047316022 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046257682 0 0) 0x7f6b280b3890 con 0x7f6b2806dee0 2024-03-29T10:14:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.298+0000 7f6b27400700 1 -- 172.21.15.12:0/3047316022 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b100057c0 con 0x7f6b2806dee0 2024-03-29T10:14:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.298+0000 7f6b27400700 1 -- 172.21.15.12:0/3047316022 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874332395 0 0) 0x7f6b280acac0 con 0x7f6b2815da50 2024-03-29T10:14:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.298+0000 7f6b27400700 1 -- 172.21.15.12:0/3047316022 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b280b3890 con 0x7f6b2815da50 2024-03-29T10:14:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.298+0000 7f6b2cc00700 1 -- 172.21.15.12:0/3047316022 >> v1:172.21.15.159:6789/0 conn(0x7f6b2815da50 legacy=0x7f6b280b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.298+0000 7f6b2cc00700 1 -- 172.21.15.12:0/3047316022 >> v1:172.21.15.12:6789/0 conn(0x7f6b2806dee0 legacy=0x7f6b2806e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.298+0000 7f6b2cc00700 1 -- 172.21.15.12:0/3047316022 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b280b35b0 con 0x7f6b2815d670 2024-03-29T10:14:30.298 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-29T10:14:30.298+0000 7f3752c00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:14:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.298+0000 7f6b2cc00700 1 --2- 172.21.15.12:0/3047316022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 0x7f6b280ac580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c009870 tx=0x7f6b1c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e790c4e0849b8a0f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.299+0000 7f6b27400700 1 -- 172.21.15.12:0/3047316022 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c011830 con 0x7f6b2815d670 2024-03-29T10:14:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.299+0000 7f6b27400700 1 -- 172.21.15.12:0/3047316022 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c011990 con 0x7f6b2815d670 2024-03-29T10:14:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.299+0000 7f6b27400700 1 -- 172.21.15.12:0/3047316022 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c011d20 con 0x7f6b2815d670 2024-03-29T10:14:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.299+0000 7f6b2f600700 1 -- 172.21.15.12:0/3047316022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 msgr2=0x7f6b280ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.299+0000 7f6b2f600700 1 --2- 172.21.15.12:0/3047316022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 0x7f6b280ac580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c009870 tx=0x7f6b1c009b80 comp rx=0 tx=0).stop 2024-03-29T10:14:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.299+0000 7f6b2f600700 1 -- 172.21.15.12:0/3047316022 shutdown_connections 2024-03-29T10:14:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.299+0000 7f6b2f600700 1 --2- 172.21.15.12:0/3047316022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 0x7f6b280ac580 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.299+0000 7f6b2f600700 1 -- 172.21.15.12:0/3047316022 >> 172.21.15.12:0/3047316022 conn(0x7f6b280cf190 msgr2=0x7f6b280cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.300+0000 7f6b2f600700 1 -- 172.21.15.12:0/3047316022 shutdown_connections 2024-03-29T10:14:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.300+0000 7f6b2f600700 1 -- 172.21.15.12:0/3047316022 wait complete. 2024-03-29T10:14:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.301+0000 7f6b2f600700 1 Processor -- start 2024-03-29T10:14:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.301+0000 7f6b2f600700 1 -- start start 2024-03-29T10:14:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.302+0000 7f6b2f600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b2806dee0 0x7f6b281f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.302+0000 7f6b2f600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 0x7f6b280c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.302+0000 7f6b2f600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2815da50 0x7f6b280ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.302+0000 7f6b2f600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b280c9410 con 0x7f6b2815da50 2024-03-29T10:14:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.302+0000 7f6b2f600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2815bc30 con 0x7f6b2806dee0 2024-03-29T10:14:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.302+0000 7f6b2f600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2815b340 con 0x7f6b2815d670 2024-03-29T10:14:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b2d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2815da50 0x7f6b280ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b2d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2815da50 0x7f6b280ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59016/0 (socket says 172.21.15.12:59016) 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b2d600700 1 -- 172.21.15.12:0/2071203693 learned_addr learned my addr 172.21.15.12:0/2071203693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b2cc00700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b2806dee0 0x7f6b281f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b27e00700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 0x7f6b280c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b2d600700 1 -- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 msgr2=0x7f6b280c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b2d600700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 0x7f6b280c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b2d600700 1 -- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b2806dee0 msgr2=0x7f6b281f2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b2d600700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b2806dee0 0x7f6b281f2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b2d600700 1 -- 172.21.15.12:0/2071203693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c0094a0 con 0x7f6b2815da50 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b27e00700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 0x7f6b280c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.303+0000 7f6b2d600700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2815da50 0x7f6b280ca110 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000cba0 tx=0x7f6b2000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.304+0000 7f6b24c00700 1 -- 172.21.15.12:0/2071203693 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2000aa70 con 0x7f6b2815da50 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.304+0000 7f6b24c00700 1 -- 172.21.15.12:0/2071203693 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2000abd0 con 0x7f6b2815da50 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.304+0000 7f6b2f600700 1 -- 172.21.15.12:0/2071203693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2815b5e0 con 0x7f6b2815da50 2024-03-29T10:14:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.304+0000 7f6b24c00700 1 -- 172.21.15.12:0/2071203693 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b20012400 con 0x7f6b2815da50 2024-03-29T10:14:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.304+0000 7f6b2f600700 1 -- 172.21.15.12:0/2071203693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b280b09f0 con 0x7f6b2815da50 2024-03-29T10:14:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.305+0000 7f6b2f600700 1 -- 172.21.15.12:0/2071203693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b28070e70 con 0x7f6b2815da50 2024-03-29T10:14:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.306+0000 7f6b24c00700 1 -- 172.21.15.12:0/2071203693 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6b2001b3b0 con 0x7f6b2815da50 2024-03-29T10:14:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.307+0000 7f6b24c00700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0806adb0 0x7f6b0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.307+0000 7f6b24c00700 1 -- 172.21.15.12:0/2071203693 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b20089270 con 0x7f6b2815da50 2024-03-29T10:14:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.307+0000 7f6b2cc00700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0806adb0 0x7f6b0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.308+0000 7f6b2cc00700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0806adb0 0x7f6b0806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c0097c0 tx=0x7f6b1c01c040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:30.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.311+0000 7f6b24c00700 1 -- 172.21.15.12:0/2071203693 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6b200542a0 con 0x7f6b2815da50 2024-03-29T10:14:30.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.523+0000 7f6b2f600700 1 -- 172.21.15.12:0/2071203693 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b2806d9c0 con 0x7f6b0806adb0 2024-03-29T10:14:30.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.534+0000 7f6b24c00700 1 -- 172.21.15.12:0/2071203693 <== mgr.8190 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289493 (secure 0 0 0) 0x7f6b1c011b00 con 0x7f6b0806adb0 2024-03-29T10:14:30.539 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:30.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 -- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0806adb0 msgr2=0x7f6b0806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0806adb0 0x7f6b0806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c0097c0 tx=0x7f6b1c01c040 comp rx=0 tx=0).stop 2024-03-29T10:14:30.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 -- 172.21.15.12:0/2071203693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2815da50 msgr2=0x7f6b280ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2815da50 0x7f6b280ca110 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000cba0 tx=0x7f6b2000ceb0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 -- 172.21.15.12:0/2071203693 shutdown_connections 2024-03-29T10:14:30.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b0806adb0 0x7f6b0806d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2815da50 0x7f6b280ca110 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b2806dee0 0x7f6b281f2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 --2- 172.21.15.12:0/2071203693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b2815d670 0x7f6b280c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 -- 172.21.15.12:0/2071203693 >> 172.21.15.12:0/2071203693 conn(0x7f6b280cf190 msgr2=0x7f6b280b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:30.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 -- 172.21.15.12:0/2071203693 shutdown_connections 2024-03-29T10:14:30.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.539+0000 7f6b2f600700 1 -- 172.21.15.12:0/2071203693 wait complete. 2024-03-29T10:14:30.542 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:14:30.556 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-29T10:14:29.960771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":435,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283,"ondisk_log_size":283,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":340512,"kb_used_data":14508,"kb_used_omap":12,"kb_used_meta":325875,"kb_avail":754634208,"statfs":{"total":773094113280,"available":772745428992,"internally_reserved":0,"allocated":14856192,"data_stored":7715223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":333696569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.643587"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.882323+0000","last_change":"2024-03-29T10:14:25.867231+0000","last_active":"2024-03-29T10:14:25.882323+0000","last_peered":"2024-03-29T10:14:25.882323+0000","last_clean":"2024-03-29T10:14:25.882323+0000","last_became_active":"2024-03-29T10:14:25.866717+0000","last_became_peered":"2024-03-29T10:14:25.866717+0000","last_unstale":"2024-03-29T10:14:25.882323+0000","last_undegraded":"2024-03-29T10:14:25.882323+0000","last_fullsized":"2024-03-29T10:14:25.882323+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:47.090759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.870000+0000","last_change":"2024-03-29T10:14:25.865931+0000","last_active":"2024-03-29T10:14:25.870000+0000","last_peered":"2024-03-29T10:14:25.870000+0000","last_clean":"2024-03-29T10:14:25.870000+0000","last_became_active":"2024-03-29T10:14:25.865258+0000","last_became_peered":"2024-03-29T10:14:25.865258+0000","last_unstale":"2024-03-29T10:14:25.870000+0000","last_undegraded":"2024-03-29T10:14:25.870000+0000","last_fullsized":"2024-03-29T10:14:25.870000+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:35.441935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876611+0000","last_change":"2024-03-29T10:14:25.874133+0000","last_active":"2024-03-29T10:14:25.876611+0000","last_peered":"2024-03-29T10:14:25.876611+0000","last_clean":"2024-03-29T10:14:25.876611+0000","last_became_active":"2024-03-29T10:14:25.871258+0000","last_became_peered":"2024-03-29T10:14:25.871258+0000","last_unstale":"2024-03-29T10:14:25.876611+0000","last_undegraded":"2024-03-29T10:14:25.876611+0000","last_fullsized":"2024-03-29T10:14:25.876611+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:06.217126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.870796+0000","last_change":"2024-03-29T10:14:25.861248+0000","last_active":"2024-03-29T10:14:25.870796+0000","last_peered":"2024-03-29T10:14:25.870796+0000","last_clean":"2024-03-29T10:14:25.870796+0000","last_became_active":"2024-03-29T10:14:25.860968+0000","last_became_peered":"2024-03-29T10:14:25.860968+0000","last_unstale":"2024-03-29T10:14:25.870796+0000","last_undegraded":"2024-03-29T10:14:25.870796+0000","last_fullsized":"2024-03-29T10:14:25.870796+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:18:09.216626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880415+0000","last_change":"2024-03-29T10:14:25.877398+0000","last_active":"2024-03-29T10:14:25.880415+0000","last_peered":"2024-03-29T10:14:25.880415+0000","last_clean":"2024-03-29T10:14:25.880415+0000","last_became_active":"2024-03-29T10:14:25.873261+0000","last_became_peered":"2024-03-29T10:14:25.873261+0000","last_unstale":"2024-03-29T10:14:25.880415+0000","last_undegraded":"2024-03-29T10:14:25.880415+0000","last_fullsized":"2024-03-29T10:14:25.880415+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:01:19.838234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.873725+0000","last_change":"2024-03-29T10:14:25.870197+0000","last_active":"2024-03-29T10:14:25.873725+0000","last_peered":"2024-03-29T10:14:25.873725+0000","last_clean":"2024-03-29T10:14:25.873725+0000","last_became_active":"2024-03-29T10:14:25.869971+0000","last_became_peered":"2024-03-29T10:14:25.869971+0000","last_unstale":"2024-03-29T10:14:25.873725+0000","last_undegraded":"2024-03-29T10:14:25.873725+0000","last_fullsized":"2024-03-29T10:14:25.873725+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:01.585597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.861267+0000","last_change":"2024-03-29T10:14:25.859122+0000","last_active":"2024-03-29T10:14:25.861267+0000","last_peered":"2024-03-29T10:14:25.861267+0000","last_clean":"2024-03-29T10:14:25.861267+0000","last_became_active":"2024-03-29T10:14:25.858871+0000","last_became_peered":"2024-03-29T10:14:25.858871+0000","last_unstale":"2024-03-29T10:14:25.861267+0000","last_undegraded":"2024-03-29T10:14:25.861267+0000","last_fullsized":"2024-03-29T10:14:25.861267+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:43.171887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876002+0000","last_change":"2024-03-29T10:14:25.873283+0000","last_active":"2024-03-29T10:14:25.876002+0000","last_peered":"2024-03-29T10:14:25.876002+0000","last_clean":"2024-03-29T10:14:25.876002+0000","last_became_active":"2024-03-29T10:14:25.870284+0000","last_became_peered":"2024-03-29T10:14:25.870284+0000","last_unstale":"2024-03-29T10:14:25.876002+0000","last_undegraded":"2024-03-29T10:14:25.876002+0000","last_fullsized":"2024-03-29T10:14:25.876002+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:13.255891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.889015+0000","last_change":"2024-03-29T10:14:25.884050+0000","last_active":"2024-03-29T10:14:25.889015+0000","last_peered":"2024-03-29T10:14:25.889015+0000","last_clean":"2024-03-29T10:14:25.889015+0000","last_became_active":"2024-03-29T10:14:25.882246+0000","last_became_peered":"2024-03-29T10:14:25.882246+0000","last_unstale":"2024-03-29T10:14:25.889015+0000","last_undegraded":"2024-03-29T10:14:25.889015+0000","last_fullsized":"2024-03-29T10:14:25.889015+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:27:20.627138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.886650+0000","last_change":"2024-03-29T10:14:25.882652+0000","last_active":"2024-03-29T10:14:25.886650+0000","last_peered":"2024-03-29T10:14:25.886650+0000","last_clean":"2024-03-29T10:14:25.886650+0000","last_became_active":"2024-03-29T10:14:25.880731+0000","last_became_peered":"2024-03-29T10:14:25.880731+0000","last_unstale":"2024-03-29T10:14:25.886650+0000","last_undegraded":"2024-03-29T10:14:25.886650+0000","last_fullsized":"2024-03-29T10:14:25.886650+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:02:32.687768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.867082+0000","last_change":"2024-03-29T10:14:25.862629+0000","last_active":"2024-03-29T10:14:25.867082+0000","last_peered":"2024-03-29T10:14:25.867082+0000","last_clean":"2024-03-29T10:14:25.867082+0000","last_became_active":"2024-03-29T10:14:25.861745+0000","last_became_peered":"2024-03-29T10:14:25.861745+0000","last_unstale":"2024-03-29T10:14:25.867082+0000","last_undegraded":"2024-03-29T10:14:25.867082+0000","last_fullsized":"2024-03-29T10:14:25.867082+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:26:15.665510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883445+0000","last_change":"2024-03-29T10:14:25.878006+0000","last_active":"2024-03-29T10:14:25.883445+0000","last_peered":"2024-03-29T10:14:25.883445+0000","last_clean":"2024-03-29T10:14:25.883445+0000","last_became_active":"2024-03-29T10:14:25.868410+0000","last_became_peered":"2024-03-29T10:14:25.868410+0000","last_unstale":"2024-03-29T10:14:25.883445+0000","last_undegraded":"2024-03-29T10:14:25.883445+0000","last_fullsized":"2024-03-29T10:14:25.883445+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:58:02.323015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881107+0000","last_change":"2024-03-29T10:14:25.880218+0000","last_active":"2024-03-29T10:14:25.881107+0000","last_peered":"2024-03-29T10:14:25.881107+0000","last_clean":"2024-03-29T10:14:25.881107+0000","last_became_active":"2024-03-29T10:14:25.879737+0000","last_became_peered":"2024-03-29T10:14:25.879737+0000","last_unstale":"2024-03-29T10:14:25.881107+0000","last_undegraded":"2024-03-29T10:14:25.881107+0000","last_fullsized":"2024-03-29T10:14:25.881107+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:13:14.177697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.875965+0000","last_change":"2024-03-29T10:14:25.873601+0000","last_active":"2024-03-29T10:14:25.875965+0000","last_peered":"2024-03-29T10:14:25.875965+0000","last_clean":"2024-03-29T10:14:25.875965+0000","last_became_active":"2024-03-29T10:14:25.873336+0000","last_became_peered":"2024-03-29T10:14:25.873336+0000","last_unstale":"2024-03-29T10:14:25.875965+0000","last_undegraded":"2024-03-29T10:14:25.875965+0000","last_fullsized":"2024-03-29T10:14:25.875965+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:37.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.872756+0000","last_change":"2024-03-29T10:14:25.870756+0000","last_active":"2024-03-29T10:14:25.872756+0000","last_peered":"2024-03-29T10:14:25.872756+0000","last_clean":"2024-03-29T10:14:25.872756+0000","last_became_active":"2024-03-29T10:14:25.870498+0000","last_became_peered":"2024-03-29T10:14:25.870498+0000","last_unstale":"2024-03-29T10:14:25.872756+0000","last_undegraded":"2024-03-29T10:14:25.872756+0000","last_fullsized":"2024-03-29T10:14:25.872756+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:22.425840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.890614+0000","last_change":"2024-03-29T10:14:25.885361+0000","last_active":"2024-03-29T10:14:25.890614+0000","last_peered":"2024-03-29T10:14:25.890614+0000","last_clean":"2024-03-29T10:14:25.890614+0000","last_became_active":"2024-03-29T10:14:25.880011+0000","last_became_peered":"2024-03-29T10:14:25.880011+0000","last_unstale":"2024-03-29T10:14:25.890614+0000","last_undegraded":"2024-03-29T10:14:25.890614+0000","last_fullsized":"2024-03-29T10:14:25.890614+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:59.099058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.864279+0000","last_change":"2024-03-29T10:14:25.861476+0000","last_active":"2024-03-29T10:14:25.864279+0000","last_peered":"2024-03-29T10:14:25.864279+0000","last_clean":"2024-03-29T10:14:25.864279+0000","last_became_active":"2024-03-29T10:14:25.860982+0000","last_became_peered":"2024-03-29T10:14:25.860982+0000","last_unstale":"2024-03-29T10:14:25.864279+0000","last_undegraded":"2024-03-29T10:14:25.864279+0000","last_fullsized":"2024-03-29T10:14:25.864279+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:13:14.177697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.865629+0000","last_change":"2024-03-29T10:14:25.861551+0000","last_active":"2024-03-29T10:14:25.865629+0000","last_peered":"2024-03-29T10:14:25.865629+0000","last_clean":"2024-03-29T10:14:25.865629+0000","last_became_active":"2024-03-29T10:14:25.861202+0000","last_became_peered":"2024-03-29T10:14:25.861202+0000","last_unstale":"2024-03-29T10:14:25.865629+0000","last_undegraded":"2024-03-29T10:14:25.865629+0000","last_fullsized":"2024-03-29T10:14:25.865629+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:55.080683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.882009+0000","last_change":"2024-03-29T10:14:25.881182+0000","last_active":"2024-03-29T10:14:25.882009+0000","last_peered":"2024-03-29T10:14:25.882009+0000","last_clean":"2024-03-29T10:14:25.882009+0000","last_became_active":"2024-03-29T10:14:25.880836+0000","last_became_peered":"2024-03-29T10:14:25.880836+0000","last_unstale":"2024-03-29T10:14:25.882009+0000","last_undegraded":"2024-03-29T10:14:25.882009+0000","last_fullsized":"2024-03-29T10:14:25.882009+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:10.510455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.870933+0000","last_change":"2024-03-29T10:14:25.868963+0000","last_active":"2024-03-29T10:14:25.870933+0000","last_peered":"2024-03-29T10:14:25.870933+0000","last_clean":"2024-03-29T10:14:25.870933+0000","last_became_active":"2024-03-29T10:14:25.868416+0000","last_became_peered":"2024-03-29T10:14:25.868416+0000","last_unstale":"2024-03-29T10:14:25.870933+0000","last_undegraded":"2024-03-29T10:14:25.870933+0000","last_fullsized":"2024-03-29T10:14:25.870933+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:06.217126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876992+0000","last_change":"2024-03-29T10:14:25.874199+0000","last_active":"2024-03-29T10:14:25.876992+0000","last_peered":"2024-03-29T10:14:25.876992+0000","last_clean":"2024-03-29T10:14:25.876992+0000","last_became_active":"2024-03-29T10:14:25.872536+0000","last_became_peered":"2024-03-29T10:14:25.872536+0000","last_unstale":"2024-03-29T10:14:25.876992+0000","last_undegraded":"2024-03-29T10:14:25.876992+0000","last_fullsized":"2024-03-29T10:14:25.876992+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:19:52.655089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883049+0000","last_change":"2024-03-29T10:14:25.878199+0000","last_active":"2024-03-29T10:14:25.883049+0000","last_peered":"2024-03-29T10:14:25.883049+0000","last_clean":"2024-03-29T10:14:25.883049+0000","last_became_active":"2024-03-29T10:14:25.873311+0000","last_became_peered":"2024-03-29T10:14:25.873311+0000","last_unstale":"2024-03-29T10:14:25.883049+0000","last_undegraded":"2024-03-29T10:14:25.883049+0000","last_fullsized":"2024-03-29T10:14:25.883049+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:04.861706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.870095+0000","last_change":"2024-03-29T10:14:25.866401+0000","last_active":"2024-03-29T10:14:25.870095+0000","last_peered":"2024-03-29T10:14:25.870095+0000","last_clean":"2024-03-29T10:14:25.870095+0000","last_became_active":"2024-03-29T10:14:25.865088+0000","last_became_peered":"2024-03-29T10:14:25.865088+0000","last_unstale":"2024-03-29T10:14:25.870095+0000","last_undegraded":"2024-03-29T10:14:25.870095+0000","last_fullsized":"2024-03-29T10:14:25.870095+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:37.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881107+0000","last_change":"2024-03-29T10:14:25.875699+0000","last_active":"2024-03-29T10:14:25.881107+0000","last_peered":"2024-03-29T10:14:25.881107+0000","last_clean":"2024-03-29T10:14:25.881107+0000","last_became_active":"2024-03-29T10:14:25.875191+0000","last_became_peered":"2024-03-29T10:14:25.875191+0000","last_unstale":"2024-03-29T10:14:25.881107+0000","last_undegraded":"2024-03-29T10:14:25.881107+0000","last_fullsized":"2024-03-29T10:14:25.881107+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:35:59.383439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.864296+0000","last_change":"2024-03-29T10:14:25.861546+0000","last_active":"2024-03-29T10:14:25.864296+0000","last_peered":"2024-03-29T10:14:25.864296+0000","last_clean":"2024-03-29T10:14:25.864296+0000","last_became_active":"2024-03-29T10:14:25.861289+0000","last_became_peered":"2024-03-29T10:14:25.861289+0000","last_unstale":"2024-03-29T10:14:25.864296+0000","last_undegraded":"2024-03-29T10:14:25.864296+0000","last_fullsized":"2024-03-29T10:14:25.864296+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:44:59.298968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883665+0000","last_change":"2024-03-29T10:14:25.871938+0000","last_active":"2024-03-29T10:14:25.883665+0000","last_peered":"2024-03-29T10:14:25.883665+0000","last_clean":"2024-03-29T10:14:25.883665+0000","last_became_active":"2024-03-29T10:14:25.866346+0000","last_became_peered":"2024-03-29T10:14:25.866346+0000","last_unstale":"2024-03-29T10:14:25.883665+0000","last_undegraded":"2024-03-29T10:14:25.883665+0000","last_fullsized":"2024-03-29T10:14:25.883665+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:36:28.316001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.888553+0000","last_change":"2024-03-29T10:14:25.887571+0000","last_active":"2024-03-29T10:14:25.888553+0000","last_peered":"2024-03-29T10:14:25.888553+0000","last_clean":"2024-03-29T10:14:25.888553+0000","last_became_active":"2024-03-29T10:14:25.886953+0000","last_became_peered":"2024-03-29T10:14:25.886953+0000","last_unstale":"2024-03-29T10:14:25.888553+0000","last_undegraded":"2024-03-29T10:14:25.888553+0000","last_fullsized":"2024-03-29T10:14:25.888553+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:01.459468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.890876+0000","last_change":"2024-03-29T10:14:25.886547+0000","last_active":"2024-03-29T10:14:25.890876+0000","last_peered":"2024-03-29T10:14:25.890876+0000","last_clean":"2024-03-29T10:14:25.890876+0000","last_became_active":"2024-03-29T10:14:25.885699+0000","last_became_peered":"2024-03-29T10:14:25.885699+0000","last_unstale":"2024-03-29T10:14:25.890876+0000","last_undegraded":"2024-03-29T10:14:25.890876+0000","last_fullsized":"2024-03-29T10:14:25.890876+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:10.510455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.870715+0000","last_change":"2024-03-29T10:14:25.867983+0000","last_active":"2024-03-29T10:14:25.870715+0000","last_peered":"2024-03-29T10:14:25.870715+0000","last_clean":"2024-03-29T10:14:25.870715+0000","last_became_active":"2024-03-29T10:14:25.867706+0000","last_became_peered":"2024-03-29T10:14:25.867706+0000","last_unstale":"2024-03-29T10:14:25.870715+0000","last_undegraded":"2024-03-29T10:14:25.870715+0000","last_fullsized":"2024-03-29T10:14:25.870715+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:10.913510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876429+0000","last_change":"2024-03-29T10:14:25.874117+0000","last_active":"2024-03-29T10:14:25.876429+0000","last_peered":"2024-03-29T10:14:25.876429+0000","last_clean":"2024-03-29T10:14:25.876429+0000","last_became_active":"2024-03-29T10:14:25.873855+0000","last_became_peered":"2024-03-29T10:14:25.873855+0000","last_unstale":"2024-03-29T10:14:25.876429+0000","last_undegraded":"2024-03-29T10:14:25.876429+0000","last_fullsized":"2024-03-29T10:14:25.876429+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:27.658902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876916+0000","last_change":"2024-03-29T10:14:25.865378+0000","last_active":"2024-03-29T10:14:25.876916+0000","last_peered":"2024-03-29T10:14:25.876916+0000","last_clean":"2024-03-29T10:14:25.876916+0000","last_became_active":"2024-03-29T10:14:25.864097+0000","last_became_peered":"2024-03-29T10:14:25.864097+0000","last_unstale":"2024-03-29T10:14:25.876916+0000","last_undegraded":"2024-03-29T10:14:25.876916+0000","last_fullsized":"2024-03-29T10:14:25.876916+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:14:17.129081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.874200+0000","last_change":"2024-03-29T10:14:25.871957+0000","last_active":"2024-03-29T10:14:25.874200+0000","last_peered":"2024-03-29T10:14:25.874200+0000","last_clean":"2024-03-29T10:14:25.874200+0000","last_became_active":"2024-03-29T10:14:25.871563+0000","last_became_peered":"2024-03-29T10:14:25.871563+0000","last_unstale":"2024-03-29T10:14:25.874200+0000","last_undegraded":"2024-03-29T10:14:25.874200+0000","last_fullsized":"2024-03-29T10:14:25.874200+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:08.083096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.879182+0000","last_change":"2024-03-29T10:14:25.876949+0000","last_active":"2024-03-29T10:14:25.879182+0000","last_peered":"2024-03-29T10:14:25.879182+0000","last_clean":"2024-03-29T10:14:25.879182+0000","last_became_active":"2024-03-29T10:14:25.876651+0000","last_became_peered":"2024-03-29T10:14:25.876651+0000","last_unstale":"2024-03-29T10:14:25.879182+0000","last_undegraded":"2024-03-29T10:14:25.879182+0000","last_fullsized":"2024-03-29T10:14:25.879182+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:57:54.331499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"139'4","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.875660+0000","last_change":"2024-03-29T10:14:25.343003+0000","last_active":"2024-03-29T10:14:25.875660+0000","last_peered":"2024-03-29T10:14:25.875660+0000","last_clean":"2024-03-29T10:14:25.875660+0000","last_became_active":"2024-03-29T10:14:25.341921+0000","last_became_peered":"2024-03-29T10:14:25.341921+0000","last_unstale":"2024-03-29T10:14:25.875660+0000","last_undegraded":"2024-03-29T10:14:25.875660+0000","last_fullsized":"2024-03-29T10:14:25.875660+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:07.072327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.871048+0000","last_change":"2024-03-29T10:14:25.863769+0000","last_active":"2024-03-29T10:14:25.871048+0000","last_peered":"2024-03-29T10:14:25.871048+0000","last_clean":"2024-03-29T10:14:25.871048+0000","last_became_active":"2024-03-29T10:14:25.863509+0000","last_became_peered":"2024-03-29T10:14:25.863509+0000","last_unstale":"2024-03-29T10:14:25.871048+0000","last_undegraded":"2024-03-29T10:14:25.871048+0000","last_fullsized":"2024-03-29T10:14:25.871048+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:26:15.665510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.877777+0000","last_change":"2024-03-29T10:14:25.345960+0000","last_active":"2024-03-29T10:14:25.877777+0000","last_peered":"2024-03-29T10:14:25.877777+0000","last_clean":"2024-03-29T10:14:25.877777+0000","last_became_active":"2024-03-29T10:14:25.345587+0000","last_became_peered":"2024-03-29T10:14:25.345587+0000","last_unstale":"2024-03-29T10:14:25.877777+0000","last_undegraded":"2024-03-29T10:14:25.877777+0000","last_fullsized":"2024-03-29T10:14:25.877777+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:40:49.403379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.875018+0000","last_change":"2024-03-29T10:14:25.872632+0000","last_active":"2024-03-29T10:14:25.875018+0000","last_peered":"2024-03-29T10:14:25.875018+0000","last_clean":"2024-03-29T10:14:25.875018+0000","last_became_active":"2024-03-29T10:14:25.872399+0000","last_became_peered":"2024-03-29T10:14:25.872399+0000","last_unstale":"2024-03-29T10:14:25.875018+0000","last_undegraded":"2024-03-29T10:14:25.875018+0000","last_fullsized":"2024-03-29T10:14:25.875018+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:08.949979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.877195+0000","last_change":"2024-03-29T10:14:25.350578+0000","last_active":"2024-03-29T10:14:25.877195+0000","last_peered":"2024-03-29T10:14:25.877195+0000","last_clean":"2024-03-29T10:14:25.877195+0000","last_became_active":"2024-03-29T10:14:25.350248+0000","last_became_peered":"2024-03-29T10:14:25.350248+0000","last_unstale":"2024-03-29T10:14:25.877195+0000","last_undegraded":"2024-03-29T10:14:25.877195+0000","last_fullsized":"2024-03-29T10:14:25.877195+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:59.777306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881597+0000","last_change":"2024-03-29T10:14:25.881080+0000","last_active":"2024-03-29T10:14:25.881597+0000","last_peered":"2024-03-29T10:14:25.881597+0000","last_clean":"2024-03-29T10:14:25.881597+0000","last_became_active":"2024-03-29T10:14:25.880807+0000","last_became_peered":"2024-03-29T10:14:25.880807+0000","last_unstale":"2024-03-29T10:14:25.881597+0000","last_undegraded":"2024-03-29T10:14:25.881597+0000","last_fullsized":"2024-03-29T10:14:25.881597+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:10.913510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876234+0000","last_change":"2024-03-29T10:14:25.349861+0000","last_active":"2024-03-29T10:14:25.876234+0000","last_peered":"2024-03-29T10:14:25.876234+0000","last_clean":"2024-03-29T10:14:25.876234+0000","last_became_active":"2024-03-29T10:14:25.349596+0000","last_became_peered":"2024-03-29T10:14:25.349596+0000","last_unstale":"2024-03-29T10:14:25.876234+0000","last_undegraded":"2024-03-29T10:14:25.876234+0000","last_fullsized":"2024-03-29T10:14:25.876234+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:37:39.878435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883564+0000","last_change":"2024-03-29T10:14:25.871609+0000","last_active":"2024-03-29T10:14:25.883564+0000","last_peered":"2024-03-29T10:14:25.883564+0000","last_clean":"2024-03-29T10:14:25.883564+0000","last_became_active":"2024-03-29T10:14:25.871365+0000","last_became_peered":"2024-03-29T10:14:25.871365+0000","last_unstale":"2024-03-29T10:14:25.883564+0000","last_undegraded":"2024-03-29T10:14:25.883564+0000","last_fullsized":"2024-03-29T10:14:25.883564+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:14:12.633868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876106+0000","last_change":"2024-03-29T10:14:25.343716+0000","last_active":"2024-03-29T10:14:25.876106+0000","last_peered":"2024-03-29T10:14:25.876106+0000","last_clean":"2024-03-29T10:14:25.876106+0000","last_became_active":"2024-03-29T10:14:25.343257+0000","last_became_peered":"2024-03-29T10:14:25.343257+0000","last_unstale":"2024-03-29T10:14:25.876106+0000","last_undegraded":"2024-03-29T10:14:25.876106+0000","last_fullsized":"2024-03-29T10:14:25.876106+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:42:03.382085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.869960+0000","last_change":"2024-03-29T10:14:25.865974+0000","last_active":"2024-03-29T10:14:25.869960+0000","last_peered":"2024-03-29T10:14:25.869960+0000","last_clean":"2024-03-29T10:14:25.869960+0000","last_became_active":"2024-03-29T10:14:25.865729+0000","last_became_peered":"2024-03-29T10:14:25.865729+0000","last_unstale":"2024-03-29T10:14:25.869960+0000","last_undegraded":"2024-03-29T10:14:25.869960+0000","last_fullsized":"2024-03-29T10:14:25.869960+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:36:57.215001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883374+0000","last_change":"2024-03-29T10:14:25.339830+0000","last_active":"2024-03-29T10:14:25.883374+0000","last_peered":"2024-03-29T10:14:25.883374+0000","last_clean":"2024-03-29T10:14:25.883374+0000","last_became_active":"2024-03-29T10:14:25.338776+0000","last_became_peered":"2024-03-29T10:14:25.338776+0000","last_unstale":"2024-03-29T10:14:25.883374+0000","last_undegraded":"2024-03-29T10:14:25.883374+0000","last_fullsized":"2024-03-29T10:14:25.883374+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:29:57.943141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.882956+0000","last_change":"2024-03-29T10:14:25.870128+0000","last_active":"2024-03-29T10:14:25.882956+0000","last_peered":"2024-03-29T10:14:25.882956+0000","last_clean":"2024-03-29T10:14:25.882956+0000","last_became_active":"2024-03-29T10:14:25.869868+0000","last_became_peered":"2024-03-29T10:14:25.869868+0000","last_unstale":"2024-03-29T10:14:25.882956+0000","last_undegraded":"2024-03-29T10:14:25.882956+0000","last_fullsized":"2024-03-29T10:14:25.882956+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:56:36.965345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.858270+0000","last_change":"2024-03-29T10:14:25.337810+0000","last_active":"2024-03-29T10:14:25.858270+0000","last_peered":"2024-03-29T10:14:25.858270+0000","last_clean":"2024-03-29T10:14:25.858270+0000","last_became_active":"2024-03-29T10:14:25.337015+0000","last_became_peered":"2024-03-29T10:14:25.337015+0000","last_unstale":"2024-03-29T10:14:25.858270+0000","last_undegraded":"2024-03-29T10:14:25.858270+0000","last_fullsized":"2024-03-29T10:14:25.858270+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:39.178639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.863977+0000","last_change":"2024-03-29T10:14:25.859870+0000","last_active":"2024-03-29T10:14:25.863977+0000","last_peered":"2024-03-29T10:14:25.863977+0000","last_clean":"2024-03-29T10:14:25.863977+0000","last_became_active":"2024-03-29T10:14:25.859161+0000","last_became_peered":"2024-03-29T10:14:25.859161+0000","last_unstale":"2024-03-29T10:14:25.863977+0000","last_undegraded":"2024-03-29T10:14:25.863977+0000","last_fullsized":"2024-03-29T10:14:25.863977+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:33:34.318626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.850760+0000","last_change":"2024-03-29T10:14:25.344294+0000","last_active":"2024-03-29T10:14:25.850760+0000","last_peered":"2024-03-29T10:14:25.850760+0000","last_clean":"2024-03-29T10:14:25.850760+0000","last_became_active":"2024-03-29T10:14:25.342979+0000","last_became_peered":"2024-03-29T10:14:25.342979+0000","last_unstale":"2024-03-29T10:14:25.850760+0000","last_undegraded":"2024-03-29T10:14:25.850760+0000","last_fullsized":"2024-03-29T10:14:25.850760+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:35:58.372670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880393+0000","last_change":"2024-03-29T10:14:25.879282+0000","last_active":"2024-03-29T10:14:25.880393+0000","last_peered":"2024-03-29T10:14:25.880393+0000","last_clean":"2024-03-29T10:14:25.880393+0000","last_became_active":"2024-03-29T10:14:25.878842+0000","last_became_peered":"2024-03-29T10:14:25.878842+0000","last_unstale":"2024-03-29T10:14:25.880393+0000","last_undegraded":"2024-03-29T10:14:25.880393+0000","last_fullsized":"2024-03-29T10:14:25.880393+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:36:36.898588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.864423+0000","last_change":"2024-03-29T10:14:25.350143+0000","last_active":"2024-03-29T10:14:25.864423+0000","last_peered":"2024-03-29T10:14:25.864423+0000","last_clean":"2024-03-29T10:14:25.864423+0000","last_became_active":"2024-03-29T10:14:25.349900+0000","last_became_peered":"2024-03-29T10:14:25.349900+0000","last_unstale":"2024-03-29T10:14:25.864423+0000","last_undegraded":"2024-03-29T10:14:25.864423+0000","last_fullsized":"2024-03-29T10:14:25.864423+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:33:33.307856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.873170+0000","last_change":"2024-03-29T10:14:25.871150+0000","last_active":"2024-03-29T10:14:25.873170+0000","last_peered":"2024-03-29T10:14:25.873170+0000","last_clean":"2024-03-29T10:14:25.873170+0000","last_became_active":"2024-03-29T10:14:25.869686+0000","last_became_peered":"2024-03-29T10:14:25.869686+0000","last_unstale":"2024-03-29T10:14:25.873170+0000","last_undegraded":"2024-03-29T10:14:25.873170+0000","last_fullsized":"2024-03-29T10:14:25.873170+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:44:59.298968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.859035+0000","last_change":"2024-03-29T10:14:25.344378+0000","last_active":"2024-03-29T10:14:25.859035+0000","last_peered":"2024-03-29T10:14:25.859035+0000","last_clean":"2024-03-29T10:14:25.859035+0000","last_became_active":"2024-03-29T10:14:25.344086+0000","last_became_peered":"2024-03-29T10:14:25.344086+0000","last_unstale":"2024-03-29T10:14:25.859035+0000","last_undegraded":"2024-03-29T10:14:25.859035+0000","last_fullsized":"2024-03-29T10:14:25.859035+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:36:38.754686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880545+0000","last_change":"2024-03-29T10:14:25.874015+0000","last_active":"2024-03-29T10:14:25.880545+0000","last_peered":"2024-03-29T10:14:25.880545+0000","last_clean":"2024-03-29T10:14:25.880545+0000","last_became_active":"2024-03-29T10:14:25.873619+0000","last_became_peered":"2024-03-29T10:14:25.873619+0000","last_unstale":"2024-03-29T10:14:25.880545+0000","last_undegraded":"2024-03-29T10:14:25.880545+0000","last_fullsized":"2024-03-29T10:14:25.880545+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:06.217126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"139'3","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.890714+0000","last_change":"2024-03-29T10:14:25.343742+0000","last_active":"2024-03-29T10:14:25.890714+0000","last_peered":"2024-03-29T10:14:25.890714+0000","last_clean":"2024-03-29T10:14:25.890714+0000","last_became_active":"2024-03-29T10:14:25.343406+0000","last_became_peered":"2024-03-29T10:14:25.343406+0000","last_unstale":"2024-03-29T10:14:25.890714+0000","last_undegraded":"2024-03-29T10:14:25.890714+0000","last_fullsized":"2024-03-29T10:14:25.890714+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:03.850937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.865864+0000","last_change":"2024-03-29T10:14:25.863196+0000","last_active":"2024-03-29T10:14:25.865864+0000","last_peered":"2024-03-29T10:14:25.865864+0000","last_clean":"2024-03-29T10:14:25.865864+0000","last_became_active":"2024-03-29T10:14:25.862909+0000","last_became_peered":"2024-03-29T10:14:25.862909+0000","last_unstale":"2024-03-29T10:14:25.865864+0000","last_undegraded":"2024-03-29T10:14:25.865864+0000","last_fullsized":"2024-03-29T10:14:25.865864+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:23.961101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"139'6","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.893835+0000","last_change":"2024-03-29T10:14:25.339655+0000","last_active":"2024-03-29T10:14:25.893835+0000","last_peered":"2024-03-29T10:14:25.893835+0000","last_clean":"2024-03-29T10:14:25.893835+0000","last_became_active":"2024-03-29T10:14:25.339373+0000","last_became_peered":"2024-03-29T10:14:25.339373+0000","last_unstale":"2024-03-29T10:14:25.893835+0000","last_undegraded":"2024-03-29T10:14:25.893835+0000","last_fullsized":"2024-03-29T10:14:25.893835+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:05.206356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881329+0000","last_change":"2024-03-29T10:14:25.876506+0000","last_active":"2024-03-29T10:14:25.881329+0000","last_peered":"2024-03-29T10:14:25.881329+0000","last_clean":"2024-03-29T10:14:25.881329+0000","last_became_active":"2024-03-29T10:14:25.867691+0000","last_became_peered":"2024-03-29T10:14:25.867691+0000","last_unstale":"2024-03-29T10:14:25.881329+0000","last_undegraded":"2024-03-29T10:14:25.881329+0000","last_fullsized":"2024-03-29T10:14:25.881329+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:22.092543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.873572+0000","last_change":"2024-03-29T10:14:25.342193+0000","last_active":"2024-03-29T10:14:25.873572+0000","last_peered":"2024-03-29T10:14:25.873572+0000","last_clean":"2024-03-29T10:14:25.873572+0000","last_became_active":"2024-03-29T10:14:25.341097+0000","last_became_peered":"2024-03-29T10:14:25.341097+0000","last_unstale":"2024-03-29T10:14:25.873572+0000","last_undegraded":"2024-03-29T10:14:25.873572+0000","last_fullsized":"2024-03-29T10:14:25.873572+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:53:50.253055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.877611+0000","last_change":"2024-03-29T10:14:25.870098+0000","last_active":"2024-03-29T10:14:25.877611+0000","last_peered":"2024-03-29T10:14:25.877611+0000","last_clean":"2024-03-29T10:14:25.877611+0000","last_became_active":"2024-03-29T10:14:25.867813+0000","last_became_peered":"2024-03-29T10:14:25.867813+0000","last_unstale":"2024-03-29T10:14:25.877611+0000","last_undegraded":"2024-03-29T10:14:25.877611+0000","last_fullsized":"2024-03-29T10:14:25.877611+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:32:26.002710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.853703+0000","last_change":"2024-03-29T10:14:25.350247+0000","last_active":"2024-03-29T10:14:25.853703+0000","last_peered":"2024-03-29T10:14:25.853703+0000","last_clean":"2024-03-29T10:14:25.853703+0000","last_became_active":"2024-03-29T10:14:25.349895+0000","last_became_peered":"2024-03-29T10:14:25.349895+0000","last_unstale":"2024-03-29T10:14:25.853703+0000","last_undegraded":"2024-03-29T10:14:25.853703+0000","last_fullsized":"2024-03-29T10:14:25.853703+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:26:43.583820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880242+0000","last_change":"2024-03-29T10:14:25.872095+0000","last_active":"2024-03-29T10:14:25.880242+0000","last_peered":"2024-03-29T10:14:25.880242+0000","last_clean":"2024-03-29T10:14:25.880242+0000","last_became_active":"2024-03-29T10:14:25.869407+0000","last_became_peered":"2024-03-29T10:14:25.869407+0000","last_unstale":"2024-03-29T10:14:25.880242+0000","last_undegraded":"2024-03-29T10:14:25.880242+0000","last_fullsized":"2024-03-29T10:14:25.880242+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:10.913510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"139'8","reported_seq":18,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.889373+0000","last_change":"2024-03-29T10:14:25.353082+0000","last_active":"2024-03-29T10:14:25.889373+0000","last_peered":"2024-03-29T10:14:25.889373+0000","last_clean":"2024-03-29T10:14:25.889373+0000","last_became_active":"2024-03-29T10:14:25.352159+0000","last_became_peered":"2024-03-29T10:14:25.352159+0000","last_unstale":"2024-03-29T10:14:25.889373+0000","last_undegraded":"2024-03-29T10:14:25.889373+0000","last_fullsized":"2024-03-29T10:14:25.889373+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:42.161117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.887903+0000","last_change":"2024-03-29T10:14:25.884768+0000","last_active":"2024-03-29T10:14:25.887903+0000","last_peered":"2024-03-29T10:14:25.887903+0000","last_clean":"2024-03-29T10:14:25.887903+0000","last_became_active":"2024-03-29T10:14:25.883667+0000","last_became_peered":"2024-03-29T10:14:25.883667+0000","last_unstale":"2024-03-29T10:14:25.887903+0000","last_undegraded":"2024-03-29T10:14:25.887903+0000","last_fullsized":"2024-03-29T10:14:25.887903+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:40.189409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.889108+0000","last_change":"2024-03-29T10:14:25.345283+0000","last_active":"2024-03-29T10:14:25.889108+0000","last_peered":"2024-03-29T10:14:25.889108+0000","last_clean":"2024-03-29T10:14:25.889108+0000","last_became_active":"2024-03-29T10:14:25.345002+0000","last_became_peered":"2024-03-29T10:14:25.345002+0000","last_unstale":"2024-03-29T10:14:25.889108+0000","last_undegraded":"2024-03-29T10:14:25.889108+0000","last_fullsized":"2024-03-29T10:14:25.889108+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:36:56.204231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.888266+0000","last_change":"2024-03-29T10:14:25.887325+0000","last_active":"2024-03-29T10:14:25.888266+0000","last_peered":"2024-03-29T10:14:25.888266+0000","last_clean":"2024-03-29T10:14:25.888266+0000","last_became_active":"2024-03-29T10:14:25.886949+0000","last_became_peered":"2024-03-29T10:14:25.886949+0000","last_unstale":"2024-03-29T10:14:25.888266+0000","last_undegraded":"2024-03-29T10:14:25.888266+0000","last_fullsized":"2024-03-29T10:14:25.888266+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:01.646920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.878735+0000","last_change":"2024-03-29T10:14:25.341791+0000","last_active":"2024-03-29T10:14:25.878735+0000","last_peered":"2024-03-29T10:14:25.878735+0000","last_clean":"2024-03-29T10:14:25.878735+0000","last_became_active":"2024-03-29T10:14:25.341510+0000","last_became_peered":"2024-03-29T10:14:25.341510+0000","last_unstale":"2024-03-29T10:14:25.878735+0000","last_undegraded":"2024-03-29T10:14:25.878735+0000","last_fullsized":"2024-03-29T10:14:25.878735+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:00.636151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.884959+0000","last_change":"2024-03-29T10:14:25.884418+0000","last_active":"2024-03-29T10:14:25.884959+0000","last_peered":"2024-03-29T10:14:25.884959+0000","last_clean":"2024-03-29T10:14:25.884959+0000","last_became_active":"2024-03-29T10:14:25.884030+0000","last_became_peered":"2024-03-29T10:14:25.884030+0000","last_unstale":"2024-03-29T10:14:25.884959+0000","last_undegraded":"2024-03-29T10:14:25.884959+0000","last_fullsized":"2024-03-29T10:14:25.884959+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:04.861706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881950+0000","last_change":"2024-03-29T10:14:25.341668+0000","last_active":"2024-03-29T10:14:25.881950+0000","last_peered":"2024-03-29T10:14:25.881950+0000","last_clean":"2024-03-29T10:14:25.881950+0000","last_became_active":"2024-03-29T10:14:25.340775+0000","last_became_peered":"2024-03-29T10:14:25.340775+0000","last_unstale":"2024-03-29T10:14:25.881950+0000","last_undegraded":"2024-03-29T10:14:25.881950+0000","last_fullsized":"2024-03-29T10:14:25.881950+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:40:49.403379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.877022+0000","last_change":"2024-03-29T10:14:25.875801+0000","last_active":"2024-03-29T10:14:25.877022+0000","last_peered":"2024-03-29T10:14:25.877022+0000","last_clean":"2024-03-29T10:14:25.877022+0000","last_became_active":"2024-03-29T10:14:25.875522+0000","last_became_peered":"2024-03-29T10:14:25.875522+0000","last_unstale":"2024-03-29T10:14:25.877022+0000","last_undegraded":"2024-03-29T10:14:25.877022+0000","last_fullsized":"2024-03-29T10:14:25.877022+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:36:57.215001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.851760+0000","last_change":"2024-03-29T10:14:25.345506+0000","last_active":"2024-03-29T10:14:25.851760+0000","last_peered":"2024-03-29T10:14:25.851760+0000","last_clean":"2024-03-29T10:14:25.851760+0000","last_became_active":"2024-03-29T10:14:25.344798+0000","last_became_peered":"2024-03-29T10:14:25.344798+0000","last_unstale":"2024-03-29T10:14:25.851760+0000","last_undegraded":"2024-03-29T10:14:25.851760+0000","last_fullsized":"2024-03-29T10:14:25.851760+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:21.415071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876596+0000","last_change":"2024-03-29T10:14:25.874606+0000","last_active":"2024-03-29T10:14:25.876596+0000","last_peered":"2024-03-29T10:14:25.876596+0000","last_clean":"2024-03-29T10:14:25.876596+0000","last_became_active":"2024-03-29T10:14:25.871321+0000","last_became_peered":"2024-03-29T10:14:25.871321+0000","last_unstale":"2024-03-29T10:14:25.876596+0000","last_undegraded":"2024-03-29T10:14:25.876596+0000","last_fullsized":"2024-03-29T10:14:25.876596+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:39.396414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881690+0000","last_change":"2024-03-29T10:14:25.345818+0000","last_active":"2024-03-29T10:14:25.881690+0000","last_peered":"2024-03-29T10:14:25.881690+0000","last_clean":"2024-03-29T10:14:25.881690+0000","last_became_active":"2024-03-29T10:14:25.345241+0000","last_became_peered":"2024-03-29T10:14:25.345241+0000","last_unstale":"2024-03-29T10:14:25.881690+0000","last_undegraded":"2024-03-29T10:14:25.881690+0000","last_fullsized":"2024-03-29T10:14:25.881690+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:29.779446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883817+0000","last_change":"2024-03-29T10:14:25.878487+0000","last_active":"2024-03-29T10:14:25.883817+0000","last_peered":"2024-03-29T10:14:25.883817+0000","last_clean":"2024-03-29T10:14:25.883817+0000","last_became_active":"2024-03-29T10:14:25.874261+0000","last_became_peered":"2024-03-29T10:14:25.874261+0000","last_unstale":"2024-03-29T10:14:25.883817+0000","last_undegraded":"2024-03-29T10:14:25.883817+0000","last_fullsized":"2024-03-29T10:14:25.883817+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:52:48.761811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.860630+0000","last_change":"2024-03-29T10:14:25.351458+0000","last_active":"2024-03-29T10:14:25.860630+0000","last_peered":"2024-03-29T10:14:25.860630+0000","last_clean":"2024-03-29T10:14:25.860630+0000","last_became_active":"2024-03-29T10:14:25.351096+0000","last_became_peered":"2024-03-29T10:14:25.351096+0000","last_unstale":"2024-03-29T10:14:25.860630+0000","last_undegraded":"2024-03-29T10:14:25.860630+0000","last_fullsized":"2024-03-29T10:14:25.860630+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:00.448698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.877745+0000","last_change":"2024-03-29T10:14:25.877034+0000","last_active":"2024-03-29T10:14:25.877745+0000","last_peered":"2024-03-29T10:14:25.877745+0000","last_clean":"2024-03-29T10:14:25.877745+0000","last_became_active":"2024-03-29T10:14:25.876692+0000","last_became_peered":"2024-03-29T10:14:25.876692+0000","last_unstale":"2024-03-29T10:14:25.877745+0000","last_undegraded":"2024-03-29T10:14:25.877745+0000","last_fullsized":"2024-03-29T10:14:25.877745+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:19:52.655089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.862158+0000","last_change":"2024-03-29T10:14:25.345499+0000","last_active":"2024-03-29T10:14:25.862158+0000","last_peered":"2024-03-29T10:14:25.862158+0000","last_clean":"2024-03-29T10:14:25.862158+0000","last_became_active":"2024-03-29T10:14:25.345215+0000","last_became_peered":"2024-03-29T10:14:25.345215+0000","last_unstale":"2024-03-29T10:14:25.862158+0000","last_undegraded":"2024-03-29T10:14:25.862158+0000","last_fullsized":"2024-03-29T10:14:25.862158+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:17.989168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.888118+0000","last_change":"2024-03-29T10:14:25.884970+0000","last_active":"2024-03-29T10:14:25.888118+0000","last_peered":"2024-03-29T10:14:25.888118+0000","last_clean":"2024-03-29T10:14:25.888118+0000","last_became_active":"2024-03-29T10:14:25.884682+0000","last_became_peered":"2024-03-29T10:14:25.884682+0000","last_unstale":"2024-03-29T10:14:25.888118+0000","last_undegraded":"2024-03-29T10:14:25.888118+0000","last_fullsized":"2024-03-29T10:14:25.888118+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:35.441935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.850871+0000","last_change":"2024-03-29T10:14:25.348713+0000","last_active":"2024-03-29T10:14:25.850871+0000","last_peered":"2024-03-29T10:14:25.850871+0000","last_clean":"2024-03-29T10:14:25.850871+0000","last_became_active":"2024-03-29T10:14:25.348073+0000","last_became_peered":"2024-03-29T10:14:25.348073+0000","last_unstale":"2024-03-29T10:14:25.850871+0000","last_undegraded":"2024-03-29T10:14:25.850871+0000","last_fullsized":"2024-03-29T10:14:25.850871+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:32:24.991941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.874313+0000","last_change":"2024-03-29T10:14:25.871775+0000","last_active":"2024-03-29T10:14:25.874313+0000","last_peered":"2024-03-29T10:14:25.874313+0000","last_clean":"2024-03-29T10:14:25.874313+0000","last_became_active":"2024-03-29T10:14:25.871479+0000","last_became_peered":"2024-03-29T10:14:25.871479+0000","last_unstale":"2024-03-29T10:14:25.874313+0000","last_undegraded":"2024-03-29T10:14:25.874313+0000","last_fullsized":"2024-03-29T10:14:25.874313+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:12.627299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.888702+0000","last_change":"2024-03-29T10:14:25.338929+0000","last_active":"2024-03-29T10:14:25.888702+0000","last_peered":"2024-03-29T10:14:25.888702+0000","last_clean":"2024-03-29T10:14:25.888702+0000","last_became_active":"2024-03-29T10:14:25.338315+0000","last_became_peered":"2024-03-29T10:14:25.338315+0000","last_unstale":"2024-03-29T10:14:25.888702+0000","last_undegraded":"2024-03-29T10:14:25.888702+0000","last_fullsized":"2024-03-29T10:14:25.888702+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:38.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.868024+0000","last_change":"2024-03-29T10:14:25.861393+0000","last_active":"2024-03-29T10:14:25.868024+0000","last_peered":"2024-03-29T10:14:25.868024+0000","last_clean":"2024-03-29T10:14:25.868024+0000","last_became_active":"2024-03-29T10:14:25.861112+0000","last_became_peered":"2024-03-29T10:14:25.861112+0000","last_unstale":"2024-03-29T10:14:25.868024+0000","last_undegraded":"2024-03-29T10:14:25.868024+0000","last_fullsized":"2024-03-29T10:14:25.868024+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:44.105000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.855241+0000","last_change":"2024-03-29T10:14:25.338178+0000","last_active":"2024-03-29T10:14:25.855241+0000","last_peered":"2024-03-29T10:14:25.855241+0000","last_clean":"2024-03-29T10:14:25.855241+0000","last_became_active":"2024-03-29T10:14:25.337650+0000","last_became_peered":"2024-03-29T10:14:25.337650+0000","last_unstale":"2024-03-29T10:14:25.855241+0000","last_undegraded":"2024-03-29T10:14:25.855241+0000","last_fullsized":"2024-03-29T10:14:25.855241+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:05.125943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.882690+0000","last_change":"2024-03-29T10:14:25.870914+0000","last_active":"2024-03-29T10:14:25.882690+0000","last_peered":"2024-03-29T10:14:25.882690+0000","last_clean":"2024-03-29T10:14:25.882690+0000","last_became_active":"2024-03-29T10:14:25.870581+0000","last_became_peered":"2024-03-29T10:14:25.870581+0000","last_unstale":"2024-03-29T10:14:25.882690+0000","last_undegraded":"2024-03-29T10:14:25.882690+0000","last_fullsized":"2024-03-29T10:14:25.882690+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:18.634376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.851485+0000","last_change":"2024-03-29T10:14:25.345972+0000","last_active":"2024-03-29T10:14:25.851485+0000","last_peered":"2024-03-29T10:14:25.851485+0000","last_clean":"2024-03-29T10:14:25.851485+0000","last_became_active":"2024-03-29T10:14:25.345702+0000","last_became_peered":"2024-03-29T10:14:25.345702+0000","last_unstale":"2024-03-29T10:14:25.851485+0000","last_undegraded":"2024-03-29T10:14:25.851485+0000","last_fullsized":"2024-03-29T10:14:25.851485+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.081774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.860910+0000","last_change":"2024-03-29T10:14:25.858097+0000","last_active":"2024-03-29T10:14:25.860910+0000","last_peered":"2024-03-29T10:14:25.860910+0000","last_clean":"2024-03-29T10:14:25.860910+0000","last_became_active":"2024-03-29T10:14:25.857805+0000","last_became_peered":"2024-03-29T10:14:25.857805+0000","last_unstale":"2024-03-29T10:14:25.860910+0000","last_undegraded":"2024-03-29T10:14:25.860910+0000","last_fullsized":"2024-03-29T10:14:25.860910+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:37.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.861182+0000","last_change":"2024-03-29T10:14:25.337296+0000","last_active":"2024-03-29T10:14:25.861182+0000","last_peered":"2024-03-29T10:14:25.861182+0000","last_clean":"2024-03-29T10:14:25.861182+0000","last_became_active":"2024-03-29T10:14:25.336687+0000","last_became_peered":"2024-03-29T10:14:25.336687+0000","last_unstale":"2024-03-29T10:14:25.861182+0000","last_undegraded":"2024-03-29T10:14:25.861182+0000","last_fullsized":"2024-03-29T10:14:25.861182+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:53:14.993560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.863696+0000","last_change":"2024-03-29T10:14:25.861238+0000","last_active":"2024-03-29T10:14:25.863696+0000","last_peered":"2024-03-29T10:14:25.863696+0000","last_clean":"2024-03-29T10:14:25.863696+0000","last_became_active":"2024-03-29T10:14:25.860916+0000","last_became_peered":"2024-03-29T10:14:25.860916+0000","last_unstale":"2024-03-29T10:14:25.863696+0000","last_undegraded":"2024-03-29T10:14:25.863696+0000","last_fullsized":"2024-03-29T10:14:25.863696+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:12.627299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.852056+0000","last_change":"2024-03-29T10:14:25.340212+0000","last_active":"2024-03-29T10:14:25.852056+0000","last_peered":"2024-03-29T10:14:25.852056+0000","last_clean":"2024-03-29T10:14:25.852056+0000","last_became_active":"2024-03-29T10:14:25.339383+0000","last_became_peered":"2024-03-29T10:14:25.339383+0000","last_unstale":"2024-03-29T10:14:25.852056+0000","last_undegraded":"2024-03-29T10:14:25.852056+0000","last_fullsized":"2024-03-29T10:14:25.852056+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:07.939210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880275+0000","last_change":"2024-03-29T10:14:25.879282+0000","last_active":"2024-03-29T10:14:25.880275+0000","last_peered":"2024-03-29T10:14:25.880275+0000","last_clean":"2024-03-29T10:14:25.880275+0000","last_became_active":"2024-03-29T10:14:25.878957+0000","last_became_peered":"2024-03-29T10:14:25.878957+0000","last_unstale":"2024-03-29T10:14:25.880275+0000","last_undegraded":"2024-03-29T10:14:25.880275+0000","last_fullsized":"2024-03-29T10:14:25.880275+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:41.642317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.857389+0000","last_change":"2024-03-29T10:14:25.341807+0000","last_active":"2024-03-29T10:14:25.857389+0000","last_peered":"2024-03-29T10:14:25.857389+0000","last_clean":"2024-03-29T10:14:25.857389+0000","last_became_active":"2024-03-29T10:14:25.341544+0000","last_became_peered":"2024-03-29T10:14:25.341544+0000","last_unstale":"2024-03-29T10:14:25.857389+0000","last_undegraded":"2024-03-29T10:14:25.857389+0000","last_fullsized":"2024-03-29T10:14:25.857389+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:47:49.900383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.873889+0000","last_change":"2024-03-29T10:14:25.871632+0000","last_active":"2024-03-29T10:14:25.873889+0000","last_peered":"2024-03-29T10:14:25.873889+0000","last_clean":"2024-03-29T10:14:25.873889+0000","last_became_active":"2024-03-29T10:14:25.871387+0000","last_became_peered":"2024-03-29T10:14:25.871387+0000","last_unstale":"2024-03-29T10:14:25.873889+0000","last_undegraded":"2024-03-29T10:14:25.873889+0000","last_fullsized":"2024-03-29T10:14:25.873889+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:26:15.665510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.859786+0000","last_change":"2024-03-29T10:14:25.346138+0000","last_active":"2024-03-29T10:14:25.859786+0000","last_peered":"2024-03-29T10:14:25.859786+0000","last_clean":"2024-03-29T10:14:25.859786+0000","last_became_active":"2024-03-29T10:14:25.345864+0000","last_became_peered":"2024-03-29T10:14:25.345864+0000","last_unstale":"2024-03-29T10:14:25.859786+0000","last_undegraded":"2024-03-29T10:14:25.859786+0000","last_fullsized":"2024-03-29T10:14:25.859786+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:21:02.783143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.882124+0000","last_change":"2024-03-29T10:14:25.869942+0000","last_active":"2024-03-29T10:14:25.882124+0000","last_peered":"2024-03-29T10:14:25.882124+0000","last_clean":"2024-03-29T10:14:25.882124+0000","last_became_active":"2024-03-29T10:14:25.869680+0000","last_became_peered":"2024-03-29T10:14:25.869680+0000","last_unstale":"2024-03-29T10:14:25.882124+0000","last_undegraded":"2024-03-29T10:14:25.882124+0000","last_fullsized":"2024-03-29T10:14:25.882124+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:14:40.091724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.851409+0000","last_change":"2024-03-29T10:14:25.348725+0000","last_active":"2024-03-29T10:14:25.851409+0000","last_peered":"2024-03-29T10:14:25.851409+0000","last_clean":"2024-03-29T10:14:25.851409+0000","last_became_active":"2024-03-29T10:14:25.348450+0000","last_became_peered":"2024-03-29T10:14:25.348450+0000","last_unstale":"2024-03-29T10:14:25.851409+0000","last_undegraded":"2024-03-29T10:14:25.851409+0000","last_fullsized":"2024-03-29T10:14:25.851409+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:12.245122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880189+0000","last_change":"2024-03-29T10:14:25.870577+0000","last_active":"2024-03-29T10:14:25.880189+0000","last_peered":"2024-03-29T10:14:25.880189+0000","last_clean":"2024-03-29T10:14:25.880189+0000","last_became_active":"2024-03-29T10:14:25.870306+0000","last_became_peered":"2024-03-29T10:14:25.870306+0000","last_unstale":"2024-03-29T10:14:25.880189+0000","last_undegraded":"2024-03-29T10:14:25.880189+0000","last_fullsized":"2024-03-29T10:14:25.880189+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:13.255891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.853969+0000","last_change":"2024-03-29T10:14:25.344453+0000","last_active":"2024-03-29T10:14:25.853969+0000","last_peered":"2024-03-29T10:14:25.853969+0000","last_clean":"2024-03-29T10:14:25.853969+0000","last_became_active":"2024-03-29T10:14:25.344131+0000","last_became_peered":"2024-03-29T10:14:25.344131+0000","last_unstale":"2024-03-29T10:14:25.853969+0000","last_undegraded":"2024-03-29T10:14:25.853969+0000","last_fullsized":"2024-03-29T10:14:25.853969+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:00.574827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"128'245","reported_seq":728,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.340630+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:14:25.340630+0000","last_peered":"2024-03-29T10:14:25.340630+0000","last_clean":"2024-03-29T10:14:25.340630+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:14:25.340630+0000","last_undegraded":"2024-03-29T10:14:25.340630+0000","last_fullsized":"2024-03-29T10:14:25.340630+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":391,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":391,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245,"ondisk_log_size":245,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575080,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37468,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334372,"statfs":{"total":96636764160,"available":96598396928,"internally_reserved":0,"allocated":2306048,"data_stored":1408875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36043212},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.66300000000000003,"15min":0.70899999999999996},"min":{"1min":0.45300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.96099999999999997,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67600000000000005,"15min":0.70799999999999996},"min":{"1min":0.45200000000000001,"5min":0.41699999999999998,"15min":0.377},"max":{"1min":0.94499999999999995,"5min":1.143,"15min":1.143},"last":0.59099999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70599999999999996,"15min":0.72299999999999998},"min":{"1min":0.50800000000000001,"5min":0.45700000000000002,"15min":0.44500000000000001},"max":{"1min":0.91000000000000003,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.52300000000000002,"5min":0.46000000000000002,"15min":0.41199999999999998},"max":{"1min":0.91500000000000004,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71299999999999997,"15min":0.73699999999999999},"min":{"1min":0.53500000000000003,"5min":0.47799999999999998,"15min":0.45200000000000001},"max":{"1min":0.92200000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.54000000000000004,"5min":0.49299999999999999,"15min":0.48799999999999999},"max":{"1min":0.98299999999999998,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.63100000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.71899999999999997,"15min":0.73999999999999999},"min":{"1min":0.55000000000000004,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.94299999999999995,"5min":1.171,"15min":1.171},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73499999999999999,"15min":0.77200000000000002},"min":{"1min":0.56799999999999995,"5min":0.52300000000000002,"15min":0.50700000000000001},"max":{"1min":0.89600000000000002,"5min":1.129,"15min":1.129},"last":0.73499999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.627,"15min":0.60599999999999998},"min":{"1min":0.48799999999999999,"5min":0.39500000000000002,"15min":0.38200000000000001},"max":{"1min":0.85199999999999998,"5min":1.25,"15min":1.25},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66200000000000003,"15min":0.64100000000000001},"min":{"1min":0.45000000000000001,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":0.95699999999999996,"5min":1.117,"15min":1.117},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.67400000000000004,"15min":0.65600000000000003},"min":{"1min":0.435,"5min":0.43099999999999999,"15min":0.40699999999999997},"max":{"1min":0.91400000000000003,"5min":1.47,"15min":1.47},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.53900000000000003,"5min":0.442,"15min":0.39400000000000002},"max":{"1min":0.97699999999999998,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.70999999999999996}]},{"osd":6,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.751,"15min":0.73099999999999998},"min":{"1min":0.55600000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.92400000000000004,"5min":1.222,"15min":1.222},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.746,"5min":0.73299999999999998,"15min":0.70899999999999996},"min":{"1min":0.52500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.94299999999999995,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.69799999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575081,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1712128,"data_stored":818507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69399999999999995,"15min":0.70699999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.876,"5min":2.7770000000000001,"15min":2.7770000000000001},"last":0.877},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.67700000000000005,"15min":0.71899999999999997},"min":{"1min":0.40999999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83899999999999997,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.76000000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65400000000000003,"15min":0.70099999999999996},"min":{"1min":0.56000000000000005,"5min":0.434,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69899999999999995,"15min":0.71999999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.89500000000000002,"5min":2.6709999999999998,"15min":2.6709999999999998},"last":0.68600000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.66100000000000003,"15min":0.70599999999999996},"min":{"1min":0.57899999999999996,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.877,"5min":1.256,"15min":1.256},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.71499999999999997,"15min":0.73899999999999999},"min":{"1min":0.56799999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.872,"5min":1.454,"15min":1.454},"last":0.91600000000000004}]},{"osd":3,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68500000000000005,"15min":0.73399999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":1.131,"15min":1.131},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73599999999999999,"15min":0.76400000000000001},"min":{"1min":0.47199999999999998,"5min":0.44500000000000001,"15min":0.39000000000000001},"max":{"1min":0.876,"5min":2.718,"15min":2.718},"last":0.748}]},{"osd":4,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.35199999999999998},"max":{"1min":0.78500000000000003,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.68000000000000005,"15min":0.68400000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.40600000000000003},"max":{"1min":0.75,"5min":2.544,"15min":2.544},"last":0.82499999999999996}]},{"osd":5,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65000000000000002,"15min":0.65100000000000002},"min":{"1min":0.45000000000000001,"5min":0.36499999999999999,"15min":0.34599999999999997},"max":{"1min":0.82599999999999996,"5min":3.2280000000000002,"15min":3.2280000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.67500000000000004,"15min":0.67800000000000005},"min":{"1min":0.46999999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78500000000000003,"5min":2.835,"15min":2.835},"last":0.80800000000000005}]},{"osd":7,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.72099999999999997,"15min":0.74199999999999999},"min":{"1min":0.54100000000000004,"5min":0.499,"15min":0.45100000000000001},"max":{"1min":0.82099999999999995,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.38900000000000001},"max":{"1min":0.79600000000000004,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.84599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575079,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54920,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316920,"statfs":{"total":96636764160,"available":96580526080,"internally_reserved":0,"allocated":1699840,"data_stored":812232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45400000000000001,"15min":0.45000000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.219},"max":{"1min":0.61499999999999999,"5min":0.81699999999999995,"15min":1.5600000000000001},"last":0.496},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.505,"15min":0.51400000000000001},"min":{"1min":0.35499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70499999999999996,"5min":0.71799999999999997,"15min":1.0860000000000001},"last":0.33700000000000002}]},{"osd":2,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.66500000000000004,"5min":1.222,"15min":1.7869999999999999},"last":0.37},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76900000000000002,"5min":1.04,"15min":1.04},"last":0.35599999999999998}]},{"osd":3,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57099999999999995,"15min":0.59199999999999997},"min":{"1min":0.39000000000000001,"5min":0.38600000000000001,"15min":0.377},"max":{"1min":0.70699999999999996,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53800000000000003,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71999999999999997,"5min":1.026,"15min":1.3049999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.76400000000000001,"15min":0.76300000000000001},"min":{"1min":0.63500000000000001,"5min":0.52900000000000003,"15min":0.497},"max":{"1min":1.1839999999999999,"5min":1.2709999999999999,"15min":1.617},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.78200000000000003,"15min":0.77500000000000002},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.48299999999999998},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.59999999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.157,"5min":1.157,"15min":1.3939999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.79800000000000004,"15min":0.79900000000000004},"min":{"1min":0.55900000000000005,"5min":0.55800000000000005,"15min":0.52900000000000003},"max":{"1min":1.292,"5min":1.3140000000000001,"15min":1.6579999999999999},"last":0.72199999999999998}]},{"osd":6,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.78700000000000003,"15min":0.79500000000000004},"min":{"1min":0.58499999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.754},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.67500000000000004,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.319,"5min":1.319,"15min":1.468},"last":0.61199999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.82599999999999996,"15min":0.83399999999999996},"min":{"1min":0.65500000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.109,"5min":1.1319999999999999,"15min":1.6000000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.82999999999999996,"15min":0.83099999999999996},"min":{"1min":0.64900000000000002,"5min":0.58399999999999996,"15min":0.57099999999999995},"max":{"1min":1.3560000000000001,"5min":1.4530000000000001,"15min":1.6919999999999999},"last":0.58399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37460,"kb_used_data":2248,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334380,"statfs":{"total":96636764160,"available":96598405120,"internally_reserved":0,"allocated":2301952,"data_stored":1407856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.438,"15min":0.45600000000000002},"min":{"1min":0.34499999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.68200000000000005,"5min":0.72299999999999998,"15min":0.93300000000000005},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.43099999999999999,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.76300000000000001,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53200000000000003,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71299999999999997,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.49299999999999999,"15min":0.498},"min":{"1min":0.379,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73399999999999999,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.441}]},{"osd":3,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54100000000000004,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.84899999999999998,"15min":1.077},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59399999999999997,"15min":0.61199999999999999},"min":{"1min":0.51000000000000001,"5min":0.40400000000000003,"15min":0.39100000000000001},"max":{"1min":0.86399999999999999,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.52500000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.78000000000000003,"15min":0.80300000000000005},"min":{"1min":0.55200000000000005,"5min":0.499,"15min":0.39200000000000002},"max":{"1min":1.254,"5min":1.254,"15min":1.625},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.80700000000000005,"15min":0.82299999999999995},"min":{"1min":0.60199999999999998,"5min":0.5,"15min":0.45800000000000002},"max":{"1min":1.115,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.84599999999999997}]},{"osd":5,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.74399999999999999,"15min":0.75900000000000001},"min":{"1min":0.63500000000000001,"5min":0.438,"15min":0.39000000000000001},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.872},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.77200000000000002,"15min":0.80000000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.41099999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.5309999999999999},"last":0.79600000000000004}]},{"osd":6,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.81799999999999995,"15min":0.82399999999999995},"min":{"1min":0.65500000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1519999999999999,"5min":1.476,"15min":1.476},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.81799999999999995,"15min":0.84199999999999997},"min":{"1min":0.5,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.601},"last":0.85399999999999998}]},{"osd":7,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.78200000000000003,"15min":0.79700000000000004},"min":{"1min":0.58799999999999997,"5min":0.48599999999999999,"15min":0.44900000000000001},"max":{"1min":1.238,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.81299999999999994,"15min":0.83899999999999997},"min":{"1min":0.625,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.294,"5min":1.296,"15min":1.5669999999999999},"last":0.91600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575080,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40984,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330856,"statfs":{"total":96636764160,"available":96594796544,"internally_reserved":0,"allocated":1712128,"data_stored":818507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.28899999999999998,"5min":0.251,"15min":0.248},"max":{"1min":0.64500000000000002,"5min":1.236,"15min":1.236},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.499,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":0.76000000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.44800000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48099999999999998,"15min":0.47399999999999998},"min":{"1min":0.313,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78500000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53600000000000003,"15min":0.55200000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":1.032},"last":0.37}]},{"osd":3,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57099999999999995,"15min":0.57799999999999996},"min":{"1min":0.42699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76500000000000001,"5min":0.86499999999999999,"15min":0.99099999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57099999999999995,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.73399999999999999,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.505}]},{"osd":4,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75600000000000001,"15min":0.77400000000000002},"min":{"1min":0.54500000000000004,"5min":0.51100000000000001,"15min":0.42799999999999999},"max":{"1min":0.92500000000000004,"5min":1.125,"15min":1.3520000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.755,"15min":0.78700000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.97399999999999998,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.77100000000000002,"15min":0.79600000000000004},"min":{"1min":0.624,"5min":0.52400000000000002,"15min":0.46999999999999997},"max":{"1min":1.093,"5min":1.1699999999999999,"15min":1.2589999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.78500000000000003,"15min":0.79700000000000004},"min":{"1min":0.57299999999999995,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.05,"5min":1.1919999999999999,"15min":1.2250000000000001},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.79700000000000004,"15min":0.81100000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":0.96899999999999997,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.79800000000000004,"15min":0.82699999999999996},"min":{"1min":0.58799999999999997,"5min":0.56200000000000006,"15min":0.55000000000000004},"max":{"1min":1.0229999999999999,"5min":1.1579999999999999,"15min":1.3560000000000001},"last":0.64800000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.82199999999999995,"15min":0.85399999999999998},"min":{"1min":0.57799999999999996,"5min":0.56899999999999995,"15min":0.53400000000000003},"max":{"1min":1.042,"5min":1.258,"15min":1.4490000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.85099999999999998,"15min":0.86899999999999999},"min":{"1min":0.70999999999999996,"5min":0.60899999999999999,"15min":0.56799999999999995},"max":{"1min":1.0640000000000001,"5min":1.226,"15min":1.4199999999999999},"last":0.68000000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1712128,"data_stored":818507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.432,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.26800000000000002,"15min":0.26500000000000001},"max":{"1min":0.67800000000000005,"5min":0.75600000000000001,"15min":0.81299999999999994},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.28399999999999997},"max":{"1min":0.68000000000000005,"5min":0.874,"15min":0.874},"last":0.75800000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.44400000000000001,"15min":0.47799999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.98699999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.437,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80400000000000005,"5min":0.82699999999999996,"15min":0.96099999999999997},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.54800000000000004,"15min":0.55800000000000005},"min":{"1min":0.42999999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.98299999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59499999999999997,"15min":0.625},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79400000000000004,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.60999999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72299999999999998,"15min":0.75600000000000001},"min":{"1min":0.504,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.93999999999999995,"5min":1.034,"15min":1.288},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.57899999999999996,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95599999999999996,"5min":1.093,"15min":1.46},"last":0.94199999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74299999999999999,"15min":0.77900000000000003},"min":{"1min":0.53500000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93600000000000005,"5min":1.1040000000000001,"15min":1.3580000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.76300000000000001,"15min":0.79200000000000004},"min":{"1min":0.60299999999999998,"5min":0.50700000000000001,"15min":0.46899999999999997},"max":{"1min":0.99099999999999999,"5min":1.1479999999999999,"15min":1.429},"last":0.63500000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.77800000000000002,"15min":0.80500000000000005},"min":{"1min":0.60399999999999998,"5min":0.54500000000000004,"15min":0.54200000000000004},"max":{"1min":1.0660000000000001,"5min":1.123,"15min":1.4730000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.749,"5min":0.76000000000000001,"15min":0.80300000000000005},"min":{"1min":0.56200000000000006,"5min":0.501,"15min":0.501},"max":{"1min":0.97299999999999998,"5min":1.0600000000000001,"15min":1.3999999999999999},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.78200000000000003,"15min":0.82099999999999995},"min":{"1min":0.64300000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.052,"5min":1.103,"15min":1.5029999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.81100000000000005,"15min":0.82999999999999996},"min":{"1min":0.69599999999999995,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.105,"5min":1.1859999999999999,"15min":1.345},"last":0.66100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575081,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59016,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312824,"statfs":{"total":96636764160,"available":96576331776,"internally_reserved":0,"allocated":1699840,"data_stored":812232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.69099999999999995},"min":{"1min":0.39400000000000002,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.88100000000000001,"5min":0.98799999999999999,"15min":1.417},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68100000000000005,"15min":0.69899999999999995},"min":{"1min":0.47499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.90100000000000002,"5min":1.0269999999999999,"15min":1.165},"last":0.67300000000000004}]},{"osd":1,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.70199999999999996},"min":{"1min":0.46500000000000002,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.98399999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69699999999999995,"15min":0.70999999999999996},"min":{"1min":0.51000000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.88600000000000001,"5min":1.075,"15min":1.333},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.76400000000000001,"15min":0.77500000000000002},"min":{"1min":0.64000000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.92300000000000004,"5min":1.014,"15min":1.03},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77800000000000002,"15min":0.77400000000000002},"min":{"1min":0.59499999999999997,"5min":0.50600000000000001,"15min":0.46200000000000002},"max":{"1min":0.95299999999999996,"5min":1.0600000000000001,"15min":1.387},"last":0.59299999999999997}]},{"osd":3,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.71499999999999997,"15min":0.72599999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95999999999999996,"5min":0.97499999999999998,"15min":1.2350000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71699999999999997,"15min":0.73399999999999999},"min":{"1min":0.52600000000000002,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.3180000000000001},"last":0.58199999999999996}]},{"osd":5,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61099999999999999,"15min":0.58899999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.38100000000000001},"max":{"1min":0.76700000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.628,"15min":0.60399999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.309},"max":{"1min":0.77400000000000002,"5min":0.98599999999999999,"15min":1.294},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.64200000000000002,"15min":0.621},"min":{"1min":0.47899999999999998,"5min":0.44,"15min":0.375},"max":{"1min":0.82299999999999995,"5min":1.026,"15min":1.026},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.67300000000000004,"15min":0.66100000000000003},"min":{"1min":0.46500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86199999999999999,"5min":1.093,"15min":1.1359999999999999},"last":0.51100000000000001}]},{"osd":7,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.68300000000000005,"15min":0.68200000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":1.04,"5min":1.0580000000000001,"15min":1.345},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77500000000000002,"15min":0.76100000000000001},"min":{"1min":0.54800000000000004,"5min":0.44,"15min":0.44},"max":{"1min":1.014,"5min":1.014,"15min":1.214},"last":0.62}]}]},{"osd":5,"up_from":13,"seq":55834575080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1712128,"data_stored":818507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.63600000000000001,"15min":0.67600000000000005},"min":{"1min":0.46000000000000002,"5min":0.36799999999999999,"15min":0.35699999999999998},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":1.1899999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.69999999999999996,"15min":0.72699999999999998},"min":{"1min":0.45700000000000002,"5min":0.38100000000000001,"15min":0.33500000000000002},"max":{"1min":0.96999999999999997,"5min":0.997,"15min":1.339},"last":0.68600000000000005}]},{"osd":1,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68799999999999994,"15min":0.71299999999999997},"min":{"1min":0.51200000000000001,"5min":0.42699999999999999,"15min":0.377},"max":{"1min":0.92300000000000004,"5min":0.96599999999999997,"15min":0.96699999999999997},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66200000000000003,"15min":0.70099999999999996},"min":{"1min":0.40200000000000002,"5min":0.39900000000000002,"15min":0.39200000000000002},"max":{"1min":0.93999999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.74399999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.70999999999999996,"15min":0.72899999999999998},"min":{"1min":0.53200000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98899999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.68200000000000005,"15min":0.70599999999999996},"min":{"1min":0.503,"5min":0.46200000000000002,"15min":0.39700000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":1.095},"last":0.83599999999999997}]},{"osd":3,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.69799999999999995,"15min":0.72599999999999998},"min":{"1min":0.55800000000000005,"5min":0.497,"15min":0.45600000000000002},"max":{"1min":0.97999999999999998,"5min":1.042,"15min":1.1419999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71799999999999997,"15min":0.746},"min":{"1min":0.53600000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.88200000000000001,"5min":0.98799999999999999,"15min":1.0980000000000001},"last":1.0029999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59999999999999998,"15min":0.60099999999999998},"min":{"1min":0.41499999999999998,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":0.87,"5min":0.91700000000000004,"15min":0.94599999999999995},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64800000000000002,"15min":0.64200000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.38900000000000001},"max":{"1min":0.877,"5min":1.097,"15min":1.097},"last":1.04}]},{"osd":6,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64700000000000002,"15min":0.65500000000000003},"min":{"1min":0.443,"5min":0.41299999999999998,"15min":0.36799999999999999},"max":{"1min":0.82499999999999996,"5min":0.93899999999999995,"15min":1.151},"last":1.083},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.66200000000000003,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.80800000000000005,"5min":0.95899999999999996,"15min":0.996},"last":0.95299999999999996}]},{"osd":7,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69499999999999995,"15min":0.70399999999999996},"min":{"1min":0.46899999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.96299999999999997,"5min":0.998,"15min":1.0289999999999999},"last":1.073},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71699999999999997,"15min":0.72999999999999998},"min":{"1min":0.52100000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.1990000000000001},"last":1.0549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:14:30.561 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:14:30.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.796+0000 7ff635c00700 1 Processor -- start 2024-03-29T10:14:30.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.796+0000 7ff635c00700 1 -- start start 2024-03-29T10:14:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.797+0000 7ff635c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff63015d640 0x7ff6300d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.797+0000 7ff635c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6301619d0 0x7ff6300b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.797+0000 7ff635c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6300d30d0 con 0x7ff6300ceec0 2024-03-29T10:14:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.797+0000 7ff635c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6300b6840 con 0x7ff63015d640 2024-03-29T10:14:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.797+0000 7ff635c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300d2880 con 0x7ff6301619d0 2024-03-29T10:14:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.798+0000 7ff62e000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff6300ceec0 0x7ff6301612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40122/0 (socket says 172.21.15.12:40122) 2024-03-29T10:14:30.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.798+0000 7ff62e000700 1 -- 172.21.15.12:0/3997967228 learned_addr learned my addr 172.21.15.12:0/3997967228 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.798+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845474116 0 0) 0x7ff6300d30d0 con 0x7ff6300ceec0 2024-03-29T10:14:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.798+0000 7ff62ea00700 1 --2- 172.21.15.12:0/3997967228 >> v2:172.21.15.159:3300/0 conn(0x7ff63015d640 0x7ff6300d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.798+0000 7ff62f400700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6301619d0 0x7ff6300b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.798+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6140035b0 con 0x7ff6300ceec0 2024-03-29T10:14:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.798+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3949147613 0 0) 0x7ff6140035b0 con 0x7ff6300ceec0 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.798+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6301619d0 msgr2=0x7ff6300b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6301619d0 0x7ff6300b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 >> v2:172.21.15.159:3300/0 conn(0x7ff63015d640 msgr2=0x7ff6300d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 --2- 172.21.15.12:0/3997967228 >> v2:172.21.15.159:3300/0 conn(0x7ff63015d640 0x7ff6300d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6300d2b30 con 0x7ff6300ceec0 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7ff61c001670 con 0x7ff6300ceec0 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 >> v1:172.21.15.12:6789/0 conn(0x7ff6300ceec0 legacy=0x7ff6301612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 0x7ff614003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff614004850 0x7ff614004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6301619d0 0x7ff614005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300b6840 con 0x7ff614003a00 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300d2880 con 0x7ff614004850 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61400ba40 con 0x7ff6301619d0 2024-03-29T10:14:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62ea00700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 0x7ff614003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62e000700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff614004850 0x7ff614004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62f400700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6301619d0 0x7ff614005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62ea00700 1 -- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6301619d0 msgr2=0x7ff614005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62ea00700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6301619d0 0x7ff614005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62ea00700 1 -- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff614004850 msgr2=0x7ff614004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62ea00700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff614004850 0x7ff614004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62ea00700 1 -- 172.21.15.12:0/3997967228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6140035b0 con 0x7ff614003a00 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.799+0000 7ff62e000700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff614004850 0x7ff614004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff62ea00700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 0x7ff614003e20 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7ff620009b60 tx=0x7ff614005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39c43c701ddc5a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000e660 con 0x7ff614003a00 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62000e7c0 con 0x7ff614003a00 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff62d600700 1 -- 172.21.15.12:0/3997967228 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000eb30 con 0x7ff614003a00 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff635c00700 1 -- 172.21.15.12:0/3997967228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 msgr2=0x7ff614003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff635c00700 1 --2- 172.21.15.12:0/3997967228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 0x7ff614003e20 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7ff620009b60 tx=0x7ff614005880 comp rx=0 tx=0).stop 2024-03-29T10:14:30.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff62f400700 1 -- 172.21.15.12:0/3997967228 reap_dead start 2024-03-29T10:14:30.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff635c00700 1 -- 172.21.15.12:0/3997967228 shutdown_connections 2024-03-29T10:14:30.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff635c00700 1 -- 172.21.15.12:0/3997967228 >> 172.21.15.12:0/3997967228 conn(0x7ff63008aff0 msgr2=0x7ff6300d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:30.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff635c00700 1 -- 172.21.15.12:0/3997967228 shutdown_connections 2024-03-29T10:14:30.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.800+0000 7ff635c00700 1 -- 172.21.15.12:0/3997967228 wait complete. 2024-03-29T10:14:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.802+0000 7ff635c00700 1 Processor -- start 2024-03-29T10:14:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.803+0000 7ff635c00700 1 -- start start 2024-03-29T10:14:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.803+0000 7ff635c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 0x7ff630154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.803+0000 7ff635c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff630160470 0x7ff630151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.803+0000 7ff635c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff63015ee10 0x7ff630152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff635c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63015f3d0 con 0x7ff614003a00 2024-03-29T10:14:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff635c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63015f6b0 con 0x7ff63015ee10 2024-03-29T10:14:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff635c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63015f990 con 0x7ff630160470 2024-03-29T10:14:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 0x7ff630154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 0x7ff630154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59024/0 (socket says 172.21.15.12:59024) 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62ea00700 1 -- 172.21.15.12:0/3428623890 learned_addr learned my addr 172.21.15.12:0/3428623890 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62e000700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff630160470 0x7ff630151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62f400700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff63015ee10 0x7ff630152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62ea00700 1 -- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff630160470 msgr2=0x7ff630151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62ea00700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff630160470 0x7ff630151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62ea00700 1 -- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff63015ee10 msgr2=0x7ff630152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62ea00700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff63015ee10 0x7ff630152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62ea00700 1 -- 172.21.15.12:0/3428623890 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63015fc70 con 0x7ff614003a00 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.804+0000 7ff62ea00700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 0x7ff630154760 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff614004770 tx=0x7ff6200047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.805+0000 7ff626a00700 1 -- 172.21.15.12:0/3428623890 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff620021070 con 0x7ff614003a00 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.805+0000 7ff626a00700 1 -- 172.21.15.12:0/3428623890 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6200045d0 con 0x7ff614003a00 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.805+0000 7ff626a00700 1 -- 172.21.15.12:0/3428623890 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000e660 con 0x7ff614003a00 2024-03-29T10:14:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.805+0000 7ff635c00700 1 -- 172.21.15.12:0/3428623890 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63015ff50 con 0x7ff614003a00 2024-03-29T10:14:30.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.805+0000 7ff635c00700 1 -- 172.21.15.12:0/3428623890 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6300ca0c0 con 0x7ff614003a00 2024-03-29T10:14:30.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.806+0000 7ff635c00700 1 -- 172.21.15.12:0/3428623890 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff630160110 con 0x7ff614003a00 2024-03-29T10:14:30.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.807+0000 7ff626a00700 1 -- 172.21.15.12:0/3428623890 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff620025070 con 0x7ff614003a00 2024-03-29T10:14:30.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.808+0000 7ff626a00700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff61006adb0 0x7ff61006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.808+0000 7ff626a00700 1 -- 172.21.15.12:0/3428623890 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff62008a510 con 0x7ff614003a00 2024-03-29T10:14:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.809+0000 7ff62e000700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff61006adb0 0x7ff61006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.809+0000 7ff62e000700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff61006adb0 0x7ff61006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff6300ceb10 tx=0x7ff61c009040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:30.813+0000 7ff626a00700 1 -- 172.21.15.12:0/3428623890 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff620055670 con 0x7ff614003a00 2024-03-29T10:14:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.026+0000 7ff635c00700 1 -- 172.21.15.12:0/3428623890 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6300b64d0 con 0x7ff61006adb0 2024-03-29T10:14:31.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.035+0000 7ff626a00700 1 -- 172.21.15.12:0/3428623890 <== mgr.8190 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289493 (secure 0 0 0) 0x7ff61c004870 con 0x7ff61006adb0 2024-03-29T10:14:31.038 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.039+0000 7ff635c00700 1 -- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff61006adb0 msgr2=0x7ff61006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.039+0000 7ff635c00700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff61006adb0 0x7ff61006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff6300ceb10 tx=0x7ff61c009040 comp rx=0 tx=0).stop 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.039+0000 7ff635c00700 1 -- 172.21.15.12:0/3428623890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 msgr2=0x7ff630154760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.039+0000 7ff635c00700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 0x7ff630154760 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff614004770 tx=0x7ff6200047c0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.039+0000 7ff635c00700 1 -- 172.21.15.12:0/3428623890 shutdown_connections 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.039+0000 7ff635c00700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff61006adb0 0x7ff61006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.039+0000 7ff635c00700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff63015ee10 0x7ff630152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.039+0000 7ff635c00700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff614003a00 0x7ff630154760 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.039+0000 7ff635c00700 1 --2- 172.21.15.12:0/3428623890 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff630160470 0x7ff630151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.039+0000 7ff635c00700 1 -- 172.21.15.12:0/3428623890 >> 172.21.15.12:0/3428623890 conn(0x7ff63008aff0 msgr2=0x7ff63015b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.040+0000 7ff635c00700 1 -- 172.21.15.12:0/3428623890 shutdown_connections 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.040+0000 7ff635c00700 1 -- 172.21.15.12:0/3428623890 wait complete. 2024-03-29T10:14:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:14:31.057 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-29T10:14:29.960771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":435,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283,"ondisk_log_size":283,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":340512,"kb_used_data":14508,"kb_used_omap":12,"kb_used_meta":325875,"kb_avail":754634208,"statfs":{"total":773094113280,"available":772745428992,"internally_reserved":0,"allocated":14856192,"data_stored":7715223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":333696569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.643587"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.882323+0000","last_change":"2024-03-29T10:14:25.867231+0000","last_active":"2024-03-29T10:14:25.882323+0000","last_peered":"2024-03-29T10:14:25.882323+0000","last_clean":"2024-03-29T10:14:25.882323+0000","last_became_active":"2024-03-29T10:14:25.866717+0000","last_became_peered":"2024-03-29T10:14:25.866717+0000","last_unstale":"2024-03-29T10:14:25.882323+0000","last_undegraded":"2024-03-29T10:14:25.882323+0000","last_fullsized":"2024-03-29T10:14:25.882323+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:47.090759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.870000+0000","last_change":"2024-03-29T10:14:25.865931+0000","last_active":"2024-03-29T10:14:25.870000+0000","last_peered":"2024-03-29T10:14:25.870000+0000","last_clean":"2024-03-29T10:14:25.870000+0000","last_became_active":"2024-03-29T10:14:25.865258+0000","last_became_peered":"2024-03-29T10:14:25.865258+0000","last_unstale":"2024-03-29T10:14:25.870000+0000","last_undegraded":"2024-03-29T10:14:25.870000+0000","last_fullsized":"2024-03-29T10:14:25.870000+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:35.441935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876611+0000","last_change":"2024-03-29T10:14:25.874133+0000","last_active":"2024-03-29T10:14:25.876611+0000","last_peered":"2024-03-29T10:14:25.876611+0000","last_clean":"2024-03-29T10:14:25.876611+0000","last_became_active":"2024-03-29T10:14:25.871258+0000","last_became_peered":"2024-03-29T10:14:25.871258+0000","last_unstale":"2024-03-29T10:14:25.876611+0000","last_undegraded":"2024-03-29T10:14:25.876611+0000","last_fullsized":"2024-03-29T10:14:25.876611+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:06.217126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.870796+0000","last_change":"2024-03-29T10:14:25.861248+0000","last_active":"2024-03-29T10:14:25.870796+0000","last_peered":"2024-03-29T10:14:25.870796+0000","last_clean":"2024-03-29T10:14:25.870796+0000","last_became_active":"2024-03-29T10:14:25.860968+0000","last_became_peered":"2024-03-29T10:14:25.860968+0000","last_unstale":"2024-03-29T10:14:25.870796+0000","last_undegraded":"2024-03-29T10:14:25.870796+0000","last_fullsized":"2024-03-29T10:14:25.870796+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:18:09.216626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880415+0000","last_change":"2024-03-29T10:14:25.877398+0000","last_active":"2024-03-29T10:14:25.880415+0000","last_peered":"2024-03-29T10:14:25.880415+0000","last_clean":"2024-03-29T10:14:25.880415+0000","last_became_active":"2024-03-29T10:14:25.873261+0000","last_became_peered":"2024-03-29T10:14:25.873261+0000","last_unstale":"2024-03-29T10:14:25.880415+0000","last_undegraded":"2024-03-29T10:14:25.880415+0000","last_fullsized":"2024-03-29T10:14:25.880415+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:01:19.838234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.873725+0000","last_change":"2024-03-29T10:14:25.870197+0000","last_active":"2024-03-29T10:14:25.873725+0000","last_peered":"2024-03-29T10:14:25.873725+0000","last_clean":"2024-03-29T10:14:25.873725+0000","last_became_active":"2024-03-29T10:14:25.869971+0000","last_became_peered":"2024-03-29T10:14:25.869971+0000","last_unstale":"2024-03-29T10:14:25.873725+0000","last_undegraded":"2024-03-29T10:14:25.873725+0000","last_fullsized":"2024-03-29T10:14:25.873725+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:01.585597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.861267+0000","last_change":"2024-03-29T10:14:25.859122+0000","last_active":"2024-03-29T10:14:25.861267+0000","last_peered":"2024-03-29T10:14:25.861267+0000","last_clean":"2024-03-29T10:14:25.861267+0000","last_became_active":"2024-03-29T10:14:25.858871+0000","last_became_peered":"2024-03-29T10:14:25.858871+0000","last_unstale":"2024-03-29T10:14:25.861267+0000","last_undegraded":"2024-03-29T10:14:25.861267+0000","last_fullsized":"2024-03-29T10:14:25.861267+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:43.171887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876002+0000","last_change":"2024-03-29T10:14:25.873283+0000","last_active":"2024-03-29T10:14:25.876002+0000","last_peered":"2024-03-29T10:14:25.876002+0000","last_clean":"2024-03-29T10:14:25.876002+0000","last_became_active":"2024-03-29T10:14:25.870284+0000","last_became_peered":"2024-03-29T10:14:25.870284+0000","last_unstale":"2024-03-29T10:14:25.876002+0000","last_undegraded":"2024-03-29T10:14:25.876002+0000","last_fullsized":"2024-03-29T10:14:25.876002+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:13.255891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.889015+0000","last_change":"2024-03-29T10:14:25.884050+0000","last_active":"2024-03-29T10:14:25.889015+0000","last_peered":"2024-03-29T10:14:25.889015+0000","last_clean":"2024-03-29T10:14:25.889015+0000","last_became_active":"2024-03-29T10:14:25.882246+0000","last_became_peered":"2024-03-29T10:14:25.882246+0000","last_unstale":"2024-03-29T10:14:25.889015+0000","last_undegraded":"2024-03-29T10:14:25.889015+0000","last_fullsized":"2024-03-29T10:14:25.889015+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:27:20.627138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.886650+0000","last_change":"2024-03-29T10:14:25.882652+0000","last_active":"2024-03-29T10:14:25.886650+0000","last_peered":"2024-03-29T10:14:25.886650+0000","last_clean":"2024-03-29T10:14:25.886650+0000","last_became_active":"2024-03-29T10:14:25.880731+0000","last_became_peered":"2024-03-29T10:14:25.880731+0000","last_unstale":"2024-03-29T10:14:25.886650+0000","last_undegraded":"2024-03-29T10:14:25.886650+0000","last_fullsized":"2024-03-29T10:14:25.886650+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:02:32.687768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.867082+0000","last_change":"2024-03-29T10:14:25.862629+0000","last_active":"2024-03-29T10:14:25.867082+0000","last_peered":"2024-03-29T10:14:25.867082+0000","last_clean":"2024-03-29T10:14:25.867082+0000","last_became_active":"2024-03-29T10:14:25.861745+0000","last_became_peered":"2024-03-29T10:14:25.861745+0000","last_unstale":"2024-03-29T10:14:25.867082+0000","last_undegraded":"2024-03-29T10:14:25.867082+0000","last_fullsized":"2024-03-29T10:14:25.867082+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:26:15.665510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883445+0000","last_change":"2024-03-29T10:14:25.878006+0000","last_active":"2024-03-29T10:14:25.883445+0000","last_peered":"2024-03-29T10:14:25.883445+0000","last_clean":"2024-03-29T10:14:25.883445+0000","last_became_active":"2024-03-29T10:14:25.868410+0000","last_became_peered":"2024-03-29T10:14:25.868410+0000","last_unstale":"2024-03-29T10:14:25.883445+0000","last_undegraded":"2024-03-29T10:14:25.883445+0000","last_fullsized":"2024-03-29T10:14:25.883445+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:58:02.323015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881107+0000","last_change":"2024-03-29T10:14:25.880218+0000","last_active":"2024-03-29T10:14:25.881107+0000","last_peered":"2024-03-29T10:14:25.881107+0000","last_clean":"2024-03-29T10:14:25.881107+0000","last_became_active":"2024-03-29T10:14:25.879737+0000","last_became_peered":"2024-03-29T10:14:25.879737+0000","last_unstale":"2024-03-29T10:14:25.881107+0000","last_undegraded":"2024-03-29T10:14:25.881107+0000","last_fullsized":"2024-03-29T10:14:25.881107+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:13:14.177697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.875965+0000","last_change":"2024-03-29T10:14:25.873601+0000","last_active":"2024-03-29T10:14:25.875965+0000","last_peered":"2024-03-29T10:14:25.875965+0000","last_clean":"2024-03-29T10:14:25.875965+0000","last_became_active":"2024-03-29T10:14:25.873336+0000","last_became_peered":"2024-03-29T10:14:25.873336+0000","last_unstale":"2024-03-29T10:14:25.875965+0000","last_undegraded":"2024-03-29T10:14:25.875965+0000","last_fullsized":"2024-03-29T10:14:25.875965+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:37.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.872756+0000","last_change":"2024-03-29T10:14:25.870756+0000","last_active":"2024-03-29T10:14:25.872756+0000","last_peered":"2024-03-29T10:14:25.872756+0000","last_clean":"2024-03-29T10:14:25.872756+0000","last_became_active":"2024-03-29T10:14:25.870498+0000","last_became_peered":"2024-03-29T10:14:25.870498+0000","last_unstale":"2024-03-29T10:14:25.872756+0000","last_undegraded":"2024-03-29T10:14:25.872756+0000","last_fullsized":"2024-03-29T10:14:25.872756+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:22.425840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.890614+0000","last_change":"2024-03-29T10:14:25.885361+0000","last_active":"2024-03-29T10:14:25.890614+0000","last_peered":"2024-03-29T10:14:25.890614+0000","last_clean":"2024-03-29T10:14:25.890614+0000","last_became_active":"2024-03-29T10:14:25.880011+0000","last_became_peered":"2024-03-29T10:14:25.880011+0000","last_unstale":"2024-03-29T10:14:25.890614+0000","last_undegraded":"2024-03-29T10:14:25.890614+0000","last_fullsized":"2024-03-29T10:14:25.890614+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:59.099058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.864279+0000","last_change":"2024-03-29T10:14:25.861476+0000","last_active":"2024-03-29T10:14:25.864279+0000","last_peered":"2024-03-29T10:14:25.864279+0000","last_clean":"2024-03-29T10:14:25.864279+0000","last_became_active":"2024-03-29T10:14:25.860982+0000","last_became_peered":"2024-03-29T10:14:25.860982+0000","last_unstale":"2024-03-29T10:14:25.864279+0000","last_undegraded":"2024-03-29T10:14:25.864279+0000","last_fullsized":"2024-03-29T10:14:25.864279+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:13:14.177697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.865629+0000","last_change":"2024-03-29T10:14:25.861551+0000","last_active":"2024-03-29T10:14:25.865629+0000","last_peered":"2024-03-29T10:14:25.865629+0000","last_clean":"2024-03-29T10:14:25.865629+0000","last_became_active":"2024-03-29T10:14:25.861202+0000","last_became_peered":"2024-03-29T10:14:25.861202+0000","last_unstale":"2024-03-29T10:14:25.865629+0000","last_undegraded":"2024-03-29T10:14:25.865629+0000","last_fullsized":"2024-03-29T10:14:25.865629+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:55.080683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.882009+0000","last_change":"2024-03-29T10:14:25.881182+0000","last_active":"2024-03-29T10:14:25.882009+0000","last_peered":"2024-03-29T10:14:25.882009+0000","last_clean":"2024-03-29T10:14:25.882009+0000","last_became_active":"2024-03-29T10:14:25.880836+0000","last_became_peered":"2024-03-29T10:14:25.880836+0000","last_unstale":"2024-03-29T10:14:25.882009+0000","last_undegraded":"2024-03-29T10:14:25.882009+0000","last_fullsized":"2024-03-29T10:14:25.882009+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:10.510455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.870933+0000","last_change":"2024-03-29T10:14:25.868963+0000","last_active":"2024-03-29T10:14:25.870933+0000","last_peered":"2024-03-29T10:14:25.870933+0000","last_clean":"2024-03-29T10:14:25.870933+0000","last_became_active":"2024-03-29T10:14:25.868416+0000","last_became_peered":"2024-03-29T10:14:25.868416+0000","last_unstale":"2024-03-29T10:14:25.870933+0000","last_undegraded":"2024-03-29T10:14:25.870933+0000","last_fullsized":"2024-03-29T10:14:25.870933+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:06.217126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876992+0000","last_change":"2024-03-29T10:14:25.874199+0000","last_active":"2024-03-29T10:14:25.876992+0000","last_peered":"2024-03-29T10:14:25.876992+0000","last_clean":"2024-03-29T10:14:25.876992+0000","last_became_active":"2024-03-29T10:14:25.872536+0000","last_became_peered":"2024-03-29T10:14:25.872536+0000","last_unstale":"2024-03-29T10:14:25.876992+0000","last_undegraded":"2024-03-29T10:14:25.876992+0000","last_fullsized":"2024-03-29T10:14:25.876992+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:19:52.655089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883049+0000","last_change":"2024-03-29T10:14:25.878199+0000","last_active":"2024-03-29T10:14:25.883049+0000","last_peered":"2024-03-29T10:14:25.883049+0000","last_clean":"2024-03-29T10:14:25.883049+0000","last_became_active":"2024-03-29T10:14:25.873311+0000","last_became_peered":"2024-03-29T10:14:25.873311+0000","last_unstale":"2024-03-29T10:14:25.883049+0000","last_undegraded":"2024-03-29T10:14:25.883049+0000","last_fullsized":"2024-03-29T10:14:25.883049+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:04.861706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.870095+0000","last_change":"2024-03-29T10:14:25.866401+0000","last_active":"2024-03-29T10:14:25.870095+0000","last_peered":"2024-03-29T10:14:25.870095+0000","last_clean":"2024-03-29T10:14:25.870095+0000","last_became_active":"2024-03-29T10:14:25.865088+0000","last_became_peered":"2024-03-29T10:14:25.865088+0000","last_unstale":"2024-03-29T10:14:25.870095+0000","last_undegraded":"2024-03-29T10:14:25.870095+0000","last_fullsized":"2024-03-29T10:14:25.870095+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:37.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881107+0000","last_change":"2024-03-29T10:14:25.875699+0000","last_active":"2024-03-29T10:14:25.881107+0000","last_peered":"2024-03-29T10:14:25.881107+0000","last_clean":"2024-03-29T10:14:25.881107+0000","last_became_active":"2024-03-29T10:14:25.875191+0000","last_became_peered":"2024-03-29T10:14:25.875191+0000","last_unstale":"2024-03-29T10:14:25.881107+0000","last_undegraded":"2024-03-29T10:14:25.881107+0000","last_fullsized":"2024-03-29T10:14:25.881107+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:35:59.383439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.864296+0000","last_change":"2024-03-29T10:14:25.861546+0000","last_active":"2024-03-29T10:14:25.864296+0000","last_peered":"2024-03-29T10:14:25.864296+0000","last_clean":"2024-03-29T10:14:25.864296+0000","last_became_active":"2024-03-29T10:14:25.861289+0000","last_became_peered":"2024-03-29T10:14:25.861289+0000","last_unstale":"2024-03-29T10:14:25.864296+0000","last_undegraded":"2024-03-29T10:14:25.864296+0000","last_fullsized":"2024-03-29T10:14:25.864296+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:44:59.298968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883665+0000","last_change":"2024-03-29T10:14:25.871938+0000","last_active":"2024-03-29T10:14:25.883665+0000","last_peered":"2024-03-29T10:14:25.883665+0000","last_clean":"2024-03-29T10:14:25.883665+0000","last_became_active":"2024-03-29T10:14:25.866346+0000","last_became_peered":"2024-03-29T10:14:25.866346+0000","last_unstale":"2024-03-29T10:14:25.883665+0000","last_undegraded":"2024-03-29T10:14:25.883665+0000","last_fullsized":"2024-03-29T10:14:25.883665+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:36:28.316001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.888553+0000","last_change":"2024-03-29T10:14:25.887571+0000","last_active":"2024-03-29T10:14:25.888553+0000","last_peered":"2024-03-29T10:14:25.888553+0000","last_clean":"2024-03-29T10:14:25.888553+0000","last_became_active":"2024-03-29T10:14:25.886953+0000","last_became_peered":"2024-03-29T10:14:25.886953+0000","last_unstale":"2024-03-29T10:14:25.888553+0000","last_undegraded":"2024-03-29T10:14:25.888553+0000","last_fullsized":"2024-03-29T10:14:25.888553+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:01.459468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.890876+0000","last_change":"2024-03-29T10:14:25.886547+0000","last_active":"2024-03-29T10:14:25.890876+0000","last_peered":"2024-03-29T10:14:25.890876+0000","last_clean":"2024-03-29T10:14:25.890876+0000","last_became_active":"2024-03-29T10:14:25.885699+0000","last_became_peered":"2024-03-29T10:14:25.885699+0000","last_unstale":"2024-03-29T10:14:25.890876+0000","last_undegraded":"2024-03-29T10:14:25.890876+0000","last_fullsized":"2024-03-29T10:14:25.890876+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:10.510455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.870715+0000","last_change":"2024-03-29T10:14:25.867983+0000","last_active":"2024-03-29T10:14:25.870715+0000","last_peered":"2024-03-29T10:14:25.870715+0000","last_clean":"2024-03-29T10:14:25.870715+0000","last_became_active":"2024-03-29T10:14:25.867706+0000","last_became_peered":"2024-03-29T10:14:25.867706+0000","last_unstale":"2024-03-29T10:14:25.870715+0000","last_undegraded":"2024-03-29T10:14:25.870715+0000","last_fullsized":"2024-03-29T10:14:25.870715+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:10.913510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876429+0000","last_change":"2024-03-29T10:14:25.874117+0000","last_active":"2024-03-29T10:14:25.876429+0000","last_peered":"2024-03-29T10:14:25.876429+0000","last_clean":"2024-03-29T10:14:25.876429+0000","last_became_active":"2024-03-29T10:14:25.873855+0000","last_became_peered":"2024-03-29T10:14:25.873855+0000","last_unstale":"2024-03-29T10:14:25.876429+0000","last_undegraded":"2024-03-29T10:14:25.876429+0000","last_fullsized":"2024-03-29T10:14:25.876429+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:27.658902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876916+0000","last_change":"2024-03-29T10:14:25.865378+0000","last_active":"2024-03-29T10:14:25.876916+0000","last_peered":"2024-03-29T10:14:25.876916+0000","last_clean":"2024-03-29T10:14:25.876916+0000","last_became_active":"2024-03-29T10:14:25.864097+0000","last_became_peered":"2024-03-29T10:14:25.864097+0000","last_unstale":"2024-03-29T10:14:25.876916+0000","last_undegraded":"2024-03-29T10:14:25.876916+0000","last_fullsized":"2024-03-29T10:14:25.876916+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:14:17.129081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.874200+0000","last_change":"2024-03-29T10:14:25.871957+0000","last_active":"2024-03-29T10:14:25.874200+0000","last_peered":"2024-03-29T10:14:25.874200+0000","last_clean":"2024-03-29T10:14:25.874200+0000","last_became_active":"2024-03-29T10:14:25.871563+0000","last_became_peered":"2024-03-29T10:14:25.871563+0000","last_unstale":"2024-03-29T10:14:25.874200+0000","last_undegraded":"2024-03-29T10:14:25.874200+0000","last_fullsized":"2024-03-29T10:14:25.874200+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:08.083096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.879182+0000","last_change":"2024-03-29T10:14:25.876949+0000","last_active":"2024-03-29T10:14:25.879182+0000","last_peered":"2024-03-29T10:14:25.879182+0000","last_clean":"2024-03-29T10:14:25.879182+0000","last_became_active":"2024-03-29T10:14:25.876651+0000","last_became_peered":"2024-03-29T10:14:25.876651+0000","last_unstale":"2024-03-29T10:14:25.879182+0000","last_undegraded":"2024-03-29T10:14:25.879182+0000","last_fullsized":"2024-03-29T10:14:25.879182+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:57:54.331499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"139'4","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.875660+0000","last_change":"2024-03-29T10:14:25.343003+0000","last_active":"2024-03-29T10:14:25.875660+0000","last_peered":"2024-03-29T10:14:25.875660+0000","last_clean":"2024-03-29T10:14:25.875660+0000","last_became_active":"2024-03-29T10:14:25.341921+0000","last_became_peered":"2024-03-29T10:14:25.341921+0000","last_unstale":"2024-03-29T10:14:25.875660+0000","last_undegraded":"2024-03-29T10:14:25.875660+0000","last_fullsized":"2024-03-29T10:14:25.875660+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:07.072327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.871048+0000","last_change":"2024-03-29T10:14:25.863769+0000","last_active":"2024-03-29T10:14:25.871048+0000","last_peered":"2024-03-29T10:14:25.871048+0000","last_clean":"2024-03-29T10:14:25.871048+0000","last_became_active":"2024-03-29T10:14:25.863509+0000","last_became_peered":"2024-03-29T10:14:25.863509+0000","last_unstale":"2024-03-29T10:14:25.871048+0000","last_undegraded":"2024-03-29T10:14:25.871048+0000","last_fullsized":"2024-03-29T10:14:25.871048+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:26:15.665510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.877777+0000","last_change":"2024-03-29T10:14:25.345960+0000","last_active":"2024-03-29T10:14:25.877777+0000","last_peered":"2024-03-29T10:14:25.877777+0000","last_clean":"2024-03-29T10:14:25.877777+0000","last_became_active":"2024-03-29T10:14:25.345587+0000","last_became_peered":"2024-03-29T10:14:25.345587+0000","last_unstale":"2024-03-29T10:14:25.877777+0000","last_undegraded":"2024-03-29T10:14:25.877777+0000","last_fullsized":"2024-03-29T10:14:25.877777+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:40:49.403379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.875018+0000","last_change":"2024-03-29T10:14:25.872632+0000","last_active":"2024-03-29T10:14:25.875018+0000","last_peered":"2024-03-29T10:14:25.875018+0000","last_clean":"2024-03-29T10:14:25.875018+0000","last_became_active":"2024-03-29T10:14:25.872399+0000","last_became_peered":"2024-03-29T10:14:25.872399+0000","last_unstale":"2024-03-29T10:14:25.875018+0000","last_undegraded":"2024-03-29T10:14:25.875018+0000","last_fullsized":"2024-03-29T10:14:25.875018+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:08.949979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.877195+0000","last_change":"2024-03-29T10:14:25.350578+0000","last_active":"2024-03-29T10:14:25.877195+0000","last_peered":"2024-03-29T10:14:25.877195+0000","last_clean":"2024-03-29T10:14:25.877195+0000","last_became_active":"2024-03-29T10:14:25.350248+0000","last_became_peered":"2024-03-29T10:14:25.350248+0000","last_unstale":"2024-03-29T10:14:25.877195+0000","last_undegraded":"2024-03-29T10:14:25.877195+0000","last_fullsized":"2024-03-29T10:14:25.877195+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:19:59.777306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881597+0000","last_change":"2024-03-29T10:14:25.881080+0000","last_active":"2024-03-29T10:14:25.881597+0000","last_peered":"2024-03-29T10:14:25.881597+0000","last_clean":"2024-03-29T10:14:25.881597+0000","last_became_active":"2024-03-29T10:14:25.880807+0000","last_became_peered":"2024-03-29T10:14:25.880807+0000","last_unstale":"2024-03-29T10:14:25.881597+0000","last_undegraded":"2024-03-29T10:14:25.881597+0000","last_fullsized":"2024-03-29T10:14:25.881597+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:10.913510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876234+0000","last_change":"2024-03-29T10:14:25.349861+0000","last_active":"2024-03-29T10:14:25.876234+0000","last_peered":"2024-03-29T10:14:25.876234+0000","last_clean":"2024-03-29T10:14:25.876234+0000","last_became_active":"2024-03-29T10:14:25.349596+0000","last_became_peered":"2024-03-29T10:14:25.349596+0000","last_unstale":"2024-03-29T10:14:25.876234+0000","last_undegraded":"2024-03-29T10:14:25.876234+0000","last_fullsized":"2024-03-29T10:14:25.876234+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:37:39.878435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883564+0000","last_change":"2024-03-29T10:14:25.871609+0000","last_active":"2024-03-29T10:14:25.883564+0000","last_peered":"2024-03-29T10:14:25.883564+0000","last_clean":"2024-03-29T10:14:25.883564+0000","last_became_active":"2024-03-29T10:14:25.871365+0000","last_became_peered":"2024-03-29T10:14:25.871365+0000","last_unstale":"2024-03-29T10:14:25.883564+0000","last_undegraded":"2024-03-29T10:14:25.883564+0000","last_fullsized":"2024-03-29T10:14:25.883564+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:14:12.633868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876106+0000","last_change":"2024-03-29T10:14:25.343716+0000","last_active":"2024-03-29T10:14:25.876106+0000","last_peered":"2024-03-29T10:14:25.876106+0000","last_clean":"2024-03-29T10:14:25.876106+0000","last_became_active":"2024-03-29T10:14:25.343257+0000","last_became_peered":"2024-03-29T10:14:25.343257+0000","last_unstale":"2024-03-29T10:14:25.876106+0000","last_undegraded":"2024-03-29T10:14:25.876106+0000","last_fullsized":"2024-03-29T10:14:25.876106+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:42:03.382085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.869960+0000","last_change":"2024-03-29T10:14:25.865974+0000","last_active":"2024-03-29T10:14:25.869960+0000","last_peered":"2024-03-29T10:14:25.869960+0000","last_clean":"2024-03-29T10:14:25.869960+0000","last_became_active":"2024-03-29T10:14:25.865729+0000","last_became_peered":"2024-03-29T10:14:25.865729+0000","last_unstale":"2024-03-29T10:14:25.869960+0000","last_undegraded":"2024-03-29T10:14:25.869960+0000","last_fullsized":"2024-03-29T10:14:25.869960+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:36:57.215001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883374+0000","last_change":"2024-03-29T10:14:25.339830+0000","last_active":"2024-03-29T10:14:25.883374+0000","last_peered":"2024-03-29T10:14:25.883374+0000","last_clean":"2024-03-29T10:14:25.883374+0000","last_became_active":"2024-03-29T10:14:25.338776+0000","last_became_peered":"2024-03-29T10:14:25.338776+0000","last_unstale":"2024-03-29T10:14:25.883374+0000","last_undegraded":"2024-03-29T10:14:25.883374+0000","last_fullsized":"2024-03-29T10:14:25.883374+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:29:57.943141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.882956+0000","last_change":"2024-03-29T10:14:25.870128+0000","last_active":"2024-03-29T10:14:25.882956+0000","last_peered":"2024-03-29T10:14:25.882956+0000","last_clean":"2024-03-29T10:14:25.882956+0000","last_became_active":"2024-03-29T10:14:25.869868+0000","last_became_peered":"2024-03-29T10:14:25.869868+0000","last_unstale":"2024-03-29T10:14:25.882956+0000","last_undegraded":"2024-03-29T10:14:25.882956+0000","last_fullsized":"2024-03-29T10:14:25.882956+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:56:36.965345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.858270+0000","last_change":"2024-03-29T10:14:25.337810+0000","last_active":"2024-03-29T10:14:25.858270+0000","last_peered":"2024-03-29T10:14:25.858270+0000","last_clean":"2024-03-29T10:14:25.858270+0000","last_became_active":"2024-03-29T10:14:25.337015+0000","last_became_peered":"2024-03-29T10:14:25.337015+0000","last_unstale":"2024-03-29T10:14:25.858270+0000","last_undegraded":"2024-03-29T10:14:25.858270+0000","last_fullsized":"2024-03-29T10:14:25.858270+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:39.178639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.863977+0000","last_change":"2024-03-29T10:14:25.859870+0000","last_active":"2024-03-29T10:14:25.863977+0000","last_peered":"2024-03-29T10:14:25.863977+0000","last_clean":"2024-03-29T10:14:25.863977+0000","last_became_active":"2024-03-29T10:14:25.859161+0000","last_became_peered":"2024-03-29T10:14:25.859161+0000","last_unstale":"2024-03-29T10:14:25.863977+0000","last_undegraded":"2024-03-29T10:14:25.863977+0000","last_fullsized":"2024-03-29T10:14:25.863977+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:33:34.318626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.850760+0000","last_change":"2024-03-29T10:14:25.344294+0000","last_active":"2024-03-29T10:14:25.850760+0000","last_peered":"2024-03-29T10:14:25.850760+0000","last_clean":"2024-03-29T10:14:25.850760+0000","last_became_active":"2024-03-29T10:14:25.342979+0000","last_became_peered":"2024-03-29T10:14:25.342979+0000","last_unstale":"2024-03-29T10:14:25.850760+0000","last_undegraded":"2024-03-29T10:14:25.850760+0000","last_fullsized":"2024-03-29T10:14:25.850760+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:35:58.372670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880393+0000","last_change":"2024-03-29T10:14:25.879282+0000","last_active":"2024-03-29T10:14:25.880393+0000","last_peered":"2024-03-29T10:14:25.880393+0000","last_clean":"2024-03-29T10:14:25.880393+0000","last_became_active":"2024-03-29T10:14:25.878842+0000","last_became_peered":"2024-03-29T10:14:25.878842+0000","last_unstale":"2024-03-29T10:14:25.880393+0000","last_undegraded":"2024-03-29T10:14:25.880393+0000","last_fullsized":"2024-03-29T10:14:25.880393+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:36:36.898588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.864423+0000","last_change":"2024-03-29T10:14:25.350143+0000","last_active":"2024-03-29T10:14:25.864423+0000","last_peered":"2024-03-29T10:14:25.864423+0000","last_clean":"2024-03-29T10:14:25.864423+0000","last_became_active":"2024-03-29T10:14:25.349900+0000","last_became_peered":"2024-03-29T10:14:25.349900+0000","last_unstale":"2024-03-29T10:14:25.864423+0000","last_undegraded":"2024-03-29T10:14:25.864423+0000","last_fullsized":"2024-03-29T10:14:25.864423+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:33:33.307856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.873170+0000","last_change":"2024-03-29T10:14:25.871150+0000","last_active":"2024-03-29T10:14:25.873170+0000","last_peered":"2024-03-29T10:14:25.873170+0000","last_clean":"2024-03-29T10:14:25.873170+0000","last_became_active":"2024-03-29T10:14:25.869686+0000","last_became_peered":"2024-03-29T10:14:25.869686+0000","last_unstale":"2024-03-29T10:14:25.873170+0000","last_undegraded":"2024-03-29T10:14:25.873170+0000","last_fullsized":"2024-03-29T10:14:25.873170+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:44:59.298968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.859035+0000","last_change":"2024-03-29T10:14:25.344378+0000","last_active":"2024-03-29T10:14:25.859035+0000","last_peered":"2024-03-29T10:14:25.859035+0000","last_clean":"2024-03-29T10:14:25.859035+0000","last_became_active":"2024-03-29T10:14:25.344086+0000","last_became_peered":"2024-03-29T10:14:25.344086+0000","last_unstale":"2024-03-29T10:14:25.859035+0000","last_undegraded":"2024-03-29T10:14:25.859035+0000","last_fullsized":"2024-03-29T10:14:25.859035+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:36:38.754686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880545+0000","last_change":"2024-03-29T10:14:25.874015+0000","last_active":"2024-03-29T10:14:25.880545+0000","last_peered":"2024-03-29T10:14:25.880545+0000","last_clean":"2024-03-29T10:14:25.880545+0000","last_became_active":"2024-03-29T10:14:25.873619+0000","last_became_peered":"2024-03-29T10:14:25.873619+0000","last_unstale":"2024-03-29T10:14:25.880545+0000","last_undegraded":"2024-03-29T10:14:25.880545+0000","last_fullsized":"2024-03-29T10:14:25.880545+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:06.217126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"139'3","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.890714+0000","last_change":"2024-03-29T10:14:25.343742+0000","last_active":"2024-03-29T10:14:25.890714+0000","last_peered":"2024-03-29T10:14:25.890714+0000","last_clean":"2024-03-29T10:14:25.890714+0000","last_became_active":"2024-03-29T10:14:25.343406+0000","last_became_peered":"2024-03-29T10:14:25.343406+0000","last_unstale":"2024-03-29T10:14:25.890714+0000","last_undegraded":"2024-03-29T10:14:25.890714+0000","last_fullsized":"2024-03-29T10:14:25.890714+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:03.850937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.865864+0000","last_change":"2024-03-29T10:14:25.863196+0000","last_active":"2024-03-29T10:14:25.865864+0000","last_peered":"2024-03-29T10:14:25.865864+0000","last_clean":"2024-03-29T10:14:25.865864+0000","last_became_active":"2024-03-29T10:14:25.862909+0000","last_became_peered":"2024-03-29T10:14:25.862909+0000","last_unstale":"2024-03-29T10:14:25.865864+0000","last_undegraded":"2024-03-29T10:14:25.865864+0000","last_fullsized":"2024-03-29T10:14:25.865864+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:07:23.961101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"139'6","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.893835+0000","last_change":"2024-03-29T10:14:25.339655+0000","last_active":"2024-03-29T10:14:25.893835+0000","last_peered":"2024-03-29T10:14:25.893835+0000","last_clean":"2024-03-29T10:14:25.893835+0000","last_became_active":"2024-03-29T10:14:25.339373+0000","last_became_peered":"2024-03-29T10:14:25.339373+0000","last_unstale":"2024-03-29T10:14:25.893835+0000","last_undegraded":"2024-03-29T10:14:25.893835+0000","last_fullsized":"2024-03-29T10:14:25.893835+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:05.206356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881329+0000","last_change":"2024-03-29T10:14:25.876506+0000","last_active":"2024-03-29T10:14:25.881329+0000","last_peered":"2024-03-29T10:14:25.881329+0000","last_clean":"2024-03-29T10:14:25.881329+0000","last_became_active":"2024-03-29T10:14:25.867691+0000","last_became_peered":"2024-03-29T10:14:25.867691+0000","last_unstale":"2024-03-29T10:14:25.881329+0000","last_undegraded":"2024-03-29T10:14:25.881329+0000","last_fullsized":"2024-03-29T10:14:25.881329+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:22.092543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.873572+0000","last_change":"2024-03-29T10:14:25.342193+0000","last_active":"2024-03-29T10:14:25.873572+0000","last_peered":"2024-03-29T10:14:25.873572+0000","last_clean":"2024-03-29T10:14:25.873572+0000","last_became_active":"2024-03-29T10:14:25.341097+0000","last_became_peered":"2024-03-29T10:14:25.341097+0000","last_unstale":"2024-03-29T10:14:25.873572+0000","last_undegraded":"2024-03-29T10:14:25.873572+0000","last_fullsized":"2024-03-29T10:14:25.873572+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:53:50.253055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.877611+0000","last_change":"2024-03-29T10:14:25.870098+0000","last_active":"2024-03-29T10:14:25.877611+0000","last_peered":"2024-03-29T10:14:25.877611+0000","last_clean":"2024-03-29T10:14:25.877611+0000","last_became_active":"2024-03-29T10:14:25.867813+0000","last_became_peered":"2024-03-29T10:14:25.867813+0000","last_unstale":"2024-03-29T10:14:25.877611+0000","last_undegraded":"2024-03-29T10:14:25.877611+0000","last_fullsized":"2024-03-29T10:14:25.877611+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:32:26.002710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.853703+0000","last_change":"2024-03-29T10:14:25.350247+0000","last_active":"2024-03-29T10:14:25.853703+0000","last_peered":"2024-03-29T10:14:25.853703+0000","last_clean":"2024-03-29T10:14:25.853703+0000","last_became_active":"2024-03-29T10:14:25.349895+0000","last_became_peered":"2024-03-29T10:14:25.349895+0000","last_unstale":"2024-03-29T10:14:25.853703+0000","last_undegraded":"2024-03-29T10:14:25.853703+0000","last_fullsized":"2024-03-29T10:14:25.853703+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:26:43.583820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880242+0000","last_change":"2024-03-29T10:14:25.872095+0000","last_active":"2024-03-29T10:14:25.880242+0000","last_peered":"2024-03-29T10:14:25.880242+0000","last_clean":"2024-03-29T10:14:25.880242+0000","last_became_active":"2024-03-29T10:14:25.869407+0000","last_became_peered":"2024-03-29T10:14:25.869407+0000","last_unstale":"2024-03-29T10:14:25.880242+0000","last_undegraded":"2024-03-29T10:14:25.880242+0000","last_fullsized":"2024-03-29T10:14:25.880242+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:20:10.913510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"139'8","reported_seq":18,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.889373+0000","last_change":"2024-03-29T10:14:25.353082+0000","last_active":"2024-03-29T10:14:25.889373+0000","last_peered":"2024-03-29T10:14:25.889373+0000","last_clean":"2024-03-29T10:14:25.889373+0000","last_became_active":"2024-03-29T10:14:25.352159+0000","last_became_peered":"2024-03-29T10:14:25.352159+0000","last_unstale":"2024-03-29T10:14:25.889373+0000","last_undegraded":"2024-03-29T10:14:25.889373+0000","last_fullsized":"2024-03-29T10:14:25.889373+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:42.161117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.887903+0000","last_change":"2024-03-29T10:14:25.884768+0000","last_active":"2024-03-29T10:14:25.887903+0000","last_peered":"2024-03-29T10:14:25.887903+0000","last_clean":"2024-03-29T10:14:25.887903+0000","last_became_active":"2024-03-29T10:14:25.883667+0000","last_became_peered":"2024-03-29T10:14:25.883667+0000","last_unstale":"2024-03-29T10:14:25.887903+0000","last_undegraded":"2024-03-29T10:14:25.887903+0000","last_fullsized":"2024-03-29T10:14:25.887903+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:40.189409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.889108+0000","last_change":"2024-03-29T10:14:25.345283+0000","last_active":"2024-03-29T10:14:25.889108+0000","last_peered":"2024-03-29T10:14:25.889108+0000","last_clean":"2024-03-29T10:14:25.889108+0000","last_became_active":"2024-03-29T10:14:25.345002+0000","last_became_peered":"2024-03-29T10:14:25.345002+0000","last_unstale":"2024-03-29T10:14:25.889108+0000","last_undegraded":"2024-03-29T10:14:25.889108+0000","last_fullsized":"2024-03-29T10:14:25.889108+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:36:56.204231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.888266+0000","last_change":"2024-03-29T10:14:25.887325+0000","last_active":"2024-03-29T10:14:25.888266+0000","last_peered":"2024-03-29T10:14:25.888266+0000","last_clean":"2024-03-29T10:14:25.888266+0000","last_became_active":"2024-03-29T10:14:25.886949+0000","last_became_peered":"2024-03-29T10:14:25.886949+0000","last_unstale":"2024-03-29T10:14:25.888266+0000","last_undegraded":"2024-03-29T10:14:25.888266+0000","last_fullsized":"2024-03-29T10:14:25.888266+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:01.646920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.878735+0000","last_change":"2024-03-29T10:14:25.341791+0000","last_active":"2024-03-29T10:14:25.878735+0000","last_peered":"2024-03-29T10:14:25.878735+0000","last_clean":"2024-03-29T10:14:25.878735+0000","last_became_active":"2024-03-29T10:14:25.341510+0000","last_became_peered":"2024-03-29T10:14:25.341510+0000","last_unstale":"2024-03-29T10:14:25.878735+0000","last_undegraded":"2024-03-29T10:14:25.878735+0000","last_fullsized":"2024-03-29T10:14:25.878735+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:00.636151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.884959+0000","last_change":"2024-03-29T10:14:25.884418+0000","last_active":"2024-03-29T10:14:25.884959+0000","last_peered":"2024-03-29T10:14:25.884959+0000","last_clean":"2024-03-29T10:14:25.884959+0000","last_became_active":"2024-03-29T10:14:25.884030+0000","last_became_peered":"2024-03-29T10:14:25.884030+0000","last_unstale":"2024-03-29T10:14:25.884959+0000","last_undegraded":"2024-03-29T10:14:25.884959+0000","last_fullsized":"2024-03-29T10:14:25.884959+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:04.861706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881950+0000","last_change":"2024-03-29T10:14:25.341668+0000","last_active":"2024-03-29T10:14:25.881950+0000","last_peered":"2024-03-29T10:14:25.881950+0000","last_clean":"2024-03-29T10:14:25.881950+0000","last_became_active":"2024-03-29T10:14:25.340775+0000","last_became_peered":"2024-03-29T10:14:25.340775+0000","last_unstale":"2024-03-29T10:14:25.881950+0000","last_undegraded":"2024-03-29T10:14:25.881950+0000","last_fullsized":"2024-03-29T10:14:25.881950+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:40:49.403379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.877022+0000","last_change":"2024-03-29T10:14:25.875801+0000","last_active":"2024-03-29T10:14:25.877022+0000","last_peered":"2024-03-29T10:14:25.877022+0000","last_clean":"2024-03-29T10:14:25.877022+0000","last_became_active":"2024-03-29T10:14:25.875522+0000","last_became_peered":"2024-03-29T10:14:25.875522+0000","last_unstale":"2024-03-29T10:14:25.877022+0000","last_undegraded":"2024-03-29T10:14:25.877022+0000","last_fullsized":"2024-03-29T10:14:25.877022+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:36:57.215001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.851760+0000","last_change":"2024-03-29T10:14:25.345506+0000","last_active":"2024-03-29T10:14:25.851760+0000","last_peered":"2024-03-29T10:14:25.851760+0000","last_clean":"2024-03-29T10:14:25.851760+0000","last_became_active":"2024-03-29T10:14:25.344798+0000","last_became_peered":"2024-03-29T10:14:25.344798+0000","last_unstale":"2024-03-29T10:14:25.851760+0000","last_undegraded":"2024-03-29T10:14:25.851760+0000","last_fullsized":"2024-03-29T10:14:25.851760+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:21.415071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.876596+0000","last_change":"2024-03-29T10:14:25.874606+0000","last_active":"2024-03-29T10:14:25.876596+0000","last_peered":"2024-03-29T10:14:25.876596+0000","last_clean":"2024-03-29T10:14:25.876596+0000","last_became_active":"2024-03-29T10:14:25.871321+0000","last_became_peered":"2024-03-29T10:14:25.871321+0000","last_unstale":"2024-03-29T10:14:25.876596+0000","last_undegraded":"2024-03-29T10:14:25.876596+0000","last_fullsized":"2024-03-29T10:14:25.876596+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:39.396414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.881690+0000","last_change":"2024-03-29T10:14:25.345818+0000","last_active":"2024-03-29T10:14:25.881690+0000","last_peered":"2024-03-29T10:14:25.881690+0000","last_clean":"2024-03-29T10:14:25.881690+0000","last_became_active":"2024-03-29T10:14:25.345241+0000","last_became_peered":"2024-03-29T10:14:25.345241+0000","last_unstale":"2024-03-29T10:14:25.881690+0000","last_undegraded":"2024-03-29T10:14:25.881690+0000","last_fullsized":"2024-03-29T10:14:25.881690+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:49:29.779446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.883817+0000","last_change":"2024-03-29T10:14:25.878487+0000","last_active":"2024-03-29T10:14:25.883817+0000","last_peered":"2024-03-29T10:14:25.883817+0000","last_clean":"2024-03-29T10:14:25.883817+0000","last_became_active":"2024-03-29T10:14:25.874261+0000","last_became_peered":"2024-03-29T10:14:25.874261+0000","last_unstale":"2024-03-29T10:14:25.883817+0000","last_undegraded":"2024-03-29T10:14:25.883817+0000","last_fullsized":"2024-03-29T10:14:25.883817+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:52:48.761811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.860630+0000","last_change":"2024-03-29T10:14:25.351458+0000","last_active":"2024-03-29T10:14:25.860630+0000","last_peered":"2024-03-29T10:14:25.860630+0000","last_clean":"2024-03-29T10:14:25.860630+0000","last_became_active":"2024-03-29T10:14:25.351096+0000","last_became_peered":"2024-03-29T10:14:25.351096+0000","last_unstale":"2024-03-29T10:14:25.860630+0000","last_undegraded":"2024-03-29T10:14:25.860630+0000","last_fullsized":"2024-03-29T10:14:25.860630+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:00.448698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.877745+0000","last_change":"2024-03-29T10:14:25.877034+0000","last_active":"2024-03-29T10:14:25.877745+0000","last_peered":"2024-03-29T10:14:25.877745+0000","last_clean":"2024-03-29T10:14:25.877745+0000","last_became_active":"2024-03-29T10:14:25.876692+0000","last_became_peered":"2024-03-29T10:14:25.876692+0000","last_unstale":"2024-03-29T10:14:25.877745+0000","last_undegraded":"2024-03-29T10:14:25.877745+0000","last_fullsized":"2024-03-29T10:14:25.877745+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:19:52.655089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.862158+0000","last_change":"2024-03-29T10:14:25.345499+0000","last_active":"2024-03-29T10:14:25.862158+0000","last_peered":"2024-03-29T10:14:25.862158+0000","last_clean":"2024-03-29T10:14:25.862158+0000","last_became_active":"2024-03-29T10:14:25.345215+0000","last_became_peered":"2024-03-29T10:14:25.345215+0000","last_unstale":"2024-03-29T10:14:25.862158+0000","last_undegraded":"2024-03-29T10:14:25.862158+0000","last_fullsized":"2024-03-29T10:14:25.862158+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:17.989168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.888118+0000","last_change":"2024-03-29T10:14:25.884970+0000","last_active":"2024-03-29T10:14:25.888118+0000","last_peered":"2024-03-29T10:14:25.888118+0000","last_clean":"2024-03-29T10:14:25.888118+0000","last_became_active":"2024-03-29T10:14:25.884682+0000","last_became_peered":"2024-03-29T10:14:25.884682+0000","last_unstale":"2024-03-29T10:14:25.888118+0000","last_undegraded":"2024-03-29T10:14:25.888118+0000","last_fullsized":"2024-03-29T10:14:25.888118+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:35.441935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.850871+0000","last_change":"2024-03-29T10:14:25.348713+0000","last_active":"2024-03-29T10:14:25.850871+0000","last_peered":"2024-03-29T10:14:25.850871+0000","last_clean":"2024-03-29T10:14:25.850871+0000","last_became_active":"2024-03-29T10:14:25.348073+0000","last_became_peered":"2024-03-29T10:14:25.348073+0000","last_unstale":"2024-03-29T10:14:25.850871+0000","last_undegraded":"2024-03-29T10:14:25.850871+0000","last_fullsized":"2024-03-29T10:14:25.850871+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:32:24.991941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.874313+0000","last_change":"2024-03-29T10:14:25.871775+0000","last_active":"2024-03-29T10:14:25.874313+0000","last_peered":"2024-03-29T10:14:25.874313+0000","last_clean":"2024-03-29T10:14:25.874313+0000","last_became_active":"2024-03-29T10:14:25.871479+0000","last_became_peered":"2024-03-29T10:14:25.871479+0000","last_unstale":"2024-03-29T10:14:25.874313+0000","last_undegraded":"2024-03-29T10:14:25.874313+0000","last_fullsized":"2024-03-29T10:14:25.874313+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:12.627299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.888702+0000","last_change":"2024-03-29T10:14:25.338929+0000","last_active":"2024-03-29T10:14:25.888702+0000","last_peered":"2024-03-29T10:14:25.888702+0000","last_clean":"2024-03-29T10:14:25.888702+0000","last_became_active":"2024-03-29T10:14:25.338315+0000","last_became_peered":"2024-03-29T10:14:25.338315+0000","last_unstale":"2024-03-29T10:14:25.888702+0000","last_undegraded":"2024-03-29T10:14:25.888702+0000","last_fullsized":"2024-03-29T10:14:25.888702+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:12:38.385644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.868024+0000","last_change":"2024-03-29T10:14:25.861393+0000","last_active":"2024-03-29T10:14:25.868024+0000","last_peered":"2024-03-29T10:14:25.868024+0000","last_clean":"2024-03-29T10:14:25.868024+0000","last_became_active":"2024-03-29T10:14:25.861112+0000","last_became_peered":"2024-03-29T10:14:25.861112+0000","last_unstale":"2024-03-29T10:14:25.868024+0000","last_undegraded":"2024-03-29T10:14:25.868024+0000","last_fullsized":"2024-03-29T10:14:25.868024+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:44.105000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.855241+0000","last_change":"2024-03-29T10:14:25.338178+0000","last_active":"2024-03-29T10:14:25.855241+0000","last_peered":"2024-03-29T10:14:25.855241+0000","last_clean":"2024-03-29T10:14:25.855241+0000","last_became_active":"2024-03-29T10:14:25.337650+0000","last_became_peered":"2024-03-29T10:14:25.337650+0000","last_unstale":"2024-03-29T10:14:25.855241+0000","last_undegraded":"2024-03-29T10:14:25.855241+0000","last_fullsized":"2024-03-29T10:14:25.855241+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:05.125943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.882690+0000","last_change":"2024-03-29T10:14:25.870914+0000","last_active":"2024-03-29T10:14:25.882690+0000","last_peered":"2024-03-29T10:14:25.882690+0000","last_clean":"2024-03-29T10:14:25.882690+0000","last_became_active":"2024-03-29T10:14:25.870581+0000","last_became_peered":"2024-03-29T10:14:25.870581+0000","last_unstale":"2024-03-29T10:14:25.882690+0000","last_undegraded":"2024-03-29T10:14:25.882690+0000","last_fullsized":"2024-03-29T10:14:25.882690+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:18.634376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.851485+0000","last_change":"2024-03-29T10:14:25.345972+0000","last_active":"2024-03-29T10:14:25.851485+0000","last_peered":"2024-03-29T10:14:25.851485+0000","last_clean":"2024-03-29T10:14:25.851485+0000","last_became_active":"2024-03-29T10:14:25.345702+0000","last_became_peered":"2024-03-29T10:14:25.345702+0000","last_unstale":"2024-03-29T10:14:25.851485+0000","last_undegraded":"2024-03-29T10:14:25.851485+0000","last_fullsized":"2024-03-29T10:14:25.851485+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.081774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.860910+0000","last_change":"2024-03-29T10:14:25.858097+0000","last_active":"2024-03-29T10:14:25.860910+0000","last_peered":"2024-03-29T10:14:25.860910+0000","last_clean":"2024-03-29T10:14:25.860910+0000","last_became_active":"2024-03-29T10:14:25.857805+0000","last_became_peered":"2024-03-29T10:14:25.857805+0000","last_unstale":"2024-03-29T10:14:25.860910+0000","last_undegraded":"2024-03-29T10:14:25.860910+0000","last_fullsized":"2024-03-29T10:14:25.860910+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:37.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.861182+0000","last_change":"2024-03-29T10:14:25.337296+0000","last_active":"2024-03-29T10:14:25.861182+0000","last_peered":"2024-03-29T10:14:25.861182+0000","last_clean":"2024-03-29T10:14:25.861182+0000","last_became_active":"2024-03-29T10:14:25.336687+0000","last_became_peered":"2024-03-29T10:14:25.336687+0000","last_unstale":"2024-03-29T10:14:25.861182+0000","last_undegraded":"2024-03-29T10:14:25.861182+0000","last_fullsized":"2024-03-29T10:14:25.861182+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:53:14.993560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.863696+0000","last_change":"2024-03-29T10:14:25.861238+0000","last_active":"2024-03-29T10:14:25.863696+0000","last_peered":"2024-03-29T10:14:25.863696+0000","last_clean":"2024-03-29T10:14:25.863696+0000","last_became_active":"2024-03-29T10:14:25.860916+0000","last_became_peered":"2024-03-29T10:14:25.860916+0000","last_unstale":"2024-03-29T10:14:25.863696+0000","last_undegraded":"2024-03-29T10:14:25.863696+0000","last_fullsized":"2024-03-29T10:14:25.863696+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:12.627299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.852056+0000","last_change":"2024-03-29T10:14:25.340212+0000","last_active":"2024-03-29T10:14:25.852056+0000","last_peered":"2024-03-29T10:14:25.852056+0000","last_clean":"2024-03-29T10:14:25.852056+0000","last_became_active":"2024-03-29T10:14:25.339383+0000","last_became_peered":"2024-03-29T10:14:25.339383+0000","last_unstale":"2024-03-29T10:14:25.852056+0000","last_undegraded":"2024-03-29T10:14:25.852056+0000","last_fullsized":"2024-03-29T10:14:25.852056+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:07.939210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880275+0000","last_change":"2024-03-29T10:14:25.879282+0000","last_active":"2024-03-29T10:14:25.880275+0000","last_peered":"2024-03-29T10:14:25.880275+0000","last_clean":"2024-03-29T10:14:25.880275+0000","last_became_active":"2024-03-29T10:14:25.878957+0000","last_became_peered":"2024-03-29T10:14:25.878957+0000","last_unstale":"2024-03-29T10:14:25.880275+0000","last_undegraded":"2024-03-29T10:14:25.880275+0000","last_fullsized":"2024-03-29T10:14:25.880275+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:41.642317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.857389+0000","last_change":"2024-03-29T10:14:25.341807+0000","last_active":"2024-03-29T10:14:25.857389+0000","last_peered":"2024-03-29T10:14:25.857389+0000","last_clean":"2024-03-29T10:14:25.857389+0000","last_became_active":"2024-03-29T10:14:25.341544+0000","last_became_peered":"2024-03-29T10:14:25.341544+0000","last_unstale":"2024-03-29T10:14:25.857389+0000","last_undegraded":"2024-03-29T10:14:25.857389+0000","last_fullsized":"2024-03-29T10:14:25.857389+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:47:49.900383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.873889+0000","last_change":"2024-03-29T10:14:25.871632+0000","last_active":"2024-03-29T10:14:25.873889+0000","last_peered":"2024-03-29T10:14:25.873889+0000","last_clean":"2024-03-29T10:14:25.873889+0000","last_became_active":"2024-03-29T10:14:25.871387+0000","last_became_peered":"2024-03-29T10:14:25.871387+0000","last_unstale":"2024-03-29T10:14:25.873889+0000","last_undegraded":"2024-03-29T10:14:25.873889+0000","last_fullsized":"2024-03-29T10:14:25.873889+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:26:15.665510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.859786+0000","last_change":"2024-03-29T10:14:25.346138+0000","last_active":"2024-03-29T10:14:25.859786+0000","last_peered":"2024-03-29T10:14:25.859786+0000","last_clean":"2024-03-29T10:14:25.859786+0000","last_became_active":"2024-03-29T10:14:25.345864+0000","last_became_peered":"2024-03-29T10:14:25.345864+0000","last_unstale":"2024-03-29T10:14:25.859786+0000","last_undegraded":"2024-03-29T10:14:25.859786+0000","last_fullsized":"2024-03-29T10:14:25.859786+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:21:02.783143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.882124+0000","last_change":"2024-03-29T10:14:25.869942+0000","last_active":"2024-03-29T10:14:25.882124+0000","last_peered":"2024-03-29T10:14:25.882124+0000","last_clean":"2024-03-29T10:14:25.882124+0000","last_became_active":"2024-03-29T10:14:25.869680+0000","last_became_peered":"2024-03-29T10:14:25.869680+0000","last_unstale":"2024-03-29T10:14:25.882124+0000","last_undegraded":"2024-03-29T10:14:25.882124+0000","last_fullsized":"2024-03-29T10:14:25.882124+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:14:40.091724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.851409+0000","last_change":"2024-03-29T10:14:25.348725+0000","last_active":"2024-03-29T10:14:25.851409+0000","last_peered":"2024-03-29T10:14:25.851409+0000","last_clean":"2024-03-29T10:14:25.851409+0000","last_became_active":"2024-03-29T10:14:25.348450+0000","last_became_peered":"2024-03-29T10:14:25.348450+0000","last_unstale":"2024-03-29T10:14:25.851409+0000","last_undegraded":"2024-03-29T10:14:25.851409+0000","last_fullsized":"2024-03-29T10:14:25.851409+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:12.245122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.880189+0000","last_change":"2024-03-29T10:14:25.870577+0000","last_active":"2024-03-29T10:14:25.880189+0000","last_peered":"2024-03-29T10:14:25.880189+0000","last_clean":"2024-03-29T10:14:25.880189+0000","last_became_active":"2024-03-29T10:14:25.870306+0000","last_became_peered":"2024-03-29T10:14:25.870306+0000","last_unstale":"2024-03-29T10:14:25.880189+0000","last_undegraded":"2024-03-29T10:14:25.880189+0000","last_fullsized":"2024-03-29T10:14:25.880189+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:25.316501+0000","last_clean_scrub_stamp":"2024-03-29T10:14:25.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:13.255891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.853969+0000","last_change":"2024-03-29T10:14:25.344453+0000","last_active":"2024-03-29T10:14:25.853969+0000","last_peered":"2024-03-29T10:14:25.853969+0000","last_clean":"2024-03-29T10:14:25.853969+0000","last_became_active":"2024-03-29T10:14:25.344131+0000","last_became_peered":"2024-03-29T10:14:25.344131+0000","last_unstale":"2024-03-29T10:14:25.853969+0000","last_undegraded":"2024-03-29T10:14:25.853969+0000","last_fullsized":"2024-03-29T10:14:25.853969+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:14:24.305731+0000","last_clean_scrub_stamp":"2024-03-29T10:14:24.305731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:00.574827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"128'245","reported_seq":728,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-29T10:14:25.340630+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:14:25.340630+0000","last_peered":"2024-03-29T10:14:25.340630+0000","last_clean":"2024-03-29T10:14:25.340630+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:14:25.340630+0000","last_undegraded":"2024-03-29T10:14:25.340630+0000","last_fullsized":"2024-03-29T10:14:25.340630+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":391,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":391,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245,"ondisk_log_size":245,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575080,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37468,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334372,"statfs":{"total":96636764160,"available":96598396928,"internally_reserved":0,"allocated":2306048,"data_stored":1408875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36043212},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.66300000000000003,"15min":0.70899999999999996},"min":{"1min":0.45300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.96099999999999997,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67600000000000005,"15min":0.70799999999999996},"min":{"1min":0.45200000000000001,"5min":0.41699999999999998,"15min":0.377},"max":{"1min":0.94499999999999995,"5min":1.143,"15min":1.143},"last":0.59099999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70599999999999996,"15min":0.72299999999999998},"min":{"1min":0.50800000000000001,"5min":0.45700000000000002,"15min":0.44500000000000001},"max":{"1min":0.91000000000000003,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.52300000000000002,"5min":0.46000000000000002,"15min":0.41199999999999998},"max":{"1min":0.91500000000000004,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71299999999999997,"15min":0.73699999999999999},"min":{"1min":0.53500000000000003,"5min":0.47799999999999998,"15min":0.45200000000000001},"max":{"1min":0.92200000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.54000000000000004,"5min":0.49299999999999999,"15min":0.48799999999999999},"max":{"1min":0.98299999999999998,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.63100000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.71899999999999997,"15min":0.73999999999999999},"min":{"1min":0.55000000000000004,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.94299999999999995,"5min":1.171,"15min":1.171},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73499999999999999,"15min":0.77200000000000002},"min":{"1min":0.56799999999999995,"5min":0.52300000000000002,"15min":0.50700000000000001},"max":{"1min":0.89600000000000002,"5min":1.129,"15min":1.129},"last":0.73499999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.627,"15min":0.60599999999999998},"min":{"1min":0.48799999999999999,"5min":0.39500000000000002,"15min":0.38200000000000001},"max":{"1min":0.85199999999999998,"5min":1.25,"15min":1.25},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66200000000000003,"15min":0.64100000000000001},"min":{"1min":0.45000000000000001,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":0.95699999999999996,"5min":1.117,"15min":1.117},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.67400000000000004,"15min":0.65600000000000003},"min":{"1min":0.435,"5min":0.43099999999999999,"15min":0.40699999999999997},"max":{"1min":0.91400000000000003,"5min":1.47,"15min":1.47},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.53900000000000003,"5min":0.442,"15min":0.39400000000000002},"max":{"1min":0.97699999999999998,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.70999999999999996}]},{"osd":6,"last update":"Fri Mar 29 10:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.751,"15min":0.73099999999999998},"min":{"1min":0.55600000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.92400000000000004,"5min":1.222,"15min":1.222},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.746,"5min":0.73299999999999998,"15min":0.70899999999999996},"min":{"1min":0.52500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.94299999999999995,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.69799999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575081,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1712128,"data_stored":818507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69399999999999995,"15min":0.70699999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.876,"5min":2.7770000000000001,"15min":2.7770000000000001},"last":0.877},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.67700000000000005,"15min":0.71899999999999997},"min":{"1min":0.40999999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83899999999999997,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.76000000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65400000000000003,"15min":0.70099999999999996},"min":{"1min":0.56000000000000005,"5min":0.434,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69899999999999995,"15min":0.71999999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.89500000000000002,"5min":2.6709999999999998,"15min":2.6709999999999998},"last":0.68600000000000005}]},{"osd":2,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.66100000000000003,"15min":0.70599999999999996},"min":{"1min":0.57899999999999996,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.877,"5min":1.256,"15min":1.256},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.71499999999999997,"15min":0.73899999999999999},"min":{"1min":0.56799999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.872,"5min":1.454,"15min":1.454},"last":0.91600000000000004}]},{"osd":3,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68500000000000005,"15min":0.73399999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":1.131,"15min":1.131},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73599999999999999,"15min":0.76400000000000001},"min":{"1min":0.47199999999999998,"5min":0.44500000000000001,"15min":0.39000000000000001},"max":{"1min":0.876,"5min":2.718,"15min":2.718},"last":0.748}]},{"osd":4,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.35199999999999998},"max":{"1min":0.78500000000000003,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.68000000000000005,"15min":0.68400000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.40600000000000003},"max":{"1min":0.75,"5min":2.544,"15min":2.544},"last":0.82499999999999996}]},{"osd":5,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65000000000000002,"15min":0.65100000000000002},"min":{"1min":0.45000000000000001,"5min":0.36499999999999999,"15min":0.34599999999999997},"max":{"1min":0.82599999999999996,"5min":3.2280000000000002,"15min":3.2280000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.67500000000000004,"15min":0.67800000000000005},"min":{"1min":0.46999999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78500000000000003,"5min":2.835,"15min":2.835},"last":0.80800000000000005}]},{"osd":7,"last update":"Fri Mar 29 10:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.72099999999999997,"15min":0.74199999999999999},"min":{"1min":0.54100000000000004,"5min":0.499,"15min":0.45100000000000001},"max":{"1min":0.82099999999999995,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.38900000000000001},"max":{"1min":0.79600000000000004,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.84599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575079,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54920,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316920,"statfs":{"total":96636764160,"available":96580526080,"internally_reserved":0,"allocated":1699840,"data_stored":812232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45400000000000001,"15min":0.45000000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.219},"max":{"1min":0.61499999999999999,"5min":0.81699999999999995,"15min":1.5600000000000001},"last":0.496},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.505,"15min":0.51400000000000001},"min":{"1min":0.35499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70499999999999996,"5min":0.71799999999999997,"15min":1.0860000000000001},"last":0.33700000000000002}]},{"osd":2,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.66500000000000004,"5min":1.222,"15min":1.7869999999999999},"last":0.37},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76900000000000002,"5min":1.04,"15min":1.04},"last":0.35599999999999998}]},{"osd":3,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57099999999999995,"15min":0.59199999999999997},"min":{"1min":0.39000000000000001,"5min":0.38600000000000001,"15min":0.377},"max":{"1min":0.70699999999999996,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53800000000000003,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71999999999999997,"5min":1.026,"15min":1.3049999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.76400000000000001,"15min":0.76300000000000001},"min":{"1min":0.63500000000000001,"5min":0.52900000000000003,"15min":0.497},"max":{"1min":1.1839999999999999,"5min":1.2709999999999999,"15min":1.617},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.78200000000000003,"15min":0.77500000000000002},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.48299999999999998},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.59999999999999998}]},{"osd":5,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.157,"5min":1.157,"15min":1.3939999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.79800000000000004,"15min":0.79900000000000004},"min":{"1min":0.55900000000000005,"5min":0.55800000000000005,"15min":0.52900000000000003},"max":{"1min":1.292,"5min":1.3140000000000001,"15min":1.6579999999999999},"last":0.72199999999999998}]},{"osd":6,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.78700000000000003,"15min":0.79500000000000004},"min":{"1min":0.58499999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.754},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.67500000000000004,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.319,"5min":1.319,"15min":1.468},"last":0.61199999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.82599999999999996,"15min":0.83399999999999996},"min":{"1min":0.65500000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.109,"5min":1.1319999999999999,"15min":1.6000000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.82999999999999996,"15min":0.83099999999999996},"min":{"1min":0.64900000000000002,"5min":0.58399999999999996,"15min":0.57099999999999995},"max":{"1min":1.3560000000000001,"5min":1.4530000000000001,"15min":1.6919999999999999},"last":0.58399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37460,"kb_used_data":2248,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334380,"statfs":{"total":96636764160,"available":96598405120,"internally_reserved":0,"allocated":2301952,"data_stored":1407856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.438,"15min":0.45600000000000002},"min":{"1min":0.34499999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.68200000000000005,"5min":0.72299999999999998,"15min":0.93300000000000005},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.43099999999999999,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.76300000000000001,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53200000000000003,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71299999999999997,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.49299999999999999,"15min":0.498},"min":{"1min":0.379,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73399999999999999,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.441}]},{"osd":3,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54100000000000004,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.84899999999999998,"15min":1.077},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59399999999999997,"15min":0.61199999999999999},"min":{"1min":0.51000000000000001,"5min":0.40400000000000003,"15min":0.39100000000000001},"max":{"1min":0.86399999999999999,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.52500000000000002}]},{"osd":4,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.78000000000000003,"15min":0.80300000000000005},"min":{"1min":0.55200000000000005,"5min":0.499,"15min":0.39200000000000002},"max":{"1min":1.254,"5min":1.254,"15min":1.625},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.80700000000000005,"15min":0.82299999999999995},"min":{"1min":0.60199999999999998,"5min":0.5,"15min":0.45800000000000002},"max":{"1min":1.115,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.84599999999999997}]},{"osd":5,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.74399999999999999,"15min":0.75900000000000001},"min":{"1min":0.63500000000000001,"5min":0.438,"15min":0.39000000000000001},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.872},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.77200000000000002,"15min":0.80000000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.41099999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.5309999999999999},"last":0.79600000000000004}]},{"osd":6,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.81799999999999995,"15min":0.82399999999999995},"min":{"1min":0.65500000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1519999999999999,"5min":1.476,"15min":1.476},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.81799999999999995,"15min":0.84199999999999997},"min":{"1min":0.5,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.601},"last":0.85399999999999998}]},{"osd":7,"last update":"Fri Mar 29 10:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.78200000000000003,"15min":0.79700000000000004},"min":{"1min":0.58799999999999997,"5min":0.48599999999999999,"15min":0.44900000000000001},"max":{"1min":1.238,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.81299999999999994,"15min":0.83899999999999997},"min":{"1min":0.625,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.294,"5min":1.296,"15min":1.5669999999999999},"last":0.91600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575080,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40984,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330856,"statfs":{"total":96636764160,"available":96594796544,"internally_reserved":0,"allocated":1712128,"data_stored":818507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.28899999999999998,"5min":0.251,"15min":0.248},"max":{"1min":0.64500000000000002,"5min":1.236,"15min":1.236},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.499,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":0.76000000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.44800000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48099999999999998,"15min":0.47399999999999998},"min":{"1min":0.313,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78500000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53600000000000003,"15min":0.55200000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":1.032},"last":0.37}]},{"osd":3,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57099999999999995,"15min":0.57799999999999996},"min":{"1min":0.42699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76500000000000001,"5min":0.86499999999999999,"15min":0.99099999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57099999999999995,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.73399999999999999,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.505}]},{"osd":4,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75600000000000001,"15min":0.77400000000000002},"min":{"1min":0.54500000000000004,"5min":0.51100000000000001,"15min":0.42799999999999999},"max":{"1min":0.92500000000000004,"5min":1.125,"15min":1.3520000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.755,"15min":0.78700000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.97399999999999998,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.77100000000000002,"15min":0.79600000000000004},"min":{"1min":0.624,"5min":0.52400000000000002,"15min":0.46999999999999997},"max":{"1min":1.093,"5min":1.1699999999999999,"15min":1.2589999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.78500000000000003,"15min":0.79700000000000004},"min":{"1min":0.57299999999999995,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.05,"5min":1.1919999999999999,"15min":1.2250000000000001},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.79700000000000004,"15min":0.81100000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":0.96899999999999997,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.79800000000000004,"15min":0.82699999999999996},"min":{"1min":0.58799999999999997,"5min":0.56200000000000006,"15min":0.55000000000000004},"max":{"1min":1.0229999999999999,"5min":1.1579999999999999,"15min":1.3560000000000001},"last":0.64800000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.82199999999999995,"15min":0.85399999999999998},"min":{"1min":0.57799999999999996,"5min":0.56899999999999995,"15min":0.53400000000000003},"max":{"1min":1.042,"5min":1.258,"15min":1.4490000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.85099999999999998,"15min":0.86899999999999999},"min":{"1min":0.70999999999999996,"5min":0.60899999999999999,"15min":0.56799999999999995},"max":{"1min":1.0640000000000001,"5min":1.226,"15min":1.4199999999999999},"last":0.68000000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1712128,"data_stored":818507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.432,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.26800000000000002,"15min":0.26500000000000001},"max":{"1min":0.67800000000000005,"5min":0.75600000000000001,"15min":0.81299999999999994},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.28399999999999997},"max":{"1min":0.68000000000000005,"5min":0.874,"15min":0.874},"last":0.75800000000000001}]},{"osd":1,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.44400000000000001,"15min":0.47799999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.98699999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.437,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80400000000000005,"5min":0.82699999999999996,"15min":0.96099999999999997},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.54800000000000004,"15min":0.55800000000000005},"min":{"1min":0.42999999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.98299999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59499999999999997,"15min":0.625},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79400000000000004,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.60999999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72299999999999998,"15min":0.75600000000000001},"min":{"1min":0.504,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.93999999999999995,"5min":1.034,"15min":1.288},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.57899999999999996,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95599999999999996,"5min":1.093,"15min":1.46},"last":0.94199999999999995}]},{"osd":5,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74299999999999999,"15min":0.77900000000000003},"min":{"1min":0.53500000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93600000000000005,"5min":1.1040000000000001,"15min":1.3580000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.76300000000000001,"15min":0.79200000000000004},"min":{"1min":0.60299999999999998,"5min":0.50700000000000001,"15min":0.46899999999999997},"max":{"1min":0.99099999999999999,"5min":1.1479999999999999,"15min":1.429},"last":0.63500000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.77800000000000002,"15min":0.80500000000000005},"min":{"1min":0.60399999999999998,"5min":0.54500000000000004,"15min":0.54200000000000004},"max":{"1min":1.0660000000000001,"5min":1.123,"15min":1.4730000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.749,"5min":0.76000000000000001,"15min":0.80300000000000005},"min":{"1min":0.56200000000000006,"5min":0.501,"15min":0.501},"max":{"1min":0.97299999999999998,"5min":1.0600000000000001,"15min":1.3999999999999999},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Mar 29 10:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.78200000000000003,"15min":0.82099999999999995},"min":{"1min":0.64300000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.052,"5min":1.103,"15min":1.5029999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.81100000000000005,"15min":0.82999999999999996},"min":{"1min":0.69599999999999995,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.105,"5min":1.1859999999999999,"15min":1.345},"last":0.66100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575081,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59016,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312824,"statfs":{"total":96636764160,"available":96576331776,"internally_reserved":0,"allocated":1699840,"data_stored":812232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.69099999999999995},"min":{"1min":0.39400000000000002,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.88100000000000001,"5min":0.98799999999999999,"15min":1.417},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68100000000000005,"15min":0.69899999999999995},"min":{"1min":0.47499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.90100000000000002,"5min":1.0269999999999999,"15min":1.165},"last":0.67300000000000004}]},{"osd":1,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.70199999999999996},"min":{"1min":0.46500000000000002,"5min":0.42599999999999999,"15min":0.39000000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.98399999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69699999999999995,"15min":0.70999999999999996},"min":{"1min":0.51000000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.88600000000000001,"5min":1.075,"15min":1.333},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.76400000000000001,"15min":0.77500000000000002},"min":{"1min":0.64000000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.92300000000000004,"5min":1.014,"15min":1.03},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77800000000000002,"15min":0.77400000000000002},"min":{"1min":0.59499999999999997,"5min":0.50600000000000001,"15min":0.46200000000000002},"max":{"1min":0.95299999999999996,"5min":1.0600000000000001,"15min":1.387},"last":0.59299999999999997}]},{"osd":3,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.71499999999999997,"15min":0.72599999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95999999999999996,"5min":0.97499999999999998,"15min":1.2350000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71699999999999997,"15min":0.73399999999999999},"min":{"1min":0.52600000000000002,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.3180000000000001},"last":0.58199999999999996}]},{"osd":5,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61099999999999999,"15min":0.58899999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.38100000000000001},"max":{"1min":0.76700000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.628,"15min":0.60399999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.309},"max":{"1min":0.77400000000000002,"5min":0.98599999999999999,"15min":1.294},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.64200000000000002,"15min":0.621},"min":{"1min":0.47899999999999998,"5min":0.44,"15min":0.375},"max":{"1min":0.82299999999999995,"5min":1.026,"15min":1.026},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.67300000000000004,"15min":0.66100000000000003},"min":{"1min":0.46500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86199999999999999,"5min":1.093,"15min":1.1359999999999999},"last":0.51100000000000001}]},{"osd":7,"last update":"Fri Mar 29 10:13:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.68300000000000005,"15min":0.68200000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":1.04,"5min":1.0580000000000001,"15min":1.345},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77500000000000002,"15min":0.76100000000000001},"min":{"1min":0.54800000000000004,"5min":0.44,"15min":0.44},"max":{"1min":1.014,"5min":1.014,"15min":1.214},"last":0.62}]}]},{"osd":5,"up_from":13,"seq":55834575080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1712128,"data_stored":818507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.63600000000000001,"15min":0.67600000000000005},"min":{"1min":0.46000000000000002,"5min":0.36799999999999999,"15min":0.35699999999999998},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":1.1899999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.69999999999999996,"15min":0.72699999999999998},"min":{"1min":0.45700000000000002,"5min":0.38100000000000001,"15min":0.33500000000000002},"max":{"1min":0.96999999999999997,"5min":0.997,"15min":1.339},"last":0.68600000000000005}]},{"osd":1,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68799999999999994,"15min":0.71299999999999997},"min":{"1min":0.51200000000000001,"5min":0.42699999999999999,"15min":0.377},"max":{"1min":0.92300000000000004,"5min":0.96599999999999997,"15min":0.96699999999999997},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66200000000000003,"15min":0.70099999999999996},"min":{"1min":0.40200000000000002,"5min":0.39900000000000002,"15min":0.39200000000000002},"max":{"1min":0.93999999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.74399999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.70999999999999996,"15min":0.72899999999999998},"min":{"1min":0.53200000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98899999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.68200000000000005,"15min":0.70599999999999996},"min":{"1min":0.503,"5min":0.46200000000000002,"15min":0.39700000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":1.095},"last":0.83599999999999997}]},{"osd":3,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.69799999999999995,"15min":0.72599999999999998},"min":{"1min":0.55800000000000005,"5min":0.497,"15min":0.45600000000000002},"max":{"1min":0.97999999999999998,"5min":1.042,"15min":1.1419999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71799999999999997,"15min":0.746},"min":{"1min":0.53600000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.88200000000000001,"5min":0.98799999999999999,"15min":1.0980000000000001},"last":1.0029999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59999999999999998,"15min":0.60099999999999998},"min":{"1min":0.41499999999999998,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":0.87,"5min":0.91700000000000004,"15min":0.94599999999999995},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64800000000000002,"15min":0.64200000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.38900000000000001},"max":{"1min":0.877,"5min":1.097,"15min":1.097},"last":1.04}]},{"osd":6,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64700000000000002,"15min":0.65500000000000003},"min":{"1min":0.443,"5min":0.41299999999999998,"15min":0.36799999999999999},"max":{"1min":0.82499999999999996,"5min":0.93899999999999995,"15min":1.151},"last":1.083},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.66200000000000003,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.80800000000000005,"5min":0.95899999999999996,"15min":0.996},"last":0.95299999999999996}]},{"osd":7,"last update":"Fri Mar 29 10:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69499999999999995,"15min":0.70399999999999996},"min":{"1min":0.46899999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.96299999999999997,"5min":0.998,"15min":1.0289999999999999},"last":1.073},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71699999999999997,"15min":0.72999999999999998},"min":{"1min":0.52100000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.1990000000000001},"last":1.0549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:14:31.062 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T10:14:31.062 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T10:14:31.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.303+0000 7f6d0ee00700 1 Processor -- start 2024-03-29T10:14:31.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.304+0000 7f6d0ee00700 1 -- start start 2024-03-29T10:14:31.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d0ee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d081611b0 0x7f6d0815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d0ee00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6d0815f4d0 0x7f6d080ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d0ee00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6d080d21f0 0x7f6d080cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d0ee00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d080cf180 con 0x7f6d080d21f0 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d0ee00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d080ce270 con 0x7f6d0815f4d0 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d0ee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0815b600 con 0x7f6d081611b0 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d0ce00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6d080d21f0 0x7f6d080cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d0ce00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6d080d21f0 0x7f6d080cec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59034/0 (socket says 172.21.15.12:59034) 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d0ce00700 1 -- 172.21.15.12:0/917835884 learned_addr learned my addr 172.21.15.12:0/917835884 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d07e00700 1 --2- 172.21.15.12:0/917835884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d081611b0 0x7f6d0815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.305+0000 7f6d07400700 1 --2- 172.21.15.12:0/917835884 >> v2:172.21.15.159:3300/0 conn(0x7f6d0815f4d0 0x7f6d080ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.306+0000 7f6d0ce00700 1 -- 172.21.15.12:0/917835884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d081611b0 msgr2=0x7f6d0815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.306+0000 7f6d0ce00700 1 --2- 172.21.15.12:0/917835884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d081611b0 0x7f6d0815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.306+0000 7f6d0ce00700 1 -- 172.21.15.12:0/917835884 >> v2:172.21.15.159:3300/0 conn(0x7f6d0815f4d0 msgr2=0x7f6d080ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.306+0000 7f6d0ce00700 1 --2- 172.21.15.12:0/917835884 >> v2:172.21.15.159:3300/0 conn(0x7f6d0815f4d0 0x7f6d080ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.306+0000 7f6d0ce00700 1 -- 172.21.15.12:0/917835884 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d080d2760 con 0x7f6d080d21f0 2024-03-29T10:14:31.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.306+0000 7f6d0ce00700 1 --2- 172.21.15.12:0/917835884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d080d21f0 0x7f6d080cec40 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f6d0808d540 tx=0x7f6d0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20352fc8870ec75 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:31.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.306+0000 7f6d06a00700 1 -- 172.21.15.12:0/917835884 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0000e070 con 0x7f6d080d21f0 2024-03-29T10:14:31.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.306+0000 7f6d06a00700 1 -- 172.21.15.12:0/917835884 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0000b910 con 0x7f6d080d21f0 2024-03-29T10:14:31.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.306+0000 7f6d06a00700 1 -- 172.21.15.12:0/917835884 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0000bcc0 con 0x7f6d080d21f0 2024-03-29T10:14:31.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.307+0000 7f6d0ee00700 1 -- 172.21.15.12:0/917835884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d080d21f0 msgr2=0x7f6d080cec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.307+0000 7f6d0ee00700 1 --2- 172.21.15.12:0/917835884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d080d21f0 0x7f6d080cec40 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f6d0808d540 tx=0x7f6d0000b330 comp rx=0 tx=0).stop 2024-03-29T10:14:31.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.307+0000 7f6d0ee00700 1 -- 172.21.15.12:0/917835884 shutdown_connections 2024-03-29T10:14:31.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.307+0000 7f6d0ee00700 1 --2- 172.21.15.12:0/917835884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d080d21f0 0x7f6d080cec40 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.307+0000 7f6d0ee00700 1 --2- 172.21.15.12:0/917835884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d081611b0 0x7f6d0815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.307+0000 7f6d0ee00700 1 --2- 172.21.15.12:0/917835884 >> v2:172.21.15.159:3300/0 conn(0x7f6d0815f4d0 0x7f6d080ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.307+0000 7f6d0ee00700 1 -- 172.21.15.12:0/917835884 >> 172.21.15.12:0/917835884 conn(0x7f6d0808aff0 msgr2=0x7f6d080d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:31.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.307+0000 7f6d0ee00700 1 -- 172.21.15.12:0/917835884 shutdown_connections 2024-03-29T10:14:31.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.307+0000 7f6d0ee00700 1 -- 172.21.15.12:0/917835884 wait complete. 2024-03-29T10:14:31.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.309+0000 7f6d0ee00700 1 Processor -- start 2024-03-29T10:14:31.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.309+0000 7f6d0ee00700 1 -- start start 2024-03-29T10:14:31.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d0ee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d080d21f0 0x7f6d081f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d0ee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0815f4d0 0x7f6d080cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d0ee00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d081611b0 0x7f6d080b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d0ee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080cd970 con 0x7f6d0815f4d0 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d0ee00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081f77b0 con 0x7f6d081611b0 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d0ee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081f6c30 con 0x7f6d080d21f0 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d07400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0815f4d0 0x7f6d080cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d07400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0815f4d0 0x7f6d080cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59044/0 (socket says 172.21.15.12:59044) 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d07400700 1 -- 172.21.15.12:0/3355971249 learned_addr learned my addr 172.21.15.12:0/3355971249 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d07e00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d080d21f0 0x7f6d081f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d0ce00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d081611b0 0x7f6d080b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d07400700 1 -- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d080d21f0 msgr2=0x7f6d081f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d07400700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d080d21f0 0x7f6d081f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d07400700 1 -- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d081611b0 msgr2=0x7f6d080b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d07400700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d081611b0 0x7f6d080b78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d07400700 1 -- 172.21.15.12:0/3355971249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d00009b30 con 0x7f6d0815f4d0 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.311+0000 7f6d0ce00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d081611b0 0x7f6d080b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:14:31.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.312+0000 7f6d07400700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0815f4d0 0x7f6d080cba90 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400e900 tx=0x7f6cf400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.312+0000 7f6cffe00700 1 -- 172.21.15.12:0/3355971249 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4009690 con 0x7f6d0815f4d0 2024-03-29T10:14:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.312+0000 7f6cffe00700 1 -- 172.21.15.12:0/3355971249 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf400c850 con 0x7f6d0815f4d0 2024-03-29T10:14:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.312+0000 7f6cffe00700 1 -- 172.21.15.12:0/3355971249 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400cc00 con 0x7f6d0815f4d0 2024-03-29T10:14:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.312+0000 7f6d0ee00700 1 -- 172.21.15.12:0/3355971249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d081f6f10 con 0x7f6d0815f4d0 2024-03-29T10:14:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.312+0000 7f6d0ee00700 1 -- 172.21.15.12:0/3355971249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d081f74d0 con 0x7f6d0815f4d0 2024-03-29T10:14:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.313+0000 7f6d0ee00700 1 -- 172.21.15.12:0/3355971249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d08067720 con 0x7f6d0815f4d0 2024-03-29T10:14:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.314+0000 7f6cffe00700 1 -- 172.21.15.12:0/3355971249 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6cf4013470 con 0x7f6d0815f4d0 2024-03-29T10:14:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.315+0000 7f6cffe00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ce806adb0 0x7f6ce806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.315+0000 7f6cffe00700 1 -- 172.21.15.12:0/3355971249 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6cf4089510 con 0x7f6d0815f4d0 2024-03-29T10:14:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.315+0000 7f6d07e00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ce806adb0 0x7f6ce806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.316+0000 7f6d07e00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ce806adb0 0x7f6ce806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6d081f0e50 tx=0x7f6cf8008040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:31.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.319+0000 7f6cffe00700 1 -- 172.21.15.12:0/3355971249 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6cf40545d0 con 0x7f6d0815f4d0 2024-03-29T10:14:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.617+0000 7f6d0ee00700 1 -- 172.21.15.12:0/3355971249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6d080b5d00 con 0x7f6d0815f4d0 2024-03-29T10:14:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.623+0000 7f6cffe00700 1 -- 172.21.15.12:0/3355971249 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6cf4057bf0 con 0x7f6d0815f4d0 2024-03-29T10:14:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 -- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ce806adb0 msgr2=0x7f6ce806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ce806adb0 0x7f6ce806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6d081f0e50 tx=0x7f6cf8008040 comp rx=0 tx=0).stop 2024-03-29T10:14:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 -- 172.21.15.12:0/3355971249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0815f4d0 msgr2=0x7f6d080cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0815f4d0 0x7f6d080cba90 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400e900 tx=0x7f6cf400ecc0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 -- 172.21.15.12:0/3355971249 shutdown_connections 2024-03-29T10:14:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ce806adb0 0x7f6ce806d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d081611b0 0x7f6d080b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d080d21f0 0x7f6d081f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 --2- 172.21.15.12:0/3355971249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d0815f4d0 0x7f6d080cba90 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 -- 172.21.15.12:0/3355971249 >> 172.21.15.12:0/3355971249 conn(0x7f6d0808aff0 msgr2=0x7f6d0806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.625+0000 7f6d0ee00700 1 -- 172.21.15.12:0/3355971249 shutdown_connections 2024-03-29T10:14:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.626+0000 7f6d0ee00700 1 -- 172.21.15.12:0/3355971249 wait complete. 2024-03-29T10:14:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T10:14:31.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.845+0000 7f83a2c00700 1 Processor -- start 2024-03-29T10:14:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.845+0000 7f83a2c00700 1 -- start start 2024-03-29T10:14:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.847+0000 7f83a2c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f839c0cebd0 0x7f839c0cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.847+0000 7f83a2c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c06f850 con 0x7f839c06a790 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.847+0000 7f83a2c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c06e3f0 con 0x7f839c0afbe0 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.847+0000 7f83a2c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c06fad0 con 0x7f839c0cebd0 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.847+0000 7f839be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f839c06a790 0x7f839c06ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40128/0 (socket says 172.21.15.12:40128) 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.847+0000 7f839be00700 1 -- 172.21.15.12:0/3903459980 learned_addr learned my addr 172.21.15.12:0/3903459980 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.847+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448144389 0 0) 0x7f839c06f850 con 0x7f839c06a790 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.847+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83800035b0 con 0x7f839c06a790 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f83a0c00700 1 --2- 172.21.15.12:0/3903459980 >> v2:172.21.15.159:3300/0 conn(0x7f839c0cebd0 0x7f839c0cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944626864 0 0) 0x7f83800035b0 con 0x7f839c06a790 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 >> v2:172.21.15.159:3300/0 conn(0x7f839c0cebd0 msgr2=0x7f839c0cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 --2- 172.21.15.12:0/3903459980 >> v2:172.21.15.159:3300/0 conn(0x7f839c0cebd0 0x7f839c0cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 >> v1:172.21.15.159:6789/0 conn(0x7f839c0afbe0 legacy=0x7f839c06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c0af970 con 0x7f839c06a790 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f838c001670 con 0x7f839c06a790 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f83a0c00700 1 --2- 172.21.15.12:0/3903459980 >> v2:172.21.15.159:3300/0 conn(0x7f839c0cebd0 0x7f839c0cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 >> v1:172.21.15.12:6789/0 conn(0x7f839c06a790 legacy=0x7f839c06ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8380003a00 0x7f8380003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8380004850 0x7f8380004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380005410 0x7f838000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c06fad0 con 0x7f8380004850 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838000c2a0 con 0x7f8380003a00 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838000c460 con 0x7f8380005410 2024-03-29T10:14:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839b400700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8380004850 0x7f8380004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f839be00700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8380003a00 0x7f8380003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.848+0000 7f83a0c00700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380005410 0x7f838000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f839b400700 1 -- 172.21.15.12:0/3903459980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380005410 msgr2=0x7f838000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f839b400700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380005410 0x7f838000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f839b400700 1 -- 172.21.15.12:0/3903459980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8380003a00 msgr2=0x7f8380003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f839b400700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8380003a00 0x7f8380003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f83a0c00700 1 -- 172.21.15.12:0/3903459980 reap_dead start 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f839b400700 1 -- 172.21.15.12:0/3903459980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83800035b0 con 0x7f8380004850 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f839be00700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8380003a00 0x7f8380003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f839b400700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8380004850 0x7f8380004cd0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f8388001800 tx=0x7f8380005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133dc7cf2d1b0439 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83880012f0 con 0x7f8380004850 2024-03-29T10:14:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8388005310 con 0x7f8380004850 2024-03-29T10:14:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.849+0000 7f839aa00700 1 -- 172.21.15.12:0/3903459980 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388006860 con 0x7f8380004850 2024-03-29T10:14:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.850+0000 7f83a2c00700 1 -- 172.21.15.12:0/3903459980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8380004850 msgr2=0x7f8380004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.850+0000 7f83a2c00700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8380004850 0x7f8380004cd0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f8388001800 tx=0x7f8380005880 comp rx=0 tx=0).stop 2024-03-29T10:14:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.850+0000 7f83a2c00700 1 -- 172.21.15.12:0/3903459980 shutdown_connections 2024-03-29T10:14:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.850+0000 7f83a2c00700 1 --2- 172.21.15.12:0/3903459980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8380004850 0x7f8380004cd0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.850+0000 7f83a2c00700 1 -- 172.21.15.12:0/3903459980 >> 172.21.15.12:0/3903459980 conn(0x7f839c0b5ab0 msgr2=0x7f839c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.850+0000 7f83a2c00700 1 -- 172.21.15.12:0/3903459980 shutdown_connections 2024-03-29T10:14:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.850+0000 7f83a2c00700 1 -- 172.21.15.12:0/3903459980 wait complete. 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.852+0000 7f83a2c00700 1 Processor -- start 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.852+0000 7f83a2c00700 1 -- start start 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.853+0000 7f83a2c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8380004850 0x7f839c09aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.853+0000 7f83a2c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f839c09a360 0x7f839c15dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.853+0000 7f83a2c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f839c15e160 0x7f839c09da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.853+0000 7f83a2c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c1f2d40 con 0x7f839c09a360 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.853+0000 7f83a2c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c15e5f0 con 0x7f8380004850 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.853+0000 7f83a2c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c155590 con 0x7f839c15e160 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.853+0000 7f839b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f839c09a360 0x7f839c15dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.853+0000 7f839b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f839c09a360 0x7f839c15dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59066/0 (socket says 172.21.15.12:59066) 2024-03-29T10:14:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.853+0000 7f839b400700 1 -- 172.21.15.12:0/1184679571 learned_addr learned my addr 172.21.15.12:0/1184679571 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.853+0000 7f839be00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8380004850 0x7f839c09aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f83a0c00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f839c15e160 0x7f839c09da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f839b400700 1 -- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f839c15e160 msgr2=0x7f839c09da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f839b400700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f839c15e160 0x7f839c09da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f839b400700 1 -- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8380004850 msgr2=0x7f839c09aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f839b400700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8380004850 0x7f839c09aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f839b400700 1 -- 172.21.15.12:0/1184679571 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c1f2f20 con 0x7f839c09a360 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f839b400700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f839c09a360 0x7f839c15dc20 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f8388000fb0 tx=0x7f8388001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f8393e00700 1 -- 172.21.15.12:0/1184679571 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388001290 con 0x7f839c09a360 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f8393e00700 1 -- 172.21.15.12:0/1184679571 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83880050d0 con 0x7f839c09a360 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f83a2c00700 1 -- 172.21.15.12:0/1184679571 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839c1f3200 con 0x7f839c09a360 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f8393e00700 1 -- 172.21.15.12:0/1184679571 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388007280 con 0x7f839c09a360 2024-03-29T10:14:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.854+0000 7f83a2c00700 1 -- 172.21.15.12:0/1184679571 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c1f37c0 con 0x7f839c09a360 2024-03-29T10:14:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.855+0000 7f83a2c00700 1 -- 172.21.15.12:0/1184679571 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c067720 con 0x7f839c09a360 2024-03-29T10:14:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.856+0000 7f8393e00700 1 -- 172.21.15.12:0/1184679571 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f83880073e0 con 0x7f839c09a360 2024-03-29T10:14:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.858+0000 7f8393e00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837c06adb0 0x7f837c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.858+0000 7f8393e00700 1 -- 172.21.15.12:0/1184679571 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f838808a190 con 0x7f839c09a360 2024-03-29T10:14:31.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.858+0000 7f839be00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837c06adb0 0x7f837c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:31.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.859+0000 7f839be00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837c06adb0 0x7f837c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f838c001200 tx=0x7f838c001c30 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:31.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:31.862+0000 7f8393e00700 1 -- 172.21.15.12:0/1184679571 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8388055250 con 0x7f839c09a360 2024-03-29T10:14:32.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.121+0000 7f83a2c00700 1 -- 172.21.15.12:0/1184679571 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f839c15ecb0 con 0x7f839c09a360 2024-03-29T10:14:32.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.122+0000 7f8393e00700 1 -- 172.21.15.12:0/1184679571 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f8388058870 con 0x7f839c09a360 2024-03-29T10:14:32.120 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:32.121 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1317631939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1317631939},{"type":"v1","addr":"172.21.15.159:6836","nonce":1317631939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1057731647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1057731647},{"type":"v1","addr":"172.21.15.159:6837","nonce":1057731647}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1567088347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1567088347},{"type":"v1","addr":"172.21.15.12:6834","nonce":1567088347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:14:25.846391+0000","modified":"2024-03-29T10:14:29.114278+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8406":{"gid":8406,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3311625077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3311625077},{"type":"v1","addr":"172.21.15.12:6835","nonce":3311625077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-29T10:14:32.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.124+0000 7f83a2c00700 1 -- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837c06adb0 msgr2=0x7f837c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.124+0000 7f83a2c00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837c06adb0 0x7f837c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f838c001200 tx=0x7f838c001c30 comp rx=0 tx=0).stop 2024-03-29T10:14:32.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.124+0000 7f83a2c00700 1 -- 172.21.15.12:0/1184679571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f839c09a360 msgr2=0x7f839c15dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.124+0000 7f83a2c00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f839c09a360 0x7f839c15dc20 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f8388000fb0 tx=0x7f8388001120 comp rx=0 tx=0).stop 2024-03-29T10:14:32.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.125+0000 7f83a2c00700 1 -- 172.21.15.12:0/1184679571 shutdown_connections 2024-03-29T10:14:32.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.125+0000 7f83a2c00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f837c06adb0 0x7f837c06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.125+0000 7f83a2c00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f839c15e160 0x7f839c09da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.125+0000 7f83a2c00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8380004850 0x7f839c09aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.125+0000 7f83a2c00700 1 --2- 172.21.15.12:0/1184679571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f839c09a360 0x7f839c15dc20 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.125+0000 7f83a2c00700 1 -- 172.21.15.12:0/1184679571 >> 172.21.15.12:0/1184679571 conn(0x7f839c0b5ab0 msgr2=0x7f839c0b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:32.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.125+0000 7f83a2c00700 1 -- 172.21.15.12:0/1184679571 shutdown_connections 2024-03-29T10:14:32.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.125+0000 7f83a2c00700 1 -- 172.21.15.12:0/1184679571 wait complete. 2024-03-29T10:14:32.124 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 64 2024-03-29T10:14:32.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T10:14:25.846391+0000', 'modified': '2024-03-29T10:14:29.114278+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8406}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8406': {'gid': 8406, 'name': 'a', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6835/3311625077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6833', 'nonce': 3311625077}, {'type': 'v1', 'addr': '172.21.15.12:6835', 'nonce': 3311625077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T10:14:32.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T10:14:32.545 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-29T10:14:32.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.746+0000 7ff4f3800700 1 Processor -- start 2024-03-29T10:14:32.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.747+0000 7ff4f3800700 1 -- start start 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.748+0000 7ff4f3800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4ec15a5c0 0x7ff4ec0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.748+0000 7ff4f3800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 0x7ff4ec0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.748+0000 7ff4f3800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ec0cb310 con 0x7ff4ec15dfe0 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.748+0000 7ff4f3800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ec070f00 con 0x7ff4ec15a5c0 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.748+0000 7ff4f3800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec0711a0 con 0x7ff4ec0ce840 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.748+0000 7ff4f0e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4ec15a5c0 0x7ff4ec0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.748+0000 7ff4f0e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4ec15a5c0 0x7ff4ec0709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59078/0 (socket says 172.21.15.12:59078) 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.748+0000 7ff4f0e00700 1 -- 172.21.15.12:0/3247190810 learned_addr learned my addr 172.21.15.12:0/3247190810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.748+0000 7ff4f1800700 1 --2- 172.21.15.12:0/3247190810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 0x7ff4ec0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.749+0000 7ff4f0e00700 1 -- 172.21.15.12:0/3247190810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 msgr2=0x7ff4ec0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.749+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/3247190810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 0x7ff4ec0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.749+0000 7ff4f0e00700 1 -- 172.21.15.12:0/3247190810 >> v1:172.21.15.159:6789/0 conn(0x7ff4ec15dfe0 legacy=0x7ff4ec0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.749+0000 7ff4f0e00700 1 -- 172.21.15.12:0/3247190810 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec071480 con 0x7ff4ec15a5c0 2024-03-29T10:14:32.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.749+0000 7ff4f1800700 1 --2- 172.21.15.12:0/3247190810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 0x7ff4ec0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:32.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.749+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/3247190810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15a5c0 0x7ff4ec0709c0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7ff4e00098f0 tx=0x7ff4e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c150e61e47273574 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:32.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.749+0000 7ff4eb400700 1 -- 172.21.15.12:0/3247190810 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e000b8a0 con 0x7ff4ec15a5c0 2024-03-29T10:14:32.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.749+0000 7ff4eb400700 1 -- 172.21.15.12:0/3247190810 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e000ba00 con 0x7ff4ec15a5c0 2024-03-29T10:14:32.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.749+0000 7ff4eb400700 1 -- 172.21.15.12:0/3247190810 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e000bd90 con 0x7ff4ec15a5c0 2024-03-29T10:14:32.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.750+0000 7ff4f3800700 1 -- 172.21.15.12:0/3247190810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15a5c0 msgr2=0x7ff4ec0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.750+0000 7ff4f3800700 1 --2- 172.21.15.12:0/3247190810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15a5c0 0x7ff4ec0709c0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7ff4e00098f0 tx=0x7ff4e0009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.750+0000 7ff4f3800700 1 -- 172.21.15.12:0/3247190810 shutdown_connections 2024-03-29T10:14:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.750+0000 7ff4f3800700 1 --2- 172.21.15.12:0/3247190810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 0x7ff4ec0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.750+0000 7ff4f3800700 1 --2- 172.21.15.12:0/3247190810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15a5c0 0x7ff4ec0709c0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.750+0000 7ff4f3800700 1 -- 172.21.15.12:0/3247190810 >> 172.21.15.12:0/3247190810 conn(0x7ff4ec0b0d70 msgr2=0x7ff4ec0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.750+0000 7ff4f3800700 1 -- 172.21.15.12:0/3247190810 shutdown_connections 2024-03-29T10:14:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.750+0000 7ff4f3800700 1 -- 172.21.15.12:0/3247190810 wait complete. 2024-03-29T10:14:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.752+0000 7ff4f3800700 1 Processor -- start 2024-03-29T10:14:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.752+0000 7ff4f3800700 1 -- start start 2024-03-29T10:14:32.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f3800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 0x7ff4ec14de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f3800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4ec15a5c0 0x7ff4ec14e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f3800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15dfe0 0x7ff4ec15cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f3800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec158b50 con 0x7ff4ec15dfe0 2024-03-29T10:14:32.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f3800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec09d0d0 con 0x7ff4ec15a5c0 2024-03-29T10:14:32.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f3800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec1581b0 con 0x7ff4ec0ce840 2024-03-29T10:14:32.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15dfe0 0x7ff4ec15cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15dfe0 0x7ff4ec15cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59094/0 (socket says 172.21.15.12:59094) 2024-03-29T10:14:32.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f1800700 1 -- 172.21.15.12:0/4047407735 learned_addr learned my addr 172.21.15.12:0/4047407735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:32.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 0x7ff4ec14de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4ec15a5c0 0x7ff4ec14e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f1800700 1 -- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 msgr2=0x7ff4ec14de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f1800700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 0x7ff4ec14de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f1800700 1 -- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4ec15a5c0 msgr2=0x7ff4ec14e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f1800700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4ec15a5c0 0x7ff4ec14e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4f1800700 1 -- 172.21.15.12:0/4047407735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e0009550 con 0x7ff4ec15dfe0 2024-03-29T10:14:32.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.753+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4ec15a5c0 0x7ff4ec14e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:32.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.754+0000 7ff4f1800700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15dfe0 0x7ff4ec15cc00 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4007ec0 tx=0x7ff4e400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:32.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.754+0000 7ff4e8c00700 1 -- 172.21.15.12:0/4047407735 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e400de30 con 0x7ff4ec15dfe0 2024-03-29T10:14:32.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.754+0000 7ff4e8c00700 1 -- 172.21.15.12:0/4047407735 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e400f040 con 0x7ff4ec15dfe0 2024-03-29T10:14:32.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.754+0000 7ff4f3800700 1 -- 172.21.15.12:0/4047407735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ec158490 con 0x7ff4ec15dfe0 2024-03-29T10:14:32.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.754+0000 7ff4e8c00700 1 -- 172.21.15.12:0/4047407735 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4003c60 con 0x7ff4ec15dfe0 2024-03-29T10:14:32.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.754+0000 7ff4f3800700 1 -- 172.21.15.12:0/4047407735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec14e3c0 con 0x7ff4ec15dfe0 2024-03-29T10:14:32.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.757+0000 7ff4e8c00700 1 -- 172.21.15.12:0/4047407735 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff4e4004030 con 0x7ff4ec15dfe0 2024-03-29T10:14:32.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.758+0000 7ff4e8c00700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff4cc06adb0 0x7ff4cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.758+0000 7ff4e8c00700 1 -- 172.21.15.12:0/4047407735 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4e4089680 con 0x7ff4ec15dfe0 2024-03-29T10:14:32.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.758+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff4cc06adb0 0x7ff4cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.759+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff4cc06adb0 0x7ff4cc06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009c40 tx=0x7ff4e0008030 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:32.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.760+0000 7ff5073d5b80 1 Processor -- start 2024-03-29T10:14:32.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.760+0000 7ff5073d5b80 1 -- start start 2024-03-29T10:14:32.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc33f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc33efa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc333d650 con 0x55acc33c3b60 2024-03-29T10:14:32.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc331ec30 con 0x55acc33f2830 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acc331c430 con 0x55acc33c5be0 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff4f1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc33efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff4f0e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc33f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff4f1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc33efa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59102/0 (socket says 172.21.15.12:59102) 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff4f1800700 1 -- 172.21.15.12:0/2340713416 learned_addr learned my addr 172.21.15.12:0/2340713416 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff4f1800700 1 -- 172.21.15.12:0/2340713416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 msgr2=0x55acc33c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff4f1800700 1 --2- 172.21.15.12:0/2340713416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff4f1800700 1 -- 172.21.15.12:0/2340713416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 msgr2=0x55acc33f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.761+0000 7ff4f1800700 1 --2- 172.21.15.12:0/2340713416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc33f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.762+0000 7ff4f1800700 1 -- 172.21.15.12:0/2340713416 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acc310b790 con 0x55acc33c3b60 2024-03-29T10:14:32.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.762+0000 7ff4f1800700 1 --2- 172.21.15.12:0/2340713416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc33efa20 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40601b0 tx=0x7ff4e4057d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f736e64d0dbbc8f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:32.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.762+0000 7ff4dd600700 1 -- 172.21.15.12:0/2340713416 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e406d030 con 0x55acc33c3b60 2024-03-29T10:14:32.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.762+0000 7ff4dd600700 1 -- 172.21.15.12:0/2340713416 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e405c120 con 0x55acc33c3b60 2024-03-29T10:14:32.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.762+0000 7ff4dd600700 1 -- 172.21.15.12:0/2340713416 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e40606e0 con 0x55acc33c3b60 2024-03-29T10:14:32.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.763+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2340713416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 msgr2=0x55acc33efa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.763+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/2340713416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc33efa20 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40601b0 tx=0x7ff4e4057d80 comp rx=0 tx=0).stop 2024-03-29T10:14:32.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.763+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2340713416 shutdown_connections 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.763+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/2340713416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc33efa20 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.763+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/2340713416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc33f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.763+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/2340713416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.763+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2340713416 >> 172.21.15.12:0/2340713416 conn(0x55acc33266a0 msgr2=0x55acc340d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.763+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2340713416 shutdown_connections 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.763+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2340713416 wait complete. 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.763+0000 7ff5073d5b80 1 -- start start 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.764+0000 7ff5073d5b80 1 Processor -- start 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc34114a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc33f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc3317d20 con 0x55acc33c3b60 2024-03-29T10:14:32.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc32b3770 con 0x55acc33f2830 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acc32d66f0 con 0x55acc33c5be0 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff5073d5b80 10 client.0 _ll_register_callbacks cb 0x55acc32bd5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff4f0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc34114a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff4f0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc34114a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59108/0 (socket says 172.21.15.12:59108) 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff4f0e00700 1 -- 172.21.15.12:0/2758318921 learned_addr learned my addr 172.21.15.12:0/2758318921 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff4f1800700 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc33f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff4f0e00700 1 -- 172.21.15.12:0/2758318921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 msgr2=0x55acc33f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff4f0e00700 1 -- 172.21.15.12:0/2758318921 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 msgr2=0x55acc33f3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc33f3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.765+0000 7ff4f0e00700 1 -- 172.21.15.12:0/2758318921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec14e3c0 con 0x55acc33c3b60 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.766+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc34114a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000b0e0 tx=0x7ff4e000e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584ea035c2b752ca server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:32.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.766+0000 7ff4d2000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.766+0000 7ff4d2000700 1 -- 172.21.15.12:0/2758318921 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e002e040 con 0x55acc33c3b60 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.766+0000 7ff4d2000700 1 -- 172.21.15.12:0/2758318921 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4e0003a70 con 0x55acc33c3b60 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.766+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2758318921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55acc33ba920 con 0x55acc33c3b60 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.766+0000 7ff4d2000700 1 -- 172.21.15.12:0/2758318921 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4e0020070 con 0x55acc33c3b60 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.766+0000 7ff5073d5b80 10 client.8550 fetch_fsmap learned FSMap version 64 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.766+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2758318921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55acc31f80a0 con 0x55acc33c3b60 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.766+0000 7ff4d2000700 1 -- 172.21.15.12:0/2758318921 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55acc33ba920 con 0x55acc33c3b60 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.767+0000 7ff4d2000700 1 -- 172.21.15.12:0/2758318921 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff4e0003c30 con 0x55acc33c3b60 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.767+0000 7ff5073d5b80 10 client.8550 fetch_fsmap finished waiting for FSMap version 64 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.767+0000 7ff5073d5b80 10 client.8550 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.767+0000 7ff5073d5b80 20 client.8550 populate_metadata read hostname 'smithi012' 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.767+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.12:6833/3311625077,v1:172.21.15.12:6835/3311625077] conn(0x55acc33ee830 0x55acc342a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.767+0000 7ff5073d5b80 4 client.8550 mds_command: new command op to 8406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.767+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2758318921 --> [v2:172.21.15.12:6833/3311625077,v1:172.21.15.12:6835/3311625077] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55acc33976f0 con 0x55acc33ee830 2024-03-29T10:14:32.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.767+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.12:6833/3311625077,v1:172.21.15.12:6835/3311625077] conn(0x55acc33ee830 0x55acc342a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.768+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.12:6833/3311625077,v1:172.21.15.12:6835/3311625077] conn(0x55acc33ee830 0x55acc342a230 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c91910a1f4c51398 server_cookie=f1fc8068a8b6620f in_seq=0 out_seq=0 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.768+0000 7ff4d2000700 10 client.8550 ms_handle_connect on v2:172.21.15.12:6833/3311625077 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.768+0000 7ff4d2000700 1 -- 172.21.15.12:0/2758318921 <== mds.0 v2:172.21.15.12:6833/3311625077 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55acc33976f0 con 0x55acc33ee830 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.768+0000 7ff4d2000700 10 client.8550 handle_command_reply: tid=0 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.768+0000 7ff5073d5b80 1 client.8550 shutdown 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2758318921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 msgr2=0x55acc34114a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc34114a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000b0e0 tx=0x7ff4e000e000 comp rx=0 tx=0).stop 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2758318921 shutdown_connections 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc33f3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.12:6833/3311625077,v1:172.21.15.12:6835/3311625077] conn(0x55acc33ee830 0x55acc342a230 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc34114a0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/2758318921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2758318921 >> 172.21.15.12:0/2758318921 conn(0x55acc33266a0 msgr2=0x55acc340cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:32.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff4d2000700 0 client.8550 ms_handle_reset on v2:172.21.15.12:6833/3311625077 2024-03-29T10:14:32.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2758318921 shutdown_connections 2024-03-29T10:14:32.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 1 -- 172.21.15.12:0/2758318921 wait complete. 2024-03-29T10:14:32.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.769+0000 7ff5073d5b80 20 client.8550 trim_cache size 0 max 16384 2024-03-29T10:14:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.787+0000 7ff5073d5b80 1 Processor -- start 2024-03-29T10:14:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.787+0000 7ff5073d5b80 1 -- start start 2024-03-29T10:14:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.788+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33c3b60 0x55acc33c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.788+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c5be0 0x55acc33c2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.788+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33f2830 0x55acc33c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.788+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc310c390 con 0x55acc33c5be0 2024-03-29T10:14:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.788+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc3319c00 con 0x55acc33c3b60 2024-03-29T10:14:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.788+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acc33be880 con 0x55acc33f2830 2024-03-29T10:14:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.788+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c5be0 0x55acc33c2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.788+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c5be0 0x55acc33c2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59116/0 (socket says 172.21.15.12:59116) 2024-03-29T10:14:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.788+0000 7ff4ebe00700 1 -- 172.21.15.12:0/1972699182 learned_addr learned my addr 172.21.15.12:0/1972699182 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.789+0000 7ff4f1800700 1 --2- 172.21.15.12:0/1972699182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33f2830 0x55acc33c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.789+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/1972699182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33c3b60 0x55acc33c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.789+0000 7ff4ebe00700 1 -- 172.21.15.12:0/1972699182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33f2830 msgr2=0x55acc33c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.789+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/1972699182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33f2830 0x55acc33c93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.789+0000 7ff4ebe00700 1 -- 172.21.15.12:0/1972699182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33c3b60 msgr2=0x55acc33c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.789+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/1972699182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33c3b60 0x55acc33c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.789+0000 7ff4ebe00700 1 -- 172.21.15.12:0/1972699182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acc32de010 con 0x55acc33c5be0 2024-03-29T10:14:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.789+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/1972699182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c5be0 0x55acc33c2640 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec150b50 tx=0x7ff4d8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05c1268980b8dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.789+0000 7ff4de000700 1 -- 172.21.15.12:0/1972699182 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8006c20 con 0x55acc33c5be0 2024-03-29T10:14:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff4de000700 1 -- 172.21.15.12:0/1972699182 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d801c040 con 0x55acc33c5be0 2024-03-29T10:14:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff4de000700 1 -- 172.21.15.12:0/1972699182 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8005110 con 0x55acc33c5be0 2024-03-29T10:14:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1972699182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c5be0 msgr2=0x55acc33c2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/1972699182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c5be0 0x55acc33c2640 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec150b50 tx=0x7ff4d8009040 comp rx=0 tx=0).stop 2024-03-29T10:14:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1972699182 shutdown_connections 2024-03-29T10:14:32.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/1972699182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33f2830 0x55acc33c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/1972699182 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33c3b60 0x55acc33c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/1972699182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c5be0 0x55acc33c2640 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1972699182 >> 172.21.15.12:0/1972699182 conn(0x55acc33266a0 msgr2=0x55acc340eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1972699182 shutdown_connections 2024-03-29T10:14:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.790+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1972699182 wait complete. 2024-03-29T10:14:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.791+0000 7ff5073d5b80 1 -- start start 2024-03-29T10:14:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.791+0000 7ff5073d5b80 1 Processor -- start 2024-03-29T10:14:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc345a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff5073d5b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc340d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc34753a0 con 0x55acc33c3b60 2024-03-29T10:14:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc34755e0 con 0x55acc33f2830 2024-03-29T10:14:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff5073d5b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acc34745d0 con 0x55acc33c5be0 2024-03-29T10:14:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff5073d5b80 10 client.0 _ll_register_callbacks cb 0x7ff4e000bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:14:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff4f0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc345a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff4f0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc345a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59130/0 (socket says 172.21.15.12:59130) 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff4f0e00700 1 -- 172.21.15.12:0/1020820451 learned_addr learned my addr 172.21.15.12:0/1020820451 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.793+0000 7ff4f1800700 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc340d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4f0e00700 1 -- 172.21.15.12:0/1020820451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 msgr2=0x55acc33c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4f0e00700 1 -- 172.21.15.12:0/1020820451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 msgr2=0x55acc340d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc340d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4f0e00700 1 -- 172.21.15.12:0/1020820451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d800f040 con 0x55acc33c3b60 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4f0e00700 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc345a230 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0034790 tx=0x7ff4e00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7e115dff54e085 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4d2000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4d2000700 1 -- 172.21.15.12:0/1020820451 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e002e070 con 0x55acc33c3b60 2024-03-29T10:14:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4d2000700 1 -- 172.21.15.12:0/1020820451 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4e003f660 con 0x55acc33c3b60 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1020820451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4e405c120 con 0x55acc33c3b60 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff4d2000700 1 -- 172.21.15.12:0/1020820451 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4e000bb70 con 0x55acc33c3b60 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff5073d5b80 10 client.8556 fetch_fsmap learned FSMap version 64 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.794+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1020820451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55acc316c1d0 con 0x55acc33c3b60 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff4d2000700 1 -- 172.21.15.12:0/1020820451 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e405c120 con 0x55acc33c3b60 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff4d2000700 1 -- 172.21.15.12:0/1020820451 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff4e003f820 con 0x55acc33c3b60 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff5073d5b80 10 client.8556 fetch_fsmap finished waiting for FSMap version 64 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff5073d5b80 10 client.8556 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff5073d5b80 20 client.8556 populate_metadata read hostname 'smithi012' 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.12:6833/3311625077,v1:172.21.15.12:6835/3311625077] conn(0x55acc33c63a0 0x55acc33c6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff5073d5b80 4 client.8556 mds_command: new command op to 8406 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1020820451 --> [v2:172.21.15.12:6833/3311625077,v1:172.21.15.12:6835/3311625077] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55acc3151040 con 0x55acc33c63a0 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.12:6833/3311625077,v1:172.21.15.12:6835/3311625077] conn(0x55acc33c63a0 0x55acc33c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff4ebe00700 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.12:6833/3311625077,v1:172.21.15.12:6835/3311625077] conn(0x55acc33c63a0 0x55acc33c6780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb3e238c6c2afff6 server_cookie=8acd6352b9344744 in_seq=0 out_seq=0 2024-03-29T10:14:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.795+0000 7ff4d2000700 10 client.8556 ms_handle_connect on v2:172.21.15.12:6833/3311625077 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff4d2000700 1 -- 172.21.15.12:0/1020820451 <== mds.0 v2:172.21.15.12:6833/3311625077 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55acc3151040 con 0x55acc33c63a0 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff4d2000700 10 client.8556 handle_command_reply: tid=0 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 client.8556 shutdown 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1020820451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 msgr2=0x55acc345a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc345a230 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0034790 tx=0x7ff4e00047c0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1020820451 shutdown_connections 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55acc33f2830 0x55acc340d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.12:6833/3311625077,v1:172.21.15.12:6835/3311625077] conn(0x55acc33c63a0 0x55acc33c6780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55acc33c3b60 0x55acc345a230 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 --2- 172.21.15.12:0/1020820451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55acc33c5be0 0x55acc33c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff4d2000700 0 client.8556 ms_handle_reset on v2:172.21.15.12:6833/3311625077 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1020820451 >> 172.21.15.12:0/1020820451 conn(0x55acc33266a0 msgr2=0x55acc3411c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1020820451 shutdown_connections 2024-03-29T10:14:32.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 1 -- 172.21.15.12:0/1020820451 wait complete. 2024-03-29T10:14:32.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.796+0000 7ff5073d5b80 20 client.8556 trim_cache size 0 max 16384 2024-03-29T10:14:32.799 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:14:32.799 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T10:14:32.799 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T10:14:32.799 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 8406, 2024-03-29T10:14:32.799 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T10:14:32.799 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T10:14:32.799 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T10:14:32.799 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 6.9479534379999999, 2024-03-29T10:14:32.800 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 64, 2024-03-29T10:14:32.800 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 140, 2024-03-29T10:14:32.800 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 140, 2024-03-29T10:14:32.800 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12.533030875 2024-03-29T10:14:32.800 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 -- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff4cc06adb0 msgr2=0x7ff4cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff4cc06adb0 0x7ff4cc06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009c40 tx=0x7ff4e0008030 comp rx=0 tx=0).stop 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 -- 172.21.15.12:0/4047407735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15dfe0 msgr2=0x7ff4ec15cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15dfe0 0x7ff4ec15cc00 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4007ec0 tx=0x7ff4e400d3b0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 -- 172.21.15.12:0/4047407735 shutdown_connections 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff4cc06adb0 0x7ff4cc06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4ec15dfe0 0x7ff4ec15cc00 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4ec0ce840 0x7ff4ec14de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 --2- 172.21.15.12:0/4047407735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4ec15a5c0 0x7ff4ec14e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 -- 172.21.15.12:0/4047407735 >> 172.21.15.12:0/4047407735 conn(0x7ff4ec0b0d70 msgr2=0x7ff4ec0cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 -- 172.21.15.12:0/4047407735 shutdown_connections 2024-03-29T10:14:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:32.797+0000 7ff4f3800700 1 -- 172.21.15.12:0/4047407735 wait complete. 2024-03-29T10:14:32.813 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T10:14:33.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.013+0000 7f988aa00700 1 Processor -- start 2024-03-29T10:14:33.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.014+0000 7f988aa00700 1 -- start start 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.015+0000 7f988aa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f988415b5d0 0x7f98840ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.015+0000 7f988aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f98840ac7d0 0x7f988406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.015+0000 7f988aa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988415fd50 con 0x7f98840cb3f0 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.015+0000 7f988aa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988415b9b0 con 0x7f98840ac7d0 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.015+0000 7f988aa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988415f140 con 0x7f988415b5d0 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.015+0000 7f9888a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f98840ac7d0 0x7f988406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.015+0000 7f9888a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f98840ac7d0 0x7f988406c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59142/0 (socket says 172.21.15.12:59142) 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.015+0000 7f9888a00700 1 -- 172.21.15.12:0/3989412347 learned_addr learned my addr 172.21.15.12:0/3989412347 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.016+0000 7f9883e00700 1 --2- 172.21.15.12:0/3989412347 >> v2:172.21.15.159:3300/0 conn(0x7f988415b5d0 0x7f98840ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.016+0000 7f9888a00700 1 -- 172.21.15.12:0/3989412347 >> v2:172.21.15.159:3300/0 conn(0x7f988415b5d0 msgr2=0x7f98840ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.016+0000 7f9888a00700 1 --2- 172.21.15.12:0/3989412347 >> v2:172.21.15.159:3300/0 conn(0x7f988415b5d0 0x7f98840ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.016+0000 7f9888a00700 1 -- 172.21.15.12:0/3989412347 >> v1:172.21.15.159:6789/0 conn(0x7f98840cb3f0 legacy=0x7f98840ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:33.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.016+0000 7f9888a00700 1 -- 172.21.15.12:0/3989412347 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988415f420 con 0x7f98840ac7d0 2024-03-29T10:14:33.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.016+0000 7f9888a00700 1 --2- 172.21.15.12:0/3989412347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98840ac7d0 0x7f988406c9d0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f988406b8b0 tx=0x7f987c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33cb9f762bb33ae server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:33.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.016+0000 7f9882a00700 1 -- 172.21.15.12:0/3989412347 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00e070 con 0x7f98840ac7d0 2024-03-29T10:14:33.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.016+0000 7f9882a00700 1 -- 172.21.15.12:0/3989412347 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c00b9c0 con 0x7f98840ac7d0 2024-03-29T10:14:33.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.016+0000 7f9882a00700 1 -- 172.21.15.12:0/3989412347 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00bd70 con 0x7f98840ac7d0 2024-03-29T10:14:33.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.017+0000 7f988aa00700 1 -- 172.21.15.12:0/3989412347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98840ac7d0 msgr2=0x7f988406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.017+0000 7f988aa00700 1 --2- 172.21.15.12:0/3989412347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98840ac7d0 0x7f988406c9d0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f988406b8b0 tx=0x7f987c00b330 comp rx=0 tx=0).stop 2024-03-29T10:14:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.017+0000 7f988aa00700 1 -- 172.21.15.12:0/3989412347 shutdown_connections 2024-03-29T10:14:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.017+0000 7f988aa00700 1 --2- 172.21.15.12:0/3989412347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98840ac7d0 0x7f988406c9d0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.017+0000 7f988aa00700 1 --2- 172.21.15.12:0/3989412347 >> v2:172.21.15.159:3300/0 conn(0x7f988415b5d0 0x7f98840ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.017+0000 7f988aa00700 1 -- 172.21.15.12:0/3989412347 >> 172.21.15.12:0/3989412347 conn(0x7f98840cf190 msgr2=0x7f98840cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.017+0000 7f988aa00700 1 -- 172.21.15.12:0/3989412347 shutdown_connections 2024-03-29T10:14:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.017+0000 7f988aa00700 1 -- 172.21.15.12:0/3989412347 wait complete. 2024-03-29T10:14:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.019+0000 7f988aa00700 1 Processor -- start 2024-03-29T10:14:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.019+0000 7f988aa00700 1 -- start start 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f988aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98840ac7d0 0x7f988406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f988aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f988415b5d0 0x7f9884152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f988aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9884156db0 0x7f9884153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f988aa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98840c8b40 con 0x7f9884156db0 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f988aa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9884155000 con 0x7f98840ac7d0 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f988aa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9884070da0 con 0x7f988415b5d0 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f9888a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9884156db0 0x7f9884153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f9888a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9884156db0 0x7f9884153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59154/0 (socket says 172.21.15.12:59154) 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f9888a00700 1 -- 172.21.15.12:0/1720901948 learned_addr learned my addr 172.21.15.12:0/1720901948 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f9883e00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98840ac7d0 0x7f988406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.020+0000 7f9883400700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f988415b5d0 0x7f9884152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f9888a00700 1 -- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f988415b5d0 msgr2=0x7f9884152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f9888a00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f988415b5d0 0x7f9884152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f9888a00700 1 -- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98840ac7d0 msgr2=0x7f988406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f9888a00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98840ac7d0 0x7f988406d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f9888a00700 1 -- 172.21.15.12:0/1720901948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c009b30 con 0x7f9884156db0 2024-03-29T10:14:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f9888a00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9884156db0 0x7f9884153140 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f988406b930 tx=0x7f987c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f987be00700 1 -- 172.21.15.12:0/1720901948 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c004930 con 0x7f9884156db0 2024-03-29T10:14:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f987be00700 1 -- 172.21.15.12:0/1720901948 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c0099b0 con 0x7f9884156db0 2024-03-29T10:14:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f987be00700 1 -- 172.21.15.12:0/1720901948 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c01c680 con 0x7f9884156db0 2024-03-29T10:14:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f988aa00700 1 -- 172.21.15.12:0/1720901948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98841549b0 con 0x7f9884156db0 2024-03-29T10:14:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.021+0000 7f988aa00700 1 -- 172.21.15.12:0/1720901948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98840709c0 con 0x7f9884156db0 2024-03-29T10:14:33.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.022+0000 7f988aa00700 1 -- 172.21.15.12:0/1720901948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9884154810 con 0x7f9884156db0 2024-03-29T10:14:33.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.023+0000 7f987be00700 1 -- 172.21.15.12:0/1720901948 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f987c01c7e0 con 0x7f9884156db0 2024-03-29T10:14:33.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.024+0000 7f987be00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f986406adb0 0x7f986406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:33.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.024+0000 7f987be00700 1 -- 172.21.15.12:0/1720901948 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f987c08b250 con 0x7f9884156db0 2024-03-29T10:14:33.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.024+0000 7f9883e00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f986406adb0 0x7f986406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:33.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.025+0000 7f9883e00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f986406adb0 0x7f986406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f98840cb070 tx=0x7f9874008040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:33.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.029+0000 7f987be00700 1 -- 172.21.15.12:0/1720901948 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9884154810 con 0x7f9884156db0 2024-03-29T10:14:33.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.289+0000 7f988aa00700 1 -- 172.21.15.12:0/1720901948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9884154810 con 0x7f9884156db0 2024-03-29T10:14:33.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.290+0000 7f987be00700 1 -- 172.21.15.12:0/1720901948 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v64) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9884154810 con 0x7f9884156db0 2024-03-29T10:14:33.288 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 -- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f986406adb0 msgr2=0x7f986406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f986406adb0 0x7f986406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f98840cb070 tx=0x7f9874008040 comp rx=0 tx=0).stop 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 -- 172.21.15.12:0/1720901948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9884156db0 msgr2=0x7f9884153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9884156db0 0x7f9884153140 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f988406b930 tx=0x7f987c004ad0 comp rx=0 tx=0).stop 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 -- 172.21.15.12:0/1720901948 shutdown_connections 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f986406adb0 0x7f986406d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9884156db0 0x7f9884153140 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98840ac7d0 0x7f988406d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 --2- 172.21.15.12:0/1720901948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f988415b5d0 0x7f9884152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 -- 172.21.15.12:0/1720901948 >> 172.21.15.12:0/1720901948 conn(0x7f98840cf190 msgr2=0x7f98840afbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 -- 172.21.15.12:0/1720901948 shutdown_connections 2024-03-29T10:14:33.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:33.292+0000 7f988aa00700 1 -- 172.21.15.12:0/1720901948 wait complete. 2024-03-29T10:14:33.309 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T10:14:33.309 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T10:14:33.309 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T10:14:33.309 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T10:14:33.309 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T10:14:33.309 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T10:14:33.309 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T10:14:33.309 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T10:14:33.309 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:14:33.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:33.310 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T10:14:33.356 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T10:14:33.356 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T10:14:33.356 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T10:14:33.356 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:14:33.356 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T10:14:33.356 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:14:33.356 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:14:33.356 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:14:33.356 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T10:14:33.356 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 40244sec preferred_lft 40244sec 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:14:33.357 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T10:14:33.358 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:14:33.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:33.359 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T10:14:33.424 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T10:14:33.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:33.427 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T10:14:33.475 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T10:14:33.475 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T10:14:33.475 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T10:14:33.475 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T10:14:33.475 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T10:14:33.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:33.476 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T10:14:33.476 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T10:14:33.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T10:14:33.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T10:14:33.477 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T10:14:33.477 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T10:14:33.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:33.640 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T10:14:33.686 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T10:14:33.687 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T10:14:33.687 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:14:33.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:33.687 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T10:14:33.795 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:14:33.795 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T10:14:33.839 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:14:33.839 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:14:33.839 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:14:33.923 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T10:14:33.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:33.923 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:14:34.065 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:14:34.066 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:14:34.091 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:14:34.092 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:14:34.092 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T10:14:34.092 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T10:14:34.092 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T10:14:34.097 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T10:14:34.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:34.097 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:14:34.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:34.166 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T10:14:34.265 INFO:teuthology.orchestra.run.smithi012.stdout:client7307 2024-03-29T10:14:34.265 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:14:34.265 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:34.310 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T10:14:34.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.551+0000 7f71f6200700 1 Processor -- start 2024-03-29T10:14:34.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.551+0000 7f71f6200700 1 -- start start 2024-03-29T10:14:34.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.552+0000 7f71f6200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f71f00d16a0 0x7f71f00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:34.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.552+0000 7f71f6200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f71f00d2370 0x7f71f00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:34.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.552+0000 7f71f6200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f0161420 con 0x7f71f015d9d0 2024-03-29T10:14:34.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.552+0000 7f71f6200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f00d4340 con 0x7f71f00d2370 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.552+0000 7f71f6200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f015edc0 con 0x7f71f00d16a0 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.552+0000 7f71ef400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f71f015d9d0 0x7f71f00d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40136/0 (socket says 172.21.15.12:40136) 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.552+0000 7f71ef400700 1 -- 172.21.15.12:0/3380368315 learned_addr learned my addr 172.21.15.12:0/3380368315 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.552+0000 7f71efe00700 1 --2- 172.21.15.12:0/3380368315 >> v2:172.21.15.12:3300/0 conn(0x7f71f00d2370 0x7f71f00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71eea00700 1 --2- 172.21.15.12:0/3380368315 >> v2:172.21.15.159:3300/0 conn(0x7f71f00d16a0 0x7f71f00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71ee000700 1 -- 172.21.15.12:0/3380368315 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493354042 0 0) 0x7f71f0161420 con 0x7f71f015d9d0 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71ee000700 1 -- 172.21.15.12:0/3380368315 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d40035b0 con 0x7f71f015d9d0 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71efe00700 1 -- 172.21.15.12:0/3380368315 >> v2:172.21.15.159:3300/0 conn(0x7f71f00d16a0 msgr2=0x7f71f00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71efe00700 1 --2- 172.21.15.12:0/3380368315 >> v2:172.21.15.159:3300/0 conn(0x7f71f00d16a0 0x7f71f00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71efe00700 1 -- 172.21.15.12:0/3380368315 >> v1:172.21.15.12:6789/0 conn(0x7f71f015d9d0 legacy=0x7f71f00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71efe00700 1 -- 172.21.15.12:0/3380368315 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f0161700 con 0x7f71f00d2370 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71eea00700 1 --2- 172.21.15.12:0/3380368315 >> v2:172.21.15.159:3300/0 conn(0x7f71f00d16a0 0x7f71f00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71efe00700 1 --2- 172.21.15.12:0/3380368315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 0x7f71f00ce360 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f71f00d12f0 tx=0x7f71e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9057a4c895e665 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71ee000700 1 -- 172.21.15.12:0/3380368315 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e800e070 con 0x7f71f00d2370 2024-03-29T10:14:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71ee000700 1 -- 172.21.15.12:0/3380368315 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e800bb20 con 0x7f71f00d2370 2024-03-29T10:14:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.553+0000 7f71ee000700 1 -- 172.21.15.12:0/3380368315 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e8012430 con 0x7f71f00d2370 2024-03-29T10:14:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.554+0000 7f71f6200700 1 -- 172.21.15.12:0/3380368315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 msgr2=0x7f71f00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.554+0000 7f71f6200700 1 --2- 172.21.15.12:0/3380368315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 0x7f71f00ce360 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f71f00d12f0 tx=0x7f71e800b330 comp rx=0 tx=0).stop 2024-03-29T10:14:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.554+0000 7f71f6200700 1 -- 172.21.15.12:0/3380368315 shutdown_connections 2024-03-29T10:14:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.554+0000 7f71f6200700 1 --2- 172.21.15.12:0/3380368315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 0x7f71f00ce360 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.554+0000 7f71f6200700 1 --2- 172.21.15.12:0/3380368315 >> v2:172.21.15.159:3300/0 conn(0x7f71f00d16a0 0x7f71f00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.554+0000 7f71f6200700 1 -- 172.21.15.12:0/3380368315 >> 172.21.15.12:0/3380368315 conn(0x7f71f008aff0 msgr2=0x7f71f00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.554+0000 7f71f6200700 1 -- 172.21.15.12:0/3380368315 shutdown_connections 2024-03-29T10:14:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.554+0000 7f71f6200700 1 -- 172.21.15.12:0/3380368315 wait complete. 2024-03-29T10:14:34.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.556+0000 7f71f6200700 1 Processor -- start 2024-03-29T10:14:34.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.556+0000 7f71f6200700 1 -- start start 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.557+0000 7f71f6200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f00d16a0 0x7f71f015ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.557+0000 7f71f6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 0x7f71f01f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.557+0000 7f71f6200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f015d9d0 0x7f71f01f70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.557+0000 7f71f6200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f00cb570 con 0x7f71f00d2370 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.557+0000 7f71f6200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f00cb850 con 0x7f71f00d16a0 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.557+0000 7f71f6200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f00cbb30 con 0x7f71f015d9d0 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.557+0000 7f71eea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 0x7f71f01f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.557+0000 7f71eea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 0x7f71f01f6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59164/0 (socket says 172.21.15.12:59164) 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.557+0000 7f71eea00700 1 -- 172.21.15.12:0/1082802738 learned_addr learned my addr 172.21.15.12:0/1082802738 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.557+0000 7f71ef400700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f00d16a0 0x7f71f015ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71efe00700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f015d9d0 0x7f71f01f70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71eea00700 1 -- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f015d9d0 msgr2=0x7f71f01f70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71eea00700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f015d9d0 0x7f71f01f70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71eea00700 1 -- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f00d16a0 msgr2=0x7f71f015ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71eea00700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f00d16a0 0x7f71f015ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71eea00700 1 -- 172.21.15.12:0/1082802738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e8009a70 con 0x7f71f00d2370 2024-03-29T10:14:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71eea00700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 0x7f71f01f6ba0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00e810 tx=0x7f71dc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71e7400700 1 -- 172.21.15.12:0/1082802738 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc00c070 con 0x7f71f00d2370 2024-03-29T10:14:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71e7400700 1 -- 172.21.15.12:0/1082802738 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71dc005c20 con 0x7f71f00d2370 2024-03-29T10:14:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71e7400700 1 -- 172.21.15.12:0/1082802738 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc006030 con 0x7f71f00d2370 2024-03-29T10:14:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.558+0000 7f71f6200700 1 -- 172.21.15.12:0/1082802738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f00cbe10 con 0x7f71f00d2370 2024-03-29T10:14:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.559+0000 7f71f6200700 1 -- 172.21.15.12:0/1082802738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f00ca820 con 0x7f71f00d2370 2024-03-29T10:14:34.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.560+0000 7f71f6200700 1 -- 172.21.15.12:0/1082802738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f00d1ca0 con 0x7f71f00d2370 2024-03-29T10:14:34.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.561+0000 7f71e7400700 1 -- 172.21.15.12:0/1082802738 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f71dc006190 con 0x7f71f00d2370 2024-03-29T10:14:34.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.562+0000 7f71e7400700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71d006adb0 0x7f71d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:34.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.562+0000 7f71e7400700 1 -- 172.21.15.12:0/1082802738 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71dc0892e0 con 0x7f71f00d2370 2024-03-29T10:14:34.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.562+0000 7f71ef400700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71d006adb0 0x7f71d006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:34.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.563+0000 7f71ef400700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71d006adb0 0x7f71d006d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f71e0007fd0 tx=0x7f71e0007dc0 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:34.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.567+0000 7f71e7400700 1 -- 172.21.15.12:0/1082802738 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71dc054200 con 0x7f71f00d2370 2024-03-29T10:14:34.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.810+0000 7f71f6200700 1 -- 172.21.15.12:0/1082802738 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f71f0161fa0 con 0x7f71d006adb0 2024-03-29T10:14:34.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.812+0000 7f71e7400700 1 -- 172.21.15.12:0/1082802738 <== mgr.8190 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f71e00058e0 con 0x7f71d006adb0 2024-03-29T10:14:34.810 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:14:34.811 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:14:34.811 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T10:14:34.811 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:14:34.811 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:14:34.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.814+0000 7f71f6200700 1 -- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71d006adb0 msgr2=0x7f71d006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:34.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.814+0000 7f71f6200700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71d006adb0 0x7f71d006d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f71e0007fd0 tx=0x7f71e0007dc0 comp rx=0 tx=0).stop 2024-03-29T10:14:34.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.814+0000 7f71f6200700 1 -- 172.21.15.12:0/1082802738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 msgr2=0x7f71f01f6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:34.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.814+0000 7f71f6200700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 0x7f71f01f6ba0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00e810 tx=0x7f71dc00eb20 comp rx=0 tx=0).stop 2024-03-29T10:14:34.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.814+0000 7f71f6200700 1 -- 172.21.15.12:0/1082802738 shutdown_connections 2024-03-29T10:14:34.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.814+0000 7f71f6200700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71d006adb0 0x7f71d006d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:34.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.814+0000 7f71f6200700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71f015d9d0 0x7f71f01f70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:34.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.814+0000 7f71f6200700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f00d16a0 0x7f71f015ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:34.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.814+0000 7f71f6200700 1 --2- 172.21.15.12:0/1082802738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71f00d2370 0x7f71f01f6ba0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:34.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.814+0000 7f71f6200700 1 -- 172.21.15.12:0/1082802738 >> 172.21.15.12:0/1082802738 conn(0x7f71f008aff0 msgr2=0x7f71f006cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:34.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.815+0000 7f71f6200700 1 -- 172.21.15.12:0/1082802738 shutdown_connections 2024-03-29T10:14:34.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:34.815+0000 7f71f6200700 1 -- 172.21.15.12:0/1082802738 wait complete. 2024-03-29T10:14:34.831 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T10:14:35.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.032+0000 7f2144000700 1 Processor -- start 2024-03-29T10:14:35.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.032+0000 7f2144000700 1 -- start start 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.033+0000 7f2144000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f213c15d9d0 0x7f213c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.033+0000 7f2144000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f213c0d16a0 0x7f213c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.033+0000 7f2144000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f213c0d2370 0x7f213c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.033+0000 7f2144000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213c0d2820 con 0x7f213c0d2370 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.033+0000 7f2144000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213c161420 con 0x7f213c15d9d0 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.033+0000 7f2144000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213c161700 con 0x7f213c0d16a0 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.033+0000 7f2142000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f213c0d2370 0x7f213c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.033+0000 7f2142000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f213c0d2370 0x7f213c15ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59180/0 (socket says 172.21.15.12:59180) 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.033+0000 7f2142000700 1 -- 172.21.15.12:0/3174027655 learned_addr learned my addr 172.21.15.12:0/3174027655 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.033+0000 7f2141600700 1 --2- 172.21.15.12:0/3174027655 >> v2:172.21.15.159:3300/0 conn(0x7f213c15d9d0 0x7f213c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.034+0000 7f2140c00700 1 --2- 172.21.15.12:0/3174027655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f213c0d16a0 0x7f213c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.034+0000 7f2142000700 1 -- 172.21.15.12:0/3174027655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f213c0d16a0 msgr2=0x7f213c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:35.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.034+0000 7f2142000700 1 --2- 172.21.15.12:0/3174027655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f213c0d16a0 0x7f213c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:35.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.034+0000 7f2142000700 1 -- 172.21.15.12:0/3174027655 >> v2:172.21.15.159:3300/0 conn(0x7f213c15d9d0 msgr2=0x7f213c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:35.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.034+0000 7f2142000700 1 --2- 172.21.15.12:0/3174027655 >> v2:172.21.15.159:3300/0 conn(0x7f213c15d9d0 0x7f213c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:35.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.034+0000 7f2142000700 1 -- 172.21.15.12:0/3174027655 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c1619e0 con 0x7f213c0d2370 2024-03-29T10:14:35.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.034+0000 7f2142000700 1 --2- 172.21.15.12:0/3174027655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d2370 0x7f213c15ddb0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f213c0d0ff0 tx=0x7f213400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ea399b5332a249 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:35.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.034+0000 7f213be00700 1 -- 172.21.15.12:0/3174027655 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213400e070 con 0x7f213c0d2370 2024-03-29T10:14:35.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.034+0000 7f213be00700 1 -- 172.21.15.12:0/3174027655 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213400b910 con 0x7f213c0d2370 2024-03-29T10:14:35.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.034+0000 7f213be00700 1 -- 172.21.15.12:0/3174027655 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213400bcc0 con 0x7f213c0d2370 2024-03-29T10:14:35.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.035+0000 7f2144000700 1 -- 172.21.15.12:0/3174027655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d2370 msgr2=0x7f213c15ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:35.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.035+0000 7f2144000700 1 --2- 172.21.15.12:0/3174027655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d2370 0x7f213c15ddb0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f213c0d0ff0 tx=0x7f213400b330 comp rx=0 tx=0).stop 2024-03-29T10:14:35.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.035+0000 7f2144000700 1 -- 172.21.15.12:0/3174027655 shutdown_connections 2024-03-29T10:14:35.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.035+0000 7f2144000700 1 --2- 172.21.15.12:0/3174027655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d2370 0x7f213c15ddb0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:35.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.035+0000 7f2144000700 1 --2- 172.21.15.12:0/3174027655 >> v2:172.21.15.159:3300/0 conn(0x7f213c15d9d0 0x7f213c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:35.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.035+0000 7f2144000700 1 --2- 172.21.15.12:0/3174027655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f213c0d16a0 0x7f213c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:35.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.035+0000 7f2144000700 1 -- 172.21.15.12:0/3174027655 >> 172.21.15.12:0/3174027655 conn(0x7f213c08aff0 msgr2=0x7f213c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:35.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.035+0000 7f2144000700 1 -- 172.21.15.12:0/3174027655 shutdown_connections 2024-03-29T10:14:35.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.035+0000 7f2144000700 1 -- 172.21.15.12:0/3174027655 wait complete. 2024-03-29T10:14:35.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.037+0000 7f2144000700 1 Processor -- start 2024-03-29T10:14:35.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.037+0000 7f2144000700 1 -- start start 2024-03-29T10:14:35.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.038+0000 7f2144000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d16a0 0x7f213c1569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:35.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.038+0000 7f2144000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f213c0d2370 0x7f213c1e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:35.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.038+0000 7f2144000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f213c15d9d0 0x7f213c15fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:35.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.038+0000 7f2144000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c1e8670 con 0x7f213c0d16a0 2024-03-29T10:14:35.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.038+0000 7f2144000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c1e8950 con 0x7f213c0d2370 2024-03-29T10:14:35.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.038+0000 7f2144000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213c0cbdd0 con 0x7f213c15d9d0 2024-03-29T10:14:35.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.038+0000 7f2141600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d16a0 0x7f213c1569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:35.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.038+0000 7f2141600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d16a0 0x7f213c1569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59194/0 (socket says 172.21.15.12:59194) 2024-03-29T10:14:35.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.038+0000 7f2141600700 1 -- 172.21.15.12:0/3437181201 learned_addr learned my addr 172.21.15.12:0/3437181201 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:35.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.038+0000 7f2140c00700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f213c0d2370 0x7f213c1e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2142000700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f213c15d9d0 0x7f213c15fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2141600700 1 -- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f213c15d9d0 msgr2=0x7f213c15fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2141600700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f213c15d9d0 0x7f213c15fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2141600700 1 -- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f213c0d2370 msgr2=0x7f213c1e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2141600700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f213c0d2370 0x7f213c1e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2141600700 1 -- 172.21.15.12:0/3437181201 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2134009b30 con 0x7f213c0d16a0 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2141600700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d16a0 0x7f213c1569d0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f213000e870 tx=0x7f213000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2139600700 1 -- 172.21.15.12:0/3437181201 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2130009590 con 0x7f213c0d16a0 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2139600700 1 -- 172.21.15.12:0/3437181201 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213000ee70 con 0x7f213c0d16a0 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2144000700 1 -- 172.21.15.12:0/3437181201 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213c0cc0b0 con 0x7f213c0d16a0 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2139600700 1 -- 172.21.15.12:0/3437181201 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213000ca50 con 0x7f213c0d16a0 2024-03-29T10:14:35.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.039+0000 7f2144000700 1 -- 172.21.15.12:0/3437181201 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213c1e8e20 con 0x7f213c0d16a0 2024-03-29T10:14:35.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.040+0000 7f2144000700 1 -- 172.21.15.12:0/3437181201 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f213c0d1ca0 con 0x7f213c0d16a0 2024-03-29T10:14:35.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.041+0000 7f2139600700 1 -- 172.21.15.12:0/3437181201 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 68) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f213000cbb0 con 0x7f213c0d16a0 2024-03-29T10:14:35.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.042+0000 7f2139600700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f211c06adb0 0x7f211c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:35.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.042+0000 7f2139600700 1 -- 172.21.15.12:0/3437181201 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2130089370 con 0x7f213c0d16a0 2024-03-29T10:14:35.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.043+0000 7f2140c00700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f211c06adb0 0x7f211c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:35.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.044+0000 7f2140c00700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f211c06adb0 0x7f211c06d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f213c160a60 tx=0x7f2128008040 comp rx=0 tx=0).ready entity=mgr.8190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:35.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.047+0000 7f2139600700 1 -- 172.21.15.12:0/3437181201 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21300544d0 con 0x7f213c0d16a0 2024-03-29T10:14:35.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:35.321+0000 7f2144000700 1 -- 172.21.15.12:0/3437181201 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f213c067720 con 0x7f213c0d16a0 2024-03-29T10:14:36.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.155+0000 7f2139600700 1 -- 172.21.15.12:0/3437181201 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 69) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f21300548a0 con 0x7f213c0d16a0 2024-03-29T10:14:36.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.156+0000 7f2139600700 1 -- 172.21.15.12:0/3437181201 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v69) v1 ==== 92+0+0 (secure 0 0 0) 0x7f21300874c0 con 0x7f213c0d16a0 2024-03-29T10:14:36.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 -- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f211c06adb0 msgr2=0x7f211c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f211c06adb0 0x7f211c06d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f213c160a60 tx=0x7f2128008040 comp rx=0 tx=0).stop 2024-03-29T10:14:36.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 -- 172.21.15.12:0/3437181201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d16a0 msgr2=0x7f213c1569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d16a0 0x7f213c1569d0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f213000e870 tx=0x7f213000ec30 comp rx=0 tx=0).stop 2024-03-29T10:14:36.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 -- 172.21.15.12:0/3437181201 shutdown_connections 2024-03-29T10:14:36.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f211c06adb0 0x7f211c06d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f213c15d9d0 0x7f213c15fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c0d16a0 0x7f213c1569d0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 --2- 172.21.15.12:0/3437181201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f213c0d2370 0x7f213c1e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 -- 172.21.15.12:0/3437181201 >> 172.21.15.12:0/3437181201 conn(0x7f213c08aff0 msgr2=0x7f213c0b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:36.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.159+0000 7f2144000700 1 -- 172.21.15.12:0/3437181201 shutdown_connections 2024-03-29T10:14:36.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.160+0000 7f2144000700 1 -- 172.21.15.12:0/3437181201 wait complete. 2024-03-29T10:14:36.179 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T10:14:36.375 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:36.378+0000 7f9e3d572240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:14:36.387 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:36.389+0000 7f56054c9240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:14:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.391+0000 7f6616400700 1 Processor -- start 2024-03-29T10:14:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.392+0000 7f6616400700 1 -- start start 2024-03-29T10:14:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f6616400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f66100b2b30 0x7f66100b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f6616400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66100b3960 0x7f66100cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f6616400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66100b3d70 con 0x7f6610157f90 2024-03-29T10:14:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f6616400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6610158370 con 0x7f66100b3960 2024-03-29T10:14:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f6616400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66100b2710 con 0x7f66100b2b30 2024-03-29T10:14:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f660fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66100b3960 0x7f66100cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f660ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f66100b2b30 0x7f66100b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f660fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66100b3960 0x7f66100cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59224/0 (socket says 172.21.15.12:59224) 2024-03-29T10:14:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f660ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f66100b2b30 0x7f66100b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:49508/0 (socket says 172.21.15.12:49508) 2024-03-29T10:14:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f660fe00700 1 -- 172.21.15.12:0/1692428451 learned_addr learned my addr 172.21.15.12:0/1692428451 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.393+0000 7f660f400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f6610157f90 0x7f66101613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:57538/0 (socket says 172.21.15.12:57538) 2024-03-29T10:14:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.394+0000 7f660e000700 1 -- 172.21.15.12:0/1692428451 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252393697 0 0) 0x7f66100b3d70 con 0x7f6610157f90 2024-03-29T10:14:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.394+0000 7f660fe00700 1 -- 172.21.15.12:0/1692428451 >> v2:172.21.15.159:3300/0 conn(0x7f66100b2b30 msgr2=0x7f66100b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.394+0000 7f660fe00700 1 --2- 172.21.15.12:0/1692428451 >> v2:172.21.15.159:3300/0 conn(0x7f66100b2b30 0x7f66100b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.394+0000 7f660fe00700 1 -- 172.21.15.12:0/1692428451 >> v1:172.21.15.159:6789/0 conn(0x7f6610157f90 legacy=0x7f66101613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.394+0000 7f660fe00700 1 -- 172.21.15.12:0/1692428451 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66100b2070 con 0x7f66100b3960 2024-03-29T10:14:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.394+0000 7f660fe00700 1 --2- 172.21.15.12:0/1692428451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 0x7f66100cd020 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f66100b4210 tx=0x7f660800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c6ac30d38c3ad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.394+0000 7f660e000700 1 -- 172.21.15.12:0/1692428451 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660800e070 con 0x7f66100b3960 2024-03-29T10:14:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.394+0000 7f660e000700 1 -- 172.21.15.12:0/1692428451 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660800b910 con 0x7f66100b3960 2024-03-29T10:14:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.394+0000 7f660e000700 1 -- 172.21.15.12:0/1692428451 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660800bcc0 con 0x7f66100b3960 2024-03-29T10:14:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.395+0000 7f6616400700 1 -- 172.21.15.12:0/1692428451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 msgr2=0x7f66100cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.395+0000 7f6616400700 1 --2- 172.21.15.12:0/1692428451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 0x7f66100cd020 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f66100b4210 tx=0x7f660800b330 comp rx=0 tx=0).stop 2024-03-29T10:14:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.395+0000 7f6616400700 1 -- 172.21.15.12:0/1692428451 shutdown_connections 2024-03-29T10:14:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.395+0000 7f6616400700 1 --2- 172.21.15.12:0/1692428451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 0x7f66100cd020 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.395+0000 7f6616400700 1 --2- 172.21.15.12:0/1692428451 >> v2:172.21.15.159:3300/0 conn(0x7f66100b2b30 0x7f66100b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.395+0000 7f6616400700 1 -- 172.21.15.12:0/1692428451 >> 172.21.15.12:0/1692428451 conn(0x7f66100b0d70 msgr2=0x7f66100b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.395+0000 7f6616400700 1 -- 172.21.15.12:0/1692428451 shutdown_connections 2024-03-29T10:14:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.395+0000 7f6616400700 1 -- 172.21.15.12:0/1692428451 wait complete. 2024-03-29T10:14:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.397+0000 7f6616400700 1 Processor -- start 2024-03-29T10:14:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.397+0000 7f6616400700 1 -- start start 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f6616400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66100b2b30 0x7f6610160870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f6616400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 0x7f661009c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f6616400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6610157f90 0x7f661006da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f6616400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6610161d20 con 0x7f66100b3960 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f6616400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66101610a0 con 0x7f6610157f90 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f6616400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6610161380 con 0x7f66100b2b30 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f660ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 0x7f661009c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f660ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 0x7f661009c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59238/0 (socket says 172.21.15.12:59238) 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f660ea00700 1 -- 172.21.15.12:0/546085083 learned_addr learned my addr 172.21.15.12:0/546085083 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f660f400700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66100b2b30 0x7f6610160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.398+0000 7f660fe00700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6610157f90 0x7f661006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.399+0000 7f660ea00700 1 -- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66100b2b30 msgr2=0x7f6610160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.399+0000 7f660ea00700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66100b2b30 0x7f6610160870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.399+0000 7f660ea00700 1 -- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6610157f90 msgr2=0x7f661006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.399+0000 7f660ea00700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6610157f90 0x7f661006da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.399+0000 7f660ea00700 1 -- 172.21.15.12:0/546085083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6608009b30 con 0x7f66100b3960 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.399+0000 7f660ea00700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 0x7f661009c8a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00e870 tx=0x7f65fc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.399+0000 7f6607400700 1 -- 172.21.15.12:0/546085083 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc009590 con 0x7f66100b3960 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.399+0000 7f6607400700 1 -- 172.21.15.12:0/546085083 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65fc013680 con 0x7f66100b3960 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.399+0000 7f6607400700 1 -- 172.21.15.12:0/546085083 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc0139f0 con 0x7f66100b3960 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.400+0000 7f6616400700 1 -- 172.21.15.12:0/546085083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6610161660 con 0x7f66100b3960 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.400+0000 7f6616400700 1 -- 172.21.15.12:0/546085083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6610160db0 con 0x7f66100b3960 2024-03-29T10:14:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.401+0000 7f6616400700 1 -- 172.21.15.12:0/546085083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6610067720 con 0x7f66100b3960 2024-03-29T10:14:36.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.402+0000 7f6607400700 1 -- 172.21.15.12:0/546085083 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 69) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f65fc013b50 con 0x7f66100b3960 2024-03-29T10:14:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.402+0000 7f6607400700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65f006ae00 0x7f65f006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.402+0000 7f6607400700 1 -- 172.21.15.12:0/546085083 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65fc089360 con 0x7f66100b3960 2024-03-29T10:14:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.403+0000 7f660f400700 1 -- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65f006ae00 msgr2=0x7f65f006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.403+0000 7f660f400700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65f006ae00 0x7f65f006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:36.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.407+0000 7f6607400700 1 -- 172.21.15.12:0/546085083 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65fc054420 con 0x7f66100b3960 2024-03-29T10:14:36.561 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:36.563+0000 7f9e3d572240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:14:36.594 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:36.595+0000 7f56054c9240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:14:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.604+0000 7f660f400700 1 -- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65f006ae00 msgr2=0x7f65f006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:36.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.604+0000 7f660f400700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65f006ae00 0x7f65f006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:36.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.660+0000 7f6616400700 1 -- 172.21.15.12:0/546085083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f66100ce2e0 con 0x7f66100b3960 2024-03-29T10:14:36.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.664+0000 7f6607400700 1 -- 172.21.15.12:0/546085083 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f65fc057a40 con 0x7f66100b3960 2024-03-29T10:14:36.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.666+0000 7f6616400700 1 -- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65f006ae00 msgr2=0x7f65f006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:36.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.666+0000 7f6616400700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65f006ae00 0x7f65f006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.666+0000 7f6616400700 1 -- 172.21.15.12:0/546085083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 msgr2=0x7f661009c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.666+0000 7f6616400700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 0x7f661009c8a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00e870 tx=0x7f65fc00eb80 comp rx=0 tx=0).stop 2024-03-29T10:14:36.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.666+0000 7f6616400700 1 -- 172.21.15.12:0/546085083 shutdown_connections 2024-03-29T10:14:36.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.666+0000 7f6616400700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65f006ae00 0x7f65f006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.666+0000 7f6616400700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6610157f90 0x7f661006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.666+0000 7f6616400700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66100b2b30 0x7f6610160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.666+0000 7f6616400700 1 --2- 172.21.15.12:0/546085083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66100b3960 0x7f661009c8a0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.666+0000 7f6616400700 1 -- 172.21.15.12:0/546085083 >> 172.21.15.12:0/546085083 conn(0x7f66100b0d70 msgr2=0x7f66100b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:36.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.667+0000 7f6616400700 1 -- 172.21.15.12:0/546085083 shutdown_connections 2024-03-29T10:14:36.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.667+0000 7f6616400700 1 -- 172.21.15.12:0/546085083 wait complete. 2024-03-29T10:14:36.683 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T10:14:36.876 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:36.878+0000 7f9e3d572240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:14:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.888+0000 7f9e61200700 1 Processor -- start 2024-03-29T10:14:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.889+0000 7f9e61200700 1 -- start start 2024-03-29T10:14:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.890+0000 7f9e61200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5c0cbb30 0x7f9e5c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.890+0000 7f9e61200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e5c0cd4a0 0x7f9e5c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.890+0000 7f9e61200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c0cc450 con 0x7f9e5c0cd020 2024-03-29T10:14:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.890+0000 7f9e61200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5c0cdae0 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.890+0000 7f9e61200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0cddc0 con 0x7f9e5c0cbb30 2024-03-29T10:14:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.890+0000 7f9e5a800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e5c0cd4a0 0x7f9e5c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.890+0000 7f9e59400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9e5c0cd020 0x7f9e5c0b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40152/0 (socket says 172.21.15.12:40152) 2024-03-29T10:14:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.890+0000 7f9e59400700 1 -- 172.21.15.12:0/2527240711 learned_addr learned my addr 172.21.15.12:0/2527240711 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.890+0000 7f9e59e00700 1 --2- 172.21.15.12:0/2527240711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5c0cbb30 0x7f9e5c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.891+0000 7f9e5a800700 1 -- 172.21.15.12:0/2527240711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5c0cbb30 msgr2=0x7f9e5c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.891+0000 7f9e5a800700 1 --2- 172.21.15.12:0/2527240711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5c0cbb30 0x7f9e5c0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.891+0000 7f9e5a800700 1 -- 172.21.15.12:0/2527240711 >> v1:172.21.15.12:6789/0 conn(0x7f9e5c0cd020 legacy=0x7f9e5c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.891+0000 7f9e5a800700 1 -- 172.21.15.12:0/2527240711 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0ce0a0 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.891+0000 7f9e58a00700 1 -- 172.21.15.12:0/2527240711 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833861855 0 0) 0x7f9e5c0cc450 con 0x7f9e5c0cd020 2024-03-29T10:14:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.891+0000 7f9e5a800700 1 --2- 172.21.15.12:0/2527240711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 0x7f9e5c06e420 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009d90 tx=0x7f9e5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5889584e5fe5139b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.891+0000 7f9e58a00700 1 -- 172.21.15.12:0/2527240711 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5400e070 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.891+0000 7f9e58a00700 1 -- 172.21.15.12:0/2527240711 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5400bb20 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.891+0000 7f9e58a00700 1 -- 172.21.15.12:0/2527240711 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54012430 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.892+0000 7f9e61200700 1 -- 172.21.15.12:0/2527240711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 msgr2=0x7f9e5c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.892+0000 7f9e61200700 1 --2- 172.21.15.12:0/2527240711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 0x7f9e5c06e420 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009d90 tx=0x7f9e5400b330 comp rx=0 tx=0).stop 2024-03-29T10:14:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.892+0000 7f9e61200700 1 -- 172.21.15.12:0/2527240711 shutdown_connections 2024-03-29T10:14:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.892+0000 7f9e61200700 1 --2- 172.21.15.12:0/2527240711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 0x7f9e5c06e420 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.892+0000 7f9e61200700 1 --2- 172.21.15.12:0/2527240711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5c0cbb30 0x7f9e5c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.892+0000 7f9e61200700 1 -- 172.21.15.12:0/2527240711 >> 172.21.15.12:0/2527240711 conn(0x7f9e5c0b0d70 msgr2=0x7f9e5c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.892+0000 7f9e61200700 1 -- 172.21.15.12:0/2527240711 shutdown_connections 2024-03-29T10:14:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.892+0000 7f9e61200700 1 -- 172.21.15.12:0/2527240711 wait complete. 2024-03-29T10:14:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.894+0000 7f9e61200700 1 Processor -- start 2024-03-29T10:14:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.894+0000 7f9e61200700 1 -- start start 2024-03-29T10:14:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e61200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e5c0cbb30 0x7f9e5c153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e61200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5c0cd020 0x7f9e5c1f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e61200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 0x7f9e5c156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e61200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15c660 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e61200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15c8b0 con 0x7f9e5c0cbb30 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e61200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15cb90 con 0x7f9e5c0cd020 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e5a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 0x7f9e5c156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e5a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 0x7f9e5c156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59252/0 (socket says 172.21.15.12:59252) 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e5a800700 1 -- 172.21.15.12:0/2551960509 learned_addr learned my addr 172.21.15.12:0/2551960509 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e59400700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5c0cd020 0x7f9e5c1f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.895+0000 7f9e59e00700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e5c0cbb30 0x7f9e5c153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.896+0000 7f9e5a800700 1 -- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5c0cd020 msgr2=0x7f9e5c1f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.896+0000 7f9e5a800700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5c0cd020 0x7f9e5c1f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.896+0000 7f9e5a800700 1 -- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e5c0cbb30 msgr2=0x7f9e5c153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.896+0000 7f9e5a800700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e5c0cbb30 0x7f9e5c153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.896+0000 7f9e5a800700 1 -- 172.21.15.12:0/2551960509 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54009a70 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.896+0000 7f9e5a800700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 0x7f9e5c156130 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c06c130 tx=0x7f9e54004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.896+0000 7f9e52000700 1 -- 172.21.15.12:0/2551960509 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54012430 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.897+0000 7f9e52000700 1 -- 172.21.15.12:0/2551960509 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e54020070 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.897+0000 7f9e52000700 1 -- 172.21.15.12:0/2551960509 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54004280 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.897+0000 7f9e61200700 1 -- 172.21.15.12:0/2551960509 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c15ce70 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.897+0000 7f9e61200700 1 -- 172.21.15.12:0/2551960509 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c1f25e0 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.898+0000 7f9e61200700 1 -- 172.21.15.12:0/2551960509 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c15d390 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.899+0000 7f9e52000700 1 -- 172.21.15.12:0/2551960509 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 69) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f9e540043e0 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.900+0000 7f9e52000700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9e3c06ae00 0x7f9e3c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:36.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.900+0000 7f9e52000700 1 -- 172.21.15.12:0/2551960509 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e54022070 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.900+0000 7f9e59e00700 1 -- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9e3c06ae00 msgr2=0x7f9e3c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:36.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.900+0000 7f9e59e00700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9e3c06ae00 0x7f9e3c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:36.905+0000 7f9e52000700 1 -- 172.21.15.12:0/2551960509 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e540561e0 con 0x7f9e5c0cd4a0 2024-03-29T10:14:36.960 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:36.962+0000 7f56054c9240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:14:37.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.101+0000 7f9e59e00700 1 -- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9e3c06ae00 msgr2=0x7f9e3c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:37.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.101+0000 7f9e59e00700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9e3c06ae00 0x7f9e3c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.151+0000 7f9e61200700 1 -- 172.21.15.12:0/2551960509 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9e5c15bda0 con 0x7f9e5c0cd4a0 2024-03-29T10:14:37.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.159+0000 7f9e52000700 1 -- 172.21.15.12:0/2551960509 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9e54059800 con 0x7f9e5c0cd4a0 2024-03-29T10:14:37.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 -- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9e3c06ae00 msgr2=0x7f9e3c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:37.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9e3c06ae00 0x7f9e3c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 -- 172.21.15.12:0/2551960509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 msgr2=0x7f9e5c156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 0x7f9e5c156130 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c06c130 tx=0x7f9e54004ad0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 -- 172.21.15.12:0/2551960509 shutdown_connections 2024-03-29T10:14:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9e3c06ae00 0x7f9e3c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5c0cd4a0 0x7f9e5c156130 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e5c0cbb30 0x7f9e5c153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 --2- 172.21.15.12:0/2551960509 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5c0cd020 0x7f9e5c1f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 -- 172.21.15.12:0/2551960509 >> 172.21.15.12:0/2551960509 conn(0x7f9e5c0b0d70 msgr2=0x7f9e5c159620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 -- 172.21.15.12:0/2551960509 shutdown_connections 2024-03-29T10:14:37.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.161+0000 7f9e61200700 1 -- 172.21.15.12:0/2551960509 wait complete. 2024-03-29T10:14:37.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:37.179 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2024-03-29T10:14:37.229 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2024-03-29T10:14:37.402 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:37.405+0000 7f9e3d572240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:14:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.442+0000 7f54c4600700 1 Processor -- start 2024-03-29T10:14:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.442+0000 7f54c4600700 1 -- start start 2024-03-29T10:14:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.443+0000 7f54c4600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc157f90 0x7f54bc0b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.444+0000 7f54c4600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f54bc0b3b30 0x7f54bc161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.444+0000 7f54c4600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc0b3070 con 0x7f54bc1613e0 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.444+0000 7f54c4600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54bc158370 con 0x7f54bc0b3b30 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.444+0000 7f54c4600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0b2710 con 0x7f54bc157f90 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.444+0000 7f54c1c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc157f90 0x7f54bc0b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.444+0000 7f54c2600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f54bc0b3b30 0x7f54bc161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.444+0000 7f54c1200700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f54bc1613e0 0x7f54bc0b3420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:57542/0 (socket says 172.21.15.12:57542) 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.444+0000 7f54c1200700 1 -- 172.21.15.12:0/2987049974 learned_addr learned my addr 172.21.15.12:0/2987049974 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.444+0000 7f54bbe00700 1 -- 172.21.15.12:0/2987049974 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812280868 0 0) 0x7f54a8001110 con 0x7f54bc1613e0 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.445+0000 7f54bbe00700 1 -- 172.21.15.12:0/2987049974 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a40035b0 con 0x7f54bc1613e0 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.445+0000 7f54c1c00700 1 -- 172.21.15.12:0/2987049974 >> v2:172.21.15.159:3300/0 conn(0x7f54bc0b3b30 msgr2=0x7f54bc161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.445+0000 7f54c1c00700 1 --2- 172.21.15.12:0/2987049974 >> v2:172.21.15.159:3300/0 conn(0x7f54bc0b3b30 0x7f54bc161860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.445+0000 7f54c1c00700 1 -- 172.21.15.12:0/2987049974 >> v1:172.21.15.159:6789/0 conn(0x7f54bc1613e0 legacy=0x7f54bc0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.445+0000 7f54c1c00700 1 -- 172.21.15.12:0/2987049974 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc0b2070 con 0x7f54bc157f90 2024-03-29T10:14:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.445+0000 7f54c2600700 1 --2- 172.21.15.12:0/2987049974 >> v2:172.21.15.159:3300/0 conn(0x7f54bc0b3b30 0x7f54bc161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.445+0000 7f54c1c00700 1 --2- 172.21.15.12:0/2987049974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc157f90 0x7f54bc0b2b30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009840 tx=0x7f54b0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f0e0178a4cdb97c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.445+0000 7f54bbe00700 1 -- 172.21.15.12:0/2987049974 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0011830 con 0x7f54bc157f90 2024-03-29T10:14:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.445+0000 7f54bbe00700 1 -- 172.21.15.12:0/2987049974 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b0011990 con 0x7f54bc157f90 2024-03-29T10:14:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.445+0000 7f54bbe00700 1 -- 172.21.15.12:0/2987049974 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0011d20 con 0x7f54bc157f90 2024-03-29T10:14:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.446+0000 7f54c4600700 1 -- 172.21.15.12:0/2987049974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc157f90 msgr2=0x7f54bc0b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.446+0000 7f54c4600700 1 --2- 172.21.15.12:0/2987049974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc157f90 0x7f54bc0b2b30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009840 tx=0x7f54b0009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.446+0000 7f54c4600700 1 -- 172.21.15.12:0/2987049974 shutdown_connections 2024-03-29T10:14:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.446+0000 7f54c4600700 1 --2- 172.21.15.12:0/2987049974 >> v2:172.21.15.159:3300/0 conn(0x7f54bc0b3b30 0x7f54bc161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.446+0000 7f54c4600700 1 --2- 172.21.15.12:0/2987049974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc157f90 0x7f54bc0b2b30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.446+0000 7f54c4600700 1 -- 172.21.15.12:0/2987049974 >> 172.21.15.12:0/2987049974 conn(0x7f54bc0b0d70 msgr2=0x7f54bc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.446+0000 7f54c4600700 1 -- 172.21.15.12:0/2987049974 shutdown_connections 2024-03-29T10:14:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.446+0000 7f54c4600700 1 -- 172.21.15.12:0/2987049974 wait complete. 2024-03-29T10:14:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.448+0000 7f54c4600700 1 Processor -- start 2024-03-29T10:14:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.448+0000 7f54c4600700 1 -- start start 2024-03-29T10:14:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c4600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc0b3b30 0x7f54bc0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c4600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc157f90 0x7f54bc0c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c4600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc1613e0 0x7f54bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c4600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0caea0 con 0x7f54bc157f90 2024-03-29T10:14:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c4600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0cb180 con 0x7f54bc1613e0 2024-03-29T10:14:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c4600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0cb460 con 0x7f54bc0b3b30 2024-03-29T10:14:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc157f90 0x7f54bc0c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c1c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc0b3b30 0x7f54bc0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c2600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc1613e0 0x7f54bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc157f90 0x7f54bc0c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59266/0 (socket says 172.21.15.12:59266) 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.449+0000 7f54c1200700 1 -- 172.21.15.12:0/3089071725 learned_addr learned my addr 172.21.15.12:0/3089071725 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54c1c00700 1 -- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc1613e0 msgr2=0x7f54bc06d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54c1c00700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc1613e0 0x7f54bc06d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54c1c00700 1 -- 172.21.15.12:0/3089071725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc157f90 msgr2=0x7f54bc0c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54c1c00700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc157f90 0x7f54bc0c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54c1c00700 1 -- 172.21.15.12:0/3089071725 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b4009a70 con 0x7f54bc0b3b30 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54c2600700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc1613e0 0x7f54bc06d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54c1c00700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc0b3b30 0x7f54bc0cf240 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009840 tx=0x7f54b0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0004550 con 0x7f54bc0b3b30 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b001f070 con 0x7f54bc0b3b30 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54c4600700 1 -- 172.21.15.12:0/3089071725 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b00094a0 con 0x7f54bc0b3b30 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.450+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b00038a0 con 0x7f54bc0b3b30 2024-03-29T10:14:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.451+0000 7f54c4600700 1 -- 172.21.15.12:0/3089071725 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc0cba20 con 0x7f54bc0b3b30 2024-03-29T10:14:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.452+0000 7f54c4600700 1 -- 172.21.15.12:0/3089071725 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54bc0cbd90 con 0x7f54bc0b3b30 2024-03-29T10:14:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.452+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 69) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f54b0003a00 con 0x7f54bc0b3b30 2024-03-29T10:14:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.453+0000 7f54b9600700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.453+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54b0021070 con 0x7f54bc0b3b30 2024-03-29T10:14:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.454+0000 7f54c1200700 1 -- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 msgr2=0x7f549c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.454+0000 7f54c1200700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:37.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.458+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54b0055d30 con 0x7f54bc0b3b30 2024-03-29T10:14:37.544 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:37.546+0000 7f56054c9240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:14:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.655+0000 7f54c1200700 1 -- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 msgr2=0x7f549c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.655+0000 7f54c1200700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:37.658 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:37.661+0000 7f9e3d572240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:14:37.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:37.700+0000 7f54c4600700 1 -- 172.21.15.12:0/3089071725 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f54bc162230 con 0x7f549c06ae00 2024-03-29T10:14:37.812 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:37.814+0000 7f56054c9240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:14:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:38.056+0000 7f54c1200700 1 -- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 msgr2=0x7f549c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:38.056+0000 7f54c1200700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T10:14:38.116 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:38.118+0000 7f9e3d572240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:14:38.293 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:38.295+0000 7f56054c9240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:14:38.821 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:38.823+0000 7f9e3d572240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:14:38.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:38.857+0000 7f54c1200700 1 -- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 msgr2=0x7f549c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:38.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:38.857+0000 7f54c1200700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T10:14:38.911 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:38.914+0000 7f9e3d572240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:14:39.039 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:39.041+0000 7f56054c9240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:14:39.071 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:39.074+0000 7f9e3d572240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:14:39.135 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:39.137+0000 7f56054c9240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:14:39.285 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:39.288+0000 7f9e3d572240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:14:39.303 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:39.305+0000 7f56054c9240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:14:39.377 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:39.379+0000 7f9e3d572240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:14:39.475 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:39.477+0000 7f9e3d572240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:14:39.523 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:39.525+0000 7f56054c9240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:14:39.574 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:39.576+0000 7f9e3d572240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:14:39.619 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:39.621+0000 7f56054c9240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:14:39.671 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:39.674+0000 7f9e3d572240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:14:39.722 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:39.724+0000 7f56054c9240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:14:39.825 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:39.827+0000 7f56054c9240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:14:39.883 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:39.885+0000 7f9e3d572240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:14:39.930 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:39.932+0000 7f56054c9240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:14:40.146 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:40.149+0000 7f9e3d572240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:14:40.152 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:40.154+0000 7f56054c9240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:14:40.338 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:40.341+0000 7f9e3d572240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:14:40.418 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:40.420+0000 7f56054c9240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:14:40.431 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:40.434+0000 7f9e3d572240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:40.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:40.459+0000 7f54c1200700 1 -- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 msgr2=0x7f549c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:40.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:40.459+0000 7f54c1200700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T10:14:40.618 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:40.619+0000 7f56054c9240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:14:40.715 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:40.717+0000 7f56054c9240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:41.160 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:41.162+0000 7f9e3d572240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:14:41.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:41.374+0000 7f9e3d572240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:14:41.484 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:41.486+0000 7f56054c9240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:14:41.576 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:41.579+0000 7f9e3d572240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:41.707 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:41.709+0000 7f56054c9240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:14:41.926 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:41.928+0000 7f56054c9240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:42.055 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:42.057+0000 7f9e3d572240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:14:42.262 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:42.265+0000 7f9e3d572240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:42.430 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:42.432+0000 7f56054c9240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:14:42.647 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:42.649+0000 7f56054c9240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:42.706 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:42.709+0000 7f9e3d572240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:14:42.818 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:42.820+0000 7f9e3d572240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:42.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:42.832+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mgrmap(e 70) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f54b0056100 con 0x7f54bc0b3b30 2024-03-29T10:14:42.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:42.832+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 msgr2=0x7f549c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:42.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:42.832+0000 7f54b9600700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:43.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:43.021+0000 7f9da5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:43.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:43.021+0000 7f9da5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:43.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:43.021+0000 7f9da5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:43.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:43.021+0000 7f9da5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:43.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:43.021+0000 7f9da5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:43.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:43.022+0000 7f9da9800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:43.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:43.022+0000 7f9da9800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:43.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:43.022+0000 7f9da9800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:43.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:43.022+0000 7f9da9800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:43.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:43.022+0000 7f9da9800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:43.119 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:43.121+0000 7f56054c9240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:14:43.236 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:43.238+0000 7f56054c9240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.836+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 <== mon.2 v2:172.21.15.159:3301/0 8 ==== mgrmap(e 71) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f54b005d4c0 con 0x7f54bc0b3b30 2024-03-29T10:14:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.837+0000 7f54b9600700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.837+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f549c00b9b0 con 0x7f549c06ae00 2024-03-29T10:14:43.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.838+0000 7f54c1200700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:43.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.840+0000 7f54c1200700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f54a8007fd0 tx=0x7f54a8007d00 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.922+0000 7f54b9600700 1 -- 172.21.15.12:0/3089071725 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f549c00b9b0 con 0x7f549c06ae00 2024-03-29T10:14:43.921 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1/minutes 2024-03-29T10:14:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 -- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 msgr2=0x7f549c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f54a8007fd0 tx=0x7f54a8007d00 comp rx=0 tx=0).stop 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 -- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc0b3b30 msgr2=0x7f54bc0cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc0b3b30 0x7f54bc0cf240 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009840 tx=0x7f54b0004750 comp rx=0 tx=0).stop 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 -- 172.21.15.12:0/3089071725 shutdown_connections 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f549c06ae00 0x7f549c06d2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc1613e0 0x7f54bc06d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc0b3b30 0x7f54bc0cf240 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 --2- 172.21.15.12:0/3089071725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc157f90 0x7f54bc0c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 -- 172.21.15.12:0/3089071725 >> 172.21.15.12:0/3089071725 conn(0x7f54bc0b0d70 msgr2=0x7f54bc15cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 -- 172.21.15.12:0/3089071725 shutdown_connections 2024-03-29T10:14:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:43.925+0000 7f54c4600700 1 -- 172.21.15.12:0/3089071725 wait complete. 2024-03-29T10:14:43.944 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2024-03-29T10:14:43.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:43.945 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2024-03-29T10:14:43.993 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2024-03-29T10:14:44.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.196+0000 7ff1cc400700 1 Processor -- start 2024-03-29T10:14:44.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.196+0000 7ff1cc400700 1 -- start start 2024-03-29T10:14:44.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1cc400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff1c415d800 0x7ff1c40725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1cc400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1c40ce230 0x7ff1c40ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1cc400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff1c40b2070 0x7ff1c40b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1cc400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c40b40f0 con 0x7ff1c40b2070 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1cc400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c40709c0 con 0x7ff1c415d800 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1cc400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c4070c80 con 0x7ff1c40ce230 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1ca400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff1c40b2070 0x7ff1c40b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1ca400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff1c40b2070 0x7ff1c40b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48518/0 (socket says 172.21.15.12:48518) 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1ca400700 1 -- 172.21.15.12:0/2862322229 learned_addr learned my addr 172.21.15.12:0/2862322229 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1c9000700 1 --2- 172.21.15.12:0/2862322229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1c40ce230 0x7ff1c40ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1c9a00700 1 --2- 172.21.15.12:0/2862322229 >> v2:172.21.15.159:3300/0 conn(0x7ff1c415d800 0x7ff1c40725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1ca400700 1 -- 172.21.15.12:0/2862322229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1c40ce230 msgr2=0x7ff1c40ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1ca400700 1 --2- 172.21.15.12:0/2862322229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1c40ce230 0x7ff1c40ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1ca400700 1 -- 172.21.15.12:0/2862322229 >> v2:172.21.15.159:3300/0 conn(0x7ff1c415d800 msgr2=0x7ff1c40725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1ca400700 1 --2- 172.21.15.12:0/2862322229 >> v2:172.21.15.159:3300/0 conn(0x7ff1c415d800 0x7ff1c40725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.198+0000 7ff1ca400700 1 -- 172.21.15.12:0/2862322229 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c4070f60 con 0x7ff1c40b2070 2024-03-29T10:14:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.199+0000 7ff1ca400700 1 --2- 172.21.15.12:0/2862322229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40b2070 0x7ff1c40b2500 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff1c406c100 tx=0x7ff1bc00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff18840748a9d878 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.199+0000 7ff1c3e00700 1 -- 172.21.15.12:0/2862322229 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc010070 con 0x7ff1c40b2070 2024-03-29T10:14:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.199+0000 7ff1c3e00700 1 -- 172.21.15.12:0/2862322229 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc00dad0 con 0x7ff1c40b2070 2024-03-29T10:14:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.199+0000 7ff1c3e00700 1 -- 172.21.15.12:0/2862322229 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc00de60 con 0x7ff1c40b2070 2024-03-29T10:14:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.199+0000 7ff1cc400700 1 -- 172.21.15.12:0/2862322229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40b2070 msgr2=0x7ff1c40b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.199+0000 7ff1cc400700 1 --2- 172.21.15.12:0/2862322229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40b2070 0x7ff1c40b2500 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff1c406c100 tx=0x7ff1bc00d430 comp rx=0 tx=0).stop 2024-03-29T10:14:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.200+0000 7ff1cc400700 1 -- 172.21.15.12:0/2862322229 shutdown_connections 2024-03-29T10:14:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.200+0000 7ff1cc400700 1 --2- 172.21.15.12:0/2862322229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40b2070 0x7ff1c40b2500 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.200+0000 7ff1cc400700 1 --2- 172.21.15.12:0/2862322229 >> v2:172.21.15.159:3300/0 conn(0x7ff1c415d800 0x7ff1c40725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.200+0000 7ff1cc400700 1 --2- 172.21.15.12:0/2862322229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1c40ce230 0x7ff1c40ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.200+0000 7ff1cc400700 1 -- 172.21.15.12:0/2862322229 >> 172.21.15.12:0/2862322229 conn(0x7ff1c40b0d70 msgr2=0x7ff1c40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:44.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.200+0000 7ff1cc400700 1 -- 172.21.15.12:0/2862322229 shutdown_connections 2024-03-29T10:14:44.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.200+0000 7ff1cc400700 1 -- 172.21.15.12:0/2862322229 wait complete. 2024-03-29T10:14:44.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.202+0000 7ff1cc400700 1 Processor -- start 2024-03-29T10:14:44.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.202+0000 7ff1cc400700 1 -- start start 2024-03-29T10:14:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.203+0000 7ff1cc400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1c40b2070 0x7ff1c414f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1cc400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40ce230 0x7ff1c41485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1cc400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1c415d800 0x7ff1c40ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1cc400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40ae0f0 con 0x7ff1c40ce230 2024-03-29T10:14:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1cc400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c4150ef0 con 0x7ff1c415d800 2024-03-29T10:14:44.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1cc400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c41511d0 con 0x7ff1c40b2070 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1c9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40ce230 0x7ff1c41485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1c9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40ce230 0x7ff1c41485e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48532/0 (socket says 172.21.15.12:48532) 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1c9000700 1 -- 172.21.15.12:0/3823193973 learned_addr learned my addr 172.21.15.12:0/3823193973 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1c9a00700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1c40b2070 0x7ff1c414f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1ca400700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1c415d800 0x7ff1c40ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1c9000700 1 -- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1c40b2070 msgr2=0x7ff1c414f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1c9000700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1c40b2070 0x7ff1c414f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1c9000700 1 -- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1c415d800 msgr2=0x7ff1c40ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1c9000700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1c415d800 0x7ff1c40ad830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1c9000700 1 -- 172.21.15.12:0/3823193973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc00bd40 con 0x7ff1c40ce230 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1ca400700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1c415d800 0x7ff1c40ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.204+0000 7ff1c9000700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40ce230 0x7ff1c41485e0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000e930 tx=0x7ff1b000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.205+0000 7ff1c1600700 1 -- 172.21.15.12:0/3823193973 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000ee80 con 0x7ff1c40ce230 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.205+0000 7ff1c1600700 1 -- 172.21.15.12:0/3823193973 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b000c850 con 0x7ff1c40ce230 2024-03-29T10:14:44.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.205+0000 7ff1c1600700 1 -- 172.21.15.12:0/3823193973 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000cbe0 con 0x7ff1c40ce230 2024-03-29T10:14:44.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.205+0000 7ff1cc400700 1 -- 172.21.15.12:0/3823193973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c41514b0 con 0x7ff1c40ce230 2024-03-29T10:14:44.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.205+0000 7ff1cc400700 1 -- 172.21.15.12:0/3823193973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c40ae3d0 con 0x7ff1c40ce230 2024-03-29T10:14:44.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.206+0000 7ff1cc400700 1 -- 172.21.15.12:0/3823193973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c4151820 con 0x7ff1c40ce230 2024-03-29T10:14:44.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.207+0000 7ff1c1600700 1 -- 172.21.15.12:0/3823193973 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 71) v1 ==== 88992+0+0 (secure 0 0 0) 0x7ff1b000cd40 con 0x7ff1c40ce230 2024-03-29T10:14:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.208+0000 7ff1c1600700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff1a406ab30 0x7ff1a406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.208+0000 7ff1c1600700 1 -- 172.21.15.12:0/3823193973 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff1b0089220 con 0x7ff1c40ce230 2024-03-29T10:14:44.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.209+0000 7ff1c9a00700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff1a406ab30 0x7ff1a406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.209+0000 7ff1c9a00700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff1a406ab30 0x7ff1a406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8009b40 tx=0x7ff1b8009210 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.213+0000 7ff1c1600700 1 -- 172.21.15.12:0/3823193973 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff1b0054510 con 0x7ff1c40ce230 2024-03-29T10:14:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.453+0000 7ff1cc400700 1 -- 172.21.15.12:0/3823193973 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1c4072b00 con 0x7ff1a406ab30 2024-03-29T10:14:44.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.528+0000 7ff1c1600700 1 -- 172.21.15.12:0/3823193973 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7ff1b8004040 con 0x7ff1a406ab30 2024-03-29T10:14:44.527 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1/hourly 2024-03-29T10:14:44.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 -- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff1a406ab30 msgr2=0x7ff1a406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:44.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff1a406ab30 0x7ff1a406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8009b40 tx=0x7ff1b8009210 comp rx=0 tx=0).stop 2024-03-29T10:14:44.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 -- 172.21.15.12:0/3823193973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40ce230 msgr2=0x7ff1c41485e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:44.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40ce230 0x7ff1c41485e0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000e930 tx=0x7ff1b000ec40 comp rx=0 tx=0).stop 2024-03-29T10:14:44.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 -- 172.21.15.12:0/3823193973 shutdown_connections 2024-03-29T10:14:44.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff1a406ab30 0x7ff1a406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1c415d800 0x7ff1c40ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1c40b2070 0x7ff1c414f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 --2- 172.21.15.12:0/3823193973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1c40ce230 0x7ff1c41485e0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 -- 172.21.15.12:0/3823193973 >> 172.21.15.12:0/3823193973 conn(0x7ff1c40b0d70 msgr2=0x7ff1c4153360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:44.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 -- 172.21.15.12:0/3823193973 shutdown_connections 2024-03-29T10:14:44.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.531+0000 7ff1cc400700 1 -- 172.21.15.12:0/3823193973 wait complete. 2024-03-29T10:14:44.547 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2024-03-29T10:14:44.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:44.548 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2024-03-29T10:14:44.594 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2024-03-29T10:14:44.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.796+0000 7f5722c00700 1 Processor -- start 2024-03-29T10:14:44.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.796+0000 7f5722c00700 1 -- start start 2024-03-29T10:14:44.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.797+0000 7f5722c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f571c15d9d0 0x7f571c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.797+0000 7f5722c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f571c0d2540 0x7f571c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.797+0000 7f5722c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c161420 con 0x7f571c0d16a0 2024-03-29T10:14:44.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.797+0000 7f5722c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571c15edc0 con 0x7f571c15d9d0 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.797+0000 7f5722c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571c161700 con 0x7f571c0d2540 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.797+0000 7f571be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f571c15d9d0 0x7f571c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.797+0000 7f571be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f571c15d9d0 0x7f571c0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48534/0 (socket says 172.21.15.12:48534) 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.797+0000 7f571be00700 1 -- 172.21.15.12:0/2085401929 learned_addr learned my addr 172.21.15.12:0/2085401929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.797+0000 7f571b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f571c0d16a0 0x7f571c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38388/0 (socket says 172.21.15.12:38388) 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f571be00700 1 -- 172.21.15.12:0/2085401929 >> v2:172.21.15.159:3300/0 conn(0x7f571c0d2540 msgr2=0x7f571c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f5720c00700 1 --2- 172.21.15.12:0/2085401929 >> v2:172.21.15.159:3300/0 conn(0x7f571c0d2540 0x7f571c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f571be00700 1 --2- 172.21.15.12:0/2085401929 >> v2:172.21.15.159:3300/0 conn(0x7f571c0d2540 0x7f571c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f571be00700 1 -- 172.21.15.12:0/2085401929 >> v1:172.21.15.12:6789/0 conn(0x7f571c0d16a0 legacy=0x7f571c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f571be00700 1 -- 172.21.15.12:0/2085401929 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c1619e0 con 0x7f571c15d9d0 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f5720c00700 1 --2- 172.21.15.12:0/2085401929 >> v2:172.21.15.159:3300/0 conn(0x7f571c0d2540 0x7f571c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f571aa00700 1 -- 172.21.15.12:0/2085401929 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055356875 0 0) 0x7f571c161420 con 0x7f571c0d16a0 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f571be00700 1 --2- 172.21.15.12:0/2085401929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c15d9d0 0x7f571c0d3c30 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f571c08ca10 tx=0x7f570c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a7a93b4e142bd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:44.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f571aa00700 1 -- 172.21.15.12:0/2085401929 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c00b870 con 0x7f571c15d9d0 2024-03-29T10:14:44.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f571aa00700 1 -- 172.21.15.12:0/2085401929 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c00b9d0 con 0x7f571c15d9d0 2024-03-29T10:14:44.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.798+0000 7f571aa00700 1 -- 172.21.15.12:0/2085401929 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c00bd60 con 0x7f571c15d9d0 2024-03-29T10:14:44.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.799+0000 7f5722c00700 1 -- 172.21.15.12:0/2085401929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c15d9d0 msgr2=0x7f571c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:44.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.799+0000 7f5722c00700 1 --2- 172.21.15.12:0/2085401929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c15d9d0 0x7f571c0d3c30 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f571c08ca10 tx=0x7f570c009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.799+0000 7f5722c00700 1 -- 172.21.15.12:0/2085401929 shutdown_connections 2024-03-29T10:14:44.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.799+0000 7f5722c00700 1 --2- 172.21.15.12:0/2085401929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c15d9d0 0x7f571c0d3c30 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.799+0000 7f5722c00700 1 --2- 172.21.15.12:0/2085401929 >> v2:172.21.15.159:3300/0 conn(0x7f571c0d2540 0x7f571c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.799+0000 7f5722c00700 1 -- 172.21.15.12:0/2085401929 >> 172.21.15.12:0/2085401929 conn(0x7f571c08aff0 msgr2=0x7f571c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:44.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.799+0000 7f5722c00700 1 -- 172.21.15.12:0/2085401929 shutdown_connections 2024-03-29T10:14:44.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.799+0000 7f5722c00700 1 -- 172.21.15.12:0/2085401929 wait complete. 2024-03-29T10:14:44.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.801+0000 7f5722c00700 1 Processor -- start 2024-03-29T10:14:44.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.801+0000 7f5722c00700 1 -- start start 2024-03-29T10:14:44.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f5722c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c0d16a0 0x7f571c0b7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f5722c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c0d2540 0x7f571c0cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f5722c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15d9d0 0x7f571c0b8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f5722c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c15cbd0 con 0x7f571c0d2540 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f5722c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0b7980 con 0x7f571c0d16a0 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f5722c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0b63a0 con 0x7f571c15d9d0 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f571b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c0d2540 0x7f571c0cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f571b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c0d2540 0x7f571c0cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48536/0 (socket says 172.21.15.12:48536) 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f571b400700 1 -- 172.21.15.12:0/432444885 learned_addr learned my addr 172.21.15.12:0/432444885 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f571be00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c0d16a0 0x7f571c0b7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f5720c00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15d9d0 0x7f571c0b8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f571b400700 1 -- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15d9d0 msgr2=0x7f571c0b8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f571b400700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15d9d0 0x7f571c0b8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f571b400700 1 -- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c0d16a0 msgr2=0x7f571c0b7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f571b400700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c0d16a0 0x7f571c0b7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f571b400700 1 -- 172.21.15.12:0/432444885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c009550 con 0x7f571c0d2540 2024-03-29T10:14:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.802+0000 7f5720c00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15d9d0 0x7f571c0b8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:44.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.803+0000 7f571b400700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c0d2540 0x7f571c0cb000 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f570800c520 tx=0x7f570800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:44.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.803+0000 7f5713e00700 1 -- 172.21.15.12:0/432444885 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57080074e0 con 0x7f571c0d2540 2024-03-29T10:14:44.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.803+0000 7f5713e00700 1 -- 172.21.15.12:0/432444885 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5708007660 con 0x7f571c0d2540 2024-03-29T10:14:44.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.803+0000 7f5713e00700 1 -- 172.21.15.12:0/432444885 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5708005a70 con 0x7f571c0d2540 2024-03-29T10:14:44.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.803+0000 7f5722c00700 1 -- 172.21.15.12:0/432444885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c0b6680 con 0x7f571c0d2540 2024-03-29T10:14:44.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.803+0000 7f5722c00700 1 -- 172.21.15.12:0/432444885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c0b6c40 con 0x7f571c0d2540 2024-03-29T10:14:44.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.804+0000 7f5722c00700 1 -- 172.21.15.12:0/432444885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571c0d1ca0 con 0x7f571c0d2540 2024-03-29T10:14:44.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.805+0000 7f5713e00700 1 -- 172.21.15.12:0/432444885 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 71) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f5708005bd0 con 0x7f571c0d2540 2024-03-29T10:14:44.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.806+0000 7f5713e00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06ab30 0x7f56fc06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:44.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.806+0000 7f5713e00700 1 -- 172.21.15.12:0/432444885 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f57080886d0 con 0x7f571c0d2540 2024-03-29T10:14:44.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.806+0000 7f571be00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06ab30 0x7f56fc06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:44.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.807+0000 7f571be00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06ab30 0x7f56fc06cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f570c014040 tx=0x7f570c008030 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:44.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.810+0000 7f5713e00700 1 -- 172.21.15.12:0/432444885 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5708053930 con 0x7f571c0d2540 2024-03-29T10:14:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:44.890+0000 7f5713e00700 1 -- 172.21.15.12:0/432444885 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 72) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f5708053d00 con 0x7f571c0d2540 2024-03-29T10:14:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.054+0000 7f5722c00700 1 -- 172.21.15.12:0/432444885 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7f571c161fa0 con 0x7f56fc06ab30 2024-03-29T10:14:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.129+0000 7f5713e00700 1 -- 172.21.15.12:0/432444885 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f570c00bb40 con 0x7f56fc06ab30 2024-03-29T10:14:45.128 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1/daily 2024-03-29T10:14:45.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 -- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06ab30 msgr2=0x7f56fc06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:45.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06ab30 0x7f56fc06cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f570c014040 tx=0x7f570c008030 comp rx=0 tx=0).stop 2024-03-29T10:14:45.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 -- 172.21.15.12:0/432444885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c0d2540 msgr2=0x7f571c0cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:45.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c0d2540 0x7f571c0cb000 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f570800c520 tx=0x7f570800c830 comp rx=0 tx=0).stop 2024-03-29T10:14:45.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 -- 172.21.15.12:0/432444885 shutdown_connections 2024-03-29T10:14:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f56fc06ab30 0x7f56fc06cff0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c15d9d0 0x7f571c0b8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c0d16a0 0x7f571c0b7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 --2- 172.21.15.12:0/432444885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f571c0d2540 0x7f571c0cb000 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 -- 172.21.15.12:0/432444885 >> 172.21.15.12:0/432444885 conn(0x7f571c08aff0 msgr2=0x7f571c0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 -- 172.21.15.12:0/432444885 shutdown_connections 2024-03-29T10:14:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.132+0000 7f5722c00700 1 -- 172.21.15.12:0/432444885 wait complete. 2024-03-29T10:14:45.150 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2024-03-29T10:14:45.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:45.150 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2024-03-29T10:14:45.196 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2024-03-29T10:14:45.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.440+0000 7f074f400700 1 Processor -- start 2024-03-29T10:14:45.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.441+0000 7f074f400700 1 -- start start 2024-03-29T10:14:45.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.442+0000 7f074f400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f07480d16a0 0x7f07480d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:45.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.442+0000 7f074f400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0748161420 con 0x7f07480d2370 2024-03-29T10:14:45.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.442+0000 7f074f400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0748161700 con 0x7f074815d9d0 2024-03-29T10:14:45.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.442+0000 7f074f400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07480d4340 con 0x7f07480d16a0 2024-03-29T10:14:45.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.442+0000 7f074d400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f07480d2370 0x7f07480ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38398/0 (socket says 172.21.15.12:38398) 2024-03-29T10:14:45.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.442+0000 7f074d400700 1 -- 172.21.15.12:0/56065410 learned_addr learned my addr 172.21.15.12:0/56065410 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:45.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.442+0000 7f0747e00700 1 --2- 172.21.15.12:0/56065410 >> v2:172.21.15.159:3300/0 conn(0x7f07480d16a0 0x7f07480d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:45.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.442+0000 7f0747400700 1 -- 172.21.15.12:0/56065410 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049320697 0 0) 0x7f0748161420 con 0x7f07480d2370 2024-03-29T10:14:45.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.442+0000 7f0747400700 1 -- 172.21.15.12:0/56065410 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07300035b0 con 0x7f07480d2370 2024-03-29T10:14:45.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.443+0000 7f0747400700 1 -- 172.21.15.12:0/56065410 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722812402 0 0) 0x7f0748161700 con 0x7f074815d9d0 2024-03-29T10:14:45.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.443+0000 7f0747400700 1 -- 172.21.15.12:0/56065410 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0748161420 con 0x7f074815d9d0 2024-03-29T10:14:45.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.443+0000 7f0747e00700 1 -- 172.21.15.12:0/56065410 >> v1:172.21.15.159:6789/0 conn(0x7f074815d9d0 legacy=0x7f07480d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:45.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.443+0000 7f0747e00700 1 -- 172.21.15.12:0/56065410 >> v1:172.21.15.12:6789/0 conn(0x7f07480d2370 legacy=0x7f07480ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:45.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.443+0000 7f0747e00700 1 -- 172.21.15.12:0/56065410 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07481619e0 con 0x7f07480d16a0 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.443+0000 7f0747e00700 1 --2- 172.21.15.12:0/56065410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07480d16a0 0x7f07480d1e30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0734009820 tx=0x7f0734009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1184fe3d1e8b0841 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.443+0000 7f0747400700 1 -- 172.21.15.12:0/56065410 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734005e30 con 0x7f07480d16a0 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.443+0000 7f0747400700 1 -- 172.21.15.12:0/56065410 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0734006030 con 0x7f07480d16a0 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.443+0000 7f0747400700 1 -- 172.21.15.12:0/56065410 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07340063c0 con 0x7f07480d16a0 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.444+0000 7f074f400700 1 -- 172.21.15.12:0/56065410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07480d16a0 msgr2=0x7f07480d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.444+0000 7f074f400700 1 --2- 172.21.15.12:0/56065410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07480d16a0 0x7f07480d1e30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0734009820 tx=0x7f0734009b30 comp rx=0 tx=0).stop 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.444+0000 7f074f400700 1 -- 172.21.15.12:0/56065410 shutdown_connections 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.444+0000 7f074f400700 1 --2- 172.21.15.12:0/56065410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07480d16a0 0x7f07480d1e30 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.444+0000 7f074f400700 1 -- 172.21.15.12:0/56065410 >> 172.21.15.12:0/56065410 conn(0x7f074808aff0 msgr2=0x7f07480d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.444+0000 7f074f400700 1 -- 172.21.15.12:0/56065410 shutdown_connections 2024-03-29T10:14:45.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.444+0000 7f074f400700 1 -- 172.21.15.12:0/56065410 wait complete. 2024-03-29T10:14:45.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.446+0000 7f074f400700 1 Processor -- start 2024-03-29T10:14:45.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.446+0000 7f074f400700 1 -- start start 2024-03-29T10:14:45.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f074f400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07480d16a0 0x7f074815c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:45.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f074f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07480d2370 0x7f074815b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:45.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f074f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f074815d9d0 0x7f07480b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:45.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f074f400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07481611a0 con 0x7f07480d2370 2024-03-29T10:14:45.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f074f400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074815bed0 con 0x7f07480d16a0 2024-03-29T10:14:45.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f074f400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07480ca790 con 0x7f074815d9d0 2024-03-29T10:14:45.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f0747e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07480d2370 0x7f074815b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:45.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f0747e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07480d2370 0x7f074815b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48552/0 (socket says 172.21.15.12:48552) 2024-03-29T10:14:45.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f0747e00700 1 -- 172.21.15.12:0/2929548767 learned_addr learned my addr 172.21.15.12:0/2929548767 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:45.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f074ca00700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07480d16a0 0x7f074815c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:45.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f074d400700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f074815d9d0 0x7f07480b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:45.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f0747e00700 1 -- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f074815d9d0 msgr2=0x7f07480b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:45.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f0747e00700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f074815d9d0 0x7f07480b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f0747e00700 1 -- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07480d16a0 msgr2=0x7f074815c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:45.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f0747e00700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07480d16a0 0x7f074815c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.447+0000 7f0747e00700 1 -- 172.21.15.12:0/2929548767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0734009430 con 0x7f07480d2370 2024-03-29T10:14:45.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.448+0000 7f0747e00700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07480d2370 0x7f074815b990 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f0734013040 tx=0x7f0734004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:45.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.448+0000 7f0744c00700 1 -- 172.21.15.12:0/2929548767 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734006870 con 0x7f07480d2370 2024-03-29T10:14:45.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.448+0000 7f0744c00700 1 -- 172.21.15.12:0/2929548767 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0734020070 con 0x7f07480d2370 2024-03-29T10:14:45.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.448+0000 7f074f400700 1 -- 172.21.15.12:0/2929548767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07480caa70 con 0x7f07480d2370 2024-03-29T10:14:45.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.448+0000 7f0744c00700 1 -- 172.21.15.12:0/2929548767 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734025670 con 0x7f07480d2370 2024-03-29T10:14:45.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.448+0000 7f074f400700 1 -- 172.21.15.12:0/2929548767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07480cb030 con 0x7f07480d2370 2024-03-29T10:14:45.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.449+0000 7f074f400700 1 -- 172.21.15.12:0/2929548767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07480d1ca0 con 0x7f07480d2370 2024-03-29T10:14:45.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.450+0000 7f0744c00700 1 -- 172.21.15.12:0/2929548767 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 72) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f0734022030 con 0x7f07480d2370 2024-03-29T10:14:45.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.451+0000 7f0744c00700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f072806ae00 0x7f072806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:45.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.452+0000 7f0744c00700 1 -- 172.21.15.12:0/2929548767 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f073408aae0 con 0x7f07480d2370 2024-03-29T10:14:45.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.452+0000 7f074ca00700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f072806ae00 0x7f072806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:45.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.452+0000 7f074ca00700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f072806ae00 0x7f072806d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f073c0011e0 tx=0x7f073c009040 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:45.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.456+0000 7f0744c00700 1 -- 172.21.15.12:0/2929548767 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0734055a40 con 0x7f07480d2370 2024-03-29T10:14:45.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.694+0000 7f074f400700 1 -- 172.21.15.12:0/2929548767 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7f0748161fa0 con 0x7f072806ae00 2024-03-29T10:14:45.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.772+0000 7f0744c00700 1 -- 172.21.15.12:0/2929548767 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f073c0014d0 con 0x7f072806ae00 2024-03-29T10:14:45.770 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1/weekly 2024-03-29T10:14:45.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.774+0000 7f074f400700 1 -- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f072806ae00 msgr2=0x7f072806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:45.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.774+0000 7f074f400700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f072806ae00 0x7f072806d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f073c0011e0 tx=0x7f073c009040 comp rx=0 tx=0).stop 2024-03-29T10:14:45.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.774+0000 7f074f400700 1 -- 172.21.15.12:0/2929548767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07480d2370 msgr2=0x7f074815b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:45.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.774+0000 7f074f400700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07480d2370 0x7f074815b990 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f0734013040 tx=0x7f0734004750 comp rx=0 tx=0).stop 2024-03-29T10:14:45.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.775+0000 7f074f400700 1 -- 172.21.15.12:0/2929548767 shutdown_connections 2024-03-29T10:14:45.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.775+0000 7f074f400700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f072806ae00 0x7f072806d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.775+0000 7f074f400700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f074815d9d0 0x7f07480b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.775+0000 7f074f400700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07480d16a0 0x7f074815c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.775+0000 7f074f400700 1 --2- 172.21.15.12:0/2929548767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07480d2370 0x7f074815b990 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:45.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.775+0000 7f074f400700 1 -- 172.21.15.12:0/2929548767 >> 172.21.15.12:0/2929548767 conn(0x7f074808aff0 msgr2=0x7f07480b5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:45.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.775+0000 7f074f400700 1 -- 172.21.15.12:0/2929548767 shutdown_connections 2024-03-29T10:14:45.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:45.775+0000 7f074f400700 1 -- 172.21.15.12:0/2929548767 wait complete. 2024-03-29T10:14:45.791 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2024-03-29T10:14:45.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:45.792 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2024-03-29T10:14:45.838 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2024-03-29T10:14:46.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.041+0000 7fc55d200700 1 Processor -- start 2024-03-29T10:14:46.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.041+0000 7fc55d200700 1 -- start start 2024-03-29T10:14:46.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.042+0000 7fc55d200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc5580d16a0 0x7fc5580d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:46.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.042+0000 7fc55d200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc558161420 con 0x7fc5580d2370 2024-03-29T10:14:46.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.042+0000 7fc55d200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc558161700 con 0x7fc55815d9d0 2024-03-29T10:14:46.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.042+0000 7fc55d200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5580d4340 con 0x7fc5580d16a0 2024-03-29T10:14:46.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.042+0000 7fc555400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc5580d16a0 0x7fc5580d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:46.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.042+0000 7fc555400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc5580d16a0 0x7fc5580d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48562/0 (socket says 172.21.15.12:48562) 2024-03-29T10:14:46.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.042+0000 7fc556800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc5580d2370 0x7fc5580ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38400/0 (socket says 172.21.15.12:38400) 2024-03-29T10:14:46.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.042+0000 7fc555400700 1 -- 172.21.15.12:0/3440569536 learned_addr learned my addr 172.21.15.12:0/3440569536 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:46.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.043+0000 7fc555400700 1 -- 172.21.15.12:0/3440569536 >> v1:172.21.15.159:6789/0 conn(0x7fc55815d9d0 legacy=0x7fc5580d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.043+0000 7fc555400700 1 -- 172.21.15.12:0/3440569536 >> v1:172.21.15.12:6789/0 conn(0x7fc5580d2370 legacy=0x7fc5580ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.043+0000 7fc555400700 1 -- 172.21.15.12:0/3440569536 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5581619e0 con 0x7fc5580d16a0 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.043+0000 7fc554a00700 1 -- 172.21.15.12:0/3440569536 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267551975 0 0) 0x7fc558161420 con 0x7fc5580d2370 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.043+0000 7fc555400700 1 --2- 172.21.15.12:0/3440569536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 0x7fc5580d1e30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fc544006fd0 tx=0x7fc544009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ddc1abca356b4a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.043+0000 7fc554a00700 1 -- 172.21.15.12:0/3440569536 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544005e30 con 0x7fc5580d16a0 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.043+0000 7fc554a00700 1 -- 172.21.15.12:0/3440569536 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc544004030 con 0x7fc5580d16a0 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.043+0000 7fc554a00700 1 -- 172.21.15.12:0/3440569536 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5440043e0 con 0x7fc5580d16a0 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.044+0000 7fc55d200700 1 -- 172.21.15.12:0/3440569536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 msgr2=0x7fc5580d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.044+0000 7fc55d200700 1 --2- 172.21.15.12:0/3440569536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 0x7fc5580d1e30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fc544006fd0 tx=0x7fc544009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.044+0000 7fc55d200700 1 -- 172.21.15.12:0/3440569536 shutdown_connections 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.044+0000 7fc55d200700 1 --2- 172.21.15.12:0/3440569536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 0x7fc5580d1e30 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.044+0000 7fc55d200700 1 -- 172.21.15.12:0/3440569536 >> 172.21.15.12:0/3440569536 conn(0x7fc55808aff0 msgr2=0x7fc5580d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.044+0000 7fc55d200700 1 -- 172.21.15.12:0/3440569536 shutdown_connections 2024-03-29T10:14:46.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.044+0000 7fc55d200700 1 -- 172.21.15.12:0/3440569536 wait complete. 2024-03-29T10:14:46.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.046+0000 7fc55d200700 1 Processor -- start 2024-03-29T10:14:46.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.046+0000 7fc55d200700 1 -- start start 2024-03-29T10:14:46.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc55d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 0x7fc5580c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:46.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc55d200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580d2370 0x7fc5581550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:46.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc55d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc55815d9d0 0x7fc5580b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:46.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc55d200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc558155b40 con 0x7fc5580d16a0 2024-03-29T10:14:46.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc55d200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc558155630 con 0x7fc5580d2370 2024-03-29T10:14:46.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc55d200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55815e370 con 0x7fc55815d9d0 2024-03-29T10:14:46.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc555e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 0x7fc5580c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:46.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc555e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 0x7fc5580c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48576/0 (socket says 172.21.15.12:48576) 2024-03-29T10:14:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc555e00700 1 -- 172.21.15.12:0/3257475009 learned_addr learned my addr 172.21.15.12:0/3257475009 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc556800700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc55815d9d0 0x7fc5580b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc555400700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580d2370 0x7fc5581550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc555e00700 1 -- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc55815d9d0 msgr2=0x7fc5580b4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc555e00700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc55815d9d0 0x7fc5580b4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc555e00700 1 -- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580d2370 msgr2=0x7fc5581550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc555e00700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580d2370 0x7fc5581550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc555e00700 1 -- 172.21.15.12:0/3257475009 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc544009500 con 0x7fc5580d16a0 2024-03-29T10:14:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.047+0000 7fc555400700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580d2370 0x7fc5581550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.048+0000 7fc555e00700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 0x7fc5580c9a60 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fc54800c450 tx=0x7fc54800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.048+0000 7fc54e000700 1 -- 172.21.15.12:0/3257475009 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5480052d0 con 0x7fc5580d16a0 2024-03-29T10:14:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.048+0000 7fc54e000700 1 -- 172.21.15.12:0/3257475009 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc548005430 con 0x7fc5580d16a0 2024-03-29T10:14:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.048+0000 7fc55d200700 1 -- 172.21.15.12:0/3257475009 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55815e650 con 0x7fc5580d16a0 2024-03-29T10:14:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.048+0000 7fc54e000700 1 -- 172.21.15.12:0/3257475009 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5480038c0 con 0x7fc5580d16a0 2024-03-29T10:14:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.048+0000 7fc55d200700 1 -- 172.21.15.12:0/3257475009 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55815ec10 con 0x7fc5580d16a0 2024-03-29T10:14:46.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.049+0000 7fc55d200700 1 -- 172.21.15.12:0/3257475009 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5580d1ca0 con 0x7fc5580d16a0 2024-03-29T10:14:46.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.050+0000 7fc54e000700 1 -- 172.21.15.12:0/3257475009 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 72) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fc548003a20 con 0x7fc5580d16a0 2024-03-29T10:14:46.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.051+0000 7fc54e000700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc53806ae00 0x7fc53806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:46.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.051+0000 7fc54e000700 1 -- 172.21.15.12:0/3257475009 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc548088a40 con 0x7fc5580d16a0 2024-03-29T10:14:46.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.052+0000 7fc555400700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc53806ae00 0x7fc53806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:46.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.052+0000 7fc555400700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc53806ae00 0x7fc53806d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc5580d1ae0 tx=0x7fc544008070 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:46.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.055+0000 7fc54e000700 1 -- 172.21.15.12:0/3257475009 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc548053910 con 0x7fc5580d16a0 2024-03-29T10:14:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.295+0000 7fc55d200700 1 -- 172.21.15.12:0/3257475009 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fc558161fa0 con 0x7fc53806ae00 2024-03-29T10:14:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.367+0000 7fc54e000700 1 -- 172.21.15.12:0/3257475009 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fc5440037a0 con 0x7fc53806ae00 2024-03-29T10:14:46.365 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1/monthly 2024-03-29T10:14:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 -- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc53806ae00 msgr2=0x7fc53806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc53806ae00 0x7fc53806d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc5580d1ae0 tx=0x7fc544008070 comp rx=0 tx=0).stop 2024-03-29T10:14:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 -- 172.21.15.12:0/3257475009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 msgr2=0x7fc5580c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 0x7fc5580c9a60 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fc54800c450 tx=0x7fc54800c760 comp rx=0 tx=0).stop 2024-03-29T10:14:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 -- 172.21.15.12:0/3257475009 shutdown_connections 2024-03-29T10:14:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc53806ae00 0x7fc53806d2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc55815d9d0 0x7fc5580b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580d16a0 0x7fc5580c9a60 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 --2- 172.21.15.12:0/3257475009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580d2370 0x7fc5581550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 -- 172.21.15.12:0/3257475009 >> 172.21.15.12:0/3257475009 conn(0x7fc55808aff0 msgr2=0x7fc5580d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 -- 172.21.15.12:0/3257475009 shutdown_connections 2024-03-29T10:14:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.370+0000 7fc55d200700 1 -- 172.21.15.12:0/3257475009 wait complete. 2024-03-29T10:14:46.386 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2024-03-29T10:14:46.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:46.386 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2024-03-29T10:14:46.433 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2024-03-29T10:14:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.635+0000 7f397fa00700 1 Processor -- start 2024-03-29T10:14:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.635+0000 7f397fa00700 1 -- start start 2024-03-29T10:14:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.636+0000 7f397fa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39780d3eb0 0x7f39780b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.636+0000 7f397fa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39780ced20 con 0x7f397815d8a0 2024-03-29T10:14:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.636+0000 7f397fa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39780d4340 con 0x7f39780cf1b0 2024-03-29T10:14:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.636+0000 7f397fa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39780ce6d0 con 0x7f39780d3eb0 2024-03-29T10:14:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.636+0000 7f3977e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39780d3eb0 0x7f39780b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.636+0000 7f397d000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f397815d8a0 0x7f397806b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38408/0 (socket says 172.21.15.12:38408) 2024-03-29T10:14:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.636+0000 7f397d000700 1 -- 172.21.15.12:0/3335051834 learned_addr learned my addr 172.21.15.12:0/3335051834 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.637+0000 7f3977e00700 1 -- 172.21.15.12:0/3335051834 >> v1:172.21.15.159:6789/0 conn(0x7f39780cf1b0 legacy=0x7f39780d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.637+0000 7f3977e00700 1 -- 172.21.15.12:0/3335051834 >> v1:172.21.15.12:6789/0 conn(0x7f397815d8a0 legacy=0x7f397806b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.637+0000 7f3977e00700 1 -- 172.21.15.12:0/3335051834 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39780ce9b0 con 0x7f39780d3eb0 2024-03-29T10:14:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.637+0000 7f3977400700 1 -- 172.21.15.12:0/3335051834 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209762151 0 0) 0x7f39780ced20 con 0x7f397815d8a0 2024-03-29T10:14:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.637+0000 7f3977e00700 1 --2- 172.21.15.12:0/3335051834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39780d3eb0 0x7f39780b2990 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3964009830 tx=0x7f3964009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d308a6c3edd9e99d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.637+0000 7f3977400700 1 -- 172.21.15.12:0/3335051834 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964004230 con 0x7f39780d3eb0 2024-03-29T10:14:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.637+0000 7f3977400700 1 -- 172.21.15.12:0/3335051834 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3964004390 con 0x7f39780d3eb0 2024-03-29T10:14:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.637+0000 7f3977400700 1 -- 172.21.15.12:0/3335051834 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39640036a0 con 0x7f39780d3eb0 2024-03-29T10:14:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.638+0000 7f397fa00700 1 -- 172.21.15.12:0/3335051834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39780d3eb0 msgr2=0x7f39780b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.638+0000 7f397fa00700 1 --2- 172.21.15.12:0/3335051834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39780d3eb0 0x7f39780b2990 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3964009830 tx=0x7f3964009b40 comp rx=0 tx=0).stop 2024-03-29T10:14:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.638+0000 7f397fa00700 1 -- 172.21.15.12:0/3335051834 shutdown_connections 2024-03-29T10:14:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.638+0000 7f397fa00700 1 --2- 172.21.15.12:0/3335051834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39780d3eb0 0x7f39780b2990 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.638+0000 7f397fa00700 1 -- 172.21.15.12:0/3335051834 >> 172.21.15.12:0/3335051834 conn(0x7f397808aff0 msgr2=0x7f39780d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.638+0000 7f397fa00700 1 -- 172.21.15.12:0/3335051834 shutdown_connections 2024-03-29T10:14:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.638+0000 7f397fa00700 1 -- 172.21.15.12:0/3335051834 wait complete. 2024-03-29T10:14:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.640+0000 7f397fa00700 1 Processor -- start 2024-03-29T10:14:46.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.640+0000 7f397fa00700 1 -- start start 2024-03-29T10:14:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.641+0000 7f397fa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39780cf1b0 0x7f39780ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.641+0000 7f397fa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397815d8a0 0x7f3978154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.641+0000 7f397fa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3978154f10 0x7f39780cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.641+0000 7f397fa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39780cac40 con 0x7f397815d8a0 2024-03-29T10:14:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.641+0000 7f397fa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39780caf20 con 0x7f3978154f10 2024-03-29T10:14:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.641+0000 7f397fa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39780cb200 con 0x7f39780cf1b0 2024-03-29T10:14:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.641+0000 7f3977e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397815d8a0 0x7f3978154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.641+0000 7f3977e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397815d8a0 0x7f3978154710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48606/0 (socket says 172.21.15.12:48606) 2024-03-29T10:14:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.641+0000 7f3977e00700 1 -- 172.21.15.12:0/1168843540 learned_addr learned my addr 172.21.15.12:0/1168843540 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f397da00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3978154f10 0x7f39780cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f397d000700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39780cf1b0 0x7f39780ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f3977e00700 1 -- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39780cf1b0 msgr2=0x7f39780ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f3977e00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39780cf1b0 0x7f39780ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f3977e00700 1 -- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3978154f10 msgr2=0x7f39780cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f3977e00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3978154f10 0x7f39780cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f3977e00700 1 -- 172.21.15.12:0/1168843540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3964009440 con 0x7f397815d8a0 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f397da00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3978154f10 0x7f39780cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f3977e00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397815d8a0 0x7f3978154710 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f3964000c00 tx=0x7f3964004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f3974c00700 1 -- 172.21.15.12:0/1168843540 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964011630 con 0x7f397815d8a0 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f3974c00700 1 -- 172.21.15.12:0/1168843540 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396401e070 con 0x7f397815d8a0 2024-03-29T10:14:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f397fa00700 1 -- 172.21.15.12:0/1168843540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39780cb4e0 con 0x7f397815d8a0 2024-03-29T10:14:46.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.642+0000 7f3974c00700 1 -- 172.21.15.12:0/1168843540 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39640119f0 con 0x7f397815d8a0 2024-03-29T10:14:46.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.643+0000 7f397fa00700 1 -- 172.21.15.12:0/1168843540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3978154c50 con 0x7f397815d8a0 2024-03-29T10:14:46.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.644+0000 7f397fa00700 1 -- 172.21.15.12:0/1168843540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3978067720 con 0x7f397815d8a0 2024-03-29T10:14:46.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.644+0000 7f3974c00700 1 -- 172.21.15.12:0/1168843540 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 72) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f3964011b50 con 0x7f397815d8a0 2024-03-29T10:14:46.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.645+0000 7f3974c00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f395806ae30 0x7f395806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:46.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.645+0000 7f3974c00700 1 -- 172.21.15.12:0/1168843540 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f396408a990 con 0x7f397815d8a0 2024-03-29T10:14:46.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.646+0000 7f397d000700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f395806ae30 0x7f395806d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:46.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.647+0000 7f397d000700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f395806ae30 0x7f395806d2f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f396c000f60 tx=0x7f396c007c90 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:46.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.650+0000 7f3974c00700 1 -- 172.21.15.12:0/1168843540 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39640558f0 con 0x7f397815d8a0 2024-03-29T10:14:46.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.893+0000 7f397fa00700 1 -- 172.21.15.12:0/1168843540 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7f39780ce380 con 0x7f395806ae30 2024-03-29T10:14:46.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.899+0000 7f3974c00700 1 -- 172.21.15.12:0/1168843540 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 73) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f3964055cc0 con 0x7f397815d8a0 2024-03-29T10:14:46.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.977+0000 7f3974c00700 1 -- 172.21.15.12:0/1168843540 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f396c013420 con 0x7f395806ae30 2024-03-29T10:14:46.976 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1/yearly 2024-03-29T10:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 -- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f395806ae30 msgr2=0x7f395806d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f395806ae30 0x7f395806d2f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f396c000f60 tx=0x7f396c007c90 comp rx=0 tx=0).stop 2024-03-29T10:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 -- 172.21.15.12:0/1168843540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397815d8a0 msgr2=0x7f3978154710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397815d8a0 0x7f3978154710 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f3964000c00 tx=0x7f3964004750 comp rx=0 tx=0).stop 2024-03-29T10:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 -- 172.21.15.12:0/1168843540 shutdown_connections 2024-03-29T10:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f395806ae30 0x7f395806d2f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3978154f10 0x7f39780cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39780cf1b0 0x7f39780ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 --2- 172.21.15.12:0/1168843540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397815d8a0 0x7f3978154710 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 -- 172.21.15.12:0/1168843540 >> 172.21.15.12:0/1168843540 conn(0x7f397808aff0 msgr2=0x7f39780b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 -- 172.21.15.12:0/1168843540 shutdown_connections 2024-03-29T10:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:46.980+0000 7f397fa00700 1 -- 172.21.15.12:0/1168843540 wait complete. 2024-03-29T10:14:46.997 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2024-03-29T10:14:46.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:46.997 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2024-03-29T10:14:47.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2024-03-29T10:14:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.248+0000 7f0cfc000700 1 Processor -- start 2024-03-29T10:14:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.248+0000 7f0cfc000700 1 -- start start 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.249+0000 7f0cfc000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf415d9d0 0x7f0cf40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.249+0000 7f0cfc000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0cf40d2540 0x7f0cf40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.249+0000 7f0cfc000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf4161420 con 0x7f0cf40d16a0 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.249+0000 7f0cfc000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf415edc0 con 0x7f0cf40d2540 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.249+0000 7f0cfc000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4161700 con 0x7f0cf415d9d0 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.249+0000 7f0cfa000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0cf40d2540 0x7f0cf40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.249+0000 7f0cfa000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0cf40d2540 0x7f0cf40ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48616/0 (socket says 172.21.15.12:48616) 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.249+0000 7f0cfa000700 1 -- 172.21.15.12:0/2124067635 learned_addr learned my addr 172.21.15.12:0/2124067635 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.250+0000 7f0cf9600700 1 --2- 172.21.15.12:0/2124067635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf415d9d0 0x7f0cf40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.250+0000 7f0cfa000700 1 -- 172.21.15.12:0/2124067635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf415d9d0 msgr2=0x7f0cf40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.250+0000 7f0cfa000700 1 --2- 172.21.15.12:0/2124067635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf415d9d0 0x7f0cf40d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.250+0000 7f0cfa000700 1 -- 172.21.15.12:0/2124067635 >> v1:172.21.15.159:6789/0 conn(0x7f0cf40d16a0 legacy=0x7f0cf40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.250+0000 7f0cfa000700 1 -- 172.21.15.12:0/2124067635 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf41619e0 con 0x7f0cf40d2540 2024-03-29T10:14:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.250+0000 7f0cfa000700 1 --2- 172.21.15.12:0/2124067635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cf40d2540 0x7f0cf40ce360 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009e80 tx=0x7f0cec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adfa52ef62e92de server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.250+0000 7f0cf3e00700 1 -- 172.21.15.12:0/2124067635 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec00e070 con 0x7f0cf40d2540 2024-03-29T10:14:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.251+0000 7f0cf3e00700 1 -- 172.21.15.12:0/2124067635 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec00b8d0 con 0x7f0cf40d2540 2024-03-29T10:14:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.251+0000 7f0cf3e00700 1 -- 172.21.15.12:0/2124067635 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec00bc80 con 0x7f0cf40d2540 2024-03-29T10:14:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.251+0000 7f0cfc000700 1 -- 172.21.15.12:0/2124067635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cf40d2540 msgr2=0x7f0cf40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.251+0000 7f0cfc000700 1 --2- 172.21.15.12:0/2124067635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cf40d2540 0x7f0cf40ce360 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009e80 tx=0x7f0cec00b330 comp rx=0 tx=0).stop 2024-03-29T10:14:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.251+0000 7f0cfc000700 1 -- 172.21.15.12:0/2124067635 shutdown_connections 2024-03-29T10:14:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.251+0000 7f0cfc000700 1 --2- 172.21.15.12:0/2124067635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cf40d2540 0x7f0cf40ce360 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.251+0000 7f0cfc000700 1 --2- 172.21.15.12:0/2124067635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf415d9d0 0x7f0cf40d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.251+0000 7f0cfc000700 1 -- 172.21.15.12:0/2124067635 >> 172.21.15.12:0/2124067635 conn(0x7f0cf408aff0 msgr2=0x7f0cf40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.251+0000 7f0cfc000700 1 -- 172.21.15.12:0/2124067635 shutdown_connections 2024-03-29T10:14:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.251+0000 7f0cfc000700 1 -- 172.21.15.12:0/2124067635 wait complete. 2024-03-29T10:14:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.253+0000 7f0cfc000700 1 Processor -- start 2024-03-29T10:14:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.253+0000 7f0cfc000700 1 -- start start 2024-03-29T10:14:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cfc000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cf40d16a0 0x7f0cf4154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cfc000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf40d2540 0x7f0cf40b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cfc000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cf415d9d0 0x7f0cf40b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cfc000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf40b60c0 con 0x7f0cf415d9d0 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cfc000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf40b63a0 con 0x7f0cf40d16a0 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cfc000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf40b6680 con 0x7f0cf40d2540 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cfa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cf415d9d0 0x7f0cf40b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cf9600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cf40d16a0 0x7f0cf4154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cf9600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cf40d16a0 0x7f0cf4154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:54540/0 (socket says 172.21.15.12:54540) 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cfa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cf415d9d0 0x7f0cf40b8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48618/0 (socket says 172.21.15.12:48618) 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cf9600700 1 -- 172.21.15.12:0/2950937605 learned_addr learned my addr 172.21.15.12:0/2950937605 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.254+0000 7f0cf8c00700 1 --2- 172.21.15.12:0/2950937605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf40d2540 0x7f0cf40b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cfa000700 1 -- 172.21.15.12:0/2950937605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf40d2540 msgr2=0x7f0cf40b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cfa000700 1 --2- 172.21.15.12:0/2950937605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf40d2540 0x7f0cf40b83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cfa000700 1 -- 172.21.15.12:0/2950937605 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cf40d16a0 msgr2=0x7f0cf4154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cfa000700 1 --2- 172.21.15.12:0/2950937605 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cf40d16a0 0x7f0cf4154480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cfa000700 1 -- 172.21.15.12:0/2950937605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec009b30 con 0x7f0cf415d9d0 2024-03-29T10:14:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cfa000700 1 --2- 172.21.15.12:0/2950937605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cf415d9d0 0x7f0cf40b8930 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f0cec015000 tx=0x7f0cec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cf1600700 1 -- 172.21.15.12:0/2950937605 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec004800 con 0x7f0cf415d9d0 2024-03-29T10:14:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cf1600700 1 -- 172.21.15.12:0/2950937605 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec0092e0 con 0x7f0cf415d9d0 2024-03-29T10:14:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cfc000700 1 -- 172.21.15.12:0/2950937605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf40b6960 con 0x7f0cf415d9d0 2024-03-29T10:14:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cf1600700 1 -- 172.21.15.12:0/2950937605 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec01c680 con 0x7f0cf415d9d0 2024-03-29T10:14:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.255+0000 7f0cfc000700 1 -- 172.21.15.12:0/2950937605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf41549c0 con 0x7f0cf415d9d0 2024-03-29T10:14:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.257+0000 7f0cfc000700 1 -- 172.21.15.12:0/2950937605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf40d1ca0 con 0x7f0cf415d9d0 2024-03-29T10:14:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.258+0000 7f0cf1600700 1 -- 172.21.15.12:0/2950937605 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 73) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f0cec01c7e0 con 0x7f0cf415d9d0 2024-03-29T10:14:47.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.259+0000 7f0cf1600700 1 --2- 172.21.15.12:0/2950937605 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0cd406ae90 0x7f0cd406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.259+0000 7f0cf1600700 1 -- 172.21.15.12:0/2950937605 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0cec08b1e0 con 0x7f0cf415d9d0 2024-03-29T10:14:47.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.259+0000 7f0cf9600700 1 --2- 172.21.15.12:0/2950937605 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0cd406ae90 0x7f0cd406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.260+0000 7f0cf9600700 1 --2- 172.21.15.12:0/2950937605 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0cd406ae90 0x7f0cd406d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009c00 tx=0x7f0ce8009210 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.264+0000 7f0cf1600700 1 -- 172.21.15.12:0/2950937605 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0cec055ec0 con 0x7f0cf415d9d0 2024-03-29T10:14:47.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.504+0000 7f0cfc000700 1 -- 172.21.15.12:0/2950937605 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cf4161fa0 con 0x7f0cd406ae90 2024-03-29T10:14:47.503 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:47.505+0000 7f9db2400700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-03-29T10:14:47.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.506+0000 7f0cf1600700 1 -- 172.21.15.12:0/2950937605 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7f0ce8004030 con 0x7f0cd406ae90 2024-03-29T10:14:47.506 INFO:teuthology.orchestra.run.smithi012.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-03-29T10:14:47.514 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-29T10:14:47.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2024-03-29T10:14:47.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.717+0000 7fd280a00700 1 Processor -- start 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.717+0000 7fd280a00700 1 -- start start 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.718+0000 7fd280a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd27c15da10 0x7fd27c0d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd280a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd27c0ce910 0x7fd27c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd280a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27c0cd050 con 0x7fd27c06a790 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd280a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd27c15ddf0 con 0x7fd27c0ce910 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd280a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c0cf2d0 con 0x7fd27c15da10 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd273e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd27c0ce910 0x7fd27c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd273e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd27c0ce910 0x7fd27c0ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48620/0 (socket says 172.21.15.12:48620) 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd27a800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd27c06a790 0x7fd27c06abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38410/0 (socket says 172.21.15.12:38410) 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd273e00700 1 -- 172.21.15.12:0/4091950151 learned_addr learned my addr 172.21.15.12:0/4091950151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd279e00700 1 --2- 172.21.15.12:0/4091950151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd27c15da10 0x7fd27c0d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd273e00700 1 -- 172.21.15.12:0/4091950151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd27c15da10 msgr2=0x7fd27c0d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd273e00700 1 --2- 172.21.15.12:0/4091950151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd27c15da10 0x7fd27c0d09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:47.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd273e00700 1 -- 172.21.15.12:0/4091950151 >> v1:172.21.15.12:6789/0 conn(0x7fd27c06a790 legacy=0x7fd27c06abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd273e00700 1 -- 172.21.15.12:0/4091950151 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c0cd330 con 0x7fd27c0ce910 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.719+0000 7fd279e00700 1 --2- 172.21.15.12:0/4091950151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd27c15da10 0x7fd27c0d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.720+0000 7fd273e00700 1 --2- 172.21.15.12:0/4091950151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd27c0ce910 0x7fd27c0ced90 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fd2640098f0 tx=0x7fd264009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c97160b741dd9d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.720+0000 7fd279400700 1 -- 172.21.15.12:0/4091950151 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264004230 con 0x7fd27c0ce910 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.720+0000 7fd279400700 1 -- 172.21.15.12:0/4091950151 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd264004390 con 0x7fd27c0ce910 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.720+0000 7fd279400700 1 -- 172.21.15.12:0/4091950151 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2640036a0 con 0x7fd27c0ce910 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.720+0000 7fd280a00700 1 -- 172.21.15.12:0/4091950151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd27c0ce910 msgr2=0x7fd27c0ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.721+0000 7fd280a00700 1 --2- 172.21.15.12:0/4091950151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd27c0ce910 0x7fd27c0ced90 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fd2640098f0 tx=0x7fd264009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.721+0000 7fd280a00700 1 -- 172.21.15.12:0/4091950151 shutdown_connections 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.721+0000 7fd280a00700 1 --2- 172.21.15.12:0/4091950151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd27c15da10 0x7fd27c0d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.721+0000 7fd280a00700 1 --2- 172.21.15.12:0/4091950151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd27c0ce910 0x7fd27c0ced90 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.721+0000 7fd280a00700 1 -- 172.21.15.12:0/4091950151 >> 172.21.15.12:0/4091950151 conn(0x7fd27c0b5820 msgr2=0x7fd27c0b5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.721+0000 7fd280a00700 1 -- 172.21.15.12:0/4091950151 shutdown_connections 2024-03-29T10:14:47.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.721+0000 7fd280a00700 1 -- 172.21.15.12:0/4091950151 wait complete. 2024-03-29T10:14:47.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.723+0000 7fd280a00700 1 Processor -- start 2024-03-29T10:14:47.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.723+0000 7fd280a00700 1 -- start start 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.724+0000 7fd280a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd27c06a790 0x7fd27c156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd280a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd27c0ce910 0x7fd27c15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd280a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd27c15da10 0x7fd27c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd280a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c0c9720 con 0x7fd27c0ce910 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd280a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c15b550 con 0x7fd27c06a790 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd280a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c0c99c0 con 0x7fd27c15da10 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd273e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd27c0ce910 0x7fd27c15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd273e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd27c0ce910 0x7fd27c15ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48626/0 (socket says 172.21.15.12:48626) 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd273e00700 1 -- 172.21.15.12:0/2496846412 learned_addr learned my addr 172.21.15.12:0/2496846412 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd27a800700 1 --2- 172.21.15.12:0/2496846412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd27c15da10 0x7fd27c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd279e00700 1 --2- 172.21.15.12:0/2496846412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd27c06a790 0x7fd27c156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd273e00700 1 -- 172.21.15.12:0/2496846412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd27c15da10 msgr2=0x7fd27c06e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd273e00700 1 --2- 172.21.15.12:0/2496846412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd27c15da10 0x7fd27c06e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd273e00700 1 -- 172.21.15.12:0/2496846412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd27c06a790 msgr2=0x7fd27c156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd273e00700 1 --2- 172.21.15.12:0/2496846412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd27c06a790 0x7fd27c156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.725+0000 7fd273e00700 1 -- 172.21.15.12:0/2496846412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c0094a0 con 0x7fd27c0ce910 2024-03-29T10:14:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.726+0000 7fd273e00700 1 --2- 172.21.15.12:0/2496846412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd27c0ce910 0x7fd27c15ac40 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fd264009130 tx=0x7fd264004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.726+0000 7fd272000700 1 -- 172.21.15.12:0/2496846412 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2640045d0 con 0x7fd27c0ce910 2024-03-29T10:14:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.726+0000 7fd272000700 1 -- 172.21.15.12:0/2496846412 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd264011630 con 0x7fd27c0ce910 2024-03-29T10:14:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.726+0000 7fd272000700 1 -- 172.21.15.12:0/2496846412 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2640119e0 con 0x7fd27c0ce910 2024-03-29T10:14:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.726+0000 7fd280a00700 1 -- 172.21.15.12:0/2496846412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd264009550 con 0x7fd27c0ce910 2024-03-29T10:14:47.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.726+0000 7fd280a00700 1 -- 172.21.15.12:0/2496846412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27c15b830 con 0x7fd27c0ce910 2024-03-29T10:14:47.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.727+0000 7fd280a00700 1 -- 172.21.15.12:0/2496846412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27c0c9ee0 con 0x7fd27c0ce910 2024-03-29T10:14:47.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.728+0000 7fd272000700 1 -- 172.21.15.12:0/2496846412 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 73) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fd264022070 con 0x7fd27c0ce910 2024-03-29T10:14:47.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.729+0000 7fd272000700 1 --2- 172.21.15.12:0/2496846412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd25806aec0 0x7fd25806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:47.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.729+0000 7fd272000700 1 -- 172.21.15.12:0/2496846412 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fd26408b1f0 con 0x7fd27c0ce910 2024-03-29T10:14:47.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.729+0000 7fd279e00700 1 --2- 172.21.15.12:0/2496846412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd25806aec0 0x7fd25806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:47.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.730+0000 7fd279e00700 1 --2- 172.21.15.12:0/2496846412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd25806aec0 0x7fd25806d380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009d70 tx=0x7fd26c009210 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:47.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.733+0000 7fd272000700 1 -- 172.21.15.12:0/2496846412 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd27c0c9ee0 con 0x7fd27c0ce910 2024-03-29T10:14:47.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.980+0000 7fd280a00700 1 -- 172.21.15.12:0/2496846412 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7fd27c0cdbd0 con 0x7fd25806aec0 2024-03-29T10:14:47.979 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:47.981+0000 7f9db2400700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-29T10:14:47.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:47.982+0000 7fd272000700 1 -- 172.21.15.12:0/2496846412 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd26c0099c0 con 0x7fd25806aec0 2024-03-29T10:14:47.982 INFO:teuthology.orchestra.run.smithi012.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-29T10:14:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-29T10:14:47.990 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2024-03-29T10:14:48.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.194+0000 7fd5b9000700 1 Processor -- start 2024-03-29T10:14:48.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.194+0000 7fd5b9000700 1 -- start start 2024-03-29T10:14:48.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.195+0000 7fd5b9000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5b415d800 0x7fd5b40725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.195+0000 7fd5b9000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd5b40ce230 0x7fd5b40ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b9000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b40709c0 con 0x7fd5b40b2070 2024-03-29T10:14:48.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b9000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b40b40f0 con 0x7fd5b415d800 2024-03-29T10:14:48.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b9000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b4070c80 con 0x7fd5b40ce230 2024-03-29T10:14:48.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5b415d800 0x7fd5b40725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5b415d800 0x7fd5b40725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48638/0 (socket says 172.21.15.12:48638) 2024-03-29T10:14:48.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b1e00700 1 -- 172.21.15.12:0/2722311076 learned_addr learned my addr 172.21.15.12:0/2722311076 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b1400700 1 --2- 172.21.15.12:0/2722311076 >> v2:172.21.15.159:3300/0 conn(0x7fd5b40ce230 0x7fd5b40ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b1e00700 1 -- 172.21.15.12:0/2722311076 >> v2:172.21.15.159:3300/0 conn(0x7fd5b40ce230 msgr2=0x7fd5b40ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b1e00700 1 --2- 172.21.15.12:0/2722311076 >> v2:172.21.15.159:3300/0 conn(0x7fd5b40ce230 0x7fd5b40ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b1e00700 1 -- 172.21.15.12:0/2722311076 >> v1:172.21.15.159:6789/0 conn(0x7fd5b40b2070 legacy=0x7fd5b415dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.196+0000 7fd5b1e00700 1 -- 172.21.15.12:0/2722311076 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4070f60 con 0x7fd5b415d800 2024-03-29T10:14:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.197+0000 7fd5b1e00700 1 --2- 172.21.15.12:0/2722311076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b415d800 0x7fd5b40725c0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40cce70 tx=0x7fd5a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20427da7d985ad4e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.197+0000 7fd5b0a00700 1 -- 172.21.15.12:0/2722311076 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a400b870 con 0x7fd5b415d800 2024-03-29T10:14:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.197+0000 7fd5b0a00700 1 -- 172.21.15.12:0/2722311076 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a400b9d0 con 0x7fd5b415d800 2024-03-29T10:14:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.197+0000 7fd5b0a00700 1 -- 172.21.15.12:0/2722311076 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a400bd60 con 0x7fd5b415d800 2024-03-29T10:14:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.197+0000 7fd5b9000700 1 -- 172.21.15.12:0/2722311076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b415d800 msgr2=0x7fd5b40725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.197+0000 7fd5b9000700 1 --2- 172.21.15.12:0/2722311076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b415d800 0x7fd5b40725c0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40cce70 tx=0x7fd5a4009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.198+0000 7fd5b9000700 1 -- 172.21.15.12:0/2722311076 shutdown_connections 2024-03-29T10:14:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.198+0000 7fd5b9000700 1 --2- 172.21.15.12:0/2722311076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b415d800 0x7fd5b40725c0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.198+0000 7fd5b9000700 1 --2- 172.21.15.12:0/2722311076 >> v2:172.21.15.159:3300/0 conn(0x7fd5b40ce230 0x7fd5b40ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.198+0000 7fd5b9000700 1 -- 172.21.15.12:0/2722311076 >> 172.21.15.12:0/2722311076 conn(0x7fd5b40b0d70 msgr2=0x7fd5b40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.198+0000 7fd5b9000700 1 -- 172.21.15.12:0/2722311076 shutdown_connections 2024-03-29T10:14:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.198+0000 7fd5b9000700 1 -- 172.21.15.12:0/2722311076 wait complete. 2024-03-29T10:14:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.199+0000 7fd5b9000700 1 Processor -- start 2024-03-29T10:14:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.200+0000 7fd5b9000700 1 -- start start 2024-03-29T10:14:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b9000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b40b2070 0x7fd5b406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b40ce230 0x7fd5b4162d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b9000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5b415d800 0x7fd5b4163260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b9000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b41643c0 con 0x7fd5b40ce230 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b9000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4164680 con 0x7fd5b415d800 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b9000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b41565d0 con 0x7fd5b40b2070 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b40ce230 0x7fd5b4162d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b40ce230 0x7fd5b4162d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48652/0 (socket says 172.21.15.12:48652) 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b1400700 1 -- 172.21.15.12:0/397020224 learned_addr learned my addr 172.21.15.12:0/397020224 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b1e00700 1 --2- 172.21.15.12:0/397020224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b40b2070 0x7fd5b406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b2800700 1 --2- 172.21.15.12:0/397020224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5b415d800 0x7fd5b4163260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b1400700 1 -- 172.21.15.12:0/397020224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b40b2070 msgr2=0x7fd5b406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b1400700 1 --2- 172.21.15.12:0/397020224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b40b2070 0x7fd5b406e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b1400700 1 -- 172.21.15.12:0/397020224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5b415d800 msgr2=0x7fd5b4163260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b1400700 1 --2- 172.21.15.12:0/397020224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5b415d800 0x7fd5b4163260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.201+0000 7fd5b1400700 1 -- 172.21.15.12:0/397020224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a4009550 con 0x7fd5b40ce230 2024-03-29T10:14:48.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.202+0000 7fd5b1400700 1 --2- 172.21.15.12:0/397020224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b40ce230 0x7fd5b4162d20 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000e810 tx=0x7fd5a000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:48.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.202+0000 7fd5aa000700 1 -- 172.21.15.12:0/397020224 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000c070 con 0x7fd5b40ce230 2024-03-29T10:14:48.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.202+0000 7fd5aa000700 1 -- 172.21.15.12:0/397020224 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a0013610 con 0x7fd5b40ce230 2024-03-29T10:14:48.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.202+0000 7fd5aa000700 1 -- 172.21.15.12:0/397020224 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a00139c0 con 0x7fd5b40ce230 2024-03-29T10:14:48.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.202+0000 7fd5b9000700 1 -- 172.21.15.12:0/397020224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b41568b0 con 0x7fd5b40ce230 2024-03-29T10:14:48.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.202+0000 7fd5b9000700 1 -- 172.21.15.12:0/397020224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b4156280 con 0x7fd5b40ce230 2024-03-29T10:14:48.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.203+0000 7fd5b9000700 1 -- 172.21.15.12:0/397020224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b4067720 con 0x7fd5b40ce230 2024-03-29T10:14:48.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.204+0000 7fd5aa000700 1 -- 172.21.15.12:0/397020224 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 73) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fd5a0013b20 con 0x7fd5b40ce230 2024-03-29T10:14:48.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.205+0000 7fd5aa000700 1 --2- 172.21.15.12:0/397020224 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd59406ae90 0x7fd59406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.205+0000 7fd5aa000700 1 -- 172.21.15.12:0/397020224 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fd5a0089710 con 0x7fd5b40ce230 2024-03-29T10:14:48.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.205+0000 7fd5b1e00700 1 --2- 172.21.15.12:0/397020224 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd59406ae90 0x7fd59406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.206+0000 7fd5b1e00700 1 --2- 172.21.15.12:0/397020224 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd59406ae90 0x7fd59406d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4014040 tx=0x7fd5a4008030 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:48.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.210+0000 7fd5aa000700 1 -- 172.21.15.12:0/397020224 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd5a0054370 con 0x7fd5b40ce230 2024-03-29T10:14:48.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.453+0000 7fd5b9000700 1 -- 172.21.15.12:0/397020224 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5b4072b00 con 0x7fd59406ae90 2024-03-29T10:14:48.453 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:48.454+0000 7f9db2400700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-29T10:14:48.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.455+0000 7fd5aa000700 1 -- 172.21.15.12:0/397020224 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd5a400bb40 con 0x7fd59406ae90 2024-03-29T10:14:48.455 INFO:teuthology.orchestra.run.smithi012.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-29T10:14:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-29T10:14:48.463 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2024-03-29T10:14:48.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.666+0000 7f38afe00700 1 Processor -- start 2024-03-29T10:14:48.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.666+0000 7f38afe00700 1 -- start start 2024-03-29T10:14:48.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.667+0000 7f38afe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f38a80cd020 0x7f38a80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.667+0000 7f38afe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 0x7f38a80b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.667+0000 7f38afe00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a80cdae0 con 0x7f38a80cbb30 2024-03-29T10:14:48.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.667+0000 7f38afe00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a80cddc0 con 0x7f38a80cd020 2024-03-29T10:14:48.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.667+0000 7f38afe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80ce0a0 con 0x7f38a80b2070 2024-03-29T10:14:48.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38aca00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f38a80cd020 0x7f38a80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38ade00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 0x7f38a80b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38ad400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f38a80cbb30 0x7f38a80cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:42938/0 (socket says 172.21.15.12:42938) 2024-03-29T10:14:48.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38aca00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f38a80cd020 0x7f38a80cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:54586/0 (socket says 172.21.15.12:54586) 2024-03-29T10:14:48.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38ad400700 1 -- 172.21.15.12:0/2559594628 learned_addr learned my addr 172.21.15.12:0/2559594628 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:48.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38aca00700 1 -- 172.21.15.12:0/2559594628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 msgr2=0x7f38a80b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:48.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38aca00700 1 --2- 172.21.15.12:0/2559594628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 0x7f38a80b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38aca00700 1 -- 172.21.15.12:0/2559594628 >> v1:172.21.15.159:6789/0 conn(0x7f38a80cbb30 legacy=0x7f38a80cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:48.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38aca00700 1 -- 172.21.15.12:0/2559594628 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a806dc20 con 0x7f38a80cd020 2024-03-29T10:14:48.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38ade00700 1 --2- 172.21.15.12:0/2559594628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 0x7f38a80b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:48.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.668+0000 7f38a7e00700 1 -- 172.21.15.12:0/2559594628 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664678626 0 0) 0x7f38a80cdae0 con 0x7f38a80cbb30 2024-03-29T10:14:48.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.669+0000 7f38aca00700 1 --2- 172.21.15.12:0/2559594628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38a80cd020 0x7f38a80cd480 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3894009840 tx=0x7f3894009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fba2788ee0bf164 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:48.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.669+0000 7f38a7e00700 1 -- 172.21.15.12:0/2559594628 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894011830 con 0x7f38a80cd020 2024-03-29T10:14:48.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.669+0000 7f38a7e00700 1 -- 172.21.15.12:0/2559594628 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3894011990 con 0x7f38a80cd020 2024-03-29T10:14:48.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.669+0000 7f38a7e00700 1 -- 172.21.15.12:0/2559594628 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894011d20 con 0x7f38a80cd020 2024-03-29T10:14:48.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.669+0000 7f38afe00700 1 -- 172.21.15.12:0/2559594628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38a80cd020 msgr2=0x7f38a80cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:48.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.669+0000 7f38afe00700 1 --2- 172.21.15.12:0/2559594628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38a80cd020 0x7f38a80cd480 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3894009840 tx=0x7f3894009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:48.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.670+0000 7f38afe00700 1 -- 172.21.15.12:0/2559594628 shutdown_connections 2024-03-29T10:14:48.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.670+0000 7f38afe00700 1 --2- 172.21.15.12:0/2559594628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 0x7f38a80b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.670+0000 7f38afe00700 1 --2- 172.21.15.12:0/2559594628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38a80cd020 0x7f38a80cd480 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.670+0000 7f38afe00700 1 -- 172.21.15.12:0/2559594628 >> 172.21.15.12:0/2559594628 conn(0x7f38a80b0d70 msgr2=0x7f38a80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:48.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.670+0000 7f38afe00700 1 -- 172.21.15.12:0/2559594628 shutdown_connections 2024-03-29T10:14:48.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.670+0000 7f38afe00700 1 -- 172.21.15.12:0/2559594628 wait complete. 2024-03-29T10:14:48.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.671+0000 7f38afe00700 1 Processor -- start 2024-03-29T10:14:48.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.671+0000 7f38afe00700 1 -- start start 2024-03-29T10:14:48.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.672+0000 7f38afe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 0x7f38a80ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.672+0000 7f38afe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38a80cbb30 0x7f38a809cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.672+0000 7f38afe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38a80cd020 0x7f38a80c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.672+0000 7f38afe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a81f3730 con 0x7f38a80cbb30 2024-03-29T10:14:48.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.672+0000 7f38afe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80c8930 con 0x7f38a80cd020 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.672+0000 7f38afe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80c8bd0 con 0x7f38a80b2070 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38aca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38a80cbb30 0x7f38a809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38aca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38a80cbb30 0x7f38a809cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48662/0 (socket says 172.21.15.12:48662) 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38aca00700 1 -- 172.21.15.12:0/573287246 learned_addr learned my addr 172.21.15.12:0/573287246 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38ad400700 1 --2- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 0x7f38a80ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38ade00700 1 --2- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38a80cd020 0x7f38a80c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38aca00700 1 -- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 msgr2=0x7f38a80ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38aca00700 1 --2- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 0x7f38a80ad510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38aca00700 1 -- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38a80cd020 msgr2=0x7f38a80c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38aca00700 1 --2- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38a80cd020 0x7f38a80c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38aca00700 1 -- 172.21.15.12:0/573287246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a0009a70 con 0x7f38a80cbb30 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38ad400700 1 --2- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38a80b2070 0x7f38a80ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38ade00700 1 --2- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38a80cd020 0x7f38a80c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:48.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.673+0000 7f38aca00700 1 --2- 172.21.15.12:0/573287246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38a80cbb30 0x7f38a809cc30 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f3894009fd0 tx=0x7f3894004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:48.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.674+0000 7f38a5600700 1 -- 172.21.15.12:0/573287246 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38940038a0 con 0x7f38a80cbb30 2024-03-29T10:14:48.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.674+0000 7f38a5600700 1 -- 172.21.15.12:0/573287246 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389401f070 con 0x7f38a80cbb30 2024-03-29T10:14:48.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.674+0000 7f38a5600700 1 -- 172.21.15.12:0/573287246 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894003c40 con 0x7f38a80cbb30 2024-03-29T10:14:48.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.674+0000 7f38afe00700 1 -- 172.21.15.12:0/573287246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38940094a0 con 0x7f38a80cbb30 2024-03-29T10:14:48.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.674+0000 7f38afe00700 1 -- 172.21.15.12:0/573287246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a80c9190 con 0x7f38a80cbb30 2024-03-29T10:14:48.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.675+0000 7f38afe00700 1 -- 172.21.15.12:0/573287246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a81f38e0 con 0x7f38a80cbb30 2024-03-29T10:14:48.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.676+0000 7f38a5600700 1 -- 172.21.15.12:0/573287246 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 73) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f3894026480 con 0x7f38a80cbb30 2024-03-29T10:14:48.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.677+0000 7f38a5600700 1 --2- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f388806ae90 0x7f388806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:48.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.677+0000 7f38a5600700 1 -- 172.21.15.12:0/573287246 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f3894021070 con 0x7f38a80cbb30 2024-03-29T10:14:48.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.677+0000 7f38ad400700 1 --2- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f388806ae90 0x7f388806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:48.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.678+0000 7f38ad400700 1 --2- 172.21.15.12:0/573287246 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f388806ae90 0x7f388806d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f389c000f60 tx=0x7f389c007ca0 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:48.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.681+0000 7f38a5600700 1 -- 172.21.15.12:0/573287246 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f38a81f38e0 con 0x7f38a80cbb30 2024-03-29T10:14:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.920+0000 7f38afe00700 1 -- 172.21.15.12:0/573287246 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f38a815bdd0 con 0x7f388806ae90 2024-03-29T10:14:48.919 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:48.921+0000 7f9db2400700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-29T10:14:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:48.922+0000 7f38a5600700 1 -- 172.21.15.12:0/573287246 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f389c001310 con 0x7f388806ae90 2024-03-29T10:14:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-29T10:14:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-29T10:14:48.930 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2024-03-29T10:14:49.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.133+0000 7f6427400700 1 Processor -- start 2024-03-29T10:14:49.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.134+0000 7f6427400700 1 -- start start 2024-03-29T10:14:49.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.135+0000 7f6427400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f642015a5c0 0x7f64200709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.135+0000 7f6427400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f64200ce840 0x7f64200cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.135+0000 7f6427400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64200cb310 con 0x7f642015dfe0 2024-03-29T10:14:49.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.135+0000 7f6427400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6420070f00 con 0x7f64200ce840 2024-03-29T10:14:49.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.135+0000 7f6427400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64200711a0 con 0x7f642015a5c0 2024-03-29T10:14:49.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.135+0000 7f6424a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f642015a5c0 0x7f64200709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.135+0000 7f641fe00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f642015dfe0 0x7f64200ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:59858/0 (socket says 172.21.15.12:59858) 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.135+0000 7f641fe00700 1 -- 172.21.15.12:0/635240294 learned_addr learned my addr 172.21.15.12:0/635240294 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.135+0000 7f6425400700 1 --2- 172.21.15.12:0/635240294 >> v2:172.21.15.159:3300/0 conn(0x7f64200ce840 0x7f64200cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.136+0000 7f641f400700 1 -- 172.21.15.12:0/635240294 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405536851 0 0) 0x7f64200cb310 con 0x7f642015dfe0 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.136+0000 7f641f400700 1 -- 172.21.15.12:0/635240294 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64080035b0 con 0x7f642015dfe0 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.136+0000 7f6425400700 1 -- 172.21.15.12:0/635240294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f642015a5c0 msgr2=0x7f64200709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.136+0000 7f6425400700 1 --2- 172.21.15.12:0/635240294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f642015a5c0 0x7f64200709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.136+0000 7f6425400700 1 -- 172.21.15.12:0/635240294 >> v1:172.21.15.159:6789/0 conn(0x7f642015dfe0 legacy=0x7f64200ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.136+0000 7f6425400700 1 -- 172.21.15.12:0/635240294 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6420071480 con 0x7f64200ce840 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.136+0000 7f6424a00700 1 --2- 172.21.15.12:0/635240294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f642015a5c0 0x7f64200709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.136+0000 7f6425400700 1 --2- 172.21.15.12:0/635240294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64200ce840 0x7f64200cba80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6418009d90 tx=0x7f641800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=528508caedf5c6ec server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f641f400700 1 -- 172.21.15.12:0/635240294 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641800e070 con 0x7f64200ce840 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f641f400700 1 -- 172.21.15.12:0/635240294 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641800bb20 con 0x7f64200ce840 2024-03-29T10:14:49.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f641f400700 1 -- 172.21.15.12:0/635240294 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6418012430 con 0x7f64200ce840 2024-03-29T10:14:49.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f6427400700 1 -- 172.21.15.12:0/635240294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64200ce840 msgr2=0x7f64200cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:49.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f6427400700 1 --2- 172.21.15.12:0/635240294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64200ce840 0x7f64200cba80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6418009d90 tx=0x7f641800b330 comp rx=0 tx=0).stop 2024-03-29T10:14:49.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f6427400700 1 -- 172.21.15.12:0/635240294 shutdown_connections 2024-03-29T10:14:49.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f6427400700 1 --2- 172.21.15.12:0/635240294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64200ce840 0x7f64200cba80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f6427400700 1 --2- 172.21.15.12:0/635240294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f642015a5c0 0x7f64200709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f6427400700 1 -- 172.21.15.12:0/635240294 >> 172.21.15.12:0/635240294 conn(0x7f64200b0d70 msgr2=0x7f64200b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:49.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f6427400700 1 -- 172.21.15.12:0/635240294 shutdown_connections 2024-03-29T10:14:49.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.137+0000 7f6427400700 1 -- 172.21.15.12:0/635240294 wait complete. 2024-03-29T10:14:49.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.139+0000 7f6427400700 1 Processor -- start 2024-03-29T10:14:49.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.139+0000 7f6427400700 1 -- start start 2024-03-29T10:14:49.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.140+0000 7f6427400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64200ce840 0x7f6420156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.140+0000 7f6427400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f642015a5c0 0x7f642014c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.140+0000 7f6427400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642015dfe0 0x7f64200ad590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.140+0000 7f6427400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6420156780 con 0x7f642015dfe0 2024-03-29T10:14:49.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.140+0000 7f6427400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64200ae070 con 0x7f642015a5c0 2024-03-29T10:14:49.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.140+0000 7f6427400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64200ae350 con 0x7f64200ce840 2024-03-29T10:14:49.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f6425400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642015dfe0 0x7f64200ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f6425400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642015dfe0 0x7f64200ad590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35814/0 (socket says 172.21.15.12:35814) 2024-03-29T10:14:49.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f6425400700 1 -- 172.21.15.12:0/991578468 learned_addr learned my addr 172.21.15.12:0/991578468 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:49.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f6424a00700 1 --2- 172.21.15.12:0/991578468 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64200ce840 0x7f6420156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f641fe00700 1 --2- 172.21.15.12:0/991578468 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f642015a5c0 0x7f642014c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f6425400700 1 -- 172.21.15.12:0/991578468 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64200ce840 msgr2=0x7f6420156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:49.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f6425400700 1 --2- 172.21.15.12:0/991578468 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64200ce840 0x7f6420156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f6425400700 1 -- 172.21.15.12:0/991578468 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f642015a5c0 msgr2=0x7f642014c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:49.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f6425400700 1 --2- 172.21.15.12:0/991578468 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f642015a5c0 0x7f642014c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f6425400700 1 -- 172.21.15.12:0/991578468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64140094a0 con 0x7f642015dfe0 2024-03-29T10:14:49.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.141+0000 7f6425400700 1 --2- 172.21.15.12:0/991578468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642015dfe0 0x7f64200ad590 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f6418014040 tx=0x7f6418004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:49.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.142+0000 7f641cc00700 1 -- 172.21.15.12:0/991578468 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64180129f0 con 0x7f642015dfe0 2024-03-29T10:14:49.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.142+0000 7f641cc00700 1 -- 172.21.15.12:0/991578468 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6418020070 con 0x7f642015dfe0 2024-03-29T10:14:49.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.142+0000 7f641cc00700 1 -- 172.21.15.12:0/991578468 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6418012db0 con 0x7f642015dfe0 2024-03-29T10:14:49.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.142+0000 7f6427400700 1 -- 172.21.15.12:0/991578468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6418009a70 con 0x7f642015dfe0 2024-03-29T10:14:49.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.142+0000 7f6427400700 1 -- 172.21.15.12:0/991578468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6420155d20 con 0x7f642015dfe0 2024-03-29T10:14:49.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.143+0000 7f6427400700 1 -- 172.21.15.12:0/991578468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64200ae590 con 0x7f642015dfe0 2024-03-29T10:14:49.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.144+0000 7f641cc00700 1 -- 172.21.15.12:0/991578468 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 73) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f6418004040 con 0x7f642015dfe0 2024-03-29T10:14:49.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.145+0000 7f641cc00700 1 --2- 172.21.15.12:0/991578468 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f640006ae90 0x7f640006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.145+0000 7f641cc00700 1 -- 172.21.15.12:0/991578468 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6418022070 con 0x7f642015dfe0 2024-03-29T10:14:49.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.146+0000 7f6424a00700 1 --2- 172.21.15.12:0/991578468 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f640006ae90 0x7f640006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.146+0000 7f6424a00700 1 --2- 172.21.15.12:0/991578468 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f640006ae90 0x7f640006d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6414009d70 tx=0x7f6414009210 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:49.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.150+0000 7f641cc00700 1 -- 172.21.15.12:0/991578468 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64180561a0 con 0x7f642015dfe0 2024-03-29T10:14:49.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.393+0000 7f6427400700 1 -- 172.21.15.12:0/991578468 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f64201558c0 con 0x7f640006ae90 2024-03-29T10:14:49.392 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:49.394+0000 7f9db2400700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-03-29T10:14:49.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.394+0000 7f641cc00700 1 -- 172.21.15.12:0/991578468 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7f64140099c0 con 0x7f640006ae90 2024-03-29T10:14:49.394 INFO:teuthology.orchestra.run.smithi012.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-03-29T10:14:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-29T10:14:49.403 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2024-03-29T10:14:49.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.605+0000 7f54a8a00700 1 Processor -- start 2024-03-29T10:14:49.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.605+0000 7f54a8a00700 1 -- start start 2024-03-29T10:14:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.606+0000 7f54a8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a40cbb30 0x7f54a40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.606+0000 7f54a8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f54a40cd020 0x7f54a40cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.606+0000 7f54a8a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f54a40b2070 0x7f54a40b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.606+0000 7f54a8a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a40cc450 con 0x7f54a40cd020 2024-03-29T10:14:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a8a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a40cdae0 con 0x7f54a40b2070 2024-03-29T10:14:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a40cddc0 con 0x7f54a40cbb30 2024-03-29T10:14:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a1400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f54a40cd020 0x7f54a40cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a1400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f54a40cd020 0x7f54a40cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35830/0 (socket says 172.21.15.12:35830) 2024-03-29T10:14:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a1400700 1 -- 172.21.15.12:0/4286692181 learned_addr learned my addr 172.21.15.12:0/4286692181 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a1e00700 1 --2- 172.21.15.12:0/4286692181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a40cbb30 0x7f54a40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a2800700 1 --2- 172.21.15.12:0/4286692181 >> v2:172.21.15.159:3300/0 conn(0x7f54a40b2070 0x7f54a40b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a1400700 1 -- 172.21.15.12:0/4286692181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a40cbb30 msgr2=0x7f54a40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a1400700 1 --2- 172.21.15.12:0/4286692181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a40cbb30 0x7f54a40cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a1400700 1 -- 172.21.15.12:0/4286692181 >> v2:172.21.15.159:3300/0 conn(0x7f54a40b2070 msgr2=0x7f54a40b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a1400700 1 --2- 172.21.15.12:0/4286692181 >> v2:172.21.15.159:3300/0 conn(0x7f54a40b2070 0x7f54a40b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a1400700 1 -- 172.21.15.12:0/4286692181 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a40ce0a0 con 0x7f54a40cd020 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.607+0000 7f54a2800700 1 --2- 172.21.15.12:0/4286692181 >> v2:172.21.15.159:3300/0 conn(0x7f54a40b2070 0x7f54a40b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a1400700 1 --2- 172.21.15.12:0/4286692181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a40cd020 0x7f54a40cd4a0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f54a406c9f0 tx=0x7f5490009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02b5c325cba25d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a0a00700 1 -- 172.21.15.12:0/4286692181 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549000b870 con 0x7f54a40cd020 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a0a00700 1 -- 172.21.15.12:0/4286692181 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549000b9d0 con 0x7f54a40cd020 2024-03-29T10:14:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a0a00700 1 -- 172.21.15.12:0/4286692181 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549000bd60 con 0x7f54a40cd020 2024-03-29T10:14:49.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a8a00700 1 -- 172.21.15.12:0/4286692181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a40cd020 msgr2=0x7f54a40cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a8a00700 1 --2- 172.21.15.12:0/4286692181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a40cd020 0x7f54a40cd4a0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f54a406c9f0 tx=0x7f5490009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a8a00700 1 -- 172.21.15.12:0/4286692181 shutdown_connections 2024-03-29T10:14:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a8a00700 1 --2- 172.21.15.12:0/4286692181 >> v2:172.21.15.159:3300/0 conn(0x7f54a40b2070 0x7f54a40b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a8a00700 1 --2- 172.21.15.12:0/4286692181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a40cbb30 0x7f54a40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a8a00700 1 --2- 172.21.15.12:0/4286692181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a40cd020 0x7f54a40cd4a0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a8a00700 1 -- 172.21.15.12:0/4286692181 >> 172.21.15.12:0/4286692181 conn(0x7f54a40b0d70 msgr2=0x7f54a40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.608+0000 7f54a8a00700 1 -- 172.21.15.12:0/4286692181 shutdown_connections 2024-03-29T10:14:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.609+0000 7f54a8a00700 1 -- 172.21.15.12:0/4286692181 wait complete. 2024-03-29T10:14:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.610+0000 7f54a8a00700 1 Processor -- start 2024-03-29T10:14:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.611+0000 7f54a8a00700 1 -- start start 2024-03-29T10:14:49.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.611+0000 7f54a8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a40b2070 0x7f54a415d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.611+0000 7f54a8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a40cbb30 0x7f54a41526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.611+0000 7f54a8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a40cd020 0x7f54a40ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a4155cf0 con 0x7f54a40cd020 2024-03-29T10:14:49.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a8a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a415dd80 con 0x7f54a40cbb30 2024-03-29T10:14:49.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a4156090 con 0x7f54a40b2070 2024-03-29T10:14:49.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a40cd020 0x7f54a40ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a40cd020 0x7f54a40ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35840/0 (socket says 172.21.15.12:35840) 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a2800700 1 -- 172.21.15.12:0/3238772191 learned_addr learned my addr 172.21.15.12:0/3238772191 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a1e00700 1 --2- 172.21.15.12:0/3238772191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a40b2070 0x7f54a415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a1400700 1 --2- 172.21.15.12:0/3238772191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a40cbb30 0x7f54a41526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a2800700 1 -- 172.21.15.12:0/3238772191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a40b2070 msgr2=0x7f54a415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a2800700 1 --2- 172.21.15.12:0/3238772191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a40b2070 0x7f54a415d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a2800700 1 -- 172.21.15.12:0/3238772191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a40cbb30 msgr2=0x7f54a41526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a2800700 1 --2- 172.21.15.12:0/3238772191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a40cbb30 0x7f54a41526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a2800700 1 -- 172.21.15.12:0/3238772191 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5490009550 con 0x7f54a40cd020 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.612+0000 7f54a2800700 1 --2- 172.21.15.12:0/3238772191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a40cd020 0x7f54a40ad1d0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f549c007e70 tx=0x7f549c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.613+0000 7f549a000700 1 -- 172.21.15.12:0/3238772191 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c00dd20 con 0x7f54a40cd020 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.613+0000 7f549a000700 1 -- 172.21.15.12:0/3238772191 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549c00f040 con 0x7f54a40cd020 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.613+0000 7f54a8a00700 1 -- 172.21.15.12:0/3238772191 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a4156370 con 0x7f54a40cd020 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.613+0000 7f549a000700 1 -- 172.21.15.12:0/3238772191 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c00dea0 con 0x7f54a40cd020 2024-03-29T10:14:49.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.613+0000 7f54a8a00700 1 -- 172.21.15.12:0/3238772191 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a4156930 con 0x7f54a40cd020 2024-03-29T10:14:49.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.614+0000 7f54a8a00700 1 -- 172.21.15.12:0/3238772191 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a4067720 con 0x7f54a40cd020 2024-03-29T10:14:49.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.615+0000 7f549a000700 1 -- 172.21.15.12:0/3238772191 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 73) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f549c003c60 con 0x7f54a40cd020 2024-03-29T10:14:49.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.616+0000 7f549a000700 1 --2- 172.21.15.12:0/3238772191 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f548406ae90 0x7f548406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:49.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.616+0000 7f549a000700 1 -- 172.21.15.12:0/3238772191 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f549c0898a0 con 0x7f54a40cd020 2024-03-29T10:14:49.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.617+0000 7f54a1e00700 1 --2- 172.21.15.12:0/3238772191 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f548406ae90 0x7f548406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:49.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.617+0000 7f54a1e00700 1 --2- 172.21.15.12:0/3238772191 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f548406ae90 0x7f548406d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5494006b20 tx=0x7f5494008040 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:49.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.620+0000 7f549a000700 1 -- 172.21.15.12:0/3238772191 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f549c054580 con 0x7f54a40cd020 2024-03-29T10:14:49.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.858+0000 7f54a8a00700 1 -- 172.21.15.12:0/3238772191 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f54a41594a0 con 0x7f548406ae90 2024-03-29T10:14:49.857 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:49.859+0000 7f9db2400700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-29T10:14:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:49.860+0000 7f549a000700 1 -- 172.21.15.12:0/3238772191 <== mgr.8275 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5494007630 con 0x7f548406ae90 2024-03-29T10:14:49.860 INFO:teuthology.orchestra.run.smithi012.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-29T10:14:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-29T10:14:49.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:49.869 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2024-03-29T10:14:49.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:49.916 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2024-03-29T10:14:49.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:49.962 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2024-03-29T10:14:50.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:50.052 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2024-03-29T10:14:50.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:50.141 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2024-03-29T10:14:50.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:50.186 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2024-03-29T10:14:50.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:50.232 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2024-03-29T10:14:50.318 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T10:14:50.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.559+0000 7f54d5200700 1 Processor -- start 2024-03-29T10:14:50.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.559+0000 7f54d5200700 1 -- start start 2024-03-29T10:14:50.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.560+0000 7f54d5200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d015d9d0 0x7f54d00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:50.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.560+0000 7f54d5200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f54d00d16a0 0x7f54d00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:50.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.560+0000 7f54d5200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d0161420 con 0x7f54d00d2370 2024-03-29T10:14:50.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.560+0000 7f54d5200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d00d2820 con 0x7f54d00d16a0 2024-03-29T10:14:50.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.560+0000 7f54d5200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0161700 con 0x7f54d015d9d0 2024-03-29T10:14:50.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.560+0000 7f54ce800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f54d00d2370 0x7f54d015ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52290/0 (socket says 172.21.15.12:52290) 2024-03-29T10:14:50.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.560+0000 7f54ce800700 1 -- 172.21.15.12:0/24166300 learned_addr learned my addr 172.21.15.12:0/24166300 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:50.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.560+0000 7f54cd400700 1 --2- 172.21.15.12:0/24166300 >> v2:172.21.15.159:3300/0 conn(0x7f54d00d16a0 0x7f54d00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:50.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.560+0000 7f54cde00700 1 --2- 172.21.15.12:0/24166300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d015d9d0 0x7f54d00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:50.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.561+0000 7f54cca00700 1 -- 172.21.15.12:0/24166300 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847189193 0 0) 0x7f54d0161420 con 0x7f54d00d2370 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.561+0000 7f54cca00700 1 -- 172.21.15.12:0/24166300 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b40035b0 con 0x7f54d00d2370 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.561+0000 7f54cd400700 1 -- 172.21.15.12:0/24166300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d015d9d0 msgr2=0x7f54d00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.561+0000 7f54cd400700 1 --2- 172.21.15.12:0/24166300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d015d9d0 0x7f54d00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.561+0000 7f54cd400700 1 -- 172.21.15.12:0/24166300 >> v1:172.21.15.12:6789/0 conn(0x7f54d00d2370 legacy=0x7f54d015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.561+0000 7f54cd400700 1 -- 172.21.15.12:0/24166300 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d01619e0 con 0x7f54d00d16a0 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.561+0000 7f54cd400700 1 --2- 172.21.15.12:0/24166300 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54d00d16a0 0x7f54d00d1e30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f54bc009820 tx=0x7f54bc009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f44e9d0c591a97d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.561+0000 7f54cca00700 1 -- 172.21.15.12:0/24166300 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc005e30 con 0x7f54d00d16a0 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.561+0000 7f54cca00700 1 -- 172.21.15.12:0/24166300 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54bc006030 con 0x7f54d00d16a0 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.561+0000 7f54cca00700 1 -- 172.21.15.12:0/24166300 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc0063c0 con 0x7f54d00d16a0 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.562+0000 7f54d5200700 1 -- 172.21.15.12:0/24166300 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54d00d16a0 msgr2=0x7f54d00d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.562+0000 7f54d5200700 1 --2- 172.21.15.12:0/24166300 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54d00d16a0 0x7f54d00d1e30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f54bc009820 tx=0x7f54bc009b30 comp rx=0 tx=0).stop 2024-03-29T10:14:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.562+0000 7f54d5200700 1 -- 172.21.15.12:0/24166300 shutdown_connections 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.562+0000 7f54d5200700 1 --2- 172.21.15.12:0/24166300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d015d9d0 0x7f54d00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.562+0000 7f54d5200700 1 --2- 172.21.15.12:0/24166300 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54d00d16a0 0x7f54d00d1e30 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.562+0000 7f54d5200700 1 -- 172.21.15.12:0/24166300 >> 172.21.15.12:0/24166300 conn(0x7f54d008aff0 msgr2=0x7f54d00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.562+0000 7f54d5200700 1 -- 172.21.15.12:0/24166300 shutdown_connections 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.562+0000 7f54d5200700 1 -- 172.21.15.12:0/24166300 wait complete. 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.564+0000 7f54d5200700 1 Processor -- start 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.565+0000 7f54d5200700 1 -- start start 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54d5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d00d16a0 0x7f54d0158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54d5200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d00d2370 0x7f54d014ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54d5200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54d015d9d0 0x7f54d00b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54d5200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d014ef40 con 0x7f54d00d16a0 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54d5200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d00b6f70 con 0x7f54d015d9d0 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54d5200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d00b74e0 con 0x7f54d00d2370 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54cde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d00d16a0 0x7f54d0158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54cde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d00d16a0 0x7f54d0158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35854/0 (socket says 172.21.15.12:35854) 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54cde00700 1 -- 172.21.15.12:0/2404613077 learned_addr learned my addr 172.21.15.12:0/2404613077 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54cd400700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d00d2370 0x7f54d014ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.566+0000 7f54ce800700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54d015d9d0 0x7f54d00b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54cde00700 1 -- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d00d2370 msgr2=0x7f54d014ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54cde00700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d00d2370 0x7f54d014ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54cde00700 1 -- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54d015d9d0 msgr2=0x7f54d00b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54cde00700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54d015d9d0 0x7f54d00b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54cde00700 1 -- 172.21.15.12:0/2404613077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc009430 con 0x7f54d00d16a0 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54ce800700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54d015d9d0 0x7f54d00b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54cde00700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d00d16a0 0x7f54d0158860 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f54c000e870 tx=0x7f54c000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54c6000700 1 -- 172.21.15.12:0/2404613077 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0009590 con 0x7f54d00d16a0 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54c6000700 1 -- 172.21.15.12:0/2404613077 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c000ee70 con 0x7f54d00d16a0 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54d5200700 1 -- 172.21.15.12:0/2404613077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d00b77c0 con 0x7f54d00d16a0 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54c6000700 1 -- 172.21.15.12:0/2404613077 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c000ca50 con 0x7f54d00d16a0 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.567+0000 7f54d5200700 1 -- 172.21.15.12:0/2404613077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d00b7d80 con 0x7f54d00d16a0 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.568+0000 7f54d5200700 1 -- 172.21.15.12:0/2404613077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d00b83d0 con 0x7f54d00d16a0 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.569+0000 7f54c6000700 1 -- 172.21.15.12:0/2404613077 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 73) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f54c000cbb0 con 0x7f54d00d16a0 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.570+0000 7f54c6000700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b006ae90 0x7f54b006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.570+0000 7f54c6000700 1 -- 172.21.15.12:0/2404613077 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f54c0089860 con 0x7f54d00d16a0 2024-03-29T10:14:50.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.571+0000 7f54cd400700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b006ae90 0x7f54b006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:50.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.572+0000 7f54cd400700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b006ae90 0x7f54b006d350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f54d00d1ae0 tx=0x7f54bc008070 comp rx=0 tx=0).ready entity=mgr.8275 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:50.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.574+0000 7f54c6000700 1 -- 172.21.15.12:0/2404613077 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54c00545c0 con 0x7f54d00d16a0 2024-03-29T10:14:50.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:50.853+0000 7f54d5200700 1 -- 172.21.15.12:0/2404613077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f54d00d1ca0 con 0x7f54d00d16a0 2024-03-29T10:14:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.125+0000 7f54c6000700 1 -- 172.21.15.12:0/2404613077 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f54c0054990 con 0x7f54d00d16a0 2024-03-29T10:14:51.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.126+0000 7f54c6000700 1 -- 172.21.15.12:0/2404613077 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v74) v1 ==== 93+0+0 (secure 0 0 0) 0x7f54c0087560 con 0x7f54d00d16a0 2024-03-29T10:14:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 -- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b006ae90 msgr2=0x7f54b006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b006ae90 0x7f54b006d350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f54d00d1ae0 tx=0x7f54bc008070 comp rx=0 tx=0).stop 2024-03-29T10:14:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 -- 172.21.15.12:0/2404613077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d00d16a0 msgr2=0x7f54d0158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d00d16a0 0x7f54d0158860 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f54c000e870 tx=0x7f54c000ec30 comp rx=0 tx=0).stop 2024-03-29T10:14:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 -- 172.21.15.12:0/2404613077 shutdown_connections 2024-03-29T10:14:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b006ae90 0x7f54b006d350 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54d015d9d0 0x7f54d00b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d00d16a0 0x7f54d0158860 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 --2- 172.21.15.12:0/2404613077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d00d2370 0x7f54d014ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 -- 172.21.15.12:0/2404613077 >> 172.21.15.12:0/2404613077 conn(0x7f54d008aff0 msgr2=0x7f54d00b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 -- 172.21.15.12:0/2404613077 shutdown_connections 2024-03-29T10:14:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.129+0000 7f54d5200700 1 -- 172.21.15.12:0/2404613077 wait complete. 2024-03-29T10:14:51.147 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:14:51.147 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:14:51.246 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:14:51.349 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:51.351+0000 7fc7e1153240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:14:51.351 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:14:51.352 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:14:51.352 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:51.362 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:51.364+0000 7fb8a456a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:14:51.398 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:14:51.398 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:51.491 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:51.547 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:51.549+0000 7fb8a456a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:14:51.556 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:51.558+0000 7fc7e1153240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:14:51.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.718+0000 7f5fea600700 1 Processor -- start 2024-03-29T10:14:51.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.719+0000 7f5fea600700 1 -- start start 2024-03-29T10:14:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.720+0000 7f5fea600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5fe40ac7d0 0x7f5fe4166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.720+0000 7f5fea600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe415fd50 con 0x7f5fe415b5d0 2024-03-29T10:14:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.720+0000 7f5fea600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe415f140 con 0x7f5fe40cb3f0 2024-03-29T10:14:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.720+0000 7f5fea600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe415b9b0 con 0x7f5fe40ac7d0 2024-03-29T10:14:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.720+0000 7f5fe3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5fe40ac7d0 0x7f5fe4166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.720+0000 7f5fe3400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5fe415b5d0 0x7f5fe40ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52310/0 (socket says 172.21.15.12:52310) 2024-03-29T10:14:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.720+0000 7f5fe3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5fe40ac7d0 0x7f5fe4166430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35858/0 (socket says 172.21.15.12:35858) 2024-03-29T10:14:51.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.720+0000 7f5fe3400700 1 -- 172.21.15.12:0/1357363709 learned_addr learned my addr 172.21.15.12:0/1357363709 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.721+0000 7f5fe3e00700 1 -- 172.21.15.12:0/1357363709 >> v1:172.21.15.159:6789/0 conn(0x7f5fe40cb3f0 legacy=0x7f5fe40ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.721+0000 7f5fe2000700 1 -- 172.21.15.12:0/1357363709 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397930769 0 0) 0x7f5fe415f140 con 0x7f5fe40cb3f0 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.721+0000 7f5fe3e00700 1 -- 172.21.15.12:0/1357363709 >> v1:172.21.15.12:6789/0 conn(0x7f5fe415b5d0 legacy=0x7f5fe40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.721+0000 7f5fe3e00700 1 -- 172.21.15.12:0/1357363709 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe415f420 con 0x7f5fe40ac7d0 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.721+0000 7f5fe3e00700 1 --2- 172.21.15.12:0/1357363709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe40ac7d0 0x7f5fe4166430 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009d90 tx=0x7f5fdc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609776fa6dabfd96 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.721+0000 7f5fe2000700 1 -- 172.21.15.12:0/1357363709 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc00e070 con 0x7f5fe40ac7d0 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.721+0000 7f5fe2000700 1 -- 172.21.15.12:0/1357363709 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc00bbd0 con 0x7f5fe40ac7d0 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.721+0000 7f5fe2000700 1 -- 172.21.15.12:0/1357363709 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc012430 con 0x7f5fe40ac7d0 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.722+0000 7f5fea600700 1 -- 172.21.15.12:0/1357363709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe40ac7d0 msgr2=0x7f5fe4166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.722+0000 7f5fea600700 1 --2- 172.21.15.12:0/1357363709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe40ac7d0 0x7f5fe4166430 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009d90 tx=0x7f5fdc00b330 comp rx=0 tx=0).stop 2024-03-29T10:14:51.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.722+0000 7f5fea600700 1 -- 172.21.15.12:0/1357363709 shutdown_connections 2024-03-29T10:14:51.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.722+0000 7f5fea600700 1 --2- 172.21.15.12:0/1357363709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe40ac7d0 0x7f5fe4166430 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.722+0000 7f5fea600700 1 -- 172.21.15.12:0/1357363709 >> 172.21.15.12:0/1357363709 conn(0x7f5fe40cf190 msgr2=0x7f5fe40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:51.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.722+0000 7f5fea600700 1 -- 172.21.15.12:0/1357363709 shutdown_connections 2024-03-29T10:14:51.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.722+0000 7f5fea600700 1 -- 172.21.15.12:0/1357363709 wait complete. 2024-03-29T10:14:51.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.724+0000 7f5fea600700 1 Processor -- start 2024-03-29T10:14:51.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.724+0000 7f5fea600700 1 -- start start 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fea600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fe40ac7d0 0x7f5fe41445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fea600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe415b5d0 0x7f5fe4151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fea600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fe40b3b90 0x7f5fe4150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fea600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe40aece0 con 0x7f5fe415b5d0 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fea600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe406d2d0 con 0x7f5fe40ac7d0 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fea600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe40aef30 con 0x7f5fe40b3b90 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fe2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe415b5d0 0x7f5fe4151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fe2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe415b5d0 0x7f5fe4151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35860/0 (socket says 172.21.15.12:35860) 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fe2a00700 1 -- 172.21.15.12:0/2402469635 learned_addr learned my addr 172.21.15.12:0/2402469635 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fe3400700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fe40ac7d0 0x7f5fe41445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:51.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fe3e00700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fe40b3b90 0x7f5fe4150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:51.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fe2a00700 1 -- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fe40b3b90 msgr2=0x7f5fe4150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:51.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fe2a00700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fe40b3b90 0x7f5fe4150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fe2a00700 1 -- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fe40ac7d0 msgr2=0x7f5fe41445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:51.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fe2a00700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fe40ac7d0 0x7f5fe41445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.725+0000 7f5fe2a00700 1 -- 172.21.15.12:0/2402469635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc009a70 con 0x7f5fe415b5d0 2024-03-29T10:14:51.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.726+0000 7f5fe2a00700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe415b5d0 0x7f5fe4151a40 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f5fd00011e0 tx=0x7f5fd000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:51.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.726+0000 7f5fdb400700 1 -- 172.21.15.12:0/2402469635 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd00054e0 con 0x7f5fe415b5d0 2024-03-29T10:14:51.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.726+0000 7f5fdb400700 1 -- 172.21.15.12:0/2402469635 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd0003680 con 0x7f5fe415b5d0 2024-03-29T10:14:51.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.726+0000 7f5fea600700 1 -- 172.21.15.12:0/2402469635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe4151520 con 0x7f5fe415b5d0 2024-03-29T10:14:51.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.726+0000 7f5fdb400700 1 -- 172.21.15.12:0/2402469635 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0003a30 con 0x7f5fe415b5d0 2024-03-29T10:14:51.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.726+0000 7f5fea600700 1 -- 172.21.15.12:0/2402469635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe4151f80 con 0x7f5fe415b5d0 2024-03-29T10:14:51.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.727+0000 7f5fea600700 1 -- 172.21.15.12:0/2402469635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe4151760 con 0x7f5fe415b5d0 2024-03-29T10:14:51.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.728+0000 7f5fdb400700 1 -- 172.21.15.12:0/2402469635 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f5fd0003b90 con 0x7f5fe415b5d0 2024-03-29T10:14:51.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.729+0000 7f5fdb400700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5fc406ae70 0x7f5fc406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:51.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.729+0000 7f5fdb400700 1 -- 172.21.15.12:0/2402469635 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f5fd0088b90 con 0x7f5fe415b5d0 2024-03-29T10:14:51.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.729+0000 7f5fe3400700 1 -- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5fc406ae70 msgr2=0x7f5fc406d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:51.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.729+0000 7f5fe3400700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5fc406ae70 0x7f5fc406d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:51.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.738+0000 7f5fdb400700 1 -- 172.21.15.12:0/2402469635 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5fe4151760 con 0x7f5fe415b5d0 2024-03-29T10:14:51.866 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:51.868+0000 7fb8a456a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:14:51.899 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:51.901+0000 7fc7e1153240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:14:51.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.930+0000 7f5fe3400700 1 -- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5fc406ae70 msgr2=0x7f5fc406d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:51.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.930+0000 7f5fe3400700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5fc406ae70 0x7f5fc406d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.955+0000 7f5fea600700 1 -- 172.21.15.12:0/2402469635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fe415dd70 con 0x7f5fe415b5d0 2024-03-29T10:14:51.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.957+0000 7f5fdb400700 1 -- 172.21.15.12:0/2402469635 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16558 (secure 0 0 0) 0x7f5fd0017020 con 0x7f5fe415b5d0 2024-03-29T10:14:51.955 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:51.955 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":141,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:14:42.823304+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:14:23.708181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-29T10:14:24.829152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3261410285":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2027263440":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/4127686095":"2024-03-30T10:14:42.823243+0000","172.21.15.159:6833/43444":"2024-03-30T10:14:42.823243+0000","172.21.15.159:6832/43444":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/3727946147":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2214680226":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2899228198":"2024-03-30T10:14:42.823243+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:51.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.959+0000 7f5fea600700 1 -- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5fc406ae70 msgr2=0x7f5fc406d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.959+0000 7f5fea600700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5fc406ae70 0x7f5fc406d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.959+0000 7f5fea600700 1 -- 172.21.15.12:0/2402469635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe415b5d0 msgr2=0x7f5fe4151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.959+0000 7f5fea600700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe415b5d0 0x7f5fe4151a40 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f5fd00011e0 tx=0x7f5fd000ca30 comp rx=0 tx=0).stop 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.959+0000 7f5fea600700 1 -- 172.21.15.12:0/2402469635 shutdown_connections 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.959+0000 7f5fea600700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5fc406ae70 0x7f5fc406d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.959+0000 7f5fea600700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fe40b3b90 0x7f5fe4150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.959+0000 7f5fea600700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fe40ac7d0 0x7f5fe41445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.959+0000 7f5fea600700 1 --2- 172.21.15.12:0/2402469635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe415b5d0 0x7f5fe4151a40 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.959+0000 7f5fea600700 1 -- 172.21.15.12:0/2402469635 >> 172.21.15.12:0/2402469635 conn(0x7f5fe40cf190 msgr2=0x7f5fe415d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.960+0000 7f5fea600700 1 -- 172.21.15.12:0/2402469635 shutdown_connections 2024-03-29T10:14:51.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:51.960+0000 7f5fea600700 1 -- 172.21.15.12:0/2402469635 wait complete. 2024-03-29T10:14:51.976 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:14:23.708181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:14:24.829152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:14:51.977 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:14:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.185+0000 7fda92000700 1 Processor -- start 2024-03-29T10:14:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.185+0000 7fda92000700 1 -- start start 2024-03-29T10:14:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.187+0000 7fda92000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda8c15a5c0 0x7fda8c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.187+0000 7fda92000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda8c0ce840 0x7fda8c0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.187+0000 7fda92000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8c0cb310 con 0x7fda8c15dfe0 2024-03-29T10:14:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.187+0000 7fda92000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8c070f00 con 0x7fda8c15a5c0 2024-03-29T10:14:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.187+0000 7fda92000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c0711a0 con 0x7fda8c0ce840 2024-03-29T10:14:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.187+0000 7fda8b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda8c15a5c0 0x7fda8c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.187+0000 7fda8b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda8c15a5c0 0x7fda8c0709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35872/0 (socket says 172.21.15.12:35872) 2024-03-29T10:14:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.187+0000 7fda8aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fda8c15dfe0 0x7fda8c0ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52314/0 (socket says 172.21.15.12:52314) 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.187+0000 7fda8b400700 1 -- 172.21.15.12:0/3249329871 learned_addr learned my addr 172.21.15.12:0/3249329871 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.187+0000 7fda8be00700 1 --2- 172.21.15.12:0/3249329871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda8c0ce840 0x7fda8c0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.188+0000 7fda8b400700 1 -- 172.21.15.12:0/3249329871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda8c0ce840 msgr2=0x7fda8c0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.188+0000 7fda8b400700 1 --2- 172.21.15.12:0/3249329871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda8c0ce840 0x7fda8c0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.188+0000 7fda8b400700 1 -- 172.21.15.12:0/3249329871 >> v1:172.21.15.12:6789/0 conn(0x7fda8c15dfe0 legacy=0x7fda8c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.188+0000 7fda8b400700 1 -- 172.21.15.12:0/3249329871 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c071480 con 0x7fda8c15a5c0 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.188+0000 7fda8a000700 1 -- 172.21.15.12:0/3249329871 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246976684 0 0) 0x7fda8c0cb310 con 0x7fda8c15dfe0 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.188+0000 7fda8b400700 1 --2- 172.21.15.12:0/3249329871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15a5c0 0x7fda8c0709c0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fda7c0098f0 tx=0x7fda7c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba442ef4bbfbbfd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.188+0000 7fda8a000700 1 -- 172.21.15.12:0/3249329871 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c011830 con 0x7fda8c15a5c0 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.188+0000 7fda8a000700 1 -- 172.21.15.12:0/3249329871 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c011990 con 0x7fda8c15a5c0 2024-03-29T10:14:52.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.188+0000 7fda8a000700 1 -- 172.21.15.12:0/3249329871 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c011d20 con 0x7fda8c15a5c0 2024-03-29T10:14:52.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.189+0000 7fda92000700 1 -- 172.21.15.12:0/3249329871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15a5c0 msgr2=0x7fda8c0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:52.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.189+0000 7fda92000700 1 --2- 172.21.15.12:0/3249329871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15a5c0 0x7fda8c0709c0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fda7c0098f0 tx=0x7fda7c009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:52.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.189+0000 7fda92000700 1 -- 172.21.15.12:0/3249329871 shutdown_connections 2024-03-29T10:14:52.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.189+0000 7fda92000700 1 --2- 172.21.15.12:0/3249329871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda8c0ce840 0x7fda8c0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.189+0000 7fda92000700 1 --2- 172.21.15.12:0/3249329871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15a5c0 0x7fda8c0709c0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.189+0000 7fda92000700 1 -- 172.21.15.12:0/3249329871 >> 172.21.15.12:0/3249329871 conn(0x7fda8c0b0d70 msgr2=0x7fda8c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:52.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.189+0000 7fda92000700 1 -- 172.21.15.12:0/3249329871 shutdown_connections 2024-03-29T10:14:52.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.189+0000 7fda92000700 1 -- 172.21.15.12:0/3249329871 wait complete. 2024-03-29T10:14:52.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.191+0000 7fda92000700 1 Processor -- start 2024-03-29T10:14:52.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.191+0000 7fda92000700 1 -- start start 2024-03-29T10:14:52.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda92000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda8c0ce840 0x7fda8c159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda92000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda8c15a5c0 0x7fda8c15aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda92000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15dfe0 0x7fda8c1554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda92000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c15af70 con 0x7fda8c15dfe0 2024-03-29T10:14:52.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda92000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c15b1a0 con 0x7fda8c15a5c0 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda92000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c15b480 con 0x7fda8c0ce840 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda8be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15dfe0 0x7fda8c1554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda8be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15dfe0 0x7fda8c1554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35886/0 (socket says 172.21.15.12:35886) 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda8be00700 1 -- 172.21.15.12:0/3781334139 learned_addr learned my addr 172.21.15.12:0/3781334139 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda8b400700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda8c0ce840 0x7fda8c159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.192+0000 7fda8aa00700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda8c15a5c0 0x7fda8c15aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda8be00700 1 -- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda8c0ce840 msgr2=0x7fda8c159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda8be00700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda8c0ce840 0x7fda8c159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda8be00700 1 -- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda8c15a5c0 msgr2=0x7fda8c15aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda8be00700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda8c15a5c0 0x7fda8c15aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda8be00700 1 -- 172.21.15.12:0/3781334139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c009550 con 0x7fda8c15dfe0 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda8be00700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15dfe0 0x7fda8c1554c0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fda8400efc0 tx=0x7fda8400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:52.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda83400700 1 -- 172.21.15.12:0/3781334139 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8400cc90 con 0x7fda8c15dfe0 2024-03-29T10:14:52.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda83400700 1 -- 172.21.15.12:0/3781334139 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8400f040 con 0x7fda8c15dfe0 2024-03-29T10:14:52.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda83400700 1 -- 172.21.15.12:0/3781334139 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda840146c0 con 0x7fda8c15dfe0 2024-03-29T10:14:52.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda92000700 1 -- 172.21.15.12:0/3781334139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c15b760 con 0x7fda8c15dfe0 2024-03-29T10:14:52.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.193+0000 7fda92000700 1 -- 172.21.15.12:0/3781334139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c15bd20 con 0x7fda8c15dfe0 2024-03-29T10:14:52.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.194+0000 7fda92000700 1 -- 172.21.15.12:0/3781334139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c157020 con 0x7fda8c15dfe0 2024-03-29T10:14:52.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.196+0000 7fda83400700 1 -- 172.21.15.12:0/3781334139 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fda84014820 con 0x7fda8c15dfe0 2024-03-29T10:14:52.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.197+0000 7fda83400700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda6c06ae40 0x7fda6c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.198+0000 7fda83400700 1 -- 172.21.15.12:0/3781334139 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fda8408a4b0 con 0x7fda8c15dfe0 2024-03-29T10:14:52.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.198+0000 7fda8b400700 1 -- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda6c06ae40 msgr2=0x7fda6c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:52.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.198+0000 7fda8b400700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda6c06ae40 0x7fda6c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:52.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.202+0000 7fda83400700 1 -- 172.21.15.12:0/3781334139 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fda84055100 con 0x7fda8c15dfe0 2024-03-29T10:14:52.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.399+0000 7fda8b400700 1 -- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda6c06ae40 msgr2=0x7fda6c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:52.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.399+0000 7fda8b400700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda6c06ae40 0x7fda6c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:52.401 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:52.403+0000 7fb8a456a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:14:52.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.420+0000 7fda92000700 1 -- 172.21.15.12:0/3781334139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda8c067720 con 0x7fda8c15dfe0 2024-03-29T10:14:52.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.421+0000 7fda83400700 1 -- 172.21.15.12:0/3781334139 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda84009ca0 con 0x7fda8c15dfe0 2024-03-29T10:14:52.419 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:14:52.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.423+0000 7fda92000700 1 -- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda6c06ae40 msgr2=0x7fda6c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:52.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.423+0000 7fda92000700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda6c06ae40 0x7fda6c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.423+0000 7fda92000700 1 -- 172.21.15.12:0/3781334139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15dfe0 msgr2=0x7fda8c1554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:52.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.423+0000 7fda92000700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15dfe0 0x7fda8c1554c0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fda8400efc0 tx=0x7fda8400c3a0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.424+0000 7fda92000700 1 -- 172.21.15.12:0/3781334139 shutdown_connections 2024-03-29T10:14:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.424+0000 7fda92000700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda6c06ae40 0x7fda6c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.424+0000 7fda92000700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda8c15dfe0 0x7fda8c1554c0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.424+0000 7fda92000700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda8c0ce840 0x7fda8c159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.424+0000 7fda92000700 1 --2- 172.21.15.12:0/3781334139 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda8c15a5c0 0x7fda8c15aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.424+0000 7fda92000700 1 -- 172.21.15.12:0/3781334139 >> 172.21.15.12:0/3781334139 conn(0x7fda8c0b0d70 msgr2=0x7fda8c06cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.424+0000 7fda92000700 1 -- 172.21.15.12:0/3781334139 shutdown_connections 2024-03-29T10:14:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.424+0000 7fda92000700 1 -- 172.21.15.12:0/3781334139 wait complete. 2024-03-29T10:14:52.441 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:14:52.470 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:52.472+0000 7fc7e1153240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:14:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.647+0000 7fb73b400700 1 Processor -- start 2024-03-29T10:14:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.647+0000 7fb73b400700 1 -- start start 2024-03-29T10:14:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.648+0000 7fb73b400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 0x7fb7340d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.648+0000 7fb73b400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb7340d16a0 0x7fb7340d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.648+0000 7fb73b400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb734161420 con 0x7fb7340d2370 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.648+0000 7fb73b400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7340d2820 con 0x7fb7340d16a0 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.648+0000 7fb73b400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb734161700 con 0x7fb73415d9d0 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.649+0000 7fb738a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 0x7fb7340d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.649+0000 7fb733e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb7340d16a0 0x7fb7340d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.649+0000 7fb738a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 0x7fb7340d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:56804/0 (socket says 172.21.15.12:56804) 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.649+0000 7fb738a00700 1 -- 172.21.15.12:0/687374831 learned_addr learned my addr 172.21.15.12:0/687374831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.649+0000 7fb738a00700 1 -- 172.21.15.12:0/687374831 >> v2:172.21.15.159:3300/0 conn(0x7fb7340d16a0 msgr2=0x7fb7340d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.649+0000 7fb738a00700 1 --2- 172.21.15.12:0/687374831 >> v2:172.21.15.159:3300/0 conn(0x7fb7340d16a0 0x7fb7340d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.649+0000 7fb738a00700 1 -- 172.21.15.12:0/687374831 >> v1:172.21.15.159:6789/0 conn(0x7fb7340d2370 legacy=0x7fb73415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.649+0000 7fb738a00700 1 -- 172.21.15.12:0/687374831 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7341619e0 con 0x7fb73415d9d0 2024-03-29T10:14:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.649+0000 7fb738a00700 1 --2- 172.21.15.12:0/687374831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 0x7fb7340d3c30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb7280098f0 tx=0x7fb728009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d9ddb0556b5801f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.650+0000 7fb733400700 1 -- 172.21.15.12:0/687374831 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728011830 con 0x7fb73415d9d0 2024-03-29T10:14:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.650+0000 7fb733400700 1 -- 172.21.15.12:0/687374831 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb728011990 con 0x7fb73415d9d0 2024-03-29T10:14:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.650+0000 7fb733400700 1 -- 172.21.15.12:0/687374831 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728011d20 con 0x7fb73415d9d0 2024-03-29T10:14:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.650+0000 7fb73b400700 1 -- 172.21.15.12:0/687374831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 msgr2=0x7fb7340d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.650+0000 7fb73b400700 1 --2- 172.21.15.12:0/687374831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 0x7fb7340d3c30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb7280098f0 tx=0x7fb728009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.650+0000 7fb73b400700 1 -- 172.21.15.12:0/687374831 shutdown_connections 2024-03-29T10:14:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.650+0000 7fb73b400700 1 --2- 172.21.15.12:0/687374831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 0x7fb7340d3c30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.650+0000 7fb73b400700 1 --2- 172.21.15.12:0/687374831 >> v2:172.21.15.159:3300/0 conn(0x7fb7340d16a0 0x7fb7340d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.650+0000 7fb73b400700 1 -- 172.21.15.12:0/687374831 >> 172.21.15.12:0/687374831 conn(0x7fb73408aff0 msgr2=0x7fb7340d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.650+0000 7fb73b400700 1 -- 172.21.15.12:0/687374831 shutdown_connections 2024-03-29T10:14:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.651+0000 7fb73b400700 1 -- 172.21.15.12:0/687374831 wait complete. 2024-03-29T10:14:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.652+0000 7fb73b400700 1 Processor -- start 2024-03-29T10:14:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.652+0000 7fb73b400700 1 -- start start 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.653+0000 7fb73b400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7340d16a0 0x7fb7341556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.653+0000 7fb73b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7340d2370 0x7fb7340b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.653+0000 7fb73b400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 0x7fb7340b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.653+0000 7fb73b400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb734155bf0 con 0x7fb7340d2370 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.653+0000 7fb73b400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340b8050 con 0x7fb7340d16a0 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.653+0000 7fb73b400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340b73d0 con 0x7fb73415d9d0 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb733e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7340d2370 0x7fb7340b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb738a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7340d16a0 0x7fb7341556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb733e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7340d2370 0x7fb7340b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35902/0 (socket says 172.21.15.12:35902) 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb733e00700 1 -- 172.21.15.12:0/2914724276 learned_addr learned my addr 172.21.15.12:0/2914724276 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb739400700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 0x7fb7340b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb733e00700 1 -- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 msgr2=0x7fb7340b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb733e00700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 0x7fb7340b4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb733e00700 1 -- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7340d16a0 msgr2=0x7fb7341556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb733e00700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7340d16a0 0x7fb7341556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb733e00700 1 -- 172.21.15.12:0/2914724276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb728009550 con 0x7fb7340d2370 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb738a00700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7340d16a0 0x7fb7341556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.654+0000 7fb733e00700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7340d2370 0x7fb7340b4a00 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fb72000e960 tx=0x7fb72000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.655+0000 7fb730c00700 1 -- 172.21.15.12:0/2914724276 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7200096f0 con 0x7fb7340d2370 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.655+0000 7fb730c00700 1 -- 172.21.15.12:0/2914724276 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb720013680 con 0x7fb7340d2370 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.655+0000 7fb73b400700 1 -- 172.21.15.12:0/2914724276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7340b76b0 con 0x7fb7340d2370 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.655+0000 7fb730c00700 1 -- 172.21.15.12:0/2914724276 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720013a30 con 0x7fb7340d2370 2024-03-29T10:14:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.655+0000 7fb73b400700 1 -- 172.21.15.12:0/2914724276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7340b7c70 con 0x7fb7340d2370 2024-03-29T10:14:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.656+0000 7fb73b400700 1 -- 172.21.15.12:0/2914724276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7340d1ca0 con 0x7fb7340d2370 2024-03-29T10:14:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.657+0000 7fb730c00700 1 -- 172.21.15.12:0/2914724276 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fb720013b90 con 0x7fb7340d2370 2024-03-29T10:14:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.658+0000 7fb730c00700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb71406ae40 0x7fb71406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.658+0000 7fb730c00700 1 -- 172.21.15.12:0/2914724276 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fb720089af0 con 0x7fb7340d2370 2024-03-29T10:14:52.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.659+0000 7fb738a00700 1 -- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb71406ae40 msgr2=0x7fb71406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:52.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.659+0000 7fb738a00700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb71406ae40 0x7fb71406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:52.658 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:52.660+0000 7fb8a456a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:14:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.662+0000 7fb730c00700 1 -- 172.21.15.12:0/2914724276 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7200548f0 con 0x7fb7340d2370 2024-03-29T10:14:52.743 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:52.745+0000 7fc7e1153240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:14:52.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.860+0000 7fb738a00700 1 -- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb71406ae40 msgr2=0x7fb71406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:52.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.860+0000 7fb738a00700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb71406ae40 0x7fb71406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:52.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.876+0000 7fb73b400700 1 -- 172.21.15.12:0/2914724276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb734067720 con 0x7fb7340d2370 2024-03-29T10:14:52.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.877+0000 7fb730c00700 1 -- 172.21.15.12:0/2914724276 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb720021020 con 0x7fb7340d2370 2024-03-29T10:14:52.875 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:14:52.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.879+0000 7fb73b400700 1 -- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb71406ae40 msgr2=0x7fb71406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:52.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.879+0000 7fb73b400700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb71406ae40 0x7fb71406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.879+0000 7fb73b400700 1 -- 172.21.15.12:0/2914724276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7340d2370 msgr2=0x7fb7340b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:52.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.879+0000 7fb73b400700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7340d2370 0x7fb7340b4a00 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fb72000e960 tx=0x7fb72000ec70 comp rx=0 tx=0).stop 2024-03-29T10:14:52.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.880+0000 7fb73b400700 1 -- 172.21.15.12:0/2914724276 shutdown_connections 2024-03-29T10:14:52.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.880+0000 7fb73b400700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb71406ae40 0x7fb71406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.880+0000 7fb73b400700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb73415d9d0 0x7fb7340b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.880+0000 7fb73b400700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7340d16a0 0x7fb7341556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.880+0000 7fb73b400700 1 --2- 172.21.15.12:0/2914724276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7340d2370 0x7fb7340b4a00 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:52.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.880+0000 7fb73b400700 1 -- 172.21.15.12:0/2914724276 >> 172.21.15.12:0/2914724276 conn(0x7fb73408aff0 msgr2=0x7fb73415bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:52.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.880+0000 7fb73b400700 1 -- 172.21.15.12:0/2914724276 shutdown_connections 2024-03-29T10:14:52.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:52.880+0000 7fb73b400700 1 -- 172.21.15.12:0/2914724276 wait complete. 2024-03-29T10:14:52.897 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:14:53.121 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:53.122+0000 7fb8a456a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:14:53.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.145+0000 7f8b9e600700 1 Processor -- start 2024-03-29T10:14:53.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.145+0000 7f8b9e600700 1 -- start start 2024-03-29T10:14:53.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.146+0000 7f8b9e600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8b980d21f0 0x7f8b980ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.146+0000 7f8b9e600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980cec40 0x7f8b98161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.146+0000 7f8b9e600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980ce270 con 0x7f8b981611b0 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.146+0000 7f8b9e600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b9815b600 con 0x7f8b980d21f0 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.146+0000 7f8b9e600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980d2760 con 0x7f8b980cec40 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.146+0000 7f8b8fe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8b980d21f0 0x7f8b980ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b97e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980cec40 0x7f8b98161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b97400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f8b981611b0 0x7f8b9815ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:59886/0 (socket says 172.21.15.12:59886) 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b8fe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8b980d21f0 0x7f8b980ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:56844/0 (socket says 172.21.15.12:56844) 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b97e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980cec40 0x7f8b98161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:56828/0 (socket says 172.21.15.12:56828) 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b97400700 1 -- 172.21.15.12:0/2025721687 learned_addr learned my addr 172.21.15.12:0/2025721687 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b96a00700 1 -- 172.21.15.12:0/2025721687 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976840716 0 0) 0x7f8b980ce270 con 0x7f8b981611b0 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b96a00700 1 -- 172.21.15.12:0/2025721687 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b7c0035b0 con 0x7f8b981611b0 2024-03-29T10:14:53.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b97e00700 1 -- 172.21.15.12:0/2025721687 >> v2:172.21.15.159:3300/0 conn(0x7f8b980d21f0 msgr2=0x7f8b980ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:53.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b97e00700 1 --2- 172.21.15.12:0/2025721687 >> v2:172.21.15.159:3300/0 conn(0x7f8b980d21f0 0x7f8b980ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b97e00700 1 -- 172.21.15.12:0/2025721687 >> v1:172.21.15.159:6789/0 conn(0x7f8b981611b0 legacy=0x7f8b9815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b97e00700 1 -- 172.21.15.12:0/2025721687 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b980d2a40 con 0x7f8b980cec40 2024-03-29T10:14:53.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b8fe00700 1 --2- 172.21.15.12:0/2025721687 >> v2:172.21.15.159:3300/0 conn(0x7f8b980d21f0 0x7f8b980ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:53.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.147+0000 7f8b97e00700 1 --2- 172.21.15.12:0/2025721687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980cec40 0x7f8b98161590 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009d90 tx=0x7f8b9000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bea3765ac0efcd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:53.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.148+0000 7f8b96a00700 1 -- 172.21.15.12:0/2025721687 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9000e070 con 0x7f8b980cec40 2024-03-29T10:14:53.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.148+0000 7f8b96a00700 1 -- 172.21.15.12:0/2025721687 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9000bb20 con 0x7f8b980cec40 2024-03-29T10:14:53.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.148+0000 7f8b96a00700 1 -- 172.21.15.12:0/2025721687 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90012430 con 0x7f8b980cec40 2024-03-29T10:14:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.148+0000 7f8b9e600700 1 -- 172.21.15.12:0/2025721687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980cec40 msgr2=0x7f8b98161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.148+0000 7f8b9e600700 1 --2- 172.21.15.12:0/2025721687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980cec40 0x7f8b98161590 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009d90 tx=0x7f8b9000b330 comp rx=0 tx=0).stop 2024-03-29T10:14:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.149+0000 7f8b9e600700 1 -- 172.21.15.12:0/2025721687 shutdown_connections 2024-03-29T10:14:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.149+0000 7f8b9e600700 1 --2- 172.21.15.12:0/2025721687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980cec40 0x7f8b98161590 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.149+0000 7f8b9e600700 1 --2- 172.21.15.12:0/2025721687 >> v2:172.21.15.159:3300/0 conn(0x7f8b980d21f0 0x7f8b980ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.149+0000 7f8b9e600700 1 -- 172.21.15.12:0/2025721687 >> 172.21.15.12:0/2025721687 conn(0x7f8b9808aff0 msgr2=0x7f8b980d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:53.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.149+0000 7f8b9e600700 1 -- 172.21.15.12:0/2025721687 shutdown_connections 2024-03-29T10:14:53.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.149+0000 7f8b9e600700 1 -- 172.21.15.12:0/2025721687 wait complete. 2024-03-29T10:14:53.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.150+0000 7f8b9e600700 1 Processor -- start 2024-03-29T10:14:53.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.151+0000 7f8b9e600700 1 -- start start 2024-03-29T10:14:53.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.151+0000 7f8b9e600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b980cec40 0x7f8b980b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.151+0000 7f8b9e600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980d21f0 0x7f8b980cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.151+0000 7f8b9e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b981611b0 0x7f8b981f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.151+0000 7f8b9e600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980b6150 con 0x7f8b981611b0 2024-03-29T10:14:53.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.151+0000 7f8b9e600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98154f00 con 0x7f8b980cec40 2024-03-29T10:14:53.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.151+0000 7f8b9e600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980cb4a0 con 0x7f8b980d21f0 2024-03-29T10:14:53.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.152+0000 7f8b97e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b981611b0 0x7f8b981f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.152+0000 7f8b97e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b981611b0 0x7f8b981f64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35904/0 (socket says 172.21.15.12:35904) 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.152+0000 7f8b97e00700 1 -- 172.21.15.12:0/868929320 learned_addr learned my addr 172.21.15.12:0/868929320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.152+0000 7f8b97400700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b980cec40 0x7f8b980b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.152+0000 7f8b8fe00700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980d21f0 0x7f8b980cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.152+0000 7f8b97e00700 1 -- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980d21f0 msgr2=0x7f8b980cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.152+0000 7f8b97e00700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980d21f0 0x7f8b980cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.152+0000 7f8b97e00700 1 -- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b980cec40 msgr2=0x7f8b980b8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.152+0000 7f8b97e00700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b980cec40 0x7f8b980b8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.152+0000 7f8b97e00700 1 -- 172.21.15.12:0/868929320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b800094a0 con 0x7f8b981611b0 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.153+0000 7f8b97e00700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b981611b0 0x7f8b981f64a0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8b90014040 tx=0x7f8b90004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.153+0000 7f8b8f400700 1 -- 172.21.15.12:0/868929320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b900129f0 con 0x7f8b981611b0 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.153+0000 7f8b8f400700 1 -- 172.21.15.12:0/868929320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b90020070 con 0x7f8b981611b0 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.153+0000 7f8b9e600700 1 -- 172.21.15.12:0/868929320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b90009a70 con 0x7f8b981611b0 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.153+0000 7f8b8f400700 1 -- 172.21.15.12:0/868929320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90012db0 con 0x7f8b981611b0 2024-03-29T10:14:53.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.153+0000 7f8b9e600700 1 -- 172.21.15.12:0/868929320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b98154b90 con 0x7f8b981611b0 2024-03-29T10:14:53.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.154+0000 7f8b9e600700 1 -- 172.21.15.12:0/868929320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b98155070 con 0x7f8b981611b0 2024-03-29T10:14:53.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.155+0000 7f8b8f400700 1 -- 172.21.15.12:0/868929320 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f8b90004040 con 0x7f8b981611b0 2024-03-29T10:14:53.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.156+0000 7f8b8f400700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8b7406ae40 0x7f8b7406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.156+0000 7f8b8f400700 1 -- 172.21.15.12:0/868929320 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f8b90022070 con 0x7f8b981611b0 2024-03-29T10:14:53.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.156+0000 7f8b97400700 1 -- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8b7406ae40 msgr2=0x7f8b7406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:53.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.156+0000 7f8b97400700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8b7406ae40 0x7f8b7406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.164+0000 7f8b8f400700 1 -- 172.21.15.12:0/868929320 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8b90056510 con 0x7f8b981611b0 2024-03-29T10:14:53.234 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:53.236+0000 7fc7e1153240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:14:53.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.357+0000 7f8b97400700 1 -- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8b7406ae40 msgr2=0x7f8b7406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:53.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.357+0000 7f8b97400700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8b7406ae40 0x7f8b7406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.378+0000 7f8b9e600700 1 -- 172.21.15.12:0/868929320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8b98067720 con 0x7f8b981611b0 2024-03-29T10:14:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.379+0000 7f8b8f400700 1 -- 172.21.15.12:0/868929320 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8b90059b30 con 0x7f8b981611b0 2024-03-29T10:14:53.377 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:14:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.381+0000 7f8b9e600700 1 -- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8b7406ae40 msgr2=0x7f8b7406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.381+0000 7f8b9e600700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8b7406ae40 0x7f8b7406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.381+0000 7f8b9e600700 1 -- 172.21.15.12:0/868929320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b981611b0 msgr2=0x7f8b981f64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.381+0000 7f8b9e600700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b981611b0 0x7f8b981f64a0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8b90014040 tx=0x7f8b90004ad0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.381+0000 7f8b9e600700 1 -- 172.21.15.12:0/868929320 shutdown_connections 2024-03-29T10:14:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.381+0000 7f8b9e600700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8b7406ae40 0x7f8b7406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.381+0000 7f8b9e600700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b981611b0 0x7f8b981f64a0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.381+0000 7f8b9e600700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b980cec40 0x7f8b980b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.381+0000 7f8b9e600700 1 --2- 172.21.15.12:0/868929320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b980d21f0 0x7f8b980cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.381+0000 7f8b9e600700 1 -- 172.21.15.12:0/868929320 >> 172.21.15.12:0/868929320 conn(0x7f8b9808aff0 msgr2=0x7f8b9806d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.382+0000 7f8b9e600700 1 -- 172.21.15.12:0/868929320 shutdown_connections 2024-03-29T10:14:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.382+0000 7f8b9e600700 1 -- 172.21.15.12:0/868929320 wait complete. 2024-03-29T10:14:53.398 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T10:14:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.600+0000 7f3eed600700 1 Processor -- start 2024-03-29T10:14:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.600+0000 7f3eed600700 1 -- start start 2024-03-29T10:14:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.601+0000 7f3eed600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3ee80cd480 0x7f3ee806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.601+0000 7f3eed600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee80cdae0 con 0x7f3ee80cbb30 2024-03-29T10:14:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3eed600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee80cddc0 con 0x7f3ee80cd020 2024-03-29T10:14:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3eed600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee80ce0a0 con 0x7f3ee80cd480 2024-03-29T10:14:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3ee6a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3ee80cbb30 0x7f3ee80cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52330/0 (socket says 172.21.15.12:52330) 2024-03-29T10:14:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3ee6a00700 1 -- 172.21.15.12:0/2381228864 learned_addr learned my addr 172.21.15.12:0/2381228864 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3ee7400700 1 --2- 172.21.15.12:0/2381228864 >> v2:172.21.15.159:3300/0 conn(0x7f3ee80cd480 0x7f3ee806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483881598 0 0) 0x7f3ee80cdae0 con 0x7f3ee80cbb30 2024-03-29T10:14:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ecc0035b0 con 0x7f3ee80cbb30 2024-03-29T10:14:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979635 0 0) 0x7f3ee80cddc0 con 0x7f3ee80cd020 2024-03-29T10:14:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ee80cdae0 con 0x7f3ee80cd020 2024-03-29T10:14:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 641719982 0 0) 0x7f3ecc0035b0 con 0x7f3ee80cbb30 2024-03-29T10:14:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 >> v2:172.21.15.159:3300/0 conn(0x7f3ee80cd480 msgr2=0x7f3ee806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.602+0000 7f3ee5600700 1 --2- 172.21.15.12:0/2381228864 >> v2:172.21.15.159:3300/0 conn(0x7f3ee80cd480 0x7f3ee806e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 >> v1:172.21.15.159:6789/0 conn(0x7f3ee80cd020 legacy=0x7f3ee80b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee806dc20 con 0x7f3ee80cbb30 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f3ed8001670 con 0x7f3ee80cbb30 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee7400700 1 --2- 172.21.15.12:0/2381228864 >> v2:172.21.15.159:3300/0 conn(0x7f3ee80cd480 0x7f3ee806e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 >> v1:172.21.15.12:6789/0 conn(0x7f3ee80cbb30 legacy=0x7f3ee80cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee5600700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc003b90 0x7f3ecc003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee5600700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ecc004850 0x7f3ecc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee5600700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ecc005210 0x7f3ecc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80ce0a0 con 0x7f3ecc003b90 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc00c090 con 0x7f3ecc004850 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc00c250 con 0x7f3ecc005210 2024-03-29T10:14:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee6a00700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc003b90 0x7f3ecc003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee7400700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ecc005210 0x7f3ecc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee6000700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ecc004850 0x7f3ecc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee6a00700 1 -- 172.21.15.12:0/2381228864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ecc005210 msgr2=0x7f3ecc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee6a00700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ecc005210 0x7f3ecc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee6a00700 1 -- 172.21.15.12:0/2381228864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ecc004850 msgr2=0x7f3ecc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee6a00700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ecc004850 0x7f3ecc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee6a00700 1 -- 172.21.15.12:0/2381228864 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc0035b0 con 0x7f3ecc003b90 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.603+0000 7f3ee7400700 1 -- 172.21.15.12:0/2381228864 reap_dead start 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.604+0000 7f3ee6a00700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc003b90 0x7f3ecc003f90 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c210 tx=0x7f3ecc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3cbcbbfe269736 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.604+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed800cdb0 con 0x7f3ecc003b90 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.604+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed8005960 con 0x7f3ecc003b90 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.604+0000 7f3ee5600700 1 -- 172.21.15.12:0/2381228864 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8005d10 con 0x7f3ecc003b90 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.605+0000 7f3eed600700 1 -- 172.21.15.12:0/2381228864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc003b90 msgr2=0x7f3ecc003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.605+0000 7f3eed600700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc003b90 0x7f3ecc003f90 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c210 tx=0x7f3ecc005880 comp rx=0 tx=0).stop 2024-03-29T10:14:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.605+0000 7f3eed600700 1 -- 172.21.15.12:0/2381228864 shutdown_connections 2024-03-29T10:14:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.605+0000 7f3eed600700 1 --2- 172.21.15.12:0/2381228864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ecc003b90 0x7f3ecc003f90 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.605+0000 7f3eed600700 1 -- 172.21.15.12:0/2381228864 >> 172.21.15.12:0/2381228864 conn(0x7f3ee80b0d70 msgr2=0x7f3ee80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.605+0000 7f3eed600700 1 -- 172.21.15.12:0/2381228864 shutdown_connections 2024-03-29T10:14:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.605+0000 7f3eed600700 1 -- 172.21.15.12:0/2381228864 wait complete. 2024-03-29T10:14:53.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.607+0000 7f3eed600700 1 Processor -- start 2024-03-29T10:14:53.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.607+0000 7f3eed600700 1 -- start start 2024-03-29T10:14:53.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3eed600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ecc003b90 0x7f3ee8155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3eed600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee80718b0 0x7f3ee80b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3eed600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ee8071d30 0x7f3ee815a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3eed600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80ca140 con 0x7f3ee80718b0 2024-03-29T10:14:53.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3eed600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80c9530 con 0x7f3ecc003b90 2024-03-29T10:14:53.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3eed600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80c9810 con 0x7f3ee8071d30 2024-03-29T10:14:53.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3ee6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee80718b0 0x7f3ee80b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3ee6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee80718b0 0x7f3ee80b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35922/0 (socket says 172.21.15.12:35922) 2024-03-29T10:14:53.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3ee6000700 1 -- 172.21.15.12:0/1007601542 learned_addr learned my addr 172.21.15.12:0/1007601542 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:53.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3ee7400700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ee8071d30 0x7f3ee815a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.608+0000 7f3ee6a00700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ecc003b90 0x7f3ee8155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:53.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3ee6000700 1 -- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ee8071d30 msgr2=0x7f3ee815a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3ee6000700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ee8071d30 0x7f3ee815a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3ee6000700 1 -- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ecc003b90 msgr2=0x7f3ee8155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3ee6000700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ecc003b90 0x7f3ee8155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3ee6000700 1 -- 172.21.15.12:0/1007601542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee80c9af0 con 0x7f3ee80718b0 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3ee7400700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ee8071d30 0x7f3ee815a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3ee6a00700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ecc003b90 0x7f3ee8155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3ee6000700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee80718b0 0x7f3ee80b1d90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400e580 tx=0x7f3ed400e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3edea00700 1 -- 172.21.15.12:0/1007601542 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40052e0 con 0x7f3ee80718b0 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3edea00700 1 -- 172.21.15.12:0/1007601542 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed4005440 con 0x7f3ee80718b0 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3edea00700 1 -- 172.21.15.12:0/1007601542 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40038c0 con 0x7f3ee80718b0 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3eed600700 1 -- 172.21.15.12:0/1007601542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee80c9dd0 con 0x7f3ee80718b0 2024-03-29T10:14:53.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.609+0000 7f3eed600700 1 -- 172.21.15.12:0/1007601542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee8155c80 con 0x7f3ee80718b0 2024-03-29T10:14:53.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.610+0000 7f3eed600700 1 -- 172.21.15.12:0/1007601542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee8067720 con 0x7f3ee80718b0 2024-03-29T10:14:53.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.611+0000 7f3edea00700 1 -- 172.21.15.12:0/1007601542 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f3ed4003a20 con 0x7f3ee80718b0 2024-03-29T10:14:53.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.612+0000 7f3edea00700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3ec806ae40 0x7f3ec806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:53.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.612+0000 7f3edea00700 1 -- 172.21.15.12:0/1007601542 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f3ed4088f30 con 0x7f3ee80718b0 2024-03-29T10:14:53.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.612+0000 7f3ee6a00700 1 -- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3ec806ae40 msgr2=0x7f3ec806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:53.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.612+0000 7f3ee6a00700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3ec806ae40 0x7f3ec806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:53.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.621+0000 7f3edea00700 1 -- 172.21.15.12:0/1007601542 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ed4053c90 con 0x7f3ee80718b0 2024-03-29T10:14:53.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.813+0000 7f3ee6a00700 1 -- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3ec806ae40 msgr2=0x7f3ec806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:53.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.813+0000 7f3ee6a00700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3ec806ae40 0x7f3ec806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:53.835 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:53.837+0000 7fb8a456a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:14:53.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.854+0000 7f3eed600700 1 -- 172.21.15.12:0/1007601542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3ee80ca280 con 0x7f3ee80718b0 2024-03-29T10:14:53.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.855+0000 7f3edea00700 1 -- 172.21.15.12:0/1007601542 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v141) v1 ==== 80+0+468 (secure 0 0 0) 0x7f3ed4019070 con 0x7f3ee80718b0 2024-03-29T10:14:53.853 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3261410285 2024-03-30T10:14:42.823243+0000 2024-03-29T10:14:53.853 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2027263440 2024-03-30T10:14:42.823243+0000 2024-03-29T10:14:53.853 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/4127686095 2024-03-30T10:14:42.823243+0000 2024-03-29T10:14:53.853 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/43444 2024-03-30T10:14:42.823243+0000 2024-03-29T10:14:53.853 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/43444 2024-03-30T10:14:42.823243+0000 2024-03-29T10:14:53.853 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3727946147 2024-03-30T10:14:42.823243+0000 2024-03-29T10:14:53.853 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2214680226 2024-03-30T10:14:42.823243+0000 2024-03-29T10:14:53.853 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2899228198 2024-03-30T10:14:42.823243+0000 2024-03-29T10:14:53.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 -- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3ec806ae40 msgr2=0x7f3ec806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:53.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3ec806ae40 0x7f3ec806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 -- 172.21.15.12:0/1007601542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee80718b0 msgr2=0x7f3ee80b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee80718b0 0x7f3ee80b1d90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400e580 tx=0x7f3ed400e890 comp rx=0 tx=0).stop 2024-03-29T10:14:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 -- 172.21.15.12:0/1007601542 shutdown_connections 2024-03-29T10:14:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3ec806ae40 0x7f3ec806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ee8071d30 0x7f3ee815a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ecc003b90 0x7f3ee8155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 --2- 172.21.15.12:0/1007601542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee80718b0 0x7f3ee80b1d90 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 -- 172.21.15.12:0/1007601542 >> 172.21.15.12:0/1007601542 conn(0x7f3ee80b0d70 msgr2=0x7f3ee815d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 -- 172.21.15.12:0/1007601542 shutdown_connections 2024-03-29T10:14:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:53.857+0000 7f3eed600700 1 -- 172.21.15.12:0/1007601542 wait complete. 2024-03-29T10:14:53.857 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T10:14:53.873 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T10:14:53.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:53.874 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T10:14:53.929 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:53.931+0000 7fb8a456a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:14:53.953 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T10:14:53.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:14:53.953 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T10:14:54.000 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:14:54.000 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:54.009 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:54.011+0000 7fc7e1153240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:14:54.050 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:14:54.050 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:14:54.050 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:54.095 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:14:54.095 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:14:54.095 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:14:54.100 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:54.101+0000 7fb8a456a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:14:54.108 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:54.109+0000 7fc7e1153240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:14:54.141 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:14:54.141 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:54.280 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:54.282+0000 7fc7e1153240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:14:54.318 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:54.320+0000 7fb8a456a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:14:54.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.344+0000 7f82f2a00700 1 Processor -- start 2024-03-29T10:14:54.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.345+0000 7f82f2a00700 1 -- start start 2024-03-29T10:14:54.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.346+0000 7f82f2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f82ec15b5d0 0x7f82ec0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:54.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.346+0000 7f82f2a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec15fd50 con 0x7f82ec0cb3f0 2024-03-29T10:14:54.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.346+0000 7f82f2a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec15f140 con 0x7f82ec0ac7d0 2024-03-29T10:14:54.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.346+0000 7f82f2a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ec15b9b0 con 0x7f82ec15b5d0 2024-03-29T10:14:54.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.346+0000 7f82ebe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f82ec15b5d0 0x7f82ec0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:54.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.346+0000 7f82eb400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f82ec0cb3f0 0x7f82ec0ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52346/0 (socket says 172.21.15.12:52346) 2024-03-29T10:14:54.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.346+0000 7f82eb400700 1 -- 172.21.15.12:0/1181562877 learned_addr learned my addr 172.21.15.12:0/1181562877 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:54.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.346+0000 7f82ebe00700 1 -- 172.21.15.12:0/1181562877 >> v1:172.21.15.159:6789/0 conn(0x7f82ec0ac7d0 legacy=0x7f82ec06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.346+0000 7f82ebe00700 1 -- 172.21.15.12:0/1181562877 >> v1:172.21.15.12:6789/0 conn(0x7f82ec0cb3f0 legacy=0x7f82ec0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.347+0000 7f82ebe00700 1 -- 172.21.15.12:0/1181562877 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec15f420 con 0x7f82ec15b5d0 2024-03-29T10:14:54.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.347+0000 7f82eaa00700 1 -- 172.21.15.12:0/1181562877 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713026575 0 0) 0x7f82ec15fd50 con 0x7f82ec0cb3f0 2024-03-29T10:14:54.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.347+0000 7f82ebe00700 1 --2- 172.21.15.12:0/1181562877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 0x7f82ec0ae3f0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009840 tx=0x7f82dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b00d6105bca0846 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:54.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.347+0000 7f82eaa00700 1 -- 172.21.15.12:0/1181562877 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc011830 con 0x7f82ec15b5d0 2024-03-29T10:14:54.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.347+0000 7f82eaa00700 1 -- 172.21.15.12:0/1181562877 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc011990 con 0x7f82ec15b5d0 2024-03-29T10:14:54.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.347+0000 7f82eaa00700 1 -- 172.21.15.12:0/1181562877 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc011d20 con 0x7f82ec15b5d0 2024-03-29T10:14:54.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.348+0000 7f82f2a00700 1 -- 172.21.15.12:0/1181562877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 msgr2=0x7f82ec0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.348+0000 7f82f2a00700 1 --2- 172.21.15.12:0/1181562877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 0x7f82ec0ae3f0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009840 tx=0x7f82dc009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:54.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.348+0000 7f82f2a00700 1 -- 172.21.15.12:0/1181562877 shutdown_connections 2024-03-29T10:14:54.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.348+0000 7f82f2a00700 1 --2- 172.21.15.12:0/1181562877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 0x7f82ec0ae3f0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.348+0000 7f82f2a00700 1 -- 172.21.15.12:0/1181562877 >> 172.21.15.12:0/1181562877 conn(0x7f82ec0cf190 msgr2=0x7f82ec0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:54.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.348+0000 7f82f2a00700 1 -- 172.21.15.12:0/1181562877 shutdown_connections 2024-03-29T10:14:54.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.348+0000 7f82f2a00700 1 -- 172.21.15.12:0/1181562877 wait complete. 2024-03-29T10:14:54.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.350+0000 7f82f2a00700 1 Processor -- start 2024-03-29T10:14:54.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.350+0000 7f82f2a00700 1 -- start start 2024-03-29T10:14:54.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82f2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82ec0ac7d0 0x7f82ec1f71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:54.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82f2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 0x7f82ec071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82f2a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82ec1f0240 0x7f82ec1f06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82f2a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec1f7730 con 0x7f82ec15b5d0 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82f2a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec1f0c30 con 0x7f82ec1f0240 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82f2a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec1f4180 con 0x7f82ec0ac7d0 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82eb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 0x7f82ec071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82eb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 0x7f82ec071c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35948/0 (socket says 172.21.15.12:35948) 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82eb400700 1 -- 172.21.15.12:0/3845715664 learned_addr learned my addr 172.21.15.12:0/3845715664 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82f0a00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82ec1f0240 0x7f82ec1f06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82ebe00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82ec0ac7d0 0x7f82ec1f71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82eb400700 1 -- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82ec0ac7d0 msgr2=0x7f82ec1f71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82eb400700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82ec0ac7d0 0x7f82ec1f71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82eb400700 1 -- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82ec1f0240 msgr2=0x7f82ec1f06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82eb400700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82ec1f0240 0x7f82ec1f06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.351+0000 7f82eb400700 1 -- 172.21.15.12:0/3845715664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc0094a0 con 0x7f82ec15b5d0 2024-03-29T10:14:54.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.352+0000 7f82eb400700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 0x7f82ec071c80 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f82d800c520 tx=0x7f82d800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:54.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.352+0000 7f82e3e00700 1 -- 172.21.15.12:0/3845715664 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8004c80 con 0x7f82ec15b5d0 2024-03-29T10:14:54.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.352+0000 7f82e3e00700 1 -- 172.21.15.12:0/3845715664 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d8004e00 con 0x7f82ec15b5d0 2024-03-29T10:14:54.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.352+0000 7f82e3e00700 1 -- 172.21.15.12:0/3845715664 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8007310 con 0x7f82ec15b5d0 2024-03-29T10:14:54.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.352+0000 7f82f2a00700 1 -- 172.21.15.12:0/3845715664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec1f3600 con 0x7f82ec15b5d0 2024-03-29T10:14:54.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.352+0000 7f82f2a00700 1 -- 172.21.15.12:0/3845715664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec1f3bc0 con 0x7f82ec15b5d0 2024-03-29T10:14:54.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.353+0000 7f82f2a00700 1 -- 172.21.15.12:0/3845715664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec0cba10 con 0x7f82ec15b5d0 2024-03-29T10:14:54.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.354+0000 7f82e3e00700 1 -- 172.21.15.12:0/3845715664 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f82d8007470 con 0x7f82ec15b5d0 2024-03-29T10:14:54.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.356+0000 7f82e3e00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82cc06ae40 0x7f82cc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:54.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.356+0000 7f82e3e00700 1 -- 172.21.15.12:0/3845715664 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f82d80892d0 con 0x7f82ec15b5d0 2024-03-29T10:14:54.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.356+0000 7f82ebe00700 1 -- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82cc06ae40 msgr2=0x7f82cc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:54.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.356+0000 7f82ebe00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82cc06ae40 0x7f82cc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:54.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.360+0000 7f82e3e00700 1 -- 172.21.15.12:0/3845715664 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82d8054160 con 0x7f82ec15b5d0 2024-03-29T10:14:54.413 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:54.414+0000 7fb8a456a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:14:54.503 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:54.505+0000 7fc7e1153240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:14:54.510 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:54.512+0000 7fb8a456a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:14:54.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.557+0000 7f82ebe00700 1 -- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82cc06ae40 msgr2=0x7f82cc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:54.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.557+0000 7f82ebe00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82cc06ae40 0x7f82cc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:54.600 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:54.602+0000 7fc7e1153240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:14:54.610 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:54.612+0000 7fb8a456a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:14:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.621+0000 7f82f2a00700 1 -- 172.21.15.12:0/3845715664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82ec067720 con 0x7f82ec15b5d0 2024-03-29T10:14:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.622+0000 7f82e3e00700 1 -- 172.21.15.12:0/3845715664 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f82d8057780 con 0x7f82ec15b5d0 2024-03-29T10:14:54.620 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:54.620 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1317631939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1317631939},{"type":"v1","addr":"172.21.15.159:6836","nonce":1317631939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1057731647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1057731647},{"type":"v1","addr":"172.21.15.159:6837","nonce":1057731647}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1567088347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1567088347},{"type":"v1","addr":"172.21.15.12:6834","nonce":1567088347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:14:25.846391+0000","modified":"2024-03-29T10:14:29.114278+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8406":{"gid":8406,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3311625077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3311625077},{"type":"v1","addr":"172.21.15.12:6835","nonce":3311625077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-29T10:14:54.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 -- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82cc06ae40 msgr2=0x7f82cc06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:54.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82cc06ae40 0x7f82cc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 -- 172.21.15.12:0/3845715664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 msgr2=0x7f82ec071c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 0x7f82ec071c80 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f82d800c520 tx=0x7f82d800c830 comp rx=0 tx=0).stop 2024-03-29T10:14:54.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 -- 172.21.15.12:0/3845715664 shutdown_connections 2024-03-29T10:14:54.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82cc06ae40 0x7f82cc06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82ec1f0240 0x7f82ec1f06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82ec0ac7d0 0x7f82ec1f71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 --2- 172.21.15.12:0/3845715664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82ec15b5d0 0x7f82ec071c80 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 -- 172.21.15.12:0/3845715664 >> 172.21.15.12:0/3845715664 conn(0x7f82ec0cf190 msgr2=0x7f82ec15e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 -- 172.21.15.12:0/3845715664 shutdown_connections 2024-03-29T10:14:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.624+0000 7f82f2a00700 1 -- 172.21.15.12:0/3845715664 wait complete. 2024-03-29T10:14:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 64 2024-03-29T10:14:54.640 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:54.708 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:54.710+0000 7fc7e1153240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:14:54.709 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:54.710+0000 7fb8a456a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:14:54.814 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:54.816+0000 7fc7e1153240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:14:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.842+0000 7f35ace00700 1 Processor -- start 2024-03-29T10:14:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.843+0000 7f35ace00700 1 -- start start 2024-03-29T10:14:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.844+0000 7f35ace00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35a80d05e0 0x7f35a80d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.844+0000 7f35ace00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a80d0f80 con 0x7f35a815da50 2024-03-29T10:14:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.844+0000 7f35ace00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a80d24e0 con 0x7f35a80b6990 2024-03-29T10:14:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.844+0000 7f35ace00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80b95d0 con 0x7f35a80d05e0 2024-03-29T10:14:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.844+0000 7f35a5400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35a80d05e0 0x7f35a80d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.844+0000 7f35a5400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35a80d05e0 0x7f35a80d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35954/0 (socket says 172.21.15.12:35954) 2024-03-29T10:14:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.844+0000 7f35a5e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f35a815da50 0x7f35a80ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52352/0 (socket says 172.21.15.12:52352) 2024-03-29T10:14:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.844+0000 7f35a5400700 1 -- 172.21.15.12:0/223147956 learned_addr learned my addr 172.21.15.12:0/223147956 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.845+0000 7f35a5400700 1 -- 172.21.15.12:0/223147956 >> v1:172.21.15.159:6789/0 conn(0x7f35a80b6990 legacy=0x7f35a80b6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.845+0000 7f35a5400700 1 -- 172.21.15.12:0/223147956 >> v1:172.21.15.12:6789/0 conn(0x7f35a815da50 legacy=0x7f35a80ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.845+0000 7f35a5400700 1 -- 172.21.15.12:0/223147956 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a80d27c0 con 0x7f35a80d05e0 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.845+0000 7f35a5400700 1 --2- 172.21.15.12:0/223147956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 0x7f35a80d0a40 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f35940098f0 tx=0x7f3594009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0b161afb84e755 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.845+0000 7f35a4a00700 1 -- 172.21.15.12:0/223147956 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359400b8a0 con 0x7f35a80d05e0 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.845+0000 7f35a4a00700 1 -- 172.21.15.12:0/223147956 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359400ba00 con 0x7f35a80d05e0 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.845+0000 7f35a4a00700 1 -- 172.21.15.12:0/223147956 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359400bd90 con 0x7f35a80d05e0 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.846+0000 7f35ace00700 1 -- 172.21.15.12:0/223147956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 msgr2=0x7f35a80d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.846+0000 7f35ace00700 1 --2- 172.21.15.12:0/223147956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 0x7f35a80d0a40 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f35940098f0 tx=0x7f3594009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.846+0000 7f35ace00700 1 -- 172.21.15.12:0/223147956 shutdown_connections 2024-03-29T10:14:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.846+0000 7f35ace00700 1 --2- 172.21.15.12:0/223147956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 0x7f35a80d0a40 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.846+0000 7f35ace00700 1 -- 172.21.15.12:0/223147956 >> 172.21.15.12:0/223147956 conn(0x7f35a80666a0 msgr2=0x7f35a8066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.846+0000 7f35ace00700 1 -- 172.21.15.12:0/223147956 shutdown_connections 2024-03-29T10:14:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.846+0000 7f35ace00700 1 -- 172.21.15.12:0/223147956 wait complete. 2024-03-29T10:14:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.848+0000 7f35ace00700 1 Processor -- start 2024-03-29T10:14:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.848+0000 7f35ace00700 1 -- start start 2024-03-29T10:14:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35ace00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35a80b6990 0x7f35a8160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35ace00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 0x7f35a80c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35ace00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35a815da50 0x7f35a81f6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35ace00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a80c9790 con 0x7f35a80d05e0 2024-03-29T10:14:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35ace00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a81f2f10 con 0x7f35a80b6990 2024-03-29T10:14:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35ace00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a80c8900 con 0x7f35a815da50 2024-03-29T10:14:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35a5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 0x7f35a80c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35a5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35a80b6990 0x7f35a8160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35a5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 0x7f35a80c83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35958/0 (socket says 172.21.15.12:35958) 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35a5400700 1 -- 172.21.15.12:0/1458855337 learned_addr learned my addr 172.21.15.12:0/1458855337 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35a6800700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35a815da50 0x7f35a81f6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35a5400700 1 -- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35a815da50 msgr2=0x7f35a81f6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35a5400700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35a815da50 0x7f35a81f6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35a5400700 1 -- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35a80b6990 msgr2=0x7f35a8160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35a5400700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35a80b6990 0x7f35a8160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.849+0000 7f35a5400700 1 -- 172.21.15.12:0/1458855337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3594009550 con 0x7f35a80d05e0 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.850+0000 7f35a5400700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 0x7f35a80c83c0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f3594009c40 tx=0x7f3594004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.850+0000 7f359e000700 1 -- 172.21.15.12:0/1458855337 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594004380 con 0x7f35a80d05e0 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.850+0000 7f359e000700 1 -- 172.21.15.12:0/1458855337 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359401f070 con 0x7f35a80d05e0 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.850+0000 7f359e000700 1 -- 172.21.15.12:0/1458855337 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35940036a0 con 0x7f35a80d05e0 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.850+0000 7f35ace00700 1 -- 172.21.15.12:0/1458855337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a80c8b80 con 0x7f35a80d05e0 2024-03-29T10:14:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.850+0000 7f35ace00700 1 -- 172.21.15.12:0/1458855337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a80c9140 con 0x7f35a80d05e0 2024-03-29T10:14:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.851+0000 7f35ace00700 1 -- 172.21.15.12:0/1458855337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a80c9940 con 0x7f35a80d05e0 2024-03-29T10:14:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.853+0000 7f359e000700 1 -- 172.21.15.12:0/1458855337 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f3594021030 con 0x7f35a80d05e0 2024-03-29T10:14:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.854+0000 7f359e000700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f358806ae40 0x7f358806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.855+0000 7f359e000700 1 -- 172.21.15.12:0/1458855337 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f359408ac30 con 0x7f35a80d05e0 2024-03-29T10:14:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.855+0000 7f35a5e00700 1 -- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f358806ae40 msgr2=0x7f358806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.855+0000 7f35a5e00700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f358806ae40 0x7f358806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:54.859+0000 7f359e000700 1 -- 172.21.15.12:0/1458855337 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f35a80c9940 con 0x7f35a80d05e0 2024-03-29T10:14:54.923 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:54.924+0000 7fb8a456a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:14:54.923 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:54.925+0000 7fc7e1153240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:14:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.056+0000 7f35a5e00700 1 -- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f358806ae40 msgr2=0x7f358806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.056+0000 7f35a5e00700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f358806ae40 0x7f358806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.077+0000 7f35ace00700 1 -- 172.21.15.12:0/1458855337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35a80c9940 con 0x7f35a80d05e0 2024-03-29T10:14:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.078+0000 7f359e000700 1 -- 172.21.15.12:0/1458855337 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16558 (secure 0 0 0) 0x7f35a80c9940 con 0x7f35a80d05e0 2024-03-29T10:14:55.076 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:55.077 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":141,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:14:42.823304+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:14:23.708181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-29T10:14:24.829152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3261410285":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2027263440":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/4127686095":"2024-03-30T10:14:42.823243+0000","172.21.15.159:6833/43444":"2024-03-30T10:14:42.823243+0000","172.21.15.159:6832/43444":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/3727946147":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2214680226":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2899228198":"2024-03-30T10:14:42.823243+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 -- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f358806ae40 msgr2=0x7f358806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f358806ae40 0x7f358806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 -- 172.21.15.12:0/1458855337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 msgr2=0x7f35a80c83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 0x7f35a80c83c0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f3594009c40 tx=0x7f3594004750 comp rx=0 tx=0).stop 2024-03-29T10:14:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 -- 172.21.15.12:0/1458855337 shutdown_connections 2024-03-29T10:14:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f358806ae40 0x7f358806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35a815da50 0x7f35a81f6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35a80b6990 0x7f35a8160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 --2- 172.21.15.12:0/1458855337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a80d05e0 0x7f35a80c83c0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 -- 172.21.15.12:0/1458855337 >> 172.21.15.12:0/1458855337 conn(0x7f35a80666a0 msgr2=0x7f35a815bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 -- 172.21.15.12:0/1458855337 shutdown_connections 2024-03-29T10:14:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.081+0000 7f35ace00700 1 -- 172.21.15.12:0/1458855337 wait complete. 2024-03-29T10:14:55.098 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:14:23.708181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:14:24.829152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:14:55.099 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:14:55.153 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:55.156+0000 7fc7e1153240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:14:55.183 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:55.185+0000 7fb8a456a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:14:55.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.309+0000 7fef0ce00700 1 Processor -- start 2024-03-29T10:14:55.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.309+0000 7fef0ce00700 1 -- start start 2024-03-29T10:14:55.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.310+0000 7fef0ce00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fef080d3f20 0x7fef080cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:55.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.311+0000 7fef0ce00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef08161230 con 0x7fef08160e50 2024-03-29T10:14:55.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.311+0000 7fef0ce00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080d2a80 con 0x7fef080b3f80 2024-03-29T10:14:55.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.311+0000 7fef0ce00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef080d43b0 con 0x7fef080d3f20 2024-03-29T10:14:55.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.311+0000 7fef05e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fef08160e50 0x7fef080b4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52354/0 (socket says 172.21.15.12:52354) 2024-03-29T10:14:55.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.311+0000 7fef05e00700 1 -- 172.21.15.12:0/4159772518 learned_addr learned my addr 172.21.15.12:0/4159772518 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:55.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.311+0000 7fef05400700 1 --2- 172.21.15.12:0/4159772518 >> v2:172.21.15.159:3300/0 conn(0x7fef080d3f20 0x7fef080cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:55.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.311+0000 7fef04a00700 1 -- 172.21.15.12:0/4159772518 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163665569 0 0) 0x7fef08161230 con 0x7fef08160e50 2024-03-29T10:14:55.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.311+0000 7fef04a00700 1 -- 172.21.15.12:0/4159772518 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeec0035b0 con 0x7fef08160e50 2024-03-29T10:14:55.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.312+0000 7fef05400700 1 -- 172.21.15.12:0/4159772518 >> v1:172.21.15.159:6789/0 conn(0x7fef080b3f80 legacy=0x7fef080ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.312+0000 7fef05400700 1 -- 172.21.15.12:0/4159772518 >> v1:172.21.15.12:6789/0 conn(0x7fef08160e50 legacy=0x7fef080b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.312+0000 7fef05400700 1 -- 172.21.15.12:0/4159772518 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080d21e0 con 0x7fef080d3f20 2024-03-29T10:14:55.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.312+0000 7fef05400700 1 --2- 172.21.15.12:0/4159772518 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef080d3f20 0x7fef080cdef0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7feef4009840 tx=0x7feef4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d64e1a07e9ea949 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:55.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.312+0000 7fef04a00700 1 -- 172.21.15.12:0/4159772518 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef4011830 con 0x7fef080d3f20 2024-03-29T10:14:55.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.312+0000 7fef04a00700 1 -- 172.21.15.12:0/4159772518 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef4011990 con 0x7fef080d3f20 2024-03-29T10:14:55.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.312+0000 7fef04a00700 1 -- 172.21.15.12:0/4159772518 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef4011d20 con 0x7fef080d3f20 2024-03-29T10:14:55.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.313+0000 7fef0ce00700 1 -- 172.21.15.12:0/4159772518 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef080d3f20 msgr2=0x7fef080cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.313+0000 7fef0ce00700 1 --2- 172.21.15.12:0/4159772518 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef080d3f20 0x7fef080cdef0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7feef4009840 tx=0x7feef4009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:55.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.313+0000 7fef0ce00700 1 -- 172.21.15.12:0/4159772518 shutdown_connections 2024-03-29T10:14:55.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.313+0000 7fef0ce00700 1 --2- 172.21.15.12:0/4159772518 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef080d3f20 0x7fef080cdef0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.313+0000 7fef0ce00700 1 -- 172.21.15.12:0/4159772518 >> 172.21.15.12:0/4159772518 conn(0x7fef0808aff0 msgr2=0x7fef080d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:55.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.313+0000 7fef0ce00700 1 -- 172.21.15.12:0/4159772518 shutdown_connections 2024-03-29T10:14:55.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.313+0000 7fef0ce00700 1 -- 172.21.15.12:0/4159772518 wait complete. 2024-03-29T10:14:55.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.315+0000 7fef0ce00700 1 Processor -- start 2024-03-29T10:14:55.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.315+0000 7fef0ce00700 1 -- start start 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef0ce00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef080b3f80 0x7fef080c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef0ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef08160e50 0x7fef080b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef0ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef081f6900 0x7fef081f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef0ce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef080b79c0 con 0x7fef08160e50 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef0ce00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0814d2b0 con 0x7fef080b3f80 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef0ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef080cd3c0 con 0x7fef081f6900 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef05400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef08160e50 0x7fef080b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef05400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef08160e50 0x7fef080b8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35968/0 (socket says 172.21.15.12:35968) 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef05400700 1 -- 172.21.15.12:0/1356671725 learned_addr learned my addr 172.21.15.12:0/1356671725 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef06800700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef081f6900 0x7fef081f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.316+0000 7fef05e00700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef080b3f80 0x7fef080c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:55.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.317+0000 7fef05400700 1 -- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef081f6900 msgr2=0x7fef081f6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.317+0000 7fef05400700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef081f6900 0x7fef081f6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.317+0000 7fef05400700 1 -- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef080b3f80 msgr2=0x7fef080c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.317+0000 7fef05400700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef080b3f80 0x7fef080c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.317+0000 7fef05400700 1 -- 172.21.15.12:0/1356671725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef40094a0 con 0x7fef08160e50 2024-03-29T10:14:55.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.317+0000 7fef05400700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef08160e50 0x7fef080b8660 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7feef4009fd0 tx=0x7feef4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:55.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.317+0000 7feefe000700 1 -- 172.21.15.12:0/1356671725 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef40036a0 con 0x7fef08160e50 2024-03-29T10:14:55.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.317+0000 7feefe000700 1 -- 172.21.15.12:0/1356671725 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef401f070 con 0x7fef08160e50 2024-03-29T10:14:55.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.317+0000 7feefe000700 1 -- 172.21.15.12:0/1356671725 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef4003a60 con 0x7fef08160e50 2024-03-29T10:14:55.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.317+0000 7fef0ce00700 1 -- 172.21.15.12:0/1356671725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef080cd920 con 0x7fef08160e50 2024-03-29T10:14:55.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.318+0000 7fef0ce00700 1 -- 172.21.15.12:0/1356671725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef080b7e80 con 0x7fef08160e50 2024-03-29T10:14:55.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.319+0000 7fef0ce00700 1 -- 172.21.15.12:0/1356671725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef08067720 con 0x7fef08160e50 2024-03-29T10:14:55.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.320+0000 7feefe000700 1 -- 172.21.15.12:0/1356671725 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7feef4003bc0 con 0x7fef08160e50 2024-03-29T10:14:55.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.321+0000 7feefe000700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feee806ae40 0x7feee806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:55.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.321+0000 7feefe000700 1 -- 172.21.15.12:0/1356671725 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7feef4021070 con 0x7fef08160e50 2024-03-29T10:14:55.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.321+0000 7fef05e00700 1 -- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feee806ae40 msgr2=0x7feee806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:55.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.321+0000 7fef05e00700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feee806ae40 0x7feee806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:55.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.326+0000 7feefe000700 1 -- 172.21.15.12:0/1356671725 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feef4055db0 con 0x7fef08160e50 2024-03-29T10:14:55.377 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:55.379+0000 7fb8a456a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:14:55.426 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:55.428+0000 7fc7e1153240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:14:55.470 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:55.472+0000 7fb8a456a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:55.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.522+0000 7fef05e00700 1 -- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feee806ae40 msgr2=0x7feee806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:55.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.522+0000 7fef05e00700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feee806ae40 0x7feee806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:55.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.541+0000 7fef0ce00700 1 -- 172.21.15.12:0/1356671725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fef080d2fc0 con 0x7fef08160e50 2024-03-29T10:14:55.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.541+0000 7feefe000700 1 -- 172.21.15.12:0/1356671725 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7feef4015020 con 0x7fef08160e50 2024-03-29T10:14:55.539 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:14:55.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 -- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feee806ae40 msgr2=0x7feee806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feee806ae40 0x7feee806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 -- 172.21.15.12:0/1356671725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef08160e50 msgr2=0x7fef080b8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef08160e50 0x7fef080b8660 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7feef4009fd0 tx=0x7feef4004750 comp rx=0 tx=0).stop 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 -- 172.21.15.12:0/1356671725 shutdown_connections 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feee806ae40 0x7feee806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef081f6900 0x7fef081f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef080b3f80 0x7fef080c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 --2- 172.21.15.12:0/1356671725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef08160e50 0x7fef080b8660 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 -- 172.21.15.12:0/1356671725 >> 172.21.15.12:0/1356671725 conn(0x7fef0808aff0 msgr2=0x7fef0815f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 -- 172.21.15.12:0/1356671725 shutdown_connections 2024-03-29T10:14:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.544+0000 7fef0ce00700 1 -- 172.21.15.12:0/1356671725 wait complete. 2024-03-29T10:14:55.561 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T10:14:55.634 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:55.636+0000 7fc7e1153240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:14:55.734 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:55.736+0000 7fc7e1153240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:55.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.765+0000 7ff596e00700 1 Processor -- start 2024-03-29T10:14:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.766+0000 7ff596e00700 1 -- start start 2024-03-29T10:14:55.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.768+0000 7ff596e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff59015dfe0 0x7ff5900ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:55.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.768+0000 7ff596e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5900cb310 con 0x7ff5900ce670 2024-03-29T10:14:55.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.768+0000 7ff596e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5900711a0 con 0x7ff59015a5c0 2024-03-29T10:14:55.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.768+0000 7ff596e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5900ceb00 con 0x7ff59015dfe0 2024-03-29T10:14:55.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.768+0000 7ff594e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff5900ce670 0x7ff5900cba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52358/0 (socket says 172.21.15.12:52358) 2024-03-29T10:14:55.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.768+0000 7ff594e00700 1 -- 172.21.15.12:0/2396097336 learned_addr learned my addr 172.21.15.12:0/2396097336 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.768+0000 7ff58f400700 1 --2- 172.21.15.12:0/2396097336 >> v2:172.21.15.159:3300/0 conn(0x7ff59015dfe0 0x7ff5900ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:55.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.769+0000 7ff58ea00700 1 -- 172.21.15.12:0/2396097336 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117701613 0 0) 0x7ff5900cb310 con 0x7ff5900ce670 2024-03-29T10:14:55.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.769+0000 7ff58ea00700 1 -- 172.21.15.12:0/2396097336 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5740035b0 con 0x7ff5900ce670 2024-03-29T10:14:55.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.769+0000 7ff58f400700 1 -- 172.21.15.12:0/2396097336 >> v1:172.21.15.159:6789/0 conn(0x7ff59015a5c0 legacy=0x7ff5900709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.769+0000 7ff58f400700 1 -- 172.21.15.12:0/2396097336 >> v1:172.21.15.12:6789/0 conn(0x7ff5900ce670 legacy=0x7ff5900cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.769+0000 7ff58f400700 1 -- 172.21.15.12:0/2396097336 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590071480 con 0x7ff59015dfe0 2024-03-29T10:14:55.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.769+0000 7ff58f400700 1 --2- 172.21.15.12:0/2396097336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff59015dfe0 0x7ff5900ce130 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009840 tx=0x7ff57c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9a7db558fc4f2d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:55.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.770+0000 7ff58ea00700 1 -- 172.21.15.12:0/2396097336 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c011830 con 0x7ff59015dfe0 2024-03-29T10:14:55.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.770+0000 7ff58ea00700 1 -- 172.21.15.12:0/2396097336 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c011990 con 0x7ff59015dfe0 2024-03-29T10:14:55.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.770+0000 7ff58ea00700 1 -- 172.21.15.12:0/2396097336 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c011d20 con 0x7ff59015dfe0 2024-03-29T10:14:55.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.770+0000 7ff596e00700 1 -- 172.21.15.12:0/2396097336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff59015dfe0 msgr2=0x7ff5900ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.770+0000 7ff596e00700 1 --2- 172.21.15.12:0/2396097336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff59015dfe0 0x7ff5900ce130 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009840 tx=0x7ff57c009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:55.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.771+0000 7ff596e00700 1 -- 172.21.15.12:0/2396097336 shutdown_connections 2024-03-29T10:14:55.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.771+0000 7ff596e00700 1 --2- 172.21.15.12:0/2396097336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff59015dfe0 0x7ff5900ce130 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.771+0000 7ff596e00700 1 -- 172.21.15.12:0/2396097336 >> 172.21.15.12:0/2396097336 conn(0x7ff5900b0d70 msgr2=0x7ff5900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:55.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.771+0000 7ff596e00700 1 -- 172.21.15.12:0/2396097336 shutdown_connections 2024-03-29T10:14:55.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.771+0000 7ff596e00700 1 -- 172.21.15.12:0/2396097336 wait complete. 2024-03-29T10:14:55.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.773+0000 7ff596e00700 1 Processor -- start 2024-03-29T10:14:55.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.774+0000 7ff596e00700 1 -- start start 2024-03-29T10:14:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.775+0000 7ff596e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5900ce670 0x7ff5900ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.775+0000 7ff596e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff59015a5c0 0x7ff590159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.775+0000 7ff596e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59015dfe0 0x7ff5900c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.775+0000 7ff596e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff590157de0 con 0x7ff59015dfe0 2024-03-29T10:14:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.775+0000 7ff596e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5901f32d0 con 0x7ff5900ce670 2024-03-29T10:14:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.775+0000 7ff596e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5900adb40 con 0x7ff59015a5c0 2024-03-29T10:14:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff594e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59015dfe0 0x7ff5900c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff594e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59015dfe0 0x7ff5900c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35982/0 (socket says 172.21.15.12:35982) 2024-03-29T10:14:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff594e00700 1 -- 172.21.15.12:0/1445420764 learned_addr learned my addr 172.21.15.12:0/1445420764 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:55.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff58f400700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff59015a5c0 0x7ff590159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:55.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff58fe00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5900ce670 0x7ff5900ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:55.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff594e00700 1 -- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff59015a5c0 msgr2=0x7ff590159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff594e00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff59015a5c0 0x7ff590159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff594e00700 1 -- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5900ce670 msgr2=0x7ff5900ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:55.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff594e00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5900ce670 0x7ff5900ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:55.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff594e00700 1 -- 172.21.15.12:0/1445420764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c0094a0 con 0x7ff59015dfe0 2024-03-29T10:14:55.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.776+0000 7ff594e00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59015dfe0 0x7ff5900c96b0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7ff58800cb70 tx=0x7ff58800ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:55.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.777+0000 7ff587e00700 1 -- 172.21.15.12:0/1445420764 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58800aa60 con 0x7ff59015dfe0 2024-03-29T10:14:55.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.777+0000 7ff587e00700 1 -- 172.21.15.12:0/1445420764 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58800abc0 con 0x7ff59015dfe0 2024-03-29T10:14:55.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.777+0000 7ff596e00700 1 -- 172.21.15.12:0/1445420764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff590159e60 con 0x7ff59015dfe0 2024-03-29T10:14:55.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.777+0000 7ff587e00700 1 -- 172.21.15.12:0/1445420764 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff588012400 con 0x7ff59015dfe0 2024-03-29T10:14:55.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.777+0000 7ff596e00700 1 -- 172.21.15.12:0/1445420764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5900ae330 con 0x7ff59015dfe0 2024-03-29T10:14:55.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.778+0000 7ff596e00700 1 -- 172.21.15.12:0/1445420764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff590157f20 con 0x7ff59015dfe0 2024-03-29T10:14:55.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.779+0000 7ff587e00700 1 -- 172.21.15.12:0/1445420764 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7ff58801b3b0 con 0x7ff59015dfe0 2024-03-29T10:14:55.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.780+0000 7ff587e00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff57006ae40 0x7ff57006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:55.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.780+0000 7ff587e00700 1 -- 172.21.15.12:0/1445420764 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff588089250 con 0x7ff59015dfe0 2024-03-29T10:14:55.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.780+0000 7ff58fe00700 1 -- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff57006ae40 msgr2=0x7ff57006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:55.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.780+0000 7ff58fe00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff57006ae40 0x7ff57006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:55.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.785+0000 7ff587e00700 1 -- 172.21.15.12:0/1445420764 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff588054060 con 0x7ff59015dfe0 2024-03-29T10:14:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.981+0000 7ff58fe00700 1 -- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff57006ae40 msgr2=0x7ff57006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.981+0000 7ff58fe00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff57006ae40 0x7ff57006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:55.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.998+0000 7ff596e00700 1 -- 172.21.15.12:0/1445420764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff590159e60 con 0x7ff59015dfe0 2024-03-29T10:14:55.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:55.999+0000 7ff587e00700 1 -- 172.21.15.12:0/1445420764 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff590159e60 con 0x7ff59015dfe0 2024-03-29T10:14:55.998 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T10:14:56.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 -- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff57006ae40 msgr2=0x7ff57006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:56.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff57006ae40 0x7ff57006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 -- 172.21.15.12:0/1445420764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59015dfe0 msgr2=0x7ff5900c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59015dfe0 0x7ff5900c96b0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7ff58800cb70 tx=0x7ff58800ce80 comp rx=0 tx=0).stop 2024-03-29T10:14:56.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 -- 172.21.15.12:0/1445420764 shutdown_connections 2024-03-29T10:14:56.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff57006ae40 0x7ff57006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59015dfe0 0x7ff5900c96b0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5900ce670 0x7ff5900ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 --2- 172.21.15.12:0/1445420764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff59015a5c0 0x7ff590159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 -- 172.21.15.12:0/1445420764 >> 172.21.15.12:0/1445420764 conn(0x7ff5900b0d70 msgr2=0x7ff5900cb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:56.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 -- 172.21.15.12:0/1445420764 shutdown_connections 2024-03-29T10:14:56.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.002+0000 7ff596e00700 1 -- 172.21.15.12:0/1445420764 wait complete. 2024-03-29T10:14:56.023 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T10:14:56.203 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:56.205+0000 7fb8a456a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:14:56.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.228+0000 7f71eec00700 1 Processor -- start 2024-03-29T10:14:56.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.229+0000 7f71eec00700 1 -- start start 2024-03-29T10:14:56.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.230+0000 7f71eec00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f71e815dfe0 0x7f71e80ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.230+0000 7f71eec00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e80cb310 con 0x7f71e80ce670 2024-03-29T10:14:56.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.230+0000 7f71eec00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e80711a0 con 0x7f71e815a5c0 2024-03-29T10:14:56.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.230+0000 7f71eec00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e80ceb00 con 0x7f71e815dfe0 2024-03-29T10:14:56.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.230+0000 7f71ecc00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f71e80ce670 0x7f71e80cba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52360/0 (socket says 172.21.15.12:52360) 2024-03-29T10:14:56.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.230+0000 7f71ecc00700 1 -- 172.21.15.12:0/2049503292 learned_addr learned my addr 172.21.15.12:0/2049503292 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:56.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.230+0000 7f71e7400700 1 --2- 172.21.15.12:0/2049503292 >> v2:172.21.15.159:3300/0 conn(0x7f71e815dfe0 0x7f71e80ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:56.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.231+0000 7f71e6a00700 1 -- 172.21.15.12:0/2049503292 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151160313 0 0) 0x7f71e80cb310 con 0x7f71e80ce670 2024-03-29T10:14:56.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.231+0000 7f71e7400700 1 -- 172.21.15.12:0/2049503292 >> v1:172.21.15.159:6789/0 conn(0x7f71e815a5c0 legacy=0x7f71e80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.231+0000 7f71e7400700 1 -- 172.21.15.12:0/2049503292 >> v1:172.21.15.12:6789/0 conn(0x7f71e80ce670 legacy=0x7f71e80cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.231+0000 7f71e7400700 1 -- 172.21.15.12:0/2049503292 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e8071480 con 0x7f71e815dfe0 2024-03-29T10:14:56.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.231+0000 7f71e7400700 1 --2- 172.21.15.12:0/2049503292 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e815dfe0 0x7f71e80ce130 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f71d4009840 tx=0x7f71d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6db77d399c2057e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:56.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.231+0000 7f71e6a00700 1 -- 172.21.15.12:0/2049503292 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d400f8a0 con 0x7f71e815dfe0 2024-03-29T10:14:56.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.231+0000 7f71e6a00700 1 -- 172.21.15.12:0/2049503292 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d400fa00 con 0x7f71e815dfe0 2024-03-29T10:14:56.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.231+0000 7f71e6a00700 1 -- 172.21.15.12:0/2049503292 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d400fd90 con 0x7f71e815dfe0 2024-03-29T10:14:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.232+0000 7f71eec00700 1 -- 172.21.15.12:0/2049503292 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e815dfe0 msgr2=0x7f71e80ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.232+0000 7f71eec00700 1 --2- 172.21.15.12:0/2049503292 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e815dfe0 0x7f71e80ce130 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f71d4009840 tx=0x7f71d4009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.232+0000 7f71eec00700 1 -- 172.21.15.12:0/2049503292 shutdown_connections 2024-03-29T10:14:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.232+0000 7f71eec00700 1 --2- 172.21.15.12:0/2049503292 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e815dfe0 0x7f71e80ce130 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.232+0000 7f71eec00700 1 -- 172.21.15.12:0/2049503292 >> 172.21.15.12:0/2049503292 conn(0x7f71e80b0d70 msgr2=0x7f71e80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.232+0000 7f71eec00700 1 -- 172.21.15.12:0/2049503292 shutdown_connections 2024-03-29T10:14:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.232+0000 7f71eec00700 1 -- 172.21.15.12:0/2049503292 wait complete. 2024-03-29T10:14:56.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.235+0000 7f71eec00700 1 Processor -- start 2024-03-29T10:14:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.235+0000 7f71eec00700 1 -- start start 2024-03-29T10:14:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71eec00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71e80ce670 0x7f71e81505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71eec00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e815a5c0 0x7f71e80c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71eec00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e815dfe0 0x7f71e80ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71eec00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80ca760 con 0x7f71e80ce670 2024-03-29T10:14:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71eec00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80adfb0 con 0x7f71e815dfe0 2024-03-29T10:14:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71eec00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e8153f30 con 0x7f71e815a5c0 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71e80ce670 0x7f71e81505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71e80ce670 0x7f71e81505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35990/0 (socket says 172.21.15.12:35990) 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71e7e00700 1 -- 172.21.15.12:0/3678568108 learned_addr learned my addr 172.21.15.12:0/3678568108 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71ecc00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e815dfe0 0x7f71e80ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.237+0000 7f71e7400700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e815a5c0 0x7f71e80c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71ecc00700 1 -- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e815a5c0 msgr2=0x7f71e80c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71ecc00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e815a5c0 0x7f71e80c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71ecc00700 1 -- 172.21.15.12:0/3678568108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71e80ce670 msgr2=0x7f71e81505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71ecc00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71e80ce670 0x7f71e81505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71ecc00700 1 -- 172.21.15.12:0/3678568108 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d40094a0 con 0x7f71e815dfe0 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71e7400700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e815a5c0 0x7f71e80c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71ecc00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e815dfe0 0x7f71e80ca0e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f71e000cbe0 tx=0x7f71e000cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71dfe00700 1 -- 172.21.15.12:0/3678568108 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e000aa40 con 0x7f71e815dfe0 2024-03-29T10:14:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71dfe00700 1 -- 172.21.15.12:0/3678568108 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e000abc0 con 0x7f71e815dfe0 2024-03-29T10:14:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71eec00700 1 -- 172.21.15.12:0/3678568108 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e80ad370 con 0x7f71e815dfe0 2024-03-29T10:14:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71dfe00700 1 -- 172.21.15.12:0/3678568108 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e00070d0 con 0x7f71e815dfe0 2024-03-29T10:14:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.238+0000 7f71eec00700 1 -- 172.21.15.12:0/3678568108 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e80ad8f0 con 0x7f71e815dfe0 2024-03-29T10:14:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.240+0000 7f71eec00700 1 -- 172.21.15.12:0/3678568108 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e8067720 con 0x7f71e815dfe0 2024-03-29T10:14:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.240+0000 7f71dfe00700 1 -- 172.21.15.12:0/3678568108 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f71e0007230 con 0x7f71e815dfe0 2024-03-29T10:14:56.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.242+0000 7f71dfe00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71c806ae40 0x7f71c806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.242+0000 7f71dfe00700 1 -- 172.21.15.12:0/3678568108 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f71e0089450 con 0x7f71e815dfe0 2024-03-29T10:14:56.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.242+0000 7f71e7e00700 1 -- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71c806ae40 msgr2=0x7f71c806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:56.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.242+0000 7f71e7e00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71c806ae40 0x7f71c806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:56.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.246+0000 7f71dfe00700 1 -- 172.21.15.12:0/3678568108 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71e0054240 con 0x7f71e815dfe0 2024-03-29T10:14:56.415 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:56.417+0000 7fb8a456a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:14:56.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.443+0000 7f71e7e00700 1 -- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71c806ae40 msgr2=0x7f71c806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:56.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.443+0000 7f71e7e00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71c806ae40 0x7f71c806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:56.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.460+0000 7f71eec00700 1 -- 172.21.15.12:0/3678568108 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71e80b1ad0 con 0x7f71e815dfe0 2024-03-29T10:14:56.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.460+0000 7f71dfe00700 1 -- 172.21.15.12:0/3678568108 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7f71e0057860 con 0x7f71e815dfe0 2024-03-29T10:14:56.459 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T10:14:56.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 -- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71c806ae40 msgr2=0x7f71c806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:56.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71c806ae40 0x7f71c806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 -- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e815dfe0 msgr2=0x7f71e80ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e815dfe0 0x7f71e80ca0e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f71e000cbe0 tx=0x7f71e000cef0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 -- 172.21.15.12:0/3678568108 shutdown_connections 2024-03-29T10:14:56.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f71c806ae40 0x7f71c806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e815dfe0 0x7f71e80ca0e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71e80ce670 0x7f71e81505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 --2- 172.21.15.12:0/3678568108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e815a5c0 0x7f71e80c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 -- 172.21.15.12:0/3678568108 >> 172.21.15.12:0/3678568108 conn(0x7f71e80b0d70 msgr2=0x7f71e806dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:56.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 -- 172.21.15.12:0/3678568108 shutdown_connections 2024-03-29T10:14:56.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.465+0000 7f71eec00700 1 -- 172.21.15.12:0/3678568108 wait complete. 2024-03-29T10:14:56.492 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:56.533 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:56.536+0000 7fc7e1153240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:14:56.620 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:56.622+0000 7fb8a456a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:56.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.697+0000 7f7ad0a00700 1 Processor -- start 2024-03-29T10:14:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.698+0000 7f7ad0a00700 1 -- start start 2024-03-29T10:14:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.699+0000 7f7ad0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc15d670 0x7f7acc0ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.699+0000 7f7ad0a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7acc15da50 0x7f7acc0b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.699+0000 7f7ad0a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc0b3890 con 0x7f7acc0b3000 2024-03-29T10:14:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.699+0000 7f7ad0a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7acc0acac0 con 0x7f7acc15da50 2024-03-29T10:14:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.699+0000 7f7ad0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc159f50 con 0x7f7acc15d670 2024-03-29T10:14:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.700+0000 7f7ac9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc15d670 0x7f7acc0ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.700+0000 7f7ac9400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7acc15da50 0x7f7acc0b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.700+0000 7f7ac9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc15d670 0x7f7acc0ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:56916/0 (socket says 172.21.15.12:56916) 2024-03-29T10:14:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.700+0000 7f7ac9e00700 1 -- 172.21.15.12:0/1288112645 learned_addr learned my addr 172.21.15.12:0/1288112645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.700+0000 7f7ac9e00700 1 -- 172.21.15.12:0/1288112645 >> v2:172.21.15.159:3300/0 conn(0x7f7acc15da50 msgr2=0x7f7acc0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.700+0000 7f7ac9e00700 1 --2- 172.21.15.12:0/1288112645 >> v2:172.21.15.159:3300/0 conn(0x7f7acc15da50 0x7f7acc0b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.700+0000 7f7ac9e00700 1 -- 172.21.15.12:0/1288112645 >> v1:172.21.15.159:6789/0 conn(0x7f7acc0b3000 legacy=0x7f7acc06dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.700+0000 7f7ac9e00700 1 -- 172.21.15.12:0/1288112645 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc0b35b0 con 0x7f7acc15d670 2024-03-29T10:14:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.701+0000 7f7ac9e00700 1 --2- 172.21.15.12:0/1288112645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc15d670 0x7f7acc0ac580 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00aa90 tx=0x7f7abc00ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11757b0829eafbdf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.701+0000 7f7ac8a00700 1 -- 172.21.15.12:0/1288112645 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc00cb70 con 0x7f7acc15d670 2024-03-29T10:14:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.701+0000 7f7ac8a00700 1 -- 172.21.15.12:0/1288112645 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc00ccd0 con 0x7f7acc15d670 2024-03-29T10:14:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.701+0000 7f7ac8a00700 1 -- 172.21.15.12:0/1288112645 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc00f690 con 0x7f7acc15d670 2024-03-29T10:14:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.702+0000 7f7ad0a00700 1 -- 172.21.15.12:0/1288112645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc15d670 msgr2=0x7f7acc0ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.702+0000 7f7ad0a00700 1 --2- 172.21.15.12:0/1288112645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc15d670 0x7f7acc0ac580 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00aa90 tx=0x7f7abc00ada0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.702+0000 7f7ad0a00700 1 -- 172.21.15.12:0/1288112645 shutdown_connections 2024-03-29T10:14:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.702+0000 7f7ad0a00700 1 --2- 172.21.15.12:0/1288112645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc15d670 0x7f7acc0ac580 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.702+0000 7f7ad0a00700 1 --2- 172.21.15.12:0/1288112645 >> v2:172.21.15.159:3300/0 conn(0x7f7acc15da50 0x7f7acc0b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.702+0000 7f7ad0a00700 1 -- 172.21.15.12:0/1288112645 >> 172.21.15.12:0/1288112645 conn(0x7f7acc0cf190 msgr2=0x7f7acc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.702+0000 7f7ad0a00700 1 -- 172.21.15.12:0/1288112645 shutdown_connections 2024-03-29T10:14:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.702+0000 7f7ad0a00700 1 -- 172.21.15.12:0/1288112645 wait complete. 2024-03-29T10:14:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.704+0000 7f7ad0a00700 1 Processor -- start 2024-03-29T10:14:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.704+0000 7f7ad0a00700 1 -- start start 2024-03-29T10:14:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7ad0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc0b3000 0x7f7acc151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7ad0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7acc15d670 0x7f7acc154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7ad0a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7acc15da50 0x7f7acc154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7ad0a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc1522b0 con 0x7f7acc15d670 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7ad0a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc152590 con 0x7f7acc15da50 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7ad0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc152870 con 0x7f7acc0b3000 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7ac9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7acc15d670 0x7f7acc154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7ac9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc0b3000 0x7f7acc151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7ac9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc0b3000 0x7f7acc151260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:56926/0 (socket says 172.21.15.12:56926) 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7ac9e00700 1 -- 172.21.15.12:0/774862981 learned_addr learned my addr 172.21.15.12:0/774862981 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.705+0000 7f7aca800700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7acc15da50 0x7f7acc154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ac9e00700 1 -- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7acc15da50 msgr2=0x7f7acc154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ac9e00700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7acc15da50 0x7f7acc154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ac9e00700 1 -- 172.21.15.12:0/774862981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7acc15d670 msgr2=0x7f7acc154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ac9e00700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7acc15d670 0x7f7acc154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ac9e00700 1 -- 172.21.15.12:0/774862981 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc00a6f0 con 0x7f7acc0b3000 2024-03-29T10:14:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ac9e00700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc0b3000 0x7f7acc151260 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00ade0 tx=0x7f7abc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ac2000700 1 -- 172.21.15.12:0/774862981 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc00a570 con 0x7f7acc0b3000 2024-03-29T10:14:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ac2000700 1 -- 172.21.15.12:0/774862981 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc00f840 con 0x7f7acc0b3000 2024-03-29T10:14:56.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ad0a00700 1 -- 172.21.15.12:0/774862981 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc152b50 con 0x7f7acc0b3000 2024-03-29T10:14:56.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ac2000700 1 -- 172.21.15.12:0/774862981 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc00fbb0 con 0x7f7acc0b3000 2024-03-29T10:14:56.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.706+0000 7f7ad0a00700 1 -- 172.21.15.12:0/774862981 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc151ef0 con 0x7f7acc0b3000 2024-03-29T10:14:56.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.708+0000 7f7ad0a00700 1 -- 172.21.15.12:0/774862981 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc152080 con 0x7f7acc0b3000 2024-03-29T10:14:56.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.708+0000 7f7ac2000700 1 -- 172.21.15.12:0/774862981 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f7abc00fd10 con 0x7f7acc0b3000 2024-03-29T10:14:56.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.710+0000 7f7ac2000700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7aac06ae40 0x7f7aac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.710+0000 7f7ac2000700 1 -- 172.21.15.12:0/774862981 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7abc013070 con 0x7f7acc0b3000 2024-03-29T10:14:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.710+0000 7f7ac9400700 1 -- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7aac06ae40 msgr2=0x7f7aac06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.710+0000 7f7ac9400700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7aac06ae40 0x7f7aac06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:56.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.714+0000 7f7ac2000700 1 -- 172.21.15.12:0/774862981 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7abc056330 con 0x7f7acc0b3000 2024-03-29T10:14:56.765 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:56.767+0000 7fc7e1153240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:14:56.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.911+0000 7f7ac9400700 1 -- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7aac06ae40 msgr2=0x7f7aac06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:56.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.911+0000 7f7ac9400700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7aac06ae40 0x7f7aac06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:56.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.975+0000 7f7ad0a00700 1 -- 172.21.15.12:0/774862981 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7acc067720 con 0x7f7acc0b3000 2024-03-29T10:14:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.976+0000 7f7ac2000700 1 -- 172.21.15.12:0/774862981 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f7abc059950 con 0x7f7acc0b3000 2024-03-29T10:14:56.974 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:56.974 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1317631939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1317631939},{"type":"v1","addr":"172.21.15.159:6836","nonce":1317631939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1057731647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1057731647},{"type":"v1","addr":"172.21.15.159:6837","nonce":1057731647}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1567088347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1567088347},{"type":"v1","addr":"172.21.15.12:6834","nonce":1567088347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:14:25.846391+0000","modified":"2024-03-29T10:14:29.114278+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8406":{"gid":8406,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3311625077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3311625077},{"type":"v1","addr":"172.21.15.12:6835","nonce":3311625077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-29T10:14:56.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.978+0000 7f7ad0a00700 1 -- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7aac06ae40 msgr2=0x7f7aac06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:56.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.978+0000 7f7ad0a00700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7aac06ae40 0x7f7aac06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.978+0000 7f7ad0a00700 1 -- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc0b3000 msgr2=0x7f7acc151260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:56.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.978+0000 7f7ad0a00700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc0b3000 0x7f7acc151260 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00ade0 tx=0x7f7abc004750 comp rx=0 tx=0).stop 2024-03-29T10:14:56.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.978+0000 7f7ad0a00700 1 -- 172.21.15.12:0/774862981 shutdown_connections 2024-03-29T10:14:56.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.978+0000 7f7ad0a00700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7aac06ae40 0x7f7aac06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.978+0000 7f7ad0a00700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7acc15da50 0x7f7acc154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.978+0000 7f7ad0a00700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7acc0b3000 0x7f7acc151260 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.978+0000 7f7ad0a00700 1 --2- 172.21.15.12:0/774862981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7acc15d670 0x7f7acc154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:56.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.979+0000 7f7ad0a00700 1 -- 172.21.15.12:0/774862981 >> 172.21.15.12:0/774862981 conn(0x7f7acc0cf190 msgr2=0x7f7acc0acf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:56.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.979+0000 7f7ad0a00700 1 -- 172.21.15.12:0/774862981 shutdown_connections 2024-03-29T10:14:56.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:56.979+0000 7f7ad0a00700 1 -- 172.21.15.12:0/774862981 wait complete. 2024-03-29T10:14:56.978 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 64 2024-03-29T10:14:56.985 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:56.987+0000 7fc7e1153240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:56.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:57.096 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.098+0000 7fb8a456a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:14:57.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.241+0000 7fd4c7e00700 1 Processor -- start 2024-03-29T10:14:57.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.241+0000 7fd4c7e00700 1 -- start start 2024-03-29T10:14:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.242+0000 7fd4c7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd4c00b27f0 0x7fd4c00ac960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.242+0000 7fd4c7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c00718e0 0x7fd4c00acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.242+0000 7fd4c7e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c00ac4d0 con 0x7fd4c0160c90 2024-03-29T10:14:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4c7e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c0071e50 con 0x7fd4c00b27f0 2024-03-29T10:14:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4c7e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c0072130 con 0x7fd4c00718e0 2024-03-29T10:14:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4c4a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd4c00b27f0 0x7fd4c00ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4c4a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd4c00b27f0 0x7fd4c00ac960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36014/0 (socket says 172.21.15.12:36014) 2024-03-29T10:14:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4c4a00700 1 -- 172.21.15.12:0/35711255 learned_addr learned my addr 172.21.15.12:0/35711255 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4c5e00700 1 --2- 172.21.15.12:0/35711255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c00718e0 0x7fd4c00acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4c4a00700 1 -- 172.21.15.12:0/35711255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c00718e0 msgr2=0x7fd4c00acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4c4a00700 1 --2- 172.21.15.12:0/35711255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c00718e0 0x7fd4c00acea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4c4a00700 1 -- 172.21.15.12:0/35711255 >> v1:172.21.15.12:6789/0 conn(0x7fd4c0160c90 legacy=0x7fd4c00b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4c4a00700 1 -- 172.21.15.12:0/35711255 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c0072410 con 0x7fd4c00b27f0 2024-03-29T10:14:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.243+0000 7fd4bfe00700 1 -- 172.21.15.12:0/35711255 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239884556 0 0) 0x7fd4c00ac4d0 con 0x7fd4c0160c90 2024-03-29T10:14:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.244+0000 7fd4c4a00700 1 --2- 172.21.15.12:0/35711255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 0x7fd4c00ac960 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac0098f0 tx=0x7fd4ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1596bf6a61baf92 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.244+0000 7fd4bfe00700 1 -- 172.21.15.12:0/35711255 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac011830 con 0x7fd4c00b27f0 2024-03-29T10:14:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.244+0000 7fd4bfe00700 1 -- 172.21.15.12:0/35711255 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac011990 con 0x7fd4c00b27f0 2024-03-29T10:14:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.244+0000 7fd4bfe00700 1 -- 172.21.15.12:0/35711255 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac011d20 con 0x7fd4c00b27f0 2024-03-29T10:14:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.244+0000 7fd4c7e00700 1 -- 172.21.15.12:0/35711255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 msgr2=0x7fd4c00ac960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.244+0000 7fd4c7e00700 1 --2- 172.21.15.12:0/35711255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 0x7fd4c00ac960 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac0098f0 tx=0x7fd4ac009c00 comp rx=0 tx=0).stop 2024-03-29T10:14:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.245+0000 7fd4c7e00700 1 -- 172.21.15.12:0/35711255 shutdown_connections 2024-03-29T10:14:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.245+0000 7fd4c7e00700 1 --2- 172.21.15.12:0/35711255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c00718e0 0x7fd4c00acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.245+0000 7fd4c7e00700 1 --2- 172.21.15.12:0/35711255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 0x7fd4c00ac960 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.245+0000 7fd4c7e00700 1 -- 172.21.15.12:0/35711255 >> 172.21.15.12:0/35711255 conn(0x7fd4c00cf190 msgr2=0x7fd4c00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.245+0000 7fd4c7e00700 1 -- 172.21.15.12:0/35711255 shutdown_connections 2024-03-29T10:14:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.245+0000 7fd4c7e00700 1 -- 172.21.15.12:0/35711255 wait complete. 2024-03-29T10:14:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.246+0000 7fd4c7e00700 1 Processor -- start 2024-03-29T10:14:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.247+0000 7fd4c7e00700 1 -- start start 2024-03-29T10:14:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.247+0000 7fd4c7e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4c00718e0 0x7fd4c00ac960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.247+0000 7fd4c7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 0x7fd4c00acea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c0160c90 0x7fd4c014ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c7e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00ad3e0 con 0x7fd4c00b27f0 2024-03-29T10:14:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c7e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00ae230 con 0x7fd4c00718e0 2024-03-29T10:14:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c7e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00ad620 con 0x7fd4c0160c90 2024-03-29T10:14:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 0x7fd4c00acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 0x7fd4c00acea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36018/0 (socket says 172.21.15.12:36018) 2024-03-29T10:14:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c4a00700 1 -- 172.21.15.12:0/1599655050 learned_addr learned my addr 172.21.15.12:0/1599655050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c5400700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4c00718e0 0x7fd4c00ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c5e00700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c0160c90 0x7fd4c014ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c4a00700 1 -- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c0160c90 msgr2=0x7fd4c014ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c4a00700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c0160c90 0x7fd4c014ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c4a00700 1 -- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4c00718e0 msgr2=0x7fd4c00ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c4a00700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4c00718e0 0x7fd4c00ac960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.248+0000 7fd4c4a00700 1 -- 172.21.15.12:0/1599655050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac009550 con 0x7fd4c00b27f0 2024-03-29T10:14:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.249+0000 7fd4c4a00700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 0x7fd4c00acea0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac009c40 tx=0x7fd4ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.249+0000 7fd4bd600700 1 -- 172.21.15.12:0/1599655050 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac004340 con 0x7fd4c00b27f0 2024-03-29T10:14:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.249+0000 7fd4bd600700 1 -- 172.21.15.12:0/1599655050 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac01f070 con 0x7fd4c00b27f0 2024-03-29T10:14:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.249+0000 7fd4c7e00700 1 -- 172.21.15.12:0/1599655050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c00ad900 con 0x7fd4c00b27f0 2024-03-29T10:14:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.249+0000 7fd4bd600700 1 -- 172.21.15.12:0/1599655050 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac0036a0 con 0x7fd4c00b27f0 2024-03-29T10:14:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.249+0000 7fd4c7e00700 1 -- 172.21.15.12:0/1599655050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c00adec0 con 0x7fd4c00b27f0 2024-03-29T10:14:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.250+0000 7fd4c7e00700 1 -- 172.21.15.12:0/1599655050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c0067720 con 0x7fd4c00b27f0 2024-03-29T10:14:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.251+0000 7fd4bd600700 1 -- 172.21.15.12:0/1599655050 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fd4ac021030 con 0x7fd4c00b27f0 2024-03-29T10:14:57.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.252+0000 7fd4bd600700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4a006ae40 0x7fd4a006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.252+0000 7fd4bd600700 1 -- 172.21.15.12:0/1599655050 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fd4ac08aad0 con 0x7fd4c00b27f0 2024-03-29T10:14:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.252+0000 7fd4c5400700 1 -- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4a006ae40 msgr2=0x7fd4a006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:57.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.252+0000 7fd4c5400700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4a006ae40 0x7fd4a006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:57.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.256+0000 7fd4bd600700 1 -- 172.21.15.12:0/1599655050 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd4ac0558e0 con 0x7fd4c00b27f0 2024-03-29T10:14:57.306 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.308+0000 7fb8a456a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:57.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.453+0000 7fd4c5400700 1 -- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4a006ae40 msgr2=0x7fd4a006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:57.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.453+0000 7fd4c5400700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4a006ae40 0x7fd4a006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:14:57.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.501+0000 7fd4c7e00700 1 -- 172.21.15.12:0/1599655050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4c015ebe0 con 0x7fd4c00b27f0 2024-03-29T10:14:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.503+0000 7fd4bd600700 1 -- 172.21.15.12:0/1599655050 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16558 (secure 0 0 0) 0x7fd4ac058f00 con 0x7fd4c00b27f0 2024-03-29T10:14:57.501 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:57.501 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":141,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:14:42.823304+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:14:23.708181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-29T10:14:24.829152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3261410285":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2027263440":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/4127686095":"2024-03-30T10:14:42.823243+0000","172.21.15.159:6833/43444":"2024-03-30T10:14:42.823243+0000","172.21.15.159:6832/43444":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/3727946147":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2214680226":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2899228198":"2024-03-30T10:14:42.823243+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 -- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4a006ae40 msgr2=0x7fd4a006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4a006ae40 0x7fd4a006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 -- 172.21.15.12:0/1599655050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 msgr2=0x7fd4c00acea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 0x7fd4c00acea0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac009c40 tx=0x7fd4ac004750 comp rx=0 tx=0).stop 2024-03-29T10:14:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 -- 172.21.15.12:0/1599655050 shutdown_connections 2024-03-29T10:14:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4a006ae40 0x7fd4a006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c0160c90 0x7fd4c014ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4c00718e0 0x7fd4c00ac960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 --2- 172.21.15.12:0/1599655050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c00b27f0 0x7fd4c00acea0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 -- 172.21.15.12:0/1599655050 >> 172.21.15.12:0/1599655050 conn(0x7fd4c00cf190 msgr2=0x7fd4c015fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 -- 172.21.15.12:0/1599655050 shutdown_connections 2024-03-29T10:14:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.507+0000 7fd4c7e00700 1 -- 172.21.15.12:0/1599655050 wait complete. 2024-03-29T10:14:57.531 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T10:14:57.531 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:57.538 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:57.541+0000 7fc7e1153240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:14:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.733+0000 7f9975c00700 1 Processor -- start 2024-03-29T10:14:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.733+0000 7f9975c00700 1 -- start start 2024-03-29T10:14:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.734+0000 7f9975c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99700cb3f0 0x7f99700ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.734+0000 7f9975c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f99700ac600 0x7f997006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.734+0000 7f9975c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997015fd50 con 0x7f997015b5d0 2024-03-29T10:14:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.734+0000 7f9975c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f997015b9b0 con 0x7f99700cb3f0 2024-03-29T10:14:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.734+0000 7f9975c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99700aca10 con 0x7f99700ac600 2024-03-29T10:14:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.734+0000 7f996e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99700cb3f0 0x7f99700ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.734+0000 7f996e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99700cb3f0 0x7f99700ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36032/0 (socket says 172.21.15.12:36032) 2024-03-29T10:14:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.734+0000 7f996e000700 1 -- 172.21.15.12:0/1229974344 learned_addr learned my addr 172.21.15.12:0/1229974344 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.734+0000 7f996f400700 1 --2- 172.21.15.12:0/1229974344 >> v2:172.21.15.159:3300/0 conn(0x7f99700ac600 0x7f997006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.735+0000 7f996e000700 1 -- 172.21.15.12:0/1229974344 >> v2:172.21.15.159:3300/0 conn(0x7f99700ac600 msgr2=0x7f997006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.735+0000 7f996e000700 1 --2- 172.21.15.12:0/1229974344 >> v2:172.21.15.159:3300/0 conn(0x7f99700ac600 0x7f997006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.735+0000 7f996e000700 1 -- 172.21.15.12:0/1229974344 >> v1:172.21.15.159:6789/0 conn(0x7f997015b5d0 legacy=0x7f99700ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.735+0000 7f996e000700 1 -- 172.21.15.12:0/1229974344 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997015f140 con 0x7f99700cb3f0 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.735+0000 7f996e000700 1 --2- 172.21.15.12:0/1229974344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99700cb3f0 0x7f99700ac0c0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f99700cfdc0 tx=0x7f995c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502af56327fdf072 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.735+0000 7f996d600700 1 -- 172.21.15.12:0/1229974344 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c011830 con 0x7f99700cb3f0 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.735+0000 7f996d600700 1 -- 172.21.15.12:0/1229974344 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995c011990 con 0x7f99700cb3f0 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.735+0000 7f996d600700 1 -- 172.21.15.12:0/1229974344 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c011d20 con 0x7f99700cb3f0 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.736+0000 7f9975c00700 1 -- 172.21.15.12:0/1229974344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99700cb3f0 msgr2=0x7f99700ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.736+0000 7f9975c00700 1 --2- 172.21.15.12:0/1229974344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99700cb3f0 0x7f99700ac0c0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f99700cfdc0 tx=0x7f995c009bd0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.736+0000 7f9975c00700 1 -- 172.21.15.12:0/1229974344 shutdown_connections 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.736+0000 7f9975c00700 1 --2- 172.21.15.12:0/1229974344 >> v2:172.21.15.159:3300/0 conn(0x7f99700ac600 0x7f997006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.736+0000 7f9975c00700 1 --2- 172.21.15.12:0/1229974344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99700cb3f0 0x7f99700ac0c0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.736+0000 7f9975c00700 1 -- 172.21.15.12:0/1229974344 >> 172.21.15.12:0/1229974344 conn(0x7f99700cf190 msgr2=0x7f99700cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.736+0000 7f9975c00700 1 -- 172.21.15.12:0/1229974344 shutdown_connections 2024-03-29T10:14:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.736+0000 7f9975c00700 1 -- 172.21.15.12:0/1229974344 wait complete. 2024-03-29T10:14:57.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.738+0000 7f9975c00700 1 Processor -- start 2024-03-29T10:14:57.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.739+0000 7f9975c00700 1 -- start start 2024-03-29T10:14:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.740+0000 7f9975c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99700ac600 0x7f99701f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.740+0000 7f9975c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f997015b5d0 0x7f9970152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.740+0000 7f9975c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99700cbba0 0x7f9970153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.740+0000 7f9975c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99700ccdc0 con 0x7f99700cbba0 2024-03-29T10:14:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.740+0000 7f9975c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99700cc140 con 0x7f99700ac600 2024-03-29T10:14:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.740+0000 7f9975c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99700cc420 con 0x7f997015b5d0 2024-03-29T10:14:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99700ac600 0x7f99701f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99700cbba0 0x7f9970153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99700ac600 0x7f99701f4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:57004/0 (socket says 172.21.15.12:57004) 2024-03-29T10:14:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99700cbba0 0x7f9970153010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36042/0 (socket says 172.21.15.12:36042) 2024-03-29T10:14:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996ea00700 1 -- 172.21.15.12:0/3038154665 learned_addr learned my addr 172.21.15.12:0/3038154665 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996e000700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f997015b5d0 0x7f9970152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996ea00700 1 -- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f997015b5d0 msgr2=0x7f9970152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996ea00700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f997015b5d0 0x7f9970152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996ea00700 1 -- 172.21.15.12:0/3038154665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99700cbba0 msgr2=0x7f9970153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996ea00700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99700cbba0 0x7f9970153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996ea00700 1 -- 172.21.15.12:0/3038154665 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c009550 con 0x7f99700ac600 2024-03-29T10:14:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.741+0000 7f996ea00700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99700ac600 0x7f99701f4260 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f996000c4e0 tx=0x7f996000c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.742+0000 7f9966a00700 1 -- 172.21.15.12:0/3038154665 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99600052d0 con 0x7f99700ac600 2024-03-29T10:14:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.742+0000 7f9966a00700 1 -- 172.21.15.12:0/3038154665 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9960005430 con 0x7f99700ac600 2024-03-29T10:14:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.742+0000 7f9975c00700 1 -- 172.21.15.12:0/3038154665 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99700cc700 con 0x7f99700ac600 2024-03-29T10:14:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.742+0000 7f9966a00700 1 -- 172.21.15.12:0/3038154665 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99600038c0 con 0x7f99700ac600 2024-03-29T10:14:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.742+0000 7f9975c00700 1 -- 172.21.15.12:0/3038154665 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99701f47a0 con 0x7f99700ac600 2024-03-29T10:14:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.743+0000 7f9975c00700 1 -- 172.21.15.12:0/3038154665 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99700cba10 con 0x7f99700ac600 2024-03-29T10:14:57.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.744+0000 7f9966a00700 1 -- 172.21.15.12:0/3038154665 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 74) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f9960003a20 con 0x7f99700ac600 2024-03-29T10:14:57.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.745+0000 7f9966a00700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995006ae40 0x7f995006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:57.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.745+0000 7f9966a00700 1 -- 172.21.15.12:0/3038154665 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f9960088bd0 con 0x7f99700ac600 2024-03-29T10:14:57.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.745+0000 7f996e000700 1 -- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995006ae40 msgr2=0x7f995006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:14:57.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.745+0000 7f996e000700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995006ae40 0x7f995006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:14:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.749+0000 7f9966a00700 1 -- 172.21.15.12:0/3038154665 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9960053930 con 0x7f99700ac600 2024-03-29T10:14:57.751 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.753+0000 7fb8a456a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:14:57.765 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:57.767+0000 7fc7e1153240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:14:57.866 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.868+0000 7fb8a456a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.881+0000 7f9966a00700 1 -- 172.21.15.12:0/3038154665 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mgrmap(e 75) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f9960053d00 con 0x7f99700ac600 2024-03-29T10:14:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.881+0000 7f9966a00700 1 -- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995006ae40 msgr2=0x7f995006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:14:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:57.881+0000 7f9966a00700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995006ae40 0x7f995006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:57.947 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.949+0000 7fb817600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:57.947 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.949+0000 7fb817600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:57.947 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.949+0000 7fb817600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:57.947 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.949+0000 7fb817600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:57.947 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.949+0000 7fb817600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:57.948 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.950+0000 7fb812600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:57.948 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.950+0000 7fb812600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:57.948 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.950+0000 7fb812600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:57.948 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.950+0000 7fb812600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:57.948 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:14:57.950+0000 7fb812600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:14:58.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.026+0000 7f9975c00700 1 -- 172.21.15.12:0/3038154665 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f997015f9a0 con 0x7f99700ac600 2024-03-29T10:14:58.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.027+0000 7f9966a00700 1 -- 172.21.15.12:0/3038154665 <== mon.1 v2:172.21.15.159:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f9960073110 con 0x7f99700ac600 2024-03-29T10:14:58.025 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:58.025 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1317631939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1317631939},{"type":"v1","addr":"172.21.15.159:6836","nonce":1317631939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1057731647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1057731647},{"type":"v1","addr":"172.21.15.159:6837","nonce":1057731647}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1567088347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1567088347},{"type":"v1","addr":"172.21.15.12:6834","nonce":1567088347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:14:25.846391+0000","modified":"2024-03-29T10:14:29.114278+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8406":{"gid":8406,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3311625077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3311625077},{"type":"v1","addr":"172.21.15.12:6835","nonce":3311625077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-29T10:14:58.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.030+0000 7f9975c00700 1 -- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99700ac600 msgr2=0x7f99701f4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.030+0000 7f9975c00700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99700ac600 0x7f99701f4260 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f996000c4e0 tx=0x7f996000c7f0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.030+0000 7f9975c00700 1 -- 172.21.15.12:0/3038154665 shutdown_connections 2024-03-29T10:14:58.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.030+0000 7f9975c00700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f995006ae40 0x7f995006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.030+0000 7f9975c00700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99700cbba0 0x7f9970153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.030+0000 7f9975c00700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99700ac600 0x7f99701f4260 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.030+0000 7f9975c00700 1 --2- 172.21.15.12:0/3038154665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f997015b5d0 0x7f9970152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.030+0000 7f9975c00700 1 -- 172.21.15.12:0/3038154665 >> 172.21.15.12:0/3038154665 conn(0x7f99700cf190 msgr2=0x7f997014c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:58.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.030+0000 7f9975c00700 1 -- 172.21.15.12:0/3038154665 shutdown_connections 2024-03-29T10:14:58.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.030+0000 7f9975c00700 1 -- 172.21.15.12:0/3038154665 wait complete. 2024-03-29T10:14:58.030 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 64 2024-03-29T10:14:58.048 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:14:58.240 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:58.242+0000 7fc7e1153240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:14:58.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.293+0000 7f6a2f000700 1 Processor -- start 2024-03-29T10:14:58.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.293+0000 7f6a2f000700 1 -- start start 2024-03-29T10:14:58.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.294+0000 7f6a2f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 0x7f6a280ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:58.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.294+0000 7f6a2f000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6a280b6a20 0x7f6a280b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:58.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.294+0000 7f6a2f000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a280ceef0 con 0x7f6a28160be0 2024-03-29T10:14:58.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.294+0000 7f6a2f000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a280b65d0 con 0x7f6a280b6a20 2024-03-29T10:14:58.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.294+0000 7f6a2f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280d2a80 con 0x7f6a280ce590 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.295+0000 7f6a27400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 0x7f6a280ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.295+0000 7f6a2d000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6a280b6a20 0x7f6a280b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.295+0000 7f6a27e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f6a28160be0 0x7f6a280cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:59962/0 (socket says 172.21.15.12:59962) 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.295+0000 7f6a2d000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6a280b6a20 0x7f6a280b6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:57016/0 (socket says 172.21.15.12:57016) 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.295+0000 7f6a27e00700 1 -- 172.21.15.12:0/2280673637 learned_addr learned my addr 172.21.15.12:0/2280673637 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.295+0000 7f6a2d000700 1 -- 172.21.15.12:0/2280673637 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 msgr2=0x7f6a280ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.295+0000 7f6a2d000700 1 --2- 172.21.15.12:0/2280673637 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 0x7f6a280ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.295+0000 7f6a2d000700 1 -- 172.21.15.12:0/2280673637 >> v1:172.21.15.159:6789/0 conn(0x7f6a28160be0 legacy=0x7f6a280cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.296+0000 7f6a2d000700 1 -- 172.21.15.12:0/2280673637 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a28160fc0 con 0x7f6a280b6a20 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.295+0000 7f6a26a00700 1 -- 172.21.15.12:0/2280673637 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88127039 0 0) 0x7f6a280ceef0 con 0x7f6a28160be0 2024-03-29T10:14:58.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.296+0000 7f6a2d000700 1 --2- 172.21.15.12:0/2280673637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a280b6a20 0x7f6a280b6eb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009d90 tx=0x7f6a2000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4748b3914ee86aa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:58.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.296+0000 7f6a26a00700 1 -- 172.21.15.12:0/2280673637 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2000e070 con 0x7f6a280b6a20 2024-03-29T10:14:58.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.296+0000 7f6a26a00700 1 -- 172.21.15.12:0/2280673637 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2000bb20 con 0x7f6a280b6a20 2024-03-29T10:14:58.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.296+0000 7f6a26a00700 1 -- 172.21.15.12:0/2280673637 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20012430 con 0x7f6a280b6a20 2024-03-29T10:14:58.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.297+0000 7f6a2f000700 1 -- 172.21.15.12:0/2280673637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a280b6a20 msgr2=0x7f6a280b6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.297+0000 7f6a2f000700 1 --2- 172.21.15.12:0/2280673637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a280b6a20 0x7f6a280b6eb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009d90 tx=0x7f6a2000b330 comp rx=0 tx=0).stop 2024-03-29T10:14:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.297+0000 7f6a2f000700 1 -- 172.21.15.12:0/2280673637 shutdown_connections 2024-03-29T10:14:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.297+0000 7f6a2f000700 1 --2- 172.21.15.12:0/2280673637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a280b6a20 0x7f6a280b6eb0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.297+0000 7f6a2f000700 1 --2- 172.21.15.12:0/2280673637 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 0x7f6a280ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.297+0000 7f6a2f000700 1 -- 172.21.15.12:0/2280673637 >> 172.21.15.12:0/2280673637 conn(0x7f6a2808aff0 msgr2=0x7f6a280d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.297+0000 7f6a2f000700 1 -- 172.21.15.12:0/2280673637 shutdown_connections 2024-03-29T10:14:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.297+0000 7f6a2f000700 1 -- 172.21.15.12:0/2280673637 wait complete. 2024-03-29T10:14:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.299+0000 7f6a2f000700 1 Processor -- start 2024-03-29T10:14:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.300+0000 7f6a2f000700 1 -- start start 2024-03-29T10:14:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a2f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a280b6a20 0x7f6a281f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a2f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 0x7f6a280b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a2f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a28160be0 0x7f6a280b8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a2f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280b5160 con 0x7f6a28160be0 2024-03-29T10:14:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a2f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2809de40 con 0x7f6a280b6a20 2024-03-29T10:14:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a2f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280ca750 con 0x7f6a280ce590 2024-03-29T10:14:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a27e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a280b6a20 0x7f6a281f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a27400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 0x7f6a280b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:58.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a27e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a280b6a20 0x7f6a281f2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:57028/0 (socket says 172.21.15.12:57028) 2024-03-29T10:14:58.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a27e00700 1 -- 172.21.15.12:0/1709166374 learned_addr learned my addr 172.21.15.12:0/1709166374 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:58.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.301+0000 7f6a2d000700 1 --2- 172.21.15.12:0/1709166374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a28160be0 0x7f6a280b8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.302+0000 7f6a27400700 1 -- 172.21.15.12:0/1709166374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a280b6a20 msgr2=0x7f6a281f2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.302+0000 7f6a27400700 1 --2- 172.21.15.12:0/1709166374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a280b6a20 0x7f6a281f2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.302+0000 7f6a27400700 1 -- 172.21.15.12:0/1709166374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a28160be0 msgr2=0x7f6a280b8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.302+0000 7f6a27400700 1 --2- 172.21.15.12:0/1709166374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a28160be0 0x7f6a280b8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.302+0000 7f6a27400700 1 -- 172.21.15.12:0/1709166374 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a20009a70 con 0x7f6a280ce590 2024-03-29T10:14:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.302+0000 7f6a27400700 1 --2- 172.21.15.12:0/1709166374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 0x7f6a280b7e50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400e900 tx=0x7f6a1400ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.302+0000 7f6a1fe00700 1 -- 172.21.15.12:0/1709166374 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14013680 con 0x7f6a280ce590 2024-03-29T10:14:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.302+0000 7f6a1fe00700 1 -- 172.21.15.12:0/1709166374 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a140137e0 con 0x7f6a280ce590 2024-03-29T10:14:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.302+0000 7f6a2f000700 1 -- 172.21.15.12:0/1709166374 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a280caa30 con 0x7f6a280ce590 2024-03-29T10:14:58.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.302+0000 7f6a1fe00700 1 -- 172.21.15.12:0/1709166374 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14013b50 con 0x7f6a280ce590 2024-03-29T10:14:58.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.303+0000 7f6a2f000700 1 -- 172.21.15.12:0/1709166374 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a280caff0 con 0x7f6a280ce590 2024-03-29T10:14:58.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.304+0000 7f6a2f000700 1 -- 172.21.15.12:0/1709166374 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a28067720 con 0x7f6a280ce590 2024-03-29T10:14:58.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.305+0000 7f6a1fe00700 1 -- 172.21.15.12:0/1709166374 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 75) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f6a14013cb0 con 0x7f6a280ce590 2024-03-29T10:14:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.306+0000 7f6a1fe00700 1 -- 172.21.15.12:0/1709166374 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6354+0+0 (secure 0 0 0) 0x7f6a14088fa0 con 0x7f6a280ce590 2024-03-29T10:14:58.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.310+0000 7f6a1fe00700 1 -- 172.21.15.12:0/1709166374 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a14054340 con 0x7f6a280ce590 2024-03-29T10:14:58.363 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:14:58.365+0000 7fc7e1153240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:14:58.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.575+0000 7f6a2f000700 1 -- 172.21.15.12:0/1709166374 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a280b5cb0 con 0x7f6a280ce590 2024-03-29T10:14:58.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.576+0000 7f6a1fe00700 1 -- 172.21.15.12:0/1709166374 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f6a14017090 con 0x7f6a280ce590 2024-03-29T10:14:58.574 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:58.575 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1317631939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1317631939},{"type":"v1","addr":"172.21.15.159:6836","nonce":1317631939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1057731647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1057731647},{"type":"v1","addr":"172.21.15.159:6837","nonce":1057731647}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1567088347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1567088347},{"type":"v1","addr":"172.21.15.12:6834","nonce":1567088347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:14:25.846391+0000","modified":"2024-03-29T10:14:29.114278+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8406":{"gid":8406,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/3311625077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3311625077},{"type":"v1","addr":"172.21.15.12:6835","nonce":3311625077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-29T10:14:58.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.579+0000 7f6a2f000700 1 -- 172.21.15.12:0/1709166374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 msgr2=0x7f6a280b7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.579+0000 7f6a2f000700 1 --2- 172.21.15.12:0/1709166374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 0x7f6a280b7e50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400e900 tx=0x7f6a1400ec10 comp rx=0 tx=0).stop 2024-03-29T10:14:58.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.579+0000 7f6a2f000700 1 -- 172.21.15.12:0/1709166374 shutdown_connections 2024-03-29T10:14:58.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.579+0000 7f6a2f000700 1 --2- 172.21.15.12:0/1709166374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a28160be0 0x7f6a280b8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.579+0000 7f6a2f000700 1 --2- 172.21.15.12:0/1709166374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a280b6a20 0x7f6a281f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.579+0000 7f6a2f000700 1 --2- 172.21.15.12:0/1709166374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a280ce590 0x7f6a280b7e50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.579+0000 7f6a2f000700 1 -- 172.21.15.12:0/1709166374 >> 172.21.15.12:0/1709166374 conn(0x7f6a2808aff0 msgr2=0x7f6a280d16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:58.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.579+0000 7f6a2f000700 1 -- 172.21.15.12:0/1709166374 shutdown_connections 2024-03-29T10:14:58.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.579+0000 7f6a2f000700 1 -- 172.21.15.12:0/1709166374 wait complete. 2024-03-29T10:14:58.578 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 64 2024-03-29T10:14:58.596 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:14:58.781 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:14:58.785 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:14:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.797+0000 7f02cc000700 1 Processor -- start 2024-03-29T10:14:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.797+0000 7f02cc000700 1 -- start start 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.798+0000 7f02cc000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f02c40709c0 0x7f02c4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.798+0000 7f02cc000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c40b2180 con 0x7f02c4072580 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.798+0000 7f02cc000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c415b100 con 0x7f02c40b3af0 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.798+0000 7f02cc000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c415b3e0 con 0x7f02c40709c0 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.798+0000 7f02c8c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f02c4072580 0x7f02c40b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52382/0 (socket says 172.21.15.12:52382) 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.798+0000 7f02c8c00700 1 -- 172.21.15.12:0/2210361290 learned_addr learned my addr 172.21.15.12:0/2210361290 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.799+0000 7f02ca000700 1 --2- 172.21.15.12:0/2210361290 >> v2:172.21.15.159:3300/0 conn(0x7f02c40709c0 0x7f02c4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.799+0000 7f02c3e00700 1 -- 172.21.15.12:0/2210361290 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339174923 0 0) 0x7f02c40b2180 con 0x7f02c4072580 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.799+0000 7f02c3e00700 1 -- 172.21.15.12:0/2210361290 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ac0035b0 con 0x7f02c4072580 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.799+0000 7f02c3e00700 1 -- 172.21.15.12:0/2210361290 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233081693 0 0) 0x7f02c415b100 con 0x7f02c40b3af0 2024-03-29T10:14:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.799+0000 7f02c3e00700 1 -- 172.21.15.12:0/2210361290 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c40b2180 con 0x7f02c40b3af0 2024-03-29T10:14:58.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.799+0000 7f02ca000700 1 -- 172.21.15.12:0/2210361290 >> v1:172.21.15.159:6789/0 conn(0x7f02c40b3af0 legacy=0x7f02c415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.799+0000 7f02ca000700 1 -- 172.21.15.12:0/2210361290 >> v1:172.21.15.12:6789/0 conn(0x7f02c4072580 legacy=0x7f02c40b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.799+0000 7f02ca000700 1 -- 172.21.15.12:0/2210361290 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c40b23f0 con 0x7f02c40709c0 2024-03-29T10:14:58.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.799+0000 7f02ca000700 1 --2- 172.21.15.12:0/2210361290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c40709c0 0x7f02c4070e00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009d90 tx=0x7f02bc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cb499c616e61a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:58.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.800+0000 7f02c3e00700 1 -- 172.21.15.12:0/2210361290 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc00e070 con 0x7f02c40709c0 2024-03-29T10:14:58.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.800+0000 7f02c3e00700 1 -- 172.21.15.12:0/2210361290 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02bc00bb20 con 0x7f02c40709c0 2024-03-29T10:14:58.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.800+0000 7f02c3e00700 1 -- 172.21.15.12:0/2210361290 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc012430 con 0x7f02c40709c0 2024-03-29T10:14:58.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.800+0000 7f02cc000700 1 -- 172.21.15.12:0/2210361290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c40709c0 msgr2=0x7f02c4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.800+0000 7f02cc000700 1 --2- 172.21.15.12:0/2210361290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c40709c0 0x7f02c4070e00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009d90 tx=0x7f02bc00b330 comp rx=0 tx=0).stop 2024-03-29T10:14:58.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.801+0000 7f02cc000700 1 -- 172.21.15.12:0/2210361290 shutdown_connections 2024-03-29T10:14:58.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.801+0000 7f02cc000700 1 --2- 172.21.15.12:0/2210361290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c40709c0 0x7f02c4070e00 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.801+0000 7f02cc000700 1 -- 172.21.15.12:0/2210361290 >> 172.21.15.12:0/2210361290 conn(0x7f02c40cf190 msgr2=0x7f02c40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:58.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.801+0000 7f02cc000700 1 -- 172.21.15.12:0/2210361290 shutdown_connections 2024-03-29T10:14:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.801+0000 7f02cc000700 1 -- 172.21.15.12:0/2210361290 wait complete. 2024-03-29T10:14:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.802+0000 7f02cc000700 1 Processor -- start 2024-03-29T10:14:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.803+0000 7f02cc000700 1 -- start start 2024-03-29T10:14:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02cc000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02c40709c0 0x7f02c41576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02cc000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c4072580 0x7f02c40afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02cc000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02c40b3af0 0x7f02c40b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02cc000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40cb2e0 con 0x7f02c40709c0 2024-03-29T10:14:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02cc000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40ca520 con 0x7f02c4072580 2024-03-29T10:14:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02cc000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40ca800 con 0x7f02c40b3af0 2024-03-29T10:14:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02c9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02c40709c0 0x7f02c41576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02c9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02c40709c0 0x7f02c41576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36064/0 (socket says 172.21.15.12:36064) 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02c9600700 1 -- 172.21.15.12:0/3926515474 learned_addr learned my addr 172.21.15.12:0/3926515474 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02ca000700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02c40b3af0 0x7f02c40b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02c8c00700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c4072580 0x7f02c40afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02c9600700 1 -- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02c40b3af0 msgr2=0x7f02c40b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02c9600700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02c40b3af0 0x7f02c40b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02c9600700 1 -- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c4072580 msgr2=0x7f02c40afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02c9600700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c4072580 0x7f02c40afdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02c9600700 1 -- 172.21.15.12:0/3926515474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc009a70 con 0x7f02c40709c0 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.804+0000 7f02c8c00700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c4072580 0x7f02c40afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.805+0000 7f02c9600700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02c40709c0 0x7f02c41576e0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f02b800c500 tx=0x7f02b800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.805+0000 7f02c1600700 1 -- 172.21.15.12:0/3926515474 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b80052d0 con 0x7f02c40709c0 2024-03-29T10:14:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.805+0000 7f02c1600700 1 -- 172.21.15.12:0/3926515474 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b8005450 con 0x7f02c40709c0 2024-03-29T10:14:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.805+0000 7f02c1600700 1 -- 172.21.15.12:0/3926515474 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b80038c0 con 0x7f02c40709c0 2024-03-29T10:14:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.805+0000 7f02cc000700 1 -- 172.21.15.12:0/3926515474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c40caae0 con 0x7f02c40709c0 2024-03-29T10:14:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.805+0000 7f02cc000700 1 -- 172.21.15.12:0/3926515474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c4157c20 con 0x7f02c40709c0 2024-03-29T10:14:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.806+0000 7f02cc000700 1 -- 172.21.15.12:0/3926515474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c40cb000 con 0x7f02c40709c0 2024-03-29T10:14:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.807+0000 7f02c1600700 1 -- 172.21.15.12:0/3926515474 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 75) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f02b8003a20 con 0x7f02c40709c0 2024-03-29T10:14:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.809+0000 7f02c1600700 1 -- 172.21.15.12:0/3926515474 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6354+0+0 (secure 0 0 0) 0x7f02b8088440 con 0x7f02c40709c0 2024-03-29T10:14:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.814+0000 7f02c1600700 1 -- 172.21.15.12:0/3926515474 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f02c40cb000 con 0x7f02c40709c0 2024-03-29T10:14:58.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.884+0000 7f02c1600700 1 -- 172.21.15.12:0/3926515474 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 76) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f02b8053290 con 0x7f02c40709c0 2024-03-29T10:14:58.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.885+0000 7f02c1600700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02a4036150 0x7f02a4072310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:58.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.885+0000 7f02c8c00700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02a4036150 0x7f02a4072310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:58.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:58.888+0000 7f02c8c00700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02a4036150 0x7f02a4072310 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f02c4148e40 tx=0x7f02b0009040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:59.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.036+0000 7f02cc000700 1 -- 172.21.15.12:0/3926515474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02c40cb000 con 0x7f02c40709c0 2024-03-29T10:14:59.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.037+0000 7f02c1600700 1 -- 172.21.15.12:0/3926515474 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+16997 (secure 0 0 0) 0x7f02c40cb000 con 0x7f02c40709c0 2024-03-29T10:14:59.035 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:14:59.036 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":142,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:14:57.872116+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:14:23.708181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-29T10:14:24.829152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/991696097":"2024-03-30T10:14:57.872052+0000","172.21.15.159:0/1372646348":"2024-03-30T10:14:57.872052+0000","172.21.15.159:0/4087946115":"2024-03-30T10:14:57.872052+0000","172.21.15.159:0/205545561":"2024-03-30T10:14:57.872052+0000","172.21.15.159:0/1994155044":"2024-03-30T10:14:57.872052+0000","172.21.15.159:0/1348266098":"2024-03-30T10:14:57.872052+0000","172.21.15.159:0/3261410285":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/1049084973":"2024-03-30T10:14:57.872052+0000","172.21.15.159:0/2027263440":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/4127686095":"2024-03-30T10:14:42.823243+0000","172.21.15.159:6833/43444":"2024-03-30T10:14:42.823243+0000","172.21.15.159:6832/43444":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/3727946147":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2214680226":"2024-03-30T10:14:42.823243+0000","172.21.15.159:0/2899228198":"2024-03-30T10:14:42.823243+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:14:59.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 -- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02a4036150 msgr2=0x7f02a4072310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02a4036150 0x7f02a4072310 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f02c4148e40 tx=0x7f02b0009040 comp rx=0 tx=0).stop 2024-03-29T10:14:59.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 -- 172.21.15.12:0/3926515474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02c40709c0 msgr2=0x7f02c41576e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02c40709c0 0x7f02c41576e0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f02b800c500 tx=0x7f02b800c810 comp rx=0 tx=0).stop 2024-03-29T10:14:59.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 -- 172.21.15.12:0/3926515474 shutdown_connections 2024-03-29T10:14:59.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f02a4036150 0x7f02a4072310 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02c40b3af0 0x7f02c40b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02c40709c0 0x7f02c41576e0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 --2- 172.21.15.12:0/3926515474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c4072580 0x7f02c40afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 -- 172.21.15.12:0/3926515474 >> 172.21.15.12:0/3926515474 conn(0x7f02c40cf190 msgr2=0x7f02c406dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:59.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 -- 172.21.15.12:0/3926515474 shutdown_connections 2024-03-29T10:14:59.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.040+0000 7f02cc000700 1 -- 172.21.15.12:0/3926515474 wait complete. 2024-03-29T10:14:59.056 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T10:14:59.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.257+0000 7f14c0400700 1 Processor -- start 2024-03-29T10:14:59.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.258+0000 7f14c0400700 1 -- start start 2024-03-29T10:14:59.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.259+0000 7f14c0400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14b80cbb30 0x7f14b80cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.259+0000 7f14c0400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f14b80cd020 0x7f14b80cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.259+0000 7f14c0400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b80cc450 con 0x7f14b80b2070 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.259+0000 7f14c0400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b80cdae0 con 0x7f14b80cd020 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.259+0000 7f14c0400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b80cddc0 con 0x7f14b80cbb30 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.259+0000 7f14bd000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f14b80cd020 0x7f14b80cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.259+0000 7f14be400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f14b80b2070 0x7f14b806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46600/0 (socket says 172.21.15.12:46600) 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.259+0000 7f14be400700 1 -- 172.21.15.12:0/1795326134 learned_addr learned my addr 172.21.15.12:0/1795326134 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.260+0000 7f14bda00700 1 --2- 172.21.15.12:0/1795326134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14b80cbb30 0x7f14b80cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.260+0000 7f14bd000700 1 -- 172.21.15.12:0/1795326134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14b80cbb30 msgr2=0x7f14b80cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.260+0000 7f14bd000700 1 --2- 172.21.15.12:0/1795326134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14b80cbb30 0x7f14b80cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.260+0000 7f14bd000700 1 -- 172.21.15.12:0/1795326134 >> v1:172.21.15.12:6789/0 conn(0x7f14b80b2070 legacy=0x7f14b806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.260+0000 7f14bd000700 1 -- 172.21.15.12:0/1795326134 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b80ce0a0 con 0x7f14b80cd020 2024-03-29T10:14:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.260+0000 7f14bd000700 1 --2- 172.21.15.12:0/1795326134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 0x7f14b80cd4a0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009840 tx=0x7f14a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcaab41f6386fe52 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.260+0000 7f14b7e00700 1 -- 172.21.15.12:0/1795326134 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4011830 con 0x7f14b80cd020 2024-03-29T10:14:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.260+0000 7f14b7e00700 1 -- 172.21.15.12:0/1795326134 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a4011990 con 0x7f14b80cd020 2024-03-29T10:14:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.260+0000 7f14b7e00700 1 -- 172.21.15.12:0/1795326134 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4011d20 con 0x7f14b80cd020 2024-03-29T10:14:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.261+0000 7f14c0400700 1 -- 172.21.15.12:0/1795326134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 msgr2=0x7f14b80cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.261+0000 7f14c0400700 1 --2- 172.21.15.12:0/1795326134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 0x7f14b80cd4a0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009840 tx=0x7f14a4009b50 comp rx=0 tx=0).stop 2024-03-29T10:14:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.261+0000 7f14c0400700 1 -- 172.21.15.12:0/1795326134 shutdown_connections 2024-03-29T10:14:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.261+0000 7f14c0400700 1 --2- 172.21.15.12:0/1795326134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14b80cbb30 0x7f14b80cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.261+0000 7f14c0400700 1 --2- 172.21.15.12:0/1795326134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 0x7f14b80cd4a0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.261+0000 7f14c0400700 1 -- 172.21.15.12:0/1795326134 >> 172.21.15.12:0/1795326134 conn(0x7f14b80b0d70 msgr2=0x7f14b80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.261+0000 7f14c0400700 1 -- 172.21.15.12:0/1795326134 shutdown_connections 2024-03-29T10:14:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.261+0000 7f14c0400700 1 -- 172.21.15.12:0/1795326134 wait complete. 2024-03-29T10:14:59.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.263+0000 7f14c0400700 1 Processor -- start 2024-03-29T10:14:59.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.263+0000 7f14c0400700 1 -- start start 2024-03-29T10:14:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14c0400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14b80b2070 0x7f14b81474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14c0400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14b80cbb30 0x7f14b815a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14c0400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 0x7f14b815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14c0400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b80acc40 con 0x7f14b80cd020 2024-03-29T10:14:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14c0400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b80ac050 con 0x7f14b80cbb30 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14c0400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b80ac2a0 con 0x7f14b80b2070 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14be400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 0x7f14b815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14bda00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14b80b2070 0x7f14b81474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14bd000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14b80cbb30 0x7f14b815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14be400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 0x7f14b815abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40584/0 (socket says 172.21.15.12:40584) 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.264+0000 7f14be400700 1 -- 172.21.15.12:0/3038738231 learned_addr learned my addr 172.21.15.12:0/3038738231 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14be400700 1 -- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14b80b2070 msgr2=0x7f14b81474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14be400700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14b80b2070 0x7f14b81474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14be400700 1 -- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14b80cbb30 msgr2=0x7f14b815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14be400700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14b80cbb30 0x7f14b815a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14be400700 1 -- 172.21.15.12:0/3038738231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a40094a0 con 0x7f14b80cd020 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14bd000700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14b80cbb30 0x7f14b815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14be400700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 0x7f14b815abb0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f14b000ca80 tx=0x7f14b000cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14b5600700 1 -- 172.21.15.12:0/3038738231 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b000a9e0 con 0x7f14b80cd020 2024-03-29T10:14:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14b5600700 1 -- 172.21.15.12:0/3038738231 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b000ab40 con 0x7f14b80cd020 2024-03-29T10:14:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14b5600700 1 -- 172.21.15.12:0/3038738231 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0012400 con 0x7f14b80cd020 2024-03-29T10:14:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14c0400700 1 -- 172.21.15.12:0/3038738231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b80ac580 con 0x7f14b80cd020 2024-03-29T10:14:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.265+0000 7f14c0400700 1 -- 172.21.15.12:0/3038738231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b81479f0 con 0x7f14b80cd020 2024-03-29T10:14:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.266+0000 7f14c0400700 1 -- 172.21.15.12:0/3038738231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b80acd80 con 0x7f14b80cd020 2024-03-29T10:14:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.267+0000 7f14b5600700 1 -- 172.21.15.12:0/3038738231 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 76) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f14b001b3b0 con 0x7f14b80cd020 2024-03-29T10:14:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.268+0000 7f14b5600700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f149806aae0 0x7f149806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.268+0000 7f14b5600700 1 -- 172.21.15.12:0/3038738231 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6354+0+0 (secure 0 0 0) 0x7f14b0088d30 con 0x7f14b80cd020 2024-03-29T10:14:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.269+0000 7f14bda00700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f149806aae0 0x7f149806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.269+0000 7f14bda00700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f149806aae0 0x7f149806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f14ac006b20 tx=0x7f14ac008040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:59.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.273+0000 7f14b5600700 1 -- 172.21.15.12:0/3038738231 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f14b80acd80 con 0x7f14b80cd020 2024-03-29T10:14:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.532+0000 7f14c0400700 1 -- 172.21.15.12:0/3038738231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f14b80acd80 con 0x7f14b80cd020 2024-03-29T10:14:59.531 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T10:14:59.533+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T10:14:59.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.543+0000 7f14b5600700 1 -- 172.21.15.12:0/3038738231 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v65) v1 ==== 161+0+0 (secure 0 0 0) 0x7f14b80acd80 con 0x7f14b80cd020 2024-03-29T10:14:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 -- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f149806aae0 msgr2=0x7f149806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f149806aae0 0x7f149806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f14ac006b20 tx=0x7f14ac008040 comp rx=0 tx=0).stop 2024-03-29T10:14:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 -- 172.21.15.12:0/3038738231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 msgr2=0x7f14b815abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 0x7f14b815abb0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f14b000ca80 tx=0x7f14b000cd90 comp rx=0 tx=0).stop 2024-03-29T10:14:59.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 -- 172.21.15.12:0/3038738231 shutdown_connections 2024-03-29T10:14:59.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f149806aae0 0x7f149806cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14b80cd020 0x7f14b815abb0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14b80b2070 0x7f14b81474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 --2- 172.21.15.12:0/3038738231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14b80cbb30 0x7f14b815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 -- 172.21.15.12:0/3038738231 >> 172.21.15.12:0/3038738231 conn(0x7f14b80b0d70 msgr2=0x7f14b80b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:59.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 -- 172.21.15.12:0/3038738231 shutdown_connections 2024-03-29T10:14:59.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.547+0000 7f14c0400700 1 -- 172.21.15.12:0/3038738231 wait complete. 2024-03-29T10:14:59.546 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T10:14:59.559 INFO:tasks.ceph.mds.a.smithi012.stderr: -9835> 2024-03-29T10:14:30.298+0000 7f3752c00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:14:59.571 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T10:14:59.621 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:14:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.777+0000 7f7a84600700 1 Processor -- start 2024-03-29T10:14:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.777+0000 7f7a84600700 1 -- start start 2024-03-29T10:14:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.779+0000 7f7a84600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c160d70 0x7f7a7c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.779+0000 7f7a84600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7c161150 con 0x7f7a7c06a790 2024-03-29T10:14:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.779+0000 7f7a84600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7c06dba0 con 0x7f7a7c15ef70 2024-03-29T10:14:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.779+0000 7f7a84600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0b4cd0 con 0x7f7a7c160d70 2024-03-29T10:14:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.779+0000 7f7a82600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7a7c06a790 0x7f7a7c06e080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46618/0 (socket says 172.21.15.12:46618) 2024-03-29T10:14:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.779+0000 7f7a82600700 1 -- 172.21.15.12:0/3020942768 learned_addr learned my addr 172.21.15.12:0/3020942768 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:14:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.779+0000 7f7a81c00700 1 --2- 172.21.15.12:0/3020942768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c160d70 0x7f7a7c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.779+0000 7f7a7be00700 1 -- 172.21.15.12:0/3020942768 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672528849 0 0) 0x7f7a7c161150 con 0x7f7a7c06a790 2024-03-29T10:14:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.779+0000 7f7a7be00700 1 -- 172.21.15.12:0/3020942768 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a640035b0 con 0x7f7a7c06a790 2024-03-29T10:14:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.780+0000 7f7a7be00700 1 -- 172.21.15.12:0/3020942768 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776860848 0 0) 0x7f7a7c06dba0 con 0x7f7a7c15ef70 2024-03-29T10:14:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.780+0000 7f7a7be00700 1 -- 172.21.15.12:0/3020942768 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a7c161150 con 0x7f7a7c15ef70 2024-03-29T10:14:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.780+0000 7f7a81c00700 1 -- 172.21.15.12:0/3020942768 >> v1:172.21.15.159:6789/0 conn(0x7f7a7c15ef70 legacy=0x7f7a7c15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.780+0000 7f7a81c00700 1 -- 172.21.15.12:0/3020942768 >> v1:172.21.15.12:6789/0 conn(0x7f7a7c06a790 legacy=0x7f7a7c06e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.780+0000 7f7a81c00700 1 -- 172.21.15.12:0/3020942768 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c0b4ee0 con 0x7f7a7c160d70 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.780+0000 7f7a81c00700 1 --2- 172.21.15.12:0/3020942768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c160d70 0x7f7a7c0b3200 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7a70009870 tx=0x7f7a70009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7407d8e18e41d4f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.780+0000 7f7a7be00700 1 -- 172.21.15.12:0/3020942768 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70011830 con 0x7f7a7c160d70 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.780+0000 7f7a7be00700 1 -- 172.21.15.12:0/3020942768 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a70011990 con 0x7f7a7c160d70 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.780+0000 7f7a7be00700 1 -- 172.21.15.12:0/3020942768 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70011d20 con 0x7f7a7c160d70 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.781+0000 7f7a84600700 1 -- 172.21.15.12:0/3020942768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c160d70 msgr2=0x7f7a7c0b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.781+0000 7f7a84600700 1 --2- 172.21.15.12:0/3020942768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c160d70 0x7f7a7c0b3200 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7a70009870 tx=0x7f7a70009b80 comp rx=0 tx=0).stop 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.781+0000 7f7a84600700 1 -- 172.21.15.12:0/3020942768 shutdown_connections 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.781+0000 7f7a84600700 1 --2- 172.21.15.12:0/3020942768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c160d70 0x7f7a7c0b3200 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.781+0000 7f7a84600700 1 -- 172.21.15.12:0/3020942768 >> 172.21.15.12:0/3020942768 conn(0x7f7a7c0b5720 msgr2=0x7f7a7c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.781+0000 7f7a84600700 1 -- 172.21.15.12:0/3020942768 shutdown_connections 2024-03-29T10:14:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.781+0000 7f7a84600700 1 -- 172.21.15.12:0/3020942768 wait complete. 2024-03-29T10:14:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.783+0000 7f7a84600700 1 Processor -- start 2024-03-29T10:14:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.783+0000 7f7a84600700 1 -- start start 2024-03-29T10:14:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a84600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c06a790 0x7f7a7c15c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a84600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c15ef70 0x7f7a7c13fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a84600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c160d70 0x7f7a7c141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a84600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c141440 con 0x7f7a7c160d70 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a84600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c141860 con 0x7f7a7c06a790 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a84600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c15cd50 con 0x7f7a7c15ef70 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a82600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c160d70 0x7f7a7c141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a82600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c160d70 0x7f7a7c141e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40608/0 (socket says 172.21.15.12:40608) 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a82600700 1 -- 172.21.15.12:0/77706424 learned_addr learned my addr 172.21.15.12:0/77706424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a81c00700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c06a790 0x7f7a7c15c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a81200700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c15ef70 0x7f7a7c13fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.784+0000 7f7a82600700 1 -- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c15ef70 msgr2=0x7f7a7c13fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a82600700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c15ef70 0x7f7a7c13fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a82600700 1 -- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c06a790 msgr2=0x7f7a7c15c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:14:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a82600700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c06a790 0x7f7a7c15c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:14:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a82600700 1 -- 172.21.15.12:0/77706424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a700094a0 con 0x7f7a7c160d70 2024-03-29T10:14:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a81200700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c15ef70 0x7f7a7c13fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:14:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a82600700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c160d70 0x7f7a7c141e30 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400cbd0 tx=0x7f7a7400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a79600700 1 -- 172.21.15.12:0/77706424 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7400aa60 con 0x7f7a7c160d70 2024-03-29T10:14:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a79600700 1 -- 172.21.15.12:0/77706424 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7400abc0 con 0x7f7a7c160d70 2024-03-29T10:14:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a84600700 1 -- 172.21.15.12:0/77706424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7c1407c0 con 0x7f7a7c160d70 2024-03-29T10:14:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a79600700 1 -- 172.21.15.12:0/77706424 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74012400 con 0x7f7a7c160d70 2024-03-29T10:14:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.785+0000 7f7a84600700 1 -- 172.21.15.12:0/77706424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7c140d80 con 0x7f7a7c160d70 2024-03-29T10:14:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.786+0000 7f7a84600700 1 -- 172.21.15.12:0/77706424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7c141580 con 0x7f7a7c160d70 2024-03-29T10:14:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.787+0000 7f7a79600700 1 -- 172.21.15.12:0/77706424 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 76) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f7a7401b3b0 con 0x7f7a7c160d70 2024-03-29T10:14:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.788+0000 7f7a79600700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a5c06aae0 0x7f7a5c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:14:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.788+0000 7f7a79600700 1 -- 172.21.15.12:0/77706424 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f7a74088ff0 con 0x7f7a7c160d70 2024-03-29T10:14:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.788+0000 7f7a81c00700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a5c06aae0 0x7f7a5c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:14:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.789+0000 7f7a81c00700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a5c06aae0 0x7f7a5c06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7a700097c0 tx=0x7f7a7001c040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:14:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:14:59.793+0000 7f7a79600700 1 -- 172.21.15.12:0/77706424 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7a740bc580 con 0x7f7a7c160d70 2024-03-29T10:15:00.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.062+0000 7f7a84600700 1 -- 172.21.15.12:0/77706424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7a7c067720 con 0x7f7a7c160d70 2024-03-29T10:15:00.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.548+0000 7f7a79600700 1 -- 172.21.15.12:0/77706424 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v66) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7a7c141580 con 0x7f7a7c160d70 2024-03-29T10:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 -- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a5c06aae0 msgr2=0x7f7a5c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a5c06aae0 0x7f7a5c06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7a700097c0 tx=0x7f7a7001c040 comp rx=0 tx=0).stop 2024-03-29T10:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 -- 172.21.15.12:0/77706424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c160d70 msgr2=0x7f7a7c141e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c160d70 0x7f7a7c141e30 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400cbd0 tx=0x7f7a7400cee0 comp rx=0 tx=0).stop 2024-03-29T10:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 -- 172.21.15.12:0/77706424 shutdown_connections 2024-03-29T10:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7a5c06aae0 0x7f7a5c06cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c160d70 0x7f7a7c141e30 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a7c06a790 0x7f7a7c15c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 --2- 172.21.15.12:0/77706424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a7c15ef70 0x7f7a7c13fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 -- 172.21.15.12:0/77706424 >> 172.21.15.12:0/77706424 conn(0x7f7a7c0b5720 msgr2=0x7f7a7c145d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 -- 172.21.15.12:0/77706424 shutdown_connections 2024-03-29T10:15:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.551+0000 7f7a84600700 1 -- 172.21.15.12:0/77706424 wait complete. 2024-03-29T10:15:00.571 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T10:15:00.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.777+0000 7ff80c000700 1 Processor -- start 2024-03-29T10:15:00.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.777+0000 7ff80c000700 1 -- start start 2024-03-29T10:15:00.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.778+0000 7ff80c000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff8040d0bd0 0x7ff80415e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:00.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.778+0000 7ff80c000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff80406aea0 0x7ff80406dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:00.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.778+0000 7ff80c000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80415ee10 con 0x7ff8040b3120 2024-03-29T10:15:00.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.778+0000 7ff80c000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80415de10 con 0x7ff8040d0bd0 2024-03-29T10:15:00.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.778+0000 7ff80c000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8040b4fd0 con 0x7ff80406aea0 2024-03-29T10:15:00.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.778+0000 7ff809600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff8040d0bd0 0x7ff80415e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:00.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.778+0000 7ff808c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff8040b3120 0x7ff80406a790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46634/0 (socket says 172.21.15.12:46634) 2024-03-29T10:15:00.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.778+0000 7ff808c00700 1 -- 172.21.15.12:0/2574480657 learned_addr learned my addr 172.21.15.12:0/2574480657 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:00.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.779+0000 7ff80a000700 1 --2- 172.21.15.12:0/2574480657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff80406aea0 0x7ff80406dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:00.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.779+0000 7ff809600700 1 -- 172.21.15.12:0/2574480657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff80406aea0 msgr2=0x7ff80406dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:00.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.779+0000 7ff809600700 1 --2- 172.21.15.12:0/2574480657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff80406aea0 0x7ff80406dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:00.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.779+0000 7ff809600700 1 -- 172.21.15.12:0/2574480657 >> v1:172.21.15.12:6789/0 conn(0x7ff8040b3120 legacy=0x7ff80406a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:00.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.779+0000 7ff809600700 1 -- 172.21.15.12:0/2574480657 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8040b52b0 con 0x7ff8040d0bd0 2024-03-29T10:15:00.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.779+0000 7ff809600700 1 --2- 172.21.15.12:0/2574480657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 0x7ff80415e8d0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8009840 tx=0x7ff7f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae544c9a0793b286 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:00.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.779+0000 7ff803e00700 1 -- 172.21.15.12:0/2574480657 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8011830 con 0x7ff8040d0bd0 2024-03-29T10:15:00.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.779+0000 7ff803e00700 1 -- 172.21.15.12:0/2574480657 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f8011990 con 0x7ff8040d0bd0 2024-03-29T10:15:00.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.780+0000 7ff803e00700 1 -- 172.21.15.12:0/2574480657 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8011d20 con 0x7ff8040d0bd0 2024-03-29T10:15:00.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.780+0000 7ff80c000700 1 -- 172.21.15.12:0/2574480657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 msgr2=0x7ff80415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:00.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.780+0000 7ff80c000700 1 --2- 172.21.15.12:0/2574480657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 0x7ff80415e8d0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8009840 tx=0x7ff7f8009b50 comp rx=0 tx=0).stop 2024-03-29T10:15:00.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.780+0000 7ff80c000700 1 -- 172.21.15.12:0/2574480657 shutdown_connections 2024-03-29T10:15:00.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.780+0000 7ff80c000700 1 --2- 172.21.15.12:0/2574480657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff80406aea0 0x7ff80406dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:00.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.780+0000 7ff80c000700 1 --2- 172.21.15.12:0/2574480657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 0x7ff80415e8d0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:00.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.780+0000 7ff80c000700 1 -- 172.21.15.12:0/2574480657 >> 172.21.15.12:0/2574480657 conn(0x7ff8040b5640 msgr2=0x7ff8040b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:00.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.780+0000 7ff80c000700 1 -- 172.21.15.12:0/2574480657 shutdown_connections 2024-03-29T10:15:00.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.780+0000 7ff80c000700 1 -- 172.21.15.12:0/2574480657 wait complete. 2024-03-29T10:15:00.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.782+0000 7ff80c000700 1 Processor -- start 2024-03-29T10:15:00.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.782+0000 7ff80c000700 1 -- start start 2024-03-29T10:15:00.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff80c000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff80406aea0 0x7ff8040d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:00.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff80c000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8040b3120 0x7ff804158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:00.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff80c000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 0x7ff80409bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:00.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff80c000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80409b4e0 con 0x7ff8040d0bd0 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff80c000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80409a860 con 0x7ff8040b3120 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff80c000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80409ab40 con 0x7ff80406aea0 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff80a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 0x7ff80409bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff80a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 0x7ff80409bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40630/0 (socket says 172.21.15.12:40630) 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff80a000700 1 -- 172.21.15.12:0/1586757029 learned_addr learned my addr 172.21.15.12:0/1586757029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff809600700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff80406aea0 0x7ff8040d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.783+0000 7ff808c00700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8040b3120 0x7ff804158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.784+0000 7ff80a000700 1 -- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff80406aea0 msgr2=0x7ff8040d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.784+0000 7ff80a000700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff80406aea0 0x7ff8040d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.784+0000 7ff80a000700 1 -- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8040b3120 msgr2=0x7ff804158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.784+0000 7ff80a000700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8040b3120 0x7ff804158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.784+0000 7ff80a000700 1 -- 172.21.15.12:0/1586757029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f80094a0 con 0x7ff8040d0bd0 2024-03-29T10:15:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.784+0000 7ff80a000700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 0x7ff80409bfb0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc007e30 tx=0x7ff7fc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.784+0000 7ff801600700 1 -- 172.21.15.12:0/1586757029 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc00ed30 con 0x7ff8040d0bd0 2024-03-29T10:15:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.784+0000 7ff801600700 1 -- 172.21.15.12:0/1586757029 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc00ee90 con 0x7ff8040d0bd0 2024-03-29T10:15:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.784+0000 7ff801600700 1 -- 172.21.15.12:0/1586757029 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc014660 con 0x7ff8040d0bd0 2024-03-29T10:15:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.784+0000 7ff80c000700 1 -- 172.21.15.12:0/1586757029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80409ae20 con 0x7ff8040d0bd0 2024-03-29T10:15:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.785+0000 7ff80c000700 1 -- 172.21.15.12:0/1586757029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff804157380 con 0x7ff8040d0bd0 2024-03-29T10:15:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.786+0000 7ff80c000700 1 -- 172.21.15.12:0/1586757029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80409b620 con 0x7ff8040d0bd0 2024-03-29T10:15:00.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.786+0000 7ff801600700 1 -- 172.21.15.12:0/1586757029 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 77) v1 ==== 89183+0+0 (secure 0 0 0) 0x7ff7fc0147c0 con 0x7ff8040d0bd0 2024-03-29T10:15:00.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.787+0000 7ff801600700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7e406ad20 0x7ff7e406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:00.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.787+0000 7ff801600700 1 -- 172.21.15.12:0/1586757029 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6440+0+0 (secure 0 0 0) 0x7ff7fc089cf0 con 0x7ff8040d0bd0 2024-03-29T10:15:00.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.788+0000 7ff809600700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7e406ad20 0x7ff7e406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:00.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.788+0000 7ff809600700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7e406ad20 0x7ff7e406d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80090b0 tx=0x7ff7f801c040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:00.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:00.792+0000 7ff801600700 1 -- 172.21.15.12:0/1586757029 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff80409b620 con 0x7ff8040d0bd0 2024-03-29T10:15:01.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.004+0000 7ff80c000700 1 -- 172.21.15.12:0/1586757029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff80409b620 con 0x7ff8040d0bd0 2024-03-29T10:15:01.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.894+0000 7ff801600700 1 -- 172.21.15.12:0/1586757029 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v144) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff80409b620 con 0x7ff8040d0bd0 2024-03-29T10:15:01.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.898+0000 7ff80c000700 1 -- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7e406ad20 msgr2=0x7ff7e406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:01.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.898+0000 7ff80c000700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7e406ad20 0x7ff7e406d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80090b0 tx=0x7ff7f801c040 comp rx=0 tx=0).stop 2024-03-29T10:15:01.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.898+0000 7ff80c000700 1 -- 172.21.15.12:0/1586757029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 msgr2=0x7ff80409bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:01.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.898+0000 7ff80c000700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 0x7ff80409bfb0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc007e30 tx=0x7ff7fc00e3c0 comp rx=0 tx=0).stop 2024-03-29T10:15:01.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.898+0000 7ff80c000700 1 -- 172.21.15.12:0/1586757029 shutdown_connections 2024-03-29T10:15:01.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.898+0000 7ff80c000700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff7e406ad20 0x7ff7e406d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:01.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.898+0000 7ff80c000700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8040d0bd0 0x7ff80409bfb0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:01.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.898+0000 7ff80c000700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff80406aea0 0x7ff8040d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:01.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.898+0000 7ff80c000700 1 --2- 172.21.15.12:0/1586757029 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8040b3120 0x7ff804158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:01.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.898+0000 7ff80c000700 1 -- 172.21.15.12:0/1586757029 >> 172.21.15.12:0/1586757029 conn(0x7ff8040b5640 msgr2=0x7ff8040b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:01.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.899+0000 7ff80c000700 1 -- 172.21.15.12:0/1586757029 shutdown_connections 2024-03-29T10:15:01.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:01.899+0000 7ff80c000700 1 -- 172.21.15.12:0/1586757029 wait complete. 2024-03-29T10:15:01.898 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T10:15:01.918 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T10:15:02.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.134+0000 7f7162e00700 1 Processor -- start 2024-03-29T10:15:02.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.134+0000 7f7162e00700 1 -- start start 2024-03-29T10:15:02.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.135+0000 7f7162e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f715c15d930 0x7f715c0d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:02.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.135+0000 7f7162e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f715c162030 0x7f715c0d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:02.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.135+0000 7f7162e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c0d2520 con 0x7f715c0d20d0 2024-03-29T10:15:02.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.135+0000 7f7162e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715c15ec40 con 0x7f715c162030 2024-03-29T10:15:02.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.135+0000 7f7162e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715c15dd10 con 0x7f715c15d930 2024-03-29T10:15:02.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.136+0000 7f715b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f715c162030 0x7f715c0d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:02.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.136+0000 7f715b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f715c162030 0x7f715c0d1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40644/0 (socket says 172.21.15.12:40644) 2024-03-29T10:15:02.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.136+0000 7f715b400700 1 -- 172.21.15.12:0/3909581050 learned_addr learned my addr 172.21.15.12:0/3909581050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:02.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.136+0000 7f715be00700 1 --2- 172.21.15.12:0/3909581050 >> v2:172.21.15.159:3300/0 conn(0x7f715c15d930 0x7f715c0d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:02.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.136+0000 7f715b400700 1 -- 172.21.15.12:0/3909581050 >> v2:172.21.15.159:3300/0 conn(0x7f715c15d930 msgr2=0x7f715c0d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:02.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.136+0000 7f715b400700 1 --2- 172.21.15.12:0/3909581050 >> v2:172.21.15.159:3300/0 conn(0x7f715c15d930 0x7f715c0d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:02.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.136+0000 7f715b400700 1 -- 172.21.15.12:0/3909581050 >> v1:172.21.15.159:6789/0 conn(0x7f715c0d20d0 legacy=0x7f715c0cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:02.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.136+0000 7f715b400700 1 -- 172.21.15.12:0/3909581050 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c15dfc0 con 0x7f715c162030 2024-03-29T10:15:02.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.136+0000 7f715b400700 1 --2- 172.21.15.12:0/3909581050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c162030 0x7f715c0d1b90 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f715c0a1430 tx=0x7f7148009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c673c6b89c4e58c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f715aa00700 1 -- 172.21.15.12:0/3909581050 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148004230 con 0x7f715c162030 2024-03-29T10:15:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f715aa00700 1 -- 172.21.15.12:0/3909581050 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7148004390 con 0x7f715c162030 2024-03-29T10:15:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f715aa00700 1 -- 172.21.15.12:0/3909581050 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71480036a0 con 0x7f715c162030 2024-03-29T10:15:02.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f7162e00700 1 -- 172.21.15.12:0/3909581050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c162030 msgr2=0x7f715c0d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:02.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f7162e00700 1 --2- 172.21.15.12:0/3909581050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c162030 0x7f715c0d1b90 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f715c0a1430 tx=0x7f7148009bd0 comp rx=0 tx=0).stop 2024-03-29T10:15:02.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f7162e00700 1 -- 172.21.15.12:0/3909581050 shutdown_connections 2024-03-29T10:15:02.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f7162e00700 1 --2- 172.21.15.12:0/3909581050 >> v2:172.21.15.159:3300/0 conn(0x7f715c15d930 0x7f715c0d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:02.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f7162e00700 1 --2- 172.21.15.12:0/3909581050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c162030 0x7f715c0d1b90 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:02.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f7162e00700 1 -- 172.21.15.12:0/3909581050 >> 172.21.15.12:0/3909581050 conn(0x7f715c08aff0 msgr2=0x7f715c0d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:02.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f7162e00700 1 -- 172.21.15.12:0/3909581050 shutdown_connections 2024-03-29T10:15:02.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.137+0000 7f7162e00700 1 -- 172.21.15.12:0/3909581050 wait complete. 2024-03-29T10:15:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.139+0000 7f7162e00700 1 Processor -- start 2024-03-29T10:15:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.139+0000 7f7162e00700 1 -- start start 2024-03-29T10:15:02.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.140+0000 7f7162e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c0d20d0 0x7f715c14f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:02.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.140+0000 7f7162e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f715c15d930 0x7f715c160ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:02.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.140+0000 7f7162e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f715c162030 0x7f715c0b1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:02.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.140+0000 7f7162e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c15f4d0 con 0x7f715c0d20d0 2024-03-29T10:15:02.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.140+0000 7f7162e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c15f7b0 con 0x7f715c15d930 2024-03-29T10:15:02.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.140+0000 7f7162e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c15fa90 con 0x7f715c162030 2024-03-29T10:15:02.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.140+0000 7f715be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c0d20d0 0x7f715c14f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:02.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.140+0000 7f715be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c0d20d0 0x7f715c14f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40646/0 (socket says 172.21.15.12:40646) 2024-03-29T10:15:02.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.140+0000 7f715be00700 1 -- 172.21.15.12:0/3481474230 learned_addr learned my addr 172.21.15.12:0/3481474230 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f715b400700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f715c15d930 0x7f715c160ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f7160e00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f715c162030 0x7f715c0b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f715be00700 1 -- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f715c162030 msgr2=0x7f715c0b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f715be00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f715c162030 0x7f715c0b1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f715be00700 1 -- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f715c15d930 msgr2=0x7f715c160ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f715be00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f715c15d930 0x7f715c160ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f715be00700 1 -- 172.21.15.12:0/3481474230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7148009550 con 0x7f715c0d20d0 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f715be00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c0d20d0 0x7f715c14f5c0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f714c00e810 tx=0x7f714c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f7153e00700 1 -- 172.21.15.12:0/3481474230 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c013610 con 0x7f715c0d20d0 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f7153e00700 1 -- 172.21.15.12:0/3481474230 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c013770 con 0x7f715c0d20d0 2024-03-29T10:15:02.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f7153e00700 1 -- 172.21.15.12:0/3481474230 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c013b00 con 0x7f715c0d20d0 2024-03-29T10:15:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.141+0000 7f7162e00700 1 -- 172.21.15.12:0/3481474230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715c15fd70 con 0x7f715c0d20d0 2024-03-29T10:15:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.142+0000 7f7162e00700 1 -- 172.21.15.12:0/3481474230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715c161020 con 0x7f715c0d20d0 2024-03-29T10:15:02.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.143+0000 7f7162e00700 1 -- 172.21.15.12:0/3481474230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c067720 con 0x7f715c0d20d0 2024-03-29T10:15:02.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.143+0000 7f7153e00700 1 -- 172.21.15.12:0/3481474230 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 77) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f714c013c60 con 0x7f715c0d20d0 2024-03-29T10:15:02.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.145+0000 7f7153e00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f713c06ad20 0x7f713c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:02.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.145+0000 7f7153e00700 1 -- 172.21.15.12:0/3481474230 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f714c089290 con 0x7f715c0d20d0 2024-03-29T10:15:02.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.145+0000 7f715b400700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f713c06ad20 0x7f713c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:02.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.146+0000 7f715b400700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f713c06ad20 0x7f713c06d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7148009130 tx=0x7f7148008030 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:02.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.149+0000 7f7153e00700 1 -- 172.21.15.12:0/3481474230 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f714c054250 con 0x7f715c0d20d0 2024-03-29T10:15:02.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.157+0000 7f7153e00700 1 -- 172.21.15.12:0/3481474230 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f714c054620 con 0x7f715c0d20d0 2024-03-29T10:15:02.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:02.367+0000 7f7162e00700 1 -- 172.21.15.12:0/3481474230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f715c0b3d30 con 0x7f715c0d20d0 2024-03-29T10:15:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.157+0000 7f7153e00700 1 -- 172.21.15.12:0/3481474230 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v145) v1 ==== 169+0+0 (secure 0 0 0) 0x7f714c057ce0 con 0x7f715c0d20d0 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.161+0000 7f7162e00700 1 -- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f713c06ad20 msgr2=0x7f713c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.161+0000 7f7162e00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f713c06ad20 0x7f713c06d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7148009130 tx=0x7f7148008030 comp rx=0 tx=0).stop 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.161+0000 7f7162e00700 1 -- 172.21.15.12:0/3481474230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c0d20d0 msgr2=0x7f715c14f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.161+0000 7f7162e00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c0d20d0 0x7f715c14f5c0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f714c00e810 tx=0x7f714c00ebd0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.162+0000 7f7162e00700 1 -- 172.21.15.12:0/3481474230 shutdown_connections 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.162+0000 7f7162e00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f713c06ad20 0x7f713c06d1e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.162+0000 7f7162e00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f715c162030 0x7f715c0b1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.162+0000 7f7162e00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f715c0d20d0 0x7f715c14f5c0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.162+0000 7f7162e00700 1 --2- 172.21.15.12:0/3481474230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f715c15d930 0x7f715c160ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.162+0000 7f7162e00700 1 -- 172.21.15.12:0/3481474230 >> 172.21.15.12:0/3481474230 conn(0x7f715c08aff0 msgr2=0x7f715c15cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.162+0000 7f7162e00700 1 -- 172.21.15.12:0/3481474230 shutdown_connections 2024-03-29T10:15:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.162+0000 7f7162e00700 1 -- 172.21.15.12:0/3481474230 wait complete. 2024-03-29T10:15:03.162 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T10:15:03.184 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T10:15:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.390+0000 7f1fce400700 1 Processor -- start 2024-03-29T10:15:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.390+0000 7f1fce400700 1 -- start start 2024-03-29T10:15:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.391+0000 7f1fce400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1fc80d16a0 0x7f1fc80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.391+0000 7f1fce400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc8161420 con 0x7f1fc815d9d0 2024-03-29T10:15:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.391+0000 7f1fce400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc8161700 con 0x7f1fc80d2370 2024-03-29T10:15:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.391+0000 7f1fce400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc80d4340 con 0x7f1fc80d16a0 2024-03-29T10:15:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.392+0000 7f1fc6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1fc80d16a0 0x7f1fc80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.392+0000 7f1fc6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1fc80d16a0 0x7f1fc80d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40648/0 (socket says 172.21.15.12:40648) 2024-03-29T10:15:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.392+0000 7f1fc6a00700 1 -- 172.21.15.12:0/1453111450 learned_addr learned my addr 172.21.15.12:0/1453111450 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.392+0000 7f1fc7400700 1 --1- 172.21.15.12:0/1453111450 >> v1:172.21.15.12:6789/0 conn(0x7f1fc815d9d0 0x7f1fc80d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46648/0 (socket says 172.21.15.12:46648) 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.392+0000 7f1fc6a00700 1 -- 172.21.15.12:0/1453111450 >> v1:172.21.15.159:6789/0 conn(0x7f1fc80d2370 legacy=0x7f1fc80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.392+0000 7f1fc6a00700 1 -- 172.21.15.12:0/1453111450 >> v1:172.21.15.12:6789/0 conn(0x7f1fc815d9d0 legacy=0x7f1fc80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.392+0000 7f1fc6a00700 1 -- 172.21.15.12:0/1453111450 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc81619e0 con 0x7f1fc80d16a0 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.392+0000 7f1fc6000700 1 -- 172.21.15.12:0/1453111450 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325582953 0 0) 0x7f1fc8161420 con 0x7f1fc815d9d0 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.392+0000 7f1fc6a00700 1 --2- 172.21.15.12:0/1453111450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc80d16a0 0x7f1fc80d1e30 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4006fd0 tx=0x7f1fb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ece5b66406ca2da server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.393+0000 7f1fc6000700 1 -- 172.21.15.12:0/1453111450 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4005e30 con 0x7f1fc80d16a0 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.393+0000 7f1fc6000700 1 -- 172.21.15.12:0/1453111450 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb4004030 con 0x7f1fc80d16a0 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.393+0000 7f1fc6000700 1 -- 172.21.15.12:0/1453111450 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb40043e0 con 0x7f1fc80d16a0 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.393+0000 7f1fce400700 1 -- 172.21.15.12:0/1453111450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc80d16a0 msgr2=0x7f1fc80d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.393+0000 7f1fce400700 1 --2- 172.21.15.12:0/1453111450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc80d16a0 0x7f1fc80d1e30 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4006fd0 tx=0x7f1fb4009bd0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.393+0000 7f1fce400700 1 -- 172.21.15.12:0/1453111450 shutdown_connections 2024-03-29T10:15:03.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.393+0000 7f1fce400700 1 --2- 172.21.15.12:0/1453111450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc80d16a0 0x7f1fc80d1e30 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.393+0000 7f1fce400700 1 -- 172.21.15.12:0/1453111450 >> 172.21.15.12:0/1453111450 conn(0x7f1fc808aff0 msgr2=0x7f1fc80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:03.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.393+0000 7f1fce400700 1 -- 172.21.15.12:0/1453111450 shutdown_connections 2024-03-29T10:15:03.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.393+0000 7f1fce400700 1 -- 172.21.15.12:0/1453111450 wait complete. 2024-03-29T10:15:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.395+0000 7f1fce400700 1 Processor -- start 2024-03-29T10:15:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.395+0000 7f1fce400700 1 -- start start 2024-03-29T10:15:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.396+0000 7f1fce400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1fc80d16a0 0x7f1fc80b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.396+0000 7f1fce400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1fc80d2370 0x7f1fc815f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.396+0000 7f1fce400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc815d9d0 0x7f1fc81550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.396+0000 7f1fce400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8160810 con 0x7f1fc815d9d0 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.396+0000 7f1fce400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc815fc00 con 0x7f1fc80d16a0 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.396+0000 7f1fce400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc815fee0 con 0x7f1fc80d2370 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.396+0000 7f1fc7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc815d9d0 0x7f1fc81550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.396+0000 7f1fc7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc815d9d0 0x7f1fc81550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40652/0 (socket says 172.21.15.12:40652) 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.396+0000 7f1fc7e00700 1 -- 172.21.15.12:0/1303386420 learned_addr learned my addr 172.21.15.12:0/1303386420 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fc7400700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1fc80d16a0 0x7f1fc80b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fc6a00700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1fc80d2370 0x7f1fc815f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fc7e00700 1 -- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1fc80d2370 msgr2=0x7f1fc815f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fc7e00700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1fc80d2370 0x7f1fc815f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fc7e00700 1 -- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1fc80d16a0 msgr2=0x7f1fc80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fc7e00700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1fc80d16a0 0x7f1fc80b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fc7e00700 1 -- 172.21.15.12:0/1303386420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4009500 con 0x7f1fc815d9d0 2024-03-29T10:15:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fc7e00700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc815d9d0 0x7f1fc81550f0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000cc50 tx=0x7f1fc000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fbf400700 1 -- 172.21.15.12:0/1303386420 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0003c00 con 0x7f1fc815d9d0 2024-03-29T10:15:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fbf400700 1 -- 172.21.15.12:0/1303386420 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc0003d60 con 0x7f1fc815d9d0 2024-03-29T10:15:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fce400700 1 -- 172.21.15.12:0/1303386420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc81601c0 con 0x7f1fc815d9d0 2024-03-29T10:15:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.397+0000 7f1fbf400700 1 -- 172.21.15.12:0/1303386420 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc000ab60 con 0x7f1fc815d9d0 2024-03-29T10:15:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.398+0000 7f1fce400700 1 -- 172.21.15.12:0/1303386420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc8155f50 con 0x7f1fc815d9d0 2024-03-29T10:15:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.398+0000 7f1fce400700 1 -- 172.21.15.12:0/1303386420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc80d1ca0 con 0x7f1fc815d9d0 2024-03-29T10:15:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.399+0000 7f1fbf400700 1 -- 172.21.15.12:0/1303386420 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1fc000acc0 con 0x7f1fc815d9d0 2024-03-29T10:15:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.400+0000 7f1fbf400700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1fa806adb0 0x7f1fa806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.400+0000 7f1fbf400700 1 -- 172.21.15.12:0/1303386420 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f1fc0089540 con 0x7f1fc815d9d0 2024-03-29T10:15:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.401+0000 7f1fc7400700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1fa806adb0 0x7f1fa806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.402+0000 7f1fc7400700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1fa806adb0 0x7f1fa806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8007c40 tx=0x7f1fb8007b90 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.405+0000 7f1fbf400700 1 -- 172.21.15.12:0/1303386420 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1fc00545e0 con 0x7f1fc815d9d0 2024-03-29T10:15:03.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.620+0000 7f1fce400700 1 -- 172.21.15.12:0/1303386420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1fc8067720 con 0x7f1fc815d9d0 2024-03-29T10:15:03.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.624+0000 7f1fbf400700 1 -- 172.21.15.12:0/1303386420 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v49) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1fc0087e60 con 0x7f1fc815d9d0 2024-03-29T10:15:03.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 -- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1fa806adb0 msgr2=0x7f1fa806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1fa806adb0 0x7f1fa806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8007c40 tx=0x7f1fb8007b90 comp rx=0 tx=0).stop 2024-03-29T10:15:03.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 -- 172.21.15.12:0/1303386420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc815d9d0 msgr2=0x7f1fc81550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc815d9d0 0x7f1fc81550f0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000cc50 tx=0x7f1fc000cf60 comp rx=0 tx=0).stop 2024-03-29T10:15:03.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 -- 172.21.15.12:0/1303386420 shutdown_connections 2024-03-29T10:15:03.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1fa806adb0 0x7f1fa806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc815d9d0 0x7f1fc81550f0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1fc80d16a0 0x7f1fc80b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 --2- 172.21.15.12:0/1303386420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1fc80d2370 0x7f1fc815f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 -- 172.21.15.12:0/1303386420 >> 172.21.15.12:0/1303386420 conn(0x7f1fc808aff0 msgr2=0x7f1fc80b0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 -- 172.21.15.12:0/1303386420 shutdown_connections 2024-03-29T10:15:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.627+0000 7f1fce400700 1 -- 172.21.15.12:0/1303386420 wait complete. 2024-03-29T10:15:03.643 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T10:15:03.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.844+0000 7fed17600700 1 Processor -- start 2024-03-29T10:15:03.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.844+0000 7fed17600700 1 -- start start 2024-03-29T10:15:03.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.845+0000 7fed17600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fed100d2540 0x7fed101687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:03.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.845+0000 7fed17600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed10161420 con 0x7fed1015d9d0 2024-03-29T10:15:03.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.845+0000 7fed17600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed10161700 con 0x7fed100d16a0 2024-03-29T10:15:03.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.845+0000 7fed17600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed100d4340 con 0x7fed100d2540 2024-03-29T10:15:03.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.845+0000 7fed14c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fed1015d9d0 0x7fed100d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46654/0 (socket says 172.21.15.12:46654) 2024-03-29T10:15:03.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.845+0000 7fed14c00700 1 -- 172.21.15.12:0/2296904879 learned_addr learned my addr 172.21.15.12:0/2296904879 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:03.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.845+0000 7fed15600700 1 --2- 172.21.15.12:0/2296904879 >> v2:172.21.15.159:3300/0 conn(0x7fed100d2540 0x7fed101687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:03.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.846+0000 7fed0f400700 1 -- 172.21.15.12:0/2296904879 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637892018 0 0) 0x7fed10161420 con 0x7fed1015d9d0 2024-03-29T10:15:03.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.846+0000 7fed0f400700 1 -- 172.21.15.12:0/2296904879 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf80035b0 con 0x7fed1015d9d0 2024-03-29T10:15:03.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.846+0000 7fed0f400700 1 -- 172.21.15.12:0/2296904879 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3181924393 0 0) 0x7fecf80035b0 con 0x7fed1015d9d0 2024-03-29T10:15:03.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.846+0000 7fed15600700 1 -- 172.21.15.12:0/2296904879 >> v1:172.21.15.159:6789/0 conn(0x7fed100d16a0 legacy=0x7fed100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.846+0000 7fed15600700 1 -- 172.21.15.12:0/2296904879 >> v1:172.21.15.12:6789/0 conn(0x7fed1015d9d0 legacy=0x7fed100d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.846+0000 7fed15600700 1 -- 172.21.15.12:0/2296904879 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed101619e0 con 0x7fed100d2540 2024-03-29T10:15:03.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.846+0000 7fed15600700 1 --2- 172.21.15.12:0/2296904879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed100d2540 0x7fed101687b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fed08009d90 tx=0x7fed0800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ad68a0ceaaae8b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:03.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.847+0000 7fed0f400700 1 -- 172.21.15.12:0/2296904879 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0800e070 con 0x7fed100d2540 2024-03-29T10:15:03.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.847+0000 7fed0f400700 1 -- 172.21.15.12:0/2296904879 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed08003a20 con 0x7fed100d2540 2024-03-29T10:15:03.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.847+0000 7fed0f400700 1 -- 172.21.15.12:0/2296904879 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04001670 con 0x7fed100d2540 2024-03-29T10:15:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.847+0000 7fed17600700 1 -- 172.21.15.12:0/2296904879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed100d2540 msgr2=0x7fed101687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.847+0000 7fed17600700 1 --2- 172.21.15.12:0/2296904879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed100d2540 0x7fed101687b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fed08009d90 tx=0x7fed0800b330 comp rx=0 tx=0).stop 2024-03-29T10:15:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.848+0000 7fed17600700 1 -- 172.21.15.12:0/2296904879 shutdown_connections 2024-03-29T10:15:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.848+0000 7fed17600700 1 --2- 172.21.15.12:0/2296904879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed100d2540 0x7fed101687b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.848+0000 7fed17600700 1 -- 172.21.15.12:0/2296904879 >> 172.21.15.12:0/2296904879 conn(0x7fed1008aff0 msgr2=0x7fed100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.848+0000 7fed17600700 1 -- 172.21.15.12:0/2296904879 shutdown_connections 2024-03-29T10:15:03.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.848+0000 7fed17600700 1 -- 172.21.15.12:0/2296904879 wait complete. 2024-03-29T10:15:03.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.849+0000 7fed17600700 1 Processor -- start 2024-03-29T10:15:03.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.849+0000 7fed17600700 1 -- start start 2024-03-29T10:15:03.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.850+0000 7fed17600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed100d16a0 0x7fed101e8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:03.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.850+0000 7fed17600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed100d2540 0x7fed1015bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:03.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.850+0000 7fed17600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1015d9d0 0x7fed100ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.850+0000 7fed17600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed100b8780 con 0x7fed100d16a0 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.850+0000 7fed17600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed100cafb0 con 0x7fed1015d9d0 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.850+0000 7fed17600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed101e9220 con 0x7fed100d2540 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed14c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed100d16a0 0x7fed101e8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed14c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed100d16a0 0x7fed101e8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40666/0 (socket says 172.21.15.12:40666) 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed14c00700 1 -- 172.21.15.12:0/1796327692 learned_addr learned my addr 172.21.15.12:0/1796327692 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed15600700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1015d9d0 0x7fed100ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed0fe00700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed100d2540 0x7fed1015bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed14c00700 1 -- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed100d2540 msgr2=0x7fed1015bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed14c00700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed100d2540 0x7fed1015bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed14c00700 1 -- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1015d9d0 msgr2=0x7fed100ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed14c00700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1015d9d0 0x7fed100ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:03.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed14c00700 1 -- 172.21.15.12:0/1796327692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08009a70 con 0x7fed100d16a0 2024-03-29T10:15:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed0fe00700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed100d2540 0x7fed1015bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.851+0000 7fed14c00700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed100d16a0 0x7fed101e8ce0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c520 tx=0x7fed0400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.852+0000 7fed0cc00700 1 -- 172.21.15.12:0/1796327692 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04004c80 con 0x7fed100d16a0 2024-03-29T10:15:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.852+0000 7fed17600700 1 -- 172.21.15.12:0/1796327692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed100b7b70 con 0x7fed100d16a0 2024-03-29T10:15:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.852+0000 7fed0cc00700 1 -- 172.21.15.12:0/1796327692 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed04004e00 con 0x7fed100d16a0 2024-03-29T10:15:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.852+0000 7fed0cc00700 1 -- 172.21.15.12:0/1796327692 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04007310 con 0x7fed100d16a0 2024-03-29T10:15:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.852+0000 7fed17600700 1 -- 172.21.15.12:0/1796327692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed100b8130 con 0x7fed100d16a0 2024-03-29T10:15:03.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.853+0000 7fed17600700 1 -- 172.21.15.12:0/1796327692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed100d1ca0 con 0x7fed100d16a0 2024-03-29T10:15:03.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.854+0000 7fed0cc00700 1 -- 172.21.15.12:0/1796327692 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fed04007470 con 0x7fed100d16a0 2024-03-29T10:15:03.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.855+0000 7fed0cc00700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf006adb0 0x7fecf006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:03.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.855+0000 7fed0cc00700 1 -- 172.21.15.12:0/1796327692 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fed040892e0 con 0x7fed100d16a0 2024-03-29T10:15:03.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.855+0000 7fed0fe00700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf006adb0 0x7fecf006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:03.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.856+0000 7fed0fe00700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf006adb0 0x7fecf006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fed10160ae0 tx=0x7fecfc009040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:03.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:03.859+0000 7fed0cc00700 1 -- 172.21.15.12:0/1796327692 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fed04054300 con 0x7fed100d16a0 2024-03-29T10:15:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.077+0000 7fed17600700 1 -- 172.21.15.12:0/1796327692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fed100cb0f0 con 0x7fed100d16a0 2024-03-29T10:15:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.081+0000 7fed0cc00700 1 -- 172.21.15.12:0/1796327692 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7fed04057920 con 0x7fed100d16a0 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 -- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf006adb0 msgr2=0x7fecf006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf006adb0 0x7fecf006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fed10160ae0 tx=0x7fecfc009040 comp rx=0 tx=0).stop 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 -- 172.21.15.12:0/1796327692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed100d16a0 msgr2=0x7fed101e8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed100d16a0 0x7fed101e8ce0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c520 tx=0x7fed0400c830 comp rx=0 tx=0).stop 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 -- 172.21.15.12:0/1796327692 shutdown_connections 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf006adb0 0x7fecf006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1015d9d0 0x7fed100ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed100d16a0 0x7fed101e8ce0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 --2- 172.21.15.12:0/1796327692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed100d2540 0x7fed1015bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 -- 172.21.15.12:0/1796327692 >> 172.21.15.12:0/1796327692 conn(0x7fed1008aff0 msgr2=0x7fed100b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 -- 172.21.15.12:0/1796327692 shutdown_connections 2024-03-29T10:15:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.084+0000 7fed17600700 1 -- 172.21.15.12:0/1796327692 wait complete. 2024-03-29T10:15:04.102 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T10:15:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.309+0000 7fdfd0a00700 1 Processor -- start 2024-03-29T10:15:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.309+0000 7fdfd0a00700 1 -- start start 2024-03-29T10:15:04.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.310+0000 7fdfd0a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdfcc0d16a0 0x7fdfcc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.311+0000 7fdfd0a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc161420 con 0x7fdfcc0d2370 2024-03-29T10:15:04.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.311+0000 7fdfd0a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc161700 con 0x7fdfcc15d9d0 2024-03-29T10:15:04.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.311+0000 7fdfd0a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfcc0d4340 con 0x7fdfcc0d16a0 2024-03-29T10:15:04.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.311+0000 7fdfca800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdfcc0d2370 0x7fdfcc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46658/0 (socket says 172.21.15.12:46658) 2024-03-29T10:15:04.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.311+0000 7fdfca800700 1 -- 172.21.15.12:0/2588328275 learned_addr learned my addr 172.21.15.12:0/2588328275 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.311+0000 7fdfc9400700 1 --2- 172.21.15.12:0/2588328275 >> v2:172.21.15.159:3300/0 conn(0x7fdfcc0d16a0 0x7fdfcc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.311+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426063429 0 0) 0x7fdfcc161420 con 0x7fdfcc0d2370 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.311+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb00035b0 con 0x7fdfcc0d2370 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958956934 0 0) 0x7fdfcc161700 con 0x7fdfcc15d9d0 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfcc161420 con 0x7fdfcc15d9d0 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269711260 0 0) 0x7fdfb00035b0 con 0x7fdfcc0d2370 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 >> v2:172.21.15.159:3300/0 conn(0x7fdfcc0d16a0 msgr2=0x7fdfcc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 --2- 172.21.15.12:0/2588328275 >> v2:172.21.15.159:3300/0 conn(0x7fdfcc0d16a0 0x7fdfcc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 >> v1:172.21.15.159:6789/0 conn(0x7fdfcc15d9d0 legacy=0x7fdfcc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc1619e0 con 0x7fdfcc0d2370 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fdfc4001970 con 0x7fdfcc0d2370 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 >> v1:172.21.15.12:6789/0 conn(0x7fdfcc0d2370 legacy=0x7fdfcc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfb0003be0 0x7fdfb0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfb0004850 0x7fdfb0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfb0005210 0x7fdfb000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb000c0e0 con 0x7fdfb0003be0 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb000c220 con 0x7fdfb0004850 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb000c3e0 con 0x7fdfb0005210 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc9e00700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfb0003be0 0x7fdfb0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc9400700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfb0004850 0x7fdfb0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfca800700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfb0005210 0x7fdfb000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc9e00700 1 -- 172.21.15.12:0/2588328275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfb0005210 msgr2=0x7fdfb000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc9e00700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfb0005210 0x7fdfb000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc9e00700 1 -- 172.21.15.12:0/2588328275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfb0004850 msgr2=0x7fdfb0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc9e00700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfb0004850 0x7fdfb0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc9e00700 1 -- 172.21.15.12:0/2588328275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb00035b0 con 0x7fdfb0003be0 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfc9400700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfb0004850 0x7fdfb0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:04.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.312+0000 7fdfca800700 1 -- 172.21.15.12:0/2588328275 reap_dead start 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.313+0000 7fdfc9e00700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfb0003be0 0x7fdfb0003fe0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc001940 tx=0x7fdfb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a734fe0a73596d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.313+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc001430 con 0x7fdfb0003be0 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.313+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc001d70 con 0x7fdfb0003be0 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.313+0000 7fdfc8a00700 1 -- 172.21.15.12:0/2588328275 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc005530 con 0x7fdfb0003be0 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.313+0000 7fdfd0a00700 1 -- 172.21.15.12:0/2588328275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfb0003be0 msgr2=0x7fdfb0003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.313+0000 7fdfd0a00700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfb0003be0 0x7fdfb0003fe0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc001940 tx=0x7fdfb0005880 comp rx=0 tx=0).stop 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.313+0000 7fdfd0a00700 1 -- 172.21.15.12:0/2588328275 shutdown_connections 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.313+0000 7fdfd0a00700 1 --2- 172.21.15.12:0/2588328275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfb0003be0 0x7fdfb0003fe0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.313+0000 7fdfd0a00700 1 -- 172.21.15.12:0/2588328275 >> 172.21.15.12:0/2588328275 conn(0x7fdfcc08aff0 msgr2=0x7fdfcc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.314+0000 7fdfd0a00700 1 -- 172.21.15.12:0/2588328275 shutdown_connections 2024-03-29T10:15:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.314+0000 7fdfd0a00700 1 -- 172.21.15.12:0/2588328275 wait complete. 2024-03-29T10:15:04.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.315+0000 7fdfd0a00700 1 Processor -- start 2024-03-29T10:15:04.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.315+0000 7fdfd0a00700 1 -- start start 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.316+0000 7fdfd0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfb0003be0 0x7fdfcc0b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.316+0000 7fdfd0a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfcc0ca5f0 0x7fdfcc152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.316+0000 7fdfd0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfcc0cbdc0 0x7fdfcc0cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.316+0000 7fdfd0a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc0cb700 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.316+0000 7fdfd0a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc15cc30 con 0x7fdfcc0ca5f0 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.316+0000 7fdfd0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc0cc790 con 0x7fdfb0003be0 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfcc0cbdc0 0x7fdfcc0cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfcc0cbdc0 0x7fdfcc0cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40682/0 (socket says 172.21.15.12:40682) 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfca800700 1 -- 172.21.15.12:0/4103999312 learned_addr learned my addr 172.21.15.12:0/4103999312 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfc9400700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfcc0ca5f0 0x7fdfcc152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfc9e00700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfb0003be0 0x7fdfcc0b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfca800700 1 -- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfb0003be0 msgr2=0x7fdfcc0b8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfca800700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfb0003be0 0x7fdfcc0b8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfca800700 1 -- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfcc0ca5f0 msgr2=0x7fdfcc152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfca800700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfcc0ca5f0 0x7fdfcc152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfca800700 1 -- 172.21.15.12:0/4103999312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc0caa80 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.317+0000 7fdfca800700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfcc0cbdc0 0x7fdfcc0cc250 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc06b5b0 tx=0x7fdfc400e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.318+0000 7fdfc2000700 1 -- 172.21.15.12:0/4103999312 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4002070 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.318+0000 7fdfc2000700 1 -- 172.21.15.12:0/4103999312 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc400ec30 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.318+0000 7fdfc2000700 1 -- 172.21.15.12:0/4103999312 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4013420 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.318+0000 7fdfd0a00700 1 -- 172.21.15.12:0/4103999312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfcc0cad60 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.318+0000 7fdfd0a00700 1 -- 172.21.15.12:0/4103999312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfcc0cb320 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.319+0000 7fdfd0a00700 1 -- 172.21.15.12:0/4103999312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfcc0cb840 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.320+0000 7fdfc2000700 1 -- 172.21.15.12:0/4103999312 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdfc4013580 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.321+0000 7fdfc2000700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdfac06adb0 0x7fdfac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.321+0000 7fdfc2000700 1 -- 172.21.15.12:0/4103999312 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fdfc4089950 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.322+0000 7fdfc9e00700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdfac06adb0 0x7fdfac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.322+0000 7fdfc9e00700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdfac06adb0 0x7fdfac06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc005040 tx=0x7fdfbc00a000 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:04.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.326+0000 7fdfc2000700 1 -- 172.21.15.12:0/4103999312 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdfcc0cb840 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.540+0000 7fdfd0a00700 1 -- 172.21.15.12:0/4103999312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fdfcc0cb840 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.545+0000 7fdfc2000700 1 -- 172.21.15.12:0/4103999312 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v51) v1 ==== 120+0+0 (secure 0 0 0) 0x7fdfcc0cb840 con 0x7fdfcc0cbdc0 2024-03-29T10:15:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 -- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdfac06adb0 msgr2=0x7fdfac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdfac06adb0 0x7fdfac06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc005040 tx=0x7fdfbc00a000 comp rx=0 tx=0).stop 2024-03-29T10:15:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 -- 172.21.15.12:0/4103999312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfcc0cbdc0 msgr2=0x7fdfcc0cc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfcc0cbdc0 0x7fdfcc0cc250 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc06b5b0 tx=0x7fdfc400e440 comp rx=0 tx=0).stop 2024-03-29T10:15:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 -- 172.21.15.12:0/4103999312 shutdown_connections 2024-03-29T10:15:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fdfac06adb0 0x7fdfac06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfcc0cbdc0 0x7fdfcc0cc250 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfb0003be0 0x7fdfcc0b8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 --2- 172.21.15.12:0/4103999312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfcc0ca5f0 0x7fdfcc152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 -- 172.21.15.12:0/4103999312 >> 172.21.15.12:0/4103999312 conn(0x7fdfcc08aff0 msgr2=0x7fdfcc0b2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 -- 172.21.15.12:0/4103999312 shutdown_connections 2024-03-29T10:15:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.547+0000 7fdfd0a00700 1 -- 172.21.15.12:0/4103999312 wait complete. 2024-03-29T10:15:04.563 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-03-29T10:15:04.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.764+0000 7fce21200700 1 Processor -- start 2024-03-29T10:15:04.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.764+0000 7fce21200700 1 -- start start 2024-03-29T10:15:04.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce21200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c0d05e0 0x7fce1c0d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce21200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fce1c0b6990 0x7fce1c0b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce21200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce1c0d0f80 con 0x7fce1c15da50 2024-03-29T10:15:04.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce21200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1c0b95d0 con 0x7fce1c0b6990 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce21200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c15de30 con 0x7fce1c0d05e0 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce19e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fce1c15da50 0x7fce1c0ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46660/0 (socket says 172.21.15.12:46660) 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce19e00700 1 -- 172.21.15.12:0/1222019701 learned_addr learned my addr 172.21.15.12:0/1222019701 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce1a800700 1 --2- 172.21.15.12:0/1222019701 >> v2:172.21.15.159:3300/0 conn(0x7fce1c0b6990 0x7fce1c0b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce19400700 1 --2- 172.21.15.12:0/1222019701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c0d05e0 0x7fce1c0d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce18a00700 1 -- 172.21.15.12:0/1222019701 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851115893 0 0) 0x7fce1c0d0f80 con 0x7fce1c15da50 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.766+0000 7fce18a00700 1 -- 172.21.15.12:0/1222019701 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce000035b0 con 0x7fce1c15da50 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.767+0000 7fce19400700 1 -- 172.21.15.12:0/1222019701 >> v2:172.21.15.159:3300/0 conn(0x7fce1c0b6990 msgr2=0x7fce1c0b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.767+0000 7fce19400700 1 --2- 172.21.15.12:0/1222019701 >> v2:172.21.15.159:3300/0 conn(0x7fce1c0b6990 0x7fce1c0b6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.767+0000 7fce18a00700 1 -- 172.21.15.12:0/1222019701 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1568816943 0 0) 0x7fce000035b0 con 0x7fce1c15da50 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.767+0000 7fce19400700 1 -- 172.21.15.12:0/1222019701 >> v1:172.21.15.12:6789/0 conn(0x7fce1c15da50 legacy=0x7fce1c0ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.767+0000 7fce19400700 1 -- 172.21.15.12:0/1222019701 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c0d24e0 con 0x7fce1c0d05e0 2024-03-29T10:15:04.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.767+0000 7fce1a800700 1 --2- 172.21.15.12:0/1222019701 >> v2:172.21.15.159:3300/0 conn(0x7fce1c0b6990 0x7fce1c0b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.767+0000 7fce19400700 1 --2- 172.21.15.12:0/1222019701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c0d05e0 0x7fce1c0d0a40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fce08009840 tx=0x7fce08009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42102e5ee0690d2d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.767+0000 7fce18a00700 1 -- 172.21.15.12:0/1222019701 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c001670 con 0x7fce1c0d05e0 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.767+0000 7fce18a00700 1 -- 172.21.15.12:0/1222019701 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce08004030 con 0x7fce1c0d05e0 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.767+0000 7fce18a00700 1 -- 172.21.15.12:0/1222019701 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce080043c0 con 0x7fce1c0d05e0 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.768+0000 7fce21200700 1 -- 172.21.15.12:0/1222019701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c0d05e0 msgr2=0x7fce1c0d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.768+0000 7fce21200700 1 --2- 172.21.15.12:0/1222019701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c0d05e0 0x7fce1c0d0a40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fce08009840 tx=0x7fce08009b50 comp rx=0 tx=0).stop 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.768+0000 7fce21200700 1 -- 172.21.15.12:0/1222019701 shutdown_connections 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.768+0000 7fce21200700 1 --2- 172.21.15.12:0/1222019701 >> v2:172.21.15.159:3300/0 conn(0x7fce1c0b6990 0x7fce1c0b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.768+0000 7fce21200700 1 --2- 172.21.15.12:0/1222019701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c0d05e0 0x7fce1c0d0a40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.768+0000 7fce21200700 1 -- 172.21.15.12:0/1222019701 >> 172.21.15.12:0/1222019701 conn(0x7fce1c0666a0 msgr2=0x7fce1c066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.768+0000 7fce21200700 1 -- 172.21.15.12:0/1222019701 shutdown_connections 2024-03-29T10:15:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.768+0000 7fce21200700 1 -- 172.21.15.12:0/1222019701 wait complete. 2024-03-29T10:15:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.770+0000 7fce21200700 1 Processor -- start 2024-03-29T10:15:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.770+0000 7fce21200700 1 -- start start 2024-03-29T10:15:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce21200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c0b6990 0x7fce1c1f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce21200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce1c0d05e0 0x7fce1c0c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce21200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c15da50 0x7fce1c0c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce21200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c0c7890 con 0x7fce1c0b6990 2024-03-29T10:15:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce21200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c0c9f30 con 0x7fce1c0d05e0 2024-03-29T10:15:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce21200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c0ca210 con 0x7fce1c15da50 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce19e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c0b6990 0x7fce1c1f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce19e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c0b6990 0x7fce1c1f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40690/0 (socket says 172.21.15.12:40690) 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce19e00700 1 -- 172.21.15.12:0/744596891 learned_addr learned my addr 172.21.15.12:0/744596891 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce19400700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce1c0d05e0 0x7fce1c0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.771+0000 7fce1a800700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c15da50 0x7fce1c0c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce19e00700 1 -- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c15da50 msgr2=0x7fce1c0c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce19e00700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c15da50 0x7fce1c0c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce19e00700 1 -- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce1c0d05e0 msgr2=0x7fce1c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce19e00700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce1c0d05e0 0x7fce1c0c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce19e00700 1 -- 172.21.15.12:0/744596891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce14009a70 con 0x7fce1c0b6990 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce19e00700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c0b6990 0x7fce1c1f3250 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c750 tx=0x7fce0c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce12000700 1 -- 172.21.15.12:0/744596891 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c004c80 con 0x7fce1c0b6990 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce12000700 1 -- 172.21.15.12:0/744596891 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0c004de0 con 0x7fce1c0b6990 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce12000700 1 -- 172.21.15.12:0/744596891 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c0072d0 con 0x7fce1c0b6990 2024-03-29T10:15:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce21200700 1 -- 172.21.15.12:0/744596891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce080094a0 con 0x7fce1c0b6990 2024-03-29T10:15:04.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.772+0000 7fce21200700 1 -- 172.21.15.12:0/744596891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1c157170 con 0x7fce1c0b6990 2024-03-29T10:15:04.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.773+0000 7fce21200700 1 -- 172.21.15.12:0/744596891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1c156cf0 con 0x7fce1c0b6990 2024-03-29T10:15:04.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.774+0000 7fce12000700 1 -- 172.21.15.12:0/744596891 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fce0c007430 con 0x7fce1c0b6990 2024-03-29T10:15:04.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.775+0000 7fce12000700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcdfc06adb0 0x7fcdfc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:04.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.775+0000 7fce12000700 1 -- 172.21.15.12:0/744596891 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fce0c089160 con 0x7fce1c0b6990 2024-03-29T10:15:04.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.776+0000 7fce19400700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcdfc06adb0 0x7fcdfc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:04.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.776+0000 7fce19400700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcdfc06adb0 0x7fcdfc06d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fce08009840 tx=0x7fce08008030 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:04.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:04.779+0000 7fce12000700 1 -- 172.21.15.12:0/744596891 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce1c156cf0 con 0x7fce1c0b6990 2024-03-29T10:15:05.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.076+0000 7fce21200700 1 -- 172.21.15.12:0/744596891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fce1c156cf0 con 0x7fce1c0b6990 2024-03-29T10:15:05.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.162+0000 7fce12000700 1 -- 172.21.15.12:0/744596891 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v784) v1 ==== 160+0+0 (secure 0 0 0) 0x7fce1c156cf0 con 0x7fce1c0b6990 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.164+0000 7fce21200700 1 -- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcdfc06adb0 msgr2=0x7fcdfc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.164+0000 7fce21200700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcdfc06adb0 0x7fcdfc06d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fce08009840 tx=0x7fce08008030 comp rx=0 tx=0).stop 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.164+0000 7fce21200700 1 -- 172.21.15.12:0/744596891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c0b6990 msgr2=0x7fce1c1f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.164+0000 7fce21200700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c0b6990 0x7fce1c1f3250 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c750 tx=0x7fce0c00ca60 comp rx=0 tx=0).stop 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.165+0000 7fce21200700 1 -- 172.21.15.12:0/744596891 shutdown_connections 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.165+0000 7fce21200700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fcdfc06adb0 0x7fcdfc06d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.165+0000 7fce21200700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce1c15da50 0x7fce1c0c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.165+0000 7fce21200700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c0b6990 0x7fce1c1f3250 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.165+0000 7fce21200700 1 --2- 172.21.15.12:0/744596891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce1c0d05e0 0x7fce1c0c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.165+0000 7fce21200700 1 -- 172.21.15.12:0/744596891 >> 172.21.15.12:0/744596891 conn(0x7fce1c0666a0 msgr2=0x7fce1c15a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.165+0000 7fce21200700 1 -- 172.21.15.12:0/744596891 shutdown_connections 2024-03-29T10:15:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.165+0000 7fce21200700 1 -- 172.21.15.12:0/744596891 wait complete. 2024-03-29T10:15:05.180 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2024-03-29T10:15:05.180 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2024-03-29T10:15:05.181 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T10:15:05.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-03-29T10:15:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.381+0000 7f7606a00700 1 Processor -- start 2024-03-29T10:15:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.382+0000 7f7606a00700 1 -- start start 2024-03-29T10:15:05.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.383+0000 7f7606a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7600160f10 0x7f76000d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:05.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.383+0000 7f7606a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600161590 0x7f7600161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:05.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.383+0000 7f7606a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76000ceda0 con 0x7f76000cf230 2024-03-29T10:15:05.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.383+0000 7f7606a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7600161f80 con 0x7f7600160f10 2024-03-29T10:15:05.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.383+0000 7f7606a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76001612f0 con 0x7f7600161590 2024-03-29T10:15:05.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.383+0000 7f75ff400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f76000cf230 0x7f76000d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46668/0 (socket says 172.21.15.12:46668) 2024-03-29T10:15:05.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.383+0000 7f75ff400700 1 -- 172.21.15.12:0/2523680566 learned_addr learned my addr 172.21.15.12:0/2523680566 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:05.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.383+0000 7f75ffe00700 1 --2- 172.21.15.12:0/2523680566 >> v2:172.21.15.159:3300/0 conn(0x7f7600160f10 0x7f76000d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:05.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.383+0000 7f7604a00700 1 --2- 172.21.15.12:0/2523680566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600161590 0x7f7600161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:05.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.383+0000 7f75fea00700 1 -- 172.21.15.12:0/2523680566 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92960854 0 0) 0x7f76000ceda0 con 0x7f76000cf230 2024-03-29T10:15:05.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.384+0000 7f75fea00700 1 -- 172.21.15.12:0/2523680566 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e40035b0 con 0x7f76000cf230 2024-03-29T10:15:05.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.384+0000 7f75ffe00700 1 -- 172.21.15.12:0/2523680566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600161590 msgr2=0x7f7600161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:05.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.384+0000 7f75ffe00700 1 --2- 172.21.15.12:0/2523680566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600161590 0x7f7600161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:05.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.384+0000 7f75ffe00700 1 -- 172.21.15.12:0/2523680566 >> v1:172.21.15.12:6789/0 conn(0x7f76000cf230 legacy=0x7f76000d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:05.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.384+0000 7f75ffe00700 1 -- 172.21.15.12:0/2523680566 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76000ce470 con 0x7f7600160f10 2024-03-29T10:15:05.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.384+0000 7f75ffe00700 1 --2- 172.21.15.12:0/2523680566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7600160f10 0x7f76000d18b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009840 tx=0x7f75f0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5897c2004425b71e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.384+0000 7f75fea00700 1 -- 172.21.15.12:0/2523680566 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0011830 con 0x7f7600160f10 2024-03-29T10:15:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.385+0000 7f75fea00700 1 -- 172.21.15.12:0/2523680566 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f0011990 con 0x7f7600160f10 2024-03-29T10:15:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.385+0000 7f75fea00700 1 -- 172.21.15.12:0/2523680566 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0011d20 con 0x7f7600160f10 2024-03-29T10:15:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.385+0000 7f7606a00700 1 -- 172.21.15.12:0/2523680566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7600160f10 msgr2=0x7f76000d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.385+0000 7f7606a00700 1 --2- 172.21.15.12:0/2523680566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7600160f10 0x7f76000d18b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009840 tx=0x7f75f0009b50 comp rx=0 tx=0).stop 2024-03-29T10:15:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.385+0000 7f7606a00700 1 -- 172.21.15.12:0/2523680566 shutdown_connections 2024-03-29T10:15:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.385+0000 7f7606a00700 1 --2- 172.21.15.12:0/2523680566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600161590 0x7f7600161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.385+0000 7f7606a00700 1 --2- 172.21.15.12:0/2523680566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7600160f10 0x7f76000d18b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.385+0000 7f7606a00700 1 -- 172.21.15.12:0/2523680566 >> 172.21.15.12:0/2523680566 conn(0x7f760008aff0 msgr2=0x7f76000d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.385+0000 7f7606a00700 1 -- 172.21.15.12:0/2523680566 shutdown_connections 2024-03-29T10:15:05.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.385+0000 7f7606a00700 1 -- 172.21.15.12:0/2523680566 wait complete. 2024-03-29T10:15:05.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.387+0000 7f7606a00700 1 Processor -- start 2024-03-29T10:15:05.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.387+0000 7f7606a00700 1 -- start start 2024-03-29T10:15:05.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.388+0000 7f7606a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76000cf230 0x7f76000b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:05.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.388+0000 7f7606a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600160f10 0x7f76001e77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:05.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.388+0000 7f7606a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7600161590 0x7f7600158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:05.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.388+0000 7f7606a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76001e7cf0 con 0x7f76000cf230 2024-03-29T10:15:05.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.388+0000 7f7606a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76001e7f50 con 0x7f7600161590 2024-03-29T10:15:05.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.388+0000 7f7606a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76001e8230 con 0x7f7600160f10 2024-03-29T10:15:05.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.388+0000 7f75ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76000cf230 0x7f76000b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:05.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76000cf230 0x7f76000b72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40696/0 (socket says 172.21.15.12:40696) 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75ffe00700 1 -- 172.21.15.12:0/1312755270 learned_addr learned my addr 172.21.15.12:0/1312755270 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f7604a00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7600161590 0x7f7600158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75ff400700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600160f10 0x7f76001e77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75ffe00700 1 -- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600160f10 msgr2=0x7f76001e77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75ffe00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600160f10 0x7f76001e77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75ffe00700 1 -- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7600161590 msgr2=0x7f7600158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75ffe00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7600161590 0x7f7600158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75ffe00700 1 -- 172.21.15.12:0/1312755270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f00094a0 con 0x7f76000cf230 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75ff400700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600160f10 0x7f76001e77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75ffe00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76000cf230 0x7f76000b72b0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009fd0 tx=0x7f75f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75f7e00700 1 -- 172.21.15.12:0/1312755270 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f00036a0 con 0x7f76000cf230 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75f7e00700 1 -- 172.21.15.12:0/1312755270 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f001e070 con 0x7f76000cf230 2024-03-29T10:15:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.389+0000 7f75f7e00700 1 -- 172.21.15.12:0/1312755270 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0003a60 con 0x7f76000cf230 2024-03-29T10:15:05.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.390+0000 7f7606a00700 1 -- 172.21.15.12:0/1312755270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76001e8510 con 0x7f76000cf230 2024-03-29T10:15:05.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.390+0000 7f7606a00700 1 -- 172.21.15.12:0/1312755270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7600158f40 con 0x7f76000cf230 2024-03-29T10:15:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.391+0000 7f7606a00700 1 -- 172.21.15.12:0/1312755270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760015e5b0 con 0x7f76000cf230 2024-03-29T10:15:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.392+0000 7f75f7e00700 1 -- 172.21.15.12:0/1312755270 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f75f0003bc0 con 0x7f76000cf230 2024-03-29T10:15:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.393+0000 7f75f7e00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75e006adb0 0x7f75e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.393+0000 7f75f7e00700 1 -- 172.21.15.12:0/1312755270 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f75f008a2b0 con 0x7f76000cf230 2024-03-29T10:15:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.393+0000 7f75ff400700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75e006adb0 0x7f75e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.394+0000 7f75ff400700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75e006adb0 0x7f75e006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f76000b7fa0 tx=0x7f75ec009040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:05.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.397+0000 7f75f7e00700 1 -- 172.21.15.12:0/1312755270 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f760015e5b0 con 0x7f76000cf230 2024-03-29T10:15:05.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:05.698+0000 7f7606a00700 1 -- 172.21.15.12:0/1312755270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f760015e5b0 con 0x7f76000cf230 2024-03-29T10:15:06.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.167+0000 7f75f7e00700 1 -- 172.21.15.12:0/1312755270 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v785) v1 ==== 181+0+0 (secure 0 0 0) 0x7f760015e5b0 con 0x7f76000cf230 2024-03-29T10:15:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.169+0000 7f7606a00700 1 -- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75e006adb0 msgr2=0x7f75e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.169+0000 7f7606a00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75e006adb0 0x7f75e006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f76000b7fa0 tx=0x7f75ec009040 comp rx=0 tx=0).stop 2024-03-29T10:15:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.169+0000 7f7606a00700 1 -- 172.21.15.12:0/1312755270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76000cf230 msgr2=0x7f76000b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:06.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.169+0000 7f7606a00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76000cf230 0x7f76000b72b0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009fd0 tx=0x7f75f0004750 comp rx=0 tx=0).stop 2024-03-29T10:15:06.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.169+0000 7f7606a00700 1 -- 172.21.15.12:0/1312755270 shutdown_connections 2024-03-29T10:15:06.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.169+0000 7f7606a00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f75e006adb0 0x7f75e006d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.169+0000 7f7606a00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7600161590 0x7f7600158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.169+0000 7f7606a00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76000cf230 0x7f76000b72b0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.169+0000 7f7606a00700 1 --2- 172.21.15.12:0/1312755270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7600160f10 0x7f76001e77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.169+0000 7f7606a00700 1 -- 172.21.15.12:0/1312755270 >> 172.21.15.12:0/1312755270 conn(0x7f760008aff0 msgr2=0x7f76000b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:06.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.170+0000 7f7606a00700 1 -- 172.21.15.12:0/1312755270 shutdown_connections 2024-03-29T10:15:06.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.170+0000 7f7606a00700 1 -- 172.21.15.12:0/1312755270 wait complete. 2024-03-29T10:15:06.187 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T10:15:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.392+0000 7f97d1e00700 1 Processor -- start 2024-03-29T10:15:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.393+0000 7f97d1e00700 1 -- start start 2024-03-29T10:15:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.394+0000 7f97d1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97cc1611b0 0x7f97cc15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.394+0000 7f97d1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97cc15f4d0 0x7f97cc0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.394+0000 7f97d1e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc0ce270 con 0x7f97cc0d21f0 2024-03-29T10:15:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.394+0000 7f97d1e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97cc15b600 con 0x7f97cc1611b0 2024-03-29T10:15:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.394+0000 7f97d1e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0d2760 con 0x7f97cc15f4d0 2024-03-29T10:15:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.394+0000 7f97caa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97cc1611b0 0x7f97cc15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.394+0000 7f97cb400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f97cc0d21f0 0x7f97cc161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46676/0 (socket says 172.21.15.12:46676) 2024-03-29T10:15:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.394+0000 7f97cb400700 1 -- 172.21.15.12:0/634938868 learned_addr learned my addr 172.21.15.12:0/634938868 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.394+0000 7f97ca000700 1 --2- 172.21.15.12:0/634938868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97cc15f4d0 0x7f97cc0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.394+0000 7f97c9600700 1 -- 172.21.15.12:0/634938868 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453646352 0 0) 0x7f97cc0ce270 con 0x7f97cc0d21f0 2024-03-29T10:15:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.395+0000 7f97c9600700 1 -- 172.21.15.12:0/634938868 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b00035b0 con 0x7f97cc0d21f0 2024-03-29T10:15:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.395+0000 7f97caa00700 1 -- 172.21.15.12:0/634938868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97cc15f4d0 msgr2=0x7f97cc0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:06.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.395+0000 7f97caa00700 1 --2- 172.21.15.12:0/634938868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97cc15f4d0 0x7f97cc0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.395+0000 7f97caa00700 1 -- 172.21.15.12:0/634938868 >> v1:172.21.15.12:6789/0 conn(0x7f97cc0d21f0 legacy=0x7f97cc161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.395+0000 7f97caa00700 1 -- 172.21.15.12:0/634938868 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc0d2a40 con 0x7f97cc1611b0 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.395+0000 7f97caa00700 1 --2- 172.21.15.12:0/634938868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 0x7f97cc15ef90 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009840 tx=0x7f97bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a345962fd0a3c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.395+0000 7f97c9600700 1 -- 172.21.15.12:0/634938868 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc011830 con 0x7f97cc1611b0 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.395+0000 7f97c9600700 1 -- 172.21.15.12:0/634938868 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc011990 con 0x7f97cc1611b0 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.395+0000 7f97c9600700 1 -- 172.21.15.12:0/634938868 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc011d00 con 0x7f97cc1611b0 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.396+0000 7f97d1e00700 1 -- 172.21.15.12:0/634938868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 msgr2=0x7f97cc15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.396+0000 7f97d1e00700 1 --2- 172.21.15.12:0/634938868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 0x7f97cc15ef90 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009840 tx=0x7f97bc009b50 comp rx=0 tx=0).stop 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.396+0000 7f97d1e00700 1 -- 172.21.15.12:0/634938868 shutdown_connections 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.396+0000 7f97d1e00700 1 --2- 172.21.15.12:0/634938868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 0x7f97cc15ef90 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.396+0000 7f97d1e00700 1 --2- 172.21.15.12:0/634938868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97cc15f4d0 0x7f97cc0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.396+0000 7f97d1e00700 1 -- 172.21.15.12:0/634938868 >> 172.21.15.12:0/634938868 conn(0x7f97cc08aff0 msgr2=0x7f97cc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:06.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.396+0000 7f97d1e00700 1 -- 172.21.15.12:0/634938868 shutdown_connections 2024-03-29T10:15:06.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.396+0000 7f97d1e00700 1 -- 172.21.15.12:0/634938868 wait complete. 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.398+0000 7f97d1e00700 1 Processor -- start 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.398+0000 7f97d1e00700 1 -- start start 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97d1e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97cc0d21f0 0x7f97cc1f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97d1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97cc15f4d0 0x7f97cc0b6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97d1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 0x7f97cc156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97d1e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0cbd30 con 0x7f97cc1611b0 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97d1e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0b77b0 con 0x7f97cc0d21f0 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97d1e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0b7a90 con 0x7f97cc15f4d0 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97cb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 0x7f97cc156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97cb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 0x7f97cc156c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40720/0 (socket says 172.21.15.12:40720) 2024-03-29T10:15:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97cb400700 1 -- 172.21.15.12:0/2796662807 learned_addr learned my addr 172.21.15.12:0/2796662807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97ca000700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97cc15f4d0 0x7f97cc0b6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97caa00700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97cc0d21f0 0x7f97cc1f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97cb400700 1 -- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97cc15f4d0 msgr2=0x7f97cc0b6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97cb400700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97cc15f4d0 0x7f97cc0b6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97cb400700 1 -- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97cc0d21f0 msgr2=0x7f97cc1f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97cb400700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97cc0d21f0 0x7f97cc1f3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.399+0000 7f97cb400700 1 -- 172.21.15.12:0/2796662807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc0094a0 con 0x7f97cc1611b0 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.400+0000 7f97cb400700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 0x7f97cc156c20 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f97c400cb40 tx=0x7f97c400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.400+0000 7f97c3400700 1 -- 172.21.15.12:0/2796662807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c400a9e0 con 0x7f97cc1611b0 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.400+0000 7f97c3400700 1 -- 172.21.15.12:0/2796662807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c400ab40 con 0x7f97cc1611b0 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.400+0000 7f97d1e00700 1 -- 172.21.15.12:0/2796662807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97cc0b7d70 con 0x7f97cc1611b0 2024-03-29T10:15:06.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.400+0000 7f97c3400700 1 -- 172.21.15.12:0/2796662807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4012400 con 0x7f97cc1611b0 2024-03-29T10:15:06.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.400+0000 7f97d1e00700 1 -- 172.21.15.12:0/2796662807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97cc0cb600 con 0x7f97cc1611b0 2024-03-29T10:15:06.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.401+0000 7f97d1e00700 1 -- 172.21.15.12:0/2796662807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97cc0b7fb0 con 0x7f97cc1611b0 2024-03-29T10:15:06.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.402+0000 7f97c3400700 1 -- 172.21.15.12:0/2796662807 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f97c401b3b0 con 0x7f97cc1611b0 2024-03-29T10:15:06.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.403+0000 7f97c3400700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97ac06adb0 0x7f97ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:06.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.403+0000 7f97c3400700 1 -- 172.21.15.12:0/2796662807 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f97c4088fb0 con 0x7f97cc1611b0 2024-03-29T10:15:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.403+0000 7f97caa00700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97ac06adb0 0x7f97ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.404+0000 7f97caa00700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97ac06adb0 0x7f97ac06d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f97bc000c00 tx=0x7f97bc01c040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:06.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.407+0000 7f97c3400700 1 -- 172.21.15.12:0/2796662807 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f97cc0b7fb0 con 0x7f97cc1611b0 2024-03-29T10:15:06.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.621+0000 7f97d1e00700 1 -- 172.21.15.12:0/2796662807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f97cc0b7fb0 con 0x7f97cc1611b0 2024-03-29T10:15:06.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.626+0000 7f97c3400700 1 -- 172.21.15.12:0/2796662807 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f97cc0b7fb0 con 0x7f97cc1611b0 2024-03-29T10:15:06.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 -- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97ac06adb0 msgr2=0x7f97ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:06.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97ac06adb0 0x7f97ac06d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f97bc000c00 tx=0x7f97bc01c040 comp rx=0 tx=0).stop 2024-03-29T10:15:06.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 -- 172.21.15.12:0/2796662807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 msgr2=0x7f97cc156c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:06.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 0x7f97cc156c20 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f97c400cb40 tx=0x7f97c400ce50 comp rx=0 tx=0).stop 2024-03-29T10:15:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 -- 172.21.15.12:0/2796662807 shutdown_connections 2024-03-29T10:15:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f97ac06adb0 0x7f97ac06d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97cc1611b0 0x7f97cc156c20 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97cc0d21f0 0x7f97cc1f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 --2- 172.21.15.12:0/2796662807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97cc15f4d0 0x7f97cc0b6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 -- 172.21.15.12:0/2796662807 >> 172.21.15.12:0/2796662807 conn(0x7f97cc08aff0 msgr2=0x7f97cc06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 -- 172.21.15.12:0/2796662807 shutdown_connections 2024-03-29T10:15:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:06.628+0000 7f97d1e00700 1 -- 172.21.15.12:0/2796662807 wait complete. 2024-03-29T10:15:06.647 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T10:15:06.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:15:06.737 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T10:15:06.839 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T10:15:06.839 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:15:06.839 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:15:06.884 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:15:06.884 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:15:06.884 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:15:06.931 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:15:06.931 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:15:06.931 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:15:06.976 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T10:15:06.976 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T10:15:06.976 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:15:07.021 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T10:15:07.021 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T10:15:07.021 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:15:07.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.222+0000 7f5dd0a00700 1 Processor -- start 2024-03-29T10:15:07.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.223+0000 7f5dd0a00700 1 -- start start 2024-03-29T10:15:07.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dd0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dcc0ce590 0x7f5dcc0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:07.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dd0a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5dcc0b6a20 0x7f5dcc0b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:07.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dd0a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc0ceef0 con 0x7f5dcc160be0 2024-03-29T10:15:07.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dd0a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dcc0b65d0 con 0x7f5dcc0b6a20 2024-03-29T10:15:07.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dd0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc0d2a80 con 0x7f5dcc0ce590 2024-03-29T10:15:07.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dca800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5dcc0b6a20 0x7f5dcc0b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:07.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dca800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5dcc0b6a20 0x7f5dcc0b6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40732/0 (socket says 172.21.15.12:40732) 2024-03-29T10:15:07.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dca800700 1 -- 172.21.15.12:0/725294520 learned_addr learned my addr 172.21.15.12:0/725294520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:07.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dc9400700 1 --2- 172.21.15.12:0/725294520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dcc0ce590 0x7f5dcc0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dca800700 1 -- 172.21.15.12:0/725294520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dcc0ce590 msgr2=0x7f5dcc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.224+0000 7f5dc8a00700 1 -- 172.21.15.12:0/725294520 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620149591 0 0) 0x7f5dcc0ceef0 con 0x7f5dcc160be0 2024-03-29T10:15:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.225+0000 7f5dca800700 1 --2- 172.21.15.12:0/725294520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dcc0ce590 0x7f5dcc0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.225+0000 7f5dca800700 1 -- 172.21.15.12:0/725294520 >> v1:172.21.15.159:6789/0 conn(0x7f5dcc160be0 legacy=0x7f5dcc0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.225+0000 7f5dca800700 1 -- 172.21.15.12:0/725294520 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc160fc0 con 0x7f5dcc0b6a20 2024-03-29T10:15:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.225+0000 7f5dca800700 1 --2- 172.21.15.12:0/725294520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc0b6a20 0x7f5dcc0b6eb0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009e50 tx=0x7f5dc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfff5749297ca9e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.225+0000 7f5dc8a00700 1 -- 172.21.15.12:0/725294520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc400e070 con 0x7f5dcc0b6a20 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.225+0000 7f5dc8a00700 1 -- 172.21.15.12:0/725294520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc400b910 con 0x7f5dcc0b6a20 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.225+0000 7f5dc8a00700 1 -- 172.21.15.12:0/725294520 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc400bcc0 con 0x7f5dcc0b6a20 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.226+0000 7f5dd0a00700 1 -- 172.21.15.12:0/725294520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc0b6a20 msgr2=0x7f5dcc0b6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.226+0000 7f5dd0a00700 1 --2- 172.21.15.12:0/725294520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc0b6a20 0x7f5dcc0b6eb0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009e50 tx=0x7f5dc400b330 comp rx=0 tx=0).stop 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.226+0000 7f5dd0a00700 1 -- 172.21.15.12:0/725294520 shutdown_connections 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.226+0000 7f5dd0a00700 1 --2- 172.21.15.12:0/725294520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc0b6a20 0x7f5dcc0b6eb0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.226+0000 7f5dd0a00700 1 --2- 172.21.15.12:0/725294520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dcc0ce590 0x7f5dcc0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.226+0000 7f5dd0a00700 1 -- 172.21.15.12:0/725294520 >> 172.21.15.12:0/725294520 conn(0x7f5dcc08aff0 msgr2=0x7f5dcc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.226+0000 7f5dd0a00700 1 -- 172.21.15.12:0/725294520 shutdown_connections 2024-03-29T10:15:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.226+0000 7f5dd0a00700 1 -- 172.21.15.12:0/725294520 wait complete. 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.228+0000 7f5dd0a00700 1 Processor -- start 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.228+0000 7f5dd0a00700 1 -- start start 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dd0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dcc0b6a20 0x7f5dcc15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dd0a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dcc0ce590 0x7f5dcc0c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dd0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc160be0 0x7f5dcc0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dd0a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc0c7a90 con 0x7f5dcc160be0 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dd0a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc0c85d0 con 0x7f5dcc0ce590 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dd0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc15b930 con 0x7f5dcc0b6a20 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc160be0 0x7f5dcc0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc160be0 0x7f5dcc0cc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40744/0 (socket says 172.21.15.12:40744) 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dca800700 1 -- 172.21.15.12:0/293801839 learned_addr learned my addr 172.21.15.12:0/293801839 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dc9e00700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dcc0b6a20 0x7f5dcc15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dc9400700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dcc0ce590 0x7f5dcc0c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dca800700 1 -- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dcc0b6a20 msgr2=0x7f5dcc15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dca800700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dcc0b6a20 0x7f5dcc15b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dca800700 1 -- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dcc0ce590 msgr2=0x7f5dcc0c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dca800700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dcc0ce590 0x7f5dcc0c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.229+0000 7f5dca800700 1 -- 172.21.15.12:0/293801839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc4009b30 con 0x7f5dcc160be0 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.230+0000 7f5dca800700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc160be0 0x7f5dcc0cc690 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400b680 tx=0x7f5dc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.230+0000 7f5dc2000700 1 -- 172.21.15.12:0/293801839 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4003ea0 con 0x7f5dcc160be0 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.230+0000 7f5dc2000700 1 -- 172.21.15.12:0/293801839 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc40048d0 con 0x7f5dcc160be0 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.230+0000 7f5dd0a00700 1 -- 172.21.15.12:0/293801839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dcc0c7ca0 con 0x7f5dcc160be0 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.230+0000 7f5dc2000700 1 -- 172.21.15.12:0/293801839 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc401b670 con 0x7f5dcc160be0 2024-03-29T10:15:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.230+0000 7f5dd0a00700 1 -- 172.21.15.12:0/293801839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dcc0c8260 con 0x7f5dcc160be0 2024-03-29T10:15:07.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.231+0000 7f5dd0a00700 1 -- 172.21.15.12:0/293801839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dcc0c8760 con 0x7f5dcc160be0 2024-03-29T10:15:07.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.232+0000 7f5dc2000700 1 -- 172.21.15.12:0/293801839 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5dc401b7d0 con 0x7f5dcc160be0 2024-03-29T10:15:07.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.233+0000 7f5dc2000700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5dac06adb0 0x7f5dac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:07.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.233+0000 7f5dc2000700 1 -- 172.21.15.12:0/293801839 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f5dc408ad10 con 0x7f5dcc160be0 2024-03-29T10:15:07.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.234+0000 7f5dc9e00700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5dac06adb0 0x7f5dac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:07.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.234+0000 7f5dc9e00700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5dac06adb0 0x7f5dac06d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc007fd0 tx=0x7f5dbc009040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:07.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.237+0000 7f5dc2000700 1 -- 172.21.15.12:0/293801839 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5dc4055d30 con 0x7f5dcc160be0 2024-03-29T10:15:07.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.493+0000 7f5dd0a00700 1 -- 172.21.15.12:0/293801839 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dcc067720 con 0x7f5dcc160be0 2024-03-29T10:15:07.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.493+0000 7f5dc2000700 1 -- 172.21.15.12:0/293801839 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f5dc408ede0 con 0x7f5dcc160be0 2024-03-29T10:15:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:07.491 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":66,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7199,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1317631939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1317631939},{"type":"v1","addr":"172.21.15.159:6836","nonce":1317631939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1057731647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1057731647},{"type":"v1","addr":"172.21.15.159:6837","nonce":1057731647}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1567088347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1567088347},{"type":"v1","addr":"172.21.15.12:6834","nonce":1567088347}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":8880,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1035700475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":1035700475},{"type":"v1","addr":"172.21.15.12:6835","nonce":1035700475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[]} 2024-03-29T10:15:07.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 -- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5dac06adb0 msgr2=0x7f5dac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:07.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5dac06adb0 0x7f5dac06d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc007fd0 tx=0x7f5dbc009040 comp rx=0 tx=0).stop 2024-03-29T10:15:07.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 -- 172.21.15.12:0/293801839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc160be0 msgr2=0x7f5dcc0cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:07.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc160be0 0x7f5dcc0cc690 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400b680 tx=0x7f5dc4004ad0 comp rx=0 tx=0).stop 2024-03-29T10:15:07.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 -- 172.21.15.12:0/293801839 shutdown_connections 2024-03-29T10:15:07.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5dac06adb0 0x7f5dac06d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:07.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dcc160be0 0x7f5dcc0cc690 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:07.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dcc0b6a20 0x7f5dcc15b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:07.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 --2- 172.21.15.12:0/293801839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dcc0ce590 0x7f5dcc0c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:07.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 -- 172.21.15.12:0/293801839 >> 172.21.15.12:0/293801839 conn(0x7f5dcc08aff0 msgr2=0x7f5dcc15fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:07.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 -- 172.21.15.12:0/293801839 shutdown_connections 2024-03-29T10:15:07.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:07.496+0000 7f5dd0a00700 1 -- 172.21.15.12:0/293801839 wait complete. 2024-03-29T10:15:07.495 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 66 2024-03-29T10:15:07.512 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T10:15:07.512 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T10:15:07.512 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T10:15:07.513 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:15:07.513 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T10:15:07.513 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T10:15:07.513 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T10:15:07.513 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:15:07.513 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T10:15:07.513 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T10:15:07.514 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T10:15:07.514 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:15:07.514 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T10:15:07.514 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T10:15:07.514 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T10:15:07.514 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:15:07.526 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T10:15:07.526 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T10:15:07.527 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T10:15:07.527 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T10:15:07.529 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T10:15:07.529 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T10:15:07.571 INFO:tasks.ceph.mds.a:Started 2024-03-29T10:15:07.571 DEBUG:teuthology.parallel:result is None 2024-03-29T10:15:07.573 INFO:tasks.ceph.mds.c:Started 2024-03-29T10:15:07.573 DEBUG:teuthology.parallel:result is None 2024-03-29T10:15:07.574 INFO:tasks.ceph.mds.d:Started 2024-03-29T10:15:07.574 DEBUG:teuthology.parallel:result is None 2024-03-29T10:15:07.715 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T10:15:07.715 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T10:15:07.717 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T10:15:09.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-29T10:15:13.529 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T10:15:13.530 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T10:15:13.534 INFO:tasks.ceph.mds.b:Started 2024-03-29T10:15:13.535 DEBUG:teuthology.parallel:result is None 2024-03-29T10:15:13.535 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T10:15:13.681 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T10:15:13.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.750+0000 7f9ed7a00700 1 Processor -- start 2024-03-29T10:15:13.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.750+0000 7f9ed7a00700 1 -- start start 2024-03-29T10:15:13.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.751+0000 7f9ed7a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed015dad0 0x7f9ed00ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:13.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.751+0000 7f9ed7a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ed00d1630 0x7f9ed00d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:13.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.751+0000 7f9ed7a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed015f820 con 0x7f9ed00d1fd0 2024-03-29T10:15:13.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.751+0000 7f9ed7a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed00cfcc0 con 0x7f9ed00d1630 2024-03-29T10:15:13.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.751+0000 7f9ed7a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00cd9f0 con 0x7f9ed015dad0 2024-03-29T10:15:13.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.751+0000 7f9ecfe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ed00d1630 0x7f9ed00d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:13.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.751+0000 7f9ecfe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ed00d1630 0x7f9ed00d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39278/0 (socket says 172.21.15.12:39278) 2024-03-29T10:15:13.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.751+0000 7f9ecfe00700 1 -- 172.21.15.12:0/560642036 learned_addr learned my addr 172.21.15.12:0/560642036 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:13.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.751+0000 7f9ed5000700 1 --2- 172.21.15.12:0/560642036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed015dad0 0x7f9ed00ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.752+0000 7f9ecfe00700 1 -- 172.21.15.12:0/560642036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed015dad0 msgr2=0x7f9ed00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.752+0000 7f9ecfe00700 1 --2- 172.21.15.12:0/560642036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed015dad0 0x7f9ed00ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.752+0000 7f9ecfe00700 1 -- 172.21.15.12:0/560642036 >> v1:172.21.15.159:6789/0 conn(0x7f9ed00d1fd0 legacy=0x7f9ed015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.752+0000 7f9ecfe00700 1 -- 172.21.15.12:0/560642036 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed00d2460 con 0x7f9ed00d1630 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.752+0000 7f9ecfe00700 1 --2- 172.21.15.12:0/560642036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed00d1630 0x7f9ed00d1a90 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc0098f0 tx=0x7f9ebc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c111ce836bd61165 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.752+0000 7f9ecf400700 1 -- 172.21.15.12:0/560642036 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc011830 con 0x7f9ed00d1630 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.752+0000 7f9ecf400700 1 -- 172.21.15.12:0/560642036 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ebc011990 con 0x7f9ed00d1630 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.752+0000 7f9ecf400700 1 -- 172.21.15.12:0/560642036 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc011d20 con 0x7f9ed00d1630 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.753+0000 7f9ed7a00700 1 -- 172.21.15.12:0/560642036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed00d1630 msgr2=0x7f9ed00d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.753+0000 7f9ed7a00700 1 --2- 172.21.15.12:0/560642036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed00d1630 0x7f9ed00d1a90 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc0098f0 tx=0x7f9ebc009c00 comp rx=0 tx=0).stop 2024-03-29T10:15:13.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.753+0000 7f9ed7a00700 1 -- 172.21.15.12:0/560642036 shutdown_connections 2024-03-29T10:15:13.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.753+0000 7f9ed7a00700 1 --2- 172.21.15.12:0/560642036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed015dad0 0x7f9ed00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:13.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.753+0000 7f9ed7a00700 1 --2- 172.21.15.12:0/560642036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed00d1630 0x7f9ed00d1a90 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:13.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.753+0000 7f9ed7a00700 1 -- 172.21.15.12:0/560642036 >> 172.21.15.12:0/560642036 conn(0x7f9ed008aff0 msgr2=0x7f9ed00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:13.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.753+0000 7f9ed7a00700 1 -- 172.21.15.12:0/560642036 shutdown_connections 2024-03-29T10:15:13.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.753+0000 7f9ed7a00700 1 -- 172.21.15.12:0/560642036 wait complete. 2024-03-29T10:15:13.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.755+0000 7f9ed7a00700 1 Processor -- start 2024-03-29T10:15:13.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.755+0000 7f9ed7a00700 1 -- start start 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ed7a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d1630 0x7f9ed01548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ed7a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed00d1fd0 0x7f9ed00b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ed7a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015dad0 0x7f9ed015e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ed7a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00b8240 con 0x7f9ed00d1fd0 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ed7a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00b8470 con 0x7f9ed015dad0 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ed7a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00b8750 con 0x7f9ed00d1630 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ecfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed00d1fd0 0x7f9ed00b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ed5000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d1630 0x7f9ed01548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ecfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed00d1fd0 0x7f9ed00b7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39288/0 (socket says 172.21.15.12:39288) 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ecfe00700 1 -- 172.21.15.12:0/1485642501 learned_addr learned my addr 172.21.15.12:0/1485642501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.756+0000 7f9ed5a00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015dad0 0x7f9ed015e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9ed5000700 1 -- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015dad0 msgr2=0x7f9ed015e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9ed5000700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015dad0 0x7f9ed015e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9ed5000700 1 -- 172.21.15.12:0/1485642501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed00d1fd0 msgr2=0x7f9ed00b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9ed5000700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed00d1fd0 0x7f9ed00b7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9ed5000700 1 -- 172.21.15.12:0/1485642501 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc009550 con 0x7f9ed00d1630 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9ed5a00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015dad0 0x7f9ed015e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9ed5000700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d1630 0x7f9ed01548b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400e870 tx=0x7f9ec400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9eccc00700 1 -- 172.21.15.12:0/1485642501 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4013680 con 0x7f9ed00d1630 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9eccc00700 1 -- 172.21.15.12:0/1485642501 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec40137e0 con 0x7f9ed00d1630 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9ed7a00700 1 -- 172.21.15.12:0/1485642501 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed00c71f0 con 0x7f9ed00d1630 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9eccc00700 1 -- 172.21.15.12:0/1485642501 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4013b70 con 0x7f9ed00d1630 2024-03-29T10:15:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.757+0000 7f9ed7a00700 1 -- 172.21.15.12:0/1485642501 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed015eb70 con 0x7f9ed00d1630 2024-03-29T10:15:13.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.758+0000 7f9ed7a00700 1 -- 172.21.15.12:0/1485642501 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed00c7050 con 0x7f9ed00d1630 2024-03-29T10:15:13.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.759+0000 7f9eccc00700 1 -- 172.21.15.12:0/1485642501 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9ec4013cd0 con 0x7f9ed00d1630 2024-03-29T10:15:13.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.760+0000 7f9eccc00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb006adb0 0x7f9eb006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:13.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.760+0000 7f9eccc00700 1 -- 172.21.15.12:0/1485642501 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f9ec4089340 con 0x7f9ed00d1630 2024-03-29T10:15:13.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.760+0000 7f9ecfe00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb006adb0 0x7f9eb006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:13.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.761+0000 7f9ecfe00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb006adb0 0x7f9eb006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00ca150 tx=0x7f9ebc008030 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:13.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.765+0000 7f9eccc00700 1 -- 172.21.15.12:0/1485642501 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ed00c7050 con 0x7f9ed00d1630 2024-03-29T10:15:13.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:13.994+0000 7f9ed7a00700 1 -- 172.21.15.12:0/1485642501 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9ed00c7050 con 0x7f9ed00d1630 2024-03-29T10:15:14.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.714+0000 7f9eccc00700 1 -- 172.21.15.12:0/1485642501 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v150) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9ed00c7050 con 0x7f9ed00d1630 2024-03-29T10:15:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 -- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb006adb0 msgr2=0x7f9eb006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb006adb0 0x7f9eb006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00ca150 tx=0x7f9ebc008030 comp rx=0 tx=0).stop 2024-03-29T10:15:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 -- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d1630 msgr2=0x7f9ed01548b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d1630 0x7f9ed01548b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400e870 tx=0x7f9ec400eb80 comp rx=0 tx=0).stop 2024-03-29T10:15:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 -- 172.21.15.12:0/1485642501 shutdown_connections 2024-03-29T10:15:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9eb006adb0 0x7f9eb006d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015dad0 0x7f9ed015e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d1630 0x7f9ed01548b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 --2- 172.21.15.12:0/1485642501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed00d1fd0 0x7f9ed00b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 -- 172.21.15.12:0/1485642501 >> 172.21.15.12:0/1485642501 conn(0x7f9ed008aff0 msgr2=0x7f9ed006b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 -- 172.21.15.12:0/1485642501 shutdown_connections 2024-03-29T10:15:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.717+0000 7f9ed7a00700 1 -- 172.21.15.12:0/1485642501 wait complete. 2024-03-29T10:15:14.716 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T10:15:14.737 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T10:15:14.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.943+0000 7f1350400700 1 Processor -- start 2024-03-29T10:15:14.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.943+0000 7f1350400700 1 -- start start 2024-03-29T10:15:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.944+0000 7f1350400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f134815b5d0 0x7f13480ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.944+0000 7f1350400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13480cb3f0 0x7f13480ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.944+0000 7f1350400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f13480ae930 0x7f13480ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.944+0000 7f1350400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134815b9b0 con 0x7f134815b5d0 2024-03-29T10:15:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.944+0000 7f1350400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134815fd50 con 0x7f13480ae930 2024-03-29T10:15:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.944+0000 7f1350400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134815f140 con 0x7f13480cb3f0 2024-03-29T10:15:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.944+0000 7f134da00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f134815b5d0 0x7f13480ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.944+0000 7f134da00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f134815b5d0 0x7f13480ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39298/0 (socket says 172.21.15.12:39298) 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.944+0000 7f134da00700 1 -- 172.21.15.12:0/314475230 learned_addr learned my addr 172.21.15.12:0/314475230 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f134d000700 1 --2- 172.21.15.12:0/314475230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13480cb3f0 0x7f13480ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f134e400700 1 --2- 172.21.15.12:0/314475230 >> v2:172.21.15.159:3300/0 conn(0x7f13480ae930 0x7f13480ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f134da00700 1 -- 172.21.15.12:0/314475230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13480cb3f0 msgr2=0x7f13480ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f134da00700 1 --2- 172.21.15.12:0/314475230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13480cb3f0 0x7f13480ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f134da00700 1 -- 172.21.15.12:0/314475230 >> v2:172.21.15.159:3300/0 conn(0x7f13480ae930 msgr2=0x7f13480ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f134da00700 1 --2- 172.21.15.12:0/314475230 >> v2:172.21.15.159:3300/0 conn(0x7f13480ae930 0x7f13480ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f134da00700 1 -- 172.21.15.12:0/314475230 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134815f420 con 0x7f134815b5d0 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f134da00700 1 --2- 172.21.15.12:0/314475230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f134815b5d0 0x7f13480ae3f0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f13480b0d70 tx=0x7f133c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ec46f93ea095f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f1347e00700 1 -- 172.21.15.12:0/314475230 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c00b870 con 0x7f134815b5d0 2024-03-29T10:15:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f1347e00700 1 -- 172.21.15.12:0/314475230 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c00b9d0 con 0x7f134815b5d0 2024-03-29T10:15:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.945+0000 7f1347e00700 1 -- 172.21.15.12:0/314475230 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c00bd60 con 0x7f134815b5d0 2024-03-29T10:15:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.946+0000 7f1350400700 1 -- 172.21.15.12:0/314475230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f134815b5d0 msgr2=0x7f13480ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.946+0000 7f1350400700 1 --2- 172.21.15.12:0/314475230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f134815b5d0 0x7f13480ae3f0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f13480b0d70 tx=0x7f133c009bd0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.946+0000 7f1350400700 1 -- 172.21.15.12:0/314475230 shutdown_connections 2024-03-29T10:15:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.946+0000 7f1350400700 1 --2- 172.21.15.12:0/314475230 >> v2:172.21.15.159:3300/0 conn(0x7f13480ae930 0x7f13480ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.946+0000 7f1350400700 1 --2- 172.21.15.12:0/314475230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f134815b5d0 0x7f13480ae3f0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.946+0000 7f1350400700 1 --2- 172.21.15.12:0/314475230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13480cb3f0 0x7f13480ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.946+0000 7f1350400700 1 -- 172.21.15.12:0/314475230 >> 172.21.15.12:0/314475230 conn(0x7f13480cf190 msgr2=0x7f13480cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.946+0000 7f1350400700 1 -- 172.21.15.12:0/314475230 shutdown_connections 2024-03-29T10:15:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.946+0000 7f1350400700 1 -- 172.21.15.12:0/314475230 wait complete. 2024-03-29T10:15:14.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.948+0000 7f1350400700 1 Processor -- start 2024-03-29T10:15:14.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.948+0000 7f1350400700 1 -- start start 2024-03-29T10:15:14.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.949+0000 7f1350400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13480ae930 0x7f1348152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:14.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.949+0000 7f1350400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f134815b5d0 0x7f13481f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:14.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.949+0000 7f1350400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13480b4390 0x7f13480b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:14.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.949+0000 7f1350400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13480cbcd0 con 0x7f13480b4390 2024-03-29T10:15:14.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.949+0000 7f1350400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13480cbfb0 con 0x7f134815b5d0 2024-03-29T10:15:14.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.949+0000 7f1350400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13480cc290 con 0x7f13480ae930 2024-03-29T10:15:14.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.949+0000 7f134e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13480b4390 0x7f13480b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:14.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.949+0000 7f134e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13480b4390 0x7f13480b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39302/0 (socket says 172.21.15.12:39302) 2024-03-29T10:15:14.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.949+0000 7f134e400700 1 -- 172.21.15.12:0/4097468961 learned_addr learned my addr 172.21.15.12:0/4097468961 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:14.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.949+0000 7f134da00700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13480ae930 0x7f1348152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:14.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f134d000700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f134815b5d0 0x7f13481f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:14.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f134e400700 1 -- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13480ae930 msgr2=0x7f1348152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:14.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f134e400700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13480ae930 0x7f1348152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f134e400700 1 -- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f134815b5d0 msgr2=0x7f13481f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:14.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f134e400700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f134815b5d0 0x7f13481f3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:14.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f134e400700 1 -- 172.21.15.12:0/4097468961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c009550 con 0x7f13480b4390 2024-03-29T10:15:14.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f134e400700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13480b4390 0x7f13480b3300 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f1340007d60 tx=0x7f134000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:14.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f1345600700 1 -- 172.21.15.12:0/4097468961 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134000ede0 con 0x7f13480b4390 2024-03-29T10:15:14.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f1345600700 1 -- 172.21.15.12:0/4097468961 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1340014420 con 0x7f13480b4390 2024-03-29T10:15:14.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f1350400700 1 -- 172.21.15.12:0/4097468961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13480cc570 con 0x7f13480b4390 2024-03-29T10:15:14.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f1345600700 1 -- 172.21.15.12:0/4097468961 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13400147d0 con 0x7f13480b4390 2024-03-29T10:15:14.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.950+0000 7f1350400700 1 -- 172.21.15.12:0/4097468961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13481f4350 con 0x7f13480b4390 2024-03-29T10:15:14.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.951+0000 7f1350400700 1 -- 172.21.15.12:0/4097468961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1348067720 con 0x7f13480b4390 2024-03-29T10:15:14.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.952+0000 7f1345600700 1 -- 172.21.15.12:0/4097468961 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1340014930 con 0x7f13480b4390 2024-03-29T10:15:14.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.954+0000 7f1345600700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f132806adb0 0x7f132806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:14.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.954+0000 7f1345600700 1 -- 172.21.15.12:0/4097468961 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f13400897e0 con 0x7f13480b4390 2024-03-29T10:15:14.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.954+0000 7f134da00700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f132806adb0 0x7f132806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:14.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.955+0000 7f134da00700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f132806adb0 0x7f132806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f133c014040 tx=0x7f133c008030 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:14.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:14.959+0000 7f1345600700 1 -- 172.21.15.12:0/4097468961 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1340054bf0 con 0x7f13480b4390 2024-03-29T10:15:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.259+0000 7f1350400700 1 -- 172.21.15.12:0/4097468961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f134806ea60 con 0x7f13480b4390 2024-03-29T10:15:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.259+0000 7f1345600700 1 -- 172.21.15.12:0/4097468961 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f1340058210 con 0x7f13480b4390 2024-03-29T10:15:15.257 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:15.257 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:15:15.257 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T10:15:15.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.257 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T10:15:15.257 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T10:15:15.257 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:15:15.258 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.259 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:15:15.260 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T10:15:15.261 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:15:15.263 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.264 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T10:15:15.265 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T10:15:15.266 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T10:15:15.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T10:15:15.268 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T10:15:15.268 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.268 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.268 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:15:15.269 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T10:15:15.270 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:15:15.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 -- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f132806adb0 msgr2=0x7f132806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f132806adb0 0x7f132806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f133c014040 tx=0x7f133c008030 comp rx=0 tx=0).stop 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 -- 172.21.15.12:0/4097468961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13480b4390 msgr2=0x7f13480b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13480b4390 0x7f13480b3300 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f1340007d60 tx=0x7f134000e3c0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 -- 172.21.15.12:0/4097468961 shutdown_connections 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f132806adb0 0x7f132806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13480b4390 0x7f13480b3300 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13480ae930 0x7f1348152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 --2- 172.21.15.12:0/4097468961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f134815b5d0 0x7f13481f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 -- 172.21.15.12:0/4097468961 >> 172.21.15.12:0/4097468961 conn(0x7f13480cf190 msgr2=0x7f134815d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 -- 172.21.15.12:0/4097468961 shutdown_connections 2024-03-29T10:15:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.262+0000 7f1350400700 1 -- 172.21.15.12:0/4097468961 wait complete. 2024-03-29T10:15:15.277 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:15:15.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.478+0000 7f96bf000700 1 Processor -- start 2024-03-29T10:15:15.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.478+0000 7f96bf000700 1 -- start start 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.479+0000 7f96bf000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96b815b5d0 0x7f96b80ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.479+0000 7f96bf000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f96b80ac7d0 0x7f96b806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.479+0000 7f96bf000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b815fd50 con 0x7f96b80cb3f0 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.479+0000 7f96bf000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b815b9b0 con 0x7f96b815b5d0 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.480+0000 7f96bf000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b815f140 con 0x7f96b80ac7d0 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.480+0000 7f96b7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96b815b5d0 0x7f96b80ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.480+0000 7f96b7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96b815b5d0 0x7f96b80ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39318/0 (socket says 172.21.15.12:39318) 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.480+0000 7f96b7e00700 1 -- 172.21.15.12:0/2809656558 learned_addr learned my addr 172.21.15.12:0/2809656558 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.480+0000 7f96bd000700 1 --2- 172.21.15.12:0/2809656558 >> v2:172.21.15.159:3300/0 conn(0x7f96b80ac7d0 0x7f96b806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.480+0000 7f96b7e00700 1 -- 172.21.15.12:0/2809656558 >> v2:172.21.15.159:3300/0 conn(0x7f96b80ac7d0 msgr2=0x7f96b806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:15.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.480+0000 7f96b7e00700 1 --2- 172.21.15.12:0/2809656558 >> v2:172.21.15.159:3300/0 conn(0x7f96b80ac7d0 0x7f96b806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.480+0000 7f96b7e00700 1 -- 172.21.15.12:0/2809656558 >> v1:172.21.15.159:6789/0 conn(0x7f96b80cb3f0 legacy=0x7f96b80ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:15.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.480+0000 7f96b7e00700 1 -- 172.21.15.12:0/2809656558 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b815f420 con 0x7f96b815b5d0 2024-03-29T10:15:15.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.481+0000 7f96b7e00700 1 --2- 172.21.15.12:0/2809656558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 0x7f96b80ae3f0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f96b80cfdc0 tx=0x7f96a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244f62481e3de95b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:15.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.481+0000 7f96b6a00700 1 -- 172.21.15.12:0/2809656558 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8004230 con 0x7f96b815b5d0 2024-03-29T10:15:15.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.481+0000 7f96b6a00700 1 -- 172.21.15.12:0/2809656558 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a8004390 con 0x7f96b815b5d0 2024-03-29T10:15:15.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.481+0000 7f96b6a00700 1 -- 172.21.15.12:0/2809656558 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a80036a0 con 0x7f96b815b5d0 2024-03-29T10:15:15.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.481+0000 7f96bf000700 1 -- 172.21.15.12:0/2809656558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 msgr2=0x7f96b80ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.481+0000 7f96bf000700 1 --2- 172.21.15.12:0/2809656558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 0x7f96b80ae3f0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f96b80cfdc0 tx=0x7f96a8009bd0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.481+0000 7f96bf000700 1 -- 172.21.15.12:0/2809656558 shutdown_connections 2024-03-29T10:15:15.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.481+0000 7f96bf000700 1 --2- 172.21.15.12:0/2809656558 >> v2:172.21.15.159:3300/0 conn(0x7f96b80ac7d0 0x7f96b806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.481+0000 7f96bf000700 1 --2- 172.21.15.12:0/2809656558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 0x7f96b80ae3f0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.481+0000 7f96bf000700 1 -- 172.21.15.12:0/2809656558 >> 172.21.15.12:0/2809656558 conn(0x7f96b80cf190 msgr2=0x7f96b80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:15.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.482+0000 7f96bf000700 1 -- 172.21.15.12:0/2809656558 shutdown_connections 2024-03-29T10:15:15.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.482+0000 7f96bf000700 1 -- 172.21.15.12:0/2809656558 wait complete. 2024-03-29T10:15:15.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.483+0000 7f96bf000700 1 Processor -- start 2024-03-29T10:15:15.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.483+0000 7f96bf000700 1 -- start start 2024-03-29T10:15:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.484+0000 7f96bf000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96b80ac7d0 0x7f96b80c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.484+0000 7f96bf000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 0x7f96b8071890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.484+0000 7f96bf000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96b80ce7b0 0x7f96b80cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.484+0000 7f96bf000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b8071dd0 con 0x7f96b815b5d0 2024-03-29T10:15:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.484+0000 7f96bf000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b80c8a40 con 0x7f96b80ce7b0 2024-03-29T10:15:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.484+0000 7f96bf000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b80c8d20 con 0x7f96b80ac7d0 2024-03-29T10:15:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96b7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 0x7f96b8071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96b7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 0x7f96b8071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39334/0 (socket says 172.21.15.12:39334) 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96b7400700 1 -- 172.21.15.12:0/3072830562 learned_addr learned my addr 172.21.15.12:0/3072830562 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96b7e00700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96b80ac7d0 0x7f96b80c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96bd000700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96b80ce7b0 0x7f96b80cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96b7400700 1 -- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96b80ac7d0 msgr2=0x7f96b80c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96b7400700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96b80ac7d0 0x7f96b80c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96b7400700 1 -- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96b80ce7b0 msgr2=0x7f96b80cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96b7400700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96b80ce7b0 0x7f96b80cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96b7400700 1 -- 172.21.15.12:0/3072830562 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a8009550 con 0x7f96b815b5d0 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96bd000700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96b80ce7b0 0x7f96b80cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.485+0000 7f96b7400700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 0x7f96b8071890 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f96a400c4d0 tx=0x7f96a400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.486+0000 7f96afe00700 1 -- 172.21.15.12:0/3072830562 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a4011790 con 0x7f96b815b5d0 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.486+0000 7f96afe00700 1 -- 172.21.15.12:0/3072830562 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a40118f0 con 0x7f96b815b5d0 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.486+0000 7f96afe00700 1 -- 172.21.15.12:0/3072830562 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a4011ca0 con 0x7f96b815b5d0 2024-03-29T10:15:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.486+0000 7f96bf000700 1 -- 172.21.15.12:0/3072830562 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b80c9000 con 0x7f96b815b5d0 2024-03-29T10:15:15.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.486+0000 7f96bf000700 1 -- 172.21.15.12:0/3072830562 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b80c95c0 con 0x7f96b815b5d0 2024-03-29T10:15:15.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.487+0000 7f96bf000700 1 -- 172.21.15.12:0/3072830562 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b8067720 con 0x7f96b815b5d0 2024-03-29T10:15:15.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.488+0000 7f96afe00700 1 -- 172.21.15.12:0/3072830562 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f96a401c460 con 0x7f96b815b5d0 2024-03-29T10:15:15.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.489+0000 7f96afe00700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f969806adb0 0x7f969806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.489+0000 7f96afe00700 1 -- 172.21.15.12:0/3072830562 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f96a4088eb0 con 0x7f96b815b5d0 2024-03-29T10:15:15.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.489+0000 7f96b7e00700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f969806adb0 0x7f969806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.490+0000 7f96b7e00700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f969806adb0 0x7f969806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009130 tx=0x7f96a8008030 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:15.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.493+0000 7f96afe00700 1 -- 172.21.15.12:0/3072830562 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f96a40542d0 con 0x7f96b815b5d0 2024-03-29T10:15:15.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.709+0000 7f96bf000700 1 -- 172.21.15.12:0/3072830562 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96b809d1a0 con 0x7f96b815b5d0 2024-03-29T10:15:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.710+0000 7f96afe00700 1 -- 172.21.15.12:0/3072830562 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+12399 (secure 0 0 0) 0x7f96a40578f0 con 0x7f96b815b5d0 2024-03-29T10:15:15.707 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:15.708 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":150,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:15:14.697305+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:15:15.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.712+0000 7f96bf000700 1 -- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f969806adb0 msgr2=0x7f969806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.712+0000 7f96bf000700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f969806adb0 0x7f969806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009130 tx=0x7f96a8008030 comp rx=0 tx=0).stop 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.712+0000 7f96bf000700 1 -- 172.21.15.12:0/3072830562 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 msgr2=0x7f96b8071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.712+0000 7f96bf000700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 0x7f96b8071890 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f96a400c4d0 tx=0x7f96a400c7e0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.713+0000 7f96bf000700 1 -- 172.21.15.12:0/3072830562 shutdown_connections 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.713+0000 7f96bf000700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f969806adb0 0x7f969806d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.713+0000 7f96bf000700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96b80ce7b0 0x7f96b80cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.713+0000 7f96bf000700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96b80ac7d0 0x7f96b80c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.713+0000 7f96bf000700 1 --2- 172.21.15.12:0/3072830562 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96b815b5d0 0x7f96b8071890 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.713+0000 7f96bf000700 1 -- 172.21.15.12:0/3072830562 >> 172.21.15.12:0/3072830562 conn(0x7f96b80cf190 msgr2=0x7f96b815dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.713+0000 7f96bf000700 1 -- 172.21.15.12:0/3072830562 shutdown_connections 2024-03-29T10:15:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.713+0000 7f96bf000700 1 -- 172.21.15.12:0/3072830562 wait complete. 2024-03-29T10:15:15.728 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T10:15:15.728 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T10:15:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.932+0000 7f6ee7400700 1 Processor -- start 2024-03-29T10:15:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.932+0000 7f6ee7400700 1 -- start start 2024-03-29T10:15:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.933+0000 7f6ee7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6ee01611b0 0x7f6ee015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.933+0000 7f6ee7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ee015f4d0 0x7f6ee00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.933+0000 7f6ee7400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee00ce270 con 0x7f6ee00d21f0 2024-03-29T10:15:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.933+0000 7f6ee7400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee015b600 con 0x7f6ee01611b0 2024-03-29T10:15:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.933+0000 7f6ee7400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00d2760 con 0x7f6ee015f4d0 2024-03-29T10:15:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.933+0000 7f6ee4a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6ee01611b0 0x7f6ee015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.933+0000 7f6ee4a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6ee01611b0 0x7f6ee015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39348/0 (socket says 172.21.15.12:39348) 2024-03-29T10:15:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.933+0000 7f6ee4a00700 1 -- 172.21.15.12:0/530285652 learned_addr learned my addr 172.21.15.12:0/530285652 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.934+0000 7f6edfe00700 1 --2- 172.21.15.12:0/530285652 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ee015f4d0 0x7f6ee00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.934+0000 7f6ee4a00700 1 -- 172.21.15.12:0/530285652 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ee015f4d0 msgr2=0x7f6ee00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.934+0000 7f6ee4a00700 1 --2- 172.21.15.12:0/530285652 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ee015f4d0 0x7f6ee00ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.934+0000 7f6ee4a00700 1 -- 172.21.15.12:0/530285652 >> v1:172.21.15.12:6789/0 conn(0x7f6ee00d21f0 legacy=0x7f6ee0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.934+0000 7f6ee4a00700 1 -- 172.21.15.12:0/530285652 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee00d2a40 con 0x7f6ee01611b0 2024-03-29T10:15:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.934+0000 7f6ee4a00700 1 --2- 172.21.15.12:0/530285652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 0x7f6ee015ef90 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40098f0 tx=0x7f6ed4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31fc0dac6ddbefef server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.934+0000 7f6edf400700 1 -- 172.21.15.12:0/530285652 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4011830 con 0x7f6ee01611b0 2024-03-29T10:15:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.935+0000 7f6edf400700 1 -- 172.21.15.12:0/530285652 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed4011990 con 0x7f6ee01611b0 2024-03-29T10:15:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.935+0000 7f6edf400700 1 -- 172.21.15.12:0/530285652 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4011d20 con 0x7f6ee01611b0 2024-03-29T10:15:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.935+0000 7f6ee7400700 1 -- 172.21.15.12:0/530285652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 msgr2=0x7f6ee015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.935+0000 7f6ee7400700 1 --2- 172.21.15.12:0/530285652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 0x7f6ee015ef90 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40098f0 tx=0x7f6ed4009c00 comp rx=0 tx=0).stop 2024-03-29T10:15:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.935+0000 7f6ee7400700 1 -- 172.21.15.12:0/530285652 shutdown_connections 2024-03-29T10:15:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.935+0000 7f6ee7400700 1 --2- 172.21.15.12:0/530285652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 0x7f6ee015ef90 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.935+0000 7f6ee7400700 1 --2- 172.21.15.12:0/530285652 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ee015f4d0 0x7f6ee00ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.935+0000 7f6ee7400700 1 -- 172.21.15.12:0/530285652 >> 172.21.15.12:0/530285652 conn(0x7f6ee008aff0 msgr2=0x7f6ee00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.935+0000 7f6ee7400700 1 -- 172.21.15.12:0/530285652 shutdown_connections 2024-03-29T10:15:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.935+0000 7f6ee7400700 1 -- 172.21.15.12:0/530285652 wait complete. 2024-03-29T10:15:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.937+0000 7f6ee7400700 1 Processor -- start 2024-03-29T10:15:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.937+0000 7f6ee7400700 1 -- start start 2024-03-29T10:15:15.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6ee7400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ee00d21f0 0x7f6ee01f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6ee7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ee015f4d0 0x7f6ee01f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6ee7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 0x7f6ee009d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6ee7400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00caf10 con 0x7f6ee01611b0 2024-03-29T10:15:15.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6ee7400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00ca600 con 0x7f6ee00d21f0 2024-03-29T10:15:15.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6ee7400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee009e130 con 0x7f6ee015f4d0 2024-03-29T10:15:15.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6ee5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 0x7f6ee009d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6ee5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 0x7f6ee009d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39364/0 (socket says 172.21.15.12:39364) 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6ee5400700 1 -- 172.21.15.12:0/3846203539 learned_addr learned my addr 172.21.15.12:0/3846203539 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6ee4a00700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ee00d21f0 0x7f6ee01f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.938+0000 7f6edfe00700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ee015f4d0 0x7f6ee01f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.939+0000 7f6ee5400700 1 -- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ee015f4d0 msgr2=0x7f6ee01f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.939+0000 7f6ee5400700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ee015f4d0 0x7f6ee01f69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.939+0000 7f6ee5400700 1 -- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ee00d21f0 msgr2=0x7f6ee01f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.939+0000 7f6ee5400700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ee00d21f0 0x7f6ee01f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.939+0000 7f6ee5400700 1 -- 172.21.15.12:0/3846203539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed4009550 con 0x7f6ee01611b0 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.939+0000 7f6ee5400700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 0x7f6ee009d3e0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f6ed800cc50 tx=0x7f6ed800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.939+0000 7f6edcc00700 1 -- 172.21.15.12:0/3846203539 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed8003c00 con 0x7f6ee01611b0 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.939+0000 7f6edcc00700 1 -- 172.21.15.12:0/3846203539 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed8003d60 con 0x7f6ee01611b0 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.939+0000 7f6ee7400700 1 -- 172.21.15.12:0/3846203539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee00ca850 con 0x7f6ee01611b0 2024-03-29T10:15:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.939+0000 7f6edcc00700 1 -- 172.21.15.12:0/3846203539 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed800aab0 con 0x7f6ee01611b0 2024-03-29T10:15:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.940+0000 7f6ee7400700 1 -- 172.21.15.12:0/3846203539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee009d920 con 0x7f6ee01611b0 2024-03-29T10:15:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.940+0000 7f6ee7400700 1 -- 172.21.15.12:0/3846203539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee015d610 con 0x7f6ee01611b0 2024-03-29T10:15:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.941+0000 7f6edcc00700 1 -- 172.21.15.12:0/3846203539 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6ed800ac10 con 0x7f6ee01611b0 2024-03-29T10:15:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.942+0000 7f6edcc00700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ec006adb0 0x7f6ec006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.942+0000 7f6edcc00700 1 -- 172.21.15.12:0/3846203539 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6ed80890f0 con 0x7f6ee01611b0 2024-03-29T10:15:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.943+0000 7f6ee4a00700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ec006adb0 0x7f6ec006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.943+0000 7f6ee4a00700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ec006adb0 0x7f6ec006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4014040 tx=0x7f6ed4008030 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:15.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:15.947+0000 7f6edcc00700 1 -- 172.21.15.12:0/3846203539 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ed8054500 con 0x7f6ee01611b0 2024-03-29T10:15:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.160+0000 7f6ee7400700 1 -- 172.21.15.12:0/3846203539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ee0067720 con 0x7f6ee01611b0 2024-03-29T10:15:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.160+0000 7f6edcc00700 1 -- 172.21.15.12:0/3846203539 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v150) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ed8057b20 con 0x7f6ee01611b0 2024-03-29T10:15:16.158 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T10:15:16.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 -- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ec006adb0 msgr2=0x7f6ec006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ec006adb0 0x7f6ec006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4014040 tx=0x7f6ed4008030 comp rx=0 tx=0).stop 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 -- 172.21.15.12:0/3846203539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 msgr2=0x7f6ee009d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 0x7f6ee009d3e0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f6ed800cc50 tx=0x7f6ed800cf60 comp rx=0 tx=0).stop 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 -- 172.21.15.12:0/3846203539 shutdown_connections 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6ec006adb0 0x7f6ec006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ee01611b0 0x7f6ee009d3e0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ee00d21f0 0x7f6ee01f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 --2- 172.21.15.12:0/3846203539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ee015f4d0 0x7f6ee01f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 -- 172.21.15.12:0/3846203539 >> 172.21.15.12:0/3846203539 conn(0x7f6ee008aff0 msgr2=0x7f6ee015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 -- 172.21.15.12:0/3846203539 shutdown_connections 2024-03-29T10:15:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.163+0000 7f6ee7400700 1 -- 172.21.15.12:0/3846203539 wait complete. 2024-03-29T10:15:16.178 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T10:15:16.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-29T10:15:16.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.381+0000 7fe294800700 1 Processor -- start 2024-03-29T10:15:16.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.381+0000 7fe294800700 1 -- start start 2024-03-29T10:15:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.382+0000 7fe294800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe28c0cb3f0 0x7fe28c0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.382+0000 7fe294800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe28c0ac600 0x7fe28c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe294800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c15fd50 con 0x7fe28c15b5d0 2024-03-29T10:15:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe294800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe28c15b9b0 con 0x7fe28c0cb3f0 2024-03-29T10:15:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe294800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe28c0aca10 con 0x7fe28c0ac600 2024-03-29T10:15:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe291400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe28c0cb3f0 0x7fe28c0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe291400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe28c0cb3f0 0x7fe28c0ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39366/0 (socket says 172.21.15.12:39366) 2024-03-29T10:15:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe291400700 1 -- 172.21.15.12:0/1253611628 learned_addr learned my addr 172.21.15.12:0/1253611628 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:16.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe292800700 1 --2- 172.21.15.12:0/1253611628 >> v2:172.21.15.159:3300/0 conn(0x7fe28c0ac600 0x7fe28c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe291400700 1 -- 172.21.15.12:0/1253611628 >> v2:172.21.15.159:3300/0 conn(0x7fe28c0ac600 msgr2=0x7fe28c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:16.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe291400700 1 --2- 172.21.15.12:0/1253611628 >> v2:172.21.15.159:3300/0 conn(0x7fe28c0ac600 0x7fe28c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe291400700 1 -- 172.21.15.12:0/1253611628 >> v1:172.21.15.159:6789/0 conn(0x7fe28c15b5d0 legacy=0x7fe28c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe291400700 1 -- 172.21.15.12:0/1253611628 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c15f140 con 0x7fe28c0cb3f0 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.383+0000 7fe292800700 1 --2- 172.21.15.12:0/1253611628 >> v2:172.21.15.159:3300/0 conn(0x7fe28c0ac600 0x7fe28c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe291400700 1 --2- 172.21.15.12:0/1253611628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe28c0cb3f0 0x7fe28c0ac0c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0cfdc0 tx=0x7fe27c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d34bf8a6f44a3bd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe290a00700 1 -- 172.21.15.12:0/1253611628 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c004230 con 0x7fe28c0cb3f0 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe290a00700 1 -- 172.21.15.12:0/1253611628 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27c004390 con 0x7fe28c0cb3f0 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe290a00700 1 -- 172.21.15.12:0/1253611628 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c0036a0 con 0x7fe28c0cb3f0 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe294800700 1 -- 172.21.15.12:0/1253611628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe28c0cb3f0 msgr2=0x7fe28c0ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe294800700 1 --2- 172.21.15.12:0/1253611628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe28c0cb3f0 0x7fe28c0ac0c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0cfdc0 tx=0x7fe27c009bd0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe294800700 1 -- 172.21.15.12:0/1253611628 shutdown_connections 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe294800700 1 --2- 172.21.15.12:0/1253611628 >> v2:172.21.15.159:3300/0 conn(0x7fe28c0ac600 0x7fe28c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe294800700 1 --2- 172.21.15.12:0/1253611628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe28c0cb3f0 0x7fe28c0ac0c0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe294800700 1 -- 172.21.15.12:0/1253611628 >> 172.21.15.12:0/1253611628 conn(0x7fe28c0cf190 msgr2=0x7fe28c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.384+0000 7fe294800700 1 -- 172.21.15.12:0/1253611628 shutdown_connections 2024-03-29T10:15:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.385+0000 7fe294800700 1 -- 172.21.15.12:0/1253611628 wait complete. 2024-03-29T10:15:16.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.386+0000 7fe294800700 1 Processor -- start 2024-03-29T10:15:16.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.386+0000 7fe294800700 1 -- start start 2024-03-29T10:15:16.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.387+0000 7fe294800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe28c0ac600 0x7fe28c09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.387+0000 7fe294800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe28c15b5d0 0x7fe28c158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.387+0000 7fe294800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe28c156510 0x7fe28c158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.387+0000 7fe294800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c1441c0 con 0x7fe28c156510 2024-03-29T10:15:16.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.387+0000 7fe294800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c143ee0 con 0x7fe28c0ac600 2024-03-29T10:15:16.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.387+0000 7fe294800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c15a3d0 con 0x7fe28c15b5d0 2024-03-29T10:15:16.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe291e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe28c0ac600 0x7fe28c09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe292800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe28c156510 0x7fe28c158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe291e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe28c0ac600 0x7fe28c09cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:54234/0 (socket says 172.21.15.12:54234) 2024-03-29T10:15:16.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe291400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe28c15b5d0 0x7fe28c158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe291e00700 1 -- 172.21.15.12:0/3325911444 learned_addr learned my addr 172.21.15.12:0/3325911444 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:16.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe292800700 1 -- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe28c15b5d0 msgr2=0x7fe28c158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe292800700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe28c15b5d0 0x7fe28c158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe292800700 1 -- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe28c0ac600 msgr2=0x7fe28c09cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe292800700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe28c0ac600 0x7fe28c09cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe292800700 1 -- 172.21.15.12:0/3325911444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c009550 con 0x7fe28c156510 2024-03-29T10:15:16.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.388+0000 7fe292800700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe28c156510 0x7fe28c158760 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe28800cc90 tx=0x7fe28800a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:16.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.389+0000 7fe286000700 1 -- 172.21.15.12:0/3325911444 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28800acf0 con 0x7fe28c156510 2024-03-29T10:15:16.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.389+0000 7fe286000700 1 -- 172.21.15.12:0/3325911444 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28800ae70 con 0x7fe28c156510 2024-03-29T10:15:16.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.389+0000 7fe294800700 1 -- 172.21.15.12:0/3325911444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c15a6b0 con 0x7fe28c156510 2024-03-29T10:15:16.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.389+0000 7fe286000700 1 -- 172.21.15.12:0/3325911444 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28800f6a0 con 0x7fe28c156510 2024-03-29T10:15:16.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.389+0000 7fe294800700 1 -- 172.21.15.12:0/3325911444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c09d0a0 con 0x7fe28c156510 2024-03-29T10:15:16.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.390+0000 7fe294800700 1 -- 172.21.15.12:0/3325911444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c15a230 con 0x7fe28c156510 2024-03-29T10:15:16.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.391+0000 7fe286000700 1 -- 172.21.15.12:0/3325911444 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe28800b040 con 0x7fe28c156510 2024-03-29T10:15:16.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.392+0000 7fe286000700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe27006adb0 0x7fe27006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.392+0000 7fe286000700 1 -- 172.21.15.12:0/3325911444 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe288089900 con 0x7fe28c156510 2024-03-29T10:15:16.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.392+0000 7fe291e00700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe27006adb0 0x7fe27006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.393+0000 7fe291e00700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe27006adb0 0x7fe27006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe280003ac0 tx=0x7fe280003a50 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:16.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.396+0000 7fe286000700 1 -- 172.21.15.12:0/3325911444 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe28c15a230 con 0x7fe28c156510 2024-03-29T10:15:16.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.613+0000 7fe294800700 1 -- 172.21.15.12:0/3325911444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe28c15a230 con 0x7fe28c156510 2024-03-29T10:15:16.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.614+0000 7fe286000700 1 -- 172.21.15.12:0/3325911444 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v150) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe28c15a230 con 0x7fe28c156510 2024-03-29T10:15:16.611 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:16.612 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:15:16.612 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:16.612 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T10:15:16.612 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T10:15:16.612 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:16.612 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:15:16.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.616+0000 7fe294800700 1 -- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe27006adb0 msgr2=0x7fe27006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.616+0000 7fe294800700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe27006adb0 0x7fe27006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe280003ac0 tx=0x7fe280003a50 comp rx=0 tx=0).stop 2024-03-29T10:15:16.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.616+0000 7fe294800700 1 -- 172.21.15.12:0/3325911444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe28c156510 msgr2=0x7fe28c158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.616+0000 7fe294800700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe28c156510 0x7fe28c158760 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe28800cc90 tx=0x7fe28800a280 comp rx=0 tx=0).stop 2024-03-29T10:15:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.616+0000 7fe294800700 1 -- 172.21.15.12:0/3325911444 shutdown_connections 2024-03-29T10:15:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.616+0000 7fe294800700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe27006adb0 0x7fe27006d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.616+0000 7fe294800700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe28c156510 0x7fe28c158760 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.616+0000 7fe294800700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe28c0ac600 0x7fe28c09cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.617+0000 7fe294800700 1 --2- 172.21.15.12:0/3325911444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe28c15b5d0 0x7fe28c158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.617+0000 7fe294800700 1 -- 172.21.15.12:0/3325911444 >> 172.21.15.12:0/3325911444 conn(0x7fe28c0cf190 msgr2=0x7fe28c0af150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.617+0000 7fe294800700 1 -- 172.21.15.12:0/3325911444 shutdown_connections 2024-03-29T10:15:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.617+0000 7fe294800700 1 -- 172.21.15.12:0/3325911444 wait complete. 2024-03-29T10:15:16.632 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T10:15:16.632 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T10:15:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.837+0000 7f4c07200700 1 Processor -- start 2024-03-29T10:15:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.838+0000 7f4c07200700 1 -- start start 2024-03-29T10:15:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.839+0000 7f4c07200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c000d28e0 0x7f4c00168680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.839+0000 7f4c07200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c000ce8a0 con 0x7f4c0015ddf0 2024-03-29T10:15:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.839+0000 7f4c07200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c000cdc90 con 0x7f4c000ced30 2024-03-29T10:15:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.839+0000 7f4c07200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0015ec60 con 0x7f4c000d28e0 2024-03-29T10:15:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.839+0000 7f4bffe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4c0015ddf0 0x7f4c0015e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56496/0 (socket says 172.21.15.12:56496) 2024-03-29T10:15:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.839+0000 7f4bffe00700 1 -- 172.21.15.12:0/538663523 learned_addr learned my addr 172.21.15.12:0/538663523 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.839+0000 7f4c05200700 1 --2- 172.21.15.12:0/538663523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c000d28e0 0x7f4c00168680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.839+0000 7f4bfea00700 1 -- 172.21.15.12:0/538663523 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404918312 0 0) 0x7f4c000ce8a0 con 0x7f4c0015ddf0 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.840+0000 7f4bfea00700 1 -- 172.21.15.12:0/538663523 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be40035b0 con 0x7f4c0015ddf0 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.840+0000 7f4bfea00700 1 -- 172.21.15.12:0/538663523 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376316390 0 0) 0x7f4c000cdc90 con 0x7f4c000ced30 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.840+0000 7f4bfea00700 1 -- 172.21.15.12:0/538663523 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c000ce8a0 con 0x7f4c000ced30 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.840+0000 7f4c05200700 1 -- 172.21.15.12:0/538663523 >> v1:172.21.15.159:6789/0 conn(0x7f4c000ced30 legacy=0x7f4c000d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.840+0000 7f4c05200700 1 -- 172.21.15.12:0/538663523 >> v1:172.21.15.12:6789/0 conn(0x7f4c0015ddf0 legacy=0x7f4c0015e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.840+0000 7f4c05200700 1 -- 172.21.15.12:0/538663523 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c000cdf70 con 0x7f4c000d28e0 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.840+0000 7f4c05200700 1 --2- 172.21.15.12:0/538663523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c000d28e0 0x7f4c00168680 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009dc0 tx=0x7f4bf800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4608c232ba1c50a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.840+0000 7f4bfea00700 1 -- 172.21.15.12:0/538663523 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf800e070 con 0x7f4c000d28e0 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.840+0000 7f4bfea00700 1 -- 172.21.15.12:0/538663523 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf800bb20 con 0x7f4c000d28e0 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.840+0000 7f4bfea00700 1 -- 172.21.15.12:0/538663523 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8012430 con 0x7f4c000d28e0 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.841+0000 7f4c07200700 1 -- 172.21.15.12:0/538663523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c000d28e0 msgr2=0x7f4c00168680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.841+0000 7f4c07200700 1 --2- 172.21.15.12:0/538663523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c000d28e0 0x7f4c00168680 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009dc0 tx=0x7f4bf800b330 comp rx=0 tx=0).stop 2024-03-29T10:15:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.841+0000 7f4c07200700 1 -- 172.21.15.12:0/538663523 shutdown_connections 2024-03-29T10:15:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.841+0000 7f4c07200700 1 --2- 172.21.15.12:0/538663523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c000d28e0 0x7f4c00168680 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.841+0000 7f4c07200700 1 -- 172.21.15.12:0/538663523 >> 172.21.15.12:0/538663523 conn(0x7f4c0008aff0 msgr2=0x7f4c000d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.841+0000 7f4c07200700 1 -- 172.21.15.12:0/538663523 shutdown_connections 2024-03-29T10:15:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.841+0000 7f4c07200700 1 -- 172.21.15.12:0/538663523 wait complete. 2024-03-29T10:15:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.843+0000 7f4c07200700 1 Processor -- start 2024-03-29T10:15:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.843+0000 7f4c07200700 1 -- start start 2024-03-29T10:15:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4c07200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c000ced30 0x7f4c000b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4c07200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c0015ddf0 0x7f4c000cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4c07200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c000ccc50 0x7f4c000ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4c07200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c001f3af0 con 0x7f4c000ccc50 2024-03-29T10:15:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4c07200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c001f3dd0 con 0x7f4c000ced30 2024-03-29T10:15:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4c07200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c001f40b0 con 0x7f4c0015ddf0 2024-03-29T10:15:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4c05200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c000ccc50 0x7f4c000ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4c05200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c000ccc50 0x7f4c000ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39388/0 (socket says 172.21.15.12:39388) 2024-03-29T10:15:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4c05200700 1 -- 172.21.15.12:0/156981237 learned_addr learned my addr 172.21.15.12:0/156981237 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4bffe00700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c000ced30 0x7f4c000b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.844+0000 7f4bff400700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c0015ddf0 0x7f4c000cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4c05200700 1 -- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c0015ddf0 msgr2=0x7f4c000cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4c05200700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c0015ddf0 0x7f4c000cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4c05200700 1 -- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c000ced30 msgr2=0x7f4c000b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4c05200700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c000ced30 0x7f4c000b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4c05200700 1 -- 172.21.15.12:0/156981237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf8009a70 con 0x7f4c000ccc50 2024-03-29T10:15:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4c05200700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c000ccc50 0x7f4c000ca5e0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f4bf800b370 tx=0x7f4bf8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4bf7e00700 1 -- 172.21.15.12:0/156981237 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf801b810 con 0x7f4c000ccc50 2024-03-29T10:15:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4bf7e00700 1 -- 172.21.15.12:0/156981237 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf8020070 con 0x7f4c000ccc50 2024-03-29T10:15:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4c07200700 1 -- 172.21.15.12:0/156981237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c000cbc30 con 0x7f4c000ccc50 2024-03-29T10:15:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4bf7e00700 1 -- 172.21.15.12:0/156981237 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf801bbf0 con 0x7f4c000ccc50 2024-03-29T10:15:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.845+0000 7f4c07200700 1 -- 172.21.15.12:0/156981237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c000cc1f0 con 0x7f4c000ccc50 2024-03-29T10:15:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.846+0000 7f4c07200700 1 -- 172.21.15.12:0/156981237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c00067720 con 0x7f4c000ccc50 2024-03-29T10:15:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.847+0000 7f4bf7e00700 1 -- 172.21.15.12:0/156981237 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4bf801bd50 con 0x7f4c000ccc50 2024-03-29T10:15:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.849+0000 7f4bf7e00700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4be006adb0 0x7f4be006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.849+0000 7f4bf7e00700 1 -- 172.21.15.12:0/156981237 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4bf8022070 con 0x7f4c000ccc50 2024-03-29T10:15:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.849+0000 7f4bffe00700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4be006adb0 0x7f4be006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.850+0000 7f4bffe00700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4be006adb0 0x7f4be006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0000f60 tx=0x7f4bf0009040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:16.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:16.853+0000 7f4bf7e00700 1 -- 172.21.15.12:0/156981237 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4bf8056d20 con 0x7f4c000ccc50 2024-03-29T10:15:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.067+0000 7f4c07200700 1 -- 172.21.15.12:0/156981237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4c000d3050 con 0x7f4c000ccc50 2024-03-29T10:15:17.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.211+0000 7f4bf7e00700 1 -- 172.21.15.12:0/156981237 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v151) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4bf805a340 con 0x7f4c000ccc50 2024-03-29T10:15:17.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.214+0000 7f4c07200700 1 -- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4be006adb0 msgr2=0x7f4be006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:17.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.214+0000 7f4c07200700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4be006adb0 0x7f4be006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0000f60 tx=0x7f4bf0009040 comp rx=0 tx=0).stop 2024-03-29T10:15:17.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.215+0000 7f4c07200700 1 -- 172.21.15.12:0/156981237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c000ccc50 msgr2=0x7f4c000ca5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:17.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.215+0000 7f4c07200700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c000ccc50 0x7f4c000ca5e0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f4bf800b370 tx=0x7f4bf8004ad0 comp rx=0 tx=0).stop 2024-03-29T10:15:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.215+0000 7f4c07200700 1 -- 172.21.15.12:0/156981237 shutdown_connections 2024-03-29T10:15:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.215+0000 7f4c07200700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4be006adb0 0x7f4be006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.215+0000 7f4c07200700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c000ccc50 0x7f4c000ca5e0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.215+0000 7f4c07200700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c000ced30 0x7f4c000b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.215+0000 7f4c07200700 1 --2- 172.21.15.12:0/156981237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c0015ddf0 0x7f4c000cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.215+0000 7f4c07200700 1 -- 172.21.15.12:0/156981237 >> 172.21.15.12:0/156981237 conn(0x7f4c0008aff0 msgr2=0x7f4c000b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.215+0000 7f4c07200700 1 -- 172.21.15.12:0/156981237 shutdown_connections 2024-03-29T10:15:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.215+0000 7f4c07200700 1 -- 172.21.15.12:0/156981237 wait complete. 2024-03-29T10:15:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T10:15:17.235 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T10:15:17.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.443+0000 7f7180a00700 1 Processor -- start 2024-03-29T10:15:17.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.443+0000 7f7180a00700 1 -- start start 2024-03-29T10:15:17.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.445+0000 7f7180a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0d06a0 0x7f717c15e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:17.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.445+0000 7f7180a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717c15eda0 con 0x7f717c06fa20 2024-03-29T10:15:17.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.445+0000 7f7180a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717c099dd0 con 0x7f717c0cd5c0 2024-03-29T10:15:17.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.445+0000 7f7180a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c09a0b0 con 0x7f717c0d06a0 2024-03-29T10:15:17.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.445+0000 7f7179400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f717c06fa20 0x7f717c15cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56510/0 (socket says 172.21.15.12:56510) 2024-03-29T10:15:17.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.445+0000 7f7179400700 1 -- 172.21.15.12:0/2786342455 learned_addr learned my addr 172.21.15.12:0/2786342455 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:17.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.445+0000 7f717a800700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0d06a0 0x7f717c15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:17.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.445+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500834720 0 0) 0x7f7168001110 con 0x7f717c06fa20 2024-03-29T10:15:17.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.445+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71600035b0 con 0x7f717c06fa20 2024-03-29T10:15:17.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621346026 0 0) 0x7f717c099dd0 con 0x7f717c0cd5c0 2024-03-29T10:15:17.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7168001110 con 0x7f717c0cd5c0 2024-03-29T10:15:17.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2539198752 0 0) 0x7f71600035b0 con 0x7f717c06fa20 2024-03-29T10:15:17.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0d06a0 msgr2=0x7f717c15e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:17.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0d06a0 0x7f717c15e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:17.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 >> v1:172.21.15.159:6789/0 conn(0x7f717c0cd5c0 legacy=0x7f717c06f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:17.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c09a390 con 0x7f717c06fa20 2024-03-29T10:15:17.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f7168001800 con 0x7f717c06fa20 2024-03-29T10:15:17.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 >> v1:172.21.15.12:6789/0 conn(0x7f717c06fa20 legacy=0x7f717c15cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:17.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 0x7f7160003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:17.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0d06a0 0x7f7160004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:17.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7160004d90 0x7f7160005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:17.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c09a0b0 con 0x7f7160003bc0 2024-03-29T10:15:17.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716000bac0 con 0x7f7160004d90 2024-03-29T10:15:17.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716000bc80 con 0x7f717c0d06a0 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7179e00700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 0x7f7160003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f7179400700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0d06a0 0x7f7160004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.446+0000 7f717a800700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7160004d90 0x7f7160005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.447+0000 7f7179e00700 1 -- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0d06a0 msgr2=0x7f7160004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.447+0000 7f7179e00700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0d06a0 0x7f7160004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.447+0000 7f7179e00700 1 -- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7160004d90 msgr2=0x7f7160005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.447+0000 7f7179e00700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7160004d90 0x7f7160005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.447+0000 7f7179e00700 1 -- 172.21.15.12:0/2786342455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71600035b0 con 0x7f7160003bc0 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.447+0000 7f7179e00700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 0x7f7160003fc0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f716c001940 tx=0x7f7160005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9669d129a0a11853 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.447+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c001430 con 0x7f7160003bc0 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.447+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716c001d70 con 0x7f7160003bc0 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.447+0000 7f7178a00700 1 -- 172.21.15.12:0/2786342455 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c005510 con 0x7f7160003bc0 2024-03-29T10:15:17.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.447+0000 7f7180a00700 1 -- 172.21.15.12:0/2786342455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 msgr2=0x7f7160003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.448+0000 7f7180a00700 1 --2- 172.21.15.12:0/2786342455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 0x7f7160003fc0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f716c001940 tx=0x7f7160005880 comp rx=0 tx=0).stop 2024-03-29T10:15:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.448+0000 7f717a800700 1 -- 172.21.15.12:0/2786342455 reap_dead start 2024-03-29T10:15:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.448+0000 7f7180a00700 1 -- 172.21.15.12:0/2786342455 shutdown_connections 2024-03-29T10:15:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.448+0000 7f7180a00700 1 -- 172.21.15.12:0/2786342455 >> 172.21.15.12:0/2786342455 conn(0x7f717c0d1370 msgr2=0x7f717c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.448+0000 7f7180a00700 1 -- 172.21.15.12:0/2786342455 shutdown_connections 2024-03-29T10:15:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.448+0000 7f7180a00700 1 -- 172.21.15.12:0/2786342455 wait complete. 2024-03-29T10:15:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.449+0000 7f7180a00700 1 Processor -- start 2024-03-29T10:15:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.450+0000 7f7180a00700 1 -- start start 2024-03-29T10:15:17.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.451+0000 7f7180a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 0x7f717c159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:17.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.451+0000 7f7180a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f717c1549d0 0x7f717c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.451+0000 7f7180a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0c76e0 0x7f717c0c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.451+0000 7f7180a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0c8280 con 0x7f7160003bc0 2024-03-29T10:15:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.451+0000 7f7180a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c1591e0 con 0x7f717c1549d0 2024-03-29T10:15:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.451+0000 7f7180a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c1594c0 con 0x7f717c0c76e0 2024-03-29T10:15:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.451+0000 7f7179e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 0x7f717c159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.451+0000 7f7179e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 0x7f717c159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39400/0 (socket says 172.21.15.12:39400) 2024-03-29T10:15:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.451+0000 7f7179e00700 1 -- 172.21.15.12:0/358066308 learned_addr learned my addr 172.21.15.12:0/358066308 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:17.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.451+0000 7f7179400700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f717c1549d0 0x7f717c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7179e00700 1 -- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0c76e0 msgr2=0x7f717c0c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f717a800700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0c76e0 0x7f717c0c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7179e00700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0c76e0 0x7f717c0c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7179e00700 1 -- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f717c1549d0 msgr2=0x7f717c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7179e00700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f717c1549d0 0x7f717c15a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7179e00700 1 -- 172.21.15.12:0/358066308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c1597a0 con 0x7f7160003bc0 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f717a800700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0c76e0 0x7f717c0c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7179400700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f717c1549d0 0x7f717c15a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7179e00700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 0x7f717c159fa0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f716c00c3c0 tx=0x7f716c0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7172000700 1 -- 172.21.15.12:0/358066308 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c006c70 con 0x7f7160003bc0 2024-03-29T10:15:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7172000700 1 -- 172.21.15.12:0/358066308 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716c006dd0 con 0x7f7160003bc0 2024-03-29T10:15:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7180a00700 1 -- 172.21.15.12:0/358066308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f717c159a80 con 0x7f7160003bc0 2024-03-29T10:15:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7172000700 1 -- 172.21.15.12:0/358066308 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c007240 con 0x7f7160003bc0 2024-03-29T10:15:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.452+0000 7f7180a00700 1 -- 172.21.15.12:0/358066308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f717c1f31f0 con 0x7f7160003bc0 2024-03-29T10:15:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.453+0000 7f7180a00700 1 -- 172.21.15.12:0/358066308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717c159df0 con 0x7f7160003bc0 2024-03-29T10:15:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.455+0000 7f7172000700 1 -- 172.21.15.12:0/358066308 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f716c0073a0 con 0x7f7160003bc0 2024-03-29T10:15:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.456+0000 7f7172000700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f715c06adb0 0x7f715c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.456+0000 7f7172000700 1 -- 172.21.15.12:0/358066308 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f716c08a540 con 0x7f7160003bc0 2024-03-29T10:15:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.456+0000 7f7179400700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f715c06adb0 0x7f715c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.457+0000 7f7179400700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f715c06adb0 0x7f715c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7168007fd0 tx=0x7f7168009040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.460+0000 7f7172000700 1 -- 172.21.15.12:0/358066308 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f716c055860 con 0x7f7160003bc0 2024-03-29T10:15:17.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:17.674+0000 7f7180a00700 1 -- 172.21.15.12:0/358066308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f717c067720 con 0x7f7160003bc0 2024-03-29T10:15:18.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.225+0000 7f7172000700 1 -- 172.21.15.12:0/358066308 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v152) v1 ==== 170+0+0 (secure 0 0 0) 0x7f716c058e80 con 0x7f7160003bc0 2024-03-29T10:15:18.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.232+0000 7f7180a00700 1 -- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f715c06adb0 msgr2=0x7f715c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.232+0000 7f7180a00700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f715c06adb0 0x7f715c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7168007fd0 tx=0x7f7168009040 comp rx=0 tx=0).stop 2024-03-29T10:15:18.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.232+0000 7f7180a00700 1 -- 172.21.15.12:0/358066308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 msgr2=0x7f717c159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.232+0000 7f7180a00700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 0x7f717c159fa0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f716c00c3c0 tx=0x7f716c0012d0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.233+0000 7f7180a00700 1 -- 172.21.15.12:0/358066308 shutdown_connections 2024-03-29T10:15:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.233+0000 7f7180a00700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f715c06adb0 0x7f715c06d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.233+0000 7f7180a00700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f717c0c76e0 0x7f717c0c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.233+0000 7f7180a00700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160003bc0 0x7f717c159fa0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.233+0000 7f7180a00700 1 --2- 172.21.15.12:0/358066308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f717c1549d0 0x7f717c15a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.233+0000 7f7180a00700 1 -- 172.21.15.12:0/358066308 >> 172.21.15.12:0/358066308 conn(0x7f717c0d1370 msgr2=0x7f717c06b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.235+0000 7f7180a00700 1 -- 172.21.15.12:0/358066308 shutdown_connections 2024-03-29T10:15:18.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.235+0000 7f7180a00700 1 -- 172.21.15.12:0/358066308 wait complete. 2024-03-29T10:15:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T10:15:18.262 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T10:15:18.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.466+0000 7f5362e00700 1 Processor -- start 2024-03-29T10:15:18.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.466+0000 7f5362e00700 1 -- start start 2024-03-29T10:15:18.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.467+0000 7f5362e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f535c0d16a0 0x7f535c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.467+0000 7f5362e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d2370 0x7f535c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.467+0000 7f5362e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535c161420 con 0x7f535c15d9d0 2024-03-29T10:15:18.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.467+0000 7f5362e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f535c0d2820 con 0x7f535c0d16a0 2024-03-29T10:15:18.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.467+0000 7f5362e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f535c0d4340 con 0x7f535c0d2370 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.468+0000 7f5360e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d2370 0x7f535c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.468+0000 7f535b400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f535c0d16a0 0x7f535c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.468+0000 7f5360e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d2370 0x7f535c0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:37704/0 (socket says 172.21.15.12:37704) 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.468+0000 7f5360e00700 1 -- 172.21.15.12:0/798334080 learned_addr learned my addr 172.21.15.12:0/798334080 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.468+0000 7f5360e00700 1 -- 172.21.15.12:0/798334080 >> v2:172.21.15.159:3300/0 conn(0x7f535c0d16a0 msgr2=0x7f535c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.468+0000 7f5360e00700 1 --2- 172.21.15.12:0/798334080 >> v2:172.21.15.159:3300/0 conn(0x7f535c0d16a0 0x7f535c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.468+0000 7f5360e00700 1 -- 172.21.15.12:0/798334080 >> v1:172.21.15.159:6789/0 conn(0x7f535c15d9d0 legacy=0x7f535c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.468+0000 7f5360e00700 1 -- 172.21.15.12:0/798334080 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535c161700 con 0x7f535c0d2370 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.468+0000 7f5360e00700 1 --2- 172.21.15.12:0/798334080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d2370 0x7f535c0ce360 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5354009e50 tx=0x7f535400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33ae0af0ffe934f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f535aa00700 1 -- 172.21.15.12:0/798334080 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535400e070 con 0x7f535c0d2370 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f535aa00700 1 -- 172.21.15.12:0/798334080 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535400b910 con 0x7f535c0d2370 2024-03-29T10:15:18.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f535aa00700 1 -- 172.21.15.12:0/798334080 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535400bcc0 con 0x7f535c0d2370 2024-03-29T10:15:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f5362e00700 1 -- 172.21.15.12:0/798334080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d2370 msgr2=0x7f535c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f5362e00700 1 --2- 172.21.15.12:0/798334080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d2370 0x7f535c0ce360 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5354009e50 tx=0x7f535400b330 comp rx=0 tx=0).stop 2024-03-29T10:15:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f5362e00700 1 -- 172.21.15.12:0/798334080 shutdown_connections 2024-03-29T10:15:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f5362e00700 1 --2- 172.21.15.12:0/798334080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d2370 0x7f535c0ce360 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f5362e00700 1 --2- 172.21.15.12:0/798334080 >> v2:172.21.15.159:3300/0 conn(0x7f535c0d16a0 0x7f535c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f5362e00700 1 -- 172.21.15.12:0/798334080 >> 172.21.15.12:0/798334080 conn(0x7f535c08aff0 msgr2=0x7f535c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f5362e00700 1 -- 172.21.15.12:0/798334080 shutdown_connections 2024-03-29T10:15:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.469+0000 7f5362e00700 1 -- 172.21.15.12:0/798334080 wait complete. 2024-03-29T10:15:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.471+0000 7f5362e00700 1 Processor -- start 2024-03-29T10:15:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.471+0000 7f5362e00700 1 -- start start 2024-03-29T10:15:18.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f5362e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d16a0 0x7f535c160220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f5362e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f535c0d2370 0x7f535c158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f5362e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f535c15d9d0 0x7f535c0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f5362e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535c159370 con 0x7f535c15d9d0 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f5362e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535c15f180 con 0x7f535c0d2370 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f5362e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f535c15f460 con 0x7f535c0d16a0 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f5360e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f535c15d9d0 0x7f535c0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f535be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d16a0 0x7f535c160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f5360e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f535c15d9d0 0x7f535c0cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39410/0 (socket says 172.21.15.12:39410) 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f535b400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f535c0d2370 0x7f535c158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.472+0000 7f5360e00700 1 -- 172.21.15.12:0/2441650845 learned_addr learned my addr 172.21.15.12:0/2441650845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.473+0000 7f535b400700 1 -- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d16a0 msgr2=0x7f535c160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.473+0000 7f535b400700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d16a0 0x7f535c160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.473+0000 7f535b400700 1 -- 172.21.15.12:0/2441650845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f535c15d9d0 msgr2=0x7f535c0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.473+0000 7f535b400700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f535c15d9d0 0x7f535c0cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.473+0000 7f535b400700 1 -- 172.21.15.12:0/2441650845 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5354009b30 con 0x7f535c0d2370 2024-03-29T10:15:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.473+0000 7f535b400700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f535c0d2370 0x7f535c158e30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f535c0d1ae0 tx=0x7f534800ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.473+0000 7f5353e00700 1 -- 172.21.15.12:0/2441650845 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348013680 con 0x7f535c0d2370 2024-03-29T10:15:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.474+0000 7f5353e00700 1 -- 172.21.15.12:0/2441650845 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53480137e0 con 0x7f535c0d2370 2024-03-29T10:15:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.474+0000 7f5362e00700 1 -- 172.21.15.12:0/2441650845 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535c15f740 con 0x7f535c0d2370 2024-03-29T10:15:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.474+0000 7f5353e00700 1 -- 172.21.15.12:0/2441650845 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348013b50 con 0x7f535c0d2370 2024-03-29T10:15:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.474+0000 7f5362e00700 1 -- 172.21.15.12:0/2441650845 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535c15fd00 con 0x7f535c0d2370 2024-03-29T10:15:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.475+0000 7f5362e00700 1 -- 172.21.15.12:0/2441650845 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f535c15fec0 con 0x7f535c0d2370 2024-03-29T10:15:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.476+0000 7f5353e00700 1 -- 172.21.15.12:0/2441650845 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5348013cb0 con 0x7f535c0d2370 2024-03-29T10:15:18.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.477+0000 7f5353e00700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f533c06adb0 0x7f533c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.477+0000 7f5353e00700 1 -- 172.21.15.12:0/2441650845 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f5348089500 con 0x7f535c0d2370 2024-03-29T10:15:18.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.477+0000 7f535be00700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f533c06adb0 0x7f533c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.478+0000 7f535be00700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f533c06adb0 0x7f533c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f535c1f6670 tx=0x7f534c009040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:18.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.481+0000 7f5353e00700 1 -- 172.21.15.12:0/2441650845 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5348054610 con 0x7f535c0d2370 2024-03-29T10:15:18.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.739+0000 7f5362e00700 1 -- 172.21.15.12:0/2441650845 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f535c0d1ca0 con 0x7f535c0d2370 2024-03-29T10:15:18.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.751+0000 7f5353e00700 1 -- 172.21.15.12:0/2441650845 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v69) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5348057c30 con 0x7f535c0d2370 2024-03-29T10:15:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.756+0000 7f5362e00700 1 -- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f533c06adb0 msgr2=0x7f533c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.756+0000 7f5362e00700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f533c06adb0 0x7f533c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f535c1f6670 tx=0x7f534c009040 comp rx=0 tx=0).stop 2024-03-29T10:15:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.756+0000 7f5362e00700 1 -- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f535c0d2370 msgr2=0x7f535c158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.756+0000 7f5362e00700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f535c0d2370 0x7f535c158e30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f535c0d1ae0 tx=0x7f534800ebb0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.757+0000 7f5362e00700 1 -- 172.21.15.12:0/2441650845 shutdown_connections 2024-03-29T10:15:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.757+0000 7f5362e00700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f533c06adb0 0x7f533c06d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.757+0000 7f5362e00700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f535c15d9d0 0x7f535c0cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.757+0000 7f5362e00700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f535c0d16a0 0x7f535c160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.757+0000 7f5362e00700 1 --2- 172.21.15.12:0/2441650845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f535c0d2370 0x7f535c158e30 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.757+0000 7f5362e00700 1 -- 172.21.15.12:0/2441650845 >> 172.21.15.12:0/2441650845 conn(0x7f535c08aff0 msgr2=0x7f535c15bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.759+0000 7f5362e00700 1 -- 172.21.15.12:0/2441650845 shutdown_connections 2024-03-29T10:15:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.762+0000 7f5362e00700 1 -- 172.21.15.12:0/2441650845 wait complete. 2024-03-29T10:15:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 18 and data pool 19 2024-03-29T10:15:18.790 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:15:18.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.991+0000 7ff7b8c00700 1 Processor -- start 2024-03-29T10:15:18.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.991+0000 7ff7b8c00700 1 -- start start 2024-03-29T10:15:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.992+0000 7ff7b8c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7b4160be0 0x7ff7b40cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.992+0000 7ff7b8c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff7b40ce0e0 0x7ff7b40ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.992+0000 7ff7b8c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b4160fc0 con 0x7ff7b40cead0 2024-03-29T10:15:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.992+0000 7ff7b8c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b40b65d0 con 0x7ff7b4160be0 2024-03-29T10:15:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.992+0000 7ff7b8c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b40d2a80 con 0x7ff7b40ce0e0 2024-03-29T10:15:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.993+0000 7ff7b1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7b4160be0 0x7ff7b40cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.993+0000 7ff7b1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7b4160be0 0x7ff7b40cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49738/0 (socket says 172.21.15.12:49738) 2024-03-29T10:15:18.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.993+0000 7ff7b1e00700 1 -- 172.21.15.12:0/2255825691 learned_addr learned my addr 172.21.15.12:0/2255825691 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:18.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.993+0000 7ff7b1400700 1 --2- 172.21.15.12:0/2255825691 >> v2:172.21.15.159:3300/0 conn(0x7ff7b40ce0e0 0x7ff7b40ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.993+0000 7ff7b1e00700 1 -- 172.21.15.12:0/2255825691 >> v2:172.21.15.159:3300/0 conn(0x7ff7b40ce0e0 msgr2=0x7ff7b40ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:18.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.993+0000 7ff7b1e00700 1 --2- 172.21.15.12:0/2255825691 >> v2:172.21.15.159:3300/0 conn(0x7ff7b40ce0e0 0x7ff7b40ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.993+0000 7ff7b1e00700 1 -- 172.21.15.12:0/2255825691 >> v1:172.21.15.159:6789/0 conn(0x7ff7b40cead0 legacy=0x7ff7b4161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.993+0000 7ff7b1e00700 1 -- 172.21.15.12:0/2255825691 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b40d21e0 con 0x7ff7b4160be0 2024-03-29T10:15:18.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.993+0000 7ff7b1400700 1 --2- 172.21.15.12:0/2255825691 >> v2:172.21.15.159:3300/0 conn(0x7ff7b40ce0e0 0x7ff7b40ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.994+0000 7ff7b1e00700 1 --2- 172.21.15.12:0/2255825691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b4160be0 0x7ff7b40cdba0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40b9600 tx=0x7ff7a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449e2f2e47df7a47 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.994+0000 7ff7b0a00700 1 -- 172.21.15.12:0/2255825691 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a400b870 con 0x7ff7b4160be0 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.994+0000 7ff7b0a00700 1 -- 172.21.15.12:0/2255825691 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a400b9d0 con 0x7ff7b4160be0 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.994+0000 7ff7b0a00700 1 -- 172.21.15.12:0/2255825691 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a400bd60 con 0x7ff7b4160be0 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.994+0000 7ff7b8c00700 1 -- 172.21.15.12:0/2255825691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b4160be0 msgr2=0x7ff7b40cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.994+0000 7ff7b8c00700 1 --2- 172.21.15.12:0/2255825691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b4160be0 0x7ff7b40cdba0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40b9600 tx=0x7ff7a4009bd0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.994+0000 7ff7b8c00700 1 -- 172.21.15.12:0/2255825691 shutdown_connections 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.994+0000 7ff7b8c00700 1 --2- 172.21.15.12:0/2255825691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b4160be0 0x7ff7b40cdba0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.994+0000 7ff7b8c00700 1 --2- 172.21.15.12:0/2255825691 >> v2:172.21.15.159:3300/0 conn(0x7ff7b40ce0e0 0x7ff7b40ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.994+0000 7ff7b8c00700 1 -- 172.21.15.12:0/2255825691 >> 172.21.15.12:0/2255825691 conn(0x7ff7b408aff0 msgr2=0x7ff7b40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.995+0000 7ff7b8c00700 1 -- 172.21.15.12:0/2255825691 shutdown_connections 2024-03-29T10:15:18.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.995+0000 7ff7b8c00700 1 -- 172.21.15.12:0/2255825691 wait complete. 2024-03-29T10:15:18.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.997+0000 7ff7b8c00700 1 Processor -- start 2024-03-29T10:15:18.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.997+0000 7ff7b8c00700 1 -- start start 2024-03-29T10:15:18.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.998+0000 7ff7b8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b40ce0e0 0x7ff7b40c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.998+0000 7ff7b8c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7b40cead0 0x7ff7b4157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.998+0000 7ff7b8c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7b4160be0 0x7ff7b406caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:18.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.998+0000 7ff7b8c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b40cd4f0 con 0x7ff7b40ce0e0 2024-03-29T10:15:18.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.998+0000 7ff7b8c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b40c7190 con 0x7ff7b40cead0 2024-03-29T10:15:18.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.998+0000 7ff7b8c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b40c7470 con 0x7ff7b4160be0 2024-03-29T10:15:18.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b40ce0e0 0x7ff7b40c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b40ce0e0 0x7ff7b40c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49752/0 (socket says 172.21.15.12:49752) 2024-03-29T10:15:18.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1e00700 1 -- 172.21.15.12:0/3189902189 learned_addr learned my addr 172.21.15.12:0/3189902189 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:18.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1400700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7b40cead0 0x7ff7b4157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b2800700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7b4160be0 0x7ff7b406caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:18.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1e00700 1 -- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7b4160be0 msgr2=0x7ff7b406caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1e00700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7b4160be0 0x7ff7b406caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1e00700 1 -- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7b40cead0 msgr2=0x7ff7b4157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:18.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1e00700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7b40cead0 0x7ff7b4157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:18.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1e00700 1 -- 172.21.15.12:0/3189902189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a4009550 con 0x7ff7b40ce0e0 2024-03-29T10:15:18.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1400700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7b40cead0 0x7ff7b4157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:15:18.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b2800700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7b4160be0 0x7ff7b406caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:15:18.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:18.999+0000 7ff7b1e00700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b40ce0e0 0x7ff7b40c8230 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4000c00 tx=0x7ff7a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:18.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.000+0000 7ff7aa000700 1 -- 172.21.15.12:0/3189902189 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4004520 con 0x7ff7b40ce0e0 2024-03-29T10:15:18.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.000+0000 7ff7aa000700 1 -- 172.21.15.12:0/3189902189 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a401e070 con 0x7ff7b40ce0e0 2024-03-29T10:15:18.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.000+0000 7ff7aa000700 1 -- 172.21.15.12:0/3189902189 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a40038a0 con 0x7ff7b40ce0e0 2024-03-29T10:15:18.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.000+0000 7ff7b8c00700 1 -- 172.21.15.12:0/3189902189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b40c7750 con 0x7ff7b40ce0e0 2024-03-29T10:15:18.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.000+0000 7ff7b8c00700 1 -- 172.21.15.12:0/3189902189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b40c7d10 con 0x7ff7b40ce0e0 2024-03-29T10:15:18.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.001+0000 7ff7b8c00700 1 -- 172.21.15.12:0/3189902189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b4067720 con 0x7ff7b40ce0e0 2024-03-29T10:15:18.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.002+0000 7ff7aa000700 1 -- 172.21.15.12:0/3189902189 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff7a4003a00 con 0x7ff7b40ce0e0 2024-03-29T10:15:19.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.003+0000 7ff7aa000700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff79406adb0 0x7ff79406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.003+0000 7ff7aa000700 1 -- 172.21.15.12:0/3189902189 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7a4020070 con 0x7ff7b40ce0e0 2024-03-29T10:15:19.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.003+0000 7ff7b1400700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff79406adb0 0x7ff79406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.004+0000 7ff7b1400700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff79406adb0 0x7ff79406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7b415b480 tx=0x7ff7a0008040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:19.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.008+0000 7ff7aa000700 1 -- 172.21.15.12:0/3189902189 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff7a4055ba0 con 0x7ff7b40ce0e0 2024-03-29T10:15:19.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.225+0000 7ff7b8c00700 1 -- 172.21.15.12:0/3189902189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7b40b5cb0 con 0x7ff7b40ce0e0 2024-03-29T10:15:19.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.226+0000 7ff7aa000700 1 -- 172.21.15.12:0/3189902189 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+16124 (secure 0 0 0) 0x7ff7a402a090 con 0x7ff7b40ce0e0 2024-03-29T10:15:19.224 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:19.224 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":153,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:15:18.741838+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:15:17.110269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-29T10:15:17.717343+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:15:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 -- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff79406adb0 msgr2=0x7ff79406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff79406adb0 0x7ff79406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7b415b480 tx=0x7ff7a0008040 comp rx=0 tx=0).stop 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 -- 172.21.15.12:0/3189902189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b40ce0e0 msgr2=0x7ff7b40c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b40ce0e0 0x7ff7b40c8230 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4000c00 tx=0x7ff7a4004750 comp rx=0 tx=0).stop 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 -- 172.21.15.12:0/3189902189 shutdown_connections 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff79406adb0 0x7ff79406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7b4160be0 0x7ff7b406caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b40ce0e0 0x7ff7b40c8230 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 --2- 172.21.15.12:0/3189902189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7b40cead0 0x7ff7b4157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 -- 172.21.15.12:0/3189902189 >> 172.21.15.12:0/3189902189 conn(0x7ff7b408aff0 msgr2=0x7ff7b415f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 -- 172.21.15.12:0/3189902189 shutdown_connections 2024-03-29T10:15:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.229+0000 7ff7b8c00700 1 -- 172.21.15.12:0/3189902189 wait complete. 2024-03-29T10:15:19.245 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:15:19.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.446+0000 7f7f9bc00700 1 Processor -- start 2024-03-29T10:15:19.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.446+0000 7f7f9bc00700 1 -- start start 2024-03-29T10:15:19.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.447+0000 7f7f9bc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940d0a40 0x7f7f940b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.447+0000 7f7f9bc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f940d0ef0 con 0x7f7f940d05e0 2024-03-29T10:15:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.447+0000 7f7f9bc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f940d24e0 con 0x7f7f9415da50 2024-03-29T10:15:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.447+0000 7f7f9bc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f940b95d0 con 0x7f7f940d0a40 2024-03-29T10:15:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.447+0000 7f7f93e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7f940d05e0 0x7f7f940b6990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43610/0 (socket says 172.21.15.12:43610) 2024-03-29T10:15:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.447+0000 7f7f93e00700 1 -- 172.21.15.12:0/546820335 learned_addr learned my addr 172.21.15.12:0/546820335 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f99c00700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940d0a40 0x7f7f940b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828406623 0 0) 0x7f7f940d0ef0 con 0x7f7f940d05e0 2024-03-29T10:15:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f7c0035b0 con 0x7f7f940d05e0 2024-03-29T10:15:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4043834078 0 0) 0x7f7f7c0035b0 con 0x7f7f940d05e0 2024-03-29T10:15:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940d0a40 msgr2=0x7f7f940b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940d0a40 0x7f7f940b70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 >> v1:172.21.15.159:6789/0 conn(0x7f7f9415da50 legacy=0x7f7f940ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f940d27c0 con 0x7f7f940d05e0 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f7f80001670 con 0x7f7f940d05e0 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 >> v1:172.21.15.12:6789/0 conn(0x7f7f940d05e0 legacy=0x7f7f940b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940d0a40 0x7f7f7c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 0x7f7f7c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f7c004d90 0x7f7f7c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c00bb20 con 0x7f7f7c004000 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c00bc90 con 0x7f7f7c004d90 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.448+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c00be50 con 0x7f7f940d0a40 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.449+0000 7f7f93e00700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 0x7f7f7c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.449+0000 7f7f99200700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940d0a40 0x7f7f7c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.449+0000 7f7f99c00700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f7c004d90 0x7f7f7c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.449+0000 7f7f93e00700 1 -- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940d0a40 msgr2=0x7f7f7c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.449+0000 7f7f93e00700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940d0a40 0x7f7f7c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.449+0000 7f7f93e00700 1 -- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f7c004d90 msgr2=0x7f7f7c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.449+0000 7f7f93e00700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f7c004d90 0x7f7f7c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.449+0000 7f7f93e00700 1 -- 172.21.15.12:0/546820335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c0035b0 con 0x7f7f7c004000 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.449+0000 7f7f93e00700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 0x7f7f7c004850 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f7f9408ba30 tx=0x7f7f7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b718a16487d34df server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.450+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80007b70 con 0x7f7f7c004000 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.450+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8000cc30 con 0x7f7f7c004000 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.450+0000 7f7f93400700 1 -- 172.21.15.12:0/546820335 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80005960 con 0x7f7f7c004000 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.450+0000 7f7f9bc00700 1 -- 172.21.15.12:0/546820335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 msgr2=0x7f7f7c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.450+0000 7f7f9bc00700 1 --2- 172.21.15.12:0/546820335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 0x7f7f7c004850 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f7f9408ba30 tx=0x7f7f7c005880 comp rx=0 tx=0).stop 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.450+0000 7f7f99c00700 1 -- 172.21.15.12:0/546820335 reap_dead start 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.450+0000 7f7f9bc00700 1 -- 172.21.15.12:0/546820335 shutdown_connections 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.450+0000 7f7f9bc00700 1 -- 172.21.15.12:0/546820335 >> 172.21.15.12:0/546820335 conn(0x7f7f940666a0 msgr2=0x7f7f94066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.450+0000 7f7f9bc00700 1 -- 172.21.15.12:0/546820335 shutdown_connections 2024-03-29T10:15:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.450+0000 7f7f9bc00700 1 -- 172.21.15.12:0/546820335 wait complete. 2024-03-29T10:15:19.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.452+0000 7f7f9bc00700 1 Processor -- start 2024-03-29T10:15:19.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.452+0000 7f7f9bc00700 1 -- start start 2024-03-29T10:15:19.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.453+0000 7f7f9bc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 0x7f7f9415de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.453+0000 7f7f9bc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f940cd190 0x7f7f94146b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.453+0000 7f7f9bc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940b7280 0x7f7f940b7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.453+0000 7f7f9bc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f940b7c40 con 0x7f7f7c004000 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.453+0000 7f7f9bc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9415f020 con 0x7f7f940cd190 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.453+0000 7f7f9bc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9415e410 con 0x7f7f940b7280 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.453+0000 7f7f99200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 0x7f7f9415de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.453+0000 7f7f99200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 0x7f7f9415de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49778/0 (socket says 172.21.15.12:49778) 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.453+0000 7f7f99200700 1 -- 172.21.15.12:0/4030481230 learned_addr learned my addr 172.21.15.12:0/4030481230 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f99c00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940b7280 0x7f7f940b7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f93e00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f940cd190 0x7f7f94146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f99200700 1 -- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940b7280 msgr2=0x7f7f940b7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f99200700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940b7280 0x7f7f940b7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f99200700 1 -- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f940cd190 msgr2=0x7f7f94146b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f99200700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f940cd190 0x7f7f94146b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f99200700 1 -- 172.21.15.12:0/4030481230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9415e6f0 con 0x7f7f7c004000 2024-03-29T10:15:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f99c00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940b7280 0x7f7f940b7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f99200700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 0x7f7f9415de30 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f7f8800f820 tx=0x7f7f8800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f90c00700 1 -- 172.21.15.12:0/4030481230 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f880052e0 con 0x7f7f7c004000 2024-03-29T10:15:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f90c00700 1 -- 172.21.15.12:0/4030481230 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f88005440 con 0x7f7f7c004000 2024-03-29T10:15:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f9bc00700 1 -- 172.21.15.12:0/4030481230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9415e9d0 con 0x7f7f7c004000 2024-03-29T10:15:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.454+0000 7f7f90c00700 1 -- 172.21.15.12:0/4030481230 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f880038c0 con 0x7f7f7c004000 2024-03-29T10:15:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.455+0000 7f7f9bc00700 1 -- 172.21.15.12:0/4030481230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f94147050 con 0x7f7f7c004000 2024-03-29T10:15:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.455+0000 7f7f9bc00700 1 -- 172.21.15.12:0/4030481230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f94067720 con 0x7f7f7c004000 2024-03-29T10:15:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.456+0000 7f7f90c00700 1 -- 172.21.15.12:0/4030481230 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7f88003a20 con 0x7f7f7c004000 2024-03-29T10:15:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.457+0000 7f7f90c00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f7406ade0 0x7f7f7406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.457+0000 7f7f90c00700 1 -- 172.21.15.12:0/4030481230 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f88089340 con 0x7f7f7c004000 2024-03-29T10:15:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.458+0000 7f7f93e00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f7406ade0 0x7f7f7406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.459+0000 7f7f93e00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f7406ade0 0x7f7f7406d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7f940ccd60 tx=0x7f7f80007000 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:19.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.462+0000 7f7f90c00700 1 -- 172.21.15.12:0/4030481230 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f880542e0 con 0x7f7f7c004000 2024-03-29T10:15:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.676+0000 7f7f9bc00700 1 -- 172.21.15.12:0/4030481230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f941e7c00 con 0x7f7f7c004000 2024-03-29T10:15:19.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.678+0000 7f7f90c00700 1 -- 172.21.15.12:0/4030481230 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+16124 (secure 0 0 0) 0x7f7f88057900 con 0x7f7f7c004000 2024-03-29T10:15:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:19.675 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":153,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:15:18.741838+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:15:17.110269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-29T10:15:17.717343+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:15:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.680+0000 7f7f9bc00700 1 -- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f7406ade0 msgr2=0x7f7f7406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.680+0000 7f7f9bc00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f7406ade0 0x7f7f7406d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7f940ccd60 tx=0x7f7f80007000 comp rx=0 tx=0).stop 2024-03-29T10:15:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.680+0000 7f7f9bc00700 1 -- 172.21.15.12:0/4030481230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 msgr2=0x7f7f9415de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.680+0000 7f7f9bc00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 0x7f7f9415de30 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f7f8800f820 tx=0x7f7f8800fb30 comp rx=0 tx=0).stop 2024-03-29T10:15:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.680+0000 7f7f9bc00700 1 -- 172.21.15.12:0/4030481230 shutdown_connections 2024-03-29T10:15:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.680+0000 7f7f9bc00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7f7406ade0 0x7f7f7406d2a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.680+0000 7f7f9bc00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f940b7280 0x7f7f940b7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.680+0000 7f7f9bc00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f7c004000 0x7f7f9415de30 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.680+0000 7f7f9bc00700 1 --2- 172.21.15.12:0/4030481230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f940cd190 0x7f7f94146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.680+0000 7f7f9bc00700 1 -- 172.21.15.12:0/4030481230 >> 172.21.15.12:0/4030481230 conn(0x7f7f940666a0 msgr2=0x7f7f940c9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.681+0000 7f7f9bc00700 1 -- 172.21.15.12:0/4030481230 shutdown_connections 2024-03-29T10:15:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.681+0000 7f7f9bc00700 1 -- 172.21.15.12:0/4030481230 wait complete. 2024-03-29T10:15:19.697 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T10:15:19.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.904+0000 7f95c6e00700 1 Processor -- start 2024-03-29T10:15:19.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.905+0000 7f95c6e00700 1 -- start start 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95c6e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f95c015d9d0 0x7f95c00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95c6e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95c00d16a0 0x7f95c00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95c6e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95c00d2370 0x7f95c015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95c6e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c00d2820 con 0x7f95c00d2370 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95c6e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c0161420 con 0x7f95c015d9d0 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95c6e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c0161700 con 0x7f95c00d16a0 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95c4e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95c00d2370 0x7f95c015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95c4e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95c00d2370 0x7f95c015ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49790/0 (socket says 172.21.15.12:49790) 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95c4e00700 1 -- 172.21.15.12:0/1263293580 learned_addr learned my addr 172.21.15.12:0/1263293580 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95bfe00700 1 --2- 172.21.15.12:0/1263293580 >> v2:172.21.15.159:3300/0 conn(0x7f95c015d9d0 0x7f95c00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.906+0000 7f95bf400700 1 --2- 172.21.15.12:0/1263293580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95c00d16a0 0x7f95c00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.907+0000 7f95c4e00700 1 -- 172.21.15.12:0/1263293580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95c00d16a0 msgr2=0x7f95c00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.907+0000 7f95c4e00700 1 --2- 172.21.15.12:0/1263293580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95c00d16a0 0x7f95c00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.907+0000 7f95c4e00700 1 -- 172.21.15.12:0/1263293580 >> v2:172.21.15.159:3300/0 conn(0x7f95c015d9d0 msgr2=0x7f95c00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.907+0000 7f95c4e00700 1 --2- 172.21.15.12:0/1263293580 >> v2:172.21.15.159:3300/0 conn(0x7f95c015d9d0 0x7f95c00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.907+0000 7f95c4e00700 1 -- 172.21.15.12:0/1263293580 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c01619e0 con 0x7f95c00d2370 2024-03-29T10:15:19.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.907+0000 7f95c4e00700 1 --2- 172.21.15.12:0/1263293580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d2370 0x7f95c015ddb0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f95c00d0ff0 tx=0x7f95b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453347053e2b6406 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:19.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.907+0000 7f95bea00700 1 -- 172.21.15.12:0/1263293580 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b800e070 con 0x7f95c00d2370 2024-03-29T10:15:19.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.907+0000 7f95bea00700 1 -- 172.21.15.12:0/1263293580 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b800b9c0 con 0x7f95c00d2370 2024-03-29T10:15:19.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.907+0000 7f95bea00700 1 -- 172.21.15.12:0/1263293580 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b800bd70 con 0x7f95c00d2370 2024-03-29T10:15:19.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.908+0000 7f95c6e00700 1 -- 172.21.15.12:0/1263293580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d2370 msgr2=0x7f95c015ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.908+0000 7f95c6e00700 1 --2- 172.21.15.12:0/1263293580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d2370 0x7f95c015ddb0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f95c00d0ff0 tx=0x7f95b800b330 comp rx=0 tx=0).stop 2024-03-29T10:15:19.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.908+0000 7f95c6e00700 1 -- 172.21.15.12:0/1263293580 shutdown_connections 2024-03-29T10:15:19.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.908+0000 7f95c6e00700 1 --2- 172.21.15.12:0/1263293580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c00d2370 0x7f95c015ddb0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.908+0000 7f95c6e00700 1 --2- 172.21.15.12:0/1263293580 >> v2:172.21.15.159:3300/0 conn(0x7f95c015d9d0 0x7f95c00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.908+0000 7f95c6e00700 1 --2- 172.21.15.12:0/1263293580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95c00d16a0 0x7f95c00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.908+0000 7f95c6e00700 1 -- 172.21.15.12:0/1263293580 >> 172.21.15.12:0/1263293580 conn(0x7f95c008aff0 msgr2=0x7f95c00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:19.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.908+0000 7f95c6e00700 1 -- 172.21.15.12:0/1263293580 shutdown_connections 2024-03-29T10:15:19.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.908+0000 7f95c6e00700 1 -- 172.21.15.12:0/1263293580 wait complete. 2024-03-29T10:15:19.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.910+0000 7f95c6e00700 1 Processor -- start 2024-03-29T10:15:19.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.910+0000 7f95c6e00700 1 -- start start 2024-03-29T10:15:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c6e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95c00d16a0 0x7f95c00b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c6e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95c00d2370 0x7f95c01f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c6e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c015d9d0 0x7f95c00cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c6e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00b8140 con 0x7f95c015d9d0 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c6e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c01f3ca0 con 0x7f95c00d16a0 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c6e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c01f3f80 con 0x7f95c00d2370 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c015d9d0 0x7f95c00cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c015d9d0 0x7f95c00cb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49804/0 (socket says 172.21.15.12:49804) 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c4e00700 1 -- 172.21.15.12:0/1546076122 learned_addr learned my addr 172.21.15.12:0/1546076122 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95bfe00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95c00d16a0 0x7f95c00b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95bf400700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95c00d2370 0x7f95c01f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c4e00700 1 -- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95c00d2370 msgr2=0x7f95c01f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c4e00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95c00d2370 0x7f95c01f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c4e00700 1 -- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95c00d16a0 msgr2=0x7f95c00b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c4e00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95c00d16a0 0x7f95c00b7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95c4e00700 1 -- 172.21.15.12:0/1546076122 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b8009b30 con 0x7f95c015d9d0 2024-03-29T10:15:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.911+0000 7f95bfe00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95c00d16a0 0x7f95c00b7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.912+0000 7f95c4e00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c015d9d0 0x7f95c00cb640 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f95c008c230 tx=0x7f95b8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.912+0000 7f95b7e00700 1 -- 172.21.15.12:0/1546076122 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8004250 con 0x7f95c015d9d0 2024-03-29T10:15:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.912+0000 7f95b7e00700 1 -- 172.21.15.12:0/1546076122 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b80043d0 con 0x7f95c015d9d0 2024-03-29T10:15:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.912+0000 7f95b7e00700 1 -- 172.21.15.12:0/1546076122 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8004780 con 0x7f95c015d9d0 2024-03-29T10:15:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.912+0000 7f95c6e00700 1 -- 172.21.15.12:0/1546076122 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c00b61a0 con 0x7f95c015d9d0 2024-03-29T10:15:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.912+0000 7f95c6e00700 1 -- 172.21.15.12:0/1546076122 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c00cd230 con 0x7f95c015d9d0 2024-03-29T10:15:19.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.913+0000 7f95c6e00700 1 -- 172.21.15.12:0/1546076122 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c00d1ca0 con 0x7f95c015d9d0 2024-03-29T10:15:19.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.914+0000 7f95b7e00700 1 -- 172.21.15.12:0/1546076122 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f95b801c440 con 0x7f95c015d9d0 2024-03-29T10:15:19.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.915+0000 7f95b7e00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95a006adb0 0x7f95a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:19.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.915+0000 7f95b7e00700 1 -- 172.21.15.12:0/1546076122 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95b808ab80 con 0x7f95c015d9d0 2024-03-29T10:15:19.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.915+0000 7f95bfe00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95a006adb0 0x7f95a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:19.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.916+0000 7f95bfe00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95a006adb0 0x7f95a006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f95b0007a50 tx=0x7f95b0008040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:19.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:19.919+0000 7f95b7e00700 1 -- 172.21.15.12:0/1546076122 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f95b8055c50 con 0x7f95c015d9d0 2024-03-29T10:15:20.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.188+0000 7f95c6e00700 1 -- 172.21.15.12:0/1546076122 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f95c00b66c0 con 0x7f95c015d9d0 2024-03-29T10:15:20.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.766+0000 7f95b7e00700 1 -- 172.21.15.12:0/1546076122 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7f95c00b66c0 con 0x7f95c015d9d0 2024-03-29T10:15:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 -- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95a006adb0 msgr2=0x7f95a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95a006adb0 0x7f95a006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f95b0007a50 tx=0x7f95b0008040 comp rx=0 tx=0).stop 2024-03-29T10:15:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 -- 172.21.15.12:0/1546076122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c015d9d0 msgr2=0x7f95c00cb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c015d9d0 0x7f95c00cb640 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f95c008c230 tx=0x7f95b8004ad0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 -- 172.21.15.12:0/1546076122 shutdown_connections 2024-03-29T10:15:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f95a006adb0 0x7f95a006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95c015d9d0 0x7f95c00cb640 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95c00d16a0 0x7f95c00b7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 --2- 172.21.15.12:0/1546076122 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95c00d2370 0x7f95c01f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 -- 172.21.15.12:0/1546076122 >> 172.21.15.12:0/1546076122 conn(0x7f95c008aff0 msgr2=0x7f95c00ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:20.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.768+0000 7f95c6e00700 1 -- 172.21.15.12:0/1546076122 shutdown_connections 2024-03-29T10:15:20.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.769+0000 7f95c6e00700 1 -- 172.21.15.12:0/1546076122 wait complete. 2024-03-29T10:15:20.786 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T10:15:20.786 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T10:15:20.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.992+0000 7f276d800700 1 Processor -- start 2024-03-29T10:15:20.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.992+0000 7f276d800700 1 -- start start 2024-03-29T10:15:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.993+0000 7f276d800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f27681611b0 0x7f276815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.993+0000 7f276d800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f276815f4d0 0x7f27680ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.993+0000 7f276d800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27680ce270 con 0x7f27680d21f0 2024-03-29T10:15:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.993+0000 7f276d800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f276815b600 con 0x7f27681611b0 2024-03-29T10:15:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.993+0000 7f276d800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27680d2760 con 0x7f276815f4d0 2024-03-29T10:15:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.993+0000 7f2766a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f27681611b0 0x7f276815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.993+0000 7f2766a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f27681611b0 0x7f276815ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49812/0 (socket says 172.21.15.12:49812) 2024-03-29T10:15:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.993+0000 7f2766a00700 1 -- 172.21.15.12:0/1409912982 learned_addr learned my addr 172.21.15.12:0/1409912982 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.993+0000 7f2766000700 1 --2- 172.21.15.12:0/1409912982 >> v2:172.21.15.159:3300/0 conn(0x7f276815f4d0 0x7f27680ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.994+0000 7f2766a00700 1 -- 172.21.15.12:0/1409912982 >> v2:172.21.15.159:3300/0 conn(0x7f276815f4d0 msgr2=0x7f27680ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:20.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.994+0000 7f2766a00700 1 --2- 172.21.15.12:0/1409912982 >> v2:172.21.15.159:3300/0 conn(0x7f276815f4d0 0x7f27680ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.994+0000 7f2766a00700 1 -- 172.21.15.12:0/1409912982 >> v1:172.21.15.159:6789/0 conn(0x7f27680d21f0 legacy=0x7f2768161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:20.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.994+0000 7f2766a00700 1 -- 172.21.15.12:0/1409912982 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27680d2a40 con 0x7f27681611b0 2024-03-29T10:15:20.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.994+0000 7f2766a00700 1 --2- 172.21.15.12:0/1409912982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27681611b0 0x7f276815ef90 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f276808c030 tx=0x7f2758009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ecbfbc5a8ce2f03 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:20.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.994+0000 7f2765600700 1 -- 172.21.15.12:0/1409912982 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2758011830 con 0x7f27681611b0 2024-03-29T10:15:20.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.994+0000 7f2765600700 1 -- 172.21.15.12:0/1409912982 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2758011990 con 0x7f27681611b0 2024-03-29T10:15:20.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.994+0000 7f2765600700 1 -- 172.21.15.12:0/1409912982 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2758011d20 con 0x7f27681611b0 2024-03-29T10:15:20.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.995+0000 7f276d800700 1 -- 172.21.15.12:0/1409912982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27681611b0 msgr2=0x7f276815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:20.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.995+0000 7f276d800700 1 --2- 172.21.15.12:0/1409912982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27681611b0 0x7f276815ef90 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f276808c030 tx=0x7f2758009bd0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.995+0000 7f276d800700 1 -- 172.21.15.12:0/1409912982 shutdown_connections 2024-03-29T10:15:20.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.995+0000 7f276d800700 1 --2- 172.21.15.12:0/1409912982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27681611b0 0x7f276815ef90 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.995+0000 7f276d800700 1 --2- 172.21.15.12:0/1409912982 >> v2:172.21.15.159:3300/0 conn(0x7f276815f4d0 0x7f27680ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.995+0000 7f276d800700 1 -- 172.21.15.12:0/1409912982 >> 172.21.15.12:0/1409912982 conn(0x7f276808aff0 msgr2=0x7f27680d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:20.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.995+0000 7f276d800700 1 -- 172.21.15.12:0/1409912982 shutdown_connections 2024-03-29T10:15:20.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.995+0000 7f276d800700 1 -- 172.21.15.12:0/1409912982 wait complete. 2024-03-29T10:15:20.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.997+0000 7f276d800700 1 Processor -- start 2024-03-29T10:15:20.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.997+0000 7f276d800700 1 -- start start 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f276d800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27680d21f0 0x7f276814d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f276d800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f276815f4d0 0x7f27680b8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f276d800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27681611b0 0x7f27680c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f276d800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276814df20 con 0x7f276815f4d0 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f276d800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27680cabc0 con 0x7f27681611b0 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f276d800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27680b8ed0 con 0x7f27680d21f0 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f2766000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f276815f4d0 0x7f27680b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f2766000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f276815f4d0 0x7f27680b8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49826/0 (socket says 172.21.15.12:49826) 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f2766000700 1 -- 172.21.15.12:0/3644258822 learned_addr learned my addr 172.21.15.12:0/3644258822 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f2766a00700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27680d21f0 0x7f276814d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.998+0000 7f2767400700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27681611b0 0x7f27680c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:20.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.999+0000 7f2766000700 1 -- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27680d21f0 msgr2=0x7f276814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:20.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.999+0000 7f2766000700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27680d21f0 0x7f276814d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.999+0000 7f2766000700 1 -- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27681611b0 msgr2=0x7f27680c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:20.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.999+0000 7f2766000700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27681611b0 0x7f27680c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:20.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.999+0000 7f2766000700 1 -- 172.21.15.12:0/3644258822 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2758009550 con 0x7f276815f4d0 2024-03-29T10:15:20.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.999+0000 7f2766000700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f276815f4d0 0x7f27680b8080 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f275400e9b0 tx=0x7f275400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:20.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.999+0000 7f275ea00700 1 -- 172.21.15.12:0/3644258822 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275400ee50 con 0x7f276815f4d0 2024-03-29T10:15:20.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.999+0000 7f275ea00700 1 -- 172.21.15.12:0/3644258822 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275400c7f0 con 0x7f276815f4d0 2024-03-29T10:15:20.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.999+0000 7f275ea00700 1 -- 172.21.15.12:0/3644258822 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275400cb60 con 0x7f276815f4d0 2024-03-29T10:15:20.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:20.999+0000 7f276d800700 1 -- 172.21.15.12:0/3644258822 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27680b85c0 con 0x7f276815f4d0 2024-03-29T10:15:20.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:21.000+0000 7f276d800700 1 -- 172.21.15.12:0/3644258822 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27680c81c0 con 0x7f276815f4d0 2024-03-29T10:15:20.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:21.001+0000 7f276d800700 1 -- 172.21.15.12:0/3644258822 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2768067720 con 0x7f276815f4d0 2024-03-29T10:15:20.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:21.002+0000 7f275ea00700 1 -- 172.21.15.12:0/3644258822 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f275400ccc0 con 0x7f276815f4d0 2024-03-29T10:15:21.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:21.002+0000 7f275ea00700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f274806ade0 0x7f274806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:21.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:21.003+0000 7f275ea00700 1 -- 172.21.15.12:0/3644258822 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27540892f0 con 0x7f276815f4d0 2024-03-29T10:15:21.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:21.003+0000 7f2766a00700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f274806ade0 0x7f274806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:21.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:21.004+0000 7f2766a00700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f274806ade0 0x7f274806d2a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2758014040 tx=0x7f2758008030 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:21.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:21.007+0000 7f275ea00700 1 -- 172.21.15.12:0/3644258822 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2754054450 con 0x7f276815f4d0 2024-03-29T10:15:21.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:21.271+0000 7f276d800700 1 -- 172.21.15.12:0/3644258822 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f27680b5d00 con 0x7f276815f4d0 2024-03-29T10:15:22.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.236+0000 7f275ea00700 1 -- 172.21.15.12:0/3644258822 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) v1 ==== 122+0+0 (secure 0 0 0) 0x7f275405ce60 con 0x7f276815f4d0 2024-03-29T10:15:22.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.239+0000 7f276d800700 1 -- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f274806ade0 msgr2=0x7f274806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.239+0000 7f276d800700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f274806ade0 0x7f274806d2a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2758014040 tx=0x7f2758008030 comp rx=0 tx=0).stop 2024-03-29T10:15:22.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.239+0000 7f276d800700 1 -- 172.21.15.12:0/3644258822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f276815f4d0 msgr2=0x7f27680b8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.239+0000 7f276d800700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f276815f4d0 0x7f27680b8080 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f275400e9b0 tx=0x7f275400ecc0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.239+0000 7f276d800700 1 -- 172.21.15.12:0/3644258822 shutdown_connections 2024-03-29T10:15:22.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.240+0000 7f276d800700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f274806ade0 0x7f274806d2a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.240+0000 7f276d800700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27681611b0 0x7f27680c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.240+0000 7f276d800700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27680d21f0 0x7f276814d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.240+0000 7f276d800700 1 --2- 172.21.15.12:0/3644258822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f276815f4d0 0x7f27680b8080 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.240+0000 7f276d800700 1 -- 172.21.15.12:0/3644258822 >> 172.21.15.12:0/3644258822 conn(0x7f276808aff0 msgr2=0x7f27680ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:22.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.240+0000 7f276d800700 1 -- 172.21.15.12:0/3644258822 shutdown_connections 2024-03-29T10:15:22.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.240+0000 7f276d800700 1 -- 172.21.15.12:0/3644258822 wait complete. 2024-03-29T10:15:22.258 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T10:15:22.258 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:15:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.464+0000 7fd77f600700 1 Processor -- start 2024-03-29T10:15:22.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.464+0000 7fd77f600700 1 -- start start 2024-03-29T10:15:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.465+0000 7fd77f600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd77815d640 0x7fd7780d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.465+0000 7fd77f600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ceec0 0x7fd7781612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.465+0000 7fd77f600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd778161800 0x7fd7780cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.465+0000 7fd77f600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd778161c90 con 0x7fd778161800 2024-03-29T10:15:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.465+0000 7fd77f600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7780d30d0 con 0x7fd77815d640 2024-03-29T10:15:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.465+0000 7fd77f600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780b6840 con 0x7fd7780ceec0 2024-03-29T10:15:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd77d600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd778161800 0x7fd7780cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd77d600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd778161800 0x7fd7780cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49828/0 (socket says 172.21.15.12:49828) 2024-03-29T10:15:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd77d600700 1 -- 172.21.15.12:0/1714288832 learned_addr learned my addr 172.21.15.12:0/1714288832 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:22.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd777e00700 1 --2- 172.21.15.12:0/1714288832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ceec0 0x7fd7781612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd77cc00700 1 --2- 172.21.15.12:0/1714288832 >> v2:172.21.15.159:3300/0 conn(0x7fd77815d640 0x7fd7780d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd77d600700 1 -- 172.21.15.12:0/1714288832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ceec0 msgr2=0x7fd7781612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd77d600700 1 --2- 172.21.15.12:0/1714288832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ceec0 0x7fd7781612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd77d600700 1 -- 172.21.15.12:0/1714288832 >> v2:172.21.15.159:3300/0 conn(0x7fd77815d640 msgr2=0x7fd7780d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd77d600700 1 --2- 172.21.15.12:0/1714288832 >> v2:172.21.15.159:3300/0 conn(0x7fd77815d640 0x7fd7780d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd77d600700 1 -- 172.21.15.12:0/1714288832 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7780d2880 con 0x7fd778161800 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd777e00700 1 --2- 172.21.15.12:0/1714288832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ceec0 0x7fd7781612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.466+0000 7fd77d600700 1 --2- 172.21.15.12:0/1714288832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd778161800 0x7fd7780cdba0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fd7780b5c20 tx=0x7fd77000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f824e70f9ceb89a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd777400700 1 -- 172.21.15.12:0/1714288832 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77000e070 con 0x7fd778161800 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd777400700 1 -- 172.21.15.12:0/1714288832 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77000b9c0 con 0x7fd778161800 2024-03-29T10:15:22.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd777400700 1 -- 172.21.15.12:0/1714288832 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77000bd70 con 0x7fd778161800 2024-03-29T10:15:22.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd77f600700 1 -- 172.21.15.12:0/1714288832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd778161800 msgr2=0x7fd7780cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd77f600700 1 --2- 172.21.15.12:0/1714288832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd778161800 0x7fd7780cdba0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fd7780b5c20 tx=0x7fd77000b330 comp rx=0 tx=0).stop 2024-03-29T10:15:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd77f600700 1 -- 172.21.15.12:0/1714288832 shutdown_connections 2024-03-29T10:15:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd77f600700 1 --2- 172.21.15.12:0/1714288832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd778161800 0x7fd7780cdba0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd77f600700 1 --2- 172.21.15.12:0/1714288832 >> v2:172.21.15.159:3300/0 conn(0x7fd77815d640 0x7fd7780d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd77f600700 1 --2- 172.21.15.12:0/1714288832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7780ceec0 0x7fd7781612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd77f600700 1 -- 172.21.15.12:0/1714288832 >> 172.21.15.12:0/1714288832 conn(0x7fd77808aff0 msgr2=0x7fd7780d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd77f600700 1 -- 172.21.15.12:0/1714288832 shutdown_connections 2024-03-29T10:15:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.467+0000 7fd77f600700 1 -- 172.21.15.12:0/1714288832 wait complete. 2024-03-29T10:15:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.469+0000 7fd77f600700 1 Processor -- start 2024-03-29T10:15:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.469+0000 7fd77f600700 1 -- start start 2024-03-29T10:15:22.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.470+0000 7fd77f600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780ceec0 0x7fd778155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.470+0000 7fd77f600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77815d640 0x7fd7781f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.470+0000 7fd77f600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd778161800 0x7fd7781f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.470+0000 7fd77f600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd778160760 con 0x7fd7780ceec0 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.470+0000 7fd77f600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77815fae0 con 0x7fd778161800 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.470+0000 7fd77f600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77815fdc0 con 0x7fd77815d640 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.470+0000 7fd77cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780ceec0 0x7fd778155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.470+0000 7fd77cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780ceec0 0x7fd778155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49844/0 (socket says 172.21.15.12:49844) 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.470+0000 7fd77cc00700 1 -- 172.21.15.12:0/3565322526 learned_addr learned my addr 172.21.15.12:0/3565322526 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd777e00700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77815d640 0x7fd7781f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd77d600700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd778161800 0x7fd7781f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd77cc00700 1 -- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77815d640 msgr2=0x7fd7781f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd77cc00700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77815d640 0x7fd7781f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd77cc00700 1 -- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd778161800 msgr2=0x7fd7781f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd77cc00700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd778161800 0x7fd7781f2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd77cc00700 1 -- 172.21.15.12:0/3565322526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd770009b30 con 0x7fd7780ceec0 2024-03-29T10:15:22.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd77d600700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd778161800 0x7fd7781f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:22.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd77cc00700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780ceec0 0x7fd778155d10 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00e810 tx=0x7fd76c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:22.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd774c00700 1 -- 172.21.15.12:0/3565322526 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c013610 con 0x7fd7780ceec0 2024-03-29T10:15:22.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd774c00700 1 -- 172.21.15.12:0/3565322526 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76c013770 con 0x7fd7780ceec0 2024-03-29T10:15:22.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.471+0000 7fd77f600700 1 -- 172.21.15.12:0/3565322526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7781600a0 con 0x7fd7780ceec0 2024-03-29T10:15:22.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.472+0000 7fd774c00700 1 -- 172.21.15.12:0/3565322526 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c013b20 con 0x7fd7780ceec0 2024-03-29T10:15:22.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.472+0000 7fd77f600700 1 -- 172.21.15.12:0/3565322526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd778156250 con 0x7fd7780ceec0 2024-03-29T10:15:22.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.473+0000 7fd77f600700 1 -- 172.21.15.12:0/3565322526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7780c7b60 con 0x7fd7780ceec0 2024-03-29T10:15:22.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.474+0000 7fd774c00700 1 -- 172.21.15.12:0/3565322526 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd76c013c80 con 0x7fd7780ceec0 2024-03-29T10:15:22.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.474+0000 7fd774c00700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.475+0000 7fd774c00700 1 -- 172.21.15.12:0/3565322526 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd76c0891d0 con 0x7fd7780ceec0 2024-03-29T10:15:22.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.475+0000 7fd777e00700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.476+0000 7fd777e00700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 0x7fd75806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd7781f6520 tx=0x7fd764008040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:22.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.479+0000 7fd774c00700 1 -- 172.21.15.12:0/3565322526 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd76c054100 con 0x7fd7780ceec0 2024-03-29T10:15:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.742+0000 7fd77f600700 1 -- 172.21.15.12:0/3565322526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd778067720 con 0x7fd7780ceec0 2024-03-29T10:15:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.743+0000 7fd774c00700 1 -- 172.21.15.12:0/3565322526 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7fd76c057720 con 0x7fd7780ceec0 2024-03-29T10:15:22.740 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:22.740 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3149931506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3149931506},{"type":"v1","addr":"172.21.15.159:6835","nonce":3149931506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/967685480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":967685480},{"type":"v1","addr":"172.21.15.159:6837","nonce":967685480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8967,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4273394318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4273394318},{"type":"v1","addr":"172.21.15.12:6834","nonce":4273394318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:15:18.742186+0000","modified":"2024-03-29T10:15:22.226050+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8970},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8970":{"gid":8970,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/512120304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":512120304},{"type":"v1","addr":"172.21.15.12:6835","nonce":512120304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-29T10:15:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 -- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 msgr2=0x7fd75806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 0x7fd75806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd7781f6520 tx=0x7fd764008040 comp rx=0 tx=0).stop 2024-03-29T10:15:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 -- 172.21.15.12:0/3565322526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780ceec0 msgr2=0x7fd778155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780ceec0 0x7fd778155d10 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00e810 tx=0x7fd76c00eb20 comp rx=0 tx=0).stop 2024-03-29T10:15:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 -- 172.21.15.12:0/3565322526 shutdown_connections 2024-03-29T10:15:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd778161800 0x7fd7781f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7780ceec0 0x7fd778155d10 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 --2- 172.21.15.12:0/3565322526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77815d640 0x7fd7781f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 -- 172.21.15.12:0/3565322526 >> 172.21.15.12:0/3565322526 conn(0x7fd77808aff0 msgr2=0x7fd7780d0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 -- 172.21.15.12:0/3565322526 shutdown_connections 2024-03-29T10:15:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.745+0000 7fd77f600700 1 -- 172.21.15.12:0/3565322526 wait complete. 2024-03-29T10:15:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 73 2024-03-29T10:15:22.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:15:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.961+0000 7fe9c5c00700 1 Processor -- start 2024-03-29T10:15:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.962+0000 7fe9c5c00700 1 -- start start 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.963+0000 7fe9c5c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9c00acda0 0x7fe9c015e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.963+0000 7fe9c5c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9c00ad180 0x7fe9c00b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.963+0000 7fe9c5c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c015eb80 con 0x7fe9c00b4690 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.963+0000 7fe9c5c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c006eef0 con 0x7fe9c00ad180 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.963+0000 7fe9c5c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00ae7b0 con 0x7fe9c00acda0 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.963+0000 7fe9be000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9c00ad180 0x7fe9c00b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.963+0000 7fe9be000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9c00ad180 0x7fe9c00b4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49846/0 (socket says 172.21.15.12:49846) 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.963+0000 7fe9be000700 1 -- 172.21.15.12:0/4190177688 learned_addr learned my addr 172.21.15.12:0/4190177688 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.963+0000 7fe9bea00700 1 --2- 172.21.15.12:0/4190177688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9c00acda0 0x7fe9c015e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.964+0000 7fe9be000700 1 -- 172.21.15.12:0/4190177688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9c00acda0 msgr2=0x7fe9c015e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.964+0000 7fe9be000700 1 --2- 172.21.15.12:0/4190177688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9c00acda0 0x7fe9c015e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.964+0000 7fe9be000700 1 -- 172.21.15.12:0/4190177688 >> v1:172.21.15.159:6789/0 conn(0x7fe9c00b4690 legacy=0x7fe9c00b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.964+0000 7fe9be000700 1 -- 172.21.15.12:0/4190177688 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c00aea90 con 0x7fe9c00ad180 2024-03-29T10:15:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.964+0000 7fe9be000700 1 --2- 172.21.15.12:0/4190177688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00ad180 0x7fe9c00b4150 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0098f0 tx=0x7fe9ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244debf05e42abe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.964+0000 7fe9bd600700 1 -- 172.21.15.12:0/4190177688 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac011830 con 0x7fe9c00ad180 2024-03-29T10:15:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.964+0000 7fe9bd600700 1 -- 172.21.15.12:0/4190177688 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac011990 con 0x7fe9c00ad180 2024-03-29T10:15:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.964+0000 7fe9bd600700 1 -- 172.21.15.12:0/4190177688 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac011d20 con 0x7fe9c00ad180 2024-03-29T10:15:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.965+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4190177688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00ad180 msgr2=0x7fe9c00b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.965+0000 7fe9c5c00700 1 --2- 172.21.15.12:0/4190177688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00ad180 0x7fe9c00b4150 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0098f0 tx=0x7fe9ac009c00 comp rx=0 tx=0).stop 2024-03-29T10:15:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.965+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4190177688 shutdown_connections 2024-03-29T10:15:22.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.965+0000 7fe9c5c00700 1 --2- 172.21.15.12:0/4190177688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9c00acda0 0x7fe9c015e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.965+0000 7fe9c5c00700 1 --2- 172.21.15.12:0/4190177688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00ad180 0x7fe9c00b4150 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.965+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4190177688 >> 172.21.15.12:0/4190177688 conn(0x7fe9c0072170 msgr2=0x7fe9c006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:22.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.965+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4190177688 shutdown_connections 2024-03-29T10:15:22.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.965+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4190177688 wait complete. 2024-03-29T10:15:22.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.967+0000 7fe9c5c00700 1 Processor -- start 2024-03-29T10:15:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.968+0000 7fe9c5c00700 1 -- start start 2024-03-29T10:15:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9c5c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9c00acda0 0x7fe9c014cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9c5c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9c00ad180 0x7fe9c0157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9c5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00b4690 0x7fe9c0160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9c5c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00ccf50 con 0x7fe9c00b4690 2024-03-29T10:15:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9c5c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0158870 con 0x7fe9c00acda0 2024-03-29T10:15:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9c5c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0158b50 con 0x7fe9c00ad180 2024-03-29T10:15:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00b4690 0x7fe9c0160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00b4690 0x7fe9c0160500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49862/0 (socket says 172.21.15.12:49862) 2024-03-29T10:15:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9bf400700 1 -- 172.21.15.12:0/4093091507 learned_addr learned my addr 172.21.15.12:0/4093091507 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9bea00700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9c00acda0 0x7fe9c014cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.969+0000 7fe9be000700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9c00ad180 0x7fe9c0157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9bf400700 1 -- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9c00ad180 msgr2=0x7fe9c0157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9bf400700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9c00ad180 0x7fe9c0157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9bf400700 1 -- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9c00acda0 msgr2=0x7fe9c014cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9bf400700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9c00acda0 0x7fe9c014cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9bf400700 1 -- 172.21.15.12:0/4093091507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac009550 con 0x7fe9c00b4690 2024-03-29T10:15:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9bf400700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00b4690 0x7fe9c0160500 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800cc50 tx=0x7fe9b800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9b6a00700 1 -- 172.21.15.12:0/4093091507 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8003c00 con 0x7fe9c00b4690 2024-03-29T10:15:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9b6a00700 1 -- 172.21.15.12:0/4093091507 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b8003d60 con 0x7fe9c00b4690 2024-03-29T10:15:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4093091507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c0158e30 con 0x7fe9c00b4690 2024-03-29T10:15:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9b6a00700 1 -- 172.21.15.12:0/4093091507 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b800aab0 con 0x7fe9c00b4690 2024-03-29T10:15:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.970+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4093091507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c01584b0 con 0x7fe9c00b4690 2024-03-29T10:15:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.971+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4093091507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c0158640 con 0x7fe9c00b4690 2024-03-29T10:15:22.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.972+0000 7fe9b6a00700 1 -- 172.21.15.12:0/4093091507 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe9b800ac10 con 0x7fe9c00b4690 2024-03-29T10:15:22.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.973+0000 7fe9b6a00700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9a006adb0 0x7fe9a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:22.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.973+0000 7fe9b6a00700 1 -- 172.21.15.12:0/4093091507 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9b8089510 con 0x7fe9c00b4690 2024-03-29T10:15:22.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.974+0000 7fe9bea00700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9a006adb0 0x7fe9a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:22.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.974+0000 7fe9bea00700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9a006adb0 0x7fe9a006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009b40 tx=0x7fe9b0009210 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:22.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:22.978+0000 7fe9b6a00700 1 -- 172.21.15.12:0/4093091507 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe9b8054540 con 0x7fe9c00b4690 2024-03-29T10:15:23.174 INFO:tasks.ceph.mds.c.smithi012.stderr:2024-03-29T10:15:23.176+0000 7fa9ff200700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T10:15:23.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.192+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4093091507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9c00cd400 con 0x7fe9c00b4690 2024-03-29T10:15:23.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.193+0000 7fe9b6a00700 1 -- 172.21.15.12:0/4093091507 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+16111 (secure 0 0 0) 0x7fe9b8057b60 con 0x7fe9c00b4690 2024-03-29T10:15:23.191 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:23.191 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":154,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T10:15:19.744212+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:15:17.110269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-29T10:15:17.717343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:15:23.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9a006adb0 msgr2=0x7fe9a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9a006adb0 0x7fe9a006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009b40 tx=0x7fe9b0009210 comp rx=0 tx=0).stop 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4093091507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00b4690 msgr2=0x7fe9c0160500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00b4690 0x7fe9c0160500 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800cc50 tx=0x7fe9b800cf60 comp rx=0 tx=0).stop 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4093091507 shutdown_connections 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9a006adb0 0x7fe9a006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00b4690 0x7fe9c0160500 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9c00acda0 0x7fe9c014cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 --2- 172.21.15.12:0/4093091507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9c00ad180 0x7fe9c0157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4093091507 >> 172.21.15.12:0/4093091507 conn(0x7fe9c0072170 msgr2=0x7fe9c00b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4093091507 shutdown_connections 2024-03-29T10:15:23.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.196+0000 7fe9c5c00700 1 -- 172.21.15.12:0/4093091507 wait complete. 2024-03-29T10:15:23.213 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T10:15:23.213 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:15:23.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.414+0000 7fa7b9a00700 1 Processor -- start 2024-03-29T10:15:23.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.414+0000 7fa7b9a00700 1 -- start start 2024-03-29T10:15:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.415+0000 7fa7b9a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7b415d760 0x7fa7b40d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.415+0000 7fa7b9a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa7b40cf3c0 0x7fa7b40cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.415+0000 7fa7b9a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b40d1330 con 0x7fa7b415db40 2024-03-29T10:15:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.415+0000 7fa7b9a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b40d1fb0 con 0x7fa7b415d760 2024-03-29T10:15:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.415+0000 7fa7b9a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b40d1610 con 0x7fa7b40cf3c0 2024-03-29T10:15:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.415+0000 7fa7b2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7b415d760 0x7fa7b40d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.415+0000 7fa7b2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7b415d760 0x7fa7b40d23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49874/0 (socket says 172.21.15.12:49874) 2024-03-29T10:15:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.415+0000 7fa7b2000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa7b415db40 0x7fa7b40cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43624/0 (socket says 172.21.15.12:43624) 2024-03-29T10:15:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.415+0000 7fa7b2a00700 1 -- 172.21.15.12:0/3256672185 learned_addr learned my addr 172.21.15.12:0/3256672185 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.416+0000 7fa7b3400700 1 --2- 172.21.15.12:0/3256672185 >> v2:172.21.15.159:3300/0 conn(0x7fa7b40cf3c0 0x7fa7b40cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.416+0000 7fa7b2a00700 1 -- 172.21.15.12:0/3256672185 >> v2:172.21.15.159:3300/0 conn(0x7fa7b40cf3c0 msgr2=0x7fa7b40cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.416+0000 7fa7b2a00700 1 --2- 172.21.15.12:0/3256672185 >> v2:172.21.15.159:3300/0 conn(0x7fa7b40cf3c0 0x7fa7b40cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.416+0000 7fa7b2a00700 1 -- 172.21.15.12:0/3256672185 >> v1:172.21.15.12:6789/0 conn(0x7fa7b415db40 legacy=0x7fa7b40cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.416+0000 7fa7b2a00700 1 -- 172.21.15.12:0/3256672185 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b40d18f0 con 0x7fa7b415d760 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.416+0000 7fa7b2a00700 1 --2- 172.21.15.12:0/3256672185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415d760 0x7fa7b40d23d0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa7b408b920 tx=0x7fa7a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470e32f6ce302e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.416+0000 7fa7b1600700 1 -- 172.21.15.12:0/3256672185 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a400b870 con 0x7fa7b415d760 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.416+0000 7fa7b1600700 1 -- 172.21.15.12:0/3256672185 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a400b9d0 con 0x7fa7b415d760 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.416+0000 7fa7b1600700 1 -- 172.21.15.12:0/3256672185 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a400bd60 con 0x7fa7b415d760 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.417+0000 7fa7b9a00700 1 -- 172.21.15.12:0/3256672185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415d760 msgr2=0x7fa7b40d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.417+0000 7fa7b9a00700 1 --2- 172.21.15.12:0/3256672185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415d760 0x7fa7b40d23d0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa7b408b920 tx=0x7fa7a4009bd0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.417+0000 7fa7b9a00700 1 -- 172.21.15.12:0/3256672185 shutdown_connections 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.417+0000 7fa7b9a00700 1 --2- 172.21.15.12:0/3256672185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415d760 0x7fa7b40d23d0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.417+0000 7fa7b9a00700 1 --2- 172.21.15.12:0/3256672185 >> v2:172.21.15.159:3300/0 conn(0x7fa7b40cf3c0 0x7fa7b40cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.417+0000 7fa7b9a00700 1 -- 172.21.15.12:0/3256672185 >> 172.21.15.12:0/3256672185 conn(0x7fa7b408aff0 msgr2=0x7fa7b40d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:23.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.417+0000 7fa7b9a00700 1 -- 172.21.15.12:0/3256672185 shutdown_connections 2024-03-29T10:15:23.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.417+0000 7fa7b9a00700 1 -- 172.21.15.12:0/3256672185 wait complete. 2024-03-29T10:15:23.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.419+0000 7fa7b9a00700 1 Processor -- start 2024-03-29T10:15:23.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.420+0000 7fa7b9a00700 1 -- start start 2024-03-29T10:15:23.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b9a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7b40cf3c0 0x7fa7b41f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b9a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7b415d760 0x7fa7b40b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b9a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415db40 0x7fa7b40b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b9a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b40b61d0 con 0x7fa7b415db40 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b9a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b40ccda0 con 0x7fa7b40cf3c0 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b9a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b40cd080 con 0x7fa7b415d760 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415db40 0x7fa7b40b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415db40 0x7fa7b40b57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49876/0 (socket says 172.21.15.12:49876) 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b3400700 1 -- 172.21.15.12:0/2307416712 learned_addr learned my addr 172.21.15.12:0/2307416712 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b2a00700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7b40cf3c0 0x7fa7b41f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.421+0000 7fa7b2000700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7b415d760 0x7fa7b40b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7b3400700 1 -- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7b415d760 msgr2=0x7fa7b40b5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7b3400700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7b415d760 0x7fa7b40b5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7b3400700 1 -- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7b40cf3c0 msgr2=0x7fa7b41f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7b3400700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7b40cf3c0 0x7fa7b41f6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7b3400700 1 -- 172.21.15.12:0/2307416712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a4009550 con 0x7fa7b415db40 2024-03-29T10:15:23.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7b3400700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415db40 0x7fa7b40b57a0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009fd0 tx=0x7fa7ac00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:23.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7aaa00700 1 -- 172.21.15.12:0/2307416712 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac007ea0 con 0x7fa7b415db40 2024-03-29T10:15:23.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7aaa00700 1 -- 172.21.15.12:0/2307416712 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac00ed30 con 0x7fa7b415db40 2024-03-29T10:15:23.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7aaa00700 1 -- 172.21.15.12:0/2307416712 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac00eea0 con 0x7fa7b415db40 2024-03-29T10:15:23.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7b9a00700 1 -- 172.21.15.12:0/2307416712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b40cd360 con 0x7fa7b415db40 2024-03-29T10:15:23.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.422+0000 7fa7b9a00700 1 -- 172.21.15.12:0/2307416712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b41f6840 con 0x7fa7b415db40 2024-03-29T10:15:23.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.423+0000 7fa7b9a00700 1 -- 172.21.15.12:0/2307416712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b40cd550 con 0x7fa7b415db40 2024-03-29T10:15:23.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.424+0000 7fa7aaa00700 1 -- 172.21.15.12:0/2307416712 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa7ac014660 con 0x7fa7b415db40 2024-03-29T10:15:23.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.425+0000 7fa7aaa00700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa79406adb0 0x7fa79406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.425+0000 7fa7aaa00700 1 -- 172.21.15.12:0/2307416712 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7ac089a60 con 0x7fa7b415db40 2024-03-29T10:15:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.426+0000 7fa7b2a00700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa79406adb0 0x7fa79406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.426+0000 7fa7b2a00700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa79406adb0 0x7fa79406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4000c00 tx=0x7fa7a4008030 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:23.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.430+0000 7fa7aaa00700 1 -- 172.21.15.12:0/2307416712 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7ac054b20 con 0x7fa7b415db40 2024-03-29T10:15:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.640+0000 7fa7b9a00700 1 -- 172.21.15.12:0/2307416712 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7b41f69d0 con 0x7fa79406adb0 2024-03-29T10:15:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.651+0000 7fa7aaa00700 1 -- 172.21.15.12:0/2307416712 <== mgr.8425 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289555 (secure 0 0 0) 0x7fa7b41f69d0 con 0x7fa79406adb0 2024-03-29T10:15:23.653 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.655+0000 7fa7b9a00700 1 -- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa79406adb0 msgr2=0x7fa79406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.655+0000 7fa7b9a00700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa79406adb0 0x7fa79406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4000c00 tx=0x7fa7a4008030 comp rx=0 tx=0).stop 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.655+0000 7fa7b9a00700 1 -- 172.21.15.12:0/2307416712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415db40 msgr2=0x7fa7b40b57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.655+0000 7fa7b9a00700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415db40 0x7fa7b40b57a0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009fd0 tx=0x7fa7ac00e3c0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.655+0000 7fa7b9a00700 1 -- 172.21.15.12:0/2307416712 shutdown_connections 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.656+0000 7fa7b9a00700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa79406adb0 0x7fa79406d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.656+0000 7fa7b9a00700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b415db40 0x7fa7b40b57a0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.656+0000 7fa7b9a00700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7b40cf3c0 0x7fa7b41f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.656+0000 7fa7b9a00700 1 --2- 172.21.15.12:0/2307416712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7b415d760 0x7fa7b40b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.656+0000 7fa7b9a00700 1 -- 172.21.15.12:0/2307416712 >> 172.21.15.12:0/2307416712 conn(0x7fa7b408aff0 msgr2=0x7fa7b406c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.656+0000 7fa7b9a00700 1 -- 172.21.15.12:0/2307416712 shutdown_connections 2024-03-29T10:15:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.656+0000 7fa7b9a00700 1 -- 172.21.15.12:0/2307416712 wait complete. 2024-03-29T10:15:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:15:23.671 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-29T10:15:21.896214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":475,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":359956,"kb_used_data":15916,"kb_used_omap":12,"kb_used_meta":343923,"kb_avail":754614764,"statfs":{"total":773094113280,"available":772725518336,"internally_reserved":0,"allocated":16297984,"data_stored":8411967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":352177721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.683073"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779029+0000","last_change":"2024-03-29T10:15:18.774403+0000","last_active":"2024-03-29T10:15:18.779029+0000","last_peered":"2024-03-29T10:15:18.779029+0000","last_clean":"2024-03-29T10:15:18.779029+0000","last_became_active":"2024-03-29T10:15:18.774039+0000","last_became_peered":"2024-03-29T10:15:18.774039+0000","last_unstale":"2024-03-29T10:15:18.779029+0000","last_undegraded":"2024-03-29T10:15:18.779029+0000","last_fullsized":"2024-03-29T10:15:18.779029+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:21.873857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.776454+0000","last_change":"2024-03-29T10:15:18.768481+0000","last_active":"2024-03-29T10:15:18.776454+0000","last_peered":"2024-03-29T10:15:18.776454+0000","last_clean":"2024-03-29T10:15:18.776454+0000","last_became_active":"2024-03-29T10:15:18.767948+0000","last_became_peered":"2024-03-29T10:15:18.767948+0000","last_unstale":"2024-03-29T10:15:18.776454+0000","last_undegraded":"2024-03-29T10:15:18.776454+0000","last_fullsized":"2024-03-29T10:15:18.776454+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:16:20.679135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.775772+0000","last_change":"2024-03-29T10:15:18.772019+0000","last_active":"2024-03-29T10:15:18.775772+0000","last_peered":"2024-03-29T10:15:18.775772+0000","last_clean":"2024-03-29T10:15:18.775772+0000","last_became_active":"2024-03-29T10:15:18.771640+0000","last_became_peered":"2024-03-29T10:15:18.771640+0000","last_unstale":"2024-03-29T10:15:18.775772+0000","last_undegraded":"2024-03-29T10:15:18.775772+0000","last_fullsized":"2024-03-29T10:15:18.775772+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:57:09.757851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.776052+0000","last_change":"2024-03-29T10:15:18.769115+0000","last_active":"2024-03-29T10:15:18.776052+0000","last_peered":"2024-03-29T10:15:18.776052+0000","last_clean":"2024-03-29T10:15:18.776052+0000","last_became_active":"2024-03-29T10:15:18.768846+0000","last_became_peered":"2024-03-29T10:15:18.768846+0000","last_unstale":"2024-03-29T10:15:18.776052+0000","last_undegraded":"2024-03-29T10:15:18.776052+0000","last_fullsized":"2024-03-29T10:15:18.776052+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:46:02.528881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.766857+0000","last_change":"2024-03-29T10:15:18.762901+0000","last_active":"2024-03-29T10:15:18.766857+0000","last_peered":"2024-03-29T10:15:18.766857+0000","last_clean":"2024-03-29T10:15:18.766857+0000","last_became_active":"2024-03-29T10:15:18.762289+0000","last_became_peered":"2024-03-29T10:15:18.762289+0000","last_unstale":"2024-03-29T10:15:18.766857+0000","last_undegraded":"2024-03-29T10:15:18.766857+0000","last_fullsized":"2024-03-29T10:15:18.766857+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:19:18.684392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.770433+0000","last_change":"2024-03-29T10:15:18.765267+0000","last_active":"2024-03-29T10:15:18.770433+0000","last_peered":"2024-03-29T10:15:18.770433+0000","last_clean":"2024-03-29T10:15:18.770433+0000","last_became_active":"2024-03-29T10:15:18.765004+0000","last_became_peered":"2024-03-29T10:15:18.765004+0000","last_unstale":"2024-03-29T10:15:18.770433+0000","last_undegraded":"2024-03-29T10:15:18.770433+0000","last_fullsized":"2024-03-29T10:15:18.770433+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:20:23.230817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.772059+0000","last_change":"2024-03-29T10:15:18.767667+0000","last_active":"2024-03-29T10:15:18.772059+0000","last_peered":"2024-03-29T10:15:18.772059+0000","last_clean":"2024-03-29T10:15:18.772059+0000","last_became_active":"2024-03-29T10:15:18.767341+0000","last_became_peered":"2024-03-29T10:15:18.767341+0000","last_unstale":"2024-03-29T10:15:18.772059+0000","last_undegraded":"2024-03-29T10:15:18.772059+0000","last_fullsized":"2024-03-29T10:15:18.772059+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:43.862113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.776337+0000","last_change":"2024-03-29T10:15:18.771632+0000","last_active":"2024-03-29T10:15:18.776337+0000","last_peered":"2024-03-29T10:15:18.776337+0000","last_clean":"2024-03-29T10:15:18.776337+0000","last_became_active":"2024-03-29T10:15:18.769530+0000","last_became_peered":"2024-03-29T10:15:18.769530+0000","last_unstale":"2024-03-29T10:15:18.776337+0000","last_undegraded":"2024-03-29T10:15:18.776337+0000","last_fullsized":"2024-03-29T10:15:18.776337+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:15:05.102932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.766089+0000","last_change":"2024-03-29T10:15:18.763137+0000","last_active":"2024-03-29T10:15:18.766089+0000","last_peered":"2024-03-29T10:15:18.766089+0000","last_clean":"2024-03-29T10:15:18.766089+0000","last_became_active":"2024-03-29T10:15:18.762489+0000","last_became_peered":"2024-03-29T10:15:18.762489+0000","last_unstale":"2024-03-29T10:15:18.766089+0000","last_undegraded":"2024-03-29T10:15:18.766089+0000","last_fullsized":"2024-03-29T10:15:18.766089+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:39.106716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.776432+0000","last_change":"2024-03-29T10:15:18.768244+0000","last_active":"2024-03-29T10:15:18.776432+0000","last_peered":"2024-03-29T10:15:18.776432+0000","last_clean":"2024-03-29T10:15:18.776432+0000","last_became_active":"2024-03-29T10:15:18.767581+0000","last_became_peered":"2024-03-29T10:15:18.767581+0000","last_unstale":"2024-03-29T10:15:18.776432+0000","last_undegraded":"2024-03-29T10:15:18.776432+0000","last_fullsized":"2024-03-29T10:15:18.776432+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:47.766149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779963+0000","last_change":"2024-03-29T10:15:18.770356+0000","last_active":"2024-03-29T10:15:18.779963+0000","last_peered":"2024-03-29T10:15:18.779963+0000","last_clean":"2024-03-29T10:15:18.779963+0000","last_became_active":"2024-03-29T10:15:18.770038+0000","last_became_peered":"2024-03-29T10:15:18.770038+0000","last_unstale":"2024-03-29T10:15:18.779963+0000","last_undegraded":"2024-03-29T10:15:18.779963+0000","last_fullsized":"2024-03-29T10:15:18.779963+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:49:11.173864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.765880+0000","last_change":"2024-03-29T10:15:18.763296+0000","last_active":"2024-03-29T10:15:18.765880+0000","last_peered":"2024-03-29T10:15:18.765880+0000","last_clean":"2024-03-29T10:15:18.765880+0000","last_became_active":"2024-03-29T10:15:18.763012+0000","last_became_peered":"2024-03-29T10:15:18.763012+0000","last_unstale":"2024-03-29T10:15:18.765880+0000","last_undegraded":"2024-03-29T10:15:18.765880+0000","last_fullsized":"2024-03-29T10:15:18.765880+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:43:11.098286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779202+0000","last_change":"2024-03-29T10:15:18.774337+0000","last_active":"2024-03-29T10:15:18.779202+0000","last_peered":"2024-03-29T10:15:18.779202+0000","last_clean":"2024-03-29T10:15:18.779202+0000","last_became_active":"2024-03-29T10:15:18.774090+0000","last_became_peered":"2024-03-29T10:15:18.774090+0000","last_unstale":"2024-03-29T10:15:18.779202+0000","last_undegraded":"2024-03-29T10:15:18.779202+0000","last_fullsized":"2024-03-29T10:15:18.779202+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:28.640052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.761215+0000","last_change":"2024-03-29T10:15:18.754459+0000","last_active":"2024-03-29T10:15:18.761215+0000","last_peered":"2024-03-29T10:15:18.761215+0000","last_clean":"2024-03-29T10:15:18.761215+0000","last_became_active":"2024-03-29T10:15:18.754056+0000","last_became_peered":"2024-03-29T10:15:18.754056+0000","last_unstale":"2024-03-29T10:15:18.761215+0000","last_undegraded":"2024-03-29T10:15:18.761215+0000","last_fullsized":"2024-03-29T10:15:18.761215+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:13.983592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779046+0000","last_change":"2024-03-29T10:15:18.769932+0000","last_active":"2024-03-29T10:15:18.779046+0000","last_peered":"2024-03-29T10:15:18.779046+0000","last_clean":"2024-03-29T10:15:18.779046+0000","last_became_active":"2024-03-29T10:15:18.769580+0000","last_became_peered":"2024-03-29T10:15:18.769580+0000","last_unstale":"2024-03-29T10:15:18.779046+0000","last_undegraded":"2024-03-29T10:15:18.779046+0000","last_fullsized":"2024-03-29T10:15:18.779046+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:16:20.679135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.773998+0000","last_change":"2024-03-29T10:15:18.768079+0000","last_active":"2024-03-29T10:15:18.773998+0000","last_peered":"2024-03-29T10:15:18.773998+0000","last_clean":"2024-03-29T10:15:18.773998+0000","last_became_active":"2024-03-29T10:15:18.767611+0000","last_became_peered":"2024-03-29T10:15:18.767611+0000","last_unstale":"2024-03-29T10:15:18.773998+0000","last_undegraded":"2024-03-29T10:15:18.773998+0000","last_fullsized":"2024-03-29T10:15:18.773998+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:20:58.998966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.775622+0000","last_change":"2024-03-29T10:15:18.768939+0000","last_active":"2024-03-29T10:15:18.775622+0000","last_peered":"2024-03-29T10:15:18.775622+0000","last_clean":"2024-03-29T10:15:18.775622+0000","last_became_active":"2024-03-29T10:15:18.768669+0000","last_became_peered":"2024-03-29T10:15:18.768669+0000","last_unstale":"2024-03-29T10:15:18.775622+0000","last_undegraded":"2024-03-29T10:15:18.775622+0000","last_fullsized":"2024-03-29T10:15:18.775622+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:28:12.237242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.775258+0000","last_change":"2024-03-29T10:15:18.772577+0000","last_active":"2024-03-29T10:15:18.775258+0000","last_peered":"2024-03-29T10:15:18.775258+0000","last_clean":"2024-03-29T10:15:18.775258+0000","last_became_active":"2024-03-29T10:15:18.771978+0000","last_became_peered":"2024-03-29T10:15:18.771978+0000","last_unstale":"2024-03-29T10:15:18.775258+0000","last_undegraded":"2024-03-29T10:15:18.775258+0000","last_fullsized":"2024-03-29T10:15:18.775258+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:38.270423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779312+0000","last_change":"2024-03-29T10:15:18.769713+0000","last_active":"2024-03-29T10:15:18.779312+0000","last_peered":"2024-03-29T10:15:18.779312+0000","last_clean":"2024-03-29T10:15:18.779312+0000","last_became_active":"2024-03-29T10:15:18.769435+0000","last_became_peered":"2024-03-29T10:15:18.769435+0000","last_unstale":"2024-03-29T10:15:18.779312+0000","last_undegraded":"2024-03-29T10:15:18.779312+0000","last_fullsized":"2024-03-29T10:15:18.779312+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:06:04.148022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.773172+0000","last_change":"2024-03-29T10:15:18.767477+0000","last_active":"2024-03-29T10:15:18.773172+0000","last_peered":"2024-03-29T10:15:18.773172+0000","last_clean":"2024-03-29T10:15:18.773172+0000","last_became_active":"2024-03-29T10:15:18.767203+0000","last_became_peered":"2024-03-29T10:15:18.767203+0000","last_unstale":"2024-03-29T10:15:18.773172+0000","last_undegraded":"2024-03-29T10:15:18.773172+0000","last_fullsized":"2024-03-29T10:15:18.773172+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:01.591227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.764678+0000","last_change":"2024-03-29T10:15:18.761179+0000","last_active":"2024-03-29T10:15:18.764678+0000","last_peered":"2024-03-29T10:15:18.764678+0000","last_clean":"2024-03-29T10:15:18.764678+0000","last_became_active":"2024-03-29T10:15:18.760891+0000","last_became_peered":"2024-03-29T10:15:18.760891+0000","last_unstale":"2024-03-29T10:15:18.764678+0000","last_undegraded":"2024-03-29T10:15:18.764678+0000","last_fullsized":"2024-03-29T10:15:18.764678+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:27:53.728925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.778733+0000","last_change":"2024-03-29T10:15:18.767659+0000","last_active":"2024-03-29T10:15:18.778733+0000","last_peered":"2024-03-29T10:15:18.778733+0000","last_clean":"2024-03-29T10:15:18.778733+0000","last_became_active":"2024-03-29T10:15:18.767362+0000","last_became_peered":"2024-03-29T10:15:18.767362+0000","last_unstale":"2024-03-29T10:15:18.778733+0000","last_undegraded":"2024-03-29T10:15:18.778733+0000","last_fullsized":"2024-03-29T10:15:18.778733+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:49.750852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.766806+0000","last_change":"2024-03-29T10:15:18.765029+0000","last_active":"2024-03-29T10:15:18.766806+0000","last_peered":"2024-03-29T10:15:18.766806+0000","last_clean":"2024-03-29T10:15:18.766806+0000","last_became_active":"2024-03-29T10:15:18.763526+0000","last_became_peered":"2024-03-29T10:15:18.763526+0000","last_unstale":"2024-03-29T10:15:18.766806+0000","last_undegraded":"2024-03-29T10:15:18.766806+0000","last_fullsized":"2024-03-29T10:15:18.766806+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:21:42.891275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.778793+0000","last_change":"2024-03-29T10:15:18.764827+0000","last_active":"2024-03-29T10:15:18.778793+0000","last_peered":"2024-03-29T10:15:18.778793+0000","last_clean":"2024-03-29T10:15:18.778793+0000","last_became_active":"2024-03-29T10:15:18.762293+0000","last_became_peered":"2024-03-29T10:15:18.762293+0000","last_unstale":"2024-03-29T10:15:18.778793+0000","last_undegraded":"2024-03-29T10:15:18.778793+0000","last_fullsized":"2024-03-29T10:15:18.778793+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:01:22.602449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.764082+0000","last_change":"2024-03-29T10:15:18.755725+0000","last_active":"2024-03-29T10:15:18.764082+0000","last_peered":"2024-03-29T10:15:18.764082+0000","last_clean":"2024-03-29T10:15:18.764082+0000","last_became_active":"2024-03-29T10:15:18.755425+0000","last_became_peered":"2024-03-29T10:15:18.755425+0000","last_unstale":"2024-03-29T10:15:18.764082+0000","last_undegraded":"2024-03-29T10:15:18.764082+0000","last_fullsized":"2024-03-29T10:15:18.764082+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:59.721555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.775927+0000","last_change":"2024-03-29T10:15:18.775040+0000","last_active":"2024-03-29T10:15:18.775927+0000","last_peered":"2024-03-29T10:15:18.775927+0000","last_clean":"2024-03-29T10:15:18.775927+0000","last_became_active":"2024-03-29T10:15:18.774772+0000","last_became_peered":"2024-03-29T10:15:18.774772+0000","last_unstale":"2024-03-29T10:15:18.775927+0000","last_undegraded":"2024-03-29T10:15:18.775927+0000","last_fullsized":"2024-03-29T10:15:18.775927+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:04:09.775205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779230+0000","last_change":"2024-03-29T10:15:18.771982+0000","last_active":"2024-03-29T10:15:18.779230+0000","last_peered":"2024-03-29T10:15:18.779230+0000","last_clean":"2024-03-29T10:15:18.779230+0000","last_became_active":"2024-03-29T10:15:18.771720+0000","last_became_peered":"2024-03-29T10:15:18.771720+0000","last_unstale":"2024-03-29T10:15:18.779230+0000","last_undegraded":"2024-03-29T10:15:18.779230+0000","last_fullsized":"2024-03-29T10:15:18.779230+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:11.882236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.772151+0000","last_change":"2024-03-29T10:15:18.767757+0000","last_active":"2024-03-29T10:15:18.772151+0000","last_peered":"2024-03-29T10:15:18.772151+0000","last_clean":"2024-03-29T10:15:18.772151+0000","last_became_active":"2024-03-29T10:15:18.766764+0000","last_became_peered":"2024-03-29T10:15:18.766764+0000","last_unstale":"2024-03-29T10:15:18.772151+0000","last_undegraded":"2024-03-29T10:15:18.772151+0000","last_fullsized":"2024-03-29T10:15:18.772151+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:40:36.977077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779170+0000","last_change":"2024-03-29T10:15:18.772644+0000","last_active":"2024-03-29T10:15:18.779170+0000","last_peered":"2024-03-29T10:15:18.779170+0000","last_clean":"2024-03-29T10:15:18.779170+0000","last_became_active":"2024-03-29T10:15:18.770500+0000","last_became_peered":"2024-03-29T10:15:18.770500+0000","last_unstale":"2024-03-29T10:15:18.779170+0000","last_undegraded":"2024-03-29T10:15:18.779170+0000","last_fullsized":"2024-03-29T10:15:18.779170+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:58.525728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.776578+0000","last_change":"2024-03-29T10:15:18.767932+0000","last_active":"2024-03-29T10:15:18.776578+0000","last_peered":"2024-03-29T10:15:18.776578+0000","last_clean":"2024-03-29T10:15:18.776578+0000","last_became_active":"2024-03-29T10:15:18.767630+0000","last_became_peered":"2024-03-29T10:15:18.767630+0000","last_unstale":"2024-03-29T10:15:18.776578+0000","last_undegraded":"2024-03-29T10:15:18.776578+0000","last_fullsized":"2024-03-29T10:15:18.776578+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:47.766149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.766580+0000","last_change":"2024-03-29T10:15:18.763321+0000","last_active":"2024-03-29T10:15:18.766580+0000","last_peered":"2024-03-29T10:15:18.766580+0000","last_clean":"2024-03-29T10:15:18.766580+0000","last_became_active":"2024-03-29T10:15:18.761584+0000","last_became_peered":"2024-03-29T10:15:18.761584+0000","last_unstale":"2024-03-29T10:15:18.766580+0000","last_undegraded":"2024-03-29T10:15:18.766580+0000","last_fullsized":"2024-03-29T10:15:18.766580+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:02.351513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.775093+0000","last_change":"2024-03-29T10:15:18.769126+0000","last_active":"2024-03-29T10:15:18.775093+0000","last_peered":"2024-03-29T10:15:18.775093+0000","last_clean":"2024-03-29T10:15:18.775093+0000","last_became_active":"2024-03-29T10:15:18.768786+0000","last_became_peered":"2024-03-29T10:15:18.768786+0000","last_unstale":"2024-03-29T10:15:18.775093+0000","last_undegraded":"2024-03-29T10:15:18.775093+0000","last_fullsized":"2024-03-29T10:15:18.775093+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:05.110226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.775240+0000","last_change":"2024-03-29T10:15:18.771968+0000","last_active":"2024-03-29T10:15:18.775240+0000","last_peered":"2024-03-29T10:15:18.775240+0000","last_clean":"2024-03-29T10:15:18.775240+0000","last_became_active":"2024-03-29T10:15:18.768925+0000","last_became_peered":"2024-03-29T10:15:18.768925+0000","last_unstale":"2024-03-29T10:15:18.775240+0000","last_undegraded":"2024-03-29T10:15:18.775240+0000","last_fullsized":"2024-03-29T10:15:18.775240+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:26:23.283920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"153'4","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781736+0000","last_change":"2024-03-29T10:15:18.248449+0000","last_active":"2024-03-29T10:15:18.781736+0000","last_peered":"2024-03-29T10:15:18.781736+0000","last_clean":"2024-03-29T10:15:18.781736+0000","last_became_active":"2024-03-29T10:15:18.248092+0000","last_became_peered":"2024-03-29T10:15:18.248092+0000","last_unstale":"2024-03-29T10:15:18.781736+0000","last_undegraded":"2024-03-29T10:15:18.781736+0000","last_fullsized":"2024-03-29T10:15:18.781736+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:16:14.096536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.766161+0000","last_change":"2024-03-29T10:15:18.763323+0000","last_active":"2024-03-29T10:15:18.766161+0000","last_peered":"2024-03-29T10:15:18.766161+0000","last_clean":"2024-03-29T10:15:18.766161+0000","last_became_active":"2024-03-29T10:15:18.762867+0000","last_became_peered":"2024-03-29T10:15:18.762867+0000","last_unstale":"2024-03-29T10:15:18.766161+0000","last_undegraded":"2024-03-29T10:15:18.766161+0000","last_fullsized":"2024-03-29T10:15:18.766161+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:05.110226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.780997+0000","last_change":"2024-03-29T10:15:18.245102+0000","last_active":"2024-03-29T10:15:18.780997+0000","last_peered":"2024-03-29T10:15:18.780997+0000","last_clean":"2024-03-29T10:15:18.780997+0000","last_became_active":"2024-03-29T10:15:18.244719+0000","last_became_peered":"2024-03-29T10:15:18.244719+0000","last_unstale":"2024-03-29T10:15:18.780997+0000","last_undegraded":"2024-03-29T10:15:18.780997+0000","last_fullsized":"2024-03-29T10:15:18.780997+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:02:37.063807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.774718+0000","last_change":"2024-03-29T10:15:18.768939+0000","last_active":"2024-03-29T10:15:18.774718+0000","last_peered":"2024-03-29T10:15:18.774718+0000","last_clean":"2024-03-29T10:15:18.774718+0000","last_became_active":"2024-03-29T10:15:18.767916+0000","last_became_peered":"2024-03-29T10:15:18.767916+0000","last_unstale":"2024-03-29T10:15:18.774718+0000","last_undegraded":"2024-03-29T10:15:18.774718+0000","last_fullsized":"2024-03-29T10:15:18.774718+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:39:20.841112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.782382+0000","last_change":"2024-03-29T10:15:18.243489+0000","last_active":"2024-03-29T10:15:18.782382+0000","last_peered":"2024-03-29T10:15:18.782382+0000","last_clean":"2024-03-29T10:15:18.782382+0000","last_became_active":"2024-03-29T10:15:18.243234+0000","last_became_peered":"2024-03-29T10:15:18.243234+0000","last_unstale":"2024-03-29T10:15:18.782382+0000","last_undegraded":"2024-03-29T10:15:18.782382+0000","last_fullsized":"2024-03-29T10:15:18.782382+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:17:10.429998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.766553+0000","last_change":"2024-03-29T10:15:18.763009+0000","last_active":"2024-03-29T10:15:18.766553+0000","last_peered":"2024-03-29T10:15:18.766553+0000","last_clean":"2024-03-29T10:15:18.766553+0000","last_became_active":"2024-03-29T10:15:18.761992+0000","last_became_peered":"2024-03-29T10:15:18.761992+0000","last_unstale":"2024-03-29T10:15:18.766553+0000","last_undegraded":"2024-03-29T10:15:18.766553+0000","last_fullsized":"2024-03-29T10:15:18.766553+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:03:07.307244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781639+0000","last_change":"2024-03-29T10:15:18.246698+0000","last_active":"2024-03-29T10:15:18.781639+0000","last_peered":"2024-03-29T10:15:18.781639+0000","last_clean":"2024-03-29T10:15:18.781639+0000","last_became_active":"2024-03-29T10:15:18.245609+0000","last_became_peered":"2024-03-29T10:15:18.245609+0000","last_unstale":"2024-03-29T10:15:18.781639+0000","last_undegraded":"2024-03-29T10:15:18.781639+0000","last_fullsized":"2024-03-29T10:15:18.781639+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:09:00.431279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.775289+0000","last_change":"2024-03-29T10:15:18.769523+0000","last_active":"2024-03-29T10:15:18.775289+0000","last_peered":"2024-03-29T10:15:18.775289+0000","last_clean":"2024-03-29T10:15:18.775289+0000","last_became_active":"2024-03-29T10:15:18.769263+0000","last_became_peered":"2024-03-29T10:15:18.769263+0000","last_unstale":"2024-03-29T10:15:18.775289+0000","last_undegraded":"2024-03-29T10:15:18.775289+0000","last_fullsized":"2024-03-29T10:15:18.775289+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:57.956507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.777943+0000","last_change":"2024-03-29T10:15:18.236651+0000","last_active":"2024-03-29T10:15:18.777943+0000","last_peered":"2024-03-29T10:15:18.777943+0000","last_clean":"2024-03-29T10:15:18.777943+0000","last_became_active":"2024-03-29T10:15:18.236279+0000","last_became_peered":"2024-03-29T10:15:18.236279+0000","last_unstale":"2024-03-29T10:15:18.777943+0000","last_undegraded":"2024-03-29T10:15:18.777943+0000","last_fullsized":"2024-03-29T10:15:18.777943+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:48:21.209404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.775852+0000","last_change":"2024-03-29T10:15:18.770581+0000","last_active":"2024-03-29T10:15:18.775852+0000","last_peered":"2024-03-29T10:15:18.775852+0000","last_clean":"2024-03-29T10:15:18.775852+0000","last_became_active":"2024-03-29T10:15:18.769791+0000","last_became_peered":"2024-03-29T10:15:18.769791+0000","last_unstale":"2024-03-29T10:15:18.775852+0000","last_undegraded":"2024-03-29T10:15:18.775852+0000","last_fullsized":"2024-03-29T10:15:18.775852+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:28.620001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779423+0000","last_change":"2024-03-29T10:15:18.239442+0000","last_active":"2024-03-29T10:15:18.779423+0000","last_peered":"2024-03-29T10:15:18.779423+0000","last_clean":"2024-03-29T10:15:18.779423+0000","last_became_active":"2024-03-29T10:15:18.236125+0000","last_became_peered":"2024-03-29T10:15:18.236125+0000","last_unstale":"2024-03-29T10:15:18.779423+0000","last_undegraded":"2024-03-29T10:15:18.779423+0000","last_fullsized":"2024-03-29T10:15:18.779423+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:08:34.640384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.776319+0000","last_change":"2024-03-29T10:15:18.770877+0000","last_active":"2024-03-29T10:15:18.776319+0000","last_peered":"2024-03-29T10:15:18.776319+0000","last_clean":"2024-03-29T10:15:18.776319+0000","last_became_active":"2024-03-29T10:15:18.769944+0000","last_became_peered":"2024-03-29T10:15:18.769944+0000","last_unstale":"2024-03-29T10:15:18.776319+0000","last_undegraded":"2024-03-29T10:15:18.776319+0000","last_fullsized":"2024-03-29T10:15:18.776319+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:43.862113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.763267+0000","last_change":"2024-03-29T10:15:18.244901+0000","last_active":"2024-03-29T10:15:18.763267+0000","last_peered":"2024-03-29T10:15:18.763267+0000","last_clean":"2024-03-29T10:15:18.763267+0000","last_became_active":"2024-03-29T10:15:18.244626+0000","last_became_peered":"2024-03-29T10:15:18.244626+0000","last_unstale":"2024-03-29T10:15:18.763267+0000","last_undegraded":"2024-03-29T10:15:18.763267+0000","last_fullsized":"2024-03-29T10:15:18.763267+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:10.873286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.764294+0000","last_change":"2024-03-29T10:15:18.760287+0000","last_active":"2024-03-29T10:15:18.764294+0000","last_peered":"2024-03-29T10:15:18.764294+0000","last_clean":"2024-03-29T10:15:18.764294+0000","last_became_active":"2024-03-29T10:15:18.759386+0000","last_became_peered":"2024-03-29T10:15:18.759386+0000","last_unstale":"2024-03-29T10:15:18.764294+0000","last_undegraded":"2024-03-29T10:15:18.764294+0000","last_fullsized":"2024-03-29T10:15:18.764294+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:44:59.846995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.748982+0000","last_change":"2024-03-29T10:15:18.238756+0000","last_active":"2024-03-29T10:15:18.748982+0000","last_peered":"2024-03-29T10:15:18.748982+0000","last_clean":"2024-03-29T10:15:18.748982+0000","last_became_active":"2024-03-29T10:15:18.238499+0000","last_became_peered":"2024-03-29T10:15:18.238499+0000","last_unstale":"2024-03-29T10:15:18.748982+0000","last_undegraded":"2024-03-29T10:15:18.748982+0000","last_fullsized":"2024-03-29T10:15:18.748982+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:41:25.038841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781395+0000","last_change":"2024-03-29T10:15:18.779956+0000","last_active":"2024-03-29T10:15:18.781395+0000","last_peered":"2024-03-29T10:15:18.781395+0000","last_clean":"2024-03-29T10:15:18.781395+0000","last_became_active":"2024-03-29T10:15:18.779231+0000","last_became_peered":"2024-03-29T10:15:18.779231+0000","last_unstale":"2024-03-29T10:15:18.781395+0000","last_undegraded":"2024-03-29T10:15:18.781395+0000","last_fullsized":"2024-03-29T10:15:18.781395+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:20.803416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.756966+0000","last_change":"2024-03-29T10:15:18.234835+0000","last_active":"2024-03-29T10:15:18.756966+0000","last_peered":"2024-03-29T10:15:18.756966+0000","last_clean":"2024-03-29T10:15:18.756966+0000","last_became_active":"2024-03-29T10:15:18.233366+0000","last_became_peered":"2024-03-29T10:15:18.233366+0000","last_unstale":"2024-03-29T10:15:18.756966+0000","last_undegraded":"2024-03-29T10:15:18.756966+0000","last_fullsized":"2024-03-29T10:15:18.756966+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:19.681575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.772215+0000","last_change":"2024-03-29T10:15:18.768888+0000","last_active":"2024-03-29T10:15:18.772215+0000","last_peered":"2024-03-29T10:15:18.772215+0000","last_clean":"2024-03-29T10:15:18.772215+0000","last_became_active":"2024-03-29T10:15:18.768603+0000","last_became_peered":"2024-03-29T10:15:18.768603+0000","last_unstale":"2024-03-29T10:15:18.772215+0000","last_undegraded":"2024-03-29T10:15:18.772215+0000","last_fullsized":"2024-03-29T10:15:18.772215+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:13.983592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.768700+0000","last_change":"2024-03-29T10:15:18.244095+0000","last_active":"2024-03-29T10:15:18.768700+0000","last_peered":"2024-03-29T10:15:18.768700+0000","last_clean":"2024-03-29T10:15:18.768700+0000","last_became_active":"2024-03-29T10:15:18.243018+0000","last_became_peered":"2024-03-29T10:15:18.243018+0000","last_unstale":"2024-03-29T10:15:18.768700+0000","last_undegraded":"2024-03-29T10:15:18.768700+0000","last_fullsized":"2024-03-29T10:15:18.768700+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:42.853163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.774952+0000","last_change":"2024-03-29T10:15:18.769534+0000","last_active":"2024-03-29T10:15:18.774952+0000","last_peered":"2024-03-29T10:15:18.774952+0000","last_clean":"2024-03-29T10:15:18.774952+0000","last_became_active":"2024-03-29T10:15:18.769285+0000","last_became_peered":"2024-03-29T10:15:18.769285+0000","last_unstale":"2024-03-29T10:15:18.774952+0000","last_undegraded":"2024-03-29T10:15:18.774952+0000","last_fullsized":"2024-03-29T10:15:18.774952+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:26:23.283920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"153'3","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.782176+0000","last_change":"2024-03-29T10:15:18.230283+0000","last_active":"2024-03-29T10:15:18.782176+0000","last_peered":"2024-03-29T10:15:18.782176+0000","last_clean":"2024-03-29T10:15:18.782176+0000","last_became_active":"2024-03-29T10:15:18.229323+0000","last_became_peered":"2024-03-29T10:15:18.229323+0000","last_unstale":"2024-03-29T10:15:18.782176+0000","last_undegraded":"2024-03-29T10:15:18.782176+0000","last_fullsized":"2024-03-29T10:15:18.782176+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:50:44.068800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.760125+0000","last_change":"2024-03-29T10:15:18.758162+0000","last_active":"2024-03-29T10:15:18.760125+0000","last_peered":"2024-03-29T10:15:18.760125+0000","last_clean":"2024-03-29T10:15:18.760125+0000","last_became_active":"2024-03-29T10:15:18.757041+0000","last_became_peered":"2024-03-29T10:15:18.757041+0000","last_unstale":"2024-03-29T10:15:18.760125+0000","last_undegraded":"2024-03-29T10:15:18.760125+0000","last_fullsized":"2024-03-29T10:15:18.760125+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:39.190584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"153'6","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.783970+0000","last_change":"2024-03-29T10:15:18.238514+0000","last_active":"2024-03-29T10:15:18.783970+0000","last_peered":"2024-03-29T10:15:18.783970+0000","last_clean":"2024-03-29T10:15:18.783970+0000","last_became_active":"2024-03-29T10:15:18.238124+0000","last_became_peered":"2024-03-29T10:15:18.238124+0000","last_unstale":"2024-03-29T10:15:18.783970+0000","last_undegraded":"2024-03-29T10:15:18.783970+0000","last_fullsized":"2024-03-29T10:15:18.783970+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:46:01.519930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779185+0000","last_change":"2024-03-29T10:15:18.774341+0000","last_active":"2024-03-29T10:15:18.779185+0000","last_peered":"2024-03-29T10:15:18.779185+0000","last_clean":"2024-03-29T10:15:18.779185+0000","last_became_active":"2024-03-29T10:15:18.770843+0000","last_became_peered":"2024-03-29T10:15:18.770843+0000","last_unstale":"2024-03-29T10:15:18.779185+0000","last_undegraded":"2024-03-29T10:15:18.779185+0000","last_fullsized":"2024-03-29T10:15:18.779185+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:43:11.098286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.756372+0000","last_change":"2024-03-29T10:15:18.240012+0000","last_active":"2024-03-29T10:15:18.756372+0000","last_peered":"2024-03-29T10:15:18.756372+0000","last_clean":"2024-03-29T10:15:18.756372+0000","last_became_active":"2024-03-29T10:15:18.239023+0000","last_became_peered":"2024-03-29T10:15:18.239023+0000","last_unstale":"2024-03-29T10:15:18.756372+0000","last_undegraded":"2024-03-29T10:15:18.756372+0000","last_fullsized":"2024-03-29T10:15:18.756372+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:12:25.936776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.763358+0000","last_change":"2024-03-29T10:15:18.754727+0000","last_active":"2024-03-29T10:15:18.763358+0000","last_peered":"2024-03-29T10:15:18.763358+0000","last_clean":"2024-03-29T10:15:18.763358+0000","last_became_active":"2024-03-29T10:15:18.754469+0000","last_became_peered":"2024-03-29T10:15:18.754469+0000","last_unstale":"2024-03-29T10:15:18.763358+0000","last_undegraded":"2024-03-29T10:15:18.763358+0000","last_fullsized":"2024-03-29T10:15:18.763358+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:34:15.821450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.749534+0000","last_change":"2024-03-29T10:15:18.245095+0000","last_active":"2024-03-29T10:15:18.749534+0000","last_peered":"2024-03-29T10:15:18.749534+0000","last_clean":"2024-03-29T10:15:18.749534+0000","last_became_active":"2024-03-29T10:15:18.244834+0000","last_became_peered":"2024-03-29T10:15:18.244834+0000","last_unstale":"2024-03-29T10:15:18.749534+0000","last_undegraded":"2024-03-29T10:15:18.749534+0000","last_fullsized":"2024-03-29T10:15:18.749534+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:27.668887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.767632+0000","last_change":"2024-03-29T10:15:18.766175+0000","last_active":"2024-03-29T10:15:18.767632+0000","last_peered":"2024-03-29T10:15:18.767632+0000","last_clean":"2024-03-29T10:15:18.767632+0000","last_became_active":"2024-03-29T10:15:18.762502+0000","last_became_peered":"2024-03-29T10:15:18.762502+0000","last_unstale":"2024-03-29T10:15:18.767632+0000","last_undegraded":"2024-03-29T10:15:18.767632+0000","last_fullsized":"2024-03-29T10:15:18.767632+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:56.375633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"153'8","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.782537+0000","last_change":"2024-03-29T10:15:18.236493+0000","last_active":"2024-03-29T10:15:18.782537+0000","last_peered":"2024-03-29T10:15:18.782537+0000","last_clean":"2024-03-29T10:15:18.782537+0000","last_became_active":"2024-03-29T10:15:18.233617+0000","last_became_peered":"2024-03-29T10:15:18.233617+0000","last_unstale":"2024-03-29T10:15:18.782537+0000","last_undegraded":"2024-03-29T10:15:18.782537+0000","last_fullsized":"2024-03-29T10:15:18.782537+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:12.974642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.764473+0000","last_change":"2024-03-29T10:15:18.758467+0000","last_active":"2024-03-29T10:15:18.764473+0000","last_peered":"2024-03-29T10:15:18.764473+0000","last_clean":"2024-03-29T10:15:18.764473+0000","last_became_active":"2024-03-29T10:15:18.758163+0000","last_became_peered":"2024-03-29T10:15:18.758163+0000","last_unstale":"2024-03-29T10:15:18.764473+0000","last_undegraded":"2024-03-29T10:15:18.764473+0000","last_fullsized":"2024-03-29T10:15:18.764473+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:08:35.649335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.778850+0000","last_change":"2024-03-29T10:15:18.240951+0000","last_active":"2024-03-29T10:15:18.778850+0000","last_peered":"2024-03-29T10:15:18.778850+0000","last_clean":"2024-03-29T10:15:18.778850+0000","last_became_active":"2024-03-29T10:15:18.240622+0000","last_became_peered":"2024-03-29T10:15:18.240622+0000","last_unstale":"2024-03-29T10:15:18.778850+0000","last_undegraded":"2024-03-29T10:15:18.778850+0000","last_fullsized":"2024-03-29T10:15:18.778850+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:12:25.936776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.772195+0000","last_change":"2024-03-29T10:15:18.765995+0000","last_active":"2024-03-29T10:15:18.772195+0000","last_peered":"2024-03-29T10:15:18.772195+0000","last_clean":"2024-03-29T10:15:18.772195+0000","last_became_active":"2024-03-29T10:15:18.765734+0000","last_became_peered":"2024-03-29T10:15:18.765734+0000","last_unstale":"2024-03-29T10:15:18.772195+0000","last_undegraded":"2024-03-29T10:15:18.772195+0000","last_fullsized":"2024-03-29T10:15:18.772195+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:46:02.528881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781906+0000","last_change":"2024-03-29T10:15:18.246087+0000","last_active":"2024-03-29T10:15:18.781906+0000","last_peered":"2024-03-29T10:15:18.781906+0000","last_clean":"2024-03-29T10:15:18.781906+0000","last_became_active":"2024-03-29T10:15:18.244538+0000","last_became_peered":"2024-03-29T10:15:18.244538+0000","last_unstale":"2024-03-29T10:15:18.781906+0000","last_undegraded":"2024-03-29T10:15:18.781906+0000","last_fullsized":"2024-03-29T10:15:18.781906+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:32:56.586878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.780559+0000","last_change":"2024-03-29T10:15:18.777418+0000","last_active":"2024-03-29T10:15:18.780559+0000","last_peered":"2024-03-29T10:15:18.780559+0000","last_clean":"2024-03-29T10:15:18.780559+0000","last_became_active":"2024-03-29T10:15:18.777127+0000","last_became_peered":"2024-03-29T10:15:18.777127+0000","last_unstale":"2024-03-29T10:15:18.780559+0000","last_undegraded":"2024-03-29T10:15:18.780559+0000","last_fullsized":"2024-03-29T10:15:18.780559+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:27:53.728925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779451+0000","last_change":"2024-03-29T10:15:18.247457+0000","last_active":"2024-03-29T10:15:18.779451+0000","last_peered":"2024-03-29T10:15:18.779451+0000","last_clean":"2024-03-29T10:15:18.779451+0000","last_became_active":"2024-03-29T10:15:18.247070+0000","last_became_peered":"2024-03-29T10:15:18.247070+0000","last_unstale":"2024-03-29T10:15:18.779451+0000","last_undegraded":"2024-03-29T10:15:18.779451+0000","last_fullsized":"2024-03-29T10:15:18.779451+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:19.794465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.773374+0000","last_change":"2024-03-29T10:15:18.767943+0000","last_active":"2024-03-29T10:15:18.773374+0000","last_peered":"2024-03-29T10:15:18.773374+0000","last_clean":"2024-03-29T10:15:18.773374+0000","last_became_active":"2024-03-29T10:15:18.764049+0000","last_became_peered":"2024-03-29T10:15:18.764049+0000","last_unstale":"2024-03-29T10:15:18.773374+0000","last_undegraded":"2024-03-29T10:15:18.773374+0000","last_fullsized":"2024-03-29T10:15:18.773374+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:12:12.821950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.757821+0000","last_change":"2024-03-29T10:15:18.242641+0000","last_active":"2024-03-29T10:15:18.757821+0000","last_peered":"2024-03-29T10:15:18.757821+0000","last_clean":"2024-03-29T10:15:18.757821+0000","last_became_active":"2024-03-29T10:15:18.242367+0000","last_became_peered":"2024-03-29T10:15:18.242367+0000","last_unstale":"2024-03-29T10:15:18.757821+0000","last_undegraded":"2024-03-29T10:15:18.757821+0000","last_fullsized":"2024-03-29T10:15:18.757821+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:12:40.712000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779101+0000","last_change":"2024-03-29T10:15:18.774216+0000","last_active":"2024-03-29T10:15:18.779101+0000","last_peered":"2024-03-29T10:15:18.779101+0000","last_clean":"2024-03-29T10:15:18.779101+0000","last_became_active":"2024-03-29T10:15:18.773560+0000","last_became_peered":"2024-03-29T10:15:18.773560+0000","last_unstale":"2024-03-29T10:15:18.779101+0000","last_undegraded":"2024-03-29T10:15:18.779101+0000","last_fullsized":"2024-03-29T10:15:18.779101+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:33.197637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781438+0000","last_change":"2024-03-29T10:15:18.232949+0000","last_active":"2024-03-29T10:15:18.781438+0000","last_peered":"2024-03-29T10:15:18.781438+0000","last_clean":"2024-03-29T10:15:18.781438+0000","last_became_active":"2024-03-29T10:15:18.232594+0000","last_became_peered":"2024-03-29T10:15:18.232594+0000","last_unstale":"2024-03-29T10:15:18.781438+0000","last_undegraded":"2024-03-29T10:15:18.781438+0000","last_fullsized":"2024-03-29T10:15:18.781438+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:25.414983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.771337+0000","last_change":"2024-03-29T10:15:18.764608+0000","last_active":"2024-03-29T10:15:18.771337+0000","last_peered":"2024-03-29T10:15:18.771337+0000","last_clean":"2024-03-29T10:15:18.771337+0000","last_became_active":"2024-03-29T10:15:18.761955+0000","last_became_peered":"2024-03-29T10:15:18.761955+0000","last_unstale":"2024-03-29T10:15:18.771337+0000","last_undegraded":"2024-03-29T10:15:18.771337+0000","last_fullsized":"2024-03-29T10:15:18.771337+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:57:09.757851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.763609+0000","last_change":"2024-03-29T10:15:18.240062+0000","last_active":"2024-03-29T10:15:18.763609+0000","last_peered":"2024-03-29T10:15:18.763609+0000","last_clean":"2024-03-29T10:15:18.763609+0000","last_became_active":"2024-03-29T10:15:18.239758+0000","last_became_peered":"2024-03-29T10:15:18.239758+0000","last_unstale":"2024-03-29T10:15:18.763609+0000","last_undegraded":"2024-03-29T10:15:18.763609+0000","last_fullsized":"2024-03-29T10:15:18.763609+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:27.611051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.769875+0000","last_change":"2024-03-29T10:15:18.765774+0000","last_active":"2024-03-29T10:15:18.769875+0000","last_peered":"2024-03-29T10:15:18.769875+0000","last_clean":"2024-03-29T10:15:18.769875+0000","last_became_active":"2024-03-29T10:15:18.765517+0000","last_became_peered":"2024-03-29T10:15:18.765517+0000","last_unstale":"2024-03-29T10:15:18.769875+0000","last_undegraded":"2024-03-29T10:15:18.769875+0000","last_fullsized":"2024-03-29T10:15:18.769875+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:16:23.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.765201+0000","last_change":"2024-03-29T10:15:18.230643+0000","last_active":"2024-03-29T10:15:18.765201+0000","last_peered":"2024-03-29T10:15:18.765201+0000","last_clean":"2024-03-29T10:15:18.765201+0000","last_became_active":"2024-03-29T10:15:18.229828+0000","last_became_peered":"2024-03-29T10:15:18.229828+0000","last_unstale":"2024-03-29T10:15:18.765201+0000","last_undegraded":"2024-03-29T10:15:18.765201+0000","last_fullsized":"2024-03-29T10:15:18.765201+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:04.101275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.769536+0000","last_change":"2024-03-29T10:15:18.767298+0000","last_active":"2024-03-29T10:15:18.769536+0000","last_peered":"2024-03-29T10:15:18.769536+0000","last_clean":"2024-03-29T10:15:18.769536+0000","last_became_active":"2024-03-29T10:15:18.766953+0000","last_became_peered":"2024-03-29T10:15:18.766953+0000","last_unstale":"2024-03-29T10:15:18.769536+0000","last_undegraded":"2024-03-29T10:15:18.769536+0000","last_fullsized":"2024-03-29T10:15:18.769536+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:57.956507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.759361+0000","last_change":"2024-03-29T10:15:18.244819+0000","last_active":"2024-03-29T10:15:18.759361+0000","last_peered":"2024-03-29T10:15:18.759361+0000","last_clean":"2024-03-29T10:15:18.759361+0000","last_became_active":"2024-03-29T10:15:18.244526+0000","last_became_peered":"2024-03-29T10:15:18.244526+0000","last_unstale":"2024-03-29T10:15:18.759361+0000","last_undegraded":"2024-03-29T10:15:18.759361+0000","last_fullsized":"2024-03-29T10:15:18.759361+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:23:52.991818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.775209+0000","last_change":"2024-03-29T10:15:18.769749+0000","last_active":"2024-03-29T10:15:18.775209+0000","last_peered":"2024-03-29T10:15:18.775209+0000","last_clean":"2024-03-29T10:15:18.775209+0000","last_became_active":"2024-03-29T10:15:18.766764+0000","last_became_peered":"2024-03-29T10:15:18.766764+0000","last_unstale":"2024-03-29T10:15:18.775209+0000","last_undegraded":"2024-03-29T10:15:18.775209+0000","last_fullsized":"2024-03-29T10:15:18.775209+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.723716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781826+0000","last_change":"2024-03-29T10:15:18.240629+0000","last_active":"2024-03-29T10:15:18.781826+0000","last_peered":"2024-03-29T10:15:18.781826+0000","last_clean":"2024-03-29T10:15:18.781826+0000","last_became_active":"2024-03-29T10:15:18.240375+0000","last_became_peered":"2024-03-29T10:15:18.240375+0000","last_unstale":"2024-03-29T10:15:18.781826+0000","last_undegraded":"2024-03-29T10:15:18.781826+0000","last_fullsized":"2024-03-29T10:15:18.781826+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:47.748627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.779007+0000","last_change":"2024-03-29T10:15:18.766950+0000","last_active":"2024-03-29T10:15:18.779007+0000","last_peered":"2024-03-29T10:15:18.779007+0000","last_clean":"2024-03-29T10:15:18.779007+0000","last_became_active":"2024-03-29T10:15:18.766669+0000","last_became_peered":"2024-03-29T10:15:18.766669+0000","last_unstale":"2024-03-29T10:15:18.779007+0000","last_undegraded":"2024-03-29T10:15:18.779007+0000","last_fullsized":"2024-03-29T10:15:18.779007+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:16:23.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.750806+0000","last_change":"2024-03-29T10:15:18.246341+0000","last_active":"2024-03-29T10:15:18.750806+0000","last_peered":"2024-03-29T10:15:18.750806+0000","last_clean":"2024-03-29T10:15:18.750806+0000","last_became_active":"2024-03-29T10:15:18.244902+0000","last_became_peered":"2024-03-29T10:15:18.244902+0000","last_unstale":"2024-03-29T10:15:18.750806+0000","last_undegraded":"2024-03-29T10:15:18.750806+0000","last_fullsized":"2024-03-29T10:15:18.750806+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:43:10.089336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.764272+0000","last_change":"2024-03-29T10:15:18.755404+0000","last_active":"2024-03-29T10:15:18.764272+0000","last_peered":"2024-03-29T10:15:18.764272+0000","last_clean":"2024-03-29T10:15:18.764272+0000","last_became_active":"2024-03-29T10:15:18.755137+0000","last_became_peered":"2024-03-29T10:15:18.755137+0000","last_unstale":"2024-03-29T10:15:18.764272+0000","last_undegraded":"2024-03-29T10:15:18.764272+0000","last_fullsized":"2024-03-29T10:15:18.764272+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:46:02.528881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.750061+0000","last_change":"2024-03-29T10:15:18.239927+0000","last_active":"2024-03-29T10:15:18.750061+0000","last_peered":"2024-03-29T10:15:18.750061+0000","last_clean":"2024-03-29T10:15:18.750061+0000","last_became_active":"2024-03-29T10:15:18.239622+0000","last_became_peered":"2024-03-29T10:15:18.239622+0000","last_unstale":"2024-03-29T10:15:18.750061+0000","last_undegraded":"2024-03-29T10:15:18.750061+0000","last_fullsized":"2024-03-29T10:15:18.750061+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:50:55.102853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781348+0000","last_change":"2024-03-29T10:15:18.778117+0000","last_active":"2024-03-29T10:15:18.781348+0000","last_peered":"2024-03-29T10:15:18.781348+0000","last_clean":"2024-03-29T10:15:18.781348+0000","last_became_active":"2024-03-29T10:15:18.777497+0000","last_became_peered":"2024-03-29T10:15:18.777497+0000","last_unstale":"2024-03-29T10:15:18.781348+0000","last_undegraded":"2024-03-29T10:15:18.781348+0000","last_fullsized":"2024-03-29T10:15:18.781348+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:39.190584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.761836+0000","last_change":"2024-03-29T10:15:18.247493+0000","last_active":"2024-03-29T10:15:18.761836+0000","last_peered":"2024-03-29T10:15:18.761836+0000","last_clean":"2024-03-29T10:15:18.761836+0000","last_became_active":"2024-03-29T10:15:18.247144+0000","last_became_peered":"2024-03-29T10:15:18.247144+0000","last_unstale":"2024-03-29T10:15:18.761836+0000","last_undegraded":"2024-03-29T10:15:18.761836+0000","last_fullsized":"2024-03-29T10:15:18.761836+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:01:05.207395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.774624+0000","last_change":"2024-03-29T10:15:18.768946+0000","last_active":"2024-03-29T10:15:18.774624+0000","last_peered":"2024-03-29T10:15:18.774624+0000","last_clean":"2024-03-29T10:15:18.774624+0000","last_became_active":"2024-03-29T10:15:18.768633+0000","last_became_peered":"2024-03-29T10:15:18.768633+0000","last_unstale":"2024-03-29T10:15:18.774624+0000","last_undegraded":"2024-03-29T10:15:18.774624+0000","last_fullsized":"2024-03-29T10:15:18.774624+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:30:45.359557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.748808+0000","last_change":"2024-03-29T10:15:18.243619+0000","last_active":"2024-03-29T10:15:18.748808+0000","last_peered":"2024-03-29T10:15:18.748808+0000","last_clean":"2024-03-29T10:15:18.748808+0000","last_became_active":"2024-03-29T10:15:18.243349+0000","last_became_peered":"2024-03-29T10:15:18.243349+0000","last_unstale":"2024-03-29T10:15:18.748808+0000","last_undegraded":"2024-03-29T10:15:18.748808+0000","last_fullsized":"2024-03-29T10:15:18.748808+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:27.193825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.765953+0000","last_change":"2024-03-29T10:15:18.760460+0000","last_active":"2024-03-29T10:15:18.765953+0000","last_peered":"2024-03-29T10:15:18.765953+0000","last_clean":"2024-03-29T10:15:18.765953+0000","last_became_active":"2024-03-29T10:15:18.760131+0000","last_became_peered":"2024-03-29T10:15:18.760131+0000","last_unstale":"2024-03-29T10:15:18.765953+0000","last_undegraded":"2024-03-29T10:15:18.765953+0000","last_fullsized":"2024-03-29T10:15:18.765953+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:59.721555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.748929+0000","last_change":"2024-03-29T10:15:18.247758+0000","last_active":"2024-03-29T10:15:18.748929+0000","last_peered":"2024-03-29T10:15:18.748929+0000","last_clean":"2024-03-29T10:15:18.748929+0000","last_became_active":"2024-03-29T10:15:18.247439+0000","last_became_peered":"2024-03-29T10:15:18.247439+0000","last_unstale":"2024-03-29T10:15:18.748929+0000","last_undegraded":"2024-03-29T10:15:18.748929+0000","last_fullsized":"2024-03-29T10:15:18.748929+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:20.864907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.765557+0000","last_change":"2024-03-29T10:15:18.763827+0000","last_active":"2024-03-29T10:15:18.765557+0000","last_peered":"2024-03-29T10:15:18.765557+0000","last_clean":"2024-03-29T10:15:18.765557+0000","last_became_active":"2024-03-29T10:15:18.763351+0000","last_became_peered":"2024-03-29T10:15:18.763351+0000","last_unstale":"2024-03-29T10:15:18.765557+0000","last_undegraded":"2024-03-29T10:15:18.765557+0000","last_fullsized":"2024-03-29T10:15:18.765557+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:48:34.366666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.764070+0000","last_change":"2024-03-29T10:15:18.239666+0000","last_active":"2024-03-29T10:15:18.764070+0000","last_peered":"2024-03-29T10:15:18.764070+0000","last_clean":"2024-03-29T10:15:18.764070+0000","last_became_active":"2024-03-29T10:15:18.236397+0000","last_became_peered":"2024-03-29T10:15:18.236397+0000","last_unstale":"2024-03-29T10:15:18.764070+0000","last_undegraded":"2024-03-29T10:15:18.764070+0000","last_fullsized":"2024-03-29T10:15:18.764070+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:57:08.748901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.764265+0000","last_change":"2024-03-29T10:15:18.757574+0000","last_active":"2024-03-29T10:15:18.764265+0000","last_peered":"2024-03-29T10:15:18.764265+0000","last_clean":"2024-03-29T10:15:18.764265+0000","last_became_active":"2024-03-29T10:15:18.757315+0000","last_became_peered":"2024-03-29T10:15:18.757315+0000","last_unstale":"2024-03-29T10:15:18.764265+0000","last_undegraded":"2024-03-29T10:15:18.764265+0000","last_fullsized":"2024-03-29T10:15:18.764265+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:58.525728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.759383+0000","last_change":"2024-03-29T10:15:18.238738+0000","last_active":"2024-03-29T10:15:18.759383+0000","last_peered":"2024-03-29T10:15:18.759383+0000","last_clean":"2024-03-29T10:15:18.759383+0000","last_became_active":"2024-03-29T10:15:18.237835+0000","last_became_peered":"2024-03-29T10:15:18.237835+0000","last_unstale":"2024-03-29T10:15:18.759383+0000","last_undegraded":"2024-03-29T10:15:18.759383+0000","last_fullsized":"2024-03-29T10:15:18.759383+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:31.906737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.759204+0000","last_change":"2024-03-29T10:15:18.755674+0000","last_active":"2024-03-29T10:15:18.759204+0000","last_peered":"2024-03-29T10:15:18.759204+0000","last_clean":"2024-03-29T10:15:18.759204+0000","last_became_active":"2024-03-29T10:15:18.755377+0000","last_became_peered":"2024-03-29T10:15:18.755377+0000","last_unstale":"2024-03-29T10:15:18.759204+0000","last_undegraded":"2024-03-29T10:15:18.759204+0000","last_fullsized":"2024-03-29T10:15:18.759204+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:48:34.366666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781938+0000","last_change":"2024-03-29T10:15:18.248847+0000","last_active":"2024-03-29T10:15:18.781938+0000","last_peered":"2024-03-29T10:15:18.781938+0000","last_clean":"2024-03-29T10:15:18.781938+0000","last_became_active":"2024-03-29T10:15:18.248547+0000","last_became_peered":"2024-03-29T10:15:18.248547+0000","last_unstale":"2024-03-29T10:15:18.781938+0000","last_undegraded":"2024-03-29T10:15:18.781938+0000","last_fullsized":"2024-03-29T10:15:18.781938+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:27.631102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"143'267","reported_seq":818,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.240146+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:15:18.240146+0000","last_peered":"2024-03-29T10:15:18.240146+0000","last_clean":"2024-03-29T10:15:18.240146+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:15:18.240146+0000","last_undegraded":"2024-03-29T10:15:18.240146+0000","last_fullsized":"2024-03-29T10:15:18.240146+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":431,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":431,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267,"ondisk_log_size":267,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575089,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37644,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334196,"statfs":{"total":96636764160,"available":96598216704,"internally_reserved":0,"allocated":2486272,"data_stored":1495968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36043212},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66500000000000004,"15min":0.69999999999999996},"min":{"1min":0.432,"5min":0.309,"15min":0.309},"max":{"1min":0.85299999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.67800000000000005,"15min":0.69899999999999995},"min":{"1min":0.502,"5min":0.41699999999999998,"15min":0.377},"max":{"1min":0.81399999999999995,"5min":1.143,"15min":1.143},"last":0.71499999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69499999999999995,"15min":0.71599999999999997},"min":{"1min":0.49099999999999999,"5min":0.45700000000000002,"15min":0.44500000000000001},"max":{"1min":0.878,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69599999999999995,"15min":0.72599999999999998},"min":{"1min":0.55400000000000005,"5min":0.46000000000000002,"15min":0.41199999999999998},"max":{"1min":0.80400000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.74299999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.70099999999999996,"15min":0.72899999999999998},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.45200000000000001},"max":{"1min":0.90200000000000002,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.72599999999999998,"15min":0.747},"min":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.48799999999999999},"max":{"1min":0.90300000000000002,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.76100000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72599999999999998,"15min":0.73099999999999998},"min":{"1min":0.49299999999999999,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.85199999999999998,"5min":1.171,"15min":1.171},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.72799999999999998,"15min":0.76300000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.89600000000000002,"5min":0.91200000000000003,"15min":1.129},"last":0.78200000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63500000000000001,"15min":0.60599999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89400000000000002,"5min":1.25,"15min":1.25},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66000000000000003,"15min":0.63900000000000001},"min":{"1min":0.46700000000000003,"5min":0.41899999999999998,"15min":0.377},"max":{"1min":0.80700000000000005,"5min":1.117,"15min":1.117},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68300000000000005,"15min":0.65500000000000003},"min":{"1min":0.443,"5min":0.43099999999999999,"15min":0.40699999999999997},"max":{"1min":1.0009999999999999,"5min":1.47,"15min":1.47},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72499999999999998,"15min":0.69199999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39400000000000002},"max":{"1min":0.95999999999999996,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.65200000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.75800000000000001,"15min":0.73399999999999999},"min":{"1min":0.52600000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.97899999999999998,"5min":1.222,"15min":1.222},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.70899999999999996},"min":{"1min":0.55100000000000005,"5min":0.46899999999999997,"15min":0.443},"max":{"1min":0.95199999999999996,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.80000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575090,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1892352,"data_stored":905600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67100000000000004,"15min":0.70099999999999996},"min":{"1min":0.441,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88700000000000001,"5min":1.006,"15min":2.7770000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66900000000000004,"15min":0.71599999999999997},"min":{"1min":0.46700000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.86499999999999999,"5min":0.93300000000000005,"15min":1.5049999999999999},"last":1.292}]},{"osd":1,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64100000000000001,"15min":0.69299999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80200000000000005,"5min":0.94799999999999995,"15min":1.6279999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68100000000000005,"15min":0.71699999999999997},"min":{"1min":0.45700000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.92800000000000005,"5min":1.0289999999999999,"15min":2.6709999999999998},"last":1.0820000000000001}]},{"osd":2,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66700000000000004,"15min":0.69999999999999996},"min":{"1min":0.46000000000000002,"5min":0.45100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81799999999999995,"5min":0.95799999999999996,"15min":1.256},"last":1.147},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70699999999999996,"15min":0.73199999999999998},"min":{"1min":0.497,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.91600000000000004,"5min":0.97099999999999997,"15min":1.454},"last":0.502}]},{"osd":3,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68500000000000005,"15min":0.72899999999999998},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.41099999999999998},"max":{"1min":0.90700000000000003,"5min":0.97899999999999998,"15min":1.131},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72099999999999997,"15min":0.76100000000000001},"min":{"1min":0.48999999999999999,"5min":0.47199999999999998,"15min":0.39000000000000001},"max":{"1min":1.0089999999999999,"5min":1.034,"15min":2.718},"last":1.048}]},{"osd":4,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57699999999999996,"15min":0.60299999999999998},"min":{"1min":0.44800000000000001,"5min":0.41299999999999998,"15min":0.35199999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":2.4780000000000002},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.65300000000000002,"15min":0.68600000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91300000000000003,"5min":0.93799999999999994,"15min":2.544},"last":1.024}]},{"osd":5,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.624,"15min":0.65100000000000002},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.34599999999999997},"max":{"1min":0.84599999999999997,"5min":0.91800000000000004,"15min":3.2280000000000002},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.48599999999999999,"5min":0.45600000000000002,"15min":0.38100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":2.835},"last":0.96499999999999997}]},{"osd":7,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.71099999999999997,"15min":0.74299999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.45100000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.6559999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.66900000000000004,"15min":0.70799999999999996},"min":{"1min":0.56100000000000005,"5min":0.45100000000000001,"15min":0.38900000000000001},"max":{"1min":0.872,"5min":0.96099999999999997,"15min":3.7170000000000001},"last":0.92700000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55112,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316728,"statfs":{"total":96636764160,"available":96580329472,"internally_reserved":0,"allocated":1892352,"data_stored":905600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.219},"max":{"1min":0.52900000000000003,"5min":0.81699999999999995,"15min":1.5600000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.502,"15min":0.50800000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67000000000000004,"5min":0.71799999999999997,"15min":1.0860000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.51900000000000002,"15min":0.53500000000000003},"min":{"1min":0.35599999999999998,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.63100000000000001,"5min":1.222,"15min":1.7869999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.91700000000000004,"5min":1.04,"15min":1.04},"last":0.45300000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56699999999999995,"15min":0.58599999999999997},"min":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.377},"max":{"1min":0.66400000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.52700000000000002,"15min":0.56000000000000005},"min":{"1min":0.375,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73299999999999998,"5min":1.026,"15min":1.3049999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.58899999999999997,"5min":0.52900000000000003,"15min":0.497},"max":{"1min":1.2070000000000001,"5min":1.2709999999999999,"15min":1.617},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.77300000000000002,"15min":0.77400000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.48299999999999998},"max":{"1min":0.98999999999999999,"5min":1.248,"15min":1.248},"last":0.55400000000000005}]},{"osd":5,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.55400000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.1160000000000001,"5min":1.157,"15min":1.3939999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.78400000000000003,"15min":0.79600000000000004},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.52900000000000003},"max":{"1min":1.016,"5min":1.3140000000000001,"15min":1.6579999999999999},"last":0.58799999999999997}]},{"osd":6,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.79200000000000004},"min":{"1min":0.55700000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80100000000000005,"15min":0.80900000000000005},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.53600000000000003},"max":{"1min":1.0920000000000001,"5min":1.319,"15min":1.468},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81599999999999995,"15min":0.83099999999999996},"min":{"1min":0.64300000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.6000000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.82299999999999995,"15min":0.82799999999999996},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.57099999999999995},"max":{"1min":1.089,"5min":1.4530000000000001,"15min":1.6919999999999999},"last":0.67300000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575090,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37624,"kb_used_data":2412,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334216,"statfs":{"total":96636764160,"available":96598237184,"internally_reserved":0,"allocated":2469888,"data_stored":1488674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.433,"15min":0.45200000000000001},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.61399999999999999,"5min":0.72299999999999998,"15min":0.93300000000000005},"last":0.435},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.27500000000000002},"max":{"1min":0.72099999999999997,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.35399999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81899999999999995,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60299999999999998,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53100000000000003,"15min":0.54600000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.33700000000000002},"max":{"1min":0.63700000000000001,"5min":0.84899999999999998,"15min":1.077},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.40400000000000003,"15min":0.39100000000000001},"max":{"1min":0.69999999999999996,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.42399999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.79000000000000004,"15min":0.79800000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.39200000000000002},"max":{"1min":0.88400000000000001,"5min":1.254,"15min":1.625},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.61799999999999999,"5min":0.60099999999999998,"15min":0.45800000000000002},"max":{"1min":0.86899999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.56000000000000005}]},{"osd":5,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.753,"15min":0.753},"min":{"1min":0.57199999999999995,"5min":0.48799999999999999,"15min":0.39000000000000001},"max":{"1min":0.872,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.75,"5min":0.78400000000000003,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.41099999999999998},"max":{"1min":1.0409999999999999,"5min":1.4970000000000001,"15min":1.5309999999999999},"last":0.751}]},{"osd":6,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.82099999999999995,"15min":0.81899999999999995},"min":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.95599999999999996,"5min":1.476,"15min":1.476},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.60199999999999998,"5min":0.5,"15min":0.45600000000000002},"max":{"1min":0.99099999999999999,"5min":1.3340000000000001,"15min":1.601},"last":0.70199999999999996}]},{"osd":7,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.78200000000000003,"15min":0.79300000000000004},"min":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.44900000000000001},"max":{"1min":0.94299999999999995,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":0.83299999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.48199999999999998},"max":{"1min":1.0149999999999999,"5min":1.296,"15min":1.5669999999999999},"last":0.72799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575089,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59204,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312636,"statfs":{"total":96636764160,"available":96576139264,"internally_reserved":0,"allocated":1892352,"data_stored":905600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.45300000000000001},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.248},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.379},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":0.41299999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.47699999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":0.48299999999999998}]},{"osd":3,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.42199999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.5},{"interface":"front","average":{"1min":0.628,"5min":0.58099999999999996,"15min":0.55800000000000005},"min":{"1min":0.40100000000000002,"5min":0.379,"15min":0.32100000000000001},"max":{"1min":2.2000000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.61099999999999999}]},{"osd":4,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.76600000000000001,"15min":0.77300000000000002},"min":{"1min":0.53600000000000003,"5min":0.51100000000000001,"15min":0.42799999999999999},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.76200000000000001,"15min":0.78300000000000003},"min":{"1min":0.53000000000000003,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.242,"5min":2.242,"15min":2.242},"last":0.77600000000000002}]},{"osd":5,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.57499999999999996,"5min":0.52400000000000002,"15min":0.46999999999999997},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.79800000000000004,"15min":0.79300000000000004},"min":{"1min":0.55800000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.0169999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.80900000000000005,"15min":0.80800000000000005},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.82299999999999995,"15min":0.82999999999999996},"min":{"1min":0.56799999999999995,"5min":0.56200000000000006,"15min":0.55000000000000004},"max":{"1min":5.2030000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.76900000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.82999999999999996,"15min":0.85099999999999998},"min":{"1min":0.63600000000000001,"5min":0.56899999999999995,"15min":0.53400000000000003},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.61699999999999999,"5min":0.60899999999999999,"15min":0.56799999999999995},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.0449999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1892352,"data_stored":905600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.44700000000000001},"min":{"1min":0.27500000000000002,"5min":0.26800000000000002,"15min":0.26500000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.81299999999999994},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.496,"15min":0.51400000000000001},"min":{"1min":0.34100000000000003,"5min":0.29899999999999999,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.874,"15min":0.874},"last":0.53800000000000003}]},{"osd":1,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44500000000000001,"15min":0.47299999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64900000000000002,"5min":0.83999999999999997,"15min":0.98699999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52900000000000003,"15min":0.55200000000000005},"min":{"1min":0.36899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.746,"5min":0.82699999999999996,"15min":0.96099999999999997},"last":0.46400000000000002}]},{"osd":2,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53800000000000003,"15min":0.55500000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68100000000000005,"5min":0.879,"15min":0.879},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58999999999999997,"15min":0.61799999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79100000000000004,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.60399999999999998}]},{"osd":4,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.71599999999999997,"15min":0.75},"min":{"1min":0.505,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96399999999999997,"5min":1.034,"15min":1.288},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.74399999999999999,"15min":0.76300000000000001},"min":{"1min":0.52600000000000002,"5min":0.504,"15min":0.45800000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.46},"last":0.67900000000000005}]},{"osd":5,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.75,"15min":0.77200000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47099999999999997},"max":{"1min":0.98799999999999999,"5min":1.1040000000000001,"15min":1.3580000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.75700000000000001,"15min":0.78900000000000003},"min":{"1min":0.54200000000000004,"5min":0.50700000000000001,"15min":0.46899999999999997},"max":{"1min":1.046,"5min":1.1479999999999999,"15min":1.429},"last":0.58799999999999997}]},{"osd":6,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.78200000000000003,"15min":0.80300000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.54200000000000004},"max":{"1min":1.0289999999999999,"5min":1.123,"15min":1.4730000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.75800000000000001,"15min":0.79700000000000004},"min":{"1min":0.52500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.3999999999999999},"last":0.81100000000000005}]},{"osd":7,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.77600000000000002,"15min":0.81799999999999995},"min":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93500000000000005,"5min":1.103,"15min":1.5029999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.087,"5min":1.1859999999999999,"15min":1.345},"last":0.84099999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575090,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59192,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312648,"statfs":{"total":96636764160,"available":96576151552,"internally_reserved":0,"allocated":1880064,"data_stored":899325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67800000000000005,"15min":0.68500000000000005},"min":{"1min":0.45000000000000001,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.86099999999999999,"5min":0.98799999999999999,"15min":1.417},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.40300000000000002,"5min":0.38,"15min":0.34000000000000002},"max":{"1min":0.88300000000000001,"5min":1.0269999999999999,"15min":1.165},"last":0.57299999999999995}]},{"osd":1,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68000000000000005,"15min":0.69699999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39000000000000001},"max":{"1min":0.872,"5min":0.97499999999999998,"15min":0.98399999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69299999999999995,"15min":0.70399999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"max":{"1min":0.91300000000000003,"5min":1.075,"15min":1.333},"last":0.65900000000000003}]},{"osd":2,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75800000000000001,"15min":0.77100000000000002},"min":{"1min":0.57599999999999996,"5min":0.433,"15min":0.433},"max":{"1min":0.87,"5min":1.014,"15min":1.03},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76900000000000002,"15min":0.76800000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.94599999999999995,"5min":1.0600000000000001,"15min":1.387},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70499999999999996,"15min":0.71899999999999997},"min":{"1min":0.48299999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.89300000000000002,"5min":0.97499999999999998,"15min":1.2350000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70999999999999996,"15min":0.72799999999999998},"min":{"1min":0.503,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.80700000000000005,"5min":1.0600000000000001,"15min":1.3180000000000001},"last":0.48899999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.61199999999999999,"15min":0.58899999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38100000000000001},"max":{"1min":0.77100000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.63100000000000001,"15min":0.60699999999999998},"min":{"1min":0.41099999999999998,"5min":0.40200000000000002,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.98599999999999999,"15min":1.294},"last":0.51800000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65300000000000002,"15min":0.624},"min":{"1min":0.46600000000000003,"5min":0.44,"15min":0.375},"max":{"1min":0.90000000000000002,"5min":1.026,"15min":1.026},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68400000000000005,"15min":0.66200000000000003},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.88400000000000001,"5min":1.093,"15min":1.1359999999999999},"last":0.54000000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69799999999999995,"15min":0.68500000000000005},"min":{"1min":0.48199999999999998,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":0.85799999999999998,"5min":1.0580000000000001,"15min":1.345},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78100000000000003,"15min":0.76200000000000001},"min":{"1min":0.53500000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.93500000000000005,"5min":1.014,"15min":1.214},"last":0.73199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1892352,"data_stored":905600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63300000000000001,"15min":0.67300000000000004},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.35699999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":1.1899999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.38100000000000001,"15min":0.33500000000000002},"max":{"1min":0.83899999999999997,"5min":0.997,"15min":1.2649999999999999},"last":0.73599999999999999}]},{"osd":1,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.71099999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96299999999999997,"5min":0.96599999999999997,"15min":0.96699999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64700000000000002,"15min":0.69399999999999995},"min":{"1min":0.46899999999999997,"5min":0.39900000000000002,"15min":0.39200000000000002},"max":{"1min":0.79500000000000004,"5min":0.94199999999999995,"15min":1.0669999999999999},"last":0.84899999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69099999999999995,"15min":0.72599999999999998},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.92100000000000004,"5min":0.98599999999999999,"15min":0.98899999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.69999999999999996},"min":{"1min":0.498,"5min":0.46999999999999997,"15min":0.39700000000000002},"max":{"1min":0.90200000000000002,"5min":0.96199999999999997,"15min":1.095},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":0.72099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.113},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.746},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0980000000000001},"last":0.755}]},{"osd":4,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60299999999999998},"min":{"1min":0.40699999999999997,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":0.89200000000000002,"5min":0.91700000000000004,"15min":0.94599999999999995},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65700000000000003,"15min":0.64700000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.38900000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.097},"last":0.64100000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65000000000000002,"15min":0.65400000000000003},"min":{"1min":0.433,"5min":0.41299999999999998,"15min":0.36799999999999999},"max":{"1min":1.083,"5min":1.083,"15min":1.151},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.67400000000000004,"15min":0.67900000000000005},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.77300000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70099999999999996,"15min":0.70599999999999996},"min":{"1min":0.49299999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.42299999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.1990000000000001},"last":0.89100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:15:23.677 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T10:15:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.914+0000 7fd4e1600700 1 Processor -- start 2024-03-29T10:15:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.914+0000 7fd4e1600700 1 -- start start 2024-03-29T10:15:23.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.915+0000 7fd4e1600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd4dc0afbe0 0x7fd4dc06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.915+0000 7fd4e1600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.915+0000 7fd4e1600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc06f850 con 0x7fd4dc06a790 2024-03-29T10:15:23.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.915+0000 7fd4e1600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc06fad0 con 0x7fd4dc0afbe0 2024-03-29T10:15:23.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.915+0000 7fd4e1600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0af970 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4da000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd4dc0afbe0 0x7fd4dc06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4db400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4db400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:44142/0 (socket says 172.21.15.12:44142) 2024-03-29T10:15:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4db400700 1 -- 172.21.15.12:0/4087380368 learned_addr learned my addr 172.21.15.12:0/4087380368 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4da000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd4dc0afbe0 0x7fd4dc06dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:50520/0 (socket says 172.21.15.12:50520) 2024-03-29T10:15:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4daa00700 1 --1- 172.21.15.12:0/4087380368 >> v1:172.21.15.159:6789/0 conn(0x7fd4dc06a790 0x7fd4dc06ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:54218/0 (socket says 172.21.15.12:54218) 2024-03-29T10:15:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4d9600700 1 -- 172.21.15.12:0/4087380368 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107301595 0 0) 0x7fd4dc06f850 con 0x7fd4dc06a790 2024-03-29T10:15:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4d9600700 1 -- 172.21.15.12:0/4087380368 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c00035b0 con 0x7fd4dc06a790 2024-03-29T10:15:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4db400700 1 -- 172.21.15.12:0/4087380368 >> v2:172.21.15.159:3300/0 conn(0x7fd4dc0afbe0 msgr2=0x7fd4dc06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4db400700 1 --2- 172.21.15.12:0/4087380368 >> v2:172.21.15.159:3300/0 conn(0x7fd4dc0afbe0 0x7fd4dc06dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4db400700 1 -- 172.21.15.12:0/4087380368 >> v1:172.21.15.159:6789/0 conn(0x7fd4dc06a790 legacy=0x7fd4dc06ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4db400700 1 -- 172.21.15.12:0/4087380368 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc0af690 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.916+0000 7fd4da000700 1 --2- 172.21.15.12:0/4087380368 >> v2:172.21.15.159:3300/0 conn(0x7fd4dc0afbe0 0x7fd4dc06dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.917+0000 7fd4db400700 1 --2- 172.21.15.12:0/4087380368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cf040 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009e50 tx=0x7fd4d400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7216a7d2126eb5f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.917+0000 7fd4d9600700 1 -- 172.21.15.12:0/4087380368 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d400e070 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.917+0000 7fd4d9600700 1 -- 172.21.15.12:0/4087380368 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d400b910 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.917+0000 7fd4d9600700 1 -- 172.21.15.12:0/4087380368 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d400bc80 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.917+0000 7fd4e1600700 1 -- 172.21.15.12:0/4087380368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0cebd0 msgr2=0x7fd4dc0cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.917+0000 7fd4e1600700 1 --2- 172.21.15.12:0/4087380368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cf040 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009e50 tx=0x7fd4d400b330 comp rx=0 tx=0).stop 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.917+0000 7fd4e1600700 1 -- 172.21.15.12:0/4087380368 shutdown_connections 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.917+0000 7fd4e1600700 1 --2- 172.21.15.12:0/4087380368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cf040 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.917+0000 7fd4e1600700 1 --2- 172.21.15.12:0/4087380368 >> v2:172.21.15.159:3300/0 conn(0x7fd4dc0afbe0 0x7fd4dc06dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.917+0000 7fd4e1600700 1 -- 172.21.15.12:0/4087380368 >> 172.21.15.12:0/4087380368 conn(0x7fd4dc0b5ab0 msgr2=0x7fd4dc0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:23.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.918+0000 7fd4e1600700 1 -- 172.21.15.12:0/4087380368 shutdown_connections 2024-03-29T10:15:23.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.918+0000 7fd4e1600700 1 -- 172.21.15.12:0/4087380368 wait complete. 2024-03-29T10:15:23.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.919+0000 7fd4e1600700 1 Processor -- start 2024-03-29T10:15:23.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.919+0000 7fd4e1600700 1 -- start start 2024-03-29T10:15:23.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.920+0000 7fd4e1600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4dc06a790 0x7fd4dc0c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.920+0000 7fd4e1600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0afbe0 0x7fd4dc1f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.920+0000 7fd4e1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.920+0000 7fd4e1600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0cac30 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.920+0000 7fd4e1600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0ca020 con 0x7fd4dc06a790 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.920+0000 7fd4e1600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0ca300 con 0x7fd4dc0afbe0 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49892/0 (socket says 172.21.15.12:49892) 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4db400700 1 -- 172.21.15.12:0/2829994693 learned_addr learned my addr 172.21.15.12:0/2829994693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4da000700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0afbe0 0x7fd4dc1f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4daa00700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4dc06a790 0x7fd4dc0c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4db400700 1 -- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0afbe0 msgr2=0x7fd4dc1f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4db400700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0afbe0 0x7fd4dc1f6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4db400700 1 -- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4dc06a790 msgr2=0x7fd4dc0c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4db400700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4dc06a790 0x7fd4dc0c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4db400700 1 -- 172.21.15.12:0/2829994693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c80094a0 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.921+0000 7fd4db400700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cb460 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400b370 tx=0x7fd4d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.922+0000 7fd4d2a00700 1 -- 172.21.15.12:0/2829994693 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4004450 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.922+0000 7fd4d2a00700 1 -- 172.21.15.12:0/2829994693 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d40045d0 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.922+0000 7fd4d2a00700 1 -- 172.21.15.12:0/2829994693 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d401c440 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.922+0000 7fd4e1600700 1 -- 172.21.15.12:0/2829994693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d4009b30 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.922+0000 7fd4e1600700 1 -- 172.21.15.12:0/2829994693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc0ca8c0 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.923+0000 7fd4e1600700 1 -- 172.21.15.12:0/2829994693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc0cada0 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.924+0000 7fd4d2a00700 1 -- 172.21.15.12:0/2829994693 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd4d401c5a0 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.925+0000 7fd4d2a00700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4bc06adb0 0x7fd4bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.925+0000 7fd4d2a00700 1 -- 172.21.15.12:0/2829994693 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4d408ae30 con 0x7fd4dc0cebd0 2024-03-29T10:15:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.925+0000 7fd4daa00700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4bc06adb0 0x7fd4bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.926+0000 7fd4daa00700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4bc06adb0 0x7fd4bc06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc007fd0 tx=0x7fd4cc007dc0 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:23.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:23.929+0000 7fd4d2a00700 1 -- 172.21.15.12:0/2829994693 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd4d4055de0 con 0x7fd4dc0cebd0 2024-03-29T10:15:24.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.137+0000 7fd4e1600700 1 -- 172.21.15.12:0/2829994693 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd4dc0cf580 con 0x7fd4bc06adb0 2024-03-29T10:15:24.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.146+0000 7fd4d2a00700 1 -- 172.21.15.12:0/2829994693 <== mgr.8425 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289563 (secure 0 0 0) 0x7fd4cc0012d0 con 0x7fd4bc06adb0 2024-03-29T10:15:24.147 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.150+0000 7fd4e1600700 1 -- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4bc06adb0 msgr2=0x7fd4bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4bc06adb0 0x7fd4bc06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc007fd0 tx=0x7fd4cc007dc0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 -- 172.21.15.12:0/2829994693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4dc0cebd0 msgr2=0x7fd4dc0cb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cb460 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400b370 tx=0x7fd4d4004ad0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 -- 172.21.15.12:0/2829994693 shutdown_connections 2024-03-29T10:15:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd4bc06adb0 0x7fd4bc06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4dc0cebd0 0x7fd4dc0cb460 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4dc06a790 0x7fd4dc0c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 --2- 172.21.15.12:0/2829994693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4dc0afbe0 0x7fd4dc1f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 -- 172.21.15.12:0/2829994693 >> 172.21.15.12:0/2829994693 conn(0x7fd4dc0b5ab0 msgr2=0x7fd4dc15b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 -- 172.21.15.12:0/2829994693 shutdown_connections 2024-03-29T10:15:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.151+0000 7fd4e1600700 1 -- 172.21.15.12:0/2829994693 wait complete. 2024-03-29T10:15:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T10:15:24.166 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-29T10:15:23.896647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":475,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":360184,"kb_used_data":16116,"kb_used_omap":12,"kb_used_meta":343923,"kb_avail":754614536,"statfs":{"total":773094113280,"available":772725284864,"internally_reserved":0,"allocated":16502784,"data_stored":8505377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":352177721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001580"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891048+0000","last_change":"2024-03-29T10:15:18.774403+0000","last_active":"2024-03-29T10:15:18.891048+0000","last_peered":"2024-03-29T10:15:18.891048+0000","last_clean":"2024-03-29T10:15:18.891048+0000","last_became_active":"2024-03-29T10:15:18.774039+0000","last_became_peered":"2024-03-29T10:15:18.774039+0000","last_unstale":"2024-03-29T10:15:18.891048+0000","last_undegraded":"2024-03-29T10:15:18.891048+0000","last_fullsized":"2024-03-29T10:15:18.891048+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:21.873857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890263+0000","last_change":"2024-03-29T10:15:18.768481+0000","last_active":"2024-03-29T10:15:18.890263+0000","last_peered":"2024-03-29T10:15:18.890263+0000","last_clean":"2024-03-29T10:15:18.890263+0000","last_became_active":"2024-03-29T10:15:18.767948+0000","last_became_peered":"2024-03-29T10:15:18.767948+0000","last_unstale":"2024-03-29T10:15:18.890263+0000","last_undegraded":"2024-03-29T10:15:18.890263+0000","last_fullsized":"2024-03-29T10:15:18.890263+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:16:20.679135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894631+0000","last_change":"2024-03-29T10:15:18.772019+0000","last_active":"2024-03-29T10:15:18.894631+0000","last_peered":"2024-03-29T10:15:18.894631+0000","last_clean":"2024-03-29T10:15:18.894631+0000","last_became_active":"2024-03-29T10:15:18.771640+0000","last_became_peered":"2024-03-29T10:15:18.771640+0000","last_unstale":"2024-03-29T10:15:18.894631+0000","last_undegraded":"2024-03-29T10:15:18.894631+0000","last_fullsized":"2024-03-29T10:15:18.894631+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:57:09.757851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891021+0000","last_change":"2024-03-29T10:15:18.769115+0000","last_active":"2024-03-29T10:15:18.891021+0000","last_peered":"2024-03-29T10:15:18.891021+0000","last_clean":"2024-03-29T10:15:18.891021+0000","last_became_active":"2024-03-29T10:15:18.768846+0000","last_became_peered":"2024-03-29T10:15:18.768846+0000","last_unstale":"2024-03-29T10:15:18.891021+0000","last_undegraded":"2024-03-29T10:15:18.891021+0000","last_fullsized":"2024-03-29T10:15:18.891021+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:46:02.528881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.766857+0000","last_change":"2024-03-29T10:15:18.762901+0000","last_active":"2024-03-29T10:15:18.766857+0000","last_peered":"2024-03-29T10:15:18.766857+0000","last_clean":"2024-03-29T10:15:18.766857+0000","last_became_active":"2024-03-29T10:15:18.762289+0000","last_became_peered":"2024-03-29T10:15:18.762289+0000","last_unstale":"2024-03-29T10:15:18.766857+0000","last_undegraded":"2024-03-29T10:15:18.766857+0000","last_fullsized":"2024-03-29T10:15:18.766857+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:19:18.684392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894166+0000","last_change":"2024-03-29T10:15:18.765267+0000","last_active":"2024-03-29T10:15:18.894166+0000","last_peered":"2024-03-29T10:15:18.894166+0000","last_clean":"2024-03-29T10:15:18.894166+0000","last_became_active":"2024-03-29T10:15:18.765004+0000","last_became_peered":"2024-03-29T10:15:18.765004+0000","last_unstale":"2024-03-29T10:15:18.894166+0000","last_undegraded":"2024-03-29T10:15:18.894166+0000","last_fullsized":"2024-03-29T10:15:18.894166+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:20:23.230817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894147+0000","last_change":"2024-03-29T10:15:18.767667+0000","last_active":"2024-03-29T10:15:18.894147+0000","last_peered":"2024-03-29T10:15:18.894147+0000","last_clean":"2024-03-29T10:15:18.894147+0000","last_became_active":"2024-03-29T10:15:18.767341+0000","last_became_peered":"2024-03-29T10:15:18.767341+0000","last_unstale":"2024-03-29T10:15:18.894147+0000","last_undegraded":"2024-03-29T10:15:18.894147+0000","last_fullsized":"2024-03-29T10:15:18.894147+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:43.862113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894568+0000","last_change":"2024-03-29T10:15:18.771632+0000","last_active":"2024-03-29T10:15:18.894568+0000","last_peered":"2024-03-29T10:15:18.894568+0000","last_clean":"2024-03-29T10:15:18.894568+0000","last_became_active":"2024-03-29T10:15:18.769530+0000","last_became_peered":"2024-03-29T10:15:18.769530+0000","last_unstale":"2024-03-29T10:15:18.894568+0000","last_undegraded":"2024-03-29T10:15:18.894568+0000","last_fullsized":"2024-03-29T10:15:18.894568+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:15:05.102932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894658+0000","last_change":"2024-03-29T10:15:18.763137+0000","last_active":"2024-03-29T10:15:18.894658+0000","last_peered":"2024-03-29T10:15:18.894658+0000","last_clean":"2024-03-29T10:15:18.894658+0000","last_became_active":"2024-03-29T10:15:18.762489+0000","last_became_peered":"2024-03-29T10:15:18.762489+0000","last_unstale":"2024-03-29T10:15:18.894658+0000","last_undegraded":"2024-03-29T10:15:18.894658+0000","last_fullsized":"2024-03-29T10:15:18.894658+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:39.106716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890273+0000","last_change":"2024-03-29T10:15:18.768244+0000","last_active":"2024-03-29T10:15:18.890273+0000","last_peered":"2024-03-29T10:15:18.890273+0000","last_clean":"2024-03-29T10:15:18.890273+0000","last_became_active":"2024-03-29T10:15:18.767581+0000","last_became_peered":"2024-03-29T10:15:18.767581+0000","last_unstale":"2024-03-29T10:15:18.890273+0000","last_undegraded":"2024-03-29T10:15:18.890273+0000","last_fullsized":"2024-03-29T10:15:18.890273+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:47.766149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894801+0000","last_change":"2024-03-29T10:15:18.770356+0000","last_active":"2024-03-29T10:15:18.894801+0000","last_peered":"2024-03-29T10:15:18.894801+0000","last_clean":"2024-03-29T10:15:18.894801+0000","last_became_active":"2024-03-29T10:15:18.770038+0000","last_became_peered":"2024-03-29T10:15:18.770038+0000","last_unstale":"2024-03-29T10:15:18.894801+0000","last_undegraded":"2024-03-29T10:15:18.894801+0000","last_fullsized":"2024-03-29T10:15:18.894801+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:49:11.173864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894612+0000","last_change":"2024-03-29T10:15:18.763296+0000","last_active":"2024-03-29T10:15:18.894612+0000","last_peered":"2024-03-29T10:15:18.894612+0000","last_clean":"2024-03-29T10:15:18.894612+0000","last_became_active":"2024-03-29T10:15:18.763012+0000","last_became_peered":"2024-03-29T10:15:18.763012+0000","last_unstale":"2024-03-29T10:15:18.894612+0000","last_undegraded":"2024-03-29T10:15:18.894612+0000","last_fullsized":"2024-03-29T10:15:18.894612+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:43:11.098286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891010+0000","last_change":"2024-03-29T10:15:18.774337+0000","last_active":"2024-03-29T10:15:18.891010+0000","last_peered":"2024-03-29T10:15:18.891010+0000","last_clean":"2024-03-29T10:15:18.891010+0000","last_became_active":"2024-03-29T10:15:18.774090+0000","last_became_peered":"2024-03-29T10:15:18.774090+0000","last_unstale":"2024-03-29T10:15:18.891010+0000","last_undegraded":"2024-03-29T10:15:18.891010+0000","last_fullsized":"2024-03-29T10:15:18.891010+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:28.640052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890188+0000","last_change":"2024-03-29T10:15:18.754459+0000","last_active":"2024-03-29T10:15:18.890188+0000","last_peered":"2024-03-29T10:15:18.890188+0000","last_clean":"2024-03-29T10:15:18.890188+0000","last_became_active":"2024-03-29T10:15:18.754056+0000","last_became_peered":"2024-03-29T10:15:18.754056+0000","last_unstale":"2024-03-29T10:15:18.890188+0000","last_undegraded":"2024-03-29T10:15:18.890188+0000","last_fullsized":"2024-03-29T10:15:18.890188+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:13.983592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890947+0000","last_change":"2024-03-29T10:15:18.769932+0000","last_active":"2024-03-29T10:15:18.890947+0000","last_peered":"2024-03-29T10:15:18.890947+0000","last_clean":"2024-03-29T10:15:18.890947+0000","last_became_active":"2024-03-29T10:15:18.769580+0000","last_became_peered":"2024-03-29T10:15:18.769580+0000","last_unstale":"2024-03-29T10:15:18.890947+0000","last_undegraded":"2024-03-29T10:15:18.890947+0000","last_fullsized":"2024-03-29T10:15:18.890947+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:16:20.679135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891186+0000","last_change":"2024-03-29T10:15:18.768079+0000","last_active":"2024-03-29T10:15:18.891186+0000","last_peered":"2024-03-29T10:15:18.891186+0000","last_clean":"2024-03-29T10:15:18.891186+0000","last_became_active":"2024-03-29T10:15:18.767611+0000","last_became_peered":"2024-03-29T10:15:18.767611+0000","last_unstale":"2024-03-29T10:15:18.891186+0000","last_undegraded":"2024-03-29T10:15:18.891186+0000","last_fullsized":"2024-03-29T10:15:18.891186+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:20:58.998966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894240+0000","last_change":"2024-03-29T10:15:18.768939+0000","last_active":"2024-03-29T10:15:18.894240+0000","last_peered":"2024-03-29T10:15:18.894240+0000","last_clean":"2024-03-29T10:15:18.894240+0000","last_became_active":"2024-03-29T10:15:18.768669+0000","last_became_peered":"2024-03-29T10:15:18.768669+0000","last_unstale":"2024-03-29T10:15:18.894240+0000","last_undegraded":"2024-03-29T10:15:18.894240+0000","last_fullsized":"2024-03-29T10:15:18.894240+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:28:12.237242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891413+0000","last_change":"2024-03-29T10:15:18.772577+0000","last_active":"2024-03-29T10:15:18.891413+0000","last_peered":"2024-03-29T10:15:18.891413+0000","last_clean":"2024-03-29T10:15:18.891413+0000","last_became_active":"2024-03-29T10:15:18.771978+0000","last_became_peered":"2024-03-29T10:15:18.771978+0000","last_unstale":"2024-03-29T10:15:18.891413+0000","last_undegraded":"2024-03-29T10:15:18.891413+0000","last_fullsized":"2024-03-29T10:15:18.891413+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:38.270423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894810+0000","last_change":"2024-03-29T10:15:18.769713+0000","last_active":"2024-03-29T10:15:18.894810+0000","last_peered":"2024-03-29T10:15:18.894810+0000","last_clean":"2024-03-29T10:15:18.894810+0000","last_became_active":"2024-03-29T10:15:18.769435+0000","last_became_peered":"2024-03-29T10:15:18.769435+0000","last_unstale":"2024-03-29T10:15:18.894810+0000","last_undegraded":"2024-03-29T10:15:18.894810+0000","last_fullsized":"2024-03-29T10:15:18.894810+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:06:04.148022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891375+0000","last_change":"2024-03-29T10:15:18.767477+0000","last_active":"2024-03-29T10:15:18.891375+0000","last_peered":"2024-03-29T10:15:18.891375+0000","last_clean":"2024-03-29T10:15:18.891375+0000","last_became_active":"2024-03-29T10:15:18.767203+0000","last_became_peered":"2024-03-29T10:15:18.767203+0000","last_unstale":"2024-03-29T10:15:18.891375+0000","last_undegraded":"2024-03-29T10:15:18.891375+0000","last_fullsized":"2024-03-29T10:15:18.891375+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:01.591227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.764678+0000","last_change":"2024-03-29T10:15:18.761179+0000","last_active":"2024-03-29T10:15:18.764678+0000","last_peered":"2024-03-29T10:15:18.764678+0000","last_clean":"2024-03-29T10:15:18.764678+0000","last_became_active":"2024-03-29T10:15:18.760891+0000","last_became_peered":"2024-03-29T10:15:18.760891+0000","last_unstale":"2024-03-29T10:15:18.764678+0000","last_undegraded":"2024-03-29T10:15:18.764678+0000","last_fullsized":"2024-03-29T10:15:18.764678+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:27:53.728925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894783+0000","last_change":"2024-03-29T10:15:18.767659+0000","last_active":"2024-03-29T10:15:18.894783+0000","last_peered":"2024-03-29T10:15:18.894783+0000","last_clean":"2024-03-29T10:15:18.894783+0000","last_became_active":"2024-03-29T10:15:18.767362+0000","last_became_peered":"2024-03-29T10:15:18.767362+0000","last_unstale":"2024-03-29T10:15:18.894783+0000","last_undegraded":"2024-03-29T10:15:18.894783+0000","last_fullsized":"2024-03-29T10:15:18.894783+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:49.750852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894588+0000","last_change":"2024-03-29T10:15:18.765029+0000","last_active":"2024-03-29T10:15:18.894588+0000","last_peered":"2024-03-29T10:15:18.894588+0000","last_clean":"2024-03-29T10:15:18.894588+0000","last_became_active":"2024-03-29T10:15:18.763526+0000","last_became_peered":"2024-03-29T10:15:18.763526+0000","last_unstale":"2024-03-29T10:15:18.894588+0000","last_undegraded":"2024-03-29T10:15:18.894588+0000","last_fullsized":"2024-03-29T10:15:18.894588+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:21:42.891275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894750+0000","last_change":"2024-03-29T10:15:18.764827+0000","last_active":"2024-03-29T10:15:18.894750+0000","last_peered":"2024-03-29T10:15:18.894750+0000","last_clean":"2024-03-29T10:15:18.894750+0000","last_became_active":"2024-03-29T10:15:18.762293+0000","last_became_peered":"2024-03-29T10:15:18.762293+0000","last_unstale":"2024-03-29T10:15:18.894750+0000","last_undegraded":"2024-03-29T10:15:18.894750+0000","last_fullsized":"2024-03-29T10:15:18.894750+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:01:22.602449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890282+0000","last_change":"2024-03-29T10:15:18.755725+0000","last_active":"2024-03-29T10:15:18.890282+0000","last_peered":"2024-03-29T10:15:18.890282+0000","last_clean":"2024-03-29T10:15:18.890282+0000","last_became_active":"2024-03-29T10:15:18.755425+0000","last_became_peered":"2024-03-29T10:15:18.755425+0000","last_unstale":"2024-03-29T10:15:18.890282+0000","last_undegraded":"2024-03-29T10:15:18.890282+0000","last_fullsized":"2024-03-29T10:15:18.890282+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:59.721555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891455+0000","last_change":"2024-03-29T10:15:18.775040+0000","last_active":"2024-03-29T10:15:18.891455+0000","last_peered":"2024-03-29T10:15:18.891455+0000","last_clean":"2024-03-29T10:15:18.891455+0000","last_became_active":"2024-03-29T10:15:18.774772+0000","last_became_peered":"2024-03-29T10:15:18.774772+0000","last_unstale":"2024-03-29T10:15:18.891455+0000","last_undegraded":"2024-03-29T10:15:18.891455+0000","last_fullsized":"2024-03-29T10:15:18.891455+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:04:09.775205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891030+0000","last_change":"2024-03-29T10:15:18.771982+0000","last_active":"2024-03-29T10:15:18.891030+0000","last_peered":"2024-03-29T10:15:18.891030+0000","last_clean":"2024-03-29T10:15:18.891030+0000","last_became_active":"2024-03-29T10:15:18.771720+0000","last_became_peered":"2024-03-29T10:15:18.771720+0000","last_unstale":"2024-03-29T10:15:18.891030+0000","last_undegraded":"2024-03-29T10:15:18.891030+0000","last_fullsized":"2024-03-29T10:15:18.891030+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:11.882236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.772151+0000","last_change":"2024-03-29T10:15:18.767757+0000","last_active":"2024-03-29T10:15:18.772151+0000","last_peered":"2024-03-29T10:15:18.772151+0000","last_clean":"2024-03-29T10:15:18.772151+0000","last_became_active":"2024-03-29T10:15:18.766764+0000","last_became_peered":"2024-03-29T10:15:18.766764+0000","last_unstale":"2024-03-29T10:15:18.772151+0000","last_undegraded":"2024-03-29T10:15:18.772151+0000","last_fullsized":"2024-03-29T10:15:18.772151+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:40:36.977077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890991+0000","last_change":"2024-03-29T10:15:18.772644+0000","last_active":"2024-03-29T10:15:18.890991+0000","last_peered":"2024-03-29T10:15:18.890991+0000","last_clean":"2024-03-29T10:15:18.890991+0000","last_became_active":"2024-03-29T10:15:18.770500+0000","last_became_peered":"2024-03-29T10:15:18.770500+0000","last_unstale":"2024-03-29T10:15:18.890991+0000","last_undegraded":"2024-03-29T10:15:18.890991+0000","last_fullsized":"2024-03-29T10:15:18.890991+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:58.525728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890972+0000","last_change":"2024-03-29T10:15:18.767932+0000","last_active":"2024-03-29T10:15:18.890972+0000","last_peered":"2024-03-29T10:15:18.890972+0000","last_clean":"2024-03-29T10:15:18.890972+0000","last_became_active":"2024-03-29T10:15:18.767630+0000","last_became_peered":"2024-03-29T10:15:18.767630+0000","last_unstale":"2024-03-29T10:15:18.890972+0000","last_undegraded":"2024-03-29T10:15:18.890972+0000","last_fullsized":"2024-03-29T10:15:18.890972+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:47.766149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.766580+0000","last_change":"2024-03-29T10:15:18.763321+0000","last_active":"2024-03-29T10:15:18.766580+0000","last_peered":"2024-03-29T10:15:18.766580+0000","last_clean":"2024-03-29T10:15:18.766580+0000","last_became_active":"2024-03-29T10:15:18.761584+0000","last_became_peered":"2024-03-29T10:15:18.761584+0000","last_unstale":"2024-03-29T10:15:18.766580+0000","last_undegraded":"2024-03-29T10:15:18.766580+0000","last_fullsized":"2024-03-29T10:15:18.766580+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:02.351513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894133+0000","last_change":"2024-03-29T10:15:18.769126+0000","last_active":"2024-03-29T10:15:18.894133+0000","last_peered":"2024-03-29T10:15:18.894133+0000","last_clean":"2024-03-29T10:15:18.894133+0000","last_became_active":"2024-03-29T10:15:18.768786+0000","last_became_peered":"2024-03-29T10:15:18.768786+0000","last_unstale":"2024-03-29T10:15:18.894133+0000","last_undegraded":"2024-03-29T10:15:18.894133+0000","last_fullsized":"2024-03-29T10:15:18.894133+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:05.110226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891434+0000","last_change":"2024-03-29T10:15:18.771968+0000","last_active":"2024-03-29T10:15:18.891434+0000","last_peered":"2024-03-29T10:15:18.891434+0000","last_clean":"2024-03-29T10:15:18.891434+0000","last_became_active":"2024-03-29T10:15:18.768925+0000","last_became_peered":"2024-03-29T10:15:18.768925+0000","last_unstale":"2024-03-29T10:15:18.891434+0000","last_undegraded":"2024-03-29T10:15:18.891434+0000","last_fullsized":"2024-03-29T10:15:18.891434+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:26:23.283920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"153'4","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781736+0000","last_change":"2024-03-29T10:15:18.248449+0000","last_active":"2024-03-29T10:15:18.781736+0000","last_peered":"2024-03-29T10:15:18.781736+0000","last_clean":"2024-03-29T10:15:18.781736+0000","last_became_active":"2024-03-29T10:15:18.248092+0000","last_became_peered":"2024-03-29T10:15:18.248092+0000","last_unstale":"2024-03-29T10:15:18.781736+0000","last_undegraded":"2024-03-29T10:15:18.781736+0000","last_fullsized":"2024-03-29T10:15:18.781736+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:16:14.096536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894681+0000","last_change":"2024-03-29T10:15:18.763323+0000","last_active":"2024-03-29T10:15:18.894681+0000","last_peered":"2024-03-29T10:15:18.894681+0000","last_clean":"2024-03-29T10:15:18.894681+0000","last_became_active":"2024-03-29T10:15:18.762867+0000","last_became_peered":"2024-03-29T10:15:18.762867+0000","last_unstale":"2024-03-29T10:15:18.894681+0000","last_undegraded":"2024-03-29T10:15:18.894681+0000","last_fullsized":"2024-03-29T10:15:18.894681+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:05.110226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.780997+0000","last_change":"2024-03-29T10:15:18.245102+0000","last_active":"2024-03-29T10:15:18.780997+0000","last_peered":"2024-03-29T10:15:18.780997+0000","last_clean":"2024-03-29T10:15:18.780997+0000","last_became_active":"2024-03-29T10:15:18.244719+0000","last_became_peered":"2024-03-29T10:15:18.244719+0000","last_unstale":"2024-03-29T10:15:18.780997+0000","last_undegraded":"2024-03-29T10:15:18.780997+0000","last_fullsized":"2024-03-29T10:15:18.780997+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:02:37.063807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.774718+0000","last_change":"2024-03-29T10:15:18.768939+0000","last_active":"2024-03-29T10:15:18.774718+0000","last_peered":"2024-03-29T10:15:18.774718+0000","last_clean":"2024-03-29T10:15:18.774718+0000","last_became_active":"2024-03-29T10:15:18.767916+0000","last_became_peered":"2024-03-29T10:15:18.767916+0000","last_unstale":"2024-03-29T10:15:18.774718+0000","last_undegraded":"2024-03-29T10:15:18.774718+0000","last_fullsized":"2024-03-29T10:15:18.774718+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:39:20.841112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"153'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894821+0000","last_change":"2024-03-29T10:15:18.243489+0000","last_active":"2024-03-29T10:15:18.894821+0000","last_peered":"2024-03-29T10:15:18.894821+0000","last_clean":"2024-03-29T10:15:18.894821+0000","last_became_active":"2024-03-29T10:15:18.243234+0000","last_became_peered":"2024-03-29T10:15:18.243234+0000","last_unstale":"2024-03-29T10:15:18.894821+0000","last_undegraded":"2024-03-29T10:15:18.894821+0000","last_fullsized":"2024-03-29T10:15:18.894821+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:17:10.429998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.766553+0000","last_change":"2024-03-29T10:15:18.763009+0000","last_active":"2024-03-29T10:15:18.766553+0000","last_peered":"2024-03-29T10:15:18.766553+0000","last_clean":"2024-03-29T10:15:18.766553+0000","last_became_active":"2024-03-29T10:15:18.761992+0000","last_became_peered":"2024-03-29T10:15:18.761992+0000","last_unstale":"2024-03-29T10:15:18.766553+0000","last_undegraded":"2024-03-29T10:15:18.766553+0000","last_fullsized":"2024-03-29T10:15:18.766553+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:03:07.307244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"153'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890233+0000","last_change":"2024-03-29T10:15:18.246698+0000","last_active":"2024-03-29T10:15:18.890233+0000","last_peered":"2024-03-29T10:15:18.890233+0000","last_clean":"2024-03-29T10:15:18.890233+0000","last_became_active":"2024-03-29T10:15:18.245609+0000","last_became_peered":"2024-03-29T10:15:18.245609+0000","last_unstale":"2024-03-29T10:15:18.890233+0000","last_undegraded":"2024-03-29T10:15:18.890233+0000","last_fullsized":"2024-03-29T10:15:18.890233+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:09:00.431279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891312+0000","last_change":"2024-03-29T10:15:18.769523+0000","last_active":"2024-03-29T10:15:18.891312+0000","last_peered":"2024-03-29T10:15:18.891312+0000","last_clean":"2024-03-29T10:15:18.891312+0000","last_became_active":"2024-03-29T10:15:18.769263+0000","last_became_peered":"2024-03-29T10:15:18.769263+0000","last_unstale":"2024-03-29T10:15:18.891312+0000","last_undegraded":"2024-03-29T10:15:18.891312+0000","last_fullsized":"2024-03-29T10:15:18.891312+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:57.956507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"153'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890201+0000","last_change":"2024-03-29T10:15:18.236651+0000","last_active":"2024-03-29T10:15:18.890201+0000","last_peered":"2024-03-29T10:15:18.890201+0000","last_clean":"2024-03-29T10:15:18.890201+0000","last_became_active":"2024-03-29T10:15:18.236279+0000","last_became_peered":"2024-03-29T10:15:18.236279+0000","last_unstale":"2024-03-29T10:15:18.890201+0000","last_undegraded":"2024-03-29T10:15:18.890201+0000","last_fullsized":"2024-03-29T10:15:18.890201+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:48:21.209404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894176+0000","last_change":"2024-03-29T10:15:18.770581+0000","last_active":"2024-03-29T10:15:18.894176+0000","last_peered":"2024-03-29T10:15:18.894176+0000","last_clean":"2024-03-29T10:15:18.894176+0000","last_became_active":"2024-03-29T10:15:18.769791+0000","last_became_peered":"2024-03-29T10:15:18.769791+0000","last_unstale":"2024-03-29T10:15:18.894176+0000","last_undegraded":"2024-03-29T10:15:18.894176+0000","last_fullsized":"2024-03-29T10:15:18.894176+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:28.620001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"153'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891334+0000","last_change":"2024-03-29T10:15:18.239442+0000","last_active":"2024-03-29T10:15:18.891334+0000","last_peered":"2024-03-29T10:15:18.891334+0000","last_clean":"2024-03-29T10:15:18.891334+0000","last_became_active":"2024-03-29T10:15:18.236125+0000","last_became_peered":"2024-03-29T10:15:18.236125+0000","last_unstale":"2024-03-29T10:15:18.891334+0000","last_undegraded":"2024-03-29T10:15:18.891334+0000","last_fullsized":"2024-03-29T10:15:18.891334+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:08:34.640384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894552+0000","last_change":"2024-03-29T10:15:18.770877+0000","last_active":"2024-03-29T10:15:18.894552+0000","last_peered":"2024-03-29T10:15:18.894552+0000","last_clean":"2024-03-29T10:15:18.894552+0000","last_became_active":"2024-03-29T10:15:18.769944+0000","last_became_peered":"2024-03-29T10:15:18.769944+0000","last_unstale":"2024-03-29T10:15:18.894552+0000","last_undegraded":"2024-03-29T10:15:18.894552+0000","last_fullsized":"2024-03-29T10:15:18.894552+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:43.862113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894157+0000","last_change":"2024-03-29T10:15:18.244901+0000","last_active":"2024-03-29T10:15:18.894157+0000","last_peered":"2024-03-29T10:15:18.894157+0000","last_clean":"2024-03-29T10:15:18.894157+0000","last_became_active":"2024-03-29T10:15:18.244626+0000","last_became_peered":"2024-03-29T10:15:18.244626+0000","last_unstale":"2024-03-29T10:15:18.894157+0000","last_undegraded":"2024-03-29T10:15:18.894157+0000","last_fullsized":"2024-03-29T10:15:18.894157+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:10.873286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890170+0000","last_change":"2024-03-29T10:15:18.760287+0000","last_active":"2024-03-29T10:15:18.890170+0000","last_peered":"2024-03-29T10:15:18.890170+0000","last_clean":"2024-03-29T10:15:18.890170+0000","last_became_active":"2024-03-29T10:15:18.759386+0000","last_became_peered":"2024-03-29T10:15:18.759386+0000","last_unstale":"2024-03-29T10:15:18.890170+0000","last_undegraded":"2024-03-29T10:15:18.890170+0000","last_fullsized":"2024-03-29T10:15:18.890170+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:44:59.846995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890151+0000","last_change":"2024-03-29T10:15:18.238756+0000","last_active":"2024-03-29T10:15:18.890151+0000","last_peered":"2024-03-29T10:15:18.890151+0000","last_clean":"2024-03-29T10:15:18.890151+0000","last_became_active":"2024-03-29T10:15:18.238499+0000","last_became_peered":"2024-03-29T10:15:18.238499+0000","last_unstale":"2024-03-29T10:15:18.890151+0000","last_undegraded":"2024-03-29T10:15:18.890151+0000","last_fullsized":"2024-03-29T10:15:18.890151+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:41:25.038841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891039+0000","last_change":"2024-03-29T10:15:18.779956+0000","last_active":"2024-03-29T10:15:18.891039+0000","last_peered":"2024-03-29T10:15:18.891039+0000","last_clean":"2024-03-29T10:15:18.891039+0000","last_became_active":"2024-03-29T10:15:18.779231+0000","last_became_peered":"2024-03-29T10:15:18.779231+0000","last_unstale":"2024-03-29T10:15:18.891039+0000","last_undegraded":"2024-03-29T10:15:18.891039+0000","last_fullsized":"2024-03-29T10:15:18.891039+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:20.803416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.756966+0000","last_change":"2024-03-29T10:15:18.234835+0000","last_active":"2024-03-29T10:15:18.756966+0000","last_peered":"2024-03-29T10:15:18.756966+0000","last_clean":"2024-03-29T10:15:18.756966+0000","last_became_active":"2024-03-29T10:15:18.233366+0000","last_became_peered":"2024-03-29T10:15:18.233366+0000","last_unstale":"2024-03-29T10:15:18.756966+0000","last_undegraded":"2024-03-29T10:15:18.756966+0000","last_fullsized":"2024-03-29T10:15:18.756966+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:46:19.681575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.772215+0000","last_change":"2024-03-29T10:15:18.768888+0000","last_active":"2024-03-29T10:15:18.772215+0000","last_peered":"2024-03-29T10:15:18.772215+0000","last_clean":"2024-03-29T10:15:18.772215+0000","last_became_active":"2024-03-29T10:15:18.768603+0000","last_became_peered":"2024-03-29T10:15:18.768603+0000","last_unstale":"2024-03-29T10:15:18.772215+0000","last_undegraded":"2024-03-29T10:15:18.772215+0000","last_fullsized":"2024-03-29T10:15:18.772215+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:13.983592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891474+0000","last_change":"2024-03-29T10:15:18.244095+0000","last_active":"2024-03-29T10:15:18.891474+0000","last_peered":"2024-03-29T10:15:18.891474+0000","last_clean":"2024-03-29T10:15:18.891474+0000","last_became_active":"2024-03-29T10:15:18.243018+0000","last_became_peered":"2024-03-29T10:15:18.243018+0000","last_unstale":"2024-03-29T10:15:18.891474+0000","last_undegraded":"2024-03-29T10:15:18.891474+0000","last_fullsized":"2024-03-29T10:15:18.891474+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:42.853163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.774952+0000","last_change":"2024-03-29T10:15:18.769534+0000","last_active":"2024-03-29T10:15:18.774952+0000","last_peered":"2024-03-29T10:15:18.774952+0000","last_clean":"2024-03-29T10:15:18.774952+0000","last_became_active":"2024-03-29T10:15:18.769285+0000","last_became_peered":"2024-03-29T10:15:18.769285+0000","last_unstale":"2024-03-29T10:15:18.774952+0000","last_undegraded":"2024-03-29T10:15:18.774952+0000","last_fullsized":"2024-03-29T10:15:18.774952+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:26:23.283920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"153'3","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.782176+0000","last_change":"2024-03-29T10:15:18.230283+0000","last_active":"2024-03-29T10:15:18.782176+0000","last_peered":"2024-03-29T10:15:18.782176+0000","last_clean":"2024-03-29T10:15:18.782176+0000","last_became_active":"2024-03-29T10:15:18.229323+0000","last_became_peered":"2024-03-29T10:15:18.229323+0000","last_unstale":"2024-03-29T10:15:18.782176+0000","last_undegraded":"2024-03-29T10:15:18.782176+0000","last_fullsized":"2024-03-29T10:15:18.782176+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:50:44.068800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.760125+0000","last_change":"2024-03-29T10:15:18.758162+0000","last_active":"2024-03-29T10:15:18.760125+0000","last_peered":"2024-03-29T10:15:18.760125+0000","last_clean":"2024-03-29T10:15:18.760125+0000","last_became_active":"2024-03-29T10:15:18.757041+0000","last_became_peered":"2024-03-29T10:15:18.757041+0000","last_unstale":"2024-03-29T10:15:18.760125+0000","last_undegraded":"2024-03-29T10:15:18.760125+0000","last_fullsized":"2024-03-29T10:15:18.760125+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:39.190584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"153'6","reported_seq":17,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894621+0000","last_change":"2024-03-29T10:15:18.238514+0000","last_active":"2024-03-29T10:15:18.894621+0000","last_peered":"2024-03-29T10:15:18.894621+0000","last_clean":"2024-03-29T10:15:18.894621+0000","last_became_active":"2024-03-29T10:15:18.238124+0000","last_became_peered":"2024-03-29T10:15:18.238124+0000","last_unstale":"2024-03-29T10:15:18.894621+0000","last_undegraded":"2024-03-29T10:15:18.894621+0000","last_fullsized":"2024-03-29T10:15:18.894621+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:46:01.519930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891000+0000","last_change":"2024-03-29T10:15:18.774341+0000","last_active":"2024-03-29T10:15:18.891000+0000","last_peered":"2024-03-29T10:15:18.891000+0000","last_clean":"2024-03-29T10:15:18.891000+0000","last_became_active":"2024-03-29T10:15:18.770843+0000","last_became_peered":"2024-03-29T10:15:18.770843+0000","last_unstale":"2024-03-29T10:15:18.891000+0000","last_undegraded":"2024-03-29T10:15:18.891000+0000","last_fullsized":"2024-03-29T10:15:18.891000+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:43:11.098286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890982+0000","last_change":"2024-03-29T10:15:18.240012+0000","last_active":"2024-03-29T10:15:18.890982+0000","last_peered":"2024-03-29T10:15:18.890982+0000","last_clean":"2024-03-29T10:15:18.890982+0000","last_became_active":"2024-03-29T10:15:18.239023+0000","last_became_peered":"2024-03-29T10:15:18.239023+0000","last_unstale":"2024-03-29T10:15:18.890982+0000","last_undegraded":"2024-03-29T10:15:18.890982+0000","last_fullsized":"2024-03-29T10:15:18.890982+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:12:25.936776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890141+0000","last_change":"2024-03-29T10:15:18.754727+0000","last_active":"2024-03-29T10:15:18.890141+0000","last_peered":"2024-03-29T10:15:18.890141+0000","last_clean":"2024-03-29T10:15:18.890141+0000","last_became_active":"2024-03-29T10:15:18.754469+0000","last_became_peered":"2024-03-29T10:15:18.754469+0000","last_unstale":"2024-03-29T10:15:18.890141+0000","last_undegraded":"2024-03-29T10:15:18.890141+0000","last_fullsized":"2024-03-29T10:15:18.890141+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:34:15.821450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890179+0000","last_change":"2024-03-29T10:15:18.245095+0000","last_active":"2024-03-29T10:15:18.890179+0000","last_peered":"2024-03-29T10:15:18.890179+0000","last_clean":"2024-03-29T10:15:18.890179+0000","last_became_active":"2024-03-29T10:15:18.244834+0000","last_became_peered":"2024-03-29T10:15:18.244834+0000","last_unstale":"2024-03-29T10:15:18.890179+0000","last_undegraded":"2024-03-29T10:15:18.890179+0000","last_fullsized":"2024-03-29T10:15:18.890179+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:27.668887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894600+0000","last_change":"2024-03-29T10:15:18.766175+0000","last_active":"2024-03-29T10:15:18.894600+0000","last_peered":"2024-03-29T10:15:18.894600+0000","last_clean":"2024-03-29T10:15:18.894600+0000","last_became_active":"2024-03-29T10:15:18.762502+0000","last_became_peered":"2024-03-29T10:15:18.762502+0000","last_unstale":"2024-03-29T10:15:18.894600+0000","last_undegraded":"2024-03-29T10:15:18.894600+0000","last_fullsized":"2024-03-29T10:15:18.894600+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:28:56.375633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"153'8","reported_seq":19,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894578+0000","last_change":"2024-03-29T10:15:18.236493+0000","last_active":"2024-03-29T10:15:18.894578+0000","last_peered":"2024-03-29T10:15:18.894578+0000","last_clean":"2024-03-29T10:15:18.894578+0000","last_became_active":"2024-03-29T10:15:18.233617+0000","last_became_peered":"2024-03-29T10:15:18.233617+0000","last_unstale":"2024-03-29T10:15:18.894578+0000","last_undegraded":"2024-03-29T10:15:18.894578+0000","last_fullsized":"2024-03-29T10:15:18.894578+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:12.974642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894641+0000","last_change":"2024-03-29T10:15:18.758467+0000","last_active":"2024-03-29T10:15:18.894641+0000","last_peered":"2024-03-29T10:15:18.894641+0000","last_clean":"2024-03-29T10:15:18.894641+0000","last_became_active":"2024-03-29T10:15:18.758163+0000","last_became_peered":"2024-03-29T10:15:18.758163+0000","last_unstale":"2024-03-29T10:15:18.894641+0000","last_undegraded":"2024-03-29T10:15:18.894641+0000","last_fullsized":"2024-03-29T10:15:18.894641+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:08:35.649335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.778850+0000","last_change":"2024-03-29T10:15:18.240951+0000","last_active":"2024-03-29T10:15:18.778850+0000","last_peered":"2024-03-29T10:15:18.778850+0000","last_clean":"2024-03-29T10:15:18.778850+0000","last_became_active":"2024-03-29T10:15:18.240622+0000","last_became_peered":"2024-03-29T10:15:18.240622+0000","last_unstale":"2024-03-29T10:15:18.778850+0000","last_undegraded":"2024-03-29T10:15:18.778850+0000","last_fullsized":"2024-03-29T10:15:18.778850+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:12:25.936776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.772195+0000","last_change":"2024-03-29T10:15:18.765995+0000","last_active":"2024-03-29T10:15:18.772195+0000","last_peered":"2024-03-29T10:15:18.772195+0000","last_clean":"2024-03-29T10:15:18.772195+0000","last_became_active":"2024-03-29T10:15:18.765734+0000","last_became_peered":"2024-03-29T10:15:18.765734+0000","last_unstale":"2024-03-29T10:15:18.772195+0000","last_undegraded":"2024-03-29T10:15:18.772195+0000","last_fullsized":"2024-03-29T10:15:18.772195+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:46:02.528881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"153'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894832+0000","last_change":"2024-03-29T10:15:18.246087+0000","last_active":"2024-03-29T10:15:18.894832+0000","last_peered":"2024-03-29T10:15:18.894832+0000","last_clean":"2024-03-29T10:15:18.894832+0000","last_became_active":"2024-03-29T10:15:18.244538+0000","last_became_peered":"2024-03-29T10:15:18.244538+0000","last_unstale":"2024-03-29T10:15:18.894832+0000","last_undegraded":"2024-03-29T10:15:18.894832+0000","last_fullsized":"2024-03-29T10:15:18.894832+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:32:56.586878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891395+0000","last_change":"2024-03-29T10:15:18.777418+0000","last_active":"2024-03-29T10:15:18.891395+0000","last_peered":"2024-03-29T10:15:18.891395+0000","last_clean":"2024-03-29T10:15:18.891395+0000","last_became_active":"2024-03-29T10:15:18.777127+0000","last_became_peered":"2024-03-29T10:15:18.777127+0000","last_unstale":"2024-03-29T10:15:18.891395+0000","last_undegraded":"2024-03-29T10:15:18.891395+0000","last_fullsized":"2024-03-29T10:15:18.891395+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:27:53.728925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"153'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894204+0000","last_change":"2024-03-29T10:15:18.247457+0000","last_active":"2024-03-29T10:15:18.894204+0000","last_peered":"2024-03-29T10:15:18.894204+0000","last_clean":"2024-03-29T10:15:18.894204+0000","last_became_active":"2024-03-29T10:15:18.247070+0000","last_became_peered":"2024-03-29T10:15:18.247070+0000","last_unstale":"2024-03-29T10:15:18.894204+0000","last_undegraded":"2024-03-29T10:15:18.894204+0000","last_fullsized":"2024-03-29T10:15:18.894204+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:39:19.794465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890242+0000","last_change":"2024-03-29T10:15:18.767943+0000","last_active":"2024-03-29T10:15:18.890242+0000","last_peered":"2024-03-29T10:15:18.890242+0000","last_clean":"2024-03-29T10:15:18.890242+0000","last_became_active":"2024-03-29T10:15:18.764049+0000","last_became_peered":"2024-03-29T10:15:18.764049+0000","last_unstale":"2024-03-29T10:15:18.890242+0000","last_undegraded":"2024-03-29T10:15:18.890242+0000","last_fullsized":"2024-03-29T10:15:18.890242+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:12:12.821950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.757821+0000","last_change":"2024-03-29T10:15:18.242641+0000","last_active":"2024-03-29T10:15:18.757821+0000","last_peered":"2024-03-29T10:15:18.757821+0000","last_clean":"2024-03-29T10:15:18.757821+0000","last_became_active":"2024-03-29T10:15:18.242367+0000","last_became_peered":"2024-03-29T10:15:18.242367+0000","last_unstale":"2024-03-29T10:15:18.757821+0000","last_undegraded":"2024-03-29T10:15:18.757821+0000","last_fullsized":"2024-03-29T10:15:18.757821+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:12:40.712000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890962+0000","last_change":"2024-03-29T10:15:18.774216+0000","last_active":"2024-03-29T10:15:18.890962+0000","last_peered":"2024-03-29T10:15:18.890962+0000","last_clean":"2024-03-29T10:15:18.890962+0000","last_became_active":"2024-03-29T10:15:18.773560+0000","last_became_peered":"2024-03-29T10:15:18.773560+0000","last_unstale":"2024-03-29T10:15:18.890962+0000","last_undegraded":"2024-03-29T10:15:18.890962+0000","last_fullsized":"2024-03-29T10:15:18.890962+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:33.197637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781438+0000","last_change":"2024-03-29T10:15:18.232949+0000","last_active":"2024-03-29T10:15:18.781438+0000","last_peered":"2024-03-29T10:15:18.781438+0000","last_clean":"2024-03-29T10:15:18.781438+0000","last_became_active":"2024-03-29T10:15:18.232594+0000","last_became_peered":"2024-03-29T10:15:18.232594+0000","last_unstale":"2024-03-29T10:15:18.781438+0000","last_undegraded":"2024-03-29T10:15:18.781438+0000","last_fullsized":"2024-03-29T10:15:18.781438+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:25.414983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894185+0000","last_change":"2024-03-29T10:15:18.764608+0000","last_active":"2024-03-29T10:15:18.894185+0000","last_peered":"2024-03-29T10:15:18.894185+0000","last_clean":"2024-03-29T10:15:18.894185+0000","last_became_active":"2024-03-29T10:15:18.761955+0000","last_became_peered":"2024-03-29T10:15:18.761955+0000","last_unstale":"2024-03-29T10:15:18.894185+0000","last_undegraded":"2024-03-29T10:15:18.894185+0000","last_fullsized":"2024-03-29T10:15:18.894185+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:57:09.757851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891355+0000","last_change":"2024-03-29T10:15:18.240062+0000","last_active":"2024-03-29T10:15:18.891355+0000","last_peered":"2024-03-29T10:15:18.891355+0000","last_clean":"2024-03-29T10:15:18.891355+0000","last_became_active":"2024-03-29T10:15:18.239758+0000","last_became_peered":"2024-03-29T10:15:18.239758+0000","last_unstale":"2024-03-29T10:15:18.891355+0000","last_undegraded":"2024-03-29T10:15:18.891355+0000","last_fullsized":"2024-03-29T10:15:18.891355+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:27.611051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890124+0000","last_change":"2024-03-29T10:15:18.765774+0000","last_active":"2024-03-29T10:15:18.890124+0000","last_peered":"2024-03-29T10:15:18.890124+0000","last_clean":"2024-03-29T10:15:18.890124+0000","last_became_active":"2024-03-29T10:15:18.765517+0000","last_became_peered":"2024-03-29T10:15:18.765517+0000","last_unstale":"2024-03-29T10:15:18.890124+0000","last_undegraded":"2024-03-29T10:15:18.890124+0000","last_fullsized":"2024-03-29T10:15:18.890124+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:16:23.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891274+0000","last_change":"2024-03-29T10:15:18.230643+0000","last_active":"2024-03-29T10:15:18.891274+0000","last_peered":"2024-03-29T10:15:18.891274+0000","last_clean":"2024-03-29T10:15:18.891274+0000","last_became_active":"2024-03-29T10:15:18.229828+0000","last_became_peered":"2024-03-29T10:15:18.229828+0000","last_unstale":"2024-03-29T10:15:18.891274+0000","last_undegraded":"2024-03-29T10:15:18.891274+0000","last_fullsized":"2024-03-29T10:15:18.891274+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:04.101275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.769536+0000","last_change":"2024-03-29T10:15:18.767298+0000","last_active":"2024-03-29T10:15:18.769536+0000","last_peered":"2024-03-29T10:15:18.769536+0000","last_clean":"2024-03-29T10:15:18.769536+0000","last_became_active":"2024-03-29T10:15:18.766953+0000","last_became_peered":"2024-03-29T10:15:18.766953+0000","last_unstale":"2024-03-29T10:15:18.769536+0000","last_undegraded":"2024-03-29T10:15:18.769536+0000","last_fullsized":"2024-03-29T10:15:18.769536+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:57.956507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894764+0000","last_change":"2024-03-29T10:15:18.244819+0000","last_active":"2024-03-29T10:15:18.894764+0000","last_peered":"2024-03-29T10:15:18.894764+0000","last_clean":"2024-03-29T10:15:18.894764+0000","last_became_active":"2024-03-29T10:15:18.244526+0000","last_became_peered":"2024-03-29T10:15:18.244526+0000","last_unstale":"2024-03-29T10:15:18.894764+0000","last_undegraded":"2024-03-29T10:15:18.894764+0000","last_fullsized":"2024-03-29T10:15:18.894764+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:23:52.991818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894215+0000","last_change":"2024-03-29T10:15:18.769749+0000","last_active":"2024-03-29T10:15:18.894215+0000","last_peered":"2024-03-29T10:15:18.894215+0000","last_clean":"2024-03-29T10:15:18.894215+0000","last_became_active":"2024-03-29T10:15:18.766764+0000","last_became_peered":"2024-03-29T10:15:18.766764+0000","last_unstale":"2024-03-29T10:15:18.894215+0000","last_undegraded":"2024-03-29T10:15:18.894215+0000","last_fullsized":"2024-03-29T10:15:18.894215+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.723716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"153'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891068+0000","last_change":"2024-03-29T10:15:18.240629+0000","last_active":"2024-03-29T10:15:18.891068+0000","last_peered":"2024-03-29T10:15:18.891068+0000","last_clean":"2024-03-29T10:15:18.891068+0000","last_became_active":"2024-03-29T10:15:18.240375+0000","last_became_peered":"2024-03-29T10:15:18.240375+0000","last_unstale":"2024-03-29T10:15:18.891068+0000","last_undegraded":"2024-03-29T10:15:18.891068+0000","last_fullsized":"2024-03-29T10:15:18.891068+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:47.748627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891056+0000","last_change":"2024-03-29T10:15:18.766950+0000","last_active":"2024-03-29T10:15:18.891056+0000","last_peered":"2024-03-29T10:15:18.891056+0000","last_clean":"2024-03-29T10:15:18.891056+0000","last_became_active":"2024-03-29T10:15:18.766669+0000","last_became_peered":"2024-03-29T10:15:18.766669+0000","last_unstale":"2024-03-29T10:15:18.891056+0000","last_undegraded":"2024-03-29T10:15:18.891056+0000","last_fullsized":"2024-03-29T10:15:18.891056+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:16:23.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.750806+0000","last_change":"2024-03-29T10:15:18.246341+0000","last_active":"2024-03-29T10:15:18.750806+0000","last_peered":"2024-03-29T10:15:18.750806+0000","last_clean":"2024-03-29T10:15:18.750806+0000","last_became_active":"2024-03-29T10:15:18.244902+0000","last_became_peered":"2024-03-29T10:15:18.244902+0000","last_unstale":"2024-03-29T10:15:18.750806+0000","last_undegraded":"2024-03-29T10:15:18.750806+0000","last_fullsized":"2024-03-29T10:15:18.750806+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:43:10.089336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890223+0000","last_change":"2024-03-29T10:15:18.755404+0000","last_active":"2024-03-29T10:15:18.890223+0000","last_peered":"2024-03-29T10:15:18.890223+0000","last_clean":"2024-03-29T10:15:18.890223+0000","last_became_active":"2024-03-29T10:15:18.755137+0000","last_became_peered":"2024-03-29T10:15:18.755137+0000","last_unstale":"2024-03-29T10:15:18.890223+0000","last_undegraded":"2024-03-29T10:15:18.890223+0000","last_fullsized":"2024-03-29T10:15:18.890223+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:46:02.528881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890251+0000","last_change":"2024-03-29T10:15:18.239927+0000","last_active":"2024-03-29T10:15:18.890251+0000","last_peered":"2024-03-29T10:15:18.890251+0000","last_clean":"2024-03-29T10:15:18.890251+0000","last_became_active":"2024-03-29T10:15:18.239622+0000","last_became_peered":"2024-03-29T10:15:18.239622+0000","last_unstale":"2024-03-29T10:15:18.890251+0000","last_undegraded":"2024-03-29T10:15:18.890251+0000","last_fullsized":"2024-03-29T10:15:18.890251+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:50:55.102853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894195+0000","last_change":"2024-03-29T10:15:18.778117+0000","last_active":"2024-03-29T10:15:18.894195+0000","last_peered":"2024-03-29T10:15:18.894195+0000","last_clean":"2024-03-29T10:15:18.894195+0000","last_became_active":"2024-03-29T10:15:18.777497+0000","last_became_peered":"2024-03-29T10:15:18.777497+0000","last_unstale":"2024-03-29T10:15:18.894195+0000","last_undegraded":"2024-03-29T10:15:18.894195+0000","last_fullsized":"2024-03-29T10:15:18.894195+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:53:39.190584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894792+0000","last_change":"2024-03-29T10:15:18.247493+0000","last_active":"2024-03-29T10:15:18.894792+0000","last_peered":"2024-03-29T10:15:18.894792+0000","last_clean":"2024-03-29T10:15:18.894792+0000","last_became_active":"2024-03-29T10:15:18.247144+0000","last_became_peered":"2024-03-29T10:15:18.247144+0000","last_unstale":"2024-03-29T10:15:18.894792+0000","last_undegraded":"2024-03-29T10:15:18.894792+0000","last_fullsized":"2024-03-29T10:15:18.894792+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:01:05.207395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.774624+0000","last_change":"2024-03-29T10:15:18.768946+0000","last_active":"2024-03-29T10:15:18.774624+0000","last_peered":"2024-03-29T10:15:18.774624+0000","last_clean":"2024-03-29T10:15:18.774624+0000","last_became_active":"2024-03-29T10:15:18.768633+0000","last_became_peered":"2024-03-29T10:15:18.768633+0000","last_unstale":"2024-03-29T10:15:18.774624+0000","last_undegraded":"2024-03-29T10:15:18.774624+0000","last_fullsized":"2024-03-29T10:15:18.774624+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:30:45.359557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890213+0000","last_change":"2024-03-29T10:15:18.243619+0000","last_active":"2024-03-29T10:15:18.890213+0000","last_peered":"2024-03-29T10:15:18.890213+0000","last_clean":"2024-03-29T10:15:18.890213+0000","last_became_active":"2024-03-29T10:15:18.243349+0000","last_became_peered":"2024-03-29T10:15:18.243349+0000","last_unstale":"2024-03-29T10:15:18.890213+0000","last_undegraded":"2024-03-29T10:15:18.890213+0000","last_fullsized":"2024-03-29T10:15:18.890213+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:27.193825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.765953+0000","last_change":"2024-03-29T10:15:18.760460+0000","last_active":"2024-03-29T10:15:18.765953+0000","last_peered":"2024-03-29T10:15:18.765953+0000","last_clean":"2024-03-29T10:15:18.765953+0000","last_became_active":"2024-03-29T10:15:18.760131+0000","last_became_peered":"2024-03-29T10:15:18.760131+0000","last_unstale":"2024-03-29T10:15:18.765953+0000","last_undegraded":"2024-03-29T10:15:18.765953+0000","last_fullsized":"2024-03-29T10:15:18.765953+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:37:59.721555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.748929+0000","last_change":"2024-03-29T10:15:18.247758+0000","last_active":"2024-03-29T10:15:18.748929+0000","last_peered":"2024-03-29T10:15:18.748929+0000","last_clean":"2024-03-29T10:15:18.748929+0000","last_became_active":"2024-03-29T10:15:18.247439+0000","last_became_peered":"2024-03-29T10:15:18.247439+0000","last_unstale":"2024-03-29T10:15:18.748929+0000","last_undegraded":"2024-03-29T10:15:18.748929+0000","last_fullsized":"2024-03-29T10:15:18.748929+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:20.864907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891250+0000","last_change":"2024-03-29T10:15:18.763827+0000","last_active":"2024-03-29T10:15:18.891250+0000","last_peered":"2024-03-29T10:15:18.891250+0000","last_clean":"2024-03-29T10:15:18.891250+0000","last_became_active":"2024-03-29T10:15:18.763351+0000","last_became_peered":"2024-03-29T10:15:18.763351+0000","last_unstale":"2024-03-29T10:15:18.891250+0000","last_undegraded":"2024-03-29T10:15:18.891250+0000","last_fullsized":"2024-03-29T10:15:18.891250+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:48:34.366666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.891295+0000","last_change":"2024-03-29T10:15:18.239666+0000","last_active":"2024-03-29T10:15:18.891295+0000","last_peered":"2024-03-29T10:15:18.891295+0000","last_clean":"2024-03-29T10:15:18.891295+0000","last_became_active":"2024-03-29T10:15:18.236397+0000","last_became_peered":"2024-03-29T10:15:18.236397+0000","last_unstale":"2024-03-29T10:15:18.891295+0000","last_undegraded":"2024-03-29T10:15:18.891295+0000","last_fullsized":"2024-03-29T10:15:18.891295+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:57:08.748901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.890161+0000","last_change":"2024-03-29T10:15:18.757574+0000","last_active":"2024-03-29T10:15:18.890161+0000","last_peered":"2024-03-29T10:15:18.890161+0000","last_clean":"2024-03-29T10:15:18.890161+0000","last_became_active":"2024-03-29T10:15:18.757315+0000","last_became_peered":"2024-03-29T10:15:18.757315+0000","last_unstale":"2024-03-29T10:15:18.890161+0000","last_undegraded":"2024-03-29T10:15:18.890161+0000","last_fullsized":"2024-03-29T10:15:18.890161+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:58.525728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.894773+0000","last_change":"2024-03-29T10:15:18.238738+0000","last_active":"2024-03-29T10:15:18.894773+0000","last_peered":"2024-03-29T10:15:18.894773+0000","last_clean":"2024-03-29T10:15:18.894773+0000","last_became_active":"2024-03-29T10:15:18.237835+0000","last_became_peered":"2024-03-29T10:15:18.237835+0000","last_unstale":"2024-03-29T10:15:18.894773+0000","last_undegraded":"2024-03-29T10:15:18.894773+0000","last_fullsized":"2024-03-29T10:15:18.894773+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:31.906737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.759204+0000","last_change":"2024-03-29T10:15:18.755674+0000","last_active":"2024-03-29T10:15:18.759204+0000","last_peered":"2024-03-29T10:15:18.759204+0000","last_clean":"2024-03-29T10:15:18.759204+0000","last_became_active":"2024-03-29T10:15:18.755377+0000","last_became_peered":"2024-03-29T10:15:18.755377+0000","last_unstale":"2024-03-29T10:15:18.759204+0000","last_undegraded":"2024-03-29T10:15:18.759204+0000","last_fullsized":"2024-03-29T10:15:18.759204+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:18.211154+0000","last_clean_scrub_stamp":"2024-03-29T10:15:18.211154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:48:34.366666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.781938+0000","last_change":"2024-03-29T10:15:18.248847+0000","last_active":"2024-03-29T10:15:18.781938+0000","last_peered":"2024-03-29T10:15:18.781938+0000","last_clean":"2024-03-29T10:15:18.781938+0000","last_became_active":"2024-03-29T10:15:18.248547+0000","last_became_peered":"2024-03-29T10:15:18.248547+0000","last_unstale":"2024-03-29T10:15:18.781938+0000","last_undegraded":"2024-03-29T10:15:18.781938+0000","last_fullsized":"2024-03-29T10:15:18.781938+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T10:15:17.202204+0000","last_clean_scrub_stamp":"2024-03-29T10:15:17.202204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:27.631102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"143'267","reported_seq":818,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-29T10:15:18.240146+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T10:15:18.240146+0000","last_peered":"2024-03-29T10:15:18.240146+0000","last_clean":"2024-03-29T10:15:18.240146+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T10:15:18.240146+0000","last_undegraded":"2024-03-29T10:15:18.240146+0000","last_fullsized":"2024-03-29T10:15:18.240146+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":431,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":431,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267,"ondisk_log_size":267,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575089,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37644,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334196,"statfs":{"total":96636764160,"available":96598216704,"internally_reserved":0,"allocated":2486272,"data_stored":1495968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":36043212},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66500000000000004,"15min":0.69999999999999996},"min":{"1min":0.432,"5min":0.309,"15min":0.309},"max":{"1min":0.85299999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.67800000000000005,"15min":0.69899999999999995},"min":{"1min":0.502,"5min":0.41699999999999998,"15min":0.377},"max":{"1min":0.81399999999999995,"5min":1.143,"15min":1.143},"last":0.71499999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69499999999999995,"15min":0.71599999999999997},"min":{"1min":0.49099999999999999,"5min":0.45700000000000002,"15min":0.44500000000000001},"max":{"1min":0.878,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69599999999999995,"15min":0.72599999999999998},"min":{"1min":0.55400000000000005,"5min":0.46000000000000002,"15min":0.41199999999999998},"max":{"1min":0.80400000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.74299999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.70099999999999996,"15min":0.72899999999999998},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.45200000000000001},"max":{"1min":0.90200000000000002,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.72599999999999998,"15min":0.747},"min":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.48799999999999999},"max":{"1min":0.90300000000000002,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.76100000000000001}]},{"osd":3,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72599999999999998,"15min":0.73099999999999998},"min":{"1min":0.49299999999999999,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.85199999999999998,"5min":1.171,"15min":1.171},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.72799999999999998,"15min":0.76300000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.89600000000000002,"5min":0.91200000000000003,"15min":1.129},"last":0.78200000000000003}]},{"osd":4,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63500000000000001,"15min":0.60599999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89400000000000002,"5min":1.25,"15min":1.25},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66000000000000003,"15min":0.63900000000000001},"min":{"1min":0.46700000000000003,"5min":0.41899999999999998,"15min":0.377},"max":{"1min":0.80700000000000005,"5min":1.117,"15min":1.117},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68300000000000005,"15min":0.65500000000000003},"min":{"1min":0.443,"5min":0.43099999999999999,"15min":0.40699999999999997},"max":{"1min":1.0009999999999999,"5min":1.47,"15min":1.47},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72499999999999998,"15min":0.69199999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39400000000000002},"max":{"1min":0.95999999999999996,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.65200000000000002}]},{"osd":6,"last update":"Fri Mar 29 10:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.75800000000000001,"15min":0.73399999999999999},"min":{"1min":0.52600000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.97899999999999998,"5min":1.222,"15min":1.222},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.70899999999999996},"min":{"1min":0.55100000000000005,"5min":0.46899999999999997,"15min":0.443},"max":{"1min":0.95199999999999996,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.80000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575091,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37104,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334736,"statfs":{"total":96636764160,"available":96598769664,"internally_reserved":0,"allocated":1929216,"data_stored":920239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67100000000000004,"15min":0.70099999999999996},"min":{"1min":0.441,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88700000000000001,"5min":1.006,"15min":2.7770000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66900000000000004,"15min":0.71599999999999997},"min":{"1min":0.46700000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.86499999999999999,"5min":0.93300000000000005,"15min":1.5049999999999999},"last":0.56899999999999995}]},{"osd":1,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64100000000000001,"15min":0.69299999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80200000000000005,"5min":0.94799999999999995,"15min":1.6279999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68100000000000005,"15min":0.71699999999999997},"min":{"1min":0.45700000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.92800000000000005,"5min":1.0289999999999999,"15min":2.6709999999999998},"last":0.65100000000000002}]},{"osd":2,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66700000000000004,"15min":0.69999999999999996},"min":{"1min":0.46000000000000002,"5min":0.45100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81799999999999995,"5min":0.95799999999999996,"15min":1.256},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70699999999999996,"15min":0.73199999999999998},"min":{"1min":0.497,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.91600000000000004,"5min":0.97099999999999997,"15min":1.454},"last":0.621}]},{"osd":3,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68500000000000005,"15min":0.72899999999999998},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.41099999999999998},"max":{"1min":0.90700000000000003,"5min":0.97899999999999998,"15min":1.131},"last":0.63},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72099999999999997,"15min":0.76100000000000001},"min":{"1min":0.48999999999999999,"5min":0.47199999999999998,"15min":0.39000000000000001},"max":{"1min":1.0089999999999999,"5min":1.034,"15min":2.718},"last":0.75600000000000001}]},{"osd":4,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57699999999999996,"15min":0.60299999999999998},"min":{"1min":0.44800000000000001,"5min":0.41299999999999998,"15min":0.35199999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":2.4780000000000002},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.65300000000000002,"15min":0.68600000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91300000000000003,"5min":0.93799999999999994,"15min":2.544},"last":0.73999999999999999}]},{"osd":5,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.624,"15min":0.65100000000000002},"min":{"1min":0.47999999999999998,"5min":0.40300000000000002,"15min":0.34599999999999997},"max":{"1min":0.84599999999999997,"5min":0.91800000000000004,"15min":3.2280000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.48599999999999999,"5min":0.45600000000000002,"15min":0.38100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":2.835},"last":0.46899999999999997}]},{"osd":7,"last update":"Fri Mar 29 10:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.71099999999999997,"15min":0.74299999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.45100000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.6559999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.66900000000000004,"15min":0.70799999999999996},"min":{"1min":0.56100000000000005,"5min":0.45100000000000001,"15min":0.38900000000000001},"max":{"1min":0.872,"5min":0.96099999999999997,"15min":3.7170000000000001},"last":0.58699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575089,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55132,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316708,"statfs":{"total":96636764160,"available":96580308992,"internally_reserved":0,"allocated":1916928,"data_stored":918675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.219},"max":{"1min":0.52900000000000003,"5min":0.81699999999999995,"15min":1.5600000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.502,"15min":0.50800000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67000000000000004,"5min":0.71799999999999997,"15min":1.0860000000000001},"last":0.34699999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.51900000000000002,"15min":0.53500000000000003},"min":{"1min":0.35599999999999998,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.63100000000000001,"5min":1.222,"15min":1.7869999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.91700000000000004,"5min":1.04,"15min":1.04},"last":0.48399999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56699999999999995,"15min":0.58599999999999997},"min":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.377},"max":{"1min":0.66400000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.52700000000000002,"15min":0.56000000000000005},"min":{"1min":0.375,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73299999999999998,"5min":1.026,"15min":1.3049999999999999},"last":0.44}]},{"osd":4,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.58899999999999997,"5min":0.52900000000000003,"15min":0.497},"max":{"1min":1.2070000000000001,"5min":1.2709999999999999,"15min":1.617},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.754,"5min":0.77300000000000002,"15min":0.77400000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.48299999999999998},"max":{"1min":0.98999999999999999,"5min":1.248,"15min":1.248},"last":0.66700000000000004}]},{"osd":5,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.55400000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.1160000000000001,"5min":1.157,"15min":1.3939999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.78400000000000003,"15min":0.79600000000000004},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.52900000000000003},"max":{"1min":1.016,"5min":1.3140000000000001,"15min":1.6579999999999999},"last":0.63800000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.79200000000000004},"min":{"1min":0.55700000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80100000000000005,"15min":0.80900000000000005},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.53600000000000003},"max":{"1min":1.0920000000000001,"5min":1.319,"15min":1.468},"last":0.72099999999999997}]},{"osd":7,"last update":"Fri Mar 29 10:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81599999999999995,"15min":0.83099999999999996},"min":{"1min":0.64300000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.6000000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.82299999999999995,"15min":0.82799999999999996},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.57099999999999995},"max":{"1min":1.089,"5min":1.4530000000000001,"15min":1.6919999999999999},"last":0.74199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575091,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37664,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334176,"statfs":{"total":96636764160,"available":96598196224,"internally_reserved":0,"allocated":2506752,"data_stored":1508024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.433,"15min":0.45200000000000001},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.61399999999999999,"5min":0.72299999999999998,"15min":0.93300000000000005},"last":0.441},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.27500000000000002},"max":{"1min":0.72099999999999997,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81899999999999995,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60299999999999998,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.46500000000000002}]},{"osd":3,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53100000000000003,"15min":0.54600000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.33700000000000002},"max":{"1min":0.63700000000000001,"5min":0.84899999999999998,"15min":1.077},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.40400000000000003,"15min":0.39100000000000001},"max":{"1min":0.69999999999999996,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.68899999999999995}]},{"osd":4,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.79000000000000004,"15min":0.79800000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.39200000000000002},"max":{"1min":0.88400000000000001,"5min":1.254,"15min":1.625},"last":0.747},{"interface":"front","average":{"1min":0.755,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.61799999999999999,"5min":0.60099999999999998,"15min":0.45800000000000002},"max":{"1min":0.86899999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.78100000000000003}]},{"osd":5,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.753,"15min":0.753},"min":{"1min":0.57199999999999995,"5min":0.48799999999999999,"15min":0.39000000000000001},"max":{"1min":0.872,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.78400000000000003,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.41099999999999998},"max":{"1min":1.0409999999999999,"5min":1.4970000000000001,"15min":1.5309999999999999},"last":0.83899999999999997}]},{"osd":6,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.82099999999999995,"15min":0.81899999999999995},"min":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.95599999999999996,"5min":1.476,"15min":1.476},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.60199999999999998,"5min":0.5,"15min":0.45600000000000002},"max":{"1min":0.99099999999999999,"5min":1.3340000000000001,"15min":1.601},"last":0.79700000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:15:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.78200000000000003,"15min":0.79300000000000004},"min":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.44900000000000001},"max":{"1min":0.94299999999999995,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":0.83299999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.48199999999999998},"max":{"1min":1.0149999999999999,"5min":1.296,"15min":1.5669999999999999},"last":0.88400000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575090,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59244,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312596,"statfs":{"total":96636764160,"available":96576098304,"internally_reserved":0,"allocated":1925120,"data_stored":919291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.45300000000000001},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.248},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.47699999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.38},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":0.42899999999999999}]},{"osd":3,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.42199999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.58099999999999996,"15min":0.55800000000000005},"min":{"1min":0.40100000000000002,"5min":0.379,"15min":0.32100000000000001},"max":{"1min":2.2000000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.495}]},{"osd":4,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.76600000000000001,"15min":0.77300000000000002},"min":{"1min":0.53600000000000003,"5min":0.51100000000000001,"15min":0.42799999999999999},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.76200000000000001,"15min":0.78300000000000003},"min":{"1min":0.53000000000000003,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.242,"5min":2.242,"15min":2.242},"last":0.76300000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.57499999999999996,"5min":0.52400000000000002,"15min":0.46999999999999997},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.79800000000000004,"15min":0.79300000000000004},"min":{"1min":0.55800000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":0.66300000000000003}]},{"osd":6,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.80900000000000005,"15min":0.80800000000000005},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.82299999999999995,"15min":0.82999999999999996},"min":{"1min":0.56799999999999995,"5min":0.56200000000000006,"15min":0.55000000000000004},"max":{"1min":5.2030000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.61399999999999999}]},{"osd":7,"last update":"Fri Mar 29 10:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.82999999999999996,"15min":0.85099999999999998},"min":{"1min":0.63600000000000001,"5min":0.56899999999999995,"15min":0.53400000000000003},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.61699999999999999,"5min":0.60899999999999999,"15min":0.56799999999999995},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.64900000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575090,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1925120,"data_stored":918835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.44700000000000001},"min":{"1min":0.27500000000000002,"5min":0.26800000000000002,"15min":0.26500000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.81299999999999994},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.496,"15min":0.51400000000000001},"min":{"1min":0.34100000000000003,"5min":0.29899999999999999,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.874,"15min":0.874},"last":0.47099999999999997}]},{"osd":1,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44500000000000001,"15min":0.47299999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64900000000000002,"5min":0.83999999999999997,"15min":0.98699999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52900000000000003,"15min":0.55200000000000005},"min":{"1min":0.36899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.746,"5min":0.82699999999999996,"15min":0.96099999999999997},"last":0.435}]},{"osd":2,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53800000000000003,"15min":0.55500000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68100000000000005,"5min":0.879,"15min":0.879},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58999999999999997,"15min":0.61799999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79100000000000004,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.71599999999999997,"15min":0.75},"min":{"1min":0.505,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96399999999999997,"5min":1.034,"15min":1.288},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.74399999999999999,"15min":0.76300000000000001},"min":{"1min":0.52600000000000002,"5min":0.504,"15min":0.45800000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.46},"last":0.63200000000000001}]},{"osd":5,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.75,"15min":0.77200000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47099999999999997},"max":{"1min":0.98799999999999999,"5min":1.1040000000000001,"15min":1.3580000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.75700000000000001,"15min":0.78900000000000003},"min":{"1min":0.54200000000000004,"5min":0.50700000000000001,"15min":0.46899999999999997},"max":{"1min":1.046,"5min":1.1479999999999999,"15min":1.429},"last":0.625}]},{"osd":6,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.78200000000000003,"15min":0.80300000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.54200000000000004},"max":{"1min":1.0289999999999999,"5min":1.123,"15min":1.4730000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.75800000000000001,"15min":0.79700000000000004},"min":{"1min":0.52500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.3999999999999999},"last":0.66900000000000004}]},{"osd":7,"last update":"Fri Mar 29 10:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.77600000000000002,"15min":0.81799999999999995},"min":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93500000000000005,"5min":1.103,"15min":1.5029999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.087,"5min":1.1859999999999999,"15min":1.345},"last":0.752}]}]},{"osd":4,"up_from":13,"seq":55834575091,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59236,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312604,"statfs":{"total":96636764160,"available":96576106496,"internally_reserved":0,"allocated":1921024,"data_stored":918745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67800000000000005,"15min":0.68500000000000005},"min":{"1min":0.45000000000000001,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.86099999999999999,"5min":0.98799999999999999,"15min":1.417},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.40300000000000002,"5min":0.38,"15min":0.34000000000000002},"max":{"1min":0.88300000000000001,"5min":1.0269999999999999,"15min":1.165},"last":0.65500000000000003}]},{"osd":1,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68000000000000005,"15min":0.69699999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39000000000000001},"max":{"1min":0.872,"5min":0.97499999999999998,"15min":0.98399999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69299999999999995,"15min":0.70399999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"max":{"1min":0.91300000000000003,"5min":1.075,"15min":1.333},"last":0.57399999999999995}]},{"osd":2,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75800000000000001,"15min":0.77100000000000002},"min":{"1min":0.57599999999999996,"5min":0.433,"15min":0.433},"max":{"1min":0.87,"5min":1.014,"15min":1.03},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76900000000000002,"15min":0.76800000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.94599999999999995,"5min":1.0600000000000001,"15min":1.387},"last":0.749}]},{"osd":3,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70499999999999996,"15min":0.71899999999999997},"min":{"1min":0.48299999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.89300000000000002,"5min":0.97499999999999998,"15min":1.2350000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70999999999999996,"15min":0.72799999999999998},"min":{"1min":0.503,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.80700000000000005,"5min":1.0600000000000001,"15min":1.3180000000000001},"last":0.67800000000000005}]},{"osd":5,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.61199999999999999,"15min":0.58899999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38100000000000001},"max":{"1min":0.77100000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.63100000000000001,"15min":0.60699999999999998},"min":{"1min":0.41099999999999998,"5min":0.40200000000000002,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.98599999999999999,"15min":1.294},"last":0.85899999999999999}]},{"osd":6,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65300000000000002,"15min":0.624},"min":{"1min":0.46600000000000003,"5min":0.44,"15min":0.375},"max":{"1min":0.90000000000000002,"5min":1.026,"15min":1.026},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68400000000000005,"15min":0.66200000000000003},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.88400000000000001,"5min":1.093,"15min":1.1359999999999999},"last":0.84199999999999997}]},{"osd":7,"last update":"Fri Mar 29 10:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69799999999999995,"15min":0.68500000000000005},"min":{"1min":0.48199999999999998,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":0.85799999999999998,"5min":1.0580000000000001,"15min":1.345},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78100000000000003,"15min":0.76200000000000001},"min":{"1min":0.53500000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.93500000000000005,"5min":1.014,"15min":1.214},"last":0.82599999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1892352,"data_stored":905600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63300000000000001,"15min":0.67300000000000004},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.35699999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":1.1899999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.38100000000000001,"15min":0.33500000000000002},"max":{"1min":0.83899999999999997,"5min":0.997,"15min":1.2649999999999999},"last":0.73599999999999999}]},{"osd":1,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.71099999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96299999999999997,"5min":0.96599999999999997,"15min":0.96699999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64700000000000002,"15min":0.69399999999999995},"min":{"1min":0.46899999999999997,"5min":0.39900000000000002,"15min":0.39200000000000002},"max":{"1min":0.79500000000000004,"5min":0.94199999999999995,"15min":1.0669999999999999},"last":0.84899999999999998}]},{"osd":2,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69099999999999995,"15min":0.72599999999999998},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.92100000000000004,"5min":0.98599999999999999,"15min":0.98899999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.69999999999999996},"min":{"1min":0.498,"5min":0.46999999999999997,"15min":0.39700000000000002},"max":{"1min":0.90200000000000002,"5min":0.96199999999999997,"15min":1.095},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":0.72099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.113},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.746},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0980000000000001},"last":0.755}]},{"osd":4,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60299999999999998},"min":{"1min":0.40699999999999997,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":0.89200000000000002,"5min":0.91700000000000004,"15min":0.94599999999999995},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65700000000000003,"15min":0.64700000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.38900000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.097},"last":0.64100000000000001}]},{"osd":6,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65000000000000002,"15min":0.65400000000000003},"min":{"1min":0.433,"5min":0.41299999999999998,"15min":0.36799999999999999},"max":{"1min":1.083,"5min":1.083,"15min":1.151},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.67400000000000004,"15min":0.67900000000000005},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.77300000000000002}]},{"osd":7,"last update":"Fri Mar 29 10:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70099999999999996,"15min":0.70599999999999996},"min":{"1min":0.49299999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.42299999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.1990000000000001},"last":0.89100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T10:15:24.171 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T10:15:24.172 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T10:15:24.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.415+0000 7f482f200700 1 Processor -- start 2024-03-29T10:15:24.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.416+0000 7f482f200700 1 -- start start 2024-03-29T10:15:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.417+0000 7f482f200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f48280d21f0 0x7f48280ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.417+0000 7f482f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48280cec40 0x7f4828161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.417+0000 7f482f200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48280ce270 con 0x7f48281611b0 2024-03-29T10:15:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.417+0000 7f482f200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482815b600 con 0x7f48280d21f0 2024-03-29T10:15:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.417+0000 7f482f200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48280d2760 con 0x7f48280cec40 2024-03-29T10:15:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f481fe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f48280d21f0 0x7f48280ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f482d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48280cec40 0x7f4828161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f482d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48280cec40 0x7f4828161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:44148/0 (socket says 172.21.15.12:44148) 2024-03-29T10:15:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f481fe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f48280d21f0 0x7f48280ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:50542/0 (socket says 172.21.15.12:50542) 2024-03-29T10:15:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f4827e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f48281611b0 0x7f482815ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:54226/0 (socket says 172.21.15.12:54226) 2024-03-29T10:15:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f482d200700 1 -- 172.21.15.12:0/576979419 learned_addr learned my addr 172.21.15.12:0/576979419 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f482d200700 1 -- 172.21.15.12:0/576979419 >> v2:172.21.15.159:3300/0 conn(0x7f48280d21f0 msgr2=0x7f48280ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f482d200700 1 --2- 172.21.15.12:0/576979419 >> v2:172.21.15.159:3300/0 conn(0x7f48280d21f0 0x7f48280ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f482d200700 1 -- 172.21.15.12:0/576979419 >> v1:172.21.15.159:6789/0 conn(0x7f48281611b0 legacy=0x7f482815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f482d200700 1 -- 172.21.15.12:0/576979419 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48280d2a40 con 0x7f48280cec40 2024-03-29T10:15:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f481fe00700 1 --2- 172.21.15.12:0/576979419 >> v2:172.21.15.159:3300/0 conn(0x7f48280d21f0 0x7f48280ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:15:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.418+0000 7f482d200700 1 --2- 172.21.15.12:0/576979419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48280cec40 0x7f4828161590 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4820009e50 tx=0x7f482000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7598a5bf1af5bc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:24.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f4827400700 1 -- 172.21.15.12:0/576979419 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482000e070 con 0x7f48280cec40 2024-03-29T10:15:24.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f4827400700 1 -- 172.21.15.12:0/576979419 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482000b910 con 0x7f48280cec40 2024-03-29T10:15:24.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f4827400700 1 -- 172.21.15.12:0/576979419 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482000bcc0 con 0x7f48280cec40 2024-03-29T10:15:24.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f482f200700 1 -- 172.21.15.12:0/576979419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48280cec40 msgr2=0x7f4828161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f482f200700 1 --2- 172.21.15.12:0/576979419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48280cec40 0x7f4828161590 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4820009e50 tx=0x7f482000b330 comp rx=0 tx=0).stop 2024-03-29T10:15:24.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f482f200700 1 -- 172.21.15.12:0/576979419 shutdown_connections 2024-03-29T10:15:24.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f482f200700 1 --2- 172.21.15.12:0/576979419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48280cec40 0x7f4828161590 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f482f200700 1 --2- 172.21.15.12:0/576979419 >> v2:172.21.15.159:3300/0 conn(0x7f48280d21f0 0x7f48280ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f482f200700 1 -- 172.21.15.12:0/576979419 >> 172.21.15.12:0/576979419 conn(0x7f482808aff0 msgr2=0x7f48280d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:24.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f482f200700 1 -- 172.21.15.12:0/576979419 shutdown_connections 2024-03-29T10:15:24.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.419+0000 7f482f200700 1 -- 172.21.15.12:0/576979419 wait complete. 2024-03-29T10:15:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.421+0000 7f482f200700 1 Processor -- start 2024-03-29T10:15:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.421+0000 7f482f200700 1 -- start start 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.422+0000 7f482f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48280cec40 0x7f48280b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.422+0000 7f482f200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48280d21f0 0x7f4828150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.422+0000 7f482f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48281611b0 0x7f48280ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.422+0000 7f482f200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48280b7590 con 0x7f48280cec40 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.422+0000 7f482f200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48280b6980 con 0x7f48280d21f0 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.422+0000 7f482f200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48280b6c60 con 0x7f48281611b0 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f4827e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48280cec40 0x7f48280b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f4827e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48280cec40 0x7f48280b6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49902/0 (socket says 172.21.15.12:49902) 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f4827e00700 1 -- 172.21.15.12:0/509694199 learned_addr learned my addr 172.21.15.12:0/509694199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f482d200700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48281611b0 0x7f48280ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f481fe00700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48280d21f0 0x7f4828150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f4827e00700 1 -- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48281611b0 msgr2=0x7f48280ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f4827e00700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48281611b0 0x7f48280ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f4827e00700 1 -- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48280d21f0 msgr2=0x7f4828150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f4827e00700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48280d21f0 0x7f4828150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f4827e00700 1 -- 172.21.15.12:0/509694199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48100094a0 con 0x7f48280cec40 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f481fe00700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48280d21f0 0x7f4828150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.423+0000 7f4827e00700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48280cec40 0x7f48280b6150 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f481800c500 tx=0x7f481800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.424+0000 7f4824c00700 1 -- 172.21.15.12:0/509694199 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4818005cb0 con 0x7f48280cec40 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.424+0000 7f4824c00700 1 -- 172.21.15.12:0/509694199 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4818005e10 con 0x7f48280cec40 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.424+0000 7f482f200700 1 -- 172.21.15.12:0/509694199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4820009b30 con 0x7f48280cec40 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.424+0000 7f4824c00700 1 -- 172.21.15.12:0/509694199 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48180072d0 con 0x7f48280cec40 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.424+0000 7f482f200700 1 -- 172.21.15.12:0/509694199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48280b7220 con 0x7f48280cec40 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.425+0000 7f482f200700 1 -- 172.21.15.12:0/509694199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4828067720 con 0x7f48280cec40 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.426+0000 7f4824c00700 1 -- 172.21.15.12:0/509694199 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4818007430 con 0x7f48280cec40 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.427+0000 7f4824c00700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f480406adb0 0x7f480406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.427+0000 7f4824c00700 1 -- 172.21.15.12:0/509694199 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4818088780 con 0x7f48280cec40 2024-03-29T10:15:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.427+0000 7f481fe00700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f480406adb0 0x7f480406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.428+0000 7f481fe00700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f480406adb0 0x7f480406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f48280d1e70 tx=0x7f4810008040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.431+0000 7f4824c00700 1 -- 172.21.15.12:0/509694199 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4818053870 con 0x7f48280cec40 2024-03-29T10:15:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.730+0000 7f482f200700 1 -- 172.21.15.12:0/509694199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f48280b73e0 con 0x7f48280cec40 2024-03-29T10:15:24.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.735+0000 7f4824c00700 1 -- 172.21.15.12:0/509694199 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4818056e90 con 0x7f48280cec40 2024-03-29T10:15:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 -- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f480406adb0 msgr2=0x7f480406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f480406adb0 0x7f480406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f48280d1e70 tx=0x7f4810008040 comp rx=0 tx=0).stop 2024-03-29T10:15:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 -- 172.21.15.12:0/509694199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48280cec40 msgr2=0x7f48280b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48280cec40 0x7f48280b6150 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f481800c500 tx=0x7f481800c810 comp rx=0 tx=0).stop 2024-03-29T10:15:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 -- 172.21.15.12:0/509694199 shutdown_connections 2024-03-29T10:15:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f480406adb0 0x7f480406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48281611b0 0x7f48280ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48280cec40 0x7f48280b6150 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 --2- 172.21.15.12:0/509694199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48280d21f0 0x7f4828150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 -- 172.21.15.12:0/509694199 >> 172.21.15.12:0/509694199 conn(0x7f482808aff0 msgr2=0x7f482815da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 -- 172.21.15.12:0/509694199 shutdown_connections 2024-03-29T10:15:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.738+0000 7f482f200700 1 -- 172.21.15.12:0/509694199 wait complete. 2024-03-29T10:15:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T10:15:24.754 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T10:15:24.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.954+0000 7f5931400700 1 Processor -- start 2024-03-29T10:15:24.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.954+0000 7f5931400700 1 -- start start 2024-03-29T10:15:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.955+0000 7f5931400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f592c09aba0 0x7f592c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.955+0000 7f5931400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f592c06c480 0x7f592c0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.955+0000 7f5931400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592c06bb40 con 0x7f592c15d710 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.955+0000 7f5931400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592c06c910 con 0x7f592c06c480 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.955+0000 7f5931400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592c099b00 con 0x7f592c09aba0 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.956+0000 7f592b200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f592c06c480 0x7f592c0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.956+0000 7f592b200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f592c06c480 0x7f592c0ccbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49908/0 (socket says 172.21.15.12:49908) 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.956+0000 7f592b200700 1 -- 172.21.15.12:0/3740740332 learned_addr learned my addr 172.21.15.12:0/3740740332 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.956+0000 7f5929e00700 1 --2- 172.21.15.12:0/3740740332 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f592c09aba0 0x7f592c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.956+0000 7f592b200700 1 -- 172.21.15.12:0/3740740332 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f592c09aba0 msgr2=0x7f592c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.956+0000 7f592b200700 1 --2- 172.21.15.12:0/3740740332 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f592c09aba0 0x7f592c06bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.956+0000 7f592b200700 1 -- 172.21.15.12:0/3740740332 >> v1:172.21.15.159:6789/0 conn(0x7f592c15d710 legacy=0x7f592c0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.956+0000 7f592b200700 1 -- 172.21.15.12:0/3740740332 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c099de0 con 0x7f592c06c480 2024-03-29T10:15:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.956+0000 7f592b200700 1 --2- 172.21.15.12:0/3740740332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 0x7f592c0ccbe0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f5924009e50 tx=0x7f592400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb90dcabf90b30c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5929400700 1 -- 172.21.15.12:0/3740740332 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592400e070 con 0x7f592c06c480 2024-03-29T10:15:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5929400700 1 -- 172.21.15.12:0/3740740332 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592400b910 con 0x7f592c06c480 2024-03-29T10:15:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5929400700 1 -- 172.21.15.12:0/3740740332 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592400bcc0 con 0x7f592c06c480 2024-03-29T10:15:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5931400700 1 -- 172.21.15.12:0/3740740332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 msgr2=0x7f592c0ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5931400700 1 --2- 172.21.15.12:0/3740740332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 0x7f592c0ccbe0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f5924009e50 tx=0x7f592400b330 comp rx=0 tx=0).stop 2024-03-29T10:15:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5931400700 1 -- 172.21.15.12:0/3740740332 shutdown_connections 2024-03-29T10:15:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5931400700 1 --2- 172.21.15.12:0/3740740332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 0x7f592c0ccbe0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5931400700 1 --2- 172.21.15.12:0/3740740332 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f592c09aba0 0x7f592c06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5931400700 1 -- 172.21.15.12:0/3740740332 >> 172.21.15.12:0/3740740332 conn(0x7f592c0d13e0 msgr2=0x7f592c0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5931400700 1 -- 172.21.15.12:0/3740740332 shutdown_connections 2024-03-29T10:15:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.957+0000 7f5931400700 1 -- 172.21.15.12:0/3740740332 wait complete. 2024-03-29T10:15:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.960+0000 7f5931400700 1 Processor -- start 2024-03-29T10:15:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.960+0000 7f5931400700 1 -- start start 2024-03-29T10:15:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.961+0000 7f5931400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 0x7f592c150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.961+0000 7f5931400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f592c09aba0 0x7f592c14f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.961+0000 7f5931400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f592c15d710 0x7f592c15ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.961+0000 7f5931400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c15e4c0 con 0x7f592c06c480 2024-03-29T10:15:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.961+0000 7f5931400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c14fda0 con 0x7f592c09aba0 2024-03-29T10:15:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.961+0000 7f5931400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0c6700 con 0x7f592c15d710 2024-03-29T10:15:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.961+0000 7f592a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 0x7f592c150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.961+0000 7f592a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 0x7f592c150190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49924/0 (socket says 172.21.15.12:49924) 2024-03-29T10:15:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.961+0000 7f592a800700 1 -- 172.21.15.12:0/3311418932 learned_addr learned my addr 172.21.15.12:0/3311418932 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f5929e00700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f592c09aba0 0x7f592c14f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f592b200700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f592c15d710 0x7f592c15ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f592a800700 1 -- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f592c15d710 msgr2=0x7f592c15ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f592a800700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f592c15d710 0x7f592c15ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f592a800700 1 -- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f592c09aba0 msgr2=0x7f592c14f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f592a800700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f592c09aba0 0x7f592c14f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f592a800700 1 -- 172.21.15.12:0/3311418932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924009b30 con 0x7f592c06c480 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f592a800700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 0x7f592c150190 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f591c00c450 tx=0x7f591c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f5922a00700 1 -- 172.21.15.12:0/3311418932 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c0052d0 con 0x7f592c06c480 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f5922a00700 1 -- 172.21.15.12:0/3311418932 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c005430 con 0x7f592c06c480 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f5931400700 1 -- 172.21.15.12:0/3311418932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592c15db90 con 0x7f592c06c480 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f5922a00700 1 -- 172.21.15.12:0/3311418932 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c0038c0 con 0x7f592c06c480 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.962+0000 7f5931400700 1 -- 172.21.15.12:0/3311418932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592c15e150 con 0x7f592c06c480 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.964+0000 7f5931400700 1 -- 172.21.15.12:0/3311418932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592c067720 con 0x7f592c06c480 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.964+0000 7f5922a00700 1 -- 172.21.15.12:0/3311418932 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f591c003a20 con 0x7f592c06c480 2024-03-29T10:15:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.966+0000 7f5922a00700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f590c06adb0 0x7f590c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.966+0000 7f5922a00700 1 -- 172.21.15.12:0/3311418932 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f591c088810 con 0x7f592c06c480 2024-03-29T10:15:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.966+0000 7f5929e00700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f590c06adb0 0x7f590c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.967+0000 7f5929e00700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f590c06adb0 0x7f590c06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f592c0cd6b0 tx=0x7f5918008040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:24.970+0000 7f5922a00700 1 -- 172.21.15.12:0/3311418932 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f591c053970 con 0x7f592c06c480 2024-03-29T10:15:25.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.227+0000 7f5931400700 1 -- 172.21.15.12:0/3311418932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f592c0d0f20 con 0x7f592c06c480 2024-03-29T10:15:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.227+0000 7f5922a00700 1 -- 172.21.15.12:0/3311418932 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f591c017090 con 0x7f592c06c480 2024-03-29T10:15:25.225 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:25.226 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3149931506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3149931506},{"type":"v1","addr":"172.21.15.159:6835","nonce":3149931506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/967685480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":967685480},{"type":"v1","addr":"172.21.15.159:6837","nonce":967685480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8967,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4273394318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4273394318},{"type":"v1","addr":"172.21.15.12:6834","nonce":4273394318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:15:18.742186+0000","modified":"2024-03-29T10:15:22.226050+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8970},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8970":{"gid":8970,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/512120304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":512120304},{"type":"v1","addr":"172.21.15.12:6835","nonce":512120304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-29T10:15:25.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 -- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f590c06adb0 msgr2=0x7f590c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f590c06adb0 0x7f590c06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f592c0cd6b0 tx=0x7f5918008040 comp rx=0 tx=0).stop 2024-03-29T10:15:25.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 -- 172.21.15.12:0/3311418932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 msgr2=0x7f592c150190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 0x7f592c150190 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f591c00c450 tx=0x7f591c00c810 comp rx=0 tx=0).stop 2024-03-29T10:15:25.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 -- 172.21.15.12:0/3311418932 shutdown_connections 2024-03-29T10:15:25.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f590c06adb0 0x7f590c06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f592c15d710 0x7f592c15ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f592c06c480 0x7f592c150190 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 --2- 172.21.15.12:0/3311418932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f592c09aba0 0x7f592c14f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 -- 172.21.15.12:0/3311418932 >> 172.21.15.12:0/3311418932 conn(0x7f592c0d13e0 msgr2=0x7f592c0af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:25.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 -- 172.21.15.12:0/3311418932 shutdown_connections 2024-03-29T10:15:25.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.230+0000 7f5931400700 1 -- 172.21.15.12:0/3311418932 wait complete. 2024-03-29T10:15:25.229 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 73 2024-03-29T10:15:25.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T10:15:18.742186+0000', 'modified': '2024-03-29T10:15:22.226050+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8970}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8970': {'gid': 8970, 'name': 'c', 'rank': 0, 'incarnation': 69, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.12:6835/512120304', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6833', 'nonce': 512120304}, {'type': 'v1', 'addr': '172.21.15.12:6835', 'nonce': 512120304}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T10:15:25.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T10:15:25.648 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-29T10:15:25.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.852+0000 7f1060000700 1 Processor -- start 2024-03-29T10:15:25.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.852+0000 7f1060000700 1 -- start start 2024-03-29T10:15:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.853+0000 7f1060000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f105815d800 0x7f10580725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.853+0000 7f1060000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f10580ce230 0x7f10580ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.853+0000 7f1060000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10580b2070 0x7f10580b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.853+0000 7f1060000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10580b40f0 con 0x7f10580ce230 2024-03-29T10:15:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.853+0000 7f1060000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10580709c0 con 0x7f105815d800 2024-03-29T10:15:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.853+0000 7f1060000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1058070c80 con 0x7f10580b2070 2024-03-29T10:15:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.854+0000 7f105cc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f10580ce230 0x7f10580ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.854+0000 7f105cc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f10580ce230 0x7f10580ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49938/0 (socket says 172.21.15.12:49938) 2024-03-29T10:15:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.854+0000 7f105cc00700 1 -- 172.21.15.12:0/1988555081 learned_addr learned my addr 172.21.15.12:0/1988555081 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:25.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.854+0000 7f105d600700 1 --2- 172.21.15.12:0/1988555081 >> v2:172.21.15.159:3300/0 conn(0x7f105815d800 0x7f10580725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.854+0000 7f105e000700 1 --2- 172.21.15.12:0/1988555081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10580b2070 0x7f10580b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.854+0000 7f105cc00700 1 -- 172.21.15.12:0/1988555081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10580b2070 msgr2=0x7f10580b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.854+0000 7f105cc00700 1 --2- 172.21.15.12:0/1988555081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10580b2070 0x7f10580b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.854+0000 7f105cc00700 1 -- 172.21.15.12:0/1988555081 >> v2:172.21.15.159:3300/0 conn(0x7f105815d800 msgr2=0x7f10580725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.854+0000 7f105cc00700 1 --2- 172.21.15.12:0/1988555081 >> v2:172.21.15.159:3300/0 conn(0x7f105815d800 0x7f10580725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.854+0000 7f105cc00700 1 -- 172.21.15.12:0/1988555081 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058070f60 con 0x7f10580ce230 2024-03-29T10:15:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.855+0000 7f105cc00700 1 --2- 172.21.15.12:0/1988555081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10580ce230 0x7f10580ce6a0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f105806c100 tx=0x7f104400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5872477f0ba00908 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.855+0000 7f1057e00700 1 -- 172.21.15.12:0/1988555081 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104400db60 con 0x7f10580ce230 2024-03-29T10:15:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.855+0000 7f1057e00700 1 -- 172.21.15.12:0/1988555081 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104400dcc0 con 0x7f10580ce230 2024-03-29T10:15:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.855+0000 7f1057e00700 1 -- 172.21.15.12:0/1988555081 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044017630 con 0x7f10580ce230 2024-03-29T10:15:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.855+0000 7f1060000700 1 -- 172.21.15.12:0/1988555081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10580ce230 msgr2=0x7f10580ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.855+0000 7f1060000700 1 --2- 172.21.15.12:0/1988555081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10580ce230 0x7f10580ce6a0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f105806c100 tx=0x7f104400be70 comp rx=0 tx=0).stop 2024-03-29T10:15:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.856+0000 7f1060000700 1 -- 172.21.15.12:0/1988555081 shutdown_connections 2024-03-29T10:15:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.856+0000 7f1060000700 1 --2- 172.21.15.12:0/1988555081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10580b2070 0x7f10580b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.856+0000 7f1060000700 1 --2- 172.21.15.12:0/1988555081 >> v2:172.21.15.159:3300/0 conn(0x7f105815d800 0x7f10580725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.856+0000 7f1060000700 1 --2- 172.21.15.12:0/1988555081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10580ce230 0x7f10580ce6a0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.856+0000 7f1060000700 1 -- 172.21.15.12:0/1988555081 >> 172.21.15.12:0/1988555081 conn(0x7f10580b0d70 msgr2=0x7f10580b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:25.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.856+0000 7f1060000700 1 -- 172.21.15.12:0/1988555081 shutdown_connections 2024-03-29T10:15:25.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.856+0000 7f1060000700 1 -- 172.21.15.12:0/1988555081 wait complete. 2024-03-29T10:15:25.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.857+0000 7f1060000700 1 Processor -- start 2024-03-29T10:15:25.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.858+0000 7f1060000700 1 -- start start 2024-03-29T10:15:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f1060000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10580b2070 0x7f1058146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f1060000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10580ce230 0x7f1058099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f1060000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105815d800 0x7f105809a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f1060000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058099910 con 0x7f105815d800 2024-03-29T10:15:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f1060000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058099300 con 0x7f10580ce230 2024-03-29T10:15:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f1060000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105809ae70 con 0x7f10580b2070 2024-03-29T10:15:25.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105815d800 0x7f105809a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105cc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10580ce230 0x7f1058099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105815d800 0x7f105809a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49954/0 (socket says 172.21.15.12:49954) 2024-03-29T10:15:25.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105d600700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10580b2070 0x7f1058146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105e000700 1 -- 172.21.15.12:0/2941874055 learned_addr learned my addr 172.21.15.12:0/2941874055 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105e000700 1 -- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10580b2070 msgr2=0x7f1058146630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105e000700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10580b2070 0x7f1058146630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105e000700 1 -- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10580ce230 msgr2=0x7f1058099da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105e000700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10580ce230 0x7f1058099da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105e000700 1 -- 172.21.15.12:0/2941874055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104400b7f0 con 0x7f105815d800 2024-03-29T10:15:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.859+0000 7f105cc00700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10580ce230 0x7f1058099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:15:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.860+0000 7f105e000700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105815d800 0x7f105809a420 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f105000eed0 tx=0x7f105000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.860+0000 7f1055600700 1 -- 172.21.15.12:0/2941874055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050010040 con 0x7f105815d800 2024-03-29T10:15:25.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.860+0000 7f1055600700 1 -- 172.21.15.12:0/2941874055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105000bc80 con 0x7f105815d800 2024-03-29T10:15:25.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.860+0000 7f1055600700 1 -- 172.21.15.12:0/2941874055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050009e40 con 0x7f105815d800 2024-03-29T10:15:25.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.860+0000 7f1060000700 1 -- 172.21.15.12:0/2941874055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10580995a0 con 0x7f105815d800 2024-03-29T10:15:25.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.860+0000 7f1060000700 1 -- 172.21.15.12:0/2941874055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10580ad2f0 con 0x7f105815d800 2024-03-29T10:15:25.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.862+0000 7f1055600700 1 -- 172.21.15.12:0/2941874055 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1050014400 con 0x7f105815d800 2024-03-29T10:15:25.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.863+0000 7f1055600700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f103806adb0 0x7f103806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.864+0000 7f1055600700 1 -- 172.21.15.12:0/2941874055 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10500899f0 con 0x7f105815d800 2024-03-29T10:15:25.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.864+0000 7f105d600700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f103806adb0 0x7f103806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.865+0000 7f105d600700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f103806adb0 0x7f103806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f104c009b20 tx=0x7f104c009210 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:25.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.866+0000 7f1073b87b80 1 Processor -- start 2024-03-29T10:15:25.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.866+0000 7f1073b87b80 1 -- start start 2024-03-29T10:15:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafb2b830 0x55eeafb2bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafafebe0 0x55eeafb01a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb28a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f1073b87b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeafa76650 con 0x55eeafb2b830 2024-03-29T10:15:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f1073b87b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeafa57c30 con 0x55eeafafebe0 2024-03-29T10:15:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f1073b87b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eeafa55430 con 0x55eeafafcb60 2024-03-29T10:15:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafb2b830 0x55eeafb2bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafb2b830 0x55eeafb2bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49962/0 (socket says 172.21.15.12:49962) 2024-03-29T10:15:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105d600700 1 -- 172.21.15.12:0/3595205464 learned_addr learned my addr 172.21.15.12:0/3595205464 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105cc00700 1 --2- 172.21.15.12:0/3595205464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafafebe0 0x55eeafb01a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105e000700 1 --2- 172.21.15.12:0/3595205464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb28a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105d600700 1 -- 172.21.15.12:0/3595205464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 msgr2=0x55eeafb28a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105d600700 1 --2- 172.21.15.12:0/3595205464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb28a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105d600700 1 -- 172.21.15.12:0/3595205464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafafebe0 msgr2=0x55eeafb01a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105d600700 1 --2- 172.21.15.12:0/3595205464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafafebe0 0x55eeafb01a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105d600700 1 -- 172.21.15.12:0/3595205464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eeaf844790 con 0x55eeafb2b830 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.867+0000 7f105e000700 1 --2- 172.21.15.12:0/3595205464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb28a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.868+0000 7f105d600700 1 --2- 172.21.15.12:0/3595205464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafb2b830 0x55eeafb2bc10 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f104c004530 tx=0x7f104c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d90ee8b55f708dd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.868+0000 7f1049c00700 1 -- 172.21.15.12:0/3595205464 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c004560 con 0x55eeafb2b830 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.868+0000 7f1049c00700 1 -- 172.21.15.12:0/3595205464 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c01b070 con 0x55eeafb2b830 2024-03-29T10:15:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.868+0000 7f1049c00700 1 -- 172.21.15.12:0/3595205464 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c003af0 con 0x55eeafb2b830 2024-03-29T10:15:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.868+0000 7f1073b87b80 1 -- 172.21.15.12:0/3595205464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafb2b830 msgr2=0x55eeafb2bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.868+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3595205464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafb2b830 0x55eeafb2bc10 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f104c004530 tx=0x7f104c009f90 comp rx=0 tx=0).stop 2024-03-29T10:15:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.869+0000 7f1073b87b80 1 -- 172.21.15.12:0/3595205464 shutdown_connections 2024-03-29T10:15:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.869+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3595205464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb28a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.869+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3595205464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafb2b830 0x55eeafb2bc10 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.869+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3595205464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafafebe0 0x55eeafb01a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.869+0000 7f1073b87b80 1 -- 172.21.15.12:0/3595205464 >> 172.21.15.12:0/3595205464 conn(0x55eeafa5f6a0 msgr2=0x55eeafb46640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.869+0000 7f1073b87b80 1 -- 172.21.15.12:0/3595205464 shutdown_connections 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.869+0000 7f1073b87b80 1 -- 172.21.15.12:0/3595205464 wait complete. 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.869+0000 7f1073b87b80 1 -- start start 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.870+0000 7f1073b87b80 1 Processor -- start 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb2e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafaff980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafb2b830 0x55eeafaffec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f1073b87b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeafa50d20 con 0x55eeafafebe0 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f1073b87b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf9ec770 con 0x55eeafb2b830 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f1073b87b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eeafa0f6f0 con 0x55eeafafcb60 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f1073b87b80 10 client.0 _ll_register_callbacks cb 0x55eeaf9f65e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f105cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafaff980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f105cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafaff980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49966/0 (socket says 172.21.15.12:49966) 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f105cc00700 1 -- 172.21.15.12:0/1410346211 learned_addr learned my addr 172.21.15.12:0/1410346211 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f105e000700 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafb2b830 0x55eeafaffec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f105d600700 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb2e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f105cc00700 1 -- 172.21.15.12:0/1410346211 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 msgr2=0x55eeafb2e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f105cc00700 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb2e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f105cc00700 1 -- 172.21.15.12:0/1410346211 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafb2b830 msgr2=0x55eeafaffec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f105cc00700 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafb2b830 0x55eeafaffec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.871+0000 7f105cc00700 1 -- 172.21.15.12:0/1410346211 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104c0041e0 con 0x55eeafafebe0 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.872+0000 7f105cc00700 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafaff980 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f104400beb0 tx=0x7f10440047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768ce8345f9041e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.872+0000 7f103e000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.872+0000 7f103e000700 1 -- 172.21.15.12:0/1410346211 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10440177c0 con 0x55eeafafebe0 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.872+0000 7f103e000700 1 -- 172.21.15.12:0/1410346211 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1044017920 con 0x55eeafafebe0 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.872+0000 7f1073b87b80 1 -- 172.21.15.12:0/1410346211 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eeafaf3920 con 0x55eeafafebe0 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.872+0000 7f103e000700 1 -- 172.21.15.12:0/1410346211 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1044017ae0 con 0x55eeafafebe0 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.872+0000 7f1073b87b80 10 client.9096 fetch_fsmap learned FSMap version 73 2024-03-29T10:15:25.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.872+0000 7f1073b87b80 1 -- 172.21.15.12:0/1410346211 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x55eeaf9310a0 con 0x55eeafafebe0 2024-03-29T10:15:25.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.873+0000 7f103e000700 1 -- 172.21.15.12:0/1410346211 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55eeafaf3920 con 0x55eeafafebe0 2024-03-29T10:15:25.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.873+0000 7f103e000700 1 -- 172.21.15.12:0/1410346211 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 73) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f104401f050 con 0x55eeafafebe0 2024-03-29T10:15:25.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.873+0000 7f1073b87b80 10 client.9096 fetch_fsmap finished waiting for FSMap version 73 2024-03-29T10:15:25.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.873+0000 7f1073b87b80 10 client.9096 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T10:15:25.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.873+0000 7f1073b87b80 20 client.9096 populate_metadata read hostname 'smithi012' 2024-03-29T10:15:25.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.873+0000 7f1073b87b80 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.12:6833/512120304,v1:172.21.15.12:6835/512120304] conn(0x55eeafb55cb0 0x55eeafb2c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.873+0000 7f1073b87b80 4 client.9096 mds_command: new command op to 8970 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:15:25.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.873+0000 7f1073b87b80 1 -- 172.21.15.12:0/1410346211 --> [v2:172.21.15.12:6833/512120304,v1:172.21.15.12:6835/512120304] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eeafad06f0 con 0x55eeafb55cb0 2024-03-29T10:15:25.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.873+0000 7f105d600700 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.12:6833/512120304,v1:172.21.15.12:6835/512120304] conn(0x55eeafb55cb0 0x55eeafb2c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.874+0000 7f105d600700 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.12:6833/512120304,v1:172.21.15.12:6835/512120304] conn(0x55eeafb55cb0 0x55eeafb2c7e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8fe30b84c6bede7 server_cookie=af07f528aac357ef in_seq=0 out_seq=0 2024-03-29T10:15:25.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.874+0000 7f103e000700 10 client.9096 ms_handle_connect on v2:172.21.15.12:6833/512120304 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.874+0000 7f103e000700 1 -- 172.21.15.12:0/1410346211 <== mds.0 v2:172.21.15.12:6833/512120304 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55eeafad06f0 con 0x55eeafb55cb0 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.874+0000 7f103e000700 10 client.9096 handle_command_reply: tid=0 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.874+0000 7f1073b87b80 1 client.9096 shutdown 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 1 -- 172.21.15.12:0/1410346211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 msgr2=0x55eeafaff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafaff980 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f104400beb0 tx=0x7f10440047c0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 1 -- 172.21.15.12:0/1410346211 shutdown_connections 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.12:6833/512120304,v1:172.21.15.12:6835/512120304] conn(0x55eeafb55cb0 0x55eeafb2c7e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafb2b830 0x55eeafaffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb2e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 1 --2- 172.21.15.12:0/1410346211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafaff980 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f103e000700 0 client.9096 ms_handle_reset on v2:172.21.15.12:6833/512120304 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 1 -- 172.21.15.12:0/1410346211 >> 172.21.15.12:0/1410346211 conn(0x55eeafa5f6a0 msgr2=0x55eeafaff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 1 -- 172.21.15.12:0/1410346211 shutdown_connections 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 1 -- 172.21.15.12:0/1410346211 wait complete. 2024-03-29T10:15:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.875+0000 7f1073b87b80 20 client.9096 trim_cache size 0 max 16384 2024-03-29T10:15:25.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.894+0000 7f1073b87b80 1 Processor -- start 2024-03-29T10:15:25.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.894+0000 7f1073b87b80 1 -- start start 2024-03-29T10:15:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.894+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb2a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.894+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafb2a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.894+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafb2b830 0x55eeafb42580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.894+0000 7f1073b87b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf845390 con 0x55eeafafebe0 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.894+0000 7f1073b87b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeafb2e3f0 con 0x55eeafb2b830 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.894+0000 7f1073b87b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eeafafdc10 con 0x55eeafafcb60 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafb2a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafb2a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49968/0 (socket says 172.21.15.12:49968) 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105cc00700 1 -- 172.21.15.12:0/3464204869 learned_addr learned my addr 172.21.15.12:0/3464204869 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105d600700 1 --2- 172.21.15.12:0/3464204869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb2a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105e000700 1 --2- 172.21.15.12:0/3464204869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafb2b830 0x55eeafb42580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105cc00700 1 -- 172.21.15.12:0/3464204869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 msgr2=0x55eeafb2a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105cc00700 1 --2- 172.21.15.12:0/3464204869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb2a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105cc00700 1 -- 172.21.15.12:0/3464204869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafb2b830 msgr2=0x55eeafb42580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105cc00700 1 --2- 172.21.15.12:0/3464204869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafb2b830 0x55eeafb42580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105cc00700 1 -- 172.21.15.12:0/3464204869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eeafb2bf80 con 0x55eeafafebe0 2024-03-29T10:15:25.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f105cc00700 1 --2- 172.21.15.12:0/3464204869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafb2a5a0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f1044006ab0 tx=0x7f1044022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c471a475f005461a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:25.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.895+0000 7f104a600700 1 -- 172.21.15.12:0/3464204869 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10440177c0 con 0x55eeafafebe0 2024-03-29T10:15:25.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f104a600700 1 -- 172.21.15.12:0/3464204869 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1044017920 con 0x55eeafafebe0 2024-03-29T10:15:25.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f104a600700 1 -- 172.21.15.12:0/3464204869 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044017a90 con 0x55eeafafebe0 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f1073b87b80 1 -- 172.21.15.12:0/3464204869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 msgr2=0x55eeafb2a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3464204869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafb2a5a0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f1044006ab0 tx=0x7f1044022000 comp rx=0 tx=0).stop 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f1073b87b80 1 -- 172.21.15.12:0/3464204869 shutdown_connections 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3464204869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafb2b830 0x55eeafb42580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3464204869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafafcb60 0x55eeafb2a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3464204869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafb2a5a0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f1073b87b80 1 -- 172.21.15.12:0/3464204869 >> 172.21.15.12:0/3464204869 conn(0x55eeafa5f6a0 msgr2=0x55eeafb68220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f1073b87b80 1 -- 172.21.15.12:0/3464204869 shutdown_connections 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.896+0000 7f1073b87b80 1 -- 172.21.15.12:0/3464204869 wait complete. 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.897+0000 7f1073b87b80 1 -- start start 2024-03-29T10:15:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.897+0000 7f1073b87b80 1 Processor -- start 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafafcb60 0x55eeafbac900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafafb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f1073b87b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafb2b830 0x55eeafafb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f1073b87b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeafb3c630 con 0x55eeafafebe0 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f1073b87b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeafb2d0b0 con 0x55eeafafcb60 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f1073b87b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eeafb3c070 con 0x55eeafb2b830 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f1073b87b80 10 client.0 _ll_register_callbacks cb 0x7f104400b240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f105cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafafb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f105cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafafb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49970/0 (socket says 172.21.15.12:49970) 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f105cc00700 1 -- 172.21.15.12:0/3461471690 learned_addr learned my addr 172.21.15.12:0/3461471690 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.898+0000 7f105d600700 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafafcb60 0x55eeafbac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f105e000700 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafb2b830 0x55eeafafb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f105cc00700 1 -- 172.21.15.12:0/3461471690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafb2b830 msgr2=0x55eeafafb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f105cc00700 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafb2b830 0x55eeafafb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f105cc00700 1 -- 172.21.15.12:0/3461471690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafafcb60 msgr2=0x55eeafbac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f105cc00700 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafafcb60 0x55eeafbac900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f105cc00700 1 -- 172.21.15.12:0/3461471690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eeaf9310a0 con 0x55eeafafebe0 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f105cc00700 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafafb360 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f104400d6d0 tx=0x7f104400b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ec622cfa94df36 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f103e000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f103e000700 1 -- 172.21.15.12:0/3461471690 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10440177c0 con 0x55eeafafebe0 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f103e000700 1 -- 172.21.15.12:0/3461471690 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1044000e40 con 0x55eeafafebe0 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.899+0000 7f1073b87b80 1 -- 172.21.15.12:0/3461471690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eeafafe4c0 con 0x55eeafafebe0 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f103e000700 1 -- 172.21.15.12:0/3461471690 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1044017e00 con 0x55eeafafebe0 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f1073b87b80 10 client.9102 fetch_fsmap learned FSMap version 73 2024-03-29T10:15:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f1073b87b80 1 -- 172.21.15.12:0/3461471690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x55eeafa17010 con 0x55eeafafebe0 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f103e000700 1 -- 172.21.15.12:0/3461471690 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55eeafafe4c0 con 0x55eeafafebe0 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f103e000700 1 -- 172.21.15.12:0/3461471690 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 73) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1044048050 con 0x55eeafafebe0 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f1073b87b80 10 client.9102 fetch_fsmap finished waiting for FSMap version 73 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f1073b87b80 10 client.9102 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f1073b87b80 20 client.9102 populate_metadata read hostname 'smithi012' 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.12:6833/512120304,v1:172.21.15.12:6835/512120304] conn(0x55eeafb00200 0x55eeafb005e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f1073b87b80 4 client.9102 mds_command: new command op to 8970 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f1073b87b80 1 -- 172.21.15.12:0/3461471690 --> [v2:172.21.15.12:6833/512120304,v1:172.21.15.12:6835/512120304] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f104c01b070 con 0x55eeafb00200 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.900+0000 7f105d600700 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.12:6833/512120304,v1:172.21.15.12:6835/512120304] conn(0x55eeafb00200 0x55eeafb005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f103e000700 10 client.9102 ms_handle_connect on v2:172.21.15.12:6833/512120304 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f105d600700 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.12:6833/512120304,v1:172.21.15.12:6835/512120304] conn(0x55eeafb00200 0x55eeafb005e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0bac8360a77478 server_cookie=ad895a5172ee8f41 in_seq=0 out_seq=0 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f103e000700 1 -- 172.21.15.12:0/3461471690 <== mds.0 v2:172.21.15.12:6833/512120304 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f104c01b070 con 0x55eeafb00200 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f103e000700 10 client.9102 handle_command_reply: tid=0 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 client.9102 shutdown 2024-03-29T10:15:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 -- 172.21.15.12:0/3461471690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 msgr2=0x55eeafafb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafafb360 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f104400d6d0 tx=0x7f104400b4d0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 -- 172.21.15.12:0/3461471690 shutdown_connections 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.12:6833/512120304,v1:172.21.15.12:6835/512120304] conn(0x55eeafb00200 0x55eeafb005e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55eeafb2b830 0x55eeafafb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55eeafafcb60 0x55eeafbac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 --2- 172.21.15.12:0/3461471690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eeafafebe0 0x55eeafafb360 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 -- 172.21.15.12:0/3461471690 >> 172.21.15.12:0/3461471690 conn(0x55eeafa5f6a0 msgr2=0x55eeafb3d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f103e000700 0 client.9102 ms_handle_reset on v2:172.21.15.12:6833/512120304 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 -- 172.21.15.12:0/3461471690 shutdown_connections 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.901+0000 7f1073b87b80 1 -- 172.21.15.12:0/3461471690 wait complete. 2024-03-29T10:15:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1073b87b80 20 client.9102 trim_cache size 0 max 16384 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 8970, 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1572474020000003, 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 73, 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 154, 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 154, 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 18.179577231 2024-03-29T10:15:25.903 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T10:15:25.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 -- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f103806adb0 msgr2=0x7f103806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f103806adb0 0x7f103806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f104c009b20 tx=0x7f104c009210 comp rx=0 tx=0).stop 2024-03-29T10:15:25.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 -- 172.21.15.12:0/2941874055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105815d800 msgr2=0x7f105809a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:25.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105815d800 0x7f105809a420 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f105000eed0 tx=0x7f105000b390 comp rx=0 tx=0).stop 2024-03-29T10:15:25.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 -- 172.21.15.12:0/2941874055 shutdown_connections 2024-03-29T10:15:25.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f103806adb0 0x7f103806d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105815d800 0x7f105809a420 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10580b2070 0x7f1058146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 --2- 172.21.15.12:0/2941874055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10580ce230 0x7f1058099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:25.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 -- 172.21.15.12:0/2941874055 >> 172.21.15.12:0/2941874055 conn(0x7f10580b0d70 msgr2=0x7f10580b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:25.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 -- 172.21.15.12:0/2941874055 shutdown_connections 2024-03-29T10:15:25.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:25.902+0000 7f1060000700 1 -- 172.21.15.12:0/2941874055 wait complete. 2024-03-29T10:15:25.917 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T10:15:26.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.117+0000 7f45d8400700 1 Processor -- start 2024-03-29T10:15:26.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.117+0000 7f45d8400700 1 -- start start 2024-03-29T10:15:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.118+0000 7f45d8400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f45d015d640 0x7f45d00d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.118+0000 7f45d8400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f45d00ceec0 0x7f45d01612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d8400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d0161800 0x7f45d00cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d8400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d0161c70 con 0x7f45d015d640 2024-03-29T10:15:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d8400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d00d30d0 con 0x7f45d00ceec0 2024-03-29T10:15:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d8400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00b6840 con 0x7f45d0161800 2024-03-29T10:15:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d5a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f45d015d640 0x7f45d00d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d5a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f45d015d640 0x7f45d00d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49980/0 (socket says 172.21.15.12:49980) 2024-03-29T10:15:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d5a00700 1 -- 172.21.15.12:0/281529357 learned_addr learned my addr 172.21.15.12:0/281529357 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:26.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d5000700 1 --2- 172.21.15.12:0/281529357 >> v2:172.21.15.159:3300/0 conn(0x7f45d00ceec0 0x7f45d01612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:26.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d6400700 1 --2- 172.21.15.12:0/281529357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d0161800 0x7f45d00cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d5a00700 1 -- 172.21.15.12:0/281529357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d0161800 msgr2=0x7f45d00cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d5a00700 1 --2- 172.21.15.12:0/281529357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d0161800 0x7f45d00cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d5a00700 1 -- 172.21.15.12:0/281529357 >> v2:172.21.15.159:3300/0 conn(0x7f45d00ceec0 msgr2=0x7f45d01612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d5a00700 1 --2- 172.21.15.12:0/281529357 >> v2:172.21.15.159:3300/0 conn(0x7f45d00ceec0 0x7f45d01612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d5a00700 1 -- 172.21.15.12:0/281529357 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d00d2880 con 0x7f45d015d640 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d5000700 1 --2- 172.21.15.12:0/281529357 >> v2:172.21.15.159:3300/0 conn(0x7f45d00ceec0 0x7f45d01612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.119+0000 7f45d6400700 1 --2- 172.21.15.12:0/281529357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d0161800 0x7f45d00cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45d5a00700 1 --2- 172.21.15.12:0/281529357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 0x7f45d00d2340 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f45d00cfbf0 tx=0x7f45c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc3377f13d4ab97 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45cfe00700 1 -- 172.21.15.12:0/281529357 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c400b870 con 0x7f45d015d640 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45cfe00700 1 -- 172.21.15.12:0/281529357 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c400b9d0 con 0x7f45d015d640 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45cfe00700 1 -- 172.21.15.12:0/281529357 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c400bd60 con 0x7f45d015d640 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45d8400700 1 -- 172.21.15.12:0/281529357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 msgr2=0x7f45d00d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45d8400700 1 --2- 172.21.15.12:0/281529357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 0x7f45d00d2340 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f45d00cfbf0 tx=0x7f45c4009bd0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45d8400700 1 -- 172.21.15.12:0/281529357 shutdown_connections 2024-03-29T10:15:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45d8400700 1 --2- 172.21.15.12:0/281529357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d0161800 0x7f45d00cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45d8400700 1 --2- 172.21.15.12:0/281529357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 0x7f45d00d2340 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45d8400700 1 --2- 172.21.15.12:0/281529357 >> v2:172.21.15.159:3300/0 conn(0x7f45d00ceec0 0x7f45d01612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.120+0000 7f45d8400700 1 -- 172.21.15.12:0/281529357 >> 172.21.15.12:0/281529357 conn(0x7f45d008aff0 msgr2=0x7f45d00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:26.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.121+0000 7f45d8400700 1 -- 172.21.15.12:0/281529357 shutdown_connections 2024-03-29T10:15:26.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.121+0000 7f45d8400700 1 -- 172.21.15.12:0/281529357 wait complete. 2024-03-29T10:15:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.122+0000 7f45d8400700 1 Processor -- start 2024-03-29T10:15:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.123+0000 7f45d8400700 1 -- start start 2024-03-29T10:15:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.123+0000 7f45d8400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d00ceec0 0x7f45d00b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.123+0000 7f45d8400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 0x7f45d00cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.123+0000 7f45d8400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45d0161800 0x7f45d01f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.123+0000 7f45d8400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00b4e00 con 0x7f45d015d640 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.123+0000 7f45d8400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00ce4d0 con 0x7f45d0161800 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.123+0000 7f45d8400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00ce7b0 con 0x7f45d00ceec0 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d5000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 0x7f45d00cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d5000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 0x7f45d00cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49984/0 (socket says 172.21.15.12:49984) 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d5000700 1 -- 172.21.15.12:0/459794035 learned_addr learned my addr 172.21.15.12:0/459794035 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d5a00700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d00ceec0 0x7f45d00b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d6400700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45d0161800 0x7f45d01f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d5000700 1 -- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d00ceec0 msgr2=0x7f45d00b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d5000700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d00ceec0 0x7f45d00b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d5000700 1 -- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45d0161800 msgr2=0x7f45d01f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:26.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d5000700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45d0161800 0x7f45d01f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d5000700 1 -- 172.21.15.12:0/459794035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c4009550 con 0x7f45d015d640 2024-03-29T10:15:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.124+0000 7f45d6400700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45d0161800 0x7f45d01f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.125+0000 7f45d5000700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 0x7f45d00cdba0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00c7f0 tx=0x7f45bc00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.125+0000 7f45cd600700 1 -- 172.21.15.12:0/459794035 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45bc011800 con 0x7f45d015d640 2024-03-29T10:15:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.125+0000 7f45cd600700 1 -- 172.21.15.12:0/459794035 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45bc011960 con 0x7f45d015d640 2024-03-29T10:15:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.125+0000 7f45cd600700 1 -- 172.21.15.12:0/459794035 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45bc011cf0 con 0x7f45d015d640 2024-03-29T10:15:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.125+0000 7f45d8400700 1 -- 172.21.15.12:0/459794035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d00cea90 con 0x7f45d015d640 2024-03-29T10:15:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.125+0000 7f45d8400700 1 -- 172.21.15.12:0/459794035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d00ca9a0 con 0x7f45d015d640 2024-03-29T10:15:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.126+0000 7f45d8400700 1 -- 172.21.15.12:0/459794035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d00cab90 con 0x7f45d015d640 2024-03-29T10:15:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.127+0000 7f45cd600700 1 -- 172.21.15.12:0/459794035 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f45bc018360 con 0x7f45d015d640 2024-03-29T10:15:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.128+0000 7f45cd600700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45b006adb0 0x7f45b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.128+0000 7f45cd600700 1 -- 172.21.15.12:0/459794035 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45bc089590 con 0x7f45d015d640 2024-03-29T10:15:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.128+0000 7f45d5a00700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45b006adb0 0x7f45b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.129+0000 7f45d5a00700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45b006adb0 0x7f45b006d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f45c4000c00 tx=0x7f45c4008030 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.132+0000 7f45cd600700 1 -- 172.21.15.12:0/459794035 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45bc0544c0 con 0x7f45d015d640 2024-03-29T10:15:26.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.397+0000 7f45d8400700 1 -- 172.21.15.12:0/459794035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f45d0067720 con 0x7f45d015d640 2024-03-29T10:15:26.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.397+0000 7f45cd600700 1 -- 172.21.15.12:0/459794035 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f45bc057ae0 con 0x7f45d015d640 2024-03-29T10:15:26.394 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T10:15:26.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 -- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45b006adb0 msgr2=0x7f45b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:26.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45b006adb0 0x7f45b006d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f45c4000c00 tx=0x7f45c4008030 comp rx=0 tx=0).stop 2024-03-29T10:15:26.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 -- 172.21.15.12:0/459794035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 msgr2=0x7f45d00cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:26.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 0x7f45d00cdba0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00c7f0 tx=0x7f45bc00cb00 comp rx=0 tx=0).stop 2024-03-29T10:15:26.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 -- 172.21.15.12:0/459794035 shutdown_connections 2024-03-29T10:15:26.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f45b006adb0 0x7f45b006d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45d0161800 0x7f45d01f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45d00ceec0 0x7f45d00b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 --2- 172.21.15.12:0/459794035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d015d640 0x7f45d00cdba0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:26.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 -- 172.21.15.12:0/459794035 >> 172.21.15.12:0/459794035 conn(0x7f45d008aff0 msgr2=0x7f45d015cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:26.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 -- 172.21.15.12:0/459794035 shutdown_connections 2024-03-29T10:15:26.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:26.400+0000 7f45d8400700 1 -- 172.21.15.12:0/459794035 wait complete. 2024-03-29T10:15:26.416 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T10:15:26.416 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T10:15:26.416 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T10:15:26.416 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T10:15:26.416 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T10:15:26.416 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T10:15:26.416 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T10:15:26.416 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T10:15:26.416 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:15:26.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:15:26.417 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T10:15:26.462 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T10:15:26.462 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T10:15:26.462 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T10:15:26.462 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:15:26.462 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T10:15:26.462 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:15:26.462 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:15:26.462 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:15:26.462 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T10:15:26.462 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 40191sec preferred_lft 40191sec 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T10:15:26.463 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T10:15:26.464 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T10:15:26.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:15:26.465 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T10:15:26.569 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T10:15:26.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:15:26.573 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T10:15:26.620 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T10:15:26.620 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T10:15:26.620 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T10:15:26.620 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T10:15:26.620 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T10:15:26.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:15:26.621 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T10:15:26.621 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T10:15:26.621 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T10:15:26.621 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T10:15:26.621 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T10:15:26.621 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T10:15:26.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:15:26.788 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T10:15:26.832 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T10:15:26.833 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T10:15:26.834 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:15:26.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:15:26.834 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T10:15:26.945 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:15:26.945 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T10:15:26.989 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T10:15:26.989 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T10:15:26.989 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:15:27.072 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T10:15:27.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:15:27.073 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:15:27.215 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:15:27.216 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:15:27.243 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T10:15:27.244 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T10:15:27.244 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T10:15:27.244 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T10:15:27.244 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T10:15:27.249 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T10:15:27.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:15:27.249 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T10:15:27.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:15:27.317 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T10:15:27.415 INFO:teuthology.orchestra.run.smithi012.stdout:client7730 2024-03-29T10:15:27.416 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T10:15:27.416 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T10:15:27.460 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T10:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.702+0000 7ff363e00700 1 Processor -- start 2024-03-29T10:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.702+0000 7ff363e00700 1 -- start start 2024-03-29T10:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.703+0000 7ff363e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff35c0cd020 0x7ff35c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.704+0000 7ff363e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0b2070 0x7ff35c0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.704+0000 7ff363e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0cdae0 con 0x7ff35c0cbb30 2024-03-29T10:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.704+0000 7ff363e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c0cddc0 con 0x7ff35c0cd020 2024-03-29T10:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.704+0000 7ff363e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0ce0a0 con 0x7ff35c0b2070 2024-03-29T10:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.704+0000 7ff361400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff35c0cbb30 0x7ff35c0cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43640/0 (socket says 172.21.15.12:43640) 2024-03-29T10:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.704+0000 7ff361400700 1 -- 172.21.15.12:0/1140806382 learned_addr learned my addr 172.21.15.12:0/1140806382 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.704+0000 7ff360a00700 1 --2- 172.21.15.12:0/1140806382 >> v2:172.21.15.159:3300/0 conn(0x7ff35c0cd020 0x7ff35c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.704+0000 7ff361e00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0b2070 0x7ff35c0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.704+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023774809 0 0) 0x7ff35c0cdae0 con 0x7ff35c0cbb30 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.704+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3440035b0 con 0x7ff35c0cbb30 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3345971349 0 0) 0x7ff3440035b0 con 0x7ff35c0cbb30 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0b2070 msgr2=0x7ff35c0b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0b2070 0x7ff35c0b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 >> v2:172.21.15.159:3300/0 conn(0x7ff35c0cd020 msgr2=0x7ff35c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 --2- 172.21.15.12:0/1140806382 >> v2:172.21.15.159:3300/0 conn(0x7ff35c0cd020 0x7ff35c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c06dc20 con 0x7ff35c0cbb30 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7ff350001670 con 0x7ff35c0cbb30 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 >> v1:172.21.15.12:6789/0 conn(0x7ff35c0cbb30 legacy=0x7ff35c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0b2070 0x7ff344003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff344003fd0 0x7ff344004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff344004d90 0x7ff344005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0cddc0 con 0x7ff344004d90 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0ce0a0 con 0x7ff344003fd0 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34400ba10 con 0x7ff35c0b2070 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff361e00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff344004d90 0x7ff344005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff360a00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff344003fd0 0x7ff344004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.705+0000 7ff361400700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0b2070 0x7ff344003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff361e00700 1 -- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0b2070 msgr2=0x7ff344003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff361e00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0b2070 0x7ff344003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff361e00700 1 -- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff344003fd0 msgr2=0x7ff344004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff361e00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff344003fd0 0x7ff344004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff361e00700 1 -- 172.21.15.12:0/1140806382 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3440035b0 con 0x7ff344004d90 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff360a00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff344003fd0 0x7ff344004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff361400700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0b2070 0x7ff344003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff361e00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff344004d90 0x7ff344005210 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7ff35400e170 tx=0x7ff344005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fbb490e82a262e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35400ec90 con 0x7ff344004d90 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35400f040 con 0x7ff344004d90 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.706+0000 7ff35be00700 1 -- 172.21.15.12:0/1140806382 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354014480 con 0x7ff344004d90 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.707+0000 7ff363e00700 1 -- 172.21.15.12:0/1140806382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff344004d90 msgr2=0x7ff344005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.707+0000 7ff363e00700 1 --2- 172.21.15.12:0/1140806382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff344004d90 0x7ff344005210 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7ff35400e170 tx=0x7ff344005880 comp rx=0 tx=0).stop 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.707+0000 7ff361e00700 1 -- 172.21.15.12:0/1140806382 reap_dead start 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.707+0000 7ff363e00700 1 -- 172.21.15.12:0/1140806382 shutdown_connections 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.707+0000 7ff363e00700 1 -- 172.21.15.12:0/1140806382 >> 172.21.15.12:0/1140806382 conn(0x7ff35c0b0d70 msgr2=0x7ff35c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.707+0000 7ff363e00700 1 -- 172.21.15.12:0/1140806382 shutdown_connections 2024-03-29T10:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.707+0000 7ff363e00700 1 -- 172.21.15.12:0/1140806382 wait complete. 2024-03-29T10:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.709+0000 7ff363e00700 1 Processor -- start 2024-03-29T10:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.709+0000 7ff363e00700 1 -- start start 2024-03-29T10:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff363e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff344004d90 0x7ff35c14e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff363e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0ac050 0x7ff35c157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff363e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35c0af8b0 0x7ff35c0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff363e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0ad240 con 0x7ff35c0af8b0 2024-03-29T10:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff363e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0b0270 con 0x7ff344004d90 2024-03-29T10:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff363e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0ac5c0 con 0x7ff35c0ac050 2024-03-29T10:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff361e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35c0af8b0 0x7ff35c0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff361e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35c0af8b0 0x7ff35c0afd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50008/0 (socket says 172.21.15.12:50008) 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff361e00700 1 -- 172.21.15.12:0/1168280284 learned_addr learned my addr 172.21.15.12:0/1168280284 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff361400700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff344004d90 0x7ff35c14e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff360a00700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0ac050 0x7ff35c157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff361e00700 1 -- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0ac050 msgr2=0x7ff35c157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff361e00700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0ac050 0x7ff35c157740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff361e00700 1 -- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff344004d90 msgr2=0x7ff35c14e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff361e00700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff344004d90 0x7ff35c14e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.711+0000 7ff361e00700 1 -- 172.21.15.12:0/1168280284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0ac8a0 con 0x7ff35c0af8b0 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.712+0000 7ff361e00700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35c0af8b0 0x7ff35c0afd30 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7ff354016000 tx=0x7ff354009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.712+0000 7ff359600700 1 -- 172.21.15.12:0/1168280284 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354007000 con 0x7ff35c0af8b0 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.712+0000 7ff359600700 1 -- 172.21.15.12:0/1168280284 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff354003e30 con 0x7ff35c0af8b0 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.712+0000 7ff359600700 1 -- 172.21.15.12:0/1168280284 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354014c20 con 0x7ff35c0af8b0 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.712+0000 7ff363e00700 1 -- 172.21.15.12:0/1168280284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35c0acb80 con 0x7ff35c0af8b0 2024-03-29T10:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.712+0000 7ff363e00700 1 -- 172.21.15.12:0/1168280284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c157c80 con 0x7ff35c0af8b0 2024-03-29T10:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.713+0000 7ff363e00700 1 -- 172.21.15.12:0/1168280284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35c0ad380 con 0x7ff35c0af8b0 2024-03-29T10:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.714+0000 7ff359600700 1 -- 172.21.15.12:0/1168280284 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff354023020 con 0x7ff35c0af8b0 2024-03-29T10:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.715+0000 7ff359600700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff33c06adb0 0x7ff33c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.715+0000 7ff359600700 1 -- 172.21.15.12:0/1168280284 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff35408aa50 con 0x7ff35c0af8b0 2024-03-29T10:15:27.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.715+0000 7ff361400700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff33c06adb0 0x7ff33c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:27.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.716+0000 7ff361400700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff33c06adb0 0x7ff33c06d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3500017a0 tx=0x7ff350007040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:27.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.719+0000 7ff359600700 1 -- 172.21.15.12:0/1168280284 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff35c0ad380 con 0x7ff35c0af8b0 2024-03-29T10:15:27.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.965+0000 7ff363e00700 1 -- 172.21.15.12:0/1168280284 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff35c15bdd0 con 0x7ff33c06adb0 2024-03-29T10:15:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.968+0000 7ff359600700 1 -- 172.21.15.12:0/1168280284 <== mgr.8425 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff350005e50 con 0x7ff33c06adb0 2024-03-29T10:15:27.965 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T10:15:27.965 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T10:15:27.965 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T10:15:27.965 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T10:15:27.965 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T10:15:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 -- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff33c06adb0 msgr2=0x7ff33c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff33c06adb0 0x7ff33c06d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3500017a0 tx=0x7ff350007040 comp rx=0 tx=0).stop 2024-03-29T10:15:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 -- 172.21.15.12:0/1168280284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35c0af8b0 msgr2=0x7ff35c0afd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35c0af8b0 0x7ff35c0afd30 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7ff354016000 tx=0x7ff354009750 comp rx=0 tx=0).stop 2024-03-29T10:15:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 -- 172.21.15.12:0/1168280284 shutdown_connections 2024-03-29T10:15:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff33c06adb0 0x7ff33c06d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff35c0af8b0 0x7ff35c0afd30 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff344004d90 0x7ff35c14e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:27.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 --2- 172.21.15.12:0/1168280284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff35c0ac050 0x7ff35c157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:27.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 -- 172.21.15.12:0/1168280284 >> 172.21.15.12:0/1168280284 conn(0x7ff35c0b0d70 msgr2=0x7ff35c15cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:27.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 -- 172.21.15.12:0/1168280284 shutdown_connections 2024-03-29T10:15:27.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:27.970+0000 7ff363e00700 1 -- 172.21.15.12:0/1168280284 wait complete. 2024-03-29T10:15:27.985 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T10:15:28.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.188+0000 7f330a400700 1 Processor -- start 2024-03-29T10:15:28.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.189+0000 7f330a400700 1 -- start start 2024-03-29T10:15:28.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.190+0000 7f330a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 0x7f33040d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:28.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.190+0000 7f330a400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f33040d16a0 0x7f33040d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:28.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.190+0000 7f330a400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3304161420 con 0x7f33040d2370 2024-03-29T10:15:28.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.190+0000 7f330a400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33040d2820 con 0x7f33040d16a0 2024-03-29T10:15:28.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.190+0000 7f330a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3304161700 con 0x7f330415d9d0 2024-03-29T10:15:28.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.190+0000 7f3303e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f33040d2370 0x7f330415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43650/0 (socket says 172.21.15.12:43650) 2024-03-29T10:15:28.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.190+0000 7f3303e00700 1 -- 172.21.15.12:0/1948280279 learned_addr learned my addr 172.21.15.12:0/1948280279 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:28.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.190+0000 7f3302a00700 1 --2- 172.21.15.12:0/1948280279 >> v2:172.21.15.159:3300/0 conn(0x7f33040d16a0 0x7f33040d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:28.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.190+0000 7f3303400700 1 --2- 172.21.15.12:0/1948280279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 0x7f33040d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:28.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.191+0000 7f3302000700 1 -- 172.21.15.12:0/1948280279 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091559669 0 0) 0x7f3304161420 con 0x7f33040d2370 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.191+0000 7f3302000700 1 -- 172.21.15.12:0/1948280279 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e80035b0 con 0x7f33040d2370 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.191+0000 7f3303400700 1 -- 172.21.15.12:0/1948280279 >> v2:172.21.15.159:3300/0 conn(0x7f33040d16a0 msgr2=0x7f33040d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.191+0000 7f3303400700 1 --2- 172.21.15.12:0/1948280279 >> v2:172.21.15.159:3300/0 conn(0x7f33040d16a0 0x7f33040d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.191+0000 7f3303400700 1 -- 172.21.15.12:0/1948280279 >> v1:172.21.15.12:6789/0 conn(0x7f33040d2370 legacy=0x7f330415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.191+0000 7f3303400700 1 -- 172.21.15.12:0/1948280279 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33041619e0 con 0x7f330415d9d0 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.191+0000 7f3302a00700 1 --2- 172.21.15.12:0/1948280279 >> v2:172.21.15.159:3300/0 conn(0x7f33040d16a0 0x7f33040d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.191+0000 7f3303400700 1 --2- 172.21.15.12:0/1948280279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 0x7f33040d3c30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009840 tx=0x7f32f4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52648c65c4ea2299 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.191+0000 7f3302000700 1 -- 172.21.15.12:0/1948280279 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4011830 con 0x7f330415d9d0 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.192+0000 7f3302000700 1 -- 172.21.15.12:0/1948280279 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f4011990 con 0x7f330415d9d0 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.192+0000 7f3302000700 1 -- 172.21.15.12:0/1948280279 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4011d20 con 0x7f330415d9d0 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.192+0000 7f330a400700 1 -- 172.21.15.12:0/1948280279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 msgr2=0x7f33040d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.192+0000 7f330a400700 1 --2- 172.21.15.12:0/1948280279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 0x7f33040d3c30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009840 tx=0x7f32f4009b50 comp rx=0 tx=0).stop 2024-03-29T10:15:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.192+0000 7f330a400700 1 -- 172.21.15.12:0/1948280279 shutdown_connections 2024-03-29T10:15:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.192+0000 7f330a400700 1 --2- 172.21.15.12:0/1948280279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 0x7f33040d3c30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.192+0000 7f330a400700 1 --2- 172.21.15.12:0/1948280279 >> v2:172.21.15.159:3300/0 conn(0x7f33040d16a0 0x7f33040d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.192+0000 7f330a400700 1 -- 172.21.15.12:0/1948280279 >> 172.21.15.12:0/1948280279 conn(0x7f330408aff0 msgr2=0x7f33040d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.192+0000 7f330a400700 1 -- 172.21.15.12:0/1948280279 shutdown_connections 2024-03-29T10:15:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.192+0000 7f330a400700 1 -- 172.21.15.12:0/1948280279 wait complete. 2024-03-29T10:15:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.194+0000 7f330a400700 1 Processor -- start 2024-03-29T10:15:28.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.194+0000 7f330a400700 1 -- start start 2024-03-29T10:15:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.195+0000 7f330a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33040d16a0 0x7f33041f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.195+0000 7f330a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33040d2370 0x7f330415e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.195+0000 7f330a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 0x7f33041588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.195+0000 7f330a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330415f5a0 con 0x7f33040d16a0 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.195+0000 7f330a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33040b7320 con 0x7f33040d2370 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.195+0000 7f330a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330415ed10 con 0x7f330415d9d0 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.195+0000 7f3303400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33040d16a0 0x7f33041f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.195+0000 7f3303400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33040d16a0 0x7f33041f4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50014/0 (socket says 172.21.15.12:50014) 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.195+0000 7f3303400700 1 -- 172.21.15.12:0/1349405041 learned_addr learned my addr 172.21.15.12:0/1349405041 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f3302a00700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33040d2370 0x7f330415e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f3303e00700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 0x7f33041588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f3303400700 1 -- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 msgr2=0x7f33041588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f3303400700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 0x7f33041588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f3303400700 1 -- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33040d2370 msgr2=0x7f330415e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:28.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f3303400700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33040d2370 0x7f330415e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:28.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f3303400700 1 -- 172.21.15.12:0/1349405041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f0009430 con 0x7f33040d16a0 2024-03-29T10:15:28.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f3302a00700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33040d2370 0x7f330415e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T10:15:28.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f3303400700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33040d16a0 0x7f33041f4220 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009fd0 tx=0x7f32f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:28.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f32fb400700 1 -- 172.21.15.12:0/1349405041 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f40036a0 con 0x7f33040d16a0 2024-03-29T10:15:28.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f32fb400700 1 -- 172.21.15.12:0/1349405041 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f401e070 con 0x7f33040d16a0 2024-03-29T10:15:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f330a400700 1 -- 172.21.15.12:0/1349405041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f40094a0 con 0x7f33040d16a0 2024-03-29T10:15:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f32fb400700 1 -- 172.21.15.12:0/1349405041 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4003a20 con 0x7f33040d16a0 2024-03-29T10:15:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.196+0000 7f330a400700 1 -- 172.21.15.12:0/1349405041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330415f2c0 con 0x7f33040d16a0 2024-03-29T10:15:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.197+0000 7f330a400700 1 -- 172.21.15.12:0/1349405041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33040d1ca0 con 0x7f33040d16a0 2024-03-29T10:15:28.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.198+0000 7f32fb400700 1 -- 172.21.15.12:0/1349405041 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f32f4003b80 con 0x7f33040d16a0 2024-03-29T10:15:28.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.199+0000 7f32fb400700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32e406adb0 0x7f32e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:28.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.199+0000 7f32fb400700 1 -- 172.21.15.12:0/1349405041 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32f408a460 con 0x7f33040d16a0 2024-03-29T10:15:28.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.199+0000 7f3302a00700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32e406adb0 0x7f32e406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:28.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.200+0000 7f3302a00700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32e406adb0 0x7f32e406d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f33040b0b50 tx=0x7f32f0008040 comp rx=0 tx=0).ready entity=mgr.8425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:28.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.204+0000 7f32fb400700 1 -- 172.21.15.12:0/1349405041 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32f4055520 con 0x7f33040d16a0 2024-03-29T10:15:28.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:28.480+0000 7f330a400700 1 -- 172.21.15.12:0/1349405041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f3304067720 con 0x7f33040d16a0 2024-03-29T10:15:28.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:15:28.940 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:15:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.258+0000 7f32fb400700 1 -- 172.21.15.12:0/1349405041 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 79) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f32f40558f0 con 0x7f33040d16a0 2024-03-29T10:15:29.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.260+0000 7f32fb400700 1 -- 172.21.15.12:0/1349405041 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v79) v1 ==== 92+0+0 (secure 0 0 0) 0x7f32f405cad0 con 0x7f33040d16a0 2024-03-29T10:15:29.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.262+0000 7f330a400700 1 -- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32e406adb0 msgr2=0x7f32e406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:29.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.262+0000 7f330a400700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32e406adb0 0x7f32e406d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f33040b0b50 tx=0x7f32f0008040 comp rx=0 tx=0).stop 2024-03-29T10:15:29.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.262+0000 7f330a400700 1 -- 172.21.15.12:0/1349405041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33040d16a0 msgr2=0x7f33041f4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:29.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.262+0000 7f330a400700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33040d16a0 0x7f33041f4220 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009fd0 tx=0x7f32f4004750 comp rx=0 tx=0).stop 2024-03-29T10:15:29.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.262+0000 7f330a400700 1 -- 172.21.15.12:0/1349405041 shutdown_connections 2024-03-29T10:15:29.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.262+0000 7f330a400700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f32e406adb0 0x7f32e406d270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.262+0000 7f330a400700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330415d9d0 0x7f33041588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.262+0000 7f330a400700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33040d16a0 0x7f33041f4220 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.262+0000 7f330a400700 1 --2- 172.21.15.12:0/1349405041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33040d2370 0x7f330415e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.262+0000 7f330a400700 1 -- 172.21.15.12:0/1349405041 >> 172.21.15.12:0/1349405041 conn(0x7f330408aff0 msgr2=0x7f33040b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.263+0000 7f330a400700 1 -- 172.21.15.12:0/1349405041 shutdown_connections 2024-03-29T10:15:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.263+0000 7f330a400700 1 -- 172.21.15.12:0/1349405041 wait complete. 2024-03-29T10:15:29.280 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T10:15:29.480 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:29.484+0000 7fb0b56df240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:15:29.488 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:29.491+0000 7feb0786b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:15:29.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.491+0000 7f9a9a400700 1 Processor -- start 2024-03-29T10:15:29.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.492+0000 7f9a9a400700 1 -- start start 2024-03-29T10:15:29.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.493+0000 7f9a9a400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9a940cee10 0x7f9a94161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:29.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.493+0000 7f9a9a400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a940ce270 con 0x7f9a940d21f0 2024-03-29T10:15:29.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.493+0000 7f9a9a400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9415b600 con 0x7f9a941611b0 2024-03-29T10:15:29.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.493+0000 7f9a9a400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a940d2760 con 0x7f9a940cee10 2024-03-29T10:15:29.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.493+0000 7f9a92a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9a940d21f0 0x7f9a940ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35086/0 (socket says 172.21.15.12:35086) 2024-03-29T10:15:29.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.493+0000 7f9a92a00700 1 -- 172.21.15.12:0/3739944873 learned_addr learned my addr 172.21.15.12:0/3739944873 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T10:15:29.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.493+0000 7f9a93e00700 1 --2- 172.21.15.12:0/3739944873 >> v2:172.21.15.159:3300/0 conn(0x7f9a940cee10 0x7f9a94161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:29.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.493+0000 7f9a92000700 1 -- 172.21.15.12:0/3739944873 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503267879 0 0) 0x7f9a940ce270 con 0x7f9a940d21f0 2024-03-29T10:15:29.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.493+0000 7f9a92000700 1 -- 172.21.15.12:0/3739944873 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a780035b0 con 0x7f9a940d21f0 2024-03-29T10:15:29.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.494+0000 7f9a92000700 1 -- 172.21.15.12:0/3739944873 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418830287 0 0) 0x7f9a9415b600 con 0x7f9a941611b0 2024-03-29T10:15:29.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.494+0000 7f9a92000700 1 -- 172.21.15.12:0/3739944873 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a940ce270 con 0x7f9a941611b0 2024-03-29T10:15:29.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.494+0000 7f9a93e00700 1 -- 172.21.15.12:0/3739944873 >> v1:172.21.15.159:6789/0 conn(0x7f9a941611b0 legacy=0x7f9a9415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:29.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.494+0000 7f9a93e00700 1 -- 172.21.15.12:0/3739944873 >> v1:172.21.15.12:6789/0 conn(0x7f9a940d21f0 legacy=0x7f9a940ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:29.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.494+0000 7f9a93e00700 1 -- 172.21.15.12:0/3739944873 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a940d2a40 con 0x7f9a940cee10 2024-03-29T10:15:29.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.494+0000 7f9a93e00700 1 --2- 172.21.15.12:0/3739944873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a940cee10 0x7f9a94161590 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009d90 tx=0x7f9a8c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc792029b67cc02 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:29.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.494+0000 7f9a92000700 1 -- 172.21.15.12:0/3739944873 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c00e070 con 0x7f9a940cee10 2024-03-29T10:15:29.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.494+0000 7f9a92000700 1 -- 172.21.15.12:0/3739944873 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8c00bb20 con 0x7f9a940cee10 2024-03-29T10:15:29.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.494+0000 7f9a92000700 1 -- 172.21.15.12:0/3739944873 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c012430 con 0x7f9a940cee10 2024-03-29T10:15:29.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.495+0000 7f9a9a400700 1 -- 172.21.15.12:0/3739944873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a940cee10 msgr2=0x7f9a94161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:29.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.495+0000 7f9a9a400700 1 --2- 172.21.15.12:0/3739944873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a940cee10 0x7f9a94161590 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009d90 tx=0x7f9a8c00b330 comp rx=0 tx=0).stop 2024-03-29T10:15:29.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.495+0000 7f9a9a400700 1 -- 172.21.15.12:0/3739944873 shutdown_connections 2024-03-29T10:15:29.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.495+0000 7f9a9a400700 1 --2- 172.21.15.12:0/3739944873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a940cee10 0x7f9a94161590 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.495+0000 7f9a9a400700 1 -- 172.21.15.12:0/3739944873 >> 172.21.15.12:0/3739944873 conn(0x7f9a9408aff0 msgr2=0x7f9a940d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:29.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.495+0000 7f9a9a400700 1 -- 172.21.15.12:0/3739944873 shutdown_connections 2024-03-29T10:15:29.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.495+0000 7f9a9a400700 1 -- 172.21.15.12:0/3739944873 wait complete. 2024-03-29T10:15:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.497+0000 7f9a9a400700 1 Processor -- start 2024-03-29T10:15:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.497+0000 7f9a9a400700 1 -- start start 2024-03-29T10:15:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a9a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a940cee10 0x7f9a940caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a9a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a940d21f0 0x7f9a9414a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a9a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a941611b0 0x7f9a940b7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a9a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940b71d0 con 0x7f9a941611b0 2024-03-29T10:15:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a9a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940cb3e0 con 0x7f9a940cee10 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a9a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9414ae40 con 0x7f9a940d21f0 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a93e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a941611b0 0x7f9a940b7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a93e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a941611b0 0x7f9a940b7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43042/0 (socket says 172.21.15.12:43042) 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a93e00700 1 -- 172.21.15.12:0/3147331353 learned_addr learned my addr 172.21.15.12:0/3147331353 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a93400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a940cee10 0x7f9a940caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.498+0000 7f9a92a00700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a940d21f0 0x7f9a9414a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.499+0000 7f9a93e00700 1 -- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a940d21f0 msgr2=0x7f9a9414a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.499+0000 7f9a93e00700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a940d21f0 0x7f9a9414a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.499+0000 7f9a93e00700 1 -- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a940cee10 msgr2=0x7f9a940caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.499+0000 7f9a93e00700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a940cee10 0x7f9a940caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.499+0000 7f9a93e00700 1 -- 172.21.15.12:0/3147331353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8c009a70 con 0x7f9a941611b0 2024-03-29T10:15:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.499+0000 7f9a93400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a940cee10 0x7f9a940caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:15:29.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.499+0000 7f9a93e00700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a941611b0 0x7f9a940b7970 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f9a9408ce10 tx=0x7f9a8c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:29.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.499+0000 7f9a8b400700 1 -- 172.21.15.12:0/3147331353 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c012430 con 0x7f9a941611b0 2024-03-29T10:15:29.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.499+0000 7f9a8b400700 1 -- 172.21.15.12:0/3147331353 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8c020070 con 0x7f9a941611b0 2024-03-29T10:15:29.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.499+0000 7f9a8b400700 1 -- 172.21.15.12:0/3147331353 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c01ba50 con 0x7f9a941611b0 2024-03-29T10:15:29.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.500+0000 7f9a9a400700 1 -- 172.21.15.12:0/3147331353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a940c7190 con 0x7f9a941611b0 2024-03-29T10:15:29.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.500+0000 7f9a9a400700 1 -- 172.21.15.12:0/3147331353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a940c7750 con 0x7f9a941611b0 2024-03-29T10:15:29.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.501+0000 7f9a9a400700 1 -- 172.21.15.12:0/3147331353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a94067720 con 0x7f9a941611b0 2024-03-29T10:15:29.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.502+0000 7f9a8b400700 1 -- 172.21.15.12:0/3147331353 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 79) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f9a8c01bbb0 con 0x7f9a941611b0 2024-03-29T10:15:29.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.503+0000 7f9a8b400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a7406ae00 0x7f9a7406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:29.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.503+0000 7f9a8b400700 1 -- 172.21.15.12:0/3147331353 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a8c022070 con 0x7f9a941611b0 2024-03-29T10:15:29.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.503+0000 7f9a93400700 1 -- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a7406ae00 msgr2=0x7f9a7406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:15:29.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.503+0000 7f9a93400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a7406ae00 0x7f9a7406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:15:29.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.507+0000 7f9a8b400700 1 -- 172.21.15.12:0/3147331353 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9a8c0564a0 con 0x7f9a941611b0 2024-03-29T10:15:29.666 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:29.669+0000 7fb0b56df240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:15:29.697 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:29.700+0000 7feb0786b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:15:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.704+0000 7f9a93400700 1 -- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a7406ae00 msgr2=0x7f9a7406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:15:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.704+0000 7f9a93400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a7406ae00 0x7f9a7406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:15:29.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.761+0000 7f9a9a400700 1 -- 172.21.15.12:0/3147331353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f9a940b5d00 con 0x7f9a941611b0 2024-03-29T10:15:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.766+0000 7f9a8b400700 1 -- 172.21.15.12:0/3147331353 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9a8c01be60 con 0x7f9a941611b0 2024-03-29T10:15:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 -- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a7406ae00 msgr2=0x7f9a7406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:15:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a7406ae00 0x7f9a7406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 -- 172.21.15.12:0/3147331353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a941611b0 msgr2=0x7f9a940b7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a941611b0 0x7f9a940b7970 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f9a9408ce10 tx=0x7f9a8c004ad0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 -- 172.21.15.12:0/3147331353 shutdown_connections 2024-03-29T10:15:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9a7406ae00 0x7f9a7406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a941611b0 0x7f9a940b7970 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a940cee10 0x7f9a940caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 --2- 172.21.15.12:0/3147331353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a940d21f0 0x7f9a9414a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 -- 172.21.15.12:0/3147331353 >> 172.21.15.12:0/3147331353 conn(0x7f9a9408aff0 msgr2=0x7f9a9406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 -- 172.21.15.12:0/3147331353 shutdown_connections 2024-03-29T10:15:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.768+0000 7f9a9a400700 1 -- 172.21.15.12:0/3147331353 wait complete. 2024-03-29T10:15:29.784 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T10:15:29.983 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:29.987+0000 7fb0b56df240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:15:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.997+0000 7f2cfbc00700 1 Processor -- start 2024-03-29T10:15:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.997+0000 7f2cfbc00700 1 -- start start 2024-03-29T10:15:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.998+0000 7f2cfbc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2cf4157f90 0x7f2cf40b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.998+0000 7f2cfbc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cf40b3b30 0x7f2cf4161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.998+0000 7f2cfbc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf40b3070 con 0x7f2cf41613e0 2024-03-29T10:15:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.998+0000 7f2cfbc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf4158370 con 0x7f2cf40b3b30 2024-03-29T10:15:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.998+0000 7f2cfbc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf40b2710 con 0x7f2cf4157f90 2024-03-29T10:15:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.999+0000 7f2cf9c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cf40b3b30 0x7f2cf4161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.999+0000 7f2cf9200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2cf4157f90 0x7f2cf40b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.999+0000 7f2cf9c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cf40b3b30 0x7f2cf4161890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43056/0 (socket says 172.21.15.12:43056) 2024-03-29T10:15:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.999+0000 7f2cf9200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2cf4157f90 0x7f2cf40b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:59578/0 (socket says 172.21.15.12:59578) 2024-03-29T10:15:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.999+0000 7f2cf9c00700 1 -- 172.21.15.12:0/2552130074 learned_addr learned my addr 172.21.15.12:0/2552130074 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.999+0000 7f2cf9c00700 1 -- 172.21.15.12:0/2552130074 >> v2:172.21.15.159:3300/0 conn(0x7f2cf4157f90 msgr2=0x7f2cf40b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.999+0000 7f2cf9c00700 1 --2- 172.21.15.12:0/2552130074 >> v2:172.21.15.159:3300/0 conn(0x7f2cf4157f90 0x7f2cf40b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.999+0000 7f2cf9c00700 1 -- 172.21.15.12:0/2552130074 >> v1:172.21.15.159:6789/0 conn(0x7f2cf41613e0 legacy=0x7f2cf40b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.999+0000 7f2cf9c00700 1 -- 172.21.15.12:0/2552130074 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf40b2070 con 0x7f2cf40b3b30 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:29.999+0000 7f2cf9200700 1 --2- 172.21.15.12:0/2552130074 >> v2:172.21.15.159:3300/0 conn(0x7f2cf4157f90 0x7f2cf40b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.000+0000 7f2cf9c00700 1 --2- 172.21.15.12:0/2552130074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf40b3b30 0x7f2cf4161890 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f2cf406fd80 tx=0x7f2cec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15006324c4d8eee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.000+0000 7f2cf3400700 1 -- 172.21.15.12:0/2552130074 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00e070 con 0x7f2cf40b3b30 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.000+0000 7f2cf3400700 1 -- 172.21.15.12:0/2552130074 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec00b910 con 0x7f2cf40b3b30 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.000+0000 7f2cf3400700 1 -- 172.21.15.12:0/2552130074 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00bcc0 con 0x7f2cf40b3b30 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.000+0000 7f2cfbc00700 1 -- 172.21.15.12:0/2552130074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf40b3b30 msgr2=0x7f2cf4161890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.000+0000 7f2cfbc00700 1 --2- 172.21.15.12:0/2552130074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf40b3b30 0x7f2cf4161890 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f2cf406fd80 tx=0x7f2cec00b330 comp rx=0 tx=0).stop 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.001+0000 7f2cfbc00700 1 -- 172.21.15.12:0/2552130074 shutdown_connections 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.001+0000 7f2cfbc00700 1 --2- 172.21.15.12:0/2552130074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf40b3b30 0x7f2cf4161890 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.001+0000 7f2cfbc00700 1 --2- 172.21.15.12:0/2552130074 >> v2:172.21.15.159:3300/0 conn(0x7f2cf4157f90 0x7f2cf40b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.001+0000 7f2cfbc00700 1 -- 172.21.15.12:0/2552130074 >> 172.21.15.12:0/2552130074 conn(0x7f2cf40b0d70 msgr2=0x7f2cf40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.001+0000 7f2cfbc00700 1 -- 172.21.15.12:0/2552130074 shutdown_connections 2024-03-29T10:15:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.001+0000 7f2cfbc00700 1 -- 172.21.15.12:0/2552130074 wait complete. 2024-03-29T10:15:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.002+0000 7f2cfbc00700 1 Processor -- start 2024-03-29T10:15:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.003+0000 7f2cfbc00700 1 -- start start 2024-03-29T10:15:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.003+0000 7f2cfbc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cf40b3b30 0x7f2cf4153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cfbc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf4157f90 0x7f2cf4159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cfbc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cf41613e0 0x7f2cf41584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cfbc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf4158e90 con 0x7f2cf4157f90 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cfbc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf4159170 con 0x7f2cf40b3b30 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cfbc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf4159450 con 0x7f2cf41613e0 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cf3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf4157f90 0x7f2cf4159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cf3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf4157f90 0x7f2cf4159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43060/0 (socket says 172.21.15.12:43060) 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cf3e00700 1 -- 172.21.15.12:0/1200801438 learned_addr learned my addr 172.21.15.12:0/1200801438 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cf9200700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cf40b3b30 0x7f2cf4153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cf9c00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cf41613e0 0x7f2cf41584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cf3e00700 1 -- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cf41613e0 msgr2=0x7f2cf41584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cf3e00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cf41613e0 0x7f2cf41584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cf3e00700 1 -- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cf40b3b30 msgr2=0x7f2cf4153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cf3e00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cf40b3b30 0x7f2cf4153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.004+0000 7f2cf3e00700 1 -- 172.21.15.12:0/1200801438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce80094a0 con 0x7f2cf4157f90 2024-03-29T10:15:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.005+0000 7f2cf3e00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf4157f90 0x7f2cf4159f30 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000c4e0 tx=0x7f2ce000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.005+0000 7f2cf0c00700 1 -- 172.21.15.12:0/1200801438 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce00052d0 con 0x7f2cf4157f90 2024-03-29T10:15:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.005+0000 7f2cf0c00700 1 -- 172.21.15.12:0/1200801438 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce0005430 con 0x7f2cf4157f90 2024-03-29T10:15:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.005+0000 7f2cf0c00700 1 -- 172.21.15.12:0/1200801438 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce00038c0 con 0x7f2cf4157f90 2024-03-29T10:15:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.005+0000 7f2cfbc00700 1 -- 172.21.15.12:0/1200801438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cec009b30 con 0x7f2cf4157f90 2024-03-29T10:15:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.005+0000 7f2cfbc00700 1 -- 172.21.15.12:0/1200801438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf4159a10 con 0x7f2cf4157f90 2024-03-29T10:15:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.006+0000 7f2cfbc00700 1 -- 172.21.15.12:0/1200801438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf4159c00 con 0x7f2cf4157f90 2024-03-29T10:15:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.007+0000 7f2cf0c00700 1 -- 172.21.15.12:0/1200801438 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 79) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f2ce0003a20 con 0x7f2cf4157f90 2024-03-29T10:15:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.008+0000 7f2cf0c00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cd406ae00 0x7f2cd406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.008+0000 7f2cf0c00700 1 -- 172.21.15.12:0/1200801438 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ce0088870 con 0x7f2cf4157f90 2024-03-29T10:15:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.008+0000 7f2cf9200700 1 -- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cd406ae00 msgr2=0x7f2cd406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:15:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.008+0000 7f2cf9200700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cd406ae00 0x7f2cd406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:15:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.012+0000 7f2cf0c00700 1 -- 172.21.15.12:0/1200801438 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ce00538a0 con 0x7f2cf4157f90 2024-03-29T10:15:30.048 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:30.051+0000 7feb0786b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:15:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.209+0000 7f2cf9200700 1 -- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cd406ae00 msgr2=0x7f2cd406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:15:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.209+0000 7f2cf9200700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cd406ae00 0x7f2cd406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:15:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.228+0000 7f2cfbc00700 1 -- 172.21.15.12:0/1200801438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f2cf40b3070 con 0x7f2cf4157f90 2024-03-29T10:15:30.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.232+0000 7f2cf0c00700 1 -- 172.21.15.12:0/1200801438 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2cf40b3070 con 0x7f2cf4157f90 2024-03-29T10:15:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 -- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cd406ae00 msgr2=0x7f2cd406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:15:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cd406ae00 0x7f2cd406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 -- 172.21.15.12:0/1200801438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf4157f90 msgr2=0x7f2cf4159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf4157f90 0x7f2cf4159f30 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000c4e0 tx=0x7f2ce000c7f0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 -- 172.21.15.12:0/1200801438 shutdown_connections 2024-03-29T10:15:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2cd406ae00 0x7f2cd406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cf41613e0 0x7f2cf41584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cf40b3b30 0x7f2cf4153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 --2- 172.21.15.12:0/1200801438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf4157f90 0x7f2cf4159f30 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 -- 172.21.15.12:0/1200801438 >> 172.21.15.12:0/1200801438 conn(0x7f2cf40b0d70 msgr2=0x7f2cf40cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 -- 172.21.15.12:0/1200801438 shutdown_connections 2024-03-29T10:15:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.235+0000 7f2cfbc00700 1 -- 172.21.15.12:0/1200801438 wait complete. 2024-03-29T10:15:30.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T10:15:30.252 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-29T10:15:30.301 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2024-03-29T10:15:30.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.509+0000 7fc30ba00700 1 Processor -- start 2024-03-29T10:15:30.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.509+0000 7fc30ba00700 1 -- start start 2024-03-29T10:15:30.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.510+0000 7fc30ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 0x7fc3040ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:30.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.510+0000 7fc30ba00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc3040d21a0 0x7fc30415fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:30.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.510+0000 7fc30ba00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30415f820 con 0x7fc3040d1630 2024-03-29T10:15:30.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.510+0000 7fc30ba00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3040cfcc0 con 0x7fc3040d21a0 2024-03-29T10:15:30.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.510+0000 7fc30ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3040cd9f0 con 0x7fc30415dad0 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.510+0000 7fc309000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 0x7fc3040ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.511+0000 7fc309a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc3040d21a0 0x7fc30415fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.511+0000 7fc309000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 0x7fc3040ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:38826/0 (socket says 172.21.15.12:38826) 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.511+0000 7fc309000700 1 -- 172.21.15.12:0/1580711923 learned_addr learned my addr 172.21.15.12:0/1580711923 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.511+0000 7fc309a00700 1 -- 172.21.15.12:0/1580711923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 msgr2=0x7fc3040ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.511+0000 7fc309a00700 1 --2- 172.21.15.12:0/1580711923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 0x7fc3040ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.511+0000 7fc309a00700 1 -- 172.21.15.12:0/1580711923 >> v1:172.21.15.159:6789/0 conn(0x7fc3040d1630 legacy=0x7fc3040d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.511+0000 7fc309a00700 1 -- 172.21.15.12:0/1580711923 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3040cf420 con 0x7fc3040d21a0 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.511+0000 7fc309000700 1 --2- 172.21.15.12:0/1580711923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 0x7fc3040ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.511+0000 7fc303400700 1 -- 172.21.15.12:0/1580711923 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939583383 0 0) 0x7fc30415f820 con 0x7fc3040d1630 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.511+0000 7fc309a00700 1 --2- 172.21.15.12:0/1580711923 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3040d21a0 0x7fc30415fcb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009d90 tx=0x7fc2fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c066e1e24357472c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:30.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc303400700 1 -- 172.21.15.12:0/1580711923 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc00e070 con 0x7fc3040d21a0 2024-03-29T10:15:30.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc303400700 1 -- 172.21.15.12:0/1580711923 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2fc00bb20 con 0x7fc3040d21a0 2024-03-29T10:15:30.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc303400700 1 -- 172.21.15.12:0/1580711923 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc012430 con 0x7fc3040d21a0 2024-03-29T10:15:30.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc30ba00700 1 -- 172.21.15.12:0/1580711923 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3040d21a0 msgr2=0x7fc30415fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:30.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc30ba00700 1 --2- 172.21.15.12:0/1580711923 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3040d21a0 0x7fc30415fcb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009d90 tx=0x7fc2fc00b330 comp rx=0 tx=0).stop 2024-03-29T10:15:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc30ba00700 1 -- 172.21.15.12:0/1580711923 shutdown_connections 2024-03-29T10:15:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc30ba00700 1 --2- 172.21.15.12:0/1580711923 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3040d21a0 0x7fc30415fcb0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc30ba00700 1 --2- 172.21.15.12:0/1580711923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 0x7fc3040ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc30ba00700 1 -- 172.21.15.12:0/1580711923 >> 172.21.15.12:0/1580711923 conn(0x7fc30408aff0 msgr2=0x7fc3040d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc30ba00700 1 -- 172.21.15.12:0/1580711923 shutdown_connections 2024-03-29T10:15:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.512+0000 7fc30ba00700 1 -- 172.21.15.12:0/1580711923 wait complete. 2024-03-29T10:15:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.514+0000 7fc30ba00700 1 Processor -- start 2024-03-29T10:15:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.514+0000 7fc30ba00700 1 -- start start 2024-03-29T10:15:30.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.515+0000 7fc30ba00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3040d1630 0x7fc3040cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:30.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.515+0000 7fc30ba00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3040d21a0 0x7fc3041f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:30.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.515+0000 7fc30ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 0x7fc304154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:30.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.515+0000 7fc30ba00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc304155850 con 0x7fc3040d1630 2024-03-29T10:15:30.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.515+0000 7fc30ba00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3040c7280 con 0x7fc3040d21a0 2024-03-29T10:15:30.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.515+0000 7fc30ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc304154fb0 con 0x7fc30415dad0 2024-03-29T10:15:30.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc309000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3040d1630 0x7fc3040cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:30.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc309000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3040d1630 0x7fc3040cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43076/0 (socket says 172.21.15.12:43076) 2024-03-29T10:15:30.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc309000700 1 -- 172.21.15.12:0/2930745907 learned_addr learned my addr 172.21.15.12:0/2930745907 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:30.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc303e00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3040d21a0 0x7fc3041f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:30.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc309a00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 0x7fc304154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:30.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc309000700 1 -- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 msgr2=0x7fc304154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:30.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc309000700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 0x7fc304154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc309000700 1 -- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3040d21a0 msgr2=0x7fc3041f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc309000700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3040d21a0 0x7fc3041f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc309000700 1 -- 172.21.15.12:0/2930745907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f8009550 con 0x7fc3040d1630 2024-03-29T10:15:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc303e00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3040d21a0 0x7fc3041f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:15:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.516+0000 7fc309000700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3040d1630 0x7fc3040cce30 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800e9b0 tx=0x7fc2f800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.517+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f800ee50 con 0x7fc3040d1630 2024-03-29T10:15:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.517+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f800c850 con 0x7fc3040d1630 2024-03-29T10:15:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.517+0000 7fc30ba00700 1 -- 172.21.15.12:0/2930745907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2fc009a70 con 0x7fc3040d1630 2024-03-29T10:15:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.517+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f800cbc0 con 0x7fc3040d1630 2024-03-29T10:15:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.517+0000 7fc30ba00700 1 -- 172.21.15.12:0/2930745907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc304155570 con 0x7fc3040d1630 2024-03-29T10:15:30.516 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:30.517+0000 7fb0b56df240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:15:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.518+0000 7fc30ba00700 1 -- 172.21.15.12:0/2930745907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3040c7050 con 0x7fc3040d1630 2024-03-29T10:15:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.519+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 79) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fc2f800cd20 con 0x7fc3040d1630 2024-03-29T10:15:30.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.520+0000 7fc300c00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:30.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.520+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2f8089530 con 0x7fc3040d1630 2024-03-29T10:15:30.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.520+0000 7fc303e00700 1 -- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 msgr2=0x7fc2e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:15:30.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.520+0000 7fc303e00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T10:15:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.529+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2f8054570 con 0x7fc3040d1630 2024-03-29T10:15:30.628 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:30.631+0000 7feb0786b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:15:30.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.721+0000 7fc303e00700 1 -- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 msgr2=0x7fc2e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:15:30.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.721+0000 7fc303e00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T10:15:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:30.770+0000 7fc30ba00700 1 -- 172.21.15.12:0/2930745907 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3040cf9e0 con 0x7fc2e406ae00 2024-03-29T10:15:30.770 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:30.774+0000 7fb0b56df240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:15:30.894 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:30.897+0000 7feb0786b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:15:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:31.122+0000 7fc303e00700 1 -- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 msgr2=0x7fc2e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:15:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:31.122+0000 7fc303e00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T10:15:31.226 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:31.229+0000 7fb0b56df240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:15:31.371 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:31.374+0000 7feb0786b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:15:31.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:31.923+0000 7fc303e00700 1 -- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 msgr2=0x7fc2e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:15:31.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:31.923+0000 7fc303e00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T10:15:31.927 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:31.931+0000 7fb0b56df240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:15:32.016 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:32.020+0000 7fb0b56df240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:15:32.116 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:32.119+0000 7feb0786b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:15:32.175 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:32.178+0000 7fb0b56df240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:15:32.213 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:32.216+0000 7feb0786b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:15:32.381 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:32.384+0000 7feb0786b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:15:32.389 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:32.392+0000 7fb0b56df240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:15:32.480 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:32.483+0000 7fb0b56df240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:15:32.577 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:32.581+0000 7fb0b56df240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:15:32.600 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:32.603+0000 7feb0786b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:15:32.675 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:32.678+0000 7fb0b56df240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:15:32.696 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:32.699+0000 7feb0786b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:15:32.775 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:32.778+0000 7fb0b56df240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:15:32.800 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:32.802+0000 7feb0786b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:15:32.902 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:32.905+0000 7feb0786b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:15:32.986 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:32.990+0000 7fb0b56df240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:15:33.005 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:33.008+0000 7feb0786b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:15:33.227 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:33.230+0000 7feb0786b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:15:33.244 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:33.248+0000 7fb0b56df240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:15:33.434 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:33.437+0000 7fb0b56df240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:15:33.491 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:33.494+0000 7feb0786b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:15:33.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:33.525+0000 7fc303e00700 1 -- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 msgr2=0x7fc2e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T10:15:33.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:33.525+0000 7fc303e00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T10:15:33.527 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:33.531+0000 7fb0b56df240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:15:33.691 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:33.694+0000 7feb0786b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:15:33.789 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:33.792+0000 7feb0786b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:15:34.258 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:34.262+0000 7fb0b56df240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:15:34.468 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:34.472+0000 7fb0b56df240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:15:34.556 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:34.559+0000 7feb0786b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:15:34.674 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:34.677+0000 7fb0b56df240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:15:34.781 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:34.784+0000 7feb0786b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:15:34.998 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:35.001+0000 7feb0786b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:15:35.150 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:35.154+0000 7fb0b56df240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:15:35.355 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:35.359+0000 7fb0b56df240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:15:35.500 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:35.503+0000 7feb0786b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:15:35.718 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:35.721+0000 7feb0786b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:15:35.797 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:35.801+0000 7fb0b56df240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:15:35.908 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:35.912+0000 7fb0b56df240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:15:35.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:35.922+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 80) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fc2f8054940 con 0x7fc3040d1630 2024-03-29T10:15:35.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:35.923+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 msgr2=0x7fc2e406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:15:35.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:35.923+0000 7fc300c00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:36.095 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:36.099+0000 7fb01de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:15:36.096 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:36.099+0000 7fb01de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:15:36.096 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:36.099+0000 7fb01de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:15:36.096 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:36.099+0000 7fb01de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:15:36.096 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:36.099+0000 7fb01de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:15:36.096 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:36.100+0000 7fb021a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:15:36.097 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:36.100+0000 7fb021a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:15:36.097 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:36.100+0000 7fb021a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:15:36.097 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:36.100+0000 7fb021a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:15:36.097 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:15:36.100+0000 7fb021a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:15:36.182 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:36.186+0000 7feb0786b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:15:36.300 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T10:15:36.303+0000 7feb0786b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:15:36.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:36.928+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 81) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc2f80879b0 con 0x7fc3040d1630 2024-03-29T10:15:36.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:36.928+0000 7fc300c00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:36.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:36.928+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2e401d2d0 con 0x7fc2e406ae00 2024-03-29T10:15:36.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:36.932+0000 7fc303e00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:36.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:36.940+0000 7fc303e00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc304151db0 tx=0x7fc2f0009040 comp rx=0 tx=0).ready entity=mgr.8824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:37.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.013+0000 7fc300c00700 1 -- 172.21.15.12:0/2930745907 <== mgr.8824 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fc2e401d2d0 con 0x7fc2e406ae00 2024-03-29T10:15:37.010 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.016+0000 7fc30ba00700 1 -- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 msgr2=0x7fc2e406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.016+0000 7fc30ba00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc304151db0 tx=0x7fc2f0009040 comp rx=0 tx=0).stop 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.016+0000 7fc30ba00700 1 -- 172.21.15.12:0/2930745907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3040d1630 msgr2=0x7fc3040cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.016+0000 7fc30ba00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3040d1630 0x7fc3040cce30 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800e9b0 tx=0x7fc2f800ecc0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.016+0000 7fc30ba00700 1 -- 172.21.15.12:0/2930745907 shutdown_connections 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.016+0000 7fc30ba00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2e406ae00 0x7fc2e406d2c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.016+0000 7fc30ba00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30415dad0 0x7fc304154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.016+0000 7fc30ba00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3040d1630 0x7fc3040cce30 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.016+0000 7fc30ba00700 1 --2- 172.21.15.12:0/2930745907 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3040d21a0 0x7fc3041f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.016+0000 7fc30ba00700 1 -- 172.21.15.12:0/2930745907 >> 172.21.15.12:0/2930745907 conn(0x7fc30408aff0 msgr2=0x7fc30415b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.017+0000 7fc30ba00700 1 -- 172.21.15.12:0/2930745907 shutdown_connections 2024-03-29T10:15:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.017+0000 7fc30ba00700 1 -- 172.21.15.12:0/2930745907 wait complete. 2024-03-29T10:15:37.035 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2024-03-29T10:15:37.035 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2024-03-29T10:15:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.243+0000 7f0434800700 1 Processor -- start 2024-03-29T10:15:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.243+0000 7f0434800700 1 -- start start 2024-03-29T10:15:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.244+0000 7f0434800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f042c15d7f0 0x7f042c0d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.244+0000 7f0434800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c0d1770 0x7f042c0b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.244+0000 7f0434800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042c0ceb70 con 0x7f042c0b5510 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.244+0000 7f0434800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042c15dbd0 con 0x7f042c15d7f0 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.244+0000 7f0434800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0b59a0 con 0x7f042c0d1770 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0431400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c0d1770 0x7f042c0b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0431400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c0d1770 0x7f042c0b4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:38854/0 (socket says 172.21.15.12:38854) 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0431400700 1 -- 172.21.15.12:0/2744307618 learned_addr learned my addr 172.21.15.12:0/2744307618 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0431e00700 1 --2- 172.21.15.12:0/2744307618 >> v2:172.21.15.159:3300/0 conn(0x7f042c15d7f0 0x7f042c0d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0430a00700 1 -- 172.21.15.12:0/2744307618 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417797981 0 0) 0x7f042c0ceb70 con 0x7f042c0b5510 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0430a00700 1 -- 172.21.15.12:0/2744307618 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04140035b0 con 0x7f042c0b5510 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0431400700 1 -- 172.21.15.12:0/2744307618 >> v2:172.21.15.159:3300/0 conn(0x7f042c15d7f0 msgr2=0x7f042c0d25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0431400700 1 --2- 172.21.15.12:0/2744307618 >> v2:172.21.15.159:3300/0 conn(0x7f042c15d7f0 0x7f042c0d25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0431400700 1 -- 172.21.15.12:0/2744307618 >> v1:172.21.15.159:6789/0 conn(0x7f042c0b5510 legacy=0x7f042c06ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0431400700 1 -- 172.21.15.12:0/2744307618 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c0cf000 con 0x7f042c0d1770 2024-03-29T10:15:37.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.245+0000 7f0431e00700 1 --2- 172.21.15.12:0/2744307618 >> v2:172.21.15.159:3300/0 conn(0x7f042c15d7f0 0x7f042c0d25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T10:15:37.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.246+0000 7f0431400700 1 --2- 172.21.15.12:0/2744307618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c0d1770 0x7f042c0b4fd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f041c0098f0 tx=0x7f041c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be20288cea0eaff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.246+0000 7f0430a00700 1 -- 172.21.15.12:0/2744307618 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c011830 con 0x7f042c0d1770 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.246+0000 7f0430a00700 1 -- 172.21.15.12:0/2744307618 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041c011990 con 0x7f042c0d1770 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.246+0000 7f0430a00700 1 -- 172.21.15.12:0/2744307618 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c011d20 con 0x7f042c0d1770 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.247+0000 7f0434800700 1 -- 172.21.15.12:0/2744307618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c0d1770 msgr2=0x7f042c0b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.247+0000 7f0434800700 1 --2- 172.21.15.12:0/2744307618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c0d1770 0x7f042c0b4fd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f041c0098f0 tx=0x7f041c009c00 comp rx=0 tx=0).stop 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.247+0000 7f0434800700 1 -- 172.21.15.12:0/2744307618 shutdown_connections 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.247+0000 7f0434800700 1 --2- 172.21.15.12:0/2744307618 >> v2:172.21.15.159:3300/0 conn(0x7f042c15d7f0 0x7f042c0d25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.247+0000 7f0434800700 1 --2- 172.21.15.12:0/2744307618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c0d1770 0x7f042c0b4fd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.247+0000 7f0434800700 1 -- 172.21.15.12:0/2744307618 >> 172.21.15.12:0/2744307618 conn(0x7f042c08aff0 msgr2=0x7f042c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.247+0000 7f0434800700 1 -- 172.21.15.12:0/2744307618 shutdown_connections 2024-03-29T10:15:37.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.247+0000 7f0434800700 1 -- 172.21.15.12:0/2744307618 wait complete. 2024-03-29T10:15:37.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.248+0000 7f0434800700 1 Processor -- start 2024-03-29T10:15:37.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.249+0000 7f0434800700 1 -- start start 2024-03-29T10:15:37.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.249+0000 7f0434800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f042c0b5510 0x7f042c1f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.249+0000 7f0434800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f042c0d1770 0x7f042c1460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.249+0000 7f0434800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c15d7f0 0x7f042c1465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.249+0000 7f0434800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0c8d90 con 0x7f042c0d1770 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.249+0000 7f0434800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0c9070 con 0x7f042c0b5510 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.249+0000 7f0434800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0c9350 con 0x7f042c15d7f0 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.250+0000 7f0431400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f042c0d1770 0x7f042c1460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.250+0000 7f0431400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f042c0d1770 0x7f042c1460a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43090/0 (socket says 172.21.15.12:43090) 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.250+0000 7f0431400700 1 -- 172.21.15.12:0/837440298 learned_addr learned my addr 172.21.15.12:0/837440298 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.250+0000 7f0431e00700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f042c0b5510 0x7f042c1f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.250+0000 7f0432800700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c15d7f0 0x7f042c1465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.250+0000 7f0431400700 1 -- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c15d7f0 msgr2=0x7f042c1465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.250+0000 7f0431400700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c15d7f0 0x7f042c1465e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.250+0000 7f0431400700 1 -- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f042c0b5510 msgr2=0x7f042c1f5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.250+0000 7f0431400700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f042c0b5510 0x7f042c1f5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.250+0000 7f0431400700 1 -- 172.21.15.12:0/837440298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04200094a0 con 0x7f042c0d1770 2024-03-29T10:15:37.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.251+0000 7f0431400700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f042c0d1770 0x7f042c1460a0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f041c000c00 tx=0x7f041c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:37.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.251+0000 7f0426000700 1 -- 172.21.15.12:0/837440298 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c004480 con 0x7f042c0d1770 2024-03-29T10:15:37.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.251+0000 7f0426000700 1 -- 172.21.15.12:0/837440298 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041c01f070 con 0x7f042c0d1770 2024-03-29T10:15:37.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.251+0000 7f0434800700 1 -- 172.21.15.12:0/837440298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041c009550 con 0x7f042c0d1770 2024-03-29T10:15:37.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.251+0000 7f0426000700 1 -- 172.21.15.12:0/837440298 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c0038a0 con 0x7f042c0d1770 2024-03-29T10:15:37.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.251+0000 7f0434800700 1 -- 172.21.15.12:0/837440298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042c0c9910 con 0x7f042c0d1770 2024-03-29T10:15:37.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.252+0000 7f0434800700 1 -- 172.21.15.12:0/837440298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042c0caa40 con 0x7f042c0d1770 2024-03-29T10:15:37.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.253+0000 7f0426000700 1 -- 172.21.15.12:0/837440298 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 81) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f041c021030 con 0x7f042c0d1770 2024-03-29T10:15:37.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.254+0000 7f0426000700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f041006ab60 0x7f041006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.254+0000 7f0426000700 1 -- 172.21.15.12:0/837440298 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f041c08a4c0 con 0x7f042c0d1770 2024-03-29T10:15:37.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.255+0000 7f0431e00700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f041006ab60 0x7f041006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.255+0000 7f0431e00700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f041006ab60 0x7f041006d020 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0420009d00 tx=0x7f0420008040 comp rx=0 tx=0).ready entity=mgr.8824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:37.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.259+0000 7f0426000700 1 -- 172.21.15.12:0/837440298 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f041c055880 con 0x7f042c0d1770 2024-03-29T10:15:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.498+0000 7f0434800700 1 -- 172.21.15.12:0/837440298 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7f042c15e6c0 con 0x7f041006ab60 2024-03-29T10:15:37.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.563+0000 7f0426000700 1 -- 172.21.15.12:0/837440298 <== mgr.8824 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7f042c15e6c0 con 0x7f041006ab60 2024-03-29T10:15:37.560 INFO:teuthology.orchestra.run.smithi012.stdout:Retention added to path /snap_test_dir1 2024-03-29T10:15:37.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 -- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f041006ab60 msgr2=0x7f041006d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f041006ab60 0x7f041006d020 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0420009d00 tx=0x7f0420008040 comp rx=0 tx=0).stop 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 -- 172.21.15.12:0/837440298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f042c0d1770 msgr2=0x7f042c1460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f042c0d1770 0x7f042c1460a0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f041c000c00 tx=0x7f041c004750 comp rx=0 tx=0).stop 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 -- 172.21.15.12:0/837440298 shutdown_connections 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f041006ab60 0x7f041006d020 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f042c15d7f0 0x7f042c1465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f042c0b5510 0x7f042c1f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 --2- 172.21.15.12:0/837440298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f042c0d1770 0x7f042c1460a0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 -- 172.21.15.12:0/837440298 >> 172.21.15.12:0/837440298 conn(0x7f042c08aff0 msgr2=0x7f042c0d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 -- 172.21.15.12:0/837440298 shutdown_connections 2024-03-29T10:15:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.565+0000 7f0434800700 1 -- 172.21.15.12:0/837440298 wait complete. 2024-03-29T10:15:37.581 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2024-03-29T10:15:37.581 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-03-29T10:15:37.581 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2024-03-29T10:15:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.786+0000 7f7b51a00700 1 Processor -- start 2024-03-29T10:15:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.786+0000 7f7b51a00700 1 -- start start 2024-03-29T10:15:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.787+0000 7f7b51a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7b4c15d9d0 0x7f7b4c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.787+0000 7f7b51a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4c0d2540 0x7f7b4c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.787+0000 7f7b51a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4c161420 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.787+0000 7f7b51a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c15edc0 con 0x7f7b4c15d9d0 2024-03-29T10:15:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.787+0000 7f7b51a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c161700 con 0x7f7b4c0d2540 2024-03-29T10:15:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.788+0000 7f7b4aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7b4c15d9d0 0x7f7b4c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.788+0000 7f7b4aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7b4c15d9d0 0x7f7b4c0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43102/0 (socket says 172.21.15.12:43102) 2024-03-29T10:15:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.788+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3284473394 learned_addr learned my addr 172.21.15.12:0/3284473394 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.788+0000 7f7b4b400700 1 --2- 172.21.15.12:0/3284473394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4c0d2540 0x7f7b4c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.788+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3284473394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4c0d2540 msgr2=0x7f7b4c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:15:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.788+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/3284473394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4c0d2540 0x7f7b4c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.788+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3284473394 >> v1:172.21.15.159:6789/0 conn(0x7f7b4c0d16a0 legacy=0x7f7b4c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.788+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3284473394 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c1619e0 con 0x7f7b4c15d9d0 2024-03-29T10:15:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.788+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/3284473394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c15d9d0 0x7f7b4c0d3c30 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0098f0 tx=0x7f7b3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad5df45331377c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b49600700 1 -- 172.21.15.12:0/3284473394 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c011830 con 0x7f7b4c15d9d0 2024-03-29T10:15:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b49600700 1 -- 172.21.15.12:0/3284473394 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3c011990 con 0x7f7b4c15d9d0 2024-03-29T10:15:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b49600700 1 -- 172.21.15.12:0/3284473394 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c011d20 con 0x7f7b4c15d9d0 2024-03-29T10:15:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b51a00700 1 -- 172.21.15.12:0/3284473394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c15d9d0 msgr2=0x7f7b4c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b51a00700 1 --2- 172.21.15.12:0/3284473394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c15d9d0 0x7f7b4c0d3c30 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0098f0 tx=0x7f7b3c009c00 comp rx=0 tx=0).stop 2024-03-29T10:15:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b51a00700 1 -- 172.21.15.12:0/3284473394 shutdown_connections 2024-03-29T10:15:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b51a00700 1 --2- 172.21.15.12:0/3284473394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4c0d2540 0x7f7b4c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b51a00700 1 --2- 172.21.15.12:0/3284473394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c15d9d0 0x7f7b4c0d3c30 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b51a00700 1 -- 172.21.15.12:0/3284473394 >> 172.21.15.12:0/3284473394 conn(0x7f7b4c08aff0 msgr2=0x7f7b4c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b51a00700 1 -- 172.21.15.12:0/3284473394 shutdown_connections 2024-03-29T10:15:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.789+0000 7f7b51a00700 1 -- 172.21.15.12:0/3284473394 wait complete. 2024-03-29T10:15:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.791+0000 7f7b51a00700 1 Processor -- start 2024-03-29T10:15:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.791+0000 7f7b51a00700 1 -- start start 2024-03-29T10:15:37.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b51a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c0d16a0 0x7f7b4c1e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b51a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4c0d2540 0x7f7b4c14f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b51a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b4c15d9d0 0x7f7b4c0b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b51a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c0cb5a0 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b51a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c1e82e0 con 0x7f7b4c15d9d0 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b51a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c1e85c0 con 0x7f7b4c0d2540 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b4aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c0d16a0 0x7f7b4c1e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b4aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c0d16a0 0x7f7b4c1e7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43112/0 (socket says 172.21.15.12:43112) 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b4aa00700 1 -- 172.21.15.12:0/1820139167 learned_addr learned my addr 172.21.15.12:0/1820139167 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b4a000700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4c0d2540 0x7f7b4c14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.792+0000 7f7b4b400700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b4c15d9d0 0x7f7b4c0b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b4aa00700 1 -- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4c0d2540 msgr2=0x7f7b4c14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4c0d2540 0x7f7b4c14f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b4aa00700 1 -- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b4c15d9d0 msgr2=0x7f7b4c0b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b4c15d9d0 0x7f7b4c0b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b4aa00700 1 -- 172.21.15.12:0/1820139167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3c009550 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c0d16a0 0x7f7b4c1e7c90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009c40 tx=0x7f7b3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b42a00700 1 -- 172.21.15.12:0/1820139167 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c004510 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b42a00700 1 -- 172.21.15.12:0/1820139167 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3c01e070 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b42a00700 1 -- 172.21.15.12:0/1820139167 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c0038e0 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b51a00700 1 -- 172.21.15.12:0/1820139167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c1e88a0 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.793+0000 7f7b51a00700 1 -- 172.21.15.12:0/1820139167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c14fd90 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.794+0000 7f7b51a00700 1 -- 172.21.15.12:0/1820139167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4c0d1ca0 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.795+0000 7f7b42a00700 1 -- 172.21.15.12:0/1820139167 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 81) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f7b3c003a40 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.796+0000 7f7b42a00700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b2c06ab30 0x7f7b2c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T10:15:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.796+0000 7f7b42a00700 1 -- 172.21.15.12:0/1820139167 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7b3c08a0e0 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.796+0000 7f7b4a000700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b2c06ab30 0x7f7b2c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T10:15:37.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.797+0000 7f7b4a000700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b2c06ab30 0x7f7b2c06cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c15c930 tx=0x7f7b38009040 comp rx=0 tx=0).ready entity=mgr.8824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T10:15:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:37.801+0000 7f7b42a00700 1 -- 172.21.15.12:0/1820139167 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b3c055350 con 0x7f7b4c0d16a0 2024-03-29T10:15:37.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.001+0000 7f7b42a00700 1 -- 172.21.15.12:0/1820139167 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 82) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f7b3c003cf0 con 0x7f7b4c0d16a0 2024-03-29T10:15:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.046+0000 7f7b51a00700 1 -- 172.21.15.12:0/1820139167 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b4c161fa0 con 0x7f7b2c06ab30 2024-03-29T10:15:38.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.054+0000 7f7b42a00700 1 -- 172.21.15.12:0/1820139167 <== mgr.8824 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f7b38001d70 con 0x7f7b2c06ab30 2024-03-29T10:15:38.050 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T10:15:38.051 INFO:teuthology.orchestra.run.smithi012.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-03-29T10:15:38.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 -- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b2c06ab30 msgr2=0x7f7b2c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b2c06ab30 0x7f7b2c06cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c15c930 tx=0x7f7b38009040 comp rx=0 tx=0).stop 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 -- 172.21.15.12:0/1820139167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c0d16a0 msgr2=0x7f7b4c1e7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c0d16a0 0x7f7b4c1e7c90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009c40 tx=0x7f7b3c004750 comp rx=0 tx=0).stop 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 -- 172.21.15.12:0/1820139167 shutdown_connections 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7b2c06ab30 0x7f7b2c06cff0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b4c15d9d0 0x7f7b4c0b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4c0d16a0 0x7f7b4c1e7c90 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 --2- 172.21.15.12:0/1820139167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4c0d2540 0x7f7b4c14f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 -- 172.21.15.12:0/1820139167 >> 172.21.15.12:0/1820139167 conn(0x7f7b4c08aff0 msgr2=0x7f7b4c0b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 -- 172.21.15.12:0/1820139167 shutdown_connections 2024-03-29T10:15:38.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T10:15:38.056+0000 7f7b51a00700 1 -- 172.21.15.12:0/1820139167 wait complete. 2024-03-29T10:15:38.072 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-03-29T10:15:38.072 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2024-03-29T10:15:59.057 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:15:59.062 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:16:00.572 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:16:00.575+0000 7fb00c600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:16:00.572 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:16:00.575+0000 7fb00c600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:16:00.572 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:16:00.575+0000 7fb00c600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:16:00.572 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:16:00.575+0000 7fb00c600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:16:00.572 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T10:16:00.575+0000 7fb00c600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T10:16:29.174 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:16:29.179 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:16:59.284 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:16:59.289 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:17:29.397 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:17:29.402 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:17:59.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:17:59.520 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:18:29.806 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:18:29.812 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:18:59.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:18:59.941 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:19:30.049 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:19:30.055 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:20:00.171 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:20:00.177 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:20:30.301 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:20:30.307 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:21:00.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:21:00.421 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:21:30.537 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:21:30.543 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:22:00.674 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:22:00.679 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:22:30.795 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:22:30.801 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:23:00.911 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:23:00.917 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:23:31.031 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:23:31.037 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:24:01.144 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:24:01.150 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:24:31.280 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:24:31.288 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:25:01.397 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:25:01.403 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:25:31.593 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:25:31.599 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:26:01.699 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:26:01.704 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:26:31.946 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:26:31.953 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:27:02.059 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:27:02.066 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:27:32.230 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:27:32.237 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:28:02.352 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:28:02.357 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:28:32.533 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:28:32.538 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:29:02.646 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:29:02.651 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:29:32.784 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:29:32.789 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:30:02.905 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:30:02.910 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:30:33.334 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:30:33.339 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:31:03.437 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:31:03.442 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:31:33.549 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:31:33.556 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:32:03.661 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:32:03.666 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:32:33.774 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:32:33.779 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:33:03.895 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:33:03.900 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:33:34.084 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:33:34.090 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:34:04.198 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:34:04.203 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:34:34.385 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:34:34.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:35:04.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:35:04.506 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:35:34.737 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:35:34.743 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:36:04.859 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:36:04.864 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:36:35.073 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:36:35.078 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:37:05.184 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:37:05.189 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:37:35.533 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:37:35.538 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:38:05.638 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:38:05.644 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:38:35.777 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:38:35.782 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:39:05.892 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:39:05.898 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:39:36.070 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:39:36.076 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:40:06.195 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:40:06.201 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:40:36.338 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:40:36.343 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:41:06.452 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:41:06.457 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:41:36.707 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:41:36.713 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:42:06.816 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:42:06.821 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:42:36.969 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:42:36.978 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:43:07.106 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:43:07.111 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:43:37.270 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:43:37.276 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:44:07.400 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:44:07.405 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:44:37.508 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:44:37.514 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:45:07.627 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:45:07.632 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:45:37.782 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:45:37.788 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:46:07.895 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:46:07.901 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:46:38.039 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:46:38.044 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:47:08.156 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:47:08.161 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:47:38.445 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:47:38.450 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:48:08.556 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:48:08.561 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:48:38.672 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:48:38.678 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:49:08.788 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:49:08.793 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:49:38.899 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:49:38.905 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:50:09.030 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:50:09.035 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:50:39.143 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:50:39.148 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:51:09.556 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:51:09.561 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:51:39.671 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:51:39.676 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:52:09.782 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:52:09.787 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:52:39.902 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:52:39.907 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:53:10.179 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:53:10.207 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:53:40.318 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:53:40.323 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:54:10.607 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:54:10.613 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:54:40.712 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:54:40.717 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:55:10.834 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:55:10.841 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:55:40.953 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:55:40.959 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:56:11.078 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:56:11.084 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:56:41.200 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:56:41.206 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:57:11.322 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:57:11.327 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:57:41.441 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:57:41.446 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:58:11.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:58:11.579 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:58:41.706 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:58:41.712 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:59:11.827 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:59:11.958 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:59:42.071 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T10:59:42.076 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:00:12.187 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:00:12.192 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:00:42.309 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:00:42.314 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:01:12.423 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:01:12.429 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:01:42.551 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:01:42.556 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:02:12.666 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:02:12.673 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:02:42.782 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:02:42.787 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:03:12.918 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:03:12.923 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:03:43.046 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:03:43.052 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:04:13.166 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:04:13.171 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:04:43.334 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:04:43.339 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:05:13.448 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:05:13.453 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:05:43.562 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:05:43.567 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:06:13.680 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:06:13.685 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:06:43.798 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:06:43.804 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:07:13.930 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:07:13.935 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:07:44.042 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:07:44.048 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:08:14.343 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:08:14.349 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:08:44.461 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:08:44.466 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:09:14.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:09:14.580 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:09:44.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:09:44.694 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:10:14.869 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:10:14.874 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:10:44.991 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:10:44.997 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:11:15.105 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:11:15.110 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:11:16.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:16.073 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:11:16.128 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_17_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_18_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_19_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_20_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_21_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_22_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_23_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_24_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_25_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_26_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_27_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_28_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_29_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_30_00_UTC 2024-03-29T11:11:16.129 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_31_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_32_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_33_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_34_01_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_35_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_36_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_37_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_38_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_39_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_40_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_41_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_42_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_43_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_44_00_UTC 2024-03-29T11:11:16.130 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_45_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_46_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_47_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_48_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_49_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_50_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_51_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_52_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_53_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_54_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_55_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_56_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_57_00_UTC 2024-03-29T11:11:16.131 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_58_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_59_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_00_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_01_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_02_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_03_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_04_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_05_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_06_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_07_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_08_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_09_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_10_00_UTC 2024-03-29T11:11:16.132 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_11_00_UTC 2024-03-29T11:11:16.134 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-10_17_00_UTC', 'scheduled-2024-03-29-10_18_00_UTC', 'scheduled-2024-03-29-10_19_00_UTC', 'scheduled-2024-03-29-10_20_00_UTC', 'scheduled-2024-03-29-10_21_00_UTC', 'scheduled-2024-03-29-10_22_00_UTC', 'scheduled-2024-03-29-10_23_00_UTC', 'scheduled-2024-03-29-10_24_00_UTC', 'scheduled-2024-03-29-10_25_00_UTC', 'scheduled-2024-03-29-10_26_00_UTC', 'scheduled-2024-03-29-10_27_00_UTC', 'scheduled-2024-03-29-10_28_00_UTC', 'scheduled-2024-03-29-10_29_00_UTC', 'scheduled-2024-03-29-10_30_00_UTC', 'scheduled-2024-03-29-10_31_00_UTC', 'scheduled-2024-03-29-10_32_00_UTC', 'scheduled-2024-03-29-10_33_00_UTC', 'scheduled-2024-03-29-10_34_01_UTC', 'scheduled-2024-03-29-10_35_00_UTC', 'scheduled-2024-03-29-10_36_00_UTC', 'scheduled-2024-03-29-10_37_00_UTC', 'scheduled-2024-03-29-10_38_00_UTC', 'scheduled-2024-03-29-10_39_00_UTC', 'scheduled-2024-03-29-10_40_00_UTC', 'scheduled-2024-03-29-10_41_00_UTC', 'scheduled-2024-03-29-10_42_00_UTC', 'scheduled-2024-03-29-10_43_00_UTC', 'scheduled-2024-03-29-10_44_00_UTC', 'scheduled-2024-03-29-10_45_00_UTC', 'scheduled-2024-03-29-10_46_00_UTC', 'scheduled-2024-03-29-10_47_00_UTC', 'scheduled-2024-03-29-10_48_00_UTC', 'scheduled-2024-03-29-10_49_00_UTC', 'scheduled-2024-03-29-10_50_00_UTC', 'scheduled-2024-03-29-10_51_00_UTC', 'scheduled-2024-03-29-10_52_00_UTC', 'scheduled-2024-03-29-10_53_00_UTC', 'scheduled-2024-03-29-10_54_00_UTC', 'scheduled-2024-03-29-10_55_00_UTC', 'scheduled-2024-03-29-10_56_00_UTC', 'scheduled-2024-03-29-10_57_00_UTC', 'scheduled-2024-03-29-10_58_00_UTC', 'scheduled-2024-03-29-10_59_00_UTC', 'scheduled-2024-03-29-11_00_00_UTC', 'scheduled-2024-03-29-11_01_00_UTC', 'scheduled-2024-03-29-11_02_00_UTC', 'scheduled-2024-03-29-11_03_00_UTC', 'scheduled-2024-03-29-11_04_00_UTC', 'scheduled-2024-03-29-11_05_00_UTC', 'scheduled-2024-03-29-11_06_00_UTC', 'scheduled-2024-03-29-11_07_00_UTC', 'scheduled-2024-03-29-11_08_00_UTC', 'scheduled-2024-03-29-11_09_00_UTC', 'scheduled-2024-03-29-11_10_00_UTC', 'scheduled-2024-03-29-11_11_00_UTC'] 2024-03-29T11:11:16.134 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2024-03-29T11:11:16.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.360+0000 7fe256a00700 1 Processor -- start 2024-03-29T11:11:16.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.361+0000 7fe256a00700 1 -- start start 2024-03-29T11:11:16.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.362+0000 7fe256a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe25015a5c0 0x7fe2500709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.362+0000 7fe256a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe25015dfe0 0x7fe2500ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.362+0000 7fe256a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2500cb310 con 0x7fe2500ce670 2024-03-29T11:11:16.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.362+0000 7fe256a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2500ceae0 con 0x7fe25015a5c0 2024-03-29T11:11:16.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.362+0000 7fe256a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe250070f00 con 0x7fe25015dfe0 2024-03-29T11:11:16.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.363+0000 7fe24fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe25015a5c0 0x7fe2500709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.363+0000 7fe24fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe25015a5c0 0x7fe2500709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54964/0 (socket says 172.21.15.12:54964) 2024-03-29T11:11:16.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.363+0000 7fe24fe00700 1 -- 172.21.15.12:0/3824280792 learned_addr learned my addr 172.21.15.12:0/3824280792 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:16.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.363+0000 7fe24f400700 1 --2- 172.21.15.12:0/3824280792 >> v2:172.21.15.159:3300/0 conn(0x7fe25015dfe0 0x7fe2500ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.363+0000 7fe24fe00700 1 -- 172.21.15.12:0/3824280792 >> v2:172.21.15.159:3300/0 conn(0x7fe25015dfe0 msgr2=0x7fe2500ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:16.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.363+0000 7fe24fe00700 1 --2- 172.21.15.12:0/3824280792 >> v2:172.21.15.159:3300/0 conn(0x7fe25015dfe0 0x7fe2500ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.363+0000 7fe24fe00700 1 -- 172.21.15.12:0/3824280792 >> v1:172.21.15.159:6789/0 conn(0x7fe2500ce670 legacy=0x7fe2500b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:16.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.363+0000 7fe24fe00700 1 -- 172.21.15.12:0/3824280792 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2500cca90 con 0x7fe25015a5c0 2024-03-29T11:11:16.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.363+0000 7fe24fe00700 1 --2- 172.21.15.12:0/3824280792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe25015a5c0 0x7fe2500709c0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fe2500d0220 tx=0x7fe240009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368ae91ad970ee0d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:16.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe24ea00700 1 -- 172.21.15.12:0/3824280792 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240004230 con 0x7fe25015a5c0 2024-03-29T11:11:16.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe24ea00700 1 -- 172.21.15.12:0/3824280792 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe240004390 con 0x7fe25015a5c0 2024-03-29T11:11:16.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe24ea00700 1 -- 172.21.15.12:0/3824280792 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2400036a0 con 0x7fe25015a5c0 2024-03-29T11:11:16.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe256a00700 1 -- 172.21.15.12:0/3824280792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe25015a5c0 msgr2=0x7fe2500709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe256a00700 1 --2- 172.21.15.12:0/3824280792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe25015a5c0 0x7fe2500709c0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fe2500d0220 tx=0x7fe240009bd0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe256a00700 1 -- 172.21.15.12:0/3824280792 shutdown_connections 2024-03-29T11:11:16.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe256a00700 1 --2- 172.21.15.12:0/3824280792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe25015a5c0 0x7fe2500709c0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe256a00700 1 --2- 172.21.15.12:0/3824280792 >> v2:172.21.15.159:3300/0 conn(0x7fe25015dfe0 0x7fe2500ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe256a00700 1 -- 172.21.15.12:0/3824280792 >> 172.21.15.12:0/3824280792 conn(0x7fe2500b0d70 msgr2=0x7fe2500b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:16.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe256a00700 1 -- 172.21.15.12:0/3824280792 shutdown_connections 2024-03-29T11:11:16.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.364+0000 7fe256a00700 1 -- 172.21.15.12:0/3824280792 wait complete. 2024-03-29T11:11:16.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.367+0000 7fe256a00700 1 Processor -- start 2024-03-29T11:11:16.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.367+0000 7fe256a00700 1 -- start start 2024-03-29T11:11:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.368+0000 7fe256a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2500ce670 0x7fe25015c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.368+0000 7fe256a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe25015a5c0 0x7fe250140e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.368+0000 7fe256a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25015dfe0 0x7fe2500cf320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe256a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25015b3f0 con 0x7fe2500ce670 2024-03-29T11:11:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe256a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25015cbc0 con 0x7fe25015dfe0 2024-03-29T11:11:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe256a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25015b5e0 con 0x7fe25015a5c0 2024-03-29T11:11:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe24fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2500ce670 0x7fe25015c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe24fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2500ce670 0x7fe25015c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54980/0 (socket says 172.21.15.12:54980) 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe24fe00700 1 -- 172.21.15.12:0/3633066258 learned_addr learned my addr 172.21.15.12:0/3633066258 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe254a00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25015dfe0 0x7fe2500cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe24f400700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe25015a5c0 0x7fe250140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe24fe00700 1 -- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe25015a5c0 msgr2=0x7fe250140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe24fe00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe25015a5c0 0x7fe250140e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe24fe00700 1 -- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25015dfe0 msgr2=0x7fe2500cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe24fe00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25015dfe0 0x7fe2500cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe24fe00700 1 -- 172.21.15.12:0/3633066258 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe240009550 con 0x7fe2500ce670 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.369+0000 7fe24fe00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2500ce670 0x7fe25015c680 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fe240009130 tx=0x7fe240004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.370+0000 7fe247e00700 1 -- 172.21.15.12:0/3633066258 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240003d40 con 0x7fe2500ce670 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.370+0000 7fe247e00700 1 -- 172.21.15.12:0/3633066258 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24001f070 con 0x7fe2500ce670 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.370+0000 7fe256a00700 1 -- 172.21.15.12:0/3633066258 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25015b8c0 con 0x7fe2500ce670 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.370+0000 7fe247e00700 1 -- 172.21.15.12:0/3633066258 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240011870 con 0x7fe2500ce670 2024-03-29T11:11:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.370+0000 7fe256a00700 1 -- 172.21.15.12:0/3633066258 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25015be80 con 0x7fe2500ce670 2024-03-29T11:11:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.371+0000 7fe256a00700 1 -- 172.21.15.12:0/3633066258 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe250067720 con 0x7fe2500ce670 2024-03-29T11:11:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.372+0000 7fe247e00700 1 -- 172.21.15.12:0/3633066258 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 85) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fe240021030 con 0x7fe2500ce670 2024-03-29T11:11:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.373+0000 7fe247e00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe23006ae00 0x7fe23006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.373+0000 7fe247e00700 1 -- 172.21.15.12:0/3633066258 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6155+0+0 (secure 0 0 0) 0x7fe24008ac50 con 0x7fe2500ce670 2024-03-29T11:11:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.373+0000 7fe24f400700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe23006ae00 0x7fe23006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.374+0000 7fe24f400700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe23006ae00 0x7fe23006d2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe250141df0 tx=0x7fe23c008040 comp rx=0 tx=0).ready entity=mgr.8824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.378+0000 7fe247e00700 1 -- 172.21.15.12:0/3633066258 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe240055b50 con 0x7fe2500ce670 2024-03-29T11:11:16.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.617+0000 7fe256a00700 1 -- 172.21.15.12:0/3633066258 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe250071760 con 0x7fe23006ae00 2024-03-29T11:11:16.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.627+0000 7fe247e00700 1 -- 172.21.15.12:0/3633066258 <== mgr.8824 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+373 (secure 0 0 0) 0x7fe23c00be50 con 0x7fe23006ae00 2024-03-29T11:11:16.606 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:16.606 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T10:15:36", "first": "2024-03-29T10:16:00", "last": "2024-03-29T11:11:00", "last_pruned": "2024-03-29T11:11:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-03-29T11:11:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 -- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe23006ae00 msgr2=0x7fe23006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe23006ae00 0x7fe23006d2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe250141df0 tx=0x7fe23c008040 comp rx=0 tx=0).stop 2024-03-29T11:11:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 -- 172.21.15.12:0/3633066258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2500ce670 msgr2=0x7fe25015c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2500ce670 0x7fe25015c680 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fe240009130 tx=0x7fe240004750 comp rx=0 tx=0).stop 2024-03-29T11:11:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 -- 172.21.15.12:0/3633066258 shutdown_connections 2024-03-29T11:11:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe23006ae00 0x7fe23006d2c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25015dfe0 0x7fe2500cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2500ce670 0x7fe25015c680 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 --2- 172.21.15.12:0/3633066258 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe25015a5c0 0x7fe250140e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 -- 172.21.15.12:0/3633066258 >> 172.21.15.12:0/3633066258 conn(0x7fe2500b0d70 msgr2=0x7fe2500b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 -- 172.21.15.12:0/3633066258 shutdown_connections 2024-03-29T11:11:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.630+0000 7fe256a00700 1 -- 172.21.15.12:0/3633066258 wait complete. 2024-03-29T11:11:16.625 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T10:15:36", "first": "2024-03-29T10:16:00", "last": "2024-03-29T11:11:00", "last_pruned": "2024-03-29T11:11:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-03-29T11:11:16.626 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T10:15:36', 'first': '2024-03-29T10:16:00', 'last': '2024-03-29T11:11:00', 'last_pruned': '2024-03-29T11:11:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2024-03-29T11:11:16.626 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2024-03-29T11:11:16.626 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2024-03-29T11:11:16.626 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2024-03-29T11:11:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.848+0000 7f04bec00700 1 Processor -- start 2024-03-29T11:11:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.848+0000 7f04bec00700 1 -- start start 2024-03-29T11:11:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.849+0000 7f04bec00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b815a5c0 0x7f04b80709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.849+0000 7f04bec00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f04b815dfe0 0x7f04b80ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.849+0000 7f04bec00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b80cb310 con 0x7f04b80ce670 2024-03-29T11:11:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.849+0000 7f04bec00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b80ceb00 con 0x7f04b815dfe0 2024-03-29T11:11:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.849+0000 7f04bec00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b8070f00 con 0x7f04b815a5c0 2024-03-29T11:11:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.850+0000 7f04b7400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f04b815dfe0 0x7f04b80ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.850+0000 7f04b7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b815a5c0 0x7f04b80709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.850+0000 7f04bcc00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f04b80ce670 0x7f04b80b2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:45730/0 (socket says 172.21.15.12:45730) 2024-03-29T11:11:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.850+0000 7f04bcc00700 1 -- 172.21.15.12:0/2868699497 learned_addr learned my addr 172.21.15.12:0/2868699497 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.850+0000 7f04b7e00700 1 -- 172.21.15.12:0/2868699497 >> v2:172.21.15.159:3300/0 conn(0x7f04b815dfe0 msgr2=0x7f04b80ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.850+0000 7f04b7e00700 1 --2- 172.21.15.12:0/2868699497 >> v2:172.21.15.159:3300/0 conn(0x7f04b815dfe0 0x7f04b80ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.850+0000 7f04b7e00700 1 -- 172.21.15.12:0/2868699497 >> v1:172.21.15.159:6789/0 conn(0x7f04b80ce670 legacy=0x7f04b80b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.850+0000 7f04b7e00700 1 -- 172.21.15.12:0/2868699497 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b80cca90 con 0x7f04b815a5c0 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.850+0000 7f04b7400700 1 --2- 172.21.15.12:0/2868699497 >> v2:172.21.15.159:3300/0 conn(0x7f04b815dfe0 0x7f04b80ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.851+0000 7f04b7e00700 1 --2- 172.21.15.12:0/2868699497 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b815a5c0 0x7f04b80709c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f04a8009840 tx=0x7f04a8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daf02d3ab095b51f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.851+0000 7f04b6a00700 1 -- 172.21.15.12:0/2868699497 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8011830 con 0x7f04b815a5c0 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.851+0000 7f04b6a00700 1 -- 172.21.15.12:0/2868699497 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a8011990 con 0x7f04b815a5c0 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.851+0000 7f04b6a00700 1 -- 172.21.15.12:0/2868699497 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8011d20 con 0x7f04b815a5c0 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.852+0000 7f04bec00700 1 -- 172.21.15.12:0/2868699497 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b815a5c0 msgr2=0x7f04b80709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.852+0000 7f04bec00700 1 --2- 172.21.15.12:0/2868699497 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b815a5c0 0x7f04b80709c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f04a8009840 tx=0x7f04a8009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.852+0000 7f04bec00700 1 -- 172.21.15.12:0/2868699497 shutdown_connections 2024-03-29T11:11:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.852+0000 7f04bec00700 1 --2- 172.21.15.12:0/2868699497 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b815a5c0 0x7f04b80709c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.852+0000 7f04bec00700 1 --2- 172.21.15.12:0/2868699497 >> v2:172.21.15.159:3300/0 conn(0x7f04b815dfe0 0x7f04b80ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.852+0000 7f04bec00700 1 -- 172.21.15.12:0/2868699497 >> 172.21.15.12:0/2868699497 conn(0x7f04b80b0d70 msgr2=0x7f04b80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.852+0000 7f04bec00700 1 -- 172.21.15.12:0/2868699497 shutdown_connections 2024-03-29T11:11:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.852+0000 7f04bec00700 1 -- 172.21.15.12:0/2868699497 wait complete. 2024-03-29T11:11:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.854+0000 7f04bec00700 1 Processor -- start 2024-03-29T11:11:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.854+0000 7f04bec00700 1 -- start start 2024-03-29T11:11:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bec00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b80ce670 0x7f04b815d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bec00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04b815a5c0 0x7f04b815b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bec00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04b815dfe0 0x7f04b815c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bec00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b815aa30 con 0x7f04b815dfe0 2024-03-29T11:11:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bec00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b815b2d0 con 0x7f04b815a5c0 2024-03-29T11:11:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bec00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b80aced0 con 0x7f04b80ce670 2024-03-29T11:11:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bcc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04b815dfe0 0x7f04b815c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bcc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04b815dfe0 0x7f04b815c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54982/0 (socket says 172.21.15.12:54982) 2024-03-29T11:11:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bcc00700 1 -- 172.21.15.12:0/3102252282 learned_addr learned my addr 172.21.15.12:0/3102252282 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04b7400700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04b815a5c0 0x7f04b815b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bcc00700 1 -- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b80ce670 msgr2=0x7f04b815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04b7e00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b80ce670 0x7f04b815d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bcc00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b80ce670 0x7f04b815d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bcc00700 1 -- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04b815a5c0 msgr2=0x7f04b815b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bcc00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04b815a5c0 0x7f04b815b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04bcc00700 1 -- 172.21.15.12:0/3102252282 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a40094a0 con 0x7f04b815dfe0 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04b7400700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04b815a5c0 0x7f04b815b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.855+0000 7f04b7e00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b80ce670 0x7f04b815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.856+0000 7f04bcc00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04b815dfe0 0x7f04b815c3c0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f04b000cbf0 tx=0x7f04b00021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.856+0000 7f04afe00700 1 -- 172.21.15.12:0/3102252282 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b000ab40 con 0x7f04b815dfe0 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.856+0000 7f04afe00700 1 -- 172.21.15.12:0/3102252282 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b000aca0 con 0x7f04b815dfe0 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.856+0000 7f04bec00700 1 -- 172.21.15.12:0/3102252282 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a80094a0 con 0x7f04b815dfe0 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.856+0000 7f04afe00700 1 -- 172.21.15.12:0/3102252282 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b0012600 con 0x7f04b815dfe0 2024-03-29T11:11:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.856+0000 7f04bec00700 1 -- 172.21.15.12:0/3102252282 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b815af60 con 0x7f04b815dfe0 2024-03-29T11:11:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.857+0000 7f04bec00700 1 -- 172.21.15.12:0/3102252282 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b8067720 con 0x7f04b815dfe0 2024-03-29T11:11:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.858+0000 7f04afe00700 1 -- 172.21.15.12:0/3102252282 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 85) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f04b001b3b0 con 0x7f04b815dfe0 2024-03-29T11:11:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.859+0000 7f04afe00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f049806ae00 0x7f049806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.859+0000 7f04afe00700 1 -- 172.21.15.12:0/3102252282 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6155+0+0 (secure 0 0 0) 0x7f04b0089320 con 0x7f04b815dfe0 2024-03-29T11:11:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.859+0000 7f04b7e00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f049806ae00 0x7f049806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.860+0000 7f04b7e00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f049806ae00 0x7f049806d2c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f04a80090b0 tx=0x7f04a8008030 comp rx=0 tx=0).ready entity=mgr.8824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:16.863+0000 7f04afe00700 1 -- 172.21.15.12:0/3102252282 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f04b00542d0 con 0x7f04b815dfe0 2024-03-29T11:11:17.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.106+0000 7f04bec00700 1 -- 172.21.15.12:0/3102252282 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f04b8071760 con 0x7f049806ae00 2024-03-29T11:11:17.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.191+0000 7f04afe00700 1 -- 172.21.15.12:0/3102252282 <== mgr.8824 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7f04a8011b80 con 0x7f049806ae00 2024-03-29T11:11:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1 2024-03-29T11:11:17.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 -- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f049806ae00 msgr2=0x7f049806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:17.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f049806ae00 0x7f049806d2c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f04a80090b0 tx=0x7f04a8008030 comp rx=0 tx=0).stop 2024-03-29T11:11:17.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 -- 172.21.15.12:0/3102252282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04b815dfe0 msgr2=0x7f04b815c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:17.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04b815dfe0 0x7f04b815c3c0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f04b000cbf0 tx=0x7f04b00021c0 comp rx=0 tx=0).stop 2024-03-29T11:11:17.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 -- 172.21.15.12:0/3102252282 shutdown_connections 2024-03-29T11:11:17.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f049806ae00 0x7f049806d2c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:17.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04b815dfe0 0x7f04b815c3c0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:17.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04b80ce670 0x7f04b815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:17.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 --2- 172.21.15.12:0/3102252282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04b815a5c0 0x7f04b815b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:17.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 -- 172.21.15.12:0/3102252282 >> 172.21.15.12:0/3102252282 conn(0x7f04b80b0d70 msgr2=0x7f04b80b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:17.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 -- 172.21.15.12:0/3102252282 shutdown_connections 2024-03-29T11:11:17.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:17.194+0000 7f04bec00700 1 -- 172.21.15.12:0/3102252282 wait complete. 2024-03-29T11:11:17.190 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2024-03-29T11:11:17.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.190 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:11:17.238 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_17_00_UTC 2024-03-29T11:11:17.238 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_18_00_UTC 2024-03-29T11:11:17.238 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_19_00_UTC 2024-03-29T11:11:17.238 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_20_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_21_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_22_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_23_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_24_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_25_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_26_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_27_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_28_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_29_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_30_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_31_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_32_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_33_00_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_34_01_UTC 2024-03-29T11:11:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_35_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_36_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_37_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_38_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_39_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_40_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_41_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_42_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_43_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_44_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_45_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_46_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_47_00_UTC 2024-03-29T11:11:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_48_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_49_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_50_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_51_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_52_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_53_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_54_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_55_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_56_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_57_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_58_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-10_59_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_00_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_01_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_02_00_UTC 2024-03-29T11:11:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_03_00_UTC 2024-03-29T11:11:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_04_00_UTC 2024-03-29T11:11:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_05_00_UTC 2024-03-29T11:11:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_06_00_UTC 2024-03-29T11:11:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_07_00_UTC 2024-03-29T11:11:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_08_00_UTC 2024-03-29T11:11:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_09_00_UTC 2024-03-29T11:11:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_10_00_UTC 2024-03-29T11:11:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_11_00_UTC 2024-03-29T11:11:17.243 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_17_00_UTC 2024-03-29T11:11:17.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.243 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_17_00_UTC) 2024-03-29T11:11:17.295 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_18_00_UTC 2024-03-29T11:11:17.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.296 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_18_00_UTC) 2024-03-29T11:11:17.347 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_19_00_UTC 2024-03-29T11:11:17.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.347 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_19_00_UTC) 2024-03-29T11:11:17.398 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_20_00_UTC 2024-03-29T11:11:17.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.398 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_20_00_UTC) 2024-03-29T11:11:17.450 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_21_00_UTC 2024-03-29T11:11:17.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.450 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_21_00_UTC) 2024-03-29T11:11:17.501 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_22_00_UTC 2024-03-29T11:11:17.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.501 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_22_00_UTC) 2024-03-29T11:11:17.552 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_23_00_UTC 2024-03-29T11:11:17.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.552 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_23_00_UTC) 2024-03-29T11:11:17.603 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_24_00_UTC 2024-03-29T11:11:17.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.603 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_24_00_UTC) 2024-03-29T11:11:17.655 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_25_00_UTC 2024-03-29T11:11:17.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.655 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_25_00_UTC) 2024-03-29T11:11:17.707 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_26_00_UTC 2024-03-29T11:11:17.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.707 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_26_00_UTC) 2024-03-29T11:11:17.759 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_27_00_UTC 2024-03-29T11:11:17.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.759 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_27_00_UTC) 2024-03-29T11:11:17.810 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_28_00_UTC 2024-03-29T11:11:17.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.811 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_28_00_UTC) 2024-03-29T11:11:17.862 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_29_00_UTC 2024-03-29T11:11:17.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.863 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_29_00_UTC) 2024-03-29T11:11:17.914 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_30_00_UTC 2024-03-29T11:11:17.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.915 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_30_00_UTC) 2024-03-29T11:11:17.966 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_31_00_UTC 2024-03-29T11:11:17.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:17.967 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_31_00_UTC) 2024-03-29T11:11:18.034 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_32_00_UTC 2024-03-29T11:11:18.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.034 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_32_00_UTC) 2024-03-29T11:11:18.093 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_33_00_UTC 2024-03-29T11:11:18.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.094 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_33_00_UTC) 2024-03-29T11:11:18.159 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_34_01_UTC 2024-03-29T11:11:18.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.159 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_34_01_UTC) 2024-03-29T11:11:18.210 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_35_00_UTC 2024-03-29T11:11:18.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.211 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_35_00_UTC) 2024-03-29T11:11:18.274 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_36_00_UTC 2024-03-29T11:11:18.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.274 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_36_00_UTC) 2024-03-29T11:11:18.326 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_37_00_UTC 2024-03-29T11:11:18.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.327 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_37_00_UTC) 2024-03-29T11:11:18.379 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_38_00_UTC 2024-03-29T11:11:18.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.380 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_38_00_UTC) 2024-03-29T11:11:18.431 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_39_00_UTC 2024-03-29T11:11:18.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.432 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_39_00_UTC) 2024-03-29T11:11:18.482 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_40_00_UTC 2024-03-29T11:11:18.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.483 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_40_00_UTC) 2024-03-29T11:11:18.534 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_41_00_UTC 2024-03-29T11:11:18.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.534 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_41_00_UTC) 2024-03-29T11:11:18.586 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_42_00_UTC 2024-03-29T11:11:18.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.586 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_42_00_UTC) 2024-03-29T11:11:18.638 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_43_00_UTC 2024-03-29T11:11:18.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.639 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_43_00_UTC) 2024-03-29T11:11:18.690 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_44_00_UTC 2024-03-29T11:11:18.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.690 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_44_00_UTC) 2024-03-29T11:11:18.741 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_45_00_UTC 2024-03-29T11:11:18.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.741 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_45_00_UTC) 2024-03-29T11:11:18.799 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_46_00_UTC 2024-03-29T11:11:18.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.799 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_46_00_UTC) 2024-03-29T11:11:18.850 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_47_00_UTC 2024-03-29T11:11:18.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.850 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_47_00_UTC) 2024-03-29T11:11:18.903 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_48_00_UTC 2024-03-29T11:11:18.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.904 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_48_00_UTC) 2024-03-29T11:11:18.955 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_49_00_UTC 2024-03-29T11:11:18.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:18.955 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_49_00_UTC) 2024-03-29T11:11:19.022 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_50_00_UTC 2024-03-29T11:11:19.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.022 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_50_00_UTC) 2024-03-29T11:11:19.078 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_51_00_UTC 2024-03-29T11:11:19.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.079 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_51_00_UTC) 2024-03-29T11:11:19.135 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_52_00_UTC 2024-03-29T11:11:19.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.135 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_52_00_UTC) 2024-03-29T11:11:19.186 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_53_00_UTC 2024-03-29T11:11:19.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.187 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_53_00_UTC) 2024-03-29T11:11:19.244 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_54_00_UTC 2024-03-29T11:11:19.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.244 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_54_00_UTC) 2024-03-29T11:11:19.294 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_55_00_UTC 2024-03-29T11:11:19.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.295 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_55_00_UTC) 2024-03-29T11:11:19.346 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_56_00_UTC 2024-03-29T11:11:19.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.347 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_56_00_UTC) 2024-03-29T11:11:19.401 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_57_00_UTC 2024-03-29T11:11:19.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.401 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_57_00_UTC) 2024-03-29T11:11:19.453 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_58_00_UTC 2024-03-29T11:11:19.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.453 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_58_00_UTC) 2024-03-29T11:11:19.503 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-10_59_00_UTC 2024-03-29T11:11:19.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.503 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-10_59_00_UTC) 2024-03-29T11:11:19.553 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_00_00_UTC 2024-03-29T11:11:19.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.554 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_00_00_UTC) 2024-03-29T11:11:19.604 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_01_00_UTC 2024-03-29T11:11:19.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.604 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_01_00_UTC) 2024-03-29T11:11:19.654 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_02_00_UTC 2024-03-29T11:11:19.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.654 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_02_00_UTC) 2024-03-29T11:11:19.703 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_03_00_UTC 2024-03-29T11:11:19.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.704 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_03_00_UTC) 2024-03-29T11:11:19.756 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_04_00_UTC 2024-03-29T11:11:19.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.756 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_04_00_UTC) 2024-03-29T11:11:19.806 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_05_00_UTC 2024-03-29T11:11:19.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.807 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_05_00_UTC) 2024-03-29T11:11:19.856 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_06_00_UTC 2024-03-29T11:11:19.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.857 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_06_00_UTC) 2024-03-29T11:11:19.906 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_07_00_UTC 2024-03-29T11:11:19.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.907 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_07_00_UTC) 2024-03-29T11:11:19.956 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_08_00_UTC 2024-03-29T11:11:19.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:19.957 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_08_00_UTC) 2024-03-29T11:11:20.007 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_09_00_UTC 2024-03-29T11:11:20.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:20.007 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_09_00_UTC) 2024-03-29T11:11:20.102 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_10_00_UTC 2024-03-29T11:11:20.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:20.102 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_10_00_UTC) 2024-03-29T11:11:20.186 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_11_00_UTC 2024-03-29T11:11:20.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:20.186 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_11_00_UTC) 2024-03-29T11:11:20.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:20.236 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-29T11:11:20.310 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T11:11:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.532+0000 7fef16800700 1 Processor -- start 2024-03-29T11:11:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.532+0000 7fef16800700 1 -- start start 2024-03-29T11:11:20.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.533+0000 7fef16800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fef100d21f0 0x7fef100ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:20.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.533+0000 7fef16800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100cec40 0x7fef10161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:20.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.533+0000 7fef16800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef100ce270 con 0x7fef101611b0 2024-03-29T11:11:20.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.533+0000 7fef16800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef1015b600 con 0x7fef100d21f0 2024-03-29T11:11:20.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.533+0000 7fef16800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef100d2760 con 0x7fef100cec40 2024-03-29T11:11:20.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fef101611b0 0x7fef1015ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47256/0 (socket says 172.21.15.12:47256) 2024-03-29T11:11:20.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0f400700 1 -- 172.21.15.12:0/3054441870 learned_addr learned my addr 172.21.15.12:0/3054441870 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0ea00700 1 --2- 172.21.15.12:0/3054441870 >> v2:172.21.15.159:3300/0 conn(0x7fef100d21f0 0x7fef100ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0fe00700 1 --2- 172.21.15.12:0/3054441870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100cec40 0x7fef10161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0e000700 1 -- 172.21.15.12:0/3054441870 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295402376 0 0) 0x7fef100ce270 con 0x7fef101611b0 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0e000700 1 -- 172.21.15.12:0/3054441870 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef40035b0 con 0x7fef101611b0 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0ea00700 1 -- 172.21.15.12:0/3054441870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100cec40 msgr2=0x7fef10161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0ea00700 1 --2- 172.21.15.12:0/3054441870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100cec40 0x7fef10161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0e000700 1 -- 172.21.15.12:0/3054441870 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2204595279 0 0) 0x7feef40035b0 con 0x7fef101611b0 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0ea00700 1 -- 172.21.15.12:0/3054441870 >> v1:172.21.15.12:6789/0 conn(0x7fef101611b0 legacy=0x7fef1015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.534+0000 7fef0ea00700 1 -- 172.21.15.12:0/3054441870 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef100d2a40 con 0x7fef100d21f0 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.535+0000 7fef0ea00700 1 --2- 172.21.15.12:0/3054441870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef100d21f0 0x7fef100ce700 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7feefc009840 tx=0x7feefc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb1991dd4b0ba1b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.535+0000 7fef0e000700 1 -- 172.21.15.12:0/3054441870 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef00001670 con 0x7fef100d21f0 2024-03-29T11:11:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.535+0000 7fef0e000700 1 -- 172.21.15.12:0/3054441870 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feefc004030 con 0x7fef100d21f0 2024-03-29T11:11:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.535+0000 7fef0e000700 1 -- 172.21.15.12:0/3054441870 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc0043c0 con 0x7fef100d21f0 2024-03-29T11:11:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.535+0000 7fef16800700 1 -- 172.21.15.12:0/3054441870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef100d21f0 msgr2=0x7fef100ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.535+0000 7fef16800700 1 --2- 172.21.15.12:0/3054441870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef100d21f0 0x7fef100ce700 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7feefc009840 tx=0x7feefc009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.536+0000 7fef16800700 1 -- 172.21.15.12:0/3054441870 shutdown_connections 2024-03-29T11:11:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.536+0000 7fef16800700 1 --2- 172.21.15.12:0/3054441870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100cec40 0x7fef10161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.536+0000 7fef16800700 1 --2- 172.21.15.12:0/3054441870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef100d21f0 0x7fef100ce700 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.536+0000 7fef16800700 1 -- 172.21.15.12:0/3054441870 >> 172.21.15.12:0/3054441870 conn(0x7fef1008aff0 msgr2=0x7fef100d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.536+0000 7fef16800700 1 -- 172.21.15.12:0/3054441870 shutdown_connections 2024-03-29T11:11:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.536+0000 7fef16800700 1 -- 172.21.15.12:0/3054441870 wait complete. 2024-03-29T11:11:20.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.537+0000 7fef16800700 1 Processor -- start 2024-03-29T11:11:20.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.538+0000 7fef16800700 1 -- start start 2024-03-29T11:11:20.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef16800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef100cec40 0x7fef1014c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:20.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef16800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100d21f0 0x7fef100ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:20.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef16800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef101611b0 0x7fef10158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef16800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef100b8740 con 0x7fef101611b0 2024-03-29T11:11:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef16800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef100b76a0 con 0x7fef100cec40 2024-03-29T11:11:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef16800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef100b7980 con 0x7fef100d21f0 2024-03-29T11:11:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef101611b0 0x7fef10158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef101611b0 0x7fef10158630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57964/0 (socket says 172.21.15.12:57964) 2024-03-29T11:11:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0fe00700 1 -- 172.21.15.12:0/38896393 learned_addr learned my addr 172.21.15.12:0/38896393 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0f400700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef100cec40 0x7fef1014c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0ea00700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100d21f0 0x7fef100ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:20.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0fe00700 1 -- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100d21f0 msgr2=0x7fef100ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0fe00700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100d21f0 0x7fef100ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0fe00700 1 -- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef100cec40 msgr2=0x7fef1014c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0fe00700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef100cec40 0x7fef1014c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0fe00700 1 -- 172.21.15.12:0/38896393 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feefc0094a0 con 0x7fef101611b0 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0f400700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef100cec40 0x7fef1014c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.539+0000 7fef0ea00700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100d21f0 0x7fef100ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.540+0000 7fef0fe00700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef101611b0 0x7fef10158630 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fef0800efc0 tx=0x7fef0800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.540+0000 7fef07400700 1 -- 172.21.15.12:0/38896393 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0800cd40 con 0x7fef101611b0 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.540+0000 7fef07400700 1 -- 172.21.15.12:0/38896393 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0800f040 con 0x7fef101611b0 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.540+0000 7fef07400700 1 -- 172.21.15.12:0/38896393 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef080146c0 con 0x7fef101611b0 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.540+0000 7fef16800700 1 -- 172.21.15.12:0/38896393 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef100b7c60 con 0x7fef101611b0 2024-03-29T11:11:20.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.540+0000 7fef16800700 1 -- 172.21.15.12:0/38896393 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef100b8220 con 0x7fef101611b0 2024-03-29T11:11:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.541+0000 7fef16800700 1 -- 172.21.15.12:0/38896393 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef10162150 con 0x7fef101611b0 2024-03-29T11:11:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.542+0000 7fef07400700 1 -- 172.21.15.12:0/38896393 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 85) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fef08014820 con 0x7fef101611b0 2024-03-29T11:11:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.543+0000 7fef07400700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feef006ae00 0x7feef006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.543+0000 7fef07400700 1 -- 172.21.15.12:0/38896393 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fef0808a4e0 con 0x7fef101611b0 2024-03-29T11:11:20.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.543+0000 7fef0f400700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feef006ae00 0x7feef006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:20.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.544+0000 7fef0f400700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feef006ae00 0x7feef006d2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fef00007fd0 tx=0x7fef00009040 comp rx=0 tx=0).ready entity=mgr.8824 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:20.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.547+0000 7fef07400700 1 -- 172.21.15.12:0/38896393 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef08055210 con 0x7fef101611b0 2024-03-29T11:11:20.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:20.855+0000 7fef16800700 1 -- 172.21.15.12:0/38896393 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fef10067720 con 0x7fef101611b0 2024-03-29T11:11:21.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.082+0000 7fef07400700 1 -- 172.21.15.12:0/38896393 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fef080555e0 con 0x7fef101611b0 2024-03-29T11:11:21.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.083+0000 7fef07400700 1 -- 172.21.15.12:0/38896393 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v86) v1 ==== 93+0+0 (secure 0 0 0) 0x7fef08087c30 con 0x7fef101611b0 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.086+0000 7fef16800700 1 -- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feef006ae00 msgr2=0x7feef006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.086+0000 7fef16800700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feef006ae00 0x7feef006d2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fef00007fd0 tx=0x7fef00009040 comp rx=0 tx=0).stop 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.086+0000 7fef16800700 1 -- 172.21.15.12:0/38896393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef101611b0 msgr2=0x7fef10158630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.086+0000 7fef16800700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef101611b0 0x7fef10158630 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fef0800efc0 tx=0x7fef0800c3a0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.086+0000 7fef16800700 1 -- 172.21.15.12:0/38896393 shutdown_connections 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.086+0000 7fef16800700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feef006ae00 0x7feef006d2c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.086+0000 7fef16800700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef101611b0 0x7fef10158630 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.086+0000 7fef16800700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef100cec40 0x7fef1014c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.086+0000 7fef16800700 1 --2- 172.21.15.12:0/38896393 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef100d21f0 0x7fef100ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.086+0000 7fef16800700 1 -- 172.21.15.12:0/38896393 >> 172.21.15.12:0/38896393 conn(0x7fef1008aff0 msgr2=0x7fef1015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.087+0000 7fef16800700 1 -- 172.21.15.12:0/38896393 shutdown_connections 2024-03-29T11:11:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.087+0000 7fef16800700 1 -- 172.21.15.12:0/38896393 wait complete. 2024-03-29T11:11:21.086 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:11:21.086 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:11:21.176 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:11:21.287 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:11:21.287 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:11:21.287 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:11:21.340 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:21.362+0000 7f6fd2963240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:11:21.347 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:21.369+0000 7f94498a8240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:11:21.354 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:11:21.354 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:11:21.430 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:21.580 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:21.602+0000 7f94498a8240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:11:21.608 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:21.630+0000 7f6fd2963240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:11:21.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.732+0000 7f2b84600700 1 Processor -- start 2024-03-29T11:11:21.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.732+0000 7f2b84600700 1 -- start start 2024-03-29T11:11:21.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.733+0000 7f2b84600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 0x7f2b7c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:21.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.733+0000 7f2b84600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7c15fd50 con 0x7f2b7c0cb3f0 2024-03-29T11:11:21.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.733+0000 7f2b84600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7c15f140 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.733+0000 7f2b84600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c15b9b0 con 0x7f2b7c15b5d0 2024-03-29T11:11:21.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.733+0000 7f2b81200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2b7c0cb3f0 0x7f2b7c0ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47266/0 (socket says 172.21.15.12:47266) 2024-03-29T11:11:21.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.733+0000 7f2b81200700 1 -- 172.21.15.12:0/1068295104 learned_addr learned my addr 172.21.15.12:0/1068295104 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:21.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.734+0000 7f2b81c00700 1 --2- 172.21.15.12:0/1068295104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 0x7f2b7c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.734+0000 7f2b7be00700 1 -- 172.21.15.12:0/1068295104 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381513348 0 0) 0x7f2b7c15fd50 con 0x7f2b7c0cb3f0 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.734+0000 7f2b7be00700 1 -- 172.21.15.12:0/1068295104 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b640035b0 con 0x7f2b7c0cb3f0 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.734+0000 7f2b7be00700 1 -- 172.21.15.12:0/1068295104 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856946270 0 0) 0x7f2b7c15f140 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.734+0000 7f2b7be00700 1 -- 172.21.15.12:0/1068295104 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b7c15fd50 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.734+0000 7f2b81c00700 1 -- 172.21.15.12:0/1068295104 >> v1:172.21.15.159:6789/0 conn(0x7f2b7c0ac7d0 legacy=0x7f2b7c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.734+0000 7f2b81c00700 1 -- 172.21.15.12:0/1068295104 >> v1:172.21.15.12:6789/0 conn(0x7f2b7c0cb3f0 legacy=0x7f2b7c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.734+0000 7f2b81c00700 1 -- 172.21.15.12:0/1068295104 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c15f420 con 0x7f2b7c15b5d0 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.734+0000 7f2b81c00700 1 --2- 172.21.15.12:0/1068295104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 0x7f2b7c0ae3f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009870 tx=0x7f2b70009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeef4467012d3b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.735+0000 7f2b7be00700 1 -- 172.21.15.12:0/1068295104 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70011830 con 0x7f2b7c15b5d0 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.735+0000 7f2b7be00700 1 -- 172.21.15.12:0/1068295104 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b70011990 con 0x7f2b7c15b5d0 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.735+0000 7f2b7be00700 1 -- 172.21.15.12:0/1068295104 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70011d00 con 0x7f2b7c15b5d0 2024-03-29T11:11:21.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.735+0000 7f2b84600700 1 -- 172.21.15.12:0/1068295104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 msgr2=0x7f2b7c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.735+0000 7f2b84600700 1 --2- 172.21.15.12:0/1068295104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 0x7f2b7c0ae3f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009870 tx=0x7f2b70009b80 comp rx=0 tx=0).stop 2024-03-29T11:11:21.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.735+0000 7f2b84600700 1 -- 172.21.15.12:0/1068295104 shutdown_connections 2024-03-29T11:11:21.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.735+0000 7f2b84600700 1 --2- 172.21.15.12:0/1068295104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 0x7f2b7c0ae3f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.735+0000 7f2b84600700 1 -- 172.21.15.12:0/1068295104 >> 172.21.15.12:0/1068295104 conn(0x7f2b7c0cf190 msgr2=0x7f2b7c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:21.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.735+0000 7f2b84600700 1 -- 172.21.15.12:0/1068295104 shutdown_connections 2024-03-29T11:11:21.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.736+0000 7f2b84600700 1 -- 172.21.15.12:0/1068295104 wait complete. 2024-03-29T11:11:21.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.738+0000 7f2b84600700 1 Processor -- start 2024-03-29T11:11:21.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.738+0000 7f2b84600700 1 -- start start 2024-03-29T11:11:21.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.739+0000 7f2b84600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b7c0ac7d0 0x7f2b7c156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:21.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.739+0000 7f2b84600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 0x7f2b7c158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:21.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.739+0000 7f2b84600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b7c0cdbd0 0x7f2b7c1f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:21.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.739+0000 7f2b84600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c157400 con 0x7f2b7c0cdbd0 2024-03-29T11:11:21.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.739+0000 7f2b84600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c1f3bd0 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.739+0000 7f2b84600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c1f3610 con 0x7f2b7c15b5d0 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b82600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b7c0cdbd0 0x7f2b7c1f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b81c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b7c0ac7d0 0x7f2b7c156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b81200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 0x7f2b7c158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b81c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b7c0ac7d0 0x7f2b7c156ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44700/0 (socket says 172.21.15.12:44700) 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b82600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b7c0cdbd0 0x7f2b7c1f2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57988/0 (socket says 172.21.15.12:57988) 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b81c00700 1 -- 172.21.15.12:0/1504793968 learned_addr learned my addr 172.21.15.12:0/1504793968 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b81c00700 1 -- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 msgr2=0x7f2b7c158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b81c00700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 0x7f2b7c158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b81c00700 1 -- 172.21.15.12:0/1504793968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b7c0cdbd0 msgr2=0x7f2b7c1f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b81c00700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b7c0cdbd0 0x7f2b7c1f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b81c00700 1 -- 172.21.15.12:0/1504793968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b700094a0 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b82600700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b7c0cdbd0 0x7f2b7c1f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:21.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.740+0000 7f2b81c00700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b7c0ac7d0 0x7f2b7c156ec0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2b700097c0 tx=0x7f2b70004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:21.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.741+0000 7f2b79600700 1 -- 172.21.15.12:0/1504793968 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70004550 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.741+0000 7f2b79600700 1 -- 172.21.15.12:0/1504793968 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7001f070 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.741+0000 7f2b84600700 1 -- 172.21.15.12:0/1504793968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7c1f38f0 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.741+0000 7f2b79600700 1 -- 172.21.15.12:0/1504793968 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b700038e0 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.741+0000 7f2b84600700 1 -- 172.21.15.12:0/1504793968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7c0cbf20 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.742+0000 7f2b84600700 1 -- 172.21.15.12:0/1504793968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7c067720 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.743+0000 7f2b79600700 1 -- 172.21.15.12:0/1504793968 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f2b70003a40 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.744+0000 7f2b79600700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2b5c06adb0 0x7f2b5c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:21.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.744+0000 7f2b79600700 1 -- 172.21.15.12:0/1504793968 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f2b70021070 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.744+0000 7f2b81200700 1 -- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2b5c06adb0 msgr2=0x7f2b5c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:21.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.744+0000 7f2b81200700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2b5c06adb0 0x7f2b5c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:21.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.749+0000 7f2b79600700 1 -- 172.21.15.12:0/1504793968 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2b70091080 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.901 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:21.922+0000 7f94498a8240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:11:21.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.945+0000 7f2b81200700 1 -- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2b5c06adb0 msgr2=0x7f2b5c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:21.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.945+0000 7f2b81200700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2b5c06adb0 0x7f2b5c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:21.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.966+0000 7f2b84600700 1 -- 172.21.15.12:0/1504793968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b7c06ea60 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.967+0000 7f2b79600700 1 -- 172.21.15.12:0/1504793968 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+16817 (secure 0 0 0) 0x7f2b7002b090 con 0x7f2b7c0ac7d0 2024-03-29T11:11:21.946 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:21.946 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":163,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:21.038087+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:15:17.110269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-29T10:15:17.717343+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":109,"snap_epoch":163,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.9","mappings":[{"from":2,"to":0}]},{"pgid":"19.23","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2146007013":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/2699841727":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/2365815480":"2024-03-30T10:15:35.915268+0000","172.21.15.159:6833/43444":"2024-03-30T10:15:35.915268+0000","172.21.15.159:6832/43444":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/3483953818":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/1508471695":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/3197130837":"2024-03-30T10:15:35.915268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":51},{"begin":59,"length":51}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":35,"length":19},{"begin":91,"length":19}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.969+0000 7f2b84600700 1 -- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2b5c06adb0 msgr2=0x7f2b5c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2b5c06adb0 0x7f2b5c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 -- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b7c0ac7d0 msgr2=0x7f2b7c156ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b7c0ac7d0 0x7f2b7c156ec0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2b700097c0 tx=0x7f2b70004750 comp rx=0 tx=0).stop 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 -- 172.21.15.12:0/1504793968 shutdown_connections 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2b5c06adb0 0x7f2b5c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b7c0cdbd0 0x7f2b7c1f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b7c0ac7d0 0x7f2b7c156ec0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 --2- 172.21.15.12:0/1504793968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b7c15b5d0 0x7f2b7c158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 -- 172.21.15.12:0/1504793968 >> 172.21.15.12:0/1504793968 conn(0x7f2b7c0cf190 msgr2=0x7f2b7c0b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 -- 172.21.15.12:0/1504793968 shutdown_connections 2024-03-29T11:11:21.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:21.970+0000 7f2b84600700 1 -- 172.21.15.12:0/1504793968 wait complete. 2024-03-29T11:11:21.951 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:21.973+0000 7f6fd2963240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:11:21.967 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:15:17.110269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:15:17.717343+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 109, 'snap_epoch': 163, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:11:21.967 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:11:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.201+0000 7f9348000700 1 Processor -- start 2024-03-29T11:11:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.201+0000 7f9348000700 1 -- start start 2024-03-29T11:11:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.203+0000 7f9348000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015dad0 0x7f93400ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.203+0000 7f9348000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f93400d21a0 0x7f934015fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.203+0000 7f9348000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934015f820 con 0x7f93400d1630 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.203+0000 7f9348000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93400cfcc0 con 0x7f93400d21a0 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.203+0000 7f9348000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93400cd9f0 con 0x7f934015dad0 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.203+0000 7f9344c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f93400d1630 0x7f93400d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47278/0 (socket says 172.21.15.12:47278) 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.203+0000 7f9344c00700 1 -- 172.21.15.12:0/669838205 learned_addr learned my addr 172.21.15.12:0/669838205 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.203+0000 7f9346000700 1 --2- 172.21.15.12:0/669838205 >> v2:172.21.15.159:3300/0 conn(0x7f93400d21a0 0x7f934015fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.203+0000 7f9345600700 1 --2- 172.21.15.12:0/669838205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015dad0 0x7f93400ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.204+0000 7f933fe00700 1 -- 172.21.15.12:0/669838205 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143560186 0 0) 0x7f934015f820 con 0x7f93400d1630 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.204+0000 7f933fe00700 1 -- 172.21.15.12:0/669838205 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93280035b0 con 0x7f93400d1630 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.204+0000 7f9346000700 1 -- 172.21.15.12:0/669838205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015dad0 msgr2=0x7f93400ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.204+0000 7f9346000700 1 --2- 172.21.15.12:0/669838205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015dad0 0x7f93400ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.204+0000 7f9346000700 1 -- 172.21.15.12:0/669838205 >> v1:172.21.15.12:6789/0 conn(0x7f93400d1630 legacy=0x7f93400d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.204+0000 7f9346000700 1 -- 172.21.15.12:0/669838205 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93400cf420 con 0x7f93400d21a0 2024-03-29T11:11:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.204+0000 7f9345600700 1 --2- 172.21.15.12:0/669838205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015dad0 0x7f93400ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.204+0000 7f9346000700 1 --2- 172.21.15.12:0/669838205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93400d21a0 0x7f934015fcb0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9338009d90 tx=0x7f933800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a085ec04ee94818 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f933fe00700 1 -- 172.21.15.12:0/669838205 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933800e070 con 0x7f93400d21a0 2024-03-29T11:11:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f933fe00700 1 -- 172.21.15.12:0/669838205 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933800bb20 con 0x7f93400d21a0 2024-03-29T11:11:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f933fe00700 1 -- 172.21.15.12:0/669838205 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9338012430 con 0x7f93400d21a0 2024-03-29T11:11:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f9348000700 1 -- 172.21.15.12:0/669838205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93400d21a0 msgr2=0x7f934015fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f9348000700 1 --2- 172.21.15.12:0/669838205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93400d21a0 0x7f934015fcb0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9338009d90 tx=0x7f933800b330 comp rx=0 tx=0).stop 2024-03-29T11:11:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f9348000700 1 -- 172.21.15.12:0/669838205 shutdown_connections 2024-03-29T11:11:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f9348000700 1 --2- 172.21.15.12:0/669838205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f934015dad0 0x7f93400ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f9348000700 1 --2- 172.21.15.12:0/669838205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93400d21a0 0x7f934015fcb0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f9348000700 1 -- 172.21.15.12:0/669838205 >> 172.21.15.12:0/669838205 conn(0x7f934008aff0 msgr2=0x7f93400d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f9348000700 1 -- 172.21.15.12:0/669838205 shutdown_connections 2024-03-29T11:11:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.205+0000 7f9348000700 1 -- 172.21.15.12:0/669838205 wait complete. 2024-03-29T11:11:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.207+0000 7f9348000700 1 Processor -- start 2024-03-29T11:11:22.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.207+0000 7f9348000700 1 -- start start 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.208+0000 7f9348000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93400d1630 0x7f93400c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.208+0000 7f9348000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d21a0 0x7f93400c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.208+0000 7f9348000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f934015dad0 0x7f93400cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.208+0000 7f9348000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93400c8e60 con 0x7f93400d1630 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.208+0000 7f9348000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93400c8490 con 0x7f934015dad0 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.208+0000 7f9348000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93400c7880 con 0x7f93400d21a0 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.208+0000 7f9345600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93400d1630 0x7f93400c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f9345600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93400d1630 0x7f93400c8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58002/0 (socket says 172.21.15.12:58002) 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f9345600700 1 -- 172.21.15.12:0/702527992 learned_addr learned my addr 172.21.15.12:0/702527992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f9344c00700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d21a0 0x7f93400c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f9346000700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f934015dad0 0x7f93400cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f9345600700 1 -- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d21a0 msgr2=0x7f93400c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f9345600700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d21a0 0x7f93400c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f9345600700 1 -- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f934015dad0 msgr2=0x7f93400cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f9345600700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f934015dad0 0x7f93400cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f9345600700 1 -- 172.21.15.12:0/702527992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340094a0 con 0x7f93400d1630 2024-03-29T11:11:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f9345600700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93400d1630 0x7f93400c8920 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f933400ebe0 tx=0x7f933400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.209+0000 7f933d600700 1 -- 172.21.15.12:0/702527992 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334009760 con 0x7f93400d1630 2024-03-29T11:11:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.210+0000 7f933d600700 1 -- 172.21.15.12:0/702527992 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933400c850 con 0x7f93400d1630 2024-03-29T11:11:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.210+0000 7f9348000700 1 -- 172.21.15.12:0/702527992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9338009a70 con 0x7f93400d1630 2024-03-29T11:11:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.210+0000 7f933d600700 1 -- 172.21.15.12:0/702527992 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933400cbe0 con 0x7f93400d1630 2024-03-29T11:11:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.210+0000 7f9348000700 1 -- 172.21.15.12:0/702527992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93400c8120 con 0x7f93400d1630 2024-03-29T11:11:22.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.211+0000 7f9348000700 1 -- 172.21.15.12:0/702527992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9340067720 con 0x7f93400d1630 2024-03-29T11:11:22.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.212+0000 7f933d600700 1 -- 172.21.15.12:0/702527992 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f933400cd40 con 0x7f93400d1630 2024-03-29T11:11:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.212+0000 7f933d600700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006adb0 0x7f932006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.213+0000 7f933d600700 1 -- 172.21.15.12:0/702527992 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f9334089700 con 0x7f93400d1630 2024-03-29T11:11:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.213+0000 7f9344c00700 1 -- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006adb0 msgr2=0x7f932006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.213+0000 7f9344c00700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006adb0 0x7f932006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.217+0000 7f933d600700 1 -- 172.21.15.12:0/702527992 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9334054540 con 0x7f93400d1630 2024-03-29T11:11:22.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.414+0000 7f9344c00700 1 -- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006adb0 msgr2=0x7f932006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:22.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.414+0000 7f9344c00700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006adb0 0x7f932006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.463+0000 7f9348000700 1 -- 172.21.15.12:0/702527992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f93400b6130 con 0x7f93400d1630 2024-03-29T11:11:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.464+0000 7f933d600700 1 -- 172.21.15.12:0/702527992 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9334057b60 con 0x7f93400d1630 2024-03-29T11:11:22.442 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:11:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.466+0000 7f9348000700 1 -- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006adb0 msgr2=0x7f932006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.466+0000 7f9348000700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006adb0 0x7f932006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.466+0000 7f9348000700 1 -- 172.21.15.12:0/702527992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93400d1630 msgr2=0x7f93400c8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.466+0000 7f9348000700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93400d1630 0x7f93400c8920 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f933400ebe0 tx=0x7f933400eef0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.466+0000 7f9348000700 1 -- 172.21.15.12:0/702527992 shutdown_connections 2024-03-29T11:11:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.466+0000 7f9348000700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f932006adb0 0x7f932006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.466+0000 7f9348000700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f934015dad0 0x7f93400cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.466+0000 7f9348000700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93400d1630 0x7f93400c8920 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.466+0000 7f9348000700 1 --2- 172.21.15.12:0/702527992 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93400d21a0 0x7f93400c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.467+0000 7f9348000700 1 -- 172.21.15.12:0/702527992 >> 172.21.15.12:0/702527992 conn(0x7f934008aff0 msgr2=0x7f934015b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.467+0000 7f9348000700 1 -- 172.21.15.12:0/702527992 shutdown_connections 2024-03-29T11:11:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.467+0000 7f9348000700 1 -- 172.21.15.12:0/702527992 wait complete. 2024-03-29T11:11:22.455 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:22.477+0000 7f94498a8240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:11:22.463 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:11:22.553 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:22.575+0000 7f6fd2963240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:11:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.688+0000 7fd971200700 1 Processor -- start 2024-03-29T11:11:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.688+0000 7fd971200700 1 -- start start 2024-03-29T11:11:22.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.689+0000 7fd971200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd96c0ced80 0x7fd96c0d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.689+0000 7fd971200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd96c15fcb0 0x7fd96c160120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.689+0000 7fd971200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96c15f820 con 0x7fd96c15dad0 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.689+0000 7fd971200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96c0cfcc0 con 0x7fd96c0ced80 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.689+0000 7fd971200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0cd9f0 con 0x7fd96c15fcb0 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.689+0000 7fd969400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd96c0ced80 0x7fd96c0d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.689+0000 7fd969e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd96c15dad0 0x7fd96c0d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47294/0 (socket says 172.21.15.12:47294) 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.689+0000 7fd969e00700 1 -- 172.21.15.12:0/2574360366 learned_addr learned my addr 172.21.15.12:0/2574360366 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.689+0000 7fd96a800700 1 --2- 172.21.15.12:0/2574360366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd96c15fcb0 0x7fd96c160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.690+0000 7fd968a00700 1 -- 172.21.15.12:0/2574360366 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752074389 0 0) 0x7fd96c15f820 con 0x7fd96c15dad0 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.690+0000 7fd968a00700 1 -- 172.21.15.12:0/2574360366 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9500035b0 con 0x7fd96c15dad0 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.690+0000 7fd969400700 1 -- 172.21.15.12:0/2574360366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd96c15fcb0 msgr2=0x7fd96c160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.690+0000 7fd969400700 1 --2- 172.21.15.12:0/2574360366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd96c15fcb0 0x7fd96c160120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.690+0000 7fd969400700 1 -- 172.21.15.12:0/2574360366 >> v1:172.21.15.12:6789/0 conn(0x7fd96c15dad0 legacy=0x7fd96c0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.690+0000 7fd969400700 1 -- 172.21.15.12:0/2574360366 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c0cf420 con 0x7fd96c0ced80 2024-03-29T11:11:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.690+0000 7fd969400700 1 --2- 172.21.15.12:0/2574360366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c0ced80 0x7fd96c0d1d40 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fd958009840 tx=0x7fd958009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f83bc16055443fc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.690+0000 7fd968a00700 1 -- 172.21.15.12:0/2574360366 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958011830 con 0x7fd96c0ced80 2024-03-29T11:11:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.691+0000 7fd968a00700 1 -- 172.21.15.12:0/2574360366 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd958011990 con 0x7fd96c0ced80 2024-03-29T11:11:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.691+0000 7fd968a00700 1 -- 172.21.15.12:0/2574360366 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958011d20 con 0x7fd96c0ced80 2024-03-29T11:11:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.691+0000 7fd971200700 1 -- 172.21.15.12:0/2574360366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c0ced80 msgr2=0x7fd96c0d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:22.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.691+0000 7fd971200700 1 --2- 172.21.15.12:0/2574360366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c0ced80 0x7fd96c0d1d40 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fd958009840 tx=0x7fd958009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:22.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.691+0000 7fd971200700 1 -- 172.21.15.12:0/2574360366 shutdown_connections 2024-03-29T11:11:22.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.691+0000 7fd971200700 1 --2- 172.21.15.12:0/2574360366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd96c15fcb0 0x7fd96c160120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.691+0000 7fd971200700 1 --2- 172.21.15.12:0/2574360366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c0ced80 0x7fd96c0d1d40 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.691+0000 7fd971200700 1 -- 172.21.15.12:0/2574360366 >> 172.21.15.12:0/2574360366 conn(0x7fd96c08aff0 msgr2=0x7fd96c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:22.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.691+0000 7fd971200700 1 -- 172.21.15.12:0/2574360366 shutdown_connections 2024-03-29T11:11:22.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.691+0000 7fd971200700 1 -- 172.21.15.12:0/2574360366 wait complete. 2024-03-29T11:11:22.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.693+0000 7fd971200700 1 Processor -- start 2024-03-29T11:11:22.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.693+0000 7fd971200700 1 -- start start 2024-03-29T11:11:22.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.694+0000 7fd971200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd96c0ced80 0x7fd96c1f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.694+0000 7fd971200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd96c15dad0 0x7fd96c1f0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.694+0000 7fd971200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c15fcb0 0x7fd96c0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.694+0000 7fd971200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0b8db0 con 0x7fd96c15fcb0 2024-03-29T11:11:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.694+0000 7fd971200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c1f0950 con 0x7fd96c15dad0 2024-03-29T11:11:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.694+0000 7fd971200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c1f2fc0 con 0x7fd96c0ced80 2024-03-29T11:11:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd96a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c15fcb0 0x7fd96c0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd96a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c15fcb0 0x7fd96c0c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58022/0 (socket says 172.21.15.12:58022) 2024-03-29T11:11:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd96a800700 1 -- 172.21.15.12:0/2644983405 learned_addr learned my addr 172.21.15.12:0/2644983405 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd969400700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd96c15dad0 0x7fd96c1f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd969e00700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd96c0ced80 0x7fd96c1f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd96a800700 1 -- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd96c0ced80 msgr2=0x7fd96c1f64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd96a800700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd96c0ced80 0x7fd96c1f64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd96a800700 1 -- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd96c15dad0 msgr2=0x7fd96c1f0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd96a800700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd96c15dad0 0x7fd96c1f0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd96a800700 1 -- 172.21.15.12:0/2644983405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9580094a0 con 0x7fd96c15fcb0 2024-03-29T11:11:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.695+0000 7fd96a800700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c15fcb0 0x7fd96c0c9620 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fd964009d30 tx=0x7fd96400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.696+0000 7fd962000700 1 -- 172.21.15.12:0/2644983405 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96400cc90 con 0x7fd96c15fcb0 2024-03-29T11:11:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.696+0000 7fd962000700 1 -- 172.21.15.12:0/2644983405 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96400f040 con 0x7fd96c15fcb0 2024-03-29T11:11:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.696+0000 7fd971200700 1 -- 172.21.15.12:0/2644983405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96c1f6a80 con 0x7fd96c15fcb0 2024-03-29T11:11:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.696+0000 7fd962000700 1 -- 172.21.15.12:0/2644983405 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964009e80 con 0x7fd96c15fcb0 2024-03-29T11:11:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.696+0000 7fd971200700 1 -- 172.21.15.12:0/2644983405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c1f7040 con 0x7fd96c15fcb0 2024-03-29T11:11:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.697+0000 7fd971200700 1 -- 172.21.15.12:0/2644983405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96c1f3190 con 0x7fd96c15fcb0 2024-03-29T11:11:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.698+0000 7fd962000700 1 -- 172.21.15.12:0/2644983405 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fd964014480 con 0x7fd96c15fcb0 2024-03-29T11:11:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.699+0000 7fd962000700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd94c06adb0 0x7fd94c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.699+0000 7fd962000700 1 -- 172.21.15.12:0/2644983405 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fd964089f80 con 0x7fd96c15fcb0 2024-03-29T11:11:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.699+0000 7fd969e00700 1 -- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd94c06adb0 msgr2=0x7fd94c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.699+0000 7fd969e00700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd94c06adb0 0x7fd94c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.708+0000 7fd962000700 1 -- 172.21.15.12:0/2644983405 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9640bd580 con 0x7fd96c15fcb0 2024-03-29T11:11:22.713 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:22.735+0000 7f94498a8240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:11:22.829 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:22.851+0000 7f6fd2963240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:11:22.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.901+0000 7fd969e00700 1 -- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd94c06adb0 msgr2=0x7fd94c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:22.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.901+0000 7fd969e00700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd94c06adb0 0x7fd94c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:22.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.921+0000 7fd971200700 1 -- 172.21.15.12:0/2644983405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd96c067720 con 0x7fd96c15fcb0 2024-03-29T11:11:22.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.922+0000 7fd962000700 1 -- 172.21.15.12:0/2644983405 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v164) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd9640180a0 con 0x7fd96c15fcb0 2024-03-29T11:11:22.900 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:11:22.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.924+0000 7fd971200700 1 -- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd94c06adb0 msgr2=0x7fd94c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:22.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.924+0000 7fd971200700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd94c06adb0 0x7fd94c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.924+0000 7fd971200700 1 -- 172.21.15.12:0/2644983405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c15fcb0 msgr2=0x7fd96c0c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:22.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.924+0000 7fd971200700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c15fcb0 0x7fd96c0c9620 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fd964009d30 tx=0x7fd96400c3a0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.924+0000 7fd971200700 1 -- 172.21.15.12:0/2644983405 shutdown_connections 2024-03-29T11:11:22.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.924+0000 7fd971200700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd94c06adb0 0x7fd94c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.924+0000 7fd971200700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd96c15fcb0 0x7fd96c0c9620 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.924+0000 7fd971200700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd96c0ced80 0x7fd96c1f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.924+0000 7fd971200700 1 --2- 172.21.15.12:0/2644983405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd96c15dad0 0x7fd96c1f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:22.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.924+0000 7fd971200700 1 -- 172.21.15.12:0/2644983405 >> 172.21.15.12:0/2644983405 conn(0x7fd96c08aff0 msgr2=0x7fd96c0d04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:22.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.925+0000 7fd971200700 1 -- 172.21.15.12:0/2644983405 shutdown_connections 2024-03-29T11:11:22.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:22.925+0000 7fd971200700 1 -- 172.21.15.12:0/2644983405 wait complete. 2024-03-29T11:11:22.921 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:11:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.146+0000 7fc2f0c00700 1 Processor -- start 2024-03-29T11:11:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.146+0000 7fc2f0c00700 1 -- start start 2024-03-29T11:11:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.147+0000 7fc2f0c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2ec0d16a0 0x7fc2ec0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.147+0000 7fc2f0c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2ec161420 con 0x7fc2ec15d9d0 2024-03-29T11:11:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.147+0000 7fc2f0c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2ec161700 con 0x7fc2ec0d2370 2024-03-29T11:11:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.147+0000 7fc2f0c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2ec0d4340 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.148+0000 7fc2e9400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2ec0d16a0 0x7fc2ec0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.148+0000 7fc2ea800700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc2ec0d2370 0x7fc2ec0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:37208/0 (socket says 172.21.15.12:37208) 2024-03-29T11:11:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.148+0000 7fc2e9400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2ec0d16a0 0x7fc2ec0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58034/0 (socket says 172.21.15.12:58034) 2024-03-29T11:11:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.148+0000 7fc2e9e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc2ec15d9d0 0x7fc2ec0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47308/0 (socket says 172.21.15.12:47308) 2024-03-29T11:11:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.148+0000 7fc2ea800700 1 -- 172.21.15.12:0/1155480578 learned_addr learned my addr 172.21.15.12:0/1155480578 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.148+0000 7fc2e9400700 1 -- 172.21.15.12:0/1155480578 >> v1:172.21.15.159:6789/0 conn(0x7fc2ec0d2370 legacy=0x7fc2ec0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.148+0000 7fc2e9400700 1 -- 172.21.15.12:0/1155480578 >> v1:172.21.15.12:6789/0 conn(0x7fc2ec15d9d0 legacy=0x7fc2ec0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.148+0000 7fc2e9400700 1 -- 172.21.15.12:0/1155480578 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec1619e0 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.148+0000 7fc2e9400700 1 --2- 172.21.15.12:0/1155480578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 0x7fc2ec0d1e30 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8009820 tx=0x7fc2d8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a652e0c880951e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.149+0000 7fc2e8a00700 1 -- 172.21.15.12:0/1155480578 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8005e30 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.149+0000 7fc2e8a00700 1 -- 172.21.15.12:0/1155480578 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d8006030 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.149+0000 7fc2e8a00700 1 -- 172.21.15.12:0/1155480578 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80063e0 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.149+0000 7fc2f0c00700 1 -- 172.21.15.12:0/1155480578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 msgr2=0x7fc2ec0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.149+0000 7fc2f0c00700 1 --2- 172.21.15.12:0/1155480578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 0x7fc2ec0d1e30 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8009820 tx=0x7fc2d8009b30 comp rx=0 tx=0).stop 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.149+0000 7fc2f0c00700 1 -- 172.21.15.12:0/1155480578 shutdown_connections 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.149+0000 7fc2f0c00700 1 --2- 172.21.15.12:0/1155480578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 0x7fc2ec0d1e30 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.149+0000 7fc2f0c00700 1 -- 172.21.15.12:0/1155480578 >> 172.21.15.12:0/1155480578 conn(0x7fc2ec08aff0 msgr2=0x7fc2ec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:23.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.149+0000 7fc2f0c00700 1 -- 172.21.15.12:0/1155480578 shutdown_connections 2024-03-29T11:11:23.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.149+0000 7fc2f0c00700 1 -- 172.21.15.12:0/1155480578 wait complete. 2024-03-29T11:11:23.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.151+0000 7fc2f0c00700 1 Processor -- start 2024-03-29T11:11:23.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.151+0000 7fc2f0c00700 1 -- start start 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.152+0000 7fc2f0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 0x7fc2ec1f5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.152+0000 7fc2f0c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2ec0d2370 0x7fc2ec0cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.152+0000 7fc2f0c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2ec15d9d0 0x7fc2ec0cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.152+0000 7fc2f0c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec0b60c0 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.152+0000 7fc2f0c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec0b63a0 con 0x7fc2ec15d9d0 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.152+0000 7fc2f0c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec0b6680 con 0x7fc2ec0d2370 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.152+0000 7fc2e9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 0x7fc2ec1f5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2e9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 0x7fc2ec1f5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58048/0 (socket says 172.21.15.12:58048) 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2e9e00700 1 -- 172.21.15.12:0/2564457882 learned_addr learned my addr 172.21.15.12:0/2564457882 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2ea800700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2ec15d9d0 0x7fc2ec0cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2e9400700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2ec0d2370 0x7fc2ec0cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:23.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2e9e00700 1 -- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2ec0d2370 msgr2=0x7fc2ec0cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2e9e00700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2ec0d2370 0x7fc2ec0cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2e9e00700 1 -- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2ec15d9d0 msgr2=0x7fc2ec0cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2e9e00700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2ec15d9d0 0x7fc2ec0cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2e9e00700 1 -- 172.21.15.12:0/2564457882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8009430 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2ea800700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2ec15d9d0 0x7fc2ec0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2e9e00700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 0x7fc2ec1f5060 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00c500 tx=0x7fc2dc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.153+0000 7fc2e2000700 1 -- 172.21.15.12:0/2564457882 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc005cb0 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.154+0000 7fc2e2000700 1 -- 172.21.15.12:0/2564457882 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc005e10 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.154+0000 7fc2e2000700 1 -- 172.21.15.12:0/2564457882 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0072d0 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.154+0000 7fc2f0c00700 1 -- 172.21.15.12:0/2564457882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ec0b6960 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.154+0000 7fc2f0c00700 1 -- 172.21.15.12:0/2564457882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ec0cc3f0 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.155+0000 7fc2f0c00700 1 -- 172.21.15.12:0/2564457882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ec0d1ca0 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.156+0000 7fc2e2000700 1 -- 172.21.15.12:0/2564457882 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fc2dc007430 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.157+0000 7fc2e2000700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2cc06adb0 0x7fc2cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.158+0000 7fc2e2000700 1 -- 172.21.15.12:0/2564457882 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fc2dc0888d0 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.158+0000 7fc2e9400700 1 -- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2cc06adb0 msgr2=0x7fc2cc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:23.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.158+0000 7fc2e9400700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2cc06adb0 0x7fc2cc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:23.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.162+0000 7fc2e2000700 1 -- 172.21.15.12:0/2564457882 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2dc053850 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.175 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:23.197+0000 7f94498a8240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:11:23.316 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:23.338+0000 7f6fd2963240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:11:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.359+0000 7fc2e9400700 1 -- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2cc06adb0 msgr2=0x7fc2cc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.359+0000 7fc2e9400700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2cc06adb0 0x7fc2cc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:23.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.376+0000 7fc2f0c00700 1 -- 172.21.15.12:0/2564457882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc2ec0b6e80 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.377+0000 7fc2e2000700 1 -- 172.21.15.12:0/2564457882 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v164) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc2ec0b6e80 con 0x7fc2ec0d16a0 2024-03-29T11:11:23.355 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:11:23.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 -- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2cc06adb0 msgr2=0x7fc2cc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:23.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2cc06adb0 0x7fc2cc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 -- 172.21.15.12:0/2564457882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 msgr2=0x7fc2ec1f5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 0x7fc2ec1f5060 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00c500 tx=0x7fc2dc00c810 comp rx=0 tx=0).stop 2024-03-29T11:11:23.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 -- 172.21.15.12:0/2564457882 shutdown_connections 2024-03-29T11:11:23.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc2cc06adb0 0x7fc2cc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2ec15d9d0 0x7fc2ec0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2ec0d16a0 0x7fc2ec1f5060 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 --2- 172.21.15.12:0/2564457882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2ec0d2370 0x7fc2ec0cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 -- 172.21.15.12:0/2564457882 >> 172.21.15.12:0/2564457882 conn(0x7fc2ec08aff0 msgr2=0x7fc2ec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:23.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 -- 172.21.15.12:0/2564457882 shutdown_connections 2024-03-29T11:11:23.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.380+0000 7fc2f0c00700 1 -- 172.21.15.12:0/2564457882 wait complete. 2024-03-29T11:11:23.376 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T11:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.606+0000 7f09e8600700 1 Processor -- start 2024-03-29T11:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.606+0000 7f09e8600700 1 -- start start 2024-03-29T11:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e8600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f09e015dad0 0x7f09e00ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e8600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1630 0x7f09e00d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e8600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09e00d1fd0 0x7f09e015deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e8600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e00cfcc0 con 0x7f09e00d1fd0 2024-03-29T11:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e8600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e00cd9f0 con 0x7f09e015dad0 2024-03-29T11:11:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e8600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e015f820 con 0x7f09e00d1630 2024-03-29T11:11:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e6600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09e00d1fd0 0x7f09e015deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e5c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f09e015dad0 0x7f09e00ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e5200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1630 0x7f09e00d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e6600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09e00d1fd0 0x7f09e015deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58062/0 (socket says 172.21.15.12:58062) 2024-03-29T11:11:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.607+0000 7f09e6600700 1 -- 172.21.15.12:0/1871452685 learned_addr learned my addr 172.21.15.12:0/1871452685 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.608+0000 7f09e6600700 1 -- 172.21.15.12:0/1871452685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1630 msgr2=0x7f09e00d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.608+0000 7f09e6600700 1 --2- 172.21.15.12:0/1871452685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1630 0x7f09e00d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.608+0000 7f09e6600700 1 -- 172.21.15.12:0/1871452685 >> v2:172.21.15.159:3300/0 conn(0x7f09e015dad0 msgr2=0x7f09e00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.608+0000 7f09e6600700 1 --2- 172.21.15.12:0/1871452685 >> v2:172.21.15.159:3300/0 conn(0x7f09e015dad0 0x7f09e00ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.608+0000 7f09e6600700 1 -- 172.21.15.12:0/1871452685 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00d2460 con 0x7f09e00d1fd0 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.608+0000 7f09e5c00700 1 --2- 172.21.15.12:0/1871452685 >> v2:172.21.15.159:3300/0 conn(0x7f09e015dad0 0x7f09e00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.608+0000 7f09e6600700 1 --2- 172.21.15.12:0/1871452685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1fd0 0x7f09e015deb0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f09e008d380 tx=0x7f09d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2e98d29dac94f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.608+0000 7f09dfe00700 1 -- 172.21.15.12:0/1871452685 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d800e070 con 0x7f09e00d1fd0 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.609+0000 7f09dfe00700 1 -- 172.21.15.12:0/1871452685 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d800b910 con 0x7f09e00d1fd0 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.609+0000 7f09dfe00700 1 -- 172.21.15.12:0/1871452685 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d800bcc0 con 0x7f09e00d1fd0 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.609+0000 7f09e8600700 1 -- 172.21.15.12:0/1871452685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1fd0 msgr2=0x7f09e015deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.609+0000 7f09e8600700 1 --2- 172.21.15.12:0/1871452685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1fd0 0x7f09e015deb0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f09e008d380 tx=0x7f09d800b330 comp rx=0 tx=0).stop 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.609+0000 7f09e8600700 1 -- 172.21.15.12:0/1871452685 shutdown_connections 2024-03-29T11:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.609+0000 7f09e8600700 1 --2- 172.21.15.12:0/1871452685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1fd0 0x7f09e015deb0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.609+0000 7f09e8600700 1 --2- 172.21.15.12:0/1871452685 >> v2:172.21.15.159:3300/0 conn(0x7f09e015dad0 0x7f09e00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.609+0000 7f09e8600700 1 --2- 172.21.15.12:0/1871452685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1630 0x7f09e00d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.609+0000 7f09e8600700 1 -- 172.21.15.12:0/1871452685 >> 172.21.15.12:0/1871452685 conn(0x7f09e008aff0 msgr2=0x7f09e00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.609+0000 7f09e8600700 1 -- 172.21.15.12:0/1871452685 shutdown_connections 2024-03-29T11:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.610+0000 7f09e8600700 1 -- 172.21.15.12:0/1871452685 wait complete. 2024-03-29T11:11:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.611+0000 7f09e8600700 1 Processor -- start 2024-03-29T11:11:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.611+0000 7f09e8600700 1 -- start start 2024-03-29T11:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e8600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1630 0x7f09e0154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e8600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1fd0 0x7f09e01f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e8600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09e015dad0 0x7f09e01f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e8600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01f7360 con 0x7f09e00d1630 2024-03-29T11:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e8600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01f6e80 con 0x7f09e015dad0 2024-03-29T11:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e8600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01f7080 con 0x7f09e00d1fd0 2024-03-29T11:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1630 0x7f09e0154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e5200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1fd0 0x7f09e01f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1630 0x7f09e0154920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58068/0 (socket says 172.21.15.12:58068) 2024-03-29T11:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e5c00700 1 -- 172.21.15.12:0/1001010430 learned_addr learned my addr 172.21.15.12:0/1001010430 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e6600700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09e015dad0 0x7f09e01f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e5200700 1 -- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09e015dad0 msgr2=0x7f09e01f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e5200700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09e015dad0 0x7f09e01f6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e5200700 1 -- 172.21.15.12:0/1001010430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1630 msgr2=0x7f09e0154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.613+0000 7f09e5200700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1630 0x7f09e0154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.614+0000 7f09e5200700 1 -- 172.21.15.12:0/1001010430 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40094a0 con 0x7f09e00d1fd0 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.614+0000 7f09e5c00700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1630 0x7f09e0154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.614+0000 7f09e5200700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1fd0 0x7f09e01f6400 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00e870 tx=0x7f09cc00eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.614+0000 7f09dd600700 1 -- 172.21.15.12:0/1001010430 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc013680 con 0x7f09e00d1fd0 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.614+0000 7f09dd600700 1 -- 172.21.15.12:0/1001010430 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc0137e0 con 0x7f09e00d1fd0 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.614+0000 7f09e8600700 1 -- 172.21.15.12:0/1001010430 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d8009b30 con 0x7f09e00d1fd0 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.614+0000 7f09dd600700 1 -- 172.21.15.12:0/1001010430 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc013b50 con 0x7f09e00d1fd0 2024-03-29T11:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.614+0000 7f09e8600700 1 -- 172.21.15.12:0/1001010430 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e0152c40 con 0x7f09e00d1fd0 2024-03-29T11:11:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.616+0000 7f09e8600700 1 -- 172.21.15.12:0/1001010430 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e0067720 con 0x7f09e00d1fd0 2024-03-29T11:11:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.616+0000 7f09dd600700 1 -- 172.21.15.12:0/1001010430 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f09cc013cb0 con 0x7f09e00d1fd0 2024-03-29T11:11:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.618+0000 7f09dd600700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f09c006adb0 0x7f09c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.618+0000 7f09dd600700 1 -- 172.21.15.12:0/1001010430 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f09cc089600 con 0x7f09e00d1fd0 2024-03-29T11:11:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.618+0000 7f09e5c00700 1 -- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f09c006adb0 msgr2=0x7f09c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.618+0000 7f09e5c00700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f09c006adb0 0x7f09c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:23.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.623+0000 7f09dd600700 1 -- 172.21.15.12:0/1001010430 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09cc054440 con 0x7f09e00d1fd0 2024-03-29T11:11:23.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.819+0000 7f09e5c00700 1 -- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f09c006adb0 msgr2=0x7f09c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:23.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.819+0000 7f09e5c00700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f09c006adb0 0x7f09c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:23.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.856+0000 7f09e8600700 1 -- 172.21.15.12:0/1001010430 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f09e01555d0 con 0x7f09e00d1fd0 2024-03-29T11:11:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.857+0000 7f09dd600700 1 -- 172.21.15.12:0/1001010430 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v164) v1 ==== 80+0+468 (secure 0 0 0) 0x7f09cc057a60 con 0x7f09e00d1fd0 2024-03-29T11:11:23.835 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2146007013 2024-03-30T10:15:35.915268+0000 2024-03-29T11:11:23.835 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2699841727 2024-03-30T10:15:35.915268+0000 2024-03-29T11:11:23.835 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2365815480 2024-03-30T10:15:35.915268+0000 2024-03-29T11:11:23.835 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/43444 2024-03-30T10:15:35.915268+0000 2024-03-29T11:11:23.835 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/43444 2024-03-30T10:15:35.915268+0000 2024-03-29T11:11:23.835 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3483953818 2024-03-30T10:15:35.915268+0000 2024-03-29T11:11:23.835 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1508471695 2024-03-30T10:15:35.915268+0000 2024-03-29T11:11:23.835 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3197130837 2024-03-30T10:15:35.915268+0000 2024-03-29T11:11:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 -- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f09c006adb0 msgr2=0x7f09c006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f09c006adb0 0x7f09c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 -- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1fd0 msgr2=0x7f09e01f6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1fd0 0x7f09e01f6400 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00e870 tx=0x7f09cc00eb80 comp rx=0 tx=0).stop 2024-03-29T11:11:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 -- 172.21.15.12:0/1001010430 shutdown_connections 2024-03-29T11:11:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f09c006adb0 0x7f09c006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09e015dad0 0x7f09e01f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09e00d1630 0x7f09e0154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 --2- 172.21.15.12:0/1001010430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09e00d1fd0 0x7f09e01f6400 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 -- 172.21.15.12:0/1001010430 >> 172.21.15.12:0/1001010430 conn(0x7f09e008aff0 msgr2=0x7f09e00ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 -- 172.21.15.12:0/1001010430 shutdown_connections 2024-03-29T11:11:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:23.860+0000 7f09e8600700 1 -- 172.21.15.12:0/1001010430 wait complete. 2024-03-29T11:11:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T11:11:23.857 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T11:11:23.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:23.857 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T11:11:23.887 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:23.909+0000 7f94498a8240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:11:23.933 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:11:23.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:23.933 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T11:11:23.982 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:24.004+0000 7f94498a8240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:11:23.984 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:11:23.984 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:11:24.028 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:11:24.029 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:11:24.029 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:11:24.079 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:11:24.079 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:11:24.079 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:11:24.100 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:24.122+0000 7f6fd2963240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:11:24.125 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:11:24.125 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:11:24.159 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:24.180+0000 7f94498a8240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:11:24.204 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:24.226+0000 7f6fd2963240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.351+0000 7f6e52400700 1 Processor -- start 2024-03-29T11:11:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.351+0000 7f6e52400700 1 -- start start 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e52400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6e4c1559e0 0x7f6e4c15c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e52400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6e4c0b5650 0x7f6e4c15d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e52400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c15f840 con 0x7f6e4c15f0a0 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e52400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c15cc70 con 0x7f6e4c1559e0 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e52400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c15fb20 con 0x7f6e4c0b5650 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e4b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6e4c1559e0 0x7f6e4c15c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e4aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6e4c15f0a0 0x7f6e4c0b4f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47322/0 (socket says 172.21.15.12:47322) 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e4b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6e4c1559e0 0x7f6e4c15c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58070/0 (socket says 172.21.15.12:58070) 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e4aa00700 1 -- 172.21.15.12:0/1321108894 learned_addr learned my addr 172.21.15.12:0/1321108894 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e4be00700 1 --2- 172.21.15.12:0/1321108894 >> v2:172.21.15.159:3300/0 conn(0x7f6e4c0b5650 0x7f6e4c15d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.353+0000 7f6e4a000700 1 -- 172.21.15.12:0/1321108894 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800532895 0 0) 0x7f6e4c15f840 con 0x7f6e4c15f0a0 2024-03-29T11:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.354+0000 7f6e4a000700 1 -- 172.21.15.12:0/1321108894 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e300035b0 con 0x7f6e4c15f0a0 2024-03-29T11:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.354+0000 7f6e4b400700 1 -- 172.21.15.12:0/1321108894 >> v2:172.21.15.159:3300/0 conn(0x7f6e4c0b5650 msgr2=0x7f6e4c15d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.354+0000 7f6e4b400700 1 --2- 172.21.15.12:0/1321108894 >> v2:172.21.15.159:3300/0 conn(0x7f6e4c0b5650 0x7f6e4c15d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.354+0000 7f6e4b400700 1 -- 172.21.15.12:0/1321108894 >> v1:172.21.15.12:6789/0 conn(0x7f6e4c15f0a0 legacy=0x7f6e4c0b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.354+0000 7f6e4b400700 1 -- 172.21.15.12:0/1321108894 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c15fe00 con 0x7f6e4c1559e0 2024-03-29T11:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.354+0000 7f6e4be00700 1 --2- 172.21.15.12:0/1321108894 >> v2:172.21.15.159:3300/0 conn(0x7f6e4c0b5650 0x7f6e4c15d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.354+0000 7f6e4b400700 1 --2- 172.21.15.12:0/1321108894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 0x7f6e4c15c680 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c06f1f0 tx=0x7f6e3c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f1df2ee71fe9c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.354+0000 7f6e4a000700 1 -- 172.21.15.12:0/1321108894 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c011830 con 0x7f6e4c1559e0 2024-03-29T11:11:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.354+0000 7f6e4a000700 1 -- 172.21.15.12:0/1321108894 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c011990 con 0x7f6e4c1559e0 2024-03-29T11:11:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.354+0000 7f6e4a000700 1 -- 172.21.15.12:0/1321108894 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c011d20 con 0x7f6e4c1559e0 2024-03-29T11:11:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.355+0000 7f6e52400700 1 -- 172.21.15.12:0/1321108894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 msgr2=0x7f6e4c15c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.355+0000 7f6e52400700 1 --2- 172.21.15.12:0/1321108894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 0x7f6e4c15c680 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c06f1f0 tx=0x7f6e3c009b20 comp rx=0 tx=0).stop 2024-03-29T11:11:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.355+0000 7f6e52400700 1 -- 172.21.15.12:0/1321108894 shutdown_connections 2024-03-29T11:11:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.355+0000 7f6e52400700 1 --2- 172.21.15.12:0/1321108894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 0x7f6e4c15c680 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.355+0000 7f6e52400700 1 --2- 172.21.15.12:0/1321108894 >> v2:172.21.15.159:3300/0 conn(0x7f6e4c0b5650 0x7f6e4c15d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.355+0000 7f6e52400700 1 -- 172.21.15.12:0/1321108894 >> 172.21.15.12:0/1321108894 conn(0x7f6e4c072170 msgr2=0x7f6e4c06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.355+0000 7f6e52400700 1 -- 172.21.15.12:0/1321108894 shutdown_connections 2024-03-29T11:11:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.355+0000 7f6e52400700 1 -- 172.21.15.12:0/1321108894 wait complete. 2024-03-29T11:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.357+0000 7f6e52400700 1 Processor -- start 2024-03-29T11:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.357+0000 7f6e52400700 1 -- start start 2024-03-29T11:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.358+0000 7f6e52400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e4c0b5650 0x7f6e4c14f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.358+0000 7f6e52400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 0x7f6e4c157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.358+0000 7f6e52400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e4c15f0a0 0x7f6e4c0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.358+0000 7f6e52400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0c8fb0 con 0x7f6e4c1559e0 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.358+0000 7f6e52400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c157970 con 0x7f6e4c0b5650 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.358+0000 7f6e52400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c157c50 con 0x7f6e4c15f0a0 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.358+0000 7f6e4aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 0x7f6e4c157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.358+0000 7f6e4aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 0x7f6e4c157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58084/0 (socket says 172.21.15.12:58084) 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.358+0000 7f6e4aa00700 1 -- 172.21.15.12:0/4186606949 learned_addr learned my addr 172.21.15.12:0/4186606949 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.358+0000 7f6e4b400700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e4c0b5650 0x7f6e4c14f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e4be00700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e4c15f0a0 0x7f6e4c0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e4aa00700 1 -- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e4c15f0a0 msgr2=0x7f6e4c0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e4aa00700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e4c15f0a0 0x7f6e4c0b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e4aa00700 1 -- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e4c0b5650 msgr2=0x7f6e4c14f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e4aa00700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e4c0b5650 0x7f6e4c14f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e4aa00700 1 -- 172.21.15.12:0/4186606949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e44009a70 con 0x7f6e4c1559e0 2024-03-29T11:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e4aa00700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 0x7f6e4c157070 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f6e38001eb0 tx=0x7f6e3800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e43400700 1 -- 172.21.15.12:0/4186606949 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3800cea0 con 0x7f6e4c1559e0 2024-03-29T11:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e43400700 1 -- 172.21.15.12:0/4186606949 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e38005d70 con 0x7f6e4c1559e0 2024-03-29T11:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e43400700 1 -- 172.21.15.12:0/4186606949 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e380072f0 con 0x7f6e4c1559e0 2024-03-29T11:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.359+0000 7f6e52400700 1 -- 172.21.15.12:0/4186606949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3c0094a0 con 0x7f6e4c1559e0 2024-03-29T11:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.360+0000 7f6e52400700 1 -- 172.21.15.12:0/4186606949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c0c94e0 con 0x7f6e4c1559e0 2024-03-29T11:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.360+0000 7f6e52400700 1 -- 172.21.15.12:0/4186606949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c157e10 con 0x7f6e4c1559e0 2024-03-29T11:11:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.362+0000 7f6e43400700 1 -- 172.21.15.12:0/4186606949 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f6e38007450 con 0x7f6e4c1559e0 2024-03-29T11:11:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.363+0000 7f6e43400700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6e2c06adb0 0x7f6e2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.363+0000 7f6e43400700 1 -- 172.21.15.12:0/4186606949 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f6e38088f00 con 0x7f6e4c1559e0 2024-03-29T11:11:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.363+0000 7f6e4b400700 1 -- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6e2c06adb0 msgr2=0x7f6e2c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.363+0000 7f6e4b400700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6e2c06adb0 0x7f6e2c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:24.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.371+0000 7f6e43400700 1 -- 172.21.15.12:0/4186606949 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e38053dc0 con 0x7f6e4c1559e0 2024-03-29T11:11:24.375 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:24.397+0000 7f94498a8240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:11:24.381 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:24.403+0000 7f6fd2963240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:11:24.468 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:24.489+0000 7f94498a8240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:11:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.564+0000 7f6e4b400700 1 -- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6e2c06adb0 msgr2=0x7f6e2c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.564+0000 7f6e4b400700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6e2c06adb0 0x7f6e2c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:24.568 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:24.589+0000 7f94498a8240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:11:24.603 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:24.625+0000 7f6fd2963240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:11:24.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.632+0000 7f6e52400700 1 -- 172.21.15.12:0/4186606949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e4c0c9700 con 0x7f6e4c1559e0 2024-03-29T11:11:24.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.632+0000 7f6e43400700 1 -- 172.21.15.12:0/4186606949 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f6e380573e0 con 0x7f6e4c1559e0 2024-03-29T11:11:24.611 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:24.611 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3149931506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3149931506},{"type":"v1","addr":"172.21.15.159:6835","nonce":3149931506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/967685480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":967685480},{"type":"v1","addr":"172.21.15.159:6837","nonce":967685480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8967,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4273394318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4273394318},{"type":"v1","addr":"172.21.15.12:6834","nonce":4273394318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:15:18.742186+0000","modified":"2024-03-29T10:15:22.226050+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8970},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8970":{"gid":8970,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/512120304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":512120304},{"type":"v1","addr":"172.21.15.12:6835","nonce":512120304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-29T11:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 -- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6e2c06adb0 msgr2=0x7f6e2c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6e2c06adb0 0x7f6e2c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 -- 172.21.15.12:0/4186606949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 msgr2=0x7f6e4c157070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 0x7f6e4c157070 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f6e38001eb0 tx=0x7f6e3800c7b0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 -- 172.21.15.12:0/4186606949 shutdown_connections 2024-03-29T11:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6e2c06adb0 0x7f6e2c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e4c15f0a0 0x7f6e4c0b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e4c0b5650 0x7f6e4c14f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 --2- 172.21.15.12:0/4186606949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c1559e0 0x7f6e4c157070 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 -- 172.21.15.12:0/4186606949 >> 172.21.15.12:0/4186606949 conn(0x7f6e4c072170 msgr2=0x7f6e4c14ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 -- 172.21.15.12:0/4186606949 shutdown_connections 2024-03-29T11:11:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.635+0000 7f6e52400700 1 -- 172.21.15.12:0/4186606949 wait complete. 2024-03-29T11:11:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 73 2024-03-29T11:11:24.632 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:24.667 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:24.689+0000 7f94498a8240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:11:24.705 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:24.727+0000 7f6fd2963240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:11:24.766 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:24.787+0000 7f94498a8240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:11:24.812 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:24.834+0000 7f6fd2963240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:11:24.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.855+0000 7ffaa1e00700 1 Processor -- start 2024-03-29T11:11:24.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.856+0000 7ffaa1e00700 1 -- start start 2024-03-29T11:11:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.857+0000 7ffaa1e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ffa9c15da50 0x7ffa9c0ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.857+0000 7ffaa1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.857+0000 7ffaa1e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c0d24e0 con 0x7ffa9c0cec70 2024-03-29T11:11:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.857+0000 7ffaa1e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa9c0b95d0 con 0x7ffa9c15da50 2024-03-29T11:11:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.857+0000 7ffaa1e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c15de30 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa9a000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ffa9c0cec70 0x7ffa9c0d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47334/0 (socket says 172.21.15.12:47334) 2024-03-29T11:11:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa9a000700 1 -- 172.21.15.12:0/403980823 learned_addr learned my addr 172.21.15.12:0/403980823 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa9b400700 1 --2- 172.21.15.12:0/403980823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa9aa00700 1 --2- 172.21.15.12:0/403980823 >> v2:172.21.15.159:3300/0 conn(0x7ffa9c15da50 0x7ffa9c0ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa99600700 1 -- 172.21.15.12:0/403980823 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402451376 0 0) 0x7ffa9c0d24e0 con 0x7ffa9c0cec70 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa99600700 1 -- 172.21.15.12:0/403980823 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa800035b0 con 0x7ffa9c0cec70 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa9aa00700 1 -- 172.21.15.12:0/403980823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0d0cf0 msgr2=0x7ffa9c0b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa9aa00700 1 --2- 172.21.15.12:0/403980823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa9aa00700 1 -- 172.21.15.12:0/403980823 >> v1:172.21.15.12:6789/0 conn(0x7ffa9c0cec70 legacy=0x7ffa9c0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa9aa00700 1 -- 172.21.15.12:0/403980823 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c0d27c0 con 0x7ffa9c15da50 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.858+0000 7ffa9b400700 1 --2- 172.21.15.12:0/403980823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.859+0000 7ffa9aa00700 1 --2- 172.21.15.12:0/403980823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa9c15da50 0x7ffa9c0ce730 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009840 tx=0x7ffa8c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d538599bf828264b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.859+0000 7ffa99600700 1 -- 172.21.15.12:0/403980823 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c011830 con 0x7ffa9c15da50 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.859+0000 7ffa99600700 1 -- 172.21.15.12:0/403980823 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8c011990 con 0x7ffa9c15da50 2024-03-29T11:11:24.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.859+0000 7ffa99600700 1 -- 172.21.15.12:0/403980823 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c011d20 con 0x7ffa9c15da50 2024-03-29T11:11:24.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.860+0000 7ffaa1e00700 1 -- 172.21.15.12:0/403980823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa9c15da50 msgr2=0x7ffa9c0ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:24.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.860+0000 7ffaa1e00700 1 --2- 172.21.15.12:0/403980823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa9c15da50 0x7ffa9c0ce730 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009840 tx=0x7ffa8c009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:24.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.860+0000 7ffaa1e00700 1 -- 172.21.15.12:0/403980823 shutdown_connections 2024-03-29T11:11:24.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.860+0000 7ffaa1e00700 1 --2- 172.21.15.12:0/403980823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.860+0000 7ffaa1e00700 1 --2- 172.21.15.12:0/403980823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa9c15da50 0x7ffa9c0ce730 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.860+0000 7ffaa1e00700 1 -- 172.21.15.12:0/403980823 >> 172.21.15.12:0/403980823 conn(0x7ffa9c0666a0 msgr2=0x7ffa9c066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:24.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.860+0000 7ffaa1e00700 1 -- 172.21.15.12:0/403980823 shutdown_connections 2024-03-29T11:11:24.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.860+0000 7ffaa1e00700 1 -- 172.21.15.12:0/403980823 wait complete. 2024-03-29T11:11:24.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.861+0000 7ffaa1e00700 1 Processor -- start 2024-03-29T11:11:24.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.862+0000 7ffaa1e00700 1 -- start start 2024-03-29T11:11:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.862+0000 7ffaa1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0cec70 0x7ffa9c151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.862+0000 7ffaa1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffaa1e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa9c15da50 0x7ffa9c15f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffaa1e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c0cc480 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffaa1e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c0cc930 con 0x7ffa9c15da50 2024-03-29T11:11:24.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffaa1e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c0cd6b0 con 0x7ffa9c0cec70 2024-03-29T11:11:24.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:24.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58100/0 (socket says 172.21.15.12:58100) 2024-03-29T11:11:24.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9a000700 1 -- 172.21.15.12:0/885448412 learned_addr learned my addr 172.21.15.12:0/885448412 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:24.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9b400700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa9c15da50 0x7ffa9c15f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:24.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9aa00700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0cec70 0x7ffa9c151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:24.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9a000700 1 -- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0cec70 msgr2=0x7ffa9c151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:24.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9a000700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0cec70 0x7ffa9c151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9a000700 1 -- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa9c15da50 msgr2=0x7ffa9c15f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9a000700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa9c15da50 0x7ffa9c15f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9a000700 1 -- 172.21.15.12:0/885448412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa94009a70 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.863+0000 7ffa9a000700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0cd170 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7ffa88005910 tx=0x7ffa8800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.864+0000 7ffa93400700 1 -- 172.21.15.12:0/885448412 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88005d70 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.864+0000 7ffa93400700 1 -- 172.21.15.12:0/885448412 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa880070d0 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.864+0000 7ffaa1e00700 1 -- 172.21.15.12:0/885448412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8c0094a0 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.864+0000 7ffa93400700 1 -- 172.21.15.12:0/885448412 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88007460 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.864+0000 7ffaa1e00700 1 -- 172.21.15.12:0/885448412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c0cb6c0 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.865+0000 7ffaa1e00700 1 -- 172.21.15.12:0/885448412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9c067720 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.866+0000 7ffa93400700 1 -- 172.21.15.12:0/885448412 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7ffa880075c0 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.867+0000 7ffa93400700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffa7c06adb0 0x7ffa7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.867+0000 7ffa93400700 1 -- 172.21.15.12:0/885448412 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7ffa88088b30 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.867+0000 7ffa9aa00700 1 -- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffa7c06adb0 msgr2=0x7ffa7c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:24.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.867+0000 7ffa9aa00700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffa7c06adb0 0x7ffa7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:24.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:24.872+0000 7ffa93400700 1 -- 172.21.15.12:0/885448412 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa88053a20 con 0x7ffa9c0d0cf0 2024-03-29T11:11:24.920 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:24.942+0000 7f6fd2963240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:11:24.979 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:25.001+0000 7f94498a8240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:11:25.024 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:25.046+0000 7f6fd2963240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:11:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.068+0000 7ffa9aa00700 1 -- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffa7c06adb0 msgr2=0x7ffa7c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.068+0000 7ffa9aa00700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffa7c06adb0 0x7ffa7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:25.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.086+0000 7ffaa1e00700 1 -- 172.21.15.12:0/885448412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa9c0b61c0 con 0x7ffa9c0d0cf0 2024-03-29T11:11:25.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.087+0000 7ffa93400700 1 -- 172.21.15.12:0/885448412 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+16816 (secure 0 0 0) 0x7ffa88057040 con 0x7ffa9c0d0cf0 2024-03-29T11:11:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:25.066 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":164,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:22.039712+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:15:17.110269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-29T10:15:17.717343+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.9","mappings":[{"from":2,"to":0}]},{"pgid":"19.23","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2146007013":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/2699841727":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/2365815480":"2024-03-30T10:15:35.915268+0000","172.21.15.159:6833/43444":"2024-03-30T10:15:35.915268+0000","172.21.15.159:6832/43444":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/3483953818":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/1508471695":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/3197130837":"2024-03-30T10:15:35.915268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":54,"length":3},{"begin":110,"length":3}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 -- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffa7c06adb0 msgr2=0x7ffa7c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffa7c06adb0 0x7ffa7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 -- 172.21.15.12:0/885448412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa9c0d0cf0 msgr2=0x7ffa9c0cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0cd170 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7ffa88005910 tx=0x7ffa8800c9e0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 -- 172.21.15.12:0/885448412 shutdown_connections 2024-03-29T11:11:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ffa7c06adb0 0x7ffa7c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa9c15da50 0x7ffa9c15f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa9c0cec70 0x7ffa9c151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 --2- 172.21.15.12:0/885448412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa9c0d0cf0 0x7ffa9c0cd170 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 -- 172.21.15.12:0/885448412 >> 172.21.15.12:0/885448412 conn(0x7ffa9c0666a0 msgr2=0x7ffa9c0b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 -- 172.21.15.12:0/885448412 shutdown_connections 2024-03-29T11:11:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.090+0000 7ffaa1e00700 1 -- 172.21.15.12:0/885448412 wait complete. 2024-03-29T11:11:25.087 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T10:15:17.110269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T10:15:17.717343+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 164, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:11:25.087 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:11:25.240 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:25.262+0000 7f94498a8240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:11:25.258 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:25.280+0000 7f6fd2963240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:11:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.325+0000 7f0426200700 1 Processor -- start 2024-03-29T11:11:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.325+0000 7f0426200700 1 -- start start 2024-03-29T11:11:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.326+0000 7f0426200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f04200cd020 0x7f04200cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.326+0000 7f0426200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04200b2070 0x7f04200b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.326+0000 7f0426200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04200cdae0 con 0x7f04200cbb30 2024-03-29T11:11:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.326+0000 7f0426200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04200cddc0 con 0x7f04200cd020 2024-03-29T11:11:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.326+0000 7f0426200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04200ce0a0 con 0x7f04200b2070 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.326+0000 7f041ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f04200cd020 0x7f04200cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.326+0000 7f041ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f04200cd020 0x7f04200cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58116/0 (socket says 172.21.15.12:58116) 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.326+0000 7f041ea00700 1 -- 172.21.15.12:0/2438223691 learned_addr learned my addr 172.21.15.12:0/2438223691 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.326+0000 7f041fe00700 1 --2- 172.21.15.12:0/2438223691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04200b2070 0x7f04200b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.327+0000 7f041ea00700 1 -- 172.21.15.12:0/2438223691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04200b2070 msgr2=0x7f04200b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.327+0000 7f041ea00700 1 --2- 172.21.15.12:0/2438223691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04200b2070 0x7f04200b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.327+0000 7f041ea00700 1 -- 172.21.15.12:0/2438223691 >> v1:172.21.15.159:6789/0 conn(0x7f04200cbb30 legacy=0x7f04200cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.327+0000 7f041ea00700 1 -- 172.21.15.12:0/2438223691 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042006dc20 con 0x7f04200cd020 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.327+0000 7f041ea00700 1 --2- 172.21.15.12:0/2438223691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cd020 0x7f04200cd480 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f040c0098f0 tx=0x7f040c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4affa894812c0bf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.327+0000 7f041e000700 1 -- 172.21.15.12:0/2438223691 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c011830 con 0x7f04200cd020 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.327+0000 7f041e000700 1 -- 172.21.15.12:0/2438223691 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c011990 con 0x7f04200cd020 2024-03-29T11:11:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.327+0000 7f041e000700 1 -- 172.21.15.12:0/2438223691 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c011d20 con 0x7f04200cd020 2024-03-29T11:11:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.328+0000 7f0426200700 1 -- 172.21.15.12:0/2438223691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cd020 msgr2=0x7f04200cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.328+0000 7f0426200700 1 --2- 172.21.15.12:0/2438223691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cd020 0x7f04200cd480 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f040c0098f0 tx=0x7f040c009c00 comp rx=0 tx=0).stop 2024-03-29T11:11:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.328+0000 7f0426200700 1 -- 172.21.15.12:0/2438223691 shutdown_connections 2024-03-29T11:11:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.328+0000 7f0426200700 1 --2- 172.21.15.12:0/2438223691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04200b2070 0x7f04200b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.328+0000 7f0426200700 1 --2- 172.21.15.12:0/2438223691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cd020 0x7f04200cd480 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.328+0000 7f0426200700 1 -- 172.21.15.12:0/2438223691 >> 172.21.15.12:0/2438223691 conn(0x7f04200b0d70 msgr2=0x7f04200b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.328+0000 7f0426200700 1 -- 172.21.15.12:0/2438223691 shutdown_connections 2024-03-29T11:11:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.328+0000 7f0426200700 1 -- 172.21.15.12:0/2438223691 wait complete. 2024-03-29T11:11:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.330+0000 7f0426200700 1 Processor -- start 2024-03-29T11:11:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.330+0000 7f0426200700 1 -- start start 2024-03-29T11:11:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.331+0000 7f0426200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04200b2070 0x7f04201f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.331+0000 7f0426200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cbb30 0x7f042009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.331+0000 7f0426200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04200cd020 0x7f04200724e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.331+0000 7f0426200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0420072a20 con 0x7f04200cbb30 2024-03-29T11:11:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.331+0000 7f0426200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04200cb360 con 0x7f04200cd020 2024-03-29T11:11:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.331+0000 7f0426200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04200caac0 con 0x7f04200b2070 2024-03-29T11:11:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.331+0000 7f041ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cbb30 0x7f042009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.331+0000 7f041f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04200b2070 0x7f04201f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.332+0000 7f041ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cbb30 0x7f042009cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58126/0 (socket says 172.21.15.12:58126) 2024-03-29T11:11:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.332+0000 7f041fe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04200cd020 0x7f04200724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.332+0000 7f041ea00700 1 -- 172.21.15.12:0/713782523 learned_addr learned my addr 172.21.15.12:0/713782523 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.332+0000 7f041ea00700 1 -- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04200b2070 msgr2=0x7f04201f3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.332+0000 7f041ea00700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04200b2070 0x7f04201f3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.332+0000 7f041ea00700 1 -- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04200cd020 msgr2=0x7f04200724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.332+0000 7f041ea00700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04200cd020 0x7f04200724e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.332+0000 7f041ea00700 1 -- 172.21.15.12:0/713782523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040c009550 con 0x7f04200cbb30 2024-03-29T11:11:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.332+0000 7f041ea00700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cbb30 0x7f042009cb60 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f040c000c00 tx=0x7f040c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.332+0000 7f0417400700 1 -- 172.21.15.12:0/713782523 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c004400 con 0x7f04200cbb30 2024-03-29T11:11:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.333+0000 7f0417400700 1 -- 172.21.15.12:0/713782523 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c01f070 con 0x7f04200cbb30 2024-03-29T11:11:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.333+0000 7f0426200700 1 -- 172.21.15.12:0/713782523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04200cada0 con 0x7f04200cbb30 2024-03-29T11:11:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.333+0000 7f0417400700 1 -- 172.21.15.12:0/713782523 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c0038e0 con 0x7f04200cbb30 2024-03-29T11:11:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.333+0000 7f0426200700 1 -- 172.21.15.12:0/713782523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04200ac1f0 con 0x7f04200cbb30 2024-03-29T11:11:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.334+0000 7f0426200700 1 -- 172.21.15.12:0/713782523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04200ac050 con 0x7f04200cbb30 2024-03-29T11:11:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.335+0000 7f0417400700 1 -- 172.21.15.12:0/713782523 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f040c021030 con 0x7f04200cbb30 2024-03-29T11:11:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.336+0000 7f0417400700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f040006adb0 0x7f040006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.336+0000 7f0417400700 1 -- 172.21.15.12:0/713782523 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f040c08a950 con 0x7f04200cbb30 2024-03-29T11:11:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.336+0000 7f041f400700 1 -- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f040006adb0 msgr2=0x7f040006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.336+0000 7f041f400700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f040006adb0 0x7f040006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:25.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.340+0000 7f0417400700 1 -- 172.21.15.12:0/713782523 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f04200ac050 con 0x7f04200cbb30 2024-03-29T11:11:25.432 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:25.454+0000 7f94498a8240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:11:25.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.537+0000 7f041f400700 1 -- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f040006adb0 msgr2=0x7f040006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:25.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.537+0000 7f041f400700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f040006adb0 0x7f040006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:25.525 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:25.546+0000 7f94498a8240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:25.536 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:25.557+0000 7f6fd2963240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:11:25.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.558+0000 7f0426200700 1 -- 172.21.15.12:0/713782523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f04200ac050 con 0x7f04200cbb30 2024-03-29T11:11:25.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.558+0000 7f0417400700 1 -- 172.21.15.12:0/713782523 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) v1 ==== 93+0+10 (secure 0 0 0) 0x7f04200ac050 con 0x7f04200cbb30 2024-03-29T11:11:25.536 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.562+0000 7f0426200700 1 -- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f040006adb0 msgr2=0x7f040006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.562+0000 7f0426200700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f040006adb0 0x7f040006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.562+0000 7f0426200700 1 -- 172.21.15.12:0/713782523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cbb30 msgr2=0x7f042009cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.562+0000 7f0426200700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cbb30 0x7f042009cb60 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f040c000c00 tx=0x7f040c004750 comp rx=0 tx=0).stop 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.562+0000 7f0426200700 1 -- 172.21.15.12:0/713782523 shutdown_connections 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.562+0000 7f0426200700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f040006adb0 0x7f040006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.562+0000 7f0426200700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04200cd020 0x7f04200724e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.562+0000 7f0426200700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04200b2070 0x7f04201f3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.562+0000 7f0426200700 1 --2- 172.21.15.12:0/713782523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04200cbb30 0x7f042009cb60 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.562+0000 7f0426200700 1 -- 172.21.15.12:0/713782523 >> 172.21.15.12:0/713782523 conn(0x7f04200b0d70 msgr2=0x7f04200cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.563+0000 7f0426200700 1 -- 172.21.15.12:0/713782523 shutdown_connections 2024-03-29T11:11:25.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.563+0000 7f0426200700 1 -- 172.21.15.12:0/713782523 wait complete. 2024-03-29T11:11:25.569 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:11:25.753 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:25.775+0000 7f6fd2963240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:11:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.794+0000 7f2074e00700 1 Processor -- start 2024-03-29T11:11:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.794+0000 7f2074e00700 1 -- start start 2024-03-29T11:11:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.795+0000 7f2074e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f20701611b0 0x7f207015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.795+0000 7f2074e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f207015f4d0 0x7f20700ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.795+0000 7f2074e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20700ce270 con 0x7f20700d21f0 2024-03-29T11:11:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.795+0000 7f2074e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f207015b600 con 0x7f20701611b0 2024-03-29T11:11:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.795+0000 7f2074e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700d2760 con 0x7f207015f4d0 2024-03-29T11:11:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.795+0000 7f206de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f20701611b0 0x7f207015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.796+0000 7f206de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f20701611b0 0x7f207015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58128/0 (socket says 172.21.15.12:58128) 2024-03-29T11:11:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.796+0000 7f206de00700 1 -- 172.21.15.12:0/243226824 learned_addr learned my addr 172.21.15.12:0/243226824 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.796+0000 7f206d400700 1 --2- 172.21.15.12:0/243226824 >> v2:172.21.15.159:3300/0 conn(0x7f207015f4d0 0x7f20700ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.796+0000 7f206de00700 1 -- 172.21.15.12:0/243226824 >> v2:172.21.15.159:3300/0 conn(0x7f207015f4d0 msgr2=0x7f20700ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:25.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.796+0000 7f206de00700 1 --2- 172.21.15.12:0/243226824 >> v2:172.21.15.159:3300/0 conn(0x7f207015f4d0 0x7f20700ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.796+0000 7f206de00700 1 -- 172.21.15.12:0/243226824 >> v1:172.21.15.159:6789/0 conn(0x7f20700d21f0 legacy=0x7f2070161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:25.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.796+0000 7f206de00700 1 -- 172.21.15.12:0/243226824 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20700d2a40 con 0x7f20701611b0 2024-03-29T11:11:25.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.796+0000 7f206de00700 1 --2- 172.21.15.12:0/243226824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20701611b0 0x7f207015ef90 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f207008c030 tx=0x7f2060009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676ba2b7f962e384 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:25.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.796+0000 7f206ca00700 1 -- 172.21.15.12:0/243226824 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2060011830 con 0x7f20701611b0 2024-03-29T11:11:25.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.797+0000 7f206ca00700 1 -- 172.21.15.12:0/243226824 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2060011990 con 0x7f20701611b0 2024-03-29T11:11:25.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.797+0000 7f206ca00700 1 -- 172.21.15.12:0/243226824 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2060011d20 con 0x7f20701611b0 2024-03-29T11:11:25.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.797+0000 7f2074e00700 1 -- 172.21.15.12:0/243226824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20701611b0 msgr2=0x7f207015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:25.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.797+0000 7f2074e00700 1 --2- 172.21.15.12:0/243226824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20701611b0 0x7f207015ef90 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f207008c030 tx=0x7f2060009bd0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.797+0000 7f2074e00700 1 -- 172.21.15.12:0/243226824 shutdown_connections 2024-03-29T11:11:25.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.797+0000 7f2074e00700 1 --2- 172.21.15.12:0/243226824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20701611b0 0x7f207015ef90 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.797+0000 7f2074e00700 1 --2- 172.21.15.12:0/243226824 >> v2:172.21.15.159:3300/0 conn(0x7f207015f4d0 0x7f20700ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.797+0000 7f2074e00700 1 -- 172.21.15.12:0/243226824 >> 172.21.15.12:0/243226824 conn(0x7f207008aff0 msgr2=0x7f20700d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:25.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.797+0000 7f2074e00700 1 -- 172.21.15.12:0/243226824 shutdown_connections 2024-03-29T11:11:25.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.797+0000 7f2074e00700 1 -- 172.21.15.12:0/243226824 wait complete. 2024-03-29T11:11:25.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.800+0000 7f2074e00700 1 Processor -- start 2024-03-29T11:11:25.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.800+0000 7f2074e00700 1 -- start start 2024-03-29T11:11:25.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.801+0000 7f2074e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20700d21f0 0x7f207014d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.801+0000 7f2074e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f207015f4d0 0x7f20700b8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.801+0000 7f2074e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20701611b0 0x7f20700c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.801+0000 7f2074e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207014df20 con 0x7f207015f4d0 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.801+0000 7f2074e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20700cabc0 con 0x7f20700d21f0 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.801+0000 7f2074e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20700b8ed0 con 0x7f20701611b0 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.801+0000 7f206d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f207015f4d0 0x7f20700b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.801+0000 7f206d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f207015f4d0 0x7f20700b8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58138/0 (socket says 172.21.15.12:58138) 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.801+0000 7f206d400700 1 -- 172.21.15.12:0/832686841 learned_addr learned my addr 172.21.15.12:0/832686841 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.801+0000 7f206de00700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20700d21f0 0x7f207014d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f206e800700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20701611b0 0x7f20700c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f206d400700 1 -- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20701611b0 msgr2=0x7f20700c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f206d400700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20701611b0 0x7f20700c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f206d400700 1 -- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20700d21f0 msgr2=0x7f207014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f206d400700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20700d21f0 0x7f207014d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f206d400700 1 -- 172.21.15.12:0/832686841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2060009550 con 0x7f207015f4d0 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f206d400700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f207015f4d0 0x7f20700b8080 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f205c00e9b0 tx=0x7f205c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f2066000700 1 -- 172.21.15.12:0/832686841 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c00c7f0 con 0x7f207015f4d0 2024-03-29T11:11:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f2066000700 1 -- 172.21.15.12:0/832686841 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205c00c950 con 0x7f207015f4d0 2024-03-29T11:11:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f2074e00700 1 -- 172.21.15.12:0/832686841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20700b85c0 con 0x7f207015f4d0 2024-03-29T11:11:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f2066000700 1 -- 172.21.15.12:0/832686841 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c00ccc0 con 0x7f207015f4d0 2024-03-29T11:11:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.802+0000 7f2074e00700 1 -- 172.21.15.12:0/832686841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20700c7f90 con 0x7f207015f4d0 2024-03-29T11:11:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.804+0000 7f2074e00700 1 -- 172.21.15.12:0/832686841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2070067720 con 0x7f207015f4d0 2024-03-29T11:11:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.804+0000 7f2066000700 1 -- 172.21.15.12:0/832686841 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f205c013470 con 0x7f207015f4d0 2024-03-29T11:11:25.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.806+0000 7f2066000700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f205006ade0 0x7f205006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:25.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.806+0000 7f2066000700 1 -- 172.21.15.12:0/832686841 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f205c089680 con 0x7f207015f4d0 2024-03-29T11:11:25.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.806+0000 7f206de00700 1 -- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f205006ade0 msgr2=0x7f205006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:25.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.806+0000 7f206de00700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f205006ade0 0x7f205006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:25.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:25.810+0000 7f2066000700 1 -- 172.21.15.12:0/832686841 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f205c054550 con 0x7f207015f4d0 2024-03-29T11:11:25.862 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:25.884+0000 7f6fd2963240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:25.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.007+0000 7f206de00700 1 -- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f205006ade0 msgr2=0x7f205006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:25.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.007+0000 7f206de00700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f205006ade0 0x7f205006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.034+0000 7f2074e00700 1 -- 172.21.15.12:0/832686841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f20700b5d00 con 0x7f207015f4d0 2024-03-29T11:11:26.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.034+0000 7f2066000700 1 -- 172.21.15.12:0/832686841 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v164) v1 ==== 104+0+11 (secure 0 0 0) 0x7f205c057b70 con 0x7f207015f4d0 2024-03-29T11:11:26.012 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:11:26.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.036+0000 7f2074e00700 1 -- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f205006ade0 msgr2=0x7f205006d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:26.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.036+0000 7f2074e00700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f205006ade0 0x7f205006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.036+0000 7f2074e00700 1 -- 172.21.15.12:0/832686841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f207015f4d0 msgr2=0x7f20700b8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.037+0000 7f2074e00700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f207015f4d0 0x7f20700b8080 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f205c00e9b0 tx=0x7f205c00ecc0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.037+0000 7f2074e00700 1 -- 172.21.15.12:0/832686841 shutdown_connections 2024-03-29T11:11:26.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.037+0000 7f2074e00700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f205006ade0 0x7f205006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.037+0000 7f2074e00700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20701611b0 0x7f20700c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.037+0000 7f2074e00700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20700d21f0 0x7f207014d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.037+0000 7f2074e00700 1 --2- 172.21.15.12:0/832686841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f207015f4d0 0x7f20700b8080 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.037+0000 7f2074e00700 1 -- 172.21.15.12:0/832686841 >> 172.21.15.12:0/832686841 conn(0x7f207008aff0 msgr2=0x7f20700ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:26.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.037+0000 7f2074e00700 1 -- 172.21.15.12:0/832686841 shutdown_connections 2024-03-29T11:11:26.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.037+0000 7f2074e00700 1 -- 172.21.15.12:0/832686841 wait complete. 2024-03-29T11:11:26.034 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:11:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.259+0000 7fed1e800700 1 Processor -- start 2024-03-29T11:11:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.260+0000 7fed1e800700 1 -- start start 2024-03-29T11:11:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.262+0000 7fed1e800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed18160d70 0x7fed180b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.262+0000 7fed1e800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fed1806a790 0x7fed1806ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.262+0000 7fed1e800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed18161150 con 0x7fed1815ef70 2024-03-29T11:11:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.262+0000 7fed1e800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed1806b180 con 0x7fed1806a790 2024-03-29T11:11:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.262+0000 7fed1e800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed180b4cd0 con 0x7fed18160d70 2024-03-29T11:11:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.262+0000 7fed17e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fed1806a790 0x7fed1806ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.262+0000 7fed17400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed18160d70 0x7fed180b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.262+0000 7fed17e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fed1806a790 0x7fed1806ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44844/0 (socket says 172.21.15.12:44844) 2024-03-29T11:11:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.262+0000 7fed17e00700 1 -- 172.21.15.12:0/401842076 learned_addr learned my addr 172.21.15.12:0/401842076 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.263+0000 7fed17e00700 1 -- 172.21.15.12:0/401842076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed18160d70 msgr2=0x7fed180b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.263+0000 7fed17e00700 1 --2- 172.21.15.12:0/401842076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed18160d70 0x7fed180b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.263+0000 7fed17e00700 1 -- 172.21.15.12:0/401842076 >> v1:172.21.15.159:6789/0 conn(0x7fed1815ef70 legacy=0x7fed1815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.263+0000 7fed17e00700 1 -- 172.21.15.12:0/401842076 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1806dba0 con 0x7fed1806a790 2024-03-29T11:11:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.263+0000 7fed17e00700 1 --2- 172.21.15.12:0/401842076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1806a790 0x7fed1806ac40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fed10009e50 tx=0x7fed1000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba9f368b6af1f371 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.263+0000 7fed16000700 1 -- 172.21.15.12:0/401842076 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000e070 con 0x7fed1806a790 2024-03-29T11:11:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.264+0000 7fed16000700 1 -- 172.21.15.12:0/401842076 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1000b910 con 0x7fed1806a790 2024-03-29T11:11:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.264+0000 7fed16000700 1 -- 172.21.15.12:0/401842076 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000bcc0 con 0x7fed1806a790 2024-03-29T11:11:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.264+0000 7fed1e800700 1 -- 172.21.15.12:0/401842076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1806a790 msgr2=0x7fed1806ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.264+0000 7fed1e800700 1 --2- 172.21.15.12:0/401842076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1806a790 0x7fed1806ac40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fed10009e50 tx=0x7fed1000b330 comp rx=0 tx=0).stop 2024-03-29T11:11:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.264+0000 7fed1e800700 1 -- 172.21.15.12:0/401842076 shutdown_connections 2024-03-29T11:11:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.264+0000 7fed1e800700 1 --2- 172.21.15.12:0/401842076 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1806a790 0x7fed1806ac40 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.264+0000 7fed1e800700 1 --2- 172.21.15.12:0/401842076 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed18160d70 0x7fed180b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.264+0000 7fed1e800700 1 -- 172.21.15.12:0/401842076 >> 172.21.15.12:0/401842076 conn(0x7fed180b5720 msgr2=0x7fed180b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.264+0000 7fed1e800700 1 -- 172.21.15.12:0/401842076 shutdown_connections 2024-03-29T11:11:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.264+0000 7fed1e800700 1 -- 172.21.15.12:0/401842076 wait complete. 2024-03-29T11:11:26.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.266+0000 7fed1e800700 1 Processor -- start 2024-03-29T11:11:26.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.266+0000 7fed1e800700 1 -- start start 2024-03-29T11:11:26.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed1e800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed1806a790 0x7fed1809cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed1e800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1815ef70 0x7fed180aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed1e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18160d70 0x7fed180af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed1e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed181333f0 con 0x7fed18160d70 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed1e800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed18132770 con 0x7fed1815ef70 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed1e800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed18132a50 con 0x7fed1806a790 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed17400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed1806a790 0x7fed1809cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed17e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18160d70 0x7fed180af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed17400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed1806a790 0x7fed1809cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:37800/0 (socket says 172.21.15.12:37800) 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed17400700 1 -- 172.21.15.12:0/3809354882 learned_addr learned my addr 172.21.15.12:0/3809354882 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.267+0000 7fed16a00700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1815ef70 0x7fed180aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.268+0000 7fed17400700 1 -- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1815ef70 msgr2=0x7fed180aeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.268+0000 7fed17400700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1815ef70 0x7fed180aeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.268+0000 7fed17400700 1 -- 172.21.15.12:0/3809354882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18160d70 msgr2=0x7fed180af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.268+0000 7fed17400700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18160d70 0x7fed180af410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.268+0000 7fed17400700 1 -- 172.21.15.12:0/3809354882 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10009b30 con 0x7fed1806a790 2024-03-29T11:11:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.268+0000 7fed17e00700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18160d70 0x7fed180af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.268+0000 7fed17400700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed1806a790 0x7fed1809cd00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fed0800e870 tx=0x7fed0800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.269+0000 7fed0f400700 1 -- 172.21.15.12:0/3809354882 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08009590 con 0x7fed1806a790 2024-03-29T11:11:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.269+0000 7fed0f400700 1 -- 172.21.15.12:0/3809354882 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed08013680 con 0x7fed1806a790 2024-03-29T11:11:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.269+0000 7fed0f400700 1 -- 172.21.15.12:0/3809354882 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed080139f0 con 0x7fed1806a790 2024-03-29T11:11:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.269+0000 7fed1e800700 1 -- 172.21.15.12:0/3809354882 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed18132d30 con 0x7fed1806a790 2024-03-29T11:11:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.269+0000 7fed1e800700 1 -- 172.21.15.12:0/3809354882 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1809d240 con 0x7fed1806a790 2024-03-29T11:11:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.270+0000 7fed1e800700 1 -- 172.21.15.12:0/3809354882 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed18133660 con 0x7fed1806a790 2024-03-29T11:11:26.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.271+0000 7fed0f400700 1 -- 172.21.15.12:0/3809354882 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fed08013b50 con 0x7fed1806a790 2024-03-29T11:11:26.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.271+0000 7fed0f400700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf806adb0 0x7fecf806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.272+0000 7fed0f400700 1 -- 172.21.15.12:0/3809354882 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fed08089590 con 0x7fed1806a790 2024-03-29T11:11:26.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.272+0000 7fed16a00700 1 -- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf806adb0 msgr2=0x7fecf806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:26.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.272+0000 7fed16a00700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf806adb0 0x7fecf806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:26.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.276+0000 7fed0f400700 1 -- 172.21.15.12:0/3809354882 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fed08054460 con 0x7fed1806a790 2024-03-29T11:11:26.261 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:26.283+0000 7f94498a8240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:11:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.473+0000 7fed16a00700 1 -- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf806adb0 msgr2=0x7fecf806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.473+0000 7fed16a00700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf806adb0 0x7fecf806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:26.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.495+0000 7fed1e800700 1 -- 172.21.15.12:0/3809354882 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed18132b90 con 0x7fed1806a790 2024-03-29T11:11:26.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.496+0000 7fed0f400700 1 -- 172.21.15.12:0/3809354882 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v164) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed18132b90 con 0x7fed1806a790 2024-03-29T11:11:26.474 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:11:26.475 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:26.497+0000 7f94498a8240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.498+0000 7fed1e800700 1 -- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf806adb0 msgr2=0x7fecf806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.498+0000 7fed1e800700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf806adb0 0x7fecf806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.498+0000 7fed1e800700 1 -- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed1806a790 msgr2=0x7fed1809cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.498+0000 7fed1e800700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed1806a790 0x7fed1809cd00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fed0800e870 tx=0x7fed0800eb80 comp rx=0 tx=0).stop 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.499+0000 7fed1e800700 1 -- 172.21.15.12:0/3809354882 shutdown_connections 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.499+0000 7fed1e800700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecf806adb0 0x7fecf806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.499+0000 7fed1e800700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18160d70 0x7fed180af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.499+0000 7fed1e800700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed1806a790 0x7fed1809cd00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.499+0000 7fed1e800700 1 --2- 172.21.15.12:0/3809354882 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed1815ef70 0x7fed180aeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.499+0000 7fed1e800700 1 -- 172.21.15.12:0/3809354882 >> 172.21.15.12:0/3809354882 conn(0x7fed180b5720 msgr2=0x7fed18146c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:26.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.499+0000 7fed1e800700 1 -- 172.21.15.12:0/3809354882 shutdown_connections 2024-03-29T11:11:26.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.499+0000 7fed1e800700 1 -- 172.21.15.12:0/3809354882 wait complete. 2024-03-29T11:11:26.496 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:11:26.650 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:26.672+0000 7f6fd2963240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:11:26.684 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:26.706+0000 7f94498a8240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:26.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.723+0000 7f3484800700 1 Processor -- start 2024-03-29T11:11:26.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.723+0000 7f3484800700 1 -- start start 2024-03-29T11:11:26.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.724+0000 7f3484800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c0d18b0 0x7f347c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.724+0000 7f3484800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f347c161590 0x7f347c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.724+0000 7f3484800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c0ceda0 con 0x7f347c160f10 2024-03-29T11:11:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.724+0000 7f3484800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c0d1d30 con 0x7f347c161590 2024-03-29T11:11:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.724+0000 7f3484800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c161f60 con 0x7f347c0d18b0 2024-03-29T11:11:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3481e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f347c160f10 0x7f347c0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47340/0 (socket says 172.21.15.12:47340) 2024-03-29T11:11:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3481e00700 1 -- 172.21.15.12:0/458300520 learned_addr learned my addr 172.21.15.12:0/458300520 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3481400700 1 --2- 172.21.15.12:0/458300520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c0d18b0 0x7f347c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3482800700 1 --2- 172.21.15.12:0/458300520 >> v2:172.21.15.159:3300/0 conn(0x7f347c161590 0x7f347c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:26.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3480a00700 1 -- 172.21.15.12:0/458300520 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327977622 0 0) 0x7f347c0ceda0 con 0x7f347c160f10 2024-03-29T11:11:26.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3480a00700 1 -- 172.21.15.12:0/458300520 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34640035b0 con 0x7f347c160f10 2024-03-29T11:11:26.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3481400700 1 -- 172.21.15.12:0/458300520 >> v2:172.21.15.159:3300/0 conn(0x7f347c161590 msgr2=0x7f347c161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:26.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3481400700 1 --2- 172.21.15.12:0/458300520 >> v2:172.21.15.159:3300/0 conn(0x7f347c161590 0x7f347c161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3481400700 1 -- 172.21.15.12:0/458300520 >> v1:172.21.15.12:6789/0 conn(0x7f347c160f10 legacy=0x7f347c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3481400700 1 -- 172.21.15.12:0/458300520 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c1612f0 con 0x7f347c0d18b0 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.725+0000 7f3482800700 1 --2- 172.21.15.12:0/458300520 >> v2:172.21.15.159:3300/0 conn(0x7f347c161590 0x7f347c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.726+0000 7f3481400700 1 --2- 172.21.15.12:0/458300520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c0d18b0 0x7f347c0cf230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f346c009870 tx=0x7f346c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18a0ba9e5df5683b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.726+0000 7f3480a00700 1 -- 172.21.15.12:0/458300520 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c011830 con 0x7f347c0d18b0 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.726+0000 7f3480a00700 1 -- 172.21.15.12:0/458300520 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346c011990 con 0x7f347c0d18b0 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.726+0000 7f3480a00700 1 -- 172.21.15.12:0/458300520 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c011d20 con 0x7f347c0d18b0 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.726+0000 7f3484800700 1 -- 172.21.15.12:0/458300520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c0d18b0 msgr2=0x7f347c0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.726+0000 7f3484800700 1 --2- 172.21.15.12:0/458300520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c0d18b0 0x7f347c0cf230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f346c009870 tx=0x7f346c009b80 comp rx=0 tx=0).stop 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.727+0000 7f3484800700 1 -- 172.21.15.12:0/458300520 shutdown_connections 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.727+0000 7f3484800700 1 --2- 172.21.15.12:0/458300520 >> v2:172.21.15.159:3300/0 conn(0x7f347c161590 0x7f347c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.727+0000 7f3484800700 1 --2- 172.21.15.12:0/458300520 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c0d18b0 0x7f347c0cf230 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.727+0000 7f3484800700 1 -- 172.21.15.12:0/458300520 >> 172.21.15.12:0/458300520 conn(0x7f347c08aff0 msgr2=0x7f347c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.727+0000 7f3484800700 1 -- 172.21.15.12:0/458300520 shutdown_connections 2024-03-29T11:11:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.727+0000 7f3484800700 1 -- 172.21.15.12:0/458300520 wait complete. 2024-03-29T11:11:26.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.729+0000 7f3484800700 1 Processor -- start 2024-03-29T11:11:26.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.729+0000 7f3484800700 1 -- start start 2024-03-29T11:11:26.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3484800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f347c0d18b0 0x7f347c14e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3484800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c160f10 0x7f347c09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3484800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f347c161590 0x7f347c09f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3484800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c159320 con 0x7f347c161590 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3484800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c09f760 con 0x7f347c0d18b0 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3484800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c14ede0 con 0x7f347c160f10 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3482800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f347c161590 0x7f347c09f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3481e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f347c0d18b0 0x7f347c14e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3482800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f347c161590 0x7f347c09f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58150/0 (socket says 172.21.15.12:58150) 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3482800700 1 -- 172.21.15.12:0/1118190549 learned_addr learned my addr 172.21.15.12:0/1118190549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3481400700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c160f10 0x7f347c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3482800700 1 -- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c160f10 msgr2=0x7f347c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3482800700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c160f10 0x7f347c09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3482800700 1 -- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f347c0d18b0 msgr2=0x7f347c14e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3482800700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f347c0d18b0 0x7f347c14e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.730+0000 7f3482800700 1 -- 172.21.15.12:0/1118190549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3478009a70 con 0x7f347c161590 2024-03-29T11:11:26.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.731+0000 7f3482800700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f347c161590 0x7f347c09f220 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f347800efc0 tx=0x7f347800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:26.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.731+0000 7f3476000700 1 -- 172.21.15.12:0/1118190549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347800ca30 con 0x7f347c161590 2024-03-29T11:11:26.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.731+0000 7f3476000700 1 -- 172.21.15.12:0/1118190549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347800f040 con 0x7f347c161590 2024-03-29T11:11:26.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.731+0000 7f3484800700 1 -- 172.21.15.12:0/1118190549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346c0094a0 con 0x7f347c161590 2024-03-29T11:11:26.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.731+0000 7f3476000700 1 -- 172.21.15.12:0/1118190549 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347800cdd0 con 0x7f347c161590 2024-03-29T11:11:26.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.731+0000 7f3484800700 1 -- 172.21.15.12:0/1118190549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c09fc70 con 0x7f347c161590 2024-03-29T11:11:26.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.732+0000 7f3484800700 1 -- 172.21.15.12:0/1118190549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c067720 con 0x7f347c161590 2024-03-29T11:11:26.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.733+0000 7f3476000700 1 -- 172.21.15.12:0/1118190549 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f3478014480 con 0x7f347c161590 2024-03-29T11:11:26.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.734+0000 7f3476000700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f346006adb0 0x7f346006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:26.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.734+0000 7f3476000700 1 -- 172.21.15.12:0/1118190549 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f3478089e20 con 0x7f347c161590 2024-03-29T11:11:26.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.734+0000 7f3481e00700 1 -- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f346006adb0 msgr2=0x7f346006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:26.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.734+0000 7f3481e00700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f346006adb0 0x7f346006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:26.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.742+0000 7f3476000700 1 -- 172.21.15.12:0/1118190549 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3478054bf0 con 0x7f347c161590 2024-03-29T11:11:26.883 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:26.905+0000 7f6fd2963240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:11:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.935+0000 7f3481e00700 1 -- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f346006adb0 msgr2=0x7f346006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:26.935+0000 7f3481e00700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f346006adb0 0x7f346006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:26.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.007+0000 7f3484800700 1 -- 172.21.15.12:0/1118190549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f347c0b5ba0 con 0x7f347c161590 2024-03-29T11:11:26.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.008+0000 7f3476000700 1 -- 172.21.15.12:0/1118190549 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f3478009cd0 con 0x7f347c161590 2024-03-29T11:11:26.986 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:26.986 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3149931506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3149931506},{"type":"v1","addr":"172.21.15.159:6835","nonce":3149931506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/967685480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":967685480},{"type":"v1","addr":"172.21.15.159:6837","nonce":967685480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8967,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4273394318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4273394318},{"type":"v1","addr":"172.21.15.12:6834","nonce":4273394318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:15:18.742186+0000","modified":"2024-03-29T10:15:22.226050+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8970},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8970":{"gid":8970,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/512120304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":512120304},{"type":"v1","addr":"172.21.15.12:6835","nonce":512120304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-29T11:11:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 -- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f346006adb0 msgr2=0x7f346006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f346006adb0 0x7f346006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 -- 172.21.15.12:0/1118190549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f347c161590 msgr2=0x7f347c09f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f347c161590 0x7f347c09f220 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f347800efc0 tx=0x7f347800c3a0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 -- 172.21.15.12:0/1118190549 shutdown_connections 2024-03-29T11:11:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f346006adb0 0x7f346006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f347c161590 0x7f347c09f220 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f347c0d18b0 0x7f347c14e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 --2- 172.21.15.12:0/1118190549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c160f10 0x7f347c09ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 -- 172.21.15.12:0/1118190549 >> 172.21.15.12:0/1118190549 conn(0x7f347c08aff0 msgr2=0x7f347c0b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 -- 172.21.15.12:0/1118190549 shutdown_connections 2024-03-29T11:11:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.010+0000 7f3484800700 1 -- 172.21.15.12:0/1118190549 wait complete. 2024-03-29T11:11:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 73 2024-03-29T11:11:27.007 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:27.111 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:27.133+0000 7f6fd2963240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:27.166 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:27.188+0000 7f94498a8240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:11:27.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.235+0000 7f7f02200700 1 Processor -- start 2024-03-29T11:11:27.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.235+0000 7f7f02200700 1 -- start start 2024-03-29T11:11:27.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.236+0000 7f7f02200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 0x7f7efc099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.236+0000 7f7f02200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7efc06e2f0 0x7f7efc0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.236+0000 7f7f02200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc06e780 con 0x7f7efc15a5a0 2024-03-29T11:11:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.236+0000 7f7f02200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7efc072440 con 0x7f7efc06e2f0 2024-03-29T11:11:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.236+0000 7f7f02200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc06f320 con 0x7f7efc0cc6f0 2024-03-29T11:11:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.236+0000 7f7efaa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 0x7f7efc099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.236+0000 7f7efbe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7efc06e2f0 0x7f7efc0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.237+0000 7f7efb400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f7efc15a5a0 0x7f7efc06dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:37236/0 (socket says 172.21.15.12:37236) 2024-03-29T11:11:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.237+0000 7f7efb400700 1 -- 172.21.15.12:0/3114297045 learned_addr learned my addr 172.21.15.12:0/3114297045 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:27.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.237+0000 7f7efaa00700 1 -- 172.21.15.12:0/3114297045 >> v2:172.21.15.159:3300/0 conn(0x7f7efc06e2f0 msgr2=0x7f7efc0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.237+0000 7f7efaa00700 1 --2- 172.21.15.12:0/3114297045 >> v2:172.21.15.159:3300/0 conn(0x7f7efc06e2f0 0x7f7efc0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.237+0000 7f7efaa00700 1 -- 172.21.15.12:0/3114297045 >> v1:172.21.15.159:6789/0 conn(0x7f7efc15a5a0 legacy=0x7f7efc06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.237+0000 7f7efaa00700 1 -- 172.21.15.12:0/3114297045 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc06e9f0 con 0x7f7efc0cc6f0 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.237+0000 7f7efbe00700 1 --2- 172.21.15.12:0/3114297045 >> v2:172.21.15.159:3300/0 conn(0x7f7efc06e2f0 0x7f7efc0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.237+0000 7f7efaa00700 1 --2- 172.21.15.12:0/3114297045 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 0x7f7efc099a60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009870 tx=0x7f7ee8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=538a864808c3051a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.238+0000 7f7efa000700 1 -- 172.21.15.12:0/3114297045 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8011830 con 0x7f7efc0cc6f0 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.238+0000 7f7efa000700 1 -- 172.21.15.12:0/3114297045 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee8011990 con 0x7f7efc0cc6f0 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.238+0000 7f7efa000700 1 -- 172.21.15.12:0/3114297045 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8011d20 con 0x7f7efc0cc6f0 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.238+0000 7f7f02200700 1 -- 172.21.15.12:0/3114297045 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 msgr2=0x7f7efc099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.238+0000 7f7f02200700 1 --2- 172.21.15.12:0/3114297045 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 0x7f7efc099a60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009870 tx=0x7f7ee8009b80 comp rx=0 tx=0).stop 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.238+0000 7f7f02200700 1 -- 172.21.15.12:0/3114297045 shutdown_connections 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.238+0000 7f7f02200700 1 --2- 172.21.15.12:0/3114297045 >> v2:172.21.15.159:3300/0 conn(0x7f7efc06e2f0 0x7f7efc0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.238+0000 7f7f02200700 1 --2- 172.21.15.12:0/3114297045 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 0x7f7efc099a60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.238+0000 7f7f02200700 1 -- 172.21.15.12:0/3114297045 >> 172.21.15.12:0/3114297045 conn(0x7f7efc0b5ab0 msgr2=0x7f7efc0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:27.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.239+0000 7f7f02200700 1 -- 172.21.15.12:0/3114297045 shutdown_connections 2024-03-29T11:11:27.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.239+0000 7f7f02200700 1 -- 172.21.15.12:0/3114297045 wait complete. 2024-03-29T11:11:27.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.240+0000 7f7f02200700 1 Processor -- start 2024-03-29T11:11:27.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.240+0000 7f7f02200700 1 -- start start 2024-03-29T11:11:27.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.241+0000 7f7f02200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7efc06e2f0 0x7f7efc0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.241+0000 7f7f02200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 0x7f7efc0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.241+0000 7f7f02200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7efc15a5a0 0x7f7efc1f2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.241+0000 7f7f02200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0ae280 con 0x7f7efc06e2f0 2024-03-29T11:11:27.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.241+0000 7f7f02200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0ad9e0 con 0x7f7efc15a5a0 2024-03-29T11:11:27.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.241+0000 7f7f02200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0adcc0 con 0x7f7efc0cc6f0 2024-03-29T11:11:27.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7efc06e2f0 0x7f7efc0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:27.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7efc06e2f0 0x7f7efc0ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58166/0 (socket says 172.21.15.12:58166) 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efb400700 1 -- 172.21.15.12:0/3311874355 learned_addr learned my addr 172.21.15.12:0/3311874355 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efaa00700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 0x7f7efc0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efbe00700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7efc15a5a0 0x7f7efc1f2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efbe00700 1 -- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 msgr2=0x7f7efc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efbe00700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 0x7f7efc0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efbe00700 1 -- 172.21.15.12:0/3311874355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7efc06e2f0 msgr2=0x7f7efc0ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efbe00700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7efc06e2f0 0x7f7efc0ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efbe00700 1 -- 172.21.15.12:0/3311874355 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef400bc80 con 0x7f7efc15a5a0 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efb400700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7efc06e2f0 0x7f7efc0ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efaa00700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 0x7f7efc0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.242+0000 7f7efbe00700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7efc15a5a0 0x7f7efc1f2680 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009370 tx=0x7f7ef400e3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.243+0000 7f7ef3400700 1 -- 172.21.15.12:0/3311874355 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4010040 con 0x7f7efc15a5a0 2024-03-29T11:11:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.243+0000 7f7ef3400700 1 -- 172.21.15.12:0/3311874355 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef4003c20 con 0x7f7efc15a5a0 2024-03-29T11:11:27.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.243+0000 7f7f02200700 1 -- 172.21.15.12:0/3311874355 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee80094a0 con 0x7f7efc15a5a0 2024-03-29T11:11:27.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.243+0000 7f7ef3400700 1 -- 172.21.15.12:0/3311874355 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef400b3a0 con 0x7f7efc15a5a0 2024-03-29T11:11:27.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.243+0000 7f7f02200700 1 -- 172.21.15.12:0/3311874355 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7efc15aba0 con 0x7f7efc15a5a0 2024-03-29T11:11:27.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.244+0000 7f7f02200700 1 -- 172.21.15.12:0/3311874355 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7efc067720 con 0x7f7efc15a5a0 2024-03-29T11:11:27.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.245+0000 7f7ef3400700 1 -- 172.21.15.12:0/3311874355 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f7ef400e940 con 0x7f7efc15a5a0 2024-03-29T11:11:27.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.246+0000 7f7ef3400700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7edc06adb0 0x7f7edc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.246+0000 7f7ef3400700 1 -- 172.21.15.12:0/3311874355 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f7ef4089bd0 con 0x7f7efc15a5a0 2024-03-29T11:11:27.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.246+0000 7f7efb400700 1 -- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7edc06adb0 msgr2=0x7f7edc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:27.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.246+0000 7f7efb400700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7edc06adb0 0x7f7edc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:27.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.250+0000 7f7ef3400700 1 -- 172.21.15.12:0/3311874355 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7ef40549a0 con 0x7f7efc15a5a0 2024-03-29T11:11:27.376 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:27.398+0000 7f94498a8240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.447+0000 7f7efb400700 1 -- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7edc06adb0 msgr2=0x7f7edc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.447+0000 7f7efb400700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7edc06adb0 0x7f7edc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:27.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.463+0000 7f7f02200700 1 -- 172.21.15.12:0/3311874355 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7efc0afdc0 con 0x7f7efc15a5a0 2024-03-29T11:11:27.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.465+0000 7f7ef3400700 1 -- 172.21.15.12:0/3311874355 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+16816 (secure 0 0 0) 0x7f7ef4057fc0 con 0x7f7efc15a5a0 2024-03-29T11:11:27.443 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:27.443 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":164,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:22.039712+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:15:17.110269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-29T10:15:17.717343+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.9","mappings":[{"from":2,"to":0}]},{"pgid":"19.23","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2146007013":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/2699841727":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/2365815480":"2024-03-30T10:15:35.915268+0000","172.21.15.159:6833/43444":"2024-03-30T10:15:35.915268+0000","172.21.15.159:6832/43444":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/3483953818":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/1508471695":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/3197130837":"2024-03-30T10:15:35.915268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":54,"length":3},{"begin":110,"length":3}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.467+0000 7f7f02200700 1 -- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7edc06adb0 msgr2=0x7f7edc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.467+0000 7f7f02200700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7edc06adb0 0x7f7edc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.467+0000 7f7f02200700 1 -- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7efc15a5a0 msgr2=0x7f7efc1f2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.467+0000 7f7f02200700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7efc15a5a0 0x7f7efc1f2680 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009370 tx=0x7f7ef400e3a0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.467+0000 7f7f02200700 1 -- 172.21.15.12:0/3311874355 shutdown_connections 2024-03-29T11:11:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.467+0000 7f7f02200700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7edc06adb0 0x7f7edc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.467+0000 7f7f02200700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7efc15a5a0 0x7f7efc1f2680 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.467+0000 7f7f02200700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7efc06e2f0 0x7f7efc0ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.467+0000 7f7f02200700 1 --2- 172.21.15.12:0/3311874355 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7efc0cc6f0 0x7f7efc0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.467+0000 7f7f02200700 1 -- 172.21.15.12:0/3311874355 >> 172.21.15.12:0/3311874355 conn(0x7f7efc0b5ab0 msgr2=0x7f7efc0b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:27.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.468+0000 7f7f02200700 1 -- 172.21.15.12:0/3311874355 shutdown_connections 2024-03-29T11:11:27.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.468+0000 7f7f02200700 1 -- 172.21.15.12:0/3311874355 wait complete. 2024-03-29T11:11:27.464 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T11:11:27.464 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:11:27.633 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:27.655+0000 7f6fd2963240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:11:27.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.693+0000 7f68afe00700 1 Processor -- start 2024-03-29T11:11:27.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.694+0000 7f68afe00700 1 -- start start 2024-03-29T11:11:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.695+0000 7f68afe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f68b00d0a00 0x7f68b00d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.695+0000 7f68afe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b00d2870 0x7f68b00d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.695+0000 7f68afe00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b00d2dc0 con 0x7f68b015d5d0 2024-03-29T11:11:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.695+0000 7f68afe00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b015d9b0 con 0x7f68b00d0a00 2024-03-29T11:11:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.695+0000 7f68afe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b00b2860 con 0x7f68b00d2870 2024-03-29T11:11:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68aea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f68b015d5d0 0x7f68b00d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47346/0 (socket says 172.21.15.12:47346) 2024-03-29T11:11:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68aea00700 1 -- 172.21.15.12:0/1605001351 learned_addr learned my addr 172.21.15.12:0/1605001351 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68ae000700 1 --2- 172.21.15.12:0/1605001351 >> v2:172.21.15.159:3300/0 conn(0x7f68b00d0a00 0x7f68b00d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68af400700 1 --2- 172.21.15.12:0/1605001351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b00d2870 0x7f68b00d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68ad600700 1 -- 172.21.15.12:0/1605001351 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503525587 0 0) 0x7f68b00d2dc0 con 0x7f68b015d5d0 2024-03-29T11:11:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68ad600700 1 -- 172.21.15.12:0/1605001351 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68940035b0 con 0x7f68b015d5d0 2024-03-29T11:11:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68ae000700 1 -- 172.21.15.12:0/1605001351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b00d2870 msgr2=0x7f68b00d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68ae000700 1 --2- 172.21.15.12:0/1605001351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b00d2870 0x7f68b00d0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68ae000700 1 -- 172.21.15.12:0/1605001351 >> v1:172.21.15.12:6789/0 conn(0x7f68b015d5d0 legacy=0x7f68b00d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68ae000700 1 -- 172.21.15.12:0/1605001351 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b00d2470 con 0x7f68b00d0a00 2024-03-29T11:11:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.696+0000 7f68af400700 1 --2- 172.21.15.12:0/1605001351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b00d2870 0x7f68b00d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.697+0000 7f68ae000700 1 --2- 172.21.15.12:0/1605001351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68b00d0a00 0x7f68b00d3e60 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f689c009840 tx=0x7f689c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b875fdb8b2f3cce1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.697+0000 7f68ad600700 1 -- 172.21.15.12:0/1605001351 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c011830 con 0x7f68b00d0a00 2024-03-29T11:11:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.697+0000 7f68ad600700 1 -- 172.21.15.12:0/1605001351 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c011990 con 0x7f68b00d0a00 2024-03-29T11:11:27.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.697+0000 7f68ad600700 1 -- 172.21.15.12:0/1605001351 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c011d20 con 0x7f68b00d0a00 2024-03-29T11:11:27.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.697+0000 7f68afe00700 1 -- 172.21.15.12:0/1605001351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68b00d0a00 msgr2=0x7f68b00d3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.697+0000 7f68afe00700 1 --2- 172.21.15.12:0/1605001351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68b00d0a00 0x7f68b00d3e60 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f689c009840 tx=0x7f689c009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:27.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.698+0000 7f68afe00700 1 -- 172.21.15.12:0/1605001351 shutdown_connections 2024-03-29T11:11:27.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.698+0000 7f68afe00700 1 --2- 172.21.15.12:0/1605001351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b00d2870 0x7f68b00d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.698+0000 7f68afe00700 1 --2- 172.21.15.12:0/1605001351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68b00d0a00 0x7f68b00d3e60 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.698+0000 7f68afe00700 1 -- 172.21.15.12:0/1605001351 >> 172.21.15.12:0/1605001351 conn(0x7f68b00668e0 msgr2=0x7f68b0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:27.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.698+0000 7f68afe00700 1 -- 172.21.15.12:0/1605001351 shutdown_connections 2024-03-29T11:11:27.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.698+0000 7f68afe00700 1 -- 172.21.15.12:0/1605001351 wait complete. 2024-03-29T11:11:27.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.700+0000 7f68afe00700 1 Processor -- start 2024-03-29T11:11:27.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.700+0000 7f68afe00700 1 -- start start 2024-03-29T11:11:27.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.701+0000 7f68afe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b00d0a00 0x7f68b01574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68afe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68b00d2870 0x7f68b01f5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68afe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b015d5d0 0x7f68b0157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68afe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b01f5fe0 con 0x7f68b00d0a00 2024-03-29T11:11:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68afe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b0159530 con 0x7f68b00d2870 2024-03-29T11:11:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68afe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b0159810 con 0x7f68b015d5d0 2024-03-29T11:11:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68aea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b00d0a00 0x7f68b01574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68aea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b00d0a00 0x7f68b01574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58170/0 (socket says 172.21.15.12:58170) 2024-03-29T11:11:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68aea00700 1 -- 172.21.15.12:0/3032647808 learned_addr learned my addr 172.21.15.12:0/3032647808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68ae000700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68b00d2870 0x7f68b01f5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68af400700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b015d5d0 0x7f68b0157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68ae000700 1 -- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b015d5d0 msgr2=0x7f68b0157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68ae000700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b015d5d0 0x7f68b0157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68ae000700 1 -- 172.21.15.12:0/3032647808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b00d0a00 msgr2=0x7f68b01574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68ae000700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b00d0a00 0x7f68b01574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68ae000700 1 -- 172.21.15.12:0/3032647808 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a8009a70 con 0x7f68b00d2870 2024-03-29T11:11:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.702+0000 7f68aea00700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b00d0a00 0x7f68b01574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.703+0000 7f68ae000700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68b00d2870 0x7f68b01f5aa0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f689c016000 tx=0x7f689c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.703+0000 7f68a6a00700 1 -- 172.21.15.12:0/3032647808 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c004550 con 0x7f68b00d2870 2024-03-29T11:11:27.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.703+0000 7f68a6a00700 1 -- 172.21.15.12:0/3032647808 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c01e070 con 0x7f68b00d2870 2024-03-29T11:11:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.703+0000 7f68afe00700 1 -- 172.21.15.12:0/3032647808 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689c0094a0 con 0x7f68b00d2870 2024-03-29T11:11:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.703+0000 7f68a6a00700 1 -- 172.21.15.12:0/3032647808 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c0038e0 con 0x7f68b00d2870 2024-03-29T11:11:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.703+0000 7f68afe00700 1 -- 172.21.15.12:0/3032647808 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b01587e0 con 0x7f68b00d2870 2024-03-29T11:11:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.704+0000 7f68afe00700 1 -- 172.21.15.12:0/3032647808 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b0159980 con 0x7f68b00d2870 2024-03-29T11:11:27.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.705+0000 7f68a6a00700 1 -- 172.21.15.12:0/3032647808 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 86) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f689c003a40 con 0x7f68b00d2870 2024-03-29T11:11:27.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.706+0000 7f68a6a00700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f689006adb0 0x7f689006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:27.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.706+0000 7f68a6a00700 1 -- 172.21.15.12:0/3032647808 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f689c08a550 con 0x7f68b00d2870 2024-03-29T11:11:27.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.706+0000 7f68aea00700 1 -- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f689006adb0 msgr2=0x7f689006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:27.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.706+0000 7f68aea00700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f689006adb0 0x7f689006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:27.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.711+0000 7f68a6a00700 1 -- 172.21.15.12:0/3032647808 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f689c055420 con 0x7f68b00d2870 2024-03-29T11:11:27.821 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:27.843+0000 7f94498a8240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:11:27.858 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:27.878+0000 7f6fd2963240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:27.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.907+0000 7f68aea00700 1 -- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f689006adb0 msgr2=0x7f689006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:27.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.907+0000 7f68aea00700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f689006adb0 0x7f689006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:27.932 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:27.954+0000 7f94498a8240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:27.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.966+0000 7f68a6a00700 1 -- 172.21.15.12:0/3032647808 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mgrmap(e 87) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f689c003cf0 con 0x7f68b00d2870 2024-03-29T11:11:27.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.966+0000 7f68a6a00700 1 -- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f689006adb0 msgr2=0x7f689006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:27.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.966+0000 7f68a6a00700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f689006adb0 0x7f689006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.985+0000 7f68afe00700 1 -- 172.21.15.12:0/3032647808 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68b004ea80 con 0x7f68b00d2870 2024-03-29T11:11:27.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.986+0000 7f68a6a00700 1 -- 172.21.15.12:0/3032647808 <== mon.1 v2:172.21.15.159:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f689c06fc00 con 0x7f68b00d2870 2024-03-29T11:11:27.964 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:27.964 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3149931506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3149931506},{"type":"v1","addr":"172.21.15.159:6835","nonce":3149931506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/967685480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":967685480},{"type":"v1","addr":"172.21.15.159:6837","nonce":967685480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8967,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4273394318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4273394318},{"type":"v1","addr":"172.21.15.12:6834","nonce":4273394318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:15:18.742186+0000","modified":"2024-03-29T10:15:22.226050+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8970},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8970":{"gid":8970,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/512120304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":512120304},{"type":"v1","addr":"172.21.15.12:6835","nonce":512120304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-29T11:11:27.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.989+0000 7f68afe00700 1 -- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68b00d2870 msgr2=0x7f68b01f5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.989+0000 7f68afe00700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68b00d2870 0x7f68b01f5aa0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f689c016000 tx=0x7f689c004750 comp rx=0 tx=0).stop 2024-03-29T11:11:27.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.989+0000 7f68afe00700 1 -- 172.21.15.12:0/3032647808 shutdown_connections 2024-03-29T11:11:27.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.989+0000 7f68afe00700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f689006adb0 0x7f689006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.989+0000 7f68afe00700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68b015d5d0 0x7f68b0157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.989+0000 7f68afe00700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b00d0a00 0x7f68b01574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.989+0000 7f68afe00700 1 --2- 172.21.15.12:0/3032647808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68b00d2870 0x7f68b01f5aa0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.989+0000 7f68afe00700 1 -- 172.21.15.12:0/3032647808 >> 172.21.15.12:0/3032647808 conn(0x7f68b00668e0 msgr2=0x7f68b00d02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.989+0000 7f68afe00700 1 -- 172.21.15.12:0/3032647808 shutdown_connections 2024-03-29T11:11:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:27.989+0000 7f68afe00700 1 -- 172.21.15.12:0/3032647808 wait complete. 2024-03-29T11:11:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 73 2024-03-29T11:11:27.993 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:11:28.025 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:28.046+0000 7f93bca00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:11:28.025 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:28.046+0000 7f93bca00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:11:28.025 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:28.046+0000 7f93bca00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:11:28.025 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:28.046+0000 7f93bca00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:11:28.025 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:28.046+0000 7f93b8400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:11:28.025 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:28.046+0000 7f93b8400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:11:28.025 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:28.046+0000 7f93bca00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:11:28.025 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:28.046+0000 7f93b8400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:11:28.025 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:28.046+0000 7f93b8400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:11:28.025 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:28.046+0000 7f93b8400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:11:28.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.230+0000 7f2fd5c00700 1 Processor -- start 2024-03-29T11:11:28.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.231+0000 7f2fd5c00700 1 -- start start 2024-03-29T11:11:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.232+0000 7f2fd5c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2fd015da50 0x7f2fd00ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.232+0000 7f2fd5c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00d24e0 con 0x7f2fd00cec70 2024-03-29T11:11:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.232+0000 7f2fd5c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00d27c0 con 0x7f2fd00d0cf0 2024-03-29T11:11:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.232+0000 7f2fd5c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd00b95d0 con 0x7f2fd015da50 2024-03-29T11:11:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.233+0000 7f2fce000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2fd00cec70 0x7f2fd00d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47362/0 (socket says 172.21.15.12:47362) 2024-03-29T11:11:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.233+0000 7f2fce000700 1 -- 172.21.15.12:0/977357600 learned_addr learned my addr 172.21.15.12:0/977357600 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.233+0000 7f2fcea00700 1 --2- 172.21.15.12:0/977357600 >> v2:172.21.15.159:3300/0 conn(0x7f2fd015da50 0x7f2fd00ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.233+0000 7f2fcea00700 1 -- 172.21.15.12:0/977357600 >> v1:172.21.15.159:6789/0 conn(0x7f2fd00d0cf0 legacy=0x7f2fd00b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.233+0000 7f2fcea00700 1 -- 172.21.15.12:0/977357600 >> v1:172.21.15.12:6789/0 conn(0x7f2fd00cec70 legacy=0x7f2fd00d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.233+0000 7f2fcea00700 1 -- 172.21.15.12:0/977357600 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd00d2aa0 con 0x7f2fd015da50 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.233+0000 7f2fcd600700 1 -- 172.21.15.12:0/977357600 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134818845 0 0) 0x7f2fd00d24e0 con 0x7f2fd00cec70 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.234+0000 7f2fcea00700 1 --2- 172.21.15.12:0/977357600 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fd015da50 0x7f2fd00ce730 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009840 tx=0x7f2fc0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e56df9951885218b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.234+0000 7f2fcd600700 1 -- 172.21.15.12:0/977357600 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0011830 con 0x7f2fd015da50 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.234+0000 7f2fcd600700 1 -- 172.21.15.12:0/977357600 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc0011990 con 0x7f2fd015da50 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.234+0000 7f2fcd600700 1 -- 172.21.15.12:0/977357600 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0011d20 con 0x7f2fd015da50 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.234+0000 7f2fd5c00700 1 -- 172.21.15.12:0/977357600 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fd015da50 msgr2=0x7f2fd00ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.234+0000 7f2fd5c00700 1 --2- 172.21.15.12:0/977357600 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fd015da50 0x7f2fd00ce730 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009840 tx=0x7f2fc0009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.235+0000 7f2fd5c00700 1 -- 172.21.15.12:0/977357600 shutdown_connections 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.235+0000 7f2fd5c00700 1 --2- 172.21.15.12:0/977357600 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fd015da50 0x7f2fd00ce730 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.235+0000 7f2fd5c00700 1 -- 172.21.15.12:0/977357600 >> 172.21.15.12:0/977357600 conn(0x7f2fd00666a0 msgr2=0x7f2fd0066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:28.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.235+0000 7f2fd5c00700 1 -- 172.21.15.12:0/977357600 shutdown_connections 2024-03-29T11:11:28.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.235+0000 7f2fd5c00700 1 -- 172.21.15.12:0/977357600 wait complete. 2024-03-29T11:11:28.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.237+0000 7f2fd5c00700 1 Processor -- start 2024-03-29T11:11:28.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.237+0000 7f2fd5c00700 1 -- start start 2024-03-29T11:11:28.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fd5c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fd00cec70 0x7f2fd01492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:28.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fd5c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fd00d0cf0 0x7f2fd0146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:28.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fd5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fd015da50 0x7f2fd01f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:28.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fd5c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd0145880 con 0x7f2fd015da50 2024-03-29T11:11:28.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fd5c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd01467d0 con 0x7f2fd00d0cf0 2024-03-29T11:11:28.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fd5c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd01f7520 con 0x7f2fd00cec70 2024-03-29T11:11:28.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fcf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fd015da50 0x7f2fd01f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:28.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fcf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fd015da50 0x7f2fd01f6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58182/0 (socket says 172.21.15.12:58182) 2024-03-29T11:11:28.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fcf400700 1 -- 172.21.15.12:0/1056903589 learned_addr learned my addr 172.21.15.12:0/1056903589 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:28.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fce000700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fd00d0cf0 0x7f2fd0146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:28.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.238+0000 7f2fcea00700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fd00cec70 0x7f2fd01492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fcf400700 1 -- 172.21.15.12:0/1056903589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fd00cec70 msgr2=0x7f2fd01492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fcf400700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fd00cec70 0x7f2fd01492c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fcf400700 1 -- 172.21.15.12:0/1056903589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fd00d0cf0 msgr2=0x7f2fd0146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fcf400700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fd00d0cf0 0x7f2fd0146290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fcf400700 1 -- 172.21.15.12:0/1056903589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00094a0 con 0x7f2fd015da50 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fce000700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fd00d0cf0 0x7f2fd0146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fcea00700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fd00cec70 0x7f2fd01492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fcf400700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fd015da50 0x7f2fd01f6fe0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800cb10 tx=0x7f2fc800ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fc6a00700 1 -- 172.21.15.12:0/1056903589 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8003c00 con 0x7f2fd015da50 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fc6a00700 1 -- 172.21.15.12:0/1056903589 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc8003d60 con 0x7f2fd015da50 2024-03-29T11:11:28.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fd5c00700 1 -- 172.21.15.12:0/1056903589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd00cd2f0 con 0x7f2fd015da50 2024-03-29T11:11:28.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fc6a00700 1 -- 172.21.15.12:0/1056903589 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc800a9e0 con 0x7f2fd015da50 2024-03-29T11:11:28.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.239+0000 7f2fd5c00700 1 -- 172.21.15.12:0/1056903589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd0145a90 con 0x7f2fd015da50 2024-03-29T11:11:28.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.241+0000 7f2fd5c00700 1 -- 172.21.15.12:0/1056903589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd0067720 con 0x7f2fd015da50 2024-03-29T11:11:28.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.242+0000 7f2fc6a00700 1 -- 172.21.15.12:0/1056903589 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 87) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f2fc800ab40 con 0x7f2fd015da50 2024-03-29T11:11:28.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.243+0000 7f2fc6a00700 1 -- 172.21.15.12:0/1056903589 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6457+0+0 (secure 0 0 0) 0x7f2fc8088f50 con 0x7f2fd015da50 2024-03-29T11:11:28.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.248+0000 7f2fc6a00700 1 -- 172.21.15.12:0/1056903589 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2fc80bc580 con 0x7f2fd015da50 2024-03-29T11:11:28.339 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:28.361+0000 7f6fd2963240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:11:28.456 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:28.478+0000 7f6fd2963240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:28.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.510+0000 7f2fd5c00700 1 -- 172.21.15.12:0/1056903589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fd00b5f20 con 0x7f2fd015da50 2024-03-29T11:11:28.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.511+0000 7f2fc6a00700 1 -- 172.21.15.12:0/1056903589 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f2fc8054340 con 0x7f2fd015da50 2024-03-29T11:11:28.489 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:28.489 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3149931506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3149931506},{"type":"v1","addr":"172.21.15.159:6835","nonce":3149931506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/967685480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":967685480},{"type":"v1","addr":"172.21.15.159:6837","nonce":967685480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8967,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4273394318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4273394318},{"type":"v1","addr":"172.21.15.12:6834","nonce":4273394318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T10:15:18.742186+0000","modified":"2024-03-29T10:15:22.226050+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8970},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8970":{"gid":8970,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.12:6835/512120304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":512120304},{"type":"v1","addr":"172.21.15.12:6835","nonce":512120304}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-29T11:11:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.513+0000 7f2fd5c00700 1 -- 172.21.15.12:0/1056903589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fd015da50 msgr2=0x7f2fd01f6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.513+0000 7f2fd5c00700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fd015da50 0x7f2fd01f6fe0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800cb10 tx=0x7f2fc800ce20 comp rx=0 tx=0).stop 2024-03-29T11:11:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.513+0000 7f2fd5c00700 1 -- 172.21.15.12:0/1056903589 shutdown_connections 2024-03-29T11:11:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.513+0000 7f2fd5c00700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fd015da50 0x7f2fd01f6fe0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.513+0000 7f2fd5c00700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fd00cec70 0x7f2fd01492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.513+0000 7f2fd5c00700 1 --2- 172.21.15.12:0/1056903589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fd00d0cf0 0x7f2fd0146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.513+0000 7f2fd5c00700 1 -- 172.21.15.12:0/1056903589 >> 172.21.15.12:0/1056903589 conn(0x7f2fd00666a0 msgr2=0x7f2fd00d3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.513+0000 7f2fd5c00700 1 -- 172.21.15.12:0/1056903589 shutdown_connections 2024-03-29T11:11:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.513+0000 7f2fd5c00700 1 -- 172.21.15.12:0/1056903589 wait complete. 2024-03-29T11:11:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 73 2024-03-29T11:11:28.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:28.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.728+0000 7f16cbe00700 1 Processor -- start 2024-03-29T11:11:28.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.729+0000 7f16cbe00700 1 -- start start 2024-03-29T11:11:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.730+0000 7f16cbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f16c40b3af0 0x7f16c415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.730+0000 7f16cbe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c4072580 0x7f16c40b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.730+0000 7f16cbe00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c40b2180 con 0x7f16c40b2e50 2024-03-29T11:11:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.730+0000 7f16cbe00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c415b100 con 0x7f16c40b3af0 2024-03-29T11:11:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.730+0000 7f16cbe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c415b3e0 con 0x7f16c4072580 2024-03-29T11:11:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.730+0000 7f16c9400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f16c40b3af0 0x7f16c415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.730+0000 7f16c9400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f16c40b3af0 0x7f16c415a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58188/0 (socket says 172.21.15.12:58188) 2024-03-29T11:11:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.730+0000 7f16c9400700 1 -- 172.21.15.12:0/3949091068 learned_addr learned my addr 172.21.15.12:0/3949091068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.730+0000 7f16c8a00700 1 --2- 172.21.15.12:0/3949091068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c4072580 0x7f16c40b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.731+0000 7f16c9400700 1 -- 172.21.15.12:0/3949091068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c4072580 msgr2=0x7f16c40b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:28.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.731+0000 7f16c9400700 1 --2- 172.21.15.12:0/3949091068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c4072580 0x7f16c40b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.731+0000 7f16c9400700 1 -- 172.21.15.12:0/3949091068 >> v1:172.21.15.159:6789/0 conn(0x7f16c40b2e50 legacy=0x7f16c40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.731+0000 7f16c9400700 1 -- 172.21.15.12:0/3949091068 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c40b23f0 con 0x7f16c40b3af0 2024-03-29T11:11:28.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.731+0000 7f16c8a00700 1 --2- 172.21.15.12:0/3949091068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c4072580 0x7f16c40b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:11:28.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.731+0000 7f16c9400700 1 --2- 172.21.15.12:0/3949091068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 0x7f16c415a860 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f16b80098f0 tx=0x7f16b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d7a919cde191af server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:28.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.731+0000 7f16c3e00700 1 -- 172.21.15.12:0/3949091068 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b800b8a0 con 0x7f16c40b3af0 2024-03-29T11:11:28.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.731+0000 7f16c3e00700 1 -- 172.21.15.12:0/3949091068 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b800ba00 con 0x7f16c40b3af0 2024-03-29T11:11:28.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.731+0000 7f16c3e00700 1 -- 172.21.15.12:0/3949091068 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b800bd90 con 0x7f16c40b3af0 2024-03-29T11:11:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.732+0000 7f16cbe00700 1 -- 172.21.15.12:0/3949091068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 msgr2=0x7f16c415a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.732+0000 7f16cbe00700 1 --2- 172.21.15.12:0/3949091068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 0x7f16c415a860 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f16b80098f0 tx=0x7f16b8009c00 comp rx=0 tx=0).stop 2024-03-29T11:11:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.732+0000 7f16cbe00700 1 -- 172.21.15.12:0/3949091068 shutdown_connections 2024-03-29T11:11:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.732+0000 7f16cbe00700 1 --2- 172.21.15.12:0/3949091068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 0x7f16c415a860 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.732+0000 7f16cbe00700 1 --2- 172.21.15.12:0/3949091068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c4072580 0x7f16c40b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.732+0000 7f16cbe00700 1 -- 172.21.15.12:0/3949091068 >> 172.21.15.12:0/3949091068 conn(0x7f16c40cf190 msgr2=0x7f16c40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.732+0000 7f16cbe00700 1 -- 172.21.15.12:0/3949091068 shutdown_connections 2024-03-29T11:11:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.732+0000 7f16cbe00700 1 -- 172.21.15.12:0/3949091068 wait complete. 2024-03-29T11:11:28.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.734+0000 7f16cbe00700 1 Processor -- start 2024-03-29T11:11:28.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.734+0000 7f16cbe00700 1 -- start start 2024-03-29T11:11:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16cbe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16c4072580 0x7f16c4159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16cbe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c40b2e50 0x7f16c4153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 0x7f16c4150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16cbe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4159d50 con 0x7f16c40b3af0 2024-03-29T11:11:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16cbe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4159ad0 con 0x7f16c4072580 2024-03-29T11:11:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16cbe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4158520 con 0x7f16c40b2e50 2024-03-29T11:11:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 0x7f16c4150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 0x7f16c4150780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58204/0 (socket says 172.21.15.12:58204) 2024-03-29T11:11:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c9e00700 1 -- 172.21.15.12:0/3427439524 learned_addr learned my addr 172.21.15.12:0/3427439524 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c9400700 1 --2- 172.21.15.12:0/3427439524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16c4072580 0x7f16c4159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c8a00700 1 --2- 172.21.15.12:0/3427439524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c40b2e50 0x7f16c4153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c9e00700 1 -- 172.21.15.12:0/3427439524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c40b2e50 msgr2=0x7f16c4153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c9e00700 1 --2- 172.21.15.12:0/3427439524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c40b2e50 0x7f16c4153640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c9e00700 1 -- 172.21.15.12:0/3427439524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16c4072580 msgr2=0x7f16c4159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c9e00700 1 --2- 172.21.15.12:0/3427439524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16c4072580 0x7f16c4159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c9e00700 1 -- 172.21.15.12:0/3427439524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b8009550 con 0x7f16c40b3af0 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.735+0000 7f16c9400700 1 --2- 172.21.15.12:0/3427439524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16c4072580 0x7f16c4159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.736+0000 7f16c9e00700 1 --2- 172.21.15.12:0/3427439524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 0x7f16c4150780 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00cb50 tx=0x7f16bc00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.736+0000 7f16c1600700 1 -- 172.21.15.12:0/3427439524 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc003c00 con 0x7f16c40b3af0 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.736+0000 7f16c1600700 1 -- 172.21.15.12:0/3427439524 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16bc003d60 con 0x7f16c40b3af0 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.736+0000 7f16c1600700 1 -- 172.21.15.12:0/3427439524 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc00ab70 con 0x7f16c40b3af0 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.736+0000 7f16cbe00700 1 -- 172.21.15.12:0/3427439524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c4158800 con 0x7f16c40b3af0 2024-03-29T11:11:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.736+0000 7f16cbe00700 1 -- 172.21.15.12:0/3427439524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c4158dc0 con 0x7f16c40b3af0 2024-03-29T11:11:28.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.737+0000 7f16cbe00700 1 -- 172.21.15.12:0/3427439524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c4067720 con 0x7f16c40b3af0 2024-03-29T11:11:28.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.738+0000 7f16c1600700 1 -- 172.21.15.12:0/3427439524 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 87) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f16bc00acd0 con 0x7f16c40b3af0 2024-03-29T11:11:28.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.739+0000 7f16c1600700 1 -- 172.21.15.12:0/3427439524 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6457+0+0 (secure 0 0 0) 0x7f16bc0891c0 con 0x7f16c40b3af0 2024-03-29T11:11:28.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.743+0000 7f16c1600700 1 -- 172.21.15.12:0/3427439524 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f16bc054410 con 0x7f16c40b3af0 2024-03-29T11:11:28.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.960+0000 7f16cbe00700 1 -- 172.21.15.12:0/3427439524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16c415a640 con 0x7f16c40b3af0 2024-03-29T11:11:28.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.963+0000 7f16c1600700 1 -- 172.21.15.12:0/3427439524 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+17193 (secure 0 0 0) 0x7f16bc057a30 con 0x7f16c40b3af0 2024-03-29T11:11:28.941 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:28.941 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":165,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:27.958149+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-29T10:15:17.110269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-29T10:15:17.717343+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.9","mappings":[{"from":2,"to":0}]},{"pgid":"19.23","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1694886292":"2024-03-30T11:11:27.958030+0000","172.21.15.159:0/208761155":"2024-03-30T11:11:27.958030+0000","172.21.15.159:0/1987351566":"2024-03-30T11:11:27.958030+0000","172.21.15.159:0/2146007013":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/1071105503":"2024-03-30T11:11:27.958030+0000","172.21.15.159:0/2699841727":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/1892051654":"2024-03-30T11:11:27.958030+0000","172.21.15.159:0/2365815480":"2024-03-30T10:15:35.915268+0000","172.21.15.159:6833/43444":"2024-03-30T10:15:35.915268+0000","172.21.15.159:6832/43444":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/3483953818":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/1508471695":"2024-03-30T10:15:35.915268+0000","172.21.15.159:0/1637997494":"2024-03-30T11:11:27.958030+0000","172.21.15.159:0/3197130837":"2024-03-30T10:15:35.915268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:28.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.965+0000 7f16cbe00700 1 -- 172.21.15.12:0/3427439524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 msgr2=0x7f16c4150780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.965+0000 7f16cbe00700 1 --2- 172.21.15.12:0/3427439524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 0x7f16c4150780 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00cb50 tx=0x7f16bc00cf10 comp rx=0 tx=0).stop 2024-03-29T11:11:28.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.965+0000 7f16cbe00700 1 -- 172.21.15.12:0/3427439524 shutdown_connections 2024-03-29T11:11:28.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.965+0000 7f16cbe00700 1 --2- 172.21.15.12:0/3427439524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16c40b3af0 0x7f16c4150780 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.965+0000 7f16cbe00700 1 --2- 172.21.15.12:0/3427439524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16c4072580 0x7f16c4159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.965+0000 7f16cbe00700 1 --2- 172.21.15.12:0/3427439524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c40b2e50 0x7f16c4153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:28.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.965+0000 7f16cbe00700 1 -- 172.21.15.12:0/3427439524 >> 172.21.15.12:0/3427439524 conn(0x7f16c40cf190 msgr2=0x7f16c406b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:28.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.966+0000 7f16cbe00700 1 -- 172.21.15.12:0/3427439524 shutdown_connections 2024-03-29T11:11:28.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:28.966+0000 7f16cbe00700 1 -- 172.21.15.12:0/3427439524 wait complete. 2024-03-29T11:11:28.968 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T11:11:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.191+0000 7fd597a00700 1 Processor -- start 2024-03-29T11:11:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.191+0000 7fd597a00700 1 -- start start 2024-03-29T11:11:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.192+0000 7fd597a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd590160e50 0x7fd5900b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.192+0000 7fd597a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5900d3f20 0x7fd5900b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.192+0000 7fd597a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd5900b3f80 0x7fd5900b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.192+0000 7fd597a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd590161230 con 0x7fd5900d3f20 2024-03-29T11:11:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.192+0000 7fd597a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5900d2a80 con 0x7fd5900b3f80 2024-03-29T11:11:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.192+0000 7fd597a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5900d43b0 con 0x7fd590160e50 2024-03-29T11:11:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.192+0000 7fd58fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5900d3f20 0x7fd5900b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd58fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5900d3f20 0x7fd5900b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38506/0 (socket says 172.21.15.12:38506) 2024-03-29T11:11:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd58fe00700 1 -- 172.21.15.12:0/4020639504 learned_addr learned my addr 172.21.15.12:0/4020639504 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd595a00700 1 --2- 172.21.15.12:0/4020639504 >> v2:172.21.15.159:3300/0 conn(0x7fd5900b3f80 0x7fd5900b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd595000700 1 --2- 172.21.15.12:0/4020639504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd590160e50 0x7fd5900b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd58fe00700 1 -- 172.21.15.12:0/4020639504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd590160e50 msgr2=0x7fd5900b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd58fe00700 1 --2- 172.21.15.12:0/4020639504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd590160e50 0x7fd5900b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd58fe00700 1 -- 172.21.15.12:0/4020639504 >> v2:172.21.15.159:3300/0 conn(0x7fd5900b3f80 msgr2=0x7fd5900b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd58fe00700 1 --2- 172.21.15.12:0/4020639504 >> v2:172.21.15.159:3300/0 conn(0x7fd5900b3f80 0x7fd5900b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd58fe00700 1 -- 172.21.15.12:0/4020639504 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5900d21e0 con 0x7fd5900d3f20 2024-03-29T11:11:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd595000700 1 --2- 172.21.15.12:0/4020639504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd590160e50 0x7fd5900b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:11:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd58fe00700 1 --2- 172.21.15.12:0/4020639504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5900d3f20 0x7fd5900b53d0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fd5900d3e70 tx=0x7fd57c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa192c2f2d1fd4e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.193+0000 7fd58f400700 1 -- 172.21.15.12:0/4020639504 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c011830 con 0x7fd5900d3f20 2024-03-29T11:11:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd58f400700 1 -- 172.21.15.12:0/4020639504 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c011990 con 0x7fd5900d3f20 2024-03-29T11:11:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd58f400700 1 -- 172.21.15.12:0/4020639504 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c011d20 con 0x7fd5900d3f20 2024-03-29T11:11:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd597a00700 1 -- 172.21.15.12:0/4020639504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5900d3f20 msgr2=0x7fd5900b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd597a00700 1 --2- 172.21.15.12:0/4020639504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5900d3f20 0x7fd5900b53d0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fd5900d3e70 tx=0x7fd57c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd597a00700 1 -- 172.21.15.12:0/4020639504 shutdown_connections 2024-03-29T11:11:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd597a00700 1 --2- 172.21.15.12:0/4020639504 >> v2:172.21.15.159:3300/0 conn(0x7fd5900b3f80 0x7fd5900b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd597a00700 1 --2- 172.21.15.12:0/4020639504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd590160e50 0x7fd5900b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd597a00700 1 --2- 172.21.15.12:0/4020639504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5900d3f20 0x7fd5900b53d0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd597a00700 1 -- 172.21.15.12:0/4020639504 >> 172.21.15.12:0/4020639504 conn(0x7fd59008aff0 msgr2=0x7fd5900d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd597a00700 1 -- 172.21.15.12:0/4020639504 shutdown_connections 2024-03-29T11:11:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.194+0000 7fd597a00700 1 -- 172.21.15.12:0/4020639504 wait complete. 2024-03-29T11:11:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.196+0000 7fd597a00700 1 Processor -- start 2024-03-29T11:11:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.196+0000 7fd597a00700 1 -- start start 2024-03-29T11:11:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.197+0000 7fd597a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5900b3f80 0x7fd59014d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.197+0000 7fd597a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd590160e50 0x7fd59014fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.197+0000 7fd597a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5901500f0 0x7fd5900b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.197+0000 7fd597a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5900a12b0 con 0x7fd5901500f0 2024-03-29T11:11:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.197+0000 7fd597a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59014e050 con 0x7fd590160e50 2024-03-29T11:11:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.197+0000 7fd597a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd590150640 con 0x7fd5900b3f80 2024-03-29T11:11:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.197+0000 7fd595a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5901500f0 0x7fd5900b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.197+0000 7fd595a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5901500f0 0x7fd5900b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38516/0 (socket says 172.21.15.12:38516) 2024-03-29T11:11:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.197+0000 7fd595a00700 1 -- 172.21.15.12:0/396867308 learned_addr learned my addr 172.21.15.12:0/396867308 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.197+0000 7fd595000700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5900b3f80 0x7fd59014d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd58fe00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd590160e50 0x7fd59014fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd595a00700 1 -- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5900b3f80 msgr2=0x7fd59014d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd595a00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5900b3f80 0x7fd59014d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd595a00700 1 -- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd590160e50 msgr2=0x7fd59014fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd595a00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd590160e50 0x7fd59014fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd595a00700 1 -- 172.21.15.12:0/396867308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c009550 con 0x7fd5901500f0 2024-03-29T11:11:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd595a00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5901500f0 0x7fd5900b4e90 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fd58800eea0 tx=0x7fd58800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd58cc00700 1 -- 172.21.15.12:0/396867308 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588010040 con 0x7fd5901500f0 2024-03-29T11:11:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd58cc00700 1 -- 172.21.15.12:0/396867308 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58800ba70 con 0x7fd5901500f0 2024-03-29T11:11:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd58cc00700 1 -- 172.21.15.12:0/396867308 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58800be20 con 0x7fd5901500f0 2024-03-29T11:11:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.198+0000 7fd597a00700 1 -- 172.21.15.12:0/396867308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd590150920 con 0x7fd5901500f0 2024-03-29T11:11:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.199+0000 7fd597a00700 1 -- 172.21.15.12:0/396867308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59014d7a0 con 0x7fd5901500f0 2024-03-29T11:11:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.200+0000 7fd597a00700 1 -- 172.21.15.12:0/396867308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd590150b60 con 0x7fd5901500f0 2024-03-29T11:11:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.201+0000 7fd58cc00700 1 -- 172.21.15.12:0/396867308 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 88) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fd58801d400 con 0x7fd5901500f0 2024-03-29T11:11:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.202+0000 7fd58cc00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd57006aae0 0x7fd57006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.202+0000 7fd58cc00700 1 -- 172.21.15.12:0/396867308 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6457+0+0 (secure 0 0 0) 0x7fd588089af0 con 0x7fd5901500f0 2024-03-29T11:11:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.202+0000 7fd595000700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd57006aae0 0x7fd57006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.203+0000 7fd595000700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd57006aae0 0x7fd57006cfa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd58400c940 tx=0x7fd584008040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.207+0000 7fd58cc00700 1 -- 172.21.15.12:0/396867308 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd588054c50 con 0x7fd5901500f0 2024-03-29T11:11:29.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.466+0000 7fd597a00700 1 -- 172.21.15.12:0/396867308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd590067720 con 0x7fd5901500f0 2024-03-29T11:11:29.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T11:11:29.467+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T11:11:29.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.477+0000 7fd58cc00700 1 -- 172.21.15.12:0/396867308 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd588018020 con 0x7fd5901500f0 2024-03-29T11:11:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 -- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd57006aae0 msgr2=0x7fd57006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd57006aae0 0x7fd57006cfa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd58400c940 tx=0x7fd584008040 comp rx=0 tx=0).stop 2024-03-29T11:11:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 -- 172.21.15.12:0/396867308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5901500f0 msgr2=0x7fd5900b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5901500f0 0x7fd5900b4e90 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fd58800eea0 tx=0x7fd58800b390 comp rx=0 tx=0).stop 2024-03-29T11:11:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 -- 172.21.15.12:0/396867308 shutdown_connections 2024-03-29T11:11:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fd57006aae0 0x7fd57006cfa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5901500f0 0x7fd5900b4e90 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5900b3f80 0x7fd59014d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 --2- 172.21.15.12:0/396867308 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd590160e50 0x7fd59014fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 -- 172.21.15.12:0/396867308 >> 172.21.15.12:0/396867308 conn(0x7fd59008aff0 msgr2=0x7fd5900ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 -- 172.21.15.12:0/396867308 shutdown_connections 2024-03-29T11:11:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.480+0000 7fd597a00700 1 -- 172.21.15.12:0/396867308 wait complete. 2024-03-29T11:11:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T11:11:29.477 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T11:11:29.524 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T11:11:29.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.705+0000 7f804d000700 1 Processor -- start 2024-03-29T11:11:29.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.705+0000 7f804d000700 1 -- start start 2024-03-29T11:11:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.706+0000 7f804d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80480d16a0 0x7f80480d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.706+0000 7f804d000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f80480d2370 0x7f80480ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.706+0000 7f804d000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8048161420 con 0x7f804815d9d0 2024-03-29T11:11:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.706+0000 7f804d000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80480d2820 con 0x7f80480d2370 2024-03-29T11:11:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.706+0000 7f804d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80480d4340 con 0x7f80480d16a0 2024-03-29T11:11:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.706+0000 7f8045e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f804815d9d0 0x7f80480d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35698/0 (socket says 172.21.15.12:35698) 2024-03-29T11:11:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.706+0000 7f8045e00700 1 -- 172.21.15.12:0/502537968 learned_addr learned my addr 172.21.15.12:0/502537968 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.706+0000 7f8045400700 1 --2- 172.21.15.12:0/502537968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80480d16a0 0x7f80480d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.706+0000 7f8046800700 1 --2- 172.21.15.12:0/502537968 >> v2:172.21.15.159:3300/0 conn(0x7f80480d2370 0x7f80480ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.707+0000 7f8044a00700 1 -- 172.21.15.12:0/502537968 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936085409 0 0) 0x7f8048161420 con 0x7f804815d9d0 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.707+0000 7f8044a00700 1 -- 172.21.15.12:0/502537968 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f802c0035b0 con 0x7f804815d9d0 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.707+0000 7f8046800700 1 -- 172.21.15.12:0/502537968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80480d16a0 msgr2=0x7f80480d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.707+0000 7f8046800700 1 --2- 172.21.15.12:0/502537968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80480d16a0 0x7f80480d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.707+0000 7f8046800700 1 -- 172.21.15.12:0/502537968 >> v1:172.21.15.12:6789/0 conn(0x7f804815d9d0 legacy=0x7f80480d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.707+0000 7f8046800700 1 -- 172.21.15.12:0/502537968 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8048161700 con 0x7f80480d2370 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.707+0000 7f8046800700 1 --2- 172.21.15.12:0/502537968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80480d2370 0x7f80480ce360 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8040009dc0 tx=0x7f804000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1f51b21bdbfb618 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.707+0000 7f8044a00700 1 -- 172.21.15.12:0/502537968 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804000e070 con 0x7f80480d2370 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.708+0000 7f8044a00700 1 -- 172.21.15.12:0/502537968 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804000bae0 con 0x7f80480d2370 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.708+0000 7f8044a00700 1 -- 172.21.15.12:0/502537968 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804000be90 con 0x7f80480d2370 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.708+0000 7f804d000700 1 -- 172.21.15.12:0/502537968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80480d2370 msgr2=0x7f80480ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.708+0000 7f804d000700 1 --2- 172.21.15.12:0/502537968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80480d2370 0x7f80480ce360 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8040009dc0 tx=0x7f804000b330 comp rx=0 tx=0).stop 2024-03-29T11:11:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.708+0000 7f804d000700 1 -- 172.21.15.12:0/502537968 shutdown_connections 2024-03-29T11:11:29.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.708+0000 7f804d000700 1 --2- 172.21.15.12:0/502537968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80480d2370 0x7f80480ce360 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.708+0000 7f804d000700 1 --2- 172.21.15.12:0/502537968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80480d16a0 0x7f80480d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.708+0000 7f804d000700 1 -- 172.21.15.12:0/502537968 >> 172.21.15.12:0/502537968 conn(0x7f804808aff0 msgr2=0x7f80480d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:29.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.708+0000 7f804d000700 1 -- 172.21.15.12:0/502537968 shutdown_connections 2024-03-29T11:11:29.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.708+0000 7f804d000700 1 -- 172.21.15.12:0/502537968 wait complete. 2024-03-29T11:11:29.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.710+0000 7f804d000700 1 Processor -- start 2024-03-29T11:11:29.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.710+0000 7f804d000700 1 -- start start 2024-03-29T11:11:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f804d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80480d16a0 0x7f80481588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f804d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80480d2370 0x7f80480ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f804d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f804815d9d0 0x7f80480b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f804d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8048158e20 con 0x7f80480d2370 2024-03-29T11:11:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f804d000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804815eff0 con 0x7f804815d9d0 2024-03-29T11:11:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f804d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804815f290 con 0x7f80480d16a0 2024-03-29T11:11:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f8045400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80480d2370 0x7f80480ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f8045400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80480d2370 0x7f80480ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38546/0 (socket says 172.21.15.12:38546) 2024-03-29T11:11:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f8045400700 1 -- 172.21.15.12:0/913602690 learned_addr learned my addr 172.21.15.12:0/913602690 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f8045e00700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80480d16a0 0x7f80481588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.711+0000 7f8046800700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f804815d9d0 0x7f80480b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f8045400700 1 -- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80480d16a0 msgr2=0x7f80481588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f8045400700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80480d16a0 0x7f80481588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f8045400700 1 -- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f804815d9d0 msgr2=0x7f80480b89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f8045400700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f804815d9d0 0x7f80480b89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f8045400700 1 -- 172.21.15.12:0/913602690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8040009a70 con 0x7f80480d2370 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f8045400700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80480d2370 0x7f80480ca5f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f803400e960 tx=0x7f803400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f803e000700 1 -- 172.21.15.12:0/913602690 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80340096f0 con 0x7f80480d2370 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f803e000700 1 -- 172.21.15.12:0/913602690 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8034013680 con 0x7f80480d2370 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f803e000700 1 -- 172.21.15.12:0/913602690 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8034013a30 con 0x7f80480d2370 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f804d000700 1 -- 172.21.15.12:0/913602690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f804815f570 con 0x7f80480d2370 2024-03-29T11:11:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.712+0000 7f804d000700 1 -- 172.21.15.12:0/913602690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f804815fb30 con 0x7f80480d2370 2024-03-29T11:11:29.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.713+0000 7f804d000700 1 -- 172.21.15.12:0/913602690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8048160180 con 0x7f80480d2370 2024-03-29T11:11:29.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.714+0000 7f803e000700 1 -- 172.21.15.12:0/913602690 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 88) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f8034013b90 con 0x7f80480d2370 2024-03-29T11:11:29.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.715+0000 7f803e000700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f802806aae0 0x7f802806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:29.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.715+0000 7f803e000700 1 -- 172.21.15.12:0/913602690 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f8034089720 con 0x7f80480d2370 2024-03-29T11:11:29.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.715+0000 7f8045e00700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f802806aae0 0x7f802806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:29.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.716+0000 7f8045e00700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f802806aae0 0x7f802806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8038007fd0 tx=0x7f8038007dc0 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:29.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.720+0000 7f803e000700 1 -- 172.21.15.12:0/913602690 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f80340547d0 con 0x7f80480d2370 2024-03-29T11:11:29.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:29.984+0000 7f804d000700 1 -- 172.21.15.12:0/913602690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f80480d1ca0 con 0x7f80480d2370 2024-03-29T11:11:30.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.481+0000 7f803e000700 1 -- 172.21.15.12:0/913602690 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 89) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f8034054ba0 con 0x7f80480d2370 2024-03-29T11:11:30.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.488+0000 7f803e000700 1 -- 172.21.15.12:0/913602690 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8034074520 con 0x7f80480d2370 2024-03-29T11:11:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 -- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f802806aae0 msgr2=0x7f802806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f802806aae0 0x7f802806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8038007fd0 tx=0x7f8038007dc0 comp rx=0 tx=0).stop 2024-03-29T11:11:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 -- 172.21.15.12:0/913602690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80480d2370 msgr2=0x7f80480ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80480d2370 0x7f80480ca5f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f803400e960 tx=0x7f803400ec70 comp rx=0 tx=0).stop 2024-03-29T11:11:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 -- 172.21.15.12:0/913602690 shutdown_connections 2024-03-29T11:11:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f802806aae0 0x7f802806cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f804815d9d0 0x7f80480b89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80480d16a0 0x7f80481588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 --2- 172.21.15.12:0/913602690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80480d2370 0x7f80480ca5f0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 -- 172.21.15.12:0/913602690 >> 172.21.15.12:0/913602690 conn(0x7f804808aff0 msgr2=0x7f804806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 -- 172.21.15.12:0/913602690 shutdown_connections 2024-03-29T11:11:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.491+0000 7f804d000700 1 -- 172.21.15.12:0/913602690 wait complete. 2024-03-29T11:11:30.487 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T11:11:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.752+0000 7f1925400700 1 Processor -- start 2024-03-29T11:11:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.753+0000 7f1925400700 1 -- start start 2024-03-29T11:11:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.754+0000 7f1925400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1920160e10 0x7f1920099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.754+0000 7f1925400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f192009ada0 con 0x7f19200b2380 2024-03-29T11:11:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.754+0000 7f1925400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19201611f0 con 0x7f19200cbc20 2024-03-29T11:11:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.754+0000 7f1925400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192006cc80 con 0x7f1920160e10 2024-03-29T11:11:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.754+0000 7f191de00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f19200b2380 0x7f1920099fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35712/0 (socket says 172.21.15.12:35712) 2024-03-29T11:11:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.754+0000 7f191de00700 1 -- 172.21.15.12:0/3721310181 learned_addr learned my addr 172.21.15.12:0/3721310181 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.754+0000 7f191e800700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1920160e10 0x7f1920099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.754+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229442390 0 0) 0x7f192009ada0 con 0x7f19200b2380 2024-03-29T11:11:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.754+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19040035b0 con 0x7f19200b2380 2024-03-29T11:11:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 375023014 0 0) 0x7f19040035b0 con 0x7f19200b2380 2024-03-29T11:11:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1920160e10 msgr2=0x7f1920099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1920160e10 0x7f1920099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 >> v1:172.21.15.159:6789/0 conn(0x7f19200cbc20 legacy=0x7f19200cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19200d0840 con 0x7f19200b2380 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f190c001670 con 0x7f19200b2380 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 >> v1:172.21.15.12:6789/0 conn(0x7f19200b2380 legacy=0x7f1920099fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1904003a30 0x7f1904003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1904004850 0x7f1904004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1920160e10 0x7f1904005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192006cc80 con 0x7f1904004850 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190400bb40 con 0x7f1904003a30 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190400bd00 con 0x7f1920160e10 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191de00700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1904004850 0x7f1904004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191e800700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1904003a30 0x7f1904003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.755+0000 7f191f200700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1920160e10 0x7f1904005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.756+0000 7f191de00700 1 -- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1920160e10 msgr2=0x7f1904005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.756+0000 7f191de00700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1920160e10 0x7f1904005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.756+0000 7f191de00700 1 -- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1904003a30 msgr2=0x7f1904003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.756+0000 7f191de00700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1904003a30 0x7f1904003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.756+0000 7f191de00700 1 -- 172.21.15.12:0/3721310181 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19040035b0 con 0x7f1904004850 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.756+0000 7f191de00700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1904004850 0x7f1904004cd0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f190c00c240 tx=0x7f1904005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b192b67d67cdbc1d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.756+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c007c10 con 0x7f1904004850 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.756+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c00ce00 con 0x7f1904004850 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.756+0000 7f191d400700 1 -- 172.21.15.12:0/3721310181 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c005ba0 con 0x7f1904004850 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.757+0000 7f1925400700 1 -- 172.21.15.12:0/3721310181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1904004850 msgr2=0x7f1904004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.757+0000 7f1925400700 1 --2- 172.21.15.12:0/3721310181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1904004850 0x7f1904004cd0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f190c00c240 tx=0x7f1904005880 comp rx=0 tx=0).stop 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.757+0000 7f191f200700 1 -- 172.21.15.12:0/3721310181 reap_dead start 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.757+0000 7f1925400700 1 -- 172.21.15.12:0/3721310181 shutdown_connections 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.757+0000 7f1925400700 1 -- 172.21.15.12:0/3721310181 >> 172.21.15.12:0/3721310181 conn(0x7f19200b2e20 msgr2=0x7f19200b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.757+0000 7f1925400700 1 -- 172.21.15.12:0/3721310181 shutdown_connections 2024-03-29T11:11:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.757+0000 7f1925400700 1 -- 172.21.15.12:0/3721310181 wait complete. 2024-03-29T11:11:30.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.759+0000 7f1925400700 1 Processor -- start 2024-03-29T11:11:30.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.759+0000 7f1925400700 1 -- start start 2024-03-29T11:11:30.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f1925400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1904004850 0x7f192015f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:30.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f1925400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f192015a860 0x7f19200b5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:30.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f1925400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19200b6500 0x7f19201f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:30.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f1925400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192015fa80 con 0x7f192015a860 2024-03-29T11:11:30.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f1925400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192015f0b0 con 0x7f1904004850 2024-03-29T11:11:30.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f1925400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19200b6980 con 0x7f19200b6500 2024-03-29T11:11:30.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f191de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f192015a860 0x7f19200b5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f191de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f192015a860 0x7f19200b5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38568/0 (socket says 172.21.15.12:38568) 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f191de00700 1 -- 172.21.15.12:0/1554107945 learned_addr learned my addr 172.21.15.12:0/1554107945 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f191e800700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1904004850 0x7f192015f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f191f200700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19200b6500 0x7f19201f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f191de00700 1 -- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19200b6500 msgr2=0x7f19201f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f191de00700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19200b6500 0x7f19201f3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f191de00700 1 -- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1904004850 msgr2=0x7f192015f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f191de00700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1904004850 0x7f192015f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.760+0000 7f191de00700 1 -- 172.21.15.12:0/1554107945 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19201f3dc0 con 0x7f192015a860 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.761+0000 7f191de00700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f192015a860 0x7f19200b5fc0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f190c00c830 tx=0x7f190c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.761+0000 7f1916a00700 1 -- 172.21.15.12:0/1554107945 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c004a70 con 0x7f192015a860 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.761+0000 7f1916a00700 1 -- 172.21.15.12:0/1554107945 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c004bd0 con 0x7f192015a860 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.761+0000 7f1925400700 1 -- 172.21.15.12:0/1554107945 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192015e780 con 0x7f192015a860 2024-03-29T11:11:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.761+0000 7f1916a00700 1 -- 172.21.15.12:0/1554107945 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c0070d0 con 0x7f192015a860 2024-03-29T11:11:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.761+0000 7f1925400700 1 -- 172.21.15.12:0/1554107945 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192015ed40 con 0x7f192015a860 2024-03-29T11:11:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.762+0000 7f1925400700 1 -- 172.21.15.12:0/1554107945 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192015ef00 con 0x7f192015a860 2024-03-29T11:11:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.763+0000 7f1916a00700 1 -- 172.21.15.12:0/1554107945 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 89) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f190c019070 con 0x7f192015a860 2024-03-29T11:11:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.764+0000 7f1916a00700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f190006ad20 0x7f190006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.764+0000 7f1916a00700 1 -- 172.21.15.12:0/1554107945 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f190c08a230 con 0x7f192015a860 2024-03-29T11:11:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.764+0000 7f191e800700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f190006ad20 0x7f190006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.765+0000 7f191e800700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f190006ad20 0x7f190006d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f19200ca8d0 tx=0x7f1910008040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.768+0000 7f1916a00700 1 -- 172.21.15.12:0/1554107945 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f190c055190 con 0x7f192015a860 2024-03-29T11:11:30.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:30.983+0000 7f1925400700 1 -- 172.21.15.12:0/1554107945 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1920067720 con 0x7f192015a860 2024-03-29T11:11:31.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.989+0000 7f1916a00700 1 -- 172.21.15.12:0/1554107945 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v167) v1 ==== 181+0+0 (secure 0 0 0) 0x7f190c007d70 con 0x7f192015a860 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.994+0000 7f1925400700 1 -- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f190006ad20 msgr2=0x7f190006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.994+0000 7f1925400700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f190006ad20 0x7f190006d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f19200ca8d0 tx=0x7f1910008040 comp rx=0 tx=0).stop 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.994+0000 7f1925400700 1 -- 172.21.15.12:0/1554107945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f192015a860 msgr2=0x7f19200b5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.994+0000 7f1925400700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f192015a860 0x7f19200b5fc0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f190c00c830 tx=0x7f190c001a00 comp rx=0 tx=0).stop 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.994+0000 7f1925400700 1 -- 172.21.15.12:0/1554107945 shutdown_connections 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.994+0000 7f1925400700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f190006ad20 0x7f190006d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.994+0000 7f1925400700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19200b6500 0x7f19201f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.994+0000 7f1925400700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1904004850 0x7f192015f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.994+0000 7f1925400700 1 --2- 172.21.15.12:0/1554107945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f192015a860 0x7f19200b5fc0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.994+0000 7f1925400700 1 -- 172.21.15.12:0/1554107945 >> 172.21.15.12:0/1554107945 conn(0x7f19200b2e20 msgr2=0x7f192015ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.995+0000 7f1925400700 1 -- 172.21.15.12:0/1554107945 shutdown_connections 2024-03-29T11:11:31.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:31.995+0000 7f1925400700 1 -- 172.21.15.12:0/1554107945 wait complete. 2024-03-29T11:11:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T11:11:31.995 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T11:11:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.229+0000 7f0f57800700 1 Processor -- start 2024-03-29T11:11:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.230+0000 7f0f57800700 1 -- start start 2024-03-29T11:11:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.231+0000 7f0f57800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015d930 0x7f0f500d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.231+0000 7f0f57800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5015dd10 con 0x7f0f500d22a0 2024-03-29T11:11:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f57800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5015dfc0 con 0x7f0f50162030 2024-03-29T11:11:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f57800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5015ec40 con 0x7f0f5015d930 2024-03-29T11:11:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f55800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0f500d22a0 0x7f0f500ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35720/0 (socket says 172.21.15.12:35720) 2024-03-29T11:11:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f55800700 1 -- 172.21.15.12:0/1315599602 learned_addr learned my addr 172.21.15.12:0/1315599602 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f54e00700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015d930 0x7f0f500d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648651752 0 0) 0x7f0f5015dd10 con 0x7f0f500d22a0 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f380035b0 con 0x7f0f500d22a0 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2943319099 0 0) 0x7f0f380035b0 con 0x7f0f500d22a0 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015d930 msgr2=0x7f0f500d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f4f400700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015d930 0x7f0f500d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 >> v1:172.21.15.159:6789/0 conn(0x7f0f50162030 legacy=0x7f0f500d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5015e2a0 con 0x7f0f500d22a0 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.232+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0f480019f0 con 0x7f0f500d22a0 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 >> v1:172.21.15.12:6789/0 conn(0x7f0f500d22a0 legacy=0x7f0f500ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4f400700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015d930 0x7f0f38003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4f400700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38003f70 0x7f0f38004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4f400700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f38004d90 0x7f0f38005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5015ec40 con 0x7f0f38003f70 2024-03-29T11:11:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3800bac0 con 0x7f0f38004d90 2024-03-29T11:11:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3800bc80 con 0x7f0f5015d930 2024-03-29T11:11:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4fe00700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38003f70 0x7f0f38004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f54e00700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015d930 0x7f0f38003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f55800700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f38004d90 0x7f0f38005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4fe00700 1 -- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015d930 msgr2=0x7f0f38003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4fe00700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015d930 0x7f0f38003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4fe00700 1 -- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f38004d90 msgr2=0x7f0f38005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4fe00700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f38004d90 0x7f0f38005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.233+0000 7f0f4fe00700 1 -- 172.21.15.12:0/1315599602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f380035b0 con 0x7f0f38003f70 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.234+0000 7f0f4fe00700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38003f70 0x7f0f38004850 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0018d0 tx=0x7f0f38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1393f7e4e0a02cee server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.234+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c00c690 con 0x7f0f38003f70 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.234+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3c00c7f0 con 0x7f0f38003f70 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.234+0000 7f0f4f400700 1 -- 172.21.15.12:0/1315599602 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c00cba0 con 0x7f0f38003f70 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.234+0000 7f0f57800700 1 -- 172.21.15.12:0/1315599602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38003f70 msgr2=0x7f0f38004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.234+0000 7f0f57800700 1 --2- 172.21.15.12:0/1315599602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38003f70 0x7f0f38004850 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0018d0 tx=0x7f0f38005880 comp rx=0 tx=0).stop 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.234+0000 7f0f55800700 1 -- 172.21.15.12:0/1315599602 reap_dead start 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.234+0000 7f0f57800700 1 -- 172.21.15.12:0/1315599602 shutdown_connections 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.234+0000 7f0f57800700 1 -- 172.21.15.12:0/1315599602 >> 172.21.15.12:0/1315599602 conn(0x7f0f5008aff0 msgr2=0x7f0f500d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.234+0000 7f0f57800700 1 -- 172.21.15.12:0/1315599602 shutdown_connections 2024-03-29T11:11:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.235+0000 7f0f57800700 1 -- 172.21.15.12:0/1315599602 wait complete. 2024-03-29T11:11:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.236+0000 7f0f57800700 1 Processor -- start 2024-03-29T11:11:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.236+0000 7f0f57800700 1 -- start start 2024-03-29T11:11:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.237+0000 7f0f57800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f38003f70 0x7f0f5015b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.237+0000 7f0f57800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f500cc440 0x7f0f5015a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.237+0000 7f0f57800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015ad00 0x7f0f500c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.237+0000 7f0f57800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5015b180 con 0x7f0f500cc440 2024-03-29T11:11:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.237+0000 7f0f57800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f500cb3a0 con 0x7f0f38003f70 2024-03-29T11:11:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.237+0000 7f0f57800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f500cb680 con 0x7f0f5015ad00 2024-03-29T11:11:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f4fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f500cc440 0x7f0f5015a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f4fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f500cc440 0x7f0f5015a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38590/0 (socket says 172.21.15.12:38590) 2024-03-29T11:11:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f4fe00700 1 -- 172.21.15.12:0/844524068 learned_addr learned my addr 172.21.15.12:0/844524068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f54e00700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f38003f70 0x7f0f5015b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f55800700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015ad00 0x7f0f500c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f4fe00700 1 -- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015ad00 msgr2=0x7f0f500c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f4fe00700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015ad00 0x7f0f500c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f4fe00700 1 -- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f38003f70 msgr2=0x7f0f5015b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f4fe00700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f38003f70 0x7f0f5015b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f4fe00700 1 -- 172.21.15.12:0/844524068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f500cb960 con 0x7f0f500cc440 2024-03-29T11:11:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.238+0000 7f0f4fe00700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f500cc440 0x7f0f5015a7c0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c005280 tx=0x7f0f3c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.239+0000 7f0f4cc00700 1 -- 172.21.15.12:0/844524068 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c00cb60 con 0x7f0f500cc440 2024-03-29T11:11:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.239+0000 7f0f4cc00700 1 -- 172.21.15.12:0/844524068 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3c021070 con 0x7f0f500cc440 2024-03-29T11:11:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.239+0000 7f0f57800700 1 -- 172.21.15.12:0/844524068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f500cbc40 con 0x7f0f500cc440 2024-03-29T11:11:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.239+0000 7f0f4cc00700 1 -- 172.21.15.12:0/844524068 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c001330 con 0x7f0f500cc440 2024-03-29T11:11:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.239+0000 7f0f57800700 1 -- 172.21.15.12:0/844524068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5015bc30 con 0x7f0f500cc440 2024-03-29T11:11:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.240+0000 7f0f57800700 1 -- 172.21.15.12:0/844524068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f50067720 con 0x7f0f500cc440 2024-03-29T11:11:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.241+0000 7f0f4cc00700 1 -- 172.21.15.12:0/844524068 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0f3c023030 con 0x7f0f500cc440 2024-03-29T11:11:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.241+0000 7f0f4cc00700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f3006adb0 0x7f0f3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.242+0000 7f0f4cc00700 1 -- 172.21.15.12:0/844524068 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6061+0+0 (secure 0 0 0) 0x7f0f3c08a8f0 con 0x7f0f500cc440 2024-03-29T11:11:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.242+0000 7f0f54e00700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f3006adb0 0x7f0f3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.243+0000 7f0f54e00700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f3006adb0 0x7f0f3006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009c60 tx=0x7f0f44009210 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.246+0000 7f0f4cc00700 1 -- 172.21.15.12:0/844524068 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f3c055990 con 0x7f0f500cc440 2024-03-29T11:11:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:32.462+0000 7f0f57800700 1 -- 172.21.15.12:0/844524068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0f500b9530 con 0x7f0f500cc440 2024-03-29T11:11:33.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.105+0000 7f0f4cc00700 1 -- 172.21.15.12:0/844524068 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v168) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0f3c058fb0 con 0x7f0f500cc440 2024-03-29T11:11:33.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 -- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f3006adb0 msgr2=0x7f0f3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f3006adb0 0x7f0f3006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009c60 tx=0x7f0f44009210 comp rx=0 tx=0).stop 2024-03-29T11:11:33.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 -- 172.21.15.12:0/844524068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f500cc440 msgr2=0x7f0f5015a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f500cc440 0x7f0f5015a7c0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c005280 tx=0x7f0f3c001230 comp rx=0 tx=0).stop 2024-03-29T11:11:33.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 -- 172.21.15.12:0/844524068 shutdown_connections 2024-03-29T11:11:33.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f3006adb0 0x7f0f3006d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f5015ad00 0x7f0f500c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f38003f70 0x7f0f5015b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 --2- 172.21.15.12:0/844524068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f500cc440 0x7f0f5015a7c0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 -- 172.21.15.12:0/844524068 >> 172.21.15.12:0/844524068 conn(0x7f0f5008aff0 msgr2=0x7f0f5015cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:33.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 -- 172.21.15.12:0/844524068 shutdown_connections 2024-03-29T11:11:33.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.110+0000 7f0f57800700 1 -- 172.21.15.12:0/844524068 wait complete. 2024-03-29T11:11:33.089 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T11:11:33.110 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T11:11:33.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.334+0000 7f2d10000700 1 Processor -- start 2024-03-29T11:11:33.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.335+0000 7f2d10000700 1 -- start start 2024-03-29T11:11:33.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.336+0000 7f2d10000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d0815d9d0 0x7f2d080d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.336+0000 7f2d10000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d080d2540 0x7f2d080ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.336+0000 7f2d10000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d08161420 con 0x7f2d080d16a0 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.336+0000 7f2d10000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d0815edc0 con 0x7f2d0815d9d0 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.336+0000 7f2d10000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08161700 con 0x7f2d080d2540 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.336+0000 7f2d0d600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d0815d9d0 0x7f2d080d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.336+0000 7f2d0d600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d0815d9d0 0x7f2d080d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38606/0 (socket says 172.21.15.12:38606) 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.336+0000 7f2d0d600700 1 -- 172.21.15.12:0/3652893147 learned_addr learned my addr 172.21.15.12:0/3652893147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.336+0000 7f2d0e000700 1 --2- 172.21.15.12:0/3652893147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d080d2540 0x7f2d080ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.337+0000 7f2d0d600700 1 -- 172.21.15.12:0/3652893147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d080d2540 msgr2=0x7f2d080ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.337+0000 7f2d0d600700 1 --2- 172.21.15.12:0/3652893147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d080d2540 0x7f2d080ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.337+0000 7f2d0d600700 1 -- 172.21.15.12:0/3652893147 >> v1:172.21.15.12:6789/0 conn(0x7f2d080d16a0 legacy=0x7f2d080d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.337+0000 7f2d0d600700 1 -- 172.21.15.12:0/3652893147 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d081619e0 con 0x7f2d0815d9d0 2024-03-29T11:11:33.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.337+0000 7f2d0d600700 1 --2- 172.21.15.12:0/3652893147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0815d9d0 0x7f2d080d3c30 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0098f0 tx=0x7f2cfc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67a3c84cc515c68 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.337+0000 7f2d07e00700 1 -- 172.21.15.12:0/3652893147 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc00b8a0 con 0x7f2d0815d9d0 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.337+0000 7f2d07e00700 1 -- 172.21.15.12:0/3652893147 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc00ba00 con 0x7f2d0815d9d0 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.337+0000 7f2d07e00700 1 -- 172.21.15.12:0/3652893147 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc00bd90 con 0x7f2d0815d9d0 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.338+0000 7f2d10000700 1 -- 172.21.15.12:0/3652893147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0815d9d0 msgr2=0x7f2d080d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.338+0000 7f2d10000700 1 --2- 172.21.15.12:0/3652893147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0815d9d0 0x7f2d080d3c30 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0098f0 tx=0x7f2cfc009c00 comp rx=0 tx=0).stop 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.338+0000 7f2d10000700 1 -- 172.21.15.12:0/3652893147 shutdown_connections 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.338+0000 7f2d10000700 1 --2- 172.21.15.12:0/3652893147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d080d2540 0x7f2d080ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.338+0000 7f2d10000700 1 --2- 172.21.15.12:0/3652893147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0815d9d0 0x7f2d080d3c30 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.338+0000 7f2d10000700 1 -- 172.21.15.12:0/3652893147 >> 172.21.15.12:0/3652893147 conn(0x7f2d0808aff0 msgr2=0x7f2d080d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.338+0000 7f2d10000700 1 -- 172.21.15.12:0/3652893147 shutdown_connections 2024-03-29T11:11:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.338+0000 7f2d10000700 1 -- 172.21.15.12:0/3652893147 wait complete. 2024-03-29T11:11:33.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.340+0000 7f2d10000700 1 Processor -- start 2024-03-29T11:11:33.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.340+0000 7f2d10000700 1 -- start start 2024-03-29T11:11:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d10000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d080d16a0 0x7f2d081e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d10000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080d2540 0x7f2d0814f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d10000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d0815d9d0 0x7f2d080b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d10000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080cb5a0 con 0x7f2d080d16a0 2024-03-29T11:11:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d10000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d081e82e0 con 0x7f2d080d2540 2024-03-29T11:11:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d10000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d081e85c0 con 0x7f2d0815d9d0 2024-03-29T11:11:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d080d16a0 0x7f2d081e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d080d16a0 0x7f2d081e7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38620/0 (socket says 172.21.15.12:38620) 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0d600700 1 -- 172.21.15.12:0/2706994589 learned_addr learned my addr 172.21.15.12:0/2706994589 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0e000700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d0815d9d0 0x7f2d080b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0cc00700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080d2540 0x7f2d0814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0d600700 1 -- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d0815d9d0 msgr2=0x7f2d080b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0d600700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d0815d9d0 0x7f2d080b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0d600700 1 -- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080d2540 msgr2=0x7f2d0814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0d600700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080d2540 0x7f2d0814f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0d600700 1 -- 172.21.15.12:0/2706994589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc009550 con 0x7f2d080d16a0 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.341+0000 7f2d0cc00700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080d2540 0x7f2d0814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.342+0000 7f2d0d600700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d080d16a0 0x7f2d081e7c90 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009c40 tx=0x7f2cfc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.342+0000 7f2d05600700 1 -- 172.21.15.12:0/2706994589 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc0036a0 con 0x7f2d080d16a0 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.342+0000 7f2d05600700 1 -- 172.21.15.12:0/2706994589 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc01e070 con 0x7f2d080d16a0 2024-03-29T11:11:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.342+0000 7f2d10000700 1 -- 172.21.15.12:0/2706994589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d081e88a0 con 0x7f2d080d16a0 2024-03-29T11:11:33.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.342+0000 7f2d05600700 1 -- 172.21.15.12:0/2706994589 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc003a60 con 0x7f2d080d16a0 2024-03-29T11:11:33.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.342+0000 7f2d10000700 1 -- 172.21.15.12:0/2706994589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0814fd90 con 0x7f2d080d16a0 2024-03-29T11:11:33.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.343+0000 7f2d10000700 1 -- 172.21.15.12:0/2706994589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d080d1ca0 con 0x7f2d080d16a0 2024-03-29T11:11:33.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.344+0000 7f2d05600700 1 -- 172.21.15.12:0/2706994589 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2cfc003bc0 con 0x7f2d080d16a0 2024-03-29T11:11:33.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.345+0000 7f2d05600700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ce806adb0 0x7f2ce806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.345+0000 7f2d05600700 1 -- 172.21.15.12:0/2706994589 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f2cfc020070 con 0x7f2d080d16a0 2024-03-29T11:11:33.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.345+0000 7f2d0cc00700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ce806adb0 0x7f2ce806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.346+0000 7f2d0cc00700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ce806adb0 0x7f2ce806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2d0815c930 tx=0x7f2cf4009040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:33.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.349+0000 7f2d05600700 1 -- 172.21.15.12:0/2706994589 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2cfc055a60 con 0x7f2d080d16a0 2024-03-29T11:11:33.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.565+0000 7f2d10000700 1 -- 172.21.15.12:0/2706994589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f2d080cb3e0 con 0x7f2d080d16a0 2024-03-29T11:11:33.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.569+0000 7f2d05600700 1 -- 172.21.15.12:0/2706994589 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v55) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2d080cb3e0 con 0x7f2d080d16a0 2024-03-29T11:11:33.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 -- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ce806adb0 msgr2=0x7f2ce806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ce806adb0 0x7f2ce806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2d0815c930 tx=0x7f2cf4009040 comp rx=0 tx=0).stop 2024-03-29T11:11:33.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 -- 172.21.15.12:0/2706994589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d080d16a0 msgr2=0x7f2d081e7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d080d16a0 0x7f2d081e7c90 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009c40 tx=0x7f2cfc004750 comp rx=0 tx=0).stop 2024-03-29T11:11:33.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 -- 172.21.15.12:0/2706994589 shutdown_connections 2024-03-29T11:11:33.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2ce806adb0 0x7f2ce806d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d0815d9d0 0x7f2d080b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d080d16a0 0x7f2d081e7c90 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 --2- 172.21.15.12:0/2706994589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080d2540 0x7f2d0814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 -- 172.21.15.12:0/2706994589 >> 172.21.15.12:0/2706994589 conn(0x7f2d0808aff0 msgr2=0x7f2d080b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:33.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 -- 172.21.15.12:0/2706994589 shutdown_connections 2024-03-29T11:11:33.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.572+0000 7f2d10000700 1 -- 172.21.15.12:0/2706994589 wait complete. 2024-03-29T11:11:33.568 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T11:11:33.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.789+0000 7f4e11600700 1 Processor -- start 2024-03-29T11:11:33.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.789+0000 7f4e11600700 1 -- start start 2024-03-29T11:11:33.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.791+0000 7f4e11600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4e0c0d2540 0x7f4e0c1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.791+0000 7f4e11600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c161420 con 0x7f4e0c0d16a0 2024-03-29T11:11:33.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.791+0000 7f4e11600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c161700 con 0x7f4e0c15d9d0 2024-03-29T11:11:33.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.791+0000 7f4e11600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0c0d4340 con 0x7f4e0c0d2540 2024-03-29T11:11:33.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.791+0000 7f4e0a000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4e0c0d16a0 0x7f4e0c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35744/0 (socket says 172.21.15.12:35744) 2024-03-29T11:11:33.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.791+0000 7f4e0a000700 1 -- 172.21.15.12:0/737403321 learned_addr learned my addr 172.21.15.12:0/737403321 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.791+0000 7f4e0b400700 1 --2- 172.21.15.12:0/737403321 >> v2:172.21.15.159:3300/0 conn(0x7f4e0c0d2540 0x7f4e0c1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.791+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540693124 0 0) 0x7f4e0c161420 con 0x7f4e0c0d16a0 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.791+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df00035b0 con 0x7f4e0c0d16a0 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928478364 0 0) 0x7f4df00035b0 con 0x7f4e0c0d16a0 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 >> v2:172.21.15.159:3300/0 conn(0x7f4e0c0d2540 msgr2=0x7f4e0c1687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 --2- 172.21.15.12:0/737403321 >> v2:172.21.15.159:3300/0 conn(0x7f4e0c0d2540 0x7f4e0c1687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 >> v1:172.21.15.159:6789/0 conn(0x7f4e0c15d9d0 legacy=0x7f4e0c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c1619e0 con 0x7f4e0c0d16a0 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f4df80017f0 con 0x7f4e0c0d16a0 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 >> v1:172.21.15.12:6789/0 conn(0x7f4e0c0d16a0 legacy=0x7f4e0c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df0003a50 0x7f4df0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4df0004850 0x7f4df0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4df0005260 0x7f4df000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df000c180 con 0x7f4df0003a50 2024-03-29T11:11:33.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df000c2f0 con 0x7f4df0005260 2024-03-29T11:11:33.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df000c4b0 con 0x7f4df0004850 2024-03-29T11:11:33.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e0aa00700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df0003a50 0x7f4df0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e0b400700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4df0005260 0x7f4df000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e0a000700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4df0004850 0x7f4df0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e0aa00700 1 -- 172.21.15.12:0/737403321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4df0004850 msgr2=0x7f4df0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:33.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e0aa00700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4df0004850 0x7f4df0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e0aa00700 1 -- 172.21.15.12:0/737403321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4df0005260 msgr2=0x7f4df000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:33.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e0aa00700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4df0005260 0x7f4df000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e0aa00700 1 -- 172.21.15.12:0/737403321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df00035b0 con 0x7f4df0003a50 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.792+0000 7f4e0b400700 1 -- 172.21.15.12:0/737403321 reap_dead start 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.793+0000 7f4e0aa00700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df0003a50 0x7f4df0003e70 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc007fd0 tx=0x7f4df0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9bc04b006c2ca4b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.793+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc0052d0 con 0x7f4df0003a50 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.793+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc005450 con 0x7f4df0003a50 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.793+0000 7f4e09600700 1 -- 172.21.15.12:0/737403321 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc0038c0 con 0x7f4df0003a50 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.793+0000 7f4e11600700 1 -- 172.21.15.12:0/737403321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df0003a50 msgr2=0x7f4df0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.793+0000 7f4e11600700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df0003a50 0x7f4df0003e70 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc007fd0 tx=0x7f4df0005880 comp rx=0 tx=0).stop 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.794+0000 7f4e11600700 1 -- 172.21.15.12:0/737403321 shutdown_connections 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.794+0000 7f4e11600700 1 --2- 172.21.15.12:0/737403321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df0003a50 0x7f4df0003e70 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.794+0000 7f4e11600700 1 -- 172.21.15.12:0/737403321 >> 172.21.15.12:0/737403321 conn(0x7f4e0c08aff0 msgr2=0x7f4e0c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.794+0000 7f4e11600700 1 -- 172.21.15.12:0/737403321 shutdown_connections 2024-03-29T11:11:33.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.794+0000 7f4e11600700 1 -- 172.21.15.12:0/737403321 wait complete. 2024-03-29T11:11:33.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.796+0000 7f4e11600700 1 Processor -- start 2024-03-29T11:11:33.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.796+0000 7f4e11600700 1 -- start start 2024-03-29T11:11:33.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.797+0000 7f4e11600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4df0003a50 0x7f4e0c0b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.797+0000 7f4e11600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e0c15f490 0x7f4e0c15f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e11600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c1555b0 0x7f4e0c155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e11600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c160bb0 con 0x7f4e0c1555b0 2024-03-29T11:11:33.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e11600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c155f70 con 0x7f4e0c15f490 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e11600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c15ff30 con 0x7f4df0003a50 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e0b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c1555b0 0x7f4e0c155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e0b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c1555b0 0x7f4e0c155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38640/0 (socket says 172.21.15.12:38640) 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e0b400700 1 -- 172.21.15.12:0/421523535 learned_addr learned my addr 172.21.15.12:0/421523535 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e0a000700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e0c15f490 0x7f4e0c15f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e0aa00700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4df0003a50 0x7f4e0c0b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e0b400700 1 -- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4df0003a50 msgr2=0x7f4e0c0b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e0b400700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4df0003a50 0x7f4e0c0b4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e0b400700 1 -- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e0c15f490 msgr2=0x7f4e0c15f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e0b400700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e0c15f490 0x7f4e0c15f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.798+0000 7f4e0b400700 1 -- 172.21.15.12:0/421523535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c160210 con 0x7f4e0c1555b0 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.799+0000 7f4e0b400700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c1555b0 0x7f4e0c155a30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0d2f40 tx=0x7f4e04011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.799+0000 7f4e02a00700 1 -- 172.21.15.12:0/421523535 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0400e040 con 0x7f4e0c1555b0 2024-03-29T11:11:33.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.799+0000 7f4e02a00700 1 -- 172.21.15.12:0/421523535 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0400f040 con 0x7f4e0c1555b0 2024-03-29T11:11:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.799+0000 7f4e02a00700 1 -- 172.21.15.12:0/421523535 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0400fbe0 con 0x7f4e0c1555b0 2024-03-29T11:11:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.799+0000 7f4e11600700 1 -- 172.21.15.12:0/421523535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0c1604f0 con 0x7f4e0c1555b0 2024-03-29T11:11:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.799+0000 7f4e11600700 1 -- 172.21.15.12:0/421523535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0c0b62c0 con 0x7f4e0c1555b0 2024-03-29T11:11:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.800+0000 7f4e11600700 1 -- 172.21.15.12:0/421523535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0c0b5b60 con 0x7f4e0c1555b0 2024-03-29T11:11:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.801+0000 7f4e02a00700 1 -- 172.21.15.12:0/421523535 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4e0400fd40 con 0x7f4e0c1555b0 2024-03-29T11:11:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.802+0000 7f4e02a00700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4dec06ade0 0x7f4dec06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.802+0000 7f4e02a00700 1 -- 172.21.15.12:0/421523535 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f4e04089a20 con 0x7f4e0c1555b0 2024-03-29T11:11:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.802+0000 7f4e0aa00700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4dec06ade0 0x7f4dec06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.803+0000 7f4e0aa00700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4dec06ade0 0x7f4dec06d2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc005040 tx=0x7f4dfc009030 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:33.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:33.806+0000 7f4e02a00700 1 -- 172.21.15.12:0/421523535 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e04054a40 con 0x7f4e0c1555b0 2024-03-29T11:11:34.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.025+0000 7f4e11600700 1 -- 172.21.15.12:0/421523535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4e0c067720 con 0x7f4e0c1555b0 2024-03-29T11:11:34.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.029+0000 7f4e02a00700 1 -- 172.21.15.12:0/421523535 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4e04058060 con 0x7f4e0c1555b0 2024-03-29T11:11:34.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.031+0000 7f4e11600700 1 -- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4dec06ade0 msgr2=0x7f4dec06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.031+0000 7f4e11600700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4dec06ade0 0x7f4dec06d2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc005040 tx=0x7f4dfc009030 comp rx=0 tx=0).stop 2024-03-29T11:11:34.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.031+0000 7f4e11600700 1 -- 172.21.15.12:0/421523535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c1555b0 msgr2=0x7f4e0c155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.031+0000 7f4e11600700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c1555b0 0x7f4e0c155a30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0d2f40 tx=0x7f4e04011ed0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.032+0000 7f4e11600700 1 -- 172.21.15.12:0/421523535 shutdown_connections 2024-03-29T11:11:34.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.032+0000 7f4e11600700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4dec06ade0 0x7f4dec06d2a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.032+0000 7f4e11600700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c1555b0 0x7f4e0c155a30 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.032+0000 7f4e11600700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4df0003a50 0x7f4e0c0b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.032+0000 7f4e11600700 1 --2- 172.21.15.12:0/421523535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e0c15f490 0x7f4e0c15f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.032+0000 7f4e11600700 1 -- 172.21.15.12:0/421523535 >> 172.21.15.12:0/421523535 conn(0x7f4e0c08aff0 msgr2=0x7f4e0c0b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.032+0000 7f4e11600700 1 -- 172.21.15.12:0/421523535 shutdown_connections 2024-03-29T11:11:34.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.032+0000 7f4e11600700 1 -- 172.21.15.12:0/421523535 wait complete. 2024-03-29T11:11:34.029 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T11:11:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.255+0000 7f908ba00700 1 Processor -- start 2024-03-29T11:11:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.255+0000 7f908ba00700 1 -- start start 2024-03-29T11:11:34.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.256+0000 7f908ba00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f90840ce840 0x7f9084165370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.256+0000 7f908ba00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840cb310 con 0x7f908415a5c0 2024-03-29T11:11:34.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.256+0000 7f908ba00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840cca90 con 0x7f908415dfe0 2024-03-29T11:11:34.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.256+0000 7f908ba00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90840711a0 con 0x7f90840ce840 2024-03-29T11:11:34.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9089000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f908415a5c0 0x7f90840709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35754/0 (socket says 172.21.15.12:35754) 2024-03-29T11:11:34.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9089000700 1 -- 172.21.15.12:0/4069526537 learned_addr learned my addr 172.21.15.12:0/4069526537 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9089a00700 1 --2- 172.21.15.12:0/4069526537 >> v2:172.21.15.159:3300/0 conn(0x7f90840ce840 0x7f9084165370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409803630 0 0) 0x7f90840cb310 con 0x7f908415a5c0 2024-03-29T11:11:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f906c0057c0 con 0x7f908415a5c0 2024-03-29T11:11:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1470910499 0 0) 0x7f906c0057c0 con 0x7f908415a5c0 2024-03-29T11:11:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 >> v2:172.21.15.159:3300/0 conn(0x7f90840ce840 msgr2=0x7f9084165370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9083400700 1 --2- 172.21.15.12:0/4069526537 >> v2:172.21.15.159:3300/0 conn(0x7f90840ce840 0x7f9084165370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 >> v1:172.21.15.159:6789/0 conn(0x7f908415dfe0 legacy=0x7f90840ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9084071480 con 0x7f908415a5c0 2024-03-29T11:11:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f9078001670 con 0x7f908415a5c0 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 >> v1:172.21.15.12:6789/0 conn(0x7f908415a5c0 legacy=0x7f90840709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.257+0000 7f9083400700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f906c005c60 0x7f906c006080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9083400700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f906c006a60 0x7f906c006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9083400700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f906c007420 0x7f906c00dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90840711a0 con 0x7f906c005c60 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c00e320 con 0x7f906c006a60 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906c00e4e0 con 0x7f906c007420 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9089000700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f906c005c60 0x7f906c006080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9083e00700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f906c006a60 0x7f906c006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9089a00700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f906c007420 0x7f906c00dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9089000700 1 -- 172.21.15.12:0/4069526537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f906c007420 msgr2=0x7f906c00dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9089000700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f906c007420 0x7f906c00dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9089000700 1 -- 172.21.15.12:0/4069526537 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f906c006a60 msgr2=0x7f906c006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9089000700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f906c006a60 0x7f906c006ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9089000700 1 -- 172.21.15.12:0/4069526537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906c0057c0 con 0x7f906c005c60 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9089a00700 1 -- 172.21.15.12:0/4069526537 reap_dead start 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.258+0000 7f9089000700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f906c005c60 0x7f906c006080 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f907800c160 tx=0x7f906c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a6e1a24fb5c9a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.259+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9078007b70 con 0x7f906c005c60 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.259+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907800cd20 con 0x7f906c005c60 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.259+0000 7f9083400700 1 -- 172.21.15.12:0/4069526537 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9078005ba0 con 0x7f906c005c60 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.259+0000 7f908ba00700 1 -- 172.21.15.12:0/4069526537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f906c005c60 msgr2=0x7f906c006080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.259+0000 7f908ba00700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f906c005c60 0x7f906c006080 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f907800c160 tx=0x7f906c007a90 comp rx=0 tx=0).stop 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.259+0000 7f908ba00700 1 -- 172.21.15.12:0/4069526537 shutdown_connections 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.259+0000 7f908ba00700 1 --2- 172.21.15.12:0/4069526537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f906c005c60 0x7f906c006080 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.259+0000 7f908ba00700 1 -- 172.21.15.12:0/4069526537 >> 172.21.15.12:0/4069526537 conn(0x7f90840b0d70 msgr2=0x7f90840b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.259+0000 7f908ba00700 1 -- 172.21.15.12:0/4069526537 shutdown_connections 2024-03-29T11:11:34.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.259+0000 7f908ba00700 1 -- 172.21.15.12:0/4069526537 wait complete. 2024-03-29T11:11:34.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.261+0000 7f908ba00700 1 Processor -- start 2024-03-29T11:11:34.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.261+0000 7f908ba00700 1 -- start start 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.262+0000 7f908ba00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f906c005c60 0x7f9084159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.262+0000 7f908ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f908415d240 0x7f908414f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.262+0000 7f908ba00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908414d630 0x7f908414dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.262+0000 7f908ba00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908414e020 con 0x7f908414d630 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.262+0000 7f908ba00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908415d9e0 con 0x7f906c005c60 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.262+0000 7f908ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90840c95a0 con 0x7f908415d240 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9089a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908414d630 0x7f908414dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9089a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908414d630 0x7f908414dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38644/0 (socket says 172.21.15.12:38644) 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9089a00700 1 -- 172.21.15.12:0/2370519032 learned_addr learned my addr 172.21.15.12:0/2370519032 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9089000700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f906c005c60 0x7f9084159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9083e00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f908415d240 0x7f908414f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9089a00700 1 -- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f908415d240 msgr2=0x7f908414f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9089a00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f908415d240 0x7f908414f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9089a00700 1 -- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f906c005c60 msgr2=0x7f9084159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9089a00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f906c005c60 0x7f9084159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9089a00700 1 -- 172.21.15.12:0/2370519032 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908415d700 con 0x7f908414d630 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.263+0000 7f9089a00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908414d630 0x7f908414dae0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f908406b650 tx=0x7f907c0111d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.264+0000 7f9080c00700 1 -- 172.21.15.12:0/2370519032 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c00e070 con 0x7f908414d630 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.264+0000 7f9080c00700 1 -- 172.21.15.12:0/2370519032 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c003cf0 con 0x7f908414d630 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.264+0000 7f908ba00700 1 -- 172.21.15.12:0/2370519032 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90840c9840 con 0x7f908414d630 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.264+0000 7f9080c00700 1 -- 172.21.15.12:0/2370519032 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c003ea0 con 0x7f908414d630 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.264+0000 7f908ba00700 1 -- 172.21.15.12:0/2370519032 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908415acc0 con 0x7f908414d630 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.265+0000 7f908ba00700 1 -- 172.21.15.12:0/2370519032 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90840c9d60 con 0x7f908414d630 2024-03-29T11:11:34.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.266+0000 7f9080c00700 1 -- 172.21.15.12:0/2370519032 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f907c011770 con 0x7f908414d630 2024-03-29T11:11:34.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.266+0000 7f9080c00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f906406adb0 0x7f906406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.267+0000 7f9080c00700 1 -- 172.21.15.12:0/2370519032 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f907c089960 con 0x7f908414d630 2024-03-29T11:11:34.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.267+0000 7f9089000700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f906406adb0 0x7f906406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.268+0000 7f9089000700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f906406adb0 0x7f906406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9078000c00 tx=0x7f9078007000 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:34.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.271+0000 7f9080c00700 1 -- 172.21.15.12:0/2370519032 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90840c9d60 con 0x7f908414d630 2024-03-29T11:11:34.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.484+0000 7f908ba00700 1 -- 172.21.15.12:0/2370519032 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f90840c9d60 con 0x7f908414d630 2024-03-29T11:11:34.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.488+0000 7f9080c00700 1 -- 172.21.15.12:0/2370519032 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v57) v1 ==== 120+0+0 (secure 0 0 0) 0x7f90840c9d60 con 0x7f908414d630 2024-03-29T11:11:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.490+0000 7f908ba00700 1 -- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f906406adb0 msgr2=0x7f906406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.490+0000 7f908ba00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f906406adb0 0x7f906406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9078000c00 tx=0x7f9078007000 comp rx=0 tx=0).stop 2024-03-29T11:11:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.490+0000 7f908ba00700 1 -- 172.21.15.12:0/2370519032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908414d630 msgr2=0x7f908414dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.490+0000 7f908ba00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908414d630 0x7f908414dae0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f908406b650 tx=0x7f907c0111d0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.491+0000 7f908ba00700 1 -- 172.21.15.12:0/2370519032 shutdown_connections 2024-03-29T11:11:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.491+0000 7f908ba00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f906406adb0 0x7f906406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.491+0000 7f908ba00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908414d630 0x7f908414dae0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.491+0000 7f908ba00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f906c005c60 0x7f9084159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.491+0000 7f908ba00700 1 --2- 172.21.15.12:0/2370519032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f908415d240 0x7f908414f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.491+0000 7f908ba00700 1 -- 172.21.15.12:0/2370519032 >> 172.21.15.12:0/2370519032 conn(0x7f90840b0d70 msgr2=0x7f908406df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.491+0000 7f908ba00700 1 -- 172.21.15.12:0/2370519032 shutdown_connections 2024-03-29T11:11:34.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.491+0000 7f908ba00700 1 -- 172.21.15.12:0/2370519032 wait complete. 2024-03-29T11:11:34.487 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-03-29T11:11:34.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.707+0000 7faf75e00700 1 Processor -- start 2024-03-29T11:11:34.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.707+0000 7faf75e00700 1 -- start start 2024-03-29T11:11:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.708+0000 7faf75e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faf7015d640 0x7faf700d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.708+0000 7faf75e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf700ceec0 0x7faf701612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.708+0000 7faf75e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf700d30d0 con 0x7faf70161800 2024-03-29T11:11:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.708+0000 7faf75e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf70161c90 con 0x7faf7015d640 2024-03-29T11:11:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.708+0000 7faf75e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf700b6840 con 0x7faf700ceec0 2024-03-29T11:11:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.708+0000 7faf6ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faf7015d640 0x7faf700d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.708+0000 7faf6ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faf7015d640 0x7faf700d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38656/0 (socket says 172.21.15.12:38656) 2024-03-29T11:11:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.708+0000 7faf6ea00700 1 -- 172.21.15.12:0/2531068009 learned_addr learned my addr 172.21.15.12:0/2531068009 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.708+0000 7faf6e000700 1 --2- 172.21.15.12:0/2531068009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf700ceec0 0x7faf701612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.709+0000 7faf6ea00700 1 -- 172.21.15.12:0/2531068009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf700ceec0 msgr2=0x7faf701612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.709+0000 7faf6ea00700 1 --2- 172.21.15.12:0/2531068009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf700ceec0 0x7faf701612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.709+0000 7faf6ea00700 1 -- 172.21.15.12:0/2531068009 >> v1:172.21.15.159:6789/0 conn(0x7faf70161800 legacy=0x7faf700cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.709+0000 7faf6ea00700 1 -- 172.21.15.12:0/2531068009 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700d2880 con 0x7faf7015d640 2024-03-29T11:11:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.709+0000 7faf6ea00700 1 --2- 172.21.15.12:0/2531068009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf7015d640 0x7faf700d2340 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7faf600098f0 tx=0x7faf60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1343c4a5412b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.709+0000 7faf6d600700 1 -- 172.21.15.12:0/2531068009 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60011830 con 0x7faf7015d640 2024-03-29T11:11:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.709+0000 7faf6d600700 1 -- 172.21.15.12:0/2531068009 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf60011990 con 0x7faf7015d640 2024-03-29T11:11:34.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.709+0000 7faf6d600700 1 -- 172.21.15.12:0/2531068009 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60011d20 con 0x7faf7015d640 2024-03-29T11:11:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.710+0000 7faf75e00700 1 -- 172.21.15.12:0/2531068009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf7015d640 msgr2=0x7faf700d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.710+0000 7faf75e00700 1 --2- 172.21.15.12:0/2531068009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf7015d640 0x7faf700d2340 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7faf600098f0 tx=0x7faf60009c00 comp rx=0 tx=0).stop 2024-03-29T11:11:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.710+0000 7faf75e00700 1 -- 172.21.15.12:0/2531068009 shutdown_connections 2024-03-29T11:11:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.710+0000 7faf75e00700 1 --2- 172.21.15.12:0/2531068009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf7015d640 0x7faf700d2340 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.710+0000 7faf75e00700 1 --2- 172.21.15.12:0/2531068009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf700ceec0 0x7faf701612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.710+0000 7faf75e00700 1 -- 172.21.15.12:0/2531068009 >> 172.21.15.12:0/2531068009 conn(0x7faf7008aff0 msgr2=0x7faf700d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.710+0000 7faf75e00700 1 -- 172.21.15.12:0/2531068009 shutdown_connections 2024-03-29T11:11:34.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.710+0000 7faf75e00700 1 -- 172.21.15.12:0/2531068009 wait complete. 2024-03-29T11:11:34.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.712+0000 7faf75e00700 1 Processor -- start 2024-03-29T11:11:34.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.712+0000 7faf75e00700 1 -- start start 2024-03-29T11:11:34.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf75e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf700ceec0 0x7faf701410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf75e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf7015d640 0x7faf70156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf75e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf70161800 0x7faf70146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf75e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf701566c0 con 0x7faf700ceec0 2024-03-29T11:11:34.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf75e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf70146fd0 con 0x7faf70161800 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf75e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf70159050 con 0x7faf7015d640 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf700ceec0 0x7faf701410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf700ceec0 0x7faf701410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38670/0 (socket says 172.21.15.12:38670) 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6ea00700 1 -- 172.21.15.12:0/319436057 learned_addr learned my addr 172.21.15.12:0/319436057 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6e000700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf7015d640 0x7faf70156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6f400700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf70161800 0x7faf70146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6ea00700 1 -- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf7015d640 msgr2=0x7faf70156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6ea00700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf7015d640 0x7faf70156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6ea00700 1 -- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf70161800 msgr2=0x7faf70146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6ea00700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf70161800 0x7faf70146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6ea00700 1 -- 172.21.15.12:0/319436057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf60009550 con 0x7faf700ceec0 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6e000700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf7015d640 0x7faf70156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.713+0000 7faf6f400700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf70161800 0x7faf70146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:34.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.714+0000 7faf6ea00700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf700ceec0 0x7faf701410e0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7faf7008c490 tx=0x7faf60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.714+0000 7faf67400700 1 -- 172.21.15.12:0/319436057 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60004530 con 0x7faf700ceec0 2024-03-29T11:11:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.714+0000 7faf67400700 1 -- 172.21.15.12:0/319436057 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6001f070 con 0x7faf700ceec0 2024-03-29T11:11:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.714+0000 7faf75e00700 1 -- 172.21.15.12:0/319436057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf70158d70 con 0x7faf700ceec0 2024-03-29T11:11:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.714+0000 7faf67400700 1 -- 172.21.15.12:0/319436057 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf600038c0 con 0x7faf700ceec0 2024-03-29T11:11:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.714+0000 7faf75e00700 1 -- 172.21.15.12:0/319436057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7015f180 con 0x7faf700ceec0 2024-03-29T11:11:34.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.715+0000 7faf75e00700 1 -- 172.21.15.12:0/319436057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf70067720 con 0x7faf700ceec0 2024-03-29T11:11:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.716+0000 7faf67400700 1 -- 172.21.15.12:0/319436057 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7faf60021030 con 0x7faf700ceec0 2024-03-29T11:11:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.717+0000 7faf67400700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faf5006adb0 0x7faf5006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.717+0000 7faf67400700 1 -- 172.21.15.12:0/319436057 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5589+0+0 (secure 0 0 0) 0x7faf6008a850 con 0x7faf700ceec0 2024-03-29T11:11:34.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.717+0000 7faf6e000700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faf5006adb0 0x7faf5006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:34.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.718+0000 7faf6e000700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faf5006adb0 0x7faf5006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faf70160e00 tx=0x7faf5c008040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:34.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:34.721+0000 7faf67400700 1 -- 172.21.15.12:0/319436057 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf60055980 con 0x7faf700ceec0 2024-03-29T11:11:35.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.024+0000 7faf75e00700 1 -- 172.21.15.12:0/319436057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7faf700b5c70 con 0x7faf700ceec0 2024-03-29T11:11:35.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.108+0000 7faf67400700 1 -- 172.21.15.12:0/319436057 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2504) v1 ==== 178+0+0 (secure 0 0 0) 0x7faf600150f0 con 0x7faf700ceec0 2024-03-29T11:11:35.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 -- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faf5006adb0 msgr2=0x7faf5006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faf5006adb0 0x7faf5006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faf70160e00 tx=0x7faf5c008040 comp rx=0 tx=0).stop 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 -- 172.21.15.12:0/319436057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf700ceec0 msgr2=0x7faf701410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf700ceec0 0x7faf701410e0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7faf7008c490 tx=0x7faf60004750 comp rx=0 tx=0).stop 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 -- 172.21.15.12:0/319436057 shutdown_connections 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7faf5006adb0 0x7faf5006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf70161800 0x7faf70146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf700ceec0 0x7faf701410e0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 --2- 172.21.15.12:0/319436057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf7015d640 0x7faf70156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 -- 172.21.15.12:0/319436057 >> 172.21.15.12:0/319436057 conn(0x7faf7008aff0 msgr2=0x7faf700d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 -- 172.21.15.12:0/319436057 shutdown_connections 2024-03-29T11:11:35.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.110+0000 7faf75e00700 1 -- 172.21.15.12:0/319436057 wait complete. 2024-03-29T11:11:35.106 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2024-03-29T11:11:35.107 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2024-03-29T11:11:35.107 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T11:11:35.107 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-03-29T11:11:35.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.326+0000 7f22a4200700 1 Processor -- start 2024-03-29T11:11:35.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.326+0000 7f22a4200700 1 -- start start 2024-03-29T11:11:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.327+0000 7f22a4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c160f10 0x7f229c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.327+0000 7f22a4200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f229c161590 0x7f229c161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.327+0000 7f22a4200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c0ceda0 con 0x7f229c0cf230 2024-03-29T11:11:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.327+0000 7f22a4200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229c161f80 con 0x7f229c161590 2024-03-29T11:11:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.327+0000 7f22a4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c1612f0 con 0x7f229c160f10 2024-03-29T11:11:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.328+0000 7f22a1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c160f10 0x7f229c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.328+0000 7f22a2200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f229c161590 0x7f229c161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.328+0000 7f22a2200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f229c161590 0x7f229c161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:53750/0 (socket says 172.21.15.12:53750) 2024-03-29T11:11:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.328+0000 7f22a1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c160f10 0x7f229c0d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:41966/0 (socket says 172.21.15.12:41966) 2024-03-29T11:11:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.328+0000 7f22a2200700 1 -- 172.21.15.12:0/4074238740 learned_addr learned my addr 172.21.15.12:0/4074238740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:35.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.328+0000 7f22a1800700 1 -- 172.21.15.12:0/4074238740 >> v2:172.21.15.159:3300/0 conn(0x7f229c161590 msgr2=0x7f229c161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:35.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.328+0000 7f22a1800700 1 --2- 172.21.15.12:0/4074238740 >> v2:172.21.15.159:3300/0 conn(0x7f229c161590 0x7f229c161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:35.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.328+0000 7f22a1800700 1 -- 172.21.15.12:0/4074238740 >> v1:172.21.15.159:6789/0 conn(0x7f229c0cf230 legacy=0x7f229c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.328+0000 7f22a1800700 1 -- 172.21.15.12:0/4074238740 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c0ce470 con 0x7f229c160f10 2024-03-29T11:11:35.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.328+0000 7f22a2200700 1 --2- 172.21.15.12:0/4074238740 >> v2:172.21.15.159:3300/0 conn(0x7f229c161590 0x7f229c161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:35.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.329+0000 7f22a1800700 1 --2- 172.21.15.12:0/4074238740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c160f10 0x7f229c0d18b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2290009870 tx=0x7f2290009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8acf7206478e21c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:35.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.329+0000 7f229be00700 1 -- 172.21.15.12:0/4074238740 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290011830 con 0x7f229c160f10 2024-03-29T11:11:35.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.329+0000 7f229be00700 1 -- 172.21.15.12:0/4074238740 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2290011990 con 0x7f229c160f10 2024-03-29T11:11:35.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.329+0000 7f229be00700 1 -- 172.21.15.12:0/4074238740 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290011d20 con 0x7f229c160f10 2024-03-29T11:11:35.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.330+0000 7f22a4200700 1 -- 172.21.15.12:0/4074238740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c160f10 msgr2=0x7f229c0d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.330+0000 7f22a4200700 1 --2- 172.21.15.12:0/4074238740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c160f10 0x7f229c0d18b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2290009870 tx=0x7f2290009b80 comp rx=0 tx=0).stop 2024-03-29T11:11:35.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.330+0000 7f22a4200700 1 -- 172.21.15.12:0/4074238740 shutdown_connections 2024-03-29T11:11:35.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.330+0000 7f22a4200700 1 --2- 172.21.15.12:0/4074238740 >> v2:172.21.15.159:3300/0 conn(0x7f229c161590 0x7f229c161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:35.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.330+0000 7f22a4200700 1 --2- 172.21.15.12:0/4074238740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c160f10 0x7f229c0d18b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:35.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.330+0000 7f22a4200700 1 -- 172.21.15.12:0/4074238740 >> 172.21.15.12:0/4074238740 conn(0x7f229c08aff0 msgr2=0x7f229c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.330+0000 7f22a4200700 1 -- 172.21.15.12:0/4074238740 shutdown_connections 2024-03-29T11:11:35.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.330+0000 7f22a4200700 1 -- 172.21.15.12:0/4074238740 wait complete. 2024-03-29T11:11:35.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.331+0000 7f22a4200700 1 Processor -- start 2024-03-29T11:11:35.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.332+0000 7f22a4200700 1 -- start start 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.332+0000 7f22a4200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229c0cf230 0x7f229c155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.332+0000 7f22a4200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f229c160f10 0x7f229c0b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.332+0000 7f22a4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c161590 0x7f229c0b86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a4200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c156520 con 0x7f229c0cf230 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a4200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c1567c0 con 0x7f229c160f10 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c156aa0 con 0x7f229c161590 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229c0cf230 0x7f229c155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229c0cf230 0x7f229c155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38682/0 (socket says 172.21.15.12:38682) 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a1800700 1 -- 172.21.15.12:0/3800879293 learned_addr learned my addr 172.21.15.12:0/3800879293 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a0e00700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f229c160f10 0x7f229c0b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a2200700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c161590 0x7f229c0b86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a1800700 1 -- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c161590 msgr2=0x7f229c0b86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a1800700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c161590 0x7f229c0b86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a1800700 1 -- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f229c160f10 msgr2=0x7f229c0b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a1800700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f229c160f10 0x7f229c0b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.333+0000 7f22a1800700 1 -- 172.21.15.12:0/3800879293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2294009b30 con 0x7f229c0cf230 2024-03-29T11:11:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.334+0000 7f22a1800700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229c0cf230 0x7f229c155150 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f2290014040 tx=0x7f2290004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.334+0000 7f2299600700 1 -- 172.21.15.12:0/3800879293 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22900038a0 con 0x7f229c0cf230 2024-03-29T11:11:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.334+0000 7f2299600700 1 -- 172.21.15.12:0/3800879293 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229001e070 con 0x7f229c0cf230 2024-03-29T11:11:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.334+0000 7f2299600700 1 -- 172.21.15.12:0/3800879293 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290003c40 con 0x7f229c0cf230 2024-03-29T11:11:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.334+0000 7f22a4200700 1 -- 172.21.15.12:0/3800879293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22900094a0 con 0x7f229c0cf230 2024-03-29T11:11:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.334+0000 7f22a4200700 1 -- 172.21.15.12:0/3800879293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229c157060 con 0x7f229c0cf230 2024-03-29T11:11:35.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.335+0000 7f22a4200700 1 -- 172.21.15.12:0/3800879293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229c067720 con 0x7f229c0cf230 2024-03-29T11:11:35.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.336+0000 7f2299600700 1 -- 172.21.15.12:0/3800879293 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f229002d450 con 0x7f229c0cf230 2024-03-29T11:11:35.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.337+0000 7f2299600700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227c06adb0 0x7f227c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:35.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.337+0000 7f2299600700 1 -- 172.21.15.12:0/3800879293 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f229008a460 con 0x7f229c0cf230 2024-03-29T11:11:35.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.337+0000 7f22a0e00700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227c06adb0 0x7f227c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:35.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.338+0000 7f22a0e00700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227c06adb0 0x7f227c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f229c15a6f0 tx=0x7f2288009040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:35.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.341+0000 7f2299600700 1 -- 172.21.15.12:0/3800879293 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2290055590 con 0x7f229c0cf230 2024-03-29T11:11:35.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:35.640+0000 7f22a4200700 1 -- 172.21.15.12:0/3800879293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f229c06cb70 con 0x7f229c0cf230 2024-03-29T11:11:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.113+0000 7f2299600700 1 -- 172.21.15.12:0/3800879293 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2505) v1 ==== 177+0+0 (secure 0 0 0) 0x7f2290058bb0 con 0x7f229c0cf230 2024-03-29T11:11:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.116+0000 7f22a4200700 1 -- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227c06adb0 msgr2=0x7f227c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.116+0000 7f22a4200700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227c06adb0 0x7f227c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f229c15a6f0 tx=0x7f2288009040 comp rx=0 tx=0).stop 2024-03-29T11:11:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.116+0000 7f22a4200700 1 -- 172.21.15.12:0/3800879293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229c0cf230 msgr2=0x7f229c155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.116+0000 7f22a4200700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229c0cf230 0x7f229c155150 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f2290014040 tx=0x7f2290004750 comp rx=0 tx=0).stop 2024-03-29T11:11:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.117+0000 7f22a4200700 1 -- 172.21.15.12:0/3800879293 shutdown_connections 2024-03-29T11:11:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.117+0000 7f22a4200700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f227c06adb0 0x7f227c06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.117+0000 7f22a4200700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f229c161590 0x7f229c0b86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.117+0000 7f22a4200700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f229c0cf230 0x7f229c155150 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.117+0000 7f22a4200700 1 --2- 172.21.15.12:0/3800879293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f229c160f10 0x7f229c0b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.117+0000 7f22a4200700 1 -- 172.21.15.12:0/3800879293 >> 172.21.15.12:0/3800879293 conn(0x7f229c08aff0 msgr2=0x7f229c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.117+0000 7f22a4200700 1 -- 172.21.15.12:0/3800879293 shutdown_connections 2024-03-29T11:11:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.117+0000 7f22a4200700 1 -- 172.21.15.12:0/3800879293 wait complete. 2024-03-29T11:11:36.116 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T11:11:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.341+0000 7f54db400700 1 Processor -- start 2024-03-29T11:11:36.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.341+0000 7f54db400700 1 -- start start 2024-03-29T11:11:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.343+0000 7f54db400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d415d710 0x7f54d40d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.343+0000 7f54db400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d40cf640 con 0x7f54d40d1b70 2024-03-29T11:11:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.343+0000 7f54db400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d40d0d30 con 0x7f54d40d1ff0 2024-03-29T11:11:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.343+0000 7f54db400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40cf220 con 0x7f54d415d710 2024-03-29T11:11:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.343+0000 7f54d3e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f54d40d1b70 0x7f54d40cff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35768/0 (socket says 172.21.15.12:35768) 2024-03-29T11:11:36.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.343+0000 7f54d3e00700 1 -- 172.21.15.12:0/13959809 learned_addr learned my addr 172.21.15.12:0/13959809 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:36.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.343+0000 7f54d8a00700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d415d710 0x7f54d40d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:36.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.343+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882642565 0 0) 0x7f54d40cf640 con 0x7f54d40d1b70 2024-03-29T11:11:36.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.343+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54bc0035b0 con 0x7f54d40d1b70 2024-03-29T11:11:36.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2333094420 0 0) 0x7f54bc0035b0 con 0x7f54d40d1b70 2024-03-29T11:11:36.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d415d710 msgr2=0x7f54d40d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:36.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d415d710 0x7f54d40d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 >> v1:172.21.15.159:6789/0 conn(0x7f54d40d1ff0 legacy=0x7f54d40d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d40ce5a0 con 0x7f54d40d1b70 2024-03-29T11:11:36.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f54c00017f0 con 0x7f54d40d1b70 2024-03-29T11:11:36.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d8a00700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d415d710 0x7f54d40d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:11:36.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 >> v1:172.21.15.12:6789/0 conn(0x7f54d40d1b70 legacy=0x7f54d40cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc003a30 0x7f54bc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:36.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc0043e0 0x7f54bc0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:36.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d415d710 0x7f54bc005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:36.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40cf220 con 0x7f54bc003a30 2024-03-29T11:11:36.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc00ba60 con 0x7f54bc0043e0 2024-03-29T11:11:36.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc00bc20 con 0x7f54d415d710 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d8a00700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc003a30 0x7f54bc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d3e00700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc0043e0 0x7f54bc0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.344+0000 7f54d9400700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d415d710 0x7f54bc005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d8a00700 1 -- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d415d710 msgr2=0x7f54bc005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d8a00700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d415d710 0x7f54bc005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d8a00700 1 -- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc0043e0 msgr2=0x7f54bc0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d8a00700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc0043e0 0x7f54bc0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d8a00700 1 -- 172.21.15.12:0/13959809 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c80094a0 con 0x7f54bc003a30 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d3e00700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc0043e0 0x7f54bc0048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d8a00700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc003a30 0x7f54bc003e10 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009d30 tx=0x7f54bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffedaf429b67cb06 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:36.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8009e80 con 0x7f54bc003a30 2024-03-29T11:11:36.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c800e9a0 con 0x7f54bc003a30 2024-03-29T11:11:36.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d3400700 1 -- 172.21.15.12:0/13959809 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c800ed10 con 0x7f54bc003a30 2024-03-29T11:11:36.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54db400700 1 -- 172.21.15.12:0/13959809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc003a30 msgr2=0x7f54bc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54db400700 1 --2- 172.21.15.12:0/13959809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54bc003a30 0x7f54bc003e10 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009d30 tx=0x7f54bc005880 comp rx=0 tx=0).stop 2024-03-29T11:11:36.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.345+0000 7f54d9400700 1 -- 172.21.15.12:0/13959809 reap_dead start 2024-03-29T11:11:36.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.346+0000 7f54db400700 1 -- 172.21.15.12:0/13959809 shutdown_connections 2024-03-29T11:11:36.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.346+0000 7f54db400700 1 -- 172.21.15.12:0/13959809 >> 172.21.15.12:0/13959809 conn(0x7f54d408aff0 msgr2=0x7f54d40d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:36.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.346+0000 7f54db400700 1 -- 172.21.15.12:0/13959809 shutdown_connections 2024-03-29T11:11:36.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.346+0000 7f54db400700 1 -- 172.21.15.12:0/13959809 wait complete. 2024-03-29T11:11:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.348+0000 7f54db400700 1 Processor -- start 2024-03-29T11:11:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.348+0000 7f54db400700 1 -- start start 2024-03-29T11:11:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.349+0000 7f54db400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc003a30 0x7f54d40c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.349+0000 7f54db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d4155c70 0x7f54d41560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.349+0000 7f54db400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d40c91c0 0x7f54d40c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:36.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54db400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40c9b80 con 0x7f54d4155c70 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54db400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d41f6930 con 0x7f54bc003a30 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54db400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d414a410 con 0x7f54d40c91c0 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54d3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d4155c70 0x7f54d41560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54d3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d4155c70 0x7f54d41560f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38710/0 (socket says 172.21.15.12:38710) 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54d3e00700 1 -- 172.21.15.12:0/2575261347 learned_addr learned my addr 172.21.15.12:0/2575261347 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54d9400700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d40c91c0 0x7f54d40c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54d8a00700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc003a30 0x7f54d40c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54d3e00700 1 -- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d40c91c0 msgr2=0x7f54d40c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54d3e00700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d40c91c0 0x7f54d40c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54d3e00700 1 -- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc003a30 msgr2=0x7f54d40c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54d3e00700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc003a30 0x7f54d40c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.350+0000 7f54d3e00700 1 -- 172.21.15.12:0/2575261347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d41f6b80 con 0x7f54d4155c70 2024-03-29T11:11:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.351+0000 7f54d3e00700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d4155c70 0x7f54d41560f0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f54c000e7b0 tx=0x7f54c000eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.351+0000 7f54d0c00700 1 -- 172.21.15.12:0/2575261347 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00072e0 con 0x7f54d4155c70 2024-03-29T11:11:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.351+0000 7f54d0c00700 1 -- 172.21.15.12:0/2575261347 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0007440 con 0x7f54d4155c70 2024-03-29T11:11:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.351+0000 7f54d0c00700 1 -- 172.21.15.12:0/2575261347 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0005830 con 0x7f54d4155c70 2024-03-29T11:11:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.351+0000 7f54db400700 1 -- 172.21.15.12:0/2575261347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d41f6e60 con 0x7f54d4155c70 2024-03-29T11:11:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.351+0000 7f54db400700 1 -- 172.21.15.12:0/2575261347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d41f7420 con 0x7f54d4155c70 2024-03-29T11:11:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.352+0000 7f54db400700 1 -- 172.21.15.12:0/2575261347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d4067e30 con 0x7f54d4155c70 2024-03-29T11:11:36.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.353+0000 7f54d0c00700 1 -- 172.21.15.12:0/2575261347 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f54c0005990 con 0x7f54d4155c70 2024-03-29T11:11:36.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.354+0000 7f54d0c00700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b406adb0 0x7f54b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:36.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.354+0000 7f54d0c00700 1 -- 172.21.15.12:0/2575261347 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f54c00892d0 con 0x7f54d4155c70 2024-03-29T11:11:36.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.354+0000 7f54d8a00700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b406adb0 0x7f54b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.355+0000 7f54d8a00700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b406adb0 0x7f54b406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54c80090b0 tx=0x7f54c8008030 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.358+0000 7f54d0c00700 1 -- 172.21.15.12:0/2575261347 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54c0054370 con 0x7f54d4155c70 2024-03-29T11:11:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.577+0000 7f54db400700 1 -- 172.21.15.12:0/2575261347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f54d40cd470 con 0x7f54d4155c70 2024-03-29T11:11:36.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.581+0000 7f54d0c00700 1 -- 172.21.15.12:0/2575261347 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f54c0057990 con 0x7f54d4155c70 2024-03-29T11:11:36.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 -- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b406adb0 msgr2=0x7f54b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b406adb0 0x7f54b406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54c80090b0 tx=0x7f54c8008030 comp rx=0 tx=0).stop 2024-03-29T11:11:36.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 -- 172.21.15.12:0/2575261347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d4155c70 msgr2=0x7f54d41560f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d4155c70 0x7f54d41560f0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f54c000e7b0 tx=0x7f54c000eb70 comp rx=0 tx=0).stop 2024-03-29T11:11:36.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 -- 172.21.15.12:0/2575261347 shutdown_connections 2024-03-29T11:11:36.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f54b406adb0 0x7f54b406d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54d40c91c0 0x7f54d40c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc003a30 0x7f54d40c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 --2- 172.21.15.12:0/2575261347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54d4155c70 0x7f54d41560f0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:36.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 -- 172.21.15.12:0/2575261347 >> 172.21.15.12:0/2575261347 conn(0x7f54d408aff0 msgr2=0x7f54d40b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:36.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 -- 172.21.15.12:0/2575261347 shutdown_connections 2024-03-29T11:11:36.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:36.584+0000 7f54db400700 1 -- 172.21.15.12:0/2575261347 wait complete. 2024-03-29T11:11:36.580 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:11:36.581 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:11:36.666 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:11:36.759 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:11:36.759 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:11:36.760 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:11:36.805 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:11:36.806 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:11:36.806 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:11:36.853 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:11:36.853 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:11:36.853 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:11:36.897 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:11:36.897 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:11:36.897 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:11:36.945 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:11:36.945 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T11:11:36.945 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:11:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.166+0000 7f30fac00700 1 Processor -- start 2024-03-29T11:11:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.166+0000 7f30fac00700 1 -- start start 2024-03-29T11:11:37.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.168+0000 7f30fac00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f30f40cebd0 0x7f30f40cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:37.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.168+0000 7f30fac00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f406f850 con 0x7f30f40afbe0 2024-03-29T11:11:37.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.168+0000 7f30fac00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f406e3f0 con 0x7f30f406a790 2024-03-29T11:11:37.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.168+0000 7f30fac00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f406fad0 con 0x7f30f40cebd0 2024-03-29T11:11:37.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.168+0000 7f30f3400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f30f40afbe0 0x7f30f406dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35770/0 (socket says 172.21.15.12:35770) 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.168+0000 7f30f3400700 1 -- 172.21.15.12:0/1854661258 learned_addr learned my addr 172.21.15.12:0/1854661258 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.168+0000 7f30f8c00700 1 --2- 172.21.15.12:0/1854661258 >> v2:172.21.15.159:3300/0 conn(0x7f30f40cebd0 0x7f30f40cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.168+0000 7f30f2a00700 1 -- 172.21.15.12:0/1854661258 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079982425 0 0) 0x7f30f406f850 con 0x7f30f40afbe0 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.169+0000 7f30f2a00700 1 -- 172.21.15.12:0/1854661258 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d80035b0 con 0x7f30f40afbe0 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.169+0000 7f30f8c00700 1 -- 172.21.15.12:0/1854661258 >> v1:172.21.15.159:6789/0 conn(0x7f30f406a790 legacy=0x7f30f406ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.169+0000 7f30f2a00700 1 -- 172.21.15.12:0/1854661258 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174893046 0 0) 0x7f30f406e3f0 con 0x7f30f406a790 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.169+0000 7f30f8c00700 1 -- 172.21.15.12:0/1854661258 >> v1:172.21.15.12:6789/0 conn(0x7f30f40afbe0 legacy=0x7f30f406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.169+0000 7f30f8c00700 1 -- 172.21.15.12:0/1854661258 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f40af970 con 0x7f30f40cebd0 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.169+0000 7f30f8c00700 1 --2- 172.21.15.12:0/1854661258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30f40cebd0 0x7f30f40cf000 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009d90 tx=0x7f30ec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7364557eb2c27252 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.169+0000 7f30f2a00700 1 -- 172.21.15.12:0/1854661258 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec00e070 con 0x7f30f40cebd0 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.169+0000 7f30f2a00700 1 -- 172.21.15.12:0/1854661258 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec00bb20 con 0x7f30f40cebd0 2024-03-29T11:11:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.169+0000 7f30f2a00700 1 -- 172.21.15.12:0/1854661258 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec00bed0 con 0x7f30f40cebd0 2024-03-29T11:11:37.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.170+0000 7f30fac00700 1 -- 172.21.15.12:0/1854661258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30f40cebd0 msgr2=0x7f30f40cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.170+0000 7f30fac00700 1 --2- 172.21.15.12:0/1854661258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30f40cebd0 0x7f30f40cf000 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009d90 tx=0x7f30ec00b330 comp rx=0 tx=0).stop 2024-03-29T11:11:37.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.170+0000 7f30fac00700 1 -- 172.21.15.12:0/1854661258 shutdown_connections 2024-03-29T11:11:37.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.170+0000 7f30fac00700 1 --2- 172.21.15.12:0/1854661258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30f40cebd0 0x7f30f40cf000 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.170+0000 7f30fac00700 1 -- 172.21.15.12:0/1854661258 >> 172.21.15.12:0/1854661258 conn(0x7f30f40b5ab0 msgr2=0x7f30f40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.170+0000 7f30fac00700 1 -- 172.21.15.12:0/1854661258 shutdown_connections 2024-03-29T11:11:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.170+0000 7f30fac00700 1 -- 172.21.15.12:0/1854661258 wait complete. 2024-03-29T11:11:37.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.172+0000 7f30fac00700 1 Processor -- start 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.172+0000 7f30fac00700 1 -- start start 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30fac00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30f406a790 0x7f30f409d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30fac00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30f40afbe0 0x7f30f4149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30fac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30f40cebd0 0x7f30f40cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30fac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f414a170 con 0x7f30f40cebd0 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30fac00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f40cd0e0 con 0x7f30f406a790 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30fac00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f40cd3c0 con 0x7f30f40afbe0 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30f8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30f40cebd0 0x7f30f40cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30f8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30f40cebd0 0x7f30f40cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38722/0 (socket says 172.21.15.12:38722) 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30f8c00700 1 -- 172.21.15.12:0/2414841908 learned_addr learned my addr 172.21.15.12:0/2414841908 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30f3e00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30f406a790 0x7f30f409d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30f3400700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30f40afbe0 0x7f30f4149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:37.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30f8c00700 1 -- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30f40afbe0 msgr2=0x7f30f4149c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30f8c00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30f40afbe0 0x7f30f4149c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30f8c00700 1 -- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30f406a790 msgr2=0x7f30f409d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30f8c00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30f406a790 0x7f30f409d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.173+0000 7f30f8c00700 1 -- 172.21.15.12:0/2414841908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec009a70 con 0x7f30f40cebd0 2024-03-29T11:11:37.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.174+0000 7f30f8c00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30f40cebd0 0x7f30f40cc800 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f30ec014040 tx=0x7f30ec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:37.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.174+0000 7f30ebe00700 1 -- 172.21.15.12:0/2414841908 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec01b810 con 0x7f30f40cebd0 2024-03-29T11:11:37.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.174+0000 7f30ebe00700 1 -- 172.21.15.12:0/2414841908 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec020070 con 0x7f30f40cebd0 2024-03-29T11:11:37.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.174+0000 7f30fac00700 1 -- 172.21.15.12:0/2414841908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f40cd6a0 con 0x7f30f40cebd0 2024-03-29T11:11:37.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.174+0000 7f30ebe00700 1 -- 172.21.15.12:0/2414841908 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec01bbd0 con 0x7f30f40cebd0 2024-03-29T11:11:37.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.174+0000 7f30fac00700 1 -- 172.21.15.12:0/2414841908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f40ccd40 con 0x7f30f40cebd0 2024-03-29T11:11:37.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.175+0000 7f30fac00700 1 -- 172.21.15.12:0/2414841908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f40c9780 con 0x7f30f40cebd0 2024-03-29T11:11:37.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.176+0000 7f30ebe00700 1 -- 172.21.15.12:0/2414841908 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f30ec01bd30 con 0x7f30f40cebd0 2024-03-29T11:11:37.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.177+0000 7f30ebe00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30d406adb0 0x7f30d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:37.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.177+0000 7f30ebe00700 1 -- 172.21.15.12:0/2414841908 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f30ec022070 con 0x7f30f40cebd0 2024-03-29T11:11:37.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.177+0000 7f30f3e00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30d406adb0 0x7f30d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:37.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.178+0000 7f30f3e00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30d406adb0 0x7f30d406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f30e4000f60 tx=0x7f30e4009040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:37.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.182+0000 7f30ebe00700 1 -- 172.21.15.12:0/2414841908 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f30f40c9780 con 0x7f30f40cebd0 2024-03-29T11:11:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.443+0000 7f30fac00700 1 -- 172.21.15.12:0/2414841908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30f40c9780 con 0x7f30f40cebd0 2024-03-29T11:11:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.443+0000 7f30ebe00700 1 -- 172.21.15.12:0/2414841908 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f30ec02cde0 con 0x7f30f40cebd0 2024-03-29T11:11:37.421 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:37.422 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3149931506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3149931506},{"type":"v1","addr":"172.21.15.159:6835","nonce":3149931506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/967685480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":967685480},{"type":"v1","addr":"172.21.15.159:6837","nonce":967685480}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8967,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/4273394318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4273394318},{"type":"v1","addr":"172.21.15.12:6834","nonce":4273394318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":9357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1055737204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":1055737204},{"type":"v1","addr":"172.21.15.12:6835","nonce":1055737204}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[]} 2024-03-29T11:11:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 -- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30d406adb0 msgr2=0x7f30d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30d406adb0 0x7f30d406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f30e4000f60 tx=0x7f30e4009040 comp rx=0 tx=0).stop 2024-03-29T11:11:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 -- 172.21.15.12:0/2414841908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30f40cebd0 msgr2=0x7f30f40cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30f40cebd0 0x7f30f40cc800 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f30ec014040 tx=0x7f30ec004ad0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 -- 172.21.15.12:0/2414841908 shutdown_connections 2024-03-29T11:11:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f30d406adb0 0x7f30d406d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30f40cebd0 0x7f30f40cc800 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30f406a790 0x7f30f409d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 --2- 172.21.15.12:0/2414841908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30f40afbe0 0x7f30f4149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 -- 172.21.15.12:0/2414841908 >> 172.21.15.12:0/2414841908 conn(0x7f30f40b5ab0 msgr2=0x7f30f406ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 -- 172.21.15.12:0/2414841908 shutdown_connections 2024-03-29T11:11:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.446+0000 7f30fac00700 1 -- 172.21.15.12:0/2414841908 wait complete. 2024-03-29T11:11:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 75 2024-03-29T11:11:37.443 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T11:11:37.443 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T11:11:37.443 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T11:11:37.443 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:37.444 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T11:11:37.444 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T11:11:37.444 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T11:11:37.444 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:37.444 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T11:11:37.444 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T11:11:37.444 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T11:11:37.445 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:37.445 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T11:11:37.445 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T11:11:37.445 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T11:11:37.445 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:37.457 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T11:11:37.457 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T11:11:37.459 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T11:11:37.459 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T11:11:37.461 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T11:11:37.462 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T11:11:37.463 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T11:11:37.463 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T11:11:37.503 INFO:tasks.ceph.mds.c:Started 2024-03-29T11:11:37.504 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:37.507 INFO:tasks.ceph.mds.b:Started 2024-03-29T11:11:37.507 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:37.508 INFO:tasks.ceph.mds.a:Started 2024-03-29T11:11:37.508 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:37.510 INFO:tasks.ceph.mds.d:Started 2024-03-29T11:11:37.510 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:37.510 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T11:11:37.650 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T11:11:37.651 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T11:11:37.651 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T11:11:37.655 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:11:37.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.746+0000 7f0d26800700 1 Processor -- start 2024-03-29T11:11:37.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.746+0000 7f0d26800700 1 -- start start 2024-03-29T11:11:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.747+0000 7f0d26800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0d20160d70 0x7f0d200b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.747+0000 7f0d26800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d20161150 con 0x7f0d2015ef70 2024-03-29T11:11:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.747+0000 7f0d26800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2006dba0 con 0x7f0d2006a790 2024-03-29T11:11:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.747+0000 7f0d26800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d200b4cd0 con 0x7f0d20160d70 2024-03-29T11:11:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.747+0000 7f0d1f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0d20160d70 0x7f0d200b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.747+0000 7f0d1ea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0d2015ef70 0x7f0d2015e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35788/0 (socket says 172.21.15.12:35788) 2024-03-29T11:11:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.747+0000 7f0d1ea00700 1 -- 172.21.15.12:0/1570186807 learned_addr learned my addr 172.21.15.12:0/1570186807 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.748+0000 7f0d1f400700 1 -- 172.21.15.12:0/1570186807 >> v1:172.21.15.159:6789/0 conn(0x7f0d2006a790 legacy=0x7f0d2006e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.748+0000 7f0d1f400700 1 -- 172.21.15.12:0/1570186807 >> v1:172.21.15.12:6789/0 conn(0x7f0d2015ef70 legacy=0x7f0d2015e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.748+0000 7f0d1f400700 1 -- 172.21.15.12:0/1570186807 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d200b4ee0 con 0x7f0d20160d70 2024-03-29T11:11:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.748+0000 7f0d1e000700 1 -- 172.21.15.12:0/1570186807 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919817263 0 0) 0x7f0d20161150 con 0x7f0d2015ef70 2024-03-29T11:11:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.748+0000 7f0d1f400700 1 --2- 172.21.15.12:0/1570186807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d20160d70 0x7f0d200b3200 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009840 tx=0x7f0d10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486b794f6e26105d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.748+0000 7f0d1e000700 1 -- 172.21.15.12:0/1570186807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10011830 con 0x7f0d20160d70 2024-03-29T11:11:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.748+0000 7f0d1e000700 1 -- 172.21.15.12:0/1570186807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d10011990 con 0x7f0d20160d70 2024-03-29T11:11:37.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.748+0000 7f0d1e000700 1 -- 172.21.15.12:0/1570186807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10011d20 con 0x7f0d20160d70 2024-03-29T11:11:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.749+0000 7f0d26800700 1 -- 172.21.15.12:0/1570186807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d20160d70 msgr2=0x7f0d200b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.749+0000 7f0d26800700 1 --2- 172.21.15.12:0/1570186807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d20160d70 0x7f0d200b3200 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009840 tx=0x7f0d10009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.749+0000 7f0d26800700 1 -- 172.21.15.12:0/1570186807 shutdown_connections 2024-03-29T11:11:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.749+0000 7f0d26800700 1 --2- 172.21.15.12:0/1570186807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d20160d70 0x7f0d200b3200 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.749+0000 7f0d26800700 1 -- 172.21.15.12:0/1570186807 >> 172.21.15.12:0/1570186807 conn(0x7f0d200b5720 msgr2=0x7f0d200b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.749+0000 7f0d26800700 1 -- 172.21.15.12:0/1570186807 shutdown_connections 2024-03-29T11:11:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.749+0000 7f0d26800700 1 -- 172.21.15.12:0/1570186807 wait complete. 2024-03-29T11:11:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.751+0000 7f0d26800700 1 Processor -- start 2024-03-29T11:11:37.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.751+0000 7f0d26800700 1 -- start start 2024-03-29T11:11:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d26800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d2006a790 0x7f0d20157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d26800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d2015ef70 0x7f0d20157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d26800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d20160d70 0x7f0d200af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d26800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2014bfa0 con 0x7f0d2006a790 2024-03-29T11:11:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d26800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d20145bd0 con 0x7f0d20160d70 2024-03-29T11:11:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d26800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d20157f30 con 0x7f0d2015ef70 2024-03-29T11:11:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d1f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d2006a790 0x7f0d20157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d1fe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d20160d70 0x7f0d200af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d1f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d2006a790 0x7f0d20157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38772/0 (socket says 172.21.15.12:38772) 2024-03-29T11:11:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d1fe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d20160d70 0x7f0d200af690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:53828/0 (socket says 172.21.15.12:53828) 2024-03-29T11:11:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d1f400700 1 -- 172.21.15.12:0/1928369257 learned_addr learned my addr 172.21.15.12:0/1928369257 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d1ea00700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d2015ef70 0x7f0d20157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d1f400700 1 -- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d2015ef70 msgr2=0x7f0d20157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.752+0000 7f0d1f400700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d2015ef70 0x7f0d20157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.753+0000 7f0d1f400700 1 -- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d20160d70 msgr2=0x7f0d200af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.753+0000 7f0d1f400700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d20160d70 0x7f0d200af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.753+0000 7f0d1f400700 1 -- 172.21.15.12:0/1928369257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d100094a0 con 0x7f0d2006a790 2024-03-29T11:11:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.753+0000 7f0d1f400700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d2006a790 0x7f0d20157350 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f0d10014040 tx=0x7f0d10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.753+0000 7f0d17400700 1 -- 172.21.15.12:0/1928369257 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1001a860 con 0x7f0d2006a790 2024-03-29T11:11:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.753+0000 7f0d17400700 1 -- 172.21.15.12:0/1928369257 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1001a9c0 con 0x7f0d2006a790 2024-03-29T11:11:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.753+0000 7f0d17400700 1 -- 172.21.15.12:0/1928369257 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1001ad70 con 0x7f0d2006a790 2024-03-29T11:11:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.753+0000 7f0d26800700 1 -- 172.21.15.12:0/1928369257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d20158210 con 0x7f0d2006a790 2024-03-29T11:11:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.753+0000 7f0d26800700 1 -- 172.21.15.12:0/1928369257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d200b1b60 con 0x7f0d2006a790 2024-03-29T11:11:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.754+0000 7f0d26800700 1 -- 172.21.15.12:0/1928369257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d200b16e0 con 0x7f0d2006a790 2024-03-29T11:11:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.755+0000 7f0d17400700 1 -- 172.21.15.12:0/1928369257 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0d10022070 con 0x7f0d2006a790 2024-03-29T11:11:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.756+0000 7f0d17400700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d0006adb0 0x7f0d0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.756+0000 7f0d17400700 1 -- 172.21.15.12:0/1928369257 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f0d1008a730 con 0x7f0d2006a790 2024-03-29T11:11:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.756+0000 7f0d1ea00700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d0006adb0 0x7f0d0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.757+0000 7f0d1ea00700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d0006adb0 0x7f0d0006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c007b10 tx=0x7f0d0c007a60 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.760+0000 7f0d17400700 1 -- 172.21.15.12:0/1928369257 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0d200b16e0 con 0x7f0d2006a790 2024-03-29T11:11:37.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:37.991+0000 7f0d26800700 1 -- 172.21.15.12:0/1928369257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0d200b16e0 con 0x7f0d2006a790 2024-03-29T11:11:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.706+0000 7f0d17400700 1 -- 172.21.15.12:0/1928369257 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v173) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0d200b16e0 con 0x7f0d2006a790 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 -- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d0006adb0 msgr2=0x7f0d0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d0006adb0 0x7f0d0006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c007b10 tx=0x7f0d0c007a60 comp rx=0 tx=0).stop 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 -- 172.21.15.12:0/1928369257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d2006a790 msgr2=0x7f0d20157350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d2006a790 0x7f0d20157350 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f0d10014040 tx=0x7f0d10004750 comp rx=0 tx=0).stop 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 -- 172.21.15.12:0/1928369257 shutdown_connections 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0d0006adb0 0x7f0d0006d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d20160d70 0x7f0d200af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d2006a790 0x7f0d20157350 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 --2- 172.21.15.12:0/1928369257 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d2015ef70 0x7f0d20157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 -- 172.21.15.12:0/1928369257 >> 172.21.15.12:0/1928369257 conn(0x7f0d200b5720 msgr2=0x7f0d2015ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.709+0000 7f0d26800700 1 -- 172.21.15.12:0/1928369257 shutdown_connections 2024-03-29T11:11:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.710+0000 7f0d26800700 1 -- 172.21.15.12:0/1928369257 wait complete. 2024-03-29T11:11:38.689 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T11:11:38.709 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T11:11:38.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.933+0000 7f5cd3000700 1 Processor -- start 2024-03-29T11:11:38.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.933+0000 7f5cd3000700 1 -- start start 2024-03-29T11:11:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.934+0000 7f5cd3000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5ccc160f10 0x7f5ccc0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.934+0000 7f5cd3000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5ccc0cf230 0x7f5ccc0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.934+0000 7f5cd3000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0ceda0 con 0x7f5ccc161590 2024-03-29T11:11:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.934+0000 7f5cd3000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc0d23d0 con 0x7f5ccc160f10 2024-03-29T11:11:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.934+0000 7f5cd3000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc1612f0 con 0x7f5ccc0cf230 2024-03-29T11:11:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.935+0000 7f5ccbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5ccc160f10 0x7f5ccc0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.935+0000 7f5ccbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5ccc160f10 0x7f5ccc0d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58068/0 (socket says 172.21.15.12:58068) 2024-03-29T11:11:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.935+0000 7f5ccbe00700 1 -- 172.21.15.12:0/3056542661 learned_addr learned my addr 172.21.15.12:0/3056542661 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.935+0000 7f5ccb400700 1 --2- 172.21.15.12:0/3056542661 >> v2:172.21.15.159:3300/0 conn(0x7f5ccc0cf230 0x7f5ccc0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.935+0000 7f5ccbe00700 1 -- 172.21.15.12:0/3056542661 >> v2:172.21.15.159:3300/0 conn(0x7f5ccc0cf230 msgr2=0x7f5ccc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.935+0000 7f5ccbe00700 1 --2- 172.21.15.12:0/3056542661 >> v2:172.21.15.159:3300/0 conn(0x7f5ccc0cf230 0x7f5ccc0d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.935+0000 7f5ccbe00700 1 -- 172.21.15.12:0/3056542661 >> v1:172.21.15.159:6789/0 conn(0x7f5ccc161590 legacy=0x7f5ccc0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.935+0000 7f5ccbe00700 1 -- 172.21.15.12:0/3056542661 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0ce470 con 0x7f5ccc160f10 2024-03-29T11:11:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.935+0000 7f5ccb400700 1 --2- 172.21.15.12:0/3056542661 >> v2:172.21.15.159:3300/0 conn(0x7f5ccc0cf230 0x7f5ccc0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.935+0000 7f5ccbe00700 1 --2- 172.21.15.12:0/3056542661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc160f10 0x7f5ccc0d18b0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc08cdb0 tx=0x7f5cbc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5a4d4bf3abfe7e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:38.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5ccaa00700 1 -- 172.21.15.12:0/3056542661 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc004230 con 0x7f5ccc160f10 2024-03-29T11:11:38.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5ccaa00700 1 -- 172.21.15.12:0/3056542661 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc004390 con 0x7f5ccc160f10 2024-03-29T11:11:38.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5ccaa00700 1 -- 172.21.15.12:0/3056542661 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc0036a0 con 0x7f5ccc160f10 2024-03-29T11:11:38.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5cd3000700 1 -- 172.21.15.12:0/3056542661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc160f10 msgr2=0x7f5ccc0d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5cd3000700 1 --2- 172.21.15.12:0/3056542661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc160f10 0x7f5ccc0d18b0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc08cdb0 tx=0x7f5cbc009bd0 comp rx=0 tx=0).stop 2024-03-29T11:11:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5cd3000700 1 -- 172.21.15.12:0/3056542661 shutdown_connections 2024-03-29T11:11:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5cd3000700 1 --2- 172.21.15.12:0/3056542661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc160f10 0x7f5ccc0d18b0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5cd3000700 1 --2- 172.21.15.12:0/3056542661 >> v2:172.21.15.159:3300/0 conn(0x7f5ccc0cf230 0x7f5ccc0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5cd3000700 1 -- 172.21.15.12:0/3056542661 >> 172.21.15.12:0/3056542661 conn(0x7f5ccc08aff0 msgr2=0x7f5ccc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5cd3000700 1 -- 172.21.15.12:0/3056542661 shutdown_connections 2024-03-29T11:11:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.936+0000 7f5cd3000700 1 -- 172.21.15.12:0/3056542661 wait complete. 2024-03-29T11:11:38.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.938+0000 7f5cd3000700 1 Processor -- start 2024-03-29T11:11:38.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.938+0000 7f5cd3000700 1 -- start start 2024-03-29T11:11:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.939+0000 7f5cd3000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc0cf230 0x7f5ccc158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.939+0000 7f5cd3000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ccc160f10 0x7f5ccc1f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.939+0000 7f5cd3000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ccc161590 0x7f5ccc1f3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.939+0000 7f5cd3000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc156bf0 con 0x7f5ccc0cf230 2024-03-29T11:11:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.939+0000 7f5cd3000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc1f31b0 con 0x7f5ccc161590 2024-03-29T11:11:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.939+0000 7f5cd3000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc1f42d0 con 0x7f5ccc160f10 2024-03-29T11:11:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.939+0000 7f5ccbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc0cf230 0x7f5ccc158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.939+0000 7f5ccbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc0cf230 0x7f5ccc158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58074/0 (socket says 172.21.15.12:58074) 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.939+0000 7f5ccbe00700 1 -- 172.21.15.12:0/1361799470 learned_addr learned my addr 172.21.15.12:0/1361799470 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5ccb400700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ccc160f10 0x7f5ccc1f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5cd1000700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ccc161590 0x7f5ccc1f3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5ccbe00700 1 -- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ccc160f10 msgr2=0x7f5ccc1f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5ccbe00700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ccc160f10 0x7f5ccc1f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5ccbe00700 1 -- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ccc161590 msgr2=0x7f5ccc1f3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5ccbe00700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ccc161590 0x7f5ccc1f3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5ccbe00700 1 -- 172.21.15.12:0/1361799470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc009550 con 0x7f5ccc0cf230 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5ccb400700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ccc160f10 0x7f5ccc1f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5cd1000700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ccc161590 0x7f5ccc1f3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5ccbe00700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc0cf230 0x7f5ccc158130 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc08c570 tx=0x7f5cbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5cc3e00700 1 -- 172.21.15.12:0/1361799470 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc011630 con 0x7f5ccc0cf230 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5cc3e00700 1 -- 172.21.15.12:0/1361799470 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc011790 con 0x7f5ccc0cf230 2024-03-29T11:11:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5cd3000700 1 -- 172.21.15.12:0/1361799470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc158730 con 0x7f5ccc0cf230 2024-03-29T11:11:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.940+0000 7f5cc3e00700 1 -- 172.21.15.12:0/1361799470 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc011b00 con 0x7f5ccc0cf230 2024-03-29T11:11:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.941+0000 7f5cd3000700 1 -- 172.21.15.12:0/1361799470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc158cf0 con 0x7f5ccc0cf230 2024-03-29T11:11:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.941+0000 7f5cd3000700 1 -- 172.21.15.12:0/1361799470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc067720 con 0x7f5ccc0cf230 2024-03-29T11:11:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.942+0000 7f5cc3e00700 1 -- 172.21.15.12:0/1361799470 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5cbc011c60 con 0x7f5ccc0cf230 2024-03-29T11:11:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.944+0000 7f5cc3e00700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cac06adb0 0x7f5cac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.944+0000 7f5cc3e00700 1 -- 172.21.15.12:0/1361799470 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f5cbc08a8c0 con 0x7f5ccc0cf230 2024-03-29T11:11:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.944+0000 7f5ccb400700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cac06adb0 0x7f5cac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.945+0000 7f5ccb400700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cac06adb0 0x7f5cac06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc1536f0 tx=0x7f5cb8008040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:38.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:38.948+0000 7f5cc3e00700 1 -- 172.21.15.12:0/1361799470 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5cbc055ca0 con 0x7f5ccc0cf230 2024-03-29T11:11:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.253+0000 7f5cd3000700 1 -- 172.21.15.12:0/1361799470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5ccc0d01f0 con 0x7f5ccc0cf230 2024-03-29T11:11:39.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.254+0000 7f5cc3e00700 1 -- 172.21.15.12:0/1361799470 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5cbc0592c0 con 0x7f5ccc0cf230 2024-03-29T11:11:39.232 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:39.232 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:11:39.232 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T11:11:39.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.232 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T11:11:39.232 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T11:11:39.232 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.232 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:11:39.232 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T11:11:39.234 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:11:39.236 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.237 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.238 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.239 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.240 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T11:11:39.242 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.243 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T11:11:39.244 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:11:39.245 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:11:39.245 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:11:39.245 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:11:39.245 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.245 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:39.245 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T11:11:39.245 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:11:39.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.256+0000 7f5cd3000700 1 -- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cac06adb0 msgr2=0x7f5cac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.256+0000 7f5cd3000700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cac06adb0 0x7f5cac06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc1536f0 tx=0x7f5cb8008040 comp rx=0 tx=0).stop 2024-03-29T11:11:39.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.256+0000 7f5cd3000700 1 -- 172.21.15.12:0/1361799470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc0cf230 msgr2=0x7f5ccc158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.256+0000 7f5cd3000700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc0cf230 0x7f5ccc158130 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc08c570 tx=0x7f5cbc004750 comp rx=0 tx=0).stop 2024-03-29T11:11:39.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.256+0000 7f5cd3000700 1 -- 172.21.15.12:0/1361799470 shutdown_connections 2024-03-29T11:11:39.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.256+0000 7f5cd3000700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f5cac06adb0 0x7f5cac06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.256+0000 7f5cd3000700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ccc161590 0x7f5ccc1f3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.256+0000 7f5cd3000700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ccc0cf230 0x7f5ccc158130 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.256+0000 7f5cd3000700 1 --2- 172.21.15.12:0/1361799470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ccc160f10 0x7f5ccc1f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.256+0000 7f5cd3000700 1 -- 172.21.15.12:0/1361799470 >> 172.21.15.12:0/1361799470 conn(0x7f5ccc08aff0 msgr2=0x7f5ccc0cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:39.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.257+0000 7f5cd3000700 1 -- 172.21.15.12:0/1361799470 shutdown_connections 2024-03-29T11:11:39.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.257+0000 7f5cd3000700 1 -- 172.21.15.12:0/1361799470 wait complete. 2024-03-29T11:11:39.253 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.474+0000 7f6f73600700 1 Processor -- start 2024-03-29T11:11:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.475+0000 7f6f73600700 1 -- start start 2024-03-29T11:11:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.476+0000 7f6f73600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6f6c15da50 0x7f6f6c0ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.476+0000 7f6f73600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f6c0d0cf0 0x7f6f6c0b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.476+0000 7f6f73600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6c0d24e0 con 0x7f6f6c0cec70 2024-03-29T11:11:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.476+0000 7f6f73600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6c0b95d0 con 0x7f6f6c15da50 2024-03-29T11:11:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.476+0000 7f6f73600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c15de30 con 0x7f6f6c0d0cf0 2024-03-29T11:11:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.476+0000 7f6f6be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6f6c0cec70 0x7f6f6c0d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60150/0 (socket says 172.21.15.12:60150) 2024-03-29T11:11:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.476+0000 7f6f6be00700 1 -- 172.21.15.12:0/70063021 learned_addr learned my addr 172.21.15.12:0/70063021 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.476+0000 7f6f70c00700 1 --2- 172.21.15.12:0/70063021 >> v2:172.21.15.159:3300/0 conn(0x7f6f6c15da50 0x7f6f6c0ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.476+0000 7f6f71600700 1 --2- 172.21.15.12:0/70063021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f6c0d0cf0 0x7f6f6c0b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.477+0000 7f6f6b400700 1 -- 172.21.15.12:0/70063021 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053065716 0 0) 0x7f6f6c0d24e0 con 0x7f6f6c0cec70 2024-03-29T11:11:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.477+0000 7f6f6b400700 1 -- 172.21.15.12:0/70063021 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f540035b0 con 0x7f6f6c0cec70 2024-03-29T11:11:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.477+0000 7f6f70c00700 1 -- 172.21.15.12:0/70063021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f6c0d0cf0 msgr2=0x7f6f6c0b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.477+0000 7f6f70c00700 1 --2- 172.21.15.12:0/70063021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f6c0d0cf0 0x7f6f6c0b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.477+0000 7f6f70c00700 1 -- 172.21.15.12:0/70063021 >> v1:172.21.15.12:6789/0 conn(0x7f6f6c0cec70 legacy=0x7f6f6c0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.477+0000 7f6f70c00700 1 -- 172.21.15.12:0/70063021 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6c0d27c0 con 0x7f6f6c15da50 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.477+0000 7f6f70c00700 1 --2- 172.21.15.12:0/70063021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f6c15da50 0x7f6f6c0ce730 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009840 tx=0x7f6f60009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1d2e965d4696d14 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f6b400700 1 -- 172.21.15.12:0/70063021 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60011830 con 0x7f6f6c15da50 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f6b400700 1 -- 172.21.15.12:0/70063021 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f60011990 con 0x7f6f6c15da50 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f6b400700 1 -- 172.21.15.12:0/70063021 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60011d20 con 0x7f6f6c15da50 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f73600700 1 -- 172.21.15.12:0/70063021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f6c15da50 msgr2=0x7f6f6c0ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f73600700 1 --2- 172.21.15.12:0/70063021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f6c15da50 0x7f6f6c0ce730 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009840 tx=0x7f6f60009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f73600700 1 -- 172.21.15.12:0/70063021 shutdown_connections 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f73600700 1 --2- 172.21.15.12:0/70063021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f6c0d0cf0 0x7f6f6c0b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f73600700 1 --2- 172.21.15.12:0/70063021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f6c15da50 0x7f6f6c0ce730 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f73600700 1 -- 172.21.15.12:0/70063021 >> 172.21.15.12:0/70063021 conn(0x7f6f6c0666a0 msgr2=0x7f6f6c066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f73600700 1 -- 172.21.15.12:0/70063021 shutdown_connections 2024-03-29T11:11:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.478+0000 7f6f73600700 1 -- 172.21.15.12:0/70063021 wait complete. 2024-03-29T11:11:39.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.480+0000 7f6f73600700 1 Processor -- start 2024-03-29T11:11:39.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.480+0000 7f6f73600700 1 -- start start 2024-03-29T11:11:39.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f73600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f6c0cec70 0x7f6f6c1f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f73600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f6c0d0cf0 0x7f6f6c15df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f73600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f6c15da50 0x7f6f6c15f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f73600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1f30a0 con 0x7f6f6c15da50 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f73600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c14c530 con 0x7f6f6c0d0cf0 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f73600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c15e4b0 con 0x7f6f6c0cec70 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f71600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f6c15da50 0x7f6f6c15f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f71600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f6c15da50 0x7f6f6c15f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58086/0 (socket says 172.21.15.12:58086) 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f71600700 1 -- 172.21.15.12:0/4248126113 learned_addr learned my addr 172.21.15.12:0/4248126113 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f70c00700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f6c0cec70 0x7f6f6c1f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.481+0000 7f6f6be00700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f6c0d0cf0 0x7f6f6c15df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.482+0000 7f6f71600700 1 -- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f6c0cec70 msgr2=0x7f6f6c1f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.482+0000 7f6f71600700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f6c0cec70 0x7f6f6c1f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.482+0000 7f6f71600700 1 -- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f6c0d0cf0 msgr2=0x7f6f6c15df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.482+0000 7f6f71600700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f6c0d0cf0 0x7f6f6c15df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.482+0000 7f6f71600700 1 -- 172.21.15.12:0/4248126113 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f600094a0 con 0x7f6f6c15da50 2024-03-29T11:11:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.482+0000 7f6f71600700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f6c15da50 0x7f6f6c15f880 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f6f64009d20 tx=0x7f6f6400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.482+0000 7f6f68c00700 1 -- 172.21.15.12:0/4248126113 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6400cc90 con 0x7f6f6c15da50 2024-03-29T11:11:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.482+0000 7f6f68c00700 1 -- 172.21.15.12:0/4248126113 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6400f040 con 0x7f6f6c15da50 2024-03-29T11:11:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.482+0000 7f6f73600700 1 -- 172.21.15.12:0/4248126113 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6c15e750 con 0x7f6f6c15da50 2024-03-29T11:11:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.482+0000 7f6f68c00700 1 -- 172.21.15.12:0/4248126113 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64009b10 con 0x7f6f6c15da50 2024-03-29T11:11:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.483+0000 7f6f73600700 1 -- 172.21.15.12:0/4248126113 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6c15ed10 con 0x7f6f6c15da50 2024-03-29T11:11:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.483+0000 7f6f73600700 1 -- 172.21.15.12:0/4248126113 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6c15f360 con 0x7f6f6c15da50 2024-03-29T11:11:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.484+0000 7f6f68c00700 1 -- 172.21.15.12:0/4248126113 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6f64014480 con 0x7f6f6c15da50 2024-03-29T11:11:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.485+0000 7f6f68c00700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f4c06adb0 0x7f6f4c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.485+0000 7f6f68c00700 1 -- 172.21.15.12:0/4248126113 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f6f64089b00 con 0x7f6f6c15da50 2024-03-29T11:11:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.486+0000 7f6f70c00700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f4c06adb0 0x7f6f4c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.486+0000 7f6f70c00700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f4c06adb0 0x7f6f4c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6f60000c00 tx=0x7f6f6001c040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.490+0000 7f6f68c00700 1 -- 172.21.15.12:0/4248126113 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f6408be60 con 0x7f6f6c15da50 2024-03-29T11:11:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.706+0000 7f6f73600700 1 -- 172.21.15.12:0/4248126113 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f6c0ca930 con 0x7f6f6c15da50 2024-03-29T11:11:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.707+0000 7f6f68c00700 1 -- 172.21.15.12:0/4248126113 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+12469 (secure 0 0 0) 0x7f6f6c0ca930 con 0x7f6f6c15da50 2024-03-29T11:11:39.685 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:39.685 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":173,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:38.700138+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.709+0000 7f6f73600700 1 -- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f4c06adb0 msgr2=0x7f6f4c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.709+0000 7f6f73600700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f4c06adb0 0x7f6f4c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6f60000c00 tx=0x7f6f6001c040 comp rx=0 tx=0).stop 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.709+0000 7f6f73600700 1 -- 172.21.15.12:0/4248126113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f6c15da50 msgr2=0x7f6f6c15f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.709+0000 7f6f73600700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f6c15da50 0x7f6f6c15f880 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f6f64009d20 tx=0x7f6f6400c3a0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.709+0000 7f6f73600700 1 -- 172.21.15.12:0/4248126113 shutdown_connections 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.709+0000 7f6f73600700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6f4c06adb0 0x7f6f4c06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.709+0000 7f6f73600700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f6c15da50 0x7f6f6c15f880 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.709+0000 7f6f73600700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f6c0cec70 0x7f6f6c1f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.709+0000 7f6f73600700 1 --2- 172.21.15.12:0/4248126113 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f6c0d0cf0 0x7f6f6c15df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.709+0000 7f6f73600700 1 -- 172.21.15.12:0/4248126113 >> 172.21.15.12:0/4248126113 conn(0x7f6f6c0666a0 msgr2=0x7f6f6c0b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.710+0000 7f6f73600700 1 -- 172.21.15.12:0/4248126113 shutdown_connections 2024-03-29T11:11:39.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.710+0000 7f6f73600700 1 -- 172.21.15.12:0/4248126113 wait complete. 2024-03-29T11:11:39.706 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:11:39.707 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:11:39.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.930+0000 7ff45e000700 1 Processor -- start 2024-03-29T11:11:39.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.930+0000 7ff45e000700 1 -- start start 2024-03-29T11:11:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.931+0000 7ff45e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4581611b0 0x7ff45815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.931+0000 7ff45e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4580cee10 0x7ff458161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.931+0000 7ff45e000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4580ce270 con 0x7ff45815f4d0 2024-03-29T11:11:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.931+0000 7ff45e000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45815b600 con 0x7ff4580cee10 2024-03-29T11:11:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.931+0000 7ff45e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580d2760 con 0x7ff4581611b0 2024-03-29T11:11:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.931+0000 7ff457e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4580cee10 0x7ff458161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.931+0000 7ff456a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff45815f4d0 0x7ff4580ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60164/0 (socket says 172.21.15.12:60164) 2024-03-29T11:11:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.931+0000 7ff456a00700 1 -- 172.21.15.12:0/376630868 learned_addr learned my addr 172.21.15.12:0/376630868 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.931+0000 7ff457400700 1 --2- 172.21.15.12:0/376630868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4581611b0 0x7ff45815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.932+0000 7ff457e00700 1 -- 172.21.15.12:0/376630868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4581611b0 msgr2=0x7ff45815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.932+0000 7ff457e00700 1 --2- 172.21.15.12:0/376630868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4581611b0 0x7ff45815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.932+0000 7ff457e00700 1 -- 172.21.15.12:0/376630868 >> v1:172.21.15.12:6789/0 conn(0x7ff45815f4d0 legacy=0x7ff4580ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.932+0000 7ff457e00700 1 -- 172.21.15.12:0/376630868 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4580d2a40 con 0x7ff4580cee10 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.932+0000 7ff456000700 1 -- 172.21.15.12:0/376630868 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202526033 0 0) 0x7ff4580ce270 con 0x7ff45815f4d0 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.932+0000 7ff457e00700 1 --2- 172.21.15.12:0/376630868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4580cee10 0x7ff458161590 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7ff450009d90 tx=0x7ff45000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ec1597c54d7bad server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.932+0000 7ff456000700 1 -- 172.21.15.12:0/376630868 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45000e070 con 0x7ff4580cee10 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.932+0000 7ff456000700 1 -- 172.21.15.12:0/376630868 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45000bb20 con 0x7ff4580cee10 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.932+0000 7ff456000700 1 -- 172.21.15.12:0/376630868 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff450012430 con 0x7ff4580cee10 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.933+0000 7ff45e000700 1 -- 172.21.15.12:0/376630868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4580cee10 msgr2=0x7ff458161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.933+0000 7ff45e000700 1 --2- 172.21.15.12:0/376630868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4580cee10 0x7ff458161590 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7ff450009d90 tx=0x7ff45000b330 comp rx=0 tx=0).stop 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.933+0000 7ff45e000700 1 -- 172.21.15.12:0/376630868 shutdown_connections 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.933+0000 7ff45e000700 1 --2- 172.21.15.12:0/376630868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4580cee10 0x7ff458161590 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.933+0000 7ff45e000700 1 --2- 172.21.15.12:0/376630868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4581611b0 0x7ff45815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.933+0000 7ff45e000700 1 -- 172.21.15.12:0/376630868 >> 172.21.15.12:0/376630868 conn(0x7ff45808aff0 msgr2=0x7ff4580d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.933+0000 7ff45e000700 1 -- 172.21.15.12:0/376630868 shutdown_connections 2024-03-29T11:11:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.933+0000 7ff45e000700 1 -- 172.21.15.12:0/376630868 wait complete. 2024-03-29T11:11:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.935+0000 7ff45e000700 1 Processor -- start 2024-03-29T11:11:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.935+0000 7ff45e000700 1 -- start start 2024-03-29T11:11:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff45e000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4580cee10 0x7ff4580c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff45e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff45815f4d0 0x7ff4580b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff45e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4581611b0 0x7ff45815f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff45e000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580b77e0 con 0x7ff4581611b0 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff45e000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580b6bd0 con 0x7ff4580cee10 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff45e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580b6eb0 con 0x7ff45815f4d0 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff457e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4581611b0 0x7ff45815f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff457e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4581611b0 0x7ff45815f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58114/0 (socket says 172.21.15.12:58114) 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff457e00700 1 -- 172.21.15.12:0/1782632840 learned_addr learned my addr 172.21.15.12:0/1782632840 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff456a00700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff45815f4d0 0x7ff4580b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff457400700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4580cee10 0x7ff4580c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff457e00700 1 -- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff45815f4d0 msgr2=0x7ff4580b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff457e00700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff45815f4d0 0x7ff4580b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff457e00700 1 -- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4580cee10 msgr2=0x7ff4580c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff457e00700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4580cee10 0x7ff4580c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff457e00700 1 -- 172.21.15.12:0/1782632840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450009a70 con 0x7ff4581611b0 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff456a00700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff45815f4d0 0x7ff4580b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.936+0000 7ff457400700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4580cee10 0x7ff4580c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:39.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.937+0000 7ff457e00700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4581611b0 0x7ff45815f8e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7ff45808ce10 tx=0x7ff450004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:39.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.937+0000 7ff44f400700 1 -- 172.21.15.12:0/1782632840 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff450012430 con 0x7ff4581611b0 2024-03-29T11:11:39.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.937+0000 7ff44f400700 1 -- 172.21.15.12:0/1782632840 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff450020070 con 0x7ff4581611b0 2024-03-29T11:11:39.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.937+0000 7ff44f400700 1 -- 172.21.15.12:0/1782632840 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff450004280 con 0x7ff4581611b0 2024-03-29T11:11:39.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.937+0000 7ff45e000700 1 -- 172.21.15.12:0/1782632840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4580b7190 con 0x7ff4581611b0 2024-03-29T11:11:39.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.937+0000 7ff45e000700 1 -- 172.21.15.12:0/1782632840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff458160bd0 con 0x7ff4581611b0 2024-03-29T11:11:39.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.938+0000 7ff45e000700 1 -- 172.21.15.12:0/1782632840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4580b7990 con 0x7ff4581611b0 2024-03-29T11:11:39.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.939+0000 7ff44f400700 1 -- 172.21.15.12:0/1782632840 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff4500043e0 con 0x7ff4581611b0 2024-03-29T11:11:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.940+0000 7ff44f400700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff43806adb0 0x7ff43806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.940+0000 7ff44f400700 1 -- 172.21.15.12:0/1782632840 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 4901+0+0 (secure 0 0 0) 0x7ff450022070 con 0x7ff4581611b0 2024-03-29T11:11:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.941+0000 7ff457400700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff43806adb0 0x7ff43806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.941+0000 7ff457400700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff43806adb0 0x7ff43806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff448009ac0 tx=0x7ff448008040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:39.945+0000 7ff44f400700 1 -- 172.21.15.12:0/1782632840 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff4500562e0 con 0x7ff4581611b0 2024-03-29T11:11:40.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.164+0000 7ff45e000700 1 -- 172.21.15.12:0/1782632840 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff458067720 con 0x7ff4581611b0 2024-03-29T11:11:40.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.164+0000 7ff44f400700 1 -- 172.21.15.12:0/1782632840 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v173) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff45002c020 con 0x7ff4581611b0 2024-03-29T11:11:40.145 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 -- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff43806adb0 msgr2=0x7ff43806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff43806adb0 0x7ff43806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff448009ac0 tx=0x7ff448008040 comp rx=0 tx=0).stop 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 -- 172.21.15.12:0/1782632840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4581611b0 msgr2=0x7ff45815f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4581611b0 0x7ff45815f8e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7ff45808ce10 tx=0x7ff450004ad0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 -- 172.21.15.12:0/1782632840 shutdown_connections 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff43806adb0 0x7ff43806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4581611b0 0x7ff45815f8e0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4580cee10 0x7ff4580c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 --2- 172.21.15.12:0/1782632840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff45815f4d0 0x7ff4580b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 -- 172.21.15.12:0/1782632840 >> 172.21.15.12:0/1782632840 conn(0x7ff45808aff0 msgr2=0x7ff458159820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 -- 172.21.15.12:0/1782632840 shutdown_connections 2024-03-29T11:11:40.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.167+0000 7ff45e000700 1 -- 172.21.15.12:0/1782632840 wait complete. 2024-03-29T11:11:40.164 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T11:11:40.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.384+0000 7f3b39a00700 1 Processor -- start 2024-03-29T11:11:40.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.384+0000 7f3b39a00700 1 -- start start 2024-03-29T11:11:40.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.385+0000 7f3b39a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b34072580 0x7f3b340b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.385+0000 7f3b39a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b340b2e50 0x7f3b340709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.385+0000 7f3b39a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b340b2180 con 0x7f3b340b3af0 2024-03-29T11:11:40.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.385+0000 7f3b39a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3415b100 con 0x7f3b340b2e50 2024-03-29T11:11:40.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.385+0000 7f3b39a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3415b3e0 con 0x7f3b34072580 2024-03-29T11:11:40.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.385+0000 7f3b32a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3b340b3af0 0x7f3b3415a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60172/0 (socket says 172.21.15.12:60172) 2024-03-29T11:11:40.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.385+0000 7f3b32a00700 1 -- 172.21.15.12:0/153180557 learned_addr learned my addr 172.21.15.12:0/153180557 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:40.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.385+0000 7f3b33400700 1 --2- 172.21.15.12:0/153180557 >> v2:172.21.15.12:3300/0 conn(0x7f3b340b2e50 0x7f3b340709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.386+0000 7f3b32000700 1 --2- 172.21.15.12:0/153180557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b34072580 0x7f3b340b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.386+0000 7f3b31600700 1 -- 172.21.15.12:0/153180557 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634882608 0 0) 0x7f3b340b2180 con 0x7f3b340b3af0 2024-03-29T11:11:40.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.386+0000 7f3b31600700 1 -- 172.21.15.12:0/153180557 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b180057c0 con 0x7f3b340b3af0 2024-03-29T11:11:40.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.386+0000 7f3b33400700 1 -- 172.21.15.12:0/153180557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b34072580 msgr2=0x7f3b340b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:40.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.386+0000 7f3b33400700 1 --2- 172.21.15.12:0/153180557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b34072580 0x7f3b340b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.386+0000 7f3b33400700 1 -- 172.21.15.12:0/153180557 >> v1:172.21.15.12:6789/0 conn(0x7f3b340b3af0 legacy=0x7f3b3415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.386+0000 7f3b33400700 1 -- 172.21.15.12:0/153180557 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b340b23f0 con 0x7f3b340b2e50 2024-03-29T11:11:40.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.386+0000 7f3b33400700 1 --2- 172.21.15.12:0/153180557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b2e50 0x7f3b340709c0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009d90 tx=0x7f3b2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcdca78bc1888582 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:40.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.386+0000 7f3b31600700 1 -- 172.21.15.12:0/153180557 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c00e070 con 0x7f3b340b2e50 2024-03-29T11:11:40.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.387+0000 7f3b31600700 1 -- 172.21.15.12:0/153180557 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c00bb20 con 0x7f3b340b2e50 2024-03-29T11:11:40.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.387+0000 7f3b31600700 1 -- 172.21.15.12:0/153180557 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c012430 con 0x7f3b340b2e50 2024-03-29T11:11:40.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.387+0000 7f3b39a00700 1 -- 172.21.15.12:0/153180557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b2e50 msgr2=0x7f3b340709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.387+0000 7f3b39a00700 1 --2- 172.21.15.12:0/153180557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b2e50 0x7f3b340709c0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009d90 tx=0x7f3b2c00b330 comp rx=0 tx=0).stop 2024-03-29T11:11:40.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.387+0000 7f3b39a00700 1 -- 172.21.15.12:0/153180557 shutdown_connections 2024-03-29T11:11:40.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.387+0000 7f3b39a00700 1 --2- 172.21.15.12:0/153180557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b2e50 0x7f3b340709c0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.387+0000 7f3b39a00700 1 --2- 172.21.15.12:0/153180557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b34072580 0x7f3b340b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.387+0000 7f3b39a00700 1 -- 172.21.15.12:0/153180557 >> 172.21.15.12:0/153180557 conn(0x7f3b340cf190 msgr2=0x7f3b340cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:40.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.387+0000 7f3b39a00700 1 -- 172.21.15.12:0/153180557 shutdown_connections 2024-03-29T11:11:40.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.387+0000 7f3b39a00700 1 -- 172.21.15.12:0/153180557 wait complete. 2024-03-29T11:11:40.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.389+0000 7f3b39a00700 1 Processor -- start 2024-03-29T11:11:40.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.389+0000 7f3b39a00700 1 -- start start 2024-03-29T11:11:40.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.390+0000 7f3b39a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b34072580 0x7f3b34151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.390+0000 7f3b39a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b340b2e50 0x7f3b340aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.390+0000 7f3b39a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b3af0 0x7f3b3414b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.390+0000 7f3b39a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34157500 con 0x7f3b340b3af0 2024-03-29T11:11:40.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.390+0000 7f3b39a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34156880 con 0x7f3b340b2e50 2024-03-29T11:11:40.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.390+0000 7f3b39a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34156b60 con 0x7f3b34072580 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b33400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b3af0 0x7f3b3414b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b32000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b340b2e50 0x7f3b340aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b33400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b3af0 0x7f3b3414b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58140/0 (socket says 172.21.15.12:58140) 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b33400700 1 -- 172.21.15.12:0/1320249356 learned_addr learned my addr 172.21.15.12:0/1320249356 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b32a00700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b34072580 0x7f3b34151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b33400700 1 -- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b34072580 msgr2=0x7f3b34151650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b33400700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b34072580 0x7f3b34151650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b33400700 1 -- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b340b2e50 msgr2=0x7f3b340aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b33400700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b340b2e50 0x7f3b340aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b33400700 1 -- 172.21.15.12:0/1320249356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c009a70 con 0x7f3b340b3af0 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b33400700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b3af0 0x7f3b3414b5a0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c014040 tx=0x7f3b2c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.391+0000 7f3b2aa00700 1 -- 172.21.15.12:0/1320249356 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c0129f0 con 0x7f3b340b3af0 2024-03-29T11:11:40.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.392+0000 7f3b2aa00700 1 -- 172.21.15.12:0/1320249356 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c020070 con 0x7f3b340b3af0 2024-03-29T11:11:40.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.392+0000 7f3b39a00700 1 -- 172.21.15.12:0/1320249356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b34156e40 con 0x7f3b340b3af0 2024-03-29T11:11:40.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.392+0000 7f3b2aa00700 1 -- 172.21.15.12:0/1320249356 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c012db0 con 0x7f3b340b3af0 2024-03-29T11:11:40.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.392+0000 7f3b39a00700 1 -- 172.21.15.12:0/1320249356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b340b0450 con 0x7f3b340b3af0 2024-03-29T11:11:40.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.393+0000 7f3b39a00700 1 -- 172.21.15.12:0/1320249356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3406b680 con 0x7f3b340b3af0 2024-03-29T11:11:40.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.394+0000 7f3b2aa00700 1 -- 172.21.15.12:0/1320249356 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3b2c004040 con 0x7f3b340b3af0 2024-03-29T11:11:40.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.394+0000 7f3b2aa00700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b1406adb0 0x7f3b1406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.395+0000 7f3b2aa00700 1 -- 172.21.15.12:0/1320249356 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f3b2c022070 con 0x7f3b340b3af0 2024-03-29T11:11:40.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.395+0000 7f3b32a00700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b1406adb0 0x7f3b1406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.396+0000 7f3b32a00700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b1406adb0 0x7f3b1406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3b240011e0 tx=0x7f3b24009040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:40.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.399+0000 7f3b2aa00700 1 -- 172.21.15.12:0/1320249356 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3b2c0561a0 con 0x7f3b340b3af0 2024-03-29T11:11:40.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.615+0000 7f3b39a00700 1 -- 172.21.15.12:0/1320249356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3b34067720 con 0x7f3b340b3af0 2024-03-29T11:11:40.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.616+0000 7f3b2aa00700 1 -- 172.21.15.12:0/1320249356 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v173) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3b2c02c090 con 0x7f3b340b3af0 2024-03-29T11:11:40.595 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:40.595 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:11:40.595 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:40.595 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T11:11:40.595 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T11:11:40.595 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:40.595 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:11:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 -- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b1406adb0 msgr2=0x7f3b1406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b1406adb0 0x7f3b1406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3b240011e0 tx=0x7f3b24009040 comp rx=0 tx=0).stop 2024-03-29T11:11:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 -- 172.21.15.12:0/1320249356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b3af0 msgr2=0x7f3b3414b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b3af0 0x7f3b3414b5a0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c014040 tx=0x7f3b2c004ad0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 -- 172.21.15.12:0/1320249356 shutdown_connections 2024-03-29T11:11:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3b1406adb0 0x7f3b1406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b340b3af0 0x7f3b3414b5a0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b34072580 0x7f3b34151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 --2- 172.21.15.12:0/1320249356 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b340b2e50 0x7f3b340aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 -- 172.21.15.12:0/1320249356 >> 172.21.15.12:0/1320249356 conn(0x7f3b340cf190 msgr2=0x7f3b340cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 -- 172.21.15.12:0/1320249356 shutdown_connections 2024-03-29T11:11:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.618+0000 7f3b39a00700 1 -- 172.21.15.12:0/1320249356 wait complete. 2024-03-29T11:11:40.614 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T11:11:40.614 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T11:11:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.837+0000 7f4bd7000700 1 Processor -- start 2024-03-29T11:11:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.838+0000 7f4bd7000700 1 -- start start 2024-03-29T11:11:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.839+0000 7f4bd7000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4bd006c910 0x7f4bd006dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.839+0000 7f4bd7000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd006da50 con 0x7f4bd0070980 2024-03-29T11:11:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.839+0000 7f4bd7000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd006ce40 con 0x7f4bd015dce0 2024-03-29T11:11:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.839+0000 7f4bd7000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd006d120 con 0x7f4bd006c910 2024-03-29T11:11:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.839+0000 7f4bcf400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4bd006c910 0x7f4bd006dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.839+0000 7f4bcfe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4bd0070980 0x7f4bd0070d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60178/0 (socket says 172.21.15.12:60178) 2024-03-29T11:11:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.839+0000 7f4bcfe00700 1 -- 172.21.15.12:0/4285221830 learned_addr learned my addr 172.21.15.12:0/4285221830 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:40.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.840+0000 7f4bcf400700 1 -- 172.21.15.12:0/4285221830 >> v1:172.21.15.159:6789/0 conn(0x7f4bd015dce0 legacy=0x7f4bd00b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.840+0000 7f4bcf400700 1 -- 172.21.15.12:0/4285221830 >> v1:172.21.15.12:6789/0 conn(0x7f4bd0070980 legacy=0x7f4bd0070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.840+0000 7f4bcf400700 1 -- 172.21.15.12:0/4285221830 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd006d400 con 0x7f4bd006c910 2024-03-29T11:11:40.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.840+0000 7f4bcea00700 1 -- 172.21.15.12:0/4285221830 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457861129 0 0) 0x7f4bd006da50 con 0x7f4bd0070980 2024-03-29T11:11:40.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.840+0000 7f4bcf400700 1 --2- 172.21.15.12:0/4285221830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd006c910 0x7f4bd006dee0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009840 tx=0x7f4bbc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a473fc2ae421ad server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:40.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.840+0000 7f4bcea00700 1 -- 172.21.15.12:0/4285221830 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc011830 con 0x7f4bd006c910 2024-03-29T11:11:40.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.840+0000 7f4bcea00700 1 -- 172.21.15.12:0/4285221830 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc011990 con 0x7f4bd006c910 2024-03-29T11:11:40.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.840+0000 7f4bcea00700 1 -- 172.21.15.12:0/4285221830 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc011d20 con 0x7f4bd006c910 2024-03-29T11:11:40.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.841+0000 7f4bd7000700 1 -- 172.21.15.12:0/4285221830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd006c910 msgr2=0x7f4bd006dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.841+0000 7f4bd7000700 1 --2- 172.21.15.12:0/4285221830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd006c910 0x7f4bd006dee0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009840 tx=0x7f4bbc009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:40.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.841+0000 7f4bd7000700 1 -- 172.21.15.12:0/4285221830 shutdown_connections 2024-03-29T11:11:40.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.841+0000 7f4bd7000700 1 --2- 172.21.15.12:0/4285221830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd006c910 0x7f4bd006dee0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.841+0000 7f4bd7000700 1 -- 172.21.15.12:0/4285221830 >> 172.21.15.12:0/4285221830 conn(0x7f4bd00cf210 msgr2=0x7f4bd00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:40.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.841+0000 7f4bd7000700 1 -- 172.21.15.12:0/4285221830 shutdown_connections 2024-03-29T11:11:40.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.841+0000 7f4bd7000700 1 -- 172.21.15.12:0/4285221830 wait complete. 2024-03-29T11:11:40.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.843+0000 7f4bd7000700 1 Processor -- start 2024-03-29T11:11:40.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.843+0000 7f4bd7000700 1 -- start start 2024-03-29T11:11:40.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd7000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4bd006c910 0x7f4bd00cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd7000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4bd0070980 0x7f4bd0156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd7000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd015dce0 0x7f4bd009bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd7000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd00992e0 con 0x7f4bd015dce0 2024-03-29T11:11:40.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd7000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd0156b50 con 0x7f4bd006c910 2024-03-29T11:11:40.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd7000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd015ac00 con 0x7f4bd0070980 2024-03-29T11:11:40.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd5000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd015dce0 0x7f4bd009bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd5000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd015dce0 0x7f4bd009bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58164/0 (socket says 172.21.15.12:58164) 2024-03-29T11:11:40.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd5000700 1 -- 172.21.15.12:0/3313877493 learned_addr learned my addr 172.21.15.12:0/3313877493 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bcfe00700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4bd006c910 0x7f4bd00cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bcf400700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4bd0070980 0x7f4bd0156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd5000700 1 -- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4bd0070980 msgr2=0x7f4bd0156610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd5000700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4bd0070980 0x7f4bd0156610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd5000700 1 -- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4bd006c910 msgr2=0x7f4bd00cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd5000700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4bd006c910 0x7f4bd00cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bd5000700 1 -- 172.21.15.12:0/3313877493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0094a0 con 0x7f4bd015dce0 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.844+0000 7f4bcf400700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4bd0070980 0x7f4bd0156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.845+0000 7f4bd5000700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd015dce0 0x7f4bd009bfc0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800cb50 tx=0x7f4bc800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.845+0000 7f4bc7e00700 1 -- 172.21.15.12:0/3313877493 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8003c00 con 0x7f4bd015dce0 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.845+0000 7f4bc7e00700 1 -- 172.21.15.12:0/3313877493 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc8003d60 con 0x7f4bd015dce0 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.845+0000 7f4bd7000700 1 -- 172.21.15.12:0/3313877493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd0099530 con 0x7f4bd015dce0 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.845+0000 7f4bc7e00700 1 -- 172.21.15.12:0/3313877493 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800ab70 con 0x7f4bd015dce0 2024-03-29T11:11:40.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.845+0000 7f4bd7000700 1 -- 172.21.15.12:0/3313877493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd0099af0 con 0x7f4bd015dce0 2024-03-29T11:11:40.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.846+0000 7f4bd7000700 1 -- 172.21.15.12:0/3313877493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd0067720 con 0x7f4bd015dce0 2024-03-29T11:11:40.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.847+0000 7f4bc7e00700 1 -- 172.21.15.12:0/3313877493 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4bc800acd0 con 0x7f4bd015dce0 2024-03-29T11:11:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.848+0000 7f4bc7e00700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bb006adb0 0x7f4bb006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.848+0000 7f4bc7e00700 1 -- 172.21.15.12:0/3313877493 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f4bc8089260 con 0x7f4bd015dce0 2024-03-29T11:11:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.848+0000 7f4bcfe00700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bb006adb0 0x7f4bb006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.849+0000 7f4bcfe00700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bb006adb0 0x7f4bb006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0007fd0 tx=0x7f4bc0007c20 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:40.852+0000 7f4bc7e00700 1 -- 172.21.15.12:0/3313877493 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4bc80545b0 con 0x7f4bd015dce0 2024-03-29T11:11:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.065+0000 7f4bd7000700 1 -- 172.21.15.12:0/3313877493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4bd00b4470 con 0x7f4bd015dce0 2024-03-29T11:11:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.138+0000 7f4bc7e00700 1 -- 172.21.15.12:0/3313877493 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v174) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4bc8057bd0 con 0x7f4bd015dce0 2024-03-29T11:11:41.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 -- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bb006adb0 msgr2=0x7f4bb006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bb006adb0 0x7f4bb006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0007fd0 tx=0x7f4bc0007c20 comp rx=0 tx=0).stop 2024-03-29T11:11:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 -- 172.21.15.12:0/3313877493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd015dce0 msgr2=0x7f4bd009bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd015dce0 0x7f4bd009bfc0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800cb50 tx=0x7f4bc800cf10 comp rx=0 tx=0).stop 2024-03-29T11:11:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 -- 172.21.15.12:0/3313877493 shutdown_connections 2024-03-29T11:11:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f4bb006adb0 0x7f4bb006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bd015dce0 0x7f4bd009bfc0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4bd006c910 0x7f4bd00cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 --2- 172.21.15.12:0/3313877493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4bd0070980 0x7f4bd0156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 -- 172.21.15.12:0/3313877493 >> 172.21.15.12:0/3313877493 conn(0x7f4bd00cf210 msgr2=0x7f4bd015cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 -- 172.21.15.12:0/3313877493 shutdown_connections 2024-03-29T11:11:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.142+0000 7f4bd7000700 1 -- 172.21.15.12:0/3313877493 wait complete. 2024-03-29T11:11:41.123 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T11:11:41.142 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T11:11:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.369+0000 7f1762000700 1 Processor -- start 2024-03-29T11:11:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.369+0000 7f1762000700 1 -- start start 2024-03-29T11:11:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.370+0000 7f1762000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f175c15d9d0 0x7f175c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.370+0000 7f1762000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f175c0d2540 0x7f175c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.370+0000 7f1762000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175c161420 con 0x7f175c0d16a0 2024-03-29T11:11:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.370+0000 7f1762000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f175c15edc0 con 0x7f175c15d9d0 2024-03-29T11:11:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.370+0000 7f1762000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f175c161700 con 0x7f175c0d2540 2024-03-29T11:11:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.370+0000 7f175aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f175c0d16a0 0x7f175c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60180/0 (socket says 172.21.15.12:60180) 2024-03-29T11:11:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.370+0000 7f175aa00700 1 -- 172.21.15.12:0/2423693608 learned_addr learned my addr 172.21.15.12:0/2423693608 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.370+0000 7f175b400700 1 --2- 172.21.15.12:0/2423693608 >> v2:172.21.15.12:3300/0 conn(0x7f175c15d9d0 0x7f175c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175be00700 1 --2- 172.21.15.12:0/2423693608 >> v2:172.21.15.159:3300/0 conn(0x7f175c0d2540 0x7f175c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175a000700 1 -- 172.21.15.12:0/2423693608 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201339776 0 0) 0x7f175c161420 con 0x7f175c0d16a0 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175a000700 1 -- 172.21.15.12:0/2423693608 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17400035b0 con 0x7f175c0d16a0 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175b400700 1 -- 172.21.15.12:0/2423693608 >> v2:172.21.15.159:3300/0 conn(0x7f175c0d2540 msgr2=0x7f175c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175b400700 1 --2- 172.21.15.12:0/2423693608 >> v2:172.21.15.159:3300/0 conn(0x7f175c0d2540 0x7f175c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175b400700 1 -- 172.21.15.12:0/2423693608 >> v1:172.21.15.12:6789/0 conn(0x7f175c0d16a0 legacy=0x7f175c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175b400700 1 -- 172.21.15.12:0/2423693608 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c1619e0 con 0x7f175c15d9d0 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175b400700 1 --2- 172.21.15.12:0/2423693608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c15d9d0 0x7f175c0d3c30 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f175c08ca10 tx=0x7f174c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe6a64f5fca89b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175a000700 1 -- 172.21.15.12:0/2423693608 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c011830 con 0x7f175c15d9d0 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175a000700 1 -- 172.21.15.12:0/2423693608 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c011990 con 0x7f175c15d9d0 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.371+0000 7f175a000700 1 -- 172.21.15.12:0/2423693608 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c011d20 con 0x7f175c15d9d0 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.372+0000 7f1762000700 1 -- 172.21.15.12:0/2423693608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c15d9d0 msgr2=0x7f175c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.372+0000 7f1762000700 1 --2- 172.21.15.12:0/2423693608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c15d9d0 0x7f175c0d3c30 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f175c08ca10 tx=0x7f174c009b20 comp rx=0 tx=0).stop 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.372+0000 7f1762000700 1 -- 172.21.15.12:0/2423693608 shutdown_connections 2024-03-29T11:11:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.372+0000 7f1762000700 1 --2- 172.21.15.12:0/2423693608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c15d9d0 0x7f175c0d3c30 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.372+0000 7f1762000700 1 --2- 172.21.15.12:0/2423693608 >> v2:172.21.15.159:3300/0 conn(0x7f175c0d2540 0x7f175c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.372+0000 7f1762000700 1 -- 172.21.15.12:0/2423693608 >> 172.21.15.12:0/2423693608 conn(0x7f175c08aff0 msgr2=0x7f175c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.372+0000 7f1762000700 1 -- 172.21.15.12:0/2423693608 shutdown_connections 2024-03-29T11:11:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.372+0000 7f1762000700 1 -- 172.21.15.12:0/2423693608 wait complete. 2024-03-29T11:11:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.374+0000 7f1762000700 1 Processor -- start 2024-03-29T11:11:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.374+0000 7f1762000700 1 -- start start 2024-03-29T11:11:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f1762000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c0d16a0 0x7f175c0b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f1762000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f175c0d2540 0x7f175c1528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f1762000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f175c15d9d0 0x7f175c0b84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f1762000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c0b6560 con 0x7f175c0d16a0 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f1762000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c0b6770 con 0x7f175c0d2540 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f1762000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175c0b6a50 con 0x7f175c15d9d0 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f175b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c0d16a0 0x7f175c0b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f175b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c0d16a0 0x7f175c0b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58174/0 (socket says 172.21.15.12:58174) 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f175b400700 1 -- 172.21.15.12:0/2664603577 learned_addr learned my addr 172.21.15.12:0/2664603577 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f175aa00700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f175c0d2540 0x7f175c1528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.375+0000 7f175be00700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f175c15d9d0 0x7f175c0b84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f175b400700 1 -- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f175c15d9d0 msgr2=0x7f175c0b84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f175b400700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f175c15d9d0 0x7f175c0b84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f175b400700 1 -- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f175c0d2540 msgr2=0x7f175c1528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f175b400700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f175c0d2540 0x7f175c1528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f175b400700 1 -- 172.21.15.12:0/2664603577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c0094a0 con 0x7f175c0d16a0 2024-03-29T11:11:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f175be00700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f175c15d9d0 0x7f175c0b84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:11:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f175b400700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c0d16a0 0x7f175c0b6020 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f174c000c00 tx=0x7f174c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f1753400700 1 -- 172.21.15.12:0/2664603577 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c0036a0 con 0x7f175c0d16a0 2024-03-29T11:11:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f1753400700 1 -- 172.21.15.12:0/2664603577 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c01e070 con 0x7f175c0d16a0 2024-03-29T11:11:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f1753400700 1 -- 172.21.15.12:0/2664603577 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c003a60 con 0x7f175c0d16a0 2024-03-29T11:11:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f1762000700 1 -- 172.21.15.12:0/2664603577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f175c0b6d30 con 0x7f175c0d16a0 2024-03-29T11:11:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.376+0000 7f1762000700 1 -- 172.21.15.12:0/2664603577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f175c0b72f0 con 0x7f175c0d16a0 2024-03-29T11:11:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.377+0000 7f1762000700 1 -- 172.21.15.12:0/2664603577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f175c0d1ca0 con 0x7f175c0d16a0 2024-03-29T11:11:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.379+0000 7f1753400700 1 -- 172.21.15.12:0/2664603577 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f174c003bc0 con 0x7f175c0d16a0 2024-03-29T11:11:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.380+0000 7f1753400700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f173c06adb0 0x7f173c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.380+0000 7f1753400700 1 -- 172.21.15.12:0/2664603577 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5271+0+0 (secure 0 0 0) 0x7f174c08a4d0 con 0x7f175c0d16a0 2024-03-29T11:11:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.380+0000 7f175aa00700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f173c06adb0 0x7f173c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.381+0000 7f175aa00700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f173c06adb0 0x7f173c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f175c0cdbd0 tx=0x7f1748007c70 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.384+0000 7f1753400700 1 -- 172.21.15.12:0/2664603577 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f174c0558c0 con 0x7f175c0d16a0 2024-03-29T11:11:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:41.594+0000 7f1762000700 1 -- 172.21.15.12:0/2664603577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f175c067720 con 0x7f175c0d16a0 2024-03-29T11:11:42.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.146+0000 7f1753400700 1 -- 172.21.15.12:0/2664603577 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v175) v1 ==== 170+0+0 (secure 0 0 0) 0x7f174c058ee0 con 0x7f175c0d16a0 2024-03-29T11:11:42.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.153+0000 7f1762000700 1 -- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f173c06adb0 msgr2=0x7f173c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.153+0000 7f1762000700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f173c06adb0 0x7f173c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f175c0cdbd0 tx=0x7f1748007c70 comp rx=0 tx=0).stop 2024-03-29T11:11:42.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.153+0000 7f1762000700 1 -- 172.21.15.12:0/2664603577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c0d16a0 msgr2=0x7f175c0b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.153+0000 7f1762000700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c0d16a0 0x7f175c0b6020 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f174c000c00 tx=0x7f174c004750 comp rx=0 tx=0).stop 2024-03-29T11:11:42.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.155+0000 7f1762000700 1 -- 172.21.15.12:0/2664603577 shutdown_connections 2024-03-29T11:11:42.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.155+0000 7f1762000700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f173c06adb0 0x7f173c06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.155+0000 7f1762000700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f175c15d9d0 0x7f175c0b84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.155+0000 7f1762000700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f175c0d16a0 0x7f175c0b6020 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.155+0000 7f1762000700 1 --2- 172.21.15.12:0/2664603577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f175c0d2540 0x7f175c1528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.155+0000 7f1762000700 1 -- 172.21.15.12:0/2664603577 >> 172.21.15.12:0/2664603577 conn(0x7f175c08aff0 msgr2=0x7f175c0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:42.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.157+0000 7f1762000700 1 -- 172.21.15.12:0/2664603577 shutdown_connections 2024-03-29T11:11:42.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.157+0000 7f1762000700 1 -- 172.21.15.12:0/2664603577 wait complete. 2024-03-29T11:11:42.138 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T11:11:42.163 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T11:11:42.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.388+0000 7f0fb8600700 1 Processor -- start 2024-03-29T11:11:42.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.388+0000 7f0fb8600700 1 -- start start 2024-03-29T11:11:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.389+0000 7f0fb8600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0fb015d9d0 0x7f0fb00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.389+0000 7f0fb8600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb0161420 con 0x7f0fb00d2540 2024-03-29T11:11:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.389+0000 7f0fb8600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb0161700 con 0x7f0fb00d16a0 2024-03-29T11:11:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.389+0000 7f0fb8600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb015edc0 con 0x7f0fb015d9d0 2024-03-29T11:11:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.390+0000 7f0fb5c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0fb015d9d0 0x7f0fb00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.390+0000 7f0fb6600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0fb00d2540 0x7f0fb00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60188/0 (socket says 172.21.15.12:60188) 2024-03-29T11:11:42.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.390+0000 7f0fb6600700 1 -- 172.21.15.12:0/2679142417 learned_addr learned my addr 172.21.15.12:0/2679142417 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:42.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.390+0000 7f0fb5c00700 1 -- 172.21.15.12:0/2679142417 >> v1:172.21.15.159:6789/0 conn(0x7f0fb00d16a0 legacy=0x7f0fb00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.390+0000 7f0fb5c00700 1 -- 172.21.15.12:0/2679142417 >> v1:172.21.15.12:6789/0 conn(0x7f0fb00d2540 legacy=0x7f0fb00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.390+0000 7f0fb5c00700 1 -- 172.21.15.12:0/2679142417 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb01619e0 con 0x7f0fb015d9d0 2024-03-29T11:11:42.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.390+0000 7f0fafe00700 1 -- 172.21.15.12:0/2679142417 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075409345 0 0) 0x7f0fb0161420 con 0x7f0fb00d2540 2024-03-29T11:11:42.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.391+0000 7f0fb5c00700 1 --2- 172.21.15.12:0/2679142417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 0x7f0fb00d3c30 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009840 tx=0x7f0fa4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed955a4d1a654718 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:42.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.391+0000 7f0fafe00700 1 -- 172.21.15.12:0/2679142417 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4011830 con 0x7f0fb015d9d0 2024-03-29T11:11:42.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.391+0000 7f0fafe00700 1 -- 172.21.15.12:0/2679142417 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa4011990 con 0x7f0fb015d9d0 2024-03-29T11:11:42.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.391+0000 7f0fafe00700 1 -- 172.21.15.12:0/2679142417 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4011d20 con 0x7f0fb015d9d0 2024-03-29T11:11:42.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.391+0000 7f0fb8600700 1 -- 172.21.15.12:0/2679142417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 msgr2=0x7f0fb00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.391+0000 7f0fb8600700 1 --2- 172.21.15.12:0/2679142417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 0x7f0fb00d3c30 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009840 tx=0x7f0fa4009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.391+0000 7f0fb8600700 1 -- 172.21.15.12:0/2679142417 shutdown_connections 2024-03-29T11:11:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.391+0000 7f0fb8600700 1 --2- 172.21.15.12:0/2679142417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 0x7f0fb00d3c30 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.391+0000 7f0fb8600700 1 -- 172.21.15.12:0/2679142417 >> 172.21.15.12:0/2679142417 conn(0x7f0fb008aff0 msgr2=0x7f0fb00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.391+0000 7f0fb8600700 1 -- 172.21.15.12:0/2679142417 shutdown_connections 2024-03-29T11:11:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.392+0000 7f0fb8600700 1 -- 172.21.15.12:0/2679142417 wait complete. 2024-03-29T11:11:42.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.394+0000 7f0fb8600700 1 Processor -- start 2024-03-29T11:11:42.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.394+0000 7f0fb8600700 1 -- start start 2024-03-29T11:11:42.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.395+0000 7f0fb8600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fb00d16a0 0x7f0fb00caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.395+0000 7f0fb8600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb00d2540 0x7f0fb00b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.395+0000 7f0fb8600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 0x7f0fb00b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.395+0000 7f0fb8600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb01e8270 con 0x7f0fb015d9d0 2024-03-29T11:11:42.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.395+0000 7f0fb8600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb00caff0 con 0x7f0fb00d2540 2024-03-29T11:11:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.395+0000 7f0fb8600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb01e7660 con 0x7f0fb00d16a0 2024-03-29T11:11:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb6600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 0x7f0fb00b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb6600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 0x7f0fb00b85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58192/0 (socket says 172.21.15.12:58192) 2024-03-29T11:11:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb6600700 1 -- 172.21.15.12:0/4111202423 learned_addr learned my addr 172.21.15.12:0/4111202423 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb5200700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb00d2540 0x7f0fb00b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb5c00700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fb00d16a0 0x7f0fb00caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb6600700 1 -- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fb00d16a0 msgr2=0x7f0fb00caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb6600700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fb00d16a0 0x7f0fb00caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb6600700 1 -- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb00d2540 msgr2=0x7f0fb00b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb6600700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb00d2540 0x7f0fb00b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb6600700 1 -- 172.21.15.12:0/4111202423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa40094a0 con 0x7f0fb015d9d0 2024-03-29T11:11:42.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb5200700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb00d2540 0x7f0fb00b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:11:42.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fb6600700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 0x7f0fb00b85a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800cc70 tx=0x7f0fa800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:42.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fad600700 1 -- 172.21.15.12:0/4111202423 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa800ad80 con 0x7f0fb015d9d0 2024-03-29T11:11:42.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.396+0000 7f0fad600700 1 -- 172.21.15.12:0/4111202423 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa800ce70 con 0x7f0fb015d9d0 2024-03-29T11:11:42.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.397+0000 7f0fb8600700 1 -- 172.21.15.12:0/4111202423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb01e7940 con 0x7f0fb015d9d0 2024-03-29T11:11:42.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.397+0000 7f0fad600700 1 -- 172.21.15.12:0/4111202423 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80072f0 con 0x7f0fb015d9d0 2024-03-29T11:11:42.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.397+0000 7f0fb8600700 1 -- 172.21.15.12:0/4111202423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb01e7f00 con 0x7f0fb015d9d0 2024-03-29T11:11:42.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.398+0000 7f0fb8600700 1 -- 172.21.15.12:0/4111202423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb00d1ca0 con 0x7f0fb015d9d0 2024-03-29T11:11:42.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.399+0000 7f0fad600700 1 -- 172.21.15.12:0/4111202423 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0fa8007450 con 0x7f0fb015d9d0 2024-03-29T11:11:42.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.400+0000 7f0fad600700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9006adb0 0x7f0f9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.400+0000 7f0fad600700 1 -- 172.21.15.12:0/4111202423 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f0fa80893d0 con 0x7f0fb015d9d0 2024-03-29T11:11:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.400+0000 7f0fb5c00700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9006adb0 0x7f0f9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.401+0000 7f0fb5c00700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9006adb0 0x7f0f9006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009790 tx=0x7f0fa401c040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:42.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.405+0000 7f0fad600700 1 -- 172.21.15.12:0/4111202423 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0fa8054430 con 0x7f0fb015d9d0 2024-03-29T11:11:42.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.672+0000 7f0fb8600700 1 -- 172.21.15.12:0/4111202423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0fb00d3c30 con 0x7f0fb015d9d0 2024-03-29T11:11:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.681+0000 7f0fad600700 1 -- 172.21.15.12:0/4111202423 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v77) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0fa8057a50 con 0x7f0fb015d9d0 2024-03-29T11:11:42.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.689+0000 7f0fb8600700 1 -- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9006adb0 msgr2=0x7f0f9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.689+0000 7f0fb8600700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9006adb0 0x7f0f9006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009790 tx=0x7f0fa401c040 comp rx=0 tx=0).stop 2024-03-29T11:11:42.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.689+0000 7f0fb8600700 1 -- 172.21.15.12:0/4111202423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 msgr2=0x7f0fb00b85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.689+0000 7f0fb8600700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 0x7f0fb00b85a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800cc70 tx=0x7f0fa800a340 comp rx=0 tx=0).stop 2024-03-29T11:11:42.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.692+0000 7f0fb8600700 1 -- 172.21.15.12:0/4111202423 shutdown_connections 2024-03-29T11:11:42.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.692+0000 7f0fb8600700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9006adb0 0x7f0f9006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.692+0000 7f0fb8600700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fb015d9d0 0x7f0fb00b85a0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.692+0000 7f0fb8600700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fb00d16a0 0x7f0fb00caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.692+0000 7f0fb8600700 1 --2- 172.21.15.12:0/4111202423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fb00d2540 0x7f0fb00b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.692+0000 7f0fb8600700 1 -- 172.21.15.12:0/4111202423 >> 172.21.15.12:0/4111202423 conn(0x7f0fb008aff0 msgr2=0x7f0fb00ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.694+0000 7f0fb8600700 1 -- 172.21.15.12:0/4111202423 shutdown_connections 2024-03-29T11:11:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.694+0000 7f0fb8600700 1 -- 172.21.15.12:0/4111202423 wait complete. 2024-03-29T11:11:42.680 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 20 and data pool 21 2024-03-29T11:11:42.703 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.924+0000 7f660e000700 1 Processor -- start 2024-03-29T11:11:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.924+0000 7f660e000700 1 -- start start 2024-03-29T11:11:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.925+0000 7f660e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f660809aba0 0x7f660806bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.925+0000 7f660e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660806c480 0x7f66080ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.925+0000 7f660e000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f660806bb40 con 0x7f660815d710 2024-03-29T11:11:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.925+0000 7f660e000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f660806c910 con 0x7f660809aba0 2024-03-29T11:11:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.925+0000 7f660e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6608099b00 con 0x7f660806c480 2024-03-29T11:11:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.925+0000 7f6606a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f660809aba0 0x7f660806bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.925+0000 7f6606a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f660809aba0 0x7f660806bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58196/0 (socket says 172.21.15.12:58196) 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.925+0000 7f6606a00700 1 -- 172.21.15.12:0/3217934166 learned_addr learned my addr 172.21.15.12:0/3217934166 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.926+0000 7f6607e00700 1 --2- 172.21.15.12:0/3217934166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660806c480 0x7f66080ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.926+0000 7f6606a00700 1 -- 172.21.15.12:0/3217934166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660806c480 msgr2=0x7f66080ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.926+0000 7f6606a00700 1 --2- 172.21.15.12:0/3217934166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660806c480 0x7f66080ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.926+0000 7f6606a00700 1 -- 172.21.15.12:0/3217934166 >> v1:172.21.15.159:6789/0 conn(0x7f660815d710 legacy=0x7f66080d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.926+0000 7f6606a00700 1 -- 172.21.15.12:0/3217934166 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6608099de0 con 0x7f660809aba0 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.926+0000 7f6606a00700 1 --2- 172.21.15.12:0/3217934166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 0x7f660806bf40 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f65f40098f0 tx=0x7f65f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24532ddd10ac02a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.926+0000 7f6606000700 1 -- 172.21.15.12:0/3217934166 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f400b8a0 con 0x7f660809aba0 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.926+0000 7f6606000700 1 -- 172.21.15.12:0/3217934166 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f400ba00 con 0x7f660809aba0 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.926+0000 7f6606000700 1 -- 172.21.15.12:0/3217934166 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f400bd90 con 0x7f660809aba0 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.927+0000 7f660e000700 1 -- 172.21.15.12:0/3217934166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 msgr2=0x7f660806bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.927+0000 7f660e000700 1 --2- 172.21.15.12:0/3217934166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 0x7f660806bf40 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f65f40098f0 tx=0x7f65f4009c00 comp rx=0 tx=0).stop 2024-03-29T11:11:42.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.927+0000 7f660e000700 1 -- 172.21.15.12:0/3217934166 shutdown_connections 2024-03-29T11:11:42.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.927+0000 7f660e000700 1 --2- 172.21.15.12:0/3217934166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660806c480 0x7f66080ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.927+0000 7f660e000700 1 --2- 172.21.15.12:0/3217934166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 0x7f660806bf40 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.927+0000 7f660e000700 1 -- 172.21.15.12:0/3217934166 >> 172.21.15.12:0/3217934166 conn(0x7f66080d13e0 msgr2=0x7f66080d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:42.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.927+0000 7f660e000700 1 -- 172.21.15.12:0/3217934166 shutdown_connections 2024-03-29T11:11:42.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.927+0000 7f660e000700 1 -- 172.21.15.12:0/3217934166 wait complete. 2024-03-29T11:11:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.929+0000 7f660e000700 1 Processor -- start 2024-03-29T11:11:42.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.929+0000 7f660e000700 1 -- start start 2024-03-29T11:11:42.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f660e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660806c480 0x7f66080b67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f660e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 0x7f660815fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f660e000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f660815d710 0x7f66080c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f660e000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66080b58b0 con 0x7f660809aba0 2024-03-29T11:11:42.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f660e000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66080b49e0 con 0x7f660815d710 2024-03-29T11:11:42.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f660e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66080b4c30 con 0x7f660806c480 2024-03-29T11:11:42.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f6606a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 0x7f660815fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f6606a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 0x7f660815fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58198/0 (socket says 172.21.15.12:58198) 2024-03-29T11:11:42.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f6606a00700 1 -- 172.21.15.12:0/505886322 learned_addr learned my addr 172.21.15.12:0/505886322 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:42.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f6607400700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660806c480 0x7f66080b67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f6607e00700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f660815d710 0x7f66080c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f6606a00700 1 -- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660806c480 msgr2=0x7f66080b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.930+0000 7f6606a00700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660806c480 0x7f66080b67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.931+0000 7f6606a00700 1 -- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f660815d710 msgr2=0x7f66080c76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:42.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.931+0000 7f6606a00700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f660815d710 0x7f66080c76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:42.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.931+0000 7f6606a00700 1 -- 172.21.15.12:0/505886322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f4009550 con 0x7f660809aba0 2024-03-29T11:11:42.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.931+0000 7f6606a00700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 0x7f660815fcd0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f65f4000c00 tx=0x7f65f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:42.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.931+0000 7f65ff400700 1 -- 172.21.15.12:0/505886322 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4004300 con 0x7f660809aba0 2024-03-29T11:11:42.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.931+0000 7f65ff400700 1 -- 172.21.15.12:0/505886322 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f401f070 con 0x7f660809aba0 2024-03-29T11:11:42.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.931+0000 7f660e000700 1 -- 172.21.15.12:0/505886322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66080b4f10 con 0x7f660809aba0 2024-03-29T11:11:42.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.931+0000 7f65ff400700 1 -- 172.21.15.12:0/505886322 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40036a0 con 0x7f660809aba0 2024-03-29T11:11:42.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.931+0000 7f660e000700 1 -- 172.21.15.12:0/505886322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66080b54d0 con 0x7f660809aba0 2024-03-29T11:11:42.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.932+0000 7f660e000700 1 -- 172.21.15.12:0/505886322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66080b59f0 con 0x7f660809aba0 2024-03-29T11:11:42.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.933+0000 7f65ff400700 1 -- 172.21.15.12:0/505886322 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f65f4021030 con 0x7f660809aba0 2024-03-29T11:11:42.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.934+0000 7f65ff400700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65e806adb0 0x7f65e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:42.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.934+0000 7f65ff400700 1 -- 172.21.15.12:0/505886322 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f65f408a870 con 0x7f660809aba0 2024-03-29T11:11:42.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.935+0000 7f6607400700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65e806adb0 0x7f65e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:42.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.935+0000 7f6607400700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65e806adb0 0x7f65e806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f65f8007fd0 tx=0x7f65f8009040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:42.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:42.939+0000 7f65ff400700 1 -- 172.21.15.12:0/505886322 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f66080b59f0 con 0x7f660809aba0 2024-03-29T11:11:43.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.159+0000 7f660e000700 1 -- 172.21.15.12:0/505886322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66080b59f0 con 0x7f660809aba0 2024-03-29T11:11:43.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.160+0000 7f65ff400700 1 -- 172.21.15.12:0/505886322 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+16194 (secure 0 0 0) 0x7f66080b59f0 con 0x7f660809aba0 2024-03-29T11:11:43.139 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:43.139 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":176,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:42.673598+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:11:41.110455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-29T11:11:41.640016+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:43.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 -- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65e806adb0 msgr2=0x7f65e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65e806adb0 0x7f65e806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f65f8007fd0 tx=0x7f65f8009040 comp rx=0 tx=0).stop 2024-03-29T11:11:43.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 -- 172.21.15.12:0/505886322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 msgr2=0x7f660815fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 0x7f660815fcd0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f65f4000c00 tx=0x7f65f4004750 comp rx=0 tx=0).stop 2024-03-29T11:11:43.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 -- 172.21.15.12:0/505886322 shutdown_connections 2024-03-29T11:11:43.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f65e806adb0 0x7f65e806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f660815d710 0x7f66080c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660806c480 0x7f66080b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 --2- 172.21.15.12:0/505886322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f660809aba0 0x7f660815fcd0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 -- 172.21.15.12:0/505886322 >> 172.21.15.12:0/505886322 conn(0x7f66080d13e0 msgr2=0x7f66080b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:43.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 -- 172.21.15.12:0/505886322 shutdown_connections 2024-03-29T11:11:43.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.163+0000 7f660e000700 1 -- 172.21.15.12:0/505886322 wait complete. 2024-03-29T11:11:43.163 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:43.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.383+0000 7fe668400700 1 Processor -- start 2024-03-29T11:11:43.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.383+0000 7fe668400700 1 -- start start 2024-03-29T11:11:43.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.384+0000 7fe668400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe66015b5d0 0x7fe6600ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.384+0000 7fe668400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66015fd50 con 0x7fe6600ac7d0 2024-03-29T11:11:43.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.384+0000 7fe668400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66015f140 con 0x7fe6600cb3f0 2024-03-29T11:11:43.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.384+0000 7fe668400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66015b9b0 con 0x7fe66015b5d0 2024-03-29T11:11:43.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.384+0000 7fe666400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe6600ac7d0 0x7fe66006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60192/0 (socket says 172.21.15.12:60192) 2024-03-29T11:11:43.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.384+0000 7fe666400700 1 -- 172.21.15.12:0/1778031959 learned_addr learned my addr 172.21.15.12:0/1778031959 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:43.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.384+0000 7fe665a00700 1 --2- 172.21.15.12:0/1778031959 >> v2:172.21.15.12:3300/0 conn(0x7fe66015b5d0 0x7fe6600ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.385+0000 7fe665a00700 1 -- 172.21.15.12:0/1778031959 >> v1:172.21.15.159:6789/0 conn(0x7fe6600cb3f0 legacy=0x7fe6600ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.385+0000 7fe665a00700 1 -- 172.21.15.12:0/1778031959 >> v1:172.21.15.12:6789/0 conn(0x7fe6600ac7d0 legacy=0x7fe66006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.385+0000 7fe65fe00700 1 -- 172.21.15.12:0/1778031959 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521948847 0 0) 0x7fe66015fd50 con 0x7fe6600ac7d0 2024-03-29T11:11:43.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.385+0000 7fe665a00700 1 -- 172.21.15.12:0/1778031959 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66015f420 con 0x7fe66015b5d0 2024-03-29T11:11:43.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.385+0000 7fe665a00700 1 --2- 172.21.15.12:0/1778031959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe66015b5d0 0x7fe6600ae3f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fe654009840 tx=0x7fe654009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5def1fee87e265 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:43.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.385+0000 7fe65fe00700 1 -- 172.21.15.12:0/1778031959 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654011830 con 0x7fe66015b5d0 2024-03-29T11:11:43.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.385+0000 7fe65fe00700 1 -- 172.21.15.12:0/1778031959 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe654011990 con 0x7fe66015b5d0 2024-03-29T11:11:43.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.385+0000 7fe65fe00700 1 -- 172.21.15.12:0/1778031959 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654011d20 con 0x7fe66015b5d0 2024-03-29T11:11:43.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.386+0000 7fe668400700 1 -- 172.21.15.12:0/1778031959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe66015b5d0 msgr2=0x7fe6600ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.386+0000 7fe668400700 1 --2- 172.21.15.12:0/1778031959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe66015b5d0 0x7fe6600ae3f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fe654009840 tx=0x7fe654009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:43.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.386+0000 7fe668400700 1 -- 172.21.15.12:0/1778031959 shutdown_connections 2024-03-29T11:11:43.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.386+0000 7fe668400700 1 --2- 172.21.15.12:0/1778031959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe66015b5d0 0x7fe6600ae3f0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.386+0000 7fe668400700 1 -- 172.21.15.12:0/1778031959 >> 172.21.15.12:0/1778031959 conn(0x7fe6600cf190 msgr2=0x7fe6600cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:43.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.386+0000 7fe668400700 1 -- 172.21.15.12:0/1778031959 shutdown_connections 2024-03-29T11:11:43.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.386+0000 7fe668400700 1 -- 172.21.15.12:0/1778031959 wait complete. 2024-03-29T11:11:43.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.388+0000 7fe668400700 1 Processor -- start 2024-03-29T11:11:43.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.388+0000 7fe668400700 1 -- start start 2024-03-29T11:11:43.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe668400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6600ac7d0 0x7fe66009cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe668400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe66015b5d0 0x7fe66009b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe668400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6600ce260 0x7fe6601579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe668400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6601551c0 con 0x7fe6600ac7d0 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe668400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66009d180 con 0x7fe6600ce260 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe668400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66009bba0 con 0x7fe66015b5d0 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe665a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6600ac7d0 0x7fe66009cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe665a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6600ac7d0 0x7fe66009cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58226/0 (socket says 172.21.15.12:58226) 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe665a00700 1 -- 172.21.15.12:0/3692875550 learned_addr learned my addr 172.21.15.12:0/3692875550 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe666400700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6600ce260 0x7fe6601579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe665000700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe66015b5d0 0x7fe66009b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe665a00700 1 -- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe66015b5d0 msgr2=0x7fe66009b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe665a00700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe66015b5d0 0x7fe66009b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe665a00700 1 -- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6600ce260 msgr2=0x7fe6601579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe665a00700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6600ce260 0x7fe6601579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe665a00700 1 -- 172.21.15.12:0/3692875550 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540094a0 con 0x7fe6600ac7d0 2024-03-29T11:11:43.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.389+0000 7fe665000700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe66015b5d0 0x7fe66009b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:43.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.390+0000 7fe665a00700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6600ac7d0 0x7fe66009cc40 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fe654014040 tx=0x7fe654004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:43.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.390+0000 7fe65d600700 1 -- 172.21.15.12:0/3692875550 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65401a900 con 0x7fe6600ac7d0 2024-03-29T11:11:43.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.390+0000 7fe65d600700 1 -- 172.21.15.12:0/3692875550 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65401aa60 con 0x7fe6600ac7d0 2024-03-29T11:11:43.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.390+0000 7fe668400700 1 -- 172.21.15.12:0/3692875550 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66009be80 con 0x7fe6600ac7d0 2024-03-29T11:11:43.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.390+0000 7fe65d600700 1 -- 172.21.15.12:0/3692875550 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65401add0 con 0x7fe6600ac7d0 2024-03-29T11:11:43.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.390+0000 7fe668400700 1 -- 172.21.15.12:0/3692875550 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66009c440 con 0x7fe6600ac7d0 2024-03-29T11:11:43.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.391+0000 7fe668400700 1 -- 172.21.15.12:0/3692875550 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6600cba10 con 0x7fe6600ac7d0 2024-03-29T11:11:43.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.392+0000 7fe65d600700 1 -- 172.21.15.12:0/3692875550 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe654022070 con 0x7fe6600ac7d0 2024-03-29T11:11:43.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.393+0000 7fe65d600700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe64006adb0 0x7fe64006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.393+0000 7fe65d600700 1 -- 172.21.15.12:0/3692875550 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fe65408a7f0 con 0x7fe6600ac7d0 2024-03-29T11:11:43.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.394+0000 7fe665000700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe64006adb0 0x7fe64006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.394+0000 7fe665000700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe64006adb0 0x7fe64006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe6600ceb70 tx=0x7fe64c009040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:43.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.398+0000 7fe65d600700 1 -- 172.21.15.12:0/3692875550 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe654055880 con 0x7fe6600ac7d0 2024-03-29T11:11:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.613+0000 7fe668400700 1 -- 172.21.15.12:0/3692875550 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe660067720 con 0x7fe6600ac7d0 2024-03-29T11:11:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.614+0000 7fe65d600700 1 -- 172.21.15.12:0/3692875550 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+16194 (secure 0 0 0) 0x7fe654006cc0 con 0x7fe6600ac7d0 2024-03-29T11:11:43.592 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:43.593 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":176,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:42.673598+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:11:41.110455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-29T11:11:41.640016+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.616+0000 7fe668400700 1 -- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe64006adb0 msgr2=0x7fe64006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.616+0000 7fe668400700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe64006adb0 0x7fe64006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe6600ceb70 tx=0x7fe64c009040 comp rx=0 tx=0).stop 2024-03-29T11:11:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.616+0000 7fe668400700 1 -- 172.21.15.12:0/3692875550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6600ac7d0 msgr2=0x7fe66009cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.616+0000 7fe668400700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6600ac7d0 0x7fe66009cc40 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fe654014040 tx=0x7fe654004750 comp rx=0 tx=0).stop 2024-03-29T11:11:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.617+0000 7fe668400700 1 -- 172.21.15.12:0/3692875550 shutdown_connections 2024-03-29T11:11:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.617+0000 7fe668400700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe64006adb0 0x7fe64006d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.617+0000 7fe668400700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6600ce260 0x7fe6601579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.617+0000 7fe668400700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6600ac7d0 0x7fe66009cc40 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.617+0000 7fe668400700 1 --2- 172.21.15.12:0/3692875550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe66015b5d0 0x7fe66009b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.617+0000 7fe668400700 1 -- 172.21.15.12:0/3692875550 >> 172.21.15.12:0/3692875550 conn(0x7fe6600cf190 msgr2=0x7fe66015dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.617+0000 7fe668400700 1 -- 172.21.15.12:0/3692875550 shutdown_connections 2024-03-29T11:11:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.617+0000 7fe668400700 1 -- 172.21.15.12:0/3692875550 wait complete. 2024-03-29T11:11:43.615 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T11:11:43.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.841+0000 7f1a57a00700 1 Processor -- start 2024-03-29T11:11:43.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.842+0000 7f1a57a00700 1 -- start start 2024-03-29T11:11:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a57a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a50160e50 0x7f1a500b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a57a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a500b3f80 0x7f1a500b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a57a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a500cef00 con 0x7f1a500d3f20 2024-03-29T11:11:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a57a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a50161230 con 0x7f1a50160e50 2024-03-29T11:11:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a57a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a500d2a80 con 0x7f1a500b3f80 2024-03-29T11:11:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a55000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a50160e50 0x7f1a500b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a55000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a50160e50 0x7f1a500b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58228/0 (socket says 172.21.15.12:58228) 2024-03-29T11:11:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a55000700 1 -- 172.21.15.12:0/885083262 learned_addr learned my addr 172.21.15.12:0/885083262 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a55a00700 1 --2- 172.21.15.12:0/885083262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a500b3f80 0x7f1a500b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a55000700 1 -- 172.21.15.12:0/885083262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a500b3f80 msgr2=0x7f1a500b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:43.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a55000700 1 --2- 172.21.15.12:0/885083262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a500b3f80 0x7f1a500b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.843+0000 7f1a55000700 1 -- 172.21.15.12:0/885083262 >> v1:172.21.15.159:6789/0 conn(0x7f1a500d3f20 legacy=0x7f1a500b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.844+0000 7f1a55000700 1 -- 172.21.15.12:0/885083262 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a500d43b0 con 0x7f1a50160e50 2024-03-29T11:11:43.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.844+0000 7f1a55000700 1 --2- 172.21.15.12:0/885083262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 0x7f1a500b4e90 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f1a440098f0 tx=0x7f1a44009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da51331628a0b86f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:43.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.844+0000 7f1a4f400700 1 -- 172.21.15.12:0/885083262 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4400b8a0 con 0x7f1a50160e50 2024-03-29T11:11:43.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.844+0000 7f1a4f400700 1 -- 172.21.15.12:0/885083262 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4400ba00 con 0x7f1a50160e50 2024-03-29T11:11:43.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.844+0000 7f1a4f400700 1 -- 172.21.15.12:0/885083262 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4400bd90 con 0x7f1a50160e50 2024-03-29T11:11:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.845+0000 7f1a57a00700 1 -- 172.21.15.12:0/885083262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 msgr2=0x7f1a500b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.845+0000 7f1a57a00700 1 --2- 172.21.15.12:0/885083262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 0x7f1a500b4e90 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f1a440098f0 tx=0x7f1a44009c00 comp rx=0 tx=0).stop 2024-03-29T11:11:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.845+0000 7f1a57a00700 1 -- 172.21.15.12:0/885083262 shutdown_connections 2024-03-29T11:11:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.845+0000 7f1a57a00700 1 --2- 172.21.15.12:0/885083262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a500b3f80 0x7f1a500b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.845+0000 7f1a57a00700 1 --2- 172.21.15.12:0/885083262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 0x7f1a500b4e90 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.845+0000 7f1a57a00700 1 -- 172.21.15.12:0/885083262 >> 172.21.15.12:0/885083262 conn(0x7f1a5008aff0 msgr2=0x7f1a500d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.845+0000 7f1a57a00700 1 -- 172.21.15.12:0/885083262 shutdown_connections 2024-03-29T11:11:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.845+0000 7f1a57a00700 1 -- 172.21.15.12:0/885083262 wait complete. 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.846+0000 7f1a57a00700 1 Processor -- start 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.847+0000 7f1a57a00700 1 -- start start 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.847+0000 7f1a57a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a500b3f80 0x7f1a5015e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.847+0000 7f1a57a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 0x7f1a5015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.847+0000 7f1a57a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a501f2fd0 0x7f1a500b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.847+0000 7f1a57a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a501f4040 con 0x7f1a50160e50 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.847+0000 7f1a57a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5015e860 con 0x7f1a500b3f80 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.847+0000 7f1a57a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5015b2a0 con 0x7f1a501f2fd0 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a4fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 0x7f1a5015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a4fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 0x7f1a5015ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58230/0 (socket says 172.21.15.12:58230) 2024-03-29T11:11:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a4fe00700 1 -- 172.21.15.12:0/1759452845 learned_addr learned my addr 172.21.15.12:0/1759452845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a55000700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a500b3f80 0x7f1a5015e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a55a00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a501f2fd0 0x7f1a500b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a4fe00700 1 -- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a501f2fd0 msgr2=0x7f1a500b8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a4fe00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a501f2fd0 0x7f1a500b8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a4fe00700 1 -- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a500b3f80 msgr2=0x7f1a5015e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a4fe00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a500b3f80 0x7f1a5015e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a4fe00700 1 -- 172.21.15.12:0/1759452845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a44009550 con 0x7f1a50160e50 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a55a00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a501f2fd0 0x7f1a500b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.848+0000 7f1a4fe00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 0x7f1a5015ad60 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00c4e0 tx=0x7f1a3c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.849+0000 7f1a4cc00700 1 -- 172.21.15.12:0/1759452845 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c0052d0 con 0x7f1a50160e50 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.849+0000 7f1a4cc00700 1 -- 172.21.15.12:0/1759452845 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3c005430 con 0x7f1a50160e50 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.849+0000 7f1a57a00700 1 -- 172.21.15.12:0/1759452845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a501f37a0 con 0x7f1a50160e50 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.849+0000 7f1a4cc00700 1 -- 172.21.15.12:0/1759452845 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c0038c0 con 0x7f1a50160e50 2024-03-29T11:11:43.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.849+0000 7f1a57a00700 1 -- 172.21.15.12:0/1759452845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a501f3d60 con 0x7f1a50160e50 2024-03-29T11:11:43.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.850+0000 7f1a57a00700 1 -- 172.21.15.12:0/1759452845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a500d2480 con 0x7f1a50160e50 2024-03-29T11:11:43.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.851+0000 7f1a4cc00700 1 -- 172.21.15.12:0/1759452845 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1a3c003a20 con 0x7f1a50160e50 2024-03-29T11:11:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.852+0000 7f1a4cc00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a3006adb0 0x7f1a3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.852+0000 7f1a4cc00700 1 -- 172.21.15.12:0/1759452845 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f1a3c0889a0 con 0x7f1a50160e50 2024-03-29T11:11:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.852+0000 7f1a55000700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a3006adb0 0x7f1a3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:43.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.853+0000 7f1a55000700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a3006adb0 0x7f1a3006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a44014040 tx=0x7f1a44008030 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:43.856+0000 7f1a4cc00700 1 -- 172.21.15.12:0/1759452845 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1a3c053a30 con 0x7f1a50160e50 2024-03-29T11:11:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.123+0000 7f1a57a00700 1 -- 172.21.15.12:0/1759452845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1a50067720 con 0x7f1a50160e50 2024-03-29T11:11:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.989+0000 7f1a4cc00700 1 -- 172.21.15.12:0/1759452845 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v80) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1a3c0170b0 con 0x7f1a50160e50 2024-03-29T11:11:44.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.992+0000 7f1a57a00700 1 -- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a3006adb0 msgr2=0x7f1a3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.992+0000 7f1a57a00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a3006adb0 0x7f1a3006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a44014040 tx=0x7f1a44008030 comp rx=0 tx=0).stop 2024-03-29T11:11:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.992+0000 7f1a57a00700 1 -- 172.21.15.12:0/1759452845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 msgr2=0x7f1a5015ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.992+0000 7f1a57a00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 0x7f1a5015ad60 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00c4e0 tx=0x7f1a3c00c8a0 comp rx=0 tx=0).stop 2024-03-29T11:11:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.992+0000 7f1a57a00700 1 -- 172.21.15.12:0/1759452845 shutdown_connections 2024-03-29T11:11:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.992+0000 7f1a57a00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1a3006adb0 0x7f1a3006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.992+0000 7f1a57a00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a501f2fd0 0x7f1a500b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.992+0000 7f1a57a00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a500b3f80 0x7f1a5015e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.992+0000 7f1a57a00700 1 --2- 172.21.15.12:0/1759452845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a50160e50 0x7f1a5015ad60 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.992+0000 7f1a57a00700 1 -- 172.21.15.12:0/1759452845 >> 172.21.15.12:0/1759452845 conn(0x7f1a5008aff0 msgr2=0x7f1a500d0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.993+0000 7f1a57a00700 1 -- 172.21.15.12:0/1759452845 shutdown_connections 2024-03-29T11:11:44.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:44.993+0000 7f1a57a00700 1 -- 172.21.15.12:0/1759452845 wait complete. 2024-03-29T11:11:44.991 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T11:11:44.991 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T11:11:45.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.220+0000 7f30a9e00700 1 Processor -- start 2024-03-29T11:11:45.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.221+0000 7f30a9e00700 1 -- start start 2024-03-29T11:11:45.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.222+0000 7f30a9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a4160d70 0x7f30a40b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.222+0000 7f30a9e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f30a415ef70 0x7f30a415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.222+0000 7f30a9e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f30a406a790 0x7f30a406ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.222+0000 7f30a9e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a406b180 con 0x7f30a406a790 2024-03-29T11:11:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.222+0000 7f30a9e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a40b4cd0 con 0x7f30a415ef70 2024-03-29T11:11:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.222+0000 7f30a9e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a4161150 con 0x7f30a4160d70 2024-03-29T11:11:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.222+0000 7f30a3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f30a406a790 0x7f30a406ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.222+0000 7f30a3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f30a406a790 0x7f30a406ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58240/0 (socket says 172.21.15.12:58240) 2024-03-29T11:11:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.222+0000 7f30a3400700 1 -- 172.21.15.12:0/2743477508 learned_addr learned my addr 172.21.15.12:0/2743477508 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a2000700 1 --2- 172.21.15.12:0/2743477508 >> v2:172.21.15.159:3300/0 conn(0x7f30a415ef70 0x7f30a415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a2a00700 1 --2- 172.21.15.12:0/2743477508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a4160d70 0x7f30a40b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a3400700 1 -- 172.21.15.12:0/2743477508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a4160d70 msgr2=0x7f30a40b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a3400700 1 --2- 172.21.15.12:0/2743477508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a4160d70 0x7f30a40b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a3400700 1 -- 172.21.15.12:0/2743477508 >> v2:172.21.15.159:3300/0 conn(0x7f30a415ef70 msgr2=0x7f30a415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a3400700 1 --2- 172.21.15.12:0/2743477508 >> v2:172.21.15.159:3300/0 conn(0x7f30a415ef70 0x7f30a415e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a3400700 1 -- 172.21.15.12:0/2743477508 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a406dba0 con 0x7f30a406a790 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a2000700 1 --2- 172.21.15.12:0/2743477508 >> v2:172.21.15.159:3300/0 conn(0x7f30a415ef70 0x7f30a415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a2a00700 1 --2- 172.21.15.12:0/2743477508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a4160d70 0x7f30a40b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a3400700 1 --2- 172.21.15.12:0/2743477508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a406a790 0x7f30a406ac40 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f30a40713b0 tx=0x7f309c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69fd27152e655f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a1600700 1 -- 172.21.15.12:0/2743477508 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c00e070 con 0x7f30a406a790 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a1600700 1 -- 172.21.15.12:0/2743477508 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c00b9c0 con 0x7f30a406a790 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.223+0000 7f30a1600700 1 -- 172.21.15.12:0/2743477508 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c00bd70 con 0x7f30a406a790 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.224+0000 7f30a9e00700 1 -- 172.21.15.12:0/2743477508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a406a790 msgr2=0x7f30a406ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.224+0000 7f30a9e00700 1 --2- 172.21.15.12:0/2743477508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a406a790 0x7f30a406ac40 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f30a40713b0 tx=0x7f309c00b330 comp rx=0 tx=0).stop 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.224+0000 7f30a9e00700 1 -- 172.21.15.12:0/2743477508 shutdown_connections 2024-03-29T11:11:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.224+0000 7f30a9e00700 1 --2- 172.21.15.12:0/2743477508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a406a790 0x7f30a406ac40 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:45.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.224+0000 7f30a9e00700 1 --2- 172.21.15.12:0/2743477508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a4160d70 0x7f30a40b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:45.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.224+0000 7f30a9e00700 1 --2- 172.21.15.12:0/2743477508 >> v2:172.21.15.159:3300/0 conn(0x7f30a415ef70 0x7f30a415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:45.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.224+0000 7f30a9e00700 1 -- 172.21.15.12:0/2743477508 >> 172.21.15.12:0/2743477508 conn(0x7f30a40b5720 msgr2=0x7f30a40b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:45.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.224+0000 7f30a9e00700 1 -- 172.21.15.12:0/2743477508 shutdown_connections 2024-03-29T11:11:45.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.224+0000 7f30a9e00700 1 -- 172.21.15.12:0/2743477508 wait complete. 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.226+0000 7f30a9e00700 1 Processor -- start 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.226+0000 7f30a9e00700 1 -- start start 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a9e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30a406a790 0x7f30a415da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a415ef70 0x7f30a409b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a4160d70 0x7f30a409cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a9e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40d0e30 con 0x7f30a4160d70 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a9e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a409c6a0 con 0x7f30a406a790 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a9e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a409ba90 con 0x7f30a415ef70 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a4160d70 0x7f30a409cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a2a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30a406a790 0x7f30a415da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:45.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a4160d70 0x7f30a409cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58250/0 (socket says 172.21.15.12:58250) 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a3400700 1 -- 172.21.15.12:0/2266686833 learned_addr learned my addr 172.21.15.12:0/2266686833 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.227+0000 7f30a2000700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a415ef70 0x7f30a409b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f30a3400700 1 -- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a415ef70 msgr2=0x7f30a409b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f30a3400700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a415ef70 0x7f30a409b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f30a3400700 1 -- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30a406a790 msgr2=0x7f30a415da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f30a3400700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30a406a790 0x7f30a415da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f30a3400700 1 -- 172.21.15.12:0/2266686833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c009b30 con 0x7f30a4160d70 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f30a3400700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a4160d70 0x7f30a409cb30 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f30a4070430 tx=0x7f309c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f309b400700 1 -- 172.21.15.12:0/2266686833 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c004250 con 0x7f30a4160d70 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f309b400700 1 -- 172.21.15.12:0/2266686833 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c0043f0 con 0x7f30a4160d70 2024-03-29T11:11:45.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f309b400700 1 -- 172.21.15.12:0/2266686833 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c0047a0 con 0x7f30a4160d70 2024-03-29T11:11:45.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f30a9e00700 1 -- 172.21.15.12:0/2266686833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a409bd70 con 0x7f30a4160d70 2024-03-29T11:11:45.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.228+0000 7f30a9e00700 1 -- 172.21.15.12:0/2266686833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a409c330 con 0x7f30a4160d70 2024-03-29T11:11:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.229+0000 7f30a9e00700 1 -- 172.21.15.12:0/2266686833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a406afa0 con 0x7f30a4160d70 2024-03-29T11:11:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.230+0000 7f309b400700 1 -- 172.21.15.12:0/2266686833 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f309c01b430 con 0x7f30a4160d70 2024-03-29T11:11:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.231+0000 7f309b400700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f308406ade0 0x7f308406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.231+0000 7f309b400700 1 -- 172.21.15.12:0/2266686833 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f309c08a6f0 con 0x7f30a4160d70 2024-03-29T11:11:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.232+0000 7f30a2a00700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f308406ade0 0x7f308406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:45.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.232+0000 7f30a2a00700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f308406ade0 0x7f308406d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3094006fd0 tx=0x7f3094008040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:45.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.236+0000 7f309b400700 1 -- 172.21.15.12:0/2266686833 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f309c055670 con 0x7f30a4160d70 2024-03-29T11:11:45.243 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:11:45.249 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:11:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:45.498+0000 7f30a9e00700 1 -- 172.21.15.12:0/2266686833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f30a4067720 con 0x7f30a4160d70 2024-03-29T11:11:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.158+0000 7f309b400700 1 -- 172.21.15.12:0/2266686833 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v81) v1 ==== 122+0+0 (secure 0 0 0) 0x7f309c058c90 con 0x7f30a4160d70 2024-03-29T11:11:46.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.161+0000 7f30a9e00700 1 -- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f308406ade0 msgr2=0x7f308406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.161+0000 7f30a9e00700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f308406ade0 0x7f308406d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3094006fd0 tx=0x7f3094008040 comp rx=0 tx=0).stop 2024-03-29T11:11:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.161+0000 7f30a9e00700 1 -- 172.21.15.12:0/2266686833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a4160d70 msgr2=0x7f30a409cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.161+0000 7f30a9e00700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a4160d70 0x7f30a409cb30 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f30a4070430 tx=0x7f309c004ad0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.162+0000 7f30a9e00700 1 -- 172.21.15.12:0/2266686833 shutdown_connections 2024-03-29T11:11:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.162+0000 7f30a9e00700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f308406ade0 0x7f308406d2a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.162+0000 7f30a9e00700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a4160d70 0x7f30a409cb30 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.162+0000 7f30a9e00700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30a406a790 0x7f30a415da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.162+0000 7f30a9e00700 1 --2- 172.21.15.12:0/2266686833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30a415ef70 0x7f30a409b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.162+0000 7f30a9e00700 1 -- 172.21.15.12:0/2266686833 >> 172.21.15.12:0/2266686833 conn(0x7f30a40b5720 msgr2=0x7f30a406df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.162+0000 7f30a9e00700 1 -- 172.21.15.12:0/2266686833 shutdown_connections 2024-03-29T11:11:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.162+0000 7f30a9e00700 1 -- 172.21.15.12:0/2266686833 wait complete. 2024-03-29T11:11:46.160 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T11:11:46.161 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:11:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.385+0000 7f7e55a00700 1 Processor -- start 2024-03-29T11:11:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.385+0000 7f7e55a00700 1 -- start start 2024-03-29T11:11:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.387+0000 7f7e55a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e500ceec0 0x7f7e501612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.387+0000 7f7e55a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e50161c50 con 0x7f7e50161800 2024-03-29T11:11:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.387+0000 7f7e55a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e500d30d0 con 0x7f7e5015d640 2024-03-29T11:11:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.387+0000 7f7e55a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e500b6840 con 0x7f7e500ceec0 2024-03-29T11:11:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.387+0000 7f7e4e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e500ceec0 0x7f7e501612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.387+0000 7f7e4f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7e50161800 0x7f7e500b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60200/0 (socket says 172.21.15.12:60200) 2024-03-29T11:11:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.387+0000 7f7e4f400700 1 -- 172.21.15.12:0/3766972454 learned_addr learned my addr 172.21.15.12:0/3766972454 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.387+0000 7f7e4e000700 1 -- 172.21.15.12:0/3766972454 >> v1:172.21.15.159:6789/0 conn(0x7f7e5015d640 legacy=0x7f7e500d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.387+0000 7f7e4e000700 1 -- 172.21.15.12:0/3766972454 >> v1:172.21.15.12:6789/0 conn(0x7f7e50161800 legacy=0x7f7e500b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.388+0000 7f7e4e000700 1 -- 172.21.15.12:0/3766972454 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e500d2b30 con 0x7f7e500ceec0 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.387+0000 7f7e4d600700 1 -- 172.21.15.12:0/3766972454 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381969443 0 0) 0x7f7e48001490 con 0x7f7e50161800 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.388+0000 7f7e4e000700 1 --2- 172.21.15.12:0/3766972454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e500ceec0 0x7f7e501612c0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009840 tx=0x7f7e3c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1933acd3ba60ccc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.388+0000 7f7e4d600700 1 -- 172.21.15.12:0/3766972454 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c011830 con 0x7f7e500ceec0 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.388+0000 7f7e4d600700 1 -- 172.21.15.12:0/3766972454 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c011990 con 0x7f7e500ceec0 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.388+0000 7f7e4d600700 1 -- 172.21.15.12:0/3766972454 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c011d20 con 0x7f7e500ceec0 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.389+0000 7f7e55a00700 1 -- 172.21.15.12:0/3766972454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e500ceec0 msgr2=0x7f7e501612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.389+0000 7f7e55a00700 1 --2- 172.21.15.12:0/3766972454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e500ceec0 0x7f7e501612c0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009840 tx=0x7f7e3c009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.389+0000 7f7e55a00700 1 -- 172.21.15.12:0/3766972454 shutdown_connections 2024-03-29T11:11:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.389+0000 7f7e55a00700 1 --2- 172.21.15.12:0/3766972454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e500ceec0 0x7f7e501612c0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.389+0000 7f7e55a00700 1 -- 172.21.15.12:0/3766972454 >> 172.21.15.12:0/3766972454 conn(0x7f7e5008aff0 msgr2=0x7f7e500d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.389+0000 7f7e55a00700 1 -- 172.21.15.12:0/3766972454 shutdown_connections 2024-03-29T11:11:46.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.389+0000 7f7e55a00700 1 -- 172.21.15.12:0/3766972454 wait complete. 2024-03-29T11:11:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.390+0000 7f7e55a00700 1 Processor -- start 2024-03-29T11:11:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.391+0000 7f7e55a00700 1 -- start start 2024-03-29T11:11:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.391+0000 7f7e55a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e500ceec0 0x7f7e500c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.391+0000 7f7e55a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5015d640 0x7f7e5015f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.391+0000 7f7e55a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e50161800 0x7f7e5015fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.391+0000 7f7e55a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50153320 con 0x7f7e5015d640 2024-03-29T11:11:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.391+0000 7f7e55a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500c9d80 con 0x7f7e500ceec0 2024-03-29T11:11:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.391+0000 7f7e55a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500c8090 con 0x7f7e50161800 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.392+0000 7f7e4e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5015d640 0x7f7e5015f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.392+0000 7f7e4e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5015d640 0x7f7e5015f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58276/0 (socket says 172.21.15.12:58276) 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.392+0000 7f7e4e000700 1 -- 172.21.15.12:0/1154635426 learned_addr learned my addr 172.21.15.12:0/1154635426 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.392+0000 7f7e4f400700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e50161800 0x7f7e5015fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.392+0000 7f7e4ea00700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e500ceec0 0x7f7e500c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.392+0000 7f7e4e000700 1 -- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e50161800 msgr2=0x7f7e5015fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.392+0000 7f7e4e000700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e50161800 0x7f7e5015fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.392+0000 7f7e4e000700 1 -- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e500ceec0 msgr2=0x7f7e500c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.392+0000 7f7e4e000700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e500ceec0 0x7f7e500c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.392+0000 7f7e4e000700 1 -- 172.21.15.12:0/1154635426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c0094a0 con 0x7f7e5015d640 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.393+0000 7f7e4e000700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5015d640 0x7f7e5015f670 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c014040 tx=0x7f7e3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.393+0000 7f7e46a00700 1 -- 172.21.15.12:0/1154635426 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c01a900 con 0x7f7e5015d640 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.393+0000 7f7e46a00700 1 -- 172.21.15.12:0/1154635426 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c01aa60 con 0x7f7e5015d640 2024-03-29T11:11:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.393+0000 7f7e46a00700 1 -- 172.21.15.12:0/1154635426 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c01add0 con 0x7f7e5015d640 2024-03-29T11:11:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.393+0000 7f7e55a00700 1 -- 172.21.15.12:0/1154635426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e500c8370 con 0x7f7e5015d640 2024-03-29T11:11:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.393+0000 7f7e55a00700 1 -- 172.21.15.12:0/1154635426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e500c8930 con 0x7f7e5015d640 2024-03-29T11:11:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.394+0000 7f7e55a00700 1 -- 172.21.15.12:0/1154635426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e50067720 con 0x7f7e5015d640 2024-03-29T11:11:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.395+0000 7f7e46a00700 1 -- 172.21.15.12:0/1154635426 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7e3c022070 con 0x7f7e5015d640 2024-03-29T11:11:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.396+0000 7f7e46a00700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e3006adb0 0x7f7e3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.396+0000 7f7e46a00700 1 -- 172.21.15.12:0/1154635426 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f7e3c08a2e0 con 0x7f7e5015d640 2024-03-29T11:11:46.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.396+0000 7f7e4ea00700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e3006adb0 0x7f7e3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:46.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.397+0000 7f7e4ea00700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e3006adb0 0x7f7e3006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7e40000f60 tx=0x7f7e40009040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:46.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.400+0000 7f7e46a00700 1 -- 172.21.15.12:0/1154635426 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7e3c055540 con 0x7f7e5015d640 2024-03-29T11:11:46.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.661+0000 7f7e55a00700 1 -- 172.21.15.12:0/1154635426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e500b5c70 con 0x7f7e5015d640 2024-03-29T11:11:46.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.662+0000 7f7e46a00700 1 -- 172.21.15.12:0/1154635426 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f7e3c058b60 con 0x7f7e5015d640 2024-03-29T11:11:46.640 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:46.641 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/652112625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":652112625},{"type":"v1","addr":"172.21.15.159:6836","nonce":652112625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9166,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2803735172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2803735172},{"type":"v1","addr":"172.21.15.159:6837","nonce":2803735172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9444,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3698805288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3698805288},{"type":"v1","addr":"172.21.15.12:6833","nonce":3698805288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:11:42.673965+0000","modified":"2024-03-29T11:11:46.149885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9450":{"gid":9450,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/2066631988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2066631988},{"type":"v1","addr":"172.21.15.12:6835","nonce":2066631988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-29T11:11:46.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 -- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e3006adb0 msgr2=0x7f7e3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e3006adb0 0x7f7e3006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7e40000f60 tx=0x7f7e40009040 comp rx=0 tx=0).stop 2024-03-29T11:11:46.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 -- 172.21.15.12:0/1154635426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5015d640 msgr2=0x7f7e5015f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5015d640 0x7f7e5015f670 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c014040 tx=0x7f7e3c004750 comp rx=0 tx=0).stop 2024-03-29T11:11:46.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 -- 172.21.15.12:0/1154635426 shutdown_connections 2024-03-29T11:11:46.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e3006adb0 0x7f7e3006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e50161800 0x7f7e5015fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e500ceec0 0x7f7e500c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 --2- 172.21.15.12:0/1154635426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5015d640 0x7f7e5015f670 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 -- 172.21.15.12:0/1154635426 >> 172.21.15.12:0/1154635426 conn(0x7f7e5008aff0 msgr2=0x7f7e500d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:46.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 -- 172.21.15.12:0/1154635426 shutdown_connections 2024-03-29T11:11:46.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.664+0000 7f7e55a00700 1 -- 172.21.15.12:0/1154635426 wait complete. 2024-03-29T11:11:46.644 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 81 2024-03-29T11:11:46.661 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:46.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.882+0000 7f6a70a00700 1 Processor -- start 2024-03-29T11:11:46.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.882+0000 7f6a70a00700 1 -- start start 2024-03-29T11:11:46.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.884+0000 7f6a70a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6a6c15d670 0x7f6a6c0ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.884+0000 7f6a70a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a6c15da50 0x7f6a6c0b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.884+0000 7f6a70a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6a6c0b3000 0x7f6a6c06dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.884+0000 7f6a70a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6c0b3890 con 0x7f6a6c15d670 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.884+0000 7f6a70a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6c0acac0 con 0x7f6a6c0b3000 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.884+0000 7f6a70a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c159f50 con 0x7f6a6c15da50 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.884+0000 7f6a69e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6a6c15d670 0x7f6a6c0ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.884+0000 7f6a69e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6a6c15d670 0x7f6a6c0ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58282/0 (socket says 172.21.15.12:58282) 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.884+0000 7f6a69e00700 1 -- 172.21.15.12:0/360955495 learned_addr learned my addr 172.21.15.12:0/360955495 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a6a800700 1 --2- 172.21.15.12:0/360955495 >> v2:172.21.15.159:3300/0 conn(0x7f6a6c0b3000 0x7f6a6c06dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a63e00700 1 --2- 172.21.15.12:0/360955495 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a6c15da50 0x7f6a6c0b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a69e00700 1 -- 172.21.15.12:0/360955495 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a6c15da50 msgr2=0x7f6a6c0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a69e00700 1 --2- 172.21.15.12:0/360955495 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a6c15da50 0x7f6a6c0b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a69e00700 1 -- 172.21.15.12:0/360955495 >> v2:172.21.15.159:3300/0 conn(0x7f6a6c0b3000 msgr2=0x7f6a6c06dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a69e00700 1 --2- 172.21.15.12:0/360955495 >> v2:172.21.15.159:3300/0 conn(0x7f6a6c0b3000 0x7f6a6c06dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a69e00700 1 -- 172.21.15.12:0/360955495 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c0b35b0 con 0x7f6a6c15d670 2024-03-29T11:11:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a69e00700 1 --2- 172.21.15.12:0/360955495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c15d670 0x7f6a6c0ac580 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c072e50 tx=0x7f6a5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683e5874094aa0b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a69400700 1 -- 172.21.15.12:0/360955495 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c00b870 con 0x7f6a6c15d670 2024-03-29T11:11:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a69400700 1 -- 172.21.15.12:0/360955495 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c00b9d0 con 0x7f6a6c15d670 2024-03-29T11:11:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.885+0000 7f6a69400700 1 -- 172.21.15.12:0/360955495 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c00bd60 con 0x7f6a6c15d670 2024-03-29T11:11:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.886+0000 7f6a70a00700 1 -- 172.21.15.12:0/360955495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c15d670 msgr2=0x7f6a6c0ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.886+0000 7f6a70a00700 1 --2- 172.21.15.12:0/360955495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c15d670 0x7f6a6c0ac580 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c072e50 tx=0x7f6a5c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.886+0000 7f6a70a00700 1 -- 172.21.15.12:0/360955495 shutdown_connections 2024-03-29T11:11:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.886+0000 7f6a70a00700 1 --2- 172.21.15.12:0/360955495 >> v2:172.21.15.159:3300/0 conn(0x7f6a6c0b3000 0x7f6a6c06dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.886+0000 7f6a70a00700 1 --2- 172.21.15.12:0/360955495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c15d670 0x7f6a6c0ac580 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.886+0000 7f6a70a00700 1 --2- 172.21.15.12:0/360955495 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a6c15da50 0x7f6a6c0b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.886+0000 7f6a70a00700 1 -- 172.21.15.12:0/360955495 >> 172.21.15.12:0/360955495 conn(0x7f6a6c0cf190 msgr2=0x7f6a6c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.886+0000 7f6a70a00700 1 -- 172.21.15.12:0/360955495 shutdown_connections 2024-03-29T11:11:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.886+0000 7f6a70a00700 1 -- 172.21.15.12:0/360955495 wait complete. 2024-03-29T11:11:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.888+0000 7f6a70a00700 1 Processor -- start 2024-03-29T11:11:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.889+0000 7f6a70a00700 1 -- start start 2024-03-29T11:11:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a70a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c0b3000 0x7f6a6c098a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a70a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a6c15d670 0x7f6a6c06a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a70a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a6c15da50 0x7f6a6c06acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a70a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c098f60 con 0x7f6a6c0b3000 2024-03-29T11:11:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a70a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c0cc720 con 0x7f6a6c15d670 2024-03-29T11:11:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a70a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c0cca00 con 0x7f6a6c15da50 2024-03-29T11:11:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a69e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c0b3000 0x7f6a6c098a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a69e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c0b3000 0x7f6a6c098a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58290/0 (socket says 172.21.15.12:58290) 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a69e00700 1 -- 172.21.15.12:0/3883306386 learned_addr learned my addr 172.21.15.12:0/3883306386 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a69e00700 1 -- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a6c15da50 msgr2=0x7f6a6c06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a69e00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a6c15da50 0x7f6a6c06acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a69e00700 1 -- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a6c15d670 msgr2=0x7f6a6c06a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a69e00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a6c15d670 0x7f6a6c06a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.890+0000 7f6a69e00700 1 -- 172.21.15.12:0/3883306386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c009550 con 0x7f6a6c0b3000 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.891+0000 7f6a69e00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c0b3000 0x7f6a6c098a20 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c000c00 tx=0x7f6a5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.891+0000 7f6a62000700 1 -- 172.21.15.12:0/3883306386 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c019df0 con 0x7f6a6c0b3000 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.891+0000 7f6a62000700 1 -- 172.21.15.12:0/3883306386 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c004050 con 0x7f6a6c0b3000 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.891+0000 7f6a70a00700 1 -- 172.21.15.12:0/3883306386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6c0ccce0 con 0x7f6a6c0b3000 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.891+0000 7f6a62000700 1 -- 172.21.15.12:0/3883306386 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c004400 con 0x7f6a6c0b3000 2024-03-29T11:11:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.891+0000 7f6a70a00700 1 -- 172.21.15.12:0/3883306386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6c0cd2a0 con 0x7f6a6c0b3000 2024-03-29T11:11:46.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.892+0000 7f6a70a00700 1 -- 172.21.15.12:0/3883306386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6c067720 con 0x7f6a6c0b3000 2024-03-29T11:11:46.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.893+0000 7f6a62000700 1 -- 172.21.15.12:0/3883306386 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6a5c0036a0 con 0x7f6a6c0b3000 2024-03-29T11:11:46.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.894+0000 7f6a62000700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a4806adb0 0x7f6a4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:46.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.894+0000 7f6a62000700 1 -- 172.21.15.12:0/3883306386 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f6a5c08aa70 con 0x7f6a6c0b3000 2024-03-29T11:11:46.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.895+0000 7f6a63e00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a4806adb0 0x7f6a4806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:46.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.895+0000 7f6a63e00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a4806adb0 0x7f6a4806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c0ad690 tx=0x7f6a54008040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:46.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:46.899+0000 7f6a62000700 1 -- 172.21.15.12:0/3883306386 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a5c055a70 con 0x7f6a6c0b3000 2024-03-29T11:11:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.113+0000 7f6a70a00700 1 -- 172.21.15.12:0/3883306386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a6c071080 con 0x7f6a6c0b3000 2024-03-29T11:11:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.114+0000 7f6a62000700 1 -- 172.21.15.12:0/3883306386 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+16181 (secure 0 0 0) 0x7f6a5c026020 con 0x7f6a6c0b3000 2024-03-29T11:11:47.093 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:47.093 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":177,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:43.676478+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:11:41.110455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-29T11:11:41.640016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 -- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a4806adb0 msgr2=0x7f6a4806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a4806adb0 0x7f6a4806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c0ad690 tx=0x7f6a54008040 comp rx=0 tx=0).stop 2024-03-29T11:11:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 -- 172.21.15.12:0/3883306386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c0b3000 msgr2=0x7f6a6c098a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c0b3000 0x7f6a6c098a20 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c000c00 tx=0x7f6a5c004750 comp rx=0 tx=0).stop 2024-03-29T11:11:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 -- 172.21.15.12:0/3883306386 shutdown_connections 2024-03-29T11:11:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6a4806adb0 0x7f6a4806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a6c15da50 0x7f6a6c06acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a6c0b3000 0x7f6a6c098a20 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 --2- 172.21.15.12:0/3883306386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a6c15d670 0x7f6a6c06a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 -- 172.21.15.12:0/3883306386 >> 172.21.15.12:0/3883306386 conn(0x7f6a6c0cf190 msgr2=0x7f6a6c158da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 -- 172.21.15.12:0/3883306386 shutdown_connections 2024-03-29T11:11:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.117+0000 7f6a70a00700 1 -- 172.21.15.12:0/3883306386 wait complete. 2024-03-29T11:11:47.114 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T11:11:47.115 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:11:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.336+0000 7f4000600700 1 Processor -- start 2024-03-29T11:11:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.336+0000 7f4000600700 1 -- start start 2024-03-29T11:11:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.337+0000 7f4000600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ff8160be0 0x7f3ff80cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.337+0000 7f4000600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3ff80ce0e0 0x7f3ff80ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.337+0000 7f4000600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff8160fc0 con 0x7f3ff80cead0 2024-03-29T11:11:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.337+0000 7f4000600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ff80cef60 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.337+0000 7f4000600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff80b65d0 con 0x7f3ff8160be0 2024-03-29T11:11:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.337+0000 7f3ffd200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3ff80ce0e0 0x7f3ff80ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.337+0000 7f3ffd200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3ff80ce0e0 0x7f3ff80ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58306/0 (socket says 172.21.15.12:58306) 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.337+0000 7f3ffd200700 1 -- 172.21.15.12:0/1530747997 learned_addr learned my addr 172.21.15.12:0/1530747997 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.337+0000 7f3ffdc00700 1 --2- 172.21.15.12:0/1530747997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ff8160be0 0x7f3ff80cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.338+0000 7f3ffd200700 1 -- 172.21.15.12:0/1530747997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ff8160be0 msgr2=0x7f3ff80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.338+0000 7f3ffd200700 1 --2- 172.21.15.12:0/1530747997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ff8160be0 0x7f3ff80cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.338+0000 7f3ffd200700 1 -- 172.21.15.12:0/1530747997 >> v1:172.21.15.159:6789/0 conn(0x7f3ff80cead0 legacy=0x7f3ff8161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.338+0000 7f3ffd200700 1 -- 172.21.15.12:0/1530747997 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff80d21e0 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.338+0000 7f3ffd200700 1 --2- 172.21.15.12:0/1530747997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 0x7f3ff80ce590 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40098f0 tx=0x7f3fe4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569d0b4b662e45c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.338+0000 7f3ff7e00700 1 -- 172.21.15.12:0/1530747997 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe400b8a0 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.338+0000 7f3ff7e00700 1 -- 172.21.15.12:0/1530747997 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe400ba00 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.338+0000 7f3ff7e00700 1 -- 172.21.15.12:0/1530747997 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe400bd90 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.339+0000 7f4000600700 1 -- 172.21.15.12:0/1530747997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 msgr2=0x7f3ff80ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.339+0000 7f4000600700 1 --2- 172.21.15.12:0/1530747997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 0x7f3ff80ce590 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40098f0 tx=0x7f3fe4009c00 comp rx=0 tx=0).stop 2024-03-29T11:11:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.339+0000 7f4000600700 1 -- 172.21.15.12:0/1530747997 shutdown_connections 2024-03-29T11:11:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.339+0000 7f4000600700 1 --2- 172.21.15.12:0/1530747997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ff8160be0 0x7f3ff80cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.339+0000 7f4000600700 1 --2- 172.21.15.12:0/1530747997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 0x7f3ff80ce590 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.339+0000 7f4000600700 1 -- 172.21.15.12:0/1530747997 >> 172.21.15.12:0/1530747997 conn(0x7f3ff808aff0 msgr2=0x7f3ff80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.339+0000 7f4000600700 1 -- 172.21.15.12:0/1530747997 shutdown_connections 2024-03-29T11:11:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.339+0000 7f4000600700 1 -- 172.21.15.12:0/1530747997 wait complete. 2024-03-29T11:11:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.341+0000 7f4000600700 1 Processor -- start 2024-03-29T11:11:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.341+0000 7f4000600700 1 -- start start 2024-03-29T11:11:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f4000600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 0x7f3ff80c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f4000600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ff80cead0 0x7f3ff8156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f4000600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ff8160be0 0x7f3ff8157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f4000600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff81575a0 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f4000600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff80c7220 con 0x7f3ff8160be0 2024-03-29T11:11:47.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f4000600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff80c7500 con 0x7f3ff80cead0 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f3ffdc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 0x7f3ff80c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f3ffdc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 0x7f3ff80c9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58308/0 (socket says 172.21.15.12:58308) 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f3ffdc00700 1 -- 172.21.15.12:0/4163873786 learned_addr learned my addr 172.21.15.12:0/4163873786 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f3ffe600700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ff8160be0 0x7f3ff8157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f3ffd200700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ff80cead0 0x7f3ff8156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f3ffdc00700 1 -- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ff80cead0 msgr2=0x7f3ff8156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f3ffdc00700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ff80cead0 0x7f3ff8156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f3ffdc00700 1 -- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ff8160be0 msgr2=0x7f3ff8157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f3ffdc00700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ff8160be0 0x7f3ff8157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.342+0000 7f3ffdc00700 1 -- 172.21.15.12:0/4163873786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe4009550 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.343+0000 7f3ffe600700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ff8160be0 0x7f3ff8157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.343+0000 7f3ffdc00700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 0x7f3ff80c9c70 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00d800 tx=0x7f3fec00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:47.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.343+0000 7f3ff5600700 1 -- 172.21.15.12:0/4163873786 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec0125f0 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.343+0000 7f3ff5600700 1 -- 172.21.15.12:0/4163873786 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fec012750 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.343+0000 7f4000600700 1 -- 172.21.15.12:0/4163873786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff80c77e0 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.343+0000 7f3ff5600700 1 -- 172.21.15.12:0/4163873786 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec012b00 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.343+0000 7f4000600700 1 -- 172.21.15.12:0/4163873786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff80ca1b0 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.344+0000 7f4000600700 1 -- 172.21.15.12:0/4163873786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff80ca370 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.345+0000 7f3ff5600700 1 -- 172.21.15.12:0/4163873786 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3fec012c60 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.346+0000 7f3ff5600700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fd806adb0 0x7f3fd806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.346+0000 7f3ff5600700 1 -- 172.21.15.12:0/4163873786 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f3fec089860 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.346+0000 7f3ffd200700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fd806adb0 0x7f3fd806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.347+0000 7f3ffd200700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fd806adb0 0x7f3fd806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009c40 tx=0x7f3fe4008030 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.350+0000 7f3ff5600700 1 -- 172.21.15.12:0/4163873786 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3fec054860 con 0x7f3ff80ce0e0 2024-03-29T11:11:47.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.559+0000 7f4000600700 1 -- 172.21.15.12:0/4163873786 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ff80c7390 con 0x7f3fd806adb0 2024-03-29T11:11:47.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.571+0000 7f3ff5600700 1 -- 172.21.15.12:0/4163873786 <== mgr.7844 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289559 (secure 0 0 0) 0x7f3ff80c7390 con 0x7f3fd806adb0 2024-03-29T11:11:47.553 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:47.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.575+0000 7f4000600700 1 -- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fd806adb0 msgr2=0x7f3fd806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.575+0000 7f4000600700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fd806adb0 0x7f3fd806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009c40 tx=0x7f3fe4008030 comp rx=0 tx=0).stop 2024-03-29T11:11:47.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.575+0000 7f4000600700 1 -- 172.21.15.12:0/4163873786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 msgr2=0x7f3ff80c9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.575+0000 7f4000600700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 0x7f3ff80c9c70 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00d800 tx=0x7f3fec00db10 comp rx=0 tx=0).stop 2024-03-29T11:11:47.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.575+0000 7f4000600700 1 -- 172.21.15.12:0/4163873786 shutdown_connections 2024-03-29T11:11:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.575+0000 7f4000600700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3fd806adb0 0x7f3fd806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.575+0000 7f4000600700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ff8160be0 0x7f3ff8157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.575+0000 7f4000600700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ff80ce0e0 0x7f3ff80c9c70 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.575+0000 7f4000600700 1 --2- 172.21.15.12:0/4163873786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ff80cead0 0x7f3ff8156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.575+0000 7f4000600700 1 -- 172.21.15.12:0/4163873786 >> 172.21.15.12:0/4163873786 conn(0x7f3ff808aff0 msgr2=0x7f3ff80b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.576+0000 7f4000600700 1 -- 172.21.15.12:0/4163873786 shutdown_connections 2024-03-29T11:11:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.576+0000 7f4000600700 1 -- 172.21.15.12:0/4163873786 wait complete. 2024-03-29T11:11:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:11:47.572 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-29T11:11:45.985799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":515,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":327,"ondisk_log_size":327,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":404336,"kb_used_data":18396,"kb_used_omap":12,"kb_used_meta":385779,"kb_avail":754570384,"statfs":{"total":773094113280,"available":772680073216,"internally_reserved":0,"allocated":18837504,"data_stored":9680009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":395038265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.844206"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898045+0000","last_change":"2024-03-29T11:11:42.696332+0000","last_active":"2024-03-29T11:11:43.898045+0000","last_peered":"2024-03-29T11:11:43.898045+0000","last_clean":"2024-03-29T11:11:43.898045+0000","last_became_active":"2024-03-29T11:11:42.696080+0000","last_became_peered":"2024-03-29T11:11:42.696080+0000","last_unstale":"2024-03-29T11:11:43.898045+0000","last_undegraded":"2024-03-29T11:11:43.898045+0000","last_fullsized":"2024-03-29T11:11:43.898045+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:38:18.000649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732251+0000","last_change":"2024-03-29T11:11:42.700813+0000","last_active":"2024-03-29T11:11:43.732251+0000","last_peered":"2024-03-29T11:11:43.732251+0000","last_clean":"2024-03-29T11:11:43.732251+0000","last_became_active":"2024-03-29T11:11:42.699888+0000","last_became_peered":"2024-03-29T11:11:42.699888+0000","last_unstale":"2024-03-29T11:11:43.732251+0000","last_undegraded":"2024-03-29T11:11:43.732251+0000","last_fullsized":"2024-03-29T11:11:43.732251+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:07:02.476319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688154+0000","last_change":"2024-03-29T11:11:42.688985+0000","last_active":"2024-03-29T11:11:43.688154+0000","last_peered":"2024-03-29T11:11:43.688154+0000","last_clean":"2024-03-29T11:11:43.688154+0000","last_became_active":"2024-03-29T11:11:42.688679+0000","last_became_peered":"2024-03-29T11:11:42.688679+0000","last_unstale":"2024-03-29T11:11:43.688154+0000","last_undegraded":"2024-03-29T11:11:43.688154+0000","last_fullsized":"2024-03-29T11:11:43.688154+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:25.203576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898469+0000","last_change":"2024-03-29T11:11:42.704758+0000","last_active":"2024-03-29T11:11:43.898469+0000","last_peered":"2024-03-29T11:11:43.898469+0000","last_clean":"2024-03-29T11:11:43.898469+0000","last_became_active":"2024-03-29T11:11:42.704501+0000","last_became_peered":"2024-03-29T11:11:42.704501+0000","last_unstale":"2024-03-29T11:11:43.898469+0000","last_undegraded":"2024-03-29T11:11:43.898469+0000","last_fullsized":"2024-03-29T11:11:43.898469+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:22:20.546194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898291+0000","last_change":"2024-03-29T11:11:42.696374+0000","last_active":"2024-03-29T11:11:43.898291+0000","last_peered":"2024-03-29T11:11:43.898291+0000","last_clean":"2024-03-29T11:11:43.898291+0000","last_became_active":"2024-03-29T11:11:42.696099+0000","last_became_peered":"2024-03-29T11:11:42.696099+0000","last_unstale":"2024-03-29T11:11:43.898291+0000","last_undegraded":"2024-03-29T11:11:43.898291+0000","last_fullsized":"2024-03-29T11:11:43.898291+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:30.693777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897402+0000","last_change":"2024-03-29T11:11:42.694844+0000","last_active":"2024-03-29T11:11:43.897402+0000","last_peered":"2024-03-29T11:11:43.897402+0000","last_clean":"2024-03-29T11:11:43.897402+0000","last_became_active":"2024-03-29T11:11:42.694543+0000","last_became_peered":"2024-03-29T11:11:42.694543+0000","last_unstale":"2024-03-29T11:11:43.897402+0000","last_undegraded":"2024-03-29T11:11:43.897402+0000","last_fullsized":"2024-03-29T11:11:43.897402+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:28.309201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687499+0000","last_change":"2024-03-29T11:11:42.708861+0000","last_active":"2024-03-29T11:11:43.687499+0000","last_peered":"2024-03-29T11:11:43.687499+0000","last_clean":"2024-03-29T11:11:43.687499+0000","last_became_active":"2024-03-29T11:11:42.708572+0000","last_became_peered":"2024-03-29T11:11:42.708572+0000","last_unstale":"2024-03-29T11:11:43.687499+0000","last_undegraded":"2024-03-29T11:11:43.687499+0000","last_fullsized":"2024-03-29T11:11:43.687499+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:46:23.042064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.896976+0000","last_change":"2024-03-29T11:11:42.690600+0000","last_active":"2024-03-29T11:11:43.896976+0000","last_peered":"2024-03-29T11:11:43.896976+0000","last_clean":"2024-03-29T11:11:43.896976+0000","last_became_active":"2024-03-29T11:11:42.688865+0000","last_became_peered":"2024-03-29T11:11:42.688865+0000","last_unstale":"2024-03-29T11:11:43.896976+0000","last_undegraded":"2024-03-29T11:11:43.896976+0000","last_fullsized":"2024-03-29T11:11:43.896976+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:33:41.185232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732305+0000","last_change":"2024-03-29T11:11:42.702419+0000","last_active":"2024-03-29T11:11:43.732305+0000","last_peered":"2024-03-29T11:11:43.732305+0000","last_clean":"2024-03-29T11:11:43.732305+0000","last_became_active":"2024-03-29T11:11:42.702115+0000","last_became_peered":"2024-03-29T11:11:42.702115+0000","last_unstale":"2024-03-29T11:11:43.732305+0000","last_undegraded":"2024-03-29T11:11:43.732305+0000","last_fullsized":"2024-03-29T11:11:43.732305+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:13:17.783041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688194+0000","last_change":"2024-03-29T11:11:42.713204+0000","last_active":"2024-03-29T11:11:43.688194+0000","last_peered":"2024-03-29T11:11:43.688194+0000","last_clean":"2024-03-29T11:11:43.688194+0000","last_became_active":"2024-03-29T11:11:42.712895+0000","last_became_peered":"2024-03-29T11:11:42.712895+0000","last_unstale":"2024-03-29T11:11:43.688194+0000","last_undegraded":"2024-03-29T11:11:43.688194+0000","last_fullsized":"2024-03-29T11:11:43.688194+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:57:33.428782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897469+0000","last_change":"2024-03-29T11:11:42.686764+0000","last_active":"2024-03-29T11:11:43.897469+0000","last_peered":"2024-03-29T11:11:43.897469+0000","last_clean":"2024-03-29T11:11:43.897469+0000","last_became_active":"2024-03-29T11:11:42.685357+0000","last_became_peered":"2024-03-29T11:11:42.685357+0000","last_unstale":"2024-03-29T11:11:43.897469+0000","last_undegraded":"2024-03-29T11:11:43.897469+0000","last_fullsized":"2024-03-29T11:11:43.897469+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:37:43.164221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687898+0000","last_change":"2024-03-29T11:11:42.698405+0000","last_active":"2024-03-29T11:11:43.687898+0000","last_peered":"2024-03-29T11:11:43.687898+0000","last_clean":"2024-03-29T11:11:43.687898+0000","last_became_active":"2024-03-29T11:11:42.698113+0000","last_became_peered":"2024-03-29T11:11:42.698113+0000","last_unstale":"2024-03-29T11:11:43.687898+0000","last_undegraded":"2024-03-29T11:11:43.687898+0000","last_fullsized":"2024-03-29T11:11:43.687898+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:28.897598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898499+0000","last_change":"2024-03-29T11:11:42.701128+0000","last_active":"2024-03-29T11:11:43.898499+0000","last_peered":"2024-03-29T11:11:43.898499+0000","last_clean":"2024-03-29T11:11:43.898499+0000","last_became_active":"2024-03-29T11:11:42.700830+0000","last_became_peered":"2024-03-29T11:11:42.700830+0000","last_unstale":"2024-03-29T11:11:43.898499+0000","last_undegraded":"2024-03-29T11:11:43.898499+0000","last_fullsized":"2024-03-29T11:11:43.898499+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:27:36.222719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898341+0000","last_change":"2024-03-29T11:11:42.705814+0000","last_active":"2024-03-29T11:11:43.898341+0000","last_peered":"2024-03-29T11:11:43.898341+0000","last_clean":"2024-03-29T11:11:43.898341+0000","last_became_active":"2024-03-29T11:11:42.705537+0000","last_became_peered":"2024-03-29T11:11:42.705537+0000","last_unstale":"2024-03-29T11:11:43.898341+0000","last_undegraded":"2024-03-29T11:11:43.898341+0000","last_fullsized":"2024-03-29T11:11:43.898341+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:50:24.866795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.686887+0000","last_change":"2024-03-29T11:11:42.694257+0000","last_active":"2024-03-29T11:11:43.686887+0000","last_peered":"2024-03-29T11:11:43.686887+0000","last_clean":"2024-03-29T11:11:43.686887+0000","last_became_active":"2024-03-29T11:11:42.693808+0000","last_became_peered":"2024-03-29T11:11:42.693808+0000","last_unstale":"2024-03-29T11:11:43.686887+0000","last_undegraded":"2024-03-29T11:11:43.686887+0000","last_fullsized":"2024-03-29T11:11:43.686887+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:03:38.816350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897089+0000","last_change":"2024-03-29T11:11:42.697410+0000","last_active":"2024-03-29T11:11:43.897089+0000","last_peered":"2024-03-29T11:11:43.897089+0000","last_clean":"2024-03-29T11:11:43.897089+0000","last_became_active":"2024-03-29T11:11:42.692357+0000","last_became_peered":"2024-03-29T11:11:42.692357+0000","last_unstale":"2024-03-29T11:11:43.897089+0000","last_undegraded":"2024-03-29T11:11:43.897089+0000","last_fullsized":"2024-03-29T11:11:43.897089+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:41:11.037823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688523+0000","last_change":"2024-03-29T11:11:42.699688+0000","last_active":"2024-03-29T11:11:43.688523+0000","last_peered":"2024-03-29T11:11:43.688523+0000","last_clean":"2024-03-29T11:11:43.688523+0000","last_became_active":"2024-03-29T11:11:42.699442+0000","last_became_peered":"2024-03-29T11:11:42.699442+0000","last_unstale":"2024-03-29T11:11:43.688523+0000","last_undegraded":"2024-03-29T11:11:43.688523+0000","last_fullsized":"2024-03-29T11:11:43.688523+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:33.099004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687615+0000","last_change":"2024-03-29T11:11:42.692192+0000","last_active":"2024-03-29T11:11:43.687615+0000","last_peered":"2024-03-29T11:11:43.687615+0000","last_clean":"2024-03-29T11:11:43.687615+0000","last_became_active":"2024-03-29T11:11:42.691817+0000","last_became_peered":"2024-03-29T11:11:42.691817+0000","last_unstale":"2024-03-29T11:11:43.687615+0000","last_undegraded":"2024-03-29T11:11:43.687615+0000","last_fullsized":"2024-03-29T11:11:43.687615+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:50:35.067769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897550+0000","last_change":"2024-03-29T11:11:42.703702+0000","last_active":"2024-03-29T11:11:43.897550+0000","last_peered":"2024-03-29T11:11:43.897550+0000","last_clean":"2024-03-29T11:11:43.897550+0000","last_became_active":"2024-03-29T11:11:42.701416+0000","last_became_peered":"2024-03-29T11:11:42.701416+0000","last_unstale":"2024-03-29T11:11:43.897550+0000","last_undegraded":"2024-03-29T11:11:43.897550+0000","last_fullsized":"2024-03-29T11:11:43.897550+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:50:52.001882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897147+0000","last_change":"2024-03-29T11:11:42.694473+0000","last_active":"2024-03-29T11:11:43.897147+0000","last_peered":"2024-03-29T11:11:43.897147+0000","last_clean":"2024-03-29T11:11:43.897147+0000","last_became_active":"2024-03-29T11:11:42.693967+0000","last_became_peered":"2024-03-29T11:11:42.693967+0000","last_unstale":"2024-03-29T11:11:43.897147+0000","last_undegraded":"2024-03-29T11:11:43.897147+0000","last_fullsized":"2024-03-29T11:11:43.897147+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:56:55.325827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687890+0000","last_change":"2024-03-29T11:11:42.713124+0000","last_active":"2024-03-29T11:11:43.687890+0000","last_peered":"2024-03-29T11:11:43.687890+0000","last_clean":"2024-03-29T11:11:43.687890+0000","last_became_active":"2024-03-29T11:11:42.710967+0000","last_became_peered":"2024-03-29T11:11:42.710967+0000","last_unstale":"2024-03-29T11:11:43.687890+0000","last_undegraded":"2024-03-29T11:11:43.687890+0000","last_fullsized":"2024-03-29T11:11:43.687890+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:11.397620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898085+0000","last_change":"2024-03-29T11:11:42.695397+0000","last_active":"2024-03-29T11:11:43.898085+0000","last_peered":"2024-03-29T11:11:43.898085+0000","last_clean":"2024-03-29T11:11:43.898085+0000","last_became_active":"2024-03-29T11:11:42.695104+0000","last_became_peered":"2024-03-29T11:11:42.695104+0000","last_unstale":"2024-03-29T11:11:43.898085+0000","last_undegraded":"2024-03-29T11:11:43.898085+0000","last_fullsized":"2024-03-29T11:11:43.898085+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:36:23.384386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897934+0000","last_change":"2024-03-29T11:11:42.697718+0000","last_active":"2024-03-29T11:11:43.897934+0000","last_peered":"2024-03-29T11:11:43.897934+0000","last_clean":"2024-03-29T11:11:43.897934+0000","last_became_active":"2024-03-29T11:11:42.696020+0000","last_became_peered":"2024-03-29T11:11:42.696020+0000","last_unstale":"2024-03-29T11:11:43.897934+0000","last_undegraded":"2024-03-29T11:11:43.897934+0000","last_fullsized":"2024-03-29T11:11:43.897934+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:15.795503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687023+0000","last_change":"2024-03-29T11:11:42.714826+0000","last_active":"2024-03-29T11:11:43.687023+0000","last_peered":"2024-03-29T11:11:43.687023+0000","last_clean":"2024-03-29T11:11:43.687023+0000","last_became_active":"2024-03-29T11:11:42.714528+0000","last_became_peered":"2024-03-29T11:11:42.714528+0000","last_unstale":"2024-03-29T11:11:43.687023+0000","last_undegraded":"2024-03-29T11:11:43.687023+0000","last_fullsized":"2024-03-29T11:11:43.687023+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:49:57.531934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897927+0000","last_change":"2024-03-29T11:11:42.702260+0000","last_active":"2024-03-29T11:11:43.897927+0000","last_peered":"2024-03-29T11:11:43.897927+0000","last_clean":"2024-03-29T11:11:43.897927+0000","last_became_active":"2024-03-29T11:11:42.701679+0000","last_became_peered":"2024-03-29T11:11:42.701679+0000","last_unstale":"2024-03-29T11:11:43.897927+0000","last_undegraded":"2024-03-29T11:11:43.897927+0000","last_fullsized":"2024-03-29T11:11:43.897927+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:38.740771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897776+0000","last_change":"2024-03-29T11:11:42.695013+0000","last_active":"2024-03-29T11:11:43.897776+0000","last_peered":"2024-03-29T11:11:43.897776+0000","last_clean":"2024-03-29T11:11:43.897776+0000","last_became_active":"2024-03-29T11:11:42.691620+0000","last_became_peered":"2024-03-29T11:11:42.691620+0000","last_unstale":"2024-03-29T11:11:43.897776+0000","last_undegraded":"2024-03-29T11:11:43.897776+0000","last_fullsized":"2024-03-29T11:11:43.897776+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:20:23.171886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897404+0000","last_change":"2024-03-29T11:11:42.702245+0000","last_active":"2024-03-29T11:11:43.897404+0000","last_peered":"2024-03-29T11:11:43.897404+0000","last_clean":"2024-03-29T11:11:43.897404+0000","last_became_active":"2024-03-29T11:11:42.701974+0000","last_became_peered":"2024-03-29T11:11:42.701974+0000","last_unstale":"2024-03-29T11:11:43.897404+0000","last_undegraded":"2024-03-29T11:11:43.897404+0000","last_fullsized":"2024-03-29T11:11:43.897404+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:00:17.163599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687564+0000","last_change":"2024-03-29T11:11:42.702315+0000","last_active":"2024-03-29T11:11:43.687564+0000","last_peered":"2024-03-29T11:11:43.687564+0000","last_clean":"2024-03-29T11:11:43.687564+0000","last_became_active":"2024-03-29T11:11:42.700860+0000","last_became_peered":"2024-03-29T11:11:42.700860+0000","last_unstale":"2024-03-29T11:11:43.687564+0000","last_undegraded":"2024-03-29T11:11:43.687564+0000","last_fullsized":"2024-03-29T11:11:43.687564+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:00:17.163599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897414+0000","last_change":"2024-03-29T11:11:42.694112+0000","last_active":"2024-03-29T11:11:43.897414+0000","last_peered":"2024-03-29T11:11:43.897414+0000","last_clean":"2024-03-29T11:11:43.897414+0000","last_became_active":"2024-03-29T11:11:42.693095+0000","last_became_peered":"2024-03-29T11:11:42.693095+0000","last_unstale":"2024-03-29T11:11:43.897414+0000","last_undegraded":"2024-03-29T11:11:43.897414+0000","last_fullsized":"2024-03-29T11:11:43.897414+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:31.936715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897944+0000","last_change":"2024-03-29T11:11:42.697353+0000","last_active":"2024-03-29T11:11:43.897944+0000","last_peered":"2024-03-29T11:11:43.897944+0000","last_clean":"2024-03-29T11:11:43.897944+0000","last_became_active":"2024-03-29T11:11:42.697058+0000","last_became_peered":"2024-03-29T11:11:42.697058+0000","last_unstale":"2024-03-29T11:11:43.897944+0000","last_undegraded":"2024-03-29T11:11:43.897944+0000","last_fullsized":"2024-03-29T11:11:43.897944+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:50:35.067769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.731961+0000","last_change":"2024-03-29T11:11:42.692838+0000","last_active":"2024-03-29T11:11:43.731961+0000","last_peered":"2024-03-29T11:11:43.731961+0000","last_clean":"2024-03-29T11:11:43.731961+0000","last_became_active":"2024-03-29T11:11:42.692568+0000","last_became_peered":"2024-03-29T11:11:42.692568+0000","last_unstale":"2024-03-29T11:11:43.731961+0000","last_undegraded":"2024-03-29T11:11:43.731961+0000","last_fullsized":"2024-03-29T11:11:43.731961+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:44:43.804682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.896962+0000","last_change":"2024-03-29T11:11:42.690613+0000","last_active":"2024-03-29T11:11:43.896962+0000","last_peered":"2024-03-29T11:11:43.896962+0000","last_clean":"2024-03-29T11:11:43.896962+0000","last_became_active":"2024-03-29T11:11:42.690355+0000","last_became_peered":"2024-03-29T11:11:42.690355+0000","last_unstale":"2024-03-29T11:11:43.896962+0000","last_undegraded":"2024-03-29T11:11:43.896962+0000","last_fullsized":"2024-03-29T11:11:43.896962+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:50:24.866795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687978+0000","last_change":"2024-03-29T11:11:42.708795+0000","last_active":"2024-03-29T11:11:43.687978+0000","last_peered":"2024-03-29T11:11:43.687978+0000","last_clean":"2024-03-29T11:11:43.687978+0000","last_became_active":"2024-03-29T11:11:42.708479+0000","last_became_peered":"2024-03-29T11:11:42.708479+0000","last_unstale":"2024-03-29T11:11:43.687978+0000","last_undegraded":"2024-03-29T11:11:43.687978+0000","last_fullsized":"2024-03-29T11:11:43.687978+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:32:14.333157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"176'4","reported_seq":15,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687998+0000","last_change":"2024-03-29T11:11:42.172711+0000","last_active":"2024-03-29T11:11:43.687998+0000","last_peered":"2024-03-29T11:11:43.687998+0000","last_clean":"2024-03-29T11:11:43.687998+0000","last_became_active":"2024-03-29T11:11:42.171990+0000","last_became_peered":"2024-03-29T11:11:42.171990+0000","last_unstale":"2024-03-29T11:11:43.687998+0000","last_undegraded":"2024-03-29T11:11:43.687998+0000","last_fullsized":"2024-03-29T11:11:43.687998+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:41:26.589206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732339+0000","last_change":"2024-03-29T11:11:42.699326+0000","last_active":"2024-03-29T11:11:43.732339+0000","last_peered":"2024-03-29T11:11:43.732339+0000","last_clean":"2024-03-29T11:11:43.732339+0000","last_became_active":"2024-03-29T11:11:42.698533+0000","last_became_peered":"2024-03-29T11:11:42.698533+0000","last_unstale":"2024-03-29T11:11:43.732339+0000","last_undegraded":"2024-03-29T11:11:43.732339+0000","last_fullsized":"2024-03-29T11:11:43.732339+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:58:46.323993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"176'1","reported_seq":12,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732337+0000","last_change":"2024-03-29T11:11:42.168645+0000","last_active":"2024-03-29T11:11:43.732337+0000","last_peered":"2024-03-29T11:11:43.732337+0000","last_clean":"2024-03-29T11:11:43.732337+0000","last_became_active":"2024-03-29T11:11:42.168324+0000","last_became_peered":"2024-03-29T11:11:42.168324+0000","last_unstale":"2024-03-29T11:11:43.732337+0000","last_undegraded":"2024-03-29T11:11:43.732337+0000","last_fullsized":"2024-03-29T11:11:43.732337+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:18.796043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897546+0000","last_change":"2024-03-29T11:11:42.706050+0000","last_active":"2024-03-29T11:11:43.897546+0000","last_peered":"2024-03-29T11:11:43.897546+0000","last_clean":"2024-03-29T11:11:43.897546+0000","last_became_active":"2024-03-29T11:11:42.705536+0000","last_became_peered":"2024-03-29T11:11:42.705536+0000","last_unstale":"2024-03-29T11:11:43.897546+0000","last_undegraded":"2024-03-29T11:11:43.897546+0000","last_fullsized":"2024-03-29T11:11:43.897546+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:38:53.794122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898865+0000","last_change":"2024-03-29T11:11:42.164554+0000","last_active":"2024-03-29T11:11:43.898865+0000","last_peered":"2024-03-29T11:11:43.898865+0000","last_clean":"2024-03-29T11:11:43.898865+0000","last_became_active":"2024-03-29T11:11:42.163066+0000","last_became_peered":"2024-03-29T11:11:42.163066+0000","last_unstale":"2024-03-29T11:11:43.898865+0000","last_undegraded":"2024-03-29T11:11:43.898865+0000","last_fullsized":"2024-03-29T11:11:43.898865+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:09:26.825675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687758+0000","last_change":"2024-03-29T11:11:42.707268+0000","last_active":"2024-03-29T11:11:43.687758+0000","last_peered":"2024-03-29T11:11:43.687758+0000","last_clean":"2024-03-29T11:11:43.687758+0000","last_became_active":"2024-03-29T11:11:42.706991+0000","last_became_peered":"2024-03-29T11:11:42.706991+0000","last_unstale":"2024-03-29T11:11:43.687758+0000","last_undegraded":"2024-03-29T11:11:43.687758+0000","last_fullsized":"2024-03-29T11:11:43.687758+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:41.619044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898746+0000","last_change":"2024-03-29T11:11:42.162842+0000","last_active":"2024-03-29T11:11:43.898746+0000","last_peered":"2024-03-29T11:11:43.898746+0000","last_clean":"2024-03-29T11:11:43.898746+0000","last_became_active":"2024-03-29T11:11:42.162077+0000","last_became_peered":"2024-03-29T11:11:42.162077+0000","last_unstale":"2024-03-29T11:11:43.898746+0000","last_undegraded":"2024-03-29T11:11:43.898746+0000","last_fullsized":"2024-03-29T11:11:43.898746+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:40.994946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732076+0000","last_change":"2024-03-29T11:11:42.693678+0000","last_active":"2024-03-29T11:11:43.732076+0000","last_peered":"2024-03-29T11:11:43.732076+0000","last_clean":"2024-03-29T11:11:43.732076+0000","last_became_active":"2024-03-29T11:11:42.693424+0000","last_became_peered":"2024-03-29T11:11:42.693424+0000","last_unstale":"2024-03-29T11:11:43.732076+0000","last_undegraded":"2024-03-29T11:11:43.732076+0000","last_fullsized":"2024-03-29T11:11:43.732076+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:20:03.594867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"176'1","reported_seq":12,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.896959+0000","last_change":"2024-03-29T11:11:42.169499+0000","last_active":"2024-03-29T11:11:43.896959+0000","last_peered":"2024-03-29T11:11:43.896959+0000","last_clean":"2024-03-29T11:11:43.896959+0000","last_became_active":"2024-03-29T11:11:42.169242+0000","last_became_peered":"2024-03-29T11:11:42.169242+0000","last_unstale":"2024-03-29T11:11:43.896959+0000","last_undegraded":"2024-03-29T11:11:43.896959+0000","last_fullsized":"2024-03-29T11:11:43.896959+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:29.684063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898404+0000","last_change":"2024-03-29T11:11:42.702189+0000","last_active":"2024-03-29T11:11:43.898404+0000","last_peered":"2024-03-29T11:11:43.898404+0000","last_clean":"2024-03-29T11:11:43.898404+0000","last_became_active":"2024-03-29T11:11:42.701886+0000","last_became_peered":"2024-03-29T11:11:42.701886+0000","last_unstale":"2024-03-29T11:11:43.898404+0000","last_undegraded":"2024-03-29T11:11:43.898404+0000","last_fullsized":"2024-03-29T11:11:43.898404+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:39.932543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"176'1","reported_seq":12,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898446+0000","last_change":"2024-03-29T11:11:42.166203+0000","last_active":"2024-03-29T11:11:43.898446+0000","last_peered":"2024-03-29T11:11:43.898446+0000","last_clean":"2024-03-29T11:11:43.898446+0000","last_became_active":"2024-03-29T11:11:42.165817+0000","last_became_peered":"2024-03-29T11:11:42.165817+0000","last_unstale":"2024-03-29T11:11:43.898446+0000","last_undegraded":"2024-03-29T11:11:43.898446+0000","last_fullsized":"2024-03-29T11:11:43.898446+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:10:56.835404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.731937+0000","last_change":"2024-03-29T11:11:42.688723+0000","last_active":"2024-03-29T11:11:43.731937+0000","last_peered":"2024-03-29T11:11:43.731937+0000","last_clean":"2024-03-29T11:11:43.731937+0000","last_became_active":"2024-03-29T11:11:42.687899+0000","last_became_peered":"2024-03-29T11:11:42.687899+0000","last_unstale":"2024-03-29T11:11:43.731937+0000","last_undegraded":"2024-03-29T11:11:43.731937+0000","last_fullsized":"2024-03-29T11:11:43.731937+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:12:35.058090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897126+0000","last_change":"2024-03-29T11:11:42.166337+0000","last_active":"2024-03-29T11:11:43.897126+0000","last_peered":"2024-03-29T11:11:43.897126+0000","last_clean":"2024-03-29T11:11:43.897126+0000","last_became_active":"2024-03-29T11:11:42.166024+0000","last_became_peered":"2024-03-29T11:11:42.166024+0000","last_unstale":"2024-03-29T11:11:43.897126+0000","last_undegraded":"2024-03-29T11:11:43.897126+0000","last_fullsized":"2024-03-29T11:11:43.897126+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:39:39.875286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898202+0000","last_change":"2024-03-29T11:11:42.684061+0000","last_active":"2024-03-29T11:11:43.898202+0000","last_peered":"2024-03-29T11:11:43.898202+0000","last_clean":"2024-03-29T11:11:43.898202+0000","last_became_active":"2024-03-29T11:11:42.683470+0000","last_became_peered":"2024-03-29T11:11:42.683470+0000","last_unstale":"2024-03-29T11:11:43.898202+0000","last_undegraded":"2024-03-29T11:11:43.898202+0000","last_fullsized":"2024-03-29T11:11:43.898202+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:01.527482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897890+0000","last_change":"2024-03-29T11:11:42.166895+0000","last_active":"2024-03-29T11:11:43.897890+0000","last_peered":"2024-03-29T11:11:43.897890+0000","last_clean":"2024-03-29T11:11:43.897890+0000","last_became_active":"2024-03-29T11:11:42.166429+0000","last_became_peered":"2024-03-29T11:11:42.166429+0000","last_unstale":"2024-03-29T11:11:43.897890+0000","last_undegraded":"2024-03-29T11:11:43.897890+0000","last_fullsized":"2024-03-29T11:11:43.897890+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:32:32.661750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898869+0000","last_change":"2024-03-29T11:11:42.693830+0000","last_active":"2024-03-29T11:11:43.898869+0000","last_peered":"2024-03-29T11:11:43.898869+0000","last_clean":"2024-03-29T11:11:43.898869+0000","last_became_active":"2024-03-29T11:11:42.693574+0000","last_became_peered":"2024-03-29T11:11:42.693574+0000","last_unstale":"2024-03-29T11:11:43.898869+0000","last_undegraded":"2024-03-29T11:11:43.898869+0000","last_fullsized":"2024-03-29T11:11:43.898869+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:37:34.185152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732355+0000","last_change":"2024-03-29T11:11:42.156169+0000","last_active":"2024-03-29T11:11:43.732355+0000","last_peered":"2024-03-29T11:11:43.732355+0000","last_clean":"2024-03-29T11:11:43.732355+0000","last_became_active":"2024-03-29T11:11:42.154063+0000","last_became_peered":"2024-03-29T11:11:42.154063+0000","last_unstale":"2024-03-29T11:11:43.732355+0000","last_undegraded":"2024-03-29T11:11:43.732355+0000","last_fullsized":"2024-03-29T11:11:43.732355+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:09:16.020857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688071+0000","last_change":"2024-03-29T11:11:42.703398+0000","last_active":"2024-03-29T11:11:43.688071+0000","last_peered":"2024-03-29T11:11:43.688071+0000","last_clean":"2024-03-29T11:11:43.688071+0000","last_became_active":"2024-03-29T11:11:42.703078+0000","last_became_peered":"2024-03-29T11:11:42.703078+0000","last_unstale":"2024-03-29T11:11:43.688071+0000","last_undegraded":"2024-03-29T11:11:43.688071+0000","last_fullsized":"2024-03-29T11:11:43.688071+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:56:35.200001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687837+0000","last_change":"2024-03-29T11:11:42.173479+0000","last_active":"2024-03-29T11:11:43.687837+0000","last_peered":"2024-03-29T11:11:43.687837+0000","last_clean":"2024-03-29T11:11:43.687837+0000","last_became_active":"2024-03-29T11:11:42.173094+0000","last_became_peered":"2024-03-29T11:11:42.173094+0000","last_unstale":"2024-03-29T11:11:43.687837+0000","last_undegraded":"2024-03-29T11:11:43.687837+0000","last_fullsized":"2024-03-29T11:11:43.687837+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:31:52.589200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897644+0000","last_change":"2024-03-29T11:11:42.697825+0000","last_active":"2024-03-29T11:11:43.897644+0000","last_peered":"2024-03-29T11:11:43.897644+0000","last_clean":"2024-03-29T11:11:43.897644+0000","last_became_active":"2024-03-29T11:11:42.697565+0000","last_became_peered":"2024-03-29T11:11:42.697565+0000","last_unstale":"2024-03-29T11:11:43.897644+0000","last_undegraded":"2024-03-29T11:11:43.897644+0000","last_fullsized":"2024-03-29T11:11:43.897644+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:17.020413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"176'6","reported_seq":17,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687776+0000","last_change":"2024-03-29T11:11:42.166993+0000","last_active":"2024-03-29T11:11:43.687776+0000","last_peered":"2024-03-29T11:11:43.687776+0000","last_clean":"2024-03-29T11:11:43.687776+0000","last_became_active":"2024-03-29T11:11:42.166199+0000","last_became_peered":"2024-03-29T11:11:42.166199+0000","last_unstale":"2024-03-29T11:11:43.687776+0000","last_undegraded":"2024-03-29T11:11:43.687776+0000","last_fullsized":"2024-03-29T11:11:43.687776+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:50:23.857081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687441+0000","last_change":"2024-03-29T11:11:42.709300+0000","last_active":"2024-03-29T11:11:43.687441+0000","last_peered":"2024-03-29T11:11:43.687441+0000","last_clean":"2024-03-29T11:11:43.687441+0000","last_became_active":"2024-03-29T11:11:42.708908+0000","last_became_peered":"2024-03-29T11:11:42.708908+0000","last_unstale":"2024-03-29T11:11:43.687441+0000","last_undegraded":"2024-03-29T11:11:43.687441+0000","last_fullsized":"2024-03-29T11:11:43.687441+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:39.932543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687543+0000","last_change":"2024-03-29T11:11:42.163983+0000","last_active":"2024-03-29T11:11:43.687543+0000","last_peered":"2024-03-29T11:11:43.687543+0000","last_clean":"2024-03-29T11:11:43.687543+0000","last_became_active":"2024-03-29T11:11:42.163515+0000","last_became_peered":"2024-03-29T11:11:42.163515+0000","last_unstale":"2024-03-29T11:11:43.687543+0000","last_undegraded":"2024-03-29T11:11:43.687543+0000","last_fullsized":"2024-03-29T11:11:43.687543+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:09:26.825675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.731968+0000","last_change":"2024-03-29T11:11:42.696305+0000","last_active":"2024-03-29T11:11:43.731968+0000","last_peered":"2024-03-29T11:11:43.731968+0000","last_clean":"2024-03-29T11:11:43.731968+0000","last_became_active":"2024-03-29T11:11:42.695997+0000","last_became_peered":"2024-03-29T11:11:42.695997+0000","last_unstale":"2024-03-29T11:11:43.731968+0000","last_undegraded":"2024-03-29T11:11:43.731968+0000","last_fullsized":"2024-03-29T11:11:43.731968+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:02:39.374556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687428+0000","last_change":"2024-03-29T11:11:42.169609+0000","last_active":"2024-03-29T11:11:43.687428+0000","last_peered":"2024-03-29T11:11:43.687428+0000","last_clean":"2024-03-29T11:11:43.687428+0000","last_became_active":"2024-03-29T11:11:42.169345+0000","last_became_peered":"2024-03-29T11:11:42.169345+0000","last_unstale":"2024-03-29T11:11:43.687428+0000","last_undegraded":"2024-03-29T11:11:43.687428+0000","last_fullsized":"2024-03-29T11:11:43.687428+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:32:32.661750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898119+0000","last_change":"2024-03-29T11:11:42.701480+0000","last_active":"2024-03-29T11:11:43.898119+0000","last_peered":"2024-03-29T11:11:43.898119+0000","last_clean":"2024-03-29T11:11:43.898119+0000","last_became_active":"2024-03-29T11:11:42.700384+0000","last_became_peered":"2024-03-29T11:11:42.700384+0000","last_unstale":"2024-03-29T11:11:43.898119+0000","last_undegraded":"2024-03-29T11:11:43.898119+0000","last_fullsized":"2024-03-29T11:11:43.898119+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:39:40.885000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897224+0000","last_change":"2024-03-29T11:11:42.159016+0000","last_active":"2024-03-29T11:11:43.897224+0000","last_peered":"2024-03-29T11:11:43.897224+0000","last_clean":"2024-03-29T11:11:43.897224+0000","last_became_active":"2024-03-29T11:11:42.158324+0000","last_became_peered":"2024-03-29T11:11:42.158324+0000","last_unstale":"2024-03-29T11:11:43.897224+0000","last_undegraded":"2024-03-29T11:11:43.897224+0000","last_fullsized":"2024-03-29T11:11:43.897224+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:50.173420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.686866+0000","last_change":"2024-03-29T11:11:42.712160+0000","last_active":"2024-03-29T11:11:43.686866+0000","last_peered":"2024-03-29T11:11:43.686866+0000","last_clean":"2024-03-29T11:11:43.686866+0000","last_became_active":"2024-03-29T11:11:42.711849+0000","last_became_peered":"2024-03-29T11:11:42.711849+0000","last_unstale":"2024-03-29T11:11:43.686866+0000","last_undegraded":"2024-03-29T11:11:43.686866+0000","last_fullsized":"2024-03-29T11:11:43.686866+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:06:09.555918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"176'8","reported_seq":19,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897056+0000","last_change":"2024-03-29T11:11:42.164069+0000","last_active":"2024-03-29T11:11:43.897056+0000","last_peered":"2024-03-29T11:11:43.897056+0000","last_clean":"2024-03-29T11:11:43.897056+0000","last_became_active":"2024-03-29T11:11:42.163452+0000","last_became_peered":"2024-03-29T11:11:42.163452+0000","last_unstale":"2024-03-29T11:11:43.897056+0000","last_undegraded":"2024-03-29T11:11:43.897056+0000","last_fullsized":"2024-03-29T11:11:43.897056+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:41.340475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688055+0000","last_change":"2024-03-29T11:11:42.699141+0000","last_active":"2024-03-29T11:11:43.688055+0000","last_peered":"2024-03-29T11:11:43.688055+0000","last_clean":"2024-03-29T11:11:43.688055+0000","last_became_active":"2024-03-29T11:11:42.696934+0000","last_became_peered":"2024-03-29T11:11:42.696934+0000","last_unstale":"2024-03-29T11:11:43.688055+0000","last_undegraded":"2024-03-29T11:11:43.688055+0000","last_fullsized":"2024-03-29T11:11:43.688055+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:38.222127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"176'1","reported_seq":12,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898666+0000","last_change":"2024-03-29T11:11:42.170635+0000","last_active":"2024-03-29T11:11:43.898666+0000","last_peered":"2024-03-29T11:11:43.898666+0000","last_clean":"2024-03-29T11:11:43.898666+0000","last_became_active":"2024-03-29T11:11:42.170193+0000","last_became_peered":"2024-03-29T11:11:42.170193+0000","last_unstale":"2024-03-29T11:11:43.898666+0000","last_undegraded":"2024-03-29T11:11:43.898666+0000","last_fullsized":"2024-03-29T11:11:43.898666+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:40:42.477104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897965+0000","last_change":"2024-03-29T11:11:42.701402+0000","last_active":"2024-03-29T11:11:43.897965+0000","last_peered":"2024-03-29T11:11:43.897965+0000","last_clean":"2024-03-29T11:11:43.897965+0000","last_became_active":"2024-03-29T11:11:42.699225+0000","last_became_peered":"2024-03-29T11:11:42.699225+0000","last_unstale":"2024-03-29T11:11:43.897965+0000","last_undegraded":"2024-03-29T11:11:43.897965+0000","last_fullsized":"2024-03-29T11:11:43.897965+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:28:18.369522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732321+0000","last_change":"2024-03-29T11:11:42.163402+0000","last_active":"2024-03-29T11:11:43.732321+0000","last_peered":"2024-03-29T11:11:43.732321+0000","last_clean":"2024-03-29T11:11:43.732321+0000","last_became_active":"2024-03-29T11:11:42.162543+0000","last_became_peered":"2024-03-29T11:11:42.162543+0000","last_unstale":"2024-03-29T11:11:43.732321+0000","last_undegraded":"2024-03-29T11:11:43.732321+0000","last_fullsized":"2024-03-29T11:11:43.732321+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:35:35.410629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687640+0000","last_change":"2024-03-29T11:11:42.709781+0000","last_active":"2024-03-29T11:11:43.687640+0000","last_peered":"2024-03-29T11:11:43.687640+0000","last_clean":"2024-03-29T11:11:43.687640+0000","last_became_active":"2024-03-29T11:11:42.708903+0000","last_became_peered":"2024-03-29T11:11:42.708903+0000","last_unstale":"2024-03-29T11:11:43.687640+0000","last_undegraded":"2024-03-29T11:11:43.687640+0000","last_fullsized":"2024-03-29T11:11:43.687640+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:15.795503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"176'1","reported_seq":12,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897835+0000","last_change":"2024-03-29T11:11:42.167931+0000","last_active":"2024-03-29T11:11:43.897835+0000","last_peered":"2024-03-29T11:11:43.897835+0000","last_clean":"2024-03-29T11:11:43.897835+0000","last_became_active":"2024-03-29T11:11:42.167649+0000","last_became_peered":"2024-03-29T11:11:42.167649+0000","last_unstale":"2024-03-29T11:11:43.897835+0000","last_undegraded":"2024-03-29T11:11:43.897835+0000","last_fullsized":"2024-03-29T11:11:43.897835+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:23.167351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897691+0000","last_change":"2024-03-29T11:11:42.692721+0000","last_active":"2024-03-29T11:11:43.897691+0000","last_peered":"2024-03-29T11:11:43.897691+0000","last_clean":"2024-03-29T11:11:43.897691+0000","last_became_active":"2024-03-29T11:11:42.692451+0000","last_became_peered":"2024-03-29T11:11:42.692451+0000","last_unstale":"2024-03-29T11:11:43.897691+0000","last_undegraded":"2024-03-29T11:11:43.897691+0000","last_fullsized":"2024-03-29T11:11:43.897691+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:46.888360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732149+0000","last_change":"2024-03-29T11:11:42.161771+0000","last_active":"2024-03-29T11:11:43.732149+0000","last_peered":"2024-03-29T11:11:43.732149+0000","last_clean":"2024-03-29T11:11:43.732149+0000","last_became_active":"2024-03-29T11:11:42.161277+0000","last_became_peered":"2024-03-29T11:11:42.161277+0000","last_unstale":"2024-03-29T11:11:43.732149+0000","last_undegraded":"2024-03-29T11:11:43.732149+0000","last_fullsized":"2024-03-29T11:11:43.732149+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:46:22.032350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898190+0000","last_change":"2024-03-29T11:11:42.696984+0000","last_active":"2024-03-29T11:11:43.898190+0000","last_peered":"2024-03-29T11:11:43.898190+0000","last_clean":"2024-03-29T11:11:43.898190+0000","last_became_active":"2024-03-29T11:11:42.696683+0000","last_became_peered":"2024-03-29T11:11:42.696683+0000","last_unstale":"2024-03-29T11:11:43.898190+0000","last_undegraded":"2024-03-29T11:11:43.898190+0000","last_fullsized":"2024-03-29T11:11:43.898190+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:25.203576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687839+0000","last_change":"2024-03-29T11:11:42.169084+0000","last_active":"2024-03-29T11:11:43.687839+0000","last_peered":"2024-03-29T11:11:43.687839+0000","last_clean":"2024-03-29T11:11:43.687839+0000","last_became_active":"2024-03-29T11:11:42.168782+0000","last_became_peered":"2024-03-29T11:11:42.168782+0000","last_unstale":"2024-03-29T11:11:43.687839+0000","last_undegraded":"2024-03-29T11:11:43.687839+0000","last_fullsized":"2024-03-29T11:11:43.687839+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:51.370586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687637+0000","last_change":"2024-03-29T11:11:42.708004+0000","last_active":"2024-03-29T11:11:43.687637+0000","last_peered":"2024-03-29T11:11:43.687637+0000","last_clean":"2024-03-29T11:11:43.687637+0000","last_became_active":"2024-03-29T11:11:42.705823+0000","last_became_peered":"2024-03-29T11:11:42.705823+0000","last_unstale":"2024-03-29T11:11:43.687637+0000","last_undegraded":"2024-03-29T11:11:43.687637+0000","last_fullsized":"2024-03-29T11:11:43.687637+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:48:37.344365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687568+0000","last_change":"2024-03-29T11:11:42.172832+0000","last_active":"2024-03-29T11:11:43.687568+0000","last_peered":"2024-03-29T11:11:43.687568+0000","last_clean":"2024-03-29T11:11:43.687568+0000","last_became_active":"2024-03-29T11:11:42.171272+0000","last_became_peered":"2024-03-29T11:11:42.171272+0000","last_unstale":"2024-03-29T11:11:43.687568+0000","last_undegraded":"2024-03-29T11:11:43.687568+0000","last_fullsized":"2024-03-29T11:11:43.687568+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:08.164844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897829+0000","last_change":"2024-03-29T11:11:42.697592+0000","last_active":"2024-03-29T11:11:43.897829+0000","last_peered":"2024-03-29T11:11:43.897829+0000","last_clean":"2024-03-29T11:11:43.897829+0000","last_became_active":"2024-03-29T11:11:42.695321+0000","last_became_peered":"2024-03-29T11:11:42.695321+0000","last_unstale":"2024-03-29T11:11:43.897829+0000","last_undegraded":"2024-03-29T11:11:43.897829+0000","last_fullsized":"2024-03-29T11:11:43.897829+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:06:09.555918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687001+0000","last_change":"2024-03-29T11:11:42.173035+0000","last_active":"2024-03-29T11:11:43.687001+0000","last_peered":"2024-03-29T11:11:43.687001+0000","last_clean":"2024-03-29T11:11:43.687001+0000","last_became_active":"2024-03-29T11:11:42.172433+0000","last_became_peered":"2024-03-29T11:11:42.172433+0000","last_unstale":"2024-03-29T11:11:43.687001+0000","last_undegraded":"2024-03-29T11:11:43.687001+0000","last_fullsized":"2024-03-29T11:11:43.687001+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:06.912154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687858+0000","last_change":"2024-03-29T11:11:42.708359+0000","last_active":"2024-03-29T11:11:43.687858+0000","last_peered":"2024-03-29T11:11:43.687858+0000","last_clean":"2024-03-29T11:11:43.687858+0000","last_became_active":"2024-03-29T11:11:42.708091+0000","last_became_peered":"2024-03-29T11:11:42.708091+0000","last_unstale":"2024-03-29T11:11:43.687858+0000","last_undegraded":"2024-03-29T11:11:43.687858+0000","last_fullsized":"2024-03-29T11:11:43.687858+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:09:17.030570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.686744+0000","last_change":"2024-03-29T11:11:42.161481+0000","last_active":"2024-03-29T11:11:43.686744+0000","last_peered":"2024-03-29T11:11:43.686744+0000","last_clean":"2024-03-29T11:11:43.686744+0000","last_became_active":"2024-03-29T11:11:42.161180+0000","last_became_peered":"2024-03-29T11:11:42.161180+0000","last_unstale":"2024-03-29T11:11:43.686744+0000","last_undegraded":"2024-03-29T11:11:43.686744+0000","last_fullsized":"2024-03-29T11:11:43.686744+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:23:12.857488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898837+0000","last_change":"2024-03-29T11:11:42.701673+0000","last_active":"2024-03-29T11:11:43.898837+0000","last_peered":"2024-03-29T11:11:43.898837+0000","last_clean":"2024-03-29T11:11:43.898837+0000","last_became_active":"2024-03-29T11:11:42.701434+0000","last_became_peered":"2024-03-29T11:11:42.701434+0000","last_unstale":"2024-03-29T11:11:43.898837+0000","last_undegraded":"2024-03-29T11:11:43.898837+0000","last_fullsized":"2024-03-29T11:11:43.898837+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:42.350189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688129+0000","last_change":"2024-03-29T11:11:42.174379+0000","last_active":"2024-03-29T11:11:43.688129+0000","last_peered":"2024-03-29T11:11:43.688129+0000","last_clean":"2024-03-29T11:11:43.688129+0000","last_became_active":"2024-03-29T11:11:42.174011+0000","last_became_peered":"2024-03-29T11:11:42.174011+0000","last_unstale":"2024-03-29T11:11:43.688129+0000","last_undegraded":"2024-03-29T11:11:43.688129+0000","last_fullsized":"2024-03-29T11:11:43.688129+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:47:52.801700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688098+0000","last_change":"2024-03-29T11:11:42.705933+0000","last_active":"2024-03-29T11:11:43.688098+0000","last_peered":"2024-03-29T11:11:43.688098+0000","last_clean":"2024-03-29T11:11:43.688098+0000","last_became_active":"2024-03-29T11:11:42.705469+0000","last_became_peered":"2024-03-29T11:11:42.705469+0000","last_unstale":"2024-03-29T11:11:43.688098+0000","last_undegraded":"2024-03-29T11:11:43.688098+0000","last_fullsized":"2024-03-29T11:11:43.688098+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:36:23.384386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897878+0000","last_change":"2024-03-29T11:11:42.168786+0000","last_active":"2024-03-29T11:11:43.897878+0000","last_peered":"2024-03-29T11:11:43.897878+0000","last_clean":"2024-03-29T11:11:43.897878+0000","last_became_active":"2024-03-29T11:11:42.168456+0000","last_became_peered":"2024-03-29T11:11:42.168456+0000","last_unstale":"2024-03-29T11:11:43.897878+0000","last_undegraded":"2024-03-29T11:11:43.897878+0000","last_fullsized":"2024-03-29T11:11:43.897878+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:14:38.572924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732113+0000","last_change":"2024-03-29T11:11:42.706818+0000","last_active":"2024-03-29T11:11:43.732113+0000","last_peered":"2024-03-29T11:11:43.732113+0000","last_clean":"2024-03-29T11:11:43.732113+0000","last_became_active":"2024-03-29T11:11:42.706565+0000","last_became_peered":"2024-03-29T11:11:42.706565+0000","last_unstale":"2024-03-29T11:11:43.732113+0000","last_undegraded":"2024-03-29T11:11:43.732113+0000","last_fullsized":"2024-03-29T11:11:43.732113+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:31:12.718930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897785+0000","last_change":"2024-03-29T11:11:42.165604+0000","last_active":"2024-03-29T11:11:43.897785+0000","last_peered":"2024-03-29T11:11:43.897785+0000","last_clean":"2024-03-29T11:11:43.897785+0000","last_became_active":"2024-03-29T11:11:42.165331+0000","last_became_peered":"2024-03-29T11:11:42.165331+0000","last_unstale":"2024-03-29T11:11:43.897785+0000","last_undegraded":"2024-03-29T11:11:43.897785+0000","last_fullsized":"2024-03-29T11:11:43.897785+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:38:12.028601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897697+0000","last_change":"2024-03-29T11:11:42.705271+0000","last_active":"2024-03-29T11:11:43.897697+0000","last_peered":"2024-03-29T11:11:43.897697+0000","last_clean":"2024-03-29T11:11:43.897697+0000","last_became_active":"2024-03-29T11:11:42.704950+0000","last_became_peered":"2024-03-29T11:11:42.704950+0000","last_unstale":"2024-03-29T11:11:43.897697+0000","last_undegraded":"2024-03-29T11:11:43.897697+0000","last_fullsized":"2024-03-29T11:11:43.897697+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:35.841005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897555+0000","last_change":"2024-03-29T11:11:42.165161+0000","last_active":"2024-03-29T11:11:43.897555+0000","last_peered":"2024-03-29T11:11:43.897555+0000","last_clean":"2024-03-29T11:11:43.897555+0000","last_became_active":"2024-03-29T11:11:42.164914+0000","last_became_peered":"2024-03-29T11:11:42.164914+0000","last_unstale":"2024-03-29T11:11:43.897555+0000","last_undegraded":"2024-03-29T11:11:43.897555+0000","last_fullsized":"2024-03-29T11:11:43.897555+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:01:26.637229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897317+0000","last_change":"2024-03-29T11:11:42.708069+0000","last_active":"2024-03-29T11:11:43.897317+0000","last_peered":"2024-03-29T11:11:43.897317+0000","last_clean":"2024-03-29T11:11:43.897317+0000","last_became_active":"2024-03-29T11:11:42.707545+0000","last_became_peered":"2024-03-29T11:11:42.707545+0000","last_unstale":"2024-03-29T11:11:43.897317+0000","last_undegraded":"2024-03-29T11:11:43.897317+0000","last_fullsized":"2024-03-29T11:11:43.897317+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:56.542489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687809+0000","last_change":"2024-03-29T11:11:42.166021+0000","last_active":"2024-03-29T11:11:43.687809+0000","last_peered":"2024-03-29T11:11:43.687809+0000","last_clean":"2024-03-29T11:11:43.687809+0000","last_became_active":"2024-03-29T11:11:42.164640+0000","last_became_peered":"2024-03-29T11:11:42.164640+0000","last_unstale":"2024-03-29T11:11:43.687809+0000","last_undegraded":"2024-03-29T11:11:43.687809+0000","last_fullsized":"2024-03-29T11:11:43.687809+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:22:19.536481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898559+0000","last_change":"2024-03-29T11:11:42.691925+0000","last_active":"2024-03-29T11:11:43.898559+0000","last_peered":"2024-03-29T11:11:43.898559+0000","last_clean":"2024-03-29T11:11:43.898559+0000","last_became_active":"2024-03-29T11:11:42.690964+0000","last_became_peered":"2024-03-29T11:11:42.690964+0000","last_unstale":"2024-03-29T11:11:43.898559+0000","last_undegraded":"2024-03-29T11:11:43.898559+0000","last_fullsized":"2024-03-29T11:11:43.898559+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:24.177064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687479+0000","last_change":"2024-03-29T11:11:42.172364+0000","last_active":"2024-03-29T11:11:43.687479+0000","last_peered":"2024-03-29T11:11:43.687479+0000","last_clean":"2024-03-29T11:11:43.687479+0000","last_became_active":"2024-03-29T11:11:42.172088+0000","last_became_peered":"2024-03-29T11:11:42.172088+0000","last_unstale":"2024-03-29T11:11:43.687479+0000","last_undegraded":"2024-03-29T11:11:43.687479+0000","last_fullsized":"2024-03-29T11:11:43.687479+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:41:09.655073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687099+0000","last_change":"2024-03-29T11:11:42.709893+0000","last_active":"2024-03-29T11:11:43.687099+0000","last_peered":"2024-03-29T11:11:43.687099+0000","last_clean":"2024-03-29T11:11:43.687099+0000","last_became_active":"2024-03-29T11:11:42.708475+0000","last_became_peered":"2024-03-29T11:11:42.708475+0000","last_unstale":"2024-03-29T11:11:43.687099+0000","last_undegraded":"2024-03-29T11:11:43.687099+0000","last_fullsized":"2024-03-29T11:11:43.687099+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:06:03.923489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.686957+0000","last_change":"2024-03-29T11:11:42.166342+0000","last_active":"2024-03-29T11:11:43.686957+0000","last_peered":"2024-03-29T11:11:43.686957+0000","last_clean":"2024-03-29T11:11:43.686957+0000","last_became_active":"2024-03-29T11:11:42.165200+0000","last_became_peered":"2024-03-29T11:11:42.165200+0000","last_unstale":"2024-03-29T11:11:43.686957+0000","last_undegraded":"2024-03-29T11:11:43.686957+0000","last_fullsized":"2024-03-29T11:11:43.686957+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:40.994946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898328+0000","last_change":"2024-03-29T11:11:42.690865+0000","last_active":"2024-03-29T11:11:43.898328+0000","last_peered":"2024-03-29T11:11:43.898328+0000","last_clean":"2024-03-29T11:11:43.898328+0000","last_became_active":"2024-03-29T11:11:42.690576+0000","last_became_peered":"2024-03-29T11:11:42.690576+0000","last_unstale":"2024-03-29T11:11:43.898328+0000","last_undegraded":"2024-03-29T11:11:43.898328+0000","last_fullsized":"2024-03-29T11:11:43.898328+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:36:27.374779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732050+0000","last_change":"2024-03-29T11:11:42.166419+0000","last_active":"2024-03-29T11:11:43.732050+0000","last_peered":"2024-03-29T11:11:43.732050+0000","last_clean":"2024-03-29T11:11:43.732050+0000","last_became_active":"2024-03-29T11:11:42.166120+0000","last_became_peered":"2024-03-29T11:11:42.166120+0000","last_unstale":"2024-03-29T11:11:43.732050+0000","last_undegraded":"2024-03-29T11:11:43.732050+0000","last_fullsized":"2024-03-29T11:11:43.732050+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:48:36.334651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687670+0000","last_change":"2024-03-29T11:11:42.708475+0000","last_active":"2024-03-29T11:11:43.687670+0000","last_peered":"2024-03-29T11:11:43.687670+0000","last_clean":"2024-03-29T11:11:43.687670+0000","last_became_active":"2024-03-29T11:11:42.707818+0000","last_became_peered":"2024-03-29T11:11:42.707818+0000","last_unstale":"2024-03-29T11:11:43.687670+0000","last_undegraded":"2024-03-29T11:11:43.687670+0000","last_fullsized":"2024-03-29T11:11:43.687670+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:57:28.272002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"176'3","reported_seq":14,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688034+0000","last_change":"2024-03-29T11:11:42.168852+0000","last_active":"2024-03-29T11:11:43.688034+0000","last_peered":"2024-03-29T11:11:43.688034+0000","last_clean":"2024-03-29T11:11:43.688034+0000","last_became_active":"2024-03-29T11:11:42.168580+0000","last_became_peered":"2024-03-29T11:11:42.168580+0000","last_unstale":"2024-03-29T11:11:43.688034+0000","last_undegraded":"2024-03-29T11:11:43.688034+0000","last_fullsized":"2024-03-29T11:11:43.688034+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:00.517768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"166'289","reported_seq":945,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-03-29T11:11:42.692990+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:11:42.692990+0000","last_peered":"2024-03-29T11:11:42.692990+0000","last_clean":"2024-03-29T11:11:42.692990+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:11:42.692990+0000","last_undegraded":"2024-03-29T11:11:42.692990+0000","last_fullsized":"2024-03-29T11:11:42.692990+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":471,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":471,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":289,"ondisk_log_size":289,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575764,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47808,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324032,"statfs":{"total":96636764160,"available":96587808768,"internally_reserved":0,"allocated":2797568,"data_stored":1651380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":46135756},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.439,"5min":0.41899999999999998,"15min":0.40000000000000002},"max":{"1min":0.90100000000000002,"5min":1.0089999999999999,"15min":1.014},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.749,"15min":0.746},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.39600000000000002},"max":{"1min":0.97099999999999997,"5min":0.98599999999999999,"15min":1.046},"last":0.629}]},{"osd":1,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.56999999999999995,"5min":0.498,"15min":0.46400000000000002},"max":{"1min":0.85899999999999999,"5min":0.92900000000000005,"15min":1.008},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76200000000000001,"15min":0.77200000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.48499999999999999},"max":{"1min":0.93600000000000005,"5min":1.0009999999999999,"15min":1.081},"last":0.46700000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.75,"15min":0.75800000000000001},"min":{"1min":0.56299999999999994,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.98599999999999999,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.59299999999999997,"5min":0.54600000000000004,"15min":0.51100000000000001},"max":{"1min":0.98099999999999998,"5min":0.995,"15min":1.2310000000000001},"last":0.67300000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.76300000000000001,"15min":0.76500000000000001},"min":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.42999999999999999},"max":{"1min":0.91000000000000003,"5min":0.95399999999999996,"15min":1.091},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.78700000000000003,"15min":0.78800000000000003},"min":{"1min":0.55800000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.96099999999999997,"5min":1.028,"15min":1.0509999999999999},"last":0.71799999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.63900000000000001,"15min":0.629},"min":{"1min":0.44800000000000001,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.85199999999999998,"5min":0.875,"15min":1.1499999999999999},"last":0.498},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.42999999999999999},"max":{"1min":0.85899999999999999,"5min":0.92500000000000004,"15min":1.208},"last":0.755}]},{"osd":5,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.66300000000000003},"min":{"1min":0.45200000000000001,"5min":0.44700000000000001,"15min":0.39100000000000001},"max":{"1min":0.92700000000000005,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.70999999999999996,"15min":0.70499999999999996},"min":{"1min":0.56999999999999995,"5min":0.48699999999999999,"15min":0.41799999999999998},"max":{"1min":0.88600000000000001,"5min":1.0580000000000001,"15min":1.107},"last":0.80000000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.75600000000000001,"15min":0.75},"min":{"1min":0.5,"5min":0.47999999999999998,"15min":0.47499999999999998},"max":{"1min":0.94099999999999995,"5min":1.008,"15min":1.101},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73699999999999999,"15min":0.72799999999999998},"min":{"1min":0.51700000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.1819999999999999},"last":0.78700000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575765,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41484,"kb_used_data":2168,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330356,"statfs":{"total":96636764160,"available":96594284544,"internally_reserved":0,"allocated":2220032,"data_stored":1066850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.73299999999999998,"15min":0.73599999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.38900000000000001},"max":{"1min":0.90000000000000002,"5min":1.002,"15min":1.002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.751,"15min":0.751},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.36799999999999999},"max":{"1min":0.86799999999999999,"5min":1.012,"15min":1.135},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.74199999999999999,"15min":0.73899999999999999},"min":{"1min":0.60299999999999998,"5min":0.47999999999999998,"15min":0.40799999999999997},"max":{"1min":0.93799999999999994,"5min":0.94599999999999995,"15min":1.01},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77700000000000002,"15min":0.77200000000000002},"min":{"1min":0.53800000000000003,"5min":0.496,"15min":0.439},"max":{"1min":0.98999999999999999,"5min":1.04,"15min":1.04},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.75800000000000001,"15min":0.75700000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48699999999999999},"max":{"1min":0.92400000000000004,"5min":0.99299999999999999,"15min":1.0189999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77800000000000002,"15min":0.77400000000000002},"min":{"1min":0.624,"5min":0.50700000000000001,"15min":0.49399999999999999},"max":{"1min":0.95699999999999996,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.73699999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.77900000000000003,"15min":0.77200000000000002},"min":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.52200000000000002},"max":{"1min":0.93899999999999995,"5min":0.96099999999999997,"15min":1.024},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80200000000000005,"15min":0.79600000000000004},"min":{"1min":0.63500000000000001,"5min":0.54600000000000004,"15min":0.54000000000000004},"max":{"1min":1.0169999999999999,"5min":1.0229999999999999,"15min":1.03},"last":0.51500000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.621,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42799999999999999,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.75},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.53400000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.80400000000000005,"5min":0.98799999999999999,"15min":1.0469999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.65500000000000003},"min":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.435},"max":{"1min":0.89100000000000001,"5min":1.0049999999999999,"15min":1.125},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.71899999999999997,"15min":0.68600000000000005},"min":{"1min":0.51700000000000002,"5min":0.438,"15min":0.40400000000000003},"max":{"1min":0.94499999999999995,"5min":1.0629999999999999,"15min":1.0980000000000001},"last":0.71099999999999997}]},{"osd":7,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77100000000000002,"15min":0.752},"min":{"1min":0.59399999999999997,"5min":0.56699999999999995,"15min":0.52800000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.077},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.745,"15min":0.72399999999999998},"min":{"1min":0.57599999999999996,"5min":0.53300000000000003,"15min":0.46200000000000002},"max":{"1min":0.875,"5min":1.0660000000000001,"15min":1.0760000000000001},"last":0.623}]}]},{"osd":1,"up_from":13,"seq":55834575763,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55412,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316428,"statfs":{"total":96636764160,"available":96580022272,"internally_reserved":0,"allocated":2199552,"data_stored":1060942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.63200000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.91000000000000003},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":0.79200000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.372,"15min":0.34399999999999997},"max":{"1min":0.70999999999999996,"5min":0.88100000000000001,"15min":0.91800000000000004},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.622,"15min":0.622},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76200000000000001,"5min":0.81599999999999995,"15min":0.97599999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.42199999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.752,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.52500000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81000000000000005,"15min":0.79900000000000004},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.53800000000000003},"max":{"1min":1.139,"5min":1.139,"15min":1.1950000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.65400000000000003,"5min":0.625,"15min":0.57699999999999996},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.68899999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.83799999999999997,"15min":0.83199999999999996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.85899999999999999,"15min":0.84299999999999997},"min":{"1min":0.68799999999999994,"5min":0.60699999999999998,"15min":0.59099999999999997},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.60699999999999998}]},{"osd":6,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.84699999999999998},"min":{"1min":0.63,"5min":0.63,"15min":0.61099999999999999},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.879,"15min":0.86299999999999999},"min":{"1min":0.67200000000000004,"5min":0.63200000000000001,"15min":0.56799999999999995},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.75}]},{"osd":7,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.89300000000000002,"15min":0.88100000000000001},"min":{"1min":0.79200000000000004,"5min":0.627,"15min":0.56000000000000005},"max":{"1min":1.1419999999999999,"5min":1.1459999999999999,"15min":1.1819999999999999},"last":0.63},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.89300000000000002,"15min":0.88},"min":{"1min":0.71699999999999997,"5min":0.57499999999999996,"15min":0.55000000000000004},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.73099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575765,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60096,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311744,"statfs":{"total":96636764160,"available":96575225856,"internally_reserved":0,"allocated":2801664,"data_stored":1651307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46700000000000003,"15min":0.45900000000000002},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58499999999999996,"5min":0.83299999999999996,"15min":0.89800000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34200000000000003},"max":{"1min":0.73999999999999999,"5min":0.79700000000000004,"15min":1.03},"last":0.34599999999999997}]},{"osd":2,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.42599999999999999,"15min":0.36799999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":1.1459999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52100000000000002,"15min":0.51400000000000001},"min":{"1min":0.39600000000000002,"5min":0.37,"15min":0.34899999999999998},"max":{"1min":0.61099999999999999,"5min":0.81599999999999995,"15min":0.88700000000000001},"last":0.68799999999999994}]},{"osd":3,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.56799999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.36699999999999999},"max":{"1min":0.80900000000000005,"5min":0.85199999999999998,"15min":0.85899999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.64100000000000001},"min":{"1min":0.53700000000000003,"5min":0.49399999999999999,"15min":0.433},"max":{"1min":0.84799999999999998,"5min":0.873,"15min":1.226},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.83099999999999996,"15min":0.82599999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.48899999999999999},"max":{"1min":1.1419999999999999,"5min":1.304,"15min":1.304},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86299999999999999,"15min":0.86499999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.50700000000000001},"max":{"1min":1.107,"5min":1.1100000000000001,"15min":1.2809999999999999},"last":0.92500000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.81200000000000006,"15min":0.79300000000000004},"min":{"1min":0.66700000000000004,"5min":0.53400000000000003,"15min":0.48399999999999999},"max":{"1min":1.125,"5min":1.127,"15min":1.129},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85799999999999998,"15min":0.84099999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.54000000000000004},"max":{"1min":1.1279999999999999,"5min":1.27,"15min":1.27},"last":0.61799999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.86099999999999999,"15min":0.86199999999999999},"min":{"1min":0.66000000000000003,"5min":0.628,"15min":0.628},"max":{"1min":1.115,"5min":1.1679999999999999,"15min":1.329},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.90300000000000002,"15min":0.89100000000000001},"min":{"1min":0.69399999999999995,"5min":0.61699999999999999,"15min":0.58099999999999996},"max":{"1min":1.2150000000000001,"5min":1.337,"15min":1.337},"last":0.95399999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.84899999999999998,"15min":0.83199999999999996},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.53400000000000003},"max":{"1min":1.0960000000000001,"5min":1.161,"15min":1.181},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89000000000000001,"15min":0.88600000000000001},"min":{"1min":0.67500000000000004,"5min":0.65100000000000002,"15min":0.56499999999999995},"max":{"1min":1.1839999999999999,"5min":1.1890000000000001,"15min":1.298},"last":1.0389999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47236,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324604,"statfs":{"total":96636764160,"available":96588394496,"internally_reserved":0,"allocated":2211840,"data_stored":1061994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.35899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.621,"5min":0.72299999999999998,"15min":0.77700000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.42299999999999999,"5min":0.36499999999999999,"15min":0.34200000000000003},"max":{"1min":0.78500000000000003,"5min":0.80000000000000004,"15min":0.878},"last":0.60699999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.38600000000000001,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":0.64700000000000002,"5min":0.68799999999999994,"15min":0.81999999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.53300000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.72499999999999998,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.496}]},{"osd":3,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61799999999999999,"15min":0.61399999999999999},"min":{"1min":0.503,"5min":0.436,"15min":0.41499999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.44900000000000001,"5min":0.39200000000000002,"15min":0.374},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.90800000000000003},"last":0.39600000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.81000000000000005,"15min":0.81799999999999995},"min":{"1min":0.66700000000000004,"5min":0.56699999999999995,"15min":0.54800000000000004},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.135},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83399999999999996,"15min":0.82299999999999995},"min":{"1min":0.68400000000000005,"5min":0.60699999999999998,"15min":0.56699999999999995},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.3680000000000001},"last":0.77100000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.84199999999999997,"15min":0.83099999999999996},"min":{"1min":0.65500000000000003,"5min":0.54300000000000004,"15min":0.5},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2709999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.61699999999999999,"5min":0.60699999999999998,"15min":0.56299999999999994},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.73499999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.85799999999999998,"15min":0.85299999999999998},"min":{"1min":0.63500000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.304,"5min":1.321,"15min":1.343},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.86299999999999999,"15min":0.85999999999999999},"min":{"1min":0.69899999999999995,"5min":0.59499999999999997,"15min":0.56899999999999995},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.622}]},{"osd":7,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.92100000000000004,"15min":0.90600000000000003},"min":{"1min":0.72899999999999998,"5min":0.67300000000000004,"15min":0.58199999999999996},"max":{"1min":1.252,"5min":1.2729999999999999,"15min":1.403},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.93000000000000005,"15min":0.91600000000000004},"min":{"1min":0.72499999999999998,"5min":0.68100000000000005,"15min":0.58699999999999997},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.86899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575764,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59524,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312316,"statfs":{"total":96636764160,"available":96575811584,"internally_reserved":0,"allocated":2211840,"data_stored":1066794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.48099999999999998,"15min":0.47699999999999998},"min":{"1min":0.33200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71299999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.45100000000000001,"5min":0.35999999999999999,"15min":0.31},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.499}]},{"osd":1,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51300000000000001,"15min":0.505},"min":{"1min":0.377,"5min":0.377,"15min":0.33100000000000002},"max":{"1min":0.746,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58399999999999996,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.40799999999999997,"15min":0.38900000000000001},"max":{"1min":0.78700000000000003,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.41399999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.38400000000000001},"max":{"1min":0.69499999999999995,"5min":0.875,"15min":0.88},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.67800000000000005,"15min":0.66500000000000004},"min":{"1min":0.58999999999999997,"5min":0.52100000000000002,"15min":0.436},"max":{"1min":0.85199999999999998,"5min":0.91900000000000004,"15min":0.98799999999999999},"last":0.53600000000000003}]},{"osd":4,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.624,"5min":0.61899999999999999,"15min":0.46000000000000002},"max":{"1min":0.98399999999999999,"5min":1.2629999999999999,"15min":1.5469999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.81100000000000005,"15min":0.81200000000000006},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.48199999999999998},"max":{"1min":0.96799999999999997,"5min":1.1519999999999999,"15min":1.5309999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.82599999999999996,"15min":0.82699999999999996},"min":{"1min":0.64900000000000002,"5min":0.55100000000000005,"15min":0.52400000000000002},"max":{"1min":1.02,"5min":1.113,"15min":1.4199999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.84099999999999997,"15min":0.84299999999999997},"min":{"1min":0.68999999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.006,"5min":1.3140000000000001,"15min":1.3740000000000001},"last":0.80900000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.85599999999999998,"15min":0.85899999999999999},"min":{"1min":0.77500000000000002,"5min":0.60999999999999999,"15min":0.56399999999999995},"max":{"1min":1.1100000000000001,"5min":1.1339999999999999,"15min":1.248},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.86399999999999999,"15min":0.85799999999999998},"min":{"1min":0.67400000000000004,"5min":0.65100000000000002,"15min":0.59399999999999997},"max":{"1min":1.0569999999999999,"5min":1.212,"15min":1.2849999999999999},"last":0.73399999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.874,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.095,"5min":1.1910000000000001,"15min":1.6020000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.90000000000000002,"15min":0.89500000000000002},"min":{"1min":0.75,"5min":0.68000000000000005,"15min":0.64500000000000002},"max":{"1min":1.0740000000000001,"5min":1.2350000000000001,"15min":1.3340000000000001},"last":0.91900000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575765,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51312,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320528,"statfs":{"total":96636764160,"available":96584220672,"internally_reserved":0,"allocated":2195456,"data_stored":1060360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74199999999999999,"15min":0.73799999999999999},"min":{"1min":0.55200000000000005,"5min":0.52000000000000002,"15min":0.33300000000000002},"max":{"1min":0.96199999999999997,"5min":1.0209999999999999,"15min":1.097},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72299999999999998,"15min":0.72799999999999998},"min":{"1min":0.51500000000000001,"5min":0.44400000000000001,"15min":0.40899999999999997},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.3300000000000001},"last":0.72899999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.76300000000000001,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.627},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74099999999999999,"15min":0.74399999999999999},"min":{"1min":0.51300000000000001,"5min":0.502,"15min":0.42599999999999999},"max":{"1min":0.86699999999999999,"5min":0.97099999999999997,"15min":1.0269999999999999},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.82599999999999996},"min":{"1min":0.60299999999999998,"5min":0.54500000000000004,"15min":0.437},"max":{"1min":1.034,"5min":1.034,"15min":1.3080000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.81100000000000005,"15min":0.81299999999999994},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.46800000000000003},"max":{"1min":0.95499999999999996,"5min":1.0349999999999999,"15min":1.1140000000000001},"last":0.71499999999999997}]},{"osd":3,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76700000000000002,"15min":0.76100000000000001},"min":{"1min":0.58699999999999997,"5min":0.58399999999999996,"15min":0.53700000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":1.177},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.75600000000000001},"min":{"1min":0.57899999999999996,"5min":0.51800000000000002,"15min":0.48699999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.51600000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60999999999999999,"15min":0.60299999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.88100000000000001,"5min":0.93100000000000005,"15min":1.054},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.44700000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.1779999999999999},"last":0.58699999999999997}]},{"osd":6,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.499,"5min":0.47499999999999998,"15min":0.443},"max":{"1min":1.038,"5min":1.038,"15min":1.0389999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68700000000000006,"15min":0.67600000000000005},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.437},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":1.254},"last":0.65200000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.69499999999999995,"15min":0.69099999999999995},"min":{"1min":0.50600000000000001,"5min":0.47899999999999998,"15min":0.44900000000000001},"max":{"1min":1.004,"5min":1.004,"15min":1.4379999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.78900000000000003,"15min":0.79000000000000004},"min":{"1min":0.67300000000000004,"5min":0.58899999999999997,"15min":0.52400000000000002},"max":{"1min":1.032,"5min":1.0580000000000001,"15min":1.482},"last":0.77000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575764,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41464,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330376,"statfs":{"total":96636764160,"available":96594305024,"internally_reserved":0,"allocated":2199552,"data_stored":1060382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.73999999999999999,"15min":0.72699999999999998},"min":{"1min":0.624,"5min":0.47199999999999998,"15min":0.40699999999999997},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.74099999999999999,"15min":0.748},"min":{"1min":0.59599999999999997,"5min":0.48099999999999998,"15min":0.45000000000000001},"max":{"1min":0.91900000000000004,"5min":0.94799999999999995,"15min":1},"last":0.58299999999999996}]},{"osd":1,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.76200000000000001,"15min":0.76800000000000002},"min":{"1min":0.67500000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.95599999999999996,"5min":1.0189999999999999,"15min":1.0549999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54000000000000004,"5min":0.52500000000000002,"15min":0.51600000000000001},"max":{"1min":0.97799999999999998,"5min":0.997,"15min":1.052},"last":0.66400000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.76700000000000002,"15min":0.76400000000000001},"min":{"1min":0.58899999999999997,"5min":0.499,"15min":0.47199999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.0609999999999999},"last":0.503},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.74199999999999999,"15min":0.746},"min":{"1min":0.53300000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.78500000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75,"15min":0.77100000000000002},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.52300000000000002},"max":{"1min":1,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.78900000000000003,"15min":0.78700000000000003},"min":{"1min":0.65200000000000002,"5min":0.55800000000000005,"15min":0.54600000000000004},"max":{"1min":0.97499999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.61599999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.47499999999999998,"5min":0.38800000000000001,"15min":0.379},"max":{"1min":0.73299999999999998,"5min":0.92700000000000005,"15min":0.93899999999999995},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.46899999999999997,"5min":0.45300000000000001,"15min":0.41199999999999998},"max":{"1min":0.78800000000000003,"5min":0.95199999999999996,"15min":0.98299999999999998},"last":0.76500000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.65000000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.439},"max":{"1min":0.90900000000000003,"5min":0.93999999999999995,"15min":0.95399999999999996},"last":0.751},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65600000000000003,"15min":0.67100000000000004},"min":{"1min":0.50900000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.84799999999999998,"5min":0.86499999999999999,"15min":0.95499999999999996},"last":0.82399999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.50900000000000001,"5min":0.46800000000000003,"15min":0.46500000000000002},"max":{"1min":0.86199999999999999,"5min":0.90500000000000003,"15min":1.01},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.746,"5min":0.71699999999999997,"15min":0.73799999999999999},"min":{"1min":0.56000000000000005,"5min":0.52500000000000002,"15min":0.49399999999999999},"max":{"1min":0.94399999999999995,"5min":0.95399999999999996,"15min":1.0369999999999999},"last":0.84699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:11:47.577 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:11:47.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.832+0000 7ff388000700 1 Processor -- start 2024-03-29T11:11:47.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.832+0000 7ff388000700 1 -- start start 2024-03-29T11:11:47.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.833+0000 7ff388000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff380161ca0 0x7ff3800cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.833+0000 7ff388000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3800ceda0 con 0x7ff3800d18b0 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.833+0000 7ff388000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3801612f0 con 0x7ff380160f10 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.833+0000 7ff388000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800d1d30 con 0x7ff380161ca0 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.833+0000 7ff384c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff3800d18b0 0x7ff380161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60216/0 (socket says 172.21.15.12:60216) 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.833+0000 7ff384c00700 1 -- 172.21.15.12:0/514567036 learned_addr learned my addr 172.21.15.12:0/514567036 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff386000700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff380161ca0 0x7ff3800cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622588088 0 0) 0x7ff3800ceda0 con 0x7ff3800d18b0 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3680035b0 con 0x7ff3800d18b0 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171499216 0 0) 0x7ff3801612f0 con 0x7ff380160f10 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3800ceda0 con 0x7ff380160f10 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3309825152 0 0) 0x7ff3680035b0 con 0x7ff3800d18b0 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff380161ca0 msgr2=0x7ff3800cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff380161ca0 0x7ff3800cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 >> v1:172.21.15.159:6789/0 conn(0x7ff380160f10 legacy=0x7ff3800d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800ce470 con 0x7ff3800d18b0 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7ff36c001670 con 0x7ff3800d18b0 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 >> v1:172.21.15.12:6789/0 conn(0x7ff3800d18b0 legacy=0x7ff380161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff368003bc0 0x7ff368003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff368004850 0x7ff368004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff380161ca0 0x7ff368005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800d1d30 con 0x7ff368003bc0 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36800bba0 con 0x7ff368004850 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36800bd60 con 0x7ff380161ca0 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.834+0000 7ff385600700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff368003bc0 0x7ff368003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff386000700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff380161ca0 0x7ff368005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff384c00700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff368004850 0x7ff368004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff385600700 1 -- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff380161ca0 msgr2=0x7ff368005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff385600700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff380161ca0 0x7ff368005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff385600700 1 -- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff368004850 msgr2=0x7ff368004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff385600700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff368004850 0x7ff368004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff385600700 1 -- 172.21.15.12:0/514567036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3680035b0 con 0x7ff368003bc0 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff385600700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff368003bc0 0x7ff368003fc0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7ff374001980 tx=0x7ff368005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cab03668c2d30a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374001470 con 0x7ff368003bc0 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3740052d0 con 0x7ff368003bc0 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.835+0000 7ff37fe00700 1 -- 172.21.15.12:0/514567036 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374003680 con 0x7ff368003bc0 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.836+0000 7ff388000700 1 -- 172.21.15.12:0/514567036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff368003bc0 msgr2=0x7ff368003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.836+0000 7ff388000700 1 --2- 172.21.15.12:0/514567036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff368003bc0 0x7ff368003fc0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7ff374001980 tx=0x7ff368005880 comp rx=0 tx=0).stop 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.836+0000 7ff386000700 1 -- 172.21.15.12:0/514567036 reap_dead start 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.836+0000 7ff388000700 1 -- 172.21.15.12:0/514567036 shutdown_connections 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.836+0000 7ff388000700 1 -- 172.21.15.12:0/514567036 >> 172.21.15.12:0/514567036 conn(0x7ff38008aff0 msgr2=0x7ff3800d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.836+0000 7ff388000700 1 -- 172.21.15.12:0/514567036 shutdown_connections 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.836+0000 7ff388000700 1 -- 172.21.15.12:0/514567036 wait complete. 2024-03-29T11:11:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.838+0000 7ff388000700 1 Processor -- start 2024-03-29T11:11:47.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.838+0000 7ff388000700 1 -- start start 2024-03-29T11:11:47.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff388000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff368003bc0 0x7ff38014b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff388000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff38009d3a0 0x7ff38009d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff388000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff38015e0a0 0x7ff38015e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff388000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38009dd60 con 0x7ff38015e0a0 2024-03-29T11:11:47.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff388000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3801e98f0 con 0x7ff38009d3a0 2024-03-29T11:11:47.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff388000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3801e8ce0 con 0x7ff368003bc0 2024-03-29T11:11:47.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff386000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff38015e0a0 0x7ff38015e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff386000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff38015e0a0 0x7ff38015e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58324/0 (socket says 172.21.15.12:58324) 2024-03-29T11:11:47.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff386000700 1 -- 172.21.15.12:0/1944365738 learned_addr learned my addr 172.21.15.12:0/1944365738 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:47.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff385600700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff368003bc0 0x7ff38014b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.839+0000 7ff384c00700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff38009d3a0 0x7ff38009d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff386000700 1 -- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff368003bc0 msgr2=0x7ff38014b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff386000700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff368003bc0 0x7ff38014b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff386000700 1 -- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff38009d3a0 msgr2=0x7ff38009d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff386000700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff38009d3a0 0x7ff38009d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff386000700 1 -- 172.21.15.12:0/1944365738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3801e8fc0 con 0x7ff38015e0a0 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff384c00700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff38009d3a0 0x7ff38009d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff386000700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff38015e0a0 0x7ff38015e520 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7ff3680047d0 tx=0x7ff37800f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff37d600700 1 -- 172.21.15.12:0/1944365738 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37800e040 con 0x7ff38015e0a0 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff37d600700 1 -- 172.21.15.12:0/1944365738 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37800f040 con 0x7ff38015e0a0 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff37d600700 1 -- 172.21.15.12:0/1944365738 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378015430 con 0x7ff38015e0a0 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff388000700 1 -- 172.21.15.12:0/1944365738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3801e92a0 con 0x7ff38015e0a0 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.840+0000 7ff388000700 1 -- 172.21.15.12:0/1944365738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38014bdb0 con 0x7ff38015e0a0 2024-03-29T11:11:47.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.841+0000 7ff388000700 1 -- 172.21.15.12:0/1944365738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff380067720 con 0x7ff38015e0a0 2024-03-29T11:11:47.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.842+0000 7ff37d600700 1 -- 172.21.15.12:0/1944365738 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff3780155b0 con 0x7ff38015e0a0 2024-03-29T11:11:47.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.843+0000 7ff37d600700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff36006adb0 0x7ff36006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:47.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.843+0000 7ff37d600700 1 -- 172.21.15.12:0/1944365738 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7ff378089be0 con 0x7ff38015e0a0 2024-03-29T11:11:47.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.844+0000 7ff385600700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff36006adb0 0x7ff36006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:47.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.844+0000 7ff385600700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff36006adb0 0x7ff36006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff374000c00 tx=0x7ff37400a000 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:47.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:47.848+0000 7ff37d600700 1 -- 172.21.15.12:0/1944365738 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff378054b60 con 0x7ff38015e0a0 2024-03-29T11:11:48.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.058+0000 7ff388000700 1 -- 172.21.15.12:0/1944365738 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3800ce120 con 0x7ff36006adb0 2024-03-29T11:11:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.067+0000 7ff37d600700 1 -- 172.21.15.12:0/1944365738 <== mgr.7844 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289559 (secure 0 0 0) 0x7ff3740050d0 con 0x7ff36006adb0 2024-03-29T11:11:48.049 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 -- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff36006adb0 msgr2=0x7ff36006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff36006adb0 0x7ff36006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff374000c00 tx=0x7ff37400a000 comp rx=0 tx=0).stop 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 -- 172.21.15.12:0/1944365738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff38015e0a0 msgr2=0x7ff38015e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff38015e0a0 0x7ff38015e520 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7ff3680047d0 tx=0x7ff37800f490 comp rx=0 tx=0).stop 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 -- 172.21.15.12:0/1944365738 shutdown_connections 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff36006adb0 0x7ff36006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff38015e0a0 0x7ff38015e520 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff368003bc0 0x7ff38014b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 --2- 172.21.15.12:0/1944365738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff38009d3a0 0x7ff38009d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 -- 172.21.15.12:0/1944365738 >> 172.21.15.12:0/1944365738 conn(0x7ff38008aff0 msgr2=0x7ff3800d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 -- 172.21.15.12:0/1944365738 shutdown_connections 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.071+0000 7ff388000700 1 -- 172.21.15.12:0/1944365738 wait complete. 2024-03-29T11:11:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:11:48.068 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-29T11:11:47.986360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":515,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":327,"ondisk_log_size":327,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":404336,"kb_used_data":18396,"kb_used_omap":12,"kb_used_meta":385779,"kb_avail":754570384,"statfs":{"total":773094113280,"available":772680073216,"internally_reserved":0,"allocated":18837504,"data_stored":9680009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":395038265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001502"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898045+0000","last_change":"2024-03-29T11:11:42.696332+0000","last_active":"2024-03-29T11:11:43.898045+0000","last_peered":"2024-03-29T11:11:43.898045+0000","last_clean":"2024-03-29T11:11:43.898045+0000","last_became_active":"2024-03-29T11:11:42.696080+0000","last_became_peered":"2024-03-29T11:11:42.696080+0000","last_unstale":"2024-03-29T11:11:43.898045+0000","last_undegraded":"2024-03-29T11:11:43.898045+0000","last_fullsized":"2024-03-29T11:11:43.898045+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:38:18.000649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732251+0000","last_change":"2024-03-29T11:11:42.700813+0000","last_active":"2024-03-29T11:11:43.732251+0000","last_peered":"2024-03-29T11:11:43.732251+0000","last_clean":"2024-03-29T11:11:43.732251+0000","last_became_active":"2024-03-29T11:11:42.699888+0000","last_became_peered":"2024-03-29T11:11:42.699888+0000","last_unstale":"2024-03-29T11:11:43.732251+0000","last_undegraded":"2024-03-29T11:11:43.732251+0000","last_fullsized":"2024-03-29T11:11:43.732251+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:07:02.476319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688154+0000","last_change":"2024-03-29T11:11:42.688985+0000","last_active":"2024-03-29T11:11:43.688154+0000","last_peered":"2024-03-29T11:11:43.688154+0000","last_clean":"2024-03-29T11:11:43.688154+0000","last_became_active":"2024-03-29T11:11:42.688679+0000","last_became_peered":"2024-03-29T11:11:42.688679+0000","last_unstale":"2024-03-29T11:11:43.688154+0000","last_undegraded":"2024-03-29T11:11:43.688154+0000","last_fullsized":"2024-03-29T11:11:43.688154+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:25.203576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898469+0000","last_change":"2024-03-29T11:11:42.704758+0000","last_active":"2024-03-29T11:11:43.898469+0000","last_peered":"2024-03-29T11:11:43.898469+0000","last_clean":"2024-03-29T11:11:43.898469+0000","last_became_active":"2024-03-29T11:11:42.704501+0000","last_became_peered":"2024-03-29T11:11:42.704501+0000","last_unstale":"2024-03-29T11:11:43.898469+0000","last_undegraded":"2024-03-29T11:11:43.898469+0000","last_fullsized":"2024-03-29T11:11:43.898469+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:22:20.546194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898291+0000","last_change":"2024-03-29T11:11:42.696374+0000","last_active":"2024-03-29T11:11:43.898291+0000","last_peered":"2024-03-29T11:11:43.898291+0000","last_clean":"2024-03-29T11:11:43.898291+0000","last_became_active":"2024-03-29T11:11:42.696099+0000","last_became_peered":"2024-03-29T11:11:42.696099+0000","last_unstale":"2024-03-29T11:11:43.898291+0000","last_undegraded":"2024-03-29T11:11:43.898291+0000","last_fullsized":"2024-03-29T11:11:43.898291+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:30.693777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897402+0000","last_change":"2024-03-29T11:11:42.694844+0000","last_active":"2024-03-29T11:11:43.897402+0000","last_peered":"2024-03-29T11:11:43.897402+0000","last_clean":"2024-03-29T11:11:43.897402+0000","last_became_active":"2024-03-29T11:11:42.694543+0000","last_became_peered":"2024-03-29T11:11:42.694543+0000","last_unstale":"2024-03-29T11:11:43.897402+0000","last_undegraded":"2024-03-29T11:11:43.897402+0000","last_fullsized":"2024-03-29T11:11:43.897402+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:41:28.309201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687499+0000","last_change":"2024-03-29T11:11:42.708861+0000","last_active":"2024-03-29T11:11:43.687499+0000","last_peered":"2024-03-29T11:11:43.687499+0000","last_clean":"2024-03-29T11:11:43.687499+0000","last_became_active":"2024-03-29T11:11:42.708572+0000","last_became_peered":"2024-03-29T11:11:42.708572+0000","last_unstale":"2024-03-29T11:11:43.687499+0000","last_undegraded":"2024-03-29T11:11:43.687499+0000","last_fullsized":"2024-03-29T11:11:43.687499+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:46:23.042064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.896976+0000","last_change":"2024-03-29T11:11:42.690600+0000","last_active":"2024-03-29T11:11:43.896976+0000","last_peered":"2024-03-29T11:11:43.896976+0000","last_clean":"2024-03-29T11:11:43.896976+0000","last_became_active":"2024-03-29T11:11:42.688865+0000","last_became_peered":"2024-03-29T11:11:42.688865+0000","last_unstale":"2024-03-29T11:11:43.896976+0000","last_undegraded":"2024-03-29T11:11:43.896976+0000","last_fullsized":"2024-03-29T11:11:43.896976+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:33:41.185232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732305+0000","last_change":"2024-03-29T11:11:42.702419+0000","last_active":"2024-03-29T11:11:43.732305+0000","last_peered":"2024-03-29T11:11:43.732305+0000","last_clean":"2024-03-29T11:11:43.732305+0000","last_became_active":"2024-03-29T11:11:42.702115+0000","last_became_peered":"2024-03-29T11:11:42.702115+0000","last_unstale":"2024-03-29T11:11:43.732305+0000","last_undegraded":"2024-03-29T11:11:43.732305+0000","last_fullsized":"2024-03-29T11:11:43.732305+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:13:17.783041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688194+0000","last_change":"2024-03-29T11:11:42.713204+0000","last_active":"2024-03-29T11:11:43.688194+0000","last_peered":"2024-03-29T11:11:43.688194+0000","last_clean":"2024-03-29T11:11:43.688194+0000","last_became_active":"2024-03-29T11:11:42.712895+0000","last_became_peered":"2024-03-29T11:11:42.712895+0000","last_unstale":"2024-03-29T11:11:43.688194+0000","last_undegraded":"2024-03-29T11:11:43.688194+0000","last_fullsized":"2024-03-29T11:11:43.688194+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:57:33.428782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897469+0000","last_change":"2024-03-29T11:11:42.686764+0000","last_active":"2024-03-29T11:11:43.897469+0000","last_peered":"2024-03-29T11:11:43.897469+0000","last_clean":"2024-03-29T11:11:43.897469+0000","last_became_active":"2024-03-29T11:11:42.685357+0000","last_became_peered":"2024-03-29T11:11:42.685357+0000","last_unstale":"2024-03-29T11:11:43.897469+0000","last_undegraded":"2024-03-29T11:11:43.897469+0000","last_fullsized":"2024-03-29T11:11:43.897469+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:37:43.164221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687898+0000","last_change":"2024-03-29T11:11:42.698405+0000","last_active":"2024-03-29T11:11:43.687898+0000","last_peered":"2024-03-29T11:11:43.687898+0000","last_clean":"2024-03-29T11:11:43.687898+0000","last_became_active":"2024-03-29T11:11:42.698113+0000","last_became_peered":"2024-03-29T11:11:42.698113+0000","last_unstale":"2024-03-29T11:11:43.687898+0000","last_undegraded":"2024-03-29T11:11:43.687898+0000","last_fullsized":"2024-03-29T11:11:43.687898+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:28.897598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898499+0000","last_change":"2024-03-29T11:11:42.701128+0000","last_active":"2024-03-29T11:11:43.898499+0000","last_peered":"2024-03-29T11:11:43.898499+0000","last_clean":"2024-03-29T11:11:43.898499+0000","last_became_active":"2024-03-29T11:11:42.700830+0000","last_became_peered":"2024-03-29T11:11:42.700830+0000","last_unstale":"2024-03-29T11:11:43.898499+0000","last_undegraded":"2024-03-29T11:11:43.898499+0000","last_fullsized":"2024-03-29T11:11:43.898499+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:27:36.222719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898341+0000","last_change":"2024-03-29T11:11:42.705814+0000","last_active":"2024-03-29T11:11:43.898341+0000","last_peered":"2024-03-29T11:11:43.898341+0000","last_clean":"2024-03-29T11:11:43.898341+0000","last_became_active":"2024-03-29T11:11:42.705537+0000","last_became_peered":"2024-03-29T11:11:42.705537+0000","last_unstale":"2024-03-29T11:11:43.898341+0000","last_undegraded":"2024-03-29T11:11:43.898341+0000","last_fullsized":"2024-03-29T11:11:43.898341+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:50:24.866795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.686887+0000","last_change":"2024-03-29T11:11:42.694257+0000","last_active":"2024-03-29T11:11:43.686887+0000","last_peered":"2024-03-29T11:11:43.686887+0000","last_clean":"2024-03-29T11:11:43.686887+0000","last_became_active":"2024-03-29T11:11:42.693808+0000","last_became_peered":"2024-03-29T11:11:42.693808+0000","last_unstale":"2024-03-29T11:11:43.686887+0000","last_undegraded":"2024-03-29T11:11:43.686887+0000","last_fullsized":"2024-03-29T11:11:43.686887+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:03:38.816350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897089+0000","last_change":"2024-03-29T11:11:42.697410+0000","last_active":"2024-03-29T11:11:43.897089+0000","last_peered":"2024-03-29T11:11:43.897089+0000","last_clean":"2024-03-29T11:11:43.897089+0000","last_became_active":"2024-03-29T11:11:42.692357+0000","last_became_peered":"2024-03-29T11:11:42.692357+0000","last_unstale":"2024-03-29T11:11:43.897089+0000","last_undegraded":"2024-03-29T11:11:43.897089+0000","last_fullsized":"2024-03-29T11:11:43.897089+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:41:11.037823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688523+0000","last_change":"2024-03-29T11:11:42.699688+0000","last_active":"2024-03-29T11:11:43.688523+0000","last_peered":"2024-03-29T11:11:43.688523+0000","last_clean":"2024-03-29T11:11:43.688523+0000","last_became_active":"2024-03-29T11:11:42.699442+0000","last_became_peered":"2024-03-29T11:11:42.699442+0000","last_unstale":"2024-03-29T11:11:43.688523+0000","last_undegraded":"2024-03-29T11:11:43.688523+0000","last_fullsized":"2024-03-29T11:11:43.688523+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:27:33.099004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687615+0000","last_change":"2024-03-29T11:11:42.692192+0000","last_active":"2024-03-29T11:11:43.687615+0000","last_peered":"2024-03-29T11:11:43.687615+0000","last_clean":"2024-03-29T11:11:43.687615+0000","last_became_active":"2024-03-29T11:11:42.691817+0000","last_became_peered":"2024-03-29T11:11:42.691817+0000","last_unstale":"2024-03-29T11:11:43.687615+0000","last_undegraded":"2024-03-29T11:11:43.687615+0000","last_fullsized":"2024-03-29T11:11:43.687615+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:50:35.067769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897550+0000","last_change":"2024-03-29T11:11:42.703702+0000","last_active":"2024-03-29T11:11:43.897550+0000","last_peered":"2024-03-29T11:11:43.897550+0000","last_clean":"2024-03-29T11:11:43.897550+0000","last_became_active":"2024-03-29T11:11:42.701416+0000","last_became_peered":"2024-03-29T11:11:42.701416+0000","last_unstale":"2024-03-29T11:11:43.897550+0000","last_undegraded":"2024-03-29T11:11:43.897550+0000","last_fullsized":"2024-03-29T11:11:43.897550+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:50:52.001882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897147+0000","last_change":"2024-03-29T11:11:42.694473+0000","last_active":"2024-03-29T11:11:43.897147+0000","last_peered":"2024-03-29T11:11:43.897147+0000","last_clean":"2024-03-29T11:11:43.897147+0000","last_became_active":"2024-03-29T11:11:42.693967+0000","last_became_peered":"2024-03-29T11:11:42.693967+0000","last_unstale":"2024-03-29T11:11:43.897147+0000","last_undegraded":"2024-03-29T11:11:43.897147+0000","last_fullsized":"2024-03-29T11:11:43.897147+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:56:55.325827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687890+0000","last_change":"2024-03-29T11:11:42.713124+0000","last_active":"2024-03-29T11:11:43.687890+0000","last_peered":"2024-03-29T11:11:43.687890+0000","last_clean":"2024-03-29T11:11:43.687890+0000","last_became_active":"2024-03-29T11:11:42.710967+0000","last_became_peered":"2024-03-29T11:11:42.710967+0000","last_unstale":"2024-03-29T11:11:43.687890+0000","last_undegraded":"2024-03-29T11:11:43.687890+0000","last_fullsized":"2024-03-29T11:11:43.687890+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:11.397620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898085+0000","last_change":"2024-03-29T11:11:42.695397+0000","last_active":"2024-03-29T11:11:43.898085+0000","last_peered":"2024-03-29T11:11:43.898085+0000","last_clean":"2024-03-29T11:11:43.898085+0000","last_became_active":"2024-03-29T11:11:42.695104+0000","last_became_peered":"2024-03-29T11:11:42.695104+0000","last_unstale":"2024-03-29T11:11:43.898085+0000","last_undegraded":"2024-03-29T11:11:43.898085+0000","last_fullsized":"2024-03-29T11:11:43.898085+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:36:23.384386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897934+0000","last_change":"2024-03-29T11:11:42.697718+0000","last_active":"2024-03-29T11:11:43.897934+0000","last_peered":"2024-03-29T11:11:43.897934+0000","last_clean":"2024-03-29T11:11:43.897934+0000","last_became_active":"2024-03-29T11:11:42.696020+0000","last_became_peered":"2024-03-29T11:11:42.696020+0000","last_unstale":"2024-03-29T11:11:43.897934+0000","last_undegraded":"2024-03-29T11:11:43.897934+0000","last_fullsized":"2024-03-29T11:11:43.897934+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:15.795503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687023+0000","last_change":"2024-03-29T11:11:42.714826+0000","last_active":"2024-03-29T11:11:43.687023+0000","last_peered":"2024-03-29T11:11:43.687023+0000","last_clean":"2024-03-29T11:11:43.687023+0000","last_became_active":"2024-03-29T11:11:42.714528+0000","last_became_peered":"2024-03-29T11:11:42.714528+0000","last_unstale":"2024-03-29T11:11:43.687023+0000","last_undegraded":"2024-03-29T11:11:43.687023+0000","last_fullsized":"2024-03-29T11:11:43.687023+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:49:57.531934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897927+0000","last_change":"2024-03-29T11:11:42.702260+0000","last_active":"2024-03-29T11:11:43.897927+0000","last_peered":"2024-03-29T11:11:43.897927+0000","last_clean":"2024-03-29T11:11:43.897927+0000","last_became_active":"2024-03-29T11:11:42.701679+0000","last_became_peered":"2024-03-29T11:11:42.701679+0000","last_unstale":"2024-03-29T11:11:43.897927+0000","last_undegraded":"2024-03-29T11:11:43.897927+0000","last_fullsized":"2024-03-29T11:11:43.897927+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:38.740771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897776+0000","last_change":"2024-03-29T11:11:42.695013+0000","last_active":"2024-03-29T11:11:43.897776+0000","last_peered":"2024-03-29T11:11:43.897776+0000","last_clean":"2024-03-29T11:11:43.897776+0000","last_became_active":"2024-03-29T11:11:42.691620+0000","last_became_peered":"2024-03-29T11:11:42.691620+0000","last_unstale":"2024-03-29T11:11:43.897776+0000","last_undegraded":"2024-03-29T11:11:43.897776+0000","last_fullsized":"2024-03-29T11:11:43.897776+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:20:23.171886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897404+0000","last_change":"2024-03-29T11:11:42.702245+0000","last_active":"2024-03-29T11:11:43.897404+0000","last_peered":"2024-03-29T11:11:43.897404+0000","last_clean":"2024-03-29T11:11:43.897404+0000","last_became_active":"2024-03-29T11:11:42.701974+0000","last_became_peered":"2024-03-29T11:11:42.701974+0000","last_unstale":"2024-03-29T11:11:43.897404+0000","last_undegraded":"2024-03-29T11:11:43.897404+0000","last_fullsized":"2024-03-29T11:11:43.897404+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:00:17.163599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687564+0000","last_change":"2024-03-29T11:11:42.702315+0000","last_active":"2024-03-29T11:11:43.687564+0000","last_peered":"2024-03-29T11:11:43.687564+0000","last_clean":"2024-03-29T11:11:43.687564+0000","last_became_active":"2024-03-29T11:11:42.700860+0000","last_became_peered":"2024-03-29T11:11:42.700860+0000","last_unstale":"2024-03-29T11:11:43.687564+0000","last_undegraded":"2024-03-29T11:11:43.687564+0000","last_fullsized":"2024-03-29T11:11:43.687564+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:00:17.163599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897414+0000","last_change":"2024-03-29T11:11:42.694112+0000","last_active":"2024-03-29T11:11:43.897414+0000","last_peered":"2024-03-29T11:11:43.897414+0000","last_clean":"2024-03-29T11:11:43.897414+0000","last_became_active":"2024-03-29T11:11:42.693095+0000","last_became_peered":"2024-03-29T11:11:42.693095+0000","last_unstale":"2024-03-29T11:11:43.897414+0000","last_undegraded":"2024-03-29T11:11:43.897414+0000","last_fullsized":"2024-03-29T11:11:43.897414+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:31.936715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897944+0000","last_change":"2024-03-29T11:11:42.697353+0000","last_active":"2024-03-29T11:11:43.897944+0000","last_peered":"2024-03-29T11:11:43.897944+0000","last_clean":"2024-03-29T11:11:43.897944+0000","last_became_active":"2024-03-29T11:11:42.697058+0000","last_became_peered":"2024-03-29T11:11:42.697058+0000","last_unstale":"2024-03-29T11:11:43.897944+0000","last_undegraded":"2024-03-29T11:11:43.897944+0000","last_fullsized":"2024-03-29T11:11:43.897944+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:50:35.067769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.731961+0000","last_change":"2024-03-29T11:11:42.692838+0000","last_active":"2024-03-29T11:11:43.731961+0000","last_peered":"2024-03-29T11:11:43.731961+0000","last_clean":"2024-03-29T11:11:43.731961+0000","last_became_active":"2024-03-29T11:11:42.692568+0000","last_became_peered":"2024-03-29T11:11:42.692568+0000","last_unstale":"2024-03-29T11:11:43.731961+0000","last_undegraded":"2024-03-29T11:11:43.731961+0000","last_fullsized":"2024-03-29T11:11:43.731961+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:44:43.804682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.896962+0000","last_change":"2024-03-29T11:11:42.690613+0000","last_active":"2024-03-29T11:11:43.896962+0000","last_peered":"2024-03-29T11:11:43.896962+0000","last_clean":"2024-03-29T11:11:43.896962+0000","last_became_active":"2024-03-29T11:11:42.690355+0000","last_became_peered":"2024-03-29T11:11:42.690355+0000","last_unstale":"2024-03-29T11:11:43.896962+0000","last_undegraded":"2024-03-29T11:11:43.896962+0000","last_fullsized":"2024-03-29T11:11:43.896962+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:50:24.866795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687978+0000","last_change":"2024-03-29T11:11:42.708795+0000","last_active":"2024-03-29T11:11:43.687978+0000","last_peered":"2024-03-29T11:11:43.687978+0000","last_clean":"2024-03-29T11:11:43.687978+0000","last_became_active":"2024-03-29T11:11:42.708479+0000","last_became_peered":"2024-03-29T11:11:42.708479+0000","last_unstale":"2024-03-29T11:11:43.687978+0000","last_undegraded":"2024-03-29T11:11:43.687978+0000","last_fullsized":"2024-03-29T11:11:43.687978+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:32:14.333157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"176'4","reported_seq":15,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687998+0000","last_change":"2024-03-29T11:11:42.172711+0000","last_active":"2024-03-29T11:11:43.687998+0000","last_peered":"2024-03-29T11:11:43.687998+0000","last_clean":"2024-03-29T11:11:43.687998+0000","last_became_active":"2024-03-29T11:11:42.171990+0000","last_became_peered":"2024-03-29T11:11:42.171990+0000","last_unstale":"2024-03-29T11:11:43.687998+0000","last_undegraded":"2024-03-29T11:11:43.687998+0000","last_fullsized":"2024-03-29T11:11:43.687998+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:41:26.589206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732339+0000","last_change":"2024-03-29T11:11:42.699326+0000","last_active":"2024-03-29T11:11:43.732339+0000","last_peered":"2024-03-29T11:11:43.732339+0000","last_clean":"2024-03-29T11:11:43.732339+0000","last_became_active":"2024-03-29T11:11:42.698533+0000","last_became_peered":"2024-03-29T11:11:42.698533+0000","last_unstale":"2024-03-29T11:11:43.732339+0000","last_undegraded":"2024-03-29T11:11:43.732339+0000","last_fullsized":"2024-03-29T11:11:43.732339+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:58:46.323993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"176'1","reported_seq":12,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732337+0000","last_change":"2024-03-29T11:11:42.168645+0000","last_active":"2024-03-29T11:11:43.732337+0000","last_peered":"2024-03-29T11:11:43.732337+0000","last_clean":"2024-03-29T11:11:43.732337+0000","last_became_active":"2024-03-29T11:11:42.168324+0000","last_became_peered":"2024-03-29T11:11:42.168324+0000","last_unstale":"2024-03-29T11:11:43.732337+0000","last_undegraded":"2024-03-29T11:11:43.732337+0000","last_fullsized":"2024-03-29T11:11:43.732337+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:18.796043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897546+0000","last_change":"2024-03-29T11:11:42.706050+0000","last_active":"2024-03-29T11:11:43.897546+0000","last_peered":"2024-03-29T11:11:43.897546+0000","last_clean":"2024-03-29T11:11:43.897546+0000","last_became_active":"2024-03-29T11:11:42.705536+0000","last_became_peered":"2024-03-29T11:11:42.705536+0000","last_unstale":"2024-03-29T11:11:43.897546+0000","last_undegraded":"2024-03-29T11:11:43.897546+0000","last_fullsized":"2024-03-29T11:11:43.897546+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:38:53.794122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898865+0000","last_change":"2024-03-29T11:11:42.164554+0000","last_active":"2024-03-29T11:11:43.898865+0000","last_peered":"2024-03-29T11:11:43.898865+0000","last_clean":"2024-03-29T11:11:43.898865+0000","last_became_active":"2024-03-29T11:11:42.163066+0000","last_became_peered":"2024-03-29T11:11:42.163066+0000","last_unstale":"2024-03-29T11:11:43.898865+0000","last_undegraded":"2024-03-29T11:11:43.898865+0000","last_fullsized":"2024-03-29T11:11:43.898865+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:09:26.825675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687758+0000","last_change":"2024-03-29T11:11:42.707268+0000","last_active":"2024-03-29T11:11:43.687758+0000","last_peered":"2024-03-29T11:11:43.687758+0000","last_clean":"2024-03-29T11:11:43.687758+0000","last_became_active":"2024-03-29T11:11:42.706991+0000","last_became_peered":"2024-03-29T11:11:42.706991+0000","last_unstale":"2024-03-29T11:11:43.687758+0000","last_undegraded":"2024-03-29T11:11:43.687758+0000","last_fullsized":"2024-03-29T11:11:43.687758+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:41.619044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898746+0000","last_change":"2024-03-29T11:11:42.162842+0000","last_active":"2024-03-29T11:11:43.898746+0000","last_peered":"2024-03-29T11:11:43.898746+0000","last_clean":"2024-03-29T11:11:43.898746+0000","last_became_active":"2024-03-29T11:11:42.162077+0000","last_became_peered":"2024-03-29T11:11:42.162077+0000","last_unstale":"2024-03-29T11:11:43.898746+0000","last_undegraded":"2024-03-29T11:11:43.898746+0000","last_fullsized":"2024-03-29T11:11:43.898746+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:40.994946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732076+0000","last_change":"2024-03-29T11:11:42.693678+0000","last_active":"2024-03-29T11:11:43.732076+0000","last_peered":"2024-03-29T11:11:43.732076+0000","last_clean":"2024-03-29T11:11:43.732076+0000","last_became_active":"2024-03-29T11:11:42.693424+0000","last_became_peered":"2024-03-29T11:11:42.693424+0000","last_unstale":"2024-03-29T11:11:43.732076+0000","last_undegraded":"2024-03-29T11:11:43.732076+0000","last_fullsized":"2024-03-29T11:11:43.732076+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:20:03.594867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"176'1","reported_seq":12,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.896959+0000","last_change":"2024-03-29T11:11:42.169499+0000","last_active":"2024-03-29T11:11:43.896959+0000","last_peered":"2024-03-29T11:11:43.896959+0000","last_clean":"2024-03-29T11:11:43.896959+0000","last_became_active":"2024-03-29T11:11:42.169242+0000","last_became_peered":"2024-03-29T11:11:42.169242+0000","last_unstale":"2024-03-29T11:11:43.896959+0000","last_undegraded":"2024-03-29T11:11:43.896959+0000","last_fullsized":"2024-03-29T11:11:43.896959+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:29.684063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898404+0000","last_change":"2024-03-29T11:11:42.702189+0000","last_active":"2024-03-29T11:11:43.898404+0000","last_peered":"2024-03-29T11:11:43.898404+0000","last_clean":"2024-03-29T11:11:43.898404+0000","last_became_active":"2024-03-29T11:11:42.701886+0000","last_became_peered":"2024-03-29T11:11:42.701886+0000","last_unstale":"2024-03-29T11:11:43.898404+0000","last_undegraded":"2024-03-29T11:11:43.898404+0000","last_fullsized":"2024-03-29T11:11:43.898404+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:39.932543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"176'1","reported_seq":12,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898446+0000","last_change":"2024-03-29T11:11:42.166203+0000","last_active":"2024-03-29T11:11:43.898446+0000","last_peered":"2024-03-29T11:11:43.898446+0000","last_clean":"2024-03-29T11:11:43.898446+0000","last_became_active":"2024-03-29T11:11:42.165817+0000","last_became_peered":"2024-03-29T11:11:42.165817+0000","last_unstale":"2024-03-29T11:11:43.898446+0000","last_undegraded":"2024-03-29T11:11:43.898446+0000","last_fullsized":"2024-03-29T11:11:43.898446+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:10:56.835404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.731937+0000","last_change":"2024-03-29T11:11:42.688723+0000","last_active":"2024-03-29T11:11:43.731937+0000","last_peered":"2024-03-29T11:11:43.731937+0000","last_clean":"2024-03-29T11:11:43.731937+0000","last_became_active":"2024-03-29T11:11:42.687899+0000","last_became_peered":"2024-03-29T11:11:42.687899+0000","last_unstale":"2024-03-29T11:11:43.731937+0000","last_undegraded":"2024-03-29T11:11:43.731937+0000","last_fullsized":"2024-03-29T11:11:43.731937+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:12:35.058090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897126+0000","last_change":"2024-03-29T11:11:42.166337+0000","last_active":"2024-03-29T11:11:43.897126+0000","last_peered":"2024-03-29T11:11:43.897126+0000","last_clean":"2024-03-29T11:11:43.897126+0000","last_became_active":"2024-03-29T11:11:42.166024+0000","last_became_peered":"2024-03-29T11:11:42.166024+0000","last_unstale":"2024-03-29T11:11:43.897126+0000","last_undegraded":"2024-03-29T11:11:43.897126+0000","last_fullsized":"2024-03-29T11:11:43.897126+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:39:39.875286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898202+0000","last_change":"2024-03-29T11:11:42.684061+0000","last_active":"2024-03-29T11:11:43.898202+0000","last_peered":"2024-03-29T11:11:43.898202+0000","last_clean":"2024-03-29T11:11:43.898202+0000","last_became_active":"2024-03-29T11:11:42.683470+0000","last_became_peered":"2024-03-29T11:11:42.683470+0000","last_unstale":"2024-03-29T11:11:43.898202+0000","last_undegraded":"2024-03-29T11:11:43.898202+0000","last_fullsized":"2024-03-29T11:11:43.898202+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:01.527482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897890+0000","last_change":"2024-03-29T11:11:42.166895+0000","last_active":"2024-03-29T11:11:43.897890+0000","last_peered":"2024-03-29T11:11:43.897890+0000","last_clean":"2024-03-29T11:11:43.897890+0000","last_became_active":"2024-03-29T11:11:42.166429+0000","last_became_peered":"2024-03-29T11:11:42.166429+0000","last_unstale":"2024-03-29T11:11:43.897890+0000","last_undegraded":"2024-03-29T11:11:43.897890+0000","last_fullsized":"2024-03-29T11:11:43.897890+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:32:32.661750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898869+0000","last_change":"2024-03-29T11:11:42.693830+0000","last_active":"2024-03-29T11:11:43.898869+0000","last_peered":"2024-03-29T11:11:43.898869+0000","last_clean":"2024-03-29T11:11:43.898869+0000","last_became_active":"2024-03-29T11:11:42.693574+0000","last_became_peered":"2024-03-29T11:11:42.693574+0000","last_unstale":"2024-03-29T11:11:43.898869+0000","last_undegraded":"2024-03-29T11:11:43.898869+0000","last_fullsized":"2024-03-29T11:11:43.898869+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:37:34.185152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732355+0000","last_change":"2024-03-29T11:11:42.156169+0000","last_active":"2024-03-29T11:11:43.732355+0000","last_peered":"2024-03-29T11:11:43.732355+0000","last_clean":"2024-03-29T11:11:43.732355+0000","last_became_active":"2024-03-29T11:11:42.154063+0000","last_became_peered":"2024-03-29T11:11:42.154063+0000","last_unstale":"2024-03-29T11:11:43.732355+0000","last_undegraded":"2024-03-29T11:11:43.732355+0000","last_fullsized":"2024-03-29T11:11:43.732355+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:09:16.020857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688071+0000","last_change":"2024-03-29T11:11:42.703398+0000","last_active":"2024-03-29T11:11:43.688071+0000","last_peered":"2024-03-29T11:11:43.688071+0000","last_clean":"2024-03-29T11:11:43.688071+0000","last_became_active":"2024-03-29T11:11:42.703078+0000","last_became_peered":"2024-03-29T11:11:42.703078+0000","last_unstale":"2024-03-29T11:11:43.688071+0000","last_undegraded":"2024-03-29T11:11:43.688071+0000","last_fullsized":"2024-03-29T11:11:43.688071+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:56:35.200001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687837+0000","last_change":"2024-03-29T11:11:42.173479+0000","last_active":"2024-03-29T11:11:43.687837+0000","last_peered":"2024-03-29T11:11:43.687837+0000","last_clean":"2024-03-29T11:11:43.687837+0000","last_became_active":"2024-03-29T11:11:42.173094+0000","last_became_peered":"2024-03-29T11:11:42.173094+0000","last_unstale":"2024-03-29T11:11:43.687837+0000","last_undegraded":"2024-03-29T11:11:43.687837+0000","last_fullsized":"2024-03-29T11:11:43.687837+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:31:52.589200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897644+0000","last_change":"2024-03-29T11:11:42.697825+0000","last_active":"2024-03-29T11:11:43.897644+0000","last_peered":"2024-03-29T11:11:43.897644+0000","last_clean":"2024-03-29T11:11:43.897644+0000","last_became_active":"2024-03-29T11:11:42.697565+0000","last_became_peered":"2024-03-29T11:11:42.697565+0000","last_unstale":"2024-03-29T11:11:43.897644+0000","last_undegraded":"2024-03-29T11:11:43.897644+0000","last_fullsized":"2024-03-29T11:11:43.897644+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:17.020413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"176'6","reported_seq":17,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687776+0000","last_change":"2024-03-29T11:11:42.166993+0000","last_active":"2024-03-29T11:11:43.687776+0000","last_peered":"2024-03-29T11:11:43.687776+0000","last_clean":"2024-03-29T11:11:43.687776+0000","last_became_active":"2024-03-29T11:11:42.166199+0000","last_became_peered":"2024-03-29T11:11:42.166199+0000","last_unstale":"2024-03-29T11:11:43.687776+0000","last_undegraded":"2024-03-29T11:11:43.687776+0000","last_fullsized":"2024-03-29T11:11:43.687776+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:50:23.857081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687441+0000","last_change":"2024-03-29T11:11:42.709300+0000","last_active":"2024-03-29T11:11:43.687441+0000","last_peered":"2024-03-29T11:11:43.687441+0000","last_clean":"2024-03-29T11:11:43.687441+0000","last_became_active":"2024-03-29T11:11:42.708908+0000","last_became_peered":"2024-03-29T11:11:42.708908+0000","last_unstale":"2024-03-29T11:11:43.687441+0000","last_undegraded":"2024-03-29T11:11:43.687441+0000","last_fullsized":"2024-03-29T11:11:43.687441+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:39.932543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687543+0000","last_change":"2024-03-29T11:11:42.163983+0000","last_active":"2024-03-29T11:11:43.687543+0000","last_peered":"2024-03-29T11:11:43.687543+0000","last_clean":"2024-03-29T11:11:43.687543+0000","last_became_active":"2024-03-29T11:11:42.163515+0000","last_became_peered":"2024-03-29T11:11:42.163515+0000","last_unstale":"2024-03-29T11:11:43.687543+0000","last_undegraded":"2024-03-29T11:11:43.687543+0000","last_fullsized":"2024-03-29T11:11:43.687543+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:09:26.825675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.731968+0000","last_change":"2024-03-29T11:11:42.696305+0000","last_active":"2024-03-29T11:11:43.731968+0000","last_peered":"2024-03-29T11:11:43.731968+0000","last_clean":"2024-03-29T11:11:43.731968+0000","last_became_active":"2024-03-29T11:11:42.695997+0000","last_became_peered":"2024-03-29T11:11:42.695997+0000","last_unstale":"2024-03-29T11:11:43.731968+0000","last_undegraded":"2024-03-29T11:11:43.731968+0000","last_fullsized":"2024-03-29T11:11:43.731968+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:02:39.374556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687428+0000","last_change":"2024-03-29T11:11:42.169609+0000","last_active":"2024-03-29T11:11:43.687428+0000","last_peered":"2024-03-29T11:11:43.687428+0000","last_clean":"2024-03-29T11:11:43.687428+0000","last_became_active":"2024-03-29T11:11:42.169345+0000","last_became_peered":"2024-03-29T11:11:42.169345+0000","last_unstale":"2024-03-29T11:11:43.687428+0000","last_undegraded":"2024-03-29T11:11:43.687428+0000","last_fullsized":"2024-03-29T11:11:43.687428+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:32:32.661750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898119+0000","last_change":"2024-03-29T11:11:42.701480+0000","last_active":"2024-03-29T11:11:43.898119+0000","last_peered":"2024-03-29T11:11:43.898119+0000","last_clean":"2024-03-29T11:11:43.898119+0000","last_became_active":"2024-03-29T11:11:42.700384+0000","last_became_peered":"2024-03-29T11:11:42.700384+0000","last_unstale":"2024-03-29T11:11:43.898119+0000","last_undegraded":"2024-03-29T11:11:43.898119+0000","last_fullsized":"2024-03-29T11:11:43.898119+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:39:40.885000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897224+0000","last_change":"2024-03-29T11:11:42.159016+0000","last_active":"2024-03-29T11:11:43.897224+0000","last_peered":"2024-03-29T11:11:43.897224+0000","last_clean":"2024-03-29T11:11:43.897224+0000","last_became_active":"2024-03-29T11:11:42.158324+0000","last_became_peered":"2024-03-29T11:11:42.158324+0000","last_unstale":"2024-03-29T11:11:43.897224+0000","last_undegraded":"2024-03-29T11:11:43.897224+0000","last_fullsized":"2024-03-29T11:11:43.897224+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:50.173420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.686866+0000","last_change":"2024-03-29T11:11:42.712160+0000","last_active":"2024-03-29T11:11:43.686866+0000","last_peered":"2024-03-29T11:11:43.686866+0000","last_clean":"2024-03-29T11:11:43.686866+0000","last_became_active":"2024-03-29T11:11:42.711849+0000","last_became_peered":"2024-03-29T11:11:42.711849+0000","last_unstale":"2024-03-29T11:11:43.686866+0000","last_undegraded":"2024-03-29T11:11:43.686866+0000","last_fullsized":"2024-03-29T11:11:43.686866+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:06:09.555918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"176'8","reported_seq":19,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897056+0000","last_change":"2024-03-29T11:11:42.164069+0000","last_active":"2024-03-29T11:11:43.897056+0000","last_peered":"2024-03-29T11:11:43.897056+0000","last_clean":"2024-03-29T11:11:43.897056+0000","last_became_active":"2024-03-29T11:11:42.163452+0000","last_became_peered":"2024-03-29T11:11:42.163452+0000","last_unstale":"2024-03-29T11:11:43.897056+0000","last_undegraded":"2024-03-29T11:11:43.897056+0000","last_fullsized":"2024-03-29T11:11:43.897056+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:41.340475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688055+0000","last_change":"2024-03-29T11:11:42.699141+0000","last_active":"2024-03-29T11:11:43.688055+0000","last_peered":"2024-03-29T11:11:43.688055+0000","last_clean":"2024-03-29T11:11:43.688055+0000","last_became_active":"2024-03-29T11:11:42.696934+0000","last_became_peered":"2024-03-29T11:11:42.696934+0000","last_unstale":"2024-03-29T11:11:43.688055+0000","last_undegraded":"2024-03-29T11:11:43.688055+0000","last_fullsized":"2024-03-29T11:11:43.688055+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:20:38.222127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"176'1","reported_seq":12,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898666+0000","last_change":"2024-03-29T11:11:42.170635+0000","last_active":"2024-03-29T11:11:43.898666+0000","last_peered":"2024-03-29T11:11:43.898666+0000","last_clean":"2024-03-29T11:11:43.898666+0000","last_became_active":"2024-03-29T11:11:42.170193+0000","last_became_peered":"2024-03-29T11:11:42.170193+0000","last_unstale":"2024-03-29T11:11:43.898666+0000","last_undegraded":"2024-03-29T11:11:43.898666+0000","last_fullsized":"2024-03-29T11:11:43.898666+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:40:42.477104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897965+0000","last_change":"2024-03-29T11:11:42.701402+0000","last_active":"2024-03-29T11:11:43.897965+0000","last_peered":"2024-03-29T11:11:43.897965+0000","last_clean":"2024-03-29T11:11:43.897965+0000","last_became_active":"2024-03-29T11:11:42.699225+0000","last_became_peered":"2024-03-29T11:11:42.699225+0000","last_unstale":"2024-03-29T11:11:43.897965+0000","last_undegraded":"2024-03-29T11:11:43.897965+0000","last_fullsized":"2024-03-29T11:11:43.897965+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:28:18.369522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732321+0000","last_change":"2024-03-29T11:11:42.163402+0000","last_active":"2024-03-29T11:11:43.732321+0000","last_peered":"2024-03-29T11:11:43.732321+0000","last_clean":"2024-03-29T11:11:43.732321+0000","last_became_active":"2024-03-29T11:11:42.162543+0000","last_became_peered":"2024-03-29T11:11:42.162543+0000","last_unstale":"2024-03-29T11:11:43.732321+0000","last_undegraded":"2024-03-29T11:11:43.732321+0000","last_fullsized":"2024-03-29T11:11:43.732321+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:35:35.410629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687640+0000","last_change":"2024-03-29T11:11:42.709781+0000","last_active":"2024-03-29T11:11:43.687640+0000","last_peered":"2024-03-29T11:11:43.687640+0000","last_clean":"2024-03-29T11:11:43.687640+0000","last_became_active":"2024-03-29T11:11:42.708903+0000","last_became_peered":"2024-03-29T11:11:42.708903+0000","last_unstale":"2024-03-29T11:11:43.687640+0000","last_undegraded":"2024-03-29T11:11:43.687640+0000","last_fullsized":"2024-03-29T11:11:43.687640+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:15.795503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"176'1","reported_seq":12,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897835+0000","last_change":"2024-03-29T11:11:42.167931+0000","last_active":"2024-03-29T11:11:43.897835+0000","last_peered":"2024-03-29T11:11:43.897835+0000","last_clean":"2024-03-29T11:11:43.897835+0000","last_became_active":"2024-03-29T11:11:42.167649+0000","last_became_peered":"2024-03-29T11:11:42.167649+0000","last_unstale":"2024-03-29T11:11:43.897835+0000","last_undegraded":"2024-03-29T11:11:43.897835+0000","last_fullsized":"2024-03-29T11:11:43.897835+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:23.167351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897691+0000","last_change":"2024-03-29T11:11:42.692721+0000","last_active":"2024-03-29T11:11:43.897691+0000","last_peered":"2024-03-29T11:11:43.897691+0000","last_clean":"2024-03-29T11:11:43.897691+0000","last_became_active":"2024-03-29T11:11:42.692451+0000","last_became_peered":"2024-03-29T11:11:42.692451+0000","last_unstale":"2024-03-29T11:11:43.897691+0000","last_undegraded":"2024-03-29T11:11:43.897691+0000","last_fullsized":"2024-03-29T11:11:43.897691+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:46.888360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732149+0000","last_change":"2024-03-29T11:11:42.161771+0000","last_active":"2024-03-29T11:11:43.732149+0000","last_peered":"2024-03-29T11:11:43.732149+0000","last_clean":"2024-03-29T11:11:43.732149+0000","last_became_active":"2024-03-29T11:11:42.161277+0000","last_became_peered":"2024-03-29T11:11:42.161277+0000","last_unstale":"2024-03-29T11:11:43.732149+0000","last_undegraded":"2024-03-29T11:11:43.732149+0000","last_fullsized":"2024-03-29T11:11:43.732149+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:46:22.032350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898190+0000","last_change":"2024-03-29T11:11:42.696984+0000","last_active":"2024-03-29T11:11:43.898190+0000","last_peered":"2024-03-29T11:11:43.898190+0000","last_clean":"2024-03-29T11:11:43.898190+0000","last_became_active":"2024-03-29T11:11:42.696683+0000","last_became_peered":"2024-03-29T11:11:42.696683+0000","last_unstale":"2024-03-29T11:11:43.898190+0000","last_undegraded":"2024-03-29T11:11:43.898190+0000","last_fullsized":"2024-03-29T11:11:43.898190+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:25.203576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687839+0000","last_change":"2024-03-29T11:11:42.169084+0000","last_active":"2024-03-29T11:11:43.687839+0000","last_peered":"2024-03-29T11:11:43.687839+0000","last_clean":"2024-03-29T11:11:43.687839+0000","last_became_active":"2024-03-29T11:11:42.168782+0000","last_became_peered":"2024-03-29T11:11:42.168782+0000","last_unstale":"2024-03-29T11:11:43.687839+0000","last_undegraded":"2024-03-29T11:11:43.687839+0000","last_fullsized":"2024-03-29T11:11:43.687839+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:51.370586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687637+0000","last_change":"2024-03-29T11:11:42.708004+0000","last_active":"2024-03-29T11:11:43.687637+0000","last_peered":"2024-03-29T11:11:43.687637+0000","last_clean":"2024-03-29T11:11:43.687637+0000","last_became_active":"2024-03-29T11:11:42.705823+0000","last_became_peered":"2024-03-29T11:11:42.705823+0000","last_unstale":"2024-03-29T11:11:43.687637+0000","last_undegraded":"2024-03-29T11:11:43.687637+0000","last_fullsized":"2024-03-29T11:11:43.687637+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:48:37.344365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687568+0000","last_change":"2024-03-29T11:11:42.172832+0000","last_active":"2024-03-29T11:11:43.687568+0000","last_peered":"2024-03-29T11:11:43.687568+0000","last_clean":"2024-03-29T11:11:43.687568+0000","last_became_active":"2024-03-29T11:11:42.171272+0000","last_became_peered":"2024-03-29T11:11:42.171272+0000","last_unstale":"2024-03-29T11:11:43.687568+0000","last_undegraded":"2024-03-29T11:11:43.687568+0000","last_fullsized":"2024-03-29T11:11:43.687568+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:08.164844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897829+0000","last_change":"2024-03-29T11:11:42.697592+0000","last_active":"2024-03-29T11:11:43.897829+0000","last_peered":"2024-03-29T11:11:43.897829+0000","last_clean":"2024-03-29T11:11:43.897829+0000","last_became_active":"2024-03-29T11:11:42.695321+0000","last_became_peered":"2024-03-29T11:11:42.695321+0000","last_unstale":"2024-03-29T11:11:43.897829+0000","last_undegraded":"2024-03-29T11:11:43.897829+0000","last_fullsized":"2024-03-29T11:11:43.897829+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:06:09.555918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687001+0000","last_change":"2024-03-29T11:11:42.173035+0000","last_active":"2024-03-29T11:11:43.687001+0000","last_peered":"2024-03-29T11:11:43.687001+0000","last_clean":"2024-03-29T11:11:43.687001+0000","last_became_active":"2024-03-29T11:11:42.172433+0000","last_became_peered":"2024-03-29T11:11:42.172433+0000","last_unstale":"2024-03-29T11:11:43.687001+0000","last_undegraded":"2024-03-29T11:11:43.687001+0000","last_fullsized":"2024-03-29T11:11:43.687001+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:06.912154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687858+0000","last_change":"2024-03-29T11:11:42.708359+0000","last_active":"2024-03-29T11:11:43.687858+0000","last_peered":"2024-03-29T11:11:43.687858+0000","last_clean":"2024-03-29T11:11:43.687858+0000","last_became_active":"2024-03-29T11:11:42.708091+0000","last_became_peered":"2024-03-29T11:11:42.708091+0000","last_unstale":"2024-03-29T11:11:43.687858+0000","last_undegraded":"2024-03-29T11:11:43.687858+0000","last_fullsized":"2024-03-29T11:11:43.687858+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:09:17.030570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.686744+0000","last_change":"2024-03-29T11:11:42.161481+0000","last_active":"2024-03-29T11:11:43.686744+0000","last_peered":"2024-03-29T11:11:43.686744+0000","last_clean":"2024-03-29T11:11:43.686744+0000","last_became_active":"2024-03-29T11:11:42.161180+0000","last_became_peered":"2024-03-29T11:11:42.161180+0000","last_unstale":"2024-03-29T11:11:43.686744+0000","last_undegraded":"2024-03-29T11:11:43.686744+0000","last_fullsized":"2024-03-29T11:11:43.686744+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:23:12.857488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898837+0000","last_change":"2024-03-29T11:11:42.701673+0000","last_active":"2024-03-29T11:11:43.898837+0000","last_peered":"2024-03-29T11:11:43.898837+0000","last_clean":"2024-03-29T11:11:43.898837+0000","last_became_active":"2024-03-29T11:11:42.701434+0000","last_became_peered":"2024-03-29T11:11:42.701434+0000","last_unstale":"2024-03-29T11:11:43.898837+0000","last_undegraded":"2024-03-29T11:11:43.898837+0000","last_fullsized":"2024-03-29T11:11:43.898837+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:27:42.350189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"176'2","reported_seq":13,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688129+0000","last_change":"2024-03-29T11:11:42.174379+0000","last_active":"2024-03-29T11:11:43.688129+0000","last_peered":"2024-03-29T11:11:43.688129+0000","last_clean":"2024-03-29T11:11:43.688129+0000","last_became_active":"2024-03-29T11:11:42.174011+0000","last_became_peered":"2024-03-29T11:11:42.174011+0000","last_unstale":"2024-03-29T11:11:43.688129+0000","last_undegraded":"2024-03-29T11:11:43.688129+0000","last_fullsized":"2024-03-29T11:11:43.688129+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:47:52.801700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688098+0000","last_change":"2024-03-29T11:11:42.705933+0000","last_active":"2024-03-29T11:11:43.688098+0000","last_peered":"2024-03-29T11:11:43.688098+0000","last_clean":"2024-03-29T11:11:43.688098+0000","last_became_active":"2024-03-29T11:11:42.705469+0000","last_became_peered":"2024-03-29T11:11:42.705469+0000","last_unstale":"2024-03-29T11:11:43.688098+0000","last_undegraded":"2024-03-29T11:11:43.688098+0000","last_fullsized":"2024-03-29T11:11:43.688098+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:36:23.384386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897878+0000","last_change":"2024-03-29T11:11:42.168786+0000","last_active":"2024-03-29T11:11:43.897878+0000","last_peered":"2024-03-29T11:11:43.897878+0000","last_clean":"2024-03-29T11:11:43.897878+0000","last_became_active":"2024-03-29T11:11:42.168456+0000","last_became_peered":"2024-03-29T11:11:42.168456+0000","last_unstale":"2024-03-29T11:11:43.897878+0000","last_undegraded":"2024-03-29T11:11:43.897878+0000","last_fullsized":"2024-03-29T11:11:43.897878+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:14:38.572924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732113+0000","last_change":"2024-03-29T11:11:42.706818+0000","last_active":"2024-03-29T11:11:43.732113+0000","last_peered":"2024-03-29T11:11:43.732113+0000","last_clean":"2024-03-29T11:11:43.732113+0000","last_became_active":"2024-03-29T11:11:42.706565+0000","last_became_peered":"2024-03-29T11:11:42.706565+0000","last_unstale":"2024-03-29T11:11:43.732113+0000","last_undegraded":"2024-03-29T11:11:43.732113+0000","last_fullsized":"2024-03-29T11:11:43.732113+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:31:12.718930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897785+0000","last_change":"2024-03-29T11:11:42.165604+0000","last_active":"2024-03-29T11:11:43.897785+0000","last_peered":"2024-03-29T11:11:43.897785+0000","last_clean":"2024-03-29T11:11:43.897785+0000","last_became_active":"2024-03-29T11:11:42.165331+0000","last_became_peered":"2024-03-29T11:11:42.165331+0000","last_unstale":"2024-03-29T11:11:43.897785+0000","last_undegraded":"2024-03-29T11:11:43.897785+0000","last_fullsized":"2024-03-29T11:11:43.897785+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:38:12.028601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897697+0000","last_change":"2024-03-29T11:11:42.705271+0000","last_active":"2024-03-29T11:11:43.897697+0000","last_peered":"2024-03-29T11:11:43.897697+0000","last_clean":"2024-03-29T11:11:43.897697+0000","last_became_active":"2024-03-29T11:11:42.704950+0000","last_became_peered":"2024-03-29T11:11:42.704950+0000","last_unstale":"2024-03-29T11:11:43.897697+0000","last_undegraded":"2024-03-29T11:11:43.897697+0000","last_fullsized":"2024-03-29T11:11:43.897697+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:35.841005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897555+0000","last_change":"2024-03-29T11:11:42.165161+0000","last_active":"2024-03-29T11:11:43.897555+0000","last_peered":"2024-03-29T11:11:43.897555+0000","last_clean":"2024-03-29T11:11:43.897555+0000","last_became_active":"2024-03-29T11:11:42.164914+0000","last_became_peered":"2024-03-29T11:11:42.164914+0000","last_unstale":"2024-03-29T11:11:43.897555+0000","last_undegraded":"2024-03-29T11:11:43.897555+0000","last_fullsized":"2024-03-29T11:11:43.897555+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:01:26.637229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.897317+0000","last_change":"2024-03-29T11:11:42.708069+0000","last_active":"2024-03-29T11:11:43.897317+0000","last_peered":"2024-03-29T11:11:43.897317+0000","last_clean":"2024-03-29T11:11:43.897317+0000","last_became_active":"2024-03-29T11:11:42.707545+0000","last_became_peered":"2024-03-29T11:11:42.707545+0000","last_unstale":"2024-03-29T11:11:43.897317+0000","last_undegraded":"2024-03-29T11:11:43.897317+0000","last_fullsized":"2024-03-29T11:11:43.897317+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:56.542489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687809+0000","last_change":"2024-03-29T11:11:42.166021+0000","last_active":"2024-03-29T11:11:43.687809+0000","last_peered":"2024-03-29T11:11:43.687809+0000","last_clean":"2024-03-29T11:11:43.687809+0000","last_became_active":"2024-03-29T11:11:42.164640+0000","last_became_peered":"2024-03-29T11:11:42.164640+0000","last_unstale":"2024-03-29T11:11:43.687809+0000","last_undegraded":"2024-03-29T11:11:43.687809+0000","last_fullsized":"2024-03-29T11:11:43.687809+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:22:19.536481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898559+0000","last_change":"2024-03-29T11:11:42.691925+0000","last_active":"2024-03-29T11:11:43.898559+0000","last_peered":"2024-03-29T11:11:43.898559+0000","last_clean":"2024-03-29T11:11:43.898559+0000","last_became_active":"2024-03-29T11:11:42.690964+0000","last_became_peered":"2024-03-29T11:11:42.690964+0000","last_unstale":"2024-03-29T11:11:43.898559+0000","last_undegraded":"2024-03-29T11:11:43.898559+0000","last_fullsized":"2024-03-29T11:11:43.898559+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:24.177064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687479+0000","last_change":"2024-03-29T11:11:42.172364+0000","last_active":"2024-03-29T11:11:43.687479+0000","last_peered":"2024-03-29T11:11:43.687479+0000","last_clean":"2024-03-29T11:11:43.687479+0000","last_became_active":"2024-03-29T11:11:42.172088+0000","last_became_peered":"2024-03-29T11:11:42.172088+0000","last_unstale":"2024-03-29T11:11:43.687479+0000","last_undegraded":"2024-03-29T11:11:43.687479+0000","last_fullsized":"2024-03-29T11:11:43.687479+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:41:09.655073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687099+0000","last_change":"2024-03-29T11:11:42.709893+0000","last_active":"2024-03-29T11:11:43.687099+0000","last_peered":"2024-03-29T11:11:43.687099+0000","last_clean":"2024-03-29T11:11:43.687099+0000","last_became_active":"2024-03-29T11:11:42.708475+0000","last_became_peered":"2024-03-29T11:11:42.708475+0000","last_unstale":"2024-03-29T11:11:43.687099+0000","last_undegraded":"2024-03-29T11:11:43.687099+0000","last_fullsized":"2024-03-29T11:11:43.687099+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:06:03.923489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.686957+0000","last_change":"2024-03-29T11:11:42.166342+0000","last_active":"2024-03-29T11:11:43.686957+0000","last_peered":"2024-03-29T11:11:43.686957+0000","last_clean":"2024-03-29T11:11:43.686957+0000","last_became_active":"2024-03-29T11:11:42.165200+0000","last_became_peered":"2024-03-29T11:11:42.165200+0000","last_unstale":"2024-03-29T11:11:43.686957+0000","last_undegraded":"2024-03-29T11:11:43.686957+0000","last_fullsized":"2024-03-29T11:11:43.686957+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:40.994946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.898328+0000","last_change":"2024-03-29T11:11:42.690865+0000","last_active":"2024-03-29T11:11:43.898328+0000","last_peered":"2024-03-29T11:11:43.898328+0000","last_clean":"2024-03-29T11:11:43.898328+0000","last_became_active":"2024-03-29T11:11:42.690576+0000","last_became_peered":"2024-03-29T11:11:42.690576+0000","last_unstale":"2024-03-29T11:11:43.898328+0000","last_undegraded":"2024-03-29T11:11:43.898328+0000","last_fullsized":"2024-03-29T11:11:43.898328+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:36:27.374779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.732050+0000","last_change":"2024-03-29T11:11:42.166419+0000","last_active":"2024-03-29T11:11:43.732050+0000","last_peered":"2024-03-29T11:11:43.732050+0000","last_clean":"2024-03-29T11:11:43.732050+0000","last_became_active":"2024-03-29T11:11:42.166120+0000","last_became_peered":"2024-03-29T11:11:42.166120+0000","last_unstale":"2024-03-29T11:11:43.732050+0000","last_undegraded":"2024-03-29T11:11:43.732050+0000","last_fullsized":"2024-03-29T11:11:43.732050+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:48:36.334651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":10,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.687670+0000","last_change":"2024-03-29T11:11:42.708475+0000","last_active":"2024-03-29T11:11:43.687670+0000","last_peered":"2024-03-29T11:11:43.687670+0000","last_clean":"2024-03-29T11:11:43.687670+0000","last_became_active":"2024-03-29T11:11:42.707818+0000","last_became_peered":"2024-03-29T11:11:42.707818+0000","last_unstale":"2024-03-29T11:11:43.687670+0000","last_undegraded":"2024-03-29T11:11:43.687670+0000","last_fullsized":"2024-03-29T11:11:43.687670+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:42.137689+0000","last_clean_scrub_stamp":"2024-03-29T11:11:42.137689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:57:28.272002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"176'3","reported_seq":14,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-03-29T11:11:43.688034+0000","last_change":"2024-03-29T11:11:42.168852+0000","last_active":"2024-03-29T11:11:43.688034+0000","last_peered":"2024-03-29T11:11:43.688034+0000","last_clean":"2024-03-29T11:11:43.688034+0000","last_became_active":"2024-03-29T11:11:42.168580+0000","last_became_peered":"2024-03-29T11:11:42.168580+0000","last_unstale":"2024-03-29T11:11:43.688034+0000","last_undegraded":"2024-03-29T11:11:43.688034+0000","last_fullsized":"2024-03-29T11:11:43.688034+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:11:41.127975+0000","last_clean_scrub_stamp":"2024-03-29T11:11:41.127975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:38:00.517768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"166'289","reported_seq":945,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-03-29T11:11:42.692990+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:11:42.692990+0000","last_peered":"2024-03-29T11:11:42.692990+0000","last_clean":"2024-03-29T11:11:42.692990+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:11:42.692990+0000","last_undegraded":"2024-03-29T11:11:42.692990+0000","last_fullsized":"2024-03-29T11:11:42.692990+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":471,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":471,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":289,"ondisk_log_size":289,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575764,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47808,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324032,"statfs":{"total":96636764160,"available":96587808768,"internally_reserved":0,"allocated":2797568,"data_stored":1651380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":46135756},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.439,"5min":0.41899999999999998,"15min":0.40000000000000002},"max":{"1min":0.90100000000000002,"5min":1.0089999999999999,"15min":1.014},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.749,"15min":0.746},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.39600000000000002},"max":{"1min":0.97099999999999997,"5min":0.98599999999999999,"15min":1.046},"last":0.629}]},{"osd":1,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.56999999999999995,"5min":0.498,"15min":0.46400000000000002},"max":{"1min":0.85899999999999999,"5min":0.92900000000000005,"15min":1.008},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76200000000000001,"15min":0.77200000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.48499999999999999},"max":{"1min":0.93600000000000005,"5min":1.0009999999999999,"15min":1.081},"last":0.46700000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.75,"15min":0.75800000000000001},"min":{"1min":0.56299999999999994,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.98599999999999999,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.59299999999999997,"5min":0.54600000000000004,"15min":0.51100000000000001},"max":{"1min":0.98099999999999998,"5min":0.995,"15min":1.2310000000000001},"last":0.67300000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.76300000000000001,"15min":0.76500000000000001},"min":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.42999999999999999},"max":{"1min":0.91000000000000003,"5min":0.95399999999999996,"15min":1.091},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.78700000000000003,"15min":0.78800000000000003},"min":{"1min":0.55800000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.96099999999999997,"5min":1.028,"15min":1.0509999999999999},"last":0.71799999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.63900000000000001,"15min":0.629},"min":{"1min":0.44800000000000001,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.85199999999999998,"5min":0.875,"15min":1.1499999999999999},"last":0.498},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.42999999999999999},"max":{"1min":0.85899999999999999,"5min":0.92500000000000004,"15min":1.208},"last":0.755}]},{"osd":5,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.66300000000000003},"min":{"1min":0.45200000000000001,"5min":0.44700000000000001,"15min":0.39100000000000001},"max":{"1min":0.92700000000000005,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.70999999999999996,"15min":0.70499999999999996},"min":{"1min":0.56999999999999995,"5min":0.48699999999999999,"15min":0.41799999999999998},"max":{"1min":0.88600000000000001,"5min":1.0580000000000001,"15min":1.107},"last":0.80000000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.75600000000000001,"15min":0.75},"min":{"1min":0.5,"5min":0.47999999999999998,"15min":0.47499999999999998},"max":{"1min":0.94099999999999995,"5min":1.008,"15min":1.101},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73699999999999999,"15min":0.72799999999999998},"min":{"1min":0.51700000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.1819999999999999},"last":0.78700000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575765,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41484,"kb_used_data":2168,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330356,"statfs":{"total":96636764160,"available":96594284544,"internally_reserved":0,"allocated":2220032,"data_stored":1066850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.73299999999999998,"15min":0.73599999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.38900000000000001},"max":{"1min":0.90000000000000002,"5min":1.002,"15min":1.002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.751,"15min":0.751},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.36799999999999999},"max":{"1min":0.86799999999999999,"5min":1.012,"15min":1.135},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.74199999999999999,"15min":0.73899999999999999},"min":{"1min":0.60299999999999998,"5min":0.47999999999999998,"15min":0.40799999999999997},"max":{"1min":0.93799999999999994,"5min":0.94599999999999995,"15min":1.01},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77700000000000002,"15min":0.77200000000000002},"min":{"1min":0.53800000000000003,"5min":0.496,"15min":0.439},"max":{"1min":0.98999999999999999,"5min":1.04,"15min":1.04},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.75800000000000001,"15min":0.75700000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48699999999999999},"max":{"1min":0.92400000000000004,"5min":0.99299999999999999,"15min":1.0189999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77800000000000002,"15min":0.77400000000000002},"min":{"1min":0.624,"5min":0.50700000000000001,"15min":0.49399999999999999},"max":{"1min":0.95699999999999996,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.73699999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.77900000000000003,"15min":0.77200000000000002},"min":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.52200000000000002},"max":{"1min":0.93899999999999995,"5min":0.96099999999999997,"15min":1.024},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80200000000000005,"15min":0.79600000000000004},"min":{"1min":0.63500000000000001,"5min":0.54600000000000004,"15min":0.54000000000000004},"max":{"1min":1.0169999999999999,"5min":1.0229999999999999,"15min":1.03},"last":0.51500000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.621,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42799999999999999,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.75},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.53400000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.80400000000000005,"5min":0.98799999999999999,"15min":1.0469999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.65500000000000003},"min":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.435},"max":{"1min":0.89100000000000001,"5min":1.0049999999999999,"15min":1.125},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.71899999999999997,"15min":0.68600000000000005},"min":{"1min":0.51700000000000002,"5min":0.438,"15min":0.40400000000000003},"max":{"1min":0.94499999999999995,"5min":1.0629999999999999,"15min":1.0980000000000001},"last":0.71099999999999997}]},{"osd":7,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77100000000000002,"15min":0.752},"min":{"1min":0.59399999999999997,"5min":0.56699999999999995,"15min":0.52800000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.077},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.745,"15min":0.72399999999999998},"min":{"1min":0.57599999999999996,"5min":0.53300000000000003,"15min":0.46200000000000002},"max":{"1min":0.875,"5min":1.0660000000000001,"15min":1.0760000000000001},"last":0.623}]}]},{"osd":1,"up_from":13,"seq":55834575763,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55412,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316428,"statfs":{"total":96636764160,"available":96580022272,"internally_reserved":0,"allocated":2199552,"data_stored":1060942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.63200000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.91000000000000003},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":0.79200000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.372,"15min":0.34399999999999997},"max":{"1min":0.70999999999999996,"5min":0.88100000000000001,"15min":0.91800000000000004},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.622,"15min":0.622},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76200000000000001,"5min":0.81599999999999995,"15min":0.97599999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.42199999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.752,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.52500000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81000000000000005,"15min":0.79900000000000004},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.53800000000000003},"max":{"1min":1.139,"5min":1.139,"15min":1.1950000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.65400000000000003,"5min":0.625,"15min":0.57699999999999996},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.68899999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.83799999999999997,"15min":0.83199999999999996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.85899999999999999,"15min":0.84299999999999997},"min":{"1min":0.68799999999999994,"5min":0.60699999999999998,"15min":0.59099999999999997},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.60699999999999998}]},{"osd":6,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.84699999999999998},"min":{"1min":0.63,"5min":0.63,"15min":0.61099999999999999},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.879,"15min":0.86299999999999999},"min":{"1min":0.67200000000000004,"5min":0.63200000000000001,"15min":0.56799999999999995},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.75}]},{"osd":7,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.89300000000000002,"15min":0.88100000000000001},"min":{"1min":0.79200000000000004,"5min":0.627,"15min":0.56000000000000005},"max":{"1min":1.1419999999999999,"5min":1.1459999999999999,"15min":1.1819999999999999},"last":0.63},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.89300000000000002,"15min":0.88},"min":{"1min":0.71699999999999997,"5min":0.57499999999999996,"15min":0.55000000000000004},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.73099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575765,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60096,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311744,"statfs":{"total":96636764160,"available":96575225856,"internally_reserved":0,"allocated":2801664,"data_stored":1651307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46700000000000003,"15min":0.45900000000000002},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58499999999999996,"5min":0.83299999999999996,"15min":0.89800000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34200000000000003},"max":{"1min":0.73999999999999999,"5min":0.79700000000000004,"15min":1.03},"last":0.34599999999999997}]},{"osd":2,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.42599999999999999,"15min":0.36799999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":1.1459999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52100000000000002,"15min":0.51400000000000001},"min":{"1min":0.39600000000000002,"5min":0.37,"15min":0.34899999999999998},"max":{"1min":0.61099999999999999,"5min":0.81599999999999995,"15min":0.88700000000000001},"last":0.68799999999999994}]},{"osd":3,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.56799999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.36699999999999999},"max":{"1min":0.80900000000000005,"5min":0.85199999999999998,"15min":0.85899999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.64100000000000001},"min":{"1min":0.53700000000000003,"5min":0.49399999999999999,"15min":0.433},"max":{"1min":0.84799999999999998,"5min":0.873,"15min":1.226},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.83099999999999996,"15min":0.82599999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.48899999999999999},"max":{"1min":1.1419999999999999,"5min":1.304,"15min":1.304},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86299999999999999,"15min":0.86499999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.50700000000000001},"max":{"1min":1.107,"5min":1.1100000000000001,"15min":1.2809999999999999},"last":0.92500000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.81200000000000006,"15min":0.79300000000000004},"min":{"1min":0.66700000000000004,"5min":0.53400000000000003,"15min":0.48399999999999999},"max":{"1min":1.125,"5min":1.127,"15min":1.129},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85799999999999998,"15min":0.84099999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.54000000000000004},"max":{"1min":1.1279999999999999,"5min":1.27,"15min":1.27},"last":0.61799999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.86099999999999999,"15min":0.86199999999999999},"min":{"1min":0.66000000000000003,"5min":0.628,"15min":0.628},"max":{"1min":1.115,"5min":1.1679999999999999,"15min":1.329},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.90300000000000002,"15min":0.89100000000000001},"min":{"1min":0.69399999999999995,"5min":0.61699999999999999,"15min":0.58099999999999996},"max":{"1min":1.2150000000000001,"5min":1.337,"15min":1.337},"last":0.95399999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:11:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.84899999999999998,"15min":0.83199999999999996},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.53400000000000003},"max":{"1min":1.0960000000000001,"5min":1.161,"15min":1.181},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89000000000000001,"15min":0.88600000000000001},"min":{"1min":0.67500000000000004,"5min":0.65100000000000002,"15min":0.56499999999999995},"max":{"1min":1.1839999999999999,"5min":1.1890000000000001,"15min":1.298},"last":1.0389999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47236,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324604,"statfs":{"total":96636764160,"available":96588394496,"internally_reserved":0,"allocated":2211840,"data_stored":1061994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.35899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.621,"5min":0.72299999999999998,"15min":0.77700000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.42299999999999999,"5min":0.36499999999999999,"15min":0.34200000000000003},"max":{"1min":0.78500000000000003,"5min":0.80000000000000004,"15min":0.878},"last":0.60699999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.38600000000000001,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":0.64700000000000002,"5min":0.68799999999999994,"15min":0.81999999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.53300000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.72499999999999998,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.496}]},{"osd":3,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61799999999999999,"15min":0.61399999999999999},"min":{"1min":0.503,"5min":0.436,"15min":0.41499999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.44900000000000001,"5min":0.39200000000000002,"15min":0.374},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.90800000000000003},"last":0.39600000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.81000000000000005,"15min":0.81799999999999995},"min":{"1min":0.66700000000000004,"5min":0.56699999999999995,"15min":0.54800000000000004},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.135},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83399999999999996,"15min":0.82299999999999995},"min":{"1min":0.68400000000000005,"5min":0.60699999999999998,"15min":0.56699999999999995},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.3680000000000001},"last":0.77100000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.84199999999999997,"15min":0.83099999999999996},"min":{"1min":0.65500000000000003,"5min":0.54300000000000004,"15min":0.5},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2709999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.61699999999999999,"5min":0.60699999999999998,"15min":0.56299999999999994},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.73499999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.85799999999999998,"15min":0.85299999999999998},"min":{"1min":0.63500000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.304,"5min":1.321,"15min":1.343},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.86299999999999999,"15min":0.85999999999999999},"min":{"1min":0.69899999999999995,"5min":0.59499999999999997,"15min":0.56899999999999995},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.622}]},{"osd":7,"last update":"Fri Mar 29 11:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.92100000000000004,"15min":0.90600000000000003},"min":{"1min":0.72899999999999998,"5min":0.67300000000000004,"15min":0.58199999999999996},"max":{"1min":1.252,"5min":1.2729999999999999,"15min":1.403},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.93000000000000005,"15min":0.91600000000000004},"min":{"1min":0.72499999999999998,"5min":0.68100000000000005,"15min":0.58699999999999997},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.86899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575764,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59524,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312316,"statfs":{"total":96636764160,"available":96575811584,"internally_reserved":0,"allocated":2211840,"data_stored":1066794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.48099999999999998,"15min":0.47699999999999998},"min":{"1min":0.33200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71299999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.45100000000000001,"5min":0.35999999999999999,"15min":0.31},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.499}]},{"osd":1,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51300000000000001,"15min":0.505},"min":{"1min":0.377,"5min":0.377,"15min":0.33100000000000002},"max":{"1min":0.746,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58399999999999996,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.40799999999999997,"15min":0.38900000000000001},"max":{"1min":0.78700000000000003,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.41399999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.38400000000000001},"max":{"1min":0.69499999999999995,"5min":0.875,"15min":0.88},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.67800000000000005,"15min":0.66500000000000004},"min":{"1min":0.58999999999999997,"5min":0.52100000000000002,"15min":0.436},"max":{"1min":0.85199999999999998,"5min":0.91900000000000004,"15min":0.98799999999999999},"last":0.53600000000000003}]},{"osd":4,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.624,"5min":0.61899999999999999,"15min":0.46000000000000002},"max":{"1min":0.98399999999999999,"5min":1.2629999999999999,"15min":1.5469999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.81100000000000005,"15min":0.81200000000000006},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.48199999999999998},"max":{"1min":0.96799999999999997,"5min":1.1519999999999999,"15min":1.5309999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.82599999999999996,"15min":0.82699999999999996},"min":{"1min":0.64900000000000002,"5min":0.55100000000000005,"15min":0.52400000000000002},"max":{"1min":1.02,"5min":1.113,"15min":1.4199999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.84099999999999997,"15min":0.84299999999999997},"min":{"1min":0.68999999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.006,"5min":1.3140000000000001,"15min":1.3740000000000001},"last":0.80900000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.85599999999999998,"15min":0.85899999999999999},"min":{"1min":0.77500000000000002,"5min":0.60999999999999999,"15min":0.56399999999999995},"max":{"1min":1.1100000000000001,"5min":1.1339999999999999,"15min":1.248},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.86399999999999999,"15min":0.85799999999999998},"min":{"1min":0.67400000000000004,"5min":0.65100000000000002,"15min":0.59399999999999997},"max":{"1min":1.0569999999999999,"5min":1.212,"15min":1.2849999999999999},"last":0.73399999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.874,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.095,"5min":1.1910000000000001,"15min":1.6020000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.90000000000000002,"15min":0.89500000000000002},"min":{"1min":0.75,"5min":0.68000000000000005,"15min":0.64500000000000002},"max":{"1min":1.0740000000000001,"5min":1.2350000000000001,"15min":1.3340000000000001},"last":0.91900000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575765,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51312,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320528,"statfs":{"total":96636764160,"available":96584220672,"internally_reserved":0,"allocated":2195456,"data_stored":1060360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74199999999999999,"15min":0.73799999999999999},"min":{"1min":0.55200000000000005,"5min":0.52000000000000002,"15min":0.33300000000000002},"max":{"1min":0.96199999999999997,"5min":1.0209999999999999,"15min":1.097},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72299999999999998,"15min":0.72799999999999998},"min":{"1min":0.51500000000000001,"5min":0.44400000000000001,"15min":0.40899999999999997},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.3300000000000001},"last":0.72899999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.76300000000000001,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.627},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74099999999999999,"15min":0.74399999999999999},"min":{"1min":0.51300000000000001,"5min":0.502,"15min":0.42599999999999999},"max":{"1min":0.86699999999999999,"5min":0.97099999999999997,"15min":1.0269999999999999},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.82599999999999996},"min":{"1min":0.60299999999999998,"5min":0.54500000000000004,"15min":0.437},"max":{"1min":1.034,"5min":1.034,"15min":1.3080000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.81100000000000005,"15min":0.81299999999999994},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.46800000000000003},"max":{"1min":0.95499999999999996,"5min":1.0349999999999999,"15min":1.1140000000000001},"last":0.71499999999999997}]},{"osd":3,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76700000000000002,"15min":0.76100000000000001},"min":{"1min":0.58699999999999997,"5min":0.58399999999999996,"15min":0.53700000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":1.177},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.75600000000000001},"min":{"1min":0.57899999999999996,"5min":0.51800000000000002,"15min":0.48699999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.51600000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60999999999999999,"15min":0.60299999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.88100000000000001,"5min":0.93100000000000005,"15min":1.054},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.44700000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.1779999999999999},"last":0.58699999999999997}]},{"osd":6,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.499,"5min":0.47499999999999998,"15min":0.443},"max":{"1min":1.038,"5min":1.038,"15min":1.0389999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68700000000000006,"15min":0.67600000000000005},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.437},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":1.254},"last":0.65200000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.69499999999999995,"15min":0.69099999999999995},"min":{"1min":0.50600000000000001,"5min":0.47899999999999998,"15min":0.44900000000000001},"max":{"1min":1.004,"5min":1.004,"15min":1.4379999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.78900000000000003,"15min":0.79000000000000004},"min":{"1min":0.67300000000000004,"5min":0.58899999999999997,"15min":0.52400000000000002},"max":{"1min":1.032,"5min":1.0580000000000001,"15min":1.482},"last":0.77000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575764,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41464,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330376,"statfs":{"total":96636764160,"available":96594305024,"internally_reserved":0,"allocated":2199552,"data_stored":1060382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.73999999999999999,"15min":0.72699999999999998},"min":{"1min":0.624,"5min":0.47199999999999998,"15min":0.40699999999999997},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.74099999999999999,"15min":0.748},"min":{"1min":0.59599999999999997,"5min":0.48099999999999998,"15min":0.45000000000000001},"max":{"1min":0.91900000000000004,"5min":0.94799999999999995,"15min":1},"last":0.58299999999999996}]},{"osd":1,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.76200000000000001,"15min":0.76800000000000002},"min":{"1min":0.67500000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.95599999999999996,"5min":1.0189999999999999,"15min":1.0549999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54000000000000004,"5min":0.52500000000000002,"15min":0.51600000000000001},"max":{"1min":0.97799999999999998,"5min":0.997,"15min":1.052},"last":0.66400000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.76700000000000002,"15min":0.76400000000000001},"min":{"1min":0.58899999999999997,"5min":0.499,"15min":0.47199999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.0609999999999999},"last":0.503},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.74199999999999999,"15min":0.746},"min":{"1min":0.53300000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.78500000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75,"15min":0.77100000000000002},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.52300000000000002},"max":{"1min":1,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.78900000000000003,"15min":0.78700000000000003},"min":{"1min":0.65200000000000002,"5min":0.55800000000000005,"15min":0.54600000000000004},"max":{"1min":0.97499999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.61599999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.47499999999999998,"5min":0.38800000000000001,"15min":0.379},"max":{"1min":0.73299999999999998,"5min":0.92700000000000005,"15min":0.93899999999999995},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.46899999999999997,"5min":0.45300000000000001,"15min":0.41199999999999998},"max":{"1min":0.78800000000000003,"5min":0.95199999999999996,"15min":0.98299999999999998},"last":0.76500000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.65000000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.439},"max":{"1min":0.90900000000000003,"5min":0.93999999999999995,"15min":0.95399999999999996},"last":0.751},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65600000000000003,"15min":0.67100000000000004},"min":{"1min":0.50900000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.84799999999999998,"5min":0.86499999999999999,"15min":0.95499999999999996},"last":0.82399999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.50900000000000001,"5min":0.46800000000000003,"15min":0.46500000000000002},"max":{"1min":0.86199999999999999,"5min":0.90500000000000003,"15min":1.01},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.746,"5min":0.71699999999999997,"15min":0.73799999999999999},"min":{"1min":0.56000000000000005,"5min":0.52500000000000002,"15min":0.49399999999999999},"max":{"1min":0.94399999999999995,"5min":0.95399999999999996,"15min":1.0369999999999999},"last":0.84699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:11:48.109 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T11:11:48.109 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T11:11:48.110 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-29T11:11:48.122+0000 7ff1c5000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:11:48.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.338+0000 7efe88400700 1 Processor -- start 2024-03-29T11:11:48.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.338+0000 7efe88400700 1 -- start start 2024-03-29T11:11:48.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.339+0000 7efe88400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe800d2540 0x7efe801687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.340+0000 7efe88400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe80161420 con 0x7efe800d16a0 2024-03-29T11:11:48.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.340+0000 7efe88400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe80161700 con 0x7efe8015d9d0 2024-03-29T11:11:48.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.340+0000 7efe88400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe800d4340 con 0x7efe800d2540 2024-03-29T11:11:48.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.340+0000 7efe85000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7efe800d16a0 0x7efe800d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60220/0 (socket says 172.21.15.12:60220) 2024-03-29T11:11:48.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.340+0000 7efe85000700 1 -- 172.21.15.12:0/1665616955 learned_addr learned my addr 172.21.15.12:0/1665616955 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:48.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.340+0000 7efe86400700 1 --2- 172.21.15.12:0/1665616955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe800d2540 0x7efe801687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.340+0000 7efe7fe00700 1 -- 172.21.15.12:0/1665616955 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466610388 0 0) 0x7efe80161420 con 0x7efe800d16a0 2024-03-29T11:11:48.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.340+0000 7efe7fe00700 1 -- 172.21.15.12:0/1665616955 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe680035b0 con 0x7efe800d16a0 2024-03-29T11:11:48.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.340+0000 7efe7fe00700 1 -- 172.21.15.12:0/1665616955 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330092668 0 0) 0x7efe80161700 con 0x7efe8015d9d0 2024-03-29T11:11:48.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.340+0000 7efe7fe00700 1 -- 172.21.15.12:0/1665616955 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe80161420 con 0x7efe8015d9d0 2024-03-29T11:11:48.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.341+0000 7efe86400700 1 -- 172.21.15.12:0/1665616955 >> v1:172.21.15.159:6789/0 conn(0x7efe8015d9d0 legacy=0x7efe800d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.341+0000 7efe86400700 1 -- 172.21.15.12:0/1665616955 >> v1:172.21.15.12:6789/0 conn(0x7efe800d16a0 legacy=0x7efe800d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.341+0000 7efe86400700 1 -- 172.21.15.12:0/1665616955 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe801619e0 con 0x7efe800d2540 2024-03-29T11:11:48.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.341+0000 7efe86400700 1 --2- 172.21.15.12:0/1665616955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe800d2540 0x7efe801687b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efe78009dc0 tx=0x7efe7800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5df71ffc1ecedb79 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:48.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.341+0000 7efe7fe00700 1 -- 172.21.15.12:0/1665616955 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7800e070 con 0x7efe800d2540 2024-03-29T11:11:48.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.341+0000 7efe7fe00700 1 -- 172.21.15.12:0/1665616955 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7800bb20 con 0x7efe800d2540 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.341+0000 7efe7fe00700 1 -- 172.21.15.12:0/1665616955 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78012430 con 0x7efe800d2540 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.342+0000 7efe88400700 1 -- 172.21.15.12:0/1665616955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe800d2540 msgr2=0x7efe801687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.342+0000 7efe88400700 1 --2- 172.21.15.12:0/1665616955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe800d2540 0x7efe801687b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efe78009dc0 tx=0x7efe7800b330 comp rx=0 tx=0).stop 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.342+0000 7efe88400700 1 -- 172.21.15.12:0/1665616955 shutdown_connections 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.342+0000 7efe88400700 1 --2- 172.21.15.12:0/1665616955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe800d2540 0x7efe801687b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.342+0000 7efe88400700 1 -- 172.21.15.12:0/1665616955 >> 172.21.15.12:0/1665616955 conn(0x7efe8008aff0 msgr2=0x7efe800d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.342+0000 7efe88400700 1 -- 172.21.15.12:0/1665616955 shutdown_connections 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.342+0000 7efe88400700 1 -- 172.21.15.12:0/1665616955 wait complete. 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.344+0000 7efe88400700 1 Processor -- start 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.344+0000 7efe88400700 1 -- start start 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe88400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe800d16a0 0x7efe801f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe88400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe800d2540 0x7efe800ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe88400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe8015d9d0 0x7efe800cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe88400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe800cbc30 con 0x7efe800d2540 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe88400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe800cafb0 con 0x7efe800d16a0 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe88400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe800cb290 con 0x7efe8015d9d0 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe85000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe800d2540 0x7efe800ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe85000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe800d2540 0x7efe800ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58334/0 (socket says 172.21.15.12:58334) 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe85000700 1 -- 172.21.15.12:0/864691222 learned_addr learned my addr 172.21.15.12:0/864691222 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe85a00700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe800d16a0 0x7efe801f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe86400700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe8015d9d0 0x7efe800cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe85000700 1 -- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe8015d9d0 msgr2=0x7efe800cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe85000700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe8015d9d0 0x7efe800cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe85000700 1 -- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe800d16a0 msgr2=0x7efe801f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe85000700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe800d16a0 0x7efe801f2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.345+0000 7efe85000700 1 -- 172.21.15.12:0/864691222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe78009a70 con 0x7efe800d2540 2024-03-29T11:11:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.346+0000 7efe85000700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe800d2540 0x7efe800ca5f0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7efe6c001260 tx=0x7efe6c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.346+0000 7efe7d600700 1 -- 172.21.15.12:0/864691222 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0074e0 con 0x7efe800d2540 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.346+0000 7efe7d600700 1 -- 172.21.15.12:0/864691222 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c007640 con 0x7efe800d2540 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.346+0000 7efe88400700 1 -- 172.21.15.12:0/864691222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe800cb570 con 0x7efe800d2540 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.346+0000 7efe7d600700 1 -- 172.21.15.12:0/864691222 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c005a70 con 0x7efe800d2540 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.346+0000 7efe88400700 1 -- 172.21.15.12:0/864691222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe801f2f40 con 0x7efe800d2540 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.347+0000 7efe88400700 1 -- 172.21.15.12:0/864691222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe800d1ca0 con 0x7efe800d2540 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.348+0000 7efe7d600700 1 -- 172.21.15.12:0/864691222 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efe6c005bd0 con 0x7efe800d2540 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.349+0000 7efe7d600700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe6006adb0 0x7efe6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.349+0000 7efe7d600700 1 -- 172.21.15.12:0/864691222 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7efe6c088a30 con 0x7efe800d2540 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.349+0000 7efe85a00700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe6006adb0 0x7efe6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.350+0000 7efe85a00700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe6006adb0 0x7efe6006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efe740011e0 tx=0x7efe74009040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.353+0000 7efe7d600700 1 -- 172.21.15.12:0/864691222 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efe6c0539b0 con 0x7efe800d2540 2024-03-29T11:11:48.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.649+0000 7efe88400700 1 -- 172.21.15.12:0/864691222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7efe800cbd70 con 0x7efe800d2540 2024-03-29T11:11:48.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.654+0000 7efe7d600700 1 -- 172.21.15.12:0/864691222 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) v1 ==== 190+0+0 (secure 0 0 0) 0x7efe800cbd70 con 0x7efe800d2540 2024-03-29T11:11:48.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 -- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe6006adb0 msgr2=0x7efe6006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe6006adb0 0x7efe6006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efe740011e0 tx=0x7efe74009040 comp rx=0 tx=0).stop 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 -- 172.21.15.12:0/864691222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe800d2540 msgr2=0x7efe800ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe800d2540 0x7efe800ca5f0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7efe6c001260 tx=0x7efe6c00c9e0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 -- 172.21.15.12:0/864691222 shutdown_connections 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe6006adb0 0x7efe6006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe8015d9d0 0x7efe800cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe800d16a0 0x7efe801f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 --2- 172.21.15.12:0/864691222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe800d2540 0x7efe800ca5f0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 -- 172.21.15.12:0/864691222 >> 172.21.15.12:0/864691222 conn(0x7efe8008aff0 msgr2=0x7efe800cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 -- 172.21.15.12:0/864691222 shutdown_connections 2024-03-29T11:11:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.656+0000 7efe88400700 1 -- 172.21.15.12:0/864691222 wait complete. 2024-03-29T11:11:48.636 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T11:11:48.652 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:11:48.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.873+0000 7fe9d6800700 1 Processor -- start 2024-03-29T11:11:48.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.873+0000 7fe9d6800700 1 -- start start 2024-03-29T11:11:48.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.874+0000 7fe9d6800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9d00cf3c0 0x7fe9d00d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.874+0000 7fe9d6800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe9d015db40 0x7fe9d00cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.874+0000 7fe9d6800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d00d1330 con 0x7fe9d015d760 2024-03-29T11:11:48.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.874+0000 7fe9d6800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d00d1fb0 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.874+0000 7fe9d6800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d00d1610 con 0x7fe9d015db40 2024-03-29T11:11:48.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.874+0000 7fe9cea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9d00cf3c0 0x7fe9d00d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.874+0000 7fe9cf400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe9d015d760 0x7fe9d00cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33182/0 (socket says 172.21.15.12:33182) 2024-03-29T11:11:48.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.874+0000 7fe9cf400700 1 -- 172.21.15.12:0/3278563094 learned_addr learned my addr 172.21.15.12:0/3278563094 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:48.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.874+0000 7fe9cfe00700 1 --2- 172.21.15.12:0/3278563094 >> v2:172.21.15.159:3300/0 conn(0x7fe9d015db40 0x7fe9d00cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.875+0000 7fe9cea00700 1 -- 172.21.15.12:0/3278563094 >> v2:172.21.15.159:3300/0 conn(0x7fe9d015db40 msgr2=0x7fe9d00cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:48.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.875+0000 7fe9cea00700 1 --2- 172.21.15.12:0/3278563094 >> v2:172.21.15.159:3300/0 conn(0x7fe9d015db40 0x7fe9d00cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.875+0000 7fe9cea00700 1 -- 172.21.15.12:0/3278563094 >> v1:172.21.15.12:6789/0 conn(0x7fe9d015d760 legacy=0x7fe9d00cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.875+0000 7fe9cea00700 1 -- 172.21.15.12:0/3278563094 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d00d18f0 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.875+0000 7fe9cea00700 1 --2- 172.21.15.12:0/3278563094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 0x7fe9d00d23d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00c6de0 tx=0x7fe9bc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fef8b59cf8fc59d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:48.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.875+0000 7fe9ce000700 1 -- 172.21.15.12:0/3278563094 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc011830 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.875+0000 7fe9ce000700 1 -- 172.21.15.12:0/3278563094 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc011990 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.875+0000 7fe9ce000700 1 -- 172.21.15.12:0/3278563094 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc011d20 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.876+0000 7fe9d6800700 1 -- 172.21.15.12:0/3278563094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 msgr2=0x7fe9d00d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.876+0000 7fe9d6800700 1 --2- 172.21.15.12:0/3278563094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 0x7fe9d00d23d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00c6de0 tx=0x7fe9bc009b20 comp rx=0 tx=0).stop 2024-03-29T11:11:48.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.876+0000 7fe9d6800700 1 -- 172.21.15.12:0/3278563094 shutdown_connections 2024-03-29T11:11:48.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.876+0000 7fe9d6800700 1 --2- 172.21.15.12:0/3278563094 >> v2:172.21.15.159:3300/0 conn(0x7fe9d015db40 0x7fe9d00cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.876+0000 7fe9d6800700 1 --2- 172.21.15.12:0/3278563094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 0x7fe9d00d23d0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.876+0000 7fe9d6800700 1 -- 172.21.15.12:0/3278563094 >> 172.21.15.12:0/3278563094 conn(0x7fe9d008aff0 msgr2=0x7fe9d00d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:48.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.876+0000 7fe9d6800700 1 -- 172.21.15.12:0/3278563094 shutdown_connections 2024-03-29T11:11:48.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.876+0000 7fe9d6800700 1 -- 172.21.15.12:0/3278563094 wait complete. 2024-03-29T11:11:48.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.878+0000 7fe9d6800700 1 Processor -- start 2024-03-29T11:11:48.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.878+0000 7fe9d6800700 1 -- start start 2024-03-29T11:11:48.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.879+0000 7fe9d6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 0x7fe9d014c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.879+0000 7fe9d6800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9d015d760 0x7fe9d00cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9d6800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9d015db40 0x7fe9d0148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9d6800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d00cd030 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9d6800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d00cc3b0 con 0x7fe9d015d760 2024-03-29T11:11:48.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9d6800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d00cc690 con 0x7fe9d015db40 2024-03-29T11:11:48.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 0x7fe9d014c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 0x7fe9d014c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46902/0 (socket says 172.21.15.12:46902) 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cf400700 1 -- 172.21.15.12:0/955593879 learned_addr learned my addr 172.21.15.12:0/955593879 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cea00700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9d015d760 0x7fe9d00cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cfe00700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9d015db40 0x7fe9d0148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cf400700 1 -- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9d015db40 msgr2=0x7fe9d0148bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cf400700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9d015db40 0x7fe9d0148bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cf400700 1 -- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9d015d760 msgr2=0x7fe9d00cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cf400700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9d015d760 0x7fe9d00cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cf400700 1 -- 172.21.15.12:0/955593879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc0094a0 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cfe00700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9d015db40 0x7fe9d0148bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.880+0000 7fe9cf400700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 0x7fe9d014c0e0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000c520 tx=0x7fe9c000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.881+0000 7fe9c7400700 1 -- 172.21.15.12:0/955593879 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0005d70 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.881+0000 7fe9c7400700 1 -- 172.21.15.12:0/955593879 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c00070d0 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.881+0000 7fe9d6800700 1 -- 172.21.15.12:0/955593879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d00cc970 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.881+0000 7fe9c7400700 1 -- 172.21.15.12:0/955593879 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0007440 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.881+0000 7fe9d6800700 1 -- 172.21.15.12:0/955593879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d0149130 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.882+0000 7fe9d6800700 1 -- 172.21.15.12:0/955593879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d0067720 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.883+0000 7fe9c7400700 1 -- 172.21.15.12:0/955593879 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe9c00075a0 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.884+0000 7fe9c7400700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9b006adb0 0x7fe9b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:48.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.884+0000 7fe9c7400700 1 -- 172.21.15.12:0/955593879 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fe9c0088860 con 0x7fe9d00cf3c0 2024-03-29T11:11:48.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.884+0000 7fe9cea00700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9b006adb0 0x7fe9b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:48.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.885+0000 7fe9cea00700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9b006adb0 0x7fe9b006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00c6de0 tx=0x7fe9bc008030 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:48.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:48.888+0000 7fe9c7400700 1 -- 172.21.15.12:0/955593879 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe9c0053a40 con 0x7fe9d00cf3c0 2024-03-29T11:11:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.147+0000 7fe9d6800700 1 -- 172.21.15.12:0/955593879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9d00b9460 con 0x7fe9d00cf3c0 2024-03-29T11:11:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.147+0000 7fe9c7400700 1 -- 172.21.15.12:0/955593879 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fe9c0017090 con 0x7fe9d00cf3c0 2024-03-29T11:11:49.126 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:11:49.126 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/652112625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":652112625},{"type":"v1","addr":"172.21.15.159:6836","nonce":652112625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9166,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2803735172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2803735172},{"type":"v1","addr":"172.21.15.159:6837","nonce":2803735172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9444,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3698805288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3698805288},{"type":"v1","addr":"172.21.15.12:6833","nonce":3698805288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:11:42.673965+0000","modified":"2024-03-29T11:11:46.149885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9450":{"gid":9450,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/2066631988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2066631988},{"type":"v1","addr":"172.21.15.12:6835","nonce":2066631988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-29T11:11:49.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 -- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9b006adb0 msgr2=0x7fe9b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9b006adb0 0x7fe9b006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00c6de0 tx=0x7fe9bc008030 comp rx=0 tx=0).stop 2024-03-29T11:11:49.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 -- 172.21.15.12:0/955593879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 msgr2=0x7fe9d014c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 0x7fe9d014c0e0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000c520 tx=0x7fe9c000c8e0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 -- 172.21.15.12:0/955593879 shutdown_connections 2024-03-29T11:11:49.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe9b006adb0 0x7fe9b006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9d015db40 0x7fe9d0148bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00cf3c0 0x7fe9d014c0e0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 --2- 172.21.15.12:0/955593879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe9d015d760 0x7fe9d00cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 -- 172.21.15.12:0/955593879 >> 172.21.15.12:0/955593879 conn(0x7fe9d008aff0 msgr2=0x7fe9d006c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:49.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 -- 172.21.15.12:0/955593879 shutdown_connections 2024-03-29T11:11:49.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.150+0000 7fe9d6800700 1 -- 172.21.15.12:0/955593879 wait complete. 2024-03-29T11:11:49.129 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 81 2024-03-29T11:11:49.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T11:11:42.673965+0000', 'modified': '2024-03-29T11:11:46.149885+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9450}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9450': {'gid': 9450, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6835/2066631988', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6834', 'nonce': 2066631988}, {'type': 'v1', 'addr': '172.21.15.12:6835', 'nonce': 2066631988}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T11:11:49.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T11:11:49.550 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-29T11:11:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.775+0000 7f6c24c00700 1 Processor -- start 2024-03-29T11:11:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.775+0000 7f6c24c00700 1 -- start start 2024-03-29T11:11:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.776+0000 7f6c24c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6c200cd5c0 0x7f6c2015e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.776+0000 7f6c24c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6c2006f310 0x7f6c2006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.776+0000 7f6c24c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c2015eda0 con 0x7f6c2015cfe0 2024-03-29T11:11:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.776+0000 7f6c24c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c2006fce0 con 0x7f6c2006f310 2024-03-29T11:11:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.776+0000 7f6c24c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c20099dd0 con 0x7f6c200cd5c0 2024-03-29T11:11:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.777+0000 7f6c1e800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6c2015cfe0 0x7f6c2015d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33196/0 (socket says 172.21.15.12:33196) 2024-03-29T11:11:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.777+0000 7f6c1e800700 1 -- 172.21.15.12:0/4183786982 learned_addr learned my addr 172.21.15.12:0/4183786982 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.777+0000 7f6c1d400700 1 --2- 172.21.15.12:0/4183786982 >> v2:172.21.15.12:3300/0 conn(0x7f6c2006f310 0x7f6c2006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.777+0000 7f6c1de00700 1 --2- 172.21.15.12:0/4183786982 >> v2:172.21.15.159:3300/0 conn(0x7f6c200cd5c0 0x7f6c2015e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.777+0000 7f6c1d400700 1 -- 172.21.15.12:0/4183786982 >> v2:172.21.15.159:3300/0 conn(0x7f6c200cd5c0 msgr2=0x7f6c2015e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.777+0000 7f6c1d400700 1 --2- 172.21.15.12:0/4183786982 >> v2:172.21.15.159:3300/0 conn(0x7f6c200cd5c0 0x7f6c2015e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.777+0000 7f6c1d400700 1 -- 172.21.15.12:0/4183786982 >> v1:172.21.15.12:6789/0 conn(0x7f6c2015cfe0 legacy=0x7f6c2015d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.777+0000 7f6c1d400700 1 -- 172.21.15.12:0/4183786982 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2009a0b0 con 0x7f6c2006f310 2024-03-29T11:11:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.777+0000 7f6c1ca00700 1 -- 172.21.15.12:0/4183786982 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320242812 0 0) 0x7f6c18001490 con 0x7f6c2015cfe0 2024-03-29T11:11:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.777+0000 7f6c1d400700 1 --2- 172.21.15.12:0/4183786982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 0x7f6c2006f7a0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009840 tx=0x7f6c0c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3084a135e50dd81 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c1ca00700 1 -- 172.21.15.12:0/4183786982 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c011830 con 0x7f6c2006f310 2024-03-29T11:11:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c1ca00700 1 -- 172.21.15.12:0/4183786982 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c011990 con 0x7f6c2006f310 2024-03-29T11:11:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c1ca00700 1 -- 172.21.15.12:0/4183786982 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c011d20 con 0x7f6c2006f310 2024-03-29T11:11:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c24c00700 1 -- 172.21.15.12:0/4183786982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 msgr2=0x7f6c2006f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c24c00700 1 --2- 172.21.15.12:0/4183786982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 0x7f6c2006f7a0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009840 tx=0x7f6c0c009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c24c00700 1 -- 172.21.15.12:0/4183786982 shutdown_connections 2024-03-29T11:11:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c24c00700 1 --2- 172.21.15.12:0/4183786982 >> v2:172.21.15.159:3300/0 conn(0x7f6c200cd5c0 0x7f6c2015e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c24c00700 1 --2- 172.21.15.12:0/4183786982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 0x7f6c2006f7a0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c24c00700 1 -- 172.21.15.12:0/4183786982 >> 172.21.15.12:0/4183786982 conn(0x7f6c200d1370 msgr2=0x7f6c200d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c24c00700 1 -- 172.21.15.12:0/4183786982 shutdown_connections 2024-03-29T11:11:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.778+0000 7f6c24c00700 1 -- 172.21.15.12:0/4183786982 wait complete. 2024-03-29T11:11:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.780+0000 7f6c24c00700 1 Processor -- start 2024-03-29T11:11:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.780+0000 7f6c24c00700 1 -- start start 2024-03-29T11:11:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.781+0000 7f6c24c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 0x7f6c20155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.781+0000 7f6c24c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c200cd5c0 0x7f6c20154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.781+0000 7f6c24c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c2015cfe0 0x7f6c200af340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.781+0000 7f6c24c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c200afbe0 con 0x7f6c2006f310 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.781+0000 7f6c24c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c20155030 con 0x7f6c2015cfe0 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.781+0000 7f6c24c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c20158930 con 0x7f6c200cd5c0 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.781+0000 7f6c1de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 0x7f6c20155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.781+0000 7f6c1de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 0x7f6c20155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46922/0 (socket says 172.21.15.12:46922) 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.781+0000 7f6c1de00700 1 -- 172.21.15.12:0/2567135996 learned_addr learned my addr 172.21.15.12:0/2567135996 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c1de00700 1 -- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c200cd5c0 msgr2=0x7f6c20154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c1e800700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c2015cfe0 0x7f6c200af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c1de00700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c200cd5c0 0x7f6c20154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c1de00700 1 -- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c2015cfe0 msgr2=0x7f6c200af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c1de00700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c2015cfe0 0x7f6c200af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c1de00700 1 -- 172.21.15.12:0/2567135996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c0094a0 con 0x7f6c2006f310 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c1e800700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c2015cfe0 0x7f6c200af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c1de00700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 0x7f6c20155210 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000e9b0 tx=0x7f6c1000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c16000700 1 -- 172.21.15.12:0/2567135996 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1000c850 con 0x7f6c2006f310 2024-03-29T11:11:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c16000700 1 -- 172.21.15.12:0/2567135996 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1000c9b0 con 0x7f6c2006f310 2024-03-29T11:11:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c16000700 1 -- 172.21.15.12:0/2567135996 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1000cd20 con 0x7f6c2006f310 2024-03-29T11:11:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.782+0000 7f6c24c00700 1 -- 172.21.15.12:0/2567135996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c20158c10 con 0x7f6c2006f310 2024-03-29T11:11:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.783+0000 7f6c24c00700 1 -- 172.21.15.12:0/2567135996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c201591d0 con 0x7f6c2006f310 2024-03-29T11:11:49.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.786+0000 7f6c16000700 1 -- 172.21.15.12:0/2567135996 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6c10013470 con 0x7f6c2006f310 2024-03-29T11:11:49.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.786+0000 7f6c16000700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c0006adb0 0x7f6c0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.786+0000 7f6c16000700 1 -- 172.21.15.12:0/2567135996 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f6c10089690 con 0x7f6c2006f310 2024-03-29T11:11:49.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.787+0000 7f6c1d400700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c0006adb0 0x7f6c0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.788+0000 7f6c1d400700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c0006adb0 0x7f6c0006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6c200b09a0 tx=0x7f6c0c009f90 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.788+0000 7f6c38813b80 1 Processor -- start 2024-03-29T11:11:49.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.788+0000 7f6c38813b80 1 -- start start 2024-03-29T11:11:49.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.789+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f920830 0x56176f920c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.789+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f8f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.789+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f8f1b60 0x56176f91da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.789+0000 7f6c38813b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176f86b650 con 0x56176f8f3be0 2024-03-29T11:11:49.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.789+0000 7f6c38813b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176f84cc30 con 0x56176f920830 2024-03-29T11:11:49.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.789+0000 7f6c38813b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56176f84a430 con 0x56176f8f1b60 2024-03-29T11:11:49.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.789+0000 7f6c1d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f8f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.789+0000 7f6c1d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f8f6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46924/0 (socket says 172.21.15.12:46924) 2024-03-29T11:11:49.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.789+0000 7f6c1d400700 1 -- 172.21.15.12:0/409976716 learned_addr learned my addr 172.21.15.12:0/409976716 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:49.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.789+0000 7f6c1de00700 1 --2- 172.21.15.12:0/409976716 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f920830 0x56176f920c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.790+0000 7f6c1e800700 1 --2- 172.21.15.12:0/409976716 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f8f1b60 0x56176f91da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.790+0000 7f6c1d400700 1 -- 172.21.15.12:0/409976716 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f8f1b60 msgr2=0x56176f91da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:49.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.790+0000 7f6c1d400700 1 --2- 172.21.15.12:0/409976716 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f8f1b60 0x56176f91da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.790+0000 7f6c1d400700 1 -- 172.21.15.12:0/409976716 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f920830 msgr2=0x56176f920c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:49.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.790+0000 7f6c1d400700 1 --2- 172.21.15.12:0/409976716 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f920830 0x56176f920c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.790+0000 7f6c1d400700 1 -- 172.21.15.12:0/409976716 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56176f639790 con 0x56176f8f3be0 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.790+0000 7f6c1d400700 1 --2- 172.21.15.12:0/409976716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f8f6a10 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c000c00 tx=0x7f6c0c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89dbb8c5c912d28a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.790+0000 7f6c0a600700 1 -- 172.21.15.12:0/409976716 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c004340 con 0x56176f8f3be0 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.790+0000 7f6c0a600700 1 -- 172.21.15.12:0/409976716 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c02a070 con 0x56176f8f3be0 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.790+0000 7f6c0a600700 1 -- 172.21.15.12:0/409976716 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c0036a0 con 0x56176f8f3be0 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.791+0000 7f6c38813b80 1 -- 172.21.15.12:0/409976716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 msgr2=0x56176f8f6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.791+0000 7f6c38813b80 1 --2- 172.21.15.12:0/409976716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f8f6a10 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c000c00 tx=0x7f6c0c008030 comp rx=0 tx=0).stop 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.791+0000 7f6c38813b80 1 -- 172.21.15.12:0/409976716 shutdown_connections 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.791+0000 7f6c38813b80 1 --2- 172.21.15.12:0/409976716 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f8f1b60 0x56176f91da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.791+0000 7f6c38813b80 1 --2- 172.21.15.12:0/409976716 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f920830 0x56176f920c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.791+0000 7f6c38813b80 1 --2- 172.21.15.12:0/409976716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f8f6a10 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.791+0000 7f6c38813b80 1 -- 172.21.15.12:0/409976716 >> 172.21.15.12:0/409976716 conn(0x56176f8546a0 msgr2=0x56176f93b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.791+0000 7f6c38813b80 1 -- 172.21.15.12:0/409976716 shutdown_connections 2024-03-29T11:11:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.791+0000 7f6c38813b80 1 -- 172.21.15.12:0/409976716 wait complete. 2024-03-29T11:11:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.792+0000 7f6c38813b80 1 -- start start 2024-03-29T11:11:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.792+0000 7f6c38813b80 1 Processor -- start 2024-03-29T11:11:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.793+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f9176d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.793+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f8f3be0 0x56176f91b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.793+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f920830 0x56176f91b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.793+0000 7f6c38813b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176f845d20 con 0x56176f8f1b60 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.793+0000 7f6c38813b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176f7e1770 con 0x56176f8f3be0 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.793+0000 7f6c38813b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56176f8046f0 con 0x56176f920830 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.793+0000 7f6c38813b80 10 client.0 _ll_register_callbacks cb 0x56176f7eb5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.793+0000 7f6c1de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f9176d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.793+0000 7f6c1de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f9176d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46938/0 (socket says 172.21.15.12:46938) 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.793+0000 7f6c1de00700 1 -- 172.21.15.12:0/111689996 learned_addr learned my addr 172.21.15.12:0/111689996 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c1d400700 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f8f3be0 0x56176f91b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c1e800700 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f920830 0x56176f91b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c1de00700 1 -- 172.21.15.12:0/111689996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f920830 msgr2=0x56176f91b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c1de00700 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f920830 0x56176f91b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c1de00700 1 -- 172.21.15.12:0/111689996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f8f3be0 msgr2=0x56176f91b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c1de00700 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f8f3be0 0x56176f91b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c1de00700 1 -- 172.21.15.12:0/111689996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2009a0b0 con 0x56176f8f1b60 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c1de00700 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f9176d0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f6c100582e0 tx=0x7f6c10058b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea67c150da25397 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6bfea00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6bfea00700 1 -- 172.21.15.12:0/111689996 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10060740 con 0x56176f8f1b60 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6bfea00700 1 -- 172.21.15.12:0/111689996 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f6c1006d070 con 0x56176f8f1b60 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c38813b80 1 -- 172.21.15.12:0/111689996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56176f8e8920 con 0x56176f8f1b60 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6bfea00700 1 -- 172.21.15.12:0/111689996 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c10057840 con 0x56176f8f1b60 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c38813b80 10 client.9600 fetch_fsmap learned FSMap version 81 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.794+0000 7f6c38813b80 1 -- 172.21.15.12:0/111689996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) v3 -- 0x56176f7260a0 con 0x56176f8f1b60 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6bfea00700 1 -- 172.21.15.12:0/111689996 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56176f8e8920 con 0x56176f8f1b60 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6bfea00700 1 -- 172.21.15.12:0/111689996 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 81) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6c100602b0 con 0x56176f8f1b60 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6c38813b80 10 client.9600 fetch_fsmap finished waiting for FSMap version 81 2024-03-29T11:11:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6c38813b80 10 client.9600 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6c38813b80 20 client.9600 populate_metadata read hostname 'smithi012' 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6c38813b80 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.12:6834/2066631988,v1:172.21.15.12:6835/2066631988] conn(0x56176f923940 0x56176f958020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6c38813b80 4 client.9600 mds_command: new command op to 9450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6c38813b80 1 -- 172.21.15.12:0/111689996 --> [v2:172.21.15.12:6834/2066631988,v1:172.21.15.12:6835/2066631988] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56176f8c56f0 con 0x56176f923940 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6c1d400700 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.12:6834/2066631988,v1:172.21.15.12:6835/2066631988] conn(0x56176f923940 0x56176f958020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6c1d400700 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.12:6834/2066631988,v1:172.21.15.12:6835/2066631988] conn(0x56176f923940 0x56176f958020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db404b3cc7196805 server_cookie=4e036413c9328067 in_seq=0 out_seq=0 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.795+0000 7f6bfea00700 10 client.9600 ms_handle_connect on v2:172.21.15.12:6834/2066631988 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.796+0000 7f6bfea00700 1 -- 172.21.15.12:0/111689996 <== mds.0 v2:172.21.15.12:6834/2066631988 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56176f8c56f0 con 0x56176f923940 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.796+0000 7f6bfea00700 10 client.9600 handle_command_reply: tid=0 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.796+0000 7f6c38813b80 1 client.9600 shutdown 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.796+0000 7f6c38813b80 1 -- 172.21.15.12:0/111689996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 msgr2=0x56176f9176d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.796+0000 7f6c38813b80 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f9176d0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f6c100582e0 tx=0x7f6c10058b80 comp rx=0 tx=0).stop 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.797+0000 7f6c38813b80 1 -- 172.21.15.12:0/111689996 shutdown_connections 2024-03-29T11:11:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.797+0000 7f6c38813b80 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.12:6834/2066631988,v1:172.21.15.12:6835/2066631988] conn(0x56176f923940 0x56176f958020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.797+0000 7f6c38813b80 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f920830 0x56176f91b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.797+0000 7f6c38813b80 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f9176d0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.797+0000 7f6c38813b80 1 --2- 172.21.15.12:0/111689996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f8f3be0 0x56176f91b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.797+0000 7f6bfea00700 0 client.9600 ms_handle_reset on v2:172.21.15.12:6834/2066631988 2024-03-29T11:11:49.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.797+0000 7f6c38813b80 1 -- 172.21.15.12:0/111689996 >> 172.21.15.12:0/111689996 conn(0x56176f8546a0 msgr2=0x56176f8f48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:49.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.797+0000 7f6c38813b80 1 -- 172.21.15.12:0/111689996 shutdown_connections 2024-03-29T11:11:49.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.797+0000 7f6c38813b80 1 -- 172.21.15.12:0/111689996 wait complete. 2024-03-29T11:11:49.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.797+0000 7f6c38813b80 20 client.9600 trim_cache size 0 max 16384 2024-03-29T11:11:49.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.815+0000 7f6c38813b80 1 Processor -- start 2024-03-29T11:11:49.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.815+0000 7f6c38813b80 1 -- start start 2024-03-29T11:11:49.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f8f1b60 0x56176f924310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f91a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f920830 0x56176f8f0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c38813b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176f8f76d0 con 0x56176f8f3be0 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c38813b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176f63a390 con 0x56176f920830 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c38813b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56176f921680 con 0x56176f8f1b60 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c1d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f91a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c1d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f91a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46940/0 (socket says 172.21.15.12:46940) 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c1d400700 1 -- 172.21.15.12:0/336628600 learned_addr learned my addr 172.21.15.12:0/336628600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c1d400700 1 -- 172.21.15.12:0/336628600 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f8f1b60 msgr2=0x56176f924310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c1d400700 1 --2- 172.21.15.12:0/336628600 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f8f1b60 0x56176f924310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c1d400700 1 -- 172.21.15.12:0/336628600 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f920830 msgr2=0x56176f8f0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c1d400700 1 --2- 172.21.15.12:0/336628600 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f920830 0x56176f8f0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.816+0000 7f6c1d400700 1 -- 172.21.15.12:0/336628600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56176f80c010 con 0x56176f8f3be0 2024-03-29T11:11:49.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.817+0000 7f6c1d400700 1 --2- 172.21.15.12:0/336628600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f91a060 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c011960 tx=0x7f6c0c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5decdc315b611b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:49.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.817+0000 7f6c0b000700 1 -- 172.21.15.12:0/336628600 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c0045c0 con 0x56176f8f3be0 2024-03-29T11:11:49.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.817+0000 7f6c0b000700 1 -- 172.21.15.12:0/336628600 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c01e070 con 0x56176f8f3be0 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.817+0000 7f6c0b000700 1 -- 172.21.15.12:0/336628600 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c03abb0 con 0x56176f8f3be0 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.817+0000 7f6c38813b80 1 -- 172.21.15.12:0/336628600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 msgr2=0x56176f91a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.817+0000 7f6c38813b80 1 --2- 172.21.15.12:0/336628600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f91a060 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c011960 tx=0x7f6c0c01c000 comp rx=0 tx=0).stop 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.818+0000 7f6c38813b80 1 -- 172.21.15.12:0/336628600 shutdown_connections 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.818+0000 7f6c38813b80 1 --2- 172.21.15.12:0/336628600 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f920830 0x56176f8f0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.818+0000 7f6c38813b80 1 --2- 172.21.15.12:0/336628600 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f8f1b60 0x56176f924310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.818+0000 7f6c38813b80 1 --2- 172.21.15.12:0/336628600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f3be0 0x56176f91a060 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.818+0000 7f6c38813b80 1 -- 172.21.15.12:0/336628600 >> 172.21.15.12:0/336628600 conn(0x56176f8546a0 msgr2=0x56176f8f2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.818+0000 7f6c38813b80 1 -- 172.21.15.12:0/336628600 shutdown_connections 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.818+0000 7f6c38813b80 1 -- 172.21.15.12:0/336628600 wait complete. 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.818+0000 7f6c38813b80 1 -- start start 2024-03-29T11:11:49.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.818+0000 7f6c38813b80 1 Processor -- start 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.819+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f988010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.819+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f8f3be0 0x56176f91c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.819+0000 7f6c38813b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f920830 0x56176f8f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.819+0000 7f6c38813b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176f93f710 con 0x56176f8f1b60 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.819+0000 7f6c38813b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176f932190 con 0x56176f8f3be0 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.819+0000 7f6c38813b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56176f91ce50 con 0x56176f920830 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.819+0000 7f6c38813b80 10 client.0 _ll_register_callbacks cb 0x7f6c10057f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f988010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f988010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46948/0 (socket says 172.21.15.12:46948) 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1de00700 1 -- 172.21.15.12:0/3700455587 learned_addr learned my addr 172.21.15.12:0/3700455587 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1d400700 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f8f3be0 0x56176f91c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1e800700 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f920830 0x56176f8f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1de00700 1 -- 172.21.15.12:0/3700455587 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f920830 msgr2=0x56176f8f09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1de00700 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f920830 0x56176f8f09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1de00700 1 -- 172.21.15.12:0/3700455587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f8f3be0 msgr2=0x56176f91c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1de00700 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f8f3be0 0x56176f91c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1de00700 1 -- 172.21.15.12:0/3700455587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56176f639790 con 0x56176f8f1b60 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6c1de00700 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f988010 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f6c100566a0 tx=0x7f6c10000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771edd33ba1f4f77 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6bfea00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.820+0000 7f6bfea00700 1 -- 172.21.15.12:0/3700455587 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1005e4a0 con 0x56176f8f1b60 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6bfea00700 1 -- 172.21.15.12:0/3700455587 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f6c10057a90 con 0x56176f8f1b60 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6c38813b80 1 -- 172.21.15.12:0/3700455587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c0c02a070 con 0x56176f8f1b60 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6bfea00700 1 -- 172.21.15.12:0/3700455587 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c10060740 con 0x56176f8f1b60 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6c38813b80 10 client.9606 fetch_fsmap learned FSMap version 81 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6c38813b80 1 -- 172.21.15.12:0/3700455587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) v3 -- 0x56176f69a1d0 con 0x56176f8f1b60 2024-03-29T11:11:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6bfea00700 1 -- 172.21.15.12:0/3700455587 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c02a070 con 0x56176f8f1b60 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6bfea00700 1 -- 172.21.15.12:0/3700455587 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 81) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6c100602b0 con 0x56176f8f1b60 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6c38813b80 10 client.9606 fetch_fsmap finished waiting for FSMap version 81 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6c38813b80 10 client.9606 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6c38813b80 20 client.9606 populate_metadata read hostname 'smithi012' 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6c38813b80 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.12:6834/2066631988,v1:172.21.15.12:6835/2066631988] conn(0x56176f930fd0 0x56176f9313b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6c38813b80 4 client.9606 mds_command: new command op to 9450 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6c38813b80 1 -- 172.21.15.12:0/3700455587 --> [v2:172.21.15.12:6834/2066631988,v1:172.21.15.12:6835/2066631988] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56176f67f040 con 0x56176f930fd0 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.821+0000 7f6c1d400700 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.12:6834/2066631988,v1:172.21.15.12:6835/2066631988] conn(0x56176f930fd0 0x56176f9313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c1d400700 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.12:6834/2066631988,v1:172.21.15.12:6835/2066631988] conn(0x56176f930fd0 0x56176f9313b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d66958fbe3486b server_cookie=2ecb1be775fe3ab7 in_seq=0 out_seq=0 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6bfea00700 10 client.9606 ms_handle_connect on v2:172.21.15.12:6834/2066631988 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6bfea00700 1 -- 172.21.15.12:0/3700455587 <== mds.0 v2:172.21.15.12:6834/2066631988 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56176f67f040 con 0x56176f930fd0 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6bfea00700 10 client.9606 handle_command_reply: tid=0 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c38813b80 1 client.9606 shutdown 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c38813b80 1 -- 172.21.15.12:0/3700455587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 msgr2=0x56176f988010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c38813b80 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f988010 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f6c100566a0 tx=0x7f6c10000bc0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c38813b80 1 -- 172.21.15.12:0/3700455587 shutdown_connections 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c38813b80 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.12:6834/2066631988,v1:172.21.15.12:6835/2066631988] conn(0x56176f930fd0 0x56176f9313b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c38813b80 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56176f920830 0x56176f8f09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c38813b80 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56176f8f1b60 0x56176f988010 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6bfea00700 0 client.9606 ms_handle_reset on v2:172.21.15.12:6834/2066631988 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c38813b80 1 --2- 172.21.15.12:0/3700455587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56176f8f3be0 0x56176f91c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c38813b80 1 -- 172.21.15.12:0/3700455587 >> 172.21.15.12:0/3700455587 conn(0x56176f8546a0 msgr2=0x56176f8f2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.822+0000 7f6c38813b80 1 -- 172.21.15.12:0/3700455587 shutdown_connections 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c38813b80 1 -- 172.21.15.12:0/3700455587 wait complete. 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c38813b80 20 client.9606 trim_cache size 0 max 16384 2024-03-29T11:11:49.804 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 9450, 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1465115829999997, 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 81, 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 177, 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 177, 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12.138894257 2024-03-29T11:11:49.805 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:11:49.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 -- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c0006adb0 msgr2=0x7f6c0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c0006adb0 0x7f6c0006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6c200b09a0 tx=0x7f6c0c009f90 comp rx=0 tx=0).stop 2024-03-29T11:11:49.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 -- 172.21.15.12:0/2567135996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 msgr2=0x7f6c20155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 0x7f6c20155210 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000e9b0 tx=0x7f6c1000ed70 comp rx=0 tx=0).stop 2024-03-29T11:11:49.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 -- 172.21.15.12:0/2567135996 shutdown_connections 2024-03-29T11:11:49.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6c0006adb0 0x7f6c0006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c2015cfe0 0x7f6c200af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c2006f310 0x7f6c20155210 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 --2- 172.21.15.12:0/2567135996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c200cd5c0 0x7f6c20154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:49.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 -- 172.21.15.12:0/2567135996 >> 172.21.15.12:0/2567135996 conn(0x7f6c200d1370 msgr2=0x7f6c2015f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:49.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.823+0000 7f6c24c00700 1 -- 172.21.15.12:0/2567135996 shutdown_connections 2024-03-29T11:11:49.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:49.824+0000 7f6c24c00700 1 -- 172.21.15.12:0/2567135996 wait complete. 2024-03-29T11:11:49.819 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T11:11:50.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.039+0000 7fbad2400700 1 Processor -- start 2024-03-29T11:11:50.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.039+0000 7fbad2400700 1 -- start start 2024-03-29T11:11:50.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbad2400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbacc160f10 0x7fbacc0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:50.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbad2400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbacc0cf230 0x7fbacc0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:50.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbad2400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc0ceda0 con 0x7fbacc161590 2024-03-29T11:11:50.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbad2400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbacc0d23d0 con 0x7fbacc160f10 2024-03-29T11:11:50.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbad2400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc1612f0 con 0x7fbacc0cf230 2024-03-29T11:11:50.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbacb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbacc160f10 0x7fbacc0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:50.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbacbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbacc161590 0x7fbacc0b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33198/0 (socket says 172.21.15.12:33198) 2024-03-29T11:11:50.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbacbe00700 1 -- 172.21.15.12:0/2956233646 learned_addr learned my addr 172.21.15.12:0/2956233646 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:50.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbacaa00700 1 --2- 172.21.15.12:0/2956233646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbacc0cf230 0x7fbacc0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:50.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbaca000700 1 -- 172.21.15.12:0/2956233646 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576376200 0 0) 0x7fbacc0ceda0 con 0x7fbacc161590 2024-03-29T11:11:50.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbaca000700 1 -- 172.21.15.12:0/2956233646 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab00035b0 con 0x7fbacc161590 2024-03-29T11:11:50.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbacb400700 1 -- 172.21.15.12:0/2956233646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbacc0cf230 msgr2=0x7fbacc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:50.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbacb400700 1 --2- 172.21.15.12:0/2956233646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbacc0cf230 0x7fbacc0d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbacb400700 1 -- 172.21.15.12:0/2956233646 >> v1:172.21.15.12:6789/0 conn(0x7fbacc161590 legacy=0x7fbacc0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.041+0000 7fbacb400700 1 -- 172.21.15.12:0/2956233646 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc0ce470 con 0x7fbacc160f10 2024-03-29T11:11:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.042+0000 7fbacb400700 1 --2- 172.21.15.12:0/2956233646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc160f10 0x7fbacc0d18b0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009840 tx=0x7fbabc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e02c39eb713b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.042+0000 7fbaca000700 1 -- 172.21.15.12:0/2956233646 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc011830 con 0x7fbacc160f10 2024-03-29T11:11:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.042+0000 7fbaca000700 1 -- 172.21.15.12:0/2956233646 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc011990 con 0x7fbacc160f10 2024-03-29T11:11:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.042+0000 7fbaca000700 1 -- 172.21.15.12:0/2956233646 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc011d00 con 0x7fbacc160f10 2024-03-29T11:11:50.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.042+0000 7fbad2400700 1 -- 172.21.15.12:0/2956233646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc160f10 msgr2=0x7fbacc0d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.042+0000 7fbad2400700 1 --2- 172.21.15.12:0/2956233646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc160f10 0x7fbacc0d18b0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009840 tx=0x7fbabc009b50 comp rx=0 tx=0).stop 2024-03-29T11:11:50.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.043+0000 7fbad2400700 1 -- 172.21.15.12:0/2956233646 shutdown_connections 2024-03-29T11:11:50.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.043+0000 7fbad2400700 1 --2- 172.21.15.12:0/2956233646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc160f10 0x7fbacc0d18b0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:50.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.043+0000 7fbad2400700 1 --2- 172.21.15.12:0/2956233646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbacc0cf230 0x7fbacc0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:50.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.043+0000 7fbad2400700 1 -- 172.21.15.12:0/2956233646 >> 172.21.15.12:0/2956233646 conn(0x7fbacc08aff0 msgr2=0x7fbacc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:50.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.043+0000 7fbad2400700 1 -- 172.21.15.12:0/2956233646 shutdown_connections 2024-03-29T11:11:50.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.043+0000 7fbad2400700 1 -- 172.21.15.12:0/2956233646 wait complete. 2024-03-29T11:11:50.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.044+0000 7fbad2400700 1 Processor -- start 2024-03-29T11:11:50.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.045+0000 7fbad2400700 1 -- start start 2024-03-29T11:11:50.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.045+0000 7fbad2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc0cf230 0x7fbacc15f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:50.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbad2400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbacc160f10 0x7fbacc15a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:50.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbad2400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbacc161590 0x7fbacc0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:50.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbad2400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc0b0290 con 0x7fbacc0cf230 2024-03-29T11:11:50.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbad2400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc15f830 con 0x7fbacc161590 2024-03-29T11:11:50.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbad2400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc160080 con 0x7fbacc160f10 2024-03-29T11:11:50.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc0cf230 0x7fbacc15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:50.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc0cf230 0x7fbacc15f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46960/0 (socket says 172.21.15.12:46960) 2024-03-29T11:11:50.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacb400700 1 -- 172.21.15.12:0/1126967690 learned_addr learned my addr 172.21.15.12:0/1126967690 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:50.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacaa00700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbacc160f10 0x7fbacc15a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:50.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacbe00700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbacc161590 0x7fbacc0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:50.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacb400700 1 -- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbacc160f10 msgr2=0x7fbacc15a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacb400700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbacc160f10 0x7fbacc15a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:50.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacb400700 1 -- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbacc161590 msgr2=0x7fbacc0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacb400700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbacc161590 0x7fbacc0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:50.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacb400700 1 -- 172.21.15.12:0/1126967690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc0094a0 con 0x7fbacc0cf230 2024-03-29T11:11:50.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.046+0000 7fbacb400700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc0cf230 0x7fbacc15f050 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009fd0 tx=0x7fbabc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:50.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.047+0000 7fbac3400700 1 -- 172.21.15.12:0/1126967690 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc0036a0 con 0x7fbacc0cf230 2024-03-29T11:11:50.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.047+0000 7fbac3400700 1 -- 172.21.15.12:0/1126967690 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc01e070 con 0x7fbacc0cf230 2024-03-29T11:11:50.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.047+0000 7fbac3400700 1 -- 172.21.15.12:0/1126967690 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc003a20 con 0x7fbacc0cf230 2024-03-29T11:11:50.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.047+0000 7fbad2400700 1 -- 172.21.15.12:0/1126967690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbacc15f590 con 0x7fbacc0cf230 2024-03-29T11:11:50.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.047+0000 7fbad2400700 1 -- 172.21.15.12:0/1126967690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbacc160600 con 0x7fbacc0cf230 2024-03-29T11:11:50.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.048+0000 7fbad2400700 1 -- 172.21.15.12:0/1126967690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbacc067720 con 0x7fbacc0cf230 2024-03-29T11:11:50.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.049+0000 7fbac3400700 1 -- 172.21.15.12:0/1126967690 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbabc003b80 con 0x7fbacc0cf230 2024-03-29T11:11:50.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.050+0000 7fbac3400700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbaac06adb0 0x7fbaac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:50.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.050+0000 7fbac3400700 1 -- 172.21.15.12:0/1126967690 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fbabc08a490 con 0x7fbacc0cf230 2024-03-29T11:11:50.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.050+0000 7fbacaa00700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbaac06adb0 0x7fbaac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:50.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.051+0000 7fbacaa00700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbaac06adb0 0x7fbaac06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0ca5a0 tx=0x7fbab8008040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:50.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.054+0000 7fbac3400700 1 -- 172.21.15.12:0/1126967690 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbabc055520 con 0x7fbacc0cf230 2024-03-29T11:11:50.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.316+0000 7fbad2400700 1 -- 172.21.15.12:0/1126967690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbacc0ccce0 con 0x7fbacc0cf230 2024-03-29T11:11:50.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.316+0000 7fbac3400700 1 -- 172.21.15.12:0/1126967690 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v81) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbabc058b40 con 0x7fbacc0cf230 2024-03-29T11:11:50.295 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T11:11:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.318+0000 7fbad2400700 1 -- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbaac06adb0 msgr2=0x7fbaac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.318+0000 7fbad2400700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbaac06adb0 0x7fbaac06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0ca5a0 tx=0x7fbab8008040 comp rx=0 tx=0).stop 2024-03-29T11:11:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.319+0000 7fbad2400700 1 -- 172.21.15.12:0/1126967690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc0cf230 msgr2=0x7fbacc15f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.319+0000 7fbad2400700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc0cf230 0x7fbacc15f050 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009fd0 tx=0x7fbabc004750 comp rx=0 tx=0).stop 2024-03-29T11:11:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.319+0000 7fbad2400700 1 -- 172.21.15.12:0/1126967690 shutdown_connections 2024-03-29T11:11:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.319+0000 7fbad2400700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fbaac06adb0 0x7fbaac06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:50.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.319+0000 7fbad2400700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbacc161590 0x7fbacc0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:50.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.319+0000 7fbad2400700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbacc0cf230 0x7fbacc15f050 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:50.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.319+0000 7fbad2400700 1 --2- 172.21.15.12:0/1126967690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbacc160f10 0x7fbacc15a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:50.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.319+0000 7fbad2400700 1 -- 172.21.15.12:0/1126967690 >> 172.21.15.12:0/1126967690 conn(0x7fbacc08aff0 msgr2=0x7fbacc0cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:50.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.319+0000 7fbad2400700 1 -- 172.21.15.12:0/1126967690 shutdown_connections 2024-03-29T11:11:50.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:50.319+0000 7fbad2400700 1 -- 172.21.15.12:0/1126967690 wait complete. 2024-03-29T11:11:50.316 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T11:11:50.316 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T11:11:50.316 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T11:11:50.316 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T11:11:50.316 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T11:11:50.316 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T11:11:50.316 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T11:11:50.316 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T11:11:50.316 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:11:50.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:50.316 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T11:11:50.363 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T11:11:50.363 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T11:11:50.363 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T11:11:50.364 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:11:50.364 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T11:11:50.364 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:11:50.364 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:11:50.364 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:11:50.364 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T11:11:50.364 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:11:50.364 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:11:50.364 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T11:11:50.364 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 36807sec preferred_lft 36807sec 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T11:11:50.365 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:11:50.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:50.367 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T11:11:50.469 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T11:11:50.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:50.472 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T11:11:50.521 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T11:11:50.522 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T11:11:50.522 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T11:11:50.522 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T11:11:50.522 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T11:11:50.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:50.523 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T11:11:50.523 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T11:11:50.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T11:11:50.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T11:11:50.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T11:11:50.523 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T11:11:50.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:50.691 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T11:11:50.738 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T11:11:50.738 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T11:11:50.738 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:11:50.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:50.739 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T11:11:50.846 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:11:50.847 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T11:11:50.892 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:11:50.893 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:11:50.893 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:11:50.977 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T11:11:50.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:50.977 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:11:51.120 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:11:51.120 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:11:51.146 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:11:51.146 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:11:51.146 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T11:11:51.146 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T11:11:51.146 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T11:11:51.152 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T11:11:51.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:51.152 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:11:51.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:51.220 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T11:11:51.318 INFO:teuthology.orchestra.run.smithi012.stdout:client9618 2024-03-29T11:11:51.318 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:11:51.318 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:11:51.362 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T11:11:51.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.623+0000 7f0b8ca00700 1 Processor -- start 2024-03-29T11:11:51.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.623+0000 7f0b8ca00700 1 -- start start 2024-03-29T11:11:51.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.624+0000 7f0b8ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b8815a5c0 0x7f0b880709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:51.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.624+0000 7f0b8ca00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b880cb310 con 0x7f0b8815dfe0 2024-03-29T11:11:51.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.624+0000 7f0b8ca00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b88070f00 con 0x7f0b880ce840 2024-03-29T11:11:51.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.624+0000 7f0b8ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880711a0 con 0x7f0b8815a5c0 2024-03-29T11:11:51.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b85400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0b8815dfe0 0x7f0b880ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33214/0 (socket says 172.21.15.12:33214) 2024-03-29T11:11:51.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b85400700 1 -- 172.21.15.12:0/2466179338 learned_addr learned my addr 172.21.15.12:0/2466179338 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b85e00700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b8815a5c0 0x7f0b880709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423337312 0 0) 0x7f0b880cb310 con 0x7f0b8815dfe0 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b6c0035b0 con 0x7f0b8815dfe0 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137143819 0 0) 0x7f0b88070f00 con 0x7f0b880ce840 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b880cb310 con 0x7f0b880ce840 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2050152942 0 0) 0x7f0b6c0035b0 con 0x7f0b8815dfe0 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b8815a5c0 msgr2=0x7f0b880709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b84a00700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b8815a5c0 0x7f0b880709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 >> v1:172.21.15.159:6789/0 conn(0x7f0b880ce840 legacy=0x7f0b880cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b88071480 con 0x7f0b8815dfe0 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.625+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0b74001670 con 0x7f0b8815dfe0 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 >> v1:172.21.15.12:6789/0 conn(0x7f0b8815dfe0 legacy=0x7f0b880ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b84a00700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b6c003bc0 0x7f0b6c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:51.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b84a00700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c004850 0x7f0b6c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:51.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b84a00700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b8815a5c0 0x7f0b6c005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:51.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880711a0 con 0x7f0b6c003bc0 2024-03-29T11:11:51.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c00bba0 con 0x7f0b6c004850 2024-03-29T11:11:51.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c00bd30 con 0x7f0b8815a5c0 2024-03-29T11:11:51.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b85e00700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b6c003bc0 0x7f0b6c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:51.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b85400700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c004850 0x7f0b6c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:51.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b86800700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b8815a5c0 0x7f0b6c005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:51.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b85e00700 1 -- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b8815a5c0 msgr2=0x7f0b6c005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:51.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b85e00700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b8815a5c0 0x7f0b6c005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:51.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b85e00700 1 -- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c004850 msgr2=0x7f0b6c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:51.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b85e00700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c004850 0x7f0b6c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:51.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b85e00700 1 -- 172.21.15.12:0/2466179338 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c0035b0 con 0x7f0b6c003bc0 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.626+0000 7f0b85400700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c004850 0x7f0b6c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.627+0000 7f0b85e00700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b6c003bc0 0x7f0b6c003fc0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009b90 tx=0x7f0b6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9524163313cce844 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.627+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7800e6c0 con 0x7f0b6c003bc0 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.627+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7800e820 con 0x7f0b6c003bc0 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.627+0000 7f0b84a00700 1 -- 172.21.15.12:0/2466179338 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7800eb90 con 0x7f0b6c003bc0 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.627+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2466179338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b6c003bc0 msgr2=0x7f0b6c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.627+0000 7f0b8ca00700 1 --2- 172.21.15.12:0/2466179338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b6c003bc0 0x7f0b6c003fc0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009b90 tx=0x7f0b6c005880 comp rx=0 tx=0).stop 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.627+0000 7f0b86800700 1 -- 172.21.15.12:0/2466179338 reap_dead start 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.627+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2466179338 shutdown_connections 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.627+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2466179338 >> 172.21.15.12:0/2466179338 conn(0x7f0b880b0d70 msgr2=0x7f0b880b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.627+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2466179338 shutdown_connections 2024-03-29T11:11:51.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.628+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2466179338 wait complete. 2024-03-29T11:11:51.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.629+0000 7f0b8ca00700 1 Processor -- start 2024-03-29T11:11:51.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.630+0000 7f0b8ca00700 1 -- start start 2024-03-29T11:11:51.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.630+0000 7f0b8ca00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c003bc0 0x7f0b880c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:51.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.630+0000 7f0b8ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b880c8be0 0x7f0b8814a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:51.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.630+0000 7f0b8ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b881500e0 0x7f0b88150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:51.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.630+0000 7f0b8ca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b88150ab0 con 0x7f0b881500e0 2024-03-29T11:11:51.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.630+0000 7f0b8ca00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880c9040 con 0x7f0b6c003bc0 2024-03-29T11:11:51.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b8ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8814d6a0 con 0x7f0b880c8be0 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b86800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b881500e0 0x7f0b88150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b86800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b881500e0 0x7f0b88150570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46976/0 (socket says 172.21.15.12:46976) 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b86800700 1 -- 172.21.15.12:0/2701110230 learned_addr learned my addr 172.21.15.12:0/2701110230 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b85e00700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c003bc0 0x7f0b880c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b85400700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b880c8be0 0x7f0b8814a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b86800700 1 -- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b880c8be0 msgr2=0x7f0b8814a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b86800700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b880c8be0 0x7f0b8814a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b86800700 1 -- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c003bc0 msgr2=0x7f0b880c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b86800700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c003bc0 0x7f0b880c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b86800700 1 -- 172.21.15.12:0/2701110230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b880c9270 con 0x7f0b881500e0 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b85400700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b880c8be0 0x7f0b8814a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.631+0000 7f0b85e00700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c003bc0 0x7f0b880c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.632+0000 7f0b86800700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b881500e0 0x7f0b88150570 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f0b880ae750 tx=0x7f0b8000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.632+0000 7f0b7e000700 1 -- 172.21.15.12:0/2701110230 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80003c10 con 0x7f0b881500e0 2024-03-29T11:11:51.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.632+0000 7f0b7e000700 1 -- 172.21.15.12:0/2701110230 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b80003d90 con 0x7f0b881500e0 2024-03-29T11:11:51.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.632+0000 7f0b7e000700 1 -- 172.21.15.12:0/2701110230 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80005310 con 0x7f0b881500e0 2024-03-29T11:11:51.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.632+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2701110230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b88156180 con 0x7f0b881500e0 2024-03-29T11:11:51.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.632+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2701110230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b88156420 con 0x7f0b881500e0 2024-03-29T11:11:51.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.633+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2701110230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b88067720 con 0x7f0b881500e0 2024-03-29T11:11:51.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.634+0000 7f0b7e000700 1 -- 172.21.15.12:0/2701110230 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0b80005470 con 0x7f0b881500e0 2024-03-29T11:11:51.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.635+0000 7f0b7e000700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b6806ade0 0x7f0b6806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:51.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.635+0000 7f0b7e000700 1 -- 172.21.15.12:0/2701110230 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f0b80089510 con 0x7f0b881500e0 2024-03-29T11:11:51.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.635+0000 7f0b85e00700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b6806ade0 0x7f0b6806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:51.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.636+0000 7f0b85e00700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b6806ade0 0x7f0b6806d2a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009b30 tx=0x7f0b78008030 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:51.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.640+0000 7f0b7e000700 1 -- 172.21.15.12:0/2701110230 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b800545a0 con 0x7f0b881500e0 2024-03-29T11:11:51.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.884+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2701110230 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b880cb560 con 0x7f0b6806ade0 2024-03-29T11:11:51.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.887+0000 7f0b7e000700 1 -- 172.21.15.12:0/2701110230 <== mgr.7844 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0b7800ea10 con 0x7f0b6806ade0 2024-03-29T11:11:51.865 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:11:51.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:11:51.865 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T11:11:51.865 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:11:51.865 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b6806ade0 msgr2=0x7f0b6806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b6806ade0 0x7f0b6806d2a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009b30 tx=0x7f0b78008030 comp rx=0 tx=0).stop 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2701110230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b881500e0 msgr2=0x7f0b88150570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b881500e0 0x7f0b88150570 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f0b880ae750 tx=0x7f0b8000ef20 comp rx=0 tx=0).stop 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2701110230 shutdown_connections 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0b6806ade0 0x7f0b6806d2a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b881500e0 0x7f0b88150570 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b6c003bc0 0x7f0b880c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 --2- 172.21.15.12:0/2701110230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b880c8be0 0x7f0b8814a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2701110230 >> 172.21.15.12:0/2701110230 conn(0x7f0b880b0d70 msgr2=0x7f0b880cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:51.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2701110230 shutdown_connections 2024-03-29T11:11:51.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:51.889+0000 7f0b8ca00700 1 -- 172.21.15.12:0/2701110230 wait complete. 2024-03-29T11:11:51.886 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T11:11:52.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.108+0000 7f524f800700 1 Processor -- start 2024-03-29T11:11:52.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.108+0000 7f524f800700 1 -- start start 2024-03-29T11:11:52.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.109+0000 7f524f800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52480d16a0 0x7f52480d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:52.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.109+0000 7f524f800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f52480d2370 0x7f52480ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:52.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.109+0000 7f524f800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5248161420 con 0x7f524815d9d0 2024-03-29T11:11:52.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.109+0000 7f524f800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52480d2820 con 0x7f52480d2370 2024-03-29T11:11:52.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.109+0000 7f524f800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52480d4340 con 0x7f52480d16a0 2024-03-29T11:11:52.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.109+0000 7f524d800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f52480d2370 0x7f52480ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:52.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.109+0000 7f524d800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f52480d2370 0x7f52480ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46978/0 (socket says 172.21.15.12:46978) 2024-03-29T11:11:52.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.109+0000 7f524d800700 1 -- 172.21.15.12:0/2749172712 learned_addr learned my addr 172.21.15.12:0/2749172712 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:52.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.110+0000 7f5247e00700 1 --2- 172.21.15.12:0/2749172712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52480d16a0 0x7f52480d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:52.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.110+0000 7f524d800700 1 -- 172.21.15.12:0/2749172712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52480d16a0 msgr2=0x7f52480d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:52.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.110+0000 7f524d800700 1 --2- 172.21.15.12:0/2749172712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52480d16a0 0x7f52480d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:52.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.110+0000 7f524d800700 1 -- 172.21.15.12:0/2749172712 >> v1:172.21.15.12:6789/0 conn(0x7f524815d9d0 legacy=0x7f52480d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:52.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.110+0000 7f524d800700 1 -- 172.21.15.12:0/2749172712 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5248161700 con 0x7f52480d2370 2024-03-29T11:11:52.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.110+0000 7f524d800700 1 --2- 172.21.15.12:0/2749172712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 0x7f52480ce360 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f5240009e80 tx=0x7f524000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128cc18263b60be7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:52.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.110+0000 7f5247400700 1 -- 172.21.15.12:0/2749172712 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524000e070 con 0x7f52480d2370 2024-03-29T11:11:52.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.111+0000 7f5247400700 1 -- 172.21.15.12:0/2749172712 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524000b980 con 0x7f52480d2370 2024-03-29T11:11:52.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.111+0000 7f5247400700 1 -- 172.21.15.12:0/2749172712 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524000bd30 con 0x7f52480d2370 2024-03-29T11:11:52.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.111+0000 7f524f800700 1 -- 172.21.15.12:0/2749172712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 msgr2=0x7f52480ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:52.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.111+0000 7f524f800700 1 --2- 172.21.15.12:0/2749172712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 0x7f52480ce360 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f5240009e80 tx=0x7f524000b330 comp rx=0 tx=0).stop 2024-03-29T11:11:52.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.111+0000 7f524f800700 1 -- 172.21.15.12:0/2749172712 shutdown_connections 2024-03-29T11:11:52.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.111+0000 7f524f800700 1 --2- 172.21.15.12:0/2749172712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 0x7f52480ce360 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:52.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.111+0000 7f524f800700 1 --2- 172.21.15.12:0/2749172712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52480d16a0 0x7f52480d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:52.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.111+0000 7f524f800700 1 -- 172.21.15.12:0/2749172712 >> 172.21.15.12:0/2749172712 conn(0x7f524808aff0 msgr2=0x7f52480d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:52.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.111+0000 7f524f800700 1 -- 172.21.15.12:0/2749172712 shutdown_connections 2024-03-29T11:11:52.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.111+0000 7f524f800700 1 -- 172.21.15.12:0/2749172712 wait complete. 2024-03-29T11:11:52.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.113+0000 7f524f800700 1 Processor -- start 2024-03-29T11:11:52.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.113+0000 7f524f800700 1 -- start start 2024-03-29T11:11:52.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f524f800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52480d16a0 0x7f52480cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:52.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f524f800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 0x7f5248159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:52.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f524f800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f524815d9d0 0x7f52480cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f524f800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480b6020 con 0x7f52480d2370 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f524f800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524815a440 con 0x7f524815d9d0 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f524f800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52480cb2e0 con 0x7f52480d16a0 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f5247e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 0x7f5248159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f5247e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 0x7f5248159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46992/0 (socket says 172.21.15.12:46992) 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f5247e00700 1 -- 172.21.15.12:0/3188483588 learned_addr learned my addr 172.21.15.12:0/3188483588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f524ce00700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52480d16a0 0x7f52480cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.114+0000 7f524d800700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f524815d9d0 0x7f52480cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f5247e00700 1 -- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52480d16a0 msgr2=0x7f52480cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f5247e00700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52480d16a0 0x7f52480cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f5247e00700 1 -- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f524815d9d0 msgr2=0x7f52480cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f5247e00700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f524815d9d0 0x7f52480cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f5247e00700 1 -- 172.21.15.12:0/3188483588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5240009b30 con 0x7f52480d2370 2024-03-29T11:11:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f5247e00700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 0x7f5248159f00 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f523400e810 tx=0x7f523400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f5244c00700 1 -- 172.21.15.12:0/3188483588 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523400c070 con 0x7f52480d2370 2024-03-29T11:11:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f5244c00700 1 -- 172.21.15.12:0/3188483588 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5234013610 con 0x7f52480d2370 2024-03-29T11:11:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f524f800700 1 -- 172.21.15.12:0/3188483588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52480cb5c0 con 0x7f52480d2370 2024-03-29T11:11:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f5244c00700 1 -- 172.21.15.12:0/3188483588 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52340139c0 con 0x7f52480d2370 2024-03-29T11:11:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.115+0000 7f524f800700 1 -- 172.21.15.12:0/3188483588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524815f490 con 0x7f52480d2370 2024-03-29T11:11:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.116+0000 7f524f800700 1 -- 172.21.15.12:0/3188483588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52480d1ca0 con 0x7f52480d2370 2024-03-29T11:11:52.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.117+0000 7f5244c00700 1 -- 172.21.15.12:0/3188483588 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5234013b20 con 0x7f52480d2370 2024-03-29T11:11:52.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.118+0000 7f5244c00700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f522806ade0 0x7f522806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:52.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.118+0000 7f5244c00700 1 -- 172.21.15.12:0/3188483588 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f52340892f0 con 0x7f52480d2370 2024-03-29T11:11:52.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.119+0000 7f524ce00700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f522806ade0 0x7f522806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:52.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.120+0000 7f524ce00700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f522806ade0 0x7f522806d2a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f523c007fd0 tx=0x7f523c009040 comp rx=0 tx=0).ready entity=mgr.7844 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:52.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.123+0000 7f5244c00700 1 -- 172.21.15.12:0/3188483588 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f52340541e0 con 0x7f52480d2370 2024-03-29T11:11:52.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:52.402+0000 7f524f800700 1 -- 172.21.15.12:0/3188483588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f524815f6b0 con 0x7f52480d2370 2024-03-29T11:11:53.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.177+0000 7f5244c00700 1 -- 172.21.15.12:0/3188483588 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 91) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f52340545b0 con 0x7f52480d2370 2024-03-29T11:11:53.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.180+0000 7f5244c00700 1 -- 172.21.15.12:0/3188483588 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v91) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5234084c90 con 0x7f52480d2370 2024-03-29T11:11:53.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 -- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f522806ade0 msgr2=0x7f522806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f522806ade0 0x7f522806d2a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f523c007fd0 tx=0x7f523c009040 comp rx=0 tx=0).stop 2024-03-29T11:11:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 -- 172.21.15.12:0/3188483588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 msgr2=0x7f5248159f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 0x7f5248159f00 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f523400e810 tx=0x7f523400eb20 comp rx=0 tx=0).stop 2024-03-29T11:11:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 -- 172.21.15.12:0/3188483588 shutdown_connections 2024-03-29T11:11:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f522806ade0 0x7f522806d2a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f524815d9d0 0x7f52480cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52480d16a0 0x7f52480cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 --2- 172.21.15.12:0/3188483588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52480d2370 0x7f5248159f00 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 -- 172.21.15.12:0/3188483588 >> 172.21.15.12:0/3188483588 conn(0x7f524808aff0 msgr2=0x7f52480b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.183+0000 7f524f800700 1 -- 172.21.15.12:0/3188483588 shutdown_connections 2024-03-29T11:11:53.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.184+0000 7f524f800700 1 -- 172.21.15.12:0/3188483588 wait complete. 2024-03-29T11:11:53.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T11:11:53.381 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:53.403+0000 7f43fa710240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:11:53.403 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:53.425+0000 7f9b2f7aa240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:11:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.427+0000 7fc689c00700 1 Processor -- start 2024-03-29T11:11:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.427+0000 7fc689c00700 1 -- start start 2024-03-29T11:11:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.429+0000 7fc689c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc68415d9d0 0x7fc6840d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.429+0000 7fc689c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6840d16a0 0x7fc6840d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.429+0000 7fc689c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc684161420 con 0x7fc6840d2370 2024-03-29T11:11:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.429+0000 7fc689c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6840d2820 con 0x7fc68415d9d0 2024-03-29T11:11:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.429+0000 7fc689c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc684161700 con 0x7fc6840d16a0 2024-03-29T11:11:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.430+0000 7fc682a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc68415d9d0 0x7fc6840d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.430+0000 7fc682a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc68415d9d0 0x7fc6840d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47010/0 (socket says 172.21.15.12:47010) 2024-03-29T11:11:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.430+0000 7fc682a00700 1 -- 172.21.15.12:0/3727219040 learned_addr learned my addr 172.21.15.12:0/3727219040 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.430+0000 7fc682000700 1 --2- 172.21.15.12:0/3727219040 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6840d16a0 0x7fc6840d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.430+0000 7fc682a00700 1 -- 172.21.15.12:0/3727219040 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6840d16a0 msgr2=0x7fc6840d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.430+0000 7fc682a00700 1 --2- 172.21.15.12:0/3727219040 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6840d16a0 0x7fc6840d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.430+0000 7fc682a00700 1 -- 172.21.15.12:0/3727219040 >> v1:172.21.15.12:6789/0 conn(0x7fc6840d2370 legacy=0x7fc68415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.430+0000 7fc682a00700 1 -- 172.21.15.12:0/3727219040 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6841619e0 con 0x7fc68415d9d0 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.430+0000 7fc682a00700 1 --2- 172.21.15.12:0/3727219040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68415d9d0 0x7fc6840d3c30 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fc6740098f0 tx=0x7fc674009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18df33d168098a39 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.431+0000 7fc681600700 1 -- 172.21.15.12:0/3727219040 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc674011830 con 0x7fc68415d9d0 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.431+0000 7fc681600700 1 -- 172.21.15.12:0/3727219040 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc674011990 con 0x7fc68415d9d0 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.431+0000 7fc681600700 1 -- 172.21.15.12:0/3727219040 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc674011d20 con 0x7fc68415d9d0 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.431+0000 7fc689c00700 1 -- 172.21.15.12:0/3727219040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68415d9d0 msgr2=0x7fc6840d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.431+0000 7fc689c00700 1 --2- 172.21.15.12:0/3727219040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68415d9d0 0x7fc6840d3c30 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fc6740098f0 tx=0x7fc674009c00 comp rx=0 tx=0).stop 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.431+0000 7fc689c00700 1 -- 172.21.15.12:0/3727219040 shutdown_connections 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.431+0000 7fc689c00700 1 --2- 172.21.15.12:0/3727219040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc68415d9d0 0x7fc6840d3c30 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.431+0000 7fc689c00700 1 --2- 172.21.15.12:0/3727219040 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6840d16a0 0x7fc6840d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.431+0000 7fc689c00700 1 -- 172.21.15.12:0/3727219040 >> 172.21.15.12:0/3727219040 conn(0x7fc68408aff0 msgr2=0x7fc6840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.432+0000 7fc689c00700 1 -- 172.21.15.12:0/3727219040 shutdown_connections 2024-03-29T11:11:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.432+0000 7fc689c00700 1 -- 172.21.15.12:0/3727219040 wait complete. 2024-03-29T11:11:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.434+0000 7fc689c00700 1 Processor -- start 2024-03-29T11:11:53.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.434+0000 7fc689c00700 1 -- start start 2024-03-29T11:11:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.435+0000 7fc689c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6840d16a0 0x7fc684155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.435+0000 7fc689c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6840d2370 0x7fc6840b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.435+0000 7fc689c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc68415d9d0 0x7fc684159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.435+0000 7fc689c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68415e810 con 0x7fc6840d2370 2024-03-29T11:11:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.435+0000 7fc689c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6840b8df0 con 0x7fc68415d9d0 2024-03-29T11:11:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.435+0000 7fc689c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc68415dee0 con 0x7fc6840d16a0 2024-03-29T11:11:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc682000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6840d2370 0x7fc6840b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc682000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6840d2370 0x7fc6840b8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47024/0 (socket says 172.21.15.12:47024) 2024-03-29T11:11:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc682000700 1 -- 172.21.15.12:0/948042715 learned_addr learned my addr 172.21.15.12:0/948042715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc682a00700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6840d16a0 0x7fc684155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc683400700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc68415d9d0 0x7fc684159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc682000700 1 -- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6840d16a0 msgr2=0x7fc684155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc682000700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6840d16a0 0x7fc684155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc682000700 1 -- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc68415d9d0 msgr2=0x7fc684159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc682000700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc68415d9d0 0x7fc684159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc682000700 1 -- 172.21.15.12:0/948042715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc674009550 con 0x7fc6840d2370 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.436+0000 7fc682000700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6840d2370 0x7fc6840b8480 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fc67000e960 tx=0x7fc67000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.437+0000 7fc67aa00700 1 -- 172.21.15.12:0/948042715 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6700096f0 con 0x7fc6840d2370 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.437+0000 7fc67aa00700 1 -- 172.21.15.12:0/948042715 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67000eeb0 con 0x7fc6840d2370 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.437+0000 7fc67aa00700 1 -- 172.21.15.12:0/948042715 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67000ca70 con 0x7fc6840d2370 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.437+0000 7fc689c00700 1 -- 172.21.15.12:0/948042715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68415e1c0 con 0x7fc6840d2370 2024-03-29T11:11:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.437+0000 7fc689c00700 1 -- 172.21.15.12:0/948042715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc684155560 con 0x7fc6840d2370 2024-03-29T11:11:53.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.438+0000 7fc689c00700 1 -- 172.21.15.12:0/948042715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6840d1ca0 con 0x7fc6840d2370 2024-03-29T11:11:53.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.439+0000 7fc67aa00700 1 -- 172.21.15.12:0/948042715 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 91) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fc67000cbd0 con 0x7fc6840d2370 2024-03-29T11:11:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.440+0000 7fc67aa00700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc66406ae00 0x7fc66406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.440+0000 7fc67aa00700 1 -- 172.21.15.12:0/948042715 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fc6700898a0 con 0x7fc6840d2370 2024-03-29T11:11:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.440+0000 7fc682a00700 1 -- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc66406ae00 msgr2=0x7fc66406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.440+0000 7fc682a00700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc66406ae00 0x7fc66406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:53.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.449+0000 7fc67aa00700 1 -- 172.21.15.12:0/948042715 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc6700549d0 con 0x7fc6840d2370 2024-03-29T11:11:53.565 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:53.586+0000 7f43fa710240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:11:53.615 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:53.637+0000 7f9b2f7aa240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:11:53.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.641+0000 7fc682a00700 1 -- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc66406ae00 msgr2=0x7fc66406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:53.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.641+0000 7fc682a00700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc66406ae00 0x7fc66406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.701+0000 7fc689c00700 1 -- 172.21.15.12:0/948042715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc684067720 con 0x7fc6840d2370 2024-03-29T11:11:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.706+0000 7fc67aa00700 1 -- 172.21.15.12:0/948042715 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc6700210d0 con 0x7fc6840d2370 2024-03-29T11:11:53.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 -- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc66406ae00 msgr2=0x7fc66406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc66406ae00 0x7fc66406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 -- 172.21.15.12:0/948042715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6840d2370 msgr2=0x7fc6840b8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6840d2370 0x7fc6840b8480 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fc67000e960 tx=0x7fc67000ec70 comp rx=0 tx=0).stop 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 -- 172.21.15.12:0/948042715 shutdown_connections 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc66406ae00 0x7fc66406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc68415d9d0 0x7fc684159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6840d16a0 0x7fc684155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 --2- 172.21.15.12:0/948042715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6840d2370 0x7fc6840b8480 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 -- 172.21.15.12:0/948042715 >> 172.21.15.12:0/948042715 conn(0x7fc68408aff0 msgr2=0x7fc6840b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 -- 172.21.15.12:0/948042715 shutdown_connections 2024-03-29T11:11:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.708+0000 7fc689c00700 1 -- 172.21.15.12:0/948042715 wait complete. 2024-03-29T11:11:53.705 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T11:11:53.882 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:53.904+0000 7f43fa710240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:11:53.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.932+0000 7f1ec4a00700 1 Processor -- start 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.932+0000 7f1ec4a00700 1 -- start start 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.933+0000 7f1ec4a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ec00cd480 0x7f1ec006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.933+0000 7f1ec4a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec00cdae0 con 0x7f1ec00cbb30 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.933+0000 7f1ec4a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec00cddc0 con 0x7f1ec00cd020 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.933+0000 7f1ec4a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec00ce0a0 con 0x7f1ec00cd480 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.933+0000 7f1ebe800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ec00cd480 0x7f1ec006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.934+0000 7f1ebe800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ec00cd480 0x7f1ec006e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47026/0 (socket says 172.21.15.12:47026) 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.934+0000 7f1ebe800700 1 -- 172.21.15.12:0/3164456110 learned_addr learned my addr 172.21.15.12:0/3164456110 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.934+0000 7f1ebde00700 1 --1- 172.21.15.12:0/3164456110 >> v1:172.21.15.12:6789/0 conn(0x7f1ec00cbb30 0x7f1ec00cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33242/0 (socket says 172.21.15.12:33242) 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.934+0000 7f1ebe800700 1 -- 172.21.15.12:0/3164456110 >> v1:172.21.15.159:6789/0 conn(0x7f1ec00cd020 legacy=0x7f1ec00b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.934+0000 7f1ebe800700 1 -- 172.21.15.12:0/3164456110 >> v1:172.21.15.12:6789/0 conn(0x7f1ec00cbb30 legacy=0x7f1ec00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.934+0000 7f1ebe800700 1 -- 172.21.15.12:0/3164456110 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec006dc20 con 0x7f1ec00cd480 2024-03-29T11:11:53.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.934+0000 7f1ebca00700 1 -- 172.21.15.12:0/3164456110 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323458690 0 0) 0x7f1ec00cdae0 con 0x7f1ec00cbb30 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.934+0000 7f1ebe800700 1 --2- 172.21.15.12:0/3164456110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 0x7f1ec006e420 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009e50 tx=0x7f1eb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beaa616acd1cbab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.935+0000 7f1ebca00700 1 -- 172.21.15.12:0/3164456110 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb800e070 con 0x7f1ec00cd480 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.935+0000 7f1ebca00700 1 -- 172.21.15.12:0/3164456110 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb800b9c0 con 0x7f1ec00cd480 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.935+0000 7f1ebca00700 1 -- 172.21.15.12:0/3164456110 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb800bd70 con 0x7f1ec00cd480 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.935+0000 7f1ec4a00700 1 -- 172.21.15.12:0/3164456110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 msgr2=0x7f1ec006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.935+0000 7f1ec4a00700 1 --2- 172.21.15.12:0/3164456110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 0x7f1ec006e420 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009e50 tx=0x7f1eb800b330 comp rx=0 tx=0).stop 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.935+0000 7f1ec4a00700 1 -- 172.21.15.12:0/3164456110 shutdown_connections 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.935+0000 7f1ec4a00700 1 --2- 172.21.15.12:0/3164456110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 0x7f1ec006e420 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.935+0000 7f1ec4a00700 1 -- 172.21.15.12:0/3164456110 >> 172.21.15.12:0/3164456110 conn(0x7f1ec00b0d70 msgr2=0x7f1ec00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.935+0000 7f1ec4a00700 1 -- 172.21.15.12:0/3164456110 shutdown_connections 2024-03-29T11:11:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.935+0000 7f1ec4a00700 1 -- 172.21.15.12:0/3164456110 wait complete. 2024-03-29T11:11:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.937+0000 7f1ec4a00700 1 Processor -- start 2024-03-29T11:11:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.937+0000 7f1ec4a00700 1 -- start start 2024-03-29T11:11:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.938+0000 7f1ec4a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ec00cbb30 0x7f1ec00ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.938+0000 7f1ec4a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ec00cd020 0x7f1ec01f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.938+0000 7f1ec4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 0x7f1ec01f1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.938+0000 7f1ec4a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00b3d50 con 0x7f1ec00cd480 2024-03-29T11:11:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.938+0000 7f1ec4a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec01f1b10 con 0x7f1ec00cd020 2024-03-29T11:11:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.938+0000 7f1ec4a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec01f1df0 con 0x7f1ec00cbb30 2024-03-29T11:11:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.938+0000 7f1ebe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 0x7f1ec01f1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.938+0000 7f1ebe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 0x7f1ec01f1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47030/0 (socket says 172.21.15.12:47030) 2024-03-29T11:11:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.938+0000 7f1ebe800700 1 -- 172.21.15.12:0/2222558775 learned_addr learned my addr 172.21.15.12:0/2222558775 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1ebd400700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ec00cd020 0x7f1ec01f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1ebde00700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ec00cbb30 0x7f1ec00ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1ebe800700 1 -- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ec00cbb30 msgr2=0x7f1ec00ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1ebe800700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ec00cbb30 0x7f1ec00ad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1ebe800700 1 -- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ec00cd020 msgr2=0x7f1ec01f0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1ebe800700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ec00cd020 0x7f1ec01f0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1ebe800700 1 -- 172.21.15.12:0/2222558775 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb8009b30 con 0x7f1ec00cd480 2024-03-29T11:11:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1ebe800700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 0x7f1ec01f1490 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800b730 tx=0x7f1eb8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1eb6000700 1 -- 172.21.15.12:0/2222558775 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb801c640 con 0x7f1ec00cd480 2024-03-29T11:11:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1eb6000700 1 -- 172.21.15.12:0/2222558775 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb801c7c0 con 0x7f1ec00cd480 2024-03-29T11:11:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.939+0000 7f1eb6000700 1 -- 172.21.15.12:0/2222558775 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb801cb70 con 0x7f1ec00cd480 2024-03-29T11:11:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.940+0000 7f1ec4a00700 1 -- 172.21.15.12:0/2222558775 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec01f45e0 con 0x7f1ec00cd480 2024-03-29T11:11:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.940+0000 7f1ec4a00700 1 -- 172.21.15.12:0/2222558775 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec01f4ba0 con 0x7f1ec00cd480 2024-03-29T11:11:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.941+0000 7f1ec4a00700 1 -- 172.21.15.12:0/2222558775 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec0067720 con 0x7f1ec00cd480 2024-03-29T11:11:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.942+0000 7f1eb6000700 1 -- 172.21.15.12:0/2222558775 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 91) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f1eb801ccd0 con 0x7f1ec00cd480 2024-03-29T11:11:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.943+0000 7f1eb6000700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ea006ae00 0x7f1ea006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.943+0000 7f1eb6000700 1 -- 172.21.15.12:0/2222558775 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f1eb808aee0 con 0x7f1ec00cd480 2024-03-29T11:11:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.943+0000 7f1ebde00700 1 -- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ea006ae00 msgr2=0x7f1ea006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.943+0000 7f1ebde00700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ea006ae00 0x7f1ea006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:53.947+0000 7f1eb6000700 1 -- 172.21.15.12:0/2222558775 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1eb8055f00 con 0x7f1ec00cd480 2024-03-29T11:11:53.957 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:53.979+0000 7f9b2f7aa240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:11:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.144+0000 7f1ebde00700 1 -- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ea006ae00 msgr2=0x7f1ea006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.144+0000 7f1ebde00700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ea006ae00 0x7f1ea006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:54.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.165+0000 7f1ec4a00700 1 -- 172.21.15.12:0/2222558775 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f1ec00b1bd0 con 0x7f1ec00cd480 2024-03-29T11:11:54.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.170+0000 7f1eb6000700 1 -- 172.21.15.12:0/2222558775 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1eb8059520 con 0x7f1ec00cd480 2024-03-29T11:11:54.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 -- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ea006ae00 msgr2=0x7f1ea006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ea006ae00 0x7f1ea006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 -- 172.21.15.12:0/2222558775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 msgr2=0x7f1ec01f1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 0x7f1ec01f1490 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800b730 tx=0x7f1eb8004ad0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 -- 172.21.15.12:0/2222558775 shutdown_connections 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ea006ae00 0x7f1ea006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ec00cd480 0x7f1ec01f1490 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ec00cbb30 0x7f1ec00ad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 --2- 172.21.15.12:0/2222558775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ec00cd020 0x7f1ec01f0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 -- 172.21.15.12:0/2222558775 >> 172.21.15.12:0/2222558775 conn(0x7f1ec00b0d70 msgr2=0x7f1ec015e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 -- 172.21.15.12:0/2222558775 shutdown_connections 2024-03-29T11:11:54.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.173+0000 7f1ec4a00700 1 -- 172.21.15.12:0/2222558775 wait complete. 2024-03-29T11:11:54.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:11:54.171 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-29T11:11:54.220 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2024-03-29T11:11:54.413 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:54.435+0000 7f43fa710240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:11:54.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.447+0000 7fecf5e00700 1 Processor -- start 2024-03-29T11:11:54.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.447+0000 7fecf5e00700 1 -- start start 2024-03-29T11:11:54.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.448+0000 7fecf5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecf0157f90 0x7fecf00b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:54.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.448+0000 7fecf5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecf01613e0 0x7fecf0161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:54.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.448+0000 7fecf5e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fecf00b3420 0x7fecf00b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:54.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.448+0000 7fecf5e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf0158370 con 0x7fecf01613e0 2024-03-29T11:11:54.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.448+0000 7fecf5e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf00b2710 con 0x7fecf00b3420 2024-03-29T11:11:54.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.448+0000 7fecf5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf00b3070 con 0x7fecf0157f90 2024-03-29T11:11:54.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7fecee000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecf01613e0 0x7fecf0161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7fecee000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecf01613e0 0x7fecf0161860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47034/0 (socket says 172.21.15.12:47034) 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7fecee000700 1 -- 172.21.15.12:0/1709140245 learned_addr learned my addr 172.21.15.12:0/1709140245 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7feceea00700 1 --2- 172.21.15.12:0/1709140245 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecf0157f90 0x7fecf00b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7fecef400700 1 --2- 172.21.15.12:0/1709140245 >> v2:172.21.15.159:3300/0 conn(0x7fecf00b3420 0x7fecf00b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7fecee000700 1 -- 172.21.15.12:0/1709140245 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecf0157f90 msgr2=0x7fecf00b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7fecee000700 1 --2- 172.21.15.12:0/1709140245 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecf0157f90 0x7fecf00b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7fecee000700 1 -- 172.21.15.12:0/1709140245 >> v2:172.21.15.159:3300/0 conn(0x7fecf00b3420 msgr2=0x7fecf00b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7fecee000700 1 --2- 172.21.15.12:0/1709140245 >> v2:172.21.15.159:3300/0 conn(0x7fecf00b3420 0x7fecf00b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7fecee000700 1 -- 172.21.15.12:0/1709140245 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf00b3db0 con 0x7fecf01613e0 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.449+0000 7fecef400700 1 --2- 172.21.15.12:0/1709140245 >> v2:172.21.15.159:3300/0 conn(0x7fecf00b3420 0x7fecf00b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.450+0000 7fecee000700 1 --2- 172.21.15.12:0/1709140245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf01613e0 0x7fecf0161860 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fecf0070950 tx=0x7fecdc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f5cb01bfeebab1a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.450+0000 7feced600700 1 -- 172.21.15.12:0/1709140245 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc011830 con 0x7fecf01613e0 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.450+0000 7feced600700 1 -- 172.21.15.12:0/1709140245 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc011990 con 0x7fecf01613e0 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.450+0000 7feced600700 1 -- 172.21.15.12:0/1709140245 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc011d20 con 0x7fecf01613e0 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.450+0000 7fecf5e00700 1 -- 172.21.15.12:0/1709140245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf01613e0 msgr2=0x7fecf0161860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.450+0000 7fecf5e00700 1 --2- 172.21.15.12:0/1709140245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf01613e0 0x7fecf0161860 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fecf0070950 tx=0x7fecdc009bd0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.451+0000 7fecf5e00700 1 -- 172.21.15.12:0/1709140245 shutdown_connections 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.451+0000 7fecf5e00700 1 --2- 172.21.15.12:0/1709140245 >> v2:172.21.15.159:3300/0 conn(0x7fecf00b3420 0x7fecf00b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.451+0000 7fecf5e00700 1 --2- 172.21.15.12:0/1709140245 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecf0157f90 0x7fecf00b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.451+0000 7fecf5e00700 1 --2- 172.21.15.12:0/1709140245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf01613e0 0x7fecf0161860 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.451+0000 7fecf5e00700 1 -- 172.21.15.12:0/1709140245 >> 172.21.15.12:0/1709140245 conn(0x7fecf00b0d70 msgr2=0x7fecf00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.451+0000 7fecf5e00700 1 -- 172.21.15.12:0/1709140245 shutdown_connections 2024-03-29T11:11:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.451+0000 7fecf5e00700 1 -- 172.21.15.12:0/1709140245 wait complete. 2024-03-29T11:11:54.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.452+0000 7fecf5e00700 1 Processor -- start 2024-03-29T11:11:54.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.453+0000 7fecf5e00700 1 -- start start 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.453+0000 7fecf5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf00b3420 0x7fecf00cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.453+0000 7fecf5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecf0157f90 0x7fecf0153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.453+0000 7fecf5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecf01613e0 0x7fecf0099bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.453+0000 7fecf5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf00cc290 con 0x7fecf00b3420 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.453+0000 7fecf5e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf0153ab0 con 0x7fecf01613e0 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7fecf5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf00cacb0 con 0x7fecf0157f90 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7feceea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf00b3420 0x7fecf00cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7feceea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf00b3420 0x7fecf00cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47048/0 (socket says 172.21.15.12:47048) 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7feceea00700 1 -- 172.21.15.12:0/1453532557 learned_addr learned my addr 172.21.15.12:0/1453532557 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7fecee000700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecf0157f90 0x7fecf0153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7fecef400700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecf01613e0 0x7fecf0099bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:11:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7feceea00700 1 -- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecf0157f90 msgr2=0x7fecf0153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7feceea00700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecf0157f90 0x7fecf0153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7feceea00700 1 -- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecf01613e0 msgr2=0x7fecf0099bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7feceea00700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecf01613e0 0x7fecf0099bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:11:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.454+0000 7feceea00700 1 -- 172.21.15.12:0/1453532557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc009550 con 0x7fecf00b3420 2024-03-29T11:11:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.455+0000 7feceea00700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf00b3420 0x7fecf00cbd50 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fece000e9b0 tx=0x7fece000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:11:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.455+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0009710 con 0x7fecf00b3420 2024-03-29T11:11:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.455+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece000ee50 con 0x7fecf00b3420 2024-03-29T11:11:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.455+0000 7fecf5e00700 1 -- 172.21.15.12:0/1453532557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf00caf90 con 0x7fecf00b3420 2024-03-29T11:11:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.455+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece000ca50 con 0x7fecf00b3420 2024-03-29T11:11:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.455+0000 7fecf5e00700 1 -- 172.21.15.12:0/1453532557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf00cb550 con 0x7fecf00b3420 2024-03-29T11:11:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.456+0000 7fecf5e00700 1 -- 172.21.15.12:0/1453532557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf00cbba0 con 0x7fecf00b3420 2024-03-29T11:11:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.457+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 91) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fece000cbb0 con 0x7fecf00b3420 2024-03-29T11:11:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.458+0000 7fece7400700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:11:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.458+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fece0089580 con 0x7fecf00b3420 2024-03-29T11:11:54.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.458+0000 7fecee000700 1 -- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 msgr2=0x7fecd006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:54.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.458+0000 7fecee000700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:11:54.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.463+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fece0054580 con 0x7fecf00b3420 2024-03-29T11:11:54.523 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:54.545+0000 7f9b2f7aa240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:11:54.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.659+0000 7fecee000700 1 -- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 msgr2=0x7fecd006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:54.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.659+0000 7fecee000700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:11:54.670 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:54.691+0000 7f43fa710240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:11:54.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:54.704+0000 7fecf5e00700 1 -- 172.21.15.12:0/1453532557 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7fecf0161f50 con 0x7fecd006ae00 2024-03-29T11:11:54.789 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:54.811+0000 7f9b2f7aa240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:11:55.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:55.060+0000 7fecee000700 1 -- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 msgr2=0x7fecd006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:55.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:55.060+0000 7fecee000700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T11:11:55.127 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:55.149+0000 7f43fa710240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:11:55.267 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:55.289+0000 7f9b2f7aa240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:11:55.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:55.861+0000 7fecee000700 1 -- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 msgr2=0x7fecd006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:55.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:55.861+0000 7fecee000700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T11:11:55.840 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:55.861+0000 7f43fa710240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:11:55.931 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:55.953+0000 7f43fa710240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:11:56.011 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:56.033+0000 7f9b2f7aa240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:11:56.090 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:56.112+0000 7f43fa710240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:11:56.108 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:56.130+0000 7f9b2f7aa240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:11:56.276 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:56.298+0000 7f9b2f7aa240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:11:56.305 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:56.326+0000 7f43fa710240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:11:56.398 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:56.419+0000 7f43fa710240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:11:56.495 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:56.517+0000 7f43fa710240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:11:56.496 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:56.518+0000 7f9b2f7aa240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:11:56.593 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:56.615+0000 7f9b2f7aa240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:11:56.594 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:56.616+0000 7f43fa710240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:11:56.692 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:56.714+0000 7f43fa710240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:11:56.698 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:56.717+0000 7f9b2f7aa240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:11:56.798 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:56.820+0000 7f9b2f7aa240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:11:56.900 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:56.923+0000 7f9b2f7aa240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:11:56.904 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:56.926+0000 7f43fa710240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:11:57.122 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:57.144+0000 7f9b2f7aa240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:11:57.161 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:57.183+0000 7f43fa710240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:11:57.353 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:57.375+0000 7f43fa710240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:11:57.389 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:57.409+0000 7f9b2f7aa240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:11:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:57.463+0000 7fecee000700 1 -- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 msgr2=0x7fecd006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:11:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:57.463+0000 7fecee000700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T11:11:57.445 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:57.467+0000 7f43fa710240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:57.588 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:57.610+0000 7f9b2f7aa240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:11:57.686 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:57.708+0000 7f9b2f7aa240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:58.177 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:58.199+0000 7f43fa710240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:11:58.390 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:58.411+0000 7f43fa710240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:11:58.452 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:58.473+0000 7f9b2f7aa240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:11:58.597 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:58.619+0000 7f43fa710240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:58.673 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:58.695+0000 7f9b2f7aa240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:11:58.894 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:58.916+0000 7f9b2f7aa240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:59.076 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:59.098+0000 7f43fa710240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:11:59.283 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:59.305+0000 7f43fa710240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:59.398 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:59.420+0000 7f9b2f7aa240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:11:59.615 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:11:59.637+0000 7f9b2f7aa240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:59.725 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:59.747+0000 7f43fa710240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:11:59.837 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:11:59.859+0000 7f43fa710240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:59.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:59.873+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 92) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fece0054950 con 0x7fecf00b3420 2024-03-29T11:11:59.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:59.873+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 msgr2=0x7fecd006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:59.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:11:59.873+0000 7fece7400700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:00.030 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:00.052+0000 7f4366a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:00.030 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:00.052+0000 7f4366a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:00.052+0000 7f4366a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:00.052+0000 7f4366a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:00.052+0000 7f4366a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:00.052+0000 7f4362e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:00.052+0000 7f4362e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:00.052+0000 7f4362e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:00.052+0000 7f4362e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:00.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:00.052+0000 7f4362e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:00.100 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:00.122+0000 7f9b2f7aa240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:12:00.218 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:00.240+0000 7f9b2f7aa240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:00.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.876+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 93) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fece006fe50 con 0x7fecf00b3420 2024-03-29T11:12:00.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.876+0000 7fece7400700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:00.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.876+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7fecd001d2d0 con 0x7fecd006ae00 2024-03-29T11:12:00.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.877+0000 7fecee000700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:00.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.883+0000 7fecee000700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009c10 tx=0x7fecdc008030 comp rx=0 tx=0).ready entity=mgr.9286 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:00.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.954+0000 7fece7400700 1 -- 172.21.15.12:0/1453532557 <== mgr.9286 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fecd001d2d0 con 0x7fecd006ae00 2024-03-29T11:12:00.933 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1 2024-03-29T11:12:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 -- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 msgr2=0x7fecd006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009c10 tx=0x7fecdc008030 comp rx=0 tx=0).stop 2024-03-29T11:12:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 -- 172.21.15.12:0/1453532557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf00b3420 msgr2=0x7fecf00cbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf00b3420 0x7fecf00cbd50 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fece000e9b0 tx=0x7fece000ecc0 comp rx=0 tx=0).stop 2024-03-29T11:12:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 -- 172.21.15.12:0/1453532557 shutdown_connections 2024-03-29T11:12:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fecd006ae00 0x7fecd006d2c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecf01613e0 0x7fecf0099bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf00b3420 0x7fecf00cbd50 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 --2- 172.21.15.12:0/1453532557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecf0157f90 0x7fecf0153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 -- 172.21.15.12:0/1453532557 >> 172.21.15.12:0/1453532557 conn(0x7fecf00b0d70 msgr2=0x7fecf0160a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 -- 172.21.15.12:0/1453532557 shutdown_connections 2024-03-29T11:12:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:00.957+0000 7fecf5e00700 1 -- 172.21.15.12:0/1453532557 wait complete. 2024-03-29T11:12:00.955 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-29T11:12:00.955 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-03-29T11:12:01.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.179+0000 7f9c77600700 1 Processor -- start 2024-03-29T11:12:01.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.179+0000 7f9c77600700 1 -- start start 2024-03-29T11:12:01.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.180+0000 7f9c77600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cd020 0x7f9c700cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:01.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.180+0000 7f9c77600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c700cdae0 con 0x7f9c700b2070 2024-03-29T11:12:01.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.180+0000 7f9c77600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c700cddc0 con 0x7f9c700cbb30 2024-03-29T11:12:01.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.180+0000 7f9c77600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c700ce0a0 con 0x7f9c700cd020 2024-03-29T11:12:01.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.180+0000 7f9c75600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9c700b2070 0x7f9c7006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40286/0 (socket says 172.21.15.12:40286) 2024-03-29T11:12:01.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.180+0000 7f9c75600700 1 -- 172.21.15.12:0/2223142127 learned_addr learned my addr 172.21.15.12:0/2223142127 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:01.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.181+0000 7f9c6fe00700 1 --2- 172.21.15.12:0/2223142127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cd020 0x7f9c700cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:01.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.181+0000 7f9c6f400700 1 -- 172.21.15.12:0/2223142127 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117688590 0 0) 0x7f9c700cdae0 con 0x7f9c700b2070 2024-03-29T11:12:01.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.181+0000 7f9c6f400700 1 -- 172.21.15.12:0/2223142127 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c580035b0 con 0x7f9c700b2070 2024-03-29T11:12:01.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.181+0000 7f9c6f400700 1 -- 172.21.15.12:0/2223142127 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936278467 0 0) 0x7f9c700cddc0 con 0x7f9c700cbb30 2024-03-29T11:12:01.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.181+0000 7f9c6f400700 1 -- 172.21.15.12:0/2223142127 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c700cdae0 con 0x7f9c700cbb30 2024-03-29T11:12:01.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.181+0000 7f9c6f400700 1 -- 172.21.15.12:0/2223142127 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2874805691 0 0) 0x7f9c580035b0 con 0x7f9c700b2070 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.181+0000 7f9c6fe00700 1 -- 172.21.15.12:0/2223142127 >> v1:172.21.15.159:6789/0 conn(0x7f9c700cbb30 legacy=0x7f9c700cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.181+0000 7f9c6fe00700 1 -- 172.21.15.12:0/2223142127 >> v1:172.21.15.12:6789/0 conn(0x7f9c700b2070 legacy=0x7f9c7006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.181+0000 7f9c6fe00700 1 -- 172.21.15.12:0/2223142127 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7006dc20 con 0x7f9c700cd020 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c6fe00700 1 --2- 172.21.15.12:0/2223142127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cd020 0x7f9c700cd480 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009870 tx=0x7f9c5c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67bc7854ee0ffa39 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c6f400700 1 -- 172.21.15.12:0/2223142127 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c680019f0 con 0x7f9c700cd020 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c6f400700 1 -- 172.21.15.12:0/2223142127 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c004030 con 0x7f9c700cd020 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c6f400700 1 -- 172.21.15.12:0/2223142127 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c0043a0 con 0x7f9c700cd020 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c77600700 1 -- 172.21.15.12:0/2223142127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cd020 msgr2=0x7f9c700cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c77600700 1 --2- 172.21.15.12:0/2223142127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cd020 0x7f9c700cd480 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009870 tx=0x7f9c5c009b80 comp rx=0 tx=0).stop 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c77600700 1 -- 172.21.15.12:0/2223142127 shutdown_connections 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c77600700 1 --2- 172.21.15.12:0/2223142127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cd020 0x7f9c700cd480 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c77600700 1 -- 172.21.15.12:0/2223142127 >> 172.21.15.12:0/2223142127 conn(0x7f9c700b0d70 msgr2=0x7f9c700b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c77600700 1 -- 172.21.15.12:0/2223142127 shutdown_connections 2024-03-29T11:12:01.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.182+0000 7f9c77600700 1 -- 172.21.15.12:0/2223142127 wait complete. 2024-03-29T11:12:01.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.184+0000 7f9c77600700 1 Processor -- start 2024-03-29T11:12:01.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.184+0000 7f9c77600700 1 -- start start 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.185+0000 7f9c77600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c700b2070 0x7f9c70070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.185+0000 7f9c77600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cbb30 0x7f9c701f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.185+0000 7f9c77600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c700cd020 0x7f9c701537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.185+0000 7f9c77600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c70152710 con 0x7f9c700cd020 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.185+0000 7f9c77600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c701529f0 con 0x7f9c700b2070 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.185+0000 7f9c77600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c70152cd0 con 0x7f9c700cbb30 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c75600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c700cd020 0x7f9c701537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c74c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c700b2070 0x7f9c70070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c6fe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cbb30 0x7f9c701f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c75600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c700cd020 0x7f9c701537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57400/0 (socket says 172.21.15.12:57400) 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c75600700 1 -- 172.21.15.12:0/3534706199 learned_addr learned my addr 172.21.15.12:0/3534706199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c6fe00700 1 -- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c700b2070 msgr2=0x7f9c70070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c6fe00700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c700b2070 0x7f9c70070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c6fe00700 1 -- 172.21.15.12:0/3534706199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c700cd020 msgr2=0x7f9c701537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c6fe00700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c700cd020 0x7f9c701537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c6fe00700 1 -- 172.21.15.12:0/3534706199 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c0094a0 con 0x7f9c700cbb30 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c75600700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c700cd020 0x7f9c701537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.186+0000 7f9c6fe00700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cbb30 0x7f9c701f2d00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c015000 tx=0x7f9c5c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:01.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.187+0000 7f9c6cc00700 1 -- 172.21.15.12:0/3534706199 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c003e60 con 0x7f9c700cbb30 2024-03-29T11:12:01.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.187+0000 7f9c6cc00700 1 -- 172.21.15.12:0/3534706199 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c01f070 con 0x7f9c700cbb30 2024-03-29T11:12:01.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.187+0000 7f9c6cc00700 1 -- 172.21.15.12:0/3534706199 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c011870 con 0x7f9c700cbb30 2024-03-29T11:12:01.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.187+0000 7f9c77600700 1 -- 172.21.15.12:0/3534706199 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c70152fb0 con 0x7f9c700cbb30 2024-03-29T11:12:01.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.187+0000 7f9c77600700 1 -- 172.21.15.12:0/3534706199 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c70151a30 con 0x7f9c700cbb30 2024-03-29T11:12:01.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.188+0000 7f9c77600700 1 -- 172.21.15.12:0/3534706199 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c70067720 con 0x7f9c700cbb30 2024-03-29T11:12:01.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.189+0000 7f9c6cc00700 1 -- 172.21.15.12:0/3534706199 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 93) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f9c5c0119d0 con 0x7f9c700cbb30 2024-03-29T11:12:01.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.190+0000 7f9c6cc00700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c5006ab30 0x7f9c5006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:01.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.190+0000 7f9c6cc00700 1 -- 172.21.15.12:0/3534706199 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f9c5c021070 con 0x7f9c700cbb30 2024-03-29T11:12:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.190+0000 7f9c74c00700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c5006ab30 0x7f9c5006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:01.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.191+0000 7f9c74c00700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c5006ab30 0x7f9c5006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9c640011e0 tx=0x7f9c64009040 comp rx=0 tx=0).ready entity=mgr.9286 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:01.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.195+0000 7f9c6cc00700 1 -- 172.21.15.12:0/3534706199 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9c5c055c80 con 0x7f9c700cbb30 2024-03-29T11:12:01.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.432+0000 7f9c77600700 1 -- 172.21.15.12:0/3534706199 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c70151bc0 con 0x7f9c5006ab30 2024-03-29T11:12:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.503+0000 7f9c6cc00700 1 -- 172.21.15.12:0/3534706199 <== mgr.9286 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9c70151bc0 con 0x7f9c5006ab30 2024-03-29T11:12:01.481 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path snap_test_dir1 2024-03-29T11:12:01.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 -- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c5006ab30 msgr2=0x7f9c5006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c5006ab30 0x7f9c5006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9c640011e0 tx=0x7f9c64009040 comp rx=0 tx=0).stop 2024-03-29T11:12:01.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 -- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cbb30 msgr2=0x7f9c701f2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cbb30 0x7f9c701f2d00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c015000 tx=0x7f9c5c004750 comp rx=0 tx=0).stop 2024-03-29T11:12:01.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 -- 172.21.15.12:0/3534706199 shutdown_connections 2024-03-29T11:12:01.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9c5006ab30 0x7f9c5006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:01.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c700cd020 0x7f9c701537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:01.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9c700b2070 0x7f9c70070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:01.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 --2- 172.21.15.12:0/3534706199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9c700cbb30 0x7f9c701f2d00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:01.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 -- 172.21.15.12:0/3534706199 >> 172.21.15.12:0/3534706199 conn(0x7f9c700b0d70 msgr2=0x7f9c7015e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:01.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 -- 172.21.15.12:0/3534706199 shutdown_connections 2024-03-29T11:12:01.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.506+0000 7f9c77600700 1 -- 172.21.15.12:0/3534706199 wait complete. 2024-03-29T11:12:01.502 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-03-29T11:12:01.502 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-03-29T11:12:01.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.723+0000 7f14b3c00700 1 Processor -- start 2024-03-29T11:12:01.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.724+0000 7f14b3c00700 1 -- start start 2024-03-29T11:12:01.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.725+0000 7f14b3c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14ac0b4a50 0x7f14ac15b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:01.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.725+0000 7f14b3c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ac06eae0 con 0x7f14ac15af30 2024-03-29T11:12:01.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.725+0000 7f14b3c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ac06f660 con 0x7f14ac06da80 2024-03-29T11:12:01.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.725+0000 7f14b3c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac06faf0 con 0x7f14ac0b4a50 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.725+0000 7f14b1200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f14ac15af30 0x7f14ac06d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40290/0 (socket says 172.21.15.12:40290) 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.725+0000 7f14b1200700 1 -- 172.21.15.12:0/961901462 learned_addr learned my addr 172.21.15.12:0/961901462 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.725+0000 7f14b1c00700 1 --2- 172.21.15.12:0/961901462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14ac0b4a50 0x7f14ac15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.725+0000 7f14ab400700 1 -- 172.21.15.12:0/961901462 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213246939 0 0) 0x7f14ac06eae0 con 0x7f14ac15af30 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.726+0000 7f14ab400700 1 -- 172.21.15.12:0/961901462 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14940035b0 con 0x7f14ac15af30 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.726+0000 7f14ab400700 1 -- 172.21.15.12:0/961901462 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311732872 0 0) 0x7f14ac06f660 con 0x7f14ac06da80 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.726+0000 7f14ab400700 1 -- 172.21.15.12:0/961901462 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14ac06eae0 con 0x7f14ac06da80 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.726+0000 7f14ab400700 1 -- 172.21.15.12:0/961901462 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991656647 0 0) 0x7f14940035b0 con 0x7f14ac15af30 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.726+0000 7f14b1c00700 1 -- 172.21.15.12:0/961901462 >> v1:172.21.15.159:6789/0 conn(0x7f14ac06da80 legacy=0x7f14ac0b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.726+0000 7f14b1c00700 1 -- 172.21.15.12:0/961901462 >> v1:172.21.15.12:6789/0 conn(0x7f14ac15af30 legacy=0x7f14ac06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.726+0000 7f14b1c00700 1 -- 172.21.15.12:0/961901462 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac06e150 con 0x7f14ac0b4a50 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.726+0000 7f14b1c00700 1 --2- 172.21.15.12:0/961901462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14ac0b4a50 0x7f14ac15b310 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009df0 tx=0x7f14a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bf404a2f2eb2ba1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.726+0000 7f14ab400700 1 -- 172.21.15.12:0/961901462 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a400e070 con 0x7f14ac0b4a50 2024-03-29T11:12:01.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.727+0000 7f14ab400700 1 -- 172.21.15.12:0/961901462 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a4003a20 con 0x7f14ac0b4a50 2024-03-29T11:12:01.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.727+0000 7f14ab400700 1 -- 172.21.15.12:0/961901462 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0001670 con 0x7f14ac0b4a50 2024-03-29T11:12:01.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.727+0000 7f14b3c00700 1 -- 172.21.15.12:0/961901462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14ac0b4a50 msgr2=0x7f14ac15b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.727+0000 7f14b3c00700 1 --2- 172.21.15.12:0/961901462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14ac0b4a50 0x7f14ac15b310 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009df0 tx=0x7f14a400b330 comp rx=0 tx=0).stop 2024-03-29T11:12:01.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.727+0000 7f14b3c00700 1 -- 172.21.15.12:0/961901462 shutdown_connections 2024-03-29T11:12:01.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.727+0000 7f14b3c00700 1 --2- 172.21.15.12:0/961901462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14ac0b4a50 0x7f14ac15b310 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:01.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.727+0000 7f14b3c00700 1 -- 172.21.15.12:0/961901462 >> 172.21.15.12:0/961901462 conn(0x7f14ac071fb0 msgr2=0x7f14ac06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:01.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.727+0000 7f14b3c00700 1 -- 172.21.15.12:0/961901462 shutdown_connections 2024-03-29T11:12:01.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.727+0000 7f14b3c00700 1 -- 172.21.15.12:0/961901462 wait complete. 2024-03-29T11:12:01.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.729+0000 7f14b3c00700 1 Processor -- start 2024-03-29T11:12:01.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.729+0000 7f14b3c00700 1 -- start start 2024-03-29T11:12:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14b3c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14ac06da80 0x7f14ac0cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14b3c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14ac0b4a50 0x7f14ac0b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14b3c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14ac15af30 0x7f14ac1f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14b3c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0b1220 con 0x7f14ac06da80 2024-03-29T11:12:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14b3c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0b0c60 con 0x7f14ac0b4a50 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14b3c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0b0f40 con 0x7f14ac15af30 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14b1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14ac06da80 0x7f14ac0cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14b1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14ac06da80 0x7f14ac0cbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57416/0 (socket says 172.21.15.12:57416) 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14b1200700 1 -- 172.21.15.12:0/3298339464 learned_addr learned my addr 172.21.15.12:0/3298339464 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14b1c00700 1 --2- 172.21.15.12:0/3298339464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14ac15af30 0x7f14ac1f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.730+0000 7f14abe00700 1 --2- 172.21.15.12:0/3298339464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14ac0b4a50 0x7f14ac0b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14b1200700 1 -- 172.21.15.12:0/3298339464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14ac15af30 msgr2=0x7f14ac1f60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14b1200700 1 --2- 172.21.15.12:0/3298339464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14ac15af30 0x7f14ac1f60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14b1200700 1 -- 172.21.15.12:0/3298339464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14ac0b4a50 msgr2=0x7f14ac0b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14b1200700 1 --2- 172.21.15.12:0/3298339464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14ac0b4a50 0x7f14ac0b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14b1200700 1 -- 172.21.15.12:0/3298339464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a4009a70 con 0x7f14ac06da80 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14abe00700 1 --2- 172.21.15.12:0/3298339464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14ac0b4a50 0x7f14ac0b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14b1200700 1 --2- 172.21.15.12:0/3298339464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14ac06da80 0x7f14ac0cbc00 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f14a000c750 tx=0x7f14a000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:01.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14a8c00700 1 -- 172.21.15.12:0/3298339464 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0004c80 con 0x7f14ac06da80 2024-03-29T11:12:01.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14a8c00700 1 -- 172.21.15.12:0/3298339464 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a0004de0 con 0x7f14ac06da80 2024-03-29T11:12:01.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14a8c00700 1 -- 172.21.15.12:0/3298339464 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0007310 con 0x7f14ac06da80 2024-03-29T11:12:01.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14b3c00700 1 -- 172.21.15.12:0/3298339464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac147500 con 0x7f14ac06da80 2024-03-29T11:12:01.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.731+0000 7f14b3c00700 1 -- 172.21.15.12:0/3298339464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac1f6d20 con 0x7f14ac06da80 2024-03-29T11:12:01.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.732+0000 7f14b3c00700 1 -- 172.21.15.12:0/3298339464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ac1f6f40 con 0x7f14ac06da80 2024-03-29T11:12:01.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.733+0000 7f14a8c00700 1 -- 172.21.15.12:0/3298339464 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 93) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f14a0007470 con 0x7f14ac06da80 2024-03-29T11:12:01.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.735+0000 7f14a8c00700 1 --2- 172.21.15.12:0/3298339464 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f148c06ab30 0x7f148c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:01.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.735+0000 7f14a8c00700 1 -- 172.21.15.12:0/3298339464 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f14a0088370 con 0x7f14ac06da80 2024-03-29T11:12:01.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.735+0000 7f14abe00700 1 --2- 172.21.15.12:0/3298339464 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f148c06ab30 0x7f148c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:01.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.736+0000 7f14abe00700 1 --2- 172.21.15.12:0/3298339464 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f148c06ab30 0x7f148c06cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f14ac15e110 tx=0x7f1498009040 comp rx=0 tx=0).ready entity=mgr.9286 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:01.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.740+0000 7f14a8c00700 1 -- 172.21.15.12:0/3298339464 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f14ac1f6f40 con 0x7f14ac06da80 2024-03-29T11:12:01.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.980+0000 7f14b3c00700 1 -- 172.21.15.12:0/3298339464 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f14ac06f2f0 con 0x7f148c06ab30 2024-03-29T11:12:01.966 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:01.988+0000 7f436f600700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-03-29T11:12:01.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:01.988+0000 7f14a8c00700 1 -- 172.21.15.12:0/3298339464 <== mgr.9286 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f14980013d0 con 0x7f148c06ab30 2024-03-29T11:12:01.967 INFO:teuthology.orchestra.run.smithi012.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-03-29T11:12:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-29T11:12:01.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:01.981 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-29T11:12:02.029 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T11:12:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.252+0000 7f090c400700 1 Processor -- start 2024-03-29T11:12:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.252+0000 7f090c400700 1 -- start start 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.253+0000 7f090c400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f090415d9d0 0x7f09040d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.253+0000 7f090c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09040d16a0 0x7f09040d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.253+0000 7f090c400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0904161420 con 0x7f09040d2370 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.253+0000 7f090c400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09040d2820 con 0x7f090415d9d0 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.253+0000 7f090c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0904161700 con 0x7f09040d16a0 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.253+0000 7f0909a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f090415d9d0 0x7f09040d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.253+0000 7f0909a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f090415d9d0 0x7f09040d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57428/0 (socket says 172.21.15.12:57428) 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.253+0000 7f0909a00700 1 -- 172.21.15.12:0/4243920256 learned_addr learned my addr 172.21.15.12:0/4243920256 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.254+0000 7f0909000700 1 --2- 172.21.15.12:0/4243920256 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09040d16a0 0x7f09040d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.254+0000 7f0909a00700 1 -- 172.21.15.12:0/4243920256 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09040d16a0 msgr2=0x7f09040d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.254+0000 7f0909a00700 1 --2- 172.21.15.12:0/4243920256 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09040d16a0 0x7f09040d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.254+0000 7f0909a00700 1 -- 172.21.15.12:0/4243920256 >> v1:172.21.15.159:6789/0 conn(0x7f09040d2370 legacy=0x7f090415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.254+0000 7f0909a00700 1 -- 172.21.15.12:0/4243920256 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09041619e0 con 0x7f090415d9d0 2024-03-29T11:12:02.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.254+0000 7f0909a00700 1 --2- 172.21.15.12:0/4243920256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090415d9d0 0x7f09040d3c30 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f08f80098f0 tx=0x7f08f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ccd9cf96ac46200 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.254+0000 7f0903e00700 1 -- 172.21.15.12:0/4243920256 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f800b8a0 con 0x7f090415d9d0 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.254+0000 7f0903e00700 1 -- 172.21.15.12:0/4243920256 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f800ba00 con 0x7f090415d9d0 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.254+0000 7f0903e00700 1 -- 172.21.15.12:0/4243920256 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f800bd90 con 0x7f090415d9d0 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.255+0000 7f090c400700 1 -- 172.21.15.12:0/4243920256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090415d9d0 msgr2=0x7f09040d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.255+0000 7f090c400700 1 --2- 172.21.15.12:0/4243920256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090415d9d0 0x7f09040d3c30 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f08f80098f0 tx=0x7f08f8009c00 comp rx=0 tx=0).stop 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.255+0000 7f090c400700 1 -- 172.21.15.12:0/4243920256 shutdown_connections 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.255+0000 7f090c400700 1 --2- 172.21.15.12:0/4243920256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090415d9d0 0x7f09040d3c30 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.255+0000 7f090c400700 1 --2- 172.21.15.12:0/4243920256 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09040d16a0 0x7f09040d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.255+0000 7f090c400700 1 -- 172.21.15.12:0/4243920256 >> 172.21.15.12:0/4243920256 conn(0x7f090408aff0 msgr2=0x7f09040d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.255+0000 7f090c400700 1 -- 172.21.15.12:0/4243920256 shutdown_connections 2024-03-29T11:12:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.255+0000 7f090c400700 1 -- 172.21.15.12:0/4243920256 wait complete. 2024-03-29T11:12:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.257+0000 7f090c400700 1 Processor -- start 2024-03-29T11:12:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.257+0000 7f090c400700 1 -- start start 2024-03-29T11:12:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f090c400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040d16a0 0x7f09040b88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f090c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09040d2370 0x7f09040b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f090c400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f090415d9d0 0x7f090414b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f090c400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09040b75b0 con 0x7f09040d16a0 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f090c400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09040b8de0 con 0x7f090415d9d0 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f090c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09040b7800 con 0x7f09040d2370 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f0909a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040d16a0 0x7f09040b88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f0909a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040d16a0 0x7f09040b88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57436/0 (socket says 172.21.15.12:57436) 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f0909000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09040d2370 0x7f09040b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f0909a00700 1 -- 172.21.15.12:0/725085037 learned_addr learned my addr 172.21.15.12:0/725085037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.258+0000 7f090a400700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f090415d9d0 0x7f090414b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f0909a00700 1 -- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09040d2370 msgr2=0x7f09040b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f0909a00700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09040d2370 0x7f09040b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f0909a00700 1 -- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f090415d9d0 msgr2=0x7f090414b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f0909a00700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f090415d9d0 0x7f090414b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f0909a00700 1 -- 172.21.15.12:0/725085037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f8009550 con 0x7f09040d16a0 2024-03-29T11:12:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f0909a00700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040d16a0 0x7f09040b88a0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009c40 tx=0x7f08f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f0901600700 1 -- 172.21.15.12:0/725085037 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8004590 con 0x7f09040d16a0 2024-03-29T11:12:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f0901600700 1 -- 172.21.15.12:0/725085037 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f801e070 con 0x7f09040d16a0 2024-03-29T11:12:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f090c400700 1 -- 172.21.15.12:0/725085037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09040b7ae0 con 0x7f09040d16a0 2024-03-29T11:12:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f0901600700 1 -- 172.21.15.12:0/725085037 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f80038e0 con 0x7f09040d16a0 2024-03-29T11:12:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.259+0000 7f090c400700 1 -- 172.21.15.12:0/725085037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09040b80a0 con 0x7f09040d16a0 2024-03-29T11:12:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.260+0000 7f090c400700 1 -- 172.21.15.12:0/725085037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09040d1ca0 con 0x7f09040d16a0 2024-03-29T11:12:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.261+0000 7f0901600700 1 -- 172.21.15.12:0/725085037 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 94) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f08f8003a40 con 0x7f09040d16a0 2024-03-29T11:12:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.262+0000 7f0901600700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08e406ae00 0x7f08e406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.262+0000 7f0901600700 1 -- 172.21.15.12:0/725085037 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f08f8020070 con 0x7f09040d16a0 2024-03-29T11:12:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.262+0000 7f0909000700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08e406ae00 0x7f08e406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.263+0000 7f0909000700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08e406ae00 0x7f08e406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f090415bc50 tx=0x7f08f0008040 comp rx=0 tx=0).ready entity=mgr.9286 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.266+0000 7f0901600700 1 -- 172.21.15.12:0/725085037 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f08f8055a00 con 0x7f09040d16a0 2024-03-29T11:12:02.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:02.541+0000 7f090c400700 1 -- 172.21.15.12:0/725085037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0904067720 con 0x7f09040d16a0 2024-03-29T11:12:02.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.011+0000 7f0901600700 1 -- 172.21.15.12:0/725085037 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f08f8003cf0 con 0x7f09040d16a0 2024-03-29T11:12:02.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.013+0000 7f0901600700 1 -- 172.21.15.12:0/725085037 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v95) v1 ==== 93+0+0 (secure 0 0 0) 0x7f08f806e650 con 0x7f09040d16a0 2024-03-29T11:12:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 -- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08e406ae00 msgr2=0x7f08e406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08e406ae00 0x7f08e406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f090415bc50 tx=0x7f08f0008040 comp rx=0 tx=0).stop 2024-03-29T11:12:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 -- 172.21.15.12:0/725085037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040d16a0 msgr2=0x7f09040b88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040d16a0 0x7f09040b88a0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009c40 tx=0x7f08f8004750 comp rx=0 tx=0).stop 2024-03-29T11:12:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 -- 172.21.15.12:0/725085037 shutdown_connections 2024-03-29T11:12:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f08e406ae00 0x7f08e406d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f090415d9d0 0x7f090414b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:02.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040d16a0 0x7f09040b88a0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:02.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 --2- 172.21.15.12:0/725085037 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09040d2370 0x7f09040b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:02.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 -- 172.21.15.12:0/725085037 >> 172.21.15.12:0/725085037 conn(0x7f090408aff0 msgr2=0x7f09040ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:02.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 -- 172.21.15.12:0/725085037 shutdown_connections 2024-03-29T11:12:02.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.016+0000 7f090c400700 1 -- 172.21.15.12:0/725085037 wait complete. 2024-03-29T11:12:03.015 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:12:03.015 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:12:03.104 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:12:03.193 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:12:03.193 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:12:03.193 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:12:03.220 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:03.242+0000 7f49d499b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:12:03.221 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:03.244+0000 7f8088449240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:12:03.238 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:12:03.238 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:12:03.324 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:03.407 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:03.429+0000 7f8088449240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:12:03.426 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:03.448+0000 7f49d499b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:12:03.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.615+0000 7f7e74400700 1 Processor -- start 2024-03-29T11:12:03.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.616+0000 7f7e74400700 1 -- start start 2024-03-29T11:12:03.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.617+0000 7f7e74400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e6c09a170 0x7f7e6c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:03.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.617+0000 7f7e74400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e6c0d08d0 con 0x7f7e6c09a910 2024-03-29T11:12:03.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.617+0000 7f7e74400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e6c06c300 con 0x7f7e6c15d920 2024-03-29T11:12:03.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.617+0000 7f7e74400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e6c06b9d0 con 0x7f7e6c09a170 2024-03-29T11:12:03.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.617+0000 7f7e71000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7e6c09a910 0x7f7e6c099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40296/0 (socket says 172.21.15.12:40296) 2024-03-29T11:12:03.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.617+0000 7f7e71000700 1 -- 172.21.15.12:0/902123632 learned_addr learned my addr 172.21.15.12:0/902123632 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:03.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.617+0000 7f7e72400700 1 --2- 172.21.15.12:0/902123632 >> v2:172.21.15.12:3300/0 conn(0x7f7e6c09a170 0x7f7e6c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:03.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.618+0000 7f7e72400700 1 -- 172.21.15.12:0/902123632 >> v1:172.21.15.159:6789/0 conn(0x7f7e6c15d920 legacy=0x7f7e6c0cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.618+0000 7f7e6be00700 1 -- 172.21.15.12:0/902123632 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944663258 0 0) 0x7f7e6c0d08d0 con 0x7f7e6c09a910 2024-03-29T11:12:03.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.618+0000 7f7e72400700 1 -- 172.21.15.12:0/902123632 >> v1:172.21.15.12:6789/0 conn(0x7f7e6c09a910 legacy=0x7f7e6c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.618+0000 7f7e72400700 1 -- 172.21.15.12:0/902123632 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c06bcb0 con 0x7f7e6c09a170 2024-03-29T11:12:03.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.618+0000 7f7e72400700 1 --2- 172.21.15.12:0/902123632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a170 0x7f7e6c06c790 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009d90 tx=0x7f7e6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74cba198f68fc146 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:03.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.618+0000 7f7e6be00700 1 -- 172.21.15.12:0/902123632 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400e070 con 0x7f7e6c09a170 2024-03-29T11:12:03.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.618+0000 7f7e6be00700 1 -- 172.21.15.12:0/902123632 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6400bbd0 con 0x7f7e6c09a170 2024-03-29T11:12:03.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.618+0000 7f7e6be00700 1 -- 172.21.15.12:0/902123632 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64012430 con 0x7f7e6c09a170 2024-03-29T11:12:03.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.619+0000 7f7e74400700 1 -- 172.21.15.12:0/902123632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a170 msgr2=0x7f7e6c06c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.619+0000 7f7e74400700 1 --2- 172.21.15.12:0/902123632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a170 0x7f7e6c06c790 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009d90 tx=0x7f7e6400b330 comp rx=0 tx=0).stop 2024-03-29T11:12:03.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.619+0000 7f7e74400700 1 -- 172.21.15.12:0/902123632 shutdown_connections 2024-03-29T11:12:03.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.619+0000 7f7e74400700 1 --2- 172.21.15.12:0/902123632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a170 0x7f7e6c06c790 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:03.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.619+0000 7f7e74400700 1 -- 172.21.15.12:0/902123632 >> 172.21.15.12:0/902123632 conn(0x7f7e6c0d1350 msgr2=0x7f7e6c0d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.619+0000 7f7e74400700 1 -- 172.21.15.12:0/902123632 shutdown_connections 2024-03-29T11:12:03.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.619+0000 7f7e74400700 1 -- 172.21.15.12:0/902123632 wait complete. 2024-03-29T11:12:03.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.621+0000 7f7e74400700 1 Processor -- start 2024-03-29T11:12:03.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.621+0000 7f7e74400700 1 -- start start 2024-03-29T11:12:03.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e74400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e6c09a170 0x7f7e6c0c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e74400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a910 0x7f7e6c15c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e74400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c15d920 0x7f7e6c1f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e74400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c0c7e70 con 0x7f7e6c09a910 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e74400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c15ce70 con 0x7f7e6c15d920 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e74400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c0c7b90 con 0x7f7e6c09a170 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e71000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a910 0x7f7e6c15c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e71000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a910 0x7f7e6c15c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57448/0 (socket says 172.21.15.12:57448) 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e71000700 1 -- 172.21.15.12:0/4074205992 learned_addr learned my addr 172.21.15.12:0/4074205992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e71a00700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e6c09a170 0x7f7e6c0c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e72400700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c15d920 0x7f7e6c1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e71000700 1 -- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e6c09a170 msgr2=0x7f7e6c0c4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.622+0000 7f7e71000700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e6c09a170 0x7f7e6c0c4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.623+0000 7f7e71000700 1 -- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c15d920 msgr2=0x7f7e6c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.623+0000 7f7e71000700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c15d920 0x7f7e6c1f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.623+0000 7f7e71000700 1 -- 172.21.15.12:0/4074205992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64009a70 con 0x7f7e6c09a910 2024-03-29T11:12:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.623+0000 7f7e72400700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c15d920 0x7f7e6c1f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.623+0000 7f7e71000700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a910 0x7f7e6c15c930 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f7e58001eb0 tx=0x7f7e5800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.623+0000 7f7e69600700 1 -- 172.21.15.12:0/4074205992 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58004c80 con 0x7f7e6c09a910 2024-03-29T11:12:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.623+0000 7f7e69600700 1 -- 172.21.15.12:0/4074205992 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e58004de0 con 0x7f7e6c09a910 2024-03-29T11:12:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.623+0000 7f7e74400700 1 -- 172.21.15.12:0/4074205992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6c0c9c40 con 0x7f7e6c09a910 2024-03-29T11:12:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.623+0000 7f7e69600700 1 -- 172.21.15.12:0/4074205992 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58007310 con 0x7f7e6c09a910 2024-03-29T11:12:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.623+0000 7f7e74400700 1 -- 172.21.15.12:0/4074205992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6c0ca200 con 0x7f7e6c09a910 2024-03-29T11:12:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.625+0000 7f7e74400700 1 -- 172.21.15.12:0/4074205992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6c0ca420 con 0x7f7e6c09a910 2024-03-29T11:12:03.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.627+0000 7f7e69600700 1 -- 172.21.15.12:0/4074205992 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f7e58007470 con 0x7f7e6c09a910 2024-03-29T11:12:03.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.627+0000 7f7e69600700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e4c06adb0 0x7f7e4c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:03.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.628+0000 7f7e69600700 1 -- 172.21.15.12:0/4074205992 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f7e580891e0 con 0x7f7e6c09a910 2024-03-29T11:12:03.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.628+0000 7f7e71a00700 1 -- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e4c06adb0 msgr2=0x7f7e4c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:03.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.628+0000 7f7e71a00700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e4c06adb0 0x7f7e4c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:03.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.633+0000 7f7e69600700 1 -- 172.21.15.12:0/4074205992 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7e58054110 con 0x7f7e6c09a910 2024-03-29T11:12:03.728 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:03.751+0000 7f8088449240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:12:03.770 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:03.793+0000 7f49d499b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:12:03.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.829+0000 7f7e71a00700 1 -- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e4c06adb0 msgr2=0x7f7e4c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:03.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.829+0000 7f7e71a00700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e4c06adb0 0x7f7e4c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:03.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.851+0000 7f7e74400700 1 -- 172.21.15.12:0/4074205992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e6c067720 con 0x7f7e6c09a910 2024-03-29T11:12:03.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.853+0000 7f7e69600700 1 -- 172.21.15.12:0/4074205992 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f7e58057730 con 0x7f7e6c09a910 2024-03-29T11:12:03.831 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:03.831 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":178,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:59.862985+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:11:41.110455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-29T11:11:41.640016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1870707179":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/499386558":"2024-03-30T11:11:59.862930+0000","172.21.15.159:6833/43444":"2024-03-30T11:11:59.862930+0000","172.21.15.159:6832/43444":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/3131465277":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/3944320011":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/998093551":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/618363904":"2024-03-30T11:11:59.862930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:03.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.855+0000 7f7e74400700 1 -- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e4c06adb0 msgr2=0x7f7e4c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:03.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.855+0000 7f7e74400700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e4c06adb0 0x7f7e4c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:03.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.855+0000 7f7e74400700 1 -- 172.21.15.12:0/4074205992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a910 msgr2=0x7f7e6c15c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.855+0000 7f7e74400700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a910 0x7f7e6c15c930 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f7e58001eb0 tx=0x7f7e5800c800 comp rx=0 tx=0).stop 2024-03-29T11:12:03.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.855+0000 7f7e74400700 1 -- 172.21.15.12:0/4074205992 shutdown_connections 2024-03-29T11:12:03.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.856+0000 7f7e74400700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e4c06adb0 0x7f7e4c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:03.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.856+0000 7f7e74400700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c15d920 0x7f7e6c1f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:03.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.856+0000 7f7e74400700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e6c09a170 0x7f7e6c0c4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:03.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.856+0000 7f7e74400700 1 --2- 172.21.15.12:0/4074205992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c09a910 0x7f7e6c15c930 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:03.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.856+0000 7f7e74400700 1 -- 172.21.15.12:0/4074205992 >> 172.21.15.12:0/4074205992 conn(0x7f7e6c0d1350 msgr2=0x7f7e6c0cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.856+0000 7f7e74400700 1 -- 172.21.15.12:0/4074205992 shutdown_connections 2024-03-29T11:12:03.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:03.856+0000 7f7e74400700 1 -- 172.21.15.12:0/4074205992 wait complete. 2024-03-29T11:12:03.852 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:11:41.110455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '176', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:11:41.640016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '177', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:12:03.853 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:12:04.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.081+0000 7f261b000700 1 Processor -- start 2024-03-29T11:12:04.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.081+0000 7f261b000700 1 -- start start 2024-03-29T11:12:04.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f261b000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f261415a5c0 0x7f26140709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f261b000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f261415dfe0 0x7f26140ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f261b000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26140cb310 con 0x7f26140ce670 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f261b000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26140ceae0 con 0x7f261415dfe0 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f261b000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2614070f00 con 0x7f261415a5c0 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f2613400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f261415dfe0 0x7f26140ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f2613400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f261415dfe0 0x7f26140ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57462/0 (socket says 172.21.15.12:57462) 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f2613400700 1 -- 172.21.15.12:0/1492363574 learned_addr learned my addr 172.21.15.12:0/1492363574 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f2613e00700 1 --2- 172.21.15.12:0/1492363574 >> v2:172.21.15.159:3300/0 conn(0x7f261415a5c0 0x7f26140709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f2613400700 1 -- 172.21.15.12:0/1492363574 >> v2:172.21.15.159:3300/0 conn(0x7f261415a5c0 msgr2=0x7f26140709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f2613400700 1 --2- 172.21.15.12:0/1492363574 >> v2:172.21.15.159:3300/0 conn(0x7f261415a5c0 0x7f26140709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f2613400700 1 -- 172.21.15.12:0/1492363574 >> v1:172.21.15.159:6789/0 conn(0x7f26140ce670 legacy=0x7f26140b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.083+0000 7f2613400700 1 -- 172.21.15.12:0/1492363574 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26140cca90 con 0x7f261415dfe0 2024-03-29T11:12:04.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.084+0000 7f2613400700 1 --2- 172.21.15.12:0/1492363574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f261415dfe0 0x7f26140ce130 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f2614072e70 tx=0x7f2600009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c2b7e5ec63e378b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:04.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.084+0000 7f2612a00700 1 -- 172.21.15.12:0/1492363574 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2600004230 con 0x7f261415dfe0 2024-03-29T11:12:04.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.084+0000 7f2612a00700 1 -- 172.21.15.12:0/1492363574 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2600004390 con 0x7f261415dfe0 2024-03-29T11:12:04.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.084+0000 7f2612a00700 1 -- 172.21.15.12:0/1492363574 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26000036a0 con 0x7f261415dfe0 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.085+0000 7f261b000700 1 -- 172.21.15.12:0/1492363574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f261415dfe0 msgr2=0x7f26140ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.085+0000 7f261b000700 1 --2- 172.21.15.12:0/1492363574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f261415dfe0 0x7f26140ce130 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f2614072e70 tx=0x7f2600009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.085+0000 7f261b000700 1 -- 172.21.15.12:0/1492363574 shutdown_connections 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.085+0000 7f261b000700 1 --2- 172.21.15.12:0/1492363574 >> v2:172.21.15.159:3300/0 conn(0x7f261415a5c0 0x7f26140709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.085+0000 7f261b000700 1 --2- 172.21.15.12:0/1492363574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f261415dfe0 0x7f26140ce130 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.085+0000 7f261b000700 1 -- 172.21.15.12:0/1492363574 >> 172.21.15.12:0/1492363574 conn(0x7f26140b0d70 msgr2=0x7f26140b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.085+0000 7f261b000700 1 -- 172.21.15.12:0/1492363574 shutdown_connections 2024-03-29T11:12:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.085+0000 7f261b000700 1 -- 172.21.15.12:0/1492363574 wait complete. 2024-03-29T11:12:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.088+0000 7f261b000700 1 Processor -- start 2024-03-29T11:12:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.088+0000 7f261b000700 1 -- start start 2024-03-29T11:12:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.089+0000 7f261b000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26140ce670 0x7f26140c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.089+0000 7f261b000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f261415a5c0 0x7f26140ac090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.089+0000 7f261b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f261415dfe0 0x7f26141450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.089+0000 7f261b000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26140aca00 con 0x7f26140ce670 2024-03-29T11:12:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.089+0000 7f261b000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26140acce0 con 0x7f261415a5c0 2024-03-29T11:12:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.089+0000 7f261b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26140acfc0 con 0x7f261415dfe0 2024-03-29T11:12:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2613e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26140ce670 0x7f26140c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2613e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26140ce670 0x7f26140c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57468/0 (socket says 172.21.15.12:57468) 2024-03-29T11:12:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2613e00700 1 -- 172.21.15.12:0/781584011 learned_addr learned my addr 172.21.15.12:0/781584011 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2613400700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f261415a5c0 0x7f26140ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2619000700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f261415dfe0 0x7f26141450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2613e00700 1 -- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f261415dfe0 msgr2=0x7f26141450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2613e00700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f261415dfe0 0x7f26141450f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2613e00700 1 -- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f261415a5c0 msgr2=0x7f26140ac090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2613e00700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f261415a5c0 0x7f26140ac090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2613e00700 1 -- 172.21.15.12:0/781584011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2600009550 con 0x7f26140ce670 2024-03-29T11:12:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.090+0000 7f2613e00700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26140ce670 0x7f26140c8ce0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f260400e810 tx=0x7f260400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.091+0000 7f260be00700 1 -- 172.21.15.12:0/781584011 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260400c070 con 0x7f26140ce670 2024-03-29T11:12:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.091+0000 7f260be00700 1 -- 172.21.15.12:0/781584011 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2604013610 con 0x7f26140ce670 2024-03-29T11:12:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.091+0000 7f260be00700 1 -- 172.21.15.12:0/781584011 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26040139c0 con 0x7f26140ce670 2024-03-29T11:12:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.091+0000 7f261b000700 1 -- 172.21.15.12:0/781584011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26140ad2a0 con 0x7f26140ce670 2024-03-29T11:12:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.091+0000 7f261b000700 1 -- 172.21.15.12:0/781584011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26140c9220 con 0x7f26140ce670 2024-03-29T11:12:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.092+0000 7f261b000700 1 -- 172.21.15.12:0/781584011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26140c9410 con 0x7f26140ce670 2024-03-29T11:12:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.093+0000 7f260be00700 1 -- 172.21.15.12:0/781584011 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f2604013b20 con 0x7f26140ce670 2024-03-29T11:12:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.094+0000 7f260be00700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f25f406adb0 0x7f25f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.094+0000 7f260be00700 1 -- 172.21.15.12:0/781584011 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f2604089560 con 0x7f26140ce670 2024-03-29T11:12:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.094+0000 7f2613400700 1 -- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f25f406adb0 msgr2=0x7f25f406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.094+0000 7f2613400700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f25f406adb0 0x7f25f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.098+0000 7f260be00700 1 -- 172.21.15.12:0/781584011 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2604054300 con 0x7f26140ce670 2024-03-29T11:12:04.259 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:04.282+0000 7f8088449240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:12:04.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.295+0000 7f2613400700 1 -- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f25f406adb0 msgr2=0x7f25f406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:04.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.295+0000 7f2613400700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f25f406adb0 0x7f25f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:04.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.337+0000 7f261b000700 1 -- 172.21.15.12:0/781584011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26140ad8f0 con 0x7f26140ce670 2024-03-29T11:12:04.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.337+0000 7f260be00700 1 -- 172.21.15.12:0/781584011 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v178) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2604057920 con 0x7f26140ce670 2024-03-29T11:12:04.315 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 -- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f25f406adb0 msgr2=0x7f25f406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f25f406adb0 0x7f25f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 -- 172.21.15.12:0/781584011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26140ce670 msgr2=0x7f26140c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26140ce670 0x7f26140c8ce0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f260400e810 tx=0x7f260400eb20 comp rx=0 tx=0).stop 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 -- 172.21.15.12:0/781584011 shutdown_connections 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f25f406adb0 0x7f25f406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f261415dfe0 0x7f26141450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26140ce670 0x7f26140c8ce0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 --2- 172.21.15.12:0/781584011 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f261415a5c0 0x7f26140ac090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 -- 172.21.15.12:0/781584011 >> 172.21.15.12:0/781584011 conn(0x7f26140b0d70 msgr2=0x7f26140cda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 -- 172.21.15.12:0/781584011 shutdown_connections 2024-03-29T11:12:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.340+0000 7f261b000700 1 -- 172.21.15.12:0/781584011 wait complete. 2024-03-29T11:12:04.337 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:12:04.359 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:04.381+0000 7f49d499b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:12:04.514 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:04.537+0000 7f8088449240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:12:04.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.564+0000 7f9e21a00700 1 Processor -- start 2024-03-29T11:12:04.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.564+0000 7f9e21a00700 1 -- start start 2024-03-29T11:12:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.566+0000 7f9e21a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9e1c15dfe0 0x7f9e1c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.566+0000 7f9e21a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e1c0ce670 0x7f9e1c0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.566+0000 7f9e21a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c0cb310 con 0x7f9e1c15a5c0 2024-03-29T11:12:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.566+0000 7f9e21a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c0ceb00 con 0x7f9e1c0ce670 2024-03-29T11:12:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.566+0000 7f9e21a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c0711a0 con 0x7f9e1c15dfe0 2024-03-29T11:12:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.566+0000 7f9e1aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9e1c15a5c0 0x7f9e1c0709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40310/0 (socket says 172.21.15.12:40310) 2024-03-29T11:12:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.566+0000 7f9e1aa00700 1 -- 172.21.15.12:0/1587187656 learned_addr learned my addr 172.21.15.12:0/1587187656 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:04.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.566+0000 7f9e1b400700 1 --2- 172.21.15.12:0/1587187656 >> v2:172.21.15.12:3300/0 conn(0x7f9e1c0ce670 0x7f9e1c0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.566+0000 7f9e1a000700 1 --2- 172.21.15.12:0/1587187656 >> v2:172.21.15.159:3300/0 conn(0x7f9e1c15dfe0 0x7f9e1c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.567+0000 7f9e1b400700 1 -- 172.21.15.12:0/1587187656 >> v2:172.21.15.159:3300/0 conn(0x7f9e1c15dfe0 msgr2=0x7f9e1c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.567+0000 7f9e19600700 1 -- 172.21.15.12:0/1587187656 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969656239 0 0) 0x7f9e1c0cb310 con 0x7f9e1c15a5c0 2024-03-29T11:12:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.567+0000 7f9e1b400700 1 --2- 172.21.15.12:0/1587187656 >> v2:172.21.15.159:3300/0 conn(0x7f9e1c15dfe0 0x7f9e1c0ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.567+0000 7f9e1b400700 1 -- 172.21.15.12:0/1587187656 >> v1:172.21.15.12:6789/0 conn(0x7f9e1c15a5c0 legacy=0x7f9e1c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.567+0000 7f9e1b400700 1 -- 172.21.15.12:0/1587187656 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c071480 con 0x7f9e1c0ce670 2024-03-29T11:12:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.567+0000 7f9e1b400700 1 --2- 172.21.15.12:0/1587187656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 0x7f9e1c0cba80 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0cb2c0 tx=0x7f9e1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839a199ee5a9d5d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.567+0000 7f9e19600700 1 -- 172.21.15.12:0/1587187656 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400bd20 con 0x7f9e1c0ce670 2024-03-29T11:12:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.567+0000 7f9e19600700 1 -- 172.21.15.12:0/1587187656 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1400be80 con 0x7f9e1c0ce670 2024-03-29T11:12:04.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.567+0000 7f9e19600700 1 -- 172.21.15.12:0/1587187656 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e140116a0 con 0x7f9e1c0ce670 2024-03-29T11:12:04.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.568+0000 7f9e21a00700 1 -- 172.21.15.12:0/1587187656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 msgr2=0x7f9e1c0cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.568+0000 7f9e21a00700 1 --2- 172.21.15.12:0/1587187656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 0x7f9e1c0cba80 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0cb2c0 tx=0x7f9e1400b330 comp rx=0 tx=0).stop 2024-03-29T11:12:04.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.568+0000 7f9e21a00700 1 -- 172.21.15.12:0/1587187656 shutdown_connections 2024-03-29T11:12:04.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.568+0000 7f9e21a00700 1 --2- 172.21.15.12:0/1587187656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 0x7f9e1c0cba80 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.568+0000 7f9e21a00700 1 --2- 172.21.15.12:0/1587187656 >> v2:172.21.15.159:3300/0 conn(0x7f9e1c15dfe0 0x7f9e1c0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.568+0000 7f9e21a00700 1 -- 172.21.15.12:0/1587187656 >> 172.21.15.12:0/1587187656 conn(0x7f9e1c0b0d70 msgr2=0x7f9e1c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:04.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.568+0000 7f9e21a00700 1 -- 172.21.15.12:0/1587187656 shutdown_connections 2024-03-29T11:12:04.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.568+0000 7f9e21a00700 1 -- 172.21.15.12:0/1587187656 wait complete. 2024-03-29T11:12:04.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.570+0000 7f9e21a00700 1 Processor -- start 2024-03-29T11:12:04.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.570+0000 7f9e21a00700 1 -- start start 2024-03-29T11:12:04.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.571+0000 7f9e21a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 0x7f9e1c14d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.571+0000 7f9e21a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e1c15a5c0 0x7f9e1c0aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.571+0000 7f9e21a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e1c15dfe0 0x7f9e1c0af200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.571+0000 7f9e21a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0ac0f0 con 0x7f9e1c0ce670 2024-03-29T11:12:04.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.571+0000 7f9e21a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0ac3d0 con 0x7f9e1c15a5c0 2024-03-29T11:12:04.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.571+0000 7f9e21a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0ac6b0 con 0x7f9e1c15dfe0 2024-03-29T11:12:04.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.571+0000 7f9e1aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 0x7f9e1c14d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.571+0000 7f9e1aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 0x7f9e1c14d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57496/0 (socket says 172.21.15.12:57496) 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.571+0000 7f9e1aa00700 1 -- 172.21.15.12:0/3998141531 learned_addr learned my addr 172.21.15.12:0/3998141531 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e1a000700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e1c15a5c0 0x7f9e1c0aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e1b400700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e1c15dfe0 0x7f9e1c0af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e1aa00700 1 -- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e1c15dfe0 msgr2=0x7f9e1c0af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e1aa00700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e1c15dfe0 0x7f9e1c0af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e1aa00700 1 -- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e1c15a5c0 msgr2=0x7f9e1c0aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e1aa00700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e1c15a5c0 0x7f9e1c0aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e1aa00700 1 -- 172.21.15.12:0/3998141531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e14009a70 con 0x7f9e1c0ce670 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e1aa00700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 0x7f9e1c14d020 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0011e0 tx=0x7f9e0c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e12a00700 1 -- 172.21.15.12:0/3998141531 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c004c80 con 0x7f9e1c0ce670 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e12a00700 1 -- 172.21.15.12:0/3998141531 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c004de0 con 0x7f9e1c0ce670 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e12a00700 1 -- 172.21.15.12:0/3998141531 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c007310 con 0x7f9e1c0ce670 2024-03-29T11:12:04.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.572+0000 7f9e21a00700 1 -- 172.21.15.12:0/3998141531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c0ac990 con 0x7f9e1c0ce670 2024-03-29T11:12:04.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.573+0000 7f9e21a00700 1 -- 172.21.15.12:0/3998141531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c14d560 con 0x7f9e1c0ce670 2024-03-29T11:12:04.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.574+0000 7f9e21a00700 1 -- 172.21.15.12:0/3998141531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c067720 con 0x7f9e1c0ce670 2024-03-29T11:12:04.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.575+0000 7f9e12a00700 1 -- 172.21.15.12:0/3998141531 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f9e0c007470 con 0x7f9e1c0ce670 2024-03-29T11:12:04.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.576+0000 7f9e12a00700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dfc06adb0 0x7f9dfc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:04.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.576+0000 7f9e12a00700 1 -- 172.21.15.12:0/3998141531 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f9e0c0893c0 con 0x7f9e1c0ce670 2024-03-29T11:12:04.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.576+0000 7f9e1a000700 1 -- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dfc06adb0 msgr2=0x7f9dfc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:04.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.576+0000 7f9e1a000700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dfc06adb0 0x7f9dfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:04.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.581+0000 7f9e12a00700 1 -- 172.21.15.12:0/3998141531 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e0c054260 con 0x7f9e1c0ce670 2024-03-29T11:12:04.632 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:04.654+0000 7f49d499b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:12:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.777+0000 7f9e1a000700 1 -- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dfc06adb0 msgr2=0x7f9dfc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.777+0000 7f9e1a000700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dfc06adb0 0x7f9dfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:04.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.798+0000 7f9e21a00700 1 -- 172.21.15.12:0/3998141531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9e1c0b1ad0 con 0x7f9e1c0ce670 2024-03-29T11:12:04.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.798+0000 7f9e12a00700 1 -- 172.21.15.12:0/3998141531 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v178) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9e0c057880 con 0x7f9e1c0ce670 2024-03-29T11:12:04.777 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:12:04.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 -- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dfc06adb0 msgr2=0x7f9dfc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:04.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dfc06adb0 0x7f9dfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 -- 172.21.15.12:0/3998141531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 msgr2=0x7f9e1c14d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:04.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 0x7f9e1c14d020 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0011e0 tx=0x7f9e0c00ca30 comp rx=0 tx=0).stop 2024-03-29T11:12:04.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 -- 172.21.15.12:0/3998141531 shutdown_connections 2024-03-29T11:12:04.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f9dfc06adb0 0x7f9dfc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e1c15dfe0 0x7f9e1c0af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e1c0ce670 0x7f9e1c14d020 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 --2- 172.21.15.12:0/3998141531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e1c15a5c0 0x7f9e1c0aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:04.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 -- 172.21.15.12:0/3998141531 >> 172.21.15.12:0/3998141531 conn(0x7f9e1c0b0d70 msgr2=0x7f9e1c0cd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:04.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 -- 172.21.15.12:0/3998141531 shutdown_connections 2024-03-29T11:12:04.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:04.801+0000 7f9e21a00700 1 -- 172.21.15.12:0/3998141531 wait complete. 2024-03-29T11:12:04.797 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:12:04.974 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:04.997+0000 7f8088449240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:12:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.067+0000 7f56b9400700 1 Processor -- start 2024-03-29T11:12:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.067+0000 7f56b9400700 1 -- start start 2024-03-29T11:12:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.068+0000 7f56b9400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f56b415a5c0 0x7f56b40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.068+0000 7f56b9400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f56b40ce840 0x7f56b40cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b9400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b40cb310 con 0x7f56b415dfe0 2024-03-29T11:12:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b9400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b4070f00 con 0x7f56b415a5c0 2024-03-29T11:12:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b9400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40711a0 con 0x7f56b40ce840 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b2800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f56b415a5c0 0x7f56b40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b2800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f56b415a5c0 0x7f56b40709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57506/0 (socket says 172.21.15.12:57506) 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b2800700 1 -- 172.21.15.12:0/807269367 learned_addr learned my addr 172.21.15.12:0/807269367 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b3200700 1 --2- 172.21.15.12:0/807269367 >> v2:172.21.15.159:3300/0 conn(0x7f56b40ce840 0x7f56b40cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b2800700 1 -- 172.21.15.12:0/807269367 >> v2:172.21.15.159:3300/0 conn(0x7f56b40ce840 msgr2=0x7f56b40cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b2800700 1 --2- 172.21.15.12:0/807269367 >> v2:172.21.15.159:3300/0 conn(0x7f56b40ce840 0x7f56b40cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b2800700 1 -- 172.21.15.12:0/807269367 >> v1:172.21.15.159:6789/0 conn(0x7f56b415dfe0 legacy=0x7f56b40ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.069+0000 7f56b2800700 1 -- 172.21.15.12:0/807269367 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b4071480 con 0x7f56b415a5c0 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.070+0000 7f56b2800700 1 --2- 172.21.15.12:0/807269367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 0x7f56b40709c0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f56b406b630 tx=0x7f56a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca048cdc3968ab4d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.070+0000 7f56b1e00700 1 -- 172.21.15.12:0/807269367 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a400b870 con 0x7f56b415a5c0 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.070+0000 7f56b1e00700 1 -- 172.21.15.12:0/807269367 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a400b9d0 con 0x7f56b415a5c0 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.070+0000 7f56b1e00700 1 -- 172.21.15.12:0/807269367 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a400bd60 con 0x7f56b415a5c0 2024-03-29T11:12:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.071+0000 7f56b9400700 1 -- 172.21.15.12:0/807269367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 msgr2=0x7f56b40709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.071+0000 7f56b9400700 1 --2- 172.21.15.12:0/807269367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 0x7f56b40709c0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f56b406b630 tx=0x7f56a4009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.071+0000 7f56b9400700 1 -- 172.21.15.12:0/807269367 shutdown_connections 2024-03-29T11:12:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.071+0000 7f56b9400700 1 --2- 172.21.15.12:0/807269367 >> v2:172.21.15.159:3300/0 conn(0x7f56b40ce840 0x7f56b40cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.071+0000 7f56b9400700 1 --2- 172.21.15.12:0/807269367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 0x7f56b40709c0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.071+0000 7f56b9400700 1 -- 172.21.15.12:0/807269367 >> 172.21.15.12:0/807269367 conn(0x7f56b40b0d70 msgr2=0x7f56b40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.071+0000 7f56b9400700 1 -- 172.21.15.12:0/807269367 shutdown_connections 2024-03-29T11:12:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.071+0000 7f56b9400700 1 -- 172.21.15.12:0/807269367 wait complete. 2024-03-29T11:12:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.073+0000 7f56b9400700 1 Processor -- start 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.073+0000 7f56b9400700 1 -- start start 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56b9400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40ce840 0x7f56b41455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56b9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 0x7f56b41484e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56b9400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56b415dfe0 0x7f56b4148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56b9400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40ad7a0 con 0x7f56b415a5c0 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56b9400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40acb90 con 0x7f56b40ce840 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56b9400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40ace70 con 0x7f56b415dfe0 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56abe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 0x7f56b41484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56abe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 0x7f56b41484e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57512/0 (socket says 172.21.15.12:57512) 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56abe00700 1 -- 172.21.15.12:0/3965450770 learned_addr learned my addr 172.21.15.12:0/3965450770 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56b2800700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40ce840 0x7f56b41455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.074+0000 7f56b3200700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56b415dfe0 0x7f56b4148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56abe00700 1 -- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56b415dfe0 msgr2=0x7f56b4148a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56abe00700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56b415dfe0 0x7f56b4148a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56abe00700 1 -- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40ce840 msgr2=0x7f56b41455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56abe00700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40ce840 0x7f56b41455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56abe00700 1 -- 172.21.15.12:0/3965450770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a4009550 con 0x7f56b415a5c0 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56abe00700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 0x7f56b41484e0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f569c00c450 tx=0x7f569c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56aaa00700 1 -- 172.21.15.12:0/3965450770 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c0052d0 con 0x7f56b415a5c0 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56aaa00700 1 -- 172.21.15.12:0/3965450770 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569c005430 con 0x7f56b415a5c0 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56aaa00700 1 -- 172.21.15.12:0/3965450770 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c0038c0 con 0x7f56b415a5c0 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56b9400700 1 -- 172.21.15.12:0/3965450770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b40ad150 con 0x7f56b415a5c0 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.075+0000 7f56b9400700 1 -- 172.21.15.12:0/3965450770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b40adc30 con 0x7f56b415a5c0 2024-03-29T11:12:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.077+0000 7f56b9400700 1 -- 172.21.15.12:0/3965450770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b40ad910 con 0x7f56b415a5c0 2024-03-29T11:12:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.077+0000 7f56aaa00700 1 -- 172.21.15.12:0/3965450770 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f569c003a20 con 0x7f56b415a5c0 2024-03-29T11:12:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.078+0000 7f56aaa00700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f569006adb0 0x7f569006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.078+0000 7f56aaa00700 1 -- 172.21.15.12:0/3965450770 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f569c0888b0 con 0x7f56b415a5c0 2024-03-29T11:12:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.078+0000 7f56b2800700 1 -- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f569006adb0 msgr2=0x7f569006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.078+0000 7f56b2800700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f569006adb0 0x7f569006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:05.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.083+0000 7f56aaa00700 1 -- 172.21.15.12:0/3965450770 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f569c053880 con 0x7f56b415a5c0 2024-03-29T11:12:05.121 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:05.143+0000 7f49d499b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:12:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.280+0000 7f56b2800700 1 -- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f569006adb0 msgr2=0x7f569006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.280+0000 7f56b2800700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f569006adb0 0x7f569006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.298+0000 7f56b9400700 1 -- 172.21.15.12:0/3965450770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f56b4067720 con 0x7f56b415a5c0 2024-03-29T11:12:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.298+0000 7f56aaa00700 1 -- 172.21.15.12:0/3965450770 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v178) v1 ==== 100+0+11 (secure 0 0 0) 0x7f569c0170a0 con 0x7f56b415a5c0 2024-03-29T11:12:05.276 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:12:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 -- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f569006adb0 msgr2=0x7f569006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f569006adb0 0x7f569006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 -- 172.21.15.12:0/3965450770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 msgr2=0x7f56b41484e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 0x7f56b41484e0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f569c00c450 tx=0x7f569c00c760 comp rx=0 tx=0).stop 2024-03-29T11:12:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 -- 172.21.15.12:0/3965450770 shutdown_connections 2024-03-29T11:12:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f569006adb0 0x7f569006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56b415dfe0 0x7f56b4148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40ce840 0x7f56b41455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 --2- 172.21.15.12:0/3965450770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56b415a5c0 0x7f56b41484e0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 -- 172.21.15.12:0/3965450770 >> 172.21.15.12:0/3965450770 conn(0x7f56b40b0d70 msgr2=0x7f56b40cdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 -- 172.21.15.12:0/3965450770 shutdown_connections 2024-03-29T11:12:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.301+0000 7f56b9400700 1 -- 172.21.15.12:0/3965450770 wait complete. 2024-03-29T11:12:05.297 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T11:12:05.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.521+0000 7f26f8200700 1 Processor -- start 2024-03-29T11:12:05.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.522+0000 7f26f8200700 1 -- start start 2024-03-29T11:12:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.523+0000 7f26f8200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f00d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.523+0000 7f26f8200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f00cf1e0 con 0x7f26f015dad0 2024-03-29T11:12:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.523+0000 7f26f8200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f015f820 con 0x7f26f015fcb0 2024-03-29T11:12:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.523+0000 7f26f8200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00cfcc0 con 0x7f26f00ced80 2024-03-29T11:12:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.523+0000 7f26f5800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f26f015dad0 0x7f26f00d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40320/0 (socket says 172.21.15.12:40320) 2024-03-29T11:12:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.523+0000 7f26f5800700 1 -- 172.21.15.12:0/1120533003 learned_addr learned my addr 172.21.15.12:0/1120533003 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.523+0000 7f26f4e00700 1 --2- 172.21.15.12:0/1120533003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f00d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.523+0000 7f26efe00700 1 -- 172.21.15.12:0/1120533003 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331743534 0 0) 0x7f26e4001110 con 0x7f26f015dad0 2024-03-29T11:12:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.524+0000 7f26efe00700 1 -- 172.21.15.12:0/1120533003 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d80035b0 con 0x7f26f015dad0 2024-03-29T11:12:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.524+0000 7f26efe00700 1 -- 172.21.15.12:0/1120533003 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597499749 0 0) 0x7f26f015f820 con 0x7f26f015fcb0 2024-03-29T11:12:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.524+0000 7f26efe00700 1 -- 172.21.15.12:0/1120533003 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e4001110 con 0x7f26f015fcb0 2024-03-29T11:12:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.524+0000 7f26f4e00700 1 -- 172.21.15.12:0/1120533003 >> v1:172.21.15.159:6789/0 conn(0x7f26f015fcb0 legacy=0x7f26f0160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.524+0000 7f26f4e00700 1 -- 172.21.15.12:0/1120533003 >> v1:172.21.15.12:6789/0 conn(0x7f26f015dad0 legacy=0x7f26f00d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.524+0000 7f26f4e00700 1 -- 172.21.15.12:0/1120533003 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f00cf420 con 0x7f26f00ced80 2024-03-29T11:12:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.524+0000 7f26f4e00700 1 --2- 172.21.15.12:0/1120533003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f00d1d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009870 tx=0x7f26dc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7909026c9910de1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.524+0000 7f26efe00700 1 -- 172.21.15.12:0/1120533003 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc011830 con 0x7f26f00ced80 2024-03-29T11:12:05.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.524+0000 7f26efe00700 1 -- 172.21.15.12:0/1120533003 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc011990 con 0x7f26f00ced80 2024-03-29T11:12:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.524+0000 7f26efe00700 1 -- 172.21.15.12:0/1120533003 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc011d20 con 0x7f26f00ced80 2024-03-29T11:12:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.525+0000 7f26f8200700 1 -- 172.21.15.12:0/1120533003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 msgr2=0x7f26f00d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.525+0000 7f26f8200700 1 --2- 172.21.15.12:0/1120533003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f00d1d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009870 tx=0x7f26dc009b80 comp rx=0 tx=0).stop 2024-03-29T11:12:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.525+0000 7f26f8200700 1 -- 172.21.15.12:0/1120533003 shutdown_connections 2024-03-29T11:12:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.525+0000 7f26f8200700 1 --2- 172.21.15.12:0/1120533003 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f00d1d40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.525+0000 7f26f8200700 1 -- 172.21.15.12:0/1120533003 >> 172.21.15.12:0/1120533003 conn(0x7f26f008aff0 msgr2=0x7f26f00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.525+0000 7f26f8200700 1 -- 172.21.15.12:0/1120533003 shutdown_connections 2024-03-29T11:12:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.525+0000 7f26f8200700 1 -- 172.21.15.12:0/1120533003 wait complete. 2024-03-29T11:12:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.527+0000 7f26f8200700 1 Processor -- start 2024-03-29T11:12:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.527+0000 7f26f8200700 1 -- start start 2024-03-29T11:12:05.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f8200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f015e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f8200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26f015dad0 0x7f26f00c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f8200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26f015fcb0 0x7f26f00c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f8200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00c7ad0 con 0x7f26f015dad0 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f8200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f0143620 con 0x7f26f015fcb0 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f8200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f01f72b0 con 0x7f26f00ced80 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26f015dad0 0x7f26f00c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f5800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f015e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f5800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f015e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:60198/0 (socket says 172.21.15.12:60198) 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f5800700 1 -- 172.21.15.12:0/967064706 learned_addr learned my addr 172.21.15.12:0/967064706 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.528+0000 7f26f6200700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26f015fcb0 0x7f26f00c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26f5800700 1 -- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26f015fcb0 msgr2=0x7f26f00c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26f5800700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26f015fcb0 0x7f26f00c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26f5800700 1 -- 172.21.15.12:0/967064706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26f015dad0 msgr2=0x7f26f00c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26f5800700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26f015dad0 0x7f26f00c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26f5800700 1 -- 172.21.15.12:0/967064706 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0094a0 con 0x7f26f00ced80 2024-03-29T11:12:05.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26f4e00700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26f015dad0 0x7f26f00c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:05.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26f5800700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f015e120 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f26e4001eb0 tx=0x7f26e400c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26ed600700 1 -- 172.21.15.12:0/967064706 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4005d70 con 0x7f26f00ced80 2024-03-29T11:12:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26ed600700 1 -- 172.21.15.12:0/967064706 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e40070d0 con 0x7f26f00ced80 2024-03-29T11:12:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26ed600700 1 -- 172.21.15.12:0/967064706 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4007460 con 0x7f26f00ced80 2024-03-29T11:12:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26f8200700 1 -- 172.21.15.12:0/967064706 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f01438e0 con 0x7f26f00ced80 2024-03-29T11:12:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.529+0000 7f26f8200700 1 -- 172.21.15.12:0/967064706 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f0143ea0 con 0x7f26f00ced80 2024-03-29T11:12:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.530+0000 7f26f8200700 1 -- 172.21.15.12:0/967064706 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f0144090 con 0x7f26f00ced80 2024-03-29T11:12:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.531+0000 7f26ed600700 1 -- 172.21.15.12:0/967064706 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f26e40075c0 con 0x7f26f00ced80 2024-03-29T11:12:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.532+0000 7f26ed600700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f26d006adb0 0x7f26d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.532+0000 7f26ed600700 1 -- 172.21.15.12:0/967064706 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f26e4088b40 con 0x7f26f00ced80 2024-03-29T11:12:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.532+0000 7f26f4e00700 1 -- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f26d006adb0 msgr2=0x7f26d006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.532+0000 7f26f4e00700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f26d006adb0 0x7f26d006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.537+0000 7f26ed600700 1 -- 172.21.15.12:0/967064706 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26e40539e0 con 0x7f26f00ced80 2024-03-29T11:12:05.690 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:05.713+0000 7f8088449240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:12:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.733+0000 7f26f4e00700 1 -- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f26d006adb0 msgr2=0x7f26d006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.733+0000 7f26f4e00700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f26d006adb0 0x7f26d006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:05.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.767+0000 7f26f8200700 1 -- 172.21.15.12:0/967064706 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f26f00cf1e0 con 0x7f26f00ced80 2024-03-29T11:12:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.768+0000 7f26ed600700 1 -- 172.21.15.12:0/967064706 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v178) v1 ==== 80+0+465 (secure 0 0 0) 0x7f26f00cf1e0 con 0x7f26f00ced80 2024-03-29T11:12:05.746 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1870707179 2024-03-30T11:11:59.862930+0000 2024-03-29T11:12:05.746 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/499386558 2024-03-30T11:11:59.862930+0000 2024-03-29T11:12:05.746 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/43444 2024-03-30T11:11:59.862930+0000 2024-03-29T11:12:05.746 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/43444 2024-03-30T11:11:59.862930+0000 2024-03-29T11:12:05.746 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3131465277 2024-03-30T11:11:59.862930+0000 2024-03-29T11:12:05.746 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3944320011 2024-03-30T11:11:59.862930+0000 2024-03-29T11:12:05.746 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/998093551 2024-03-30T11:11:59.862930+0000 2024-03-29T11:12:05.746 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/618363904 2024-03-30T11:11:59.862930+0000 2024-03-29T11:12:05.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 -- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f26d006adb0 msgr2=0x7f26d006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f26d006adb0 0x7f26d006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 -- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 msgr2=0x7f26f015e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f015e120 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f26e4001eb0 tx=0x7f26e400c9e0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 -- 172.21.15.12:0/967064706 shutdown_connections 2024-03-29T11:12:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f26d006adb0 0x7f26d006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26f015fcb0 0x7f26f00c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26f00ced80 0x7f26f015e120 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 --2- 172.21.15.12:0/967064706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26f015dad0 0x7f26f00c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 -- 172.21.15.12:0/967064706 >> 172.21.15.12:0/967064706 conn(0x7f26f008aff0 msgr2=0x7f26f006c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 -- 172.21.15.12:0/967064706 shutdown_connections 2024-03-29T11:12:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:05.770+0000 7f26f8200700 1 -- 172.21.15.12:0/967064706 wait complete. 2024-03-29T11:12:05.751 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T11:12:05.766 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T11:12:05.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:05.767 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T11:12:05.784 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:05.806+0000 7f8088449240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:12:05.846 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:12:05.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:05.847 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T11:12:05.892 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:05.914+0000 7f49d499b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:12:05.896 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:12:05.896 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:12:05.951 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:12:05.952 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:12:05.952 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:12:05.955 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:05.972+0000 7f8088449240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:12:05.991 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:06.013+0000 7f49d499b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:12:05.998 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:12:05.998 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:12:05.998 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:12:06.044 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:12:06.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:12:06.167 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:06.189+0000 7f49d499b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:12:06.174 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:06.197+0000 7f8088449240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:12:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.272+0000 7fba3dc00700 1 Processor -- start 2024-03-29T11:12:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.272+0000 7fba3dc00700 1 -- start start 2024-03-29T11:12:06.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.273+0000 7fba3dc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fba38160c90 0x7fba380b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.273+0000 7fba3dc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fba380ad970 0x7fba380b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.273+0000 7fba3dc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba380ac4d0 con 0x7fba380718e0 2024-03-29T11:12:06.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.273+0000 7fba3dc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba38071e50 con 0x7fba38160c90 2024-03-29T11:12:06.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.273+0000 7fba3dc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba38072130 con 0x7fba380ad970 2024-03-29T11:12:06.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.273+0000 7fba36000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fba380718e0 0x7fba3806cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40336/0 (socket says 172.21.15.12:40336) 2024-03-29T11:12:06.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.273+0000 7fba36000700 1 -- 172.21.15.12:0/1271607688 learned_addr learned my addr 172.21.15.12:0/1271607688 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:06.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.273+0000 7fba36a00700 1 --2- 172.21.15.12:0/1271607688 >> v2:172.21.15.12:3300/0 conn(0x7fba38160c90 0x7fba380b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:06.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.273+0000 7fba37400700 1 --2- 172.21.15.12:0/1271607688 >> v2:172.21.15.159:3300/0 conn(0x7fba380ad970 0x7fba380b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:06.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.274+0000 7fba36a00700 1 -- 172.21.15.12:0/1271607688 >> v2:172.21.15.159:3300/0 conn(0x7fba380ad970 msgr2=0x7fba380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:06.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.274+0000 7fba36a00700 1 --2- 172.21.15.12:0/1271607688 >> v2:172.21.15.159:3300/0 conn(0x7fba380ad970 0x7fba380b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.274+0000 7fba36a00700 1 -- 172.21.15.12:0/1271607688 >> v1:172.21.15.12:6789/0 conn(0x7fba380718e0 legacy=0x7fba3806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.274+0000 7fba36a00700 1 -- 172.21.15.12:0/1271607688 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba38072410 con 0x7fba38160c90 2024-03-29T11:12:06.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.274+0000 7fba37400700 1 --2- 172.21.15.12:0/1271607688 >> v2:172.21.15.159:3300/0 conn(0x7fba380ad970 0x7fba380b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:06.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.274+0000 7fba36a00700 1 --2- 172.21.15.12:0/1271607688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 0x7fba380b20e0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fba38071d60 tx=0x7fba28009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b59887358e843c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.274+0000 7fba35600700 1 -- 172.21.15.12:0/1271607688 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28011830 con 0x7fba38160c90 2024-03-29T11:12:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.274+0000 7fba35600700 1 -- 172.21.15.12:0/1271607688 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba28011990 con 0x7fba38160c90 2024-03-29T11:12:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.274+0000 7fba35600700 1 -- 172.21.15.12:0/1271607688 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28011d20 con 0x7fba38160c90 2024-03-29T11:12:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.275+0000 7fba3dc00700 1 -- 172.21.15.12:0/1271607688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 msgr2=0x7fba380b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.275+0000 7fba3dc00700 1 --2- 172.21.15.12:0/1271607688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 0x7fba380b20e0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fba38071d60 tx=0x7fba28009b20 comp rx=0 tx=0).stop 2024-03-29T11:12:06.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.275+0000 7fba3dc00700 1 -- 172.21.15.12:0/1271607688 shutdown_connections 2024-03-29T11:12:06.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.275+0000 7fba3dc00700 1 --2- 172.21.15.12:0/1271607688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 0x7fba380b20e0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.275+0000 7fba3dc00700 1 --2- 172.21.15.12:0/1271607688 >> v2:172.21.15.159:3300/0 conn(0x7fba380ad970 0x7fba380b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.275+0000 7fba3dc00700 1 -- 172.21.15.12:0/1271607688 >> 172.21.15.12:0/1271607688 conn(0x7fba380cf190 msgr2=0x7fba380cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:06.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.275+0000 7fba3dc00700 1 -- 172.21.15.12:0/1271607688 shutdown_connections 2024-03-29T11:12:06.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.275+0000 7fba3dc00700 1 -- 172.21.15.12:0/1271607688 wait complete. 2024-03-29T11:12:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.278+0000 7fba3dc00700 1 Processor -- start 2024-03-29T11:12:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.278+0000 7fba3dc00700 1 -- start start 2024-03-29T11:12:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.279+0000 7fba3dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fba380718e0 0x7fba38158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.279+0000 7fba3dc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba380ad970 0x7fba38153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.279+0000 7fba3dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 0x7fba38158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.279+0000 7fba3dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba381554a0 con 0x7fba38160c90 2024-03-29T11:12:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba3dc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba38153980 con 0x7fba380ad970 2024-03-29T11:12:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba3dc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba38154890 con 0x7fba380718e0 2024-03-29T11:12:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba37400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 0x7fba38158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba37400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 0x7fba38158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57548/0 (socket says 172.21.15.12:57548) 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba37400700 1 -- 172.21.15.12:0/613558935 learned_addr learned my addr 172.21.15.12:0/613558935 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba36000700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba380ad970 0x7fba38153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba36a00700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fba380718e0 0x7fba38158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba37400700 1 -- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fba380718e0 msgr2=0x7fba38158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba37400700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fba380718e0 0x7fba38158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba37400700 1 -- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba380ad970 msgr2=0x7fba38153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba37400700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba380ad970 0x7fba38153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.280+0000 7fba37400700 1 -- 172.21.15.12:0/613558935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba280094a0 con 0x7fba38160c90 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.281+0000 7fba37400700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 0x7fba38158df0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fba30007ef0 tx=0x7fba3000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.281+0000 7fba2ea00700 1 -- 172.21.15.12:0/613558935 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3000dd80 con 0x7fba38160c90 2024-03-29T11:12:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.281+0000 7fba2ea00700 1 -- 172.21.15.12:0/613558935 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3000f040 con 0x7fba38160c90 2024-03-29T11:12:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.281+0000 7fba2ea00700 1 -- 172.21.15.12:0/613558935 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba30003c60 con 0x7fba38160c90 2024-03-29T11:12:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.281+0000 7fba3dc00700 1 -- 172.21.15.12:0/613558935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba38154b70 con 0x7fba38160c90 2024-03-29T11:12:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.281+0000 7fba3dc00700 1 -- 172.21.15.12:0/613558935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba38155130 con 0x7fba38160c90 2024-03-29T11:12:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.282+0000 7fba3dc00700 1 -- 172.21.15.12:0/613558935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba38155650 con 0x7fba38160c90 2024-03-29T11:12:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.283+0000 7fba2ea00700 1 -- 172.21.15.12:0/613558935 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fba30004030 con 0x7fba38160c90 2024-03-29T11:12:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.285+0000 7fba2ea00700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fba1806adb0 0x7fba1806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.285+0000 7fba2ea00700 1 -- 172.21.15.12:0/613558935 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fba30089710 con 0x7fba38160c90 2024-03-29T11:12:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.285+0000 7fba36a00700 1 -- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fba1806adb0 msgr2=0x7fba1806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.285+0000 7fba36a00700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fba1806adb0 0x7fba1806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:06.266 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:06.289+0000 7f8088449240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:12:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.289+0000 7fba2ea00700 1 -- 172.21.15.12:0/613558935 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba38155650 con 0x7fba38160c90 2024-03-29T11:12:06.365 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:06.388+0000 7f8088449240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:12:06.391 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:06.413+0000 7f49d499b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:12:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.486+0000 7fba36a00700 1 -- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fba1806adb0 msgr2=0x7fba1806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.486+0000 7fba36a00700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fba1806adb0 0x7fba1806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:06.464 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:06.487+0000 7f8088449240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:12:06.487 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:06.509+0000 7f49d499b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:12:06.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.548+0000 7fba3dc00700 1 -- 172.21.15.12:0/613558935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba38067720 con 0x7fba38160c90 2024-03-29T11:12:06.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.548+0000 7fba2ea00700 1 -- 172.21.15.12:0/613558935 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fba300545d0 con 0x7fba38160c90 2024-03-29T11:12:06.527 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:06.527 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/652112625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":652112625},{"type":"v1","addr":"172.21.15.159:6836","nonce":652112625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9166,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2803735172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2803735172},{"type":"v1","addr":"172.21.15.159:6837","nonce":2803735172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9444,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3698805288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3698805288},{"type":"v1","addr":"172.21.15.12:6833","nonce":3698805288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:11:42.673965+0000","modified":"2024-03-29T11:11:46.149885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9450":{"gid":9450,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/2066631988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2066631988},{"type":"v1","addr":"172.21.15.12:6835","nonce":2066631988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-29T11:12:06.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 -- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fba1806adb0 msgr2=0x7fba1806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fba1806adb0 0x7fba1806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 -- 172.21.15.12:0/613558935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 msgr2=0x7fba38158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 0x7fba38158df0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fba30007ef0 tx=0x7fba3000d3b0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 -- 172.21.15.12:0/613558935 shutdown_connections 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fba1806adb0 0x7fba1806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba38160c90 0x7fba38158df0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fba380718e0 0x7fba38158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 --2- 172.21.15.12:0/613558935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba380ad970 0x7fba38153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 -- 172.21.15.12:0/613558935 >> 172.21.15.12:0/613558935 conn(0x7fba380cf190 msgr2=0x7fba380aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 -- 172.21.15.12:0/613558935 shutdown_connections 2024-03-29T11:12:06.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.551+0000 7fba3dc00700 1 -- 172.21.15.12:0/613558935 wait complete. 2024-03-29T11:12:06.531 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 81 2024-03-29T11:12:06.550 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:06.562 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:06.584+0000 7f8088449240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:12:06.593 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:06.616+0000 7f49d499b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:12:06.703 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:06.726+0000 7f49d499b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:12:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.781+0000 7efeb1000700 1 Processor -- start 2024-03-29T11:12:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.781+0000 7efeb1000700 1 -- start start 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.782+0000 7efeb1000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efeac15da50 0x7efeac0ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.782+0000 7efeb1000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efeac0cec70 0x7efeac0d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.782+0000 7efeb1000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac0d24e0 con 0x7efeac0d0b20 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.782+0000 7efeb1000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeac0d0fd0 con 0x7efeac15da50 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.782+0000 7efeb1000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeac0b95d0 con 0x7efeac0cec70 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.782+0000 7efea9e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efeac15da50 0x7efeac0ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.782+0000 7efea9e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efeac15da50 0x7efeac0ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57560/0 (socket says 172.21.15.12:57560) 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.782+0000 7efea9e00700 1 -- 172.21.15.12:0/373360776 learned_addr learned my addr 172.21.15.12:0/373360776 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.782+0000 7efea9400700 1 --2- 172.21.15.12:0/373360776 >> v2:172.21.15.159:3300/0 conn(0x7efeac0cec70 0x7efeac0d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.783+0000 7efea9e00700 1 -- 172.21.15.12:0/373360776 >> v2:172.21.15.159:3300/0 conn(0x7efeac0cec70 msgr2=0x7efeac0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.783+0000 7efea9e00700 1 --2- 172.21.15.12:0/373360776 >> v2:172.21.15.159:3300/0 conn(0x7efeac0cec70 0x7efeac0d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.783+0000 7efea9e00700 1 -- 172.21.15.12:0/373360776 >> v1:172.21.15.159:6789/0 conn(0x7efeac0d0b20 legacy=0x7efeac0b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.783+0000 7efea9e00700 1 -- 172.21.15.12:0/373360776 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac0d27c0 con 0x7efeac15da50 2024-03-29T11:12:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.783+0000 7efea9e00700 1 --2- 172.21.15.12:0/373360776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac15da50 0x7efeac0ce730 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7efeac0740a0 tx=0x7efe9c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3e89b3368e2927 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.783+0000 7efea8a00700 1 -- 172.21.15.12:0/373360776 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c004230 con 0x7efeac15da50 2024-03-29T11:12:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.783+0000 7efea8a00700 1 -- 172.21.15.12:0/373360776 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9c004390 con 0x7efeac15da50 2024-03-29T11:12:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.783+0000 7efea8a00700 1 -- 172.21.15.12:0/373360776 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c0036a0 con 0x7efeac15da50 2024-03-29T11:12:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.784+0000 7efeb1000700 1 -- 172.21.15.12:0/373360776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac15da50 msgr2=0x7efeac0ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.784+0000 7efeb1000700 1 --2- 172.21.15.12:0/373360776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac15da50 0x7efeac0ce730 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7efeac0740a0 tx=0x7efe9c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.784+0000 7efeb1000700 1 -- 172.21.15.12:0/373360776 shutdown_connections 2024-03-29T11:12:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.784+0000 7efeb1000700 1 --2- 172.21.15.12:0/373360776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac15da50 0x7efeac0ce730 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.784+0000 7efeb1000700 1 --2- 172.21.15.12:0/373360776 >> v2:172.21.15.159:3300/0 conn(0x7efeac0cec70 0x7efeac0d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.784+0000 7efeb1000700 1 -- 172.21.15.12:0/373360776 >> 172.21.15.12:0/373360776 conn(0x7efeac0666a0 msgr2=0x7efeac066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.784+0000 7efeb1000700 1 -- 172.21.15.12:0/373360776 shutdown_connections 2024-03-29T11:12:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.784+0000 7efeb1000700 1 -- 172.21.15.12:0/373360776 wait complete. 2024-03-29T11:12:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.786+0000 7efeb1000700 1 Processor -- start 2024-03-29T11:12:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.786+0000 7efeb1000700 1 -- start start 2024-03-29T11:12:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.787+0000 7efeb1000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeac0cec70 0x7efeac146390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.787+0000 7efeb1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac0d0b20 0x7efeac0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.787+0000 7efeb1000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efeac15da50 0x7efeac15ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.787+0000 7efeb1000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac143760 con 0x7efeac0d0b20 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.787+0000 7efeb1000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac0cb000 con 0x7efeac15da50 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.787+0000 7efeb1000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeac1439b0 con 0x7efeac0cec70 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.787+0000 7efea9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac0d0b20 0x7efeac0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.787+0000 7efea9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac0d0b20 0x7efeac0caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57574/0 (socket says 172.21.15.12:57574) 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.787+0000 7efea9400700 1 -- 172.21.15.12:0/333354694 learned_addr learned my addr 172.21.15.12:0/333354694 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.788+0000 7efea9e00700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeac0cec70 0x7efeac146390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.788+0000 7efeaa800700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efeac15da50 0x7efeac15ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.788+0000 7efea9400700 1 -- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeac0cec70 msgr2=0x7efeac146390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.788+0000 7efea9400700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeac0cec70 0x7efeac146390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.788+0000 7efea9400700 1 -- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efeac15da50 msgr2=0x7efeac15ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.788+0000 7efea9400700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efeac15da50 0x7efeac15ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.788+0000 7efea9400700 1 -- 172.21.15.12:0/333354694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c009550 con 0x7efeac0d0b20 2024-03-29T11:12:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.788+0000 7efea9400700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac0d0b20 0x7efeac0caac0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7efe9800e7e0 tx=0x7efe9800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.788+0000 7efea2000700 1 -- 172.21.15.12:0/333354694 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9800c070 con 0x7efeac0d0b20 2024-03-29T11:12:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.789+0000 7efea2000700 1 -- 172.21.15.12:0/333354694 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe98013610 con 0x7efeac0d0b20 2024-03-29T11:12:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.789+0000 7efeb1000700 1 -- 172.21.15.12:0/333354694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeac143c90 con 0x7efeac0d0b20 2024-03-29T11:12:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.789+0000 7efea2000700 1 -- 172.21.15.12:0/333354694 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98013980 con 0x7efeac0d0b20 2024-03-29T11:12:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.789+0000 7efeb1000700 1 -- 172.21.15.12:0/333354694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac144250 con 0x7efeac0d0b20 2024-03-29T11:12:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.790+0000 7efeb1000700 1 -- 172.21.15.12:0/333354694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac1445c0 con 0x7efeac0d0b20 2024-03-29T11:12:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.791+0000 7efea2000700 1 -- 172.21.15.12:0/333354694 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7efe98013ae0 con 0x7efeac0d0b20 2024-03-29T11:12:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.792+0000 7efea2000700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe8c06adb0 0x7efe8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.792+0000 7efea2000700 1 -- 172.21.15.12:0/333354694 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7efe98089520 con 0x7efeac0d0b20 2024-03-29T11:12:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.792+0000 7efea9e00700 1 -- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe8c06adb0 msgr2=0x7efe8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.792+0000 7efea9e00700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe8c06adb0 0x7efe8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:06.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.796+0000 7efea2000700 1 -- 172.21.15.12:0/333354694 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efe980542c0 con 0x7efeac0d0b20 2024-03-29T11:12:06.780 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:06.803+0000 7f8088449240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:12:06.813 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:06.834+0000 7f49d499b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:12:06.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.993+0000 7efea9e00700 1 -- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe8c06adb0 msgr2=0x7efe8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:06.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:06.993+0000 7efea9e00700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe8c06adb0 0x7efe8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:06.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.016+0000 7efeb1000700 1 -- 172.21.15.12:0/333354694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efeac067720 con 0x7efeac0d0b20 2024-03-29T11:12:06.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.017+0000 7efea2000700 1 -- 172.21.15.12:0/333354694 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+16677 (secure 0 0 0) 0x7efe980578e0 con 0x7efeac0d0b20 2024-03-29T11:12:06.995 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:06.995 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":178,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:59.862985+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:11:41.110455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-29T11:11:41.640016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1870707179":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/499386558":"2024-03-30T11:11:59.862930+0000","172.21.15.159:6833/43444":"2024-03-30T11:11:59.862930+0000","172.21.15.159:6832/43444":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/3131465277":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/3944320011":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/998093551":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/618363904":"2024-03-30T11:11:59.862930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:06.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 -- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe8c06adb0 msgr2=0x7efe8c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:06.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe8c06adb0 0x7efe8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 -- 172.21.15.12:0/333354694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac0d0b20 msgr2=0x7efeac0caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac0d0b20 0x7efeac0caac0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7efe9800e7e0 tx=0x7efe9800eaf0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 -- 172.21.15.12:0/333354694 shutdown_connections 2024-03-29T11:12:06.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7efe8c06adb0 0x7efe8c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efeac15da50 0x7efeac15ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeac0cec70 0x7efeac146390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 --2- 172.21.15.12:0/333354694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efeac0d0b20 0x7efeac0caac0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:06.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 -- 172.21.15.12:0/333354694 >> 172.21.15.12:0/333354694 conn(0x7efeac0666a0 msgr2=0x7efeac0b32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:06.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 -- 172.21.15.12:0/333354694 shutdown_connections 2024-03-29T11:12:06.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.020+0000 7efeb1000700 1 -- 172.21.15.12:0/333354694 wait complete. 2024-03-29T11:12:07.017 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:11:41.110455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '176', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:11:41.640016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '177', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:12:07.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:12:07.039 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:07.061+0000 7f49d499b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:12:07.040 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:07.062+0000 7f8088449240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:12:07.230 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:07.253+0000 7f8088449240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:12:07.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.268+0000 7f05e4e00700 1 Processor -- start 2024-03-29T11:12:07.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.268+0000 7f05e4e00700 1 -- start start 2024-03-29T11:12:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.269+0000 7f05e4e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05e01611b0 0x7f05e015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.269+0000 7f05e4e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f05e00cee10 0x7f05e0161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.269+0000 7f05e4e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e00ce270 con 0x7f05e015f4d0 2024-03-29T11:12:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.269+0000 7f05e4e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05e015b600 con 0x7f05e01611b0 2024-03-29T11:12:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.269+0000 7f05e4e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05e00d2760 con 0x7f05e00cee10 2024-03-29T11:12:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.270+0000 7f05dde00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05e01611b0 0x7f05e015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.270+0000 7f05dde00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05e01611b0 0x7f05e015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57582/0 (socket says 172.21.15.12:57582) 2024-03-29T11:12:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.270+0000 7f05dde00700 1 -- 172.21.15.12:0/2355115210 learned_addr learned my addr 172.21.15.12:0/2355115210 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.270+0000 7f05de800700 1 --2- 172.21.15.12:0/2355115210 >> v2:172.21.15.159:3300/0 conn(0x7f05e00cee10 0x7f05e0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.270+0000 7f05dde00700 1 -- 172.21.15.12:0/2355115210 >> v2:172.21.15.159:3300/0 conn(0x7f05e00cee10 msgr2=0x7f05e0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.270+0000 7f05dde00700 1 --2- 172.21.15.12:0/2355115210 >> v2:172.21.15.159:3300/0 conn(0x7f05e00cee10 0x7f05e0161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.270+0000 7f05dde00700 1 -- 172.21.15.12:0/2355115210 >> v1:172.21.15.159:6789/0 conn(0x7f05e015f4d0 legacy=0x7f05e00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.270+0000 7f05dde00700 1 -- 172.21.15.12:0/2355115210 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e00d2a40 con 0x7f05e01611b0 2024-03-29T11:12:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.271+0000 7f05dde00700 1 --2- 172.21.15.12:0/2355115210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05e01611b0 0x7f05e015ef90 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f05e00d2f60 tx=0x7f05d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2882e89cb321e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.271+0000 7f05dca00700 1 -- 172.21.15.12:0/2355115210 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d000b870 con 0x7f05e01611b0 2024-03-29T11:12:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.271+0000 7f05dca00700 1 -- 172.21.15.12:0/2355115210 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d000b9d0 con 0x7f05e01611b0 2024-03-29T11:12:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.271+0000 7f05dca00700 1 -- 172.21.15.12:0/2355115210 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d000bd60 con 0x7f05e01611b0 2024-03-29T11:12:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.271+0000 7f05e4e00700 1 -- 172.21.15.12:0/2355115210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05e01611b0 msgr2=0x7f05e015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.271+0000 7f05e4e00700 1 --2- 172.21.15.12:0/2355115210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05e01611b0 0x7f05e015ef90 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f05e00d2f60 tx=0x7f05d0009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.271+0000 7f05e4e00700 1 -- 172.21.15.12:0/2355115210 shutdown_connections 2024-03-29T11:12:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.271+0000 7f05e4e00700 1 --2- 172.21.15.12:0/2355115210 >> v2:172.21.15.159:3300/0 conn(0x7f05e00cee10 0x7f05e0161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.271+0000 7f05e4e00700 1 --2- 172.21.15.12:0/2355115210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05e01611b0 0x7f05e015ef90 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.271+0000 7f05e4e00700 1 -- 172.21.15.12:0/2355115210 >> 172.21.15.12:0/2355115210 conn(0x7f05e008aff0 msgr2=0x7f05e00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.272+0000 7f05e4e00700 1 -- 172.21.15.12:0/2355115210 shutdown_connections 2024-03-29T11:12:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.272+0000 7f05e4e00700 1 -- 172.21.15.12:0/2355115210 wait complete. 2024-03-29T11:12:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.273+0000 7f05e4e00700 1 Processor -- start 2024-03-29T11:12:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.274+0000 7f05e4e00700 1 -- start start 2024-03-29T11:12:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.274+0000 7f05e4e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05e00cee10 0x7f05e0152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.274+0000 7f05e4e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05e015f4d0 0x7f05e00cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05e4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05e01611b0 0x7f05e00c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05e4e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e00cc5f0 con 0x7f05e01611b0 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05e4e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e00cc030 con 0x7f05e00cee10 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05e4e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e00cc310 con 0x7f05e015f4d0 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05dde00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05e00cee10 0x7f05e0152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05de800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05e01611b0 0x7f05e00c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05dde00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05e00cee10 0x7f05e0152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:49180/0 (socket says 172.21.15.12:49180) 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05dde00700 1 -- 172.21.15.12:0/3178273289 learned_addr learned my addr 172.21.15.12:0/3178273289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05dd400700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05e015f4d0 0x7f05e00cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05de800700 1 -- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05e015f4d0 msgr2=0x7f05e00cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05de800700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05e015f4d0 0x7f05e00cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05de800700 1 -- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05e00cee10 msgr2=0x7f05e0152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05de800700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05e00cee10 0x7f05e0152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.275+0000 7f05de800700 1 -- 172.21.15.12:0/3178273289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d0009550 con 0x7f05e01611b0 2024-03-29T11:12:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.276+0000 7f05de800700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05e01611b0 0x7f05e00c93c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f05d8007d90 tx=0x7f05d800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.276+0000 7f05d6000700 1 -- 172.21.15.12:0/3178273289 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d8018070 con 0x7f05e01611b0 2024-03-29T11:12:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.276+0000 7f05d6000700 1 -- 172.21.15.12:0/3178273289 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d800eb30 con 0x7f05e01611b0 2024-03-29T11:12:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.276+0000 7f05e4e00700 1 -- 172.21.15.12:0/3178273289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e00c7100 con 0x7f05e01611b0 2024-03-29T11:12:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.276+0000 7f05d6000700 1 -- 172.21.15.12:0/3178273289 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d8003a20 con 0x7f05e01611b0 2024-03-29T11:12:07.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.276+0000 7f05e4e00700 1 -- 172.21.15.12:0/3178273289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e01529b0 con 0x7f05e01611b0 2024-03-29T11:12:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.277+0000 7f05e4e00700 1 -- 172.21.15.12:0/3178273289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e00c7620 con 0x7f05e01611b0 2024-03-29T11:12:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.278+0000 7f05d6000700 1 -- 172.21.15.12:0/3178273289 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f05d8014420 con 0x7f05e01611b0 2024-03-29T11:12:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.279+0000 7f05d6000700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05c006adb0 0x7f05c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.279+0000 7f05d6000700 1 -- 172.21.15.12:0/3178273289 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f05d8089620 con 0x7f05e01611b0 2024-03-29T11:12:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.279+0000 7f05dde00700 1 -- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05c006adb0 msgr2=0x7f05c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.280+0000 7f05dde00700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05c006adb0 0x7f05c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.283+0000 7f05d6000700 1 -- 172.21.15.12:0/3178273289 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f05e00c7620 con 0x7f05e01611b0 2024-03-29T11:12:07.315 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:07.337+0000 7f49d499b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:12:07.324 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:07.347+0000 7f8088449240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.481+0000 7f05dde00700 1 -- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05c006adb0 msgr2=0x7f05c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.481+0000 7f05dde00700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05c006adb0 0x7f05c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.496+0000 7f05e4e00700 1 -- 172.21.15.12:0/3178273289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05e00c7620 con 0x7f05e01611b0 2024-03-29T11:12:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.497+0000 7f05d6000700 1 -- 172.21.15.12:0/3178273289 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v178) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05e00c7620 con 0x7f05e01611b0 2024-03-29T11:12:07.475 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:12:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.499+0000 7f05e4e00700 1 -- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05c006adb0 msgr2=0x7f05c006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.499+0000 7f05e4e00700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05c006adb0 0x7f05c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.499+0000 7f05e4e00700 1 -- 172.21.15.12:0/3178273289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05e01611b0 msgr2=0x7f05e00c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.499+0000 7f05e4e00700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05e01611b0 0x7f05e00c93c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f05d8007d90 tx=0x7f05d800e3c0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.500+0000 7f05e4e00700 1 -- 172.21.15.12:0/3178273289 shutdown_connections 2024-03-29T11:12:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.500+0000 7f05e4e00700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f05c006adb0 0x7f05c006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.500+0000 7f05e4e00700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05e01611b0 0x7f05e00c93c0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.500+0000 7f05e4e00700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05e00cee10 0x7f05e0152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.500+0000 7f05e4e00700 1 --2- 172.21.15.12:0/3178273289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05e015f4d0 0x7f05e00cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.500+0000 7f05e4e00700 1 -- 172.21.15.12:0/3178273289 >> 172.21.15.12:0/3178273289 conn(0x7f05e008aff0 msgr2=0x7f05e006c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.500+0000 7f05e4e00700 1 -- 172.21.15.12:0/3178273289 shutdown_connections 2024-03-29T11:12:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.500+0000 7f05e4e00700 1 -- 172.21.15.12:0/3178273289 wait complete. 2024-03-29T11:12:07.496 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:12:07.522 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:07.544+0000 7f49d499b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:12:07.630 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:07.652+0000 7f49d499b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.733+0000 7ff2a5600700 1 Processor -- start 2024-03-29T11:12:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.734+0000 7ff2a5600700 1 -- start start 2024-03-29T11:12:07.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.735+0000 7ff2a5600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 0x7ff2a00b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.735+0000 7ff2a5600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a008b6d0 con 0x7ff2a0161200 2024-03-29T11:12:07.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.735+0000 7ff2a5600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a00cf6a0 con 0x7ff2a015fe10 2024-03-29T11:12:07.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.735+0000 7ff2a5600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00d2240 con 0x7ff2a00b1190 2024-03-29T11:12:07.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.735+0000 7ff29e000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff2a0161200 0x7ff2a00d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40348/0 (socket says 172.21.15.12:40348) 2024-03-29T11:12:07.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.735+0000 7ff29e000700 1 -- 172.21.15.12:0/626044396 learned_addr learned my addr 172.21.15.12:0/626044396 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:07.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.735+0000 7ff29f400700 1 --2- 172.21.15.12:0/626044396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 0x7ff2a00b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:07.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.735+0000 7ff29d600700 1 -- 172.21.15.12:0/626044396 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229266820 0 0) 0x7ff2a008b6d0 con 0x7ff2a0161200 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.736+0000 7ff29d600700 1 -- 172.21.15.12:0/626044396 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2840035b0 con 0x7ff2a0161200 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.736+0000 7ff29d600700 1 -- 172.21.15.12:0/626044396 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834736794 0 0) 0x7ff2a00cf6a0 con 0x7ff2a015fe10 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.736+0000 7ff29d600700 1 -- 172.21.15.12:0/626044396 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a008b6d0 con 0x7ff2a015fe10 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.736+0000 7ff29f400700 1 -- 172.21.15.12:0/626044396 >> v1:172.21.15.159:6789/0 conn(0x7ff2a015fe10 legacy=0x7ff2a00d1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.736+0000 7ff29f400700 1 -- 172.21.15.12:0/626044396 >> v1:172.21.15.12:6789/0 conn(0x7ff2a0161200 legacy=0x7ff2a00d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.736+0000 7ff29f400700 1 -- 172.21.15.12:0/626044396 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00b2bf0 con 0x7ff2a00b1190 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.736+0000 7ff29f400700 1 --2- 172.21.15.12:0/626044396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 0x7ff2a00b24f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff298009dc0 tx=0x7ff29800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be2511e26ee0e334 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.736+0000 7ff29d600700 1 -- 172.21.15.12:0/626044396 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29800e070 con 0x7ff2a00b1190 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.736+0000 7ff29d600700 1 -- 172.21.15.12:0/626044396 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29800bb20 con 0x7ff2a00b1190 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.736+0000 7ff29d600700 1 -- 172.21.15.12:0/626044396 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298012430 con 0x7ff2a00b1190 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.737+0000 7ff2a5600700 1 -- 172.21.15.12:0/626044396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 msgr2=0x7ff2a00b24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.737+0000 7ff2a5600700 1 --2- 172.21.15.12:0/626044396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 0x7ff2a00b24f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff298009dc0 tx=0x7ff29800b330 comp rx=0 tx=0).stop 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.737+0000 7ff2a5600700 1 -- 172.21.15.12:0/626044396 shutdown_connections 2024-03-29T11:12:07.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.737+0000 7ff2a5600700 1 --2- 172.21.15.12:0/626044396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 0x7ff2a00b24f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.737+0000 7ff2a5600700 1 -- 172.21.15.12:0/626044396 >> 172.21.15.12:0/626044396 conn(0x7ff2a008aff0 msgr2=0x7ff2a00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:07.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.737+0000 7ff2a5600700 1 -- 172.21.15.12:0/626044396 shutdown_connections 2024-03-29T11:12:07.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.737+0000 7ff2a5600700 1 -- 172.21.15.12:0/626044396 wait complete. 2024-03-29T11:12:07.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.739+0000 7ff2a5600700 1 Processor -- start 2024-03-29T11:12:07.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.739+0000 7ff2a5600700 1 -- start start 2024-03-29T11:12:07.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff2a5600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 0x7ff2a006c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff2a5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2a015fe10 0x7ff2a01531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff2a5600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2a0161200 0x7ff2a00c0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff2a5600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00c0d70 con 0x7ff2a015fe10 2024-03-29T11:12:07.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff2a5600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00c65f0 con 0x7ff2a0161200 2024-03-29T11:12:07.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff2a5600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00c6310 con 0x7ff2a00b1190 2024-03-29T11:12:07.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff29e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2a015fe10 0x7ff2a01531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff29e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2a015fe10 0x7ff2a01531b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57610/0 (socket says 172.21.15.12:57610) 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff29e000700 1 -- 172.21.15.12:0/555635027 learned_addr learned my addr 172.21.15.12:0/555635027 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff29ea00700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 0x7ff2a006c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.740+0000 7ff29f400700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2a0161200 0x7ff2a00c0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff29e000700 1 -- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 msgr2=0x7ff2a006c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff29e000700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 0x7ff2a006c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff29e000700 1 -- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2a0161200 msgr2=0x7ff2a00c0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff29e000700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2a0161200 0x7ff2a00c0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff29e000700 1 -- 172.21.15.12:0/555635027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff298009a70 con 0x7ff2a015fe10 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff29e000700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2a015fe10 0x7ff2a01531b0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7ff28c005910 tx=0x7ff28c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff296a00700 1 -- 172.21.15.12:0/555635027 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c005d70 con 0x7ff2a015fe10 2024-03-29T11:12:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff296a00700 1 -- 172.21.15.12:0/555635027 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28c0070d0 con 0x7ff2a015fe10 2024-03-29T11:12:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff2a5600700 1 -- 172.21.15.12:0/555635027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a00b61e0 con 0x7ff2a015fe10 2024-03-29T11:12:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff296a00700 1 -- 172.21.15.12:0/555635027 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c007440 con 0x7ff2a015fe10 2024-03-29T11:12:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.741+0000 7ff2a5600700 1 -- 172.21.15.12:0/555635027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a0153a90 con 0x7ff2a015fe10 2024-03-29T11:12:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.742+0000 7ff2a5600700 1 -- 172.21.15.12:0/555635027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a00ca440 con 0x7ff2a015fe10 2024-03-29T11:12:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.743+0000 7ff296a00700 1 -- 172.21.15.12:0/555635027 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7ff28c0075a0 con 0x7ff2a015fe10 2024-03-29T11:12:07.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.744+0000 7ff296a00700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff28006adb0 0x7ff28006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:07.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.744+0000 7ff296a00700 1 -- 172.21.15.12:0/555635027 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7ff28c088a30 con 0x7ff2a015fe10 2024-03-29T11:12:07.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.744+0000 7ff29ea00700 1 -- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff28006adb0 msgr2=0x7ff28006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.744+0000 7ff29ea00700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff28006adb0 0x7ff28006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.748+0000 7ff296a00700 1 -- 172.21.15.12:0/555635027 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff28c053900 con 0x7ff2a015fe10 2024-03-29T11:12:07.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.945+0000 7ff29ea00700 1 -- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff28006adb0 msgr2=0x7ff28006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:07.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.945+0000 7ff29ea00700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff28006adb0 0x7ff28006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:07.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.968+0000 7ff2a5600700 1 -- 172.21.15.12:0/555635027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff2a0067720 con 0x7ff2a015fe10 2024-03-29T11:12:07.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.968+0000 7ff296a00700 1 -- 172.21.15.12:0/555635027 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v178) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff28c056f20 con 0x7ff2a015fe10 2024-03-29T11:12:07.947 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:12:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 -- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff28006adb0 msgr2=0x7ff28006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff28006adb0 0x7ff28006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 -- 172.21.15.12:0/555635027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2a015fe10 msgr2=0x7ff2a01531b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2a015fe10 0x7ff2a01531b0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7ff28c005910 tx=0x7ff28c00c7b0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 -- 172.21.15.12:0/555635027 shutdown_connections 2024-03-29T11:12:07.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff28006adb0 0x7ff28006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2a0161200 0x7ff2a00c0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2a00b1190 0x7ff2a006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 --2- 172.21.15.12:0/555635027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2a015fe10 0x7ff2a01531b0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:07.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 -- 172.21.15.12:0/555635027 >> 172.21.15.12:0/555635027 conn(0x7ff2a008aff0 msgr2=0x7ff2a0158ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:07.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 -- 172.21.15.12:0/555635027 shutdown_connections 2024-03-29T11:12:07.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:07.971+0000 7ff2a5600700 1 -- 172.21.15.12:0/555635027 wait complete. 2024-03-29T11:12:07.968 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:12:08.061 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:08.083+0000 7f8088449240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:12:08.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.233+0000 7fa215600700 1 Processor -- start 2024-03-29T11:12:08.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.233+0000 7fa215600700 1 -- start start 2024-03-29T11:12:08.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.234+0000 7fa215600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa21015d9d0 0x7fa2100d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.234+0000 7fa215600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2100d2540 0x7fa2100ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.234+0000 7fa215600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa210161420 con 0x7fa2100d16a0 2024-03-29T11:12:08.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.234+0000 7fa215600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21015edc0 con 0x7fa21015d9d0 2024-03-29T11:12:08.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.234+0000 7fa215600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa210161700 con 0x7fa2100d2540 2024-03-29T11:12:08.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.234+0000 7fa20ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa21015d9d0 0x7fa2100d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:08.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.234+0000 7fa20ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa21015d9d0 0x7fa2100d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57616/0 (socket says 172.21.15.12:57616) 2024-03-29T11:12:08.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.234+0000 7fa20ea00700 1 -- 172.21.15.12:0/393859653 learned_addr learned my addr 172.21.15.12:0/393859653 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:08.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.234+0000 7fa20f400700 1 --2- 172.21.15.12:0/393859653 >> v2:172.21.15.159:3300/0 conn(0x7fa2100d2540 0x7fa2100ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:08.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.235+0000 7fa20ea00700 1 -- 172.21.15.12:0/393859653 >> v2:172.21.15.159:3300/0 conn(0x7fa2100d2540 msgr2=0x7fa2100ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:08.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.235+0000 7fa20ea00700 1 --2- 172.21.15.12:0/393859653 >> v2:172.21.15.159:3300/0 conn(0x7fa2100d2540 0x7fa2100ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.235+0000 7fa20ea00700 1 -- 172.21.15.12:0/393859653 >> v1:172.21.15.159:6789/0 conn(0x7fa2100d16a0 legacy=0x7fa2100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.235+0000 7fa20ea00700 1 -- 172.21.15.12:0/393859653 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2101619e0 con 0x7fa21015d9d0 2024-03-29T11:12:08.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.235+0000 7fa20ea00700 1 --2- 172.21.15.12:0/393859653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 0x7fa2100d3c30 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa21008ca10 tx=0x7fa200009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e384bf36b23df20 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:08.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.235+0000 7fa20d600700 1 -- 172.21.15.12:0/393859653 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200011830 con 0x7fa21015d9d0 2024-03-29T11:12:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.235+0000 7fa20d600700 1 -- 172.21.15.12:0/393859653 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa200011990 con 0x7fa21015d9d0 2024-03-29T11:12:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.235+0000 7fa20d600700 1 -- 172.21.15.12:0/393859653 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200011d20 con 0x7fa21015d9d0 2024-03-29T11:12:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.236+0000 7fa215600700 1 -- 172.21.15.12:0/393859653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 msgr2=0x7fa2100d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.236+0000 7fa215600700 1 --2- 172.21.15.12:0/393859653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 0x7fa2100d3c30 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa21008ca10 tx=0x7fa200009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.236+0000 7fa215600700 1 -- 172.21.15.12:0/393859653 shutdown_connections 2024-03-29T11:12:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.236+0000 7fa215600700 1 --2- 172.21.15.12:0/393859653 >> v2:172.21.15.159:3300/0 conn(0x7fa2100d2540 0x7fa2100ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.236+0000 7fa215600700 1 --2- 172.21.15.12:0/393859653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 0x7fa2100d3c30 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.236+0000 7fa215600700 1 -- 172.21.15.12:0/393859653 >> 172.21.15.12:0/393859653 conn(0x7fa21008aff0 msgr2=0x7fa2100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.236+0000 7fa215600700 1 -- 172.21.15.12:0/393859653 shutdown_connections 2024-03-29T11:12:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.236+0000 7fa215600700 1 -- 172.21.15.12:0/393859653 wait complete. 2024-03-29T11:12:08.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.238+0000 7fa215600700 1 Processor -- start 2024-03-29T11:12:08.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.238+0000 7fa215600700 1 -- start start 2024-03-29T11:12:08.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa215600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2100d16a0 0x7fa2101557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa215600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100d2540 0x7fa2100b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa215600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 0x7fa2101551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa215600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100b4560 con 0x7fa21015d9d0 2024-03-29T11:12:08.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa215600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100b4840 con 0x7fa2100d2540 2024-03-29T11:12:08.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa215600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100b4b20 con 0x7fa2100d16a0 2024-03-29T11:12:08.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa20f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 0x7fa2101551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:08.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa20f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 0x7fa2101551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57618/0 (socket says 172.21.15.12:57618) 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa20e000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100d2540 0x7fa2100b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa20f400700 1 -- 172.21.15.12:0/1011587319 learned_addr learned my addr 172.21.15.12:0/1011587319 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa20ea00700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2100d16a0 0x7fa2101557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa20f400700 1 -- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2100d16a0 msgr2=0x7fa2101557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.239+0000 7fa20f400700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2100d16a0 0x7fa2101557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.240+0000 7fa20f400700 1 -- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100d2540 msgr2=0x7fa2100b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.240+0000 7fa20f400700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100d2540 0x7fa2100b5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.240+0000 7fa20f400700 1 -- 172.21.15.12:0/1011587319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa200009550 con 0x7fa21015d9d0 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.240+0000 7fa20f400700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 0x7fa2101551b0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa20800efc0 tx=0x7fa20800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.240+0000 7fa206a00700 1 -- 172.21.15.12:0/1011587319 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20800cc90 con 0x7fa21015d9d0 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.240+0000 7fa206a00700 1 -- 172.21.15.12:0/1011587319 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20800f040 con 0x7fa21015d9d0 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.240+0000 7fa206a00700 1 -- 172.21.15.12:0/1011587319 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2080146c0 con 0x7fa21015d9d0 2024-03-29T11:12:08.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.240+0000 7fa215600700 1 -- 172.21.15.12:0/1011587319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2100b4e00 con 0x7fa21015d9d0 2024-03-29T11:12:08.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.240+0000 7fa215600700 1 -- 172.21.15.12:0/1011587319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa210155d80 con 0x7fa21015d9d0 2024-03-29T11:12:08.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.241+0000 7fa215600700 1 -- 172.21.15.12:0/1011587319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2100d1ca0 con 0x7fa21015d9d0 2024-03-29T11:12:08.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.242+0000 7fa206a00700 1 -- 172.21.15.12:0/1011587319 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fa208014820 con 0x7fa21015d9d0 2024-03-29T11:12:08.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.244+0000 7fa206a00700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1f006adb0 0x7fa1f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.244+0000 7fa206a00700 1 -- 172.21.15.12:0/1011587319 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fa20808a340 con 0x7fa21015d9d0 2024-03-29T11:12:08.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.244+0000 7fa20ea00700 1 -- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1f006adb0 msgr2=0x7fa1f006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:08.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.244+0000 7fa20ea00700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1f006adb0 0x7fa1f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:08.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.248+0000 7fa206a00700 1 -- 172.21.15.12:0/1011587319 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2080550d0 con 0x7fa21015d9d0 2024-03-29T11:12:08.275 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:08.297+0000 7f8088449240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:12:08.412 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:08.434+0000 7f49d499b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:12:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.445+0000 7fa20ea00700 1 -- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1f006adb0 msgr2=0x7fa1f006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.445+0000 7fa20ea00700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1f006adb0 0x7fa1f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:08.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.461+0000 7fa215600700 1 -- 172.21.15.12:0/1011587319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa2100b5320 con 0x7fa21015d9d0 2024-03-29T11:12:08.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.461+0000 7fa206a00700 1 -- 172.21.15.12:0/1011587319 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v178) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa208009ca0 con 0x7fa21015d9d0 2024-03-29T11:12:08.439 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:12:08.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 -- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1f006adb0 msgr2=0x7fa1f006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:08.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1f006adb0 0x7fa1f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 -- 172.21.15.12:0/1011587319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 msgr2=0x7fa2101551b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 0x7fa2101551b0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa20800efc0 tx=0x7fa20800c3a0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 -- 172.21.15.12:0/1011587319 shutdown_connections 2024-03-29T11:12:08.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1f006adb0 0x7fa1f006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa21015d9d0 0x7fa2101551b0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2100d16a0 0x7fa2101557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 --2- 172.21.15.12:0/1011587319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100d2540 0x7fa2100b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 -- 172.21.15.12:0/1011587319 >> 172.21.15.12:0/1011587319 conn(0x7fa21008aff0 msgr2=0x7fa2100d1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:08.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 -- 172.21.15.12:0/1011587319 shutdown_connections 2024-03-29T11:12:08.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.464+0000 7fa215600700 1 -- 172.21.15.12:0/1011587319 wait complete. 2024-03-29T11:12:08.460 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:12:08.480 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:08.503+0000 7f8088449240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:08.641 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:08.664+0000 7f49d499b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:12:08.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.684+0000 7f2fcf600700 1 Processor -- start 2024-03-29T11:12:08.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.684+0000 7f2fcf600700 1 -- start start 2024-03-29T11:12:08.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.685+0000 7f2fcf600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 0x7f2fc80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.685+0000 7f2fcf600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc80ceda0 con 0x7f2fc80d18b0 2024-03-29T11:12:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.685+0000 7f2fcf600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc81612f0 con 0x7f2fc8160f10 2024-03-29T11:12:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.685+0000 7f2fcf600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80d1d30 con 0x7f2fc8161ca0 2024-03-29T11:12:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.686+0000 7f2fc7e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2fc80d18b0 0x7f2fc8161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40352/0 (socket says 172.21.15.12:40352) 2024-03-29T11:12:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.686+0000 7f2fc7e00700 1 -- 172.21.15.12:0/4210806380 learned_addr learned my addr 172.21.15.12:0/4210806380 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.686+0000 7f2fcd600700 1 --2- 172.21.15.12:0/4210806380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 0x7f2fc80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.686+0000 7f2fc7400700 1 -- 172.21.15.12:0/4210806380 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066261530 0 0) 0x7f2fc80ceda0 con 0x7f2fc80d18b0 2024-03-29T11:12:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.686+0000 7f2fc7400700 1 -- 172.21.15.12:0/4210806380 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb00035b0 con 0x7f2fc80d18b0 2024-03-29T11:12:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.686+0000 7f2fc7400700 1 -- 172.21.15.12:0/4210806380 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994779210 0 0) 0x7f2fc81612f0 con 0x7f2fc8160f10 2024-03-29T11:12:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.686+0000 7f2fc7400700 1 -- 172.21.15.12:0/4210806380 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc80ceda0 con 0x7f2fc8160f10 2024-03-29T11:12:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.686+0000 7f2fcd600700 1 -- 172.21.15.12:0/4210806380 >> v1:172.21.15.159:6789/0 conn(0x7f2fc8160f10 legacy=0x7f2fc80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.686+0000 7f2fcd600700 1 -- 172.21.15.12:0/4210806380 >> v1:172.21.15.12:6789/0 conn(0x7f2fc80d18b0 legacy=0x7f2fc8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.686+0000 7f2fcd600700 1 -- 172.21.15.12:0/4210806380 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80ce470 con 0x7f2fc8161ca0 2024-03-29T11:12:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.687+0000 7f2fcd600700 1 --2- 172.21.15.12:0/4210806380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 0x7f2fc80cf230 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009dc0 tx=0x7f2fc000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=647f90bfd3f3852f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.687+0000 7f2fc7400700 1 -- 172.21.15.12:0/4210806380 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc000e070 con 0x7f2fc8161ca0 2024-03-29T11:12:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.687+0000 7f2fc7400700 1 -- 172.21.15.12:0/4210806380 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc000bb20 con 0x7f2fc8161ca0 2024-03-29T11:12:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.687+0000 7f2fc7400700 1 -- 172.21.15.12:0/4210806380 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0012430 con 0x7f2fc8161ca0 2024-03-29T11:12:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.688+0000 7f2fcf600700 1 -- 172.21.15.12:0/4210806380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 msgr2=0x7f2fc80cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.688+0000 7f2fcf600700 1 --2- 172.21.15.12:0/4210806380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 0x7f2fc80cf230 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009dc0 tx=0x7f2fc000b330 comp rx=0 tx=0).stop 2024-03-29T11:12:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.688+0000 7f2fcf600700 1 -- 172.21.15.12:0/4210806380 shutdown_connections 2024-03-29T11:12:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.688+0000 7f2fcf600700 1 --2- 172.21.15.12:0/4210806380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 0x7f2fc80cf230 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.688+0000 7f2fcf600700 1 -- 172.21.15.12:0/4210806380 >> 172.21.15.12:0/4210806380 conn(0x7f2fc808aff0 msgr2=0x7f2fc80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.688+0000 7f2fcf600700 1 -- 172.21.15.12:0/4210806380 shutdown_connections 2024-03-29T11:12:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.688+0000 7f2fcf600700 1 -- 172.21.15.12:0/4210806380 wait complete. 2024-03-29T11:12:08.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.690+0000 7f2fcf600700 1 Processor -- start 2024-03-29T11:12:08.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.690+0000 7f2fcf600700 1 -- start start 2024-03-29T11:12:08.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fcf600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc80d18b0 0x7f2fc815a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fcf600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc8160f10 0x7f2fc806bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fcf600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 0x7f2fc806c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fcf600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc815f7c0 con 0x7f2fc80d18b0 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fcf600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc81588b0 con 0x7f2fc8160f10 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fcf600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc815eb40 con 0x7f2fc8161ca0 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fccc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc80d18b0 0x7f2fc815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fccc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc80d18b0 0x7f2fc815a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57632/0 (socket says 172.21.15.12:57632) 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fccc00700 1 -- 172.21.15.12:0/2336459956 learned_addr learned my addr 172.21.15.12:0/2336459956 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fc7e00700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc8160f10 0x7f2fc806bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fcd600700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 0x7f2fc806c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fccc00700 1 -- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 msgr2=0x7f2fc806c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fccc00700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 0x7f2fc806c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.691+0000 7f2fccc00700 1 -- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc8160f10 msgr2=0x7f2fc806bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.692+0000 7f2fccc00700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc8160f10 0x7f2fc806bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.692+0000 7f2fccc00700 1 -- 172.21.15.12:0/2336459956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc0009a70 con 0x7f2fc80d18b0 2024-03-29T11:12:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.692+0000 7f2fccc00700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc80d18b0 0x7f2fc815a030 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00c500 tx=0x7f2fbc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.692+0000 7f2fc4c00700 1 -- 172.21.15.12:0/2336459956 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0052d0 con 0x7f2fc80d18b0 2024-03-29T11:12:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.692+0000 7f2fc4c00700 1 -- 172.21.15.12:0/2336459956 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc005450 con 0x7f2fc80d18b0 2024-03-29T11:12:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.692+0000 7f2fc4c00700 1 -- 172.21.15.12:0/2336459956 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0038c0 con 0x7f2fc80d18b0 2024-03-29T11:12:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.692+0000 7f2fcf600700 1 -- 172.21.15.12:0/2336459956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc815ee20 con 0x7f2fc80d18b0 2024-03-29T11:12:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.692+0000 7f2fcf600700 1 -- 172.21.15.12:0/2336459956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc815f3e0 con 0x7f2fc80d18b0 2024-03-29T11:12:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.693+0000 7f2fcf600700 1 -- 172.21.15.12:0/2336459956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc815f570 con 0x7f2fc80d18b0 2024-03-29T11:12:08.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.696+0000 7f2fc4c00700 1 -- 172.21.15.12:0/2336459956 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f2fbc003a20 con 0x7f2fc80d18b0 2024-03-29T11:12:08.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.697+0000 7f2fc4c00700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fa806adb0 0x7f2fa806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:08.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.697+0000 7f2fc4c00700 1 -- 172.21.15.12:0/2336459956 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f2fbc0889b0 con 0x7f2fc80d18b0 2024-03-29T11:12:08.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.697+0000 7f2fc7e00700 1 -- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fa806adb0 msgr2=0x7f2fa806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:08.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.697+0000 7f2fc7e00700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fa806adb0 0x7f2fa806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:08.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.702+0000 7f2fc4c00700 1 -- 172.21.15.12:0/2336459956 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2fbc0538f0 con 0x7f2fc80d18b0 2024-03-29T11:12:08.868 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:08.890+0000 7f49d499b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.898+0000 7f2fc7e00700 1 -- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fa806adb0 msgr2=0x7f2fa806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.898+0000 7f2fc7e00700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fa806adb0 0x7f2fa806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:08.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.962+0000 7f2fcf600700 1 -- 172.21.15.12:0/2336459956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fc8067720 con 0x7f2fc80d18b0 2024-03-29T11:12:08.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.962+0000 7f2fc4c00700 1 -- 172.21.15.12:0/2336459956 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f2fbc017090 con 0x7f2fc80d18b0 2024-03-29T11:12:08.940 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:08.940 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/652112625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":652112625},{"type":"v1","addr":"172.21.15.159:6836","nonce":652112625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9166,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2803735172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2803735172},{"type":"v1","addr":"172.21.15.159:6837","nonce":2803735172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9444,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3698805288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3698805288},{"type":"v1","addr":"172.21.15.12:6833","nonce":3698805288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:11:42.673965+0000","modified":"2024-03-29T11:11:46.149885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9450":{"gid":9450,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/2066631988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2066631988},{"type":"v1","addr":"172.21.15.12:6835","nonce":2066631988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-29T11:12:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 -- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fa806adb0 msgr2=0x7f2fa806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fa806adb0 0x7f2fa806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 -- 172.21.15.12:0/2336459956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc80d18b0 msgr2=0x7f2fc815a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc80d18b0 0x7f2fc815a030 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00c500 tx=0x7f2fbc00c810 comp rx=0 tx=0).stop 2024-03-29T11:12:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 -- 172.21.15.12:0/2336459956 shutdown_connections 2024-03-29T11:12:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f2fa806adb0 0x7f2fa806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc8161ca0 0x7f2fc806c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc80d18b0 0x7f2fc815a030 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 --2- 172.21.15.12:0/2336459956 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc8160f10 0x7f2fc806bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 -- 172.21.15.12:0/2336459956 >> 172.21.15.12:0/2336459956 conn(0x7f2fc808aff0 msgr2=0x7f2fc80caeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 -- 172.21.15.12:0/2336459956 shutdown_connections 2024-03-29T11:12:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:08.965+0000 7f2fcf600700 1 -- 172.21.15.12:0/2336459956 wait complete. 2024-03-29T11:12:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 81 2024-03-29T11:12:08.956 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:08.979+0000 7f8088449240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:12:08.961 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:09.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.185+0000 7fdac3800700 1 Processor -- start 2024-03-29T11:12:09.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.185+0000 7fdac3800700 1 -- start start 2024-03-29T11:12:09.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.187+0000 7fdac3800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdabc161ca0 0x7fdabc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.187+0000 7fdac3800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc0ceda0 con 0x7fdabc160f10 2024-03-29T11:12:09.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.187+0000 7fdac3800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc1612f0 con 0x7fdabc0d18b0 2024-03-29T11:12:09.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.187+0000 7fdac3800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdabc0d1d30 con 0x7fdabc161ca0 2024-03-29T11:12:09.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.187+0000 7fdac0e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdabc160f10 0x7fdabc0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39530/0 (socket says 172.21.15.12:39530) 2024-03-29T11:12:09.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.187+0000 7fdac0e00700 1 -- 172.21.15.12:0/1478314511 learned_addr learned my addr 172.21.15.12:0/1478314511 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:09.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.187+0000 7fdac1800700 1 --2- 172.21.15.12:0/1478314511 >> v2:172.21.15.159:3300/0 conn(0x7fdabc161ca0 0x7fdabc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:09.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.187+0000 7fdabb400700 1 -- 172.21.15.12:0/1478314511 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077671900 0 0) 0x7fdabc0ceda0 con 0x7fdabc160f10 2024-03-29T11:12:09.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.188+0000 7fdabb400700 1 -- 172.21.15.12:0/1478314511 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa40035b0 con 0x7fdabc160f10 2024-03-29T11:12:09.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.188+0000 7fdac1800700 1 -- 172.21.15.12:0/1478314511 >> v1:172.21.15.159:6789/0 conn(0x7fdabc0d18b0 legacy=0x7fdabc161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.188+0000 7fdac1800700 1 -- 172.21.15.12:0/1478314511 >> v1:172.21.15.12:6789/0 conn(0x7fdabc160f10 legacy=0x7fdabc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.188+0000 7fdac1800700 1 -- 172.21.15.12:0/1478314511 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc0ce470 con 0x7fdabc161ca0 2024-03-29T11:12:09.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.188+0000 7fdac1800700 1 --2- 172.21.15.12:0/1478314511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc161ca0 0x7fdabc0cf230 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009d90 tx=0x7fdab400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=913eba33156c4c45 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:09.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.188+0000 7fdabb400700 1 -- 172.21.15.12:0/1478314511 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab400e070 con 0x7fdabc161ca0 2024-03-29T11:12:09.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.188+0000 7fdabb400700 1 -- 172.21.15.12:0/1478314511 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab400bb20 con 0x7fdabc161ca0 2024-03-29T11:12:09.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.188+0000 7fdabb400700 1 -- 172.21.15.12:0/1478314511 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab4012430 con 0x7fdabc161ca0 2024-03-29T11:12:09.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.189+0000 7fdac3800700 1 -- 172.21.15.12:0/1478314511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc161ca0 msgr2=0x7fdabc0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.189+0000 7fdac3800700 1 --2- 172.21.15.12:0/1478314511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc161ca0 0x7fdabc0cf230 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009d90 tx=0x7fdab400b330 comp rx=0 tx=0).stop 2024-03-29T11:12:09.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.189+0000 7fdac3800700 1 -- 172.21.15.12:0/1478314511 shutdown_connections 2024-03-29T11:12:09.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.189+0000 7fdac3800700 1 --2- 172.21.15.12:0/1478314511 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc161ca0 0x7fdabc0cf230 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.189+0000 7fdac3800700 1 -- 172.21.15.12:0/1478314511 >> 172.21.15.12:0/1478314511 conn(0x7fdabc08aff0 msgr2=0x7fdabc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:09.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.189+0000 7fdac3800700 1 -- 172.21.15.12:0/1478314511 shutdown_connections 2024-03-29T11:12:09.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.189+0000 7fdac3800700 1 -- 172.21.15.12:0/1478314511 wait complete. 2024-03-29T11:12:09.169 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.191+0000 7f8088449240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:09.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.191+0000 7fdac3800700 1 Processor -- start 2024-03-29T11:12:09.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.191+0000 7fdac3800700 1 -- start start 2024-03-29T11:12:09.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdac3800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc0d18b0 0x7fdabc15f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdac3800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdabc160f10 0x7fdabc15d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdac3800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc161ca0 0x7fdabc15d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdac3800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc15dcf0 con 0x7fdabc161ca0 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdac3800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc15dfa0 con 0x7fdabc0d18b0 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdac3800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc15e280 con 0x7fdabc160f10 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdac0e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc0d18b0 0x7fdabc15f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdac1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc161ca0 0x7fdabc15d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdac0e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc0d18b0 0x7fdabc15f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:42348/0 (socket says 172.21.15.12:42348) 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdac0e00700 1 -- 172.21.15.12:0/3985978886 learned_addr learned my addr 172.21.15.12:0/3985978886 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.192+0000 7fdabbe00700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdabc160f10 0x7fdabc15d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.193+0000 7fdac0e00700 1 -- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdabc160f10 msgr2=0x7fdabc15d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.193+0000 7fdac0e00700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdabc160f10 0x7fdabc15d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.193+0000 7fdac0e00700 1 -- 172.21.15.12:0/3985978886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc161ca0 msgr2=0x7fdabc15d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.193+0000 7fdac0e00700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc161ca0 0x7fdabc15d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.193+0000 7fdac0e00700 1 -- 172.21.15.12:0/3985978886 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab4009a70 con 0x7fdabc0d18b0 2024-03-29T11:12:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.193+0000 7fdac0e00700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc0d18b0 0x7fdabc15f5b0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fdab0001eb0 tx=0x7fdab000c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.193+0000 7fdab8c00700 1 -- 172.21.15.12:0/3985978886 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0005d70 con 0x7fdabc0d18b0 2024-03-29T11:12:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.193+0000 7fdab8c00700 1 -- 172.21.15.12:0/3985978886 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab00070d0 con 0x7fdabc0d18b0 2024-03-29T11:12:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.193+0000 7fdac3800700 1 -- 172.21.15.12:0/3985978886 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdabc15e560 con 0x7fdabc0d18b0 2024-03-29T11:12:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.193+0000 7fdab8c00700 1 -- 172.21.15.12:0/3985978886 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0007460 con 0x7fdabc0d18b0 2024-03-29T11:12:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.194+0000 7fdac3800700 1 -- 172.21.15.12:0/3985978886 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdabc15eb20 con 0x7fdabc0d18b0 2024-03-29T11:12:09.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.194+0000 7fdac3800700 1 -- 172.21.15.12:0/3985978886 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdabc067720 con 0x7fdabc0d18b0 2024-03-29T11:12:09.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.195+0000 7fdab8c00700 1 -- 172.21.15.12:0/3985978886 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fdab00075c0 con 0x7fdabc0d18b0 2024-03-29T11:12:09.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.196+0000 7fdab8c00700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda9c06adb0 0x7fda9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.196+0000 7fdab8c00700 1 -- 172.21.15.12:0/3985978886 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fdab0088a20 con 0x7fdabc0d18b0 2024-03-29T11:12:09.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.196+0000 7fdabbe00700 1 -- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda9c06adb0 msgr2=0x7fda9c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:09.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.196+0000 7fdabbe00700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda9c06adb0 0x7fda9c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.201+0000 7fdab8c00700 1 -- 172.21.15.12:0/3985978886 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdab0053950 con 0x7fdabc0d18b0 2024-03-29T11:12:09.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.397+0000 7fdabbe00700 1 -- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda9c06adb0 msgr2=0x7fda9c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:09.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.397+0000 7fdabbe00700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda9c06adb0 0x7fda9c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:09.387 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:09.410+0000 7f49d499b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:12:09.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.416+0000 7fdac3800700 1 -- 172.21.15.12:0/3985978886 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdabc08b760 con 0x7fdabc0d18b0 2024-03-29T11:12:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.418+0000 7fdab8c00700 1 -- 172.21.15.12:0/3985978886 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+16677 (secure 0 0 0) 0x7fdab0017090 con 0x7fdabc0d18b0 2024-03-29T11:12:09.396 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:09.396 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":178,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:11:59.862985+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:11:41.110455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-29T11:11:41.640016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1870707179":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/499386558":"2024-03-30T11:11:59.862930+0000","172.21.15.159:6833/43444":"2024-03-30T11:11:59.862930+0000","172.21.15.159:6832/43444":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/3131465277":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/3944320011":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/998093551":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/618363904":"2024-03-30T11:11:59.862930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:09.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 -- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda9c06adb0 msgr2=0x7fda9c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:09.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda9c06adb0 0x7fda9c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 -- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc0d18b0 msgr2=0x7fdabc15f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc0d18b0 0x7fdabc15f5b0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fdab0001eb0 tx=0x7fdab000c7b0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 -- 172.21.15.12:0/3985978886 shutdown_connections 2024-03-29T11:12:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fda9c06adb0 0x7fda9c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc161ca0 0x7fdabc15d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdabc0d18b0 0x7fdabc15f5b0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 --2- 172.21.15.12:0/3985978886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdabc160f10 0x7fdabc15d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 -- 172.21.15.12:0/3985978886 >> 172.21.15.12:0/3985978886 conn(0x7fdabc08aff0 msgr2=0x7fdabc0d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 -- 172.21.15.12:0/3985978886 shutdown_connections 2024-03-29T11:12:09.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.420+0000 7fdac3800700 1 -- 172.21.15.12:0/3985978886 wait complete. 2024-03-29T11:12:09.417 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T11:12:09.417 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:12:09.612 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:09.634+0000 7f49d499b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:09.613 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.636+0000 7f8088449240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:12:09.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.640+0000 7f82d0200700 1 Processor -- start 2024-03-29T11:12:09.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.641+0000 7f82d0200700 1 -- start start 2024-03-29T11:12:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.642+0000 7f82d0200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c8160f10 0x7f82c80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.642+0000 7f82d0200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f82c8161590 0x7f82c8161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.642+0000 7f82d0200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c80ceda0 con 0x7f82c80cf230 2024-03-29T11:12:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.642+0000 7f82d0200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c8161f80 con 0x7f82c8161590 2024-03-29T11:12:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.642+0000 7f82d0200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c81612f0 con 0x7f82c8160f10 2024-03-29T11:12:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.642+0000 7f82ce200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f82c8161590 0x7f82c8161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.642+0000 7f82cd800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c8160f10 0x7f82c80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.642+0000 7f82cd800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c8160f10 0x7f82c80d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:60406/0 (socket says 172.21.15.12:60406) 2024-03-29T11:12:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.642+0000 7f82ce200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f82c8161590 0x7f82c8161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:42358/0 (socket says 172.21.15.12:42358) 2024-03-29T11:12:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.642+0000 7f82cd800700 1 -- 172.21.15.12:0/620707291 learned_addr learned my addr 172.21.15.12:0/620707291 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.643+0000 7f82ce200700 1 -- 172.21.15.12:0/620707291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c8160f10 msgr2=0x7f82c80d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.643+0000 7f82ce200700 1 --2- 172.21.15.12:0/620707291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c8160f10 0x7f82c80d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.643+0000 7f82ce200700 1 -- 172.21.15.12:0/620707291 >> v1:172.21.15.159:6789/0 conn(0x7f82c80cf230 legacy=0x7f82c80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.643+0000 7f82ce200700 1 -- 172.21.15.12:0/620707291 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c80ce470 con 0x7f82c8161590 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.643+0000 7f82cd800700 1 --2- 172.21.15.12:0/620707291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c8160f10 0x7f82c80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.643+0000 7f82ce200700 1 --2- 172.21.15.12:0/620707291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82c8161590 0x7f82c8161a40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f82c0009dc0 tx=0x7f82c000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7eedbf36d9943557 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.643+0000 7f82c7e00700 1 -- 172.21.15.12:0/620707291 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c000e070 con 0x7f82c8161590 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.643+0000 7f82c7e00700 1 -- 172.21.15.12:0/620707291 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c000bae0 con 0x7f82c8161590 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.643+0000 7f82c7e00700 1 -- 172.21.15.12:0/620707291 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c000be90 con 0x7f82c8161590 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.644+0000 7f82d0200700 1 -- 172.21.15.12:0/620707291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82c8161590 msgr2=0x7f82c8161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.644+0000 7f82d0200700 1 --2- 172.21.15.12:0/620707291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82c8161590 0x7f82c8161a40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f82c0009dc0 tx=0x7f82c000b330 comp rx=0 tx=0).stop 2024-03-29T11:12:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.644+0000 7f82d0200700 1 -- 172.21.15.12:0/620707291 shutdown_connections 2024-03-29T11:12:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.644+0000 7f82d0200700 1 --2- 172.21.15.12:0/620707291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82c8161590 0x7f82c8161a40 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.644+0000 7f82d0200700 1 --2- 172.21.15.12:0/620707291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c8160f10 0x7f82c80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.644+0000 7f82d0200700 1 -- 172.21.15.12:0/620707291 >> 172.21.15.12:0/620707291 conn(0x7f82c808aff0 msgr2=0x7f82c80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.644+0000 7f82d0200700 1 -- 172.21.15.12:0/620707291 shutdown_connections 2024-03-29T11:12:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.644+0000 7f82d0200700 1 -- 172.21.15.12:0/620707291 wait complete. 2024-03-29T11:12:09.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.646+0000 7f82d0200700 1 Processor -- start 2024-03-29T11:12:09.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.646+0000 7f82d0200700 1 -- start start 2024-03-29T11:12:09.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.647+0000 7f82d0200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c80cf230 0x7f82c814ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.647+0000 7f82d0200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82c8160f10 0x7f82c80d0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.647+0000 7f82d0200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82c8161590 0x7f82c80d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.647+0000 7f82d0200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c814bf50 con 0x7f82c8160f10 2024-03-29T11:12:09.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.647+0000 7f82d0200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80b6f20 con 0x7f82c8161590 2024-03-29T11:12:09.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.647+0000 7f82d0200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80b7200 con 0x7f82c80cf230 2024-03-29T11:12:09.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.647+0000 7f82cce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82c8160f10 0x7f82c80d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:09.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.647+0000 7f82cd800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c80cf230 0x7f82c814ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:09.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.648+0000 7f82cce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82c8160f10 0x7f82c80d0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48120/0 (socket says 172.21.15.12:48120) 2024-03-29T11:12:09.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.648+0000 7f82cce00700 1 -- 172.21.15.12:0/2252298091 learned_addr learned my addr 172.21.15.12:0/2252298091 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:09.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.648+0000 7f82ce200700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82c8161590 0x7f82c80d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:09.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.648+0000 7f82cce00700 1 -- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c80cf230 msgr2=0x7f82c814ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.648+0000 7f82cce00700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c80cf230 0x7f82c814ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.648+0000 7f82cce00700 1 -- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82c8161590 msgr2=0x7f82c80d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.648+0000 7f82cce00700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82c8161590 0x7f82c80d07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.648+0000 7f82cce00700 1 -- 172.21.15.12:0/2252298091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc009550 con 0x7f82c8160f10 2024-03-29T11:12:09.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.648+0000 7f82cce00700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82c8160f10 0x7f82c80d0280 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f82b400c420 tx=0x7f82b400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:09.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.648+0000 7f82c5600700 1 -- 172.21.15.12:0/2252298091 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b40052d0 con 0x7f82c8160f10 2024-03-29T11:12:09.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.649+0000 7f82c5600700 1 -- 172.21.15.12:0/2252298091 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b4005430 con 0x7f82c8160f10 2024-03-29T11:12:09.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.649+0000 7f82d0200700 1 -- 172.21.15.12:0/2252298091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c0009a70 con 0x7f82c8160f10 2024-03-29T11:12:09.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.649+0000 7f82c5600700 1 -- 172.21.15.12:0/2252298091 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b40038c0 con 0x7f82c8160f10 2024-03-29T11:12:09.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.649+0000 7f82d0200700 1 -- 172.21.15.12:0/2252298091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c80b77c0 con 0x7f82c8160f10 2024-03-29T11:12:09.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.650+0000 7f82d0200700 1 -- 172.21.15.12:0/2252298091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c8067720 con 0x7f82c8160f10 2024-03-29T11:12:09.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.651+0000 7f82c5600700 1 -- 172.21.15.12:0/2252298091 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 95) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f82b4003a20 con 0x7f82c8160f10 2024-03-29T11:12:09.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.652+0000 7f82c5600700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82a806adb0 0x7f82a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:09.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.652+0000 7f82c5600700 1 -- 172.21.15.12:0/2252298091 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f82b40889b0 con 0x7f82c8160f10 2024-03-29T11:12:09.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.652+0000 7f82cd800700 1 -- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82a806adb0 msgr2=0x7f82a806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:09.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.652+0000 7f82cd800700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82a806adb0 0x7f82a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:09.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.656+0000 7f82c5600700 1 -- 172.21.15.12:0/2252298091 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82b4053850 con 0x7f82c8160f10 2024-03-29T11:12:09.727 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.750+0000 7f8088449240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.767+0000 7f82c5600700 1 -- 172.21.15.12:0/2252298091 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 96) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f82b4053c20 con 0x7f82c8160f10 2024-03-29T11:12:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.767+0000 7f82c5600700 1 -- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82a806adb0 msgr2=0x7f82a806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.767+0000 7f82c5600700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82a806adb0 0x7f82a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.809 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.832+0000 7f7ffa200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:09.809 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.832+0000 7f7ffa200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:09.809 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.832+0000 7f7ffa200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:09.809 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.832+0000 7f7ffa200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:09.809 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.832+0000 7f7ffa200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:09.810 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.832+0000 7f7ff7a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:09.810 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.832+0000 7f7ff7a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:09.810 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.832+0000 7f7ff7a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:09.811 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.832+0000 7f7ff7a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:09.811 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:09.832+0000 7f7ff7a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:09.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.931+0000 7f82d0200700 1 -- 172.21.15.12:0/2252298091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82c80b5e80 con 0x7f82c8160f10 2024-03-29T11:12:09.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.931+0000 7f82c5600700 1 -- 172.21.15.12:0/2252298091 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f82b4073030 con 0x7f82c8160f10 2024-03-29T11:12:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:09.909 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/652112625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":652112625},{"type":"v1","addr":"172.21.15.159:6836","nonce":652112625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9166,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2803735172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2803735172},{"type":"v1","addr":"172.21.15.159:6837","nonce":2803735172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9444,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3698805288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3698805288},{"type":"v1","addr":"172.21.15.12:6833","nonce":3698805288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:11:42.673965+0000","modified":"2024-03-29T11:11:46.149885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9450":{"gid":9450,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/2066631988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2066631988},{"type":"v1","addr":"172.21.15.12:6835","nonce":2066631988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-29T11:12:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.934+0000 7f82d0200700 1 -- 172.21.15.12:0/2252298091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82c8160f10 msgr2=0x7f82c80d0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.934+0000 7f82d0200700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82c8160f10 0x7f82c80d0280 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f82b400c420 tx=0x7f82b400c730 comp rx=0 tx=0).stop 2024-03-29T11:12:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.934+0000 7f82d0200700 1 -- 172.21.15.12:0/2252298091 shutdown_connections 2024-03-29T11:12:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.934+0000 7f82d0200700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f82a806adb0 0x7f82a806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.934+0000 7f82d0200700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82c8161590 0x7f82c80d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.934+0000 7f82d0200700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82c80cf230 0x7f82c814ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.934+0000 7f82d0200700 1 --2- 172.21.15.12:0/2252298091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82c8160f10 0x7f82c80d0280 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.934+0000 7f82d0200700 1 -- 172.21.15.12:0/2252298091 >> 172.21.15.12:0/2252298091 conn(0x7f82c808aff0 msgr2=0x7f82c815ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.934+0000 7f82d0200700 1 -- 172.21.15.12:0/2252298091 shutdown_connections 2024-03-29T11:12:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:09.934+0000 7f82d0200700 1 -- 172.21.15.12:0/2252298091 wait complete. 2024-03-29T11:12:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 81 2024-03-29T11:12:09.931 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:12:10.081 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:10.103+0000 7f49d499b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:12:10.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.155+0000 7fbc7ea00700 1 Processor -- start 2024-03-29T11:12:10.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.155+0000 7fbc7ea00700 1 -- start start 2024-03-29T11:12:10.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.156+0000 7fbc7ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc780718e0 0x7fbc780ad070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:10.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.157+0000 7fbc7ea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc780ac4d0 con 0x7fbc78160c90 2024-03-29T11:12:10.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.157+0000 7fbc7ea00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc78071e50 con 0x7fbc780b27f0 2024-03-29T11:12:10.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.157+0000 7fbc7ea00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc78072130 con 0x7fbc780718e0 2024-03-29T11:12:10.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.157+0000 7fbc7ca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc780718e0 0x7fbc780ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:10.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.157+0000 7fbc77e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbc78160c90 0x7fbc780b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39538/0 (socket says 172.21.15.12:39538) 2024-03-29T11:12:10.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.157+0000 7fbc77e00700 1 -- 172.21.15.12:0/3306536048 learned_addr learned my addr 172.21.15.12:0/3306536048 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.157+0000 7fbc7ca00700 1 -- 172.21.15.12:0/3306536048 >> v1:172.21.15.159:6789/0 conn(0x7fbc780b27f0 legacy=0x7fbc780ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.157+0000 7fbc7ca00700 1 -- 172.21.15.12:0/3306536048 >> v1:172.21.15.12:6789/0 conn(0x7fbc78160c90 legacy=0x7fbc780b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.157+0000 7fbc7ca00700 1 -- 172.21.15.12:0/3306536048 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78072410 con 0x7fbc780718e0 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.157+0000 7fbc77400700 1 -- 172.21.15.12:0/3306536048 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343957366 0 0) 0x7fbc780ac4d0 con 0x7fbc78160c90 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.158+0000 7fbc7ca00700 1 --2- 172.21.15.12:0/3306536048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 0x7fbc780ad070 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009d90 tx=0x7fbc7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b3e2b75f06e999 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.158+0000 7fbc77400700 1 -- 172.21.15.12:0/3306536048 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7000e070 con 0x7fbc780718e0 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.158+0000 7fbc77400700 1 -- 172.21.15.12:0/3306536048 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7000bbd0 con 0x7fbc780718e0 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.158+0000 7fbc77400700 1 -- 172.21.15.12:0/3306536048 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc70012430 con 0x7fbc780718e0 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.159+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3306536048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 msgr2=0x7fbc780ad070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.159+0000 7fbc7ea00700 1 --2- 172.21.15.12:0/3306536048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 0x7fbc780ad070 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009d90 tx=0x7fbc7000b330 comp rx=0 tx=0).stop 2024-03-29T11:12:10.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.159+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3306536048 shutdown_connections 2024-03-29T11:12:10.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.159+0000 7fbc7ea00700 1 --2- 172.21.15.12:0/3306536048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 0x7fbc780ad070 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.159+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3306536048 >> 172.21.15.12:0/3306536048 conn(0x7fbc780cf190 msgr2=0x7fbc780cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:10.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.159+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3306536048 shutdown_connections 2024-03-29T11:12:10.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.159+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3306536048 wait complete. 2024-03-29T11:12:10.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.161+0000 7fbc7ea00700 1 Processor -- start 2024-03-29T11:12:10.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.161+0000 7fbc7ea00700 1 -- start start 2024-03-29T11:12:10.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc7ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 0x7fbc78159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:10.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc7ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc780b27f0 0x7fbc78157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:10.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc7ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc78160c90 0x7fbc7809bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:10.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc7ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78159b80 con 0x7fbc780718e0 2024-03-29T11:12:10.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc7ea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc781581b0 con 0x7fbc78160c90 2024-03-29T11:12:10.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc7ea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78158490 con 0x7fbc780b27f0 2024-03-29T11:12:10.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc77e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 0x7fbc78159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:10.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc77e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 0x7fbc78159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48136/0 (socket says 172.21.15.12:48136) 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc77e00700 1 -- 172.21.15.12:0/3932618057 learned_addr learned my addr 172.21.15.12:0/3932618057 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc6f400700 1 --2- 172.21.15.12:0/3932618057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc780b27f0 0x7fbc78157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc7ca00700 1 --2- 172.21.15.12:0/3932618057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc78160c90 0x7fbc7809bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc77e00700 1 -- 172.21.15.12:0/3932618057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc780b27f0 msgr2=0x7fbc78157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc77e00700 1 --2- 172.21.15.12:0/3932618057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc780b27f0 0x7fbc78157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc77e00700 1 -- 172.21.15.12:0/3932618057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc78160c90 msgr2=0x7fbc7809bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc77e00700 1 --2- 172.21.15.12:0/3932618057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc78160c90 0x7fbc7809bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc77e00700 1 -- 172.21.15.12:0/3932618057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc70009a70 con 0x7fbc780718e0 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.163+0000 7fbc7ca00700 1 --2- 172.21.15.12:0/3932618057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc78160c90 0x7fbc7809bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.164+0000 7fbc77e00700 1 --2- 172.21.15.12:0/3932618057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 0x7fbc78159640 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800c520 tx=0x7fbc6800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.164+0000 7fbc74c00700 1 -- 172.21.15.12:0/3932618057 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68004c80 con 0x7fbc780718e0 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.164+0000 7fbc74c00700 1 -- 172.21.15.12:0/3932618057 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc68004e20 con 0x7fbc780718e0 2024-03-29T11:12:10.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.164+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3932618057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc78158770 con 0x7fbc780718e0 2024-03-29T11:12:10.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.164+0000 7fbc74c00700 1 -- 172.21.15.12:0/3932618057 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68007310 con 0x7fbc780718e0 2024-03-29T11:12:10.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.164+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3932618057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc781f2fd0 con 0x7fbc780718e0 2024-03-29T11:12:10.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.165+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3932618057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc78067720 con 0x7fbc780718e0 2024-03-29T11:12:10.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.166+0000 7fbc74c00700 1 -- 172.21.15.12:0/3932618057 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 96) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fbc68007470 con 0x7fbc780718e0 2024-03-29T11:12:10.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.167+0000 7fbc74c00700 1 -- 172.21.15.12:0/3932618057 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6355+0+0 (secure 0 0 0) 0x7fbc68088e50 con 0x7fbc780718e0 2024-03-29T11:12:10.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.171+0000 7fbc74c00700 1 -- 172.21.15.12:0/3932618057 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc68054100 con 0x7fbc780718e0 2024-03-29T11:12:10.200 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:10.223+0000 7f49d499b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:10.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.436+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3932618057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc7806daf0 con 0x7fbc780718e0 2024-03-29T11:12:10.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.436+0000 7fbc74c00700 1 -- 172.21.15.12:0/3932618057 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fbc68057720 con 0x7fbc780718e0 2024-03-29T11:12:10.414 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:10.415 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/652112625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":652112625},{"type":"v1","addr":"172.21.15.159:6836","nonce":652112625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9166,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2803735172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2803735172},{"type":"v1","addr":"172.21.15.159:6837","nonce":2803735172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9444,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3698805288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3698805288},{"type":"v1","addr":"172.21.15.12:6833","nonce":3698805288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:11:42.673965+0000","modified":"2024-03-29T11:11:46.149885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9450":{"gid":9450,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6835/2066631988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2066631988},{"type":"v1","addr":"172.21.15.12:6835","nonce":2066631988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-29T11:12:10.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.438+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3932618057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 msgr2=0x7fbc78159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.438+0000 7fbc7ea00700 1 --2- 172.21.15.12:0/3932618057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 0x7fbc78159640 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800c520 tx=0x7fbc6800c8e0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.439+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3932618057 shutdown_connections 2024-03-29T11:12:10.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.439+0000 7fbc7ea00700 1 --2- 172.21.15.12:0/3932618057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc78160c90 0x7fbc7809bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.439+0000 7fbc7ea00700 1 --2- 172.21.15.12:0/3932618057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc780718e0 0x7fbc78159640 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.439+0000 7fbc7ea00700 1 --2- 172.21.15.12:0/3932618057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc780b27f0 0x7fbc78157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.439+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3932618057 >> 172.21.15.12:0/3932618057 conn(0x7fbc780cf190 msgr2=0x7fbc7806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:10.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.439+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3932618057 shutdown_connections 2024-03-29T11:12:10.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.439+0000 7fbc7ea00700 1 -- 172.21.15.12:0/3932618057 wait complete. 2024-03-29T11:12:10.418 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 81 2024-03-29T11:12:10.435 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:10.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.654+0000 7f3831a00700 1 Processor -- start 2024-03-29T11:12:10.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.654+0000 7f3831a00700 1 -- start start 2024-03-29T11:12:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.655+0000 7f3831a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f382c06a790 0x7f382c06ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.655+0000 7f3831a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0afbe0 0x7f382c06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.655+0000 7f3831a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382c06f850 con 0x7f382c0cebd0 2024-03-29T11:12:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.655+0000 7f3831a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f382c06b0b0 con 0x7f382c06a790 2024-03-29T11:12:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.655+0000 7f3831a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382c06fad0 con 0x7f382c0afbe0 2024-03-29T11:12:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f382b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f382c0cebd0 0x7f382c0cf040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39550/0 (socket says 172.21.15.12:39550) 2024-03-29T11:12:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f382b400700 1 -- 172.21.15.12:0/3733777658 learned_addr learned my addr 172.21.15.12:0/3733777658 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f382a000700 1 --2- 172.21.15.12:0/3733777658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0afbe0 0x7f382c06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f382aa00700 1 --2- 172.21.15.12:0/3733777658 >> v2:172.21.15.159:3300/0 conn(0x7f382c06a790 0x7f382c06ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f3829600700 1 -- 172.21.15.12:0/3733777658 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544001848 0 0) 0x7f382c06f850 con 0x7f382c0cebd0 2024-03-29T11:12:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f3829600700 1 -- 172.21.15.12:0/3733777658 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38100035b0 con 0x7f382c0cebd0 2024-03-29T11:12:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f382aa00700 1 -- 172.21.15.12:0/3733777658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0afbe0 msgr2=0x7f382c06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f382aa00700 1 --2- 172.21.15.12:0/3733777658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0afbe0 0x7f382c06dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f382aa00700 1 -- 172.21.15.12:0/3733777658 >> v1:172.21.15.12:6789/0 conn(0x7f382c0cebd0 legacy=0x7f382c0cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f382aa00700 1 -- 172.21.15.12:0/3733777658 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c0af970 con 0x7f382c06a790 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.656+0000 7f382a000700 1 --2- 172.21.15.12:0/3733777658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0afbe0 0x7f382c06dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.657+0000 7f382aa00700 1 --2- 172.21.15.12:0/3733777658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f382c06a790 0x7f382c06ab70 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f381c009840 tx=0x7f381c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1a5c0168b3a8f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.657+0000 7f3829600700 1 -- 172.21.15.12:0/3733777658 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c011830 con 0x7f382c06a790 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.657+0000 7f3829600700 1 -- 172.21.15.12:0/3733777658 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381c011990 con 0x7f382c06a790 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.657+0000 7f3829600700 1 -- 172.21.15.12:0/3733777658 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c011d20 con 0x7f382c06a790 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.658+0000 7f3831a00700 1 -- 172.21.15.12:0/3733777658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f382c06a790 msgr2=0x7f382c06ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.658+0000 7f3831a00700 1 --2- 172.21.15.12:0/3733777658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f382c06a790 0x7f382c06ab70 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f381c009840 tx=0x7f381c009b50 comp rx=0 tx=0).stop 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.658+0000 7f3831a00700 1 -- 172.21.15.12:0/3733777658 shutdown_connections 2024-03-29T11:12:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.658+0000 7f3831a00700 1 --2- 172.21.15.12:0/3733777658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f382c06a790 0x7f382c06ab70 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.658+0000 7f3831a00700 1 --2- 172.21.15.12:0/3733777658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0afbe0 0x7f382c06dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.658+0000 7f3831a00700 1 -- 172.21.15.12:0/3733777658 >> 172.21.15.12:0/3733777658 conn(0x7f382c0b5ab0 msgr2=0x7f382c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:10.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.658+0000 7f3831a00700 1 -- 172.21.15.12:0/3733777658 shutdown_connections 2024-03-29T11:12:10.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.658+0000 7f3831a00700 1 -- 172.21.15.12:0/3733777658 wait complete. 2024-03-29T11:12:10.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.659+0000 7f3831a00700 1 Processor -- start 2024-03-29T11:12:10.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.660+0000 7f3831a00700 1 -- start start 2024-03-29T11:12:10.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.660+0000 7f3831a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f382c06a790 0x7f382c152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:10.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.660+0000 7f3831a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f382c0afbe0 0x7f382c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:10.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.660+0000 7f3831a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0cebd0 0x7f382c0b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:10.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f3831a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c09b3a0 con 0x7f382c0afbe0 2024-03-29T11:12:10.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f3831a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c09b5d0 con 0x7f382c06a790 2024-03-29T11:12:10.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f3831a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382c09b8b0 con 0x7f382c0cebd0 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f382c0afbe0 0x7f382c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f382c0afbe0 0x7f382c0b5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48148/0 (socket says 172.21.15.12:48148) 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382a000700 1 -- 172.21.15.12:0/4222056022 learned_addr learned my addr 172.21.15.12:0/4222056022 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382aa00700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f382c06a790 0x7f382c152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382b400700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0cebd0 0x7f382c0b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382a000700 1 -- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0cebd0 msgr2=0x7f382c0b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382a000700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0cebd0 0x7f382c0b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382a000700 1 -- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f382c06a790 msgr2=0x7f382c152cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382a000700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f382c06a790 0x7f382c152cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382a000700 1 -- 172.21.15.12:0/4222056022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38180094a0 con 0x7f382c0afbe0 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.661+0000 7f382a000700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f382c0afbe0 0x7f382c0b5000 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f381800ebe0 tx=0x7f381800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.662+0000 7f3822a00700 1 -- 172.21.15.12:0/4222056022 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381800c850 con 0x7f382c0afbe0 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.662+0000 7f3822a00700 1 -- 172.21.15.12:0/4222056022 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381800c9b0 con 0x7f382c0afbe0 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.662+0000 7f3831a00700 1 -- 172.21.15.12:0/4222056022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381c0094a0 con 0x7f382c0afbe0 2024-03-29T11:12:10.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.662+0000 7f3822a00700 1 -- 172.21.15.12:0/4222056022 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381800cd40 con 0x7f382c0afbe0 2024-03-29T11:12:10.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.662+0000 7f3831a00700 1 -- 172.21.15.12:0/4222056022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382c09be70 con 0x7f382c0afbe0 2024-03-29T11:12:10.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.663+0000 7f3831a00700 1 -- 172.21.15.12:0/4222056022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382c09c4c0 con 0x7f382c0afbe0 2024-03-29T11:12:10.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.664+0000 7f3822a00700 1 -- 172.21.15.12:0/4222056022 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 96) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f3818013470 con 0x7f382c0afbe0 2024-03-29T11:12:10.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.665+0000 7f3822a00700 1 -- 172.21.15.12:0/4222056022 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f38180890c0 con 0x7f382c0afbe0 2024-03-29T11:12:10.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.669+0000 7f3822a00700 1 -- 172.21.15.12:0/4222056022 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3818054460 con 0x7f382c0afbe0 2024-03-29T11:12:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.764+0000 7f3822a00700 1 -- 172.21.15.12:0/4222056022 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 97) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f3818053f10 con 0x7f382c0afbe0 2024-03-29T11:12:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.765+0000 7f3822a00700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f380c036150 0x7f380c072350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.768+0000 7f382aa00700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f380c036150 0x7f380c072350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.769+0000 7f382aa00700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f380c036150 0x7f380c072350 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f381c000c00 tx=0x7f381c01c040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.889+0000 7f3831a00700 1 -- 172.21.15.12:0/4222056022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f382c067720 con 0x7f382c0afbe0 2024-03-29T11:12:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.891+0000 7f3822a00700 1 -- 172.21.15.12:0/4222056022 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+17055 (secure 0 0 0) 0x7f381806ff00 con 0x7f382c0afbe0 2024-03-29T11:12:10.869 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:10.869 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":179,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:12:09.753424+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:11:41.110455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-29T11:11:41.640016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3147228644":"2024-03-30T11:12:09.753371+0000","172.21.15.159:0/2849765854":"2024-03-30T11:12:09.753371+0000","172.21.15.159:0/3290985110":"2024-03-30T11:12:09.753371+0000","172.21.15.159:0/1401286467":"2024-03-30T11:12:09.753371+0000","172.21.15.159:0/3615298828":"2024-03-30T11:12:09.753371+0000","172.21.15.159:0/1870707179":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/499386558":"2024-03-30T11:11:59.862930+0000","172.21.15.159:6833/43444":"2024-03-30T11:11:59.862930+0000","172.21.15.159:6832/43444":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/2867550590":"2024-03-30T11:12:09.753371+0000","172.21.15.159:0/3131465277":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/3944320011":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/998093551":"2024-03-30T11:11:59.862930+0000","172.21.15.159:0/618363904":"2024-03-30T11:11:59.862930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 -- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f380c036150 msgr2=0x7f380c072350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f380c036150 0x7f380c072350 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f381c000c00 tx=0x7f381c01c040 comp rx=0 tx=0).stop 2024-03-29T11:12:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 -- 172.21.15.12:0/4222056022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f382c0afbe0 msgr2=0x7f382c0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f382c0afbe0 0x7f382c0b5000 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f381800ebe0 tx=0x7f381800eef0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 -- 172.21.15.12:0/4222056022 shutdown_connections 2024-03-29T11:12:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f380c036150 0x7f380c072350 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f382c0cebd0 0x7f382c0b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f382c06a790 0x7f382c152cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 --2- 172.21.15.12:0/4222056022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f382c0afbe0 0x7f382c0b5000 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 -- 172.21.15.12:0/4222056022 >> 172.21.15.12:0/4222056022 conn(0x7f382c0b5ab0 msgr2=0x7f382c0ce380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 -- 172.21.15.12:0/4222056022 shutdown_connections 2024-03-29T11:12:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:10.893+0000 7f3831a00700 1 -- 172.21.15.12:0/4222056022 wait complete. 2024-03-29T11:12:10.890 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T11:12:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.112+0000 7fa20fc00700 1 Processor -- start 2024-03-29T11:12:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.112+0000 7fa20fc00700 1 -- start start 2024-03-29T11:12:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.113+0000 7fa20fc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2080cd480 0x7fa20806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.113+0000 7fa20fc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2080cdae0 con 0x7fa2080cbb30 2024-03-29T11:12:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.113+0000 7fa20fc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2080cddc0 con 0x7fa2080cd020 2024-03-29T11:12:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.114+0000 7fa20fc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2080ce0a0 con 0x7fa2080cd480 2024-03-29T11:12:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.114+0000 7fa20d200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa2080cbb30 0x7fa2080cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39566/0 (socket says 172.21.15.12:39566) 2024-03-29T11:12:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.114+0000 7fa20d200700 1 -- 172.21.15.12:0/242773563 learned_addr learned my addr 172.21.15.12:0/242773563 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.114+0000 7fa20dc00700 1 --2- 172.21.15.12:0/242773563 >> v2:172.21.15.159:3300/0 conn(0x7fa2080cd480 0x7fa20806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.114+0000 7fa207400700 1 -- 172.21.15.12:0/242773563 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538608373 0 0) 0x7fa2080cdae0 con 0x7fa2080cbb30 2024-03-29T11:12:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.114+0000 7fa207400700 1 -- 172.21.15.12:0/242773563 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f00035b0 con 0x7fa2080cbb30 2024-03-29T11:12:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.114+0000 7fa207400700 1 -- 172.21.15.12:0/242773563 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336598298 0 0) 0x7fa2080cddc0 con 0x7fa2080cd020 2024-03-29T11:12:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.114+0000 7fa207400700 1 -- 172.21.15.12:0/242773563 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2080cdae0 con 0x7fa2080cd020 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.115+0000 7fa20dc00700 1 -- 172.21.15.12:0/242773563 >> v1:172.21.15.159:6789/0 conn(0x7fa2080cd020 legacy=0x7fa2080b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.115+0000 7fa20dc00700 1 -- 172.21.15.12:0/242773563 >> v1:172.21.15.12:6789/0 conn(0x7fa2080cbb30 legacy=0x7fa2080cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.115+0000 7fa20dc00700 1 -- 172.21.15.12:0/242773563 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20806dc20 con 0x7fa2080cd480 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.115+0000 7fa20dc00700 1 --2- 172.21.15.12:0/242773563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2080cd480 0x7fa20806e420 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa200009d90 tx=0x7fa20000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8ed90c845c72ac server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.115+0000 7fa207400700 1 -- 172.21.15.12:0/242773563 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20000e070 con 0x7fa2080cd480 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.115+0000 7fa207400700 1 -- 172.21.15.12:0/242773563 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20000bb20 con 0x7fa2080cd480 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.115+0000 7fa207400700 1 -- 172.21.15.12:0/242773563 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200012430 con 0x7fa2080cd480 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.116+0000 7fa20fc00700 1 -- 172.21.15.12:0/242773563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2080cd480 msgr2=0x7fa20806e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.116+0000 7fa20fc00700 1 --2- 172.21.15.12:0/242773563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2080cd480 0x7fa20806e420 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa200009d90 tx=0x7fa20000b330 comp rx=0 tx=0).stop 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.116+0000 7fa20fc00700 1 -- 172.21.15.12:0/242773563 shutdown_connections 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.116+0000 7fa20fc00700 1 --2- 172.21.15.12:0/242773563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2080cd480 0x7fa20806e420 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.116+0000 7fa20fc00700 1 -- 172.21.15.12:0/242773563 >> 172.21.15.12:0/242773563 conn(0x7fa2080b0d70 msgr2=0x7fa2080b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.116+0000 7fa20fc00700 1 -- 172.21.15.12:0/242773563 shutdown_connections 2024-03-29T11:12:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.116+0000 7fa20fc00700 1 -- 172.21.15.12:0/242773563 wait complete. 2024-03-29T11:12:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.118+0000 7fa20fc00700 1 Processor -- start 2024-03-29T11:12:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.118+0000 7fa20fc00700 1 -- start start 2024-03-29T11:12:11.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20fc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2080cbb30 0x7fa2081f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20fc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2080cd020 0x7fa208151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20fc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cd480 0x7fa2080adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20fc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080ad180 con 0x7fa2080cd480 2024-03-29T11:12:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20fc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080ad460 con 0x7fa2080cd020 2024-03-29T11:12:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20fc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080ad740 con 0x7fa2080cbb30 2024-03-29T11:12:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cd480 0x7fa2080adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cd480 0x7fa2080adbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48150/0 (socket says 172.21.15.12:48150) 2024-03-29T11:12:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20dc00700 1 -- 172.21.15.12:0/1530246797 learned_addr learned my addr 172.21.15.12:0/1530246797 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa207e00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2080cd020 0x7fa208151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20d200700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2080cbb30 0x7fa2081f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20dc00700 1 -- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2080cbb30 msgr2=0x7fa2081f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20dc00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2080cbb30 0x7fa2081f34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20dc00700 1 -- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2080cd020 msgr2=0x7fa208151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20dc00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2080cd020 0x7fa208151190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.120+0000 7fa20dc00700 1 -- 172.21.15.12:0/1530246797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa200009a70 con 0x7fa2080cd480 2024-03-29T11:12:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.121+0000 7fa20dc00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cd480 0x7fa2080adbe0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fa20806c130 tx=0x7fa200004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.121+0000 7fa204c00700 1 -- 172.21.15.12:0/1530246797 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200012430 con 0x7fa2080cd480 2024-03-29T11:12:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.121+0000 7fa204c00700 1 -- 172.21.15.12:0/1530246797 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa200020070 con 0x7fa2080cd480 2024-03-29T11:12:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.121+0000 7fa204c00700 1 -- 172.21.15.12:0/1530246797 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20001ba50 con 0x7fa2080cd480 2024-03-29T11:12:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.121+0000 7fa20fc00700 1 -- 172.21.15.12:0/1530246797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2080c8ea0 con 0x7fa2080cd480 2024-03-29T11:12:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.121+0000 7fa20fc00700 1 -- 172.21.15.12:0/1530246797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa208152170 con 0x7fa2080cd480 2024-03-29T11:12:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.122+0000 7fa20fc00700 1 -- 172.21.15.12:0/1530246797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa208067720 con 0x7fa2080cd480 2024-03-29T11:12:11.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.123+0000 7fa204c00700 1 -- 172.21.15.12:0/1530246797 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 97) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fa20001bbb0 con 0x7fa2080cd480 2024-03-29T11:12:11.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.124+0000 7fa204c00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806aae0 0x7fa1e806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.124+0000 7fa204c00700 1 -- 172.21.15.12:0/1530246797 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6355+0+0 (secure 0 0 0) 0x7fa200022070 con 0x7fa2080cd480 2024-03-29T11:12:11.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.124+0000 7fa20d200700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806aae0 0x7fa1e806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.125+0000 7fa20d200700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806aae0 0x7fa1e806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc007fd0 tx=0x7fa1fc007dc0 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:11.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.130+0000 7fa204c00700 1 -- 172.21.15.12:0/1530246797 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa200056350 con 0x7fa2080cd480 2024-03-29T11:12:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.391+0000 7fa20fc00700 1 -- 172.21.15.12:0/1530246797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa2080739a0 con 0x7fa2080cd480 2024-03-29T11:12:11.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T11:12:11.392+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T11:12:11.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.405+0000 7fa204c00700 1 -- 172.21.15.12:0/1530246797 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v82) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa20001be60 con 0x7fa2080cd480 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.407+0000 7fa20fc00700 1 -- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806aae0 msgr2=0x7fa1e806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.407+0000 7fa20fc00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806aae0 0x7fa1e806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc007fd0 tx=0x7fa1fc007dc0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.407+0000 7fa20fc00700 1 -- 172.21.15.12:0/1530246797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cd480 msgr2=0x7fa2080adbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.407+0000 7fa20fc00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cd480 0x7fa2080adbe0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fa20806c130 tx=0x7fa200004ad0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.407+0000 7fa20fc00700 1 -- 172.21.15.12:0/1530246797 shutdown_connections 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.407+0000 7fa20fc00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa1e806aae0 0x7fa1e806cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.407+0000 7fa20fc00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2080cd480 0x7fa2080adbe0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.407+0000 7fa20fc00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2080cbb30 0x7fa2081f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.407+0000 7fa20fc00700 1 --2- 172.21.15.12:0/1530246797 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2080cd020 0x7fa208151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.407+0000 7fa20fc00700 1 -- 172.21.15.12:0/1530246797 >> 172.21.15.12:0/1530246797 conn(0x7fa2080b0d70 msgr2=0x7fa20815cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.408+0000 7fa20fc00700 1 -- 172.21.15.12:0/1530246797 shutdown_connections 2024-03-29T11:12:11.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.408+0000 7fa20fc00700 1 -- 172.21.15.12:0/1530246797 wait complete. 2024-03-29T11:12:11.387 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T11:12:11.392 INFO:tasks.ceph.mds.a.smithi012.stderr: -5027> 2024-03-29T11:11:48.122+0000 7ff1c5000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:12:11.404 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T11:12:11.447 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:12:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.630+0000 7f0fc3000700 1 Processor -- start 2024-03-29T11:12:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.630+0000 7f0fc3000700 1 -- start start 2024-03-29T11:12:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.631+0000 7f0fc3000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0fbc0d16a0 0x7f0fbc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.631+0000 7f0fc3000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc161420 con 0x7f0fbc15d9d0 2024-03-29T11:12:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.631+0000 7f0fc3000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc161700 con 0x7f0fbc0d2370 2024-03-29T11:12:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.631+0000 7f0fc3000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fbc0d4340 con 0x7f0fbc0d16a0 2024-03-29T11:12:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.631+0000 7f0fbbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0fbc15d9d0 0x7f0fbc0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39576/0 (socket says 172.21.15.12:39576) 2024-03-29T11:12:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.631+0000 7f0fbbe00700 1 -- 172.21.15.12:0/2456881994 learned_addr learned my addr 172.21.15.12:0/2456881994 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbb400700 1 --2- 172.21.15.12:0/2456881994 >> v2:172.21.15.159:3300/0 conn(0x7f0fbc0d16a0 0x7f0fbc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861798290 0 0) 0x7f0fbc161420 con 0x7f0fbc15d9d0 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa00035b0 con 0x7f0fbc15d9d0 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3303711679 0 0) 0x7f0fa00035b0 con 0x7f0fbc15d9d0 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 >> v2:172.21.15.159:3300/0 conn(0x7f0fbc0d16a0 msgr2=0x7f0fbc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 --2- 172.21.15.12:0/2456881994 >> v2:172.21.15.159:3300/0 conn(0x7f0fbc0d16a0 0x7f0fbc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 >> v1:172.21.15.159:6789/0 conn(0x7f0fbc0d2370 legacy=0x7f0fbc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc1619e0 con 0x7f0fbc15d9d0 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0fac001670 con 0x7f0fbc15d9d0 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 >> v1:172.21.15.12:6789/0 conn(0x7f0fbc15d9d0 legacy=0x7f0fbc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fa0003a00 0x7f0fa0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fa0004850 0x7f0fa0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0005210 0x7f0fa000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa000c080 con 0x7f0fa0005210 2024-03-29T11:12:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa000c1f0 con 0x7f0fa0004850 2024-03-29T11:12:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa000c3b0 con 0x7f0fa0003a00 2024-03-29T11:12:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.632+0000 7f0fbb400700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fa0004850 0x7f0fa0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fc1000700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0005210 0x7f0fa000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fbbe00700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fa0003a00 0x7f0fa0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fbb400700 1 -- 172.21.15.12:0/2456881994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fa0003a00 msgr2=0x7f0fa0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fbb400700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fa0003a00 0x7f0fa0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fbb400700 1 -- 172.21.15.12:0/2456881994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0005210 msgr2=0x7f0fa000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fbb400700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0005210 0x7f0fa000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fbb400700 1 -- 172.21.15.12:0/2456881994 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa00035b0 con 0x7f0fa0004850 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fc1000700 1 -- 172.21.15.12:0/2456881994 reap_dead start 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fbb400700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fa0004850 0x7f0fa0004cd0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009b10 tx=0x7f0fa0005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fde2e6857f90c9a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8005a20 con 0x7f0fa0004850 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa8005b80 con 0x7f0fa0004850 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.633+0000 7f0fbaa00700 1 -- 172.21.15.12:0/2456881994 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa800e6c0 con 0x7f0fa0004850 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.634+0000 7f0fc3000700 1 -- 172.21.15.12:0/2456881994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fa0004850 msgr2=0x7f0fa0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.634+0000 7f0fc3000700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fa0004850 0x7f0fa0004cd0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009b10 tx=0x7f0fa0005880 comp rx=0 tx=0).stop 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.634+0000 7f0fc3000700 1 -- 172.21.15.12:0/2456881994 shutdown_connections 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.634+0000 7f0fc3000700 1 --2- 172.21.15.12:0/2456881994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fa0004850 0x7f0fa0004cd0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.634+0000 7f0fc3000700 1 -- 172.21.15.12:0/2456881994 >> 172.21.15.12:0/2456881994 conn(0x7f0fbc08aff0 msgr2=0x7f0fbc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.634+0000 7f0fc3000700 1 -- 172.21.15.12:0/2456881994 shutdown_connections 2024-03-29T11:12:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.634+0000 7f0fc3000700 1 -- 172.21.15.12:0/2456881994 wait complete. 2024-03-29T11:12:11.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.636+0000 7f0fc3000700 1 Processor -- start 2024-03-29T11:12:11.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.636+0000 7f0fc3000700 1 -- start start 2024-03-29T11:12:11.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fc3000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0004850 0x7f0fbc1584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fc3000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fbc0ca7b0 0x7f0fbc0b7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fc3000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fbc0b7960 0x7f0fbc0b4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fc3000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc1589e0 con 0x7f0fa0004850 2024-03-29T11:12:11.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fc3000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc15f930 con 0x7f0fbc0b7960 2024-03-29T11:12:11.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fc3000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc15fbd0 con 0x7f0fbc0ca7b0 2024-03-29T11:12:11.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fbbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0004850 0x7f0fbc1584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fbbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0004850 0x7f0fbc1584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48192/0 (socket says 172.21.15.12:48192) 2024-03-29T11:12:11.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fbbe00700 1 -- 172.21.15.12:0/1249536002 learned_addr learned my addr 172.21.15.12:0/1249536002 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:11.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fc1000700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fbc0b7960 0x7f0fbc0b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fbb400700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fbc0ca7b0 0x7f0fbc0b7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fbbe00700 1 -- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fbc0ca7b0 msgr2=0x7f0fbc0b7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fbbe00700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fbc0ca7b0 0x7f0fbc0b7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fbbe00700 1 -- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fbc0b7960 msgr2=0x7f0fbc0b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fbbe00700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fbc0b7960 0x7f0fbc0b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:11.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.637+0000 7f0fbbe00700 1 -- 172.21.15.12:0/1249536002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc15feb0 con 0x7f0fa0004850 2024-03-29T11:12:11.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.638+0000 7f0fbbe00700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0004850 0x7f0fbc1584a0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f0fac007290 tx=0x7f0fac007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:11.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.638+0000 7f0fb3e00700 1 -- 172.21.15.12:0/1249536002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac00f640 con 0x7f0fa0004850 2024-03-29T11:12:11.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.638+0000 7f0fb3e00700 1 -- 172.21.15.12:0/1249536002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac00f7a0 con 0x7f0fa0004850 2024-03-29T11:12:11.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.638+0000 7f0fc3000700 1 -- 172.21.15.12:0/1249536002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fbc160190 con 0x7f0fa0004850 2024-03-29T11:12:11.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.638+0000 7f0fb3e00700 1 -- 172.21.15.12:0/1249536002 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac00fb30 con 0x7f0fa0004850 2024-03-29T11:12:11.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.638+0000 7f0fc3000700 1 -- 172.21.15.12:0/1249536002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fbc0b7f00 con 0x7f0fa0004850 2024-03-29T11:12:11.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.639+0000 7f0fc3000700 1 -- 172.21.15.12:0/1249536002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fbc1603d0 con 0x7f0fa0004850 2024-03-29T11:12:11.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.640+0000 7f0fb3e00700 1 -- 172.21.15.12:0/1249536002 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 97) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f0fac00fc90 con 0x7f0fa0004850 2024-03-29T11:12:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.641+0000 7f0fb3e00700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9c06aae0 0x7f0f9c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.641+0000 7f0fb3e00700 1 -- 172.21.15.12:0/1249536002 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f0fac089180 con 0x7f0fa0004850 2024-03-29T11:12:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.642+0000 7f0fbb400700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9c06aae0 0x7f0f9c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.642+0000 7f0fbb400700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9c06aae0 0x7f0f9c06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8000c00 tx=0x7f0fa8008030 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:11.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.646+0000 7f0fb3e00700 1 -- 172.21.15.12:0/1249536002 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0fbc1603d0 con 0x7f0fa0004850 2024-03-29T11:12:11.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:11.913+0000 7f0fc3000700 1 -- 172.21.15.12:0/1249536002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0fbc1603d0 con 0x7f0fa0004850 2024-03-29T11:12:12.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.408+0000 7f0fb3e00700 1 -- 172.21.15.12:0/1249536002 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v83) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0fbc1603d0 con 0x7f0fa0004850 2024-03-29T11:12:12.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.411+0000 7f0fc3000700 1 -- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9c06aae0 msgr2=0x7f0f9c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9c06aae0 0x7f0f9c06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8000c00 tx=0x7f0fa8008030 comp rx=0 tx=0).stop 2024-03-29T11:12:12.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 -- 172.21.15.12:0/1249536002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0004850 msgr2=0x7f0fbc1584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0004850 0x7f0fbc1584a0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f0fac007290 tx=0x7f0fac007650 comp rx=0 tx=0).stop 2024-03-29T11:12:12.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 -- 172.21.15.12:0/1249536002 shutdown_connections 2024-03-29T11:12:12.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f0f9c06aae0 0x7f0f9c06cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:12.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fbc0b7960 0x7f0fbc0b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:12.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fa0004850 0x7f0fbc1584a0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:12.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 --2- 172.21.15.12:0/1249536002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fbc0ca7b0 0x7f0fbc0b7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:12.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 -- 172.21.15.12:0/1249536002 >> 172.21.15.12:0/1249536002 conn(0x7f0fbc08aff0 msgr2=0x7f0fbc15ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:12.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 -- 172.21.15.12:0/1249536002 shutdown_connections 2024-03-29T11:12:12.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.412+0000 7f0fc3000700 1 -- 172.21.15.12:0/1249536002 wait complete. 2024-03-29T11:12:12.413 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T11:12:12.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.680+0000 7f6d86800700 1 Processor -- start 2024-03-29T11:12:12.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.680+0000 7f6d86800700 1 -- start start 2024-03-29T11:12:12.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.681+0000 7f6d86800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6d8015c700 0x7f6d800adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:12.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.681+0000 7f6d86800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d800ae4a0 0x7f6d8015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:12.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.681+0000 7f6d86800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8006dd00 con 0x7f6d80155ac0 2024-03-29T11:12:12.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.681+0000 7f6d86800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8006d0f0 con 0x7f6d8015c700 2024-03-29T11:12:12.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.681+0000 7f6d86800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8006d3d0 con 0x7f6d800ae4a0 2024-03-29T11:12:12.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.681+0000 7f6d7f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6d80155ac0 0x7f6d8006c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39578/0 (socket says 172.21.15.12:39578) 2024-03-29T11:12:12.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.681+0000 7f6d7f400700 1 -- 172.21.15.12:0/2994058623 learned_addr learned my addr 172.21.15.12:0/2994058623 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.681+0000 7f6d7fe00700 1 --2- 172.21.15.12:0/2994058623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d800ae4a0 0x7f6d8015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.681+0000 7f6d7ea00700 1 --2- 172.21.15.12:0/2994058623 >> v2:172.21.15.159:3300/0 conn(0x7f6d8015c700 0x7f6d800adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.682+0000 7f6d7e000700 1 -- 172.21.15.12:0/2994058623 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158249077 0 0) 0x7f6d8006dd00 con 0x7f6d80155ac0 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.682+0000 7f6d7e000700 1 -- 172.21.15.12:0/2994058623 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d640035b0 con 0x7f6d80155ac0 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.682+0000 7f6d7e000700 1 -- 172.21.15.12:0/2994058623 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3921894204 0 0) 0x7f6d640035b0 con 0x7f6d80155ac0 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.682+0000 7f6d7fe00700 1 -- 172.21.15.12:0/2994058623 >> v2:172.21.15.159:3300/0 conn(0x7f6d8015c700 msgr2=0x7f6d800adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.682+0000 7f6d7fe00700 1 --2- 172.21.15.12:0/2994058623 >> v2:172.21.15.159:3300/0 conn(0x7f6d8015c700 0x7f6d800adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.682+0000 7f6d7fe00700 1 -- 172.21.15.12:0/2994058623 >> v1:172.21.15.12:6789/0 conn(0x7f6d80155ac0 legacy=0x7f6d8006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.682+0000 7f6d7fe00700 1 -- 172.21.15.12:0/2994058623 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8006d6b0 con 0x7f6d800ae4a0 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.682+0000 7f6d7ea00700 1 --2- 172.21.15.12:0/2994058623 >> v2:172.21.15.159:3300/0 conn(0x7f6d8015c700 0x7f6d800adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.682+0000 7f6d7fe00700 1 --2- 172.21.15.12:0/2994058623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d800ae4a0 0x7f6d8015ccb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009d90 tx=0x7f6d7800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c74695edb9233c0c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.683+0000 7f6d7e000700 1 -- 172.21.15.12:0/2994058623 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7800e070 con 0x7f6d800ae4a0 2024-03-29T11:12:12.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.683+0000 7f6d7e000700 1 -- 172.21.15.12:0/2994058623 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d78003a20 con 0x7f6d800ae4a0 2024-03-29T11:12:12.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.683+0000 7f6d7e000700 1 -- 172.21.15.12:0/2994058623 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70001670 con 0x7f6d800ae4a0 2024-03-29T11:12:12.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.683+0000 7f6d86800700 1 -- 172.21.15.12:0/2994058623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d800ae4a0 msgr2=0x7f6d8015ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.683+0000 7f6d86800700 1 --2- 172.21.15.12:0/2994058623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d800ae4a0 0x7f6d8015ccb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009d90 tx=0x7f6d7800b330 comp rx=0 tx=0).stop 2024-03-29T11:12:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.684+0000 7f6d86800700 1 -- 172.21.15.12:0/2994058623 shutdown_connections 2024-03-29T11:12:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.684+0000 7f6d86800700 1 --2- 172.21.15.12:0/2994058623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d800ae4a0 0x7f6d8015ccb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.684+0000 7f6d86800700 1 --2- 172.21.15.12:0/2994058623 >> v2:172.21.15.159:3300/0 conn(0x7f6d8015c700 0x7f6d800adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.684+0000 7f6d86800700 1 -- 172.21.15.12:0/2994058623 >> 172.21.15.12:0/2994058623 conn(0x7f6d800cf210 msgr2=0x7f6d800cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.684+0000 7f6d86800700 1 -- 172.21.15.12:0/2994058623 shutdown_connections 2024-03-29T11:12:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.684+0000 7f6d86800700 1 -- 172.21.15.12:0/2994058623 wait complete. 2024-03-29T11:12:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.686+0000 7f6d86800700 1 Processor -- start 2024-03-29T11:12:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.686+0000 7f6d86800700 1 -- start start 2024-03-29T11:12:12.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.687+0000 7f6d86800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d800ae4a0 0x7f6d8014d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:12.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.687+0000 7f6d86800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d80155ac0 0x7f6d80150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:12.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.687+0000 7f6d86800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d8015c700 0x7f6d80153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:12.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.687+0000 7f6d86800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d80151230 con 0x7f6d8015c700 2024-03-29T11:12:12.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.687+0000 7f6d86800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800aeb10 con 0x7f6d800ae4a0 2024-03-29T11:12:12.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.687+0000 7f6d86800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d80153a70 con 0x7f6d80155ac0 2024-03-29T11:12:12.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d8015c700 0x7f6d80153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:12.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d8015c700 0x7f6d80153530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48198/0 (socket says 172.21.15.12:48198) 2024-03-29T11:12:12.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7ea00700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d80155ac0 0x7f6d80150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:12.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7f400700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d800ae4a0 0x7f6d8014d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:12.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7fe00700 1 -- 172.21.15.12:0/2557811892 learned_addr learned my addr 172.21.15.12:0/2557811892 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:12.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7fe00700 1 -- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d80155ac0 msgr2=0x7f6d80150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7fe00700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d80155ac0 0x7f6d80150cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:12.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7fe00700 1 -- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d800ae4a0 msgr2=0x7f6d8014d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7fe00700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d800ae4a0 0x7f6d8014d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:12.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7fe00700 1 -- 172.21.15.12:0/2557811892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c0094a0 con 0x7f6d8015c700 2024-03-29T11:12:12.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.688+0000 7f6d7fe00700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d8015c700 0x7f6d80153530 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f6d8006d510 tx=0x7f6d78004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:12.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.689+0000 7f6d77400700 1 -- 172.21.15.12:0/2557811892 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78004540 con 0x7f6d8015c700 2024-03-29T11:12:12.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.689+0000 7f6d77400700 1 -- 172.21.15.12:0/2557811892 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d780046e0 con 0x7f6d8015c700 2024-03-29T11:12:12.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.689+0000 7f6d77400700 1 -- 172.21.15.12:0/2557811892 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7801c440 con 0x7f6d8015c700 2024-03-29T11:12:12.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.689+0000 7f6d86800700 1 -- 172.21.15.12:0/2557811892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d78009a70 con 0x7f6d8015c700 2024-03-29T11:12:12.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.689+0000 7f6d86800700 1 -- 172.21.15.12:0/2557811892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d80156020 con 0x7f6d8015c700 2024-03-29T11:12:12.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.690+0000 7f6d86800700 1 -- 172.21.15.12:0/2557811892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d80067720 con 0x7f6d8015c700 2024-03-29T11:12:12.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.691+0000 7f6d77400700 1 -- 172.21.15.12:0/2557811892 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 98) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f6d7801c5a0 con 0x7f6d8015c700 2024-03-29T11:12:12.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.692+0000 7f6d77400700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6d6006ad20 0x7f6d6006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:12.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.692+0000 7f6d77400700 1 -- 172.21.15.12:0/2557811892 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f6d7808b3b0 con 0x7f6d8015c700 2024-03-29T11:12:12.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.692+0000 7f6d7f400700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6d6006ad20 0x7f6d6006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:12.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.693+0000 7f6d7f400700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6d6006ad20 0x7f6d6006d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d70001eb0 tx=0x7f6d70001e40 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:12.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.697+0000 7f6d77400700 1 -- 172.21.15.12:0/2557811892 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6d78056260 con 0x7f6d8015c700 2024-03-29T11:12:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:12.912+0000 7f6d86800700 1 -- 172.21.15.12:0/2557811892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6d8006e950 con 0x7f6d8015c700 2024-03-29T11:12:13.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.771+0000 7f6d77400700 1 -- 172.21.15.12:0/2557811892 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v181) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6d78059880 con 0x7f6d8015c700 2024-03-29T11:12:13.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 -- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6d6006ad20 msgr2=0x7f6d6006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6d6006ad20 0x7f6d6006d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d70001eb0 tx=0x7f6d70001e40 comp rx=0 tx=0).stop 2024-03-29T11:12:13.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 -- 172.21.15.12:0/2557811892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d8015c700 msgr2=0x7f6d80153530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d8015c700 0x7f6d80153530 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f6d8006d510 tx=0x7f6d78004ad0 comp rx=0 tx=0).stop 2024-03-29T11:12:13.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 -- 172.21.15.12:0/2557811892 shutdown_connections 2024-03-29T11:12:13.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6d6006ad20 0x7f6d6006d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:13.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d8015c700 0x7f6d80153530 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:13.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d800ae4a0 0x7f6d8014d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:13.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 --2- 172.21.15.12:0/2557811892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d80155ac0 0x7f6d80150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:13.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 -- 172.21.15.12:0/2557811892 >> 172.21.15.12:0/2557811892 conn(0x7f6d800cf210 msgr2=0x7f6d800b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:13.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 -- 172.21.15.12:0/2557811892 shutdown_connections 2024-03-29T11:12:13.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:13.774+0000 7f6d86800700 1 -- 172.21.15.12:0/2557811892 wait complete. 2024-03-29T11:12:13.754 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T11:12:13.775 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T11:12:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.011+0000 7f79a5400700 1 Processor -- start 2024-03-29T11:12:13.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.012+0000 7f79a5400700 1 -- start start 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.013+0000 7f79a5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a00d0bd0 0x7f79a015e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.013+0000 7f79a5400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f79a00b3120 0x7f79a006a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.013+0000 7f79a5400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a015ee10 con 0x7f79a006acd0 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.013+0000 7f79a5400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79a006b140 con 0x7f79a00b3120 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.013+0000 7f79a5400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a015de10 con 0x7f79a00d0bd0 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.013+0000 7f799de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f79a00b3120 0x7f79a006a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.013+0000 7f799de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f79a00b3120 0x7f79a006a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48204/0 (socket says 172.21.15.12:48204) 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.013+0000 7f799de00700 1 -- 172.21.15.12:0/2791914937 learned_addr learned my addr 172.21.15.12:0/2791914937 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.013+0000 7f799e800700 1 --2- 172.21.15.12:0/2791914937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a00d0bd0 0x7f79a015e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.014+0000 7f799de00700 1 -- 172.21.15.12:0/2791914937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a00d0bd0 msgr2=0x7f79a015e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.014+0000 7f799de00700 1 --2- 172.21.15.12:0/2791914937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a00d0bd0 0x7f79a015e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:13.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.014+0000 7f799de00700 1 -- 172.21.15.12:0/2791914937 >> v1:172.21.15.12:6789/0 conn(0x7f79a006acd0 legacy=0x7f79a006dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.014+0000 7f799de00700 1 -- 172.21.15.12:0/2791914937 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a00b4fd0 con 0x7f79a00b3120 2024-03-29T11:12:13.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.014+0000 7f799de00700 1 --2- 172.21.15.12:0/2791914937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 0x7f79a006a790 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f798c0098f0 tx=0x7f798c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeed93322302e9ab server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.014+0000 7f799d400700 1 -- 172.21.15.12:0/2791914937 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c011830 con 0x7f79a00b3120 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.014+0000 7f799d400700 1 -- 172.21.15.12:0/2791914937 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c011990 con 0x7f79a00b3120 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.014+0000 7f799d400700 1 -- 172.21.15.12:0/2791914937 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c011d20 con 0x7f79a00b3120 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.015+0000 7f79a5400700 1 -- 172.21.15.12:0/2791914937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 msgr2=0x7f79a006a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.015+0000 7f79a5400700 1 --2- 172.21.15.12:0/2791914937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 0x7f79a006a790 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f798c0098f0 tx=0x7f798c009c00 comp rx=0 tx=0).stop 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.015+0000 7f79a5400700 1 -- 172.21.15.12:0/2791914937 shutdown_connections 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.015+0000 7f79a5400700 1 --2- 172.21.15.12:0/2791914937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a00d0bd0 0x7f79a015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.015+0000 7f79a5400700 1 --2- 172.21.15.12:0/2791914937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 0x7f79a006a790 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.015+0000 7f79a5400700 1 -- 172.21.15.12:0/2791914937 >> 172.21.15.12:0/2791914937 conn(0x7f79a00b5640 msgr2=0x7f79a00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.015+0000 7f79a5400700 1 -- 172.21.15.12:0/2791914937 shutdown_connections 2024-03-29T11:12:13.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.015+0000 7f79a5400700 1 -- 172.21.15.12:0/2791914937 wait complete. 2024-03-29T11:12:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.017+0000 7f79a5400700 1 Processor -- start 2024-03-29T11:12:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.017+0000 7f79a5400700 1 -- start start 2024-03-29T11:12:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.018+0000 7f79a5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a006acd0 0x7f79a0146cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.018+0000 7f79a5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 0x7f79a0159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.018+0000 7f79a5400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79a00d0bd0 0x7f79a015a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.018+0000 7f79a5400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a01471f0 con 0x7f79a00b3120 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.018+0000 7f79a5400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a0159440 con 0x7f79a00d0bd0 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.018+0000 7f79a5400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a0154940 con 0x7f79a006acd0 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.018+0000 7f799de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 0x7f79a0159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.018+0000 7f799de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 0x7f79a0159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48218/0 (socket says 172.21.15.12:48218) 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.018+0000 7f799de00700 1 -- 172.21.15.12:0/1005383817 learned_addr learned my addr 172.21.15.12:0/1005383817 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.018+0000 7f799e800700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a006acd0 0x7f79a0146cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f799f200700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79a00d0bd0 0x7f79a015a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f799de00700 1 -- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a006acd0 msgr2=0x7f79a0146cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f799de00700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a006acd0 0x7f79a0146cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f799de00700 1 -- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79a00d0bd0 msgr2=0x7f79a015a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f799de00700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79a00d0bd0 0x7f79a015a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f799de00700 1 -- 172.21.15.12:0/1005383817 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c009550 con 0x7f79a00b3120 2024-03-29T11:12:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f799e800700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a006acd0 0x7f79a0146cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f799de00700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 0x7f79a0159ac0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f798c000c00 tx=0x7f798c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f7996a00700 1 -- 172.21.15.12:0/1005383817 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c004570 con 0x7f79a00b3120 2024-03-29T11:12:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f7996a00700 1 -- 172.21.15.12:0/1005383817 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c01f070 con 0x7f79a00b3120 2024-03-29T11:12:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f7996a00700 1 -- 172.21.15.12:0/1005383817 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c0038a0 con 0x7f79a00b3120 2024-03-29T11:12:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f79a5400700 1 -- 172.21.15.12:0/1005383817 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a0154c20 con 0x7f79a00b3120 2024-03-29T11:12:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.019+0000 7f79a5400700 1 -- 172.21.15.12:0/1005383817 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a0147700 con 0x7f79a00b3120 2024-03-29T11:12:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.020+0000 7f79a5400700 1 -- 172.21.15.12:0/1005383817 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a00b3970 con 0x7f79a00b3120 2024-03-29T11:12:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.021+0000 7f7996a00700 1 -- 172.21.15.12:0/1005383817 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 98) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f798c021030 con 0x7f79a00b3120 2024-03-29T11:12:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.022+0000 7f7996a00700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f798006ad20 0x7f798006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.022+0000 7f7996a00700 1 -- 172.21.15.12:0/1005383817 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6003+0+0 (secure 0 0 0) 0x7f798c08ab00 con 0x7f79a00b3120 2024-03-29T11:12:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.023+0000 7f799e800700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f798006ad20 0x7f798006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.023+0000 7f799e800700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f798006ad20 0x7f798006d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7990009c30 tx=0x7f7990008040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.027+0000 7f7996a00700 1 -- 172.21.15.12:0/1005383817 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f798c08ce60 con 0x7f79a00b3120 2024-03-29T11:12:14.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.039+0000 7f7996a00700 1 -- 172.21.15.12:0/1005383817 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f798c056040 con 0x7f79a00b3120 2024-03-29T11:12:14.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:14.247+0000 7f79a5400700 1 -- 172.21.15.12:0/1005383817 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f79a0147890 con 0x7f79a00b3120 2024-03-29T11:12:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.038+0000 7f7996a00700 1 -- 172.21.15.12:0/1005383817 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v182) v1 ==== 169+0+0 (secure 0 0 0) 0x7f798c05d370 con 0x7f79a00b3120 2024-03-29T11:12:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 -- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f798006ad20 msgr2=0x7f798006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f798006ad20 0x7f798006d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7990009c30 tx=0x7f7990008040 comp rx=0 tx=0).stop 2024-03-29T11:12:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 -- 172.21.15.12:0/1005383817 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 msgr2=0x7f79a0159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 0x7f79a0159ac0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f798c000c00 tx=0x7f798c004750 comp rx=0 tx=0).stop 2024-03-29T11:12:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 -- 172.21.15.12:0/1005383817 shutdown_connections 2024-03-29T11:12:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f798006ad20 0x7f798006d1e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79a00d0bd0 0x7f79a015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79a006acd0 0x7f79a0146cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 --2- 172.21.15.12:0/1005383817 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79a00b3120 0x7f79a0159ac0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 -- 172.21.15.12:0/1005383817 >> 172.21.15.12:0/1005383817 conn(0x7f79a00b5640 msgr2=0x7f79a015f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 -- 172.21.15.12:0/1005383817 shutdown_connections 2024-03-29T11:12:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.041+0000 7f79a5400700 1 -- 172.21.15.12:0/1005383817 wait complete. 2024-03-29T11:12:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T11:12:15.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T11:12:15.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.267+0000 7f7e38800700 1 Processor -- start 2024-03-29T11:12:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.268+0000 7f7e38800700 1 -- start start 2024-03-29T11:12:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.269+0000 7f7e38800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e300d2540 0x7f7e301687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.269+0000 7f7e38800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e30161420 con 0x7f7e300d16a0 2024-03-29T11:12:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.269+0000 7f7e38800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e30161700 con 0x7f7e3015d9d0 2024-03-29T11:12:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.269+0000 7f7e38800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e300d4340 con 0x7f7e300d2540 2024-03-29T11:12:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.270+0000 7f7e35400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7e300d16a0 0x7f7e300d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39602/0 (socket says 172.21.15.12:39602) 2024-03-29T11:12:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.270+0000 7f7e35400700 1 -- 172.21.15.12:0/1514322818 learned_addr learned my addr 172.21.15.12:0/1514322818 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.270+0000 7f7e36800700 1 --2- 172.21.15.12:0/1514322818 >> v2:172.21.15.12:3300/0 conn(0x7f7e300d2540 0x7f7e301687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.270+0000 7f7e36800700 1 -- 172.21.15.12:0/1514322818 >> v1:172.21.15.159:6789/0 conn(0x7f7e3015d9d0 legacy=0x7f7e300d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.270+0000 7f7e36800700 1 -- 172.21.15.12:0/1514322818 >> v1:172.21.15.12:6789/0 conn(0x7f7e300d16a0 legacy=0x7f7e300d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.270+0000 7f7e36800700 1 -- 172.21.15.12:0/1514322818 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e301619e0 con 0x7f7e300d2540 2024-03-29T11:12:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.270+0000 7f7e36800700 1 --2- 172.21.15.12:0/1514322818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 0x7f7e301687b0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009d90 tx=0x7f7e2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dee408ad1e8994 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.271+0000 7f7e34a00700 1 -- 172.21.15.12:0/1514322818 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c00e070 con 0x7f7e300d2540 2024-03-29T11:12:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.271+0000 7f7e34a00700 1 -- 172.21.15.12:0/1514322818 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2c00bbd0 con 0x7f7e300d2540 2024-03-29T11:12:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.271+0000 7f7e34a00700 1 -- 172.21.15.12:0/1514322818 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c012430 con 0x7f7e300d2540 2024-03-29T11:12:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.271+0000 7f7e38800700 1 -- 172.21.15.12:0/1514322818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 msgr2=0x7f7e301687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.271+0000 7f7e38800700 1 --2- 172.21.15.12:0/1514322818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 0x7f7e301687b0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009d90 tx=0x7f7e2c00b330 comp rx=0 tx=0).stop 2024-03-29T11:12:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.271+0000 7f7e38800700 1 -- 172.21.15.12:0/1514322818 shutdown_connections 2024-03-29T11:12:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.271+0000 7f7e38800700 1 --2- 172.21.15.12:0/1514322818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 0x7f7e301687b0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.271+0000 7f7e38800700 1 -- 172.21.15.12:0/1514322818 >> 172.21.15.12:0/1514322818 conn(0x7f7e3008aff0 msgr2=0x7f7e300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.271+0000 7f7e38800700 1 -- 172.21.15.12:0/1514322818 shutdown_connections 2024-03-29T11:12:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.271+0000 7f7e38800700 1 -- 172.21.15.12:0/1514322818 wait complete. 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.274+0000 7f7e38800700 1 Processor -- start 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.274+0000 7f7e38800700 1 -- start start 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.275+0000 7f7e38800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e300d16a0 0x7f7e300b7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.275+0000 7f7e38800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 0x7f7e301e82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.275+0000 7f7e38800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e3015d9d0 0x7f7e301e8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.275+0000 7f7e38800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300cae20 con 0x7f7e300d2540 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.275+0000 7f7e38800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300cb0a0 con 0x7f7e300d16a0 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.275+0000 7f7e38800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300cb380 con 0x7f7e3015d9d0 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.275+0000 7f7e35400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 0x7f7e301e82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.275+0000 7f7e35400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 0x7f7e301e82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48228/0 (socket says 172.21.15.12:48228) 2024-03-29T11:12:15.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.275+0000 7f7e35400700 1 -- 172.21.15.12:0/624853365 learned_addr learned my addr 172.21.15.12:0/624853365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e35e00700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e300d16a0 0x7f7e300b7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e35400700 1 -- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e3015d9d0 msgr2=0x7f7e301e8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e35400700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e3015d9d0 0x7f7e301e8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e35400700 1 -- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e300d16a0 msgr2=0x7f7e300b7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e35400700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e300d16a0 0x7f7e300b7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e35400700 1 -- 172.21.15.12:0/624853365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c009a70 con 0x7f7e300d2540 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e35e00700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e300d16a0 0x7f7e300b7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e35400700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 0x7f7e301e82c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000c4e0 tx=0x7f7e2000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e2a000700 1 -- 172.21.15.12:0/624853365 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e200074e0 con 0x7f7e300d2540 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e2a000700 1 -- 172.21.15.12:0/624853365 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e20007640 con 0x7f7e300d2540 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e38800700 1 -- 172.21.15.12:0/624853365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e300cb660 con 0x7f7e300d2540 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e2a000700 1 -- 172.21.15.12:0/624853365 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20005a50 con 0x7f7e300d2540 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.276+0000 7f7e38800700 1 -- 172.21.15.12:0/624853365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e300b8400 con 0x7f7e300d2540 2024-03-29T11:12:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.277+0000 7f7e38800700 1 -- 172.21.15.12:0/624853365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e300d1ca0 con 0x7f7e300d2540 2024-03-29T11:12:15.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.278+0000 7f7e2a000700 1 -- 172.21.15.12:0/624853365 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7e20005bb0 con 0x7f7e300d2540 2024-03-29T11:12:15.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.279+0000 7f7e2a000700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e1406adb0 0x7f7e1406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:15.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.279+0000 7f7e2a000700 1 -- 172.21.15.12:0/624853365 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f7e20088950 con 0x7f7e300d2540 2024-03-29T11:12:15.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.280+0000 7f7e35e00700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e1406adb0 0x7f7e1406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:15.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.281+0000 7f7e35e00700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e1406adb0 0x7f7e1406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e24000f60 tx=0x7f7e24009040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:15.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.284+0000 7f7e2a000700 1 -- 172.21.15.12:0/624853365 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7e20053a80 con 0x7f7e300d2540 2024-03-29T11:12:15.442 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:12:15.447 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:12:15.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.497+0000 7f7e38800700 1 -- 172.21.15.12:0/624853365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f7e300cbe60 con 0x7f7e300d2540 2024-03-29T11:12:15.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.502+0000 7f7e2a000700 1 -- 172.21.15.12:0/624853365 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v61) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7e300cbe60 con 0x7f7e300d2540 2024-03-29T11:12:15.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 -- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e1406adb0 msgr2=0x7f7e1406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e1406adb0 0x7f7e1406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e24000f60 tx=0x7f7e24009040 comp rx=0 tx=0).stop 2024-03-29T11:12:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 -- 172.21.15.12:0/624853365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 msgr2=0x7f7e301e82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 0x7f7e301e82c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000c4e0 tx=0x7f7e2000c8a0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 -- 172.21.15.12:0/624853365 shutdown_connections 2024-03-29T11:12:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f7e1406adb0 0x7f7e1406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e3015d9d0 0x7f7e301e8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e300d16a0 0x7f7e300b7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 --2- 172.21.15.12:0/624853365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e300d2540 0x7f7e301e82c0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 -- 172.21.15.12:0/624853365 >> 172.21.15.12:0/624853365 conn(0x7f7e3008aff0 msgr2=0x7f7e3006cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 -- 172.21.15.12:0/624853365 shutdown_connections 2024-03-29T11:12:15.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.504+0000 7f7e38800700 1 -- 172.21.15.12:0/624853365 wait complete. 2024-03-29T11:12:15.502 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T11:12:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.803+0000 7feeefa00700 1 Processor -- start 2024-03-29T11:12:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.803+0000 7feeefa00700 1 -- start start 2024-03-29T11:12:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.804+0000 7feeefa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feee815dba0 0x7feee815e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.804+0000 7feeefa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee815ec10 con 0x7feee8071080 2024-03-29T11:12:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.804+0000 7feeefa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee815e650 con 0x7feee806ab80 2024-03-29T11:12:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.804+0000 7feeefa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee81581a0 con 0x7feee815dba0 2024-03-29T11:12:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.804+0000 7feee7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feee815dba0 0x7feee815e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.804+0000 7feee7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feee815dba0 0x7feee815e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48238/0 (socket says 172.21.15.12:48238) 2024-03-29T11:12:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.804+0000 7feee7e00700 1 -- 172.21.15.12:0/1177771567 learned_addr learned my addr 172.21.15.12:0/1177771567 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.804+0000 7feeed000700 1 --1- 172.21.15.12:0/1177771567 >> v1:172.21.15.12:6789/0 conn(0x7feee8071080 0x7feee8161b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39604/0 (socket says 172.21.15.12:39604) 2024-03-29T11:12:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.805+0000 7feee7e00700 1 -- 172.21.15.12:0/1177771567 >> v1:172.21.15.159:6789/0 conn(0x7feee806ab80 legacy=0x7feee806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.805+0000 7feee7e00700 1 -- 172.21.15.12:0/1177771567 >> v1:172.21.15.12:6789/0 conn(0x7feee8071080 legacy=0x7feee8161b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.805+0000 7feee7e00700 1 -- 172.21.15.12:0/1177771567 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee815e930 con 0x7feee815dba0 2024-03-29T11:12:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.805+0000 7feee7e00700 1 --2- 172.21.15.12:0/1177771567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 0x7feee815e000 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7feed40098f0 tx=0x7feed4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ed6ab16239fed3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.805+0000 7feee7400700 1 -- 172.21.15.12:0/1177771567 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed400b8a0 con 0x7feee815dba0 2024-03-29T11:12:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.805+0000 7feee7400700 1 -- 172.21.15.12:0/1177771567 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed400ba00 con 0x7feee815dba0 2024-03-29T11:12:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.805+0000 7feee7400700 1 -- 172.21.15.12:0/1177771567 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed400bd90 con 0x7feee815dba0 2024-03-29T11:12:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.806+0000 7feeefa00700 1 -- 172.21.15.12:0/1177771567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 msgr2=0x7feee815e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.806+0000 7feeefa00700 1 --2- 172.21.15.12:0/1177771567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 0x7feee815e000 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7feed40098f0 tx=0x7feed4009c00 comp rx=0 tx=0).stop 2024-03-29T11:12:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.806+0000 7feeefa00700 1 -- 172.21.15.12:0/1177771567 shutdown_connections 2024-03-29T11:12:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.806+0000 7feeefa00700 1 --2- 172.21.15.12:0/1177771567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 0x7feee815e000 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.806+0000 7feeefa00700 1 -- 172.21.15.12:0/1177771567 >> 172.21.15.12:0/1177771567 conn(0x7feee80b1230 msgr2=0x7feee80b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.806+0000 7feeefa00700 1 -- 172.21.15.12:0/1177771567 shutdown_connections 2024-03-29T11:12:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.806+0000 7feeefa00700 1 -- 172.21.15.12:0/1177771567 wait complete. 2024-03-29T11:12:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.808+0000 7feeefa00700 1 Processor -- start 2024-03-29T11:12:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.808+0000 7feeefa00700 1 -- start start 2024-03-29T11:12:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeefa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feee806ab80 0x7feee80ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeefa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feee8071080 0x7feee81f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeefa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 0x7feee81f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeefa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee8161720 con 0x7feee815dba0 2024-03-29T11:12:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeefa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee81516d0 con 0x7feee8071080 2024-03-29T11:12:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeefa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee80ac220 con 0x7feee806ab80 2024-03-29T11:12:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeeda00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 0x7feee81f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeeda00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 0x7feee81f3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48248/0 (socket says 172.21.15.12:48248) 2024-03-29T11:12:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeeda00700 1 -- 172.21.15.12:0/141619920 learned_addr learned my addr 172.21.15.12:0/141619920 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeed000700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feee806ab80 0x7feee80ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feee7e00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feee8071080 0x7feee81f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeeda00700 1 -- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feee806ab80 msgr2=0x7feee80ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeeda00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feee806ab80 0x7feee80ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeeda00700 1 -- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feee8071080 msgr2=0x7feee81f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeeda00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feee8071080 0x7feee81f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.809+0000 7feeeda00700 1 -- 172.21.15.12:0/141619920 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed4009550 con 0x7feee815dba0 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.810+0000 7feeeda00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 0x7feee81f3cd0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7feee000cc50 tx=0x7feee000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.810+0000 7feee4c00700 1 -- 172.21.15.12:0/141619920 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0003c00 con 0x7feee815dba0 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.810+0000 7feee4c00700 1 -- 172.21.15.12:0/141619920 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee0003d60 con 0x7feee815dba0 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.810+0000 7feeefa00700 1 -- 172.21.15.12:0/141619920 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee80ac500 con 0x7feee815dba0 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.810+0000 7feee4c00700 1 -- 172.21.15.12:0/141619920 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee000aab0 con 0x7feee815dba0 2024-03-29T11:12:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.810+0000 7feeefa00700 1 -- 172.21.15.12:0/141619920 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee80acac0 con 0x7feee815dba0 2024-03-29T11:12:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.811+0000 7feeefa00700 1 -- 172.21.15.12:0/141619920 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee8067720 con 0x7feee815dba0 2024-03-29T11:12:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.812+0000 7feee4c00700 1 -- 172.21.15.12:0/141619920 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7feee000ac10 con 0x7feee815dba0 2024-03-29T11:12:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.813+0000 7feee4c00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feec806adb0 0x7feec806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.813+0000 7feee4c00700 1 -- 172.21.15.12:0/141619920 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5589+0+0 (secure 0 0 0) 0x7feee0089430 con 0x7feee815dba0 2024-03-29T11:12:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.813+0000 7feeed000700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feec806adb0 0x7feec806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.814+0000 7feeed000700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feec806adb0 0x7feec806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feedc000f60 tx=0x7feedc007ed0 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:15.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:15.817+0000 7feee4c00700 1 -- 172.21.15.12:0/141619920 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feee00544d0 con 0x7feee815dba0 2024-03-29T11:12:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.038+0000 7feeefa00700 1 -- 172.21.15.12:0/141619920 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7feee80b4260 con 0x7feee815dba0 2024-03-29T11:12:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.044+0000 7feee4c00700 1 -- 172.21.15.12:0/141619920 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7feee0057af0 con 0x7feee815dba0 2024-03-29T11:12:16.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.046+0000 7feeefa00700 1 -- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feec806adb0 msgr2=0x7feec806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.046+0000 7feeefa00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feec806adb0 0x7feec806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feedc000f60 tx=0x7feedc007ed0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.046+0000 7feeefa00700 1 -- 172.21.15.12:0/141619920 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 msgr2=0x7feee81f3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.046+0000 7feeefa00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 0x7feee81f3cd0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7feee000cc50 tx=0x7feee000cf60 comp rx=0 tx=0).stop 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.046+0000 7feeefa00700 1 -- 172.21.15.12:0/141619920 shutdown_connections 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.046+0000 7feeefa00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7feec806adb0 0x7feec806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.046+0000 7feeefa00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feee815dba0 0x7feee81f3cd0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.046+0000 7feeefa00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feee806ab80 0x7feee80ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.046+0000 7feeefa00700 1 --2- 172.21.15.12:0/141619920 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feee8071080 0x7feee81f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.046+0000 7feeefa00700 1 -- 172.21.15.12:0/141619920 >> 172.21.15.12:0/141619920 conn(0x7feee80b1230 msgr2=0x7feee80cf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.047+0000 7feeefa00700 1 -- 172.21.15.12:0/141619920 shutdown_connections 2024-03-29T11:12:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.047+0000 7feeefa00700 1 -- 172.21.15.12:0/141619920 wait complete. 2024-03-29T11:12:16.042 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T11:12:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.271+0000 7f076b000700 1 Processor -- start 2024-03-29T11:12:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.271+0000 7f076b000700 1 -- start start 2024-03-29T11:12:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.272+0000 7f076b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07640d16a0 0x7f07640d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.272+0000 7f076b000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0764161420 con 0x7f076415d9d0 2024-03-29T11:12:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.272+0000 7f076b000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0764161700 con 0x7f07640d2370 2024-03-29T11:12:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.272+0000 7f076b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07640d2820 con 0x7f07640d16a0 2024-03-29T11:12:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.272+0000 7f0763e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f076415d9d0 0x7f07640d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39616/0 (socket says 172.21.15.12:39616) 2024-03-29T11:12:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.272+0000 7f0763e00700 1 -- 172.21.15.12:0/3999791555 learned_addr learned my addr 172.21.15.12:0/3999791555 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0763400700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07640d16a0 0x7f07640d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201065477 0 0) 0x7f0764161420 con 0x7f076415d9d0 2024-03-29T11:12:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07480035b0 con 0x7f076415d9d0 2024-03-29T11:12:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766496718 0 0) 0x7f0764161700 con 0x7f07640d2370 2024-03-29T11:12:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0764161420 con 0x7f07640d2370 2024-03-29T11:12:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2911574175 0 0) 0x7f07480035b0 con 0x7f076415d9d0 2024-03-29T11:12:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07640d16a0 msgr2=0x7f07640d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07640d16a0 0x7f07640d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 >> v1:172.21.15.159:6789/0 conn(0x7f07640d2370 legacy=0x7f07640ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07641619e0 con 0x7f076415d9d0 2024-03-29T11:12:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0754001670 con 0x7f076415d9d0 2024-03-29T11:12:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 >> v1:172.21.15.12:6789/0 conn(0x7f076415d9d0 legacy=0x7f07640d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748003bc0 0x7f0748003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0748004850 0x7f0748004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07640d16a0 0x7f0748005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07640d2820 con 0x7f0748003bc0 2024-03-29T11:12:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074800bac0 con 0x7f0748004850 2024-03-29T11:12:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.273+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074800bc50 con 0x7f07640d16a0 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0763e00700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748003bc0 0x7f0748003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0763400700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0748004850 0x7f0748004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0769000700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07640d16a0 0x7f0748005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0763e00700 1 -- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07640d16a0 msgr2=0x7f0748005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0763e00700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07640d16a0 0x7f0748005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0763e00700 1 -- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0748004850 msgr2=0x7f0748004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0763e00700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0748004850 0x7f0748004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0763e00700 1 -- 172.21.15.12:0/3999791555 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07480035b0 con 0x7f0748003bc0 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0769000700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07640d16a0 0x7f0748005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0763400700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0748004850 0x7f0748004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0763e00700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748003bc0 0x7f0748003fc0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f075400c210 tx=0x7f0748005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7457f178a364b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075400cd70 con 0x7f0748003bc0 2024-03-29T11:12:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0754001af0 con 0x7f0748003bc0 2024-03-29T11:12:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.274+0000 7f0762a00700 1 -- 172.21.15.12:0/3999791555 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754005ba0 con 0x7f0748003bc0 2024-03-29T11:12:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.275+0000 7f076b000700 1 -- 172.21.15.12:0/3999791555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748003bc0 msgr2=0x7f0748003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.275+0000 7f076b000700 1 --2- 172.21.15.12:0/3999791555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748003bc0 0x7f0748003fc0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f075400c210 tx=0x7f0748005880 comp rx=0 tx=0).stop 2024-03-29T11:12:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.275+0000 7f0769000700 1 -- 172.21.15.12:0/3999791555 reap_dead start 2024-03-29T11:12:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.275+0000 7f076b000700 1 -- 172.21.15.12:0/3999791555 shutdown_connections 2024-03-29T11:12:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.275+0000 7f076b000700 1 -- 172.21.15.12:0/3999791555 >> 172.21.15.12:0/3999791555 conn(0x7f076408aff0 msgr2=0x7f07640d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.275+0000 7f076b000700 1 -- 172.21.15.12:0/3999791555 shutdown_connections 2024-03-29T11:12:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.275+0000 7f076b000700 1 -- 172.21.15.12:0/3999791555 wait complete. 2024-03-29T11:12:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.277+0000 7f076b000700 1 Processor -- start 2024-03-29T11:12:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.277+0000 7f076b000700 1 -- start start 2024-03-29T11:12:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f076b000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0748003bc0 0x7f076415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f076b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07641f3ee0 0x7f0764158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f076b000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07641550f0 0x7f07640b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f076b000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07640b6020 con 0x7f07641550f0 2024-03-29T11:12:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f076b000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076415e2f0 con 0x7f0748003bc0 2024-03-29T11:12:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f076b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07641f4460 con 0x7f07641f3ee0 2024-03-29T11:12:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f0769000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07641550f0 0x7f07640b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f0769000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07641550f0 0x7f07640b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48260/0 (socket says 172.21.15.12:48260) 2024-03-29T11:12:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f0769000700 1 -- 172.21.15.12:0/2876577351 learned_addr learned my addr 172.21.15.12:0/2876577351 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f0763e00700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0748003bc0 0x7f076415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.278+0000 7f0763400700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07641f3ee0 0x7f0764158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f0769000700 1 -- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07641f3ee0 msgr2=0x7f0764158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f0769000700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07641f3ee0 0x7f0764158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f0769000700 1 -- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0748003bc0 msgr2=0x7f076415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f0769000700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0748003bc0 0x7f076415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f0769000700 1 -- 172.21.15.12:0/2876577351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07641f4740 con 0x7f07641550f0 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f0763400700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07641f3ee0 0x7f0764158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f0769000700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07641550f0 0x7f07640b49d0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f075c001a50 tx=0x7f075c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f075be00700 1 -- 172.21.15.12:0/2876577351 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c003c10 con 0x7f07641550f0 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f075be00700 1 -- 172.21.15.12:0/2876577351 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c003d90 con 0x7f07641550f0 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f075be00700 1 -- 172.21.15.12:0/2876577351 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c005310 con 0x7f07641550f0 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f076b000700 1 -- 172.21.15.12:0/2876577351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07641f4a20 con 0x7f07641550f0 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.279+0000 7f076b000700 1 -- 172.21.15.12:0/2876577351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07641594e0 con 0x7f07641550f0 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.281+0000 7f076b000700 1 -- 172.21.15.12:0/2876577351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07640b6190 con 0x7f07641550f0 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.281+0000 7f075be00700 1 -- 172.21.15.12:0/2876577351 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f075c005470 con 0x7f07641550f0 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.282+0000 7f075be00700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074406adb0 0x7f074406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.282+0000 7f075be00700 1 -- 172.21.15.12:0/2876577351 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f075c0894a0 con 0x7f07641550f0 2024-03-29T11:12:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.283+0000 7f0763e00700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074406adb0 0x7f074406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.283+0000 7f0763e00700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074406adb0 0x7f074406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f07540018b0 tx=0x7f075400a000 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.287+0000 7f075be00700 1 -- 172.21.15.12:0/2876577351 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f07640b6190 con 0x7f07641550f0 2024-03-29T11:12:16.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.500+0000 7f076b000700 1 -- 172.21.15.12:0/2876577351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f07640b6190 con 0x7f07641550f0 2024-03-29T11:12:16.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.506+0000 7f075be00700 1 -- 172.21.15.12:0/2876577351 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v63) v1 ==== 120+0+0 (secure 0 0 0) 0x7f07640b6190 con 0x7f07641550f0 2024-03-29T11:12:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.508+0000 7f076b000700 1 -- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074406adb0 msgr2=0x7f074406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074406adb0 0x7f074406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f07540018b0 tx=0x7f075400a000 comp rx=0 tx=0).stop 2024-03-29T11:12:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 -- 172.21.15.12:0/2876577351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07641550f0 msgr2=0x7f07640b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07641550f0 0x7f07640b49d0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f075c001a50 tx=0x7f075c00ee90 comp rx=0 tx=0).stop 2024-03-29T11:12:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 -- 172.21.15.12:0/2876577351 shutdown_connections 2024-03-29T11:12:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f074406adb0 0x7f074406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07641550f0 0x7f07640b49d0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0748003bc0 0x7f076415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 --2- 172.21.15.12:0/2876577351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07641f3ee0 0x7f0764158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 -- 172.21.15.12:0/2876577351 >> 172.21.15.12:0/2876577351 conn(0x7f076408aff0 msgr2=0x7f07640ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 -- 172.21.15.12:0/2876577351 shutdown_connections 2024-03-29T11:12:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.509+0000 7f076b000700 1 -- 172.21.15.12:0/2876577351 wait complete. 2024-03-29T11:12:16.505 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-03-29T11:12:16.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.726+0000 7fe032800700 1 Processor -- start 2024-03-29T11:12:16.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.726+0000 7fe032800700 1 -- start start 2024-03-29T11:12:16.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.727+0000 7fe032800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe02c160f10 0x7fe02c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.727+0000 7fe032800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe02c0cf230 0x7fe02c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.727+0000 7fe032800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c0ceda0 con 0x7fe02c161590 2024-03-29T11:12:16.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.727+0000 7fe032800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02c0d23d0 con 0x7fe02c160f10 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.727+0000 7fe032800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c1612f0 con 0x7fe02c0cf230 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.727+0000 7fe02b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe02c160f10 0x7fe02c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.727+0000 7fe02b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe02c160f10 0x7fe02c0d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48266/0 (socket says 172.21.15.12:48266) 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.727+0000 7fe02b400700 1 -- 172.21.15.12:0/1375407858 learned_addr learned my addr 172.21.15.12:0/1375407858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.728+0000 7fe02aa00700 1 --2- 172.21.15.12:0/1375407858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe02c0cf230 0x7fe02c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.728+0000 7fe02b400700 1 -- 172.21.15.12:0/1375407858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe02c0cf230 msgr2=0x7fe02c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.728+0000 7fe02b400700 1 --2- 172.21.15.12:0/1375407858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe02c0cf230 0x7fe02c0d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.728+0000 7fe02b400700 1 -- 172.21.15.12:0/1375407858 >> v1:172.21.15.159:6789/0 conn(0x7fe02c161590 legacy=0x7fe02c0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.728+0000 7fe02b400700 1 -- 172.21.15.12:0/1375407858 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c0ce470 con 0x7fe02c160f10 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.728+0000 7fe02b400700 1 --2- 172.21.15.12:0/1375407858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c160f10 0x7fe02c0d18b0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fe01c0098f0 tx=0x7fe01c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53c8e21403eaf38 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.728+0000 7fe02a000700 1 -- 172.21.15.12:0/1375407858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c00b8a0 con 0x7fe02c160f10 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.728+0000 7fe02a000700 1 -- 172.21.15.12:0/1375407858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c00ba00 con 0x7fe02c160f10 2024-03-29T11:12:16.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.729+0000 7fe02a000700 1 -- 172.21.15.12:0/1375407858 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c00bd90 con 0x7fe02c160f10 2024-03-29T11:12:16.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.729+0000 7fe032800700 1 -- 172.21.15.12:0/1375407858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c160f10 msgr2=0x7fe02c0d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.729+0000 7fe032800700 1 --2- 172.21.15.12:0/1375407858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c160f10 0x7fe02c0d18b0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fe01c0098f0 tx=0x7fe01c009c00 comp rx=0 tx=0).stop 2024-03-29T11:12:16.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.729+0000 7fe032800700 1 -- 172.21.15.12:0/1375407858 shutdown_connections 2024-03-29T11:12:16.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.729+0000 7fe032800700 1 --2- 172.21.15.12:0/1375407858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c160f10 0x7fe02c0d18b0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.729+0000 7fe032800700 1 --2- 172.21.15.12:0/1375407858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe02c0cf230 0x7fe02c0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.729+0000 7fe032800700 1 -- 172.21.15.12:0/1375407858 >> 172.21.15.12:0/1375407858 conn(0x7fe02c08aff0 msgr2=0x7fe02c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:16.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.729+0000 7fe032800700 1 -- 172.21.15.12:0/1375407858 shutdown_connections 2024-03-29T11:12:16.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.729+0000 7fe032800700 1 -- 172.21.15.12:0/1375407858 wait complete. 2024-03-29T11:12:16.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.731+0000 7fe032800700 1 Processor -- start 2024-03-29T11:12:16.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.731+0000 7fe032800700 1 -- start start 2024-03-29T11:12:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe032800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe02c0cf230 0x7fe02c159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe032800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe02c160f10 0x7fe02c0b81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe032800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c161590 0x7fe02c1580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe032800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c150320 con 0x7fe02c161590 2024-03-29T11:12:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe032800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c154d10 con 0x7fe02c0cf230 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe032800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0b7700 con 0x7fe02c160f10 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe02be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c161590 0x7fe02c1580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe02be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c161590 0x7fe02c1580e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48278/0 (socket says 172.21.15.12:48278) 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe02be00700 1 -- 172.21.15.12:0/343680230 learned_addr learned my addr 172.21.15.12:0/343680230 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe02b400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe02c0cf230 0x7fe02c159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.732+0000 7fe02aa00700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe02c160f10 0x7fe02c0b81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.733+0000 7fe02be00700 1 -- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe02c160f10 msgr2=0x7fe02c0b81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.733+0000 7fe02be00700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe02c160f10 0x7fe02c0b81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.733+0000 7fe02be00700 1 -- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe02c0cf230 msgr2=0x7fe02c159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.733+0000 7fe02be00700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe02c0cf230 0x7fe02c159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.733+0000 7fe02be00700 1 -- 172.21.15.12:0/343680230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c009550 con 0x7fe02c161590 2024-03-29T11:12:16.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.733+0000 7fe02be00700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c161590 0x7fe02c1580e0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fe02400cb50 tx=0x7fe02400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.733+0000 7fe023400700 1 -- 172.21.15.12:0/343680230 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024003c00 con 0x7fe02c161590 2024-03-29T11:12:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.733+0000 7fe023400700 1 -- 172.21.15.12:0/343680230 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe024003d60 con 0x7fe02c161590 2024-03-29T11:12:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.733+0000 7fe032800700 1 -- 172.21.15.12:0/343680230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c0b79e0 con 0x7fe02c161590 2024-03-29T11:12:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.733+0000 7fe023400700 1 -- 172.21.15.12:0/343680230 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02400aa20 con 0x7fe02c161590 2024-03-29T11:12:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.734+0000 7fe032800700 1 -- 172.21.15.12:0/343680230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c152d00 con 0x7fe02c161590 2024-03-29T11:12:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.735+0000 7fe032800700 1 -- 172.21.15.12:0/343680230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c0b7f00 con 0x7fe02c161590 2024-03-29T11:12:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.736+0000 7fe023400700 1 -- 172.21.15.12:0/343680230 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe02400ab80 con 0x7fe02c161590 2024-03-29T11:12:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.736+0000 7fe023400700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe00c06adb0 0x7fe00c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.736+0000 7fe023400700 1 -- 172.21.15.12:0/343680230 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5589+0+0 (secure 0 0 0) 0x7fe0240893e0 con 0x7fe02c161590 2024-03-29T11:12:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.737+0000 7fe02b400700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe00c06adb0 0x7fe00c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:16.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.738+0000 7fe02b400700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe00c06adb0 0x7fe00c06d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe01c014040 tx=0x7fe01c008030 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:16.741+0000 7fe023400700 1 -- 172.21.15.12:0/343680230 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe02c0b7f00 con 0x7fe02c161590 2024-03-29T11:12:17.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:17.041+0000 7fe032800700 1 -- 172.21.15.12:0/343680230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7fe02c0b7f00 con 0x7fe02c161590 2024-03-29T11:12:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.047+0000 7fe023400700 1 -- 172.21.15.12:0/343680230 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2538) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe02c0b7f00 con 0x7fe02c161590 2024-03-29T11:12:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.050+0000 7fe032800700 1 -- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe00c06adb0 msgr2=0x7fe00c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.050+0000 7fe032800700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe00c06adb0 0x7fe00c06d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe01c014040 tx=0x7fe01c008030 comp rx=0 tx=0).stop 2024-03-29T11:12:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.050+0000 7fe032800700 1 -- 172.21.15.12:0/343680230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c161590 msgr2=0x7fe02c1580e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.050+0000 7fe032800700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c161590 0x7fe02c1580e0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fe02400cb50 tx=0x7fe02400ce60 comp rx=0 tx=0).stop 2024-03-29T11:12:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.051+0000 7fe032800700 1 -- 172.21.15.12:0/343680230 shutdown_connections 2024-03-29T11:12:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.051+0000 7fe032800700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fe00c06adb0 0x7fe00c06d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.051+0000 7fe032800700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c161590 0x7fe02c1580e0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.051+0000 7fe032800700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe02c0cf230 0x7fe02c159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.051+0000 7fe032800700 1 --2- 172.21.15.12:0/343680230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe02c160f10 0x7fe02c0b81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.051+0000 7fe032800700 1 -- 172.21.15.12:0/343680230 >> 172.21.15.12:0/343680230 conn(0x7fe02c08aff0 msgr2=0x7fe02c0cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:18.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.051+0000 7fe032800700 1 -- 172.21.15.12:0/343680230 shutdown_connections 2024-03-29T11:12:18.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.051+0000 7fe032800700 1 -- 172.21.15.12:0/343680230 wait complete. 2024-03-29T11:12:18.049 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2024-03-29T11:12:18.049 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2024-03-29T11:12:18.050 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T11:12:18.050 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-03-29T11:12:18.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.275+0000 7f4808000700 1 Processor -- start 2024-03-29T11:12:18.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.275+0000 7f4808000700 1 -- start start 2024-03-29T11:12:18.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.276+0000 7f4808000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f48000d18b0 0x7f48000cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:18.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.276+0000 7f4808000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4800161590 0x7f4800161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:18.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.276+0000 7f4808000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48000ceda0 con 0x7f4800160f10 2024-03-29T11:12:18.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.276+0000 7f4808000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48000d1d30 con 0x7f48000d18b0 2024-03-29T11:12:18.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.276+0000 7f4808000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4800161f60 con 0x7f4800161590 2024-03-29T11:12:18.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f4804c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f48000d18b0 0x7f48000cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f4806000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4800161590 0x7f4800161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f4805600700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f4800160f10 0x7f48000d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:48582/0 (socket says 172.21.15.12:48582) 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f4805600700 1 -- 172.21.15.12:0/2536062976 learned_addr learned my addr 172.21.15.12:0/2536062976 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f47ffe00700 1 -- 172.21.15.12:0/2536062976 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358633771 0 0) 0x7f48000ceda0 con 0x7f4800160f10 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f47ffe00700 1 -- 172.21.15.12:0/2536062976 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e80035b0 con 0x7f4800160f10 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f4804c00700 1 -- 172.21.15.12:0/2536062976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4800161590 msgr2=0x7f4800161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f4804c00700 1 --2- 172.21.15.12:0/2536062976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4800161590 0x7f4800161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f4804c00700 1 -- 172.21.15.12:0/2536062976 >> v1:172.21.15.159:6789/0 conn(0x7f4800160f10 legacy=0x7f48000d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f4804c00700 1 -- 172.21.15.12:0/2536062976 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48001612f0 con 0x7f48000d18b0 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.277+0000 7f4806000700 1 --2- 172.21.15.12:0/2536062976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4800161590 0x7f4800161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.278+0000 7f4804c00700 1 --2- 172.21.15.12:0/2536062976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48000d18b0 0x7f48000cf230 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009870 tx=0x7f47ec009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0bef5976ee298d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:18.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.278+0000 7f47ffe00700 1 -- 172.21.15.12:0/2536062976 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec011830 con 0x7f48000d18b0 2024-03-29T11:12:18.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.278+0000 7f47ffe00700 1 -- 172.21.15.12:0/2536062976 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec011990 con 0x7f48000d18b0 2024-03-29T11:12:18.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.278+0000 7f47ffe00700 1 -- 172.21.15.12:0/2536062976 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec011d20 con 0x7f48000d18b0 2024-03-29T11:12:18.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.278+0000 7f4808000700 1 -- 172.21.15.12:0/2536062976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48000d18b0 msgr2=0x7f48000cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.279+0000 7f4808000700 1 --2- 172.21.15.12:0/2536062976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48000d18b0 0x7f48000cf230 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009870 tx=0x7f47ec009b80 comp rx=0 tx=0).stop 2024-03-29T11:12:18.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.279+0000 7f4808000700 1 -- 172.21.15.12:0/2536062976 shutdown_connections 2024-03-29T11:12:18.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.279+0000 7f4808000700 1 --2- 172.21.15.12:0/2536062976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4800161590 0x7f4800161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:18.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.279+0000 7f4808000700 1 --2- 172.21.15.12:0/2536062976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48000d18b0 0x7f48000cf230 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:18.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.279+0000 7f4808000700 1 -- 172.21.15.12:0/2536062976 >> 172.21.15.12:0/2536062976 conn(0x7f480008aff0 msgr2=0x7f48000d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:18.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.279+0000 7f4808000700 1 -- 172.21.15.12:0/2536062976 shutdown_connections 2024-03-29T11:12:18.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.279+0000 7f4808000700 1 -- 172.21.15.12:0/2536062976 wait complete. 2024-03-29T11:12:18.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.280+0000 7f4808000700 1 Processor -- start 2024-03-29T11:12:18.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.281+0000 7f4808000700 1 -- start start 2024-03-29T11:12:18.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.281+0000 7f4808000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48000d18b0 0x7f4800156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:18.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.281+0000 7f4808000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4800160f10 0x7f48001f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:18.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.281+0000 7f4808000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4800161590 0x7f48001f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:18.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.281+0000 7f4808000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000b7050 con 0x7f4800161590 2024-03-29T11:12:18.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4808000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000b8090 con 0x7f4800160f10 2024-03-29T11:12:18.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4808000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48000b72d0 con 0x7f48000d18b0 2024-03-29T11:12:18.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4806000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4800161590 0x7f48001f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4806000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4800161590 0x7f48001f4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48292/0 (socket says 172.21.15.12:48292) 2024-03-29T11:12:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4806000700 1 -- 172.21.15.12:0/3240338740 learned_addr learned my addr 172.21.15.12:0/3240338740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4804c00700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4800160f10 0x7f48001f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4805600700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48000d18b0 0x7f4800156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4806000700 1 -- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48000d18b0 msgr2=0x7f4800156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4806000700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48000d18b0 0x7f4800156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4806000700 1 -- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4800160f10 msgr2=0x7f48001f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4806000700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4800160f10 0x7f48001f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4806000700 1 -- 172.21.15.12:0/3240338740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f8009a70 con 0x7f4800161590 2024-03-29T11:12:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.282+0000 7f4804c00700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4800160f10 0x7f48001f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.283+0000 7f4806000700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4800161590 0x7f48001f4a10 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f47f800efc0 tx=0x7f47f800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.283+0000 7f47fd600700 1 -- 172.21.15.12:0/3240338740 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f800ca90 con 0x7f4800161590 2024-03-29T11:12:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.283+0000 7f47fd600700 1 -- 172.21.15.12:0/3240338740 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f800f040 con 0x7f4800161590 2024-03-29T11:12:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.283+0000 7f47fd600700 1 -- 172.21.15.12:0/3240338740 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f800ce30 con 0x7f4800161590 2024-03-29T11:12:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.283+0000 7f4808000700 1 -- 172.21.15.12:0/3240338740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec0094a0 con 0x7f4800161590 2024-03-29T11:12:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.283+0000 7f4808000700 1 -- 172.21.15.12:0/3240338740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48000b7890 con 0x7f4800161590 2024-03-29T11:12:18.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.284+0000 7f4808000700 1 -- 172.21.15.12:0/3240338740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4800067720 con 0x7f4800161590 2024-03-29T11:12:18.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.285+0000 7f47fd600700 1 -- 172.21.15.12:0/3240338740 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f47f8014480 con 0x7f4800161590 2024-03-29T11:12:18.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.286+0000 7f47fd600700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f47e006ade0 0x7f47e006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:18.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.286+0000 7f47fd600700 1 -- 172.21.15.12:0/3240338740 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f47f8089ad0 con 0x7f4800161590 2024-03-29T11:12:18.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.286+0000 7f4805600700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f47e006ade0 0x7f47e006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.287+0000 7f4805600700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f47e006ade0 0x7f47e006d2a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f47f4007fd0 tx=0x7f47f4007dc0 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:18.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.290+0000 7f47fd600700 1 -- 172.21.15.12:0/3240338740 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f47f8054b90 con 0x7f4800161590 2024-03-29T11:12:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:18.590+0000 7f4808000700 1 -- 172.21.15.12:0/3240338740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f48000b4860 con 0x7f4800161590 2024-03-29T11:12:19.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.048+0000 7f47fd600700 1 -- 172.21.15.12:0/3240338740 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2539) v1 ==== 168+0+0 (secure 0 0 0) 0x7f47f8009cd0 con 0x7f4800161590 2024-03-29T11:12:19.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 -- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f47e006ade0 msgr2=0x7f47e006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f47e006ade0 0x7f47e006d2a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f47f4007fd0 tx=0x7f47f4007dc0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 -- 172.21.15.12:0/3240338740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4800161590 msgr2=0x7f48001f4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4800161590 0x7f48001f4a10 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f47f800efc0 tx=0x7f47f800c3a0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 -- 172.21.15.12:0/3240338740 shutdown_connections 2024-03-29T11:12:19.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f47e006ade0 0x7f47e006d2a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4800161590 0x7f48001f4a10 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48000d18b0 0x7f4800156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 --2- 172.21.15.12:0/3240338740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4800160f10 0x7f48001f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 -- 172.21.15.12:0/3240338740 >> 172.21.15.12:0/3240338740 conn(0x7f480008aff0 msgr2=0x7f480015c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:19.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 -- 172.21.15.12:0/3240338740 shutdown_connections 2024-03-29T11:12:19.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.051+0000 7f4808000700 1 -- 172.21.15.12:0/3240338740 wait complete. 2024-03-29T11:12:19.048 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T11:12:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.274+0000 7f58e4600700 1 Processor -- start 2024-03-29T11:12:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.274+0000 7f58e4600700 1 -- start start 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.275+0000 7f58e4600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58dc15d9d0 0x7f58dc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.275+0000 7f58e4600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f58dc0d16a0 0x7f58dc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.275+0000 7f58e4600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc161420 con 0x7f58dc0d2370 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.275+0000 7f58e4600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58dc0d27e0 con 0x7f58dc15d9d0 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.275+0000 7f58e4600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58dc161700 con 0x7f58dc0d16a0 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.275+0000 7f58e1c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58dc15d9d0 0x7f58dc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.275+0000 7f58e1c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58dc15d9d0 0x7f58dc0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37294/0 (socket says 172.21.15.12:37294) 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.275+0000 7f58e1c00700 1 -- 172.21.15.12:0/3292028133 learned_addr learned my addr 172.21.15.12:0/3292028133 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.276+0000 7f58e1200700 1 --2- 172.21.15.12:0/3292028133 >> v2:172.21.15.159:3300/0 conn(0x7f58dc0d16a0 0x7f58dc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.276+0000 7f58e1c00700 1 -- 172.21.15.12:0/3292028133 >> v2:172.21.15.159:3300/0 conn(0x7f58dc0d16a0 msgr2=0x7f58dc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.276+0000 7f58e1c00700 1 --2- 172.21.15.12:0/3292028133 >> v2:172.21.15.159:3300/0 conn(0x7f58dc0d16a0 0x7f58dc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.276+0000 7f58e1c00700 1 -- 172.21.15.12:0/3292028133 >> v1:172.21.15.159:6789/0 conn(0x7f58dc0d2370 legacy=0x7f58dc15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.276+0000 7f58e1c00700 1 -- 172.21.15.12:0/3292028133 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc1619e0 con 0x7f58dc15d9d0 2024-03-29T11:12:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.276+0000 7f58e1c00700 1 --2- 172.21.15.12:0/3292028133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 0x7f58dc0d3c30 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0d12f0 tx=0x7f58d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f2d37e046e5b74 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:19.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.276+0000 7f58dbe00700 1 -- 172.21.15.12:0/3292028133 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d000b870 con 0x7f58dc15d9d0 2024-03-29T11:12:19.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.276+0000 7f58dbe00700 1 -- 172.21.15.12:0/3292028133 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d000b9d0 con 0x7f58dc15d9d0 2024-03-29T11:12:19.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.276+0000 7f58dbe00700 1 -- 172.21.15.12:0/3292028133 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d000bd60 con 0x7f58dc15d9d0 2024-03-29T11:12:19.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.277+0000 7f58e4600700 1 -- 172.21.15.12:0/3292028133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 msgr2=0x7f58dc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.277+0000 7f58e4600700 1 --2- 172.21.15.12:0/3292028133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 0x7f58dc0d3c30 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0d12f0 tx=0x7f58d0009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.277+0000 7f58e4600700 1 -- 172.21.15.12:0/3292028133 shutdown_connections 2024-03-29T11:12:19.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.277+0000 7f58e4600700 1 --2- 172.21.15.12:0/3292028133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 0x7f58dc0d3c30 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.277+0000 7f58e4600700 1 --2- 172.21.15.12:0/3292028133 >> v2:172.21.15.159:3300/0 conn(0x7f58dc0d16a0 0x7f58dc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.277+0000 7f58e4600700 1 -- 172.21.15.12:0/3292028133 >> 172.21.15.12:0/3292028133 conn(0x7f58dc08aff0 msgr2=0x7f58dc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:19.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.277+0000 7f58e4600700 1 -- 172.21.15.12:0/3292028133 shutdown_connections 2024-03-29T11:12:19.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.277+0000 7f58e4600700 1 -- 172.21.15.12:0/3292028133 wait complete. 2024-03-29T11:12:19.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.280+0000 7f58e4600700 1 Processor -- start 2024-03-29T11:12:19.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.280+0000 7f58e4600700 1 -- start start 2024-03-29T11:12:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e4600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58dc0d16a0 0x7f58dc154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e4600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58dc0d2370 0x7f58dc15e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e4600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 0x7f58dc15b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e4600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc155460 con 0x7f58dc15d9d0 2024-03-29T11:12:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e4600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc15eb70 con 0x7f58dc0d16a0 2024-03-29T11:12:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e4600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc15be90 con 0x7f58dc0d2370 2024-03-29T11:12:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e2600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 0x7f58dc15b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e1c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58dc0d16a0 0x7f58dc154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e1c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58dc0d16a0 0x7f58dc154f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:56586/0 (socket says 172.21.15.12:56586) 2024-03-29T11:12:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e1c00700 1 -- 172.21.15.12:0/2240501548 learned_addr learned my addr 172.21.15.12:0/2240501548 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e2600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 0x7f58dc15b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37300/0 (socket says 172.21.15.12:37300) 2024-03-29T11:12:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.281+0000 7f58e1200700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58dc0d2370 0x7f58dc15e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58e2600700 1 -- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58dc0d2370 msgr2=0x7f58dc15e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58e2600700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58dc0d2370 0x7f58dc15e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58e2600700 1 -- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58dc0d16a0 msgr2=0x7f58dc154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58e2600700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58dc0d16a0 0x7f58dc154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58e2600700 1 -- 172.21.15.12:0/2240501548 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d0009550 con 0x7f58dc15d9d0 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58e2600700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 0x7f58dc15b3a0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f58d400cb90 tx=0x7f58d400cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58d9600700 1 -- 172.21.15.12:0/2240501548 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d4003c00 con 0x7f58dc15d9d0 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58d9600700 1 -- 172.21.15.12:0/2240501548 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d4003d60 con 0x7f58dc15d9d0 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58e4600700 1 -- 172.21.15.12:0/2240501548 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58dc1563c0 con 0x7f58dc15d9d0 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58d9600700 1 -- 172.21.15.12:0/2240501548 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d400aaa0 con 0x7f58dc15d9d0 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.282+0000 7f58e4600700 1 -- 172.21.15.12:0/2240501548 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc156980 con 0x7f58dc15d9d0 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.283+0000 7f58e4600700 1 -- 172.21.15.12:0/2240501548 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc0d1ca0 con 0x7f58dc15d9d0 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.284+0000 7f58d9600700 1 -- 172.21.15.12:0/2240501548 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f58d400ac00 con 0x7f58dc15d9d0 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.285+0000 7f58d9600700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58bc06adb0 0x7f58bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.285+0000 7f58d9600700 1 -- 172.21.15.12:0/2240501548 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f58d4089440 con 0x7f58dc15d9d0 2024-03-29T11:12:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.285+0000 7f58e1c00700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58bc06adb0 0x7f58bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.286+0000 7f58e1c00700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58bc06adb0 0x7f58bc06d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009c10 tx=0x7f58d0008030 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.290+0000 7f58d9600700 1 -- 172.21.15.12:0/2240501548 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f58d40544f0 con 0x7f58dc15d9d0 2024-03-29T11:12:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.504+0000 7f58e4600700 1 -- 172.21.15.12:0/2240501548 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f58dc156220 con 0x7f58dc15d9d0 2024-03-29T11:12:19.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.508+0000 7f58d9600700 1 -- 172.21.15.12:0/2240501548 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f58dc156220 con 0x7f58dc15d9d0 2024-03-29T11:12:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 -- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58bc06adb0 msgr2=0x7f58bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58bc06adb0 0x7f58bc06d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009c10 tx=0x7f58d0008030 comp rx=0 tx=0).stop 2024-03-29T11:12:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 -- 172.21.15.12:0/2240501548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 msgr2=0x7f58dc15b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 0x7f58dc15b3a0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f58d400cb90 tx=0x7f58d400cea0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 -- 172.21.15.12:0/2240501548 shutdown_connections 2024-03-29T11:12:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f58bc06adb0 0x7f58bc06d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58dc15d9d0 0x7f58dc15b3a0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58dc0d16a0 0x7f58dc154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 --2- 172.21.15.12:0/2240501548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58dc0d2370 0x7f58dc15e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 -- 172.21.15.12:0/2240501548 >> 172.21.15.12:0/2240501548 conn(0x7f58dc08aff0 msgr2=0x7f58dc06bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 -- 172.21.15.12:0/2240501548 shutdown_connections 2024-03-29T11:12:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:19.511+0000 7f58e4600700 1 -- 172.21.15.12:0/2240501548 wait complete. 2024-03-29T11:12:19.507 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:12:19.507 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:12:19.591 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:12:19.689 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:12:19.689 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:12:19.689 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:12:19.733 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:12:19.733 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:12:19.734 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:12:19.780 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:12:19.781 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:12:19.781 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:12:19.825 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:12:19.825 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:12:19.825 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:12:19.871 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:12:19.871 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T11:12:19.871 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:12:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.092+0000 7f95be800700 1 Processor -- start 2024-03-29T11:12:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.092+0000 7f95be800700 1 -- start start 2024-03-29T11:12:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.093+0000 7f95be800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f95b815d640 0x7f95b80d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.093+0000 7f95be800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95b81619d0 0x7f95b80b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.093+0000 7f95be800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b80d30d0 con 0x7f95b80ceec0 2024-03-29T11:12:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.093+0000 7f95be800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b80b6840 con 0x7f95b81619d0 2024-03-29T11:12:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.093+0000 7f95be800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b80d2880 con 0x7f95b815d640 2024-03-29T11:12:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.093+0000 7f95b7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95b81619d0 0x7f95b80b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.093+0000 7f95b7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95b81619d0 0x7f95b80b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37302/0 (socket says 172.21.15.12:37302) 2024-03-29T11:12:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.093+0000 7f95b6a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f95b80ceec0 0x7f95b81612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59610/0 (socket says 172.21.15.12:59610) 2024-03-29T11:12:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.093+0000 7f95b7e00700 1 -- 172.21.15.12:0/2496504154 learned_addr learned my addr 172.21.15.12:0/2496504154 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.093+0000 7f95b7400700 1 --2- 172.21.15.12:0/2496504154 >> v2:172.21.15.159:3300/0 conn(0x7f95b815d640 0x7f95b80d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.094+0000 7f95b7e00700 1 -- 172.21.15.12:0/2496504154 >> v2:172.21.15.159:3300/0 conn(0x7f95b815d640 msgr2=0x7f95b80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.094+0000 7f95b7e00700 1 --2- 172.21.15.12:0/2496504154 >> v2:172.21.15.159:3300/0 conn(0x7f95b815d640 0x7f95b80d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.094+0000 7f95b7e00700 1 -- 172.21.15.12:0/2496504154 >> v1:172.21.15.12:6789/0 conn(0x7f95b80ceec0 legacy=0x7f95b81612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.094+0000 7f95b7e00700 1 -- 172.21.15.12:0/2496504154 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b80d2b30 con 0x7f95b81619d0 2024-03-29T11:12:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.094+0000 7f95b7400700 1 --2- 172.21.15.12:0/2496504154 >> v2:172.21.15.159:3300/0 conn(0x7f95b815d640 0x7f95b80d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.094+0000 7f95b7e00700 1 --2- 172.21.15.12:0/2496504154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b81619d0 0x7f95b80b6cd0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f95b80b5c20 tx=0x7f95b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a946b5ac1ff8b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.094+0000 7f95b6000700 1 -- 172.21.15.12:0/2496504154 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b000e070 con 0x7f95b81619d0 2024-03-29T11:12:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.094+0000 7f95b6000700 1 -- 172.21.15.12:0/2496504154 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b000b9c0 con 0x7f95b81619d0 2024-03-29T11:12:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.094+0000 7f95b6000700 1 -- 172.21.15.12:0/2496504154 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b000bd70 con 0x7f95b81619d0 2024-03-29T11:12:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.095+0000 7f95be800700 1 -- 172.21.15.12:0/2496504154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b81619d0 msgr2=0x7f95b80b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.095+0000 7f95be800700 1 --2- 172.21.15.12:0/2496504154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b81619d0 0x7f95b80b6cd0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f95b80b5c20 tx=0x7f95b000b330 comp rx=0 tx=0).stop 2024-03-29T11:12:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.095+0000 7f95be800700 1 -- 172.21.15.12:0/2496504154 shutdown_connections 2024-03-29T11:12:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.095+0000 7f95be800700 1 --2- 172.21.15.12:0/2496504154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b81619d0 0x7f95b80b6cd0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.095+0000 7f95be800700 1 --2- 172.21.15.12:0/2496504154 >> v2:172.21.15.159:3300/0 conn(0x7f95b815d640 0x7f95b80d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.095+0000 7f95be800700 1 -- 172.21.15.12:0/2496504154 >> 172.21.15.12:0/2496504154 conn(0x7f95b808aff0 msgr2=0x7f95b80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.095+0000 7f95be800700 1 -- 172.21.15.12:0/2496504154 shutdown_connections 2024-03-29T11:12:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.095+0000 7f95be800700 1 -- 172.21.15.12:0/2496504154 wait complete. 2024-03-29T11:12:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.096+0000 7f95be800700 1 Processor -- start 2024-03-29T11:12:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.097+0000 7f95be800700 1 -- start start 2024-03-29T11:12:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.097+0000 7f95be800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95b80ceec0 0x7f95b80c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95be800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b815d640 0x7f95b80c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95be800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95b81619d0 0x7f95b814f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95be800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b80c75c0 con 0x7f95b815d640 2024-03-29T11:12:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95be800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b80a0f90 con 0x7f95b80ceec0 2024-03-29T11:12:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95be800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b81f67c0 con 0x7f95b81619d0 2024-03-29T11:12:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b815d640 0x7f95b80c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b815d640 0x7f95b80c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37310/0 (socket says 172.21.15.12:37310) 2024-03-29T11:12:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95b6a00700 1 -- 172.21.15.12:0/286040977 learned_addr learned my addr 172.21.15.12:0/286040977 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95b7e00700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95b81619d0 0x7f95b814f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95b7400700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95b80ceec0 0x7f95b80c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95b6a00700 1 -- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95b81619d0 msgr2=0x7f95b814f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95b6a00700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95b81619d0 0x7f95b814f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95b6a00700 1 -- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95b80ceec0 msgr2=0x7f95b80c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95b6a00700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95b80ceec0 0x7f95b80c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.098+0000 7f95b6a00700 1 -- 172.21.15.12:0/286040977 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b0009b30 con 0x7f95b815d640 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.099+0000 7f95b6a00700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b815d640 0x7f95b80c7080 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f95a400c500 tx=0x7f95a400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.099+0000 7f95af400700 1 -- 172.21.15.12:0/286040977 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a4005cb0 con 0x7f95b815d640 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.099+0000 7f95af400700 1 -- 172.21.15.12:0/286040977 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a4005e10 con 0x7f95b815d640 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.099+0000 7f95af400700 1 -- 172.21.15.12:0/286040977 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a40072d0 con 0x7f95b815d640 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.099+0000 7f95be800700 1 -- 172.21.15.12:0/286040977 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b81f6aa0 con 0x7f95b815d640 2024-03-29T11:12:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.099+0000 7f95be800700 1 -- 172.21.15.12:0/286040977 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b81f2d20 con 0x7f95b815d640 2024-03-29T11:12:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.100+0000 7f95be800700 1 -- 172.21.15.12:0/286040977 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b81f2f10 con 0x7f95b815d640 2024-03-29T11:12:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.101+0000 7f95af400700 1 -- 172.21.15.12:0/286040977 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f95a4007430 con 0x7f95b815d640 2024-03-29T11:12:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.102+0000 7f95af400700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f959806adb0 0x7f959806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.103+0000 7f95af400700 1 -- 172.21.15.12:0/286040977 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5589+0+0 (secure 0 0 0) 0x7f95a40886f0 con 0x7f95b815d640 2024-03-29T11:12:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.103+0000 7f95b7400700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f959806adb0 0x7f959806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.104+0000 7f95b7400700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f959806adb0 0x7f959806d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f95a8006f10 tx=0x7f95a8008040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.107+0000 7f95af400700 1 -- 172.21.15.12:0/286040977 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f95a4053840 con 0x7f95b815d640 2024-03-29T11:12:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.365+0000 7f95be800700 1 -- 172.21.15.12:0/286040977 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95b8067720 con 0x7f95b815d640 2024-03-29T11:12:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.365+0000 7f95af400700 1 -- 172.21.15.12:0/286040977 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f95a4017090 con 0x7f95b815d640 2024-03-29T11:12:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:20.344 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":83,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/652112625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":652112625},{"type":"v1","addr":"172.21.15.159:6836","nonce":652112625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9166,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2803735172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2803735172},{"type":"v1","addr":"172.21.15.159:6837","nonce":2803735172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9444,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/3698805288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3698805288},{"type":"v1","addr":"172.21.15.12:6833","nonce":3698805288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9876,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3064210860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3064210860},{"type":"v1","addr":"172.21.15.12:6835","nonce":3064210860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[]} 2024-03-29T11:12:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 -- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f959806adb0 msgr2=0x7f959806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f959806adb0 0x7f959806d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f95a8006f10 tx=0x7f95a8008040 comp rx=0 tx=0).stop 2024-03-29T11:12:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 -- 172.21.15.12:0/286040977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b815d640 msgr2=0x7f95b80c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b815d640 0x7f95b80c7080 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f95a400c500 tx=0x7f95a400c810 comp rx=0 tx=0).stop 2024-03-29T11:12:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 -- 172.21.15.12:0/286040977 shutdown_connections 2024-03-29T11:12:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f959806adb0 0x7f959806d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95b81619d0 0x7f95b814f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f95b80ceec0 0x7f95b80c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 --2- 172.21.15.12:0/286040977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b815d640 0x7f95b80c7080 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 -- 172.21.15.12:0/286040977 >> 172.21.15.12:0/286040977 conn(0x7f95b808aff0 msgr2=0x7f95b815cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 -- 172.21.15.12:0/286040977 shutdown_connections 2024-03-29T11:12:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.368+0000 7f95be800700 1 -- 172.21.15.12:0/286040977 wait complete. 2024-03-29T11:12:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 83 2024-03-29T11:12:20.365 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T11:12:20.365 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T11:12:20.365 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T11:12:20.365 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:12:20.365 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T11:12:20.365 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T11:12:20.365 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T11:12:20.366 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:12:20.366 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T11:12:20.366 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T11:12:20.366 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T11:12:20.366 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:12:20.366 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T11:12:20.366 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T11:12:20.367 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T11:12:20.367 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:12:20.382 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T11:12:20.383 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T11:12:20.383 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T11:12:20.384 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T11:12:20.385 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T11:12:20.385 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T11:12:20.386 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T11:12:20.386 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T11:12:20.427 INFO:tasks.ceph.mds.b:Started 2024-03-29T11:12:20.427 DEBUG:teuthology.parallel:result is None 2024-03-29T11:12:20.429 INFO:tasks.ceph.mds.d:Started 2024-03-29T11:12:20.429 DEBUG:teuthology.parallel:result is None 2024-03-29T11:12:20.430 INFO:tasks.ceph.mds.c:Started 2024-03-29T11:12:20.430 DEBUG:teuthology.parallel:result is None 2024-03-29T11:12:20.432 INFO:tasks.ceph.mds.a:Started 2024-03-29T11:12:20.432 DEBUG:teuthology.parallel:result is None 2024-03-29T11:12:20.432 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T11:12:20.575 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T11:12:20.576 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T11:12:20.580 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T11:12:20.581 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:12:20.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.666+0000 7fc41ec00700 1 Processor -- start 2024-03-29T11:12:20.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.666+0000 7fc41ec00700 1 -- start start 2024-03-29T11:12:20.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.667+0000 7fc41ec00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 0x7fc4180d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.667+0000 7fc41ec00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc4180d16a0 0x7fc4180d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.667+0000 7fc41ec00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc4180d2370 0x7fc41815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.667+0000 7fc41ec00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4180d2820 con 0x7fc4180d16a0 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.667+0000 7fc41ec00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc418161420 con 0x7fc4180d2370 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.667+0000 7fc41ec00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc418161700 con 0x7fc41815d9d0 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.668+0000 7fc417400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc4180d16a0 0x7fc4180d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.668+0000 7fc417400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc4180d16a0 0x7fc4180d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37344/0 (socket says 172.21.15.12:37344) 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.668+0000 7fc417400700 1 -- 172.21.15.12:0/621440593 learned_addr learned my addr 172.21.15.12:0/621440593 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.668+0000 7fc417e00700 1 --2- 172.21.15.12:0/621440593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 0x7fc4180d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.668+0000 7fc41cc00700 1 --2- 172.21.15.12:0/621440593 >> v2:172.21.15.159:3300/0 conn(0x7fc4180d2370 0x7fc41815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.668+0000 7fc417400700 1 -- 172.21.15.12:0/621440593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 msgr2=0x7fc4180d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.668+0000 7fc417400700 1 --2- 172.21.15.12:0/621440593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 0x7fc4180d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.668+0000 7fc417400700 1 -- 172.21.15.12:0/621440593 >> v2:172.21.15.159:3300/0 conn(0x7fc4180d2370 msgr2=0x7fc41815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.668+0000 7fc417400700 1 --2- 172.21.15.12:0/621440593 >> v2:172.21.15.159:3300/0 conn(0x7fc4180d2370 0x7fc41815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.668+0000 7fc417400700 1 -- 172.21.15.12:0/621440593 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4181619e0 con 0x7fc4180d16a0 2024-03-29T11:12:20.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc417400700 1 --2- 172.21.15.12:0/621440593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 0x7fc4180d1e30 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fc4180cdc00 tx=0x7fc404009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04d740b98c93673 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:20.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc416a00700 1 -- 172.21.15.12:0/621440593 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404005e30 con 0x7fc4180d16a0 2024-03-29T11:12:20.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc416a00700 1 -- 172.21.15.12:0/621440593 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc404004030 con 0x7fc4180d16a0 2024-03-29T11:12:20.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc416a00700 1 -- 172.21.15.12:0/621440593 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4040043e0 con 0x7fc4180d16a0 2024-03-29T11:12:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc41ec00700 1 -- 172.21.15.12:0/621440593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 msgr2=0x7fc4180d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc41ec00700 1 --2- 172.21.15.12:0/621440593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 0x7fc4180d1e30 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fc4180cdc00 tx=0x7fc404009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc41ec00700 1 -- 172.21.15.12:0/621440593 shutdown_connections 2024-03-29T11:12:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc41ec00700 1 --2- 172.21.15.12:0/621440593 >> v2:172.21.15.159:3300/0 conn(0x7fc4180d2370 0x7fc41815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc41ec00700 1 --2- 172.21.15.12:0/621440593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 0x7fc4180d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc41ec00700 1 --2- 172.21.15.12:0/621440593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 0x7fc4180d1e30 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.669+0000 7fc41ec00700 1 -- 172.21.15.12:0/621440593 >> 172.21.15.12:0/621440593 conn(0x7fc41808aff0 msgr2=0x7fc4180d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.670+0000 7fc41ec00700 1 -- 172.21.15.12:0/621440593 shutdown_connections 2024-03-29T11:12:20.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.670+0000 7fc41ec00700 1 -- 172.21.15.12:0/621440593 wait complete. 2024-03-29T11:12:20.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.671+0000 7fc41ec00700 1 Processor -- start 2024-03-29T11:12:20.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.671+0000 7fc41ec00700 1 -- start start 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.672+0000 7fc41ec00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 0x7fc41815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.672+0000 7fc41ec00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4180d2370 0x7fc4180b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.672+0000 7fc41ec00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 0x7fc4180b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.672+0000 7fc41ec00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4180cb9f0 con 0x7fc4180d16a0 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.672+0000 7fc41ec00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4180b8e80 con 0x7fc4180d2370 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.672+0000 7fc41ec00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4180b85e0 con 0x7fc41815d9d0 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 0x7fc41815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 0x7fc41815a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37350/0 (socket says 172.21.15.12:37350) 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417e00700 1 -- 172.21.15.12:0/4172097493 learned_addr learned my addr 172.21.15.12:0/4172097493 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417400700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4180d2370 0x7fc4180b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc41cc00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 0x7fc4180b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417e00700 1 -- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 msgr2=0x7fc4180b7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417e00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 0x7fc4180b7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417e00700 1 -- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4180d2370 msgr2=0x7fc4180b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417e00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4180d2370 0x7fc4180b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417e00700 1 -- 172.21.15.12:0/4172097493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc404009500 con 0x7fc4180d16a0 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417400700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4180d2370 0x7fc4180b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc41cc00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 0x7fc4180b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.673+0000 7fc417e00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 0x7fc41815a0e0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fc40800e8a0 tx=0x7fc40800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.674+0000 7fc40fe00700 1 -- 172.21.15.12:0/4172097493 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408009590 con 0x7fc4180d16a0 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.674+0000 7fc40fe00700 1 -- 172.21.15.12:0/4172097493 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40800eea0 con 0x7fc4180d16a0 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.674+0000 7fc40fe00700 1 -- 172.21.15.12:0/4172097493 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40800ca50 con 0x7fc4180d16a0 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.674+0000 7fc41ec00700 1 -- 172.21.15.12:0/4172097493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4180b88c0 con 0x7fc4180d16a0 2024-03-29T11:12:20.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.674+0000 7fc41ec00700 1 -- 172.21.15.12:0/4172097493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4180cbc00 con 0x7fc4180d16a0 2024-03-29T11:12:20.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.675+0000 7fc41ec00700 1 -- 172.21.15.12:0/4172097493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4180d1ca0 con 0x7fc4180d16a0 2024-03-29T11:12:20.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.676+0000 7fc40fe00700 1 -- 172.21.15.12:0/4172097493 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc40800cbb0 con 0x7fc4180d16a0 2024-03-29T11:12:20.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.676+0000 7fc40fe00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc3f806adb0 0x7fc3f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:20.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.677+0000 7fc40fe00700 1 -- 172.21.15.12:0/4172097493 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5589+0+0 (secure 0 0 0) 0x7fc4080893d0 con 0x7fc4180d16a0 2024-03-29T11:12:20.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.677+0000 7fc417400700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc3f806adb0 0x7fc3f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:20.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.678+0000 7fc417400700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc3f806adb0 0x7fc3f806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc4180d1ae0 tx=0x7fc404008070 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:20.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.681+0000 7fc40fe00700 1 -- 172.21.15.12:0/4172097493 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc408054510 con 0x7fc4180d16a0 2024-03-29T11:12:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:20.914+0000 7fc41ec00700 1 -- 172.21.15.12:0/4172097493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc418067720 con 0x7fc4180d16a0 2024-03-29T11:12:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.636+0000 7fc40fe00700 1 -- 172.21.15.12:0/4172097493 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v187) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc408057b30 con 0x7fc4180d16a0 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.639+0000 7fc41ec00700 1 -- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc3f806adb0 msgr2=0x7fc3f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.639+0000 7fc41ec00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc3f806adb0 0x7fc3f806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc4180d1ae0 tx=0x7fc404008070 comp rx=0 tx=0).stop 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.639+0000 7fc41ec00700 1 -- 172.21.15.12:0/4172097493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 msgr2=0x7fc41815a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.639+0000 7fc41ec00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 0x7fc41815a0e0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fc40800e8a0 tx=0x7fc40800ec60 comp rx=0 tx=0).stop 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.639+0000 7fc41ec00700 1 -- 172.21.15.12:0/4172097493 shutdown_connections 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.639+0000 7fc41ec00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc3f806adb0 0x7fc3f806d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.639+0000 7fc41ec00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc41815d9d0 0x7fc4180b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.639+0000 7fc41ec00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4180d16a0 0x7fc41815a0e0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.639+0000 7fc41ec00700 1 --2- 172.21.15.12:0/4172097493 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc4180d2370 0x7fc4180b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.639+0000 7fc41ec00700 1 -- 172.21.15.12:0/4172097493 >> 172.21.15.12:0/4172097493 conn(0x7fc41808aff0 msgr2=0x7fc4180ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.640+0000 7fc41ec00700 1 -- 172.21.15.12:0/4172097493 shutdown_connections 2024-03-29T11:12:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.640+0000 7fc41ec00700 1 -- 172.21.15.12:0/4172097493 wait complete. 2024-03-29T11:12:21.619 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T11:12:21.639 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T11:12:21.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.865+0000 7ff842a00700 1 Processor -- start 2024-03-29T11:12:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.866+0000 7ff842a00700 1 -- start start 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff842a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff83c15b5d0 0x7ff83c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff842a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff83c0cb3f0 0x7ff83c0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff842a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0ae930 0x7ff83c0ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff842a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c15b9b0 con 0x7ff83c15b5d0 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff842a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c15fd50 con 0x7ff83c0cb3f0 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff842a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c15f140 con 0x7ff83c0ae930 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff83be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff83c15b5d0 0x7ff83c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff83be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff83c15b5d0 0x7ff83c0ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37366/0 (socket says 172.21.15.12:37366) 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff83be00700 1 -- 172.21.15.12:0/2457767256 learned_addr learned my addr 172.21.15.12:0/2457767256 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff840a00700 1 --2- 172.21.15.12:0/2457767256 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0ae930 0x7ff83c0ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff83b400700 1 --2- 172.21.15.12:0/2457767256 >> v2:172.21.15.159:3300/0 conn(0x7ff83c0cb3f0 0x7ff83c0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff83be00700 1 -- 172.21.15.12:0/2457767256 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0ae930 msgr2=0x7ff83c0ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff83be00700 1 --2- 172.21.15.12:0/2457767256 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0ae930 0x7ff83c0ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff83be00700 1 -- 172.21.15.12:0/2457767256 >> v2:172.21.15.159:3300/0 conn(0x7ff83c0cb3f0 msgr2=0x7ff83c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff83be00700 1 --2- 172.21.15.12:0/2457767256 >> v2:172.21.15.159:3300/0 conn(0x7ff83c0cb3f0 0x7ff83c0ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.867+0000 7ff83be00700 1 -- 172.21.15.12:0/2457767256 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c15f420 con 0x7ff83c15b5d0 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.868+0000 7ff840a00700 1 --2- 172.21.15.12:0/2457767256 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0ae930 0x7ff83c0ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.868+0000 7ff83b400700 1 --2- 172.21.15.12:0/2457767256 >> v2:172.21.15.159:3300/0 conn(0x7ff83c0cb3f0 0x7ff83c0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.868+0000 7ff83be00700 1 --2- 172.21.15.12:0/2457767256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 0x7ff83c0ae3f0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7ff83c06c6c0 tx=0x7ff82c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f36df3a7a19d4db server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.868+0000 7ff83aa00700 1 -- 172.21.15.12:0/2457767256 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00b870 con 0x7ff83c15b5d0 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.868+0000 7ff83aa00700 1 -- 172.21.15.12:0/2457767256 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c00b9d0 con 0x7ff83c15b5d0 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.868+0000 7ff83aa00700 1 -- 172.21.15.12:0/2457767256 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00bd60 con 0x7ff83c15b5d0 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.869+0000 7ff842a00700 1 -- 172.21.15.12:0/2457767256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 msgr2=0x7ff83c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.869+0000 7ff842a00700 1 --2- 172.21.15.12:0/2457767256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 0x7ff83c0ae3f0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7ff83c06c6c0 tx=0x7ff82c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.869+0000 7ff842a00700 1 -- 172.21.15.12:0/2457767256 shutdown_connections 2024-03-29T11:12:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.869+0000 7ff842a00700 1 --2- 172.21.15.12:0/2457767256 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0ae930 0x7ff83c0ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.869+0000 7ff842a00700 1 --2- 172.21.15.12:0/2457767256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 0x7ff83c0ae3f0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.869+0000 7ff842a00700 1 --2- 172.21.15.12:0/2457767256 >> v2:172.21.15.159:3300/0 conn(0x7ff83c0cb3f0 0x7ff83c0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.869+0000 7ff842a00700 1 -- 172.21.15.12:0/2457767256 >> 172.21.15.12:0/2457767256 conn(0x7ff83c0cf190 msgr2=0x7ff83c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.869+0000 7ff842a00700 1 -- 172.21.15.12:0/2457767256 shutdown_connections 2024-03-29T11:12:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.869+0000 7ff842a00700 1 -- 172.21.15.12:0/2457767256 wait complete. 2024-03-29T11:12:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.870+0000 7ff842a00700 1 Processor -- start 2024-03-29T11:12:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.871+0000 7ff842a00700 1 -- start start 2024-03-29T11:12:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.871+0000 7ff842a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff83c0ae930 0x7ff83c14dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.871+0000 7ff842a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 0x7ff83c0721a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff842a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0726e0 0x7ff83c0c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:21.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff842a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0ccb50 con 0x7ff83c15b5d0 2024-03-29T11:12:21.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff842a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c1f3160 con 0x7ff83c0ae930 2024-03-29T11:12:21.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff842a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0cbed0 con 0x7ff83c0726e0 2024-03-29T11:12:21.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff83b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 0x7ff83c0721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:21.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff83b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 0x7ff83c0721a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37374/0 (socket says 172.21.15.12:37374) 2024-03-29T11:12:21.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff83b400700 1 -- 172.21.15.12:0/82944774 learned_addr learned my addr 172.21.15.12:0/82944774 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:21.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff840a00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0726e0 0x7ff83c0c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:21.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff83be00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff83c0ae930 0x7ff83c14dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:21.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff83b400700 1 -- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0726e0 msgr2=0x7ff83c0c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:21.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff83b400700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0726e0 0x7ff83c0c94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff83b400700 1 -- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff83c0ae930 msgr2=0x7ff83c14dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:21.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff83b400700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff83c0ae930 0x7ff83c14dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:21.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.872+0000 7ff83b400700 1 -- 172.21.15.12:0/82944774 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c009550 con 0x7ff83c15b5d0 2024-03-29T11:12:21.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.873+0000 7ff83b400700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 0x7ff83c0721a0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7ff82800c750 tx=0x7ff82800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:21.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.873+0000 7ff833e00700 1 -- 172.21.15.12:0/82944774 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828011800 con 0x7ff83c15b5d0 2024-03-29T11:12:21.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.873+0000 7ff833e00700 1 -- 172.21.15.12:0/82944774 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff828011960 con 0x7ff83c15b5d0 2024-03-29T11:12:21.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.873+0000 7ff833e00700 1 -- 172.21.15.12:0/82944774 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828011cd0 con 0x7ff83c15b5d0 2024-03-29T11:12:21.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.873+0000 7ff842a00700 1 -- 172.21.15.12:0/82944774 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c0cc1b0 con 0x7ff83c15b5d0 2024-03-29T11:12:21.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.873+0000 7ff842a00700 1 -- 172.21.15.12:0/82944774 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c0cc770 con 0x7ff83c15b5d0 2024-03-29T11:12:21.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.874+0000 7ff842a00700 1 -- 172.21.15.12:0/82944774 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83c0ccc90 con 0x7ff83c15b5d0 2024-03-29T11:12:21.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.875+0000 7ff833e00700 1 -- 172.21.15.12:0/82944774 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff828021350 con 0x7ff83c15b5d0 2024-03-29T11:12:21.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.876+0000 7ff833e00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81c06adb0 0x7ff81c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:21.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.876+0000 7ff833e00700 1 -- 172.21.15.12:0/82944774 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 4901+0+0 (secure 0 0 0) 0x7ff828089220 con 0x7ff83c15b5d0 2024-03-29T11:12:21.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.876+0000 7ff83be00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81c06adb0 0x7ff81c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:21.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.877+0000 7ff83be00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81c06adb0 0x7ff81c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff82c014040 tx=0x7ff82c008030 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:21.880+0000 7ff833e00700 1 -- 172.21.15.12:0/82944774 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff83c0ccc90 con 0x7ff83c15b5d0 2024-03-29T11:12:22.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.180+0000 7ff842a00700 1 -- 172.21.15.12:0/82944774 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff83c0ccc90 con 0x7ff83c15b5d0 2024-03-29T11:12:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.181+0000 7ff833e00700 1 -- 172.21.15.12:0/82944774 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff83c0ccc90 con 0x7ff83c15b5d0 2024-03-29T11:12:22.159 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:22.159 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:12:22.159 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T11:12:22.160 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:12:22.161 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T11:12:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.164 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.165 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.166 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T11:12:22.167 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T11:12:22.168 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T11:12:22.169 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.170 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:12:22.171 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T11:12:22.172 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T11:12:22.173 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:12:22.173 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:12:22.173 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:12:22.173 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:12:22.173 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.173 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:22.173 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T11:12:22.173 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:12:22.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 -- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81c06adb0 msgr2=0x7ff81c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81c06adb0 0x7ff81c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff82c014040 tx=0x7ff82c008030 comp rx=0 tx=0).stop 2024-03-29T11:12:22.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 -- 172.21.15.12:0/82944774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 msgr2=0x7ff83c0721a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 0x7ff83c0721a0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7ff82800c750 tx=0x7ff82800cb10 comp rx=0 tx=0).stop 2024-03-29T11:12:22.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 -- 172.21.15.12:0/82944774 shutdown_connections 2024-03-29T11:12:22.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff81c06adb0 0x7ff81c06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff83c0726e0 0x7ff83c0c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff83c0ae930 0x7ff83c14dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 --2- 172.21.15.12:0/82944774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff83c15b5d0 0x7ff83c0721a0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 -- 172.21.15.12:0/82944774 >> 172.21.15.12:0/82944774 conn(0x7ff83c0cf190 msgr2=0x7ff83c0aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:22.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 -- 172.21.15.12:0/82944774 shutdown_connections 2024-03-29T11:12:22.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.183+0000 7ff842a00700 1 -- 172.21.15.12:0/82944774 wait complete. 2024-03-29T11:12:22.180 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.399+0000 7f05c0a00700 1 Processor -- start 2024-03-29T11:12:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.399+0000 7f05c0a00700 1 -- start start 2024-03-29T11:12:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.400+0000 7f05c0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05bc0ceec0 0x7f05bc1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.400+0000 7f05c0a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f05bc161800 0x7f05bc0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.400+0000 7f05c0a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc0d30d0 con 0x7f05bc15d640 2024-03-29T11:12:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.400+0000 7f05c0a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc161c90 con 0x7f05bc161800 2024-03-29T11:12:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.400+0000 7f05c0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0b6840 con 0x7f05bc0ceec0 2024-03-29T11:12:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.401+0000 7f05b9e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f05bc15d640 0x7f05bc0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59632/0 (socket says 172.21.15.12:59632) 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.401+0000 7f05b9e00700 1 -- 172.21.15.12:0/184870306 learned_addr learned my addr 172.21.15.12:0/184870306 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.401+0000 7f05ba800700 1 --2- 172.21.15.12:0/184870306 >> v2:172.21.15.159:3300/0 conn(0x7f05bc161800 0x7f05bc0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.401+0000 7f05b3e00700 1 --2- 172.21.15.12:0/184870306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05bc0ceec0 0x7f05bc1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.401+0000 7f05b9400700 1 -- 172.21.15.12:0/184870306 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493805845 0 0) 0x7f05bc0d30d0 con 0x7f05bc15d640 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.401+0000 7f05b9400700 1 -- 172.21.15.12:0/184870306 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a00035b0 con 0x7f05bc15d640 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.401+0000 7f05ba800700 1 -- 172.21.15.12:0/184870306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05bc0ceec0 msgr2=0x7f05bc1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.401+0000 7f05ba800700 1 --2- 172.21.15.12:0/184870306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05bc0ceec0 0x7f05bc1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.401+0000 7f05ba800700 1 -- 172.21.15.12:0/184870306 >> v1:172.21.15.12:6789/0 conn(0x7f05bc15d640 legacy=0x7f05bc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.401+0000 7f05ba800700 1 -- 172.21.15.12:0/184870306 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc0d2b30 con 0x7f05bc161800 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.402+0000 7f05ba800700 1 --2- 172.21.15.12:0/184870306 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05bc161800 0x7f05bc0b6cd0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009d90 tx=0x7f05b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bae0cdcb8e4d6c34 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.402+0000 7f05b9400700 1 -- 172.21.15.12:0/184870306 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b400e070 con 0x7f05bc161800 2024-03-29T11:12:22.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.402+0000 7f05b9400700 1 -- 172.21.15.12:0/184870306 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b400bb20 con 0x7f05bc161800 2024-03-29T11:12:22.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.402+0000 7f05b9400700 1 -- 172.21.15.12:0/184870306 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b400be90 con 0x7f05bc161800 2024-03-29T11:12:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.403+0000 7f05c0a00700 1 -- 172.21.15.12:0/184870306 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05bc161800 msgr2=0x7f05bc0b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.403+0000 7f05c0a00700 1 --2- 172.21.15.12:0/184870306 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05bc161800 0x7f05bc0b6cd0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009d90 tx=0x7f05b400b330 comp rx=0 tx=0).stop 2024-03-29T11:12:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.403+0000 7f05c0a00700 1 -- 172.21.15.12:0/184870306 shutdown_connections 2024-03-29T11:12:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.403+0000 7f05c0a00700 1 --2- 172.21.15.12:0/184870306 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05bc161800 0x7f05bc0b6cd0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.403+0000 7f05c0a00700 1 --2- 172.21.15.12:0/184870306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05bc0ceec0 0x7f05bc1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.403+0000 7f05c0a00700 1 -- 172.21.15.12:0/184870306 >> 172.21.15.12:0/184870306 conn(0x7f05bc08aff0 msgr2=0x7f05bc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:22.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.403+0000 7f05c0a00700 1 -- 172.21.15.12:0/184870306 shutdown_connections 2024-03-29T11:12:22.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.403+0000 7f05c0a00700 1 -- 172.21.15.12:0/184870306 wait complete. 2024-03-29T11:12:22.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.404+0000 7f05c0a00700 1 Processor -- start 2024-03-29T11:12:22.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.405+0000 7f05c0a00700 1 -- start start 2024-03-29T11:12:22.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.405+0000 7f05c0a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05bc0ceec0 0x7f05bc09d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05c0a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05bc15d640 0x7f05bc14cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05c0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05bc161800 0x7f05bc0c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05c0a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc14d810 con 0x7f05bc161800 2024-03-29T11:12:22.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05c0a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc14daf0 con 0x7f05bc0ceec0 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05c0a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc14ddd0 con 0x7f05bc15d640 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05ba800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05bc161800 0x7f05bc0c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05ba800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05bc161800 0x7f05bc0c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37384/0 (socket says 172.21.15.12:37384) 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05ba800700 1 -- 172.21.15.12:0/1279454548 learned_addr learned my addr 172.21.15.12:0/1279454548 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05b3e00700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05bc15d640 0x7f05bc14cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05b9e00700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05bc0ceec0 0x7f05bc09d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05ba800700 1 -- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05bc15d640 msgr2=0x7f05bc14cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05ba800700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05bc15d640 0x7f05bc14cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05ba800700 1 -- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05bc0ceec0 msgr2=0x7f05bc09d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05ba800700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05bc0ceec0 0x7f05bc09d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.406+0000 7f05ba800700 1 -- 172.21.15.12:0/1279454548 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b4009a70 con 0x7f05bc161800 2024-03-29T11:12:22.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.407+0000 7f05ba800700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05bc161800 0x7f05bc0c8f80 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f05b4014040 tx=0x7f05b4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:22.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.407+0000 7f05b2000700 1 -- 172.21.15.12:0/1279454548 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4003e10 con 0x7f05bc161800 2024-03-29T11:12:22.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.407+0000 7f05b2000700 1 -- 172.21.15.12:0/1279454548 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4020070 con 0x7f05bc161800 2024-03-29T11:12:22.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.407+0000 7f05b2000700 1 -- 172.21.15.12:0/1279454548 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b40276c0 con 0x7f05bc161800 2024-03-29T11:12:22.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.407+0000 7f05c0a00700 1 -- 172.21.15.12:0/1279454548 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc14e0b0 con 0x7f05bc161800 2024-03-29T11:12:22.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.407+0000 7f05c0a00700 1 -- 172.21.15.12:0/1279454548 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc14d0a0 con 0x7f05bc161800 2024-03-29T11:12:22.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.408+0000 7f05c0a00700 1 -- 172.21.15.12:0/1279454548 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc067720 con 0x7f05bc161800 2024-03-29T11:12:22.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.409+0000 7f05b2000700 1 -- 172.21.15.12:0/1279454548 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f05b4012430 con 0x7f05bc161800 2024-03-29T11:12:22.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.410+0000 7f05b2000700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f059806adb0 0x7f059806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.410+0000 7f05b2000700 1 -- 172.21.15.12:0/1279454548 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f05b4022070 con 0x7f05bc161800 2024-03-29T11:12:22.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.410+0000 7f05b9e00700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f059806adb0 0x7f059806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.411+0000 7f05b9e00700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f059806adb0 0x7f059806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f05ac005910 tx=0x7f05ac009040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:22.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.414+0000 7f05b2000700 1 -- 172.21.15.12:0/1279454548 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f05b4090020 con 0x7f05bc161800 2024-03-29T11:12:22.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.628+0000 7f05c0a00700 1 -- 172.21.15.12:0/1279454548 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05bc0b5c70 con 0x7f05bc161800 2024-03-29T11:12:22.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.629+0000 7f05b2000700 1 -- 172.21.15.12:0/1279454548 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+12469 (secure 0 0 0) 0x7f05b402c090 con 0x7f05bc161800 2024-03-29T11:12:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:22.607 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":187,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:12:21.632453+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 -- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f059806adb0 msgr2=0x7f059806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f059806adb0 0x7f059806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f05ac005910 tx=0x7f05ac009040 comp rx=0 tx=0).stop 2024-03-29T11:12:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 -- 172.21.15.12:0/1279454548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05bc161800 msgr2=0x7f05bc0c8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05bc161800 0x7f05bc0c8f80 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f05b4014040 tx=0x7f05b4004ad0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 -- 172.21.15.12:0/1279454548 shutdown_connections 2024-03-29T11:12:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f059806adb0 0x7f059806d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05bc161800 0x7f05bc0c8f80 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05bc0ceec0 0x7f05bc09d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 --2- 172.21.15.12:0/1279454548 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05bc15d640 0x7f05bc14cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 -- 172.21.15.12:0/1279454548 >> 172.21.15.12:0/1279454548 conn(0x7f05bc08aff0 msgr2=0x7f05bc0d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:22.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 -- 172.21.15.12:0/1279454548 shutdown_connections 2024-03-29T11:12:22.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.632+0000 7f05c0a00700 1 -- 172.21.15.12:0/1279454548 wait complete. 2024-03-29T11:12:22.629 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:12:22.630 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:12:22.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.854+0000 7ff416200700 1 Processor -- start 2024-03-29T11:12:22.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.854+0000 7ff416200700 1 -- start start 2024-03-29T11:12:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.855+0000 7ff416200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff41015d9d0 0x7ff4100d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.855+0000 7ff416200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff410161420 con 0x7ff4100d16a0 2024-03-29T11:12:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.855+0000 7ff416200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff410161700 con 0x7ff4100d2540 2024-03-29T11:12:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.855+0000 7ff416200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41015edc0 con 0x7ff41015d9d0 2024-03-29T11:12:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.855+0000 7ff40ea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff4100d16a0 0x7ff4100d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59634/0 (socket says 172.21.15.12:59634) 2024-03-29T11:12:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.855+0000 7ff40f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff41015d9d0 0x7ff4100d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.855+0000 7ff40ea00700 1 -- 172.21.15.12:0/3263154348 learned_addr learned my addr 172.21.15.12:0/3263154348 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.856+0000 7ff40f400700 1 -- 172.21.15.12:0/3263154348 >> v1:172.21.15.159:6789/0 conn(0x7ff4100d2540 legacy=0x7ff4100ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.856+0000 7ff40f400700 1 -- 172.21.15.12:0/3263154348 >> v1:172.21.15.12:6789/0 conn(0x7ff4100d16a0 legacy=0x7ff4100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.856+0000 7ff40f400700 1 -- 172.21.15.12:0/3263154348 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4101619e0 con 0x7ff41015d9d0 2024-03-29T11:12:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.856+0000 7ff40f400700 1 --2- 172.21.15.12:0/3263154348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 0x7ff4100d3c30 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7ff400009840 tx=0x7ff400009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d17b5f80b0e6bb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.856+0000 7ff40e000700 1 -- 172.21.15.12:0/3263154348 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400011830 con 0x7ff41015d9d0 2024-03-29T11:12:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.856+0000 7ff40e000700 1 -- 172.21.15.12:0/3263154348 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff400011990 con 0x7ff41015d9d0 2024-03-29T11:12:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.856+0000 7ff40e000700 1 -- 172.21.15.12:0/3263154348 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400011d20 con 0x7ff41015d9d0 2024-03-29T11:12:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.857+0000 7ff416200700 1 -- 172.21.15.12:0/3263154348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 msgr2=0x7ff4100d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.857+0000 7ff416200700 1 --2- 172.21.15.12:0/3263154348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 0x7ff4100d3c30 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7ff400009840 tx=0x7ff400009b50 comp rx=0 tx=0).stop 2024-03-29T11:12:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.857+0000 7ff416200700 1 -- 172.21.15.12:0/3263154348 shutdown_connections 2024-03-29T11:12:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.857+0000 7ff416200700 1 --2- 172.21.15.12:0/3263154348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 0x7ff4100d3c30 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.857+0000 7ff416200700 1 -- 172.21.15.12:0/3263154348 >> 172.21.15.12:0/3263154348 conn(0x7ff41008aff0 msgr2=0x7ff4100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.857+0000 7ff416200700 1 -- 172.21.15.12:0/3263154348 shutdown_connections 2024-03-29T11:12:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.857+0000 7ff416200700 1 -- 172.21.15.12:0/3263154348 wait complete. 2024-03-29T11:12:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.859+0000 7ff416200700 1 Processor -- start 2024-03-29T11:12:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.859+0000 7ff416200700 1 -- start start 2024-03-29T11:12:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff416200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4100d16a0 0x7ff4100b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff416200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4100d2540 0x7ff4101e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff416200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 0x7ff4101e77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff416200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100b3980 con 0x7ff41015d9d0 2024-03-29T11:12:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff416200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100b7b20 con 0x7ff4100d16a0 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff416200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100b3050 con 0x7ff4100d2540 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff40fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 0x7ff4101e77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff40fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 0x7ff4101e77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37392/0 (socket says 172.21.15.12:37392) 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff40fe00700 1 -- 172.21.15.12:0/1522979225 learned_addr learned my addr 172.21.15.12:0/1522979225 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff40ea00700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4100d2540 0x7ff4101e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.860+0000 7ff40f400700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4100d16a0 0x7ff4100b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff40fe00700 1 -- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4100d2540 msgr2=0x7ff4101e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff40fe00700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4100d2540 0x7ff4101e7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff40fe00700 1 -- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4100d16a0 msgr2=0x7ff4100b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff40fe00700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4100d16a0 0x7ff4100b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff40fe00700 1 -- 172.21.15.12:0/1522979225 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4000094a0 con 0x7ff41015d9d0 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff40f400700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4100d16a0 0x7ff4100b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff40ea00700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4100d2540 0x7ff4101e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff40fe00700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 0x7ff4101e77c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7ff40800cbc0 tx=0x7ff40800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff407400700 1 -- 172.21.15.12:0/1522979225 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff408003c00 con 0x7ff41015d9d0 2024-03-29T11:12:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff407400700 1 -- 172.21.15.12:0/1522979225 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff408003d60 con 0x7ff41015d9d0 2024-03-29T11:12:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff416200700 1 -- 172.21.15.12:0/1522979225 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4100b3330 con 0x7ff41015d9d0 2024-03-29T11:12:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff407400700 1 -- 172.21.15.12:0/1522979225 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800abf0 con 0x7ff41015d9d0 2024-03-29T11:12:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.861+0000 7ff416200700 1 -- 172.21.15.12:0/1522979225 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4100b4350 con 0x7ff41015d9d0 2024-03-29T11:12:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.862+0000 7ff416200700 1 -- 172.21.15.12:0/1522979225 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4100d1ca0 con 0x7ff41015d9d0 2024-03-29T11:12:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.863+0000 7ff407400700 1 -- 172.21.15.12:0/1522979225 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff40800ad50 con 0x7ff41015d9d0 2024-03-29T11:12:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.864+0000 7ff407400700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3f006adb0 0x7ff3f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.864+0000 7ff407400700 1 -- 172.21.15.12:0/1522979225 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 4901+0+0 (secure 0 0 0) 0x7ff408089240 con 0x7ff41015d9d0 2024-03-29T11:12:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.865+0000 7ff40f400700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3f006adb0 0x7ff3f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.865+0000 7ff40f400700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3f006adb0 0x7ff3f006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff400009790 tx=0x7ff40001c040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:22.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:22.869+0000 7ff407400700 1 -- 172.21.15.12:0/1522979225 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff408054620 con 0x7ff41015d9d0 2024-03-29T11:12:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.085+0000 7ff416200700 1 -- 172.21.15.12:0/1522979225 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff410067720 con 0x7ff41015d9d0 2024-03-29T11:12:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.085+0000 7ff407400700 1 -- 172.21.15.12:0/1522979225 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v187) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff408057c40 con 0x7ff41015d9d0 2024-03-29T11:12:23.064 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:12:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 -- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3f006adb0 msgr2=0x7ff3f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3f006adb0 0x7ff3f006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff400009790 tx=0x7ff40001c040 comp rx=0 tx=0).stop 2024-03-29T11:12:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 -- 172.21.15.12:0/1522979225 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 msgr2=0x7ff4101e77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 0x7ff4101e77c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7ff40800cbc0 tx=0x7ff40800cf80 comp rx=0 tx=0).stop 2024-03-29T11:12:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 -- 172.21.15.12:0/1522979225 shutdown_connections 2024-03-29T11:12:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff3f006adb0 0x7ff3f006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41015d9d0 0x7ff4101e77c0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4100d16a0 0x7ff4100b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 --2- 172.21.15.12:0/1522979225 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4100d2540 0x7ff4101e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 -- 172.21.15.12:0/1522979225 >> 172.21.15.12:0/1522979225 conn(0x7ff41008aff0 msgr2=0x7ff41015ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 -- 172.21.15.12:0/1522979225 shutdown_connections 2024-03-29T11:12:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.088+0000 7ff416200700 1 -- 172.21.15.12:0/1522979225 wait complete. 2024-03-29T11:12:23.084 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T11:12:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.305+0000 7f1d52c00700 1 Processor -- start 2024-03-29T11:12:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.305+0000 7f1d52c00700 1 -- start start 2024-03-29T11:12:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.306+0000 7f1d52c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1d4c09a910 0x7f1d4c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.306+0000 7f1d52c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4c0d08d0 con 0x7f1d4c099a60 2024-03-29T11:12:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.306+0000 7f1d52c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4c06c300 con 0x7f1d4c15d920 2024-03-29T11:12:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.306+0000 7f1d52c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4c06b9d0 con 0x7f1d4c09a910 2024-03-29T11:12:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.306+0000 7f1d50c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1d4c099a60 0x7f1d4c099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59642/0 (socket says 172.21.15.12:59642) 2024-03-29T11:12:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.306+0000 7f1d50c00700 1 -- 172.21.15.12:0/3680930684 learned_addr learned my addr 172.21.15.12:0/3680930684 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:23.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.306+0000 7f1d4b400700 1 --2- 172.21.15.12:0/3680930684 >> v2:172.21.15.159:3300/0 conn(0x7f1d4c09a910 0x7f1d4c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.307+0000 7f1d4aa00700 1 -- 172.21.15.12:0/3680930684 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175041442 0 0) 0x7f1d4c0d08d0 con 0x7f1d4c099a60 2024-03-29T11:12:23.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.307+0000 7f1d4aa00700 1 -- 172.21.15.12:0/3680930684 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d300035b0 con 0x7f1d4c099a60 2024-03-29T11:12:23.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.307+0000 7f1d4aa00700 1 -- 172.21.15.12:0/3680930684 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057997163 0 0) 0x7f1d4c06c300 con 0x7f1d4c15d920 2024-03-29T11:12:23.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.307+0000 7f1d4aa00700 1 -- 172.21.15.12:0/3680930684 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d4c0d08d0 con 0x7f1d4c15d920 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.307+0000 7f1d4aa00700 1 -- 172.21.15.12:0/3680930684 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 830748231 0 0) 0x7f1d300035b0 con 0x7f1d4c099a60 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.307+0000 7f1d4b400700 1 -- 172.21.15.12:0/3680930684 >> v1:172.21.15.159:6789/0 conn(0x7f1d4c15d920 legacy=0x7f1d4c0cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.307+0000 7f1d4b400700 1 -- 172.21.15.12:0/3680930684 >> v1:172.21.15.12:6789/0 conn(0x7f1d4c099a60 legacy=0x7f1d4c099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.307+0000 7f1d4b400700 1 -- 172.21.15.12:0/3680930684 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4c06bcb0 con 0x7f1d4c09a910 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.307+0000 7f1d4b400700 1 --2- 172.21.15.12:0/3680930684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d4c09a910 0x7f1d4c06c790 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009840 tx=0x7f1d38009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1d4975390e7326b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.308+0000 7f1d4aa00700 1 -- 172.21.15.12:0/3680930684 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d440019f0 con 0x7f1d4c09a910 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.308+0000 7f1d4aa00700 1 -- 172.21.15.12:0/3680930684 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d38004030 con 0x7f1d4c09a910 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.308+0000 7f1d4aa00700 1 -- 172.21.15.12:0/3680930684 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d380043c0 con 0x7f1d4c09a910 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.308+0000 7f1d52c00700 1 -- 172.21.15.12:0/3680930684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d4c09a910 msgr2=0x7f1d4c06c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.308+0000 7f1d52c00700 1 --2- 172.21.15.12:0/3680930684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d4c09a910 0x7f1d4c06c790 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009840 tx=0x7f1d38009b50 comp rx=0 tx=0).stop 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.308+0000 7f1d52c00700 1 -- 172.21.15.12:0/3680930684 shutdown_connections 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.308+0000 7f1d52c00700 1 --2- 172.21.15.12:0/3680930684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d4c09a910 0x7f1d4c06c790 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.308+0000 7f1d52c00700 1 -- 172.21.15.12:0/3680930684 >> 172.21.15.12:0/3680930684 conn(0x7f1d4c0d1350 msgr2=0x7f1d4c0d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.308+0000 7f1d52c00700 1 -- 172.21.15.12:0/3680930684 shutdown_connections 2024-03-29T11:12:23.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.308+0000 7f1d52c00700 1 -- 172.21.15.12:0/3680930684 wait complete. 2024-03-29T11:12:23.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.310+0000 7f1d52c00700 1 Processor -- start 2024-03-29T11:12:23.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.310+0000 7f1d52c00700 1 -- start start 2024-03-29T11:12:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.311+0000 7f1d52c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d4c099a60 0x7f1d4c15f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.311+0000 7f1d52c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d4c09a910 0x7f1d4c15f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.311+0000 7f1d52c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d4c15d920 0x7f1d4c09ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.311+0000 7f1d52c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c09f0e0 con 0x7f1d4c099a60 2024-03-29T11:12:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.311+0000 7f1d52c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c0c9b10 con 0x7f1d4c15d920 2024-03-29T11:12:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.311+0000 7f1d52c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c09eb20 con 0x7f1d4c09a910 2024-03-29T11:12:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d4be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d4c099a60 0x7f1d4c15f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d4be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d4c099a60 0x7f1d4c15f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37402/0 (socket says 172.21.15.12:37402) 2024-03-29T11:12:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d4be00700 1 -- 172.21.15.12:0/2420910306 learned_addr learned my addr 172.21.15.12:0/2420910306 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d50c00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d4c15d920 0x7f1d4c09ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d4b400700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d4c09a910 0x7f1d4c15f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d4be00700 1 -- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d4c09a910 msgr2=0x7f1d4c15f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d4be00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d4c09a910 0x7f1d4c15f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d4be00700 1 -- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d4c15d920 msgr2=0x7f1d4c09ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d4be00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d4c15d920 0x7f1d4c09ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d4be00700 1 -- 172.21.15.12:0/2420910306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d380094a0 con 0x7f1d4c099a60 2024-03-29T11:12:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d50c00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d4c15d920 0x7f1d4c09ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.312+0000 7f1d4be00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d4c099a60 0x7f1d4c15f260 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c00c530 tx=0x7f1d3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.313+0000 7f1d43e00700 1 -- 172.21.15.12:0/2420910306 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0052d0 con 0x7f1d4c099a60 2024-03-29T11:12:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.313+0000 7f1d43e00700 1 -- 172.21.15.12:0/2420910306 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c005450 con 0x7f1d4c099a60 2024-03-29T11:12:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.313+0000 7f1d52c00700 1 -- 172.21.15.12:0/2420910306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4c09ee00 con 0x7f1d4c099a60 2024-03-29T11:12:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.313+0000 7f1d43e00700 1 -- 172.21.15.12:0/2420910306 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0038c0 con 0x7f1d4c099a60 2024-03-29T11:12:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.313+0000 7f1d52c00700 1 -- 172.21.15.12:0/2420910306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4c1591d0 con 0x7f1d4c099a60 2024-03-29T11:12:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.314+0000 7f1d52c00700 1 -- 172.21.15.12:0/2420910306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4c0c9600 con 0x7f1d4c099a60 2024-03-29T11:12:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.315+0000 7f1d43e00700 1 -- 172.21.15.12:0/2420910306 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1d3c003a20 con 0x7f1d4c099a60 2024-03-29T11:12:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.316+0000 7f1d43e00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d2c06adb0 0x7f1d2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.316+0000 7f1d43e00700 1 -- 172.21.15.12:0/2420910306 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f1d3c088670 con 0x7f1d4c099a60 2024-03-29T11:12:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.316+0000 7f1d4b400700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d2c06adb0 0x7f1d2c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.317+0000 7f1d4b400700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d2c06adb0 0x7f1d2c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c09ad30 tx=0x7f1d38008030 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.320+0000 7f1d43e00700 1 -- 172.21.15.12:0/2420910306 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1d4c0c9600 con 0x7f1d4c099a60 2024-03-29T11:12:23.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.536+0000 7f1d52c00700 1 -- 172.21.15.12:0/2420910306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1d4c0c9600 con 0x7f1d4c099a60 2024-03-29T11:12:23.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.536+0000 7f1d43e00700 1 -- 172.21.15.12:0/2420910306 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v187) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1d3c017060 con 0x7f1d4c099a60 2024-03-29T11:12:23.514 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:23.514 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:12:23.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:23.515 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T11:12:23.515 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T11:12:23.515 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:23.515 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:12:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 -- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d2c06adb0 msgr2=0x7f1d2c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d2c06adb0 0x7f1d2c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c09ad30 tx=0x7f1d38008030 comp rx=0 tx=0).stop 2024-03-29T11:12:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 -- 172.21.15.12:0/2420910306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d4c099a60 msgr2=0x7f1d4c15f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d4c099a60 0x7f1d4c15f260 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c00c530 tx=0x7f1d3c00c8f0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 -- 172.21.15.12:0/2420910306 shutdown_connections 2024-03-29T11:12:23.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1d2c06adb0 0x7f1d2c06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d4c15d920 0x7f1d4c09ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d4c099a60 0x7f1d4c15f260 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 --2- 172.21.15.12:0/2420910306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d4c09a910 0x7f1d4c15f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 -- 172.21.15.12:0/2420910306 >> 172.21.15.12:0/2420910306 conn(0x7f1d4c0d1350 msgr2=0x7f1d4c0cca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:23.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 -- 172.21.15.12:0/2420910306 shutdown_connections 2024-03-29T11:12:23.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.539+0000 7f1d52c00700 1 -- 172.21.15.12:0/2420910306 wait complete. 2024-03-29T11:12:23.536 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T11:12:23.536 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T11:12:23.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.762+0000 7f8ee5800700 1 Processor -- start 2024-03-29T11:12:23.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.762+0000 7f8ee5800700 1 -- start start 2024-03-29T11:12:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.763+0000 7f8ee5800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ee00b56e0 0x7f8ee00d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.763+0000 7f8ee5800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8ee00d1770 0x7f8ee006ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.763+0000 7f8ee5800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee00ceb70 con 0x7f8ee015d7f0 2024-03-29T11:12:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.763+0000 7f8ee5800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee015dbd0 con 0x7f8ee00b56e0 2024-03-29T11:12:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.763+0000 7f8ee5800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee00cf000 con 0x7f8ee00d1770 2024-03-29T11:12:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.763+0000 7f8edea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8ee015d7f0 0x7f8ee00b4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59652/0 (socket says 172.21.15.12:59652) 2024-03-29T11:12:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.763+0000 7f8edea00700 1 -- 172.21.15.12:0/2788206957 learned_addr learned my addr 172.21.15.12:0/2788206957 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.763+0000 7f8ede000700 1 --2- 172.21.15.12:0/2788206957 >> v2:172.21.15.12:3300/0 conn(0x7f8ee00b56e0 0x7f8ee00d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.764+0000 7f8edf400700 1 --2- 172.21.15.12:0/2788206957 >> v2:172.21.15.159:3300/0 conn(0x7f8ee00d1770 0x7f8ee006ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.764+0000 7f8edd600700 1 -- 172.21.15.12:0/2788206957 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024629600 0 0) 0x7f8ee00ceb70 con 0x7f8ee015d7f0 2024-03-29T11:12:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.764+0000 7f8ede000700 1 -- 172.21.15.12:0/2788206957 >> v2:172.21.15.159:3300/0 conn(0x7f8ee00d1770 msgr2=0x7f8ee006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.764+0000 7f8ede000700 1 --2- 172.21.15.12:0/2788206957 >> v2:172.21.15.159:3300/0 conn(0x7f8ee00d1770 0x7f8ee006ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.764+0000 7f8ede000700 1 -- 172.21.15.12:0/2788206957 >> v1:172.21.15.12:6789/0 conn(0x7f8ee015d7f0 legacy=0x7f8ee00b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.764+0000 7f8ede000700 1 -- 172.21.15.12:0/2788206957 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee00cdf60 con 0x7f8ee00b56e0 2024-03-29T11:12:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.764+0000 7f8edf400700 1 --2- 172.21.15.12:0/2788206957 >> v2:172.21.15.159:3300/0 conn(0x7f8ee00d1770 0x7f8ee006ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.764+0000 7f8ede000700 1 --2- 172.21.15.12:0/2788206957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee00b56e0 0x7f8ee00d25d0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00b6120 tx=0x7f8ecc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f6543ea8541082 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.764+0000 7f8edd600700 1 -- 172.21.15.12:0/2788206957 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc011830 con 0x7f8ee00b56e0 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.765+0000 7f8edd600700 1 -- 172.21.15.12:0/2788206957 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc011990 con 0x7f8ee00b56e0 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.765+0000 7f8edd600700 1 -- 172.21.15.12:0/2788206957 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc011d20 con 0x7f8ee00b56e0 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.765+0000 7f8ee5800700 1 -- 172.21.15.12:0/2788206957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee00b56e0 msgr2=0x7f8ee00d25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.765+0000 7f8ee5800700 1 --2- 172.21.15.12:0/2788206957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee00b56e0 0x7f8ee00d25d0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00b6120 tx=0x7f8ecc009b20 comp rx=0 tx=0).stop 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.765+0000 7f8ee5800700 1 -- 172.21.15.12:0/2788206957 shutdown_connections 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.765+0000 7f8ee5800700 1 --2- 172.21.15.12:0/2788206957 >> v2:172.21.15.159:3300/0 conn(0x7f8ee00d1770 0x7f8ee006ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.765+0000 7f8ee5800700 1 --2- 172.21.15.12:0/2788206957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee00b56e0 0x7f8ee00d25d0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.765+0000 7f8ee5800700 1 -- 172.21.15.12:0/2788206957 >> 172.21.15.12:0/2788206957 conn(0x7f8ee008aff0 msgr2=0x7f8ee00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.765+0000 7f8ee5800700 1 -- 172.21.15.12:0/2788206957 shutdown_connections 2024-03-29T11:12:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.765+0000 7f8ee5800700 1 -- 172.21.15.12:0/2788206957 wait complete. 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.767+0000 7f8ee5800700 1 Processor -- start 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.767+0000 7f8ee5800700 1 -- start start 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.768+0000 7f8ee5800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee00b56e0 0x7f8ee00c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.768+0000 7f8ee5800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ee00d1770 0x7f8ee015eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.768+0000 7f8ee5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee015d7f0 0x7f8ee01f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.768+0000 7f8ee5800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee00c7780 con 0x7f8ee015d7f0 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.768+0000 7f8ee5800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee015f0c0 con 0x7f8ee00d1770 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.768+0000 7f8ee5800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee015fba0 con 0x7f8ee00b56e0 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.768+0000 7f8edf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee015d7f0 0x7f8ee01f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.768+0000 7f8edf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee015d7f0 0x7f8ee01f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37428/0 (socket says 172.21.15.12:37428) 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.768+0000 7f8edf400700 1 -- 172.21.15.12:0/3246874254 learned_addr learned my addr 172.21.15.12:0/3246874254 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8edea00700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee00b56e0 0x7f8ee00c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8ede000700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ee00d1770 0x7f8ee015eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8edf400700 1 -- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee00b56e0 msgr2=0x7f8ee00c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8edf400700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee00b56e0 0x7f8ee00c7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8edf400700 1 -- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ee00d1770 msgr2=0x7f8ee015eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8edf400700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ee00d1770 0x7f8ee015eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8edf400700 1 -- 172.21.15.12:0/3246874254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc0094a0 con 0x7f8ee015d7f0 2024-03-29T11:12:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8ede000700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ee00d1770 0x7f8ee015eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8edea00700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee00b56e0 0x7f8ee00c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8edf400700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee015d7f0 0x7f8ee01f6a40 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8007ef0 tx=0x7f8ed800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8ed6a00700 1 -- 172.21.15.12:0/3246874254 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed800de60 con 0x7f8ee015d7f0 2024-03-29T11:12:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.769+0000 7f8ed6a00700 1 -- 172.21.15.12:0/3246874254 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed800f040 con 0x7f8ee015d7f0 2024-03-29T11:12:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.770+0000 7f8ed6a00700 1 -- 172.21.15.12:0/3246874254 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8003c60 con 0x7f8ee015d7f0 2024-03-29T11:12:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.770+0000 7f8ee5800700 1 -- 172.21.15.12:0/3246874254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee015f300 con 0x7f8ee015d7f0 2024-03-29T11:12:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.770+0000 7f8ee5800700 1 -- 172.21.15.12:0/3246874254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee015f8c0 con 0x7f8ee015d7f0 2024-03-29T11:12:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.771+0000 7f8ee5800700 1 -- 172.21.15.12:0/3246874254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee00c69d0 con 0x7f8ee015d7f0 2024-03-29T11:12:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.772+0000 7f8ed6a00700 1 -- 172.21.15.12:0/3246874254 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8ed8004030 con 0x7f8ee015d7f0 2024-03-29T11:12:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.772+0000 7f8ed6a00700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ec006adb0 0x7f8ec006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.773+0000 7f8ed6a00700 1 -- 172.21.15.12:0/3246874254 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f8ed8089470 con 0x7f8ee015d7f0 2024-03-29T11:12:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.773+0000 7f8edea00700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ec006adb0 0x7f8ec006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.774+0000 7f8edea00700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ec006adb0 0x7f8ec006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000f60 tx=0x7f8ed0007cc0 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:23.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.777+0000 7f8ed6a00700 1 -- 172.21.15.12:0/3246874254 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ed8054740 con 0x7f8ee015d7f0 2024-03-29T11:12:23.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:23.991+0000 7f8ee5800700 1 -- 172.21.15.12:0/3246874254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8ee0067720 con 0x7f8ee015d7f0 2024-03-29T11:12:24.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:24.996+0000 7f8ed6a00700 1 -- 172.21.15.12:0/3246874254 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v188) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8ed8057d60 con 0x7f8ee015d7f0 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.000+0000 7f8ee5800700 1 -- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ec006adb0 msgr2=0x7f8ec006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.000+0000 7f8ee5800700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ec006adb0 0x7f8ec006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000f60 tx=0x7f8ed0007cc0 comp rx=0 tx=0).stop 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.000+0000 7f8ee5800700 1 -- 172.21.15.12:0/3246874254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee015d7f0 msgr2=0x7f8ee01f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.000+0000 7f8ee5800700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee015d7f0 0x7f8ee01f6a40 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8007ef0 tx=0x7f8ed800d3b0 comp rx=0 tx=0).stop 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.001+0000 7f8ee5800700 1 -- 172.21.15.12:0/3246874254 shutdown_connections 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.001+0000 7f8ee5800700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8ec006adb0 0x7f8ec006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.001+0000 7f8ee5800700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee015d7f0 0x7f8ee01f6a40 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.001+0000 7f8ee5800700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee00b56e0 0x7f8ee00c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.001+0000 7f8ee5800700 1 --2- 172.21.15.12:0/3246874254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ee00d1770 0x7f8ee015eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.001+0000 7f8ee5800700 1 -- 172.21.15.12:0/3246874254 >> 172.21.15.12:0/3246874254 conn(0x7f8ee008aff0 msgr2=0x7f8ee00b4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.001+0000 7f8ee5800700 1 -- 172.21.15.12:0/3246874254 shutdown_connections 2024-03-29T11:12:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.001+0000 7f8ee5800700 1 -- 172.21.15.12:0/3246874254 wait complete. 2024-03-29T11:12:24.983 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T11:12:25.004 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T11:12:25.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.235+0000 7fc0eb000700 1 Processor -- start 2024-03-29T11:12:25.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.236+0000 7fc0eb000700 1 -- start start 2024-03-29T11:12:25.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.237+0000 7fc0eb000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc0e40d2540 0x7fc0e41687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:25.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.237+0000 7fc0eb000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e4161420 con 0x7fc0e415d9d0 2024-03-29T11:12:25.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.237+0000 7fc0eb000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e4161700 con 0x7fc0e40d16a0 2024-03-29T11:12:25.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.237+0000 7fc0eb000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e40d4340 con 0x7fc0e40d2540 2024-03-29T11:12:25.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.237+0000 7fc0e9000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc0e40d2540 0x7fc0e41687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:25.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.237+0000 7fc0e3e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc0e415d9d0 0x7fc0e40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59654/0 (socket says 172.21.15.12:59654) 2024-03-29T11:12:25.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.237+0000 7fc0e3e00700 1 -- 172.21.15.12:0/3424176938 learned_addr learned my addr 172.21.15.12:0/3424176938 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:25.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.238+0000 7fc0e9000700 1 -- 172.21.15.12:0/3424176938 >> v1:172.21.15.159:6789/0 conn(0x7fc0e40d16a0 legacy=0x7fc0e40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.238+0000 7fc0e9000700 1 -- 172.21.15.12:0/3424176938 >> v1:172.21.15.12:6789/0 conn(0x7fc0e415d9d0 legacy=0x7fc0e40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.238+0000 7fc0e9000700 1 -- 172.21.15.12:0/3424176938 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e41619e0 con 0x7fc0e40d2540 2024-03-29T11:12:25.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.238+0000 7fc0e2a00700 1 -- 172.21.15.12:0/3424176938 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289412175 0 0) 0x7fc0e4161420 con 0x7fc0e415d9d0 2024-03-29T11:12:25.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.238+0000 7fc0e9000700 1 --2- 172.21.15.12:0/3424176938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 0x7fc0e41687b0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009d90 tx=0x7fc0dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1b5d10968959d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:25.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.238+0000 7fc0e2a00700 1 -- 172.21.15.12:0/3424176938 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc00e070 con 0x7fc0e40d2540 2024-03-29T11:12:25.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.238+0000 7fc0e2a00700 1 -- 172.21.15.12:0/3424176938 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0dc00bbd0 con 0x7fc0e40d2540 2024-03-29T11:12:25.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.238+0000 7fc0e2a00700 1 -- 172.21.15.12:0/3424176938 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc012430 con 0x7fc0e40d2540 2024-03-29T11:12:25.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.239+0000 7fc0eb000700 1 -- 172.21.15.12:0/3424176938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 msgr2=0x7fc0e41687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.239+0000 7fc0eb000700 1 --2- 172.21.15.12:0/3424176938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 0x7fc0e41687b0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009d90 tx=0x7fc0dc00b330 comp rx=0 tx=0).stop 2024-03-29T11:12:25.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.239+0000 7fc0eb000700 1 -- 172.21.15.12:0/3424176938 shutdown_connections 2024-03-29T11:12:25.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.239+0000 7fc0eb000700 1 --2- 172.21.15.12:0/3424176938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 0x7fc0e41687b0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:25.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.239+0000 7fc0eb000700 1 -- 172.21.15.12:0/3424176938 >> 172.21.15.12:0/3424176938 conn(0x7fc0e408aff0 msgr2=0x7fc0e40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:25.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.239+0000 7fc0eb000700 1 -- 172.21.15.12:0/3424176938 shutdown_connections 2024-03-29T11:12:25.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.239+0000 7fc0eb000700 1 -- 172.21.15.12:0/3424176938 wait complete. 2024-03-29T11:12:25.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.241+0000 7fc0eb000700 1 Processor -- start 2024-03-29T11:12:25.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.241+0000 7fc0eb000700 1 -- start start 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0eb000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0e40d16a0 0x7fc0e40b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0eb000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 0x7fc0e40b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0eb000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0e415d9d0 0x7fc0e415c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0eb000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40cbb20 con 0x7fc0e40d2540 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0eb000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40b8700 con 0x7fc0e40d16a0 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0eb000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e415c660 con 0x7fc0e415d9d0 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0e3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 0x7fc0e40b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0e3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 0x7fc0e40b81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37448/0 (socket says 172.21.15.12:37448) 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0e3400700 1 -- 172.21.15.12:0/335463760 learned_addr learned my addr 172.21.15.12:0/335463760 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0e3e00700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0e40d16a0 0x7fc0e40b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.242+0000 7fc0e9000700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0e415d9d0 0x7fc0e415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:25.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0e3400700 1 -- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0e415d9d0 msgr2=0x7fc0e415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0e3400700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0e415d9d0 0x7fc0e415c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:25.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0e3400700 1 -- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0e40d16a0 msgr2=0x7fc0e40b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0e3400700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0e40d16a0 0x7fc0e40b8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:25.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0e3400700 1 -- 172.21.15.12:0/335463760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc009a70 con 0x7fc0e40d2540 2024-03-29T11:12:25.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0e3400700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 0x7fc0e40b81c0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000c500 tx=0x7fc0d000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:25.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0dbe00700 1 -- 172.21.15.12:0/335463760 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d00052d0 con 0x7fc0e40d2540 2024-03-29T11:12:25.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0dbe00700 1 -- 172.21.15.12:0/335463760 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d0005450 con 0x7fc0e40d2540 2024-03-29T11:12:25.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0eb000700 1 -- 172.21.15.12:0/335463760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e40cb180 con 0x7fc0e40d2540 2024-03-29T11:12:25.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0dbe00700 1 -- 172.21.15.12:0/335463760 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0003910 con 0x7fc0e40d2540 2024-03-29T11:12:25.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.243+0000 7fc0eb000700 1 -- 172.21.15.12:0/335463760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e40cb740 con 0x7fc0e40d2540 2024-03-29T11:12:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.244+0000 7fc0eb000700 1 -- 172.21.15.12:0/335463760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e40d1ca0 con 0x7fc0e40d2540 2024-03-29T11:12:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.246+0000 7fc0dbe00700 1 -- 172.21.15.12:0/335463760 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc0d0003a70 con 0x7fc0e40d2540 2024-03-29T11:12:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.246+0000 7fc0dbe00700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc0c406adb0 0x7fc0c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.247+0000 7fc0dbe00700 1 -- 172.21.15.12:0/335463760 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5271+0+0 (secure 0 0 0) 0x7fc0d0088a80 con 0x7fc0e40d2540 2024-03-29T11:12:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.247+0000 7fc0e3e00700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc0c406adb0 0x7fc0c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:25.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.248+0000 7fc0e3e00700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc0c406adb0 0x7fc0c406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4000f60 tx=0x7fc0d4004a50 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:25.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.251+0000 7fc0dbe00700 1 -- 172.21.15.12:0/335463760 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0d0053d50 con 0x7fc0e40d2540 2024-03-29T11:12:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:25.464+0000 7fc0eb000700 1 -- 172.21.15.12:0/335463760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc0e40cbc60 con 0x7fc0e40d2540 2024-03-29T11:12:25.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.005+0000 7fc0dbe00700 1 -- 172.21.15.12:0/335463760 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v189) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc0e40cbc60 con 0x7fc0e40d2540 2024-03-29T11:12:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.012+0000 7fc0eb000700 1 -- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc0c406adb0 msgr2=0x7fc0c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.012+0000 7fc0eb000700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc0c406adb0 0x7fc0c406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4000f60 tx=0x7fc0d4004a50 comp rx=0 tx=0).stop 2024-03-29T11:12:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.012+0000 7fc0eb000700 1 -- 172.21.15.12:0/335463760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 msgr2=0x7fc0e40b81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.012+0000 7fc0eb000700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 0x7fc0e40b81c0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000c500 tx=0x7fc0d000c810 comp rx=0 tx=0).stop 2024-03-29T11:12:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.012+0000 7fc0eb000700 1 -- 172.21.15.12:0/335463760 shutdown_connections 2024-03-29T11:12:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.012+0000 7fc0eb000700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc0c406adb0 0x7fc0c406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.012+0000 7fc0eb000700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0e415d9d0 0x7fc0e415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.012+0000 7fc0eb000700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0e40d16a0 0x7fc0e40b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.012+0000 7fc0eb000700 1 --2- 172.21.15.12:0/335463760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e40d2540 0x7fc0e40b81c0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.012+0000 7fc0eb000700 1 -- 172.21.15.12:0/335463760 >> 172.21.15.12:0/335463760 conn(0x7fc0e408aff0 msgr2=0x7fc0e40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.015+0000 7fc0eb000700 1 -- 172.21.15.12:0/335463760 shutdown_connections 2024-03-29T11:12:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.016+0000 7fc0eb000700 1 -- 172.21.15.12:0/335463760 wait complete. 2024-03-29T11:12:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T11:12:26.022 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T11:12:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.247+0000 7f334be00700 1 Processor -- start 2024-03-29T11:12:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.247+0000 7f334be00700 1 -- start start 2024-03-29T11:12:26.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.248+0000 7f334be00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f33440d21f0 0x7f33440ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.248+0000 7f334be00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33440ce270 con 0x7f33441611b0 2024-03-29T11:12:26.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.248+0000 7f334be00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334415b600 con 0x7f33440cec40 2024-03-29T11:12:26.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.248+0000 7f334be00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33440d2760 con 0x7f33440d21f0 2024-03-29T11:12:26.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.248+0000 7f3349400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f33441611b0 0x7f334415ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59656/0 (socket says 172.21.15.12:59656) 2024-03-29T11:12:26.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.248+0000 7f3349400700 1 -- 172.21.15.12:0/2045049718 learned_addr learned my addr 172.21.15.12:0/2045049718 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:26.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611149240 0 0) 0x7f33440ce270 con 0x7f33441611b0 2024-03-29T11:12:26.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3348a00700 1 --2- 172.21.15.12:0/2045049718 >> v2:172.21.15.159:3300/0 conn(0x7f33440d21f0 0x7f33440ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f332c0035b0 con 0x7f33441611b0 2024-03-29T11:12:26.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1043287091 0 0) 0x7f332c0035b0 con 0x7f33441611b0 2024-03-29T11:12:26.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 >> v2:172.21.15.159:3300/0 conn(0x7f33440d21f0 msgr2=0x7f33440ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 --2- 172.21.15.12:0/2045049718 >> v2:172.21.15.159:3300/0 conn(0x7f33440d21f0 0x7f33440ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 >> v1:172.21.15.159:6789/0 conn(0x7f33440cec40 legacy=0x7f3344161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33440d2a40 con 0x7f33441611b0 2024-03-29T11:12:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f3338001670 con 0x7f33441611b0 2024-03-29T11:12:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 >> v1:172.21.15.12:6789/0 conn(0x7f33441611b0 legacy=0x7f334415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f332c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f332c004850 0x7f332c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f332c005210 0x7f332c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33440d2760 con 0x7f332c003a00 2024-03-29T11:12:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c00c0b0 con 0x7f332c005210 2024-03-29T11:12:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.249+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c00c270 con 0x7f332c004850 2024-03-29T11:12:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3349400700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f332c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3348a00700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f332c004850 0x7f332c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3349e00700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f332c005210 0x7f332c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3349400700 1 -- 172.21.15.12:0/2045049718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f332c004850 msgr2=0x7f332c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3349400700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f332c004850 0x7f332c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3349400700 1 -- 172.21.15.12:0/2045049718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f332c005210 msgr2=0x7f332c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3349400700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f332c005210 0x7f332c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3349400700 1 -- 172.21.15.12:0/2045049718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c0035b0 con 0x7f332c003a00 2024-03-29T11:12:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3349e00700 1 -- 172.21.15.12:0/2045049718 reap_dead start 2024-03-29T11:12:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3349400700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f332c003e20 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f333800c240 tx=0x7f332c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25563ae5eb4321f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338007c10 con 0x7f332c003a00 2024-03-29T11:12:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333800ce00 con 0x7f332c003a00 2024-03-29T11:12:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.250+0000 7f3343e00700 1 -- 172.21.15.12:0/2045049718 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338005ba0 con 0x7f332c003a00 2024-03-29T11:12:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.251+0000 7f334be00700 1 -- 172.21.15.12:0/2045049718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 msgr2=0x7f332c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.251+0000 7f334be00700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f332c003e20 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f333800c240 tx=0x7f332c005880 comp rx=0 tx=0).stop 2024-03-29T11:12:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.251+0000 7f334be00700 1 -- 172.21.15.12:0/2045049718 shutdown_connections 2024-03-29T11:12:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.251+0000 7f334be00700 1 --2- 172.21.15.12:0/2045049718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f332c003e20 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.251+0000 7f334be00700 1 -- 172.21.15.12:0/2045049718 >> 172.21.15.12:0/2045049718 conn(0x7f334408aff0 msgr2=0x7f33440d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.251+0000 7f334be00700 1 -- 172.21.15.12:0/2045049718 shutdown_connections 2024-03-29T11:12:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.251+0000 7f334be00700 1 -- 172.21.15.12:0/2045049718 wait complete. 2024-03-29T11:12:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.253+0000 7f334be00700 1 Processor -- start 2024-03-29T11:12:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.253+0000 7f334be00700 1 -- start start 2024-03-29T11:12:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.254+0000 7f334be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f33441f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.254+0000 7f334be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f334414e160 0x7f33440cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.254+0000 7f334be00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33440cd780 0x7f3344148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.254+0000 7f334be00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33440cc1a0 con 0x7f332c003a00 2024-03-29T11:12:26.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.254+0000 7f334be00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33440cc480 con 0x7f33440cd780 2024-03-29T11:12:26.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.254+0000 7f334be00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33440cc760 con 0x7f334414e160 2024-03-29T11:12:26.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.254+0000 7f3349400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f33441f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.254+0000 7f3349400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f33441f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37462/0 (socket says 172.21.15.12:37462) 2024-03-29T11:12:26.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.254+0000 7f3349400700 1 -- 172.21.15.12:0/1932789088 learned_addr learned my addr 172.21.15.12:0/1932789088 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:26.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3349e00700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33440cd780 0x7f3344148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3349400700 1 -- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f334414e160 msgr2=0x7f33440cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3349400700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f334414e160 0x7f33440cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3349400700 1 -- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33440cd780 msgr2=0x7f3344148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3349400700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33440cd780 0x7f3344148610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3349400700 1 -- 172.21.15.12:0/1932789088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33440cca40 con 0x7f332c003a00 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3349e00700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33440cd780 0x7f3344148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3349400700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f33441f2840 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f332c00c080 tx=0x7f3338001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3341600700 1 -- 172.21.15.12:0/1932789088 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338001670 con 0x7f332c003a00 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3341600700 1 -- 172.21.15.12:0/1932789088 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33380070d0 con 0x7f332c003a00 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f334be00700 1 -- 172.21.15.12:0/1932789088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33440ccd20 con 0x7f332c003a00 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f3341600700 1 -- 172.21.15.12:0/1932789088 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338007480 con 0x7f332c003a00 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.255+0000 7f334be00700 1 -- 172.21.15.12:0/1932789088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33440cbe50 con 0x7f332c003a00 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.256+0000 7f334be00700 1 -- 172.21.15.12:0/1932789088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3344067720 con 0x7f332c003a00 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.258+0000 7f3341600700 1 -- 172.21.15.12:0/1932789088 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f33380075e0 con 0x7f332c003a00 2024-03-29T11:12:26.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.259+0000 7f3341600700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f332406adb0 0x7f332406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.259+0000 7f3341600700 1 -- 172.21.15.12:0/1932789088 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f333808ac50 con 0x7f332c003a00 2024-03-29T11:12:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.259+0000 7f3348a00700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f332406adb0 0x7f332406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.260+0000 7f3348a00700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f332406adb0 0x7f332406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3330006f10 tx=0x7f3330006e80 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.263+0000 7f3341600700 1 -- 172.21.15.12:0/1932789088 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3338055dc0 con 0x7f332c003a00 2024-03-29T11:12:26.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.522+0000 7f334be00700 1 -- 172.21.15.12:0/1932789088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f33440b94b0 con 0x7f332c003a00 2024-03-29T11:12:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.531+0000 7f3341600700 1 -- 172.21.15.12:0/1932789088 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v85) v1 ==== 174+0+0 (secure 0 0 0) 0x7f33380593e0 con 0x7f332c003a00 2024-03-29T11:12:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.536+0000 7f334be00700 1 -- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f332406adb0 msgr2=0x7f332406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.536+0000 7f334be00700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f332406adb0 0x7f332406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3330006f10 tx=0x7f3330006e80 comp rx=0 tx=0).stop 2024-03-29T11:12:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.536+0000 7f334be00700 1 -- 172.21.15.12:0/1932789088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 msgr2=0x7f33441f2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.536+0000 7f334be00700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f33441f2840 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f332c00c080 tx=0x7f3338001a00 comp rx=0 tx=0).stop 2024-03-29T11:12:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.537+0000 7f334be00700 1 -- 172.21.15.12:0/1932789088 shutdown_connections 2024-03-29T11:12:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.537+0000 7f334be00700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f332406adb0 0x7f332406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.537+0000 7f334be00700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33440cd780 0x7f3344148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.537+0000 7f334be00700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f332c003a00 0x7f33441f2840 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.537+0000 7f334be00700 1 --2- 172.21.15.12:0/1932789088 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f334414e160 0x7f33440cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.537+0000 7f334be00700 1 -- 172.21.15.12:0/1932789088 >> 172.21.15.12:0/1932789088 conn(0x7f334408aff0 msgr2=0x7f334415b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.538+0000 7f334be00700 1 -- 172.21.15.12:0/1932789088 shutdown_connections 2024-03-29T11:12:26.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.540+0000 7f334be00700 1 -- 172.21.15.12:0/1932789088 wait complete. 2024-03-29T11:12:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 22 and data pool 23 2024-03-29T11:12:26.549 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:26.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.770+0000 7f1b4c200700 1 Processor -- start 2024-03-29T11:12:26.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.770+0000 7f1b4c200700 1 -- start start 2024-03-29T11:12:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.771+0000 7f1b4c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b44160f10 0x7f1b440d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.771+0000 7f1b4c200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440ceda0 con 0x7f1b44161590 2024-03-29T11:12:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.771+0000 7f1b4c200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b441612f0 con 0x7f1b440cf230 2024-03-29T11:12:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.771+0000 7f1b4c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b440ce470 con 0x7f1b44160f10 2024-03-29T11:12:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.771+0000 7f1b4a200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1b44161590 0x7f1b440b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59662/0 (socket says 172.21.15.12:59662) 2024-03-29T11:12:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.771+0000 7f1b4a200700 1 -- 172.21.15.12:0/2499714022 learned_addr learned my addr 172.21.15.12:0/2499714022 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b49800700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b44160f10 0x7f1b440d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755271620 0 0) 0x7f1b440ceda0 con 0x7f1b44161590 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b2c0035b0 con 0x7f1b44161590 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123745478 0 0) 0x7f1b441612f0 con 0x7f1b440cf230 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b440ceda0 con 0x7f1b440cf230 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 657493863 0 0) 0x7f1b2c0035b0 con 0x7f1b44161590 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b44160f10 msgr2=0x7f1b440d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b44160f10 0x7f1b440d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 >> v1:172.21.15.159:6789/0 conn(0x7f1b440cf230 legacy=0x7f1b440d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b440ce750 con 0x7f1b44161590 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f1b3c0019f0 con 0x7f1b44161590 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b49800700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b44160f10 0x7f1b440d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 >> v1:172.21.15.12:6789/0 conn(0x7f1b44161590 legacy=0x7f1b440b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b2c003bc0 0x7f1b2c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2c004850 0x7f1b2c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b44160f10 0x7f1b2c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b440ce470 con 0x7f1b2c004850 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c00baf0 con 0x7f1b2c003bc0 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.772+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c00bcb0 con 0x7f1b44160f10 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.773+0000 7f1b48e00700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2c004850 0x7f1b2c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.773+0000 7f1b49800700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b2c003bc0 0x7f1b2c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.773+0000 7f1b4a200700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b44160f10 0x7f1b2c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.773+0000 7f1b48e00700 1 -- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b44160f10 msgr2=0x7f1b2c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.773+0000 7f1b48e00700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b44160f10 0x7f1b2c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.773+0000 7f1b48e00700 1 -- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b2c003bc0 msgr2=0x7f1b2c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.773+0000 7f1b48e00700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b2c003bc0 0x7f1b2c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.773+0000 7f1b48e00700 1 -- 172.21.15.12:0/2499714022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b380094a0 con 0x7f1b2c004850 2024-03-29T11:12:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.773+0000 7f1b4a200700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b44160f10 0x7f1b2c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.773+0000 7f1b48e00700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2c004850 0x7f1b2c004cd0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f1b30007fd0 tx=0x7f1b2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca070635fe4ccdf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.774+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30001430 con 0x7f1b2c004850 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.774+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b300052d0 con 0x7f1b2c004850 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.774+0000 7f1b43e00700 1 -- 172.21.15.12:0/2499714022 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30001d40 con 0x7f1b2c004850 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.774+0000 7f1b4c200700 1 -- 172.21.15.12:0/2499714022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2c004850 msgr2=0x7f1b2c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.774+0000 7f1b4c200700 1 --2- 172.21.15.12:0/2499714022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2c004850 0x7f1b2c004cd0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f1b30007fd0 tx=0x7f1b2c005880 comp rx=0 tx=0).stop 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.774+0000 7f1b4a200700 1 -- 172.21.15.12:0/2499714022 reap_dead start 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.774+0000 7f1b4c200700 1 -- 172.21.15.12:0/2499714022 shutdown_connections 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.774+0000 7f1b4c200700 1 -- 172.21.15.12:0/2499714022 >> 172.21.15.12:0/2499714022 conn(0x7f1b4408aff0 msgr2=0x7f1b440d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.774+0000 7f1b4c200700 1 -- 172.21.15.12:0/2499714022 shutdown_connections 2024-03-29T11:12:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.774+0000 7f1b4c200700 1 -- 172.21.15.12:0/2499714022 wait complete. 2024-03-29T11:12:26.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.776+0000 7f1b4c200700 1 Processor -- start 2024-03-29T11:12:26.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.776+0000 7f1b4c200700 1 -- start start 2024-03-29T11:12:26.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.777+0000 7f1b4c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b2c004850 0x7f1b440b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.777+0000 7f1b4c200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b440b21a0 0x7f1b4414c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.777+0000 7f1b4c200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b440b84b0 0x7f1b440b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.777+0000 7f1b4c200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b440b3500 con 0x7f1b440b84b0 2024-03-29T11:12:26.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.777+0000 7f1b4c200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b440b2620 con 0x7f1b440b21a0 2024-03-29T11:12:26.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.777+0000 7f1b4c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b440b2880 con 0x7f1b2c004850 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.777+0000 7f1b4a200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b440b84b0 0x7f1b440b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.777+0000 7f1b4a200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b440b84b0 0x7f1b440b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37474/0 (socket says 172.21.15.12:37474) 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.777+0000 7f1b4a200700 1 -- 172.21.15.12:0/2063651871 learned_addr learned my addr 172.21.15.12:0/2063651871 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b49800700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b2c004850 0x7f1b440b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b48e00700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b440b21a0 0x7f1b4414c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b4a200700 1 -- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b2c004850 msgr2=0x7f1b440b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b4a200700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b2c004850 0x7f1b440b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b4a200700 1 -- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b440b21a0 msgr2=0x7f1b4414c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b4a200700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b440b21a0 0x7f1b4414c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b4a200700 1 -- 172.21.15.12:0/2063651871 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b440b2b60 con 0x7f1b440b84b0 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b49800700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b2c004850 0x7f1b440b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b4a200700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b440b84b0 0x7f1b440b0290 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c000f80 tx=0x7f1b3c007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b41600700 1 -- 172.21.15.12:0/2063651871 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c00eda0 con 0x7f1b440b84b0 2024-03-29T11:12:26.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b41600700 1 -- 172.21.15.12:0/2063651871 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c013420 con 0x7f1b440b84b0 2024-03-29T11:12:26.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b4c200700 1 -- 172.21.15.12:0/2063651871 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b440b2e40 con 0x7f1b440b84b0 2024-03-29T11:12:26.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b41600700 1 -- 172.21.15.12:0/2063651871 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c0137d0 con 0x7f1b440b84b0 2024-03-29T11:12:26.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.778+0000 7f1b4c200700 1 -- 172.21.15.12:0/2063651871 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4414ce60 con 0x7f1b440b84b0 2024-03-29T11:12:26.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.779+0000 7f1b4c200700 1 -- 172.21.15.12:0/2063651871 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b440b3640 con 0x7f1b440b84b0 2024-03-29T11:12:26.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.780+0000 7f1b41600700 1 -- 172.21.15.12:0/2063651871 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1b3c013930 con 0x7f1b440b84b0 2024-03-29T11:12:26.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.781+0000 7f1b41600700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b2406adb0 0x7f1b2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:26.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.781+0000 7f1b41600700 1 -- 172.21.15.12:0/2063651871 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f1b3c089b40 con 0x7f1b440b84b0 2024-03-29T11:12:26.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.782+0000 7f1b49800700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b2406adb0 0x7f1b2406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:26.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.783+0000 7f1b49800700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b2406adb0 0x7f1b2406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009c10 tx=0x7f1b38008040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.786+0000 7f1b41600700 1 -- 172.21.15.12:0/2063651871 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b440b3640 con 0x7f1b440b84b0 2024-03-29T11:12:26.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:26.999+0000 7f1b4c200700 1 -- 172.21.15.12:0/2063651871 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b440b3640 con 0x7f1b440b84b0 2024-03-29T11:12:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.000+0000 7f1b41600700 1 -- 172.21.15.12:0/2063651871 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v190) v1 ==== 74+0+16186 (secure 0 0 0) 0x7f1b440b3640 con 0x7f1b440b84b0 2024-03-29T11:12:26.978 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:26.978 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":190,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:12:26.523498+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:12:24.038574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-03-29T11:12:25.509384+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.002+0000 7f1b4c200700 1 -- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b2406adb0 msgr2=0x7f1b2406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.002+0000 7f1b4c200700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b2406adb0 0x7f1b2406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009c10 tx=0x7f1b38008040 comp rx=0 tx=0).stop 2024-03-29T11:12:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.002+0000 7f1b4c200700 1 -- 172.21.15.12:0/2063651871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b440b84b0 msgr2=0x7f1b440b0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.002+0000 7f1b4c200700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b440b84b0 0x7f1b440b0290 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c000f80 tx=0x7f1b3c007c90 comp rx=0 tx=0).stop 2024-03-29T11:12:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.002+0000 7f1b4c200700 1 -- 172.21.15.12:0/2063651871 shutdown_connections 2024-03-29T11:12:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.002+0000 7f1b4c200700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1b2406adb0 0x7f1b2406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.002+0000 7f1b4c200700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b440b84b0 0x7f1b440b0290 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.002+0000 7f1b4c200700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b2c004850 0x7f1b440b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.002+0000 7f1b4c200700 1 --2- 172.21.15.12:0/2063651871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b440b21a0 0x7f1b4414c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.002+0000 7f1b4c200700 1 -- 172.21.15.12:0/2063651871 >> 172.21.15.12:0/2063651871 conn(0x7f1b4408aff0 msgr2=0x7f1b440cb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.003+0000 7f1b4c200700 1 -- 172.21.15.12:0/2063651871 shutdown_connections 2024-03-29T11:12:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.003+0000 7f1b4c200700 1 -- 172.21.15.12:0/2063651871 wait complete. 2024-03-29T11:12:27.000 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:27.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.219+0000 7fa5a9c00700 1 Processor -- start 2024-03-29T11:12:27.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.219+0000 7fa5a9c00700 1 -- start start 2024-03-29T11:12:27.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.221+0000 7fa5a9c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa5a4160be0 0x7fa5a40cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.221+0000 7fa5a9c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 0x7fa5a40b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.221+0000 7fa5a9c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a4160fc0 con 0x7fa5a40ce0e0 2024-03-29T11:12:27.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.221+0000 7fa5a9c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a40b65d0 con 0x7fa5a4160be0 2024-03-29T11:12:27.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.221+0000 7fa5a9c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a40d2a80 con 0x7fa5a40ceca0 2024-03-29T11:12:27.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.221+0000 7fa5a3400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 0x7fa5a40b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.221+0000 7fa5a3400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 0x7fa5a40b6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:43176/0 (socket says 172.21.15.12:43176) 2024-03-29T11:12:27.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.221+0000 7fa5a2000700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa5a40ce0e0 0x7fa5a40ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:53912/0 (socket says 172.21.15.12:53912) 2024-03-29T11:12:27.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.221+0000 7fa5a2a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa5a4160be0 0x7fa5a40cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.221+0000 7fa5a3400700 1 -- 172.21.15.12:0/2331013036 learned_addr learned my addr 172.21.15.12:0/2331013036 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:27.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.222+0000 7fa5a2a00700 1 -- 172.21.15.12:0/2331013036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 msgr2=0x7fa5a40b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:27.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.222+0000 7fa5a2a00700 1 --2- 172.21.15.12:0/2331013036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 0x7fa5a40b6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.222+0000 7fa5a1600700 1 -- 172.21.15.12:0/2331013036 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699941858 0 0) 0x7fa5a4160fc0 con 0x7fa5a40ce0e0 2024-03-29T11:12:27.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.222+0000 7fa5a2a00700 1 -- 172.21.15.12:0/2331013036 >> v1:172.21.15.159:6789/0 conn(0x7fa5a40ce0e0 legacy=0x7fa5a40ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.222+0000 7fa5a2a00700 1 -- 172.21.15.12:0/2331013036 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a40d21e0 con 0x7fa5a4160be0 2024-03-29T11:12:27.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.222+0000 7fa5a3400700 1 --2- 172.21.15.12:0/2331013036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 0x7fa5a40b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:27.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.222+0000 7fa5a2a00700 1 --2- 172.21.15.12:0/2331013036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a4160be0 0x7fa5a40cdba0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa594009840 tx=0x7fa594009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f4f6e96128f17e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:27.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.222+0000 7fa5a1600700 1 -- 172.21.15.12:0/2331013036 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594011830 con 0x7fa5a4160be0 2024-03-29T11:12:27.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.222+0000 7fa5a1600700 1 -- 172.21.15.12:0/2331013036 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa594011990 con 0x7fa5a4160be0 2024-03-29T11:12:27.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.222+0000 7fa5a1600700 1 -- 172.21.15.12:0/2331013036 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594011d20 con 0x7fa5a4160be0 2024-03-29T11:12:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.223+0000 7fa5a9c00700 1 -- 172.21.15.12:0/2331013036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a4160be0 msgr2=0x7fa5a40cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.223+0000 7fa5a9c00700 1 --2- 172.21.15.12:0/2331013036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a4160be0 0x7fa5a40cdba0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa594009840 tx=0x7fa594009b50 comp rx=0 tx=0).stop 2024-03-29T11:12:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.223+0000 7fa5a9c00700 1 -- 172.21.15.12:0/2331013036 shutdown_connections 2024-03-29T11:12:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.223+0000 7fa5a9c00700 1 --2- 172.21.15.12:0/2331013036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 0x7fa5a40b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.223+0000 7fa5a9c00700 1 --2- 172.21.15.12:0/2331013036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a4160be0 0x7fa5a40cdba0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.223+0000 7fa5a9c00700 1 -- 172.21.15.12:0/2331013036 >> 172.21.15.12:0/2331013036 conn(0x7fa5a408aff0 msgr2=0x7fa5a40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.223+0000 7fa5a9c00700 1 -- 172.21.15.12:0/2331013036 shutdown_connections 2024-03-29T11:12:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.223+0000 7fa5a9c00700 1 -- 172.21.15.12:0/2331013036 wait complete. 2024-03-29T11:12:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.225+0000 7fa5a9c00700 1 Processor -- start 2024-03-29T11:12:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.225+0000 7fa5a9c00700 1 -- start start 2024-03-29T11:12:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a9c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a40ce0e0 0x7fa5a414cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a9c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 0x7fa5a40cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5a4160be0 0x7fa5a40c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a9c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a40c8aa0 con 0x7fa5a4160be0 2024-03-29T11:12:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a9c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a414d320 con 0x7fa5a40ce0e0 2024-03-29T11:12:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a9c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a4158ca0 con 0x7fa5a40ceca0 2024-03-29T11:12:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5a4160be0 0x7fa5a40c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5a4160be0 0x7fa5a40c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37490/0 (socket says 172.21.15.12:37490) 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a3400700 1 -- 172.21.15.12:0/220615370 learned_addr learned my addr 172.21.15.12:0/220615370 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a2000700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 0x7fa5a40cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.226+0000 7fa5a2a00700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a40ce0e0 0x7fa5a414cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa5a3400700 1 -- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 msgr2=0x7fa5a40cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa5a3400700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 0x7fa5a40cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa5a3400700 1 -- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a40ce0e0 msgr2=0x7fa5a414cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa5a3400700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a40ce0e0 0x7fa5a414cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa5a3400700 1 -- 172.21.15.12:0/220615370 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c009b30 con 0x7fa5a4160be0 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa5a3400700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5a4160be0 0x7fa5a40c9360 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00ef90 tx=0x7fa59c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa59aa00700 1 -- 172.21.15.12:0/220615370 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c018070 con 0x7fa5a4160be0 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa59aa00700 1 -- 172.21.15.12:0/220615370 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59c00f040 con 0x7fa5a4160be0 2024-03-29T11:12:27.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa59aa00700 1 -- 172.21.15.12:0/220615370 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c003c60 con 0x7fa5a4160be0 2024-03-29T11:12:27.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa5a9c00700 1 -- 172.21.15.12:0/220615370 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5940094a0 con 0x7fa5a4160be0 2024-03-29T11:12:27.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.227+0000 7fa5a9c00700 1 -- 172.21.15.12:0/220615370 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a40c8200 con 0x7fa5a4160be0 2024-03-29T11:12:27.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.228+0000 7fa5a9c00700 1 -- 172.21.15.12:0/220615370 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a4067720 con 0x7fa5a4160be0 2024-03-29T11:12:27.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.229+0000 7fa59aa00700 1 -- 172.21.15.12:0/220615370 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa59c00c880 con 0x7fa5a4160be0 2024-03-29T11:12:27.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.230+0000 7fa59aa00700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa58406adb0 0x7fa58406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.230+0000 7fa59aa00700 1 -- 172.21.15.12:0/220615370 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fa59c089710 con 0x7fa5a4160be0 2024-03-29T11:12:27.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.231+0000 7fa5a2a00700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa58406adb0 0x7fa58406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.232+0000 7fa5a2a00700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa58406adb0 0x7fa58406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa5940090b0 tx=0x7fa59401c040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.235+0000 7fa59aa00700 1 -- 172.21.15.12:0/220615370 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa59c054690 con 0x7fa5a4160be0 2024-03-29T11:12:27.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.449+0000 7fa5a9c00700 1 -- 172.21.15.12:0/220615370 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5a40b5cb0 con 0x7fa5a4160be0 2024-03-29T11:12:27.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.451+0000 7fa59aa00700 1 -- 172.21.15.12:0/220615370 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v190) v1 ==== 74+0+16186 (secure 0 0 0) 0x7fa59c057cb0 con 0x7fa5a4160be0 2024-03-29T11:12:27.431 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:27.431 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":190,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:12:26.523498+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:12:24.038574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-03-29T11:12:25.509384+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 -- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa58406adb0 msgr2=0x7fa58406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa58406adb0 0x7fa58406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa5940090b0 tx=0x7fa59401c040 comp rx=0 tx=0).stop 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 -- 172.21.15.12:0/220615370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5a4160be0 msgr2=0x7fa5a40c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5a4160be0 0x7fa5a40c9360 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00ef90 tx=0x7fa59c00c3a0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 -- 172.21.15.12:0/220615370 shutdown_connections 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fa58406adb0 0x7fa58406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5a4160be0 0x7fa5a40c9360 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a40ce0e0 0x7fa5a414cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 --2- 172.21.15.12:0/220615370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a40ceca0 0x7fa5a40cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 -- 172.21.15.12:0/220615370 >> 172.21.15.12:0/220615370 conn(0x7fa5a408aff0 msgr2=0x7fa5a40b7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 -- 172.21.15.12:0/220615370 shutdown_connections 2024-03-29T11:12:27.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.453+0000 7fa5a9c00700 1 -- 172.21.15.12:0/220615370 wait complete. 2024-03-29T11:12:27.449 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T11:12:27.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.674+0000 7f3967600700 1 Processor -- start 2024-03-29T11:12:27.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.674+0000 7f3967600700 1 -- start start 2024-03-29T11:12:27.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.675+0000 7f3967600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f396015d800 0x7f39600725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.675+0000 7f3967600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f39600ce230 0x7f39600ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.675+0000 7f3967600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39600b2070 0x7f39600b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.675+0000 7f3967600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39600b40f0 con 0x7f396015d800 2024-03-29T11:12:27.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.675+0000 7f3967600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39600709c0 con 0x7f39600ce230 2024-03-29T11:12:27.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.675+0000 7f3967600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3960070c80 con 0x7f39600b2070 2024-03-29T11:12:27.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f3964c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f396015d800 0x7f39600725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f3964c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f396015d800 0x7f39600725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37494/0 (socket says 172.21.15.12:37494) 2024-03-29T11:12:27.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f3964c00700 1 -- 172.21.15.12:0/3136722623 learned_addr learned my addr 172.21.15.12:0/3136722623 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:27.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f395fe00700 1 --2- 172.21.15.12:0/3136722623 >> v2:172.21.15.159:3300/0 conn(0x7f39600ce230 0x7f39600ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f3965600700 1 --2- 172.21.15.12:0/3136722623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39600b2070 0x7f39600b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f3964c00700 1 -- 172.21.15.12:0/3136722623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39600b2070 msgr2=0x7f39600b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f3964c00700 1 --2- 172.21.15.12:0/3136722623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39600b2070 0x7f39600b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f3964c00700 1 -- 172.21.15.12:0/3136722623 >> v2:172.21.15.159:3300/0 conn(0x7f39600ce230 msgr2=0x7f39600ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f3964c00700 1 --2- 172.21.15.12:0/3136722623 >> v2:172.21.15.159:3300/0 conn(0x7f39600ce230 0x7f39600ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f3964c00700 1 -- 172.21.15.12:0/3136722623 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3960070f60 con 0x7f396015d800 2024-03-29T11:12:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.676+0000 7f3964c00700 1 --2- 172.21.15.12:0/3136722623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f396015d800 0x7f39600725c0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f39600cce70 tx=0x7f3954009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceae272edc69d25 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f395f400700 1 -- 172.21.15.12:0/3136722623 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395400f8a0 con 0x7f396015d800 2024-03-29T11:12:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f395f400700 1 -- 172.21.15.12:0/3136722623 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395400fa00 con 0x7f396015d800 2024-03-29T11:12:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f395f400700 1 -- 172.21.15.12:0/3136722623 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395400fd90 con 0x7f396015d800 2024-03-29T11:12:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f3967600700 1 -- 172.21.15.12:0/3136722623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f396015d800 msgr2=0x7f39600725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f3967600700 1 --2- 172.21.15.12:0/3136722623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f396015d800 0x7f39600725c0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f39600cce70 tx=0x7f3954009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f3967600700 1 -- 172.21.15.12:0/3136722623 shutdown_connections 2024-03-29T11:12:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f3967600700 1 --2- 172.21.15.12:0/3136722623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39600b2070 0x7f39600b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f3967600700 1 --2- 172.21.15.12:0/3136722623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f396015d800 0x7f39600725c0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f3967600700 1 --2- 172.21.15.12:0/3136722623 >> v2:172.21.15.159:3300/0 conn(0x7f39600ce230 0x7f39600ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f3967600700 1 -- 172.21.15.12:0/3136722623 >> 172.21.15.12:0/3136722623 conn(0x7f39600b0d70 msgr2=0x7f39600b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f3967600700 1 -- 172.21.15.12:0/3136722623 shutdown_connections 2024-03-29T11:12:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.677+0000 7f3967600700 1 -- 172.21.15.12:0/3136722623 wait complete. 2024-03-29T11:12:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.679+0000 7f3967600700 1 Processor -- start 2024-03-29T11:12:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.679+0000 7f3967600700 1 -- start start 2024-03-29T11:12:27.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.680+0000 7f3967600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39600b2070 0x7f3960148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.680+0000 7f3967600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39600ce230 0x7f396009cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.680+0000 7f3967600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f396015d800 0x7f396014cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.680+0000 7f3967600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396006e510 con 0x7f39600b2070 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.680+0000 7f3967600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396014c070 con 0x7f39600ce230 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.680+0000 7f3967600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396014ffc0 con 0x7f396015d800 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.680+0000 7f3964c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39600b2070 0x7f3960148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f395fe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39600ce230 0x7f396009cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f395fe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39600ce230 0x7f396009cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:56824/0 (socket says 172.21.15.12:56824) 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f395fe00700 1 -- 172.21.15.12:0/4030195881 learned_addr learned my addr 172.21.15.12:0/4030195881 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f3964c00700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39600b2070 0x7f3960148e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37496/0 (socket says 172.21.15.12:37496) 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f3965600700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f396015d800 0x7f396014cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f3964c00700 1 -- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f396015d800 msgr2=0x7f396014cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f3964c00700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f396015d800 0x7f396014cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f3964c00700 1 -- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39600ce230 msgr2=0x7f396009cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f3964c00700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39600ce230 0x7f396009cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:27.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f3964c00700 1 -- 172.21.15.12:0/4030195881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3954009550 con 0x7f39600b2070 2024-03-29T11:12:27.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f3964c00700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39600b2070 0x7f3960148e20 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f3954018000 tx=0x7f3954004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:27.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.681+0000 7f395cc00700 1 -- 172.21.15.12:0/4030195881 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954004460 con 0x7f39600b2070 2024-03-29T11:12:27.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.682+0000 7f395cc00700 1 -- 172.21.15.12:0/4030195881 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39540045c0 con 0x7f39600b2070 2024-03-29T11:12:27.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.682+0000 7f3967600700 1 -- 172.21.15.12:0/4030195881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39601502a0 con 0x7f39600b2070 2024-03-29T11:12:27.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.682+0000 7f395cc00700 1 -- 172.21.15.12:0/4030195881 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954010b00 con 0x7f39600b2070 2024-03-29T11:12:27.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.682+0000 7f3967600700 1 -- 172.21.15.12:0/4030195881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3960150860 con 0x7f39600b2070 2024-03-29T11:12:27.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.683+0000 7f3967600700 1 -- 172.21.15.12:0/4030195881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3960067720 con 0x7f39600b2070 2024-03-29T11:12:27.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.684+0000 7f395cc00700 1 -- 172.21.15.12:0/4030195881 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f39540036a0 con 0x7f39600b2070 2024-03-29T11:12:27.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.685+0000 7f395cc00700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f394006adb0 0x7f394006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:27.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.685+0000 7f395cc00700 1 -- 172.21.15.12:0/4030195881 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f395408af40 con 0x7f39600b2070 2024-03-29T11:12:27.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.686+0000 7f395fe00700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f394006adb0 0x7f394006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:27.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.686+0000 7f395fe00700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f394006adb0 0x7f394006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f396014fba0 tx=0x7f394c008040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:27.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.690+0000 7f395cc00700 1 -- 172.21.15.12:0/4030195881 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3954055fd0 con 0x7f39600b2070 2024-03-29T11:12:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:27.957+0000 7f3967600700 1 -- 172.21.15.12:0/4030195881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f39600b31b0 con 0x7f39600b2070 2024-03-29T11:12:28.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.542+0000 7f395cc00700 1 -- 172.21.15.12:0/4030195881 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v88) v1 ==== 118+0+0 (secure 0 0 0) 0x7f39540093b0 con 0x7f39600b2070 2024-03-29T11:12:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 -- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f394006adb0 msgr2=0x7f394006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f394006adb0 0x7f394006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f396014fba0 tx=0x7f394c008040 comp rx=0 tx=0).stop 2024-03-29T11:12:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 -- 172.21.15.12:0/4030195881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39600b2070 msgr2=0x7f3960148e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39600b2070 0x7f3960148e20 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f3954018000 tx=0x7f3954004750 comp rx=0 tx=0).stop 2024-03-29T11:12:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 -- 172.21.15.12:0/4030195881 shutdown_connections 2024-03-29T11:12:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f394006adb0 0x7f394006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f396015d800 0x7f396014cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39600b2070 0x7f3960148e20 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 --2- 172.21.15.12:0/4030195881 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39600ce230 0x7f396009cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 -- 172.21.15.12:0/4030195881 >> 172.21.15.12:0/4030195881 conn(0x7f39600b0d70 msgr2=0x7f39600aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 -- 172.21.15.12:0/4030195881 shutdown_connections 2024-03-29T11:12:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.545+0000 7f3967600700 1 -- 172.21.15.12:0/4030195881 wait complete. 2024-03-29T11:12:28.543 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T11:12:28.543 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T11:12:28.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.768+0000 7f3820400700 1 Processor -- start 2024-03-29T11:12:28.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.769+0000 7f3820400700 1 -- start start 2024-03-29T11:12:28.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.770+0000 7f3820400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f38180d21f0 0x7f38180ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:28.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.770+0000 7f3820400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38180ce270 con 0x7f38181611b0 2024-03-29T11:12:28.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.770+0000 7f3820400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381815b600 con 0x7f38180cec40 2024-03-29T11:12:28.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.770+0000 7f3820400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38180d2760 con 0x7f38180d21f0 2024-03-29T11:12:28.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.770+0000 7f381d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f38180d21f0 0x7f38180ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:28.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.770+0000 7f381da00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f38181611b0 0x7f381815ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59666/0 (socket says 172.21.15.12:59666) 2024-03-29T11:12:28.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.770+0000 7f381da00700 1 -- 172.21.15.12:0/139464038 learned_addr learned my addr 172.21.15.12:0/139464038 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.771+0000 7f381d000700 1 -- 172.21.15.12:0/139464038 >> v1:172.21.15.159:6789/0 conn(0x7f38180cec40 legacy=0x7f3818161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.771+0000 7f381d000700 1 -- 172.21.15.12:0/139464038 >> v1:172.21.15.12:6789/0 conn(0x7f38181611b0 legacy=0x7f381815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.771+0000 7f381d000700 1 -- 172.21.15.12:0/139464038 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38180d2a40 con 0x7f38180d21f0 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.771+0000 7f3817e00700 1 -- 172.21.15.12:0/139464038 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967133965 0 0) 0x7f38180ce270 con 0x7f38181611b0 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.771+0000 7f381d000700 1 --2- 172.21.15.12:0/139464038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180d21f0 0x7f38180ce700 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f3804009840 tx=0x7f3804009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8222af195ddbe8e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.771+0000 7f3817e00700 1 -- 172.21.15.12:0/139464038 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3804011830 con 0x7f38180d21f0 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.771+0000 7f3817e00700 1 -- 172.21.15.12:0/139464038 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3804011990 con 0x7f38180d21f0 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.771+0000 7f3817e00700 1 -- 172.21.15.12:0/139464038 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3804011d20 con 0x7f38180d21f0 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.772+0000 7f3820400700 1 -- 172.21.15.12:0/139464038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180d21f0 msgr2=0x7f38180ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.772+0000 7f3820400700 1 --2- 172.21.15.12:0/139464038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180d21f0 0x7f38180ce700 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f3804009840 tx=0x7f3804009b50 comp rx=0 tx=0).stop 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.772+0000 7f3820400700 1 -- 172.21.15.12:0/139464038 shutdown_connections 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.772+0000 7f3820400700 1 --2- 172.21.15.12:0/139464038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180d21f0 0x7f38180ce700 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.772+0000 7f3820400700 1 -- 172.21.15.12:0/139464038 >> 172.21.15.12:0/139464038 conn(0x7f381808aff0 msgr2=0x7f38180d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.772+0000 7f3820400700 1 -- 172.21.15.12:0/139464038 shutdown_connections 2024-03-29T11:12:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.772+0000 7f3820400700 1 -- 172.21.15.12:0/139464038 wait complete. 2024-03-29T11:12:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.774+0000 7f3820400700 1 Processor -- start 2024-03-29T11:12:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.774+0000 7f3820400700 1 -- start start 2024-03-29T11:12:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f3820400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180cec40 0x7f38181e8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f3820400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38180d21f0 0x7f38181e6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f3820400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38181611b0 0x7f38180b7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f3820400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38180cc580 con 0x7f38180cec40 2024-03-29T11:12:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f3820400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38181e9f00 con 0x7f38180d21f0 2024-03-29T11:12:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f3820400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38180cb970 con 0x7f38181611b0 2024-03-29T11:12:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f381da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180cec40 0x7f38181e8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f381da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180cec40 0x7f38181e8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37522/0 (socket says 172.21.15.12:37522) 2024-03-29T11:12:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f381da00700 1 -- 172.21.15.12:0/4114435686 learned_addr learned my addr 172.21.15.12:0/4114435686 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f381d000700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38180d21f0 0x7f38181e6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.775+0000 7f381e400700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38181611b0 0x7f38180b7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f381da00700 1 -- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38181611b0 msgr2=0x7f38180b7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f381da00700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38181611b0 0x7f38180b7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f381da00700 1 -- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38180d21f0 msgr2=0x7f38181e6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f381da00700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38180d21f0 0x7f38181e6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f381da00700 1 -- 172.21.15.12:0/4114435686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38040094a0 con 0x7f38180cec40 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f381d000700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38180d21f0 0x7f38181e6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f381e400700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38181611b0 0x7f38180b7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f381da00700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180cec40 0x7f38181e8f50 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f380c00c520 tx=0x7f380c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f3815600700 1 -- 172.21.15.12:0/4114435686 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c004c80 con 0x7f38180cec40 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f3815600700 1 -- 172.21.15.12:0/4114435686 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380c004e00 con 0x7f38180cec40 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f3820400700 1 -- 172.21.15.12:0/4114435686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38180cbc50 con 0x7f38180cec40 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f3815600700 1 -- 172.21.15.12:0/4114435686 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c007310 con 0x7f38180cec40 2024-03-29T11:12:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.776+0000 7f3820400700 1 -- 172.21.15.12:0/4114435686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38180cc210 con 0x7f38180cec40 2024-03-29T11:12:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.777+0000 7f3820400700 1 -- 172.21.15.12:0/4114435686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3818067720 con 0x7f38180cec40 2024-03-29T11:12:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.778+0000 7f3815600700 1 -- 172.21.15.12:0/4114435686 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f380c007470 con 0x7f38180cec40 2024-03-29T11:12:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.779+0000 7f3815600700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f37f806ade0 0x7f37f806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.779+0000 7f3815600700 1 -- 172.21.15.12:0/4114435686 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f380c089340 con 0x7f38180cec40 2024-03-29T11:12:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.779+0000 7f381d000700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f37f806ade0 0x7f37f806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.780+0000 7f381d000700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f37f806ade0 0x7f37f806d2a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f38180b55f0 tx=0x7f380401c040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:28.783+0000 7f3815600700 1 -- 172.21.15.12:0/4114435686 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f380c0542b0 con 0x7f38180cec40 2024-03-29T11:12:29.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:29.048+0000 7f3820400700 1 -- 172.21.15.12:0/4114435686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f38180b5d00 con 0x7f38180cec40 2024-03-29T11:12:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.015+0000 7f3815600700 1 -- 172.21.15.12:0/4114435686 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v89) v1 ==== 122+0+0 (secure 0 0 0) 0x7f380c0578d0 con 0x7f38180cec40 2024-03-29T11:12:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 -- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f37f806ade0 msgr2=0x7f37f806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f37f806ade0 0x7f37f806d2a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f38180b55f0 tx=0x7f380401c040 comp rx=0 tx=0).stop 2024-03-29T11:12:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 -- 172.21.15.12:0/4114435686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180cec40 msgr2=0x7f38181e8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180cec40 0x7f38181e8f50 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f380c00c520 tx=0x7f380c00c8e0 comp rx=0 tx=0).stop 2024-03-29T11:12:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 -- 172.21.15.12:0/4114435686 shutdown_connections 2024-03-29T11:12:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f37f806ade0 0x7f37f806d2a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38181611b0 0x7f38180b7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38180cec40 0x7f38181e8f50 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 --2- 172.21.15.12:0/4114435686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38180d21f0 0x7f38181e6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 -- 172.21.15.12:0/4114435686 >> 172.21.15.12:0/4114435686 conn(0x7f381808aff0 msgr2=0x7f3818161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 -- 172.21.15.12:0/4114435686 shutdown_connections 2024-03-29T11:12:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.018+0000 7f3820400700 1 -- 172.21.15.12:0/4114435686 wait complete. 2024-03-29T11:12:30.018 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T11:12:30.018 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:12:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.242+0000 7f3f7fa00700 1 Processor -- start 2024-03-29T11:12:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.243+0000 7f3f7fa00700 1 -- start start 2024-03-29T11:12:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f7fa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f7815d640 0x7f3f780d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f7fa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3f780ceec0 0x7f3f781612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f7fa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f780d30d0 con 0x7f3f78161800 2024-03-29T11:12:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f7fa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f78161c90 con 0x7f3f780ceec0 2024-03-29T11:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f7fa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780b6840 con 0x7f3f7815d640 2024-03-29T11:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f77e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3f780ceec0 0x7f3f781612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f77e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3f780ceec0 0x7f3f781612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42624/0 (socket says 172.21.15.12:42624) 2024-03-29T11:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f77e00700 1 -- 172.21.15.12:0/3052838658 learned_addr learned my addr 172.21.15.12:0/3052838658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f7d000700 1 --2- 172.21.15.12:0/3052838658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f7815d640 0x7f3f780d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f77e00700 1 -- 172.21.15.12:0/3052838658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f7815d640 msgr2=0x7f3f780d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f77e00700 1 --2- 172.21.15.12:0/3052838658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f7815d640 0x7f3f780d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f77e00700 1 -- 172.21.15.12:0/3052838658 >> v1:172.21.15.159:6789/0 conn(0x7f3f78161800 legacy=0x7f3f780cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.244+0000 7f3f77e00700 1 -- 172.21.15.12:0/3052838658 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f780d2880 con 0x7f3f780ceec0 2024-03-29T11:12:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.245+0000 7f3f77e00700 1 --2- 172.21.15.12:0/3052838658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f780ceec0 0x7f3f781612c0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f3f640098f0 tx=0x7f3f64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ff295225ea3f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.245+0000 7f3f77400700 1 -- 172.21.15.12:0/3052838658 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6400b8a0 con 0x7f3f780ceec0 2024-03-29T11:12:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.245+0000 7f3f77400700 1 -- 172.21.15.12:0/3052838658 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6400ba00 con 0x7f3f780ceec0 2024-03-29T11:12:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.245+0000 7f3f77400700 1 -- 172.21.15.12:0/3052838658 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6400bd90 con 0x7f3f780ceec0 2024-03-29T11:12:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.245+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3052838658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f780ceec0 msgr2=0x7f3f781612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.245+0000 7f3f7fa00700 1 --2- 172.21.15.12:0/3052838658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f780ceec0 0x7f3f781612c0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f3f640098f0 tx=0x7f3f64009c00 comp rx=0 tx=0).stop 2024-03-29T11:12:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.246+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3052838658 shutdown_connections 2024-03-29T11:12:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.246+0000 7f3f7fa00700 1 --2- 172.21.15.12:0/3052838658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f7815d640 0x7f3f780d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.246+0000 7f3f7fa00700 1 --2- 172.21.15.12:0/3052838658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f780ceec0 0x7f3f781612c0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.246+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3052838658 >> 172.21.15.12:0/3052838658 conn(0x7f3f7808aff0 msgr2=0x7f3f780d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.246+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3052838658 shutdown_connections 2024-03-29T11:12:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.246+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3052838658 wait complete. 2024-03-29T11:12:30.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.247+0000 7f3f7fa00700 1 Processor -- start 2024-03-29T11:12:30.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.248+0000 7f3f7fa00700 1 -- start start 2024-03-29T11:12:30.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.248+0000 7f3f7fa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f780ceec0 0x7f3f781f2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.248+0000 7f3f7fa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f7815d640 0x7f3f780c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.248+0000 7f3f7fa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f78161800 0x7f3f780cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.248+0000 7f3f7fa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7809eba0 con 0x7f3f7815d640 2024-03-29T11:12:30.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.248+0000 7f3f7fa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7809e2e0 con 0x7f3f78161800 2024-03-29T11:12:30.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.248+0000 7f3f7fa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7809e550 con 0x7f3f780ceec0 2024-03-29T11:12:30.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f77e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f7815d640 0x7f3f780c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f77e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f7815d640 0x7f3f780c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42634/0 (socket says 172.21.15.12:42634) 2024-03-29T11:12:30.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f77e00700 1 -- 172.21.15.12:0/3217586770 learned_addr learned my addr 172.21.15.12:0/3217586770 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:30.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f7da00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f78161800 0x7f3f780cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f7d000700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f780ceec0 0x7f3f781f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f77e00700 1 -- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f780ceec0 msgr2=0x7f3f781f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f77e00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f780ceec0 0x7f3f781f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f77e00700 1 -- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f78161800 msgr2=0x7f3f780cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f77e00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f78161800 0x7f3f780cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f77e00700 1 -- 172.21.15.12:0/3217586770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f64009550 con 0x7f3f7815d640 2024-03-29T11:12:30.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.249+0000 7f3f77e00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f7815d640 0x7f3f780c8fc0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f3f64014040 tx=0x7f3f64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:30.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.250+0000 7f3f74c00700 1 -- 172.21.15.12:0/3217586770 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64004510 con 0x7f3f7815d640 2024-03-29T11:12:30.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.250+0000 7f3f74c00700 1 -- 172.21.15.12:0/3217586770 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6401e070 con 0x7f3f7815d640 2024-03-29T11:12:30.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.250+0000 7f3f74c00700 1 -- 172.21.15.12:0/3217586770 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f640038e0 con 0x7f3f7815d640 2024-03-29T11:12:30.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.250+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3217586770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7809e830 con 0x7f3f7815d640 2024-03-29T11:12:30.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.250+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3217586770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f780c9960 con 0x7f3f7815d640 2024-03-29T11:12:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.251+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3217586770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7809ed50 con 0x7f3f7815d640 2024-03-29T11:12:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.252+0000 7f3f74c00700 1 -- 172.21.15.12:0/3217586770 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3f64003a40 con 0x7f3f7815d640 2024-03-29T11:12:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.253+0000 7f3f74c00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f5806adb0 0x7f3f5806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.253+0000 7f3f74c00700 1 -- 172.21.15.12:0/3217586770 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f3f64020070 con 0x7f3f7815d640 2024-03-29T11:12:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.253+0000 7f3f7d000700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f5806adb0 0x7f3f5806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.254+0000 7f3f7d000700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f5806adb0 0x7f3f5806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3f780c9ce0 tx=0x7f3f6c008040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:30.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.257+0000 7f3f74c00700 1 -- 172.21.15.12:0/3217586770 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f7809ed50 con 0x7f3f7815d640 2024-03-29T11:12:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.516+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3217586770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f78067720 con 0x7f3f7815d640 2024-03-29T11:12:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.516+0000 7f3f74c00700 1 -- 172.21.15.12:0/3217586770 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f3f640558e0 con 0x7f3f7815d640 2024-03-29T11:12:30.494 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:30.494 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2592133465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2592133465},{"type":"v1","addr":"172.21.15.159:6837","nonce":2592133465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":8381,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/204651369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":204651369},{"type":"v1","addr":"172.21.15.12:6835","nonce":204651369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9616,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2547917697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2547917697},{"type":"v1","addr":"172.21.15.159:6836","nonce":2547917697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:12:26.523825+0000","modified":"2024-03-29T11:12:30.007709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9975":{"gid":9975,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/4176455926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4176455926},{"type":"v1","addr":"172.21.15.12:6833","nonce":4176455926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-29T11:12:30.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f5806adb0 msgr2=0x7f3f5806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f5806adb0 0x7f3f5806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3f780c9ce0 tx=0x7f3f6c008040 comp rx=0 tx=0).stop 2024-03-29T11:12:30.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3217586770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f7815d640 msgr2=0x7f3f780c8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f7815d640 0x7f3f780c8fc0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f3f64014040 tx=0x7f3f64004750 comp rx=0 tx=0).stop 2024-03-29T11:12:30.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3217586770 shutdown_connections 2024-03-29T11:12:30.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f3f5806adb0 0x7f3f5806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f78161800 0x7f3f780cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f780ceec0 0x7f3f781f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 --2- 172.21.15.12:0/3217586770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f7815d640 0x7f3f780c8fc0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3217586770 >> 172.21.15.12:0/3217586770 conn(0x7f3f7808aff0 msgr2=0x7f3f780d3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:30.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3217586770 shutdown_connections 2024-03-29T11:12:30.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.519+0000 7f3f7fa00700 1 -- 172.21.15.12:0/3217586770 wait complete. 2024-03-29T11:12:30.498 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 89 2024-03-29T11:12:30.517 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.737+0000 7fcce5800700 1 Processor -- start 2024-03-29T11:12:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.737+0000 7fcce5800700 1 -- start start 2024-03-29T11:12:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.738+0000 7fcce5800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce0160f10 0x7fcce00d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.738+0000 7fcce5800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcce0161590 0x7fcce0161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.738+0000 7fcce5800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce00ceda0 con 0x7fcce00cf230 2024-03-29T11:12:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.738+0000 7fcce5800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce0161f80 con 0x7fcce0161590 2024-03-29T11:12:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.738+0000 7fcce5800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce01612f0 con 0x7fcce0160f10 2024-03-29T11:12:30.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.738+0000 7fccdea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce0160f10 0x7fcce00d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.738+0000 7fccdf400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcce0161590 0x7fcce0161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.738+0000 7fccdea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce0160f10 0x7fcce00d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:34930/0 (socket says 172.21.15.12:34930) 2024-03-29T11:12:30.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.738+0000 7fccdea00700 1 -- 172.21.15.12:0/3255636375 learned_addr learned my addr 172.21.15.12:0/3255636375 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:30.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.739+0000 7fccdd600700 1 -- 172.21.15.12:0/3255636375 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591402370 0 0) 0x7fcce00ceda0 con 0x7fcce00cf230 2024-03-29T11:12:30.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.739+0000 7fccdd600700 1 -- 172.21.15.12:0/3255636375 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc40035b0 con 0x7fcce00cf230 2024-03-29T11:12:30.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.739+0000 7fccdea00700 1 -- 172.21.15.12:0/3255636375 >> v2:172.21.15.159:3300/0 conn(0x7fcce0161590 msgr2=0x7fcce0161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:30.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.739+0000 7fccdea00700 1 --2- 172.21.15.12:0/3255636375 >> v2:172.21.15.159:3300/0 conn(0x7fcce0161590 0x7fcce0161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.739+0000 7fccdea00700 1 -- 172.21.15.12:0/3255636375 >> v1:172.21.15.159:6789/0 conn(0x7fcce00cf230 legacy=0x7fcce00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.739+0000 7fccdea00700 1 -- 172.21.15.12:0/3255636375 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce00ce470 con 0x7fcce0160f10 2024-03-29T11:12:30.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.739+0000 7fccdf400700 1 --2- 172.21.15.12:0/3255636375 >> v2:172.21.15.159:3300/0 conn(0x7fcce0161590 0x7fcce0161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:30.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.739+0000 7fccdea00700 1 --2- 172.21.15.12:0/3255636375 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce0160f10 0x7fcce00d18b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009920 tx=0x7fccd0009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f24a503010a0687 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:30.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fccdd600700 1 -- 172.21.15.12:0/3255636375 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd0011830 con 0x7fcce0160f10 2024-03-29T11:12:30.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fccdd600700 1 -- 172.21.15.12:0/3255636375 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd0011990 con 0x7fcce0160f10 2024-03-29T11:12:30.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fccdd600700 1 -- 172.21.15.12:0/3255636375 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd0011d20 con 0x7fcce0160f10 2024-03-29T11:12:30.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fcce5800700 1 -- 172.21.15.12:0/3255636375 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce0160f10 msgr2=0x7fcce00d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fcce5800700 1 --2- 172.21.15.12:0/3255636375 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce0160f10 0x7fcce00d18b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009920 tx=0x7fccd0009c30 comp rx=0 tx=0).stop 2024-03-29T11:12:30.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fcce5800700 1 -- 172.21.15.12:0/3255636375 shutdown_connections 2024-03-29T11:12:30.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fcce5800700 1 --2- 172.21.15.12:0/3255636375 >> v2:172.21.15.159:3300/0 conn(0x7fcce0161590 0x7fcce0161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fcce5800700 1 --2- 172.21.15.12:0/3255636375 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce0160f10 0x7fcce00d18b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fcce5800700 1 -- 172.21.15.12:0/3255636375 >> 172.21.15.12:0/3255636375 conn(0x7fcce008aff0 msgr2=0x7fcce00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:30.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fcce5800700 1 -- 172.21.15.12:0/3255636375 shutdown_connections 2024-03-29T11:12:30.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.740+0000 7fcce5800700 1 -- 172.21.15.12:0/3255636375 wait complete. 2024-03-29T11:12:30.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.742+0000 7fcce5800700 1 Processor -- start 2024-03-29T11:12:30.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.742+0000 7fcce5800700 1 -- start start 2024-03-29T11:12:30.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.743+0000 7fcce5800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce00cf230 0x7fcce0158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.743+0000 7fcce5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce0160f10 0x7fcce015fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.743+0000 7fcce5800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcce0161590 0x7fcce015ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.743+0000 7fcce5800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce009feb0 con 0x7fcce0160f10 2024-03-29T11:12:30.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.743+0000 7fcce5800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00a0190 con 0x7fcce0161590 2024-03-29T11:12:30.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.743+0000 7fcce5800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00a0470 con 0x7fcce00cf230 2024-03-29T11:12:30.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.743+0000 7fccde000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce0160f10 0x7fcce015fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccde000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce0160f10 0x7fcce015fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42648/0 (socket says 172.21.15.12:42648) 2024-03-29T11:12:30.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccde000700 1 -- 172.21.15.12:0/3801231769 learned_addr learned my addr 172.21.15.12:0/3801231769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:30.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccdf400700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcce0161590 0x7fcce015ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccdea00700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce00cf230 0x7fcce0158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccde000700 1 -- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce00cf230 msgr2=0x7fcce0158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccde000700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce00cf230 0x7fcce0158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccde000700 1 -- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcce0161590 msgr2=0x7fcce015ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccde000700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcce0161590 0x7fcce015ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccde000700 1 -- 172.21.15.12:0/3801231769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd8009a70 con 0x7fcce0160f10 2024-03-29T11:12:30.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccde000700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce0160f10 0x7fcce015fa80 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c4d0 tx=0x7fcccc00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.744+0000 7fccd6a00700 1 -- 172.21.15.12:0/3801231769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0052d0 con 0x7fcce0160f10 2024-03-29T11:12:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.745+0000 7fccd6a00700 1 -- 172.21.15.12:0/3801231769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc005430 con 0x7fcce0160f10 2024-03-29T11:12:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.745+0000 7fccd6a00700 1 -- 172.21.15.12:0/3801231769 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc003880 con 0x7fcce0160f10 2024-03-29T11:12:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.745+0000 7fcce5800700 1 -- 172.21.15.12:0/3801231769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd0009550 con 0x7fcce0160f10 2024-03-29T11:12:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.745+0000 7fcce5800700 1 -- 172.21.15.12:0/3801231769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce00a0a30 con 0x7fcce0160f10 2024-03-29T11:12:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.746+0000 7fcce5800700 1 -- 172.21.15.12:0/3801231769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce00d3070 con 0x7fcce0160f10 2024-03-29T11:12:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.747+0000 7fccd6a00700 1 -- 172.21.15.12:0/3801231769 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcccc0039e0 con 0x7fcce0160f10 2024-03-29T11:12:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.747+0000 7fccd6a00700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fccc006adb0 0x7fccc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.748+0000 7fccd6a00700 1 -- 172.21.15.12:0/3801231769 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fcccc088750 con 0x7fcce0160f10 2024-03-29T11:12:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.748+0000 7fccdea00700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fccc006adb0 0x7fccc006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.749+0000 7fccdea00700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fccc006adb0 0x7fccc006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fccd0000c00 tx=0x7fccd0008030 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.752+0000 7fccd6a00700 1 -- 172.21.15.12:0/3801231769 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcccc053800 con 0x7fcce0160f10 2024-03-29T11:12:30.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.970+0000 7fcce5800700 1 -- 172.21.15.12:0/3801231769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcce0067720 con 0x7fcce0160f10 2024-03-29T11:12:30.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.971+0000 7fccd6a00700 1 -- 172.21.15.12:0/3801231769 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) v1 ==== 74+0+16173 (secure 0 0 0) 0x7fcccc056e20 con 0x7fcce0160f10 2024-03-29T11:12:30.950 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:30.950 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":191,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:12:27.526460+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:12:24.038574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-03-29T11:12:25.509384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:30.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 -- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fccc006adb0 msgr2=0x7fccc006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fccc006adb0 0x7fccc006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fccd0000c00 tx=0x7fccd0008030 comp rx=0 tx=0).stop 2024-03-29T11:12:30.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 -- 172.21.15.12:0/3801231769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce0160f10 msgr2=0x7fcce015fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:30.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce0160f10 0x7fcce015fa80 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c4d0 tx=0x7fcccc00c7e0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 -- 172.21.15.12:0/3801231769 shutdown_connections 2024-03-29T11:12:30.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fccc006adb0 0x7fccc006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcce0161590 0x7fcce015ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcce00cf230 0x7fcce0158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 --2- 172.21.15.12:0/3801231769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce0160f10 0x7fcce015fa80 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:30.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 -- 172.21.15.12:0/3801231769 >> 172.21.15.12:0/3801231769 conn(0x7fcce008aff0 msgr2=0x7fcce00ca8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:30.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 -- 172.21.15.12:0/3801231769 shutdown_connections 2024-03-29T11:12:30.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:30.974+0000 7fcce5800700 1 -- 172.21.15.12:0/3801231769 wait complete. 2024-03-29T11:12:30.970 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T11:12:30.970 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:12:31.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.191+0000 7f8e52c00700 1 Processor -- start 2024-03-29T11:12:31.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.192+0000 7f8e52c00700 1 -- start start 2024-03-29T11:12:31.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e52c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8e4c15d640 0x7f8e4c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e52c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8e4c0ceec0 0x7f8e4c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e52c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0d30d0 con 0x7f8e4c161800 2024-03-29T11:12:31.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e52c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4c0b6840 con 0x7f8e4c0ceec0 2024-03-29T11:12:31.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e52c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4c161c20 con 0x7f8e4c15d640 2024-03-29T11:12:31.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e50c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8e4c161800 0x7f8e4c0cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57052/0 (socket says 172.21.15.12:57052) 2024-03-29T11:12:31.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e50c00700 1 -- 172.21.15.12:0/3652073167 learned_addr learned my addr 172.21.15.12:0/3652073167 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:31.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e4b400700 1 --2- 172.21.15.12:0/3652073167 >> v2:172.21.15.12:3300/0 conn(0x7f8e4c0ceec0 0x7f8e4c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e4be00700 1 --2- 172.21.15.12:0/3652073167 >> v2:172.21.15.159:3300/0 conn(0x7f8e4c15d640 0x7f8e4c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e4aa00700 1 -- 172.21.15.12:0/3652073167 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376383333 0 0) 0x7f8e4c0d30d0 con 0x7f8e4c161800 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e4aa00700 1 -- 172.21.15.12:0/3652073167 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e300035b0 con 0x7f8e4c161800 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e4b400700 1 -- 172.21.15.12:0/3652073167 >> v2:172.21.15.159:3300/0 conn(0x7f8e4c15d640 msgr2=0x7f8e4c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e4b400700 1 --2- 172.21.15.12:0/3652073167 >> v2:172.21.15.159:3300/0 conn(0x7f8e4c15d640 0x7f8e4c0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e4b400700 1 -- 172.21.15.12:0/3652073167 >> v1:172.21.15.12:6789/0 conn(0x7f8e4c161800 legacy=0x7f8e4c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.193+0000 7f8e4b400700 1 -- 172.21.15.12:0/3652073167 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0d2880 con 0x7f8e4c0ceec0 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.194+0000 7f8e4b400700 1 --2- 172.21.15.12:0/3652073167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c0ceec0 0x7f8e4c1612c0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0b5c20 tx=0x7f8e38009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea8806eb6706bc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.194+0000 7f8e4aa00700 1 -- 172.21.15.12:0/3652073167 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38011830 con 0x7f8e4c0ceec0 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.194+0000 7f8e4aa00700 1 -- 172.21.15.12:0/3652073167 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e38011990 con 0x7f8e4c0ceec0 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.194+0000 7f8e4aa00700 1 -- 172.21.15.12:0/3652073167 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38011d20 con 0x7f8e4c0ceec0 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.194+0000 7f8e52c00700 1 -- 172.21.15.12:0/3652073167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c0ceec0 msgr2=0x7f8e4c1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.194+0000 7f8e52c00700 1 --2- 172.21.15.12:0/3652073167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c0ceec0 0x7f8e4c1612c0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0b5c20 tx=0x7f8e38009b20 comp rx=0 tx=0).stop 2024-03-29T11:12:31.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.195+0000 7f8e52c00700 1 -- 172.21.15.12:0/3652073167 shutdown_connections 2024-03-29T11:12:31.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.195+0000 7f8e52c00700 1 --2- 172.21.15.12:0/3652073167 >> v2:172.21.15.159:3300/0 conn(0x7f8e4c15d640 0x7f8e4c0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.195+0000 7f8e52c00700 1 --2- 172.21.15.12:0/3652073167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c0ceec0 0x7f8e4c1612c0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.195+0000 7f8e52c00700 1 -- 172.21.15.12:0/3652073167 >> 172.21.15.12:0/3652073167 conn(0x7f8e4c08aff0 msgr2=0x7f8e4c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:31.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.195+0000 7f8e52c00700 1 -- 172.21.15.12:0/3652073167 shutdown_connections 2024-03-29T11:12:31.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.195+0000 7f8e52c00700 1 -- 172.21.15.12:0/3652073167 wait complete. 2024-03-29T11:12:31.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.196+0000 7f8e52c00700 1 Processor -- start 2024-03-29T11:12:31.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.197+0000 7f8e52c00700 1 -- start start 2024-03-29T11:12:31.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.197+0000 7f8e52c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e4c0ceec0 0x7f8e4c0a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.197+0000 7f8e52c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e4c15d640 0x7f8e4c1f3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.197+0000 7f8e52c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c161800 0x7f8e4c06ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.197+0000 7f8e52c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c0a0a70 con 0x7f8e4c161800 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.197+0000 7f8e52c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c09e9c0 con 0x7f8e4c0ceec0 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.197+0000 7f8e52c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c09fdf0 con 0x7f8e4c15d640 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e50c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c161800 0x7f8e4c06ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e50c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c161800 0x7f8e4c06ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42660/0 (socket says 172.21.15.12:42660) 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e50c00700 1 -- 172.21.15.12:0/1544495678 learned_addr learned my addr 172.21.15.12:0/1544495678 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e4be00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e4c0ceec0 0x7f8e4c0a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e4b400700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e4c15d640 0x7f8e4c1f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e50c00700 1 -- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e4c15d640 msgr2=0x7f8e4c1f3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e50c00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e4c15d640 0x7f8e4c1f3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e50c00700 1 -- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e4c0ceec0 msgr2=0x7f8e4c0a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e50c00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e4c0ceec0 0x7f8e4c0a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e50c00700 1 -- 172.21.15.12:0/1544495678 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e380094a0 con 0x7f8e4c161800 2024-03-29T11:12:31.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.198+0000 7f8e50c00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c161800 0x7f8e4c06ca40 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400cb40 tx=0x7f8e4400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:31.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.199+0000 7f8e43e00700 1 -- 172.21.15.12:0/1544495678 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4400ab50 con 0x7f8e4c161800 2024-03-29T11:12:31.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.199+0000 7f8e43e00700 1 -- 172.21.15.12:0/1544495678 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4400acb0 con 0x7f8e4c161800 2024-03-29T11:12:31.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.199+0000 7f8e52c00700 1 -- 172.21.15.12:0/1544495678 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4c0a00d0 con 0x7f8e4c161800 2024-03-29T11:12:31.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.199+0000 7f8e43e00700 1 -- 172.21.15.12:0/1544495678 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44012620 con 0x7f8e4c161800 2024-03-29T11:12:31.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.199+0000 7f8e52c00700 1 -- 172.21.15.12:0/1544495678 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4c0a0690 con 0x7f8e4c161800 2024-03-29T11:12:31.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.200+0000 7f8e52c00700 1 -- 172.21.15.12:0/1544495678 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4c0a0bb0 con 0x7f8e4c161800 2024-03-29T11:12:31.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.201+0000 7f8e43e00700 1 -- 172.21.15.12:0/1544495678 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8e4401b3b0 con 0x7f8e4c161800 2024-03-29T11:12:31.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.202+0000 7f8e43e00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e2c06adb0 0x7f8e2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.202+0000 7f8e43e00700 1 -- 172.21.15.12:0/1544495678 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f8e440892e0 con 0x7f8e4c161800 2024-03-29T11:12:31.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.202+0000 7f8e4be00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e2c06adb0 0x7f8e2c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.203+0000 7f8e4be00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e2c06adb0 0x7f8e2c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009930 tx=0x7f8e3c008040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:31.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.206+0000 7f8e43e00700 1 -- 172.21.15.12:0/1544495678 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8e4c0a0bb0 con 0x7f8e4c161800 2024-03-29T11:12:31.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.414+0000 7f8e52c00700 1 -- 172.21.15.12:0/1544495678 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8e4c0b64d0 con 0x7f8e2c06adb0 2024-03-29T11:12:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.425+0000 7f8e43e00700 1 -- 172.21.15.12:0/1544495678 <== mgr.8216 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278454 (secure 0 0 0) 0x7f8e3c009e00 con 0x7f8e2c06adb0 2024-03-29T11:12:31.407 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.429+0000 7f8e52c00700 1 -- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e2c06adb0 msgr2=0x7f8e2c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.429+0000 7f8e52c00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e2c06adb0 0x7f8e2c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009930 tx=0x7f8e3c008040 comp rx=0 tx=0).stop 2024-03-29T11:12:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.429+0000 7f8e52c00700 1 -- 172.21.15.12:0/1544495678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c161800 msgr2=0x7f8e4c06ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.429+0000 7f8e52c00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c161800 0x7f8e4c06ca40 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400cb40 tx=0x7f8e4400cf00 comp rx=0 tx=0).stop 2024-03-29T11:12:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.429+0000 7f8e52c00700 1 -- 172.21.15.12:0/1544495678 shutdown_connections 2024-03-29T11:12:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.429+0000 7f8e52c00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f8e2c06adb0 0x7f8e2c06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.429+0000 7f8e52c00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e4c161800 0x7f8e4c06ca40 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.429+0000 7f8e52c00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e4c0ceec0 0x7f8e4c0a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.429+0000 7f8e52c00700 1 --2- 172.21.15.12:0/1544495678 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e4c15d640 0x7f8e4c1f3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.429+0000 7f8e52c00700 1 -- 172.21.15.12:0/1544495678 >> 172.21.15.12:0/1544495678 conn(0x7f8e4c08aff0 msgr2=0x7f8e4c0b8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:31.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.430+0000 7f8e52c00700 1 -- 172.21.15.12:0/1544495678 shutdown_connections 2024-03-29T11:12:31.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.430+0000 7f8e52c00700 1 -- 172.21.15.12:0/1544495678 wait complete. 2024-03-29T11:12:31.411 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:12:31.426 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-29T11:12:29.778456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":511,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":311,"ondisk_log_size":311,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":409508,"kb_used_data":19500,"kb_used_omap":12,"kb_used_meta":389875,"kb_avail":754565212,"statfs":{"total":773094113280,"available":772674777088,"internally_reserved":0,"allocated":19968000,"data_stored":10244067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":399232569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.781884"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.003463+0000","last_change":"2024-03-29T11:12:25.003463+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.003463+0000","last_undegraded":"2024-03-29T11:12:25.003463+0000","last_fullsized":"2024-03-29T11:12:25.003463+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:08:46.259447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.002873+0000","last_change":"2024-03-29T11:12:25.002873+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.002873+0000","last_undegraded":"2024-03-29T11:12:25.002873+0000","last_fullsized":"2024-03-29T11:12:25.002873+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:29:47.416031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.003621+0000","last_change":"2024-03-29T11:12:25.003621+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.003621+0000","last_undegraded":"2024-03-29T11:12:25.003621+0000","last_fullsized":"2024-03-29T11:12:25.003621+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:01:24.473402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.004402+0000","last_change":"2024-03-29T11:12:25.004402+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.004402+0000","last_undegraded":"2024-03-29T11:12:25.004402+0000","last_fullsized":"2024-03-29T11:12:25.004402+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:04:21.670707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:24.998840+0000","last_change":"2024-03-29T11:12:24.998840+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:24.998840+0000","last_undegraded":"2024-03-29T11:12:24.998840+0000","last_fullsized":"2024-03-29T11:12:24.998840+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:31:54.820703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.003063+0000","last_change":"2024-03-29T11:12:25.003063+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.003063+0000","last_undegraded":"2024-03-29T11:12:25.003063+0000","last_fullsized":"2024-03-29T11:12:25.003063+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:58:16.283140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:24.999958+0000","last_change":"2024-03-29T11:12:24.999958+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:24.999958+0000","last_undegraded":"2024-03-29T11:12:24.999958+0000","last_fullsized":"2024-03-29T11:12:24.999958+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:18:59.006987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.005910+0000","last_change":"2024-03-29T11:12:25.005910+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.005910+0000","last_undegraded":"2024-03-29T11:12:25.005910+0000","last_fullsized":"2024-03-29T11:12:25.005910+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:43.500449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:24.998339+0000","last_change":"2024-03-29T11:12:24.998339+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:24.998339+0000","last_undegraded":"2024-03-29T11:12:24.998339+0000","last_fullsized":"2024-03-29T11:12:24.998339+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:56:16.709598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:24.998330+0000","last_change":"2024-03-29T11:12:24.998330+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:24.998330+0000","last_undegraded":"2024-03-29T11:12:24.998330+0000","last_fullsized":"2024-03-29T11:12:24.998330+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:43.500449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.002818+0000","last_change":"2024-03-29T11:12:25.002818+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.002818+0000","last_undegraded":"2024-03-29T11:12:25.002818+0000","last_fullsized":"2024-03-29T11:12:25.002818+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:10:15.529111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.004417+0000","last_change":"2024-03-29T11:12:25.004417+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.004417+0000","last_undegraded":"2024-03-29T11:12:25.004417+0000","last_fullsized":"2024-03-29T11:12:25.004417+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:06:46.777846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.001804+0000","last_change":"2024-03-29T11:12:25.001804+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.001804+0000","last_undegraded":"2024-03-29T11:12:25.001804+0000","last_fullsized":"2024-03-29T11:12:25.001804+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:48.259813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.004178+0000","last_change":"2024-03-29T11:12:25.004178+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.004178+0000","last_undegraded":"2024-03-29T11:12:25.004178+0000","last_fullsized":"2024-03-29T11:12:25.004178+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:18:11.751956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.004417+0000","last_change":"2024-03-29T11:12:25.004417+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.004417+0000","last_undegraded":"2024-03-29T11:12:25.004417+0000","last_fullsized":"2024-03-29T11:12:25.004417+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:43.500449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:24.999021+0000","last_change":"2024-03-29T11:12:24.999021+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:24.999021+0000","last_undegraded":"2024-03-29T11:12:24.999021+0000","last_fullsized":"2024-03-29T11:12:24.999021+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:23:33.363505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:24.998919+0000","last_change":"2024-03-29T11:12:24.998919+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:24.998919+0000","last_undegraded":"2024-03-29T11:12:24.998919+0000","last_fullsized":"2024-03-29T11:12:24.998919+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.539453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.002025+0000","last_change":"2024-03-29T11:12:25.002025+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.002025+0000","last_undegraded":"2024-03-29T11:12:25.002025+0000","last_fullsized":"2024-03-29T11:12:25.002025+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:22.939828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.002031+0000","last_change":"2024-03-29T11:12:25.002031+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.002031+0000","last_undegraded":"2024-03-29T11:12:25.002031+0000","last_fullsized":"2024-03-29T11:12:25.002031+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:39:02.038526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.004932+0000","last_change":"2024-03-29T11:12:25.004932+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.004932+0000","last_undegraded":"2024-03-29T11:12:25.004932+0000","last_fullsized":"2024-03-29T11:12:25.004932+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:50:40.386291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.002018+0000","last_change":"2024-03-29T11:12:25.002018+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.002018+0000","last_undegraded":"2024-03-29T11:12:25.002018+0000","last_fullsized":"2024-03-29T11:12:25.002018+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:38.551929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.002544+0000","last_change":"2024-03-29T11:12:25.002544+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.002544+0000","last_undegraded":"2024-03-29T11:12:25.002544+0000","last_fullsized":"2024-03-29T11:12:25.002544+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:21:21.076484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.002340+0000","last_change":"2024-03-29T11:12:25.002340+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.002340+0000","last_undegraded":"2024-03-29T11:12:25.002340+0000","last_fullsized":"2024-03-29T11:12:25.002340+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:34.078365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.001797+0000","last_change":"2024-03-29T11:12:25.001797+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.001797+0000","last_undegraded":"2024-03-29T11:12:25.001797+0000","last_fullsized":"2024-03-29T11:12:25.001797+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:17.048108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:24.999654+0000","last_change":"2024-03-29T11:12:24.999654+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:24.999654+0000","last_undegraded":"2024-03-29T11:12:24.999654+0000","last_fullsized":"2024-03-29T11:12:24.999654+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:57.014805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:24.998742+0000","last_change":"2024-03-29T11:12:24.998742+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:24.998742+0000","last_undegraded":"2024-03-29T11:12:24.998742+0000","last_fullsized":"2024-03-29T11:12:24.998742+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:21:17.353455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.001692+0000","last_change":"2024-03-29T11:12:25.001692+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.001692+0000","last_undegraded":"2024-03-29T11:12:25.001692+0000","last_fullsized":"2024-03-29T11:12:25.001692+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:22:03.070734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.003047+0000","last_change":"2024-03-29T11:12:25.003047+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.003047+0000","last_undegraded":"2024-03-29T11:12:25.003047+0000","last_fullsized":"2024-03-29T11:12:25.003047+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:44.789023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.003843+0000","last_change":"2024-03-29T11:12:25.003843+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.003843+0000","last_undegraded":"2024-03-29T11:12:25.003843+0000","last_fullsized":"2024-03-29T11:12:25.003843+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:54.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:24.999381+0000","last_change":"2024-03-29T11:12:24.999381+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:24.999381+0000","last_undegraded":"2024-03-29T11:12:24.999381+0000","last_fullsized":"2024-03-29T11:12:24.999381+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:59:43.776320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.002835+0000","last_change":"2024-03-29T11:12:25.002835+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.002835+0000","last_undegraded":"2024-03-29T11:12:25.002835+0000","last_fullsized":"2024-03-29T11:12:25.002835+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:50:38.495397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-29T11:12:25.995407+0000","last_change":"2024-03-29T11:12:25.995407+0000","last_active":"2024-03-29T11:12:25.995407+0000","last_peered":"2024-03-29T11:12:25.995407+0000","last_clean":"2024-03-29T11:12:25.995407+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.995407+0000","last_undegraded":"2024-03-29T11:12:25.995407+0000","last_fullsized":"2024-03-29T11:12:25.995407+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1,"reported_epoch":188,"state":"creating+peering","last_fresh":"2024-03-29T11:12:25.004277+0000","last_change":"2024-03-29T11:12:25.004277+0000","last_active":"2024-03-29T11:12:24.992047+0000","last_peered":"2024-03-29T11:12:24.992047+0000","last_clean":"2024-03-29T11:12:24.992047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-29T11:12:25.004277+0000","last_undegraded":"2024-03-29T11:12:25.004277+0000","last_fullsized":"2024-03-29T11:12:25.004277+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:56:16.709598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"180'311","reported_seq":1031,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-29T11:12:21.895801+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:12:21.895801+0000","last_peered":"2024-03-29T11:12:21.895801+0000","last_clean":"2024-03-29T11:12:21.895801+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:12:21.895801+0000","last_undegraded":"2024-03-29T11:12:21.895801+0000","last_fullsized":"2024-03-29T11:12:21.895801+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":511,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":511,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":311,"ondisk_log_size":311,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575770,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47944,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323896,"statfs":{"total":96636764160,"available":96587669504,"internally_reserved":0,"allocated":2945024,"data_stored":1724877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":46135756},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.71899999999999997,"15min":0.72399999999999998},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.40000000000000002},"max":{"1min":0.93999999999999995,"5min":1.0089999999999999,"15min":1.014},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.67200000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":1.008},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.746,"15min":0.76600000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.081},"last":0.48499999999999999}]},{"osd":2,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.73199999999999998,"15min":0.749},"min":{"1min":0.502,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.92900000000000005,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.76300000000000001,"15min":0.77000000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.2310000000000001},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.74299999999999999,"15min":0.75700000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.42999999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.091},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77800000000000002,"15min":0.78200000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0049999999999999,"5min":1.028,"15min":1.028},"last":0.58099999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63900000000000001,"15min":0.628},"min":{"1min":0.45600000000000002,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.80500000000000005,"5min":0.875,"15min":1.1499999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.52300000000000002,"5min":0.46300000000000002,"15min":0.42999999999999999},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.208},"last":0.50600000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67600000000000005,"15min":0.66100000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.39100000000000001},"max":{"1min":0.82899999999999996,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.628},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71399999999999997,"15min":0.70499999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.41799999999999998},"max":{"1min":0.90900000000000003,"5min":1.0580000000000001,"15min":1.1020000000000001},"last":0.70199999999999996}]},{"osd":6,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.56799999999999995,"5min":0.47999999999999998,"15min":0.47499999999999998},"max":{"1min":1,"5min":1.008,"15min":1.101},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.749,"15min":0.72599999999999998},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.68600000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575771,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41604,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330236,"statfs":{"total":96636764160,"available":96594161664,"internally_reserved":0,"allocated":2351104,"data_stored":1134509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.70999999999999996,"15min":0.72899999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.38900000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.628},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.752,"15min":0.751},"min":{"1min":0.55800000000000005,"5min":0.50800000000000001,"15min":0.36799999999999999},"max":{"1min":0.999,"5min":1.012,"15min":1.135},"last":0.58899999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.40799999999999997},"max":{"1min":0.89000000000000001,"5min":0.94599999999999995,"15min":1.01},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.76400000000000001,"15min":0.77000000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.439},"max":{"1min":1.016,"5min":1.016,"15min":1.04},"last":0.68799999999999994}]},{"osd":2,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73599999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.51300000000000001,"15min":0.48699999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98099999999999998,"5min":1.0029999999999999,"15min":1.0720000000000001},"last":0.77900000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.77900000000000003,"15min":0.77400000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.71699999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61099999999999999,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76200000000000001,"5min":0.89000000000000001,"15min":0.93899999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70199999999999996,"15min":0.69099999999999995},"min":{"1min":0.54100000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95099999999999996,"5min":0.98799999999999999,"15min":1.0469999999999999},"last":0.79500000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66000000000000003,"15min":0.65600000000000003},"min":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.435},"max":{"1min":0.996,"5min":1.0049999999999999,"15min":1.125},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.70799999999999996,"15min":0.68700000000000006},"min":{"1min":0.432,"5min":0.432,"15min":0.40400000000000003},"max":{"1min":0.98299999999999998,"5min":1.0629999999999999,"15min":1.0980000000000001},"last":0.752}]},{"osd":7,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.75800000000000001,"15min":0.752},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.52800000000000002},"max":{"1min":0.95699999999999996,"5min":0.97999999999999998,"15min":1.077},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73299999999999998,"15min":0.72399999999999998},"min":{"1min":0.57199999999999995,"5min":0.53300000000000003,"15min":0.46200000000000002},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.61099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59644,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312196,"statfs":{"total":96636764160,"available":96575688704,"internally_reserved":0,"allocated":2338816,"data_stored":1128648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.63200000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.91000000000000003},"last":0.58599999999999997}]},{"osd":2,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":0.79200000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.372,"15min":0.34399999999999997},"max":{"1min":0.70999999999999996,"5min":0.88100000000000001,"15min":0.91800000000000004},"last":0.55500000000000005}]},{"osd":3,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.622,"15min":0.622},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76200000000000001,"5min":0.81599999999999995,"15min":0.97599999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.42199999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.752,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.65300000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81000000000000005,"15min":0.79900000000000004},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.53800000000000003},"max":{"1min":1.139,"5min":1.139,"15min":1.1950000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.65400000000000003,"5min":0.625,"15min":0.57699999999999996},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.71199999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.83799999999999997,"15min":0.83199999999999996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.85899999999999999,"15min":0.84299999999999997},"min":{"1min":0.68799999999999994,"5min":0.60699999999999998,"15min":0.59099999999999997},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.57299999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.84699999999999998},"min":{"1min":0.63,"5min":0.63,"15min":0.61099999999999999},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.879,"15min":0.86299999999999999},"min":{"1min":0.67200000000000004,"5min":0.63200000000000001,"15min":0.56799999999999995},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.70399999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.89300000000000002,"15min":0.88100000000000001},"min":{"1min":0.79200000000000004,"5min":0.627,"15min":0.56000000000000005},"max":{"1min":1.1419999999999999,"5min":1.1459999999999999,"15min":1.1819999999999999},"last":0.745},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.89300000000000002,"15min":0.88},"min":{"1min":0.71699999999999997,"5min":0.57499999999999996,"15min":0.55000000000000004},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.66900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60228,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311612,"statfs":{"total":96636764160,"available":96575090688,"internally_reserved":0,"allocated":2940928,"data_stored":1723858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.47199999999999998,"15min":0.46200000000000002},"min":{"1min":0.32300000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89800000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.51900000000000002,"15min":0.53400000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70099999999999996,"5min":0.73999999999999999,"15min":1.03},"last":0.56699999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.36799999999999999},"max":{"1min":0.751,"5min":0.80100000000000005,"15min":1.1459999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68799999999999994,"5min":0.81599999999999995,"15min":0.88700000000000001},"last":0.53600000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.36699999999999999},"max":{"1min":0.79000000000000004,"5min":0.85199999999999998,"15min":0.85899999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.433},"max":{"1min":0.80800000000000005,"5min":0.873,"15min":1.226},"last":0.496}]},{"osd":4,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82199999999999995,"15min":0.82699999999999996},"min":{"1min":0.629,"5min":0.56200000000000006,"15min":0.48899999999999999},"max":{"1min":1.123,"5min":1.304,"15min":1.304},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.83999999999999997,"15min":0.85799999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.50700000000000001},"max":{"1min":1.0820000000000001,"5min":1.1100000000000001,"15min":1.2809999999999999},"last":0.77200000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.80500000000000005,"15min":0.79300000000000004},"min":{"1min":0.54100000000000004,"5min":0.53400000000000003,"15min":0.48399999999999999},"max":{"1min":0.98599999999999999,"5min":1.127,"15min":1.129},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.84099999999999997,"15min":0.83599999999999997},"min":{"1min":0.60899999999999999,"5min":0.56100000000000005,"15min":0.54000000000000004},"max":{"1min":0.99299999999999999,"5min":1.27,"15min":1.27},"last":0.78000000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.84099999999999997,"15min":0.85499999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.018,"5min":1.1679999999999999,"15min":1.329},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.88600000000000001,"15min":0.88900000000000001},"min":{"1min":0.65000000000000002,"5min":0.61699999999999999,"15min":0.58099999999999996},"max":{"1min":1.0589999999999999,"5min":1.337,"15min":1.337},"last":0.90700000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.82999999999999996,"15min":0.83199999999999996},"min":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.53400000000000003},"max":{"1min":1.0149999999999999,"5min":1.161,"15min":1.181},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.56499999999999995},"max":{"1min":1.159,"5min":1.1890000000000001,"15min":1.298},"last":0.746}]}]},{"osd":2,"up_from":13,"seq":55834575770,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47368,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324472,"statfs":{"total":96636764160,"available":96588259328,"internally_reserved":0,"allocated":2351104,"data_stored":1134509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46600000000000003,"15min":0.47399999999999998},"min":{"1min":0.29999999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.77700000000000002},"last":0.443},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.53500000000000003,"15min":0.54300000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67600000000000005,"5min":0.80000000000000004,"15min":0.878},"last":0.33400000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.63,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46700000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.44400000000000001,"5min":0.436,"15min":0.41499999999999998},"max":{"1min":0.65600000000000003,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58199999999999996},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.80800000000000005,"15min":0.90800000000000003},"last":0.496}]},{"osd":4,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.79300000000000004,"15min":0.81399999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55200000000000005},"max":{"1min":0.95199999999999996,"5min":1.0640000000000001,"15min":1.135},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.81200000000000006,"15min":0.81799999999999995},"min":{"1min":0.61499999999999999,"5min":0.60699999999999998,"15min":0.56699999999999995},"max":{"1min":0.99299999999999999,"5min":1.1419999999999999,"15min":1.3680000000000001},"last":0.67100000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.5},"max":{"1min":1.0580000000000001,"5min":1.2350000000000001,"15min":1.2709999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.81699999999999995,"15min":0.82799999999999996},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.56299999999999994},"max":{"1min":1.022,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.55800000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.83699999999999997,"15min":0.84599999999999997},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.042,"5min":1.321,"15min":1.343},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.84199999999999997,"15min":0.85199999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1,"5min":1.321,"15min":1.321},"last":0.60999999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.90200000000000002,"15min":0.90100000000000002},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.58199999999999996},"max":{"1min":1.089,"5min":1.2729999999999999,"15min":1.403},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.91300000000000003,"15min":0.91100000000000003},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.58699999999999997},"max":{"1min":1.099,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.69999999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575770,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59656,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312184,"statfs":{"total":96636764160,"available":96575676416,"internally_reserved":0,"allocated":2338816,"data_stored":1128648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48199999999999998,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55600000000000005,"15min":0.55100000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.31},"max":{"1min":0.80000000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.501,"15min":0.503},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.72499999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.40799999999999997,"15min":0.38900000000000001},"max":{"1min":0.77800000000000002,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.88},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.66600000000000004,"15min":0.66200000000000003},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.436},"max":{"1min":0.89400000000000002,"5min":0.91900000000000004,"15min":0.98799999999999999},"last":0.62}]},{"osd":4,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.79300000000000004,"15min":0.80500000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.97399999999999998,"5min":1.2629999999999999,"15min":1.5469999999999999},"last":0.755},{"interface":"front","average":{"1min":0.748,"5min":0.80600000000000005,"15min":0.80800000000000005},"min":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.51300000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.5309999999999999},"last":0.88900000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.80800000000000005,"15min":0.82199999999999995},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.52400000000000002},"max":{"1min":1.0780000000000001,"5min":1.113,"15min":1.4199999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.83299999999999996,"15min":0.83899999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.53200000000000003},"max":{"1min":1.175,"5min":1.3140000000000001,"15min":1.3740000000000001},"last":0.79700000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.84599999999999997,"15min":0.85499999999999998},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.56399999999999995},"max":{"1min":1.117,"5min":1.1339999999999999,"15min":1.248},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84699999999999998,"15min":0.85299999999999998},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.113,"5min":1.212,"15min":1.2849999999999999},"last":0.83399999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.86899999999999999,"15min":0.872},"min":{"1min":0.67100000000000004,"5min":0.64300000000000002,"15min":0.58199999999999996},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.6020000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.89500000000000002,"15min":0.89300000000000002},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.157,"5min":1.2350000000000001,"15min":1.3340000000000001},"last":0.77300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575771,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51460,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320380,"statfs":{"total":96636764160,"available":96584069120,"internally_reserved":0,"allocated":2351104,"data_stored":1134509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.72599999999999998,"15min":0.72799999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.33300000000000002},"max":{"1min":0.85699999999999998,"5min":1.0209999999999999,"15min":1.097},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.69999999999999996,"15min":0.72199999999999998},"min":{"1min":0.45700000000000002,"5min":0.44400000000000001,"15min":0.40899999999999997},"max":{"1min":0.84499999999999997,"5min":0.93899999999999995,"15min":1.3300000000000001},"last":0.503}]},{"osd":1,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.74299999999999999,"15min":0.747},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.78400000000000003,"5min":1.105,"15min":1.105},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.71599999999999997,"15min":0.73299999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86799999999999999,"5min":0.94599999999999995,"15min":1.0209999999999999},"last":0.67700000000000005}]},{"osd":2,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.79800000000000004,"15min":0.81799999999999995},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.437},"max":{"1min":0.90400000000000003,"5min":1.034,"15min":1.3080000000000001},"last":0.623},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.79300000000000004,"15min":0.80300000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.46800000000000003},"max":{"1min":0.86699999999999999,"5min":1.0349999999999999,"15min":1.1140000000000001},"last":0.64500000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.754,"15min":0.755},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.53700000000000003},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":1.177},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.74199999999999999,"15min":0.751},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1319999999999999,"5min":1.137,"15min":1.137},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.91300000000000003,"5min":0.93100000000000005,"15min":1.054},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61499999999999999,"15min":0.61799999999999999},"min":{"1min":0.46300000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.88600000000000001,"5min":0.98199999999999998,"15min":1.1779999999999999},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63700000000000001,"15min":0.63200000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.94899999999999995,"5min":1.038,"15min":1.0389999999999999},"last":0.751},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67700000000000005,"15min":0.67300000000000004},"min":{"1min":0.45900000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":1.254},"last":0.73399999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.70499999999999996,"15min":0.69499999999999995},"min":{"1min":0.57099999999999995,"5min":0.47899999999999998,"15min":0.44900000000000001},"max":{"1min":0.98599999999999999,"5min":1.004,"15min":1.4379999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.52400000000000002},"max":{"1min":1.01,"5min":1.0580000000000001,"15min":1.482},"last":0.78900000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575770,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41604,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330236,"statfs":{"total":96636764160,"available":96594161664,"internally_reserved":0,"allocated":2351104,"data_stored":1134509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.72699999999999998,"15min":0.72399999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.40699999999999997},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71599999999999997,"15min":0.74199999999999999},"min":{"1min":0.58299999999999996,"5min":0.48099999999999998,"15min":0.45000000000000001},"max":{"1min":0.84499999999999997,"5min":0.94799999999999995,"15min":1},"last":0.69099999999999995}]},{"osd":1,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.73399999999999999,"15min":0.752},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90500000000000003,"5min":0.997,"15min":1.052},"last":0.76100000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.749,"15min":0.76200000000000001},"min":{"1min":0.503,"5min":0.499,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.016,"15min":1.0609999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73199999999999998,"15min":0.745},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.55900000000000005}]},{"osd":3,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.747,"15min":0.76700000000000002},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.52300000000000002},"max":{"1min":0.93000000000000005,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.745},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.78800000000000003},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.66600000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.38800000000000001,"15min":0.379},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.5},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64700000000000002,"15min":0.64300000000000002},"min":{"1min":0.501,"5min":0.46300000000000002,"15min":0.41199999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98299999999999998},"last":0.54100000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.5,"5min":0.48699999999999999,"15min":0.439},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.66600000000000004,"15min":0.67100000000000004},"min":{"1min":0.502,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.67700000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.71299999999999997,"15min":0.71099999999999997},"min":{"1min":0.55800000000000005,"5min":0.46800000000000003,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71899999999999997,"15min":0.73699999999999999},"min":{"1min":0.59599999999999997,"5min":0.52500000000000002,"15min":0.49399999999999999},"max":{"1min":0.93100000000000005,"5min":0.95399999999999996,"15min":1.0369999999999999},"last":0.80900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:12:31.431 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:12:31.499 INFO:tasks.ceph.mds.c.smithi012.stderr:2024-03-29T11:12:31.518+0000 7f57fe400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:12:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.687+0000 7f67da200700 1 Processor -- start 2024-03-29T11:12:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.687+0000 7f67da200700 1 -- start start 2024-03-29T11:12:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.688+0000 7f67da200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f67d415d710 0x7f67d40d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.688+0000 7f67da200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d406bb40 con 0x7f67d406c650 2024-03-29T11:12:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.688+0000 7f67da200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d4099b00 con 0x7f67d409aba0 2024-03-29T11:12:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.688+0000 7f67da200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d4099de0 con 0x7f67d415d710 2024-03-29T11:12:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.689+0000 7f67d3e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f67d406c650 0x7f67d40ccbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57066/0 (socket says 172.21.15.12:57066) 2024-03-29T11:12:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.689+0000 7f67d3e00700 1 -- 172.21.15.12:0/2896067115 learned_addr learned my addr 172.21.15.12:0/2896067115 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.689+0000 7f67d3400700 1 --2- 172.21.15.12:0/2896067115 >> v2:172.21.15.159:3300/0 conn(0x7f67d415d710 0x7f67d40d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.689+0000 7f67d2000700 1 -- 172.21.15.12:0/2896067115 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101450220 0 0) 0x7f67d406bb40 con 0x7f67d406c650 2024-03-29T11:12:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.689+0000 7f67d2000700 1 -- 172.21.15.12:0/2896067115 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b80035b0 con 0x7f67d406c650 2024-03-29T11:12:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.689+0000 7f67d3400700 1 -- 172.21.15.12:0/2896067115 >> v1:172.21.15.159:6789/0 conn(0x7f67d409aba0 legacy=0x7f67d406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.689+0000 7f67d3400700 1 -- 172.21.15.12:0/2896067115 >> v1:172.21.15.12:6789/0 conn(0x7f67d406c650 legacy=0x7f67d40ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.689+0000 7f67d3400700 1 -- 172.21.15.12:0/2896067115 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d409a0c0 con 0x7f67d415d710 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.690+0000 7f67d3400700 1 --2- 172.21.15.12:0/2896067115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d415d710 0x7f67d40d0130 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009840 tx=0x7f67c4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51565f77b5d31912 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.690+0000 7f67d2000700 1 -- 172.21.15.12:0/2896067115 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4011830 con 0x7f67d415d710 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.690+0000 7f67d2000700 1 -- 172.21.15.12:0/2896067115 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c4011990 con 0x7f67d415d710 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.690+0000 7f67d2000700 1 -- 172.21.15.12:0/2896067115 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4011d00 con 0x7f67d415d710 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.690+0000 7f67da200700 1 -- 172.21.15.12:0/2896067115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d415d710 msgr2=0x7f67d40d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.690+0000 7f67da200700 1 --2- 172.21.15.12:0/2896067115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d415d710 0x7f67d40d0130 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009840 tx=0x7f67c4009b50 comp rx=0 tx=0).stop 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.690+0000 7f67da200700 1 -- 172.21.15.12:0/2896067115 shutdown_connections 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.690+0000 7f67da200700 1 --2- 172.21.15.12:0/2896067115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d415d710 0x7f67d40d0130 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.690+0000 7f67da200700 1 -- 172.21.15.12:0/2896067115 >> 172.21.15.12:0/2896067115 conn(0x7f67d40d13e0 msgr2=0x7f67d40d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.691+0000 7f67da200700 1 -- 172.21.15.12:0/2896067115 shutdown_connections 2024-03-29T11:12:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.691+0000 7f67da200700 1 -- 172.21.15.12:0/2896067115 wait complete. 2024-03-29T11:12:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.692+0000 7f67da200700 1 Processor -- start 2024-03-29T11:12:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.692+0000 7f67da200700 1 -- start start 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.693+0000 7f67da200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d406c650 0x7f67d40b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.693+0000 7f67da200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d409aba0 0x7f67d40c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.693+0000 7f67da200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d415d710 0x7f67d40b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.693+0000 7f67da200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d40ca300 con 0x7f67d406c650 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.693+0000 7f67da200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d415a230 con 0x7f67d415d710 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.693+0000 7f67da200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d40c7e50 con 0x7f67d409aba0 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d406c650 0x7f67d40b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d406c650 0x7f67d40b3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42668/0 (socket says 172.21.15.12:42668) 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3400700 1 -- 172.21.15.12:0/4234756424 learned_addr learned my addr 172.21.15.12:0/4234756424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3e00700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d415d710 0x7f67d40b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d2a00700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d409aba0 0x7f67d40c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3400700 1 -- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d409aba0 msgr2=0x7f67d40c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3400700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d409aba0 0x7f67d40c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3400700 1 -- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d415d710 msgr2=0x7f67d40b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3400700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d415d710 0x7f67d40b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3400700 1 -- 172.21.15.12:0/4234756424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c40094a0 con 0x7f67d406c650 2024-03-29T11:12:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3e00700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d415d710 0x7f67d40b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.694+0000 7f67d3400700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d406c650 0x7f67d40b3fd0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f67c4000c00 tx=0x7f67c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.695+0000 7f67cb400700 1 -- 172.21.15.12:0/4234756424 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c40036a0 con 0x7f67d406c650 2024-03-29T11:12:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.695+0000 7f67cb400700 1 -- 172.21.15.12:0/4234756424 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c401e070 con 0x7f67d406c650 2024-03-29T11:12:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.695+0000 7f67cb400700 1 -- 172.21.15.12:0/4234756424 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4003a20 con 0x7f67d406c650 2024-03-29T11:12:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.695+0000 7f67da200700 1 -- 172.21.15.12:0/4234756424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d40c9680 con 0x7f67d406c650 2024-03-29T11:12:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.695+0000 7f67da200700 1 -- 172.21.15.12:0/4234756424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d40c9c40 con 0x7f67d406c650 2024-03-29T11:12:31.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.696+0000 7f67da200700 1 -- 172.21.15.12:0/4234756424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d406d460 con 0x7f67d406c650 2024-03-29T11:12:31.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.697+0000 7f67cb400700 1 -- 172.21.15.12:0/4234756424 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f67c4003b80 con 0x7f67d406c650 2024-03-29T11:12:31.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.697+0000 7f67cb400700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b406adb0 0x7f67b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:31.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.698+0000 7f67cb400700 1 -- 172.21.15.12:0/4234756424 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f67c408a4e0 con 0x7f67d406c650 2024-03-29T11:12:31.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.698+0000 7f67d2a00700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b406adb0 0x7f67b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:31.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.699+0000 7f67d2a00700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b406adb0 0x7f67b406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f67d41611d0 tx=0x7f67c0009040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:31.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.702+0000 7f67cb400700 1 -- 172.21.15.12:0/4234756424 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67c4055570 con 0x7f67d406c650 2024-03-29T11:12:31.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.911+0000 7f67da200700 1 -- 172.21.15.12:0/4234756424 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67d409a680 con 0x7f67b406adb0 2024-03-29T11:12:31.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.921+0000 7f67cb400700 1 -- 172.21.15.12:0/4234756424 <== mgr.8216 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289524 (secure 0 0 0) 0x7f67c00050d0 con 0x7f67b406adb0 2024-03-29T11:12:31.903 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:31.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 -- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b406adb0 msgr2=0x7f67b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b406adb0 0x7f67b406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f67d41611d0 tx=0x7f67c0009040 comp rx=0 tx=0).stop 2024-03-29T11:12:31.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 -- 172.21.15.12:0/4234756424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d406c650 msgr2=0x7f67d40b3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:31.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d406c650 0x7f67d40b3fd0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f67c4000c00 tx=0x7f67c4004750 comp rx=0 tx=0).stop 2024-03-29T11:12:31.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 -- 172.21.15.12:0/4234756424 shutdown_connections 2024-03-29T11:12:31.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f67b406adb0 0x7f67b406d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67d415d710 0x7f67d40b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67d406c650 0x7f67d40b3fd0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 --2- 172.21.15.12:0/4234756424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67d409aba0 0x7f67d40c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:31.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 -- 172.21.15.12:0/4234756424 >> 172.21.15.12:0/4234756424 conn(0x7f67d40d13e0 msgr2=0x7f67d40cfd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:31.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 -- 172.21.15.12:0/4234756424 shutdown_connections 2024-03-29T11:12:31.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:31.925+0000 7f67da200700 1 -- 172.21.15.12:0/4234756424 wait complete. 2024-03-29T11:12:31.906 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:12:31.924 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-29T11:12:31.778893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":556,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":350,"ondisk_log_size":350,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413976,"kb_used_data":19876,"kb_used_omap":12,"kb_used_meta":393971,"kb_avail":754560744,"statfs":{"total":773094113280,"available":772670201856,"internally_reserved":0,"allocated":20353024,"data_stored":10416369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":403426873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001350"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.697649+0000","last_change":"2024-03-29T11:12:26.559935+0000","last_active":"2024-03-29T11:12:27.697649+0000","last_peered":"2024-03-29T11:12:27.697649+0000","last_clean":"2024-03-29T11:12:27.697649+0000","last_became_active":"2024-03-29T11:12:26.559089+0000","last_became_peered":"2024-03-29T11:12:26.559089+0000","last_unstale":"2024-03-29T11:12:27.697649+0000","last_undegraded":"2024-03-29T11:12:27.697649+0000","last_fullsized":"2024-03-29T11:12:27.697649+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:06:12.438960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.542430+0000","last_change":"2024-03-29T11:12:26.545820+0000","last_active":"2024-03-29T11:12:27.542430+0000","last_peered":"2024-03-29T11:12:27.542430+0000","last_clean":"2024-03-29T11:12:27.542430+0000","last_became_active":"2024-03-29T11:12:26.545501+0000","last_became_peered":"2024-03-29T11:12:26.545501+0000","last_unstale":"2024-03-29T11:12:27.542430+0000","last_undegraded":"2024-03-29T11:12:27.542430+0000","last_fullsized":"2024-03-29T11:12:27.542430+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:54.248899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540694+0000","last_change":"2024-03-29T11:12:26.559351+0000","last_active":"2024-03-29T11:12:27.540694+0000","last_peered":"2024-03-29T11:12:27.540694+0000","last_clean":"2024-03-29T11:12:27.540694+0000","last_became_active":"2024-03-29T11:12:26.556940+0000","last_became_peered":"2024-03-29T11:12:26.556940+0000","last_unstale":"2024-03-29T11:12:27.540694+0000","last_undegraded":"2024-03-29T11:12:27.540694+0000","last_fullsized":"2024-03-29T11:12:27.540694+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:42:10.216327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.536551+0000","last_change":"2024-03-29T11:12:26.561965+0000","last_active":"2024-03-29T11:12:27.536551+0000","last_peered":"2024-03-29T11:12:27.536551+0000","last_clean":"2024-03-29T11:12:27.536551+0000","last_became_active":"2024-03-29T11:12:26.559893+0000","last_became_peered":"2024-03-29T11:12:26.559893+0000","last_unstale":"2024-03-29T11:12:27.536551+0000","last_undegraded":"2024-03-29T11:12:27.536551+0000","last_fullsized":"2024-03-29T11:12:27.536551+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:58.018165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.542100+0000","last_change":"2024-03-29T11:12:26.547664+0000","last_active":"2024-03-29T11:12:27.542100+0000","last_peered":"2024-03-29T11:12:27.542100+0000","last_clean":"2024-03-29T11:12:27.542100+0000","last_became_active":"2024-03-29T11:12:26.545461+0000","last_became_peered":"2024-03-29T11:12:26.545461+0000","last_unstale":"2024-03-29T11:12:27.542100+0000","last_undegraded":"2024-03-29T11:12:27.542100+0000","last_fullsized":"2024-03-29T11:12:27.542100+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:01.186607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511834+0000","last_change":"2024-03-29T11:12:26.551982+0000","last_active":"2024-03-29T11:12:29.511834+0000","last_peered":"2024-03-29T11:12:29.511834+0000","last_clean":"2024-03-29T11:12:29.511834+0000","last_became_active":"2024-03-29T11:12:26.550274+0000","last_became_peered":"2024-03-29T11:12:26.550274+0000","last_unstale":"2024-03-29T11:12:29.511834+0000","last_undegraded":"2024-03-29T11:12:29.511834+0000","last_fullsized":"2024-03-29T11:12:29.511834+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:04.837609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.702385+0000","last_change":"2024-03-29T11:12:26.548203+0000","last_active":"2024-03-29T11:12:27.702385+0000","last_peered":"2024-03-29T11:12:27.702385+0000","last_clean":"2024-03-29T11:12:27.702385+0000","last_became_active":"2024-03-29T11:12:26.547707+0000","last_became_peered":"2024-03-29T11:12:26.547707+0000","last_unstale":"2024-03-29T11:12:27.702385+0000","last_undegraded":"2024-03-29T11:12:27.702385+0000","last_fullsized":"2024-03-29T11:12:27.702385+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:43.290223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.542070+0000","last_change":"2024-03-29T11:12:26.541122+0000","last_active":"2024-03-29T11:12:27.542070+0000","last_peered":"2024-03-29T11:12:27.542070+0000","last_clean":"2024-03-29T11:12:27.542070+0000","last_became_active":"2024-03-29T11:12:26.539965+0000","last_became_peered":"2024-03-29T11:12:26.539965+0000","last_unstale":"2024-03-29T11:12:27.542070+0000","last_undegraded":"2024-03-29T11:12:27.542070+0000","last_fullsized":"2024-03-29T11:12:27.542070+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:36.177398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.542396+0000","last_change":"2024-03-29T11:12:26.547919+0000","last_active":"2024-03-29T11:12:27.542396+0000","last_peered":"2024-03-29T11:12:27.542396+0000","last_clean":"2024-03-29T11:12:27.542396+0000","last_became_active":"2024-03-29T11:12:26.547318+0000","last_became_peered":"2024-03-29T11:12:26.547318+0000","last_unstale":"2024-03-29T11:12:27.542396+0000","last_undegraded":"2024-03-29T11:12:27.542396+0000","last_fullsized":"2024-03-29T11:12:27.542396+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:58:27.138713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.542465+0000","last_change":"2024-03-29T11:12:26.556373+0000","last_active":"2024-03-29T11:12:27.542465+0000","last_peered":"2024-03-29T11:12:27.542465+0000","last_clean":"2024-03-29T11:12:27.542465+0000","last_became_active":"2024-03-29T11:12:26.555370+0000","last_became_peered":"2024-03-29T11:12:26.555370+0000","last_unstale":"2024-03-29T11:12:27.542465+0000","last_undegraded":"2024-03-29T11:12:27.542465+0000","last_fullsized":"2024-03-29T11:12:27.542465+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:45:07.990370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.542342+0000","last_change":"2024-03-29T11:12:26.551140+0000","last_active":"2024-03-29T11:12:27.542342+0000","last_peered":"2024-03-29T11:12:27.542342+0000","last_clean":"2024-03-29T11:12:27.542342+0000","last_became_active":"2024-03-29T11:12:26.550827+0000","last_became_peered":"2024-03-29T11:12:26.550827+0000","last_unstale":"2024-03-29T11:12:27.542342+0000","last_undegraded":"2024-03-29T11:12:27.542342+0000","last_fullsized":"2024-03-29T11:12:27.542342+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:13.137676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.542260+0000","last_change":"2024-03-29T11:12:26.542763+0000","last_active":"2024-03-29T11:12:27.542260+0000","last_peered":"2024-03-29T11:12:27.542260+0000","last_clean":"2024-03-29T11:12:27.542260+0000","last_became_active":"2024-03-29T11:12:26.542394+0000","last_became_peered":"2024-03-29T11:12:26.542394+0000","last_unstale":"2024-03-29T11:12:27.542260+0000","last_undegraded":"2024-03-29T11:12:27.542260+0000","last_fullsized":"2024-03-29T11:12:27.542260+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:42:10.216327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540466+0000","last_change":"2024-03-29T11:12:26.550042+0000","last_active":"2024-03-29T11:12:27.540466+0000","last_peered":"2024-03-29T11:12:27.540466+0000","last_clean":"2024-03-29T11:12:27.540466+0000","last_became_active":"2024-03-29T11:12:26.549116+0000","last_became_peered":"2024-03-29T11:12:26.549116+0000","last_unstale":"2024-03-29T11:12:27.540466+0000","last_undegraded":"2024-03-29T11:12:27.540466+0000","last_fullsized":"2024-03-29T11:12:27.540466+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:13.137676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540290+0000","last_change":"2024-03-29T11:12:26.548492+0000","last_active":"2024-03-29T11:12:27.540290+0000","last_peered":"2024-03-29T11:12:27.540290+0000","last_clean":"2024-03-29T11:12:27.540290+0000","last_became_active":"2024-03-29T11:12:26.544382+0000","last_became_peered":"2024-03-29T11:12:26.544382+0000","last_unstale":"2024-03-29T11:12:27.540290+0000","last_undegraded":"2024-03-29T11:12:27.540290+0000","last_fullsized":"2024-03-29T11:12:27.540290+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:10:21.521863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.697282+0000","last_change":"2024-03-29T11:12:26.554491+0000","last_active":"2024-03-29T11:12:27.697282+0000","last_peered":"2024-03-29T11:12:27.697282+0000","last_clean":"2024-03-29T11:12:27.697282+0000","last_became_active":"2024-03-29T11:12:26.554181+0000","last_became_peered":"2024-03-29T11:12:26.554181+0000","last_unstale":"2024-03-29T11:12:27.697282+0000","last_undegraded":"2024-03-29T11:12:27.697282+0000","last_fullsized":"2024-03-29T11:12:27.697282+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:39.406651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.702270+0000","last_change":"2024-03-29T11:12:26.547908+0000","last_active":"2024-03-29T11:12:27.702270+0000","last_peered":"2024-03-29T11:12:27.702270+0000","last_clean":"2024-03-29T11:12:27.702270+0000","last_became_active":"2024-03-29T11:12:26.546342+0000","last_became_peered":"2024-03-29T11:12:26.546342+0000","last_unstale":"2024-03-29T11:12:27.702270+0000","last_undegraded":"2024-03-29T11:12:27.702270+0000","last_fullsized":"2024-03-29T11:12:27.702270+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:49:23.621862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511910+0000","last_change":"2024-03-29T11:12:26.543454+0000","last_active":"2024-03-29T11:12:29.511910+0000","last_peered":"2024-03-29T11:12:29.511910+0000","last_clean":"2024-03-29T11:12:29.511910+0000","last_became_active":"2024-03-29T11:12:26.541765+0000","last_became_peered":"2024-03-29T11:12:26.541765+0000","last_unstale":"2024-03-29T11:12:29.511910+0000","last_undegraded":"2024-03-29T11:12:29.511910+0000","last_fullsized":"2024-03-29T11:12:29.511910+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:52:01.366224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511895+0000","last_change":"2024-03-29T11:12:26.550482+0000","last_active":"2024-03-29T11:12:29.511895+0000","last_peered":"2024-03-29T11:12:29.511895+0000","last_clean":"2024-03-29T11:12:29.511895+0000","last_became_active":"2024-03-29T11:12:26.550226+0000","last_became_peered":"2024-03-29T11:12:26.550226+0000","last_unstale":"2024-03-29T11:12:29.511895+0000","last_undegraded":"2024-03-29T11:12:29.511895+0000","last_fullsized":"2024-03-29T11:12:29.511895+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:35.081725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511709+0000","last_change":"2024-03-29T11:12:26.557758+0000","last_active":"2024-03-29T11:12:29.511709+0000","last_peered":"2024-03-29T11:12:29.511709+0000","last_clean":"2024-03-29T11:12:29.511709+0000","last_became_active":"2024-03-29T11:12:26.557291+0000","last_became_peered":"2024-03-29T11:12:26.557291+0000","last_unstale":"2024-03-29T11:12:29.511709+0000","last_undegraded":"2024-03-29T11:12:29.511709+0000","last_fullsized":"2024-03-29T11:12:29.511709+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:43:45.321870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540559+0000","last_change":"2024-03-29T11:12:26.552697+0000","last_active":"2024-03-29T11:12:27.540559+0000","last_peered":"2024-03-29T11:12:27.540559+0000","last_clean":"2024-03-29T11:12:27.540559+0000","last_became_active":"2024-03-29T11:12:26.552416+0000","last_became_peered":"2024-03-29T11:12:26.552416+0000","last_unstale":"2024-03-29T11:12:27.540559+0000","last_undegraded":"2024-03-29T11:12:27.540559+0000","last_fullsized":"2024-03-29T11:12:27.540559+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:43:45.321870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511704+0000","last_change":"2024-03-29T11:12:26.548551+0000","last_active":"2024-03-29T11:12:29.511704+0000","last_peered":"2024-03-29T11:12:29.511704+0000","last_clean":"2024-03-29T11:12:29.511704+0000","last_became_active":"2024-03-29T11:12:26.545636+0000","last_became_peered":"2024-03-29T11:12:26.545636+0000","last_unstale":"2024-03-29T11:12:29.511704+0000","last_undegraded":"2024-03-29T11:12:29.511704+0000","last_fullsized":"2024-03-29T11:12:29.511704+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:18:37.644980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.542101+0000","last_change":"2024-03-29T11:12:26.543130+0000","last_active":"2024-03-29T11:12:27.542101+0000","last_peered":"2024-03-29T11:12:27.542101+0000","last_clean":"2024-03-29T11:12:27.542101+0000","last_became_active":"2024-03-29T11:12:26.542858+0000","last_became_peered":"2024-03-29T11:12:26.542858+0000","last_unstale":"2024-03-29T11:12:27.542101+0000","last_undegraded":"2024-03-29T11:12:27.542101+0000","last_fullsized":"2024-03-29T11:12:27.542101+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:10:21.521863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511487+0000","last_change":"2024-03-29T11:12:26.547654+0000","last_active":"2024-03-29T11:12:29.511487+0000","last_peered":"2024-03-29T11:12:29.511487+0000","last_clean":"2024-03-29T11:12:29.511487+0000","last_became_active":"2024-03-29T11:12:26.547283+0000","last_became_peered":"2024-03-29T11:12:26.547283+0000","last_unstale":"2024-03-29T11:12:29.511487+0000","last_undegraded":"2024-03-29T11:12:29.511487+0000","last_fullsized":"2024-03-29T11:12:29.511487+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:16.747337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.542133+0000","last_change":"2024-03-29T11:12:26.542880+0000","last_active":"2024-03-29T11:12:27.542133+0000","last_peered":"2024-03-29T11:12:27.542133+0000","last_clean":"2024-03-29T11:12:27.542133+0000","last_became_active":"2024-03-29T11:12:26.541882+0000","last_became_peered":"2024-03-29T11:12:26.541882+0000","last_unstale":"2024-03-29T11:12:27.542133+0000","last_undegraded":"2024-03-29T11:12:27.542133+0000","last_fullsized":"2024-03-29T11:12:27.542133+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:24:35.794902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.541026+0000","last_change":"2024-03-29T11:12:26.548819+0000","last_active":"2024-03-29T11:12:27.541026+0000","last_peered":"2024-03-29T11:12:27.541026+0000","last_clean":"2024-03-29T11:12:27.541026+0000","last_became_active":"2024-03-29T11:12:26.548451+0000","last_became_peered":"2024-03-29T11:12:26.548451+0000","last_unstale":"2024-03-29T11:12:27.541026+0000","last_undegraded":"2024-03-29T11:12:27.541026+0000","last_fullsized":"2024-03-29T11:12:27.541026+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:45:07.990370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.702688+0000","last_change":"2024-03-29T11:12:26.559007+0000","last_active":"2024-03-29T11:12:27.702688+0000","last_peered":"2024-03-29T11:12:27.702688+0000","last_clean":"2024-03-29T11:12:27.702688+0000","last_became_active":"2024-03-29T11:12:26.558749+0000","last_became_peered":"2024-03-29T11:12:26.558749+0000","last_unstale":"2024-03-29T11:12:27.702688+0000","last_undegraded":"2024-03-29T11:12:27.702688+0000","last_fullsized":"2024-03-29T11:12:27.702688+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:03:09.737683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540892+0000","last_change":"2024-03-29T11:12:26.558806+0000","last_active":"2024-03-29T11:12:27.540892+0000","last_peered":"2024-03-29T11:12:27.540892+0000","last_clean":"2024-03-29T11:12:27.540892+0000","last_became_active":"2024-03-29T11:12:26.557885+0000","last_became_peered":"2024-03-29T11:12:26.557885+0000","last_unstale":"2024-03-29T11:12:27.540892+0000","last_undegraded":"2024-03-29T11:12:27.540892+0000","last_fullsized":"2024-03-29T11:12:27.540892+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:24:35.794902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.536648+0000","last_change":"2024-03-29T11:12:26.549180+0000","last_active":"2024-03-29T11:12:27.536648+0000","last_peered":"2024-03-29T11:12:27.536648+0000","last_clean":"2024-03-29T11:12:27.536648+0000","last_became_active":"2024-03-29T11:12:26.548878+0000","last_became_peered":"2024-03-29T11:12:26.548878+0000","last_unstale":"2024-03-29T11:12:27.536648+0000","last_undegraded":"2024-03-29T11:12:27.536648+0000","last_fullsized":"2024-03-29T11:12:27.536648+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:23:34.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540577+0000","last_change":"2024-03-29T11:12:26.558434+0000","last_active":"2024-03-29T11:12:27.540577+0000","last_peered":"2024-03-29T11:12:27.540577+0000","last_clean":"2024-03-29T11:12:27.540577+0000","last_became_active":"2024-03-29T11:12:26.556832+0000","last_became_peered":"2024-03-29T11:12:26.556832+0000","last_unstale":"2024-03-29T11:12:27.540577+0000","last_undegraded":"2024-03-29T11:12:27.540577+0000","last_fullsized":"2024-03-29T11:12:27.540577+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:50:39.498757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511322+0000","last_change":"2024-03-29T11:12:26.543662+0000","last_active":"2024-03-29T11:12:29.511322+0000","last_peered":"2024-03-29T11:12:29.511322+0000","last_clean":"2024-03-29T11:12:29.511322+0000","last_became_active":"2024-03-29T11:12:26.543390+0000","last_became_peered":"2024-03-29T11:12:26.543390+0000","last_unstale":"2024-03-29T11:12:29.511322+0000","last_undegraded":"2024-03-29T11:12:29.511322+0000","last_fullsized":"2024-03-29T11:12:29.511322+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:52:01.366224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511349+0000","last_change":"2024-03-29T11:12:26.546314+0000","last_active":"2024-03-29T11:12:29.511349+0000","last_peered":"2024-03-29T11:12:29.511349+0000","last_clean":"2024-03-29T11:12:29.511349+0000","last_became_active":"2024-03-29T11:12:26.546010+0000","last_became_peered":"2024-03-29T11:12:26.546010+0000","last_unstale":"2024-03-29T11:12:29.511349+0000","last_undegraded":"2024-03-29T11:12:29.511349+0000","last_fullsized":"2024-03-29T11:12:29.511349+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:18:37.644980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511249+0000","last_change":"2024-03-29T11:12:26.546495+0000","last_active":"2024-03-29T11:12:29.511249+0000","last_peered":"2024-03-29T11:12:29.511249+0000","last_clean":"2024-03-29T11:12:29.511249+0000","last_became_active":"2024-03-29T11:12:26.546247+0000","last_became_peered":"2024-03-29T11:12:26.546247+0000","last_unstale":"2024-03-29T11:12:29.511249+0000","last_undegraded":"2024-03-29T11:12:29.511249+0000","last_fullsized":"2024-03-29T11:12:29.511249+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:50:36.800920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.702796+0000","last_change":"2024-03-29T11:12:26.558241+0000","last_active":"2024-03-29T11:12:27.702796+0000","last_peered":"2024-03-29T11:12:27.702796+0000","last_clean":"2024-03-29T11:12:27.702796+0000","last_became_active":"2024-03-29T11:12:26.558000+0000","last_became_peered":"2024-03-29T11:12:26.558000+0000","last_unstale":"2024-03-29T11:12:27.702796+0000","last_undegraded":"2024-03-29T11:12:27.702796+0000","last_fullsized":"2024-03-29T11:12:27.702796+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:16.747337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"190'4","reported_seq":15,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.559611+0000","last_change":"2024-03-29T11:12:26.028492+0000","last_active":"2024-03-29T11:12:26.559611+0000","last_peered":"2024-03-29T11:12:26.559611+0000","last_clean":"2024-03-29T11:12:26.559611+0000","last_became_active":"2024-03-29T11:12:26.028222+0000","last_became_peered":"2024-03-29T11:12:26.028222+0000","last_unstale":"2024-03-29T11:12:26.559611+0000","last_undegraded":"2024-03-29T11:12:26.559611+0000","last_fullsized":"2024-03-29T11:12:26.559611+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:08:46.259447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511785+0000","last_change":"2024-03-29T11:12:26.557353+0000","last_active":"2024-03-29T11:12:29.511785+0000","last_peered":"2024-03-29T11:12:29.511785+0000","last_clean":"2024-03-29T11:12:29.511785+0000","last_became_active":"2024-03-29T11:12:26.556973+0000","last_became_peered":"2024-03-29T11:12:26.556973+0000","last_unstale":"2024-03-29T11:12:29.511785+0000","last_undegraded":"2024-03-29T11:12:29.511785+0000","last_fullsized":"2024-03-29T11:12:29.511785+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:42:10.216327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"190'1","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.558121+0000","last_change":"2024-03-29T11:12:26.025418+0000","last_active":"2024-03-29T11:12:26.558121+0000","last_peered":"2024-03-29T11:12:26.558121+0000","last_clean":"2024-03-29T11:12:26.558121+0000","last_became_active":"2024-03-29T11:12:26.025125+0000","last_became_peered":"2024-03-29T11:12:26.025125+0000","last_unstale":"2024-03-29T11:12:26.558121+0000","last_undegraded":"2024-03-29T11:12:26.558121+0000","last_fullsized":"2024-03-29T11:12:26.558121+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:29:47.416031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.512042+0000","last_change":"2024-03-29T11:12:26.553478+0000","last_active":"2024-03-29T11:12:29.512042+0000","last_peered":"2024-03-29T11:12:29.512042+0000","last_clean":"2024-03-29T11:12:29.512042+0000","last_became_active":"2024-03-29T11:12:26.553227+0000","last_became_peered":"2024-03-29T11:12:26.553227+0000","last_unstale":"2024-03-29T11:12:29.512042+0000","last_undegraded":"2024-03-29T11:12:29.512042+0000","last_fullsized":"2024-03-29T11:12:29.512042+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:03:09.737683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"190'2","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.553386+0000","last_change":"2024-03-29T11:12:26.025469+0000","last_active":"2024-03-29T11:12:26.553386+0000","last_peered":"2024-03-29T11:12:26.553386+0000","last_clean":"2024-03-29T11:12:26.553386+0000","last_became_active":"2024-03-29T11:12:26.025188+0000","last_became_peered":"2024-03-29T11:12:26.025188+0000","last_unstale":"2024-03-29T11:12:26.553386+0000","last_undegraded":"2024-03-29T11:12:26.553386+0000","last_fullsized":"2024-03-29T11:12:26.553386+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:01:24.473402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540804+0000","last_change":"2024-03-29T11:12:26.545492+0000","last_active":"2024-03-29T11:12:27.540804+0000","last_peered":"2024-03-29T11:12:27.540804+0000","last_clean":"2024-03-29T11:12:27.540804+0000","last_became_active":"2024-03-29T11:12:26.544640+0000","last_became_peered":"2024-03-29T11:12:26.544640+0000","last_unstale":"2024-03-29T11:12:27.540804+0000","last_undegraded":"2024-03-29T11:12:27.540804+0000","last_fullsized":"2024-03-29T11:12:27.540804+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:01.186607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"190'2","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.562843+0000","last_change":"2024-03-29T11:12:26.023982+0000","last_active":"2024-03-29T11:12:26.562843+0000","last_peered":"2024-03-29T11:12:26.562843+0000","last_clean":"2024-03-29T11:12:26.562843+0000","last_became_active":"2024-03-29T11:12:26.023716+0000","last_became_peered":"2024-03-29T11:12:26.023716+0000","last_unstale":"2024-03-29T11:12:26.562843+0000","last_undegraded":"2024-03-29T11:12:26.562843+0000","last_fullsized":"2024-03-29T11:12:26.562843+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:04:21.670707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.702587+0000","last_change":"2024-03-29T11:12:26.556617+0000","last_active":"2024-03-29T11:12:27.702587+0000","last_peered":"2024-03-29T11:12:27.702587+0000","last_clean":"2024-03-29T11:12:27.702587+0000","last_became_active":"2024-03-29T11:12:26.556374+0000","last_became_peered":"2024-03-29T11:12:26.556374+0000","last_unstale":"2024-03-29T11:12:27.702587+0000","last_undegraded":"2024-03-29T11:12:27.702587+0000","last_fullsized":"2024-03-29T11:12:27.702587+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:18:37.644980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"190'1","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.555782+0000","last_change":"2024-03-29T11:12:26.022002+0000","last_active":"2024-03-29T11:12:26.555782+0000","last_peered":"2024-03-29T11:12:26.555782+0000","last_clean":"2024-03-29T11:12:26.555782+0000","last_became_active":"2024-03-29T11:12:26.021196+0000","last_became_peered":"2024-03-29T11:12:26.021196+0000","last_unstale":"2024-03-29T11:12:26.555782+0000","last_undegraded":"2024-03-29T11:12:26.555782+0000","last_fullsized":"2024-03-29T11:12:26.555782+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:31:54.820703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.536634+0000","last_change":"2024-03-29T11:12:26.557902+0000","last_active":"2024-03-29T11:12:27.536634+0000","last_peered":"2024-03-29T11:12:27.536634+0000","last_clean":"2024-03-29T11:12:27.536634+0000","last_became_active":"2024-03-29T11:12:26.554204+0000","last_became_peered":"2024-03-29T11:12:26.554204+0000","last_unstale":"2024-03-29T11:12:27.536634+0000","last_undegraded":"2024-03-29T11:12:27.536634+0000","last_fullsized":"2024-03-29T11:12:27.536634+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:31:55.824063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"190'1","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.562000+0000","last_change":"2024-03-29T11:12:26.027337+0000","last_active":"2024-03-29T11:12:26.562000+0000","last_peered":"2024-03-29T11:12:26.562000+0000","last_clean":"2024-03-29T11:12:26.562000+0000","last_became_active":"2024-03-29T11:12:26.026899+0000","last_became_peered":"2024-03-29T11:12:26.026899+0000","last_unstale":"2024-03-29T11:12:26.562000+0000","last_undegraded":"2024-03-29T11:12:26.562000+0000","last_fullsized":"2024-03-29T11:12:26.562000+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:58:16.283140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.536355+0000","last_change":"2024-03-29T11:12:26.557795+0000","last_active":"2024-03-29T11:12:27.536355+0000","last_peered":"2024-03-29T11:12:27.536355+0000","last_clean":"2024-03-29T11:12:27.536355+0000","last_became_active":"2024-03-29T11:12:26.557521+0000","last_became_peered":"2024-03-29T11:12:26.557521+0000","last_unstale":"2024-03-29T11:12:27.536355+0000","last_undegraded":"2024-03-29T11:12:27.536355+0000","last_fullsized":"2024-03-29T11:12:27.536355+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:34:23.963780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.531384+0000","last_change":"2024-03-29T11:12:26.023943+0000","last_active":"2024-03-29T11:12:26.531384+0000","last_peered":"2024-03-29T11:12:26.531384+0000","last_clean":"2024-03-29T11:12:26.531384+0000","last_became_active":"2024-03-29T11:12:26.023517+0000","last_became_peered":"2024-03-29T11:12:26.023517+0000","last_unstale":"2024-03-29T11:12:26.531384+0000","last_undegraded":"2024-03-29T11:12:26.531384+0000","last_fullsized":"2024-03-29T11:12:26.531384+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:18:59.006987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511593+0000","last_change":"2024-03-29T11:12:26.547470+0000","last_active":"2024-03-29T11:12:29.511593+0000","last_peered":"2024-03-29T11:12:29.511593+0000","last_clean":"2024-03-29T11:12:29.511593+0000","last_became_active":"2024-03-29T11:12:26.544918+0000","last_became_peered":"2024-03-29T11:12:26.544918+0000","last_unstale":"2024-03-29T11:12:29.511593+0000","last_undegraded":"2024-03-29T11:12:29.511593+0000","last_fullsized":"2024-03-29T11:12:29.511593+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:44:48.770096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.535731+0000","last_change":"2024-03-29T11:12:26.019224+0000","last_active":"2024-03-29T11:12:26.535731+0000","last_peered":"2024-03-29T11:12:26.535731+0000","last_clean":"2024-03-29T11:12:26.535731+0000","last_became_active":"2024-03-29T11:12:26.018963+0000","last_became_peered":"2024-03-29T11:12:26.018963+0000","last_unstale":"2024-03-29T11:12:26.535731+0000","last_undegraded":"2024-03-29T11:12:26.535731+0000","last_fullsized":"2024-03-29T11:12:26.535731+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:43.500449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511734+0000","last_change":"2024-03-29T11:12:26.555370+0000","last_active":"2024-03-29T11:12:29.511734+0000","last_peered":"2024-03-29T11:12:29.511734+0000","last_clean":"2024-03-29T11:12:29.511734+0000","last_became_active":"2024-03-29T11:12:26.554279+0000","last_became_peered":"2024-03-29T11:12:26.554279+0000","last_unstale":"2024-03-29T11:12:29.511734+0000","last_undegraded":"2024-03-29T11:12:29.511734+0000","last_fullsized":"2024-03-29T11:12:29.511734+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:00.564428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.534233+0000","last_change":"2024-03-29T11:12:26.020177+0000","last_active":"2024-03-29T11:12:26.534233+0000","last_peered":"2024-03-29T11:12:26.534233+0000","last_clean":"2024-03-29T11:12:26.534233+0000","last_became_active":"2024-03-29T11:12:26.019744+0000","last_became_peered":"2024-03-29T11:12:26.019744+0000","last_unstale":"2024-03-29T11:12:26.534233+0000","last_undegraded":"2024-03-29T11:12:26.534233+0000","last_fullsized":"2024-03-29T11:12:26.534233+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:56:16.709598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511801+0000","last_change":"2024-03-29T11:12:26.562701+0000","last_active":"2024-03-29T11:12:29.511801+0000","last_peered":"2024-03-29T11:12:29.511801+0000","last_clean":"2024-03-29T11:12:29.511801+0000","last_became_active":"2024-03-29T11:12:26.562418+0000","last_became_peered":"2024-03-29T11:12:26.562418+0000","last_unstale":"2024-03-29T11:12:29.511801+0000","last_undegraded":"2024-03-29T11:12:29.511801+0000","last_fullsized":"2024-03-29T11:12:29.511801+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:15:43.269632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"190'3","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.557465+0000","last_change":"2024-03-29T11:12:26.018603+0000","last_active":"2024-03-29T11:12:26.557465+0000","last_peered":"2024-03-29T11:12:26.557465+0000","last_clean":"2024-03-29T11:12:26.557465+0000","last_became_active":"2024-03-29T11:12:26.017679+0000","last_became_peered":"2024-03-29T11:12:26.017679+0000","last_unstale":"2024-03-29T11:12:26.557465+0000","last_undegraded":"2024-03-29T11:12:26.557465+0000","last_fullsized":"2024-03-29T11:12:26.557465+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:43.500449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511767+0000","last_change":"2024-03-29T11:12:26.552288+0000","last_active":"2024-03-29T11:12:29.511767+0000","last_peered":"2024-03-29T11:12:29.511767+0000","last_clean":"2024-03-29T11:12:29.511767+0000","last_became_active":"2024-03-29T11:12:26.550370+0000","last_became_peered":"2024-03-29T11:12:26.550370+0000","last_unstale":"2024-03-29T11:12:29.511767+0000","last_undegraded":"2024-03-29T11:12:29.511767+0000","last_fullsized":"2024-03-29T11:12:29.511767+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:10:21.521863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"190'6","reported_seq":17,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.564568+0000","last_change":"2024-03-29T11:12:26.021805+0000","last_active":"2024-03-29T11:12:26.564568+0000","last_peered":"2024-03-29T11:12:26.564568+0000","last_clean":"2024-03-29T11:12:26.564568+0000","last_became_active":"2024-03-29T11:12:26.021427+0000","last_became_peered":"2024-03-29T11:12:26.021427+0000","last_unstale":"2024-03-29T11:12:26.564568+0000","last_undegraded":"2024-03-29T11:12:26.564568+0000","last_fullsized":"2024-03-29T11:12:26.564568+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:10:15.529111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511339+0000","last_change":"2024-03-29T11:12:26.554357+0000","last_active":"2024-03-29T11:12:29.511339+0000","last_peered":"2024-03-29T11:12:29.511339+0000","last_clean":"2024-03-29T11:12:29.511339+0000","last_became_active":"2024-03-29T11:12:26.554031+0000","last_became_peered":"2024-03-29T11:12:26.554031+0000","last_unstale":"2024-03-29T11:12:29.511339+0000","last_undegraded":"2024-03-29T11:12:29.511339+0000","last_fullsized":"2024-03-29T11:12:29.511339+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:22:54.194819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"190'2","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.562083+0000","last_change":"2024-03-29T11:12:26.019583+0000","last_active":"2024-03-29T11:12:26.562083+0000","last_peered":"2024-03-29T11:12:26.562083+0000","last_clean":"2024-03-29T11:12:26.562083+0000","last_became_active":"2024-03-29T11:12:26.019298+0000","last_became_peered":"2024-03-29T11:12:26.019298+0000","last_unstale":"2024-03-29T11:12:26.562083+0000","last_undegraded":"2024-03-29T11:12:26.562083+0000","last_fullsized":"2024-03-29T11:12:26.562083+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:06:46.777846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.702617+0000","last_change":"2024-03-29T11:12:26.544664+0000","last_active":"2024-03-29T11:12:27.702617+0000","last_peered":"2024-03-29T11:12:27.702617+0000","last_clean":"2024-03-29T11:12:27.702617+0000","last_became_active":"2024-03-29T11:12:26.544396+0000","last_became_peered":"2024-03-29T11:12:26.544396+0000","last_unstale":"2024-03-29T11:12:27.702617+0000","last_undegraded":"2024-03-29T11:12:27.702617+0000","last_fullsized":"2024-03-29T11:12:27.702617+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:04.837609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.544746+0000","last_change":"2024-03-29T11:12:26.014725+0000","last_active":"2024-03-29T11:12:26.544746+0000","last_peered":"2024-03-29T11:12:26.544746+0000","last_clean":"2024-03-29T11:12:26.544746+0000","last_became_active":"2024-03-29T11:12:26.014240+0000","last_became_peered":"2024-03-29T11:12:26.014240+0000","last_unstale":"2024-03-29T11:12:26.544746+0000","last_undegraded":"2024-03-29T11:12:26.544746+0000","last_fullsized":"2024-03-29T11:12:26.544746+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:48.259813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.536361+0000","last_change":"2024-03-29T11:12:26.548403+0000","last_active":"2024-03-29T11:12:27.536361+0000","last_peered":"2024-03-29T11:12:27.536361+0000","last_clean":"2024-03-29T11:12:27.536361+0000","last_became_active":"2024-03-29T11:12:26.544506+0000","last_became_peered":"2024-03-29T11:12:26.544506+0000","last_unstale":"2024-03-29T11:12:27.536361+0000","last_undegraded":"2024-03-29T11:12:27.536361+0000","last_fullsized":"2024-03-29T11:12:27.536361+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:47.542813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.543434+0000","last_change":"2024-03-29T11:12:26.021486+0000","last_active":"2024-03-29T11:12:26.543434+0000","last_peered":"2024-03-29T11:12:26.543434+0000","last_clean":"2024-03-29T11:12:26.543434+0000","last_became_active":"2024-03-29T11:12:26.021213+0000","last_became_peered":"2024-03-29T11:12:26.021213+0000","last_unstale":"2024-03-29T11:12:26.543434+0000","last_undegraded":"2024-03-29T11:12:26.543434+0000","last_fullsized":"2024-03-29T11:12:26.543434+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:18:11.751956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540614+0000","last_change":"2024-03-29T11:12:26.538598+0000","last_active":"2024-03-29T11:12:27.540614+0000","last_peered":"2024-03-29T11:12:27.540614+0000","last_clean":"2024-03-29T11:12:27.540614+0000","last_became_active":"2024-03-29T11:12:26.538319+0000","last_became_peered":"2024-03-29T11:12:26.538319+0000","last_unstale":"2024-03-29T11:12:27.540614+0000","last_undegraded":"2024-03-29T11:12:27.540614+0000","last_fullsized":"2024-03-29T11:12:27.540614+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:42:36.177398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"190'8","reported_seq":19,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.564112+0000","last_change":"2024-03-29T11:12:26.022274+0000","last_active":"2024-03-29T11:12:26.564112+0000","last_peered":"2024-03-29T11:12:26.564112+0000","last_clean":"2024-03-29T11:12:26.564112+0000","last_became_active":"2024-03-29T11:12:26.021322+0000","last_became_peered":"2024-03-29T11:12:26.021322+0000","last_unstale":"2024-03-29T11:12:26.564112+0000","last_undegraded":"2024-03-29T11:12:26.564112+0000","last_fullsized":"2024-03-29T11:12:26.564112+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:43.500449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.536928+0000","last_change":"2024-03-29T11:12:26.556791+0000","last_active":"2024-03-29T11:12:27.536928+0000","last_peered":"2024-03-29T11:12:27.536928+0000","last_clean":"2024-03-29T11:12:27.536928+0000","last_became_active":"2024-03-29T11:12:26.554165+0000","last_became_peered":"2024-03-29T11:12:26.554165+0000","last_unstale":"2024-03-29T11:12:27.536928+0000","last_undegraded":"2024-03-29T11:12:27.536928+0000","last_fullsized":"2024-03-29T11:12:27.536928+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:54:05.839788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"190'1","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.556401+0000","last_change":"2024-03-29T11:12:26.019991+0000","last_active":"2024-03-29T11:12:26.556401+0000","last_peered":"2024-03-29T11:12:26.556401+0000","last_clean":"2024-03-29T11:12:26.556401+0000","last_became_active":"2024-03-29T11:12:26.019637+0000","last_became_peered":"2024-03-29T11:12:26.019637+0000","last_unstale":"2024-03-29T11:12:26.556401+0000","last_undegraded":"2024-03-29T11:12:26.556401+0000","last_fullsized":"2024-03-29T11:12:26.556401+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:23:33.363505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.512013+0000","last_change":"2024-03-29T11:12:26.555996+0000","last_active":"2024-03-29T11:12:29.512013+0000","last_peered":"2024-03-29T11:12:29.512013+0000","last_clean":"2024-03-29T11:12:29.512013+0000","last_became_active":"2024-03-29T11:12:26.555733+0000","last_became_peered":"2024-03-29T11:12:26.555733+0000","last_unstale":"2024-03-29T11:12:29.512013+0000","last_undegraded":"2024-03-29T11:12:29.512013+0000","last_fullsized":"2024-03-29T11:12:29.512013+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:14:58.912360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"190'2","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.557846+0000","last_change":"2024-03-29T11:12:26.021870+0000","last_active":"2024-03-29T11:12:26.557846+0000","last_peered":"2024-03-29T11:12:26.557846+0000","last_clean":"2024-03-29T11:12:26.557846+0000","last_became_active":"2024-03-29T11:12:26.021169+0000","last_became_peered":"2024-03-29T11:12:26.021169+0000","last_unstale":"2024-03-29T11:12:26.557846+0000","last_undegraded":"2024-03-29T11:12:26.557846+0000","last_fullsized":"2024-03-29T11:12:26.557846+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.539453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.536856+0000","last_change":"2024-03-29T11:12:26.555857+0000","last_active":"2024-03-29T11:12:27.536856+0000","last_peered":"2024-03-29T11:12:27.536856+0000","last_clean":"2024-03-29T11:12:27.536856+0000","last_became_active":"2024-03-29T11:12:26.552154+0000","last_became_peered":"2024-03-29T11:12:26.552154+0000","last_unstale":"2024-03-29T11:12:27.536856+0000","last_undegraded":"2024-03-29T11:12:27.536856+0000","last_fullsized":"2024-03-29T11:12:27.536856+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:59:44.779680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"190'1","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.553859+0000","last_change":"2024-03-29T11:12:26.027912+0000","last_active":"2024-03-29T11:12:26.553859+0000","last_peered":"2024-03-29T11:12:26.553859+0000","last_clean":"2024-03-29T11:12:26.553859+0000","last_became_active":"2024-03-29T11:12:26.027609+0000","last_became_peered":"2024-03-29T11:12:26.027609+0000","last_unstale":"2024-03-29T11:12:26.553859+0000","last_undegraded":"2024-03-29T11:12:26.553859+0000","last_fullsized":"2024-03-29T11:12:26.553859+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:44:22.939828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.702621+0000","last_change":"2024-03-29T11:12:26.555607+0000","last_active":"2024-03-29T11:12:27.702621+0000","last_peered":"2024-03-29T11:12:27.702621+0000","last_clean":"2024-03-29T11:12:27.702621+0000","last_became_active":"2024-03-29T11:12:26.555360+0000","last_became_peered":"2024-03-29T11:12:26.555360+0000","last_unstale":"2024-03-29T11:12:27.702621+0000","last_undegraded":"2024-03-29T11:12:27.702621+0000","last_fullsized":"2024-03-29T11:12:27.702621+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:10:21.521863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.540298+0000","last_change":"2024-03-29T11:12:26.028063+0000","last_active":"2024-03-29T11:12:26.540298+0000","last_peered":"2024-03-29T11:12:26.540298+0000","last_clean":"2024-03-29T11:12:26.540298+0000","last_became_active":"2024-03-29T11:12:26.027367+0000","last_became_peered":"2024-03-29T11:12:26.027367+0000","last_unstale":"2024-03-29T11:12:26.540298+0000","last_undegraded":"2024-03-29T11:12:26.540298+0000","last_fullsized":"2024-03-29T11:12:26.540298+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:39:02.038526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540289+0000","last_change":"2024-03-29T11:12:26.553554+0000","last_active":"2024-03-29T11:12:27.540289+0000","last_peered":"2024-03-29T11:12:27.540289+0000","last_clean":"2024-03-29T11:12:27.540289+0000","last_became_active":"2024-03-29T11:12:26.553135+0000","last_became_peered":"2024-03-29T11:12:26.553135+0000","last_unstale":"2024-03-29T11:12:27.540289+0000","last_undegraded":"2024-03-29T11:12:27.540289+0000","last_fullsized":"2024-03-29T11:12:27.540289+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:54.248899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"190'2","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.564170+0000","last_change":"2024-03-29T11:12:26.023219+0000","last_active":"2024-03-29T11:12:26.564170+0000","last_peered":"2024-03-29T11:12:26.564170+0000","last_clean":"2024-03-29T11:12:26.564170+0000","last_became_active":"2024-03-29T11:12:26.022948+0000","last_became_peered":"2024-03-29T11:12:26.022948+0000","last_unstale":"2024-03-29T11:12:26.564170+0000","last_undegraded":"2024-03-29T11:12:26.564170+0000","last_fullsized":"2024-03-29T11:12:26.564170+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:50:40.386291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.702655+0000","last_change":"2024-03-29T11:12:26.560819+0000","last_active":"2024-03-29T11:12:27.702655+0000","last_peered":"2024-03-29T11:12:27.702655+0000","last_clean":"2024-03-29T11:12:27.702655+0000","last_became_active":"2024-03-29T11:12:26.560574+0000","last_became_peered":"2024-03-29T11:12:26.560574+0000","last_unstale":"2024-03-29T11:12:27.702655+0000","last_undegraded":"2024-03-29T11:12:27.702655+0000","last_fullsized":"2024-03-29T11:12:27.702655+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:22:39.566319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.539770+0000","last_change":"2024-03-29T11:12:26.024796+0000","last_active":"2024-03-29T11:12:26.539770+0000","last_peered":"2024-03-29T11:12:26.539770+0000","last_clean":"2024-03-29T11:12:26.539770+0000","last_became_active":"2024-03-29T11:12:26.024534+0000","last_became_peered":"2024-03-29T11:12:26.024534+0000","last_unstale":"2024-03-29T11:12:26.539770+0000","last_undegraded":"2024-03-29T11:12:26.539770+0000","last_fullsized":"2024-03-29T11:12:26.539770+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:38.551929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.697412+0000","last_change":"2024-03-29T11:12:26.552219+0000","last_active":"2024-03-29T11:12:27.697412+0000","last_peered":"2024-03-29T11:12:27.697412+0000","last_clean":"2024-03-29T11:12:27.697412+0000","last_became_active":"2024-03-29T11:12:26.551599+0000","last_became_peered":"2024-03-29T11:12:26.551599+0000","last_unstale":"2024-03-29T11:12:27.697412+0000","last_undegraded":"2024-03-29T11:12:27.697412+0000","last_fullsized":"2024-03-29T11:12:27.697412+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:22:54.194819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.543459+0000","last_change":"2024-03-29T11:12:26.022117+0000","last_active":"2024-03-29T11:12:26.543459+0000","last_peered":"2024-03-29T11:12:26.543459+0000","last_clean":"2024-03-29T11:12:26.543459+0000","last_became_active":"2024-03-29T11:12:26.021815+0000","last_became_peered":"2024-03-29T11:12:26.021815+0000","last_unstale":"2024-03-29T11:12:26.543459+0000","last_undegraded":"2024-03-29T11:12:26.543459+0000","last_fullsized":"2024-03-29T11:12:26.543459+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:21:21.076484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511597+0000","last_change":"2024-03-29T11:12:26.548252+0000","last_active":"2024-03-29T11:12:29.511597+0000","last_peered":"2024-03-29T11:12:29.511597+0000","last_clean":"2024-03-29T11:12:29.511597+0000","last_became_active":"2024-03-29T11:12:26.547645+0000","last_became_peered":"2024-03-29T11:12:26.547645+0000","last_unstale":"2024-03-29T11:12:29.511597+0000","last_undegraded":"2024-03-29T11:12:29.511597+0000","last_fullsized":"2024-03-29T11:12:29.511597+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:43.290223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.538549+0000","last_change":"2024-03-29T11:12:26.027875+0000","last_active":"2024-03-29T11:12:26.538549+0000","last_peered":"2024-03-29T11:12:26.538549+0000","last_clean":"2024-03-29T11:12:26.538549+0000","last_became_active":"2024-03-29T11:12:26.027580+0000","last_became_peered":"2024-03-29T11:12:26.027580+0000","last_unstale":"2024-03-29T11:12:26.538549+0000","last_undegraded":"2024-03-29T11:12:26.538549+0000","last_fullsized":"2024-03-29T11:12:26.538549+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:34.078365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.512104+0000","last_change":"2024-03-29T11:12:26.563352+0000","last_active":"2024-03-29T11:12:29.512104+0000","last_peered":"2024-03-29T11:12:29.512104+0000","last_clean":"2024-03-29T11:12:29.512104+0000","last_became_active":"2024-03-29T11:12:26.563066+0000","last_became_peered":"2024-03-29T11:12:26.563066+0000","last_unstale":"2024-03-29T11:12:29.512104+0000","last_undegraded":"2024-03-29T11:12:29.512104+0000","last_fullsized":"2024-03-29T11:12:29.512104+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:32:10.452543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"190'2","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.556997+0000","last_change":"2024-03-29T11:12:26.019246+0000","last_active":"2024-03-29T11:12:26.556997+0000","last_peered":"2024-03-29T11:12:26.556997+0000","last_clean":"2024-03-29T11:12:26.556997+0000","last_became_active":"2024-03-29T11:12:26.018978+0000","last_became_peered":"2024-03-29T11:12:26.018978+0000","last_unstale":"2024-03-29T11:12:26.556997+0000","last_undegraded":"2024-03-29T11:12:26.556997+0000","last_fullsized":"2024-03-29T11:12:26.556997+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:38:17.048108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.702806+0000","last_change":"2024-03-29T11:12:26.547390+0000","last_active":"2024-03-29T11:12:27.702806+0000","last_peered":"2024-03-29T11:12:27.702806+0000","last_clean":"2024-03-29T11:12:27.702806+0000","last_became_active":"2024-03-29T11:12:26.547131+0000","last_became_peered":"2024-03-29T11:12:26.547131+0000","last_unstale":"2024-03-29T11:12:27.702806+0000","last_undegraded":"2024-03-29T11:12:27.702806+0000","last_fullsized":"2024-03-29T11:12:27.702806+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:52:01.366224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"191'1","reported_seq":12,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540828+0000","last_change":"2024-03-29T11:12:26.022064+0000","last_active":"2024-03-29T11:12:27.540828+0000","last_peered":"2024-03-29T11:12:27.540828+0000","last_clean":"2024-03-29T11:12:27.540828+0000","last_became_active":"2024-03-29T11:12:26.021581+0000","last_became_peered":"2024-03-29T11:12:26.021581+0000","last_unstale":"2024-03-29T11:12:27.540828+0000","last_undegraded":"2024-03-29T11:12:27.540828+0000","last_fullsized":"2024-03-29T11:12:27.540828+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:57.014805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.512141+0000","last_change":"2024-03-29T11:12:26.557073+0000","last_active":"2024-03-29T11:12:29.512141+0000","last_peered":"2024-03-29T11:12:29.512141+0000","last_clean":"2024-03-29T11:12:29.512141+0000","last_became_active":"2024-03-29T11:12:26.556812+0000","last_became_peered":"2024-03-29T11:12:26.556812+0000","last_unstale":"2024-03-29T11:12:29.512141+0000","last_undegraded":"2024-03-29T11:12:29.512141+0000","last_fullsized":"2024-03-29T11:12:29.512141+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:13.137676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.533455+0000","last_change":"2024-03-29T11:12:26.023077+0000","last_active":"2024-03-29T11:12:26.533455+0000","last_peered":"2024-03-29T11:12:26.533455+0000","last_clean":"2024-03-29T11:12:26.533455+0000","last_became_active":"2024-03-29T11:12:26.022814+0000","last_became_peered":"2024-03-29T11:12:26.022814+0000","last_unstale":"2024-03-29T11:12:26.533455+0000","last_undegraded":"2024-03-29T11:12:26.533455+0000","last_fullsized":"2024-03-29T11:12:26.533455+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:21:17.353455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511598+0000","last_change":"2024-03-29T11:12:26.558988+0000","last_active":"2024-03-29T11:12:29.511598+0000","last_peered":"2024-03-29T11:12:29.511598+0000","last_clean":"2024-03-29T11:12:29.511598+0000","last_became_active":"2024-03-29T11:12:26.558721+0000","last_became_peered":"2024-03-29T11:12:26.558721+0000","last_unstale":"2024-03-29T11:12:29.511598+0000","last_undegraded":"2024-03-29T11:12:29.511598+0000","last_fullsized":"2024-03-29T11:12:29.511598+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:08:47.262807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.545917+0000","last_change":"2024-03-29T11:12:26.020194+0000","last_active":"2024-03-29T11:12:26.545917+0000","last_peered":"2024-03-29T11:12:26.545917+0000","last_clean":"2024-03-29T11:12:26.545917+0000","last_became_active":"2024-03-29T11:12:26.019929+0000","last_became_peered":"2024-03-29T11:12:26.019929+0000","last_unstale":"2024-03-29T11:12:26.545917+0000","last_undegraded":"2024-03-29T11:12:26.545917+0000","last_fullsized":"2024-03-29T11:12:26.545917+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:22:03.070734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511348+0000","last_change":"2024-03-29T11:12:26.553228+0000","last_active":"2024-03-29T11:12:29.511348+0000","last_peered":"2024-03-29T11:12:29.511348+0000","last_clean":"2024-03-29T11:12:29.511348+0000","last_became_active":"2024-03-29T11:12:26.552960+0000","last_became_peered":"2024-03-29T11:12:26.552960+0000","last_unstale":"2024-03-29T11:12:29.511348+0000","last_undegraded":"2024-03-29T11:12:29.511348+0000","last_fullsized":"2024-03-29T11:12:29.511348+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:39.555289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.541677+0000","last_change":"2024-03-29T11:12:26.021160+0000","last_active":"2024-03-29T11:12:26.541677+0000","last_peered":"2024-03-29T11:12:26.541677+0000","last_clean":"2024-03-29T11:12:26.541677+0000","last_became_active":"2024-03-29T11:12:26.020357+0000","last_became_peered":"2024-03-29T11:12:26.020357+0000","last_unstale":"2024-03-29T11:12:26.541677+0000","last_undegraded":"2024-03-29T11:12:26.541677+0000","last_fullsized":"2024-03-29T11:12:26.541677+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:44.789023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.511388+0000","last_change":"2024-03-29T11:12:26.550975+0000","last_active":"2024-03-29T11:12:29.511388+0000","last_peered":"2024-03-29T11:12:29.511388+0000","last_clean":"2024-03-29T11:12:29.511388+0000","last_became_active":"2024-03-29T11:12:26.550548+0000","last_became_peered":"2024-03-29T11:12:26.550548+0000","last_unstale":"2024-03-29T11:12:29.511388+0000","last_undegraded":"2024-03-29T11:12:29.511388+0000","last_fullsized":"2024-03-29T11:12:29.511388+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:43:45.321870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.540369+0000","last_change":"2024-03-29T11:12:26.021825+0000","last_active":"2024-03-29T11:12:26.540369+0000","last_peered":"2024-03-29T11:12:26.540369+0000","last_clean":"2024-03-29T11:12:26.540369+0000","last_became_active":"2024-03-29T11:12:26.021554+0000","last_became_peered":"2024-03-29T11:12:26.021554+0000","last_unstale":"2024-03-29T11:12:26.540369+0000","last_undegraded":"2024-03-29T11:12:26.540369+0000","last_fullsized":"2024-03-29T11:12:26.540369+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:54.251978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.702359+0000","last_change":"2024-03-29T11:12:26.553821+0000","last_active":"2024-03-29T11:12:27.702359+0000","last_peered":"2024-03-29T11:12:27.702359+0000","last_clean":"2024-03-29T11:12:27.702359+0000","last_became_active":"2024-03-29T11:12:26.553547+0000","last_became_peered":"2024-03-29T11:12:26.553547+0000","last_unstale":"2024-03-29T11:12:27.702359+0000","last_undegraded":"2024-03-29T11:12:27.702359+0000","last_fullsized":"2024-03-29T11:12:27.702359+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:44:48.770096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.531518+0000","last_change":"2024-03-29T11:12:26.019069+0000","last_active":"2024-03-29T11:12:26.531518+0000","last_peered":"2024-03-29T11:12:26.531518+0000","last_clean":"2024-03-29T11:12:26.531518+0000","last_became_active":"2024-03-29T11:12:26.018793+0000","last_became_peered":"2024-03-29T11:12:26.018793+0000","last_unstale":"2024-03-29T11:12:26.531518+0000","last_undegraded":"2024-03-29T11:12:26.531518+0000","last_fullsized":"2024-03-29T11:12:26.531518+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:59:43.776320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:27.540380+0000","last_change":"2024-03-29T11:12:26.542896+0000","last_active":"2024-03-29T11:12:27.540380+0000","last_peered":"2024-03-29T11:12:27.540380+0000","last_clean":"2024-03-29T11:12:27.540380+0000","last_became_active":"2024-03-29T11:12:26.542629+0000","last_became_peered":"2024-03-29T11:12:26.542629+0000","last_unstale":"2024-03-29T11:12:27.540380+0000","last_undegraded":"2024-03-29T11:12:27.540380+0000","last_fullsized":"2024-03-29T11:12:27.540380+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:58:27.138713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.535930+0000","last_change":"2024-03-29T11:12:26.026019+0000","last_active":"2024-03-29T11:12:26.535930+0000","last_peered":"2024-03-29T11:12:26.535930+0000","last_clean":"2024-03-29T11:12:26.535930+0000","last_became_active":"2024-03-29T11:12:26.025636+0000","last_became_peered":"2024-03-29T11:12:26.025636+0000","last_unstale":"2024-03-29T11:12:26.535930+0000","last_undegraded":"2024-03-29T11:12:26.535930+0000","last_fullsized":"2024-03-29T11:12:26.535930+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:50:38.495397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-29T11:12:29.512006+0000","last_change":"2024-03-29T11:12:26.564173+0000","last_active":"2024-03-29T11:12:29.512006+0000","last_peered":"2024-03-29T11:12:29.512006+0000","last_clean":"2024-03-29T11:12:29.512006+0000","last_became_active":"2024-03-29T11:12:26.563864+0000","last_became_peered":"2024-03-29T11:12:26.563864+0000","last_unstale":"2024-03-29T11:12:29.512006+0000","last_undegraded":"2024-03-29T11:12:29.512006+0000","last_fullsized":"2024-03-29T11:12:29.512006+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:25.995407+0000","last_clean_scrub_stamp":"2024-03-29T11:12:25.995407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:49.664291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.541681+0000","last_change":"2024-03-29T11:12:26.023490+0000","last_active":"2024-03-29T11:12:26.541681+0000","last_peered":"2024-03-29T11:12:26.541681+0000","last_clean":"2024-03-29T11:12:26.541681+0000","last_became_active":"2024-03-29T11:12:26.023162+0000","last_became_peered":"2024-03-29T11:12:26.023162+0000","last_unstale":"2024-03-29T11:12:26.541681+0000","last_undegraded":"2024-03-29T11:12:26.541681+0000","last_fullsized":"2024-03-29T11:12:26.541681+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:12:24.992047+0000","last_clean_scrub_stamp":"2024-03-29T11:12:24.992047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:56:16.709598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"180'311","reported_seq":1034,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-29T11:12:26.538572+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:12:26.538572+0000","last_peered":"2024-03-29T11:12:26.538572+0000","last_clean":"2024-03-29T11:12:26.538572+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:12:26.538572+0000","last_undegraded":"2024-03-29T11:12:26.538572+0000","last_fullsized":"2024-03-29T11:12:26.538572+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":511,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":511,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":311,"ondisk_log_size":311,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575771,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48012,"kb_used_data":2936,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323828,"statfs":{"total":96636764160,"available":96587599872,"internally_reserved":0,"allocated":3006464,"data_stored":1746665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":46135756},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.71899999999999997,"15min":0.72399999999999998},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.40000000000000002},"max":{"1min":0.93999999999999995,"5min":1.0089999999999999,"15min":1.014},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.69799999999999995}]},{"osd":1,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":1.008},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.746,"15min":0.76600000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.081},"last":0.872}]},{"osd":2,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.73199999999999998,"15min":0.749},"min":{"1min":0.502,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.92900000000000005,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.76300000000000001,"15min":0.77000000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.2310000000000001},"last":0.76700000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.74299999999999999,"15min":0.75700000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.42999999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.091},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77800000000000002,"15min":0.78200000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0049999999999999,"5min":1.028,"15min":1.028},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63900000000000001,"15min":0.628},"min":{"1min":0.45600000000000002,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.80500000000000005,"5min":0.875,"15min":1.1499999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.52300000000000002,"5min":0.46300000000000002,"15min":0.42999999999999999},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.208},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67600000000000005,"15min":0.66100000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.39100000000000001},"max":{"1min":0.82899999999999996,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71399999999999997,"15min":0.70499999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.41799999999999998},"max":{"1min":0.90900000000000003,"5min":1.0580000000000001,"15min":1.1020000000000001},"last":0.62}]},{"osd":6,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.56799999999999995,"5min":0.47999999999999998,"15min":0.47499999999999998},"max":{"1min":1,"5min":1.008,"15min":1.101},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.749,"15min":0.72599999999999998},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.50800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575772,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41648,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330192,"statfs":{"total":96636764160,"available":96594116608,"internally_reserved":0,"allocated":2392064,"data_stored":1154013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.70999999999999996,"15min":0.72899999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.38900000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.752,"15min":0.751},"min":{"1min":0.55800000000000005,"5min":0.50800000000000001,"15min":0.36799999999999999},"max":{"1min":0.999,"5min":1.012,"15min":1.135},"last":0.63700000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.40799999999999997},"max":{"1min":0.89000000000000001,"5min":0.94599999999999995,"15min":1.01},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.76400000000000001,"15min":0.77000000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.439},"max":{"1min":1.016,"5min":1.016,"15min":1.04},"last":0.77000000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73599999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.51300000000000001,"15min":0.48699999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98099999999999998,"5min":1.0029999999999999,"15min":1.0720000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.77900000000000003,"15min":0.77400000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.71099999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61099999999999999,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76200000000000001,"5min":0.89000000000000001,"15min":0.93899999999999995},"last":0.498},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70199999999999996,"15min":0.69099999999999995},"min":{"1min":0.54100000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95099999999999996,"5min":0.98799999999999999,"15min":1.0469999999999999},"last":0.68500000000000005}]},{"osd":5,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66000000000000003,"15min":0.65600000000000003},"min":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.435},"max":{"1min":0.996,"5min":1.0049999999999999,"15min":1.125},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.70799999999999996,"15min":0.68700000000000006},"min":{"1min":0.432,"5min":0.432,"15min":0.40400000000000003},"max":{"1min":0.98299999999999998,"5min":1.0629999999999999,"15min":1.0980000000000001},"last":0.80100000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.75800000000000001,"15min":0.752},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.52800000000000002},"max":{"1min":0.95699999999999996,"5min":0.97999999999999998,"15min":1.077},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73299999999999998,"15min":0.72399999999999998},"min":{"1min":0.57199999999999995,"5min":0.53300000000000003,"15min":0.46200000000000002},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.74399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575770,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59692,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312148,"statfs":{"total":96636764160,"available":96575639552,"internally_reserved":0,"allocated":2392064,"data_stored":1154025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.63200000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.435},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.91000000000000003},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":0.79200000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.372,"15min":0.34399999999999997},"max":{"1min":0.70999999999999996,"5min":0.88100000000000001,"15min":0.91800000000000004},"last":0.59399999999999997}]},{"osd":3,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.622,"15min":0.622},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76200000000000001,"5min":0.81599999999999995,"15min":0.97599999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.42199999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.752,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.52600000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81000000000000005,"15min":0.79900000000000004},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.53800000000000003},"max":{"1min":1.139,"5min":1.139,"15min":1.1950000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.65400000000000003,"5min":0.625,"15min":0.57699999999999996},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.79800000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.83799999999999997,"15min":0.83199999999999996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.85899999999999999,"15min":0.84299999999999997},"min":{"1min":0.68799999999999994,"5min":0.60699999999999998,"15min":0.59099999999999997},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.88200000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.84699999999999998},"min":{"1min":0.63,"5min":0.63,"15min":0.61099999999999999},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.879,"15min":0.86299999999999999},"min":{"1min":0.67200000000000004,"5min":0.63200000000000001,"15min":0.56799999999999995},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.92100000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.89300000000000002,"15min":0.88100000000000001},"min":{"1min":0.79200000000000004,"5min":0.627,"15min":0.56000000000000005},"max":{"1min":1.1419999999999999,"5min":1.1459999999999999,"15min":1.1819999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.89300000000000002,"15min":0.88},"min":{"1min":0.71699999999999997,"5min":0.57499999999999996,"15min":0.55000000000000004},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.94099999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575772,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60264,"kb_used_data":2908,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311576,"statfs":{"total":96636764160,"available":96575053824,"internally_reserved":0,"allocated":2977792,"data_stored":1743340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.47199999999999998,"15min":0.46200000000000002},"min":{"1min":0.32300000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89800000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.51900000000000002,"15min":0.53400000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70099999999999996,"5min":0.73999999999999999,"15min":1.03},"last":0.60099999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.36799999999999999},"max":{"1min":0.751,"5min":0.80100000000000005,"15min":1.1459999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68799999999999994,"5min":0.81599999999999995,"15min":0.88700000000000001},"last":0.30499999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.36699999999999999},"max":{"1min":0.79000000000000004,"5min":0.85199999999999998,"15min":0.85899999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.433},"max":{"1min":0.80800000000000005,"5min":0.873,"15min":1.226},"last":0.65400000000000003}]},{"osd":4,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82199999999999995,"15min":0.82699999999999996},"min":{"1min":0.629,"5min":0.56200000000000006,"15min":0.48899999999999999},"max":{"1min":1.123,"5min":1.304,"15min":1.304},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.83999999999999997,"15min":0.85799999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.50700000000000001},"max":{"1min":1.0820000000000001,"5min":1.1100000000000001,"15min":1.2809999999999999},"last":0.77800000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.80500000000000005,"15min":0.79300000000000004},"min":{"1min":0.54100000000000004,"5min":0.53400000000000003,"15min":0.48399999999999999},"max":{"1min":0.98599999999999999,"5min":1.127,"15min":1.129},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.84099999999999997,"15min":0.83599999999999997},"min":{"1min":0.60899999999999999,"5min":0.56100000000000005,"15min":0.54000000000000004},"max":{"1min":0.99299999999999999,"5min":1.27,"15min":1.27},"last":0.54900000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.84099999999999997,"15min":0.85499999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.018,"5min":1.1679999999999999,"15min":1.329},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.88600000000000001,"15min":0.88900000000000001},"min":{"1min":0.65000000000000002,"5min":0.61699999999999999,"15min":0.58099999999999996},"max":{"1min":1.0589999999999999,"5min":1.337,"15min":1.337},"last":0.626}]},{"osd":7,"last update":"Fri Mar 29 11:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.82999999999999996,"15min":0.83199999999999996},"min":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.53400000000000003},"max":{"1min":1.0149999999999999,"5min":1.161,"15min":1.181},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.56499999999999995},"max":{"1min":1.159,"5min":1.1890000000000001,"15min":1.298},"last":0.68999999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575771,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47404,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324436,"statfs":{"total":96636764160,"available":96588222464,"internally_reserved":0,"allocated":2392064,"data_stored":1154081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46600000000000003,"15min":0.47399999999999998},"min":{"1min":0.29999999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.77700000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.53500000000000003,"15min":0.54300000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67600000000000005,"5min":0.80000000000000004,"15min":0.878},"last":0.54000000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.63,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.5}]},{"osd":3,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.44400000000000001,"5min":0.436,"15min":0.41499999999999998},"max":{"1min":0.65600000000000003,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58199999999999996},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.80800000000000005,"15min":0.90800000000000003},"last":0.52200000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.79300000000000004,"15min":0.81399999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55200000000000005},"max":{"1min":0.95199999999999996,"5min":1.0640000000000001,"15min":1.135},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.81200000000000006,"15min":0.81799999999999995},"min":{"1min":0.61499999999999999,"5min":0.60699999999999998,"15min":0.56699999999999995},"max":{"1min":0.99299999999999999,"5min":1.1419999999999999,"15min":1.3680000000000001},"last":0.66000000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.5},"max":{"1min":1.0580000000000001,"5min":1.2350000000000001,"15min":1.2709999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.81699999999999995,"15min":0.82799999999999996},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.56299999999999994},"max":{"1min":1.022,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.88800000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.83699999999999997,"15min":0.84599999999999997},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.042,"5min":1.321,"15min":1.343},"last":0.75},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.84199999999999997,"15min":0.85199999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1,"5min":1.321,"15min":1.321},"last":0.86699999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.90200000000000002,"15min":0.90100000000000002},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.58199999999999996},"max":{"1min":1.089,"5min":1.2729999999999999,"15min":1.403},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.91300000000000003,"15min":0.91100000000000003},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.58699999999999997},"max":{"1min":1.099,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.91000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575771,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59692,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312148,"statfs":{"total":96636764160,"available":96575639552,"internally_reserved":0,"allocated":2392064,"data_stored":1154025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48199999999999998,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55600000000000005,"15min":0.55100000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.31},"max":{"1min":0.80000000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.501,"15min":0.503},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.72499999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.40799999999999997,"15min":0.38900000000000001},"max":{"1min":0.77800000000000002,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.88},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.66600000000000004,"15min":0.66200000000000003},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.436},"max":{"1min":0.89400000000000002,"5min":0.91900000000000004,"15min":0.98799999999999999},"last":0.62}]},{"osd":4,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.79300000000000004,"15min":0.80500000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.97399999999999998,"5min":1.2629999999999999,"15min":1.5469999999999999},"last":0.755},{"interface":"front","average":{"1min":0.748,"5min":0.80600000000000005,"15min":0.80800000000000005},"min":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.51300000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.5309999999999999},"last":0.88900000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.80800000000000005,"15min":0.82199999999999995},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.52400000000000002},"max":{"1min":1.0780000000000001,"5min":1.113,"15min":1.4199999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.83299999999999996,"15min":0.83899999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.53200000000000003},"max":{"1min":1.175,"5min":1.3140000000000001,"15min":1.3740000000000001},"last":0.79700000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.84599999999999997,"15min":0.85499999999999998},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.56399999999999995},"max":{"1min":1.117,"5min":1.1339999999999999,"15min":1.248},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84699999999999998,"15min":0.85299999999999998},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.113,"5min":1.212,"15min":1.2849999999999999},"last":0.83399999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.86899999999999999,"15min":0.872},"min":{"1min":0.67100000000000004,"5min":0.64300000000000002,"15min":0.58199999999999996},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.6020000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.89500000000000002,"15min":0.89300000000000002},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.157,"5min":1.2350000000000001,"15min":1.3340000000000001},"last":0.77300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575772,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55608,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316232,"statfs":{"total":96636764160,"available":96579821568,"internally_reserved":0,"allocated":2404352,"data_stored":1155625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.72599999999999998,"15min":0.72799999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.33300000000000002},"max":{"1min":0.85699999999999998,"5min":1.0209999999999999,"15min":1.097},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.69999999999999996,"15min":0.72199999999999998},"min":{"1min":0.45700000000000002,"5min":0.44400000000000001,"15min":0.40899999999999997},"max":{"1min":0.84499999999999997,"5min":0.93899999999999995,"15min":1.3300000000000001},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.74299999999999999,"15min":0.747},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.78400000000000003,"5min":1.105,"15min":1.105},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.71599999999999997,"15min":0.73299999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86799999999999999,"5min":0.94599999999999995,"15min":1.0209999999999999},"last":0.53100000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.79800000000000004,"15min":0.81799999999999995},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.437},"max":{"1min":0.90400000000000003,"5min":1.034,"15min":1.3080000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.79300000000000004,"15min":0.80300000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.46800000000000003},"max":{"1min":0.86699999999999999,"5min":1.0349999999999999,"15min":1.1140000000000001},"last":0.66700000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.754,"15min":0.755},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.53700000000000003},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":1.177},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.74199999999999999,"15min":0.751},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1319999999999999,"5min":1.137,"15min":1.137},"last":0.627}]},{"osd":5,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.91300000000000003,"5min":0.93100000000000005,"15min":1.054},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61499999999999999,"15min":0.61799999999999999},"min":{"1min":0.46300000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.88600000000000001,"5min":0.98199999999999998,"15min":1.1779999999999999},"last":0.503}]},{"osd":6,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63700000000000001,"15min":0.63200000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.94899999999999995,"5min":1.038,"15min":1.0389999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67700000000000005,"15min":0.67300000000000004},"min":{"1min":0.45900000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":1.254},"last":0.82399999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.70499999999999996,"15min":0.69499999999999995},"min":{"1min":0.57099999999999995,"5min":0.47899999999999998,"15min":0.44900000000000001},"max":{"1min":0.98599999999999999,"5min":1.004,"15min":1.4379999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.52400000000000002},"max":{"1min":1.01,"5min":1.0580000000000001,"15min":1.482},"last":0.89900000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575771,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41656,"kb_used_data":2340,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330184,"statfs":{"total":96636764160,"available":96594108416,"internally_reserved":0,"allocated":2396160,"data_stored":1154595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.72699999999999998,"15min":0.72399999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.40699999999999997},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71599999999999997,"15min":0.74199999999999999},"min":{"1min":0.58299999999999996,"5min":0.48099999999999998,"15min":0.45000000000000001},"max":{"1min":0.84499999999999997,"5min":0.94799999999999995,"15min":1},"last":0.80000000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.73399999999999999,"15min":0.752},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90500000000000003,"5min":0.997,"15min":1.052},"last":0.82199999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.749,"15min":0.76200000000000001},"min":{"1min":0.503,"5min":0.499,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.016,"15min":1.0609999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73199999999999998,"15min":0.745},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.94199999999999995}]},{"osd":3,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.747,"15min":0.76700000000000002},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.52300000000000002},"max":{"1min":0.93000000000000005,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.78800000000000003},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.77200000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.38800000000000001,"15min":0.379},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64700000000000002,"15min":0.64300000000000002},"min":{"1min":0.501,"5min":0.46300000000000002,"15min":0.41199999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98299999999999998},"last":0.64600000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.5,"5min":0.48699999999999999,"15min":0.439},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.625},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.66600000000000004,"15min":0.67100000000000004},"min":{"1min":0.502,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.69999999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.71299999999999997,"15min":0.71099999999999997},"min":{"1min":0.55800000000000005,"5min":0.46800000000000003,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71899999999999997,"15min":0.73699999999999999},"min":{"1min":0.59599999999999997,"5min":0.52500000000000002,"15min":0.49399999999999999},"max":{"1min":0.93100000000000005,"5min":0.95399999999999996,"15min":1.0369999999999999},"last":0.72599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:12:31.929 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T11:12:31.929 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T11:12:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.192+0000 7fefce600700 1 Processor -- start 2024-03-29T11:12:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.192+0000 7fefce600700 1 -- start start 2024-03-29T11:12:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.193+0000 7fefce600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc815d8a0 0x7fefc80b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.193+0000 7fefce600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fefc80d3eb0 0x7fefc806b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.193+0000 7fefce600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc80ced20 con 0x7fefc806bdd0 2024-03-29T11:12:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.193+0000 7fefce600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc80d4340 con 0x7fefc80d3eb0 2024-03-29T11:12:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.193+0000 7fefce600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc80ce6d0 con 0x7fefc815d8a0 2024-03-29T11:12:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.194+0000 7fefc6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fefc80d3eb0 0x7fefc806b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.194+0000 7fefc6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fefc80d3eb0 0x7fefc806b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42676/0 (socket says 172.21.15.12:42676) 2024-03-29T11:12:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.194+0000 7fefc6a00700 1 -- 172.21.15.12:0/2319141772 learned_addr learned my addr 172.21.15.12:0/2319141772 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.194+0000 7fefc7400700 1 --2- 172.21.15.12:0/2319141772 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc815d8a0 0x7fefc80b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.194+0000 7fefc6a00700 1 -- 172.21.15.12:0/2319141772 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc815d8a0 msgr2=0x7fefc80b2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.194+0000 7fefc6a00700 1 --2- 172.21.15.12:0/2319141772 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc815d8a0 0x7fefc80b2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.194+0000 7fefc6a00700 1 -- 172.21.15.12:0/2319141772 >> v1:172.21.15.159:6789/0 conn(0x7fefc806bdd0 legacy=0x7fefc80cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.194+0000 7fefc6a00700 1 -- 172.21.15.12:0/2319141772 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc80ce9b0 con 0x7fefc80d3eb0 2024-03-29T11:12:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.194+0000 7fefc6a00700 1 --2- 172.21.15.12:0/2319141772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc80d3eb0 0x7fefc806b890 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fefb4006fd0 tx=0x7fefb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5914d69d6a16cab server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.195+0000 7fefc6000700 1 -- 172.21.15.12:0/2319141772 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4005e30 con 0x7fefc80d3eb0 2024-03-29T11:12:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.195+0000 7fefc6000700 1 -- 172.21.15.12:0/2319141772 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb4004030 con 0x7fefc80d3eb0 2024-03-29T11:12:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.195+0000 7fefc6000700 1 -- 172.21.15.12:0/2319141772 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb40043e0 con 0x7fefc80d3eb0 2024-03-29T11:12:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.195+0000 7fefce600700 1 -- 172.21.15.12:0/2319141772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc80d3eb0 msgr2=0x7fefc806b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.195+0000 7fefce600700 1 --2- 172.21.15.12:0/2319141772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc80d3eb0 0x7fefc806b890 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fefb4006fd0 tx=0x7fefb4009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.195+0000 7fefce600700 1 -- 172.21.15.12:0/2319141772 shutdown_connections 2024-03-29T11:12:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.195+0000 7fefce600700 1 --2- 172.21.15.12:0/2319141772 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc815d8a0 0x7fefc80b2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.195+0000 7fefce600700 1 --2- 172.21.15.12:0/2319141772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc80d3eb0 0x7fefc806b890 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.195+0000 7fefce600700 1 -- 172.21.15.12:0/2319141772 >> 172.21.15.12:0/2319141772 conn(0x7fefc808aff0 msgr2=0x7fefc80d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.196+0000 7fefce600700 1 -- 172.21.15.12:0/2319141772 shutdown_connections 2024-03-29T11:12:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.196+0000 7fefce600700 1 -- 172.21.15.12:0/2319141772 wait complete. 2024-03-29T11:12:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.197+0000 7fefce600700 1 Processor -- start 2024-03-29T11:12:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.197+0000 7fefce600700 1 -- start start 2024-03-29T11:12:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.198+0000 7fefce600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc806bdd0 0x7fefc81545f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.198+0000 7fefce600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc815d8a0 0x7fefc815ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.198+0000 7fefce600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefc81593f0 0x7fefc80d1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.198+0000 7fefce600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc815b5d0 con 0x7fefc815d8a0 2024-03-29T11:12:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.198+0000 7fefce600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc81f3120 con 0x7fefc81593f0 2024-03-29T11:12:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.198+0000 7fefce600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc81f3400 con 0x7fefc806bdd0 2024-03-29T11:12:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc815d8a0 0x7fefc815ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc815d8a0 0x7fefc815ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42686/0 (socket says 172.21.15.12:42686) 2024-03-29T11:12:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc6a00700 1 -- 172.21.15.12:0/94691706 learned_addr learned my addr 172.21.15.12:0/94691706 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc7400700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc806bdd0 0x7fefc81545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc7e00700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefc81593f0 0x7fefc80d1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc6a00700 1 -- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc806bdd0 msgr2=0x7fefc81545f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc6a00700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc806bdd0 0x7fefc81545f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc6a00700 1 -- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefc81593f0 msgr2=0x7fefc80d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc6a00700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefc81593f0 0x7fefc80d1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc6a00700 1 -- 172.21.15.12:0/94691706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb4009500 con 0x7fefc815d8a0 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc7400700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc806bdd0 0x7fefc81545f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc7e00700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefc81593f0 0x7fefc80d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.199+0000 7fefc6a00700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc815d8a0 0x7fefc815ad80 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009c10 tx=0x7fefb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.200+0000 7fefbf400700 1 -- 172.21.15.12:0/94691706 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4003d90 con 0x7fefc815d8a0 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.200+0000 7fefbf400700 1 -- 172.21.15.12:0/94691706 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb4011420 con 0x7fefc815d8a0 2024-03-29T11:12:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.200+0000 7fefbf400700 1 -- 172.21.15.12:0/94691706 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb40117d0 con 0x7fefc815d8a0 2024-03-29T11:12:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.200+0000 7fefce600700 1 -- 172.21.15.12:0/94691706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc81f36e0 con 0x7fefc815d8a0 2024-03-29T11:12:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.200+0000 7fefce600700 1 -- 172.21.15.12:0/94691706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc815b2c0 con 0x7fefc815d8a0 2024-03-29T11:12:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.201+0000 7fefce600700 1 -- 172.21.15.12:0/94691706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc8067720 con 0x7fefc815d8a0 2024-03-29T11:12:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.202+0000 7fefbf400700 1 -- 172.21.15.12:0/94691706 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fefb4011930 con 0x7fefc815d8a0 2024-03-29T11:12:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.203+0000 7fefbf400700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefa806adb0 0x7fefa806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.203+0000 7fefbf400700 1 -- 172.21.15.12:0/94691706 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fefb408a7d0 con 0x7fefc815d8a0 2024-03-29T11:12:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.203+0000 7fefc7400700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefa806adb0 0x7fefa806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.204+0000 7fefc7400700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefa806adb0 0x7fefa806d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009820 tx=0x7fefb8008040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.207+0000 7fefbf400700 1 -- 172.21.15.12:0/94691706 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fefb4055860 con 0x7fefc815d8a0 2024-03-29T11:12:32.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.504+0000 7fefce600700 1 -- 172.21.15.12:0/94691706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fefc80d1950 con 0x7fefc815d8a0 2024-03-29T11:12:32.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.509+0000 7fefbf400700 1 -- 172.21.15.12:0/94691706 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v43) v1 ==== 190+0+0 (secure 0 0 0) 0x7fefb4058e80 con 0x7fefc815d8a0 2024-03-29T11:12:32.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.511+0000 7fefce600700 1 -- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefa806adb0 msgr2=0x7fefa806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.511+0000 7fefce600700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefa806adb0 0x7fefa806d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009820 tx=0x7fefb8008040 comp rx=0 tx=0).stop 2024-03-29T11:12:32.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.511+0000 7fefce600700 1 -- 172.21.15.12:0/94691706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc815d8a0 msgr2=0x7fefc815ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.511+0000 7fefce600700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc815d8a0 0x7fefc815ad80 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009c10 tx=0x7fefb4004750 comp rx=0 tx=0).stop 2024-03-29T11:12:32.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.511+0000 7fefce600700 1 -- 172.21.15.12:0/94691706 shutdown_connections 2024-03-29T11:12:32.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.511+0000 7fefce600700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fefa806adb0 0x7fefa806d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.511+0000 7fefce600700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fefc81593f0 0x7fefc80d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.511+0000 7fefce600700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fefc806bdd0 0x7fefc81545f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.511+0000 7fefce600700 1 --2- 172.21.15.12:0/94691706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefc815d8a0 0x7fefc815ad80 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.511+0000 7fefce600700 1 -- 172.21.15.12:0/94691706 >> 172.21.15.12:0/94691706 conn(0x7fefc808aff0 msgr2=0x7fefc80b6150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:32.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.512+0000 7fefce600700 1 -- 172.21.15.12:0/94691706 shutdown_connections 2024-03-29T11:12:32.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.512+0000 7fefce600700 1 -- 172.21.15.12:0/94691706 wait complete. 2024-03-29T11:12:32.491 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T11:12:32.508 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:12:32.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.727+0000 7fc56c200700 1 Processor -- start 2024-03-29T11:12:32.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.728+0000 7fc56c200700 1 -- start start 2024-03-29T11:12:32.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.729+0000 7fc56c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc56415b5d0 0x7fc5640ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.729+0000 7fc56c200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc5640ac7d0 0x7fc56406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.729+0000 7fc56c200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56415fd50 con 0x7fc5640cb3f0 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.729+0000 7fc56c200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56415b9b0 con 0x7fc5640ac7d0 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.729+0000 7fc56c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56415f140 con 0x7fc56415b5d0 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.729+0000 7fc56a200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc5640ac7d0 0x7fc56406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.729+0000 7fc56a200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc5640ac7d0 0x7fc56406c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42698/0 (socket says 172.21.15.12:42698) 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.729+0000 7fc56a200700 1 -- 172.21.15.12:0/3005768401 learned_addr learned my addr 172.21.15.12:0/3005768401 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.729+0000 7fc569800700 1 --2- 172.21.15.12:0/3005768401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc56415b5d0 0x7fc5640ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.730+0000 7fc56a200700 1 -- 172.21.15.12:0/3005768401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc56415b5d0 msgr2=0x7fc5640ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.730+0000 7fc56a200700 1 --2- 172.21.15.12:0/3005768401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc56415b5d0 0x7fc5640ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.730+0000 7fc56a200700 1 -- 172.21.15.12:0/3005768401 >> v1:172.21.15.159:6789/0 conn(0x7fc5640cb3f0 legacy=0x7fc5640ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.730+0000 7fc56a200700 1 -- 172.21.15.12:0/3005768401 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56415f420 con 0x7fc5640ac7d0 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.730+0000 7fc56a200700 1 --2- 172.21.15.12:0/3005768401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ac7d0 0x7fc56406c9d0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009e50 tx=0x7fc55c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb9cc66b74f3fdf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.730+0000 7fc563e00700 1 -- 172.21.15.12:0/3005768401 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c00e070 con 0x7fc5640ac7d0 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.730+0000 7fc563e00700 1 -- 172.21.15.12:0/3005768401 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c00b910 con 0x7fc5640ac7d0 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.730+0000 7fc563e00700 1 -- 172.21.15.12:0/3005768401 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c00bcc0 con 0x7fc5640ac7d0 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.731+0000 7fc56c200700 1 -- 172.21.15.12:0/3005768401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ac7d0 msgr2=0x7fc56406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.731+0000 7fc56c200700 1 --2- 172.21.15.12:0/3005768401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ac7d0 0x7fc56406c9d0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009e50 tx=0x7fc55c00b330 comp rx=0 tx=0).stop 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.731+0000 7fc56c200700 1 -- 172.21.15.12:0/3005768401 shutdown_connections 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.731+0000 7fc56c200700 1 --2- 172.21.15.12:0/3005768401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ac7d0 0x7fc56406c9d0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.731+0000 7fc56c200700 1 --2- 172.21.15.12:0/3005768401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc56415b5d0 0x7fc5640ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.731+0000 7fc56c200700 1 -- 172.21.15.12:0/3005768401 >> 172.21.15.12:0/3005768401 conn(0x7fc5640cf190 msgr2=0x7fc5640cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.731+0000 7fc56c200700 1 -- 172.21.15.12:0/3005768401 shutdown_connections 2024-03-29T11:12:32.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.731+0000 7fc56c200700 1 -- 172.21.15.12:0/3005768401 wait complete. 2024-03-29T11:12:32.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.733+0000 7fc56c200700 1 Processor -- start 2024-03-29T11:12:32.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.733+0000 7fc56c200700 1 -- start start 2024-03-29T11:12:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56c200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5640ac7d0 0x7fc5640b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc56415b5d0 0x7fc5640ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56c200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ce7e0 0x7fc5641485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56c200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640b2ea0 con 0x7fc5640ce7e0 2024-03-29T11:12:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56c200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56414b630 con 0x7fc5640ac7d0 2024-03-29T11:12:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640b2220 con 0x7fc56415b5d0 2024-03-29T11:12:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56a200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ce7e0 0x7fc5641485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56a200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ce7e0 0x7fc5641485f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42702/0 (socket says 172.21.15.12:42702) 2024-03-29T11:12:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56a200700 1 -- 172.21.15.12:0/1127861883 learned_addr learned my addr 172.21.15.12:0/1127861883 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc568e00700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc56415b5d0 0x7fc5640ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc569800700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5640ac7d0 0x7fc5640b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56a200700 1 -- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc56415b5d0 msgr2=0x7fc5640ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56a200700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc56415b5d0 0x7fc5640ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56a200700 1 -- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5640ac7d0 msgr2=0x7fc5640b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56a200700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5640ac7d0 0x7fc5640b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.734+0000 7fc56a200700 1 -- 172.21.15.12:0/1127861883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c009b30 con 0x7fc5640ce7e0 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.735+0000 7fc56a200700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ce7e0 0x7fc5641485f0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fc55c012000 tx=0x7fc55c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.735+0000 7fc561600700 1 -- 172.21.15.12:0/1127861883 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c004960 con 0x7fc5640ce7e0 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.735+0000 7fc561600700 1 -- 172.21.15.12:0/1127861883 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c0099b0 con 0x7fc5640ce7e0 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.735+0000 7fc561600700 1 -- 172.21.15.12:0/1127861883 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c01c680 con 0x7fc5640ce7e0 2024-03-29T11:12:32.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.735+0000 7fc56c200700 1 -- 172.21.15.12:0/1127861883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5640b2500 con 0x7fc5640ce7e0 2024-03-29T11:12:32.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.735+0000 7fc56c200700 1 -- 172.21.15.12:0/1127861883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5640b2ac0 con 0x7fc5640ce7e0 2024-03-29T11:12:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.736+0000 7fc56c200700 1 -- 172.21.15.12:0/1127861883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc564067720 con 0x7fc5640ce7e0 2024-03-29T11:12:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.737+0000 7fc561600700 1 -- 172.21.15.12:0/1127861883 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc55c01c7e0 con 0x7fc5640ce7e0 2024-03-29T11:12:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.738+0000 7fc561600700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc54406adb0 0x7fc54406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.738+0000 7fc561600700 1 -- 172.21.15.12:0/1127861883 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fc55c08afb0 con 0x7fc5640ce7e0 2024-03-29T11:12:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.738+0000 7fc569800700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc54406adb0 0x7fc54406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:32.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.739+0000 7fc569800700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc54406adb0 0x7fc54406d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc56406df90 tx=0x7fc558008040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:32.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:32.742+0000 7fc561600700 1 -- 172.21.15.12:0/1127861883 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc55c055f30 con 0x7fc5640ce7e0 2024-03-29T11:12:32.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.010+0000 7fc56c200700 1 -- 172.21.15.12:0/1127861883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc56406ea60 con 0x7fc5640ce7e0 2024-03-29T11:12:32.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.011+0000 7fc561600700 1 -- 172.21.15.12:0/1127861883 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fc55c059550 con 0x7fc5640ce7e0 2024-03-29T11:12:32.989 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:12:32.989 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2592133465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2592133465},{"type":"v1","addr":"172.21.15.159:6837","nonce":2592133465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":8381,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/204651369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":204651369},{"type":"v1","addr":"172.21.15.12:6835","nonce":204651369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9616,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2547917697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2547917697},{"type":"v1","addr":"172.21.15.159:6836","nonce":2547917697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:12:26.523825+0000","modified":"2024-03-29T11:12:30.007709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9975":{"gid":9975,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/4176455926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4176455926},{"type":"v1","addr":"172.21.15.12:6833","nonce":4176455926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-29T11:12:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 -- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc54406adb0 msgr2=0x7fc54406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc54406adb0 0x7fc54406d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc56406df90 tx=0x7fc558008040 comp rx=0 tx=0).stop 2024-03-29T11:12:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 -- 172.21.15.12:0/1127861883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ce7e0 msgr2=0x7fc5641485f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ce7e0 0x7fc5641485f0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fc55c012000 tx=0x7fc55c004ad0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 -- 172.21.15.12:0/1127861883 shutdown_connections 2024-03-29T11:12:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fc54406adb0 0x7fc54406d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5640ce7e0 0x7fc5641485f0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5640ac7d0 0x7fc5640b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 --2- 172.21.15.12:0/1127861883 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc56415b5d0 0x7fc5640ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 -- 172.21.15.12:0/1127861883 >> 172.21.15.12:0/1127861883 conn(0x7fc5640cf190 msgr2=0x7fc5640aede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 -- 172.21.15.12:0/1127861883 shutdown_connections 2024-03-29T11:12:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.014+0000 7fc56c200700 1 -- 172.21.15.12:0/1127861883 wait complete. 2024-03-29T11:12:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 89 2024-03-29T11:12:33.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T11:12:26.523825+0000', 'modified': '2024-03-29T11:12:30.007709+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9975}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9975': {'gid': 9975, 'name': 'c', 'rank': 0, 'incarnation': 85, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6833/4176455926', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6832', 'nonce': 4176455926}, {'type': 'v1', 'addr': '172.21.15.12:6833', 'nonce': 4176455926}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T11:12:33.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T11:12:33.415 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-29T11:12:33.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.637+0000 7f5651e00700 1 Processor -- start 2024-03-29T11:12:33.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.638+0000 7f5651e00700 1 -- start start 2024-03-29T11:12:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f5651e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f564c0cbb30 0x7f564c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f5651e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f564c0cd480 0x7f564c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f5651e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f564c0cc450 con 0x7f564c0cd020 2024-03-29T11:12:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f5651e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c0cd8f0 con 0x7f564c0cbb30 2024-03-29T11:12:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f5651e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c0cdae0 con 0x7f564c0cd480 2024-03-29T11:12:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f564aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f564c0cbb30 0x7f564c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f564a000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f564c0cd020 0x7f564c0b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57072/0 (socket says 172.21.15.12:57072) 2024-03-29T11:12:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f564a000700 1 -- 172.21.15.12:0/900234013 learned_addr learned my addr 172.21.15.12:0/900234013 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f564b400700 1 --2- 172.21.15.12:0/900234013 >> v2:172.21.15.159:3300/0 conn(0x7f564c0cd480 0x7f564c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f564aa00700 1 -- 172.21.15.12:0/900234013 >> v2:172.21.15.159:3300/0 conn(0x7f564c0cd480 msgr2=0x7f564c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.639+0000 7f5649600700 1 -- 172.21.15.12:0/900234013 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149824771 0 0) 0x7f564c0cc450 con 0x7f564c0cd020 2024-03-29T11:12:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.640+0000 7f564aa00700 1 --2- 172.21.15.12:0/900234013 >> v2:172.21.15.159:3300/0 conn(0x7f564c0cd480 0x7f564c06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.640+0000 7f564aa00700 1 -- 172.21.15.12:0/900234013 >> v1:172.21.15.12:6789/0 conn(0x7f564c0cd020 legacy=0x7f564c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.640+0000 7f564aa00700 1 -- 172.21.15.12:0/900234013 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c0cddc0 con 0x7f564c0cbb30 2024-03-29T11:12:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.640+0000 7f564b400700 1 --2- 172.21.15.12:0/900234013 >> v2:172.21.15.159:3300/0 conn(0x7f564c0cd480 0x7f564c06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.640+0000 7f564aa00700 1 --2- 172.21.15.12:0/900234013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 0x7f564c0cbf10 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f564c06b6d0 tx=0x7f563c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb93f7783093e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.640+0000 7f5649600700 1 -- 172.21.15.12:0/900234013 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c011830 con 0x7f564c0cbb30 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.640+0000 7f5649600700 1 -- 172.21.15.12:0/900234013 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c011990 con 0x7f564c0cbb30 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.640+0000 7f5649600700 1 -- 172.21.15.12:0/900234013 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c011d20 con 0x7f564c0cbb30 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.641+0000 7f5651e00700 1 -- 172.21.15.12:0/900234013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 msgr2=0x7f564c0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.641+0000 7f5651e00700 1 --2- 172.21.15.12:0/900234013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 0x7f564c0cbf10 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f564c06b6d0 tx=0x7f563c009b20 comp rx=0 tx=0).stop 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.641+0000 7f5651e00700 1 -- 172.21.15.12:0/900234013 shutdown_connections 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.641+0000 7f5651e00700 1 --2- 172.21.15.12:0/900234013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 0x7f564c0cbf10 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.641+0000 7f5651e00700 1 --2- 172.21.15.12:0/900234013 >> v2:172.21.15.159:3300/0 conn(0x7f564c0cd480 0x7f564c06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.641+0000 7f5651e00700 1 -- 172.21.15.12:0/900234013 >> 172.21.15.12:0/900234013 conn(0x7f564c0b0d70 msgr2=0x7f564c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.641+0000 7f5651e00700 1 -- 172.21.15.12:0/900234013 shutdown_connections 2024-03-29T11:12:33.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.641+0000 7f5651e00700 1 -- 172.21.15.12:0/900234013 wait complete. 2024-03-29T11:12:33.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.642+0000 7f5651e00700 1 Processor -- start 2024-03-29T11:12:33.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.643+0000 7f5651e00700 1 -- start start 2024-03-29T11:12:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.643+0000 7f5651e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 0x7f564c0c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f5651e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f564c0cd020 0x7f564c0c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f5651e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f564c0cd480 0x7f564c0afa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f5651e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c1562e0 con 0x7f564c0cbb30 2024-03-29T11:12:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f5651e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c072970 con 0x7f564c0cd480 2024-03-29T11:12:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f5651e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0c8930 con 0x7f564c0cd020 2024-03-29T11:12:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 0x7f564c0c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 0x7f564c0c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42728/0 (socket says 172.21.15.12:42728) 2024-03-29T11:12:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564aa00700 1 -- 172.21.15.12:0/3625354860 learned_addr learned my addr 172.21.15.12:0/3625354860 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564a000700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f564c0cd020 0x7f564c0c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564aa00700 1 -- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f564c0cd020 msgr2=0x7f564c0c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564b400700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f564c0cd480 0x7f564c0afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564aa00700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f564c0cd020 0x7f564c0c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564aa00700 1 -- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f564c0cd480 msgr2=0x7f564c0afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564aa00700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f564c0cd480 0x7f564c0afa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564aa00700 1 -- 172.21.15.12:0/3625354860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c0094a0 con 0x7f564c0cbb30 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564b400700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f564c0cd480 0x7f564c0afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564a000700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f564c0cd020 0x7f564c0c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.644+0000 7f564aa00700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 0x7f564c0c9ee0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f563c014040 tx=0x7f563c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.645+0000 7f5643400700 1 -- 172.21.15.12:0/3625354860 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c0038a0 con 0x7f564c0cbb30 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.645+0000 7f5643400700 1 -- 172.21.15.12:0/3625354860 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c01e070 con 0x7f564c0cbb30 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.645+0000 7f5651e00700 1 -- 172.21.15.12:0/3625354860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564c0723b0 con 0x7f564c0cbb30 2024-03-29T11:12:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.645+0000 7f5643400700 1 -- 172.21.15.12:0/3625354860 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c003c40 con 0x7f564c0cbb30 2024-03-29T11:12:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.645+0000 7f5651e00700 1 -- 172.21.15.12:0/3625354860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564c155f70 con 0x7f564c0cbb30 2024-03-29T11:12:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.648+0000 7f5643400700 1 -- 172.21.15.12:0/3625354860 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f563c02d450 con 0x7f564c0cbb30 2024-03-29T11:12:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.649+0000 7f5643400700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f562c06adb0 0x7f562c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.649+0000 7f5643400700 1 -- 172.21.15.12:0/3625354860 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f563c08a660 con 0x7f564c0cbb30 2024-03-29T11:12:33.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.649+0000 7f564a000700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f562c06adb0 0x7f562c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.650+0000 7f564a000700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f562c06adb0 0x7f562c06d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5638007fd0 tx=0x7f5638009040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:33.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.650+0000 7f566598bb80 1 Processor -- start 2024-03-29T11:12:33.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.650+0000 7f566598bb80 1 -- start start 2024-03-29T11:12:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.651+0000 7f566598bb80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3458ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.651+0000 7f566598bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b34560a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.651+0000 7f566598bb80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34587a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.651+0000 7f566598bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b344d5650 con 0x560b3455dbe0 2024-03-29T11:12:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.651+0000 7f566598bb80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b344b6c30 con 0x560b3455bb60 2024-03-29T11:12:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.651+0000 7f566598bb80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b344b4430 con 0x560b3458a830 2024-03-29T11:12:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b34560a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b34560a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42730/0 (socket says 172.21.15.12:42730) 2024-03-29T11:12:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564a000700 1 -- 172.21.15.12:0/1127434069 learned_addr learned my addr 172.21.15.12:0/1127434069 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564aa00700 1 --2- 172.21.15.12:0/1127434069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3458ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564b400700 1 --2- 172.21.15.12:0/1127434069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34587a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564a000700 1 -- 172.21.15.12:0/1127434069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 msgr2=0x560b3458ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564a000700 1 --2- 172.21.15.12:0/1127434069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3458ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564a000700 1 -- 172.21.15.12:0/1127434069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 msgr2=0x560b34587a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564a000700 1 --2- 172.21.15.12:0/1127434069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34587a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564a000700 1 -- 172.21.15.12:0/1127434069 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b342a3790 con 0x560b3455dbe0 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f564a000700 1 --2- 172.21.15.12:0/1127434069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b34560a10 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f5638005c80 tx=0x7f5638010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb49e19351cd30f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.652+0000 7f5637400700 1 -- 172.21.15.12:0/1127434069 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638004a70 con 0x560b3455dbe0 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f5637400700 1 -- 172.21.15.12:0/1127434069 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5638004bd0 con 0x560b3455dbe0 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f5637400700 1 -- 172.21.15.12:0/1127434069 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638005050 con 0x560b3455dbe0 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f566598bb80 1 -- 172.21.15.12:0/1127434069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 msgr2=0x560b34560a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f566598bb80 1 --2- 172.21.15.12:0/1127434069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b34560a10 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f5638005c80 tx=0x7f5638010000 comp rx=0 tx=0).stop 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f566598bb80 1 -- 172.21.15.12:0/1127434069 shutdown_connections 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f566598bb80 1 --2- 172.21.15.12:0/1127434069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34587a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f566598bb80 1 --2- 172.21.15.12:0/1127434069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3458ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f566598bb80 1 --2- 172.21.15.12:0/1127434069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b34560a10 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f566598bb80 1 -- 172.21.15.12:0/1127434069 >> 172.21.15.12:0/1127434069 conn(0x560b344be6a0 msgr2=0x560b345a5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f566598bb80 1 -- 172.21.15.12:0/1127434069 shutdown_connections 2024-03-29T11:12:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.653+0000 7f566598bb80 1 -- 172.21.15.12:0/1127434069 wait complete. 2024-03-29T11:12:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.654+0000 7f566598bb80 1 -- start start 2024-03-29T11:12:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.654+0000 7f566598bb80 1 Processor -- start 2024-03-29T11:12:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.655+0000 7f566598bb80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34577f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.655+0000 7f566598bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b345a74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.655+0000 7f566598bb80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3458ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.655+0000 7f566598bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b344afd20 con 0x560b3455dbe0 2024-03-29T11:12:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.655+0000 7f566598bb80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b3444b770 con 0x560b3455bb60 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.655+0000 7f566598bb80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b3446e6f0 con 0x560b3458a830 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.655+0000 7f566598bb80 10 client.0 _ll_register_callbacks cb 0x560b344555e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b345a74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b345a74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42746/0 (socket says 172.21.15.12:42746) 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564a000700 1 -- 172.21.15.12:0/3112115439 learned_addr learned my addr 172.21.15.12:0/3112115439 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564a000700 1 -- 172.21.15.12:0/3112115439 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 msgr2=0x560b3458ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564a000700 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3458ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564aa00700 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34577f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564a000700 1 -- 172.21.15.12:0/3112115439 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 msgr2=0x560b34577f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564a000700 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34577f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564a000700 1 -- 172.21.15.12:0/3112115439 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56380059c0 con 0x560b3455dbe0 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564aa00700 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34577f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f564a000700 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b345a74d0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f5638013000 tx=0x7f5638000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346159ecf528fe89 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.656+0000 7f562be00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f562be00700 1 -- 172.21.15.12:0/3112115439 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638028070 con 0x560b3455dbe0 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f562be00700 1 -- 172.21.15.12:0/3112115439 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f5638005670 con 0x560b3455dbe0 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f566598bb80 1 -- 172.21.15.12:0/3112115439 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560b34552920 con 0x560b3455dbe0 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f562be00700 1 -- 172.21.15.12:0/3112115439 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5638005830 con 0x560b3455dbe0 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f566598bb80 10 client.10110 fetch_fsmap learned FSMap version 89 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f566598bb80 1 -- 172.21.15.12:0/3112115439 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x560b343900a0 con 0x560b3455dbe0 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f562be00700 1 -- 172.21.15.12:0/3112115439 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560b34552920 con 0x560b3455dbe0 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f562be00700 1 -- 172.21.15.12:0/3112115439 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 89) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5638021460 con 0x560b3455dbe0 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f566598bb80 10 client.10110 fetch_fsmap finished waiting for FSMap version 89 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f566598bb80 10 client.10110 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f566598bb80 20 client.10110 populate_metadata read hostname 'smithi012' 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f566598bb80 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.12:6832/4176455926,v1:172.21.15.12:6833/4176455926] conn(0x560b3458c7c0 0x560b3458cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f566598bb80 4 client.10110 mds_command: new command op to 9975 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.657+0000 7f566598bb80 1 -- 172.21.15.12:0/3112115439 --> [v2:172.21.15.12:6832/4176455926,v1:172.21.15.12:6833/4176455926] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b3452f6f0 con 0x560b3458c7c0 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.658+0000 7f564aa00700 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.12:6832/4176455926,v1:172.21.15.12:6833/4176455926] conn(0x560b3458c7c0 0x560b3458cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.658+0000 7f564aa00700 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.12:6832/4176455926,v1:172.21.15.12:6833/4176455926] conn(0x560b3458c7c0 0x560b3458cba0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d7125ab68fc05ed server_cookie=21d4d75e31bc012c in_seq=0 out_seq=0 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.658+0000 7f562be00700 10 client.10110 ms_handle_connect on v2:172.21.15.12:6832/4176455926 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f562be00700 1 -- 172.21.15.12:0/3112115439 <== mds.0 v2:172.21.15.12:6832/4176455926 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560b3452f6f0 con 0x560b3458c7c0 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f562be00700 10 client.10110 handle_command_reply: tid=0 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 client.10110 shutdown 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 -- 172.21.15.12:0/3112115439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 msgr2=0x560b345a74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b345a74d0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f5638013000 tx=0x7f5638000ed0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 -- 172.21.15.12:0/3112115439 shutdown_connections 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.12:6832/4176455926,v1:172.21.15.12:6833/4176455926] conn(0x560b3458c7c0 0x560b3458cba0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3458ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34577f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 --2- 172.21.15.12:0/3112115439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b345a74d0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f562be00700 0 client.10110 ms_handle_reset on v2:172.21.15.12:6832/4176455926 2024-03-29T11:12:33.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 -- 172.21.15.12:0/3112115439 >> 172.21.15.12:0/3112115439 conn(0x560b344be6a0 msgr2=0x560b3455fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:33.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 -- 172.21.15.12:0/3112115439 shutdown_connections 2024-03-29T11:12:33.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 1 -- 172.21.15.12:0/3112115439 wait complete. 2024-03-29T11:12:33.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.659+0000 7f566598bb80 20 client.10110 trim_cache size 0 max 16384 2024-03-29T11:12:33.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.678+0000 7f566598bb80 1 Processor -- start 2024-03-29T11:12:33.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.678+0000 7f566598bb80 1 -- start start 2024-03-29T11:12:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.679+0000 7f566598bb80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34603250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.679+0000 7f566598bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b3455a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.679+0000 7f566598bb80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3455a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.679+0000 7f566598bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b342a4390 con 0x560b3455dbe0 2024-03-29T11:12:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.679+0000 7f566598bb80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b3458d410 con 0x560b3455bb60 2024-03-29T11:12:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.679+0000 7f566598bb80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b3455cd90 con 0x560b3458a830 2024-03-29T11:12:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b3455a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b3455a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42750/0 (socket says 172.21.15.12:42750) 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564a000700 1 -- 172.21.15.12:0/2037725112 learned_addr learned my addr 172.21.15.12:0/2037725112 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564aa00700 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34603250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564b400700 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3455a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564a000700 1 -- 172.21.15.12:0/2037725112 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 msgr2=0x560b3455a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564a000700 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3455a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564a000700 1 -- 172.21.15.12:0/2037725112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 msgr2=0x560b34603250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564a000700 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34603250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564a000700 1 -- 172.21.15.12:0/2037725112 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b34476010 con 0x560b3455dbe0 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564aa00700 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34603250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564b400700 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3455a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.680+0000 7f564a000700 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b3455a360 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f56380013b0 tx=0x7f5638028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e35bad05a216478 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f5637e00700 1 -- 172.21.15.12:0/2037725112 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638004ed0 con 0x560b3455dbe0 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f5637e00700 1 -- 172.21.15.12:0/2037725112 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5638021970 con 0x560b3455dbe0 2024-03-29T11:12:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f5637e00700 1 -- 172.21.15.12:0/2037725112 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638005670 con 0x560b3455dbe0 2024-03-29T11:12:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f566598bb80 1 -- 172.21.15.12:0/2037725112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 msgr2=0x560b3455a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f566598bb80 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b3455a360 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f56380013b0 tx=0x7f5638028000 comp rx=0 tx=0).stop 2024-03-29T11:12:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f566598bb80 1 -- 172.21.15.12:0/2037725112 shutdown_connections 2024-03-29T11:12:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f566598bb80 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3458a830 0x560b3455a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f566598bb80 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455bb60 0x560b34603250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f566598bb80 1 --2- 172.21.15.12:0/2037725112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3455dbe0 0x560b3455a360 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f566598bb80 1 -- 172.21.15.12:0/2037725112 >> 172.21.15.12:0/2037725112 conn(0x560b344be6a0 msgr2=0x560b34589630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f566598bb80 1 -- 172.21.15.12:0/2037725112 shutdown_connections 2024-03-29T11:12:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.681+0000 7f566598bb80 1 -- 172.21.15.12:0/2037725112 wait complete. 2024-03-29T11:12:33.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.682+0000 7f566598bb80 1 -- start start 2024-03-29T11:12:33.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.682+0000 7f566598bb80 1 Processor -- start 2024-03-29T11:12:33.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.684+0000 7f566598bb80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3455bb60 0x560b345a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.684+0000 7f566598bb80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455dbe0 0x560b345a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.684+0000 7f566598bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3458a830 0x560b345b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.684+0000 7f566598bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b345a3ce0 con 0x560b3458a830 2024-03-29T11:12:33.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.684+0000 7f566598bb80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b345b6d30 con 0x560b3455dbe0 2024-03-29T11:12:33.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.684+0000 7f566598bb80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b3455dfc0 con 0x560b3455bb60 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.684+0000 7f566598bb80 10 client.0 _ll_register_callbacks cb 0x7f5638005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.684+0000 7f564b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3458a830 0x560b345b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.684+0000 7f564b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3458a830 0x560b345b71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42762/0 (socket says 172.21.15.12:42762) 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.684+0000 7f564b400700 1 -- 172.21.15.12:0/3287484372 learned_addr learned my addr 172.21.15.12:0/3287484372 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f564a000700 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455dbe0 0x560b345a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f564aa00700 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3455bb60 0x560b345a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f564b400700 1 -- 172.21.15.12:0/3287484372 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3455bb60 msgr2=0x560b345a5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f564b400700 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3455bb60 0x560b345a5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f564b400700 1 -- 172.21.15.12:0/3287484372 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455dbe0 msgr2=0x560b345a31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f564b400700 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455dbe0 0x560b345a31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f564b400700 1 -- 172.21.15.12:0/3287484372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b343900a0 con 0x560b3458a830 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f564aa00700 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3455bb60 0x560b345a5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f564b400700 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3458a830 0x560b345b71c0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f564400eaa0 tx=0x7f564400ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1166f41db6adfc87 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:33.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f562be00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f562be00700 1 -- 172.21.15.12:0/3287484372 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644013070 con 0x560b3458a830 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f562be00700 1 -- 172.21.15.12:0/3287484372 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f5644019b00 con 0x560b3458a830 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.685+0000 7f566598bb80 1 -- 172.21.15.12:0/3287484372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5638004bd0 con 0x560b3458a830 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f562be00700 1 -- 172.21.15.12:0/3287484372 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56440079f0 con 0x560b3458a830 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f566598bb80 10 client.10116 fetch_fsmap learned FSMap version 89 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f566598bb80 1 -- 172.21.15.12:0/3287484372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x560b343041d0 con 0x560b3458a830 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f562be00700 1 -- 172.21.15.12:0/3287484372 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5638004bd0 con 0x560b3458a830 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f562be00700 1 -- 172.21.15.12:0/3287484372 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 89) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5644019cc0 con 0x560b3458a830 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f566598bb80 10 client.10116 fetch_fsmap finished waiting for FSMap version 89 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f566598bb80 10 client.10116 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f566598bb80 20 client.10116 populate_metadata read hostname 'smithi012' 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f566598bb80 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.12:6832/4176455926,v1:172.21.15.12:6833/4176455926] conn(0x560b34589630 0x560b34589a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f566598bb80 4 client.10116 mds_command: new command op to 9975 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T11:12:33.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f566598bb80 1 -- 172.21.15.12:0/3287484372 --> [v2:172.21.15.12:6832/4176455926,v1:172.21.15.12:6833/4176455926] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560b342e9040 con 0x560b34589630 2024-03-29T11:12:33.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.686+0000 7f564aa00700 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.12:6832/4176455926,v1:172.21.15.12:6833/4176455926] conn(0x560b34589630 0x560b34589a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f564aa00700 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.12:6832/4176455926,v1:172.21.15.12:6833/4176455926] conn(0x560b34589630 0x560b34589a10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be6b9a8d1b33e5eb server_cookie=5c979106d307e699 in_seq=0 out_seq=0 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f562be00700 10 client.10116 ms_handle_connect on v2:172.21.15.12:6832/4176455926 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f562be00700 1 -- 172.21.15.12:0/3287484372 <== mds.0 v2:172.21.15.12:6832/4176455926 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x560b342e9040 con 0x560b34589630 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f562be00700 10 client.10116 handle_command_reply: tid=0 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 client.10116 shutdown 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 -- 172.21.15.12:0/3287484372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3458a830 msgr2=0x560b345b71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3458a830 0x560b345b71c0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f564400eaa0 tx=0x7f564400ead0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 -- 172.21.15.12:0/3287484372 shutdown_connections 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3458a830 0x560b345b71c0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.12:6832/4176455926,v1:172.21.15.12:6833/4176455926] conn(0x560b34589630 0x560b34589a10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x560b3455bb60 0x560b345a5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 --2- 172.21.15.12:0/3287484372 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x560b3455dbe0 0x560b345a31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 -- 172.21.15.12:0/3287484372 >> 172.21.15.12:0/3287484372 conn(0x560b344be6a0 msgr2=0x560b345597b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:33.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f562be00700 0 client.10116 ms_handle_reset on v2:172.21.15.12:6832/4176455926 2024-03-29T11:12:33.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 -- 172.21.15.12:0/3287484372 shutdown_connections 2024-03-29T11:12:33.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.687+0000 7f566598bb80 1 -- 172.21.15.12:0/3287484372 wait complete. 2024-03-29T11:12:33.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f566598bb80 20 client.10116 trim_cache size 0 max 16384 2024-03-29T11:12:33.669 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:12:33.669 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T11:12:33.669 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T11:12:33.669 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 9975, 2024-03-29T11:12:33.669 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T11:12:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T11:12:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T11:12:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1614098159999999, 2024-03-29T11:12:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 89, 2024-03-29T11:12:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 191, 2024-03-29T11:12:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 191, 2024-03-29T11:12:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 13.080865289 2024-03-29T11:12:33.670 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:12:33.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 -- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f562c06adb0 msgr2=0x7f562c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f562c06adb0 0x7f562c06d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5638007fd0 tx=0x7f5638009040 comp rx=0 tx=0).stop 2024-03-29T11:12:33.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 -- 172.21.15.12:0/3625354860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 msgr2=0x7f564c0c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 0x7f564c0c9ee0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f563c014040 tx=0x7f563c004750 comp rx=0 tx=0).stop 2024-03-29T11:12:33.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 -- 172.21.15.12:0/3625354860 shutdown_connections 2024-03-29T11:12:33.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f562c06adb0 0x7f562c06d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f564c0cd480 0x7f564c0afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f564c0cbb30 0x7f564c0c9ee0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 --2- 172.21.15.12:0/3625354860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f564c0cd020 0x7f564c0c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 -- 172.21.15.12:0/3625354860 >> 172.21.15.12:0/3625354860 conn(0x7f564c0b0d70 msgr2=0x7f564c15d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:33.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 -- 172.21.15.12:0/3625354860 shutdown_connections 2024-03-29T11:12:33.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.688+0000 7f5651e00700 1 -- 172.21.15.12:0/3625354860 wait complete. 2024-03-29T11:12:33.686 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T11:12:33.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.907+0000 7f1aed200700 1 Processor -- start 2024-03-29T11:12:33.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.907+0000 7f1aed200700 1 -- start start 2024-03-29T11:12:33.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.908+0000 7f1aed200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1ae81619d0 0x7f1ae81685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1aed200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae80d30d0 con 0x7f1ae80ceec0 2024-03-29T11:12:33.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1aed200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae80b6840 con 0x7f1ae815d640 2024-03-29T11:12:33.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1aed200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae80d2b30 con 0x7f1ae81619d0 2024-03-29T11:12:33.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1ae5400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1ae80ceec0 0x7f1ae81612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57078/0 (socket says 172.21.15.12:57078) 2024-03-29T11:12:33.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1ae5400700 1 -- 172.21.15.12:0/859517782 learned_addr learned my addr 172.21.15.12:0/859517782 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:33.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1ae6800700 1 --2- 172.21.15.12:0/859517782 >> v2:172.21.15.159:3300/0 conn(0x7f1ae81619d0 0x7f1ae81685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1ae4a00700 1 -- 172.21.15.12:0/859517782 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762876770 0 0) 0x7f1ae80d30d0 con 0x7f1ae80ceec0 2024-03-29T11:12:33.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1ae4a00700 1 -- 172.21.15.12:0/859517782 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1acc0035b0 con 0x7f1ae80ceec0 2024-03-29T11:12:33.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1ae6800700 1 -- 172.21.15.12:0/859517782 >> v1:172.21.15.159:6789/0 conn(0x7f1ae815d640 legacy=0x7f1ae80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1ae6800700 1 -- 172.21.15.12:0/859517782 >> v1:172.21.15.12:6789/0 conn(0x7f1ae80ceec0 legacy=0x7f1ae81612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.909+0000 7f1ae6800700 1 -- 172.21.15.12:0/859517782 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae80d2df0 con 0x7f1ae81619d0 2024-03-29T11:12:33.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.910+0000 7f1ae6800700 1 --2- 172.21.15.12:0/859517782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae81619d0 0x7f1ae81685f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009d90 tx=0x7f1ae000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49157d8e1f77469c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:33.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.910+0000 7f1ae4a00700 1 -- 172.21.15.12:0/859517782 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae000e070 con 0x7f1ae81619d0 2024-03-29T11:12:33.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.910+0000 7f1ae4a00700 1 -- 172.21.15.12:0/859517782 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae000bb20 con 0x7f1ae81619d0 2024-03-29T11:12:33.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.910+0000 7f1ae4a00700 1 -- 172.21.15.12:0/859517782 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0012430 con 0x7f1ae81619d0 2024-03-29T11:12:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.911+0000 7f1aed200700 1 -- 172.21.15.12:0/859517782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae81619d0 msgr2=0x7f1ae81685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.911+0000 7f1aed200700 1 --2- 172.21.15.12:0/859517782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae81619d0 0x7f1ae81685f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009d90 tx=0x7f1ae000b330 comp rx=0 tx=0).stop 2024-03-29T11:12:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.911+0000 7f1aed200700 1 -- 172.21.15.12:0/859517782 shutdown_connections 2024-03-29T11:12:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.911+0000 7f1aed200700 1 --2- 172.21.15.12:0/859517782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae81619d0 0x7f1ae81685f0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.911+0000 7f1aed200700 1 -- 172.21.15.12:0/859517782 >> 172.21.15.12:0/859517782 conn(0x7f1ae808aff0 msgr2=0x7f1ae80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.911+0000 7f1aed200700 1 -- 172.21.15.12:0/859517782 shutdown_connections 2024-03-29T11:12:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.911+0000 7f1aed200700 1 -- 172.21.15.12:0/859517782 wait complete. 2024-03-29T11:12:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.912+0000 7f1aed200700 1 Processor -- start 2024-03-29T11:12:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.913+0000 7f1aed200700 1 -- start start 2024-03-29T11:12:33.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.913+0000 7f1aed200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80ceec0 0x7f1ae80ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.913+0000 7f1aed200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae815d640 0x7f1ae815c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.913+0000 7f1aed200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae81619d0 0x7f1ae80cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.913+0000 7f1aed200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80cbef0 con 0x7f1ae81619d0 2024-03-29T11:12:33.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1aed200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80cdc20 con 0x7f1ae815d640 2024-03-29T11:12:33.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1aed200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80cbc10 con 0x7f1ae80ceec0 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae81619d0 0x7f1ae80cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae81619d0 0x7f1ae80cb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42764/0 (socket says 172.21.15.12:42764) 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae6800700 1 -- 172.21.15.12:0/2767702239 learned_addr learned my addr 172.21.15.12:0/2767702239 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae5e00700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80ceec0 0x7f1ae80ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae5400700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae815d640 0x7f1ae815c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae6800700 1 -- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80ceec0 msgr2=0x7f1ae80ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae6800700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80ceec0 0x7f1ae80ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae6800700 1 -- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae815d640 msgr2=0x7f1ae815c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae6800700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae815d640 0x7f1ae815c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae6800700 1 -- 172.21.15.12:0/2767702239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae0009a70 con 0x7f1ae81619d0 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae5e00700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80ceec0 0x7f1ae80ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.914+0000 7f1ae5400700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae815d640 0x7f1ae815c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.915+0000 7f1ae6800700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae81619d0 0x7f1ae80cb250 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f1ae808c490 tx=0x7f1ae0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.915+0000 7f1ade000700 1 -- 172.21.15.12:0/2767702239 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0012430 con 0x7f1ae81619d0 2024-03-29T11:12:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.915+0000 7f1ade000700 1 -- 172.21.15.12:0/2767702239 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae0020070 con 0x7f1ae81619d0 2024-03-29T11:12:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.915+0000 7f1ade000700 1 -- 172.21.15.12:0/2767702239 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0004280 con 0x7f1ae81619d0 2024-03-29T11:12:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.915+0000 7f1aed200700 1 -- 172.21.15.12:0/2767702239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae80cd5d0 con 0x7f1ae81619d0 2024-03-29T11:12:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.915+0000 7f1aed200700 1 -- 172.21.15.12:0/2767702239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae80cd2c0 con 0x7f1ae81619d0 2024-03-29T11:12:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.916+0000 7f1aed200700 1 -- 172.21.15.12:0/2767702239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae80cddb0 con 0x7f1ae81619d0 2024-03-29T11:12:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.917+0000 7f1ade000700 1 -- 172.21.15.12:0/2767702239 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1ae00043e0 con 0x7f1ae81619d0 2024-03-29T11:12:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.918+0000 7f1ade000700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ac806adb0 0x7f1ac806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.918+0000 7f1ade000700 1 -- 172.21.15.12:0/2767702239 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f1ae0022070 con 0x7f1ae81619d0 2024-03-29T11:12:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.918+0000 7f1ae5e00700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ac806adb0 0x7f1ac806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.919+0000 7f1ae5e00700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ac806adb0 0x7f1ac806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8000f60 tx=0x7f1ad8009040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:33.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:33.922+0000 7f1ade000700 1 -- 172.21.15.12:0/2767702239 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ae00562e0 con 0x7f1ae81619d0 2024-03-29T11:12:34.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.187+0000 7f1aed200700 1 -- 172.21.15.12:0/2767702239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1ae8067720 con 0x7f1ae81619d0 2024-03-29T11:12:34.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.187+0000 7f1ade000700 1 -- 172.21.15.12:0/2767702239 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v89) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1ae0059900 con 0x7f1ae81619d0 2024-03-29T11:12:34.165 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.189+0000 7f1aed200700 1 -- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ac806adb0 msgr2=0x7f1ac806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.189+0000 7f1aed200700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ac806adb0 0x7f1ac806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8000f60 tx=0x7f1ad8009040 comp rx=0 tx=0).stop 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.189+0000 7f1aed200700 1 -- 172.21.15.12:0/2767702239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae81619d0 msgr2=0x7f1ae80cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.189+0000 7f1aed200700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae81619d0 0x7f1ae80cb250 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f1ae808c490 tx=0x7f1ae0004ad0 comp rx=0 tx=0).stop 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.190+0000 7f1aed200700 1 -- 172.21.15.12:0/2767702239 shutdown_connections 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.190+0000 7f1aed200700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f1ac806adb0 0x7f1ac806d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.190+0000 7f1aed200700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae81619d0 0x7f1ae80cb250 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.190+0000 7f1aed200700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80ceec0 0x7f1ae80ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.190+0000 7f1aed200700 1 --2- 172.21.15.12:0/2767702239 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae815d640 0x7f1ae815c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.190+0000 7f1aed200700 1 -- 172.21.15.12:0/2767702239 >> 172.21.15.12:0/2767702239 conn(0x7f1ae808aff0 msgr2=0x7f1ae806ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.190+0000 7f1aed200700 1 -- 172.21.15.12:0/2767702239 shutdown_connections 2024-03-29T11:12:34.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:34.190+0000 7f1aed200700 1 -- 172.21.15.12:0/2767702239 wait complete. 2024-03-29T11:12:34.186 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T11:12:34.186 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T11:12:34.186 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T11:12:34.186 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T11:12:34.187 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T11:12:34.187 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T11:12:34.187 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T11:12:34.187 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T11:12:34.187 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:12:34.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:34.187 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T11:12:34.233 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T11:12:34.233 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T11:12:34.233 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T11:12:34.233 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:12:34.233 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T11:12:34.234 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 36763sec preferred_lft 36763sec 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T11:12:34.235 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:12:34.236 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:12:34.236 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T11:12:34.236 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T11:12:34.236 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:12:34.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:34.236 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T11:12:34.338 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T11:12:34.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:34.342 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T11:12:34.389 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T11:12:34.389 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T11:12:34.389 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T11:12:34.389 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T11:12:34.389 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T11:12:34.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:34.394 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T11:12:34.394 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T11:12:34.394 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T11:12:34.394 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T11:12:34.394 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T11:12:34.395 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T11:12:34.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:34.561 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T11:12:34.607 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T11:12:34.607 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T11:12:34.607 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:12:34.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:34.608 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T11:12:34.716 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:12:34.716 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T11:12:34.760 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:12:34.760 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:12:34.760 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:12:34.844 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T11:12:34.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:34.845 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:12:34.987 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:12:34.987 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:12:35.018 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:12:35.018 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:12:35.018 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T11:12:35.018 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T11:12:35.018 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T11:12:35.022 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T11:12:35.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:35.022 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:12:35.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:35.090 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T11:12:35.192 INFO:teuthology.orchestra.run.smithi012.stdout:client8462 2024-03-29T11:12:35.193 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:12:35.193 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:12:35.243 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T11:12:35.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.504+0000 7f20cd400700 1 Processor -- start 2024-03-29T11:12:35.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.504+0000 7f20cd400700 1 -- start start 2024-03-29T11:12:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20cd400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f20c80cbb30 0x7f20c80cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20cd400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f20c80cd020 0x7f20c80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20cd400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 0x7f20c80b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20cd400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c80cc450 con 0x7f20c80cd020 2024-03-29T11:12:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20cd400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c80cdae0 con 0x7f20c80cbb30 2024-03-29T11:12:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20cd400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c80cddc0 con 0x7f20c80b2070 2024-03-29T11:12:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f20c80cd020 0x7f20c80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f20c80cd020 0x7f20c80cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42776/0 (socket says 172.21.15.12:42776) 2024-03-29T11:12:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c5e00700 1 -- 172.21.15.12:0/1176641313 learned_addr learned my addr 172.21.15.12:0/1176641313 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c6800700 1 --2- 172.21.15.12:0/1176641313 >> v2:172.21.15.159:3300/0 conn(0x7f20c80cbb30 0x7f20c80cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c7200700 1 --2- 172.21.15.12:0/1176641313 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 0x7f20c80b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c5e00700 1 -- 172.21.15.12:0/1176641313 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 msgr2=0x7f20c80b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c5e00700 1 --2- 172.21.15.12:0/1176641313 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 0x7f20c80b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c5e00700 1 -- 172.21.15.12:0/1176641313 >> v2:172.21.15.159:3300/0 conn(0x7f20c80cbb30 msgr2=0x7f20c80cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c5e00700 1 --2- 172.21.15.12:0/1176641313 >> v2:172.21.15.159:3300/0 conn(0x7f20c80cbb30 0x7f20c80cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c5e00700 1 -- 172.21.15.12:0/1176641313 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c80ce0a0 con 0x7f20c80cd020 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.506+0000 7f20c7200700 1 --2- 172.21.15.12:0/1176641313 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 0x7f20c80b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20c5e00700 1 --2- 172.21.15.12:0/1176641313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cd020 0x7f20c80cd480 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f20c806fd00 tx=0x7f20b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91c5f20c6e4d1bb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20c5400700 1 -- 172.21.15.12:0/1176641313 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b400b870 con 0x7f20c80cd020 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20c5400700 1 -- 172.21.15.12:0/1176641313 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b400b9d0 con 0x7f20c80cd020 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20c5400700 1 -- 172.21.15.12:0/1176641313 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b400bd60 con 0x7f20c80cd020 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20cd400700 1 -- 172.21.15.12:0/1176641313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cd020 msgr2=0x7f20c80cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20cd400700 1 --2- 172.21.15.12:0/1176641313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cd020 0x7f20c80cd480 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f20c806fd00 tx=0x7f20b4009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20cd400700 1 -- 172.21.15.12:0/1176641313 shutdown_connections 2024-03-29T11:12:35.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20cd400700 1 --2- 172.21.15.12:0/1176641313 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 0x7f20c80b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20cd400700 1 --2- 172.21.15.12:0/1176641313 >> v2:172.21.15.159:3300/0 conn(0x7f20c80cbb30 0x7f20c80cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20cd400700 1 --2- 172.21.15.12:0/1176641313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cd020 0x7f20c80cd480 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.507+0000 7f20cd400700 1 -- 172.21.15.12:0/1176641313 >> 172.21.15.12:0/1176641313 conn(0x7f20c80b0d70 msgr2=0x7f20c80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.508+0000 7f20cd400700 1 -- 172.21.15.12:0/1176641313 shutdown_connections 2024-03-29T11:12:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.508+0000 7f20cd400700 1 -- 172.21.15.12:0/1176641313 wait complete. 2024-03-29T11:12:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.510+0000 7f20cd400700 1 Processor -- start 2024-03-29T11:12:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.510+0000 7f20cd400700 1 -- start start 2024-03-29T11:12:35.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.511+0000 7f20cd400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 0x7f20c80ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.511+0000 7f20cd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cbb30 0x7f20c81484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.511+0000 7f20cd400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20c80cd020 0x7f20c81f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.511+0000 7f20cd400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c80ad8a0 con 0x7f20c80cbb30 2024-03-29T11:12:35.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.511+0000 7f20cd400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c8153a50 con 0x7f20c80cd020 2024-03-29T11:12:35.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.511+0000 7f20cd400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c80cae30 con 0x7f20c80b2070 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cbb30 0x7f20c81484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cbb30 0x7f20c81484c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42784/0 (socket says 172.21.15.12:42784) 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c5e00700 1 -- 172.21.15.12:0/3472274986 learned_addr learned my addr 172.21.15.12:0/3472274986 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c6800700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 0x7f20c80ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c7200700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20c80cd020 0x7f20c81f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c5e00700 1 -- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 msgr2=0x7f20c80ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c5e00700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 0x7f20c80ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c5e00700 1 -- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20c80cd020 msgr2=0x7f20c81f3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c5e00700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20c80cd020 0x7f20c81f3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c5e00700 1 -- 172.21.15.12:0/3472274986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b4009550 con 0x7f20c80cbb30 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c7200700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20c80cd020 0x7f20c81f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.512+0000 7f20c5e00700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cbb30 0x7f20c81484c0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009c10 tx=0x7f20b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.513+0000 7f20bea00700 1 -- 172.21.15.12:0/3472274986 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4004590 con 0x7f20c80cbb30 2024-03-29T11:12:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.513+0000 7f20bea00700 1 -- 172.21.15.12:0/3472274986 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b40036a0 con 0x7f20c80cbb30 2024-03-29T11:12:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.513+0000 7f20bea00700 1 -- 172.21.15.12:0/3472274986 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4003a50 con 0x7f20c80cbb30 2024-03-29T11:12:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.513+0000 7f20cd400700 1 -- 172.21.15.12:0/3472274986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c8148a00 con 0x7f20c80cbb30 2024-03-29T11:12:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.513+0000 7f20cd400700 1 -- 172.21.15.12:0/3472274986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c80acf70 con 0x7f20c80cbb30 2024-03-29T11:12:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.514+0000 7f20cd400700 1 -- 172.21.15.12:0/3472274986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c80ada10 con 0x7f20c80cbb30 2024-03-29T11:12:35.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.515+0000 7f20bea00700 1 -- 172.21.15.12:0/3472274986 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f20b4003bb0 con 0x7f20c80cbb30 2024-03-29T11:12:35.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.516+0000 7f20bea00700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f20a806adb0 0x7f20a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.516+0000 7f20bea00700 1 -- 172.21.15.12:0/3472274986 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f20b408a890 con 0x7f20c80cbb30 2024-03-29T11:12:35.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.516+0000 7f20c6800700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f20a806adb0 0x7f20a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.517+0000 7f20c6800700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f20a806adb0 0x7f20a806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f20b8006b20 tx=0x7f20b8008040 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:35.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.521+0000 7f20bea00700 1 -- 172.21.15.12:0/3472274986 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f20b4055ae0 con 0x7f20c80cbb30 2024-03-29T11:12:35.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.766+0000 7f20cd400700 1 -- 172.21.15.12:0/3472274986 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f20c81594a0 con 0x7f20a806adb0 2024-03-29T11:12:35.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.768+0000 7f20bea00700 1 -- 172.21.15.12:0/3472274986 <== mgr.8216 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f20b8007630 con 0x7f20a806adb0 2024-03-29T11:12:35.746 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:12:35.746 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:12:35.746 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T11:12:35.747 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:12:35.747 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:12:35.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 -- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f20a806adb0 msgr2=0x7f20a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f20a806adb0 0x7f20a806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f20b8006b20 tx=0x7f20b8008040 comp rx=0 tx=0).stop 2024-03-29T11:12:35.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 -- 172.21.15.12:0/3472274986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cbb30 msgr2=0x7f20c81484c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cbb30 0x7f20c81484c0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009c10 tx=0x7f20b4004750 comp rx=0 tx=0).stop 2024-03-29T11:12:35.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 -- 172.21.15.12:0/3472274986 shutdown_connections 2024-03-29T11:12:35.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f20a806adb0 0x7f20a806d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20c80cd020 0x7f20c81f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20c80b2070 0x7f20c80ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 --2- 172.21.15.12:0/3472274986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20c80cbb30 0x7f20c81484c0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 -- 172.21.15.12:0/3472274986 >> 172.21.15.12:0/3472274986 conn(0x7f20c80b0d70 msgr2=0x7f20c815b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:35.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 -- 172.21.15.12:0/3472274986 shutdown_connections 2024-03-29T11:12:35.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.771+0000 7f20cd400700 1 -- 172.21.15.12:0/3472274986 wait complete. 2024-03-29T11:12:35.767 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T11:12:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.990+0000 7ff0a5a00700 1 Processor -- start 2024-03-29T11:12:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.990+0000 7ff0a5a00700 1 -- start start 2024-03-29T11:12:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.991+0000 7ff0a5a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00d3eb0 0x7ff0a00b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.991+0000 7ff0a5a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff0a00cf1b0 0x7ff0a00cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.992+0000 7ff0a5a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a00ced20 con 0x7ff0a015d8a0 2024-03-29T11:12:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.992+0000 7ff0a5a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a00d4340 con 0x7ff0a00cf1b0 2024-03-29T11:12:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.992+0000 7ff0a5a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00ce6d0 con 0x7ff0a00d3eb0 2024-03-29T11:12:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.992+0000 7ff09e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00d3eb0 0x7ff0a00b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.992+0000 7ff09f400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff0a00cf1b0 0x7ff0a00cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.992+0000 7ff09e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00d3eb0 0x7ff0a00b2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:35086/0 (socket says 172.21.15.12:35086) 2024-03-29T11:12:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.992+0000 7ff09e000700 1 -- 172.21.15.12:0/2857533675 learned_addr learned my addr 172.21.15.12:0/2857533675 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.992+0000 7ff09ea00700 1 --1- 172.21.15.12:0/2857533675 >> v1:172.21.15.159:6789/0 conn(0x7ff0a015d8a0 0x7ff0a006b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:41076/0 (socket says 172.21.15.12:41076) 2024-03-29T11:12:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.993+0000 7ff09e000700 1 -- 172.21.15.12:0/2857533675 >> v2:172.21.15.159:3300/0 conn(0x7ff0a00cf1b0 msgr2=0x7ff0a00cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.993+0000 7ff09e000700 1 --2- 172.21.15.12:0/2857533675 >> v2:172.21.15.159:3300/0 conn(0x7ff0a00cf1b0 0x7ff0a00cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.993+0000 7ff09e000700 1 -- 172.21.15.12:0/2857533675 >> v1:172.21.15.159:6789/0 conn(0x7ff0a015d8a0 legacy=0x7ff0a006b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.993+0000 7ff09e000700 1 -- 172.21.15.12:0/2857533675 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a00ce9b0 con 0x7ff0a00d3eb0 2024-03-29T11:12:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.993+0000 7ff09f400700 1 --2- 172.21.15.12:0/2857533675 >> v2:172.21.15.159:3300/0 conn(0x7ff0a00cf1b0 0x7ff0a00cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.993+0000 7ff09e000700 1 --2- 172.21.15.12:0/2857533675 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00d3eb0 0x7ff0a00b2990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff08c006fd0 tx=0x7ff08c009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c89a102fd2d45a7c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.993+0000 7ff09d600700 1 -- 172.21.15.12:0/2857533675 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c005e30 con 0x7ff0a00d3eb0 2024-03-29T11:12:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.993+0000 7ff09d600700 1 -- 172.21.15.12:0/2857533675 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c004030 con 0x7ff0a00d3eb0 2024-03-29T11:12:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.993+0000 7ff09d600700 1 -- 172.21.15.12:0/2857533675 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c0043e0 con 0x7ff0a00d3eb0 2024-03-29T11:12:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.994+0000 7ff0a5a00700 1 -- 172.21.15.12:0/2857533675 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00d3eb0 msgr2=0x7ff0a00b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.994+0000 7ff0a5a00700 1 --2- 172.21.15.12:0/2857533675 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00d3eb0 0x7ff0a00b2990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff08c006fd0 tx=0x7ff08c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.994+0000 7ff0a5a00700 1 -- 172.21.15.12:0/2857533675 shutdown_connections 2024-03-29T11:12:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.994+0000 7ff0a5a00700 1 --2- 172.21.15.12:0/2857533675 >> v2:172.21.15.159:3300/0 conn(0x7ff0a00cf1b0 0x7ff0a00cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.994+0000 7ff0a5a00700 1 --2- 172.21.15.12:0/2857533675 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00d3eb0 0x7ff0a00b2990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.994+0000 7ff0a5a00700 1 -- 172.21.15.12:0/2857533675 >> 172.21.15.12:0/2857533675 conn(0x7ff0a008aff0 msgr2=0x7ff0a00d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.994+0000 7ff0a5a00700 1 -- 172.21.15.12:0/2857533675 shutdown_connections 2024-03-29T11:12:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.994+0000 7ff0a5a00700 1 -- 172.21.15.12:0/2857533675 wait complete. 2024-03-29T11:12:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.996+0000 7ff0a5a00700 1 Processor -- start 2024-03-29T11:12:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.996+0000 7ff0a5a00700 1 -- start start 2024-03-29T11:12:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff0a5a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00cf1b0 0x7ff0a00c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff0a5a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0a015d8a0 0x7ff0a01597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff0a5a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0a00a0950 0x7ff0a00b8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff0a5a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00cd210 con 0x7ff0a015d8a0 2024-03-29T11:12:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff0a5a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a0159d10 con 0x7ff0a00a0950 2024-03-29T11:12:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff0a5a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00cc600 con 0x7ff0a00cf1b0 2024-03-29T11:12:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff09e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0a015d8a0 0x7ff0a01597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff09e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0a015d8a0 0x7ff0a01597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42790/0 (socket says 172.21.15.12:42790) 2024-03-29T11:12:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff09e000700 1 -- 172.21.15.12:0/4111822160 learned_addr learned my addr 172.21.15.12:0/4111822160 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff09ea00700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00cf1b0 0x7ff0a00c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.997+0000 7ff09f400700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0a00a0950 0x7ff0a00b8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff09e000700 1 -- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00cf1b0 msgr2=0x7ff0a00c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff09e000700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00cf1b0 0x7ff0a00c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff09e000700 1 -- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0a00a0950 msgr2=0x7ff0a00b8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff09e000700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0a00a0950 0x7ff0a00b8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff09e000700 1 -- 172.21.15.12:0/4111822160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff098009a70 con 0x7ff0a015d8a0 2024-03-29T11:12:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff09e000700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0a015d8a0 0x7ff0a01597d0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7ff08c004000 tx=0x7ff08c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff096a00700 1 -- 172.21.15.12:0/4111822160 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c003d80 con 0x7ff0a015d8a0 2024-03-29T11:12:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff096a00700 1 -- 172.21.15.12:0/4111822160 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c01c420 con 0x7ff0a015d8a0 2024-03-29T11:12:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff0a5a00700 1 -- 172.21.15.12:0/4111822160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c009500 con 0x7ff0a015d8a0 2024-03-29T11:12:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff096a00700 1 -- 172.21.15.12:0/4111822160 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c01c7d0 con 0x7ff0a015d8a0 2024-03-29T11:12:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.998+0000 7ff0a5a00700 1 -- 172.21.15.12:0/4111822160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a00ccbc0 con 0x7ff0a015d8a0 2024-03-29T11:12:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:35.999+0000 7ff0a5a00700 1 -- 172.21.15.12:0/4111822160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a00ca6f0 con 0x7ff0a015d8a0 2024-03-29T11:12:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:36.000+0000 7ff096a00700 1 -- 172.21.15.12:0/4111822160 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 99) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff08c01c930 con 0x7ff0a015d8a0 2024-03-29T11:12:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:36.001+0000 7ff096a00700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff08006adb0 0x7ff08006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:35.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:36.001+0000 7ff096a00700 1 -- 172.21.15.12:0/4111822160 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7ff08c08a400 con 0x7ff0a015d8a0 2024-03-29T11:12:35.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:36.002+0000 7ff09ea00700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff08006adb0 0x7ff08006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:35.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:36.003+0000 7ff09ea00700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff08006adb0 0x7ff08006d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff090000f60 tx=0x7ff090004a50 comp rx=0 tx=0).ready entity=mgr.8216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:35.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:36.006+0000 7ff096a00700 1 -- 172.21.15.12:0/4111822160 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff0a00ca6f0 con 0x7ff0a015d8a0 2024-03-29T11:12:36.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:36.282+0000 7ff0a5a00700 1 -- 172.21.15.12:0/4111822160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ff0a00ca6f0 con 0x7ff0a015d8a0 2024-03-29T11:12:37.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.037+0000 7ff096a00700 1 -- 172.21.15.12:0/4111822160 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 100) v1 ==== 89256+0+0 (secure 0 0 0) 0x7ff08c055860 con 0x7ff0a015d8a0 2024-03-29T11:12:37.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.038+0000 7ff096a00700 1 -- 172.21.15.12:0/4111822160 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v100) v1 ==== 92+0+0 (secure 0 0 0) 0x7ff08c06d680 con 0x7ff0a015d8a0 2024-03-29T11:12:37.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 -- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff08006adb0 msgr2=0x7ff08006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff08006adb0 0x7ff08006d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff090000f60 tx=0x7ff090004a50 comp rx=0 tx=0).stop 2024-03-29T11:12:37.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 -- 172.21.15.12:0/4111822160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0a015d8a0 msgr2=0x7ff0a01597d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0a015d8a0 0x7ff0a01597d0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7ff08c004000 tx=0x7ff08c004750 comp rx=0 tx=0).stop 2024-03-29T11:12:37.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 -- 172.21.15.12:0/4111822160 shutdown_connections 2024-03-29T11:12:37.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7ff08006adb0 0x7ff08006d270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0a00a0950 0x7ff0a00b8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0a00cf1b0 0x7ff0a00c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 --2- 172.21.15.12:0/4111822160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0a015d8a0 0x7ff0a01597d0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 -- 172.21.15.12:0/4111822160 >> 172.21.15.12:0/4111822160 conn(0x7ff0a008aff0 msgr2=0x7ff0a00d04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:37.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 -- 172.21.15.12:0/4111822160 shutdown_connections 2024-03-29T11:12:37.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.041+0000 7ff0a5a00700 1 -- 172.21.15.12:0/4111822160 wait complete. 2024-03-29T11:12:37.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T11:12:37.239 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:37.261+0000 7fb646f1f240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:12:37.248 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:37.271+0000 7fb422ec0240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:12:37.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.278+0000 7fb88a800700 1 Processor -- start 2024-03-29T11:12:37.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.279+0000 7fb88a800700 1 -- start start 2024-03-29T11:12:37.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.280+0000 7fb88a800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb8840d2540 0x7fb8841687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:37.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.280+0000 7fb88a800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb884161420 con 0x7fb8840d16a0 2024-03-29T11:12:37.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.280+0000 7fb88a800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb884161700 con 0x7fb88415d9d0 2024-03-29T11:12:37.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.280+0000 7fb88a800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8840d4340 con 0x7fb8840d2540 2024-03-29T11:12:37.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.280+0000 7fb882a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb8840d16a0 0x7fb8840d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57100/0 (socket says 172.21.15.12:57100) 2024-03-29T11:12:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.280+0000 7fb882a00700 1 -- 172.21.15.12:0/3314346388 learned_addr learned my addr 172.21.15.12:0/3314346388 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.280+0000 7fb883e00700 1 --2- 172.21.15.12:0/3314346388 >> v2:172.21.15.159:3300/0 conn(0x7fb8840d2540 0x7fb8841687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.280+0000 7fb882000700 1 -- 172.21.15.12:0/3314346388 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599462476 0 0) 0x7fb884161420 con 0x7fb8840d16a0 2024-03-29T11:12:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb882000700 1 -- 172.21.15.12:0/3314346388 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8680035b0 con 0x7fb8840d16a0 2024-03-29T11:12:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb882000700 1 -- 172.21.15.12:0/3314346388 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369590652 0 0) 0x7fb884161700 con 0x7fb88415d9d0 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb882000700 1 -- 172.21.15.12:0/3314346388 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb884161420 con 0x7fb88415d9d0 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb882000700 1 -- 172.21.15.12:0/3314346388 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2718322349 0 0) 0x7fb8680035b0 con 0x7fb8840d16a0 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb883e00700 1 -- 172.21.15.12:0/3314346388 >> v1:172.21.15.159:6789/0 conn(0x7fb88415d9d0 legacy=0x7fb8840d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb883e00700 1 -- 172.21.15.12:0/3314346388 >> v1:172.21.15.12:6789/0 conn(0x7fb8840d16a0 legacy=0x7fb8840d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb883e00700 1 -- 172.21.15.12:0/3314346388 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8841619e0 con 0x7fb8840d2540 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb883e00700 1 --2- 172.21.15.12:0/3314346388 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8840d2540 0x7fb8841687b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009d90 tx=0x7fb87c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9ef239e1e5d333b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb882000700 1 -- 172.21.15.12:0/3314346388 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c00e070 con 0x7fb8840d2540 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb882000700 1 -- 172.21.15.12:0/3314346388 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c003a20 con 0x7fb8840d2540 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.281+0000 7fb882000700 1 -- 172.21.15.12:0/3314346388 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8700017f0 con 0x7fb8840d2540 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.282+0000 7fb88a800700 1 -- 172.21.15.12:0/3314346388 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8840d2540 msgr2=0x7fb8841687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.282+0000 7fb88a800700 1 --2- 172.21.15.12:0/3314346388 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8840d2540 0x7fb8841687b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009d90 tx=0x7fb87c00b330 comp rx=0 tx=0).stop 2024-03-29T11:12:37.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.282+0000 7fb88a800700 1 -- 172.21.15.12:0/3314346388 shutdown_connections 2024-03-29T11:12:37.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.282+0000 7fb88a800700 1 --2- 172.21.15.12:0/3314346388 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8840d2540 0x7fb8841687b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.282+0000 7fb88a800700 1 -- 172.21.15.12:0/3314346388 >> 172.21.15.12:0/3314346388 conn(0x7fb88408aff0 msgr2=0x7fb8840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:37.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.282+0000 7fb88a800700 1 -- 172.21.15.12:0/3314346388 shutdown_connections 2024-03-29T11:12:37.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.282+0000 7fb88a800700 1 -- 172.21.15.12:0/3314346388 wait complete. 2024-03-29T11:12:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.284+0000 7fb88a800700 1 Processor -- start 2024-03-29T11:12:37.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.284+0000 7fb88a800700 1 -- start start 2024-03-29T11:12:37.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb88a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8840d16a0 0x7fb884152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:37.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb88a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8840d2540 0x7fb8840b8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:37.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb88a800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb88415d9d0 0x7fb8840cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:37.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb88a800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb884152f40 con 0x7fb8840d16a0 2024-03-29T11:12:37.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb88a800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8840b8df0 con 0x7fb88415d9d0 2024-03-29T11:12:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb88a800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8840ca5f0 con 0x7fb8840d2540 2024-03-29T11:12:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb883400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8840d16a0 0x7fb884152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb883400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8840d16a0 0x7fb884152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42812/0 (socket says 172.21.15.12:42812) 2024-03-29T11:12:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb883400700 1 -- 172.21.15.12:0/2370539872 learned_addr learned my addr 172.21.15.12:0/2370539872 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:37.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb883e00700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb88415d9d0 0x7fb8840cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:37.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.285+0000 7fb882a00700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8840d2540 0x7fb8840b8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:37.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb883400700 1 -- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8840d2540 msgr2=0x7fb8840b8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb883400700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8840d2540 0x7fb8840b8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb883400700 1 -- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb88415d9d0 msgr2=0x7fb8840cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb883400700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb88415d9d0 0x7fb8840cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb883400700 1 -- 172.21.15.12:0/2370539872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c009a70 con 0x7fb8840d16a0 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb883400700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8840d16a0 0x7fb884152a00 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fb87400c500 tx=0x7fb87400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb87b400700 1 -- 172.21.15.12:0/2370539872 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8740052d0 con 0x7fb8840d16a0 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb87b400700 1 -- 172.21.15.12:0/2370539872 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb874005450 con 0x7fb8840d16a0 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb87b400700 1 -- 172.21.15.12:0/2370539872 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8740038c0 con 0x7fb8840d16a0 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb88a800700 1 -- 172.21.15.12:0/2370539872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8840ca8d0 con 0x7fb8840d16a0 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.286+0000 7fb88a800700 1 -- 172.21.15.12:0/2370539872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8840cae90 con 0x7fb8840d16a0 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.288+0000 7fb88a800700 1 -- 172.21.15.12:0/2370539872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8840cb0b0 con 0x7fb8840d16a0 2024-03-29T11:12:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.288+0000 7fb87b400700 1 -- 172.21.15.12:0/2370539872 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 100) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fb874003a20 con 0x7fb8840d16a0 2024-03-29T11:12:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.289+0000 7fb87b400700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb86406ae00 0x7fb86406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.289+0000 7fb87b400700 1 -- 172.21.15.12:0/2370539872 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fb874088840 con 0x7fb8840d16a0 2024-03-29T11:12:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.289+0000 7fb882a00700 1 -- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb86406ae00 msgr2=0x7fb86406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.289+0000 7fb882a00700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb86406ae00 0x7fb86406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.293+0000 7fb87b400700 1 -- 172.21.15.12:0/2370539872 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb874053970 con 0x7fb8840d16a0 2024-03-29T11:12:37.427 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:37.448+0000 7fb646f1f240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:12:37.453 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:37.475+0000 7fb422ec0240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:12:37.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.490+0000 7fb882a00700 1 -- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb86406ae00 msgr2=0x7fb86406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:37.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.490+0000 7fb882a00700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb86406ae00 0x7fb86406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:37.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.547+0000 7fb88a800700 1 -- 172.21.15.12:0/2370539872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fb884067720 con 0x7fb8840d16a0 2024-03-29T11:12:37.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.551+0000 7fb87b400700 1 -- 172.21.15.12:0/2370539872 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb8740170d0 con 0x7fb8840d16a0 2024-03-29T11:12:37.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 -- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb86406ae00 msgr2=0x7fb86406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:37.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb86406ae00 0x7fb86406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 -- 172.21.15.12:0/2370539872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8840d16a0 msgr2=0x7fb884152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8840d16a0 0x7fb884152a00 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fb87400c500 tx=0x7fb87400c810 comp rx=0 tx=0).stop 2024-03-29T11:12:37.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 -- 172.21.15.12:0/2370539872 shutdown_connections 2024-03-29T11:12:37.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fb86406ae00 0x7fb86406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb88415d9d0 0x7fb8840cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8840d16a0 0x7fb884152a00 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 --2- 172.21.15.12:0/2370539872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8840d2540 0x7fb8840b8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 -- 172.21.15.12:0/2370539872 >> 172.21.15.12:0/2370539872 conn(0x7fb88408aff0 msgr2=0x7fb8840b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:37.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 -- 172.21.15.12:0/2370539872 shutdown_connections 2024-03-29T11:12:37.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.554+0000 7fb88a800700 1 -- 172.21.15.12:0/2370539872 wait complete. 2024-03-29T11:12:37.550 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T11:12:37.749 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:37.771+0000 7fb646f1f240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:12:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.777+0000 7eff85800700 1 Processor -- start 2024-03-29T11:12:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.777+0000 7eff85800700 1 -- start start 2024-03-29T11:12:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.778+0000 7eff85800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7eff8015dab0 0x7eff800b0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.778+0000 7eff85800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff80162100 con 0x7eff8015d6d0 2024-03-29T11:12:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.778+0000 7eff85800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff800ce960 con 0x7eff800d40f0 2024-03-29T11:12:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.778+0000 7eff85800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff800b2d20 con 0x7eff8015dab0 2024-03-29T11:12:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.779+0000 7eff7ea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7eff8015d6d0 0x7eff80161430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57104/0 (socket says 172.21.15.12:57104) 2024-03-29T11:12:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.779+0000 7eff7ea00700 1 -- 172.21.15.12:0/4194811086 learned_addr learned my addr 172.21.15.12:0/4194811086 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.779+0000 7eff7f400700 1 --2- 172.21.15.12:0/4194811086 >> v2:172.21.15.159:3300/0 conn(0x7eff8015dab0 0x7eff800b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.779+0000 7eff7d600700 1 -- 172.21.15.12:0/4194811086 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666854655 0 0) 0x7eff80162100 con 0x7eff8015d6d0 2024-03-29T11:12:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.779+0000 7eff7d600700 1 -- 172.21.15.12:0/4194811086 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff640035b0 con 0x7eff8015d6d0 2024-03-29T11:12:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.779+0000 7eff7d600700 1 -- 172.21.15.12:0/4194811086 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075547726 0 0) 0x7eff800ce960 con 0x7eff800d40f0 2024-03-29T11:12:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.779+0000 7eff7d600700 1 -- 172.21.15.12:0/4194811086 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff80162100 con 0x7eff800d40f0 2024-03-29T11:12:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.779+0000 7eff7f400700 1 -- 172.21.15.12:0/4194811086 >> v1:172.21.15.159:6789/0 conn(0x7eff800d40f0 legacy=0x7eff800b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.779+0000 7eff7f400700 1 -- 172.21.15.12:0/4194811086 >> v1:172.21.15.12:6789/0 conn(0x7eff8015d6d0 legacy=0x7eff80161430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.779+0000 7eff7f400700 1 -- 172.21.15.12:0/4194811086 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff800cec40 con 0x7eff8015dab0 2024-03-29T11:12:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.780+0000 7eff7f400700 1 --2- 172.21.15.12:0/4194811086 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff8015dab0 0x7eff800b0830 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7eff78009d90 tx=0x7eff7800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1db375846b0325eb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.780+0000 7eff7d600700 1 -- 172.21.15.12:0/4194811086 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7800e070 con 0x7eff8015dab0 2024-03-29T11:12:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.780+0000 7eff7d600700 1 -- 172.21.15.12:0/4194811086 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7800bae0 con 0x7eff8015dab0 2024-03-29T11:12:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.780+0000 7eff7d600700 1 -- 172.21.15.12:0/4194811086 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7800be90 con 0x7eff8015dab0 2024-03-29T11:12:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.780+0000 7eff85800700 1 -- 172.21.15.12:0/4194811086 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff8015dab0 msgr2=0x7eff800b0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.780+0000 7eff85800700 1 --2- 172.21.15.12:0/4194811086 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff8015dab0 0x7eff800b0830 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7eff78009d90 tx=0x7eff7800b330 comp rx=0 tx=0).stop 2024-03-29T11:12:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.780+0000 7eff85800700 1 -- 172.21.15.12:0/4194811086 shutdown_connections 2024-03-29T11:12:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.780+0000 7eff85800700 1 --2- 172.21.15.12:0/4194811086 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff8015dab0 0x7eff800b0830 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.780+0000 7eff85800700 1 -- 172.21.15.12:0/4194811086 >> 172.21.15.12:0/4194811086 conn(0x7eff8008aff0 msgr2=0x7eff800d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.781+0000 7eff85800700 1 -- 172.21.15.12:0/4194811086 shutdown_connections 2024-03-29T11:12:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.781+0000 7eff85800700 1 -- 172.21.15.12:0/4194811086 wait complete. 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.782+0000 7eff85800700 1 Processor -- start 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.782+0000 7eff85800700 1 -- start start 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.783+0000 7eff85800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800d40f0 0x7eff801f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.783+0000 7eff85800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff8015d6d0 0x7eff800b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.783+0000 7eff85800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8015dab0 0x7eff800b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.783+0000 7eff85800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff801623e0 con 0x7eff8015dab0 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.783+0000 7eff85800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff801f7a00 con 0x7eff8015d6d0 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.783+0000 7eff85800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff801f7160 con 0x7eff800d40f0 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8015dab0 0x7eff800b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8015dab0 0x7eff800b0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42816/0 (socket says 172.21.15.12:42816) 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7f400700 1 -- 172.21.15.12:0/1970340457 learned_addr learned my addr 172.21.15.12:0/1970340457 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7e000700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff8015d6d0 0x7eff800b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7ea00700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800d40f0 0x7eff801f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7f400700 1 -- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800d40f0 msgr2=0x7eff801f6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7f400700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800d40f0 0x7eff801f6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7f400700 1 -- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff8015d6d0 msgr2=0x7eff800b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7f400700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff8015d6d0 0x7eff800b0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7f400700 1 -- 172.21.15.12:0/1970340457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff78009a70 con 0x7eff8015dab0 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.784+0000 7eff7f400700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8015dab0 0x7eff800b0c50 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7eff78004000 tx=0x7eff78004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.785+0000 7eff76a00700 1 -- 172.21.15.12:0/1970340457 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff78004540 con 0x7eff8015dab0 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.785+0000 7eff76a00700 1 -- 172.21.15.12:0/1970340457 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff780046a0 con 0x7eff8015dab0 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.785+0000 7eff76a00700 1 -- 172.21.15.12:0/1970340457 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff78012430 con 0x7eff8015dab0 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.785+0000 7eff85800700 1 -- 172.21.15.12:0/1970340457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff801f7440 con 0x7eff8015dab0 2024-03-29T11:12:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.785+0000 7eff85800700 1 -- 172.21.15.12:0/1970340457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff801f4760 con 0x7eff8015dab0 2024-03-29T11:12:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.786+0000 7eff85800700 1 -- 172.21.15.12:0/1970340457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff80067720 con 0x7eff8015dab0 2024-03-29T11:12:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.787+0000 7eff76a00700 1 -- 172.21.15.12:0/1970340457 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 100) v1 ==== 89256+0+0 (secure 0 0 0) 0x7eff78012590 con 0x7eff8015dab0 2024-03-29T11:12:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.788+0000 7eff76a00700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae00 0x7eff6006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.788+0000 7eff76a00700 1 -- 172.21.15.12:0/1970340457 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7eff7808b1f0 con 0x7eff8015dab0 2024-03-29T11:12:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.788+0000 7eff7ea00700 1 -- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae00 msgr2=0x7eff6006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.788+0000 7eff7ea00700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae00 0x7eff6006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.793+0000 7eff76a00700 1 -- 172.21.15.12:0/1970340457 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff78059c00 con 0x7eff8015dab0 2024-03-29T11:12:37.801 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:37.823+0000 7fb422ec0240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:12:37.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.989+0000 7eff7ea00700 1 -- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae00 msgr2=0x7eff6006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:37.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:37.989+0000 7eff7ea00700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae00 0x7eff6006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:37.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.011+0000 7eff85800700 1 -- 172.21.15.12:0/1970340457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7eff800cdc70 con 0x7eff8015dab0 2024-03-29T11:12:37.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.015+0000 7eff76a00700 1 -- 172.21.15.12:0/1970340457 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7eff78059790 con 0x7eff8015dab0 2024-03-29T11:12:37.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 -- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae00 msgr2=0x7eff6006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:37.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae00 0x7eff6006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 -- 172.21.15.12:0/1970340457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8015dab0 msgr2=0x7eff800b0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:37.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8015dab0 0x7eff800b0c50 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7eff78004000 tx=0x7eff78004ad0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 -- 172.21.15.12:0/1970340457 shutdown_connections 2024-03-29T11:12:37.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7eff6006ae00 0x7eff6006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8015dab0 0x7eff800b0c50 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800d40f0 0x7eff801f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 --2- 172.21.15.12:0/1970340457 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff8015d6d0 0x7eff800b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:37.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 -- 172.21.15.12:0/1970340457 >> 172.21.15.12:0/1970340457 conn(0x7eff8008aff0 msgr2=0x7eff8015abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:37.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 -- 172.21.15.12:0/1970340457 shutdown_connections 2024-03-29T11:12:37.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.018+0000 7eff85800700 1 -- 172.21.15.12:0/1970340457 wait complete. 2024-03-29T11:12:38.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:38.016 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-29T11:12:38.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:12:38.066 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2024-03-29T11:12:38.114 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2024-03-29T11:12:38.285 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:38.306+0000 7fb646f1f240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:12:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.347+0000 7f6b8da00700 1 Processor -- start 2024-03-29T11:12:38.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.347+0000 7f6b8da00700 1 -- start start 2024-03-29T11:12:38.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.348+0000 7f6b8da00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6b8815a5c0 0x7f6b880709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:38.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.348+0000 7f6b8da00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b880ce840 0x7f6b880cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:38.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.348+0000 7f6b8da00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b880cb310 con 0x7f6b8815dfe0 2024-03-29T11:12:38.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.348+0000 7f6b8da00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b88070f00 con 0x7f6b8815a5c0 2024-03-29T11:12:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.348+0000 7f6b8da00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b880711a0 con 0x7f6b880ce840 2024-03-29T11:12:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.348+0000 7f6b86000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6b8815dfe0 0x7f6b880ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57112/0 (socket says 172.21.15.12:57112) 2024-03-29T11:12:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.348+0000 7f6b86000700 1 -- 172.21.15.12:0/3253510111 learned_addr learned my addr 172.21.15.12:0/3253510111 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.348+0000 7f6b86a00700 1 --2- 172.21.15.12:0/3253510111 >> v2:172.21.15.12:3300/0 conn(0x7f6b8815a5c0 0x7f6b880709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.348+0000 7f6b87400700 1 --2- 172.21.15.12:0/3253510111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b880ce840 0x7f6b880cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.349+0000 7f6b86a00700 1 -- 172.21.15.12:0/3253510111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b880ce840 msgr2=0x7f6b880cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.349+0000 7f6b86a00700 1 --2- 172.21.15.12:0/3253510111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b880ce840 0x7f6b880cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.349+0000 7f6b86a00700 1 -- 172.21.15.12:0/3253510111 >> v1:172.21.15.12:6789/0 conn(0x7f6b8815dfe0 legacy=0x7f6b880ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.349+0000 7f6b86a00700 1 -- 172.21.15.12:0/3253510111 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b88071480 con 0x7f6b8815a5c0 2024-03-29T11:12:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.349+0000 7f6b87400700 1 --2- 172.21.15.12:0/3253510111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b880ce840 0x7f6b880cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:12:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.349+0000 7f6b85600700 1 -- 172.21.15.12:0/3253510111 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254994536 0 0) 0x7f6b880cb310 con 0x7f6b8815dfe0 2024-03-29T11:12:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.349+0000 7f6b86a00700 1 --2- 172.21.15.12:0/3253510111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b8815a5c0 0x7f6b880709c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009840 tx=0x7f6b78009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f7dac43699e1b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.349+0000 7f6b85600700 1 -- 172.21.15.12:0/3253510111 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78011830 con 0x7f6b8815a5c0 2024-03-29T11:12:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.349+0000 7f6b85600700 1 -- 172.21.15.12:0/3253510111 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b78011990 con 0x7f6b8815a5c0 2024-03-29T11:12:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.350+0000 7f6b85600700 1 -- 172.21.15.12:0/3253510111 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78011d20 con 0x7f6b8815a5c0 2024-03-29T11:12:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.350+0000 7f6b8da00700 1 -- 172.21.15.12:0/3253510111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b8815a5c0 msgr2=0x7f6b880709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.350+0000 7f6b8da00700 1 --2- 172.21.15.12:0/3253510111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b8815a5c0 0x7f6b880709c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009840 tx=0x7f6b78009b50 comp rx=0 tx=0).stop 2024-03-29T11:12:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.350+0000 7f6b8da00700 1 -- 172.21.15.12:0/3253510111 shutdown_connections 2024-03-29T11:12:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.350+0000 7f6b8da00700 1 --2- 172.21.15.12:0/3253510111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b880ce840 0x7f6b880cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.350+0000 7f6b8da00700 1 --2- 172.21.15.12:0/3253510111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b8815a5c0 0x7f6b880709c0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.350+0000 7f6b8da00700 1 -- 172.21.15.12:0/3253510111 >> 172.21.15.12:0/3253510111 conn(0x7f6b880b0d70 msgr2=0x7f6b880b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.350+0000 7f6b8da00700 1 -- 172.21.15.12:0/3253510111 shutdown_connections 2024-03-29T11:12:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.350+0000 7f6b8da00700 1 -- 172.21.15.12:0/3253510111 wait complete. 2024-03-29T11:12:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.352+0000 7f6b8da00700 1 Processor -- start 2024-03-29T11:12:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.352+0000 7f6b8da00700 1 -- start start 2024-03-29T11:12:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.353+0000 7f6b8da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b880ce840 0x7f6b880af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.353+0000 7f6b8da00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b8815a5c0 0x7f6b880b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.353+0000 7f6b8da00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b8815dfe0 0x7f6b8815aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.353+0000 7f6b8da00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8814ca00 con 0x7f6b880ce840 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.353+0000 7f6b8da00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b880b07f0 con 0x7f6b8815a5c0 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.353+0000 7f6b8da00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b880b0a90 con 0x7f6b8815dfe0 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.353+0000 7f6b86a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b880ce840 0x7f6b880af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.353+0000 7f6b86a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b880ce840 0x7f6b880af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42844/0 (socket says 172.21.15.12:42844) 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.353+0000 7f6b86a00700 1 -- 172.21.15.12:0/757790053 learned_addr learned my addr 172.21.15.12:0/757790053 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b87400700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b8815dfe0 0x7f6b8815aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b86000700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b8815a5c0 0x7f6b880b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b86a00700 1 -- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b8815dfe0 msgr2=0x7f6b8815aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b86a00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b8815dfe0 0x7f6b8815aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b86a00700 1 -- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b8815a5c0 msgr2=0x7f6b880b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b86a00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b8815a5c0 0x7f6b880b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b86a00700 1 -- 172.21.15.12:0/757790053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b80009a70 con 0x7f6b880ce840 2024-03-29T11:12:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b86000700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b8815a5c0 0x7f6b880b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:12:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b86a00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b880ce840 0x7f6b880af290 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009fd0 tx=0x7f6b78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b780036a0 con 0x7f6b880ce840 2024-03-29T11:12:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7801f070 con 0x7f6b880ce840 2024-03-29T11:12:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78003a40 con 0x7f6b880ce840 2024-03-29T11:12:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.354+0000 7f6b8da00700 1 -- 172.21.15.12:0/757790053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b780094a0 con 0x7f6b880ce840 2024-03-29T11:12:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.355+0000 7f6b8da00700 1 -- 172.21.15.12:0/757790053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8814c340 con 0x7f6b880ce840 2024-03-29T11:12:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.356+0000 7f6b8da00700 1 -- 172.21.15.12:0/757790053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b88067720 con 0x7f6b880ce840 2024-03-29T11:12:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.357+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 100) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f6b78003ba0 con 0x7f6b880ce840 2024-03-29T11:12:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.358+0000 7f6b7ea00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.358+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f6b78021070 con 0x7f6b880ce840 2024-03-29T11:12:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.358+0000 7f6b86000700 1 -- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 msgr2=0x7f6b6806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.358+0000 7f6b86000700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:12:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.362+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6b78055e80 con 0x7f6b880ce840 2024-03-29T11:12:38.369 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:38.391+0000 7fb422ec0240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:12:38.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.559+0000 7f6b86000700 1 -- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 msgr2=0x7f6b6806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:38.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.559+0000 7f6b86000700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:12:38.542 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:38.564+0000 7fb646f1f240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:12:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.612+0000 7f6b8da00700 1 -- 172.21.15.12:0/757790053 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b880cb530 con 0x7f6b6806ae00 2024-03-29T11:12:38.636 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:38.658+0000 7fb422ec0240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:12:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.960+0000 7f6b86000700 1 -- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 msgr2=0x7f6b6806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:38.960+0000 7f6b86000700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T11:12:39.002 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:39.024+0000 7fb646f1f240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:12:39.116 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:39.138+0000 7fb422ec0240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:12:39.716 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:39.738+0000 7fb646f1f240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:12:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:39.761+0000 7f6b86000700 1 -- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 msgr2=0x7f6b6806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:39.761+0000 7f6b86000700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T11:12:39.807 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:39.829+0000 7fb646f1f240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:12:39.861 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:39.882+0000 7fb422ec0240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:12:39.957 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:39.979+0000 7fb422ec0240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:12:39.967 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:39.989+0000 7fb646f1f240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:12:40.126 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:40.148+0000 7fb422ec0240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:12:40.181 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:40.203+0000 7fb646f1f240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:12:40.274 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:40.296+0000 7fb646f1f240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:12:40.345 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:40.368+0000 7fb422ec0240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:12:40.374 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:40.396+0000 7fb646f1f240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:12:40.441 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:40.463+0000 7fb422ec0240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:12:40.473 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:40.495+0000 7fb646f1f240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:12:40.545 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:40.566+0000 7fb422ec0240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:12:40.570 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:40.592+0000 7fb646f1f240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:12:40.665 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:40.687+0000 7fb422ec0240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:12:40.785 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:40.807+0000 7fb646f1f240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:12:40.796 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:40.818+0000 7fb422ec0240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:12:41.019 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:41.041+0000 7fb422ec0240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:12:41.044 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:41.065+0000 7fb646f1f240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:12:41.234 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:41.256+0000 7fb646f1f240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:12:41.282 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:41.305+0000 7fb422ec0240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:12:41.328 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:41.350+0000 7fb646f1f240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:41.363+0000 7f6b86000700 1 -- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 msgr2=0x7f6b6806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/43444 2024-03-29T11:12:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:41.363+0000 7f6b86000700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T11:12:41.482 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:41.504+0000 7fb422ec0240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:12:41.578 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:41.600+0000 7fb422ec0240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:42.065 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:42.087+0000 7fb646f1f240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:12:42.277 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:42.299+0000 7fb646f1f240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:12:42.343 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:42.366+0000 7fb422ec0240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:12:42.487 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:42.509+0000 7fb646f1f240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:42.568 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:42.590+0000 7fb422ec0240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:12:42.786 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:42.808+0000 7fb422ec0240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:42.969 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:42.991+0000 7fb646f1f240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:12:43.177 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.199+0000 7fb646f1f240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:43.285 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:43.308+0000 7fb422ec0240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:12:43.503 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:43.525+0000 7fb422ec0240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:43.619 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.641+0000 7fb646f1f240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:12:43.732 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.754+0000 7fb646f1f240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:43.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:43.765+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 101) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f6b78056250 con 0x7f6b880ce840 2024-03-29T11:12:43.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:43.766+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 msgr2=0x7f6b6806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:43.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:43.766+0000 7f6b7ea00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:43.917 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.939+0000 7fb5b3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:43.918 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.939+0000 7fb5b3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:43.918 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.939+0000 7fb5b3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:43.918 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.939+0000 7fb5b3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:43.918 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.939+0000 7fb5b3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:43.918 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.939+0000 7fb5af600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:43.918 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.939+0000 7fb5af600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:43.918 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.939+0000 7fb5af600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:43.918 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.939+0000 7fb5af600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:43.918 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:12:43.939+0000 7fb5af600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:12:43.967 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:43.989+0000 7fb422ec0240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:12:44.084 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:12:44.107+0000 7fb422ec0240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:44.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.771+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 102) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f6b78089240 con 0x7f6b880ce840 2024-03-29T11:12:44.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.771+0000 7f6b7ea00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:44.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.771+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b880cb530 con 0x7f6b6806ae00 2024-03-29T11:12:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.772+0000 7f6b86000700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:44.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.774+0000 7f6b86000700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6b74007fd0 tx=0x7f6b74005880 comp rx=0 tx=0).ready entity=mgr.9772 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:44.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.854+0000 7f6b7ea00700 1 -- 172.21.15.12:0/757790053 <== mgr.9772 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6b880cb530 con 0x7f6b6806ae00 2024-03-29T11:12:44.833 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /snap_test_dir1/test_restart 2024-03-29T11:12:44.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 -- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 msgr2=0x7f6b6806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:44.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6b74007fd0 tx=0x7f6b74005880 comp rx=0 tx=0).stop 2024-03-29T11:12:44.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 -- 172.21.15.12:0/757790053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b880ce840 msgr2=0x7f6b880af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:44.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b880ce840 0x7f6b880af290 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009fd0 tx=0x7f6b78004750 comp rx=0 tx=0).stop 2024-03-29T11:12:44.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 -- 172.21.15.12:0/757790053 shutdown_connections 2024-03-29T11:12:44.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f6b6806ae00 0x7f6b6806d2c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:44.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b8815dfe0 0x7f6b8815aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:44.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b880ce840 0x7f6b880af290 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:44.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 --2- 172.21.15.12:0/757790053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b8815a5c0 0x7f6b880b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:44.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 -- 172.21.15.12:0/757790053 >> 172.21.15.12:0/757790053 conn(0x7f6b880b0d70 msgr2=0x7f6b880b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:44.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 -- 172.21.15.12:0/757790053 shutdown_connections 2024-03-29T11:12:44.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:44.857+0000 7f6b8da00700 1 -- 172.21.15.12:0/757790053 wait complete. 2024-03-29T11:12:44.857 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2024-03-29T11:12:44.858 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2024-03-29T11:12:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.086+0000 7f84b7c00700 1 Processor -- start 2024-03-29T11:12:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.086+0000 7f84b7c00700 1 -- start start 2024-03-29T11:12:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.087+0000 7f84b7c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f84b0160cc0 0x7f84b0168730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.087+0000 7f84b7c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b015f820 con 0x7f84b015dad0 2024-03-29T11:12:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.087+0000 7f84b7c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b00d2450 con 0x7f84b00ced80 2024-03-29T11:12:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.087+0000 7f84b7c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b00cfcc0 con 0x7f84b0160cc0 2024-03-29T11:12:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.087+0000 7f84b5200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f84b015dad0 0x7f84b00d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52166/0 (socket says 172.21.15.12:52166) 2024-03-29T11:12:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.087+0000 7f84b5200700 1 -- 172.21.15.12:0/3828228782 learned_addr learned my addr 172.21.15.12:0/3828228782 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:12:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.087+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524130792 0 0) 0x7f84b015f820 con 0x7f84b015dad0 2024-03-29T11:12:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84980035b0 con 0x7f84b015dad0 2024-03-29T11:12:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84b5c00700 1 --2- 172.21.15.12:0/3828228782 >> v2:172.21.15.159:3300/0 conn(0x7f84b0160cc0 0x7f84b0168730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3799428277 0 0) 0x7f84980035b0 con 0x7f84b015dad0 2024-03-29T11:12:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 >> v2:172.21.15.159:3300/0 conn(0x7f84b0160cc0 msgr2=0x7f84b0168730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 --2- 172.21.15.12:0/3828228782 >> v2:172.21.15.159:3300/0 conn(0x7f84b0160cc0 0x7f84b0168730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 >> v1:172.21.15.159:6789/0 conn(0x7f84b00ced80 legacy=0x7f84b00d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00cf420 con 0x7f84b015dad0 2024-03-29T11:12:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f84a4001670 con 0x7f84b015dad0 2024-03-29T11:12:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 >> v1:172.21.15.12:6789/0 conn(0x7f84b015dad0 legacy=0x7f84b00d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8498003a50 0x7f8498003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f8498004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8498005260 0x7f849800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00cfcc0 con 0x7f8498004850 2024-03-29T11:12:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849800c1b0 con 0x7f8498003a50 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849800c370 con 0x7f8498005260 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84b5200700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8498003a50 0x7f8498003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84afe00700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f8498004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.088+0000 7f84b5c00700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8498005260 0x7f849800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.089+0000 7f84afe00700 1 -- 172.21.15.12:0/3828228782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8498005260 msgr2=0x7f849800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.089+0000 7f84afe00700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8498005260 0x7f849800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.089+0000 7f84afe00700 1 -- 172.21.15.12:0/3828228782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8498003a50 msgr2=0x7f8498003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.089+0000 7f84afe00700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8498003a50 0x7f8498003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.089+0000 7f84b5c00700 1 -- 172.21.15.12:0/3828228782 reap_dead start 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.089+0000 7f84afe00700 1 -- 172.21.15.12:0/3828228782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84980035b0 con 0x7f8498004850 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.089+0000 7f84afe00700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f8498004cd0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f849c007fd0 tx=0x7f8498005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d74dab6afb4f8e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:45.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.089+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849c005220 con 0x7f8498004850 2024-03-29T11:12:45.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.089+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849c0053a0 con 0x7f8498004850 2024-03-29T11:12:45.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.089+0000 7f84af400700 1 -- 172.21.15.12:0/3828228782 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849c006aa0 con 0x7f8498004850 2024-03-29T11:12:45.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.090+0000 7f84b7c00700 1 -- 172.21.15.12:0/3828228782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 msgr2=0x7f8498004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:45.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.090+0000 7f84b7c00700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f8498004cd0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f849c007fd0 tx=0x7f8498005880 comp rx=0 tx=0).stop 2024-03-29T11:12:45.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.090+0000 7f84b7c00700 1 -- 172.21.15.12:0/3828228782 shutdown_connections 2024-03-29T11:12:45.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.090+0000 7f84b7c00700 1 --2- 172.21.15.12:0/3828228782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f8498004cd0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:45.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.090+0000 7f84b7c00700 1 -- 172.21.15.12:0/3828228782 >> 172.21.15.12:0/3828228782 conn(0x7f84b008aff0 msgr2=0x7f84b00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:45.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.090+0000 7f84b7c00700 1 -- 172.21.15.12:0/3828228782 shutdown_connections 2024-03-29T11:12:45.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.090+0000 7f84b7c00700 1 -- 172.21.15.12:0/3828228782 wait complete. 2024-03-29T11:12:45.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.092+0000 7f84b7c00700 1 Processor -- start 2024-03-29T11:12:45.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.092+0000 7f84b7c00700 1 -- start start 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84b7c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f84b0159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84b7c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84b00b8650 0x7f84b00b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84b7c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84b00b6ca0 0x7f84b00cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84b7c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00b6880 con 0x7f8498004850 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84b7c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00b9180 con 0x7f84b00b8650 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84b7c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00b8bc0 con 0x7f84b00b6ca0 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84b5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f84b0159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84b5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f84b0159180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41246/0 (socket says 172.21.15.12:41246) 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84b5200700 1 -- 172.21.15.12:0/4134614244 learned_addr learned my addr 172.21.15.12:0/4134614244 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84afe00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84b00b8650 0x7f84b00b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.093+0000 7f84b5c00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84b00b6ca0 0x7f84b00cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84b5200700 1 -- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84b00b6ca0 msgr2=0x7f84b00cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84b5200700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84b00b6ca0 0x7f84b00cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:45.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84b5200700 1 -- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84b00b8650 msgr2=0x7f84b00b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84b5200700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84b00b8650 0x7f84b00b7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84b5200700 1 -- 172.21.15.12:0/4134614244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00b8ea0 con 0x7f8498004850 2024-03-29T11:12:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84b5200700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f84b0159180 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f84a4000c00 tx=0x7f84a4007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84acc00700 1 -- 172.21.15.12:0/4134614244 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4011580 con 0x7f8498004850 2024-03-29T11:12:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84acc00700 1 -- 172.21.15.12:0/4134614244 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a40116e0 con 0x7f8498004850 2024-03-29T11:12:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84acc00700 1 -- 172.21.15.12:0/4134614244 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4011a90 con 0x7f8498004850 2024-03-29T11:12:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84b7c00700 1 -- 172.21.15.12:0/4134614244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b00b64a0 con 0x7f8498004850 2024-03-29T11:12:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.094+0000 7f84b7c00700 1 -- 172.21.15.12:0/4134614244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b00b8270 con 0x7f8498004850 2024-03-29T11:12:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.095+0000 7f84b7c00700 1 -- 172.21.15.12:0/4134614244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b00b69c0 con 0x7f8498004850 2024-03-29T11:12:45.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.096+0000 7f84acc00700 1 -- 172.21.15.12:0/4134614244 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 102) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f84a4011bf0 con 0x7f8498004850 2024-03-29T11:12:45.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.097+0000 7f84acc00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f849006ab30 0x7f849006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:12:45.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.097+0000 7f84acc00700 1 -- 172.21.15.12:0/4134614244 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f84a40891d0 con 0x7f8498004850 2024-03-29T11:12:45.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.098+0000 7f84afe00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f849006ab30 0x7f849006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:12:45.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.098+0000 7f84afe00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f849006ab30 0x7f849006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f849c000c00 tx=0x7f849c009030 comp rx=0 tx=0).ready entity=mgr.9772 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:12:45.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.102+0000 7f84acc00700 1 -- 172.21.15.12:0/4134614244 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f84b00b69c0 con 0x7f8498004850 2024-03-29T11:12:45.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.344+0000 7f84b7c00700 1 -- 172.21.15.12:0/4134614244 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f84b00cf9e0 con 0x7f849006ab30 2024-03-29T11:12:45.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.404+0000 7f84acc00700 1 -- 172.21.15.12:0/4134614244 <== mgr.9772 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f849c0050c0 con 0x7f849006ab30 2024-03-29T11:12:45.382 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path /snap_test_dir1/test_restart 2024-03-29T11:12:45.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 -- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f849006ab30 msgr2=0x7f849006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:45.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f849006ab30 0x7f849006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f849c000c00 tx=0x7f849c009030 comp rx=0 tx=0).stop 2024-03-29T11:12:45.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 -- 172.21.15.12:0/4134614244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 msgr2=0x7f84b0159180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:45.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f84b0159180 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f84a4000c00 tx=0x7f84a4007570 comp rx=0 tx=0).stop 2024-03-29T11:12:45.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 -- 172.21.15.12:0/4134614244 shutdown_connections 2024-03-29T11:12:45.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f849006ab30 0x7f849006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:45.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84b00b6ca0 0x7f84b00cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:45.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8498004850 0x7f84b0159180 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:45.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 --2- 172.21.15.12:0/4134614244 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84b00b8650 0x7f84b00b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:12:45.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 -- 172.21.15.12:0/4134614244 >> 172.21.15.12:0/4134614244 conn(0x7f84b008aff0 msgr2=0x7f84b0159cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:45.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.406+0000 7f84b7c00700 1 -- 172.21.15.12:0/4134614244 shutdown_connections 2024-03-29T11:12:45.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:12:45.407+0000 7f84b7c00700 1 -- 172.21.15.12:0/4134614244 wait complete. 2024-03-29T11:12:45.403 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2024-03-29T11:12:45.564 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:12:45.568 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:13:00.395 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:13:00.417+0000 7fb59de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:13:00.395 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:13:00.417+0000 7fb59de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:13:00.395 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:13:00.417+0000 7fb59de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:13:00.395 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:13:00.417+0000 7fb59de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:13:00.395 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:13:00.417+0000 7fb59de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:13:15.674 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:13:15.681 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:13:45.787 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:13:45.793 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:14:15.914 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:14:15.924 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:14:46.045 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:14:46.050 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:15:16.282 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:15:16.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:15:46.396 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:15:46.401 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:16:16.592 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:16:16.598 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:16:46.706 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:16:46.712 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:17:16.870 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:17:16.876 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:17:47.001 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:17:47.007 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:18:17.144 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:18:17.150 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:18:47.255 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:18:47.261 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:19:17.383 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:19:17.388 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:19:47.516 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:19:47.522 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:20:17.702 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:20:17.707 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:20:47.823 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:20:47.829 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:21:17.946 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:21:17.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:21:48.059 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:21:48.065 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:22:18.265 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:22:18.270 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:22:48.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:22:48.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:23:16.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:23:16.405 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-03-29T11:23:16.457 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_13_00_UTC 2024-03-29T11:23:16.457 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_14_00_UTC 2024-03-29T11:23:16.457 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_15_00_UTC 2024-03-29T11:23:16.457 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_16_00_UTC 2024-03-29T11:23:16.458 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_17_00_UTC 2024-03-29T11:23:16.458 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_18_00_UTC 2024-03-29T11:23:16.458 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_19_00_UTC 2024-03-29T11:23:16.458 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_20_00_UTC 2024-03-29T11:23:16.458 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_21_00_UTC 2024-03-29T11:23:16.458 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_22_00_UTC 2024-03-29T11:23:16.458 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_23_00_UTC 2024-03-29T11:23:16.459 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-11_13_00_UTC', 'scheduled-2024-03-29-11_14_00_UTC', 'scheduled-2024-03-29-11_15_00_UTC', 'scheduled-2024-03-29-11_16_00_UTC', 'scheduled-2024-03-29-11_17_00_UTC', 'scheduled-2024-03-29-11_18_00_UTC', 'scheduled-2024-03-29-11_19_00_UTC', 'scheduled-2024-03-29-11_20_00_UTC', 'scheduled-2024-03-29-11_21_00_UTC', 'scheduled-2024-03-29-11_22_00_UTC', 'scheduled-2024-03-29-11_23_00_UTC'] 2024-03-29T11:23:16.459 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-03-29T11:23:16.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.686+0000 7f76c5c00700 1 Processor -- start 2024-03-29T11:23:16.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.687+0000 7f76c5c00700 1 -- start start 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.688+0000 7f76c5c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cbb30 0x7f76c00cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.688+0000 7f76c5c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f76c00cd4a0 0x7f76c006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.688+0000 7f76c5c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c00cc450 con 0x7f76c00cd020 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.688+0000 7f76c5c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c00cdae0 con 0x7f76c00cd4a0 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.688+0000 7f76c5c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00cddc0 con 0x7f76c00cbb30 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.688+0000 7f76bea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cbb30 0x7f76c00cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.688+0000 7f76bf400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f76c00cd4a0 0x7f76c006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.688+0000 7f76bf400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f76c00cd4a0 0x7f76c006e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:39316/0 (socket says 172.21.15.12:39316) 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.688+0000 7f76bf400700 1 -- 172.21.15.12:0/1754480911 learned_addr learned my addr 172.21.15.12:0/1754480911 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.688+0000 7f76bea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cbb30 0x7f76c00cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:55652/0 (socket says 172.21.15.12:55652) 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.689+0000 7f76bf400700 1 -- 172.21.15.12:0/1754480911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cbb30 msgr2=0x7f76c00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.689+0000 7f76bf400700 1 --2- 172.21.15.12:0/1754480911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cbb30 0x7f76c00cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.689+0000 7f76bf400700 1 -- 172.21.15.12:0/1754480911 >> v1:172.21.15.159:6789/0 conn(0x7f76c00cd020 legacy=0x7f76c00b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.689+0000 7f76bf400700 1 -- 172.21.15.12:0/1754480911 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c00ce0a0 con 0x7f76c00cd4a0 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.689+0000 7f76bf400700 1 --2- 172.21.15.12:0/1754480911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76c00cd4a0 0x7f76c006e420 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009e50 tx=0x7f76b800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a34ebf8980c96ef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.689+0000 7f76bd600700 1 -- 172.21.15.12:0/1754480911 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b800e070 con 0x7f76c00cd4a0 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.689+0000 7f76bd600700 1 -- 172.21.15.12:0/1754480911 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b800b910 con 0x7f76c00cd4a0 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.690+0000 7f76bd600700 1 -- 172.21.15.12:0/1754480911 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b800bcc0 con 0x7f76c00cd4a0 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.690+0000 7f76c5c00700 1 -- 172.21.15.12:0/1754480911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76c00cd4a0 msgr2=0x7f76c006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.690+0000 7f76c5c00700 1 --2- 172.21.15.12:0/1754480911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76c00cd4a0 0x7f76c006e420 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009e50 tx=0x7f76b800b330 comp rx=0 tx=0).stop 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.690+0000 7f76c5c00700 1 -- 172.21.15.12:0/1754480911 shutdown_connections 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.690+0000 7f76c5c00700 1 --2- 172.21.15.12:0/1754480911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76c00cd4a0 0x7f76c006e420 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.690+0000 7f76c5c00700 1 --2- 172.21.15.12:0/1754480911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cbb30 0x7f76c00cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.690+0000 7f76c5c00700 1 -- 172.21.15.12:0/1754480911 >> 172.21.15.12:0/1754480911 conn(0x7f76c00b0d70 msgr2=0x7f76c00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.690+0000 7f76c5c00700 1 -- 172.21.15.12:0/1754480911 shutdown_connections 2024-03-29T11:23:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.690+0000 7f76c5c00700 1 -- 172.21.15.12:0/1754480911 wait complete. 2024-03-29T11:23:16.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.692+0000 7f76c5c00700 1 Processor -- start 2024-03-29T11:23:16.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.692+0000 7f76c5c00700 1 -- start start 2024-03-29T11:23:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.693+0000 7f76c5c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76c00cbb30 0x7f76c01484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.693+0000 7f76c5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76c00cd020 0x7f76c00caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.693+0000 7f76c5c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cd4a0 0x7f76c00c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.693+0000 7f76c5c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00acf70 con 0x7f76c00cd020 2024-03-29T11:23:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.693+0000 7f76c5c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00ad250 con 0x7f76c00cbb30 2024-03-29T11:23:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.693+0000 7f76c5c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00ad530 con 0x7f76c00cd4a0 2024-03-29T11:23:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.693+0000 7f76be000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76c00cd020 0x7f76c00caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.693+0000 7f76bea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76c00cbb30 0x7f76c01484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.693+0000 7f76be000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76c00cd020 0x7f76c00caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57008/0 (socket says 172.21.15.12:57008) 2024-03-29T11:23:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76be000700 1 -- 172.21.15.12:0/1089973544 learned_addr learned my addr 172.21.15.12:0/1089973544 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:23:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76bf400700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cd4a0 0x7f76c00c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76be000700 1 -- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cd4a0 msgr2=0x7f76c00c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76be000700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cd4a0 0x7f76c00c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76be000700 1 -- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76c00cbb30 msgr2=0x7f76c01484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76be000700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76c00cbb30 0x7f76c01484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76be000700 1 -- 172.21.15.12:0/1089973544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b8009b30 con 0x7f76c00cd020 2024-03-29T11:23:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76be000700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76c00cd020 0x7f76c00caff0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00c450 tx=0x7f76ac00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:23:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76b6a00700 1 -- 172.21.15.12:0/1089973544 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0052d0 con 0x7f76c00cd020 2024-03-29T11:23:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76b6a00700 1 -- 172.21.15.12:0/1089973544 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ac005430 con 0x7f76c00cd020 2024-03-29T11:23:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76b6a00700 1 -- 172.21.15.12:0/1089973544 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0038c0 con 0x7f76c00cd020 2024-03-29T11:23:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.694+0000 7f76c5c00700 1 -- 172.21.15.12:0/1089973544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c00ad810 con 0x7f76c00cd020 2024-03-29T11:23:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.695+0000 7f76c5c00700 1 -- 172.21.15.12:0/1089973544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c0148a00 con 0x7f76c00cd020 2024-03-29T11:23:16.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.695+0000 7f76c5c00700 1 -- 172.21.15.12:0/1089973544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c0067720 con 0x7f76c00cd020 2024-03-29T11:23:16.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.697+0000 7f76b6a00700 1 -- 172.21.15.12:0/1089973544 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 106) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f76ac003a20 con 0x7f76c00cd020 2024-03-29T11:23:16.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.697+0000 7f76b6a00700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76a006ae00 0x7f76a006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:16.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.697+0000 7f76b6a00700 1 -- 172.21.15.12:0/1089973544 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f76ac0889b0 con 0x7f76c00cd020 2024-03-29T11:23:16.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.698+0000 7f76bea00700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76a006ae00 0x7f76a006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:16.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.699+0000 7f76bea00700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76a006ae00 0x7f76a006d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009b40 tx=0x7f76b0009210 comp rx=0 tx=0).ready entity=mgr.9772 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:23:16.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.702+0000 7f76b6a00700 1 -- 172.21.15.12:0/1089973544 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76ac0538f0 con 0x7f76c00cd020 2024-03-29T11:23:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.945+0000 7f76c5c00700 1 -- 172.21.15.12:0/1089973544 --> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f76c00ad3c0 con 0x7f76a006ae00 2024-03-29T11:23:16.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.954+0000 7f76b6a00700 1 -- 172.21.15.12:0/1089973544 <== mgr.9772 v2:172.21.15.159:6832/43444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+375 (secure 0 0 0) 0x7f76c00ad3c0 con 0x7f76a006ae00 2024-03-29T11:23:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:23:16.933 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T11:12:44", "first": "2024-03-29T11:13:00", "last": "2024-03-29T11:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.956+0000 7f76c5c00700 1 -- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76a006ae00 msgr2=0x7f76a006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.956+0000 7f76c5c00700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76a006ae00 0x7f76a006d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009b40 tx=0x7f76b0009210 comp rx=0 tx=0).stop 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.956+0000 7f76c5c00700 1 -- 172.21.15.12:0/1089973544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76c00cd020 msgr2=0x7f76c00caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.956+0000 7f76c5c00700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76c00cd020 0x7f76c00caff0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00c450 tx=0x7f76ac00c760 comp rx=0 tx=0).stop 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.957+0000 7f76c5c00700 1 -- 172.21.15.12:0/1089973544 shutdown_connections 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.957+0000 7f76c5c00700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7f76a006ae00 0x7f76a006d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.957+0000 7f76c5c00700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76c00cd4a0 0x7f76c00c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.957+0000 7f76c5c00700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76c00cbb30 0x7f76c01484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.957+0000 7f76c5c00700 1 --2- 172.21.15.12:0/1089973544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76c00cd020 0x7f76c00caff0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.957+0000 7f76c5c00700 1 -- 172.21.15.12:0/1089973544 >> 172.21.15.12:0/1089973544 conn(0x7f76c00b0d70 msgr2=0x7f76c015a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.957+0000 7f76c5c00700 1 -- 172.21.15.12:0/1089973544 shutdown_connections 2024-03-29T11:23:16.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:16.957+0000 7f76c5c00700 1 -- 172.21.15.12:0/1089973544 wait complete. 2024-03-29T11:23:16.953 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T11:12:44", "first": "2024-03-29T11:13:00", "last": "2024-03-29T11:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-03-29T11:23:16.953 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T11:12:44', 'first': '2024-03-29T11:13:00', 'last': '2024-03-29T11:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2024-03-29T11:23:16.953 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-29T11:23:16.953 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-29T11:23:16.953 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-29T11:23:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.172+0000 7fee96a00700 1 Processor -- start 2024-03-29T11:23:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.172+0000 7fee96a00700 1 -- start start 2024-03-29T11:23:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.173+0000 7fee96a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fee9015da50 0x7fee900ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.173+0000 7fee96a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee900cec70 0x7fee900d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.173+0000 7fee96a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee900d24e0 con 0x7fee900d0b20 2024-03-29T11:23:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.173+0000 7fee96a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee900b95d0 con 0x7fee9015da50 2024-03-29T11:23:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.173+0000 7fee96a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9015de30 con 0x7fee900cec70 2024-03-29T11:23:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee8f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee900cec70 0x7fee900d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee8fe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fee9015da50 0x7fee900ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee94a00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fee900d0b20 0x7fee900b52e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:44328/0 (socket says 172.21.15.12:44328) 2024-03-29T11:23:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee8f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee900cec70 0x7fee900d05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:55668/0 (socket says 172.21.15.12:55668) 2024-03-29T11:23:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee94a00700 1 -- 172.21.15.12:0/652119035 learned_addr learned my addr 172.21.15.12:0/652119035 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:23:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee8fe00700 1 -- 172.21.15.12:0/652119035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee900cec70 msgr2=0x7fee900d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee8fe00700 1 --2- 172.21.15.12:0/652119035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee900cec70 0x7fee900d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee8fe00700 1 -- 172.21.15.12:0/652119035 >> v1:172.21.15.159:6789/0 conn(0x7fee900d0b20 legacy=0x7fee900b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee8fe00700 1 -- 172.21.15.12:0/652119035 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee900d27c0 con 0x7fee9015da50 2024-03-29T11:23:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee8ea00700 1 -- 172.21.15.12:0/652119035 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407707776 0 0) 0x7fee900d24e0 con 0x7fee900d0b20 2024-03-29T11:23:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.174+0000 7fee8fe00700 1 --2- 172.21.15.12:0/652119035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee9015da50 0x7fee900ce730 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fee80009840 tx=0x7fee80009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eaa75f937a63ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:23:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee8ea00700 1 -- 172.21.15.12:0/652119035 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80011830 con 0x7fee9015da50 2024-03-29T11:23:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee8ea00700 1 -- 172.21.15.12:0/652119035 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee80011990 con 0x7fee9015da50 2024-03-29T11:23:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee8ea00700 1 -- 172.21.15.12:0/652119035 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80011d20 con 0x7fee9015da50 2024-03-29T11:23:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee96a00700 1 -- 172.21.15.12:0/652119035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee9015da50 msgr2=0x7fee900ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee96a00700 1 --2- 172.21.15.12:0/652119035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee9015da50 0x7fee900ce730 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fee80009840 tx=0x7fee80009b50 comp rx=0 tx=0).stop 2024-03-29T11:23:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee96a00700 1 -- 172.21.15.12:0/652119035 shutdown_connections 2024-03-29T11:23:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee96a00700 1 --2- 172.21.15.12:0/652119035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee9015da50 0x7fee900ce730 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee96a00700 1 --2- 172.21.15.12:0/652119035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee900cec70 0x7fee900d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee96a00700 1 -- 172.21.15.12:0/652119035 >> 172.21.15.12:0/652119035 conn(0x7fee900666a0 msgr2=0x7fee90066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee96a00700 1 -- 172.21.15.12:0/652119035 shutdown_connections 2024-03-29T11:23:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.175+0000 7fee96a00700 1 -- 172.21.15.12:0/652119035 wait complete. 2024-03-29T11:23:17.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.177+0000 7fee96a00700 1 Processor -- start 2024-03-29T11:23:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.177+0000 7fee96a00700 1 -- start start 2024-03-29T11:23:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.178+0000 7fee96a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee900cec70 0x7fee9015cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.178+0000 7fee96a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee900d0b20 0x7fee900c6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.178+0000 7fee96a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee9015da50 0x7fee900b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.178+0000 7fee96a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee900cbe00 con 0x7fee900cec70 2024-03-29T11:23:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.178+0000 7fee96a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee900c6a80 con 0x7fee900d0b20 2024-03-29T11:23:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.178+0000 7fee96a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee900b4210 con 0x7fee9015da50 2024-03-29T11:23:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.178+0000 7fee8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee900cec70 0x7fee9015cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee900cec70 0x7fee9015cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57012/0 (socket says 172.21.15.12:57012) 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee8fe00700 1 -- 172.21.15.12:0/1707429524 learned_addr learned my addr 172.21.15.12:0/1707429524 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee94a00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee9015da50 0x7fee900b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee8f400700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee900d0b20 0x7fee900c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee8fe00700 1 -- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee9015da50 msgr2=0x7fee900b3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee8fe00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee9015da50 0x7fee900b3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee8fe00700 1 -- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee900d0b20 msgr2=0x7fee900c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee8fe00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee900d0b20 0x7fee900c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee8fe00700 1 -- 172.21.15.12:0/1707429524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee800094a0 con 0x7fee900cec70 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee8f400700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee900d0b20 0x7fee900c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee8fe00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee900cec70 0x7fee9015cdb0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fee80009b90 tx=0x7fee80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee87e00700 1 -- 172.21.15.12:0/1707429524 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee800038a0 con 0x7fee900cec70 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee87e00700 1 -- 172.21.15.12:0/1707429524 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8001e070 con 0x7fee900cec70 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.179+0000 7fee96a00700 1 -- 172.21.15.12:0/1707429524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee900b44b0 con 0x7fee900cec70 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.180+0000 7fee87e00700 1 -- 172.21.15.12:0/1707429524 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80003c40 con 0x7fee900cec70 2024-03-29T11:23:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.180+0000 7fee96a00700 1 -- 172.21.15.12:0/1707429524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee900cba20 con 0x7fee900cec70 2024-03-29T11:23:17.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.181+0000 7fee96a00700 1 -- 172.21.15.12:0/1707429524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee90067720 con 0x7fee900cec70 2024-03-29T11:23:17.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.182+0000 7fee87e00700 1 -- 172.21.15.12:0/1707429524 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 106) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fee8002d450 con 0x7fee900cec70 2024-03-29T11:23:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.182+0000 7fee87e00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fee7006ae00 0x7fee7006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:23:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.183+0000 7fee87e00700 1 -- 172.21.15.12:0/1707429524 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fee8008a610 con 0x7fee900cec70 2024-03-29T11:23:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.183+0000 7fee8f400700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fee7006ae00 0x7fee7006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:23:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.184+0000 7fee8f400700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fee7006ae00 0x7fee7006d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fee901e3ec0 tx=0x7fee7c008040 comp rx=0 tx=0).ready entity=mgr.9772 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:23:17.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.187+0000 7fee87e00700 1 -- 172.21.15.12:0/1707429524 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee800555e0 con 0x7fee900cec70 2024-03-29T11:23:17.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.465+0000 7fee96a00700 1 -- 172.21.15.12:0/1707429524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fee90066f20 con 0x7fee900cec70 2024-03-29T11:23:17.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.469+0000 7fee87e00700 1 -- 172.21.15.12:0/1707429524 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v106) v1 ==== 74+0+171143 (secure 0 0 0) 0x7fee80058c00 con 0x7fee900cec70 2024-03-29T11:23:17.450 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:23:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.472+0000 7fee96a00700 1 -- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fee7006ae00 msgr2=0x7fee7006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.472+0000 7fee96a00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fee7006ae00 0x7fee7006d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fee901e3ec0 tx=0x7fee7c008040 comp rx=0 tx=0).stop 2024-03-29T11:23:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.472+0000 7fee96a00700 1 -- 172.21.15.12:0/1707429524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee900cec70 msgr2=0x7fee9015cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.472+0000 7fee96a00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee900cec70 0x7fee9015cdb0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fee80009b90 tx=0x7fee80004750 comp rx=0 tx=0).stop 2024-03-29T11:23:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.472+0000 7fee96a00700 1 -- 172.21.15.12:0/1707429524 shutdown_connections 2024-03-29T11:23:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.472+0000 7fee96a00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:6832/43444,v1:172.21.15.159:6833/43444] conn(0x7fee7006ae00 0x7fee7006d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.472+0000 7fee96a00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee9015da50 0x7fee900b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.472+0000 7fee96a00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee900cec70 0x7fee9015cdb0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.472+0000 7fee96a00700 1 --2- 172.21.15.12:0/1707429524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee900d0b20 0x7fee900c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:23:17.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.472+0000 7fee96a00700 1 -- 172.21.15.12:0/1707429524 >> 172.21.15.12:0/1707429524 conn(0x7fee900666a0 msgr2=0x7fee901e2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:17.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.473+0000 7fee96a00700 1 -- 172.21.15.12:0/1707429524 shutdown_connections 2024-03-29T11:23:17.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:23:17.473+0000 7fee96a00700 1 -- 172.21.15.12:0/1707429524 wait complete. 2024-03-29T11:23:17.469 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":106,"active_gid":9772,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":43444},{"type":"v1","addr":"172.21.15.159:6833","nonce":43444}]},"active_addr":"172.21.15.159:6833/43444","active_change":"2024-03-29T11:12:43.758179+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":10146,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":192,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":479652442}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":3819915206}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":3631785847}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":750511589}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":3123015140}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":2791060580}]}]} 2024-03-29T11:23:17.473 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2024-03-29T11:23:17.473 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:23:17.473 INFO:tasks.ceph.mgr.x:Stopping old one... 2024-03-29T11:23:17.473 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:23:17.473 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:17.550 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:23:17.550 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:23:17.554 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:23:17.821 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:17.842+0000 7fad37884240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:23:18.006 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:18.027+0000 7fad37884240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:23:18.325 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:18.346+0000 7fad37884240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:23:18.534 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:23:18.539 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:23:18.864 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:18.885+0000 7fad37884240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:23:19.121 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:19.143+0000 7fad37884240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:23:19.579 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:19.601+0000 7fad37884240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:23:20.289 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:20.311+0000 7fad37884240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:23:20.380 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:20.403+0000 7fad37884240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:23:20.539 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:20.561+0000 7fad37884240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:23:20.754 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:20.775+0000 7fad37884240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:23:20.845 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:20.867+0000 7fad37884240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:23:20.945 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:20.966+0000 7fad37884240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:23:21.042 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:21.064+0000 7fad37884240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:23:21.139 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:21.161+0000 7fad37884240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:23:21.352 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:21.374+0000 7fad37884240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:23:21.608 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:21.630+0000 7fad37884240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:23:21.798 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:21.820+0000 7fad37884240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:23:21.891 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:21.913+0000 7fad37884240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:23:22.628 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:22.650+0000 7fad37884240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:23:22.838 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:22.860+0000 7fad37884240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:23:23.045 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:23.067+0000 7fad37884240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:23:23.522 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:23.544+0000 7fad37884240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:23:23.730 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:23.752+0000 7fad37884240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:23:24.176 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.198+0000 7fad37884240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:23:24.289 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.311+0000 7fad37884240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:23:24.475 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.497+0000 7faca2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:23:24.475 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.497+0000 7faca2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:23:24.475 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.497+0000 7faca2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:23:24.475 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.497+0000 7faca2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:23:24.475 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.497+0000 7faca2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:23:24.476 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.498+0000 7fac9f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:23:24.476 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.498+0000 7fac9f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:23:24.476 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.498+0000 7fac9f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:23:24.476 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.498+0000 7fac9f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:23:24.476 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:23:24.498+0000 7fac9f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:23:48.650 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:23:48.655 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:24:00.471 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:24:00.490+0000 7facad200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:24:00.471 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:24:00.490+0000 7facad200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:24:00.471 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:24:00.490+0000 7facad200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:24:00.472 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:24:00.490+0000 7facad200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:24:00.472 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:24:00.490+0000 7facad200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:24:18.892 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:24:18.897 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:24:49.005 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:24:49.013 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:25:19.178 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:25:19.184 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:25:49.286 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:25:49.291 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:26:19.501 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:26:19.506 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:26:49.619 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:26:49.625 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:27:19.738 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:27:19.743 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:27:49.861 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:27:49.868 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:28:17.558 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2024-03-29T11:28:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.780+0000 7f3f36400700 1 Processor -- start 2024-03-29T11:28:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.780+0000 7f3f36400700 1 -- start start 2024-03-29T11:28:17.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.781+0000 7f3f36400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f300cbb30 0x7f3f300cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:17.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.781+0000 7f3f36400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f300cc450 con 0x7f3f300cd4a0 2024-03-29T11:28:17.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.781+0000 7f3f36400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f300cdae0 con 0x7f3f300cd020 2024-03-29T11:28:17.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.781+0000 7f3f36400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f300cddc0 con 0x7f3f300cbb30 2024-03-29T11:28:17.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.781+0000 7f3f2fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3f300cd4a0 0x7f3f3006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33352/0 (socket says 172.21.15.12:33352) 2024-03-29T11:28:17.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2fe00700 1 -- 172.21.15.12:0/166221523 learned_addr learned my addr 172.21.15.12:0/166221523 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:17.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2f400700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f300cbb30 0x7f3f300cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:17.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076829978 0 0) 0x7f3f300cc450 con 0x7f3f300cd4a0 2024-03-29T11:28:17.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f140035b0 con 0x7f3f300cd4a0 2024-03-29T11:28:17.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491045755 0 0) 0x7f3f300cdae0 con 0x7f3f300cd020 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f300cc450 con 0x7f3f300cd020 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1100558076 0 0) 0x7f3f140035b0 con 0x7f3f300cd4a0 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f300cbb30 msgr2=0x7f3f300cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f300cbb30 0x7f3f300cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 >> v1:172.21.15.159:6789/0 conn(0x7f3f300cd020 legacy=0x7f3f300b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f300ce0a0 con 0x7f3f300cd4a0 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f3f280019f0 con 0x7f3f300cd4a0 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 >> v1:172.21.15.12:6789/0 conn(0x7f3f300cd4a0 legacy=0x7f3f3006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 0x7f3f14003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.782+0000 7f3f2e000700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f300cbb30 0x7f3f14004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2e000700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f14004d90 0x7f3f14005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f300cddc0 con 0x7f3f14003bc0 2024-03-29T11:28:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1400bac0 con 0x7f3f14004d90 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1400bc30 con 0x7f3f300cbb30 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2f400700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 0x7f3f14003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2fe00700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f14004d90 0x7f3f14005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2ea00700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f300cbb30 0x7f3f14004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2f400700 1 -- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f300cbb30 msgr2=0x7f3f14004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2f400700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f300cbb30 0x7f3f14004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2f400700 1 -- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f14004d90 msgr2=0x7f3f14005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2f400700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f14004d90 0x7f3f14005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2f400700 1 -- 172.21.15.12:0/166221523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f140035b0 con 0x7f3f14003bc0 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.783+0000 7f3f2f400700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 0x7f3f14003fc0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009b60 tx=0x7f3f14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da06df8f0a663c77 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.784+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2000e6c0 con 0x7f3f14003bc0 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.784+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2000e820 con 0x7f3f14003bc0 2024-03-29T11:28:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.784+0000 7f3f2e000700 1 -- 172.21.15.12:0/166221523 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2000eb90 con 0x7f3f14003bc0 2024-03-29T11:28:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.784+0000 7f3f36400700 1 -- 172.21.15.12:0/166221523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 msgr2=0x7f3f14003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.784+0000 7f3f36400700 1 --2- 172.21.15.12:0/166221523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 0x7f3f14003fc0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009b60 tx=0x7f3f14005880 comp rx=0 tx=0).stop 2024-03-29T11:28:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.784+0000 7f3f2fe00700 1 -- 172.21.15.12:0/166221523 reap_dead start 2024-03-29T11:28:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.784+0000 7f3f36400700 1 -- 172.21.15.12:0/166221523 shutdown_connections 2024-03-29T11:28:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.784+0000 7f3f36400700 1 -- 172.21.15.12:0/166221523 >> 172.21.15.12:0/166221523 conn(0x7f3f300b0d70 msgr2=0x7f3f300b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.784+0000 7f3f36400700 1 -- 172.21.15.12:0/166221523 shutdown_connections 2024-03-29T11:28:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.784+0000 7f3f36400700 1 -- 172.21.15.12:0/166221523 wait complete. 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.786+0000 7f3f36400700 1 Processor -- start 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.786+0000 7f3f36400700 1 -- start start 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.787+0000 7f3f36400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 0x7f3f3014e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.787+0000 7f3f36400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f30072500 0x7f3f3015d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.787+0000 7f3f36400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3015d610 0x7f3f300ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.787+0000 7f3f36400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30071920 con 0x7f3f14003bc0 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.787+0000 7f3f36400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3015da90 con 0x7f3f3015d610 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.787+0000 7f3f36400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f300c94c0 con 0x7f3f30072500 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 0x7f3f3014e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 0x7f3f3014e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46282/0 (socket says 172.21.15.12:46282) 2024-03-29T11:28:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2f400700 1 -- 172.21.15.12:0/1902908686 learned_addr learned my addr 172.21.15.12:0/1902908686 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2fe00700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3015d610 0x7f3f300ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2ea00700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f30072500 0x7f3f3015d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2f400700 1 -- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f30072500 msgr2=0x7f3f3015d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2f400700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f30072500 0x7f3f3015d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2f400700 1 -- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3015d610 msgr2=0x7f3f300ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2f400700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3015d610 0x7f3f300ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2f400700 1 -- 172.21.15.12:0/1902908686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f30070da0 con 0x7f3f14003bc0 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f2f400700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 0x7f3f3014e540 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f3f14005830 tx=0x7f3f200047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.788+0000 7f3f27400700 1 -- 172.21.15.12:0/1902908686 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f20021070 con 0x7f3f14003bc0 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.789+0000 7f3f27400700 1 -- 172.21.15.12:0/1902908686 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f200045d0 con 0x7f3f14003bc0 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.789+0000 7f3f36400700 1 -- 172.21.15.12:0/1902908686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f30071080 con 0x7f3f14003bc0 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.789+0000 7f3f27400700 1 -- 172.21.15.12:0/1902908686 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2000e6c0 con 0x7f3f14003bc0 2024-03-29T11:28:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.789+0000 7f3f36400700 1 -- 172.21.15.12:0/1902908686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f30071640 con 0x7f3f14003bc0 2024-03-29T11:28:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.790+0000 7f3f36400700 1 -- 172.21.15.12:0/1902908686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f30071860 con 0x7f3f14003bc0 2024-03-29T11:28:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.791+0000 7f3f27400700 1 -- 172.21.15.12:0/1902908686 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 112) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f3f20025070 con 0x7f3f14003bc0 2024-03-29T11:28:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.792+0000 7f3f27400700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3f1006ae00 0x7f3f1006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.792+0000 7f3f27400700 1 -- 172.21.15.12:0/1902908686 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f3f2008a970 con 0x7f3f14003bc0 2024-03-29T11:28:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.792+0000 7f3f2ea00700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3f1006ae00 0x7f3f1006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.793+0000 7f3f2ea00700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3f1006ae00 0x7f3f1006d2c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3f300c9250 tx=0x7f3f1c009040 comp rx=0 tx=0).ready entity=mgr.9874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:17.797+0000 7f3f27400700 1 -- 172.21.15.12:0/1902908686 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f200557a0 con 0x7f3f14003bc0 2024-03-29T11:28:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.037+0000 7f3f36400700 1 -- 172.21.15.12:0/1902908686 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f301594a0 con 0x7f3f1006ae00 2024-03-29T11:28:18.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.114+0000 7f3f27400700 1 -- 172.21.15.12:0/1902908686 <== mgr.9874 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f3f1c005170 con 0x7f3f1006ae00 2024-03-29T11:28:18.103 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2024-03-29T11:28:18.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.115+0000 7f3f36400700 1 -- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3f1006ae00 msgr2=0x7f3f1006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.115+0000 7f3f36400700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3f1006ae00 0x7f3f1006d2c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3f300c9250 tx=0x7f3f1c009040 comp rx=0 tx=0).stop 2024-03-29T11:28:18.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.115+0000 7f3f36400700 1 -- 172.21.15.12:0/1902908686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 msgr2=0x7f3f3014e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.115+0000 7f3f36400700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 0x7f3f3014e540 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f3f14005830 tx=0x7f3f200047c0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.116+0000 7f3f36400700 1 -- 172.21.15.12:0/1902908686 shutdown_connections 2024-03-29T11:28:18.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.116+0000 7f3f36400700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3f1006ae00 0x7f3f1006d2c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.116+0000 7f3f36400700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3015d610 0x7f3f300ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.116+0000 7f3f36400700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f14003bc0 0x7f3f3014e540 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.116+0000 7f3f36400700 1 --2- 172.21.15.12:0/1902908686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f30072500 0x7f3f3015d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.116+0000 7f3f36400700 1 -- 172.21.15.12:0/1902908686 >> 172.21.15.12:0/1902908686 conn(0x7f3f300b0d70 msgr2=0x7f3f3015ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:18.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.116+0000 7f3f36400700 1 -- 172.21.15.12:0/1902908686 shutdown_connections 2024-03-29T11:28:18.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.116+0000 7f3f36400700 1 -- 172.21.15.12:0/1902908686 wait complete. 2024-03-29T11:28:18.124 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2024-03-29T11:28:18.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:18.124 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-03-29T11:28:18.170 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_13_00_UTC 2024-03-29T11:28:18.170 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_14_00_UTC 2024-03-29T11:28:18.170 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_15_00_UTC 2024-03-29T11:28:18.170 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_16_00_UTC 2024-03-29T11:28:18.170 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_17_00_UTC 2024-03-29T11:28:18.170 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_18_00_UTC 2024-03-29T11:28:18.170 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_19_00_UTC 2024-03-29T11:28:18.170 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_20_00_UTC 2024-03-29T11:28:18.170 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_21_00_UTC 2024-03-29T11:28:18.171 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_22_00_UTC 2024-03-29T11:28:18.171 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_23_00_UTC 2024-03-29T11:28:18.171 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_24_00_UTC 2024-03-29T11:28:18.171 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_25_00_UTC 2024-03-29T11:28:18.171 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_26_00_UTC 2024-03-29T11:28:18.171 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_27_00_UTC 2024-03-29T11:28:18.171 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_28_00_UTC 2024-03-29T11:28:18.172 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-29-11_13_00_UTC', 'scheduled-2024-03-29-11_14_00_UTC', 'scheduled-2024-03-29-11_15_00_UTC', 'scheduled-2024-03-29-11_16_00_UTC', 'scheduled-2024-03-29-11_17_00_UTC', 'scheduled-2024-03-29-11_18_00_UTC', 'scheduled-2024-03-29-11_19_00_UTC', 'scheduled-2024-03-29-11_20_00_UTC', 'scheduled-2024-03-29-11_21_00_UTC', 'scheduled-2024-03-29-11_22_00_UTC', 'scheduled-2024-03-29-11_23_00_UTC', 'scheduled-2024-03-29-11_24_00_UTC', 'scheduled-2024-03-29-11_25_00_UTC', 'scheduled-2024-03-29-11_26_00_UTC', 'scheduled-2024-03-29-11_27_00_UTC', 'scheduled-2024-03-29-11_28_00_UTC'] 2024-03-29T11:28:18.172 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-03-29T11:28:18.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.422+0000 7f26eaa00700 1 Processor -- start 2024-03-29T11:28:18.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.423+0000 7f26eaa00700 1 -- start start 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.424+0000 7f26eaa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f26e4160e50 0x7f26e40b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.424+0000 7f26eaa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e40d3f20 0x7f26e40b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.424+0000 7f26eaa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e40cef00 con 0x7f26e40b3f80 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.424+0000 7f26eaa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e4161230 con 0x7f26e4160e50 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.424+0000 7f26eaa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e40d2a80 con 0x7f26e40d3f20 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.424+0000 7f26e8a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f26e40b3f80 0x7f26e40b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33360/0 (socket says 172.21.15.12:33360) 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.424+0000 7f26e8a00700 1 -- 172.21.15.12:0/2814757962 learned_addr learned my addr 172.21.15.12:0/2814757962 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.424+0000 7f26e3e00700 1 --2- 172.21.15.12:0/2814757962 >> v2:172.21.15.159:3300/0 conn(0x7f26e4160e50 0x7f26e40b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.424+0000 7f26e3400700 1 --2- 172.21.15.12:0/2814757962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e40d3f20 0x7f26e40b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.424+0000 7f26e2a00700 1 -- 172.21.15.12:0/2814757962 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128694961 0 0) 0x7f26e40cef00 con 0x7f26e40b3f80 2024-03-29T11:28:18.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.425+0000 7f26e2a00700 1 -- 172.21.15.12:0/2814757962 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c80035b0 con 0x7f26e40b3f80 2024-03-29T11:28:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.425+0000 7f26e3e00700 1 -- 172.21.15.12:0/2814757962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e40d3f20 msgr2=0x7f26e40b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.425+0000 7f26e3e00700 1 --2- 172.21.15.12:0/2814757962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e40d3f20 0x7f26e40b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.425+0000 7f26e3e00700 1 -- 172.21.15.12:0/2814757962 >> v1:172.21.15.12:6789/0 conn(0x7f26e40b3f80 legacy=0x7f26e40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.425+0000 7f26e3e00700 1 -- 172.21.15.12:0/2814757962 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e40d43b0 con 0x7f26e4160e50 2024-03-29T11:28:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.425+0000 7f26e3e00700 1 --2- 172.21.15.12:0/2814757962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e4160e50 0x7f26e40b4e90 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009840 tx=0x7f26d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32a6236a282631e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.425+0000 7f26e2a00700 1 -- 172.21.15.12:0/2814757962 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d4011830 con 0x7f26e4160e50 2024-03-29T11:28:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.425+0000 7f26e2a00700 1 -- 172.21.15.12:0/2814757962 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d4011990 con 0x7f26e4160e50 2024-03-29T11:28:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.425+0000 7f26e2a00700 1 -- 172.21.15.12:0/2814757962 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d4011d20 con 0x7f26e4160e50 2024-03-29T11:28:18.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.426+0000 7f26eaa00700 1 -- 172.21.15.12:0/2814757962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e4160e50 msgr2=0x7f26e40b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.426+0000 7f26eaa00700 1 --2- 172.21.15.12:0/2814757962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e4160e50 0x7f26e40b4e90 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009840 tx=0x7f26d4009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.426+0000 7f26eaa00700 1 -- 172.21.15.12:0/2814757962 shutdown_connections 2024-03-29T11:28:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.426+0000 7f26eaa00700 1 --2- 172.21.15.12:0/2814757962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e4160e50 0x7f26e40b4e90 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.426+0000 7f26eaa00700 1 --2- 172.21.15.12:0/2814757962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e40d3f20 0x7f26e40b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.426+0000 7f26eaa00700 1 -- 172.21.15.12:0/2814757962 >> 172.21.15.12:0/2814757962 conn(0x7f26e408aff0 msgr2=0x7f26e40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.426+0000 7f26eaa00700 1 -- 172.21.15.12:0/2814757962 shutdown_connections 2024-03-29T11:28:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.426+0000 7f26eaa00700 1 -- 172.21.15.12:0/2814757962 wait complete. 2024-03-29T11:28:18.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.428+0000 7f26eaa00700 1 Processor -- start 2024-03-29T11:28:18.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.428+0000 7f26eaa00700 1 -- start start 2024-03-29T11:28:18.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.429+0000 7f26eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e40b3f80 0x7f26e414cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.429+0000 7f26eaa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e4160e50 0x7f26e40b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.429+0000 7f26eaa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e40b0970 0x7f26e40c9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.429+0000 7f26eaa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e40b0e20 con 0x7f26e40b3f80 2024-03-29T11:28:18.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.429+0000 7f26eaa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e414d120 con 0x7f26e40b0970 2024-03-29T11:28:18.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.429+0000 7f26eaa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e40cacb0 con 0x7f26e4160e50 2024-03-29T11:28:18.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.429+0000 7f26e3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e40b3f80 0x7f26e414cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.429+0000 7f26e3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e40b3f80 0x7f26e414cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46290/0 (socket says 172.21.15.12:46290) 2024-03-29T11:28:18.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.429+0000 7f26e3e00700 1 -- 172.21.15.12:0/953408146 learned_addr learned my addr 172.21.15.12:0/953408146 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:18.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26e3400700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e4160e50 0x7f26e40b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26e8a00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e40b0970 0x7f26e40c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26e3e00700 1 -- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e4160e50 msgr2=0x7f26e40b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26e3e00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e4160e50 0x7f26e40b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26e3e00700 1 -- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e40b0970 msgr2=0x7f26e40c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26e3e00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e40b0970 0x7f26e40c9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26e3e00700 1 -- 172.21.15.12:0/953408146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d40094a0 con 0x7f26e40b3f80 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26e3400700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e4160e50 0x7f26e40b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26e8a00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e40b0970 0x7f26e40c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26e3e00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e40b3f80 0x7f26e414cbe0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009fd0 tx=0x7f26d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26dbe00700 1 -- 172.21.15.12:0/953408146 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d40036a0 con 0x7f26e40b3f80 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26dbe00700 1 -- 172.21.15.12:0/953408146 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d401e070 con 0x7f26e40b3f80 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26eaa00700 1 -- 172.21.15.12:0/953408146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e40ca6a0 con 0x7f26e40b3f80 2024-03-29T11:28:18.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26dbe00700 1 -- 172.21.15.12:0/953408146 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d4003a60 con 0x7f26e40b3f80 2024-03-29T11:28:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.430+0000 7f26eaa00700 1 -- 172.21.15.12:0/953408146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e40ca940 con 0x7f26e40b3f80 2024-03-29T11:28:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.431+0000 7f26eaa00700 1 -- 172.21.15.12:0/953408146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e4067720 con 0x7f26e40b3f80 2024-03-29T11:28:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.432+0000 7f26dbe00700 1 -- 172.21.15.12:0/953408146 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 112) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f26d4003bc0 con 0x7f26e40b3f80 2024-03-29T11:28:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.433+0000 7f26dbe00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c406ae00 0x7f26c406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.433+0000 7f26dbe00700 1 -- 172.21.15.12:0/953408146 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f26d408a690 con 0x7f26e40b3f80 2024-03-29T11:28:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.433+0000 7f26e3400700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c406ae00 0x7f26c406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.434+0000 7f26e3400700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c406ae00 0x7f26c406d2c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f26e415d250 tx=0x7f26d0008040 comp rx=0 tx=0).ready entity=mgr.9874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.438+0000 7f26dbe00700 1 -- 172.21.15.12:0/953408146 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26d40554c0 con 0x7f26e40b3f80 2024-03-29T11:28:18.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.683+0000 7f26eaa00700 1 -- 172.21.15.12:0/953408146 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26e40d27a0 con 0x7f26c406ae00 2024-03-29T11:28:18.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.692+0000 7f26dbe00700 1 -- 172.21.15.12:0/953408146 <== mgr.9874 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+376 (secure 0 0 0) 0x7f26d00106b0 con 0x7f26c406ae00 2024-03-29T11:28:18.682 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:18.683 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T11:12:44", "first": "2024-03-29T11:13:00", "last": "2024-03-29T11:28:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-03-29T11:28:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 -- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c406ae00 msgr2=0x7f26c406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c406ae00 0x7f26c406d2c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f26e415d250 tx=0x7f26d0008040 comp rx=0 tx=0).stop 2024-03-29T11:28:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 -- 172.21.15.12:0/953408146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e40b3f80 msgr2=0x7f26e414cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e40b3f80 0x7f26e414cbe0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009fd0 tx=0x7f26d4004750 comp rx=0 tx=0).stop 2024-03-29T11:28:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 -- 172.21.15.12:0/953408146 shutdown_connections 2024-03-29T11:28:18.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c406ae00 0x7f26c406d2c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e40b0970 0x7f26e40c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e40b3f80 0x7f26e414cbe0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 --2- 172.21.15.12:0/953408146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e4160e50 0x7f26e40b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 -- 172.21.15.12:0/953408146 >> 172.21.15.12:0/953408146 conn(0x7f26e408aff0 msgr2=0x7f26e415f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:18.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 -- 172.21.15.12:0/953408146 shutdown_connections 2024-03-29T11:28:18.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.694+0000 7f26eaa00700 1 -- 172.21.15.12:0/953408146 wait complete. 2024-03-29T11:28:18.703 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T11:12:44", "first": "2024-03-29T11:13:00", "last": "2024-03-29T11:28:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-03-29T11:28:18.703 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-03-29T00:00:00', 'created': '2024-03-29T11:12:44', 'first': '2024-03-29T11:13:00', 'last': '2024-03-29T11:28:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2024-03-29T11:28:18.703 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2024-03-29T11:28:18.703 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2024-03-29T11:28:18.703 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2024-03-29T11:28:18.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.914+0000 7f10be400700 1 Processor -- start 2024-03-29T11:28:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.914+0000 7f10be400700 1 -- start start 2024-03-29T11:28:18.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.915+0000 7f10be400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b80ced80 0x7f10b80d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.915+0000 7f10be400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f10b815fcb0 0x7f10b8160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.915+0000 7f10be400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b80cf1e0 con 0x7f10b815dad0 2024-03-29T11:28:18.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.915+0000 7f10be400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b80cfcc0 con 0x7f10b815fcb0 2024-03-29T11:28:18.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.915+0000 7f10be400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80cd9f0 con 0x7f10b80ced80 2024-03-29T11:28:18.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b7400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f10b815dad0 0x7f10b80d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52840/0 (socket says 172.21.15.12:52840) 2024-03-29T11:28:18.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b7400700 1 -- 172.21.15.12:0/3131144725 learned_addr learned my addr 172.21.15.12:0/3131144725 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:18.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6a00700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b80ced80 0x7f10b80d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b7e00700 1 --2- 172.21.15.12:0/3131144725 >> v2:172.21.15.159:3300/0 conn(0x7f10b815fcb0 0x7f10b8160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315818065 0 0) 0x7f10a8001110 con 0x7f10b815dad0 2024-03-29T11:28:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f109c0035b0 con 0x7f10b815dad0 2024-03-29T11:28:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536990243 0 0) 0x7f109c0035b0 con 0x7f10b815dad0 2024-03-29T11:28:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b80ced80 msgr2=0x7f10b80d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6000700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b80ced80 0x7f10b80d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 >> v2:172.21.15.159:3300/0 conn(0x7f10b815fcb0 msgr2=0x7f10b8160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6000700 1 --2- 172.21.15.12:0/3131144725 >> v2:172.21.15.159:3300/0 conn(0x7f10b815fcb0 0x7f10b8160140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b815f820 con 0x7f10b815dad0 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f10a8001800 con 0x7f10b815dad0 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.916+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 >> v1:172.21.15.12:6789/0 conn(0x7f10b815dad0 legacy=0x7f10b80d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b6000700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 0x7f109c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b6000700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f109c004850 0x7f109c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b6000700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b80ced80 0x7f109c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80cfcc0 con 0x7f109c003a00 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80cd9f0 con 0x7f109c004850 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f109c00ba10 con 0x7f10b80ced80 2024-03-29T11:28:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b7400700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 0x7f109c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b6a00700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f109c004850 0x7f109c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b7e00700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b80ced80 0x7f109c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b7400700 1 -- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b80ced80 msgr2=0x7f109c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b7400700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b80ced80 0x7f109c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b7400700 1 -- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f109c004850 msgr2=0x7f109c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b7400700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f109c004850 0x7f109c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b7400700 1 -- 172.21.15.12:0/3131144725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109c0035b0 con 0x7f109c003a00 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b7e00700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b80ced80 0x7f109c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b6a00700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f109c004850 0x7f109c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.917+0000 7f10b7400700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 0x7f109c003e20 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f10a800c240 tx=0x7f109c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3551c1ce3ffbc86 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.918+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8007db0 con 0x7f109c003a00 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.918+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a800ce00 con 0x7f109c003a00 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.918+0000 7f10b6000700 1 -- 172.21.15.12:0/3131144725 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8005ba0 con 0x7f109c003a00 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.918+0000 7f10be400700 1 -- 172.21.15.12:0/3131144725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 msgr2=0x7f109c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.918+0000 7f10be400700 1 --2- 172.21.15.12:0/3131144725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 0x7f109c003e20 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f10a800c240 tx=0x7f109c005880 comp rx=0 tx=0).stop 2024-03-29T11:28:18.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.918+0000 7f10b7e00700 1 -- 172.21.15.12:0/3131144725 reap_dead start 2024-03-29T11:28:18.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.918+0000 7f10be400700 1 -- 172.21.15.12:0/3131144725 shutdown_connections 2024-03-29T11:28:18.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.918+0000 7f10be400700 1 -- 172.21.15.12:0/3131144725 >> 172.21.15.12:0/3131144725 conn(0x7f10b808aff0 msgr2=0x7f10b80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:18.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.918+0000 7f10be400700 1 -- 172.21.15.12:0/3131144725 shutdown_connections 2024-03-29T11:28:18.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.918+0000 7f10be400700 1 -- 172.21.15.12:0/3131144725 wait complete. 2024-03-29T11:28:18.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.920+0000 7f10be400700 1 Processor -- start 2024-03-29T11:28:18.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.920+0000 7f10be400700 1 -- start start 2024-03-29T11:28:18.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.921+0000 7f10be400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 0x7f10b8151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.921+0000 7f10be400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b80b73c0 0x7f10b80c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.921+0000 7f10be400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b8153ab0 0x7f10b80cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.921+0000 7f10be400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80c8ab0 con 0x7f109c003a00 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.921+0000 7f10be400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80b83d0 con 0x7f10b80b73c0 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.921+0000 7f10be400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80b7850 con 0x7f10b8153ab0 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.921+0000 7f10b7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 0x7f10b8151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.921+0000 7f10b7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 0x7f10b8151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40918/0 (socket says 172.21.15.12:40918) 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.921+0000 7f10b7400700 1 -- 172.21.15.12:0/181307203 learned_addr learned my addr 172.21.15.12:0/181307203 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10b6a00700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b80b73c0 0x7f10b80c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10b7e00700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b8153ab0 0x7f10b80cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10b7400700 1 -- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b8153ab0 msgr2=0x7f10b80cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10b7400700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b8153ab0 0x7f10b80cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10b7400700 1 -- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b80b73c0 msgr2=0x7f10b80c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10b7400700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b80b73c0 0x7f10b80c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10b7400700 1 -- 172.21.15.12:0/181307203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80b7b30 con 0x7f109c003a00 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10b6a00700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b80b73c0 0x7f10b80c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10b7400700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 0x7f10b8151dc0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f109c0046d0 tx=0x7f10b806b0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:18.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10af400700 1 -- 172.21.15.12:0/181307203 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8004d40 con 0x7f109c003a00 2024-03-29T11:28:18.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10af400700 1 -- 172.21.15.12:0/181307203 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a80070d0 con 0x7f109c003a00 2024-03-29T11:28:18.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10be400700 1 -- 172.21.15.12:0/181307203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b80b7e10 con 0x7f109c003a00 2024-03-29T11:28:18.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.922+0000 7f10af400700 1 -- 172.21.15.12:0/181307203 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8004ea0 con 0x7f109c003a00 2024-03-29T11:28:18.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.923+0000 7f10be400700 1 -- 172.21.15.12:0/181307203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b8152300 con 0x7f109c003a00 2024-03-29T11:28:18.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.923+0000 7f10be400700 1 -- 172.21.15.12:0/181307203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b8067720 con 0x7f109c003a00 2024-03-29T11:28:18.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.925+0000 7f10af400700 1 -- 172.21.15.12:0/181307203 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 112) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f10a8019070 con 0x7f109c003a00 2024-03-29T11:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.925+0000 7f10af400700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f109806ae00 0x7f109806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.925+0000 7f10af400700 1 -- 172.21.15.12:0/181307203 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f10a8089ce0 con 0x7f109c003a00 2024-03-29T11:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.926+0000 7f10b6a00700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f109806ae00 0x7f109806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.927+0000 7f10b6a00700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f109806ae00 0x7f109806d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f10b8155900 tx=0x7f10a4008040 comp rx=0 tx=0).ready entity=mgr.9874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:18.930+0000 7f10af400700 1 -- 172.21.15.12:0/181307203 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f10a8054b10 con 0x7f109c003a00 2024-03-29T11:28:19.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.172+0000 7f10be400700 1 -- 172.21.15.12:0/181307203 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f10b80cf9e0 con 0x7f109806ae00 2024-03-29T11:28:19.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.253+0000 7f10af400700 1 -- 172.21.15.12:0/181307203 <== mgr.9874 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f10a4004020 con 0x7f109806ae00 2024-03-29T11:28:19.243 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path /snap_test_dir1/test_restart 2024-03-29T11:28:19.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 -- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f109806ae00 msgr2=0x7f109806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f109806ae00 0x7f109806d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f10b8155900 tx=0x7f10a4008040 comp rx=0 tx=0).stop 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 -- 172.21.15.12:0/181307203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 msgr2=0x7f10b8151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 0x7f10b8151dc0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f109c0046d0 tx=0x7f10b806b0a0 comp rx=0 tx=0).stop 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 -- 172.21.15.12:0/181307203 shutdown_connections 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f109806ae00 0x7f109806d2c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10b8153ab0 0x7f10b80cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f109c003a00 0x7f10b8151dc0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 --2- 172.21.15.12:0/181307203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10b80b73c0 0x7f10b80c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 -- 172.21.15.12:0/181307203 >> 172.21.15.12:0/181307203 conn(0x7f10b808aff0 msgr2=0x7f10b815beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 -- 172.21.15.12:0/181307203 shutdown_connections 2024-03-29T11:28:19.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:19.255+0000 7f10be400700 1 -- 172.21.15.12:0/181307203 wait complete. 2024-03-29T11:28:19.263 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2024-03-29T11:28:19.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.263 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-03-29T11:28:19.309 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_13_00_UTC 2024-03-29T11:28:19.309 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_14_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_15_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_16_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_17_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_18_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_19_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_20_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_21_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_22_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_23_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_24_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_25_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_26_00_UTC 2024-03-29T11:28:19.310 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_27_00_UTC 2024-03-29T11:28:19.311 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_28_00_UTC 2024-03-29T11:28:19.311 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_13_00_UTC 2024-03-29T11:28:19.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.312 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_13_00_UTC) 2024-03-29T11:28:19.403 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_14_00_UTC 2024-03-29T11:28:19.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.403 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_14_00_UTC) 2024-03-29T11:28:19.453 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_15_00_UTC 2024-03-29T11:28:19.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.453 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_15_00_UTC) 2024-03-29T11:28:19.545 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_16_00_UTC 2024-03-29T11:28:19.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.545 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_16_00_UTC) 2024-03-29T11:28:19.595 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_17_00_UTC 2024-03-29T11:28:19.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.596 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_17_00_UTC) 2024-03-29T11:28:19.646 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_18_00_UTC 2024-03-29T11:28:19.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.646 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_18_00_UTC) 2024-03-29T11:28:19.697 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_19_00_UTC 2024-03-29T11:28:19.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.697 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_19_00_UTC) 2024-03-29T11:28:19.748 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_20_00_UTC 2024-03-29T11:28:19.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.748 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_20_00_UTC) 2024-03-29T11:28:19.798 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_21_00_UTC 2024-03-29T11:28:19.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.799 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_21_00_UTC) 2024-03-29T11:28:19.848 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_22_00_UTC 2024-03-29T11:28:19.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.849 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_22_00_UTC) 2024-03-29T11:28:19.899 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_23_00_UTC 2024-03-29T11:28:19.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.900 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_23_00_UTC) 2024-03-29T11:28:19.950 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_24_00_UTC 2024-03-29T11:28:19.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:19.950 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_24_00_UTC) 2024-03-29T11:28:19.976 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:28:19.999 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:28:20.000 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_25_00_UTC 2024-03-29T11:28:20.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:20.001 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_25_00_UTC) 2024-03-29T11:28:20.092 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_26_00_UTC 2024-03-29T11:28:20.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:20.092 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_26_00_UTC) 2024-03-29T11:28:20.184 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_27_00_UTC 2024-03-29T11:28:20.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:20.184 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_27_00_UTC) 2024-03-29T11:28:20.236 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_28_00_UTC 2024-03-29T11:28:20.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:20.236 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-29-11_28_00_UTC) 2024-03-29T11:28:20.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:20.287 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2024-03-29T11:28:20.334 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T11:28:20.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.547+0000 7f5609800700 1 Processor -- start 2024-03-29T11:28:20.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.547+0000 7f5609800700 1 -- start start 2024-03-29T11:28:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.548+0000 7f5609800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f560415dfe0 0x7f56040ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.548+0000 7f5609800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 0x7f56040cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.548+0000 7f5609800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56040cb310 con 0x7f560415a5c0 2024-03-29T11:28:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.548+0000 7f5609800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56040ceb00 con 0x7f560415dfe0 2024-03-29T11:28:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.548+0000 7f5609800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56040711a0 con 0x7f56040ce670 2024-03-29T11:28:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.548+0000 7f5602000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f560415dfe0 0x7f56040ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.548+0000 7f5602a00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f560415a5c0 0x7f56040709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:53830/0 (socket says 172.21.15.12:53830) 2024-03-29T11:28:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.548+0000 7f5602000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f560415dfe0 0x7f56040ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:36266/0 (socket says 172.21.15.12:36266) 2024-03-29T11:28:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.548+0000 7f5603400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 0x7f56040cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.548+0000 7f5602a00700 1 -- 172.21.15.12:0/1584394410 learned_addr learned my addr 172.21.15.12:0/1584394410 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5601600700 1 -- 172.21.15.12:0/1584394410 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058398555 0 0) 0x7f56040cb310 con 0x7f560415a5c0 2024-03-29T11:28:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5601600700 1 -- 172.21.15.12:0/1584394410 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55e80035b0 con 0x7f560415a5c0 2024-03-29T11:28:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5603400700 1 -- 172.21.15.12:0/1584394410 >> v2:172.21.15.159:3300/0 conn(0x7f560415dfe0 msgr2=0x7f56040ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5603400700 1 --2- 172.21.15.12:0/1584394410 >> v2:172.21.15.159:3300/0 conn(0x7f560415dfe0 0x7f56040ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5603400700 1 -- 172.21.15.12:0/1584394410 >> v1:172.21.15.159:6789/0 conn(0x7f560415a5c0 legacy=0x7f56040709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5603400700 1 -- 172.21.15.12:0/1584394410 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5604071480 con 0x7f56040ce670 2024-03-29T11:28:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5602000700 1 --2- 172.21.15.12:0/1584394410 >> v2:172.21.15.159:3300/0 conn(0x7f560415dfe0 0x7f56040ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5603400700 1 --2- 172.21.15.12:0/1584394410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 0x7f56040cba80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009d90 tx=0x7f55fc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d86f4da184ffd32 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5601600700 1 -- 172.21.15.12:0/1584394410 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc00e070 con 0x7f56040ce670 2024-03-29T11:28:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5601600700 1 -- 172.21.15.12:0/1584394410 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55fc00bb20 con 0x7f56040ce670 2024-03-29T11:28:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.549+0000 7f5601600700 1 -- 172.21.15.12:0/1584394410 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc012430 con 0x7f56040ce670 2024-03-29T11:28:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.550+0000 7f5609800700 1 -- 172.21.15.12:0/1584394410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 msgr2=0x7f56040cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.550+0000 7f5609800700 1 --2- 172.21.15.12:0/1584394410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 0x7f56040cba80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009d90 tx=0x7f55fc00b330 comp rx=0 tx=0).stop 2024-03-29T11:28:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.550+0000 7f5609800700 1 -- 172.21.15.12:0/1584394410 shutdown_connections 2024-03-29T11:28:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.550+0000 7f5609800700 1 --2- 172.21.15.12:0/1584394410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 0x7f56040cba80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.550+0000 7f5609800700 1 --2- 172.21.15.12:0/1584394410 >> v2:172.21.15.159:3300/0 conn(0x7f560415dfe0 0x7f56040ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.550+0000 7f5609800700 1 -- 172.21.15.12:0/1584394410 >> 172.21.15.12:0/1584394410 conn(0x7f56040b0d70 msgr2=0x7f56040b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.550+0000 7f5609800700 1 -- 172.21.15.12:0/1584394410 shutdown_connections 2024-03-29T11:28:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.550+0000 7f5609800700 1 -- 172.21.15.12:0/1584394410 wait complete. 2024-03-29T11:28:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.552+0000 7f5609800700 1 Processor -- start 2024-03-29T11:28:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.552+0000 7f5609800700 1 -- start start 2024-03-29T11:28:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5609800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 0x7f560414eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5609800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560415a5c0 0x7f560414c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5609800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560415dfe0 0x7f56041554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5609800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560414f0c0 con 0x7f560415dfe0 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5609800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560414dae0 con 0x7f560415a5c0 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5609800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560414ddc0 con 0x7f56040ce670 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5603400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560415dfe0 0x7f56041554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5603400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560415dfe0 0x7f56041554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40922/0 (socket says 172.21.15.12:40922) 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5603400700 1 -- 172.21.15.12:0/206172087 learned_addr learned my addr 172.21.15.12:0/206172087 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5602000700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560415a5c0 0x7f560414c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.553+0000 7f5602a00700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 0x7f560414eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f5603400700 1 -- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 msgr2=0x7f560414eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f5603400700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 0x7f560414eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f5603400700 1 -- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560415a5c0 msgr2=0x7f560414c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f5603400700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560415a5c0 0x7f560414c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f5603400700 1 -- 172.21.15.12:0/206172087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f000b740 con 0x7f560415dfe0 2024-03-29T11:28:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f5602000700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560415a5c0 0x7f560414c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f5603400700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560415dfe0 0x7f56041554c0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f55fc014040 tx=0x7f55fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f55faa00700 1 -- 172.21.15.12:0/206172087 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc0129f0 con 0x7f560415dfe0 2024-03-29T11:28:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f55faa00700 1 -- 172.21.15.12:0/206172087 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55fc020070 con 0x7f560415dfe0 2024-03-29T11:28:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f5609800700 1 -- 172.21.15.12:0/206172087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc009a70 con 0x7f560415dfe0 2024-03-29T11:28:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f55faa00700 1 -- 172.21.15.12:0/206172087 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc012db0 con 0x7f560415dfe0 2024-03-29T11:28:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.554+0000 7f5609800700 1 -- 172.21.15.12:0/206172087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560414e380 con 0x7f560415dfe0 2024-03-29T11:28:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.555+0000 7f5609800700 1 -- 172.21.15.12:0/206172087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5604067720 con 0x7f560415dfe0 2024-03-29T11:28:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.556+0000 7f55faa00700 1 -- 172.21.15.12:0/206172087 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 112) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f55fc004040 con 0x7f560415dfe0 2024-03-29T11:28:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.557+0000 7f55faa00700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55e406ae00 0x7f55e406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.557+0000 7f55faa00700 1 -- 172.21.15.12:0/206172087 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f55fc022070 con 0x7f560415dfe0 2024-03-29T11:28:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.558+0000 7f5602a00700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55e406ae00 0x7f55e406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.558+0000 7f5602a00700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55e406ae00 0x7f55e406d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f55f4007fd0 tx=0x7f55f4007dc0 comp rx=0 tx=0).ready entity=mgr.9874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.562+0000 7f55faa00700 1 -- 172.21.15.12:0/206172087 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55fc056580 con 0x7f560415dfe0 2024-03-29T11:28:20.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:20.842+0000 7f5609800700 1 -- 172.21.15.12:0/206172087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f56040b1ad0 con 0x7f560415dfe0 2024-03-29T11:28:21.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.481+0000 7f55faa00700 1 -- 172.21.15.12:0/206172087 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f55fc056950 con 0x7f560415dfe0 2024-03-29T11:28:21.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.482+0000 7f55faa00700 1 -- 172.21.15.12:0/206172087 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v113) v1 ==== 93+0+0 (secure 0 0 0) 0x7f55fc05dab0 con 0x7f560415dfe0 2024-03-29T11:28:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 -- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55e406ae00 msgr2=0x7f55e406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55e406ae00 0x7f55e406d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f55f4007fd0 tx=0x7f55f4007dc0 comp rx=0 tx=0).stop 2024-03-29T11:28:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 -- 172.21.15.12:0/206172087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560415dfe0 msgr2=0x7f56041554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560415dfe0 0x7f56041554c0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f55fc014040 tx=0x7f55fc004ad0 comp rx=0 tx=0).stop 2024-03-29T11:28:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 -- 172.21.15.12:0/206172087 shutdown_connections 2024-03-29T11:28:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55e406ae00 0x7f55e406d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f560415dfe0 0x7f56041554c0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56040ce670 0x7f560414eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 --2- 172.21.15.12:0/206172087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f560415a5c0 0x7f560414c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 -- 172.21.15.12:0/206172087 >> 172.21.15.12:0/206172087 conn(0x7f56040b0d70 msgr2=0x7f560406dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 -- 172.21.15.12:0/206172087 shutdown_connections 2024-03-29T11:28:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:21.486+0000 7f5609800700 1 -- 172.21.15.12:0/206172087 wait complete. 2024-03-29T11:28:21.510 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:28:21.511 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:28:21.691 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:28:21.779 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:28:21.779 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:28:21.779 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:28:21.783 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:21.793+0000 7f7ab52c7240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:28:21.805 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:21.814+0000 7fed302d6240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:28:21.825 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:28:21.825 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:28:21.913 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:21.969 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:21.979+0000 7f7ab52c7240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:28:22.011 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:22.018+0000 7fed302d6240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:28:22.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.191+0000 7f6e81200700 1 Processor -- start 2024-03-29T11:28:22.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.191+0000 7f6e81200700 1 -- start start 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.192+0000 7f6e81200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6e7c160c90 0x7f6e7c0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.192+0000 7f6e81200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6e7c0ad970 0x7f6e7c0b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.192+0000 7f6e81200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7c0ac4d0 con 0x7f6e7c0718e0 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.192+0000 7f6e81200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7c071e50 con 0x7f6e7c160c90 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.192+0000 7f6e81200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7c072130 con 0x7f6e7c0ad970 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.192+0000 7f6e79e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6e7c160c90 0x7f6e7c0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.192+0000 7f6e79e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6e7c160c90 0x7f6e7c0b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40944/0 (socket says 172.21.15.12:40944) 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.192+0000 7f6e79e00700 1 -- 172.21.15.12:0/1412166242 learned_addr learned my addr 172.21.15.12:0/1412166242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.192+0000 7f6e79400700 1 --1- 172.21.15.12:0/1412166242 >> v1:172.21.15.12:6789/0 conn(0x7f6e7c0718e0 0x7f6e7c06cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52854/0 (socket says 172.21.15.12:52854) 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.193+0000 7f6e7a800700 1 --2- 172.21.15.12:0/1412166242 >> v2:172.21.15.159:3300/0 conn(0x7f6e7c0ad970 0x7f6e7c0b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.193+0000 7f6e79e00700 1 -- 172.21.15.12:0/1412166242 >> v2:172.21.15.159:3300/0 conn(0x7f6e7c0ad970 msgr2=0x7f6e7c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.193+0000 7f6e79e00700 1 --2- 172.21.15.12:0/1412166242 >> v2:172.21.15.159:3300/0 conn(0x7f6e7c0ad970 0x7f6e7c0b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.193+0000 7f6e79e00700 1 -- 172.21.15.12:0/1412166242 >> v1:172.21.15.12:6789/0 conn(0x7f6e7c0718e0 legacy=0x7f6e7c06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.193+0000 7f6e79e00700 1 -- 172.21.15.12:0/1412166242 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c072410 con 0x7f6e7c160c90 2024-03-29T11:28:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.193+0000 7f6e79e00700 1 --2- 172.21.15.12:0/1412166242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7c160c90 0x7f6e7c0b20e0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c071d60 tx=0x7f6e6c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d00d297ee015966 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.193+0000 7f6e78a00700 1 -- 172.21.15.12:0/1412166242 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c00b870 con 0x7f6e7c160c90 2024-03-29T11:28:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.193+0000 7f6e78a00700 1 -- 172.21.15.12:0/1412166242 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c00b9d0 con 0x7f6e7c160c90 2024-03-29T11:28:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.193+0000 7f6e78a00700 1 -- 172.21.15.12:0/1412166242 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c00bd60 con 0x7f6e7c160c90 2024-03-29T11:28:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.194+0000 7f6e81200700 1 -- 172.21.15.12:0/1412166242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7c160c90 msgr2=0x7f6e7c0b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.194+0000 7f6e81200700 1 --2- 172.21.15.12:0/1412166242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7c160c90 0x7f6e7c0b20e0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c071d60 tx=0x7f6e6c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.194+0000 7f6e81200700 1 -- 172.21.15.12:0/1412166242 shutdown_connections 2024-03-29T11:28:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.194+0000 7f6e81200700 1 --2- 172.21.15.12:0/1412166242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7c160c90 0x7f6e7c0b20e0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.194+0000 7f6e81200700 1 --2- 172.21.15.12:0/1412166242 >> v2:172.21.15.159:3300/0 conn(0x7f6e7c0ad970 0x7f6e7c0b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.194+0000 7f6e81200700 1 -- 172.21.15.12:0/1412166242 >> 172.21.15.12:0/1412166242 conn(0x7f6e7c0cf190 msgr2=0x7f6e7c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.194+0000 7f6e81200700 1 -- 172.21.15.12:0/1412166242 shutdown_connections 2024-03-29T11:28:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.194+0000 7f6e81200700 1 -- 172.21.15.12:0/1412166242 wait complete. 2024-03-29T11:28:22.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.196+0000 7f6e81200700 1 Processor -- start 2024-03-29T11:28:22.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.196+0000 7f6e81200700 1 -- start start 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e81200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e7c0718e0 0x7f6e7c156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e81200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e7c0ad970 0x7f6e7c154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e81200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7c160c90 0x7f6e7c09cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e81200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0991d0 con 0x7f6e7c160c90 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e81200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c155f20 con 0x7f6e7c0718e0 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e81200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c157c40 con 0x7f6e7c0ad970 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e79e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e7c0718e0 0x7f6e7c156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e7a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7c160c90 0x7f6e7c09cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e79400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e7c0ad970 0x7f6e7c154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e79e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e7c0718e0 0x7f6e7c156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:36312/0 (socket says 172.21.15.12:36312) 2024-03-29T11:28:22.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.197+0000 7f6e79e00700 1 -- 172.21.15.12:0/3652622204 learned_addr learned my addr 172.21.15.12:0/3652622204 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:22.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.198+0000 7f6e79e00700 1 -- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e7c0ad970 msgr2=0x7f6e7c154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.198+0000 7f6e79e00700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e7c0ad970 0x7f6e7c154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.198+0000 7f6e79e00700 1 -- 172.21.15.12:0/3652622204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7c160c90 msgr2=0x7f6e7c09cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.198+0000 7f6e79e00700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7c160c90 0x7f6e7c09cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.198+0000 7f6e79e00700 1 -- 172.21.15.12:0/3652622204 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c009550 con 0x7f6e7c0718e0 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.198+0000 7f6e7a800700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7c160c90 0x7f6e7c09cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.198+0000 7f6e79e00700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e7c0718e0 0x7f6e7c156280 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c000c00 tx=0x7f6e6c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.198+0000 7f6e72000700 1 -- 172.21.15.12:0/3652622204 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c004310 con 0x7f6e7c0718e0 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.199+0000 7f6e72000700 1 -- 172.21.15.12:0/3652622204 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c01e070 con 0x7f6e7c0718e0 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.199+0000 7f6e72000700 1 -- 172.21.15.12:0/3652622204 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c0036a0 con 0x7f6e7c0718e0 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.199+0000 7f6e81200700 1 -- 172.21.15.12:0/3652622204 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7c1553e0 con 0x7f6e7c0718e0 2024-03-29T11:28:22.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.199+0000 7f6e81200700 1 -- 172.21.15.12:0/3652622204 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c155680 con 0x7f6e7c0718e0 2024-03-29T11:28:22.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.200+0000 7f6e81200700 1 -- 172.21.15.12:0/3652622204 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7c157a10 con 0x7f6e7c0718e0 2024-03-29T11:28:22.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.201+0000 7f6e72000700 1 -- 172.21.15.12:0/3652622204 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f6e6c020030 con 0x7f6e7c0718e0 2024-03-29T11:28:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.201+0000 7f6e72000700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6e5c06ade0 0x7f6e5c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.201+0000 7f6e72000700 1 -- 172.21.15.12:0/3652622204 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6443+0+0 (secure 0 0 0) 0x7f6e6c08a6f0 con 0x7f6e7c0718e0 2024-03-29T11:28:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.202+0000 7f6e79400700 1 -- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6e5c06ade0 msgr2=0x7f6e5c06d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.202+0000 7f6e79400700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6e5c06ade0 0x7f6e5c06d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.207+0000 7f6e72000700 1 -- 172.21.15.12:0/3652622204 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e7c157a10 con 0x7f6e7c0718e0 2024-03-29T11:28:22.285 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:22.295+0000 7f7ab52c7240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:28:22.356 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:22.366+0000 7fed302d6240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:28:22.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.403+0000 7f6e79400700 1 -- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6e5c06ade0 msgr2=0x7f6e5c06d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:22.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.403+0000 7f6e79400700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6e5c06ade0 0x7f6e5c06d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:22.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.433+0000 7f6e81200700 1 -- 172.21.15.12:0/3652622204 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e7c157a10 con 0x7f6e7c0718e0 2024-03-29T11:28:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.435+0000 7f6e72000700 1 -- 172.21.15.12:0/3652622204 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) v1 ==== 74+0+17212 (secure 0 0 0) 0x7f6e7c157a10 con 0x7f6e7c0718e0 2024-03-29T11:28:22.425 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:22.425 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":194,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:28:21.476107+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:12:24.038574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-03-29T11:12:25.509384+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":32,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3631785847":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/750511589":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/2791060580":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/479652442":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/180231134":"2024-03-30T11:12:43.757966+0000","172.21.15.159:6833/43444":"2024-03-30T11:12:43.757966+0000","172.21.15.159:6832/43444":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/2825856150":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3819915206":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/2176927052":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3123015140":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/1699207236":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3816982205":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/4147178761":"2024-03-30T11:12:43.757966+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":15},{"begin":18,"length":15}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":2,"length":15},{"begin":18,"length":15}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.437+0000 7f6e81200700 1 -- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6e5c06ade0 msgr2=0x7f6e5c06d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.437+0000 7f6e81200700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6e5c06ade0 0x7f6e5c06d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.437+0000 7f6e81200700 1 -- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e7c0718e0 msgr2=0x7f6e7c156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.437+0000 7f6e81200700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e7c0718e0 0x7f6e7c156280 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c000c00 tx=0x7f6e6c004750 comp rx=0 tx=0).stop 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.437+0000 7f6e81200700 1 -- 172.21.15.12:0/3652622204 shutdown_connections 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.437+0000 7f6e81200700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6e5c06ade0 0x7f6e5c06d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.437+0000 7f6e81200700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7c160c90 0x7f6e7c09cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.437+0000 7f6e81200700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e7c0718e0 0x7f6e7c156280 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.437+0000 7f6e81200700 1 --2- 172.21.15.12:0/3652622204 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e7c0ad970 0x7f6e7c154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.437+0000 7f6e81200700 1 -- 172.21.15.12:0/3652622204 >> 172.21.15.12:0/3652622204 conn(0x7f6e7c0cf190 msgr2=0x7f6e7c06dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.438+0000 7f6e81200700 1 -- 172.21.15.12:0/3652622204 shutdown_connections 2024-03-29T11:28:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.438+0000 7f6e81200700 1 -- 172.21.15.12:0/3652622204 wait complete. 2024-03-29T11:28:22.446 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:12:24.038574+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '190', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:12:25.509384+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 32, 'snap_epoch': 194, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:28:22.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:28:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.666+0000 7f5e7fc00700 1 Processor -- start 2024-03-29T11:28:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.666+0000 7f5e7fc00700 1 -- start start 2024-03-29T11:28:22.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7fc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5e7815d9d0 0x7f5e780d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7fc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e780d2540 0x7f5e780ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7fc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e78161420 con 0x7f5e780d16a0 2024-03-29T11:28:22.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7fc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e7815edc0 con 0x7f5e7815d9d0 2024-03-29T11:28:22.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7fc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e78161700 con 0x7f5e780d2540 2024-03-29T11:28:22.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7d200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5e7815d9d0 0x7f5e780d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:22.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7d200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5e7815d9d0 0x7f5e780d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40972/0 (socket says 172.21.15.12:40972) 2024-03-29T11:28:22.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7d200700 1 -- 172.21.15.12:0/3708765824 learned_addr learned my addr 172.21.15.12:0/3708765824 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:22.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e77e00700 1 --1- 172.21.15.12:0/3708765824 >> v1:172.21.15.12:6789/0 conn(0x7f5e780d16a0 0x7f5e780d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52862/0 (socket says 172.21.15.12:52862) 2024-03-29T11:28:22.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7dc00700 1 --2- 172.21.15.12:0/3708765824 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e780d2540 0x7f5e780ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:22.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7d200700 1 -- 172.21.15.12:0/3708765824 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e780d2540 msgr2=0x7f5e780ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:22.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7d200700 1 --2- 172.21.15.12:0/3708765824 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e780d2540 0x7f5e780ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7d200700 1 -- 172.21.15.12:0/3708765824 >> v1:172.21.15.12:6789/0 conn(0x7f5e780d16a0 legacy=0x7f5e780d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.668+0000 7f5e7d200700 1 -- 172.21.15.12:0/3708765824 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e781619e0 con 0x7f5e7815d9d0 2024-03-29T11:28:22.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.669+0000 7f5e7d200700 1 --2- 172.21.15.12:0/3708765824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 0x7f5e780d3c30 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0098f0 tx=0x7f5e6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94426a0057734944 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:22.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.669+0000 7f5e77400700 1 -- 172.21.15.12:0/3708765824 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c011830 con 0x7f5e7815d9d0 2024-03-29T11:28:22.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.669+0000 7f5e77400700 1 -- 172.21.15.12:0/3708765824 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c011990 con 0x7f5e7815d9d0 2024-03-29T11:28:22.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.669+0000 7f5e77400700 1 -- 172.21.15.12:0/3708765824 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c011d20 con 0x7f5e7815d9d0 2024-03-29T11:28:22.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.670+0000 7f5e7fc00700 1 -- 172.21.15.12:0/3708765824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 msgr2=0x7f5e780d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.670+0000 7f5e7fc00700 1 --2- 172.21.15.12:0/3708765824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 0x7f5e780d3c30 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0098f0 tx=0x7f5e6c009c00 comp rx=0 tx=0).stop 2024-03-29T11:28:22.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.670+0000 7f5e7fc00700 1 -- 172.21.15.12:0/3708765824 shutdown_connections 2024-03-29T11:28:22.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.670+0000 7f5e7fc00700 1 --2- 172.21.15.12:0/3708765824 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e780d2540 0x7f5e780ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.670+0000 7f5e7fc00700 1 --2- 172.21.15.12:0/3708765824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 0x7f5e780d3c30 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.670+0000 7f5e7fc00700 1 -- 172.21.15.12:0/3708765824 >> 172.21.15.12:0/3708765824 conn(0x7f5e7808aff0 msgr2=0x7f5e780d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.670+0000 7f5e7fc00700 1 -- 172.21.15.12:0/3708765824 shutdown_connections 2024-03-29T11:28:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.670+0000 7f5e7fc00700 1 -- 172.21.15.12:0/3708765824 wait complete. 2024-03-29T11:28:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.671+0000 7f5e7fc00700 1 Processor -- start 2024-03-29T11:28:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.672+0000 7f5e7fc00700 1 -- start start 2024-03-29T11:28:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7fc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e780d16a0 0x7f5e781e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7fc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e780d2540 0x7f5e7814f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7fc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 0x7f5e780b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7fc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e780cb5a0 con 0x7f5e7815d9d0 2024-03-29T11:28:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7fc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e781e82e0 con 0x7f5e780d16a0 2024-03-29T11:28:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7fc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e781e85c0 con 0x7f5e780d2540 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 0x7f5e780b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7d200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e780d16a0 0x7f5e781e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 0x7f5e780b7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40988/0 (socket says 172.21.15.12:40988) 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7dc00700 1 -- 172.21.15.12:0/2410833712 learned_addr learned my addr 172.21.15.12:0/2410833712 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e77e00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e780d2540 0x7f5e7814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7d200700 1 -- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e780d2540 msgr2=0x7f5e7814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7d200700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e780d2540 0x7f5e7814f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7d200700 1 -- 172.21.15.12:0/2410833712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 msgr2=0x7f5e780b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7d200700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 0x7f5e780b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7d200700 1 -- 172.21.15.12:0/2410833712 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c009550 con 0x7f5e780d16a0 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.673+0000 7f5e7dc00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 0x7f5e780b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.674+0000 7f5e7d200700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e780d16a0 0x7f5e781e7c90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c016000 tx=0x7f5e6c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.674+0000 7f5e74c00700 1 -- 172.21.15.12:0/2410833712 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c004450 con 0x7f5e780d16a0 2024-03-29T11:28:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.674+0000 7f5e74c00700 1 -- 172.21.15.12:0/2410833712 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c01e070 con 0x7f5e780d16a0 2024-03-29T11:28:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.674+0000 7f5e7fc00700 1 -- 172.21.15.12:0/2410833712 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e781e88a0 con 0x7f5e780d16a0 2024-03-29T11:28:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.674+0000 7f5e74c00700 1 -- 172.21.15.12:0/2410833712 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c0038e0 con 0x7f5e780d16a0 2024-03-29T11:28:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.674+0000 7f5e7fc00700 1 -- 172.21.15.12:0/2410833712 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7814fd90 con 0x7f5e780d16a0 2024-03-29T11:28:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.675+0000 7f5e7fc00700 1 -- 172.21.15.12:0/2410833712 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e780d1ca0 con 0x7f5e780d16a0 2024-03-29T11:28:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.676+0000 7f5e74c00700 1 -- 172.21.15.12:0/2410833712 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f5e6c003a40 con 0x7f5e780d16a0 2024-03-29T11:28:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.677+0000 7f5e74c00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e5806adb0 0x7f5e5806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.677+0000 7f5e74c00700 1 -- 172.21.15.12:0/2410833712 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f5e6c08a7d0 con 0x7f5e780d16a0 2024-03-29T11:28:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.677+0000 7f5e77e00700 1 -- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e5806adb0 msgr2=0x7f5e5806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.677+0000 7f5e77e00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e5806adb0 0x7f5e5806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.682+0000 7f5e74c00700 1 -- 172.21.15.12:0/2410833712 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e6c0555c0 con 0x7f5e780d16a0 2024-03-29T11:28:22.827 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:22.837+0000 7f7ab52c7240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:28:22.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.878+0000 7f5e77e00700 1 -- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e5806adb0 msgr2=0x7f5e5806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:22.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.878+0000 7f5e77e00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e5806adb0 0x7f5e5806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.902+0000 7f5e7fc00700 1 -- 172.21.15.12:0/2410833712 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e780cb3e0 con 0x7f5e780d16a0 2024-03-29T11:28:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.903+0000 7f5e74c00700 1 -- 172.21.15.12:0/2410833712 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v195) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5e780cb3e0 con 0x7f5e780d16a0 2024-03-29T11:28:22.892 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:28:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 -- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e5806adb0 msgr2=0x7f5e5806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e5806adb0 0x7f5e5806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 -- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e780d16a0 msgr2=0x7f5e781e7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e780d16a0 0x7f5e781e7c90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c016000 tx=0x7f5e6c004750 comp rx=0 tx=0).stop 2024-03-29T11:28:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 -- 172.21.15.12:0/2410833712 shutdown_connections 2024-03-29T11:28:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e5806adb0 0x7f5e5806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e7815d9d0 0x7f5e780b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e780d16a0 0x7f5e781e7c90 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 --2- 172.21.15.12:0/2410833712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e780d2540 0x7f5e7814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:22.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 -- 172.21.15.12:0/2410833712 >> 172.21.15.12:0/2410833712 conn(0x7f5e7808aff0 msgr2=0x7f5e780b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:22.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 -- 172.21.15.12:0/2410833712 shutdown_connections 2024-03-29T11:28:22.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:22.905+0000 7f5e7fc00700 1 -- 172.21.15.12:0/2410833712 wait complete. 2024-03-29T11:28:22.913 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:28:22.938 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:22.949+0000 7fed302d6240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:28:23.083 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:23.093+0000 7f7ab52c7240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:28:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.126+0000 7f2201200700 1 Processor -- start 2024-03-29T11:28:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.126+0000 7f2201200700 1 -- start start 2024-03-29T11:28:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.127+0000 7f2201200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 0x7f21fc0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.127+0000 7f2201200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f21fc0cd020 0x7f21fc0cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.127+0000 7f2201200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc0cc450 con 0x7f21fc0b2070 2024-03-29T11:28:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.127+0000 7f2201200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21fc0cdae0 con 0x7f21fc0cd020 2024-03-29T11:28:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.127+0000 7f2201200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0cddc0 con 0x7f21fc0cbb30 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21f9400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f21fc0cd020 0x7f21fc0cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21fa800700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f21fc0b2070 0x7f21fc06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:53836/0 (socket says 172.21.15.12:53836) 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21f9400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f21fc0cd020 0x7f21fc0cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:36326/0 (socket says 172.21.15.12:36326) 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21fa800700 1 -- 172.21.15.12:0/3069328747 learned_addr learned my addr 172.21.15.12:0/3069328747 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21f9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 0x7f21fc0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21f8a00700 1 -- 172.21.15.12:0/3069328747 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602604716 0 0) 0x7f21fc0cc450 con 0x7f21fc0b2070 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21f9e00700 1 -- 172.21.15.12:0/3069328747 >> v2:172.21.15.159:3300/0 conn(0x7f21fc0cd020 msgr2=0x7f21fc0cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21f9e00700 1 --2- 172.21.15.12:0/3069328747 >> v2:172.21.15.159:3300/0 conn(0x7f21fc0cd020 0x7f21fc0cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21f9e00700 1 -- 172.21.15.12:0/3069328747 >> v1:172.21.15.159:6789/0 conn(0x7f21fc0b2070 legacy=0x7f21fc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21f9e00700 1 -- 172.21.15.12:0/3069328747 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc0ce0a0 con 0x7f21fc0cbb30 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21f9400700 1 --2- 172.21.15.12:0/3069328747 >> v2:172.21.15.159:3300/0 conn(0x7f21fc0cd020 0x7f21fc0cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.128+0000 7f21f9e00700 1 --2- 172.21.15.12:0/3069328747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 0x7f21fc0cbf10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009840 tx=0x7f21ec009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f14291dfcde7945c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f21f8a00700 1 -- 172.21.15.12:0/3069328747 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec011830 con 0x7f21fc0cbb30 2024-03-29T11:28:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f21f8a00700 1 -- 172.21.15.12:0/3069328747 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec011990 con 0x7f21fc0cbb30 2024-03-29T11:28:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f21f8a00700 1 -- 172.21.15.12:0/3069328747 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec011d20 con 0x7f21fc0cbb30 2024-03-29T11:28:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f2201200700 1 -- 172.21.15.12:0/3069328747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 msgr2=0x7f21fc0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f2201200700 1 --2- 172.21.15.12:0/3069328747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 0x7f21fc0cbf10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009840 tx=0x7f21ec009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f2201200700 1 -- 172.21.15.12:0/3069328747 shutdown_connections 2024-03-29T11:28:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f2201200700 1 --2- 172.21.15.12:0/3069328747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 0x7f21fc0cbf10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f2201200700 1 --2- 172.21.15.12:0/3069328747 >> v2:172.21.15.159:3300/0 conn(0x7f21fc0cd020 0x7f21fc0cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f2201200700 1 -- 172.21.15.12:0/3069328747 >> 172.21.15.12:0/3069328747 conn(0x7f21fc0b0d70 msgr2=0x7f21fc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f2201200700 1 -- 172.21.15.12:0/3069328747 shutdown_connections 2024-03-29T11:28:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.129+0000 7f2201200700 1 -- 172.21.15.12:0/3069328747 wait complete. 2024-03-29T11:28:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.131+0000 7f2201200700 1 Processor -- start 2024-03-29T11:28:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.131+0000 7f2201200700 1 -- start start 2024-03-29T11:28:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.132+0000 7f2201200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21fc0b2070 0x7f21fc14a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.132+0000 7f2201200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 0x7f21fc0b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.132+0000 7f2201200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21fc0cd020 0x7f21fc0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.132+0000 7f2201200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc154850 con 0x7f21fc0cd020 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.132+0000 7f2201200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc155640 con 0x7f21fc0b2070 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.132+0000 7f2201200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc14a980 con 0x7f21fc0cbb30 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.132+0000 7f21f9e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21fc0b2070 0x7f21fc14a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21f9e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21fc0b2070 0x7f21fc14a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:36328/0 (socket says 172.21.15.12:36328) 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21fa800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21fc0cd020 0x7f21fc0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21f9e00700 1 -- 172.21.15.12:0/1837683427 learned_addr learned my addr 172.21.15.12:0/1837683427 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21f9400700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 0x7f21fc0b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21f9e00700 1 -- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 msgr2=0x7f21fc0b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21f9e00700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 0x7f21fc0b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21f9e00700 1 -- 172.21.15.12:0/1837683427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21fc0cd020 msgr2=0x7f21fc0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21f9e00700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21fc0cd020 0x7f21fc0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21f9e00700 1 -- 172.21.15.12:0/1837683427 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e80094a0 con 0x7f21fc0b2070 2024-03-29T11:28:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21f9e00700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21fc0b2070 0x7f21fc14a440 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009b90 tx=0x7f21ec004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.133+0000 7f21f2000700 1 -- 172.21.15.12:0/1837683427 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec0036a0 con 0x7f21fc0b2070 2024-03-29T11:28:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.134+0000 7f21f2000700 1 -- 172.21.15.12:0/1837683427 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec01e070 con 0x7f21fc0b2070 2024-03-29T11:28:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.134+0000 7f21f2000700 1 -- 172.21.15.12:0/1837683427 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec003a60 con 0x7f21fc0b2070 2024-03-29T11:28:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.134+0000 7f2201200700 1 -- 172.21.15.12:0/1837683427 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ec0094a0 con 0x7f21fc0b2070 2024-03-29T11:28:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.134+0000 7f2201200700 1 -- 172.21.15.12:0/1837683427 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21fc154d10 con 0x7f21fc0b2070 2024-03-29T11:28:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.135+0000 7f2201200700 1 -- 172.21.15.12:0/1837683427 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21fc067720 con 0x7f21fc0b2070 2024-03-29T11:28:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.136+0000 7f21f2000700 1 -- 172.21.15.12:0/1837683427 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f21ec003bc0 con 0x7f21fc0b2070 2024-03-29T11:28:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.136+0000 7f21f2000700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21dc06adb0 0x7f21dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.137+0000 7f21f2000700 1 -- 172.21.15.12:0/1837683427 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f21ec08a640 con 0x7f21fc0b2070 2024-03-29T11:28:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.137+0000 7f21f9400700 1 -- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21dc06adb0 msgr2=0x7f21dc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.137+0000 7f21f9400700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21dc06adb0 0x7f21dc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:23.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.141+0000 7f21f2000700 1 -- 172.21.15.12:0/1837683427 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21ec055430 con 0x7f21fc0b2070 2024-03-29T11:28:23.214 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:23.225+0000 7fed302d6240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:28:23.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.338+0000 7f21f9400700 1 -- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21dc06adb0 msgr2=0x7f21dc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:23.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.338+0000 7f21f9400700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21dc06adb0 0x7f21dc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:23.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.357+0000 7f2201200700 1 -- 172.21.15.12:0/1837683427 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f21fc15bda0 con 0x7f21fc0b2070 2024-03-29T11:28:23.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.358+0000 7f21f2000700 1 -- 172.21.15.12:0/1837683427 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v195) v1 ==== 104+0+11 (secure 0 0 0) 0x7f21ec058a50 con 0x7f21fc0b2070 2024-03-29T11:28:23.348 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:28:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.360+0000 7f2201200700 1 -- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21dc06adb0 msgr2=0x7f21dc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.360+0000 7f2201200700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21dc06adb0 0x7f21dc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.360+0000 7f2201200700 1 -- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21fc0b2070 msgr2=0x7f21fc14a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.360+0000 7f2201200700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21fc0b2070 0x7f21fc14a440 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009b90 tx=0x7f21ec004750 comp rx=0 tx=0).stop 2024-03-29T11:28:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.361+0000 7f2201200700 1 -- 172.21.15.12:0/1837683427 shutdown_connections 2024-03-29T11:28:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.361+0000 7f2201200700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21dc06adb0 0x7f21dc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.361+0000 7f2201200700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21fc0cd020 0x7f21fc0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.361+0000 7f2201200700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21fc0b2070 0x7f21fc14a440 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.361+0000 7f2201200700 1 --2- 172.21.15.12:0/1837683427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21fc0cbb30 0x7f21fc0b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.361+0000 7f2201200700 1 -- 172.21.15.12:0/1837683427 >> 172.21.15.12:0/1837683427 conn(0x7f21fc0b0d70 msgr2=0x7f21fc15c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:23.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.361+0000 7f2201200700 1 -- 172.21.15.12:0/1837683427 shutdown_connections 2024-03-29T11:28:23.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.361+0000 7f2201200700 1 -- 172.21.15.12:0/1837683427 wait complete. 2024-03-29T11:28:23.369 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:28:23.544 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:23.555+0000 7f7ab52c7240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:28:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.625+0000 7fe7ca400700 1 Processor -- start 2024-03-29T11:28:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.626+0000 7fe7ca400700 1 -- start start 2024-03-29T11:28:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.627+0000 7fe7ca400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7c4099a60 0x7fe7c40d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.627+0000 7fe7ca400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c415d990 0x7fe7c4099e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.627+0000 7fe7ca400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c40d0970 con 0x7fe7c409a750 2024-03-29T11:28:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.627+0000 7fe7ca400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c406c020 con 0x7fe7c4099a60 2024-03-29T11:28:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.627+0000 7fe7ca400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c409a380 con 0x7fe7c415d990 2024-03-29T11:28:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.627+0000 7fe7c3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7c4099a60 0x7fe7c40d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.627+0000 7fe7c3e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe7c409a750 0x7fe7c406c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52878/0 (socket says 172.21.15.12:52878) 2024-03-29T11:28:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.627+0000 7fe7c3e00700 1 -- 172.21.15.12:0/2104932146 learned_addr learned my addr 172.21.15.12:0/2104932146 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.627+0000 7fe7c2a00700 1 --2- 172.21.15.12:0/2104932146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c415d990 0x7fe7c4099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.628+0000 7fe7c3400700 1 -- 172.21.15.12:0/2104932146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c415d990 msgr2=0x7fe7c4099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.628+0000 7fe7c3400700 1 --2- 172.21.15.12:0/2104932146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c415d990 0x7fe7c4099e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.628+0000 7fe7c3400700 1 -- 172.21.15.12:0/2104932146 >> v1:172.21.15.12:6789/0 conn(0x7fe7c409a750 legacy=0x7fe7c406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.628+0000 7fe7c3400700 1 -- 172.21.15.12:0/2104932146 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40c81a0 con 0x7fe7c4099a60 2024-03-29T11:28:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.628+0000 7fe7c3400700 1 --2- 172.21.15.12:0/2104932146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c4099a60 0x7fe7c40d0060 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009840 tx=0x7fe7b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c3c311756904ac server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.628+0000 7fe7c2000700 1 -- 172.21.15.12:0/2104932146 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4011830 con 0x7fe7c4099a60 2024-03-29T11:28:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.628+0000 7fe7c2000700 1 -- 172.21.15.12:0/2104932146 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b4011990 con 0x7fe7c4099a60 2024-03-29T11:28:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.628+0000 7fe7c2000700 1 -- 172.21.15.12:0/2104932146 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4011d20 con 0x7fe7c4099a60 2024-03-29T11:28:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.629+0000 7fe7ca400700 1 -- 172.21.15.12:0/2104932146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c4099a60 msgr2=0x7fe7c40d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.629+0000 7fe7ca400700 1 --2- 172.21.15.12:0/2104932146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c4099a60 0x7fe7c40d0060 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009840 tx=0x7fe7b4009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.629+0000 7fe7ca400700 1 -- 172.21.15.12:0/2104932146 shutdown_connections 2024-03-29T11:28:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.629+0000 7fe7ca400700 1 --2- 172.21.15.12:0/2104932146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c4099a60 0x7fe7c40d0060 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.629+0000 7fe7ca400700 1 --2- 172.21.15.12:0/2104932146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c415d990 0x7fe7c4099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.629+0000 7fe7ca400700 1 -- 172.21.15.12:0/2104932146 >> 172.21.15.12:0/2104932146 conn(0x7fe7c40b2e20 msgr2=0x7fe7c40b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.629+0000 7fe7ca400700 1 -- 172.21.15.12:0/2104932146 shutdown_connections 2024-03-29T11:28:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.629+0000 7fe7ca400700 1 -- 172.21.15.12:0/2104932146 wait complete. 2024-03-29T11:28:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.632+0000 7fe7ca400700 1 Processor -- start 2024-03-29T11:28:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.632+0000 7fe7ca400700 1 -- start start 2024-03-29T11:28:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.633+0000 7fe7ca400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7c4099a60 0x7fe7c4153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.633+0000 7fe7ca400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c409a750 0x7fe7c415eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.633+0000 7fe7ca400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c415d990 0x7fe7c40ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.633+0000 7fe7ca400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c40c7940 con 0x7fe7c409a750 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.633+0000 7fe7ca400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c40c9ad0 con 0x7fe7c4099a60 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.633+0000 7fe7ca400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c40c4f30 con 0x7fe7c415d990 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.633+0000 7fe7c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c409a750 0x7fe7c415eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.633+0000 7fe7c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c409a750 0x7fe7c415eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41006/0 (socket says 172.21.15.12:41006) 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.633+0000 7fe7c2a00700 1 -- 172.21.15.12:0/2508014843 learned_addr learned my addr 172.21.15.12:0/2508014843 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7c3400700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7c4099a60 0x7fe7c4153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7c3e00700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c415d990 0x7fe7c40ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7c2a00700 1 -- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c415d990 msgr2=0x7fe7c40ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7c2a00700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c415d990 0x7fe7c40ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7c2a00700 1 -- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7c4099a60 msgr2=0x7fe7c4153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7c2a00700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7c4099a60 0x7fe7c4153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7c2a00700 1 -- 172.21.15.12:0/2508014843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b40094a0 con 0x7fe7c409a750 2024-03-29T11:28:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7c2a00700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c409a750 0x7fe7c415eec0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000e870 tx=0x7fe7b000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7bb400700 1 -- 172.21.15.12:0/2508014843 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0009590 con 0x7fe7c409a750 2024-03-29T11:28:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7bb400700 1 -- 172.21.15.12:0/2508014843 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b0013680 con 0x7fe7c409a750 2024-03-29T11:28:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.634+0000 7fe7ca400700 1 -- 172.21.15.12:0/2508014843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c4154530 con 0x7fe7c409a750 2024-03-29T11:28:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.635+0000 7fe7bb400700 1 -- 172.21.15.12:0/2508014843 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0013a30 con 0x7fe7c409a750 2024-03-29T11:28:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.635+0000 7fe7ca400700 1 -- 172.21.15.12:0/2508014843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c40c54b0 con 0x7fe7c409a750 2024-03-29T11:28:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.636+0000 7fe7ca400700 1 -- 172.21.15.12:0/2508014843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c4067720 con 0x7fe7c409a750 2024-03-29T11:28:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.637+0000 7fe7bb400700 1 -- 172.21.15.12:0/2508014843 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fe7b0013b90 con 0x7fe7c409a750 2024-03-29T11:28:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.637+0000 7fe7bb400700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a406adb0 0x7fe7a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.638+0000 7fe7bb400700 1 -- 172.21.15.12:0/2508014843 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fe7b0089560 con 0x7fe7c409a750 2024-03-29T11:28:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.638+0000 7fe7c3400700 1 -- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a406adb0 msgr2=0x7fe7a406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.638+0000 7fe7c3400700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a406adb0 0x7fe7a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.642+0000 7fe7bb400700 1 -- 172.21.15.12:0/2508014843 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe7b0054350 con 0x7fe7c409a750 2024-03-29T11:28:23.708 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:23.719+0000 7fed302d6240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:28:23.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.839+0000 7fe7c3400700 1 -- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a406adb0 msgr2=0x7fe7a406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:23.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.839+0000 7fe7c3400700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a406adb0 0x7fe7a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:23.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.858+0000 7fe7ca400700 1 -- 172.21.15.12:0/2508014843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe7c40b2560 con 0x7fe7c409a750 2024-03-29T11:28:23.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.858+0000 7fe7bb400700 1 -- 172.21.15.12:0/2508014843 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v195) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe7b0057970 con 0x7fe7c409a750 2024-03-29T11:28:23.848 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 -- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a406adb0 msgr2=0x7fe7a406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a406adb0 0x7fe7a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 -- 172.21.15.12:0/2508014843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c409a750 msgr2=0x7fe7c415eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c409a750 0x7fe7c415eec0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000e870 tx=0x7fe7b000eb80 comp rx=0 tx=0).stop 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 -- 172.21.15.12:0/2508014843 shutdown_connections 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a406adb0 0x7fe7a406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c415d990 0x7fe7c40ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7c4099a60 0x7fe7c4153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 --2- 172.21.15.12:0/2508014843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c409a750 0x7fe7c415eec0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 -- 172.21.15.12:0/2508014843 >> 172.21.15.12:0/2508014843 conn(0x7fe7c40b2e20 msgr2=0x7fe7c40b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 -- 172.21.15.12:0/2508014843 shutdown_connections 2024-03-29T11:28:23.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:23.861+0000 7fe7ca400700 1 -- 172.21.15.12:0/2508014843 wait complete. 2024-03-29T11:28:23.871 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T11:28:24.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.082+0000 7fb463c00700 1 Processor -- start 2024-03-29T11:28:24.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.082+0000 7fb463c00700 1 -- start start 2024-03-29T11:28:24.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.084+0000 7fb463c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb45c06aea0 0x7fb45c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.084+0000 7fb463c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c161150 con 0x7fb45c160d70 2024-03-29T11:28:24.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.084+0000 7fb463c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c06dba0 con 0x7fb45c15ef70 2024-03-29T11:28:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.084+0000 7fb463c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45c0b4cd0 con 0x7fb45c06aea0 2024-03-29T11:28:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.084+0000 7fb461200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb45c160d70 0x7fb45c15e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52888/0 (socket says 172.21.15.12:52888) 2024-03-29T11:28:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.084+0000 7fb461200700 1 -- 172.21.15.12:0/2550842850 learned_addr learned my addr 172.21.15.12:0/2550842850 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.084+0000 7fb461c00700 1 --2- 172.21.15.12:0/2550842850 >> v2:172.21.15.12:3300/0 conn(0x7fb45c06aea0 0x7fb45c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.084+0000 7fb45b400700 1 -- 172.21.15.12:0/2550842850 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024545230 0 0) 0x7fb45c161150 con 0x7fb45c160d70 2024-03-29T11:28:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.084+0000 7fb45b400700 1 -- 172.21.15.12:0/2550842850 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4440035b0 con 0x7fb45c160d70 2024-03-29T11:28:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.085+0000 7fb45b400700 1 -- 172.21.15.12:0/2550842850 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157265238 0 0) 0x7fb45c06dba0 con 0x7fb45c15ef70 2024-03-29T11:28:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.085+0000 7fb45b400700 1 -- 172.21.15.12:0/2550842850 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45c161150 con 0x7fb45c15ef70 2024-03-29T11:28:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.085+0000 7fb461c00700 1 -- 172.21.15.12:0/2550842850 >> v1:172.21.15.159:6789/0 conn(0x7fb45c15ef70 legacy=0x7fb45c06a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.085+0000 7fb461c00700 1 -- 172.21.15.12:0/2550842850 >> v1:172.21.15.12:6789/0 conn(0x7fb45c160d70 legacy=0x7fb45c15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.085+0000 7fb45b400700 1 -- 172.21.15.12:0/2550842850 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2821957850 0 0) 0x7fb4440035b0 con 0x7fb45c160d70 2024-03-29T11:28:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.085+0000 7fb461c00700 1 -- 172.21.15.12:0/2550842850 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c0b4ee0 con 0x7fb45c06aea0 2024-03-29T11:28:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.085+0000 7fb461c00700 1 --2- 172.21.15.12:0/2550842850 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 0x7fb45c0b3200 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fb454009d90 tx=0x7fb45400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0f8d85c829ad server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.085+0000 7fb45b400700 1 -- 172.21.15.12:0/2550842850 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45400e070 con 0x7fb45c06aea0 2024-03-29T11:28:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.085+0000 7fb45b400700 1 -- 172.21.15.12:0/2550842850 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb454003a20 con 0x7fb45c06aea0 2024-03-29T11:28:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.085+0000 7fb45b400700 1 -- 172.21.15.12:0/2550842850 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb450001670 con 0x7fb45c06aea0 2024-03-29T11:28:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.086+0000 7fb463c00700 1 -- 172.21.15.12:0/2550842850 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 msgr2=0x7fb45c0b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.086+0000 7fb463c00700 1 --2- 172.21.15.12:0/2550842850 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 0x7fb45c0b3200 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fb454009d90 tx=0x7fb45400b330 comp rx=0 tx=0).stop 2024-03-29T11:28:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.086+0000 7fb463c00700 1 -- 172.21.15.12:0/2550842850 shutdown_connections 2024-03-29T11:28:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.086+0000 7fb463c00700 1 --2- 172.21.15.12:0/2550842850 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 0x7fb45c0b3200 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.086+0000 7fb463c00700 1 -- 172.21.15.12:0/2550842850 >> 172.21.15.12:0/2550842850 conn(0x7fb45c0b5720 msgr2=0x7fb45c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.086+0000 7fb463c00700 1 -- 172.21.15.12:0/2550842850 shutdown_connections 2024-03-29T11:28:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.086+0000 7fb463c00700 1 -- 172.21.15.12:0/2550842850 wait complete. 2024-03-29T11:28:24.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.088+0000 7fb463c00700 1 Processor -- start 2024-03-29T11:28:24.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.088+0000 7fb463c00700 1 -- start start 2024-03-29T11:28:24.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.089+0000 7fb463c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 0x7fb45c1519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.089+0000 7fb463c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb45c15ef70 0x7fb45c159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.089+0000 7fb463c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb45c160d70 0x7fb45c1f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.089+0000 7fb463c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c14da20 con 0x7fb45c06aea0 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.089+0000 7fb463c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c15a070 con 0x7fb45c15ef70 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.089+0000 7fb463c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c09cd40 con 0x7fb45c160d70 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.089+0000 7fb461200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 0x7fb45c1519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.089+0000 7fb461200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 0x7fb45c1519b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41030/0 (socket says 172.21.15.12:41030) 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.089+0000 7fb461200700 1 -- 172.21.15.12:0/2740899628 learned_addr learned my addr 172.21.15.12:0/2740899628 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.089+0000 7fb45be00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb45c15ef70 0x7fb45c159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb461c00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb45c160d70 0x7fb45c1f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb461200700 1 -- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb45c160d70 msgr2=0x7fb45c1f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb461200700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb45c160d70 0x7fb45c1f6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb461200700 1 -- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb45c15ef70 msgr2=0x7fb45c159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb461200700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb45c15ef70 0x7fb45c159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb461200700 1 -- 172.21.15.12:0/2740899628 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb454009a70 con 0x7fb45c06aea0 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb461200700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 0x7fb45c1519b0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fb45000c730 tx=0x7fb45000ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb458c00700 1 -- 172.21.15.12:0/2740899628 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb450004c80 con 0x7fb45c06aea0 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb458c00700 1 -- 172.21.15.12:0/2740899628 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb450004de0 con 0x7fb45c06aea0 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb458c00700 1 -- 172.21.15.12:0/2740899628 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4500072d0 con 0x7fb45c06aea0 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.090+0000 7fb463c00700 1 -- 172.21.15.12:0/2740899628 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c09d020 con 0x7fb45c06aea0 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.091+0000 7fb463c00700 1 -- 172.21.15.12:0/2740899628 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c152070 con 0x7fb45c06aea0 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.092+0000 7fb463c00700 1 -- 172.21.15.12:0/2740899628 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45c067720 con 0x7fb45c06aea0 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.093+0000 7fb458c00700 1 -- 172.21.15.12:0/2740899628 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fb450007430 con 0x7fb45c06aea0 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.094+0000 7fb458c00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb43c06adb0 0x7fb43c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.094+0000 7fb458c00700 1 -- 172.21.15.12:0/2740899628 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fb450089400 con 0x7fb45c06aea0 2024-03-29T11:28:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.094+0000 7fb45be00700 1 -- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb43c06adb0 msgr2=0x7fb43c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:24.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.094+0000 7fb45be00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb43c06adb0 0x7fb43c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:24.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.098+0000 7fb458c00700 1 -- 172.21.15.12:0/2740899628 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb450054200 con 0x7fb45c06aea0 2024-03-29T11:28:24.256 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:24.266+0000 7f7ab52c7240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:28:24.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.295+0000 7fb45be00700 1 -- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb43c06adb0 msgr2=0x7fb43c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:24.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.295+0000 7fb45be00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb43c06adb0 0x7fb43c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:24.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.328+0000 7fb463c00700 1 -- 172.21.15.12:0/2740899628 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb45c0d0800 con 0x7fb45c06aea0 2024-03-29T11:28:24.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.328+0000 7fb458c00700 1 -- 172.21.15.12:0/2740899628 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 14 entries v195) v1 ==== 81+0+819 (secure 0 0 0) 0x7fb450057820 con 0x7fb45c06aea0 2024-03-29T11:28:24.318 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3631785847 2024-03-30T11:23:24.314524+0000 2024-03-29T11:28:24.318 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/750511589 2024-03-30T11:23:24.314524+0000 2024-03-29T11:28:24.318 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2791060580 2024-03-30T11:23:24.314524+0000 2024-03-29T11:28:24.318 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/479652442 2024-03-30T11:23:24.314524+0000 2024-03-29T11:28:24.318 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/180231134 2024-03-30T11:12:43.757966+0000 2024-03-29T11:28:24.318 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/43444 2024-03-30T11:12:43.757966+0000 2024-03-29T11:28:24.318 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/43444 2024-03-30T11:12:43.757966+0000 2024-03-29T11:28:24.319 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2825856150 2024-03-30T11:12:43.757966+0000 2024-03-29T11:28:24.319 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3819915206 2024-03-30T11:23:24.314524+0000 2024-03-29T11:28:24.319 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2176927052 2024-03-30T11:12:43.757966+0000 2024-03-29T11:28:24.319 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3123015140 2024-03-30T11:23:24.314524+0000 2024-03-29T11:28:24.319 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1699207236 2024-03-30T11:12:43.757966+0000 2024-03-29T11:28:24.319 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3816982205 2024-03-30T11:12:43.757966+0000 2024-03-29T11:28:24.319 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/4147178761 2024-03-30T11:12:43.757966+0000 2024-03-29T11:28:24.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.330+0000 7fb463c00700 1 -- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb43c06adb0 msgr2=0x7fb43c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:24.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.330+0000 7fb463c00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb43c06adb0 0x7fb43c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.330+0000 7fb463c00700 1 -- 172.21.15.12:0/2740899628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 msgr2=0x7fb45c1519b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.330+0000 7fb463c00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 0x7fb45c1519b0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fb45000c730 tx=0x7fb45000ca40 comp rx=0 tx=0).stop 2024-03-29T11:28:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.330+0000 7fb463c00700 1 -- 172.21.15.12:0/2740899628 shutdown_connections 2024-03-29T11:28:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.330+0000 7fb463c00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb43c06adb0 0x7fb43c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.330+0000 7fb463c00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb45c160d70 0x7fb45c1f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.330+0000 7fb463c00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45c06aea0 0x7fb45c1519b0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.330+0000 7fb463c00700 1 --2- 172.21.15.12:0/2740899628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb45c15ef70 0x7fb45c159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.330+0000 7fb463c00700 1 -- 172.21.15.12:0/2740899628 >> 172.21.15.12:0/2740899628 conn(0x7fb45c0b5720 msgr2=0x7fb45c161330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.331+0000 7fb463c00700 1 -- 172.21.15.12:0/2740899628 shutdown_connections 2024-03-29T11:28:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.331+0000 7fb463c00700 1 -- 172.21.15.12:0/2740899628 wait complete. 2024-03-29T11:28:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:listed 14 entries 2024-03-29T11:28:24.340 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T11:28:24.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:24.340 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T11:28:24.347 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:24.357+0000 7f7ab52c7240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:28:24.420 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:28:24.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:24.421 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T11:28:24.467 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:28:24.467 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:28:24.472 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:24.482+0000 7fed302d6240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:28:24.512 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:28:24.512 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:28:24.513 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:28:24.513 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:24.523+0000 7f7ab52c7240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:28:24.557 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:28:24.558 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:28:24.558 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:28:24.571 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:24.582+0000 7fed302d6240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:28:24.604 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:28:24.604 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:28:24.732 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:24.742+0000 7f7ab52c7240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:28:24.744 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:24.754+0000 7fed302d6240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:28:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.816+0000 7feff5e00700 1 Processor -- start 2024-03-29T11:28:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.816+0000 7feff5e00700 1 -- start start 2024-03-29T11:28:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.818+0000 7feff5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff015d710 0x7feff00d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.818+0000 7feff5e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7feff009aba0 0x7feff00d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.818+0000 7feff5e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff006bb40 con 0x7feff006bf40 2024-03-29T11:28:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.818+0000 7feff5e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff0099b00 con 0x7feff009aba0 2024-03-29T11:28:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.818+0000 7feff5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff0099de0 con 0x7feff015d710 2024-03-29T11:28:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.819+0000 7fefee000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7feff009aba0 0x7feff00d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:24.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.819+0000 7fefeea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff015d710 0x7feff00d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:24.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.819+0000 7fefef400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7feff006bf40 0x7feff00ccbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:53854/0 (socket says 172.21.15.12:53854) 2024-03-29T11:28:24.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.819+0000 7fefef400700 1 -- 172.21.15.12:0/923683009 learned_addr learned my addr 172.21.15.12:0/923683009 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:24.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.819+0000 7fefed600700 1 -- 172.21.15.12:0/923683009 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118008316 0 0) 0x7feff006bb40 con 0x7feff006bf40 2024-03-29T11:28:24.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.819+0000 7fefed600700 1 -- 172.21.15.12:0/923683009 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd40035b0 con 0x7feff006bf40 2024-03-29T11:28:24.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.819+0000 7fefee000700 1 -- 172.21.15.12:0/923683009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff015d710 msgr2=0x7feff00d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:24.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.819+0000 7fefee000700 1 --2- 172.21.15.12:0/923683009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff015d710 0x7feff00d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.819+0000 7fefee000700 1 -- 172.21.15.12:0/923683009 >> v1:172.21.15.159:6789/0 conn(0x7feff006bf40 legacy=0x7feff00ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.819+0000 7fefee000700 1 -- 172.21.15.12:0/923683009 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff009a0c0 con 0x7feff009aba0 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.820+0000 7fefee000700 1 --2- 172.21.15.12:0/923683009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff009aba0 0x7feff00d0670 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009840 tx=0x7fefdc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33ad28650bba1480 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.820+0000 7fefed600700 1 -- 172.21.15.12:0/923683009 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc011830 con 0x7feff009aba0 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.820+0000 7fefed600700 1 -- 172.21.15.12:0/923683009 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefdc011990 con 0x7feff009aba0 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.820+0000 7fefed600700 1 -- 172.21.15.12:0/923683009 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc011d20 con 0x7feff009aba0 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.820+0000 7feff5e00700 1 -- 172.21.15.12:0/923683009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff009aba0 msgr2=0x7feff00d0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.820+0000 7feff5e00700 1 --2- 172.21.15.12:0/923683009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff009aba0 0x7feff00d0670 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009840 tx=0x7fefdc009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.820+0000 7feff5e00700 1 -- 172.21.15.12:0/923683009 shutdown_connections 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.820+0000 7feff5e00700 1 --2- 172.21.15.12:0/923683009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff015d710 0x7feff00d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.821+0000 7feff5e00700 1 --2- 172.21.15.12:0/923683009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff009aba0 0x7feff00d0670 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.821+0000 7feff5e00700 1 -- 172.21.15.12:0/923683009 >> 172.21.15.12:0/923683009 conn(0x7feff00d13e0 msgr2=0x7feff00d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.821+0000 7feff5e00700 1 -- 172.21.15.12:0/923683009 shutdown_connections 2024-03-29T11:28:24.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.821+0000 7feff5e00700 1 -- 172.21.15.12:0/923683009 wait complete. 2024-03-29T11:28:24.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.822+0000 7feff5e00700 1 Processor -- start 2024-03-29T11:28:24.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.822+0000 7feff5e00700 1 -- start start 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.823+0000 7feff5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feff006bf40 0x7feff014e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.823+0000 7feff5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff009aba0 0x7feff00c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.823+0000 7feff5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff015d710 0x7feff00af830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.823+0000 7feff5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff014ee00 con 0x7feff006bf40 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.823+0000 7feff5e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff00ae300 con 0x7feff015d710 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.823+0000 7feff5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff00ae5e0 con 0x7feff009aba0 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefeea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feff006bf40 0x7feff014e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefeea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feff006bf40 0x7feff014e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41032/0 (socket says 172.21.15.12:41032) 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefeea00700 1 -- 172.21.15.12:0/2764762682 learned_addr learned my addr 172.21.15.12:0/2764762682 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefee000700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff009aba0 0x7feff00c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefef400700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff015d710 0x7feff00af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefeea00700 1 -- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff009aba0 msgr2=0x7feff00c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefeea00700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff009aba0 0x7feff00c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefeea00700 1 -- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff015d710 msgr2=0x7feff00af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefeea00700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff015d710 0x7feff00af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:24.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefeea00700 1 -- 172.21.15.12:0/2764762682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc0094a0 con 0x7feff006bf40 2024-03-29T11:28:24.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.824+0000 7fefeea00700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feff006bf40 0x7feff014e8c0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fefe000e870 tx=0x7fefe000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:24.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.825+0000 7fefe7400700 1 -- 172.21.15.12:0/2764762682 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0009590 con 0x7feff006bf40 2024-03-29T11:28:24.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.825+0000 7fefe7400700 1 -- 172.21.15.12:0/2764762682 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe0013680 con 0x7feff006bf40 2024-03-29T11:28:24.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.825+0000 7fefe7400700 1 -- 172.21.15.12:0/2764762682 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe00139f0 con 0x7feff006bf40 2024-03-29T11:28:24.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.825+0000 7feff5e00700 1 -- 172.21.15.12:0/2764762682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff00ae8c0 con 0x7feff006bf40 2024-03-29T11:28:24.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.825+0000 7feff5e00700 1 -- 172.21.15.12:0/2764762682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff00aee80 con 0x7feff006bf40 2024-03-29T11:28:24.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.826+0000 7feff5e00700 1 -- 172.21.15.12:0/2764762682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff0067720 con 0x7feff006bf40 2024-03-29T11:28:24.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.827+0000 7fefe7400700 1 -- 172.21.15.12:0/2764762682 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fefe0013b50 con 0x7feff006bf40 2024-03-29T11:28:24.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.828+0000 7fefe7400700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fefd006adb0 0x7fefd006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:24.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.828+0000 7fefe7400700 1 -- 172.21.15.12:0/2764762682 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fefe0089a30 con 0x7feff006bf40 2024-03-29T11:28:24.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.828+0000 7fefee000700 1 -- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fefd006adb0 msgr2=0x7fefd006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:24.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.829+0000 7fefee000700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fefd006adb0 0x7fefd006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:24.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:24.833+0000 7fefe7400700 1 -- 172.21.15.12:0/2764762682 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fefe0054790 con 0x7feff006bf40 2024-03-29T11:28:24.825 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:24.835+0000 7f7ab52c7240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:28:24.924 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:24.935+0000 7f7ab52c7240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:28:24.967 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:24.977+0000 7fed302d6240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:28:25.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.030+0000 7fefee000700 1 -- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fefd006adb0 msgr2=0x7fefd006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:25.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.030+0000 7fefee000700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fefd006adb0 0x7fefd006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:25.021 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:25.032+0000 7f7ab52c7240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:28:25.064 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:25.074+0000 7fed302d6240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:28:25.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.095+0000 7feff5e00700 1 -- 172.21.15.12:0/2764762682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feff00d0f20 con 0x7feff006bf40 2024-03-29T11:28:25.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.095+0000 7fefe7400700 1 -- 172.21.15.12:0/2764762682 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fefe0017090 con 0x7feff006bf40 2024-03-29T11:28:25.085 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:25.085 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2592133465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2592133465},{"type":"v1","addr":"172.21.15.159:6837","nonce":2592133465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":8381,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/204651369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":204651369},{"type":"v1","addr":"172.21.15.12:6835","nonce":204651369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9616,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2547917697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2547917697},{"type":"v1","addr":"172.21.15.159:6836","nonce":2547917697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:12:26.523825+0000","modified":"2024-03-29T11:12:30.007709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9975":{"gid":9975,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/4176455926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4176455926},{"type":"v1","addr":"172.21.15.12:6833","nonce":4176455926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-29T11:28:25.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 -- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fefd006adb0 msgr2=0x7fefd006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:25.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fefd006adb0 0x7fefd006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 -- 172.21.15.12:0/2764762682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feff006bf40 msgr2=0x7feff014e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feff006bf40 0x7feff014e8c0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fefe000e870 tx=0x7fefe000eb80 comp rx=0 tx=0).stop 2024-03-29T11:28:25.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 -- 172.21.15.12:0/2764762682 shutdown_connections 2024-03-29T11:28:25.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fefd006adb0 0x7fefd006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feff015d710 0x7feff00af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feff006bf40 0x7feff014e8c0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 --2- 172.21.15.12:0/2764762682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feff009aba0 0x7feff00c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 -- 172.21.15.12:0/2764762682 >> 172.21.15.12:0/2764762682 conn(0x7feff00d13e0 msgr2=0x7feff006c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:25.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 -- 172.21.15.12:0/2764762682 shutdown_connections 2024-03-29T11:28:25.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.098+0000 7feff5e00700 1 -- 172.21.15.12:0/2764762682 wait complete. 2024-03-29T11:28:25.089 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 89 2024-03-29T11:28:25.106 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:25.119 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:25.129+0000 7f7ab52c7240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:28:25.172 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:25.182+0000 7fed302d6240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:28:25.277 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:25.288+0000 7fed302d6240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:28:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.316+0000 7ff741000700 1 Processor -- start 2024-03-29T11:28:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.316+0000 7ff741000700 1 -- start start 2024-03-29T11:28:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.317+0000 7ff741000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff73c15d640 0x7ff73c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.317+0000 7ff741000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff73c1619d0 0x7ff73c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.317+0000 7ff741000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73c0d30d0 con 0x7ff73c0ceec0 2024-03-29T11:28:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.317+0000 7ff741000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73c0b6840 con 0x7ff73c15d640 2024-03-29T11:28:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.318+0000 7ff741000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0d2880 con 0x7ff73c1619d0 2024-03-29T11:28:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.318+0000 7ff739e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff73c15d640 0x7ff73c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.318+0000 7ff73a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff73c1619d0 0x7ff73c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.318+0000 7ff739e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff73c15d640 0x7ff73c0d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:36358/0 (socket says 172.21.15.12:36358) 2024-03-29T11:28:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.318+0000 7ff739e00700 1 -- 172.21.15.12:0/836193205 learned_addr learned my addr 172.21.15.12:0/836193205 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.318+0000 7ff739e00700 1 -- 172.21.15.12:0/836193205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff73c1619d0 msgr2=0x7ff73c0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.318+0000 7ff739e00700 1 --2- 172.21.15.12:0/836193205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff73c1619d0 0x7ff73c0b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.318+0000 7ff739e00700 1 -- 172.21.15.12:0/836193205 >> v1:172.21.15.159:6789/0 conn(0x7ff73c0ceec0 legacy=0x7ff73c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.318+0000 7ff739e00700 1 -- 172.21.15.12:0/836193205 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c0d2b30 con 0x7ff73c15d640 2024-03-29T11:28:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.319+0000 7ff739e00700 1 --2- 172.21.15.12:0/836193205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c15d640 0x7ff73c0d2340 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff72c0098f0 tx=0x7ff72c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfe1757d564e00ca server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.319+0000 7ff738a00700 1 -- 172.21.15.12:0/836193205 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c011830 con 0x7ff73c15d640 2024-03-29T11:28:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.319+0000 7ff738a00700 1 -- 172.21.15.12:0/836193205 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c011990 con 0x7ff73c15d640 2024-03-29T11:28:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.319+0000 7ff738a00700 1 -- 172.21.15.12:0/836193205 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c011d20 con 0x7ff73c15d640 2024-03-29T11:28:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.319+0000 7ff741000700 1 -- 172.21.15.12:0/836193205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c15d640 msgr2=0x7ff73c0d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.319+0000 7ff741000700 1 --2- 172.21.15.12:0/836193205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c15d640 0x7ff73c0d2340 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff72c0098f0 tx=0x7ff72c009c00 comp rx=0 tx=0).stop 2024-03-29T11:28:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.320+0000 7ff741000700 1 -- 172.21.15.12:0/836193205 shutdown_connections 2024-03-29T11:28:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.320+0000 7ff741000700 1 --2- 172.21.15.12:0/836193205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff73c1619d0 0x7ff73c0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.320+0000 7ff741000700 1 --2- 172.21.15.12:0/836193205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c15d640 0x7ff73c0d2340 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.320+0000 7ff741000700 1 -- 172.21.15.12:0/836193205 >> 172.21.15.12:0/836193205 conn(0x7ff73c08aff0 msgr2=0x7ff73c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.320+0000 7ff741000700 1 -- 172.21.15.12:0/836193205 shutdown_connections 2024-03-29T11:28:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.320+0000 7ff741000700 1 -- 172.21.15.12:0/836193205 wait complete. 2024-03-29T11:28:25.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.321+0000 7ff741000700 1 Processor -- start 2024-03-29T11:28:25.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.322+0000 7ff741000700 1 -- start start 2024-03-29T11:28:25.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff741000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c0ceec0 0x7ff73c1f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff741000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff73c15d640 0x7ff73c0ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff741000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff73c1619d0 0x7ff73c0cab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff741000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0cce70 con 0x7ff73c1619d0 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff741000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0cc860 con 0x7ff73c0ceec0 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff741000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c15f660 con 0x7ff73c15d640 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff739e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c0ceec0 0x7ff73c1f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff73a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff73c1619d0 0x7ff73c0cab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff739e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c0ceec0 0x7ff73c1f2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:36360/0 (socket says 172.21.15.12:36360) 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff739e00700 1 -- 172.21.15.12:0/2750657195 learned_addr learned my addr 172.21.15.12:0/2750657195 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff739400700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff73c15d640 0x7ff73c0ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff739e00700 1 -- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff73c15d640 msgr2=0x7ff73c0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff739e00700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff73c15d640 0x7ff73c0ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff739e00700 1 -- 172.21.15.12:0/2750657195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff73c1619d0 msgr2=0x7ff73c0cab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff739e00700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff73c1619d0 0x7ff73c0cab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff739e00700 1 -- 172.21.15.12:0/2750657195 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c009550 con 0x7ff73c0ceec0 2024-03-29T11:28:25.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.323+0000 7ff73a800700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff73c1619d0 0x7ff73c0cab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:25.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.324+0000 7ff739e00700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c0ceec0 0x7ff73c1f2830 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff72c000c00 tx=0x7ff72c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:25.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.324+0000 7ff732000700 1 -- 172.21.15.12:0/2750657195 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c004400 con 0x7ff73c0ceec0 2024-03-29T11:28:25.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.324+0000 7ff732000700 1 -- 172.21.15.12:0/2750657195 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c01f070 con 0x7ff73c0ceec0 2024-03-29T11:28:25.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.324+0000 7ff741000700 1 -- 172.21.15.12:0/2750657195 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73c0ccb00 con 0x7ff73c0ceec0 2024-03-29T11:28:25.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.324+0000 7ff732000700 1 -- 172.21.15.12:0/2750657195 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c0038e0 con 0x7ff73c0ceec0 2024-03-29T11:28:25.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.324+0000 7ff741000700 1 -- 172.21.15.12:0/2750657195 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73c1f2d70 con 0x7ff73c0ceec0 2024-03-29T11:28:25.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.325+0000 7ff741000700 1 -- 172.21.15.12:0/2750657195 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73c0cc6c0 con 0x7ff73c0ceec0 2024-03-29T11:28:25.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.326+0000 7ff732000700 1 -- 172.21.15.12:0/2750657195 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7ff72c021030 con 0x7ff73c0ceec0 2024-03-29T11:28:25.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.328+0000 7ff732000700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff71c06adb0 0x7ff71c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.328+0000 7ff732000700 1 -- 172.21.15.12:0/2750657195 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7ff72c08aa40 con 0x7ff73c0ceec0 2024-03-29T11:28:25.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.328+0000 7ff739400700 1 -- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff71c06adb0 msgr2=0x7ff71c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:25.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.328+0000 7ff739400700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff71c06adb0 0x7ff71c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:25.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.332+0000 7ff732000700 1 -- 172.21.15.12:0/2750657195 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff73c0cc6c0 con 0x7ff73c0ceec0 2024-03-29T11:28:25.332 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:25.342+0000 7f7ab52c7240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:28:25.384 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:25.395+0000 7fed302d6240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:28:25.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.529+0000 7ff739400700 1 -- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff71c06adb0 msgr2=0x7ff71c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:25.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.529+0000 7ff739400700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff71c06adb0 0x7ff71c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:25.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.548+0000 7ff741000700 1 -- 172.21.15.12:0/2750657195 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff73c0cc6c0 con 0x7ff73c0ceec0 2024-03-29T11:28:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.550+0000 7ff732000700 1 -- 172.21.15.12:0/2750657195 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+17186 (secure 0 0 0) 0x7ff73c0cc6c0 con 0x7ff73c0ceec0 2024-03-29T11:28:25.540 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:25.540 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":195,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:28:22.488235+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:12:24.038574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-03-29T11:12:25.509384+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":195,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3631785847":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/750511589":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/2791060580":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/479652442":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/180231134":"2024-03-30T11:12:43.757966+0000","172.21.15.159:6833/43444":"2024-03-30T11:12:43.757966+0000","172.21.15.159:6832/43444":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/2825856150":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3819915206":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/2176927052":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3123015140":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/1699207236":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3816982205":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/4147178761":"2024-03-30T11:12:43.757966+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.552+0000 7ff741000700 1 -- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff71c06adb0 msgr2=0x7ff71c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.552+0000 7ff741000700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff71c06adb0 0x7ff71c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.552+0000 7ff741000700 1 -- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c0ceec0 msgr2=0x7ff73c1f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.552+0000 7ff741000700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c0ceec0 0x7ff73c1f2830 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff72c000c00 tx=0x7ff72c004750 comp rx=0 tx=0).stop 2024-03-29T11:28:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.552+0000 7ff741000700 1 -- 172.21.15.12:0/2750657195 shutdown_connections 2024-03-29T11:28:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.552+0000 7ff741000700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff71c06adb0 0x7ff71c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.552+0000 7ff741000700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff73c1619d0 0x7ff73c0cab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.552+0000 7ff741000700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff73c0ceec0 0x7ff73c1f2830 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.552+0000 7ff741000700 1 --2- 172.21.15.12:0/2750657195 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff73c15d640 0x7ff73c0ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.552+0000 7ff741000700 1 -- 172.21.15.12:0/2750657195 >> 172.21.15.12:0/2750657195 conn(0x7ff73c08aff0 msgr2=0x7ff73c0b8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:25.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.553+0000 7ff741000700 1 -- 172.21.15.12:0/2750657195 shutdown_connections 2024-03-29T11:28:25.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.553+0000 7ff741000700 1 -- 172.21.15.12:0/2750657195 wait complete. 2024-03-29T11:28:25.568 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:12:24.038574+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '190', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:12:25.509384+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 195, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:28:25.570 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:28:25.590 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:25.600+0000 7f7ab52c7240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:28:25.617 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:25.628+0000 7fed302d6240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:28:25.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.788+0000 7f373ec00700 1 Processor -- start 2024-03-29T11:28:25.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.788+0000 7f373ec00700 1 -- start start 2024-03-29T11:28:25.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.789+0000 7f373ec00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f37380d1b70 0x7f37380d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.789+0000 7f373ec00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380cff10 0x7f37380d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.789+0000 7f373ec00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37380cf640 con 0x7f373815d710 2024-03-29T11:28:25.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.789+0000 7f373ec00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37380cf220 con 0x7f37380d1b70 2024-03-29T11:28:25.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.789+0000 7f373ec00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37380d2560 con 0x7f37380cff10 2024-03-29T11:28:25.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.789+0000 7f3737400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f37380d1b70 0x7f37380d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:25.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.789+0000 7f373cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380cff10 0x7f37380d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:25.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.789+0000 7f3737400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f37380d1b70 0x7f37380d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41048/0 (socket says 172.21.15.12:41048) 2024-03-29T11:28:25.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.789+0000 7f3737400700 1 -- 172.21.15.12:0/2455914155 learned_addr learned my addr 172.21.15.12:0/2455914155 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.790+0000 7f373cc00700 1 -- 172.21.15.12:0/2455914155 >> v2:172.21.15.12:3300/0 conn(0x7f37380d1b70 msgr2=0x7f37380d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.790+0000 7f373cc00700 1 --2- 172.21.15.12:0/2455914155 >> v2:172.21.15.12:3300/0 conn(0x7f37380d1b70 0x7f37380d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.790+0000 7f373cc00700 1 -- 172.21.15.12:0/2455914155 >> v1:172.21.15.159:6789/0 conn(0x7f373815d710 legacy=0x7f37380d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.790+0000 7f373cc00700 1 -- 172.21.15.12:0/2455914155 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37380ce5a0 con 0x7f37380cff10 2024-03-29T11:28:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.790+0000 7f3736a00700 1 -- 172.21.15.12:0/2455914155 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220643774 0 0) 0x7f37380cf640 con 0x7f373815d710 2024-03-29T11:28:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.790+0000 7f3737400700 1 --2- 172.21.15.12:0/2455914155 >> v2:172.21.15.12:3300/0 conn(0x7f37380d1b70 0x7f37380d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.790+0000 7f373cc00700 1 --2- 172.21.15.12:0/2455914155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380cff10 0x7f37380d0330 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3730009d90 tx=0x7f373000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=102155673856b6f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f3736a00700 1 -- 172.21.15.12:0/2455914155 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373000e070 con 0x7f37380cff10 2024-03-29T11:28:25.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f3736a00700 1 -- 172.21.15.12:0/2455914155 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373000bb20 con 0x7f37380cff10 2024-03-29T11:28:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f3736a00700 1 -- 172.21.15.12:0/2455914155 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730012430 con 0x7f37380cff10 2024-03-29T11:28:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f373ec00700 1 -- 172.21.15.12:0/2455914155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380cff10 msgr2=0x7f37380d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f373ec00700 1 --2- 172.21.15.12:0/2455914155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380cff10 0x7f37380d0330 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3730009d90 tx=0x7f373000b330 comp rx=0 tx=0).stop 2024-03-29T11:28:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f373ec00700 1 -- 172.21.15.12:0/2455914155 shutdown_connections 2024-03-29T11:28:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f373ec00700 1 --2- 172.21.15.12:0/2455914155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380cff10 0x7f37380d0330 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f373ec00700 1 --2- 172.21.15.12:0/2455914155 >> v2:172.21.15.12:3300/0 conn(0x7f37380d1b70 0x7f37380d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f373ec00700 1 -- 172.21.15.12:0/2455914155 >> 172.21.15.12:0/2455914155 conn(0x7f373808aff0 msgr2=0x7f37380d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f373ec00700 1 -- 172.21.15.12:0/2455914155 shutdown_connections 2024-03-29T11:28:25.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.791+0000 7f373ec00700 1 -- 172.21.15.12:0/2455914155 wait complete. 2024-03-29T11:28:25.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.793+0000 7f373ec00700 1 Processor -- start 2024-03-29T11:28:25.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.793+0000 7f373ec00700 1 -- start start 2024-03-29T11:28:25.783 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:25.793+0000 7f7ab52c7240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:28:25.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.794+0000 7f373ec00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37380cff10 0x7f373815a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.794+0000 7f373ec00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380d1b70 0x7f37380cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.794+0000 7f373ec00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f373815d710 0x7f373815fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.794+0000 7f373ec00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373815acd0 con 0x7f37380cff10 2024-03-29T11:28:25.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.794+0000 7f373ec00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373815a6f0 con 0x7f373815d710 2024-03-29T11:28:25.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.794+0000 7f373ec00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37380b4020 con 0x7f37380d1b70 2024-03-29T11:28:25.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.794+0000 7f3737e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37380cff10 0x7f373815a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:25.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f3737400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380d1b70 0x7f37380cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:25.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f3737e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37380cff10 0x7f373815a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41062/0 (socket says 172.21.15.12:41062) 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f3737e00700 1 -- 172.21.15.12:0/177146467 learned_addr learned my addr 172.21.15.12:0/177146467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f373cc00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f373815d710 0x7f373815fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f3737e00700 1 -- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380d1b70 msgr2=0x7f37380cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f3737e00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380d1b70 0x7f37380cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f3737e00700 1 -- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f373815d710 msgr2=0x7f373815fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f3737e00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f373815d710 0x7f373815fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f3737e00700 1 -- 172.21.15.12:0/177146467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37240094f0 con 0x7f37380cff10 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f3737e00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37380cff10 0x7f373815a1b0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f372800c550 tx=0x7f372800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f372fe00700 1 -- 172.21.15.12:0/177146467 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37280054e0 con 0x7f37380cff10 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.795+0000 7f372fe00700 1 -- 172.21.15.12:0/177146467 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3728003680 con 0x7f37380cff10 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.796+0000 7f373ec00700 1 -- 172.21.15.12:0/177146467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3730009a70 con 0x7f37380cff10 2024-03-29T11:28:25.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.796+0000 7f372fe00700 1 -- 172.21.15.12:0/177146467 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728003a30 con 0x7f37380cff10 2024-03-29T11:28:25.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.796+0000 7f373ec00700 1 -- 172.21.15.12:0/177146467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37380b45e0 con 0x7f37380cff10 2024-03-29T11:28:25.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.797+0000 7f373ec00700 1 -- 172.21.15.12:0/177146467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3738067e30 con 0x7f37380cff10 2024-03-29T11:28:25.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.798+0000 7f372fe00700 1 -- 172.21.15.12:0/177146467 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f3728003b90 con 0x7f37380cff10 2024-03-29T11:28:25.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.798+0000 7f372fe00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f371806adb0 0x7f371806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:25.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.799+0000 7f372fe00700 1 -- 172.21.15.12:0/177146467 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f3728088bc0 con 0x7f37380cff10 2024-03-29T11:28:25.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.799+0000 7f3737400700 1 -- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f371806adb0 msgr2=0x7f371806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:25.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.799+0000 7f3737400700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f371806adb0 0x7f371806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:25.803+0000 7f372fe00700 1 -- 172.21.15.12:0/177146467 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f37280538a0 con 0x7f37380cff10 2024-03-29T11:28:25.877 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:25.887+0000 7f7ab52c7240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:28:25.887 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:25.897+0000 7fed302d6240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:28:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.000+0000 7f3737400700 1 -- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f371806adb0 msgr2=0x7f371806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.000+0000 7f3737400700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f371806adb0 0x7f371806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:26.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.017+0000 7f373ec00700 1 -- 172.21.15.12:0/177146467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37380b9510 con 0x7f37380cff10 2024-03-29T11:28:26.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.018+0000 7f372fe00700 1 -- 172.21.15.12:0/177146467 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v195) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37280170a0 con 0x7f37380cff10 2024-03-29T11:28:26.008 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:28:26.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 -- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f371806adb0 msgr2=0x7f371806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:26.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f371806adb0 0x7f371806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 -- 172.21.15.12:0/177146467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37380cff10 msgr2=0x7f373815a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37380cff10 0x7f373815a1b0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f372800c550 tx=0x7f372800c860 comp rx=0 tx=0).stop 2024-03-29T11:28:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 -- 172.21.15.12:0/177146467 shutdown_connections 2024-03-29T11:28:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f371806adb0 0x7f371806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f373815d710 0x7f373815fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37380cff10 0x7f373815a1b0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 --2- 172.21.15.12:0/177146467 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37380d1b70 0x7f37380cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 -- 172.21.15.12:0/177146467 >> 172.21.15.12:0/177146467 conn(0x7f373808aff0 msgr2=0x7f3738073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 -- 172.21.15.12:0/177146467 shutdown_connections 2024-03-29T11:28:26.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.021+0000 7f373ec00700 1 -- 172.21.15.12:0/177146467 wait complete. 2024-03-29T11:28:26.029 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:28:26.096 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:26.106+0000 7fed302d6240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:28:26.197 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:26.208+0000 7fed302d6240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:28:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.243+0000 7f244f000700 1 Processor -- start 2024-03-29T11:28:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.243+0000 7f244f000700 1 -- start start 2024-03-29T11:28:26.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.244+0000 7f244f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f244815d9d0 0x7f24480d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.244+0000 7f244f000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f24480d16a0 0x7f24480d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.244+0000 7f244f000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f24480d2370 0x7f244815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.244+0000 7f244f000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24480d2820 con 0x7f24480d2370 2024-03-29T11:28:26.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.244+0000 7f244f000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2448161420 con 0x7f24480d16a0 2024-03-29T11:28:26.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.244+0000 7f244f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2448161700 con 0x7f244815d9d0 2024-03-29T11:28:26.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f244d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f24480d2370 0x7f244815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:26.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f244d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f24480d2370 0x7f244815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41068/0 (socket says 172.21.15.12:41068) 2024-03-29T11:28:26.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f244d000700 1 -- 172.21.15.12:0/2117984410 learned_addr learned my addr 172.21.15.12:0/2117984410 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:26.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f2447400700 1 --2- 172.21.15.12:0/2117984410 >> v2:172.21.15.159:3300/0 conn(0x7f24480d16a0 0x7f24480d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:26.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f2447e00700 1 --2- 172.21.15.12:0/2117984410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f244815d9d0 0x7f24480d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:26.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f244d000700 1 -- 172.21.15.12:0/2117984410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f244815d9d0 msgr2=0x7f24480d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:26.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f244d000700 1 --2- 172.21.15.12:0/2117984410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f244815d9d0 0x7f24480d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f244d000700 1 -- 172.21.15.12:0/2117984410 >> v2:172.21.15.159:3300/0 conn(0x7f24480d16a0 msgr2=0x7f24480d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:26.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f244d000700 1 --2- 172.21.15.12:0/2117984410 >> v2:172.21.15.159:3300/0 conn(0x7f24480d16a0 0x7f24480d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f244d000700 1 -- 172.21.15.12:0/2117984410 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24481619e0 con 0x7f24480d2370 2024-03-29T11:28:26.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.245+0000 7f244d000700 1 --2- 172.21.15.12:0/2117984410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24480d2370 0x7f244815ddb0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f244806b5e0 tx=0x7f244000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befcdb9419774786 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:26.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.246+0000 7f2446a00700 1 -- 172.21.15.12:0/2117984410 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244000e070 con 0x7f24480d2370 2024-03-29T11:28:26.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.246+0000 7f2446a00700 1 -- 172.21.15.12:0/2117984410 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244000b910 con 0x7f24480d2370 2024-03-29T11:28:26.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.246+0000 7f2446a00700 1 -- 172.21.15.12:0/2117984410 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244000bcc0 con 0x7f24480d2370 2024-03-29T11:28:26.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.246+0000 7f244f000700 1 -- 172.21.15.12:0/2117984410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24480d2370 msgr2=0x7f244815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.246+0000 7f244f000700 1 --2- 172.21.15.12:0/2117984410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24480d2370 0x7f244815ddb0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f244806b5e0 tx=0x7f244000b330 comp rx=0 tx=0).stop 2024-03-29T11:28:26.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.246+0000 7f244f000700 1 -- 172.21.15.12:0/2117984410 shutdown_connections 2024-03-29T11:28:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.246+0000 7f244f000700 1 --2- 172.21.15.12:0/2117984410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24480d2370 0x7f244815ddb0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.246+0000 7f244f000700 1 --2- 172.21.15.12:0/2117984410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f244815d9d0 0x7f24480d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.246+0000 7f244f000700 1 --2- 172.21.15.12:0/2117984410 >> v2:172.21.15.159:3300/0 conn(0x7f24480d16a0 0x7f24480d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.246+0000 7f244f000700 1 -- 172.21.15.12:0/2117984410 >> 172.21.15.12:0/2117984410 conn(0x7f244808aff0 msgr2=0x7f24480d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.247+0000 7f244f000700 1 -- 172.21.15.12:0/2117984410 shutdown_connections 2024-03-29T11:28:26.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.247+0000 7f244f000700 1 -- 172.21.15.12:0/2117984410 wait complete. 2024-03-29T11:28:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.248+0000 7f244f000700 1 Processor -- start 2024-03-29T11:28:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.248+0000 7f244f000700 1 -- start start 2024-03-29T11:28:26.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.249+0000 7f244f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24480d16a0 0x7f244815c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.249+0000 7f244f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24480d2370 0x7f24480b8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.249+0000 7f244f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244815d9d0 0x7f24480b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.249+0000 7f244f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24480b7b40 con 0x7f244815d9d0 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.249+0000 7f244f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244815cdb0 con 0x7f24480d2370 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.249+0000 7f244f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24480b6fc0 con 0x7f24480d16a0 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f2447e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24480d16a0 0x7f244815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f244d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244815d9d0 0x7f24480b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f2447e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24480d16a0 0x7f244815c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:45688/0 (socket says 172.21.15.12:45688) 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f2447e00700 1 -- 172.21.15.12:0/2551269691 learned_addr learned my addr 172.21.15.12:0/2551269691 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f2447400700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24480d2370 0x7f24480b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f2447e00700 1 -- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24480d2370 msgr2=0x7f24480b8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f2447e00700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24480d2370 0x7f24480b8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f2447e00700 1 -- 172.21.15.12:0/2551269691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244815d9d0 msgr2=0x7f24480b8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f2447e00700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244815d9d0 0x7f24480b8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f2447e00700 1 -- 172.21.15.12:0/2551269691 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2440009b30 con 0x7f24480d16a0 2024-03-29T11:28:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f244d000700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244815d9d0 0x7f24480b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.250+0000 7f2447e00700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24480d16a0 0x7f244815c870 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f243800e870 tx=0x7f243800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.251+0000 7f243fe00700 1 -- 172.21.15.12:0/2551269691 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438009590 con 0x7f24480d16a0 2024-03-29T11:28:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.251+0000 7f243fe00700 1 -- 172.21.15.12:0/2551269691 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2438013680 con 0x7f24480d16a0 2024-03-29T11:28:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.251+0000 7f243fe00700 1 -- 172.21.15.12:0/2551269691 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24380139f0 con 0x7f24480d16a0 2024-03-29T11:28:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.251+0000 7f244f000700 1 -- 172.21.15.12:0/2551269691 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24480b72a0 con 0x7f24480d16a0 2024-03-29T11:28:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.251+0000 7f244f000700 1 -- 172.21.15.12:0/2551269691 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24480b7860 con 0x7f24480d16a0 2024-03-29T11:28:26.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.252+0000 7f244f000700 1 -- 172.21.15.12:0/2551269691 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24480d1ca0 con 0x7f24480d16a0 2024-03-29T11:28:26.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.253+0000 7f243fe00700 1 -- 172.21.15.12:0/2551269691 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f2438013b50 con 0x7f24480d16a0 2024-03-29T11:28:26.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.254+0000 7f243fe00700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f242806adb0 0x7f242806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.254+0000 7f243fe00700 1 -- 172.21.15.12:0/2551269691 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f2438089700 con 0x7f24480d16a0 2024-03-29T11:28:26.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.254+0000 7f2447400700 1 -- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f242806adb0 msgr2=0x7f242806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:26.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.254+0000 7f2447400700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f242806adb0 0x7f242806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.259+0000 7f243fe00700 1 -- 172.21.15.12:0/2551269691 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2438054460 con 0x7f24480d16a0 2024-03-29T11:28:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.455+0000 7f2447400700 1 -- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f242806adb0 msgr2=0x7f242806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.455+0000 7f2447400700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f242806adb0 0x7f242806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.473+0000 7f244f000700 1 -- 172.21.15.12:0/2551269691 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2448067720 con 0x7f24480d16a0 2024-03-29T11:28:26.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.473+0000 7f243fe00700 1 -- 172.21.15.12:0/2551269691 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v195) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2438057a80 con 0x7f24480d16a0 2024-03-29T11:28:26.463 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:28:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 -- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f242806adb0 msgr2=0x7f242806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f242806adb0 0x7f242806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 -- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24480d16a0 msgr2=0x7f244815c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24480d16a0 0x7f244815c870 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f243800e870 tx=0x7f243800eb80 comp rx=0 tx=0).stop 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 -- 172.21.15.12:0/2551269691 shutdown_connections 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f242806adb0 0x7f242806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f244815d9d0 0x7f24480b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24480d16a0 0x7f244815c870 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 --2- 172.21.15.12:0/2551269691 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24480d2370 0x7f24480b8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 -- 172.21.15.12:0/2551269691 >> 172.21.15.12:0/2551269691 conn(0x7f244808aff0 msgr2=0x7f24480b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 -- 172.21.15.12:0/2551269691 shutdown_connections 2024-03-29T11:28:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.477+0000 7f244f000700 1 -- 172.21.15.12:0/2551269691 wait complete. 2024-03-29T11:28:26.487 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:28:26.612 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:26.622+0000 7f7ab52c7240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:28:26.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.702+0000 7f5e3ea00700 1 Processor -- start 2024-03-29T11:28:26.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.702+0000 7f5e3ea00700 1 -- start start 2024-03-29T11:28:26.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.703+0000 7f5e3ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5e381611b0 0x7f5e3815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.703+0000 7f5e3ea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e380ce270 con 0x7f5e3815f4d0 2024-03-29T11:28:26.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.703+0000 7f5e3ea00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3815b600 con 0x7f5e380cee10 2024-03-29T11:28:26.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.703+0000 7f5e3ea00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e380d2760 con 0x7f5e381611b0 2024-03-29T11:28:26.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.704+0000 7f5e37400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5e3815f4d0 0x7f5e380ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52894/0 (socket says 172.21.15.12:52894) 2024-03-29T11:28:26.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.704+0000 7f5e37400700 1 -- 172.21.15.12:0/2851181813 learned_addr learned my addr 172.21.15.12:0/2851181813 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.704+0000 7f5e37e00700 1 --2- 172.21.15.12:0/2851181813 >> v2:172.21.15.159:3300/0 conn(0x7f5e381611b0 0x7f5e3815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.704+0000 7f5e36a00700 1 -- 172.21.15.12:0/2851181813 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021642082 0 0) 0x7f5e380ce270 con 0x7f5e3815f4d0 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.704+0000 7f5e36a00700 1 -- 172.21.15.12:0/2851181813 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1c0035b0 con 0x7f5e3815f4d0 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.704+0000 7f5e36a00700 1 -- 172.21.15.12:0/2851181813 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094160752 0 0) 0x7f5e3815b600 con 0x7f5e380cee10 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.704+0000 7f5e36a00700 1 -- 172.21.15.12:0/2851181813 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e380ce270 con 0x7f5e380cee10 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.704+0000 7f5e37e00700 1 -- 172.21.15.12:0/2851181813 >> v1:172.21.15.159:6789/0 conn(0x7f5e380cee10 legacy=0x7f5e38161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.704+0000 7f5e37e00700 1 -- 172.21.15.12:0/2851181813 >> v1:172.21.15.12:6789/0 conn(0x7f5e3815f4d0 legacy=0x7f5e380ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.704+0000 7f5e37e00700 1 -- 172.21.15.12:0/2851181813 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e380d2a40 con 0x7f5e381611b0 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.705+0000 7f5e37e00700 1 --2- 172.21.15.12:0/2851181813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e381611b0 0x7f5e3815ef90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009840 tx=0x7f5e28009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cb4bac307d086ef server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.705+0000 7f5e36a00700 1 -- 172.21.15.12:0/2851181813 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28011830 con 0x7f5e381611b0 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.705+0000 7f5e36a00700 1 -- 172.21.15.12:0/2851181813 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e28011990 con 0x7f5e381611b0 2024-03-29T11:28:26.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.705+0000 7f5e36a00700 1 -- 172.21.15.12:0/2851181813 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28011d00 con 0x7f5e381611b0 2024-03-29T11:28:26.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.706+0000 7f5e3ea00700 1 -- 172.21.15.12:0/2851181813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e381611b0 msgr2=0x7f5e3815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.706+0000 7f5e3ea00700 1 --2- 172.21.15.12:0/2851181813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e381611b0 0x7f5e3815ef90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009840 tx=0x7f5e28009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:26.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.706+0000 7f5e3ea00700 1 -- 172.21.15.12:0/2851181813 shutdown_connections 2024-03-29T11:28:26.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.706+0000 7f5e3ea00700 1 --2- 172.21.15.12:0/2851181813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e381611b0 0x7f5e3815ef90 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.706+0000 7f5e3ea00700 1 -- 172.21.15.12:0/2851181813 >> 172.21.15.12:0/2851181813 conn(0x7f5e3808aff0 msgr2=0x7f5e380d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.706+0000 7f5e3ea00700 1 -- 172.21.15.12:0/2851181813 shutdown_connections 2024-03-29T11:28:26.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.706+0000 7f5e3ea00700 1 -- 172.21.15.12:0/2851181813 wait complete. 2024-03-29T11:28:26.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.707+0000 7f5e3ea00700 1 Processor -- start 2024-03-29T11:28:26.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.708+0000 7f5e3ea00700 1 -- start start 2024-03-29T11:28:26.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.708+0000 7f5e3ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e380cee10 0x7f5e380c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.708+0000 7f5e3ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e3815f4d0 0x7f5e380cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.708+0000 7f5e3ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e381611b0 0x7f5e38154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.708+0000 7f5e3ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380cbdb0 con 0x7f5e381611b0 2024-03-29T11:28:26.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.708+0000 7f5e3ea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380cc090 con 0x7f5e380cee10 2024-03-29T11:28:26.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e3ea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380cc370 con 0x7f5e3815f4d0 2024-03-29T11:28:26.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e3ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e381611b0 0x7f5e38154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:26.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e37e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e380cee10 0x7f5e380c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:26.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e37e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e380cee10 0x7f5e380c86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:36410/0 (socket says 172.21.15.12:36410) 2024-03-29T11:28:26.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e37400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e3815f4d0 0x7f5e380cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:26.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e37e00700 1 -- 172.21.15.12:0/587397682 learned_addr learned my addr 172.21.15.12:0/587397682 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:26.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e3ca00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e381611b0 0x7f5e38154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41088/0 (socket says 172.21.15.12:41088) 2024-03-29T11:28:26.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e3ca00700 1 -- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e3815f4d0 msgr2=0x7f5e380cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e3ca00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e3815f4d0 0x7f5e380cb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e3ca00700 1 -- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e380cee10 msgr2=0x7f5e380c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e3ca00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e380cee10 0x7f5e380c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.709+0000 7f5e3ca00700 1 -- 172.21.15.12:0/587397682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e280094a0 con 0x7f5e381611b0 2024-03-29T11:28:26.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.710+0000 7f5e3ca00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e381611b0 0x7f5e38154a70 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000cb10 tx=0x7f5e3000ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:26.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.710+0000 7f5e2fe00700 1 -- 172.21.15.12:0/587397682 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30003c00 con 0x7f5e381611b0 2024-03-29T11:28:26.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.710+0000 7f5e2fe00700 1 -- 172.21.15.12:0/587397682 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e30003d60 con 0x7f5e381611b0 2024-03-29T11:28:26.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.710+0000 7f5e3ea00700 1 -- 172.21.15.12:0/587397682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e380cc650 con 0x7f5e381611b0 2024-03-29T11:28:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.710+0000 7f5e2fe00700 1 -- 172.21.15.12:0/587397682 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3000aa20 con 0x7f5e381611b0 2024-03-29T11:28:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.710+0000 7f5e3ea00700 1 -- 172.21.15.12:0/587397682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e380cba50 con 0x7f5e381611b0 2024-03-29T11:28:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.711+0000 7f5e3ea00700 1 -- 172.21.15.12:0/587397682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e380ccb70 con 0x7f5e381611b0 2024-03-29T11:28:26.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.713+0000 7f5e2fe00700 1 -- 172.21.15.12:0/587397682 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f5e3000ab80 con 0x7f5e381611b0 2024-03-29T11:28:26.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.714+0000 7f5e2fe00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e1806ade0 0x7f5e1806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:26.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.714+0000 7f5e2fe00700 1 -- 172.21.15.12:0/587397682 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f5e300895a0 con 0x7f5e381611b0 2024-03-29T11:28:26.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.714+0000 7f5e37e00700 1 -- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e1806ade0 msgr2=0x7f5e1806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:26.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.714+0000 7f5e37e00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e1806ade0 0x7f5e1806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:26.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.719+0000 7f5e2fe00700 1 -- 172.21.15.12:0/587397682 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e380ccb70 con 0x7f5e381611b0 2024-03-29T11:28:26.825 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:26.835+0000 7f7ab52c7240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:28:26.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.915+0000 7f5e37e00700 1 -- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e1806ade0 msgr2=0x7f5e1806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:26.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.915+0000 7f5e37e00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e1806ade0 0x7f5e1806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.932+0000 7f5e3ea00700 1 -- 172.21.15.12:0/587397682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5e380ccb70 con 0x7f5e381611b0 2024-03-29T11:28:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.933+0000 7f5e2fe00700 1 -- 172.21.15.12:0/587397682 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v195) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5e380ccb70 con 0x7f5e381611b0 2024-03-29T11:28:26.922 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:28:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 -- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e1806ade0 msgr2=0x7f5e1806d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e1806ade0 0x7f5e1806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 -- 172.21.15.12:0/587397682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e381611b0 msgr2=0x7f5e38154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e381611b0 0x7f5e38154a70 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000cb10 tx=0x7f5e3000ce20 comp rx=0 tx=0).stop 2024-03-29T11:28:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 -- 172.21.15.12:0/587397682 shutdown_connections 2024-03-29T11:28:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e1806ade0 0x7f5e1806d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e381611b0 0x7f5e38154a70 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e380cee10 0x7f5e380c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 --2- 172.21.15.12:0/587397682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e3815f4d0 0x7f5e380cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:26.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 -- 172.21.15.12:0/587397682 >> 172.21.15.12:0/587397682 conn(0x7f5e3808aff0 msgr2=0x7f5e380d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 -- 172.21.15.12:0/587397682 shutdown_connections 2024-03-29T11:28:26.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:26.935+0000 7f5e3ea00700 1 -- 172.21.15.12:0/587397682 wait complete. 2024-03-29T11:28:26.943 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:28:26.991 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:27.002+0000 7fed302d6240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:28:27.030 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:27.040+0000 7f7ab52c7240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:28:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.154+0000 7ff089a00700 1 Processor -- start 2024-03-29T11:28:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.154+0000 7ff089a00700 1 -- start start 2024-03-29T11:28:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.155+0000 7ff089a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840cb3f0 0x7ff0840ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.156+0000 7ff089a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff0840ac600 0x7ff08406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.156+0000 7ff089a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08415fd50 con 0x7ff08415b5d0 2024-03-29T11:28:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.156+0000 7ff089a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08415b9b0 con 0x7ff0840ac600 2024-03-29T11:28:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.156+0000 7ff089a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0840aca70 con 0x7ff0840cb3f0 2024-03-29T11:28:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.156+0000 7ff082a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff08415b5d0 0x7ff0840ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52900/0 (socket says 172.21.15.12:52900) 2024-03-29T11:28:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.156+0000 7ff082a00700 1 -- 172.21.15.12:0/2282579479 learned_addr learned my addr 172.21.15.12:0/2282579479 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.156+0000 7ff083400700 1 --2- 172.21.15.12:0/2282579479 >> v2:172.21.15.159:3300/0 conn(0x7ff0840ac600 0x7ff08406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.156+0000 7ff082000700 1 --2- 172.21.15.12:0/2282579479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840cb3f0 0x7ff0840ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.156+0000 7ff081600700 1 -- 172.21.15.12:0/2282579479 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578998797 0 0) 0x7ff08415fd50 con 0x7ff08415b5d0 2024-03-29T11:28:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.156+0000 7ff081600700 1 -- 172.21.15.12:0/2282579479 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0680035b0 con 0x7ff08415b5d0 2024-03-29T11:28:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.157+0000 7ff082000700 1 -- 172.21.15.12:0/2282579479 >> v2:172.21.15.159:3300/0 conn(0x7ff0840ac600 msgr2=0x7ff08406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.157+0000 7ff082000700 1 --2- 172.21.15.12:0/2282579479 >> v2:172.21.15.159:3300/0 conn(0x7ff0840ac600 0x7ff08406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.157+0000 7ff082000700 1 -- 172.21.15.12:0/2282579479 >> v1:172.21.15.12:6789/0 conn(0x7ff08415b5d0 legacy=0x7ff0840ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.157+0000 7ff082000700 1 -- 172.21.15.12:0/2282579479 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08415f140 con 0x7ff0840cb3f0 2024-03-29T11:28:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.157+0000 7ff083400700 1 --2- 172.21.15.12:0/2282579479 >> v2:172.21.15.159:3300/0 conn(0x7ff0840ac600 0x7ff08406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.157+0000 7ff082000700 1 --2- 172.21.15.12:0/2282579479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840cb3f0 0x7ff0840ac0c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff070009840 tx=0x7ff070009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d69ac7e13d09984e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.157+0000 7ff081600700 1 -- 172.21.15.12:0/2282579479 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070011830 con 0x7ff0840cb3f0 2024-03-29T11:28:27.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.157+0000 7ff081600700 1 -- 172.21.15.12:0/2282579479 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff070011990 con 0x7ff0840cb3f0 2024-03-29T11:28:27.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.157+0000 7ff081600700 1 -- 172.21.15.12:0/2282579479 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070011d20 con 0x7ff0840cb3f0 2024-03-29T11:28:27.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.158+0000 7ff089a00700 1 -- 172.21.15.12:0/2282579479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840cb3f0 msgr2=0x7ff0840ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.158+0000 7ff089a00700 1 --2- 172.21.15.12:0/2282579479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840cb3f0 0x7ff0840ac0c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff070009840 tx=0x7ff070009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:27.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.158+0000 7ff089a00700 1 -- 172.21.15.12:0/2282579479 shutdown_connections 2024-03-29T11:28:27.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.158+0000 7ff089a00700 1 --2- 172.21.15.12:0/2282579479 >> v2:172.21.15.159:3300/0 conn(0x7ff0840ac600 0x7ff08406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.158+0000 7ff089a00700 1 --2- 172.21.15.12:0/2282579479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840cb3f0 0x7ff0840ac0c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.158+0000 7ff089a00700 1 -- 172.21.15.12:0/2282579479 >> 172.21.15.12:0/2282579479 conn(0x7ff0840cf190 msgr2=0x7ff0840cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.158+0000 7ff089a00700 1 -- 172.21.15.12:0/2282579479 shutdown_connections 2024-03-29T11:28:27.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.158+0000 7ff089a00700 1 -- 172.21.15.12:0/2282579479 wait complete. 2024-03-29T11:28:27.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.160+0000 7ff089a00700 1 Processor -- start 2024-03-29T11:28:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.160+0000 7ff089a00700 1 -- start start 2024-03-29T11:28:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff089a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0840ac600 0x7ff0840aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff089a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff08415b5d0 0x7ff084152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff089a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840b3050 0x7ff084153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff089a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0840c9dd0 con 0x7ff08415b5d0 2024-03-29T11:28:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff089a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0841533b0 con 0x7ff0840ac600 2024-03-29T11:28:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff089a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0840c8e70 con 0x7ff0840b3050 2024-03-29T11:28:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff082000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff08415b5d0 0x7ff084152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff082a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0840ac600 0x7ff0840aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff082000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff08415b5d0 0x7ff084152e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41098/0 (socket says 172.21.15.12:41098) 2024-03-29T11:28:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff082000700 1 -- 172.21.15.12:0/711861301 learned_addr learned my addr 172.21.15.12:0/711861301 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.161+0000 7ff083400700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840b3050 0x7ff084153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.162+0000 7ff082000700 1 -- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840b3050 msgr2=0x7ff084153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.162+0000 7ff082000700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840b3050 0x7ff084153c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.162+0000 7ff082000700 1 -- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0840ac600 msgr2=0x7ff0840aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.162+0000 7ff082000700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0840ac600 0x7ff0840aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.162+0000 7ff082000700 1 -- 172.21.15.12:0/711861301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c009a70 con 0x7ff08415b5d0 2024-03-29T11:28:27.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.162+0000 7ff082000700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff08415b5d0 0x7ff084152e70 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7ff0700090b0 tx=0x7ff070004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:27.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.162+0000 7ff07aa00700 1 -- 172.21.15.12:0/711861301 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070004550 con 0x7ff08415b5d0 2024-03-29T11:28:27.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.163+0000 7ff07aa00700 1 -- 172.21.15.12:0/711861301 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07001f070 con 0x7ff08415b5d0 2024-03-29T11:28:27.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.163+0000 7ff089a00700 1 -- 172.21.15.12:0/711861301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0700094a0 con 0x7ff08415b5d0 2024-03-29T11:28:27.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.163+0000 7ff07aa00700 1 -- 172.21.15.12:0/711861301 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700038e0 con 0x7ff08415b5d0 2024-03-29T11:28:27.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.163+0000 7ff089a00700 1 -- 172.21.15.12:0/711861301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0840c9430 con 0x7ff08415b5d0 2024-03-29T11:28:27.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.164+0000 7ff089a00700 1 -- 172.21.15.12:0/711861301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0840c9f10 con 0x7ff08415b5d0 2024-03-29T11:28:27.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.165+0000 7ff07aa00700 1 -- 172.21.15.12:0/711861301 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7ff070003a40 con 0x7ff08415b5d0 2024-03-29T11:28:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.166+0000 7ff07aa00700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff06406adb0 0x7ff06406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.166+0000 7ff07aa00700 1 -- 172.21.15.12:0/711861301 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7ff070021070 con 0x7ff08415b5d0 2024-03-29T11:28:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.166+0000 7ff082a00700 1 -- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff06406adb0 msgr2=0x7ff06406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:27.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.166+0000 7ff082a00700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff06406adb0 0x7ff06406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:27.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.170+0000 7ff07aa00700 1 -- 172.21.15.12:0/711861301 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff0840c9f10 con 0x7ff08415b5d0 2024-03-29T11:28:27.222 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:27.233+0000 7fed302d6240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:28:27.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.367+0000 7ff082a00700 1 -- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff06406adb0 msgr2=0x7ff06406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:27.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.367+0000 7ff082a00700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff06406adb0 0x7ff06406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:27.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.431+0000 7ff089a00700 1 -- 172.21.15.12:0/711861301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0840c9f10 con 0x7ff08415b5d0 2024-03-29T11:28:27.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.432+0000 7ff07aa00700 1 -- 172.21.15.12:0/711861301 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7ff0840c9f10 con 0x7ff08415b5d0 2024-03-29T11:28:27.422 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:27.422 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2592133465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2592133465},{"type":"v1","addr":"172.21.15.159:6837","nonce":2592133465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":8381,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/204651369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":204651369},{"type":"v1","addr":"172.21.15.12:6835","nonce":204651369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9616,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2547917697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2547917697},{"type":"v1","addr":"172.21.15.159:6836","nonce":2547917697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:12:26.523825+0000","modified":"2024-03-29T11:12:30.007709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9975":{"gid":9975,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/4176455926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4176455926},{"type":"v1","addr":"172.21.15.12:6833","nonce":4176455926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-29T11:28:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 -- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff06406adb0 msgr2=0x7ff06406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff06406adb0 0x7ff06406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 -- 172.21.15.12:0/711861301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff08415b5d0 msgr2=0x7ff084152e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff08415b5d0 0x7ff084152e70 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7ff0700090b0 tx=0x7ff070004750 comp rx=0 tx=0).stop 2024-03-29T11:28:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 -- 172.21.15.12:0/711861301 shutdown_connections 2024-03-29T11:28:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff06406adb0 0x7ff06406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0840b3050 0x7ff084153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0840ac600 0x7ff0840aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 --2- 172.21.15.12:0/711861301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff08415b5d0 0x7ff084152e70 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 -- 172.21.15.12:0/711861301 >> 172.21.15.12:0/711861301 conn(0x7ff0840cf190 msgr2=0x7ff08415d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 -- 172.21.15.12:0/711861301 shutdown_connections 2024-03-29T11:28:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.434+0000 7ff089a00700 1 -- 172.21.15.12:0/711861301 wait complete. 2024-03-29T11:28:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 89 2024-03-29T11:28:27.442 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:27.448 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:27.456+0000 7fed302d6240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:28:27.504 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:27.514+0000 7f7ab52c7240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:28:27.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.657+0000 7fa13c400700 1 Processor -- start 2024-03-29T11:28:27.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.658+0000 7fa13c400700 1 -- start start 2024-03-29T11:28:27.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.659+0000 7fa13c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13415b5d0 0x7fa1340ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.659+0000 7fa13c400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1340ac7d0 0x7fa13406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.659+0000 7fa13c400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13415fd50 con 0x7fa1340cb3f0 2024-03-29T11:28:27.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.659+0000 7fa13c400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa13415b9b0 con 0x7fa1340ac7d0 2024-03-29T11:28:27.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.659+0000 7fa13c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13415f140 con 0x7fa13415b5d0 2024-03-29T11:28:27.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.659+0000 7fa13a400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1340ac7d0 0x7fa13406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:27.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.659+0000 7fa13a400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1340ac7d0 0x7fa13406c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41104/0 (socket says 172.21.15.12:41104) 2024-03-29T11:28:27.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.659+0000 7fa13a400700 1 -- 172.21.15.12:0/2803506035 learned_addr learned my addr 172.21.15.12:0/2803506035 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:27.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.659+0000 7fa139a00700 1 --2- 172.21.15.12:0/2803506035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13415b5d0 0x7fa1340ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.660+0000 7fa13a400700 1 -- 172.21.15.12:0/2803506035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13415b5d0 msgr2=0x7fa1340ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.660+0000 7fa13a400700 1 --2- 172.21.15.12:0/2803506035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13415b5d0 0x7fa1340ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.660+0000 7fa13a400700 1 -- 172.21.15.12:0/2803506035 >> v1:172.21.15.159:6789/0 conn(0x7fa1340cb3f0 legacy=0x7fa1340ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.660+0000 7fa13a400700 1 -- 172.21.15.12:0/2803506035 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13415f420 con 0x7fa1340ac7d0 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.660+0000 7fa13a400700 1 --2- 172.21.15.12:0/2803506035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 0x7fa13406c9d0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009e50 tx=0x7fa12c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466abe32fa306292 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.660+0000 7fa133e00700 1 -- 172.21.15.12:0/2803506035 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c00e070 con 0x7fa1340ac7d0 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.660+0000 7fa133e00700 1 -- 172.21.15.12:0/2803506035 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12c00b910 con 0x7fa1340ac7d0 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.660+0000 7fa133e00700 1 -- 172.21.15.12:0/2803506035 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c00bcc0 con 0x7fa1340ac7d0 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.661+0000 7fa13c400700 1 -- 172.21.15.12:0/2803506035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 msgr2=0x7fa13406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.661+0000 7fa13c400700 1 --2- 172.21.15.12:0/2803506035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 0x7fa13406c9d0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009e50 tx=0x7fa12c00b330 comp rx=0 tx=0).stop 2024-03-29T11:28:27.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.661+0000 7fa13c400700 1 -- 172.21.15.12:0/2803506035 shutdown_connections 2024-03-29T11:28:27.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.661+0000 7fa13c400700 1 --2- 172.21.15.12:0/2803506035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 0x7fa13406c9d0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.661+0000 7fa13c400700 1 --2- 172.21.15.12:0/2803506035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13415b5d0 0x7fa1340ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.661+0000 7fa13c400700 1 -- 172.21.15.12:0/2803506035 >> 172.21.15.12:0/2803506035 conn(0x7fa1340cf190 msgr2=0x7fa1340cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.661+0000 7fa13c400700 1 -- 172.21.15.12:0/2803506035 shutdown_connections 2024-03-29T11:28:27.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.661+0000 7fa13c400700 1 -- 172.21.15.12:0/2803506035 wait complete. 2024-03-29T11:28:27.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.663+0000 7fa13c400700 1 Processor -- start 2024-03-29T11:28:27.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.664+0000 7fa13c400700 1 -- start start 2024-03-29T11:28:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa13c400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 0x7fa1340709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa13c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13415b5d0 0x7fa1340ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa13c400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1340c9680 0x7fa1340c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa13c400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1341549b0 con 0x7fa1340ac7d0 2024-03-29T11:28:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa13c400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1340cad80 con 0x7fa1340c9680 2024-03-29T11:28:27.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa13c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa134155510 con 0x7fa13415b5d0 2024-03-29T11:28:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa139a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 0x7fa1340709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa139a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 0x7fa1340709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41112/0 (socket says 172.21.15.12:41112) 2024-03-29T11:28:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa139a00700 1 -- 172.21.15.12:0/3259483474 learned_addr learned my addr 172.21.15.12:0/3259483474 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa139000700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13415b5d0 0x7fa1340ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.665+0000 7fa13a400700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1340c9680 0x7fa1340c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa139a00700 1 -- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13415b5d0 msgr2=0x7fa1340ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa139a00700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13415b5d0 0x7fa1340ca570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa139a00700 1 -- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1340c9680 msgr2=0x7fa1340c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa139a00700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1340c9680 0x7fa1340c86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa139a00700 1 -- 172.21.15.12:0/3259483474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12c009b30 con 0x7fa1340ac7d0 2024-03-29T11:28:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa139a00700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 0x7fa1340709c0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fa12800e870 tx=0x7fa12800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa131600700 1 -- 172.21.15.12:0/3259483474 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa128009590 con 0x7fa1340ac7d0 2024-03-29T11:28:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa131600700 1 -- 172.21.15.12:0/3259483474 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa128013680 con 0x7fa1340ac7d0 2024-03-29T11:28:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa131600700 1 -- 172.21.15.12:0/3259483474 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1280139f0 con 0x7fa1340ac7d0 2024-03-29T11:28:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa13c400700 1 -- 172.21.15.12:0/3259483474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1340caab0 con 0x7fa1340ac7d0 2024-03-29T11:28:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.666+0000 7fa13c400700 1 -- 172.21.15.12:0/3259483474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa134154ec0 con 0x7fa1340ac7d0 2024-03-29T11:28:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.667+0000 7fa13c400700 1 -- 172.21.15.12:0/3259483474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa134067720 con 0x7fa1340ac7d0 2024-03-29T11:28:27.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.669+0000 7fa131600700 1 -- 172.21.15.12:0/3259483474 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fa128013b50 con 0x7fa1340ac7d0 2024-03-29T11:28:27.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.670+0000 7fa131600700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa11406adb0 0x7fa11406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:27.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.670+0000 7fa131600700 1 -- 172.21.15.12:0/3259483474 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fa128089ca0 con 0x7fa1340ac7d0 2024-03-29T11:28:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.670+0000 7fa139000700 1 -- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa11406adb0 msgr2=0x7fa11406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:27.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.670+0000 7fa139000700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa11406adb0 0x7fa11406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:27.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.674+0000 7fa131600700 1 -- 172.21.15.12:0/3259483474 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa128054b80 con 0x7fa1340ac7d0 2024-03-29T11:28:27.712 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:27.722+0000 7f7ab52c7240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:28:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.871+0000 7fa139000700 1 -- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa11406adb0 msgr2=0x7fa11406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.871+0000 7fa139000700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa11406adb0 0x7fa11406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:28:27.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.890+0000 7fa13c400700 1 -- 172.21.15.12:0/3259483474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1340cb8e0 con 0x7fa1340ac7d0 2024-03-29T11:28:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.891+0000 7fa131600700 1 -- 172.21.15.12:0/3259483474 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+17186 (secure 0 0 0) 0x7fa1280581a0 con 0x7fa1340ac7d0 2024-03-29T11:28:27.881 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:27.881 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":195,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:28:22.488235+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:12:24.038574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-03-29T11:12:25.509384+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":195,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/3631785847":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/750511589":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/2791060580":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/479652442":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/180231134":"2024-03-30T11:12:43.757966+0000","172.21.15.159:6833/43444":"2024-03-30T11:12:43.757966+0000","172.21.15.159:6832/43444":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/2825856150":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3819915206":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/2176927052":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3123015140":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/1699207236":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3816982205":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/4147178761":"2024-03-30T11:12:43.757966+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 -- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa11406adb0 msgr2=0x7fa11406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa11406adb0 0x7fa11406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 -- 172.21.15.12:0/3259483474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 msgr2=0x7fa1340709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 0x7fa1340709c0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fa12800e870 tx=0x7fa12800eb80 comp rx=0 tx=0).stop 2024-03-29T11:28:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 -- 172.21.15.12:0/3259483474 shutdown_connections 2024-03-29T11:28:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa11406adb0 0x7fa11406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1340c9680 0x7fa1340c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1340ac7d0 0x7fa1340709c0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 --2- 172.21.15.12:0/3259483474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13415b5d0 0x7fa1340ca570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 -- 172.21.15.12:0/3259483474 >> 172.21.15.12:0/3259483474 conn(0x7fa1340cf190 msgr2=0x7fa1340aedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 -- 172.21.15.12:0/3259483474 shutdown_connections 2024-03-29T11:28:27.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:27.894+0000 7fa13c400700 1 -- 172.21.15.12:0/3259483474 wait complete. 2024-03-29T11:28:27.902 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T11:28:27.902 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:28:27.970 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:27.980+0000 7fed302d6240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:28:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.112+0000 7fa0d3000700 1 Processor -- start 2024-03-29T11:28:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.113+0000 7fa0d3000700 1 -- start start 2024-03-29T11:28:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.114+0000 7fa0d3000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa0cc15d640 0x7fa0cc0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.114+0000 7fa0d3000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc0d30d0 con 0x7fa0cc0ceec0 2024-03-29T11:28:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.114+0000 7fa0d3000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc0b6840 con 0x7fa0cc1619d0 2024-03-29T11:28:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.114+0000 7fa0d3000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0cc0d2880 con 0x7fa0cc15d640 2024-03-29T11:28:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.114+0000 7fa0cb400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa0cc0ceec0 0x7fa0cc1612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52916/0 (socket says 172.21.15.12:52916) 2024-03-29T11:28:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.114+0000 7fa0cb400700 1 -- 172.21.15.12:0/804117232 learned_addr learned my addr 172.21.15.12:0/804117232 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.114+0000 7fa0cbe00700 1 --2- 172.21.15.12:0/804117232 >> v2:172.21.15.159:3300/0 conn(0x7fa0cc15d640 0x7fa0cc0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.115+0000 7fa0caa00700 1 -- 172.21.15.12:0/804117232 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996466480 0 0) 0x7fa0cc0d30d0 con 0x7fa0cc0ceec0 2024-03-29T11:28:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.115+0000 7fa0caa00700 1 -- 172.21.15.12:0/804117232 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b00035b0 con 0x7fa0cc0ceec0 2024-03-29T11:28:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.115+0000 7fa0caa00700 1 -- 172.21.15.12:0/804117232 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345058295 0 0) 0x7fa0cc0b6840 con 0x7fa0cc1619d0 2024-03-29T11:28:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.115+0000 7fa0caa00700 1 -- 172.21.15.12:0/804117232 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0cc0d30d0 con 0x7fa0cc1619d0 2024-03-29T11:28:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.115+0000 7fa0cbe00700 1 -- 172.21.15.12:0/804117232 >> v1:172.21.15.159:6789/0 conn(0x7fa0cc1619d0 legacy=0x7fa0cc0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.115+0000 7fa0cbe00700 1 -- 172.21.15.12:0/804117232 >> v1:172.21.15.12:6789/0 conn(0x7fa0cc0ceec0 legacy=0x7fa0cc1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.115+0000 7fa0caa00700 1 -- 172.21.15.12:0/804117232 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3939192885 0 0) 0x7fa0b00035b0 con 0x7fa0cc0ceec0 2024-03-29T11:28:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.115+0000 7fa0cbe00700 1 -- 172.21.15.12:0/804117232 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc0d2b30 con 0x7fa0cc15d640 2024-03-29T11:28:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.115+0000 7fa0cbe00700 1 --2- 172.21.15.12:0/804117232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 0x7fa0cc0d2340 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009840 tx=0x7fa0bc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6048efbe27c531ee server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.116+0000 7fa0caa00700 1 -- 172.21.15.12:0/804117232 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8001670 con 0x7fa0cc15d640 2024-03-29T11:28:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.116+0000 7fa0caa00700 1 -- 172.21.15.12:0/804117232 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc004030 con 0x7fa0cc15d640 2024-03-29T11:28:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.116+0000 7fa0caa00700 1 -- 172.21.15.12:0/804117232 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0043c0 con 0x7fa0cc15d640 2024-03-29T11:28:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.116+0000 7fa0d3000700 1 -- 172.21.15.12:0/804117232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 msgr2=0x7fa0cc0d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.116+0000 7fa0d3000700 1 --2- 172.21.15.12:0/804117232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 0x7fa0cc0d2340 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009840 tx=0x7fa0bc009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.116+0000 7fa0d3000700 1 -- 172.21.15.12:0/804117232 shutdown_connections 2024-03-29T11:28:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.116+0000 7fa0d3000700 1 --2- 172.21.15.12:0/804117232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 0x7fa0cc0d2340 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.116+0000 7fa0d3000700 1 -- 172.21.15.12:0/804117232 >> 172.21.15.12:0/804117232 conn(0x7fa0cc08aff0 msgr2=0x7fa0cc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.116+0000 7fa0d3000700 1 -- 172.21.15.12:0/804117232 shutdown_connections 2024-03-29T11:28:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.117+0000 7fa0d3000700 1 -- 172.21.15.12:0/804117232 wait complete. 2024-03-29T11:28:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.118+0000 7fa0d3000700 1 Processor -- start 2024-03-29T11:28:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.118+0000 7fa0d3000700 1 -- start start 2024-03-29T11:28:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.119+0000 7fa0d3000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0cc0ceec0 0x7fa0cc1f7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.119+0000 7fa0d3000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 0x7fa0cc0d17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.119+0000 7fa0d3000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc1619d0 0x7fa0cc1f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.119+0000 7fa0d3000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc0d2320 con 0x7fa0cc1619d0 2024-03-29T11:28:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.119+0000 7fa0d3000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc0d2040 con 0x7fa0cc15d640 2024-03-29T11:28:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.119+0000 7fa0d3000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc1f6740 con 0x7fa0cc0ceec0 2024-03-29T11:28:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0cb400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 0x7fa0cc0d17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0d1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc1619d0 0x7fa0cc1f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0cbe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0cc0ceec0 0x7fa0cc1f7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0cb400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 0x7fa0cc0d17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:36466/0 (socket says 172.21.15.12:36466) 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0cb400700 1 -- 172.21.15.12:0/837328954 learned_addr learned my addr 172.21.15.12:0/837328954 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0cb400700 1 -- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0cc0ceec0 msgr2=0x7fa0cc1f7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0cb400700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0cc0ceec0 0x7fa0cc1f7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0cb400700 1 -- 172.21.15.12:0/837328954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc1619d0 msgr2=0x7fa0cc1f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0cb400700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc1619d0 0x7fa0cc1f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0cb400700 1 -- 172.21.15.12:0/837328954 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0094a0 con 0x7fa0cc15d640 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0cbe00700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0cc0ceec0 0x7fa0cc1f7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.120+0000 7fa0d1000700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc1619d0 0x7fa0cc1f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.121+0000 7fa0cb400700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 0x7fa0cc0d17f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800c750 tx=0x7fa0b800ca60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.121+0000 7fa0c3e00700 1 -- 172.21.15.12:0/837328954 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8004c80 con 0x7fa0cc15d640 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.121+0000 7fa0c3e00700 1 -- 172.21.15.12:0/837328954 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b8004de0 con 0x7fa0cc15d640 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.121+0000 7fa0c3e00700 1 -- 172.21.15.12:0/837328954 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b80072d0 con 0x7fa0cc15d640 2024-03-29T11:28:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.121+0000 7fa0d3000700 1 -- 172.21.15.12:0/837328954 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc1f6a20 con 0x7fa0cc15d640 2024-03-29T11:28:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.121+0000 7fa0d3000700 1 -- 172.21.15.12:0/837328954 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc0d1d30 con 0x7fa0cc15d640 2024-03-29T11:28:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.123+0000 7fa0d3000700 1 -- 172.21.15.12:0/837328954 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0cc1582c0 con 0x7fa0cc15d640 2024-03-29T11:28:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.123+0000 7fa0c3e00700 1 -- 172.21.15.12:0/837328954 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 113) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fa0b8007430 con 0x7fa0cc15d640 2024-03-29T11:28:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.124+0000 7fa0c3e00700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa0ac06adb0 0x7fa0ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.124+0000 7fa0c3e00700 1 -- 172.21.15.12:0/837328954 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fa0b8089600 con 0x7fa0cc15d640 2024-03-29T11:28:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.124+0000 7fa0cbe00700 1 -- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa0ac06adb0 msgr2=0x7fa0ac06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:28:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.124+0000 7fa0cbe00700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa0ac06adb0 0x7fa0ac06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:28:28.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.129+0000 7fa0c3e00700 1 -- 172.21.15.12:0/837328954 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa0b8054360 con 0x7fa0cc15d640 2024-03-29T11:28:28.158 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.168+0000 7f7ab52c7240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:28:28.196 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:28.206+0000 7fed302d6240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:28:28.270 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.280+0000 7f7ab52c7240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:28:28.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.292+0000 7fa0c3e00700 1 -- 172.21.15.12:0/837328954 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mgrmap(e 114) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fa0b8054730 con 0x7fa0cc15d640 2024-03-29T11:28:28.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.293+0000 7fa0c3e00700 1 -- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa0ac06adb0 msgr2=0x7fa0ac06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:28.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.293+0000 7fa0c3e00700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa0ac06adb0 0x7fa0ac06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.351 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.360+0000 7f7a28400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:28:28.351 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.361+0000 7f7a28400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:28:28.351 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.361+0000 7f7a28400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:28:28.351 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.361+0000 7f7a28400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:28:28.351 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.361+0000 7f7a28400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:28:28.352 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.361+0000 7f7a24800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:28:28.352 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.361+0000 7f7a24800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:28:28.352 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.361+0000 7f7a24800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:28:28.352 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.361+0000 7f7a24800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:28:28.352 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:28:28.361+0000 7f7a24800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:28:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.399+0000 7fa0d3000700 1 -- 172.21.15.12:0/837328954 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0cc1f6f40 con 0x7fa0cc15d640 2024-03-29T11:28:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.400+0000 7fa0c3e00700 1 -- 172.21.15.12:0/837328954 <== mon.1 v2:172.21.15.159:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fa0cc1f6f40 con 0x7fa0cc15d640 2024-03-29T11:28:28.389 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:28.390 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2592133465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2592133465},{"type":"v1","addr":"172.21.15.159:6837","nonce":2592133465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":8381,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/204651369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":204651369},{"type":"v1","addr":"172.21.15.12:6835","nonce":204651369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9616,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2547917697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2547917697},{"type":"v1","addr":"172.21.15.159:6836","nonce":2547917697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:12:26.523825+0000","modified":"2024-03-29T11:12:30.007709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9975":{"gid":9975,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/4176455926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4176455926},{"type":"v1","addr":"172.21.15.12:6833","nonce":4176455926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-29T11:28:28.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.402+0000 7fa0d3000700 1 -- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 msgr2=0x7fa0cc0d17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.402+0000 7fa0d3000700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 0x7fa0cc0d17f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800c750 tx=0x7fa0b800ca60 comp rx=0 tx=0).stop 2024-03-29T11:28:28.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.402+0000 7fa0d3000700 1 -- 172.21.15.12:0/837328954 shutdown_connections 2024-03-29T11:28:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.402+0000 7fa0d3000700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa0ac06adb0 0x7fa0ac06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.402+0000 7fa0d3000700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc1619d0 0x7fa0cc1f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.402+0000 7fa0d3000700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0cc0ceec0 0x7fa0cc1f7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.402+0000 7fa0d3000700 1 --2- 172.21.15.12:0/837328954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0cc15d640 0x7fa0cc0d17f0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.402+0000 7fa0d3000700 1 -- 172.21.15.12:0/837328954 >> 172.21.15.12:0/837328954 conn(0x7fa0cc08aff0 msgr2=0x7fa0cc0b7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.403+0000 7fa0d3000700 1 -- 172.21.15.12:0/837328954 shutdown_connections 2024-03-29T11:28:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.403+0000 7fa0d3000700 1 -- 172.21.15.12:0/837328954 wait complete. 2024-03-29T11:28:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 89 2024-03-29T11:28:28.416 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:28:28.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.639+0000 7fe251600700 1 Processor -- start 2024-03-29T11:28:28.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.639+0000 7fe251600700 1 -- start start 2024-03-29T11:28:28.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe251600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe24c160c90 0x7fe24c0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe251600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe24c0718e0 0x7fe24c0b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe251600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0709c0 0x7fe24c06cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe251600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24c0ac4d0 con 0x7fe24c160c90 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe251600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24c071e50 con 0x7fe24c0718e0 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe251600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c072130 con 0x7fe24c0709c0 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe24aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe24c160c90 0x7fe24c0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe24aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe24c160c90 0x7fe24c0b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41140/0 (socket says 172.21.15.12:41140) 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe24aa00700 1 -- 172.21.15.12:0/3792405523 learned_addr learned my addr 172.21.15.12:0/3792405523 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe24a000700 1 --2- 172.21.15.12:0/3792405523 >> v2:172.21.15.159:3300/0 conn(0x7fe24c0718e0 0x7fe24c0b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.640+0000 7fe24b400700 1 --2- 172.21.15.12:0/3792405523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0709c0 0x7fe24c06cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.641+0000 7fe24aa00700 1 -- 172.21.15.12:0/3792405523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0709c0 msgr2=0x7fe24c06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.641+0000 7fe24aa00700 1 --2- 172.21.15.12:0/3792405523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0709c0 0x7fe24c06cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.641+0000 7fe24aa00700 1 -- 172.21.15.12:0/3792405523 >> v2:172.21.15.159:3300/0 conn(0x7fe24c0718e0 msgr2=0x7fe24c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.641+0000 7fe24aa00700 1 --2- 172.21.15.12:0/3792405523 >> v2:172.21.15.159:3300/0 conn(0x7fe24c0718e0 0x7fe24c0b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.641+0000 7fe24aa00700 1 -- 172.21.15.12:0/3792405523 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c072410 con 0x7fe24c160c90 2024-03-29T11:28:28.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.641+0000 7fe24b400700 1 --2- 172.21.15.12:0/3792405523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0709c0 0x7fe24c06cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:28.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.641+0000 7fe24aa00700 1 --2- 172.21.15.12:0/3792405523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 0x7fe24c0b20e0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fe24c06c530 tx=0x7fe23c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ef2c7778e6590c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.641+0000 7fe249600700 1 -- 172.21.15.12:0/3792405523 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c011830 con 0x7fe24c160c90 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.641+0000 7fe249600700 1 -- 172.21.15.12:0/3792405523 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23c011990 con 0x7fe24c160c90 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.641+0000 7fe249600700 1 -- 172.21.15.12:0/3792405523 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c011d20 con 0x7fe24c160c90 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.642+0000 7fe251600700 1 -- 172.21.15.12:0/3792405523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 msgr2=0x7fe24c0b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.642+0000 7fe251600700 1 --2- 172.21.15.12:0/3792405523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 0x7fe24c0b20e0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fe24c06c530 tx=0x7fe23c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.642+0000 7fe251600700 1 -- 172.21.15.12:0/3792405523 shutdown_connections 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.642+0000 7fe251600700 1 --2- 172.21.15.12:0/3792405523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0709c0 0x7fe24c06cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.642+0000 7fe251600700 1 --2- 172.21.15.12:0/3792405523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 0x7fe24c0b20e0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.642+0000 7fe251600700 1 --2- 172.21.15.12:0/3792405523 >> v2:172.21.15.159:3300/0 conn(0x7fe24c0718e0 0x7fe24c0b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.642+0000 7fe251600700 1 -- 172.21.15.12:0/3792405523 >> 172.21.15.12:0/3792405523 conn(0x7fe24c0cf190 msgr2=0x7fe24c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.642+0000 7fe251600700 1 -- 172.21.15.12:0/3792405523 shutdown_connections 2024-03-29T11:28:28.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.642+0000 7fe251600700 1 -- 172.21.15.12:0/3792405523 wait complete. 2024-03-29T11:28:28.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.645+0000 7fe251600700 1 Processor -- start 2024-03-29T11:28:28.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.645+0000 7fe251600700 1 -- start start 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.646+0000 7fe251600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24c0709c0 0x7fe24c14c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.646+0000 7fe251600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0718e0 0x7fe24c148f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe251600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 0x7fe24c15bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe251600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c09d2b0 con 0x7fe24c160c90 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe251600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c149d20 con 0x7fe24c0709c0 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe251600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c14c780 con 0x7fe24c0718e0 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 0x7fe24c15bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 0x7fe24c15bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41154/0 (socket says 172.21.15.12:41154) 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24b400700 1 -- 172.21.15.12:0/1694321554 learned_addr learned my addr 172.21.15.12:0/1694321554 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24aa00700 1 --2- 172.21.15.12:0/1694321554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24c0709c0 0x7fe24c14c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24a000700 1 --2- 172.21.15.12:0/1694321554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0718e0 0x7fe24c148f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24aa00700 1 -- 172.21.15.12:0/1694321554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0718e0 msgr2=0x7fe24c148f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24aa00700 1 --2- 172.21.15.12:0/1694321554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0718e0 0x7fe24c148f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24aa00700 1 -- 172.21.15.12:0/1694321554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 msgr2=0x7fe24c15bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24aa00700 1 --2- 172.21.15.12:0/1694321554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 0x7fe24c15bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24aa00700 1 -- 172.21.15.12:0/1694321554 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23c009550 con 0x7fe24c0709c0 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.647+0000 7fe24b400700 1 --2- 172.21.15.12:0/1694321554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 0x7fe24c15bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.648+0000 7fe24aa00700 1 --2- 172.21.15.12:0/1694321554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24c0709c0 0x7fe24c14c240 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe23c016000 tx=0x7fe23c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.648+0000 7fe242a00700 1 -- 172.21.15.12:0/1694321554 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c004510 con 0x7fe24c0709c0 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.648+0000 7fe242a00700 1 -- 172.21.15.12:0/1694321554 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23c01e070 con 0x7fe24c0709c0 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.648+0000 7fe242a00700 1 -- 172.21.15.12:0/1694321554 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c0038e0 con 0x7fe24c0709c0 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.648+0000 7fe251600700 1 -- 172.21.15.12:0/1694321554 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24c149490 con 0x7fe24c0709c0 2024-03-29T11:28:28.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.648+0000 7fe251600700 1 -- 172.21.15.12:0/1694321554 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24c09ccf0 con 0x7fe24c0709c0 2024-03-29T11:28:28.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.649+0000 7fe251600700 1 -- 172.21.15.12:0/1694321554 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24c067720 con 0x7fe24c0709c0 2024-03-29T11:28:28.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.650+0000 7fe242a00700 1 -- 172.21.15.12:0/1694321554 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 114) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fe23c003a40 con 0x7fe24c0709c0 2024-03-29T11:28:28.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.651+0000 7fe242a00700 1 -- 172.21.15.12:0/1694321554 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6727+0+0 (secure 0 0 0) 0x7fe23c089fe0 con 0x7fe24c0709c0 2024-03-29T11:28:28.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.656+0000 7fe242a00700 1 -- 172.21.15.12:0/1694321554 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe23c055120 con 0x7fe24c0709c0 2024-03-29T11:28:28.678 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:28.687+0000 7fed302d6240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:28:28.796 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:28:28.806+0000 7fed302d6240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:28:28.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.920+0000 7fe251600700 1 -- 172.21.15.12:0/1694321554 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe24c15fa50 con 0x7fe24c0709c0 2024-03-29T11:28:28.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.920+0000 7fe242a00700 1 -- 172.21.15.12:0/1694321554 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fe23c020090 con 0x7fe24c0709c0 2024-03-29T11:28:28.910 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:28.910 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2592133465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2592133465},{"type":"v1","addr":"172.21.15.159:6837","nonce":2592133465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":8381,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/204651369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":204651369},{"type":"v1","addr":"172.21.15.12:6835","nonce":204651369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9616,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2547917697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2547917697},{"type":"v1","addr":"172.21.15.159:6836","nonce":2547917697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:12:26.523825+0000","modified":"2024-03-29T11:12:30.007709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9975":{"gid":9975,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/4176455926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":4176455926},{"type":"v1","addr":"172.21.15.12:6833","nonce":4176455926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-29T11:28:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.923+0000 7fe251600700 1 -- 172.21.15.12:0/1694321554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24c0709c0 msgr2=0x7fe24c14c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.923+0000 7fe251600700 1 --2- 172.21.15.12:0/1694321554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24c0709c0 0x7fe24c14c240 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe23c016000 tx=0x7fe23c004750 comp rx=0 tx=0).stop 2024-03-29T11:28:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.923+0000 7fe251600700 1 -- 172.21.15.12:0/1694321554 shutdown_connections 2024-03-29T11:28:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.923+0000 7fe251600700 1 --2- 172.21.15.12:0/1694321554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe24c160c90 0x7fe24c15bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.923+0000 7fe251600700 1 --2- 172.21.15.12:0/1694321554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24c0709c0 0x7fe24c14c240 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.923+0000 7fe251600700 1 --2- 172.21.15.12:0/1694321554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe24c0718e0 0x7fe24c148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.923+0000 7fe251600700 1 -- 172.21.15.12:0/1694321554 >> 172.21.15.12:0/1694321554 conn(0x7fe24c0cf190 msgr2=0x7fe24c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.923+0000 7fe251600700 1 -- 172.21.15.12:0/1694321554 shutdown_connections 2024-03-29T11:28:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:28.923+0000 7fe251600700 1 -- 172.21.15.12:0/1694321554 wait complete. 2024-03-29T11:28:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 89 2024-03-29T11:28:28.931 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:29.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.139+0000 7f8d34a00700 1 Processor -- start 2024-03-29T11:28:29.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.140+0000 7f8d34a00700 1 -- start start 2024-03-29T11:28:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.141+0000 7f8d34a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8d300b27f0 0x7f8d300ac960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.141+0000 7f8d34a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d300ac4d0 con 0x7f8d30160c90 2024-03-29T11:28:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.141+0000 7f8d34a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d30071e50 con 0x7f8d300718e0 2024-03-29T11:28:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.141+0000 7f8d34a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d30072130 con 0x7f8d300b27f0 2024-03-29T11:28:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.141+0000 7f8d2de00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8d30160c90 0x7f8d300b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36282/0 (socket says 172.21.15.12:36282) 2024-03-29T11:28:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.141+0000 7f8d2de00700 1 -- 172.21.15.12:0/1150773557 learned_addr learned my addr 172.21.15.12:0/1150773557 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.141+0000 7f8d2d400700 1 --2- 172.21.15.12:0/1150773557 >> v2:172.21.15.159:3300/0 conn(0x7f8d300b27f0 0x7f8d300ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.141+0000 7f8d2ca00700 1 -- 172.21.15.12:0/1150773557 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969266643 0 0) 0x7f8d300ac4d0 con 0x7f8d30160c90 2024-03-29T11:28:29.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.142+0000 7f8d2ca00700 1 -- 172.21.15.12:0/1150773557 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d140035b0 con 0x7f8d30160c90 2024-03-29T11:28:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.142+0000 7f8d2ca00700 1 -- 172.21.15.12:0/1150773557 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4248302600 0 0) 0x7f8d140035b0 con 0x7f8d30160c90 2024-03-29T11:28:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.142+0000 7f8d2d400700 1 -- 172.21.15.12:0/1150773557 >> v1:172.21.15.159:6789/0 conn(0x7f8d300718e0 legacy=0x7f8d300acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.142+0000 7f8d2d400700 1 -- 172.21.15.12:0/1150773557 >> v1:172.21.15.12:6789/0 conn(0x7f8d30160c90 legacy=0x7f8d300b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.142+0000 7f8d2d400700 1 -- 172.21.15.12:0/1150773557 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d30072410 con 0x7f8d300b27f0 2024-03-29T11:28:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.142+0000 7f8d2d400700 1 --2- 172.21.15.12:0/1150773557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d300b27f0 0x7f8d300ac960 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009840 tx=0x7f8d1c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8872530df1c93d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.142+0000 7f8d2ca00700 1 -- 172.21.15.12:0/1150773557 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20001670 con 0x7f8d300b27f0 2024-03-29T11:28:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.142+0000 7f8d2ca00700 1 -- 172.21.15.12:0/1150773557 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c004030 con 0x7f8d300b27f0 2024-03-29T11:28:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.142+0000 7f8d2ca00700 1 -- 172.21.15.12:0/1150773557 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c0043c0 con 0x7f8d300b27f0 2024-03-29T11:28:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.143+0000 7f8d34a00700 1 -- 172.21.15.12:0/1150773557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d300b27f0 msgr2=0x7f8d300ac960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.143+0000 7f8d34a00700 1 --2- 172.21.15.12:0/1150773557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d300b27f0 0x7f8d300ac960 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009840 tx=0x7f8d1c009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.143+0000 7f8d34a00700 1 -- 172.21.15.12:0/1150773557 shutdown_connections 2024-03-29T11:28:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.143+0000 7f8d34a00700 1 --2- 172.21.15.12:0/1150773557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d300b27f0 0x7f8d300ac960 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.143+0000 7f8d34a00700 1 -- 172.21.15.12:0/1150773557 >> 172.21.15.12:0/1150773557 conn(0x7f8d300cf190 msgr2=0x7f8d300cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.143+0000 7f8d34a00700 1 -- 172.21.15.12:0/1150773557 shutdown_connections 2024-03-29T11:28:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.143+0000 7f8d34a00700 1 -- 172.21.15.12:0/1150773557 wait complete. 2024-03-29T11:28:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.145+0000 7f8d34a00700 1 Processor -- start 2024-03-29T11:28:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.145+0000 7f8d34a00700 1 -- start start 2024-03-29T11:28:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d34a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d300718e0 0x7f8d3014a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d34a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d300b27f0 0x7f8d3015a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d34a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d30160c90 0x7f8d301f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d34a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d30159b70 con 0x7f8d30160c90 2024-03-29T11:28:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d34a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301f3cd0 con 0x7f8d300b27f0 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d34a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d30158b60 con 0x7f8d300718e0 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d2de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d300718e0 0x7f8d3014a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d2e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d30160c90 0x7f8d301f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d2de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d300718e0 0x7f8d3014a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:41224/0 (socket says 172.21.15.12:41224) 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d2de00700 1 -- 172.21.15.12:0/3242832459 learned_addr learned my addr 172.21.15.12:0/3242832459 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.146+0000 7f8d2d400700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d300b27f0 0x7f8d3015a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d2de00700 1 -- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d300b27f0 msgr2=0x7f8d3015a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d2de00700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d300b27f0 0x7f8d3015a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d2de00700 1 -- 172.21.15.12:0/3242832459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d30160c90 msgr2=0x7f8d301f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d2de00700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d30160c90 0x7f8d301f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d2de00700 1 -- 172.21.15.12:0/3242832459 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c0094a0 con 0x7f8d300718e0 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d2de00700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d300718e0 0x7f8d3014a6e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000c520 tx=0x7f8d2000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d26000700 1 -- 172.21.15.12:0/3242832459 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20004c80 con 0x7f8d300718e0 2024-03-29T11:28:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d26000700 1 -- 172.21.15.12:0/3242832459 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d20004de0 con 0x7f8d300718e0 2024-03-29T11:28:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d34a00700 1 -- 172.21.15.12:0/3242832459 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d301596e0 con 0x7f8d300718e0 2024-03-29T11:28:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d26000700 1 -- 172.21.15.12:0/3242832459 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20007310 con 0x7f8d300718e0 2024-03-29T11:28:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.147+0000 7f8d34a00700 1 -- 172.21.15.12:0/3242832459 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d30159090 con 0x7f8d300718e0 2024-03-29T11:28:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.148+0000 7f8d34a00700 1 -- 172.21.15.12:0/3242832459 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d30067720 con 0x7f8d300718e0 2024-03-29T11:28:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.149+0000 7f8d26000700 1 -- 172.21.15.12:0/3242832459 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 114) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f8d20007470 con 0x7f8d300718e0 2024-03-29T11:28:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.151+0000 7f8d26000700 1 -- 172.21.15.12:0/3242832459 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f8d20089390 con 0x7f8d300718e0 2024-03-29T11:28:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.154+0000 7f8d26000700 1 -- 172.21.15.12:0/3242832459 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d20054550 con 0x7f8d300718e0 2024-03-29T11:28:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.296+0000 7f8d26000700 1 -- 172.21.15.12:0/3242832459 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mgrmap(e 115) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f8d2001ad90 con 0x7f8d300718e0 2024-03-29T11:28:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.296+0000 7f8d26000700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d10036150 0x7f8d100726d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.297+0000 7f8d2d400700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d10036150 0x7f8d100726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.301+0000 7f8d2d400700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d10036150 0x7f8d100726d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0090b0 tx=0x7f8d1c008030 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:29.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.372+0000 7f8d34a00700 1 -- 172.21.15.12:0/3242832459 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d3015a470 con 0x7f8d300718e0 2024-03-29T11:28:29.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.374+0000 7f8d26000700 1 -- 172.21.15.12:0/3242832459 <== mon.2 v2:172.21.15.159:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+17615 (secure 0 0 0) 0x7f8d20077130 con 0x7f8d300718e0 2024-03-29T11:28:29.364 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:29.364 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":196,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:28:28.284421+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:12:24.038574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-03-29T11:12:25.509384+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":195,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6832/62646":"2024-03-30T11:28:28.284377+0000","172.21.15.159:0/4242656175":"2024-03-30T11:28:28.284377+0000","172.21.15.159:0/230282561":"2024-03-30T11:28:28.284377+0000","172.21.15.159:0/2468643523":"2024-03-30T11:28:28.284377+0000","172.21.15.159:0/2938369341":"2024-03-30T11:28:28.284377+0000","172.21.15.159:0/3347839760":"2024-03-30T11:28:28.284377+0000","172.21.15.159:6833/62646":"2024-03-30T11:28:28.284377+0000","172.21.15.159:0/3631785847":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/750511589":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/2791060580":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/479652442":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/180231134":"2024-03-30T11:12:43.757966+0000","172.21.15.159:6833/43444":"2024-03-30T11:12:43.757966+0000","172.21.15.159:6832/43444":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/2825856150":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3819915206":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/2176927052":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/3123015140":"2024-03-30T11:23:24.314524+0000","172.21.15.159:0/1699207236":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/712920848":"2024-03-30T11:28:28.284377+0000","172.21.15.159:0/3816982205":"2024-03-30T11:12:43.757966+0000","172.21.15.159:0/4147178761":"2024-03-30T11:12:43.757966+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:29.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 -- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d10036150 msgr2=0x7f8d100726d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d10036150 0x7f8d100726d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0090b0 tx=0x7f8d1c008030 comp rx=0 tx=0).stop 2024-03-29T11:28:29.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 -- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d300718e0 msgr2=0x7f8d3014a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d300718e0 0x7f8d3014a6e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000c520 tx=0x7f8d2000c830 comp rx=0 tx=0).stop 2024-03-29T11:28:29.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 -- 172.21.15.12:0/3242832459 shutdown_connections 2024-03-29T11:28:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d10036150 0x7f8d100726d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d30160c90 0x7f8d301f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d300718e0 0x7f8d3014a6e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 --2- 172.21.15.12:0/3242832459 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d300b27f0 0x7f8d3015a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 -- 172.21.15.12:0/3242832459 >> 172.21.15.12:0/3242832459 conn(0x7f8d300cf190 msgr2=0x7f8d300b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 -- 172.21.15.12:0/3242832459 shutdown_connections 2024-03-29T11:28:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.376+0000 7f8d34a00700 1 -- 172.21.15.12:0/3242832459 wait complete. 2024-03-29T11:28:29.385 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T11:28:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.595+0000 7f8289a00700 1 Processor -- start 2024-03-29T11:28:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.595+0000 7f8289a00700 1 -- start start 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.596+0000 7f8289a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f828415dad0 0x7f82840ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.596+0000 7f8289a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f82840d1630 0x7f82840d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.596+0000 7f8289a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828415f820 con 0x7f82840d1fd0 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.596+0000 7f8289a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82840cfcc0 con 0x7f82840d1630 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.596+0000 7f8289a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82840cd9f0 con 0x7f828415dad0 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.596+0000 7f8282000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f82840d1630 0x7f82840d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.596+0000 7f8282000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f82840d1630 0x7f82840d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42710/0 (socket says 172.21.15.12:42710) 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.596+0000 7f8282000700 1 -- 172.21.15.12:0/3378671512 learned_addr learned my addr 172.21.15.12:0/3378671512 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.596+0000 7f8282a00700 1 --2- 172.21.15.12:0/3378671512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f828415dad0 0x7f82840ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.597+0000 7f8282000700 1 -- 172.21.15.12:0/3378671512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f828415dad0 msgr2=0x7f82840ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.597+0000 7f8282000700 1 --2- 172.21.15.12:0/3378671512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f828415dad0 0x7f82840ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.597+0000 7f8282000700 1 -- 172.21.15.12:0/3378671512 >> v1:172.21.15.159:6789/0 conn(0x7f82840d1fd0 legacy=0x7f828415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.597+0000 7f8282000700 1 -- 172.21.15.12:0/3378671512 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82840d2460 con 0x7f82840d1630 2024-03-29T11:28:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.597+0000 7f8282000700 1 --2- 172.21.15.12:0/3378671512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1630 0x7f82840d1a90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f82700098f0 tx=0x7f8270009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68e52f8799bdd5e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.597+0000 7f8281600700 1 -- 172.21.15.12:0/3378671512 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270011830 con 0x7f82840d1630 2024-03-29T11:28:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.597+0000 7f8281600700 1 -- 172.21.15.12:0/3378671512 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8270011990 con 0x7f82840d1630 2024-03-29T11:28:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.597+0000 7f8281600700 1 -- 172.21.15.12:0/3378671512 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270011d20 con 0x7f82840d1630 2024-03-29T11:28:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.598+0000 7f8289a00700 1 -- 172.21.15.12:0/3378671512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1630 msgr2=0x7f82840d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.598+0000 7f8289a00700 1 --2- 172.21.15.12:0/3378671512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1630 0x7f82840d1a90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f82700098f0 tx=0x7f8270009c00 comp rx=0 tx=0).stop 2024-03-29T11:28:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.598+0000 7f8289a00700 1 -- 172.21.15.12:0/3378671512 shutdown_connections 2024-03-29T11:28:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.598+0000 7f8289a00700 1 --2- 172.21.15.12:0/3378671512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f828415dad0 0x7f82840ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.598+0000 7f8289a00700 1 --2- 172.21.15.12:0/3378671512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1630 0x7f82840d1a90 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.598+0000 7f8289a00700 1 -- 172.21.15.12:0/3378671512 >> 172.21.15.12:0/3378671512 conn(0x7f828408aff0 msgr2=0x7f82840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.598+0000 7f8289a00700 1 -- 172.21.15.12:0/3378671512 shutdown_connections 2024-03-29T11:28:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.598+0000 7f8289a00700 1 -- 172.21.15.12:0/3378671512 wait complete. 2024-03-29T11:28:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.600+0000 7f8289a00700 1 Processor -- start 2024-03-29T11:28:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.600+0000 7f8289a00700 1 -- start start 2024-03-29T11:28:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8289a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82840d1630 0x7f82841548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8289a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1fd0 0x7f82840b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8289a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f828415dad0 0x7f828415e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8289a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82840b8240 con 0x7f82840d1fd0 2024-03-29T11:28:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8289a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82840b8470 con 0x7f82840d1630 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8289a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82840b8750 con 0x7f828415dad0 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8282000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1fd0 0x7f82840b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8282000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1fd0 0x7f82840b7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42724/0 (socket says 172.21.15.12:42724) 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8282000700 1 -- 172.21.15.12:0/175793531 learned_addr learned my addr 172.21.15.12:0/175793531 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8282a00700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82840d1630 0x7f82841548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8283400700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f828415dad0 0x7f828415e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8282000700 1 -- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f828415dad0 msgr2=0x7f828415e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8282000700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f828415dad0 0x7f828415e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8282000700 1 -- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82840d1630 msgr2=0x7f82841548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8282000700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82840d1630 0x7f82841548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.602+0000 7f8282000700 1 -- 172.21.15.12:0/175793531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8270009550 con 0x7f82840d1fd0 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.603+0000 7f8282000700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1fd0 0x7f82840b7d00 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f8270000c00 tx=0x7f8270004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.603+0000 7f827aa00700 1 -- 172.21.15.12:0/175793531 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270004400 con 0x7f82840d1fd0 2024-03-29T11:28:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.603+0000 7f827aa00700 1 -- 172.21.15.12:0/175793531 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827001f070 con 0x7f82840d1fd0 2024-03-29T11:28:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.603+0000 7f827aa00700 1 -- 172.21.15.12:0/175793531 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82700038e0 con 0x7f82840d1fd0 2024-03-29T11:28:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.603+0000 7f8289a00700 1 -- 172.21.15.12:0/175793531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82840c71f0 con 0x7f82840d1fd0 2024-03-29T11:28:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.603+0000 7f8289a00700 1 -- 172.21.15.12:0/175793531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828415eb70 con 0x7f82840d1fd0 2024-03-29T11:28:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.604+0000 7f8289a00700 1 -- 172.21.15.12:0/175793531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82840c7050 con 0x7f82840d1fd0 2024-03-29T11:28:29.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.605+0000 7f827aa00700 1 -- 172.21.15.12:0/175793531 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 115) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f8270021030 con 0x7f82840d1fd0 2024-03-29T11:28:29.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.606+0000 7f827aa00700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f826406aae0 0x7f826406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:29.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.606+0000 7f827aa00700 1 -- 172.21.15.12:0/175793531 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f827008a7f0 con 0x7f82840d1fd0 2024-03-29T11:28:29.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.606+0000 7f8282a00700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f826406aae0 0x7f826406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:29.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.607+0000 7f8282a00700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f826406aae0 0x7f826406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f82840d1f40 tx=0x7f8274009210 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.611+0000 7f827aa00700 1 -- 172.21.15.12:0/175793531 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82840c7050 con 0x7f82840d1fd0 2024-03-29T11:28:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.875+0000 7f8289a00700 1 -- 172.21.15.12:0/175793531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f82840c7710 con 0x7f82840d1fd0 2024-03-29T11:28:29.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T11:28:29.876+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T11:28:29.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.884+0000 7f827aa00700 1 -- 172.21.15.12:0/175793531 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v90) v1 ==== 161+0+0 (secure 0 0 0) 0x7f82840c7710 con 0x7f82840d1fd0 2024-03-29T11:28:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.886+0000 7f8289a00700 1 -- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f826406aae0 msgr2=0x7f826406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.886+0000 7f8289a00700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f826406aae0 0x7f826406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f82840d1f40 tx=0x7f8274009210 comp rx=0 tx=0).stop 2024-03-29T11:28:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.886+0000 7f8289a00700 1 -- 172.21.15.12:0/175793531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1fd0 msgr2=0x7f82840b7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.886+0000 7f8289a00700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1fd0 0x7f82840b7d00 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f8270000c00 tx=0x7f8270004750 comp rx=0 tx=0).stop 2024-03-29T11:28:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.886+0000 7f8289a00700 1 -- 172.21.15.12:0/175793531 shutdown_connections 2024-03-29T11:28:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.886+0000 7f8289a00700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f826406aae0 0x7f826406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.886+0000 7f8289a00700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f828415dad0 0x7f828415e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.886+0000 7f8289a00700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82840d1630 0x7f82841548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.886+0000 7f8289a00700 1 --2- 172.21.15.12:0/175793531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82840d1fd0 0x7f82840b7d00 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.886+0000 7f8289a00700 1 -- 172.21.15.12:0/175793531 >> 172.21.15.12:0/175793531 conn(0x7f828408aff0 msgr2=0x7f828406b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.887+0000 7f8289a00700 1 -- 172.21.15.12:0/175793531 shutdown_connections 2024-03-29T11:28:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:29.887+0000 7f8289a00700 1 -- 172.21.15.12:0/175793531 wait complete. 2024-03-29T11:28:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T11:28:29.896 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T11:28:29.942 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T11:28:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.108+0000 7f7ea0e00700 1 Processor -- start 2024-03-29T11:28:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.108+0000 7f7ea0e00700 1 -- start start 2024-03-29T11:28:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.110+0000 7f7ea0e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c0d16a0 0x7f7e9c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.110+0000 7f7ea0e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c161420 con 0x7f7e9c15d9d0 2024-03-29T11:28:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.110+0000 7f7ea0e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c161700 con 0x7f7e9c0d2370 2024-03-29T11:28:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.110+0000 7f7ea0e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c0d2820 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.110+0000 7f7e99e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7e9c15d9d0 0x7f7e9c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36300/0 (socket says 172.21.15.12:36300) 2024-03-29T11:28:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.110+0000 7f7e99e00700 1 -- 172.21.15.12:0/3818568553 learned_addr learned my addr 172.21.15.12:0/3818568553 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.110+0000 7f7e99400700 1 --2- 172.21.15.12:0/3818568553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c0d16a0 0x7f7e9c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.110+0000 7f7e98a00700 1 -- 172.21.15.12:0/3818568553 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647853705 0 0) 0x7f7e9c161420 con 0x7f7e9c15d9d0 2024-03-29T11:28:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.110+0000 7f7e98a00700 1 -- 172.21.15.12:0/3818568553 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e800035b0 con 0x7f7e9c15d9d0 2024-03-29T11:28:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.111+0000 7f7e98a00700 1 -- 172.21.15.12:0/3818568553 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941880247 0 0) 0x7f7e9c161700 con 0x7f7e9c0d2370 2024-03-29T11:28:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.111+0000 7f7e98a00700 1 -- 172.21.15.12:0/3818568553 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e9c161420 con 0x7f7e9c0d2370 2024-03-29T11:28:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.111+0000 7f7e99400700 1 -- 172.21.15.12:0/3818568553 >> v1:172.21.15.159:6789/0 conn(0x7f7e9c0d2370 legacy=0x7f7e9c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.111+0000 7f7e99400700 1 -- 172.21.15.12:0/3818568553 >> v1:172.21.15.12:6789/0 conn(0x7f7e9c15d9d0 legacy=0x7f7e9c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.111+0000 7f7e98a00700 1 -- 172.21.15.12:0/3818568553 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1905529577 0 0) 0x7f7e800035b0 con 0x7f7e9c15d9d0 2024-03-29T11:28:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.111+0000 7f7e99400700 1 -- 172.21.15.12:0/3818568553 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c1619e0 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.111+0000 7f7e99400700 1 --2- 172.21.15.12:0/3818568553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c0d16a0 0x7f7e9c0d1e30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7e88009850 tx=0x7f7e88009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=382902c138448d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.111+0000 7f7e98a00700 1 -- 172.21.15.12:0/3818568553 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c001670 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.111+0000 7f7e98a00700 1 -- 172.21.15.12:0/3818568553 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e88005c30 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.111+0000 7f7e98a00700 1 -- 172.21.15.12:0/3818568553 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e88004030 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.112+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3818568553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c0d16a0 msgr2=0x7f7e9c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.112+0000 7f7ea0e00700 1 --2- 172.21.15.12:0/3818568553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c0d16a0 0x7f7e9c0d1e30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7e88009850 tx=0x7f7e88009b60 comp rx=0 tx=0).stop 2024-03-29T11:28:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.112+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3818568553 shutdown_connections 2024-03-29T11:28:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.112+0000 7f7ea0e00700 1 --2- 172.21.15.12:0/3818568553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c0d16a0 0x7f7e9c0d1e30 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.112+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3818568553 >> 172.21.15.12:0/3818568553 conn(0x7f7e9c08aff0 msgr2=0x7f7e9c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.112+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3818568553 shutdown_connections 2024-03-29T11:28:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.112+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3818568553 wait complete. 2024-03-29T11:28:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.114+0000 7f7ea0e00700 1 Processor -- start 2024-03-29T11:28:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.114+0000 7f7ea0e00700 1 -- start start 2024-03-29T11:28:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7ea0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9c0d16a0 0x7f7e9c0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7ea0e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e9c0d2370 0x7f7e9c1e8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7ea0e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c15d9d0 0x7f7e9c1e8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7ea0e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c1551d0 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7ea0e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c1e8dd0 con 0x7f7e9c0d2370 2024-03-29T11:28:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7ea0e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c1545c0 con 0x7f7e9c15d9d0 2024-03-29T11:28:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7e99e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9c0d16a0 0x7f7e9c0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7e99e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9c0d16a0 0x7f7e9c0ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42750/0 (socket says 172.21.15.12:42750) 2024-03-29T11:28:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7e99e00700 1 -- 172.21.15.12:0/3273086261 learned_addr learned my addr 172.21.15.12:0/3273086261 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7e9a800700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c15d9d0 0x7f7e9c1e8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7e99400700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e9c0d2370 0x7f7e9c1e8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7e99e00700 1 -- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c15d9d0 msgr2=0x7f7e9c1e8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7e99e00700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c15d9d0 0x7f7e9c1e8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7e99e00700 1 -- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e9c0d2370 msgr2=0x7f7e9c1e8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7e99e00700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e9c0d2370 0x7f7e9c1e8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.115+0000 7f7e99e00700 1 -- 172.21.15.12:0/3273086261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e88009430 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.116+0000 7f7e99e00700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9c0d16a0 0x7f7e9c0ccbd0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00c520 tx=0x7f7e8c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.116+0000 7f7e92000700 1 -- 172.21.15.12:0/3273086261 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c004c80 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.116+0000 7f7e92000700 1 -- 172.21.15.12:0/3273086261 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c004de0 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.116+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3273086261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9c1548a0 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.116+0000 7f7e92000700 1 -- 172.21.15.12:0/3273086261 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c007310 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.116+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3273086261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9c154e60 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.117+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3273086261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9c0d1ca0 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.118+0000 7f7e92000700 1 -- 172.21.15.12:0/3273086261 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 115) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f7e8c007470 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.119+0000 7f7e92000700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e7c06aae0 0x7f7e7c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.119+0000 7f7e92000700 1 -- 172.21.15.12:0/3273086261 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f7e8c0890e0 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.120+0000 7f7e99400700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e7c06aae0 0x7f7e7c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.120+0000 7f7e99400700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e7c06aae0 0x7f7e7c06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0d1ae0 tx=0x7f7e88008070 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.123+0000 7f7e92000700 1 -- 172.21.15.12:0/3273086261 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7e8c0540e0 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.392+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3273086261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7e9c067720 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.886+0000 7f7e92000700 1 -- 172.21.15.12:0/3273086261 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v91) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7e8c057700 con 0x7f7e9c0d16a0 2024-03-29T11:28:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.889+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e7c06aae0 msgr2=0x7f7e7c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.889+0000 7f7ea0e00700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e7c06aae0 0x7f7e7c06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0d1ae0 tx=0x7f7e88008070 comp rx=0 tx=0).stop 2024-03-29T11:28:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.889+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3273086261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9c0d16a0 msgr2=0x7f7e9c0ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.889+0000 7f7ea0e00700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9c0d16a0 0x7f7e9c0ccbd0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00c520 tx=0x7f7e8c00c830 comp rx=0 tx=0).stop 2024-03-29T11:28:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.889+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3273086261 shutdown_connections 2024-03-29T11:28:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.889+0000 7f7ea0e00700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e7c06aae0 0x7f7e7c06cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.889+0000 7f7ea0e00700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9c15d9d0 0x7f7e9c1e8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.889+0000 7f7ea0e00700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9c0d16a0 0x7f7e9c0ccbd0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.889+0000 7f7ea0e00700 1 --2- 172.21.15.12:0/3273086261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e9c0d2370 0x7f7e9c1e8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.889+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3273086261 >> 172.21.15.12:0/3273086261 conn(0x7f7e9c08aff0 msgr2=0x7f7e9c0d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.890+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3273086261 shutdown_connections 2024-03-29T11:28:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:30.890+0000 7f7ea0e00700 1 -- 172.21.15.12:0/3273086261 wait complete. 2024-03-29T11:28:30.902 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T11:28:31.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.156+0000 7fa666600700 1 Processor -- start 2024-03-29T11:28:31.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.157+0000 7fa666600700 1 -- start start 2024-03-29T11:28:31.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa666600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6600b56e0 0x7fa6600d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:31.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa666600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa6600d1770 0x7fa66006ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:31.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa666600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6600ceb70 con 0x7fa66015d7f0 2024-03-29T11:28:31.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa666600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66015dbd0 con 0x7fa6600d1770 2024-03-29T11:28:31.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa666600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6600cf000 con 0x7fa6600b56e0 2024-03-29T11:28:31.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa65f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa66015d7f0 0x7fa6600b4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36312/0 (socket says 172.21.15.12:36312) 2024-03-29T11:28:31.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa65f400700 1 -- 172.21.15.12:0/879841507 learned_addr learned my addr 172.21.15.12:0/879841507 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa65fe00700 1 --2- 172.21.15.12:0/879841507 >> v2:172.21.15.159:3300/0 conn(0x7fa6600d1770 0x7fa66006ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa65ea00700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6600b56e0 0x7fa6600d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617471355 0 0) 0x7fa6600ceb70 con 0x7fa66015d7f0 2024-03-29T11:28:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.158+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6440035b0 con 0x7fa66015d7f0 2024-03-29T11:28:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3895527053 0 0) 0x7fa6440035b0 con 0x7fa66015d7f0 2024-03-29T11:28:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6600b56e0 msgr2=0x7fa6600d25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6600b56e0 0x7fa6600d25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 >> v2:172.21.15.159:3300/0 conn(0x7fa6600d1770 msgr2=0x7fa66006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 --2- 172.21.15.12:0/879841507 >> v2:172.21.15.159:3300/0 conn(0x7fa6600d1770 0x7fa66006ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6600cdf60 con 0x7fa66015d7f0 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fa650001670 con 0x7fa66015d7f0 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 >> v1:172.21.15.12:6789/0 conn(0x7fa66015d7f0 legacy=0x7fa6600b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa644003a00 0x7fa644003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa644004850 0x7fa644004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6600b56e0 0x7fa644005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6600cf000 con 0x7fa644004850 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64400ba10 con 0x7fa644003a00 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64400bb50 con 0x7fa6600b56e0 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65fe00700 1 --2- 172.21.15.12:0/879841507 >> v2:172.21.15.159:3300/0 conn(0x7fa6600d1770 0x7fa66006ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65ea00700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa644004850 0x7fa644004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65f400700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa644003a00 0x7fa644003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.159+0000 7fa65fe00700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6600b56e0 0x7fa644005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.160+0000 7fa65ea00700 1 -- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6600b56e0 msgr2=0x7fa644005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.160+0000 7fa65ea00700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6600b56e0 0x7fa644005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.160+0000 7fa65ea00700 1 -- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa644003a00 msgr2=0x7fa644003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.160+0000 7fa65ea00700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa644003a00 0x7fa644003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.160+0000 7fa65ea00700 1 -- 172.21.15.12:0/879841507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6440035b0 con 0x7fa644004850 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.160+0000 7fa65fe00700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6600b56e0 0x7fa644005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.160+0000 7fa65ea00700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa644004850 0x7fa644004cd0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009b60 tx=0x7fa644005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5a39445d62fa09 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.160+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00e660 con 0x7fa644004850 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.160+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c00e7c0 con 0x7fa644004850 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.160+0000 7fa65e000700 1 -- 172.21.15.12:0/879841507 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00eb30 con 0x7fa644004850 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.161+0000 7fa666600700 1 -- 172.21.15.12:0/879841507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa644004850 msgr2=0x7fa644004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.161+0000 7fa666600700 1 --2- 172.21.15.12:0/879841507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa644004850 0x7fa644004cd0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009b60 tx=0x7fa644005880 comp rx=0 tx=0).stop 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.161+0000 7fa65fe00700 1 -- 172.21.15.12:0/879841507 reap_dead start 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.161+0000 7fa666600700 1 -- 172.21.15.12:0/879841507 shutdown_connections 2024-03-29T11:28:31.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.161+0000 7fa666600700 1 -- 172.21.15.12:0/879841507 >> 172.21.15.12:0/879841507 conn(0x7fa66008aff0 msgr2=0x7fa6600d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.161+0000 7fa666600700 1 -- 172.21.15.12:0/879841507 shutdown_connections 2024-03-29T11:28:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.161+0000 7fa666600700 1 -- 172.21.15.12:0/879841507 wait complete. 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.163+0000 7fa666600700 1 Processor -- start 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.163+0000 7fa666600700 1 -- start start 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa666600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa644004850 0x7fa6600b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa666600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6600b7b10 0x7fa6600b6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa666600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6600b65d0 0x7fa660151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa666600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6600b6a70 con 0x7fa6600b7b10 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa666600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6600b6d50 con 0x7fa6600b65d0 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa666600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6600b7030 con 0x7fa644004850 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa65ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6600b7b10 0x7fa6600b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa65ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6600b7b10 0x7fa6600b6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42756/0 (socket says 172.21.15.12:42756) 2024-03-29T11:28:31.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa65ea00700 1 -- 172.21.15.12:0/3911094645 learned_addr learned my addr 172.21.15.12:0/3911094645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa65ea00700 1 -- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa644004850 msgr2=0x7fa6600b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa65fe00700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6600b65d0 0x7fa660151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa65ea00700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa644004850 0x7fa6600b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa65ea00700 1 -- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6600b65d0 msgr2=0x7fa660151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa65ea00700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6600b65d0 0x7fa660151d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa65ea00700 1 -- 172.21.15.12:0/3911094645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6600b7310 con 0x7fa6600b7b10 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.164+0000 7fa65fe00700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6600b65d0 0x7fa660151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.165+0000 7fa65ea00700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6600b7b10 0x7fa6600b6090 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000c00 tx=0x7fa64c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.165+0000 7fa657400700 1 -- 172.21.15.12:0/3911094645 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c022070 con 0x7fa6600b7b10 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.165+0000 7fa657400700 1 -- 172.21.15.12:0/3911094645 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c0036a0 con 0x7fa6600b7b10 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.165+0000 7fa657400700 1 -- 172.21.15.12:0/3911094645 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00e660 con 0x7fa6600b7b10 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.165+0000 7fa666600700 1 -- 172.21.15.12:0/3911094645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6600b75f0 con 0x7fa6600b7b10 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.165+0000 7fa666600700 1 -- 172.21.15.12:0/3911094645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6600b8ba0 con 0x7fa6600b7b10 2024-03-29T11:28:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.166+0000 7fa666600700 1 -- 172.21.15.12:0/3911094645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6600b8d90 con 0x7fa6600b7b10 2024-03-29T11:28:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.167+0000 7fa657400700 1 -- 172.21.15.12:0/3911094645 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 116) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fa64c003a30 con 0x7fa6600b7b10 2024-03-29T11:28:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.168+0000 7fa657400700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa64006ad20 0x7fa64006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.168+0000 7fa657400700 1 -- 172.21.15.12:0/3911094645 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa64c08a7d0 con 0x7fa6600b7b10 2024-03-29T11:28:31.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.168+0000 7fa65f400700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa64006ad20 0x7fa64006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:31.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.169+0000 7fa65f400700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa64006ad20 0x7fa64006d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa6600b3fa0 tx=0x7fa650000f50 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:31.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.172+0000 7fa657400700 1 -- 172.21.15.12:0/3911094645 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa64c0555b0 con 0x7fa6600b7b10 2024-03-29T11:28:31.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:31.387+0000 7fa666600700 1 -- 172.21.15.12:0/3911094645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa660067720 con 0x7fa6600b7b10 2024-03-29T11:28:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.299+0000 7fa657400700 1 -- 172.21.15.12:0/3911094645 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v198) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa64c058bd0 con 0x7fa6600b7b10 2024-03-29T11:28:32.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 -- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa64006ad20 msgr2=0x7fa64006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:32.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa64006ad20 0x7fa64006d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa6600b3fa0 tx=0x7fa650000f50 comp rx=0 tx=0).stop 2024-03-29T11:28:32.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 -- 172.21.15.12:0/3911094645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6600b7b10 msgr2=0x7fa6600b6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:32.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6600b7b10 0x7fa6600b6090 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000c00 tx=0x7fa64c0047c0 comp rx=0 tx=0).stop 2024-03-29T11:28:32.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 -- 172.21.15.12:0/3911094645 shutdown_connections 2024-03-29T11:28:32.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa64006ad20 0x7fa64006d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:32.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6600b65d0 0x7fa660151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:32.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa644004850 0x7fa6600b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:32.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 --2- 172.21.15.12:0/3911094645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6600b7b10 0x7fa6600b6090 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:32.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 -- 172.21.15.12:0/3911094645 >> 172.21.15.12:0/3911094645 conn(0x7fa66008aff0 msgr2=0x7fa6600b48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:32.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 -- 172.21.15.12:0/3911094645 shutdown_connections 2024-03-29T11:28:32.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.303+0000 7fa666600700 1 -- 172.21.15.12:0/3911094645 wait complete. 2024-03-29T11:28:32.294 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T11:28:32.314 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T11:28:32.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.532+0000 7f1e07c00700 1 Processor -- start 2024-03-29T11:28:32.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.532+0000 7f1e07c00700 1 -- start start 2024-03-29T11:28:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.534+0000 7f1e07c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1e001580c0 0x7f1e0006e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.534+0000 7f1e07c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e0006e7f0 0x7f1e0015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.534+0000 7f1e07c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0006ec50 con 0x7f1e0015b160 2024-03-29T11:28:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.534+0000 7f1e07c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e00071570 con 0x7f1e001580c0 2024-03-29T11:28:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.534+0000 7f1e07c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e00071990 con 0x7f1e0006e7f0 2024-03-29T11:28:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.534+0000 7f1e05c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1e0015b160 0x7f1e0015df60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36324/0 (socket says 172.21.15.12:36324) 2024-03-29T11:28:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.534+0000 7f1e05c00700 1 -- 172.21.15.12:0/2454266542 learned_addr learned my addr 172.21.15.12:0/2454266542 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.534+0000 7f1e05200700 1 --2- 172.21.15.12:0/2454266542 >> v2:172.21.15.159:3300/0 conn(0x7f1e001580c0 0x7f1e0006e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.534+0000 7f1dffe00700 1 --2- 172.21.15.12:0/2454266542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e0006e7f0 0x7f1e0015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.534+0000 7f1dff400700 1 -- 172.21.15.12:0/2454266542 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999292147 0 0) 0x7f1e0006ec50 con 0x7f1e0015b160 2024-03-29T11:28:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1dff400700 1 -- 172.21.15.12:0/2454266542 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1de80035b0 con 0x7f1e0015b160 2024-03-29T11:28:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1e05200700 1 -- 172.21.15.12:0/2454266542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e0006e7f0 msgr2=0x7f1e0015ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1e05200700 1 --2- 172.21.15.12:0/2454266542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e0006e7f0 0x7f1e0015ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1dff400700 1 -- 172.21.15.12:0/2454266542 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 57836938 0 0) 0x7f1de80035b0 con 0x7f1e0015b160 2024-03-29T11:28:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1e05200700 1 -- 172.21.15.12:0/2454266542 >> v1:172.21.15.12:6789/0 conn(0x7f1e0015b160 legacy=0x7f1e0015df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1e05200700 1 -- 172.21.15.12:0/2454266542 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e00070980 con 0x7f1e001580c0 2024-03-29T11:28:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1dffe00700 1 --2- 172.21.15.12:0/2454266542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e0006e7f0 0x7f1e0015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1e05200700 1 --2- 172.21.15.12:0/2454266542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e001580c0 0x7f1e0006e2b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009870 tx=0x7f1df4009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae4c79a2045bcece server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1dff400700 1 -- 172.21.15.12:0/2454266542 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df80019f0 con 0x7f1e001580c0 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1dff400700 1 -- 172.21.15.12:0/2454266542 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df4004030 con 0x7f1e001580c0 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.535+0000 7f1dff400700 1 -- 172.21.15.12:0/2454266542 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df40043c0 con 0x7f1e001580c0 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.536+0000 7f1e07c00700 1 -- 172.21.15.12:0/2454266542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e001580c0 msgr2=0x7f1e0006e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.536+0000 7f1e07c00700 1 --2- 172.21.15.12:0/2454266542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e001580c0 0x7f1e0006e2b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009870 tx=0x7f1df4009b80 comp rx=0 tx=0).stop 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.536+0000 7f1e07c00700 1 -- 172.21.15.12:0/2454266542 shutdown_connections 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.536+0000 7f1e07c00700 1 --2- 172.21.15.12:0/2454266542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e001580c0 0x7f1e0006e2b0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.536+0000 7f1e07c00700 1 --2- 172.21.15.12:0/2454266542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e0006e7f0 0x7f1e0015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.536+0000 7f1e07c00700 1 -- 172.21.15.12:0/2454266542 >> 172.21.15.12:0/2454266542 conn(0x7f1e000cf210 msgr2=0x7f1e000cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.536+0000 7f1e07c00700 1 -- 172.21.15.12:0/2454266542 shutdown_connections 2024-03-29T11:28:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.536+0000 7f1e07c00700 1 -- 172.21.15.12:0/2454266542 wait complete. 2024-03-29T11:28:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.538+0000 7f1e07c00700 1 Processor -- start 2024-03-29T11:28:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.538+0000 7f1e07c00700 1 -- start start 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1e07c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e0006e7f0 0x7f1e0015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1e07c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e001580c0 0x7f1e00150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1e07c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0015b160 0x7f1e00151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1e07c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e001526f0 con 0x7f1e0015b160 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1e07c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e00153010 con 0x7f1e0006e7f0 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1e07c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e00151ae0 con 0x7f1e001580c0 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1e05c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0015b160 0x7f1e00151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1e05c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0015b160 0x7f1e00151440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42768/0 (socket says 172.21.15.12:42768) 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1e05c00700 1 -- 172.21.15.12:0/1999656884 learned_addr learned my addr 172.21.15.12:0/1999656884 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1e05200700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e0006e7f0 0x7f1e0015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.539+0000 7f1dffe00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e001580c0 0x7f1e00150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1e05c00700 1 -- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e001580c0 msgr2=0x7f1e00150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1e05c00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e001580c0 0x7f1e00150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1e05c00700 1 -- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e0006e7f0 msgr2=0x7f1e0015bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1e05c00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e0006e7f0 0x7f1e0015bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1e05c00700 1 -- 172.21.15.12:0/1999656884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec0094a0 con 0x7f1e0015b160 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1e05200700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e0006e7f0 0x7f1e0015bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1e05c00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0015b160 0x7f1e00151440 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f1df800cbe0 tx=0x7f1df800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1dfcc00700 1 -- 172.21.15.12:0/1999656884 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df800aaa0 con 0x7f1e0015b160 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1dfcc00700 1 -- 172.21.15.12:0/1999656884 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df800ac00 con 0x7f1e0015b160 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1e07c00700 1 -- 172.21.15.12:0/1999656884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df40094a0 con 0x7f1e0015b160 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1dfcc00700 1 -- 172.21.15.12:0/1999656884 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df80070d0 con 0x7f1e0015b160 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.540+0000 7f1e07c00700 1 -- 172.21.15.12:0/1999656884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e001520a0 con 0x7f1e0015b160 2024-03-29T11:28:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.541+0000 7f1e07c00700 1 -- 172.21.15.12:0/1999656884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e000b4710 con 0x7f1e0015b160 2024-03-29T11:28:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.542+0000 7f1dfcc00700 1 -- 172.21.15.12:0/1999656884 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1df8007230 con 0x7f1e0015b160 2024-03-29T11:28:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.543+0000 7f1dfcc00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1de006adb0 0x7f1de006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.543+0000 7f1dfcc00700 1 -- 172.21.15.12:0/1999656884 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f1df8089630 con 0x7f1e0015b160 2024-03-29T11:28:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.544+0000 7f1e05200700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1de006adb0 0x7f1de006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.544+0000 7f1e05200700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1de006adb0 0x7f1de006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1df40097c0 tx=0x7f1df401c040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.548+0000 7f1dfcc00700 1 -- 172.21.15.12:0/1999656884 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1df8054330 con 0x7f1e0015b160 2024-03-29T11:28:32.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:32.760+0000 7f1e07c00700 1 -- 172.21.15.12:0/1999656884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1e00067720 con 0x7f1e0015b160 2024-03-29T11:28:33.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.514+0000 7f1dfcc00700 1 -- 172.21.15.12:0/1999656884 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v199) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1df80180d0 con 0x7f1e0015b160 2024-03-29T11:28:33.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.519+0000 7f1e07c00700 1 -- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1de006adb0 msgr2=0x7f1de006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.519+0000 7f1e07c00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1de006adb0 0x7f1de006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1df40097c0 tx=0x7f1df401c040 comp rx=0 tx=0).stop 2024-03-29T11:28:33.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.519+0000 7f1e07c00700 1 -- 172.21.15.12:0/1999656884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0015b160 msgr2=0x7f1e00151440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.519+0000 7f1e07c00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0015b160 0x7f1e00151440 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f1df800cbe0 tx=0x7f1df800cef0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.519+0000 7f1e07c00700 1 -- 172.21.15.12:0/1999656884 shutdown_connections 2024-03-29T11:28:33.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.519+0000 7f1e07c00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1de006adb0 0x7f1de006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.519+0000 7f1e07c00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0015b160 0x7f1e00151440 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.519+0000 7f1e07c00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e0006e7f0 0x7f1e0015bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.519+0000 7f1e07c00700 1 --2- 172.21.15.12:0/1999656884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e001580c0 0x7f1e00150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.519+0000 7f1e07c00700 1 -- 172.21.15.12:0/1999656884 >> 172.21.15.12:0/1999656884 conn(0x7f1e000cf210 msgr2=0x7f1e000b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:33.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.520+0000 7f1e07c00700 1 -- 172.21.15.12:0/1999656884 shutdown_connections 2024-03-29T11:28:33.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.520+0000 7f1e07c00700 1 -- 172.21.15.12:0/1999656884 wait complete. 2024-03-29T11:28:33.512 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T11:28:33.534 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T11:28:33.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.747+0000 7f3d45600700 1 Processor -- start 2024-03-29T11:28:33.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.748+0000 7f3d45600700 1 -- start start 2024-03-29T11:28:33.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d45600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3d4015dfe0 0x7f3d400ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:33.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d45600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3d400ce670 0x7f3d400cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:33.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d45600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d400cb310 con 0x7f3d4015a5c0 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d45600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d400ceb00 con 0x7f3d4015dfe0 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d45600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d400711a0 con 0x7f3d400ce670 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d3e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3d4015dfe0 0x7f3d400ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d3e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3d4015dfe0 0x7f3d400ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42774/0 (socket says 172.21.15.12:42774) 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d3e000700 1 -- 172.21.15.12:0/342099427 learned_addr learned my addr 172.21.15.12:0/342099427 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d3f400700 1 --2- 172.21.15.12:0/342099427 >> v2:172.21.15.159:3300/0 conn(0x7f3d400ce670 0x7f3d400cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d3e000700 1 -- 172.21.15.12:0/342099427 >> v2:172.21.15.159:3300/0 conn(0x7f3d400ce670 msgr2=0x7f3d400cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.749+0000 7f3d3e000700 1 --2- 172.21.15.12:0/342099427 >> v2:172.21.15.159:3300/0 conn(0x7f3d400ce670 0x7f3d400cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.750+0000 7f3d3e000700 1 -- 172.21.15.12:0/342099427 >> v1:172.21.15.159:6789/0 conn(0x7f3d4015a5c0 legacy=0x7f3d400709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.750+0000 7f3d3e000700 1 -- 172.21.15.12:0/342099427 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d40071480 con 0x7f3d4015dfe0 2024-03-29T11:28:33.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.750+0000 7f3d3e000700 1 --2- 172.21.15.12:0/342099427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d4015dfe0 0x7f3d400ce130 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f3d40070950 tx=0x7f3d2c00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c13cd8a2022d46 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:33.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.750+0000 7f3d3d600700 1 -- 172.21.15.12:0/342099427 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00dcf0 con 0x7f3d4015dfe0 2024-03-29T11:28:33.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.750+0000 7f3d3d600700 1 -- 172.21.15.12:0/342099427 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c00de50 con 0x7f3d4015dfe0 2024-03-29T11:28:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.750+0000 7f3d3d600700 1 -- 172.21.15.12:0/342099427 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c004250 con 0x7f3d4015dfe0 2024-03-29T11:28:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.751+0000 7f3d45600700 1 -- 172.21.15.12:0/342099427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d4015dfe0 msgr2=0x7f3d400ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.751+0000 7f3d45600700 1 --2- 172.21.15.12:0/342099427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d4015dfe0 0x7f3d400ce130 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f3d40070950 tx=0x7f3d2c00be70 comp rx=0 tx=0).stop 2024-03-29T11:28:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.751+0000 7f3d45600700 1 -- 172.21.15.12:0/342099427 shutdown_connections 2024-03-29T11:28:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.751+0000 7f3d45600700 1 --2- 172.21.15.12:0/342099427 >> v2:172.21.15.159:3300/0 conn(0x7f3d400ce670 0x7f3d400cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.751+0000 7f3d45600700 1 --2- 172.21.15.12:0/342099427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d4015dfe0 0x7f3d400ce130 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.751+0000 7f3d45600700 1 -- 172.21.15.12:0/342099427 >> 172.21.15.12:0/342099427 conn(0x7f3d400b0d70 msgr2=0x7f3d400b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.751+0000 7f3d45600700 1 -- 172.21.15.12:0/342099427 shutdown_connections 2024-03-29T11:28:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.751+0000 7f3d45600700 1 -- 172.21.15.12:0/342099427 wait complete. 2024-03-29T11:28:33.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.753+0000 7f3d45600700 1 Processor -- start 2024-03-29T11:28:33.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.753+0000 7f3d45600700 1 -- start start 2024-03-29T11:28:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d45600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d400ce670 0x7f3d40147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d45600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d4015a5c0 0x7f3d400c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d45600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d4015dfe0 0x7f3d401585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d45600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d400c9e10 con 0x7f3d400ce670 2024-03-29T11:28:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d45600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d40154e50 con 0x7f3d4015dfe0 2024-03-29T11:28:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d45600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d400c9850 con 0x7f3d4015a5c0 2024-03-29T11:28:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d3ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d400ce670 0x7f3d40147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d3ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d400ce670 0x7f3d40147860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42784/0 (socket says 172.21.15.12:42784) 2024-03-29T11:28:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d3ea00700 1 -- 172.21.15.12:0/2014773598 learned_addr learned my addr 172.21.15.12:0/2014773598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d3f400700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d4015dfe0 0x7f3d401585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.754+0000 7f3d3e000700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d4015a5c0 0x7f3d400c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d3ea00700 1 -- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d4015a5c0 msgr2=0x7f3d400c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d3ea00700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d4015a5c0 0x7f3d400c8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d3ea00700 1 -- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d4015dfe0 msgr2=0x7f3d401585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d3ea00700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d4015dfe0 0x7f3d401585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d3ea00700 1 -- 172.21.15.12:0/2014773598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c00b7f0 con 0x7f3d400ce670 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d3ea00700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d400ce670 0x7f3d40147860 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000c4d0 tx=0x7f3d3000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d36a00700 1 -- 172.21.15.12:0/2014773598 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30011790 con 0x7f3d400ce670 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d36a00700 1 -- 172.21.15.12:0/2014773598 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d300118f0 con 0x7f3d400ce670 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d36a00700 1 -- 172.21.15.12:0/2014773598 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30011ca0 con 0x7f3d400ce670 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d45600700 1 -- 172.21.15.12:0/2014773598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d400c9b30 con 0x7f3d400ce670 2024-03-29T11:28:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.755+0000 7f3d45600700 1 -- 172.21.15.12:0/2014773598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d40147da0 con 0x7f3d400ce670 2024-03-29T11:28:33.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.756+0000 7f3d45600700 1 -- 172.21.15.12:0/2014773598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d40154940 con 0x7f3d400ce670 2024-03-29T11:28:33.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.757+0000 7f3d36a00700 1 -- 172.21.15.12:0/2014773598 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3d3001c460 con 0x7f3d400ce670 2024-03-29T11:28:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.758+0000 7f3d36a00700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3d2006adb0 0x7f3d2006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.758+0000 7f3d36a00700 1 -- 172.21.15.12:0/2014773598 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5961+0+0 (secure 0 0 0) 0x7f3d30089290 con 0x7f3d400ce670 2024-03-29T11:28:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.759+0000 7f3d3e000700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3d2006adb0 0x7f3d2006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.760+0000 7f3d3e000700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3d2006adb0 0x7f3d2006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00b340 tx=0x7f3d2c01c040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.763+0000 7f3d36a00700 1 -- 172.21.15.12:0/2014773598 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3d40154940 con 0x7f3d400ce670 2024-03-29T11:28:33.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.982+0000 7f3d45600700 1 -- 172.21.15.12:0/2014773598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f3d40154940 con 0x7f3d400ce670 2024-03-29T11:28:33.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.986+0000 7f3d36a00700 1 -- 172.21.15.12:0/2014773598 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v67) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3d40154940 con 0x7f3d400ce670 2024-03-29T11:28:33.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 -- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3d2006adb0 msgr2=0x7f3d2006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3d2006adb0 0x7f3d2006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00b340 tx=0x7f3d2c01c040 comp rx=0 tx=0).stop 2024-03-29T11:28:33.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 -- 172.21.15.12:0/2014773598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d400ce670 msgr2=0x7f3d40147860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d400ce670 0x7f3d40147860 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000c4d0 tx=0x7f3d3000c7e0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 -- 172.21.15.12:0/2014773598 shutdown_connections 2024-03-29T11:28:33.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3d2006adb0 0x7f3d2006d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d4015dfe0 0x7f3d401585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d400ce670 0x7f3d40147860 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 --2- 172.21.15.12:0/2014773598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d4015a5c0 0x7f3d400c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:33.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 -- 172.21.15.12:0/2014773598 >> 172.21.15.12:0/2014773598 conn(0x7f3d400b0d70 msgr2=0x7f3d400cf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:33.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 -- 172.21.15.12:0/2014773598 shutdown_connections 2024-03-29T11:28:33.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:33.988+0000 7f3d45600700 1 -- 172.21.15.12:0/2014773598 wait complete. 2024-03-29T11:28:33.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T11:28:34.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.206+0000 7f5543000700 1 Processor -- start 2024-03-29T11:28:34.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.206+0000 7f5543000700 1 -- start start 2024-03-29T11:28:34.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.207+0000 7f5543000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c0b3f80 0x7f553c0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.207+0000 7f5543000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c161230 con 0x7f553c0d3f20 2024-03-29T11:28:34.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.207+0000 7f5543000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c0d2a80 con 0x7f553c160e50 2024-03-29T11:28:34.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.207+0000 7f5543000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0d43b0 con 0x7f553c0b3f80 2024-03-29T11:28:34.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.207+0000 7f553b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f553c0d3f20 0x7f553c0cdef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36332/0 (socket says 172.21.15.12:36332) 2024-03-29T11:28:34.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.207+0000 7f553b400700 1 -- 172.21.15.12:0/3237378152 learned_addr learned my addr 172.21.15.12:0/3237378152 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:34.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.207+0000 7f5541000700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c0b3f80 0x7f553c0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669287287 0 0) 0x7f553c161230 con 0x7f553c0d3f20 2024-03-29T11:28:34.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55200035b0 con 0x7f553c0d3f20 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128412014 0 0) 0x7f553c0d2a80 con 0x7f553c160e50 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f553c161230 con 0x7f553c160e50 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3334902273 0 0) 0x7f55200035b0 con 0x7f553c0d3f20 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c0b3f80 msgr2=0x7f553c0ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c0b3f80 0x7f553c0ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 >> v1:172.21.15.159:6789/0 conn(0x7f553c160e50 legacy=0x7f553c0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c0d21e0 con 0x7f553c0d3f20 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f5528001670 con 0x7f553c0d3f20 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 >> v1:172.21.15.12:6789/0 conn(0x7f553c0d3f20 legacy=0x7f553c0cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c0b3f80 0x7f5520003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5520004170 0x7f5520004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 0x7f5520005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0d43b0 con 0x7f5520004d90 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552000bbd0 con 0x7f5520004170 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552000bd90 con 0x7f553c0b3f80 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.208+0000 7f5541000700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 0x7f5520005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f553be00700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c0b3f80 0x7f5520003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f553b400700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5520004170 0x7f5520004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f5541000700 1 -- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c0b3f80 msgr2=0x7f5520003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f5541000700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c0b3f80 0x7f5520003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f5541000700 1 -- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5520004170 msgr2=0x7f5520004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f5541000700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5520004170 0x7f5520004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f5541000700 1 -- 172.21.15.12:0/3237378152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55200035b0 con 0x7f5520004d90 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f553b400700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5520004170 0x7f5520004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f5541000700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 0x7f5520005210 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f5534009f90 tx=0x7f5520005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50278eb2e976318f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553400ec30 con 0x7f5520004d90 2024-03-29T11:28:34.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553400f040 con 0x7f5520004d90 2024-03-29T11:28:34.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.209+0000 7f553aa00700 1 -- 172.21.15.12:0/3237378152 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5534014480 con 0x7f5520004d90 2024-03-29T11:28:34.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.210+0000 7f5543000700 1 -- 172.21.15.12:0/3237378152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 msgr2=0x7f5520005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.210+0000 7f5543000700 1 --2- 172.21.15.12:0/3237378152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 0x7f5520005210 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f5534009f90 tx=0x7f5520005880 comp rx=0 tx=0).stop 2024-03-29T11:28:34.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.210+0000 7f5541000700 1 -- 172.21.15.12:0/3237378152 reap_dead start 2024-03-29T11:28:34.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.210+0000 7f5543000700 1 -- 172.21.15.12:0/3237378152 shutdown_connections 2024-03-29T11:28:34.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.210+0000 7f5543000700 1 -- 172.21.15.12:0/3237378152 >> 172.21.15.12:0/3237378152 conn(0x7f553c08aff0 msgr2=0x7f553c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:34.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.210+0000 7f5543000700 1 -- 172.21.15.12:0/3237378152 shutdown_connections 2024-03-29T11:28:34.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.210+0000 7f5543000700 1 -- 172.21.15.12:0/3237378152 wait complete. 2024-03-29T11:28:34.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.212+0000 7f5543000700 1 Processor -- start 2024-03-29T11:28:34.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.212+0000 7f5543000700 1 -- start start 2024-03-29T11:28:34.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f5543000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 0x7f553c1f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f5543000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f553c0b6230 0x7f553c0b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f5543000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c15e970 0x7f553c15ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f5543000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c15f340 con 0x7f5520004d90 2024-03-29T11:28:34.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f5543000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0b83a0 con 0x7f553c0b6230 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f5543000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0b8680 con 0x7f553c15e970 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f553be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 0x7f553c1f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f553be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 0x7f553c1f62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42808/0 (socket says 172.21.15.12:42808) 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f553be00700 1 -- 172.21.15.12:0/2261446390 learned_addr learned my addr 172.21.15.12:0/2261446390 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f5541000700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c15e970 0x7f553c15ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f553b400700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f553c0b6230 0x7f553c0b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f553be00700 1 -- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c15e970 msgr2=0x7f553c15ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f553be00700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c15e970 0x7f553c15ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f553be00700 1 -- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f553c0b6230 msgr2=0x7f553c0b7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f553be00700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f553c0b6230 0x7f553c0b7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f553be00700 1 -- 172.21.15.12:0/2261446390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c0b8960 con 0x7f5520004d90 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f553b400700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f553c0b6230 0x7f553c0b7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.213+0000 7f5541000700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c15e970 0x7f553c15ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.214+0000 7f553be00700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 0x7f553c1f62f0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f552c00e7b0 tx=0x7f552c00eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:34.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.214+0000 7f5533e00700 1 -- 172.21.15.12:0/2261446390 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c0052e0 con 0x7f5520004d90 2024-03-29T11:28:34.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.214+0000 7f5533e00700 1 -- 172.21.15.12:0/2261446390 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c005440 con 0x7f5520004d90 2024-03-29T11:28:34.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.214+0000 7f5533e00700 1 -- 172.21.15.12:0/2261446390 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c006aa0 con 0x7f5520004d90 2024-03-29T11:28:34.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.214+0000 7f5543000700 1 -- 172.21.15.12:0/2261446390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553c0b8c40 con 0x7f5520004d90 2024-03-29T11:28:34.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.214+0000 7f5543000700 1 -- 172.21.15.12:0/2261446390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553c1f6830 con 0x7f5520004d90 2024-03-29T11:28:34.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.215+0000 7f5543000700 1 -- 172.21.15.12:0/2261446390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553c0b9160 con 0x7f5520004d90 2024-03-29T11:28:34.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.216+0000 7f5533e00700 1 -- 172.21.15.12:0/2261446390 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f552c006c00 con 0x7f5520004d90 2024-03-29T11:28:34.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.217+0000 7f5533e00700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f551c06ade0 0x7f551c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.217+0000 7f5533e00700 1 -- 172.21.15.12:0/2261446390 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5961+0+0 (secure 0 0 0) 0x7f552c089130 con 0x7f5520004d90 2024-03-29T11:28:34.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.218+0000 7f553b400700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f551c06ade0 0x7f551c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.218+0000 7f553b400700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f551c06ade0 0x7f551c06d2a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5528001200 tx=0x7f5528009040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:34.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.222+0000 7f5533e00700 1 -- 172.21.15.12:0/2261446390 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f553c0b9160 con 0x7f5520004d90 2024-03-29T11:28:34.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.437+0000 7f5543000700 1 -- 172.21.15.12:0/2261446390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f553c0b9160 con 0x7f5520004d90 2024-03-29T11:28:34.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.441+0000 7f5533e00700 1 -- 172.21.15.12:0/2261446390 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f553c0b9160 con 0x7f5520004d90 2024-03-29T11:28:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 -- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f551c06ade0 msgr2=0x7f551c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f551c06ade0 0x7f551c06d2a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5528001200 tx=0x7f5528009040 comp rx=0 tx=0).stop 2024-03-29T11:28:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 -- 172.21.15.12:0/2261446390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 msgr2=0x7f553c1f62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 0x7f553c1f62f0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f552c00e7b0 tx=0x7f552c00eb70 comp rx=0 tx=0).stop 2024-03-29T11:28:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 -- 172.21.15.12:0/2261446390 shutdown_connections 2024-03-29T11:28:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f551c06ade0 0x7f551c06d2a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f553c15e970 0x7f553c15ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5520004d90 0x7f553c1f62f0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 --2- 172.21.15.12:0/2261446390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f553c0b6230 0x7f553c0b7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 -- 172.21.15.12:0/2261446390 >> 172.21.15.12:0/2261446390 conn(0x7f553c08aff0 msgr2=0x7f553c0d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:34.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 -- 172.21.15.12:0/2261446390 shutdown_connections 2024-03-29T11:28:34.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.444+0000 7f5543000700 1 -- 172.21.15.12:0/2261446390 wait complete. 2024-03-29T11:28:34.452 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T11:28:34.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.666+0000 7fe26ee00700 1 Processor -- start 2024-03-29T11:28:34.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.666+0000 7fe26ee00700 1 -- start start 2024-03-29T11:28:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe26ee00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2680ced80 0x7fe2680d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe26ee00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26815f820 con 0x7fe26815fcb0 2024-03-29T11:28:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe26ee00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2680cf420 con 0x7fe26815dad0 2024-03-29T11:28:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe26ee00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2680cfcc0 con 0x7fe2680ced80 2024-03-29T11:28:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe267400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2680ced80 0x7fe2680d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe26ce00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe26815fcb0 0x7fe2681600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36336/0 (socket says 172.21.15.12:36336) 2024-03-29T11:28:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe26ce00700 1 -- 172.21.15.12:0/2319528182 learned_addr learned my addr 172.21.15.12:0/2319528182 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe267400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2680ced80 0x7fe2680d1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42812/0 (socket says 172.21.15.12:42812) 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe267400700 1 -- 172.21.15.12:0/2319528182 >> v1:172.21.15.159:6789/0 conn(0x7fe26815dad0 legacy=0x7fe2680d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe267400700 1 -- 172.21.15.12:0/2319528182 >> v1:172.21.15.12:6789/0 conn(0x7fe26815fcb0 legacy=0x7fe2681600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe267400700 1 -- 172.21.15.12:0/2319528182 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2680cf700 con 0x7fe2680ced80 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.668+0000 7fe266a00700 1 -- 172.21.15.12:0/2319528182 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429234703 0 0) 0x7fe26815f820 con 0x7fe26815fcb0 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.669+0000 7fe267400700 1 --2- 172.21.15.12:0/2319528182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2680ced80 0x7fe2680d1d40 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fe254009840 tx=0x7fe254009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84a9e97d4bd1eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.669+0000 7fe266a00700 1 -- 172.21.15.12:0/2319528182 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe254011830 con 0x7fe2680ced80 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.669+0000 7fe266a00700 1 -- 172.21.15.12:0/2319528182 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe254011990 con 0x7fe2680ced80 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.669+0000 7fe266a00700 1 -- 172.21.15.12:0/2319528182 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe254011d20 con 0x7fe2680ced80 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.669+0000 7fe26ee00700 1 -- 172.21.15.12:0/2319528182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2680ced80 msgr2=0x7fe2680d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.669+0000 7fe26ee00700 1 --2- 172.21.15.12:0/2319528182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2680ced80 0x7fe2680d1d40 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fe254009840 tx=0x7fe254009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.669+0000 7fe26ee00700 1 -- 172.21.15.12:0/2319528182 shutdown_connections 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.669+0000 7fe26ee00700 1 --2- 172.21.15.12:0/2319528182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2680ced80 0x7fe2680d1d40 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.669+0000 7fe26ee00700 1 -- 172.21.15.12:0/2319528182 >> 172.21.15.12:0/2319528182 conn(0x7fe26808aff0 msgr2=0x7fe2680d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.670+0000 7fe26ee00700 1 -- 172.21.15.12:0/2319528182 shutdown_connections 2024-03-29T11:28:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.670+0000 7fe26ee00700 1 -- 172.21.15.12:0/2319528182 wait complete. 2024-03-29T11:28:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.671+0000 7fe26ee00700 1 Processor -- start 2024-03-29T11:28:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.671+0000 7fe26ee00700 1 -- start start 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.672+0000 7fe26ee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2680ced80 0x7fe268155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.672+0000 7fe26ee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26815dad0 0x7fe2680cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.672+0000 7fe26ee00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe26815fcb0 0x7fe2680ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.672+0000 7fe26ee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe268159400 con 0x7fe26815dad0 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.672+0000 7fe26ee00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680b6d80 con 0x7fe26815fcb0 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.672+0000 7fe26ee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680b7060 con 0x7fe2680ced80 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.673+0000 7fe267400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26815dad0 0x7fe2680cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.673+0000 7fe267400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26815dad0 0x7fe2680cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42814/0 (socket says 172.21.15.12:42814) 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.673+0000 7fe267400700 1 -- 172.21.15.12:0/2075521128 learned_addr learned my addr 172.21.15.12:0/2075521128 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.673+0000 7fe267400700 1 -- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2680ced80 msgr2=0x7fe268155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.673+0000 7fe267400700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2680ced80 0x7fe268155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.673+0000 7fe267400700 1 -- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe26815fcb0 msgr2=0x7fe2680ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.673+0000 7fe267400700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe26815fcb0 0x7fe2680ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.673+0000 7fe267400700 1 -- 172.21.15.12:0/2075521128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2540094a0 con 0x7fe26815dad0 2024-03-29T11:28:34.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.673+0000 7fe267400700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26815dad0 0x7fe2680cc600 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fe2540090b0 tx=0x7fe254004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.673+0000 7fe25fe00700 1 -- 172.21.15.12:0/2075521128 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25401a900 con 0x7fe26815dad0 2024-03-29T11:28:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.674+0000 7fe25fe00700 1 -- 172.21.15.12:0/2075521128 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25401aa60 con 0x7fe26815dad0 2024-03-29T11:28:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.674+0000 7fe26ee00700 1 -- 172.21.15.12:0/2075521128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2680b7340 con 0x7fe26815dad0 2024-03-29T11:28:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.674+0000 7fe25fe00700 1 -- 172.21.15.12:0/2075521128 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25401add0 con 0x7fe26815dad0 2024-03-29T11:28:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.674+0000 7fe26ee00700 1 -- 172.21.15.12:0/2075521128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe268156700 con 0x7fe26815dad0 2024-03-29T11:28:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.675+0000 7fe26ee00700 1 -- 172.21.15.12:0/2075521128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe268067720 con 0x7fe26815dad0 2024-03-29T11:28:34.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.676+0000 7fe25fe00700 1 -- 172.21.15.12:0/2075521128 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe254022070 con 0x7fe26815dad0 2024-03-29T11:28:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.676+0000 7fe25fe00700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24806adb0 0x7fe24806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.676+0000 7fe25fe00700 1 -- 172.21.15.12:0/2075521128 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5961+0+0 (secure 0 0 0) 0x7fe25408a7e0 con 0x7fe26815dad0 2024-03-29T11:28:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.677+0000 7fe267e00700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24806adb0 0x7fe24806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:34.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.677+0000 7fe267e00700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24806adb0 0x7fe24806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe258000f60 tx=0x7fe258007a40 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:34.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.681+0000 7fe25fe00700 1 -- 172.21.15.12:0/2075521128 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe254055850 con 0x7fe26815dad0 2024-03-29T11:28:34.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.903+0000 7fe26ee00700 1 -- 172.21.15.12:0/2075521128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe26806a840 con 0x7fe26815dad0 2024-03-29T11:28:34.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.908+0000 7fe25fe00700 1 -- 172.21.15.12:0/2075521128 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v69) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe254006cc0 con 0x7fe26815dad0 2024-03-29T11:28:34.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.910+0000 7fe26ee00700 1 -- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24806adb0 msgr2=0x7fe24806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.910+0000 7fe26ee00700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24806adb0 0x7fe24806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe258000f60 tx=0x7fe258007a40 comp rx=0 tx=0).stop 2024-03-29T11:28:34.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.910+0000 7fe26ee00700 1 -- 172.21.15.12:0/2075521128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26815dad0 msgr2=0x7fe2680cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.910+0000 7fe26ee00700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26815dad0 0x7fe2680cc600 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fe2540090b0 tx=0x7fe254004750 comp rx=0 tx=0).stop 2024-03-29T11:28:34.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.911+0000 7fe26ee00700 1 -- 172.21.15.12:0/2075521128 shutdown_connections 2024-03-29T11:28:34.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.911+0000 7fe26ee00700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24806adb0 0x7fe24806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.911+0000 7fe26ee00700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe26815fcb0 0x7fe2680ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.911+0000 7fe26ee00700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2680ced80 0x7fe268155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.911+0000 7fe26ee00700 1 --2- 172.21.15.12:0/2075521128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26815dad0 0x7fe2680cc600 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:34.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.911+0000 7fe26ee00700 1 -- 172.21.15.12:0/2075521128 >> 172.21.15.12:0/2075521128 conn(0x7fe26808aff0 msgr2=0x7fe26815ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:34.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.911+0000 7fe26ee00700 1 -- 172.21.15.12:0/2075521128 shutdown_connections 2024-03-29T11:28:34.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:34.911+0000 7fe26ee00700 1 -- 172.21.15.12:0/2075521128 wait complete. 2024-03-29T11:28:34.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-03-29T11:28:35.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.129+0000 7f9ac5400700 1 Processor -- start 2024-03-29T11:28:35.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.129+0000 7f9ac5400700 1 -- start start 2024-03-29T11:28:35.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.130+0000 7f9ac5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d18b0 0x7f9ac00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.130+0000 7f9ac5400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9ac0161590 0x7f9ac0161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.130+0000 7f9ac5400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac00ceda0 con 0x7f9ac0160f10 2024-03-29T11:28:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.130+0000 7f9ac5400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac00d1d30 con 0x7f9ac0161590 2024-03-29T11:28:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.130+0000 7f9ac5400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0161f60 con 0x7f9ac00d18b0 2024-03-29T11:28:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.131+0000 7f9abf200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9ac0161590 0x7f9ac0161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.131+0000 7f9abde00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d18b0 0x7f9ac00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.131+0000 7f9abf200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9ac0161590 0x7f9ac0161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:39038/0 (socket says 172.21.15.12:39038) 2024-03-29T11:28:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.131+0000 7f9abf200700 1 -- 172.21.15.12:0/3227190001 learned_addr learned my addr 172.21.15.12:0/3227190001 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:35.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.131+0000 7f9abe800700 1 --1- 172.21.15.12:0/3227190001 >> v1:172.21.15.159:6789/0 conn(0x7f9ac0160f10 0x7f9ac00d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:47950/0 (socket says 172.21.15.12:47950) 2024-03-29T11:28:35.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.131+0000 7f9abf200700 1 -- 172.21.15.12:0/3227190001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d18b0 msgr2=0x7f9ac00cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:35.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.131+0000 7f9abf200700 1 --2- 172.21.15.12:0/3227190001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d18b0 0x7f9ac00cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.131+0000 7f9abf200700 1 -- 172.21.15.12:0/3227190001 >> v1:172.21.15.159:6789/0 conn(0x7f9ac0160f10 legacy=0x7f9ac00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.131+0000 7f9abf200700 1 -- 172.21.15.12:0/3227190001 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac01612f0 con 0x7f9ac0161590 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.131+0000 7f9abf200700 1 --2- 172.21.15.12:0/3227190001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac0161590 0x7f9ac0161a20 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009e80 tx=0x7f9ab800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3885c9f2cd2e84e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9abd400700 1 -- 172.21.15.12:0/3227190001 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab800e070 con 0x7f9ac0161590 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9abd400700 1 -- 172.21.15.12:0/3227190001 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab800b8d0 con 0x7f9ac0161590 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9abd400700 1 -- 172.21.15.12:0/3227190001 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab800bc80 con 0x7f9ac0161590 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9ac5400700 1 -- 172.21.15.12:0/3227190001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac0161590 msgr2=0x7f9ac0161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9ac5400700 1 --2- 172.21.15.12:0/3227190001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac0161590 0x7f9ac0161a20 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009e80 tx=0x7f9ab800b330 comp rx=0 tx=0).stop 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9ac5400700 1 -- 172.21.15.12:0/3227190001 shutdown_connections 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9ac5400700 1 --2- 172.21.15.12:0/3227190001 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac0161590 0x7f9ac0161a20 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9ac5400700 1 --2- 172.21.15.12:0/3227190001 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d18b0 0x7f9ac00cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9ac5400700 1 -- 172.21.15.12:0/3227190001 >> 172.21.15.12:0/3227190001 conn(0x7f9ac008aff0 msgr2=0x7f9ac00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9ac5400700 1 -- 172.21.15.12:0/3227190001 shutdown_connections 2024-03-29T11:28:35.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.132+0000 7f9ac5400700 1 -- 172.21.15.12:0/3227190001 wait complete. 2024-03-29T11:28:35.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.134+0000 7f9ac5400700 1 Processor -- start 2024-03-29T11:28:35.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.134+0000 7f9ac5400700 1 -- start start 2024-03-29T11:28:35.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.135+0000 7f9ac5400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac00d18b0 0x7f9ac0158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.135+0000 7f9ac5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac0160f10 0x7f9ac00b75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.135+0000 7f9ac5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac0161590 0x7f9ac0156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.135+0000 7f9ac5400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00b85b0 con 0x7f9ac0161590 2024-03-29T11:28:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.135+0000 7f9ac5400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00b7ff0 con 0x7f9ac00d18b0 2024-03-29T11:28:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.135+0000 7f9ac5400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00b82d0 con 0x7f9ac0160f10 2024-03-29T11:28:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abf200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac0161590 0x7f9ac0156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abf200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac0161590 0x7f9ac0156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42820/0 (socket says 172.21.15.12:42820) 2024-03-29T11:28:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abf200700 1 -- 172.21.15.12:0/1957903994 learned_addr learned my addr 172.21.15.12:0/1957903994 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abe800700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac00d18b0 0x7f9ac0158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abde00700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac0160f10 0x7f9ac00b75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abf200700 1 -- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac0160f10 msgr2=0x7f9ac00b75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abf200700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac0160f10 0x7f9ac00b75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abf200700 1 -- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac00d18b0 msgr2=0x7f9ac0158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abf200700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac00d18b0 0x7f9ac0158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abf200700 1 -- 172.21.15.12:0/1957903994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8009b30 con 0x7f9ac0161590 2024-03-29T11:28:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abde00700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac0160f10 0x7f9ac00b75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9abf200700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac0161590 0x7f9ac0156f10 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f9ac008c570 tx=0x7f9ab8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.136+0000 7f9ab6a00700 1 -- 172.21.15.12:0/1957903994 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80048b0 con 0x7f9ac0161590 2024-03-29T11:28:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.137+0000 7f9ab6a00700 1 -- 172.21.15.12:0/1957903994 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab80099b0 con 0x7f9ac0161590 2024-03-29T11:28:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.137+0000 7f9ab6a00700 1 -- 172.21.15.12:0/1957903994 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab801c680 con 0x7f9ac0161590 2024-03-29T11:28:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.137+0000 7f9ac5400700 1 -- 172.21.15.12:0/1957903994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac0156710 con 0x7f9ac0161590 2024-03-29T11:28:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.137+0000 7f9ac5400700 1 -- 172.21.15.12:0/1957903994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac015d1f0 con 0x7f9ac0161590 2024-03-29T11:28:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.138+0000 7f9ac5400700 1 -- 172.21.15.12:0/1957903994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac015d380 con 0x7f9ac0161590 2024-03-29T11:28:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.139+0000 7f9ab6a00700 1 -- 172.21.15.12:0/1957903994 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9ab801c7e0 con 0x7f9ac0161590 2024-03-29T11:28:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.140+0000 7f9ab6a00700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006adb0 0x7f9aa006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.140+0000 7f9ab6a00700 1 -- 172.21.15.12:0/1957903994 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5961+0+0 (secure 0 0 0) 0x7f9ab808afb0 con 0x7f9ac0161590 2024-03-29T11:28:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.140+0000 7f9abe800700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006adb0 0x7f9aa006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.141+0000 7f9abe800700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006adb0 0x7f9aa006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0000f60 tx=0x7f9ab0007ed0 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.145+0000 7f9ab6a00700 1 -- 172.21.15.12:0/1957903994 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ab80be580 con 0x7f9ac0161590 2024-03-29T11:28:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.446+0000 7f9ac5400700 1 -- 172.21.15.12:0/1957903994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f9ac0156c30 con 0x7f9ac0161590 2024-03-29T11:28:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.520+0000 7f9ab6a00700 1 -- 172.21.15.12:0/1957903994 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3039) v1 ==== 165+0+0 (secure 0 0 0) 0x7f9ac0156c30 con 0x7f9ac0161590 2024-03-29T11:28:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 -- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006adb0 msgr2=0x7f9aa006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006adb0 0x7f9aa006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0000f60 tx=0x7f9ab0007ed0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 -- 172.21.15.12:0/1957903994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac0161590 msgr2=0x7f9ac0156f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac0161590 0x7f9ac0156f10 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f9ac008c570 tx=0x7f9ab8004ad0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 -- 172.21.15.12:0/1957903994 shutdown_connections 2024-03-29T11:28:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006adb0 0x7f9aa006d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac0161590 0x7f9ac0156f10 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac00d18b0 0x7f9ac0158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 --2- 172.21.15.12:0/1957903994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac0160f10 0x7f9ac00b75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 -- 172.21.15.12:0/1957903994 >> 172.21.15.12:0/1957903994 conn(0x7f9ac008aff0 msgr2=0x7f9ac006bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 -- 172.21.15.12:0/1957903994 shutdown_connections 2024-03-29T11:28:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.523+0000 7f9ac5400700 1 -- 172.21.15.12:0/1957903994 wait complete. 2024-03-29T11:28:35.531 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2024-03-29T11:28:35.531 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2024-03-29T11:28:35.531 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-29T11:28:35.532 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-03-29T11:28:35.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.740+0000 7f144b000700 1 Processor -- start 2024-03-29T11:28:35.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.740+0000 7f144b000700 1 -- start start 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.742+0000 7f144b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 0x7f14440d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.742+0000 7f144b000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f14441619d0 0x7f14440b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.742+0000 7f144b000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14440d30d0 con 0x7f14440ceec0 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.742+0000 7f144b000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14440b6840 con 0x7f14441619d0 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.742+0000 7f144b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14440d2880 con 0x7f144415d640 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.742+0000 7f1443400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f14440ceec0 0x7f14441612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36342/0 (socket says 172.21.15.12:36342) 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.742+0000 7f1443400700 1 -- 172.21.15.12:0/2003362577 learned_addr learned my addr 172.21.15.12:0/2003362577 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.742+0000 7f1449000700 1 --2- 172.21.15.12:0/2003362577 >> v2:172.21.15.12:3300/0 conn(0x7f14441619d0 0x7f14440b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.742+0000 7f1443e00700 1 --2- 172.21.15.12:0/2003362577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 0x7f14440d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.742+0000 7f1442a00700 1 -- 172.21.15.12:0/2003362577 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551469298 0 0) 0x7f14440d30d0 con 0x7f14440ceec0 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.743+0000 7f1442a00700 1 -- 172.21.15.12:0/2003362577 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14280035b0 con 0x7f14440ceec0 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.743+0000 7f1449000700 1 -- 172.21.15.12:0/2003362577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 msgr2=0x7f14440d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:35.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.743+0000 7f1449000700 1 --2- 172.21.15.12:0/2003362577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 0x7f14440d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.743+0000 7f1449000700 1 -- 172.21.15.12:0/2003362577 >> v1:172.21.15.12:6789/0 conn(0x7f14440ceec0 legacy=0x7f14441612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.743+0000 7f1449000700 1 -- 172.21.15.12:0/2003362577 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14440d2b30 con 0x7f14441619d0 2024-03-29T11:28:35.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.743+0000 7f1449000700 1 --2- 172.21.15.12:0/2003362577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 0x7f14440b6cd0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f143c009d90 tx=0x7f143c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd3f8e1b7a30e95 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.743+0000 7f1442a00700 1 -- 172.21.15.12:0/2003362577 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c00e070 con 0x7f14441619d0 2024-03-29T11:28:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.743+0000 7f1442a00700 1 -- 172.21.15.12:0/2003362577 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143c00bb20 con 0x7f14441619d0 2024-03-29T11:28:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.743+0000 7f1442a00700 1 -- 172.21.15.12:0/2003362577 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c012430 con 0x7f14441619d0 2024-03-29T11:28:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.744+0000 7f144b000700 1 -- 172.21.15.12:0/2003362577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 msgr2=0x7f14440b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.744+0000 7f144b000700 1 --2- 172.21.15.12:0/2003362577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 0x7f14440b6cd0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f143c009d90 tx=0x7f143c00b330 comp rx=0 tx=0).stop 2024-03-29T11:28:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.744+0000 7f144b000700 1 -- 172.21.15.12:0/2003362577 shutdown_connections 2024-03-29T11:28:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.744+0000 7f144b000700 1 --2- 172.21.15.12:0/2003362577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 0x7f14440b6cd0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.744+0000 7f144b000700 1 --2- 172.21.15.12:0/2003362577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 0x7f14440d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.744+0000 7f144b000700 1 -- 172.21.15.12:0/2003362577 >> 172.21.15.12:0/2003362577 conn(0x7f144408aff0 msgr2=0x7f14440d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:35.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.744+0000 7f144b000700 1 -- 172.21.15.12:0/2003362577 shutdown_connections 2024-03-29T11:28:35.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.744+0000 7f144b000700 1 -- 172.21.15.12:0/2003362577 wait complete. 2024-03-29T11:28:35.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.746+0000 7f144b000700 1 Processor -- start 2024-03-29T11:28:35.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.746+0000 7f144b000700 1 -- start start 2024-03-29T11:28:35.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f144b000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14440ceec0 0x7f1444150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f144b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 0x7f14440c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f144b000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 0x7f144409fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f144b000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144409f340 con 0x7f14441619d0 2024-03-29T11:28:35.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f144b000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144409e6c0 con 0x7f14440ceec0 2024-03-29T11:28:35.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f144b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144409e9a0 con 0x7f144415d640 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f1449000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 0x7f144409fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f1449000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 0x7f144409fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42848/0 (socket says 172.21.15.12:42848) 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f1449000700 1 -- 172.21.15.12:0/1909290251 learned_addr learned my addr 172.21.15.12:0/1909290251 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f1443400700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 0x7f14440c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.747+0000 7f1443e00700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14440ceec0 0x7f1444150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f1449000700 1 -- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 msgr2=0x7f14440c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f1449000700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 0x7f14440c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f1449000700 1 -- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14440ceec0 msgr2=0x7f1444150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f1449000700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14440ceec0 0x7f1444150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f1449000700 1 -- 172.21.15.12:0/1909290251 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143c009a70 con 0x7f14441619d0 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f1443400700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 0x7f14440c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f1443e00700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14440ceec0 0x7f1444150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f1449000700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 0x7f144409fe80 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f143c014040 tx=0x7f143c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f143be00700 1 -- 172.21.15.12:0/1909290251 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c012a70 con 0x7f14441619d0 2024-03-29T11:28:35.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f143be00700 1 -- 172.21.15.12:0/1909290251 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143c020070 con 0x7f14441619d0 2024-03-29T11:28:35.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f143be00700 1 -- 172.21.15.12:0/1909290251 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c012e30 con 0x7f14441619d0 2024-03-29T11:28:35.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f144b000700 1 -- 172.21.15.12:0/1909290251 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144409ec80 con 0x7f14441619d0 2024-03-29T11:28:35.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.748+0000 7f144b000700 1 -- 172.21.15.12:0/1909290251 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1444150d40 con 0x7f14441619d0 2024-03-29T11:28:35.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.749+0000 7f144b000700 1 -- 172.21.15.12:0/1909290251 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1444067720 con 0x7f14441619d0 2024-03-29T11:28:35.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.750+0000 7f143be00700 1 -- 172.21.15.12:0/1909290251 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f143c004040 con 0x7f14441619d0 2024-03-29T11:28:35.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.751+0000 7f143be00700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f142406adb0 0x7f142406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:35.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.751+0000 7f143be00700 1 -- 172.21.15.12:0/1909290251 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5961+0+0 (secure 0 0 0) 0x7f143c022070 con 0x7f14441619d0 2024-03-29T11:28:35.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.752+0000 7f1443e00700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f142406adb0 0x7f142406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:35.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.752+0000 7f1443e00700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f142406adb0 0x7f142406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1434009b90 tx=0x7f1434008040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:35.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:35.756+0000 7f143be00700 1 -- 172.21.15.12:0/1909290251 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f143c059da0 con 0x7f14441619d0 2024-03-29T11:28:36.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.050+0000 7f144b000700 1 -- 172.21.15.12:0/1909290251 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f14440b5c70 con 0x7f14441619d0 2024-03-29T11:28:36.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.525+0000 7f143be00700 1 -- 172.21.15.12:0/1909290251 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3040) v1 ==== 166+0+0 (secure 0 0 0) 0x7f143c059930 con 0x7f14441619d0 2024-03-29T11:28:36.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 -- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f142406adb0 msgr2=0x7f142406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f142406adb0 0x7f142406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1434009b90 tx=0x7f1434008040 comp rx=0 tx=0).stop 2024-03-29T11:28:36.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 -- 172.21.15.12:0/1909290251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 msgr2=0x7f144409fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 0x7f144409fe80 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f143c014040 tx=0x7f143c004ad0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 -- 172.21.15.12:0/1909290251 shutdown_connections 2024-03-29T11:28:36.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f142406adb0 0x7f142406d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14441619d0 0x7f144409fe80 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14440ceec0 0x7f1444150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 --2- 172.21.15.12:0/1909290251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f144415d640 0x7f14440c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 -- 172.21.15.12:0/1909290251 >> 172.21.15.12:0/1909290251 conn(0x7f144408aff0 msgr2=0x7f144415ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:36.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 -- 172.21.15.12:0/1909290251 shutdown_connections 2024-03-29T11:28:36.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.528+0000 7f144b000700 1 -- 172.21.15.12:0/1909290251 wait complete. 2024-03-29T11:28:36.538 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T11:28:36.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.751+0000 7fe597c00700 1 Processor -- start 2024-03-29T11:28:36.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.751+0000 7fe597c00700 1 -- start start 2024-03-29T11:28:36.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.752+0000 7fe597c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe5900cd020 0x7fe5900cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:36.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.752+0000 7fe597c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5900b2070 0x7fe5900b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:36.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.752+0000 7fe597c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5900cdae0 con 0x7fe5900cbb30 2024-03-29T11:28:36.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.752+0000 7fe597c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5900cddc0 con 0x7fe5900b2070 2024-03-29T11:28:36.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.752+0000 7fe597c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5900ce0a0 con 0x7fe5900cd020 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.752+0000 7fe595200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe5900cbb30 0x7fe5900cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36344/0 (socket says 172.21.15.12:36344) 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.752+0000 7fe595c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5900b2070 0x7fe5900b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.753+0000 7fe595200700 1 -- 172.21.15.12:0/1372049161 learned_addr learned my addr 172.21.15.12:0/1372049161 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.753+0000 7fe58fe00700 1 --2- 172.21.15.12:0/1372049161 >> v2:172.21.15.159:3300/0 conn(0x7fe5900cd020 0x7fe5900cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.753+0000 7fe595c00700 1 -- 172.21.15.12:0/1372049161 >> v2:172.21.15.159:3300/0 conn(0x7fe5900cd020 msgr2=0x7fe5900cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.753+0000 7fe595c00700 1 --2- 172.21.15.12:0/1372049161 >> v2:172.21.15.159:3300/0 conn(0x7fe5900cd020 0x7fe5900cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.753+0000 7fe595c00700 1 -- 172.21.15.12:0/1372049161 >> v1:172.21.15.12:6789/0 conn(0x7fe5900cbb30 legacy=0x7fe5900cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.753+0000 7fe595c00700 1 -- 172.21.15.12:0/1372049161 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59006dc20 con 0x7fe5900b2070 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.753+0000 7fe58f400700 1 -- 172.21.15.12:0/1372049161 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800128171 0 0) 0x7fe5900cdae0 con 0x7fe5900cbb30 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.753+0000 7fe595c00700 1 --2- 172.21.15.12:0/1372049161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900b2070 0x7fe5900b2500 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fe5900b1e30 tx=0x7fe58800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a959ce8be8aa135f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.753+0000 7fe58f400700 1 -- 172.21.15.12:0/1372049161 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58800e070 con 0x7fe5900b2070 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.754+0000 7fe58f400700 1 -- 172.21.15.12:0/1372049161 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58800bb20 con 0x7fe5900b2070 2024-03-29T11:28:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.754+0000 7fe58f400700 1 -- 172.21.15.12:0/1372049161 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe588012430 con 0x7fe5900b2070 2024-03-29T11:28:36.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.754+0000 7fe597c00700 1 -- 172.21.15.12:0/1372049161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900b2070 msgr2=0x7fe5900b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.754+0000 7fe597c00700 1 --2- 172.21.15.12:0/1372049161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900b2070 0x7fe5900b2500 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fe5900b1e30 tx=0x7fe58800b330 comp rx=0 tx=0).stop 2024-03-29T11:28:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.754+0000 7fe597c00700 1 -- 172.21.15.12:0/1372049161 shutdown_connections 2024-03-29T11:28:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.754+0000 7fe597c00700 1 --2- 172.21.15.12:0/1372049161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900b2070 0x7fe5900b2500 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.754+0000 7fe597c00700 1 --2- 172.21.15.12:0/1372049161 >> v2:172.21.15.159:3300/0 conn(0x7fe5900cd020 0x7fe5900cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.754+0000 7fe597c00700 1 -- 172.21.15.12:0/1372049161 >> 172.21.15.12:0/1372049161 conn(0x7fe5900b0d70 msgr2=0x7fe5900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.754+0000 7fe597c00700 1 -- 172.21.15.12:0/1372049161 shutdown_connections 2024-03-29T11:28:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.754+0000 7fe597c00700 1 -- 172.21.15.12:0/1372049161 wait complete. 2024-03-29T11:28:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.756+0000 7fe597c00700 1 Processor -- start 2024-03-29T11:28:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.756+0000 7fe597c00700 1 -- start start 2024-03-29T11:28:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.757+0000 7fe597c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5900b2070 0x7fe5900ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.757+0000 7fe597c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900cbb30 0x7fe590151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.757+0000 7fe597c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5900cd020 0x7fe590157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.757+0000 7fe597c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5900c95e0 con 0x7fe5900cbb30 2024-03-29T11:28:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.757+0000 7fe597c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5900cab10 con 0x7fe5900b2070 2024-03-29T11:28:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.757+0000 7fe597c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5900c9810 con 0x7fe5900cd020 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.757+0000 7fe58fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900cbb30 0x7fe590151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.757+0000 7fe58fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900cbb30 0x7fe590151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42868/0 (socket says 172.21.15.12:42868) 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.757+0000 7fe58fe00700 1 -- 172.21.15.12:0/2309817879 learned_addr learned my addr 172.21.15.12:0/2309817879 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe595200700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5900b2070 0x7fe5900ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe595c00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5900cd020 0x7fe590157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe58fe00700 1 -- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5900cd020 msgr2=0x7fe590157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe58fe00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5900cd020 0x7fe590157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe58fe00700 1 -- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5900b2070 msgr2=0x7fe5900ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe58fe00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5900b2070 0x7fe5900ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe58fe00700 1 -- 172.21.15.12:0/2309817879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe588009a70 con 0x7fe5900cbb30 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe58fe00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900cbb30 0x7fe590151ba0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00e870 tx=0x7fe57c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe58cc00700 1 -- 172.21.15.12:0/2309817879 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c009590 con 0x7fe5900cbb30 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe58cc00700 1 -- 172.21.15.12:0/2309817879 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57c00ee70 con 0x7fe5900cbb30 2024-03-29T11:28:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe597c00700 1 -- 172.21.15.12:0/2309817879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5900c9af0 con 0x7fe5900cbb30 2024-03-29T11:28:36.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe58cc00700 1 -- 172.21.15.12:0/2309817879 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c00ca50 con 0x7fe5900cbb30 2024-03-29T11:28:36.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.758+0000 7fe597c00700 1 -- 172.21.15.12:0/2309817879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5900ca0b0 con 0x7fe5900cbb30 2024-03-29T11:28:36.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.759+0000 7fe597c00700 1 -- 172.21.15.12:0/2309817879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59006d0a0 con 0x7fe5900cbb30 2024-03-29T11:28:36.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.760+0000 7fe58cc00700 1 -- 172.21.15.12:0/2309817879 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe57c00cbb0 con 0x7fe5900cbb30 2024-03-29T11:28:36.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.761+0000 7fe58cc00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57006adb0 0x7fe57006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:36.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.761+0000 7fe58cc00700 1 -- 172.21.15.12:0/2309817879 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5961+0+0 (secure 0 0 0) 0x7fe57c089560 con 0x7fe5900cbb30 2024-03-29T11:28:36.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.762+0000 7fe595200700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57006adb0 0x7fe57006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:36.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.762+0000 7fe595200700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57006adb0 0x7fe57006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe584000f60 tx=0x7fe584007be0 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:36.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.766+0000 7fe58cc00700 1 -- 172.21.15.12:0/2309817879 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe57c054520 con 0x7fe5900cbb30 2024-03-29T11:28:36.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.979+0000 7fe597c00700 1 -- 172.21.15.12:0/2309817879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe590067720 con 0x7fe5900cbb30 2024-03-29T11:28:36.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.983+0000 7fe58cc00700 1 -- 172.21.15.12:0/2309817879 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe57c00ce60 con 0x7fe5900cbb30 2024-03-29T11:28:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 -- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57006adb0 msgr2=0x7fe57006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57006adb0 0x7fe57006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe584000f60 tx=0x7fe584007be0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 -- 172.21.15.12:0/2309817879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900cbb30 msgr2=0x7fe590151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900cbb30 0x7fe590151ba0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00e870 tx=0x7fe57c00ec30 comp rx=0 tx=0).stop 2024-03-29T11:28:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 -- 172.21.15.12:0/2309817879 shutdown_connections 2024-03-29T11:28:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57006adb0 0x7fe57006d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5900cd020 0x7fe590157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5900b2070 0x7fe5900ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 --2- 172.21.15.12:0/2309817879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5900cbb30 0x7fe590151ba0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 -- 172.21.15.12:0/2309817879 >> 172.21.15.12:0/2309817879 conn(0x7fe5900b0d70 msgr2=0x7fe59015bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 -- 172.21.15.12:0/2309817879 shutdown_connections 2024-03-29T11:28:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:36.986+0000 7fe597c00700 1 -- 172.21.15.12:0/2309817879 wait complete. 2024-03-29T11:28:36.994 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:28:36.995 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:28:37.078 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:28:37.182 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:28:37.183 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:28:37.183 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:28:37.227 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:28:37.228 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:28:37.228 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:28:37.275 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:28:37.275 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:28:37.275 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:28:37.319 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:28:37.319 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:28:37.319 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:28:37.365 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:28:37.365 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T11:28:37.365 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:28:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.572+0000 7f582d000700 1 Processor -- start 2024-03-29T11:28:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.573+0000 7f582d000700 1 -- start start 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f582d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f582815da50 0x7f58280ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f582d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 0x7f58280b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f582d000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58280d24e0 con 0x7f58280cec70 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f582d000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58280b95d0 con 0x7f582815da50 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f582d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582815de30 con 0x7f58280d0cf0 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f5825e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f582815da50 0x7f58280ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f5825e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f582815da50 0x7f58280ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42878/0 (socket says 172.21.15.12:42878) 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f5825e00700 1 -- 172.21.15.12:0/2902872664 learned_addr learned my addr 172.21.15.12:0/2902872664 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f5826800700 1 --2- 172.21.15.12:0/2902872664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 0x7f58280b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f5825e00700 1 -- 172.21.15.12:0/2902872664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 msgr2=0x7f58280b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f5825e00700 1 --2- 172.21.15.12:0/2902872664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 0x7f58280b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:37.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f5825e00700 1 -- 172.21.15.12:0/2902872664 >> v1:172.21.15.159:6789/0 conn(0x7f58280cec70 legacy=0x7f58280d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:37.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f5825e00700 1 -- 172.21.15.12:0/2902872664 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58280d27c0 con 0x7f582815da50 2024-03-29T11:28:37.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.574+0000 7f5826800700 1 --2- 172.21.15.12:0/2902872664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 0x7f58280b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:37.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.575+0000 7f5825e00700 1 --2- 172.21.15.12:0/2902872664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 0x7f58280ce730 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f58180098f0 tx=0x7f5818009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122e42a2ed7b4f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:37.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.575+0000 7f5824a00700 1 -- 172.21.15.12:0/2902872664 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581800b8a0 con 0x7f582815da50 2024-03-29T11:28:37.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.575+0000 7f5824a00700 1 -- 172.21.15.12:0/2902872664 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581800ba00 con 0x7f582815da50 2024-03-29T11:28:37.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.575+0000 7f5824a00700 1 -- 172.21.15.12:0/2902872664 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581800bd90 con 0x7f582815da50 2024-03-29T11:28:37.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.575+0000 7f582d000700 1 -- 172.21.15.12:0/2902872664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 msgr2=0x7f58280ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.575+0000 7f582d000700 1 --2- 172.21.15.12:0/2902872664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 0x7f58280ce730 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f58180098f0 tx=0x7f5818009c00 comp rx=0 tx=0).stop 2024-03-29T11:28:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.575+0000 7f582d000700 1 -- 172.21.15.12:0/2902872664 shutdown_connections 2024-03-29T11:28:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.575+0000 7f582d000700 1 --2- 172.21.15.12:0/2902872664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 0x7f58280b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.575+0000 7f582d000700 1 --2- 172.21.15.12:0/2902872664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 0x7f58280ce730 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.576+0000 7f582d000700 1 -- 172.21.15.12:0/2902872664 >> 172.21.15.12:0/2902872664 conn(0x7f58280666a0 msgr2=0x7f5828066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.576+0000 7f582d000700 1 -- 172.21.15.12:0/2902872664 shutdown_connections 2024-03-29T11:28:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.576+0000 7f582d000700 1 -- 172.21.15.12:0/2902872664 wait complete. 2024-03-29T11:28:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.577+0000 7f582d000700 1 Processor -- start 2024-03-29T11:28:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.577+0000 7f582d000700 1 -- start start 2024-03-29T11:28:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.578+0000 7f582d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58280cec70 0x7f582815df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.578+0000 7f582d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 0x7f58281f0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.578+0000 7f582d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 0x7f58281f7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.578+0000 7f582d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5828152590 con 0x7f582815da50 2024-03-29T11:28:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.578+0000 7f582d000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5828160a70 con 0x7f58280cec70 2024-03-29T11:28:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.578+0000 7f582d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582815e4b0 con 0x7f58280d0cf0 2024-03-29T11:28:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5826800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 0x7f58281f7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5826800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 0x7f58281f7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42888/0 (socket says 172.21.15.12:42888) 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5826800700 1 -- 172.21.15.12:0/2183606935 learned_addr learned my addr 172.21.15.12:0/2183606935 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5825400700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 0x7f58281f0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5825e00700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58280cec70 0x7f582815df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5826800700 1 -- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 msgr2=0x7f58281f0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5826800700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 0x7f58281f0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5826800700 1 -- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58280cec70 msgr2=0x7f582815df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5826800700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58280cec70 0x7f582815df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5826800700 1 -- 172.21.15.12:0/2183606935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5818009550 con 0x7f582815da50 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f5826800700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 0x7f58281f7400 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f5820007e90 tx=0x7f582000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.579+0000 7f581e000700 1 -- 172.21.15.12:0/2183606935 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582000dd20 con 0x7f582815da50 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.580+0000 7f581e000700 1 -- 172.21.15.12:0/2183606935 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582000f040 con 0x7f582815da50 2024-03-29T11:28:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.580+0000 7f581e000700 1 -- 172.21.15.12:0/2183606935 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5820003c60 con 0x7f582815da50 2024-03-29T11:28:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.580+0000 7f582d000700 1 -- 172.21.15.12:0/2183606935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58281f0e80 con 0x7f582815da50 2024-03-29T11:28:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.580+0000 7f582d000700 1 -- 172.21.15.12:0/2183606935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58281f7940 con 0x7f582815da50 2024-03-29T11:28:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.581+0000 7f582d000700 1 -- 172.21.15.12:0/2183606935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58280cfd00 con 0x7f582815da50 2024-03-29T11:28:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.582+0000 7f581e000700 1 -- 172.21.15.12:0/2183606935 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5820004030 con 0x7f582815da50 2024-03-29T11:28:37.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.582+0000 7f581e000700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f580806adb0 0x7f580806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:37.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.583+0000 7f581e000700 1 -- 172.21.15.12:0/2183606935 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5961+0+0 (secure 0 0 0) 0x7f5820089670 con 0x7f582815da50 2024-03-29T11:28:37.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.583+0000 7f5825e00700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f580806adb0 0x7f580806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:37.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.583+0000 7f5825e00700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f580806adb0 0x7f580806d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5818014040 tx=0x7f5818008030 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:37.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.587+0000 7f581e000700 1 -- 172.21.15.12:0/2183606935 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5820054520 con 0x7f582815da50 2024-03-29T11:28:37.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.847+0000 7f582d000700 1 -- 172.21.15.12:0/2183606935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5828067720 con 0x7f582815da50 2024-03-29T11:28:37.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.847+0000 7f581e000700 1 -- 172.21.15.12:0/2183606935 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+3533 (secure 0 0 0) 0x7f582001ade0 con 0x7f582815da50 2024-03-29T11:28:37.838 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:37.838 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":91,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2592133465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2592133465},{"type":"v1","addr":"172.21.15.159:6837","nonce":2592133465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":8381,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/204651369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":204651369},{"type":"v1","addr":"172.21.15.12:6835","nonce":204651369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9616,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2547917697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2547917697},{"type":"v1","addr":"172.21.15.159:6836","nonce":2547917697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":10425,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/2459180530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2459180530},{"type":"v1","addr":"172.21.15.12:6833","nonce":2459180530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[]} 2024-03-29T11:28:37.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 -- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f580806adb0 msgr2=0x7f580806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:37.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f580806adb0 0x7f580806d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5818014040 tx=0x7f5818008030 comp rx=0 tx=0).stop 2024-03-29T11:28:37.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 -- 172.21.15.12:0/2183606935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 msgr2=0x7f58281f7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:37.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 0x7f58281f7400 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f5820007e90 tx=0x7f582000d3b0 comp rx=0 tx=0).stop 2024-03-29T11:28:37.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 -- 172.21.15.12:0/2183606935 shutdown_connections 2024-03-29T11:28:37.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f580806adb0 0x7f580806d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:37.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582815da50 0x7f58281f7400 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:37.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58280cec70 0x7f582815df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:37.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 --2- 172.21.15.12:0/2183606935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58280d0cf0 0x7f58281f0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:37.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 -- 172.21.15.12:0/2183606935 >> 172.21.15.12:0/2183606935 conn(0x7f58280666a0 msgr2=0x7f582815c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:37.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 -- 172.21.15.12:0/2183606935 shutdown_connections 2024-03-29T11:28:37.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:37.850+0000 7f582d000700 1 -- 172.21.15.12:0/2183606935 wait complete. 2024-03-29T11:28:37.841 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 91 2024-03-29T11:28:37.859 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T11:28:37.859 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T11:28:37.860 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T11:28:37.860 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:28:37.860 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T11:28:37.860 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T11:28:37.860 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T11:28:37.860 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:28:37.860 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T11:28:37.861 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T11:28:37.861 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T11:28:37.861 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:28:37.861 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T11:28:37.861 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T11:28:37.861 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T11:28:37.861 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:28:37.874 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T11:28:37.874 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T11:28:37.875 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T11:28:37.875 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T11:28:37.880 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T11:28:37.880 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T11:28:37.918 INFO:tasks.ceph.mds.c:Started 2024-03-29T11:28:37.918 DEBUG:teuthology.parallel:result is None 2024-03-29T11:28:37.920 INFO:tasks.ceph.mds.a:Started 2024-03-29T11:28:37.921 DEBUG:teuthology.parallel:result is None 2024-03-29T11:28:37.925 INFO:tasks.ceph.mds.d:Started 2024-03-29T11:28:37.925 DEBUG:teuthology.parallel:result is None 2024-03-29T11:28:38.063 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T11:28:38.064 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:28:38.067 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T11:28:39.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-29T11:28:43.880 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T11:28:43.881 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T11:28:43.885 INFO:tasks.ceph.mds.b:Started 2024-03-29T11:28:43.886 DEBUG:teuthology.parallel:result is None 2024-03-29T11:28:43.886 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T11:28:44.036 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T11:28:44.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.107+0000 7fbfa6400700 1 Processor -- start 2024-03-29T11:28:44.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.107+0000 7fbfa6400700 1 -- start start 2024-03-29T11:28:44.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.108+0000 7fbfa6400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbfa01611b0 0x7fbfa015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:44.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.108+0000 7fbfa6400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 0x7fbfa0161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:44.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.108+0000 7fbfa6400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa00ce270 con 0x7fbfa015f4d0 2024-03-29T11:28:44.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.108+0000 7fbfa6400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa015b600 con 0x7fbfa01611b0 2024-03-29T11:28:44.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.108+0000 7fbfa6400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00d2760 con 0x7fbfa00cee10 2024-03-29T11:28:44.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.108+0000 7fbf9f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbfa01611b0 0x7fbfa015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.108+0000 7fbf9f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbfa01611b0 0x7fbfa015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44532/0 (socket says 172.21.15.12:44532) 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9f400700 1 -- 172.21.15.12:0/4236589005 learned_addr learned my addr 172.21.15.12:0/4236589005 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9fe00700 1 --2- 172.21.15.12:0/4236589005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 0x7fbfa0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9f400700 1 -- 172.21.15.12:0/4236589005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 msgr2=0x7fbfa0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9f400700 1 --2- 172.21.15.12:0/4236589005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 0x7fbfa0161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9f400700 1 -- 172.21.15.12:0/4236589005 >> v1:172.21.15.159:6789/0 conn(0x7fbfa015f4d0 legacy=0x7fbfa00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9f400700 1 -- 172.21.15.12:0/4236589005 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa00d2a40 con 0x7fbfa01611b0 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9fe00700 1 --2- 172.21.15.12:0/4236589005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 0x7fbfa0161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9f400700 1 --2- 172.21.15.12:0/4236589005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 0x7fbfa015ef90 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fbf900098f0 tx=0x7fbf90009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f65d64b7818cd263 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9e000700 1 -- 172.21.15.12:0/4236589005 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9000b8a0 con 0x7fbfa01611b0 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9e000700 1 -- 172.21.15.12:0/4236589005 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9000ba00 con 0x7fbfa01611b0 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.109+0000 7fbf9e000700 1 -- 172.21.15.12:0/4236589005 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9000bd90 con 0x7fbfa01611b0 2024-03-29T11:28:44.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.110+0000 7fbfa6400700 1 -- 172.21.15.12:0/4236589005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 msgr2=0x7fbfa015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:44.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.110+0000 7fbfa6400700 1 --2- 172.21.15.12:0/4236589005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 0x7fbfa015ef90 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fbf900098f0 tx=0x7fbf90009c00 comp rx=0 tx=0).stop 2024-03-29T11:28:44.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.110+0000 7fbfa6400700 1 -- 172.21.15.12:0/4236589005 shutdown_connections 2024-03-29T11:28:44.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.110+0000 7fbfa6400700 1 --2- 172.21.15.12:0/4236589005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 0x7fbfa0161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:44.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.110+0000 7fbfa6400700 1 --2- 172.21.15.12:0/4236589005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 0x7fbfa015ef90 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:44.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.110+0000 7fbfa6400700 1 -- 172.21.15.12:0/4236589005 >> 172.21.15.12:0/4236589005 conn(0x7fbfa008aff0 msgr2=0x7fbfa00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:44.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.110+0000 7fbfa6400700 1 -- 172.21.15.12:0/4236589005 shutdown_connections 2024-03-29T11:28:44.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.110+0000 7fbfa6400700 1 -- 172.21.15.12:0/4236589005 wait complete. 2024-03-29T11:28:44.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.112+0000 7fbfa6400700 1 Processor -- start 2024-03-29T11:28:44.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.112+0000 7fbfa6400700 1 -- start start 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbfa6400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 0x7fbfa014d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbfa6400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfa015f4d0 0x7fbfa00b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbfa6400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 0x7fbfa00b8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbfa6400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00b7790 con 0x7fbfa01611b0 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbfa6400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00b7c20 con 0x7fbfa015f4d0 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbfa6400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00b6b80 con 0x7fbfa00cee10 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbf9fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 0x7fbfa00b8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbf9f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 0x7fbfa014d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbf9fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 0x7fbfa00b8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44542/0 (socket says 172.21.15.12:44542) 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbf9fe00700 1 -- 172.21.15.12:0/136786917 learned_addr learned my addr 172.21.15.12:0/136786917 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.113+0000 7fbf9ea00700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfa015f4d0 0x7fbfa00b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbf9fe00700 1 -- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 msgr2=0x7fbfa014d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbf9fe00700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 0x7fbfa014d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:44.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbf9fe00700 1 -- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfa015f4d0 msgr2=0x7fbfa00b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:44.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbf9fe00700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfa015f4d0 0x7fbfa00b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:44.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbf9fe00700 1 -- 172.21.15.12:0/136786917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf90009550 con 0x7fbfa01611b0 2024-03-29T11:28:44.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbf9fe00700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 0x7fbfa00b8950 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fbf98007ef0 tx=0x7fbf9800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:44.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbf97400700 1 -- 172.21.15.12:0/136786917 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9800dd80 con 0x7fbfa01611b0 2024-03-29T11:28:44.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbf97400700 1 -- 172.21.15.12:0/136786917 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9800f040 con 0x7fbfa01611b0 2024-03-29T11:28:44.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbf97400700 1 -- 172.21.15.12:0/136786917 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf98003c60 con 0x7fbfa01611b0 2024-03-29T11:28:44.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbfa6400700 1 -- 172.21.15.12:0/136786917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa00b6e60 con 0x7fbfa01611b0 2024-03-29T11:28:44.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.114+0000 7fbfa6400700 1 -- 172.21.15.12:0/136786917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa00b7420 con 0x7fbfa01611b0 2024-03-29T11:28:44.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.115+0000 7fbfa6400700 1 -- 172.21.15.12:0/136786917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa0162150 con 0x7fbfa01611b0 2024-03-29T11:28:44.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.116+0000 7fbf97400700 1 -- 172.21.15.12:0/136786917 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbf98004030 con 0x7fbfa01611b0 2024-03-29T11:28:44.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.117+0000 7fbf97400700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf8006ade0 0x7fbf8006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:44.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.117+0000 7fbf97400700 1 -- 172.21.15.12:0/136786917 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5961+0+0 (secure 0 0 0) 0x7fbf98089790 con 0x7fbfa01611b0 2024-03-29T11:28:44.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.117+0000 7fbf9f400700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf8006ade0 0x7fbf8006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:44.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.118+0000 7fbf9f400700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf8006ade0 0x7fbf8006d2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf90014040 tx=0x7fbf90008030 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:44.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.122+0000 7fbf97400700 1 -- 172.21.15.12:0/136786917 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf980545b0 con 0x7fbfa01611b0 2024-03-29T11:28:44.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:44.356+0000 7fbfa6400700 1 -- 172.21.15.12:0/136786917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbfa0067720 con 0x7fbfa01611b0 2024-03-29T11:28:45.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.255+0000 7fbf97400700 1 -- 172.21.15.12:0/136786917 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v204) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbf98057bd0 con 0x7fbfa01611b0 2024-03-29T11:28:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 -- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf8006ade0 msgr2=0x7fbf8006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf8006ade0 0x7fbf8006d2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf90014040 tx=0x7fbf90008030 comp rx=0 tx=0).stop 2024-03-29T11:28:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 -- 172.21.15.12:0/136786917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 msgr2=0x7fbfa00b8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 0x7fbfa00b8950 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fbf98007ef0 tx=0x7fbf9800d3b0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 -- 172.21.15.12:0/136786917 shutdown_connections 2024-03-29T11:28:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf8006ade0 0x7fbf8006d2a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfa01611b0 0x7fbfa00b8950 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa00cee10 0x7fbfa014d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 --2- 172.21.15.12:0/136786917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfa015f4d0 0x7fbfa00b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 -- 172.21.15.12:0/136786917 >> 172.21.15.12:0/136786917 conn(0x7fbfa008aff0 msgr2=0x7fbfa006c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 -- 172.21.15.12:0/136786917 shutdown_connections 2024-03-29T11:28:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.258+0000 7fbfa6400700 1 -- 172.21.15.12:0/136786917 wait complete. 2024-03-29T11:28:45.250 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T11:28:45.270 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T11:28:45.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.483+0000 7f6803200700 1 Processor -- start 2024-03-29T11:28:45.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.483+0000 7f6803200700 1 -- start start 2024-03-29T11:28:45.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.484+0000 7f6803200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc15d640 0x7f67fc0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:45.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.484+0000 7f6803200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc0d30d0 con 0x7f67fc0ceec0 2024-03-29T11:28:45.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.484+0000 7f6803200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc0b6840 con 0x7f67fc1619d0 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.484+0000 7f6803200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc0d2880 con 0x7f67fc15d640 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.485+0000 7f67fb400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f67fc0ceec0 0x7f67fc1612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54030/0 (socket says 172.21.15.12:54030) 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.485+0000 7f67fb400700 1 -- 172.21.15.12:0/867852513 learned_addr learned my addr 172.21.15.12:0/867852513 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.485+0000 7f67fbe00700 1 --2- 172.21.15.12:0/867852513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc15d640 0x7f67fc0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.485+0000 7f67faa00700 1 -- 172.21.15.12:0/867852513 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316482334 0 0) 0x7f67fc0d30d0 con 0x7f67fc0ceec0 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.485+0000 7f67faa00700 1 -- 172.21.15.12:0/867852513 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e00035b0 con 0x7f67fc0ceec0 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.485+0000 7f67faa00700 1 -- 172.21.15.12:0/867852513 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911770680 0 0) 0x7f67fc0b6840 con 0x7f67fc1619d0 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.485+0000 7f67faa00700 1 -- 172.21.15.12:0/867852513 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67fc0d30d0 con 0x7f67fc1619d0 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.485+0000 7f67fbe00700 1 -- 172.21.15.12:0/867852513 >> v1:172.21.15.159:6789/0 conn(0x7f67fc1619d0 legacy=0x7f67fc0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.485+0000 7f67fbe00700 1 -- 172.21.15.12:0/867852513 >> v1:172.21.15.12:6789/0 conn(0x7f67fc0ceec0 legacy=0x7f67fc1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.485+0000 7f67fbe00700 1 -- 172.21.15.12:0/867852513 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc0d2b30 con 0x7f67fc15d640 2024-03-29T11:28:45.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.486+0000 7f67fbe00700 1 --2- 172.21.15.12:0/867852513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc15d640 0x7f67fc0d2340 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009870 tx=0x7f67ec009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43ef233282245061 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.486+0000 7f67faa00700 1 -- 172.21.15.12:0/867852513 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec011830 con 0x7f67fc15d640 2024-03-29T11:28:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.486+0000 7f67faa00700 1 -- 172.21.15.12:0/867852513 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ec011990 con 0x7f67fc15d640 2024-03-29T11:28:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.486+0000 7f67faa00700 1 -- 172.21.15.12:0/867852513 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec011d20 con 0x7f67fc15d640 2024-03-29T11:28:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.486+0000 7f6803200700 1 -- 172.21.15.12:0/867852513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc15d640 msgr2=0x7f67fc0d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.486+0000 7f6803200700 1 --2- 172.21.15.12:0/867852513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc15d640 0x7f67fc0d2340 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009870 tx=0x7f67ec009b80 comp rx=0 tx=0).stop 2024-03-29T11:28:45.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.486+0000 7f6803200700 1 -- 172.21.15.12:0/867852513 shutdown_connections 2024-03-29T11:28:45.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.486+0000 7f6803200700 1 --2- 172.21.15.12:0/867852513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc15d640 0x7f67fc0d2340 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.486+0000 7f6803200700 1 -- 172.21.15.12:0/867852513 >> 172.21.15.12:0/867852513 conn(0x7f67fc08aff0 msgr2=0x7f67fc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:45.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.487+0000 7f6803200700 1 -- 172.21.15.12:0/867852513 shutdown_connections 2024-03-29T11:28:45.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.487+0000 7f6803200700 1 -- 172.21.15.12:0/867852513 wait complete. 2024-03-29T11:28:45.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.488+0000 7f6803200700 1 Processor -- start 2024-03-29T11:28:45.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.488+0000 7f6803200700 1 -- start start 2024-03-29T11:28:45.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.489+0000 7f6803200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc0ceec0 0x7f67fc158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:45.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.489+0000 7f6803200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67fc15d640 0x7f67fc09d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.489+0000 7f6803200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67fc1619d0 0x7f67fc15f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.489+0000 7f6803200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc09dc10 con 0x7f67fc1619d0 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.489+0000 7f6803200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc09def0 con 0x7f67fc15d640 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.489+0000 7f6803200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc09e590 con 0x7f67fc0ceec0 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f6801200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67fc1619d0 0x7f67fc15f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f6801200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67fc1619d0 0x7f67fc15f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44558/0 (socket says 172.21.15.12:44558) 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f6801200700 1 -- 172.21.15.12:0/2988171361 learned_addr learned my addr 172.21.15.12:0/2988171361 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f67fb400700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67fc15d640 0x7f67fc09d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f67fbe00700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc0ceec0 0x7f67fc158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f6801200700 1 -- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc0ceec0 msgr2=0x7f67fc158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f6801200700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc0ceec0 0x7f67fc158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f6801200700 1 -- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67fc15d640 msgr2=0x7f67fc09d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f6801200700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67fc15d640 0x7f67fc09d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f6801200700 1 -- 172.21.15.12:0/2988171361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec0094a0 con 0x7f67fc1619d0 2024-03-29T11:28:45.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f6801200700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67fc1619d0 0x7f67fc15f790 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f67f400cbc0 tx=0x7f67f400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:45.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.490+0000 7f67f3e00700 1 -- 172.21.15.12:0/2988171361 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4003c00 con 0x7f67fc1619d0 2024-03-29T11:28:45.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.491+0000 7f67f3e00700 1 -- 172.21.15.12:0/2988171361 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f4003d60 con 0x7f67fc1619d0 2024-03-29T11:28:45.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.491+0000 7f67f3e00700 1 -- 172.21.15.12:0/2988171361 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f400ab90 con 0x7f67fc1619d0 2024-03-29T11:28:45.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.491+0000 7f6803200700 1 -- 172.21.15.12:0/2988171361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67fc09e870 con 0x7f67fc1619d0 2024-03-29T11:28:45.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.491+0000 7f6803200700 1 -- 172.21.15.12:0/2988171361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67fc15c020 con 0x7f67fc1619d0 2024-03-29T11:28:45.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.492+0000 7f6803200700 1 -- 172.21.15.12:0/2988171361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67fc15c240 con 0x7f67fc1619d0 2024-03-29T11:28:45.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.493+0000 7f67f3e00700 1 -- 172.21.15.12:0/2988171361 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f67f400acf0 con 0x7f67fc1619d0 2024-03-29T11:28:45.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.494+0000 7f67f3e00700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67dc06adb0 0x7f67dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:45.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.494+0000 7f67f3e00700 1 -- 172.21.15.12:0/2988171361 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f67f40892f0 con 0x7f67fc1619d0 2024-03-29T11:28:45.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.494+0000 7f67fbe00700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67dc06adb0 0x7f67dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:45.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.495+0000 7f67fbe00700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67dc06adb0 0x7f67dc06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f67ec000c00 tx=0x7f67ec01c040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:45.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.499+0000 7f67f3e00700 1 -- 172.21.15.12:0/2988171361 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67fc15c240 con 0x7f67fc1619d0 2024-03-29T11:28:45.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.797+0000 7f6803200700 1 -- 172.21.15.12:0/2988171361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f67fc15c240 con 0x7f67fc1619d0 2024-03-29T11:28:45.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.798+0000 7f67f3e00700 1 -- 172.21.15.12:0/2988171361 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v47) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f67fc15c240 con 0x7f67fc1619d0 2024-03-29T11:28:45.788 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:45.788 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:28:45.788 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T11:28:45.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.788 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.789 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:28:45.790 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.792 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T11:28:45.792 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T11:28:45.792 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.792 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:28:45.792 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:28:45.792 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T11:28:45.793 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:28:45.794 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.795 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T11:28:45.796 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T11:28:45.796 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.796 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:28:45.796 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:28:45.796 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.796 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.796 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.797 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T11:28:45.798 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.799 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:28:45.800 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.801 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T11:28:45.802 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:28:45.803 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:28:45.803 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:28:45.803 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:28:45.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:45.803 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T11:28:45.803 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:28:45.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 -- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67dc06adb0 msgr2=0x7f67dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:45.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67dc06adb0 0x7f67dc06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f67ec000c00 tx=0x7f67ec01c040 comp rx=0 tx=0).stop 2024-03-29T11:28:45.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 -- 172.21.15.12:0/2988171361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67fc1619d0 msgr2=0x7f67fc15f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:45.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67fc1619d0 0x7f67fc15f790 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f67f400cbc0 tx=0x7f67f400cf80 comp rx=0 tx=0).stop 2024-03-29T11:28:45.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 -- 172.21.15.12:0/2988171361 shutdown_connections 2024-03-29T11:28:45.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67dc06adb0 0x7f67dc06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67fc1619d0 0x7f67fc15f790 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67fc0ceec0 0x7f67fc158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 --2- 172.21.15.12:0/2988171361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67fc15d640 0x7f67fc09d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:45.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 -- 172.21.15.12:0/2988171361 >> 172.21.15.12:0/2988171361 conn(0x7f67fc08aff0 msgr2=0x7f67fc06d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:45.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 -- 172.21.15.12:0/2988171361 shutdown_connections 2024-03-29T11:28:45.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:45.800+0000 7f6803200700 1 -- 172.21.15.12:0/2988171361 wait complete. 2024-03-29T11:28:45.809 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.017+0000 7f5f84c00700 1 Processor -- start 2024-03-29T11:28:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.017+0000 7f5f84c00700 1 -- start start 2024-03-29T11:28:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.018+0000 7f5f84c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5f80160f10 0x7f5f800d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.018+0000 7f5f84c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5f800cf230 0x7f5f800d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.018+0000 7f5f84c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80161590 0x7f5f80161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.018+0000 7f5f84c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f80161f60 con 0x7f5f80160f10 2024-03-29T11:28:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.018+0000 7f5f84c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f800ceda0 con 0x7f5f800cf230 2024-03-29T11:28:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f84c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f800d23d0 con 0x7f5f80161590 2024-03-29T11:28:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5f80160f10 0x7f5f800d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5f80160f10 0x7f5f800d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44568/0 (socket says 172.21.15.12:44568) 2024-03-29T11:28:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7de00700 1 -- 172.21.15.12:0/2492213925 learned_addr learned my addr 172.21.15.12:0/2492213925 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7e800700 1 --2- 172.21.15.12:0/2492213925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80161590 0x7f5f80161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7d400700 1 --2- 172.21.15.12:0/2492213925 >> v2:172.21.15.159:3300/0 conn(0x7f5f800cf230 0x7f5f800d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7de00700 1 -- 172.21.15.12:0/2492213925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80161590 msgr2=0x7f5f80161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7de00700 1 --2- 172.21.15.12:0/2492213925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80161590 0x7f5f80161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7de00700 1 -- 172.21.15.12:0/2492213925 >> v2:172.21.15.159:3300/0 conn(0x7f5f800cf230 msgr2=0x7f5f800d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7de00700 1 --2- 172.21.15.12:0/2492213925 >> v2:172.21.15.159:3300/0 conn(0x7f5f800cf230 0x7f5f800d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7de00700 1 -- 172.21.15.12:0/2492213925 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f801612f0 con 0x7f5f80160f10 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7e800700 1 --2- 172.21.15.12:0/2492213925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80161590 0x7f5f80161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.019+0000 7f5f7d400700 1 --2- 172.21.15.12:0/2492213925 >> v2:172.21.15.159:3300/0 conn(0x7f5f800cf230 0x7f5f800d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f7de00700 1 --2- 172.21.15.12:0/2492213925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f80160f10 0x7f5f800d18b0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f5f8008cdb0 tx=0x7f5f70009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72db8bcf8c066a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f7ca00700 1 -- 172.21.15.12:0/2492213925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000b870 con 0x7f5f80160f10 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f7ca00700 1 -- 172.21.15.12:0/2492213925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7000b9d0 con 0x7f5f80160f10 2024-03-29T11:28:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f7ca00700 1 -- 172.21.15.12:0/2492213925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000bd60 con 0x7f5f80160f10 2024-03-29T11:28:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f84c00700 1 -- 172.21.15.12:0/2492213925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f80160f10 msgr2=0x7f5f800d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f84c00700 1 --2- 172.21.15.12:0/2492213925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f80160f10 0x7f5f800d18b0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f5f8008cdb0 tx=0x7f5f70009bd0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f84c00700 1 -- 172.21.15.12:0/2492213925 shutdown_connections 2024-03-29T11:28:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f84c00700 1 --2- 172.21.15.12:0/2492213925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80161590 0x7f5f80161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f84c00700 1 --2- 172.21.15.12:0/2492213925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f80160f10 0x7f5f800d18b0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f84c00700 1 --2- 172.21.15.12:0/2492213925 >> v2:172.21.15.159:3300/0 conn(0x7f5f800cf230 0x7f5f800d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.020+0000 7f5f84c00700 1 -- 172.21.15.12:0/2492213925 >> 172.21.15.12:0/2492213925 conn(0x7f5f8008aff0 msgr2=0x7f5f800d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.021+0000 7f5f84c00700 1 -- 172.21.15.12:0/2492213925 shutdown_connections 2024-03-29T11:28:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.021+0000 7f5f84c00700 1 -- 172.21.15.12:0/2492213925 wait complete. 2024-03-29T11:28:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.022+0000 7f5f84c00700 1 Processor -- start 2024-03-29T11:28:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.022+0000 7f5f84c00700 1 -- start start 2024-03-29T11:28:46.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.023+0000 7f5f84c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f800cf230 0x7f5f8015d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.023+0000 7f5f84c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80160f10 0x7f5f8015f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.023+0000 7f5f84c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f80161590 0x7f5f8009e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.023+0000 7f5f84c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8015d590 con 0x7f5f800cf230 2024-03-29T11:28:46.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.023+0000 7f5f84c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8015d860 con 0x7f5f80161590 2024-03-29T11:28:46.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.023+0000 7f5f84c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8015db40 con 0x7f5f80160f10 2024-03-29T11:28:46.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f800cf230 0x7f5f8015d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f800cf230 0x7f5f8015d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44574/0 (socket says 172.21.15.12:44574) 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7de00700 1 -- 172.21.15.12:0/1711232283 learned_addr learned my addr 172.21.15.12:0/1711232283 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7e800700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f80161590 0x7f5f8009e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7d400700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80160f10 0x7f5f8015f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7de00700 1 -- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80160f10 msgr2=0x7f5f8015f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7de00700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80160f10 0x7f5f8015f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7de00700 1 -- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f80161590 msgr2=0x7f5f8009e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7de00700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f80161590 0x7f5f8009e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7de00700 1 -- 172.21.15.12:0/1711232283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f70009550 con 0x7f5f800cf230 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7e800700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f80161590 0x7f5f8009e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7d400700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80160f10 0x7f5f8015f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.024+0000 7f5f7de00700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f800cf230 0x7f5f8015d050 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f5f70014040 tx=0x7f5f70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.025+0000 7f5f76000700 1 -- 172.21.15.12:0/1711232283 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70004520 con 0x7f5f800cf230 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.025+0000 7f5f76000700 1 -- 172.21.15.12:0/1711232283 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7001e070 con 0x7f5f800cf230 2024-03-29T11:28:46.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.025+0000 7f5f84c00700 1 -- 172.21.15.12:0/1711232283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8015de20 con 0x7f5f800cf230 2024-03-29T11:28:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.025+0000 7f5f76000700 1 -- 172.21.15.12:0/1711232283 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f700038e0 con 0x7f5f800cf230 2024-03-29T11:28:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.025+0000 7f5f84c00700 1 -- 172.21.15.12:0/1711232283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8015e3e0 con 0x7f5f800cf230 2024-03-29T11:28:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.026+0000 7f5f84c00700 1 -- 172.21.15.12:0/1711232283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f80067720 con 0x7f5f800cf230 2024-03-29T11:28:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.027+0000 7f5f76000700 1 -- 172.21.15.12:0/1711232283 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5f70003a40 con 0x7f5f800cf230 2024-03-29T11:28:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.028+0000 7f5f76000700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5f6006adb0 0x7f5f6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.028+0000 7f5f76000700 1 -- 172.21.15.12:0/1711232283 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f5f70020070 con 0x7f5f800cf230 2024-03-29T11:28:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.028+0000 7f5f7d400700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5f6006adb0 0x7f5f6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.029+0000 7f5f7d400700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5f6006adb0 0x7f5f6006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5f800ca070 tx=0x7f5f6c008040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.032+0000 7f5f76000700 1 -- 172.21.15.12:0/1711232283 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5f70055c30 con 0x7f5f800cf230 2024-03-29T11:28:46.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.247+0000 7f5f84c00700 1 -- 172.21.15.12:0/1711232283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f80161d20 con 0x7f5f800cf230 2024-03-29T11:28:46.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.248+0000 7f5f76000700 1 -- 172.21.15.12:0/1711232283 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+12515 (secure 0 0 0) 0x7f5f70015090 con 0x7f5f800cf230 2024-03-29T11:28:46.238 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:46.238 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":204,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:28:45.247254+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:46.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 -- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5f6006adb0 msgr2=0x7f5f6006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5f6006adb0 0x7f5f6006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5f800ca070 tx=0x7f5f6c008040 comp rx=0 tx=0).stop 2024-03-29T11:28:46.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 -- 172.21.15.12:0/1711232283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f800cf230 msgr2=0x7f5f8015d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f800cf230 0x7f5f8015d050 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f5f70014040 tx=0x7f5f70004750 comp rx=0 tx=0).stop 2024-03-29T11:28:46.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 -- 172.21.15.12:0/1711232283 shutdown_connections 2024-03-29T11:28:46.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5f6006adb0 0x7f5f6006d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f80161590 0x7f5f8009e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f800cf230 0x7f5f8015d050 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 --2- 172.21.15.12:0/1711232283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f80160f10 0x7f5f8015f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 -- 172.21.15.12:0/1711232283 >> 172.21.15.12:0/1711232283 conn(0x7f5f8008aff0 msgr2=0x7f5f8015bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:46.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 -- 172.21.15.12:0/1711232283 shutdown_connections 2024-03-29T11:28:46.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.250+0000 7f5f84c00700 1 -- 172.21.15.12:0/1711232283 wait complete. 2024-03-29T11:28:46.259 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:28:46.259 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:28:46.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.469+0000 7fa5d3600700 1 Processor -- start 2024-03-29T11:28:46.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.470+0000 7fa5d3600700 1 -- start start 2024-03-29T11:28:46.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.471+0000 7fa5d3600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc0cee10 0x7fa5cc161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.471+0000 7fa5d3600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc0ce270 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.471+0000 7fa5d3600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc15b600 con 0x7fa5cc1611b0 2024-03-29T11:28:46.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.471+0000 7fa5d3600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0d2760 con 0x7fa5cc0cee10 2024-03-29T11:28:46.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.471+0000 7fa5cbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa5cc0d21f0 0x7fa5cc0ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54040/0 (socket says 172.21.15.12:54040) 2024-03-29T11:28:46.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.471+0000 7fa5cbe00700 1 -- 172.21.15.12:0/3354503731 learned_addr learned my addr 172.21.15.12:0/3354503731 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:46.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.471+0000 7fa5d1600700 1 --2- 172.21.15.12:0/3354503731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc0cee10 0x7fa5cc161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.471+0000 7fa5cb400700 1 -- 172.21.15.12:0/3354503731 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951558656 0 0) 0x7fa5cc0ce270 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.471+0000 7fa5cb400700 1 -- 172.21.15.12:0/3354503731 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b40035b0 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.472+0000 7fa5cb400700 1 -- 172.21.15.12:0/3354503731 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569505036 0 0) 0x7fa5cc15b600 con 0x7fa5cc1611b0 2024-03-29T11:28:46.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.472+0000 7fa5cb400700 1 -- 172.21.15.12:0/3354503731 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5cc0ce270 con 0x7fa5cc1611b0 2024-03-29T11:28:46.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.472+0000 7fa5cb400700 1 -- 172.21.15.12:0/3354503731 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 763337354 0 0) 0x7fa5b40035b0 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.472+0000 7fa5d1600700 1 -- 172.21.15.12:0/3354503731 >> v1:172.21.15.159:6789/0 conn(0x7fa5cc1611b0 legacy=0x7fa5cc15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.472+0000 7fa5d1600700 1 -- 172.21.15.12:0/3354503731 >> v1:172.21.15.12:6789/0 conn(0x7fa5cc0d21f0 legacy=0x7fa5cc0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.472+0000 7fa5d1600700 1 -- 172.21.15.12:0/3354503731 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0d2a40 con 0x7fa5cc0cee10 2024-03-29T11:28:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.472+0000 7fa5d1600700 1 --2- 172.21.15.12:0/3354503731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc0cee10 0x7fa5cc161590 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009dc0 tx=0x7fa5c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f55a0c9e0083a2f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.473+0000 7fa5cb400700 1 -- 172.21.15.12:0/3354503731 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c400e070 con 0x7fa5cc0cee10 2024-03-29T11:28:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.473+0000 7fa5cb400700 1 -- 172.21.15.12:0/3354503731 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c4003a20 con 0x7fa5cc0cee10 2024-03-29T11:28:46.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.473+0000 7fa5cb400700 1 -- 172.21.15.12:0/3354503731 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8001670 con 0x7fa5cc0cee10 2024-03-29T11:28:46.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.473+0000 7fa5d3600700 1 -- 172.21.15.12:0/3354503731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc0cee10 msgr2=0x7fa5cc161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.473+0000 7fa5d3600700 1 --2- 172.21.15.12:0/3354503731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc0cee10 0x7fa5cc161590 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009dc0 tx=0x7fa5c400b330 comp rx=0 tx=0).stop 2024-03-29T11:28:46.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.473+0000 7fa5d3600700 1 -- 172.21.15.12:0/3354503731 shutdown_connections 2024-03-29T11:28:46.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.473+0000 7fa5d3600700 1 --2- 172.21.15.12:0/3354503731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc0cee10 0x7fa5cc161590 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.473+0000 7fa5d3600700 1 -- 172.21.15.12:0/3354503731 >> 172.21.15.12:0/3354503731 conn(0x7fa5cc08aff0 msgr2=0x7fa5cc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:46.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.473+0000 7fa5d3600700 1 -- 172.21.15.12:0/3354503731 shutdown_connections 2024-03-29T11:28:46.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.473+0000 7fa5d3600700 1 -- 172.21.15.12:0/3354503731 wait complete. 2024-03-29T11:28:46.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.475+0000 7fa5d3600700 1 Processor -- start 2024-03-29T11:28:46.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.475+0000 7fa5d3600700 1 -- start start 2024-03-29T11:28:46.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.476+0000 7fa5d3600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5cc0cee10 0x7fa5cc0c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.476+0000 7fa5d3600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5cc0d21f0 0x7fa5cc15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.476+0000 7fa5d3600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc1611b0 0x7fa5cc0cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.476+0000 7fa5d3600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0cd0e0 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.476+0000 7fa5d3600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0c8b80 con 0x7fa5cc0cee10 2024-03-29T11:28:46.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.476+0000 7fa5d3600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0c9930 con 0x7fa5cc1611b0 2024-03-29T11:28:46.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.476+0000 7fa5cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5cc0d21f0 0x7fa5cc15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.476+0000 7fa5cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5cc0d21f0 0x7fa5cc15aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44586/0 (socket says 172.21.15.12:44586) 2024-03-29T11:28:46.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.476+0000 7fa5cbe00700 1 -- 172.21.15.12:0/954072838 learned_addr learned my addr 172.21.15.12:0/954072838 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:46.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.476+0000 7fa5d1600700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc1611b0 0x7fa5cc0cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5d0c00700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5cc0cee10 0x7fa5cc0c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5cbe00700 1 -- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc1611b0 msgr2=0x7fa5cc0cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5cbe00700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc1611b0 0x7fa5cc0cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5cbe00700 1 -- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5cc0cee10 msgr2=0x7fa5cc0c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5cbe00700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5cc0cee10 0x7fa5cc0c8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5cbe00700 1 -- 172.21.15.12:0/954072838 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c4009a70 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5d0c00700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5cc0cee10 0x7fa5cc0c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5cbe00700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5cc0d21f0 0x7fa5cc15aa20 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c750 tx=0x7fa5b800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5c8c00700 1 -- 172.21.15.12:0/954072838 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8004c80 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5c8c00700 1 -- 172.21.15.12:0/954072838 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8004e00 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5d3600700 1 -- 172.21.15.12:0/954072838 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc0c8db0 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5c8c00700 1 -- 172.21.15.12:0/954072838 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8007310 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.477+0000 7fa5d3600700 1 -- 172.21.15.12:0/954072838 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc0c9370 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.478+0000 7fa5d3600700 1 -- 172.21.15.12:0/954072838 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc1f3660 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.479+0000 7fa5c8c00700 1 -- 172.21.15.12:0/954072838 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa5b8007470 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.480+0000 7fa5c8c00700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5ac06adb0 0x7fa5ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.480+0000 7fa5c8c00700 1 -- 172.21.15.12:0/954072838 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fa5b8088f30 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.481+0000 7fa5d0c00700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5ac06adb0 0x7fa5ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.481+0000 7fa5d0c00700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5ac06adb0 0x7fa5ac06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0001710 tx=0x7fa5c0009040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:46.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.485+0000 7fa5c8c00700 1 -- 172.21.15.12:0/954072838 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa5cc1f3660 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-29T11:28:46.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.698+0000 7fa5d3600700 1 -- 172.21.15.12:0/954072838 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5cc067720 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.698+0000 7fa5c8c00700 1 -- 172.21.15.12:0/954072838 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v204) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5b80542f0 con 0x7fa5cc0d21f0 2024-03-29T11:28:46.689 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:28:46.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 -- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5ac06adb0 msgr2=0x7fa5ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5ac06adb0 0x7fa5ac06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0001710 tx=0x7fa5c0009040 comp rx=0 tx=0).stop 2024-03-29T11:28:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 -- 172.21.15.12:0/954072838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5cc0d21f0 msgr2=0x7fa5cc15aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5cc0d21f0 0x7fa5cc15aa20 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c750 tx=0x7fa5b800ca60 comp rx=0 tx=0).stop 2024-03-29T11:28:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 -- 172.21.15.12:0/954072838 shutdown_connections 2024-03-29T11:28:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5ac06adb0 0x7fa5ac06d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5cc1611b0 0x7fa5cc0cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5cc0cee10 0x7fa5cc0c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 --2- 172.21.15.12:0/954072838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5cc0d21f0 0x7fa5cc15aa20 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 -- 172.21.15.12:0/954072838 >> 172.21.15.12:0/954072838 conn(0x7fa5cc08aff0 msgr2=0x7fa5cc0ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 -- 172.21.15.12:0/954072838 shutdown_connections 2024-03-29T11:28:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.701+0000 7fa5d3600700 1 -- 172.21.15.12:0/954072838 wait complete. 2024-03-29T11:28:46.710 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T11:28:46.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.918+0000 7ffa19400700 1 Processor -- start 2024-03-29T11:28:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.918+0000 7ffa19400700 1 -- start start 2024-03-29T11:28:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.919+0000 7ffa19400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffa140ac580 0x7ffa140b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.919+0000 7ffa19400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 0x7ffa1406dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.919+0000 7ffa19400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140b3890 con 0x7ffa1415d670 2024-03-29T11:28:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa19400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa14159f50 con 0x7ffa140ac580 2024-03-29T11:28:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa19400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140b35b0 con 0x7ffa140b3000 2024-03-29T11:28:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa11e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffa140ac580 0x7ffa140b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa12800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ffa1415d670 0x7ffa1415da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54046/0 (socket says 172.21.15.12:54046) 2024-03-29T11:28:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa12800700 1 -- 172.21.15.12:0/3493153885 learned_addr learned my addr 172.21.15.12:0/3493153885 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa13200700 1 --2- 172.21.15.12:0/3493153885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 0x7ffa1406dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa11e00700 1 -- 172.21.15.12:0/3493153885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 msgr2=0x7ffa1406dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa11e00700 1 --2- 172.21.15.12:0/3493153885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 0x7ffa1406dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa11e00700 1 -- 172.21.15.12:0/3493153885 >> v1:172.21.15.12:6789/0 conn(0x7ffa1415d670 legacy=0x7ffa1415da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa11e00700 1 -- 172.21.15.12:0/3493153885 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1406d400 con 0x7ffa140ac580 2024-03-29T11:28:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa13200700 1 --2- 172.21.15.12:0/3493153885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 0x7ffa1406dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.920+0000 7ffa11400700 1 -- 172.21.15.12:0/3493153885 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432952550 0 0) 0x7ffa140b3890 con 0x7ffa1415d670 2024-03-29T11:28:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa11e00700 1 --2- 172.21.15.12:0/3493153885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa140ac580 0x7ffa140b2ac0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009840 tx=0x7ffa00009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a6f2277e2ef84f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa11400700 1 -- 172.21.15.12:0/3493153885 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00011830 con 0x7ffa140ac580 2024-03-29T11:28:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa11400700 1 -- 172.21.15.12:0/3493153885 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa00011990 con 0x7ffa140ac580 2024-03-29T11:28:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa11400700 1 -- 172.21.15.12:0/3493153885 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00011d20 con 0x7ffa140ac580 2024-03-29T11:28:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa19400700 1 -- 172.21.15.12:0/3493153885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa140ac580 msgr2=0x7ffa140b2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa19400700 1 --2- 172.21.15.12:0/3493153885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa140ac580 0x7ffa140b2ac0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009840 tx=0x7ffa00009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa19400700 1 -- 172.21.15.12:0/3493153885 shutdown_connections 2024-03-29T11:28:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa19400700 1 --2- 172.21.15.12:0/3493153885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 0x7ffa1406dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa19400700 1 --2- 172.21.15.12:0/3493153885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa140ac580 0x7ffa140b2ac0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa19400700 1 -- 172.21.15.12:0/3493153885 >> 172.21.15.12:0/3493153885 conn(0x7ffa140cf190 msgr2=0x7ffa140cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.921+0000 7ffa19400700 1 -- 172.21.15.12:0/3493153885 shutdown_connections 2024-03-29T11:28:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.922+0000 7ffa19400700 1 -- 172.21.15.12:0/3493153885 wait complete. 2024-03-29T11:28:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.923+0000 7ffa19400700 1 Processor -- start 2024-03-29T11:28:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.923+0000 7ffa19400700 1 -- start start 2024-03-29T11:28:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.924+0000 7ffa19400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa140ac580 0x7ffa140ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.924+0000 7ffa19400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 0x7ffa140ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.924+0000 7ffa19400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa1415d670 0x7ffa141f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.924+0000 7ffa19400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140c8e70 con 0x7ffa1415d670 2024-03-29T11:28:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.924+0000 7ffa19400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140c9cd0 con 0x7ffa140ac580 2024-03-29T11:28:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.924+0000 7ffa19400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140c90c0 con 0x7ffa140b3000 2024-03-29T11:28:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa13200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa1415d670 0x7ffa141f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa13200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa1415d670 0x7ffa141f3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44602/0 (socket says 172.21.15.12:44602) 2024-03-29T11:28:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa13200700 1 -- 172.21.15.12:0/1422862304 learned_addr learned my addr 172.21.15.12:0/1422862304 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa12800700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa140ac580 0x7ffa140ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa11e00700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 0x7ffa140ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa13200700 1 -- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 msgr2=0x7ffa140ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa13200700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 0x7ffa140ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa13200700 1 -- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa140ac580 msgr2=0x7ffa140ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa13200700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa140ac580 0x7ffa140ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa13200700 1 -- 172.21.15.12:0/1422862304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa000094a0 con 0x7ffa1415d670 2024-03-29T11:28:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.925+0000 7ffa13200700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa1415d670 0x7ffa141f3060 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c007de0 tx=0x7ffa0c00d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.926+0000 7ffa0aa00700 1 -- 172.21.15.12:0/1422862304 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c00f040 con 0x7ffa1415d670 2024-03-29T11:28:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.926+0000 7ffa0aa00700 1 -- 172.21.15.12:0/1422862304 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0c003c20 con 0x7ffa1415d670 2024-03-29T11:28:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.926+0000 7ffa19400700 1 -- 172.21.15.12:0/1422862304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa140c93a0 con 0x7ffa1415d670 2024-03-29T11:28:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.926+0000 7ffa0aa00700 1 -- 172.21.15.12:0/1422862304 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c00a2e0 con 0x7ffa1415d670 2024-03-29T11:28:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.926+0000 7ffa19400700 1 -- 172.21.15.12:0/1422862304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa140c9960 con 0x7ffa1415d670 2024-03-29T11:28:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.927+0000 7ffa19400700 1 -- 172.21.15.12:0/1422862304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa140c9e60 con 0x7ffa1415d670 2024-03-29T11:28:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.928+0000 7ffa0aa00700 1 -- 172.21.15.12:0/1422862304 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ffa0c00d900 con 0x7ffa1415d670 2024-03-29T11:28:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.929+0000 7ffa0aa00700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9f406ade0 0x7ff9f406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.929+0000 7ffa0aa00700 1 -- 172.21.15.12:0/1422862304 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 4929+0+0 (secure 0 0 0) 0x7ffa0c0892d0 con 0x7ffa1415d670 2024-03-29T11:28:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.929+0000 7ffa12800700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9f406ade0 0x7ff9f406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.930+0000 7ffa12800700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9f406ade0 0x7ff9f406d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffa04000f60 tx=0x7ffa04007e70 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:46.934+0000 7ffa0aa00700 1 -- 172.21.15.12:0/1422862304 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa0c054580 con 0x7ffa1415d670 2024-03-29T11:28:47.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.149+0000 7ffa19400700 1 -- 172.21.15.12:0/1422862304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ffa14067720 con 0x7ffa1415d670 2024-03-29T11:28:47.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.149+0000 7ffa0aa00700 1 -- 172.21.15.12:0/1422862304 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v204) v1 ==== 84+0+65 (secure 0 0 0) 0x7ffa0c01a300 con 0x7ffa1415d670 2024-03-29T11:28:47.140 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:47.140 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:28:47.140 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:47.140 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T11:28:47.140 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T11:28:47.140 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:47.140 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:28:47.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.151+0000 7ffa19400700 1 -- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9f406ade0 msgr2=0x7ff9f406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9f406ade0 0x7ff9f406d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffa04000f60 tx=0x7ffa04007e70 comp rx=0 tx=0).stop 2024-03-29T11:28:47.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 -- 172.21.15.12:0/1422862304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa1415d670 msgr2=0x7ffa141f3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa1415d670 0x7ffa141f3060 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c007de0 tx=0x7ffa0c00d3a0 comp rx=0 tx=0).stop 2024-03-29T11:28:47.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 -- 172.21.15.12:0/1422862304 shutdown_connections 2024-03-29T11:28:47.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9f406ade0 0x7ff9f406d2a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:47.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa1415d670 0x7ffa141f3060 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:47.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa140ac580 0x7ffa140ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:47.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 --2- 172.21.15.12:0/1422862304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa140b3000 0x7ffa140ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:47.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 -- 172.21.15.12:0/1422862304 >> 172.21.15.12:0/1422862304 conn(0x7ffa140cf190 msgr2=0x7ffa14072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:47.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 -- 172.21.15.12:0/1422862304 shutdown_connections 2024-03-29T11:28:47.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.152+0000 7ffa19400700 1 -- 172.21.15.12:0/1422862304 wait complete. 2024-03-29T11:28:47.160 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T11:28:47.161 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T11:28:47.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.374+0000 7fd8f2e00700 1 Processor -- start 2024-03-29T11:28:47.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.374+0000 7fd8f2e00700 1 -- start start 2024-03-29T11:28:47.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.375+0000 7fd8f2e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:47.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.375+0000 7fd8f2e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd8ec15e1d0 0x7fd8ec0d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:47.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.375+0000 7fd8f2e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec0ce8a0 con 0x7fd8ec0d2710 2024-03-29T11:28:47.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.375+0000 7fd8f2e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec0d2b80 con 0x7fd8ec15e1d0 2024-03-29T11:28:47.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.375+0000 7fd8f2e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec15ec60 con 0x7fd8ec15ddf0 2024-03-29T11:28:47.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.376+0000 7fd8ebe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:47.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.376+0000 7fd8eb400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd8ec15e1d0 0x7fd8ec0d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:47.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.376+0000 7fd8ebe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec0ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:57852/0 (socket says 172.21.15.12:57852) 2024-03-29T11:28:47.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.376+0000 7fd8f0e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fd8ec0d2710 0x7fd8ec1612f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:51642/0 (socket says 172.21.15.12:51642) 2024-03-29T11:28:47.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.376+0000 7fd8ebe00700 1 -- 172.21.15.12:0/4034727687 learned_addr learned my addr 172.21.15.12:0/4034727687 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.376+0000 7fd8ebe00700 1 -- 172.21.15.12:0/4034727687 >> v2:172.21.15.159:3300/0 conn(0x7fd8ec15e1d0 msgr2=0x7fd8ec0d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.376+0000 7fd8ebe00700 1 --2- 172.21.15.12:0/4034727687 >> v2:172.21.15.159:3300/0 conn(0x7fd8ec15e1d0 0x7fd8ec0d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.376+0000 7fd8ebe00700 1 -- 172.21.15.12:0/4034727687 >> v1:172.21.15.159:6789/0 conn(0x7fd8ec0d2710 legacy=0x7fd8ec1612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.376+0000 7fd8ebe00700 1 -- 172.21.15.12:0/4034727687 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec0cdc90 con 0x7fd8ec15ddf0 2024-03-29T11:28:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.376+0000 7fd8eaa00700 1 -- 172.21.15.12:0/4034727687 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562613897 0 0) 0x7fd8ec0ce8a0 con 0x7fd8ec0d2710 2024-03-29T11:28:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8ebe00700 1 --2- 172.21.15.12:0/4034727687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec0ced30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0098f0 tx=0x7fd8dc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95974789a7586f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8eaa00700 1 -- 172.21.15.12:0/4034727687 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc011830 con 0x7fd8ec15ddf0 2024-03-29T11:28:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8eaa00700 1 -- 172.21.15.12:0/4034727687 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc011990 con 0x7fd8ec15ddf0 2024-03-29T11:28:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8eaa00700 1 -- 172.21.15.12:0/4034727687 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc011d20 con 0x7fd8ec15ddf0 2024-03-29T11:28:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8f2e00700 1 -- 172.21.15.12:0/4034727687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 msgr2=0x7fd8ec0ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8f2e00700 1 --2- 172.21.15.12:0/4034727687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec0ced30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0098f0 tx=0x7fd8dc009c00 comp rx=0 tx=0).stop 2024-03-29T11:28:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8f2e00700 1 -- 172.21.15.12:0/4034727687 shutdown_connections 2024-03-29T11:28:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8f2e00700 1 --2- 172.21.15.12:0/4034727687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec0ced30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8f2e00700 1 --2- 172.21.15.12:0/4034727687 >> v2:172.21.15.159:3300/0 conn(0x7fd8ec15e1d0 0x7fd8ec0d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8f2e00700 1 -- 172.21.15.12:0/4034727687 >> 172.21.15.12:0/4034727687 conn(0x7fd8ec08aff0 msgr2=0x7fd8ec0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.377+0000 7fd8f2e00700 1 -- 172.21.15.12:0/4034727687 shutdown_connections 2024-03-29T11:28:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.378+0000 7fd8f2e00700 1 -- 172.21.15.12:0/4034727687 wait complete. 2024-03-29T11:28:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.379+0000 7fd8f2e00700 1 Processor -- start 2024-03-29T11:28:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.379+0000 7fd8f2e00700 1 -- start start 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.380+0000 7fd8f2e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8ec0d2710 0x7fd8ec148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.380+0000 7fd8f2e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec1e7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.380+0000 7fd8f2e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8ec15e1d0 0x7fd8ec1e7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.380+0000 7fd8f2e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec0c98d0 con 0x7fd8ec0d2710 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.380+0000 7fd8f2e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec149350 con 0x7fd8ec15e1d0 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.380+0000 7fd8f2e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec149630 con 0x7fd8ec15ddf0 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.380+0000 7fd8ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8ec0d2710 0x7fd8ec148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8ec0d2710 0x7fd8ec148a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44604/0 (socket says 172.21.15.12:44604) 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8ebe00700 1 -- 172.21.15.12:0/3831133858 learned_addr learned my addr 172.21.15.12:0/3831133858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8eb400700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec1e7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8f0e00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8ec15e1d0 0x7fd8ec1e7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8ebe00700 1 -- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 msgr2=0x7fd8ec1e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8ebe00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec1e7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8ebe00700 1 -- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8ec15e1d0 msgr2=0x7fd8ec1e7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8ebe00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8ec15e1d0 0x7fd8ec1e7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8ebe00700 1 -- 172.21.15.12:0/3831133858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc009550 con 0x7fd8ec0d2710 2024-03-29T11:28:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8eb400700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec1e7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8ebe00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8ec0d2710 0x7fd8ec148a90 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009c40 tx=0x7fd8dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8e3e00700 1 -- 172.21.15.12:0/3831133858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc004480 con 0x7fd8ec0d2710 2024-03-29T11:28:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.381+0000 7fd8e3e00700 1 -- 172.21.15.12:0/3831133858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc01f070 con 0x7fd8ec0d2710 2024-03-29T11:28:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.382+0000 7fd8f2e00700 1 -- 172.21.15.12:0/3831133858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec149910 con 0x7fd8ec0d2710 2024-03-29T11:28:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.382+0000 7fd8e3e00700 1 -- 172.21.15.12:0/3831133858 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc0038c0 con 0x7fd8ec0d2710 2024-03-29T11:28:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.382+0000 7fd8f2e00700 1 -- 172.21.15.12:0/3831133858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec148fd0 con 0x7fd8ec0d2710 2024-03-29T11:28:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.383+0000 7fd8f2e00700 1 -- 172.21.15.12:0/3831133858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ec067720 con 0x7fd8ec0d2710 2024-03-29T11:28:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.384+0000 7fd8e3e00700 1 -- 172.21.15.12:0/3831133858 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd8dc021030 con 0x7fd8ec0d2710 2024-03-29T11:28:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.385+0000 7fd8e3e00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd8cc06adb0 0x7fd8cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.385+0000 7fd8e3e00700 1 -- 172.21.15.12:0/3831133858 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fd8dc08a690 con 0x7fd8ec0d2710 2024-03-29T11:28:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.385+0000 7fd8eb400700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd8cc06adb0 0x7fd8cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.386+0000 7fd8eb400700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd8cc06adb0 0x7fd8cc06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009b90 tx=0x7fd8d8008040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:47.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.389+0000 7fd8e3e00700 1 -- 172.21.15.12:0/3831133858 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8dc055b70 con 0x7fd8ec0d2710 2024-03-29T11:28:47.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:47.605+0000 7fd8f2e00700 1 -- 172.21.15.12:0/3831133858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd8ec0b5ce0 con 0x7fd8ec0d2710 2024-03-29T11:28:48.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.565+0000 7fd8e3e00700 1 -- 172.21.15.12:0/3831133858 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v205) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd8dc0150b0 con 0x7fd8ec0d2710 2024-03-29T11:28:48.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.571+0000 7fd8f2e00700 1 -- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd8cc06adb0 msgr2=0x7fd8cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.571+0000 7fd8f2e00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd8cc06adb0 0x7fd8cc06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009b90 tx=0x7fd8d8008040 comp rx=0 tx=0).stop 2024-03-29T11:28:48.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.571+0000 7fd8f2e00700 1 -- 172.21.15.12:0/3831133858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8ec0d2710 msgr2=0x7fd8ec148a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.571+0000 7fd8f2e00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8ec0d2710 0x7fd8ec148a90 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009c40 tx=0x7fd8dc004750 comp rx=0 tx=0).stop 2024-03-29T11:28:48.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.571+0000 7fd8f2e00700 1 -- 172.21.15.12:0/3831133858 shutdown_connections 2024-03-29T11:28:48.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.571+0000 7fd8f2e00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd8cc06adb0 0x7fd8cc06d270 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009b90 tx=0x7fd8d8008040 comp rx=0 tx=0).stop 2024-03-29T11:28:48.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.571+0000 7fd8f2e00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8ec15e1d0 0x7fd8ec1e7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:48.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.571+0000 7fd8f2e00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8ec0d2710 0x7fd8ec148a90 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:48.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.571+0000 7fd8f2e00700 1 --2- 172.21.15.12:0/3831133858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8ec15ddf0 0x7fd8ec1e7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:48.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.571+0000 7fd8f2e00700 1 -- 172.21.15.12:0/3831133858 >> 172.21.15.12:0/3831133858 conn(0x7fd8ec08aff0 msgr2=0x7fd8ec0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:48.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.572+0000 7fd8f2e00700 1 -- 172.21.15.12:0/3831133858 shutdown_connections 2024-03-29T11:28:48.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.572+0000 7fd8f2e00700 1 -- 172.21.15.12:0/3831133858 wait complete. 2024-03-29T11:28:48.565 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T11:28:48.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T11:28:48.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.804+0000 7f1248200700 1 Processor -- start 2024-03-29T11:28:48.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.805+0000 7f1248200700 1 -- start start 2024-03-29T11:28:48.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.806+0000 7f1248200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1240160d70 0x7f12400b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:48.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.806+0000 7f1248200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1240161150 con 0x7f124015ef70 2024-03-29T11:28:48.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.806+0000 7f1248200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124006dba0 con 0x7f124006a790 2024-03-29T11:28:48.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.806+0000 7f1248200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12400b4cd0 con 0x7f1240160d70 2024-03-29T11:28:48.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.806+0000 7f1245800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1240160d70 0x7f12400b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:48.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.806+0000 7f1244e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f124015ef70 0x7f124015e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54052/0 (socket says 172.21.15.12:54052) 2024-03-29T11:28:48.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.806+0000 7f1244e00700 1 -- 172.21.15.12:0/811819747 learned_addr learned my addr 172.21.15.12:0/811819747 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:48.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.807+0000 7f1245800700 1 -- 172.21.15.12:0/811819747 >> v1:172.21.15.159:6789/0 conn(0x7f124006a790 legacy=0x7f124006e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.807+0000 7f1245800700 1 -- 172.21.15.12:0/811819747 >> v1:172.21.15.12:6789/0 conn(0x7f124015ef70 legacy=0x7f124015e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.807+0000 7f1245800700 1 -- 172.21.15.12:0/811819747 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12400b4ee0 con 0x7f1240160d70 2024-03-29T11:28:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.807+0000 7f1245800700 1 --2- 172.21.15.12:0/811819747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1240160d70 0x7f12400b3200 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f1234009840 tx=0x7f1234009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b03b671d56fc260 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.807+0000 7f123fe00700 1 -- 172.21.15.12:0/811819747 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234011830 con 0x7f1240160d70 2024-03-29T11:28:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.807+0000 7f123fe00700 1 -- 172.21.15.12:0/811819747 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1234011990 con 0x7f1240160d70 2024-03-29T11:28:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.807+0000 7f123fe00700 1 -- 172.21.15.12:0/811819747 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234011d20 con 0x7f1240160d70 2024-03-29T11:28:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.808+0000 7f1248200700 1 -- 172.21.15.12:0/811819747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1240160d70 msgr2=0x7f12400b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.808+0000 7f1248200700 1 --2- 172.21.15.12:0/811819747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1240160d70 0x7f12400b3200 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f1234009840 tx=0x7f1234009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.808+0000 7f1248200700 1 -- 172.21.15.12:0/811819747 shutdown_connections 2024-03-29T11:28:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.808+0000 7f1248200700 1 --2- 172.21.15.12:0/811819747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1240160d70 0x7f12400b3200 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.808+0000 7f1248200700 1 -- 172.21.15.12:0/811819747 >> 172.21.15.12:0/811819747 conn(0x7f12400b5720 msgr2=0x7f12400b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.808+0000 7f1248200700 1 -- 172.21.15.12:0/811819747 shutdown_connections 2024-03-29T11:28:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.808+0000 7f1248200700 1 -- 172.21.15.12:0/811819747 wait complete. 2024-03-29T11:28:48.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.810+0000 7f1248200700 1 Processor -- start 2024-03-29T11:28:48.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.810+0000 7f1248200700 1 -- start start 2024-03-29T11:28:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1248200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f124006a790 0x7f12400ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1248200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f124015ef70 0x7f1240149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1248200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1240160d70 0x7f124014b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1248200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1240147bc0 con 0x7f1240160d70 2024-03-29T11:28:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1248200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1240148fe0 con 0x7f124015ef70 2024-03-29T11:28:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1248200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12400aed10 con 0x7f124006a790 2024-03-29T11:28:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1245800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f124006a790 0x7f12400ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1246200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1240160d70 0x7f124014b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:48.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1245800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f124006a790 0x7f12400ae7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:57868/0 (socket says 172.21.15.12:57868) 2024-03-29T11:28:48.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1245800700 1 -- 172.21.15.12:0/3737953077 learned_addr learned my addr 172.21.15.12:0/3737953077 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:48.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.811+0000 7f1244e00700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f124015ef70 0x7f1240149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:48.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.812+0000 7f1245800700 1 -- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f124015ef70 msgr2=0x7f1240149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.812+0000 7f1245800700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f124015ef70 0x7f1240149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:48.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.812+0000 7f1245800700 1 -- 172.21.15.12:0/3737953077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1240160d70 msgr2=0x7f124014b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.812+0000 7f1245800700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1240160d70 0x7f124014b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:48.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.812+0000 7f1245800700 1 -- 172.21.15.12:0/3737953077 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12340094a0 con 0x7f124006a790 2024-03-29T11:28:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.812+0000 7f1245800700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f124006a790 0x7f12400ae7d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1234000c00 tx=0x7f1234004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.813+0000 7f123d600700 1 -- 172.21.15.12:0/3737953077 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12340036a0 con 0x7f124006a790 2024-03-29T11:28:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.813+0000 7f123d600700 1 -- 172.21.15.12:0/3737953077 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123401e070 con 0x7f124006a790 2024-03-29T11:28:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.813+0000 7f123d600700 1 -- 172.21.15.12:0/3737953077 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234003a40 con 0x7f124006a790 2024-03-29T11:28:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.813+0000 7f1248200700 1 -- 172.21.15.12:0/3737953077 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12401483d0 con 0x7f124006a790 2024-03-29T11:28:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.813+0000 7f1248200700 1 -- 172.21.15.12:0/3737953077 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1240148990 con 0x7f124006a790 2024-03-29T11:28:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.814+0000 7f1248200700 1 -- 172.21.15.12:0/3737953077 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1240067720 con 0x7f124006a790 2024-03-29T11:28:48.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.815+0000 7f123d600700 1 -- 172.21.15.12:0/3737953077 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1234003ba0 con 0x7f124006a790 2024-03-29T11:28:48.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.816+0000 7f123d600700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f122006adb0 0x7f122006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:48.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.816+0000 7f123d600700 1 -- 172.21.15.12:0/3737953077 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5299+0+0 (secure 0 0 0) 0x7f123408a730 con 0x7f124006a790 2024-03-29T11:28:48.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.816+0000 7f1244e00700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f122006adb0 0x7f122006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:48.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.817+0000 7f1244e00700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f122006adb0 0x7f122006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f12400c9630 tx=0x7f122c009040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:48.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:48.820+0000 7f123d600700 1 -- 172.21.15.12:0/3737953077 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1234055a00 con 0x7f124006a790 2024-03-29T11:28:49.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.037+0000 7f1248200700 1 -- 172.21.15.12:0/3737953077 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f124015cb90 con 0x7f124006a790 2024-03-29T11:28:49.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.575+0000 7f123d600700 1 -- 172.21.15.12:0/3737953077 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v206) v1 ==== 170+0+0 (secure 0 0 0) 0x7f12340200d0 con 0x7f124006a790 2024-03-29T11:28:49.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.583+0000 7f1248200700 1 -- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f122006adb0 msgr2=0x7f122006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.583+0000 7f1248200700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f122006adb0 0x7f122006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f12400c9630 tx=0x7f122c009040 comp rx=0 tx=0).stop 2024-03-29T11:28:49.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.583+0000 7f1248200700 1 -- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f124006a790 msgr2=0x7f12400ae7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.583+0000 7f1248200700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f124006a790 0x7f12400ae7d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1234000c00 tx=0x7f1234004750 comp rx=0 tx=0).stop 2024-03-29T11:28:49.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.584+0000 7f1248200700 1 -- 172.21.15.12:0/3737953077 shutdown_connections 2024-03-29T11:28:49.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.584+0000 7f1248200700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f122006adb0 0x7f122006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:49.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.584+0000 7f1248200700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1240160d70 0x7f124014b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:49.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.584+0000 7f1248200700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f124006a790 0x7f12400ae7d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:49.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.584+0000 7f1248200700 1 --2- 172.21.15.12:0/3737953077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f124015ef70 0x7f1240149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:49.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.584+0000 7f1248200700 1 -- 172.21.15.12:0/3737953077 >> 172.21.15.12:0/3737953077 conn(0x7f12400b5720 msgr2=0x7f124009c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:49.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.585+0000 7f1248200700 1 -- 172.21.15.12:0/3737953077 shutdown_connections 2024-03-29T11:28:49.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.586+0000 7f1248200700 1 -- 172.21.15.12:0/3737953077 wait complete. 2024-03-29T11:28:49.581 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T11:28:49.602 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T11:28:49.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.814+0000 7f0c32e00700 1 Processor -- start 2024-03-29T11:28:49.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.814+0000 7f0c32e00700 1 -- start start 2024-03-29T11:28:49.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.815+0000 7f0c32e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0c2c15d9d0 0x7f0c2c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:49.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.815+0000 7f0c32e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c2c0d2540 0x7f0c2c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:49.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.815+0000 7f0c32e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2c161420 con 0x7f0c2c0d16a0 2024-03-29T11:28:49.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.815+0000 7f0c32e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c2c15edc0 con 0x7f0c2c15d9d0 2024-03-29T11:28:49.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.815+0000 7f0c32e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c161700 con 0x7f0c2c0d2540 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.815+0000 7f0c2b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0c2c0d16a0 0x7f0c2c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54158/0 (socket says 172.21.15.12:54158) 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.815+0000 7f0c2b400700 1 -- 172.21.15.12:0/3259368305 learned_addr learned my addr 172.21.15.12:0/3259368305 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2be00700 1 --2- 172.21.15.12:0/3259368305 >> v2:172.21.15.159:3300/0 conn(0x7f0c2c15d9d0 0x7f0c2c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c30e00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c2c0d2540 0x7f0c2c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399193167 0 0) 0x7f0c2c161420 con 0x7f0c2c0d16a0 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c100035b0 con 0x7f0c2c0d16a0 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3663541315 0 0) 0x7f0c100035b0 con 0x7f0c2c0d16a0 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c2c0d2540 msgr2=0x7f0c2c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c2c0d2540 0x7f0c2c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 >> v2:172.21.15.159:3300/0 conn(0x7f0c2c15d9d0 msgr2=0x7f0c2c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 --2- 172.21.15.12:0/3259368305 >> v2:172.21.15.159:3300/0 conn(0x7f0c2c15d9d0 0x7f0c2c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:49.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c1619e0 con 0x7f0c2c0d16a0 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0c180017f0 con 0x7f0c2c0d16a0 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2be00700 1 --2- 172.21.15.12:0/3259368305 >> v2:172.21.15.159:3300/0 conn(0x7f0c2c15d9d0 0x7f0c2c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 >> v1:172.21.15.12:6789/0 conn(0x7f0c2c0d16a0 legacy=0x7f0c2c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c10003a00 0x7f0c10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c10004850 0x7f0c10004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c10005210 0x7f0c1000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c161700 con 0x7f0c10003a00 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1000bf50 con 0x7f0c10004850 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1000c090 con 0x7f0c10005210 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.816+0000 7f0c30e00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c2c0d2540 0x7f0c2c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.817+0000 7f0c2be00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c10003a00 0x7f0c10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.817+0000 7f0c2b400700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c10004850 0x7f0c10004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.817+0000 7f0c30e00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c10005210 0x7f0c1000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.817+0000 7f0c2be00700 1 -- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c10005210 msgr2=0x7f0c1000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.817+0000 7f0c2be00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c10005210 0x7f0c1000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.817+0000 7f0c2be00700 1 -- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c10004850 msgr2=0x7f0c10004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.817+0000 7f0c2be00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c10004850 0x7f0c10004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.817+0000 7f0c2be00700 1 -- 172.21.15.12:0/3259368305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c100035b0 con 0x7f0c10003a00 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.817+0000 7f0c2be00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c10003a00 0x7f0c10003e20 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009d60 tx=0x7f0c10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39973437059e32 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.818+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c00e9a0 con 0x7f0c10003a00 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.818+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1c00eb00 con 0x7f0c10003a00 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.818+0000 7f0c2aa00700 1 -- 172.21.15.12:0/3259368305 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c00eeb0 con 0x7f0c10003a00 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.818+0000 7f0c32e00700 1 -- 172.21.15.12:0/3259368305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c10003a00 msgr2=0x7f0c10003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.818+0000 7f0c32e00700 1 --2- 172.21.15.12:0/3259368305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c10003a00 0x7f0c10003e20 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009d60 tx=0x7f0c10005880 comp rx=0 tx=0).stop 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.818+0000 7f0c30e00700 1 -- 172.21.15.12:0/3259368305 reap_dead start 2024-03-29T11:28:49.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.818+0000 7f0c32e00700 1 -- 172.21.15.12:0/3259368305 shutdown_connections 2024-03-29T11:28:49.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.818+0000 7f0c32e00700 1 -- 172.21.15.12:0/3259368305 >> 172.21.15.12:0/3259368305 conn(0x7f0c2c08aff0 msgr2=0x7f0c2c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:49.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.818+0000 7f0c32e00700 1 -- 172.21.15.12:0/3259368305 shutdown_connections 2024-03-29T11:28:49.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.818+0000 7f0c32e00700 1 -- 172.21.15.12:0/3259368305 wait complete. 2024-03-29T11:28:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.820+0000 7f0c32e00700 1 Processor -- start 2024-03-29T11:28:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.820+0000 7f0c32e00700 1 -- start start 2024-03-29T11:28:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.821+0000 7f0c32e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c10003a00 0x7f0c2c160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.821+0000 7f0c32e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c2c0b6020 0x7f0c2c0b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.821+0000 7f0c32e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c2c14ea00 0x7f0c2c06c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.821+0000 7f0c32e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0b6850 con 0x7f0c2c0b6020 2024-03-29T11:28:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.821+0000 7f0c32e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0b6b30 con 0x7f0c10003a00 2024-03-29T11:28:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.821+0000 7f0c32e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0b6e10 con 0x7f0c2c14ea00 2024-03-29T11:28:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.821+0000 7f0c2b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c2c0b6020 0x7f0c2c0b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.821+0000 7f0c2b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c2c0b6020 0x7f0c2c0b78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54504/0 (socket says 172.21.15.12:54504) 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.821+0000 7f0c2b400700 1 -- 172.21.15.12:0/348148277 learned_addr learned my addr 172.21.15.12:0/348148277 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.821+0000 7f0c2be00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c10003a00 0x7f0c2c160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.822+0000 7f0c30e00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c2c14ea00 0x7f0c2c06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.822+0000 7f0c2be00700 1 -- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c2c14ea00 msgr2=0x7f0c2c06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.822+0000 7f0c2be00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c2c14ea00 0x7f0c2c06c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.822+0000 7f0c2be00700 1 -- 172.21.15.12:0/348148277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c2c0b6020 msgr2=0x7f0c2c0b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.822+0000 7f0c2be00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c2c0b6020 0x7f0c2c0b78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.822+0000 7f0c2be00700 1 -- 172.21.15.12:0/348148277 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c0b70f0 con 0x7f0c10003a00 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.822+0000 7f0c2b400700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c2c0b6020 0x7f0c2c0b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.822+0000 7f0c2be00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c10003a00 0x7f0c2c160350 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c019000 tx=0x7f0c1c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.823+0000 7f0c23e00700 1 -- 172.21.15.12:0/348148277 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c00e9a0 con 0x7f0c10003a00 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.823+0000 7f0c23e00700 1 -- 172.21.15.12:0/348148277 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1c01c8a0 con 0x7f0c10003a00 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.823+0000 7f0c32e00700 1 -- 172.21.15.12:0/348148277 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2c0b73d0 con 0x7f0c10003a00 2024-03-29T11:28:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.823+0000 7f0c23e00700 1 -- 172.21.15.12:0/348148277 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c006e60 con 0x7f0c10003a00 2024-03-29T11:28:49.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.823+0000 7f0c32e00700 1 -- 172.21.15.12:0/348148277 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2c0b64a0 con 0x7f0c10003a00 2024-03-29T11:28:49.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.824+0000 7f0c32e00700 1 -- 172.21.15.12:0/348148277 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2c0b6b30 con 0x7f0c10003a00 2024-03-29T11:28:49.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.825+0000 7f0c23e00700 1 -- 172.21.15.12:0/348148277 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0c1c01cc50 con 0x7f0c10003a00 2024-03-29T11:28:49.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.826+0000 7f0c23e00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c0c06adb0 0x7f0c0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:49.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.826+0000 7f0c23e00700 1 -- 172.21.15.12:0/348148277 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5681+0+0 (secure 0 0 0) 0x7f0c1c08b250 con 0x7f0c10003a00 2024-03-29T11:28:49.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.826+0000 7f0c2b400700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c0c06adb0 0x7f0c0c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:49.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.827+0000 7f0c2b400700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c0c06adb0 0x7f0c0c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0cdbd0 tx=0x7f0c18009040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:49.831+0000 7f0c23e00700 1 -- 172.21.15.12:0/348148277 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c1c056440 con 0x7f0c10003a00 2024-03-29T11:28:50.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.093+0000 7f0c32e00700 1 -- 172.21.15.12:0/348148277 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0c2c0d1ca0 con 0x7f0c10003a00 2024-03-29T11:28:50.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.107+0000 7f0c23e00700 1 -- 172.21.15.12:0/348148277 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v94) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0c1c059a60 con 0x7f0c10003a00 2024-03-29T11:28:50.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.115+0000 7f0c32e00700 1 -- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c0c06adb0 msgr2=0x7f0c0c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.115+0000 7f0c32e00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c0c06adb0 0x7f0c0c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0cdbd0 tx=0x7f0c18009040 comp rx=0 tx=0).stop 2024-03-29T11:28:50.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.115+0000 7f0c32e00700 1 -- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c10003a00 msgr2=0x7f0c2c160350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.115+0000 7f0c32e00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c10003a00 0x7f0c2c160350 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c019000 tx=0x7f0c1c0047c0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.107 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:28:50.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.117+0000 7f0c32e00700 1 -- 172.21.15.12:0/348148277 shutdown_connections 2024-03-29T11:28:50.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.117+0000 7f0c32e00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c0c06adb0 0x7f0c0c06d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.117+0000 7f0c32e00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c2c14ea00 0x7f0c2c06c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.117+0000 7f0c32e00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c10003a00 0x7f0c2c160350 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.117+0000 7f0c32e00700 1 --2- 172.21.15.12:0/348148277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c2c0b6020 0x7f0c2c0b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.117+0000 7f0c32e00700 1 -- 172.21.15.12:0/348148277 >> 172.21.15.12:0/348148277 conn(0x7f0c2c08aff0 msgr2=0x7f0c2c15bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:50.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.117+0000 7f0c32e00700 1 -- 172.21.15.12:0/348148277 shutdown_connections 2024-03-29T11:28:50.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.117+0000 7f0c32e00700 1 -- 172.21.15.12:0/348148277 wait complete. 2024-03-29T11:28:50.118 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:28:50.122 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 24 and data pool 25 2024-03-29T11:28:50.138 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:50.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.387+0000 7fa26f000700 1 Processor -- start 2024-03-29T11:28:50.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.387+0000 7fa26f000700 1 -- start start 2024-03-29T11:28:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.388+0000 7fa26f000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa26815d760 0x7fa2680d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.388+0000 7fa26f000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa26815db40 0x7fa2680cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.388+0000 7fa26f000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2680d1330 con 0x7fa2680cf1f0 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.388+0000 7fa26f000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2680d1fb0 con 0x7fa26815db40 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.388+0000 7fa26f000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2680cf660 con 0x7fa26815d760 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.388+0000 7fa267400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa26815db40 0x7fa2680cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.388+0000 7fa26d000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa2680cf1f0 0x7fa2680cffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54168/0 (socket says 172.21.15.12:54168) 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.388+0000 7fa26d000700 1 -- 172.21.15.12:0/1233981745 learned_addr learned my addr 172.21.15.12:0/1233981745 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.389+0000 7fa267e00700 1 --2- 172.21.15.12:0/1233981745 >> v2:172.21.15.159:3300/0 conn(0x7fa26815d760 0x7fa2680d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.389+0000 7fa267400700 1 -- 172.21.15.12:0/1233981745 >> v2:172.21.15.159:3300/0 conn(0x7fa26815d760 msgr2=0x7fa2680d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.389+0000 7fa267400700 1 --2- 172.21.15.12:0/1233981745 >> v2:172.21.15.159:3300/0 conn(0x7fa26815d760 0x7fa2680d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.389+0000 7fa267400700 1 -- 172.21.15.12:0/1233981745 >> v1:172.21.15.12:6789/0 conn(0x7fa2680cf1f0 legacy=0x7fa2680cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.389+0000 7fa267400700 1 -- 172.21.15.12:0/1233981745 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2680d1610 con 0x7fa26815db40 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.389+0000 7fa266a00700 1 -- 172.21.15.12:0/1233981745 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108001520 0 0) 0x7fa2680d1330 con 0x7fa2680cf1f0 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.389+0000 7fa267400700 1 --2- 172.21.15.12:0/1233981745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa26815db40 0x7fa2680cecb0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fa2680b61d0 tx=0x7fa254009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68342ef8d323010 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.389+0000 7fa266a00700 1 -- 172.21.15.12:0/1233981745 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254011830 con 0x7fa26815db40 2024-03-29T11:28:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.389+0000 7fa266a00700 1 -- 172.21.15.12:0/1233981745 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa254011990 con 0x7fa26815db40 2024-03-29T11:28:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.389+0000 7fa266a00700 1 -- 172.21.15.12:0/1233981745 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254011d20 con 0x7fa26815db40 2024-03-29T11:28:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.390+0000 7fa26f000700 1 -- 172.21.15.12:0/1233981745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa26815db40 msgr2=0x7fa2680cecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.390+0000 7fa26f000700 1 --2- 172.21.15.12:0/1233981745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa26815db40 0x7fa2680cecb0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fa2680b61d0 tx=0x7fa254009b20 comp rx=0 tx=0).stop 2024-03-29T11:28:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.390+0000 7fa26f000700 1 -- 172.21.15.12:0/1233981745 shutdown_connections 2024-03-29T11:28:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.390+0000 7fa26f000700 1 --2- 172.21.15.12:0/1233981745 >> v2:172.21.15.159:3300/0 conn(0x7fa26815d760 0x7fa2680d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.390+0000 7fa26f000700 1 --2- 172.21.15.12:0/1233981745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa26815db40 0x7fa2680cecb0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.390+0000 7fa26f000700 1 -- 172.21.15.12:0/1233981745 >> 172.21.15.12:0/1233981745 conn(0x7fa26808aff0 msgr2=0x7fa2680d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.390+0000 7fa26f000700 1 -- 172.21.15.12:0/1233981745 shutdown_connections 2024-03-29T11:28:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.390+0000 7fa26f000700 1 -- 172.21.15.12:0/1233981745 wait complete. 2024-03-29T11:28:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.392+0000 7fa26f000700 1 Processor -- start 2024-03-29T11:28:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.392+0000 7fa26f000700 1 -- start start 2024-03-29T11:28:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa26f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2680cf1f0 0x7fa268159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa26f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa26815d760 0x7fa2680b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa26f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa26815db40 0x7fa2680cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa26f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26815f370 con 0x7fa2680cf1f0 2024-03-29T11:28:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa26f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2680cc7d0 con 0x7fa26815d760 2024-03-29T11:28:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa26f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2680cbf30 con 0x7fa26815db40 2024-03-29T11:28:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa267e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2680cf1f0 0x7fa268159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa267e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2680cf1f0 0x7fa268159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54526/0 (socket says 172.21.15.12:54526) 2024-03-29T11:28:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa267e00700 1 -- 172.21.15.12:0/3728764326 learned_addr learned my addr 172.21.15.12:0/3728764326 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa26d000700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa26815db40 0x7fa2680cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa267400700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa26815d760 0x7fa2680b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa267e00700 1 -- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa26815db40 msgr2=0x7fa2680cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.393+0000 7fa267e00700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa26815db40 0x7fa2680cb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.394+0000 7fa267e00700 1 -- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa26815d760 msgr2=0x7fa2680b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.394+0000 7fa267e00700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa26815d760 0x7fa2680b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.394+0000 7fa267e00700 1 -- 172.21.15.12:0/3728764326 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2540094a0 con 0x7fa2680cf1f0 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.394+0000 7fa267e00700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2680cf1f0 0x7fa268159460 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fa25800e9b0 tx=0x7fa25800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.394+0000 7fa25fe00700 1 -- 172.21.15.12:0/3728764326 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258009710 con 0x7fa2680cf1f0 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.394+0000 7fa25fe00700 1 -- 172.21.15.12:0/3728764326 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25800ee50 con 0x7fa2680cf1f0 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.394+0000 7fa25fe00700 1 -- 172.21.15.12:0/3728764326 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25800ca50 con 0x7fa2680cf1f0 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.394+0000 7fa26f000700 1 -- 172.21.15.12:0/3728764326 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2680cc210 con 0x7fa2680cf1f0 2024-03-29T11:28:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.394+0000 7fa26f000700 1 -- 172.21.15.12:0/3728764326 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26815f5a0 con 0x7fa2680cf1f0 2024-03-29T11:28:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.395+0000 7fa26f000700 1 -- 172.21.15.12:0/3728764326 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26815f910 con 0x7fa2680cf1f0 2024-03-29T11:28:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.396+0000 7fa25fe00700 1 -- 172.21.15.12:0/3728764326 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa25800cbb0 con 0x7fa2680cf1f0 2024-03-29T11:28:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.397+0000 7fa25fe00700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa24806adb0 0x7fa24806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.397+0000 7fa25fe00700 1 -- 172.21.15.12:0/3728764326 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fa2580894a0 con 0x7fa2680cf1f0 2024-03-29T11:28:50.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.398+0000 7fa267400700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa24806adb0 0x7fa24806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.399+0000 7fa267400700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa24806adb0 0x7fa24806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2680b61d0 tx=0x7fa25401c040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.403+0000 7fa25fe00700 1 -- 172.21.15.12:0/3728764326 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa258054510 con 0x7fa2680cf1f0 2024-03-29T11:28:50.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.619+0000 7fa26f000700 1 -- 172.21.15.12:0/3728764326 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa268067720 con 0x7fa2680cf1f0 2024-03-29T11:28:50.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.620+0000 7fa25fe00700 1 -- 172.21.15.12:0/3728764326 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16240 (secure 0 0 0) 0x7fa25800ce60 con 0x7fa2680cf1f0 2024-03-29T11:28:50.611 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:50.611 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":207,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:28:50.095017+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:28:47.651679+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-03-29T11:28:49.085953+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.622+0000 7fa26f000700 1 -- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa24806adb0 msgr2=0x7fa24806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.622+0000 7fa26f000700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa24806adb0 0x7fa24806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2680b61d0 tx=0x7fa25401c040 comp rx=0 tx=0).stop 2024-03-29T11:28:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.622+0000 7fa26f000700 1 -- 172.21.15.12:0/3728764326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2680cf1f0 msgr2=0x7fa268159460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.622+0000 7fa26f000700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2680cf1f0 0x7fa268159460 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fa25800e9b0 tx=0x7fa25800ecc0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.622+0000 7fa26f000700 1 -- 172.21.15.12:0/3728764326 shutdown_connections 2024-03-29T11:28:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.622+0000 7fa26f000700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa24806adb0 0x7fa24806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.622+0000 7fa26f000700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa26815db40 0x7fa2680cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.622+0000 7fa26f000700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2680cf1f0 0x7fa268159460 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.622+0000 7fa26f000700 1 --2- 172.21.15.12:0/3728764326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa26815d760 0x7fa2680b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.622+0000 7fa26f000700 1 -- 172.21.15.12:0/3728764326 >> 172.21.15.12:0/3728764326 conn(0x7fa26808aff0 msgr2=0x7fa2680b3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:50.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.623+0000 7fa26f000700 1 -- 172.21.15.12:0/3728764326 shutdown_connections 2024-03-29T11:28:50.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.623+0000 7fa26f000700 1 -- 172.21.15.12:0/3728764326 wait complete. 2024-03-29T11:28:50.635 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:50.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.843+0000 7ff8a5200700 1 Processor -- start 2024-03-29T11:28:50.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.843+0000 7ff8a5200700 1 -- start start 2024-03-29T11:28:50.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.844+0000 7ff8a5200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff8a015d920 0x7ff8a006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.844+0000 7ff8a5200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00d08d0 con 0x7ff8a0099a60 2024-03-29T11:28:50.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.844+0000 7ff8a5200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a006c300 con 0x7ff8a009a910 2024-03-29T11:28:50.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.844+0000 7ff8a5200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a006b9d0 con 0x7ff8a015d920 2024-03-29T11:28:50.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.845+0000 7ff89e800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff8a0099a60 0x7ff8a0099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54174/0 (socket says 172.21.15.12:54174) 2024-03-29T11:28:50.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.845+0000 7ff89e800700 1 -- 172.21.15.12:0/3414969441 learned_addr learned my addr 172.21.15.12:0/3414969441 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.845+0000 7ff89de00700 1 --2- 172.21.15.12:0/3414969441 >> v2:172.21.15.159:3300/0 conn(0x7ff8a015d920 0x7ff8a006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.845+0000 7ff89ca00700 1 -- 172.21.15.12:0/3414969441 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554851797 0 0) 0x7ff8a00d08d0 con 0x7ff8a0099a60 2024-03-29T11:28:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.845+0000 7ff89ca00700 1 -- 172.21.15.12:0/3414969441 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8840035b0 con 0x7ff8a0099a60 2024-03-29T11:28:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.845+0000 7ff89ca00700 1 -- 172.21.15.12:0/3414969441 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662473276 0 0) 0x7ff8a006c300 con 0x7ff8a009a910 2024-03-29T11:28:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.845+0000 7ff89ca00700 1 -- 172.21.15.12:0/3414969441 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a00d08d0 con 0x7ff8a009a910 2024-03-29T11:28:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.845+0000 7ff89de00700 1 -- 172.21.15.12:0/3414969441 >> v1:172.21.15.159:6789/0 conn(0x7ff8a009a910 legacy=0x7ff8a00cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.845+0000 7ff89ca00700 1 -- 172.21.15.12:0/3414969441 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2798727369 0 0) 0x7ff8840035b0 con 0x7ff8a0099a60 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.845+0000 7ff89de00700 1 -- 172.21.15.12:0/3414969441 >> v1:172.21.15.12:6789/0 conn(0x7ff8a0099a60 legacy=0x7ff8a0099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.846+0000 7ff89de00700 1 -- 172.21.15.12:0/3414969441 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a006bcb0 con 0x7ff8a015d920 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.846+0000 7ff89de00700 1 --2- 172.21.15.12:0/3414969441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8a015d920 0x7ff8a006c790 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff890009840 tx=0x7ff890009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c60ad7d2cb9bd8a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.846+0000 7ff89ca00700 1 -- 172.21.15.12:0/3414969441 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8980019f0 con 0x7ff8a015d920 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.846+0000 7ff89ca00700 1 -- 172.21.15.12:0/3414969441 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff890004030 con 0x7ff8a015d920 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.846+0000 7ff89ca00700 1 -- 172.21.15.12:0/3414969441 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8900043c0 con 0x7ff8a015d920 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.847+0000 7ff8a5200700 1 -- 172.21.15.12:0/3414969441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8a015d920 msgr2=0x7ff8a006c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.847+0000 7ff8a5200700 1 --2- 172.21.15.12:0/3414969441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8a015d920 0x7ff8a006c790 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff890009840 tx=0x7ff890009b50 comp rx=0 tx=0).stop 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.847+0000 7ff8a5200700 1 -- 172.21.15.12:0/3414969441 shutdown_connections 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.847+0000 7ff8a5200700 1 --2- 172.21.15.12:0/3414969441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8a015d920 0x7ff8a006c790 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.847+0000 7ff8a5200700 1 -- 172.21.15.12:0/3414969441 >> 172.21.15.12:0/3414969441 conn(0x7ff8a00d1350 msgr2=0x7ff8a00d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.847+0000 7ff8a5200700 1 -- 172.21.15.12:0/3414969441 shutdown_connections 2024-03-29T11:28:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.847+0000 7ff8a5200700 1 -- 172.21.15.12:0/3414969441 wait complete. 2024-03-29T11:28:50.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.848+0000 7ff8a5200700 1 Processor -- start 2024-03-29T11:28:50.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.849+0000 7ff8a5200700 1 -- start start 2024-03-29T11:28:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.849+0000 7ff8a5200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8a0099a60 0x7ff8a0156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff8a5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8a009a910 0x7ff8a01f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff8a5200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8a015d920 0x7ff8a006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff8a5200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a015e050 con 0x7ff8a009a910 2024-03-29T11:28:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff8a5200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a015e330 con 0x7ff8a0099a60 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff8a5200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a015e610 con 0x7ff8a015d920 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8a009a910 0x7ff8a01f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8a009a910 0x7ff8a01f2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54530/0 (socket says 172.21.15.12:54530) 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89d400700 1 -- 172.21.15.12:0/2618567649 learned_addr learned my addr 172.21.15.12:0/2618567649 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89de00700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8a0099a60 0x7ff8a0156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89e800700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8a015d920 0x7ff8a006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89d400700 1 -- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8a015d920 msgr2=0x7ff8a006ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89d400700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8a015d920 0x7ff8a006ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89d400700 1 -- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8a0099a60 msgr2=0x7ff8a0156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89d400700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8a0099a60 0x7ff8a0156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89d400700 1 -- 172.21.15.12:0/2618567649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8900094a0 con 0x7ff8a009a910 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.850+0000 7ff89de00700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8a0099a60 0x7ff8a0156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.851+0000 7ff89d400700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8a009a910 0x7ff8a01f2c40 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00c4f0 tx=0x7ff88c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.851+0000 7ff896000700 1 -- 172.21.15.12:0/2618567649 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c0052d0 con 0x7ff8a009a910 2024-03-29T11:28:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.851+0000 7ff896000700 1 -- 172.21.15.12:0/2618567649 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88c005430 con 0x7ff8a009a910 2024-03-29T11:28:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.851+0000 7ff8a5200700 1 -- 172.21.15.12:0/2618567649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a015e8f0 con 0x7ff8a009a910 2024-03-29T11:28:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.851+0000 7ff896000700 1 -- 172.21.15.12:0/2618567649 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c003880 con 0x7ff8a009a910 2024-03-29T11:28:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.851+0000 7ff8a5200700 1 -- 172.21.15.12:0/2618567649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a015dd00 con 0x7ff8a009a910 2024-03-29T11:28:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.852+0000 7ff8a5200700 1 -- 172.21.15.12:0/2618567649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a015ee10 con 0x7ff8a009a910 2024-03-29T11:28:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.853+0000 7ff896000700 1 -- 172.21.15.12:0/2618567649 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff88c0039e0 con 0x7ff8a009a910 2024-03-29T11:28:50.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.854+0000 7ff896000700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff88006adb0 0x7ff88006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:50.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.854+0000 7ff896000700 1 -- 172.21.15.12:0/2618567649 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5781+0+0 (secure 0 0 0) 0x7ff88c088990 con 0x7ff8a009a910 2024-03-29T11:28:50.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.855+0000 7ff89de00700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff88006adb0 0x7ff88006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:50.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.855+0000 7ff89de00700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff88006adb0 0x7ff88006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff8900090b0 tx=0x7ff890013040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:50.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:50.859+0000 7ff896000700 1 -- 172.21.15.12:0/2618567649 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff8a015ee10 con 0x7ff8a009a910 2024-03-29T11:28:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.074+0000 7ff8a5200700 1 -- 172.21.15.12:0/2618567649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8a015ee10 con 0x7ff8a009a910 2024-03-29T11:28:51.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.075+0000 7ff896000700 1 -- 172.21.15.12:0/2618567649 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16240 (secure 0 0 0) 0x7ff8a015ee10 con 0x7ff8a009a910 2024-03-29T11:28:51.066 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:51.066 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":207,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:28:50.095017+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:28:47.651679+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-03-29T11:28:49.085953+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 -- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff88006adb0 msgr2=0x7ff88006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff88006adb0 0x7ff88006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff8900090b0 tx=0x7ff890013040 comp rx=0 tx=0).stop 2024-03-29T11:28:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 -- 172.21.15.12:0/2618567649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8a009a910 msgr2=0x7ff8a01f2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8a009a910 0x7ff8a01f2c40 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00c4f0 tx=0x7ff88c00c8b0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 -- 172.21.15.12:0/2618567649 shutdown_connections 2024-03-29T11:28:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff88006adb0 0x7ff88006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff8a015d920 0x7ff8a006ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff8a0099a60 0x7ff8a0156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 --2- 172.21.15.12:0/2618567649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8a009a910 0x7ff8a01f2c40 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 -- 172.21.15.12:0/2618567649 >> 172.21.15.12:0/2618567649 conn(0x7ff8a00d1350 msgr2=0x7ff8a00cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 -- 172.21.15.12:0/2618567649 shutdown_connections 2024-03-29T11:28:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.078+0000 7ff8a5200700 1 -- 172.21.15.12:0/2618567649 wait complete. 2024-03-29T11:28:51.088 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T11:28:51.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.300+0000 7fcaa1e00700 1 Processor -- start 2024-03-29T11:28:51.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.300+0000 7fcaa1e00700 1 -- start start 2024-03-29T11:28:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.301+0000 7fcaa1e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fca9c0d2540 0x7fca9c1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.301+0000 7fcaa1e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c161420 con 0x7fca9c0d16a0 2024-03-29T11:28:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.301+0000 7fcaa1e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c161700 con 0x7fca9c15d9d0 2024-03-29T11:28:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.301+0000 7fcaa1e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca9c0d4340 con 0x7fca9c0d2540 2024-03-29T11:28:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca9a000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fca9c0d16a0 0x7fca9c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54186/0 (socket says 172.21.15.12:54186) 2024-03-29T11:28:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca9a000700 1 -- 172.21.15.12:0/2490237785 learned_addr learned my addr 172.21.15.12:0/2490237785 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca9b400700 1 --2- 172.21.15.12:0/2490237785 >> v2:172.21.15.159:3300/0 conn(0x7fca9c0d2540 0x7fca9c1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116221925 0 0) 0x7fca9c161420 con 0x7fca9c0d16a0 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca800035b0 con 0x7fca9c0d16a0 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3101244999 0 0) 0x7fca800035b0 con 0x7fca9c0d16a0 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 >> v2:172.21.15.159:3300/0 conn(0x7fca9c0d2540 msgr2=0x7fca9c1687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca99600700 1 --2- 172.21.15.12:0/2490237785 >> v2:172.21.15.159:3300/0 conn(0x7fca9c0d2540 0x7fca9c1687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 >> v1:172.21.15.159:6789/0 conn(0x7fca9c15d9d0 legacy=0x7fca9c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c1619e0 con 0x7fca9c0d16a0 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.302+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fca880017f0 con 0x7fca9c0d16a0 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 >> v1:172.21.15.12:6789/0 conn(0x7fca9c0d16a0 legacy=0x7fca9c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca99600700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca80003a00 0x7fca80003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca99600700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca80004400 0x7fca800048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca99600700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca80005170 0x7fca8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8000c080 con 0x7fca80003a00 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8000c1f0 con 0x7fca80005170 2024-03-29T11:28:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8000c3b0 con 0x7fca80004400 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9aa00700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca80003a00 0x7fca80003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9b400700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca80005170 0x7fca8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9a000700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca80004400 0x7fca800048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9aa00700 1 -- 172.21.15.12:0/2490237785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca80004400 msgr2=0x7fca800048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9aa00700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca80004400 0x7fca800048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9aa00700 1 -- 172.21.15.12:0/2490237785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca80005170 msgr2=0x7fca8000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9aa00700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca80005170 0x7fca8000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9aa00700 1 -- 172.21.15.12:0/2490237785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca800035b0 con 0x7fca80003a00 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9a000700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca80004400 0x7fca800048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9b400700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca80005170 0x7fca8000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9b400700 1 -- 172.21.15.12:0/2490237785 reap_dead start 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.303+0000 7fca9aa00700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca80003a00 0x7fca80003de0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fca8c001980 tx=0x7fca80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ddb00df32c205fa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:51.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.304+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c001420 con 0x7fca80003a00 2024-03-29T11:28:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.304+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c0052d0 con 0x7fca80003a00 2024-03-29T11:28:51.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.304+0000 7fca99600700 1 -- 172.21.15.12:0/2490237785 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c003680 con 0x7fca80003a00 2024-03-29T11:28:51.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.304+0000 7fcaa1e00700 1 -- 172.21.15.12:0/2490237785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca80003a00 msgr2=0x7fca80003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.304+0000 7fcaa1e00700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca80003a00 0x7fca80003de0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fca8c001980 tx=0x7fca80005880 comp rx=0 tx=0).stop 2024-03-29T11:28:51.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.304+0000 7fcaa1e00700 1 -- 172.21.15.12:0/2490237785 shutdown_connections 2024-03-29T11:28:51.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.304+0000 7fcaa1e00700 1 --2- 172.21.15.12:0/2490237785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca80003a00 0x7fca80003de0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.304+0000 7fcaa1e00700 1 -- 172.21.15.12:0/2490237785 >> 172.21.15.12:0/2490237785 conn(0x7fca9c08aff0 msgr2=0x7fca9c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:51.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.304+0000 7fcaa1e00700 1 -- 172.21.15.12:0/2490237785 shutdown_connections 2024-03-29T11:28:51.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.304+0000 7fcaa1e00700 1 -- 172.21.15.12:0/2490237785 wait complete. 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.306+0000 7fcaa1e00700 1 Processor -- start 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.306+0000 7fcaa1e00700 1 -- start start 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.307+0000 7fcaa1e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca80003a00 0x7fca9c0b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.307+0000 7fcaa1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca9c0b5600 0x7fca9c1f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.307+0000 7fcaa1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca9c0b42e0 0x7fca9c0b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.307+0000 7fcaa1e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c0cafb0 con 0x7fca9c0b5600 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.307+0000 7fcaa1e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c0b4ca0 con 0x7fca80003a00 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.307+0000 7fcaa1e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c0b67a0 con 0x7fca9c0b42e0 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca9c0b5600 0x7fca9c1f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca9c0b5600 0x7fca9c1f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54542/0 (socket says 172.21.15.12:54542) 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9a000700 1 -- 172.21.15.12:0/4050534356 learned_addr learned my addr 172.21.15.12:0/4050534356 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9aa00700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca80003a00 0x7fca9c0b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9b400700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca9c0b42e0 0x7fca9c0b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:51.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9a000700 1 -- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca9c0b42e0 msgr2=0x7fca9c0b4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9a000700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca9c0b42e0 0x7fca9c0b4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9a000700 1 -- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca80003a00 msgr2=0x7fca9c0b8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9a000700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca80003a00 0x7fca9c0b8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9a000700 1 -- 172.21.15.12:0/4050534356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c0b7ee0 con 0x7fca9c0b5600 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9b400700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca9c0b42e0 0x7fca9c0b4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca9a000700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca9c0b5600 0x7fca9c1f28a0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fca88000ef0 tx=0x7fca8800db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.308+0000 7fca93400700 1 -- 172.21.15.12:0/4050534356 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8800b070 con 0x7fca9c0b5600 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.309+0000 7fca93400700 1 -- 172.21.15.12:0/4050534356 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca88007500 con 0x7fca9c0b5600 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.309+0000 7fca93400700 1 -- 172.21.15.12:0/4050534356 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca88005a70 con 0x7fca9c0b5600 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.309+0000 7fcaa1e00700 1 -- 172.21.15.12:0/4050534356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca9c0cb1c0 con 0x7fca9c0b5600 2024-03-29T11:28:51.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.309+0000 7fcaa1e00700 1 -- 172.21.15.12:0/4050534356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca9c0b6430 con 0x7fca9c0b5600 2024-03-29T11:28:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.310+0000 7fcaa1e00700 1 -- 172.21.15.12:0/4050534356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9c0b65f0 con 0x7fca9c0b5600 2024-03-29T11:28:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.311+0000 7fca93400700 1 -- 172.21.15.12:0/4050534356 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fca88005bd0 con 0x7fca9c0b5600 2024-03-29T11:28:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.312+0000 7fca93400700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fca7c06adb0 0x7fca7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.312+0000 7fca93400700 1 -- 172.21.15.12:0/4050534356 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fca88088d20 con 0x7fca9c0b5600 2024-03-29T11:28:51.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.312+0000 7fca9aa00700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fca7c06adb0 0x7fca7c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:51.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.313+0000 7fca9aa00700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fca7c06adb0 0x7fca7c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fca8c015040 tx=0x7fca8c00a000 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:51.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.316+0000 7fca93400700 1 -- 172.21.15.12:0/4050534356 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fca88053d30 con 0x7fca9c0b5600 2024-03-29T11:28:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:51.575+0000 7fcaa1e00700 1 -- 172.21.15.12:0/4050534356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fca9c067720 con 0x7fca9c0b5600 2024-03-29T11:28:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.585+0000 7fca93400700 1 -- 172.21.15.12:0/4050534356 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v97) v1 ==== 118+0+0 (secure 0 0 0) 0x7fca88018020 con 0x7fca9c0b5600 2024-03-29T11:28:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.588+0000 7fcaa1e00700 1 -- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fca7c06adb0 msgr2=0x7fca7c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.588+0000 7fcaa1e00700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fca7c06adb0 0x7fca7c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fca8c015040 tx=0x7fca8c00a000 comp rx=0 tx=0).stop 2024-03-29T11:28:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.588+0000 7fcaa1e00700 1 -- 172.21.15.12:0/4050534356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca9c0b5600 msgr2=0x7fca9c1f28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.588+0000 7fcaa1e00700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca9c0b5600 0x7fca9c1f28a0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fca88000ef0 tx=0x7fca8800db00 comp rx=0 tx=0).stop 2024-03-29T11:28:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.588+0000 7fcaa1e00700 1 -- 172.21.15.12:0/4050534356 shutdown_connections 2024-03-29T11:28:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.588+0000 7fcaa1e00700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fca7c06adb0 0x7fca7c06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.588+0000 7fcaa1e00700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca9c0b42e0 0x7fca9c0b4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.588+0000 7fcaa1e00700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca80003a00 0x7fca9c0b8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.588+0000 7fcaa1e00700 1 --2- 172.21.15.12:0/4050534356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca9c0b5600 0x7fca9c1f28a0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.588+0000 7fcaa1e00700 1 -- 172.21.15.12:0/4050534356 >> 172.21.15.12:0/4050534356 conn(0x7fca9c08aff0 msgr2=0x7fca9c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.589+0000 7fcaa1e00700 1 -- 172.21.15.12:0/4050534356 shutdown_connections 2024-03-29T11:28:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.589+0000 7fcaa1e00700 1 -- 172.21.15.12:0/4050534356 wait complete. 2024-03-29T11:28:52.601 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T11:28:52.601 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T11:28:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.816+0000 7f7346600700 1 Processor -- start 2024-03-29T11:28:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.816+0000 7f7346600700 1 -- start start 2024-03-29T11:28:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.818+0000 7f7346600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400d2540 0x7f73401687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.818+0000 7f7346600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7340161420 con 0x7f734015d9d0 2024-03-29T11:28:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.818+0000 7f7346600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7340161700 con 0x7f73400d16a0 2024-03-29T11:28:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.818+0000 7f7346600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73400d4340 con 0x7f73400d2540 2024-03-29T11:28:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.818+0000 7f733f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f734015d9d0 0x7f73400d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54188/0 (socket says 172.21.15.12:54188) 2024-03-29T11:28:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.818+0000 7f733f400700 1 -- 172.21.15.12:0/2743347991 learned_addr learned my addr 172.21.15.12:0/2743347991 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.818+0000 7f733fe00700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400d2540 0x7f73401687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.818+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361387255 0 0) 0x7f7340161420 con 0x7f734015d9d0 2024-03-29T11:28:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.818+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73240035b0 con 0x7f734015d9d0 2024-03-29T11:28:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2120664640 0 0) 0x7f73240035b0 con 0x7f734015d9d0 2024-03-29T11:28:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400d2540 msgr2=0x7f73401687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400d2540 0x7f73401687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 >> v1:172.21.15.159:6789/0 conn(0x7f73400d16a0 legacy=0x7f73400d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73401619e0 con 0x7f734015d9d0 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f7330001670 con 0x7f734015d9d0 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 >> v1:172.21.15.12:6789/0 conn(0x7f734015d9d0 legacy=0x7f73400d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400d2540 0x7f7324003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7324003f70 0x7f7324004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7324004d90 0x7f7324005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732400ba40 con 0x7f7324004d90 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732400bbb0 con 0x7f7324003f70 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732400bd70 con 0x7f73400d2540 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733fe00700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7324004d90 0x7f7324005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733f400700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400d2540 0x7f7324003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.819+0000 7f733ea00700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7324003f70 0x7f7324004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.820+0000 7f733fe00700 1 -- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400d2540 msgr2=0x7f7324003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.820+0000 7f733fe00700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400d2540 0x7f7324003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.820+0000 7f733fe00700 1 -- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7324003f70 msgr2=0x7f7324004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.820+0000 7f733fe00700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7324003f70 0x7f7324004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.820+0000 7f733fe00700 1 -- 172.21.15.12:0/2743347991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73240035b0 con 0x7f7324004d90 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.820+0000 7f733fe00700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7324004d90 0x7f7324005210 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f734008c320 tx=0x7f7324005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209aadf413bac946 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.820+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733800eba0 con 0x7f7324004d90 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.820+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733800f040 con 0x7f7324004d90 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.820+0000 7f733e000700 1 -- 172.21.15.12:0/2743347991 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7338014480 con 0x7f7324004d90 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.821+0000 7f7346600700 1 -- 172.21.15.12:0/2743347991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7324004d90 msgr2=0x7f7324005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.821+0000 7f7346600700 1 --2- 172.21.15.12:0/2743347991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7324004d90 0x7f7324005210 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f734008c320 tx=0x7f7324005880 comp rx=0 tx=0).stop 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.821+0000 7f733fe00700 1 -- 172.21.15.12:0/2743347991 reap_dead start 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.821+0000 7f7346600700 1 -- 172.21.15.12:0/2743347991 shutdown_connections 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.821+0000 7f7346600700 1 -- 172.21.15.12:0/2743347991 >> 172.21.15.12:0/2743347991 conn(0x7f734008aff0 msgr2=0x7f73400d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.821+0000 7f7346600700 1 -- 172.21.15.12:0/2743347991 shutdown_connections 2024-03-29T11:28:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.821+0000 7f7346600700 1 -- 172.21.15.12:0/2743347991 wait complete. 2024-03-29T11:28:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.823+0000 7f7346600700 1 Processor -- start 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.823+0000 7f7346600700 1 -- start start 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.823+0000 7f7346600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7340155470 0x7f73400b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f7346600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f734015f140 0x7f73400b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f7346600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400b4a00 0x7f734015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f7346600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73400b4eb0 con 0x7f7340155470 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f7346600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734015f9d0 con 0x7f734015f140 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f7346600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734015fcb0 con 0x7f73400b4a00 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f733f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7340155470 0x7f73400b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f733f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7340155470 0x7f73400b8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54570/0 (socket says 172.21.15.12:54570) 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f733f400700 1 -- 172.21.15.12:0/511622295 learned_addr learned my addr 172.21.15.12:0/511622295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f733ea00700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f734015f140 0x7f73400b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f733fe00700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400b4a00 0x7f734015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f733f400700 1 -- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400b4a00 msgr2=0x7f734015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f733f400700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400b4a00 0x7f734015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f733f400700 1 -- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f734015f140 msgr2=0x7f73400b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f733f400700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f734015f140 0x7f73400b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.824+0000 7f733f400700 1 -- 172.21.15.12:0/511622295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734015ff90 con 0x7f7340155470 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.825+0000 7f733f400700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7340155470 0x7f73400b8d30 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f73300072c0 tx=0x7f73300075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.825+0000 7f7337400700 1 -- 172.21.15.12:0/511622295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330012560 con 0x7f7340155470 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.825+0000 7f7337400700 1 -- 172.21.15.12:0/511622295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73300126e0 con 0x7f7340155470 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.825+0000 7f7346600700 1 -- 172.21.15.12:0/511622295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7340160270 con 0x7f7340155470 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.825+0000 7f7337400700 1 -- 172.21.15.12:0/511622295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330012a90 con 0x7f7340155470 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.825+0000 7f7346600700 1 -- 172.21.15.12:0/511622295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f734015f5c0 con 0x7f7340155470 2024-03-29T11:28:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.826+0000 7f7346600700 1 -- 172.21.15.12:0/511622295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73400d1ca0 con 0x7f7340155470 2024-03-29T11:28:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.827+0000 7f7337400700 1 -- 172.21.15.12:0/511622295 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7330012bf0 con 0x7f7340155470 2024-03-29T11:28:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.828+0000 7f7337400700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f732006adb0 0x7f732006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.828+0000 7f7337400700 1 -- 172.21.15.12:0/511622295 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f7330089400 con 0x7f7340155470 2024-03-29T11:28:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.828+0000 7f733ea00700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f732006adb0 0x7f732006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.829+0000 7f733ea00700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f732006adb0 0x7f732006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f73400a1420 tx=0x7f732c009040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:52.833+0000 7f7337400700 1 -- 172.21.15.12:0/511622295 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7330054470 con 0x7f7340155470 2024-03-29T11:28:53.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.099+0000 7f7346600700 1 -- 172.21.15.12:0/511622295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f734015f780 con 0x7f7340155470 2024-03-29T11:28:53.530 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-29T11:28:53.539+0000 7fd34a200700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:28:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.587+0000 7f7337400700 1 -- 172.21.15.12:0/511622295 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v98) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7330057a90 con 0x7f7340155470 2024-03-29T11:28:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.590+0000 7f7346600700 1 -- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f732006adb0 msgr2=0x7f732006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.590+0000 7f7346600700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f732006adb0 0x7f732006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f73400a1420 tx=0x7f732c009040 comp rx=0 tx=0).stop 2024-03-29T11:28:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.590+0000 7f7346600700 1 -- 172.21.15.12:0/511622295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7340155470 msgr2=0x7f73400b8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.590+0000 7f7346600700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7340155470 0x7f73400b8d30 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f73300072c0 tx=0x7f73300075d0 comp rx=0 tx=0).stop 2024-03-29T11:28:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.591+0000 7f7346600700 1 -- 172.21.15.12:0/511622295 shutdown_connections 2024-03-29T11:28:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.591+0000 7f7346600700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f732006adb0 0x7f732006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.591+0000 7f7346600700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73400b4a00 0x7f734015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.591+0000 7f7346600700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7340155470 0x7f73400b8d30 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.591+0000 7f7346600700 1 --2- 172.21.15.12:0/511622295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f734015f140 0x7f73400b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.591+0000 7f7346600700 1 -- 172.21.15.12:0/511622295 >> 172.21.15.12:0/511622295 conn(0x7f734008aff0 msgr2=0x7f73400b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.591+0000 7f7346600700 1 -- 172.21.15.12:0/511622295 shutdown_connections 2024-03-29T11:28:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.591+0000 7f7346600700 1 -- 172.21.15.12:0/511622295 wait complete. 2024-03-29T11:28:53.601 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T11:28:53.601 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:28:53.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.812+0000 7f4273200700 1 Processor -- start 2024-03-29T11:28:53.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.812+0000 7f4273200700 1 -- start start 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.813+0000 7f4273200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f426c0d18b0 0x7f426c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.813+0000 7f4273200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f426c161590 0x7f426c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.813+0000 7f4273200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c0ceda0 con 0x7f426c160f10 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.813+0000 7f4273200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426c0d1d30 con 0x7f426c161590 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.813+0000 7f4273200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426c161f60 con 0x7f426c0d18b0 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.813+0000 7f4271200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f426c161590 0x7f426c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.813+0000 7f4271200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f426c161590 0x7f426c161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54574/0 (socket says 172.21.15.12:54574) 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.814+0000 7f4271200700 1 -- 172.21.15.12:0/1373139153 learned_addr learned my addr 172.21.15.12:0/1373139153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.814+0000 7f426b400700 1 --2- 172.21.15.12:0/1373139153 >> v2:172.21.15.159:3300/0 conn(0x7f426c0d18b0 0x7f426c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.814+0000 7f4271200700 1 -- 172.21.15.12:0/1373139153 >> v2:172.21.15.159:3300/0 conn(0x7f426c0d18b0 msgr2=0x7f426c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.814+0000 7f4271200700 1 --2- 172.21.15.12:0/1373139153 >> v2:172.21.15.159:3300/0 conn(0x7f426c0d18b0 0x7f426c0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.814+0000 7f4271200700 1 -- 172.21.15.12:0/1373139153 >> v1:172.21.15.159:6789/0 conn(0x7f426c160f10 legacy=0x7f426c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.814+0000 7f4271200700 1 -- 172.21.15.12:0/1373139153 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c1612f0 con 0x7f426c161590 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.814+0000 7f4271200700 1 --2- 172.21.15.12:0/1373139153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c161590 0x7f426c161a20 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f426c08ca30 tx=0x7f426400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65cc1292da800682 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.814+0000 7f426aa00700 1 -- 172.21.15.12:0/1373139153 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426400e070 con 0x7f426c161590 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.814+0000 7f426aa00700 1 -- 172.21.15.12:0/1373139153 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4264003a20 con 0x7f426c161590 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.814+0000 7f426aa00700 1 -- 172.21.15.12:0/1373139153 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264003dd0 con 0x7f426c161590 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.815+0000 7f4273200700 1 -- 172.21.15.12:0/1373139153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c161590 msgr2=0x7f426c161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.815+0000 7f4273200700 1 --2- 172.21.15.12:0/1373139153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c161590 0x7f426c161a20 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f426c08ca30 tx=0x7f426400b330 comp rx=0 tx=0).stop 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.815+0000 7f4273200700 1 -- 172.21.15.12:0/1373139153 shutdown_connections 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.815+0000 7f4273200700 1 --2- 172.21.15.12:0/1373139153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c161590 0x7f426c161a20 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.815+0000 7f4273200700 1 --2- 172.21.15.12:0/1373139153 >> v2:172.21.15.159:3300/0 conn(0x7f426c0d18b0 0x7f426c0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.815+0000 7f4273200700 1 -- 172.21.15.12:0/1373139153 >> 172.21.15.12:0/1373139153 conn(0x7f426c08aff0 msgr2=0x7f426c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.815+0000 7f4273200700 1 -- 172.21.15.12:0/1373139153 shutdown_connections 2024-03-29T11:28:53.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.815+0000 7f4273200700 1 -- 172.21.15.12:0/1373139153 wait complete. 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.817+0000 7f4273200700 1 Processor -- start 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.817+0000 7f4273200700 1 -- start start 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f4273200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f426c0d18b0 0x7f426c1525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f4273200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c160f10 0x7f426c0b7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f4273200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f426c161590 0x7f426c1565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f4273200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0b6670 con 0x7f426c160f10 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f4273200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0b7470 con 0x7f426c0d18b0 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f4273200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0b6860 con 0x7f426c161590 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f426b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c160f10 0x7f426c0b7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f426b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c160f10 0x7f426c0b7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54590/0 (socket says 172.21.15.12:54590) 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f426b400700 1 -- 172.21.15.12:0/3837768444 learned_addr learned my addr 172.21.15.12:0/3837768444 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f426be00700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f426c0d18b0 0x7f426c1525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.818+0000 7f4271200700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f426c161590 0x7f426c1565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f426b400700 1 -- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f426c161590 msgr2=0x7f426c1565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f426b400700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f426c161590 0x7f426c1565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f426b400700 1 -- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f426c0d18b0 msgr2=0x7f426c1525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f426b400700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f426c0d18b0 0x7f426c1525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f426b400700 1 -- 172.21.15.12:0/3837768444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4264009b30 con 0x7f426c160f10 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f426b400700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c160f10 0x7f426c0b7900 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f425800d800 tx=0x7f425800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f4263e00700 1 -- 172.21.15.12:0/3837768444 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42580125f0 con 0x7f426c160f10 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f4263e00700 1 -- 172.21.15.12:0/3837768444 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4258012750 con 0x7f426c160f10 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f4263e00700 1 -- 172.21.15.12:0/3837768444 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4258012b00 con 0x7f426c160f10 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f4273200700 1 -- 172.21.15.12:0/3837768444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426c0b6b40 con 0x7f426c160f10 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.819+0000 7f4273200700 1 -- 172.21.15.12:0/3837768444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c0b7100 con 0x7f426c160f10 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.820+0000 7f4273200700 1 -- 172.21.15.12:0/3837768444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c0b7600 con 0x7f426c160f10 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.821+0000 7f4263e00700 1 -- 172.21.15.12:0/3837768444 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4258012c60 con 0x7f426c160f10 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.822+0000 7f4263e00700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f424c06adb0 0x7f424c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:53.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.822+0000 7f4263e00700 1 -- 172.21.15.12:0/3837768444 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f4258089590 con 0x7f426c160f10 2024-03-29T11:28:53.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.823+0000 7f426be00700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f424c06adb0 0x7f424c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:53.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.823+0000 7f426be00700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f424c06adb0 0x7f424c06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f425c009960 tx=0x7f425c0090e0 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:53.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:53.827+0000 7f4263e00700 1 -- 172.21.15.12:0/3837768444 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4258054600 con 0x7f426c160f10 2024-03-29T11:28:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.087+0000 7f4273200700 1 -- 172.21.15.12:0/3837768444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f426c067720 con 0x7f426c160f10 2024-03-29T11:28:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.087+0000 7f4263e00700 1 -- 172.21.15.12:0/3837768444 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f4258057c20 con 0x7f426c160f10 2024-03-29T11:28:54.078 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:54.078 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/395980670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":395980670},{"type":"v1","addr":"172.21.15.159:6837","nonce":395980670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":93},{"gid":10183,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/531080306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":531080306},{"type":"v1","addr":"172.21.15.159:6835","nonce":531080306}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10509,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2709565550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2709565550},{"type":"v1","addr":"172.21.15.12:6834","nonce":2709565550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:28:50.095373+0000","modified":"2024-03-29T11:28:53.584378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10512":{"gid":10512,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.12:6835/99601797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":99601797},{"type":"v1","addr":"172.21.15.12:6835","nonce":99601797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-29T11:28:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 -- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f424c06adb0 msgr2=0x7f424c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f424c06adb0 0x7f424c06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f425c009960 tx=0x7f425c0090e0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 -- 172.21.15.12:0/3837768444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c160f10 msgr2=0x7f426c0b7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c160f10 0x7f426c0b7900 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f425800d800 tx=0x7f425800db10 comp rx=0 tx=0).stop 2024-03-29T11:28:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 -- 172.21.15.12:0/3837768444 shutdown_connections 2024-03-29T11:28:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f424c06adb0 0x7f424c06d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f426c161590 0x7f426c1565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f426c0d18b0 0x7f426c1525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 --2- 172.21.15.12:0/3837768444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f426c160f10 0x7f426c0b7900 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 -- 172.21.15.12:0/3837768444 >> 172.21.15.12:0/3837768444 conn(0x7f426c08aff0 msgr2=0x7f426c06cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 -- 172.21.15.12:0/3837768444 shutdown_connections 2024-03-29T11:28:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.090+0000 7f4273200700 1 -- 172.21.15.12:0/3837768444 wait complete. 2024-03-29T11:28:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 98 2024-03-29T11:28:54.099 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:54.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.310+0000 7f7e71600700 1 Processor -- start 2024-03-29T11:28:54.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.310+0000 7f7e71600700 1 -- start start 2024-03-29T11:28:54.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.311+0000 7f7e71600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7e6c15da50 0x7f7e6c0ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.311+0000 7f7e71600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e6c0cec70 0x7f7e6c0d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.311+0000 7f7e71600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e6c0d24e0 con 0x7f7e6c0d0b20 2024-03-29T11:28:54.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.311+0000 7f7e71600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e6c0d0fd0 con 0x7f7e6c0cec70 2024-03-29T11:28:54.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.311+0000 7f7e71600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e6c0b95d0 con 0x7f7e6c15da50 2024-03-29T11:28:54.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.312+0000 7f7e6a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e6c0cec70 0x7f7e6c0d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.312+0000 7f7e6aa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7e6c15da50 0x7f7e6c0ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.312+0000 7f7e6a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e6c0cec70 0x7f7e6c0d05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54598/0 (socket says 172.21.15.12:54598) 2024-03-29T11:28:54.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.312+0000 7f7e6a000700 1 -- 172.21.15.12:0/568327095 learned_addr learned my addr 172.21.15.12:0/568327095 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.312+0000 7f7e6a000700 1 -- 172.21.15.12:0/568327095 >> v2:172.21.15.159:3300/0 conn(0x7f7e6c15da50 msgr2=0x7f7e6c0ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.312+0000 7f7e6a000700 1 --2- 172.21.15.12:0/568327095 >> v2:172.21.15.159:3300/0 conn(0x7f7e6c15da50 0x7f7e6c0ce730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.312+0000 7f7e6a000700 1 -- 172.21.15.12:0/568327095 >> v1:172.21.15.159:6789/0 conn(0x7f7e6c0d0b20 legacy=0x7f7e6c0b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.312+0000 7f7e6a000700 1 -- 172.21.15.12:0/568327095 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c0d27c0 con 0x7f7e6c0cec70 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.312+0000 7f7e6a000700 1 --2- 172.21.15.12:0/568327095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c0cec70 0x7f7e6c0d05e0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0cfbb0 tx=0x7f7e58009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1787b5633f41036 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e69600700 1 -- 172.21.15.12:0/568327095 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58011830 con 0x7f7e6c0cec70 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e69600700 1 -- 172.21.15.12:0/568327095 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e58011990 con 0x7f7e6c0cec70 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e69600700 1 -- 172.21.15.12:0/568327095 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58011d20 con 0x7f7e6c0cec70 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e71600700 1 -- 172.21.15.12:0/568327095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c0cec70 msgr2=0x7f7e6c0d05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e71600700 1 --2- 172.21.15.12:0/568327095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c0cec70 0x7f7e6c0d05e0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0cfbb0 tx=0x7f7e58009bd0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e71600700 1 -- 172.21.15.12:0/568327095 shutdown_connections 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e71600700 1 --2- 172.21.15.12:0/568327095 >> v2:172.21.15.159:3300/0 conn(0x7f7e6c15da50 0x7f7e6c0ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e71600700 1 --2- 172.21.15.12:0/568327095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c0cec70 0x7f7e6c0d05e0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e71600700 1 -- 172.21.15.12:0/568327095 >> 172.21.15.12:0/568327095 conn(0x7f7e6c0666a0 msgr2=0x7f7e6c066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:54.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e71600700 1 -- 172.21.15.12:0/568327095 shutdown_connections 2024-03-29T11:28:54.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.313+0000 7f7e71600700 1 -- 172.21.15.12:0/568327095 wait complete. 2024-03-29T11:28:54.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.315+0000 7f7e71600700 1 Processor -- start 2024-03-29T11:28:54.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.315+0000 7f7e71600700 1 -- start start 2024-03-29T11:28:54.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.316+0000 7f7e71600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e6c0cec70 0x7f7e6c1538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.316+0000 7f7e71600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c0d0b20 0x7f7e6c0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.316+0000 7f7e71600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c15da50 0x7f7e6c0c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.316+0000 7f7e71600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c0c7c80 con 0x7f7e6c15da50 2024-03-29T11:28:54.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.316+0000 7f7e71600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c0c7f60 con 0x7f7e6c0d0b20 2024-03-29T11:28:54.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.316+0000 7f7e71600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c0c8240 con 0x7f7e6c0cec70 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.316+0000 7f7e6b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c15da50 0x7f7e6c0c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.316+0000 7f7e6b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c15da50 0x7f7e6c0c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54608/0 (socket says 172.21.15.12:54608) 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.316+0000 7f7e6b400700 1 -- 172.21.15.12:0/2091728400 learned_addr learned my addr 172.21.15.12:0/2091728400 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e6aa00700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e6c0cec70 0x7f7e6c1538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e6a000700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c0d0b20 0x7f7e6c0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e6b400700 1 -- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e6c0cec70 msgr2=0x7f7e6c1538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e6b400700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e6c0cec70 0x7f7e6c1538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e6b400700 1 -- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c0d0b20 msgr2=0x7f7e6c0c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e6b400700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c0d0b20 0x7f7e6c0c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e6b400700 1 -- 172.21.15.12:0/2091728400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e58009550 con 0x7f7e6c15da50 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e6b400700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c15da50 0x7f7e6c0c7370 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400cb40 tx=0x7f7e6400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e62a00700 1 -- 172.21.15.12:0/2091728400 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64003c00 con 0x7f7e6c15da50 2024-03-29T11:28:54.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e62a00700 1 -- 172.21.15.12:0/2091728400 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e64003d60 con 0x7f7e6c15da50 2024-03-29T11:28:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e62a00700 1 -- 172.21.15.12:0/2091728400 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400aa20 con 0x7f7e6c15da50 2024-03-29T11:28:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.317+0000 7f7e71600700 1 -- 172.21.15.12:0/2091728400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6c0c8520 con 0x7f7e6c15da50 2024-03-29T11:28:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.318+0000 7f7e71600700 1 -- 172.21.15.12:0/2091728400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6c0c9260 con 0x7f7e6c15da50 2024-03-29T11:28:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.319+0000 7f7e71600700 1 -- 172.21.15.12:0/2091728400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6c0c8a40 con 0x7f7e6c15da50 2024-03-29T11:28:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.320+0000 7f7e62a00700 1 -- 172.21.15.12:0/2091728400 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7e6400ab80 con 0x7f7e6c15da50 2024-03-29T11:28:54.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.321+0000 7f7e62a00700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e4c06adb0 0x7f7e4c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.321+0000 7f7e62a00700 1 -- 172.21.15.12:0/2091728400 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f7e640893e0 con 0x7f7e6c15da50 2024-03-29T11:28:54.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.321+0000 7f7e6aa00700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e4c06adb0 0x7f7e4c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.322+0000 7f7e6aa00700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e4c06adb0 0x7f7e4c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009b40 tx=0x7f7e5c009210 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:54.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.325+0000 7f7e62a00700 1 -- 172.21.15.12:0/2091728400 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7e6c0c8a40 con 0x7f7e6c15da50 2024-03-29T11:28:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.541+0000 7f7e71600700 1 -- 172.21.15.12:0/2091728400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e6c0c8a40 con 0x7f7e6c15da50 2024-03-29T11:28:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.542+0000 7f7e62a00700 1 -- 172.21.15.12:0/2091728400 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+16227 (secure 0 0 0) 0x7f7e6c0c8a40 con 0x7f7e6c15da50 2024-03-29T11:28:54.533 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:54.533 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":208,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:28:51.097916+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:28:47.651679+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-03-29T11:28:49.085953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.544+0000 7f7e71600700 1 -- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e4c06adb0 msgr2=0x7f7e4c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.544+0000 7f7e71600700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e4c06adb0 0x7f7e4c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009b40 tx=0x7f7e5c009210 comp rx=0 tx=0).stop 2024-03-29T11:28:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.544+0000 7f7e71600700 1 -- 172.21.15.12:0/2091728400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c15da50 msgr2=0x7f7e6c0c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.544+0000 7f7e71600700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c15da50 0x7f7e6c0c7370 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400cb40 tx=0x7f7e6400ce50 comp rx=0 tx=0).stop 2024-03-29T11:28:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.545+0000 7f7e71600700 1 -- 172.21.15.12:0/2091728400 shutdown_connections 2024-03-29T11:28:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.545+0000 7f7e71600700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7e4c06adb0 0x7f7e4c06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.545+0000 7f7e71600700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6c15da50 0x7f7e6c0c7370 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.545+0000 7f7e71600700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e6c0cec70 0x7f7e6c1538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.545+0000 7f7e71600700 1 --2- 172.21.15.12:0/2091728400 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e6c0d0b20 0x7f7e6c0c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.545+0000 7f7e71600700 1 -- 172.21.15.12:0/2091728400 >> 172.21.15.12:0/2091728400 conn(0x7f7e6c0666a0 msgr2=0x7f7e6c0b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.545+0000 7f7e71600700 1 -- 172.21.15.12:0/2091728400 shutdown_connections 2024-03-29T11:28:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.545+0000 7f7e71600700 1 -- 172.21.15.12:0/2091728400 wait complete. 2024-03-29T11:28:54.554 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T11:28:54.554 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:28:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.762+0000 7f528f200700 1 Processor -- start 2024-03-29T11:28:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.762+0000 7f528f200700 1 -- start start 2024-03-29T11:28:54.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.763+0000 7f528f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52880cf3c0 0x7f52880d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.763+0000 7f528f200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f528815db40 0x7f52880cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.763+0000 7f528f200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52880d1330 con 0x7f528815d760 2024-03-29T11:28:54.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.763+0000 7f528f200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52880d1fb0 con 0x7f528815db40 2024-03-29T11:28:54.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.763+0000 7f528f200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52880d1610 con 0x7f52880cf3c0 2024-03-29T11:28:54.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5287e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f528815d760 0x7f52880cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54204/0 (socket says 172.21.15.12:54204) 2024-03-29T11:28:54.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5287e00700 1 -- 172.21.15.12:0/191736954 learned_addr learned my addr 172.21.15.12:0/191736954 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:54.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5287400700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52880cf3c0 0x7f52880d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f528d200700 1 --2- 172.21.15.12:0/191736954 >> v2:172.21.15.159:3300/0 conn(0x7f528815db40 0x7f52880cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197597915 0 0) 0x7f52880d1330 con 0x7f528815d760 2024-03-29T11:28:54.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f526c0035b0 con 0x7f528815d760 2024-03-29T11:28:54.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2101503992 0 0) 0x7f526c0035b0 con 0x7f528815d760 2024-03-29T11:28:54.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52880cf3c0 msgr2=0x7f52880d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5286a00700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52880cf3c0 0x7f52880d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 >> v2:172.21.15.159:3300/0 conn(0x7f528815db40 msgr2=0x7f52880cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5286a00700 1 --2- 172.21.15.12:0/191736954 >> v2:172.21.15.159:3300/0 conn(0x7f528815db40 0x7f52880cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52880d18f0 con 0x7f528815d760 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.764+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f5278001670 con 0x7f528815d760 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 >> v1:172.21.15.12:6789/0 conn(0x7f528815d760 legacy=0x7f52880cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5286a00700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f526c003a00 0x7f526c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5286a00700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f526c004850 0x7f526c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5286a00700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52880cf3c0 0x7f526c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52880d1fb0 con 0x7f526c003a00 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52880d1610 con 0x7f526c004850 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526c00ba60 con 0x7f52880cf3c0 2024-03-29T11:28:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5287e00700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f526c003a00 0x7f526c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5287400700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f526c004850 0x7f526c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f528d200700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52880cf3c0 0x7f526c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5287e00700 1 -- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52880cf3c0 msgr2=0x7f526c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5287e00700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52880cf3c0 0x7f526c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5287e00700 1 -- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f526c004850 msgr2=0x7f526c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5287e00700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f526c004850 0x7f526c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.765+0000 7f5287e00700 1 -- 172.21.15.12:0/191736954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c0035b0 con 0x7f526c003a00 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.766+0000 7f5287e00700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f526c003a00 0x7f526c003e20 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f527800c1c0 tx=0x7f526c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0fcdf14cf43a1ee server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.766+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278007b70 con 0x7f526c003a00 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.766+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527800cd20 con 0x7f526c003a00 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.766+0000 7f5286a00700 1 -- 172.21.15.12:0/191736954 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278005ba0 con 0x7f526c003a00 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.766+0000 7f528f200700 1 -- 172.21.15.12:0/191736954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f526c003a00 msgr2=0x7f526c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.766+0000 7f528f200700 1 --2- 172.21.15.12:0/191736954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f526c003a00 0x7f526c003e20 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f527800c1c0 tx=0x7f526c005880 comp rx=0 tx=0).stop 2024-03-29T11:28:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.766+0000 7f528d200700 1 -- 172.21.15.12:0/191736954 reap_dead start 2024-03-29T11:28:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.767+0000 7f528f200700 1 -- 172.21.15.12:0/191736954 shutdown_connections 2024-03-29T11:28:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.767+0000 7f528f200700 1 -- 172.21.15.12:0/191736954 >> 172.21.15.12:0/191736954 conn(0x7f528808aff0 msgr2=0x7f52880d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.767+0000 7f528f200700 1 -- 172.21.15.12:0/191736954 shutdown_connections 2024-03-29T11:28:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.767+0000 7f528f200700 1 -- 172.21.15.12:0/191736954 wait complete. 2024-03-29T11:28:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.768+0000 7f528f200700 1 Processor -- start 2024-03-29T11:28:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.769+0000 7f528f200700 1 -- start start 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.769+0000 7f528f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f526c003a00 0x7f5288159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.769+0000 7f528f200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f528815a500 0x7f52881f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.769+0000 7f528f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52880b6450 0x7f52880b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.769+0000 7f528f200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52880b6e40 con 0x7f52880b6450 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.769+0000 7f528f200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528815a2b0 con 0x7f528815a500 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.769+0000 7f528f200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52880cc5b0 con 0x7f526c003a00 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f528d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52880b6450 0x7f52880b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f528d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52880b6450 0x7f52880b6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54626/0 (socket says 172.21.15.12:54626) 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f528d200700 1 -- 172.21.15.12:0/698769177 learned_addr learned my addr 172.21.15.12:0/698769177 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f5287e00700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f526c003a00 0x7f5288159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f5287400700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f528815a500 0x7f52881f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f528d200700 1 -- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f526c003a00 msgr2=0x7f5288159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f528d200700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f526c003a00 0x7f5288159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f528d200700 1 -- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f528815a500 msgr2=0x7f52881f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f528d200700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f528815a500 0x7f52881f38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f528d200700 1 -- 172.21.15.12:0/698769177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528806b950 con 0x7f52880b6450 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.770+0000 7f528d200700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52880b6450 0x7f52880b6900 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f528808c1d0 tx=0x7f5280011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.771+0000 7f527fe00700 1 -- 172.21.15.12:0/698769177 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528000e040 con 0x7f52880b6450 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.771+0000 7f527fe00700 1 -- 172.21.15.12:0/698769177 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528000f040 con 0x7f52880b6450 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.771+0000 7f527fe00700 1 -- 172.21.15.12:0/698769177 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528000fc40 con 0x7f52880b6450 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.771+0000 7f528f200700 1 -- 172.21.15.12:0/698769177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528806bc30 con 0x7f52880b6450 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.771+0000 7f528f200700 1 -- 172.21.15.12:0/698769177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528806c1f0 con 0x7f52880b6450 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.772+0000 7f528f200700 1 -- 172.21.15.12:0/698769177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5288067720 con 0x7f52880b6450 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.773+0000 7f527fe00700 1 -- 172.21.15.12:0/698769177 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5280022460 con 0x7f52880b6450 2024-03-29T11:28:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.774+0000 7f527fe00700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f526806adb0 0x7f526806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.774+0000 7f527fe00700 1 -- 172.21.15.12:0/698769177 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f52800899e0 con 0x7f52880b6450 2024-03-29T11:28:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.774+0000 7f5287e00700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f526806adb0 0x7f526806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.775+0000 7f5287e00700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f526806adb0 0x7f526806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f527800c160 tx=0x7f527800a000 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.778+0000 7f527fe00700 1 -- 172.21.15.12:0/698769177 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5280054950 con 0x7f52880b6450 2024-03-29T11:28:54.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:54.990+0000 7f528f200700 1 -- 172.21.15.12:0/698769177 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52880b5ef0 con 0x7f526806adb0 2024-03-29T11:28:54.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.001+0000 7f527fe00700 1 -- 172.21.15.12:0/698769177 <== mgr.9958 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289658 (secure 0 0 0) 0x7f52880b5ef0 con 0x7f526806adb0 2024-03-29T11:28:54.996 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:55.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 -- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f526806adb0 msgr2=0x7f526806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f526806adb0 0x7f526806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f527800c160 tx=0x7f527800a000 comp rx=0 tx=0).stop 2024-03-29T11:28:55.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 -- 172.21.15.12:0/698769177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52880b6450 msgr2=0x7f52880b6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52880b6450 0x7f52880b6900 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f528808c1d0 tx=0x7f5280011ed0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 -- 172.21.15.12:0/698769177 shutdown_connections 2024-03-29T11:28:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f526806adb0 0x7f526806d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52880b6450 0x7f52880b6900 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f526c003a00 0x7f5288159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 --2- 172.21.15.12:0/698769177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f528815a500 0x7f52881f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 -- 172.21.15.12:0/698769177 >> 172.21.15.12:0/698769177 conn(0x7f528808aff0 msgr2=0x7f52880b28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 -- 172.21.15.12:0/698769177 shutdown_connections 2024-03-29T11:28:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.006+0000 7f528f200700 1 -- 172.21.15.12:0/698769177 wait complete. 2024-03-29T11:28:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:28:55.015 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-29T11:28:54.310362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":616,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":383,"ondisk_log_size":383,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":432044,"kb_used_data":21572,"kb_used_omap":12,"kb_used_meta":410355,"kb_avail":754542676,"statfs":{"total":773094113280,"available":772651700224,"internally_reserved":0,"allocated":22089728,"data_stored":11310145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":420204089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.744745"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108610+0000","last_change":"2024-03-29T11:28:50.136657+0000","last_active":"2024-03-29T11:28:51.108610+0000","last_peered":"2024-03-29T11:28:51.108610+0000","last_clean":"2024-03-29T11:28:51.108610+0000","last_became_active":"2024-03-29T11:28:50.136349+0000","last_became_peered":"2024-03-29T11:28:50.136349+0000","last_unstale":"2024-03-29T11:28:51.108610+0000","last_undegraded":"2024-03-29T11:28:51.108610+0000","last_fullsized":"2024-03-29T11:28:51.108610+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:20:14.802317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108296+0000","last_change":"2024-03-29T11:28:50.118192+0000","last_active":"2024-03-29T11:28:51.108296+0000","last_peered":"2024-03-29T11:28:51.108296+0000","last_clean":"2024-03-29T11:28:51.108296+0000","last_became_active":"2024-03-29T11:28:50.117471+0000","last_became_peered":"2024-03-29T11:28:50.117471+0000","last_unstale":"2024-03-29T11:28:51.108296+0000","last_undegraded":"2024-03-29T11:28:51.108296+0000","last_fullsized":"2024-03-29T11:28:51.108296+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:22:01.558596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122122+0000","last_change":"2024-03-29T11:28:50.115075+0000","last_active":"2024-03-29T11:28:52.122122+0000","last_peered":"2024-03-29T11:28:52.122122+0000","last_clean":"2024-03-29T11:28:52.122122+0000","last_became_active":"2024-03-29T11:28:50.114782+0000","last_became_peered":"2024-03-29T11:28:50.114782+0000","last_unstale":"2024-03-29T11:28:52.122122+0000","last_undegraded":"2024-03-29T11:28:52.122122+0000","last_fullsized":"2024-03-29T11:28:52.122122+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:18:37.749501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107848+0000","last_change":"2024-03-29T11:28:50.115403+0000","last_active":"2024-03-29T11:28:51.107848+0000","last_peered":"2024-03-29T11:28:51.107848+0000","last_clean":"2024-03-29T11:28:51.107848+0000","last_became_active":"2024-03-29T11:28:50.114779+0000","last_became_peered":"2024-03-29T11:28:50.114779+0000","last_unstale":"2024-03-29T11:28:51.107848+0000","last_undegraded":"2024-03-29T11:28:51.107848+0000","last_fullsized":"2024-03-29T11:28:51.107848+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:54.062257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121655+0000","last_change":"2024-03-29T11:28:50.127626+0000","last_active":"2024-03-29T11:28:52.121655+0000","last_peered":"2024-03-29T11:28:52.121655+0000","last_clean":"2024-03-29T11:28:52.121655+0000","last_became_active":"2024-03-29T11:28:50.127387+0000","last_became_peered":"2024-03-29T11:28:50.127387+0000","last_unstale":"2024-03-29T11:28:52.121655+0000","last_undegraded":"2024-03-29T11:28:52.121655+0000","last_fullsized":"2024-03-29T11:28:52.121655+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:00:23.484211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121364+0000","last_change":"2024-03-29T11:28:50.123709+0000","last_active":"2024-03-29T11:28:52.121364+0000","last_peered":"2024-03-29T11:28:52.121364+0000","last_clean":"2024-03-29T11:28:52.121364+0000","last_became_active":"2024-03-29T11:28:50.123419+0000","last_became_peered":"2024-03-29T11:28:50.123419+0000","last_unstale":"2024-03-29T11:28:52.121364+0000","last_undegraded":"2024-03-29T11:28:52.121364+0000","last_fullsized":"2024-03-29T11:28:52.121364+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:29.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121264+0000","last_change":"2024-03-29T11:28:50.135863+0000","last_active":"2024-03-29T11:28:52.121264+0000","last_peered":"2024-03-29T11:28:52.121264+0000","last_clean":"2024-03-29T11:28:52.121264+0000","last_became_active":"2024-03-29T11:28:50.135597+0000","last_became_peered":"2024-03-29T11:28:50.135597+0000","last_unstale":"2024-03-29T11:28:52.121264+0000","last_undegraded":"2024-03-29T11:28:52.121264+0000","last_fullsized":"2024-03-29T11:28:52.121264+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:09:33.745584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108450+0000","last_change":"2024-03-29T11:28:50.117493+0000","last_active":"2024-03-29T11:28:51.108450+0000","last_peered":"2024-03-29T11:28:51.108450+0000","last_clean":"2024-03-29T11:28:51.108450+0000","last_became_active":"2024-03-29T11:28:50.117220+0000","last_became_peered":"2024-03-29T11:28:50.117220+0000","last_unstale":"2024-03-29T11:28:51.108450+0000","last_undegraded":"2024-03-29T11:28:51.108450+0000","last_fullsized":"2024-03-29T11:28:51.108450+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:32:45.260558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122118+0000","last_change":"2024-03-29T11:28:50.114782+0000","last_active":"2024-03-29T11:28:52.122118+0000","last_peered":"2024-03-29T11:28:52.122118+0000","last_clean":"2024-03-29T11:28:52.122118+0000","last_became_active":"2024-03-29T11:28:50.114516+0000","last_became_peered":"2024-03-29T11:28:50.114516+0000","last_unstale":"2024-03-29T11:28:52.122118+0000","last_undegraded":"2024-03-29T11:28:52.122118+0000","last_fullsized":"2024-03-29T11:28:52.122118+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:22:06.717277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122034+0000","last_change":"2024-03-29T11:28:50.124454+0000","last_active":"2024-03-29T11:28:52.122034+0000","last_peered":"2024-03-29T11:28:52.122034+0000","last_clean":"2024-03-29T11:28:52.122034+0000","last_became_active":"2024-03-29T11:28:50.124188+0000","last_became_peered":"2024-03-29T11:28:50.124188+0000","last_unstale":"2024-03-29T11:28:52.122034+0000","last_undegraded":"2024-03-29T11:28:52.122034+0000","last_fullsized":"2024-03-29T11:28:52.122034+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:00:23.484211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122011+0000","last_change":"2024-03-29T11:28:50.125654+0000","last_active":"2024-03-29T11:28:52.122011+0000","last_peered":"2024-03-29T11:28:52.122011+0000","last_clean":"2024-03-29T11:28:52.122011+0000","last_became_active":"2024-03-29T11:28:50.123621+0000","last_became_peered":"2024-03-29T11:28:50.123621+0000","last_unstale":"2024-03-29T11:28:52.122011+0000","last_undegraded":"2024-03-29T11:28:52.122011+0000","last_fullsized":"2024-03-29T11:28:52.122011+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:22:01.558596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107833+0000","last_change":"2024-03-29T11:28:50.127197+0000","last_active":"2024-03-29T11:28:51.107833+0000","last_peered":"2024-03-29T11:28:51.107833+0000","last_clean":"2024-03-29T11:28:51.107833+0000","last_became_active":"2024-03-29T11:28:50.126921+0000","last_became_peered":"2024-03-29T11:28:50.126921+0000","last_unstale":"2024-03-29T11:28:51.107833+0000","last_undegraded":"2024-03-29T11:28:51.107833+0000","last_fullsized":"2024-03-29T11:28:51.107833+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:38:03.520426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107734+0000","last_change":"2024-03-29T11:28:50.121346+0000","last_active":"2024-03-29T11:28:51.107734+0000","last_peered":"2024-03-29T11:28:51.107734+0000","last_clean":"2024-03-29T11:28:51.107734+0000","last_became_active":"2024-03-29T11:28:50.121079+0000","last_became_peered":"2024-03-29T11:28:50.121079+0000","last_unstale":"2024-03-29T11:28:51.107734+0000","last_undegraded":"2024-03-29T11:28:51.107734+0000","last_fullsized":"2024-03-29T11:28:51.107734+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:00.575122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121371+0000","last_change":"2024-03-29T11:28:50.129893+0000","last_active":"2024-03-29T11:28:52.121371+0000","last_peered":"2024-03-29T11:28:52.121371+0000","last_clean":"2024-03-29T11:28:52.121371+0000","last_became_active":"2024-03-29T11:28:50.129628+0000","last_became_peered":"2024-03-29T11:28:50.129628+0000","last_unstale":"2024-03-29T11:28:52.121371+0000","last_undegraded":"2024-03-29T11:28:52.121371+0000","last_fullsized":"2024-03-29T11:28:52.121371+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:54:55.931798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107854+0000","last_change":"2024-03-29T11:28:50.124047+0000","last_active":"2024-03-29T11:28:51.107854+0000","last_peered":"2024-03-29T11:28:51.107854+0000","last_clean":"2024-03-29T11:28:51.107854+0000","last_became_active":"2024-03-29T11:28:50.123782+0000","last_became_peered":"2024-03-29T11:28:50.123782+0000","last_unstale":"2024-03-29T11:28:51.107854+0000","last_undegraded":"2024-03-29T11:28:51.107854+0000","last_fullsized":"2024-03-29T11:28:51.107854+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:59.117506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108578+0000","last_change":"2024-03-29T11:28:50.115095+0000","last_active":"2024-03-29T11:28:51.108578+0000","last_peered":"2024-03-29T11:28:51.108578+0000","last_clean":"2024-03-29T11:28:51.108578+0000","last_became_active":"2024-03-29T11:28:50.114828+0000","last_became_peered":"2024-03-29T11:28:50.114828+0000","last_unstale":"2024-03-29T11:28:51.108578+0000","last_undegraded":"2024-03-29T11:28:51.108578+0000","last_fullsized":"2024-03-29T11:28:51.108578+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:45:40.534936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122583+0000","last_change":"2024-03-29T11:28:50.130157+0000","last_active":"2024-03-29T11:28:52.122583+0000","last_peered":"2024-03-29T11:28:52.122583+0000","last_clean":"2024-03-29T11:28:52.122583+0000","last_became_active":"2024-03-29T11:28:50.129188+0000","last_became_peered":"2024-03-29T11:28:50.129188+0000","last_unstale":"2024-03-29T11:28:52.122583+0000","last_undegraded":"2024-03-29T11:28:52.122583+0000","last_fullsized":"2024-03-29T11:28:52.122583+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:28.540927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108439+0000","last_change":"2024-03-29T11:28:50.128160+0000","last_active":"2024-03-29T11:28:51.108439+0000","last_peered":"2024-03-29T11:28:51.108439+0000","last_clean":"2024-03-29T11:28:51.108439+0000","last_became_active":"2024-03-29T11:28:50.127881+0000","last_became_peered":"2024-03-29T11:28:50.127881+0000","last_unstale":"2024-03-29T11:28:51.108439+0000","last_undegraded":"2024-03-29T11:28:51.108439+0000","last_fullsized":"2024-03-29T11:28:51.108439+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:18:56.271072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121271+0000","last_change":"2024-03-29T11:28:50.119640+0000","last_active":"2024-03-29T11:28:52.121271+0000","last_peered":"2024-03-29T11:28:52.121271+0000","last_clean":"2024-03-29T11:28:52.121271+0000","last_became_active":"2024-03-29T11:28:50.119350+0000","last_became_peered":"2024-03-29T11:28:50.119350+0000","last_unstale":"2024-03-29T11:28:52.121271+0000","last_undegraded":"2024-03-29T11:28:52.121271+0000","last_fullsized":"2024-03-29T11:28:52.121271+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:50.272182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121655+0000","last_change":"2024-03-29T11:28:50.126965+0000","last_active":"2024-03-29T11:28:52.121655+0000","last_peered":"2024-03-29T11:28:52.121655+0000","last_clean":"2024-03-29T11:28:52.121655+0000","last_became_active":"2024-03-29T11:28:50.126694+0000","last_became_peered":"2024-03-29T11:28:50.126694+0000","last_unstale":"2024-03-29T11:28:52.121655+0000","last_undegraded":"2024-03-29T11:28:52.121655+0000","last_fullsized":"2024-03-29T11:28:52.121655+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:12:57.697724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121768+0000","last_change":"2024-03-29T11:28:50.125404+0000","last_active":"2024-03-29T11:28:52.121768+0000","last_peered":"2024-03-29T11:28:52.121768+0000","last_clean":"2024-03-29T11:28:52.121768+0000","last_became_active":"2024-03-29T11:28:50.125162+0000","last_became_peered":"2024-03-29T11:28:50.125162+0000","last_unstale":"2024-03-29T11:28:52.121768+0000","last_undegraded":"2024-03-29T11:28:52.121768+0000","last_fullsized":"2024-03-29T11:28:52.121768+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:40.139442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121005+0000","last_change":"2024-03-29T11:28:50.127463+0000","last_active":"2024-03-29T11:28:52.121005+0000","last_peered":"2024-03-29T11:28:52.121005+0000","last_clean":"2024-03-29T11:28:52.121005+0000","last_became_active":"2024-03-29T11:28:50.126885+0000","last_became_peered":"2024-03-29T11:28:50.126885+0000","last_unstale":"2024-03-29T11:28:52.121005+0000","last_undegraded":"2024-03-29T11:28:52.121005+0000","last_fullsized":"2024-03-29T11:28:52.121005+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:12:05.408813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121343+0000","last_change":"2024-03-29T11:28:50.128429+0000","last_active":"2024-03-29T11:28:52.121343+0000","last_peered":"2024-03-29T11:28:52.121343+0000","last_clean":"2024-03-29T11:28:52.121343+0000","last_became_active":"2024-03-29T11:28:50.128142+0000","last_became_peered":"2024-03-29T11:28:50.128142+0000","last_unstale":"2024-03-29T11:28:52.121343+0000","last_undegraded":"2024-03-29T11:28:52.121343+0000","last_fullsized":"2024-03-29T11:28:52.121343+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:53:22.040285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108676+0000","last_change":"2024-03-29T11:28:50.114117+0000","last_active":"2024-03-29T11:28:51.108676+0000","last_peered":"2024-03-29T11:28:51.108676+0000","last_clean":"2024-03-29T11:28:51.108676+0000","last_became_active":"2024-03-29T11:28:50.113845+0000","last_became_peered":"2024-03-29T11:28:50.113845+0000","last_unstale":"2024-03-29T11:28:51.108676+0000","last_undegraded":"2024-03-29T11:28:51.108676+0000","last_fullsized":"2024-03-29T11:28:51.108676+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:18.491683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108343+0000","last_change":"2024-03-29T11:28:50.129416+0000","last_active":"2024-03-29T11:28:51.108343+0000","last_peered":"2024-03-29T11:28:51.108343+0000","last_clean":"2024-03-29T11:28:51.108343+0000","last_became_active":"2024-03-29T11:28:50.129025+0000","last_became_peered":"2024-03-29T11:28:50.129025+0000","last_unstale":"2024-03-29T11:28:51.108343+0000","last_undegraded":"2024-03-29T11:28:51.108343+0000","last_fullsized":"2024-03-29T11:28:51.108343+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:01.581391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122002+0000","last_change":"2024-03-29T11:28:50.128708+0000","last_active":"2024-03-29T11:28:52.122002+0000","last_peered":"2024-03-29T11:28:52.122002+0000","last_clean":"2024-03-29T11:28:52.122002+0000","last_became_active":"2024-03-29T11:28:50.128382+0000","last_became_peered":"2024-03-29T11:28:50.128382+0000","last_unstale":"2024-03-29T11:28:52.122002+0000","last_undegraded":"2024-03-29T11:28:52.122002+0000","last_fullsized":"2024-03-29T11:28:52.122002+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:33:08.316656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122219+0000","last_change":"2024-03-29T11:28:50.122564+0000","last_active":"2024-03-29T11:28:52.122219+0000","last_peered":"2024-03-29T11:28:52.122219+0000","last_clean":"2024-03-29T11:28:52.122219+0000","last_became_active":"2024-03-29T11:28:50.122289+0000","last_became_peered":"2024-03-29T11:28:50.122289+0000","last_unstale":"2024-03-29T11:28:52.122219+0000","last_undegraded":"2024-03-29T11:28:52.122219+0000","last_fullsized":"2024-03-29T11:28:52.122219+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:29:34.801860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121767+0000","last_change":"2024-03-29T11:28:50.132362+0000","last_active":"2024-03-29T11:28:52.121767+0000","last_peered":"2024-03-29T11:28:52.121767+0000","last_clean":"2024-03-29T11:28:52.121767+0000","last_became_active":"2024-03-29T11:28:50.132115+0000","last_became_peered":"2024-03-29T11:28:50.132115+0000","last_unstale":"2024-03-29T11:28:52.121767+0000","last_undegraded":"2024-03-29T11:28:52.121767+0000","last_fullsized":"2024-03-29T11:28:52.121767+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:56:00.889542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107891+0000","last_change":"2024-03-29T11:28:50.125084+0000","last_active":"2024-03-29T11:28:51.107891+0000","last_peered":"2024-03-29T11:28:51.107891+0000","last_clean":"2024-03-29T11:28:51.107891+0000","last_became_active":"2024-03-29T11:28:50.124728+0000","last_became_peered":"2024-03-29T11:28:50.124728+0000","last_unstale":"2024-03-29T11:28:51.107891+0000","last_undegraded":"2024-03-29T11:28:51.107891+0000","last_fullsized":"2024-03-29T11:28:51.107891+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:09:33.745584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121513+0000","last_change":"2024-03-29T11:28:50.120774+0000","last_active":"2024-03-29T11:28:52.121513+0000","last_peered":"2024-03-29T11:28:52.121513+0000","last_clean":"2024-03-29T11:28:52.121513+0000","last_became_active":"2024-03-29T11:28:50.120480+0000","last_became_peered":"2024-03-29T11:28:50.120480+0000","last_unstale":"2024-03-29T11:28:52.121513+0000","last_undegraded":"2024-03-29T11:28:52.121513+0000","last_fullsized":"2024-03-29T11:28:52.121513+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:47.048409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121344+0000","last_change":"2024-03-29T11:28:50.115547+0000","last_active":"2024-03-29T11:28:52.121344+0000","last_peered":"2024-03-29T11:28:52.121344+0000","last_clean":"2024-03-29T11:28:52.121344+0000","last_became_active":"2024-03-29T11:28:50.115296+0000","last_became_peered":"2024-03-29T11:28:50.115296+0000","last_unstale":"2024-03-29T11:28:52.121344+0000","last_undegraded":"2024-03-29T11:28:52.121344+0000","last_fullsized":"2024-03-29T11:28:52.121344+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:35.771515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.120945+0000","last_change":"2024-03-29T11:28:50.119485+0000","last_active":"2024-03-29T11:28:52.120945+0000","last_peered":"2024-03-29T11:28:52.120945+0000","last_clean":"2024-03-29T11:28:52.120945+0000","last_became_active":"2024-03-29T11:28:50.118790+0000","last_became_peered":"2024-03-29T11:28:50.118790+0000","last_unstale":"2024-03-29T11:28:52.120945+0000","last_undegraded":"2024-03-29T11:28:52.120945+0000","last_fullsized":"2024-03-29T11:28:52.120945+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:37:12.615160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.109156+0000","last_change":"2024-03-29T11:28:50.119766+0000","last_active":"2024-03-29T11:28:51.109156+0000","last_peered":"2024-03-29T11:28:51.109156+0000","last_clean":"2024-03-29T11:28:51.109156+0000","last_became_active":"2024-03-29T11:28:50.118403+0000","last_became_peered":"2024-03-29T11:28:50.118403+0000","last_unstale":"2024-03-29T11:28:51.109156+0000","last_undegraded":"2024-03-29T11:28:51.109156+0000","last_fullsized":"2024-03-29T11:28:51.109156+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:17:40.762889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"207'4","reported_seq":15,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.138600+0000","last_change":"2024-03-29T11:28:49.592987+0000","last_active":"2024-03-29T11:28:50.138600+0000","last_peered":"2024-03-29T11:28:50.138600+0000","last_clean":"2024-03-29T11:28:50.138600+0000","last_became_active":"2024-03-29T11:28:49.592681+0000","last_became_peered":"2024-03-29T11:28:49.592681+0000","last_unstale":"2024-03-29T11:28:50.138600+0000","last_undegraded":"2024-03-29T11:28:50.138600+0000","last_fullsized":"2024-03-29T11:28:50.138600+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:33:07.308817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122170+0000","last_change":"2024-03-29T11:28:50.123685+0000","last_active":"2024-03-29T11:28:52.122170+0000","last_peered":"2024-03-29T11:28:52.122170+0000","last_clean":"2024-03-29T11:28:52.122170+0000","last_became_active":"2024-03-29T11:28:50.121913+0000","last_became_peered":"2024-03-29T11:28:50.121913+0000","last_unstale":"2024-03-29T11:28:52.122170+0000","last_undegraded":"2024-03-29T11:28:52.122170+0000","last_fullsized":"2024-03-29T11:28:52.122170+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:15.868140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"207'1","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.123238+0000","last_change":"2024-03-29T11:28:49.588251+0000","last_active":"2024-03-29T11:28:50.123238+0000","last_peered":"2024-03-29T11:28:50.123238+0000","last_clean":"2024-03-29T11:28:50.123238+0000","last_became_active":"2024-03-29T11:28:49.587958+0000","last_became_peered":"2024-03-29T11:28:49.587958+0000","last_unstale":"2024-03-29T11:28:50.123238+0000","last_undegraded":"2024-03-29T11:28:50.123238+0000","last_fullsized":"2024-03-29T11:28:50.123238+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:56:22.069036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121613+0000","last_change":"2024-03-29T11:28:50.121894+0000","last_active":"2024-03-29T11:28:52.121613+0000","last_peered":"2024-03-29T11:28:52.121613+0000","last_clean":"2024-03-29T11:28:52.121613+0000","last_became_active":"2024-03-29T11:28:50.121173+0000","last_became_peered":"2024-03-29T11:28:50.121173+0000","last_unstale":"2024-03-29T11:28:52.121613+0000","last_undegraded":"2024-03-29T11:28:52.121613+0000","last_fullsized":"2024-03-29T11:28:52.121613+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:54.062257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.132652+0000","last_change":"2024-03-29T11:28:49.585583+0000","last_active":"2024-03-29T11:28:50.132652+0000","last_peered":"2024-03-29T11:28:50.132652+0000","last_clean":"2024-03-29T11:28:50.132652+0000","last_became_active":"2024-03-29T11:28:49.585327+0000","last_became_peered":"2024-03-29T11:28:49.585327+0000","last_unstale":"2024-03-29T11:28:50.132652+0000","last_undegraded":"2024-03-29T11:28:50.132652+0000","last_fullsized":"2024-03-29T11:28:50.132652+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:47:35.454167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121973+0000","last_change":"2024-03-29T11:28:50.121616+0000","last_active":"2024-03-29T11:28:52.121973+0000","last_peered":"2024-03-29T11:28:52.121973+0000","last_clean":"2024-03-29T11:28:52.121973+0000","last_became_active":"2024-03-29T11:28:50.119166+0000","last_became_peered":"2024-03-29T11:28:50.119166+0000","last_unstale":"2024-03-29T11:28:52.121973+0000","last_undegraded":"2024-03-29T11:28:52.121973+0000","last_fullsized":"2024-03-29T11:28:52.121973+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:01:39.484194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.129339+0000","last_change":"2024-03-29T11:28:49.590475+0000","last_active":"2024-03-29T11:28:50.129339+0000","last_peered":"2024-03-29T11:28:50.129339+0000","last_clean":"2024-03-29T11:28:50.129339+0000","last_became_active":"2024-03-29T11:28:49.590183+0000","last_became_peered":"2024-03-29T11:28:49.590183+0000","last_unstale":"2024-03-29T11:28:50.129339+0000","last_undegraded":"2024-03-29T11:28:50.129339+0000","last_fullsized":"2024-03-29T11:28:50.129339+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:34.763676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121240+0000","last_change":"2024-03-29T11:28:50.127619+0000","last_active":"2024-03-29T11:28:52.121240+0000","last_peered":"2024-03-29T11:28:52.121240+0000","last_clean":"2024-03-29T11:28:52.121240+0000","last_became_active":"2024-03-29T11:28:50.127223+0000","last_became_peered":"2024-03-29T11:28:50.127223+0000","last_unstale":"2024-03-29T11:28:52.121240+0000","last_undegraded":"2024-03-29T11:28:52.121240+0000","last_fullsized":"2024-03-29T11:28:52.121240+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:40.139442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"207'1","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.138994+0000","last_change":"2024-03-29T11:28:49.592533+0000","last_active":"2024-03-29T11:28:50.138994+0000","last_peered":"2024-03-29T11:28:50.138994+0000","last_clean":"2024-03-29T11:28:50.138994+0000","last_became_active":"2024-03-29T11:28:49.592270+0000","last_became_peered":"2024-03-29T11:28:49.592270+0000","last_unstale":"2024-03-29T11:28:50.138994+0000","last_undegraded":"2024-03-29T11:28:50.138994+0000","last_fullsized":"2024-03-29T11:28:50.138994+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:00.573552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108951+0000","last_change":"2024-03-29T11:28:50.121103+0000","last_active":"2024-03-29T11:28:51.108951+0000","last_peered":"2024-03-29T11:28:51.108951+0000","last_clean":"2024-03-29T11:28:51.108951+0000","last_became_active":"2024-03-29T11:28:50.120827+0000","last_became_peered":"2024-03-29T11:28:50.120827+0000","last_unstale":"2024-03-29T11:28:51.108951+0000","last_undegraded":"2024-03-29T11:28:51.108951+0000","last_fullsized":"2024-03-29T11:28:51.108951+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:26:24.715913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"207'1","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.129759+0000","last_change":"2024-03-29T11:28:49.588026+0000","last_active":"2024-03-29T11:28:50.129759+0000","last_peered":"2024-03-29T11:28:50.129759+0000","last_clean":"2024-03-29T11:28:50.129759+0000","last_became_active":"2024-03-29T11:28:49.586022+0000","last_became_peered":"2024-03-29T11:28:49.586022+0000","last_unstale":"2024-03-29T11:28:50.129759+0000","last_undegraded":"2024-03-29T11:28:50.129759+0000","last_fullsized":"2024-03-29T11:28:50.129759+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:24.683283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121485+0000","last_change":"2024-03-29T11:28:50.123628+0000","last_active":"2024-03-29T11:28:52.121485+0000","last_peered":"2024-03-29T11:28:52.121485+0000","last_clean":"2024-03-29T11:28:52.121485+0000","last_became_active":"2024-03-29T11:28:50.123349+0000","last_became_peered":"2024-03-29T11:28:50.123349+0000","last_unstale":"2024-03-29T11:28:52.121485+0000","last_undegraded":"2024-03-29T11:28:52.121485+0000","last_fullsized":"2024-03-29T11:28:52.121485+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:56:23.076874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.113652+0000","last_change":"2024-03-29T11:28:49.592976+0000","last_active":"2024-03-29T11:28:50.113652+0000","last_peered":"2024-03-29T11:28:50.113652+0000","last_clean":"2024-03-29T11:28:50.113652+0000","last_became_active":"2024-03-29T11:28:49.592693+0000","last_became_peered":"2024-03-29T11:28:49.592693+0000","last_unstale":"2024-03-29T11:28:50.113652+0000","last_undegraded":"2024-03-29T11:28:50.113652+0000","last_fullsized":"2024-03-29T11:28:50.113652+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:27.533088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122488+0000","last_change":"2024-03-29T11:28:50.120407+0000","last_active":"2024-03-29T11:28:52.122488+0000","last_peered":"2024-03-29T11:28:52.122488+0000","last_clean":"2024-03-29T11:28:52.122488+0000","last_became_active":"2024-03-29T11:28:50.119801+0000","last_became_peered":"2024-03-29T11:28:50.119801+0000","last_unstale":"2024-03-29T11:28:52.122488+0000","last_undegraded":"2024-03-29T11:28:52.122488+0000","last_fullsized":"2024-03-29T11:28:52.122488+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:54.413793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.110995+0000","last_change":"2024-03-29T11:28:49.587659+0000","last_active":"2024-03-29T11:28:50.110995+0000","last_peered":"2024-03-29T11:28:50.110995+0000","last_clean":"2024-03-29T11:28:50.110995+0000","last_became_active":"2024-03-29T11:28:49.587184+0000","last_became_peered":"2024-03-29T11:28:49.587184+0000","last_unstale":"2024-03-29T11:28:50.110995+0000","last_undegraded":"2024-03-29T11:28:50.110995+0000","last_fullsized":"2024-03-29T11:28:50.110995+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:53.405955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.109247+0000","last_change":"2024-03-29T11:28:50.122423+0000","last_active":"2024-03-29T11:28:51.109247+0000","last_peered":"2024-03-29T11:28:51.109247+0000","last_clean":"2024-03-29T11:28:51.109247+0000","last_became_active":"2024-03-29T11:28:50.121286+0000","last_became_peered":"2024-03-29T11:28:50.121286+0000","last_unstale":"2024-03-29T11:28:51.109247+0000","last_undegraded":"2024-03-29T11:28:51.109247+0000","last_fullsized":"2024-03-29T11:28:51.109247+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:57:06.474859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.115997+0000","last_change":"2024-03-29T11:28:49.595035+0000","last_active":"2024-03-29T11:28:50.115997+0000","last_peered":"2024-03-29T11:28:50.115997+0000","last_clean":"2024-03-29T11:28:50.115997+0000","last_became_active":"2024-03-29T11:28:49.592835+0000","last_became_peered":"2024-03-29T11:28:49.592835+0000","last_unstale":"2024-03-29T11:28:50.115997+0000","last_undegraded":"2024-03-29T11:28:50.115997+0000","last_fullsized":"2024-03-29T11:28:50.115997+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:24.687334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122002+0000","last_change":"2024-03-29T11:28:50.126055+0000","last_active":"2024-03-29T11:28:52.122002+0000","last_peered":"2024-03-29T11:28:52.122002+0000","last_clean":"2024-03-29T11:28:52.122002+0000","last_became_active":"2024-03-29T11:28:50.123504+0000","last_became_peered":"2024-03-29T11:28:50.123504+0000","last_unstale":"2024-03-29T11:28:52.122002+0000","last_undegraded":"2024-03-29T11:28:52.122002+0000","last_fullsized":"2024-03-29T11:28:52.122002+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:38:03.520426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"207'3","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.138330+0000","last_change":"2024-03-29T11:28:49.586030+0000","last_active":"2024-03-29T11:28:50.138330+0000","last_peered":"2024-03-29T11:28:50.138330+0000","last_clean":"2024-03-29T11:28:50.138330+0000","last_became_active":"2024-03-29T11:28:49.585755+0000","last_became_peered":"2024-03-29T11:28:49.585755+0000","last_unstale":"2024-03-29T11:28:50.138330+0000","last_undegraded":"2024-03-29T11:28:50.138330+0000","last_fullsized":"2024-03-29T11:28:50.138330+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:59.567283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121411+0000","last_change":"2024-03-29T11:28:50.123318+0000","last_active":"2024-03-29T11:28:52.121411+0000","last_peered":"2024-03-29T11:28:52.121411+0000","last_clean":"2024-03-29T11:28:52.121411+0000","last_became_active":"2024-03-29T11:28:50.122803+0000","last_became_peered":"2024-03-29T11:28:50.122803+0000","last_unstale":"2024-03-29T11:28:52.121411+0000","last_undegraded":"2024-03-29T11:28:52.121411+0000","last_fullsized":"2024-03-29T11:28:52.121411+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:19:05.469618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"207'6","reported_seq":17,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.140745+0000","last_change":"2024-03-29T11:28:49.589530+0000","last_active":"2024-03-29T11:28:50.140745+0000","last_peered":"2024-03-29T11:28:50.140745+0000","last_clean":"2024-03-29T11:28:50.140745+0000","last_became_active":"2024-03-29T11:28:49.588514+0000","last_became_peered":"2024-03-29T11:28:49.588514+0000","last_unstale":"2024-03-29T11:28:50.140745+0000","last_undegraded":"2024-03-29T11:28:50.140745+0000","last_fullsized":"2024-03-29T11:28:50.140745+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:16:20.743757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121524+0000","last_change":"2024-03-29T11:28:50.129973+0000","last_active":"2024-03-29T11:28:52.121524+0000","last_peered":"2024-03-29T11:28:52.121524+0000","last_clean":"2024-03-29T11:28:52.121524+0000","last_became_active":"2024-03-29T11:28:50.129689+0000","last_became_peered":"2024-03-29T11:28:50.129689+0000","last_unstale":"2024-03-29T11:28:52.121524+0000","last_undegraded":"2024-03-29T11:28:52.121524+0000","last_fullsized":"2024-03-29T11:28:52.121524+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:58:04.762023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.136780+0000","last_change":"2024-03-29T11:28:49.593870+0000","last_active":"2024-03-29T11:28:50.136780+0000","last_peered":"2024-03-29T11:28:50.136780+0000","last_clean":"2024-03-29T11:28:50.136780+0000","last_became_active":"2024-03-29T11:28:49.593606+0000","last_became_peered":"2024-03-29T11:28:49.593606+0000","last_unstale":"2024-03-29T11:28:50.136780+0000","last_undegraded":"2024-03-29T11:28:50.136780+0000","last_fullsized":"2024-03-29T11:28:50.136780+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:14.860302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121604+0000","last_change":"2024-03-29T11:28:50.129419+0000","last_active":"2024-03-29T11:28:52.121604+0000","last_peered":"2024-03-29T11:28:52.121604+0000","last_clean":"2024-03-29T11:28:52.121604+0000","last_became_active":"2024-03-29T11:28:50.128142+0000","last_became_peered":"2024-03-29T11:28:50.128142+0000","last_unstale":"2024-03-29T11:28:52.121604+0000","last_undegraded":"2024-03-29T11:28:52.121604+0000","last_fullsized":"2024-03-29T11:28:52.121604+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:59.117506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.112375+0000","last_change":"2024-03-29T11:28:49.596513+0000","last_active":"2024-03-29T11:28:50.112375+0000","last_peered":"2024-03-29T11:28:50.112375+0000","last_clean":"2024-03-29T11:28:50.112375+0000","last_became_active":"2024-03-29T11:28:49.596189+0000","last_became_peered":"2024-03-29T11:28:49.596189+0000","last_unstale":"2024-03-29T11:28:50.112375+0000","last_undegraded":"2024-03-29T11:28:50.112375+0000","last_fullsized":"2024-03-29T11:28:50.112375+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:53.054418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121721+0000","last_change":"2024-03-29T11:28:50.120384+0000","last_active":"2024-03-29T11:28:52.121721+0000","last_peered":"2024-03-29T11:28:52.121721+0000","last_clean":"2024-03-29T11:28:52.121721+0000","last_became_active":"2024-03-29T11:28:50.120076+0000","last_became_peered":"2024-03-29T11:28:50.120076+0000","last_unstale":"2024-03-29T11:28:52.121721+0000","last_undegraded":"2024-03-29T11:28:52.121721+0000","last_fullsized":"2024-03-29T11:28:52.121721+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:17:49.619853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.101737+0000","last_change":"2024-03-29T11:28:49.591931+0000","last_active":"2024-03-29T11:28:50.101737+0000","last_peered":"2024-03-29T11:28:50.101737+0000","last_clean":"2024-03-29T11:28:50.101737+0000","last_became_active":"2024-03-29T11:28:49.590295+0000","last_became_peered":"2024-03-29T11:28:49.590295+0000","last_unstale":"2024-03-29T11:28:50.101737+0000","last_undegraded":"2024-03-29T11:28:50.101737+0000","last_fullsized":"2024-03-29T11:28:50.101737+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:22:50.946710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.120865+0000","last_change":"2024-03-29T11:28:50.126351+0000","last_active":"2024-03-29T11:28:52.120865+0000","last_peered":"2024-03-29T11:28:52.120865+0000","last_clean":"2024-03-29T11:28:52.120865+0000","last_became_active":"2024-03-29T11:28:50.125201+0000","last_became_peered":"2024-03-29T11:28:50.125201+0000","last_unstale":"2024-03-29T11:28:52.120865+0000","last_undegraded":"2024-03-29T11:28:52.120865+0000","last_fullsized":"2024-03-29T11:28:52.120865+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:37:12.615160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"207'8","reported_seq":19,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.136927+0000","last_change":"2024-03-29T11:28:49.593142+0000","last_active":"2024-03-29T11:28:50.136927+0000","last_peered":"2024-03-29T11:28:50.136927+0000","last_clean":"2024-03-29T11:28:50.136927+0000","last_became_active":"2024-03-29T11:28:49.592684+0000","last_became_peered":"2024-03-29T11:28:49.592684+0000","last_unstale":"2024-03-29T11:28:50.136927+0000","last_undegraded":"2024-03-29T11:28:50.136927+0000","last_fullsized":"2024-03-29T11:28:50.136927+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:00:22.476372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122078+0000","last_change":"2024-03-29T11:28:50.126263+0000","last_active":"2024-03-29T11:28:52.122078+0000","last_peered":"2024-03-29T11:28:52.122078+0000","last_clean":"2024-03-29T11:28:52.122078+0000","last_became_active":"2024-03-29T11:28:50.125942+0000","last_became_peered":"2024-03-29T11:28:50.125942+0000","last_unstale":"2024-03-29T11:28:52.122078+0000","last_undegraded":"2024-03-29T11:28:52.122078+0000","last_fullsized":"2024-03-29T11:28:52.122078+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:56:00.889542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"207'1","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.139354+0000","last_change":"2024-03-29T11:28:49.592749+0000","last_active":"2024-03-29T11:28:50.139354+0000","last_peered":"2024-03-29T11:28:50.139354+0000","last_clean":"2024-03-29T11:28:50.139354+0000","last_became_active":"2024-03-29T11:28:49.592474+0000","last_became_peered":"2024-03-29T11:28:49.592474+0000","last_unstale":"2024-03-29T11:28:50.139354+0000","last_undegraded":"2024-03-29T11:28:50.139354+0000","last_fullsized":"2024-03-29T11:28:50.139354+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:38:02.512587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122170+0000","last_change":"2024-03-29T11:28:50.120220+0000","last_active":"2024-03-29T11:28:52.122170+0000","last_peered":"2024-03-29T11:28:52.122170+0000","last_clean":"2024-03-29T11:28:52.122170+0000","last_became_active":"2024-03-29T11:28:50.119893+0000","last_became_peered":"2024-03-29T11:28:50.119893+0000","last_unstale":"2024-03-29T11:28:52.122170+0000","last_undegraded":"2024-03-29T11:28:52.122170+0000","last_fullsized":"2024-03-29T11:28:52.122170+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:25.695173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.127794+0000","last_change":"2024-03-29T11:28:49.587442+0000","last_active":"2024-03-29T11:28:50.127794+0000","last_peered":"2024-03-29T11:28:50.127794+0000","last_clean":"2024-03-29T11:28:50.127794+0000","last_became_active":"2024-03-29T11:28:49.587094+0000","last_became_peered":"2024-03-29T11:28:49.587094+0000","last_unstale":"2024-03-29T11:28:50.127794+0000","last_undegraded":"2024-03-29T11:28:50.127794+0000","last_fullsized":"2024-03-29T11:28:50.127794+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:39.131603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121513+0000","last_change":"2024-03-29T11:28:50.129029+0000","last_active":"2024-03-29T11:28:52.121513+0000","last_peered":"2024-03-29T11:28:52.121513+0000","last_clean":"2024-03-29T11:28:52.121513+0000","last_became_active":"2024-03-29T11:28:50.128768+0000","last_became_peered":"2024-03-29T11:28:50.128768+0000","last_unstale":"2024-03-29T11:28:52.121513+0000","last_undegraded":"2024-03-29T11:28:52.121513+0000","last_fullsized":"2024-03-29T11:28:52.121513+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:56:55.880048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"207'1","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.133833+0000","last_change":"2024-03-29T11:28:49.593388+0000","last_active":"2024-03-29T11:28:50.133833+0000","last_peered":"2024-03-29T11:28:50.133833+0000","last_clean":"2024-03-29T11:28:50.133833+0000","last_became_active":"2024-03-29T11:28:49.592789+0000","last_became_peered":"2024-03-29T11:28:49.592789+0000","last_unstale":"2024-03-29T11:28:50.133833+0000","last_undegraded":"2024-03-29T11:28:50.133833+0000","last_fullsized":"2024-03-29T11:28:50.133833+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:12:56.689885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108042+0000","last_change":"2024-03-29T11:28:50.115210+0000","last_active":"2024-03-29T11:28:51.108042+0000","last_peered":"2024-03-29T11:28:51.108042+0000","last_clean":"2024-03-29T11:28:51.108042+0000","last_became_active":"2024-03-29T11:28:50.114928+0000","last_became_peered":"2024-03-29T11:28:50.114928+0000","last_unstale":"2024-03-29T11:28:51.108042+0000","last_undegraded":"2024-03-29T11:28:51.108042+0000","last_fullsized":"2024-03-29T11:28:51.108042+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:14:21.419257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.110790+0000","last_change":"2024-03-29T11:28:49.589826+0000","last_active":"2024-03-29T11:28:50.110790+0000","last_peered":"2024-03-29T11:28:50.110790+0000","last_clean":"2024-03-29T11:28:50.110790+0000","last_became_active":"2024-03-29T11:28:49.589410+0000","last_became_peered":"2024-03-29T11:28:49.589410+0000","last_unstale":"2024-03-29T11:28:50.110790+0000","last_undegraded":"2024-03-29T11:28:50.110790+0000","last_fullsized":"2024-03-29T11:28:50.110790+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:18:36.741662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121003+0000","last_change":"2024-03-29T11:28:50.121655+0000","last_active":"2024-03-29T11:28:52.121003+0000","last_peered":"2024-03-29T11:28:52.121003+0000","last_clean":"2024-03-29T11:28:52.121003+0000","last_became_active":"2024-03-29T11:28:50.121394+0000","last_became_peered":"2024-03-29T11:28:50.121394+0000","last_unstale":"2024-03-29T11:28:52.121003+0000","last_undegraded":"2024-03-29T11:28:52.121003+0000","last_fullsized":"2024-03-29T11:28:52.121003+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:12:57.697724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.119663+0000","last_change":"2024-03-29T11:28:49.595068+0000","last_active":"2024-03-29T11:28:50.119663+0000","last_peered":"2024-03-29T11:28:50.119663+0000","last_clean":"2024-03-29T11:28:50.119663+0000","last_became_active":"2024-03-29T11:28:49.594049+0000","last_became_peered":"2024-03-29T11:28:49.594049+0000","last_unstale":"2024-03-29T11:28:50.119663+0000","last_undegraded":"2024-03-29T11:28:50.119663+0000","last_fullsized":"2024-03-29T11:28:50.119663+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:44:41.821484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121718+0000","last_change":"2024-03-29T11:28:50.115276+0000","last_active":"2024-03-29T11:28:52.121718+0000","last_peered":"2024-03-29T11:28:52.121718+0000","last_clean":"2024-03-29T11:28:52.121718+0000","last_became_active":"2024-03-29T11:28:50.114053+0000","last_became_peered":"2024-03-29T11:28:50.114053+0000","last_unstale":"2024-03-29T11:28:52.121718+0000","last_undegraded":"2024-03-29T11:28:52.121718+0000","last_fullsized":"2024-03-29T11:28:52.121718+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:14:21.419257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.108899+0000","last_change":"2024-03-29T11:28:49.591960+0000","last_active":"2024-03-29T11:28:50.108899+0000","last_peered":"2024-03-29T11:28:50.108899+0000","last_clean":"2024-03-29T11:28:50.108899+0000","last_became_active":"2024-03-29T11:28:49.591696+0000","last_became_peered":"2024-03-29T11:28:49.591696+0000","last_unstale":"2024-03-29T11:28:50.108899+0000","last_undegraded":"2024-03-29T11:28:50.108899+0000","last_fullsized":"2024-03-29T11:28:50.108899+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:53.054418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107599+0000","last_change":"2024-03-29T11:28:50.127864+0000","last_active":"2024-03-29T11:28:51.107599+0000","last_peered":"2024-03-29T11:28:51.107599+0000","last_clean":"2024-03-29T11:28:51.107599+0000","last_became_active":"2024-03-29T11:28:50.127558+0000","last_became_peered":"2024-03-29T11:28:50.127558+0000","last_unstale":"2024-03-29T11:28:51.107599+0000","last_undegraded":"2024-03-29T11:28:51.107599+0000","last_fullsized":"2024-03-29T11:28:51.107599+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:33:08.316656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.106933+0000","last_change":"2024-03-29T11:28:49.591597+0000","last_active":"2024-03-29T11:28:50.106933+0000","last_peered":"2024-03-29T11:28:50.106933+0000","last_clean":"2024-03-29T11:28:50.106933+0000","last_became_active":"2024-03-29T11:28:49.591333+0000","last_became_peered":"2024-03-29T11:28:49.591333+0000","last_unstale":"2024-03-29T11:28:50.106933+0000","last_undegraded":"2024-03-29T11:28:50.106933+0000","last_fullsized":"2024-03-29T11:28:50.106933+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:57:18.331610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121202+0000","last_change":"2024-03-29T11:28:50.121034+0000","last_active":"2024-03-29T11:28:52.121202+0000","last_peered":"2024-03-29T11:28:52.121202+0000","last_clean":"2024-03-29T11:28:52.121202+0000","last_became_active":"2024-03-29T11:28:50.120761+0000","last_became_peered":"2024-03-29T11:28:50.120761+0000","last_unstale":"2024-03-29T11:28:52.121202+0000","last_undegraded":"2024-03-29T11:28:52.121202+0000","last_fullsized":"2024-03-29T11:28:52.121202+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:19:05.469618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.109525+0000","last_change":"2024-03-29T11:28:49.592219+0000","last_active":"2024-03-29T11:28:50.109525+0000","last_peered":"2024-03-29T11:28:50.109525+0000","last_clean":"2024-03-29T11:28:50.109525+0000","last_became_active":"2024-03-29T11:28:49.591806+0000","last_became_peered":"2024-03-29T11:28:49.591806+0000","last_unstale":"2024-03-29T11:28:50.109525+0000","last_undegraded":"2024-03-29T11:28:50.109525+0000","last_fullsized":"2024-03-29T11:28:50.109525+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:32.966697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122279+0000","last_change":"2024-03-29T11:28:50.121269+0000","last_active":"2024-03-29T11:28:52.122279+0000","last_peered":"2024-03-29T11:28:52.122279+0000","last_clean":"2024-03-29T11:28:52.122279+0000","last_became_active":"2024-03-29T11:28:50.121033+0000","last_became_peered":"2024-03-29T11:28:50.121033+0000","last_unstale":"2024-03-29T11:28:52.122279+0000","last_undegraded":"2024-03-29T11:28:52.122279+0000","last_fullsized":"2024-03-29T11:28:52.122279+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:12:05.408813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.136977+0000","last_change":"2024-03-29T11:28:49.596075+0000","last_active":"2024-03-29T11:28:50.136977+0000","last_peered":"2024-03-29T11:28:50.136977+0000","last_clean":"2024-03-29T11:28:50.136977+0000","last_became_active":"2024-03-29T11:28:49.595756+0000","last_became_peered":"2024-03-29T11:28:49.595756+0000","last_unstale":"2024-03-29T11:28:50.136977+0000","last_undegraded":"2024-03-29T11:28:50.136977+0000","last_fullsized":"2024-03-29T11:28:50.136977+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:40:30.202152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122221+0000","last_change":"2024-03-29T11:28:50.133779+0000","last_active":"2024-03-29T11:28:52.122221+0000","last_peered":"2024-03-29T11:28:52.122221+0000","last_clean":"2024-03-29T11:28:52.122221+0000","last_became_active":"2024-03-29T11:28:50.133293+0000","last_became_peered":"2024-03-29T11:28:50.133293+0000","last_unstale":"2024-03-29T11:28:52.122221+0000","last_undegraded":"2024-03-29T11:28:52.122221+0000","last_fullsized":"2024-03-29T11:28:52.122221+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:44.130979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"208'1","reported_seq":12,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122124+0000","last_change":"2024-03-29T11:28:49.589012+0000","last_active":"2024-03-29T11:28:52.122124+0000","last_peered":"2024-03-29T11:28:52.122124+0000","last_clean":"2024-03-29T11:28:52.122124+0000","last_became_active":"2024-03-29T11:28:49.588740+0000","last_became_peered":"2024-03-29T11:28:49.588740+0000","last_unstale":"2024-03-29T11:28:52.122124+0000","last_undegraded":"2024-03-29T11:28:52.122124+0000","last_fullsized":"2024-03-29T11:28:52.122124+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:09:32.737746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.109026+0000","last_change":"2024-03-29T11:28:50.122423+0000","last_active":"2024-03-29T11:28:51.109026+0000","last_peered":"2024-03-29T11:28:51.109026+0000","last_clean":"2024-03-29T11:28:51.109026+0000","last_became_active":"2024-03-29T11:28:50.122157+0000","last_became_peered":"2024-03-29T11:28:50.122157+0000","last_unstale":"2024-03-29T11:28:51.109026+0000","last_undegraded":"2024-03-29T11:28:51.109026+0000","last_fullsized":"2024-03-29T11:28:51.109026+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:35:38.671731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.110065+0000","last_change":"2024-03-29T11:28:49.590468+0000","last_active":"2024-03-29T11:28:50.110065+0000","last_peered":"2024-03-29T11:28:50.110065+0000","last_clean":"2024-03-29T11:28:50.110065+0000","last_became_active":"2024-03-29T11:28:49.590196+0000","last_became_peered":"2024-03-29T11:28:49.590196+0000","last_unstale":"2024-03-29T11:28:50.110065+0000","last_undegraded":"2024-03-29T11:28:50.110065+0000","last_fullsized":"2024-03-29T11:28:50.110065+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:33:07.308817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.109189+0000","last_change":"2024-03-29T11:28:50.113553+0000","last_active":"2024-03-29T11:28:51.109189+0000","last_peered":"2024-03-29T11:28:51.109189+0000","last_clean":"2024-03-29T11:28:51.109189+0000","last_became_active":"2024-03-29T11:28:50.113276+0000","last_became_peered":"2024-03-29T11:28:50.113276+0000","last_unstale":"2024-03-29T11:28:51.109189+0000","last_undegraded":"2024-03-29T11:28:51.109189+0000","last_fullsized":"2024-03-29T11:28:51.109189+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:52:47.722790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.102176+0000","last_change":"2024-03-29T11:28:49.586785+0000","last_active":"2024-03-29T11:28:50.102176+0000","last_peered":"2024-03-29T11:28:50.102176+0000","last_clean":"2024-03-29T11:28:50.102176+0000","last_became_active":"2024-03-29T11:28:49.586409+0000","last_became_peered":"2024-03-29T11:28:49.586409+0000","last_unstale":"2024-03-29T11:28:50.102176+0000","last_undegraded":"2024-03-29T11:28:50.102176+0000","last_fullsized":"2024-03-29T11:28:50.102176+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:49.264343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108285+0000","last_change":"2024-03-29T11:28:50.132786+0000","last_active":"2024-03-29T11:28:51.108285+0000","last_peered":"2024-03-29T11:28:51.108285+0000","last_clean":"2024-03-29T11:28:51.108285+0000","last_became_active":"2024-03-29T11:28:50.132492+0000","last_became_peered":"2024-03-29T11:28:50.132492+0000","last_unstale":"2024-03-29T11:28:51.108285+0000","last_undegraded":"2024-03-29T11:28:51.108285+0000","last_fullsized":"2024-03-29T11:28:51.108285+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:32:45.260558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.109748+0000","last_change":"2024-03-29T11:28:49.581723+0000","last_active":"2024-03-29T11:28:50.109748+0000","last_peered":"2024-03-29T11:28:50.109748+0000","last_clean":"2024-03-29T11:28:50.109748+0000","last_became_active":"2024-03-29T11:28:49.581336+0000","last_became_peered":"2024-03-29T11:28:49.581336+0000","last_unstale":"2024-03-29T11:28:50.109748+0000","last_undegraded":"2024-03-29T11:28:50.109748+0000","last_fullsized":"2024-03-29T11:28:50.109748+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:22:00.550757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108547+0000","last_change":"2024-03-29T11:28:50.128076+0000","last_active":"2024-03-29T11:28:51.108547+0000","last_peered":"2024-03-29T11:28:51.108547+0000","last_clean":"2024-03-29T11:28:51.108547+0000","last_became_active":"2024-03-29T11:28:50.127795+0000","last_became_peered":"2024-03-29T11:28:50.127795+0000","last_unstale":"2024-03-29T11:28:51.108547+0000","last_undegraded":"2024-03-29T11:28:51.108547+0000","last_fullsized":"2024-03-29T11:28:51.108547+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:18.491683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.109884+0000","last_change":"2024-03-29T11:28:49.590029+0000","last_active":"2024-03-29T11:28:50.109884+0000","last_peered":"2024-03-29T11:28:50.109884+0000","last_clean":"2024-03-29T11:28:50.109884+0000","last_became_active":"2024-03-29T11:28:49.589322+0000","last_became_peered":"2024-03-29T11:28:49.589322+0000","last_unstale":"2024-03-29T11:28:50.109884+0000","last_undegraded":"2024-03-29T11:28:50.109884+0000","last_fullsized":"2024-03-29T11:28:50.109884+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:40.940699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107888+0000","last_change":"2024-03-29T11:28:50.130534+0000","last_active":"2024-03-29T11:28:51.107888+0000","last_peered":"2024-03-29T11:28:51.107888+0000","last_clean":"2024-03-29T11:28:51.107888+0000","last_became_active":"2024-03-29T11:28:50.130284+0000","last_became_peered":"2024-03-29T11:28:50.130284+0000","last_unstale":"2024-03-29T11:28:51.107888+0000","last_undegraded":"2024-03-29T11:28:51.107888+0000","last_fullsized":"2024-03-29T11:28:51.107888+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:56:23.076874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.110026+0000","last_change":"2024-03-29T11:28:49.586706+0000","last_active":"2024-03-29T11:28:50.110026+0000","last_peered":"2024-03-29T11:28:50.110026+0000","last_clean":"2024-03-29T11:28:50.110026+0000","last_became_active":"2024-03-29T11:28:49.586218+0000","last_became_peered":"2024-03-29T11:28:49.586218+0000","last_unstale":"2024-03-29T11:28:50.110026+0000","last_undegraded":"2024-03-29T11:28:50.110026+0000","last_fullsized":"2024-03-29T11:28:50.110026+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:29:33.794021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121270+0000","last_change":"2024-03-29T11:28:50.127095+0000","last_active":"2024-03-29T11:28:52.121270+0000","last_peered":"2024-03-29T11:28:52.121270+0000","last_clean":"2024-03-29T11:28:52.121270+0000","last_became_active":"2024-03-29T11:28:50.126842+0000","last_became_peered":"2024-03-29T11:28:50.126842+0000","last_unstale":"2024-03-29T11:28:52.121270+0000","last_undegraded":"2024-03-29T11:28:52.121270+0000","last_fullsized":"2024-03-29T11:28:52.121270+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:44.130979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.107653+0000","last_change":"2024-03-29T11:28:49.591978+0000","last_active":"2024-03-29T11:28:50.107653+0000","last_peered":"2024-03-29T11:28:50.107653+0000","last_clean":"2024-03-29T11:28:50.107653+0000","last_became_active":"2024-03-29T11:28:49.591702+0000","last_became_peered":"2024-03-29T11:28:49.591702+0000","last_unstale":"2024-03-29T11:28:50.107653+0000","last_undegraded":"2024-03-29T11:28:50.107653+0000","last_fullsized":"2024-03-29T11:28:50.107653+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:06:43.501520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.120603+0000","last_change":"2024-03-29T11:28:50.116491+0000","last_active":"2024-03-29T11:28:52.120603+0000","last_peered":"2024-03-29T11:28:52.120603+0000","last_clean":"2024-03-29T11:28:52.120603+0000","last_became_active":"2024-03-29T11:28:50.115323+0000","last_became_peered":"2024-03-29T11:28:50.115323+0000","last_unstale":"2024-03-29T11:28:52.120603+0000","last_undegraded":"2024-03-29T11:28:52.120603+0000","last_fullsized":"2024-03-29T11:28:52.120603+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:28.540927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.121975+0000","last_change":"2024-03-29T11:28:49.593317+0000","last_active":"2024-03-29T11:28:50.121975+0000","last_peered":"2024-03-29T11:28:50.121975+0000","last_clean":"2024-03-29T11:28:50.121975+0000","last_became_active":"2024-03-29T11:28:49.592981+0000","last_became_peered":"2024-03-29T11:28:49.592981+0000","last_unstale":"2024-03-29T11:28:50.121975+0000","last_undegraded":"2024-03-29T11:28:50.121975+0000","last_fullsized":"2024-03-29T11:28:50.121975+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.123140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"197'344","reported_seq":1175,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.113849+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:28:50.113849+0000","last_peered":"2024-03-29T11:28:50.113849+0000","last_clean":"2024-03-29T11:28:50.113849+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:28:50.113849+0000","last_undegraded":"2024-03-29T11:28:50.113849+0000","last_fullsized":"2024-03-29T11:28:50.113849+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":571,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":571,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":344,"ondisk_log_size":344,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575965,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52296,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319544,"statfs":{"total":96636764160,"available":96583213056,"internally_reserved":0,"allocated":3207168,"data_stored":1856185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":50330060},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.54200000000000004,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.1459999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.749,"15min":0.751},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.40300000000000002},"max":{"1min":0.96999999999999997,"5min":1.119,"15min":1.119},"last":0.85799999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73999999999999999,"15min":0.73699999999999999},"min":{"1min":0.51400000000000001,"5min":0.45000000000000001,"15min":0.44700000000000001},"max":{"1min":0.878,"5min":1.478,"15min":1.478},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77300000000000002,"15min":0.78000000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.47699999999999998},"max":{"1min":0.98399999999999999,"5min":1.393,"15min":1.393},"last":0.76300000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.90200000000000002,"5min":1.089,"15min":1.284},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.78000000000000003,"15min":0.78600000000000003},"min":{"1min":0.52800000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.90200000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.77900000000000003,"15min":0.76900000000000002},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.47799999999999998},"max":{"1min":0.94899999999999995,"5min":1.0580000000000001,"15min":1.069},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.78400000000000003,"15min":0.79400000000000004},"min":{"1min":0.65500000000000003,"5min":0.55300000000000005,"15min":0.52000000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.3169999999999999},"last":0.94999999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.63700000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41999999999999998},"max":{"1min":0.87,"5min":1.024,"15min":1.3779999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.48399999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83799999999999997,"5min":0.95299999999999996,"15min":1.335},"last":0.71099999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.50800000000000001,"5min":0.41999999999999998,"15min":0.38400000000000001},"max":{"1min":0.94299999999999995,"5min":0.95999999999999996,"15min":1.347},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70399999999999996,"15min":0.70899999999999996},"min":{"1min":0.57099999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94299999999999995,"5min":0.96099999999999997,"15min":1.1759999999999999},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.754,"15min":0.76500000000000001},"min":{"1min":0.53800000000000003,"5min":0.52500000000000002,"15min":0.47899999999999998},"max":{"1min":0.98899999999999999,"5min":0.99099999999999999,"15min":1.2070000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.53500000000000003,"5min":0.442,"15min":0.435},"max":{"1min":0.97499999999999998,"5min":1.0640000000000001,"15min":1.069},"last":0.83599999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575966,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45948,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325892,"statfs":{"total":96636764160,"available":96589713408,"internally_reserved":0,"allocated":2605056,"data_stored":1265713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44431815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72199999999999998,"15min":0.753},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.76100000000000001,"15min":0.77600000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.747,"15min":0.75},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.44800000000000001},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.70099999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.55900000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76600000000000001,"15min":0.77600000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.46200000000000002},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.76000000000000001,"15min":0.78400000000000003},"min":{"1min":0.57999999999999996,"5min":0.54400000000000004,"15min":0.499},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":0.622},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.47799999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.70899999999999996,"15min":0.70699999999999996},"min":{"1min":0.53700000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.6629999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":0.67500000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66300000000000003,"15min":0.65400000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70799999999999996,"15min":0.70899999999999996},"min":{"1min":0.46999999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.76300000000000001,"15min":0.76500000000000001},"min":{"1min":0.54800000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.7050000000000001,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73799999999999999,"15min":0.74199999999999999},"min":{"1min":0.504,"5min":0.499,"15min":0.47199999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.76900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575964,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59912,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311928,"statfs":{"total":96636764160,"available":96575414272,"internally_reserved":0,"allocated":2617344,"data_stored":1267277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.60899999999999999,"5min":0.80300000000000005,"15min":0.81100000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52900000000000003,"15min":0.54200000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.625,"5min":0.85299999999999998,"15min":0.89000000000000001},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.42899999999999999,"5min":0.40500000000000003,"15min":0.40000000000000002},"max":{"1min":0.76100000000000001,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52500000000000002,"15min":0.54100000000000004},"min":{"1min":0.36199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.90300000000000002},"last":0.373}]},{"osd":3,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59599999999999997,"15min":0.60699999999999998},"min":{"1min":0.45300000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86199999999999999,"5min":0.876,"15min":0.995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.36499999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.878},"last":0.52700000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.78200000000000003,"15min":0.80600000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.97999999999999998,"5min":1.119,"15min":1.3660000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78600000000000003,"15min":0.80900000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.93200000000000005,"5min":1.304,"15min":1.304},"last":0.70299999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.79800000000000004,"15min":0.82499999999999996},"min":{"1min":0.57799999999999996,"5min":0.57699999999999996,"15min":0.48199999999999998},"max":{"1min":0.93999999999999995,"5min":1.3380000000000001,"15min":1.3720000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.82699999999999996,"15min":0.84999999999999998},"min":{"1min":0.65500000000000003,"5min":0.61299999999999999,"15min":0.59099999999999997},"max":{"1min":1.109,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.67500000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.80700000000000005,"15min":0.83199999999999996},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.98199999999999998,"5min":1.175,"15min":1.1919999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82999999999999996,"15min":0.85799999999999998},"min":{"1min":0.61599999999999999,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.0129999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"last":0.78200000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.84899999999999998,"15min":0.88200000000000001},"min":{"1min":0.66500000000000004,"5min":0.626,"15min":0.61199999999999999},"max":{"1min":1.0700000000000001,"5min":1.1479999999999999,"15min":1.3540000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.85699999999999998,"15min":0.88},"min":{"1min":0.60299999999999998,"5min":0.59999999999999998,"15min":0.59699999999999998},"max":{"1min":1.056,"5min":1.22,"15min":1.3420000000000001},"last":0.78800000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575966,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60480,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311360,"statfs":{"total":96636764160,"available":96574832640,"internally_reserved":0,"allocated":3198976,"data_stored":1855084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46999999999999997},"min":{"1min":0.308,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.087},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55600000000000005,"15min":0.54700000000000004},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.32400000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.38200000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.36099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.5289999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33200000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.63300000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.45300000000000001,"5min":0.40899999999999997,"15min":0.36299999999999999},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65800000000000003,"15min":0.64700000000000002},"min":{"1min":0.53000000000000003,"5min":0.46000000000000002,"15min":0.42199999999999999},"max":{"1min":0.83999999999999997,"5min":0.90000000000000002,"15min":1.3899999999999999},"last":0.58399999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.81899999999999995,"15min":0.81799999999999995},"min":{"1min":0.58699999999999997,"5min":0.495,"15min":0.495},"max":{"1min":1.0960000000000001,"5min":1.232,"15min":1.232},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.86099999999999999,"15min":0.85399999999999998},"min":{"1min":0.66800000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.99299999999999999,"5min":1.1890000000000001,"15min":1.5820000000000001},"last":0.72099999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.79900000000000004,"15min":0.78900000000000003},"min":{"1min":0.65700000000000003,"5min":0.53600000000000003,"15min":0.51200000000000001},"max":{"1min":1.0700000000000001,"5min":1.1870000000000001,"15min":1.3440000000000001},"last":0.754},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.82599999999999996},"min":{"1min":0.60999999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.73399999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.873,"15min":0.86199999999999999},"min":{"1min":0.63100000000000001,"5min":0.62,"15min":0.55000000000000004},"max":{"1min":1.343,"5min":1.343,"15min":1.6240000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.89400000000000002,"15min":0.89100000000000001},"min":{"1min":0.58499999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3240000000000001,"5min":1.337,"15min":1.337},"last":0.82199999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.83699999999999997,"15min":0.83099999999999996},"min":{"1min":0.629,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.115,"5min":1.214,"15min":1.214},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89700000000000002,"15min":0.88600000000000001},"min":{"1min":0.66500000000000004,"5min":0.66300000000000003,"15min":0.60399999999999998},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.67900000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575965,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47628,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324212,"statfs":{"total":96636764160,"available":96587993088,"internally_reserved":0,"allocated":2617344,"data_stored":1266407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47299999999999998,"15min":0.47599999999999998},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.35499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68700000000000006,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.53100000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48499999999999999,"15min":0.48799999999999999},"min":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.746,"5min":0.79000000000000004,"15min":0.80700000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.58799999999999997,"15min":0.58499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36199999999999999},"max":{"1min":0.77200000000000002,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61099999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60199999999999998,"15min":0.61099999999999999},"min":{"1min":0.45200000000000001,"5min":0.41999999999999998,"15min":0.41799999999999998},"max":{"1min":0.75600000000000001,"5min":0.80700000000000005,"15min":0.90400000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58099999999999996,"15min":0.58799999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.68799999999999994,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.59699999999999998,"5min":0.58699999999999997,"15min":0.55100000000000005},"max":{"1min":1.091,"5min":1.169,"15min":1.1970000000000001},"last":0.624},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81899999999999995,"15min":0.82699999999999996},"min":{"1min":0.63,"5min":0.57299999999999995,"15min":0.52400000000000002},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.67100000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.81499999999999995,"15min":0.83299999999999996},"min":{"1min":0.63900000000000001,"5min":0.56499999999999995,"15min":0.53400000000000003},"max":{"1min":1.1599999999999999,"5min":1.256,"15min":1.256},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.83799999999999997,"15min":0.83699999999999997},"min":{"1min":0.60899999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.143,"5min":1.24,"15min":1.3140000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.86599999999999999,"15min":0.85999999999999999},"min":{"1min":0.59599999999999997,"5min":0.58899999999999997,"15min":0.57799999999999996},"max":{"1min":1.1459999999999999,"5min":1.208,"15min":1.24},"last":0.871},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85899999999999999,"15min":0.85999999999999999},"min":{"1min":0.71599999999999997,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.075,"5min":1.1879999999999999,"15min":1.329},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.71599999999999997,"5min":0.65100000000000002,"15min":0.64600000000000002},"max":{"1min":1.1910000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.66400000000000003,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.0980000000000001,"5min":1.304,"15min":1.304},"last":0.82399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575965,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64008,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307832,"statfs":{"total":96636764160,"available":96571219968,"internally_reserved":0,"allocated":2617344,"data_stored":1265907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":62912968},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.35399999999999998,"5min":0.33000000000000002,"15min":0.313},"max":{"1min":0.55500000000000005,"5min":0.748,"15min":0.75900000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.318},"max":{"1min":0.82299999999999995,"5min":0.84799999999999998,"15min":0.96099999999999997},"last":0.40899999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.498,"15min":0.499},"min":{"1min":0.35899999999999999,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":0.58499999999999996,"5min":0.71599999999999997,"15min":0.77100000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.503,"5min":0.57299999999999995,"15min":0.57899999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.65900000000000003,"5min":1.046,"15min":1.046},"last":0.51600000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.44,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":0.67300000000000004,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.66000000000000003,"15min":0.66400000000000003},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.442},"max":{"1min":0.77900000000000003,"5min":0.89700000000000002,"15min":1.01},"last":0.58599999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.79800000000000004},"min":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.1339999999999999,"5min":1.266,"15min":1.2889999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81699999999999995,"15min":0.81599999999999995},"min":{"1min":0.65900000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0680000000000001,"5min":1.1830000000000001,"15min":1.216},"last":0.80900000000000005}]},{"osd":5,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.81499999999999995,"15min":0.81200000000000006},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.502},"max":{"1min":1.1160000000000001,"5min":1.1399999999999999,"15min":1.252},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.81000000000000005,"15min":0.82599999999999996},"min":{"1min":0.66500000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.98799999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.77500000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.84499999999999997,"15min":0.85499999999999998},"min":{"1min":0.71599999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.095,"5min":1.288,"15min":1.288},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.63200000000000001,"5min":0.63100000000000001,"15min":0.59099999999999997},"max":{"1min":1.0289999999999999,"5min":1.248,"15min":1.248},"last":1.147}]},{"osd":7,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85799999999999998,"15min":0.86599999999999999},"min":{"1min":0.54500000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1870000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":1.052},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.88100000000000001,"15min":0.89200000000000002},"min":{"1min":0.69999999999999996,"5min":0.58099999999999996,"15min":0.57899999999999996},"max":{"1min":1.0289999999999999,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":1.113}]}]},{"osd":4,"up_from":13,"seq":55834575966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59916,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311924,"statfs":{"total":96636764160,"available":96575410176,"internally_reserved":0,"allocated":2621440,"data_stored":1267859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72899999999999998,"15min":0.73699999999999999},"min":{"1min":0.503,"5min":0.39900000000000002,"15min":0.371},"max":{"1min":0.95499999999999996,"5min":1.0309999999999999,"15min":1.0509999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.53500000000000003,"5min":0.49099999999999999,"15min":0.38600000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.1459999999999999},"last":0.64300000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.75,"15min":0.76500000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.39900000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.74399999999999999,"15min":0.76100000000000001},"min":{"1min":0.54500000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.002},"last":0.59799999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.82799999999999996,"15min":0.84299999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82199999999999995,"15min":0.82699999999999996},"min":{"1min":0.67700000000000005,"5min":0.58099999999999996,"15min":0.53900000000000003},"max":{"1min":1.01,"5min":1.0529999999999999,"15min":1.1299999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.76700000000000002,"15min":0.77900000000000003},"min":{"1min":0.54800000000000004,"5min":0.52200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.181},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.78400000000000003,"15min":0.77700000000000002},"min":{"1min":0.55300000000000005,"5min":0.51300000000000001,"15min":0.48699999999999999},"max":{"1min":0.995,"5min":0.995,"15min":1.2490000000000001},"last":0.84399999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.626},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82599999999999996,"5min":0.93000000000000005,"15min":1.004},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.64900000000000002},"min":{"1min":0.45500000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.84999999999999998,"5min":1.071,"15min":1.169},"last":0.79000000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66200000000000003},"min":{"1min":0.436,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0800000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70199999999999996,"15min":0.71399999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.44700000000000001},"max":{"1min":1.0089999999999999,"5min":1.024,"15min":1.028},"last":0.71399999999999997}]},{"osd":7,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.71499999999999997,"15min":0.72499999999999998},"min":{"1min":0.56000000000000005,"5min":0.436,"15min":0.436},"max":{"1min":0.96699999999999997,"5min":1.052,"15min":1.518},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81899999999999995,"15min":0.82399999999999995},"min":{"1min":0.65400000000000003,"5min":0.54400000000000004,"15min":0.54300000000000004},"max":{"1min":1.0289999999999999,"5min":1.0860000000000001,"15min":1.3440000000000001},"last":0.877}]}]},{"osd":5,"up_from":13,"seq":55834575965,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41856,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329984,"statfs":{"total":96636764160,"available":96593903616,"internally_reserved":0,"allocated":2605056,"data_stored":1265713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.72399999999999998,"15min":0.71999999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86599999999999999,"5min":0.95999999999999996,"15min":1.0149999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.745,"15min":0.749},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.92700000000000005,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77500000000000002,"15min":0.76800000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92500000000000004,"5min":0.997,"15min":1.0740000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.753,"15min":0.751},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.47399999999999998},"max":{"1min":0.93999999999999995,"5min":0.98999999999999999,"15min":1.085},"last":0.504}]},{"osd":2,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77300000000000002,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.42799999999999999},"max":{"1min":0.95299999999999996,"5min":1.04,"15min":1.052},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.76200000000000001,"15min":0.75900000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.48399999999999999},"max":{"1min":0.94399999999999995,"5min":1.0129999999999999,"15min":1.0740000000000001},"last":0.79200000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.502,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.92200000000000004,"5min":1.077,"15min":1.0980000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77700000000000002,"15min":0.78100000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.51800000000000002},"max":{"1min":0.88100000000000001,"5min":1.046,"15min":1.046},"last":0.63400000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.38700000000000001},"max":{"1min":0.76100000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64400000000000002,"15min":0.63800000000000001},"min":{"1min":0.45100000000000001,"5min":0.45000000000000001,"15min":0.40600000000000003},"max":{"1min":0.76700000000000002,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42699999999999999},"max":{"1min":0.77800000000000002,"5min":1.038,"15min":1.038},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.42499999999999999},"max":{"1min":0.90900000000000003,"5min":1.107,"15min":1.107},"last":0.89400000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.72899999999999998,"15min":0.71199999999999997},"min":{"1min":0.52900000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88400000000000001,"5min":1.129,"15min":1.153},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.76900000000000002,"15min":0.75600000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1539999999999999},"last":0.82099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:28:55.022 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:28:55.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.263+0000 7f4e34800700 1 Processor -- start 2024-03-29T11:28:55.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.263+0000 7f4e34800700 1 -- start start 2024-03-29T11:28:55.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.264+0000 7f4e34800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4e2c072580 0x7f4e2c0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.265+0000 7f4e34800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c0b2e50 0x7f4e2c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.265+0000 7f4e34800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c0b2180 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.265+0000 7f4e34800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2c15b100 con 0x7f4e2c072580 2024-03-29T11:28:55.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.265+0000 7f4e34800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c15b3e0 con 0x7f4e2c0b2e50 2024-03-29T11:28:55.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.265+0000 7f4e32800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c0b2e50 0x7f4e2c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.265+0000 7f4e31400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4e2c072580 0x7f4e2c0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.265+0000 7f4e32800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c0b2e50 0x7f4e2c0709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:56784/0 (socket says 172.21.15.12:56784) 2024-03-29T11:28:55.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.265+0000 7f4e32800700 1 -- 172.21.15.12:0/2322716828 learned_addr learned my addr 172.21.15.12:0/2322716828 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:55.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.266+0000 7f4e32800700 1 -- 172.21.15.12:0/2322716828 >> v2:172.21.15.159:3300/0 conn(0x7f4e2c072580 msgr2=0x7f4e2c0b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:55.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.266+0000 7f4e32800700 1 --2- 172.21.15.12:0/2322716828 >> v2:172.21.15.159:3300/0 conn(0x7f4e2c072580 0x7f4e2c0b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.266+0000 7f4e32800700 1 -- 172.21.15.12:0/2322716828 >> v1:172.21.15.159:6789/0 conn(0x7f4e2c0b3af0 legacy=0x7f4e2c15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.266+0000 7f4e32800700 1 -- 172.21.15.12:0/2322716828 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c0b23f0 con 0x7f4e2c0b2e50 2024-03-29T11:28:55.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.266+0000 7f4e31400700 1 --2- 172.21.15.12:0/2322716828 >> v2:172.21.15.159:3300/0 conn(0x7f4e2c072580 0x7f4e2c0b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:55.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.266+0000 7f4e32800700 1 --2- 172.21.15.12:0/2322716828 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c0b2e50 0x7f4e2c0709c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800d150 tx=0x7f4e2800d460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b67a1a173aa859a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:55.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.266+0000 7f4e30a00700 1 -- 172.21.15.12:0/2322716828 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28010070 con 0x7f4e2c0b2e50 2024-03-29T11:28:55.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.266+0000 7f4e30a00700 1 -- 172.21.15.12:0/2322716828 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2800dd10 con 0x7f4e2c0b2e50 2024-03-29T11:28:55.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.266+0000 7f4e30a00700 1 -- 172.21.15.12:0/2322716828 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2800b310 con 0x7f4e2c0b2e50 2024-03-29T11:28:55.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.267+0000 7f4e34800700 1 -- 172.21.15.12:0/2322716828 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c0b2e50 msgr2=0x7f4e2c0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.267+0000 7f4e34800700 1 --2- 172.21.15.12:0/2322716828 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c0b2e50 0x7f4e2c0709c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800d150 tx=0x7f4e2800d460 comp rx=0 tx=0).stop 2024-03-29T11:28:55.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.267+0000 7f4e34800700 1 -- 172.21.15.12:0/2322716828 shutdown_connections 2024-03-29T11:28:55.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.267+0000 7f4e34800700 1 --2- 172.21.15.12:0/2322716828 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c0b2e50 0x7f4e2c0709c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.267+0000 7f4e34800700 1 --2- 172.21.15.12:0/2322716828 >> v2:172.21.15.159:3300/0 conn(0x7f4e2c072580 0x7f4e2c0b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.267+0000 7f4e34800700 1 -- 172.21.15.12:0/2322716828 >> 172.21.15.12:0/2322716828 conn(0x7f4e2c0cf190 msgr2=0x7f4e2c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:55.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.267+0000 7f4e34800700 1 -- 172.21.15.12:0/2322716828 shutdown_connections 2024-03-29T11:28:55.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.267+0000 7f4e34800700 1 -- 172.21.15.12:0/2322716828 wait complete. 2024-03-29T11:28:55.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.269+0000 7f4e34800700 1 Processor -- start 2024-03-29T11:28:55.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.269+0000 7f4e34800700 1 -- start start 2024-03-29T11:28:55.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e34800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c072580 0x7f4e2c1473f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e34800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e2c0b2e50 0x7f4e2c1598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e34800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2c0b3af0 0x7f4e2c159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e34800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c150580 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e34800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c154e70 con 0x7f4e2c0b2e50 2024-03-29T11:28:55.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e34800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c14fa00 con 0x7f4e2c072580 2024-03-29T11:28:55.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e32800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2c0b3af0 0x7f4e2c159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e31e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c072580 0x7f4e2c1473f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e32800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2c0b3af0 0x7f4e2c159e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54642/0 (socket says 172.21.15.12:54642) 2024-03-29T11:28:55.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e32800700 1 -- 172.21.15.12:0/613268294 learned_addr learned my addr 172.21.15.12:0/613268294 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:55.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.270+0000 7f4e31400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e2c0b2e50 0x7f4e2c1598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e32800700 1 -- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c072580 msgr2=0x7f4e2c1473f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e32800700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c072580 0x7f4e2c1473f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e32800700 1 -- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e2c0b2e50 msgr2=0x7f4e2c1598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e32800700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e2c0b2e50 0x7f4e2c1598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e32800700 1 -- 172.21.15.12:0/613268294 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c0094a0 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e32800700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2c0b3af0 0x7f4e2c159e30 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c06b930 tx=0x7f4e28004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:55.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e26000700 1 -- 172.21.15.12:0/613268294 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2800bbc0 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e26000700 1 -- 172.21.15.12:0/613268294 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2800b4c0 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e26000700 1 -- 172.21.15.12:0/613268294 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2800ae70 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e34800700 1 -- 172.21.15.12:0/613268294 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2800bd40 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.271+0000 7f4e34800700 1 -- 172.21.15.12:0/613268294 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c14ffc0 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.272+0000 7f4e34800700 1 -- 172.21.15.12:0/613268294 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2c155030 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.273+0000 7f4e26000700 1 -- 172.21.15.12:0/613268294 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4e28004040 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.274+0000 7f4e26000700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e1006adb0 0x7f4e1006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.274+0000 7f4e26000700 1 -- 172.21.15.12:0/613268294 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f4e2808d0a0 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.275+0000 7f4e31e00700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e1006adb0 0x7f4e1006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.275+0000 7f4e31e00700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e1006adb0 0x7f4e1006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c1f27c0 tx=0x7f4e20009040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:55.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.278+0000 7f4e26000700 1 -- 172.21.15.12:0/613268294 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e280c0580 con 0x7f4e2c0b3af0 2024-03-29T11:28:55.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.488+0000 7f4e34800700 1 -- 172.21.15.12:0/613268294 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e2c070f00 con 0x7f4e1006adb0 2024-03-29T11:28:55.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.497+0000 7f4e26000700 1 -- 172.21.15.12:0/613268294 <== mgr.9958 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289658 (secure 0 0 0) 0x7f4e20001cc0 con 0x7f4e1006adb0 2024-03-29T11:28:55.491 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 -- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e1006adb0 msgr2=0x7f4e1006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e1006adb0 0x7f4e1006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c1f27c0 tx=0x7f4e20009040 comp rx=0 tx=0).stop 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 -- 172.21.15.12:0/613268294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2c0b3af0 msgr2=0x7f4e2c159e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2c0b3af0 0x7f4e2c159e30 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c06b930 tx=0x7f4e28004ad0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 -- 172.21.15.12:0/613268294 shutdown_connections 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e1006adb0 0x7f4e1006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2c0b3af0 0x7f4e2c159e30 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e2c072580 0x7f4e2c1473f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 --2- 172.21.15.12:0/613268294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e2c0b2e50 0x7f4e2c1598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 -- 172.21.15.12:0/613268294 >> 172.21.15.12:0/613268294 conn(0x7f4e2c0cf190 msgr2=0x7f4e2c15dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 -- 172.21.15.12:0/613268294 shutdown_connections 2024-03-29T11:28:55.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.501+0000 7f4e34800700 1 -- 172.21.15.12:0/613268294 wait complete. 2024-03-29T11:28:55.495 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:28:55.510 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-29T11:28:54.310362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":616,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":383,"ondisk_log_size":383,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":432044,"kb_used_data":21572,"kb_used_omap":12,"kb_used_meta":410355,"kb_avail":754542676,"statfs":{"total":773094113280,"available":772651700224,"internally_reserved":0,"allocated":22089728,"data_stored":11310145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":420204089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.744745"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108610+0000","last_change":"2024-03-29T11:28:50.136657+0000","last_active":"2024-03-29T11:28:51.108610+0000","last_peered":"2024-03-29T11:28:51.108610+0000","last_clean":"2024-03-29T11:28:51.108610+0000","last_became_active":"2024-03-29T11:28:50.136349+0000","last_became_peered":"2024-03-29T11:28:50.136349+0000","last_unstale":"2024-03-29T11:28:51.108610+0000","last_undegraded":"2024-03-29T11:28:51.108610+0000","last_fullsized":"2024-03-29T11:28:51.108610+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:20:14.802317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108296+0000","last_change":"2024-03-29T11:28:50.118192+0000","last_active":"2024-03-29T11:28:51.108296+0000","last_peered":"2024-03-29T11:28:51.108296+0000","last_clean":"2024-03-29T11:28:51.108296+0000","last_became_active":"2024-03-29T11:28:50.117471+0000","last_became_peered":"2024-03-29T11:28:50.117471+0000","last_unstale":"2024-03-29T11:28:51.108296+0000","last_undegraded":"2024-03-29T11:28:51.108296+0000","last_fullsized":"2024-03-29T11:28:51.108296+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:22:01.558596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122122+0000","last_change":"2024-03-29T11:28:50.115075+0000","last_active":"2024-03-29T11:28:52.122122+0000","last_peered":"2024-03-29T11:28:52.122122+0000","last_clean":"2024-03-29T11:28:52.122122+0000","last_became_active":"2024-03-29T11:28:50.114782+0000","last_became_peered":"2024-03-29T11:28:50.114782+0000","last_unstale":"2024-03-29T11:28:52.122122+0000","last_undegraded":"2024-03-29T11:28:52.122122+0000","last_fullsized":"2024-03-29T11:28:52.122122+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:18:37.749501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107848+0000","last_change":"2024-03-29T11:28:50.115403+0000","last_active":"2024-03-29T11:28:51.107848+0000","last_peered":"2024-03-29T11:28:51.107848+0000","last_clean":"2024-03-29T11:28:51.107848+0000","last_became_active":"2024-03-29T11:28:50.114779+0000","last_became_peered":"2024-03-29T11:28:50.114779+0000","last_unstale":"2024-03-29T11:28:51.107848+0000","last_undegraded":"2024-03-29T11:28:51.107848+0000","last_fullsized":"2024-03-29T11:28:51.107848+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:54.062257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121655+0000","last_change":"2024-03-29T11:28:50.127626+0000","last_active":"2024-03-29T11:28:52.121655+0000","last_peered":"2024-03-29T11:28:52.121655+0000","last_clean":"2024-03-29T11:28:52.121655+0000","last_became_active":"2024-03-29T11:28:50.127387+0000","last_became_peered":"2024-03-29T11:28:50.127387+0000","last_unstale":"2024-03-29T11:28:52.121655+0000","last_undegraded":"2024-03-29T11:28:52.121655+0000","last_fullsized":"2024-03-29T11:28:52.121655+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:00:23.484211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121364+0000","last_change":"2024-03-29T11:28:50.123709+0000","last_active":"2024-03-29T11:28:52.121364+0000","last_peered":"2024-03-29T11:28:52.121364+0000","last_clean":"2024-03-29T11:28:52.121364+0000","last_became_active":"2024-03-29T11:28:50.123419+0000","last_became_peered":"2024-03-29T11:28:50.123419+0000","last_unstale":"2024-03-29T11:28:52.121364+0000","last_undegraded":"2024-03-29T11:28:52.121364+0000","last_fullsized":"2024-03-29T11:28:52.121364+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:29.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121264+0000","last_change":"2024-03-29T11:28:50.135863+0000","last_active":"2024-03-29T11:28:52.121264+0000","last_peered":"2024-03-29T11:28:52.121264+0000","last_clean":"2024-03-29T11:28:52.121264+0000","last_became_active":"2024-03-29T11:28:50.135597+0000","last_became_peered":"2024-03-29T11:28:50.135597+0000","last_unstale":"2024-03-29T11:28:52.121264+0000","last_undegraded":"2024-03-29T11:28:52.121264+0000","last_fullsized":"2024-03-29T11:28:52.121264+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:09:33.745584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108450+0000","last_change":"2024-03-29T11:28:50.117493+0000","last_active":"2024-03-29T11:28:51.108450+0000","last_peered":"2024-03-29T11:28:51.108450+0000","last_clean":"2024-03-29T11:28:51.108450+0000","last_became_active":"2024-03-29T11:28:50.117220+0000","last_became_peered":"2024-03-29T11:28:50.117220+0000","last_unstale":"2024-03-29T11:28:51.108450+0000","last_undegraded":"2024-03-29T11:28:51.108450+0000","last_fullsized":"2024-03-29T11:28:51.108450+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:32:45.260558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122118+0000","last_change":"2024-03-29T11:28:50.114782+0000","last_active":"2024-03-29T11:28:52.122118+0000","last_peered":"2024-03-29T11:28:52.122118+0000","last_clean":"2024-03-29T11:28:52.122118+0000","last_became_active":"2024-03-29T11:28:50.114516+0000","last_became_peered":"2024-03-29T11:28:50.114516+0000","last_unstale":"2024-03-29T11:28:52.122118+0000","last_undegraded":"2024-03-29T11:28:52.122118+0000","last_fullsized":"2024-03-29T11:28:52.122118+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:22:06.717277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122034+0000","last_change":"2024-03-29T11:28:50.124454+0000","last_active":"2024-03-29T11:28:52.122034+0000","last_peered":"2024-03-29T11:28:52.122034+0000","last_clean":"2024-03-29T11:28:52.122034+0000","last_became_active":"2024-03-29T11:28:50.124188+0000","last_became_peered":"2024-03-29T11:28:50.124188+0000","last_unstale":"2024-03-29T11:28:52.122034+0000","last_undegraded":"2024-03-29T11:28:52.122034+0000","last_fullsized":"2024-03-29T11:28:52.122034+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:00:23.484211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122011+0000","last_change":"2024-03-29T11:28:50.125654+0000","last_active":"2024-03-29T11:28:52.122011+0000","last_peered":"2024-03-29T11:28:52.122011+0000","last_clean":"2024-03-29T11:28:52.122011+0000","last_became_active":"2024-03-29T11:28:50.123621+0000","last_became_peered":"2024-03-29T11:28:50.123621+0000","last_unstale":"2024-03-29T11:28:52.122011+0000","last_undegraded":"2024-03-29T11:28:52.122011+0000","last_fullsized":"2024-03-29T11:28:52.122011+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:22:01.558596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107833+0000","last_change":"2024-03-29T11:28:50.127197+0000","last_active":"2024-03-29T11:28:51.107833+0000","last_peered":"2024-03-29T11:28:51.107833+0000","last_clean":"2024-03-29T11:28:51.107833+0000","last_became_active":"2024-03-29T11:28:50.126921+0000","last_became_peered":"2024-03-29T11:28:50.126921+0000","last_unstale":"2024-03-29T11:28:51.107833+0000","last_undegraded":"2024-03-29T11:28:51.107833+0000","last_fullsized":"2024-03-29T11:28:51.107833+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:38:03.520426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107734+0000","last_change":"2024-03-29T11:28:50.121346+0000","last_active":"2024-03-29T11:28:51.107734+0000","last_peered":"2024-03-29T11:28:51.107734+0000","last_clean":"2024-03-29T11:28:51.107734+0000","last_became_active":"2024-03-29T11:28:50.121079+0000","last_became_peered":"2024-03-29T11:28:50.121079+0000","last_unstale":"2024-03-29T11:28:51.107734+0000","last_undegraded":"2024-03-29T11:28:51.107734+0000","last_fullsized":"2024-03-29T11:28:51.107734+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:00.575122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121371+0000","last_change":"2024-03-29T11:28:50.129893+0000","last_active":"2024-03-29T11:28:52.121371+0000","last_peered":"2024-03-29T11:28:52.121371+0000","last_clean":"2024-03-29T11:28:52.121371+0000","last_became_active":"2024-03-29T11:28:50.129628+0000","last_became_peered":"2024-03-29T11:28:50.129628+0000","last_unstale":"2024-03-29T11:28:52.121371+0000","last_undegraded":"2024-03-29T11:28:52.121371+0000","last_fullsized":"2024-03-29T11:28:52.121371+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:54:55.931798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107854+0000","last_change":"2024-03-29T11:28:50.124047+0000","last_active":"2024-03-29T11:28:51.107854+0000","last_peered":"2024-03-29T11:28:51.107854+0000","last_clean":"2024-03-29T11:28:51.107854+0000","last_became_active":"2024-03-29T11:28:50.123782+0000","last_became_peered":"2024-03-29T11:28:50.123782+0000","last_unstale":"2024-03-29T11:28:51.107854+0000","last_undegraded":"2024-03-29T11:28:51.107854+0000","last_fullsized":"2024-03-29T11:28:51.107854+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:59.117506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108578+0000","last_change":"2024-03-29T11:28:50.115095+0000","last_active":"2024-03-29T11:28:51.108578+0000","last_peered":"2024-03-29T11:28:51.108578+0000","last_clean":"2024-03-29T11:28:51.108578+0000","last_became_active":"2024-03-29T11:28:50.114828+0000","last_became_peered":"2024-03-29T11:28:50.114828+0000","last_unstale":"2024-03-29T11:28:51.108578+0000","last_undegraded":"2024-03-29T11:28:51.108578+0000","last_fullsized":"2024-03-29T11:28:51.108578+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:45:40.534936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122583+0000","last_change":"2024-03-29T11:28:50.130157+0000","last_active":"2024-03-29T11:28:52.122583+0000","last_peered":"2024-03-29T11:28:52.122583+0000","last_clean":"2024-03-29T11:28:52.122583+0000","last_became_active":"2024-03-29T11:28:50.129188+0000","last_became_peered":"2024-03-29T11:28:50.129188+0000","last_unstale":"2024-03-29T11:28:52.122583+0000","last_undegraded":"2024-03-29T11:28:52.122583+0000","last_fullsized":"2024-03-29T11:28:52.122583+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:28.540927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108439+0000","last_change":"2024-03-29T11:28:50.128160+0000","last_active":"2024-03-29T11:28:51.108439+0000","last_peered":"2024-03-29T11:28:51.108439+0000","last_clean":"2024-03-29T11:28:51.108439+0000","last_became_active":"2024-03-29T11:28:50.127881+0000","last_became_peered":"2024-03-29T11:28:50.127881+0000","last_unstale":"2024-03-29T11:28:51.108439+0000","last_undegraded":"2024-03-29T11:28:51.108439+0000","last_fullsized":"2024-03-29T11:28:51.108439+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:18:56.271072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121271+0000","last_change":"2024-03-29T11:28:50.119640+0000","last_active":"2024-03-29T11:28:52.121271+0000","last_peered":"2024-03-29T11:28:52.121271+0000","last_clean":"2024-03-29T11:28:52.121271+0000","last_became_active":"2024-03-29T11:28:50.119350+0000","last_became_peered":"2024-03-29T11:28:50.119350+0000","last_unstale":"2024-03-29T11:28:52.121271+0000","last_undegraded":"2024-03-29T11:28:52.121271+0000","last_fullsized":"2024-03-29T11:28:52.121271+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:50.272182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121655+0000","last_change":"2024-03-29T11:28:50.126965+0000","last_active":"2024-03-29T11:28:52.121655+0000","last_peered":"2024-03-29T11:28:52.121655+0000","last_clean":"2024-03-29T11:28:52.121655+0000","last_became_active":"2024-03-29T11:28:50.126694+0000","last_became_peered":"2024-03-29T11:28:50.126694+0000","last_unstale":"2024-03-29T11:28:52.121655+0000","last_undegraded":"2024-03-29T11:28:52.121655+0000","last_fullsized":"2024-03-29T11:28:52.121655+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:12:57.697724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121768+0000","last_change":"2024-03-29T11:28:50.125404+0000","last_active":"2024-03-29T11:28:52.121768+0000","last_peered":"2024-03-29T11:28:52.121768+0000","last_clean":"2024-03-29T11:28:52.121768+0000","last_became_active":"2024-03-29T11:28:50.125162+0000","last_became_peered":"2024-03-29T11:28:50.125162+0000","last_unstale":"2024-03-29T11:28:52.121768+0000","last_undegraded":"2024-03-29T11:28:52.121768+0000","last_fullsized":"2024-03-29T11:28:52.121768+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:40.139442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121005+0000","last_change":"2024-03-29T11:28:50.127463+0000","last_active":"2024-03-29T11:28:52.121005+0000","last_peered":"2024-03-29T11:28:52.121005+0000","last_clean":"2024-03-29T11:28:52.121005+0000","last_became_active":"2024-03-29T11:28:50.126885+0000","last_became_peered":"2024-03-29T11:28:50.126885+0000","last_unstale":"2024-03-29T11:28:52.121005+0000","last_undegraded":"2024-03-29T11:28:52.121005+0000","last_fullsized":"2024-03-29T11:28:52.121005+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:12:05.408813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121343+0000","last_change":"2024-03-29T11:28:50.128429+0000","last_active":"2024-03-29T11:28:52.121343+0000","last_peered":"2024-03-29T11:28:52.121343+0000","last_clean":"2024-03-29T11:28:52.121343+0000","last_became_active":"2024-03-29T11:28:50.128142+0000","last_became_peered":"2024-03-29T11:28:50.128142+0000","last_unstale":"2024-03-29T11:28:52.121343+0000","last_undegraded":"2024-03-29T11:28:52.121343+0000","last_fullsized":"2024-03-29T11:28:52.121343+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:53:22.040285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108676+0000","last_change":"2024-03-29T11:28:50.114117+0000","last_active":"2024-03-29T11:28:51.108676+0000","last_peered":"2024-03-29T11:28:51.108676+0000","last_clean":"2024-03-29T11:28:51.108676+0000","last_became_active":"2024-03-29T11:28:50.113845+0000","last_became_peered":"2024-03-29T11:28:50.113845+0000","last_unstale":"2024-03-29T11:28:51.108676+0000","last_undegraded":"2024-03-29T11:28:51.108676+0000","last_fullsized":"2024-03-29T11:28:51.108676+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:18.491683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108343+0000","last_change":"2024-03-29T11:28:50.129416+0000","last_active":"2024-03-29T11:28:51.108343+0000","last_peered":"2024-03-29T11:28:51.108343+0000","last_clean":"2024-03-29T11:28:51.108343+0000","last_became_active":"2024-03-29T11:28:50.129025+0000","last_became_peered":"2024-03-29T11:28:50.129025+0000","last_unstale":"2024-03-29T11:28:51.108343+0000","last_undegraded":"2024-03-29T11:28:51.108343+0000","last_fullsized":"2024-03-29T11:28:51.108343+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:01.581391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122002+0000","last_change":"2024-03-29T11:28:50.128708+0000","last_active":"2024-03-29T11:28:52.122002+0000","last_peered":"2024-03-29T11:28:52.122002+0000","last_clean":"2024-03-29T11:28:52.122002+0000","last_became_active":"2024-03-29T11:28:50.128382+0000","last_became_peered":"2024-03-29T11:28:50.128382+0000","last_unstale":"2024-03-29T11:28:52.122002+0000","last_undegraded":"2024-03-29T11:28:52.122002+0000","last_fullsized":"2024-03-29T11:28:52.122002+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:33:08.316656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122219+0000","last_change":"2024-03-29T11:28:50.122564+0000","last_active":"2024-03-29T11:28:52.122219+0000","last_peered":"2024-03-29T11:28:52.122219+0000","last_clean":"2024-03-29T11:28:52.122219+0000","last_became_active":"2024-03-29T11:28:50.122289+0000","last_became_peered":"2024-03-29T11:28:50.122289+0000","last_unstale":"2024-03-29T11:28:52.122219+0000","last_undegraded":"2024-03-29T11:28:52.122219+0000","last_fullsized":"2024-03-29T11:28:52.122219+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:29:34.801860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121767+0000","last_change":"2024-03-29T11:28:50.132362+0000","last_active":"2024-03-29T11:28:52.121767+0000","last_peered":"2024-03-29T11:28:52.121767+0000","last_clean":"2024-03-29T11:28:52.121767+0000","last_became_active":"2024-03-29T11:28:50.132115+0000","last_became_peered":"2024-03-29T11:28:50.132115+0000","last_unstale":"2024-03-29T11:28:52.121767+0000","last_undegraded":"2024-03-29T11:28:52.121767+0000","last_fullsized":"2024-03-29T11:28:52.121767+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:56:00.889542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107891+0000","last_change":"2024-03-29T11:28:50.125084+0000","last_active":"2024-03-29T11:28:51.107891+0000","last_peered":"2024-03-29T11:28:51.107891+0000","last_clean":"2024-03-29T11:28:51.107891+0000","last_became_active":"2024-03-29T11:28:50.124728+0000","last_became_peered":"2024-03-29T11:28:50.124728+0000","last_unstale":"2024-03-29T11:28:51.107891+0000","last_undegraded":"2024-03-29T11:28:51.107891+0000","last_fullsized":"2024-03-29T11:28:51.107891+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:09:33.745584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121513+0000","last_change":"2024-03-29T11:28:50.120774+0000","last_active":"2024-03-29T11:28:52.121513+0000","last_peered":"2024-03-29T11:28:52.121513+0000","last_clean":"2024-03-29T11:28:52.121513+0000","last_became_active":"2024-03-29T11:28:50.120480+0000","last_became_peered":"2024-03-29T11:28:50.120480+0000","last_unstale":"2024-03-29T11:28:52.121513+0000","last_undegraded":"2024-03-29T11:28:52.121513+0000","last_fullsized":"2024-03-29T11:28:52.121513+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:18:47.048409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121344+0000","last_change":"2024-03-29T11:28:50.115547+0000","last_active":"2024-03-29T11:28:52.121344+0000","last_peered":"2024-03-29T11:28:52.121344+0000","last_clean":"2024-03-29T11:28:52.121344+0000","last_became_active":"2024-03-29T11:28:50.115296+0000","last_became_peered":"2024-03-29T11:28:50.115296+0000","last_unstale":"2024-03-29T11:28:52.121344+0000","last_undegraded":"2024-03-29T11:28:52.121344+0000","last_fullsized":"2024-03-29T11:28:52.121344+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:35.771515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.120945+0000","last_change":"2024-03-29T11:28:50.119485+0000","last_active":"2024-03-29T11:28:52.120945+0000","last_peered":"2024-03-29T11:28:52.120945+0000","last_clean":"2024-03-29T11:28:52.120945+0000","last_became_active":"2024-03-29T11:28:50.118790+0000","last_became_peered":"2024-03-29T11:28:50.118790+0000","last_unstale":"2024-03-29T11:28:52.120945+0000","last_undegraded":"2024-03-29T11:28:52.120945+0000","last_fullsized":"2024-03-29T11:28:52.120945+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:37:12.615160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.109156+0000","last_change":"2024-03-29T11:28:50.119766+0000","last_active":"2024-03-29T11:28:51.109156+0000","last_peered":"2024-03-29T11:28:51.109156+0000","last_clean":"2024-03-29T11:28:51.109156+0000","last_became_active":"2024-03-29T11:28:50.118403+0000","last_became_peered":"2024-03-29T11:28:50.118403+0000","last_unstale":"2024-03-29T11:28:51.109156+0000","last_undegraded":"2024-03-29T11:28:51.109156+0000","last_fullsized":"2024-03-29T11:28:51.109156+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:17:40.762889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"207'4","reported_seq":15,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.138600+0000","last_change":"2024-03-29T11:28:49.592987+0000","last_active":"2024-03-29T11:28:50.138600+0000","last_peered":"2024-03-29T11:28:50.138600+0000","last_clean":"2024-03-29T11:28:50.138600+0000","last_became_active":"2024-03-29T11:28:49.592681+0000","last_became_peered":"2024-03-29T11:28:49.592681+0000","last_unstale":"2024-03-29T11:28:50.138600+0000","last_undegraded":"2024-03-29T11:28:50.138600+0000","last_fullsized":"2024-03-29T11:28:50.138600+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:33:07.308817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122170+0000","last_change":"2024-03-29T11:28:50.123685+0000","last_active":"2024-03-29T11:28:52.122170+0000","last_peered":"2024-03-29T11:28:52.122170+0000","last_clean":"2024-03-29T11:28:52.122170+0000","last_became_active":"2024-03-29T11:28:50.121913+0000","last_became_peered":"2024-03-29T11:28:50.121913+0000","last_unstale":"2024-03-29T11:28:52.122170+0000","last_undegraded":"2024-03-29T11:28:52.122170+0000","last_fullsized":"2024-03-29T11:28:52.122170+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:15.868140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"207'1","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.123238+0000","last_change":"2024-03-29T11:28:49.588251+0000","last_active":"2024-03-29T11:28:50.123238+0000","last_peered":"2024-03-29T11:28:50.123238+0000","last_clean":"2024-03-29T11:28:50.123238+0000","last_became_active":"2024-03-29T11:28:49.587958+0000","last_became_peered":"2024-03-29T11:28:49.587958+0000","last_unstale":"2024-03-29T11:28:50.123238+0000","last_undegraded":"2024-03-29T11:28:50.123238+0000","last_fullsized":"2024-03-29T11:28:50.123238+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:56:22.069036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121613+0000","last_change":"2024-03-29T11:28:50.121894+0000","last_active":"2024-03-29T11:28:52.121613+0000","last_peered":"2024-03-29T11:28:52.121613+0000","last_clean":"2024-03-29T11:28:52.121613+0000","last_became_active":"2024-03-29T11:28:50.121173+0000","last_became_peered":"2024-03-29T11:28:50.121173+0000","last_unstale":"2024-03-29T11:28:52.121613+0000","last_undegraded":"2024-03-29T11:28:52.121613+0000","last_fullsized":"2024-03-29T11:28:52.121613+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:54.062257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.132652+0000","last_change":"2024-03-29T11:28:49.585583+0000","last_active":"2024-03-29T11:28:50.132652+0000","last_peered":"2024-03-29T11:28:50.132652+0000","last_clean":"2024-03-29T11:28:50.132652+0000","last_became_active":"2024-03-29T11:28:49.585327+0000","last_became_peered":"2024-03-29T11:28:49.585327+0000","last_unstale":"2024-03-29T11:28:50.132652+0000","last_undegraded":"2024-03-29T11:28:50.132652+0000","last_fullsized":"2024-03-29T11:28:50.132652+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:47:35.454167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121973+0000","last_change":"2024-03-29T11:28:50.121616+0000","last_active":"2024-03-29T11:28:52.121973+0000","last_peered":"2024-03-29T11:28:52.121973+0000","last_clean":"2024-03-29T11:28:52.121973+0000","last_became_active":"2024-03-29T11:28:50.119166+0000","last_became_peered":"2024-03-29T11:28:50.119166+0000","last_unstale":"2024-03-29T11:28:52.121973+0000","last_undegraded":"2024-03-29T11:28:52.121973+0000","last_fullsized":"2024-03-29T11:28:52.121973+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:01:39.484194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.129339+0000","last_change":"2024-03-29T11:28:49.590475+0000","last_active":"2024-03-29T11:28:50.129339+0000","last_peered":"2024-03-29T11:28:50.129339+0000","last_clean":"2024-03-29T11:28:50.129339+0000","last_became_active":"2024-03-29T11:28:49.590183+0000","last_became_peered":"2024-03-29T11:28:49.590183+0000","last_unstale":"2024-03-29T11:28:50.129339+0000","last_undegraded":"2024-03-29T11:28:50.129339+0000","last_fullsized":"2024-03-29T11:28:50.129339+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:34.763676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121240+0000","last_change":"2024-03-29T11:28:50.127619+0000","last_active":"2024-03-29T11:28:52.121240+0000","last_peered":"2024-03-29T11:28:52.121240+0000","last_clean":"2024-03-29T11:28:52.121240+0000","last_became_active":"2024-03-29T11:28:50.127223+0000","last_became_peered":"2024-03-29T11:28:50.127223+0000","last_unstale":"2024-03-29T11:28:52.121240+0000","last_undegraded":"2024-03-29T11:28:52.121240+0000","last_fullsized":"2024-03-29T11:28:52.121240+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:40.139442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"207'1","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.138994+0000","last_change":"2024-03-29T11:28:49.592533+0000","last_active":"2024-03-29T11:28:50.138994+0000","last_peered":"2024-03-29T11:28:50.138994+0000","last_clean":"2024-03-29T11:28:50.138994+0000","last_became_active":"2024-03-29T11:28:49.592270+0000","last_became_peered":"2024-03-29T11:28:49.592270+0000","last_unstale":"2024-03-29T11:28:50.138994+0000","last_undegraded":"2024-03-29T11:28:50.138994+0000","last_fullsized":"2024-03-29T11:28:50.138994+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:00.573552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108951+0000","last_change":"2024-03-29T11:28:50.121103+0000","last_active":"2024-03-29T11:28:51.108951+0000","last_peered":"2024-03-29T11:28:51.108951+0000","last_clean":"2024-03-29T11:28:51.108951+0000","last_became_active":"2024-03-29T11:28:50.120827+0000","last_became_peered":"2024-03-29T11:28:50.120827+0000","last_unstale":"2024-03-29T11:28:51.108951+0000","last_undegraded":"2024-03-29T11:28:51.108951+0000","last_fullsized":"2024-03-29T11:28:51.108951+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:26:24.715913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"207'1","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.129759+0000","last_change":"2024-03-29T11:28:49.588026+0000","last_active":"2024-03-29T11:28:50.129759+0000","last_peered":"2024-03-29T11:28:50.129759+0000","last_clean":"2024-03-29T11:28:50.129759+0000","last_became_active":"2024-03-29T11:28:49.586022+0000","last_became_peered":"2024-03-29T11:28:49.586022+0000","last_unstale":"2024-03-29T11:28:50.129759+0000","last_undegraded":"2024-03-29T11:28:50.129759+0000","last_fullsized":"2024-03-29T11:28:50.129759+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:24.683283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121485+0000","last_change":"2024-03-29T11:28:50.123628+0000","last_active":"2024-03-29T11:28:52.121485+0000","last_peered":"2024-03-29T11:28:52.121485+0000","last_clean":"2024-03-29T11:28:52.121485+0000","last_became_active":"2024-03-29T11:28:50.123349+0000","last_became_peered":"2024-03-29T11:28:50.123349+0000","last_unstale":"2024-03-29T11:28:52.121485+0000","last_undegraded":"2024-03-29T11:28:52.121485+0000","last_fullsized":"2024-03-29T11:28:52.121485+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:56:23.076874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.113652+0000","last_change":"2024-03-29T11:28:49.592976+0000","last_active":"2024-03-29T11:28:50.113652+0000","last_peered":"2024-03-29T11:28:50.113652+0000","last_clean":"2024-03-29T11:28:50.113652+0000","last_became_active":"2024-03-29T11:28:49.592693+0000","last_became_peered":"2024-03-29T11:28:49.592693+0000","last_unstale":"2024-03-29T11:28:50.113652+0000","last_undegraded":"2024-03-29T11:28:50.113652+0000","last_fullsized":"2024-03-29T11:28:50.113652+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:27.533088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122488+0000","last_change":"2024-03-29T11:28:50.120407+0000","last_active":"2024-03-29T11:28:52.122488+0000","last_peered":"2024-03-29T11:28:52.122488+0000","last_clean":"2024-03-29T11:28:52.122488+0000","last_became_active":"2024-03-29T11:28:50.119801+0000","last_became_peered":"2024-03-29T11:28:50.119801+0000","last_unstale":"2024-03-29T11:28:52.122488+0000","last_undegraded":"2024-03-29T11:28:52.122488+0000","last_fullsized":"2024-03-29T11:28:52.122488+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:54.413793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.110995+0000","last_change":"2024-03-29T11:28:49.587659+0000","last_active":"2024-03-29T11:28:50.110995+0000","last_peered":"2024-03-29T11:28:50.110995+0000","last_clean":"2024-03-29T11:28:50.110995+0000","last_became_active":"2024-03-29T11:28:49.587184+0000","last_became_peered":"2024-03-29T11:28:49.587184+0000","last_unstale":"2024-03-29T11:28:50.110995+0000","last_undegraded":"2024-03-29T11:28:50.110995+0000","last_fullsized":"2024-03-29T11:28:50.110995+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:54:53.405955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.109247+0000","last_change":"2024-03-29T11:28:50.122423+0000","last_active":"2024-03-29T11:28:51.109247+0000","last_peered":"2024-03-29T11:28:51.109247+0000","last_clean":"2024-03-29T11:28:51.109247+0000","last_became_active":"2024-03-29T11:28:50.121286+0000","last_became_peered":"2024-03-29T11:28:50.121286+0000","last_unstale":"2024-03-29T11:28:51.109247+0000","last_undegraded":"2024-03-29T11:28:51.109247+0000","last_fullsized":"2024-03-29T11:28:51.109247+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:57:06.474859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.115997+0000","last_change":"2024-03-29T11:28:49.595035+0000","last_active":"2024-03-29T11:28:50.115997+0000","last_peered":"2024-03-29T11:28:50.115997+0000","last_clean":"2024-03-29T11:28:50.115997+0000","last_became_active":"2024-03-29T11:28:49.592835+0000","last_became_peered":"2024-03-29T11:28:49.592835+0000","last_unstale":"2024-03-29T11:28:50.115997+0000","last_undegraded":"2024-03-29T11:28:50.115997+0000","last_fullsized":"2024-03-29T11:28:50.115997+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:24.687334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122002+0000","last_change":"2024-03-29T11:28:50.126055+0000","last_active":"2024-03-29T11:28:52.122002+0000","last_peered":"2024-03-29T11:28:52.122002+0000","last_clean":"2024-03-29T11:28:52.122002+0000","last_became_active":"2024-03-29T11:28:50.123504+0000","last_became_peered":"2024-03-29T11:28:50.123504+0000","last_unstale":"2024-03-29T11:28:52.122002+0000","last_undegraded":"2024-03-29T11:28:52.122002+0000","last_fullsized":"2024-03-29T11:28:52.122002+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:38:03.520426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"207'3","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.138330+0000","last_change":"2024-03-29T11:28:49.586030+0000","last_active":"2024-03-29T11:28:50.138330+0000","last_peered":"2024-03-29T11:28:50.138330+0000","last_clean":"2024-03-29T11:28:50.138330+0000","last_became_active":"2024-03-29T11:28:49.585755+0000","last_became_peered":"2024-03-29T11:28:49.585755+0000","last_unstale":"2024-03-29T11:28:50.138330+0000","last_undegraded":"2024-03-29T11:28:50.138330+0000","last_fullsized":"2024-03-29T11:28:50.138330+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:59.567283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121411+0000","last_change":"2024-03-29T11:28:50.123318+0000","last_active":"2024-03-29T11:28:52.121411+0000","last_peered":"2024-03-29T11:28:52.121411+0000","last_clean":"2024-03-29T11:28:52.121411+0000","last_became_active":"2024-03-29T11:28:50.122803+0000","last_became_peered":"2024-03-29T11:28:50.122803+0000","last_unstale":"2024-03-29T11:28:52.121411+0000","last_undegraded":"2024-03-29T11:28:52.121411+0000","last_fullsized":"2024-03-29T11:28:52.121411+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:19:05.469618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"207'6","reported_seq":17,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.140745+0000","last_change":"2024-03-29T11:28:49.589530+0000","last_active":"2024-03-29T11:28:50.140745+0000","last_peered":"2024-03-29T11:28:50.140745+0000","last_clean":"2024-03-29T11:28:50.140745+0000","last_became_active":"2024-03-29T11:28:49.588514+0000","last_became_peered":"2024-03-29T11:28:49.588514+0000","last_unstale":"2024-03-29T11:28:50.140745+0000","last_undegraded":"2024-03-29T11:28:50.140745+0000","last_fullsized":"2024-03-29T11:28:50.140745+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:16:20.743757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121524+0000","last_change":"2024-03-29T11:28:50.129973+0000","last_active":"2024-03-29T11:28:52.121524+0000","last_peered":"2024-03-29T11:28:52.121524+0000","last_clean":"2024-03-29T11:28:52.121524+0000","last_became_active":"2024-03-29T11:28:50.129689+0000","last_became_peered":"2024-03-29T11:28:50.129689+0000","last_unstale":"2024-03-29T11:28:52.121524+0000","last_undegraded":"2024-03-29T11:28:52.121524+0000","last_fullsized":"2024-03-29T11:28:52.121524+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:58:04.762023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.136780+0000","last_change":"2024-03-29T11:28:49.593870+0000","last_active":"2024-03-29T11:28:50.136780+0000","last_peered":"2024-03-29T11:28:50.136780+0000","last_clean":"2024-03-29T11:28:50.136780+0000","last_became_active":"2024-03-29T11:28:49.593606+0000","last_became_peered":"2024-03-29T11:28:49.593606+0000","last_unstale":"2024-03-29T11:28:50.136780+0000","last_undegraded":"2024-03-29T11:28:50.136780+0000","last_fullsized":"2024-03-29T11:28:50.136780+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:14.860302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121604+0000","last_change":"2024-03-29T11:28:50.129419+0000","last_active":"2024-03-29T11:28:52.121604+0000","last_peered":"2024-03-29T11:28:52.121604+0000","last_clean":"2024-03-29T11:28:52.121604+0000","last_became_active":"2024-03-29T11:28:50.128142+0000","last_became_peered":"2024-03-29T11:28:50.128142+0000","last_unstale":"2024-03-29T11:28:52.121604+0000","last_undegraded":"2024-03-29T11:28:52.121604+0000","last_fullsized":"2024-03-29T11:28:52.121604+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:05:59.117506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.112375+0000","last_change":"2024-03-29T11:28:49.596513+0000","last_active":"2024-03-29T11:28:50.112375+0000","last_peered":"2024-03-29T11:28:50.112375+0000","last_clean":"2024-03-29T11:28:50.112375+0000","last_became_active":"2024-03-29T11:28:49.596189+0000","last_became_peered":"2024-03-29T11:28:49.596189+0000","last_unstale":"2024-03-29T11:28:50.112375+0000","last_undegraded":"2024-03-29T11:28:50.112375+0000","last_fullsized":"2024-03-29T11:28:50.112375+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:53.054418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121721+0000","last_change":"2024-03-29T11:28:50.120384+0000","last_active":"2024-03-29T11:28:52.121721+0000","last_peered":"2024-03-29T11:28:52.121721+0000","last_clean":"2024-03-29T11:28:52.121721+0000","last_became_active":"2024-03-29T11:28:50.120076+0000","last_became_peered":"2024-03-29T11:28:50.120076+0000","last_unstale":"2024-03-29T11:28:52.121721+0000","last_undegraded":"2024-03-29T11:28:52.121721+0000","last_fullsized":"2024-03-29T11:28:52.121721+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:17:49.619853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.101737+0000","last_change":"2024-03-29T11:28:49.591931+0000","last_active":"2024-03-29T11:28:50.101737+0000","last_peered":"2024-03-29T11:28:50.101737+0000","last_clean":"2024-03-29T11:28:50.101737+0000","last_became_active":"2024-03-29T11:28:49.590295+0000","last_became_peered":"2024-03-29T11:28:49.590295+0000","last_unstale":"2024-03-29T11:28:50.101737+0000","last_undegraded":"2024-03-29T11:28:50.101737+0000","last_fullsized":"2024-03-29T11:28:50.101737+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:22:50.946710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.120865+0000","last_change":"2024-03-29T11:28:50.126351+0000","last_active":"2024-03-29T11:28:52.120865+0000","last_peered":"2024-03-29T11:28:52.120865+0000","last_clean":"2024-03-29T11:28:52.120865+0000","last_became_active":"2024-03-29T11:28:50.125201+0000","last_became_peered":"2024-03-29T11:28:50.125201+0000","last_unstale":"2024-03-29T11:28:52.120865+0000","last_undegraded":"2024-03-29T11:28:52.120865+0000","last_fullsized":"2024-03-29T11:28:52.120865+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:37:12.615160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"207'8","reported_seq":19,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.136927+0000","last_change":"2024-03-29T11:28:49.593142+0000","last_active":"2024-03-29T11:28:50.136927+0000","last_peered":"2024-03-29T11:28:50.136927+0000","last_clean":"2024-03-29T11:28:50.136927+0000","last_became_active":"2024-03-29T11:28:49.592684+0000","last_became_peered":"2024-03-29T11:28:49.592684+0000","last_unstale":"2024-03-29T11:28:50.136927+0000","last_undegraded":"2024-03-29T11:28:50.136927+0000","last_fullsized":"2024-03-29T11:28:50.136927+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:00:22.476372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122078+0000","last_change":"2024-03-29T11:28:50.126263+0000","last_active":"2024-03-29T11:28:52.122078+0000","last_peered":"2024-03-29T11:28:52.122078+0000","last_clean":"2024-03-29T11:28:52.122078+0000","last_became_active":"2024-03-29T11:28:50.125942+0000","last_became_peered":"2024-03-29T11:28:50.125942+0000","last_unstale":"2024-03-29T11:28:52.122078+0000","last_undegraded":"2024-03-29T11:28:52.122078+0000","last_fullsized":"2024-03-29T11:28:52.122078+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:56:00.889542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"207'1","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.139354+0000","last_change":"2024-03-29T11:28:49.592749+0000","last_active":"2024-03-29T11:28:50.139354+0000","last_peered":"2024-03-29T11:28:50.139354+0000","last_clean":"2024-03-29T11:28:50.139354+0000","last_became_active":"2024-03-29T11:28:49.592474+0000","last_became_peered":"2024-03-29T11:28:49.592474+0000","last_unstale":"2024-03-29T11:28:50.139354+0000","last_undegraded":"2024-03-29T11:28:50.139354+0000","last_fullsized":"2024-03-29T11:28:50.139354+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:38:02.512587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122170+0000","last_change":"2024-03-29T11:28:50.120220+0000","last_active":"2024-03-29T11:28:52.122170+0000","last_peered":"2024-03-29T11:28:52.122170+0000","last_clean":"2024-03-29T11:28:52.122170+0000","last_became_active":"2024-03-29T11:28:50.119893+0000","last_became_peered":"2024-03-29T11:28:50.119893+0000","last_unstale":"2024-03-29T11:28:52.122170+0000","last_undegraded":"2024-03-29T11:28:52.122170+0000","last_fullsized":"2024-03-29T11:28:52.122170+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:25.695173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.127794+0000","last_change":"2024-03-29T11:28:49.587442+0000","last_active":"2024-03-29T11:28:50.127794+0000","last_peered":"2024-03-29T11:28:50.127794+0000","last_clean":"2024-03-29T11:28:50.127794+0000","last_became_active":"2024-03-29T11:28:49.587094+0000","last_became_peered":"2024-03-29T11:28:49.587094+0000","last_unstale":"2024-03-29T11:28:50.127794+0000","last_undegraded":"2024-03-29T11:28:50.127794+0000","last_fullsized":"2024-03-29T11:28:50.127794+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:39.131603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121513+0000","last_change":"2024-03-29T11:28:50.129029+0000","last_active":"2024-03-29T11:28:52.121513+0000","last_peered":"2024-03-29T11:28:52.121513+0000","last_clean":"2024-03-29T11:28:52.121513+0000","last_became_active":"2024-03-29T11:28:50.128768+0000","last_became_peered":"2024-03-29T11:28:50.128768+0000","last_unstale":"2024-03-29T11:28:52.121513+0000","last_undegraded":"2024-03-29T11:28:52.121513+0000","last_fullsized":"2024-03-29T11:28:52.121513+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:56:55.880048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"207'1","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.133833+0000","last_change":"2024-03-29T11:28:49.593388+0000","last_active":"2024-03-29T11:28:50.133833+0000","last_peered":"2024-03-29T11:28:50.133833+0000","last_clean":"2024-03-29T11:28:50.133833+0000","last_became_active":"2024-03-29T11:28:49.592789+0000","last_became_peered":"2024-03-29T11:28:49.592789+0000","last_unstale":"2024-03-29T11:28:50.133833+0000","last_undegraded":"2024-03-29T11:28:50.133833+0000","last_fullsized":"2024-03-29T11:28:50.133833+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:12:56.689885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108042+0000","last_change":"2024-03-29T11:28:50.115210+0000","last_active":"2024-03-29T11:28:51.108042+0000","last_peered":"2024-03-29T11:28:51.108042+0000","last_clean":"2024-03-29T11:28:51.108042+0000","last_became_active":"2024-03-29T11:28:50.114928+0000","last_became_peered":"2024-03-29T11:28:50.114928+0000","last_unstale":"2024-03-29T11:28:51.108042+0000","last_undegraded":"2024-03-29T11:28:51.108042+0000","last_fullsized":"2024-03-29T11:28:51.108042+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:14:21.419257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.110790+0000","last_change":"2024-03-29T11:28:49.589826+0000","last_active":"2024-03-29T11:28:50.110790+0000","last_peered":"2024-03-29T11:28:50.110790+0000","last_clean":"2024-03-29T11:28:50.110790+0000","last_became_active":"2024-03-29T11:28:49.589410+0000","last_became_peered":"2024-03-29T11:28:49.589410+0000","last_unstale":"2024-03-29T11:28:50.110790+0000","last_undegraded":"2024-03-29T11:28:50.110790+0000","last_fullsized":"2024-03-29T11:28:50.110790+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:18:36.741662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121003+0000","last_change":"2024-03-29T11:28:50.121655+0000","last_active":"2024-03-29T11:28:52.121003+0000","last_peered":"2024-03-29T11:28:52.121003+0000","last_clean":"2024-03-29T11:28:52.121003+0000","last_became_active":"2024-03-29T11:28:50.121394+0000","last_became_peered":"2024-03-29T11:28:50.121394+0000","last_unstale":"2024-03-29T11:28:52.121003+0000","last_undegraded":"2024-03-29T11:28:52.121003+0000","last_fullsized":"2024-03-29T11:28:52.121003+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:12:57.697724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.119663+0000","last_change":"2024-03-29T11:28:49.595068+0000","last_active":"2024-03-29T11:28:50.119663+0000","last_peered":"2024-03-29T11:28:50.119663+0000","last_clean":"2024-03-29T11:28:50.119663+0000","last_became_active":"2024-03-29T11:28:49.594049+0000","last_became_peered":"2024-03-29T11:28:49.594049+0000","last_unstale":"2024-03-29T11:28:50.119663+0000","last_undegraded":"2024-03-29T11:28:50.119663+0000","last_fullsized":"2024-03-29T11:28:50.119663+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:44:41.821484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121718+0000","last_change":"2024-03-29T11:28:50.115276+0000","last_active":"2024-03-29T11:28:52.121718+0000","last_peered":"2024-03-29T11:28:52.121718+0000","last_clean":"2024-03-29T11:28:52.121718+0000","last_became_active":"2024-03-29T11:28:50.114053+0000","last_became_peered":"2024-03-29T11:28:50.114053+0000","last_unstale":"2024-03-29T11:28:52.121718+0000","last_undegraded":"2024-03-29T11:28:52.121718+0000","last_fullsized":"2024-03-29T11:28:52.121718+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:14:21.419257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.108899+0000","last_change":"2024-03-29T11:28:49.591960+0000","last_active":"2024-03-29T11:28:50.108899+0000","last_peered":"2024-03-29T11:28:50.108899+0000","last_clean":"2024-03-29T11:28:50.108899+0000","last_became_active":"2024-03-29T11:28:49.591696+0000","last_became_peered":"2024-03-29T11:28:49.591696+0000","last_unstale":"2024-03-29T11:28:50.108899+0000","last_undegraded":"2024-03-29T11:28:50.108899+0000","last_fullsized":"2024-03-29T11:28:50.108899+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:43:53.054418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107599+0000","last_change":"2024-03-29T11:28:50.127864+0000","last_active":"2024-03-29T11:28:51.107599+0000","last_peered":"2024-03-29T11:28:51.107599+0000","last_clean":"2024-03-29T11:28:51.107599+0000","last_became_active":"2024-03-29T11:28:50.127558+0000","last_became_peered":"2024-03-29T11:28:50.127558+0000","last_unstale":"2024-03-29T11:28:51.107599+0000","last_undegraded":"2024-03-29T11:28:51.107599+0000","last_fullsized":"2024-03-29T11:28:51.107599+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:33:08.316656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.106933+0000","last_change":"2024-03-29T11:28:49.591597+0000","last_active":"2024-03-29T11:28:50.106933+0000","last_peered":"2024-03-29T11:28:50.106933+0000","last_clean":"2024-03-29T11:28:50.106933+0000","last_became_active":"2024-03-29T11:28:49.591333+0000","last_became_peered":"2024-03-29T11:28:49.591333+0000","last_unstale":"2024-03-29T11:28:50.106933+0000","last_undegraded":"2024-03-29T11:28:50.106933+0000","last_fullsized":"2024-03-29T11:28:50.106933+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:57:18.331610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121202+0000","last_change":"2024-03-29T11:28:50.121034+0000","last_active":"2024-03-29T11:28:52.121202+0000","last_peered":"2024-03-29T11:28:52.121202+0000","last_clean":"2024-03-29T11:28:52.121202+0000","last_became_active":"2024-03-29T11:28:50.120761+0000","last_became_peered":"2024-03-29T11:28:50.120761+0000","last_unstale":"2024-03-29T11:28:52.121202+0000","last_undegraded":"2024-03-29T11:28:52.121202+0000","last_fullsized":"2024-03-29T11:28:52.121202+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:19:05.469618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.109525+0000","last_change":"2024-03-29T11:28:49.592219+0000","last_active":"2024-03-29T11:28:50.109525+0000","last_peered":"2024-03-29T11:28:50.109525+0000","last_clean":"2024-03-29T11:28:50.109525+0000","last_became_active":"2024-03-29T11:28:49.591806+0000","last_became_peered":"2024-03-29T11:28:49.591806+0000","last_unstale":"2024-03-29T11:28:50.109525+0000","last_undegraded":"2024-03-29T11:28:50.109525+0000","last_fullsized":"2024-03-29T11:28:50.109525+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:32.966697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122279+0000","last_change":"2024-03-29T11:28:50.121269+0000","last_active":"2024-03-29T11:28:52.122279+0000","last_peered":"2024-03-29T11:28:52.122279+0000","last_clean":"2024-03-29T11:28:52.122279+0000","last_became_active":"2024-03-29T11:28:50.121033+0000","last_became_peered":"2024-03-29T11:28:50.121033+0000","last_unstale":"2024-03-29T11:28:52.122279+0000","last_undegraded":"2024-03-29T11:28:52.122279+0000","last_fullsized":"2024-03-29T11:28:52.122279+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:12:05.408813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"207'2","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.136977+0000","last_change":"2024-03-29T11:28:49.596075+0000","last_active":"2024-03-29T11:28:50.136977+0000","last_peered":"2024-03-29T11:28:50.136977+0000","last_clean":"2024-03-29T11:28:50.136977+0000","last_became_active":"2024-03-29T11:28:49.595756+0000","last_became_peered":"2024-03-29T11:28:49.595756+0000","last_unstale":"2024-03-29T11:28:50.136977+0000","last_undegraded":"2024-03-29T11:28:50.136977+0000","last_fullsized":"2024-03-29T11:28:50.136977+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:40:30.202152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122221+0000","last_change":"2024-03-29T11:28:50.133779+0000","last_active":"2024-03-29T11:28:52.122221+0000","last_peered":"2024-03-29T11:28:52.122221+0000","last_clean":"2024-03-29T11:28:52.122221+0000","last_became_active":"2024-03-29T11:28:50.133293+0000","last_became_peered":"2024-03-29T11:28:50.133293+0000","last_unstale":"2024-03-29T11:28:52.122221+0000","last_undegraded":"2024-03-29T11:28:52.122221+0000","last_fullsized":"2024-03-29T11:28:52.122221+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:44.130979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"208'1","reported_seq":12,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.122124+0000","last_change":"2024-03-29T11:28:49.589012+0000","last_active":"2024-03-29T11:28:52.122124+0000","last_peered":"2024-03-29T11:28:52.122124+0000","last_clean":"2024-03-29T11:28:52.122124+0000","last_became_active":"2024-03-29T11:28:49.588740+0000","last_became_peered":"2024-03-29T11:28:49.588740+0000","last_unstale":"2024-03-29T11:28:52.122124+0000","last_undegraded":"2024-03-29T11:28:52.122124+0000","last_fullsized":"2024-03-29T11:28:52.122124+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:09:32.737746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.109026+0000","last_change":"2024-03-29T11:28:50.122423+0000","last_active":"2024-03-29T11:28:51.109026+0000","last_peered":"2024-03-29T11:28:51.109026+0000","last_clean":"2024-03-29T11:28:51.109026+0000","last_became_active":"2024-03-29T11:28:50.122157+0000","last_became_peered":"2024-03-29T11:28:50.122157+0000","last_unstale":"2024-03-29T11:28:51.109026+0000","last_undegraded":"2024-03-29T11:28:51.109026+0000","last_fullsized":"2024-03-29T11:28:51.109026+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:35:38.671731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.110065+0000","last_change":"2024-03-29T11:28:49.590468+0000","last_active":"2024-03-29T11:28:50.110065+0000","last_peered":"2024-03-29T11:28:50.110065+0000","last_clean":"2024-03-29T11:28:50.110065+0000","last_became_active":"2024-03-29T11:28:49.590196+0000","last_became_peered":"2024-03-29T11:28:49.590196+0000","last_unstale":"2024-03-29T11:28:50.110065+0000","last_undegraded":"2024-03-29T11:28:50.110065+0000","last_fullsized":"2024-03-29T11:28:50.110065+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:33:07.308817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.109189+0000","last_change":"2024-03-29T11:28:50.113553+0000","last_active":"2024-03-29T11:28:51.109189+0000","last_peered":"2024-03-29T11:28:51.109189+0000","last_clean":"2024-03-29T11:28:51.109189+0000","last_became_active":"2024-03-29T11:28:50.113276+0000","last_became_peered":"2024-03-29T11:28:50.113276+0000","last_unstale":"2024-03-29T11:28:51.109189+0000","last_undegraded":"2024-03-29T11:28:51.109189+0000","last_fullsized":"2024-03-29T11:28:51.109189+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:52:47.722790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.102176+0000","last_change":"2024-03-29T11:28:49.586785+0000","last_active":"2024-03-29T11:28:50.102176+0000","last_peered":"2024-03-29T11:28:50.102176+0000","last_clean":"2024-03-29T11:28:50.102176+0000","last_became_active":"2024-03-29T11:28:49.586409+0000","last_became_peered":"2024-03-29T11:28:49.586409+0000","last_unstale":"2024-03-29T11:28:50.102176+0000","last_undegraded":"2024-03-29T11:28:50.102176+0000","last_fullsized":"2024-03-29T11:28:50.102176+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:49.264343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108285+0000","last_change":"2024-03-29T11:28:50.132786+0000","last_active":"2024-03-29T11:28:51.108285+0000","last_peered":"2024-03-29T11:28:51.108285+0000","last_clean":"2024-03-29T11:28:51.108285+0000","last_became_active":"2024-03-29T11:28:50.132492+0000","last_became_peered":"2024-03-29T11:28:50.132492+0000","last_unstale":"2024-03-29T11:28:51.108285+0000","last_undegraded":"2024-03-29T11:28:51.108285+0000","last_fullsized":"2024-03-29T11:28:51.108285+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:32:45.260558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.109748+0000","last_change":"2024-03-29T11:28:49.581723+0000","last_active":"2024-03-29T11:28:50.109748+0000","last_peered":"2024-03-29T11:28:50.109748+0000","last_clean":"2024-03-29T11:28:50.109748+0000","last_became_active":"2024-03-29T11:28:49.581336+0000","last_became_peered":"2024-03-29T11:28:49.581336+0000","last_unstale":"2024-03-29T11:28:50.109748+0000","last_undegraded":"2024-03-29T11:28:50.109748+0000","last_fullsized":"2024-03-29T11:28:50.109748+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:22:00.550757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.108547+0000","last_change":"2024-03-29T11:28:50.128076+0000","last_active":"2024-03-29T11:28:51.108547+0000","last_peered":"2024-03-29T11:28:51.108547+0000","last_clean":"2024-03-29T11:28:51.108547+0000","last_became_active":"2024-03-29T11:28:50.127795+0000","last_became_peered":"2024-03-29T11:28:50.127795+0000","last_unstale":"2024-03-29T11:28:51.108547+0000","last_undegraded":"2024-03-29T11:28:51.108547+0000","last_fullsized":"2024-03-29T11:28:51.108547+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:18.491683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.109884+0000","last_change":"2024-03-29T11:28:49.590029+0000","last_active":"2024-03-29T11:28:50.109884+0000","last_peered":"2024-03-29T11:28:50.109884+0000","last_clean":"2024-03-29T11:28:50.109884+0000","last_became_active":"2024-03-29T11:28:49.589322+0000","last_became_peered":"2024-03-29T11:28:49.589322+0000","last_unstale":"2024-03-29T11:28:50.109884+0000","last_undegraded":"2024-03-29T11:28:50.109884+0000","last_fullsized":"2024-03-29T11:28:50.109884+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:35:40.940699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:51.107888+0000","last_change":"2024-03-29T11:28:50.130534+0000","last_active":"2024-03-29T11:28:51.107888+0000","last_peered":"2024-03-29T11:28:51.107888+0000","last_clean":"2024-03-29T11:28:51.107888+0000","last_became_active":"2024-03-29T11:28:50.130284+0000","last_became_peered":"2024-03-29T11:28:50.130284+0000","last_unstale":"2024-03-29T11:28:51.107888+0000","last_undegraded":"2024-03-29T11:28:51.107888+0000","last_fullsized":"2024-03-29T11:28:51.107888+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:56:23.076874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.110026+0000","last_change":"2024-03-29T11:28:49.586706+0000","last_active":"2024-03-29T11:28:50.110026+0000","last_peered":"2024-03-29T11:28:50.110026+0000","last_clean":"2024-03-29T11:28:50.110026+0000","last_became_active":"2024-03-29T11:28:49.586218+0000","last_became_peered":"2024-03-29T11:28:49.586218+0000","last_unstale":"2024-03-29T11:28:50.110026+0000","last_undegraded":"2024-03-29T11:28:50.110026+0000","last_fullsized":"2024-03-29T11:28:50.110026+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:29:33.794021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.121270+0000","last_change":"2024-03-29T11:28:50.127095+0000","last_active":"2024-03-29T11:28:52.121270+0000","last_peered":"2024-03-29T11:28:52.121270+0000","last_clean":"2024-03-29T11:28:52.121270+0000","last_became_active":"2024-03-29T11:28:50.126842+0000","last_became_peered":"2024-03-29T11:28:50.126842+0000","last_unstale":"2024-03-29T11:28:52.121270+0000","last_undegraded":"2024-03-29T11:28:52.121270+0000","last_fullsized":"2024-03-29T11:28:52.121270+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:44.130979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.107653+0000","last_change":"2024-03-29T11:28:49.591978+0000","last_active":"2024-03-29T11:28:50.107653+0000","last_peered":"2024-03-29T11:28:50.107653+0000","last_clean":"2024-03-29T11:28:50.107653+0000","last_became_active":"2024-03-29T11:28:49.591702+0000","last_became_peered":"2024-03-29T11:28:49.591702+0000","last_unstale":"2024-03-29T11:28:50.107653+0000","last_undegraded":"2024-03-29T11:28:50.107653+0000","last_fullsized":"2024-03-29T11:28:50.107653+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:06:43.501520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-03-29T11:28:52.120603+0000","last_change":"2024-03-29T11:28:50.116491+0000","last_active":"2024-03-29T11:28:52.120603+0000","last_peered":"2024-03-29T11:28:52.120603+0000","last_clean":"2024-03-29T11:28:52.120603+0000","last_became_active":"2024-03-29T11:28:50.115323+0000","last_became_peered":"2024-03-29T11:28:50.115323+0000","last_unstale":"2024-03-29T11:28:52.120603+0000","last_undegraded":"2024-03-29T11:28:52.120603+0000","last_fullsized":"2024-03-29T11:28:52.120603+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:49.564587+0000","last_clean_scrub_stamp":"2024-03-29T11:28:49.564587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:52:28.540927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.121975+0000","last_change":"2024-03-29T11:28:49.593317+0000","last_active":"2024-03-29T11:28:50.121975+0000","last_peered":"2024-03-29T11:28:50.121975+0000","last_clean":"2024-03-29T11:28:50.121975+0000","last_became_active":"2024-03-29T11:28:49.592981+0000","last_became_peered":"2024-03-29T11:28:49.592981+0000","last_unstale":"2024-03-29T11:28:50.121975+0000","last_undegraded":"2024-03-29T11:28:50.121975+0000","last_fullsized":"2024-03-29T11:28:50.121975+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:28:48.556749+0000","last_clean_scrub_stamp":"2024-03-29T11:28:48.556749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:24:43.123140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"197'344","reported_seq":1175,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-29T11:28:50.113849+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:28:50.113849+0000","last_peered":"2024-03-29T11:28:50.113849+0000","last_clean":"2024-03-29T11:28:50.113849+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:28:50.113849+0000","last_undegraded":"2024-03-29T11:28:50.113849+0000","last_fullsized":"2024-03-29T11:28:50.113849+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":571,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":571,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":344,"ondisk_log_size":344,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575965,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52296,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319544,"statfs":{"total":96636764160,"available":96583213056,"internally_reserved":0,"allocated":3207168,"data_stored":1856185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":50330060},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.54200000000000004,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.1459999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.749,"15min":0.751},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.40300000000000002},"max":{"1min":0.96999999999999997,"5min":1.119,"15min":1.119},"last":0.85799999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73999999999999999,"15min":0.73699999999999999},"min":{"1min":0.51400000000000001,"5min":0.45000000000000001,"15min":0.44700000000000001},"max":{"1min":0.878,"5min":1.478,"15min":1.478},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77300000000000002,"15min":0.78000000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.47699999999999998},"max":{"1min":0.98399999999999999,"5min":1.393,"15min":1.393},"last":0.76300000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.90200000000000002,"5min":1.089,"15min":1.284},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.78000000000000003,"15min":0.78600000000000003},"min":{"1min":0.52800000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.90200000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.77900000000000003,"15min":0.76900000000000002},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.47799999999999998},"max":{"1min":0.94899999999999995,"5min":1.0580000000000001,"15min":1.069},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.78400000000000003,"15min":0.79400000000000004},"min":{"1min":0.65500000000000003,"5min":0.55300000000000005,"15min":0.52000000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.3169999999999999},"last":0.94999999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.63700000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41999999999999998},"max":{"1min":0.87,"5min":1.024,"15min":1.3779999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.48399999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83799999999999997,"5min":0.95299999999999996,"15min":1.335},"last":0.71099999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.50800000000000001,"5min":0.41999999999999998,"15min":0.38400000000000001},"max":{"1min":0.94299999999999995,"5min":0.95999999999999996,"15min":1.347},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70399999999999996,"15min":0.70899999999999996},"min":{"1min":0.57099999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94299999999999995,"5min":0.96099999999999997,"15min":1.1759999999999999},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.754,"15min":0.76500000000000001},"min":{"1min":0.53800000000000003,"5min":0.52500000000000002,"15min":0.47899999999999998},"max":{"1min":0.98899999999999999,"5min":0.99099999999999999,"15min":1.2070000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.53500000000000003,"5min":0.442,"15min":0.435},"max":{"1min":0.97499999999999998,"5min":1.0640000000000001,"15min":1.069},"last":0.83599999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575966,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45948,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325892,"statfs":{"total":96636764160,"available":96589713408,"internally_reserved":0,"allocated":2605056,"data_stored":1265713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44431815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72199999999999998,"15min":0.753},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.76100000000000001,"15min":0.77600000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.747,"15min":0.75},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.44800000000000001},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.70099999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.55900000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76600000000000001,"15min":0.77600000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.46200000000000002},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.76000000000000001,"15min":0.78400000000000003},"min":{"1min":0.57999999999999996,"5min":0.54400000000000004,"15min":0.499},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":0.622},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.47799999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.70899999999999996,"15min":0.70699999999999996},"min":{"1min":0.53700000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.6629999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":0.67500000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66300000000000003,"15min":0.65400000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70799999999999996,"15min":0.70899999999999996},"min":{"1min":0.46999999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.76300000000000001,"15min":0.76500000000000001},"min":{"1min":0.54800000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.7050000000000001,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73799999999999999,"15min":0.74199999999999999},"min":{"1min":0.504,"5min":0.499,"15min":0.47199999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.76900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575964,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59912,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311928,"statfs":{"total":96636764160,"available":96575414272,"internally_reserved":0,"allocated":2617344,"data_stored":1267277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.60899999999999999,"5min":0.80300000000000005,"15min":0.81100000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52900000000000003,"15min":0.54200000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.625,"5min":0.85299999999999998,"15min":0.89000000000000001},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.42899999999999999,"5min":0.40500000000000003,"15min":0.40000000000000002},"max":{"1min":0.76100000000000001,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52500000000000002,"15min":0.54100000000000004},"min":{"1min":0.36199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.90300000000000002},"last":0.373}]},{"osd":3,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59599999999999997,"15min":0.60699999999999998},"min":{"1min":0.45300000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86199999999999999,"5min":0.876,"15min":0.995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.36499999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.878},"last":0.52700000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.78200000000000003,"15min":0.80600000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.97999999999999998,"5min":1.119,"15min":1.3660000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78600000000000003,"15min":0.80900000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.93200000000000005,"5min":1.304,"15min":1.304},"last":0.70299999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.79800000000000004,"15min":0.82499999999999996},"min":{"1min":0.57799999999999996,"5min":0.57699999999999996,"15min":0.48199999999999998},"max":{"1min":0.93999999999999995,"5min":1.3380000000000001,"15min":1.3720000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.82699999999999996,"15min":0.84999999999999998},"min":{"1min":0.65500000000000003,"5min":0.61299999999999999,"15min":0.59099999999999997},"max":{"1min":1.109,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.67500000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.80700000000000005,"15min":0.83199999999999996},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.98199999999999998,"5min":1.175,"15min":1.1919999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82999999999999996,"15min":0.85799999999999998},"min":{"1min":0.61599999999999999,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.0129999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"last":0.78200000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.84899999999999998,"15min":0.88200000000000001},"min":{"1min":0.66500000000000004,"5min":0.626,"15min":0.61199999999999999},"max":{"1min":1.0700000000000001,"5min":1.1479999999999999,"15min":1.3540000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.85699999999999998,"15min":0.88},"min":{"1min":0.60299999999999998,"5min":0.59999999999999998,"15min":0.59699999999999998},"max":{"1min":1.056,"5min":1.22,"15min":1.3420000000000001},"last":0.78800000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575966,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60480,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311360,"statfs":{"total":96636764160,"available":96574832640,"internally_reserved":0,"allocated":3198976,"data_stored":1855084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46999999999999997},"min":{"1min":0.308,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.087},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55600000000000005,"15min":0.54700000000000004},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.32400000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.38200000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.36099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.5289999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33200000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.63300000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.45300000000000001,"5min":0.40899999999999997,"15min":0.36299999999999999},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65800000000000003,"15min":0.64700000000000002},"min":{"1min":0.53000000000000003,"5min":0.46000000000000002,"15min":0.42199999999999999},"max":{"1min":0.83999999999999997,"5min":0.90000000000000002,"15min":1.3899999999999999},"last":0.58399999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.81899999999999995,"15min":0.81799999999999995},"min":{"1min":0.58699999999999997,"5min":0.495,"15min":0.495},"max":{"1min":1.0960000000000001,"5min":1.232,"15min":1.232},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.86099999999999999,"15min":0.85399999999999998},"min":{"1min":0.66800000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.99299999999999999,"5min":1.1890000000000001,"15min":1.5820000000000001},"last":0.72099999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.79900000000000004,"15min":0.78900000000000003},"min":{"1min":0.65700000000000003,"5min":0.53600000000000003,"15min":0.51200000000000001},"max":{"1min":1.0700000000000001,"5min":1.1870000000000001,"15min":1.3440000000000001},"last":0.754},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.82599999999999996},"min":{"1min":0.60999999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.73399999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.873,"15min":0.86199999999999999},"min":{"1min":0.63100000000000001,"5min":0.62,"15min":0.55000000000000004},"max":{"1min":1.343,"5min":1.343,"15min":1.6240000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.89400000000000002,"15min":0.89100000000000001},"min":{"1min":0.58499999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3240000000000001,"5min":1.337,"15min":1.337},"last":0.82199999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.83699999999999997,"15min":0.83099999999999996},"min":{"1min":0.629,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.115,"5min":1.214,"15min":1.214},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89700000000000002,"15min":0.88600000000000001},"min":{"1min":0.66500000000000004,"5min":0.66300000000000003,"15min":0.60399999999999998},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.67900000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575965,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47628,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324212,"statfs":{"total":96636764160,"available":96587993088,"internally_reserved":0,"allocated":2617344,"data_stored":1266407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47299999999999998,"15min":0.47599999999999998},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.35499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68700000000000006,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.53100000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48499999999999999,"15min":0.48799999999999999},"min":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.746,"5min":0.79000000000000004,"15min":0.80700000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.58799999999999997,"15min":0.58499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36199999999999999},"max":{"1min":0.77200000000000002,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61099999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60199999999999998,"15min":0.61099999999999999},"min":{"1min":0.45200000000000001,"5min":0.41999999999999998,"15min":0.41799999999999998},"max":{"1min":0.75600000000000001,"5min":0.80700000000000005,"15min":0.90400000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58099999999999996,"15min":0.58799999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.68799999999999994,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.59699999999999998,"5min":0.58699999999999997,"15min":0.55100000000000005},"max":{"1min":1.091,"5min":1.169,"15min":1.1970000000000001},"last":0.624},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81899999999999995,"15min":0.82699999999999996},"min":{"1min":0.63,"5min":0.57299999999999995,"15min":0.52400000000000002},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.67100000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.81499999999999995,"15min":0.83299999999999996},"min":{"1min":0.63900000000000001,"5min":0.56499999999999995,"15min":0.53400000000000003},"max":{"1min":1.1599999999999999,"5min":1.256,"15min":1.256},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.83799999999999997,"15min":0.83699999999999997},"min":{"1min":0.60899999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.143,"5min":1.24,"15min":1.3140000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.86599999999999999,"15min":0.85999999999999999},"min":{"1min":0.59599999999999997,"5min":0.58899999999999997,"15min":0.57799999999999996},"max":{"1min":1.1459999999999999,"5min":1.208,"15min":1.24},"last":0.871},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85899999999999999,"15min":0.85999999999999999},"min":{"1min":0.71599999999999997,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.075,"5min":1.1879999999999999,"15min":1.329},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.71599999999999997,"5min":0.65100000000000002,"15min":0.64600000000000002},"max":{"1min":1.1910000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.66400000000000003,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.0980000000000001,"5min":1.304,"15min":1.304},"last":0.82399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575965,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64008,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307832,"statfs":{"total":96636764160,"available":96571219968,"internally_reserved":0,"allocated":2617344,"data_stored":1265907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":62912968},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.35399999999999998,"5min":0.33000000000000002,"15min":0.313},"max":{"1min":0.55500000000000005,"5min":0.748,"15min":0.75900000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.318},"max":{"1min":0.82299999999999995,"5min":0.84799999999999998,"15min":0.96099999999999997},"last":0.40899999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.498,"15min":0.499},"min":{"1min":0.35899999999999999,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":0.58499999999999996,"5min":0.71599999999999997,"15min":0.77100000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.503,"5min":0.57299999999999995,"15min":0.57899999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.65900000000000003,"5min":1.046,"15min":1.046},"last":0.51600000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.44,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":0.67300000000000004,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.66000000000000003,"15min":0.66400000000000003},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.442},"max":{"1min":0.77900000000000003,"5min":0.89700000000000002,"15min":1.01},"last":0.58599999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.79800000000000004},"min":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.1339999999999999,"5min":1.266,"15min":1.2889999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81699999999999995,"15min":0.81599999999999995},"min":{"1min":0.65900000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0680000000000001,"5min":1.1830000000000001,"15min":1.216},"last":0.80900000000000005}]},{"osd":5,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.81499999999999995,"15min":0.81200000000000006},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.502},"max":{"1min":1.1160000000000001,"5min":1.1399999999999999,"15min":1.252},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.81000000000000005,"15min":0.82599999999999996},"min":{"1min":0.66500000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.98799999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.77500000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.84499999999999997,"15min":0.85499999999999998},"min":{"1min":0.71599999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.095,"5min":1.288,"15min":1.288},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.63200000000000001,"5min":0.63100000000000001,"15min":0.59099999999999997},"max":{"1min":1.0289999999999999,"5min":1.248,"15min":1.248},"last":1.147}]},{"osd":7,"last update":"Fri Mar 29 11:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85799999999999998,"15min":0.86599999999999999},"min":{"1min":0.54500000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1870000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":1.052},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.88100000000000001,"15min":0.89200000000000002},"min":{"1min":0.69999999999999996,"5min":0.58099999999999996,"15min":0.57899999999999996},"max":{"1min":1.0289999999999999,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":1.113}]}]},{"osd":4,"up_from":13,"seq":55834575966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59916,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311924,"statfs":{"total":96636764160,"available":96575410176,"internally_reserved":0,"allocated":2621440,"data_stored":1267859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72899999999999998,"15min":0.73699999999999999},"min":{"1min":0.503,"5min":0.39900000000000002,"15min":0.371},"max":{"1min":0.95499999999999996,"5min":1.0309999999999999,"15min":1.0509999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.53500000000000003,"5min":0.49099999999999999,"15min":0.38600000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.1459999999999999},"last":0.64300000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.75,"15min":0.76500000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.39900000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.74399999999999999,"15min":0.76100000000000001},"min":{"1min":0.54500000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.002},"last":0.59799999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.82799999999999996,"15min":0.84299999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82199999999999995,"15min":0.82699999999999996},"min":{"1min":0.67700000000000005,"5min":0.58099999999999996,"15min":0.53900000000000003},"max":{"1min":1.01,"5min":1.0529999999999999,"15min":1.1299999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.76700000000000002,"15min":0.77900000000000003},"min":{"1min":0.54800000000000004,"5min":0.52200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.181},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.78400000000000003,"15min":0.77700000000000002},"min":{"1min":0.55300000000000005,"5min":0.51300000000000001,"15min":0.48699999999999999},"max":{"1min":0.995,"5min":0.995,"15min":1.2490000000000001},"last":0.84399999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.626},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82599999999999996,"5min":0.93000000000000005,"15min":1.004},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.64900000000000002},"min":{"1min":0.45500000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.84999999999999998,"5min":1.071,"15min":1.169},"last":0.79000000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66200000000000003},"min":{"1min":0.436,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0800000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70199999999999996,"15min":0.71399999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.44700000000000001},"max":{"1min":1.0089999999999999,"5min":1.024,"15min":1.028},"last":0.71399999999999997}]},{"osd":7,"last update":"Fri Mar 29 11:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.71499999999999997,"15min":0.72499999999999998},"min":{"1min":0.56000000000000005,"5min":0.436,"15min":0.436},"max":{"1min":0.96699999999999997,"5min":1.052,"15min":1.518},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81899999999999995,"15min":0.82399999999999995},"min":{"1min":0.65400000000000003,"5min":0.54400000000000004,"15min":0.54300000000000004},"max":{"1min":1.0289999999999999,"5min":1.0860000000000001,"15min":1.3440000000000001},"last":0.877}]}]},{"osd":5,"up_from":13,"seq":55834575965,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41856,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329984,"statfs":{"total":96636764160,"available":96593903616,"internally_reserved":0,"allocated":2605056,"data_stored":1265713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.72399999999999998,"15min":0.71999999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86599999999999999,"5min":0.95999999999999996,"15min":1.0149999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.745,"15min":0.749},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.92700000000000005,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77500000000000002,"15min":0.76800000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92500000000000004,"5min":0.997,"15min":1.0740000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.753,"15min":0.751},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.47399999999999998},"max":{"1min":0.93999999999999995,"5min":0.98999999999999999,"15min":1.085},"last":0.504}]},{"osd":2,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77300000000000002,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.42799999999999999},"max":{"1min":0.95299999999999996,"5min":1.04,"15min":1.052},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.76200000000000001,"15min":0.75900000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.48399999999999999},"max":{"1min":0.94399999999999995,"5min":1.0129999999999999,"15min":1.0740000000000001},"last":0.79200000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.502,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.92200000000000004,"5min":1.077,"15min":1.0980000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77700000000000002,"15min":0.78100000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.51800000000000002},"max":{"1min":0.88100000000000001,"5min":1.046,"15min":1.046},"last":0.63400000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.38700000000000001},"max":{"1min":0.76100000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64400000000000002,"15min":0.63800000000000001},"min":{"1min":0.45100000000000001,"5min":0.45000000000000001,"15min":0.40600000000000003},"max":{"1min":0.76700000000000002,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42699999999999999},"max":{"1min":0.77800000000000002,"5min":1.038,"15min":1.038},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.42499999999999999},"max":{"1min":0.90900000000000003,"5min":1.107,"15min":1.107},"last":0.89400000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.72899999999999998,"15min":0.71199999999999997},"min":{"1min":0.52900000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88400000000000001,"5min":1.129,"15min":1.153},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.76900000000000002,"15min":0.75600000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1539999999999999},"last":0.82099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:28:55.515 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T11:28:55.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T11:28:55.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.765+0000 7f7247000700 1 Processor -- start 2024-03-29T11:28:55.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.765+0000 7f7247000700 1 -- start start 2024-03-29T11:28:55.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.766+0000 7f7247000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f72400cd020 0x7f72400cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.767+0000 7f7247000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400b2070 0x7f72400b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.767+0000 7f7247000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72400cdae0 con 0x7f72400cbb30 2024-03-29T11:28:55.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.767+0000 7f7247000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72400cddc0 con 0x7f72400cd020 2024-03-29T11:28:55.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.767+0000 7f7247000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72400ce0a0 con 0x7f72400b2070 2024-03-29T11:28:55.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.767+0000 7f723f400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f72400cd020 0x7f72400cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.767+0000 7f723f400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f72400cd020 0x7f72400cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:39216/0 (socket says 172.21.15.12:39216) 2024-03-29T11:28:55.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.767+0000 7f7245000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400b2070 0x7f72400b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.767+0000 7f723f400700 1 -- 172.21.15.12:0/3681453466 learned_addr learned my addr 172.21.15.12:0/3681453466 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:55.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.767+0000 7f723f400700 1 -- 172.21.15.12:0/3681453466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400b2070 msgr2=0x7f72400b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.767+0000 7f723f400700 1 --2- 172.21.15.12:0/3681453466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400b2070 0x7f72400b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.768+0000 7f723f400700 1 -- 172.21.15.12:0/3681453466 >> v1:172.21.15.159:6789/0 conn(0x7f72400cbb30 legacy=0x7f72400cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.768+0000 7f723f400700 1 -- 172.21.15.12:0/3681453466 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724006dc20 con 0x7f72400cd020 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.768+0000 7f723f400700 1 --2- 172.21.15.12:0/3681453466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72400cd020 0x7f72400cd480 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f722c0098f0 tx=0x7f722c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12d94c4924d9bd16 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.768+0000 7f723ea00700 1 -- 172.21.15.12:0/3681453466 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c011830 con 0x7f72400cd020 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.768+0000 7f723ea00700 1 -- 172.21.15.12:0/3681453466 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722c011990 con 0x7f72400cd020 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.768+0000 7f723ea00700 1 -- 172.21.15.12:0/3681453466 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c011d20 con 0x7f72400cd020 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.769+0000 7f7247000700 1 -- 172.21.15.12:0/3681453466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72400cd020 msgr2=0x7f72400cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.769+0000 7f7247000700 1 --2- 172.21.15.12:0/3681453466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72400cd020 0x7f72400cd480 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f722c0098f0 tx=0x7f722c009c00 comp rx=0 tx=0).stop 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.769+0000 7f7247000700 1 -- 172.21.15.12:0/3681453466 shutdown_connections 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.769+0000 7f7247000700 1 --2- 172.21.15.12:0/3681453466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400b2070 0x7f72400b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.769+0000 7f7247000700 1 --2- 172.21.15.12:0/3681453466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72400cd020 0x7f72400cd480 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.769+0000 7f7247000700 1 -- 172.21.15.12:0/3681453466 >> 172.21.15.12:0/3681453466 conn(0x7f72400b0d70 msgr2=0x7f72400b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.769+0000 7f7247000700 1 -- 172.21.15.12:0/3681453466 shutdown_connections 2024-03-29T11:28:55.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.769+0000 7f7247000700 1 -- 172.21.15.12:0/3681453466 wait complete. 2024-03-29T11:28:55.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.770+0000 7f7247000700 1 Processor -- start 2024-03-29T11:28:55.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.771+0000 7f7247000700 1 -- start start 2024-03-29T11:28:55.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.771+0000 7f7247000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72400b2070 0x7f72400c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.771+0000 7f7247000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72400cbb30 0x7f7240157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.771+0000 7f7247000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400cd020 0x7f7240071cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.771+0000 7f7247000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72400c8930 con 0x7f72400cbb30 2024-03-29T11:28:55.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.771+0000 7f7247000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72400c8bb0 con 0x7f72400b2070 2024-03-29T11:28:55.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.771+0000 7f7247000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72400c8e90 con 0x7f72400cd020 2024-03-29T11:28:55.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f723f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72400cbb30 0x7f7240157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f723f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72400cbb30 0x7f7240157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54658/0 (socket says 172.21.15.12:54658) 2024-03-29T11:28:55.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f723f400700 1 -- 172.21.15.12:0/2696569322 learned_addr learned my addr 172.21.15.12:0/2696569322 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f723fe00700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72400b2070 0x7f72400c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f723f400700 1 -- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400cd020 msgr2=0x7f7240071cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f7245000700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400cd020 0x7f7240071cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f723f400700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400cd020 0x7f7240071cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f723f400700 1 -- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72400b2070 msgr2=0x7f72400c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f723f400700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72400b2070 0x7f72400c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f723f400700 1 -- 172.21.15.12:0/2696569322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c009550 con 0x7f72400cbb30 2024-03-29T11:28:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f7245000700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400cd020 0x7f7240071cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.772+0000 7f723fe00700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72400b2070 0x7f72400c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.773+0000 7f723f400700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72400cbb30 0x7f7240157c20 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f722c000c00 tx=0x7f722c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.773+0000 7f7237e00700 1 -- 172.21.15.12:0/2696569322 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c004530 con 0x7f72400cbb30 2024-03-29T11:28:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.773+0000 7f7237e00700 1 -- 172.21.15.12:0/2696569322 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722c01f070 con 0x7f72400cbb30 2024-03-29T11:28:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.773+0000 7f7247000700 1 -- 172.21.15.12:0/2696569322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72400c9170 con 0x7f72400cbb30 2024-03-29T11:28:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.773+0000 7f7237e00700 1 -- 172.21.15.12:0/2696569322 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c0038e0 con 0x7f72400cbb30 2024-03-29T11:28:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.773+0000 7f7247000700 1 -- 172.21.15.12:0/2696569322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72400c9730 con 0x7f72400cbb30 2024-03-29T11:28:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.774+0000 7f7247000700 1 -- 172.21.15.12:0/2696569322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72400c9920 con 0x7f72400cbb30 2024-03-29T11:28:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.775+0000 7f7237e00700 1 -- 172.21.15.12:0/2696569322 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f722c021030 con 0x7f72400cbb30 2024-03-29T11:28:55.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.776+0000 7f7237e00700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f722006adb0 0x7f722006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:55.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.776+0000 7f7237e00700 1 -- 172.21.15.12:0/2696569322 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f722c08a8d0 con 0x7f72400cbb30 2024-03-29T11:28:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.777+0000 7f723fe00700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f722006adb0 0x7f722006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:55.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.777+0000 7f723fe00700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f722006adb0 0x7f722006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7230000f60 tx=0x7f7230009040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:55.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:55.780+0000 7f7237e00700 1 -- 172.21.15.12:0/2696569322 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f722c0559e0 con 0x7f72400cbb30 2024-03-29T11:28:56.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.078+0000 7f7247000700 1 -- 172.21.15.12:0/2696569322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7240067720 con 0x7f72400cbb30 2024-03-29T11:28:56.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.083+0000 7f7237e00700 1 -- 172.21.15.12:0/2696569322 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v48) v1 ==== 190+0+0 (secure 0 0 0) 0x7f722c0150e0 con 0x7f72400cbb30 2024-03-29T11:28:56.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 -- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f722006adb0 msgr2=0x7f722006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f722006adb0 0x7f722006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7230000f60 tx=0x7f7230009040 comp rx=0 tx=0).stop 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 -- 172.21.15.12:0/2696569322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72400cbb30 msgr2=0x7f7240157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72400cbb30 0x7f7240157c20 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f722c000c00 tx=0x7f722c004750 comp rx=0 tx=0).stop 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 -- 172.21.15.12:0/2696569322 shutdown_connections 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f722006adb0 0x7f722006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72400cd020 0x7f7240071cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72400b2070 0x7f72400c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 --2- 172.21.15.12:0/2696569322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72400cbb30 0x7f7240157c20 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 -- 172.21.15.12:0/2696569322 >> 172.21.15.12:0/2696569322 conn(0x7f72400b0d70 msgr2=0x7f72400afbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 -- 172.21.15.12:0/2696569322 shutdown_connections 2024-03-29T11:28:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.086+0000 7f7247000700 1 -- 172.21.15.12:0/2696569322 wait complete. 2024-03-29T11:28:56.078 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T11:28:56.095 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:28:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.301+0000 7fce67800700 1 Processor -- start 2024-03-29T11:28:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.302+0000 7fce67800700 1 -- start start 2024-03-29T11:28:56.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.303+0000 7fce67800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fce60072580 0x7fce600b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.303+0000 7fce67800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fce600b2e50 0x7fce600709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.303+0000 7fce67800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce600b2180 con 0x7fce600b3af0 2024-03-29T11:28:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.303+0000 7fce67800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6015b100 con 0x7fce60072580 2024-03-29T11:28:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.303+0000 7fce67800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6015b3e0 con 0x7fce600b2e50 2024-03-29T11:28:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.303+0000 7fce64e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fce600b3af0 0x7fce6015a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54216/0 (socket says 172.21.15.12:54216) 2024-03-29T11:28:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.303+0000 7fce64e00700 1 -- 172.21.15.12:0/498194363 learned_addr learned my addr 172.21.15.12:0/498194363 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.303+0000 7fce5fe00700 1 --2- 172.21.15.12:0/498194363 >> v2:172.21.15.12:3300/0 conn(0x7fce60072580 0x7fce600b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.303+0000 7fce65800700 1 --2- 172.21.15.12:0/498194363 >> v2:172.21.15.159:3300/0 conn(0x7fce600b2e50 0x7fce600709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.304+0000 7fce5fe00700 1 -- 172.21.15.12:0/498194363 >> v2:172.21.15.159:3300/0 conn(0x7fce600b2e50 msgr2=0x7fce600709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.304+0000 7fce5fe00700 1 --2- 172.21.15.12:0/498194363 >> v2:172.21.15.159:3300/0 conn(0x7fce600b2e50 0x7fce600709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.304+0000 7fce5fe00700 1 -- 172.21.15.12:0/498194363 >> v1:172.21.15.12:6789/0 conn(0x7fce600b3af0 legacy=0x7fce6015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.304+0000 7fce5fe00700 1 -- 172.21.15.12:0/498194363 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce600b23f0 con 0x7fce60072580 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.304+0000 7fce65800700 1 --2- 172.21.15.12:0/498194363 >> v2:172.21.15.159:3300/0 conn(0x7fce600b2e50 0x7fce600709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.304+0000 7fce5fe00700 1 --2- 172.21.15.12:0/498194363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce60072580 0x7fce600b2910 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fce60158390 tx=0x7fce4c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a548c0480177585 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.304+0000 7fce5f400700 1 -- 172.21.15.12:0/498194363 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c011830 con 0x7fce60072580 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.304+0000 7fce5f400700 1 -- 172.21.15.12:0/498194363 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4c011990 con 0x7fce60072580 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.304+0000 7fce5f400700 1 -- 172.21.15.12:0/498194363 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c011d20 con 0x7fce60072580 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.305+0000 7fce67800700 1 -- 172.21.15.12:0/498194363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce60072580 msgr2=0x7fce600b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.305+0000 7fce67800700 1 --2- 172.21.15.12:0/498194363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce60072580 0x7fce600b2910 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fce60158390 tx=0x7fce4c009b20 comp rx=0 tx=0).stop 2024-03-29T11:28:56.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.305+0000 7fce67800700 1 -- 172.21.15.12:0/498194363 shutdown_connections 2024-03-29T11:28:56.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.305+0000 7fce67800700 1 --2- 172.21.15.12:0/498194363 >> v2:172.21.15.159:3300/0 conn(0x7fce600b2e50 0x7fce600709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.305+0000 7fce67800700 1 --2- 172.21.15.12:0/498194363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce60072580 0x7fce600b2910 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.305+0000 7fce67800700 1 -- 172.21.15.12:0/498194363 >> 172.21.15.12:0/498194363 conn(0x7fce600cf190 msgr2=0x7fce600cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:56.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.305+0000 7fce67800700 1 -- 172.21.15.12:0/498194363 shutdown_connections 2024-03-29T11:28:56.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.305+0000 7fce67800700 1 -- 172.21.15.12:0/498194363 wait complete. 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.307+0000 7fce67800700 1 Processor -- start 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.307+0000 7fce67800700 1 -- start start 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce67800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce60072580 0x7fce6015d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce67800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce600b2e50 0x7fce601e9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce67800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce600b3af0 0x7fce60152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce67800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce601e99d0 con 0x7fce600b3af0 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce67800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6015bdd0 con 0x7fce600b2e50 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce67800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6015c0b0 con 0x7fce60072580 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce65800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce600b3af0 0x7fce60152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce65800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce600b3af0 0x7fce60152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54686/0 (socket says 172.21.15.12:54686) 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce65800700 1 -- 172.21.15.12:0/794389517 learned_addr learned my addr 172.21.15.12:0/794389517 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:56.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce5fe00700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce600b2e50 0x7fce601e9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce64e00700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce60072580 0x7fce6015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce65800700 1 -- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce60072580 msgr2=0x7fce6015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce65800700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce60072580 0x7fce6015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce65800700 1 -- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce600b2e50 msgr2=0x7fce601e9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce65800700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce600b2e50 0x7fce601e9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.308+0000 7fce65800700 1 -- 172.21.15.12:0/794389517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c0094a0 con 0x7fce600b3af0 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.309+0000 7fce65800700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce600b3af0 0x7fce60152ab0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fce58009fa0 tx=0x7fce5800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.309+0000 7fce5cc00700 1 -- 172.21.15.12:0/794389517 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5800ddb0 con 0x7fce600b3af0 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.309+0000 7fce5cc00700 1 -- 172.21.15.12:0/794389517 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5800f040 con 0x7fce600b3af0 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.309+0000 7fce67800700 1 -- 172.21.15.12:0/794389517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6015c390 con 0x7fce600b3af0 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.309+0000 7fce5cc00700 1 -- 172.21.15.12:0/794389517 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58003c60 con 0x7fce600b3af0 2024-03-29T11:28:56.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.309+0000 7fce67800700 1 -- 172.21.15.12:0/794389517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6015c950 con 0x7fce600b3af0 2024-03-29T11:28:56.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.310+0000 7fce67800700 1 -- 172.21.15.12:0/794389517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6015cb70 con 0x7fce600b3af0 2024-03-29T11:28:56.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.311+0000 7fce5cc00700 1 -- 172.21.15.12:0/794389517 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fce58004030 con 0x7fce600b3af0 2024-03-29T11:28:56.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.312+0000 7fce5cc00700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fce4006adb0 0x7fce4006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:56.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.312+0000 7fce5cc00700 1 -- 172.21.15.12:0/794389517 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fce580896a0 con 0x7fce600b3af0 2024-03-29T11:28:56.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.312+0000 7fce64e00700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fce4006adb0 0x7fce4006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:56.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.313+0000 7fce64e00700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fce4006adb0 0x7fce4006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fce54005910 tx=0x7fce54009040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:56.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.316+0000 7fce5cc00700 1 -- 172.21.15.12:0/794389517 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce58054600 con 0x7fce600b3af0 2024-03-29T11:28:56.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.576+0000 7fce67800700 1 -- 172.21.15.12:0/794389517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce60067720 con 0x7fce600b3af0 2024-03-29T11:28:56.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.577+0000 7fce5cc00700 1 -- 172.21.15.12:0/794389517 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fce58057c20 con 0x7fce600b3af0 2024-03-29T11:28:56.567 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:28:56.567 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/395980670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":395980670},{"type":"v1","addr":"172.21.15.159:6837","nonce":395980670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":93},{"gid":10183,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/531080306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":531080306},{"type":"v1","addr":"172.21.15.159:6835","nonce":531080306}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10509,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2709565550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2709565550},{"type":"v1","addr":"172.21.15.12:6834","nonce":2709565550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:28:50.095373+0000","modified":"2024-03-29T11:28:53.584378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10512":{"gid":10512,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.12:6835/99601797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":99601797},{"type":"v1","addr":"172.21.15.12:6835","nonce":99601797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-29T11:28:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.579+0000 7fce67800700 1 -- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fce4006adb0 msgr2=0x7fce4006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.579+0000 7fce67800700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fce4006adb0 0x7fce4006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fce54005910 tx=0x7fce54009040 comp rx=0 tx=0).stop 2024-03-29T11:28:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.579+0000 7fce67800700 1 -- 172.21.15.12:0/794389517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce600b3af0 msgr2=0x7fce60152ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.579+0000 7fce67800700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce600b3af0 0x7fce60152ab0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fce58009fa0 tx=0x7fce5800d3b0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.579+0000 7fce67800700 1 -- 172.21.15.12:0/794389517 shutdown_connections 2024-03-29T11:28:56.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.579+0000 7fce67800700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fce4006adb0 0x7fce4006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.579+0000 7fce67800700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce600b3af0 0x7fce60152ab0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.579+0000 7fce67800700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce60072580 0x7fce6015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.579+0000 7fce67800700 1 --2- 172.21.15.12:0/794389517 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce600b2e50 0x7fce601e9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:56.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.579+0000 7fce67800700 1 -- 172.21.15.12:0/794389517 >> 172.21.15.12:0/794389517 conn(0x7fce600cf190 msgr2=0x7fce600acde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:56.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.580+0000 7fce67800700 1 -- 172.21.15.12:0/794389517 shutdown_connections 2024-03-29T11:28:56.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:56.580+0000 7fce67800700 1 -- 172.21.15.12:0/794389517 wait complete. 2024-03-29T11:28:56.571 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 98 2024-03-29T11:28:56.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T11:28:50.095373+0000', 'modified': '2024-03-29T11:28:53.584378+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10512}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10512': {'gid': 10512, 'name': 'a', 'rank': 0, 'incarnation': 94, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.12:6835/99601797', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6833', 'nonce': 99601797}, {'type': 'v1', 'addr': '172.21.15.12:6835', 'nonce': 99601797}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T11:28:56.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T11:28:56.994 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-29T11:28:57.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.206+0000 7f865f000700 1 Processor -- start 2024-03-29T11:28:57.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.206+0000 7f865f000700 1 -- start start 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.207+0000 7f865f000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f86580cd5c0 0x7f865815e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.207+0000 7f865f000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f865806f310 0x7f865806f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.207+0000 7f865f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f865815cfe0 0x7f865815d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.207+0000 7f865f000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865815eda0 con 0x7f86580cd5c0 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.207+0000 7f865f000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865806fce0 con 0x7f865806f310 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.207+0000 7f865f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8658099dd0 con 0x7f865815cfe0 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.207+0000 7f8657e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f86580cd5c0 0x7f865815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.207+0000 7f8657e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f86580cd5c0 0x7f865815e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54690/0 (socket says 172.21.15.12:54690) 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.207+0000 7f8657e00700 1 -- 172.21.15.12:0/2412378486 learned_addr learned my addr 172.21.15.12:0/2412378486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f865d000700 1 --2- 172.21.15.12:0/2412378486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f865815cfe0 0x7f865815d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f8657400700 1 --2- 172.21.15.12:0/2412378486 >> v2:172.21.15.159:3300/0 conn(0x7f865806f310 0x7f865806f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f8657e00700 1 -- 172.21.15.12:0/2412378486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f865815cfe0 msgr2=0x7f865815d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f8657e00700 1 --2- 172.21.15.12:0/2412378486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f865815cfe0 0x7f865815d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f8657e00700 1 -- 172.21.15.12:0/2412378486 >> v2:172.21.15.159:3300/0 conn(0x7f865806f310 msgr2=0x7f865806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f8657e00700 1 --2- 172.21.15.12:0/2412378486 >> v2:172.21.15.159:3300/0 conn(0x7f865806f310 0x7f865806f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f8657e00700 1 -- 172.21.15.12:0/2412378486 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865809a0b0 con 0x7f86580cd5c0 2024-03-29T11:28:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f8657e00700 1 --2- 172.21.15.12:0/2412378486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 0x7f865815e860 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f86480098f0 tx=0x7f8648009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b40ed4f1a5a15d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f8656a00700 1 -- 172.21.15.12:0/2412378486 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864800b860 con 0x7f86580cd5c0 2024-03-29T11:28:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f8656a00700 1 -- 172.21.15.12:0/2412378486 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864800b9c0 con 0x7f86580cd5c0 2024-03-29T11:28:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.208+0000 7f8656a00700 1 -- 172.21.15.12:0/2412378486 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864800bd50 con 0x7f86580cd5c0 2024-03-29T11:28:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.209+0000 7f865f000700 1 -- 172.21.15.12:0/2412378486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 msgr2=0x7f865815e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.209+0000 7f865f000700 1 --2- 172.21.15.12:0/2412378486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 0x7f865815e860 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f86480098f0 tx=0x7f8648009c00 comp rx=0 tx=0).stop 2024-03-29T11:28:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.209+0000 7f865f000700 1 -- 172.21.15.12:0/2412378486 shutdown_connections 2024-03-29T11:28:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.209+0000 7f865f000700 1 --2- 172.21.15.12:0/2412378486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f865815cfe0 0x7f865815d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.209+0000 7f865f000700 1 --2- 172.21.15.12:0/2412378486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 0x7f865815e860 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.209+0000 7f865f000700 1 --2- 172.21.15.12:0/2412378486 >> v2:172.21.15.159:3300/0 conn(0x7f865806f310 0x7f865806f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.209+0000 7f865f000700 1 -- 172.21.15.12:0/2412378486 >> 172.21.15.12:0/2412378486 conn(0x7f86580d1370 msgr2=0x7f86580d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.209+0000 7f865f000700 1 -- 172.21.15.12:0/2412378486 shutdown_connections 2024-03-29T11:28:57.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.209+0000 7f865f000700 1 -- 172.21.15.12:0/2412378486 wait complete. 2024-03-29T11:28:57.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.211+0000 7f865f000700 1 Processor -- start 2024-03-29T11:28:57.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.211+0000 7f865f000700 1 -- start start 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f865f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f865806f310 0x7f86581f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f865f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 0x7f86580ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f865f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f865815cfe0 0x7f86580ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f865f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86581f4150 con 0x7f86580cd5c0 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f865f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86581f34d0 con 0x7f865815cfe0 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f865f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86581f37b0 con 0x7f865806f310 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f8657400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 0x7f86580ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f8657400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 0x7f86580ae160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54696/0 (socket says 172.21.15.12:54696) 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f8657400700 1 -- 172.21.15.12:0/3418626350 learned_addr learned my addr 172.21.15.12:0/3418626350 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f8657e00700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f865806f310 0x7f86581f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.212+0000 7f865d000700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f865815cfe0 0x7f86580ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f8657400700 1 -- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f865806f310 msgr2=0x7f86581f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f8657400700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f865806f310 0x7f86581f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f8657400700 1 -- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f865815cfe0 msgr2=0x7f86580ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f8657400700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f865815cfe0 0x7f86580ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f8657400700 1 -- 172.21.15.12:0/3418626350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8648009550 con 0x7f86580cd5c0 2024-03-29T11:28:57.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f865d000700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f865815cfe0 0x7f86580ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:57.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f8657400700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 0x7f86580ae160 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f864400d800 tx=0x7f864400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:57.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f864fe00700 1 -- 172.21.15.12:0/3418626350 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86440125f0 con 0x7f86580cd5c0 2024-03-29T11:28:57.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f864fe00700 1 -- 172.21.15.12:0/3418626350 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8644012750 con 0x7f86580cd5c0 2024-03-29T11:28:57.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f865f000700 1 -- 172.21.15.12:0/3418626350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86581f3a90 con 0x7f86580cd5c0 2024-03-29T11:28:57.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f864fe00700 1 -- 172.21.15.12:0/3418626350 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644012b00 con 0x7f86580cd5c0 2024-03-29T11:28:57.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.213+0000 7f865f000700 1 -- 172.21.15.12:0/3418626350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86581f31b0 con 0x7f86580cd5c0 2024-03-29T11:28:57.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.215+0000 7f864fe00700 1 -- 172.21.15.12:0/3418626350 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8644012c60 con 0x7f86580cd5c0 2024-03-29T11:28:57.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.217+0000 7f864fe00700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f863806adb0 0x7f863806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.217+0000 7f864fe00700 1 -- 172.21.15.12:0/3418626350 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f8644089880 con 0x7f86580cd5c0 2024-03-29T11:28:57.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.217+0000 7f8657e00700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f863806adb0 0x7f863806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.218+0000 7f8657e00700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f863806adb0 0x7f863806d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8648009840 tx=0x7f8648008030 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:57.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.219+0000 7f8672bc8b80 1 Processor -- start 2024-03-29T11:28:57.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.219+0000 7f8672bc8b80 1 -- start start 2024-03-29T11:28:57.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 0x563d1f6fec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f6d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d1f649650 con 0x563d1f6cfb60 2024-03-29T11:28:57.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d1f62ac30 con 0x563d1f6fe830 2024-03-29T11:28:57.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d1f628430 con 0x563d1f6d1be0 2024-03-29T11:28:57.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f865d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f865d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54702/0 (socket says 172.21.15.12:54702) 2024-03-29T11:28:57.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f865d000700 1 -- 172.21.15.12:0/1338697748 learned_addr learned my addr 172.21.15.12:0/1338697748 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:57.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f8657400700 1 --2- 172.21.15.12:0/1338697748 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f6d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f8657e00700 1 --2- 172.21.15.12:0/1338697748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 0x563d1f6fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f865d000700 1 -- 172.21.15.12:0/1338697748 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 msgr2=0x563d1f6d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f865d000700 1 --2- 172.21.15.12:0/1338697748 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f6d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f865d000700 1 -- 172.21.15.12:0/1338697748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 msgr2=0x563d1f6fec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f865d000700 1 --2- 172.21.15.12:0/1338697748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 0x563d1f6fec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f865d000700 1 -- 172.21.15.12:0/1338697748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d1f417790 con 0x563d1f6cfb60 2024-03-29T11:28:57.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.220+0000 7f8657e00700 1 --2- 172.21.15.12:0/1338697748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 0x563d1f6fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:57.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.221+0000 7f865d000700 1 --2- 172.21.15.12:0/1338697748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fba20 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f8650007430 tx=0x7f8650007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58d983f6fd94846 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:57.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.221+0000 7f864cc00700 1 -- 172.21.15.12:0/1338697748 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650015670 con 0x563d1f6cfb60 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.221+0000 7f864cc00700 1 -- 172.21.15.12:0/1338697748 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865000f050 con 0x563d1f6cfb60 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.221+0000 7f864cc00700 1 -- 172.21.15.12:0/1338697748 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650015a10 con 0x563d1f6cfb60 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.222+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1338697748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 msgr2=0x563d1f6fba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.222+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/1338697748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fba20 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f8650007430 tx=0x7f8650007740 comp rx=0 tx=0).stop 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.222+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1338697748 shutdown_connections 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.222+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/1338697748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fba20 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.222+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/1338697748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 0x563d1f6fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.222+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/1338697748 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f6d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.222+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1338697748 >> 172.21.15.12:0/1338697748 conn(0x563d1f6326a0 msgr2=0x563d1f719640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.222+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1338697748 shutdown_connections 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.222+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1338697748 wait complete. 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.223+0000 7f8672bc8b80 1 -- start start 2024-03-29T11:28:57.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.223+0000 7f8672bc8b80 1 Processor -- start 2024-03-29T11:28:57.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f6fde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 0x563d1f71ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d1f623d20 con 0x563d1f6cfb60 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d1f5bf770 con 0x563d1f6fe830 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d1f5e26f0 con 0x563d1f6d1be0 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8672bc8b80 10 client.0 _ll_register_callbacks cb 0x563d1f5c95e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8657e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8657e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54712/0 (socket says 172.21.15.12:54712) 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8657e00700 1 -- 172.21.15.12:0/1173887107 learned_addr learned my addr 172.21.15.12:0/1173887107 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.224+0000 7f8657400700 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f6fde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f865d000700 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 0x563d1f71ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8657e00700 1 -- 172.21.15.12:0/1173887107 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 msgr2=0x563d1f6fde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8657e00700 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f6fde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8657e00700 1 -- 172.21.15.12:0/1173887107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 msgr2=0x563d1f71ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8657e00700 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 0x563d1f71ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8657e00700 1 -- 172.21.15.12:0/1173887107 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8650009cf0 con 0x563d1f6cfb60 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8657400700 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f6fde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8657e00700 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fce60 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f864800b7b0 tx=0x7f86480047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e5bacda1007946 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8641600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8641600700 1 -- 172.21.15.12:0/1173887107 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864802e070 con 0x563d1f6cfb60 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8641600700 1 -- 172.21.15.12:0/1173887107 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f8648034b20 con 0x563d1f6cfb60 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.225+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1173887107 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d1f6c6920 con 0x563d1f6cfb60 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8641600700 1 -- 172.21.15.12:0/1173887107 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8648020070 con 0x563d1f6cfb60 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8672bc8b80 10 client.10644 fetch_fsmap learned FSMap version 98 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1173887107 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x563d1f5040a0 con 0x563d1f6cfb60 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8641600700 1 -- 172.21.15.12:0/1173887107 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563d1f6c6920 con 0x563d1f6cfb60 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8641600700 1 -- 172.21.15.12:0/1173887107 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 98) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8648034ce0 con 0x563d1f6cfb60 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8672bc8b80 10 client.10644 fetch_fsmap finished waiting for FSMap version 98 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8672bc8b80 10 client.10644 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T11:28:57.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8672bc8b80 20 client.10644 populate_metadata read hostname 'smithi012' 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.12:6833/99601797,v1:172.21.15.12:6835/99601797] conn(0x563d1f701a70 0x563d1f6ffcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8672bc8b80 4 client.10644 mds_command: new command op to 10512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1173887107 --> [v2:172.21.15.12:6833/99601797,v1:172.21.15.12:6835/99601797] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d1f6a36f0 con 0x563d1f701a70 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.226+0000 7f8657400700 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.12:6833/99601797,v1:172.21.15.12:6835/99601797] conn(0x563d1f701a70 0x563d1f6ffcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.227+0000 7f8657400700 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.12:6833/99601797,v1:172.21.15.12:6835/99601797] conn(0x563d1f701a70 0x563d1f6ffcb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f43bab2c21dacb60 server_cookie=360c52236d98a380 in_seq=0 out_seq=0 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.227+0000 7f8641600700 10 client.10644 ms_handle_connect on v2:172.21.15.12:6833/99601797 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.227+0000 7f8641600700 1 -- 172.21.15.12:0/1173887107 <== mds.0 v2:172.21.15.12:6833/99601797 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563d1f6a36f0 con 0x563d1f701a70 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.227+0000 7f8641600700 10 client.10644 handle_command_reply: tid=0 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 client.10644 shutdown 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1173887107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 msgr2=0x563d1f6fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fce60 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f864800b7b0 tx=0x7f86480047c0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1173887107 shutdown_connections 2024-03-29T11:28:57.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.12:6833/99601797,v1:172.21.15.12:6835/99601797] conn(0x563d1f701a70 0x563d1f6ffcb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6fe830 0x563d1f71ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6cfb60 0x563d1f6fce60 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8641600700 0 client.10644 ms_handle_reset on v2:172.21.15.12:6833/99601797 2024-03-29T11:28:57.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/1173887107 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f6fde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1173887107 >> 172.21.15.12:0/1173887107 conn(0x563d1f6326a0 msgr2=0x563d1f6d3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1173887107 shutdown_connections 2024-03-29T11:28:57.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 1 -- 172.21.15.12:0/1173887107 wait complete. 2024-03-29T11:28:57.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.228+0000 7f8672bc8b80 20 client.10644 trim_cache size 0 max 16384 2024-03-29T11:28:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.247+0000 7f8672bc8b80 1 Processor -- start 2024-03-29T11:28:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.247+0000 7f8672bc8b80 1 -- start start 2024-03-29T11:28:57.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6cfb60 0x563d1f71dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6d1be0 0x563d1f702170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6d5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d1f7016f0 con 0x563d1f6fe830 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d1f418390 con 0x563d1f6d1be0 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d1f6f9620 con 0x563d1f6cfb60 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f865d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6d5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f865d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6d5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54722/0 (socket says 172.21.15.12:54722) 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f865d000700 1 -- 172.21.15.12:0/2564352759 learned_addr learned my addr 172.21.15.12:0/2564352759 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f8657e00700 1 --2- 172.21.15.12:0/2564352759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6cfb60 0x563d1f71dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.248+0000 7f8657400700 1 --2- 172.21.15.12:0/2564352759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6d1be0 0x563d1f702170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.249+0000 7f865d000700 1 -- 172.21.15.12:0/2564352759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6cfb60 msgr2=0x563d1f71dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.249+0000 7f865d000700 1 --2- 172.21.15.12:0/2564352759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6cfb60 0x563d1f71dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.249+0000 7f865d000700 1 -- 172.21.15.12:0/2564352759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6d1be0 msgr2=0x563d1f702170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.249+0000 7f865d000700 1 --2- 172.21.15.12:0/2564352759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6d1be0 0x563d1f702170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.249+0000 7f865d000700 1 -- 172.21.15.12:0/2564352759 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d1f5ea010 con 0x563d1f6fe830 2024-03-29T11:28:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.249+0000 7f865d000700 1 --2- 172.21.15.12:0/2564352759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6d5310 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f8650016000 tx=0x7f865000c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859e21fa8b1b5683 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.249+0000 7f864d600700 1 -- 172.21.15.12:0/2564352759 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650015670 con 0x563d1f6fe830 2024-03-29T11:28:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.249+0000 7f864d600700 1 -- 172.21.15.12:0/2564352759 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86500157d0 con 0x563d1f6fe830 2024-03-29T11:28:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.249+0000 7f864d600700 1 -- 172.21.15.12:0/2564352759 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650015b60 con 0x563d1f6fe830 2024-03-29T11:28:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.250+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2564352759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 msgr2=0x563d1f6d5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.250+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/2564352759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6d5310 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f8650016000 tx=0x7f865000c000 comp rx=0 tx=0).stop 2024-03-29T11:28:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.250+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2564352759 shutdown_connections 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.250+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/2564352759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6d5310 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.250+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/2564352759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6cfb60 0x563d1f71dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.250+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/2564352759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6d1be0 0x563d1f702170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.250+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2564352759 >> 172.21.15.12:0/2564352759 conn(0x563d1f6326a0 msgr2=0x563d1f6fb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.250+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2564352759 shutdown_connections 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.250+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2564352759 wait complete. 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.250+0000 7f8672bc8b80 1 -- start start 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.251+0000 7f8672bc8b80 1 Processor -- start 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6cfb60 0x563d1f77ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f72a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f8672bc8b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6f8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d1f6fa8e0 con 0x563d1f6fe830 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d1f6d38d0 con 0x563d1f6cfb60 2024-03-29T11:28:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f8672bc8b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d1f6d3a70 con 0x563d1f6d1be0 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f8672bc8b80 10 client.0 _ll_register_callbacks cb 0x7f864800bc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f865d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6f8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f865d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6f8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54726/0 (socket says 172.21.15.12:54726) 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f865d000700 1 -- 172.21.15.12:0/2903049064 learned_addr learned my addr 172.21.15.12:0/2903049064 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f8657e00700 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6cfb60 0x563d1f77ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f865d000700 1 -- 172.21.15.12:0/2903049064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 msgr2=0x563d1f72a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f865d000700 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f72a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f865d000700 1 -- 172.21.15.12:0/2903049064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6cfb60 msgr2=0x563d1f77ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f865d000700 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6cfb60 0x563d1f77ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f865d000700 1 -- 172.21.15.12:0/2903049064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d1f417790 con 0x563d1f6fe830 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.252+0000 7f8657e00700 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6cfb60 0x563d1f77ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f865d000700 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6f8da0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f8650000f80 tx=0x7f8650004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7902fe2f02f14af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8641600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8641600700 1 -- 172.21.15.12:0/2903049064 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650015d50 con 0x563d1f6fe830 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8641600700 1 -- 172.21.15.12:0/2903049064 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f8650032460 con 0x563d1f6fe830 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2903049064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f865000f050 con 0x563d1f6fe830 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8641600700 1 -- 172.21.15.12:0/2903049064 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8650007250 con 0x563d1f6fe830 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8672bc8b80 10 client.10650 fetch_fsmap learned FSMap version 98 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2903049064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x563d1f4781d0 con 0x563d1f6fe830 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8641600700 1 -- 172.21.15.12:0/2903049064 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865000f050 con 0x563d1f6fe830 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8641600700 1 -- 172.21.15.12:0/2903049064 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 98) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f865002e070 con 0x563d1f6fe830 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8672bc8b80 10 client.10650 fetch_fsmap finished waiting for FSMap version 98 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8672bc8b80 10 client.10650 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8672bc8b80 20 client.10650 populate_metadata read hostname 'smithi012' 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.12:6833/99601797,v1:172.21.15.12:6835/99601797] conn(0x563d1f6f8400 0x563d1f70f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8672bc8b80 4 client.10650 mds_command: new command op to 10512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T11:28:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.253+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2903049064 --> [v2:172.21.15.12:6833/99601797,v1:172.21.15.12:6835/99601797] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563d1f45d040 con 0x563d1f6f8400 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.254+0000 7f8657e00700 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.12:6833/99601797,v1:172.21.15.12:6835/99601797] conn(0x563d1f6f8400 0x563d1f70f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.254+0000 7f8657e00700 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.12:6833/99601797,v1:172.21.15.12:6835/99601797] conn(0x563d1f6f8400 0x563d1f70f7d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3761cd560e5918f7 server_cookie=a8907320a4ada2fd in_seq=0 out_seq=0 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.254+0000 7f8641600700 10 client.10650 ms_handle_connect on v2:172.21.15.12:6833/99601797 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.254+0000 7f8641600700 1 -- 172.21.15.12:0/2903049064 <== mds.0 v2:172.21.15.12:6833/99601797 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x563d1f45d040 con 0x563d1f6f8400 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.254+0000 7f8641600700 10 client.10650 handle_command_reply: tid=0 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.254+0000 7f8672bc8b80 1 client.10650 shutdown 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.254+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2903049064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 msgr2=0x563d1f6f8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.254+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6f8da0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f8650000f80 tx=0x7f8650004b40 comp rx=0 tx=0).stop 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2903049064 shutdown_connections 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.12:6833/99601797,v1:172.21.15.12:6835/99601797] conn(0x563d1f6f8400 0x563d1f70f7d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563d1f6fe830 0x563d1f6f8da0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x563d1f6cfb60 0x563d1f77ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f8672bc8b80 1 --2- 172.21.15.12:0/2903049064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x563d1f6d1be0 0x563d1f72a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f8641600700 0 client.10650 ms_handle_reset on v2:172.21.15.12:6833/99601797 2024-03-29T11:28:57.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2903049064 >> 172.21.15.12:0/2903049064 conn(0x563d1f6326a0 msgr2=0x563d1f6d2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2903049064 shutdown_connections 2024-03-29T11:28:57.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f8672bc8b80 1 -- 172.21.15.12:0/2903049064 wait complete. 2024-03-29T11:28:57.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f8672bc8b80 20 client.10650 trim_cache size 0 max 16384 2024-03-29T11:28:57.250 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:28:57.250 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T11:28:57.250 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T11:28:57.250 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 10512, 2024-03-29T11:28:57.250 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T11:28:57.250 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T11:28:57.250 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T11:28:57.251 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.157472415, 2024-03-29T11:28:57.251 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 98, 2024-03-29T11:28:57.251 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 208, 2024-03-29T11:28:57.251 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 208, 2024-03-29T11:28:57.251 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 19.177135058000001 2024-03-29T11:28:57.251 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f865f000700 1 -- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f863806adb0 msgr2=0x7f863806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f865f000700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f863806adb0 0x7f863806d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8648009840 tx=0x7f8648008030 comp rx=0 tx=0).stop 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f865f000700 1 -- 172.21.15.12:0/3418626350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 msgr2=0x7f86580ae160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.255+0000 7f865f000700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 0x7f86580ae160 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f864400d800 tx=0x7f864400db10 comp rx=0 tx=0).stop 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.256+0000 7f865f000700 1 -- 172.21.15.12:0/3418626350 shutdown_connections 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.256+0000 7f865f000700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f863806adb0 0x7f863806d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.256+0000 7f865f000700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f865815cfe0 0x7f86580ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.256+0000 7f865f000700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f865806f310 0x7f86581f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.256+0000 7f865f000700 1 --2- 172.21.15.12:0/3418626350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86580cd5c0 0x7f86580ae160 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.256+0000 7f865f000700 1 -- 172.21.15.12:0/3418626350 >> 172.21.15.12:0/3418626350 conn(0x7f86580d1370 msgr2=0x7f865806b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.256+0000 7f865f000700 1 -- 172.21.15.12:0/3418626350 shutdown_connections 2024-03-29T11:28:57.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.256+0000 7f865f000700 1 -- 172.21.15.12:0/3418626350 wait complete. 2024-03-29T11:28:57.264 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T11:28:57.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.471+0000 7f5d8ea00700 1 Processor -- start 2024-03-29T11:28:57.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.471+0000 7f5d8ea00700 1 -- start start 2024-03-29T11:28:57.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.472+0000 7f5d8ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d880ceec0 0x7f5d881612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.472+0000 7f5d8ea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d880d30d0 con 0x7f5d8815d640 2024-03-29T11:28:57.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.472+0000 7f5d8ea00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d880b6840 con 0x7f5d88161800 2024-03-29T11:28:57.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.472+0000 7f5d8ea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d88161c90 con 0x7f5d880ceec0 2024-03-29T11:28:57.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.472+0000 7f5d87e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5d8815d640 0x7f5d880d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54224/0 (socket says 172.21.15.12:54224) 2024-03-29T11:28:57.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.472+0000 7f5d87e00700 1 -- 172.21.15.12:0/3775587271 learned_addr learned my addr 172.21.15.12:0/3775587271 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:57.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d87400700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d880ceec0 0x7f5d881612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151295475 0 0) 0x7f5d880d30d0 con 0x7f5d8815d640 2024-03-29T11:28:57.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d6c0035b0 con 0x7f5d8815d640 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805918549 0 0) 0x7f5d880b6840 con 0x7f5d88161800 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d880d30d0 con 0x7f5d88161800 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3013694318 0 0) 0x7f5d6c0035b0 con 0x7f5d8815d640 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d880ceec0 msgr2=0x7f5d881612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d880ceec0 0x7f5d881612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 >> v1:172.21.15.159:6789/0 conn(0x7f5d88161800 legacy=0x7f5d880b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d880d2b30 con 0x7f5d8815d640 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f5d78001670 con 0x7f5d8815d640 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d87400700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d880ceec0 0x7f5d881612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 >> v1:172.21.15.12:6789/0 conn(0x7f5d8815d640 legacy=0x7f5d880d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c003bc0 0x7f5d6c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d880ceec0 0x7f5d6c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c004d90 0x7f5d6c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d88161c90 con 0x7f5d6c003bc0 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c00bb20 con 0x7f5d6c004d90 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.473+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c00bce0 con 0x7f5d880ceec0 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.474+0000 7f5d87e00700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c003bc0 0x7f5d6c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.474+0000 7f5d87400700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d880ceec0 0x7f5d6c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.474+0000 7f5d8ca00700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c004d90 0x7f5d6c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.474+0000 7f5d87e00700 1 -- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d880ceec0 msgr2=0x7f5d6c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.474+0000 7f5d87e00700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d880ceec0 0x7f5d6c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.474+0000 7f5d87e00700 1 -- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c004d90 msgr2=0x7f5d6c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.474+0000 7f5d87e00700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c004d90 0x7f5d6c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.474+0000 7f5d87e00700 1 -- 172.21.15.12:0/3775587271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d740094a0 con 0x7f5d6c003bc0 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.474+0000 7f5d87e00700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c003bc0 0x7f5d6c003fc0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800c110 tx=0x7f5d6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea476337406b9558 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.474+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7800cc50 con 0x7f5d6c003bc0 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.475+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7800cdd0 con 0x7f5d6c003bc0 2024-03-29T11:28:57.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.475+0000 7f5d86a00700 1 -- 172.21.15.12:0/3775587271 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78005ba0 con 0x7f5d6c003bc0 2024-03-29T11:28:57.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.475+0000 7f5d8ea00700 1 -- 172.21.15.12:0/3775587271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c003bc0 msgr2=0x7f5d6c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.475+0000 7f5d8ea00700 1 --2- 172.21.15.12:0/3775587271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d6c003bc0 0x7f5d6c003fc0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800c110 tx=0x7f5d6c005880 comp rx=0 tx=0).stop 2024-03-29T11:28:57.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.475+0000 7f5d8ca00700 1 -- 172.21.15.12:0/3775587271 reap_dead start 2024-03-29T11:28:57.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.475+0000 7f5d8ea00700 1 -- 172.21.15.12:0/3775587271 shutdown_connections 2024-03-29T11:28:57.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.475+0000 7f5d8ea00700 1 -- 172.21.15.12:0/3775587271 >> 172.21.15.12:0/3775587271 conn(0x7f5d8808aff0 msgr2=0x7f5d880d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.475+0000 7f5d8ea00700 1 -- 172.21.15.12:0/3775587271 shutdown_connections 2024-03-29T11:28:57.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.475+0000 7f5d8ea00700 1 -- 172.21.15.12:0/3775587271 wait complete. 2024-03-29T11:28:57.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.477+0000 7f5d8ea00700 1 Processor -- start 2024-03-29T11:28:57.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.477+0000 7f5d8ea00700 1 -- start start 2024-03-29T11:28:57.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.478+0000 7f5d8ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c003bc0 0x7f5d88158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.478+0000 7f5d8ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d8806cd20 0x7f5d880cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.478+0000 7f5d8ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d880cae10 0x7f5d880cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.478+0000 7f5d8ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880ce5d0 con 0x7f5d880cae10 2024-03-29T11:28:57.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.478+0000 7f5d8ea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d88159080 con 0x7f5d6c003bc0 2024-03-29T11:28:57.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.478+0000 7f5d8ea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880cb7e0 con 0x7f5d8806cd20 2024-03-29T11:28:57.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.478+0000 7f5d8ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d880cae10 0x7f5d880cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.478+0000 7f5d87e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c003bc0 0x7f5d88158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.479+0000 7f5d8ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d880cae10 0x7f5d880cb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54742/0 (socket says 172.21.15.12:54742) 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.479+0000 7f5d8ca00700 1 -- 172.21.15.12:0/2293002921 learned_addr learned my addr 172.21.15.12:0/2293002921 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.479+0000 7f5d87400700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d8806cd20 0x7f5d880cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.479+0000 7f5d8ca00700 1 -- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d8806cd20 msgr2=0x7f5d880cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.479+0000 7f5d8ca00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d8806cd20 0x7f5d880cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.479+0000 7f5d8ca00700 1 -- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c003bc0 msgr2=0x7f5d88158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.479+0000 7f5d8ca00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c003bc0 0x7f5d88158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.479+0000 7f5d8ca00700 1 -- 172.21.15.12:0/2293002921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d880c71f0 con 0x7f5d880cae10 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.479+0000 7f5d8ca00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d880cae10 0x7f5d880cb2a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f5d88161be0 tx=0x7f5d8000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.479+0000 7f5d7fe00700 1 -- 172.21.15.12:0/2293002921 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80003c10 con 0x7f5d880cae10 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.480+0000 7f5d7fe00700 1 -- 172.21.15.12:0/2293002921 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d80003d90 con 0x7f5d880cae10 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.480+0000 7f5d7fe00700 1 -- 172.21.15.12:0/2293002921 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80005310 con 0x7f5d880cae10 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.480+0000 7f5d8ea00700 1 -- 172.21.15.12:0/2293002921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d880c74d0 con 0x7f5d880cae10 2024-03-29T11:28:57.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.480+0000 7f5d8ea00700 1 -- 172.21.15.12:0/2293002921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d880c7a90 con 0x7f5d880cae10 2024-03-29T11:28:57.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.481+0000 7f5d8ea00700 1 -- 172.21.15.12:0/2293002921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d88067720 con 0x7f5d880cae10 2024-03-29T11:28:57.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.482+0000 7f5d7fe00700 1 -- 172.21.15.12:0/2293002921 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5d80005470 con 0x7f5d880cae10 2024-03-29T11:28:57.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.483+0000 7f5d7fe00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5d6806adb0 0x7f5d6806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:57.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.483+0000 7f5d7fe00700 1 -- 172.21.15.12:0/2293002921 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f5d80089480 con 0x7f5d880cae10 2024-03-29T11:28:57.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.483+0000 7f5d87e00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5d6806adb0 0x7f5d6806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:57.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.484+0000 7f5d87e00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5d6806adb0 0x7f5d6806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5d78015040 tx=0x7f5d78007000 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:57.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.487+0000 7f5d7fe00700 1 -- 172.21.15.12:0/2293002921 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5d80054500 con 0x7f5d880cae10 2024-03-29T11:28:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.750+0000 7f5d8ea00700 1 -- 172.21.15.12:0/2293002921 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5d881620f0 con 0x7f5d880cae10 2024-03-29T11:28:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.751+0000 7f5d7fe00700 1 -- 172.21.15.12:0/2293002921 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v98) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5d80057b20 con 0x7f5d880cae10 2024-03-29T11:28:57.742 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T11:28:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.753+0000 7f5d8ea00700 1 -- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5d6806adb0 msgr2=0x7f5d6806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.753+0000 7f5d8ea00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5d6806adb0 0x7f5d6806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5d78015040 tx=0x7f5d78007000 comp rx=0 tx=0).stop 2024-03-29T11:28:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.753+0000 7f5d8ea00700 1 -- 172.21.15.12:0/2293002921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d880cae10 msgr2=0x7f5d880cb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.753+0000 7f5d8ea00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d880cae10 0x7f5d880cb2a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f5d88161be0 tx=0x7f5d8000ee70 comp rx=0 tx=0).stop 2024-03-29T11:28:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.753+0000 7f5d8ea00700 1 -- 172.21.15.12:0/2293002921 shutdown_connections 2024-03-29T11:28:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.753+0000 7f5d8ea00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5d6806adb0 0x7f5d6806d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.753+0000 7f5d8ea00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d880cae10 0x7f5d880cb2a0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.753+0000 7f5d8ea00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d6c003bc0 0x7f5d88158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.753+0000 7f5d8ea00700 1 --2- 172.21.15.12:0/2293002921 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d8806cd20 0x7f5d880cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.753+0000 7f5d8ea00700 1 -- 172.21.15.12:0/2293002921 >> 172.21.15.12:0/2293002921 conn(0x7f5d8808aff0 msgr2=0x7f5d880b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.754+0000 7f5d8ea00700 1 -- 172.21.15.12:0/2293002921 shutdown_connections 2024-03-29T11:28:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:57.754+0000 7f5d8ea00700 1 -- 172.21.15.12:0/2293002921 wait complete. 2024-03-29T11:28:57.763 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T11:28:57.763 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T11:28:57.763 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T11:28:57.763 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T11:28:57.763 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T11:28:57.763 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T11:28:57.763 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T11:28:57.763 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T11:28:57.764 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:28:57.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:57.764 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T11:28:57.809 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T11:28:57.809 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T11:28:57.809 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T11:28:57.810 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 35780sec preferred_lft 35780sec 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T11:28:57.811 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:28:57.812 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:28:57.812 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T11:28:57.812 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T11:28:57.812 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:28:57.812 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:28:57.812 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T11:28:57.812 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T11:28:57.812 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:28:57.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:57.812 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T11:28:57.914 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T11:28:57.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:57.917 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T11:28:57.964 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T11:28:57.964 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T11:28:57.965 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T11:28:57.965 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T11:28:57.965 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T11:28:57.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:57.966 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T11:28:57.966 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T11:28:57.966 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T11:28:57.966 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T11:28:57.966 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T11:28:57.966 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T11:28:58.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:58.133 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T11:28:58.178 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T11:28:58.179 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T11:28:58.179 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:28:58.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:58.179 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T11:28:58.288 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:28:58.288 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T11:28:58.332 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:28:58.332 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:28:58.332 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:28:58.416 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T11:28:58.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:58.417 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:28:58.559 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:28:58.559 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:28:58.587 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:28:58.587 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:28:58.587 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T11:28:58.587 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T11:28:58.588 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T11:28:58.593 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T11:28:58.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:58.594 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:28:58.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:28:58.662 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T11:28:58.760 INFO:teuthology.orchestra.run.smithi012.stdout:client10327 2024-03-29T11:28:58.761 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:28:58.761 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:28:58.805 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T11:28:59.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.054+0000 7f98c6e00700 1 Processor -- start 2024-03-29T11:28:59.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.054+0000 7f98c6e00700 1 -- start start 2024-03-29T11:28:59.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.056+0000 7f98c6e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 0x7f98c00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.056+0000 7f98c6e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c0161420 con 0x7f98c00d16a0 2024-03-29T11:28:59.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.056+0000 7f98c6e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c0161700 con 0x7f98c00d2540 2024-03-29T11:28:59.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.056+0000 7f98c6e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c015edc0 con 0x7f98c015d9d0 2024-03-29T11:28:59.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.056+0000 7f98bf400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f98c00d16a0 0x7f98c00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33538/0 (socket says 172.21.15.12:33538) 2024-03-29T11:28:59.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.056+0000 7f98bf400700 1 -- 172.21.15.12:0/1227283792 learned_addr learned my addr 172.21.15.12:0/1227283792 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:28:59.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.056+0000 7f98bfe00700 1 --2- 172.21.15.12:0/1227283792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 0x7f98c00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:59.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.057+0000 7f98bfe00700 1 -- 172.21.15.12:0/1227283792 >> v1:172.21.15.159:6789/0 conn(0x7f98c00d2540 legacy=0x7f98c00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.057+0000 7f98bfe00700 1 -- 172.21.15.12:0/1227283792 >> v1:172.21.15.12:6789/0 conn(0x7f98c00d16a0 legacy=0x7f98c00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.057+0000 7f98bfe00700 1 -- 172.21.15.12:0/1227283792 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c01619e0 con 0x7f98c015d9d0 2024-03-29T11:28:59.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.057+0000 7f98bea00700 1 -- 172.21.15.12:0/1227283792 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542946759 0 0) 0x7f98c0161420 con 0x7f98c00d16a0 2024-03-29T11:28:59.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.057+0000 7f98bfe00700 1 --2- 172.21.15.12:0/1227283792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 0x7f98c00d3c30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f98b00098a0 tx=0x7f98b0009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e9f2fda4f8105b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:59.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.057+0000 7f98bea00700 1 -- 172.21.15.12:0/1227283792 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b0011830 con 0x7f98c015d9d0 2024-03-29T11:28:59.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.057+0000 7f98bea00700 1 -- 172.21.15.12:0/1227283792 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b0011990 con 0x7f98c015d9d0 2024-03-29T11:28:59.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.057+0000 7f98bea00700 1 -- 172.21.15.12:0/1227283792 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b0011d40 con 0x7f98c015d9d0 2024-03-29T11:28:59.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.058+0000 7f98c6e00700 1 -- 172.21.15.12:0/1227283792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 msgr2=0x7f98c00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.058+0000 7f98c6e00700 1 --2- 172.21.15.12:0/1227283792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 0x7f98c00d3c30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f98b00098a0 tx=0x7f98b0009bb0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.058+0000 7f98c6e00700 1 -- 172.21.15.12:0/1227283792 shutdown_connections 2024-03-29T11:28:59.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.058+0000 7f98c6e00700 1 --2- 172.21.15.12:0/1227283792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 0x7f98c00d3c30 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.058+0000 7f98c6e00700 1 -- 172.21.15.12:0/1227283792 >> 172.21.15.12:0/1227283792 conn(0x7f98c008aff0 msgr2=0x7f98c00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:59.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.058+0000 7f98c6e00700 1 -- 172.21.15.12:0/1227283792 shutdown_connections 2024-03-29T11:28:59.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.058+0000 7f98c6e00700 1 -- 172.21.15.12:0/1227283792 wait complete. 2024-03-29T11:28:59.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.060+0000 7f98c6e00700 1 Processor -- start 2024-03-29T11:28:59.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.060+0000 7f98c6e00700 1 -- start start 2024-03-29T11:28:59.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.061+0000 7f98c6e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98c00d16a0 0x7f98c00ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.061+0000 7f98c6e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98c00d2540 0x7f98c00a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.061+0000 7f98c6e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 0x7f98c01f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.061+0000 7f98c6e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c00cab30 con 0x7f98c00d16a0 2024-03-29T11:28:59.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.061+0000 7f98c6e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c00cad60 con 0x7f98c00d2540 2024-03-29T11:28:59.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.061+0000 7f98c6e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c00cb040 con 0x7f98c015d9d0 2024-03-29T11:28:59.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.061+0000 7f98bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98c00d16a0 0x7f98c00ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:59.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.061+0000 7f98bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98c00d16a0 0x7f98c00ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59612/0 (socket says 172.21.15.12:59612) 2024-03-29T11:28:59.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.061+0000 7f98bfe00700 1 -- 172.21.15.12:0/2841758297 learned_addr learned my addr 172.21.15.12:0/2841758297 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:59.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98c4e00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 0x7f98c01f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98bfe00700 1 -- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 msgr2=0x7f98c01f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98bfe00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 0x7f98c01f3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98bfe00700 1 -- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98c00d2540 msgr2=0x7f98c00a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98bfe00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98c00d2540 0x7f98c00a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98bfe00700 1 -- 172.21.15.12:0/2841758297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b00094a0 con 0x7f98c00d16a0 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98bfe00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98c00d16a0 0x7f98c00ca5f0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f98b0014040 tx=0x7f98b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98b7e00700 1 -- 172.21.15.12:0/2841758297 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b00036a0 con 0x7f98c00d16a0 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98b7e00700 1 -- 172.21.15.12:0/2841758297 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b001e070 con 0x7f98c00d16a0 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98b7e00700 1 -- 172.21.15.12:0/2841758297 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b0003a20 con 0x7f98c00d16a0 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.062+0000 7f98c6e00700 1 -- 172.21.15.12:0/2841758297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c00cb320 con 0x7f98c00d16a0 2024-03-29T11:28:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.063+0000 7f98c6e00700 1 -- 172.21.15.12:0/2841758297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c00cb8e0 con 0x7f98c00d16a0 2024-03-29T11:28:59.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.064+0000 7f98c6e00700 1 -- 172.21.15.12:0/2841758297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c00d1ca0 con 0x7f98c00d16a0 2024-03-29T11:28:59.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.064+0000 7f98b7e00700 1 -- 172.21.15.12:0/2841758297 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f98b0003b80 con 0x7f98c00d16a0 2024-03-29T11:28:59.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.065+0000 7f98b7e00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f98a006adb0 0x7f98a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.065+0000 7f98b7e00700 1 -- 172.21.15.12:0/2841758297 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f98b008a870 con 0x7f98c00d16a0 2024-03-29T11:28:59.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.066+0000 7f98bf400700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f98a006adb0 0x7f98a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:59.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.067+0000 7f98bf400700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f98a006adb0 0x7f98a006d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f98ac001360 tx=0x7f98ac001080 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:59.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.070+0000 7f98b7e00700 1 -- 172.21.15.12:0/2841758297 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f98b00558e0 con 0x7f98c00d16a0 2024-03-29T11:28:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.314+0000 7f98c6e00700 1 -- 172.21.15.12:0/2841758297 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f98c0161fa0 con 0x7f98a006adb0 2024-03-29T11:28:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.316+0000 7f98b7e00700 1 -- 172.21.15.12:0/2841758297 <== mgr.9958 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f98ac005830 con 0x7f98a006adb0 2024-03-29T11:28:59.307 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:28:59.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:28:59.307 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T11:28:59.307 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:28:59.307 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:28:59.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 -- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f98a006adb0 msgr2=0x7f98a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f98a006adb0 0x7f98a006d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f98ac001360 tx=0x7f98ac001080 comp rx=0 tx=0).stop 2024-03-29T11:28:59.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 -- 172.21.15.12:0/2841758297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98c00d16a0 msgr2=0x7f98c00ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98c00d16a0 0x7f98c00ca5f0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f98b0014040 tx=0x7f98b0004750 comp rx=0 tx=0).stop 2024-03-29T11:28:59.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 -- 172.21.15.12:0/2841758297 shutdown_connections 2024-03-29T11:28:59.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f98a006adb0 0x7f98a006d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98c015d9d0 0x7f98c01f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98c00d16a0 0x7f98c00ca5f0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 --2- 172.21.15.12:0/2841758297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98c00d2540 0x7f98c00a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 -- 172.21.15.12:0/2841758297 >> 172.21.15.12:0/2841758297 conn(0x7f98c008aff0 msgr2=0x7f98c00cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:59.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 -- 172.21.15.12:0/2841758297 shutdown_connections 2024-03-29T11:28:59.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.319+0000 7f98c6e00700 1 -- 172.21.15.12:0/2841758297 wait complete. 2024-03-29T11:28:59.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T11:28:59.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.538+0000 7f1d0ca00700 1 Processor -- start 2024-03-29T11:28:59.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.538+0000 7f1d0ca00700 1 -- start start 2024-03-29T11:28:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.539+0000 7f1d0ca00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1d0815d9d0 0x7f1d080d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.539+0000 7f1d0ca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1d080d2540 0x7f1d080ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.539+0000 7f1d0ca00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d08161420 con 0x7f1d080d16a0 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.539+0000 7f1d0ca00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0815edc0 con 0x7f1d080d2540 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.539+0000 7f1d0ca00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d08161700 con 0x7f1d0815d9d0 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.539+0000 7f1d06800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1d080d2540 0x7f1d080ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.540+0000 7f1d06800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1d080d2540 0x7f1d080ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59622/0 (socket says 172.21.15.12:59622) 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.540+0000 7f1d06800700 1 -- 172.21.15.12:0/3202140236 learned_addr learned my addr 172.21.15.12:0/3202140236 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.540+0000 7f1d05e00700 1 --2- 172.21.15.12:0/3202140236 >> v2:172.21.15.159:3300/0 conn(0x7f1d0815d9d0 0x7f1d080d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.540+0000 7f1d06800700 1 -- 172.21.15.12:0/3202140236 >> v2:172.21.15.159:3300/0 conn(0x7f1d0815d9d0 msgr2=0x7f1d080d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.540+0000 7f1d06800700 1 --2- 172.21.15.12:0/3202140236 >> v2:172.21.15.159:3300/0 conn(0x7f1d0815d9d0 0x7f1d080d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.540+0000 7f1d06800700 1 -- 172.21.15.12:0/3202140236 >> v1:172.21.15.159:6789/0 conn(0x7f1d080d16a0 legacy=0x7f1d080d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.540+0000 7f1d06800700 1 -- 172.21.15.12:0/3202140236 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d081619e0 con 0x7f1d080d2540 2024-03-29T11:28:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.540+0000 7f1d06800700 1 --2- 172.21.15.12:0/3202140236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 0x7f1d080ce360 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f1d080d0ff0 tx=0x7f1d0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf57480999326a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:59.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.540+0000 7f1d04a00700 1 -- 172.21.15.12:0/3202140236 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0000e070 con 0x7f1d080d2540 2024-03-29T11:28:59.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.540+0000 7f1d04a00700 1 -- 172.21.15.12:0/3202140236 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0000b9c0 con 0x7f1d080d2540 2024-03-29T11:28:59.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.541+0000 7f1d04a00700 1 -- 172.21.15.12:0/3202140236 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0000bd70 con 0x7f1d080d2540 2024-03-29T11:28:59.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.541+0000 7f1d0ca00700 1 -- 172.21.15.12:0/3202140236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 msgr2=0x7f1d080ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.541+0000 7f1d0ca00700 1 --2- 172.21.15.12:0/3202140236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 0x7f1d080ce360 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f1d080d0ff0 tx=0x7f1d0000b330 comp rx=0 tx=0).stop 2024-03-29T11:28:59.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.541+0000 7f1d0ca00700 1 -- 172.21.15.12:0/3202140236 shutdown_connections 2024-03-29T11:28:59.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.541+0000 7f1d0ca00700 1 --2- 172.21.15.12:0/3202140236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 0x7f1d080ce360 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.541+0000 7f1d0ca00700 1 --2- 172.21.15.12:0/3202140236 >> v2:172.21.15.159:3300/0 conn(0x7f1d0815d9d0 0x7f1d080d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.541+0000 7f1d0ca00700 1 -- 172.21.15.12:0/3202140236 >> 172.21.15.12:0/3202140236 conn(0x7f1d0808aff0 msgr2=0x7f1d080d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:59.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.541+0000 7f1d0ca00700 1 -- 172.21.15.12:0/3202140236 shutdown_connections 2024-03-29T11:28:59.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.541+0000 7f1d0ca00700 1 -- 172.21.15.12:0/3202140236 wait complete. 2024-03-29T11:28:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.543+0000 7f1d0ca00700 1 Processor -- start 2024-03-29T11:28:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.543+0000 7f1d0ca00700 1 -- start start 2024-03-29T11:28:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d0ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d080d16a0 0x7f1d081f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d0ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 0x7f1d081f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d0ca00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d0815d9d0 0x7f1d080cc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d0ca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d080cc740 con 0x7f1d080d2540 2024-03-29T11:28:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d0ca00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d080ccc30 con 0x7f1d0815d9d0 2024-03-29T11:28:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d0ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d080ccf10 con 0x7f1d080d16a0 2024-03-29T11:28:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d05400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 0x7f1d081f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d05400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 0x7f1d081f66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59624/0 (socket says 172.21.15.12:59624) 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d05400700 1 -- 172.21.15.12:0/1437087192 learned_addr learned my addr 172.21.15.12:0/1437087192 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d05e00700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d080d16a0 0x7f1d081f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.544+0000 7f1d06800700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d0815d9d0 0x7f1d080cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1d05400700 1 -- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d080d16a0 msgr2=0x7f1d081f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1d05400700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d080d16a0 0x7f1d081f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1d05400700 1 -- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d0815d9d0 msgr2=0x7f1d080cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1d05400700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d0815d9d0 0x7f1d080cc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1d05400700 1 -- 172.21.15.12:0/1437087192 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d00009b30 con 0x7f1d080d2540 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1d05400700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 0x7f1d081f66e0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400c500 tx=0x7f1cf400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1cfe000700 1 -- 172.21.15.12:0/1437087192 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf40052d0 con 0x7f1d080d2540 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1cfe000700 1 -- 172.21.15.12:0/1437087192 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf4005450 con 0x7f1d080d2540 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1cfe000700 1 -- 172.21.15.12:0/1437087192 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4003910 con 0x7f1d080d2540 2024-03-29T11:28:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1d0ca00700 1 -- 172.21.15.12:0/1437087192 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d080b6ad0 con 0x7f1d080d2540 2024-03-29T11:28:59.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.545+0000 7f1d0ca00700 1 -- 172.21.15.12:0/1437087192 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d080cd1f0 con 0x7f1d080d2540 2024-03-29T11:28:59.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.547+0000 7f1d0ca00700 1 -- 172.21.15.12:0/1437087192 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d080d1ca0 con 0x7f1d080d2540 2024-03-29T11:28:59.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.548+0000 7f1cfe000700 1 -- 172.21.15.12:0/1437087192 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 117) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1cf4003a70 con 0x7f1d080d2540 2024-03-29T11:28:59.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.549+0000 7f1cfe000700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ce806adb0 0x7f1ce806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:28:59.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.549+0000 7f1cfe000700 1 -- 172.21.15.12:0/1437087192 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f1cf40888b0 con 0x7f1d080d2540 2024-03-29T11:28:59.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.550+0000 7f1d05e00700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ce806adb0 0x7f1ce806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:28:59.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.550+0000 7f1d05e00700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ce806adb0 0x7f1ce806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8006b20 tx=0x7f1cf8008040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:28:59.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.554+0000 7f1cfe000700 1 -- 172.21.15.12:0/1437087192 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1cf40538a0 con 0x7f1d080d2540 2024-03-29T11:28:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:28:59.828+0000 7f1d0ca00700 1 -- 172.21.15.12:0/1437087192 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f1d08067720 con 0x7f1d080d2540 2024-03-29T11:29:00.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.621+0000 7f1cfe000700 1 -- 172.21.15.12:0/1437087192 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 118) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f1cf4086d80 con 0x7f1d080d2540 2024-03-29T11:29:00.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.623+0000 7f1cfe000700 1 -- 172.21.15.12:0/1437087192 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v118) v1 ==== 92+0+0 (secure 0 0 0) 0x7f1cf4073c50 con 0x7f1d080d2540 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 -- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ce806adb0 msgr2=0x7f1ce806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ce806adb0 0x7f1ce806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8006b20 tx=0x7f1cf8008040 comp rx=0 tx=0).stop 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 -- 172.21.15.12:0/1437087192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 msgr2=0x7f1d081f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 0x7f1d081f66e0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400c500 tx=0x7f1cf400c810 comp rx=0 tx=0).stop 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 -- 172.21.15.12:0/1437087192 shutdown_connections 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ce806adb0 0x7f1ce806d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d0815d9d0 0x7f1d080cc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d080d16a0 0x7f1d081f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 --2- 172.21.15.12:0/1437087192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d080d2540 0x7f1d081f66e0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 -- 172.21.15.12:0/1437087192 >> 172.21.15.12:0/1437087192 conn(0x7f1d0808aff0 msgr2=0x7f1d080b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 -- 172.21.15.12:0/1437087192 shutdown_connections 2024-03-29T11:29:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.626+0000 7f1d0ca00700 1 -- 172.21.15.12:0/1437087192 wait complete. 2024-03-29T11:29:00.637 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T11:29:00.839 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:00.846+0000 7f9efada2240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:29:00.840 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:00.850+0000 7fe4a3ed8240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:29:00.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.855+0000 7f3cdbc00700 1 Processor -- start 2024-03-29T11:29:00.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.856+0000 7f3cdbc00700 1 -- start start 2024-03-29T11:29:00.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.857+0000 7f3cdbc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3cd406aea0 0x7f3cd40b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:00.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.857+0000 7f3cdbc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd4161150 con 0x7f3cd4160d70 2024-03-29T11:29:00.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.857+0000 7f3cdbc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd406dba0 con 0x7f3cd415ef70 2024-03-29T11:29:00.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.857+0000 7f3cdbc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd40b4cd0 con 0x7f3cd406aea0 2024-03-29T11:29:00.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.857+0000 7f3cd9200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3cd4160d70 0x7f3cd415e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33542/0 (socket says 172.21.15.12:33542) 2024-03-29T11:29:00.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.857+0000 7f3cd9200700 1 -- 172.21.15.12:0/349916776 learned_addr learned my addr 172.21.15.12:0/349916776 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:29:00.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.857+0000 7f3cd9c00700 1 --2- 172.21.15.12:0/349916776 >> v2:172.21.15.159:3300/0 conn(0x7f3cd406aea0 0x7f3cd40b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:00.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.858+0000 7f3cd3400700 1 -- 172.21.15.12:0/349916776 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249871486 0 0) 0x7f3cd4161150 con 0x7f3cd4160d70 2024-03-29T11:29:00.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.858+0000 7f3cd3400700 1 -- 172.21.15.12:0/349916776 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc0035b0 con 0x7f3cd4160d70 2024-03-29T11:29:00.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.858+0000 7f3cd3400700 1 -- 172.21.15.12:0/349916776 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494623271 0 0) 0x7f3cd406dba0 con 0x7f3cd415ef70 2024-03-29T11:29:00.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.858+0000 7f3cd3400700 1 -- 172.21.15.12:0/349916776 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd4161150 con 0x7f3cd415ef70 2024-03-29T11:29:00.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.858+0000 7f3cd9c00700 1 -- 172.21.15.12:0/349916776 >> v1:172.21.15.159:6789/0 conn(0x7f3cd415ef70 legacy=0x7f3cd406a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.858+0000 7f3cd9c00700 1 -- 172.21.15.12:0/349916776 >> v1:172.21.15.12:6789/0 conn(0x7f3cd4160d70 legacy=0x7f3cd415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.858+0000 7f3cd9c00700 1 -- 172.21.15.12:0/349916776 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd40b4ee0 con 0x7f3cd406aea0 2024-03-29T11:29:00.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.858+0000 7f3cd9c00700 1 --2- 172.21.15.12:0/349916776 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cd406aea0 0x7f3cd40b3200 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009d90 tx=0x7f3ccc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e928a413b1b9296f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:00.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.859+0000 7f3cd3400700 1 -- 172.21.15.12:0/349916776 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc00e070 con 0x7f3cd406aea0 2024-03-29T11:29:00.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.859+0000 7f3cd3400700 1 -- 172.21.15.12:0/349916776 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc00bb20 con 0x7f3cd406aea0 2024-03-29T11:29:00.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.859+0000 7f3cd3400700 1 -- 172.21.15.12:0/349916776 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc012430 con 0x7f3cd406aea0 2024-03-29T11:29:00.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.859+0000 7f3cdbc00700 1 -- 172.21.15.12:0/349916776 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cd406aea0 msgr2=0x7f3cd40b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.859+0000 7f3cdbc00700 1 --2- 172.21.15.12:0/349916776 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cd406aea0 0x7f3cd40b3200 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009d90 tx=0x7f3ccc00b330 comp rx=0 tx=0).stop 2024-03-29T11:29:00.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.859+0000 7f3cdbc00700 1 -- 172.21.15.12:0/349916776 shutdown_connections 2024-03-29T11:29:00.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.859+0000 7f3cdbc00700 1 --2- 172.21.15.12:0/349916776 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cd406aea0 0x7f3cd40b3200 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:00.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.859+0000 7f3cdbc00700 1 -- 172.21.15.12:0/349916776 >> 172.21.15.12:0/349916776 conn(0x7f3cd40b5720 msgr2=0x7f3cd40b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:00.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.859+0000 7f3cdbc00700 1 -- 172.21.15.12:0/349916776 shutdown_connections 2024-03-29T11:29:00.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.859+0000 7f3cdbc00700 1 -- 172.21.15.12:0/349916776 wait complete. 2024-03-29T11:29:00.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.861+0000 7f3cdbc00700 1 Processor -- start 2024-03-29T11:29:00.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.861+0000 7f3cdbc00700 1 -- start start 2024-03-29T11:29:00.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.862+0000 7f3cdbc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd406aea0 0x7f3cd41550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:00.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.862+0000 7f3cdbc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cd415ef70 0x7f3cd415c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:00.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.862+0000 7f3cdbc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cd4160d70 0x7f3cd415f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:00.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.862+0000 7f3cdbc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd409b640 con 0x7f3cd406aea0 2024-03-29T11:29:00.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.862+0000 7f3cdbc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd409b360 con 0x7f3cd4160d70 2024-03-29T11:29:00.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.862+0000 7f3cdbc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd415bb80 con 0x7f3cd415ef70 2024-03-29T11:29:00.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.862+0000 7f3cd9200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd406aea0 0x7f3cd41550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:00.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd9200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd406aea0 0x7f3cd41550b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59642/0 (socket says 172.21.15.12:59642) 2024-03-29T11:29:00.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd9200700 1 -- 172.21.15.12:0/739426242 learned_addr learned my addr 172.21.15.12:0/739426242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:29:00.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd3e00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cd415ef70 0x7f3cd415c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:00.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd9c00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cd4160d70 0x7f3cd415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:00.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd9200700 1 -- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cd415ef70 msgr2=0x7f3cd415c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd9200700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cd415ef70 0x7f3cd415c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:00.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd9200700 1 -- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cd4160d70 msgr2=0x7f3cd415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd9200700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cd4160d70 0x7f3cd415f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:00.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd9200700 1 -- 172.21.15.12:0/739426242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc009a70 con 0x7f3cd406aea0 2024-03-29T11:29:00.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd9200700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd406aea0 0x7f3cd41550b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8001eb0 tx=0x7f3cc800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:00.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.863+0000 7f3cd0c00700 1 -- 172.21.15.12:0/739426242 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8005d70 con 0x7f3cd406aea0 2024-03-29T11:29:00.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.864+0000 7f3cd0c00700 1 -- 172.21.15.12:0/739426242 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc80070d0 con 0x7f3cd406aea0 2024-03-29T11:29:00.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.864+0000 7f3cdbc00700 1 -- 172.21.15.12:0/739426242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd415be60 con 0x7f3cd406aea0 2024-03-29T11:29:00.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.864+0000 7f3cd0c00700 1 -- 172.21.15.12:0/739426242 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8007440 con 0x7f3cd406aea0 2024-03-29T11:29:00.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.864+0000 7f3cdbc00700 1 -- 172.21.15.12:0/739426242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd415cba0 con 0x7f3cd406aea0 2024-03-29T11:29:00.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.865+0000 7f3cdbc00700 1 -- 172.21.15.12:0/739426242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd4067720 con 0x7f3cd406aea0 2024-03-29T11:29:00.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.866+0000 7f3cd0c00700 1 -- 172.21.15.12:0/739426242 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 118) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f3cc80075a0 con 0x7f3cd406aea0 2024-03-29T11:29:00.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.867+0000 7f3cd0c00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3cb406ae00 0x7f3cb406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:00.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.867+0000 7f3cd0c00700 1 -- 172.21.15.12:0/739426242 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f3cc8088800 con 0x7f3cd406aea0 2024-03-29T11:29:00.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.867+0000 7f3cd3e00700 1 -- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3cb406ae00 msgr2=0x7f3cb406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:29:00.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.867+0000 7f3cd3e00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3cb406ae00 0x7f3cb406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:29:00.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:00.871+0000 7f3cd0c00700 1 -- 172.21.15.12:0/739426242 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3cc8053920 con 0x7f3cd406aea0 2024-03-29T11:29:01.024 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:01.033+0000 7f9efada2240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:29:01.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.068+0000 7f3cd3e00700 1 -- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3cb406ae00 msgr2=0x7f3cb406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:29:01.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.068+0000 7f3cd3e00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3cb406ae00 0x7f3cb406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:29:01.068 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:01.078+0000 7fe4a3ed8240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:29:01.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.149+0000 7f3cdbc00700 1 -- 172.21.15.12:0/739426242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3cd40d0dc0 con 0x7f3cd406aea0 2024-03-29T11:29:01.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.153+0000 7f3cd0c00700 1 -- 172.21.15.12:0/739426242 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3cc8056f40 con 0x7f3cd406aea0 2024-03-29T11:29:01.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 -- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3cb406ae00 msgr2=0x7f3cb406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3cb406ae00 0x7f3cb406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 -- 172.21.15.12:0/739426242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd406aea0 msgr2=0x7f3cd41550b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd406aea0 0x7f3cd41550b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8001eb0 tx=0x7f3cc800c7b0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 -- 172.21.15.12:0/739426242 shutdown_connections 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3cb406ae00 0x7f3cb406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cd4160d70 0x7f3cd415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd406aea0 0x7f3cd41550b0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 --2- 172.21.15.12:0/739426242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cd415ef70 0x7f3cd415c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 -- 172.21.15.12:0/739426242 >> 172.21.15.12:0/739426242 conn(0x7f3cd40b5720 msgr2=0x7f3cd409a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 -- 172.21.15.12:0/739426242 shutdown_connections 2024-03-29T11:29:01.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.156+0000 7f3cdbc00700 1 -- 172.21.15.12:0/739426242 wait complete. 2024-03-29T11:29:01.167 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T11:29:01.342 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:01.351+0000 7f9efada2240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:29:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.379+0000 7fbc66c00700 1 Processor -- start 2024-03-29T11:29:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.380+0000 7fbc66c00700 1 -- start start 2024-03-29T11:29:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.381+0000 7fbc66c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc600cd480 0x7fbc6006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.381+0000 7fbc66c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600cdae0 con 0x7fbc600cbb30 2024-03-29T11:29:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.381+0000 7fbc66c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600cddc0 con 0x7fbc600cd020 2024-03-29T11:29:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.381+0000 7fbc66c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc600ce0a0 con 0x7fbc600cd480 2024-03-29T11:29:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.381+0000 7fbc5fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbc600cbb30 0x7fbc600cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33556/0 (socket says 172.21.15.12:33556) 2024-03-29T11:29:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.381+0000 7fbc5fe00700 1 -- 172.21.15.12:0/2140430681 learned_addr learned my addr 172.21.15.12:0/2140430681 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:29:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.381+0000 7fbc64c00700 1 --2- 172.21.15.12:0/2140430681 >> v2:172.21.15.12:3300/0 conn(0x7fbc600cd480 0x7fbc6006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.382+0000 7fbc5ea00700 1 -- 172.21.15.12:0/2140430681 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608809351 0 0) 0x7fbc600cdae0 con 0x7fbc600cbb30 2024-03-29T11:29:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.382+0000 7fbc5ea00700 1 -- 172.21.15.12:0/2140430681 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc440035b0 con 0x7fbc600cbb30 2024-03-29T11:29:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.382+0000 7fbc64c00700 1 -- 172.21.15.12:0/2140430681 >> v1:172.21.15.159:6789/0 conn(0x7fbc600cd020 legacy=0x7fbc600b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.382+0000 7fbc5ea00700 1 -- 172.21.15.12:0/2140430681 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821811392 0 0) 0x7fbc600cddc0 con 0x7fbc600cd020 2024-03-29T11:29:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.382+0000 7fbc64c00700 1 -- 172.21.15.12:0/2140430681 >> v1:172.21.15.12:6789/0 conn(0x7fbc600cbb30 legacy=0x7fbc600cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.382+0000 7fbc64c00700 1 -- 172.21.15.12:0/2140430681 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6006dc20 con 0x7fbc600cd480 2024-03-29T11:29:01.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.382+0000 7fbc64c00700 1 --2- 172.21.15.12:0/2140430681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 0x7fbc6006e420 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009d90 tx=0x7fbc5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea6e560a3a006ab server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:01.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.382+0000 7fbc5ea00700 1 -- 172.21.15.12:0/2140430681 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5800e070 con 0x7fbc600cd480 2024-03-29T11:29:01.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.382+0000 7fbc5ea00700 1 -- 172.21.15.12:0/2140430681 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5800bb20 con 0x7fbc600cd480 2024-03-29T11:29:01.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.382+0000 7fbc5ea00700 1 -- 172.21.15.12:0/2140430681 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5800bed0 con 0x7fbc600cd480 2024-03-29T11:29:01.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.383+0000 7fbc66c00700 1 -- 172.21.15.12:0/2140430681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 msgr2=0x7fbc6006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.383+0000 7fbc66c00700 1 --2- 172.21.15.12:0/2140430681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 0x7fbc6006e420 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009d90 tx=0x7fbc5800b330 comp rx=0 tx=0).stop 2024-03-29T11:29:01.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.383+0000 7fbc66c00700 1 -- 172.21.15.12:0/2140430681 shutdown_connections 2024-03-29T11:29:01.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.383+0000 7fbc66c00700 1 --2- 172.21.15.12:0/2140430681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 0x7fbc6006e420 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.383+0000 7fbc66c00700 1 -- 172.21.15.12:0/2140430681 >> 172.21.15.12:0/2140430681 conn(0x7fbc600b0d70 msgr2=0x7fbc600b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:01.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.383+0000 7fbc66c00700 1 -- 172.21.15.12:0/2140430681 shutdown_connections 2024-03-29T11:29:01.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.383+0000 7fbc66c00700 1 -- 172.21.15.12:0/2140430681 wait complete. 2024-03-29T11:29:01.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.385+0000 7fbc66c00700 1 Processor -- start 2024-03-29T11:29:01.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.386+0000 7fbc66c00700 1 -- start start 2024-03-29T11:29:01.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc66c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc600cbb30 0x7fbc60157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc66c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600cd020 0x7fbc6014a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc66c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 0x7fbc6014a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc66c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60156d10 con 0x7fbc600cd480 2024-03-29T11:29:01.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc66c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600c8660 con 0x7fbc600cbb30 2024-03-29T11:29:01.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc66c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60156100 con 0x7fbc600cd020 2024-03-29T11:29:01.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc64c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 0x7fbc6014a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:01.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc5fe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc600cbb30 0x7fbc60157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:01.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc64c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 0x7fbc6014a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59668/0 (socket says 172.21.15.12:59668) 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc64c00700 1 -- 172.21.15.12:0/1711729508 learned_addr learned my addr 172.21.15.12:0/1711729508 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.387+0000 7fbc5f400700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600cd020 0x7fbc6014a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.388+0000 7fbc64c00700 1 -- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600cd020 msgr2=0x7fbc6014a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.388+0000 7fbc64c00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600cd020 0x7fbc6014a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.388+0000 7fbc64c00700 1 -- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc600cbb30 msgr2=0x7fbc60157ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.388+0000 7fbc64c00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc600cbb30 0x7fbc60157ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.388+0000 7fbc64c00700 1 -- 172.21.15.12:0/1711729508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc58009a70 con 0x7fbc600cd480 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.388+0000 7fbc64c00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 0x7fbc6014a6a0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fbc6006c130 tx=0x7fbc58004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.388+0000 7fbc57e00700 1 -- 172.21.15.12:0/1711729508 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5801c810 con 0x7fbc600cd480 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.388+0000 7fbc57e00700 1 -- 172.21.15.12:0/1711729508 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc58020070 con 0x7fbc600cd480 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.388+0000 7fbc57e00700 1 -- 172.21.15.12:0/1711729508 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5801cbb0 con 0x7fbc600cd480 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.388+0000 7fbc66c00700 1 -- 172.21.15.12:0/1711729508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc601563e0 con 0x7fbc600cd480 2024-03-29T11:29:01.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.389+0000 7fbc66c00700 1 -- 172.21.15.12:0/1711729508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc601569a0 con 0x7fbc600cd480 2024-03-29T11:29:01.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.390+0000 7fbc66c00700 1 -- 172.21.15.12:0/1711729508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc60156ec0 con 0x7fbc600cd480 2024-03-29T11:29:01.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.391+0000 7fbc57e00700 1 -- 172.21.15.12:0/1711729508 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 118) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fbc5801cd10 con 0x7fbc600cd480 2024-03-29T11:29:01.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.392+0000 7fbc57e00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006ae00 0x7fbc4006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.392+0000 7fbc57e00700 1 -- 172.21.15.12:0/1711729508 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fbc58022070 con 0x7fbc600cd480 2024-03-29T11:29:01.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.392+0000 7fbc5fe00700 1 -- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006ae00 msgr2=0x7fbc4006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:29:01.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.392+0000 7fbc5fe00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006ae00 0x7fbc4006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:29:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.397+0000 7fbc57e00700 1 -- 172.21.15.12:0/1711729508 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc58056530 con 0x7fbc600cd480 2024-03-29T11:29:01.414 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:01.423+0000 7fe4a3ed8240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:29:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.593+0000 7fbc5fe00700 1 -- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006ae00 msgr2=0x7fbc4006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:29:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.593+0000 7fbc5fe00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006ae00 0x7fbc4006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:29:01.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.610+0000 7fbc66c00700 1 -- 172.21.15.12:0/1711729508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fbc601570a0 con 0x7fbc600cd480 2024-03-29T11:29:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.616+0000 7fbc57e00700 1 -- 172.21.15.12:0/1711729508 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7fbc58059b50 con 0x7fbc600cd480 2024-03-29T11:29:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 -- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006ae00 msgr2=0x7fbc4006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006ae00 0x7fbc4006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 -- 172.21.15.12:0/1711729508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 msgr2=0x7fbc6014a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 0x7fbc6014a6a0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fbc6006c130 tx=0x7fbc58004ad0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 -- 172.21.15.12:0/1711729508 shutdown_connections 2024-03-29T11:29:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006ae00 0x7fbc4006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc600cd480 0x7fbc6014a6a0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc600cbb30 0x7fbc60157ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 --2- 172.21.15.12:0/1711729508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600cd020 0x7fbc6014a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 -- 172.21.15.12:0/1711729508 >> 172.21.15.12:0/1711729508 conn(0x7fbc600b0d70 msgr2=0x7fbc600b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:01.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 -- 172.21.15.12:0/1711729508 shutdown_connections 2024-03-29T11:29:01.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.619+0000 7fbc66c00700 1 -- 172.21.15.12:0/1711729508 wait complete. 2024-03-29T11:29:01.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:01.630 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-29T11:29:01.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-03-29T11:29:01.871 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:01.880+0000 7f9efada2240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:29:01.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.896+0000 7f306a600700 1 Processor -- start 2024-03-29T11:29:01.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.896+0000 7f306a600700 1 -- start start 2024-03-29T11:29:01.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.897+0000 7f306a600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 0x7f30640cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.897+0000 7f306a600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f30640b2070 0x7f30640b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f306a600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30640cdae0 con 0x7f30640cbb30 2024-03-29T11:29:01.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f306a600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30640cddc0 con 0x7f30640b2070 2024-03-29T11:29:01.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f306a600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30640ce0a0 con 0x7f30640cd020 2024-03-29T11:29:01.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f3063400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f30640cbb30 0x7f30640cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33570/0 (socket says 172.21.15.12:33570) 2024-03-29T11:29:01.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f3063400700 1 -- 172.21.15.12:0/102097382 learned_addr learned my addr 172.21.15.12:0/102097382 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:29:01.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f3063e00700 1 --2- 172.21.15.12:0/102097382 >> v2:172.21.15.159:3300/0 conn(0x7f30640b2070 0x7f30640b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:01.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f3062a00700 1 --2- 172.21.15.12:0/102097382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 0x7f30640cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:01.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f3062000700 1 -- 172.21.15.12:0/102097382 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746086244 0 0) 0x7f30640cdae0 con 0x7f30640cbb30 2024-03-29T11:29:01.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f3062000700 1 -- 172.21.15.12:0/102097382 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30480035b0 con 0x7f30640cbb30 2024-03-29T11:29:01.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f3062a00700 1 -- 172.21.15.12:0/102097382 >> v2:172.21.15.159:3300/0 conn(0x7f30640b2070 msgr2=0x7f30640b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:01.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f3062a00700 1 --2- 172.21.15.12:0/102097382 >> v2:172.21.15.159:3300/0 conn(0x7f30640b2070 0x7f30640b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.898+0000 7f3062a00700 1 -- 172.21.15.12:0/102097382 >> v1:172.21.15.12:6789/0 conn(0x7f30640cbb30 legacy=0x7f30640cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.899+0000 7f3062a00700 1 -- 172.21.15.12:0/102097382 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306406dc20 con 0x7f30640cd020 2024-03-29T11:29:01.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.899+0000 7f3063e00700 1 --2- 172.21.15.12:0/102097382 >> v2:172.21.15.159:3300/0 conn(0x7f30640b2070 0x7f30640b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:29:01.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.899+0000 7f3062a00700 1 --2- 172.21.15.12:0/102097382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 0x7f30640cd480 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3050009840 tx=0x7f3050009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60a0fd43d0246b5f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:01.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.899+0000 7f3062000700 1 -- 172.21.15.12:0/102097382 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050011830 con 0x7f30640cd020 2024-03-29T11:29:01.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.899+0000 7f3062000700 1 -- 172.21.15.12:0/102097382 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3050011990 con 0x7f30640cd020 2024-03-29T11:29:01.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.899+0000 7f3062000700 1 -- 172.21.15.12:0/102097382 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050011d20 con 0x7f30640cd020 2024-03-29T11:29:01.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.900+0000 7f306a600700 1 -- 172.21.15.12:0/102097382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 msgr2=0x7f30640cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.900+0000 7f306a600700 1 --2- 172.21.15.12:0/102097382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 0x7f30640cd480 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3050009840 tx=0x7f3050009b50 comp rx=0 tx=0).stop 2024-03-29T11:29:01.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.900+0000 7f306a600700 1 -- 172.21.15.12:0/102097382 shutdown_connections 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.900+0000 7f306a600700 1 --2- 172.21.15.12:0/102097382 >> v2:172.21.15.159:3300/0 conn(0x7f30640b2070 0x7f30640b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.900+0000 7f306a600700 1 --2- 172.21.15.12:0/102097382 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 0x7f30640cd480 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.900+0000 7f306a600700 1 -- 172.21.15.12:0/102097382 >> 172.21.15.12:0/102097382 conn(0x7f30640b0d70 msgr2=0x7f30640b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.900+0000 7f306a600700 1 -- 172.21.15.12:0/102097382 shutdown_connections 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.900+0000 7f306a600700 1 -- 172.21.15.12:0/102097382 wait complete. 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.902+0000 7f306a600700 1 Processor -- start 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.902+0000 7f306a600700 1 -- start start 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f306a600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30640b2070 0x7f306415d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f306a600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30640cbb30 0x7f30640721f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f306a600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 0x7f30640add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f306a600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30640709c0 con 0x7f30640cbb30 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f306a600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306414ade0 con 0x7f30640b2070 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f306a600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3064070c10 con 0x7f30640cd020 2024-03-29T11:29:01.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f3062a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30640cbb30 0x7f30640721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f3062a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30640cbb30 0x7f30640721f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59682/0 (socket says 172.21.15.12:59682) 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f3063400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30640b2070 0x7f306415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f3062a00700 1 -- 172.21.15.12:0/1525075765 learned_addr learned my addr 172.21.15.12:0/1525075765 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.903+0000 7f3063e00700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 0x7f30640add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f3062a00700 1 -- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 msgr2=0x7f30640add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f3062a00700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 0x7f30640add70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f3062a00700 1 -- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30640b2070 msgr2=0x7f306415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f3062a00700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30640b2070 0x7f306415d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f3062a00700 1 -- 172.21.15.12:0/1525075765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c009a70 con 0x7f30640cbb30 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f3063400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30640b2070 0x7f306415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f3063e00700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 0x7f30640add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f3062a00700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30640cbb30 0x7f30640721f0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f30500090b0 tx=0x7f3050004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:01.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050004550 con 0x7f30640cbb30 2024-03-29T11:29:01.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305001e070 con 0x7f30640cbb30 2024-03-29T11:29:01.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f306a600700 1 -- 172.21.15.12:0/1525075765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30500094a0 con 0x7f30640cbb30 2024-03-29T11:29:01.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.904+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30500038e0 con 0x7f30640cbb30 2024-03-29T11:29:01.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.905+0000 7f306a600700 1 -- 172.21.15.12:0/1525075765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30640711d0 con 0x7f30640cbb30 2024-03-29T11:29:01.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.906+0000 7f306a600700 1 -- 172.21.15.12:0/1525075765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3064067720 con 0x7f30640cbb30 2024-03-29T11:29:01.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.908+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 118) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f3050003a40 con 0x7f30640cbb30 2024-03-29T11:29:01.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.909+0000 7f305b400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:01.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.909+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f305008a2d0 con 0x7f30640cbb30 2024-03-29T11:29:01.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.909+0000 7f3063400700 1 -- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 msgr2=0x7f304406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:29:01.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.909+0000 7f3063400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:29:01.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:01.913+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f30500553e0 con 0x7f30640cbb30 2024-03-29T11:29:01.982 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:01.991+0000 7fe4a3ed8240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:29:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:02.110+0000 7f3063400700 1 -- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 msgr2=0x7f304406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:29:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:02.110+0000 7f3063400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:29:02.128 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:02.137+0000 7f9efada2240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:29:02.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:02.157+0000 7f306a600700 1 -- 172.21.15.12:0/1525075765 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f306415bdd0 con 0x7f304406ae00 2024-03-29T11:29:02.248 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:02.257+0000 7fe4a3ed8240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:29:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:02.511+0000 7f3063400700 1 -- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 msgr2=0x7f304406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:29:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:02.511+0000 7f3063400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T11:29:02.582 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:02.591+0000 7f9efada2240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:29:02.725 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:02.735+0000 7fe4a3ed8240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:29:03.287 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:03.296+0000 7f9efada2240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:29:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:03.312+0000 7f3063400700 1 -- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 msgr2=0x7f304406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:29:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:03.312+0000 7f3063400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T11:29:03.379 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:03.388+0000 7f9efada2240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:29:03.469 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:03.479+0000 7fe4a3ed8240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:29:03.537 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:03.546+0000 7f9efada2240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:29:03.566 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:03.575+0000 7fe4a3ed8240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:29:03.734 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:03.744+0000 7fe4a3ed8240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:29:03.752 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:03.761+0000 7f9efada2240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:29:03.844 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:03.853+0000 7f9efada2240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:29:03.943 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:03.952+0000 7f9efada2240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:29:03.953 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:03.962+0000 7fe4a3ed8240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:29:04.042 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:04.051+0000 7f9efada2240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:29:04.048 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:04.058+0000 7fe4a3ed8240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:29:04.141 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:04.150+0000 7f9efada2240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:29:04.151 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:04.161+0000 7fe4a3ed8240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:29:04.255 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:04.264+0000 7fe4a3ed8240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:29:04.354 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:04.364+0000 7f9efada2240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:29:04.358 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:04.367+0000 7fe4a3ed8240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:29:04.581 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:04.590+0000 7fe4a3ed8240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:29:04.612 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:04.621+0000 7f9efada2240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:29:04.802 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:04.811+0000 7f9efada2240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:29:04.844 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:04.853+0000 7fe4a3ed8240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:29:04.894 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:04.903+0000 7f9efada2240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:29:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:04.914+0000 7f3063400700 1 -- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 msgr2=0x7f304406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:29:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:04.914+0000 7f3063400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T11:29:05.044 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:05.053+0000 7fe4a3ed8240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:29:05.140 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:05.150+0000 7fe4a3ed8240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:29:05.626 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:05.635+0000 7f9efada2240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:29:05.837 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:05.846+0000 7f9efada2240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:29:05.906 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:05.915+0000 7fe4a3ed8240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:29:06.044 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:06.053+0000 7f9efada2240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:29:06.132 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:06.141+0000 7fe4a3ed8240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:29:06.346 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:06.356+0000 7fe4a3ed8240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:29:06.518 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:06.527+0000 7f9efada2240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:29:06.723 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:06.733+0000 7f9efada2240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:29:06.847 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:06.857+0000 7fe4a3ed8240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:29:07.065 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:07.075+0000 7fe4a3ed8240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:29:07.164 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.173+0000 7f9efada2240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:29:07.275 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.284+0000 7f9efada2240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:29:07.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:07.295+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 119) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f3050003cf0 con 0x7f30640cbb30 2024-03-29T11:29:07.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:07.295+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 msgr2=0x7f304406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:07.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:07.295+0000 7f305b400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:07.467 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.475+0000 7f9e62a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:29:07.467 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.475+0000 7f9e62a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:29:07.467 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.475+0000 7f9e62a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:29:07.467 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.475+0000 7f9e62a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:29:07.467 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.476+0000 7f9e67000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:29:07.467 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.475+0000 7f9e62a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:29:07.468 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.476+0000 7f9e67000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:29:07.468 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.476+0000 7f9e67000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:29:07.468 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.476+0000 7f9e67000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:29:07.468 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:29:07.476+0000 7f9e67000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:29:07.537 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:07.547+0000 7fe4a3ed8240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:29:07.654 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:29:07.663+0000 7fe4a3ed8240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:29:08.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.301+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 120) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f30500557b0 con 0x7f30640cbb30 2024-03-29T11:29:08.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.301+0000 7f305b400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:08.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.301+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3044035990 con 0x7f304406ae00 2024-03-29T11:29:08.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.304+0000 7f3063400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:08.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.308+0000 7f3063400700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3054007fd0 tx=0x7f3054007dc0 comp rx=0 tx=0).ready entity=mgr.10354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.386+0000 7f305b400700 1 -- 172.21.15.12:0/1525075765 <== mgr.10354 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3044035990 con 0x7f304406ae00 2024-03-29T11:29:08.377 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1 2024-03-29T11:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 -- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 msgr2=0x7f304406d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3054007fd0 tx=0x7f3054007dc0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 -- 172.21.15.12:0/1525075765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30640cbb30 msgr2=0x7f30640721f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30640cbb30 0x7f30640721f0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f30500090b0 tx=0x7f3050004750 comp rx=0 tx=0).stop 2024-03-29T11:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 -- 172.21.15.12:0/1525075765 shutdown_connections 2024-03-29T11:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f304406ae00 0x7f304406d240 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30640cd020 0x7f30640add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30640b2070 0x7f306415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 --2- 172.21.15.12:0/1525075765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30640cbb30 0x7f30640721f0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 -- 172.21.15.12:0/1525075765 >> 172.21.15.12:0/1525075765 conn(0x7f30640b0d70 msgr2=0x7f306415cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 -- 172.21.15.12:0/1525075765 shutdown_connections 2024-03-29T11:29:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.389+0000 7f306a600700 1 -- 172.21.15.12:0/1525075765 wait complete. 2024-03-29T11:29:08.399 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-29T11:29:08.400 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-03-29T11:29:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.610+0000 7fe25f000700 1 Processor -- start 2024-03-29T11:29:08.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.611+0000 7fe25f000700 1 -- start start 2024-03-29T11:29:08.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.612+0000 7fe25f000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe258155ac0 0x7fe25815c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:08.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.612+0000 7fe25f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ae670 0x7fe25815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:08.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.612+0000 7fe25f000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe25806dd00 con 0x7fe25806c910 2024-03-29T11:29:08.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.612+0000 7fe25f000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25806d0f0 con 0x7fe258155ac0 2024-03-29T11:29:08.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.612+0000 7fe25f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25806d3d0 con 0x7fe2580ae670 2024-03-29T11:29:08.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.612+0000 7fe257e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe258155ac0 0x7fe25815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:08.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.612+0000 7fe257400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe25806c910 0x7fe2580adf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33584/0 (socket says 172.21.15.12:33584) 2024-03-29T11:29:08.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.612+0000 7fe257e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe258155ac0 0x7fe25815c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59696/0 (socket says 172.21.15.12:59696) 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.612+0000 7fe257400700 1 -- 172.21.15.12:0/2348495581 learned_addr learned my addr 172.21.15.12:0/2348495581 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe25d000700 1 --2- 172.21.15.12:0/2348495581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ae670 0x7fe25815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe257e00700 1 -- 172.21.15.12:0/2348495581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ae670 msgr2=0x7fe25815ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe257e00700 1 --2- 172.21.15.12:0/2348495581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ae670 0x7fe25815ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe257e00700 1 -- 172.21.15.12:0/2348495581 >> v1:172.21.15.12:6789/0 conn(0x7fe25806c910 legacy=0x7fe2580adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe257e00700 1 -- 172.21.15.12:0/2348495581 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25806d6b0 con 0x7fe258155ac0 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe25d000700 1 --2- 172.21.15.12:0/2348495581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ae670 0x7fe25815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe256a00700 1 -- 172.21.15.12:0/2348495581 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362532777 0 0) 0x7fe25806dd00 con 0x7fe25806c910 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe257e00700 1 --2- 172.21.15.12:0/2348495581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe258155ac0 0x7fe25815c700 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fe248009840 tx=0x7fe248009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1cc4e48f133b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe256a00700 1 -- 172.21.15.12:0/2348495581 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248011830 con 0x7fe258155ac0 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe256a00700 1 -- 172.21.15.12:0/2348495581 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe248011990 con 0x7fe258155ac0 2024-03-29T11:29:08.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.613+0000 7fe256a00700 1 -- 172.21.15.12:0/2348495581 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248011d20 con 0x7fe258155ac0 2024-03-29T11:29:08.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.614+0000 7fe25f000700 1 -- 172.21.15.12:0/2348495581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe258155ac0 msgr2=0x7fe25815c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.614+0000 7fe25f000700 1 --2- 172.21.15.12:0/2348495581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe258155ac0 0x7fe25815c700 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fe248009840 tx=0x7fe248009b50 comp rx=0 tx=0).stop 2024-03-29T11:29:08.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.614+0000 7fe25f000700 1 -- 172.21.15.12:0/2348495581 shutdown_connections 2024-03-29T11:29:08.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.614+0000 7fe25f000700 1 --2- 172.21.15.12:0/2348495581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ae670 0x7fe25815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.614+0000 7fe25f000700 1 --2- 172.21.15.12:0/2348495581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe258155ac0 0x7fe25815c700 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.614+0000 7fe25f000700 1 -- 172.21.15.12:0/2348495581 >> 172.21.15.12:0/2348495581 conn(0x7fe2580cf210 msgr2=0x7fe2580cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:08.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.614+0000 7fe25f000700 1 -- 172.21.15.12:0/2348495581 shutdown_connections 2024-03-29T11:29:08.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.614+0000 7fe25f000700 1 -- 172.21.15.12:0/2348495581 wait complete. 2024-03-29T11:29:08.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.616+0000 7fe25f000700 1 Processor -- start 2024-03-29T11:29:08.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.616+0000 7fe25f000700 1 -- start start 2024-03-29T11:29:08.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe25f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25806c910 0x7fe25814b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:08.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe25f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2580ae670 0x7fe25815f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:08.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe25f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe258155ac0 0x7fe25815b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:08.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe25f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25814cb70 con 0x7fe2580ae670 2024-03-29T11:29:08.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe25f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2581521e0 con 0x7fe25806c910 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe25f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25815fc50 con 0x7fe258155ac0 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe257400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2580ae670 0x7fe25815f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe257400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2580ae670 0x7fe25815f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59708/0 (socket says 172.21.15.12:59708) 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe257400700 1 -- 172.21.15.12:0/1238329791 learned_addr learned my addr 172.21.15.12:0/1238329791 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe25d000700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe258155ac0 0x7fe25815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe257e00700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25806c910 0x7fe25814b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe257400700 1 -- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe258155ac0 msgr2=0x7fe25815b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe257400700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe258155ac0 0x7fe25815b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe257400700 1 -- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25806c910 msgr2=0x7fe25814b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe257400700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25806c910 0x7fe25814b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe257400700 1 -- 172.21.15.12:0/1238329791 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2480094a0 con 0x7fe2580ae670 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe257e00700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25806c910 0x7fe25814b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.617+0000 7fe25d000700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe258155ac0 0x7fe25815b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:29:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.618+0000 7fe257400700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2580ae670 0x7fe25815f710 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fe24400c750 tx=0x7fe24400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.618+0000 7fe24fe00700 1 -- 172.21.15.12:0/1238329791 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe244004c80 con 0x7fe2580ae670 2024-03-29T11:29:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.618+0000 7fe24fe00700 1 -- 172.21.15.12:0/1238329791 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe244004e00 con 0x7fe2580ae670 2024-03-29T11:29:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.618+0000 7fe24fe00700 1 -- 172.21.15.12:0/1238329791 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe244007310 con 0x7fe2580ae670 2024-03-29T11:29:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.618+0000 7fe25f000700 1 -- 172.21.15.12:0/1238329791 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe258156020 con 0x7fe2580ae670 2024-03-29T11:29:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.618+0000 7fe25f000700 1 -- 172.21.15.12:0/1238329791 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe258151560 con 0x7fe2580ae670 2024-03-29T11:29:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.619+0000 7fe25f000700 1 -- 172.21.15.12:0/1238329791 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe258067720 con 0x7fe2580ae670 2024-03-29T11:29:08.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.620+0000 7fe24fe00700 1 -- 172.21.15.12:0/1238329791 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 120) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fe244007470 con 0x7fe2580ae670 2024-03-29T11:29:08.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.621+0000 7fe24fe00700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe23806ab30 0x7fe23806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:08.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.621+0000 7fe24fe00700 1 -- 172.21.15.12:0/1238329791 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fe244088ea0 con 0x7fe2580ae670 2024-03-29T11:29:08.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.621+0000 7fe257e00700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe23806ab30 0x7fe23806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:08.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.622+0000 7fe257e00700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe23806ab30 0x7fe23806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe2480090b0 tx=0x7fe24801c040 comp rx=0 tx=0).ready entity=mgr.10354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:08.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.626+0000 7fe24fe00700 1 -- 172.21.15.12:0/1238329791 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe244054220 con 0x7fe2580ae670 2024-03-29T11:29:08.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.866+0000 7fe25f000700 1 -- 172.21.15.12:0/1238329791 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe258155ea0 con 0x7fe23806ab30 2024-03-29T11:29:08.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.940+0000 7fe24fe00700 1 -- 172.21.15.12:0/1238329791 <== mgr.10354 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fe2480117d0 con 0x7fe23806ab30 2024-03-29T11:29:08.932 INFO:teuthology.orchestra.run.smithi012.stdout:Retention added to path snap_test_dir1 2024-03-29T11:29:08.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 -- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe23806ab30 msgr2=0x7fe23806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe23806ab30 0x7fe23806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe2480090b0 tx=0x7fe24801c040 comp rx=0 tx=0).stop 2024-03-29T11:29:08.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 -- 172.21.15.12:0/1238329791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2580ae670 msgr2=0x7fe25815f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2580ae670 0x7fe25815f710 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fe24400c750 tx=0x7fe24400cb10 comp rx=0 tx=0).stop 2024-03-29T11:29:08.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 -- 172.21.15.12:0/1238329791 shutdown_connections 2024-03-29T11:29:08.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe23806ab30 0x7fe23806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe258155ac0 0x7fe25815b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25806c910 0x7fe25814b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 --2- 172.21.15.12:0/1238329791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2580ae670 0x7fe25815f710 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:08.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 -- 172.21.15.12:0/1238329791 >> 172.21.15.12:0/1238329791 conn(0x7fe2580cf210 msgr2=0x7fe2581534c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:08.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 -- 172.21.15.12:0/1238329791 shutdown_connections 2024-03-29T11:29:08.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:08.943+0000 7fe25f000700 1 -- 172.21.15.12:0/1238329791 wait complete. 2024-03-29T11:29:08.953 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-03-29T11:29:08.953 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-03-29-11_30_00 in ~52s... 2024-03-29T11:29:08.953 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-03-29T11:29:08.953 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-03-29T11:29:09.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.163+0000 7fa6ec000700 1 Processor -- start 2024-03-29T11:29:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.163+0000 7fa6ec000700 1 -- start start 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6ec000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6e415d9d0 0x7fa6e40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6ec000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6e40d16a0 0x7fa6e40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6ec000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa6e40d2370 0x7fa6e415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6ec000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e40d2820 con 0x7fa6e40d16a0 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6ec000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e4161420 con 0x7fa6e40d2370 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6ec000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e4161700 con 0x7fa6e415d9d0 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6e8c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6e40d16a0 0x7fa6e40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6e8c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6e40d16a0 0x7fa6e40d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55228/0 (socket says 172.21.15.12:55228) 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6e8c00700 1 -- 172.21.15.12:0/2769726957 learned_addr learned my addr 172.21.15.12:0/2769726957 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6e9600700 1 --2- 172.21.15.12:0/2769726957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6e415d9d0 0x7fa6e40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.164+0000 7fa6ea000700 1 --2- 172.21.15.12:0/2769726957 >> v2:172.21.15.159:3300/0 conn(0x7fa6e40d2370 0x7fa6e415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.165+0000 7fa6e8c00700 1 -- 172.21.15.12:0/2769726957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6e415d9d0 msgr2=0x7fa6e40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.165+0000 7fa6e8c00700 1 --2- 172.21.15.12:0/2769726957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6e415d9d0 0x7fa6e40d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.165+0000 7fa6e8c00700 1 -- 172.21.15.12:0/2769726957 >> v2:172.21.15.159:3300/0 conn(0x7fa6e40d2370 msgr2=0x7fa6e415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.165+0000 7fa6e8c00700 1 --2- 172.21.15.12:0/2769726957 >> v2:172.21.15.159:3300/0 conn(0x7fa6e40d2370 0x7fa6e415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.165+0000 7fa6e8c00700 1 -- 172.21.15.12:0/2769726957 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e41619e0 con 0x7fa6e40d16a0 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.165+0000 7fa6e8c00700 1 --2- 172.21.15.12:0/2769726957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 0x7fa6e40d1e30 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40cdc00 tx=0x7fa6d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293ccde008edd059 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.165+0000 7fa6e3e00700 1 -- 172.21.15.12:0/2769726957 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0005e30 con 0x7fa6e40d16a0 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.165+0000 7fa6e3e00700 1 -- 172.21.15.12:0/2769726957 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d0004030 con 0x7fa6e40d16a0 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.165+0000 7fa6e3e00700 1 -- 172.21.15.12:0/2769726957 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d00043e0 con 0x7fa6e40d16a0 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.166+0000 7fa6ec000700 1 -- 172.21.15.12:0/2769726957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 msgr2=0x7fa6e40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.166+0000 7fa6ec000700 1 --2- 172.21.15.12:0/2769726957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 0x7fa6e40d1e30 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40cdc00 tx=0x7fa6d0009bd0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.166+0000 7fa6ec000700 1 -- 172.21.15.12:0/2769726957 shutdown_connections 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.166+0000 7fa6ec000700 1 --2- 172.21.15.12:0/2769726957 >> v2:172.21.15.159:3300/0 conn(0x7fa6e40d2370 0x7fa6e415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.166+0000 7fa6ec000700 1 --2- 172.21.15.12:0/2769726957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6e415d9d0 0x7fa6e40d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.166+0000 7fa6ec000700 1 --2- 172.21.15.12:0/2769726957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 0x7fa6e40d1e30 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.166+0000 7fa6ec000700 1 -- 172.21.15.12:0/2769726957 >> 172.21.15.12:0/2769726957 conn(0x7fa6e408aff0 msgr2=0x7fa6e40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.166+0000 7fa6ec000700 1 -- 172.21.15.12:0/2769726957 shutdown_connections 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.166+0000 7fa6ec000700 1 -- 172.21.15.12:0/2769726957 wait complete. 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.168+0000 7fa6ec000700 1 Processor -- start 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.168+0000 7fa6ec000700 1 -- start start 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6ec000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 0x7fa6e415a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6ec000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6e40d2370 0x7fa6e40b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6ec000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6e415d9d0 0x7fa6e40b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6ec000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e40cb9f0 con 0x7fa6e40d16a0 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6ec000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e40b8e80 con 0x7fa6e415d9d0 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6ec000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e40b85e0 con 0x7fa6e40d2370 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6e9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 0x7fa6e415a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6e9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 0x7fa6e415a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55234/0 (socket says 172.21.15.12:55234) 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6e9600700 1 -- 172.21.15.12:0/435127461 learned_addr learned my addr 172.21.15.12:0/435127461 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6ea000700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6e415d9d0 0x7fa6e40b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.169+0000 7fa6e8c00700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6e40d2370 0x7fa6e40b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6e9600700 1 -- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6e40d2370 msgr2=0x7fa6e40b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6e9600700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6e40d2370 0x7fa6e40b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6e9600700 1 -- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6e415d9d0 msgr2=0x7fa6e40b7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6e9600700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6e415d9d0 0x7fa6e40b7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6e9600700 1 -- 172.21.15.12:0/435127461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d0009500 con 0x7fa6e40d16a0 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6e9600700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 0x7fa6e415a0e0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800e8a0 tx=0x7fa6d800ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6e1600700 1 -- 172.21.15.12:0/435127461 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8009590 con 0x7fa6e40d16a0 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6e1600700 1 -- 172.21.15.12:0/435127461 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d8013680 con 0x7fa6e40d16a0 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6e1600700 1 -- 172.21.15.12:0/435127461 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d80139f0 con 0x7fa6e40d16a0 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6ec000700 1 -- 172.21.15.12:0/435127461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e40b88c0 con 0x7fa6e40d16a0 2024-03-29T11:29:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.170+0000 7fa6ec000700 1 -- 172.21.15.12:0/435127461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e40cbc00 con 0x7fa6e40d16a0 2024-03-29T11:29:09.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.171+0000 7fa6ec000700 1 -- 172.21.15.12:0/435127461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e40d1ca0 con 0x7fa6e40d16a0 2024-03-29T11:29:09.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.172+0000 7fa6e1600700 1 -- 172.21.15.12:0/435127461 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 120) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fa6d8013b50 con 0x7fa6e40d16a0 2024-03-29T11:29:09.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.173+0000 7fa6e1600700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa6c406ab30 0x7fa6c406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:29:09.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.173+0000 7fa6e1600700 1 -- 172.21.15.12:0/435127461 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fa6d80890f0 con 0x7fa6e40d16a0 2024-03-29T11:29:09.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.174+0000 7fa6e8c00700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa6c406ab30 0x7fa6c406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:29:09.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.174+0000 7fa6e8c00700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa6c406ab30 0x7fa6c406cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40d1ae0 tx=0x7fa6d0008070 comp rx=0 tx=0).ready entity=mgr.10354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:29:09.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.178+0000 7fa6e1600700 1 -- 172.21.15.12:0/435127461 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa6d80543a0 con 0x7fa6e40d16a0 2024-03-29T11:29:09.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.419+0000 7fa6ec000700 1 -- 172.21.15.12:0/435127461 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa6e4161fa0 con 0x7fa6c406ab30 2024-03-29T11:29:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.427+0000 7fa6e1600700 1 -- 172.21.15.12:0/435127461 <== mgr.10354 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fa6d0006c00 con 0x7fa6c406ab30 2024-03-29T11:29:09.418 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:29:09.418 INFO:teuthology.orchestra.run.smithi012.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-03-29T11:29:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 -- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa6c406ab30 msgr2=0x7fa6c406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa6c406ab30 0x7fa6c406cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40d1ae0 tx=0x7fa6d0008070 comp rx=0 tx=0).stop 2024-03-29T11:29:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 -- 172.21.15.12:0/435127461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 msgr2=0x7fa6e415a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 0x7fa6e415a0e0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800e8a0 tx=0x7fa6d800ebb0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 -- 172.21.15.12:0/435127461 shutdown_connections 2024-03-29T11:29:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa6c406ab30 0x7fa6c406cff0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6e415d9d0 0x7fa6e40b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e40d16a0 0x7fa6e415a0e0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 --2- 172.21.15.12:0/435127461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6e40d2370 0x7fa6e40b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:29:09.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 -- 172.21.15.12:0/435127461 >> 172.21.15.12:0/435127461 conn(0x7fa6e408aff0 msgr2=0x7fa6e40ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:09.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 -- 172.21.15.12:0/435127461 shutdown_connections 2024-03-29T11:29:09.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:29:09.430+0000 7fa6ec000700 1 -- 172.21.15.12:0/435127461 wait complete. 2024-03-29T11:29:09.439 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-03-29T11:29:09.440 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-03-29T11:29:09.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:09.440 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:09.486 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:09.486 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:09.486 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:10.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:10.488 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:10.540 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:10.540 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:10.540 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:11.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:11.541 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:11.593 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:11.593 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:11.593 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:12.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:12.594 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:12.646 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:12.650 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:12.650 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:13.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:13.651 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:13.703 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:13.704 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:13.704 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:14.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:14.705 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:14.761 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:14.761 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:14.761 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:15.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:15.762 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:15.814 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:15.814 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:15.814 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:16.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:16.815 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:16.868 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:16.868 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:16.868 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:17.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:17.870 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:17.922 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:17.923 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:17.923 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:18.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:18.924 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:18.976 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:18.976 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:18.976 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:19.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:19.977 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:20.029 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:20.029 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:20.029 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:20.231 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:29:20.236 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:29:21.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:21.030 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:21.083 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:21.083 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:21.083 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:22.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:22.084 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:22.138 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:22.139 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:22.139 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:23.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:23.140 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:23.192 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:23.192 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:23.192 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:24.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:24.194 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:24.246 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:24.246 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:24.246 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:25.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:25.248 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:25.300 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:25.300 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:25.300 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:26.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:26.302 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:26.353 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:26.353 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:26.354 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:27.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:27.354 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:27.407 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:27.407 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:27.408 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:28.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:28.409 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:28.460 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:28.460 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:28.460 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:29.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:29.462 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:29.514 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:29.514 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:29.515 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:30.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:30.516 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:30.569 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:30.569 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:30.569 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:31.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:31.570 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:31.622 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:31.622 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:31.622 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:32.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:32.623 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:32.674 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:32.675 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:32.675 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:33.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:33.675 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:33.727 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:33.727 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:33.727 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:34.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:34.729 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:34.781 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:34.781 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:34.781 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:35.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:35.783 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:35.835 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:35.835 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:35.835 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:36.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:36.836 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:36.888 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:36.888 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:36.888 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:37.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:37.890 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:37.942 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:37.942 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:37.942 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:38.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:38.943 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:38.995 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:38.995 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:38.995 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:39.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:39.997 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:40.048 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:40.048 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:40.048 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:41.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:41.049 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:41.101 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:41.101 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:41.101 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:42.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:42.103 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:42.155 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:42.155 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:42.155 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:43.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:43.156 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:43.208 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:43.208 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:43.208 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:44.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:44.209 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:44.261 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:44.261 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:44.261 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:45.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:45.262 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:45.314 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:45.315 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:45.315 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:46.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:46.316 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:46.368 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:46.368 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:46.368 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:47.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:47.369 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:47.422 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:47.422 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:47.422 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:48.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:48.424 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:48.476 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:48.476 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:48.476 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:49.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:49.477 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:49.529 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:49.529 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:49.529 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:50.346 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:29:50.351 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:29:50.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:50.530 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:50.579 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:50.579 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:50.579 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:51.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:51.580 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:51.632 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:51.633 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:51.633 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:52.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:52.633 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:52.689 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:52.689 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:52.689 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:53.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:53.690 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:53.742 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:53.742 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:53.742 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:54.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:54.744 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:54.796 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:54.796 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:54.796 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:55.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:55.797 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:55.850 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:55.850 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:55.850 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:56.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:56.851 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:56.903 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:56.903 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:56.903 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:57.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:57.905 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:57.960 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:57.960 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:57.960 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:29:58.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:29:58.961 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:29:59.013 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:29:59.013 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:29:59.013 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:00.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:00.014 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:00.066 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-29T11:30:00.066 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:00.067 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:00.946 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:30:00.953+0000 7f9e51c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:30:00.946 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:30:00.953+0000 7f9e51c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:30:00.946 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:30:00.953+0000 7f9e51c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:30:00.946 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:30:00.953+0000 7f9e51c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:30:00.946 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:30:00.953+0000 7f9e51c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:30:01.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:01.067 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:01.114 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:01.116 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:01.116 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:01.116 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:01.116 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-03-29-11_30_00_UTC'] 2024-03-29T11:30:01.116 DEBUG:tasks.cephfs.test_snap_schedules:exec=1711711748.9533854, now = 1711711801.1164315, timo = 52 2024-03-29T11:30:02.117 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2024-03-29-11_30_00 in ~112s... 2024-03-29T11:30:02.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:02.117 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:02.168 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:02.169 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:02.169 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:02.169 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:03.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:03.170 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:03.221 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:03.222 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:03.222 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:03.222 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:04.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:04.223 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:04.274 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:04.276 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:04.276 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:04.276 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:05.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:05.277 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:05.325 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:05.326 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:05.326 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:05.326 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:06.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:06.327 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:06.378 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:06.379 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:06.379 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:06.379 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:07.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:07.381 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:07.431 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:07.432 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:07.432 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:07.432 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:08.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:08.434 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:08.485 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:08.486 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:08.486 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:08.486 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:09.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:09.487 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:09.538 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:09.540 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:09.540 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:09.540 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:10.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:10.541 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:10.592 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:10.593 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:10.593 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:10.593 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:11.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:11.594 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:11.645 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:11.646 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:11.646 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:11.646 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:12.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:12.647 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:12.703 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:12.703 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:12.703 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:12.703 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:13.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:13.704 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:13.755 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:13.756 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:13.756 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:13.756 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:14.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:14.758 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:14.806 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:14.807 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:14.808 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:14.808 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:15.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:15.809 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:15.860 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:15.861 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:15.861 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:15.861 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:16.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:16.862 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:16.913 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:16.914 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:16.914 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:16.914 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:17.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:17.915 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:17.965 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:17.966 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:17.967 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:17.967 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:18.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:18.967 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:19.020 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:19.021 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:19.021 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:19.021 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:20.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:20.022 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:20.073 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:20.075 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:20.075 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:20.075 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:20.468 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:30:20.473 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:30:21.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:21.077 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:21.127 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:21.128 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:21.129 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:21.129 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:22.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:22.129 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:22.180 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:22.181 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:22.181 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:22.181 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:23.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:23.183 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:23.234 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:23.236 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:23.236 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:23.236 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:24.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:24.237 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:24.288 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:24.289 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:24.289 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:24.289 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:25.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:25.291 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:25.341 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:25.342 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:25.342 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:25.343 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:26.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:26.343 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:26.395 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:26.396 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:26.397 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:26.397 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:27.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:27.398 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:27.449 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:27.450 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:27.450 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:27.450 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:28.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:28.451 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:28.504 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:28.504 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:28.504 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:29.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:29.505 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:29.556 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:29.557 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:29.557 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:29.557 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:30.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:30.558 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:30.609 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:30.610 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:30.610 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:30.611 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:31.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:31.612 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:31.663 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:31.664 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:31.664 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:31.664 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:32.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:32.665 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:32.716 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:32.717 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:32.717 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:32.717 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:33.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:33.718 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:33.769 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:33.770 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:33.770 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:33.770 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:34.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:34.771 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:34.822 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:34.823 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:34.824 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:34.824 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:35.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:35.825 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:35.878 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:35.879 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:35.879 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:35.879 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:36.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:36.880 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:36.931 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:36.932 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:36.932 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:36.932 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:37.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:37.934 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:37.984 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:37.985 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:37.985 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:37.986 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:38.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:38.987 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:39.038 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:39.040 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:39.040 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:39.040 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:40.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:40.041 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:40.092 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:40.093 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:40.093 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:40.093 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:41.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:41.094 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:41.145 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:41.146 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:41.146 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:41.146 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:42.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:42.147 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:42.197 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:42.198 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:42.198 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:42.198 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:43.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:43.199 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:43.250 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:43.254 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:43.255 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:43.255 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:44.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:44.256 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:44.307 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:44.308 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:44.308 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:44.309 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:45.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:45.310 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:45.360 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:45.361 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:45.362 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:45.362 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:46.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:46.363 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:46.414 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:46.415 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:46.415 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:46.415 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:47.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:47.416 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:47.467 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:47.468 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:47.468 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:47.468 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:48.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:48.469 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:48.520 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:48.521 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:48.521 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:48.522 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:49.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:49.523 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:49.574 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:49.575 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:49.575 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:49.575 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:50.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:50.576 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:50.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:30:50.626 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:30:50.628 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:50.628 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:50.628 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:50.628 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:51.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:51.629 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:51.680 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:51.681 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:51.681 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:51.681 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:52.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:52.685 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:52.736 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:52.737 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:52.738 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:52.738 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:53.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:53.738 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:53.790 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:53.791 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:53.791 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:53.791 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:54.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:54.792 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:54.843 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:54.844 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:54.844 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:54.844 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:55.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:55.845 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:55.895 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:55.896 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:55.897 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:55.897 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:56.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:56.897 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:56.949 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:56.950 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:56.950 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:56.951 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:57.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:57.951 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:58.002 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:58.003 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:58.003 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:58.003 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:30:59.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:30:59.004 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:30:59.055 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:30:59.056 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:30:59.056 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:30:59.056 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:31:00.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:00.057 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:31:00.111 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_30_00_UTC 2024-03-29T11:31:00.112 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_31_00_UTC 2024-03-29T11:31:00.113 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_31_00_UTC', 'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:31:00.113 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-03-29-11_31_00_UTC'} 2024-03-29T11:31:00.113 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-29T11:31:01.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:01.114 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:31:01.164 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_31_00_UTC 2024-03-29T11:31:01.165 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-29-11_31_00_UTC'} 2024-03-29T11:31:01.165 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-29T11:31:01.166 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2024-03-29-11_30_00_UTC'} 2024-03-29T11:31:01.166 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2024-03-29-11_30_00_UTC'] 2024-03-29T11:31:01.166 DEBUG:tasks.cephfs.test_snap_schedules:exec=1711711748.9533854, now = 1711711861.166215, timo = 112 2024-03-29T11:31:02.167 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.386+0000 7f06fa800700 1 Processor -- start 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.386+0000 7f06fa800700 1 -- start start 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.387+0000 7f06fa800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415a5c0 0x7f06f40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.387+0000 7f06fa800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06f415dfe0 0x7f06f40ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.387+0000 7f06fa800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f40cb310 con 0x7f06f40ce670 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.387+0000 7f06fa800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f40ceb00 con 0x7f06f415dfe0 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.387+0000 7f06fa800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f4070f00 con 0x7f06f415a5c0 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.387+0000 7f06f2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06f415dfe0 0x7f06f40ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.388+0000 7f06f2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06f415dfe0 0x7f06f40ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39336/0 (socket says 172.21.15.12:39336) 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.388+0000 7f06f2a00700 1 -- 172.21.15.12:0/1964878492 learned_addr learned my addr 172.21.15.12:0/1964878492 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.388+0000 7f06f3400700 1 --2- 172.21.15.12:0/1964878492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415a5c0 0x7f06f40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.388+0000 7f06f2a00700 1 -- 172.21.15.12:0/1964878492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415a5c0 msgr2=0x7f06f40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.388+0000 7f06f2a00700 1 --2- 172.21.15.12:0/1964878492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415a5c0 0x7f06f40709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.388+0000 7f06f2a00700 1 -- 172.21.15.12:0/1964878492 >> v1:172.21.15.159:6789/0 conn(0x7f06f40ce670 legacy=0x7f06f40b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.388+0000 7f06f2a00700 1 -- 172.21.15.12:0/1964878492 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f40cca90 con 0x7f06f415dfe0 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.388+0000 7f06f3400700 1 --2- 172.21.15.12:0/1964878492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415a5c0 0x7f06f40709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.388+0000 7f06f2a00700 1 --2- 172.21.15.12:0/1964878492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f415dfe0 0x7f06f40ce130 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f06e00098f0 tx=0x7f06e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787cd7f6b69b963c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06f2000700 1 -- 172.21.15.12:0/1964878492 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e000b8a0 con 0x7f06f415dfe0 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06f2000700 1 -- 172.21.15.12:0/1964878492 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e000ba00 con 0x7f06f415dfe0 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06f2000700 1 -- 172.21.15.12:0/1964878492 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e000bd90 con 0x7f06f415dfe0 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06fa800700 1 -- 172.21.15.12:0/1964878492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f415dfe0 msgr2=0x7f06f40ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06fa800700 1 --2- 172.21.15.12:0/1964878492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f415dfe0 0x7f06f40ce130 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f06e00098f0 tx=0x7f06e0009c00 comp rx=0 tx=0).stop 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06fa800700 1 -- 172.21.15.12:0/1964878492 shutdown_connections 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06fa800700 1 --2- 172.21.15.12:0/1964878492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415a5c0 0x7f06f40709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06fa800700 1 --2- 172.21.15.12:0/1964878492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f415dfe0 0x7f06f40ce130 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06fa800700 1 -- 172.21.15.12:0/1964878492 >> 172.21.15.12:0/1964878492 conn(0x7f06f40b0d70 msgr2=0x7f06f40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06fa800700 1 -- 172.21.15.12:0/1964878492 shutdown_connections 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.389+0000 7f06fa800700 1 -- 172.21.15.12:0/1964878492 wait complete. 2024-03-29T11:31:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.391+0000 7f06fa800700 1 Processor -- start 2024-03-29T11:31:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.391+0000 7f06fa800700 1 -- start start 2024-03-29T11:31:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.392+0000 7f06fa800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f40ce670 0x7f06f415ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.392+0000 7f06fa800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06f415a5c0 0x7f06f415c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.392+0000 7f06fa800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415dfe0 0x7f06f40b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.392+0000 7f06fa800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f414ef70 con 0x7f06f40ce670 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.392+0000 7f06fa800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f415d060 con 0x7f06f415a5c0 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.392+0000 7f06fa800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f415d340 con 0x7f06f415dfe0 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.392+0000 7f06f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f40ce670 0x7f06f415ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f40ce670 0x7f06f415ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39342/0 (socket says 172.21.15.12:39342) 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f3400700 1 -- 172.21.15.12:0/1000078410 learned_addr learned my addr 172.21.15.12:0/1000078410 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f2a00700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06f415a5c0 0x7f06f415c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f3e00700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415dfe0 0x7f06f40b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f3400700 1 -- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415dfe0 msgr2=0x7f06f40b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f3400700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415dfe0 0x7f06f40b2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f3400700 1 -- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06f415a5c0 msgr2=0x7f06f415c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f3400700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06f415a5c0 0x7f06f415c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f3400700 1 -- 172.21.15.12:0/1000078410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e0009550 con 0x7f06f40ce670 2024-03-29T11:31:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f3e00700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415dfe0 0x7f06f40b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06f3400700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f40ce670 0x7f06f415ae50 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f06e400e810 tx=0x7f06e400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.393+0000 7f06eb400700 1 -- 172.21.15.12:0/1000078410 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4004230 con 0x7f06f40ce670 2024-03-29T11:31:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.394+0000 7f06eb400700 1 -- 172.21.15.12:0/1000078410 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e40043b0 con 0x7f06f40ce670 2024-03-29T11:31:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.394+0000 7f06eb400700 1 -- 172.21.15.12:0/1000078410 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e40038e0 con 0x7f06f40ce670 2024-03-29T11:31:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.394+0000 7f06fa800700 1 -- 172.21.15.12:0/1000078410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f415d620 con 0x7f06f40ce670 2024-03-29T11:31:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.394+0000 7f06fa800700 1 -- 172.21.15.12:0/1000078410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f415cc50 con 0x7f06f40ce670 2024-03-29T11:31:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.395+0000 7f06fa800700 1 -- 172.21.15.12:0/1000078410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f4067720 con 0x7f06f40ce670 2024-03-29T11:31:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.396+0000 7f06eb400700 1 -- 172.21.15.12:0/1000078410 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 124) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f06e4003a40 con 0x7f06f40ce670 2024-03-29T11:31:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.397+0000 7f06eb400700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06d406ae00 0x7f06d406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.397+0000 7f06eb400700 1 -- 172.21.15.12:0/1000078410 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f06e40891e0 con 0x7f06f40ce670 2024-03-29T11:31:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.397+0000 7f06f2a00700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06d406ae00 0x7f06d406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.398+0000 7f06f2a00700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06d406ae00 0x7f06d406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f06e0014040 tx=0x7f06e0008030 comp rx=0 tx=0).ready entity=mgr.10354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.403+0000 7f06eb400700 1 -- 172.21.15.12:0/1000078410 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06e4054050 con 0x7f06f40ce670 2024-03-29T11:31:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.645+0000 7f06fa800700 1 -- 172.21.15.12:0/1000078410 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f06f4071760 con 0x7f06d406ae00 2024-03-29T11:31:02.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.727+0000 7f06eb400700 1 -- 172.21.15.12:0/1000078410 <== mgr.10354 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f06e000bb70 con 0x7f06d406ae00 2024-03-29T11:31:02.721 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path snap_test_dir1 2024-03-29T11:31:02.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 -- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06d406ae00 msgr2=0x7f06d406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06d406ae00 0x7f06d406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f06e0014040 tx=0x7f06e0008030 comp rx=0 tx=0).stop 2024-03-29T11:31:02.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 -- 172.21.15.12:0/1000078410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f40ce670 msgr2=0x7f06f415ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f40ce670 0x7f06f415ae50 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f06e400e810 tx=0x7f06e400eb20 comp rx=0 tx=0).stop 2024-03-29T11:31:02.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 -- 172.21.15.12:0/1000078410 shutdown_connections 2024-03-29T11:31:02.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06d406ae00 0x7f06d406d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:02.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06f415dfe0 0x7f06f40b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:02.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06f40ce670 0x7f06f415ae50 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:02.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 --2- 172.21.15.12:0/1000078410 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06f415a5c0 0x7f06f415c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:02.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 -- 172.21.15.12:0/1000078410 >> 172.21.15.12:0/1000078410 conn(0x7f06f40b0d70 msgr2=0x7f06f40b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:02.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 -- 172.21.15.12:0/1000078410 shutdown_connections 2024-03-29T11:31:02.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:02.730+0000 7f06fa800700 1 -- 172.21.15.12:0/1000078410 wait complete. 2024-03-29T11:31:02.741 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-03-29T11:31:02.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:02.741 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:31:02.786 INFO:teuthology.orchestra.run.smithi012.stdout:scheduled-2024-03-29-11_31_00_UTC 2024-03-29T11:31:02.787 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-29-11_31_00_UTC 2024-03-29T11:31:02.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:02.787 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-29-11_31_00_UTC) 2024-03-29T11:31:02.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:02.877 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-29T11:31:02.926 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T11:31:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.132+0000 7f48be200700 1 Processor -- start 2024-03-29T11:31:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.132+0000 7f48be200700 1 -- start start 2024-03-29T11:31:03.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.133+0000 7f48be200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80d21f0 0x7f48b80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:03.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.133+0000 7f48be200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b80ce270 con 0x7f48b80cec40 2024-03-29T11:31:03.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.133+0000 7f48be200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b815b600 con 0x7f48b81611b0 2024-03-29T11:31:03.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.133+0000 7f48be200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80cf0b0 con 0x7f48b80d21f0 2024-03-29T11:31:03.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.134+0000 7f48b7e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f48b80cec40 0x7f48b8161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45694/0 (socket says 172.21.15.12:45694) 2024-03-29T11:31:03.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.134+0000 7f48b7e00700 1 -- 172.21.15.12:0/1162750664 learned_addr learned my addr 172.21.15.12:0/1162750664 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:03.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.134+0000 7f48b6a00700 1 --2- 172.21.15.12:0/1162750664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80d21f0 0x7f48b80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:03.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.134+0000 7f48b6000700 1 -- 172.21.15.12:0/1162750664 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843869362 0 0) 0x7f48b80ce270 con 0x7f48b80cec40 2024-03-29T11:31:03.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.134+0000 7f48b6000700 1 -- 172.21.15.12:0/1162750664 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f489c0035b0 con 0x7f48b80cec40 2024-03-29T11:31:03.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.134+0000 7f48b6000700 1 -- 172.21.15.12:0/1162750664 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649710801 0 0) 0x7f48b815b600 con 0x7f48b81611b0 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.134+0000 7f48b6000700 1 -- 172.21.15.12:0/1162750664 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48b80ce270 con 0x7f48b81611b0 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.135+0000 7f48b6a00700 1 -- 172.21.15.12:0/1162750664 >> v1:172.21.15.159:6789/0 conn(0x7f48b81611b0 legacy=0x7f48b815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.135+0000 7f48b6a00700 1 -- 172.21.15.12:0/1162750664 >> v1:172.21.15.12:6789/0 conn(0x7f48b80cec40 legacy=0x7f48b8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.135+0000 7f48b6000700 1 -- 172.21.15.12:0/1162750664 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4181410079 0 0) 0x7f489c0035b0 con 0x7f48b80cec40 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.135+0000 7f48b6a00700 1 -- 172.21.15.12:0/1162750664 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b80d2760 con 0x7f48b80d21f0 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.135+0000 7f48b6a00700 1 --2- 172.21.15.12:0/1162750664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80d21f0 0x7f48b80ce700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009870 tx=0x7f48a4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ea7ed5c4b036086 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.135+0000 7f48b6000700 1 -- 172.21.15.12:0/1162750664 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b00019f0 con 0x7f48b80d21f0 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.135+0000 7f48b6000700 1 -- 172.21.15.12:0/1162750664 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a4004030 con 0x7f48b80d21f0 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.135+0000 7f48b6000700 1 -- 172.21.15.12:0/1162750664 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a40043c0 con 0x7f48b80d21f0 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.136+0000 7f48be200700 1 -- 172.21.15.12:0/1162750664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80d21f0 msgr2=0x7f48b80ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.136+0000 7f48be200700 1 --2- 172.21.15.12:0/1162750664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80d21f0 0x7f48b80ce700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009870 tx=0x7f48a4009b80 comp rx=0 tx=0).stop 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.136+0000 7f48be200700 1 -- 172.21.15.12:0/1162750664 shutdown_connections 2024-03-29T11:31:03.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.136+0000 7f48be200700 1 --2- 172.21.15.12:0/1162750664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80d21f0 0x7f48b80ce700 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:03.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.136+0000 7f48be200700 1 -- 172.21.15.12:0/1162750664 >> 172.21.15.12:0/1162750664 conn(0x7f48b808aff0 msgr2=0x7f48b80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:03.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.136+0000 7f48be200700 1 -- 172.21.15.12:0/1162750664 shutdown_connections 2024-03-29T11:31:03.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.136+0000 7f48be200700 1 -- 172.21.15.12:0/1162750664 wait complete. 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.138+0000 7f48be200700 1 Processor -- start 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.138+0000 7f48be200700 1 -- start start 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48be200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80cec40 0x7f48b81f6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48be200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48b80d21f0 0x7f48b80c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48be200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48b81611b0 0x7f48b80cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48be200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b8150470 con 0x7f48b81611b0 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48be200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80cb410 con 0x7f48b80d21f0 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48be200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80cb6f0 con 0x7f48b80cec40 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48b7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48b81611b0 0x7f48b80cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48b7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48b81611b0 0x7f48b80cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39350/0 (socket says 172.21.15.12:39350) 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48b7e00700 1 -- 172.21.15.12:0/652321619 learned_addr learned my addr 172.21.15.12:0/652321619 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48b6a00700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48b80d21f0 0x7f48b80c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48b7400700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80cec40 0x7f48b81f6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48b7e00700 1 -- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80cec40 msgr2=0x7f48b81f6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48b7e00700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80cec40 0x7f48b81f6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48b7e00700 1 -- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48b80d21f0 msgr2=0x7f48b80c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48b7e00700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48b80d21f0 0x7f48b80c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.139+0000 7f48b7e00700 1 -- 172.21.15.12:0/652321619 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a40094a0 con 0x7f48b81611b0 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.140+0000 7f48b7e00700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48b81611b0 0x7f48b80cc4b0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f48b0001700 tx=0x7f48b000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.140+0000 7f48af400700 1 -- 172.21.15.12:0/652321619 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b000ad80 con 0x7f48b81611b0 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.140+0000 7f48af400700 1 -- 172.21.15.12:0/652321619 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b00070d0 con 0x7f48b81611b0 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.140+0000 7f48be200700 1 -- 172.21.15.12:0/652321619 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b80cb9d0 con 0x7f48b81611b0 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.140+0000 7f48af400700 1 -- 172.21.15.12:0/652321619 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b0007480 con 0x7f48b81611b0 2024-03-29T11:31:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.140+0000 7f48be200700 1 -- 172.21.15.12:0/652321619 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b80cbf90 con 0x7f48b81611b0 2024-03-29T11:31:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.141+0000 7f48be200700 1 -- 172.21.15.12:0/652321619 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b8067720 con 0x7f48b81611b0 2024-03-29T11:31:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.142+0000 7f48af400700 1 -- 172.21.15.12:0/652321619 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 124) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f48b00075e0 con 0x7f48b81611b0 2024-03-29T11:31:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.143+0000 7f48af400700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489806ae00 0x7f489806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.143+0000 7f48af400700 1 -- 172.21.15.12:0/652321619 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f48b0089630 con 0x7f48b81611b0 2024-03-29T11:31:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.144+0000 7f48b7400700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489806ae00 0x7f489806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.144+0000 7f48b7400700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489806ae00 0x7f489806d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f48a8001220 tx=0x7f48a8009040 comp rx=0 tx=0).ready entity=mgr.10354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.147+0000 7f48af400700 1 -- 172.21.15.12:0/652321619 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f48b00545b0 con 0x7f48b81611b0 2024-03-29T11:31:03.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:03.429+0000 7f48be200700 1 -- 172.21.15.12:0/652321619 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f48b80b5d00 con 0x7f48b81611b0 2024-03-29T11:31:04.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.416+0000 7f48af400700 1 -- 172.21.15.12:0/652321619 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f48b0054980 con 0x7f48b81611b0 2024-03-29T11:31:04.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.417+0000 7f48af400700 1 -- 172.21.15.12:0/652321619 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v125) v1 ==== 93+0+0 (secure 0 0 0) 0x7f48b0074140 con 0x7f48b81611b0 2024-03-29T11:31:04.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 -- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489806ae00 msgr2=0x7f489806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489806ae00 0x7f489806d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f48a8001220 tx=0x7f48a8009040 comp rx=0 tx=0).stop 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 -- 172.21.15.12:0/652321619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48b81611b0 msgr2=0x7f48b80cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48b81611b0 0x7f48b80cc4b0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f48b0001700 tx=0x7f48b000a340 comp rx=0 tx=0).stop 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 -- 172.21.15.12:0/652321619 shutdown_connections 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489806ae00 0x7f489806d2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48b81611b0 0x7f48b80cc4b0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48b80cec40 0x7f48b81f6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 --2- 172.21.15.12:0/652321619 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48b80d21f0 0x7f48b80c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 -- 172.21.15.12:0/652321619 >> 172.21.15.12:0/652321619 conn(0x7f48b808aff0 msgr2=0x7f48b815f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 -- 172.21.15.12:0/652321619 shutdown_connections 2024-03-29T11:31:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:04.421+0000 7f48be200700 1 -- 172.21.15.12:0/652321619 wait complete. 2024-03-29T11:31:04.437 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:31:04.437 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:31:04.531 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:31:04.620 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:31:04.621 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:31:04.621 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:31:04.642 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:04.648+0000 7ff414758240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:31:04.651 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:04.657+0000 7fb0bba5a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:31:04.666 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:31:04.666 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:31:04.757 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:04.840 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:04.846+0000 7fb0bba5a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:31:04.859 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:04.866+0000 7ff414758240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:31:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.035+0000 7f4c24c00700 1 Processor -- start 2024-03-29T11:31:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.035+0000 7f4c24c00700 1 -- start start 2024-03-29T11:31:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.037+0000 7f4c24c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4c200b3af0 0x7f4c2015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.037+0000 7f4c24c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4c20072580 0x7f4c200b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.037+0000 7f4c24c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c200b2180 con 0x7f4c200b2e50 2024-03-29T11:31:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.037+0000 7f4c24c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2015b100 con 0x7f4c20072580 2024-03-29T11:31:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.037+0000 7f4c24c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2015b3e0 con 0x7f4c200b3af0 2024-03-29T11:31:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.037+0000 7f4c1d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4c20072580 0x7f4c200b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.037+0000 7f4c1e800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4c200b2e50 0x7f4c200709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45722/0 (socket says 172.21.15.12:45722) 2024-03-29T11:31:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.037+0000 7f4c1e800700 1 -- 172.21.15.12:0/1751733554 learned_addr learned my addr 172.21.15.12:0/1751733554 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.037+0000 7f4c1de00700 1 --2- 172.21.15.12:0/1751733554 >> v2:172.21.15.159:3300/0 conn(0x7f4c200b3af0 0x7f4c2015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.038+0000 7f4c1d400700 1 -- 172.21.15.12:0/1751733554 >> v2:172.21.15.159:3300/0 conn(0x7f4c200b3af0 msgr2=0x7f4c2015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.038+0000 7f4c1d400700 1 --2- 172.21.15.12:0/1751733554 >> v2:172.21.15.159:3300/0 conn(0x7f4c200b3af0 0x7f4c2015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.038+0000 7f4c1d400700 1 -- 172.21.15.12:0/1751733554 >> v1:172.21.15.12:6789/0 conn(0x7f4c200b2e50 legacy=0x7f4c200709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.038+0000 7f4c1d400700 1 -- 172.21.15.12:0/1751733554 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c200b23f0 con 0x7f4c20072580 2024-03-29T11:31:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.038+0000 7f4c1ca00700 1 -- 172.21.15.12:0/1751733554 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711771821 0 0) 0x7f4c200b2180 con 0x7f4c200b2e50 2024-03-29T11:31:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.038+0000 7f4c1d400700 1 --2- 172.21.15.12:0/1751733554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c20072580 0x7f4c200b2910 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f4c2006b860 tx=0x7f4c0c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a59ef5e237ef6b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.038+0000 7f4c1ca00700 1 -- 172.21.15.12:0/1751733554 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c011830 con 0x7f4c20072580 2024-03-29T11:31:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.038+0000 7f4c1ca00700 1 -- 172.21.15.12:0/1751733554 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0c011990 con 0x7f4c20072580 2024-03-29T11:31:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.038+0000 7f4c1ca00700 1 -- 172.21.15.12:0/1751733554 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c011d20 con 0x7f4c20072580 2024-03-29T11:31:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.039+0000 7f4c24c00700 1 -- 172.21.15.12:0/1751733554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c20072580 msgr2=0x7f4c200b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.039+0000 7f4c24c00700 1 --2- 172.21.15.12:0/1751733554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c20072580 0x7f4c200b2910 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f4c2006b860 tx=0x7f4c0c009b20 comp rx=0 tx=0).stop 2024-03-29T11:31:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.039+0000 7f4c24c00700 1 -- 172.21.15.12:0/1751733554 shutdown_connections 2024-03-29T11:31:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.039+0000 7f4c24c00700 1 --2- 172.21.15.12:0/1751733554 >> v2:172.21.15.159:3300/0 conn(0x7f4c200b3af0 0x7f4c2015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.039+0000 7f4c24c00700 1 --2- 172.21.15.12:0/1751733554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c20072580 0x7f4c200b2910 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.039+0000 7f4c24c00700 1 -- 172.21.15.12:0/1751733554 >> 172.21.15.12:0/1751733554 conn(0x7f4c200cf190 msgr2=0x7f4c200cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.039+0000 7f4c24c00700 1 -- 172.21.15.12:0/1751733554 shutdown_connections 2024-03-29T11:31:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.039+0000 7f4c24c00700 1 -- 172.21.15.12:0/1751733554 wait complete. 2024-03-29T11:31:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.042+0000 7f4c24c00700 1 Processor -- start 2024-03-29T11:31:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.042+0000 7f4c24c00700 1 -- start start 2024-03-29T11:31:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c24c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c20072580 0x7f4c20149870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c24c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c200b2e50 0x7f4c200cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c24c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c200b3af0 0x7f4c2006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c24c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c200ce2c0 con 0x7f4c200b2e50 2024-03-29T11:31:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c24c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c200cd6b0 con 0x7f4c20072580 2024-03-29T11:31:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c24c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c200cd990 con 0x7f4c200b3af0 2024-03-29T11:31:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c1d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c200b2e50 0x7f4c200cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c1d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c200b2e50 0x7f4c200cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39388/0 (socket says 172.21.15.12:39388) 2024-03-29T11:31:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c1d400700 1 -- 172.21.15.12:0/36912423 learned_addr learned my addr 172.21.15.12:0/36912423 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c1de00700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c20072580 0x7f4c20149870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.044+0000 7f4c1e800700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c200b3af0 0x7f4c2006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.045+0000 7f4c1d400700 1 -- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c200b3af0 msgr2=0x7f4c2006d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.045+0000 7f4c1d400700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c200b3af0 0x7f4c2006d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.045+0000 7f4c1d400700 1 -- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c20072580 msgr2=0x7f4c20149870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.045+0000 7f4c1d400700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c20072580 0x7f4c20149870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.045+0000 7f4c1d400700 1 -- 172.21.15.12:0/36912423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c0094a0 con 0x7f4c200b2e50 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.045+0000 7f4c1d400700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c200b2e50 0x7f4c200cccd0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c0090b0 tx=0x7f4c0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.045+0000 7f4c16000700 1 -- 172.21.15.12:0/36912423 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c0036a0 con 0x7f4c200b2e50 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.046+0000 7f4c16000700 1 -- 172.21.15.12:0/36912423 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0c01f070 con 0x7f4c200b2e50 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.046+0000 7f4c24c00700 1 -- 172.21.15.12:0/36912423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c200cdc70 con 0x7f4c200b2e50 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.046+0000 7f4c16000700 1 -- 172.21.15.12:0/36912423 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c003a60 con 0x7f4c200b2e50 2024-03-29T11:31:05.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.046+0000 7f4c24c00700 1 -- 172.21.15.12:0/36912423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c20154f80 con 0x7f4c200b2e50 2024-03-29T11:31:05.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.047+0000 7f4c24c00700 1 -- 172.21.15.12:0/36912423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c200ce470 con 0x7f4c200b2e50 2024-03-29T11:31:05.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.048+0000 7f4c16000700 1 -- 172.21.15.12:0/36912423 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f4c0c003bc0 con 0x7f4c200b2e50 2024-03-29T11:31:05.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.049+0000 7f4c16000700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c0006ade0 0x7f4c0006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.049+0000 7f4c16000700 1 -- 172.21.15.12:0/36912423 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f4c0c021070 con 0x7f4c200b2e50 2024-03-29T11:31:05.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.049+0000 7f4c1de00700 1 -- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c0006ade0 msgr2=0x7f4c0006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:05.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.049+0000 7f4c1de00700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c0006ade0 0x7f4c0006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.053+0000 7f4c16000700 1 -- 172.21.15.12:0/36912423 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c200ce470 con 0x7f4c200b2e50 2024-03-29T11:31:05.157 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:05.163+0000 7fb0bba5a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:31:05.231 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:05.238+0000 7ff414758240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:31:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.250+0000 7f4c1de00700 1 -- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c0006ade0 msgr2=0x7f4c0006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.250+0000 7f4c1de00700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c0006ade0 0x7f4c0006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:05.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.306+0000 7f4c24c00700 1 -- 172.21.15.12:0/36912423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c20155140 con 0x7f4c200b2e50 2024-03-29T11:31:05.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.307+0000 7f4c16000700 1 -- 172.21.15.12:0/36912423 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+16900 (secure 0 0 0) 0x7f4c0c015020 con 0x7f4c200b2e50 2024-03-29T11:31:05.301 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:05.301 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":213,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:04.402813+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:28:47.651679+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-03-29T11:28:49.085953+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":213,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.18","mappings":[{"from":0,"to":2}]},{"pgid":"25.1f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2122872060":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/2280623603":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/3600559047":"2024-03-30T11:29:07.287915+0000","172.21.15.159:6833/62646":"2024-03-30T11:29:07.287915+0000","172.21.15.159:6832/62646":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/4050963121":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/67497648":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/343093900":"2024-03-30T11:29:07.287915+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 -- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c0006ade0 msgr2=0x7f4c0006d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c0006ade0 0x7f4c0006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 -- 172.21.15.12:0/36912423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c200b2e50 msgr2=0x7f4c200cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c200b2e50 0x7f4c200cccd0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c0090b0 tx=0x7f4c0c004750 comp rx=0 tx=0).stop 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 -- 172.21.15.12:0/36912423 shutdown_connections 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c0006ade0 0x7f4c0006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c200b3af0 0x7f4c2006d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c20072580 0x7f4c20149870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 --2- 172.21.15.12:0/36912423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c200b2e50 0x7f4c200cccd0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 -- 172.21.15.12:0/36912423 >> 172.21.15.12:0/36912423 conn(0x7f4c200cf190 msgr2=0x7f4c2015e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 -- 172.21.15.12:0/36912423 shutdown_connections 2024-03-29T11:31:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.310+0000 7f4c24c00700 1 -- 172.21.15.12:0/36912423 wait complete. 2024-03-29T11:31:05.322 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:28:47.651679+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '207', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:28:49.085953+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 213, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:31:05.323 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:31:05.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.532+0000 7fd9b1400700 1 Processor -- start 2024-03-29T11:31:05.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.533+0000 7fd9b1400700 1 -- start start 2024-03-29T11:31:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.534+0000 7fd9b1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac15d800 0x7fd9ac0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.534+0000 7fd9b1400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9ac0ce230 0x7fd9ac0ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.534+0000 7fd9b1400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0709c0 con 0x7fd9ac0b2070 2024-03-29T11:31:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.534+0000 7fd9b1400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac0b40f0 con 0x7fd9ac0ce230 2024-03-29T11:31:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.534+0000 7fd9b1400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac070c80 con 0x7fd9ac15d800 2024-03-29T11:31:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.534+0000 7fd9a9e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9ac0ce230 0x7fd9ac0ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.534+0000 7fd9a9e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9ac0ce230 0x7fd9ac0ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39394/0 (socket says 172.21.15.12:39394) 2024-03-29T11:31:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.534+0000 7fd9aa800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac15d800 0x7fd9ac0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.534+0000 7fd9a9e00700 1 -- 172.21.15.12:0/1198332980 learned_addr learned my addr 172.21.15.12:0/1198332980 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.535+0000 7fd9aa800700 1 -- 172.21.15.12:0/1198332980 >> v2:172.21.15.12:3300/0 conn(0x7fd9ac0ce230 msgr2=0x7fd9ac0ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.535+0000 7fd9aa800700 1 --2- 172.21.15.12:0/1198332980 >> v2:172.21.15.12:3300/0 conn(0x7fd9ac0ce230 0x7fd9ac0ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.535+0000 7fd9aa800700 1 -- 172.21.15.12:0/1198332980 >> v1:172.21.15.159:6789/0 conn(0x7fd9ac0b2070 legacy=0x7fd9ac15dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.535+0000 7fd9aa800700 1 -- 172.21.15.12:0/1198332980 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac070f60 con 0x7fd9ac15d800 2024-03-29T11:31:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.535+0000 7fd9aa800700 1 --2- 172.21.15.12:0/1198332980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac15d800 0x7fd9ac0725c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009840 tx=0x7fd99c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b957917362c0f131 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.535+0000 7fd9a9400700 1 -- 172.21.15.12:0/1198332980 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c011830 con 0x7fd9ac15d800 2024-03-29T11:31:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.535+0000 7fd9a9400700 1 -- 172.21.15.12:0/1198332980 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c011990 con 0x7fd9ac15d800 2024-03-29T11:31:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.535+0000 7fd9a9400700 1 -- 172.21.15.12:0/1198332980 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c011d20 con 0x7fd9ac15d800 2024-03-29T11:31:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.536+0000 7fd9b1400700 1 -- 172.21.15.12:0/1198332980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac15d800 msgr2=0x7fd9ac0725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.536+0000 7fd9b1400700 1 --2- 172.21.15.12:0/1198332980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac15d800 0x7fd9ac0725c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009840 tx=0x7fd99c009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.536+0000 7fd9b1400700 1 -- 172.21.15.12:0/1198332980 shutdown_connections 2024-03-29T11:31:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.536+0000 7fd9b1400700 1 --2- 172.21.15.12:0/1198332980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac15d800 0x7fd9ac0725c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.536+0000 7fd9b1400700 1 --2- 172.21.15.12:0/1198332980 >> v2:172.21.15.12:3300/0 conn(0x7fd9ac0ce230 0x7fd9ac0ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.536+0000 7fd9b1400700 1 -- 172.21.15.12:0/1198332980 >> 172.21.15.12:0/1198332980 conn(0x7fd9ac0b0d70 msgr2=0x7fd9ac0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.536+0000 7fd9b1400700 1 -- 172.21.15.12:0/1198332980 shutdown_connections 2024-03-29T11:31:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.536+0000 7fd9b1400700 1 -- 172.21.15.12:0/1198332980 wait complete. 2024-03-29T11:31:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.538+0000 7fd9b1400700 1 Processor -- start 2024-03-29T11:31:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.538+0000 7fd9b1400700 1 -- start start 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9b1400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9ac0b2070 0x7fd9ac154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9b1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac0ce230 0x7fd9ac1ee740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9b1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9ac15d800 0x7fd9ac15a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9b1400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac1553c0 con 0x7fd9ac15d800 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9b1400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac15b100 con 0x7fd9ac0b2070 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9b1400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac1eec80 con 0x7fd9ac0ce230 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9ab200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9ac15d800 0x7fd9ac15a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9aa800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9ac0b2070 0x7fd9ac154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9aa800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9ac0b2070 0x7fd9ac154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:60344/0 (socket says 172.21.15.12:60344) 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9aa800700 1 -- 172.21.15.12:0/2396064501 learned_addr learned my addr 172.21.15.12:0/2396064501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.539+0000 7fd9a9e00700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac0ce230 0x7fd9ac1ee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9ab200700 1 -- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac0ce230 msgr2=0x7fd9ac1ee740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9ab200700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac0ce230 0x7fd9ac1ee740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9ab200700 1 -- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9ac0b2070 msgr2=0x7fd9ac154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9ab200700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9ac0b2070 0x7fd9ac154e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9ab200700 1 -- 172.21.15.12:0/2396064501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c0094a0 con 0x7fd9ac15d800 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9a9e00700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac0ce230 0x7fd9ac1ee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9ab200700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9ac15d800 0x7fd9ac15a540 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400cb10 tx=0x7fd9a400ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9a2a00700 1 -- 172.21.15.12:0/2396064501 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4003c00 con 0x7fd9ac15d800 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9a2a00700 1 -- 172.21.15.12:0/2396064501 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a4003d60 con 0x7fd9ac15d800 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9a2a00700 1 -- 172.21.15.12:0/2396064501 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a400a9e0 con 0x7fd9ac15d800 2024-03-29T11:31:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.540+0000 7fd9b1400700 1 -- 172.21.15.12:0/2396064501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac06cae0 con 0x7fd9ac15d800 2024-03-29T11:31:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.541+0000 7fd9b1400700 1 -- 172.21.15.12:0/2396064501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac06d0a0 con 0x7fd9ac15d800 2024-03-29T11:31:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.542+0000 7fd9b1400700 1 -- 172.21.15.12:0/2396064501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac067720 con 0x7fd9ac15d800 2024-03-29T11:31:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.543+0000 7fd9a2a00700 1 -- 172.21.15.12:0/2396064501 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fd9a400ab40 con 0x7fd9ac15d800 2024-03-29T11:31:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.544+0000 7fd9a2a00700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd98c06adb0 0x7fd98c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.544+0000 7fd9a2a00700 1 -- 172.21.15.12:0/2396064501 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fd9a4089470 con 0x7fd9ac15d800 2024-03-29T11:31:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.544+0000 7fd9aa800700 1 -- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd98c06adb0 msgr2=0x7fd98c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.544+0000 7fd9aa800700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd98c06adb0 0x7fd98c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.548+0000 7fd9a2a00700 1 -- 172.21.15.12:0/2396064501 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9a40543b0 con 0x7fd9ac15d800 2024-03-29T11:31:05.686 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:05.692+0000 7fb0bba5a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:31:05.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.745+0000 7fd9aa800700 1 -- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd98c06adb0 msgr2=0x7fd98c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:05.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.745+0000 7fd9aa800700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd98c06adb0 0x7fd98c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:05.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.762+0000 7fd9b1400700 1 -- 172.21.15.12:0/2396064501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9ac06b9a0 con 0x7fd9ac15d800 2024-03-29T11:31:05.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.763+0000 7fd9a2a00700 1 -- 172.21.15.12:0/2396064501 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v213) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9a40579d0 con 0x7fd9ac15d800 2024-03-29T11:31:05.756 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:31:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 -- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd98c06adb0 msgr2=0x7fd98c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd98c06adb0 0x7fd98c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 -- 172.21.15.12:0/2396064501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9ac15d800 msgr2=0x7fd9ac15a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9ac15d800 0x7fd9ac15a540 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400cb10 tx=0x7fd9a400ce20 comp rx=0 tx=0).stop 2024-03-29T11:31:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 -- 172.21.15.12:0/2396064501 shutdown_connections 2024-03-29T11:31:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd98c06adb0 0x7fd98c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9ac15d800 0x7fd9ac15a540 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9ac0b2070 0x7fd9ac154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 --2- 172.21.15.12:0/2396064501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9ac0ce230 0x7fd9ac1ee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 -- 172.21.15.12:0/2396064501 >> 172.21.15.12:0/2396064501 conn(0x7fd9ac0b0d70 msgr2=0x7fd9ac15dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 -- 172.21.15.12:0/2396064501 shutdown_connections 2024-03-29T11:31:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.765+0000 7fd9b1400700 1 -- 172.21.15.12:0/2396064501 wait complete. 2024-03-29T11:31:05.778 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:31:05.806 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:05.813+0000 7ff414758240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:31:05.942 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:05.948+0000 7fb0bba5a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:31:05.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.988+0000 7ff421400700 1 Processor -- start 2024-03-29T11:31:05.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.988+0000 7ff421400700 1 -- start start 2024-03-29T11:31:05.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.989+0000 7ff421400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff41c15d9d0 0x7ff41c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.989+0000 7ff421400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff41c0d2540 0x7ff41c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.989+0000 7ff421400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c161420 con 0x7ff41c0d16a0 2024-03-29T11:31:05.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.989+0000 7ff421400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41c15edc0 con 0x7ff41c0d2540 2024-03-29T11:31:05.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.989+0000 7ff421400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41c161700 con 0x7ff41c15d9d0 2024-03-29T11:31:05.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.989+0000 7ff419e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff41c0d16a0 0x7ff41c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45732/0 (socket says 172.21.15.12:45732) 2024-03-29T11:31:05.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.989+0000 7ff419e00700 1 -- 172.21.15.12:0/3324923915 learned_addr learned my addr 172.21.15.12:0/3324923915 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:05.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.989+0000 7ff41b200700 1 --2- 172.21.15.12:0/3324923915 >> v2:172.21.15.12:3300/0 conn(0x7ff41c0d2540 0x7ff41c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.989+0000 7ff41a800700 1 --2- 172.21.15.12:0/3324923915 >> v2:172.21.15.159:3300/0 conn(0x7ff41c15d9d0 0x7ff41c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.990+0000 7ff419400700 1 -- 172.21.15.12:0/3324923915 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658997362 0 0) 0x7ff41c161420 con 0x7ff41c0d16a0 2024-03-29T11:31:05.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.990+0000 7ff419400700 1 -- 172.21.15.12:0/3324923915 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4000035b0 con 0x7ff41c0d16a0 2024-03-29T11:31:05.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.990+0000 7ff41b200700 1 -- 172.21.15.12:0/3324923915 >> v2:172.21.15.159:3300/0 conn(0x7ff41c15d9d0 msgr2=0x7ff41c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:05.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.990+0000 7ff41b200700 1 --2- 172.21.15.12:0/3324923915 >> v2:172.21.15.159:3300/0 conn(0x7ff41c15d9d0 0x7ff41c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.990+0000 7ff41b200700 1 -- 172.21.15.12:0/3324923915 >> v1:172.21.15.12:6789/0 conn(0x7ff41c0d16a0 legacy=0x7ff41c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.990+0000 7ff41b200700 1 -- 172.21.15.12:0/3324923915 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c1619e0 con 0x7ff41c0d2540 2024-03-29T11:31:05.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.990+0000 7ff41b200700 1 --2- 172.21.15.12:0/3324923915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c0d2540 0x7ff41c0ce360 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7ff41c0d0ff0 tx=0x7ff41400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7280d2990502e8ec server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:05.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.990+0000 7ff419400700 1 -- 172.21.15.12:0/3324923915 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41400e070 con 0x7ff41c0d2540 2024-03-29T11:31:05.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.991+0000 7ff419400700 1 -- 172.21.15.12:0/3324923915 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41400bb20 con 0x7ff41c0d2540 2024-03-29T11:31:05.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.991+0000 7ff419400700 1 -- 172.21.15.12:0/3324923915 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414012430 con 0x7ff41c0d2540 2024-03-29T11:31:05.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.991+0000 7ff421400700 1 -- 172.21.15.12:0/3324923915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c0d2540 msgr2=0x7ff41c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.991+0000 7ff421400700 1 --2- 172.21.15.12:0/3324923915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c0d2540 0x7ff41c0ce360 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7ff41c0d0ff0 tx=0x7ff41400b330 comp rx=0 tx=0).stop 2024-03-29T11:31:05.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.991+0000 7ff421400700 1 -- 172.21.15.12:0/3324923915 shutdown_connections 2024-03-29T11:31:05.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.991+0000 7ff421400700 1 --2- 172.21.15.12:0/3324923915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c0d2540 0x7ff41c0ce360 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.991+0000 7ff421400700 1 --2- 172.21.15.12:0/3324923915 >> v2:172.21.15.159:3300/0 conn(0x7ff41c15d9d0 0x7ff41c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.991+0000 7ff421400700 1 -- 172.21.15.12:0/3324923915 >> 172.21.15.12:0/3324923915 conn(0x7ff41c08aff0 msgr2=0x7ff41c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:05.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.991+0000 7ff421400700 1 -- 172.21.15.12:0/3324923915 shutdown_connections 2024-03-29T11:31:05.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.991+0000 7ff421400700 1 -- 172.21.15.12:0/3324923915 wait complete. 2024-03-29T11:31:05.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.993+0000 7ff421400700 1 Processor -- start 2024-03-29T11:31:05.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.993+0000 7ff421400700 1 -- start start 2024-03-29T11:31:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.994+0000 7ff421400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff41c0d16a0 0x7ff41c0b7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.994+0000 7ff421400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff41c0d2540 0x7ff41c1f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.994+0000 7ff421400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c15d9d0 0x7ff41c1f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.994+0000 7ff421400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c1f77a0 con 0x7ff41c15d9d0 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.994+0000 7ff421400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0b7f80 con 0x7ff41c0d2540 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.994+0000 7ff421400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0caae0 con 0x7ff41c0d16a0 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff41b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c15d9d0 0x7ff41c1f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff41a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff41c0d16a0 0x7ff41c0b7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff41b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c15d9d0 0x7ff41c1f33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39416/0 (socket says 172.21.15.12:39416) 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff41b200700 1 -- 172.21.15.12:0/96905171 learned_addr learned my addr 172.21.15.12:0/96905171 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff419e00700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff41c0d2540 0x7ff41c1f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff41b200700 1 -- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff41c0d16a0 msgr2=0x7ff41c0b7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff41b200700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff41c0d16a0 0x7ff41c0b7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff41b200700 1 -- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff41c0d2540 msgr2=0x7ff41c1f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:05.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff41b200700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff41c0d2540 0x7ff41c1f6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:05.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff41b200700 1 -- 172.21.15.12:0/96905171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff414009a70 con 0x7ff41c15d9d0 2024-03-29T11:31:05.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.995+0000 7ff41b200700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c15d9d0 0x7ff41c1f33b0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7ff414014040 tx=0x7ff414004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:05.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.996+0000 7ff412a00700 1 -- 172.21.15.12:0/96905171 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4140129f0 con 0x7ff41c15d9d0 2024-03-29T11:31:05.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.996+0000 7ff412a00700 1 -- 172.21.15.12:0/96905171 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff414020070 con 0x7ff41c15d9d0 2024-03-29T11:31:05.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.996+0000 7ff412a00700 1 -- 172.21.15.12:0/96905171 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414012d90 con 0x7ff41c15d9d0 2024-03-29T11:31:05.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.996+0000 7ff421400700 1 -- 172.21.15.12:0/96905171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c0cadc0 con 0x7ff41c15d9d0 2024-03-29T11:31:05.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.996+0000 7ff421400700 1 -- 172.21.15.12:0/96905171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41c0cb380 con 0x7ff41c15d9d0 2024-03-29T11:31:05.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.997+0000 7ff421400700 1 -- 172.21.15.12:0/96905171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41c0d1ca0 con 0x7ff41c15d9d0 2024-03-29T11:31:05.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.998+0000 7ff412a00700 1 -- 172.21.15.12:0/96905171 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7ff414004040 con 0x7ff41c15d9d0 2024-03-29T11:31:05.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.999+0000 7ff412a00700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff3fc06adb0 0x7ff3fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:05.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.999+0000 7ff412a00700 1 -- 172.21.15.12:0/96905171 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7ff414022070 con 0x7ff41c15d9d0 2024-03-29T11:31:05.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.999+0000 7ff41a800700 1 -- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff3fc06adb0 msgr2=0x7ff3fc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:05.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:05.999+0000 7ff41a800700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff3fc06adb0 0x7ff3fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:05.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.003+0000 7ff412a00700 1 -- 172.21.15.12:0/96905171 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff414056160 con 0x7ff41c15d9d0 2024-03-29T11:31:06.082 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:06.089+0000 7ff414758240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:31:06.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.200+0000 7ff41a800700 1 -- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff3fc06adb0 msgr2=0x7ff3fc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:06.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.200+0000 7ff41a800700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff3fc06adb0 0x7ff3fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:06.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.222+0000 7ff421400700 1 -- 172.21.15.12:0/96905171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff41c067720 con 0x7ff41c15d9d0 2024-03-29T11:31:06.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.223+0000 7ff412a00700 1 -- 172.21.15.12:0/96905171 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v213) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff414059780 con 0x7ff41c15d9d0 2024-03-29T11:31:06.216 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:31:06.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.225+0000 7ff421400700 1 -- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff3fc06adb0 msgr2=0x7ff3fc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:06.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.225+0000 7ff421400700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff3fc06adb0 0x7ff3fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.225+0000 7ff421400700 1 -- 172.21.15.12:0/96905171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c15d9d0 msgr2=0x7ff41c1f33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.225+0000 7ff421400700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c15d9d0 0x7ff41c1f33b0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7ff414014040 tx=0x7ff414004ad0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.226+0000 7ff421400700 1 -- 172.21.15.12:0/96905171 shutdown_connections 2024-03-29T11:31:06.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.226+0000 7ff421400700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff3fc06adb0 0x7ff3fc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.226+0000 7ff421400700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff41c15d9d0 0x7ff41c1f33b0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.226+0000 7ff421400700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff41c0d16a0 0x7ff41c0b7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.226+0000 7ff421400700 1 --2- 172.21.15.12:0/96905171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff41c0d2540 0x7ff41c1f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.226+0000 7ff421400700 1 -- 172.21.15.12:0/96905171 >> 172.21.15.12:0/96905171 conn(0x7ff41c08aff0 msgr2=0x7ff41c0b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:06.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.226+0000 7ff421400700 1 -- 172.21.15.12:0/96905171 shutdown_connections 2024-03-29T11:31:06.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.226+0000 7ff421400700 1 -- 172.21.15.12:0/96905171 wait complete. 2024-03-29T11:31:06.237 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:31:06.416 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:06.422+0000 7fb0bba5a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:31:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.496+0000 7fc408a00700 1 Processor -- start 2024-03-29T11:31:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.496+0000 7fc408a00700 1 -- start start 2024-03-29T11:31:06.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.498+0000 7fc408a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc404160d70 0x7fc4040b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.498+0000 7fc408a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc404161150 con 0x7fc40406a790 2024-03-29T11:31:06.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.498+0000 7fc408a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40406dba0 con 0x7fc40415ef70 2024-03-29T11:31:06.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.498+0000 7fc408a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4040b4cd0 con 0x7fc404160d70 2024-03-29T11:31:06.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.498+0000 7fc401e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc404160d70 0x7fc4040b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:06.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.498+0000 7fc402800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc40406a790 0x7fc40406e080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45740/0 (socket says 172.21.15.12:45740) 2024-03-29T11:31:06.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.498+0000 7fc401e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc404160d70 0x7fc4040b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39430/0 (socket says 172.21.15.12:39430) 2024-03-29T11:31:06.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.498+0000 7fc402800700 1 -- 172.21.15.12:0/319806705 learned_addr learned my addr 172.21.15.12:0/319806705 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:06.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.499+0000 7fc401e00700 1 -- 172.21.15.12:0/319806705 >> v1:172.21.15.159:6789/0 conn(0x7fc40415ef70 legacy=0x7fc40415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.499+0000 7fc401e00700 1 -- 172.21.15.12:0/319806705 >> v1:172.21.15.12:6789/0 conn(0x7fc40406a790 legacy=0x7fc40406e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.499+0000 7fc401e00700 1 -- 172.21.15.12:0/319806705 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4040b4ee0 con 0x7fc404160d70 2024-03-29T11:31:06.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.499+0000 7fc401e00700 1 --2- 172.21.15.12:0/319806705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc404160d70 0x7fc4040b3200 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009840 tx=0x7fc3f4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373b7c4367cafd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:06.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.499+0000 7fc400a00700 1 -- 172.21.15.12:0/319806705 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4011830 con 0x7fc404160d70 2024-03-29T11:31:06.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.499+0000 7fc400a00700 1 -- 172.21.15.12:0/319806705 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f4011990 con 0x7fc404160d70 2024-03-29T11:31:06.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.499+0000 7fc400a00700 1 -- 172.21.15.12:0/319806705 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4011d20 con 0x7fc404160d70 2024-03-29T11:31:06.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.500+0000 7fc408a00700 1 -- 172.21.15.12:0/319806705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc404160d70 msgr2=0x7fc4040b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.500+0000 7fc408a00700 1 --2- 172.21.15.12:0/319806705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc404160d70 0x7fc4040b3200 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009840 tx=0x7fc3f4009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:06.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.500+0000 7fc408a00700 1 -- 172.21.15.12:0/319806705 shutdown_connections 2024-03-29T11:31:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.500+0000 7fc408a00700 1 --2- 172.21.15.12:0/319806705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc404160d70 0x7fc4040b3200 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.500+0000 7fc408a00700 1 -- 172.21.15.12:0/319806705 >> 172.21.15.12:0/319806705 conn(0x7fc4040b5720 msgr2=0x7fc4040b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.500+0000 7fc408a00700 1 -- 172.21.15.12:0/319806705 shutdown_connections 2024-03-29T11:31:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.500+0000 7fc408a00700 1 -- 172.21.15.12:0/319806705 wait complete. 2024-03-29T11:31:06.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.502+0000 7fc408a00700 1 Processor -- start 2024-03-29T11:31:06.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.503+0000 7fc408a00700 1 -- start start 2024-03-29T11:31:06.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc408a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc40406a790 0x7fc4041484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc408a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40415ef70 0x7fc40414aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc408a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc404160d70 0x7fc404149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc408a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4041489e0 con 0x7fc40415ef70 2024-03-29T11:31:06.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc408a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40414b430 con 0x7fc40406a790 2024-03-29T11:31:06.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc408a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc404148c40 con 0x7fc404160d70 2024-03-29T11:31:06.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc401400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40415ef70 0x7fc40414aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc401e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc40406a790 0x7fc4041484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc401e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc40406a790 0x7fc4041484a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:60376/0 (socket says 172.21.15.12:60376) 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc401e00700 1 -- 172.21.15.12:0/1799645007 learned_addr learned my addr 172.21.15.12:0/1799645007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.504+0000 7fc402800700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc404160d70 0x7fc404149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc401400700 1 -- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc404160d70 msgr2=0x7fc404149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc401400700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc404160d70 0x7fc404149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc401400700 1 -- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc40406a790 msgr2=0x7fc4041484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc401400700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc40406a790 0x7fc4041484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc401400700 1 -- 172.21.15.12:0/1799645007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40094a0 con 0x7fc40415ef70 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc402800700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc404160d70 0x7fc404149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc401e00700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc40406a790 0x7fc4041484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc401400700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40415ef70 0x7fc40414aef0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fc40415f3b0 tx=0x7fc3f000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc3fa000700 1 -- 172.21.15.12:0/1799645007 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00052d0 con 0x7fc40415ef70 2024-03-29T11:31:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc3fa000700 1 -- 172.21.15.12:0/1799645007 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f0005430 con 0x7fc40415ef70 2024-03-29T11:31:06.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc408a00700 1 -- 172.21.15.12:0/1799645007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc404148f20 con 0x7fc40415ef70 2024-03-29T11:31:06.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.505+0000 7fc3fa000700 1 -- 172.21.15.12:0/1799645007 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0003880 con 0x7fc40415ef70 2024-03-29T11:31:06.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.506+0000 7fc408a00700 1 -- 172.21.15.12:0/1799645007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4041494e0 con 0x7fc40415ef70 2024-03-29T11:31:06.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.507+0000 7fc408a00700 1 -- 172.21.15.12:0/1799645007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc404067720 con 0x7fc40415ef70 2024-03-29T11:31:06.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.508+0000 7fc3fa000700 1 -- 172.21.15.12:0/1799645007 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fc3f00039e0 con 0x7fc40415ef70 2024-03-29T11:31:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.509+0000 7fc3fa000700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc3e406adb0 0x7fc3e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.509+0000 7fc3fa000700 1 -- 172.21.15.12:0/1799645007 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fc3f0088800 con 0x7fc40415ef70 2024-03-29T11:31:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.509+0000 7fc401e00700 1 -- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc3e406adb0 msgr2=0x7fc3e406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.509+0000 7fc401e00700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc3e406adb0 0x7fc3e406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.513+0000 7fc3fa000700 1 -- 172.21.15.12:0/1799645007 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc3f00537f0 con 0x7fc40415ef70 2024-03-29T11:31:06.579 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:06.586+0000 7ff414758240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:31:06.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.710+0000 7fc401e00700 1 -- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc3e406adb0 msgr2=0x7fc3e406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:06.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.710+0000 7fc401e00700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc3e406adb0 0x7fc3e406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:06.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.727+0000 7fc408a00700 1 -- 172.21.15.12:0/1799645007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc40409a670 con 0x7fc40415ef70 2024-03-29T11:31:06.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.727+0000 7fc3fa000700 1 -- 172.21.15.12:0/1799645007 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v213) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc3f0017020 con 0x7fc40415ef70 2024-03-29T11:31:06.721 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:31:06.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.729+0000 7fc408a00700 1 -- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc3e406adb0 msgr2=0x7fc3e406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.729+0000 7fc408a00700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc3e406adb0 0x7fc3e406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.729+0000 7fc408a00700 1 -- 172.21.15.12:0/1799645007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40415ef70 msgr2=0x7fc40414aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.729+0000 7fc408a00700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40415ef70 0x7fc40414aef0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fc40415f3b0 tx=0x7fc3f000c7b0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.730+0000 7fc408a00700 1 -- 172.21.15.12:0/1799645007 shutdown_connections 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.730+0000 7fc408a00700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc3e406adb0 0x7fc3e406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.730+0000 7fc408a00700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc404160d70 0x7fc404149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.730+0000 7fc408a00700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc40406a790 0x7fc4041484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.730+0000 7fc408a00700 1 --2- 172.21.15.12:0/1799645007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40415ef70 0x7fc40414aef0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.730+0000 7fc408a00700 1 -- 172.21.15.12:0/1799645007 >> 172.21.15.12:0/1799645007 conn(0x7fc4040b5720 msgr2=0x7fc40409c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.730+0000 7fc408a00700 1 -- 172.21.15.12:0/1799645007 shutdown_connections 2024-03-29T11:31:06.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.730+0000 7fc408a00700 1 -- 172.21.15.12:0/1799645007 wait complete. 2024-03-29T11:31:06.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T11:31:06.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.949+0000 7f54e7200700 1 Processor -- start 2024-03-29T11:31:06.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.949+0000 7f54e7200700 1 -- start start 2024-03-29T11:31:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.950+0000 7f54e7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54e00cd020 0x7f54e00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.950+0000 7f54e7200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f54e00b2070 0x7f54e00b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.950+0000 7f54e7200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e00cdae0 con 0x7f54e00cbb30 2024-03-29T11:31:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.950+0000 7f54e7200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e00cddc0 con 0x7f54e00b2070 2024-03-29T11:31:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.950+0000 7f54e7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e00ce0a0 con 0x7f54e00cd020 2024-03-29T11:31:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.950+0000 7f54e5200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f54e00b2070 0x7f54e00b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.950+0000 7f54dfe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f54e00cbb30 0x7f54e00cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45748/0 (socket says 172.21.15.12:45748) 2024-03-29T11:31:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.950+0000 7f54dfe00700 1 -- 172.21.15.12:0/2953210900 learned_addr learned my addr 172.21.15.12:0/2953210900 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.950+0000 7f54df400700 1 --2- 172.21.15.12:0/2953210900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54e00cd020 0x7f54e00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:06.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.951+0000 7f54e5200700 1 -- 172.21.15.12:0/2953210900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54e00cd020 msgr2=0x7f54e00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:06.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.951+0000 7f54e5200700 1 --2- 172.21.15.12:0/2953210900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54e00cd020 0x7f54e00cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.951+0000 7f54e5200700 1 -- 172.21.15.12:0/2953210900 >> v1:172.21.15.12:6789/0 conn(0x7f54e00cbb30 legacy=0x7f54e00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.951+0000 7f54e5200700 1 -- 172.21.15.12:0/2953210900 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e006dc20 con 0x7f54e00b2070 2024-03-29T11:31:06.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.951+0000 7f54dea00700 1 -- 172.21.15.12:0/2953210900 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642785474 0 0) 0x7f54e00cdae0 con 0x7f54e00cbb30 2024-03-29T11:31:06.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.951+0000 7f54e5200700 1 --2- 172.21.15.12:0/2953210900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00b2070 0x7f54e00b2500 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009d90 tx=0x7f54d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a94e0299dd86cc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:06.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.951+0000 7f54dea00700 1 -- 172.21.15.12:0/2953210900 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d800e070 con 0x7f54e00b2070 2024-03-29T11:31:06.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.951+0000 7f54dea00700 1 -- 172.21.15.12:0/2953210900 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d800bb20 con 0x7f54e00b2070 2024-03-29T11:31:06.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.951+0000 7f54dea00700 1 -- 172.21.15.12:0/2953210900 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d8012430 con 0x7f54e00b2070 2024-03-29T11:31:06.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.952+0000 7f54e7200700 1 -- 172.21.15.12:0/2953210900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00b2070 msgr2=0x7f54e00b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.952+0000 7f54e7200700 1 --2- 172.21.15.12:0/2953210900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00b2070 0x7f54e00b2500 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009d90 tx=0x7f54d800b330 comp rx=0 tx=0).stop 2024-03-29T11:31:06.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.952+0000 7f54e7200700 1 -- 172.21.15.12:0/2953210900 shutdown_connections 2024-03-29T11:31:06.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.952+0000 7f54e7200700 1 --2- 172.21.15.12:0/2953210900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00b2070 0x7f54e00b2500 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.952+0000 7f54e7200700 1 --2- 172.21.15.12:0/2953210900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54e00cd020 0x7f54e00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.952+0000 7f54e7200700 1 -- 172.21.15.12:0/2953210900 >> 172.21.15.12:0/2953210900 conn(0x7f54e00b0d70 msgr2=0x7f54e00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:06.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.952+0000 7f54e7200700 1 -- 172.21.15.12:0/2953210900 shutdown_connections 2024-03-29T11:31:06.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.952+0000 7f54e7200700 1 -- 172.21.15.12:0/2953210900 wait complete. 2024-03-29T11:31:06.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.954+0000 7f54e7200700 1 Processor -- start 2024-03-29T11:31:06.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.954+0000 7f54e7200700 1 -- start start 2024-03-29T11:31:06.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54e7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54e00b2070 0x7f54e00ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54e7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00cbb30 0x7f54e015de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54e7200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54e00cd020 0x7f54e00ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54e7200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e00ad520 con 0x7f54e00cbb30 2024-03-29T11:31:06.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54e7200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e00ac8a0 con 0x7f54e00cd020 2024-03-29T11:31:06.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54e7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e00acb80 con 0x7f54e00b2070 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54df400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00cbb30 0x7f54e015de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54df400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00cbb30 0x7f54e015de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39448/0 (socket says 172.21.15.12:39448) 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54df400700 1 -- 172.21.15.12:0/2776106892 learned_addr learned my addr 172.21.15.12:0/2776106892 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54dfe00700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54e00b2070 0x7f54e00ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.955+0000 7f54e5200700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54e00cd020 0x7f54e00ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54df400700 1 -- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54e00b2070 msgr2=0x7f54e00ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54df400700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54e00b2070 0x7f54e00ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54df400700 1 -- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54e00cd020 msgr2=0x7f54e00ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54df400700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54e00cd020 0x7f54e00ad940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54df400700 1 -- 172.21.15.12:0/2776106892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d8009a70 con 0x7f54e00cbb30 2024-03-29T11:31:06.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54df400700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00cbb30 0x7f54e015de70 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00e870 tx=0x7f54cc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:06.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54d7e00700 1 -- 172.21.15.12:0/2776106892 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc009590 con 0x7f54e00cbb30 2024-03-29T11:31:06.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54d7e00700 1 -- 172.21.15.12:0/2776106892 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc013680 con 0x7f54e00cbb30 2024-03-29T11:31:06.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54d7e00700 1 -- 172.21.15.12:0/2776106892 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc0139f0 con 0x7f54e00cbb30 2024-03-29T11:31:06.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54e7200700 1 -- 172.21.15.12:0/2776106892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e00ace60 con 0x7f54e00cbb30 2024-03-29T11:31:06.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.956+0000 7f54e7200700 1 -- 172.21.15.12:0/2776106892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e00ac590 con 0x7f54e00cbb30 2024-03-29T11:31:06.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.957+0000 7f54e7200700 1 -- 172.21.15.12:0/2776106892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e00ad660 con 0x7f54e00cbb30 2024-03-29T11:31:06.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.958+0000 7f54d7e00700 1 -- 172.21.15.12:0/2776106892 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f54cc013b50 con 0x7f54e00cbb30 2024-03-29T11:31:06.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.959+0000 7f54d7e00700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f54c006adb0 0x7f54c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:06.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.959+0000 7f54d7e00700 1 -- 172.21.15.12:0/2776106892 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f54cc0894f0 con 0x7f54e00cbb30 2024-03-29T11:31:06.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.960+0000 7f54dfe00700 1 -- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f54c006adb0 msgr2=0x7f54c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:06.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.960+0000 7f54dfe00700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f54c006adb0 0x7f54c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:06.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:06.964+0000 7f54d7e00700 1 -- 172.21.15.12:0/2776106892 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54e00ad660 con 0x7f54e00cbb30 2024-03-29T11:31:07.124 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:07.130+0000 7fb0bba5a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:31:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.161+0000 7f54dfe00700 1 -- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f54c006adb0 msgr2=0x7f54c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.161+0000 7f54dfe00700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f54c006adb0 0x7f54c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.197+0000 7f54e7200700 1 -- 172.21.15.12:0/2776106892 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f54e00ad660 con 0x7f54e00cbb30 2024-03-29T11:31:07.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.197+0000 7f54d7e00700 1 -- 172.21.15.12:0/2776106892 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v213) v1 ==== 80+0+465 (secure 0 0 0) 0x7f54e00ad660 con 0x7f54e00cbb30 2024-03-29T11:31:07.191 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2122872060 2024-03-30T11:29:07.287915+0000 2024-03-29T11:31:07.191 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2280623603 2024-03-30T11:29:07.287915+0000 2024-03-29T11:31:07.191 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3600559047 2024-03-30T11:29:07.287915+0000 2024-03-29T11:31:07.191 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/62646 2024-03-30T11:29:07.287915+0000 2024-03-29T11:31:07.191 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/62646 2024-03-30T11:29:07.287915+0000 2024-03-29T11:31:07.191 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/4050963121 2024-03-30T11:29:07.287915+0000 2024-03-29T11:31:07.191 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/67497648 2024-03-30T11:29:07.287915+0000 2024-03-29T11:31:07.191 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/343093900 2024-03-30T11:29:07.287915+0000 2024-03-29T11:31:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 -- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f54c006adb0 msgr2=0x7f54c006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f54c006adb0 0x7f54c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 -- 172.21.15.12:0/2776106892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00cbb30 msgr2=0x7f54e015de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00cbb30 0x7f54e015de70 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00e870 tx=0x7f54cc00eb80 comp rx=0 tx=0).stop 2024-03-29T11:31:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 -- 172.21.15.12:0/2776106892 shutdown_connections 2024-03-29T11:31:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f54c006adb0 0x7f54c006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54e00cd020 0x7f54e00ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54e00b2070 0x7f54e00ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 --2- 172.21.15.12:0/2776106892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54e00cbb30 0x7f54e015de70 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 -- 172.21.15.12:0/2776106892 >> 172.21.15.12:0/2776106892 conn(0x7f54e00b0d70 msgr2=0x7f54e00b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:07.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 -- 172.21.15.12:0/2776106892 shutdown_connections 2024-03-29T11:31:07.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.200+0000 7f54e7200700 1 -- 172.21.15.12:0/2776106892 wait complete. 2024-03-29T11:31:07.195 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T11:31:07.212 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T11:31:07.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:07.213 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T11:31:07.215 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:07.221+0000 7fb0bba5a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:31:07.289 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:31:07.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:07.289 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T11:31:07.334 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:31:07.334 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:31:07.354 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:07.360+0000 7ff414758240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:31:07.377 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:07.383+0000 7fb0bba5a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:31:07.379 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:31:07.379 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:31:07.379 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:31:07.426 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:31:07.427 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:31:07.427 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:31:07.456 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:07.462+0000 7ff414758240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:31:07.472 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:31:07.473 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:31:07.597 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:07.603+0000 7fb0bba5a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:31:07.629 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:07.636+0000 7ff414758240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:31:07.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.679+0000 7fa687400700 1 Processor -- start 2024-03-29T11:31:07.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.680+0000 7fa687400700 1 -- start start 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.681+0000 7fa687400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6800d05e0 0x7fa6800d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.681+0000 7fa687400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6800b6990 0x7fa6800b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.681+0000 7fa687400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800d0f80 con 0x7fa68015da50 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.681+0000 7fa687400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6800b95d0 con 0x7fa6800b6990 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.681+0000 7fa687400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68015de30 con 0x7fa6800d05e0 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.681+0000 7fa685400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6800b6990 0x7fa6800b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.681+0000 7fa685400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6800b6990 0x7fa6800b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39450/0 (socket says 172.21.15.12:39450) 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.681+0000 7fa685400700 1 -- 172.21.15.12:0/1366866326 learned_addr learned my addr 172.21.15.12:0/1366866326 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.681+0000 7fa67fe00700 1 --2- 172.21.15.12:0/1366866326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6800d05e0 0x7fa6800d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.682+0000 7fa685400700 1 -- 172.21.15.12:0/1366866326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6800d05e0 msgr2=0x7fa6800d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:07.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.682+0000 7fa685400700 1 --2- 172.21.15.12:0/1366866326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6800d05e0 0x7fa6800d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.682+0000 7fa685400700 1 -- 172.21.15.12:0/1366866326 >> v1:172.21.15.159:6789/0 conn(0x7fa68015da50 legacy=0x7fa6800ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:07.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.682+0000 7fa685400700 1 -- 172.21.15.12:0/1366866326 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6800d24e0 con 0x7fa6800b6990 2024-03-29T11:31:07.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.682+0000 7fa67f400700 1 -- 172.21.15.12:0/1366866326 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627708743 0 0) 0x7fa6800d0f80 con 0x7fa68015da50 2024-03-29T11:31:07.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.682+0000 7fa685400700 1 --2- 172.21.15.12:0/1366866326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6800b6990 0x7fa6800b6e10 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fa678009e50 tx=0x7fa67800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd1fce19f1d3b1e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:07.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.682+0000 7fa67f400700 1 -- 172.21.15.12:0/1366866326 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67800e070 con 0x7fa6800b6990 2024-03-29T11:31:07.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.682+0000 7fa67f400700 1 -- 172.21.15.12:0/1366866326 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67800b910 con 0x7fa6800b6990 2024-03-29T11:31:07.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.682+0000 7fa67f400700 1 -- 172.21.15.12:0/1366866326 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67800bcc0 con 0x7fa6800b6990 2024-03-29T11:31:07.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.683+0000 7fa687400700 1 -- 172.21.15.12:0/1366866326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6800b6990 msgr2=0x7fa6800b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:07.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.683+0000 7fa687400700 1 --2- 172.21.15.12:0/1366866326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6800b6990 0x7fa6800b6e10 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fa678009e50 tx=0x7fa67800b330 comp rx=0 tx=0).stop 2024-03-29T11:31:07.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.683+0000 7fa687400700 1 -- 172.21.15.12:0/1366866326 shutdown_connections 2024-03-29T11:31:07.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.683+0000 7fa687400700 1 --2- 172.21.15.12:0/1366866326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6800b6990 0x7fa6800b6e10 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.683+0000 7fa687400700 1 --2- 172.21.15.12:0/1366866326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6800d05e0 0x7fa6800d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.683+0000 7fa687400700 1 -- 172.21.15.12:0/1366866326 >> 172.21.15.12:0/1366866326 conn(0x7fa6800666a0 msgr2=0x7fa680066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:07.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.683+0000 7fa687400700 1 -- 172.21.15.12:0/1366866326 shutdown_connections 2024-03-29T11:31:07.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.683+0000 7fa687400700 1 -- 172.21.15.12:0/1366866326 wait complete. 2024-03-29T11:31:07.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.685+0000 7fa687400700 1 Processor -- start 2024-03-29T11:31:07.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.685+0000 7fa687400700 1 -- start start 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa687400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6800b6990 0x7fa68014fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa687400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6800d05e0 0x7fa6801f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa687400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa68015da50 0x7fa6800b8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa687400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800b5e00 con 0x7fa68015da50 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa687400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6801f3e50 con 0x7fa6800d05e0 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa687400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa680151250 con 0x7fa6800b6990 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa685400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa68015da50 0x7fa6800b8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa685400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa68015da50 0x7fa6800b8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39462/0 (socket says 172.21.15.12:39462) 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa685400700 1 -- 172.21.15.12:0/3478572094 learned_addr learned my addr 172.21.15.12:0/3478572094 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa684a00700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6800b6990 0x7fa68014fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:07.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa67fe00700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6800d05e0 0x7fa6801f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa685400700 1 -- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6800b6990 msgr2=0x7fa68014fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa685400700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6800b6990 0x7fa68014fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa685400700 1 -- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6800d05e0 msgr2=0x7fa6801f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa685400700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6800d05e0 0x7fa6801f3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.686+0000 7fa685400700 1 -- 172.21.15.12:0/3478572094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa678009b30 con 0x7fa68015da50 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.687+0000 7fa685400700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa68015da50 0x7fa6800b8320 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fa67800b680 tx=0x7fa678004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.687+0000 7fa67cc00700 1 -- 172.21.15.12:0/3478572094 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6780048b0 con 0x7fa68015da50 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.687+0000 7fa67cc00700 1 -- 172.21.15.12:0/3478572094 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6780099b0 con 0x7fa68015da50 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.687+0000 7fa687400700 1 -- 172.21.15.12:0/3478572094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6801501b0 con 0x7fa68015da50 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.687+0000 7fa67cc00700 1 -- 172.21.15.12:0/3478572094 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67801c680 con 0x7fa68015da50 2024-03-29T11:31:07.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.687+0000 7fa687400700 1 -- 172.21.15.12:0/3478572094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa680150770 con 0x7fa68015da50 2024-03-29T11:31:07.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.689+0000 7fa687400700 1 -- 172.21.15.12:0/3478572094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6800d0050 con 0x7fa68015da50 2024-03-29T11:31:07.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.689+0000 7fa67cc00700 1 -- 172.21.15.12:0/3478572094 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fa67801c7e0 con 0x7fa68015da50 2024-03-29T11:31:07.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.691+0000 7fa67cc00700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa66006adb0 0x7fa66006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:07.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.691+0000 7fa67cc00700 1 -- 172.21.15.12:0/3478572094 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fa67808afe0 con 0x7fa68015da50 2024-03-29T11:31:07.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.691+0000 7fa684a00700 1 -- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa66006adb0 msgr2=0x7fa66006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:07.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.691+0000 7fa684a00700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa66006adb0 0x7fa66006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:07.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.695+0000 7fa67cc00700 1 -- 172.21.15.12:0/3478572094 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa678059800 con 0x7fa68015da50 2024-03-29T11:31:07.689 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:07.696+0000 7fb0bba5a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:31:07.787 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:07.793+0000 7fb0bba5a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:31:07.867 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:07.873+0000 7ff414758240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:31:07.885 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:07.892+0000 7fb0bba5a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:31:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.892+0000 7fa684a00700 1 -- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa66006adb0 msgr2=0x7fa66006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:07.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.892+0000 7fa684a00700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa66006adb0 0x7fa66006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:07.963 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:07.969+0000 7ff414758240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:31:07.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.977+0000 7fa687400700 1 -- 172.21.15.12:0/3478572094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa680067720 con 0x7fa68015da50 2024-03-29T11:31:07.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.978+0000 7fa67cc00700 1 -- 172.21.15.12:0/3478572094 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fa678059390 con 0x7fa68015da50 2024-03-29T11:31:07.972 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:07.972 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/395980670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":395980670},{"type":"v1","addr":"172.21.15.159:6837","nonce":395980670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":93},{"gid":10183,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/531080306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":531080306},{"type":"v1","addr":"172.21.15.159:6835","nonce":531080306}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10509,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2709565550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2709565550},{"type":"v1","addr":"172.21.15.12:6834","nonce":2709565550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:28:50.095373+0000","modified":"2024-03-29T11:28:53.584378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10512":{"gid":10512,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.12:6835/99601797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":99601797},{"type":"v1","addr":"172.21.15.12:6835","nonce":99601797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.980+0000 7fa687400700 1 -- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa66006adb0 msgr2=0x7fa66006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.980+0000 7fa687400700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa66006adb0 0x7fa66006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.980+0000 7fa687400700 1 -- 172.21.15.12:0/3478572094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa68015da50 msgr2=0x7fa6800b8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.980+0000 7fa687400700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa68015da50 0x7fa6800b8320 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fa67800b680 tx=0x7fa678004ad0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.981+0000 7fa687400700 1 -- 172.21.15.12:0/3478572094 shutdown_connections 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.981+0000 7fa687400700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa66006adb0 0x7fa66006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.981+0000 7fa687400700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa68015da50 0x7fa6800b8320 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.981+0000 7fa687400700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6800b6990 0x7fa68014fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.981+0000 7fa687400700 1 --2- 172.21.15.12:0/3478572094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6800d05e0 0x7fa6801f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.981+0000 7fa687400700 1 -- 172.21.15.12:0/3478572094 >> 172.21.15.12:0/3478572094 conn(0x7fa6800666a0 msgr2=0x7fa6800b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.981+0000 7fa687400700 1 -- 172.21.15.12:0/3478572094 shutdown_connections 2024-03-29T11:31:07.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:07.981+0000 7fa687400700 1 -- 172.21.15.12:0/3478572094 wait complete. 2024-03-29T11:31:07.976 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 98 2024-03-29T11:31:07.983 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:07.991+0000 7fb0bba5a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:31:07.994 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:08.076 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:08.083+0000 7ff414758240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:31:08.185 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:08.191+0000 7ff414758240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:31:08.195 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:08.202+0000 7fb0bba5a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:31:08.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.202+0000 7fb049000700 1 Processor -- start 2024-03-29T11:31:08.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.202+0000 7fb049000700 1 -- start start 2024-03-29T11:31:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.203+0000 7fb049000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb04415d640 0x7fb0440d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.203+0000 7fb049000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb0441619d0 0x7fb0440b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.203+0000 7fb049000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0440d30d0 con 0x7fb0440ceec0 2024-03-29T11:31:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.203+0000 7fb049000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0440b6840 con 0x7fb04415d640 2024-03-29T11:31:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.203+0000 7fb049000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0440d2880 con 0x7fb0441619d0 2024-03-29T11:31:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.203+0000 7fb041e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb04415d640 0x7fb0440d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.203+0000 7fb041e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb04415d640 0x7fb0440d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39468/0 (socket says 172.21.15.12:39468) 2024-03-29T11:31:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.203+0000 7fb041e00700 1 -- 172.21.15.12:0/3019576884 learned_addr learned my addr 172.21.15.12:0/3019576884 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.203+0000 7fb042800700 1 --2- 172.21.15.12:0/3019576884 >> v2:172.21.15.159:3300/0 conn(0x7fb0441619d0 0x7fb0440b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:08.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.204+0000 7fb041e00700 1 -- 172.21.15.12:0/3019576884 >> v2:172.21.15.159:3300/0 conn(0x7fb0441619d0 msgr2=0x7fb0440b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.204+0000 7fb041e00700 1 --2- 172.21.15.12:0/3019576884 >> v2:172.21.15.159:3300/0 conn(0x7fb0441619d0 0x7fb0440b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.204+0000 7fb041e00700 1 -- 172.21.15.12:0/3019576884 >> v1:172.21.15.159:6789/0 conn(0x7fb0440ceec0 legacy=0x7fb0441612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.204+0000 7fb041e00700 1 -- 172.21.15.12:0/3019576884 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0440d2b30 con 0x7fb04415d640 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.204+0000 7fb041e00700 1 --2- 172.21.15.12:0/3019576884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04415d640 0x7fb0440d2340 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fb0440c7020 tx=0x7fb034009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418c4ae3bce15a18 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.204+0000 7fb040a00700 1 -- 172.21.15.12:0/3019576884 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034011830 con 0x7fb04415d640 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.204+0000 7fb040a00700 1 -- 172.21.15.12:0/3019576884 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb034011990 con 0x7fb04415d640 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.204+0000 7fb040a00700 1 -- 172.21.15.12:0/3019576884 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034011d20 con 0x7fb04415d640 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.205+0000 7fb049000700 1 -- 172.21.15.12:0/3019576884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04415d640 msgr2=0x7fb0440d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.205+0000 7fb049000700 1 --2- 172.21.15.12:0/3019576884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04415d640 0x7fb0440d2340 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fb0440c7020 tx=0x7fb034009bd0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.205+0000 7fb049000700 1 -- 172.21.15.12:0/3019576884 shutdown_connections 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.205+0000 7fb049000700 1 --2- 172.21.15.12:0/3019576884 >> v2:172.21.15.159:3300/0 conn(0x7fb0441619d0 0x7fb0440b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.205+0000 7fb049000700 1 --2- 172.21.15.12:0/3019576884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04415d640 0x7fb0440d2340 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.205+0000 7fb049000700 1 -- 172.21.15.12:0/3019576884 >> 172.21.15.12:0/3019576884 conn(0x7fb04408aff0 msgr2=0x7fb0440d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.205+0000 7fb049000700 1 -- 172.21.15.12:0/3019576884 shutdown_connections 2024-03-29T11:31:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.205+0000 7fb049000700 1 -- 172.21.15.12:0/3019576884 wait complete. 2024-03-29T11:31:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.207+0000 7fb049000700 1 Processor -- start 2024-03-29T11:31:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.207+0000 7fb049000700 1 -- start start 2024-03-29T11:31:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb049000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0440ceec0 0x7fb0440c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb049000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04415d640 0x7fb0440a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb049000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0441619d0 0x7fb04409df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb049000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04409d230 con 0x7fb0441619d0 2024-03-29T11:31:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb049000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04409dad0 con 0x7fb04415d640 2024-03-29T11:31:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb049000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440c7f00 con 0x7fb0440ceec0 2024-03-29T11:31:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb042800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0441619d0 0x7fb04409df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb042800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0441619d0 0x7fb04409df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39474/0 (socket says 172.21.15.12:39474) 2024-03-29T11:31:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb042800700 1 -- 172.21.15.12:0/2706324629 learned_addr learned my addr 172.21.15.12:0/2706324629 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb041e00700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0440ceec0 0x7fb0440c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.208+0000 7fb041400700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04415d640 0x7fb0440a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb042800700 1 -- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0440ceec0 msgr2=0x7fb0440c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb042800700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0440ceec0 0x7fb0440c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb042800700 1 -- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04415d640 msgr2=0x7fb0440a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb042800700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04415d640 0x7fb0440a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb042800700 1 -- 172.21.15.12:0/2706324629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb034009550 con 0x7fb0441619d0 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb041e00700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0440ceec0 0x7fb0440c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb042800700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0441619d0 0x7fb04409df60 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00efc0 tx=0x7fb03c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb03a000700 1 -- 172.21.15.12:0/2706324629 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c009ea0 con 0x7fb0441619d0 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb03a000700 1 -- 172.21.15.12:0/2706324629 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c00f040 con 0x7fb0441619d0 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb03a000700 1 -- 172.21.15.12:0/2706324629 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c00ced0 con 0x7fb0441619d0 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb049000700 1 -- 172.21.15.12:0/2706324629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04409d480 con 0x7fb0441619d0 2024-03-29T11:31:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.209+0000 7fb049000700 1 -- 172.21.15.12:0/2706324629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0440a1230 con 0x7fb0441619d0 2024-03-29T11:31:08.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.211+0000 7fb049000700 1 -- 172.21.15.12:0/2706324629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04409dc40 con 0x7fb0441619d0 2024-03-29T11:31:08.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.212+0000 7fb03a000700 1 -- 172.21.15.12:0/2706324629 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fb03c014480 con 0x7fb0441619d0 2024-03-29T11:31:08.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.212+0000 7fb03a000700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02406adb0 0x7fb02406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.213+0000 7fb03a000700 1 -- 172.21.15.12:0/2706324629 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fb03c089fe0 con 0x7fb0441619d0 2024-03-29T11:31:08.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.213+0000 7fb041e00700 1 -- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02406adb0 msgr2=0x7fb02406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:08.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.213+0000 7fb041e00700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02406adb0 0x7fb02406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:08.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.217+0000 7fb03a000700 1 -- 172.21.15.12:0/2706324629 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb03c054d70 con 0x7fb0441619d0 2024-03-29T11:31:08.293 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:08.299+0000 7ff414758240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:31:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.414+0000 7fb041e00700 1 -- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02406adb0 msgr2=0x7fb02406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.414+0000 7fb041e00700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02406adb0 0x7fb02406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.437+0000 7fb049000700 1 -- 172.21.15.12:0/2706324629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb044067720 con 0x7fb0441619d0 2024-03-29T11:31:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.439+0000 7fb03a000700 1 -- 172.21.15.12:0/2706324629 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+16900 (secure 0 0 0) 0x7fb03c018020 con 0x7fb0441619d0 2024-03-29T11:31:08.433 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:08.433 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":213,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:04.402813+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:28:47.651679+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-03-29T11:28:49.085953+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":213,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.18","mappings":[{"from":0,"to":2}]},{"pgid":"25.1f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2122872060":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/2280623603":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/3600559047":"2024-03-30T11:29:07.287915+0000","172.21.15.159:6833/62646":"2024-03-30T11:29:07.287915+0000","172.21.15.159:6832/62646":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/4050963121":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/67497648":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/343093900":"2024-03-30T11:29:07.287915+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 -- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02406adb0 msgr2=0x7fb02406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02406adb0 0x7fb02406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 -- 172.21.15.12:0/2706324629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0441619d0 msgr2=0x7fb04409df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0441619d0 0x7fb04409df60 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00efc0 tx=0x7fb03c00c3a0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 -- 172.21.15.12:0/2706324629 shutdown_connections 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02406adb0 0x7fb02406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0441619d0 0x7fb04409df60 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0440ceec0 0x7fb0440c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 --2- 172.21.15.12:0/2706324629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04415d640 0x7fb0440a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 -- 172.21.15.12:0/2706324629 >> 172.21.15.12:0/2706324629 conn(0x7fb04408aff0 msgr2=0x7fb0440b7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 -- 172.21.15.12:0/2706324629 shutdown_connections 2024-03-29T11:31:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.442+0000 7fb049000700 1 -- 172.21.15.12:0/2706324629 wait complete. 2024-03-29T11:31:08.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:28:47.651679+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '207', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:28:49.085953+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 213, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:31:08.455 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:31:08.457 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:08.464+0000 7fb0bba5a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:31:08.523 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:08.530+0000 7ff414758240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:31:08.646 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:08.653+0000 7fb0bba5a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:31:08.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.668+0000 7f6af2a00700 1 Processor -- start 2024-03-29T11:31:08.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.668+0000 7f6af2a00700 1 -- start start 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.669+0000 7f6af2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6aec15d9d0 0x7f6aec0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.669+0000 7f6af2a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6aec0d2540 0x7f6aec0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.669+0000 7f6af2a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec161420 con 0x7f6aec0d16a0 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.669+0000 7f6af2a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aec15edc0 con 0x7f6aec15d9d0 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.669+0000 7f6af2a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aec161700 con 0x7f6aec0d2540 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.670+0000 7f6aebe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6aec15d9d0 0x7f6aec0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.670+0000 7f6aebe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6aec15d9d0 0x7f6aec0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39486/0 (socket says 172.21.15.12:39486) 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.670+0000 7f6aebe00700 1 -- 172.21.15.12:0/1759279579 learned_addr learned my addr 172.21.15.12:0/1759279579 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.670+0000 7f6af0a00700 1 --2- 172.21.15.12:0/1759279579 >> v2:172.21.15.159:3300/0 conn(0x7f6aec0d2540 0x7f6aec0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.670+0000 7f6aebe00700 1 -- 172.21.15.12:0/1759279579 >> v2:172.21.15.159:3300/0 conn(0x7f6aec0d2540 msgr2=0x7f6aec0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:08.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.670+0000 7f6aebe00700 1 --2- 172.21.15.12:0/1759279579 >> v2:172.21.15.159:3300/0 conn(0x7f6aec0d2540 0x7f6aec0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.670+0000 7f6aebe00700 1 -- 172.21.15.12:0/1759279579 >> v1:172.21.15.159:6789/0 conn(0x7f6aec0d16a0 legacy=0x7f6aec0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.670+0000 7f6aebe00700 1 -- 172.21.15.12:0/1759279579 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec1619e0 con 0x7f6aec15d9d0 2024-03-29T11:31:08.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.670+0000 7f6aebe00700 1 --2- 172.21.15.12:0/1759279579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec15d9d0 0x7f6aec0d3c30 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f6aec08ca10 tx=0x7f6adc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b552ce3fc980398 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.671+0000 7f6aeaa00700 1 -- 172.21.15.12:0/1759279579 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc011830 con 0x7f6aec15d9d0 2024-03-29T11:31:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.671+0000 7f6aeaa00700 1 -- 172.21.15.12:0/1759279579 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc011990 con 0x7f6aec15d9d0 2024-03-29T11:31:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.671+0000 7f6aeaa00700 1 -- 172.21.15.12:0/1759279579 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc011d20 con 0x7f6aec15d9d0 2024-03-29T11:31:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.671+0000 7f6af2a00700 1 -- 172.21.15.12:0/1759279579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec15d9d0 msgr2=0x7f6aec0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.671+0000 7f6af2a00700 1 --2- 172.21.15.12:0/1759279579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec15d9d0 0x7f6aec0d3c30 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f6aec08ca10 tx=0x7f6adc009bd0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.672+0000 7f6af2a00700 1 -- 172.21.15.12:0/1759279579 shutdown_connections 2024-03-29T11:31:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.672+0000 7f6af2a00700 1 --2- 172.21.15.12:0/1759279579 >> v2:172.21.15.159:3300/0 conn(0x7f6aec0d2540 0x7f6aec0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.672+0000 7f6af2a00700 1 --2- 172.21.15.12:0/1759279579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec15d9d0 0x7f6aec0d3c30 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.672+0000 7f6af2a00700 1 -- 172.21.15.12:0/1759279579 >> 172.21.15.12:0/1759279579 conn(0x7f6aec08aff0 msgr2=0x7f6aec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:08.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.672+0000 7f6af2a00700 1 -- 172.21.15.12:0/1759279579 shutdown_connections 2024-03-29T11:31:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.672+0000 7f6af2a00700 1 -- 172.21.15.12:0/1759279579 wait complete. 2024-03-29T11:31:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.674+0000 7f6af2a00700 1 Processor -- start 2024-03-29T11:31:08.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.674+0000 7f6af2a00700 1 -- start start 2024-03-29T11:31:08.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6af2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec0d16a0 0x7f6aec1557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6af2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6aec0d2540 0x7f6aec0b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6af2a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6aec15d9d0 0x7f6aec1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6af2a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0b4560 con 0x7f6aec0d16a0 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6af2a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0b4840 con 0x7f6aec15d9d0 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6af2a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0b4b20 con 0x7f6aec0d2540 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6aebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec0d16a0 0x7f6aec1557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6aebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec0d16a0 0x7f6aec1557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39502/0 (socket says 172.21.15.12:39502) 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6aebe00700 1 -- 172.21.15.12:0/1310452014 learned_addr learned my addr 172.21.15.12:0/1310452014 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6af0a00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6aec15d9d0 0x7f6aec1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.675+0000 7f6aeb400700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6aec0d2540 0x7f6aec0b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6aebe00700 1 -- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6aec0d2540 msgr2=0x7f6aec0b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6aebe00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6aec0d2540 0x7f6aec0b5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6aebe00700 1 -- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6aec15d9d0 msgr2=0x7f6aec1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6aebe00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6aec15d9d0 0x7f6aec1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6aebe00700 1 -- 172.21.15.12:0/1310452014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc009550 con 0x7f6aec0d16a0 2024-03-29T11:31:08.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6aebe00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec0d16a0 0x7f6aec1557c0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009c10 tx=0x7f6adc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6ae3e00700 1 -- 172.21.15.12:0/1310452014 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc004510 con 0x7f6aec0d16a0 2024-03-29T11:31:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6ae3e00700 1 -- 172.21.15.12:0/1310452014 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc01e070 con 0x7f6aec0d16a0 2024-03-29T11:31:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6ae3e00700 1 -- 172.21.15.12:0/1310452014 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0038e0 con 0x7f6aec0d16a0 2024-03-29T11:31:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6af2a00700 1 -- 172.21.15.12:0/1310452014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec0b4e00 con 0x7f6aec0d16a0 2024-03-29T11:31:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.676+0000 7f6af2a00700 1 -- 172.21.15.12:0/1310452014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec155d80 con 0x7f6aec0d16a0 2024-03-29T11:31:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.677+0000 7f6af2a00700 1 -- 172.21.15.12:0/1310452014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aec0d1ca0 con 0x7f6aec0d16a0 2024-03-29T11:31:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.679+0000 7f6ae3e00700 1 -- 172.21.15.12:0/1310452014 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f6adc003a40 con 0x7f6aec0d16a0 2024-03-29T11:31:08.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.680+0000 7f6ae3e00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6acc06adb0 0x7f6acc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:08.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.680+0000 7f6ae3e00700 1 -- 172.21.15.12:0/1310452014 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f6adc08a450 con 0x7f6aec0d16a0 2024-03-29T11:31:08.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.680+0000 7f6aeb400700 1 -- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6acc06adb0 msgr2=0x7f6acc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:08.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.680+0000 7f6aeb400700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6acc06adb0 0x7f6acc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.685+0000 7f6ae3e00700 1 -- 172.21.15.12:0/1310452014 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6adc055390 con 0x7f6aec0d16a0 2024-03-29T11:31:08.740 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:08.747+0000 7fb0bba5a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:08.793 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:08.800+0000 7ff414758240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.881+0000 7f6aeb400700 1 -- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6acc06adb0 msgr2=0x7f6acc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.881+0000 7f6aeb400700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6acc06adb0 0x7f6acc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.902+0000 7f6af2a00700 1 -- 172.21.15.12:0/1310452014 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6aec0b5320 con 0x7f6aec0d16a0 2024-03-29T11:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.902+0000 7f6ae3e00700 1 -- 172.21.15.12:0/1310452014 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v213) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6adc0589b0 con 0x7f6aec0d16a0 2024-03-29T11:31:08.896 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 -- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6acc06adb0 msgr2=0x7f6acc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6acc06adb0 0x7f6acc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 -- 172.21.15.12:0/1310452014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec0d16a0 msgr2=0x7f6aec1557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec0d16a0 0x7f6aec1557c0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009c10 tx=0x7f6adc004750 comp rx=0 tx=0).stop 2024-03-29T11:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 -- 172.21.15.12:0/1310452014 shutdown_connections 2024-03-29T11:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6acc06adb0 0x7f6acc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6aec15d9d0 0x7f6aec1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6aec0d16a0 0x7f6aec1557c0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 --2- 172.21.15.12:0/1310452014 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6aec0d2540 0x7f6aec0b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 -- 172.21.15.12:0/1310452014 >> 172.21.15.12:0/1310452014 conn(0x7f6aec08aff0 msgr2=0x7f6aec0d1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.905+0000 7f6af2a00700 1 -- 172.21.15.12:0/1310452014 shutdown_connections 2024-03-29T11:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:08.906+0000 7f6af2a00700 1 -- 172.21.15.12:0/1310452014 wait complete. 2024-03-29T11:31:08.917 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:31:09.004 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:09.011+0000 7ff414758240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:31:09.104 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:09.110+0000 7ff414758240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:09.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.128+0000 7ff55d000700 1 Processor -- start 2024-03-29T11:31:09.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.128+0000 7ff55d000700 1 -- start start 2024-03-29T11:31:09.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.130+0000 7ff55d000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff55806fa20 0x7ff55815e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.130+0000 7ff55d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 0x7ff55815d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.130+0000 7ff55d000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55815eda0 con 0x7ff5580cd5c0 2024-03-29T11:31:09.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.130+0000 7ff55d000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff558099dd0 con 0x7ff55806fa20 2024-03-29T11:31:09.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.130+0000 7ff55d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55809a0b0 con 0x7ff55815cfe0 2024-03-29T11:31:09.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff555e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff5580cd5c0 0x7ff55806f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34616/0 (socket says 172.21.15.12:34616) 2024-03-29T11:31:09.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff555e00700 1 -- 172.21.15.12:0/1718436323 learned_addr learned my addr 172.21.15.12:0/1718436323 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:09.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff555400700 1 --2- 172.21.15.12:0/1718436323 >> v2:172.21.15.159:3300/0 conn(0x7ff55806fa20 0x7ff55815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:09.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff556800700 1 --2- 172.21.15.12:0/1718436323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 0x7ff55815d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff554a00700 1 -- 172.21.15.12:0/1718436323 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555670097 0 0) 0x7ff548001110 con 0x7ff5580cd5c0 2024-03-29T11:31:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff554a00700 1 -- 172.21.15.12:0/1718436323 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff53c0035b0 con 0x7ff5580cd5c0 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff555400700 1 -- 172.21.15.12:0/1718436323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 msgr2=0x7ff55815d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff555400700 1 --2- 172.21.15.12:0/1718436323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 0x7ff55815d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff555400700 1 -- 172.21.15.12:0/1718436323 >> v1:172.21.15.12:6789/0 conn(0x7ff5580cd5c0 legacy=0x7ff55806f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff555400700 1 -- 172.21.15.12:0/1718436323 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55809a390 con 0x7ff55806fa20 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.131+0000 7ff556800700 1 --2- 172.21.15.12:0/1718436323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 0x7ff55815d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.132+0000 7ff555400700 1 --2- 172.21.15.12:0/1718436323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff55806fa20 0x7ff55815e860 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff544009870 tx=0x7ff544009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60f19d0ea9377d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.132+0000 7ff554a00700 1 -- 172.21.15.12:0/1718436323 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544011830 con 0x7ff55806fa20 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.132+0000 7ff554a00700 1 -- 172.21.15.12:0/1718436323 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff544011990 con 0x7ff55806fa20 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.132+0000 7ff554a00700 1 -- 172.21.15.12:0/1718436323 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544011d20 con 0x7ff55806fa20 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.133+0000 7ff55d000700 1 -- 172.21.15.12:0/1718436323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff55806fa20 msgr2=0x7ff55815e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.133+0000 7ff55d000700 1 --2- 172.21.15.12:0/1718436323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff55806fa20 0x7ff55815e860 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff544009870 tx=0x7ff544009b80 comp rx=0 tx=0).stop 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.133+0000 7ff55d000700 1 -- 172.21.15.12:0/1718436323 shutdown_connections 2024-03-29T11:31:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.133+0000 7ff55d000700 1 --2- 172.21.15.12:0/1718436323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 0x7ff55815d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.133+0000 7ff55d000700 1 --2- 172.21.15.12:0/1718436323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff55806fa20 0x7ff55815e860 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.133+0000 7ff55d000700 1 -- 172.21.15.12:0/1718436323 >> 172.21.15.12:0/1718436323 conn(0x7ff5580d1370 msgr2=0x7ff5580d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.133+0000 7ff55d000700 1 -- 172.21.15.12:0/1718436323 shutdown_connections 2024-03-29T11:31:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.133+0000 7ff55d000700 1 -- 172.21.15.12:0/1718436323 wait complete. 2024-03-29T11:31:09.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.136+0000 7ff55d000700 1 Processor -- start 2024-03-29T11:31:09.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.136+0000 7ff55d000700 1 -- start start 2024-03-29T11:31:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.137+0000 7ff55d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff55806fa20 0x7ff5580b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.137+0000 7ff55d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5580cd5c0 0x7ff5580b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.137+0000 7ff55d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 0x7ff5581569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff55d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580b5470 con 0x7ff5580cd5c0 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff55d000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580b0b30 con 0x7ff55806fa20 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff55d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580b0e10 con 0x7ff55815cfe0 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff555400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5580cd5c0 0x7ff5580b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff555e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff55806fa20 0x7ff5580b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff555e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff55806fa20 0x7ff5580b04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44226/0 (socket says 172.21.15.12:44226) 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff555400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5580cd5c0 0x7ff5580b1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58174/0 (socket says 172.21.15.12:58174) 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff555e00700 1 -- 172.21.15.12:0/2674148058 learned_addr learned my addr 172.21.15.12:0/2674148058 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff556800700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 0x7ff5581569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff555400700 1 -- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 msgr2=0x7ff5581569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff555400700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 0x7ff5581569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff555400700 1 -- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff55806fa20 msgr2=0x7ff5580b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff555400700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff55806fa20 0x7ff5580b04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.138+0000 7ff555400700 1 -- 172.21.15.12:0/2674148058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff550009a70 con 0x7ff5580cd5c0 2024-03-29T11:31:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.139+0000 7ff555400700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5580cd5c0 0x7ff5580b1bd0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7ff544016000 tx=0x7ff544004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.139+0000 7ff54e000700 1 -- 172.21.15.12:0/2674148058 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544004450 con 0x7ff5580cd5c0 2024-03-29T11:31:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.139+0000 7ff54e000700 1 -- 172.21.15.12:0/2674148058 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54401f070 con 0x7ff5580cd5c0 2024-03-29T11:31:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.139+0000 7ff55d000700 1 -- 172.21.15.12:0/2674148058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5440094a0 con 0x7ff5580cd5c0 2024-03-29T11:31:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.139+0000 7ff54e000700 1 -- 172.21.15.12:0/2674148058 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5440038c0 con 0x7ff5580cd5c0 2024-03-29T11:31:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.139+0000 7ff55d000700 1 -- 172.21.15.12:0/2674148058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5580b13d0 con 0x7ff5580cd5c0 2024-03-29T11:31:09.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.140+0000 7ff55d000700 1 -- 172.21.15.12:0/2674148058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5580cc830 con 0x7ff5580cd5c0 2024-03-29T11:31:09.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.141+0000 7ff54e000700 1 -- 172.21.15.12:0/2674148058 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7ff544003a20 con 0x7ff5580cd5c0 2024-03-29T11:31:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.142+0000 7ff54e000700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff53806adb0 0x7ff53806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.142+0000 7ff54e000700 1 -- 172.21.15.12:0/2674148058 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7ff544021070 con 0x7ff5580cd5c0 2024-03-29T11:31:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.142+0000 7ff555e00700 1 -- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff53806adb0 msgr2=0x7ff53806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.142+0000 7ff555e00700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff53806adb0 0x7ff53806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:09.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.146+0000 7ff54e000700 1 -- 172.21.15.12:0/2674148058 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff544055ae0 con 0x7ff5580cd5c0 2024-03-29T11:31:09.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.344+0000 7ff555e00700 1 -- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff53806adb0 msgr2=0x7ff53806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:09.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.344+0000 7ff555e00700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff53806adb0 0x7ff53806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:09.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.363+0000 7ff55d000700 1 -- 172.21.15.12:0/2674148058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff558067720 con 0x7ff5580cd5c0 2024-03-29T11:31:09.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.364+0000 7ff54e000700 1 -- 172.21.15.12:0/2674148058 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v213) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff5440150a0 con 0x7ff5580cd5c0 2024-03-29T11:31:09.358 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:31:09.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 -- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff53806adb0 msgr2=0x7ff53806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:09.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff53806adb0 0x7ff53806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 -- 172.21.15.12:0/2674148058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5580cd5c0 msgr2=0x7ff5580b1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5580cd5c0 0x7ff5580b1bd0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7ff544016000 tx=0x7ff544004750 comp rx=0 tx=0).stop 2024-03-29T11:31:09.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 -- 172.21.15.12:0/2674148058 shutdown_connections 2024-03-29T11:31:09.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff53806adb0 0x7ff53806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff55815cfe0 0x7ff5581569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff55806fa20 0x7ff5580b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 --2- 172.21.15.12:0/2674148058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5580cd5c0 0x7ff5580b1bd0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 -- 172.21.15.12:0/2674148058 >> 172.21.15.12:0/2674148058 conn(0x7ff5580d1370 msgr2=0x7ff55806c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:09.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 -- 172.21.15.12:0/2674148058 shutdown_connections 2024-03-29T11:31:09.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.368+0000 7ff55d000700 1 -- 172.21.15.12:0/2674148058 wait complete. 2024-03-29T11:31:09.382 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:31:09.475 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:09.482+0000 7fb0bba5a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:31:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.592+0000 7f1821800700 1 Processor -- start 2024-03-29T11:31:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.592+0000 7f1821800700 1 -- start start 2024-03-29T11:31:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.593+0000 7f1821800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f181c1611b0 0x7f181c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.593+0000 7f1821800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c0ce270 con 0x7f181c15f4d0 2024-03-29T11:31:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.593+0000 7f1821800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c15b600 con 0x7f181c0cee10 2024-03-29T11:31:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.593+0000 7f1821800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c0d2760 con 0x7f181c1611b0 2024-03-29T11:31:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.593+0000 7f181aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f181c1611b0 0x7f181c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.594+0000 7f181a000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f181c15f4d0 0x7f181c0ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34626/0 (socket says 172.21.15.12:34626) 2024-03-29T11:31:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.594+0000 7f181a000700 1 -- 172.21.15.12:0/2651774693 learned_addr learned my addr 172.21.15.12:0/2651774693 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:09.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.594+0000 7f181aa00700 1 -- 172.21.15.12:0/2651774693 >> v1:172.21.15.159:6789/0 conn(0x7f181c0cee10 legacy=0x7f181c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.594+0000 7f181aa00700 1 -- 172.21.15.12:0/2651774693 >> v1:172.21.15.12:6789/0 conn(0x7f181c15f4d0 legacy=0x7f181c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.594+0000 7f181aa00700 1 -- 172.21.15.12:0/2651774693 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c0d2a40 con 0x7f181c1611b0 2024-03-29T11:31:09.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.594+0000 7f181aa00700 1 --2- 172.21.15.12:0/2651774693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c1611b0 0x7f181c15ef90 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f180c009840 tx=0x7f180c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a55af3935aa4a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:09.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.594+0000 7f1819600700 1 -- 172.21.15.12:0/2651774693 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c011830 con 0x7f181c1611b0 2024-03-29T11:31:09.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.595+0000 7f1819600700 1 -- 172.21.15.12:0/2651774693 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c011990 con 0x7f181c1611b0 2024-03-29T11:31:09.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.595+0000 7f1819600700 1 -- 172.21.15.12:0/2651774693 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c011d20 con 0x7f181c1611b0 2024-03-29T11:31:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.595+0000 7f1821800700 1 -- 172.21.15.12:0/2651774693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c1611b0 msgr2=0x7f181c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.595+0000 7f1821800700 1 --2- 172.21.15.12:0/2651774693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c1611b0 0x7f181c15ef90 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f180c009840 tx=0x7f180c009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.595+0000 7f1821800700 1 -- 172.21.15.12:0/2651774693 shutdown_connections 2024-03-29T11:31:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.595+0000 7f1821800700 1 --2- 172.21.15.12:0/2651774693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c1611b0 0x7f181c15ef90 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.595+0000 7f1821800700 1 -- 172.21.15.12:0/2651774693 >> 172.21.15.12:0/2651774693 conn(0x7f181c08aff0 msgr2=0x7f181c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.595+0000 7f1821800700 1 -- 172.21.15.12:0/2651774693 shutdown_connections 2024-03-29T11:31:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.595+0000 7f1821800700 1 -- 172.21.15.12:0/2651774693 wait complete. 2024-03-29T11:31:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.597+0000 7f1821800700 1 Processor -- start 2024-03-29T11:31:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.597+0000 7f1821800700 1 -- start start 2024-03-29T11:31:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f1821800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c0cee10 0x7f181c1f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f1821800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f181c15f4d0 0x7f181c0c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f1821800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f181c1611b0 0x7f181c0cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f1821800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c1f4000 con 0x7f181c0cee10 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f1821800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0cd3e0 con 0x7f181c15f4d0 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f1821800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0b7560 con 0x7f181c1611b0 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f181aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c0cee10 0x7f181c1f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f181aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c0cee10 0x7f181c1f3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58194/0 (socket says 172.21.15.12:58194) 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f181aa00700 1 -- 172.21.15.12:0/2155350034 learned_addr learned my addr 172.21.15.12:0/2155350034 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f181a000700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f181c15f4d0 0x7f181c0c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.598+0000 7f181b400700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f181c1611b0 0x7f181c0cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.599+0000 7f181aa00700 1 -- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f181c1611b0 msgr2=0x7f181c0cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.599+0000 7f181aa00700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f181c1611b0 0x7f181c0cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.599+0000 7f181aa00700 1 -- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f181c15f4d0 msgr2=0x7f181c0c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.599+0000 7f181aa00700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f181c15f4d0 0x7f181c0c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.599+0000 7f181aa00700 1 -- 172.21.15.12:0/2155350034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c0094a0 con 0x7f181c0cee10 2024-03-29T11:31:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.599+0000 7f181aa00700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c0cee10 0x7f181c1f3ac0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f180c0090b0 tx=0x7f180c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.599+0000 7f1812a00700 1 -- 172.21.15.12:0/2155350034 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c0036a0 con 0x7f181c0cee10 2024-03-29T11:31:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.599+0000 7f1812a00700 1 -- 172.21.15.12:0/2155350034 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c01f070 con 0x7f181c0cee10 2024-03-29T11:31:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.599+0000 7f1821800700 1 -- 172.21.15.12:0/2155350034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c0b7800 con 0x7f181c0cee10 2024-03-29T11:31:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.599+0000 7f1812a00700 1 -- 172.21.15.12:0/2155350034 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c003a60 con 0x7f181c0cee10 2024-03-29T11:31:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.600+0000 7f1821800700 1 -- 172.21.15.12:0/2155350034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c0b7dc0 con 0x7f181c0cee10 2024-03-29T11:31:09.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.601+0000 7f1821800700 1 -- 172.21.15.12:0/2155350034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c06c440 con 0x7f181c0cee10 2024-03-29T11:31:09.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.602+0000 7f1812a00700 1 -- 172.21.15.12:0/2155350034 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f180c003bc0 con 0x7f181c0cee10 2024-03-29T11:31:09.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.602+0000 7f1812a00700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17fc06adb0 0x7f17fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:09.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.602+0000 7f1812a00700 1 -- 172.21.15.12:0/2155350034 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f180c021070 con 0x7f181c0cee10 2024-03-29T11:31:09.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.603+0000 7f181a000700 1 -- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17fc06adb0 msgr2=0x7f17fc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:09.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.603+0000 7f181a000700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17fc06adb0 0x7f17fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:09.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.607+0000 7f1812a00700 1 -- 172.21.15.12:0/2155350034 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f180c055f80 con 0x7f181c0cee10 2024-03-29T11:31:09.686 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:09.693+0000 7fb0bba5a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:31:09.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.804+0000 7f181a000700 1 -- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17fc06adb0 msgr2=0x7f17fc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:09.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.804+0000 7f181a000700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17fc06adb0 0x7f17fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.822+0000 7f1821800700 1 -- 172.21.15.12:0/2155350034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f181c0ce6e0 con 0x7f181c0cee10 2024-03-29T11:31:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.823+0000 7f1812a00700 1 -- 172.21.15.12:0/2155350034 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v213) v1 ==== 100+0+11 (secure 0 0 0) 0x7f180c0150a0 con 0x7f181c0cee10 2024-03-29T11:31:09.817 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:31:09.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 -- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17fc06adb0 msgr2=0x7f17fc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:09.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17fc06adb0 0x7f17fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 -- 172.21.15.12:0/2155350034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c0cee10 msgr2=0x7f181c1f3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c0cee10 0x7f181c1f3ac0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f180c0090b0 tx=0x7f180c004750 comp rx=0 tx=0).stop 2024-03-29T11:31:09.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 -- 172.21.15.12:0/2155350034 shutdown_connections 2024-03-29T11:31:09.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17fc06adb0 0x7f17fc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f181c1611b0 0x7f181c0cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181c0cee10 0x7f181c1f3ac0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 --2- 172.21.15.12:0/2155350034 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f181c15f4d0 0x7f181c0c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:09.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 -- 172.21.15.12:0/2155350034 >> 172.21.15.12:0/2155350034 conn(0x7f181c08aff0 msgr2=0x7f181c159b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:09.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 -- 172.21.15.12:0/2155350034 shutdown_connections 2024-03-29T11:31:09.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:09.826+0000 7f1821800700 1 -- 172.21.15.12:0/2155350034 wait complete. 2024-03-29T11:31:09.838 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:31:09.891 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:09.898+0000 7fb0bba5a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:09.900 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:09.907+0000 7ff414758240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:31:10.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.045+0000 7fe3b3e00700 1 Processor -- start 2024-03-29T11:31:10.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.045+0000 7fe3b3e00700 1 -- start start 2024-03-29T11:31:10.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3b3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 0x7fe3ac070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:10.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3b3e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac0b2180 con 0x7fe3ac0b3af0 2024-03-29T11:31:10.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3b3e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac15b100 con 0x7fe3ac072580 2024-03-29T11:31:10.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3b3e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac15b3e0 con 0x7fe3ac0709c0 2024-03-29T11:31:10.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3b1400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe3ac0b3af0 0x7fe3ac15a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34632/0 (socket says 172.21.15.12:34632) 2024-03-29T11:31:10.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3b1400700 1 -- 172.21.15.12:0/458291863 learned_addr learned my addr 172.21.15.12:0/458291863 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:10.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3b1e00700 1 --2- 172.21.15.12:0/458291863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 0x7fe3ac070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:10.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3abe00700 1 -- 172.21.15.12:0/458291863 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274531800 0 0) 0x7fe3ac0b2180 con 0x7fe3ac0b3af0 2024-03-29T11:31:10.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3abe00700 1 -- 172.21.15.12:0/458291863 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3940035b0 con 0x7fe3ac0b3af0 2024-03-29T11:31:10.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3abe00700 1 -- 172.21.15.12:0/458291863 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144864357 0 0) 0x7fe3ac15b100 con 0x7fe3ac072580 2024-03-29T11:31:10.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.047+0000 7fe3abe00700 1 -- 172.21.15.12:0/458291863 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3ac0b2180 con 0x7fe3ac072580 2024-03-29T11:31:10.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.048+0000 7fe3b1e00700 1 -- 172.21.15.12:0/458291863 >> v1:172.21.15.159:6789/0 conn(0x7fe3ac072580 legacy=0x7fe3ac0b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.048+0000 7fe3b1e00700 1 -- 172.21.15.12:0/458291863 >> v1:172.21.15.12:6789/0 conn(0x7fe3ac0b3af0 legacy=0x7fe3ac15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.048+0000 7fe3b1e00700 1 -- 172.21.15.12:0/458291863 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac0b23f0 con 0x7fe3ac0709c0 2024-03-29T11:31:10.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.048+0000 7fe3b1e00700 1 --2- 172.21.15.12:0/458291863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 0x7fe3ac070e50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009dc0 tx=0x7fe3a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68c01947947898dc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:10.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.048+0000 7fe3abe00700 1 -- 172.21.15.12:0/458291863 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a400e070 con 0x7fe3ac0709c0 2024-03-29T11:31:10.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.048+0000 7fe3abe00700 1 -- 172.21.15.12:0/458291863 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a400bb20 con 0x7fe3ac0709c0 2024-03-29T11:31:10.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.048+0000 7fe3abe00700 1 -- 172.21.15.12:0/458291863 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4012430 con 0x7fe3ac0709c0 2024-03-29T11:31:10.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.049+0000 7fe3b3e00700 1 -- 172.21.15.12:0/458291863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 msgr2=0x7fe3ac070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.049+0000 7fe3b3e00700 1 --2- 172.21.15.12:0/458291863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 0x7fe3ac070e50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009dc0 tx=0x7fe3a400b330 comp rx=0 tx=0).stop 2024-03-29T11:31:10.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.049+0000 7fe3b3e00700 1 -- 172.21.15.12:0/458291863 shutdown_connections 2024-03-29T11:31:10.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.049+0000 7fe3b3e00700 1 --2- 172.21.15.12:0/458291863 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 0x7fe3ac070e50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.049+0000 7fe3b3e00700 1 -- 172.21.15.12:0/458291863 >> 172.21.15.12:0/458291863 conn(0x7fe3ac0cf190 msgr2=0x7fe3ac0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:10.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.049+0000 7fe3b3e00700 1 -- 172.21.15.12:0/458291863 shutdown_connections 2024-03-29T11:31:10.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.049+0000 7fe3b3e00700 1 -- 172.21.15.12:0/458291863 wait complete. 2024-03-29T11:31:10.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.051+0000 7fe3b3e00700 1 Processor -- start 2024-03-29T11:31:10.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.051+0000 7fe3b3e00700 1 -- start start 2024-03-29T11:31:10.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.052+0000 7fe3b3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 0x7fe3ac1550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:10.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.052+0000 7fe3b3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ac072580 0x7fe3ac1f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:10.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.052+0000 7fe3b3e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ac0b3af0 0x7fe3ac1f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:10.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.052+0000 7fe3b3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac142070 con 0x7fe3ac072580 2024-03-29T11:31:10.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.052+0000 7fe3b3e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac1f2ee0 con 0x7fe3ac0b3af0 2024-03-29T11:31:10.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.052+0000 7fe3b3e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac157b70 con 0x7fe3ac0709c0 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.052+0000 7fe3b0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ac072580 0x7fe3ac1f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.052+0000 7fe3b1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 0x7fe3ac1550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.052+0000 7fe3b0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ac072580 0x7fe3ac1f2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58208/0 (socket says 172.21.15.12:58208) 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.052+0000 7fe3b0a00700 1 -- 172.21.15.12:0/2197843039 learned_addr learned my addr 172.21.15.12:0/2197843039 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.053+0000 7fe3b1e00700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ac0b3af0 0x7fe3ac1f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.053+0000 7fe3b0a00700 1 -- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 msgr2=0x7fe3ac1550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.053+0000 7fe3b0a00700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 0x7fe3ac1550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.053+0000 7fe3b0a00700 1 -- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ac0b3af0 msgr2=0x7fe3ac1f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.053+0000 7fe3b0a00700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ac0b3af0 0x7fe3ac1f29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.053+0000 7fe3b0a00700 1 -- 172.21.15.12:0/2197843039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a4009a70 con 0x7fe3ac072580 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.053+0000 7fe3b0a00700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ac072580 0x7fe3ac1f2460 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fe39800c4f0 tx=0x7fe39800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.053+0000 7fe3a9600700 1 -- 172.21.15.12:0/2197843039 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980052d0 con 0x7fe3ac072580 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.053+0000 7fe3a9600700 1 -- 172.21.15.12:0/2197843039 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe398005430 con 0x7fe3ac072580 2024-03-29T11:31:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.054+0000 7fe3a9600700 1 -- 172.21.15.12:0/2197843039 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398003880 con 0x7fe3ac072580 2024-03-29T11:31:10.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.054+0000 7fe3b3e00700 1 -- 172.21.15.12:0/2197843039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ac1422f0 con 0x7fe3ac072580 2024-03-29T11:31:10.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.054+0000 7fe3b3e00700 1 -- 172.21.15.12:0/2197843039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ac1428b0 con 0x7fe3ac072580 2024-03-29T11:31:10.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.055+0000 7fe3b3e00700 1 -- 172.21.15.12:0/2197843039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ac142ad0 con 0x7fe3ac072580 2024-03-29T11:31:10.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.057+0000 7fe3a9600700 1 -- 172.21.15.12:0/2197843039 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fe3980039e0 con 0x7fe3ac072580 2024-03-29T11:31:10.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.058+0000 7fe3a9600700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe38c06adb0 0x7fe38c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:10.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.058+0000 7fe3a9600700 1 -- 172.21.15.12:0/2197843039 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fe398088a20 con 0x7fe3ac072580 2024-03-29T11:31:10.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.058+0000 7fe3b1400700 1 -- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe38c06adb0 msgr2=0x7fe38c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:10.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.058+0000 7fe3b1400700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe38c06adb0 0x7fe38c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.062+0000 7fe3a9600700 1 -- 172.21.15.12:0/2197843039 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe3980538e0 con 0x7fe3ac072580 2024-03-29T11:31:10.131 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:10.137+0000 7ff414758240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:31:10.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.259+0000 7fe3b1400700 1 -- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe38c06adb0 msgr2=0x7fe38c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:10.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.259+0000 7fe3b1400700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe38c06adb0 0x7fe38c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:10.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.325+0000 7fe3b3e00700 1 -- 172.21.15.12:0/2197843039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3ac067720 con 0x7fe3ac072580 2024-03-29T11:31:10.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.326+0000 7fe3a9600700 1 -- 172.21.15.12:0/2197843039 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fe398017090 con 0x7fe3ac072580 2024-03-29T11:31:10.319 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:10.320 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/395980670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":395980670},{"type":"v1","addr":"172.21.15.159:6837","nonce":395980670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":93},{"gid":10183,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/531080306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":531080306},{"type":"v1","addr":"172.21.15.159:6835","nonce":531080306}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10509,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2709565550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2709565550},{"type":"v1","addr":"172.21.15.12:6834","nonce":2709565550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:28:50.095373+0000","modified":"2024-03-29T11:28:53.584378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10512":{"gid":10512,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.12:6835/99601797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":99601797},{"type":"v1","addr":"172.21.15.12:6835","nonce":99601797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-29T11:31:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 -- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe38c06adb0 msgr2=0x7fe38c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe38c06adb0 0x7fe38c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 -- 172.21.15.12:0/2197843039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ac072580 msgr2=0x7fe3ac1f2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ac072580 0x7fe3ac1f2460 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fe39800c4f0 tx=0x7fe39800c800 comp rx=0 tx=0).stop 2024-03-29T11:31:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 -- 172.21.15.12:0/2197843039 shutdown_connections 2024-03-29T11:31:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe38c06adb0 0x7fe38c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ac0b3af0 0x7fe3ac1f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac0709c0 0x7fe3ac1550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 --2- 172.21.15.12:0/2197843039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3ac072580 0x7fe3ac1f2460 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 -- 172.21.15.12:0/2197843039 >> 172.21.15.12:0/2197843039 conn(0x7fe3ac0cf190 msgr2=0x7fe3ac06d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 -- 172.21.15.12:0/2197843039 shutdown_connections 2024-03-29T11:31:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.328+0000 7fe3b3e00700 1 -- 172.21.15.12:0/2197843039 wait complete. 2024-03-29T11:31:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 98 2024-03-29T11:31:10.340 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:10.356 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:10.362+0000 7ff414758240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:10.368 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:10.375+0000 7fb0bba5a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:31:10.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.553+0000 7f5ec3a00700 1 Processor -- start 2024-03-29T11:31:10.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.553+0000 7f5ec3a00700 1 -- start start 2024-03-29T11:31:10.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.554+0000 7f5ec3a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5ebc0ac7d0 0x7f5ebc166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:10.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.554+0000 7f5ec3a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc15fd50 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.554+0000 7f5ec3a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc15f140 con 0x7f5ebc0cb3f0 2024-03-29T11:31:10.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.554+0000 7f5ec3a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ebc15b9b0 con 0x7f5ebc0ac7d0 2024-03-29T11:31:10.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.554+0000 7f5ec1a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5ebc0ac7d0 0x7f5ebc166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:10.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.554+0000 7f5ec1a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5ebc0ac7d0 0x7f5ebc166430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58210/0 (socket says 172.21.15.12:58210) 2024-03-29T11:31:10.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.554+0000 7f5ec1a00700 1 -- 172.21.15.12:0/1036868062 learned_addr learned my addr 172.21.15.12:0/1036868062 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:10.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.554+0000 7f5ec1000700 1 --1- 172.21.15.12:0/1036868062 >> v1:172.21.15.12:6789/0 conn(0x7f5ebc15b5d0 0x7f5ebc0ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34636/0 (socket says 172.21.15.12:34636) 2024-03-29T11:31:10.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.555+0000 7f5ec1a00700 1 -- 172.21.15.12:0/1036868062 >> v1:172.21.15.159:6789/0 conn(0x7f5ebc0cb3f0 legacy=0x7f5ebc0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.555+0000 7f5ec1a00700 1 -- 172.21.15.12:0/1036868062 >> v1:172.21.15.12:6789/0 conn(0x7f5ebc15b5d0 legacy=0x7f5ebc0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.555+0000 7f5ec1a00700 1 -- 172.21.15.12:0/1036868062 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc15f420 con 0x7f5ebc0ac7d0 2024-03-29T11:31:10.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.555+0000 7f5ec1a00700 1 --2- 172.21.15.12:0/1036868062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0ac7d0 0x7f5ebc166430 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009e50 tx=0x7f5eb400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d4fb8dd64b1c97 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:10.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.555+0000 7f5ebb400700 1 -- 172.21.15.12:0/1036868062 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb400e070 con 0x7f5ebc0ac7d0 2024-03-29T11:31:10.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.555+0000 7f5ebb400700 1 -- 172.21.15.12:0/1036868062 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb400b9c0 con 0x7f5ebc0ac7d0 2024-03-29T11:31:10.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.555+0000 7f5ebb400700 1 -- 172.21.15.12:0/1036868062 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb400bd70 con 0x7f5ebc0ac7d0 2024-03-29T11:31:10.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.556+0000 7f5ec3a00700 1 -- 172.21.15.12:0/1036868062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0ac7d0 msgr2=0x7f5ebc166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.556+0000 7f5ec3a00700 1 --2- 172.21.15.12:0/1036868062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0ac7d0 0x7f5ebc166430 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009e50 tx=0x7f5eb400b330 comp rx=0 tx=0).stop 2024-03-29T11:31:10.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.556+0000 7f5ec3a00700 1 -- 172.21.15.12:0/1036868062 shutdown_connections 2024-03-29T11:31:10.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.556+0000 7f5ec3a00700 1 --2- 172.21.15.12:0/1036868062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0ac7d0 0x7f5ebc166430 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.556+0000 7f5ec3a00700 1 -- 172.21.15.12:0/1036868062 >> 172.21.15.12:0/1036868062 conn(0x7f5ebc0cf190 msgr2=0x7f5ebc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:10.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.556+0000 7f5ec3a00700 1 -- 172.21.15.12:0/1036868062 shutdown_connections 2024-03-29T11:31:10.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.557+0000 7f5ec3a00700 1 -- 172.21.15.12:0/1036868062 wait complete. 2024-03-29T11:31:10.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.559+0000 7f5ec3a00700 1 Processor -- start 2024-03-29T11:31:10.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.559+0000 7f5ec3a00700 1 -- start start 2024-03-29T11:31:10.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.561+0000 7f5ec3a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ebc0ac7d0 0x7f5ebc0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:10.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.561+0000 7f5ec3a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ebc15b5d0 0x7f5ebc0b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:10.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.561+0000 7f5ec3a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0b2410 0x7f5ebc0b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:10.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.561+0000 7f5ec3a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc142730 con 0x7f5ebc0b2410 2024-03-29T11:31:10.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.561+0000 7f5ec3a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc142fd0 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.561+0000 7f5ec3a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc0cbba0 con 0x7f5ebc0ac7d0 2024-03-29T11:31:10.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.561+0000 7f5ec1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0b2410 0x7f5ebc0b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:10.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.561+0000 7f5ebbe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ebc15b5d0 0x7f5ebc0b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:10.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.561+0000 7f5ec1000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ebc0ac7d0 0x7f5ebc0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:10.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.562+0000 7f5ebbe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ebc15b5d0 0x7f5ebc0b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:44264/0 (socket says 172.21.15.12:44264) 2024-03-29T11:31:10.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.562+0000 7f5ec1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0b2410 0x7f5ebc0b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58216/0 (socket says 172.21.15.12:58216) 2024-03-29T11:31:10.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.562+0000 7f5ebbe00700 1 -- 172.21.15.12:0/338754081 learned_addr learned my addr 172.21.15.12:0/338754081 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:10.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.562+0000 7f5ebbe00700 1 -- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ebc0ac7d0 msgr2=0x7f5ebc0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.562+0000 7f5ebbe00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ebc0ac7d0 0x7f5ebc0b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.562+0000 7f5ebbe00700 1 -- 172.21.15.12:0/338754081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0b2410 msgr2=0x7f5ebc0b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.562+0000 7f5ebbe00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0b2410 0x7f5ebc0b28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.562+0000 7f5ebbe00700 1 -- 172.21.15.12:0/338754081 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb4009b30 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.562+0000 7f5ec1a00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0b2410 0x7f5ebc0b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:10.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.562+0000 7f5ebbe00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ebc15b5d0 0x7f5ebc0b1ed0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800c510 tx=0x7f5ea800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:10.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.563+0000 7f5eb8c00700 1 -- 172.21.15.12:0/338754081 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea80052d0 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.563+0000 7f5eb8c00700 1 -- 172.21.15.12:0/338754081 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea8005430 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.563+0000 7f5eb8c00700 1 -- 172.21.15.12:0/338754081 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea80038c0 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.563+0000 7f5ec3a00700 1 -- 172.21.15.12:0/338754081 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ebc15a980 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.563+0000 7f5ec3a00700 1 -- 172.21.15.12:0/338754081 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ebc142c60 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.564+0000 7f5ec3a00700 1 -- 172.21.15.12:0/338754081 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ebc142e20 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.565+0000 7f5eb8c00700 1 -- 172.21.15.12:0/338754081 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f5ea8003a20 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.566+0000 7f5eb8c00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e9c06adb0 0x7f5e9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:10.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.566+0000 7f5eb8c00700 1 -- 172.21.15.12:0/338754081 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f5ea8088a40 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.566+0000 7f5ec1000700 1 -- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e9c06adb0 msgr2=0x7f5e9c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:10.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.566+0000 7f5ec1000700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e9c06adb0 0x7f5e9c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:10.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.570+0000 7f5eb8c00700 1 -- 172.21.15.12:0/338754081 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ea80538e0 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.577 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:10.584+0000 7fb0bba5a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.767+0000 7f5ec1000700 1 -- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e9c06adb0 msgr2=0x7f5e9c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.767+0000 7f5ec1000700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e9c06adb0 0x7f5e9c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.789+0000 7f5ec3a00700 1 -- 172.21.15.12:0/338754081 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ebc067720 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.791+0000 7f5eb8c00700 1 -- 172.21.15.12:0/338754081 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+16900 (secure 0 0 0) 0x7f5ea8017090 con 0x7f5ebc15b5d0 2024-03-29T11:31:10.785 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:10.785 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":213,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:04.402813+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:28:47.651679+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-03-29T11:28:49.085953+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":213,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.18","mappings":[{"from":0,"to":2}]},{"pgid":"25.1f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2122872060":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/2280623603":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/3600559047":"2024-03-30T11:29:07.287915+0000","172.21.15.159:6833/62646":"2024-03-30T11:29:07.287915+0000","172.21.15.159:6832/62646":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/4050963121":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/67497648":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/343093900":"2024-03-30T11:29:07.287915+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:10.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 -- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e9c06adb0 msgr2=0x7f5e9c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:10.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e9c06adb0 0x7f5e9c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 -- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ebc15b5d0 msgr2=0x7f5ebc0b1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ebc15b5d0 0x7f5ebc0b1ed0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800c510 tx=0x7f5ea800c820 comp rx=0 tx=0).stop 2024-03-29T11:31:10.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 -- 172.21.15.12:0/338754081 shutdown_connections 2024-03-29T11:31:10.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e9c06adb0 0x7f5e9c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ebc0b2410 0x7f5ebc0b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ebc0ac7d0 0x7f5ebc0b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 --2- 172.21.15.12:0/338754081 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ebc15b5d0 0x7f5ebc0b1ed0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:10.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 -- 172.21.15.12:0/338754081 >> 172.21.15.12:0/338754081 conn(0x7f5ebc0cf190 msgr2=0x7f5ebc15dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:10.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 -- 172.21.15.12:0/338754081 shutdown_connections 2024-03-29T11:31:10.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:10.793+0000 7f5ec3a00700 1 -- 172.21.15.12:0/338754081 wait complete. 2024-03-29T11:31:10.807 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T11:31:10.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:31:10.877 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:10.883+0000 7ff414758240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:31:11.021 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.028+0000 7fb0bba5a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:31:11.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.053+0000 7f7b4aa00700 1 Processor -- start 2024-03-29T11:31:11.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.053+0000 7f7b4aa00700 1 -- start start 2024-03-29T11:31:11.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.055+0000 7f7b4aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b440cb3f0 0x7f7b440ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:11.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.055+0000 7f7b4aa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4415fd50 con 0x7f7b4415b5d0 2024-03-29T11:31:11.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.055+0000 7f7b4aa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4415f140 con 0x7f7b440ac600 2024-03-29T11:31:11.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.055+0000 7f7b4aa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4415b9b0 con 0x7f7b440cb3f0 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.055+0000 7f7b43e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7b4415b5d0 0x7f7b440ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34648/0 (socket says 172.21.15.12:34648) 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.055+0000 7f7b43e00700 1 -- 172.21.15.12:0/2331603958 learned_addr learned my addr 172.21.15.12:0/2331603958 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.055+0000 7f7b43400700 1 --2- 172.21.15.12:0/2331603958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b440cb3f0 0x7f7b440ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.055+0000 7f7b42a00700 1 -- 172.21.15.12:0/2331603958 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203123313 0 0) 0x7f7b4415fd50 con 0x7f7b4415b5d0 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.055+0000 7f7b42a00700 1 -- 172.21.15.12:0/2331603958 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b280035b0 con 0x7f7b4415b5d0 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.056+0000 7f7b42a00700 1 -- 172.21.15.12:0/2331603958 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381756639 0 0) 0x7f7b4415f140 con 0x7f7b440ac600 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.056+0000 7f7b42a00700 1 -- 172.21.15.12:0/2331603958 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b4415fd50 con 0x7f7b440ac600 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.056+0000 7f7b43400700 1 -- 172.21.15.12:0/2331603958 >> v1:172.21.15.159:6789/0 conn(0x7f7b440ac600 legacy=0x7f7b4406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.056+0000 7f7b43400700 1 -- 172.21.15.12:0/2331603958 >> v1:172.21.15.12:6789/0 conn(0x7f7b4415b5d0 legacy=0x7f7b440ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.056+0000 7f7b43400700 1 -- 172.21.15.12:0/2331603958 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4415f420 con 0x7f7b440cb3f0 2024-03-29T11:31:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.056+0000 7f7b43400700 1 --2- 172.21.15.12:0/2331603958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b440cb3f0 0x7f7b440ac0c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009870 tx=0x7f7b30009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d30c4a0c4b95443 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:11.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.056+0000 7f7b42a00700 1 -- 172.21.15.12:0/2331603958 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30011830 con 0x7f7b440cb3f0 2024-03-29T11:31:11.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.056+0000 7f7b42a00700 1 -- 172.21.15.12:0/2331603958 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b30011990 con 0x7f7b440cb3f0 2024-03-29T11:31:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.056+0000 7f7b42a00700 1 -- 172.21.15.12:0/2331603958 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30011d20 con 0x7f7b440cb3f0 2024-03-29T11:31:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.057+0000 7f7b4aa00700 1 -- 172.21.15.12:0/2331603958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b440cb3f0 msgr2=0x7f7b440ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.057+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/2331603958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b440cb3f0 0x7f7b440ac0c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009870 tx=0x7f7b30009b80 comp rx=0 tx=0).stop 2024-03-29T11:31:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.057+0000 7f7b4aa00700 1 -- 172.21.15.12:0/2331603958 shutdown_connections 2024-03-29T11:31:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.057+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/2331603958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b440cb3f0 0x7f7b440ac0c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.057+0000 7f7b4aa00700 1 -- 172.21.15.12:0/2331603958 >> 172.21.15.12:0/2331603958 conn(0x7f7b440cf190 msgr2=0x7f7b440cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.057+0000 7f7b4aa00700 1 -- 172.21.15.12:0/2331603958 shutdown_connections 2024-03-29T11:31:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.057+0000 7f7b4aa00700 1 -- 172.21.15.12:0/2331603958 wait complete. 2024-03-29T11:31:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.059+0000 7f7b4aa00700 1 Processor -- start 2024-03-29T11:31:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.059+0000 7f7b4aa00700 1 -- start start 2024-03-29T11:31:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b4aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b440ac600 0x7f7b440af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b4aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4415b5d0 0x7f7b441576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b4aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4409cd10 0x7f7b441580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b4aa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4406da50 con 0x7f7b4409cd10 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b4aa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4415ad90 con 0x7f7b440ac600 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b4aa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4409d190 con 0x7f7b4415b5d0 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b48a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4409cd10 0x7f7b441580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b48a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4409cd10 0x7f7b441580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58224/0 (socket says 172.21.15.12:58224) 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b48a00700 1 -- 172.21.15.12:0/3444877684 learned_addr learned my addr 172.21.15.12:0/3444877684 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b43e00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b440ac600 0x7f7b440af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.060+0000 7f7b43400700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4415b5d0 0x7f7b441576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b48a00700 1 -- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4415b5d0 msgr2=0x7f7b441576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b48a00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4415b5d0 0x7f7b441576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b48a00700 1 -- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b440ac600 msgr2=0x7f7b440af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b48a00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b440ac600 0x7f7b440af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b48a00700 1 -- 172.21.15.12:0/3444877684 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300094a0 con 0x7f7b4409cd10 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b43e00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b440ac600 0x7f7b440af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b48a00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4409cd10 0x7f7b441580b0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00cbc0 tx=0x7f7b3c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:11.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b3be00700 1 -- 172.21.15.12:0/3444877684 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c003c00 con 0x7f7b4409cd10 2024-03-29T11:31:11.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b3be00700 1 -- 172.21.15.12:0/3444877684 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3c003d60 con 0x7f7b4409cd10 2024-03-29T11:31:11.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b3be00700 1 -- 172.21.15.12:0/3444877684 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00aaa0 con 0x7f7b4409cd10 2024-03-29T11:31:11.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3444877684 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4415aab0 con 0x7f7b4409cd10 2024-03-29T11:31:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.061+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3444877684 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b44154ae0 con 0x7f7b4409cd10 2024-03-29T11:31:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.062+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3444877684 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b44067720 con 0x7f7b4409cd10 2024-03-29T11:31:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.063+0000 7f7b3be00700 1 -- 172.21.15.12:0/3444877684 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 125) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f7b3c00ac00 con 0x7f7b4409cd10 2024-03-29T11:31:11.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.064+0000 7f7b3be00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b2406adb0 0x7f7b2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:11.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.064+0000 7f7b3be00700 1 -- 172.21.15.12:0/3444877684 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f7b3c089720 con 0x7f7b4409cd10 2024-03-29T11:31:11.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.064+0000 7f7b43e00700 1 -- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b2406adb0 msgr2=0x7f7b2406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:11.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.065+0000 7f7b43e00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b2406adb0 0x7f7b2406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:11.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.070+0000 7f7b3be00700 1 -- 172.21.15.12:0/3444877684 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b3c054530 con 0x7f7b4409cd10 2024-03-29T11:31:11.103 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:11.109+0000 7ff414758240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:11.134 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.141+0000 7fb0bba5a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:11.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.155+0000 7f7b3be00700 1 -- 172.21.15.12:0/3444877684 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 126) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f7b3c054900 con 0x7f7b4409cd10 2024-03-29T11:31:11.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.155+0000 7f7b3be00700 1 -- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b2406adb0 msgr2=0x7f7b2406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:11.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.155+0000 7f7b3be00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b2406adb0 0x7f7b2406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.211 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.218+0000 7fb02d800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:11.211 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.218+0000 7fb02d800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:11.211 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.218+0000 7fb02d800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:11.212 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.218+0000 7fb02d800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:11.212 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.218+0000 7fb02d800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:11.212 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.219+0000 7fb029c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:11.212 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.219+0000 7fb029c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:11.212 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.219+0000 7fb029c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:11.212 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.219+0000 7fb029c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:11.212 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:11.219+0000 7fb029c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.338+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3444877684 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b4406ea60 con 0x7f7b4409cd10 2024-03-29T11:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.339+0000 7f7b3be00700 1 -- 172.21.15.12:0/3444877684 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f7b3c075650 con 0x7f7b4409cd10 2024-03-29T11:31:11.333 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:11.333 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/395980670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":395980670},{"type":"v1","addr":"172.21.15.159:6837","nonce":395980670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":93},{"gid":10183,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/531080306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":531080306},{"type":"v1","addr":"172.21.15.159:6835","nonce":531080306}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10509,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2709565550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2709565550},{"type":"v1","addr":"172.21.15.12:6834","nonce":2709565550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:28:50.095373+0000","modified":"2024-03-29T11:28:53.584378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10512":{"gid":10512,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.12:6835/99601797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":99601797},{"type":"v1","addr":"172.21.15.12:6835","nonce":99601797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-29T11:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.342+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3444877684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4409cd10 msgr2=0x7f7b441580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.342+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4409cd10 0x7f7b441580b0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00cbc0 tx=0x7f7b3c00ced0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.342+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3444877684 shutdown_connections 2024-03-29T11:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.342+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b2406adb0 0x7f7b2406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.342+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b4409cd10 0x7f7b441580b0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.342+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b440ac600 0x7f7b440af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.342+0000 7f7b4aa00700 1 --2- 172.21.15.12:0/3444877684 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b4415b5d0 0x7f7b441576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.342+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3444877684 >> 172.21.15.12:0/3444877684 conn(0x7f7b440cf190 msgr2=0x7f7b4415bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.342+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3444877684 shutdown_connections 2024-03-29T11:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.342+0000 7f7b4aa00700 1 -- 172.21.15.12:0/3444877684 wait complete. 2024-03-29T11:31:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 98 2024-03-29T11:31:11.355 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:31:11.571 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:11.577+0000 7ff414758240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:31:11.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.603+0000 7f4460c00700 1 Processor -- start 2024-03-29T11:31:11.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.603+0000 7f4460c00700 1 -- start start 2024-03-29T11:31:11.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.606+0000 7f4460c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f445c1661e0 0x7f445c1685d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:11.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.606+0000 7f4460c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445c0ae7b0 con 0x7f445c0b5160 2024-03-29T11:31:11.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.606+0000 7f4460c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445c0aea90 con 0x7f445c0acda0 2024-03-29T11:31:11.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.606+0000 7f4460c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445c0b55e0 con 0x7f445c1661e0 2024-03-29T11:31:11.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.606+0000 7f4459400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f445c0b5160 0x7f445c0ad180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34654/0 (socket says 172.21.15.12:34654) 2024-03-29T11:31:11.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.606+0000 7f4459400700 1 -- 172.21.15.12:0/121629374 learned_addr learned my addr 172.21.15.12:0/121629374 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:11.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.606+0000 7f445a800700 1 --2- 172.21.15.12:0/121629374 >> v2:172.21.15.159:3300/0 conn(0x7f445c1661e0 0x7f445c1685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:11.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.606+0000 7f4458a00700 1 -- 172.21.15.12:0/121629374 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916322484 0 0) 0x7f445c0ae7b0 con 0x7f445c0b5160 2024-03-29T11:31:11.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.607+0000 7f4458a00700 1 -- 172.21.15.12:0/121629374 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44400035b0 con 0x7f445c0b5160 2024-03-29T11:31:11.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.607+0000 7f4458a00700 1 -- 172.21.15.12:0/121629374 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178951071 0 0) 0x7f445c0aea90 con 0x7f445c0acda0 2024-03-29T11:31:11.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.607+0000 7f4458a00700 1 -- 172.21.15.12:0/121629374 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f445c0ae7b0 con 0x7f445c0acda0 2024-03-29T11:31:11.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.607+0000 7f445a800700 1 -- 172.21.15.12:0/121629374 >> v1:172.21.15.159:6789/0 conn(0x7f445c0acda0 legacy=0x7f445c15e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.607+0000 7f445a800700 1 -- 172.21.15.12:0/121629374 >> v1:172.21.15.12:6789/0 conn(0x7f445c0b5160 legacy=0x7f445c0ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.607+0000 7f445a800700 1 -- 172.21.15.12:0/121629374 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c0aed70 con 0x7f445c1661e0 2024-03-29T11:31:11.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.607+0000 7f445a800700 1 --2- 172.21.15.12:0/121629374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f445c1661e0 0x7f445c1685d0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f4454009dc0 tx=0x7f445400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56d004af8a8888e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:11.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.607+0000 7f4458a00700 1 -- 172.21.15.12:0/121629374 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445400e070 con 0x7f445c1661e0 2024-03-29T11:31:11.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.607+0000 7f4458a00700 1 -- 172.21.15.12:0/121629374 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445400bae0 con 0x7f445c1661e0 2024-03-29T11:31:11.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.607+0000 7f4458a00700 1 -- 172.21.15.12:0/121629374 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445400be90 con 0x7f445c1661e0 2024-03-29T11:31:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.608+0000 7f4460c00700 1 -- 172.21.15.12:0/121629374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f445c1661e0 msgr2=0x7f445c1685d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.608+0000 7f4460c00700 1 --2- 172.21.15.12:0/121629374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f445c1661e0 0x7f445c1685d0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f4454009dc0 tx=0x7f445400b330 comp rx=0 tx=0).stop 2024-03-29T11:31:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.608+0000 7f4460c00700 1 -- 172.21.15.12:0/121629374 shutdown_connections 2024-03-29T11:31:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.608+0000 7f4460c00700 1 --2- 172.21.15.12:0/121629374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f445c1661e0 0x7f445c1685d0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.608+0000 7f4460c00700 1 -- 172.21.15.12:0/121629374 >> 172.21.15.12:0/121629374 conn(0x7f445c072170 msgr2=0x7f445c06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.608+0000 7f4460c00700 1 -- 172.21.15.12:0/121629374 shutdown_connections 2024-03-29T11:31:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.608+0000 7f4460c00700 1 -- 172.21.15.12:0/121629374 wait complete. 2024-03-29T11:31:11.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.611+0000 7f4460c00700 1 Processor -- start 2024-03-29T11:31:11.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.611+0000 7f4460c00700 1 -- start start 2024-03-29T11:31:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.613+0000 7f4460c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0acda0 0x7f445c156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.613+0000 7f4460c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f445c0b5160 0x7f445c15ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.613+0000 7f4460c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f445c1661e0 0x7f445c15e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.613+0000 7f4460c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c1585c0 con 0x7f445c0acda0 2024-03-29T11:31:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.613+0000 7f4460c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c1588a0 con 0x7f445c0b5160 2024-03-29T11:31:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.613+0000 7f4460c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445c158b80 con 0x7f445c1661e0 2024-03-29T11:31:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.613+0000 7f4459e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0acda0 0x7f445c156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.613+0000 7f4459e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0acda0 0x7f445c156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58234/0 (socket says 172.21.15.12:58234) 2024-03-29T11:31:11.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.613+0000 7f4459e00700 1 -- 172.21.15.12:0/4102370951 learned_addr learned my addr 172.21.15.12:0/4102370951 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4459400700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f445c0b5160 0x7f445c15ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f445a800700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f445c1661e0 0x7f445c15e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4459e00700 1 -- 172.21.15.12:0/4102370951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f445c1661e0 msgr2=0x7f445c15e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4459e00700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f445c1661e0 0x7f445c15e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4459e00700 1 -- 172.21.15.12:0/4102370951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f445c0b5160 msgr2=0x7f445c15ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4459e00700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f445c0b5160 0x7f445c15ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4459e00700 1 -- 172.21.15.12:0/4102370951 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4454009a70 con 0x7f445c0acda0 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4459400700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f445c0b5160 0x7f445c15ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f445a800700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f445c1661e0 0x7f445c15e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4459e00700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0acda0 0x7f445c156cb0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f444c00c520 tx=0x7f444c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4452000700 1 -- 172.21.15.12:0/4102370951 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c0052d0 con 0x7f445c0acda0 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4452000700 1 -- 172.21.15.12:0/4102370951 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c005430 con 0x7f445c0acda0 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4460c00700 1 -- 172.21.15.12:0/4102370951 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445c158e60 con 0x7f445c0acda0 2024-03-29T11:31:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4452000700 1 -- 172.21.15.12:0/4102370951 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c0038c0 con 0x7f445c0acda0 2024-03-29T11:31:11.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.614+0000 7f4460c00700 1 -- 172.21.15.12:0/4102370951 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445c1581e0 con 0x7f445c0acda0 2024-03-29T11:31:11.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.616+0000 7f4460c00700 1 -- 172.21.15.12:0/4102370951 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445c067720 con 0x7f445c0acda0 2024-03-29T11:31:11.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.617+0000 7f4452000700 1 -- 172.21.15.12:0/4102370951 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 126) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f444c003a20 con 0x7f445c0acda0 2024-03-29T11:31:11.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.619+0000 7f4452000700 1 -- 172.21.15.12:0/4102370951 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f444c0885f0 con 0x7f445c0acda0 2024-03-29T11:31:11.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.622+0000 7f4452000700 1 -- 172.21.15.12:0/4102370951 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f444c053920 con 0x7f445c0acda0 2024-03-29T11:31:11.695 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:11.701+0000 7ff414758240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:11.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.880+0000 7f4460c00700 1 -- 172.21.15.12:0/4102370951 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f445c06e1d0 con 0x7f445c0acda0 2024-03-29T11:31:11.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.881+0000 7f4452000700 1 -- 172.21.15.12:0/4102370951 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f444c056f40 con 0x7f445c0acda0 2024-03-29T11:31:11.875 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:11.875 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/395980670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":395980670},{"type":"v1","addr":"172.21.15.159:6837","nonce":395980670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":93},{"gid":10183,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/531080306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":531080306},{"type":"v1","addr":"172.21.15.159:6835","nonce":531080306}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10509,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2709565550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2709565550},{"type":"v1","addr":"172.21.15.12:6834","nonce":2709565550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:28:50.095373+0000","modified":"2024-03-29T11:28:53.584378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10512":{"gid":10512,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.12:6835/99601797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":99601797},{"type":"v1","addr":"172.21.15.12:6835","nonce":99601797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-29T11:31:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.884+0000 7f4460c00700 1 -- 172.21.15.12:0/4102370951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0acda0 msgr2=0x7f445c156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.884+0000 7f4460c00700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0acda0 0x7f445c156cb0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f444c00c520 tx=0x7f444c00c8e0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.884+0000 7f4460c00700 1 -- 172.21.15.12:0/4102370951 shutdown_connections 2024-03-29T11:31:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.884+0000 7f4460c00700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f445c1661e0 0x7f445c15e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.884+0000 7f4460c00700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0acda0 0x7f445c156cb0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.884+0000 7f4460c00700 1 --2- 172.21.15.12:0/4102370951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f445c0b5160 0x7f445c15ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.884+0000 7f4460c00700 1 -- 172.21.15.12:0/4102370951 >> 172.21.15.12:0/4102370951 conn(0x7f445c072170 msgr2=0x7f445c0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.884+0000 7f4460c00700 1 -- 172.21.15.12:0/4102370951 shutdown_connections 2024-03-29T11:31:11.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:11.884+0000 7f4460c00700 1 -- 172.21.15.12:0/4102370951 wait complete. 2024-03-29T11:31:11.879 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 98 2024-03-29T11:31:11.895 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:12.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.098+0000 7f113b200700 1 Processor -- start 2024-03-29T11:31:12.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.099+0000 7f113b200700 1 -- start start 2024-03-29T11:31:12.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.100+0000 7f113b200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11341619d0 0x7f11341685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.100+0000 7f113b200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340d30d0 con 0x7f113415d640 2024-03-29T11:31:12.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.100+0000 7f113b200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340b6840 con 0x7f11340ceec0 2024-03-29T11:31:12.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.100+0000 7f113b200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11340d2b30 con 0x7f11341619d0 2024-03-29T11:31:12.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.100+0000 7f1133e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f113415d640 0x7f11340d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34670/0 (socket says 172.21.15.12:34670) 2024-03-29T11:31:12.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.100+0000 7f1133e00700 1 -- 172.21.15.12:0/1750208263 learned_addr learned my addr 172.21.15.12:0/1750208263 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:12.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.100+0000 7f1139200700 1 --2- 172.21.15.12:0/1750208263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11341619d0 0x7f11341685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1132a00700 1 -- 172.21.15.12:0/1750208263 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441513933 0 0) 0x7f11340d30d0 con 0x7f113415d640 2024-03-29T11:31:12.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1132a00700 1 -- 172.21.15.12:0/1750208263 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11180035b0 con 0x7f113415d640 2024-03-29T11:31:12.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1132a00700 1 -- 172.21.15.12:0/1750208263 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870064486 0 0) 0x7f11340b6840 con 0x7f11340ceec0 2024-03-29T11:31:12.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1132a00700 1 -- 172.21.15.12:0/1750208263 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11340d30d0 con 0x7f11340ceec0 2024-03-29T11:31:12.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1139200700 1 -- 172.21.15.12:0/1750208263 >> v1:172.21.15.159:6789/0 conn(0x7f11340ceec0 legacy=0x7f11341612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1139200700 1 -- 172.21.15.12:0/1750208263 >> v1:172.21.15.12:6789/0 conn(0x7f113415d640 legacy=0x7f11340d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1139200700 1 -- 172.21.15.12:0/1750208263 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11340d2df0 con 0x7f11341619d0 2024-03-29T11:31:12.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1139200700 1 --2- 172.21.15.12:0/1750208263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11341619d0 0x7f11341685f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f112c009dc0 tx=0x7f112c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f3e31fd3b91489b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:12.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1132a00700 1 -- 172.21.15.12:0/1750208263 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00e070 con 0x7f11341619d0 2024-03-29T11:31:12.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1132a00700 1 -- 172.21.15.12:0/1750208263 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c00bb20 con 0x7f11341619d0 2024-03-29T11:31:12.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.101+0000 7f1132a00700 1 -- 172.21.15.12:0/1750208263 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c012430 con 0x7f11341619d0 2024-03-29T11:31:12.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.102+0000 7f113b200700 1 -- 172.21.15.12:0/1750208263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11341619d0 msgr2=0x7f11341685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.102+0000 7f113b200700 1 --2- 172.21.15.12:0/1750208263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11341619d0 0x7f11341685f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f112c009dc0 tx=0x7f112c00b330 comp rx=0 tx=0).stop 2024-03-29T11:31:12.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.102+0000 7f113b200700 1 -- 172.21.15.12:0/1750208263 shutdown_connections 2024-03-29T11:31:12.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.102+0000 7f113b200700 1 --2- 172.21.15.12:0/1750208263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11341619d0 0x7f11341685f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.102+0000 7f113b200700 1 -- 172.21.15.12:0/1750208263 >> 172.21.15.12:0/1750208263 conn(0x7f113408aff0 msgr2=0x7f11340d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:12.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.102+0000 7f113b200700 1 -- 172.21.15.12:0/1750208263 shutdown_connections 2024-03-29T11:31:12.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.102+0000 7f113b200700 1 -- 172.21.15.12:0/1750208263 wait complete. 2024-03-29T11:31:12.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.104+0000 7f113b200700 1 Processor -- start 2024-03-29T11:31:12.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.104+0000 7f113b200700 1 -- start start 2024-03-29T11:31:12.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f113b200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11340ceec0 0x7f113415c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f113b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f113415d640 0x7f1134158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f113b200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11341619d0 0x7f11340cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f113b200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113415c100 con 0x7f113415d640 2024-03-29T11:31:12.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f113b200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1134159260 con 0x7f11341619d0 2024-03-29T11:31:12.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f113b200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113415b480 con 0x7f11340ceec0 2024-03-29T11:31:12.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f1133400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f113415d640 0x7f1134158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f1133400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f113415d640 0x7f1134158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58244/0 (socket says 172.21.15.12:58244) 2024-03-29T11:31:12.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f1133400700 1 -- 172.21.15.12:0/1564390303 learned_addr learned my addr 172.21.15.12:0/1564390303 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f1139200700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11341619d0 0x7f11340cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f1133e00700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11340ceec0 0x7f113415c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f1133400700 1 -- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11340ceec0 msgr2=0x7f113415c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.105+0000 7f1133400700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11340ceec0 0x7f113415c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.106+0000 7f1133400700 1 -- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11341619d0 msgr2=0x7f11340cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.106+0000 7f1133400700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11341619d0 0x7f11340cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.106+0000 7f1133400700 1 -- 172.21.15.12:0/1564390303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c009a70 con 0x7f113415d640 2024-03-29T11:31:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.106+0000 7f1139200700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11341619d0 0x7f11340cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.106+0000 7f1133400700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f113415d640 0x7f1134158d20 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f112000c520 tx=0x7f112000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:12.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.106+0000 7f112be00700 1 -- 172.21.15.12:0/1564390303 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11200052d0 con 0x7f113415d640 2024-03-29T11:31:12.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.106+0000 7f112be00700 1 -- 172.21.15.12:0/1564390303 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1120005430 con 0x7f113415d640 2024-03-29T11:31:12.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.106+0000 7f112be00700 1 -- 172.21.15.12:0/1564390303 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11200038a0 con 0x7f113415d640 2024-03-29T11:31:12.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.106+0000 7f113b200700 1 -- 172.21.15.12:0/1564390303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113415b760 con 0x7f113415d640 2024-03-29T11:31:12.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.106+0000 7f113b200700 1 -- 172.21.15.12:0/1564390303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113415bd20 con 0x7f113415d640 2024-03-29T11:31:12.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.107+0000 7f113b200700 1 -- 172.21.15.12:0/1564390303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113415c240 con 0x7f113415d640 2024-03-29T11:31:12.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.108+0000 7f112be00700 1 -- 172.21.15.12:0/1564390303 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 126) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f1120003a00 con 0x7f113415d640 2024-03-29T11:31:12.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.109+0000 7f112be00700 1 -- 172.21.15.12:0/1564390303 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f1120088580 con 0x7f113415d640 2024-03-29T11:31:12.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.114+0000 7f112be00700 1 -- 172.21.15.12:0/1564390303 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f113415c240 con 0x7f113415d640 2024-03-29T11:31:12.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.155+0000 7f112be00700 1 -- 172.21.15.12:0/1564390303 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 127) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f1120053360 con 0x7f113415d640 2024-03-29T11:31:12.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.156+0000 7f112be00700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f11140704e0 0x7f11140728d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.158+0000 7f1133e00700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f11140704e0 0x7f11140728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.160+0000 7f1133e00700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f11140704e0 0x7f11140728d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1124007fd0 tx=0x7f1124007dc0 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.330+0000 7f113b200700 1 -- 172.21.15.12:0/1564390303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f113415c240 con 0x7f113415d640 2024-03-29T11:31:12.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.331+0000 7f112be00700 1 -- 172.21.15.12:0/1564390303 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+17209 (secure 0 0 0) 0x7f113415c240 con 0x7f113415d640 2024-03-29T11:31:12.325 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:12.325 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":214,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:11.144207+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:28:47.651679+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-03-29T11:28:49.085953+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":213,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.18","mappings":[{"from":0,"to":2}]},{"pgid":"25.1f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/187236055":"2024-03-30T11:31:11.144150+0000","172.21.15.159:0/1125652430":"2024-03-30T11:31:11.144150+0000","172.21.15.159:0/974124420":"2024-03-30T11:31:11.144150+0000","172.21.15.159:0/3612634932":"2024-03-30T11:31:11.144150+0000","172.21.15.159:0/2193792293":"2024-03-30T11:31:11.144150+0000","172.21.15.159:0/2122872060":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/2280623603":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/3600559047":"2024-03-30T11:29:07.287915+0000","172.21.15.159:6833/62646":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/1758254515":"2024-03-30T11:31:11.144150+0000","172.21.15.159:6832/62646":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/4050963121":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/67497648":"2024-03-30T11:29:07.287915+0000","172.21.15.159:0/343093900":"2024-03-30T11:29:07.287915+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 -- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f11140704e0 msgr2=0x7f11140728d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f11140704e0 0x7f11140728d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1124007fd0 tx=0x7f1124007dc0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 -- 172.21.15.12:0/1564390303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f113415d640 msgr2=0x7f1134158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f113415d640 0x7f1134158d20 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f112000c520 tx=0x7f112000c830 comp rx=0 tx=0).stop 2024-03-29T11:31:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 -- 172.21.15.12:0/1564390303 shutdown_connections 2024-03-29T11:31:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f11140704e0 0x7f11140728d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11341619d0 0x7f11340cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11340ceec0 0x7f113415c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 --2- 172.21.15.12:0/1564390303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f113415d640 0x7f1134158d20 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 -- 172.21.15.12:0/1564390303 >> 172.21.15.12:0/1564390303 conn(0x7f113408aff0 msgr2=0x7f113406cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 -- 172.21.15.12:0/1564390303 shutdown_connections 2024-03-29T11:31:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.334+0000 7f113b200700 1 -- 172.21.15.12:0/1564390303 wait complete. 2024-03-29T11:31:12.346 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T11:31:12.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.553+0000 7f1ee5c00700 1 Processor -- start 2024-03-29T11:31:12.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.553+0000 7f1ee5c00700 1 -- start start 2024-03-29T11:31:12.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.554+0000 7f1ee5c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1ee00d3f20 0x7f1ee00cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.554+0000 7f1ee5c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ee00b3f80 0x7f1ee00ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.554+0000 7f1ee5c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee0161230 con 0x7f1ee0160e50 2024-03-29T11:31:12.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.554+0000 7f1ee5c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee00d2a80 con 0x7f1ee00b3f80 2024-03-29T11:31:12.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.554+0000 7f1ee5c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee00d43b0 con 0x7f1ee00d3f20 2024-03-29T11:31:12.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1edf400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ee00b3f80 0x7f1ee00ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1edf400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ee00b3f80 0x7f1ee00ce430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58252/0 (socket says 172.21.15.12:58252) 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1edea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1ee0160e50 0x7f1ee00b4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34684/0 (socket says 172.21.15.12:34684) 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1edf400700 1 -- 172.21.15.12:0/2876869637 learned_addr learned my addr 172.21.15.12:0/2876869637 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1ede000700 1 --2- 172.21.15.12:0/2876869637 >> v2:172.21.15.159:3300/0 conn(0x7f1ee00d3f20 0x7f1ee00cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1edf400700 1 -- 172.21.15.12:0/2876869637 >> v2:172.21.15.159:3300/0 conn(0x7f1ee00d3f20 msgr2=0x7f1ee00cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1edf400700 1 --2- 172.21.15.12:0/2876869637 >> v2:172.21.15.159:3300/0 conn(0x7f1ee00d3f20 0x7f1ee00cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1edf400700 1 -- 172.21.15.12:0/2876869637 >> v1:172.21.15.12:6789/0 conn(0x7f1ee0160e50 legacy=0x7f1ee00b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1edf400700 1 -- 172.21.15.12:0/2876869637 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee00d21e0 con 0x7f1ee00b3f80 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1edd600700 1 -- 172.21.15.12:0/2876869637 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939543596 0 0) 0x7f1ee0161230 con 0x7f1ee0160e50 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.555+0000 7f1edf400700 1 --2- 172.21.15.12:0/2876869637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 0x7f1ee00ce430 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f1ee008b910 tx=0x7f1ed800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43355849399a2ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1edd600700 1 -- 172.21.15.12:0/2876869637 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed800e070 con 0x7f1ee00b3f80 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1edd600700 1 -- 172.21.15.12:0/2876869637 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed800b9c0 con 0x7f1ee00b3f80 2024-03-29T11:31:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1edd600700 1 -- 172.21.15.12:0/2876869637 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed800bd70 con 0x7f1ee00b3f80 2024-03-29T11:31:12.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2876869637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 msgr2=0x7f1ee00ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1ee5c00700 1 --2- 172.21.15.12:0/2876869637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 0x7f1ee00ce430 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f1ee008b910 tx=0x7f1ed800b330 comp rx=0 tx=0).stop 2024-03-29T11:31:12.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2876869637 shutdown_connections 2024-03-29T11:31:12.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1ee5c00700 1 --2- 172.21.15.12:0/2876869637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 0x7f1ee00ce430 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1ee5c00700 1 --2- 172.21.15.12:0/2876869637 >> v2:172.21.15.159:3300/0 conn(0x7f1ee00d3f20 0x7f1ee00cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2876869637 >> 172.21.15.12:0/2876869637 conn(0x7f1ee008aff0 msgr2=0x7f1ee00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:12.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2876869637 shutdown_connections 2024-03-29T11:31:12.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.556+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2876869637 wait complete. 2024-03-29T11:31:12.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.558+0000 7f1ee5c00700 1 Processor -- start 2024-03-29T11:31:12.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.558+0000 7f1ee5c00700 1 -- start start 2024-03-29T11:31:12.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.559+0000 7f1ee5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 0x7f1ee00cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.559+0000 7f1ee5c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ee0160e50 0x7f1ee00cff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.559+0000 7f1ee5c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ee00ca0e0 0x7f1ee00d0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.559+0000 7f1ee5c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee015c390 con 0x7f1ee00b3f80 2024-03-29T11:31:12.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.559+0000 7f1ee5c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00c7980 con 0x7f1ee00ca0e0 2024-03-29T11:31:12.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.559+0000 7f1ee5c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00cbea0 con 0x7f1ee0160e50 2024-03-29T11:31:12.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.559+0000 7f1edea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 0x7f1ee00cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.559+0000 7f1edea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 0x7f1ee00cb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58262/0 (socket says 172.21.15.12:58262) 2024-03-29T11:31:12.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.559+0000 7f1edea00700 1 -- 172.21.15.12:0/2512704955 learned_addr learned my addr 172.21.15.12:0/2512704955 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1edf400700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ee00ca0e0 0x7f1ee00d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1ede000700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ee0160e50 0x7f1ee00cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1edea00700 1 -- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ee0160e50 msgr2=0x7f1ee00cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1edea00700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ee0160e50 0x7f1ee00cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1edea00700 1 -- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ee00ca0e0 msgr2=0x7f1ee00d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1edea00700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ee00ca0e0 0x7f1ee00d0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1edea00700 1 -- 172.21.15.12:0/2512704955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed8009b30 con 0x7f1ee00b3f80 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1ede000700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ee0160e50 0x7f1ee00cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1edea00700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 0x7f1ee00cb650 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000c550 tx=0x7f1ed000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1ed6a00700 1 -- 172.21.15.12:0/2512704955 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0004c80 con 0x7f1ee00b3f80 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1ed6a00700 1 -- 172.21.15.12:0/2512704955 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed0004e00 con 0x7f1ee00b3f80 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1ed6a00700 1 -- 172.21.15.12:0/2512704955 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0007310 con 0x7f1ee00b3f80 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.560+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2512704955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee00cc180 con 0x7f1ee00b3f80 2024-03-29T11:31:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.561+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2512704955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee00c7c00 con 0x7f1ee00b3f80 2024-03-29T11:31:12.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.562+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2512704955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee00cc6a0 con 0x7f1ee00b3f80 2024-03-29T11:31:12.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.562+0000 7f1ed6a00700 1 -- 172.21.15.12:0/2512704955 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 127) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f1ed0007470 con 0x7f1ee00b3f80 2024-03-29T11:31:12.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.563+0000 7f1ed6a00700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ec006aae0 0x7f1ec006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:12.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.563+0000 7f1ed6a00700 1 -- 172.21.15.12:0/2512704955 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f1ed0088fb0 con 0x7f1ee00b3f80 2024-03-29T11:31:12.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.564+0000 7f1ede000700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ec006aae0 0x7f1ec006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:12.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.564+0000 7f1ede000700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ec006aae0 0x7f1ec006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0099f0 tx=0x7f1ecc008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:12.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.568+0000 7f1ed6a00700 1 -- 172.21.15.12:0/2512704955 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ee00cc6a0 con 0x7f1ee00b3f80 2024-03-29T11:31:12.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.834+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2512704955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1ee00cc6a0 con 0x7f1ee00b3f80 2024-03-29T11:31:12.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T11:31:12.835+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T11:31:12.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.847+0000 7f1ed6a00700 1 -- 172.21.15.12:0/2512704955 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v99) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1ee00cc6a0 con 0x7f1ee00b3f80 2024-03-29T11:31:12.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ec006aae0 msgr2=0x7f1ec006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ec006aae0 0x7f1ec006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0099f0 tx=0x7f1ecc008040 comp rx=0 tx=0).stop 2024-03-29T11:31:12.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2512704955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 msgr2=0x7f1ee00cb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:12.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 0x7f1ee00cb650 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000c550 tx=0x7f1ed000c860 comp rx=0 tx=0).stop 2024-03-29T11:31:12.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2512704955 shutdown_connections 2024-03-29T11:31:12.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ec006aae0 0x7f1ec006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ee00ca0e0 0x7f1ee00d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ee00b3f80 0x7f1ee00cb650 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 --2- 172.21.15.12:0/2512704955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ee0160e50 0x7f1ee00cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:12.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2512704955 >> 172.21.15.12:0/2512704955 conn(0x7f1ee008aff0 msgr2=0x7f1ee00ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:12.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2512704955 shutdown_connections 2024-03-29T11:31:12.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:12.851+0000 7f1ee5c00700 1 -- 172.21.15.12:0/2512704955 wait complete. 2024-03-29T11:31:12.846 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T11:31:12.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T11:31:12.919 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:31:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.082+0000 7f26e6e00700 1 Processor -- start 2024-03-29T11:31:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.083+0000 7f26e6e00700 1 -- start start 2024-03-29T11:31:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.084+0000 7f26e6e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26e00cd020 0x7f26e00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.084+0000 7f26e6e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e00cdae0 con 0x7f26e00b2070 2024-03-29T11:31:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.084+0000 7f26e6e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e00cddc0 con 0x7f26e00cbb30 2024-03-29T11:31:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.084+0000 7f26e6e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00ce0a0 con 0x7f26e00cd020 2024-03-29T11:31:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.084+0000 7f26df400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26e00cd020 0x7f26e00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.084+0000 7f26df400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26e00cd020 0x7f26e00cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58290/0 (socket says 172.21.15.12:58290) 2024-03-29T11:31:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.084+0000 7f26df400700 1 -- 172.21.15.12:0/3857407972 learned_addr learned my addr 172.21.15.12:0/3857407972 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.084+0000 7f26df400700 1 -- 172.21.15.12:0/3857407972 >> v1:172.21.15.159:6789/0 conn(0x7f26e00cbb30 legacy=0x7f26e00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.084+0000 7f26df400700 1 -- 172.21.15.12:0/3857407972 >> v1:172.21.15.12:6789/0 conn(0x7f26e00b2070 legacy=0x7f26e006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.084+0000 7f26df400700 1 -- 172.21.15.12:0/3857407972 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e006dc20 con 0x7f26e00cd020 2024-03-29T11:31:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.085+0000 7f26df400700 1 --2- 172.21.15.12:0/3857407972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00cd020 0x7f26e00cd480 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0098f0 tx=0x7f26cc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff458515dd369257 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.085+0000 7f26dea00700 1 -- 172.21.15.12:0/3857407972 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc004230 con 0x7f26e00cd020 2024-03-29T11:31:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.085+0000 7f26dea00700 1 -- 172.21.15.12:0/3857407972 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26cc004390 con 0x7f26e00cd020 2024-03-29T11:31:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.085+0000 7f26dea00700 1 -- 172.21.15.12:0/3857407972 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc0036a0 con 0x7f26e00cd020 2024-03-29T11:31:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.085+0000 7f26e6e00700 1 -- 172.21.15.12:0/3857407972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00cd020 msgr2=0x7f26e00cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.085+0000 7f26e6e00700 1 --2- 172.21.15.12:0/3857407972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00cd020 0x7f26e00cd480 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0098f0 tx=0x7f26cc009c00 comp rx=0 tx=0).stop 2024-03-29T11:31:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.085+0000 7f26e6e00700 1 -- 172.21.15.12:0/3857407972 shutdown_connections 2024-03-29T11:31:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.085+0000 7f26e6e00700 1 --2- 172.21.15.12:0/3857407972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00cd020 0x7f26e00cd480 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.086+0000 7f26e6e00700 1 -- 172.21.15.12:0/3857407972 >> 172.21.15.12:0/3857407972 conn(0x7f26e00b0d70 msgr2=0x7f26e00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.086+0000 7f26e6e00700 1 -- 172.21.15.12:0/3857407972 shutdown_connections 2024-03-29T11:31:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.086+0000 7f26e6e00700 1 -- 172.21.15.12:0/3857407972 wait complete. 2024-03-29T11:31:13.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.087+0000 7f26e6e00700 1 Processor -- start 2024-03-29T11:31:13.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.087+0000 7f26e6e00700 1 -- start start 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.088+0000 7f26e6e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00b2070 0x7f26e01505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.088+0000 7f26e6e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e00cbb30 0x7f26e0150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.088+0000 7f26e6e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e00cd020 0x7f26e01f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.088+0000 7f26e6e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0070fb0 con 0x7f26e00b2070 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.088+0000 7f26e6e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00cb380 con 0x7f26e00cd020 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.088+0000 7f26e6e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0155970 con 0x7f26e00cbb30 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26dfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00b2070 0x7f26e01505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26dfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00b2070 0x7f26e01505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58306/0 (socket says 172.21.15.12:58306) 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26dfe00700 1 -- 172.21.15.12:0/1070152136 learned_addr learned my addr 172.21.15.12:0/1070152136 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26dfe00700 1 -- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e00cbb30 msgr2=0x7f26e0150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26df400700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e00cbb30 0x7f26e0150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:13.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26e4e00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e00cd020 0x7f26e01f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26dfe00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e00cbb30 0x7f26e0150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26dfe00700 1 -- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e00cd020 msgr2=0x7f26e01f6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26dfe00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e00cd020 0x7f26e01f6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26dfe00700 1 -- 172.21.15.12:0/1070152136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc009550 con 0x7f26e00b2070 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26df400700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e00cbb30 0x7f26e0150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26e4e00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e00cd020 0x7f26e01f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.089+0000 7f26dfe00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00b2070 0x7f26e01505f0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b4a0 tx=0x7f26d000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.090+0000 7f26d7e00700 1 -- 172.21.15.12:0/1070152136 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0011830 con 0x7f26e00b2070 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.090+0000 7f26d7e00700 1 -- 172.21.15.12:0/1070152136 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d0011990 con 0x7f26e00b2070 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.090+0000 7f26d7e00700 1 -- 172.21.15.12:0/1070152136 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0011d40 con 0x7f26e00b2070 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.090+0000 7f26e6e00700 1 -- 172.21.15.12:0/1070152136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e00c8500 con 0x7f26e00b2070 2024-03-29T11:31:13.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.090+0000 7f26e6e00700 1 -- 172.21.15.12:0/1070152136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e01553b0 con 0x7f26e00b2070 2024-03-29T11:31:13.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.091+0000 7f26e6e00700 1 -- 172.21.15.12:0/1070152136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e0067720 con 0x7f26e00b2070 2024-03-29T11:31:13.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.092+0000 7f26d7e00700 1 -- 172.21.15.12:0/1070152136 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 127) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f26d001a400 con 0x7f26e00b2070 2024-03-29T11:31:13.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.092+0000 7f26d7e00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c006aae0 0x7f26c006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:13.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.093+0000 7f26d7e00700 1 -- 172.21.15.12:0/1070152136 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6555+0+0 (secure 0 0 0) 0x7f26d0089090 con 0x7f26e00b2070 2024-03-29T11:31:13.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.093+0000 7f26df400700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c006aae0 0x7f26c006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:13.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.094+0000 7f26df400700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c006aae0 0x7f26c006cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f26e00ae260 tx=0x7f26cc008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:13.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.097+0000 7f26d7e00700 1 -- 172.21.15.12:0/1070152136 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26d0054240 con 0x7f26e00b2070 2024-03-29T11:31:13.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.358+0000 7f26e6e00700 1 -- 172.21.15.12:0/1070152136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f26e015c3c0 con 0x7f26e00b2070 2024-03-29T11:31:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.851+0000 7f26d7e00700 1 -- 172.21.15.12:0/1070152136 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v100) v1 ==== 104+0+0 (secure 0 0 0) 0x7f26d0057860 con 0x7f26e00b2070 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 -- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c006aae0 msgr2=0x7f26c006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c006aae0 0x7f26c006cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f26e00ae260 tx=0x7f26cc008030 comp rx=0 tx=0).stop 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 -- 172.21.15.12:0/1070152136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00b2070 msgr2=0x7f26e01505f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00b2070 0x7f26e01505f0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b4a0 tx=0x7f26d000b860 comp rx=0 tx=0).stop 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 -- 172.21.15.12:0/1070152136 shutdown_connections 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f26c006aae0 0x7f26c006cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26e00cd020 0x7f26e01f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26e00b2070 0x7f26e01505f0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 --2- 172.21.15.12:0/1070152136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26e00cbb30 0x7f26e0150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 -- 172.21.15.12:0/1070152136 >> 172.21.15.12:0/1070152136 conn(0x7f26e00b0d70 msgr2=0x7f26e0158220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 -- 172.21.15.12:0/1070152136 shutdown_connections 2024-03-29T11:31:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:13.854+0000 7f26e6e00700 1 -- 172.21.15.12:0/1070152136 wait complete. 2024-03-29T11:31:13.870 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T11:31:14.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.080+0000 7f6f81400700 1 Processor -- start 2024-03-29T11:31:14.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.080+0000 7f6f81400700 1 -- start start 2024-03-29T11:31:14.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f81400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f7c071340 0x7f6f7c0b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:14.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f81400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f7c159d10 0x7f6f7c071720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:14.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f81400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6f7c070980 0x7f6f7c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:14.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f81400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7c0b2ff0 con 0x7f6f7c071340 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f81400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7c0b3e50 con 0x7f6f7c070980 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f81400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c071c60 con 0x7f6f7c159d10 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f7a800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f7c071340 0x7f6f7c0b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f7a800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f7c071340 0x7f6f7c0b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58322/0 (socket says 172.21.15.12:58322) 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f7a800700 1 -- 172.21.15.12:0/2713590949 learned_addr learned my addr 172.21.15.12:0/2713590949 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f7a800700 1 -- 172.21.15.12:0/2713590949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f7c159d10 msgr2=0x7f6f7c071720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f7a800700 1 --2- 172.21.15.12:0/2713590949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f7c159d10 0x7f6f7c071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f7a800700 1 -- 172.21.15.12:0/2713590949 >> v2:172.21.15.159:3300/0 conn(0x7f6f7c070980 msgr2=0x7f6f7c06d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f7a800700 1 --2- 172.21.15.12:0/2713590949 >> v2:172.21.15.159:3300/0 conn(0x7f6f7c070980 0x7f6f7c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.081+0000 7f6f7a800700 1 -- 172.21.15.12:0/2713590949 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c0b3240 con 0x7f6f7c071340 2024-03-29T11:31:14.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.082+0000 7f6f7a800700 1 --2- 172.21.15.12:0/2713590949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 0x7f6f7c0b42e0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0cbb70 tx=0x7f6f6c00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14be19b14655173 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:14.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.082+0000 7f6f79400700 1 -- 172.21.15.12:0/2713590949 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c00dc20 con 0x7f6f7c071340 2024-03-29T11:31:14.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.082+0000 7f6f79400700 1 -- 172.21.15.12:0/2713590949 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c00dd80 con 0x7f6f7c071340 2024-03-29T11:31:14.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.082+0000 7f6f79400700 1 -- 172.21.15.12:0/2713590949 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c017630 con 0x7f6f7c071340 2024-03-29T11:31:14.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.083+0000 7f6f81400700 1 -- 172.21.15.12:0/2713590949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 msgr2=0x7f6f7c0b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:14.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.083+0000 7f6f81400700 1 --2- 172.21.15.12:0/2713590949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 0x7f6f7c0b42e0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0cbb70 tx=0x7f6f6c00be70 comp rx=0 tx=0).stop 2024-03-29T11:31:14.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.083+0000 7f6f81400700 1 -- 172.21.15.12:0/2713590949 shutdown_connections 2024-03-29T11:31:14.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.083+0000 7f6f81400700 1 --2- 172.21.15.12:0/2713590949 >> v2:172.21.15.159:3300/0 conn(0x7f6f7c070980 0x7f6f7c06d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:14.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.083+0000 7f6f81400700 1 --2- 172.21.15.12:0/2713590949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 0x7f6f7c0b42e0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:14.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.083+0000 7f6f81400700 1 --2- 172.21.15.12:0/2713590949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f7c159d10 0x7f6f7c071720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:14.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.083+0000 7f6f81400700 1 -- 172.21.15.12:0/2713590949 >> 172.21.15.12:0/2713590949 conn(0x7f6f7c0cf210 msgr2=0x7f6f7c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:14.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.083+0000 7f6f81400700 1 -- 172.21.15.12:0/2713590949 shutdown_connections 2024-03-29T11:31:14.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.083+0000 7f6f81400700 1 -- 172.21.15.12:0/2713590949 wait complete. 2024-03-29T11:31:14.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.084+0000 7f6f81400700 1 Processor -- start 2024-03-29T11:31:14.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.085+0000 7f6f81400700 1 -- start start 2024-03-29T11:31:14.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.085+0000 7f6f81400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f7c070980 0x7f6f7c15b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:14.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f81400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 0x7f6f7c154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f81400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f7c159d10 0x7f6f7c145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f81400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0ccfa0 con 0x7f6f7c071340 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f81400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0cd280 con 0x7f6f7c070980 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f81400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0cd560 con 0x7f6f7c159d10 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f79e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 0x7f6f7c154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f79e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 0x7f6f7c154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58336/0 (socket says 172.21.15.12:58336) 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f79e00700 1 -- 172.21.15.12:0/2652111874 learned_addr learned my addr 172.21.15.12:0/2652111874 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f7a800700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f7c070980 0x7f6f7c15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f7b200700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f7c159d10 0x7f6f7c145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f79e00700 1 -- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f7c159d10 msgr2=0x7f6f7c145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f79e00700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f7c159d10 0x7f6f7c145d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f79e00700 1 -- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f7c070980 msgr2=0x7f6f7c15b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f79e00700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f7c070980 0x7f6f7c15b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f79e00700 1 -- 172.21.15.12:0/2652111874 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6c00b7f0 con 0x7f6f7c071340 2024-03-29T11:31:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.086+0000 7f6f79e00700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 0x7f6f7c154180 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800b530 tx=0x7f6f6800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.087+0000 7f6f72a00700 1 -- 172.21.15.12:0/2652111874 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f68011830 con 0x7f6f7c071340 2024-03-29T11:31:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.087+0000 7f6f72a00700 1 -- 172.21.15.12:0/2652111874 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f68011990 con 0x7f6f7c071340 2024-03-29T11:31:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.087+0000 7f6f81400700 1 -- 172.21.15.12:0/2652111874 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c0cd840 con 0x7f6f7c071340 2024-03-29T11:31:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.087+0000 7f6f72a00700 1 -- 172.21.15.12:0/2652111874 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f68011d20 con 0x7f6f7c071340 2024-03-29T11:31:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.087+0000 7f6f81400700 1 -- 172.21.15.12:0/2652111874 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c0ccc50 con 0x7f6f7c071340 2024-03-29T11:31:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.088+0000 7f6f81400700 1 -- 172.21.15.12:0/2652111874 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7c0cdd60 con 0x7f6f7c071340 2024-03-29T11:31:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.089+0000 7f6f72a00700 1 -- 172.21.15.12:0/2652111874 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 128) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f6f6801a400 con 0x7f6f7c071340 2024-03-29T11:31:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.090+0000 7f6f72a00700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f5c06ad20 0x7f6f5c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.090+0000 7f6f72a00700 1 -- 172.21.15.12:0/2652111874 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6555+0+0 (secure 0 0 0) 0x7f6f680897f0 con 0x7f6f7c071340 2024-03-29T11:31:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.090+0000 7f6f7a800700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f5c06ad20 0x7f6f5c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.091+0000 7f6f7a800700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f5c06ad20 0x7f6f5c06d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c019040 tx=0x7f6f6c00bf90 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:14.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.094+0000 7f6f72a00700 1 -- 172.21.15.12:0/2652111874 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f7c0cdd60 con 0x7f6f7c071340 2024-03-29T11:31:14.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:14.319+0000 7f6f81400700 1 -- 172.21.15.12:0/2652111874 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6f7c0cdd60 con 0x7f6f7c071340 2024-03-29T11:31:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.300+0000 7f6f72a00700 1 -- 172.21.15.12:0/2652111874 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v217) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6f7c0cdd60 con 0x7f6f7c071340 2024-03-29T11:31:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.304+0000 7f6f81400700 1 -- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f5c06ad20 msgr2=0x7f6f5c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.304+0000 7f6f81400700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f5c06ad20 0x7f6f5c06d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c019040 tx=0x7f6f6c00bf90 comp rx=0 tx=0).stop 2024-03-29T11:31:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.304+0000 7f6f81400700 1 -- 172.21.15.12:0/2652111874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 msgr2=0x7f6f7c154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.304+0000 7f6f81400700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 0x7f6f7c154180 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800b530 tx=0x7f6f6800b840 comp rx=0 tx=0).stop 2024-03-29T11:31:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.305+0000 7f6f81400700 1 -- 172.21.15.12:0/2652111874 shutdown_connections 2024-03-29T11:31:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.305+0000 7f6f81400700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f5c06ad20 0x7f6f5c06d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.305+0000 7f6f81400700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f7c159d10 0x7f6f7c145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.305+0000 7f6f81400700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f7c070980 0x7f6f7c15b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.305+0000 7f6f81400700 1 --2- 172.21.15.12:0/2652111874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7c071340 0x7f6f7c154180 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.305+0000 7f6f81400700 1 -- 172.21.15.12:0/2652111874 >> 172.21.15.12:0/2652111874 conn(0x7f6f7c0cf210 msgr2=0x7f6f7c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.305+0000 7f6f81400700 1 -- 172.21.15.12:0/2652111874 shutdown_connections 2024-03-29T11:31:15.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.305+0000 7f6f81400700 1 -- 172.21.15.12:0/2652111874 wait complete. 2024-03-29T11:31:15.300 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T11:31:15.319 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T11:31:15.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.535+0000 7f5e65400700 1 Processor -- start 2024-03-29T11:31:15.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.535+0000 7f5e65400700 1 -- start start 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.537+0000 7f5e65400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5e600d0bd0 0x7f5e6015e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.537+0000 7f5e65400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5e600b3120 0x7f5e6006a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.537+0000 7f5e65400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6015ee10 con 0x7f5e6006acd0 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.537+0000 7f5e65400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6006b0e0 con 0x7f5e600b3120 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.537+0000 7f5e65400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6015de10 con 0x7f5e600d0bd0 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.537+0000 7f5e57e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5e600b3120 0x7f5e6006a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.538+0000 7f5e5f200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5e6006acd0 0x7f5e6006dae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34726/0 (socket says 172.21.15.12:34726) 2024-03-29T11:31:15.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.538+0000 7f5e5f200700 1 -- 172.21.15.12:0/2363340123 learned_addr learned my addr 172.21.15.12:0/2363340123 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.538+0000 7f5e57e00700 1 -- 172.21.15.12:0/2363340123 >> v2:172.21.15.159:3300/0 conn(0x7f5e600d0bd0 msgr2=0x7f5e6015e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.538+0000 7f5e57e00700 1 --2- 172.21.15.12:0/2363340123 >> v2:172.21.15.159:3300/0 conn(0x7f5e600d0bd0 0x7f5e6015e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.538+0000 7f5e57e00700 1 -- 172.21.15.12:0/2363340123 >> v1:172.21.15.12:6789/0 conn(0x7f5e6006acd0 legacy=0x7f5e6006dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.538+0000 7f5e57e00700 1 -- 172.21.15.12:0/2363340123 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e600b4fd0 con 0x7f5e600b3120 2024-03-29T11:31:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.538+0000 7f5e57e00700 1 --2- 172.21.15.12:0/2363340123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600b3120 0x7f5e6006a790 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f5e600b2e70 tx=0x7f5e48009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb772db8ccdd4a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.538+0000 7f5e5de00700 1 -- 172.21.15.12:0/2363340123 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48011830 con 0x7f5e600b3120 2024-03-29T11:31:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.538+0000 7f5e5de00700 1 -- 172.21.15.12:0/2363340123 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e48011990 con 0x7f5e600b3120 2024-03-29T11:31:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.539+0000 7f5e5de00700 1 -- 172.21.15.12:0/2363340123 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48011d20 con 0x7f5e600b3120 2024-03-29T11:31:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.539+0000 7f5e65400700 1 -- 172.21.15.12:0/2363340123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600b3120 msgr2=0x7f5e6006a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.539+0000 7f5e65400700 1 --2- 172.21.15.12:0/2363340123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600b3120 0x7f5e6006a790 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f5e600b2e70 tx=0x7f5e48009b20 comp rx=0 tx=0).stop 2024-03-29T11:31:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.539+0000 7f5e65400700 1 -- 172.21.15.12:0/2363340123 shutdown_connections 2024-03-29T11:31:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.539+0000 7f5e65400700 1 --2- 172.21.15.12:0/2363340123 >> v2:172.21.15.159:3300/0 conn(0x7f5e600d0bd0 0x7f5e6015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.539+0000 7f5e65400700 1 --2- 172.21.15.12:0/2363340123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600b3120 0x7f5e6006a790 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.539+0000 7f5e65400700 1 -- 172.21.15.12:0/2363340123 >> 172.21.15.12:0/2363340123 conn(0x7f5e600b5640 msgr2=0x7f5e600b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.539+0000 7f5e65400700 1 -- 172.21.15.12:0/2363340123 shutdown_connections 2024-03-29T11:31:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.539+0000 7f5e65400700 1 -- 172.21.15.12:0/2363340123 wait complete. 2024-03-29T11:31:15.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.541+0000 7f5e65400700 1 Processor -- start 2024-03-29T11:31:15.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.541+0000 7f5e65400700 1 -- start start 2024-03-29T11:31:15.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.542+0000 7f5e65400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e6006acd0 0x7f5e601435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:15.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.542+0000 7f5e65400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e600b3120 0x7f5e60142b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:15.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.542+0000 7f5e65400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600d0bd0 0x7f5e601598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.542+0000 7f5e65400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e601545d0 con 0x7f5e600d0bd0 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.542+0000 7f5e65400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e601548b0 con 0x7f5e6006acd0 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.542+0000 7f5e65400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60154b90 con 0x7f5e600b3120 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.542+0000 7f5e5f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600d0bd0 0x7f5e601598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.542+0000 7f5e5f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600d0bd0 0x7f5e601598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58358/0 (socket says 172.21.15.12:58358) 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.542+0000 7f5e5f200700 1 -- 172.21.15.12:0/146831698 learned_addr learned my addr 172.21.15.12:0/146831698 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e5e800700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e6006acd0 0x7f5e601435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e57e00700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e600b3120 0x7f5e60142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e5f200700 1 -- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e600b3120 msgr2=0x7f5e60142b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e5f200700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e600b3120 0x7f5e60142b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e5f200700 1 -- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e6006acd0 msgr2=0x7f5e601435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e5f200700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e6006acd0 0x7f5e601435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:15.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e5f200700 1 -- 172.21.15.12:0/146831698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e480094a0 con 0x7f5e600d0bd0 2024-03-29T11:31:15.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e5f200700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600d0bd0 0x7f5e601598c0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800cbc0 tx=0x7f5e5800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:15.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e56a00700 1 -- 172.21.15.12:0/146831698 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5800a9e0 con 0x7f5e600d0bd0 2024-03-29T11:31:15.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e56a00700 1 -- 172.21.15.12:0/146831698 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5800ab40 con 0x7f5e600d0bd0 2024-03-29T11:31:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e56a00700 1 -- 172.21.15.12:0/146831698 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58012400 con 0x7f5e600d0bd0 2024-03-29T11:31:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.543+0000 7f5e65400700 1 -- 172.21.15.12:0/146831698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e60154e70 con 0x7f5e600d0bd0 2024-03-29T11:31:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.544+0000 7f5e65400700 1 -- 172.21.15.12:0/146831698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e60143080 con 0x7f5e600d0bd0 2024-03-29T11:31:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.545+0000 7f5e65400700 1 -- 172.21.15.12:0/146831698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e601550b0 con 0x7f5e600d0bd0 2024-03-29T11:31:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.546+0000 7f5e56a00700 1 -- 172.21.15.12:0/146831698 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5e5801b3b0 con 0x7f5e600d0bd0 2024-03-29T11:31:15.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.547+0000 7f5e56a00700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e3c06adb0 0x7f5e3c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:15.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.547+0000 7f5e56a00700 1 -- 172.21.15.12:0/146831698 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6089+0+0 (secure 0 0 0) 0x7f5e58088ee0 con 0x7f5e600d0bd0 2024-03-29T11:31:15.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.547+0000 7f5e5e800700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e3c06adb0 0x7f5e3c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:15.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.548+0000 7f5e5e800700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e3c06adb0 0x7f5e3c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009970 tx=0x7f5e50009110 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:15.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.551+0000 7f5e56a00700 1 -- 172.21.15.12:0/146831698 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e601550b0 con 0x7f5e600d0bd0 2024-03-29T11:31:15.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:15.765+0000 7f5e65400700 1 -- 172.21.15.12:0/146831698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5e6015ee10 con 0x7f5e600d0bd0 2024-03-29T11:31:16.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.440+0000 7f5e56a00700 1 -- 172.21.15.12:0/146831698 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v218) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5e6015ee10 con 0x7f5e600d0bd0 2024-03-29T11:31:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 -- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e3c06adb0 msgr2=0x7f5e3c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e3c06adb0 0x7f5e3c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009970 tx=0x7f5e50009110 comp rx=0 tx=0).stop 2024-03-29T11:31:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 -- 172.21.15.12:0/146831698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600d0bd0 msgr2=0x7f5e601598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600d0bd0 0x7f5e601598c0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800cbc0 tx=0x7f5e5800ced0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 -- 172.21.15.12:0/146831698 shutdown_connections 2024-03-29T11:31:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5e3c06adb0 0x7f5e3c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e600d0bd0 0x7f5e601598c0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e6006acd0 0x7f5e601435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 --2- 172.21.15.12:0/146831698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e600b3120 0x7f5e60142b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 -- 172.21.15.12:0/146831698 >> 172.21.15.12:0/146831698 conn(0x7f5e600b5640 msgr2=0x7f5e6015efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 -- 172.21.15.12:0/146831698 shutdown_connections 2024-03-29T11:31:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.445+0000 7f5e65400700 1 -- 172.21.15.12:0/146831698 wait complete. 2024-03-29T11:31:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T11:31:16.458 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T11:31:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.675+0000 7f320c800700 1 Processor -- start 2024-03-29T11:31:16.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.675+0000 7f320c800700 1 -- start start 2024-03-29T11:31:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.676+0000 7f320c800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f32040cd020 0x7f32040cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.676+0000 7f320c800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32040cdae0 con 0x7f32040b2070 2024-03-29T11:31:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.676+0000 7f320c800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32040cddc0 con 0x7f32040cbb30 2024-03-29T11:31:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.676+0000 7f320c800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32040ce0a0 con 0x7f32040cd020 2024-03-29T11:31:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f320a800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f32040b2070 0x7f320406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34742/0 (socket says 172.21.15.12:34742) 2024-03-29T11:31:16.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f320a800700 1 -- 172.21.15.12:0/1791408925 learned_addr learned my addr 172.21.15.12:0/1791408925 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:16.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3209400700 1 --2- 172.21.15.12:0/1791408925 >> v2:172.21.15.159:3300/0 conn(0x7f32040cd020 0x7f32040cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:16.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617350410 0 0) 0x7f32040cdae0 con 0x7f32040b2070 2024-03-29T11:31:16.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec0035b0 con 0x7f32040b2070 2024-03-29T11:31:16.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892393520 0 0) 0x7f32040cddc0 con 0x7f32040cbb30 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32040cdae0 con 0x7f32040cbb30 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1951210740 0 0) 0x7f31ec0035b0 con 0x7f32040b2070 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 >> v2:172.21.15.159:3300/0 conn(0x7f32040cd020 msgr2=0x7f32040cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 --2- 172.21.15.12:0/1791408925 >> v2:172.21.15.159:3300/0 conn(0x7f32040cd020 0x7f32040cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 >> v1:172.21.15.159:6789/0 conn(0x7f32040cbb30 legacy=0x7f32040cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320406dc20 con 0x7f32040b2070 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f32000019f0 con 0x7f32040b2070 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 >> v1:172.21.15.12:6789/0 conn(0x7f32040b2070 legacy=0x7f320406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.677+0000 7f3208a00700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31ec003be0 0x7f31ec003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3208a00700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ec004850 0x7f31ec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3208a00700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31ec005210 0x7f31ec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:16.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040ce0a0 con 0x7f31ec004850 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec00c0e0 con 0x7f31ec005210 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec00c2a0 con 0x7f31ec003be0 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3209400700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ec004850 0x7f31ec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f320a800700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31ec005210 0x7f31ec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3209e00700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31ec003be0 0x7f31ec003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3209400700 1 -- 172.21.15.12:0/1791408925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31ec003be0 msgr2=0x7f31ec003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3209400700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31ec003be0 0x7f31ec003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3209400700 1 -- 172.21.15.12:0/1791408925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31ec005210 msgr2=0x7f31ec00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3209400700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31ec005210 0x7f31ec00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3209400700 1 -- 172.21.15.12:0/1791408925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec0035b0 con 0x7f31ec004850 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f320a800700 1 -- 172.21.15.12:0/1791408925 reap_dead start 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.678+0000 7f3209e00700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31ec003be0 0x7f31ec003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.679+0000 7f3209400700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ec004850 0x7f31ec004cd0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f31f4009b60 tx=0x7f31ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e551cd50a5b94821 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:16.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.679+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f400e6c0 con 0x7f31ec004850 2024-03-29T11:31:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.679+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f400e820 con 0x7f31ec004850 2024-03-29T11:31:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.679+0000 7f3208a00700 1 -- 172.21.15.12:0/1791408925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f400eb90 con 0x7f31ec004850 2024-03-29T11:31:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.679+0000 7f320c800700 1 -- 172.21.15.12:0/1791408925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ec004850 msgr2=0x7f31ec004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.679+0000 7f320c800700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ec004850 0x7f31ec004cd0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f31f4009b60 tx=0x7f31ec005880 comp rx=0 tx=0).stop 2024-03-29T11:31:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.679+0000 7f320c800700 1 -- 172.21.15.12:0/1791408925 shutdown_connections 2024-03-29T11:31:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.679+0000 7f320c800700 1 --2- 172.21.15.12:0/1791408925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ec004850 0x7f31ec004cd0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.679+0000 7f320c800700 1 -- 172.21.15.12:0/1791408925 >> 172.21.15.12:0/1791408925 conn(0x7f32040b0d70 msgr2=0x7f32040b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.679+0000 7f320c800700 1 -- 172.21.15.12:0/1791408925 shutdown_connections 2024-03-29T11:31:16.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.680+0000 7f320c800700 1 -- 172.21.15.12:0/1791408925 wait complete. 2024-03-29T11:31:16.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.681+0000 7f320c800700 1 Processor -- start 2024-03-29T11:31:16.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.681+0000 7f320c800700 1 -- start start 2024-03-29T11:31:16.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.682+0000 7f320c800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31ec004850 0x7f3204153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:16.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.682+0000 7f320c800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ad090 0x7f32040ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:16.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.682+0000 7f320c800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3204145670 0x7f3204145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:16.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.682+0000 7f320c800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32041542f0 con 0x7f32040ad090 2024-03-29T11:31:16.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.682+0000 7f320c800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040ada50 con 0x7f3204145670 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.682+0000 7f320c800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32041445d0 con 0x7f31ec004850 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.682+0000 7f3209400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ad090 0x7f32040ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.683+0000 7f3209400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ad090 0x7f32040ad510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58372/0 (socket says 172.21.15.12:58372) 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.683+0000 7f3209400700 1 -- 172.21.15.12:0/3534810857 learned_addr learned my addr 172.21.15.12:0/3534810857 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.683+0000 7f3209e00700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31ec004850 0x7f3204153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.683+0000 7f320a800700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3204145670 0x7f3204145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.683+0000 7f3209400700 1 -- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31ec004850 msgr2=0x7f3204153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.683+0000 7f3209400700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31ec004850 0x7f3204153db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.683+0000 7f3209400700 1 -- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3204145670 msgr2=0x7f3204145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.683+0000 7f3209400700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3204145670 0x7f3204145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.683+0000 7f3209400700 1 -- 172.21.15.12:0/3534810857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32041448b0 con 0x7f32040ad090 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.683+0000 7f3209400700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ad090 0x7f32040ad510 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0047d0 tx=0x7f31f40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:16.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.684+0000 7f31fe000700 1 -- 172.21.15.12:0/3534810857 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f400e6c0 con 0x7f32040ad090 2024-03-29T11:31:16.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.684+0000 7f31fe000700 1 -- 172.21.15.12:0/3534810857 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f40038e0 con 0x7f32040ad090 2024-03-29T11:31:16.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.684+0000 7f31fe000700 1 -- 172.21.15.12:0/3534810857 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4003c90 con 0x7f32040ad090 2024-03-29T11:31:16.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.684+0000 7f320c800700 1 -- 172.21.15.12:0/3534810857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3204144b90 con 0x7f32040ad090 2024-03-29T11:31:16.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.684+0000 7f320c800700 1 -- 172.21.15.12:0/3534810857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3204145150 con 0x7f32040ad090 2024-03-29T11:31:16.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.685+0000 7f320c800700 1 -- 172.21.15.12:0/3534810857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32041454c0 con 0x7f32040ad090 2024-03-29T11:31:16.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.686+0000 7f31fe000700 1 -- 172.21.15.12:0/3534810857 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f31f4013470 con 0x7f32040ad090 2024-03-29T11:31:16.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.687+0000 7f31fe000700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f31e806adb0 0x7f31e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:16.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.687+0000 7f31fe000700 1 -- 172.21.15.12:0/3534810857 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5617+0+0 (secure 0 0 0) 0x7f31f408ab20 con 0x7f32040ad090 2024-03-29T11:31:16.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.687+0000 7f3209e00700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f31e806adb0 0x7f31e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:16.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.688+0000 7f3209e00700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f31e806adb0 0x7f31e806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31f8007b70 tx=0x7f31f8009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:16.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.691+0000 7f31fe000700 1 -- 172.21.15.12:0/3534810857 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31f4055d50 con 0x7f32040ad090 2024-03-29T11:31:16.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.910+0000 7f320c800700 1 -- 172.21.15.12:0/3534810857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f32041542f0 con 0x7f32040ad090 2024-03-29T11:31:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.914+0000 7f31fe000700 1 -- 172.21.15.12:0/3534810857 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v73) v1 ==== 115+0+0 (secure 0 0 0) 0x7f31f40170b0 con 0x7f32040ad090 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 -- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f31e806adb0 msgr2=0x7f31e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f31e806adb0 0x7f31e806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31f8007b70 tx=0x7f31f8009040 comp rx=0 tx=0).stop 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 -- 172.21.15.12:0/3534810857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ad090 msgr2=0x7f32040ad510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ad090 0x7f32040ad510 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0047d0 tx=0x7f31f40047c0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 -- 172.21.15.12:0/3534810857 shutdown_connections 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f31e806adb0 0x7f31e806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3204145670 0x7f3204145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31ec004850 0x7f3204153db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 --2- 172.21.15.12:0/3534810857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ad090 0x7f32040ad510 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 -- 172.21.15.12:0/3534810857 >> 172.21.15.12:0/3534810857 conn(0x7f32040b0d70 msgr2=0x7f32040b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 -- 172.21.15.12:0/3534810857 shutdown_connections 2024-03-29T11:31:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:16.917+0000 7f320c800700 1 -- 172.21.15.12:0/3534810857 wait complete. 2024-03-29T11:31:16.929 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T11:31:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.134+0000 7fe768400700 1 Processor -- start 2024-03-29T11:31:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.134+0000 7fe768400700 1 -- start start 2024-03-29T11:31:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.135+0000 7fe768400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe760160e50 0x7fe7600b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.135+0000 7fe768400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe7600d3f20 0x7fe7600b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.135+0000 7fe768400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600cef00 con 0x7fe7600b3f80 2024-03-29T11:31:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.135+0000 7fe768400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe760161230 con 0x7fe760160e50 2024-03-29T11:31:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.135+0000 7fe768400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7600d2a80 con 0x7fe7600d3f20 2024-03-29T11:31:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.135+0000 7fe765a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe760160e50 0x7fe7600b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.135+0000 7fe766400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe7600b3f80 0x7fe7600b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34744/0 (socket says 172.21.15.12:34744) 2024-03-29T11:31:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.135+0000 7fe766400700 1 -- 172.21.15.12:0/3956831491 learned_addr learned my addr 172.21.15.12:0/3956831491 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:17.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.136+0000 7fe765000700 1 --2- 172.21.15.12:0/3956831491 >> v2:172.21.15.159:3300/0 conn(0x7fe7600d3f20 0x7fe7600b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.136+0000 7fe765a00700 1 -- 172.21.15.12:0/3956831491 >> v2:172.21.15.159:3300/0 conn(0x7fe7600d3f20 msgr2=0x7fe7600b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.136+0000 7fe765a00700 1 --2- 172.21.15.12:0/3956831491 >> v2:172.21.15.159:3300/0 conn(0x7fe7600d3f20 0x7fe7600b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.136+0000 7fe765a00700 1 -- 172.21.15.12:0/3956831491 >> v1:172.21.15.12:6789/0 conn(0x7fe7600b3f80 legacy=0x7fe7600b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.136+0000 7fe765a00700 1 -- 172.21.15.12:0/3956831491 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600d43b0 con 0x7fe760160e50 2024-03-29T11:31:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.136+0000 7fe765a00700 1 --2- 172.21.15.12:0/3956831491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 0x7fe7600b4e90 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fe76008b910 tx=0x7fe754009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335bb90f483724f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.136+0000 7fe75fe00700 1 -- 172.21.15.12:0/3956831491 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754011830 con 0x7fe760160e50 2024-03-29T11:31:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.136+0000 7fe75fe00700 1 -- 172.21.15.12:0/3956831491 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe754011990 con 0x7fe760160e50 2024-03-29T11:31:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.136+0000 7fe75fe00700 1 -- 172.21.15.12:0/3956831491 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754011d20 con 0x7fe760160e50 2024-03-29T11:31:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.137+0000 7fe768400700 1 -- 172.21.15.12:0/3956831491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 msgr2=0x7fe7600b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.137+0000 7fe768400700 1 --2- 172.21.15.12:0/3956831491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 0x7fe7600b4e90 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fe76008b910 tx=0x7fe754009b20 comp rx=0 tx=0).stop 2024-03-29T11:31:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.137+0000 7fe768400700 1 -- 172.21.15.12:0/3956831491 shutdown_connections 2024-03-29T11:31:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.137+0000 7fe768400700 1 --2- 172.21.15.12:0/3956831491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 0x7fe7600b4e90 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.137+0000 7fe768400700 1 --2- 172.21.15.12:0/3956831491 >> v2:172.21.15.159:3300/0 conn(0x7fe7600d3f20 0x7fe7600b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.137+0000 7fe768400700 1 -- 172.21.15.12:0/3956831491 >> 172.21.15.12:0/3956831491 conn(0x7fe76008aff0 msgr2=0x7fe7600d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.137+0000 7fe768400700 1 -- 172.21.15.12:0/3956831491 shutdown_connections 2024-03-29T11:31:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.137+0000 7fe768400700 1 -- 172.21.15.12:0/3956831491 wait complete. 2024-03-29T11:31:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.139+0000 7fe768400700 1 Processor -- start 2024-03-29T11:31:17.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.139+0000 7fe768400700 1 -- start start 2024-03-29T11:31:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe768400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7600b3f80 0x7fe7601f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe768400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 0x7fe7600b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe768400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7601f41b0 0x7fe7600b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe768400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600ca950 con 0x7fe760160e50 2024-03-29T11:31:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe768400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601f6340 con 0x7fe7601f41b0 2024-03-29T11:31:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe768400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600b7180 con 0x7fe7600b3f80 2024-03-29T11:31:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe765000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 0x7fe7600b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe765000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 0x7fe7600b4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58398/0 (socket says 172.21.15.12:58398) 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe765000700 1 -- 172.21.15.12:0/2950719764 learned_addr learned my addr 172.21.15.12:0/2950719764 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe765a00700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7600b3f80 0x7fe7601f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.140+0000 7fe766400700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7601f41b0 0x7fe7600b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe765000700 1 -- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7600b3f80 msgr2=0x7fe7601f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe765000700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7600b3f80 0x7fe7601f5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe765000700 1 -- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7601f41b0 msgr2=0x7fe7600b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe765000700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7601f41b0 0x7fe7600b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe765000700 1 -- 172.21.15.12:0/2950719764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7540094a0 con 0x7fe760160e50 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe766400700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7601f41b0 0x7fe7600b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe765000700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 0x7fe7600b4d70 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00e810 tx=0x7fe74c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe75d600700 1 -- 172.21.15.12:0/2950719764 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c013610 con 0x7fe760160e50 2024-03-29T11:31:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe75d600700 1 -- 172.21.15.12:0/2950719764 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c013770 con 0x7fe760160e50 2024-03-29T11:31:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe768400700 1 -- 172.21.15.12:0/2950719764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7600b7460 con 0x7fe760160e50 2024-03-29T11:31:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe75d600700 1 -- 172.21.15.12:0/2950719764 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c013b20 con 0x7fe760160e50 2024-03-29T11:31:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.141+0000 7fe768400700 1 -- 172.21.15.12:0/2950719764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7600b7a20 con 0x7fe760160e50 2024-03-29T11:31:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.142+0000 7fe768400700 1 -- 172.21.15.12:0/2950719764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76006d010 con 0x7fe760160e50 2024-03-29T11:31:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.143+0000 7fe75d600700 1 -- 172.21.15.12:0/2950719764 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe74c013c80 con 0x7fe760160e50 2024-03-29T11:31:17.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.144+0000 7fe75d600700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe74006adb0 0x7fe74006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.144+0000 7fe75d600700 1 -- 172.21.15.12:0/2950719764 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fe74c089250 con 0x7fe760160e50 2024-03-29T11:31:17.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.145+0000 7fe765a00700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe74006adb0 0x7fe74006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.145+0000 7fe765a00700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe74006adb0 0x7fe74006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe7540090b0 tx=0x7fe754008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:17.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.148+0000 7fe75d600700 1 -- 172.21.15.12:0/2950719764 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe74c0541d0 con 0x7fe760160e50 2024-03-29T11:31:17.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.361+0000 7fe768400700 1 -- 172.21.15.12:0/2950719764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe760067720 con 0x7fe760160e50 2024-03-29T11:31:17.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.365+0000 7fe75d600700 1 -- 172.21.15.12:0/2950719764 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe74c0577f0 con 0x7fe760160e50 2024-03-29T11:31:17.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 -- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe74006adb0 msgr2=0x7fe74006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe74006adb0 0x7fe74006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe7540090b0 tx=0x7fe754008030 comp rx=0 tx=0).stop 2024-03-29T11:31:17.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 -- 172.21.15.12:0/2950719764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 msgr2=0x7fe7600b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 0x7fe7600b4d70 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00e810 tx=0x7fe74c00eb20 comp rx=0 tx=0).stop 2024-03-29T11:31:17.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 -- 172.21.15.12:0/2950719764 shutdown_connections 2024-03-29T11:31:17.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe74006adb0 0x7fe74006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7601f41b0 0x7fe7600b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7600b3f80 0x7fe7601f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 --2- 172.21.15.12:0/2950719764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe760160e50 0x7fe7600b4d70 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 -- 172.21.15.12:0/2950719764 >> 172.21.15.12:0/2950719764 conn(0x7fe76008aff0 msgr2=0x7fe7600cfa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:17.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 -- 172.21.15.12:0/2950719764 shutdown_connections 2024-03-29T11:31:17.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.368+0000 7fe768400700 1 -- 172.21.15.12:0/2950719764 wait complete. 2024-03-29T11:31:17.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T11:31:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.585+0000 7f046a000700 1 Processor -- start 2024-03-29T11:31:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.586+0000 7f046a000700 1 -- start start 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f046a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f046415a5c0 0x7f04640709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f046a000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f046415dfe0 0x7f04640ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f046a000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640cb310 con 0x7f04640ce670 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f046a000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04640ceae0 con 0x7f046415a5c0 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f046a000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0464070f00 con 0x7f046415dfe0 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f0463400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f046415a5c0 0x7f04640709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f0463400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f046415a5c0 0x7f04640709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58410/0 (socket says 172.21.15.12:58410) 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f0463400700 1 -- 172.21.15.12:0/2257949355 learned_addr learned my addr 172.21.15.12:0/2257949355 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f0462a00700 1 --2- 172.21.15.12:0/2257949355 >> v2:172.21.15.159:3300/0 conn(0x7f046415dfe0 0x7f04640ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f0463400700 1 -- 172.21.15.12:0/2257949355 >> v2:172.21.15.159:3300/0 conn(0x7f046415dfe0 msgr2=0x7f04640ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.587+0000 7f0463400700 1 --2- 172.21.15.12:0/2257949355 >> v2:172.21.15.159:3300/0 conn(0x7f046415dfe0 0x7f04640ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.588+0000 7f0463400700 1 -- 172.21.15.12:0/2257949355 >> v1:172.21.15.159:6789/0 conn(0x7f04640ce670 legacy=0x7f04640b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.588+0000 7f0463400700 1 -- 172.21.15.12:0/2257949355 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04640cca90 con 0x7f046415a5c0 2024-03-29T11:31:17.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.588+0000 7f0463400700 1 --2- 172.21.15.12:0/2257949355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415a5c0 0x7f04640709c0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f04640d0220 tx=0x7f0454009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41e9ce2db249aa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:17.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.588+0000 7f0462000700 1 -- 172.21.15.12:0/2257949355 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045400b870 con 0x7f046415a5c0 2024-03-29T11:31:17.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.588+0000 7f0462000700 1 -- 172.21.15.12:0/2257949355 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045400b9d0 con 0x7f046415a5c0 2024-03-29T11:31:17.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.588+0000 7f0462000700 1 -- 172.21.15.12:0/2257949355 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045400bd60 con 0x7f046415a5c0 2024-03-29T11:31:17.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.589+0000 7f046a000700 1 -- 172.21.15.12:0/2257949355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415a5c0 msgr2=0x7f04640709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.589+0000 7f046a000700 1 --2- 172.21.15.12:0/2257949355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415a5c0 0x7f04640709c0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f04640d0220 tx=0x7f0454009bd0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.589+0000 7f046a000700 1 -- 172.21.15.12:0/2257949355 shutdown_connections 2024-03-29T11:31:17.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.589+0000 7f046a000700 1 --2- 172.21.15.12:0/2257949355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f046415a5c0 0x7f04640709c0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.589+0000 7f046a000700 1 --2- 172.21.15.12:0/2257949355 >> v2:172.21.15.159:3300/0 conn(0x7f046415dfe0 0x7f04640ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.589+0000 7f046a000700 1 -- 172.21.15.12:0/2257949355 >> 172.21.15.12:0/2257949355 conn(0x7f04640b0d70 msgr2=0x7f04640b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:17.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.589+0000 7f046a000700 1 -- 172.21.15.12:0/2257949355 shutdown_connections 2024-03-29T11:31:17.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.589+0000 7f046a000700 1 -- 172.21.15.12:0/2257949355 wait complete. 2024-03-29T11:31:17.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.590+0000 7f046a000700 1 Processor -- start 2024-03-29T11:31:17.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.591+0000 7f046a000700 1 -- start start 2024-03-29T11:31:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.591+0000 7f046a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04640ce670 0x7f04640c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f046a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f046415a5c0 0x7f0464147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f046a000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046415dfe0 0x7f04640affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f046a000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0464147ea0 con 0x7f04640ce670 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f046a000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640b0500 con 0x7f046415dfe0 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f046a000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04640c9690 con 0x7f046415a5c0 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04640ce670 0x7f04640c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04640ce670 0x7f04640c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58420/0 (socket says 172.21.15.12:58420) 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463400700 1 -- 172.21.15.12:0/802368218 learned_addr learned my addr 172.21.15.12:0/802368218 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0462a00700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f046415a5c0 0x7f0464147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463e00700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046415dfe0 0x7f04640affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463400700 1 -- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f046415a5c0 msgr2=0x7f0464147960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463400700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f046415a5c0 0x7f0464147960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463400700 1 -- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046415dfe0 msgr2=0x7f04640affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463400700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046415dfe0 0x7f04640affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463400700 1 -- 172.21.15.12:0/802368218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0454009550 con 0x7f04640ce670 2024-03-29T11:31:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0462a00700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f046415a5c0 0x7f0464147960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463e00700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046415dfe0 0x7f04640affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.592+0000 7f0463400700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04640ce670 0x7f04640c83f0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f046406c130 tx=0x7f0454004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.593+0000 7f045b400700 1 -- 172.21.15.12:0/802368218 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454004540 con 0x7f04640ce670 2024-03-29T11:31:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.593+0000 7f045b400700 1 -- 172.21.15.12:0/802368218 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04540036a0 con 0x7f04640ce670 2024-03-29T11:31:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.593+0000 7f045b400700 1 -- 172.21.15.12:0/802368218 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454003a30 con 0x7f04640ce670 2024-03-29T11:31:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.593+0000 7f046a000700 1 -- 172.21.15.12:0/802368218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04640c8a80 con 0x7f04640ce670 2024-03-29T11:31:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.593+0000 7f046a000700 1 -- 172.21.15.12:0/802368218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04640c9040 con 0x7f04640ce670 2024-03-29T11:31:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.594+0000 7f046a000700 1 -- 172.21.15.12:0/802368218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0464067f20 con 0x7f04640ce670 2024-03-29T11:31:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.595+0000 7f045b400700 1 -- 172.21.15.12:0/802368218 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0454021070 con 0x7f04640ce670 2024-03-29T11:31:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.596+0000 7f045b400700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f044406adb0 0x7f044406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.596+0000 7f045b400700 1 -- 172.21.15.12:0/802368218 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5617+0+0 (secure 0 0 0) 0x7f045408ab80 con 0x7f04640ce670 2024-03-29T11:31:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.596+0000 7f0462a00700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f044406adb0 0x7f044406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.597+0000 7f0462a00700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f044406adb0 0x7f044406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f046414a1f0 tx=0x7f0450008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:17.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.600+0000 7f045b400700 1 -- 172.21.15.12:0/802368218 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0454055c90 con 0x7f04640ce670 2024-03-29T11:31:17.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.812+0000 7f046a000700 1 -- 172.21.15.12:0/802368218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f0464067720 con 0x7f04640ce670 2024-03-29T11:31:17.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.816+0000 7f045b400700 1 -- 172.21.15.12:0/802368218 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v75) v1 ==== 120+0+0 (secure 0 0 0) 0x7f04540150b0 con 0x7f04640ce670 2024-03-29T11:31:17.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.818+0000 7f046a000700 1 -- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f044406adb0 msgr2=0x7f044406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.818+0000 7f046a000700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f044406adb0 0x7f044406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f046414a1f0 tx=0x7f0450008040 comp rx=0 tx=0).stop 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.818+0000 7f046a000700 1 -- 172.21.15.12:0/802368218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04640ce670 msgr2=0x7f04640c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.818+0000 7f046a000700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04640ce670 0x7f04640c83f0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f046406c130 tx=0x7f0454004750 comp rx=0 tx=0).stop 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.818+0000 7f046a000700 1 -- 172.21.15.12:0/802368218 shutdown_connections 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.818+0000 7f046a000700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f044406adb0 0x7f044406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.818+0000 7f046a000700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f046415dfe0 0x7f04640affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.818+0000 7f046a000700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04640ce670 0x7f04640c83f0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.818+0000 7f046a000700 1 --2- 172.21.15.12:0/802368218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f046415a5c0 0x7f0464147960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.818+0000 7f046a000700 1 -- 172.21.15.12:0/802368218 >> 172.21.15.12:0/802368218 conn(0x7f04640b0d70 msgr2=0x7f04641593e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.819+0000 7f046a000700 1 -- 172.21.15.12:0/802368218 shutdown_connections 2024-03-29T11:31:17.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:17.819+0000 7f046a000700 1 -- 172.21.15.12:0/802368218 wait complete. 2024-03-29T11:31:17.831 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-03-29T11:31:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.036+0000 7fcd5ae00700 1 Processor -- start 2024-03-29T11:31:18.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.036+0000 7fcd5ae00700 1 -- start start 2024-03-29T11:31:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.037+0000 7fcd5ae00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 0x7fcd540b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.037+0000 7fcd5ae00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcd540b2e50 0x7fcd540709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.037+0000 7fcd5ae00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd540b2180 con 0x7fcd540b3af0 2024-03-29T11:31:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.037+0000 7fcd5ae00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd5415b100 con 0x7fcd540b2e50 2024-03-29T11:31:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.037+0000 7fcd5ae00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5415b3e0 con 0x7fcd54072580 2024-03-29T11:31:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.037+0000 7fcd53e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcd540b3af0 0x7fcd5415a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34758/0 (socket says 172.21.15.12:34758) 2024-03-29T11:31:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.037+0000 7fcd53e00700 1 -- 172.21.15.12:0/2139964027 learned_addr learned my addr 172.21.15.12:0/2139964027 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.037+0000 7fcd53400700 1 --2- 172.21.15.12:0/2139964027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 0x7fcd540b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.037+0000 7fcd58e00700 1 --2- 172.21.15.12:0/2139964027 >> v2:172.21.15.159:3300/0 conn(0x7fcd540b2e50 0x7fcd540709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.038+0000 7fcd52a00700 1 -- 172.21.15.12:0/2139964027 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216689952 0 0) 0x7fcd540b2180 con 0x7fcd540b3af0 2024-03-29T11:31:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.038+0000 7fcd52a00700 1 -- 172.21.15.12:0/2139964027 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd380035b0 con 0x7fcd540b3af0 2024-03-29T11:31:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.038+0000 7fcd53400700 1 -- 172.21.15.12:0/2139964027 >> v2:172.21.15.159:3300/0 conn(0x7fcd540b2e50 msgr2=0x7fcd540709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.038+0000 7fcd53400700 1 --2- 172.21.15.12:0/2139964027 >> v2:172.21.15.159:3300/0 conn(0x7fcd540b2e50 0x7fcd540709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.038+0000 7fcd53400700 1 -- 172.21.15.12:0/2139964027 >> v1:172.21.15.12:6789/0 conn(0x7fcd540b3af0 legacy=0x7fcd5415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.038+0000 7fcd53400700 1 -- 172.21.15.12:0/2139964027 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd540b23f0 con 0x7fcd54072580 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.038+0000 7fcd53400700 1 --2- 172.21.15.12:0/2139964027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 0x7fcd540b2910 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009840 tx=0x7fcd40009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff8d6fcb3f1b4b98 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.038+0000 7fcd52a00700 1 -- 172.21.15.12:0/2139964027 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40011830 con 0x7fcd54072580 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.038+0000 7fcd52a00700 1 -- 172.21.15.12:0/2139964027 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd40011990 con 0x7fcd54072580 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.039+0000 7fcd52a00700 1 -- 172.21.15.12:0/2139964027 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40011d20 con 0x7fcd54072580 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.039+0000 7fcd5ae00700 1 -- 172.21.15.12:0/2139964027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 msgr2=0x7fcd540b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.039+0000 7fcd5ae00700 1 --2- 172.21.15.12:0/2139964027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 0x7fcd540b2910 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009840 tx=0x7fcd40009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.039+0000 7fcd5ae00700 1 -- 172.21.15.12:0/2139964027 shutdown_connections 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.039+0000 7fcd5ae00700 1 --2- 172.21.15.12:0/2139964027 >> v2:172.21.15.159:3300/0 conn(0x7fcd540b2e50 0x7fcd540709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.039+0000 7fcd5ae00700 1 --2- 172.21.15.12:0/2139964027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 0x7fcd540b2910 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.039+0000 7fcd5ae00700 1 -- 172.21.15.12:0/2139964027 >> 172.21.15.12:0/2139964027 conn(0x7fcd540cf190 msgr2=0x7fcd540cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.039+0000 7fcd5ae00700 1 -- 172.21.15.12:0/2139964027 shutdown_connections 2024-03-29T11:31:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.039+0000 7fcd5ae00700 1 -- 172.21.15.12:0/2139964027 wait complete. 2024-03-29T11:31:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.041+0000 7fcd5ae00700 1 Processor -- start 2024-03-29T11:31:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.041+0000 7fcd5ae00700 1 -- start start 2024-03-29T11:31:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.042+0000 7fcd5ae00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 0x7fcd54152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.042+0000 7fcd5ae00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd540b2e50 0x7fcd54153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.042+0000 7fcd5ae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd540b3af0 0x7fcd54159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.042+0000 7fcd5ae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd540ccbe0 con 0x7fcd540b3af0 2024-03-29T11:31:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.042+0000 7fcd5ae00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd54153410 con 0x7fcd540b2e50 2024-03-29T11:31:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.042+0000 7fcd5ae00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd540cbf60 con 0x7fcd54072580 2024-03-29T11:31:18.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.042+0000 7fcd58e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd540b3af0 0x7fcd54159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.043+0000 7fcd53e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 0x7fcd54152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.042+0000 7fcd58e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd540b3af0 0x7fcd54159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58422/0 (socket says 172.21.15.12:58422) 2024-03-29T11:31:18.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.043+0000 7fcd58e00700 1 -- 172.21.15.12:0/3855715065 learned_addr learned my addr 172.21.15.12:0/3855715065 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:18.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.043+0000 7fcd53400700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd540b2e50 0x7fcd54153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.043+0000 7fcd58e00700 1 -- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 msgr2=0x7fcd54152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.043+0000 7fcd58e00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 0x7fcd54152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.043+0000 7fcd58e00700 1 -- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd540b2e50 msgr2=0x7fcd54153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.043+0000 7fcd58e00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd540b2e50 0x7fcd54153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.043+0000 7fcd58e00700 1 -- 172.21.15.12:0/3855715065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd400094a0 con 0x7fcd540b3af0 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.043+0000 7fcd58e00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd540b3af0 0x7fcd54159440 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c00efc0 tx=0x7fcd4c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.043+0000 7fcd4be00700 1 -- 172.21.15.12:0/3855715065 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c009ea0 con 0x7fcd540b3af0 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.044+0000 7fcd4be00700 1 -- 172.21.15.12:0/3855715065 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4c00f040 con 0x7fcd540b3af0 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.044+0000 7fcd4be00700 1 -- 172.21.15.12:0/3855715065 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c00ced0 con 0x7fcd540b3af0 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.044+0000 7fcd5ae00700 1 -- 172.21.15.12:0/3855715065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd540cc240 con 0x7fcd540b3af0 2024-03-29T11:31:18.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.044+0000 7fcd5ae00700 1 -- 172.21.15.12:0/3855715065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd540cc800 con 0x7fcd540b3af0 2024-03-29T11:31:18.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.045+0000 7fcd5ae00700 1 -- 172.21.15.12:0/3855715065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd540ccd20 con 0x7fcd540b3af0 2024-03-29T11:31:18.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.046+0000 7fcd4be00700 1 -- 172.21.15.12:0/3855715065 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcd4c014480 con 0x7fcd540b3af0 2024-03-29T11:31:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.047+0000 7fcd4be00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcd3406adb0 0x7fcd3406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.047+0000 7fcd4be00700 1 -- 172.21.15.12:0/3855715065 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fcd4c089e70 con 0x7fcd540b3af0 2024-03-29T11:31:18.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.047+0000 7fcd53e00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcd3406adb0 0x7fcd3406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.048+0000 7fcd53e00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcd3406adb0 0x7fcd3406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcd44007fd0 tx=0x7fcd44007dc0 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:18.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.051+0000 7fcd4be00700 1 -- 172.21.15.12:0/3855715065 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd540ccd20 con 0x7fcd540b3af0 2024-03-29T11:31:18.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.345+0000 7fcd5ae00700 1 -- 172.21.15.12:0/3855715065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7fcd540ccd20 con 0x7fcd540b3af0 2024-03-29T11:31:18.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.443+0000 7fcd4be00700 1 -- 172.21.15.12:0/3855715065 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3143) v1 ==== 163+0+0 (secure 0 0 0) 0x7fcd540ccd20 con 0x7fcd540b3af0 2024-03-29T11:31:18.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.445+0000 7fcd5ae00700 1 -- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcd3406adb0 msgr2=0x7fcd3406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:18.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.445+0000 7fcd5ae00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcd3406adb0 0x7fcd3406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcd44007fd0 tx=0x7fcd44007dc0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.445+0000 7fcd5ae00700 1 -- 172.21.15.12:0/3855715065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd540b3af0 msgr2=0x7fcd54159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.445+0000 7fcd5ae00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd540b3af0 0x7fcd54159440 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c00efc0 tx=0x7fcd4c00c3a0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.445+0000 7fcd5ae00700 1 -- 172.21.15.12:0/3855715065 shutdown_connections 2024-03-29T11:31:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.445+0000 7fcd5ae00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcd3406adb0 0x7fcd3406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.445+0000 7fcd5ae00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd540b3af0 0x7fcd54159440 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.445+0000 7fcd5ae00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd54072580 0x7fcd54152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.445+0000 7fcd5ae00700 1 --2- 172.21.15.12:0/3855715065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd540b2e50 0x7fcd54153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.445+0000 7fcd5ae00700 1 -- 172.21.15.12:0/3855715065 >> 172.21.15.12:0/3855715065 conn(0x7fcd540cf190 msgr2=0x7fcd540cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.446+0000 7fcd5ae00700 1 -- 172.21.15.12:0/3855715065 shutdown_connections 2024-03-29T11:31:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.446+0000 7fcd5ae00700 1 -- 172.21.15.12:0/3855715065 wait complete. 2024-03-29T11:31:18.458 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2024-03-29T11:31:18.458 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2024-03-29T11:31:18.458 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-03-29T11:31:18.459 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-03-29T11:31:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.664+0000 7f9ea0400700 1 Processor -- start 2024-03-29T11:31:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.664+0000 7f9ea0400700 1 -- start start 2024-03-29T11:31:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.665+0000 7f9ea0400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e9815d640 0x7f9e980d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.665+0000 7f9ea0400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9e980ceec0 0x7f9e981612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.665+0000 7f9ea0400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e980d30d0 con 0x7f9e98161800 2024-03-29T11:31:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.665+0000 7f9ea0400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e980b6840 con 0x7f9e9815d640 2024-03-29T11:31:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.665+0000 7f9ea0400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e98161c20 con 0x7f9e980ceec0 2024-03-29T11:31:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.665+0000 7f9e9da00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e9815d640 0x7f9e980d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.665+0000 7f9e9da00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e9815d640 0x7f9e980d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58426/0 (socket says 172.21.15.12:58426) 2024-03-29T11:31:18.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.665+0000 7f9e9da00700 1 -- 172.21.15.12:0/3076598661 learned_addr learned my addr 172.21.15.12:0/3076598661 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:18.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.665+0000 7f9e9d000700 1 --2- 172.21.15.12:0/3076598661 >> v2:172.21.15.159:3300/0 conn(0x7f9e980ceec0 0x7f9e981612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.666+0000 7f9e9da00700 1 -- 172.21.15.12:0/3076598661 >> v2:172.21.15.159:3300/0 conn(0x7f9e980ceec0 msgr2=0x7f9e981612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:18.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.666+0000 7f9e9da00700 1 --2- 172.21.15.12:0/3076598661 >> v2:172.21.15.159:3300/0 conn(0x7f9e980ceec0 0x7f9e981612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.666+0000 7f9e9da00700 1 -- 172.21.15.12:0/3076598661 >> v1:172.21.15.159:6789/0 conn(0x7f9e98161800 legacy=0x7f9e980cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:18.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.666+0000 7f9e9da00700 1 -- 172.21.15.12:0/3076598661 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e980d2880 con 0x7f9e9815d640 2024-03-29T11:31:18.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.666+0000 7f9e9da00700 1 --2- 172.21.15.12:0/3076598661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 0x7f9e980d2340 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f9e980cfbf0 tx=0x7f9e8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=550952b45a05a380 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:18.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.666+0000 7f9e97e00700 1 -- 172.21.15.12:0/3076598661 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c011830 con 0x7f9e9815d640 2024-03-29T11:31:18.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.666+0000 7f9e97e00700 1 -- 172.21.15.12:0/3076598661 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c011990 con 0x7f9e9815d640 2024-03-29T11:31:18.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.666+0000 7f9e97e00700 1 -- 172.21.15.12:0/3076598661 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c011d20 con 0x7f9e9815d640 2024-03-29T11:31:18.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.667+0000 7f9ea0400700 1 -- 172.21.15.12:0/3076598661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 msgr2=0x7f9e980d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:18.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.667+0000 7f9ea0400700 1 --2- 172.21.15.12:0/3076598661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 0x7f9e980d2340 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f9e980cfbf0 tx=0x7f9e8c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.667+0000 7f9ea0400700 1 -- 172.21.15.12:0/3076598661 shutdown_connections 2024-03-29T11:31:18.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.667+0000 7f9ea0400700 1 --2- 172.21.15.12:0/3076598661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 0x7f9e980d2340 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.667+0000 7f9ea0400700 1 --2- 172.21.15.12:0/3076598661 >> v2:172.21.15.159:3300/0 conn(0x7f9e980ceec0 0x7f9e981612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.667+0000 7f9ea0400700 1 -- 172.21.15.12:0/3076598661 >> 172.21.15.12:0/3076598661 conn(0x7f9e9808aff0 msgr2=0x7f9e980d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:18.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.667+0000 7f9ea0400700 1 -- 172.21.15.12:0/3076598661 shutdown_connections 2024-03-29T11:31:18.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.667+0000 7f9ea0400700 1 -- 172.21.15.12:0/3076598661 wait complete. 2024-03-29T11:31:18.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.669+0000 7f9ea0400700 1 Processor -- start 2024-03-29T11:31:18.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.669+0000 7f9ea0400700 1 -- start start 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9ea0400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e980ceec0 0x7f9e9815bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9ea0400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 0x7f9e980a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9ea0400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e98161800 0x7f9e981f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9ea0400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e980a1260 con 0x7f9e9815d640 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9ea0400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9809e4e0 con 0x7f9e98161800 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9ea0400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9809e7c0 con 0x7f9e980ceec0 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 0x7f9e980a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 0x7f9e980a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58436/0 (socket says 172.21.15.12:58436) 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9d000700 1 -- 172.21.15.12:0/2096117727 learned_addr learned my addr 172.21.15.12:0/2096117727 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9da00700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e980ceec0 0x7f9e9815bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9e400700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e98161800 0x7f9e981f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9d000700 1 -- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e980ceec0 msgr2=0x7f9e9815bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9d000700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e980ceec0 0x7f9e9815bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9d000700 1 -- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e98161800 msgr2=0x7f9e981f42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9d000700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e98161800 0x7f9e981f42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9d000700 1 -- 172.21.15.12:0/2096117727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c009550 con 0x7f9e9815d640 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.670+0000 7f9e9e400700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e98161800 0x7f9e981f42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.671+0000 7f9e9d000700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 0x7f9e980a0d20 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400e870 tx=0x7f9e8400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.671+0000 7f9e95600700 1 -- 172.21.15.12:0/2096117727 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84009590 con 0x7f9e9815d640 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.671+0000 7f9e95600700 1 -- 172.21.15.12:0/2096117727 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e84013680 con 0x7f9e9815d640 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.671+0000 7f9ea0400700 1 -- 172.21.15.12:0/2096117727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9809eaa0 con 0x7f9e9815d640 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.671+0000 7f9e95600700 1 -- 172.21.15.12:0/2096117727 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840139f0 con 0x7f9e9815d640 2024-03-29T11:31:18.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.671+0000 7f9ea0400700 1 -- 172.21.15.12:0/2096117727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9809f060 con 0x7f9e9815d640 2024-03-29T11:31:18.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.672+0000 7f9ea0400700 1 -- 172.21.15.12:0/2096117727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9809f6b0 con 0x7f9e9815d640 2024-03-29T11:31:18.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.673+0000 7f9e95600700 1 -- 172.21.15.12:0/2096117727 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9e84013b50 con 0x7f9e9815d640 2024-03-29T11:31:18.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.674+0000 7f9e95600700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e7806adb0 0x7f9e7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:18.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.674+0000 7f9e95600700 1 -- 172.21.15.12:0/2096117727 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5617+0+0 (secure 0 0 0) 0x7f9e84089430 con 0x7f9e9815d640 2024-03-29T11:31:18.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.674+0000 7f9e9da00700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e7806adb0 0x7f9e7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:18.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.675+0000 7f9e9da00700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e7806adb0 0x7f9e7806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009c10 tx=0x7f9e8c008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:18.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.678+0000 7f9e95600700 1 -- 172.21.15.12:0/2096117727 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e840544b0 con 0x7f9e9815d640 2024-03-29T11:31:18.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:18.978+0000 7f9ea0400700 1 -- 172.21.15.12:0/2096117727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f9e98067720 con 0x7f9e9815d640 2024-03-29T11:31:19.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.452+0000 7f9e95600700 1 -- 172.21.15.12:0/2096117727 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3144) v1 ==== 189+0+0 (secure 0 0 0) 0x7f9e84057ad0 con 0x7f9e9815d640 2024-03-29T11:31:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 -- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e7806adb0 msgr2=0x7f9e7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e7806adb0 0x7f9e7806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009c10 tx=0x7f9e8c008030 comp rx=0 tx=0).stop 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 -- 172.21.15.12:0/2096117727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 msgr2=0x7f9e980a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 0x7f9e980a0d20 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400e870 tx=0x7f9e8400eb80 comp rx=0 tx=0).stop 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 -- 172.21.15.12:0/2096117727 shutdown_connections 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e7806adb0 0x7f9e7806d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e98161800 0x7f9e981f42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e980ceec0 0x7f9e9815bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 --2- 172.21.15.12:0/2096117727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9815d640 0x7f9e980a0d20 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 -- 172.21.15.12:0/2096117727 >> 172.21.15.12:0/2096117727 conn(0x7f9e9808aff0 msgr2=0x7f9e980b8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 -- 172.21.15.12:0/2096117727 shutdown_connections 2024-03-29T11:31:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.455+0000 7f9ea0400700 1 -- 172.21.15.12:0/2096117727 wait complete. 2024-03-29T11:31:19.469 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T11:31:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.679+0000 7fcb6e200700 1 Processor -- start 2024-03-29T11:31:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.679+0000 7fcb6e200700 1 -- start start 2024-03-29T11:31:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.680+0000 7fcb6e200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815dab0 0x7fcb680b0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.680+0000 7fcb6e200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb68162100 con 0x7fcb680d40f0 2024-03-29T11:31:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.680+0000 7fcb6e200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb680ce960 con 0x7fcb6815d6d0 2024-03-29T11:31:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.680+0000 7fcb6e200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb680b2d20 con 0x7fcb6815dab0 2024-03-29T11:31:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb66a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcb680d40f0 0x7fcb680b0120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38114/0 (socket says 172.21.15.12:38114) 2024-03-29T11:31:19.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb66a00700 1 -- 172.21.15.12:0/2176438859 learned_addr learned my addr 172.21.15.12:0/2176438859 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb67e00700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815dab0 0x7fcb680b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938070722 0 0) 0x7fcb68162100 con 0x7fcb680d40f0 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb4c0035b0 con 0x7fcb680d40f0 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 809429425 0 0) 0x7fcb4c0035b0 con 0x7fcb680d40f0 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815dab0 msgr2=0x7fcb680b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb66000700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815dab0 0x7fcb680b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 >> v1:172.21.15.159:6789/0 conn(0x7fcb6815d6d0 legacy=0x7fcb68161430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb680cec40 con 0x7fcb680d40f0 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.681+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fcb54001670 con 0x7fcb680d40f0 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 >> v1:172.21.15.12:6789/0 conn(0x7fcb680d40f0 legacy=0x7fcb680b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb66000700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb4c003a30 0x7fcb4c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb66000700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815dab0 0x7fcb4c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb66000700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4c004d90 0x7fcb4c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb680b2d20 con 0x7fcb4c004d90 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c00baf0 con 0x7fcb4c003a30 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c00bcb0 con 0x7fcb6815dab0 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb67e00700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4c004d90 0x7fcb4c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb67400700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb4c003a30 0x7fcb4c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb66a00700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815dab0 0x7fcb4c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb67e00700 1 -- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815dab0 msgr2=0x7fcb4c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb67e00700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815dab0 0x7fcb4c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb67e00700 1 -- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb4c003a30 msgr2=0x7fcb4c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb67e00700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb4c003a30 0x7fcb4c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb67e00700 1 -- 172.21.15.12:0/2176438859 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c0035b0 con 0x7fcb4c004d90 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.682+0000 7fcb66a00700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815dab0 0x7fcb4c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.683+0000 7fcb67e00700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4c004d90 0x7fcb4c005210 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fcb6815df00 tx=0x7fcb4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fe5f34fa41fb5c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.683+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6000ebd0 con 0x7fcb4c004d90 2024-03-29T11:31:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.683+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6000f040 con 0x7fcb4c004d90 2024-03-29T11:31:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.683+0000 7fcb66000700 1 -- 172.21.15.12:0/2176438859 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb60014480 con 0x7fcb4c004d90 2024-03-29T11:31:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.683+0000 7fcb6e200700 1 -- 172.21.15.12:0/2176438859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4c004d90 msgr2=0x7fcb4c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.683+0000 7fcb6e200700 1 --2- 172.21.15.12:0/2176438859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4c004d90 0x7fcb4c005210 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fcb6815df00 tx=0x7fcb4c005880 comp rx=0 tx=0).stop 2024-03-29T11:31:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.683+0000 7fcb67e00700 1 -- 172.21.15.12:0/2176438859 reap_dead start 2024-03-29T11:31:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.683+0000 7fcb6e200700 1 -- 172.21.15.12:0/2176438859 shutdown_connections 2024-03-29T11:31:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.683+0000 7fcb6e200700 1 -- 172.21.15.12:0/2176438859 >> 172.21.15.12:0/2176438859 conn(0x7fcb6808aff0 msgr2=0x7fcb680d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.683+0000 7fcb6e200700 1 -- 172.21.15.12:0/2176438859 shutdown_connections 2024-03-29T11:31:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.684+0000 7fcb6e200700 1 -- 172.21.15.12:0/2176438859 wait complete. 2024-03-29T11:31:19.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.685+0000 7fcb6e200700 1 Processor -- start 2024-03-29T11:31:19.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.685+0000 7fcb6e200700 1 -- start start 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.686+0000 7fcb6e200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb4c004d90 0x7fcb68157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.686+0000 7fcb6e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb68159d70 0x7fcb680b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.686+0000 7fcb6e200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815a1f0 0x7fcb680b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.686+0000 7fcb6e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb681623e0 con 0x7fcb68159d70 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.686+0000 7fcb6e200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6806d630 con 0x7fcb4c004d90 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.686+0000 7fcb6e200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb681592d0 con 0x7fcb6815a1f0 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.686+0000 7fcb66a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb68159d70 0x7fcb680b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb66a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb68159d70 0x7fcb680b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55554/0 (socket says 172.21.15.12:55554) 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb66a00700 1 -- 172.21.15.12:0/1629625615 learned_addr learned my addr 172.21.15.12:0/1629625615 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb67e00700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815a1f0 0x7fcb680b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb67400700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb4c004d90 0x7fcb68157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb66a00700 1 -- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815a1f0 msgr2=0x7fcb680b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb66a00700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815a1f0 0x7fcb680b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb66a00700 1 -- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb4c004d90 msgr2=0x7fcb68157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb66a00700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb4c004d90 0x7fcb68157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb66a00700 1 -- 172.21.15.12:0/1629625615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb68160a20 con 0x7fcb68159d70 2024-03-29T11:31:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb66a00700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb68159d70 0x7fcb680b0120 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400eaf0 tx=0x7fcb5400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.687+0000 7fcb5f400700 1 -- 172.21.15.12:0/1629625615 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5400c750 con 0x7fcb68159d70 2024-03-29T11:31:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.688+0000 7fcb5f400700 1 -- 172.21.15.12:0/1629625615 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5400c8b0 con 0x7fcb68159d70 2024-03-29T11:31:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.688+0000 7fcb6e200700 1 -- 172.21.15.12:0/1629625615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb68159570 con 0x7fcb68159d70 2024-03-29T11:31:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.688+0000 7fcb5f400700 1 -- 172.21.15.12:0/1629625615 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5400cc20 con 0x7fcb68159d70 2024-03-29T11:31:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.688+0000 7fcb6e200700 1 -- 172.21.15.12:0/1629625615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb68160d00 con 0x7fcb68159d70 2024-03-29T11:31:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.689+0000 7fcb6e200700 1 -- 172.21.15.12:0/1629625615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb68159a90 con 0x7fcb68159d70 2024-03-29T11:31:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.690+0000 7fcb5f400700 1 -- 172.21.15.12:0/1629625615 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcb5401d460 con 0x7fcb68159d70 2024-03-29T11:31:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.690+0000 7fcb5f400700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcb4806adb0 0x7fcb4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.691+0000 7fcb5f400700 1 -- 172.21.15.12:0/1629625615 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fcb540893a0 con 0x7fcb68159d70 2024-03-29T11:31:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.691+0000 7fcb67400700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcb4806adb0 0x7fcb4806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.692+0000 7fcb67400700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcb4806adb0 0x7fcb4806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcb58000c00 tx=0x7fcb58009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:19.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.695+0000 7fcb5f400700 1 -- 172.21.15.12:0/1629625615 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb68159a90 con 0x7fcb68159d70 2024-03-29T11:31:19.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.913+0000 7fcb6e200700 1 -- 172.21.15.12:0/1629625615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcb68159a90 con 0x7fcb68159d70 2024-03-29T11:31:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.918+0000 7fcb5f400700 1 -- 172.21.15.12:0/1629625615 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcb68159a90 con 0x7fcb68159d70 2024-03-29T11:31:19.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 -- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcb4806adb0 msgr2=0x7fcb4806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcb4806adb0 0x7fcb4806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcb58000c00 tx=0x7fcb58009040 comp rx=0 tx=0).stop 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 -- 172.21.15.12:0/1629625615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb68159d70 msgr2=0x7fcb680b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb68159d70 0x7fcb680b0120 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400eaf0 tx=0x7fcb5400eeb0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 -- 172.21.15.12:0/1629625615 shutdown_connections 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcb4806adb0 0x7fcb4806d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb6815a1f0 0x7fcb680b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb4c004d90 0x7fcb68157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 --2- 172.21.15.12:0/1629625615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb68159d70 0x7fcb680b0120 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 -- 172.21.15.12:0/1629625615 >> 172.21.15.12:0/1629625615 conn(0x7fcb6808aff0 msgr2=0x7fcb6815aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.920+0000 7fcb6e200700 1 -- 172.21.15.12:0/1629625615 shutdown_connections 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:19.921+0000 7fcb6e200700 1 -- 172.21.15.12:0/1629625615 wait complete. 2024-03-29T11:31:19.932 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:31:19.932 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:31:20.016 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:31:20.110 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:31:20.110 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:31:20.110 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:31:20.155 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:31:20.155 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:31:20.155 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:31:20.202 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:31:20.202 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:31:20.202 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:31:20.249 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:31:20.249 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:31:20.249 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:31:20.296 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:31:20.296 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T11:31:20.297 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:31:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.501+0000 7fc765a00700 1 Processor -- start 2024-03-29T11:31:20.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.502+0000 7fc765a00700 1 -- start start 2024-03-29T11:31:20.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.503+0000 7fc765a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7600ceec0 0x7fc7601612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:20.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.503+0000 7fc765a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc760161800 0x7fc7600b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:20.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.503+0000 7fc765a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7600d30d0 con 0x7fc76015d640 2024-03-29T11:31:20.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.503+0000 7fc765a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc760161c90 con 0x7fc760161800 2024-03-29T11:31:20.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.503+0000 7fc765a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7600b6840 con 0x7fc7600ceec0 2024-03-29T11:31:20.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.503+0000 7fc75ea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc76015d640 0x7fc7600d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38126/0 (socket says 172.21.15.12:38126) 2024-03-29T11:31:20.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.503+0000 7fc75ea00700 1 -- 172.21.15.12:0/645914185 learned_addr learned my addr 172.21.15.12:0/645914185 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:20.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.503+0000 7fc75f400700 1 --2- 172.21.15.12:0/645914185 >> v2:172.21.15.159:3300/0 conn(0x7fc760161800 0x7fc7600b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:20.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.503+0000 7fc75e000700 1 --2- 172.21.15.12:0/645914185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7600ceec0 0x7fc7601612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:20.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75d600700 1 -- 172.21.15.12:0/645914185 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154209318 0 0) 0x7fc7600d30d0 con 0x7fc76015d640 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75d600700 1 -- 172.21.15.12:0/645914185 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7440035b0 con 0x7fc76015d640 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75e000700 1 -- 172.21.15.12:0/645914185 >> v2:172.21.15.159:3300/0 conn(0x7fc760161800 msgr2=0x7fc7600b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75e000700 1 --2- 172.21.15.12:0/645914185 >> v2:172.21.15.159:3300/0 conn(0x7fc760161800 0x7fc7600b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75e000700 1 -- 172.21.15.12:0/645914185 >> v1:172.21.15.12:6789/0 conn(0x7fc76015d640 legacy=0x7fc7600d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75e000700 1 -- 172.21.15.12:0/645914185 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7600d2b30 con 0x7fc7600ceec0 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75f400700 1 --2- 172.21.15.12:0/645914185 >> v2:172.21.15.159:3300/0 conn(0x7fc760161800 0x7fc7600b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75e000700 1 --2- 172.21.15.12:0/645914185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7600ceec0 0x7fc7601612c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009840 tx=0x7fc74c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6406f910fb7960c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75d600700 1 -- 172.21.15.12:0/645914185 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c011830 con 0x7fc7600ceec0 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75d600700 1 -- 172.21.15.12:0/645914185 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c011990 con 0x7fc7600ceec0 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.504+0000 7fc75d600700 1 -- 172.21.15.12:0/645914185 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c011d20 con 0x7fc7600ceec0 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.505+0000 7fc765a00700 1 -- 172.21.15.12:0/645914185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7600ceec0 msgr2=0x7fc7601612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.505+0000 7fc765a00700 1 --2- 172.21.15.12:0/645914185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7600ceec0 0x7fc7601612c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009840 tx=0x7fc74c009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.505+0000 7fc765a00700 1 -- 172.21.15.12:0/645914185 shutdown_connections 2024-03-29T11:31:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.505+0000 7fc765a00700 1 --2- 172.21.15.12:0/645914185 >> v2:172.21.15.159:3300/0 conn(0x7fc760161800 0x7fc7600b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.505+0000 7fc765a00700 1 --2- 172.21.15.12:0/645914185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7600ceec0 0x7fc7601612c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.505+0000 7fc765a00700 1 -- 172.21.15.12:0/645914185 >> 172.21.15.12:0/645914185 conn(0x7fc76008aff0 msgr2=0x7fc7600d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.505+0000 7fc765a00700 1 -- 172.21.15.12:0/645914185 shutdown_connections 2024-03-29T11:31:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.505+0000 7fc765a00700 1 -- 172.21.15.12:0/645914185 wait complete. 2024-03-29T11:31:20.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.507+0000 7fc765a00700 1 Processor -- start 2024-03-29T11:31:20.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.507+0000 7fc765a00700 1 -- start start 2024-03-29T11:31:20.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc765a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7600ceec0 0x7fc7600b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:20.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc765a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76015d640 0x7fc760160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:20.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc765a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc760161800 0x7fc760158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:20.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc765a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760160790 con 0x7fc760161800 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc765a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7601560d0 con 0x7fc7600ceec0 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc765a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc760146b70 con 0x7fc76015d640 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc75f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc760161800 0x7fc760158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc75ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7600ceec0 0x7fc7600b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc75f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc760161800 0x7fc760158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55564/0 (socket says 172.21.15.12:55564) 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc75f400700 1 -- 172.21.15.12:0/3340644372 learned_addr learned my addr 172.21.15.12:0/3340644372 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc75e000700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76015d640 0x7fc760160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc75f400700 1 -- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76015d640 msgr2=0x7fc760160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc75f400700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76015d640 0x7fc760160250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc75f400700 1 -- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7600ceec0 msgr2=0x7fc7600b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc75f400700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7600ceec0 0x7fc7600b7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.508+0000 7fc75f400700 1 -- 172.21.15.12:0/3340644372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758009a70 con 0x7fc760161800 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.509+0000 7fc75f400700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc760161800 0x7fc760158a90 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fc758009f00 tx=0x7fc75800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:20.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.509+0000 7fc756a00700 1 -- 172.21.15.12:0/3340644372 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758009d70 con 0x7fc760161800 2024-03-29T11:31:20.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.509+0000 7fc756a00700 1 -- 172.21.15.12:0/3340644372 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75800f040 con 0x7fc760161800 2024-03-29T11:31:20.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.509+0000 7fc765a00700 1 -- 172.21.15.12:0/3340644372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74c0094a0 con 0x7fc760161800 2024-03-29T11:31:20.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.509+0000 7fc756a00700 1 -- 172.21.15.12:0/3340644372 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75800cd70 con 0x7fc760161800 2024-03-29T11:31:20.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.509+0000 7fc765a00700 1 -- 172.21.15.12:0/3340644372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7601552b0 con 0x7fc760161800 2024-03-29T11:31:20.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.510+0000 7fc765a00700 1 -- 172.21.15.12:0/3340644372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc760156240 con 0x7fc760161800 2024-03-29T11:31:20.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.511+0000 7fc756a00700 1 -- 172.21.15.12:0/3340644372 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc758014480 con 0x7fc760161800 2024-03-29T11:31:20.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.512+0000 7fc756a00700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc74006adb0 0x7fc74006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:20.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.512+0000 7fc756a00700 1 -- 172.21.15.12:0/3340644372 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fc758089b40 con 0x7fc760161800 2024-03-29T11:31:20.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.513+0000 7fc75ea00700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc74006adb0 0x7fc74006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.513+0000 7fc75ea00700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc74006adb0 0x7fc74006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc750007fd0 tx=0x7fc750007dc0 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.516+0000 7fc756a00700 1 -- 172.21.15.12:0/3340644372 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc760156240 con 0x7fc760161800 2024-03-29T11:31:20.740 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:31:20.744 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:31:20.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.786+0000 7fc765a00700 1 -- 172.21.15.12:0/3340644372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc760156240 con 0x7fc760161800 2024-03-29T11:31:20.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.786+0000 7fc756a00700 1 -- 172.21.15.12:0/3340644372 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3531 (secure 0 0 0) 0x7fc758018de0 con 0x7fc760161800 2024-03-29T11:31:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:20.780 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":100,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/395980670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":395980670},{"type":"v1","addr":"172.21.15.159:6837","nonce":395980670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":93},{"gid":10183,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/531080306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":531080306},{"type":"v1","addr":"172.21.15.159:6835","nonce":531080306}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10509,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2709565550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2709565550},{"type":"v1","addr":"172.21.15.12:6834","nonce":2709565550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10944,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/397963435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":397963435},{"type":"v1","addr":"172.21.15.12:6835","nonce":397963435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[]} 2024-03-29T11:31:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 -- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc74006adb0 msgr2=0x7fc74006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc74006adb0 0x7fc74006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc750007fd0 tx=0x7fc750007dc0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 -- 172.21.15.12:0/3340644372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc760161800 msgr2=0x7fc760158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc760161800 0x7fc760158a90 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fc758009f00 tx=0x7fc75800c3a0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 -- 172.21.15.12:0/3340644372 shutdown_connections 2024-03-29T11:31:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc74006adb0 0x7fc74006d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc760161800 0x7fc760158a90 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7600ceec0 0x7fc7600b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 --2- 172.21.15.12:0/3340644372 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76015d640 0x7fc760160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 -- 172.21.15.12:0/3340644372 >> 172.21.15.12:0/3340644372 conn(0x7fc76008aff0 msgr2=0x7fc7600d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 -- 172.21.15.12:0/3340644372 shutdown_connections 2024-03-29T11:31:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:20.789+0000 7fc765a00700 1 -- 172.21.15.12:0/3340644372 wait complete. 2024-03-29T11:31:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 100 2024-03-29T11:31:20.801 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T11:31:20.802 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T11:31:20.802 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T11:31:20.802 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:20.802 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T11:31:20.802 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T11:31:20.802 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T11:31:20.802 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:20.803 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T11:31:20.803 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T11:31:20.803 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T11:31:20.803 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:20.803 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T11:31:20.803 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T11:31:20.804 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T11:31:20.804 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:20.818 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T11:31:20.818 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T11:31:20.819 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T11:31:20.819 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T11:31:20.856 INFO:tasks.ceph.mds.d:Started 2024-03-29T11:31:20.856 DEBUG:teuthology.parallel:result is None 2024-03-29T11:31:20.859 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T11:31:20.859 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T11:31:20.860 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T11:31:20.860 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T11:31:20.861 INFO:tasks.ceph.mds.b:Started 2024-03-29T11:31:20.861 DEBUG:teuthology.parallel:result is None 2024-03-29T11:31:20.904 INFO:tasks.ceph.mds.a:Started 2024-03-29T11:31:20.905 DEBUG:teuthology.parallel:result is None 2024-03-29T11:31:20.906 INFO:tasks.ceph.mds.c:Started 2024-03-29T11:31:20.907 DEBUG:teuthology.parallel:result is None 2024-03-29T11:31:20.907 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T11:31:21.000 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T11:31:21.001 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T11:31:21.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-29T11:31:21.058 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:31:21.060 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T11:31:21.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.131+0000 7fcc79800700 1 Processor -- start 2024-03-29T11:31:21.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.131+0000 7fcc79800700 1 -- start start 2024-03-29T11:31:21.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.132+0000 7fcc79800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406ba20 0x7fcc740d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:21.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.132+0000 7fcc79800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcc740b2180 0x7fcc7406be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:21.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.132+0000 7fcc79800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7406c780 con 0x7fcc7406cc10 2024-03-29T11:31:21.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.132+0000 7fcc79800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7406d6b0 con 0x7fcc740b2180 2024-03-29T11:31:21.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.132+0000 7fcc79800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7406d0f0 con 0x7fcc7406ba20 2024-03-29T11:31:21.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc73e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406ba20 0x7fcc740d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:21.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc78e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcc740b2180 0x7fcc7406be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:21.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc73400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fcc7406cc10 0x7fcc74160920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:53750/0 (socket says 172.21.15.12:53750) 2024-03-29T11:31:21.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc73400700 1 -- 172.21.15.12:0/18528363 learned_addr learned my addr 172.21.15.12:0/18528363 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:21.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc73e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406ba20 0x7fcc740d3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:42618/0 (socket says 172.21.15.12:42618) 2024-03-29T11:31:21.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc78e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcc740b2180 0x7fcc7406be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:37616/0 (socket says 172.21.15.12:37616) 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc73e00700 1 -- 172.21.15.12:0/18528363 >> v2:172.21.15.159:3300/0 conn(0x7fcc740b2180 msgr2=0x7fcc7406be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc73e00700 1 --2- 172.21.15.12:0/18528363 >> v2:172.21.15.159:3300/0 conn(0x7fcc740b2180 0x7fcc7406be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc73e00700 1 -- 172.21.15.12:0/18528363 >> v1:172.21.15.159:6789/0 conn(0x7fcc7406cc10 legacy=0x7fcc74160920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc73e00700 1 -- 172.21.15.12:0/18528363 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7406d3d0 con 0x7fcc7406ba20 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc78e00700 1 --2- 172.21.15.12:0/18528363 >> v2:172.21.15.159:3300/0 conn(0x7fcc740b2180 0x7fcc7406be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.133+0000 7fcc73e00700 1 --2- 172.21.15.12:0/18528363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406ba20 0x7fcc740d3f60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009840 tx=0x7fcc64009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2baa0e84a4c934b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc72a00700 1 -- 172.21.15.12:0/18528363 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64011830 con 0x7fcc7406ba20 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc72a00700 1 -- 172.21.15.12:0/18528363 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc64011990 con 0x7fcc7406ba20 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc72a00700 1 -- 172.21.15.12:0/18528363 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64011d20 con 0x7fcc7406ba20 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc79800700 1 -- 172.21.15.12:0/18528363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406ba20 msgr2=0x7fcc740d3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc79800700 1 --2- 172.21.15.12:0/18528363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406ba20 0x7fcc740d3f60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009840 tx=0x7fcc64009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc79800700 1 -- 172.21.15.12:0/18528363 shutdown_connections 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc79800700 1 --2- 172.21.15.12:0/18528363 >> v2:172.21.15.159:3300/0 conn(0x7fcc740b2180 0x7fcc7406be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:21.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc79800700 1 --2- 172.21.15.12:0/18528363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406ba20 0x7fcc740d3f60 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:21.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc79800700 1 -- 172.21.15.12:0/18528363 >> 172.21.15.12:0/18528363 conn(0x7fcc7409c6c0 msgr2=0x7fcc740d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:21.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc79800700 1 -- 172.21.15.12:0/18528363 shutdown_connections 2024-03-29T11:31:21.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.134+0000 7fcc79800700 1 -- 172.21.15.12:0/18528363 wait complete. 2024-03-29T11:31:21.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.136+0000 7fcc79800700 1 Processor -- start 2024-03-29T11:31:21.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.136+0000 7fcc79800700 1 -- start start 2024-03-29T11:31:21.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.137+0000 7fcc79800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7406ba20 0x7fcc740c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:21.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.137+0000 7fcc79800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406cc10 0x7fcc7414d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:21.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.137+0000 7fcc79800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc740b2180 0x7fcc7414d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:21.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.137+0000 7fcc79800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740c9680 con 0x7fcc7406ba20 2024-03-29T11:31:21.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.137+0000 7fcc79800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740c9960 con 0x7fcc740b2180 2024-03-29T11:31:21.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.137+0000 7fcc79800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740c9c40 con 0x7fcc7406cc10 2024-03-29T11:31:21.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.137+0000 7fcc73e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7406ba20 0x7fcc740c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:21.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.137+0000 7fcc73e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7406ba20 0x7fcc740c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55596/0 (socket says 172.21.15.12:55596) 2024-03-29T11:31:21.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.137+0000 7fcc73e00700 1 -- 172.21.15.12:0/248865485 learned_addr learned my addr 172.21.15.12:0/248865485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:21.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc73400700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406cc10 0x7fcc7414d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:21.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc78e00700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc740b2180 0x7fcc7414d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:21.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc73e00700 1 -- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406cc10 msgr2=0x7fcc7414d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:21.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc73e00700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406cc10 0x7fcc7414d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:21.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc73e00700 1 -- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc740b2180 msgr2=0x7fcc7414d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:21.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc73e00700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc740b2180 0x7fcc7414d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:21.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc73e00700 1 -- 172.21.15.12:0/248865485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c009a70 con 0x7fcc7406ba20 2024-03-29T11:31:21.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc73e00700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7406ba20 0x7fcc740c69f0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009fd0 tx=0x7fcc64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:21.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc6be00700 1 -- 172.21.15.12:0/248865485 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc640038a0 con 0x7fcc7406ba20 2024-03-29T11:31:21.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc6be00700 1 -- 172.21.15.12:0/248865485 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6401e070 con 0x7fcc7406ba20 2024-03-29T11:31:21.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc6be00700 1 -- 172.21.15.12:0/248865485 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64003c40 con 0x7fcc7406ba20 2024-03-29T11:31:21.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc79800700 1 -- 172.21.15.12:0/248865485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc640094a0 con 0x7fcc7406ba20 2024-03-29T11:31:21.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.138+0000 7fcc79800700 1 -- 172.21.15.12:0/248865485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc740c6f30 con 0x7fcc7406ba20 2024-03-29T11:31:21.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.139+0000 7fcc79800700 1 -- 172.21.15.12:0/248865485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7404ea80 con 0x7fcc7406ba20 2024-03-29T11:31:21.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.140+0000 7fcc6be00700 1 -- 172.21.15.12:0/248865485 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcc6402d450 con 0x7fcc7406ba20 2024-03-29T11:31:21.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.141+0000 7fcc6be00700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc5406adb0 0x7fcc5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:21.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.141+0000 7fcc6be00700 1 -- 172.21.15.12:0/248865485 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fcc6408a410 con 0x7fcc7406ba20 2024-03-29T11:31:21.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.142+0000 7fcc73400700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc5406adb0 0x7fcc5406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:21.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.142+0000 7fcc73400700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc5406adb0 0x7fcc5406d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcc741f60d0 tx=0x7fcc60009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:21.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.145+0000 7fcc6be00700 1 -- 172.21.15.12:0/248865485 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcc640555c0 con 0x7fcc7406ba20 2024-03-29T11:31:21.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:21.382+0000 7fcc79800700 1 -- 172.21.15.12:0/248865485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcc740c9d80 con 0x7fcc7406ba20 2024-03-29T11:31:22.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.094+0000 7fcc6be00700 1 -- 172.21.15.12:0/248865485 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v223) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcc740c9d80 con 0x7fcc7406ba20 2024-03-29T11:31:22.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 -- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc5406adb0 msgr2=0x7fcc5406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc5406adb0 0x7fcc5406d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcc741f60d0 tx=0x7fcc60009040 comp rx=0 tx=0).stop 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 -- 172.21.15.12:0/248865485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7406ba20 msgr2=0x7fcc740c69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7406ba20 0x7fcc740c69f0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009fd0 tx=0x7fcc64004750 comp rx=0 tx=0).stop 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 -- 172.21.15.12:0/248865485 shutdown_connections 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc5406adb0 0x7fcc5406d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc740b2180 0x7fcc7414d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7406ba20 0x7fcc740c69f0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 --2- 172.21.15.12:0/248865485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7406cc10 0x7fcc7414d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 -- 172.21.15.12:0/248865485 >> 172.21.15.12:0/248865485 conn(0x7fcc7409c6c0 msgr2=0x7fcc740b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.097+0000 7fcc79800700 1 -- 172.21.15.12:0/248865485 shutdown_connections 2024-03-29T11:31:22.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.098+0000 7fcc79800700 1 -- 172.21.15.12:0/248865485 wait complete. 2024-03-29T11:31:22.093 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T11:31:22.113 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T11:31:22.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.321+0000 7f8273e00700 1 Processor -- start 2024-03-29T11:31:22.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.321+0000 7f8273e00700 1 -- start start 2024-03-29T11:31:22.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.322+0000 7f8273e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f826c072580 0x7f826c0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.322+0000 7f8273e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f826c0b2e50 0x7f826c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.322+0000 7f8273e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0b2180 con 0x7f826c0b3af0 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.322+0000 7f8273e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c15b100 con 0x7f826c072580 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.322+0000 7f8273e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c15b3e0 con 0x7f826c0b2e50 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.323+0000 7f8270a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f826c072580 0x7f826c0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.323+0000 7f8271400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f826c0b3af0 0x7f826c15a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38148/0 (socket says 172.21.15.12:38148) 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.323+0000 7f8270a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f826c072580 0x7f826c0b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55604/0 (socket says 172.21.15.12:55604) 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.323+0000 7f8271400700 1 -- 172.21.15.12:0/74080149 learned_addr learned my addr 172.21.15.12:0/74080149 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.323+0000 7f8271e00700 1 --2- 172.21.15.12:0/74080149 >> v2:172.21.15.159:3300/0 conn(0x7f826c0b2e50 0x7f826c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.323+0000 7f8270a00700 1 -- 172.21.15.12:0/74080149 >> v2:172.21.15.159:3300/0 conn(0x7f826c0b2e50 msgr2=0x7f826c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.323+0000 7f8270a00700 1 --2- 172.21.15.12:0/74080149 >> v2:172.21.15.159:3300/0 conn(0x7f826c0b2e50 0x7f826c0709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.323+0000 7f8270a00700 1 -- 172.21.15.12:0/74080149 >> v1:172.21.15.12:6789/0 conn(0x7f826c0b3af0 legacy=0x7f826c15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.323+0000 7f8270a00700 1 -- 172.21.15.12:0/74080149 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0b23f0 con 0x7f826c072580 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.323+0000 7f826be00700 1 -- 172.21.15.12:0/74080149 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420103835 0 0) 0x7f826c0b2180 con 0x7f826c0b3af0 2024-03-29T11:31:22.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f8270a00700 1 --2- 172.21.15.12:0/74080149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 0x7f826c0b2910 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f826c158390 tx=0x7f8258009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4283fc46c4404ab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f826be00700 1 -- 172.21.15.12:0/74080149 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8258011830 con 0x7f826c072580 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f826be00700 1 -- 172.21.15.12:0/74080149 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8258011990 con 0x7f826c072580 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f826be00700 1 -- 172.21.15.12:0/74080149 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8258011d20 con 0x7f826c072580 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f8273e00700 1 -- 172.21.15.12:0/74080149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 msgr2=0x7f826c0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f8273e00700 1 --2- 172.21.15.12:0/74080149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 0x7f826c0b2910 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f826c158390 tx=0x7f8258009b20 comp rx=0 tx=0).stop 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f8273e00700 1 -- 172.21.15.12:0/74080149 shutdown_connections 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f8273e00700 1 --2- 172.21.15.12:0/74080149 >> v2:172.21.15.159:3300/0 conn(0x7f826c0b2e50 0x7f826c0709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f8273e00700 1 --2- 172.21.15.12:0/74080149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 0x7f826c0b2910 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f8273e00700 1 -- 172.21.15.12:0/74080149 >> 172.21.15.12:0/74080149 conn(0x7f826c0cf190 msgr2=0x7f826c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f8273e00700 1 -- 172.21.15.12:0/74080149 shutdown_connections 2024-03-29T11:31:22.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.324+0000 7f8273e00700 1 -- 172.21.15.12:0/74080149 wait complete. 2024-03-29T11:31:22.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.327+0000 7f8273e00700 1 Processor -- start 2024-03-29T11:31:22.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.327+0000 7f8273e00700 1 -- start start 2024-03-29T11:31:22.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.328+0000 7f8273e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 0x7f826c1598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.328+0000 7f8273e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f826c0b2e50 0x7f826c0ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.328+0000 7f8273e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f826c0b3af0 0x7f826c0ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.328+0000 7f8273e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c153090 con 0x7f826c072580 2024-03-29T11:31:22.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.328+0000 7f8273e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0caed0 con 0x7f826c0b3af0 2024-03-29T11:31:22.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.328+0000 7f8273e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c159df0 con 0x7f826c0b2e50 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.328+0000 7f8271400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 0x7f826c1598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.329+0000 7f8271400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 0x7f826c1598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55618/0 (socket says 172.21.15.12:55618) 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.329+0000 7f8271400700 1 -- 172.21.15.12:0/1870609757 learned_addr learned my addr 172.21.15.12:0/1870609757 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.329+0000 7f8271e00700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f826c0b3af0 0x7f826c0ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.329+0000 7f8270a00700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f826c0b2e50 0x7f826c0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.329+0000 7f8271400700 1 -- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f826c0b2e50 msgr2=0x7f826c0ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.329+0000 7f8271400700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f826c0b2e50 0x7f826c0ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.329+0000 7f8271400700 1 -- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f826c0b3af0 msgr2=0x7f826c0ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.329+0000 7f8271400700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f826c0b3af0 0x7f826c0ca990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.329+0000 7f8271400700 1 -- 172.21.15.12:0/1870609757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82580094a0 con 0x7f826c072580 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.329+0000 7f8271400700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 0x7f826c1598b0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f826000c580 tx=0x7f826000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.330+0000 7f8269600700 1 -- 172.21.15.12:0/1870609757 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260004c80 con 0x7f826c072580 2024-03-29T11:31:22.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.330+0000 7f8269600700 1 -- 172.21.15.12:0/1870609757 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8260004e00 con 0x7f826c072580 2024-03-29T11:31:22.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.330+0000 7f8273e00700 1 -- 172.21.15.12:0/1870609757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c152480 con 0x7f826c072580 2024-03-29T11:31:22.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.330+0000 7f8269600700 1 -- 172.21.15.12:0/1870609757 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260007310 con 0x7f826c072580 2024-03-29T11:31:22.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.330+0000 7f8273e00700 1 -- 172.21.15.12:0/1870609757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c152a40 con 0x7f826c072580 2024-03-29T11:31:22.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.331+0000 7f8273e00700 1 -- 172.21.15.12:0/1870609757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c15ecf0 con 0x7f826c072580 2024-03-29T11:31:22.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.332+0000 7f8269600700 1 -- 172.21.15.12:0/1870609757 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8260007470 con 0x7f826c072580 2024-03-29T11:31:22.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.333+0000 7f8269600700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f824c06adb0 0x7f824c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.333+0000 7f8269600700 1 -- 172.21.15.12:0/1870609757 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f8260088da0 con 0x7f826c072580 2024-03-29T11:31:22.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.333+0000 7f8270a00700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f824c06adb0 0x7f824c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.334+0000 7f8270a00700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f824c06adb0 0x7f824c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f826c158390 tx=0x7f8258008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:22.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.337+0000 7f8269600700 1 -- 172.21.15.12:0/1870609757 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8260054160 con 0x7f826c072580 2024-03-29T11:31:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.637+0000 7f8273e00700 1 -- 172.21.15.12:0/1870609757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f826c067720 con 0x7f826c072580 2024-03-29T11:31:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.637+0000 7f8269600700 1 -- 172.21.15.12:0/1870609757 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v49) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8260057780 con 0x7f826c072580 2024-03-29T11:31:22.631 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:22.631 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:31:22.631 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T11:31:22.631 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.632 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:31:22.633 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.634 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:31:22.635 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.636 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.637 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T11:31:22.638 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T11:31:22.642 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:31:22.643 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T11:31:22.644 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:31:22.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 -- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f824c06adb0 msgr2=0x7f824c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f824c06adb0 0x7f824c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f826c158390 tx=0x7f8258008030 comp rx=0 tx=0).stop 2024-03-29T11:31:22.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 -- 172.21.15.12:0/1870609757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 msgr2=0x7f826c1598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 0x7f826c1598b0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f826000c580 tx=0x7f826000c890 comp rx=0 tx=0).stop 2024-03-29T11:31:22.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 -- 172.21.15.12:0/1870609757 shutdown_connections 2024-03-29T11:31:22.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f824c06adb0 0x7f824c06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f826c0b3af0 0x7f826c0ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f826c072580 0x7f826c1598b0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 --2- 172.21.15.12:0/1870609757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f826c0b2e50 0x7f826c0ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 -- 172.21.15.12:0/1870609757 >> 172.21.15.12:0/1870609757 conn(0x7f826c0cf190 msgr2=0x7f826c0aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:22.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 -- 172.21.15.12:0/1870609757 shutdown_connections 2024-03-29T11:31:22.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.640+0000 7f8273e00700 1 -- 172.21.15.12:0/1870609757 wait complete. 2024-03-29T11:31:22.652 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:22.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.856+0000 7ff651c00700 1 Processor -- start 2024-03-29T11:31:22.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.856+0000 7ff651c00700 1 -- start start 2024-03-29T11:31:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.857+0000 7ff651c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff64c15d640 0x7ff64c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.857+0000 7ff651c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff64c0ceec0 0x7ff64c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.857+0000 7ff651c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c0d30d0 con 0x7ff64c161800 2024-03-29T11:31:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.857+0000 7ff651c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64c0b6840 con 0x7ff64c0ceec0 2024-03-29T11:31:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.858+0000 7ff651c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64c161c20 con 0x7ff64c15d640 2024-03-29T11:31:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.858+0000 7ff643e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff64c0ceec0 0x7ff64c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.858+0000 7ff643e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff64c0ceec0 0x7ff64c1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55632/0 (socket says 172.21.15.12:55632) 2024-03-29T11:31:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.858+0000 7ff643e00700 1 -- 172.21.15.12:0/810523612 learned_addr learned my addr 172.21.15.12:0/810523612 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.858+0000 7ff64aa00700 1 --2- 172.21.15.12:0/810523612 >> v2:172.21.15.159:3300/0 conn(0x7ff64c15d640 0x7ff64c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.858+0000 7ff643e00700 1 -- 172.21.15.12:0/810523612 >> v2:172.21.15.159:3300/0 conn(0x7ff64c15d640 msgr2=0x7ff64c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.858+0000 7ff643e00700 1 --2- 172.21.15.12:0/810523612 >> v2:172.21.15.159:3300/0 conn(0x7ff64c15d640 0x7ff64c0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.858+0000 7ff643e00700 1 -- 172.21.15.12:0/810523612 >> v1:172.21.15.159:6789/0 conn(0x7ff64c161800 legacy=0x7ff64c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.858+0000 7ff643e00700 1 -- 172.21.15.12:0/810523612 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c0d2880 con 0x7ff64c0ceec0 2024-03-29T11:31:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.859+0000 7ff643e00700 1 --2- 172.21.15.12:0/810523612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 0x7ff64c1612c0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0b5c20 tx=0x7ff634009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b356f66b578e41 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.859+0000 7ff64a000700 1 -- 172.21.15.12:0/810523612 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff634004230 con 0x7ff64c0ceec0 2024-03-29T11:31:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.859+0000 7ff64a000700 1 -- 172.21.15.12:0/810523612 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff634004390 con 0x7ff64c0ceec0 2024-03-29T11:31:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.859+0000 7ff64a000700 1 -- 172.21.15.12:0/810523612 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6340036a0 con 0x7ff64c0ceec0 2024-03-29T11:31:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.859+0000 7ff651c00700 1 -- 172.21.15.12:0/810523612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 msgr2=0x7ff64c1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.859+0000 7ff651c00700 1 --2- 172.21.15.12:0/810523612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 0x7ff64c1612c0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0b5c20 tx=0x7ff634009bd0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.859+0000 7ff651c00700 1 -- 172.21.15.12:0/810523612 shutdown_connections 2024-03-29T11:31:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.859+0000 7ff651c00700 1 --2- 172.21.15.12:0/810523612 >> v2:172.21.15.159:3300/0 conn(0x7ff64c15d640 0x7ff64c0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.859+0000 7ff651c00700 1 --2- 172.21.15.12:0/810523612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 0x7ff64c1612c0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.859+0000 7ff651c00700 1 -- 172.21.15.12:0/810523612 >> 172.21.15.12:0/810523612 conn(0x7ff64c08aff0 msgr2=0x7ff64c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.860+0000 7ff651c00700 1 -- 172.21.15.12:0/810523612 shutdown_connections 2024-03-29T11:31:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.860+0000 7ff651c00700 1 -- 172.21.15.12:0/810523612 wait complete. 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.861+0000 7ff651c00700 1 Processor -- start 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.861+0000 7ff651c00700 1 -- start start 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.862+0000 7ff651c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 0x7ff64c158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.862+0000 7ff651c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff64c15d640 0x7ff64c15f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.862+0000 7ff651c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff64c161800 0x7ff64c15f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.862+0000 7ff651c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c09fff0 con 0x7ff64c0ceec0 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.862+0000 7ff651c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c0a0b50 con 0x7ff64c161800 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.862+0000 7ff651c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c0a0220 con 0x7ff64c15d640 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 0x7ff64c158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 0x7ff64c158b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55638/0 (socket says 172.21.15.12:55638) 2024-03-29T11:31:22.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64aa00700 1 -- 172.21.15.12:0/363081314 learned_addr learned my addr 172.21.15.12:0/363081314 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64b400700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff64c161800 0x7ff64c15f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff643e00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff64c15d640 0x7ff64c15f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64aa00700 1 -- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff64c15d640 msgr2=0x7ff64c15f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64aa00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff64c15d640 0x7ff64c15f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64aa00700 1 -- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff64c161800 msgr2=0x7ff64c15f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64aa00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff64c161800 0x7ff64c15f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64aa00700 1 -- 172.21.15.12:0/363081314 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff634009550 con 0x7ff64c0ceec0 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64b400700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff64c161800 0x7ff64c15f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff64aa00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 0x7ff64c158b20 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00e810 tx=0x7ff63c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff642a00700 1 -- 172.21.15.12:0/363081314 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c013610 con 0x7ff64c0ceec0 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff642a00700 1 -- 172.21.15.12:0/363081314 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c013770 con 0x7ff64c0ceec0 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff642a00700 1 -- 172.21.15.12:0/363081314 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c013b20 con 0x7ff64c0ceec0 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.863+0000 7ff651c00700 1 -- 172.21.15.12:0/363081314 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c0a0500 con 0x7ff64c0ceec0 2024-03-29T11:31:22.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.864+0000 7ff651c00700 1 -- 172.21.15.12:0/363081314 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c09eac0 con 0x7ff64c0ceec0 2024-03-29T11:31:22.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.865+0000 7ff651c00700 1 -- 172.21.15.12:0/363081314 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c0a0e30 con 0x7ff64c0ceec0 2024-03-29T11:31:22.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.866+0000 7ff642a00700 1 -- 172.21.15.12:0/363081314 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff63c013c80 con 0x7ff64c0ceec0 2024-03-29T11:31:22.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.866+0000 7ff642a00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff62806adb0 0x7ff62806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:22.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.867+0000 7ff642a00700 1 -- 172.21.15.12:0/363081314 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 4929+0+0 (secure 0 0 0) 0x7ff63c089060 con 0x7ff64c0ceec0 2024-03-29T11:31:22.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.867+0000 7ff643e00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff62806adb0 0x7ff62806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:22.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.868+0000 7ff643e00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff62806adb0 0x7ff62806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff634009130 tx=0x7ff634008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:22.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:22.870+0000 7ff642a00700 1 -- 172.21.15.12:0/363081314 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff63c054290 con 0x7ff64c0ceec0 2024-03-29T11:31:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.084+0000 7ff651c00700 1 -- 172.21.15.12:0/363081314 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff64c067720 con 0x7ff64c0ceec0 2024-03-29T11:31:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.085+0000 7ff642a00700 1 -- 172.21.15.12:0/363081314 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+12515 (secure 0 0 0) 0x7ff63c0578b0 con 0x7ff64c0ceec0 2024-03-29T11:31:23.079 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:23.079 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":223,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:22.088017+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 -- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff62806adb0 msgr2=0x7ff62806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff62806adb0 0x7ff62806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff634009130 tx=0x7ff634008030 comp rx=0 tx=0).stop 2024-03-29T11:31:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 -- 172.21.15.12:0/363081314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 msgr2=0x7ff64c158b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 0x7ff64c158b20 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00e810 tx=0x7ff63c00ebd0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 -- 172.21.15.12:0/363081314 shutdown_connections 2024-03-29T11:31:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff62806adb0 0x7ff62806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff64c161800 0x7ff64c15f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64c0ceec0 0x7ff64c158b20 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 --2- 172.21.15.12:0/363081314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff64c15d640 0x7ff64c15f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 -- 172.21.15.12:0/363081314 >> 172.21.15.12:0/363081314 conn(0x7ff64c08aff0 msgr2=0x7ff64c0d05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 -- 172.21.15.12:0/363081314 shutdown_connections 2024-03-29T11:31:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.087+0000 7ff651c00700 1 -- 172.21.15.12:0/363081314 wait complete. 2024-03-29T11:31:23.099 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:31:23.100 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:31:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.306+0000 7f1b01e00700 1 Processor -- start 2024-03-29T11:31:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.306+0000 7f1b01e00700 1 -- start start 2024-03-29T11:31:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1b01e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1afc160e50 0x7f1afc0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1b01e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1afc0d3f20 0x7f1afc0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1b01e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1afc0b3f80 0x7f1afc0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1b01e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1afc161230 con 0x7f1afc0d3f20 2024-03-29T11:31:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1b01e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1afc0d2a80 con 0x7f1afc160e50 2024-03-29T11:31:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1b01e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0d43b0 con 0x7f1afc0b3f80 2024-03-29T11:31:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1afa000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1afc0d3f20 0x7f1afc0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1afa000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1afc0d3f20 0x7f1afc0b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55642/0 (socket says 172.21.15.12:55642) 2024-03-29T11:31:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1afa000700 1 -- 172.21.15.12:0/355675830 learned_addr learned my addr 172.21.15.12:0/355675830 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1afb400700 1 --2- 172.21.15.12:0/355675830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1afc0b3f80 0x7f1afc0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.307+0000 7f1afaa00700 1 --2- 172.21.15.12:0/355675830 >> v2:172.21.15.159:3300/0 conn(0x7f1afc160e50 0x7f1afc0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.308+0000 7f1afa000700 1 -- 172.21.15.12:0/355675830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1afc0b3f80 msgr2=0x7f1afc0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.308+0000 7f1afa000700 1 --2- 172.21.15.12:0/355675830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1afc0b3f80 0x7f1afc0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.308+0000 7f1afa000700 1 -- 172.21.15.12:0/355675830 >> v2:172.21.15.159:3300/0 conn(0x7f1afc160e50 msgr2=0x7f1afc0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.308+0000 7f1afa000700 1 --2- 172.21.15.12:0/355675830 >> v2:172.21.15.159:3300/0 conn(0x7f1afc160e50 0x7f1afc0b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.308+0000 7f1afa000700 1 -- 172.21.15.12:0/355675830 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc0d21e0 con 0x7f1afc0d3f20 2024-03-29T11:31:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.308+0000 7f1afa000700 1 --2- 172.21.15.12:0/355675830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc0d3f20 0x7f1afc0b53d0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f1afc0cff10 tx=0x7f1ae8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80cd52e04cf9766 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:23.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.308+0000 7f1af9600700 1 -- 172.21.15.12:0/355675830 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8011830 con 0x7f1afc0d3f20 2024-03-29T11:31:23.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.308+0000 7f1af9600700 1 -- 172.21.15.12:0/355675830 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae8011990 con 0x7f1afc0d3f20 2024-03-29T11:31:23.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.309+0000 7f1af9600700 1 -- 172.21.15.12:0/355675830 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8011d20 con 0x7f1afc0d3f20 2024-03-29T11:31:23.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.309+0000 7f1b01e00700 1 -- 172.21.15.12:0/355675830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc0d3f20 msgr2=0x7f1afc0b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.309+0000 7f1b01e00700 1 --2- 172.21.15.12:0/355675830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc0d3f20 0x7f1afc0b53d0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f1afc0cff10 tx=0x7f1ae8009bd0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.309+0000 7f1b01e00700 1 -- 172.21.15.12:0/355675830 shutdown_connections 2024-03-29T11:31:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.309+0000 7f1b01e00700 1 --2- 172.21.15.12:0/355675830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1afc0b3f80 0x7f1afc0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.309+0000 7f1b01e00700 1 --2- 172.21.15.12:0/355675830 >> v2:172.21.15.159:3300/0 conn(0x7f1afc160e50 0x7f1afc0b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.309+0000 7f1b01e00700 1 --2- 172.21.15.12:0/355675830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc0d3f20 0x7f1afc0b53d0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.309+0000 7f1b01e00700 1 -- 172.21.15.12:0/355675830 >> 172.21.15.12:0/355675830 conn(0x7f1afc08aff0 msgr2=0x7f1afc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.309+0000 7f1b01e00700 1 -- 172.21.15.12:0/355675830 shutdown_connections 2024-03-29T11:31:23.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.309+0000 7f1b01e00700 1 -- 172.21.15.12:0/355675830 wait complete. 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.311+0000 7f1b01e00700 1 Processor -- start 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.311+0000 7f1b01e00700 1 -- start start 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.312+0000 7f1b01e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1afc0b3f80 0x7f1afc0b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.312+0000 7f1b01e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc160e50 0x7f1afc15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.312+0000 7f1b01e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1afc06c3d0 0x7f1afc06c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.312+0000 7f1b01e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc06cda0 con 0x7f1afc160e50 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.312+0000 7f1b01e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc15bf90 con 0x7f1afc06c3d0 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.312+0000 7f1b01e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0cc430 con 0x7f1afc0b3f80 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.312+0000 7f1afa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc160e50 0x7f1afc15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.312+0000 7f1afa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc160e50 0x7f1afc15ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55656/0 (socket says 172.21.15.12:55656) 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.312+0000 7f1afa000700 1 -- 172.21.15.12:0/1792931126 learned_addr learned my addr 172.21.15.12:0/1792931126 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1afaa00700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1afc0b3f80 0x7f1afc0b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1afb400700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1afc06c3d0 0x7f1afc06c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1afa000700 1 -- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1afc0b3f80 msgr2=0x7f1afc0b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1afa000700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1afc0b3f80 0x7f1afc0b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1afa000700 1 -- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1afc06c3d0 msgr2=0x7f1afc06c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1afa000700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1afc06c3d0 0x7f1afc06c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1afa000700 1 -- 172.21.15.12:0/1792931126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae8009550 con 0x7f1afc160e50 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1afa000700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc160e50 0x7f1afc15ba50 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8000c00 tx=0x7f1ae8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1af3400700 1 -- 172.21.15.12:0/1792931126 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8004300 con 0x7f1afc160e50 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1af3400700 1 -- 172.21.15.12:0/1792931126 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae801f070 con 0x7f1afc160e50 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1b01e00700 1 -- 172.21.15.12:0/1792931126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1afc0b6150 con 0x7f1afc160e50 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.313+0000 7f1af3400700 1 -- 172.21.15.12:0/1792931126 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae80036a0 con 0x7f1afc160e50 2024-03-29T11:31:23.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.314+0000 7f1b01e00700 1 -- 172.21.15.12:0/1792931126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1afc0b6710 con 0x7f1afc160e50 2024-03-29T11:31:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.315+0000 7f1b01e00700 1 -- 172.21.15.12:0/1792931126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1afc0d11c0 con 0x7f1afc160e50 2024-03-29T11:31:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.315+0000 7f1af3400700 1 -- 172.21.15.12:0/1792931126 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1ae8021030 con 0x7f1afc160e50 2024-03-29T11:31:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.317+0000 7f1af3400700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1adc06adb0 0x7f1adc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.317+0000 7f1af3400700 1 -- 172.21.15.12:0/1792931126 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f1ae808a470 con 0x7f1afc160e50 2024-03-29T11:31:23.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.317+0000 7f1afaa00700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1adc06adb0 0x7f1adc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.318+0000 7f1afaa00700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1adc06adb0 0x7f1adc06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009ad0 tx=0x7f1aec008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.321+0000 7f1af3400700 1 -- 172.21.15.12:0/1792931126 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ae8055950 con 0x7f1afc160e50 2024-03-29T11:31:23.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.534+0000 7f1b01e00700 1 -- 172.21.15.12:0/1792931126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1afc067720 con 0x7f1afc160e50 2024-03-29T11:31:23.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.534+0000 7f1af3400700 1 -- 172.21.15.12:0/1792931126 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v223) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ae80150a0 con 0x7f1afc160e50 2024-03-29T11:31:23.528 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:31:23.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 -- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1adc06adb0 msgr2=0x7f1adc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1adc06adb0 0x7f1adc06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009ad0 tx=0x7f1aec008040 comp rx=0 tx=0).stop 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 -- 172.21.15.12:0/1792931126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc160e50 msgr2=0x7f1afc15ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc160e50 0x7f1afc15ba50 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8000c00 tx=0x7f1ae8004750 comp rx=0 tx=0).stop 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 -- 172.21.15.12:0/1792931126 shutdown_connections 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1adc06adb0 0x7f1adc06d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1afc06c3d0 0x7f1afc06c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1afc0b3f80 0x7f1afc0b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 --2- 172.21.15.12:0/1792931126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1afc160e50 0x7f1afc15ba50 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 -- 172.21.15.12:0/1792931126 >> 172.21.15.12:0/1792931126 conn(0x7f1afc08aff0 msgr2=0x7f1afc0cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 -- 172.21.15.12:0/1792931126 shutdown_connections 2024-03-29T11:31:23.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.536+0000 7f1b01e00700 1 -- 172.21.15.12:0/1792931126 wait complete. 2024-03-29T11:31:23.549 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T11:31:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.752+0000 7f0c35400700 1 Processor -- start 2024-03-29T11:31:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.753+0000 7f0c35400700 1 -- start start 2024-03-29T11:31:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.754+0000 7f0c35400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0c301619d0 0x7f0c301685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.754+0000 7f0c35400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c300d30d0 con 0x7f0c3015d640 2024-03-29T11:31:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.754+0000 7f0c35400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c300b6840 con 0x7f0c300ceec0 2024-03-29T11:31:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.754+0000 7f0c35400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c300d2b30 con 0x7f0c301619d0 2024-03-29T11:31:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.754+0000 7f0c2e800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0c3015d640 0x7f0c300d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38162/0 (socket says 172.21.15.12:38162) 2024-03-29T11:31:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.754+0000 7f0c2e800700 1 -- 172.21.15.12:0/3086266002 learned_addr learned my addr 172.21.15.12:0/3086266002 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.754+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225734635 0 0) 0x7f0c300d30d0 con 0x7f0c3015d640 2024-03-29T11:31:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.754+0000 7f0c2f200700 1 --2- 172.21.15.12:0/3086266002 >> v2:172.21.15.159:3300/0 conn(0x7f0c301619d0 0x7f0c301685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.754+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c140035b0 con 0x7f0c3015d640 2024-03-29T11:31:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3590546064 0 0) 0x7f0c140035b0 con 0x7f0c3015d640 2024-03-29T11:31:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 >> v2:172.21.15.159:3300/0 conn(0x7f0c301619d0 msgr2=0x7f0c301685f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 --2- 172.21.15.12:0/3086266002 >> v2:172.21.15.159:3300/0 conn(0x7f0c301619d0 0x7f0c301685f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 >> v1:172.21.15.159:6789/0 conn(0x7f0c300ceec0 legacy=0x7f0c301612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c300d2df0 con 0x7f0c3015d640 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0c20001670 con 0x7f0c3015d640 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 >> v1:172.21.15.12:6789/0 conn(0x7f0c3015d640 legacy=0x7f0c300d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c14003a00 0x7f0c14003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c14004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c14005210 0x7f0c1400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300d2b30 con 0x7f0c14004850 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1400c1b0 con 0x7f0c14005210 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1400c370 con 0x7f0c14003a00 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2de00700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c14004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2e800700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c14003a00 0x7f0c14003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.755+0000 7f0c2f200700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c14005210 0x7f0c1400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.756+0000 7f0c2de00700 1 -- 172.21.15.12:0/3086266002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c14003a00 msgr2=0x7f0c14003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.756+0000 7f0c2de00700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c14003a00 0x7f0c14003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.756+0000 7f0c2de00700 1 -- 172.21.15.12:0/3086266002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c14005210 msgr2=0x7f0c1400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.756+0000 7f0c2de00700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c14005210 0x7f0c1400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.756+0000 7f0c2de00700 1 -- 172.21.15.12:0/3086266002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c140035b0 con 0x7f0c14004850 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.756+0000 7f0c2f200700 1 -- 172.21.15.12:0/3086266002 reap_dead start 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.756+0000 7f0c2de00700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c14004cd0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0017c0 tx=0x7f0c14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b605b1e187ae9795 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.756+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0052d0 con 0x7f0c14004850 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.756+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1c005450 con 0x7f0c14004850 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.756+0000 7f0c2d400700 1 -- 172.21.15.12:0/3086266002 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0038c0 con 0x7f0c14004850 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.757+0000 7f0c35400700 1 -- 172.21.15.12:0/3086266002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 msgr2=0x7f0c14004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.757+0000 7f0c35400700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c14004cd0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0017c0 tx=0x7f0c14005880 comp rx=0 tx=0).stop 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.757+0000 7f0c35400700 1 -- 172.21.15.12:0/3086266002 shutdown_connections 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.757+0000 7f0c35400700 1 --2- 172.21.15.12:0/3086266002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c14004cd0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.757+0000 7f0c35400700 1 -- 172.21.15.12:0/3086266002 >> 172.21.15.12:0/3086266002 conn(0x7f0c3008aff0 msgr2=0x7f0c300d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.757+0000 7f0c35400700 1 -- 172.21.15.12:0/3086266002 shutdown_connections 2024-03-29T11:31:23.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.757+0000 7f0c35400700 1 -- 172.21.15.12:0/3086266002 wait complete. 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.759+0000 7f0c35400700 1 Processor -- start 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.759+0000 7f0c35400700 1 -- start start 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c35400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c3009fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c35400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c3009e690 0x7f0c301f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c35400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c300a0250 0x7f0c301f68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c35400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3009d5f0 con 0x7f0c14004850 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c35400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3009d8d0 con 0x7f0c3009e690 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c35400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3009dbb0 con 0x7f0c300a0250 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c2e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c3009fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c2e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c3009fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55670/0 (socket says 172.21.15.12:55670) 2024-03-29T11:31:23.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c2e800700 1 -- 172.21.15.12:0/4175139254 learned_addr learned my addr 172.21.15.12:0/4175139254 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c2f200700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c300a0250 0x7f0c301f68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.760+0000 7f0c2de00700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c3009e690 0x7f0c301f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c2e800700 1 -- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c300a0250 msgr2=0x7f0c301f68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c2e800700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c300a0250 0x7f0c301f68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c2e800700 1 -- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c3009e690 msgr2=0x7f0c301f63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c2e800700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c3009e690 0x7f0c301f63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c2e800700 1 -- 172.21.15.12:0/4175139254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3009de90 con 0x7f0c14004850 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c2e800700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c3009fd10 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f0c200072c0 tx=0x7f0c200075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c26a00700 1 -- 172.21.15.12:0/4175139254 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20011580 con 0x7f0c14004850 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c26a00700 1 -- 172.21.15.12:0/4175139254 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c20011700 con 0x7f0c14004850 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c26a00700 1 -- 172.21.15.12:0/4175139254 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20011ab0 con 0x7f0c14004850 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c35400700 1 -- 172.21.15.12:0/4175139254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3009e170 con 0x7f0c14004850 2024-03-29T11:31:23.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.761+0000 7f0c35400700 1 -- 172.21.15.12:0/4175139254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3015a170 con 0x7f0c14004850 2024-03-29T11:31:23.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.762+0000 7f0c35400700 1 -- 172.21.15.12:0/4175139254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c30067720 con 0x7f0c14004850 2024-03-29T11:31:23.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.763+0000 7f0c26a00700 1 -- 172.21.15.12:0/4175139254 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0c20011c10 con 0x7f0c14004850 2024-03-29T11:31:23.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.764+0000 7f0c26a00700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c1006adb0 0x7f0c1006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:23.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.764+0000 7f0c26a00700 1 -- 172.21.15.12:0/4175139254 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f0c20089120 con 0x7f0c14004850 2024-03-29T11:31:23.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.764+0000 7f0c2de00700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c1006adb0 0x7f0c1006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:23.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.765+0000 7f0c2de00700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c1006adb0 0x7f0c1006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c005040 tx=0x7f0c1c009030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:23.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.768+0000 7f0c26a00700 1 -- 172.21.15.12:0/4175139254 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c20054450 con 0x7f0c14004850 2024-03-29T11:31:23.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.983+0000 7f0c35400700 1 -- 172.21.15.12:0/4175139254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0c300b5c70 con 0x7f0c14004850 2024-03-29T11:31:23.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.984+0000 7f0c26a00700 1 -- 172.21.15.12:0/4175139254 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v223) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0c2005ce60 con 0x7f0c14004850 2024-03-29T11:31:23.978 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:23.978 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:31:23.978 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:23.978 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T11:31:23.978 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T11:31:23.979 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:23.979 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:31:23.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 -- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c1006adb0 msgr2=0x7f0c1006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c1006adb0 0x7f0c1006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c005040 tx=0x7f0c1c009030 comp rx=0 tx=0).stop 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 -- 172.21.15.12:0/4175139254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 msgr2=0x7f0c3009fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c3009fd10 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f0c200072c0 tx=0x7f0c200075d0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 -- 172.21.15.12:0/4175139254 shutdown_connections 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0c1006adb0 0x7f0c1006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c300a0250 0x7f0c301f68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c14004850 0x7f0c3009fd10 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 --2- 172.21.15.12:0/4175139254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c3009e690 0x7f0c301f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 -- 172.21.15.12:0/4175139254 >> 172.21.15.12:0/4175139254 conn(0x7f0c3008aff0 msgr2=0x7f0c3015b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 -- 172.21.15.12:0/4175139254 shutdown_connections 2024-03-29T11:31:23.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:23.986+0000 7f0c35400700 1 -- 172.21.15.12:0/4175139254 wait complete. 2024-03-29T11:31:23.998 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T11:31:23.999 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T11:31:24.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.207+0000 7f2e5dc00700 1 Processor -- start 2024-03-29T11:31:24.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.207+0000 7f2e5dc00700 1 -- start start 2024-03-29T11:31:24.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.208+0000 7f2e5dc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2e5815ddf0 0x7f2e580ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:24.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.208+0000 7f2e5dc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e580ce8a0 con 0x7f2e5815e1d0 2024-03-29T11:31:24.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.208+0000 7f2e5dc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e580cdc90 con 0x7f2e580d28e0 2024-03-29T11:31:24.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.208+0000 7f2e5dc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e5815ec60 con 0x7f2e5815ddf0 2024-03-29T11:31:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.208+0000 7f2e56000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2e5815e1d0 0x7f2e580d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38178/0 (socket says 172.21.15.12:38178) 2024-03-29T11:31:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.208+0000 7f2e56000700 1 -- 172.21.15.12:0/3189727062 learned_addr learned my addr 172.21.15.12:0/3189727062 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.208+0000 7f2e56a00700 1 --2- 172.21.15.12:0/3189727062 >> v2:172.21.15.159:3300/0 conn(0x7f2e5815ddf0 0x7f2e580ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.209+0000 7f2e55600700 1 -- 172.21.15.12:0/3189727062 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111996094 0 0) 0x7f2e580ce8a0 con 0x7f2e5815e1d0 2024-03-29T11:31:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.209+0000 7f2e55600700 1 -- 172.21.15.12:0/3189727062 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e3c0035b0 con 0x7f2e5815e1d0 2024-03-29T11:31:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.209+0000 7f2e56a00700 1 -- 172.21.15.12:0/3189727062 >> v1:172.21.15.159:6789/0 conn(0x7f2e580d28e0 legacy=0x7f2e5815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.209+0000 7f2e56a00700 1 -- 172.21.15.12:0/3189727062 >> v1:172.21.15.12:6789/0 conn(0x7f2e5815e1d0 legacy=0x7f2e580d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.209+0000 7f2e56a00700 1 -- 172.21.15.12:0/3189727062 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e580cdf70 con 0x7f2e5815ddf0 2024-03-29T11:31:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.209+0000 7f2e56a00700 1 --2- 172.21.15.12:0/3189727062 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e5815ddf0 0x7f2e580ced30 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009840 tx=0x7f2e48009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beb6304e5b64f301 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:24.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.210+0000 7f2e55600700 1 -- 172.21.15.12:0/3189727062 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e48011830 con 0x7f2e5815ddf0 2024-03-29T11:31:24.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.210+0000 7f2e55600700 1 -- 172.21.15.12:0/3189727062 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e48011990 con 0x7f2e5815ddf0 2024-03-29T11:31:24.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.210+0000 7f2e55600700 1 -- 172.21.15.12:0/3189727062 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e48011d20 con 0x7f2e5815ddf0 2024-03-29T11:31:24.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.210+0000 7f2e5dc00700 1 -- 172.21.15.12:0/3189727062 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e5815ddf0 msgr2=0x7f2e580ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.210+0000 7f2e5dc00700 1 --2- 172.21.15.12:0/3189727062 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e5815ddf0 0x7f2e580ced30 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009840 tx=0x7f2e48009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:24.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.210+0000 7f2e5dc00700 1 -- 172.21.15.12:0/3189727062 shutdown_connections 2024-03-29T11:31:24.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.210+0000 7f2e5dc00700 1 --2- 172.21.15.12:0/3189727062 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e5815ddf0 0x7f2e580ced30 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:24.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.210+0000 7f2e5dc00700 1 -- 172.21.15.12:0/3189727062 >> 172.21.15.12:0/3189727062 conn(0x7f2e5808aff0 msgr2=0x7f2e580d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:24.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.210+0000 7f2e5dc00700 1 -- 172.21.15.12:0/3189727062 shutdown_connections 2024-03-29T11:31:24.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.210+0000 7f2e5dc00700 1 -- 172.21.15.12:0/3189727062 wait complete. 2024-03-29T11:31:24.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.212+0000 7f2e5dc00700 1 Processor -- start 2024-03-29T11:31:24.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.212+0000 7f2e5dc00700 1 -- start start 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.213+0000 7f2e5dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e5815ddf0 0x7f2e581e9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.213+0000 7f2e5dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e5815e1d0 0x7f2e580b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.213+0000 7f2e5dc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e580b27d0 0x7f2e580b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.213+0000 7f2e5dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e580b2d20 con 0x7f2e5815e1d0 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.213+0000 7f2e5dc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e580b3000 con 0x7f2e580b27d0 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.213+0000 7f2e5dc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e580b32e0 con 0x7f2e5815ddf0 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.213+0000 7f2e56000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e5815e1d0 0x7f2e580b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e56000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e5815e1d0 0x7f2e580b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55676/0 (socket says 172.21.15.12:55676) 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e56000700 1 -- 172.21.15.12:0/182682540 learned_addr learned my addr 172.21.15.12:0/182682540 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e57400700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e580b27d0 0x7f2e580b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:24.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e56a00700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e5815ddf0 0x7f2e581e9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e56000700 1 -- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e5815ddf0 msgr2=0x7f2e581e9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e56000700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e5815ddf0 0x7f2e581e9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e56000700 1 -- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e580b27d0 msgr2=0x7f2e580b4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e56000700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e580b27d0 0x7f2e580b4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e56000700 1 -- 172.21.15.12:0/182682540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e480094a0 con 0x7f2e5815e1d0 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e56000700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e5815e1d0 0x7f2e580b3dc0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400c700 tx=0x7f2e4400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e4ea00700 1 -- 172.21.15.12:0/182682540 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e44005d70 con 0x7f2e5815e1d0 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e4ea00700 1 -- 172.21.15.12:0/182682540 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e440070d0 con 0x7f2e5815e1d0 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.214+0000 7f2e5dc00700 1 -- 172.21.15.12:0/182682540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e580b35c0 con 0x7f2e5815e1d0 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.215+0000 7f2e4ea00700 1 -- 172.21.15.12:0/182682540 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e44007460 con 0x7f2e5815e1d0 2024-03-29T11:31:24.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.215+0000 7f2e5dc00700 1 -- 172.21.15.12:0/182682540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e581e9780 con 0x7f2e5815e1d0 2024-03-29T11:31:24.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.216+0000 7f2e5dc00700 1 -- 172.21.15.12:0/182682540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e581e9910 con 0x7f2e5815e1d0 2024-03-29T11:31:24.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.217+0000 7f2e4ea00700 1 -- 172.21.15.12:0/182682540 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2e440075c0 con 0x7f2e5815e1d0 2024-03-29T11:31:24.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.217+0000 7f2e4ea00700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2e3806adb0 0x7f2e3806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:24.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.218+0000 7f2e4ea00700 1 -- 172.21.15.12:0/182682540 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f2e44088890 con 0x7f2e5815e1d0 2024-03-29T11:31:24.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.218+0000 7f2e56a00700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2e3806adb0 0x7f2e3806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:24.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.219+0000 7f2e56a00700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2e3806adb0 0x7f2e3806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2e480090b0 tx=0x7f2e48008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:24.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.222+0000 7f2e4ea00700 1 -- 172.21.15.12:0/182682540 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e581e9910 con 0x7f2e5815e1d0 2024-03-29T11:31:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:24.437+0000 7f2e5dc00700 1 -- 172.21.15.12:0/182682540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2e581e9910 con 0x7f2e5815e1d0 2024-03-29T11:31:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.296+0000 7f2e4ea00700 1 -- 172.21.15.12:0/182682540 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v224) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2e581e9910 con 0x7f2e5815e1d0 2024-03-29T11:31:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.302+0000 7f2e5dc00700 1 -- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2e3806adb0 msgr2=0x7f2e3806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.302+0000 7f2e5dc00700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2e3806adb0 0x7f2e3806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2e480090b0 tx=0x7f2e48008030 comp rx=0 tx=0).stop 2024-03-29T11:31:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.302+0000 7f2e5dc00700 1 -- 172.21.15.12:0/182682540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e5815e1d0 msgr2=0x7f2e580b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.302+0000 7f2e5dc00700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e5815e1d0 0x7f2e580b3dc0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400c700 tx=0x7f2e4400cac0 comp rx=0 tx=0).stop 2024-03-29T11:31:25.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.303+0000 7f2e5dc00700 1 -- 172.21.15.12:0/182682540 shutdown_connections 2024-03-29T11:31:25.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.303+0000 7f2e5dc00700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2e3806adb0 0x7f2e3806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:25.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.303+0000 7f2e5dc00700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e580b27d0 0x7f2e580b4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:25.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.303+0000 7f2e5dc00700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e5815ddf0 0x7f2e581e9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:25.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.303+0000 7f2e5dc00700 1 --2- 172.21.15.12:0/182682540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e5815e1d0 0x7f2e580b3dc0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:25.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.303+0000 7f2e5dc00700 1 -- 172.21.15.12:0/182682540 >> 172.21.15.12:0/182682540 conn(0x7f2e5808aff0 msgr2=0x7f2e580b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:25.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.303+0000 7f2e5dc00700 1 -- 172.21.15.12:0/182682540 shutdown_connections 2024-03-29T11:31:25.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.303+0000 7f2e5dc00700 1 -- 172.21.15.12:0/182682540 wait complete. 2024-03-29T11:31:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T11:31:25.321 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T11:31:25.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.541+0000 7fe61e000700 1 Processor -- start 2024-03-29T11:31:25.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.541+0000 7fe61e000700 1 -- start start 2024-03-29T11:31:25.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.542+0000 7fe61e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6180ced80 0x7fe6180d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:25.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.542+0000 7fe61e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe61815fcb0 0x7fe618160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.542+0000 7fe61e000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180cf1e0 con 0x7fe61815dad0 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.542+0000 7fe61e000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6180cfcc0 con 0x7fe61815fcb0 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.542+0000 7fe61e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180cd9f0 con 0x7fe6180ced80 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.542+0000 7fe617e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe61815fcb0 0x7fe618160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.542+0000 7fe617e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe61815fcb0 0x7fe618160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55678/0 (socket says 172.21.15.12:55678) 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.542+0000 7fe617e00700 1 -- 172.21.15.12:0/1230810129 learned_addr learned my addr 172.21.15.12:0/1230810129 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.542+0000 7fe616a00700 1 --2- 172.21.15.12:0/1230810129 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6180ced80 0x7fe6180d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.543+0000 7fe617e00700 1 -- 172.21.15.12:0/1230810129 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6180ced80 msgr2=0x7fe6180d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.543+0000 7fe617e00700 1 --2- 172.21.15.12:0/1230810129 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6180ced80 0x7fe6180d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.543+0000 7fe617e00700 1 -- 172.21.15.12:0/1230810129 >> v1:172.21.15.159:6789/0 conn(0x7fe61815dad0 legacy=0x7fe6180d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:25.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.543+0000 7fe617e00700 1 -- 172.21.15.12:0/1230810129 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61815f820 con 0x7fe61815fcb0 2024-03-29T11:31:25.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.543+0000 7fe617e00700 1 --2- 172.21.15.12:0/1230810129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 0x7fe618160140 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe610009e50 tx=0x7fe61000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b64d02901cef55 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe616000700 1 -- 172.21.15.12:0/1230810129 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000e070 con 0x7fe61815fcb0 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe616000700 1 -- 172.21.15.12:0/1230810129 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61000b910 con 0x7fe61815fcb0 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe616000700 1 -- 172.21.15.12:0/1230810129 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000bcc0 con 0x7fe61815fcb0 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe61e000700 1 -- 172.21.15.12:0/1230810129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 msgr2=0x7fe618160140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe61e000700 1 --2- 172.21.15.12:0/1230810129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 0x7fe618160140 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe610009e50 tx=0x7fe61000b330 comp rx=0 tx=0).stop 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe61e000700 1 -- 172.21.15.12:0/1230810129 shutdown_connections 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe61e000700 1 --2- 172.21.15.12:0/1230810129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 0x7fe618160140 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe61e000700 1 --2- 172.21.15.12:0/1230810129 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6180ced80 0x7fe6180d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe61e000700 1 -- 172.21.15.12:0/1230810129 >> 172.21.15.12:0/1230810129 conn(0x7fe61808aff0 msgr2=0x7fe6180d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe61e000700 1 -- 172.21.15.12:0/1230810129 shutdown_connections 2024-03-29T11:31:25.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.544+0000 7fe61e000700 1 -- 172.21.15.12:0/1230810129 wait complete. 2024-03-29T11:31:25.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.546+0000 7fe61e000700 1 Processor -- start 2024-03-29T11:31:25.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.546+0000 7fe61e000700 1 -- start start 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.547+0000 7fe61e000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6180ced80 0x7fe6180c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.547+0000 7fe61e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe61815dad0 0x7fe61814a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.547+0000 7fe61e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 0x7fe6181f2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.547+0000 7fe61e000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe618156170 con 0x7fe61815fcb0 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.547+0000 7fe61e000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61814a890 con 0x7fe6180ced80 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.547+0000 7fe61e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe618157c30 con 0x7fe61815dad0 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 0x7fe6181f2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 0x7fe6181f2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55680/0 (socket says 172.21.15.12:55680) 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617e00700 1 -- 172.21.15.12:0/3377443671 learned_addr learned my addr 172.21.15.12:0/3377443671 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617400700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6180ced80 0x7fe6180c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe616a00700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe61815dad0 0x7fe61814a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617e00700 1 -- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe61815dad0 msgr2=0x7fe61814a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617e00700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe61815dad0 0x7fe61814a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617e00700 1 -- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6180ced80 msgr2=0x7fe6180c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617e00700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6180ced80 0x7fe6180c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617e00700 1 -- 172.21.15.12:0/3377443671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe610009b30 con 0x7fe61815fcb0 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617400700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6180ced80 0x7fe6180c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.548+0000 7fe617e00700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 0x7fe6181f2bb0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fe61808ce40 tx=0x7fe610004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.549+0000 7fe60f400700 1 -- 172.21.15.12:0/3377443671 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6100048b0 con 0x7fe61815fcb0 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.549+0000 7fe60f400700 1 -- 172.21.15.12:0/3377443671 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6100092e0 con 0x7fe61815fcb0 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.549+0000 7fe61e000700 1 -- 172.21.15.12:0/3377443671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe618157f10 con 0x7fe61815fcb0 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.549+0000 7fe60f400700 1 -- 172.21.15.12:0/3377443671 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61001b650 con 0x7fe61815fcb0 2024-03-29T11:31:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.549+0000 7fe61e000700 1 -- 172.21.15.12:0/3377443671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6181584d0 con 0x7fe61815fcb0 2024-03-29T11:31:25.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.550+0000 7fe61e000700 1 -- 172.21.15.12:0/3377443671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe618067720 con 0x7fe61815fcb0 2024-03-29T11:31:25.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.551+0000 7fe60f400700 1 -- 172.21.15.12:0/3377443671 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe61001b7b0 con 0x7fe61815fcb0 2024-03-29T11:31:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.552+0000 7fe60f400700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5f806adb0 0x7fe5f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.552+0000 7fe60f400700 1 -- 172.21.15.12:0/3377443671 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 5299+0+0 (secure 0 0 0) 0x7fe61008ad10 con 0x7fe61815fcb0 2024-03-29T11:31:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.552+0000 7fe617400700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5f806adb0 0x7fe5f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.553+0000 7fe617400700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5f806adb0 0x7fe5f806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe608000f60 tx=0x7fe608009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:25.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.557+0000 7fe60f400700 1 -- 172.21.15.12:0/3377443671 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe610055e50 con 0x7fe61815fcb0 2024-03-29T11:31:25.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:25.776+0000 7fe61e000700 1 -- 172.21.15.12:0/3377443671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe6180d07b0 con 0x7fe61815fcb0 2024-03-29T11:31:26.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.298+0000 7fe60f400700 1 -- 172.21.15.12:0/3377443671 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v225) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe610059470 con 0x7fe61815fcb0 2024-03-29T11:31:26.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.304+0000 7fe61e000700 1 -- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5f806adb0 msgr2=0x7fe5f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:26.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.304+0000 7fe61e000700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5f806adb0 0x7fe5f806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe608000f60 tx=0x7fe608009040 comp rx=0 tx=0).stop 2024-03-29T11:31:26.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.304+0000 7fe61e000700 1 -- 172.21.15.12:0/3377443671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 msgr2=0x7fe6181f2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:26.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.304+0000 7fe61e000700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 0x7fe6181f2bb0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fe61808ce40 tx=0x7fe610004ad0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.305+0000 7fe61e000700 1 -- 172.21.15.12:0/3377443671 shutdown_connections 2024-03-29T11:31:26.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.305+0000 7fe61e000700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5f806adb0 0x7fe5f806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.305+0000 7fe61e000700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe61815fcb0 0x7fe6181f2bb0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.305+0000 7fe61e000700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6180ced80 0x7fe6180c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.305+0000 7fe61e000700 1 --2- 172.21.15.12:0/3377443671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe61815dad0 0x7fe61814a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.305+0000 7fe61e000700 1 -- 172.21.15.12:0/3377443671 >> 172.21.15.12:0/3377443671 conn(0x7fe61808aff0 msgr2=0x7fe6180cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:26.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.306+0000 7fe61e000700 1 -- 172.21.15.12:0/3377443671 shutdown_connections 2024-03-29T11:31:26.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.307+0000 7fe61e000700 1 -- 172.21.15.12:0/3377443671 wait complete. 2024-03-29T11:31:26.308 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T11:31:26.331 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T11:31:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.538+0000 7fc5c6c00700 1 Processor -- start 2024-03-29T11:31:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.539+0000 7fc5c6c00700 1 -- start start 2024-03-29T11:31:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.540+0000 7fc5c6c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 0x7fc5c0165370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.540+0000 7fc5c6c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c00cb310 con 0x7fc5c015dfe0 2024-03-29T11:31:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.540+0000 7fc5c6c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c00cca90 con 0x7fc5c015a5c0 2024-03-29T11:31:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.540+0000 7fc5c6c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c00711a0 con 0x7fc5c00ce840 2024-03-29T11:31:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.540+0000 7fc5bf400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc5c015dfe0 0x7fc5c00ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38192/0 (socket says 172.21.15.12:38192) 2024-03-29T11:31:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.540+0000 7fc5bf400700 1 -- 172.21.15.12:0/3285128949 learned_addr learned my addr 172.21.15.12:0/3285128949 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.540+0000 7fc5c4c00700 1 --2- 172.21.15.12:0/3285128949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 0x7fc5c0165370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.541+0000 7fc5bea00700 1 -- 172.21.15.12:0/3285128949 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942735799 0 0) 0x7fc5c00cb310 con 0x7fc5c015dfe0 2024-03-29T11:31:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.541+0000 7fc5bea00700 1 -- 172.21.15.12:0/3285128949 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a40035b0 con 0x7fc5c015dfe0 2024-03-29T11:31:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.541+0000 7fc5c4c00700 1 -- 172.21.15.12:0/3285128949 >> v1:172.21.15.159:6789/0 conn(0x7fc5c015a5c0 legacy=0x7fc5c00709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.541+0000 7fc5bea00700 1 -- 172.21.15.12:0/3285128949 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005152650 0 0) 0x7fc5c00cca90 con 0x7fc5c015a5c0 2024-03-29T11:31:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.541+0000 7fc5c4c00700 1 -- 172.21.15.12:0/3285128949 >> v1:172.21.15.12:6789/0 conn(0x7fc5c015dfe0 legacy=0x7fc5c00ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.541+0000 7fc5c4c00700 1 -- 172.21.15.12:0/3285128949 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c0071480 con 0x7fc5c00ce840 2024-03-29T11:31:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.541+0000 7fc5c4c00700 1 --2- 172.21.15.12:0/3285128949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 0x7fc5c0165370 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009dc0 tx=0x7fc5b800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e497b7273697a2e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.542+0000 7fc5bea00700 1 -- 172.21.15.12:0/3285128949 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800e070 con 0x7fc5c00ce840 2024-03-29T11:31:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.542+0000 7fc5bea00700 1 -- 172.21.15.12:0/3285128949 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b800bb20 con 0x7fc5c00ce840 2024-03-29T11:31:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.542+0000 7fc5bea00700 1 -- 172.21.15.12:0/3285128949 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800be90 con 0x7fc5c00ce840 2024-03-29T11:31:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.542+0000 7fc5c6c00700 1 -- 172.21.15.12:0/3285128949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 msgr2=0x7fc5c0165370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.542+0000 7fc5c6c00700 1 --2- 172.21.15.12:0/3285128949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 0x7fc5c0165370 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009dc0 tx=0x7fc5b800b330 comp rx=0 tx=0).stop 2024-03-29T11:31:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.542+0000 7fc5c6c00700 1 -- 172.21.15.12:0/3285128949 shutdown_connections 2024-03-29T11:31:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.542+0000 7fc5c6c00700 1 --2- 172.21.15.12:0/3285128949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 0x7fc5c0165370 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.542+0000 7fc5c6c00700 1 -- 172.21.15.12:0/3285128949 >> 172.21.15.12:0/3285128949 conn(0x7fc5c00b0d70 msgr2=0x7fc5c00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.542+0000 7fc5c6c00700 1 -- 172.21.15.12:0/3285128949 shutdown_connections 2024-03-29T11:31:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.542+0000 7fc5c6c00700 1 -- 172.21.15.12:0/3285128949 wait complete. 2024-03-29T11:31:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.544+0000 7fc5c6c00700 1 Processor -- start 2024-03-29T11:31:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.544+0000 7fc5c6c00700 1 -- start start 2024-03-29T11:31:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.545+0000 7fc5c6c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 0x7fc5c0159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.545+0000 7fc5c6c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5c015a5c0 0x7fc5c01f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.545+0000 7fc5c6c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5c015dfe0 0x7fc5c014dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.545+0000 7fc5c6c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c01583b0 con 0x7fc5c015a5c0 2024-03-29T11:31:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.545+0000 7fc5c6c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0158690 con 0x7fc5c015dfe0 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.545+0000 7fc5c6c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0158970 con 0x7fc5c00ce840 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.545+0000 7fc5bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5c015a5c0 0x7fc5c01f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.545+0000 7fc5bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5c015a5c0 0x7fc5c01f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55696/0 (socket says 172.21.15.12:55696) 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.545+0000 7fc5bf400700 1 -- 172.21.15.12:0/1347108006 learned_addr learned my addr 172.21.15.12:0/1347108006 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5c4c00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5c015dfe0 0x7fc5c014dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5bfe00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 0x7fc5c0159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5bf400700 1 -- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 msgr2=0x7fc5c0159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5bf400700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 0x7fc5c0159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5bf400700 1 -- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5c015dfe0 msgr2=0x7fc5c014dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5bf400700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5c015dfe0 0x7fc5c014dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5bf400700 1 -- 172.21.15.12:0/1347108006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b8009a70 con 0x7fc5c015a5c0 2024-03-29T11:31:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5bfe00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 0x7fc5c0159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5bf400700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5c015a5c0 0x7fc5c01f2db0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00c420 tx=0x7fc5ac00c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5b7e00700 1 -- 172.21.15.12:0/1347108006 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac005d70 con 0x7fc5c015a5c0 2024-03-29T11:31:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5b7e00700 1 -- 172.21.15.12:0/1347108006 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac0070d0 con 0x7fc5c015a5c0 2024-03-29T11:31:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5c6c00700 1 -- 172.21.15.12:0/1347108006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c0158c50 con 0x7fc5c015a5c0 2024-03-29T11:31:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.546+0000 7fc5b7e00700 1 -- 172.21.15.12:0/1347108006 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac007440 con 0x7fc5c015a5c0 2024-03-29T11:31:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.547+0000 7fc5c6c00700 1 -- 172.21.15.12:0/1347108006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c0159210 con 0x7fc5c015a5c0 2024-03-29T11:31:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.548+0000 7fc5c6c00700 1 -- 172.21.15.12:0/1347108006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c0067720 con 0x7fc5c015a5c0 2024-03-29T11:31:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.549+0000 7fc5b7e00700 1 -- 172.21.15.12:0/1347108006 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc5ac0075a0 con 0x7fc5c015a5c0 2024-03-29T11:31:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.550+0000 7fc5b7e00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc5a006adb0 0x7fc5a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.550+0000 7fc5b7e00700 1 -- 172.21.15.12:0/1347108006 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5681+0+0 (secure 0 0 0) 0x7fc5ac088bd0 con 0x7fc5c015a5c0 2024-03-29T11:31:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.550+0000 7fc5bfe00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc5a006adb0 0x7fc5a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.551+0000 7fc5bfe00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc5a006adb0 0x7fc5a006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0007fd0 tx=0x7fc5b0009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:26.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.554+0000 7fc5b7e00700 1 -- 172.21.15.12:0/1347108006 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc5ac053dd0 con 0x7fc5c015a5c0 2024-03-29T11:31:26.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.814+0000 7fc5c6c00700 1 -- 172.21.15.12:0/1347108006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc5c006f2d0 con 0x7fc5c015a5c0 2024-03-29T11:31:26.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.821+0000 7fc5b7e00700 1 -- 172.21.15.12:0/1347108006 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v102) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc5ac0170c0 con 0x7fc5c015a5c0 2024-03-29T11:31:26.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.826+0000 7fc5c6c00700 1 -- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc5a006adb0 msgr2=0x7fc5a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.826+0000 7fc5c6c00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc5a006adb0 0x7fc5a006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0007fd0 tx=0x7fc5b0009040 comp rx=0 tx=0).stop 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.826+0000 7fc5c6c00700 1 -- 172.21.15.12:0/1347108006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5c015a5c0 msgr2=0x7fc5c01f2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.826+0000 7fc5c6c00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5c015a5c0 0x7fc5c01f2db0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00c420 tx=0x7fc5ac00c730 comp rx=0 tx=0).stop 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.827+0000 7fc5c6c00700 1 -- 172.21.15.12:0/1347108006 shutdown_connections 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.827+0000 7fc5c6c00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc5a006adb0 0x7fc5a006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.827+0000 7fc5c6c00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5c015dfe0 0x7fc5c014dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.827+0000 7fc5c6c00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5c00ce840 0x7fc5c0159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.827+0000 7fc5c6c00700 1 --2- 172.21.15.12:0/1347108006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5c015a5c0 0x7fc5c01f2db0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.827+0000 7fc5c6c00700 1 -- 172.21.15.12:0/1347108006 >> 172.21.15.12:0/1347108006 conn(0x7fc5c00b0d70 msgr2=0x7fc5c01554c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.828+0000 7fc5c6c00700 1 -- 172.21.15.12:0/1347108006 shutdown_connections 2024-03-29T11:31:26.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:26.828+0000 7fc5c6c00700 1 -- 172.21.15.12:0/1347108006 wait complete. 2024-03-29T11:31:26.827 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 26 and data pool 27 2024-03-29T11:31:26.854 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.059+0000 7f68ade00700 1 Processor -- start 2024-03-29T11:31:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.059+0000 7f68ade00700 1 -- start start 2024-03-29T11:31:27.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68ade00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a815d640 0x7f68a80d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68ade00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f68a80ceec0 0x7f68a81612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68ade00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68a8161800 0x7f68a80cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68ade00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a8161c90 con 0x7f68a8161800 2024-03-29T11:31:27.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68ade00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a80d30d0 con 0x7f68a80ceec0 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68ade00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80b6840 con 0x7f68a815d640 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68a7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68a8161800 0x7f68a80cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68a7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68a8161800 0x7f68a80cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55700/0 (socket says 172.21.15.12:55700) 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68a7400700 1 -- 172.21.15.12:0/665655876 learned_addr learned my addr 172.21.15.12:0/665655876 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68a6a00700 1 --2- 172.21.15.12:0/665655876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a815d640 0x7f68a80d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.060+0000 7f68a6000700 1 --2- 172.21.15.12:0/665655876 >> v2:172.21.15.159:3300/0 conn(0x7f68a80ceec0 0x7f68a81612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.061+0000 7f68a7400700 1 -- 172.21.15.12:0/665655876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a815d640 msgr2=0x7f68a80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.061+0000 7f68a7400700 1 --2- 172.21.15.12:0/665655876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a815d640 0x7f68a80d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.061+0000 7f68a7400700 1 -- 172.21.15.12:0/665655876 >> v2:172.21.15.159:3300/0 conn(0x7f68a80ceec0 msgr2=0x7f68a81612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.061+0000 7f68a7400700 1 --2- 172.21.15.12:0/665655876 >> v2:172.21.15.159:3300/0 conn(0x7f68a80ceec0 0x7f68a81612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.061+0000 7f68a7400700 1 -- 172.21.15.12:0/665655876 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a80d2880 con 0x7f68a8161800 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.061+0000 7f68a6000700 1 --2- 172.21.15.12:0/665655876 >> v2:172.21.15.159:3300/0 conn(0x7f68a80ceec0 0x7f68a81612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.061+0000 7f68a7400700 1 --2- 172.21.15.12:0/665655876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a8161800 0x7f68a80cdba0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f68a808c1c0 tx=0x7f68a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef7d92e27c8a8e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.061+0000 7f68a5600700 1 -- 172.21.15.12:0/665655876 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a000e070 con 0x7f68a8161800 2024-03-29T11:31:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.061+0000 7f68a5600700 1 -- 172.21.15.12:0/665655876 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a000b910 con 0x7f68a8161800 2024-03-29T11:31:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.061+0000 7f68a5600700 1 -- 172.21.15.12:0/665655876 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a000bcc0 con 0x7f68a8161800 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.062+0000 7f68ade00700 1 -- 172.21.15.12:0/665655876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a8161800 msgr2=0x7f68a80cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.062+0000 7f68ade00700 1 --2- 172.21.15.12:0/665655876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a8161800 0x7f68a80cdba0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f68a808c1c0 tx=0x7f68a000b330 comp rx=0 tx=0).stop 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.062+0000 7f68ade00700 1 -- 172.21.15.12:0/665655876 shutdown_connections 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.062+0000 7f68ade00700 1 --2- 172.21.15.12:0/665655876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a8161800 0x7f68a80cdba0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.062+0000 7f68ade00700 1 --2- 172.21.15.12:0/665655876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a815d640 0x7f68a80d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.062+0000 7f68ade00700 1 --2- 172.21.15.12:0/665655876 >> v2:172.21.15.159:3300/0 conn(0x7f68a80ceec0 0x7f68a81612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.062+0000 7f68ade00700 1 -- 172.21.15.12:0/665655876 >> 172.21.15.12:0/665655876 conn(0x7f68a808aff0 msgr2=0x7f68a80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.062+0000 7f68ade00700 1 -- 172.21.15.12:0/665655876 shutdown_connections 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.062+0000 7f68ade00700 1 -- 172.21.15.12:0/665655876 wait complete. 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.064+0000 7f68ade00700 1 Processor -- start 2024-03-29T11:31:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.064+0000 7f68ade00700 1 -- start start 2024-03-29T11:31:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68ade00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a80ceec0 0x7f68a815a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68ade00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a815d640 0x7f68a8151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68ade00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a8161800 0x7f68a8154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68ade00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8156c90 con 0x7f68a80ceec0 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68ade00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8151fa0 con 0x7f68a815d640 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68ade00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a81569b0 con 0x7f68a8161800 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68a6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a80ceec0 0x7f68a815a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68a6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a80ceec0 0x7f68a815a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55710/0 (socket says 172.21.15.12:55710) 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68a6a00700 1 -- 172.21.15.12:0/1990919325 learned_addr learned my addr 172.21.15.12:0/1990919325 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68a6000700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a815d640 0x7f68a8151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.065+0000 7f68a7400700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a8161800 0x7f68a8154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f68a6a00700 1 -- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a8161800 msgr2=0x7f68a8154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f68a6a00700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a8161800 0x7f68a8154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f68a6a00700 1 -- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a815d640 msgr2=0x7f68a8151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f68a6a00700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a815d640 0x7f68a8151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f68a6a00700 1 -- 172.21.15.12:0/1990919325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a0009b30 con 0x7f68a80ceec0 2024-03-29T11:31:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f68a6a00700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a80ceec0 0x7f68a815a190 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f689800e9b0 tx=0x7f689800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f689f400700 1 -- 172.21.15.12:0/1990919325 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898009710 con 0x7f68a80ceec0 2024-03-29T11:31:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f689f400700 1 -- 172.21.15.12:0/1990919325 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689800ee50 con 0x7f68a80ceec0 2024-03-29T11:31:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f68ade00700 1 -- 172.21.15.12:0/1990919325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a80c7220 con 0x7f68a80ceec0 2024-03-29T11:31:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f689f400700 1 -- 172.21.15.12:0/1990919325 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689800ca50 con 0x7f68a80ceec0 2024-03-29T11:31:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.066+0000 7f68ade00700 1 -- 172.21.15.12:0/1990919325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a80c77e0 con 0x7f68a80ceec0 2024-03-29T11:31:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.067+0000 7f68ade00700 1 -- 172.21.15.12:0/1990919325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a80c7b50 con 0x7f68a80ceec0 2024-03-29T11:31:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.068+0000 7f689f400700 1 -- 172.21.15.12:0/1990919325 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f689800cbb0 con 0x7f68a80ceec0 2024-03-29T11:31:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.069+0000 7f689f400700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f688806adb0 0x7f688806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.069+0000 7f689f400700 1 -- 172.21.15.12:0/1990919325 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f6898089540 con 0x7f68a80ceec0 2024-03-29T11:31:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.070+0000 7f68a6000700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f688806adb0 0x7f688806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.070+0000 7f68a6000700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f688806adb0 0x7f688806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68a81f66a0 tx=0x7f6894008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.074+0000 7f689f400700 1 -- 172.21.15.12:0/1990919325 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6898054520 con 0x7f68a80ceec0 2024-03-29T11:31:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.288+0000 7f68ade00700 1 -- 172.21.15.12:0/1990919325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68a80c7080 con 0x7f68a80ceec0 2024-03-29T11:31:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.290+0000 7f689f400700 1 -- 172.21.15.12:0/1990919325 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+16244 (secure 0 0 0) 0x7f68a80c7080 con 0x7f68a80ceec0 2024-03-29T11:31:27.284 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:27.284 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":226,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:26.815902+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.292+0000 7f68ade00700 1 -- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f688806adb0 msgr2=0x7f688806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.292+0000 7f68ade00700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f688806adb0 0x7f688806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68a81f66a0 tx=0x7f6894008040 comp rx=0 tx=0).stop 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.292+0000 7f68ade00700 1 -- 172.21.15.12:0/1990919325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a80ceec0 msgr2=0x7f68a815a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.292+0000 7f68ade00700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a80ceec0 0x7f68a815a190 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f689800e9b0 tx=0x7f689800ecc0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.292+0000 7f68ade00700 1 -- 172.21.15.12:0/1990919325 shutdown_connections 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.292+0000 7f68ade00700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f688806adb0 0x7f688806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.292+0000 7f68ade00700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a8161800 0x7f68a8154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.292+0000 7f68ade00700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68a80ceec0 0x7f68a815a190 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.292+0000 7f68ade00700 1 --2- 172.21.15.12:0/1990919325 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a815d640 0x7f68a8151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.292+0000 7f68ade00700 1 -- 172.21.15.12:0/1990919325 >> 172.21.15.12:0/1990919325 conn(0x7f68a808aff0 msgr2=0x7f68a815cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.293+0000 7f68ade00700 1 -- 172.21.15.12:0/1990919325 shutdown_connections 2024-03-29T11:31:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.293+0000 7f68ade00700 1 -- 172.21.15.12:0/1990919325 wait complete. 2024-03-29T11:31:27.306 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:27.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.510+0000 7ff5e3000700 1 Processor -- start 2024-03-29T11:31:27.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.511+0000 7ff5e3000700 1 -- start start 2024-03-29T11:31:27.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.512+0000 7ff5e3000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff5dc0cf3c0 0x7ff5dc0d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.512+0000 7ff5e3000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc0d1330 con 0x7ff5dc15db40 2024-03-29T11:31:27.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.512+0000 7ff5e3000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc0d1610 con 0x7ff5dc15d760 2024-03-29T11:31:27.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.512+0000 7ff5e3000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5dc0d1fb0 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.513+0000 7ff5e1000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff5dc15db40 0x7ff5dc0cffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38204/0 (socket says 172.21.15.12:38204) 2024-03-29T11:31:27.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.513+0000 7ff5e1000700 1 -- 172.21.15.12:0/2770691554 learned_addr learned my addr 172.21.15.12:0/2770691554 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:27.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.513+0000 7ff5db400700 1 --2- 172.21.15.12:0/2770691554 >> v2:172.21.15.12:3300/0 conn(0x7ff5dc0cf3c0 0x7ff5dc0d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.513+0000 7ff5daa00700 1 -- 172.21.15.12:0/2770691554 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832355214 0 0) 0x7ff5dc0d1330 con 0x7ff5dc15db40 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.513+0000 7ff5daa00700 1 -- 172.21.15.12:0/2770691554 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c00035b0 con 0x7ff5dc15db40 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.514+0000 7ff5daa00700 1 -- 172.21.15.12:0/2770691554 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553508456 0 0) 0x7ff5dc0d1610 con 0x7ff5dc15d760 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.514+0000 7ff5daa00700 1 -- 172.21.15.12:0/2770691554 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5dc0d1330 con 0x7ff5dc15d760 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.514+0000 7ff5db400700 1 -- 172.21.15.12:0/2770691554 >> v1:172.21.15.159:6789/0 conn(0x7ff5dc15d760 legacy=0x7ff5dc0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.514+0000 7ff5db400700 1 -- 172.21.15.12:0/2770691554 >> v1:172.21.15.12:6789/0 conn(0x7ff5dc15db40 legacy=0x7ff5dc0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.514+0000 7ff5db400700 1 -- 172.21.15.12:0/2770691554 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc0d18f0 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.514+0000 7ff5db400700 1 --2- 172.21.15.12:0/2770691554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 0x7ff5dc0d23d0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009840 tx=0x7ff5c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ed8c392770336 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.514+0000 7ff5daa00700 1 -- 172.21.15.12:0/2770691554 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8011830 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.514+0000 7ff5daa00700 1 -- 172.21.15.12:0/2770691554 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c8011990 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.514+0000 7ff5daa00700 1 -- 172.21.15.12:0/2770691554 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8011d20 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.515+0000 7ff5e3000700 1 -- 172.21.15.12:0/2770691554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 msgr2=0x7ff5dc0d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.515+0000 7ff5e3000700 1 --2- 172.21.15.12:0/2770691554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 0x7ff5dc0d23d0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009840 tx=0x7ff5c8009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:27.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.515+0000 7ff5e3000700 1 -- 172.21.15.12:0/2770691554 shutdown_connections 2024-03-29T11:31:27.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.515+0000 7ff5e3000700 1 --2- 172.21.15.12:0/2770691554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 0x7ff5dc0d23d0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.515+0000 7ff5e3000700 1 -- 172.21.15.12:0/2770691554 >> 172.21.15.12:0/2770691554 conn(0x7ff5dc08aff0 msgr2=0x7ff5dc0d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:27.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.515+0000 7ff5e3000700 1 -- 172.21.15.12:0/2770691554 shutdown_connections 2024-03-29T11:31:27.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.515+0000 7ff5e3000700 1 -- 172.21.15.12:0/2770691554 wait complete. 2024-03-29T11:31:27.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.517+0000 7ff5e3000700 1 Processor -- start 2024-03-29T11:31:27.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.517+0000 7ff5e3000700 1 -- start start 2024-03-29T11:31:27.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5e3000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 0x7ff5dc154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5e3000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc15d760 0x7ff5dc155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5e3000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc15db40 0x7ff5dc151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5e3000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0ca6b0 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5e3000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc157d00 con 0x7ff5dc15db40 2024-03-29T11:31:27.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5e3000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0ca900 con 0x7ff5dc15d760 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5dbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 0x7ff5dc154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5dbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 0x7ff5dc154b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55720/0 (socket says 172.21.15.12:55720) 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5dbe00700 1 -- 172.21.15.12:0/3756392692 learned_addr learned my addr 172.21.15.12:0/3756392692 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5dbe00700 1 -- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc15d760 msgr2=0x7ff5dc155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5db400700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc15d760 0x7ff5dc155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5e1000700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc15db40 0x7ff5dc151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5dbe00700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc15d760 0x7ff5dc155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5dbe00700 1 -- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc15db40 msgr2=0x7ff5dc151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5dbe00700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc15db40 0x7ff5dc151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5dbe00700 1 -- 172.21.15.12:0/3756392692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c80094a0 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5db400700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc15d760 0x7ff5dc155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:27.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.518+0000 7ff5e1000700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc15db40 0x7ff5dc151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:27.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.519+0000 7ff5dbe00700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 0x7ff5dc154b50 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00c520 tx=0x7ff5cc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:27.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.519+0000 7ff5d3e00700 1 -- 172.21.15.12:0/3756392692 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc0052d0 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.519+0000 7ff5d3e00700 1 -- 172.21.15.12:0/3756392692 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5cc005430 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.519+0000 7ff5d3e00700 1 -- 172.21.15.12:0/3756392692 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc0038c0 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.519+0000 7ff5e3000700 1 -- 172.21.15.12:0/3756392692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc0cabe0 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.519+0000 7ff5e3000700 1 -- 172.21.15.12:0/3756392692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc0b64b0 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.520+0000 7ff5e3000700 1 -- 172.21.15.12:0/3756392692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc067720 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.521+0000 7ff5d3e00700 1 -- 172.21.15.12:0/3756392692 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff5cc003a20 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.522+0000 7ff5d3e00700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff5bc06adb0 0x7ff5bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.522+0000 7ff5d3e00700 1 -- 172.21.15.12:0/3756392692 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5781+0+0 (secure 0 0 0) 0x7ff5cc088ab0 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.522+0000 7ff5db400700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff5bc06adb0 0x7ff5bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.523+0000 7ff5db400700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff5bc06adb0 0x7ff5bc06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009840 tx=0x7ff5c801c040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:27.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.526+0000 7ff5d3e00700 1 -- 172.21.15.12:0/3756392692 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff5cc053b20 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-29T11:31:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.743+0000 7ff5e3000700 1 -- 172.21.15.12:0/3756392692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5dc0b9460 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.744+0000 7ff5d3e00700 1 -- 172.21.15.12:0/3756392692 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+16244 (secure 0 0 0) 0x7ff5cc017090 con 0x7ff5dc0cf3c0 2024-03-29T11:31:27.738 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:27.738 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":226,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:26.815902+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:27.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 -- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff5bc06adb0 msgr2=0x7ff5bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff5bc06adb0 0x7ff5bc06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009840 tx=0x7ff5c801c040 comp rx=0 tx=0).stop 2024-03-29T11:31:27.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 -- 172.21.15.12:0/3756392692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 msgr2=0x7ff5dc154b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 0x7ff5dc154b50 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00c520 tx=0x7ff5cc00c8e0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 -- 172.21.15.12:0/3756392692 shutdown_connections 2024-03-29T11:31:27.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff5bc06adb0 0x7ff5bc06d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc15db40 0x7ff5dc151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5dc0cf3c0 0x7ff5dc154b50 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 --2- 172.21.15.12:0/3756392692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc15d760 0x7ff5dc155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 -- 172.21.15.12:0/3756392692 >> 172.21.15.12:0/3756392692 conn(0x7ff5dc08aff0 msgr2=0x7ff5dc158650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:27.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 -- 172.21.15.12:0/3756392692 shutdown_connections 2024-03-29T11:31:27.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.747+0000 7ff5e3000700 1 -- 172.21.15.12:0/3756392692 wait complete. 2024-03-29T11:31:27.759 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T11:31:27.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.969+0000 7f780e200700 1 Processor -- start 2024-03-29T11:31:27.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.969+0000 7f780e200700 1 -- start start 2024-03-29T11:31:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.970+0000 7f780e200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f78080b3f80 0x7f78080ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.971+0000 7f780e200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7808161230 con 0x7f78080d3f20 2024-03-29T11:31:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.971+0000 7f780e200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78080d2a80 con 0x7f7808160e50 2024-03-29T11:31:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.971+0000 7f780e200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78080d43b0 con 0x7f78080b3f80 2024-03-29T11:31:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.971+0000 7f7806a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f78080d3f20 0x7f78080cdef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38206/0 (socket says 172.21.15.12:38206) 2024-03-29T11:31:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.971+0000 7f7806a00700 1 -- 172.21.15.12:0/1737281090 learned_addr learned my addr 172.21.15.12:0/1737281090 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.971+0000 7f7807e00700 1 --2- 172.21.15.12:0/1737281090 >> v2:172.21.15.159:3300/0 conn(0x7f78080b3f80 0x7f78080ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.971+0000 7f7806000700 1 -- 172.21.15.12:0/1737281090 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924951794 0 0) 0x7f7808161230 con 0x7f78080d3f20 2024-03-29T11:31:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.971+0000 7f7806000700 1 -- 172.21.15.12:0/1737281090 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77ec0035b0 con 0x7f78080d3f20 2024-03-29T11:31:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.972+0000 7f7807e00700 1 -- 172.21.15.12:0/1737281090 >> v1:172.21.15.159:6789/0 conn(0x7f7808160e50 legacy=0x7f78080b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.972+0000 7f7807e00700 1 -- 172.21.15.12:0/1737281090 >> v1:172.21.15.12:6789/0 conn(0x7f78080d3f20 legacy=0x7f78080cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.972+0000 7f7807e00700 1 -- 172.21.15.12:0/1737281090 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78080d21e0 con 0x7f78080b3f80 2024-03-29T11:31:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.972+0000 7f7807e00700 1 --2- 172.21.15.12:0/1737281090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78080b3f80 0x7f78080ce600 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f7800009d90 tx=0x7f780000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffc8a7799db7afc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.972+0000 7f7806000700 1 -- 172.21.15.12:0/1737281090 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780000e070 con 0x7f78080b3f80 2024-03-29T11:31:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.972+0000 7f7806000700 1 -- 172.21.15.12:0/1737281090 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780000bb20 con 0x7f78080b3f80 2024-03-29T11:31:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.972+0000 7f7806000700 1 -- 172.21.15.12:0/1737281090 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7800012430 con 0x7f78080b3f80 2024-03-29T11:31:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.973+0000 7f780e200700 1 -- 172.21.15.12:0/1737281090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78080b3f80 msgr2=0x7f78080ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.973+0000 7f780e200700 1 --2- 172.21.15.12:0/1737281090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78080b3f80 0x7f78080ce600 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f7800009d90 tx=0x7f780000b330 comp rx=0 tx=0).stop 2024-03-29T11:31:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.973+0000 7f780e200700 1 -- 172.21.15.12:0/1737281090 shutdown_connections 2024-03-29T11:31:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.973+0000 7f780e200700 1 --2- 172.21.15.12:0/1737281090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78080b3f80 0x7f78080ce600 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.973+0000 7f780e200700 1 -- 172.21.15.12:0/1737281090 >> 172.21.15.12:0/1737281090 conn(0x7f780808aff0 msgr2=0x7f78080d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:27.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.973+0000 7f780e200700 1 -- 172.21.15.12:0/1737281090 shutdown_connections 2024-03-29T11:31:27.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.973+0000 7f780e200700 1 -- 172.21.15.12:0/1737281090 wait complete. 2024-03-29T11:31:27.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.975+0000 7f780e200700 1 Processor -- start 2024-03-29T11:31:27.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.975+0000 7f780e200700 1 -- start start 2024-03-29T11:31:27.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f780e200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78080b3f80 0x7f7808152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f780e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7808160e50 0x7f78080cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f780e200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78080b8b60 0x7f78080cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f780e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78080c7c60 con 0x7f7808160e50 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f780e200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78080cbd10 con 0x7f78080b3f80 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f780e200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78080b61d0 con 0x7f78080b8b60 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f7806a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7808160e50 0x7f78080cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f7806a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7808160e50 0x7f78080cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55734/0 (socket says 172.21.15.12:55734) 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f7806a00700 1 -- 172.21.15.12:0/1043618121 learned_addr learned my addr 172.21.15.12:0/1043618121 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f7807400700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78080b3f80 0x7f7808152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f7807e00700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78080b8b60 0x7f78080cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f7806a00700 1 -- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78080b8b60 msgr2=0x7f78080cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f7806a00700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78080b8b60 0x7f78080cc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f7806a00700 1 -- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78080b3f80 msgr2=0x7f7808152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.976+0000 7f7806a00700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78080b3f80 0x7f7808152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.977+0000 7f7806a00700 1 -- 172.21.15.12:0/1043618121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7800009a70 con 0x7f7808160e50 2024-03-29T11:31:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.977+0000 7f7806a00700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7808160e50 0x7f78080cb4e0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f77f400c700 tx=0x7f77f400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.977+0000 7f77ff400700 1 -- 172.21.15.12:0/1043618121 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4005d70 con 0x7f7808160e50 2024-03-29T11:31:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.977+0000 7f77ff400700 1 -- 172.21.15.12:0/1043618121 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f40070d0 con 0x7f7808160e50 2024-03-29T11:31:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.977+0000 7f77ff400700 1 -- 172.21.15.12:0/1043618121 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4007460 con 0x7f7808160e50 2024-03-29T11:31:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.977+0000 7f780e200700 1 -- 172.21.15.12:0/1043618121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78080b64b0 con 0x7f7808160e50 2024-03-29T11:31:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.977+0000 7f780e200700 1 -- 172.21.15.12:0/1043618121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78080cba20 con 0x7f7808160e50 2024-03-29T11:31:27.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.978+0000 7f780e200700 1 -- 172.21.15.12:0/1043618121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78080b69d0 con 0x7f7808160e50 2024-03-29T11:31:27.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.979+0000 7f77ff400700 1 -- 172.21.15.12:0/1043618121 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f77f40075c0 con 0x7f7808160e50 2024-03-29T11:31:27.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.980+0000 7f77ff400700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f77e806adb0 0x7f77e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:27.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.980+0000 7f77ff400700 1 -- 172.21.15.12:0/1043618121 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f77f4088960 con 0x7f7808160e50 2024-03-29T11:31:27.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.981+0000 7f7807400700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f77e806adb0 0x7f77e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:27.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.981+0000 7f7807400700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f77e806adb0 0x7f77e806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f77f8007fd0 tx=0x7f77f8009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:27.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:27.984+0000 7f77ff400700 1 -- 172.21.15.12:0/1043618121 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f78080b69d0 con 0x7f7808160e50 2024-03-29T11:31:28.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.250+0000 7f780e200700 1 -- 172.21.15.12:0/1043618121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f78080b69d0 con 0x7f7808160e50 2024-03-29T11:31:28.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.827+0000 7f77ff400700 1 -- 172.21.15.12:0/1043618121 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v105) v1 ==== 118+0+0 (secure 0 0 0) 0x7f78080b69d0 con 0x7f7808160e50 2024-03-29T11:31:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.829+0000 7f780e200700 1 -- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f77e806adb0 msgr2=0x7f77e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.829+0000 7f780e200700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f77e806adb0 0x7f77e806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f77f8007fd0 tx=0x7f77f8009040 comp rx=0 tx=0).stop 2024-03-29T11:31:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.829+0000 7f780e200700 1 -- 172.21.15.12:0/1043618121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7808160e50 msgr2=0x7f78080cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.829+0000 7f780e200700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7808160e50 0x7f78080cb4e0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f77f400c700 tx=0x7f77f400ca10 comp rx=0 tx=0).stop 2024-03-29T11:31:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.829+0000 7f780e200700 1 -- 172.21.15.12:0/1043618121 shutdown_connections 2024-03-29T11:31:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.829+0000 7f780e200700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f77e806adb0 0x7f77e806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.829+0000 7f780e200700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78080b8b60 0x7f78080cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.829+0000 7f780e200700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78080b3f80 0x7f7808152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.829+0000 7f780e200700 1 --2- 172.21.15.12:0/1043618121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7808160e50 0x7f78080cb4e0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.829+0000 7f780e200700 1 -- 172.21.15.12:0/1043618121 >> 172.21.15.12:0/1043618121 conn(0x7f780808aff0 msgr2=0x7f78080d05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.830+0000 7f780e200700 1 -- 172.21.15.12:0/1043618121 shutdown_connections 2024-03-29T11:31:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:28.830+0000 7f780e200700 1 -- 172.21.15.12:0/1043618121 wait complete. 2024-03-29T11:31:28.843 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T11:31:28.844 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T11:31:29.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.055+0000 7fc382e00700 1 Processor -- start 2024-03-29T11:31:29.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.055+0000 7fc382e00700 1 -- start start 2024-03-29T11:31:29.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.056+0000 7fc382e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15a5c0 0x7fc37c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:29.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.056+0000 7fc382e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c0cb310 con 0x7fc37c0ce840 2024-03-29T11:31:29.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.056+0000 7fc382e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c070f00 con 0x7fc37c15dfe0 2024-03-29T11:31:29.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.056+0000 7fc382e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c0711a0 con 0x7fc37c15a5c0 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.056+0000 7fc380e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc37c0ce840 0x7fc37c0cba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57634/0 (socket says 172.21.15.12:57634) 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.056+0000 7fc380e00700 1 -- 172.21.15.12:0/1405203168 learned_addr learned my addr 172.21.15.12:0/1405203168 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.057+0000 7fc37be00700 1 --2- 172.21.15.12:0/1405203168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15a5c0 0x7fc37c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.057+0000 7fc37aa00700 1 -- 172.21.15.12:0/1405203168 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639918323 0 0) 0x7fc37c0cb310 con 0x7fc37c0ce840 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.057+0000 7fc37aa00700 1 -- 172.21.15.12:0/1405203168 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3600035b0 con 0x7fc37c0ce840 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.057+0000 7fc37aa00700 1 -- 172.21.15.12:0/1405203168 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526436052 0 0) 0x7fc37c070f00 con 0x7fc37c15dfe0 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.057+0000 7fc37aa00700 1 -- 172.21.15.12:0/1405203168 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c0cb310 con 0x7fc37c15dfe0 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.057+0000 7fc37be00700 1 -- 172.21.15.12:0/1405203168 >> v1:172.21.15.159:6789/0 conn(0x7fc37c15dfe0 legacy=0x7fc37c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.057+0000 7fc37be00700 1 -- 172.21.15.12:0/1405203168 >> v1:172.21.15.12:6789/0 conn(0x7fc37c0ce840 legacy=0x7fc37c0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.057+0000 7fc37be00700 1 -- 172.21.15.12:0/1405203168 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c071480 con 0x7fc37c15a5c0 2024-03-29T11:31:29.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.058+0000 7fc37be00700 1 --2- 172.21.15.12:0/1405203168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15a5c0 0x7fc37c0709c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009870 tx=0x7fc36c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=661d90b4a599e2aa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:29.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.058+0000 7fc37aa00700 1 -- 172.21.15.12:0/1405203168 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c011830 con 0x7fc37c15a5c0 2024-03-29T11:31:29.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.058+0000 7fc37aa00700 1 -- 172.21.15.12:0/1405203168 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c011990 con 0x7fc37c15a5c0 2024-03-29T11:31:29.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.058+0000 7fc37aa00700 1 -- 172.21.15.12:0/1405203168 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c011d20 con 0x7fc37c15a5c0 2024-03-29T11:31:29.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.058+0000 7fc382e00700 1 -- 172.21.15.12:0/1405203168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15a5c0 msgr2=0x7fc37c0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:29.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.058+0000 7fc382e00700 1 --2- 172.21.15.12:0/1405203168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15a5c0 0x7fc37c0709c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009870 tx=0x7fc36c009b80 comp rx=0 tx=0).stop 2024-03-29T11:31:29.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.058+0000 7fc382e00700 1 -- 172.21.15.12:0/1405203168 shutdown_connections 2024-03-29T11:31:29.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.058+0000 7fc382e00700 1 --2- 172.21.15.12:0/1405203168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15a5c0 0x7fc37c0709c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:29.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.058+0000 7fc382e00700 1 -- 172.21.15.12:0/1405203168 >> 172.21.15.12:0/1405203168 conn(0x7fc37c0b0d70 msgr2=0x7fc37c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:29.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.058+0000 7fc382e00700 1 -- 172.21.15.12:0/1405203168 shutdown_connections 2024-03-29T11:31:29.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.059+0000 7fc382e00700 1 -- 172.21.15.12:0/1405203168 wait complete. 2024-03-29T11:31:29.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.060+0000 7fc382e00700 1 Processor -- start 2024-03-29T11:31:29.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.060+0000 7fc382e00700 1 -- start start 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.061+0000 7fc382e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc37c0ce840 0x7fc37c1f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.061+0000 7fc382e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc37c15a5c0 0x7fc37c14ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.061+0000 7fc382e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15dfe0 0x7fc37c14f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.061+0000 7fc382e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c1f3180 con 0x7fc37c0ce840 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.061+0000 7fc382e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c14d920 con 0x7fc37c15a5c0 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.061+0000 7fc382e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c14dc00 con 0x7fc37c15dfe0 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc37c0ce840 0x7fc37c1f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc37c0ce840 0x7fc37c1f2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36782/0 (socket says 172.21.15.12:36782) 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37be00700 1 -- 172.21.15.12:0/2516940115 learned_addr learned my addr 172.21.15.12:0/2516940115 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc380e00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15dfe0 0x7fc37c14f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37b400700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc37c15a5c0 0x7fc37c14ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37be00700 1 -- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15dfe0 msgr2=0x7fc37c14f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37be00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15dfe0 0x7fc37c14f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37be00700 1 -- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc37c15a5c0 msgr2=0x7fc37c14ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37be00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc37c15a5c0 0x7fc37c14ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37be00700 1 -- 172.21.15.12:0/2516940115 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0094a0 con 0x7fc37c0ce840 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37b400700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc37c15a5c0 0x7fc37c14ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.062+0000 7fc37be00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc37c0ce840 0x7fc37c1f2c40 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0090b0 tx=0x7fc36c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.063+0000 7fc373e00700 1 -- 172.21.15.12:0/2516940115 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c01a860 con 0x7fc37c0ce840 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.063+0000 7fc373e00700 1 -- 172.21.15.12:0/2516940115 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c01a9c0 con 0x7fc37c0ce840 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.063+0000 7fc382e00700 1 -- 172.21.15.12:0/2516940115 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c14dee0 con 0x7fc37c0ce840 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.063+0000 7fc373e00700 1 -- 172.21.15.12:0/2516940115 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c01ad30 con 0x7fc37c0ce840 2024-03-29T11:31:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.063+0000 7fc382e00700 1 -- 172.21.15.12:0/2516940115 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37c14e4a0 con 0x7fc37c0ce840 2024-03-29T11:31:29.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.064+0000 7fc382e00700 1 -- 172.21.15.12:0/2516940115 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37c067720 con 0x7fc37c0ce840 2024-03-29T11:31:29.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.065+0000 7fc373e00700 1 -- 172.21.15.12:0/2516940115 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc36c022070 con 0x7fc37c0ce840 2024-03-29T11:31:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.065+0000 7fc373e00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35c06adb0 0x7fc35c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.066+0000 7fc373e00700 1 -- 172.21.15.12:0/2516940115 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fc36c08a320 con 0x7fc37c0ce840 2024-03-29T11:31:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.066+0000 7fc37b400700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35c06adb0 0x7fc35c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.067+0000 7fc37b400700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35c06adb0 0x7fc35c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0b0790 tx=0x7fc368009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:29.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.070+0000 7fc373e00700 1 -- 172.21.15.12:0/2516940115 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc36c055560 con 0x7fc37c0ce840 2024-03-29T11:31:29.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:29.337+0000 7fc382e00700 1 -- 172.21.15.12:0/2516940115 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc37c0b1ad0 con 0x7fc37c0ce840 2024-03-29T11:31:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.294+0000 7fc373e00700 1 -- 172.21.15.12:0/2516940115 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v106) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc36c058b80 con 0x7fc37c0ce840 2024-03-29T11:31:30.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.298+0000 7fc382e00700 1 -- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35c06adb0 msgr2=0x7fc35c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.298+0000 7fc382e00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35c06adb0 0x7fc35c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0b0790 tx=0x7fc368009040 comp rx=0 tx=0).stop 2024-03-29T11:31:30.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.298+0000 7fc382e00700 1 -- 172.21.15.12:0/2516940115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc37c0ce840 msgr2=0x7fc37c1f2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.298+0000 7fc382e00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc37c0ce840 0x7fc37c1f2c40 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0090b0 tx=0x7fc36c004750 comp rx=0 tx=0).stop 2024-03-29T11:31:30.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.298+0000 7fc382e00700 1 -- 172.21.15.12:0/2516940115 shutdown_connections 2024-03-29T11:31:30.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.298+0000 7fc382e00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35c06adb0 0x7fc35c06d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.298+0000 7fc382e00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc37c15dfe0 0x7fc37c14f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.298+0000 7fc382e00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc37c0ce840 0x7fc37c1f2c40 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.298+0000 7fc382e00700 1 --2- 172.21.15.12:0/2516940115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc37c15a5c0 0x7fc37c14ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.298+0000 7fc382e00700 1 -- 172.21.15.12:0/2516940115 >> 172.21.15.12:0/2516940115 conn(0x7fc37c0b0d70 msgr2=0x7fc37c0cda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:30.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.299+0000 7fc382e00700 1 -- 172.21.15.12:0/2516940115 shutdown_connections 2024-03-29T11:31:30.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.299+0000 7fc382e00700 1 -- 172.21.15.12:0/2516940115 wait complete. 2024-03-29T11:31:30.316 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T11:31:30.317 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:31:30.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.527+0000 7fd34ac00700 1 Processor -- start 2024-03-29T11:31:30.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.528+0000 7fd34ac00700 1 -- start start 2024-03-29T11:31:30.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.529+0000 7fd34ac00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd3440ce590 0x7fd3440ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:30.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.529+0000 7fd34ac00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3440b6a20 0x7fd3440b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:30.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.529+0000 7fd34ac00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3440ceef0 con 0x7fd344160be0 2024-03-29T11:31:30.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.529+0000 7fd34ac00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3440b65d0 con 0x7fd3440ce590 2024-03-29T11:31:30.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.529+0000 7fd34ac00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440d2a80 con 0x7fd3440b6a20 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.529+0000 7fd343400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd3440ce590 0x7fd3440ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.529+0000 7fd343400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd3440ce590 0x7fd3440ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36788/0 (socket says 172.21.15.12:36788) 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.529+0000 7fd343e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd344160be0 0x7fd3440cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57636/0 (socket says 172.21.15.12:57636) 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.529+0000 7fd343400700 1 -- 172.21.15.12:0/2018403385 learned_addr learned my addr 172.21.15.12:0/2018403385 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.529+0000 7fd348c00700 1 --2- 172.21.15.12:0/2018403385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3440b6a20 0x7fd3440b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.530+0000 7fd343400700 1 -- 172.21.15.12:0/2018403385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3440b6a20 msgr2=0x7fd3440b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.530+0000 7fd343400700 1 --2- 172.21.15.12:0/2018403385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3440b6a20 0x7fd3440b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.530+0000 7fd343400700 1 -- 172.21.15.12:0/2018403385 >> v1:172.21.15.12:6789/0 conn(0x7fd344160be0 legacy=0x7fd3440cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.530+0000 7fd343400700 1 -- 172.21.15.12:0/2018403385 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd344160fc0 con 0x7fd3440ce590 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.530+0000 7fd343400700 1 --2- 172.21.15.12:0/2018403385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 0x7fd3440ce9b0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fd3300098f0 tx=0x7fd330009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f603c5150af85e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.530+0000 7fd342a00700 1 -- 172.21.15.12:0/2018403385 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330004230 con 0x7fd3440ce590 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.530+0000 7fd342a00700 1 -- 172.21.15.12:0/2018403385 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd330004390 con 0x7fd3440ce590 2024-03-29T11:31:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.530+0000 7fd342a00700 1 -- 172.21.15.12:0/2018403385 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3300036a0 con 0x7fd3440ce590 2024-03-29T11:31:30.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.531+0000 7fd34ac00700 1 -- 172.21.15.12:0/2018403385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 msgr2=0x7fd3440ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.531+0000 7fd34ac00700 1 --2- 172.21.15.12:0/2018403385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 0x7fd3440ce9b0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fd3300098f0 tx=0x7fd330009c00 comp rx=0 tx=0).stop 2024-03-29T11:31:30.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.531+0000 7fd34ac00700 1 -- 172.21.15.12:0/2018403385 shutdown_connections 2024-03-29T11:31:30.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.531+0000 7fd34ac00700 1 --2- 172.21.15.12:0/2018403385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3440b6a20 0x7fd3440b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.531+0000 7fd34ac00700 1 --2- 172.21.15.12:0/2018403385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 0x7fd3440ce9b0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.531+0000 7fd34ac00700 1 -- 172.21.15.12:0/2018403385 >> 172.21.15.12:0/2018403385 conn(0x7fd34408aff0 msgr2=0x7fd3440d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:30.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.531+0000 7fd34ac00700 1 -- 172.21.15.12:0/2018403385 shutdown_connections 2024-03-29T11:31:30.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.531+0000 7fd34ac00700 1 -- 172.21.15.12:0/2018403385 wait complete. 2024-03-29T11:31:30.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.533+0000 7fd34ac00700 1 Processor -- start 2024-03-29T11:31:30.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.533+0000 7fd34ac00700 1 -- start start 2024-03-29T11:31:30.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd34ac00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3440b6a20 0x7fd3440a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:30.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd34ac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 0x7fd34406bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:30.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd34ac00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd344160be0 0x7fd34406c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd34ac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440cd2d0 con 0x7fd3440ce590 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd34ac00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440cd5b0 con 0x7fd3440b6a20 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd34ac00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440cd890 con 0x7fd344160be0 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd343400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 0x7fd34406bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd343400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 0x7fd34406bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36794/0 (socket says 172.21.15.12:36794) 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd343400700 1 -- 172.21.15.12:0/519740010 learned_addr learned my addr 172.21.15.12:0/519740010 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd343e00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3440b6a20 0x7fd3440a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd348c00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd344160be0 0x7fd34406c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd343400700 1 -- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd344160be0 msgr2=0x7fd34406c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd343400700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd344160be0 0x7fd34406c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd343400700 1 -- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3440b6a20 msgr2=0x7fd3440a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd343400700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3440b6a20 0x7fd3440a0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.534+0000 7fd343400700 1 -- 172.21.15.12:0/519740010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd330009550 con 0x7fd3440ce590 2024-03-29T11:31:30.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.535+0000 7fd343400700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 0x7fd34406bf10 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fd330009130 tx=0x7fd330004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:30.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.535+0000 7fd33be00700 1 -- 172.21.15.12:0/519740010 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3300045d0 con 0x7fd3440ce590 2024-03-29T11:31:30.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.535+0000 7fd33be00700 1 -- 172.21.15.12:0/519740010 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd330011630 con 0x7fd3440ce590 2024-03-29T11:31:30.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.535+0000 7fd34ac00700 1 -- 172.21.15.12:0/519740010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3440cdb70 con 0x7fd3440ce590 2024-03-29T11:31:30.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.535+0000 7fd33be00700 1 -- 172.21.15.12:0/519740010 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3300119e0 con 0x7fd3440ce590 2024-03-29T11:31:30.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.535+0000 7fd34ac00700 1 -- 172.21.15.12:0/519740010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3440a0f90 con 0x7fd3440ce590 2024-03-29T11:31:30.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.536+0000 7fd34ac00700 1 -- 172.21.15.12:0/519740010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd344067720 con 0x7fd3440ce590 2024-03-29T11:31:30.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.537+0000 7fd33be00700 1 -- 172.21.15.12:0/519740010 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd330022070 con 0x7fd3440ce590 2024-03-29T11:31:30.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.538+0000 7fd33be00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd32406adb0 0x7fd32406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:30.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.538+0000 7fd33be00700 1 -- 172.21.15.12:0/519740010 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fd33008b180 con 0x7fd3440ce590 2024-03-29T11:31:30.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.539+0000 7fd343e00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd32406adb0 0x7fd32406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:30.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.539+0000 7fd343e00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd32406adb0 0x7fd32406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd334007c20 tx=0x7fd334007b70 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:30.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.543+0000 7fd33be00700 1 -- 172.21.15.12:0/519740010 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd330056310 con 0x7fd3440ce590 2024-03-29T11:31:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.802+0000 7fd34ac00700 1 -- 172.21.15.12:0/519740010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3440b5cb0 con 0x7fd3440ce590 2024-03-29T11:31:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.802+0000 7fd33be00700 1 -- 172.21.15.12:0/519740010 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4755 (secure 0 0 0) 0x7fd330059930 con 0x7fd3440ce590 2024-03-29T11:31:30.797 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:30.797 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":106,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":11025,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:26.816217+0000","modified":"2024-03-29T11:31:30.290952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/2722218561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2722218561},{"type":"v1","addr":"172.21.15.12:6833","nonce":2722218561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-03-29T11:31:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 -- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd32406adb0 msgr2=0x7fd32406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd32406adb0 0x7fd32406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd334007c20 tx=0x7fd334007b70 comp rx=0 tx=0).stop 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 -- 172.21.15.12:0/519740010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 msgr2=0x7fd34406bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 0x7fd34406bf10 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fd330009130 tx=0x7fd330004750 comp rx=0 tx=0).stop 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 -- 172.21.15.12:0/519740010 shutdown_connections 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd32406adb0 0x7fd32406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd344160be0 0x7fd34406c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3440b6a20 0x7fd3440a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 --2- 172.21.15.12:0/519740010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3440ce590 0x7fd34406bf10 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 -- 172.21.15.12:0/519740010 >> 172.21.15.12:0/519740010 conn(0x7fd34408aff0 msgr2=0x7fd3440b6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 -- 172.21.15.12:0/519740010 shutdown_connections 2024-03-29T11:31:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:30.805+0000 7fd34ac00700 1 -- 172.21.15.12:0/519740010 wait complete. 2024-03-29T11:31:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 106 2024-03-29T11:31:30.817 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:31.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.021+0000 7f0fe6200700 1 Processor -- start 2024-03-29T11:31:31.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.022+0000 7f0fe6200700 1 -- start start 2024-03-29T11:31:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.023+0000 7f0fe6200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0fe00afbe0 0x7f0fe006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.023+0000 7f0fe6200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0fe00cebd0 0x7f0fe00cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.023+0000 7f0fe6200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe006f850 con 0x7f0fe006a790 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.023+0000 7f0fe6200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe006fad0 con 0x7f0fe00afbe0 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.023+0000 7f0fe6200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe00af970 con 0x7f0fe00cebd0 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.023+0000 7f0fdea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0fe00afbe0 0x7f0fe006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.023+0000 7f0fdf400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0fe006a790 0x7f0fe006ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57652/0 (socket says 172.21.15.12:57652) 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.023+0000 7f0fdf400700 1 -- 172.21.15.12:0/2134365122 learned_addr learned my addr 172.21.15.12:0/2134365122 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.023+0000 7f0fdfe00700 1 --2- 172.21.15.12:0/2134365122 >> v2:172.21.15.159:3300/0 conn(0x7f0fe00cebd0 0x7f0fe00cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.024+0000 7f0fdea00700 1 -- 172.21.15.12:0/2134365122 >> v2:172.21.15.159:3300/0 conn(0x7f0fe00cebd0 msgr2=0x7f0fe00cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.024+0000 7f0fdea00700 1 --2- 172.21.15.12:0/2134365122 >> v2:172.21.15.159:3300/0 conn(0x7f0fe00cebd0 0x7f0fe00cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.024+0000 7f0fdea00700 1 -- 172.21.15.12:0/2134365122 >> v1:172.21.15.12:6789/0 conn(0x7f0fe006a790 legacy=0x7f0fe006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.024+0000 7f0fdea00700 1 -- 172.21.15.12:0/2134365122 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe00af690 con 0x7f0fe00afbe0 2024-03-29T11:31:31.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.024+0000 7f0fdea00700 1 --2- 172.21.15.12:0/2134365122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe00afbe0 0x7f0fe006dce0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0071ad0 tx=0x7f0fcc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bab79761eb9d253 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:31.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.024+0000 7f0fde000700 1 -- 172.21.15.12:0/2134365122 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc011830 con 0x7f0fe00afbe0 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.024+0000 7f0fde000700 1 -- 172.21.15.12:0/2134365122 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fcc011990 con 0x7f0fe00afbe0 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.024+0000 7f0fde000700 1 -- 172.21.15.12:0/2134365122 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc011d20 con 0x7f0fe00afbe0 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.025+0000 7f0fe6200700 1 -- 172.21.15.12:0/2134365122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe00afbe0 msgr2=0x7f0fe006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.025+0000 7f0fe6200700 1 --2- 172.21.15.12:0/2134365122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe00afbe0 0x7f0fe006dce0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0071ad0 tx=0x7f0fcc009b20 comp rx=0 tx=0).stop 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.025+0000 7f0fe6200700 1 -- 172.21.15.12:0/2134365122 shutdown_connections 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.025+0000 7f0fe6200700 1 --2- 172.21.15.12:0/2134365122 >> v2:172.21.15.159:3300/0 conn(0x7f0fe00cebd0 0x7f0fe00cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.025+0000 7f0fe6200700 1 --2- 172.21.15.12:0/2134365122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe00afbe0 0x7f0fe006dce0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.025+0000 7f0fe6200700 1 -- 172.21.15.12:0/2134365122 >> 172.21.15.12:0/2134365122 conn(0x7f0fe00b5ab0 msgr2=0x7f0fe00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.025+0000 7f0fe6200700 1 -- 172.21.15.12:0/2134365122 shutdown_connections 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.025+0000 7f0fe6200700 1 -- 172.21.15.12:0/2134365122 wait complete. 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.027+0000 7f0fe6200700 1 Processor -- start 2024-03-29T11:31:31.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.027+0000 7f0fe6200700 1 -- start start 2024-03-29T11:31:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fe6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe006a790 0x7f0fe014cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fe6200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fe00afbe0 0x7f0fe0155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fe6200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fe00cebd0 0x7f0fe01579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fe6200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe014d180 con 0x7f0fe006a790 2024-03-29T11:31:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fe6200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe01571a0 con 0x7f0fe00cebd0 2024-03-29T11:31:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fe6200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe0156100 con 0x7f0fe00afbe0 2024-03-29T11:31:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe006a790 0x7f0fe014cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe006a790 0x7f0fe014cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36812/0 (socket says 172.21.15.12:36812) 2024-03-29T11:31:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdf400700 1 -- 172.21.15.12:0/1031273407 learned_addr learned my addr 172.21.15.12:0/1031273407 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdea00700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fe00afbe0 0x7f0fe0155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdfe00700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fe00cebd0 0x7f0fe01579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdf400700 1 -- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fe00afbe0 msgr2=0x7f0fe0155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdf400700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fe00afbe0 0x7f0fe0155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdf400700 1 -- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fe00cebd0 msgr2=0x7f0fe01579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdf400700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fe00cebd0 0x7f0fe01579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdf400700 1 -- 172.21.15.12:0/1031273407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc0094a0 con 0x7f0fe006a790 2024-03-29T11:31:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdea00700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fe00afbe0 0x7f0fe0155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.028+0000 7f0fdfe00700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fe00cebd0 0x7f0fe01579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.029+0000 7f0fdf400700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe006a790 0x7f0fe014cc40 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000c520 tx=0x7f0fd000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.029+0000 7f0fd7400700 1 -- 172.21.15.12:0/1031273407 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0005d70 con 0x7f0fe006a790 2024-03-29T11:31:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.029+0000 7f0fd7400700 1 -- 172.21.15.12:0/1031273407 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd00070d0 con 0x7f0fe006a790 2024-03-29T11:31:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.029+0000 7f0fd7400700 1 -- 172.21.15.12:0/1031273407 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0007440 con 0x7f0fe006a790 2024-03-29T11:31:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.029+0000 7f0fe6200700 1 -- 172.21.15.12:0/1031273407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe00cf6c0 con 0x7f0fe006a790 2024-03-29T11:31:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.029+0000 7f0fe6200700 1 -- 172.21.15.12:0/1031273407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe01566c0 con 0x7f0fe006a790 2024-03-29T11:31:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.030+0000 7f0fe6200700 1 -- 172.21.15.12:0/1031273407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe0067720 con 0x7f0fe006a790 2024-03-29T11:31:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.031+0000 7f0fd7400700 1 -- 172.21.15.12:0/1031273407 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0fd00075a0 con 0x7f0fe006a790 2024-03-29T11:31:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.032+0000 7f0fd7400700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0fc006adb0 0x7f0fc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.032+0000 7f0fd7400700 1 -- 172.21.15.12:0/1031273407 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f0fd0088910 con 0x7f0fe006a790 2024-03-29T11:31:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.032+0000 7f0fdea00700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0fc006adb0 0x7f0fc006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.033+0000 7f0fdea00700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0fc006adb0 0x7f0fc006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0090b0 tx=0x7f0fcc008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:31.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.037+0000 7f0fd7400700 1 -- 172.21.15.12:0/1031273407 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0fd0053a40 con 0x7f0fe006a790 2024-03-29T11:31:31.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.254+0000 7f0fe6200700 1 -- 172.21.15.12:0/1031273407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fe0155370 con 0x7f0fe006a790 2024-03-29T11:31:31.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.255+0000 7f0fd7400700 1 -- 172.21.15.12:0/1031273407 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v227) v1 ==== 74+0+16231 (secure 0 0 0) 0x7f0fd0017090 con 0x7f0fe006a790 2024-03-29T11:31:31.250 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:31.250 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":227,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:27.818673+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:31.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 -- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0fc006adb0 msgr2=0x7f0fc006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0fc006adb0 0x7f0fc006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0090b0 tx=0x7f0fcc008030 comp rx=0 tx=0).stop 2024-03-29T11:31:31.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 -- 172.21.15.12:0/1031273407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe006a790 msgr2=0x7f0fe014cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe006a790 0x7f0fe014cc40 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000c520 tx=0x7f0fd000c8e0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 -- 172.21.15.12:0/1031273407 shutdown_connections 2024-03-29T11:31:31.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0fc006adb0 0x7f0fc006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0fe00cebd0 0x7f0fe01579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fe006a790 0x7f0fe014cc40 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 --2- 172.21.15.12:0/1031273407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0fe00afbe0 0x7f0fe0155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 -- 172.21.15.12:0/1031273407 >> 172.21.15.12:0/1031273407 conn(0x7f0fe00b5ab0 msgr2=0x7f0fe015e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:31.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 -- 172.21.15.12:0/1031273407 shutdown_connections 2024-03-29T11:31:31.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.258+0000 7f0fe6200700 1 -- 172.21.15.12:0/1031273407 wait complete. 2024-03-29T11:31:31.271 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T11:31:31.271 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:31:31.458 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-29T11:31:31.464+0000 7f6d0f800700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:31:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.476+0000 7f618ae00700 1 Processor -- start 2024-03-29T11:31:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.476+0000 7f618ae00700 1 -- start start 2024-03-29T11:31:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.477+0000 7f618ae00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6184160f10 0x7f61840d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.477+0000 7f618ae00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61840cf230 0x7f61840d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.477+0000 7f618ae00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61840ceda0 con 0x7f6184161590 2024-03-29T11:31:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.477+0000 7f618ae00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61840d23d0 con 0x7f6184160f10 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.477+0000 7f618ae00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61841612f0 con 0x7f61840cf230 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.478+0000 7f6183e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6184160f10 0x7f61840d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.478+0000 7f6183e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6184160f10 0x7f61840d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36818/0 (socket says 172.21.15.12:36818) 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.478+0000 7f6183e00700 1 -- 172.21.15.12:0/617561106 learned_addr learned my addr 172.21.15.12:0/617561106 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.478+0000 7f6183e00700 1 -- 172.21.15.12:0/617561106 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61840cf230 msgr2=0x7f61840d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.478+0000 7f6183400700 1 --2- 172.21.15.12:0/617561106 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61840cf230 0x7f61840d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.478+0000 7f6183e00700 1 --2- 172.21.15.12:0/617561106 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61840cf230 0x7f61840d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.478+0000 7f6183e00700 1 -- 172.21.15.12:0/617561106 >> v1:172.21.15.159:6789/0 conn(0x7f6184161590 legacy=0x7f61840b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.478+0000 7f6183e00700 1 -- 172.21.15.12:0/617561106 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61840ce470 con 0x7f6184160f10 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.478+0000 7f6183400700 1 --2- 172.21.15.12:0/617561106 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61840cf230 0x7f61840d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.479+0000 7f6183e00700 1 --2- 172.21.15.12:0/617561106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 0x7f61840d18b0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f61740098f0 tx=0x7f6174009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a769030524ade1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.479+0000 7f6182a00700 1 -- 172.21.15.12:0/617561106 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174004230 con 0x7f6184160f10 2024-03-29T11:31:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.479+0000 7f6182a00700 1 -- 172.21.15.12:0/617561106 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6174004390 con 0x7f6184160f10 2024-03-29T11:31:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.479+0000 7f6182a00700 1 -- 172.21.15.12:0/617561106 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61740036a0 con 0x7f6184160f10 2024-03-29T11:31:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.479+0000 7f618ae00700 1 -- 172.21.15.12:0/617561106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 msgr2=0x7f61840d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.479+0000 7f618ae00700 1 --2- 172.21.15.12:0/617561106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 0x7f61840d18b0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f61740098f0 tx=0x7f6174009c00 comp rx=0 tx=0).stop 2024-03-29T11:31:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.479+0000 7f618ae00700 1 -- 172.21.15.12:0/617561106 shutdown_connections 2024-03-29T11:31:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.479+0000 7f618ae00700 1 --2- 172.21.15.12:0/617561106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 0x7f61840d18b0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.479+0000 7f618ae00700 1 --2- 172.21.15.12:0/617561106 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61840cf230 0x7f61840d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.479+0000 7f618ae00700 1 -- 172.21.15.12:0/617561106 >> 172.21.15.12:0/617561106 conn(0x7f618408aff0 msgr2=0x7f61840d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.480+0000 7f618ae00700 1 -- 172.21.15.12:0/617561106 shutdown_connections 2024-03-29T11:31:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.480+0000 7f618ae00700 1 -- 172.21.15.12:0/617561106 wait complete. 2024-03-29T11:31:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.481+0000 7f618ae00700 1 Processor -- start 2024-03-29T11:31:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.481+0000 7f618ae00700 1 -- start start 2024-03-29T11:31:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.482+0000 7f618ae00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61840cf230 0x7f61841585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.482+0000 7f618ae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 0x7f61841f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.482+0000 7f618ae00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6184161590 0x7f61841f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.482+0000 7f618ae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61841f31e0 con 0x7f6184160f10 2024-03-29T11:31:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.482+0000 7f618ae00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6184157ba0 con 0x7f61840cf230 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.482+0000 7f618ae00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61840b8c20 con 0x7f6184161590 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6183400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 0x7f61841f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6183400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 0x7f61841f6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36820/0 (socket says 172.21.15.12:36820) 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6183400700 1 -- 172.21.15.12:0/4215597219 learned_addr learned my addr 172.21.15.12:0/4215597219 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6183e00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61840cf230 0x7f61841585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6188e00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6184161590 0x7f61841f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6183400700 1 -- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6184161590 msgr2=0x7f61841f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6183400700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6184161590 0x7f61841f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6183400700 1 -- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61840cf230 msgr2=0x7f61841585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6183400700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61840cf230 0x7f61841585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6183400700 1 -- 172.21.15.12:0/4215597219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6174009550 con 0x7f6184160f10 2024-03-29T11:31:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.483+0000 7f6183400700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 0x7f61841f6890 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f617000c7f0 tx=0x7f617000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.484+0000 7f617be00700 1 -- 172.21.15.12:0/4215597219 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170011800 con 0x7f6184160f10 2024-03-29T11:31:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.484+0000 7f617be00700 1 -- 172.21.15.12:0/4215597219 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6170011960 con 0x7f6184160f10 2024-03-29T11:31:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.484+0000 7f617be00700 1 -- 172.21.15.12:0/4215597219 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170011cf0 con 0x7f6184160f10 2024-03-29T11:31:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.484+0000 7f618ae00700 1 -- 172.21.15.12:0/4215597219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61840b8f00 con 0x7f6184160f10 2024-03-29T11:31:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.484+0000 7f618ae00700 1 -- 172.21.15.12:0/4215597219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61841580c0 con 0x7f6184160f10 2024-03-29T11:31:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.485+0000 7f618ae00700 1 -- 172.21.15.12:0/4215597219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61841582e0 con 0x7f6184160f10 2024-03-29T11:31:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.486+0000 7f617be00700 1 -- 172.21.15.12:0/4215597219 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6170018360 con 0x7f6184160f10 2024-03-29T11:31:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.487+0000 7f617be00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f616406adb0 0x7f616406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.487+0000 7f617be00700 1 -- 172.21.15.12:0/4215597219 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f61700893e0 con 0x7f6184160f10 2024-03-29T11:31:31.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.488+0000 7f6183e00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f616406adb0 0x7f616406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.489+0000 7f6183e00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f616406adb0 0x7f616406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6174009130 tx=0x7f6174008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:31.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.492+0000 7f617be00700 1 -- 172.21.15.12:0/4215597219 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6170054340 con 0x7f6184160f10 2024-03-29T11:31:31.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.699+0000 7f618ae00700 1 -- 172.21.15.12:0/4215597219 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61840cf6b0 con 0x7f616406adb0 2024-03-29T11:31:31.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.710+0000 7f617be00700 1 -- 172.21.15.12:0/4215597219 <== mgr.10459 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289840 (secure 0 0 0) 0x7f6174003720 con 0x7f616406adb0 2024-03-29T11:31:31.709 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:31.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 -- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f616406adb0 msgr2=0x7f616406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f616406adb0 0x7f616406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6174009130 tx=0x7f6174008030 comp rx=0 tx=0).stop 2024-03-29T11:31:31.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 -- 172.21.15.12:0/4215597219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 msgr2=0x7f61841f6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 0x7f61841f6890 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f617000c7f0 tx=0x7f617000cb00 comp rx=0 tx=0).stop 2024-03-29T11:31:31.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 -- 172.21.15.12:0/4215597219 shutdown_connections 2024-03-29T11:31:31.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f616406adb0 0x7f616406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6184161590 0x7f61841f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61840cf230 0x7f61841585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 --2- 172.21.15.12:0/4215597219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6184160f10 0x7f61841f6890 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 -- 172.21.15.12:0/4215597219 >> 172.21.15.12:0/4215597219 conn(0x7f618408aff0 msgr2=0x7f6184160070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:31.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 -- 172.21.15.12:0/4215597219 shutdown_connections 2024-03-29T11:31:31.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.715+0000 7f618ae00700 1 -- 172.21.15.12:0/4215597219 wait complete. 2024-03-29T11:31:31.713 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:31:31.728 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-29T11:31:31.167937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":655,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":433732,"kb_used_data":23236,"kb_used_omap":12,"kb_used_meta":410355,"kb_avail":754540988,"statfs":{"total":773094113280,"available":772649971712,"internally_reserved":0,"allocated":23793664,"data_stored":12185923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":420204089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.873721"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854714+0000","last_change":"2024-03-29T11:31:26.850587+0000","last_active":"2024-03-29T11:31:26.854714+0000","last_peered":"2024-03-29T11:31:26.854714+0000","last_clean":"2024-03-29T11:31:26.854714+0000","last_became_active":"2024-03-29T11:31:26.850150+0000","last_became_peered":"2024-03-29T11:31:26.850150+0000","last_unstale":"2024-03-29T11:31:26.854714+0000","last_undegraded":"2024-03-29T11:31:26.854714+0000","last_fullsized":"2024-03-29T11:31:26.854714+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:49.204368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854837+0000","last_change":"2024-03-29T11:31:26.848512+0000","last_active":"2024-03-29T11:31:26.854837+0000","last_peered":"2024-03-29T11:31:26.854837+0000","last_clean":"2024-03-29T11:31:26.854837+0000","last_became_active":"2024-03-29T11:31:26.846361+0000","last_became_peered":"2024-03-29T11:31:26.846361+0000","last_unstale":"2024-03-29T11:31:26.854837+0000","last_undegraded":"2024-03-29T11:31:26.854837+0000","last_fullsized":"2024-03-29T11:31:26.854837+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:49:25.037675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853062+0000","last_change":"2024-03-29T11:31:26.851998+0000","last_active":"2024-03-29T11:31:26.853062+0000","last_peered":"2024-03-29T11:31:26.853062+0000","last_clean":"2024-03-29T11:31:26.853062+0000","last_became_active":"2024-03-29T11:31:26.850781+0000","last_became_peered":"2024-03-29T11:31:26.850781+0000","last_unstale":"2024-03-29T11:31:26.853062+0000","last_undegraded":"2024-03-29T11:31:26.853062+0000","last_fullsized":"2024-03-29T11:31:26.853062+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:57.876055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854117+0000","last_change":"2024-03-29T11:31:26.852752+0000","last_active":"2024-03-29T11:31:26.854117+0000","last_peered":"2024-03-29T11:31:26.854117+0000","last_clean":"2024-03-29T11:31:26.854117+0000","last_became_active":"2024-03-29T11:31:26.852491+0000","last_became_peered":"2024-03-29T11:31:26.852491+0000","last_unstale":"2024-03-29T11:31:26.854117+0000","last_undegraded":"2024-03-29T11:31:26.854117+0000","last_fullsized":"2024-03-29T11:31:26.854117+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:32.613551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845568+0000","last_change":"2024-03-29T11:31:26.841559+0000","last_active":"2024-03-29T11:31:26.845568+0000","last_peered":"2024-03-29T11:31:26.845568+0000","last_clean":"2024-03-29T11:31:26.845568+0000","last_became_active":"2024-03-29T11:31:26.841323+0000","last_became_peered":"2024-03-29T11:31:26.841323+0000","last_unstale":"2024-03-29T11:31:26.845568+0000","last_undegraded":"2024-03-29T11:31:26.845568+0000","last_fullsized":"2024-03-29T11:31:26.845568+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:40:17.980944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.840870+0000","last_change":"2024-03-29T11:31:26.836146+0000","last_active":"2024-03-29T11:31:26.840870+0000","last_peered":"2024-03-29T11:31:26.840870+0000","last_clean":"2024-03-29T11:31:26.840870+0000","last_became_active":"2024-03-29T11:31:26.835185+0000","last_became_peered":"2024-03-29T11:31:26.835185+0000","last_unstale":"2024-03-29T11:31:26.840870+0000","last_undegraded":"2024-03-29T11:31:26.840870+0000","last_fullsized":"2024-03-29T11:31:26.840870+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:15:49.012984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.846028+0000","last_change":"2024-03-29T11:31:26.836892+0000","last_active":"2024-03-29T11:31:26.846028+0000","last_peered":"2024-03-29T11:31:26.846028+0000","last_clean":"2024-03-29T11:31:26.846028+0000","last_became_active":"2024-03-29T11:31:26.832344+0000","last_became_peered":"2024-03-29T11:31:26.832344+0000","last_unstale":"2024-03-29T11:31:26.846028+0000","last_undegraded":"2024-03-29T11:31:26.846028+0000","last_fullsized":"2024-03-29T11:31:26.846028+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:27:29.490312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.840636+0000","last_change":"2024-03-29T11:31:26.833794+0000","last_active":"2024-03-29T11:31:26.840636+0000","last_peered":"2024-03-29T11:31:26.840636+0000","last_clean":"2024-03-29T11:31:26.840636+0000","last_became_active":"2024-03-29T11:31:26.833440+0000","last_became_peered":"2024-03-29T11:31:26.833440+0000","last_unstale":"2024-03-29T11:31:26.840636+0000","last_undegraded":"2024-03-29T11:31:26.840636+0000","last_fullsized":"2024-03-29T11:31:26.840636+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:07.336827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854459+0000","last_change":"2024-03-29T11:31:26.851850+0000","last_active":"2024-03-29T11:31:26.854459+0000","last_peered":"2024-03-29T11:31:26.854459+0000","last_clean":"2024-03-29T11:31:26.854459+0000","last_became_active":"2024-03-29T11:31:26.848888+0000","last_became_peered":"2024-03-29T11:31:26.848888+0000","last_unstale":"2024-03-29T11:31:26.854459+0000","last_undegraded":"2024-03-29T11:31:26.854459+0000","last_fullsized":"2024-03-29T11:31:26.854459+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:26:36.183185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855076+0000","last_change":"2024-03-29T11:31:26.853942+0000","last_active":"2024-03-29T11:31:26.855076+0000","last_peered":"2024-03-29T11:31:26.855076+0000","last_clean":"2024-03-29T11:31:26.855076+0000","last_became_active":"2024-03-29T11:31:26.853648+0000","last_became_peered":"2024-03-29T11:31:26.853648+0000","last_unstale":"2024-03-29T11:31:26.855076+0000","last_undegraded":"2024-03-29T11:31:26.855076+0000","last_fullsized":"2024-03-29T11:31:26.855076+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:24.230359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.848145+0000","last_change":"2024-03-29T11:31:26.841262+0000","last_active":"2024-03-29T11:31:26.848145+0000","last_peered":"2024-03-29T11:31:26.848145+0000","last_clean":"2024-03-29T11:31:26.848145+0000","last_became_active":"2024-03-29T11:31:26.841004+0000","last_became_peered":"2024-03-29T11:31:26.841004+0000","last_unstale":"2024-03-29T11:31:26.848145+0000","last_undegraded":"2024-03-29T11:31:26.848145+0000","last_fullsized":"2024-03-29T11:31:26.848145+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:54.107881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853190+0000","last_change":"2024-03-29T11:31:26.848052+0000","last_active":"2024-03-29T11:31:26.853190+0000","last_peered":"2024-03-29T11:31:26.853190+0000","last_clean":"2024-03-29T11:31:26.853190+0000","last_became_active":"2024-03-29T11:31:26.847789+0000","last_became_peered":"2024-03-29T11:31:26.847789+0000","last_unstale":"2024-03-29T11:31:26.853190+0000","last_undegraded":"2024-03-29T11:31:26.853190+0000","last_fullsized":"2024-03-29T11:31:26.853190+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:23:23.236057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853771+0000","last_change":"2024-03-29T11:31:26.848377+0000","last_active":"2024-03-29T11:31:26.853771+0000","last_peered":"2024-03-29T11:31:26.853771+0000","last_clean":"2024-03-29T11:31:26.853771+0000","last_became_active":"2024-03-29T11:31:26.847564+0000","last_became_peered":"2024-03-29T11:31:26.847564+0000","last_unstale":"2024-03-29T11:31:26.853771+0000","last_undegraded":"2024-03-29T11:31:26.853771+0000","last_fullsized":"2024-03-29T11:31:26.853771+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:40:17.980944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853815+0000","last_change":"2024-03-29T11:31:26.848452+0000","last_active":"2024-03-29T11:31:26.853815+0000","last_peered":"2024-03-29T11:31:26.853815+0000","last_clean":"2024-03-29T11:31:26.853815+0000","last_became_active":"2024-03-29T11:31:26.846605+0000","last_became_peered":"2024-03-29T11:31:26.846605+0000","last_unstale":"2024-03-29T11:31:26.853815+0000","last_undegraded":"2024-03-29T11:31:26.853815+0000","last_fullsized":"2024-03-29T11:31:26.853815+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:57.163908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852282+0000","last_change":"2024-03-29T11:31:26.843972+0000","last_active":"2024-03-29T11:31:26.852282+0000","last_peered":"2024-03-29T11:31:26.852282+0000","last_clean":"2024-03-29T11:31:26.852282+0000","last_became_active":"2024-03-29T11:31:26.843713+0000","last_became_peered":"2024-03-29T11:31:26.843713+0000","last_unstale":"2024-03-29T11:31:26.852282+0000","last_undegraded":"2024-03-29T11:31:26.852282+0000","last_fullsized":"2024-03-29T11:31:26.852282+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847412+0000","last_change":"2024-03-29T11:31:26.843040+0000","last_active":"2024-03-29T11:31:26.847412+0000","last_peered":"2024-03-29T11:31:26.847412+0000","last_clean":"2024-03-29T11:31:26.847412+0000","last_became_active":"2024-03-29T11:31:26.841805+0000","last_became_peered":"2024-03-29T11:31:26.841805+0000","last_unstale":"2024-03-29T11:31:26.847412+0000","last_undegraded":"2024-03-29T11:31:26.847412+0000","last_fullsized":"2024-03-29T11:31:26.847412+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:05.909043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854695+0000","last_change":"2024-03-29T11:31:26.850388+0000","last_active":"2024-03-29T11:31:26.854695+0000","last_peered":"2024-03-29T11:31:26.854695+0000","last_clean":"2024-03-29T11:31:26.854695+0000","last_became_active":"2024-03-29T11:31:26.849826+0000","last_became_peered":"2024-03-29T11:31:26.849826+0000","last_unstale":"2024-03-29T11:31:26.854695+0000","last_undegraded":"2024-03-29T11:31:26.854695+0000","last_fullsized":"2024-03-29T11:31:26.854695+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:55:45.796741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845675+0000","last_change":"2024-03-29T11:31:26.840048+0000","last_active":"2024-03-29T11:31:26.845675+0000","last_peered":"2024-03-29T11:31:26.845675+0000","last_clean":"2024-03-29T11:31:26.845675+0000","last_became_active":"2024-03-29T11:31:26.838725+0000","last_became_peered":"2024-03-29T11:31:26.838725+0000","last_unstale":"2024-03-29T11:31:26.845675+0000","last_undegraded":"2024-03-29T11:31:26.845675+0000","last_fullsized":"2024-03-29T11:31:26.845675+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:09.172222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855793+0000","last_change":"2024-03-29T11:31:26.854890+0000","last_active":"2024-03-29T11:31:26.855793+0000","last_peered":"2024-03-29T11:31:26.855793+0000","last_clean":"2024-03-29T11:31:26.855793+0000","last_became_active":"2024-03-29T11:31:26.854625+0000","last_became_peered":"2024-03-29T11:31:26.854625+0000","last_unstale":"2024-03-29T11:31:26.855793+0000","last_undegraded":"2024-03-29T11:31:26.855793+0000","last_fullsized":"2024-03-29T11:31:26.855793+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.831388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845779+0000","last_change":"2024-03-29T11:31:26.843448+0000","last_active":"2024-03-29T11:31:26.845779+0000","last_peered":"2024-03-29T11:31:26.845779+0000","last_clean":"2024-03-29T11:31:26.845779+0000","last_became_active":"2024-03-29T11:31:26.843183+0000","last_became_peered":"2024-03-29T11:31:26.843183+0000","last_unstale":"2024-03-29T11:31:26.845779+0000","last_undegraded":"2024-03-29T11:31:26.845779+0000","last_fullsized":"2024-03-29T11:31:26.845779+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:23:23.236057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.857836+0000","last_change":"2024-03-29T11:31:26.856449+0000","last_active":"2024-03-29T11:31:26.857836+0000","last_peered":"2024-03-29T11:31:26.857836+0000","last_clean":"2024-03-29T11:31:26.857836+0000","last_became_active":"2024-03-29T11:31:26.856166+0000","last_became_peered":"2024-03-29T11:31:26.856166+0000","last_unstale":"2024-03-29T11:31:26.857836+0000","last_undegraded":"2024-03-29T11:31:26.857836+0000","last_fullsized":"2024-03-29T11:31:26.857836+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:27.122635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.841326+0000","last_change":"2024-03-29T11:31:26.834482+0000","last_active":"2024-03-29T11:31:26.841326+0000","last_peered":"2024-03-29T11:31:26.841326+0000","last_clean":"2024-03-29T11:31:26.841326+0000","last_became_active":"2024-03-29T11:31:26.833118+0000","last_became_peered":"2024-03-29T11:31:26.833118+0000","last_unstale":"2024-03-29T11:31:26.841326+0000","last_undegraded":"2024-03-29T11:31:26.841326+0000","last_fullsized":"2024-03-29T11:31:26.841326+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:42.629666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845477+0000","last_change":"2024-03-29T11:31:26.842715+0000","last_active":"2024-03-29T11:31:26.845477+0000","last_peered":"2024-03-29T11:31:26.845477+0000","last_clean":"2024-03-29T11:31:26.845477+0000","last_became_active":"2024-03-29T11:31:26.842354+0000","last_became_peered":"2024-03-29T11:31:26.842354+0000","last_unstale":"2024-03-29T11:31:26.845477+0000","last_undegraded":"2024-03-29T11:31:26.845477+0000","last_fullsized":"2024-03-29T11:31:26.845477+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:58.653895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.857509+0000","last_change":"2024-03-29T11:31:26.855506+0000","last_active":"2024-03-29T11:31:26.857509+0000","last_peered":"2024-03-29T11:31:26.857509+0000","last_clean":"2024-03-29T11:31:26.857509+0000","last_became_active":"2024-03-29T11:31:26.852090+0000","last_became_peered":"2024-03-29T11:31:26.852090+0000","last_unstale":"2024-03-29T11:31:26.857509+0000","last_undegraded":"2024-03-29T11:31:26.857509+0000","last_fullsized":"2024-03-29T11:31:26.857509+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:09.172222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854104+0000","last_change":"2024-03-29T11:31:26.847195+0000","last_active":"2024-03-29T11:31:26.854104+0000","last_peered":"2024-03-29T11:31:26.854104+0000","last_clean":"2024-03-29T11:31:26.854104+0000","last_became_active":"2024-03-29T11:31:26.846876+0000","last_became_peered":"2024-03-29T11:31:26.846876+0000","last_unstale":"2024-03-29T11:31:26.854104+0000","last_undegraded":"2024-03-29T11:31:26.854104+0000","last_fullsized":"2024-03-29T11:31:26.854104+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:41.350201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852325+0000","last_change":"2024-03-29T11:31:26.843275+0000","last_active":"2024-03-29T11:31:26.852325+0000","last_peered":"2024-03-29T11:31:26.852325+0000","last_clean":"2024-03-29T11:31:26.852325+0000","last_became_active":"2024-03-29T11:31:26.842588+0000","last_became_peered":"2024-03-29T11:31:26.842588+0000","last_unstale":"2024-03-29T11:31:26.852325+0000","last_undegraded":"2024-03-29T11:31:26.852325+0000","last_fullsized":"2024-03-29T11:31:26.852325+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:57.876055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854434+0000","last_change":"2024-03-29T11:31:26.852411+0000","last_active":"2024-03-29T11:31:26.854434+0000","last_peered":"2024-03-29T11:31:26.854434+0000","last_clean":"2024-03-29T11:31:26.854434+0000","last_became_active":"2024-03-29T11:31:26.852100+0000","last_became_peered":"2024-03-29T11:31:26.852100+0000","last_unstale":"2024-03-29T11:31:26.854434+0000","last_undegraded":"2024-03-29T11:31:26.854434+0000","last_fullsized":"2024-03-29T11:31:26.854434+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:15:49.012984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847430+0000","last_change":"2024-03-29T11:31:26.838393+0000","last_active":"2024-03-29T11:31:26.847430+0000","last_peered":"2024-03-29T11:31:26.847430+0000","last_clean":"2024-03-29T11:31:26.847430+0000","last_became_active":"2024-03-29T11:31:26.838003+0000","last_became_peered":"2024-03-29T11:31:26.838003+0000","last_unstale":"2024-03-29T11:31:26.847430+0000","last_undegraded":"2024-03-29T11:31:26.847430+0000","last_fullsized":"2024-03-29T11:31:26.847430+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:47.195933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853791+0000","last_change":"2024-03-29T11:31:26.848961+0000","last_active":"2024-03-29T11:31:26.853791+0000","last_peered":"2024-03-29T11:31:26.853791+0000","last_clean":"2024-03-29T11:31:26.853791+0000","last_became_active":"2024-03-29T11:31:26.848637+0000","last_became_peered":"2024-03-29T11:31:26.848637+0000","last_unstale":"2024-03-29T11:31:26.853791+0000","last_undegraded":"2024-03-29T11:31:26.853791+0000","last_fullsized":"2024-03-29T11:31:26.853791+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:31:51.171001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.846678+0000","last_change":"2024-03-29T11:31:26.842077+0000","last_active":"2024-03-29T11:31:26.846678+0000","last_peered":"2024-03-29T11:31:26.846678+0000","last_clean":"2024-03-29T11:31:26.846678+0000","last_became_active":"2024-03-29T11:31:26.841818+0000","last_became_peered":"2024-03-29T11:31:26.841818+0000","last_unstale":"2024-03-29T11:31:26.846678+0000","last_undegraded":"2024-03-29T11:31:26.846678+0000","last_fullsized":"2024-03-29T11:31:26.846678+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:42:57.193431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.851202+0000","last_change":"2024-03-29T11:31:26.847737+0000","last_active":"2024-03-29T11:31:26.851202+0000","last_peered":"2024-03-29T11:31:26.851202+0000","last_clean":"2024-03-29T11:31:26.851202+0000","last_became_active":"2024-03-29T11:31:26.847473+0000","last_became_peered":"2024-03-29T11:31:26.847473+0000","last_unstale":"2024-03-29T11:31:26.851202+0000","last_undegraded":"2024-03-29T11:31:26.851202+0000","last_fullsized":"2024-03-29T11:31:26.851202+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:36:05.541971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847682+0000","last_change":"2024-03-29T11:31:26.839138+0000","last_active":"2024-03-29T11:31:26.847682+0000","last_peered":"2024-03-29T11:31:26.847682+0000","last_clean":"2024-03-29T11:31:26.847682+0000","last_became_active":"2024-03-29T11:31:26.838904+0000","last_became_peered":"2024-03-29T11:31:26.838904+0000","last_unstale":"2024-03-29T11:31:26.847682+0000","last_undegraded":"2024-03-29T11:31:26.847682+0000","last_fullsized":"2024-03-29T11:31:26.847682+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:54.107881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.848635+0000","last_change":"2024-03-29T11:31:26.844474+0000","last_active":"2024-03-29T11:31:26.848635+0000","last_peered":"2024-03-29T11:31:26.848635+0000","last_clean":"2024-03-29T11:31:26.848635+0000","last_became_active":"2024-03-29T11:31:26.844195+0000","last_became_peered":"2024-03-29T11:31:26.844195+0000","last_unstale":"2024-03-29T11:31:26.848635+0000","last_undegraded":"2024-03-29T11:31:26.848635+0000","last_fullsized":"2024-03-29T11:31:26.848635+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:38:37.668282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"226'4","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855147+0000","last_change":"2024-03-29T11:31:26.315269+0000","last_active":"2024-03-29T11:31:26.855147+0000","last_peered":"2024-03-29T11:31:26.855147+0000","last_clean":"2024-03-29T11:31:26.855147+0000","last_became_active":"2024-03-29T11:31:26.314956+0000","last_became_peered":"2024-03-29T11:31:26.314956+0000","last_unstale":"2024-03-29T11:31:26.855147+0000","last_undegraded":"2024-03-29T11:31:26.855147+0000","last_fullsized":"2024-03-29T11:31:26.855147+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:18.302941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847446+0000","last_change":"2024-03-29T11:31:26.843531+0000","last_active":"2024-03-29T11:31:26.847446+0000","last_peered":"2024-03-29T11:31:26.847446+0000","last_clean":"2024-03-29T11:31:26.847446+0000","last_became_active":"2024-03-29T11:31:26.843164+0000","last_became_peered":"2024-03-29T11:31:26.843164+0000","last_unstale":"2024-03-29T11:31:26.847446+0000","last_undegraded":"2024-03-29T11:31:26.847446+0000","last_fullsized":"2024-03-29T11:31:26.847446+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:29:23.515574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855908+0000","last_change":"2024-03-29T11:31:26.320463+0000","last_active":"2024-03-29T11:31:26.855908+0000","last_peered":"2024-03-29T11:31:26.855908+0000","last_clean":"2024-03-29T11:31:26.855908+0000","last_became_active":"2024-03-29T11:31:26.319713+0000","last_became_peered":"2024-03-29T11:31:26.319713+0000","last_unstale":"2024-03-29T11:31:26.855908+0000","last_undegraded":"2024-03-29T11:31:26.855908+0000","last_fullsized":"2024-03-29T11:31:26.855908+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:32:47.003355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854970+0000","last_change":"2024-03-29T11:31:26.850407+0000","last_active":"2024-03-29T11:31:26.854970+0000","last_peered":"2024-03-29T11:31:26.854970+0000","last_clean":"2024-03-29T11:31:26.854970+0000","last_became_active":"2024-03-29T11:31:26.849457+0000","last_became_peered":"2024-03-29T11:31:26.849457+0000","last_unstale":"2024-03-29T11:31:26.854970+0000","last_undegraded":"2024-03-29T11:31:26.854970+0000","last_fullsized":"2024-03-29T11:31:26.854970+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:38:50.961636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854522+0000","last_change":"2024-03-29T11:31:26.319707+0000","last_active":"2024-03-29T11:31:26.854522+0000","last_peered":"2024-03-29T11:31:26.854522+0000","last_clean":"2024-03-29T11:31:26.854522+0000","last_became_active":"2024-03-29T11:31:26.316121+0000","last_became_peered":"2024-03-29T11:31:26.316121+0000","last_unstale":"2024-03-29T11:31:26.854522+0000","last_undegraded":"2024-03-29T11:31:26.854522+0000","last_fullsized":"2024-03-29T11:31:26.854522+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:12:51.039260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845866+0000","last_change":"2024-03-29T11:31:26.840358+0000","last_active":"2024-03-29T11:31:26.845866+0000","last_peered":"2024-03-29T11:31:26.845866+0000","last_clean":"2024-03-29T11:31:26.845866+0000","last_became_active":"2024-03-29T11:31:26.838477+0000","last_became_peered":"2024-03-29T11:31:26.838477+0000","last_unstale":"2024-03-29T11:31:26.845866+0000","last_undegraded":"2024-03-29T11:31:26.845866+0000","last_fullsized":"2024-03-29T11:31:26.845866+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:27.122635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854050+0000","last_change":"2024-03-29T11:31:26.321235+0000","last_active":"2024-03-29T11:31:26.854050+0000","last_peered":"2024-03-29T11:31:26.854050+0000","last_clean":"2024-03-29T11:31:26.854050+0000","last_became_active":"2024-03-29T11:31:26.320454+0000","last_became_peered":"2024-03-29T11:31:26.320454+0000","last_unstale":"2024-03-29T11:31:26.854050+0000","last_undegraded":"2024-03-29T11:31:26.854050+0000","last_fullsized":"2024-03-29T11:31:26.854050+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:27.238049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.856497+0000","last_change":"2024-03-29T11:31:26.853179+0000","last_active":"2024-03-29T11:31:26.856497+0000","last_peered":"2024-03-29T11:31:26.856497+0000","last_clean":"2024-03-29T11:31:26.856497+0000","last_became_active":"2024-03-29T11:31:26.848186+0000","last_became_peered":"2024-03-29T11:31:26.848186+0000","last_unstale":"2024-03-29T11:31:26.856497+0000","last_undegraded":"2024-03-29T11:31:26.856497+0000","last_fullsized":"2024-03-29T11:31:26.856497+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:35:08.770095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.851789+0000","last_change":"2024-03-29T11:31:26.311383+0000","last_active":"2024-03-29T11:31:26.851789+0000","last_peered":"2024-03-29T11:31:26.851789+0000","last_clean":"2024-03-29T11:31:26.851789+0000","last_became_active":"2024-03-29T11:31:26.310630+0000","last_became_peered":"2024-03-29T11:31:26.310630+0000","last_unstale":"2024-03-29T11:31:26.851789+0000","last_undegraded":"2024-03-29T11:31:26.851789+0000","last_fullsized":"2024-03-29T11:31:26.851789+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:51:16.023346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852437+0000","last_change":"2024-03-29T11:31:26.850244+0000","last_active":"2024-03-29T11:31:26.852437+0000","last_peered":"2024-03-29T11:31:26.852437+0000","last_clean":"2024-03-29T11:31:26.852437+0000","last_became_active":"2024-03-29T11:31:26.849736+0000","last_became_peered":"2024-03-29T11:31:26.849736+0000","last_unstale":"2024-03-29T11:31:26.852437+0000","last_undegraded":"2024-03-29T11:31:26.852437+0000","last_fullsized":"2024-03-29T11:31:26.852437+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:57.906301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.831185+0000","last_change":"2024-03-29T11:31:26.310235+0000","last_active":"2024-03-29T11:31:26.831185+0000","last_peered":"2024-03-29T11:31:26.831185+0000","last_clean":"2024-03-29T11:31:26.831185+0000","last_became_active":"2024-03-29T11:31:26.309932+0000","last_became_peered":"2024-03-29T11:31:26.309932+0000","last_unstale":"2024-03-29T11:31:26.831185+0000","last_undegraded":"2024-03-29T11:31:26.831185+0000","last_fullsized":"2024-03-29T11:31:26.831185+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:31:30.532498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855515+0000","last_change":"2024-03-29T11:31:26.847903+0000","last_active":"2024-03-29T11:31:26.855515+0000","last_peered":"2024-03-29T11:31:26.855515+0000","last_clean":"2024-03-29T11:31:26.855515+0000","last_became_active":"2024-03-29T11:31:26.846171+0000","last_became_peered":"2024-03-29T11:31:26.846171+0000","last_unstale":"2024-03-29T11:31:26.855515+0000","last_undegraded":"2024-03-29T11:31:26.855515+0000","last_fullsized":"2024-03-29T11:31:26.855515+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:24.230359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.826484+0000","last_change":"2024-03-29T11:31:26.320631+0000","last_active":"2024-03-29T11:31:26.826484+0000","last_peered":"2024-03-29T11:31:26.826484+0000","last_clean":"2024-03-29T11:31:26.826484+0000","last_became_active":"2024-03-29T11:31:26.320350+0000","last_became_peered":"2024-03-29T11:31:26.320350+0000","last_unstale":"2024-03-29T11:31:26.826484+0000","last_undegraded":"2024-03-29T11:31:26.826484+0000","last_fullsized":"2024-03-29T11:31:26.826484+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:57.806178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.857532+0000","last_change":"2024-03-29T11:31:26.855805+0000","last_active":"2024-03-29T11:31:26.857532+0000","last_peered":"2024-03-29T11:31:26.857532+0000","last_clean":"2024-03-29T11:31:26.857532+0000","last_became_active":"2024-03-29T11:31:26.854618+0000","last_became_peered":"2024-03-29T11:31:26.854618+0000","last_unstale":"2024-03-29T11:31:26.857532+0000","last_undegraded":"2024-03-29T11:31:26.857532+0000","last_fullsized":"2024-03-29T11:31:26.857532+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.831388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.824673+0000","last_change":"2024-03-29T11:31:26.317297+0000","last_active":"2024-03-29T11:31:26.824673+0000","last_peered":"2024-03-29T11:31:26.824673+0000","last_clean":"2024-03-29T11:31:26.824673+0000","last_became_active":"2024-03-29T11:31:26.316956+0000","last_became_peered":"2024-03-29T11:31:26.316956+0000","last_unstale":"2024-03-29T11:31:26.824673+0000","last_undegraded":"2024-03-29T11:31:26.824673+0000","last_fullsized":"2024-03-29T11:31:26.824673+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:18.302941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.849098+0000","last_change":"2024-03-29T11:31:26.846344+0000","last_active":"2024-03-29T11:31:26.849098+0000","last_peered":"2024-03-29T11:31:26.849098+0000","last_clean":"2024-03-29T11:31:26.849098+0000","last_became_active":"2024-03-29T11:31:26.846066+0000","last_became_peered":"2024-03-29T11:31:26.846066+0000","last_unstale":"2024-03-29T11:31:26.849098+0000","last_undegraded":"2024-03-29T11:31:26.849098+0000","last_fullsized":"2024-03-29T11:31:26.849098+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:57.163908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.826341+0000","last_change":"2024-03-29T11:31:26.316324+0000","last_active":"2024-03-29T11:31:26.826341+0000","last_peered":"2024-03-29T11:31:26.826341+0000","last_clean":"2024-03-29T11:31:26.826341+0000","last_became_active":"2024-03-29T11:31:26.316066+0000","last_became_peered":"2024-03-29T11:31:26.316066+0000","last_unstale":"2024-03-29T11:31:26.826341+0000","last_undegraded":"2024-03-29T11:31:26.826341+0000","last_fullsized":"2024-03-29T11:31:26.826341+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:24:19.700336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.857974+0000","last_change":"2024-03-29T11:31:26.857148+0000","last_active":"2024-03-29T11:31:26.857974+0000","last_peered":"2024-03-29T11:31:26.857974+0000","last_clean":"2024-03-29T11:31:26.857974+0000","last_became_active":"2024-03-29T11:31:26.856870+0000","last_became_peered":"2024-03-29T11:31:26.856870+0000","last_unstale":"2024-03-29T11:31:26.857974+0000","last_undegraded":"2024-03-29T11:31:26.857974+0000","last_fullsized":"2024-03-29T11:31:26.857974+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"226'3","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854630+0000","last_change":"2024-03-29T11:31:26.320127+0000","last_active":"2024-03-29T11:31:26.854630+0000","last_peered":"2024-03-29T11:31:26.854630+0000","last_clean":"2024-03-29T11:31:26.854630+0000","last_became_active":"2024-03-29T11:31:26.319346+0000","last_became_peered":"2024-03-29T11:31:26.319346+0000","last_unstale":"2024-03-29T11:31:26.854630+0000","last_undegraded":"2024-03-29T11:31:26.854630+0000","last_fullsized":"2024-03-29T11:31:26.854630+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:32:50.637683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853294+0000","last_change":"2024-03-29T11:31:26.844802+0000","last_active":"2024-03-29T11:31:26.853294+0000","last_peered":"2024-03-29T11:31:26.853294+0000","last_clean":"2024-03-29T11:31:26.853294+0000","last_became_active":"2024-03-29T11:31:26.844538+0000","last_became_peered":"2024-03-29T11:31:26.844538+0000","last_unstale":"2024-03-29T11:31:26.853294+0000","last_undegraded":"2024-03-29T11:31:26.853294+0000","last_fullsized":"2024-03-29T11:31:26.853294+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:05.909043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"226'6","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.858524+0000","last_change":"2024-03-29T11:31:26.319966+0000","last_active":"2024-03-29T11:31:26.858524+0000","last_peered":"2024-03-29T11:31:26.858524+0000","last_clean":"2024-03-29T11:31:26.858524+0000","last_became_active":"2024-03-29T11:31:26.319700+0000","last_became_peered":"2024-03-29T11:31:26.319700+0000","last_unstale":"2024-03-29T11:31:26.858524+0000","last_undegraded":"2024-03-29T11:31:26.858524+0000","last_fullsized":"2024-03-29T11:31:26.858524+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:56.161261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.842254+0000","last_change":"2024-03-29T11:31:26.840312+0000","last_active":"2024-03-29T11:31:26.842254+0000","last_peered":"2024-03-29T11:31:26.842254+0000","last_clean":"2024-03-29T11:31:26.842254+0000","last_became_active":"2024-03-29T11:31:26.840020+0000","last_became_peered":"2024-03-29T11:31:26.840020+0000","last_unstale":"2024-03-29T11:31:26.842254+0000","last_undegraded":"2024-03-29T11:31:26.842254+0000","last_fullsized":"2024-03-29T11:31:26.842254+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:31:51.171001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852521+0000","last_change":"2024-03-29T11:31:26.313768+0000","last_active":"2024-03-29T11:31:26.852521+0000","last_peered":"2024-03-29T11:31:26.852521+0000","last_clean":"2024-03-29T11:31:26.852521+0000","last_became_active":"2024-03-29T11:31:26.313458+0000","last_became_peered":"2024-03-29T11:31:26.313458+0000","last_unstale":"2024-03-29T11:31:26.852521+0000","last_undegraded":"2024-03-29T11:31:26.852521+0000","last_fullsized":"2024-03-29T11:31:26.852521+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:06.334180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.834985+0000","last_change":"2024-03-29T11:31:26.833527+0000","last_active":"2024-03-29T11:31:26.834985+0000","last_peered":"2024-03-29T11:31:26.834985+0000","last_clean":"2024-03-29T11:31:26.834985+0000","last_became_active":"2024-03-29T11:31:26.833280+0000","last_became_peered":"2024-03-29T11:31:26.833280+0000","last_unstale":"2024-03-29T11:31:26.834985+0000","last_undegraded":"2024-03-29T11:31:26.834985+0000","last_fullsized":"2024-03-29T11:31:26.834985+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:35:08.770095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.827275+0000","last_change":"2024-03-29T11:31:26.315775+0000","last_active":"2024-03-29T11:31:26.827275+0000","last_peered":"2024-03-29T11:31:26.827275+0000","last_clean":"2024-03-29T11:31:26.827275+0000","last_became_active":"2024-03-29T11:31:26.315493+0000","last_became_peered":"2024-03-29T11:31:26.315493+0000","last_unstale":"2024-03-29T11:31:26.827275+0000","last_undegraded":"2024-03-29T11:31:26.827275+0000","last_fullsized":"2024-03-29T11:31:26.827275+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:32:47.003355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.849908+0000","last_change":"2024-03-29T11:31:26.846893+0000","last_active":"2024-03-29T11:31:26.849908+0000","last_peered":"2024-03-29T11:31:26.849908+0000","last_clean":"2024-03-29T11:31:26.849908+0000","last_became_active":"2024-03-29T11:31:26.845341+0000","last_became_peered":"2024-03-29T11:31:26.845341+0000","last_unstale":"2024-03-29T11:31:26.849908+0000","last_undegraded":"2024-03-29T11:31:26.849908+0000","last_fullsized":"2024-03-29T11:31:26.849908+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:28.240696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.828686+0000","last_change":"2024-03-29T11:31:26.314471+0000","last_active":"2024-03-29T11:31:26.828686+0000","last_peered":"2024-03-29T11:31:26.828686+0000","last_clean":"2024-03-29T11:31:26.828686+0000","last_became_active":"2024-03-29T11:31:26.314204+0000","last_became_peered":"2024-03-29T11:31:26.314204+0000","last_unstale":"2024-03-29T11:31:26.828686+0000","last_undegraded":"2024-03-29T11:31:26.828686+0000","last_fullsized":"2024-03-29T11:31:26.828686+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:49:13.676544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852263+0000","last_change":"2024-03-29T11:31:26.843461+0000","last_active":"2024-03-29T11:31:26.852263+0000","last_peered":"2024-03-29T11:31:26.852263+0000","last_clean":"2024-03-29T11:31:26.852263+0000","last_became_active":"2024-03-29T11:31:26.839816+0000","last_became_peered":"2024-03-29T11:31:26.839816+0000","last_unstale":"2024-03-29T11:31:26.852263+0000","last_undegraded":"2024-03-29T11:31:26.852263+0000","last_fullsized":"2024-03-29T11:31:26.852263+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:32.997118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"226'8","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.857280+0000","last_change":"2024-03-29T11:31:26.320392+0000","last_active":"2024-03-29T11:31:26.857280+0000","last_peered":"2024-03-29T11:31:26.857280+0000","last_clean":"2024-03-29T11:31:26.857280+0000","last_became_active":"2024-03-29T11:31:26.319139+0000","last_became_peered":"2024-03-29T11:31:26.319139+0000","last_unstale":"2024-03-29T11:31:26.857280+0000","last_undegraded":"2024-03-29T11:31:26.857280+0000","last_fullsized":"2024-03-29T11:31:26.857280+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:57.651247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855756+0000","last_change":"2024-03-29T11:31:26.854468+0000","last_active":"2024-03-29T11:31:26.855756+0000","last_peered":"2024-03-29T11:31:26.855756+0000","last_clean":"2024-03-29T11:31:26.855756+0000","last_became_active":"2024-03-29T11:31:26.851644+0000","last_became_peered":"2024-03-29T11:31:26.851644+0000","last_unstale":"2024-03-29T11:31:26.855756+0000","last_undegraded":"2024-03-29T11:31:26.855756+0000","last_fullsized":"2024-03-29T11:31:26.855756+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:16.143243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.849979+0000","last_change":"2024-03-29T11:31:26.320455+0000","last_active":"2024-03-29T11:31:26.849979+0000","last_peered":"2024-03-29T11:31:26.849979+0000","last_clean":"2024-03-29T11:31:26.849979+0000","last_became_active":"2024-03-29T11:31:26.320175+0000","last_became_peered":"2024-03-29T11:31:26.320175+0000","last_unstale":"2024-03-29T11:31:26.849979+0000","last_undegraded":"2024-03-29T11:31:26.849979+0000","last_fullsized":"2024-03-29T11:31:26.849979+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:25:12.671618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.834947+0000","last_change":"2024-03-29T11:31:26.833432+0000","last_active":"2024-03-29T11:31:26.834947+0000","last_peered":"2024-03-29T11:31:26.834947+0000","last_clean":"2024-03-29T11:31:26.834947+0000","last_became_active":"2024-03-29T11:31:26.833078+0000","last_became_peered":"2024-03-29T11:31:26.833078+0000","last_unstale":"2024-03-29T11:31:26.834947+0000","last_undegraded":"2024-03-29T11:31:26.834947+0000","last_fullsized":"2024-03-29T11:31:26.834947+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:17.353377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854785+0000","last_change":"2024-03-29T11:31:26.317437+0000","last_active":"2024-03-29T11:31:26.854785+0000","last_peered":"2024-03-29T11:31:26.854785+0000","last_clean":"2024-03-29T11:31:26.854785+0000","last_became_active":"2024-03-29T11:31:26.317128+0000","last_became_peered":"2024-03-29T11:31:26.317128+0000","last_unstale":"2024-03-29T11:31:26.854785+0000","last_undegraded":"2024-03-29T11:31:26.854785+0000","last_fullsized":"2024-03-29T11:31:26.854785+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:57.806178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.838242+0000","last_change":"2024-03-29T11:31:26.833306+0000","last_active":"2024-03-29T11:31:26.838242+0000","last_peered":"2024-03-29T11:31:26.838242+0000","last_clean":"2024-03-29T11:31:26.838242+0000","last_became_active":"2024-03-29T11:31:26.833041+0000","last_became_peered":"2024-03-29T11:31:26.833041+0000","last_unstale":"2024-03-29T11:31:26.838242+0000","last_undegraded":"2024-03-29T11:31:26.838242+0000","last_fullsized":"2024-03-29T11:31:26.838242+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:15:11.283200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.848252+0000","last_change":"2024-03-29T11:31:26.319951+0000","last_active":"2024-03-29T11:31:26.848252+0000","last_peered":"2024-03-29T11:31:26.848252+0000","last_clean":"2024-03-29T11:31:26.848252+0000","last_became_active":"2024-03-29T11:31:26.318226+0000","last_became_peered":"2024-03-29T11:31:26.318226+0000","last_unstale":"2024-03-29T11:31:26.848252+0000","last_undegraded":"2024-03-29T11:31:26.848252+0000","last_fullsized":"2024-03-29T11:31:26.848252+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:22.562759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.851358+0000","last_change":"2024-03-29T11:31:26.848108+0000","last_active":"2024-03-29T11:31:26.851358+0000","last_peered":"2024-03-29T11:31:26.851358+0000","last_clean":"2024-03-29T11:31:26.851358+0000","last_became_active":"2024-03-29T11:31:26.847705+0000","last_became_peered":"2024-03-29T11:31:26.847705+0000","last_unstale":"2024-03-29T11:31:26.851358+0000","last_undegraded":"2024-03-29T11:31:26.851358+0000","last_fullsized":"2024-03-29T11:31:26.851358+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:53:43.020271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.820860+0000","last_change":"2024-03-29T11:31:26.313158+0000","last_active":"2024-03-29T11:31:26.820860+0000","last_peered":"2024-03-29T11:31:26.820860+0000","last_clean":"2024-03-29T11:31:26.820860+0000","last_became_active":"2024-03-29T11:31:26.311492+0000","last_became_peered":"2024-03-29T11:31:26.311492+0000","last_unstale":"2024-03-29T11:31:26.820860+0000","last_undegraded":"2024-03-29T11:31:26.820860+0000","last_fullsized":"2024-03-29T11:31:26.820860+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:40.347554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847445+0000","last_change":"2024-03-29T11:31:26.844232+0000","last_active":"2024-03-29T11:31:26.847445+0000","last_peered":"2024-03-29T11:31:26.847445+0000","last_clean":"2024-03-29T11:31:26.847445+0000","last_became_active":"2024-03-29T11:31:26.843977+0000","last_became_peered":"2024-03-29T11:31:26.843977+0000","last_unstale":"2024-03-29T11:31:26.847445+0000","last_undegraded":"2024-03-29T11:31:26.847445+0000","last_fullsized":"2024-03-29T11:31:26.847445+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.849162+0000","last_change":"2024-03-29T11:31:26.316435+0000","last_active":"2024-03-29T11:31:26.849162+0000","last_peered":"2024-03-29T11:31:26.849162+0000","last_clean":"2024-03-29T11:31:26.849162+0000","last_became_active":"2024-03-29T11:31:26.316124+0000","last_became_peered":"2024-03-29T11:31:26.316124+0000","last_unstale":"2024-03-29T11:31:26.849162+0000","last_undegraded":"2024-03-29T11:31:26.849162+0000","last_fullsized":"2024-03-29T11:31:26.849162+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:15:10.280553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847286+0000","last_change":"2024-03-29T11:31:26.836881+0000","last_active":"2024-03-29T11:31:26.847286+0000","last_peered":"2024-03-29T11:31:26.847286+0000","last_clean":"2024-03-29T11:31:26.847286+0000","last_became_active":"2024-03-29T11:31:26.836613+0000","last_became_peered":"2024-03-29T11:31:26.836613+0000","last_unstale":"2024-03-29T11:31:26.847286+0000","last_undegraded":"2024-03-29T11:31:26.847286+0000","last_fullsized":"2024-03-29T11:31:26.847286+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:23.565406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.830838+0000","last_change":"2024-03-29T11:31:26.312485+0000","last_active":"2024-03-29T11:31:26.830838+0000","last_peered":"2024-03-29T11:31:26.830838+0000","last_clean":"2024-03-29T11:31:26.830838+0000","last_became_active":"2024-03-29T11:31:26.311989+0000","last_became_peered":"2024-03-29T11:31:26.311989+0000","last_unstale":"2024-03-29T11:31:26.830838+0000","last_undegraded":"2024-03-29T11:31:26.830838+0000","last_fullsized":"2024-03-29T11:31:26.830838+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:42:56.190784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845507+0000","last_change":"2024-03-29T11:31:26.835978+0000","last_active":"2024-03-29T11:31:26.845507+0000","last_peered":"2024-03-29T11:31:26.845507+0000","last_clean":"2024-03-29T11:31:26.845507+0000","last_became_active":"2024-03-29T11:31:26.835400+0000","last_became_peered":"2024-03-29T11:31:26.835400+0000","last_unstale":"2024-03-29T11:31:26.845507+0000","last_undegraded":"2024-03-29T11:31:26.845507+0000","last_fullsized":"2024-03-29T11:31:26.845507+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:31.783000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.828017+0000","last_change":"2024-03-29T11:31:26.312962+0000","last_active":"2024-03-29T11:31:26.828017+0000","last_peered":"2024-03-29T11:31:26.828017+0000","last_clean":"2024-03-29T11:31:26.828017+0000","last_became_active":"2024-03-29T11:31:26.312668+0000","last_became_peered":"2024-03-29T11:31:26.312668+0000","last_unstale":"2024-03-29T11:31:26.828017+0000","last_undegraded":"2024-03-29T11:31:26.828017+0000","last_fullsized":"2024-03-29T11:31:26.828017+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:46.870101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845496+0000","last_change":"2024-03-29T11:31:26.843420+0000","last_active":"2024-03-29T11:31:26.845496+0000","last_peered":"2024-03-29T11:31:26.845496+0000","last_clean":"2024-03-29T11:31:26.845496+0000","last_became_active":"2024-03-29T11:31:26.842734+0000","last_became_peered":"2024-03-29T11:31:26.842734+0000","last_unstale":"2024-03-29T11:31:26.845496+0000","last_undegraded":"2024-03-29T11:31:26.845496+0000","last_fullsized":"2024-03-29T11:31:26.845496+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:46.006140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.825409+0000","last_change":"2024-03-29T11:31:26.321619+0000","last_active":"2024-03-29T11:31:26.825409+0000","last_peered":"2024-03-29T11:31:26.825409+0000","last_clean":"2024-03-29T11:31:26.825409+0000","last_became_active":"2024-03-29T11:31:26.321349+0000","last_became_peered":"2024-03-29T11:31:26.321349+0000","last_unstale":"2024-03-29T11:31:26.825409+0000","last_undegraded":"2024-03-29T11:31:26.825409+0000","last_fullsized":"2024-03-29T11:31:26.825409+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:09.446187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854485+0000","last_change":"2024-03-29T11:31:26.851615+0000","last_active":"2024-03-29T11:31:26.854485+0000","last_peered":"2024-03-29T11:31:26.854485+0000","last_clean":"2024-03-29T11:31:26.854485+0000","last_became_active":"2024-03-29T11:31:26.851328+0000","last_became_peered":"2024-03-29T11:31:26.851328+0000","last_unstale":"2024-03-29T11:31:26.854485+0000","last_undegraded":"2024-03-29T11:31:26.854485+0000","last_fullsized":"2024-03-29T11:31:26.854485+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:57.906301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.849180+0000","last_change":"2024-03-29T11:31:26.310244+0000","last_active":"2024-03-29T11:31:26.849180+0000","last_peered":"2024-03-29T11:31:26.849180+0000","last_clean":"2024-03-29T11:31:26.849180+0000","last_became_active":"2024-03-29T11:31:26.309981+0000","last_became_peered":"2024-03-29T11:31:26.309981+0000","last_unstale":"2024-03-29T11:31:26.849180+0000","last_undegraded":"2024-03-29T11:31:26.849180+0000","last_fullsized":"2024-03-29T11:31:26.849180+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:16.350729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853300+0000","last_change":"2024-03-29T11:31:26.845919+0000","last_active":"2024-03-29T11:31:26.853300+0000","last_peered":"2024-03-29T11:31:26.853300+0000","last_clean":"2024-03-29T11:31:26.853300+0000","last_became_active":"2024-03-29T11:31:26.845630+0000","last_became_peered":"2024-03-29T11:31:26.845630+0000","last_unstale":"2024-03-29T11:31:26.853300+0000","last_undegraded":"2024-03-29T11:31:26.853300+0000","last_fullsized":"2024-03-29T11:31:26.853300+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:47.195933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.844537+0000","last_change":"2024-03-29T11:31:26.316463+0000","last_active":"2024-03-29T11:31:26.844537+0000","last_peered":"2024-03-29T11:31:26.844537+0000","last_clean":"2024-03-29T11:31:26.844537+0000","last_became_active":"2024-03-29T11:31:26.315640+0000","last_became_peered":"2024-03-29T11:31:26.315640+0000","last_unstale":"2024-03-29T11:31:26.844537+0000","last_undegraded":"2024-03-29T11:31:26.844537+0000","last_fullsized":"2024-03-29T11:31:26.844537+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:31.994471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854449+0000","last_change":"2024-03-29T11:31:26.849336+0000","last_active":"2024-03-29T11:31:26.854449+0000","last_peered":"2024-03-29T11:31:26.854449+0000","last_clean":"2024-03-29T11:31:26.854449+0000","last_became_active":"2024-03-29T11:31:26.849042+0000","last_became_peered":"2024-03-29T11:31:26.849042+0000","last_unstale":"2024-03-29T11:31:26.854449+0000","last_undegraded":"2024-03-29T11:31:26.854449+0000","last_fullsized":"2024-03-29T11:31:26.854449+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:58.653895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.831304+0000","last_change":"2024-03-29T11:31:26.317291+0000","last_active":"2024-03-29T11:31:26.831304+0000","last_peered":"2024-03-29T11:31:26.831304+0000","last_clean":"2024-03-29T11:31:26.831304+0000","last_became_active":"2024-03-29T11:31:26.317003+0000","last_became_peered":"2024-03-29T11:31:26.317003+0000","last_unstale":"2024-03-29T11:31:26.831304+0000","last_undegraded":"2024-03-29T11:31:26.831304+0000","last_fullsized":"2024-03-29T11:31:26.831304+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:55:21.607497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847388+0000","last_change":"2024-03-29T11:31:26.844018+0000","last_active":"2024-03-29T11:31:26.847388+0000","last_peered":"2024-03-29T11:31:26.847388+0000","last_clean":"2024-03-29T11:31:26.847388+0000","last_became_active":"2024-03-29T11:31:26.843237+0000","last_became_peered":"2024-03-29T11:31:26.843237+0000","last_unstale":"2024-03-29T11:31:26.847388+0000","last_undegraded":"2024-03-29T11:31:26.847388+0000","last_fullsized":"2024-03-29T11:31:26.847388+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:12:52.041908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.821002+0000","last_change":"2024-03-29T11:31:26.311780+0000","last_active":"2024-03-29T11:31:26.821002+0000","last_peered":"2024-03-29T11:31:26.821002+0000","last_clean":"2024-03-29T11:31:26.821002+0000","last_became_active":"2024-03-29T11:31:26.311128+0000","last_became_peered":"2024-03-29T11:31:26.311128+0000","last_unstale":"2024-03-29T11:31:26.821002+0000","last_undegraded":"2024-03-29T11:31:26.821002+0000","last_fullsized":"2024-03-29T11:31:26.821002+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:46.870101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853833+0000","last_change":"2024-03-29T11:31:26.848609+0000","last_active":"2024-03-29T11:31:26.853833+0000","last_peered":"2024-03-29T11:31:26.853833+0000","last_clean":"2024-03-29T11:31:26.853833+0000","last_became_active":"2024-03-29T11:31:26.848202+0000","last_became_peered":"2024-03-29T11:31:26.848202+0000","last_unstale":"2024-03-29T11:31:26.853833+0000","last_undegraded":"2024-03-29T11:31:26.853833+0000","last_fullsized":"2024-03-29T11:31:26.853833+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:28.240696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.838927+0000","last_change":"2024-03-29T11:31:26.320428+0000","last_active":"2024-03-29T11:31:26.838927+0000","last_peered":"2024-03-29T11:31:26.838927+0000","last_clean":"2024-03-29T11:31:26.838927+0000","last_became_active":"2024-03-29T11:31:26.320099+0000","last_became_peered":"2024-03-29T11:31:26.320099+0000","last_unstale":"2024-03-29T11:31:26.838927+0000","last_undegraded":"2024-03-29T11:31:26.838927+0000","last_fullsized":"2024-03-29T11:31:26.838927+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:56.873408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.843113+0000","last_change":"2024-03-29T11:31:26.838532+0000","last_active":"2024-03-29T11:31:26.843113+0000","last_peered":"2024-03-29T11:31:26.843113+0000","last_clean":"2024-03-29T11:31:26.843113+0000","last_became_active":"2024-03-29T11:31:26.838267+0000","last_became_peered":"2024-03-29T11:31:26.838267+0000","last_unstale":"2024-03-29T11:31:26.843113+0000","last_undegraded":"2024-03-29T11:31:26.843113+0000","last_fullsized":"2024-03-29T11:31:26.843113+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:17:41.239491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.825907+0000","last_change":"2024-03-29T11:31:26.322425+0000","last_active":"2024-03-29T11:31:26.825907+0000","last_peered":"2024-03-29T11:31:26.825907+0000","last_clean":"2024-03-29T11:31:26.825907+0000","last_became_active":"2024-03-29T11:31:26.321981+0000","last_became_peered":"2024-03-29T11:31:26.321981+0000","last_unstale":"2024-03-29T11:31:26.825907+0000","last_undegraded":"2024-03-29T11:31:26.825907+0000","last_fullsized":"2024-03-29T11:31:26.825907+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:36.434947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.848339+0000","last_change":"2024-03-29T11:31:26.842129+0000","last_active":"2024-03-29T11:31:26.848339+0000","last_peered":"2024-03-29T11:31:26.848339+0000","last_clean":"2024-03-29T11:31:26.848339+0000","last_became_active":"2024-03-29T11:31:26.841888+0000","last_became_peered":"2024-03-29T11:31:26.841888+0000","last_unstale":"2024-03-29T11:31:26.848339+0000","last_undegraded":"2024-03-29T11:31:26.848339+0000","last_fullsized":"2024-03-29T11:31:26.848339+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:32.997118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.827539+0000","last_change":"2024-03-29T11:31:26.310072+0000","last_active":"2024-03-29T11:31:26.827539+0000","last_peered":"2024-03-29T11:31:26.827539+0000","last_clean":"2024-03-29T11:31:26.827539+0000","last_became_active":"2024-03-29T11:31:26.309254+0000","last_became_peered":"2024-03-29T11:31:26.309254+0000","last_unstale":"2024-03-29T11:31:26.827539+0000","last_undegraded":"2024-03-29T11:31:26.827539+0000","last_fullsized":"2024-03-29T11:31:26.827539+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:00:36.402074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847166+0000","last_change":"2024-03-29T11:31:26.843285+0000","last_active":"2024-03-29T11:31:26.847166+0000","last_peered":"2024-03-29T11:31:26.847166+0000","last_clean":"2024-03-29T11:31:26.847166+0000","last_became_active":"2024-03-29T11:31:26.841561+0000","last_became_peered":"2024-03-29T11:31:26.841561+0000","last_unstale":"2024-03-29T11:31:26.847166+0000","last_undegraded":"2024-03-29T11:31:26.847166+0000","last_fullsized":"2024-03-29T11:31:26.847166+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:47.872748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.836521+0000","last_change":"2024-03-29T11:31:26.318569+0000","last_active":"2024-03-29T11:31:26.836521+0000","last_peered":"2024-03-29T11:31:26.836521+0000","last_clean":"2024-03-29T11:31:26.836521+0000","last_became_active":"2024-03-29T11:31:26.318298+0000","last_became_peered":"2024-03-29T11:31:26.318298+0000","last_unstale":"2024-03-29T11:31:26.836521+0000","last_undegraded":"2024-03-29T11:31:26.836521+0000","last_fullsized":"2024-03-29T11:31:26.836521+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:05:48.527536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852031+0000","last_change":"2024-03-29T11:31:26.844835+0000","last_active":"2024-03-29T11:31:26.852031+0000","last_peered":"2024-03-29T11:31:26.852031+0000","last_clean":"2024-03-29T11:31:26.852031+0000","last_became_active":"2024-03-29T11:31:26.844570+0000","last_became_peered":"2024-03-29T11:31:26.844570+0000","last_unstale":"2024-03-29T11:31:26.852031+0000","last_undegraded":"2024-03-29T11:31:26.852031+0000","last_fullsized":"2024-03-29T11:31:26.852031+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:53:43.020271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.850572+0000","last_change":"2024-03-29T11:31:26.318923+0000","last_active":"2024-03-29T11:31:26.850572+0000","last_peered":"2024-03-29T11:31:26.850572+0000","last_clean":"2024-03-29T11:31:26.850572+0000","last_became_active":"2024-03-29T11:31:26.318658+0000","last_became_peered":"2024-03-29T11:31:26.318658+0000","last_unstale":"2024-03-29T11:31:26.850572+0000","last_undegraded":"2024-03-29T11:31:26.850572+0000","last_fullsized":"2024-03-29T11:31:26.850572+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:25:12.671618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"215'366","reported_seq":1270,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.309655+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:31:26.309655+0000","last_peered":"2024-03-29T11:31:26.309655+0000","last_clean":"2024-03-29T11:31:26.309655+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:31:26.309655+0000","last_undegraded":"2024-03-29T11:31:26.309655+0000","last_fullsized":"2024-03-29T11:31:26.309655+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575994,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52504,"kb_used_data":3336,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319336,"statfs":{"total":96636764160,"available":96583000064,"internally_reserved":0,"allocated":3416064,"data_stored":1964224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":50330060},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72299999999999998,"15min":0.72999999999999998},"min":{"1min":0.47799999999999998,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":1.0229999999999999,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.72699999999999998,"15min":0.747},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.40300000000000002},"max":{"1min":0.98499999999999999,"5min":1.119,"15min":1.119},"last":0.627}]},{"osd":1,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.73099999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.96799999999999997,"5min":1.002,"15min":1.478},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.749,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97099999999999997,"5min":0.98699999999999999,"15min":1.393},"last":0.59399999999999997}]},{"osd":2,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.55800000000000005,"5min":0.443,"15min":0.443},"max":{"1min":0.89900000000000002,"5min":1.089,"15min":1.284},"last":0.436},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76500000000000001,"15min":0.77800000000000002},"min":{"1min":0.52900000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.032,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75900000000000001,"15min":0.76400000000000001},"min":{"1min":0.52000000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.96899999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.78600000000000003,"15min":0.79400000000000004},"min":{"1min":0.65200000000000002,"5min":0.55900000000000005,"15min":0.55300000000000005},"max":{"1min":1.0069999999999999,"5min":1.071,"15min":1.3169999999999999},"last":0.64200000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.627,"15min":0.63700000000000001},"min":{"1min":0.45300000000000001,"5min":0.44900000000000001,"15min":0.41999999999999998},"max":{"1min":0.88100000000000001,"5min":1.024,"15min":1.3779999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.48199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89900000000000002,"5min":0.95299999999999996,"15min":1.335},"last":0.46800000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.69199999999999995,"15min":0.67900000000000005},"min":{"1min":0.50800000000000001,"5min":0.46700000000000003,"15min":0.38400000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.347},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.72699999999999998,"15min":0.71499999999999997},"min":{"1min":0.55400000000000005,"5min":0.50900000000000001,"15min":0.44900000000000001},"max":{"1min":1.006,"5min":1.0169999999999999,"15min":1.1759999999999999},"last":0.73999999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.59999999999999998,"5min":0.48499999999999999,"15min":0.47899999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.2070000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.75700000000000001,"15min":0.73799999999999999},"min":{"1min":0.53200000000000003,"5min":0.442,"15min":0.435},"max":{"1min":1.014,"5min":1.071,"15min":1.071},"last":0.61199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46184,"kb_used_data":2772,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325656,"statfs":{"total":96636764160,"available":96589471744,"internally_reserved":0,"allocated":2838528,"data_stored":1381285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44431815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72699999999999998,"15min":0.746},"min":{"1min":0.52600000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89100000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.745,"15min":0.76800000000000002},"min":{"1min":0.61299999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.96399999999999997,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.621}]},{"osd":1,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73099999999999998,"15min":0.74399999999999999},"min":{"1min":0.61199999999999999,"5min":0.47599999999999998,"15min":0.44800000000000001},"max":{"1min":0.88,"5min":1.98,"15min":1.98},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76000000000000001,"15min":0.77900000000000003},"min":{"1min":0.51100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94499999999999995,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.69999999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74099999999999999,"15min":0.75700000000000001},"min":{"1min":0.54800000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.96199999999999997,"5min":1.669,"15min":1.669},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76200000000000001,"15min":0.77100000000000002},"min":{"1min":0.56899999999999995,"5min":0.49099999999999999,"15min":0.46200000000000002},"max":{"1min":1.016,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.58099999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.76000000000000001,"15min":0.77900000000000003},"min":{"1min":0.59199999999999997,"5min":0.54200000000000004,"15min":0.499},"max":{"1min":0.96099999999999997,"5min":2.52,"15min":2.52},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.79100000000000004,"15min":0.80500000000000005},"min":{"1min":0.69999999999999996,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.95099999999999996,"5min":1.877,"15min":1.877},"last":0.67600000000000005}]},{"osd":4,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.40699999999999997,"15min":0.40300000000000002},"max":{"1min":0.93999999999999995,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.69999999999999996,"15min":0.70899999999999996},"min":{"1min":0.55400000000000005,"5min":0.47299999999999998,"15min":0.45900000000000002},"max":{"1min":0.93500000000000005,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":0.504}]},{"osd":5,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65400000000000003},"min":{"1min":0.42599999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96099999999999997,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.70999999999999996,"15min":0.71199999999999997},"min":{"1min":0.503,"5min":0.46999999999999997,"15min":0.42199999999999999},"max":{"1min":0.98399999999999999,"5min":1.925,"15min":1.925},"last":0.64600000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.754,"15min":0.76400000000000001},"min":{"1min":0.59399999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.73399999999999999,"15min":0.74099999999999999},"min":{"1min":0.53600000000000003,"5min":0.504,"15min":0.47499999999999998},"max":{"1min":1.0009999999999999,"5min":1.726,"15min":1.726},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60132,"kb_used_data":2768,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311708,"statfs":{"total":96636764160,"available":96575188992,"internally_reserved":0,"allocated":2834432,"data_stored":1380703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.46899999999999997},"min":{"1min":0.33900000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.81100000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.41999999999999998,"5min":0.35299999999999998,"15min":0.34799999999999998},"max":{"1min":0.75900000000000001,"5min":0.81399999999999995,"15min":0.90300000000000002},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.42999999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77000000000000002,"5min":0.86199999999999999,"15min":0.995},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.49199999999999999,"5min":0.38900000000000001,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.83999999999999997,"15min":0.878},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75900000000000001,"15min":0.79200000000000004},"min":{"1min":0.55100000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.995,"5min":1.153,"15min":1.3660000000000001},"last":1.022},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.78300000000000003,"15min":0.80500000000000005},"min":{"1min":0.60199999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.96399999999999997,"5min":1.304,"15min":1.304},"last":0.97699999999999998}]},{"osd":5,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80100000000000005,"15min":0.82199999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.48199999999999998},"max":{"1min":0.97799999999999998,"5min":1.3380000000000001,"15min":1.3720000000000001},"last":0.997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.80900000000000005,"15min":0.83799999999999997},"min":{"1min":0.65300000000000002,"5min":0.59199999999999997,"15min":0.59099999999999997},"max":{"1min":1.0549999999999999,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.92600000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.79000000000000004,"15min":0.82299999999999995},"min":{"1min":0.67100000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.95599999999999996,"5min":1.1890000000000001,"15min":1.1919999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.84199999999999997,"15min":0.85499999999999998},"min":{"1min":0.61899999999999999,"5min":0.60999999999999999,"15min":0.53100000000000003},"max":{"1min":1.0569999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"last":1.042}]},{"osd":7,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.65900000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.034,"5min":1.327,"15min":1.3540000000000001},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.84399999999999997,"15min":0.87},"min":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":1.0700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60692,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311148,"statfs":{"total":96636764160,"available":96574615552,"internally_reserved":0,"allocated":3411968,"data_stored":1963205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.35799999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.67200000000000004,"5min":0.95899999999999996,"15min":1.087},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.40799999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.85799999999999998,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.40500000000000003,"5min":0.39200000000000002,"15min":0.36099999999999999},"max":{"1min":0.78100000000000003,"5min":1.028,"15min":1.5289999999999999},"last":0.371},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52100000000000002,"15min":0.52200000000000002},"min":{"1min":0.39300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.66800000000000004,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.40400000000000003,"15min":0.36299999999999999},"max":{"1min":0.84699999999999998,"5min":1.222,"15min":1.222},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.64700000000000002},"min":{"1min":0.51500000000000001,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.47299999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.81299999999999994,"15min":0.82199999999999995},"min":{"1min":0.621,"5min":0.495,"15min":0.495},"max":{"1min":1.022,"5min":1.1719999999999999,"15min":1.232},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.84399999999999997,"15min":0.85499999999999998},"min":{"1min":0.69399999999999995,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.96099999999999997,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.67900000000000005}]},{"osd":5,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.78700000000000003,"15min":0.79200000000000004},"min":{"1min":0.59699999999999998,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.004,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.82199999999999995,"15min":0.82799999999999996},"min":{"1min":0.57299999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.98499999999999999,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.56299999999999994}]},{"osd":6,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85999999999999999,"15min":0.86399999999999999},"min":{"1min":0.69799999999999995,"5min":0.62,"15min":0.60999999999999999},"max":{"1min":1.155,"5min":1.343,"15min":1.6240000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.88400000000000001,"15min":0.89500000000000002},"min":{"1min":0.63900000000000001,"5min":0.58499999999999996,"15min":0.58199999999999996},"max":{"1min":1.1399999999999999,"5min":1.3240000000000001,"15min":1.337},"last":0.74299999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82299999999999995,"15min":0.82999999999999996},"min":{"1min":0.56000000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.96499999999999997,"5min":1.1299999999999999,"15min":1.214},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88400000000000001},"min":{"1min":0.65400000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.103,"5min":1.365,"15min":1.365},"last":0.77100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575994,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47824,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324016,"statfs":{"total":96636764160,"available":96587792384,"internally_reserved":0,"allocated":2822144,"data_stored":1373856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.377,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67800000000000005,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73999999999999999,"5min":0.75600000000000001,"15min":0.96799999999999997},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.79000000000000004,"15min":0.80700000000000005},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.872,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.439}]},{"osd":3,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76700000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.44,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76300000000000001,"5min":1.038,"15min":1.038},"last":0.46200000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79100000000000004,"15min":0.81100000000000005},"min":{"1min":0.61599999999999999,"5min":0.57899999999999996,"15min":0.55100000000000005},"max":{"1min":0.89200000000000002,"5min":1.169,"15min":1.1970000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78500000000000003,"15min":0.81599999999999995},"min":{"1min":0.64800000000000002,"5min":0.57099999999999995,"15min":0.52400000000000002},"max":{"1min":1.157,"5min":1.244,"15min":1.244},"last":0.67500000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.61799999999999999,"5min":0.59799999999999998,"15min":0.53400000000000003},"max":{"1min":1.1200000000000001,"5min":1.256,"15min":1.256},"last":0.753},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81299999999999994,"15min":0.82999999999999996},"min":{"1min":0.72799999999999998,"5min":0.57799999999999996,"15min":0.56000000000000005},"max":{"1min":0.97499999999999998,"5min":1.24,"15min":1.3140000000000001},"last":0.60099999999999998}]},{"osd":6,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82799999999999996,"15min":0.85199999999999998},"min":{"1min":0.64800000000000002,"5min":0.58999999999999997,"15min":0.57799999999999996},"max":{"1min":1.105,"5min":1.1459999999999999,"15min":1.24},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.623,"5min":0.58599999999999997,"15min":0.58199999999999996},"max":{"1min":1.0880000000000001,"5min":1.1879999999999999,"15min":1.329},"last":0.624}]},{"osd":7,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.877,"15min":0.90000000000000002},"min":{"1min":0.67400000000000004,"5min":0.66800000000000004,"15min":0.64600000000000002},"max":{"1min":1.0209999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88900000000000001,"15min":0.91200000000000003},"min":{"1min":0.68300000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.071,"5min":1.304,"15min":1.304},"last":0.70899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575994,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64196,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307644,"statfs":{"total":96636764160,"available":96571027456,"internally_reserved":0,"allocated":2809856,"data_stored":1367509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":62912968},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.71299999999999997,"15min":0.751},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53900000000000003,"15min":0.55100000000000005},"min":{"1min":0.36099999999999999,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48899999999999999,"15min":0.498},"min":{"1min":0.36699999999999999,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":0.70199999999999996,"5min":0.72099999999999997,"15min":0.748},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":1.046},"last":0.44700000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.58099999999999996,"15min":0.59199999999999997},"min":{"1min":0.48799999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.72599999999999998,"5min":0.82599999999999996,"15min":0.96699999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.55700000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90800000000000003,"5min":1.0049999999999999,"15min":1.01},"last":0.62}]},{"osd":4,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.79100000000000004,"15min":0.80400000000000005},"min":{"1min":0.66600000000000004,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.1000000000000001,"5min":1.266,"15min":1.2889999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.56999999999999995,"5min":0.52400000000000002,"15min":0.51400000000000001},"max":{"1min":1.0680000000000001,"5min":1.306,"15min":1.306},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.81100000000000005,"15min":0.81599999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.502},"max":{"1min":1.048,"5min":1.123,"15min":1.252},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.81599999999999995,"15min":0.82799999999999996},"min":{"1min":0.59899999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.173,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.69499999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.85099999999999998,"15min":0.85899999999999999},"min":{"1min":0.67500000000000004,"5min":0.61199999999999999,"15min":0.57799999999999996},"max":{"1min":1.0900000000000001,"5min":1.288,"15min":1.288},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84199999999999997,"15min":0.84599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1779999999999999,"5min":1.248,"15min":1.248},"last":0.68400000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84399999999999997,"15min":0.86499999999999999},"min":{"1min":0.65300000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.137,"5min":1.331,"15min":1.331},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.88400000000000001,"15min":0.89300000000000002},"min":{"1min":0.70399999999999996,"5min":0.64900000000000002,"15min":0.57899999999999996},"max":{"1min":1.1559999999999999,"5min":1.286,"15min":1.286},"last":0.78100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60120,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311720,"statfs":{"total":96636764160,"available":96575201280,"internally_reserved":0,"allocated":2822144,"data_stored":1373856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70599999999999996,"15min":0.73199999999999998},"min":{"1min":0.50700000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85199999999999998,"5min":0.95499999999999996,"15min":1.0509999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.70599999999999996,"15min":0.73299999999999998},"min":{"1min":0.50800000000000001,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.86599999999999999,"5min":0.99299999999999999,"15min":1.1459999999999999},"last":0.62}]},{"osd":1,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71999999999999997,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":0.875,"5min":1.1000000000000001,"15min":1.1910000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72499999999999998,"15min":0.753},"min":{"1min":0.53200000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93899999999999995,"5min":0.94899999999999995,"15min":1.002},"last":0.69299999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.81200000000000006,"15min":0.83599999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.92500000000000004,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.80200000000000005,"15min":0.82199999999999995},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.90900000000000003,"5min":1.224,"15min":1.224},"last":0.54100000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73999999999999999,"15min":0.77000000000000002},"min":{"1min":0.56799999999999995,"5min":0.44,"15min":0.44},"max":{"1min":0.88300000000000001,"5min":1.0580000000000001,"15min":1.181},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77600000000000002,"15min":0.77700000000000002},"min":{"1min":0.58499999999999996,"5min":0.54700000000000004,"15min":0.48699999999999999},"max":{"1min":0.91300000000000003,"5min":0.995,"15min":1.2490000000000001},"last":0.66600000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.624,"15min":0.623},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86099999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.64200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45900000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83099999999999996,"5min":1.0920000000000001,"15min":1.169},"last":0.80700000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.45200000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":0.82299999999999995,"5min":1.0509999999999999,"15min":1.0800000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71399999999999997,"15min":0.71499999999999997},"min":{"1min":0.52000000000000002,"5min":0.48199999999999998,"15min":0.47699999999999998},"max":{"1min":0.95599999999999996,"5min":1.024,"15min":1.028},"last":0.77100000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.54200000000000004,"5min":0.438,"15min":0.436},"max":{"1min":0.83499999999999996,"5min":1.0389999999999999,"15min":1.518},"last":0.873},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.82399999999999995,"15min":0.82699999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.54400000000000004},"max":{"1min":0.96099999999999997,"5min":1.1619999999999999,"15min":1.3440000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575994,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42080,"kb_used_data":2772,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329760,"statfs":{"total":96636764160,"available":96593674240,"internally_reserved":0,"allocated":2838528,"data_stored":1381285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70499999999999996,"15min":0.71399999999999997},"min":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.95999999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.877,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76700000000000002},"min":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93600000000000005,"5min":0.95999999999999996,"15min":1.0740000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73399999999999999,"15min":0.73999999999999999},"min":{"1min":0.56899999999999995,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91800000000000004,"5min":1.123,"15min":1.123},"last":0.65100000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77400000000000002,"15min":0.76000000000000001},"min":{"1min":0.69299999999999995,"5min":0.51500000000000001,"15min":0.48299999999999998},"max":{"1min":0.91600000000000004,"5min":0.95299999999999996,"15min":1.04},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.748,"15min":0.755},"min":{"1min":0.627,"5min":0.51200000000000001,"15min":0.48399999999999999},"max":{"1min":0.91000000000000003,"5min":0.98499999999999999,"15min":1.0129999999999999},"last":0.72399999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.99099999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.77500000000000002,"15min":0.78000000000000003},"min":{"1min":0.66200000000000003,"5min":0.52000000000000002,"15min":0.51800000000000002},"max":{"1min":0.97699999999999998,"5min":1.0409999999999999,"15min":1.046},"last":0.84199999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.627,"15min":0.60299999999999998},"min":{"1min":0.52000000000000002,"5min":0.41799999999999998,"15min":0.38700000000000001},"max":{"1min":0.88300000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66900000000000004,"15min":0.64800000000000002},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.40600000000000003},"max":{"1min":0.95999999999999996,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.80200000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.68000000000000005,"15min":0.66300000000000003},"min":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42699999999999999},"max":{"1min":1.046,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.71399999999999997,"15min":0.69799999999999995},"min":{"1min":0.51400000000000001,"5min":0.51000000000000001,"15min":0.42499999999999999},"max":{"1min":1.0680000000000001,"5min":1.107,"15min":1.107},"last":0.78100000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.752,"15min":0.72399999999999998},"min":{"1min":0.56699999999999995,"5min":0.52900000000000003,"15min":0.42299999999999999},"max":{"1min":1.036,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.78900000000000003,"15min":0.76600000000000001},"min":{"1min":0.54800000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1000000000000001,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.98199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:31:31.734 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:31:31.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.972+0000 7fcfafe00700 1 Processor -- start 2024-03-29T11:31:31.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.972+0000 7fcfafe00700 1 -- start start 2024-03-29T11:31:31.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.973+0000 7fcfafe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcfa80ceec0 0x7fcfa81612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.973+0000 7fcfafe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcfa8161800 0x7fcfa80b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.973+0000 7fcfafe00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa8161c50 con 0x7fcfa815d640 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.973+0000 7fcfafe00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa80d30d0 con 0x7fcfa80ceec0 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.973+0000 7fcfafe00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa80b6840 con 0x7fcfa8161800 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.973+0000 7fcfaca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcfa80ceec0 0x7fcfa81612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfaca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcfa80ceec0 0x7fcfa81612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36832/0 (socket says 172.21.15.12:36832) 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfaca00700 1 -- 172.21.15.12:0/2847026853 learned_addr learned my addr 172.21.15.12:0/2847026853 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfade00700 1 --2- 172.21.15.12:0/2847026853 >> v2:172.21.15.159:3300/0 conn(0x7fcfa8161800 0x7fcfa80b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfaca00700 1 -- 172.21.15.12:0/2847026853 >> v2:172.21.15.159:3300/0 conn(0x7fcfa8161800 msgr2=0x7fcfa80b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfaca00700 1 --2- 172.21.15.12:0/2847026853 >> v2:172.21.15.159:3300/0 conn(0x7fcfa8161800 0x7fcfa80b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfaca00700 1 -- 172.21.15.12:0/2847026853 >> v1:172.21.15.159:6789/0 conn(0x7fcfa815d640 legacy=0x7fcfa80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:31.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfaca00700 1 -- 172.21.15.12:0/2847026853 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa80d2b30 con 0x7fcfa80ceec0 2024-03-29T11:31:31.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfade00700 1 --2- 172.21.15.12:0/2847026853 >> v2:172.21.15.159:3300/0 conn(0x7fcfa8161800 0x7fcfa80b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:31.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfaca00700 1 --2- 172.21.15.12:0/2847026853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80ceec0 0x7fcfa81612c0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8162360 tx=0x7fcf94009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff58d7ac46335c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:31.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfa7e00700 1 -- 172.21.15.12:0/2847026853 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94004230 con 0x7fcfa80ceec0 2024-03-29T11:31:31.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfa7e00700 1 -- 172.21.15.12:0/2847026853 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf94004390 con 0x7fcfa80ceec0 2024-03-29T11:31:31.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.974+0000 7fcfa7e00700 1 -- 172.21.15.12:0/2847026853 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf940036a0 con 0x7fcfa80ceec0 2024-03-29T11:31:31.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.975+0000 7fcfafe00700 1 -- 172.21.15.12:0/2847026853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80ceec0 msgr2=0x7fcfa81612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.975+0000 7fcfafe00700 1 --2- 172.21.15.12:0/2847026853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80ceec0 0x7fcfa81612c0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8162360 tx=0x7fcf94009bd0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.975+0000 7fcfafe00700 1 -- 172.21.15.12:0/2847026853 shutdown_connections 2024-03-29T11:31:31.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.975+0000 7fcfafe00700 1 --2- 172.21.15.12:0/2847026853 >> v2:172.21.15.159:3300/0 conn(0x7fcfa8161800 0x7fcfa80b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.975+0000 7fcfafe00700 1 --2- 172.21.15.12:0/2847026853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80ceec0 0x7fcfa81612c0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.975+0000 7fcfafe00700 1 -- 172.21.15.12:0/2847026853 >> 172.21.15.12:0/2847026853 conn(0x7fcfa808aff0 msgr2=0x7fcfa80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.975+0000 7fcfafe00700 1 -- 172.21.15.12:0/2847026853 shutdown_connections 2024-03-29T11:31:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.975+0000 7fcfafe00700 1 -- 172.21.15.12:0/2847026853 wait complete. 2024-03-29T11:31:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.977+0000 7fcfafe00700 1 Processor -- start 2024-03-29T11:31:31.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.977+0000 7fcfafe00700 1 -- start start 2024-03-29T11:31:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfafe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcfa80ceec0 0x7fcfa81f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfafe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcfa815d640 0x7fcfa80a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfafe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa8161800 0x7fcfa80c9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfafe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80ca9b0 con 0x7fcfa8161800 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfafe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81f2de0 con 0x7fcfa80ceec0 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfafe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa814c320 con 0x7fcfa815d640 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfade00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa8161800 0x7fcfa80c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfade00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa8161800 0x7fcfa80c9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36834/0 (socket says 172.21.15.12:36834) 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfade00700 1 -- 172.21.15.12:0/2630445574 learned_addr learned my addr 172.21.15.12:0/2630445574 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfaca00700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcfa815d640 0x7fcfa80a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.978+0000 7fcfad400700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcfa80ceec0 0x7fcfa81f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfade00700 1 -- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcfa815d640 msgr2=0x7fcfa80a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfade00700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcfa815d640 0x7fcfa80a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfade00700 1 -- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcfa80ceec0 msgr2=0x7fcfa81f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfade00700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcfa80ceec0 0x7fcfa81f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfade00700 1 -- 172.21.15.12:0/2630445574 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf94009550 con 0x7fcfa8161800 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfade00700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa8161800 0x7fcfa80c9210 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000cef0 tx=0x7fcfa000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:31.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfa5600700 1 -- 172.21.15.12:0/2630445574 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa000ad00 con 0x7fcfa8161800 2024-03-29T11:31:31.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfa5600700 1 -- 172.21.15.12:0/2630445574 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa000ae60 con 0x7fcfa8161800 2024-03-29T11:31:31.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfa5600700 1 -- 172.21.15.12:0/2630445574 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0015680 con 0x7fcfa8161800 2024-03-29T11:31:31.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfafe00700 1 -- 172.21.15.12:0/2630445574 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa814c5c0 con 0x7fcfa8161800 2024-03-29T11:31:31.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.979+0000 7fcfafe00700 1 -- 172.21.15.12:0/2630445574 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa814cb80 con 0x7fcfa8161800 2024-03-29T11:31:31.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.980+0000 7fcfafe00700 1 -- 172.21.15.12:0/2630445574 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa8067720 con 0x7fcfa8161800 2024-03-29T11:31:31.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.981+0000 7fcfa5600700 1 -- 172.21.15.12:0/2630445574 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcfa000d430 con 0x7fcfa8161800 2024-03-29T11:31:31.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.982+0000 7fcfa5600700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcf8806adb0 0x7fcf8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:31.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.982+0000 7fcfa5600700 1 -- 172.21.15.12:0/2630445574 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fcfa0089b00 con 0x7fcfa8161800 2024-03-29T11:31:31.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.983+0000 7fcfad400700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcf8806adb0 0x7fcf8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:31.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.983+0000 7fcfad400700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcf8806adb0 0x7fcf8806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c007fd0 tx=0x7fcf9c009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:31.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:31.987+0000 7fcfa5600700 1 -- 172.21.15.12:0/2630445574 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcfa008be60 con 0x7fcfa8161800 2024-03-29T11:31:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.195+0000 7fcfafe00700 1 -- 172.21.15.12:0/2630445574 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcfa80b64d0 con 0x7fcf8806adb0 2024-03-29T11:31:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.204+0000 7fcfa5600700 1 -- 172.21.15.12:0/2630445574 <== mgr.10459 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289840 (secure 0 0 0) 0x7fcf9c001500 con 0x7fcf8806adb0 2024-03-29T11:31:32.204 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 -- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcf8806adb0 msgr2=0x7fcf8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcf8806adb0 0x7fcf8806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c007fd0 tx=0x7fcf9c009040 comp rx=0 tx=0).stop 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 -- 172.21.15.12:0/2630445574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa8161800 msgr2=0x7fcfa80c9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa8161800 0x7fcfa80c9210 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000cef0 tx=0x7fcfa000cf20 comp rx=0 tx=0).stop 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 -- 172.21.15.12:0/2630445574 shutdown_connections 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcf8806adb0 0x7fcf8806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa8161800 0x7fcfa80c9210 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcfa80ceec0 0x7fcfa81f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 --2- 172.21.15.12:0/2630445574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcfa815d640 0x7fcfa80a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 -- 172.21.15.12:0/2630445574 >> 172.21.15.12:0/2630445574 conn(0x7fcfa808aff0 msgr2=0x7fcfa80b7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 -- 172.21.15.12:0/2630445574 shutdown_connections 2024-03-29T11:31:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.208+0000 7fcfafe00700 1 -- 172.21.15.12:0/2630445574 wait complete. 2024-03-29T11:31:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:31:32.222 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-29T11:31:31.167937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":655,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":433732,"kb_used_data":23236,"kb_used_omap":12,"kb_used_meta":410355,"kb_avail":754540988,"statfs":{"total":773094113280,"available":772649971712,"internally_reserved":0,"allocated":23793664,"data_stored":12185923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":420204089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.873721"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854714+0000","last_change":"2024-03-29T11:31:26.850587+0000","last_active":"2024-03-29T11:31:26.854714+0000","last_peered":"2024-03-29T11:31:26.854714+0000","last_clean":"2024-03-29T11:31:26.854714+0000","last_became_active":"2024-03-29T11:31:26.850150+0000","last_became_peered":"2024-03-29T11:31:26.850150+0000","last_unstale":"2024-03-29T11:31:26.854714+0000","last_undegraded":"2024-03-29T11:31:26.854714+0000","last_fullsized":"2024-03-29T11:31:26.854714+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:49.204368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854837+0000","last_change":"2024-03-29T11:31:26.848512+0000","last_active":"2024-03-29T11:31:26.854837+0000","last_peered":"2024-03-29T11:31:26.854837+0000","last_clean":"2024-03-29T11:31:26.854837+0000","last_became_active":"2024-03-29T11:31:26.846361+0000","last_became_peered":"2024-03-29T11:31:26.846361+0000","last_unstale":"2024-03-29T11:31:26.854837+0000","last_undegraded":"2024-03-29T11:31:26.854837+0000","last_fullsized":"2024-03-29T11:31:26.854837+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:49:25.037675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853062+0000","last_change":"2024-03-29T11:31:26.851998+0000","last_active":"2024-03-29T11:31:26.853062+0000","last_peered":"2024-03-29T11:31:26.853062+0000","last_clean":"2024-03-29T11:31:26.853062+0000","last_became_active":"2024-03-29T11:31:26.850781+0000","last_became_peered":"2024-03-29T11:31:26.850781+0000","last_unstale":"2024-03-29T11:31:26.853062+0000","last_undegraded":"2024-03-29T11:31:26.853062+0000","last_fullsized":"2024-03-29T11:31:26.853062+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:57.876055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854117+0000","last_change":"2024-03-29T11:31:26.852752+0000","last_active":"2024-03-29T11:31:26.854117+0000","last_peered":"2024-03-29T11:31:26.854117+0000","last_clean":"2024-03-29T11:31:26.854117+0000","last_became_active":"2024-03-29T11:31:26.852491+0000","last_became_peered":"2024-03-29T11:31:26.852491+0000","last_unstale":"2024-03-29T11:31:26.854117+0000","last_undegraded":"2024-03-29T11:31:26.854117+0000","last_fullsized":"2024-03-29T11:31:26.854117+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:32.613551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845568+0000","last_change":"2024-03-29T11:31:26.841559+0000","last_active":"2024-03-29T11:31:26.845568+0000","last_peered":"2024-03-29T11:31:26.845568+0000","last_clean":"2024-03-29T11:31:26.845568+0000","last_became_active":"2024-03-29T11:31:26.841323+0000","last_became_peered":"2024-03-29T11:31:26.841323+0000","last_unstale":"2024-03-29T11:31:26.845568+0000","last_undegraded":"2024-03-29T11:31:26.845568+0000","last_fullsized":"2024-03-29T11:31:26.845568+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:40:17.980944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.840870+0000","last_change":"2024-03-29T11:31:26.836146+0000","last_active":"2024-03-29T11:31:26.840870+0000","last_peered":"2024-03-29T11:31:26.840870+0000","last_clean":"2024-03-29T11:31:26.840870+0000","last_became_active":"2024-03-29T11:31:26.835185+0000","last_became_peered":"2024-03-29T11:31:26.835185+0000","last_unstale":"2024-03-29T11:31:26.840870+0000","last_undegraded":"2024-03-29T11:31:26.840870+0000","last_fullsized":"2024-03-29T11:31:26.840870+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:15:49.012984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.846028+0000","last_change":"2024-03-29T11:31:26.836892+0000","last_active":"2024-03-29T11:31:26.846028+0000","last_peered":"2024-03-29T11:31:26.846028+0000","last_clean":"2024-03-29T11:31:26.846028+0000","last_became_active":"2024-03-29T11:31:26.832344+0000","last_became_peered":"2024-03-29T11:31:26.832344+0000","last_unstale":"2024-03-29T11:31:26.846028+0000","last_undegraded":"2024-03-29T11:31:26.846028+0000","last_fullsized":"2024-03-29T11:31:26.846028+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:27:29.490312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.840636+0000","last_change":"2024-03-29T11:31:26.833794+0000","last_active":"2024-03-29T11:31:26.840636+0000","last_peered":"2024-03-29T11:31:26.840636+0000","last_clean":"2024-03-29T11:31:26.840636+0000","last_became_active":"2024-03-29T11:31:26.833440+0000","last_became_peered":"2024-03-29T11:31:26.833440+0000","last_unstale":"2024-03-29T11:31:26.840636+0000","last_undegraded":"2024-03-29T11:31:26.840636+0000","last_fullsized":"2024-03-29T11:31:26.840636+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:07.336827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854459+0000","last_change":"2024-03-29T11:31:26.851850+0000","last_active":"2024-03-29T11:31:26.854459+0000","last_peered":"2024-03-29T11:31:26.854459+0000","last_clean":"2024-03-29T11:31:26.854459+0000","last_became_active":"2024-03-29T11:31:26.848888+0000","last_became_peered":"2024-03-29T11:31:26.848888+0000","last_unstale":"2024-03-29T11:31:26.854459+0000","last_undegraded":"2024-03-29T11:31:26.854459+0000","last_fullsized":"2024-03-29T11:31:26.854459+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:26:36.183185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855076+0000","last_change":"2024-03-29T11:31:26.853942+0000","last_active":"2024-03-29T11:31:26.855076+0000","last_peered":"2024-03-29T11:31:26.855076+0000","last_clean":"2024-03-29T11:31:26.855076+0000","last_became_active":"2024-03-29T11:31:26.853648+0000","last_became_peered":"2024-03-29T11:31:26.853648+0000","last_unstale":"2024-03-29T11:31:26.855076+0000","last_undegraded":"2024-03-29T11:31:26.855076+0000","last_fullsized":"2024-03-29T11:31:26.855076+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:24.230359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.848145+0000","last_change":"2024-03-29T11:31:26.841262+0000","last_active":"2024-03-29T11:31:26.848145+0000","last_peered":"2024-03-29T11:31:26.848145+0000","last_clean":"2024-03-29T11:31:26.848145+0000","last_became_active":"2024-03-29T11:31:26.841004+0000","last_became_peered":"2024-03-29T11:31:26.841004+0000","last_unstale":"2024-03-29T11:31:26.848145+0000","last_undegraded":"2024-03-29T11:31:26.848145+0000","last_fullsized":"2024-03-29T11:31:26.848145+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:54.107881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853190+0000","last_change":"2024-03-29T11:31:26.848052+0000","last_active":"2024-03-29T11:31:26.853190+0000","last_peered":"2024-03-29T11:31:26.853190+0000","last_clean":"2024-03-29T11:31:26.853190+0000","last_became_active":"2024-03-29T11:31:26.847789+0000","last_became_peered":"2024-03-29T11:31:26.847789+0000","last_unstale":"2024-03-29T11:31:26.853190+0000","last_undegraded":"2024-03-29T11:31:26.853190+0000","last_fullsized":"2024-03-29T11:31:26.853190+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:23:23.236057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853771+0000","last_change":"2024-03-29T11:31:26.848377+0000","last_active":"2024-03-29T11:31:26.853771+0000","last_peered":"2024-03-29T11:31:26.853771+0000","last_clean":"2024-03-29T11:31:26.853771+0000","last_became_active":"2024-03-29T11:31:26.847564+0000","last_became_peered":"2024-03-29T11:31:26.847564+0000","last_unstale":"2024-03-29T11:31:26.853771+0000","last_undegraded":"2024-03-29T11:31:26.853771+0000","last_fullsized":"2024-03-29T11:31:26.853771+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:40:17.980944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853815+0000","last_change":"2024-03-29T11:31:26.848452+0000","last_active":"2024-03-29T11:31:26.853815+0000","last_peered":"2024-03-29T11:31:26.853815+0000","last_clean":"2024-03-29T11:31:26.853815+0000","last_became_active":"2024-03-29T11:31:26.846605+0000","last_became_peered":"2024-03-29T11:31:26.846605+0000","last_unstale":"2024-03-29T11:31:26.853815+0000","last_undegraded":"2024-03-29T11:31:26.853815+0000","last_fullsized":"2024-03-29T11:31:26.853815+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:57.163908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852282+0000","last_change":"2024-03-29T11:31:26.843972+0000","last_active":"2024-03-29T11:31:26.852282+0000","last_peered":"2024-03-29T11:31:26.852282+0000","last_clean":"2024-03-29T11:31:26.852282+0000","last_became_active":"2024-03-29T11:31:26.843713+0000","last_became_peered":"2024-03-29T11:31:26.843713+0000","last_unstale":"2024-03-29T11:31:26.852282+0000","last_undegraded":"2024-03-29T11:31:26.852282+0000","last_fullsized":"2024-03-29T11:31:26.852282+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847412+0000","last_change":"2024-03-29T11:31:26.843040+0000","last_active":"2024-03-29T11:31:26.847412+0000","last_peered":"2024-03-29T11:31:26.847412+0000","last_clean":"2024-03-29T11:31:26.847412+0000","last_became_active":"2024-03-29T11:31:26.841805+0000","last_became_peered":"2024-03-29T11:31:26.841805+0000","last_unstale":"2024-03-29T11:31:26.847412+0000","last_undegraded":"2024-03-29T11:31:26.847412+0000","last_fullsized":"2024-03-29T11:31:26.847412+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:05.909043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854695+0000","last_change":"2024-03-29T11:31:26.850388+0000","last_active":"2024-03-29T11:31:26.854695+0000","last_peered":"2024-03-29T11:31:26.854695+0000","last_clean":"2024-03-29T11:31:26.854695+0000","last_became_active":"2024-03-29T11:31:26.849826+0000","last_became_peered":"2024-03-29T11:31:26.849826+0000","last_unstale":"2024-03-29T11:31:26.854695+0000","last_undegraded":"2024-03-29T11:31:26.854695+0000","last_fullsized":"2024-03-29T11:31:26.854695+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:55:45.796741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845675+0000","last_change":"2024-03-29T11:31:26.840048+0000","last_active":"2024-03-29T11:31:26.845675+0000","last_peered":"2024-03-29T11:31:26.845675+0000","last_clean":"2024-03-29T11:31:26.845675+0000","last_became_active":"2024-03-29T11:31:26.838725+0000","last_became_peered":"2024-03-29T11:31:26.838725+0000","last_unstale":"2024-03-29T11:31:26.845675+0000","last_undegraded":"2024-03-29T11:31:26.845675+0000","last_fullsized":"2024-03-29T11:31:26.845675+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:09.172222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855793+0000","last_change":"2024-03-29T11:31:26.854890+0000","last_active":"2024-03-29T11:31:26.855793+0000","last_peered":"2024-03-29T11:31:26.855793+0000","last_clean":"2024-03-29T11:31:26.855793+0000","last_became_active":"2024-03-29T11:31:26.854625+0000","last_became_peered":"2024-03-29T11:31:26.854625+0000","last_unstale":"2024-03-29T11:31:26.855793+0000","last_undegraded":"2024-03-29T11:31:26.855793+0000","last_fullsized":"2024-03-29T11:31:26.855793+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.831388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845779+0000","last_change":"2024-03-29T11:31:26.843448+0000","last_active":"2024-03-29T11:31:26.845779+0000","last_peered":"2024-03-29T11:31:26.845779+0000","last_clean":"2024-03-29T11:31:26.845779+0000","last_became_active":"2024-03-29T11:31:26.843183+0000","last_became_peered":"2024-03-29T11:31:26.843183+0000","last_unstale":"2024-03-29T11:31:26.845779+0000","last_undegraded":"2024-03-29T11:31:26.845779+0000","last_fullsized":"2024-03-29T11:31:26.845779+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:23:23.236057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.857836+0000","last_change":"2024-03-29T11:31:26.856449+0000","last_active":"2024-03-29T11:31:26.857836+0000","last_peered":"2024-03-29T11:31:26.857836+0000","last_clean":"2024-03-29T11:31:26.857836+0000","last_became_active":"2024-03-29T11:31:26.856166+0000","last_became_peered":"2024-03-29T11:31:26.856166+0000","last_unstale":"2024-03-29T11:31:26.857836+0000","last_undegraded":"2024-03-29T11:31:26.857836+0000","last_fullsized":"2024-03-29T11:31:26.857836+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:27.122635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.841326+0000","last_change":"2024-03-29T11:31:26.834482+0000","last_active":"2024-03-29T11:31:26.841326+0000","last_peered":"2024-03-29T11:31:26.841326+0000","last_clean":"2024-03-29T11:31:26.841326+0000","last_became_active":"2024-03-29T11:31:26.833118+0000","last_became_peered":"2024-03-29T11:31:26.833118+0000","last_unstale":"2024-03-29T11:31:26.841326+0000","last_undegraded":"2024-03-29T11:31:26.841326+0000","last_fullsized":"2024-03-29T11:31:26.841326+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:42.629666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845477+0000","last_change":"2024-03-29T11:31:26.842715+0000","last_active":"2024-03-29T11:31:26.845477+0000","last_peered":"2024-03-29T11:31:26.845477+0000","last_clean":"2024-03-29T11:31:26.845477+0000","last_became_active":"2024-03-29T11:31:26.842354+0000","last_became_peered":"2024-03-29T11:31:26.842354+0000","last_unstale":"2024-03-29T11:31:26.845477+0000","last_undegraded":"2024-03-29T11:31:26.845477+0000","last_fullsized":"2024-03-29T11:31:26.845477+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:58.653895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.857509+0000","last_change":"2024-03-29T11:31:26.855506+0000","last_active":"2024-03-29T11:31:26.857509+0000","last_peered":"2024-03-29T11:31:26.857509+0000","last_clean":"2024-03-29T11:31:26.857509+0000","last_became_active":"2024-03-29T11:31:26.852090+0000","last_became_peered":"2024-03-29T11:31:26.852090+0000","last_unstale":"2024-03-29T11:31:26.857509+0000","last_undegraded":"2024-03-29T11:31:26.857509+0000","last_fullsized":"2024-03-29T11:31:26.857509+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:09.172222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854104+0000","last_change":"2024-03-29T11:31:26.847195+0000","last_active":"2024-03-29T11:31:26.854104+0000","last_peered":"2024-03-29T11:31:26.854104+0000","last_clean":"2024-03-29T11:31:26.854104+0000","last_became_active":"2024-03-29T11:31:26.846876+0000","last_became_peered":"2024-03-29T11:31:26.846876+0000","last_unstale":"2024-03-29T11:31:26.854104+0000","last_undegraded":"2024-03-29T11:31:26.854104+0000","last_fullsized":"2024-03-29T11:31:26.854104+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:41.350201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852325+0000","last_change":"2024-03-29T11:31:26.843275+0000","last_active":"2024-03-29T11:31:26.852325+0000","last_peered":"2024-03-29T11:31:26.852325+0000","last_clean":"2024-03-29T11:31:26.852325+0000","last_became_active":"2024-03-29T11:31:26.842588+0000","last_became_peered":"2024-03-29T11:31:26.842588+0000","last_unstale":"2024-03-29T11:31:26.852325+0000","last_undegraded":"2024-03-29T11:31:26.852325+0000","last_fullsized":"2024-03-29T11:31:26.852325+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:57.876055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854434+0000","last_change":"2024-03-29T11:31:26.852411+0000","last_active":"2024-03-29T11:31:26.854434+0000","last_peered":"2024-03-29T11:31:26.854434+0000","last_clean":"2024-03-29T11:31:26.854434+0000","last_became_active":"2024-03-29T11:31:26.852100+0000","last_became_peered":"2024-03-29T11:31:26.852100+0000","last_unstale":"2024-03-29T11:31:26.854434+0000","last_undegraded":"2024-03-29T11:31:26.854434+0000","last_fullsized":"2024-03-29T11:31:26.854434+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:15:49.012984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847430+0000","last_change":"2024-03-29T11:31:26.838393+0000","last_active":"2024-03-29T11:31:26.847430+0000","last_peered":"2024-03-29T11:31:26.847430+0000","last_clean":"2024-03-29T11:31:26.847430+0000","last_became_active":"2024-03-29T11:31:26.838003+0000","last_became_peered":"2024-03-29T11:31:26.838003+0000","last_unstale":"2024-03-29T11:31:26.847430+0000","last_undegraded":"2024-03-29T11:31:26.847430+0000","last_fullsized":"2024-03-29T11:31:26.847430+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:47.195933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853791+0000","last_change":"2024-03-29T11:31:26.848961+0000","last_active":"2024-03-29T11:31:26.853791+0000","last_peered":"2024-03-29T11:31:26.853791+0000","last_clean":"2024-03-29T11:31:26.853791+0000","last_became_active":"2024-03-29T11:31:26.848637+0000","last_became_peered":"2024-03-29T11:31:26.848637+0000","last_unstale":"2024-03-29T11:31:26.853791+0000","last_undegraded":"2024-03-29T11:31:26.853791+0000","last_fullsized":"2024-03-29T11:31:26.853791+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:31:51.171001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.846678+0000","last_change":"2024-03-29T11:31:26.842077+0000","last_active":"2024-03-29T11:31:26.846678+0000","last_peered":"2024-03-29T11:31:26.846678+0000","last_clean":"2024-03-29T11:31:26.846678+0000","last_became_active":"2024-03-29T11:31:26.841818+0000","last_became_peered":"2024-03-29T11:31:26.841818+0000","last_unstale":"2024-03-29T11:31:26.846678+0000","last_undegraded":"2024-03-29T11:31:26.846678+0000","last_fullsized":"2024-03-29T11:31:26.846678+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:42:57.193431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.851202+0000","last_change":"2024-03-29T11:31:26.847737+0000","last_active":"2024-03-29T11:31:26.851202+0000","last_peered":"2024-03-29T11:31:26.851202+0000","last_clean":"2024-03-29T11:31:26.851202+0000","last_became_active":"2024-03-29T11:31:26.847473+0000","last_became_peered":"2024-03-29T11:31:26.847473+0000","last_unstale":"2024-03-29T11:31:26.851202+0000","last_undegraded":"2024-03-29T11:31:26.851202+0000","last_fullsized":"2024-03-29T11:31:26.851202+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:36:05.541971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847682+0000","last_change":"2024-03-29T11:31:26.839138+0000","last_active":"2024-03-29T11:31:26.847682+0000","last_peered":"2024-03-29T11:31:26.847682+0000","last_clean":"2024-03-29T11:31:26.847682+0000","last_became_active":"2024-03-29T11:31:26.838904+0000","last_became_peered":"2024-03-29T11:31:26.838904+0000","last_unstale":"2024-03-29T11:31:26.847682+0000","last_undegraded":"2024-03-29T11:31:26.847682+0000","last_fullsized":"2024-03-29T11:31:26.847682+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:54.107881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.848635+0000","last_change":"2024-03-29T11:31:26.844474+0000","last_active":"2024-03-29T11:31:26.848635+0000","last_peered":"2024-03-29T11:31:26.848635+0000","last_clean":"2024-03-29T11:31:26.848635+0000","last_became_active":"2024-03-29T11:31:26.844195+0000","last_became_peered":"2024-03-29T11:31:26.844195+0000","last_unstale":"2024-03-29T11:31:26.848635+0000","last_undegraded":"2024-03-29T11:31:26.848635+0000","last_fullsized":"2024-03-29T11:31:26.848635+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:38:37.668282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"226'4","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855147+0000","last_change":"2024-03-29T11:31:26.315269+0000","last_active":"2024-03-29T11:31:26.855147+0000","last_peered":"2024-03-29T11:31:26.855147+0000","last_clean":"2024-03-29T11:31:26.855147+0000","last_became_active":"2024-03-29T11:31:26.314956+0000","last_became_peered":"2024-03-29T11:31:26.314956+0000","last_unstale":"2024-03-29T11:31:26.855147+0000","last_undegraded":"2024-03-29T11:31:26.855147+0000","last_fullsized":"2024-03-29T11:31:26.855147+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:18.302941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847446+0000","last_change":"2024-03-29T11:31:26.843531+0000","last_active":"2024-03-29T11:31:26.847446+0000","last_peered":"2024-03-29T11:31:26.847446+0000","last_clean":"2024-03-29T11:31:26.847446+0000","last_became_active":"2024-03-29T11:31:26.843164+0000","last_became_peered":"2024-03-29T11:31:26.843164+0000","last_unstale":"2024-03-29T11:31:26.847446+0000","last_undegraded":"2024-03-29T11:31:26.847446+0000","last_fullsized":"2024-03-29T11:31:26.847446+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:29:23.515574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855908+0000","last_change":"2024-03-29T11:31:26.320463+0000","last_active":"2024-03-29T11:31:26.855908+0000","last_peered":"2024-03-29T11:31:26.855908+0000","last_clean":"2024-03-29T11:31:26.855908+0000","last_became_active":"2024-03-29T11:31:26.319713+0000","last_became_peered":"2024-03-29T11:31:26.319713+0000","last_unstale":"2024-03-29T11:31:26.855908+0000","last_undegraded":"2024-03-29T11:31:26.855908+0000","last_fullsized":"2024-03-29T11:31:26.855908+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:32:47.003355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854970+0000","last_change":"2024-03-29T11:31:26.850407+0000","last_active":"2024-03-29T11:31:26.854970+0000","last_peered":"2024-03-29T11:31:26.854970+0000","last_clean":"2024-03-29T11:31:26.854970+0000","last_became_active":"2024-03-29T11:31:26.849457+0000","last_became_peered":"2024-03-29T11:31:26.849457+0000","last_unstale":"2024-03-29T11:31:26.854970+0000","last_undegraded":"2024-03-29T11:31:26.854970+0000","last_fullsized":"2024-03-29T11:31:26.854970+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:38:50.961636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854522+0000","last_change":"2024-03-29T11:31:26.319707+0000","last_active":"2024-03-29T11:31:26.854522+0000","last_peered":"2024-03-29T11:31:26.854522+0000","last_clean":"2024-03-29T11:31:26.854522+0000","last_became_active":"2024-03-29T11:31:26.316121+0000","last_became_peered":"2024-03-29T11:31:26.316121+0000","last_unstale":"2024-03-29T11:31:26.854522+0000","last_undegraded":"2024-03-29T11:31:26.854522+0000","last_fullsized":"2024-03-29T11:31:26.854522+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:12:51.039260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845866+0000","last_change":"2024-03-29T11:31:26.840358+0000","last_active":"2024-03-29T11:31:26.845866+0000","last_peered":"2024-03-29T11:31:26.845866+0000","last_clean":"2024-03-29T11:31:26.845866+0000","last_became_active":"2024-03-29T11:31:26.838477+0000","last_became_peered":"2024-03-29T11:31:26.838477+0000","last_unstale":"2024-03-29T11:31:26.845866+0000","last_undegraded":"2024-03-29T11:31:26.845866+0000","last_fullsized":"2024-03-29T11:31:26.845866+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:27.122635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854050+0000","last_change":"2024-03-29T11:31:26.321235+0000","last_active":"2024-03-29T11:31:26.854050+0000","last_peered":"2024-03-29T11:31:26.854050+0000","last_clean":"2024-03-29T11:31:26.854050+0000","last_became_active":"2024-03-29T11:31:26.320454+0000","last_became_peered":"2024-03-29T11:31:26.320454+0000","last_unstale":"2024-03-29T11:31:26.854050+0000","last_undegraded":"2024-03-29T11:31:26.854050+0000","last_fullsized":"2024-03-29T11:31:26.854050+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:27.238049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.856497+0000","last_change":"2024-03-29T11:31:26.853179+0000","last_active":"2024-03-29T11:31:26.856497+0000","last_peered":"2024-03-29T11:31:26.856497+0000","last_clean":"2024-03-29T11:31:26.856497+0000","last_became_active":"2024-03-29T11:31:26.848186+0000","last_became_peered":"2024-03-29T11:31:26.848186+0000","last_unstale":"2024-03-29T11:31:26.856497+0000","last_undegraded":"2024-03-29T11:31:26.856497+0000","last_fullsized":"2024-03-29T11:31:26.856497+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:35:08.770095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.851789+0000","last_change":"2024-03-29T11:31:26.311383+0000","last_active":"2024-03-29T11:31:26.851789+0000","last_peered":"2024-03-29T11:31:26.851789+0000","last_clean":"2024-03-29T11:31:26.851789+0000","last_became_active":"2024-03-29T11:31:26.310630+0000","last_became_peered":"2024-03-29T11:31:26.310630+0000","last_unstale":"2024-03-29T11:31:26.851789+0000","last_undegraded":"2024-03-29T11:31:26.851789+0000","last_fullsized":"2024-03-29T11:31:26.851789+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:51:16.023346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852437+0000","last_change":"2024-03-29T11:31:26.850244+0000","last_active":"2024-03-29T11:31:26.852437+0000","last_peered":"2024-03-29T11:31:26.852437+0000","last_clean":"2024-03-29T11:31:26.852437+0000","last_became_active":"2024-03-29T11:31:26.849736+0000","last_became_peered":"2024-03-29T11:31:26.849736+0000","last_unstale":"2024-03-29T11:31:26.852437+0000","last_undegraded":"2024-03-29T11:31:26.852437+0000","last_fullsized":"2024-03-29T11:31:26.852437+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:57.906301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.831185+0000","last_change":"2024-03-29T11:31:26.310235+0000","last_active":"2024-03-29T11:31:26.831185+0000","last_peered":"2024-03-29T11:31:26.831185+0000","last_clean":"2024-03-29T11:31:26.831185+0000","last_became_active":"2024-03-29T11:31:26.309932+0000","last_became_peered":"2024-03-29T11:31:26.309932+0000","last_unstale":"2024-03-29T11:31:26.831185+0000","last_undegraded":"2024-03-29T11:31:26.831185+0000","last_fullsized":"2024-03-29T11:31:26.831185+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:31:30.532498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855515+0000","last_change":"2024-03-29T11:31:26.847903+0000","last_active":"2024-03-29T11:31:26.855515+0000","last_peered":"2024-03-29T11:31:26.855515+0000","last_clean":"2024-03-29T11:31:26.855515+0000","last_became_active":"2024-03-29T11:31:26.846171+0000","last_became_peered":"2024-03-29T11:31:26.846171+0000","last_unstale":"2024-03-29T11:31:26.855515+0000","last_undegraded":"2024-03-29T11:31:26.855515+0000","last_fullsized":"2024-03-29T11:31:26.855515+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:24.230359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.826484+0000","last_change":"2024-03-29T11:31:26.320631+0000","last_active":"2024-03-29T11:31:26.826484+0000","last_peered":"2024-03-29T11:31:26.826484+0000","last_clean":"2024-03-29T11:31:26.826484+0000","last_became_active":"2024-03-29T11:31:26.320350+0000","last_became_peered":"2024-03-29T11:31:26.320350+0000","last_unstale":"2024-03-29T11:31:26.826484+0000","last_undegraded":"2024-03-29T11:31:26.826484+0000","last_fullsized":"2024-03-29T11:31:26.826484+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:57.806178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.857532+0000","last_change":"2024-03-29T11:31:26.855805+0000","last_active":"2024-03-29T11:31:26.857532+0000","last_peered":"2024-03-29T11:31:26.857532+0000","last_clean":"2024-03-29T11:31:26.857532+0000","last_became_active":"2024-03-29T11:31:26.854618+0000","last_became_peered":"2024-03-29T11:31:26.854618+0000","last_unstale":"2024-03-29T11:31:26.857532+0000","last_undegraded":"2024-03-29T11:31:26.857532+0000","last_fullsized":"2024-03-29T11:31:26.857532+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.831388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.824673+0000","last_change":"2024-03-29T11:31:26.317297+0000","last_active":"2024-03-29T11:31:26.824673+0000","last_peered":"2024-03-29T11:31:26.824673+0000","last_clean":"2024-03-29T11:31:26.824673+0000","last_became_active":"2024-03-29T11:31:26.316956+0000","last_became_peered":"2024-03-29T11:31:26.316956+0000","last_unstale":"2024-03-29T11:31:26.824673+0000","last_undegraded":"2024-03-29T11:31:26.824673+0000","last_fullsized":"2024-03-29T11:31:26.824673+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:18.302941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.849098+0000","last_change":"2024-03-29T11:31:26.846344+0000","last_active":"2024-03-29T11:31:26.849098+0000","last_peered":"2024-03-29T11:31:26.849098+0000","last_clean":"2024-03-29T11:31:26.849098+0000","last_became_active":"2024-03-29T11:31:26.846066+0000","last_became_peered":"2024-03-29T11:31:26.846066+0000","last_unstale":"2024-03-29T11:31:26.849098+0000","last_undegraded":"2024-03-29T11:31:26.849098+0000","last_fullsized":"2024-03-29T11:31:26.849098+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:57.163908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.826341+0000","last_change":"2024-03-29T11:31:26.316324+0000","last_active":"2024-03-29T11:31:26.826341+0000","last_peered":"2024-03-29T11:31:26.826341+0000","last_clean":"2024-03-29T11:31:26.826341+0000","last_became_active":"2024-03-29T11:31:26.316066+0000","last_became_peered":"2024-03-29T11:31:26.316066+0000","last_unstale":"2024-03-29T11:31:26.826341+0000","last_undegraded":"2024-03-29T11:31:26.826341+0000","last_fullsized":"2024-03-29T11:31:26.826341+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:24:19.700336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.857974+0000","last_change":"2024-03-29T11:31:26.857148+0000","last_active":"2024-03-29T11:31:26.857974+0000","last_peered":"2024-03-29T11:31:26.857974+0000","last_clean":"2024-03-29T11:31:26.857974+0000","last_became_active":"2024-03-29T11:31:26.856870+0000","last_became_peered":"2024-03-29T11:31:26.856870+0000","last_unstale":"2024-03-29T11:31:26.857974+0000","last_undegraded":"2024-03-29T11:31:26.857974+0000","last_fullsized":"2024-03-29T11:31:26.857974+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"226'3","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854630+0000","last_change":"2024-03-29T11:31:26.320127+0000","last_active":"2024-03-29T11:31:26.854630+0000","last_peered":"2024-03-29T11:31:26.854630+0000","last_clean":"2024-03-29T11:31:26.854630+0000","last_became_active":"2024-03-29T11:31:26.319346+0000","last_became_peered":"2024-03-29T11:31:26.319346+0000","last_unstale":"2024-03-29T11:31:26.854630+0000","last_undegraded":"2024-03-29T11:31:26.854630+0000","last_fullsized":"2024-03-29T11:31:26.854630+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:32:50.637683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853294+0000","last_change":"2024-03-29T11:31:26.844802+0000","last_active":"2024-03-29T11:31:26.853294+0000","last_peered":"2024-03-29T11:31:26.853294+0000","last_clean":"2024-03-29T11:31:26.853294+0000","last_became_active":"2024-03-29T11:31:26.844538+0000","last_became_peered":"2024-03-29T11:31:26.844538+0000","last_unstale":"2024-03-29T11:31:26.853294+0000","last_undegraded":"2024-03-29T11:31:26.853294+0000","last_fullsized":"2024-03-29T11:31:26.853294+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:05.909043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"226'6","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.858524+0000","last_change":"2024-03-29T11:31:26.319966+0000","last_active":"2024-03-29T11:31:26.858524+0000","last_peered":"2024-03-29T11:31:26.858524+0000","last_clean":"2024-03-29T11:31:26.858524+0000","last_became_active":"2024-03-29T11:31:26.319700+0000","last_became_peered":"2024-03-29T11:31:26.319700+0000","last_unstale":"2024-03-29T11:31:26.858524+0000","last_undegraded":"2024-03-29T11:31:26.858524+0000","last_fullsized":"2024-03-29T11:31:26.858524+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:56.161261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.842254+0000","last_change":"2024-03-29T11:31:26.840312+0000","last_active":"2024-03-29T11:31:26.842254+0000","last_peered":"2024-03-29T11:31:26.842254+0000","last_clean":"2024-03-29T11:31:26.842254+0000","last_became_active":"2024-03-29T11:31:26.840020+0000","last_became_peered":"2024-03-29T11:31:26.840020+0000","last_unstale":"2024-03-29T11:31:26.842254+0000","last_undegraded":"2024-03-29T11:31:26.842254+0000","last_fullsized":"2024-03-29T11:31:26.842254+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:31:51.171001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852521+0000","last_change":"2024-03-29T11:31:26.313768+0000","last_active":"2024-03-29T11:31:26.852521+0000","last_peered":"2024-03-29T11:31:26.852521+0000","last_clean":"2024-03-29T11:31:26.852521+0000","last_became_active":"2024-03-29T11:31:26.313458+0000","last_became_peered":"2024-03-29T11:31:26.313458+0000","last_unstale":"2024-03-29T11:31:26.852521+0000","last_undegraded":"2024-03-29T11:31:26.852521+0000","last_fullsized":"2024-03-29T11:31:26.852521+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:06.334180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.834985+0000","last_change":"2024-03-29T11:31:26.833527+0000","last_active":"2024-03-29T11:31:26.834985+0000","last_peered":"2024-03-29T11:31:26.834985+0000","last_clean":"2024-03-29T11:31:26.834985+0000","last_became_active":"2024-03-29T11:31:26.833280+0000","last_became_peered":"2024-03-29T11:31:26.833280+0000","last_unstale":"2024-03-29T11:31:26.834985+0000","last_undegraded":"2024-03-29T11:31:26.834985+0000","last_fullsized":"2024-03-29T11:31:26.834985+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:35:08.770095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.827275+0000","last_change":"2024-03-29T11:31:26.315775+0000","last_active":"2024-03-29T11:31:26.827275+0000","last_peered":"2024-03-29T11:31:26.827275+0000","last_clean":"2024-03-29T11:31:26.827275+0000","last_became_active":"2024-03-29T11:31:26.315493+0000","last_became_peered":"2024-03-29T11:31:26.315493+0000","last_unstale":"2024-03-29T11:31:26.827275+0000","last_undegraded":"2024-03-29T11:31:26.827275+0000","last_fullsized":"2024-03-29T11:31:26.827275+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:32:47.003355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.849908+0000","last_change":"2024-03-29T11:31:26.846893+0000","last_active":"2024-03-29T11:31:26.849908+0000","last_peered":"2024-03-29T11:31:26.849908+0000","last_clean":"2024-03-29T11:31:26.849908+0000","last_became_active":"2024-03-29T11:31:26.845341+0000","last_became_peered":"2024-03-29T11:31:26.845341+0000","last_unstale":"2024-03-29T11:31:26.849908+0000","last_undegraded":"2024-03-29T11:31:26.849908+0000","last_fullsized":"2024-03-29T11:31:26.849908+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:28.240696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.828686+0000","last_change":"2024-03-29T11:31:26.314471+0000","last_active":"2024-03-29T11:31:26.828686+0000","last_peered":"2024-03-29T11:31:26.828686+0000","last_clean":"2024-03-29T11:31:26.828686+0000","last_became_active":"2024-03-29T11:31:26.314204+0000","last_became_peered":"2024-03-29T11:31:26.314204+0000","last_unstale":"2024-03-29T11:31:26.828686+0000","last_undegraded":"2024-03-29T11:31:26.828686+0000","last_fullsized":"2024-03-29T11:31:26.828686+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:49:13.676544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852263+0000","last_change":"2024-03-29T11:31:26.843461+0000","last_active":"2024-03-29T11:31:26.852263+0000","last_peered":"2024-03-29T11:31:26.852263+0000","last_clean":"2024-03-29T11:31:26.852263+0000","last_became_active":"2024-03-29T11:31:26.839816+0000","last_became_peered":"2024-03-29T11:31:26.839816+0000","last_unstale":"2024-03-29T11:31:26.852263+0000","last_undegraded":"2024-03-29T11:31:26.852263+0000","last_fullsized":"2024-03-29T11:31:26.852263+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:32.997118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"226'8","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.857280+0000","last_change":"2024-03-29T11:31:26.320392+0000","last_active":"2024-03-29T11:31:26.857280+0000","last_peered":"2024-03-29T11:31:26.857280+0000","last_clean":"2024-03-29T11:31:26.857280+0000","last_became_active":"2024-03-29T11:31:26.319139+0000","last_became_peered":"2024-03-29T11:31:26.319139+0000","last_unstale":"2024-03-29T11:31:26.857280+0000","last_undegraded":"2024-03-29T11:31:26.857280+0000","last_fullsized":"2024-03-29T11:31:26.857280+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:57.651247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.855756+0000","last_change":"2024-03-29T11:31:26.854468+0000","last_active":"2024-03-29T11:31:26.855756+0000","last_peered":"2024-03-29T11:31:26.855756+0000","last_clean":"2024-03-29T11:31:26.855756+0000","last_became_active":"2024-03-29T11:31:26.851644+0000","last_became_peered":"2024-03-29T11:31:26.851644+0000","last_unstale":"2024-03-29T11:31:26.855756+0000","last_undegraded":"2024-03-29T11:31:26.855756+0000","last_fullsized":"2024-03-29T11:31:26.855756+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:16.143243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.849979+0000","last_change":"2024-03-29T11:31:26.320455+0000","last_active":"2024-03-29T11:31:26.849979+0000","last_peered":"2024-03-29T11:31:26.849979+0000","last_clean":"2024-03-29T11:31:26.849979+0000","last_became_active":"2024-03-29T11:31:26.320175+0000","last_became_peered":"2024-03-29T11:31:26.320175+0000","last_unstale":"2024-03-29T11:31:26.849979+0000","last_undegraded":"2024-03-29T11:31:26.849979+0000","last_fullsized":"2024-03-29T11:31:26.849979+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:25:12.671618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.834947+0000","last_change":"2024-03-29T11:31:26.833432+0000","last_active":"2024-03-29T11:31:26.834947+0000","last_peered":"2024-03-29T11:31:26.834947+0000","last_clean":"2024-03-29T11:31:26.834947+0000","last_became_active":"2024-03-29T11:31:26.833078+0000","last_became_peered":"2024-03-29T11:31:26.833078+0000","last_unstale":"2024-03-29T11:31:26.834947+0000","last_undegraded":"2024-03-29T11:31:26.834947+0000","last_fullsized":"2024-03-29T11:31:26.834947+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:17.353377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854785+0000","last_change":"2024-03-29T11:31:26.317437+0000","last_active":"2024-03-29T11:31:26.854785+0000","last_peered":"2024-03-29T11:31:26.854785+0000","last_clean":"2024-03-29T11:31:26.854785+0000","last_became_active":"2024-03-29T11:31:26.317128+0000","last_became_peered":"2024-03-29T11:31:26.317128+0000","last_unstale":"2024-03-29T11:31:26.854785+0000","last_undegraded":"2024-03-29T11:31:26.854785+0000","last_fullsized":"2024-03-29T11:31:26.854785+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:57.806178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.838242+0000","last_change":"2024-03-29T11:31:26.833306+0000","last_active":"2024-03-29T11:31:26.838242+0000","last_peered":"2024-03-29T11:31:26.838242+0000","last_clean":"2024-03-29T11:31:26.838242+0000","last_became_active":"2024-03-29T11:31:26.833041+0000","last_became_peered":"2024-03-29T11:31:26.833041+0000","last_unstale":"2024-03-29T11:31:26.838242+0000","last_undegraded":"2024-03-29T11:31:26.838242+0000","last_fullsized":"2024-03-29T11:31:26.838242+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:15:11.283200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.848252+0000","last_change":"2024-03-29T11:31:26.319951+0000","last_active":"2024-03-29T11:31:26.848252+0000","last_peered":"2024-03-29T11:31:26.848252+0000","last_clean":"2024-03-29T11:31:26.848252+0000","last_became_active":"2024-03-29T11:31:26.318226+0000","last_became_peered":"2024-03-29T11:31:26.318226+0000","last_unstale":"2024-03-29T11:31:26.848252+0000","last_undegraded":"2024-03-29T11:31:26.848252+0000","last_fullsized":"2024-03-29T11:31:26.848252+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:22.562759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.851358+0000","last_change":"2024-03-29T11:31:26.848108+0000","last_active":"2024-03-29T11:31:26.851358+0000","last_peered":"2024-03-29T11:31:26.851358+0000","last_clean":"2024-03-29T11:31:26.851358+0000","last_became_active":"2024-03-29T11:31:26.847705+0000","last_became_peered":"2024-03-29T11:31:26.847705+0000","last_unstale":"2024-03-29T11:31:26.851358+0000","last_undegraded":"2024-03-29T11:31:26.851358+0000","last_fullsized":"2024-03-29T11:31:26.851358+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:53:43.020271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.820860+0000","last_change":"2024-03-29T11:31:26.313158+0000","last_active":"2024-03-29T11:31:26.820860+0000","last_peered":"2024-03-29T11:31:26.820860+0000","last_clean":"2024-03-29T11:31:26.820860+0000","last_became_active":"2024-03-29T11:31:26.311492+0000","last_became_peered":"2024-03-29T11:31:26.311492+0000","last_unstale":"2024-03-29T11:31:26.820860+0000","last_undegraded":"2024-03-29T11:31:26.820860+0000","last_fullsized":"2024-03-29T11:31:26.820860+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:40.347554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847445+0000","last_change":"2024-03-29T11:31:26.844232+0000","last_active":"2024-03-29T11:31:26.847445+0000","last_peered":"2024-03-29T11:31:26.847445+0000","last_clean":"2024-03-29T11:31:26.847445+0000","last_became_active":"2024-03-29T11:31:26.843977+0000","last_became_peered":"2024-03-29T11:31:26.843977+0000","last_unstale":"2024-03-29T11:31:26.847445+0000","last_undegraded":"2024-03-29T11:31:26.847445+0000","last_fullsized":"2024-03-29T11:31:26.847445+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.849162+0000","last_change":"2024-03-29T11:31:26.316435+0000","last_active":"2024-03-29T11:31:26.849162+0000","last_peered":"2024-03-29T11:31:26.849162+0000","last_clean":"2024-03-29T11:31:26.849162+0000","last_became_active":"2024-03-29T11:31:26.316124+0000","last_became_peered":"2024-03-29T11:31:26.316124+0000","last_unstale":"2024-03-29T11:31:26.849162+0000","last_undegraded":"2024-03-29T11:31:26.849162+0000","last_fullsized":"2024-03-29T11:31:26.849162+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:15:10.280553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847286+0000","last_change":"2024-03-29T11:31:26.836881+0000","last_active":"2024-03-29T11:31:26.847286+0000","last_peered":"2024-03-29T11:31:26.847286+0000","last_clean":"2024-03-29T11:31:26.847286+0000","last_became_active":"2024-03-29T11:31:26.836613+0000","last_became_peered":"2024-03-29T11:31:26.836613+0000","last_unstale":"2024-03-29T11:31:26.847286+0000","last_undegraded":"2024-03-29T11:31:26.847286+0000","last_fullsized":"2024-03-29T11:31:26.847286+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:23.565406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.830838+0000","last_change":"2024-03-29T11:31:26.312485+0000","last_active":"2024-03-29T11:31:26.830838+0000","last_peered":"2024-03-29T11:31:26.830838+0000","last_clean":"2024-03-29T11:31:26.830838+0000","last_became_active":"2024-03-29T11:31:26.311989+0000","last_became_peered":"2024-03-29T11:31:26.311989+0000","last_unstale":"2024-03-29T11:31:26.830838+0000","last_undegraded":"2024-03-29T11:31:26.830838+0000","last_fullsized":"2024-03-29T11:31:26.830838+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:42:56.190784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845507+0000","last_change":"2024-03-29T11:31:26.835978+0000","last_active":"2024-03-29T11:31:26.845507+0000","last_peered":"2024-03-29T11:31:26.845507+0000","last_clean":"2024-03-29T11:31:26.845507+0000","last_became_active":"2024-03-29T11:31:26.835400+0000","last_became_peered":"2024-03-29T11:31:26.835400+0000","last_unstale":"2024-03-29T11:31:26.845507+0000","last_undegraded":"2024-03-29T11:31:26.845507+0000","last_fullsized":"2024-03-29T11:31:26.845507+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:31.783000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.828017+0000","last_change":"2024-03-29T11:31:26.312962+0000","last_active":"2024-03-29T11:31:26.828017+0000","last_peered":"2024-03-29T11:31:26.828017+0000","last_clean":"2024-03-29T11:31:26.828017+0000","last_became_active":"2024-03-29T11:31:26.312668+0000","last_became_peered":"2024-03-29T11:31:26.312668+0000","last_unstale":"2024-03-29T11:31:26.828017+0000","last_undegraded":"2024-03-29T11:31:26.828017+0000","last_fullsized":"2024-03-29T11:31:26.828017+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:46.870101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.845496+0000","last_change":"2024-03-29T11:31:26.843420+0000","last_active":"2024-03-29T11:31:26.845496+0000","last_peered":"2024-03-29T11:31:26.845496+0000","last_clean":"2024-03-29T11:31:26.845496+0000","last_became_active":"2024-03-29T11:31:26.842734+0000","last_became_peered":"2024-03-29T11:31:26.842734+0000","last_unstale":"2024-03-29T11:31:26.845496+0000","last_undegraded":"2024-03-29T11:31:26.845496+0000","last_fullsized":"2024-03-29T11:31:26.845496+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:46.006140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.825409+0000","last_change":"2024-03-29T11:31:26.321619+0000","last_active":"2024-03-29T11:31:26.825409+0000","last_peered":"2024-03-29T11:31:26.825409+0000","last_clean":"2024-03-29T11:31:26.825409+0000","last_became_active":"2024-03-29T11:31:26.321349+0000","last_became_peered":"2024-03-29T11:31:26.321349+0000","last_unstale":"2024-03-29T11:31:26.825409+0000","last_undegraded":"2024-03-29T11:31:26.825409+0000","last_fullsized":"2024-03-29T11:31:26.825409+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:09.446187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854485+0000","last_change":"2024-03-29T11:31:26.851615+0000","last_active":"2024-03-29T11:31:26.854485+0000","last_peered":"2024-03-29T11:31:26.854485+0000","last_clean":"2024-03-29T11:31:26.854485+0000","last_became_active":"2024-03-29T11:31:26.851328+0000","last_became_peered":"2024-03-29T11:31:26.851328+0000","last_unstale":"2024-03-29T11:31:26.854485+0000","last_undegraded":"2024-03-29T11:31:26.854485+0000","last_fullsized":"2024-03-29T11:31:26.854485+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:57.906301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.849180+0000","last_change":"2024-03-29T11:31:26.310244+0000","last_active":"2024-03-29T11:31:26.849180+0000","last_peered":"2024-03-29T11:31:26.849180+0000","last_clean":"2024-03-29T11:31:26.849180+0000","last_became_active":"2024-03-29T11:31:26.309981+0000","last_became_peered":"2024-03-29T11:31:26.309981+0000","last_unstale":"2024-03-29T11:31:26.849180+0000","last_undegraded":"2024-03-29T11:31:26.849180+0000","last_fullsized":"2024-03-29T11:31:26.849180+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:16.350729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853300+0000","last_change":"2024-03-29T11:31:26.845919+0000","last_active":"2024-03-29T11:31:26.853300+0000","last_peered":"2024-03-29T11:31:26.853300+0000","last_clean":"2024-03-29T11:31:26.853300+0000","last_became_active":"2024-03-29T11:31:26.845630+0000","last_became_peered":"2024-03-29T11:31:26.845630+0000","last_unstale":"2024-03-29T11:31:26.853300+0000","last_undegraded":"2024-03-29T11:31:26.853300+0000","last_fullsized":"2024-03-29T11:31:26.853300+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:47.195933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.844537+0000","last_change":"2024-03-29T11:31:26.316463+0000","last_active":"2024-03-29T11:31:26.844537+0000","last_peered":"2024-03-29T11:31:26.844537+0000","last_clean":"2024-03-29T11:31:26.844537+0000","last_became_active":"2024-03-29T11:31:26.315640+0000","last_became_peered":"2024-03-29T11:31:26.315640+0000","last_unstale":"2024-03-29T11:31:26.844537+0000","last_undegraded":"2024-03-29T11:31:26.844537+0000","last_fullsized":"2024-03-29T11:31:26.844537+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:31.994471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.854449+0000","last_change":"2024-03-29T11:31:26.849336+0000","last_active":"2024-03-29T11:31:26.854449+0000","last_peered":"2024-03-29T11:31:26.854449+0000","last_clean":"2024-03-29T11:31:26.854449+0000","last_became_active":"2024-03-29T11:31:26.849042+0000","last_became_peered":"2024-03-29T11:31:26.849042+0000","last_unstale":"2024-03-29T11:31:26.854449+0000","last_undegraded":"2024-03-29T11:31:26.854449+0000","last_fullsized":"2024-03-29T11:31:26.854449+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:58.653895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.831304+0000","last_change":"2024-03-29T11:31:26.317291+0000","last_active":"2024-03-29T11:31:26.831304+0000","last_peered":"2024-03-29T11:31:26.831304+0000","last_clean":"2024-03-29T11:31:26.831304+0000","last_became_active":"2024-03-29T11:31:26.317003+0000","last_became_peered":"2024-03-29T11:31:26.317003+0000","last_unstale":"2024-03-29T11:31:26.831304+0000","last_undegraded":"2024-03-29T11:31:26.831304+0000","last_fullsized":"2024-03-29T11:31:26.831304+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:55:21.607497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847388+0000","last_change":"2024-03-29T11:31:26.844018+0000","last_active":"2024-03-29T11:31:26.847388+0000","last_peered":"2024-03-29T11:31:26.847388+0000","last_clean":"2024-03-29T11:31:26.847388+0000","last_became_active":"2024-03-29T11:31:26.843237+0000","last_became_peered":"2024-03-29T11:31:26.843237+0000","last_unstale":"2024-03-29T11:31:26.847388+0000","last_undegraded":"2024-03-29T11:31:26.847388+0000","last_fullsized":"2024-03-29T11:31:26.847388+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:12:52.041908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.821002+0000","last_change":"2024-03-29T11:31:26.311780+0000","last_active":"2024-03-29T11:31:26.821002+0000","last_peered":"2024-03-29T11:31:26.821002+0000","last_clean":"2024-03-29T11:31:26.821002+0000","last_became_active":"2024-03-29T11:31:26.311128+0000","last_became_peered":"2024-03-29T11:31:26.311128+0000","last_unstale":"2024-03-29T11:31:26.821002+0000","last_undegraded":"2024-03-29T11:31:26.821002+0000","last_fullsized":"2024-03-29T11:31:26.821002+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:46.870101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.853833+0000","last_change":"2024-03-29T11:31:26.848609+0000","last_active":"2024-03-29T11:31:26.853833+0000","last_peered":"2024-03-29T11:31:26.853833+0000","last_clean":"2024-03-29T11:31:26.853833+0000","last_became_active":"2024-03-29T11:31:26.848202+0000","last_became_peered":"2024-03-29T11:31:26.848202+0000","last_unstale":"2024-03-29T11:31:26.853833+0000","last_undegraded":"2024-03-29T11:31:26.853833+0000","last_fullsized":"2024-03-29T11:31:26.853833+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:28.240696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.838927+0000","last_change":"2024-03-29T11:31:26.320428+0000","last_active":"2024-03-29T11:31:26.838927+0000","last_peered":"2024-03-29T11:31:26.838927+0000","last_clean":"2024-03-29T11:31:26.838927+0000","last_became_active":"2024-03-29T11:31:26.320099+0000","last_became_peered":"2024-03-29T11:31:26.320099+0000","last_unstale":"2024-03-29T11:31:26.838927+0000","last_undegraded":"2024-03-29T11:31:26.838927+0000","last_fullsized":"2024-03-29T11:31:26.838927+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:56.873408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.843113+0000","last_change":"2024-03-29T11:31:26.838532+0000","last_active":"2024-03-29T11:31:26.843113+0000","last_peered":"2024-03-29T11:31:26.843113+0000","last_clean":"2024-03-29T11:31:26.843113+0000","last_became_active":"2024-03-29T11:31:26.838267+0000","last_became_peered":"2024-03-29T11:31:26.838267+0000","last_unstale":"2024-03-29T11:31:26.843113+0000","last_undegraded":"2024-03-29T11:31:26.843113+0000","last_fullsized":"2024-03-29T11:31:26.843113+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:17:41.239491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.825907+0000","last_change":"2024-03-29T11:31:26.322425+0000","last_active":"2024-03-29T11:31:26.825907+0000","last_peered":"2024-03-29T11:31:26.825907+0000","last_clean":"2024-03-29T11:31:26.825907+0000","last_became_active":"2024-03-29T11:31:26.321981+0000","last_became_peered":"2024-03-29T11:31:26.321981+0000","last_unstale":"2024-03-29T11:31:26.825907+0000","last_undegraded":"2024-03-29T11:31:26.825907+0000","last_fullsized":"2024-03-29T11:31:26.825907+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:36.434947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.848339+0000","last_change":"2024-03-29T11:31:26.842129+0000","last_active":"2024-03-29T11:31:26.848339+0000","last_peered":"2024-03-29T11:31:26.848339+0000","last_clean":"2024-03-29T11:31:26.848339+0000","last_became_active":"2024-03-29T11:31:26.841888+0000","last_became_peered":"2024-03-29T11:31:26.841888+0000","last_unstale":"2024-03-29T11:31:26.848339+0000","last_undegraded":"2024-03-29T11:31:26.848339+0000","last_fullsized":"2024-03-29T11:31:26.848339+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:32.997118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.827539+0000","last_change":"2024-03-29T11:31:26.310072+0000","last_active":"2024-03-29T11:31:26.827539+0000","last_peered":"2024-03-29T11:31:26.827539+0000","last_clean":"2024-03-29T11:31:26.827539+0000","last_became_active":"2024-03-29T11:31:26.309254+0000","last_became_peered":"2024-03-29T11:31:26.309254+0000","last_unstale":"2024-03-29T11:31:26.827539+0000","last_undegraded":"2024-03-29T11:31:26.827539+0000","last_fullsized":"2024-03-29T11:31:26.827539+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:00:36.402074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.847166+0000","last_change":"2024-03-29T11:31:26.843285+0000","last_active":"2024-03-29T11:31:26.847166+0000","last_peered":"2024-03-29T11:31:26.847166+0000","last_clean":"2024-03-29T11:31:26.847166+0000","last_became_active":"2024-03-29T11:31:26.841561+0000","last_became_peered":"2024-03-29T11:31:26.841561+0000","last_unstale":"2024-03-29T11:31:26.847166+0000","last_undegraded":"2024-03-29T11:31:26.847166+0000","last_fullsized":"2024-03-29T11:31:26.847166+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:47.872748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.836521+0000","last_change":"2024-03-29T11:31:26.318569+0000","last_active":"2024-03-29T11:31:26.836521+0000","last_peered":"2024-03-29T11:31:26.836521+0000","last_clean":"2024-03-29T11:31:26.836521+0000","last_became_active":"2024-03-29T11:31:26.318298+0000","last_became_peered":"2024-03-29T11:31:26.318298+0000","last_unstale":"2024-03-29T11:31:26.836521+0000","last_undegraded":"2024-03-29T11:31:26.836521+0000","last_fullsized":"2024-03-29T11:31:26.836521+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:05:48.527536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.852031+0000","last_change":"2024-03-29T11:31:26.844835+0000","last_active":"2024-03-29T11:31:26.852031+0000","last_peered":"2024-03-29T11:31:26.852031+0000","last_clean":"2024-03-29T11:31:26.852031+0000","last_became_active":"2024-03-29T11:31:26.844570+0000","last_became_peered":"2024-03-29T11:31:26.844570+0000","last_unstale":"2024-03-29T11:31:26.852031+0000","last_undegraded":"2024-03-29T11:31:26.852031+0000","last_fullsized":"2024-03-29T11:31:26.852031+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:53:43.020271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.850572+0000","last_change":"2024-03-29T11:31:26.318923+0000","last_active":"2024-03-29T11:31:26.850572+0000","last_peered":"2024-03-29T11:31:26.850572+0000","last_clean":"2024-03-29T11:31:26.850572+0000","last_became_active":"2024-03-29T11:31:26.318658+0000","last_became_peered":"2024-03-29T11:31:26.318658+0000","last_unstale":"2024-03-29T11:31:26.850572+0000","last_undegraded":"2024-03-29T11:31:26.850572+0000","last_fullsized":"2024-03-29T11:31:26.850572+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:25:12.671618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"215'366","reported_seq":1270,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-29T11:31:26.309655+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:31:26.309655+0000","last_peered":"2024-03-29T11:31:26.309655+0000","last_clean":"2024-03-29T11:31:26.309655+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:31:26.309655+0000","last_undegraded":"2024-03-29T11:31:26.309655+0000","last_fullsized":"2024-03-29T11:31:26.309655+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575994,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52504,"kb_used_data":3336,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319336,"statfs":{"total":96636764160,"available":96583000064,"internally_reserved":0,"allocated":3416064,"data_stored":1964224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":50330060},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72299999999999998,"15min":0.72999999999999998},"min":{"1min":0.47799999999999998,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":1.0229999999999999,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.72699999999999998,"15min":0.747},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.40300000000000002},"max":{"1min":0.98499999999999999,"5min":1.119,"15min":1.119},"last":0.627}]},{"osd":1,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.73099999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.96799999999999997,"5min":1.002,"15min":1.478},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.749,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97099999999999997,"5min":0.98699999999999999,"15min":1.393},"last":0.59399999999999997}]},{"osd":2,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.55800000000000005,"5min":0.443,"15min":0.443},"max":{"1min":0.89900000000000002,"5min":1.089,"15min":1.284},"last":0.436},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76500000000000001,"15min":0.77800000000000002},"min":{"1min":0.52900000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.032,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75900000000000001,"15min":0.76400000000000001},"min":{"1min":0.52000000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.96899999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.78600000000000003,"15min":0.79400000000000004},"min":{"1min":0.65200000000000002,"5min":0.55900000000000005,"15min":0.55300000000000005},"max":{"1min":1.0069999999999999,"5min":1.071,"15min":1.3169999999999999},"last":0.64200000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.627,"15min":0.63700000000000001},"min":{"1min":0.45300000000000001,"5min":0.44900000000000001,"15min":0.41999999999999998},"max":{"1min":0.88100000000000001,"5min":1.024,"15min":1.3779999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.48199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89900000000000002,"5min":0.95299999999999996,"15min":1.335},"last":0.46800000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.69199999999999995,"15min":0.67900000000000005},"min":{"1min":0.50800000000000001,"5min":0.46700000000000003,"15min":0.38400000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.347},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.72699999999999998,"15min":0.71499999999999997},"min":{"1min":0.55400000000000005,"5min":0.50900000000000001,"15min":0.44900000000000001},"max":{"1min":1.006,"5min":1.0169999999999999,"15min":1.1759999999999999},"last":0.73999999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.59999999999999998,"5min":0.48499999999999999,"15min":0.47899999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.2070000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.75700000000000001,"15min":0.73799999999999999},"min":{"1min":0.53200000000000003,"5min":0.442,"15min":0.435},"max":{"1min":1.014,"5min":1.071,"15min":1.071},"last":0.61199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46184,"kb_used_data":2772,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325656,"statfs":{"total":96636764160,"available":96589471744,"internally_reserved":0,"allocated":2838528,"data_stored":1381285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44431815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72699999999999998,"15min":0.746},"min":{"1min":0.52600000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89100000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.745,"15min":0.76800000000000002},"min":{"1min":0.61299999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.96399999999999997,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.621}]},{"osd":1,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73099999999999998,"15min":0.74399999999999999},"min":{"1min":0.61199999999999999,"5min":0.47599999999999998,"15min":0.44800000000000001},"max":{"1min":0.88,"5min":1.98,"15min":1.98},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76000000000000001,"15min":0.77900000000000003},"min":{"1min":0.51100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94499999999999995,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.69999999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74099999999999999,"15min":0.75700000000000001},"min":{"1min":0.54800000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.96199999999999997,"5min":1.669,"15min":1.669},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76200000000000001,"15min":0.77100000000000002},"min":{"1min":0.56899999999999995,"5min":0.49099999999999999,"15min":0.46200000000000002},"max":{"1min":1.016,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.58099999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.76000000000000001,"15min":0.77900000000000003},"min":{"1min":0.59199999999999997,"5min":0.54200000000000004,"15min":0.499},"max":{"1min":0.96099999999999997,"5min":2.52,"15min":2.52},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.79100000000000004,"15min":0.80500000000000005},"min":{"1min":0.69999999999999996,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.95099999999999996,"5min":1.877,"15min":1.877},"last":0.67600000000000005}]},{"osd":4,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.40699999999999997,"15min":0.40300000000000002},"max":{"1min":0.93999999999999995,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.69999999999999996,"15min":0.70899999999999996},"min":{"1min":0.55400000000000005,"5min":0.47299999999999998,"15min":0.45900000000000002},"max":{"1min":0.93500000000000005,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":0.504}]},{"osd":5,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65400000000000003},"min":{"1min":0.42599999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96099999999999997,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.70999999999999996,"15min":0.71199999999999997},"min":{"1min":0.503,"5min":0.46999999999999997,"15min":0.42199999999999999},"max":{"1min":0.98399999999999999,"5min":1.925,"15min":1.925},"last":0.64600000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.754,"15min":0.76400000000000001},"min":{"1min":0.59399999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.73399999999999999,"15min":0.74099999999999999},"min":{"1min":0.53600000000000003,"5min":0.504,"15min":0.47499999999999998},"max":{"1min":1.0009999999999999,"5min":1.726,"15min":1.726},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60132,"kb_used_data":2768,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311708,"statfs":{"total":96636764160,"available":96575188992,"internally_reserved":0,"allocated":2834432,"data_stored":1380703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.46899999999999997},"min":{"1min":0.33900000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.81100000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.41999999999999998,"5min":0.35299999999999998,"15min":0.34799999999999998},"max":{"1min":0.75900000000000001,"5min":0.81399999999999995,"15min":0.90300000000000002},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.42999999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77000000000000002,"5min":0.86199999999999999,"15min":0.995},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.49199999999999999,"5min":0.38900000000000001,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.83999999999999997,"15min":0.878},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75900000000000001,"15min":0.79200000000000004},"min":{"1min":0.55100000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.995,"5min":1.153,"15min":1.3660000000000001},"last":1.022},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.78300000000000003,"15min":0.80500000000000005},"min":{"1min":0.60199999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.96399999999999997,"5min":1.304,"15min":1.304},"last":0.97699999999999998}]},{"osd":5,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80100000000000005,"15min":0.82199999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.48199999999999998},"max":{"1min":0.97799999999999998,"5min":1.3380000000000001,"15min":1.3720000000000001},"last":0.997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.80900000000000005,"15min":0.83799999999999997},"min":{"1min":0.65300000000000002,"5min":0.59199999999999997,"15min":0.59099999999999997},"max":{"1min":1.0549999999999999,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.92600000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.79000000000000004,"15min":0.82299999999999995},"min":{"1min":0.67100000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.95599999999999996,"5min":1.1890000000000001,"15min":1.1919999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.84199999999999997,"15min":0.85499999999999998},"min":{"1min":0.61899999999999999,"5min":0.60999999999999999,"15min":0.53100000000000003},"max":{"1min":1.0569999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"last":1.042}]},{"osd":7,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.65900000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.034,"5min":1.327,"15min":1.3540000000000001},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.84399999999999997,"15min":0.87},"min":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":1.0700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60692,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311148,"statfs":{"total":96636764160,"available":96574615552,"internally_reserved":0,"allocated":3411968,"data_stored":1963205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.35799999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.67200000000000004,"5min":0.95899999999999996,"15min":1.087},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.40799999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.85799999999999998,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.40500000000000003,"5min":0.39200000000000002,"15min":0.36099999999999999},"max":{"1min":0.78100000000000003,"5min":1.028,"15min":1.5289999999999999},"last":0.371},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52100000000000002,"15min":0.52200000000000002},"min":{"1min":0.39300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.66800000000000004,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.40400000000000003,"15min":0.36299999999999999},"max":{"1min":0.84699999999999998,"5min":1.222,"15min":1.222},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.64700000000000002},"min":{"1min":0.51500000000000001,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.47299999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.81299999999999994,"15min":0.82199999999999995},"min":{"1min":0.621,"5min":0.495,"15min":0.495},"max":{"1min":1.022,"5min":1.1719999999999999,"15min":1.232},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.84399999999999997,"15min":0.85499999999999998},"min":{"1min":0.69399999999999995,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.96099999999999997,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.67900000000000005}]},{"osd":5,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.78700000000000003,"15min":0.79200000000000004},"min":{"1min":0.59699999999999998,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.004,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.82199999999999995,"15min":0.82799999999999996},"min":{"1min":0.57299999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.98499999999999999,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.56299999999999994}]},{"osd":6,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85999999999999999,"15min":0.86399999999999999},"min":{"1min":0.69799999999999995,"5min":0.62,"15min":0.60999999999999999},"max":{"1min":1.155,"5min":1.343,"15min":1.6240000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.88400000000000001,"15min":0.89500000000000002},"min":{"1min":0.63900000000000001,"5min":0.58499999999999996,"15min":0.58199999999999996},"max":{"1min":1.1399999999999999,"5min":1.3240000000000001,"15min":1.337},"last":0.74299999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82299999999999995,"15min":0.82999999999999996},"min":{"1min":0.56000000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.96499999999999997,"5min":1.1299999999999999,"15min":1.214},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88400000000000001},"min":{"1min":0.65400000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.103,"5min":1.365,"15min":1.365},"last":0.77100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575994,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47824,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324016,"statfs":{"total":96636764160,"available":96587792384,"internally_reserved":0,"allocated":2822144,"data_stored":1373856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.377,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67800000000000005,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73999999999999999,"5min":0.75600000000000001,"15min":0.96799999999999997},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.79000000000000004,"15min":0.80700000000000005},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.872,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.439}]},{"osd":3,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76700000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.44,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76300000000000001,"5min":1.038,"15min":1.038},"last":0.46200000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79100000000000004,"15min":0.81100000000000005},"min":{"1min":0.61599999999999999,"5min":0.57899999999999996,"15min":0.55100000000000005},"max":{"1min":0.89200000000000002,"5min":1.169,"15min":1.1970000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78500000000000003,"15min":0.81599999999999995},"min":{"1min":0.64800000000000002,"5min":0.57099999999999995,"15min":0.52400000000000002},"max":{"1min":1.157,"5min":1.244,"15min":1.244},"last":0.67500000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.61799999999999999,"5min":0.59799999999999998,"15min":0.53400000000000003},"max":{"1min":1.1200000000000001,"5min":1.256,"15min":1.256},"last":0.753},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81299999999999994,"15min":0.82999999999999996},"min":{"1min":0.72799999999999998,"5min":0.57799999999999996,"15min":0.56000000000000005},"max":{"1min":0.97499999999999998,"5min":1.24,"15min":1.3140000000000001},"last":0.60099999999999998}]},{"osd":6,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82799999999999996,"15min":0.85199999999999998},"min":{"1min":0.64800000000000002,"5min":0.58999999999999997,"15min":0.57799999999999996},"max":{"1min":1.105,"5min":1.1459999999999999,"15min":1.24},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.623,"5min":0.58599999999999997,"15min":0.58199999999999996},"max":{"1min":1.0880000000000001,"5min":1.1879999999999999,"15min":1.329},"last":0.624}]},{"osd":7,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.877,"15min":0.90000000000000002},"min":{"1min":0.67400000000000004,"5min":0.66800000000000004,"15min":0.64600000000000002},"max":{"1min":1.0209999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88900000000000001,"15min":0.91200000000000003},"min":{"1min":0.68300000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.071,"5min":1.304,"15min":1.304},"last":0.70899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575994,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64196,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307644,"statfs":{"total":96636764160,"available":96571027456,"internally_reserved":0,"allocated":2809856,"data_stored":1367509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":62912968},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.71299999999999997,"15min":0.751},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53900000000000003,"15min":0.55100000000000005},"min":{"1min":0.36099999999999999,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48899999999999999,"15min":0.498},"min":{"1min":0.36699999999999999,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":0.70199999999999996,"5min":0.72099999999999997,"15min":0.748},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":1.046},"last":0.44700000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.58099999999999996,"15min":0.59199999999999997},"min":{"1min":0.48799999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.72599999999999998,"5min":0.82599999999999996,"15min":0.96699999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.55700000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90800000000000003,"5min":1.0049999999999999,"15min":1.01},"last":0.62}]},{"osd":4,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.79100000000000004,"15min":0.80400000000000005},"min":{"1min":0.66600000000000004,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.1000000000000001,"5min":1.266,"15min":1.2889999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.56999999999999995,"5min":0.52400000000000002,"15min":0.51400000000000001},"max":{"1min":1.0680000000000001,"5min":1.306,"15min":1.306},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.81100000000000005,"15min":0.81599999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.502},"max":{"1min":1.048,"5min":1.123,"15min":1.252},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.81599999999999995,"15min":0.82799999999999996},"min":{"1min":0.59899999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.173,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.69499999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.85099999999999998,"15min":0.85899999999999999},"min":{"1min":0.67500000000000004,"5min":0.61199999999999999,"15min":0.57799999999999996},"max":{"1min":1.0900000000000001,"5min":1.288,"15min":1.288},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84199999999999997,"15min":0.84599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1779999999999999,"5min":1.248,"15min":1.248},"last":0.68400000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84399999999999997,"15min":0.86499999999999999},"min":{"1min":0.65300000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.137,"5min":1.331,"15min":1.331},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.88400000000000001,"15min":0.89300000000000002},"min":{"1min":0.70399999999999996,"5min":0.64900000000000002,"15min":0.57899999999999996},"max":{"1min":1.1559999999999999,"5min":1.286,"15min":1.286},"last":0.78100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60120,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311720,"statfs":{"total":96636764160,"available":96575201280,"internally_reserved":0,"allocated":2822144,"data_stored":1373856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70599999999999996,"15min":0.73199999999999998},"min":{"1min":0.50700000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85199999999999998,"5min":0.95499999999999996,"15min":1.0509999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.70599999999999996,"15min":0.73299999999999998},"min":{"1min":0.50800000000000001,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.86599999999999999,"5min":0.99299999999999999,"15min":1.1459999999999999},"last":0.62}]},{"osd":1,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71999999999999997,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":0.875,"5min":1.1000000000000001,"15min":1.1910000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72499999999999998,"15min":0.753},"min":{"1min":0.53200000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93899999999999995,"5min":0.94899999999999995,"15min":1.002},"last":0.69299999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.81200000000000006,"15min":0.83599999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.92500000000000004,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.80200000000000005,"15min":0.82199999999999995},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.90900000000000003,"5min":1.224,"15min":1.224},"last":0.54100000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73999999999999999,"15min":0.77000000000000002},"min":{"1min":0.56799999999999995,"5min":0.44,"15min":0.44},"max":{"1min":0.88300000000000001,"5min":1.0580000000000001,"15min":1.181},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77600000000000002,"15min":0.77700000000000002},"min":{"1min":0.58499999999999996,"5min":0.54700000000000004,"15min":0.48699999999999999},"max":{"1min":0.91300000000000003,"5min":0.995,"15min":1.2490000000000001},"last":0.66600000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.624,"15min":0.623},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86099999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.64200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45900000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83099999999999996,"5min":1.0920000000000001,"15min":1.169},"last":0.80700000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.45200000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":0.82299999999999995,"5min":1.0509999999999999,"15min":1.0800000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71399999999999997,"15min":0.71499999999999997},"min":{"1min":0.52000000000000002,"5min":0.48199999999999998,"15min":0.47699999999999998},"max":{"1min":0.95599999999999996,"5min":1.024,"15min":1.028},"last":0.77100000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.54200000000000004,"5min":0.438,"15min":0.436},"max":{"1min":0.83499999999999996,"5min":1.0389999999999999,"15min":1.518},"last":0.873},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.82399999999999995,"15min":0.82699999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.54400000000000004},"max":{"1min":0.96099999999999997,"5min":1.1619999999999999,"15min":1.3440000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575994,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42080,"kb_used_data":2772,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329760,"statfs":{"total":96636764160,"available":96593674240,"internally_reserved":0,"allocated":2838528,"data_stored":1381285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70499999999999996,"15min":0.71399999999999997},"min":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.95999999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.877,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76700000000000002},"min":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93600000000000005,"5min":0.95999999999999996,"15min":1.0740000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73399999999999999,"15min":0.73999999999999999},"min":{"1min":0.56899999999999995,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91800000000000004,"5min":1.123,"15min":1.123},"last":0.65100000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77400000000000002,"15min":0.76000000000000001},"min":{"1min":0.69299999999999995,"5min":0.51500000000000001,"15min":0.48299999999999998},"max":{"1min":0.91600000000000004,"5min":0.95299999999999996,"15min":1.04},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.748,"15min":0.755},"min":{"1min":0.627,"5min":0.51200000000000001,"15min":0.48399999999999999},"max":{"1min":0.91000000000000003,"5min":0.98499999999999999,"15min":1.0129999999999999},"last":0.72399999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.99099999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.77500000000000002,"15min":0.78000000000000003},"min":{"1min":0.66200000000000003,"5min":0.52000000000000002,"15min":0.51800000000000002},"max":{"1min":0.97699999999999998,"5min":1.0409999999999999,"15min":1.046},"last":0.84199999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.627,"15min":0.60299999999999998},"min":{"1min":0.52000000000000002,"5min":0.41799999999999998,"15min":0.38700000000000001},"max":{"1min":0.88300000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66900000000000004,"15min":0.64800000000000002},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.40600000000000003},"max":{"1min":0.95999999999999996,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.80200000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.68000000000000005,"15min":0.66300000000000003},"min":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42699999999999999},"max":{"1min":1.046,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.71399999999999997,"15min":0.69799999999999995},"min":{"1min":0.51400000000000001,"5min":0.51000000000000001,"15min":0.42499999999999999},"max":{"1min":1.0680000000000001,"5min":1.107,"15min":1.107},"last":0.78100000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.752,"15min":0.72399999999999998},"min":{"1min":0.56699999999999995,"5min":0.52900000000000003,"15min":0.42299999999999999},"max":{"1min":1.036,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.78900000000000003,"15min":0.76600000000000001},"min":{"1min":0.54800000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1000000000000001,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.98199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:31:32.227 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T11:31:32.227 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T11:31:32.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.473+0000 7fbfe8c00700 1 Processor -- start 2024-03-29T11:31:32.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.474+0000 7fbfe8c00700 1 -- start start 2024-03-29T11:31:32.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.475+0000 7fbfe8c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbfe415d9d0 0x7fbfe40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:32.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.475+0000 7fbfe8c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbfe40d16a0 0x7fbfe40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:32.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.475+0000 7fbfe8c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe4161420 con 0x7fbfe40d2370 2024-03-29T11:31:32.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.475+0000 7fbfe8c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe40d27e0 con 0x7fbfe415d9d0 2024-03-29T11:31:32.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.475+0000 7fbfe8c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe4161700 con 0x7fbfe40d16a0 2024-03-29T11:31:32.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.475+0000 7fbfe1e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbfe415d9d0 0x7fbfe40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:32.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.475+0000 7fbfe2800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbfe40d2370 0x7fbfe415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57654/0 (socket says 172.21.15.12:57654) 2024-03-29T11:31:32.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.475+0000 7fbfe2800700 1 -- 172.21.15.12:0/4024808700 learned_addr learned my addr 172.21.15.12:0/4024808700 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:32.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.475+0000 7fbfe1400700 1 --2- 172.21.15.12:0/4024808700 >> v2:172.21.15.159:3300/0 conn(0x7fbfe40d16a0 0x7fbfe40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:32.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.476+0000 7fbfe1e00700 1 -- 172.21.15.12:0/4024808700 >> v2:172.21.15.159:3300/0 conn(0x7fbfe40d16a0 msgr2=0x7fbfe40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:32.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.476+0000 7fbfe1e00700 1 --2- 172.21.15.12:0/4024808700 >> v2:172.21.15.159:3300/0 conn(0x7fbfe40d16a0 0x7fbfe40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.476+0000 7fbfe1e00700 1 -- 172.21.15.12:0/4024808700 >> v1:172.21.15.12:6789/0 conn(0x7fbfe40d2370 legacy=0x7fbfe415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.476+0000 7fbfe0a00700 1 -- 172.21.15.12:0/4024808700 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312684632 0 0) 0x7fbfe4161420 con 0x7fbfe40d2370 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.476+0000 7fbfe1e00700 1 -- 172.21.15.12:0/4024808700 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe41619e0 con 0x7fbfe415d9d0 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.476+0000 7fbfe1e00700 1 --2- 172.21.15.12:0/4024808700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe415d9d0 0x7fbfe40d3c30 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40d12f0 tx=0x7fbfd4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91ef5a15aa77043 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.476+0000 7fbfe0a00700 1 -- 172.21.15.12:0/4024808700 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4011830 con 0x7fbfe415d9d0 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.476+0000 7fbfe0a00700 1 -- 172.21.15.12:0/4024808700 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd4011990 con 0x7fbfe415d9d0 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.476+0000 7fbfe0a00700 1 -- 172.21.15.12:0/4024808700 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4011d20 con 0x7fbfe415d9d0 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.477+0000 7fbfe8c00700 1 -- 172.21.15.12:0/4024808700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe415d9d0 msgr2=0x7fbfe40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.477+0000 7fbfe8c00700 1 --2- 172.21.15.12:0/4024808700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe415d9d0 0x7fbfe40d3c30 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40d12f0 tx=0x7fbfd4009b20 comp rx=0 tx=0).stop 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.477+0000 7fbfe8c00700 1 -- 172.21.15.12:0/4024808700 shutdown_connections 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.477+0000 7fbfe8c00700 1 --2- 172.21.15.12:0/4024808700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe415d9d0 0x7fbfe40d3c30 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.477+0000 7fbfe8c00700 1 --2- 172.21.15.12:0/4024808700 >> v2:172.21.15.159:3300/0 conn(0x7fbfe40d16a0 0x7fbfe40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.477+0000 7fbfe8c00700 1 -- 172.21.15.12:0/4024808700 >> 172.21.15.12:0/4024808700 conn(0x7fbfe408aff0 msgr2=0x7fbfe40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.477+0000 7fbfe8c00700 1 -- 172.21.15.12:0/4024808700 shutdown_connections 2024-03-29T11:31:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.477+0000 7fbfe8c00700 1 -- 172.21.15.12:0/4024808700 wait complete. 2024-03-29T11:31:32.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.479+0000 7fbfe8c00700 1 Processor -- start 2024-03-29T11:31:32.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.479+0000 7fbfe8c00700 1 -- start start 2024-03-29T11:31:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe40d16a0 0x7fbfe4155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe8c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfe40d2370 0x7fbfe40b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe8c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfe415d9d0 0x7fbfe40b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe8c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40b7fb0 con 0x7fbfe40d16a0 2024-03-29T11:31:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe8c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40b73a0 con 0x7fbfe415d9d0 2024-03-29T11:31:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe8c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40b7680 con 0x7fbfe40d2370 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe40d16a0 0x7fbfe4155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe40d16a0 0x7fbfe4155c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36854/0 (socket says 172.21.15.12:36854) 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe1e00700 1 -- 172.21.15.12:0/1316756343 learned_addr learned my addr 172.21.15.12:0/1316756343 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe2800700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfe415d9d0 0x7fbfe40b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe1400700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfe40d2370 0x7fbfe40b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe1e00700 1 -- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfe40d2370 msgr2=0x7fbfe40b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe1e00700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfe40d2370 0x7fbfe40b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe1e00700 1 -- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfe415d9d0 msgr2=0x7fbfe40b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe1e00700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfe415d9d0 0x7fbfe40b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.480+0000 7fbfe1e00700 1 -- 172.21.15.12:0/1316756343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd40094a0 con 0x7fbfe40d16a0 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.481+0000 7fbfe1e00700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe40d16a0 0x7fbfe4155c00 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4000c00 tx=0x7fbfd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.481+0000 7fbfda000700 1 -- 172.21.15.12:0/1316756343 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd40036a0 con 0x7fbfe40d16a0 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.481+0000 7fbfda000700 1 -- 172.21.15.12:0/1316756343 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd401e070 con 0x7fbfe40d16a0 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.481+0000 7fbfda000700 1 -- 172.21.15.12:0/1316756343 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4003a60 con 0x7fbfe40d16a0 2024-03-29T11:31:32.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.481+0000 7fbfe8c00700 1 -- 172.21.15.12:0/1316756343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe40b7960 con 0x7fbfe40d16a0 2024-03-29T11:31:32.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.481+0000 7fbfe8c00700 1 -- 172.21.15.12:0/1316756343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe40b6f70 con 0x7fbfe40d16a0 2024-03-29T11:31:32.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.482+0000 7fbfe8c00700 1 -- 172.21.15.12:0/1316756343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe40d1ca0 con 0x7fbfe40d16a0 2024-03-29T11:31:32.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.483+0000 7fbfda000700 1 -- 172.21.15.12:0/1316756343 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbfd4003bc0 con 0x7fbfe40d16a0 2024-03-29T11:31:32.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.484+0000 7fbfda000700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbfc406adb0 0x7fbfc406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:32.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.484+0000 7fbfda000700 1 -- 172.21.15.12:0/1316756343 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fbfd408a410 con 0x7fbfe40d16a0 2024-03-29T11:31:32.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.484+0000 7fbfe1400700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbfc406adb0 0x7fbfc406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:32.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.485+0000 7fbfe1400700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbfc406adb0 0x7fbfc406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4156140 tx=0x7fbfd0008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:32.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.488+0000 7fbfda000700 1 -- 172.21.15.12:0/1316756343 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbfd4055520 con 0x7fbfe40d16a0 2024-03-29T11:31:32.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.785+0000 7fbfe8c00700 1 -- 172.21.15.12:0/1316756343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbfe40b7160 con 0x7fbfe40d16a0 2024-03-29T11:31:32.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.789+0000 7fbfda000700 1 -- 172.21.15.12:0/1316756343 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v50) v1 ==== 190+0+0 (secure 0 0 0) 0x7fbfe40b7160 con 0x7fbfe40d16a0 2024-03-29T11:31:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 -- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbfc406adb0 msgr2=0x7fbfc406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbfc406adb0 0x7fbfc406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4156140 tx=0x7fbfd0008040 comp rx=0 tx=0).stop 2024-03-29T11:31:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 -- 172.21.15.12:0/1316756343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe40d16a0 msgr2=0x7fbfe4155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe40d16a0 0x7fbfe4155c00 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4000c00 tx=0x7fbfd4004750 comp rx=0 tx=0).stop 2024-03-29T11:31:32.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 -- 172.21.15.12:0/1316756343 shutdown_connections 2024-03-29T11:31:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbfc406adb0 0x7fbfc406d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfe415d9d0 0x7fbfe40b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfe40d16a0 0x7fbfe4155c00 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 --2- 172.21.15.12:0/1316756343 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfe40d2370 0x7fbfe40b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 -- 172.21.15.12:0/1316756343 >> 172.21.15.12:0/1316756343 conn(0x7fbfe408aff0 msgr2=0x7fbfe40b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 -- 172.21.15.12:0/1316756343 shutdown_connections 2024-03-29T11:31:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:32.792+0000 7fbfe8c00700 1 -- 172.21.15.12:0/1316756343 wait complete. 2024-03-29T11:31:32.787 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T11:31:32.804 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:31:33.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.008+0000 7ff51c800700 1 Processor -- start 2024-03-29T11:31:33.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.009+0000 7ff51c800700 1 -- start start 2024-03-29T11:31:33.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.010+0000 7ff51c800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5140b3af0 0x7ff51415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.010+0000 7ff51c800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5140b2180 con 0x7ff514072580 2024-03-29T11:31:33.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.010+0000 7ff51c800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51415b100 con 0x7ff5140709c0 2024-03-29T11:31:33.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.010+0000 7ff51c800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51415b3e0 con 0x7ff5140b3af0 2024-03-29T11:31:33.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.010+0000 7ff519400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff514072580 0x7ff5140b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57666/0 (socket says 172.21.15.12:57666) 2024-03-29T11:31:33.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.010+0000 7ff519400700 1 -- 172.21.15.12:0/441468980 learned_addr learned my addr 172.21.15.12:0/441468980 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:33.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.010+0000 7ff519e00700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5140b3af0 0x7ff51415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.010+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284375359 0 0) 0x7ff5140b2180 con 0x7ff514072580 2024-03-29T11:31:33.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4fc0035b0 con 0x7ff514072580 2024-03-29T11:31:33.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543804818 0 0) 0x7ff51415b100 con 0x7ff5140709c0 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5140b2180 con 0x7ff5140709c0 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2079288089 0 0) 0x7ff4fc0035b0 con 0x7ff514072580 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5140b3af0 msgr2=0x7ff51415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5140b3af0 0x7ff51415a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 >> v1:172.21.15.159:6789/0 conn(0x7ff5140709c0 legacy=0x7ff51406e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140b23f0 con 0x7ff514072580 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7ff504001670 con 0x7ff514072580 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 >> v1:172.21.15.12:6789/0 conn(0x7ff514072580 legacy=0x7ff5140b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4fc003bc0 0x7ff4fc003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4fc004850 0x7ff4fc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc005210 0x7ff4fc00dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51415b3e0 con 0x7ff4fc004850 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4fc00e240 con 0x7ff4fc003bc0 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4fc00e400 con 0x7ff4fc005210 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff519e00700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5140b3af0 0x7ff51415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff519400700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4fc004850 0x7ff4fc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.011+0000 7ff51a800700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc005210 0x7ff4fc00dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff519e00700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4fc003bc0 0x7ff4fc003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff519400700 1 -- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc005210 msgr2=0x7ff4fc00dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff519400700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc005210 0x7ff4fc00dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff519400700 1 -- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4fc003bc0 msgr2=0x7ff4fc003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff519400700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4fc003bc0 0x7ff4fc003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff519400700 1 -- 172.21.15.12:0/441468980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc0035b0 con 0x7ff4fc004850 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff51a800700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc005210 0x7ff4fc00dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff519400700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4fc004850 0x7ff4fc004cd0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7ff50400c160 tx=0x7ff4fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee8133f90ef537c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50400cca0 con 0x7ff4fc004850 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50400ce20 con 0x7ff4fc004850 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.012+0000 7ff518a00700 1 -- 172.21.15.12:0/441468980 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504005ba0 con 0x7ff4fc004850 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.013+0000 7ff51c800700 1 -- 172.21.15.12:0/441468980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4fc004850 msgr2=0x7ff4fc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.013+0000 7ff51c800700 1 --2- 172.21.15.12:0/441468980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4fc004850 0x7ff4fc004cd0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7ff50400c160 tx=0x7ff4fc005880 comp rx=0 tx=0).stop 2024-03-29T11:31:33.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.013+0000 7ff51a800700 1 -- 172.21.15.12:0/441468980 reap_dead start 2024-03-29T11:31:33.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.013+0000 7ff51c800700 1 -- 172.21.15.12:0/441468980 shutdown_connections 2024-03-29T11:31:33.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.013+0000 7ff51c800700 1 -- 172.21.15.12:0/441468980 >> 172.21.15.12:0/441468980 conn(0x7ff5140cf190 msgr2=0x7ff5140cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:33.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.013+0000 7ff51c800700 1 -- 172.21.15.12:0/441468980 shutdown_connections 2024-03-29T11:31:33.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.013+0000 7ff51c800700 1 -- 172.21.15.12:0/441468980 wait complete. 2024-03-29T11:31:33.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.015+0000 7ff51c800700 1 Processor -- start 2024-03-29T11:31:33.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.015+0000 7ff51c800700 1 -- start start 2024-03-29T11:31:33.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff51c800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc004850 0x7ff5141523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff51c800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff51414a1d0 0x7ff51414a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff51c800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff514158c60 0x7ff5141590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff51c800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51414ab90 con 0x7ff51414a1d0 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff51c800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff514154f90 con 0x7ff514158c60 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff51c800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5141599d0 con 0x7ff4fc004850 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff519400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff51414a1d0 0x7ff51414a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff519400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff51414a1d0 0x7ff51414a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36858/0 (socket says 172.21.15.12:36858) 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff519400700 1 -- 172.21.15.12:0/3302685647 learned_addr learned my addr 172.21.15.12:0/3302685647 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff519e00700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc004850 0x7ff5141523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff51a800700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff514158c60 0x7ff5141590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff519400700 1 -- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc004850 msgr2=0x7ff5141523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff519400700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc004850 0x7ff5141523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff519400700 1 -- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff514158c60 msgr2=0x7ff5141590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff519400700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff514158c60 0x7ff5141590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff519400700 1 -- 172.21.15.12:0/3302685647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff514159cb0 con 0x7ff51414a1d0 2024-03-29T11:31:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.016+0000 7ff519e00700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc004850 0x7ff5141523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.017+0000 7ff519400700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff51414a1d0 0x7ff51414a650 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00e210 tx=0x7ff504001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.017+0000 7ff50e000700 1 -- 172.21.15.12:0/3302685647 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504000ed0 con 0x7ff51414a1d0 2024-03-29T11:31:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.017+0000 7ff50e000700 1 -- 172.21.15.12:0/3302685647 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5040070d0 con 0x7ff51414a1d0 2024-03-29T11:31:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.017+0000 7ff50e000700 1 -- 172.21.15.12:0/3302685647 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504007460 con 0x7ff51414a1d0 2024-03-29T11:31:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.017+0000 7ff51c800700 1 -- 172.21.15.12:0/3302685647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff514159f90 con 0x7ff51414a1d0 2024-03-29T11:31:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.017+0000 7ff51c800700 1 -- 172.21.15.12:0/3302685647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff514159620 con 0x7ff51414a1d0 2024-03-29T11:31:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.018+0000 7ff51c800700 1 -- 172.21.15.12:0/3302685647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff514155190 con 0x7ff51414a1d0 2024-03-29T11:31:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.019+0000 7ff50e000700 1 -- 172.21.15.12:0/3302685647 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff504019030 con 0x7ff51414a1d0 2024-03-29T11:31:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.020+0000 7ff50e000700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff4f806ade0 0x7ff4f806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.020+0000 7ff50e000700 1 -- 172.21.15.12:0/3302685647 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7ff50408a6c0 con 0x7ff51414a1d0 2024-03-29T11:31:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.021+0000 7ff519e00700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff4f806ade0 0x7ff4f806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.022+0000 7ff519e00700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff4f806ade0 0x7ff4f806d2a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff51406b860 tx=0x7ff508008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.025+0000 7ff50e000700 1 -- 172.21.15.12:0/3302685647 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff514155190 con 0x7ff51414a1d0 2024-03-29T11:31:33.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.284+0000 7ff51c800700 1 -- 172.21.15.12:0/3302685647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff514155150 con 0x7ff51414a1d0 2024-03-29T11:31:33.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.285+0000 7ff50e000700 1 -- 172.21.15.12:0/3302685647 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4755 (secure 0 0 0) 0x7ff504055730 con 0x7ff51414a1d0 2024-03-29T11:31:33.279 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:33.279 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":106,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":11025,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:26.816217+0000","modified":"2024-03-29T11:31:30.290952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/2722218561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2722218561},{"type":"v1","addr":"172.21.15.12:6833","nonce":2722218561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-03-29T11:31:33.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 -- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff4f806ade0 msgr2=0x7ff4f806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff4f806ade0 0x7ff4f806d2a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff51406b860 tx=0x7ff508008040 comp rx=0 tx=0).stop 2024-03-29T11:31:33.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 -- 172.21.15.12:0/3302685647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff51414a1d0 msgr2=0x7ff51414a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff51414a1d0 0x7ff51414a650 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00e210 tx=0x7ff504001930 comp rx=0 tx=0).stop 2024-03-29T11:31:33.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 -- 172.21.15.12:0/3302685647 shutdown_connections 2024-03-29T11:31:33.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff4f806ade0 0x7ff4f806d2a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff514158c60 0x7ff5141590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff4fc004850 0x7ff5141523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 --2- 172.21.15.12:0/3302685647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff51414a1d0 0x7ff51414a650 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 -- 172.21.15.12:0/3302685647 >> 172.21.15.12:0/3302685647 conn(0x7ff5140cf190 msgr2=0x7ff51406e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:33.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 -- 172.21.15.12:0/3302685647 shutdown_connections 2024-03-29T11:31:33.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.288+0000 7ff51c800700 1 -- 172.21.15.12:0/3302685647 wait complete. 2024-03-29T11:31:33.284 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 106 2024-03-29T11:31:33.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T11:31:26.816217+0000', 'modified': '2024-03-29T11:31:30.290952+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11034}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11034': {'gid': 11034, 'name': 'a', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6833/2722218561', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6832', 'nonce': 2722218561}, {'type': 'v1', 'addr': '172.21.15.12:6833', 'nonce': 2722218561}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T11:31:33.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T11:31:33.705 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-29T11:31:33.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.914+0000 7f7904400700 1 Processor -- start 2024-03-29T11:31:33.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.914+0000 7f7904400700 1 -- start start 2024-03-29T11:31:33.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.915+0000 7f7904400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78fc15d9d0 0x7f78fc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.915+0000 7f7904400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f78fc0d16a0 0x7f78fc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.915+0000 7f7904400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f78fc0d2370 0x7f78fc15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.915+0000 7f7904400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc0d2820 con 0x7f78fc0d2370 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.915+0000 7f7904400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc161420 con 0x7f78fc0d16a0 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.915+0000 7f7904400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc161700 con 0x7f78fc15d9d0 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7902400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f78fc0d2370 0x7f78fc15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7902400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f78fc0d2370 0x7f78fc15ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36868/0 (socket says 172.21.15.12:36868) 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7902400700 1 -- 172.21.15.12:0/275281641 learned_addr learned my addr 172.21.15.12:0/275281641 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7901a00700 1 --2- 172.21.15.12:0/275281641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78fc15d9d0 0x7f78fc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7901000700 1 --2- 172.21.15.12:0/275281641 >> v2:172.21.15.159:3300/0 conn(0x7f78fc0d16a0 0x7f78fc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7902400700 1 -- 172.21.15.12:0/275281641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78fc15d9d0 msgr2=0x7f78fc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7902400700 1 --2- 172.21.15.12:0/275281641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78fc15d9d0 0x7f78fc0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7902400700 1 -- 172.21.15.12:0/275281641 >> v2:172.21.15.159:3300/0 conn(0x7f78fc0d16a0 msgr2=0x7f78fc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7902400700 1 --2- 172.21.15.12:0/275281641 >> v2:172.21.15.159:3300/0 conn(0x7f78fc0d16a0 0x7f78fc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7902400700 1 -- 172.21.15.12:0/275281641 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc1619e0 con 0x7f78fc0d2370 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7901000700 1 --2- 172.21.15.12:0/275281641 >> v2:172.21.15.159:3300/0 conn(0x7f78fc0d16a0 0x7f78fc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.916+0000 7f7902400700 1 --2- 172.21.15.12:0/275281641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc0d2370 0x7f78fc15ddb0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f78fc06b5e0 tx=0x7f78f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1842cf55369c9f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f78fbe00700 1 -- 172.21.15.12:0/275281641 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f400e070 con 0x7f78fc0d2370 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f78fbe00700 1 -- 172.21.15.12:0/275281641 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f400b9c0 con 0x7f78fc0d2370 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f78fbe00700 1 -- 172.21.15.12:0/275281641 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f400bd70 con 0x7f78fc0d2370 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f7904400700 1 -- 172.21.15.12:0/275281641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc0d2370 msgr2=0x7f78fc15ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f7904400700 1 --2- 172.21.15.12:0/275281641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc0d2370 0x7f78fc15ddb0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f78fc06b5e0 tx=0x7f78f400b330 comp rx=0 tx=0).stop 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f7904400700 1 -- 172.21.15.12:0/275281641 shutdown_connections 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f7904400700 1 --2- 172.21.15.12:0/275281641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc0d2370 0x7f78fc15ddb0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f7904400700 1 --2- 172.21.15.12:0/275281641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78fc15d9d0 0x7f78fc0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f7904400700 1 --2- 172.21.15.12:0/275281641 >> v2:172.21.15.159:3300/0 conn(0x7f78fc0d16a0 0x7f78fc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f7904400700 1 -- 172.21.15.12:0/275281641 >> 172.21.15.12:0/275281641 conn(0x7f78fc08aff0 msgr2=0x7f78fc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:33.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f7904400700 1 -- 172.21.15.12:0/275281641 shutdown_connections 2024-03-29T11:31:33.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.917+0000 7f7904400700 1 -- 172.21.15.12:0/275281641 wait complete. 2024-03-29T11:31:33.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.920+0000 7f7904400700 1 Processor -- start 2024-03-29T11:31:33.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.920+0000 7f7904400700 1 -- start start 2024-03-29T11:31:33.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.921+0000 7f7904400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78fc0d16a0 0x7f78fc1f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.921+0000 7f7904400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78fc0d2370 0x7f78fc0cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.921+0000 7f7904400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc15d9d0 0x7f78fc0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.921+0000 7f7904400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0b7260 con 0x7f78fc15d9d0 2024-03-29T11:31:33.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.921+0000 7f7904400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0b65e0 con 0x7f78fc0d2370 2024-03-29T11:31:33.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.921+0000 7f7904400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0b68c0 con 0x7f78fc0d16a0 2024-03-29T11:31:33.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.921+0000 7f7902400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc15d9d0 0x7f78fc0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.921+0000 7f7902400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc15d9d0 0x7f78fc0ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36876/0 (socket says 172.21.15.12:36876) 2024-03-29T11:31:33.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f7902400700 1 -- 172.21.15.12:0/124953841 learned_addr learned my addr 172.21.15.12:0/124953841 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:33.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f7901a00700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78fc0d16a0 0x7f78fc1f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f7901000700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78fc0d2370 0x7f78fc0cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f7902400700 1 -- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78fc0d16a0 msgr2=0x7f78fc1f2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f7902400700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78fc0d16a0 0x7f78fc1f2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f7902400700 1 -- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78fc0d2370 msgr2=0x7f78fc0cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f7902400700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78fc0d2370 0x7f78fc0cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f7902400700 1 -- 172.21.15.12:0/124953841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f4009b30 con 0x7f78fc15d9d0 2024-03-29T11:31:33.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f7902400700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc15d9d0 0x7f78fc0ca5f0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f78fc08c230 tx=0x7f78f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:33.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f78f9600700 1 -- 172.21.15.12:0/124953841 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f40099b0 con 0x7f78fc15d9d0 2024-03-29T11:31:33.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.922+0000 7f78f9600700 1 -- 172.21.15.12:0/124953841 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f4004250 con 0x7f78fc15d9d0 2024-03-29T11:31:33.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.923+0000 7f7904400700 1 -- 172.21.15.12:0/124953841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc0b6ba0 con 0x7f78fc15d9d0 2024-03-29T11:31:33.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.923+0000 7f78f9600700 1 -- 172.21.15.12:0/124953841 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4004600 con 0x7f78fc15d9d0 2024-03-29T11:31:33.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.923+0000 7f7904400700 1 -- 172.21.15.12:0/124953841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc0cc550 con 0x7f78fc15d9d0 2024-03-29T11:31:33.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.925+0000 7f78f9600700 1 -- 172.21.15.12:0/124953841 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f78f4004760 con 0x7f78fc15d9d0 2024-03-29T11:31:33.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.926+0000 7f78f9600700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f78dc06adb0 0x7f78dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.926+0000 7f78f9600700 1 -- 172.21.15.12:0/124953841 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f78f408aa40 con 0x7f78fc15d9d0 2024-03-29T11:31:33.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.926+0000 7f7901a00700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f78dc06adb0 0x7f78dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.927+0000 7f7901a00700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f78dc06adb0 0x7f78dc06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f78f0006b20 tx=0x7f78f0008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:33.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.928+0000 7f7917e2ab80 1 Processor -- start 2024-03-29T11:31:33.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.928+0000 7f7917e2ab80 1 -- start start 2024-03-29T11:31:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.928+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa8a2e830 0x55caa8a2ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a04a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa89ffb60 0x55caa8a2ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa8979650 con 0x55caa8a01be0 2024-03-29T11:31:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa895ac30 con 0x55caa89ffb60 2024-03-29T11:31:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55caa8958430 con 0x55caa8a2e830 2024-03-29T11:31:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7901000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a04a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7901000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a04a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36890/0 (socket says 172.21.15.12:36890) 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7901000700 1 -- 172.21.15.12:0/475672760 learned_addr learned my addr 172.21.15.12:0/475672760 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7901a00700 1 --2- 172.21.15.12:0/475672760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa8a2e830 0x55caa8a2ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7902400700 1 --2- 172.21.15.12:0/475672760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa89ffb60 0x55caa8a2ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7901000700 1 -- 172.21.15.12:0/475672760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa8a2e830 msgr2=0x55caa8a2ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7901000700 1 --2- 172.21.15.12:0/475672760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa8a2e830 0x55caa8a2ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7901000700 1 -- 172.21.15.12:0/475672760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa89ffb60 msgr2=0x55caa8a2ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7901000700 1 --2- 172.21.15.12:0/475672760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa89ffb60 0x55caa8a2ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7901000700 1 -- 172.21.15.12:0/475672760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55caa8747790 con 0x55caa8a01be0 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.929+0000 7f7901000700 1 --2- 172.21.15.12:0/475672760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a04a10 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f78e800abd0 tx=0x7f78e800aee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22938d29801eee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.930+0000 7f78ee000700 1 -- 172.21.15.12:0/475672760 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e800fc00 con 0x55caa8a01be0 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.930+0000 7f78ee000700 1 -- 172.21.15.12:0/475672760 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e800c040 con 0x55caa8a01be0 2024-03-29T11:31:33.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.930+0000 7f78ee000700 1 -- 172.21.15.12:0/475672760 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8013440 con 0x55caa8a01be0 2024-03-29T11:31:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.930+0000 7f7917e2ab80 1 -- 172.21.15.12:0/475672760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 msgr2=0x55caa8a04a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.930+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/475672760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a04a10 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f78e800abd0 tx=0x7f78e800aee0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.930+0000 7f7917e2ab80 1 -- 172.21.15.12:0/475672760 shutdown_connections 2024-03-29T11:31:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.930+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/475672760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa89ffb60 0x55caa8a2ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.930+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/475672760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa8a2e830 0x55caa8a2ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.930+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/475672760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a04a10 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.931+0000 7f7917e2ab80 1 -- 172.21.15.12:0/475672760 >> 172.21.15.12:0/475672760 conn(0x55caa89626a0 msgr2=0x55caa8a49640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.931+0000 7f7917e2ab80 1 -- 172.21.15.12:0/475672760 shutdown_connections 2024-03-29T11:31:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.931+0000 7f7917e2ab80 1 -- 172.21.15.12:0/475672760 wait complete. 2024-03-29T11:31:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.931+0000 7f7917e2ab80 1 -- start start 2024-03-29T11:31:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.932+0000 7f7917e2ab80 1 Processor -- start 2024-03-29T11:31:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.932+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8a31290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.932+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a59260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 0x55caa8a4aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa8953d20 con 0x55caa8a01be0 2024-03-29T11:31:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa88ef770 con 0x55caa8a2e830 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55caa89126f0 con 0x55caa89ffb60 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7917e2ab80 10 client.0 _ll_register_callbacks cb 0x55caa88f95e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7901000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a59260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7901000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a59260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36896/0 (socket says 172.21.15.12:36896) 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7901000700 1 -- 172.21.15.12:0/2842399464 learned_addr learned my addr 172.21.15.12:0/2842399464 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7901a00700 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8a31290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7902400700 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 0x55caa8a4aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7901000700 1 -- 172.21.15.12:0/2842399464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 msgr2=0x55caa8a31290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7901000700 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8a31290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7901000700 1 -- 172.21.15.12:0/2842399464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 msgr2=0x55caa8a4aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7901000700 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 0x55caa8a4aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.933+0000 7f7901000700 1 -- 172.21.15.12:0/2842399464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e800a800 con 0x55caa8a01be0 2024-03-29T11:31:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f7901000700 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a59260 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f78e800afd0 tx=0x7f78e80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876acb174acdd018 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f78e2a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f78e2a00700 1 -- 172.21.15.12:0/2842399464 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8020070 con 0x55caa8a01be0 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f78e2a00700 1 -- 172.21.15.12:0/2842399464 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f78e8005e20 con 0x55caa8a01be0 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2842399464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55caa89f6920 con 0x55caa8a01be0 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f78e2a00700 1 -- 172.21.15.12:0/2842399464 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78e800fe90 con 0x55caa8a01be0 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f7917e2ab80 10 client.11169 fetch_fsmap learned FSMap version 106 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2842399464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x55caa88340a0 con 0x55caa8a01be0 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f78e2a00700 1 -- 172.21.15.12:0/2842399464 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55caa89f6920 con 0x55caa8a01be0 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f78e2a00700 1 -- 172.21.15.12:0/2842399464 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 106) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f78e8019020 con 0x55caa8a01be0 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f7917e2ab80 10 client.11169 fetch_fsmap finished waiting for FSMap version 106 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.934+0000 7f7917e2ab80 10 client.11169 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.935+0000 7f7917e2ab80 20 client.11169 populate_metadata read hostname 'smithi012' 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.935+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.12:6832/2722218561,v1:172.21.15.12:6833/2722218561] conn(0x55caa8a5c490 0x55caa8a5c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.935+0000 7f7917e2ab80 4 client.11169 mds_command: new command op to 11034 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.935+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2842399464 --> [v2:172.21.15.12:6832/2722218561,v1:172.21.15.12:6833/2722218561] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55caa89d36f0 con 0x55caa8a5c490 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.935+0000 7f7901a00700 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.12:6832/2722218561,v1:172.21.15.12:6833/2722218561] conn(0x55caa8a5c490 0x55caa8a5c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.935+0000 7f7901a00700 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.12:6832/2722218561,v1:172.21.15.12:6833/2722218561] conn(0x55caa8a5c490 0x55caa8a5c870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdf3e94db8e72a95 server_cookie=69ba3dcc5d77374f in_seq=0 out_seq=0 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.935+0000 7f78e2a00700 10 client.11169 ms_handle_connect on v2:172.21.15.12:6832/2722218561 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f78e2a00700 1 -- 172.21.15.12:0/2842399464 <== mds.0 v2:172.21.15.12:6832/2722218561 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55caa89d36f0 con 0x55caa8a5c490 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f78e2a00700 10 client.11169 handle_command_reply: tid=0 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f7917e2ab80 1 client.11169 shutdown 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2842399464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 msgr2=0x55caa8a59260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a59260 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f78e800afd0 tx=0x7f78e80047c0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2842399464 shutdown_connections 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 0x55caa8a4aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.12:6832/2722218561,v1:172.21.15.12:6833/2722218561] conn(0x55caa8a5c490 0x55caa8a5c870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8a31290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2842399464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa8a59260 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2842399464 >> 172.21.15.12:0/2842399464 conn(0x55caa89626a0 msgr2=0x55caa8a2d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.936+0000 7f78e2a00700 0 client.11169 ms_handle_reset on v2:172.21.15.12:6832/2722218561 2024-03-29T11:31:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.937+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2842399464 shutdown_connections 2024-03-29T11:31:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.937+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2842399464 wait complete. 2024-03-29T11:31:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.937+0000 7f7917e2ab80 20 client.11169 trim_cache size 0 max 16384 2024-03-29T11:31:33.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.955+0000 7f7917e2ab80 1 Processor -- start 2024-03-29T11:31:33.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.956+0000 7f7917e2ab80 1 -- start start 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8a03710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa89fe350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 0x55caa89fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa8748390 con 0x55caa8a01be0 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa8955c00 con 0x55caa8a2e830 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55caa89ff730 con 0x55caa89ffb60 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7901000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa89fe350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7901000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa89fe350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36912/0 (socket says 172.21.15.12:36912) 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7901000700 1 -- 172.21.15.12:0/2725323858 learned_addr learned my addr 172.21.15.12:0/2725323858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7902400700 1 --2- 172.21.15.12:0/2725323858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 0x55caa89fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7901000700 1 -- 172.21.15.12:0/2725323858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 msgr2=0x55caa8a03710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7901000700 1 --2- 172.21.15.12:0/2725323858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8a03710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7901000700 1 -- 172.21.15.12:0/2725323858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 msgr2=0x55caa89fe890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7901000700 1 --2- 172.21.15.12:0/2725323858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 0x55caa89fe890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7901000700 1 -- 172.21.15.12:0/2725323858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55caa8a4db60 con 0x55caa8a01be0 2024-03-29T11:31:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.957+0000 7f7902400700 1 --2- 172.21.15.12:0/2725323858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 0x55caa89fe890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f7901000700 1 --2- 172.21.15.12:0/2725323858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa89fe350 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f78e8000c00 tx=0x7f78e8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a8da96acab7dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f78eea00700 1 -- 172.21.15.12:0/2725323858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e800fd90 con 0x55caa8a01be0 2024-03-29T11:31:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f78eea00700 1 -- 172.21.15.12:0/2725323858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e80036a0 con 0x55caa8a01be0 2024-03-29T11:31:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f78eea00700 1 -- 172.21.15.12:0/2725323858 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8003a50 con 0x55caa8a01be0 2024-03-29T11:31:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2725323858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 msgr2=0x55caa89fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2725323858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa89fe350 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f78e8000c00 tx=0x7f78e8020000 comp rx=0 tx=0).stop 2024-03-29T11:31:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2725323858 shutdown_connections 2024-03-29T11:31:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2725323858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a2e830 0x55caa89fe890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2725323858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8a03710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2725323858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a01be0 0x55caa89fe350 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.958+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2725323858 >> 172.21.15.12:0/2725323858 conn(0x55caa89626a0 msgr2=0x55caa8a6a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.959+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2725323858 shutdown_connections 2024-03-29T11:31:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.959+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2725323858 wait complete. 2024-03-29T11:31:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.959+0000 7f7917e2ab80 1 -- start start 2024-03-29T11:31:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.959+0000 7f7917e2ab80 1 Processor -- start 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.960+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8ab2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.960+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a01be0 0x55caa8a03500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.960+0000 7f7917e2ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a2e830 0x55caa8a03a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.960+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa8a2df60 con 0x55caa8a2e830 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.960+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa8ab2c70 con 0x55caa8a01be0 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.960+0000 7f7917e2ab80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55caa8a45df0 con 0x55caa89ffb60 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.960+0000 7f7917e2ab80 10 client.0 _ll_register_callbacks cb 0x7f78f00076b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7902400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a2e830 0x55caa8a03a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7902400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a2e830 0x55caa8a03a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36924/0 (socket says 172.21.15.12:36924) 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7902400700 1 -- 172.21.15.12:0/2752715933 learned_addr learned my addr 172.21.15.12:0/2752715933 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7901a00700 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8ab2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7901000700 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a01be0 0x55caa8a03500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7902400700 1 -- 172.21.15.12:0/2752715933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 msgr2=0x55caa8ab2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7902400700 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8ab2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7902400700 1 -- 172.21.15.12:0/2752715933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a01be0 msgr2=0x55caa8a03500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7902400700 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a01be0 0x55caa8a03500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7902400700 1 -- 172.21.15.12:0/2752715933 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55caa88340a0 con 0x55caa8a2e830 2024-03-29T11:31:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7901a00700 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8ab2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f7902400700 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a2e830 0x55caa8a03a40 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f78f40595f0 tx=0x7f78f4059e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce848ddb870102 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.961+0000 7f78e2a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:31:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f78e2a00700 1 -- 172.21.15.12:0/2752715933 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4089080 con 0x55caa8a2e830 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f78e2a00700 1 -- 172.21.15.12:0/2752715933 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f78f405d490 con 0x55caa8a2e830 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2752715933 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78e800c040 con 0x55caa8a2e830 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f78e2a00700 1 -- 172.21.15.12:0/2752715933 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78f405d2b0 con 0x55caa8a2e830 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f7917e2ab80 10 client.11175 fetch_fsmap learned FSMap version 106 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2752715933 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x55caa891a010 con 0x55caa8a2e830 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f78e2a00700 1 -- 172.21.15.12:0/2752715933 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e800c040 con 0x55caa8a2e830 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f78e2a00700 1 -- 172.21.15.12:0/2752715933 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 106) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f78f40598c0 con 0x55caa8a2e830 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f7917e2ab80 10 client.11175 fetch_fsmap finished waiting for FSMap version 106 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f7917e2ab80 10 client.11175 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f7917e2ab80 20 client.11175 populate_metadata read hostname 'smithi012' 2024-03-29T11:31:33.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.12:6832/2722218561,v1:172.21.15.12:6833/2722218561] conn(0x55caa8a497d0 0x55caa8a49bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f7917e2ab80 4 client.11175 mds_command: new command op to 11034 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2752715933 --> [v2:172.21.15.12:6832/2722218561,v1:172.21.15.12:6833/2722218561] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55caa878d040 con 0x55caa8a497d0 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.962+0000 7f7901a00700 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.12:6832/2722218561,v1:172.21.15.12:6833/2722218561] conn(0x55caa8a497d0 0x55caa8a49bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f78e2a00700 10 client.11175 ms_handle_connect on v2:172.21.15.12:6832/2722218561 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f7901a00700 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.12:6832/2722218561,v1:172.21.15.12:6833/2722218561] conn(0x55caa8a497d0 0x55caa8a49bb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7da960b67f752165 server_cookie=9f41ddd7bd921757 in_seq=0 out_seq=0 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f78e2a00700 1 -- 172.21.15.12:0/2752715933 <== mds.0 v2:172.21.15.12:6832/2722218561 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55caa878d040 con 0x55caa8a497d0 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f78e2a00700 10 client.11175 handle_command_reply: tid=0 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f7917e2ab80 1 client.11175 shutdown 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2752715933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a2e830 msgr2=0x55caa8a03a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a2e830 0x55caa8a03a40 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f78f40595f0 tx=0x7f78f4059e10 comp rx=0 tx=0).stop 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2752715933 shutdown_connections 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55caa8a2e830 0x55caa8a03a40 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.12:6832/2722218561,v1:172.21.15.12:6833/2722218561] conn(0x55caa8a497d0 0x55caa8a49bb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55caa89ffb60 0x55caa8ab2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f7917e2ab80 1 --2- 172.21.15.12:0/2752715933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55caa8a01be0 0x55caa8a03500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f78e2a00700 0 client.11175 ms_handle_reset on v2:172.21.15.12:6832/2722218561 2024-03-29T11:31:33.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.963+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2752715933 >> 172.21.15.12:0/2752715933 conn(0x55caa89626a0 msgr2=0x55caa89fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:33.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2752715933 shutdown_connections 2024-03-29T11:31:33.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7917e2ab80 1 -- 172.21.15.12:0/2752715933 wait complete. 2024-03-29T11:31:33.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7917e2ab80 20 client.11175 trim_cache size 0 max 16384 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 11034, 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1454118949999996, 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 106, 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 227, 2024-03-29T11:31:33.964 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 227, 2024-03-29T11:31:33.965 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12.894006082000001 2024-03-29T11:31:33.965 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7904400700 1 -- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f78dc06adb0 msgr2=0x7f78dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7904400700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f78dc06adb0 0x7f78dc06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f78f0006b20 tx=0x7f78f0008040 comp rx=0 tx=0).stop 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7904400700 1 -- 172.21.15.12:0/124953841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc15d9d0 msgr2=0x7f78fc0ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7904400700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc15d9d0 0x7f78fc0ca5f0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f78fc08c230 tx=0x7f78f4004ad0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7904400700 1 -- 172.21.15.12:0/124953841 shutdown_connections 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7904400700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f78dc06adb0 0x7f78dc06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7904400700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78fc15d9d0 0x7f78fc0ca5f0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7904400700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78fc0d16a0 0x7f78fc1f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7904400700 1 --2- 172.21.15.12:0/124953841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78fc0d2370 0x7f78fc0cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.964+0000 7f7904400700 1 -- 172.21.15.12:0/124953841 >> 172.21.15.12:0/124953841 conn(0x7f78fc08aff0 msgr2=0x7f78fc0b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:33.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.965+0000 7f7904400700 1 -- 172.21.15.12:0/124953841 shutdown_connections 2024-03-29T11:31:33.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:33.965+0000 7f7904400700 1 -- 172.21.15.12:0/124953841 wait complete. 2024-03-29T11:31:33.976 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T11:31:34.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.181+0000 7f1297400700 1 Processor -- start 2024-03-29T11:31:34.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.181+0000 7f1297400700 1 -- start start 2024-03-29T11:31:34.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.182+0000 7f1297400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f129015d920 0x7f129006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:34.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.182+0000 7f1297400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f129009a910 0x7f12900cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:34.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.182+0000 7f1297400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12900d08d0 con 0x7f1290099a60 2024-03-29T11:31:34.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.182+0000 7f1297400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f129006c300 con 0x7f129015d920 2024-03-29T11:31:34.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.182+0000 7f1297400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129006b9d0 con 0x7f129009a910 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.182+0000 7f1294a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f129015d920 0x7f129006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.182+0000 7f1294a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f129015d920 0x7f129006c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36938/0 (socket says 172.21.15.12:36938) 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.182+0000 7f1294a00700 1 -- 172.21.15.12:0/816262825 learned_addr learned my addr 172.21.15.12:0/816262825 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.183+0000 7f128fe00700 1 --2- 172.21.15.12:0/816262825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f129009a910 0x7f12900cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.183+0000 7f1294a00700 1 -- 172.21.15.12:0/816262825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f129009a910 msgr2=0x7f12900cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.183+0000 7f1294a00700 1 --2- 172.21.15.12:0/816262825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f129009a910 0x7f12900cfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.183+0000 7f1294a00700 1 -- 172.21.15.12:0/816262825 >> v1:172.21.15.159:6789/0 conn(0x7f1290099a60 legacy=0x7f12900cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.183+0000 7f1294a00700 1 -- 172.21.15.12:0/816262825 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129006bcb0 con 0x7f129015d920 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.183+0000 7f1294a00700 1 --2- 172.21.15.12:0/816262825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129015d920 0x7f129006c790 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f12840098f0 tx=0x7f1284009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f7ca5e103151a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.183+0000 7f128f400700 1 -- 172.21.15.12:0/816262825 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284011830 con 0x7f129015d920 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.183+0000 7f128f400700 1 -- 172.21.15.12:0/816262825 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1284011990 con 0x7f129015d920 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.184+0000 7f128f400700 1 -- 172.21.15.12:0/816262825 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284011d20 con 0x7f129015d920 2024-03-29T11:31:34.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.184+0000 7f1297400700 1 -- 172.21.15.12:0/816262825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129015d920 msgr2=0x7f129006c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.184+0000 7f1297400700 1 --2- 172.21.15.12:0/816262825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129015d920 0x7f129006c790 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f12840098f0 tx=0x7f1284009c00 comp rx=0 tx=0).stop 2024-03-29T11:31:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.184+0000 7f1297400700 1 -- 172.21.15.12:0/816262825 shutdown_connections 2024-03-29T11:31:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.184+0000 7f1297400700 1 --2- 172.21.15.12:0/816262825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129015d920 0x7f129006c790 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.184+0000 7f1297400700 1 --2- 172.21.15.12:0/816262825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f129009a910 0x7f12900cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.184+0000 7f1297400700 1 -- 172.21.15.12:0/816262825 >> 172.21.15.12:0/816262825 conn(0x7f12900d1350 msgr2=0x7f12900d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.184+0000 7f1297400700 1 -- 172.21.15.12:0/816262825 shutdown_connections 2024-03-29T11:31:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.184+0000 7f1297400700 1 -- 172.21.15.12:0/816262825 wait complete. 2024-03-29T11:31:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.186+0000 7f1297400700 1 Processor -- start 2024-03-29T11:31:34.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.186+0000 7f1297400700 1 -- start start 2024-03-29T11:31:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.187+0000 7f1297400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1290099a60 0x7f1290160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.187+0000 7f1297400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f129009a910 0x7f12900afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.187+0000 7f1297400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f129015d920 0x7f12900b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.187+0000 7f1297400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12900b6fa0 con 0x7f1290099a60 2024-03-29T11:31:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.187+0000 7f1297400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12901f4200 con 0x7f129015d920 2024-03-29T11:31:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.187+0000 7f1297400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12901f35f0 con 0x7f129009a910 2024-03-29T11:31:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.187+0000 7f1294a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1290099a60 0x7f1290160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.187+0000 7f1294a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1290099a60 0x7f1290160520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36944/0 (socket says 172.21.15.12:36944) 2024-03-29T11:31:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.187+0000 7f1294a00700 1 -- 172.21.15.12:0/2794903828 learned_addr learned my addr 172.21.15.12:0/2794903828 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.187+0000 7f128fe00700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f129009a910 0x7f12900afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f1295400700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f129015d920 0x7f12900b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f1294a00700 1 -- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f129009a910 msgr2=0x7f12900afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f1294a00700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f129009a910 0x7f12900afcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f1294a00700 1 -- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f129015d920 msgr2=0x7f12900b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f1294a00700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f129015d920 0x7f12900b0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f1294a00700 1 -- 172.21.15.12:0/2794903828 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1284009550 con 0x7f1290099a60 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f1295400700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f129015d920 0x7f12900b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f1294a00700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1290099a60 0x7f1290160520 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f1284009c40 tx=0x7f1284004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f128cc00700 1 -- 172.21.15.12:0/2794903828 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284004590 con 0x7f1290099a60 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f128cc00700 1 -- 172.21.15.12:0/2794903828 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128401e070 con 0x7f1290099a60 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f128cc00700 1 -- 172.21.15.12:0/2794903828 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12840038e0 con 0x7f1290099a60 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.188+0000 7f1297400700 1 -- 172.21.15.12:0/2794903828 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12901f38d0 con 0x7f1290099a60 2024-03-29T11:31:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.189+0000 7f1297400700 1 -- 172.21.15.12:0/2794903828 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12901f3e90 con 0x7f1290099a60 2024-03-29T11:31:34.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.190+0000 7f1297400700 1 -- 172.21.15.12:0/2794903828 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1290067720 con 0x7f1290099a60 2024-03-29T11:31:34.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.190+0000 7f128cc00700 1 -- 172.21.15.12:0/2794903828 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1284003a40 con 0x7f1290099a60 2024-03-29T11:31:34.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.191+0000 7f128cc00700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f127006adb0 0x7f127006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:34.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.191+0000 7f128cc00700 1 -- 172.21.15.12:0/2794903828 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f128408a370 con 0x7f1290099a60 2024-03-29T11:31:34.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.192+0000 7f128fe00700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f127006adb0 0x7f127006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:34.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.192+0000 7f128fe00700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f127006adb0 0x7f127006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1290161dd0 tx=0x7f127c009210 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:34.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.196+0000 7f128cc00700 1 -- 172.21.15.12:0/2794903828 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1284055480 con 0x7f1290099a60 2024-03-29T11:31:34.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.462+0000 7f1297400700 1 -- 172.21.15.12:0/2794903828 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f12900d0de0 con 0x7f1290099a60 2024-03-29T11:31:34.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.462+0000 7f128cc00700 1 -- 172.21.15.12:0/2794903828 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v106) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1284058aa0 con 0x7f1290099a60 2024-03-29T11:31:34.456 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T11:31:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.464+0000 7f1297400700 1 -- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f127006adb0 msgr2=0x7f127006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.464+0000 7f1297400700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f127006adb0 0x7f127006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1290161dd0 tx=0x7f127c009210 comp rx=0 tx=0).stop 2024-03-29T11:31:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.465+0000 7f1297400700 1 -- 172.21.15.12:0/2794903828 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1290099a60 msgr2=0x7f1290160520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.465+0000 7f1297400700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1290099a60 0x7f1290160520 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f1284009c40 tx=0x7f1284004750 comp rx=0 tx=0).stop 2024-03-29T11:31:34.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.465+0000 7f1297400700 1 -- 172.21.15.12:0/2794903828 shutdown_connections 2024-03-29T11:31:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.465+0000 7f1297400700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f127006adb0 0x7f127006d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.465+0000 7f1297400700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f129015d920 0x7f12900b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.465+0000 7f1297400700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1290099a60 0x7f1290160520 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.465+0000 7f1297400700 1 --2- 172.21.15.12:0/2794903828 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f129009a910 0x7f12900afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.465+0000 7f1297400700 1 -- 172.21.15.12:0/2794903828 >> 172.21.15.12:0/2794903828 conn(0x7f12900d1350 msgr2=0x7f12900b0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.465+0000 7f1297400700 1 -- 172.21.15.12:0/2794903828 shutdown_connections 2024-03-29T11:31:34.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:34.465+0000 7f1297400700 1 -- 172.21.15.12:0/2794903828 wait complete. 2024-03-29T11:31:34.476 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T11:31:34.477 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T11:31:34.477 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T11:31:34.477 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T11:31:34.477 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T11:31:34.477 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T11:31:34.477 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T11:31:34.477 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T11:31:34.477 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:31:34.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:34.478 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T11:31:34.523 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T11:31:34.523 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T11:31:34.523 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T11:31:34.523 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:31:34.523 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 35623sec preferred_lft 35623sec 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:31:34.524 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:31:34.525 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:31:34.526 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T11:31:34.526 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T11:31:34.526 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:31:34.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:34.526 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T11:31:34.629 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T11:31:34.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:34.633 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T11:31:34.681 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T11:31:34.681 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T11:31:34.681 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T11:31:34.681 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T11:31:34.681 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T11:31:34.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:34.682 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T11:31:34.682 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T11:31:34.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T11:31:34.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T11:31:34.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T11:31:34.682 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T11:31:34.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:34.887 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T11:31:34.933 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T11:31:34.933 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T11:31:34.933 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:31:34.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:34.934 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T11:31:35.043 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:31:35.043 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T11:31:35.087 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:31:35.088 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:31:35.088 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:31:35.172 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T11:31:35.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:35.172 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:31:35.315 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:31:35.315 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:31:35.343 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:31:35.343 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:31:35.343 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T11:31:35.343 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T11:31:35.343 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T11:31:35.350 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T11:31:35.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:35.350 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:31:35.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:35.417 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T11:31:35.522 INFO:teuthology.orchestra.run.smithi012.stdout:client10804 2024-03-29T11:31:35.522 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:31:35.522 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:31:35.566 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-03-29T11:31:35.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.815+0000 7f409ac00700 1 Processor -- start 2024-03-29T11:31:35.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.816+0000 7f409ac00700 1 -- start start 2024-03-29T11:31:35.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f409ac00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f409406e9c0 0x7f409415ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:35.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f409ac00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f409415b160 0x7f40940b2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:35.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f409ac00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4094070980 con 0x7f40941580c0 2024-03-29T11:31:35.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f409ac00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4094071570 con 0x7f409415b160 2024-03-29T11:31:35.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f409ac00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4094071990 con 0x7f409406e9c0 2024-03-29T11:31:35.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f4098c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f409415b160 0x7f40940b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:35.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f4098c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f409415b160 0x7f40940b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36948/0 (socket says 172.21.15.12:36948) 2024-03-29T11:31:35.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f4098c00700 1 -- 172.21.15.12:0/1230268730 learned_addr learned my addr 172.21.15.12:0/1230268730 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:35.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f4093400700 1 --2- 172.21.15.12:0/1230268730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f409406e9c0 0x7f409415ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:35.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f4098c00700 1 -- 172.21.15.12:0/1230268730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f409406e9c0 msgr2=0x7f409415ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:35.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f4098c00700 1 --2- 172.21.15.12:0/1230268730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f409406e9c0 0x7f409415ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:35.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f4098c00700 1 -- 172.21.15.12:0/1230268730 >> v1:172.21.15.159:6789/0 conn(0x7f40941580c0 legacy=0x7f409406e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:35.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.817+0000 7f4098c00700 1 -- 172.21.15.12:0/1230268730 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4094070bd0 con 0x7f409415b160 2024-03-29T11:31:35.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.818+0000 7f4098c00700 1 --2- 172.21.15.12:0/1230268730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f409415b160 0x7f40940b2180 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f408c009e50 tx=0x7f408c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e05cf27e6b12313 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:35.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.818+0000 7f4092a00700 1 -- 172.21.15.12:0/1230268730 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c00e070 con 0x7f409415b160 2024-03-29T11:31:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.818+0000 7f4092a00700 1 -- 172.21.15.12:0/1230268730 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408c00b910 con 0x7f409415b160 2024-03-29T11:31:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.818+0000 7f4092a00700 1 -- 172.21.15.12:0/1230268730 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c00bcc0 con 0x7f409415b160 2024-03-29T11:31:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.818+0000 7f409ac00700 1 -- 172.21.15.12:0/1230268730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f409415b160 msgr2=0x7f40940b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.818+0000 7f409ac00700 1 --2- 172.21.15.12:0/1230268730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f409415b160 0x7f40940b2180 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f408c009e50 tx=0x7f408c00b330 comp rx=0 tx=0).stop 2024-03-29T11:31:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.819+0000 7f409ac00700 1 -- 172.21.15.12:0/1230268730 shutdown_connections 2024-03-29T11:31:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.819+0000 7f409ac00700 1 --2- 172.21.15.12:0/1230268730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f409415b160 0x7f40940b2180 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.819+0000 7f409ac00700 1 --2- 172.21.15.12:0/1230268730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f409406e9c0 0x7f409415ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.819+0000 7f409ac00700 1 -- 172.21.15.12:0/1230268730 >> 172.21.15.12:0/1230268730 conn(0x7f40940cf210 msgr2=0x7f40940cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.819+0000 7f409ac00700 1 -- 172.21.15.12:0/1230268730 shutdown_connections 2024-03-29T11:31:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.819+0000 7f409ac00700 1 -- 172.21.15.12:0/1230268730 wait complete. 2024-03-29T11:31:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.821+0000 7f409ac00700 1 Processor -- start 2024-03-29T11:31:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.821+0000 7f409ac00700 1 -- start start 2024-03-29T11:31:35.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f409ac00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f409406e9c0 0x7f40941513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:35.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f409ac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40941580c0 0x7f40940b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f409ac00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f409415b160 0x7f40940b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f409ac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409415bc20 con 0x7f40941580c0 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f409ac00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409415bf00 con 0x7f409406e9c0 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f409ac00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409415c1e0 con 0x7f409415b160 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f4093400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40941580c0 0x7f40940b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f4093400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40941580c0 0x7f40940b0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36956/0 (socket says 172.21.15.12:36956) 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f4093400700 1 -- 172.21.15.12:0/1683722770 learned_addr learned my addr 172.21.15.12:0/1683722770 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f4093e00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f409406e9c0 0x7f40941513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f4098c00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f409415b160 0x7f40940b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f4093400700 1 -- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f409415b160 msgr2=0x7f40940b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.822+0000 7f4093400700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f409415b160 0x7f40940b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.823+0000 7f4093400700 1 -- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f409406e9c0 msgr2=0x7f40941513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.823+0000 7f4093400700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f409406e9c0 0x7f40941513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.823+0000 7f4093400700 1 -- 172.21.15.12:0/1683722770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c009b30 con 0x7f40941580c0 2024-03-29T11:31:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.823+0000 7f4093400700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40941580c0 0x7f40940b0150 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f408000e9b0 tx=0x7f408000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.823+0000 7f408be00700 1 -- 172.21.15.12:0/1683722770 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080009710 con 0x7f40941580c0 2024-03-29T11:31:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.823+0000 7f408be00700 1 -- 172.21.15.12:0/1683722770 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408000ee50 con 0x7f40941580c0 2024-03-29T11:31:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.823+0000 7f409ac00700 1 -- 172.21.15.12:0/1683722770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409415c4c0 con 0x7f40941580c0 2024-03-29T11:31:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.823+0000 7f408be00700 1 -- 172.21.15.12:0/1683722770 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408000ca50 con 0x7f40941580c0 2024-03-29T11:31:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.823+0000 7f409ac00700 1 -- 172.21.15.12:0/1683722770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409415d090 con 0x7f40941580c0 2024-03-29T11:31:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.824+0000 7f409ac00700 1 -- 172.21.15.12:0/1683722770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40940b4710 con 0x7f40941580c0 2024-03-29T11:31:35.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.825+0000 7f408be00700 1 -- 172.21.15.12:0/1683722770 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f408000cbb0 con 0x7f40941580c0 2024-03-29T11:31:35.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.826+0000 7f408be00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f407406adb0 0x7f407406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:35.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.826+0000 7f408be00700 1 -- 172.21.15.12:0/1683722770 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f40800894b0 con 0x7f40941580c0 2024-03-29T11:31:35.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.826+0000 7f4093e00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f407406adb0 0x7f407406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:35.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.827+0000 7f4093e00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f407406adb0 0x7f407406d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f409415ab90 tx=0x7f4084009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:35.831+0000 7f408be00700 1 -- 172.21.15.12:0/1683722770 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4080054520 con 0x7f40941580c0 2024-03-29T11:31:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.097+0000 7f409ac00700 1 -- 172.21.15.12:0/1683722770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4094067720 con 0x7f40941580c0 2024-03-29T11:31:36.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.311+0000 7f408be00700 1 -- 172.21.15.12:0/1683722770 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v107) v1 ==== 136+0+0 (secure 0 0 0) 0x7f408000ce60 con 0x7f40941580c0 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.314+0000 7f409ac00700 1 -- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f407406adb0 msgr2=0x7f407406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.314+0000 7f409ac00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f407406adb0 0x7f407406d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f409415ab90 tx=0x7f4084009040 comp rx=0 tx=0).stop 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.314+0000 7f409ac00700 1 -- 172.21.15.12:0/1683722770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40941580c0 msgr2=0x7f40940b0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.314+0000 7f409ac00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40941580c0 0x7f40940b0150 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f408000e9b0 tx=0x7f408000ecc0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.314+0000 7f409ac00700 1 -- 172.21.15.12:0/1683722770 shutdown_connections 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.314+0000 7f409ac00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f407406adb0 0x7f407406d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.314+0000 7f409ac00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f409415b160 0x7f40940b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.314+0000 7f409ac00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f409406e9c0 0x7f40941513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.314+0000 7f409ac00700 1 --2- 172.21.15.12:0/1683722770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40941580c0 0x7f40940b0150 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.314+0000 7f409ac00700 1 -- 172.21.15.12:0/1683722770 >> 172.21.15.12:0/1683722770 conn(0x7f40940cf210 msgr2=0x7f409406aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.315+0000 7f409ac00700 1 -- 172.21.15.12:0/1683722770 shutdown_connections 2024-03-29T11:31:36.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.315+0000 7f409ac00700 1 -- 172.21.15.12:0/1683722770 wait complete. 2024-03-29T11:31:36.329 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:36.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.533+0000 7f9acfc00700 1 Processor -- start 2024-03-29T11:31:36.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.533+0000 7f9acfc00700 1 -- start start 2024-03-29T11:31:36.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.534+0000 7f9acfc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac80ce590 0x7f9ac80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.534+0000 7f9acfc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac80ceef0 con 0x7f9ac80b6a20 2024-03-29T11:31:36.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.534+0000 7f9acfc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac8160fc0 con 0x7f9ac8160be0 2024-03-29T11:31:36.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.534+0000 7f9acfc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80b65d0 con 0x7f9ac80ce590 2024-03-29T11:31:36.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.534+0000 7f9acdc00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9ac80b6a20 0x7f9ac80b6eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57674/0 (socket says 172.21.15.12:57674) 2024-03-29T11:31:36.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.534+0000 7f9acdc00700 1 -- 172.21.15.12:0/2548672348 learned_addr learned my addr 172.21.15.12:0/2548672348 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:36.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.534+0000 7f9ac7e00700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac80ce590 0x7f9ac80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454071148 0 0) 0x7f9ac80ceef0 con 0x7f9ac80b6a20 2024-03-29T11:31:36.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab00035b0 con 0x7f9ac80b6a20 2024-03-29T11:31:36.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137662971 0 0) 0x7f9ac8160fc0 con 0x7f9ac8160be0 2024-03-29T11:31:36.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac80ceef0 con 0x7f9ac8160be0 2024-03-29T11:31:36.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478300394 0 0) 0x7f9ab00035b0 con 0x7f9ac80b6a20 2024-03-29T11:31:36.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac80ce590 msgr2=0x7f9ac80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:36.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac80ce590 0x7f9ac80ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 >> v1:172.21.15.159:6789/0 conn(0x7f9ac8160be0 legacy=0x7f9ac80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac80d21e0 con 0x7f9ac80b6a20 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f9ac00019f0 con 0x7f9ac80b6a20 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7e00700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac80ce590 0x7f9ac80ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 >> v1:172.21.15.12:6789/0 conn(0x7f9ac80b6a20 legacy=0x7f9ac80b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab0003bc0 0x7f9ab0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac80ce590 0x7f9ab0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ab0004d90 0x7f9ab0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80b65d0 con 0x7f9ab0004d90 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab000bac0 con 0x7f9ab0003bc0 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab000bc00 con 0x7f9ac80ce590 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.535+0000 7f9acdc00700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ab0004d90 0x7f9ab0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9ac7e00700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac80ce590 0x7f9ab0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9acd200700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab0003bc0 0x7f9ab0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9acdc00700 1 -- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac80ce590 msgr2=0x7f9ab0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9acdc00700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac80ce590 0x7f9ab0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9acdc00700 1 -- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab0003bc0 msgr2=0x7f9ab0003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9acdc00700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab0003bc0 0x7f9ab0003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9acdc00700 1 -- 172.21.15.12:0/2548672348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab40094a0 con 0x7f9ab0004d90 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9acdc00700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ab0004d90 0x7f9ab0005210 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f9ac000a7d0 tx=0x7f9ab0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c2cfb6689c7c3a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0012600 con 0x7f9ab0004d90 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac0012780 con 0x7f9ab0004d90 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.536+0000 7f9ac7400700 1 -- 172.21.15.12:0/2548672348 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0012b30 con 0x7f9ab0004d90 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.537+0000 7f9acfc00700 1 -- 172.21.15.12:0/2548672348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ab0004d90 msgr2=0x7f9ab0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.537+0000 7f9acfc00700 1 --2- 172.21.15.12:0/2548672348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ab0004d90 0x7f9ab0005210 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f9ac000a7d0 tx=0x7f9ab0005880 comp rx=0 tx=0).stop 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.537+0000 7f9acdc00700 1 -- 172.21.15.12:0/2548672348 reap_dead start 2024-03-29T11:31:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.537+0000 7f9acfc00700 1 -- 172.21.15.12:0/2548672348 shutdown_connections 2024-03-29T11:31:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.537+0000 7f9acfc00700 1 -- 172.21.15.12:0/2548672348 >> 172.21.15.12:0/2548672348 conn(0x7f9ac808aff0 msgr2=0x7f9ac80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.537+0000 7f9acfc00700 1 -- 172.21.15.12:0/2548672348 shutdown_connections 2024-03-29T11:31:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.537+0000 7f9acfc00700 1 -- 172.21.15.12:0/2548672348 wait complete. 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.539+0000 7f9acfc00700 1 Processor -- start 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.539+0000 7f9acfc00700 1 -- start start 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.540+0000 7f9acfc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab0004d90 0x7f9ac8158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.540+0000 7f9acfc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac80c9af0 0x7f9ac81f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.540+0000 7f9acfc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac81562e0 0x7f9ac8156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.540+0000 7f9acfc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac8159580 con 0x7f9ac80c9af0 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.540+0000 7f9acfc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac814b980 con 0x7f9ab0004d90 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.540+0000 7f9acfc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac8156ca0 con 0x7f9ac81562e0 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.540+0000 7f9ac7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac80c9af0 0x7f9ac81f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.540+0000 7f9ac7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac80c9af0 0x7f9ac81f70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36980/0 (socket says 172.21.15.12:36980) 2024-03-29T11:31:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.540+0000 7f9ac7e00700 1 -- 172.21.15.12:0/2890581754 learned_addr learned my addr 172.21.15.12:0/2890581754 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9acd200700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab0004d90 0x7f9ac8158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9acdc00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac81562e0 0x7f9ac8156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9ac7e00700 1 -- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac81562e0 msgr2=0x7f9ac8156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9ac7e00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac81562e0 0x7f9ac8156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9ac7e00700 1 -- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab0004d90 msgr2=0x7f9ac8158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9ac7e00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab0004d90 0x7f9ac8158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9ac7e00700 1 -- 172.21.15.12:0/2890581754 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac8158900 con 0x7f9ac80c9af0 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9acdc00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac81562e0 0x7f9ac8156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9acd200700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab0004d90 0x7f9ac8158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9ac7e00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac80c9af0 0x7f9ac81f70a0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4011cd0 tx=0x7f9ab400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9ac4c00700 1 -- 172.21.15.12:0/2890581754 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4011e70 con 0x7f9ac80c9af0 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9ac4c00700 1 -- 172.21.15.12:0/2890581754 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab400fd40 con 0x7f9ac80c9af0 2024-03-29T11:31:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9acfc00700 1 -- 172.21.15.12:0/2890581754 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac8158be0 con 0x7f9ac80c9af0 2024-03-29T11:31:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9ac4c00700 1 -- 172.21.15.12:0/2890581754 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4004240 con 0x7f9ac80c9af0 2024-03-29T11:31:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.541+0000 7f9acfc00700 1 -- 172.21.15.12:0/2890581754 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac81591a0 con 0x7f9ac80c9af0 2024-03-29T11:31:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.542+0000 7f9acfc00700 1 -- 172.21.15.12:0/2890581754 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac8067720 con 0x7f9ac80c9af0 2024-03-29T11:31:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.543+0000 7f9ac4c00700 1 -- 172.21.15.12:0/2890581754 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9ab40043a0 con 0x7f9ac80c9af0 2024-03-29T11:31:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.544+0000 7f9ac4c00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa806adb0 0x7f9aa806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.544+0000 7f9ac4c00700 1 -- 172.21.15.12:0/2890581754 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f9ab4089c70 con 0x7f9ac80c9af0 2024-03-29T11:31:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.545+0000 7f9acd200700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa806adb0 0x7f9aa806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.545+0000 7f9acd200700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa806adb0 0x7f9aa806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9abc007fd0 tx=0x7f9abc009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:36.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.549+0000 7f9ac4c00700 1 -- 172.21.15.12:0/2890581754 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ab4054ce0 con 0x7f9ac80c9af0 2024-03-29T11:31:36.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.763+0000 7f9acfc00700 1 -- 172.21.15.12:0/2890581754 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ac815faa0 con 0x7f9ac80c9af0 2024-03-29T11:31:36.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.764+0000 7f9ac4c00700 1 -- 172.21.15.12:0/2890581754 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v227) v1 ==== 74+0+16231 (secure 0 0 0) 0x7f9ab4022070 con 0x7f9ac80c9af0 2024-03-29T11:31:36.758 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:36.758 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":227,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:27.818673+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.766+0000 7f9acfc00700 1 -- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa806adb0 msgr2=0x7f9aa806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.766+0000 7f9acfc00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa806adb0 0x7f9aa806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9abc007fd0 tx=0x7f9abc009040 comp rx=0 tx=0).stop 2024-03-29T11:31:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.766+0000 7f9acfc00700 1 -- 172.21.15.12:0/2890581754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac80c9af0 msgr2=0x7f9ac81f70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.766+0000 7f9acfc00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac80c9af0 0x7f9ac81f70a0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4011cd0 tx=0x7f9ab400f520 comp rx=0 tx=0).stop 2024-03-29T11:31:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.767+0000 7f9acfc00700 1 -- 172.21.15.12:0/2890581754 shutdown_connections 2024-03-29T11:31:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.767+0000 7f9acfc00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa806adb0 0x7f9aa806d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.767+0000 7f9acfc00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac81562e0 0x7f9ac8156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.767+0000 7f9acfc00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab0004d90 0x7f9ac8158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.767+0000 7f9acfc00700 1 --2- 172.21.15.12:0/2890581754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac80c9af0 0x7f9ac81f70a0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.767+0000 7f9acfc00700 1 -- 172.21.15.12:0/2890581754 >> 172.21.15.12:0/2890581754 conn(0x7f9ac808aff0 msgr2=0x7f9ac80b8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:36.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.767+0000 7f9acfc00700 1 -- 172.21.15.12:0/2890581754 shutdown_connections 2024-03-29T11:31:36.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.767+0000 7f9acfc00700 1 -- 172.21.15.12:0/2890581754 wait complete. 2024-03-29T11:31:36.779 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:31:24.485419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:31:25.823318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:31:36.779 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:31:36.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.988+0000 7f933ce00700 1 Processor -- start 2024-03-29T11:31:36.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.988+0000 7f933ce00700 1 -- start start 2024-03-29T11:31:36.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.989+0000 7f933ce00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f933815d9d0 0x7f93380d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.989+0000 7f933ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93380d16a0 0x7f93380d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.989+0000 7f933ce00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9338161420 con 0x7f93380d2370 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.989+0000 7f933ce00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93380d2820 con 0x7f933815d9d0 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.989+0000 7f933ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9338161700 con 0x7f93380d16a0 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.989+0000 7f9335e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f933815d9d0 0x7f93380d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9335e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f933815d9d0 0x7f93380d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36996/0 (socket says 172.21.15.12:36996) 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9336800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f93380d2370 0x7f933815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57690/0 (socket says 172.21.15.12:57690) 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9335e00700 1 -- 172.21.15.12:0/3021582795 learned_addr learned my addr 172.21.15.12:0/3021582795 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9335400700 1 --2- 172.21.15.12:0/3021582795 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93380d16a0 0x7f93380d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9335e00700 1 -- 172.21.15.12:0/3021582795 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93380d16a0 msgr2=0x7f93380d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9335e00700 1 --2- 172.21.15.12:0/3021582795 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93380d16a0 0x7f93380d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9335e00700 1 -- 172.21.15.12:0/3021582795 >> v1:172.21.15.12:6789/0 conn(0x7f93380d2370 legacy=0x7f933815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9335e00700 1 -- 172.21.15.12:0/3021582795 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93381619e0 con 0x7f933815d9d0 2024-03-29T11:31:36.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9334a00700 1 -- 172.21.15.12:0/3021582795 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250197646 0 0) 0x7f9338161420 con 0x7f93380d2370 2024-03-29T11:31:36.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9335e00700 1 --2- 172.21.15.12:0/3021582795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933815d9d0 0x7f93380d3c30 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f93280098f0 tx=0x7f9328009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd83b8c1cfd9f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:36.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.990+0000 7f9334a00700 1 -- 172.21.15.12:0/3021582795 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932800b8a0 con 0x7f933815d9d0 2024-03-29T11:31:36.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.991+0000 7f9334a00700 1 -- 172.21.15.12:0/3021582795 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932800ba00 con 0x7f933815d9d0 2024-03-29T11:31:36.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.991+0000 7f9334a00700 1 -- 172.21.15.12:0/3021582795 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932800bd90 con 0x7f933815d9d0 2024-03-29T11:31:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.991+0000 7f933ce00700 1 -- 172.21.15.12:0/3021582795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933815d9d0 msgr2=0x7f93380d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.991+0000 7f933ce00700 1 --2- 172.21.15.12:0/3021582795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933815d9d0 0x7f93380d3c30 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f93280098f0 tx=0x7f9328009c00 comp rx=0 tx=0).stop 2024-03-29T11:31:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.991+0000 7f933ce00700 1 -- 172.21.15.12:0/3021582795 shutdown_connections 2024-03-29T11:31:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.991+0000 7f933ce00700 1 --2- 172.21.15.12:0/3021582795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933815d9d0 0x7f93380d3c30 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.991+0000 7f933ce00700 1 --2- 172.21.15.12:0/3021582795 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93380d16a0 0x7f93380d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.991+0000 7f933ce00700 1 -- 172.21.15.12:0/3021582795 >> 172.21.15.12:0/3021582795 conn(0x7f933808aff0 msgr2=0x7f93380d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.991+0000 7f933ce00700 1 -- 172.21.15.12:0/3021582795 shutdown_connections 2024-03-29T11:31:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.991+0000 7f933ce00700 1 -- 172.21.15.12:0/3021582795 wait complete. 2024-03-29T11:31:36.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.993+0000 7f933ce00700 1 Processor -- start 2024-03-29T11:31:36.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.993+0000 7f933ce00700 1 -- start start 2024-03-29T11:31:36.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.994+0000 7f933ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93380d16a0 0x7f9338158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.994+0000 7f933ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93380d2370 0x7f93380b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.994+0000 7f933ce00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f933815d9d0 0x7f93380b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.994+0000 7f933ce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93380b75a0 con 0x7f93380d2370 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.994+0000 7f933ce00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9338160390 con 0x7f933815d9d0 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.994+0000 7f933ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93380b7790 con 0x7f93380d16a0 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.994+0000 7f9335400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93380d2370 0x7f93380b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.994+0000 7f9335400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93380d2370 0x7f93380b7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36998/0 (socket says 172.21.15.12:36998) 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.994+0000 7f9335400700 1 -- 172.21.15.12:0/665937816 learned_addr learned my addr 172.21.15.12:0/665937816 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f9335e00700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93380d16a0 0x7f9338158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f9336800700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f933815d9d0 0x7f93380b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f9335400700 1 -- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93380d16a0 msgr2=0x7f9338158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f9335400700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93380d16a0 0x7f9338158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f9335400700 1 -- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f933815d9d0 msgr2=0x7f93380b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:36.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f9335400700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f933815d9d0 0x7f93380b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:36.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f9335400700 1 -- 172.21.15.12:0/665937816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9328009550 con 0x7f93380d2370 2024-03-29T11:31:36.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f9335400700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93380d2370 0x7f93380b7060 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f932400e810 tx=0x7f932400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:36.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f932e000700 1 -- 172.21.15.12:0/665937816 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932400c070 con 0x7f93380d2370 2024-03-29T11:31:36.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f932e000700 1 -- 172.21.15.12:0/665937816 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9324013610 con 0x7f93380d2370 2024-03-29T11:31:36.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f932e000700 1 -- 172.21.15.12:0/665937816 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93240139c0 con 0x7f93380d2370 2024-03-29T11:31:36.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.995+0000 7f933ce00700 1 -- 172.21.15.12:0/665937816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933815de50 con 0x7f93380d2370 2024-03-29T11:31:36.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.996+0000 7f933ce00700 1 -- 172.21.15.12:0/665937816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933815e1e0 con 0x7f93380d2370 2024-03-29T11:31:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.997+0000 7f933ce00700 1 -- 172.21.15.12:0/665937816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93380d1ca0 con 0x7f93380d2370 2024-03-29T11:31:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.998+0000 7f932e000700 1 -- 172.21.15.12:0/665937816 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9324013b20 con 0x7f93380d2370 2024-03-29T11:31:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.998+0000 7f932e000700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f931806adb0 0x7f931806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.998+0000 7f932e000700 1 -- 172.21.15.12:0/665937816 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f9324089240 con 0x7f93380d2370 2024-03-29T11:31:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:36.999+0000 7f9335e00700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f931806adb0 0x7f931806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.000+0000 7f9335e00700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f931806adb0 0x7f931806d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9328014040 tx=0x7f9328008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:36.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.003+0000 7f932e000700 1 -- 172.21.15.12:0/665937816 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9324054240 con 0x7f93380d2370 2024-03-29T11:31:37.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.217+0000 7f933ce00700 1 -- 172.21.15.12:0/665937816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f933815e620 con 0x7f93380d2370 2024-03-29T11:31:37.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.217+0000 7f932e000700 1 -- 172.21.15.12:0/665937816 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v227) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9324057860 con 0x7f93380d2370 2024-03-29T11:31:37.212 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:31:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.219+0000 7f933ce00700 1 -- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f931806adb0 msgr2=0x7f931806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.219+0000 7f933ce00700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f931806adb0 0x7f931806d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9328014040 tx=0x7f9328008030 comp rx=0 tx=0).stop 2024-03-29T11:31:37.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.219+0000 7f933ce00700 1 -- 172.21.15.12:0/665937816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93380d2370 msgr2=0x7f93380b7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.219+0000 7f933ce00700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93380d2370 0x7f93380b7060 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f932400e810 tx=0x7f932400eb20 comp rx=0 tx=0).stop 2024-03-29T11:31:37.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.220+0000 7f933ce00700 1 -- 172.21.15.12:0/665937816 shutdown_connections 2024-03-29T11:31:37.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.220+0000 7f933ce00700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f931806adb0 0x7f931806d270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.220+0000 7f933ce00700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f933815d9d0 0x7f93380b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.220+0000 7f933ce00700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93380d16a0 0x7f9338158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.220+0000 7f933ce00700 1 --2- 172.21.15.12:0/665937816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93380d2370 0x7f93380b7060 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.220+0000 7f933ce00700 1 -- 172.21.15.12:0/665937816 >> 172.21.15.12:0/665937816 conn(0x7f933808aff0 msgr2=0x7f93380ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:37.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.220+0000 7f933ce00700 1 -- 172.21.15.12:0/665937816 shutdown_connections 2024-03-29T11:31:37.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.220+0000 7f933ce00700 1 -- 172.21.15.12:0/665937816 wait complete. 2024-03-29T11:31:37.232 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:31:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.441+0000 7f49b5a00700 1 Processor -- start 2024-03-29T11:31:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.441+0000 7f49b5a00700 1 -- start start 2024-03-29T11:31:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.442+0000 7f49b5a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cbb30 0x7f49b00cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.442+0000 7f49b5a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f49b00cd4a0 0x7f49b006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.442+0000 7f49b5a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b00cc450 con 0x7f49b00cd020 2024-03-29T11:31:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.442+0000 7f49b5a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b00cdae0 con 0x7f49b00cd4a0 2024-03-29T11:31:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.442+0000 7f49b5a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00cddc0 con 0x7f49b00cbb30 2024-03-29T11:31:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.443+0000 7f49af400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f49b00cd4a0 0x7f49b006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.443+0000 7f49af400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f49b00cd4a0 0x7f49b006e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37000/0 (socket says 172.21.15.12:37000) 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.443+0000 7f49af400700 1 -- 172.21.15.12:0/1853185516 learned_addr learned my addr 172.21.15.12:0/1853185516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.443+0000 7f49aea00700 1 --2- 172.21.15.12:0/1853185516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cbb30 0x7f49b00cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.443+0000 7f49af400700 1 -- 172.21.15.12:0/1853185516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cbb30 msgr2=0x7f49b00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.443+0000 7f49af400700 1 --2- 172.21.15.12:0/1853185516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cbb30 0x7f49b00cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.443+0000 7f49af400700 1 -- 172.21.15.12:0/1853185516 >> v1:172.21.15.159:6789/0 conn(0x7f49b00cd020 legacy=0x7f49b00b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.443+0000 7f49af400700 1 -- 172.21.15.12:0/1853185516 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b00ce0a0 con 0x7f49b00cd4a0 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.444+0000 7f49af400700 1 --2- 172.21.15.12:0/1853185516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 0x7f49b006e420 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009e50 tx=0x7f49a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2605a0e3df4f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.444+0000 7f49ad600700 1 -- 172.21.15.12:0/1853185516 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a800e070 con 0x7f49b00cd4a0 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.444+0000 7f49ad600700 1 -- 172.21.15.12:0/1853185516 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a800b9c0 con 0x7f49b00cd4a0 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.444+0000 7f49ad600700 1 -- 172.21.15.12:0/1853185516 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a800bd70 con 0x7f49b00cd4a0 2024-03-29T11:31:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.444+0000 7f49b5a00700 1 -- 172.21.15.12:0/1853185516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 msgr2=0x7f49b006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.444+0000 7f49b5a00700 1 --2- 172.21.15.12:0/1853185516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 0x7f49b006e420 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009e50 tx=0x7f49a800b330 comp rx=0 tx=0).stop 2024-03-29T11:31:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.444+0000 7f49b5a00700 1 -- 172.21.15.12:0/1853185516 shutdown_connections 2024-03-29T11:31:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.444+0000 7f49b5a00700 1 --2- 172.21.15.12:0/1853185516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 0x7f49b006e420 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.444+0000 7f49b5a00700 1 --2- 172.21.15.12:0/1853185516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cbb30 0x7f49b00cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.444+0000 7f49b5a00700 1 -- 172.21.15.12:0/1853185516 >> 172.21.15.12:0/1853185516 conn(0x7f49b00b0d70 msgr2=0x7f49b00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.445+0000 7f49b5a00700 1 -- 172.21.15.12:0/1853185516 shutdown_connections 2024-03-29T11:31:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.445+0000 7f49b5a00700 1 -- 172.21.15.12:0/1853185516 wait complete. 2024-03-29T11:31:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.446+0000 7f49b5a00700 1 Processor -- start 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.446+0000 7f49b5a00700 1 -- start start 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.447+0000 7f49b5a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f49b00cbb30 0x7f49b00accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.447+0000 7f49b5a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cd020 0x7f49b014ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.447+0000 7f49b5a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 0x7f49b00ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.447+0000 7f49b5a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b014f3d0 con 0x7f49b00cd4a0 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.447+0000 7f49b5a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00add00 con 0x7f49b00cbb30 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.447+0000 7f49b5a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00adf30 con 0x7f49b00cd020 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49af400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 0x7f49b00ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49af400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 0x7f49b00ad7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37006/0 (socket says 172.21.15.12:37006) 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49af400700 1 -- 172.21.15.12:0/2991741636 learned_addr learned my addr 172.21.15.12:0/2991741636 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49aea00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f49b00cbb30 0x7f49b00accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49af400700 1 -- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cd020 msgr2=0x7f49b014ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49ae000700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cd020 0x7f49b014ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49af400700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cd020 0x7f49b014ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49af400700 1 -- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f49b00cbb30 msgr2=0x7f49b00accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49af400700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f49b00cbb30 0x7f49b00accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49af400700 1 -- 172.21.15.12:0/2991741636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a8009b30 con 0x7f49b00cd4a0 2024-03-29T11:31:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49aea00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f49b00cbb30 0x7f49b00accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49ae000700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cd020 0x7f49b014ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.448+0000 7f49af400700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 0x7f49b00ad7c0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f49a800b730 tx=0x7f49a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.449+0000 7f49a6a00700 1 -- 172.21.15.12:0/2991741636 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a80099b0 con 0x7f49b00cd4a0 2024-03-29T11:31:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.449+0000 7f49a6a00700 1 -- 172.21.15.12:0/2991741636 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a80092e0 con 0x7f49b00cd4a0 2024-03-29T11:31:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.449+0000 7f49a6a00700 1 -- 172.21.15.12:0/2991741636 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a801c640 con 0x7f49b00cd4a0 2024-03-29T11:31:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.449+0000 7f49b5a00700 1 -- 172.21.15.12:0/2991741636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b0070a60 con 0x7f49b00cd4a0 2024-03-29T11:31:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.449+0000 7f49b5a00700 1 -- 172.21.15.12:0/2991741636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b0071020 con 0x7f49b00cd4a0 2024-03-29T11:31:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.450+0000 7f49b5a00700 1 -- 172.21.15.12:0/2991741636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b00ce540 con 0x7f49b00cd4a0 2024-03-29T11:31:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.451+0000 7f49a6a00700 1 -- 172.21.15.12:0/2991741636 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f49a801c7a0 con 0x7f49b00cd4a0 2024-03-29T11:31:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.452+0000 7f49a6a00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f499006adb0 0x7f499006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.452+0000 7f49a6a00700 1 -- 172.21.15.12:0/2991741636 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f49a808b350 con 0x7f49b00cd4a0 2024-03-29T11:31:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.452+0000 7f49aea00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f499006adb0 0x7f499006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.453+0000 7f49aea00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f499006adb0 0x7f499006d270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009820 tx=0x7f49a0008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.457+0000 7f49a6a00700 1 -- 172.21.15.12:0/2991741636 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f49a80562b0 con 0x7f49b00cd4a0 2024-03-29T11:31:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.673+0000 7f49b5a00700 1 -- 172.21.15.12:0/2991741636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f49b0067720 con 0x7f49b00cd4a0 2024-03-29T11:31:37.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.673+0000 7f49a6a00700 1 -- 172.21.15.12:0/2991741636 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v227) v1 ==== 104+0+11 (secure 0 0 0) 0x7f49a80598d0 con 0x7f49b00cd4a0 2024-03-29T11:31:37.668 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:31:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.675+0000 7f49b5a00700 1 -- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f499006adb0 msgr2=0x7f499006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.675+0000 7f49b5a00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f499006adb0 0x7f499006d270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009820 tx=0x7f49a0008040 comp rx=0 tx=0).stop 2024-03-29T11:31:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.675+0000 7f49b5a00700 1 -- 172.21.15.12:0/2991741636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 msgr2=0x7f49b00ad7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.675+0000 7f49b5a00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 0x7f49b00ad7c0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f49a800b730 tx=0x7f49a8004ad0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.676+0000 7f49b5a00700 1 -- 172.21.15.12:0/2991741636 shutdown_connections 2024-03-29T11:31:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.676+0000 7f49b5a00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f499006adb0 0x7f499006d270 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.676+0000 7f49b5a00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b00cd4a0 0x7f49b00ad7c0 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.676+0000 7f49b5a00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f49b00cbb30 0x7f49b00accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.676+0000 7f49b5a00700 1 --2- 172.21.15.12:0/2991741636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f49b00cd020 0x7f49b014ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.676+0000 7f49b5a00700 1 -- 172.21.15.12:0/2991741636 >> 172.21.15.12:0/2991741636 conn(0x7f49b00b0d70 msgr2=0x7f49b006e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.676+0000 7f49b5a00700 1 -- 172.21.15.12:0/2991741636 shutdown_connections 2024-03-29T11:31:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.676+0000 7f49b5a00700 1 -- 172.21.15.12:0/2991741636 wait complete. 2024-03-29T11:31:37.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:31:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.898+0000 7f4227000700 1 Processor -- start 2024-03-29T11:31:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.898+0000 7f4227000700 1 -- start start 2024-03-29T11:31:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.899+0000 7f4227000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f422015d9d0 0x7f42200d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.899+0000 7f4227000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f42200d2540 0x7f42200ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.899+0000 7f4227000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4220161420 con 0x7f42200d16a0 2024-03-29T11:31:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.899+0000 7f4227000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422015edc0 con 0x7f422015d9d0 2024-03-29T11:31:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.899+0000 7f4227000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4220161700 con 0x7f42200d2540 2024-03-29T11:31:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.899+0000 7f421fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f422015d9d0 0x7f42200d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.899+0000 7f421fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f422015d9d0 0x7f42200d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37022/0 (socket says 172.21.15.12:37022) 2024-03-29T11:31:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.899+0000 7f421fe00700 1 -- 172.21.15.12:0/3644945604 learned_addr learned my addr 172.21.15.12:0/3644945604 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.899+0000 7f4225000700 1 --2- 172.21.15.12:0/3644945604 >> v2:172.21.15.159:3300/0 conn(0x7f42200d2540 0x7f42200ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.900+0000 7f421fe00700 1 -- 172.21.15.12:0/3644945604 >> v2:172.21.15.159:3300/0 conn(0x7f42200d2540 msgr2=0x7f42200ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.900+0000 7f421fe00700 1 --2- 172.21.15.12:0/3644945604 >> v2:172.21.15.159:3300/0 conn(0x7f42200d2540 0x7f42200ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.900+0000 7f421fe00700 1 -- 172.21.15.12:0/3644945604 >> v1:172.21.15.12:6789/0 conn(0x7f42200d16a0 legacy=0x7f42200d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.900+0000 7f421fe00700 1 -- 172.21.15.12:0/3644945604 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42201619e0 con 0x7f422015d9d0 2024-03-29T11:31:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.900+0000 7f421fe00700 1 --2- 172.21.15.12:0/3644945604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422015d9d0 0x7f42200d3c30 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f422008ca10 tx=0x7f4210009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0617866bccbf2d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.900+0000 7f421ea00700 1 -- 172.21.15.12:0/3644945604 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210011830 con 0x7f422015d9d0 2024-03-29T11:31:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.900+0000 7f421ea00700 1 -- 172.21.15.12:0/3644945604 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4210011990 con 0x7f422015d9d0 2024-03-29T11:31:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.900+0000 7f421ea00700 1 -- 172.21.15.12:0/3644945604 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210011d20 con 0x7f422015d9d0 2024-03-29T11:31:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.901+0000 7f4227000700 1 -- 172.21.15.12:0/3644945604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422015d9d0 msgr2=0x7f42200d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.901+0000 7f4227000700 1 --2- 172.21.15.12:0/3644945604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422015d9d0 0x7f42200d3c30 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f422008ca10 tx=0x7f4210009bd0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.901+0000 7f4227000700 1 -- 172.21.15.12:0/3644945604 shutdown_connections 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.901+0000 7f4227000700 1 --2- 172.21.15.12:0/3644945604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422015d9d0 0x7f42200d3c30 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.901+0000 7f4227000700 1 --2- 172.21.15.12:0/3644945604 >> v2:172.21.15.159:3300/0 conn(0x7f42200d2540 0x7f42200ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.901+0000 7f4227000700 1 -- 172.21.15.12:0/3644945604 >> 172.21.15.12:0/3644945604 conn(0x7f422008aff0 msgr2=0x7f42200d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.901+0000 7f4227000700 1 -- 172.21.15.12:0/3644945604 shutdown_connections 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.901+0000 7f4227000700 1 -- 172.21.15.12:0/3644945604 wait complete. 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.903+0000 7f4227000700 1 Processor -- start 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.903+0000 7f4227000700 1 -- start start 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f4227000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42200d16a0 0x7f42201f37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f4227000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42200d2540 0x7f4220154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f4227000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f422015d9d0 0x7f422015a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f4227000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42200b4090 con 0x7f42200d16a0 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f4227000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42200b4930 con 0x7f422015d9d0 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f4227000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42200b77b0 con 0x7f42200d2540 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f421fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42200d16a0 0x7f42201f37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f421fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42200d16a0 0x7f42201f37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37038/0 (socket says 172.21.15.12:37038) 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f421fe00700 1 -- 172.21.15.12:0/1500173330 learned_addr learned my addr 172.21.15.12:0/1500173330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f421f400700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42200d2540 0x7f4220154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f4225000700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f422015d9d0 0x7f422015a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f421fe00700 1 -- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42200d2540 msgr2=0x7f4220154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f421fe00700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42200d2540 0x7f4220154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f421fe00700 1 -- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f422015d9d0 msgr2=0x7f422015a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f421fe00700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f422015d9d0 0x7f422015a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f421fe00700 1 -- 172.21.15.12:0/1500173330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4210009550 con 0x7f42200d16a0 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f4225000700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f422015d9d0 0x7f422015a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.904+0000 7f421f400700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42200d2540 0x7f4220154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.905+0000 7f421fe00700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42200d16a0 0x7f42201f37d0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f4210014040 tx=0x7f4210004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.905+0000 7f4217e00700 1 -- 172.21.15.12:0/1500173330 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42100036a0 con 0x7f42200d16a0 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.905+0000 7f4217e00700 1 -- 172.21.15.12:0/1500173330 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421001e070 con 0x7f42200d16a0 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.905+0000 7f4227000700 1 -- 172.21.15.12:0/1500173330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42200b42e0 con 0x7f42200d16a0 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.905+0000 7f4217e00700 1 -- 172.21.15.12:0/1500173330 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210003a60 con 0x7f42200d16a0 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.905+0000 7f4227000700 1 -- 172.21.15.12:0/1500173330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42201f3d10 con 0x7f42200d16a0 2024-03-29T11:31:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.906+0000 7f4227000700 1 -- 172.21.15.12:0/1500173330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42200d1ca0 con 0x7f42200d16a0 2024-03-29T11:31:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.907+0000 7f4217e00700 1 -- 172.21.15.12:0/1500173330 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4210003bc0 con 0x7f42200d16a0 2024-03-29T11:31:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.908+0000 7f4217e00700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f420006ade0 0x7f420006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.908+0000 7f4217e00700 1 -- 172.21.15.12:0/1500173330 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f421008a470 con 0x7f42200d16a0 2024-03-29T11:31:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.908+0000 7f421f400700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f420006ade0 0x7f420006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.909+0000 7f421f400700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f420006ade0 0x7f420006d2a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4220156140 tx=0x7f420c009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:37.912+0000 7f4217e00700 1 -- 172.21.15.12:0/1500173330 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f42100554e0 con 0x7f42200d16a0 2024-03-29T11:31:38.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.125+0000 7f4227000700 1 -- 172.21.15.12:0/1500173330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4220067720 con 0x7f42200d16a0 2024-03-29T11:31:38.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.125+0000 7f4217e00700 1 -- 172.21.15.12:0/1500173330 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v227) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4210058b00 con 0x7f42200d16a0 2024-03-29T11:31:38.120 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:31:38.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 -- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f420006ade0 msgr2=0x7f420006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f420006ade0 0x7f420006d2a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4220156140 tx=0x7f420c009040 comp rx=0 tx=0).stop 2024-03-29T11:31:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 -- 172.21.15.12:0/1500173330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42200d16a0 msgr2=0x7f42201f37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42200d16a0 0x7f42201f37d0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f4210014040 tx=0x7f4210004750 comp rx=0 tx=0).stop 2024-03-29T11:31:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 -- 172.21.15.12:0/1500173330 shutdown_connections 2024-03-29T11:31:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f420006ade0 0x7f420006d2a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f422015d9d0 0x7f422015a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42200d16a0 0x7f42201f37d0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 --2- 172.21.15.12:0/1500173330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42200d2540 0x7f4220154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 -- 172.21.15.12:0/1500173330 >> 172.21.15.12:0/1500173330 conn(0x7f422008aff0 msgr2=0x7f42200d1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 -- 172.21.15.12:0/1500173330 shutdown_connections 2024-03-29T11:31:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.128+0000 7f4227000700 1 -- 172.21.15.12:0/1500173330 wait complete. 2024-03-29T11:31:38.142 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T11:31:38.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.346+0000 7f104b800700 1 Processor -- start 2024-03-29T11:31:38.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.346+0000 7f104b800700 1 -- start start 2024-03-29T11:31:38.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.347+0000 7f104b800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f10440d18b0 0x7f10440cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.347+0000 7f104b800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10440ceda0 con 0x7f1044160f10 2024-03-29T11:31:38.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.347+0000 7f104b800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10441612f0 con 0x7f1044161590 2024-03-29T11:31:38.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.347+0000 7f104b800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10440d1d30 con 0x7f10440d18b0 2024-03-29T11:31:38.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.347+0000 7f1048e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1044160f10 0x7f10440d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57714/0 (socket says 172.21.15.12:57714) 2024-03-29T11:31:38.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.347+0000 7f1048e00700 1 -- 172.21.15.12:0/3340864579 learned_addr learned my addr 172.21.15.12:0/3340864579 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:38.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.348+0000 7f1043e00700 1 --2- 172.21.15.12:0/3340864579 >> v2:172.21.15.159:3300/0 conn(0x7f10440d18b0 0x7f10440cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.348+0000 7f1043400700 1 -- 172.21.15.12:0/3340864579 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703535265 0 0) 0x7f10440ceda0 con 0x7f1044160f10 2024-03-29T11:31:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.348+0000 7f1043400700 1 -- 172.21.15.12:0/3340864579 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f102c0035b0 con 0x7f1044160f10 2024-03-29T11:31:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.348+0000 7f1043400700 1 -- 172.21.15.12:0/3340864579 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806038050 0 0) 0x7f10441612f0 con 0x7f1044161590 2024-03-29T11:31:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.348+0000 7f1043400700 1 -- 172.21.15.12:0/3340864579 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10440ceda0 con 0x7f1044161590 2024-03-29T11:31:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.348+0000 7f1043e00700 1 -- 172.21.15.12:0/3340864579 >> v1:172.21.15.159:6789/0 conn(0x7f1044161590 legacy=0x7f10440b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.348+0000 7f1043e00700 1 -- 172.21.15.12:0/3340864579 >> v1:172.21.15.12:6789/0 conn(0x7f1044160f10 legacy=0x7f10440d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.348+0000 7f1043400700 1 -- 172.21.15.12:0/3340864579 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2034859732 0 0) 0x7f102c0035b0 con 0x7f1044160f10 2024-03-29T11:31:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.348+0000 7f1043e00700 1 -- 172.21.15.12:0/3340864579 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10440ce470 con 0x7f10440d18b0 2024-03-29T11:31:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.348+0000 7f1043e00700 1 --2- 172.21.15.12:0/3340864579 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10440d18b0 0x7f10440cf230 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1030009840 tx=0x7f1030009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47fa80717e2485bd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.349+0000 7f1043400700 1 -- 172.21.15.12:0/3340864579 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038001670 con 0x7f10440d18b0 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.349+0000 7f1043400700 1 -- 172.21.15.12:0/3340864579 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1030004030 con 0x7f10440d18b0 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.349+0000 7f1043400700 1 -- 172.21.15.12:0/3340864579 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10300043c0 con 0x7f10440d18b0 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.349+0000 7f104b800700 1 -- 172.21.15.12:0/3340864579 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10440d18b0 msgr2=0x7f10440cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.349+0000 7f104b800700 1 --2- 172.21.15.12:0/3340864579 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10440d18b0 0x7f10440cf230 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1030009840 tx=0x7f1030009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.349+0000 7f104b800700 1 -- 172.21.15.12:0/3340864579 shutdown_connections 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.349+0000 7f104b800700 1 --2- 172.21.15.12:0/3340864579 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10440d18b0 0x7f10440cf230 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.349+0000 7f104b800700 1 -- 172.21.15.12:0/3340864579 >> 172.21.15.12:0/3340864579 conn(0x7f104408aff0 msgr2=0x7f10440d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.349+0000 7f104b800700 1 -- 172.21.15.12:0/3340864579 shutdown_connections 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.349+0000 7f104b800700 1 -- 172.21.15.12:0/3340864579 wait complete. 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.351+0000 7f104b800700 1 Processor -- start 2024-03-29T11:31:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.351+0000 7f104b800700 1 -- start start 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.352+0000 7f104b800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10440d18b0 0x7f1044153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.352+0000 7f104b800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1044160f10 0x7f10441592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.352+0000 7f104b800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1044161590 0x7f104415a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.352+0000 7f104b800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104415add0 con 0x7f10440d18b0 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.352+0000 7f104b800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1044154290 con 0x7f1044160f10 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.352+0000 7f104b800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10441567b0 con 0x7f1044161590 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1048e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10440d18b0 0x7f1044153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1048e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10440d18b0 0x7f1044153c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37048/0 (socket says 172.21.15.12:37048) 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1048e00700 1 -- 172.21.15.12:0/2296123488 learned_addr learned my addr 172.21.15.12:0/2296123488 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1043e00700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1044160f10 0x7f10441592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1049800700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1044161590 0x7f104415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1048e00700 1 -- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1044161590 msgr2=0x7f104415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1048e00700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1044161590 0x7f104415a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1048e00700 1 -- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1044160f10 msgr2=0x7f10441592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1048e00700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1044160f10 0x7f10441592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1048e00700 1 -- 172.21.15.12:0/2296123488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10300094a0 con 0x7f10440d18b0 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1043e00700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1044160f10 0x7f10441592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1049800700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1044161590 0x7f104415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.353+0000 7f1048e00700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10440d18b0 0x7f1044153c40 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f103800c500 tx=0x7f103800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.354+0000 7f1040c00700 1 -- 172.21.15.12:0/2296123488 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038004c80 con 0x7f10440d18b0 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.354+0000 7f1040c00700 1 -- 172.21.15.12:0/2296123488 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1038004e00 con 0x7f10440d18b0 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.354+0000 7f1040c00700 1 -- 172.21.15.12:0/2296123488 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038007310 con 0x7f10440d18b0 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.354+0000 7f104b800700 1 -- 172.21.15.12:0/2296123488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1044156a90 con 0x7f10440d18b0 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.354+0000 7f104b800700 1 -- 172.21.15.12:0/2296123488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1044157050 con 0x7f10440d18b0 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.355+0000 7f104b800700 1 -- 172.21.15.12:0/2296123488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1044067720 con 0x7f10440d18b0 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.356+0000 7f1040c00700 1 -- 172.21.15.12:0/2296123488 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1038007470 con 0x7f10440d18b0 2024-03-29T11:31:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.357+0000 7f1040c00700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f102406adb0 0x7f102406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.357+0000 7f1040c00700 1 -- 172.21.15.12:0/2296123488 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f1038089250 con 0x7f10440d18b0 2024-03-29T11:31:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.357+0000 7f1043e00700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f102406adb0 0x7f102406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.358+0000 7f1043e00700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f102406adb0 0x7f102406d270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f10440a14b0 tx=0x7f1030013040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:38.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.361+0000 7f1040c00700 1 -- 172.21.15.12:0/2296123488 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1038054340 con 0x7f10440d18b0 2024-03-29T11:31:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.576+0000 7f104b800700 1 -- 172.21.15.12:0/2296123488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f104408b760 con 0x7f10440d18b0 2024-03-29T11:31:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.577+0000 7f1040c00700 1 -- 172.21.15.12:0/2296123488 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v227) v1 ==== 84+0+209 (secure 0 0 0) 0x7f1038057960 con 0x7f10440d18b0 2024-03-29T11:31:38.572 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:38.572 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:31:38.572 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:38.572 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T11:31:38.572 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T11:31:38.572 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:38.572 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:38.573 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 26, 2024-03-29T11:31:38.573 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": "cephfs_metadata" 2024-03-29T11:31:38.573 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:38.573 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:38.573 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 27, 2024-03-29T11:31:38.573 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": "cephfs_data" 2024-03-29T11:31:38.574 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:38.574 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:31:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 -- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f102406adb0 msgr2=0x7f102406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f102406adb0 0x7f102406d270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f10440a14b0 tx=0x7f1030013040 comp rx=0 tx=0).stop 2024-03-29T11:31:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 -- 172.21.15.12:0/2296123488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10440d18b0 msgr2=0x7f1044153c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10440d18b0 0x7f1044153c40 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f103800c500 tx=0x7f103800c8c0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 -- 172.21.15.12:0/2296123488 shutdown_connections 2024-03-29T11:31:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f102406adb0 0x7f102406d270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1044161590 0x7f104415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10440d18b0 0x7f1044153c40 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 --2- 172.21.15.12:0/2296123488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1044160f10 0x7f10441592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 -- 172.21.15.12:0/2296123488 >> 172.21.15.12:0/2296123488 conn(0x7f104408aff0 msgr2=0x7f1044152670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 -- 172.21.15.12:0/2296123488 shutdown_connections 2024-03-29T11:31:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.579+0000 7f104b800700 1 -- 172.21.15.12:0/2296123488 wait complete. 2024-03-29T11:31:38.592 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-03-29T11:31:38.592 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-03-29T11:31:38.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.800+0000 7f854a800700 1 Processor -- start 2024-03-29T11:31:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.800+0000 7f854a800700 1 -- start start 2024-03-29T11:31:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.801+0000 7f854a800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f85440ced30 0x7f85440d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.801+0000 7f854a800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f85440d2710 0x7f854415f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.801+0000 7f854a800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85440ce8a0 con 0x7f854415ddf0 2024-03-29T11:31:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.801+0000 7f854a800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854415ec60 con 0x7f85440d2710 2024-03-29T11:31:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.801+0000 7f854a800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85440d2b60 con 0x7f85440ced30 2024-03-29T11:31:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.802+0000 7f8543e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f85440d2710 0x7f854415f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.802+0000 7f8543e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f85440d2710 0x7f854415f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37054/0 (socket says 172.21.15.12:37054) 2024-03-29T11:31:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.802+0000 7f8543e00700 1 -- 172.21.15.12:0/1348296798 learned_addr learned my addr 172.21.15.12:0/1348296798 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.802+0000 7f8542a00700 1 --2- 172.21.15.12:0/1348296798 >> v2:172.21.15.159:3300/0 conn(0x7f85440ced30 0x7f85440d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.802+0000 7f8543e00700 1 -- 172.21.15.12:0/1348296798 >> v2:172.21.15.159:3300/0 conn(0x7f85440ced30 msgr2=0x7f85440d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:38.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.802+0000 7f8543e00700 1 --2- 172.21.15.12:0/1348296798 >> v2:172.21.15.159:3300/0 conn(0x7f85440ced30 0x7f85440d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.802+0000 7f8543e00700 1 -- 172.21.15.12:0/1348296798 >> v1:172.21.15.159:6789/0 conn(0x7f854415ddf0 legacy=0x7f854415e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:38.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.802+0000 7f8543e00700 1 -- 172.21.15.12:0/1348296798 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85440cdc90 con 0x7f85440d2710 2024-03-29T11:31:38.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.802+0000 7f8543e00700 1 --2- 172.21.15.12:0/1348296798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 0x7f854415f0f0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f85440d0fe0 tx=0x7f853c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb316622838e8e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:38.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f8542000700 1 -- 172.21.15.12:0/1348296798 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c00e070 con 0x7f85440d2710 2024-03-29T11:31:38.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f8542000700 1 -- 172.21.15.12:0/1348296798 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c00b9c0 con 0x7f85440d2710 2024-03-29T11:31:38.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f8542000700 1 -- 172.21.15.12:0/1348296798 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c00bd70 con 0x7f85440d2710 2024-03-29T11:31:38.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f854a800700 1 -- 172.21.15.12:0/1348296798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 msgr2=0x7f854415f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f854a800700 1 --2- 172.21.15.12:0/1348296798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 0x7f854415f0f0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f85440d0fe0 tx=0x7f853c00b330 comp rx=0 tx=0).stop 2024-03-29T11:31:38.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f854a800700 1 -- 172.21.15.12:0/1348296798 shutdown_connections 2024-03-29T11:31:38.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f854a800700 1 --2- 172.21.15.12:0/1348296798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 0x7f854415f0f0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f854a800700 1 --2- 172.21.15.12:0/1348296798 >> v2:172.21.15.159:3300/0 conn(0x7f85440ced30 0x7f85440d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f854a800700 1 -- 172.21.15.12:0/1348296798 >> 172.21.15.12:0/1348296798 conn(0x7f854408aff0 msgr2=0x7f85440d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:38.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f854a800700 1 -- 172.21.15.12:0/1348296798 shutdown_connections 2024-03-29T11:31:38.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.803+0000 7f854a800700 1 -- 172.21.15.12:0/1348296798 wait complete. 2024-03-29T11:31:38.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.805+0000 7f854a800700 1 Processor -- start 2024-03-29T11:31:38.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.805+0000 7f854a800700 1 -- start start 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f854a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85440ced30 0x7f85441f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f854a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 0x7f85440caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f854a800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f854415ddf0 0x7f85440cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f854a800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85440c7020 con 0x7f85440d2710 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f854a800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85440c75e0 con 0x7f854415ddf0 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f854a800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85441f7620 con 0x7f85440ced30 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f8542a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 0x7f85440caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f8542a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 0x7f85440caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37060/0 (socket says 172.21.15.12:37060) 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f8542a00700 1 -- 172.21.15.12:0/2578066291 learned_addr learned my addr 172.21.15.12:0/2578066291 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f8543400700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85440ced30 0x7f85441f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.806+0000 7f8543e00700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f854415ddf0 0x7f85440cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f8542a00700 1 -- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85440ced30 msgr2=0x7f85441f6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f8542a00700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85440ced30 0x7f85441f6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f8542a00700 1 -- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f854415ddf0 msgr2=0x7f85440cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f8542a00700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f854415ddf0 0x7f85440cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f8542a00700 1 -- 172.21.15.12:0/2578066291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c009b30 con 0x7f85440d2710 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f8543400700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85440ced30 0x7f85441f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f8542a00700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 0x7f85440caff0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f853000e840 tx=0x7f853000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f853b400700 1 -- 172.21.15.12:0/2578066291 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8530013610 con 0x7f85440d2710 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f853b400700 1 -- 172.21.15.12:0/2578066291 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8530013770 con 0x7f85440d2710 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f853b400700 1 -- 172.21.15.12:0/2578066291 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8530013b20 con 0x7f85440d2710 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f854a800700 1 -- 172.21.15.12:0/2578066291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85440cbb60 con 0x7f85440d2710 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.807+0000 7f854a800700 1 -- 172.21.15.12:0/2578066291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85440c7270 con 0x7f85440d2710 2024-03-29T11:31:38.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.808+0000 7f854a800700 1 -- 172.21.15.12:0/2578066291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85440c7770 con 0x7f85440d2710 2024-03-29T11:31:38.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.809+0000 7f853b400700 1 -- 172.21.15.12:0/2578066291 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8530013c80 con 0x7f85440d2710 2024-03-29T11:31:38.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.810+0000 7f853b400700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f852406adb0 0x7f852406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:38.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.810+0000 7f853b400700 1 -- 172.21.15.12:0/2578066291 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f85300892a0 con 0x7f85440d2710 2024-03-29T11:31:38.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.811+0000 7f8543400700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f852406adb0 0x7f852406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:38.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.811+0000 7f8543400700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f852406adb0 0x7f852406d270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f854406d540 tx=0x7f8534009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:38.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:38.814+0000 7f853b400700 1 -- 172.21.15.12:0/2578066291 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8530054180 con 0x7f85440d2710 2024-03-29T11:31:39.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.028+0000 7f854a800700 1 -- 172.21.15.12:0/2578066291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8544067720 con 0x7f85440d2710 2024-03-29T11:31:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.335+0000 7f853b400700 1 -- 172.21.15.12:0/2578066291 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v228) v1 ==== 144+0+0 (secure 0 0 0) 0x7f85300577a0 con 0x7f85440d2710 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 -- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f852406adb0 msgr2=0x7f852406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f852406adb0 0x7f852406d270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f854406d540 tx=0x7f8534009040 comp rx=0 tx=0).stop 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 -- 172.21.15.12:0/2578066291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 msgr2=0x7f85440caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 0x7f85440caff0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f853000e840 tx=0x7f853000eb50 comp rx=0 tx=0).stop 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 -- 172.21.15.12:0/2578066291 shutdown_connections 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f852406adb0 0x7f852406d270 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f854415ddf0 0x7f85440cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85440ced30 0x7f85441f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 --2- 172.21.15.12:0/2578066291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85440d2710 0x7f85440caff0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 -- 172.21.15.12:0/2578066291 >> 172.21.15.12:0/2578066291 conn(0x7f854408aff0 msgr2=0x7f85440b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 -- 172.21.15.12:0/2578066291 shutdown_connections 2024-03-29T11:31:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.338+0000 7f854a800700 1 -- 172.21.15.12:0/2578066291 wait complete. 2024-03-29T11:31:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'backup_fs_metadata' created 2024-03-29T11:31:39.356 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T11:31:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.566+0000 7fc173600700 1 Processor -- start 2024-03-29T11:31:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.567+0000 7fc173600700 1 -- start start 2024-03-29T11:31:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.568+0000 7fc173600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 0x7fc16c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.568+0000 7fc173600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc16c0b2070 0x7fc16c0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.568+0000 7fc173600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16c0cdae0 con 0x7fc16c0cbb30 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.568+0000 7fc173600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c0cddc0 con 0x7fc16c0b2070 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.568+0000 7fc173600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c0ce0a0 con 0x7fc16c0cd020 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.568+0000 7fc170c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc16c0cbb30 0x7fc16c0cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47180/0 (socket says 172.21.15.12:47180) 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.568+0000 7fc170c00700 1 -- 172.21.15.12:0/3518312330 learned_addr learned my addr 172.21.15.12:0/3518312330 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.568+0000 7fc171600700 1 --2- 172.21.15.12:0/3518312330 >> v2:172.21.15.159:3300/0 conn(0x7fc16c0b2070 0x7fc16c0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.568+0000 7fc16be00700 1 --2- 172.21.15.12:0/3518312330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 0x7fc16c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.568+0000 7fc16b400700 1 -- 172.21.15.12:0/3518312330 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219125642 0 0) 0x7fc16c0cdae0 con 0x7fc16c0cbb30 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.569+0000 7fc16b400700 1 -- 172.21.15.12:0/3518312330 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1540035b0 con 0x7fc16c0cbb30 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.569+0000 7fc16b400700 1 -- 172.21.15.12:0/3518312330 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1345872309 0 0) 0x7fc1540035b0 con 0x7fc16c0cbb30 2024-03-29T11:31:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.569+0000 7fc16be00700 1 -- 172.21.15.12:0/3518312330 >> v2:172.21.15.159:3300/0 conn(0x7fc16c0b2070 msgr2=0x7fc16c0b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.569+0000 7fc16be00700 1 --2- 172.21.15.12:0/3518312330 >> v2:172.21.15.159:3300/0 conn(0x7fc16c0b2070 0x7fc16c0b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.569+0000 7fc16be00700 1 -- 172.21.15.12:0/3518312330 >> v1:172.21.15.12:6789/0 conn(0x7fc16c0cbb30 legacy=0x7fc16c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.569+0000 7fc16be00700 1 -- 172.21.15.12:0/3518312330 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c06dc20 con 0x7fc16c0cd020 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.569+0000 7fc16be00700 1 --2- 172.21.15.12:0/3518312330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 0x7fc16c0cd480 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc158009840 tx=0x7fc158009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66967e762a38a195 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.569+0000 7fc16b400700 1 -- 172.21.15.12:0/3518312330 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc160001670 con 0x7fc16c0cd020 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.570+0000 7fc16b400700 1 -- 172.21.15.12:0/3518312330 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc158004030 con 0x7fc16c0cd020 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.570+0000 7fc16b400700 1 -- 172.21.15.12:0/3518312330 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1580043a0 con 0x7fc16c0cd020 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.570+0000 7fc173600700 1 -- 172.21.15.12:0/3518312330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 msgr2=0x7fc16c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.570+0000 7fc173600700 1 --2- 172.21.15.12:0/3518312330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 0x7fc16c0cd480 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc158009840 tx=0x7fc158009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.570+0000 7fc173600700 1 -- 172.21.15.12:0/3518312330 shutdown_connections 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.570+0000 7fc173600700 1 --2- 172.21.15.12:0/3518312330 >> v2:172.21.15.159:3300/0 conn(0x7fc16c0b2070 0x7fc16c0b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.570+0000 7fc173600700 1 --2- 172.21.15.12:0/3518312330 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 0x7fc16c0cd480 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:39.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.570+0000 7fc173600700 1 -- 172.21.15.12:0/3518312330 >> 172.21.15.12:0/3518312330 conn(0x7fc16c0b0d70 msgr2=0x7fc16c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:39.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.570+0000 7fc173600700 1 -- 172.21.15.12:0/3518312330 shutdown_connections 2024-03-29T11:31:39.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.570+0000 7fc173600700 1 -- 172.21.15.12:0/3518312330 wait complete. 2024-03-29T11:31:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.572+0000 7fc173600700 1 Processor -- start 2024-03-29T11:31:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.572+0000 7fc173600700 1 -- start start 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.573+0000 7fc173600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc16c0b2070 0x7fc16c1521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.573+0000 7fc173600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc16c0cbb30 0x7fc16c0c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.573+0000 7fc173600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 0x7fc16c0718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.573+0000 7fc173600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c0ca360 con 0x7fc16c0cbb30 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.573+0000 7fc173600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c070af0 con 0x7fc16c0b2070 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.573+0000 7fc173600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c070dd0 con 0x7fc16c0cd020 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.573+0000 7fc16be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc16c0cbb30 0x7fc16c0c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.573+0000 7fc16be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc16c0cbb30 0x7fc16c0c9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39414/0 (socket says 172.21.15.12:39414) 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.573+0000 7fc16be00700 1 -- 172.21.15.12:0/2916987676 learned_addr learned my addr 172.21.15.12:0/2916987676 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.574+0000 7fc170c00700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc16c0b2070 0x7fc16c1521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.574+0000 7fc171600700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 0x7fc16c0718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.574+0000 7fc16be00700 1 -- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 msgr2=0x7fc16c0718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.574+0000 7fc16be00700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 0x7fc16c0718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.574+0000 7fc16be00700 1 -- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc16c0b2070 msgr2=0x7fc16c1521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.574+0000 7fc16be00700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc16c0b2070 0x7fc16c1521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.574+0000 7fc16be00700 1 -- 172.21.15.12:0/2916987676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1580094a0 con 0x7fc16c0cbb30 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.574+0000 7fc171600700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 0x7fc16c0718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.574+0000 7fc16be00700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc16c0cbb30 0x7fc16c0c9a80 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc158009b90 tx=0x7fc158004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.574+0000 7fc168c00700 1 -- 172.21.15.12:0/2916987676 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158003d40 con 0x7fc16c0cbb30 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.575+0000 7fc168c00700 1 -- 172.21.15.12:0/2916987676 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15801f070 con 0x7fc16c0cbb30 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.575+0000 7fc168c00700 1 -- 172.21.15.12:0/2916987676 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158003ea0 con 0x7fc16c0cbb30 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.575+0000 7fc173600700 1 -- 172.21.15.12:0/2916987676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16c0710b0 con 0x7fc16c0cbb30 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.575+0000 7fc173600700 1 -- 172.21.15.12:0/2916987676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16c152720 con 0x7fc16c0cbb30 2024-03-29T11:31:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.576+0000 7fc173600700 1 -- 172.21.15.12:0/2916987676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16c067720 con 0x7fc16c0cbb30 2024-03-29T11:31:39.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.577+0000 7fc168c00700 1 -- 172.21.15.12:0/2916987676 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc158021030 con 0x7fc16c0cbb30 2024-03-29T11:31:39.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.578+0000 7fc168c00700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc14c06adb0 0x7fc14c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:39.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.578+0000 7fc168c00700 1 -- 172.21.15.12:0/2916987676 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 6154+0+0 (secure 0 0 0) 0x7fc15808a920 con 0x7fc16c0cbb30 2024-03-29T11:31:39.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.578+0000 7fc170c00700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc14c06adb0 0x7fc14c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:39.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.579+0000 7fc170c00700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc14c06adb0 0x7fc14c06d270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc160001eb0 tx=0x7fc160001710 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:39.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.583+0000 7fc168c00700 1 -- 172.21.15.12:0/2916987676 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc1580593d0 con 0x7fc16c0cbb30 2024-03-29T11:31:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:39.799+0000 7fc173600700 1 -- 172.21.15.12:0/2916987676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc16c0ce2e0 con 0x7fc16c0cbb30 2024-03-29T11:31:40.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.346+0000 7fc168c00700 1 -- 172.21.15.12:0/2916987676 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v229) v1 ==== 176+0+0 (secure 0 0 0) 0x7fc158058f60 con 0x7fc16c0cbb30 2024-03-29T11:31:40.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.354+0000 7fc173600700 1 -- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc14c06adb0 msgr2=0x7fc14c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:40.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.354+0000 7fc173600700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc14c06adb0 0x7fc14c06d270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc160001eb0 tx=0x7fc160001710 comp rx=0 tx=0).stop 2024-03-29T11:31:40.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.354+0000 7fc173600700 1 -- 172.21.15.12:0/2916987676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc16c0cbb30 msgr2=0x7fc16c0c9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:40.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.354+0000 7fc173600700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc16c0cbb30 0x7fc16c0c9a80 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc158009b90 tx=0x7fc158004750 comp rx=0 tx=0).stop 2024-03-29T11:31:40.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.354+0000 7fc173600700 1 -- 172.21.15.12:0/2916987676 shutdown_connections 2024-03-29T11:31:40.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.354+0000 7fc173600700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc14c06adb0 0x7fc14c06d270 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc160001eb0 tx=0x7fc160001710 comp rx=0 tx=0).stop 2024-03-29T11:31:40.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.354+0000 7fc173600700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc16c0cd020 0x7fc16c0718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:40.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.354+0000 7fc173600700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc16c0b2070 0x7fc16c1521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:40.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.354+0000 7fc173600700 1 --2- 172.21.15.12:0/2916987676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc16c0cbb30 0x7fc16c0c9a80 secure :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc158009b90 tx=0x7fc158004750 comp rx=0 tx=0).stop 2024-03-29T11:31:40.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.354+0000 7fc173600700 1 -- 172.21.15.12:0/2916987676 >> 172.21.15.12:0/2916987676 conn(0x7fc16c0b0d70 msgr2=0x7fc16c0b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:40.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.356+0000 7fc173600700 1 -- 172.21.15.12:0/2916987676 shutdown_connections 2024-03-29T11:31:40.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.356+0000 7fc173600700 1 -- 172.21.15.12:0/2916987676 wait complete. 2024-03-29T11:31:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'backup_fs_data' created 2024-03-29T11:31:40.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-03-29T11:31:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.587+0000 7f0717400700 1 Processor -- start 2024-03-29T11:31:40.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.587+0000 7f0717400700 1 -- start start 2024-03-29T11:31:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.588+0000 7f0717400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f071015d9d0 0x7f07100d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.589+0000 7f0717400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0710161420 con 0x7f07100d2540 2024-03-29T11:31:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.589+0000 7f0717400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0710161700 con 0x7f07100d16a0 2024-03-29T11:31:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.589+0000 7f0717400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f071015edc0 con 0x7f071015d9d0 2024-03-29T11:31:40.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.589+0000 7f0714a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f071015d9d0 0x7f07100d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:40.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.589+0000 7f0714a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f071015d9d0 0x7f07100d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39426/0 (socket says 172.21.15.12:39426) 2024-03-29T11:31:40.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.589+0000 7f0714a00700 1 -- 172.21.15.12:0/2607092272 learned_addr learned my addr 172.21.15.12:0/2607092272 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:40.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.589+0000 7f0714a00700 1 -- 172.21.15.12:0/2607092272 >> v1:172.21.15.159:6789/0 conn(0x7f07100d16a0 legacy=0x7f07100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:40.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.589+0000 7f070f400700 1 -- 172.21.15.12:0/2607092272 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488813745 0 0) 0x7f0710161420 con 0x7f07100d2540 2024-03-29T11:31:40.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.589+0000 7f0714a00700 1 -- 172.21.15.12:0/2607092272 >> v1:172.21.15.12:6789/0 conn(0x7f07100d2540 legacy=0x7f07100ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:40.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.589+0000 7f0714a00700 1 -- 172.21.15.12:0/2607092272 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07101619e0 con 0x7f071015d9d0 2024-03-29T11:31:40.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.590+0000 7f0714a00700 1 --2- 172.21.15.12:0/2607092272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f071015d9d0 0x7f07100d3c30 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f07040098f0 tx=0x7f0704009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f43bde133c0b000 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:40.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.590+0000 7f070f400700 1 -- 172.21.15.12:0/2607092272 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070400b8a0 con 0x7f071015d9d0 2024-03-29T11:31:40.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.590+0000 7f070f400700 1 -- 172.21.15.12:0/2607092272 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070400ba00 con 0x7f071015d9d0 2024-03-29T11:31:40.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.590+0000 7f070f400700 1 -- 172.21.15.12:0/2607092272 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070400bd90 con 0x7f071015d9d0 2024-03-29T11:31:40.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.590+0000 7f0717400700 1 -- 172.21.15.12:0/2607092272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f071015d9d0 msgr2=0x7f07100d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.590+0000 7f0717400700 1 --2- 172.21.15.12:0/2607092272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f071015d9d0 0x7f07100d3c30 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f07040098f0 tx=0x7f0704009c00 comp rx=0 tx=0).stop 2024-03-29T11:31:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.590+0000 7f0717400700 1 -- 172.21.15.12:0/2607092272 shutdown_connections 2024-03-29T11:31:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.590+0000 7f0717400700 1 --2- 172.21.15.12:0/2607092272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f071015d9d0 0x7f07100d3c30 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.590+0000 7f0717400700 1 -- 172.21.15.12:0/2607092272 >> 172.21.15.12:0/2607092272 conn(0x7f071008aff0 msgr2=0x7f07100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.591+0000 7f0717400700 1 -- 172.21.15.12:0/2607092272 shutdown_connections 2024-03-29T11:31:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.591+0000 7f0717400700 1 -- 172.21.15.12:0/2607092272 wait complete. 2024-03-29T11:31:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.592+0000 7f0717400700 1 Processor -- start 2024-03-29T11:31:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.593+0000 7f0717400700 1 -- start start 2024-03-29T11:31:40.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.593+0000 7f0717400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07100d16a0 0x7f07101e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:40.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.593+0000 7f0717400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07100d2540 0x7f07101e9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:40.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.593+0000 7f0717400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f071015d9d0 0x7f0710160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:40.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.593+0000 7f0717400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07101e8600 con 0x7f07100d2540 2024-03-29T11:31:40.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.593+0000 7f0717400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07101e8b50 con 0x7f07100d16a0 2024-03-29T11:31:40.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f0717400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07101e8e30 con 0x7f071015d9d0 2024-03-29T11:31:40.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f070fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07100d2540 0x7f07101e9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:40.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f070fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07100d2540 0x7f07101e9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39436/0 (socket says 172.21.15.12:39436) 2024-03-29T11:31:40.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f070fe00700 1 -- 172.21.15.12:0/3957085172 learned_addr learned my addr 172.21.15.12:0/3957085172 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:40.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f0714a00700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07100d16a0 0x7f07101e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:40.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f0715400700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f071015d9d0 0x7f0710160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:40.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f070fe00700 1 -- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f071015d9d0 msgr2=0x7f0710160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:40.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f070fe00700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f071015d9d0 0x7f0710160680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:40.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f070fe00700 1 -- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07100d16a0 msgr2=0x7f07101e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f070fe00700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07100d16a0 0x7f07101e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f070fe00700 1 -- 172.21.15.12:0/3957085172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0704009550 con 0x7f07100d2540 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.594+0000 7f070fe00700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07100d2540 0x7f07101e9910 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00c500 tx=0x7f06fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.595+0000 7f070cc00700 1 -- 172.21.15.12:0/3957085172 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0052d0 con 0x7f07100d2540 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.595+0000 7f070cc00700 1 -- 172.21.15.12:0/3957085172 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc005450 con 0x7f07100d2540 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.595+0000 7f0717400700 1 -- 172.21.15.12:0/3957085172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07101e9110 con 0x7f07100d2540 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.595+0000 7f070cc00700 1 -- 172.21.15.12:0/3957085172 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc003910 con 0x7f07100d2540 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.595+0000 7f0717400700 1 -- 172.21.15.12:0/3957085172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07101e81d0 con 0x7f07100d2540 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.596+0000 7f0717400700 1 -- 172.21.15.12:0/3957085172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07101e8390 con 0x7f07100d2540 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.598+0000 7f070cc00700 1 -- 172.21.15.12:0/3957085172 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f06fc003a70 con 0x7f07100d2540 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.598+0000 7f070cc00700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06f006adb0 0x7f06f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.599+0000 7f070cc00700 1 -- 172.21.15.12:0/3957085172 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f06fc089050 con 0x7f07100d2540 2024-03-29T11:31:40.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.599+0000 7f0714a00700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06f006adb0 0x7f06f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.600+0000 7f0714a00700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06f006adb0 0x7f06f006d270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f071008b580 tx=0x7f070401c040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:40.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.603+0000 7f070cc00700 1 -- 172.21.15.12:0/3957085172 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06fc053ec0 con 0x7f07100d2540 2024-03-29T11:31:40.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.861+0000 7f0717400700 1 -- 172.21.15.12:0/3957085172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f0710067720 con 0x7f07100d2540 2024-03-29T11:31:40.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.870+0000 7f070cc00700 1 -- 172.21.15.12:0/3957085172 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 28 and data pool 29 v108) v1 ==== 183+0+0 (secure 0 0 0) 0x7f06fc0574e0 con 0x7f07100d2540 2024-03-29T11:31:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.877+0000 7f0717400700 1 -- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06f006adb0 msgr2=0x7f06f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.877+0000 7f0717400700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06f006adb0 0x7f06f006d270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f071008b580 tx=0x7f070401c040 comp rx=0 tx=0).stop 2024-03-29T11:31:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.877+0000 7f0717400700 1 -- 172.21.15.12:0/3957085172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07100d2540 msgr2=0x7f07101e9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.877+0000 7f0717400700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07100d2540 0x7f07101e9910 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00c500 tx=0x7f06fc00c8c0 comp rx=0 tx=0).stop 2024-03-29T11:31:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.878+0000 7f0717400700 1 -- 172.21.15.12:0/3957085172 shutdown_connections 2024-03-29T11:31:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.878+0000 7f0717400700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f06f006adb0 0x7f06f006d270 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.878+0000 7f0717400700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f071015d9d0 0x7f0710160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.878+0000 7f0717400700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07100d16a0 0x7f07101e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.878+0000 7f0717400700 1 --2- 172.21.15.12:0/3957085172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07100d2540 0x7f07101e9910 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.878+0000 7f0717400700 1 -- 172.21.15.12:0/3957085172 >> 172.21.15.12:0/3957085172 conn(0x7f071008aff0 msgr2=0x7f07100ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.878+0000 7f0717400700 1 -- 172.21.15.12:0/3957085172 shutdown_connections 2024-03-29T11:31:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:40.878+0000 7f0717400700 1 -- 172.21.15.12:0/3957085172 wait complete. 2024-03-29T11:31:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 28 and data pool 29 2024-03-29T11:31:40.906 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.112+0000 7f225e600700 1 Processor -- start 2024-03-29T11:31:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.112+0000 7f225e600700 1 -- start start 2024-03-29T11:31:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.113+0000 7f225e600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2258160be0 0x7f22580cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.113+0000 7f225e600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22580ce0e0 0x7f22580ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.113+0000 7f225e600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2258160fc0 con 0x7f22580cead0 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.113+0000 7f225e600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22580cef60 con 0x7f2258160be0 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.113+0000 7f225e600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22580b65d0 con 0x7f22580ce0e0 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.113+0000 7f2257400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2258160be0 0x7f22580cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.113+0000 7f2257400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2258160be0 0x7f22580cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39444/0 (socket says 172.21.15.12:39444) 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.113+0000 7f2257400700 1 -- 172.21.15.12:0/1489345426 learned_addr learned my addr 172.21.15.12:0/1489345426 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.113+0000 7f2256a00700 1 --2- 172.21.15.12:0/1489345426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22580ce0e0 0x7f22580ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.114+0000 7f2257400700 1 -- 172.21.15.12:0/1489345426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22580ce0e0 msgr2=0x7f22580ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.114+0000 7f2257400700 1 --2- 172.21.15.12:0/1489345426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22580ce0e0 0x7f22580ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.114+0000 7f2257400700 1 -- 172.21.15.12:0/1489345426 >> v1:172.21.15.159:6789/0 conn(0x7f22580cead0 legacy=0x7f2258161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.114+0000 7f2257400700 1 -- 172.21.15.12:0/1489345426 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22580d21e0 con 0x7f2258160be0 2024-03-29T11:31:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.114+0000 7f2257400700 1 --2- 172.21.15.12:0/1489345426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258160be0 0x7f22580cdba0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f22480098f0 tx=0x7f2248009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5d1d7e6305117f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.114+0000 7f2256000700 1 -- 172.21.15.12:0/1489345426 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2248011830 con 0x7f2258160be0 2024-03-29T11:31:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.114+0000 7f2256000700 1 -- 172.21.15.12:0/1489345426 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2248011990 con 0x7f2258160be0 2024-03-29T11:31:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.114+0000 7f2256000700 1 -- 172.21.15.12:0/1489345426 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2248011d20 con 0x7f2258160be0 2024-03-29T11:31:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.115+0000 7f225e600700 1 -- 172.21.15.12:0/1489345426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258160be0 msgr2=0x7f22580cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.115+0000 7f225e600700 1 --2- 172.21.15.12:0/1489345426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258160be0 0x7f22580cdba0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f22480098f0 tx=0x7f2248009c00 comp rx=0 tx=0).stop 2024-03-29T11:31:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.115+0000 7f225e600700 1 -- 172.21.15.12:0/1489345426 shutdown_connections 2024-03-29T11:31:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.115+0000 7f225e600700 1 --2- 172.21.15.12:0/1489345426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258160be0 0x7f22580cdba0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.115+0000 7f225e600700 1 --2- 172.21.15.12:0/1489345426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22580ce0e0 0x7f22580ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.115+0000 7f225e600700 1 -- 172.21.15.12:0/1489345426 >> 172.21.15.12:0/1489345426 conn(0x7f225808aff0 msgr2=0x7f22580d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.115+0000 7f225e600700 1 -- 172.21.15.12:0/1489345426 shutdown_connections 2024-03-29T11:31:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.115+0000 7f225e600700 1 -- 172.21.15.12:0/1489345426 wait complete. 2024-03-29T11:31:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.117+0000 7f225e600700 1 Processor -- start 2024-03-29T11:31:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.117+0000 7f225e600700 1 -- start start 2024-03-29T11:31:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f225e600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22580ce0e0 0x7f2258158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f225e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580cead0 0x7f22580c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f225e600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2258160be0 0x7f225815d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f225e600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22580cbc70 con 0x7f22580cead0 2024-03-29T11:31:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f225e600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22580c80c0 con 0x7f2258160be0 2024-03-29T11:31:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f225e600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22580c7b00 con 0x7f22580ce0e0 2024-03-29T11:31:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f2256a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580cead0 0x7f22580c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f2256a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580cead0 0x7f22580c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39452/0 (socket says 172.21.15.12:39452) 2024-03-29T11:31:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f2256a00700 1 -- 172.21.15.12:0/3077558060 learned_addr learned my addr 172.21.15.12:0/3077558060 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f2256a00700 1 -- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22580ce0e0 msgr2=0x7f2258158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f2256a00700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22580ce0e0 0x7f2258158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f2256a00700 1 -- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2258160be0 msgr2=0x7f225815d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f2257e00700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2258160be0 0x7f225815d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f2256a00700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2258160be0 0x7f225815d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.118+0000 7f2256a00700 1 -- 172.21.15.12:0/3077558060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2248009550 con 0x7f22580cead0 2024-03-29T11:31:41.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.119+0000 7f2256a00700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580cead0 0x7f22580c7080 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f224400e9b0 tx=0x7f224400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:41.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.119+0000 7f224f400700 1 -- 172.21.15.12:0/3077558060 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2244009710 con 0x7f22580cead0 2024-03-29T11:31:41.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.119+0000 7f224f400700 1 -- 172.21.15.12:0/3077558060 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224400c850 con 0x7f22580cead0 2024-03-29T11:31:41.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.119+0000 7f225e600700 1 -- 172.21.15.12:0/3077558060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22580c7de0 con 0x7f22580cead0 2024-03-29T11:31:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.119+0000 7f224f400700 1 -- 172.21.15.12:0/3077558060 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224400cbc0 con 0x7f22580cead0 2024-03-29T11:31:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.119+0000 7f225e600700 1 -- 172.21.15.12:0/3077558060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225815adc0 con 0x7f22580cead0 2024-03-29T11:31:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.120+0000 7f225e600700 1 -- 172.21.15.12:0/3077558060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2258067720 con 0x7f22580cead0 2024-03-29T11:31:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.121+0000 7f224f400700 1 -- 172.21.15.12:0/3077558060 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f224400cd20 con 0x7f22580cead0 2024-03-29T11:31:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.122+0000 7f224f400700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f223806adb0 0x7f223806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.122+0000 7f224f400700 1 -- 172.21.15.12:0/3077558060 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f2244089960 con 0x7f22580cead0 2024-03-29T11:31:41.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.123+0000 7f2257400700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f223806adb0 0x7f223806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.123+0000 7f2257400700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f223806adb0 0x7f223806d270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f22580d35a0 tx=0x7f2248010000 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.127+0000 7f224f400700 1 -- 172.21.15.12:0/3077558060 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2244054670 con 0x7f22580cead0 2024-03-29T11:31:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.344+0000 7f225e600700 1 -- 172.21.15.12:0/3077558060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22580cbdb0 con 0x7f22580cead0 2024-03-29T11:31:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.347+0000 7f224f400700 1 -- 172.21.15.12:0/3077558060 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+19968 (secure 0 0 0) 0x7f22580cbdb0 con 0x7f22580cead0 2024-03-29T11:31:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:41.353 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":230,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:40.862882+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:31:39.075407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:31:39.845704+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.349+0000 7f225e600700 1 -- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f223806adb0 msgr2=0x7f223806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f223806adb0 0x7f223806d270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f22580d35a0 tx=0x7f2248010000 comp rx=0 tx=0).stop 2024-03-29T11:31:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 -- 172.21.15.12:0/3077558060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580cead0 msgr2=0x7f22580c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580cead0 0x7f22580c7080 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f224400e9b0 tx=0x7f224400ed70 comp rx=0 tx=0).stop 2024-03-29T11:31:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 -- 172.21.15.12:0/3077558060 shutdown_connections 2024-03-29T11:31:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f223806adb0 0x7f223806d270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2258160be0 0x7f225815d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22580ce0e0 0x7f2258158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 --2- 172.21.15.12:0/3077558060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580cead0 0x7f22580c7080 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 -- 172.21.15.12:0/3077558060 >> 172.21.15.12:0/3077558060 conn(0x7f225808aff0 msgr2=0x7f22580b7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 -- 172.21.15.12:0/3077558060 shutdown_connections 2024-03-29T11:31:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.350+0000 7f225e600700 1 -- 172.21.15.12:0/3077558060 wait complete. 2024-03-29T11:31:41.364 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.575+0000 7ff251a00700 1 Processor -- start 2024-03-29T11:31:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.575+0000 7ff251a00700 1 -- start start 2024-03-29T11:31:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.576+0000 7ff251a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff24c160f10 0x7ff24c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.576+0000 7ff251a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff24c161590 0x7ff24c161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.576+0000 7ff251a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c0ceda0 con 0x7ff24c0cf230 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.576+0000 7ff251a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24c161f80 con 0x7ff24c161590 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.576+0000 7ff251a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c1612f0 con 0x7ff24c160f10 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.576+0000 7ff24a000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff24c0cf230 0x7ff24c0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47210/0 (socket says 172.21.15.12:47210) 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.577+0000 7ff24a000700 1 -- 172.21.15.12:0/2021302283 learned_addr learned my addr 172.21.15.12:0/2021302283 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.577+0000 7ff24b400700 1 --2- 172.21.15.12:0/2021302283 >> v2:172.21.15.12:3300/0 conn(0x7ff24c161590 0x7ff24c161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.577+0000 7ff24aa00700 1 --2- 172.21.15.12:0/2021302283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff24c160f10 0x7ff24c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.577+0000 7ff249600700 1 -- 172.21.15.12:0/2021302283 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974220563 0 0) 0x7ff24c0ceda0 con 0x7ff24c0cf230 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.577+0000 7ff249600700 1 -- 172.21.15.12:0/2021302283 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2300035b0 con 0x7ff24c0cf230 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.577+0000 7ff24b400700 1 -- 172.21.15.12:0/2021302283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff24c160f10 msgr2=0x7ff24c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.577+0000 7ff24b400700 1 --2- 172.21.15.12:0/2021302283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff24c160f10 0x7ff24c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.577+0000 7ff24b400700 1 -- 172.21.15.12:0/2021302283 >> v1:172.21.15.12:6789/0 conn(0x7ff24c0cf230 legacy=0x7ff24c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.577+0000 7ff24b400700 1 -- 172.21.15.12:0/2021302283 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0ce470 con 0x7ff24c161590 2024-03-29T11:31:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.577+0000 7ff24b400700 1 --2- 172.21.15.12:0/2021302283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c161590 0x7ff24c161a40 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7ff244009dc0 tx=0x7ff24400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95de01ce6d1b9676 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff249600700 1 -- 172.21.15.12:0/2021302283 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24400e070 con 0x7ff24c161590 2024-03-29T11:31:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff249600700 1 -- 172.21.15.12:0/2021302283 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24400bae0 con 0x7ff24c161590 2024-03-29T11:31:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff249600700 1 -- 172.21.15.12:0/2021302283 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24400be90 con 0x7ff24c161590 2024-03-29T11:31:41.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff251a00700 1 -- 172.21.15.12:0/2021302283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c161590 msgr2=0x7ff24c161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff251a00700 1 --2- 172.21.15.12:0/2021302283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c161590 0x7ff24c161a40 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7ff244009dc0 tx=0x7ff24400b330 comp rx=0 tx=0).stop 2024-03-29T11:31:41.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff251a00700 1 -- 172.21.15.12:0/2021302283 shutdown_connections 2024-03-29T11:31:41.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff251a00700 1 --2- 172.21.15.12:0/2021302283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c161590 0x7ff24c161a40 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff251a00700 1 --2- 172.21.15.12:0/2021302283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff24c160f10 0x7ff24c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff251a00700 1 -- 172.21.15.12:0/2021302283 >> 172.21.15.12:0/2021302283 conn(0x7ff24c08aff0 msgr2=0x7ff24c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:41.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff251a00700 1 -- 172.21.15.12:0/2021302283 shutdown_connections 2024-03-29T11:31:41.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.578+0000 7ff251a00700 1 -- 172.21.15.12:0/2021302283 wait complete. 2024-03-29T11:31:41.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.580+0000 7ff251a00700 1 Processor -- start 2024-03-29T11:31:41.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.580+0000 7ff251a00700 1 -- start start 2024-03-29T11:31:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.581+0000 7ff251a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff24c0cf230 0x7ff24c14c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.581+0000 7ff251a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c160f10 0x7ff24c15f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.581+0000 7ff251a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff24c161590 0x7ff24c0b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.581+0000 7ff251a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c15fe90 con 0x7ff24c160f10 2024-03-29T11:31:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.581+0000 7ff251a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c14caf0 con 0x7ff24c0cf230 2024-03-29T11:31:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.581+0000 7ff251a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c14cdd0 con 0x7ff24c161590 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c160f10 0x7ff24c15f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c160f10 0x7ff24c15f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39470/0 (socket says 172.21.15.12:39470) 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24a000700 1 -- 172.21.15.12:0/3243847027 learned_addr learned my addr 172.21.15.12:0/3243847027 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24aa00700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff24c0cf230 0x7ff24c14c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24b400700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff24c161590 0x7ff24c0b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24a000700 1 -- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff24c161590 msgr2=0x7ff24c0b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24a000700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff24c161590 0x7ff24c0b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24a000700 1 -- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff24c0cf230 msgr2=0x7ff24c14c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24a000700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff24c0cf230 0x7ff24c14c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24a000700 1 -- 172.21.15.12:0/3243847027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff244009a70 con 0x7ff24c160f10 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.582+0000 7ff24a000700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c160f10 0x7ff24c15f950 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7ff238005910 tx=0x7ff23800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.583+0000 7ff242a00700 1 -- 172.21.15.12:0/3243847027 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238005d70 con 0x7ff24c160f10 2024-03-29T11:31:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.583+0000 7ff242a00700 1 -- 172.21.15.12:0/3243847027 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2380070d0 con 0x7ff24c160f10 2024-03-29T11:31:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.583+0000 7ff242a00700 1 -- 172.21.15.12:0/3243847027 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238007460 con 0x7ff24c160f10 2024-03-29T11:31:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.583+0000 7ff251a00700 1 -- 172.21.15.12:0/3243847027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c14d0b0 con 0x7ff24c160f10 2024-03-29T11:31:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.583+0000 7ff251a00700 1 -- 172.21.15.12:0/3243847027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c14d670 con 0x7ff24c160f10 2024-03-29T11:31:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.584+0000 7ff251a00700 1 -- 172.21.15.12:0/3243847027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff24c067720 con 0x7ff24c160f10 2024-03-29T11:31:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.585+0000 7ff242a00700 1 -- 172.21.15.12:0/3243847027 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff2380075c0 con 0x7ff24c160f10 2024-03-29T11:31:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.586+0000 7ff242a00700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff22c06adb0 0x7ff22c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.586+0000 7ff242a00700 1 -- 172.21.15.12:0/3243847027 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6645+0+0 (secure 0 0 0) 0x7ff238088d50 con 0x7ff24c160f10 2024-03-29T11:31:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.586+0000 7ff24aa00700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff22c06adb0 0x7ff22c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:41.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.587+0000 7ff24aa00700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff22c06adb0 0x7ff22c06d270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009b40 tx=0x7ff23c009210 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.590+0000 7ff242a00700 1 -- 172.21.15.12:0/3243847027 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff2380539d0 con 0x7ff24c160f10 2024-03-29T11:31:41.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.805+0000 7ff251a00700 1 -- 172.21.15.12:0/3243847027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff24c14d830 con 0x7ff24c160f10 2024-03-29T11:31:41.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.807+0000 7ff242a00700 1 -- 172.21.15.12:0/3243847027 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+19968 (secure 0 0 0) 0x7ff238017090 con 0x7ff24c160f10 2024-03-29T11:31:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:41.806 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":230,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:40.862882+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:31:39.075407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:31:39.845704+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:41.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 -- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff22c06adb0 msgr2=0x7ff22c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff22c06adb0 0x7ff22c06d270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009b40 tx=0x7ff23c009210 comp rx=0 tx=0).stop 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 -- 172.21.15.12:0/3243847027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c160f10 msgr2=0x7ff24c15f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c160f10 0x7ff24c15f950 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7ff238005910 tx=0x7ff23800c7b0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 -- 172.21.15.12:0/3243847027 shutdown_connections 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff22c06adb0 0x7ff22c06d270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff24c161590 0x7ff24c0b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff24c0cf230 0x7ff24c14c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 --2- 172.21.15.12:0/3243847027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c160f10 0x7ff24c15f950 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 -- 172.21.15.12:0/3243847027 >> 172.21.15.12:0/3243847027 conn(0x7ff24c08aff0 msgr2=0x7ff24c0cc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 -- 172.21.15.12:0/3243847027 shutdown_connections 2024-03-29T11:31:41.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:41.809+0000 7ff251a00700 1 -- 172.21.15.12:0/3243847027 wait complete. 2024-03-29T11:31:41.823 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-03-29T11:31:42.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.041+0000 7f8eefc00700 1 Processor -- start 2024-03-29T11:31:42.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.042+0000 7f8eefc00700 1 -- start start 2024-03-29T11:31:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eefc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8ee81611b0 0x7f8ee815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eefc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee815f4d0 0x7f8ee80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eefc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ee80d21f0 0x7f8ee80cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eefc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee80cf180 con 0x7f8ee80d21f0 2024-03-29T11:31:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eefc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee80ce270 con 0x7f8ee81611b0 2024-03-29T11:31:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eefc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee815b600 con 0x7f8ee815f4d0 2024-03-29T11:31:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eedc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ee80d21f0 0x7f8ee80cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eedc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ee80d21f0 0x7f8ee80cec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39488/0 (socket says 172.21.15.12:39488) 2024-03-29T11:31:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eedc00700 1 -- 172.21.15.12:0/963923894 learned_addr learned my addr 172.21.15.12:0/963923894 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8ee7e00700 1 --2- 172.21.15.12:0/963923894 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee815f4d0 0x7f8ee80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eed200700 1 --2- 172.21.15.12:0/963923894 >> v2:172.21.15.159:3300/0 conn(0x7f8ee81611b0 0x7f8ee815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eedc00700 1 -- 172.21.15.12:0/963923894 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee815f4d0 msgr2=0x7f8ee80ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eedc00700 1 --2- 172.21.15.12:0/963923894 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee815f4d0 0x7f8ee80ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eedc00700 1 -- 172.21.15.12:0/963923894 >> v2:172.21.15.159:3300/0 conn(0x7f8ee81611b0 msgr2=0x7f8ee815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eedc00700 1 --2- 172.21.15.12:0/963923894 >> v2:172.21.15.159:3300/0 conn(0x7f8ee81611b0 0x7f8ee815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eedc00700 1 -- 172.21.15.12:0/963923894 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee80d2760 con 0x7f8ee80d21f0 2024-03-29T11:31:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8ee7e00700 1 --2- 172.21.15.12:0/963923894 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee815f4d0 0x7f8ee80ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.043+0000 7f8eed200700 1 --2- 172.21.15.12:0/963923894 >> v2:172.21.15.159:3300/0 conn(0x7f8ee81611b0 0x7f8ee815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.044+0000 7f8eedc00700 1 --2- 172.21.15.12:0/963923894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 0x7f8ee80cec40 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f8ee808b520 tx=0x7f8ee000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd2f524a66f3b2e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.044+0000 7f8ee7400700 1 -- 172.21.15.12:0/963923894 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee000e070 con 0x7f8ee80d21f0 2024-03-29T11:31:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.044+0000 7f8ee7400700 1 -- 172.21.15.12:0/963923894 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee000b9c0 con 0x7f8ee80d21f0 2024-03-29T11:31:42.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.044+0000 7f8ee7400700 1 -- 172.21.15.12:0/963923894 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee000bd70 con 0x7f8ee80d21f0 2024-03-29T11:31:42.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.044+0000 7f8eefc00700 1 -- 172.21.15.12:0/963923894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 msgr2=0x7f8ee80cec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:42.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.044+0000 7f8eefc00700 1 --2- 172.21.15.12:0/963923894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 0x7f8ee80cec40 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f8ee808b520 tx=0x7f8ee000b330 comp rx=0 tx=0).stop 2024-03-29T11:31:42.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.045+0000 7f8eefc00700 1 -- 172.21.15.12:0/963923894 shutdown_connections 2024-03-29T11:31:42.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.045+0000 7f8eefc00700 1 --2- 172.21.15.12:0/963923894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 0x7f8ee80cec40 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.045+0000 7f8eefc00700 1 --2- 172.21.15.12:0/963923894 >> v2:172.21.15.159:3300/0 conn(0x7f8ee81611b0 0x7f8ee815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.045+0000 7f8eefc00700 1 --2- 172.21.15.12:0/963923894 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee815f4d0 0x7f8ee80ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.045+0000 7f8eefc00700 1 -- 172.21.15.12:0/963923894 >> 172.21.15.12:0/963923894 conn(0x7f8ee808aff0 msgr2=0x7f8ee80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:42.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.045+0000 7f8eefc00700 1 -- 172.21.15.12:0/963923894 shutdown_connections 2024-03-29T11:31:42.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.045+0000 7f8eefc00700 1 -- 172.21.15.12:0/963923894 wait complete. 2024-03-29T11:31:42.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.046+0000 7f8eefc00700 1 Processor -- start 2024-03-29T11:31:42.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.046+0000 7f8eefc00700 1 -- start start 2024-03-29T11:31:42.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.047+0000 7f8eefc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 0x7f8ee80cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:42.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.047+0000 7f8eefc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee815f4d0 0x7f8ee8157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:42.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.047+0000 7f8eefc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ee81611b0 0x7f8ee80cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.047+0000 7f8eefc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80cc1a0 con 0x7f8ee80d21f0 2024-03-29T11:31:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.047+0000 7f8eefc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80cc480 con 0x7f8ee81611b0 2024-03-29T11:31:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.047+0000 7f8eefc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80cc760 con 0x7f8ee815f4d0 2024-03-29T11:31:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8eed200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 0x7f8ee80cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8eed200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 0x7f8ee80cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39494/0 (socket says 172.21.15.12:39494) 2024-03-29T11:31:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8eed200700 1 -- 172.21.15.12:0/3644093260 learned_addr learned my addr 172.21.15.12:0/3644093260 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8eed200700 1 -- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee815f4d0 msgr2=0x7f8ee8157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8eed200700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee815f4d0 0x7f8ee8157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8eed200700 1 -- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ee81611b0 msgr2=0x7f8ee80cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8eed200700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ee81611b0 0x7f8ee80cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8eed200700 1 -- 172.21.15.12:0/3644093260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee0009b30 con 0x7f8ee80d21f0 2024-03-29T11:31:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8eed200700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 0x7f8ee80cd240 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00e810 tx=0x7f8edc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8ee4c00700 1 -- 172.21.15.12:0/3644093260 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc00c020 con 0x7f8ee80d21f0 2024-03-29T11:31:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8ee4c00700 1 -- 172.21.15.12:0/3644093260 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc004250 con 0x7f8ee80d21f0 2024-03-29T11:31:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.048+0000 7f8eefc00700 1 -- 172.21.15.12:0/3644093260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee80cca40 con 0x7f8ee80d21f0 2024-03-29T11:31:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.049+0000 7f8ee4c00700 1 -- 172.21.15.12:0/3644093260 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc0036a0 con 0x7f8ee80d21f0 2024-03-29T11:31:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.049+0000 7f8eefc00700 1 -- 172.21.15.12:0/3644093260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee8158200 con 0x7f8ee80d21f0 2024-03-29T11:31:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.050+0000 7f8eefc00700 1 -- 172.21.15.12:0/3644093260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee80cd090 con 0x7f8ee80d21f0 2024-03-29T11:31:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.051+0000 7f8ee4c00700 1 -- 172.21.15.12:0/3644093260 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8edc003800 con 0x7f8ee80d21f0 2024-03-29T11:31:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.051+0000 7f8ee4c00700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8ec806adb0 0x7f8ec806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.052+0000 7f8ee4c00700 1 -- 172.21.15.12:0/3644093260 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f8edc089700 con 0x7f8ee80d21f0 2024-03-29T11:31:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.052+0000 7f8ee7e00700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8ec806adb0 0x7f8ec806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.053+0000 7f8ee7e00700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8ec806adb0 0x7f8ec806d270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4006fd0 tx=0x7f8ed4008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:42.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.056+0000 7f8ee4c00700 1 -- 172.21.15.12:0/3644093260 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8edc054310 con 0x7f8ee80d21f0 2024-03-29T11:31:42.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.318+0000 7f8eefc00700 1 -- 172.21.15.12:0/3644093260 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8ee8067720 con 0x7f8ee80d21f0 2024-03-29T11:31:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.884+0000 7f8ee4c00700 1 -- 172.21.15.12:0/3644093260 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v111) v1 ==== 121+0+0 (secure 0 0 0) 0x7f8edc057930 con 0x7f8ee80d21f0 2024-03-29T11:31:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.887+0000 7f8eefc00700 1 -- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8ec806adb0 msgr2=0x7f8ec806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.887+0000 7f8eefc00700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8ec806adb0 0x7f8ec806d270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4006fd0 tx=0x7f8ed4008040 comp rx=0 tx=0).stop 2024-03-29T11:31:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.887+0000 7f8eefc00700 1 -- 172.21.15.12:0/3644093260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 msgr2=0x7f8ee80cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.887+0000 7f8eefc00700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 0x7f8ee80cd240 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00e810 tx=0x7f8edc00ebd0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.887+0000 7f8eefc00700 1 -- 172.21.15.12:0/3644093260 shutdown_connections 2024-03-29T11:31:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.887+0000 7f8eefc00700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8ec806adb0 0x7f8ec806d270 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.887+0000 7f8eefc00700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ee81611b0 0x7f8ee80cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.887+0000 7f8eefc00700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ee80d21f0 0x7f8ee80cd240 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.887+0000 7f8eefc00700 1 --2- 172.21.15.12:0/3644093260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ee815f4d0 0x7f8ee8157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.887+0000 7f8eefc00700 1 -- 172.21.15.12:0/3644093260 >> 172.21.15.12:0/3644093260 conn(0x7f8ee808aff0 msgr2=0x7f8ee806c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.888+0000 7f8eefc00700 1 -- 172.21.15.12:0/3644093260 shutdown_connections 2024-03-29T11:31:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:42.888+0000 7f8eefc00700 1 -- 172.21.15.12:0/3644093260 wait complete. 2024-03-29T11:31:42.903 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T11:31:42.903 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-03-29T11:31:43.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.117+0000 7f6c98800700 1 Processor -- start 2024-03-29T11:31:43.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.117+0000 7f6c98800700 1 -- start start 2024-03-29T11:31:43.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.118+0000 7f6c98800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015a5c0 0x7f6c900709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:43.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.118+0000 7f6c98800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6c9015dfe0 0x7f6c900ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:43.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.118+0000 7f6c98800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900cb310 con 0x7f6c900ce670 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.118+0000 7f6c98800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c900ceb00 con 0x7f6c9015dfe0 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.118+0000 7f6c98800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90070f00 con 0x7f6c9015a5c0 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c95400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6c9015dfe0 0x7f6c900ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c96800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6c900ce670 0x7f6c900b2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47224/0 (socket says 172.21.15.12:47224) 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c95400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6c9015dfe0 0x7f6c900ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39508/0 (socket says 172.21.15.12:39508) 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c96800700 1 -- 172.21.15.12:0/3229083744 learned_addr learned my addr 172.21.15.12:0/3229083744 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c95e00700 1 --2- 172.21.15.12:0/3229083744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015a5c0 0x7f6c900709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c95400700 1 -- 172.21.15.12:0/3229083744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015a5c0 msgr2=0x7f6c900709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c95400700 1 --2- 172.21.15.12:0/3229083744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015a5c0 0x7f6c900709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c95400700 1 -- 172.21.15.12:0/3229083744 >> v1:172.21.15.12:6789/0 conn(0x7f6c900ce670 legacy=0x7f6c900b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c95400700 1 -- 172.21.15.12:0/3229083744 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900cca90 con 0x7f6c9015dfe0 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c94a00700 1 -- 172.21.15.12:0/3229083744 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444842103 0 0) 0x7f6c900cb310 con 0x7f6c900ce670 2024-03-29T11:31:43.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.119+0000 7f6c95400700 1 --2- 172.21.15.12:0/3229083744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015dfe0 0x7f6c900ce130 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009840 tx=0x7f6c80009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c587d22e99d3e80c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:43.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c94a00700 1 -- 172.21.15.12:0/3229083744 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80011830 con 0x7f6c9015dfe0 2024-03-29T11:31:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c94a00700 1 -- 172.21.15.12:0/3229083744 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c80011990 con 0x7f6c9015dfe0 2024-03-29T11:31:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c94a00700 1 -- 172.21.15.12:0/3229083744 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80011d20 con 0x7f6c9015dfe0 2024-03-29T11:31:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c98800700 1 -- 172.21.15.12:0/3229083744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015dfe0 msgr2=0x7f6c900ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c98800700 1 --2- 172.21.15.12:0/3229083744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015dfe0 0x7f6c900ce130 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009840 tx=0x7f6c80009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c98800700 1 -- 172.21.15.12:0/3229083744 shutdown_connections 2024-03-29T11:31:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c98800700 1 --2- 172.21.15.12:0/3229083744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015a5c0 0x7f6c900709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c98800700 1 --2- 172.21.15.12:0/3229083744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015dfe0 0x7f6c900ce130 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c98800700 1 -- 172.21.15.12:0/3229083744 >> 172.21.15.12:0/3229083744 conn(0x7f6c900b0d70 msgr2=0x7f6c900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c98800700 1 -- 172.21.15.12:0/3229083744 shutdown_connections 2024-03-29T11:31:43.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.120+0000 7f6c98800700 1 -- 172.21.15.12:0/3229083744 wait complete. 2024-03-29T11:31:43.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.122+0000 7f6c98800700 1 Processor -- start 2024-03-29T11:31:43.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.122+0000 7f6c98800700 1 -- start start 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.123+0000 7f6c98800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c900ce670 0x7f6c900cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.123+0000 7f6c98800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015a5c0 0x7f6c901f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.123+0000 7f6c98800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015dfe0 0x7f6c9015cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.123+0000 7f6c98800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900cb110 con 0x7f6c9015a5c0 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.123+0000 7f6c98800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901f4090 con 0x7f6c900ce670 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.123+0000 7f6c98800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901f6c40 con 0x7f6c9015dfe0 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.123+0000 7f6c95400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015a5c0 0x7f6c901f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.123+0000 7f6c95400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015a5c0 0x7f6c901f3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39518/0 (socket says 172.21.15.12:39518) 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.123+0000 7f6c95400700 1 -- 172.21.15.12:0/528779763 learned_addr learned my addr 172.21.15.12:0/528779763 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c95e00700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c900ce670 0x7f6c900cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c96800700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015dfe0 0x7f6c9015cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c95400700 1 -- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015dfe0 msgr2=0x7f6c9015cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c95400700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015dfe0 0x7f6c9015cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c95400700 1 -- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c900ce670 msgr2=0x7f6c900cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c95400700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c900ce670 0x7f6c900cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c95400700 1 -- 172.21.15.12:0/528779763 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c800094a0 con 0x7f6c9015a5c0 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c95e00700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c900ce670 0x7f6c900cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c96800700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015dfe0 0x7f6c9015cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c95400700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015a5c0 0x7f6c901f3b50 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009f50 tx=0x7f6c80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c8a000700 1 -- 172.21.15.12:0/528779763 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c800038a0 con 0x7f6c9015a5c0 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c8a000700 1 -- 172.21.15.12:0/528779763 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8001f070 con 0x7f6c9015a5c0 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c98800700 1 -- 172.21.15.12:0/528779763 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c901f6f20 con 0x7f6c9015a5c0 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.124+0000 7f6c8a000700 1 -- 172.21.15.12:0/528779763 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80003c40 con 0x7f6c9015a5c0 2024-03-29T11:31:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.125+0000 7f6c98800700 1 -- 172.21.15.12:0/528779763 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c901f74e0 con 0x7f6c9015a5c0 2024-03-29T11:31:43.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.126+0000 7f6c98800700 1 -- 172.21.15.12:0/528779763 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c90067720 con 0x7f6c9015a5c0 2024-03-29T11:31:43.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.126+0000 7f6c8a000700 1 -- 172.21.15.12:0/528779763 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6c80026480 con 0x7f6c9015a5c0 2024-03-29T11:31:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.128+0000 7f6c8a000700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7406adb0 0x7f6c7406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.128+0000 7f6c8a000700 1 -- 172.21.15.12:0/528779763 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f6c80021070 con 0x7f6c9015a5c0 2024-03-29T11:31:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.128+0000 7f6c95e00700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7406adb0 0x7f6c7406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:43.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.129+0000 7f6c95e00700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7406adb0 0x7f6c7406d270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009a00 tx=0x7f6c84008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:43.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.132+0000 7f6c8a000700 1 -- 172.21.15.12:0/528779763 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c80055fe0 con 0x7f6c9015a5c0 2024-03-29T11:31:43.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.393+0000 7f6c98800700 1 -- 172.21.15.12:0/528779763 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6c900b1ad0 con 0x7f6c9015a5c0 2024-03-29T11:31:43.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.892+0000 7f6c8a000700 1 -- 172.21.15.12:0/528779763 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v112) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6c80059600 con 0x7f6c9015a5c0 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 -- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7406adb0 msgr2=0x7f6c7406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7406adb0 0x7f6c7406d270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009a00 tx=0x7f6c84008040 comp rx=0 tx=0).stop 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 -- 172.21.15.12:0/528779763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015a5c0 msgr2=0x7f6c901f3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015a5c0 0x7f6c901f3b50 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009f50 tx=0x7f6c80004750 comp rx=0 tx=0).stop 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 -- 172.21.15.12:0/528779763 shutdown_connections 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7406adb0 0x7f6c7406d270 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9015dfe0 0x7f6c9015cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c900ce670 0x7f6c900cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 --2- 172.21.15.12:0/528779763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9015a5c0 0x7f6c901f3b50 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 -- 172.21.15.12:0/528779763 >> 172.21.15.12:0/528779763 conn(0x7f6c900b0d70 msgr2=0x7f6c900aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:43.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.895+0000 7f6c98800700 1 -- 172.21.15.12:0/528779763 shutdown_connections 2024-03-29T11:31:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:43.896+0000 7f6c98800700 1 -- 172.21.15.12:0/528779763 wait complete. 2024-03-29T11:31:43.910 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-03-29T11:31:43.910 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:31:44.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.116+0000 7f6808200700 1 Processor -- start 2024-03-29T11:31:44.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.116+0000 7f6808200700 1 -- start start 2024-03-29T11:31:44.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.117+0000 7f6808200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6800161ca0 0x7f68000cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.117+0000 7f6808200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68000ceda0 con 0x7f6800160f10 2024-03-29T11:31:44.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.117+0000 7f6808200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68001612f0 con 0x7f68000d18b0 2024-03-29T11:31:44.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.117+0000 7f6808200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68000d1d30 con 0x7f6800161ca0 2024-03-29T11:31:44.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.117+0000 7f6805800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6800160f10 0x7f68000d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47234/0 (socket says 172.21.15.12:47234) 2024-03-29T11:31:44.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.117+0000 7f6805800700 1 -- 172.21.15.12:0/1498097984 learned_addr learned my addr 172.21.15.12:0/1498097984 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:44.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f6806200700 1 --2- 172.21.15.12:0/1498097984 >> v2:172.21.15.159:3300/0 conn(0x7f6800161ca0 0x7f68000cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303078031 0 0) 0x7f68000ceda0 con 0x7f6800160f10 2024-03-29T11:31:44.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e80035b0 con 0x7f6800160f10 2024-03-29T11:31:44.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936519078 0 0) 0x7f68001612f0 con 0x7f68000d18b0 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68000ceda0 con 0x7f68000d18b0 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1671532569 0 0) 0x7f67e80035b0 con 0x7f6800160f10 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 >> v2:172.21.15.159:3300/0 conn(0x7f6800161ca0 msgr2=0x7f68000cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 --2- 172.21.15.12:0/1498097984 >> v2:172.21.15.159:3300/0 conn(0x7f6800161ca0 0x7f68000cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 >> v1:172.21.15.159:6789/0 conn(0x7f68000d18b0 legacy=0x7f6800161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68000ce470 con 0x7f6800160f10 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f67f4001670 con 0x7f6800160f10 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 >> v1:172.21.15.12:6789/0 conn(0x7f6800160f10 legacy=0x7f68000d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67e8003b90 0x7f67e8003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67e8004850 0x7f67e8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67e8005210 0x7f67e800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000d1d30 con 0x7f67e8004850 2024-03-29T11:31:44.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e800c140 con 0x7f67e8003b90 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.118+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e800c300 con 0x7f67e8005210 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.119+0000 7f6804e00700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67e8004850 0x7f67e8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.119+0000 7f6805800700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67e8003b90 0x7f67e8003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.119+0000 7f6806200700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67e8005210 0x7f67e800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.119+0000 7f6804e00700 1 -- 172.21.15.12:0/1498097984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67e8005210 msgr2=0x7f67e800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.119+0000 7f6804e00700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67e8005210 0x7f67e800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.119+0000 7f6804e00700 1 -- 172.21.15.12:0/1498097984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67e8003b90 msgr2=0x7f67e8003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.119+0000 7f6804e00700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67e8003b90 0x7f67e8003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.119+0000 7f6804e00700 1 -- 172.21.15.12:0/1498097984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e80035b0 con 0x7f67e8004850 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.119+0000 7f6806200700 1 -- 172.21.15.12:0/1498097984 reap_dead start 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.119+0000 7f6804e00700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67e8004850 0x7f67e8004cd0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f67ec001940 tx=0x7f67e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e064b06466cb0f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.120+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec001430 con 0x7f67e8004850 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.120+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ec001d70 con 0x7f67e8004850 2024-03-29T11:31:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.120+0000 7f67ffe00700 1 -- 172.21.15.12:0/1498097984 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec005510 con 0x7f67e8004850 2024-03-29T11:31:44.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.120+0000 7f6808200700 1 -- 172.21.15.12:0/1498097984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67e8004850 msgr2=0x7f67e8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.120+0000 7f6808200700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67e8004850 0x7f67e8004cd0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f67ec001940 tx=0x7f67e8005880 comp rx=0 tx=0).stop 2024-03-29T11:31:44.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.120+0000 7f6808200700 1 -- 172.21.15.12:0/1498097984 shutdown_connections 2024-03-29T11:31:44.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.120+0000 7f6808200700 1 --2- 172.21.15.12:0/1498097984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67e8004850 0x7f67e8004cd0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.120+0000 7f6808200700 1 -- 172.21.15.12:0/1498097984 >> 172.21.15.12:0/1498097984 conn(0x7f680008aff0 msgr2=0x7f68000d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:44.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.120+0000 7f6808200700 1 -- 172.21.15.12:0/1498097984 shutdown_connections 2024-03-29T11:31:44.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.120+0000 7f6808200700 1 -- 172.21.15.12:0/1498097984 wait complete. 2024-03-29T11:31:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.123+0000 7f6808200700 1 Processor -- start 2024-03-29T11:31:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.123+0000 7f6808200700 1 -- start start 2024-03-29T11:31:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.124+0000 7f6808200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67e8004850 0x7f680015fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.124+0000 7f6808200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68000b5fe0 0x7f680009de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.124+0000 7f6808200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f680009e370 0x7f680015e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.124+0000 7f6808200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680009f500 con 0x7f680009e370 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.124+0000 7f6808200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680009e880 con 0x7f68000b5fe0 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.124+0000 7f6808200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680009eb60 con 0x7f67e8004850 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.124+0000 7f6806200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f680009e370 0x7f680015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f6806200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f680009e370 0x7f680015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39536/0 (socket says 172.21.15.12:39536) 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f6806200700 1 -- 172.21.15.12:0/2891410217 learned_addr learned my addr 172.21.15.12:0/2891410217 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f6804e00700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68000b5fe0 0x7f680009de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f6805800700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67e8004850 0x7f680015fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f6806200700 1 -- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67e8004850 msgr2=0x7f680015fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f6806200700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67e8004850 0x7f680015fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f6806200700 1 -- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68000b5fe0 msgr2=0x7f680009de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f6806200700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68000b5fe0 0x7f680009de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f6806200700 1 -- 172.21.15.12:0/2891410217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680009ee40 con 0x7f680009e370 2024-03-29T11:31:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f6806200700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f680009e370 0x7f680015e680 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f68000d1030 tx=0x7f67f8011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.125+0000 7f67fd600700 1 -- 172.21.15.12:0/2891410217 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f800e040 con 0x7f680009e370 2024-03-29T11:31:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.126+0000 7f67fd600700 1 -- 172.21.15.12:0/2891410217 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f800f040 con 0x7f680009e370 2024-03-29T11:31:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.126+0000 7f6808200700 1 -- 172.21.15.12:0/2891410217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680009f120 con 0x7f680009e370 2024-03-29T11:31:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.126+0000 7f67fd600700 1 -- 172.21.15.12:0/2891410217 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f800fdb0 con 0x7f680009e370 2024-03-29T11:31:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.126+0000 7f6808200700 1 -- 172.21.15.12:0/2891410217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680014efc0 con 0x7f680009e370 2024-03-29T11:31:44.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.127+0000 7f6808200700 1 -- 172.21.15.12:0/2891410217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680009f640 con 0x7f680009e370 2024-03-29T11:31:44.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.128+0000 7f67fd600700 1 -- 172.21.15.12:0/2891410217 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f67f8022460 con 0x7f680009e370 2024-03-29T11:31:44.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.128+0000 7f67fd600700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67e006adb0 0x7f67e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.129+0000 7f67fd600700 1 -- 172.21.15.12:0/2891410217 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f67f8089ef0 con 0x7f680009e370 2024-03-29T11:31:44.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.129+0000 7f6805800700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67e006adb0 0x7f67e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.130+0000 7f6805800700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67e006adb0 0x7f67e006d270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67f4001eb0 tx=0x7f67f4009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:44.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.133+0000 7f67fd600700 1 -- 172.21.15.12:0/2891410217 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f680009f640 con 0x7f680009e370 2024-03-29T11:31:44.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.393+0000 7f6808200700 1 -- 172.21.15.12:0/2891410217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f680009f640 con 0x7f680009e370 2024-03-29T11:31:44.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.394+0000 7f67fd600700 1 -- 172.21.15.12:0/2891410217 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5971 (secure 0 0 0) 0x7f680009f640 con 0x7f680009e370 2024-03-29T11:31:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:44.388 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":112,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:26.816217+0000","modified":"2024-03-29T11:31:30.290952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/2722218561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2722218561},{"type":"v1","addr":"172.21.15.12:6833","nonce":2722218561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:40.863222+0000","modified":"2024-03-29T11:31:43.883199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 -- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67e006adb0 msgr2=0x7f67e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67e006adb0 0x7f67e006d270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67f4001eb0 tx=0x7f67f4009040 comp rx=0 tx=0).stop 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 -- 172.21.15.12:0/2891410217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f680009e370 msgr2=0x7f680015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f680009e370 0x7f680015e680 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f68000d1030 tx=0x7f67f8011f80 comp rx=0 tx=0).stop 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 -- 172.21.15.12:0/2891410217 shutdown_connections 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67e006adb0 0x7f67e006d270 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f680009e370 0x7f680015e680 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67e8004850 0x7f680015fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 --2- 172.21.15.12:0/2891410217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68000b5fe0 0x7f680009de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 -- 172.21.15.12:0/2891410217 >> 172.21.15.12:0/2891410217 conn(0x7f680008aff0 msgr2=0x7f68000ca710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:44.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 -- 172.21.15.12:0/2891410217 shutdown_connections 2024-03-29T11:31:44.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.396+0000 7f6808200700 1 -- 172.21.15.12:0/2891410217 wait complete. 2024-03-29T11:31:44.392 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 112 2024-03-29T11:31:44.409 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:44.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.612+0000 7fc029800700 1 Processor -- start 2024-03-29T11:31:44.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.613+0000 7fc029800700 1 -- start start 2024-03-29T11:31:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc029800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc02415d760 0x7fc0240d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc029800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc02415db40 0x7fc0240cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc029800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0240d1330 con 0x7fc0240cf1f0 2024-03-29T11:31:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc029800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0240d1fb0 con 0x7fc02415db40 2024-03-29T11:31:44.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc029800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0240cf660 con 0x7fc02415d760 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc022000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc02415db40 0x7fc0240cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc023400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc0240cf1f0 0x7fc0240cffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47242/0 (socket says 172.21.15.12:47242) 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc023400700 1 -- 172.21.15.12:0/2838790431 learned_addr learned my addr 172.21.15.12:0/2838790431 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc022a00700 1 --2- 172.21.15.12:0/2838790431 >> v2:172.21.15.159:3300/0 conn(0x7fc02415d760 0x7fc0240d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc022000700 1 -- 172.21.15.12:0/2838790431 >> v2:172.21.15.159:3300/0 conn(0x7fc02415d760 msgr2=0x7fc0240d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc022000700 1 --2- 172.21.15.12:0/2838790431 >> v2:172.21.15.159:3300/0 conn(0x7fc02415d760 0x7fc0240d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.614+0000 7fc022000700 1 -- 172.21.15.12:0/2838790431 >> v1:172.21.15.12:6789/0 conn(0x7fc0240cf1f0 legacy=0x7fc0240cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.615+0000 7fc022000700 1 -- 172.21.15.12:0/2838790431 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0240d1610 con 0x7fc02415db40 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.615+0000 7fc022000700 1 --2- 172.21.15.12:0/2838790431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc02415db40 0x7fc0240cecb0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fc0240b61d0 tx=0x7fc010009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55f9ecd5beaddbb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.615+0000 7fc021600700 1 -- 172.21.15.12:0/2838790431 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010011830 con 0x7fc02415db40 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.615+0000 7fc021600700 1 -- 172.21.15.12:0/2838790431 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc010011990 con 0x7fc02415db40 2024-03-29T11:31:44.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.615+0000 7fc021600700 1 -- 172.21.15.12:0/2838790431 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010011d20 con 0x7fc02415db40 2024-03-29T11:31:44.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.615+0000 7fc029800700 1 -- 172.21.15.12:0/2838790431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc02415db40 msgr2=0x7fc0240cecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.615+0000 7fc029800700 1 --2- 172.21.15.12:0/2838790431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc02415db40 0x7fc0240cecb0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fc0240b61d0 tx=0x7fc010009b20 comp rx=0 tx=0).stop 2024-03-29T11:31:44.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.616+0000 7fc029800700 1 -- 172.21.15.12:0/2838790431 shutdown_connections 2024-03-29T11:31:44.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.616+0000 7fc029800700 1 --2- 172.21.15.12:0/2838790431 >> v2:172.21.15.159:3300/0 conn(0x7fc02415d760 0x7fc0240d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.616+0000 7fc029800700 1 --2- 172.21.15.12:0/2838790431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc02415db40 0x7fc0240cecb0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.616+0000 7fc029800700 1 -- 172.21.15.12:0/2838790431 >> 172.21.15.12:0/2838790431 conn(0x7fc02408aff0 msgr2=0x7fc0240d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:44.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.616+0000 7fc029800700 1 -- 172.21.15.12:0/2838790431 shutdown_connections 2024-03-29T11:31:44.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.616+0000 7fc029800700 1 -- 172.21.15.12:0/2838790431 wait complete. 2024-03-29T11:31:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.617+0000 7fc029800700 1 Processor -- start 2024-03-29T11:31:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.618+0000 7fc029800700 1 -- start start 2024-03-29T11:31:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.618+0000 7fc029800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0240cf1f0 0x7fc0240c7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.618+0000 7fc029800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc02415d760 0x7fc0240b8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.618+0000 7fc029800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc02415db40 0x7fc0240b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.618+0000 7fc029800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240b9300 con 0x7fc0240cf1f0 2024-03-29T11:31:44.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.618+0000 7fc029800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240b69e0 con 0x7fc02415db40 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.618+0000 7fc029800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240b9020 con 0x7fc02415d760 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc022a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0240cf1f0 0x7fc0240c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc022a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0240cf1f0 0x7fc0240c7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39546/0 (socket says 172.21.15.12:39546) 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc022a00700 1 -- 172.21.15.12:0/4135736561 learned_addr learned my addr 172.21.15.12:0/4135736561 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc022000700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc02415d760 0x7fc0240b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc023400700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc02415db40 0x7fc0240b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc022a00700 1 -- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc02415d760 msgr2=0x7fc0240b8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc022a00700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc02415d760 0x7fc0240b8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc022a00700 1 -- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc02415db40 msgr2=0x7fc0240b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc022a00700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc02415db40 0x7fc0240b64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc022a00700 1 -- 172.21.15.12:0/4135736561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0100094a0 con 0x7fc0240cf1f0 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc023400700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc02415db40 0x7fc0240b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.619+0000 7fc022a00700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0240cf1f0 0x7fc0240c7150 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fc01400e810 tx=0x7fc01400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.620+0000 7fc01aa00700 1 -- 172.21.15.12:0/4135736561 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014013610 con 0x7fc0240cf1f0 2024-03-29T11:31:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.620+0000 7fc01aa00700 1 -- 172.21.15.12:0/4135736561 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc014013770 con 0x7fc0240cf1f0 2024-03-29T11:31:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.620+0000 7fc029800700 1 -- 172.21.15.12:0/4135736561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0240b41b0 con 0x7fc0240cf1f0 2024-03-29T11:31:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.620+0000 7fc01aa00700 1 -- 172.21.15.12:0/4135736561 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014013b20 con 0x7fc0240cf1f0 2024-03-29T11:31:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.620+0000 7fc029800700 1 -- 172.21.15.12:0/4135736561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0240b4770 con 0x7fc0240cf1f0 2024-03-29T11:31:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.621+0000 7fc029800700 1 -- 172.21.15.12:0/4135736561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc024067720 con 0x7fc0240cf1f0 2024-03-29T11:31:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.622+0000 7fc01aa00700 1 -- 172.21.15.12:0/4135736561 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc014013c80 con 0x7fc0240cf1f0 2024-03-29T11:31:44.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.623+0000 7fc01aa00700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc00406adb0 0x7fc00406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:44.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.623+0000 7fc01aa00700 1 -- 172.21.15.12:0/4135736561 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7fc0140895f0 con 0x7fc0240cf1f0 2024-03-29T11:31:44.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.623+0000 7fc022000700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc00406adb0 0x7fc00406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:44.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.624+0000 7fc022000700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc00406adb0 0x7fc00406d270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc0240b61d0 tx=0x7fc01001c040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.627+0000 7fc01aa00700 1 -- 172.21.15.12:0/4135736561 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc014054170 con 0x7fc0240cf1f0 2024-03-29T11:31:44.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.841+0000 7fc029800700 1 -- 172.21.15.12:0/4135736561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0240d0d40 con 0x7fc0240cf1f0 2024-03-29T11:31:44.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.842+0000 7fc01aa00700 1 -- 172.21.15.12:0/4135736561 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+19955 (secure 0 0 0) 0x7fc014057790 con 0x7fc0240cf1f0 2024-03-29T11:31:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:44.837 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":231,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:41.866013+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:31:39.075407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:31:39.845704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:44.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.844+0000 7fc029800700 1 -- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc00406adb0 msgr2=0x7fc00406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.844+0000 7fc029800700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc00406adb0 0x7fc00406d270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc0240b61d0 tx=0x7fc01001c040 comp rx=0 tx=0).stop 2024-03-29T11:31:44.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.844+0000 7fc029800700 1 -- 172.21.15.12:0/4135736561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0240cf1f0 msgr2=0x7fc0240c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:44.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.844+0000 7fc029800700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0240cf1f0 0x7fc0240c7150 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fc01400e810 tx=0x7fc01400eb20 comp rx=0 tx=0).stop 2024-03-29T11:31:44.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.844+0000 7fc029800700 1 -- 172.21.15.12:0/4135736561 shutdown_connections 2024-03-29T11:31:44.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.844+0000 7fc029800700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc00406adb0 0x7fc00406d270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.844+0000 7fc029800700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc02415db40 0x7fc0240b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.844+0000 7fc029800700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0240cf1f0 0x7fc0240c7150 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.845+0000 7fc029800700 1 --2- 172.21.15.12:0/4135736561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc02415d760 0x7fc0240b8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:44.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.845+0000 7fc029800700 1 -- 172.21.15.12:0/4135736561 >> 172.21.15.12:0/4135736561 conn(0x7fc02408aff0 msgr2=0x7fc0240b0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:44.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.845+0000 7fc029800700 1 -- 172.21.15.12:0/4135736561 shutdown_connections 2024-03-29T11:31:44.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:44.845+0000 7fc029800700 1 -- 172.21.15.12:0/4135736561 wait complete. 2024-03-29T11:31:44.857 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T11:31:44.858 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:31:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.063+0000 7fe65a800700 1 Processor -- start 2024-03-29T11:31:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.063+0000 7fe65a800700 1 -- start start 2024-03-29T11:31:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.064+0000 7fe65a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe65406a790 0x7fe65406ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.065+0000 7fe65a800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65406f850 con 0x7fe6540afbe0 2024-03-29T11:31:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.065+0000 7fe65a800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65406e3f0 con 0x7fe6540cebd0 2024-03-29T11:31:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.065+0000 7fe65a800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65406b0b0 con 0x7fe65406a790 2024-03-29T11:31:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.065+0000 7fe652a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe6540afbe0 0x7fe65406dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47258/0 (socket says 172.21.15.12:47258) 2024-03-29T11:31:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.065+0000 7fe652a00700 1 -- 172.21.15.12:0/1011441553 learned_addr learned my addr 172.21.15.12:0/1011441553 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.065+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771593791 0 0) 0x7fe65406f850 con 0x7fe6540afbe0 2024-03-29T11:31:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.065+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6380035b0 con 0x7fe6540afbe0 2024-03-29T11:31:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.065+0000 7fe653400700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe65406a790 0x7fe65406ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2790840718 0 0) 0x7fe6380035b0 con 0x7fe6540afbe0 2024-03-29T11:31:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe65406a790 msgr2=0x7fe65406ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe65406a790 0x7fe65406ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 >> v1:172.21.15.159:6789/0 conn(0x7fe6540cebd0 legacy=0x7fe6540cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65406fad0 con 0x7fe6540afbe0 2024-03-29T11:31:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fe640001670 con 0x7fe6540afbe0 2024-03-29T11:31:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 >> v1:172.21.15.12:6789/0 conn(0x7fe6540afbe0 legacy=0x7fe65406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe638003a30 0x7fe638003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe638004850 0x7fe638004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe65406a790 0x7fe638005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65406b0b0 con 0x7fe638003a30 2024-03-29T11:31:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63800bb40 con 0x7fe638004850 2024-03-29T11:31:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63800bd00 con 0x7fe65406a790 2024-03-29T11:31:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe653400700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe638003a30 0x7fe638003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe653e00700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe65406a790 0x7fe638005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe652a00700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe638004850 0x7fe638004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe653400700 1 -- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe65406a790 msgr2=0x7fe638005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe653400700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe65406a790 0x7fe638005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe653400700 1 -- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe638004850 msgr2=0x7fe638004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe653400700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe638004850 0x7fe638004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe653400700 1 -- 172.21.15.12:0/1011441553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6380035b0 con 0x7fe638003a30 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.066+0000 7fe653e00700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe65406a790 0x7fe638005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.067+0000 7fe653400700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe638003a30 0x7fe638003e10 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fe644009b60 tx=0x7fe638005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61035cc8c3751736 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.067+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64400e6c0 con 0x7fe638003a30 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.067+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64400e820 con 0x7fe638003a30 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.067+0000 7fe652000700 1 -- 172.21.15.12:0/1011441553 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64400eb90 con 0x7fe638003a30 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.067+0000 7fe65a800700 1 -- 172.21.15.12:0/1011441553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe638003a30 msgr2=0x7fe638003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.067+0000 7fe65a800700 1 --2- 172.21.15.12:0/1011441553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe638003a30 0x7fe638003e10 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fe644009b60 tx=0x7fe638005880 comp rx=0 tx=0).stop 2024-03-29T11:31:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.067+0000 7fe653e00700 1 -- 172.21.15.12:0/1011441553 reap_dead start 2024-03-29T11:31:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.068+0000 7fe65a800700 1 -- 172.21.15.12:0/1011441553 shutdown_connections 2024-03-29T11:31:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.068+0000 7fe65a800700 1 -- 172.21.15.12:0/1011441553 >> 172.21.15.12:0/1011441553 conn(0x7fe6540b5ab0 msgr2=0x7fe6540b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.068+0000 7fe65a800700 1 -- 172.21.15.12:0/1011441553 shutdown_connections 2024-03-29T11:31:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.068+0000 7fe65a800700 1 -- 172.21.15.12:0/1011441553 wait complete. 2024-03-29T11:31:45.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.069+0000 7fe65a800700 1 Processor -- start 2024-03-29T11:31:45.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.070+0000 7fe65a800700 1 -- start start 2024-03-29T11:31:45.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.070+0000 7fe65a800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe638003a30 0x7fe654153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.070+0000 7fe65a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6540b1030 0x7fe65415c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.070+0000 7fe65a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe65415a9a0 0x7fe65415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.070+0000 7fe65a800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65415b370 con 0x7fe65415a9a0 2024-03-29T11:31:45.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.070+0000 7fe65a800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe654153750 con 0x7fe638003a30 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.070+0000 7fe65a800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540b14f0 con 0x7fe6540b1030 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe653e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe65415a9a0 0x7fe65415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe653e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe65415a9a0 0x7fe65415ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39562/0 (socket says 172.21.15.12:39562) 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe653e00700 1 -- 172.21.15.12:0/1153075926 learned_addr learned my addr 172.21.15.12:0/1153075926 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe652a00700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6540b1030 0x7fe65415c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe653400700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe638003a30 0x7fe654153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe653e00700 1 -- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6540b1030 msgr2=0x7fe65415c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe653e00700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6540b1030 0x7fe65415c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe653e00700 1 -- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe638003a30 msgr2=0x7fe654153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe653e00700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe638003a30 0x7fe654153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe653e00700 1 -- 172.21.15.12:0/1153075926 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540b17d0 con 0x7fe65415a9a0 2024-03-29T11:31:45.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe653e00700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe65415a9a0 0x7fe65415ae30 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fe6540cefb0 tx=0x7fe64c00c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.071+0000 7fe64b400700 1 -- 172.21.15.12:0/1153075926 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0020b0 con 0x7fe65415a9a0 2024-03-29T11:31:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.072+0000 7fe64b400700 1 -- 172.21.15.12:0/1153075926 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c003c10 con 0x7fe65415a9a0 2024-03-29T11:31:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.072+0000 7fe65a800700 1 -- 172.21.15.12:0/1153075926 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6540b1ab0 con 0x7fe65415a9a0 2024-03-29T11:31:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.072+0000 7fe64b400700 1 -- 172.21.15.12:0/1153075926 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c00c9a0 con 0x7fe65415a9a0 2024-03-29T11:31:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.072+0000 7fe65a800700 1 -- 172.21.15.12:0/1153075926 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65415cb60 con 0x7fe65415a9a0 2024-03-29T11:31:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.073+0000 7fe65a800700 1 -- 172.21.15.12:0/1153075926 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6540b5900 con 0x7fe65415a9a0 2024-03-29T11:31:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.074+0000 7fe64b400700 1 -- 172.21.15.12:0/1153075926 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe64c00cb00 con 0x7fe65415a9a0 2024-03-29T11:31:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.074+0000 7fe64b400700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe63406adb0 0x7fe63406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.075+0000 7fe64b400700 1 -- 172.21.15.12:0/1153075926 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7fe64c089b10 con 0x7fe65415a9a0 2024-03-29T11:31:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.075+0000 7fe653400700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe63406adb0 0x7fe63406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.076+0000 7fe653400700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe63406adb0 0x7fe63406d270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe6440090b0 tx=0x7fe644008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.079+0000 7fe64b400700 1 -- 172.21.15.12:0/1153075926 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe64c054790 con 0x7fe65415a9a0 2024-03-29T11:31:45.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.292+0000 7fe65a800700 1 -- 172.21.15.12:0/1153075926 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe6540cfb70 con 0x7fe63406adb0 2024-03-29T11:31:45.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.309+0000 7fe64b400700 1 -- 172.21.15.12:0/1153075926 <== mgr.10459 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528658 (secure 0 0 0) 0x7fe644022040 con 0x7fe63406adb0 2024-03-29T11:31:45.312 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:45.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.314+0000 7fe65a800700 1 -- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe63406adb0 msgr2=0x7fe63406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.314+0000 7fe65a800700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe63406adb0 0x7fe63406d270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe6440090b0 tx=0x7fe644008030 comp rx=0 tx=0).stop 2024-03-29T11:31:45.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.314+0000 7fe65a800700 1 -- 172.21.15.12:0/1153075926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe65415a9a0 msgr2=0x7fe65415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.314+0000 7fe65a800700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe65415a9a0 0x7fe65415ae30 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fe6540cefb0 tx=0x7fe64c00c3c0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.314+0000 7fe65a800700 1 -- 172.21.15.12:0/1153075926 shutdown_connections 2024-03-29T11:31:45.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.314+0000 7fe65a800700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe63406adb0 0x7fe63406d270 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.314+0000 7fe65a800700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe65415a9a0 0x7fe65415ae30 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.314+0000 7fe65a800700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe638003a30 0x7fe654153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.314+0000 7fe65a800700 1 --2- 172.21.15.12:0/1153075926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6540b1030 0x7fe65415c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.314+0000 7fe65a800700 1 -- 172.21.15.12:0/1153075926 >> 172.21.15.12:0/1153075926 conn(0x7fe6540b5ab0 msgr2=0x7fe6540ae740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:45.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.315+0000 7fe65a800700 1 -- 172.21.15.12:0/1153075926 shutdown_connections 2024-03-29T11:31:45.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.315+0000 7fe65a800700 1 -- 172.21.15.12:0/1153075926 wait complete. 2024-03-29T11:31:45.319 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:31:45.327 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-29T11:31:45.171760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596333,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":702,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":417976,"kb_used_data":23848,"kb_used_omap":12,"kb_used_meta":393971,"kb_avail":754556744,"statfs":{"total":773094113280,"available":772666105856,"internally_reserved":0,"allocated":24420352,"data_stored":12483340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":403426873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.835751"},"pg_stats":[{"pgid":"1.0","version":"215'366","reported_seq":1275,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.871423+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:31:40.871423+0000","last_peered":"2024-03-29T11:31:40.871423+0000","last_clean":"2024-03-29T11:31:40.871423+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:31:40.871423+0000","last_undegraded":"2024-03-29T11:31:40.871423+0000","last_fullsized":"2024-03-29T11:31:40.871423+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.874693+0000","last_change":"2024-03-29T11:31:26.318923+0000","last_active":"2024-03-29T11:31:40.874693+0000","last_peered":"2024-03-29T11:31:40.874693+0000","last_clean":"2024-03-29T11:31:40.874693+0000","last_became_active":"2024-03-29T11:31:26.318658+0000","last_became_peered":"2024-03-29T11:31:26.318658+0000","last_unstale":"2024-03-29T11:31:40.874693+0000","last_undegraded":"2024-03-29T11:31:40.874693+0000","last_fullsized":"2024-03-29T11:31:40.874693+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:25:12.671618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873178+0000","last_change":"2024-03-29T11:31:26.844835+0000","last_active":"2024-03-29T11:31:40.873178+0000","last_peered":"2024-03-29T11:31:40.873178+0000","last_clean":"2024-03-29T11:31:40.873178+0000","last_became_active":"2024-03-29T11:31:26.844570+0000","last_became_peered":"2024-03-29T11:31:26.844570+0000","last_unstale":"2024-03-29T11:31:40.873178+0000","last_undegraded":"2024-03-29T11:31:40.873178+0000","last_fullsized":"2024-03-29T11:31:40.873178+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:53:43.020271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950330+0000","last_change":"2024-03-29T11:31:40.368551+0000","last_active":"2024-03-29T11:31:41.950330+0000","last_peered":"2024-03-29T11:31:41.950330+0000","last_clean":"2024-03-29T11:31:41.950330+0000","last_became_active":"2024-03-29T11:31:40.368254+0000","last_became_peered":"2024-03-29T11:31:40.368254+0000","last_unstale":"2024-03-29T11:31:41.950330+0000","last_undegraded":"2024-03-29T11:31:41.950330+0000","last_fullsized":"2024-03-29T11:31:41.950330+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:35:21.805809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.878868+0000","last_change":"2024-03-29T11:31:40.887274+0000","last_active":"2024-03-29T11:31:41.878868+0000","last_peered":"2024-03-29T11:31:41.878868+0000","last_clean":"2024-03-29T11:31:41.878868+0000","last_became_active":"2024-03-29T11:31:40.886918+0000","last_became_peered":"2024-03-29T11:31:40.886918+0000","last_unstale":"2024-03-29T11:31:41.878868+0000","last_undegraded":"2024-03-29T11:31:41.878868+0000","last_fullsized":"2024-03-29T11:31:41.878868+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:50:30.787108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.886076+0000","last_change":"2024-03-29T11:31:26.318569+0000","last_active":"2024-03-29T11:31:40.886076+0000","last_peered":"2024-03-29T11:31:40.886076+0000","last_clean":"2024-03-29T11:31:40.886076+0000","last_became_active":"2024-03-29T11:31:26.318298+0000","last_became_peered":"2024-03-29T11:31:26.318298+0000","last_unstale":"2024-03-29T11:31:40.886076+0000","last_undegraded":"2024-03-29T11:31:40.886076+0000","last_fullsized":"2024-03-29T11:31:40.886076+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:05:48.527536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885182+0000","last_change":"2024-03-29T11:31:26.843285+0000","last_active":"2024-03-29T11:31:40.885182+0000","last_peered":"2024-03-29T11:31:40.885182+0000","last_clean":"2024-03-29T11:31:40.885182+0000","last_became_active":"2024-03-29T11:31:26.841561+0000","last_became_peered":"2024-03-29T11:31:26.841561+0000","last_unstale":"2024-03-29T11:31:40.885182+0000","last_undegraded":"2024-03-29T11:31:40.885182+0000","last_fullsized":"2024-03-29T11:31:40.885182+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:47.872748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879251+0000","last_change":"2024-03-29T11:31:40.367954+0000","last_active":"2024-03-29T11:31:41.879251+0000","last_peered":"2024-03-29T11:31:41.879251+0000","last_clean":"2024-03-29T11:31:41.879251+0000","last_became_active":"2024-03-29T11:31:40.367669+0000","last_became_peered":"2024-03-29T11:31:40.367669+0000","last_unstale":"2024-03-29T11:31:41.879251+0000","last_undegraded":"2024-03-29T11:31:41.879251+0000","last_fullsized":"2024-03-29T11:31:41.879251+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:58:10.780397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.881307+0000","last_change":"2024-03-29T11:31:40.897163+0000","last_active":"2024-03-29T11:31:41.881307+0000","last_peered":"2024-03-29T11:31:41.881307+0000","last_clean":"2024-03-29T11:31:41.881307+0000","last_became_active":"2024-03-29T11:31:40.896742+0000","last_became_peered":"2024-03-29T11:31:40.896742+0000","last_unstale":"2024-03-29T11:31:41.881307+0000","last_undegraded":"2024-03-29T11:31:41.881307+0000","last_fullsized":"2024-03-29T11:31:41.881307+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:56:07.738932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885453+0000","last_change":"2024-03-29T11:31:26.310072+0000","last_active":"2024-03-29T11:31:40.885453+0000","last_peered":"2024-03-29T11:31:40.885453+0000","last_clean":"2024-03-29T11:31:40.885453+0000","last_became_active":"2024-03-29T11:31:26.309254+0000","last_became_peered":"2024-03-29T11:31:26.309254+0000","last_unstale":"2024-03-29T11:31:40.885453+0000","last_undegraded":"2024-03-29T11:31:40.885453+0000","last_fullsized":"2024-03-29T11:31:40.885453+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:00:36.402074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882760+0000","last_change":"2024-03-29T11:31:26.842129+0000","last_active":"2024-03-29T11:31:40.882760+0000","last_peered":"2024-03-29T11:31:40.882760+0000","last_clean":"2024-03-29T11:31:40.882760+0000","last_became_active":"2024-03-29T11:31:26.841888+0000","last_became_peered":"2024-03-29T11:31:26.841888+0000","last_unstale":"2024-03-29T11:31:40.882760+0000","last_undegraded":"2024-03-29T11:31:40.882760+0000","last_fullsized":"2024-03-29T11:31:40.882760+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:32.997118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951488+0000","last_change":"2024-03-29T11:31:40.372014+0000","last_active":"2024-03-29T11:31:41.951488+0000","last_peered":"2024-03-29T11:31:41.951488+0000","last_clean":"2024-03-29T11:31:41.951488+0000","last_became_active":"2024-03-29T11:31:40.369598+0000","last_became_peered":"2024-03-29T11:31:40.369598+0000","last_unstale":"2024-03-29T11:31:41.951488+0000","last_undegraded":"2024-03-29T11:31:41.951488+0000","last_fullsized":"2024-03-29T11:31:41.951488+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:55:10.199622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951120+0000","last_change":"2024-03-29T11:31:40.901391+0000","last_active":"2024-03-29T11:31:41.951120+0000","last_peered":"2024-03-29T11:31:41.951120+0000","last_clean":"2024-03-29T11:31:41.951120+0000","last_became_active":"2024-03-29T11:31:40.899158+0000","last_became_peered":"2024-03-29T11:31:40.899158+0000","last_unstale":"2024-03-29T11:31:41.951120+0000","last_undegraded":"2024-03-29T11:31:41.951120+0000","last_fullsized":"2024-03-29T11:31:41.951120+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:00:19.784413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882738+0000","last_change":"2024-03-29T11:31:26.322425+0000","last_active":"2024-03-29T11:31:40.882738+0000","last_peered":"2024-03-29T11:31:40.882738+0000","last_clean":"2024-03-29T11:31:40.882738+0000","last_became_active":"2024-03-29T11:31:26.321981+0000","last_became_peered":"2024-03-29T11:31:26.321981+0000","last_unstale":"2024-03-29T11:31:40.882738+0000","last_undegraded":"2024-03-29T11:31:40.882738+0000","last_fullsized":"2024-03-29T11:31:40.882738+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:36.434947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876100+0000","last_change":"2024-03-29T11:31:26.838532+0000","last_active":"2024-03-29T11:31:40.876100+0000","last_peered":"2024-03-29T11:31:40.876100+0000","last_clean":"2024-03-29T11:31:40.876100+0000","last_became_active":"2024-03-29T11:31:26.838267+0000","last_became_peered":"2024-03-29T11:31:26.838267+0000","last_unstale":"2024-03-29T11:31:40.876100+0000","last_undegraded":"2024-03-29T11:31:40.876100+0000","last_fullsized":"2024-03-29T11:31:40.876100+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:17:41.239491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.881113+0000","last_change":"2024-03-29T11:31:40.369144+0000","last_active":"2024-03-29T11:31:41.881113+0000","last_peered":"2024-03-29T11:31:41.881113+0000","last_clean":"2024-03-29T11:31:41.881113+0000","last_became_active":"2024-03-29T11:31:40.368341+0000","last_became_peered":"2024-03-29T11:31:40.368341+0000","last_unstale":"2024-03-29T11:31:41.881113+0000","last_undegraded":"2024-03-29T11:31:41.881113+0000","last_fullsized":"2024-03-29T11:31:41.881113+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:41.276410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950821+0000","last_change":"2024-03-29T11:31:40.903386+0000","last_active":"2024-03-29T11:31:41.950821+0000","last_peered":"2024-03-29T11:31:41.950821+0000","last_clean":"2024-03-29T11:31:41.950821+0000","last_became_active":"2024-03-29T11:31:40.901220+0000","last_became_peered":"2024-03-29T11:31:40.901220+0000","last_unstale":"2024-03-29T11:31:41.950821+0000","last_undegraded":"2024-03-29T11:31:41.950821+0000","last_fullsized":"2024-03-29T11:31:41.950821+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:22.511071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872900+0000","last_change":"2024-03-29T11:31:26.320428+0000","last_active":"2024-03-29T11:31:40.872900+0000","last_peered":"2024-03-29T11:31:40.872900+0000","last_clean":"2024-03-29T11:31:40.872900+0000","last_became_active":"2024-03-29T11:31:26.320099+0000","last_became_peered":"2024-03-29T11:31:26.320099+0000","last_unstale":"2024-03-29T11:31:40.872900+0000","last_undegraded":"2024-03-29T11:31:40.872900+0000","last_fullsized":"2024-03-29T11:31:40.872900+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:56.873408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873231+0000","last_change":"2024-03-29T11:31:26.848609+0000","last_active":"2024-03-29T11:31:40.873231+0000","last_peered":"2024-03-29T11:31:40.873231+0000","last_clean":"2024-03-29T11:31:40.873231+0000","last_became_active":"2024-03-29T11:31:26.848202+0000","last_became_peered":"2024-03-29T11:31:26.848202+0000","last_unstale":"2024-03-29T11:31:40.873231+0000","last_undegraded":"2024-03-29T11:31:40.873231+0000","last_fullsized":"2024-03-29T11:31:40.873231+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:28.240696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879309+0000","last_change":"2024-03-29T11:31:40.367052+0000","last_active":"2024-03-29T11:31:41.879309+0000","last_peered":"2024-03-29T11:31:41.879309+0000","last_clean":"2024-03-29T11:31:41.879309+0000","last_became_active":"2024-03-29T11:31:40.365737+0000","last_became_peered":"2024-03-29T11:31:40.365737+0000","last_unstale":"2024-03-29T11:31:41.879309+0000","last_undegraded":"2024-03-29T11:31:41.879309+0000","last_fullsized":"2024-03-29T11:31:41.879309+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:48:45.870078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880460+0000","last_change":"2024-03-29T11:31:40.889850+0000","last_active":"2024-03-29T11:31:41.880460+0000","last_peered":"2024-03-29T11:31:41.880460+0000","last_clean":"2024-03-29T11:31:41.880460+0000","last_became_active":"2024-03-29T11:31:40.889554+0000","last_became_peered":"2024-03-29T11:31:40.889554+0000","last_unstale":"2024-03-29T11:31:41.880460+0000","last_undegraded":"2024-03-29T11:31:41.880460+0000","last_fullsized":"2024-03-29T11:31:41.880460+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:09:49.649528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883808+0000","last_change":"2024-03-29T11:31:26.311780+0000","last_active":"2024-03-29T11:31:40.883808+0000","last_peered":"2024-03-29T11:31:40.883808+0000","last_clean":"2024-03-29T11:31:40.883808+0000","last_became_active":"2024-03-29T11:31:26.311128+0000","last_became_peered":"2024-03-29T11:31:26.311128+0000","last_unstale":"2024-03-29T11:31:40.883808+0000","last_undegraded":"2024-03-29T11:31:40.883808+0000","last_fullsized":"2024-03-29T11:31:40.883808+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:46.870101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884996+0000","last_change":"2024-03-29T11:31:26.844018+0000","last_active":"2024-03-29T11:31:40.884996+0000","last_peered":"2024-03-29T11:31:40.884996+0000","last_clean":"2024-03-29T11:31:40.884996+0000","last_became_active":"2024-03-29T11:31:26.843237+0000","last_became_peered":"2024-03-29T11:31:26.843237+0000","last_unstale":"2024-03-29T11:31:40.884996+0000","last_undegraded":"2024-03-29T11:31:40.884996+0000","last_fullsized":"2024-03-29T11:31:40.884996+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:12:52.041908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949932+0000","last_change":"2024-03-29T11:31:40.364392+0000","last_active":"2024-03-29T11:31:41.949932+0000","last_peered":"2024-03-29T11:31:41.949932+0000","last_clean":"2024-03-29T11:31:41.949932+0000","last_became_active":"2024-03-29T11:31:40.364108+0000","last_became_peered":"2024-03-29T11:31:40.364108+0000","last_unstale":"2024-03-29T11:31:41.949932+0000","last_undegraded":"2024-03-29T11:31:41.949932+0000","last_fullsized":"2024-03-29T11:31:41.949932+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:54.274308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880566+0000","last_change":"2024-03-29T11:31:40.889918+0000","last_active":"2024-03-29T11:31:41.880566+0000","last_peered":"2024-03-29T11:31:41.880566+0000","last_clean":"2024-03-29T11:31:41.880566+0000","last_became_active":"2024-03-29T11:31:40.889132+0000","last_became_peered":"2024-03-29T11:31:40.889132+0000","last_unstale":"2024-03-29T11:31:41.880566+0000","last_undegraded":"2024-03-29T11:31:41.880566+0000","last_fullsized":"2024-03-29T11:31:41.880566+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:44:04.775529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.892399+0000","last_change":"2024-03-29T11:31:26.317291+0000","last_active":"2024-03-29T11:31:40.892399+0000","last_peered":"2024-03-29T11:31:40.892399+0000","last_clean":"2024-03-29T11:31:40.892399+0000","last_became_active":"2024-03-29T11:31:26.317003+0000","last_became_peered":"2024-03-29T11:31:26.317003+0000","last_unstale":"2024-03-29T11:31:40.892399+0000","last_undegraded":"2024-03-29T11:31:40.892399+0000","last_fullsized":"2024-03-29T11:31:40.892399+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:55:21.607497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.887195+0000","last_change":"2024-03-29T11:31:26.849336+0000","last_active":"2024-03-29T11:31:40.887195+0000","last_peered":"2024-03-29T11:31:40.887195+0000","last_clean":"2024-03-29T11:31:40.887195+0000","last_became_active":"2024-03-29T11:31:26.849042+0000","last_became_peered":"2024-03-29T11:31:26.849042+0000","last_unstale":"2024-03-29T11:31:40.887195+0000","last_undegraded":"2024-03-29T11:31:40.887195+0000","last_fullsized":"2024-03-29T11:31:40.887195+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:58.653895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949597+0000","last_change":"2024-03-29T11:31:40.367967+0000","last_active":"2024-03-29T11:31:41.949597+0000","last_peered":"2024-03-29T11:31:41.949597+0000","last_clean":"2024-03-29T11:31:41.949597+0000","last_became_active":"2024-03-29T11:31:40.367700+0000","last_became_peered":"2024-03-29T11:31:40.367700+0000","last_unstale":"2024-03-29T11:31:41.949597+0000","last_undegraded":"2024-03-29T11:31:41.949597+0000","last_fullsized":"2024-03-29T11:31:41.949597+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:12.867102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950152+0000","last_change":"2024-03-29T11:31:40.894878+0000","last_active":"2024-03-29T11:31:41.950152+0000","last_peered":"2024-03-29T11:31:41.950152+0000","last_clean":"2024-03-29T11:31:41.950152+0000","last_became_active":"2024-03-29T11:31:40.894613+0000","last_became_peered":"2024-03-29T11:31:40.894613+0000","last_unstale":"2024-03-29T11:31:41.950152+0000","last_undegraded":"2024-03-29T11:31:41.950152+0000","last_fullsized":"2024-03-29T11:31:41.950152+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:41.165586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.6","version":"227'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.877289+0000","last_change":"2024-03-29T11:31:26.316463+0000","last_active":"2024-03-29T11:31:40.877289+0000","last_peered":"2024-03-29T11:31:40.877289+0000","last_clean":"2024-03-29T11:31:40.877289+0000","last_became_active":"2024-03-29T11:31:26.315640+0000","last_became_peered":"2024-03-29T11:31:26.315640+0000","last_unstale":"2024-03-29T11:31:40.877289+0000","last_undegraded":"2024-03-29T11:31:40.877289+0000","last_fullsized":"2024-03-29T11:31:40.877289+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:31.994471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872913+0000","last_change":"2024-03-29T11:31:26.845919+0000","last_active":"2024-03-29T11:31:40.872913+0000","last_peered":"2024-03-29T11:31:40.872913+0000","last_clean":"2024-03-29T11:31:40.872913+0000","last_became_active":"2024-03-29T11:31:26.845630+0000","last_became_peered":"2024-03-29T11:31:26.845630+0000","last_unstale":"2024-03-29T11:31:40.872913+0000","last_undegraded":"2024-03-29T11:31:40.872913+0000","last_fullsized":"2024-03-29T11:31:40.872913+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:47.195933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879707+0000","last_change":"2024-03-29T11:31:40.366767+0000","last_active":"2024-03-29T11:31:41.879707+0000","last_peered":"2024-03-29T11:31:41.879707+0000","last_clean":"2024-03-29T11:31:41.879707+0000","last_became_active":"2024-03-29T11:31:40.366265+0000","last_became_peered":"2024-03-29T11:31:40.366265+0000","last_unstale":"2024-03-29T11:31:41.879707+0000","last_undegraded":"2024-03-29T11:31:41.879707+0000","last_fullsized":"2024-03-29T11:31:41.879707+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:17:10.911769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.878471+0000","last_change":"2024-03-29T11:31:40.889807+0000","last_active":"2024-03-29T11:31:41.878471+0000","last_peered":"2024-03-29T11:31:41.878471+0000","last_clean":"2024-03-29T11:31:41.878471+0000","last_became_active":"2024-03-29T11:31:40.889492+0000","last_became_peered":"2024-03-29T11:31:40.889492+0000","last_unstale":"2024-03-29T11:31:41.878471+0000","last_undegraded":"2024-03-29T11:31:41.878471+0000","last_fullsized":"2024-03-29T11:31:41.878471+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:21:21.715338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.7","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885315+0000","last_change":"2024-03-29T11:31:26.310244+0000","last_active":"2024-03-29T11:31:40.885315+0000","last_peered":"2024-03-29T11:31:40.885315+0000","last_clean":"2024-03-29T11:31:40.885315+0000","last_became_active":"2024-03-29T11:31:26.309981+0000","last_became_peered":"2024-03-29T11:31:26.309981+0000","last_unstale":"2024-03-29T11:31:40.885315+0000","last_undegraded":"2024-03-29T11:31:40.885315+0000","last_fullsized":"2024-03-29T11:31:40.885315+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:16.350729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.879270+0000","last_change":"2024-03-29T11:31:26.851615+0000","last_active":"2024-03-29T11:31:40.879270+0000","last_peered":"2024-03-29T11:31:40.879270+0000","last_clean":"2024-03-29T11:31:40.879270+0000","last_became_active":"2024-03-29T11:31:26.851328+0000","last_became_peered":"2024-03-29T11:31:26.851328+0000","last_unstale":"2024-03-29T11:31:40.879270+0000","last_undegraded":"2024-03-29T11:31:40.879270+0000","last_fullsized":"2024-03-29T11:31:40.879270+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:57.906301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949694+0000","last_change":"2024-03-29T11:31:40.367420+0000","last_active":"2024-03-29T11:31:41.949694+0000","last_peered":"2024-03-29T11:31:41.949694+0000","last_clean":"2024-03-29T11:31:41.949694+0000","last_became_active":"2024-03-29T11:31:40.367079+0000","last_became_peered":"2024-03-29T11:31:40.367079+0000","last_unstale":"2024-03-29T11:31:41.949694+0000","last_undegraded":"2024-03-29T11:31:41.949694+0000","last_fullsized":"2024-03-29T11:31:41.949694+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:36:00.231647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879030+0000","last_change":"2024-03-29T11:31:40.905299+0000","last_active":"2024-03-29T11:31:41.879030+0000","last_peered":"2024-03-29T11:31:41.879030+0000","last_clean":"2024-03-29T11:31:41.879030+0000","last_became_active":"2024-03-29T11:31:40.904494+0000","last_became_peered":"2024-03-29T11:31:40.904494+0000","last_unstale":"2024-03-29T11:31:41.879030+0000","last_undegraded":"2024-03-29T11:31:41.879030+0000","last_fullsized":"2024-03-29T11:31:41.879030+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:30.611972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884063+0000","last_change":"2024-03-29T11:31:26.321619+0000","last_active":"2024-03-29T11:31:40.884063+0000","last_peered":"2024-03-29T11:31:40.884063+0000","last_clean":"2024-03-29T11:31:40.884063+0000","last_became_active":"2024-03-29T11:31:26.321349+0000","last_became_peered":"2024-03-29T11:31:26.321349+0000","last_unstale":"2024-03-29T11:31:40.884063+0000","last_undegraded":"2024-03-29T11:31:40.884063+0000","last_fullsized":"2024-03-29T11:31:40.884063+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:09.446187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885595+0000","last_change":"2024-03-29T11:31:26.843420+0000","last_active":"2024-03-29T11:31:40.885595+0000","last_peered":"2024-03-29T11:31:40.885595+0000","last_clean":"2024-03-29T11:31:40.885595+0000","last_became_active":"2024-03-29T11:31:26.842734+0000","last_became_peered":"2024-03-29T11:31:26.842734+0000","last_unstale":"2024-03-29T11:31:40.885595+0000","last_undegraded":"2024-03-29T11:31:40.885595+0000","last_fullsized":"2024-03-29T11:31:40.885595+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:46.006140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950604+0000","last_change":"2024-03-29T11:31:40.371935+0000","last_active":"2024-03-29T11:31:41.950604+0000","last_peered":"2024-03-29T11:31:41.950604+0000","last_clean":"2024-03-29T11:31:41.950604+0000","last_became_active":"2024-03-29T11:31:40.371172+0000","last_became_peered":"2024-03-29T11:31:40.371172+0000","last_unstale":"2024-03-29T11:31:41.950604+0000","last_undegraded":"2024-03-29T11:31:41.950604+0000","last_fullsized":"2024-03-29T11:31:41.950604+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:10:45.255586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874890+0000","last_change":"2024-03-29T11:31:40.895452+0000","last_active":"2024-03-29T11:31:41.874890+0000","last_peered":"2024-03-29T11:31:41.874890+0000","last_clean":"2024-03-29T11:31:41.874890+0000","last_became_active":"2024-03-29T11:31:40.895193+0000","last_became_peered":"2024-03-29T11:31:40.895193+0000","last_unstale":"2024-03-29T11:31:41.874890+0000","last_undegraded":"2024-03-29T11:31:41.874890+0000","last_fullsized":"2024-03-29T11:31:41.874890+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:50:48.434922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870436+0000","last_change":"2024-03-29T11:31:26.312962+0000","last_active":"2024-03-29T11:31:40.870436+0000","last_peered":"2024-03-29T11:31:40.870436+0000","last_clean":"2024-03-29T11:31:40.870436+0000","last_became_active":"2024-03-29T11:31:26.312668+0000","last_became_peered":"2024-03-29T11:31:26.312668+0000","last_unstale":"2024-03-29T11:31:40.870436+0000","last_undegraded":"2024-03-29T11:31:40.870436+0000","last_fullsized":"2024-03-29T11:31:40.870436+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:46.870101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.881073+0000","last_change":"2024-03-29T11:31:26.835978+0000","last_active":"2024-03-29T11:31:40.881073+0000","last_peered":"2024-03-29T11:31:40.881073+0000","last_clean":"2024-03-29T11:31:40.881073+0000","last_became_active":"2024-03-29T11:31:26.835400+0000","last_became_peered":"2024-03-29T11:31:26.835400+0000","last_unstale":"2024-03-29T11:31:40.881073+0000","last_undegraded":"2024-03-29T11:31:40.881073+0000","last_fullsized":"2024-03-29T11:31:40.881073+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:31.783000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.881318+0000","last_change":"2024-03-29T11:31:40.368134+0000","last_active":"2024-03-29T11:31:41.881318+0000","last_peered":"2024-03-29T11:31:41.881318+0000","last_clean":"2024-03-29T11:31:41.881318+0000","last_became_active":"2024-03-29T11:31:40.367120+0000","last_became_peered":"2024-03-29T11:31:40.367120+0000","last_unstale":"2024-03-29T11:31:41.881318+0000","last_undegraded":"2024-03-29T11:31:41.881318+0000","last_fullsized":"2024-03-29T11:31:41.881318+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:55:58.832455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951430+0000","last_change":"2024-03-29T11:31:40.886320+0000","last_active":"2024-03-29T11:31:41.951430+0000","last_peered":"2024-03-29T11:31:41.951430+0000","last_clean":"2024-03-29T11:31:41.951430+0000","last_became_active":"2024-03-29T11:31:40.885837+0000","last_became_peered":"2024-03-29T11:31:40.885837+0000","last_unstale":"2024-03-29T11:31:41.951430+0000","last_undegraded":"2024-03-29T11:31:41.951430+0000","last_fullsized":"2024-03-29T11:31:41.951430+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:10:46.262823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873091+0000","last_change":"2024-03-29T11:31:26.312485+0000","last_active":"2024-03-29T11:31:40.873091+0000","last_peered":"2024-03-29T11:31:40.873091+0000","last_clean":"2024-03-29T11:31:40.873091+0000","last_became_active":"2024-03-29T11:31:26.311989+0000","last_became_peered":"2024-03-29T11:31:26.311989+0000","last_unstale":"2024-03-29T11:31:40.873091+0000","last_undegraded":"2024-03-29T11:31:40.873091+0000","last_fullsized":"2024-03-29T11:31:40.873091+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:42:56.190784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883080+0000","last_change":"2024-03-29T11:31:26.836881+0000","last_active":"2024-03-29T11:31:40.883080+0000","last_peered":"2024-03-29T11:31:40.883080+0000","last_clean":"2024-03-29T11:31:40.883080+0000","last_became_active":"2024-03-29T11:31:26.836613+0000","last_became_peered":"2024-03-29T11:31:26.836613+0000","last_unstale":"2024-03-29T11:31:40.883080+0000","last_undegraded":"2024-03-29T11:31:40.883080+0000","last_fullsized":"2024-03-29T11:31:40.883080+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:23.565406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874497+0000","last_change":"2024-03-29T11:31:40.357898+0000","last_active":"2024-03-29T11:31:41.874497+0000","last_peered":"2024-03-29T11:31:41.874497+0000","last_clean":"2024-03-29T11:31:41.874497+0000","last_became_active":"2024-03-29T11:31:40.357355+0000","last_became_peered":"2024-03-29T11:31:40.357355+0000","last_unstale":"2024-03-29T11:31:41.874497+0000","last_undegraded":"2024-03-29T11:31:41.874497+0000","last_fullsized":"2024-03-29T11:31:41.874497+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:54.052209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880921+0000","last_change":"2024-03-29T11:31:40.897342+0000","last_active":"2024-03-29T11:31:41.880921+0000","last_peered":"2024-03-29T11:31:41.880921+0000","last_clean":"2024-03-29T11:31:41.880921+0000","last_became_active":"2024-03-29T11:31:40.896970+0000","last_became_peered":"2024-03-29T11:31:40.896970+0000","last_unstale":"2024-03-29T11:31:41.880921+0000","last_undegraded":"2024-03-29T11:31:41.880921+0000","last_fullsized":"2024-03-29T11:31:41.880921+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:32:36.867108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875822+0000","last_change":"2024-03-29T11:31:26.316435+0000","last_active":"2024-03-29T11:31:40.875822+0000","last_peered":"2024-03-29T11:31:40.875822+0000","last_clean":"2024-03-29T11:31:40.875822+0000","last_became_active":"2024-03-29T11:31:26.316124+0000","last_became_peered":"2024-03-29T11:31:26.316124+0000","last_unstale":"2024-03-29T11:31:40.875822+0000","last_undegraded":"2024-03-29T11:31:40.875822+0000","last_fullsized":"2024-03-29T11:31:40.875822+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:15:10.280553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875924+0000","last_change":"2024-03-29T11:31:26.844232+0000","last_active":"2024-03-29T11:31:40.875924+0000","last_peered":"2024-03-29T11:31:40.875924+0000","last_clean":"2024-03-29T11:31:40.875924+0000","last_became_active":"2024-03-29T11:31:26.843977+0000","last_became_peered":"2024-03-29T11:31:26.843977+0000","last_unstale":"2024-03-29T11:31:40.875924+0000","last_undegraded":"2024-03-29T11:31:40.875924+0000","last_fullsized":"2024-03-29T11:31:40.875924+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879088+0000","last_change":"2024-03-29T11:31:40.364785+0000","last_active":"2024-03-29T11:31:41.879088+0000","last_peered":"2024-03-29T11:31:41.879088+0000","last_clean":"2024-03-29T11:31:41.879088+0000","last_became_active":"2024-03-29T11:31:40.360188+0000","last_became_peered":"2024-03-29T11:31:40.360188+0000","last_unstale":"2024-03-29T11:31:41.879088+0000","last_undegraded":"2024-03-29T11:31:41.879088+0000","last_fullsized":"2024-03-29T11:31:41.879088+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:54:08.167891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880604+0000","last_change":"2024-03-29T11:31:40.901353+0000","last_active":"2024-03-29T11:31:41.880604+0000","last_peered":"2024-03-29T11:31:41.880604+0000","last_clean":"2024-03-29T11:31:41.880604+0000","last_became_active":"2024-03-29T11:31:40.901082+0000","last_became_peered":"2024-03-29T11:31:40.901082+0000","last_unstale":"2024-03-29T11:31:41.880604+0000","last_undegraded":"2024-03-29T11:31:41.880604+0000","last_fullsized":"2024-03-29T11:31:41.880604+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:00:19.784413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883643+0000","last_change":"2024-03-29T11:31:26.313158+0000","last_active":"2024-03-29T11:31:40.883643+0000","last_peered":"2024-03-29T11:31:40.883643+0000","last_clean":"2024-03-29T11:31:40.883643+0000","last_became_active":"2024-03-29T11:31:26.311492+0000","last_became_peered":"2024-03-29T11:31:26.311492+0000","last_unstale":"2024-03-29T11:31:40.883643+0000","last_undegraded":"2024-03-29T11:31:40.883643+0000","last_fullsized":"2024-03-29T11:31:40.883643+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:40.347554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876039+0000","last_change":"2024-03-29T11:31:26.848108+0000","last_active":"2024-03-29T11:31:40.876039+0000","last_peered":"2024-03-29T11:31:40.876039+0000","last_clean":"2024-03-29T11:31:40.876039+0000","last_became_active":"2024-03-29T11:31:26.847705+0000","last_became_peered":"2024-03-29T11:31:26.847705+0000","last_unstale":"2024-03-29T11:31:40.876039+0000","last_undegraded":"2024-03-29T11:31:40.876039+0000","last_fullsized":"2024-03-29T11:31:40.876039+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:53:43.020271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949828+0000","last_change":"2024-03-29T11:31:40.363049+0000","last_active":"2024-03-29T11:31:41.949828+0000","last_peered":"2024-03-29T11:31:41.949828+0000","last_clean":"2024-03-29T11:31:41.949828+0000","last_became_active":"2024-03-29T11:31:40.361460+0000","last_became_peered":"2024-03-29T11:31:40.361460+0000","last_unstale":"2024-03-29T11:31:41.949828+0000","last_undegraded":"2024-03-29T11:31:41.949828+0000","last_fullsized":"2024-03-29T11:31:41.949828+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:37.266073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880725+0000","last_change":"2024-03-29T11:31:40.882355+0000","last_active":"2024-03-29T11:31:41.880725+0000","last_peered":"2024-03-29T11:31:41.880725+0000","last_clean":"2024-03-29T11:31:41.880725+0000","last_became_active":"2024-03-29T11:31:40.882036+0000","last_became_peered":"2024-03-29T11:31:40.882036+0000","last_unstale":"2024-03-29T11:31:41.880725+0000","last_undegraded":"2024-03-29T11:31:41.880725+0000","last_fullsized":"2024-03-29T11:31:41.880725+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:26:19.372364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.d","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876933+0000","last_change":"2024-03-29T11:31:26.319951+0000","last_active":"2024-03-29T11:31:40.876933+0000","last_peered":"2024-03-29T11:31:40.876933+0000","last_clean":"2024-03-29T11:31:40.876933+0000","last_became_active":"2024-03-29T11:31:26.318226+0000","last_became_peered":"2024-03-29T11:31:26.318226+0000","last_unstale":"2024-03-29T11:31:40.876933+0000","last_undegraded":"2024-03-29T11:31:40.876933+0000","last_fullsized":"2024-03-29T11:31:40.876933+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:22.562759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883723+0000","last_change":"2024-03-29T11:31:26.833306+0000","last_active":"2024-03-29T11:31:40.883723+0000","last_peered":"2024-03-29T11:31:40.883723+0000","last_clean":"2024-03-29T11:31:40.883723+0000","last_became_active":"2024-03-29T11:31:26.833041+0000","last_became_peered":"2024-03-29T11:31:26.833041+0000","last_unstale":"2024-03-29T11:31:40.883723+0000","last_undegraded":"2024-03-29T11:31:40.883723+0000","last_fullsized":"2024-03-29T11:31:40.883723+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:15:11.283200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.881180+0000","last_change":"2024-03-29T11:31:40.365002+0000","last_active":"2024-03-29T11:31:41.881180+0000","last_peered":"2024-03-29T11:31:41.881180+0000","last_clean":"2024-03-29T11:31:41.881180+0000","last_became_active":"2024-03-29T11:31:40.364715+0000","last_became_peered":"2024-03-29T11:31:40.364715+0000","last_unstale":"2024-03-29T11:31:41.881180+0000","last_undegraded":"2024-03-29T11:31:41.881180+0000","last_fullsized":"2024-03-29T11:31:41.881180+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:47:07.143595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950062+0000","last_change":"2024-03-29T11:31:40.896742+0000","last_active":"2024-03-29T11:31:41.950062+0000","last_peered":"2024-03-29T11:31:41.950062+0000","last_clean":"2024-03-29T11:31:41.950062+0000","last_became_active":"2024-03-29T11:31:40.896478+0000","last_became_peered":"2024-03-29T11:31:40.896478+0000","last_unstale":"2024-03-29T11:31:41.950062+0000","last_undegraded":"2024-03-29T11:31:41.950062+0000","last_fullsized":"2024-03-29T11:31:41.950062+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:23.215173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884317+0000","last_change":"2024-03-29T11:31:26.317437+0000","last_active":"2024-03-29T11:31:40.884317+0000","last_peered":"2024-03-29T11:31:40.884317+0000","last_clean":"2024-03-29T11:31:40.884317+0000","last_became_active":"2024-03-29T11:31:26.317128+0000","last_became_peered":"2024-03-29T11:31:26.317128+0000","last_unstale":"2024-03-29T11:31:40.884317+0000","last_undegraded":"2024-03-29T11:31:40.884317+0000","last_fullsized":"2024-03-29T11:31:40.884317+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:57.806178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.889312+0000","last_change":"2024-03-29T11:31:26.833432+0000","last_active":"2024-03-29T11:31:40.889312+0000","last_peered":"2024-03-29T11:31:40.889312+0000","last_clean":"2024-03-29T11:31:40.889312+0000","last_became_active":"2024-03-29T11:31:26.833078+0000","last_became_peered":"2024-03-29T11:31:26.833078+0000","last_unstale":"2024-03-29T11:31:40.889312+0000","last_undegraded":"2024-03-29T11:31:40.889312+0000","last_fullsized":"2024-03-29T11:31:40.889312+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:17.353377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879603+0000","last_change":"2024-03-29T11:31:40.366329+0000","last_active":"2024-03-29T11:31:41.879603+0000","last_peered":"2024-03-29T11:31:41.879603+0000","last_clean":"2024-03-29T11:31:41.879603+0000","last_became_active":"2024-03-29T11:31:40.365971+0000","last_became_peered":"2024-03-29T11:31:40.365971+0000","last_unstale":"2024-03-29T11:31:41.879603+0000","last_undegraded":"2024-03-29T11:31:41.879603+0000","last_fullsized":"2024-03-29T11:31:41.879603+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:03.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.948828+0000","last_change":"2024-03-29T11:31:40.896076+0000","last_active":"2024-03-29T11:31:41.948828+0000","last_peered":"2024-03-29T11:31:41.948828+0000","last_clean":"2024-03-29T11:31:41.948828+0000","last_became_active":"2024-03-29T11:31:40.895757+0000","last_became_peered":"2024-03-29T11:31:40.895757+0000","last_unstale":"2024-03-29T11:31:41.948828+0000","last_undegraded":"2024-03-29T11:31:41.948828+0000","last_fullsized":"2024-03-29T11:31:41.948828+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:28:36.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.f","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.893768+0000","last_change":"2024-03-29T11:31:26.320455+0000","last_active":"2024-03-29T11:31:40.893768+0000","last_peered":"2024-03-29T11:31:40.893768+0000","last_clean":"2024-03-29T11:31:40.893768+0000","last_became_active":"2024-03-29T11:31:26.320175+0000","last_became_peered":"2024-03-29T11:31:26.320175+0000","last_unstale":"2024-03-29T11:31:40.893768+0000","last_undegraded":"2024-03-29T11:31:40.893768+0000","last_fullsized":"2024-03-29T11:31:40.893768+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:25:12.671618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884065+0000","last_change":"2024-03-29T11:31:26.854468+0000","last_active":"2024-03-29T11:31:40.884065+0000","last_peered":"2024-03-29T11:31:40.884065+0000","last_clean":"2024-03-29T11:31:40.884065+0000","last_became_active":"2024-03-29T11:31:26.851644+0000","last_became_peered":"2024-03-29T11:31:26.851644+0000","last_unstale":"2024-03-29T11:31:40.884065+0000","last_undegraded":"2024-03-29T11:31:40.884065+0000","last_fullsized":"2024-03-29T11:31:40.884065+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:16.143243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949199+0000","last_change":"2024-03-29T11:31:40.366948+0000","last_active":"2024-03-29T11:31:41.949199+0000","last_peered":"2024-03-29T11:31:41.949199+0000","last_clean":"2024-03-29T11:31:41.949199+0000","last_became_active":"2024-03-29T11:31:40.366626+0000","last_became_peered":"2024-03-29T11:31:40.366626+0000","last_unstale":"2024-03-29T11:31:41.949199+0000","last_undegraded":"2024-03-29T11:31:41.949199+0000","last_fullsized":"2024-03-29T11:31:41.949199+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:40.158349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949088+0000","last_change":"2024-03-29T11:31:40.888358+0000","last_active":"2024-03-29T11:31:41.949088+0000","last_peered":"2024-03-29T11:31:41.949088+0000","last_clean":"2024-03-29T11:31:41.949088+0000","last_became_active":"2024-03-29T11:31:40.888044+0000","last_became_peered":"2024-03-29T11:31:40.888044+0000","last_unstale":"2024-03-29T11:31:41.949088+0000","last_undegraded":"2024-03-29T11:31:41.949088+0000","last_fullsized":"2024-03-29T11:31:41.949088+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:11:35.554749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.10","version":"226'8","reported_seq":23,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882877+0000","last_change":"2024-03-29T11:31:26.320392+0000","last_active":"2024-03-29T11:31:40.882877+0000","last_peered":"2024-03-29T11:31:40.882877+0000","last_clean":"2024-03-29T11:31:40.882877+0000","last_became_active":"2024-03-29T11:31:26.319139+0000","last_became_peered":"2024-03-29T11:31:26.319139+0000","last_unstale":"2024-03-29T11:31:40.882877+0000","last_undegraded":"2024-03-29T11:31:40.882877+0000","last_fullsized":"2024-03-29T11:31:40.882877+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:57.651247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870353+0000","last_change":"2024-03-29T11:31:26.843461+0000","last_active":"2024-03-29T11:31:40.870353+0000","last_peered":"2024-03-29T11:31:40.870353+0000","last_clean":"2024-03-29T11:31:40.870353+0000","last_became_active":"2024-03-29T11:31:26.839816+0000","last_became_peered":"2024-03-29T11:31:26.839816+0000","last_unstale":"2024-03-29T11:31:40.870353+0000","last_undegraded":"2024-03-29T11:31:40.870353+0000","last_fullsized":"2024-03-29T11:31:40.870353+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:32.997118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950691+0000","last_change":"2024-03-29T11:31:40.372288+0000","last_active":"2024-03-29T11:31:41.950691+0000","last_peered":"2024-03-29T11:31:41.950691+0000","last_clean":"2024-03-29T11:31:41.950691+0000","last_became_active":"2024-03-29T11:31:40.370805+0000","last_became_peered":"2024-03-29T11:31:40.370805+0000","last_unstale":"2024-03-29T11:31:41.950691+0000","last_undegraded":"2024-03-29T11:31:41.950691+0000","last_fullsized":"2024-03-29T11:31:41.950691+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:54.052209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880516+0000","last_change":"2024-03-29T11:31:40.896839+0000","last_active":"2024-03-29T11:31:41.880516+0000","last_peered":"2024-03-29T11:31:41.880516+0000","last_clean":"2024-03-29T11:31:41.880516+0000","last_became_active":"2024-03-29T11:31:40.896543+0000","last_became_peered":"2024-03-29T11:31:40.896543+0000","last_unstale":"2024-03-29T11:31:41.880516+0000","last_undegraded":"2024-03-29T11:31:41.880516+0000","last_fullsized":"2024-03-29T11:31:41.880516+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:21.739901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875590+0000","last_change":"2024-03-29T11:31:26.314471+0000","last_active":"2024-03-29T11:31:40.875590+0000","last_peered":"2024-03-29T11:31:40.875590+0000","last_clean":"2024-03-29T11:31:40.875590+0000","last_became_active":"2024-03-29T11:31:26.314204+0000","last_became_peered":"2024-03-29T11:31:26.314204+0000","last_unstale":"2024-03-29T11:31:40.875590+0000","last_undegraded":"2024-03-29T11:31:40.875590+0000","last_fullsized":"2024-03-29T11:31:40.875590+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:49:13.676544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882678+0000","last_change":"2024-03-29T11:31:26.846893+0000","last_active":"2024-03-29T11:31:40.882678+0000","last_peered":"2024-03-29T11:31:40.882678+0000","last_clean":"2024-03-29T11:31:40.882678+0000","last_became_active":"2024-03-29T11:31:26.845341+0000","last_became_peered":"2024-03-29T11:31:26.845341+0000","last_unstale":"2024-03-29T11:31:40.882678+0000","last_undegraded":"2024-03-29T11:31:40.882678+0000","last_fullsized":"2024-03-29T11:31:40.882678+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:28.240696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880418+0000","last_change":"2024-03-29T11:31:40.367705+0000","last_active":"2024-03-29T11:31:41.880418+0000","last_peered":"2024-03-29T11:31:41.880418+0000","last_clean":"2024-03-29T11:31:41.880418+0000","last_became_active":"2024-03-29T11:31:40.367405+0000","last_became_peered":"2024-03-29T11:31:40.367405+0000","last_unstale":"2024-03-29T11:31:41.880418+0000","last_undegraded":"2024-03-29T11:31:41.880418+0000","last_fullsized":"2024-03-29T11:31:41.880418+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:54.052209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951245+0000","last_change":"2024-03-29T11:31:40.901915+0000","last_active":"2024-03-29T11:31:41.951245+0000","last_peered":"2024-03-29T11:31:41.951245+0000","last_clean":"2024-03-29T11:31:41.951245+0000","last_became_active":"2024-03-29T11:31:40.900676+0000","last_became_peered":"2024-03-29T11:31:40.900676+0000","last_unstale":"2024-03-29T11:31:41.951245+0000","last_undegraded":"2024-03-29T11:31:41.951245+0000","last_fullsized":"2024-03-29T11:31:41.951245+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:16:12.619950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885741+0000","last_change":"2024-03-29T11:31:26.315775+0000","last_active":"2024-03-29T11:31:40.885741+0000","last_peered":"2024-03-29T11:31:40.885741+0000","last_clean":"2024-03-29T11:31:40.885741+0000","last_became_active":"2024-03-29T11:31:26.315493+0000","last_became_peered":"2024-03-29T11:31:26.315493+0000","last_unstale":"2024-03-29T11:31:40.885741+0000","last_undegraded":"2024-03-29T11:31:40.885741+0000","last_fullsized":"2024-03-29T11:31:40.885741+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:32:47.003355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876143+0000","last_change":"2024-03-29T11:31:26.833527+0000","last_active":"2024-03-29T11:31:40.876143+0000","last_peered":"2024-03-29T11:31:40.876143+0000","last_clean":"2024-03-29T11:31:40.876143+0000","last_became_active":"2024-03-29T11:31:26.833280+0000","last_became_peered":"2024-03-29T11:31:26.833280+0000","last_unstale":"2024-03-29T11:31:40.876143+0000","last_undegraded":"2024-03-29T11:31:40.876143+0000","last_fullsized":"2024-03-29T11:31:40.876143+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:35:08.770095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"230'6","reported_seq":17,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874583+0000","last_change":"2024-03-29T11:31:40.367200+0000","last_active":"2024-03-29T11:31:41.874583+0000","last_peered":"2024-03-29T11:31:41.874583+0000","last_clean":"2024-03-29T11:31:41.874583+0000","last_became_active":"2024-03-29T11:31:40.366932+0000","last_became_peered":"2024-03-29T11:31:40.366932+0000","last_unstale":"2024-03-29T11:31:41.874583+0000","last_undegraded":"2024-03-29T11:31:41.874583+0000","last_fullsized":"2024-03-29T11:31:41.874583+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:12:33.891267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879198+0000","last_change":"2024-03-29T11:31:40.889854+0000","last_active":"2024-03-29T11:31:41.879198+0000","last_peered":"2024-03-29T11:31:41.879198+0000","last_clean":"2024-03-29T11:31:41.879198+0000","last_became_active":"2024-03-29T11:31:40.889514+0000","last_became_peered":"2024-03-29T11:31:40.889514+0000","last_unstale":"2024-03-29T11:31:41.879198+0000","last_undegraded":"2024-03-29T11:31:41.879198+0000","last_fullsized":"2024-03-29T11:31:41.879198+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:22:49.143818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.13","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885809+0000","last_change":"2024-03-29T11:31:26.313768+0000","last_active":"2024-03-29T11:31:40.885809+0000","last_peered":"2024-03-29T11:31:40.885809+0000","last_clean":"2024-03-29T11:31:40.885809+0000","last_became_active":"2024-03-29T11:31:26.313458+0000","last_became_peered":"2024-03-29T11:31:26.313458+0000","last_unstale":"2024-03-29T11:31:40.885809+0000","last_undegraded":"2024-03-29T11:31:40.885809+0000","last_fullsized":"2024-03-29T11:31:40.885809+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:06.334180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883806+0000","last_change":"2024-03-29T11:31:26.840312+0000","last_active":"2024-03-29T11:31:40.883806+0000","last_peered":"2024-03-29T11:31:40.883806+0000","last_clean":"2024-03-29T11:31:40.883806+0000","last_became_active":"2024-03-29T11:31:26.840020+0000","last_became_peered":"2024-03-29T11:31:26.840020+0000","last_unstale":"2024-03-29T11:31:40.883806+0000","last_undegraded":"2024-03-29T11:31:40.883806+0000","last_fullsized":"2024-03-29T11:31:40.883806+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:31:51.171001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"230'3","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950922+0000","last_change":"2024-03-29T11:31:40.370725+0000","last_active":"2024-03-29T11:31:41.950922+0000","last_peered":"2024-03-29T11:31:41.950922+0000","last_clean":"2024-03-29T11:31:41.950922+0000","last_became_active":"2024-03-29T11:31:40.370456+0000","last_became_peered":"2024-03-29T11:31:40.370456+0000","last_unstale":"2024-03-29T11:31:41.950922+0000","last_undegraded":"2024-03-29T11:31:41.950922+0000","last_fullsized":"2024-03-29T11:31:41.950922+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:46.032832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949653+0000","last_change":"2024-03-29T11:31:40.902923+0000","last_active":"2024-03-29T11:31:41.949653+0000","last_peered":"2024-03-29T11:31:41.949653+0000","last_clean":"2024-03-29T11:31:41.949653+0000","last_became_active":"2024-03-29T11:31:40.902661+0000","last_became_peered":"2024-03-29T11:31:40.902661+0000","last_unstale":"2024-03-29T11:31:41.949653+0000","last_undegraded":"2024-03-29T11:31:41.949653+0000","last_fullsized":"2024-03-29T11:31:41.949653+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:26:19.372364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.14","version":"227'8","reported_seq":23,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.874187+0000","last_change":"2024-03-29T11:31:26.319966+0000","last_active":"2024-03-29T11:31:40.874187+0000","last_peered":"2024-03-29T11:31:40.874187+0000","last_clean":"2024-03-29T11:31:40.874187+0000","last_became_active":"2024-03-29T11:31:26.319700+0000","last_became_peered":"2024-03-29T11:31:26.319700+0000","last_unstale":"2024-03-29T11:31:40.874187+0000","last_undegraded":"2024-03-29T11:31:40.874187+0000","last_fullsized":"2024-03-29T11:31:40.874187+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:56.161261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873111+0000","last_change":"2024-03-29T11:31:26.844802+0000","last_active":"2024-03-29T11:31:40.873111+0000","last_peered":"2024-03-29T11:31:40.873111+0000","last_clean":"2024-03-29T11:31:40.873111+0000","last_became_active":"2024-03-29T11:31:26.844538+0000","last_became_peered":"2024-03-29T11:31:26.844538+0000","last_unstale":"2024-03-29T11:31:40.873111+0000","last_undegraded":"2024-03-29T11:31:40.873111+0000","last_fullsized":"2024-03-29T11:31:40.873111+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:05.909043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949757+0000","last_change":"2024-03-29T11:31:40.369428+0000","last_active":"2024-03-29T11:31:41.949757+0000","last_peered":"2024-03-29T11:31:41.949757+0000","last_clean":"2024-03-29T11:31:41.949757+0000","last_became_active":"2024-03-29T11:31:40.367133+0000","last_became_peered":"2024-03-29T11:31:40.367133+0000","last_unstale":"2024-03-29T11:31:41.949757+0000","last_undegraded":"2024-03-29T11:31:41.949757+0000","last_fullsized":"2024-03-29T11:31:41.949757+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:11:34.547512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880456+0000","last_change":"2024-03-29T11:31:40.889479+0000","last_active":"2024-03-29T11:31:41.880456+0000","last_peered":"2024-03-29T11:31:41.880456+0000","last_clean":"2024-03-29T11:31:41.880456+0000","last_became_active":"2024-03-29T11:31:40.884735+0000","last_became_peered":"2024-03-29T11:31:40.884735+0000","last_unstale":"2024-03-29T11:31:41.880456+0000","last_undegraded":"2024-03-29T11:31:41.880456+0000","last_fullsized":"2024-03-29T11:31:41.880456+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:48:28.663360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.15","version":"226'3","reported_seq":18,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876826+0000","last_change":"2024-03-29T11:31:26.320127+0000","last_active":"2024-03-29T11:31:40.876826+0000","last_peered":"2024-03-29T11:31:40.876826+0000","last_clean":"2024-03-29T11:31:40.876826+0000","last_became_active":"2024-03-29T11:31:26.319346+0000","last_became_peered":"2024-03-29T11:31:26.319346+0000","last_unstale":"2024-03-29T11:31:40.876826+0000","last_undegraded":"2024-03-29T11:31:40.876826+0000","last_fullsized":"2024-03-29T11:31:40.876826+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:32:50.637683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.874388+0000","last_change":"2024-03-29T11:31:26.857148+0000","last_active":"2024-03-29T11:31:40.874388+0000","last_peered":"2024-03-29T11:31:40.874388+0000","last_clean":"2024-03-29T11:31:40.874388+0000","last_became_active":"2024-03-29T11:31:26.856870+0000","last_became_peered":"2024-03-29T11:31:26.856870+0000","last_unstale":"2024-03-29T11:31:40.874388+0000","last_undegraded":"2024-03-29T11:31:40.874388+0000","last_fullsized":"2024-03-29T11:31:40.874388+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951631+0000","last_change":"2024-03-29T11:31:40.367974+0000","last_active":"2024-03-29T11:31:41.951631+0000","last_peered":"2024-03-29T11:31:41.951631+0000","last_clean":"2024-03-29T11:31:41.951631+0000","last_became_active":"2024-03-29T11:31:40.367528+0000","last_became_peered":"2024-03-29T11:31:40.367528+0000","last_unstale":"2024-03-29T11:31:41.951631+0000","last_undegraded":"2024-03-29T11:31:41.951631+0000","last_fullsized":"2024-03-29T11:31:41.951631+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:00:18.777176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949596+0000","last_change":"2024-03-29T11:31:40.883630+0000","last_active":"2024-03-29T11:31:41.949596+0000","last_peered":"2024-03-29T11:31:41.949596+0000","last_clean":"2024-03-29T11:31:41.949596+0000","last_became_active":"2024-03-29T11:31:40.883371+0000","last_became_peered":"2024-03-29T11:31:40.883371+0000","last_unstale":"2024-03-29T11:31:41.949596+0000","last_undegraded":"2024-03-29T11:31:41.949596+0000","last_fullsized":"2024-03-29T11:31:41.949596+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:24:49.255586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885781+0000","last_change":"2024-03-29T11:31:26.316324+0000","last_active":"2024-03-29T11:31:40.885781+0000","last_peered":"2024-03-29T11:31:40.885781+0000","last_clean":"2024-03-29T11:31:40.885781+0000","last_became_active":"2024-03-29T11:31:26.316066+0000","last_became_peered":"2024-03-29T11:31:26.316066+0000","last_unstale":"2024-03-29T11:31:40.885781+0000","last_undegraded":"2024-03-29T11:31:40.885781+0000","last_fullsized":"2024-03-29T11:31:40.885781+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:24:19.700336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884501+0000","last_change":"2024-03-29T11:31:26.846344+0000","last_active":"2024-03-29T11:31:40.884501+0000","last_peered":"2024-03-29T11:31:40.884501+0000","last_clean":"2024-03-29T11:31:40.884501+0000","last_became_active":"2024-03-29T11:31:26.846066+0000","last_became_peered":"2024-03-29T11:31:26.846066+0000","last_unstale":"2024-03-29T11:31:40.884501+0000","last_undegraded":"2024-03-29T11:31:40.884501+0000","last_fullsized":"2024-03-29T11:31:40.884501+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:57.163908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"230'8","reported_seq":19,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949749+0000","last_change":"2024-03-29T11:31:40.361408+0000","last_active":"2024-03-29T11:31:41.949749+0000","last_peered":"2024-03-29T11:31:41.949749+0000","last_clean":"2024-03-29T11:31:41.949749+0000","last_became_active":"2024-03-29T11:31:40.360360+0000","last_became_peered":"2024-03-29T11:31:40.360360+0000","last_unstale":"2024-03-29T11:31:41.949749+0000","last_undegraded":"2024-03-29T11:31:41.949749+0000","last_fullsized":"2024-03-29T11:31:41.949749+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:47:07.143595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949367+0000","last_change":"2024-03-29T11:31:40.891301+0000","last_active":"2024-03-29T11:31:41.949367+0000","last_peered":"2024-03-29T11:31:41.949367+0000","last_clean":"2024-03-29T11:31:41.949367+0000","last_became_active":"2024-03-29T11:31:40.891017+0000","last_became_peered":"2024-03-29T11:31:40.891017+0000","last_unstale":"2024-03-29T11:31:41.949367+0000","last_undegraded":"2024-03-29T11:31:41.949367+0000","last_fullsized":"2024-03-29T11:31:41.949367+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:07.759471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.891837+0000","last_change":"2024-03-29T11:31:26.317297+0000","last_active":"2024-03-29T11:31:40.891837+0000","last_peered":"2024-03-29T11:31:40.891837+0000","last_clean":"2024-03-29T11:31:40.891837+0000","last_became_active":"2024-03-29T11:31:26.316956+0000","last_became_peered":"2024-03-29T11:31:26.316956+0000","last_unstale":"2024-03-29T11:31:40.891837+0000","last_undegraded":"2024-03-29T11:31:40.891837+0000","last_fullsized":"2024-03-29T11:31:40.891837+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:18.302941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.877171+0000","last_change":"2024-03-29T11:31:26.855805+0000","last_active":"2024-03-29T11:31:40.877171+0000","last_peered":"2024-03-29T11:31:40.877171+0000","last_clean":"2024-03-29T11:31:40.877171+0000","last_became_active":"2024-03-29T11:31:26.854618+0000","last_became_peered":"2024-03-29T11:31:26.854618+0000","last_unstale":"2024-03-29T11:31:40.877171+0000","last_undegraded":"2024-03-29T11:31:40.877171+0000","last_fullsized":"2024-03-29T11:31:40.877171+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.831388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949535+0000","last_change":"2024-03-29T11:31:40.360159+0000","last_active":"2024-03-29T11:31:41.949535+0000","last_peered":"2024-03-29T11:31:41.949535+0000","last_clean":"2024-03-29T11:31:41.949535+0000","last_became_active":"2024-03-29T11:31:40.359689+0000","last_became_peered":"2024-03-29T11:31:40.359689+0000","last_unstale":"2024-03-29T11:31:41.949535+0000","last_undegraded":"2024-03-29T11:31:41.949535+0000","last_fullsized":"2024-03-29T11:31:41.949535+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:41.276410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879003+0000","last_change":"2024-03-29T11:31:40.905540+0000","last_active":"2024-03-29T11:31:41.879003+0000","last_peered":"2024-03-29T11:31:41.879003+0000","last_clean":"2024-03-29T11:31:41.879003+0000","last_became_active":"2024-03-29T11:31:40.904961+0000","last_became_peered":"2024-03-29T11:31:40.904961+0000","last_unstale":"2024-03-29T11:31:41.879003+0000","last_undegraded":"2024-03-29T11:31:41.879003+0000","last_fullsized":"2024-03-29T11:31:41.879003+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:43.064595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.894268+0000","last_change":"2024-03-29T11:31:26.320631+0000","last_active":"2024-03-29T11:31:40.894268+0000","last_peered":"2024-03-29T11:31:40.894268+0000","last_clean":"2024-03-29T11:31:40.894268+0000","last_became_active":"2024-03-29T11:31:26.320350+0000","last_became_peered":"2024-03-29T11:31:26.320350+0000","last_unstale":"2024-03-29T11:31:40.894268+0000","last_undegraded":"2024-03-29T11:31:40.894268+0000","last_fullsized":"2024-03-29T11:31:40.894268+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:57.806178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872517+0000","last_change":"2024-03-29T11:31:26.847903+0000","last_active":"2024-03-29T11:31:40.872517+0000","last_peered":"2024-03-29T11:31:40.872517+0000","last_clean":"2024-03-29T11:31:40.872517+0000","last_became_active":"2024-03-29T11:31:26.846171+0000","last_became_peered":"2024-03-29T11:31:26.846171+0000","last_unstale":"2024-03-29T11:31:40.872517+0000","last_undegraded":"2024-03-29T11:31:40.872517+0000","last_fullsized":"2024-03-29T11:31:40.872517+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:24.230359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951196+0000","last_change":"2024-03-29T11:31:40.372638+0000","last_active":"2024-03-29T11:31:41.951196+0000","last_peered":"2024-03-29T11:31:41.951196+0000","last_clean":"2024-03-29T11:31:41.951196+0000","last_became_active":"2024-03-29T11:31:40.372368+0000","last_became_peered":"2024-03-29T11:31:40.372368+0000","last_unstale":"2024-03-29T11:31:41.951196+0000","last_undegraded":"2024-03-29T11:31:41.951196+0000","last_fullsized":"2024-03-29T11:31:41.951196+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:27:42.526026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951223+0000","last_change":"2024-03-29T11:31:40.900805+0000","last_active":"2024-03-29T11:31:41.951223+0000","last_peered":"2024-03-29T11:31:41.951223+0000","last_clean":"2024-03-29T11:31:41.951223+0000","last_became_active":"2024-03-29T11:31:40.900541+0000","last_became_peered":"2024-03-29T11:31:40.900541+0000","last_unstale":"2024-03-29T11:31:41.951223+0000","last_undegraded":"2024-03-29T11:31:41.951223+0000","last_fullsized":"2024-03-29T11:31:41.951223+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:30.611972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882346+0000","last_change":"2024-03-29T11:31:26.310235+0000","last_active":"2024-03-29T11:31:40.882346+0000","last_peered":"2024-03-29T11:31:40.882346+0000","last_clean":"2024-03-29T11:31:40.882346+0000","last_became_active":"2024-03-29T11:31:26.309932+0000","last_became_peered":"2024-03-29T11:31:26.309932+0000","last_unstale":"2024-03-29T11:31:40.882346+0000","last_undegraded":"2024-03-29T11:31:40.882346+0000","last_fullsized":"2024-03-29T11:31:40.882346+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:31:30.532498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883923+0000","last_change":"2024-03-29T11:31:26.850244+0000","last_active":"2024-03-29T11:31:40.883923+0000","last_peered":"2024-03-29T11:31:40.883923+0000","last_clean":"2024-03-29T11:31:40.883923+0000","last_became_active":"2024-03-29T11:31:26.849736+0000","last_became_peered":"2024-03-29T11:31:26.849736+0000","last_unstale":"2024-03-29T11:31:40.883923+0000","last_undegraded":"2024-03-29T11:31:40.883923+0000","last_fullsized":"2024-03-29T11:31:40.883923+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:57.906301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"230'4","reported_seq":15,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950525+0000","last_change":"2024-03-29T11:31:40.366981+0000","last_active":"2024-03-29T11:31:41.950525+0000","last_peered":"2024-03-29T11:31:41.950525+0000","last_clean":"2024-03-29T11:31:41.950525+0000","last_became_active":"2024-03-29T11:31:40.366421+0000","last_became_peered":"2024-03-29T11:31:40.366421+0000","last_unstale":"2024-03-29T11:31:41.950525+0000","last_undegraded":"2024-03-29T11:31:41.950525+0000","last_fullsized":"2024-03-29T11:31:41.950525+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:42.057358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880833+0000","last_change":"2024-03-29T11:31:40.898286+0000","last_active":"2024-03-29T11:31:41.880833+0000","last_peered":"2024-03-29T11:31:41.880833+0000","last_clean":"2024-03-29T11:31:41.880833+0000","last_became_active":"2024-03-29T11:31:40.897891+0000","last_became_peered":"2024-03-29T11:31:40.897891+0000","last_unstale":"2024-03-29T11:31:41.880833+0000","last_undegraded":"2024-03-29T11:31:41.880833+0000","last_fullsized":"2024-03-29T11:31:41.880833+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:38:17.107942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1a","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882899+0000","last_change":"2024-03-29T11:31:26.311383+0000","last_active":"2024-03-29T11:31:40.882899+0000","last_peered":"2024-03-29T11:31:40.882899+0000","last_clean":"2024-03-29T11:31:40.882899+0000","last_became_active":"2024-03-29T11:31:26.310630+0000","last_became_peered":"2024-03-29T11:31:26.310630+0000","last_unstale":"2024-03-29T11:31:40.882899+0000","last_undegraded":"2024-03-29T11:31:40.882899+0000","last_fullsized":"2024-03-29T11:31:40.882899+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:51:16.023346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872991+0000","last_change":"2024-03-29T11:31:26.853179+0000","last_active":"2024-03-29T11:31:40.872991+0000","last_peered":"2024-03-29T11:31:40.872991+0000","last_clean":"2024-03-29T11:31:40.872991+0000","last_became_active":"2024-03-29T11:31:26.848186+0000","last_became_peered":"2024-03-29T11:31:26.848186+0000","last_unstale":"2024-03-29T11:31:40.872991+0000","last_undegraded":"2024-03-29T11:31:40.872991+0000","last_fullsized":"2024-03-29T11:31:40.872991+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:35:08.770095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950726+0000","last_change":"2024-03-29T11:31:40.366984+0000","last_active":"2024-03-29T11:31:41.950726+0000","last_peered":"2024-03-29T11:31:41.950726+0000","last_clean":"2024-03-29T11:31:41.950726+0000","last_became_active":"2024-03-29T11:31:40.366706+0000","last_became_peered":"2024-03-29T11:31:40.366706+0000","last_unstale":"2024-03-29T11:31:41.950726+0000","last_undegraded":"2024-03-29T11:31:41.950726+0000","last_fullsized":"2024-03-29T11:31:41.950726+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:03.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950153+0000","last_change":"2024-03-29T11:31:40.904479+0000","last_active":"2024-03-29T11:31:41.950153+0000","last_peered":"2024-03-29T11:31:41.950153+0000","last_clean":"2024-03-29T11:31:41.950153+0000","last_became_active":"2024-03-29T11:31:40.904022+0000","last_became_peered":"2024-03-29T11:31:40.904022+0000","last_unstale":"2024-03-29T11:31:41.950153+0000","last_undegraded":"2024-03-29T11:31:41.950153+0000","last_fullsized":"2024-03-29T11:31:41.950153+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:44:04.775529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1c","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.874299+0000","last_change":"2024-03-29T11:31:26.321235+0000","last_active":"2024-03-29T11:31:40.874299+0000","last_peered":"2024-03-29T11:31:40.874299+0000","last_clean":"2024-03-29T11:31:40.874299+0000","last_became_active":"2024-03-29T11:31:26.320454+0000","last_became_peered":"2024-03-29T11:31:26.320454+0000","last_unstale":"2024-03-29T11:31:40.874299+0000","last_undegraded":"2024-03-29T11:31:40.874299+0000","last_fullsized":"2024-03-29T11:31:40.874299+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:27.238049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875962+0000","last_change":"2024-03-29T11:31:26.840358+0000","last_active":"2024-03-29T11:31:40.875962+0000","last_peered":"2024-03-29T11:31:40.875962+0000","last_clean":"2024-03-29T11:31:40.875962+0000","last_became_active":"2024-03-29T11:31:26.838477+0000","last_became_peered":"2024-03-29T11:31:26.838477+0000","last_unstale":"2024-03-29T11:31:40.875962+0000","last_undegraded":"2024-03-29T11:31:40.875962+0000","last_fullsized":"2024-03-29T11:31:40.875962+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:27.122635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880281+0000","last_change":"2024-03-29T11:31:40.366196+0000","last_active":"2024-03-29T11:31:41.880281+0000","last_peered":"2024-03-29T11:31:41.880281+0000","last_clean":"2024-03-29T11:31:41.880281+0000","last_became_active":"2024-03-29T11:31:40.365887+0000","last_became_peered":"2024-03-29T11:31:40.365887+0000","last_unstale":"2024-03-29T11:31:41.880281+0000","last_undegraded":"2024-03-29T11:31:41.880281+0000","last_fullsized":"2024-03-29T11:31:41.880281+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:27:42.526026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950426+0000","last_change":"2024-03-29T11:31:40.897244+0000","last_active":"2024-03-29T11:31:41.950426+0000","last_peered":"2024-03-29T11:31:41.950426+0000","last_clean":"2024-03-29T11:31:41.950426+0000","last_became_active":"2024-03-29T11:31:40.896998+0000","last_became_peered":"2024-03-29T11:31:40.896998+0000","last_unstale":"2024-03-29T11:31:41.950426+0000","last_undegraded":"2024-03-29T11:31:41.950426+0000","last_fullsized":"2024-03-29T11:31:41.950426+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:46.656502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1d","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883866+0000","last_change":"2024-03-29T11:31:26.319707+0000","last_active":"2024-03-29T11:31:40.883866+0000","last_peered":"2024-03-29T11:31:40.883866+0000","last_clean":"2024-03-29T11:31:40.883866+0000","last_became_active":"2024-03-29T11:31:26.316121+0000","last_became_peered":"2024-03-29T11:31:26.316121+0000","last_unstale":"2024-03-29T11:31:40.883866+0000","last_undegraded":"2024-03-29T11:31:40.883866+0000","last_fullsized":"2024-03-29T11:31:40.883866+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:12:51.039260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872617+0000","last_change":"2024-03-29T11:31:26.850407+0000","last_active":"2024-03-29T11:31:40.872617+0000","last_peered":"2024-03-29T11:31:40.872617+0000","last_clean":"2024-03-29T11:31:40.872617+0000","last_became_active":"2024-03-29T11:31:26.849457+0000","last_became_peered":"2024-03-29T11:31:26.849457+0000","last_unstale":"2024-03-29T11:31:40.872617+0000","last_undegraded":"2024-03-29T11:31:40.872617+0000","last_fullsized":"2024-03-29T11:31:40.872617+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:38:50.961636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951718+0000","last_change":"2024-03-29T11:31:40.367779+0000","last_active":"2024-03-29T11:31:41.951718+0000","last_peered":"2024-03-29T11:31:41.951718+0000","last_clean":"2024-03-29T11:31:41.951718+0000","last_became_active":"2024-03-29T11:31:40.367158+0000","last_became_peered":"2024-03-29T11:31:40.367158+0000","last_unstale":"2024-03-29T11:31:41.951718+0000","last_undegraded":"2024-03-29T11:31:41.951718+0000","last_fullsized":"2024-03-29T11:31:41.951718+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:38:16.100705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879473+0000","last_change":"2024-03-29T11:31:40.891827+0000","last_active":"2024-03-29T11:31:41.879473+0000","last_peered":"2024-03-29T11:31:41.879473+0000","last_clean":"2024-03-29T11:31:41.879473+0000","last_became_active":"2024-03-29T11:31:40.891565+0000","last_became_peered":"2024-03-29T11:31:40.891565+0000","last_unstale":"2024-03-29T11:31:41.879473+0000","last_undegraded":"2024-03-29T11:31:41.879473+0000","last_fullsized":"2024-03-29T11:31:41.879473+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:58.211565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1e","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883958+0000","last_change":"2024-03-29T11:31:26.320463+0000","last_active":"2024-03-29T11:31:40.883958+0000","last_peered":"2024-03-29T11:31:40.883958+0000","last_clean":"2024-03-29T11:31:40.883958+0000","last_became_active":"2024-03-29T11:31:26.319713+0000","last_became_peered":"2024-03-29T11:31:26.319713+0000","last_unstale":"2024-03-29T11:31:40.883958+0000","last_undegraded":"2024-03-29T11:31:40.883958+0000","last_fullsized":"2024-03-29T11:31:40.883958+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:32:47.003355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885062+0000","last_change":"2024-03-29T11:31:26.843531+0000","last_active":"2024-03-29T11:31:40.885062+0000","last_peered":"2024-03-29T11:31:40.885062+0000","last_clean":"2024-03-29T11:31:40.885062+0000","last_became_active":"2024-03-29T11:31:26.843164+0000","last_became_peered":"2024-03-29T11:31:26.843164+0000","last_unstale":"2024-03-29T11:31:40.885062+0000","last_undegraded":"2024-03-29T11:31:40.885062+0000","last_fullsized":"2024-03-29T11:31:40.885062+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:29:23.515574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.873648+0000","last_change":"2024-03-29T11:31:40.366096+0000","last_active":"2024-03-29T11:31:41.873648+0000","last_peered":"2024-03-29T11:31:41.873648+0000","last_clean":"2024-03-29T11:31:41.873648+0000","last_became_active":"2024-03-29T11:31:40.365769+0000","last_became_peered":"2024-03-29T11:31:40.365769+0000","last_unstale":"2024-03-29T11:31:41.873648+0000","last_undegraded":"2024-03-29T11:31:41.873648+0000","last_fullsized":"2024-03-29T11:31:41.873648+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:10:45.255586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949865+0000","last_change":"2024-03-29T11:31:40.893845+0000","last_active":"2024-03-29T11:31:41.949865+0000","last_peered":"2024-03-29T11:31:41.949865+0000","last_clean":"2024-03-29T11:31:41.949865+0000","last_became_active":"2024-03-29T11:31:40.891813+0000","last_became_peered":"2024-03-29T11:31:40.891813+0000","last_unstale":"2024-03-29T11:31:41.949865+0000","last_undegraded":"2024-03-29T11:31:41.949865+0000","last_fullsized":"2024-03-29T11:31:41.949865+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:38.273310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1f","version":"226'4","reported_seq":19,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875640+0000","last_change":"2024-03-29T11:31:26.315269+0000","last_active":"2024-03-29T11:31:40.875640+0000","last_peered":"2024-03-29T11:31:40.875640+0000","last_clean":"2024-03-29T11:31:40.875640+0000","last_became_active":"2024-03-29T11:31:26.314956+0000","last_became_peered":"2024-03-29T11:31:26.314956+0000","last_unstale":"2024-03-29T11:31:40.875640+0000","last_undegraded":"2024-03-29T11:31:40.875640+0000","last_fullsized":"2024-03-29T11:31:40.875640+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:18.302941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884150+0000","last_change":"2024-03-29T11:31:26.844474+0000","last_active":"2024-03-29T11:31:40.884150+0000","last_peered":"2024-03-29T11:31:40.884150+0000","last_clean":"2024-03-29T11:31:40.884150+0000","last_became_active":"2024-03-29T11:31:26.844195+0000","last_became_peered":"2024-03-29T11:31:26.844195+0000","last_unstale":"2024-03-29T11:31:40.884150+0000","last_undegraded":"2024-03-29T11:31:40.884150+0000","last_fullsized":"2024-03-29T11:31:40.884150+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:38:37.668282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949371+0000","last_change":"2024-03-29T11:31:40.368937+0000","last_active":"2024-03-29T11:31:41.949371+0000","last_peered":"2024-03-29T11:31:41.949371+0000","last_clean":"2024-03-29T11:31:41.949371+0000","last_became_active":"2024-03-29T11:31:40.368617+0000","last_became_peered":"2024-03-29T11:31:40.368617+0000","last_unstale":"2024-03-29T11:31:41.949371+0000","last_undegraded":"2024-03-29T11:31:41.949371+0000","last_fullsized":"2024-03-29T11:31:41.949371+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:22.207936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.873719+0000","last_change":"2024-03-29T11:31:40.898328+0000","last_active":"2024-03-29T11:31:41.873719+0000","last_peered":"2024-03-29T11:31:41.873719+0000","last_clean":"2024-03-29T11:31:41.873719+0000","last_became_active":"2024-03-29T11:31:40.898058+0000","last_became_peered":"2024-03-29T11:31:40.898058+0000","last_unstale":"2024-03-29T11:31:41.873719+0000","last_undegraded":"2024-03-29T11:31:41.873719+0000","last_fullsized":"2024-03-29T11:31:41.873719+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:58:55.577093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951337+0000","last_change":"2024-03-29T11:31:40.891542+0000","last_active":"2024-03-29T11:31:41.951337+0000","last_peered":"2024-03-29T11:31:41.951337+0000","last_clean":"2024-03-29T11:31:41.951337+0000","last_became_active":"2024-03-29T11:31:40.889902+0000","last_became_peered":"2024-03-29T11:31:40.889902+0000","last_unstale":"2024-03-29T11:31:41.951337+0000","last_undegraded":"2024-03-29T11:31:41.951337+0000","last_fullsized":"2024-03-29T11:31:41.951337+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:55.281545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870195+0000","last_change":"2024-03-29T11:31:26.839138+0000","last_active":"2024-03-29T11:31:40.870195+0000","last_peered":"2024-03-29T11:31:40.870195+0000","last_clean":"2024-03-29T11:31:40.870195+0000","last_became_active":"2024-03-29T11:31:26.838904+0000","last_became_peered":"2024-03-29T11:31:26.838904+0000","last_unstale":"2024-03-29T11:31:40.870195+0000","last_undegraded":"2024-03-29T11:31:40.870195+0000","last_fullsized":"2024-03-29T11:31:40.870195+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:54.107881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880429+0000","last_change":"2024-03-29T11:31:40.904048+0000","last_active":"2024-03-29T11:31:41.880429+0000","last_peered":"2024-03-29T11:31:41.880429+0000","last_clean":"2024-03-29T11:31:41.880429+0000","last_became_active":"2024-03-29T11:31:40.903770+0000","last_became_peered":"2024-03-29T11:31:40.903770+0000","last_unstale":"2024-03-29T11:31:41.880429+0000","last_undegraded":"2024-03-29T11:31:41.880429+0000","last_fullsized":"2024-03-29T11:31:41.880429+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:17.301421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875501+0000","last_change":"2024-03-29T11:31:26.847737+0000","last_active":"2024-03-29T11:31:40.875501+0000","last_peered":"2024-03-29T11:31:40.875501+0000","last_clean":"2024-03-29T11:31:40.875501+0000","last_became_active":"2024-03-29T11:31:26.847473+0000","last_became_peered":"2024-03-29T11:31:26.847473+0000","last_unstale":"2024-03-29T11:31:40.875501+0000","last_undegraded":"2024-03-29T11:31:40.875501+0000","last_fullsized":"2024-03-29T11:31:40.875501+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:36:05.541971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.891946+0000","last_change":"2024-03-29T11:31:26.842077+0000","last_active":"2024-03-29T11:31:40.891946+0000","last_peered":"2024-03-29T11:31:40.891946+0000","last_clean":"2024-03-29T11:31:40.891946+0000","last_became_active":"2024-03-29T11:31:26.841818+0000","last_became_peered":"2024-03-29T11:31:26.841818+0000","last_unstale":"2024-03-29T11:31:40.891946+0000","last_undegraded":"2024-03-29T11:31:40.891946+0000","last_fullsized":"2024-03-29T11:31:40.891946+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:42:57.193431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874594+0000","last_change":"2024-03-29T11:31:40.904727+0000","last_active":"2024-03-29T11:31:41.874594+0000","last_peered":"2024-03-29T11:31:41.874594+0000","last_clean":"2024-03-29T11:31:41.874594+0000","last_became_active":"2024-03-29T11:31:40.904081+0000","last_became_peered":"2024-03-29T11:31:40.904081+0000","last_unstale":"2024-03-29T11:31:41.874594+0000","last_undegraded":"2024-03-29T11:31:41.874594+0000","last_fullsized":"2024-03-29T11:31:41.874594+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:42.265960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.878841+0000","last_change":"2024-03-29T11:31:40.907166+0000","last_active":"2024-03-29T11:31:41.878841+0000","last_peered":"2024-03-29T11:31:41.878841+0000","last_clean":"2024-03-29T11:31:41.878841+0000","last_became_active":"2024-03-29T11:31:40.905090+0000","last_became_peered":"2024-03-29T11:31:40.905090+0000","last_unstale":"2024-03-29T11:31:41.878841+0000","last_undegraded":"2024-03-29T11:31:41.878841+0000","last_fullsized":"2024-03-29T11:31:41.878841+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:36.816868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875554+0000","last_change":"2024-03-29T11:31:26.848961+0000","last_active":"2024-03-29T11:31:40.875554+0000","last_peered":"2024-03-29T11:31:40.875554+0000","last_clean":"2024-03-29T11:31:40.875554+0000","last_became_active":"2024-03-29T11:31:26.848637+0000","last_became_peered":"2024-03-29T11:31:26.848637+0000","last_unstale":"2024-03-29T11:31:40.875554+0000","last_undegraded":"2024-03-29T11:31:40.875554+0000","last_fullsized":"2024-03-29T11:31:40.875554+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:31:51.171001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883480+0000","last_change":"2024-03-29T11:31:26.838393+0000","last_active":"2024-03-29T11:31:40.883480+0000","last_peered":"2024-03-29T11:31:40.883480+0000","last_clean":"2024-03-29T11:31:40.883480+0000","last_became_active":"2024-03-29T11:31:26.838003+0000","last_became_peered":"2024-03-29T11:31:26.838003+0000","last_unstale":"2024-03-29T11:31:40.883480+0000","last_undegraded":"2024-03-29T11:31:40.883480+0000","last_fullsized":"2024-03-29T11:31:40.883480+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:47.195933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879316+0000","last_change":"2024-03-29T11:31:40.892720+0000","last_active":"2024-03-29T11:31:41.879316+0000","last_peered":"2024-03-29T11:31:41.879316+0000","last_clean":"2024-03-29T11:31:41.879316+0000","last_became_active":"2024-03-29T11:31:40.892425+0000","last_became_peered":"2024-03-29T11:31:40.892425+0000","last_unstale":"2024-03-29T11:31:41.879316+0000","last_undegraded":"2024-03-29T11:31:41.879316+0000","last_fullsized":"2024-03-29T11:31:41.879316+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:46.155852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.879257+0000","last_change":"2024-03-29T11:31:26.852411+0000","last_active":"2024-03-29T11:31:40.879257+0000","last_peered":"2024-03-29T11:31:40.879257+0000","last_clean":"2024-03-29T11:31:40.879257+0000","last_became_active":"2024-03-29T11:31:26.852100+0000","last_became_peered":"2024-03-29T11:31:26.852100+0000","last_unstale":"2024-03-29T11:31:40.879257+0000","last_undegraded":"2024-03-29T11:31:40.879257+0000","last_fullsized":"2024-03-29T11:31:40.879257+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:15:49.012984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949498+0000","last_change":"2024-03-29T11:31:40.904368+0000","last_active":"2024-03-29T11:31:41.949498+0000","last_peered":"2024-03-29T11:31:41.949498+0000","last_clean":"2024-03-29T11:31:41.949498+0000","last_became_active":"2024-03-29T11:31:40.903294+0000","last_became_peered":"2024-03-29T11:31:40.903294+0000","last_unstale":"2024-03-29T11:31:41.949498+0000","last_undegraded":"2024-03-29T11:31:41.949498+0000","last_fullsized":"2024-03-29T11:31:41.949498+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:38:17.107942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872844+0000","last_change":"2024-03-29T11:31:26.843275+0000","last_active":"2024-03-29T11:31:40.872844+0000","last_peered":"2024-03-29T11:31:40.872844+0000","last_clean":"2024-03-29T11:31:40.872844+0000","last_became_active":"2024-03-29T11:31:26.842588+0000","last_became_peered":"2024-03-29T11:31:26.842588+0000","last_unstale":"2024-03-29T11:31:40.872844+0000","last_undegraded":"2024-03-29T11:31:40.872844+0000","last_fullsized":"2024-03-29T11:31:40.872844+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:57.876055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.948965+0000","last_change":"2024-03-29T11:31:40.890546+0000","last_active":"2024-03-29T11:31:41.948965+0000","last_peered":"2024-03-29T11:31:41.948965+0000","last_clean":"2024-03-29T11:31:41.948965+0000","last_became_active":"2024-03-29T11:31:40.890297+0000","last_became_peered":"2024-03-29T11:31:40.890297+0000","last_unstale":"2024-03-29T11:31:41.948965+0000","last_undegraded":"2024-03-29T11:31:41.948965+0000","last_fullsized":"2024-03-29T11:31:41.948965+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:36:29.757453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.889285+0000","last_change":"2024-03-29T11:31:26.847195+0000","last_active":"2024-03-29T11:31:40.889285+0000","last_peered":"2024-03-29T11:31:40.889285+0000","last_clean":"2024-03-29T11:31:40.889285+0000","last_became_active":"2024-03-29T11:31:26.846876+0000","last_became_peered":"2024-03-29T11:31:26.846876+0000","last_unstale":"2024-03-29T11:31:40.889285+0000","last_undegraded":"2024-03-29T11:31:40.889285+0000","last_fullsized":"2024-03-29T11:31:40.889285+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:41.350201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.948680+0000","last_change":"2024-03-29T11:31:40.890331+0000","last_active":"2024-03-29T11:31:41.948680+0000","last_peered":"2024-03-29T11:31:41.948680+0000","last_clean":"2024-03-29T11:31:41.948680+0000","last_became_active":"2024-03-29T11:31:40.886395+0000","last_became_peered":"2024-03-29T11:31:40.886395+0000","last_unstale":"2024-03-29T11:31:41.948680+0000","last_undegraded":"2024-03-29T11:31:41.948680+0000","last_fullsized":"2024-03-29T11:31:41.948680+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:38.273310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874691+0000","last_change":"2024-03-29T11:31:40.893585+0000","last_active":"2024-03-29T11:31:41.874691+0000","last_peered":"2024-03-29T11:31:41.874691+0000","last_clean":"2024-03-29T11:31:41.874691+0000","last_became_active":"2024-03-29T11:31:40.893333+0000","last_became_peered":"2024-03-29T11:31:40.893333+0000","last_unstale":"2024-03-29T11:31:41.874691+0000","last_undegraded":"2024-03-29T11:31:41.874691+0000","last_fullsized":"2024-03-29T11:31:41.874691+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:37.010261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870753+0000","last_change":"2024-03-29T11:31:26.855506+0000","last_active":"2024-03-29T11:31:40.870753+0000","last_peered":"2024-03-29T11:31:40.870753+0000","last_clean":"2024-03-29T11:31:40.870753+0000","last_became_active":"2024-03-29T11:31:26.852090+0000","last_became_peered":"2024-03-29T11:31:26.852090+0000","last_unstale":"2024-03-29T11:31:40.870753+0000","last_undegraded":"2024-03-29T11:31:40.870753+0000","last_fullsized":"2024-03-29T11:31:40.870753+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:09.172222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951113+0000","last_change":"2024-03-29T11:31:40.896412+0000","last_active":"2024-03-29T11:31:41.951113+0000","last_peered":"2024-03-29T11:31:41.951113+0000","last_clean":"2024-03-29T11:31:41.951113+0000","last_became_active":"2024-03-29T11:31:40.896123+0000","last_became_peered":"2024-03-29T11:31:40.896123+0000","last_unstale":"2024-03-29T11:31:41.951113+0000","last_undegraded":"2024-03-29T11:31:41.951113+0000","last_fullsized":"2024-03-29T11:31:41.951113+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:48:28.663360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.886751+0000","last_change":"2024-03-29T11:31:26.842715+0000","last_active":"2024-03-29T11:31:40.886751+0000","last_peered":"2024-03-29T11:31:40.886751+0000","last_clean":"2024-03-29T11:31:40.886751+0000","last_became_active":"2024-03-29T11:31:26.842354+0000","last_became_peered":"2024-03-29T11:31:26.842354+0000","last_unstale":"2024-03-29T11:31:40.886751+0000","last_undegraded":"2024-03-29T11:31:40.886751+0000","last_fullsized":"2024-03-29T11:31:40.886751+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:58.653895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874662+0000","last_change":"2024-03-29T11:31:40.878065+0000","last_active":"2024-03-29T11:31:41.874662+0000","last_peered":"2024-03-29T11:31:41.874662+0000","last_clean":"2024-03-29T11:31:41.874662+0000","last_became_active":"2024-03-29T11:31:40.877784+0000","last_became_peered":"2024-03-29T11:31:40.877784+0000","last_unstale":"2024-03-29T11:31:41.874662+0000","last_undegraded":"2024-03-29T11:31:41.874662+0000","last_fullsized":"2024-03-29T11:31:41.874662+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:13:00.089904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885512+0000","last_change":"2024-03-29T11:31:26.834482+0000","last_active":"2024-03-29T11:31:40.885512+0000","last_peered":"2024-03-29T11:31:40.885512+0000","last_clean":"2024-03-29T11:31:40.885512+0000","last_became_active":"2024-03-29T11:31:26.833118+0000","last_became_peered":"2024-03-29T11:31:26.833118+0000","last_unstale":"2024-03-29T11:31:40.885512+0000","last_undegraded":"2024-03-29T11:31:40.885512+0000","last_fullsized":"2024-03-29T11:31:40.885512+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:42.629666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950997+0000","last_change":"2024-03-29T11:31:40.902550+0000","last_active":"2024-03-29T11:31:41.950997+0000","last_peered":"2024-03-29T11:31:41.950997+0000","last_clean":"2024-03-29T11:31:41.950997+0000","last_became_active":"2024-03-29T11:31:40.899855+0000","last_became_peered":"2024-03-29T11:31:40.899855+0000","last_unstale":"2024-03-29T11:31:41.950997+0000","last_undegraded":"2024-03-29T11:31:41.950997+0000","last_fullsized":"2024-03-29T11:31:41.950997+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:36:51.759675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873153+0000","last_change":"2024-03-29T11:31:26.856449+0000","last_active":"2024-03-29T11:31:40.873153+0000","last_peered":"2024-03-29T11:31:40.873153+0000","last_clean":"2024-03-29T11:31:40.873153+0000","last_became_active":"2024-03-29T11:31:26.856166+0000","last_became_peered":"2024-03-29T11:31:26.856166+0000","last_unstale":"2024-03-29T11:31:40.873153+0000","last_undegraded":"2024-03-29T11:31:40.873153+0000","last_fullsized":"2024-03-29T11:31:40.873153+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:27.122635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949406+0000","last_change":"2024-03-29T11:31:40.890141+0000","last_active":"2024-03-29T11:31:41.949406+0000","last_peered":"2024-03-29T11:31:41.949406+0000","last_clean":"2024-03-29T11:31:41.949406+0000","last_became_active":"2024-03-29T11:31:40.889862+0000","last_became_peered":"2024-03-29T11:31:40.889862+0000","last_unstale":"2024-03-29T11:31:41.949406+0000","last_undegraded":"2024-03-29T11:31:41.949406+0000","last_fullsized":"2024-03-29T11:31:41.949406+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:28:27.804989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.892869+0000","last_change":"2024-03-29T11:31:26.843448+0000","last_active":"2024-03-29T11:31:40.892869+0000","last_peered":"2024-03-29T11:31:40.892869+0000","last_clean":"2024-03-29T11:31:40.892869+0000","last_became_active":"2024-03-29T11:31:26.843183+0000","last_became_peered":"2024-03-29T11:31:26.843183+0000","last_unstale":"2024-03-29T11:31:40.892869+0000","last_undegraded":"2024-03-29T11:31:40.892869+0000","last_fullsized":"2024-03-29T11:31:40.892869+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:23:23.236057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950159+0000","last_change":"2024-03-29T11:31:40.901765+0000","last_active":"2024-03-29T11:31:41.950159+0000","last_peered":"2024-03-29T11:31:41.950159+0000","last_clean":"2024-03-29T11:31:41.950159+0000","last_became_active":"2024-03-29T11:31:40.899905+0000","last_became_peered":"2024-03-29T11:31:40.899905+0000","last_unstale":"2024-03-29T11:31:41.950159+0000","last_undegraded":"2024-03-29T11:31:41.950159+0000","last_fullsized":"2024-03-29T11:31:41.950159+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:13.874339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875882+0000","last_change":"2024-03-29T11:31:26.854890+0000","last_active":"2024-03-29T11:31:40.875882+0000","last_peered":"2024-03-29T11:31:40.875882+0000","last_clean":"2024-03-29T11:31:40.875882+0000","last_became_active":"2024-03-29T11:31:26.854625+0000","last_became_peered":"2024-03-29T11:31:26.854625+0000","last_unstale":"2024-03-29T11:31:40.875882+0000","last_undegraded":"2024-03-29T11:31:40.875882+0000","last_fullsized":"2024-03-29T11:31:40.875882+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.831388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.948696+0000","last_change":"2024-03-29T11:31:40.885062+0000","last_active":"2024-03-29T11:31:41.948696+0000","last_peered":"2024-03-29T11:31:41.948696+0000","last_clean":"2024-03-29T11:31:41.948696+0000","last_became_active":"2024-03-29T11:31:40.884786+0000","last_became_peered":"2024-03-29T11:31:40.884786+0000","last_unstale":"2024-03-29T11:31:41.948696+0000","last_undegraded":"2024-03-29T11:31:41.948696+0000","last_fullsized":"2024-03-29T11:31:41.948696+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:09:49.649528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876629+0000","last_change":"2024-03-29T11:31:26.840048+0000","last_active":"2024-03-29T11:31:40.876629+0000","last_peered":"2024-03-29T11:31:40.876629+0000","last_clean":"2024-03-29T11:31:40.876629+0000","last_became_active":"2024-03-29T11:31:26.838725+0000","last_became_peered":"2024-03-29T11:31:26.838725+0000","last_unstale":"2024-03-29T11:31:40.876629+0000","last_undegraded":"2024-03-29T11:31:40.876629+0000","last_fullsized":"2024-03-29T11:31:40.876629+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:09.172222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.878352+0000","last_change":"2024-03-29T11:31:40.884649+0000","last_active":"2024-03-29T11:31:41.878352+0000","last_peered":"2024-03-29T11:31:41.878352+0000","last_clean":"2024-03-29T11:31:41.878352+0000","last_became_active":"2024-03-29T11:31:40.884374+0000","last_became_peered":"2024-03-29T11:31:40.884374+0000","last_unstale":"2024-03-29T11:31:41.878352+0000","last_undegraded":"2024-03-29T11:31:41.878352+0000","last_fullsized":"2024-03-29T11:31:41.878352+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:57:25.207829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872907+0000","last_change":"2024-03-29T11:31:26.850388+0000","last_active":"2024-03-29T11:31:40.872907+0000","last_peered":"2024-03-29T11:31:40.872907+0000","last_clean":"2024-03-29T11:31:40.872907+0000","last_became_active":"2024-03-29T11:31:26.849826+0000","last_became_peered":"2024-03-29T11:31:26.849826+0000","last_unstale":"2024-03-29T11:31:40.872907+0000","last_undegraded":"2024-03-29T11:31:40.872907+0000","last_fullsized":"2024-03-29T11:31:40.872907+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:55:45.796741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874662+0000","last_change":"2024-03-29T11:31:40.884799+0000","last_active":"2024-03-29T11:31:41.874662+0000","last_peered":"2024-03-29T11:31:41.874662+0000","last_clean":"2024-03-29T11:31:41.874662+0000","last_became_active":"2024-03-29T11:31:40.884544+0000","last_became_peered":"2024-03-29T11:31:40.884544+0000","last_unstale":"2024-03-29T11:31:41.874662+0000","last_undegraded":"2024-03-29T11:31:41.874662+0000","last_fullsized":"2024-03-29T11:31:41.874662+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:51.749739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875157+0000","last_change":"2024-03-29T11:31:26.843040+0000","last_active":"2024-03-29T11:31:40.875157+0000","last_peered":"2024-03-29T11:31:40.875157+0000","last_clean":"2024-03-29T11:31:40.875157+0000","last_became_active":"2024-03-29T11:31:26.841805+0000","last_became_peered":"2024-03-29T11:31:26.841805+0000","last_unstale":"2024-03-29T11:31:40.875157+0000","last_undegraded":"2024-03-29T11:31:40.875157+0000","last_fullsized":"2024-03-29T11:31:40.875157+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:05.909043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.881086+0000","last_change":"2024-03-29T11:31:40.890262+0000","last_active":"2024-03-29T11:31:41.881086+0000","last_peered":"2024-03-29T11:31:41.881086+0000","last_clean":"2024-03-29T11:31:41.881086+0000","last_became_active":"2024-03-29T11:31:40.889385+0000","last_became_peered":"2024-03-29T11:31:40.889385+0000","last_unstale":"2024-03-29T11:31:41.881086+0000","last_undegraded":"2024-03-29T11:31:41.881086+0000","last_fullsized":"2024-03-29T11:31:41.881086+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:28:36.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870273+0000","last_change":"2024-03-29T11:31:26.843972+0000","last_active":"2024-03-29T11:31:40.870273+0000","last_peered":"2024-03-29T11:31:40.870273+0000","last_clean":"2024-03-29T11:31:40.870273+0000","last_became_active":"2024-03-29T11:31:26.843713+0000","last_became_peered":"2024-03-29T11:31:26.843713+0000","last_unstale":"2024-03-29T11:31:40.870273+0000","last_undegraded":"2024-03-29T11:31:40.870273+0000","last_fullsized":"2024-03-29T11:31:40.870273+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950667+0000","last_change":"2024-03-29T11:31:40.908373+0000","last_active":"2024-03-29T11:31:41.950667+0000","last_peered":"2024-03-29T11:31:41.950667+0000","last_clean":"2024-03-29T11:31:41.950667+0000","last_became_active":"2024-03-29T11:31:40.906929+0000","last_became_peered":"2024-03-29T11:31:40.906929+0000","last_unstale":"2024-03-29T11:31:41.950667+0000","last_undegraded":"2024-03-29T11:31:41.950667+0000","last_fullsized":"2024-03-29T11:31:41.950667+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:28:36.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873322+0000","last_change":"2024-03-29T11:31:26.848452+0000","last_active":"2024-03-29T11:31:40.873322+0000","last_peered":"2024-03-29T11:31:40.873322+0000","last_clean":"2024-03-29T11:31:40.873322+0000","last_became_active":"2024-03-29T11:31:26.846605+0000","last_became_peered":"2024-03-29T11:31:26.846605+0000","last_unstale":"2024-03-29T11:31:40.873322+0000","last_undegraded":"2024-03-29T11:31:40.873322+0000","last_fullsized":"2024-03-29T11:31:40.873322+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:57.163908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.878649+0000","last_change":"2024-03-29T11:31:40.905176+0000","last_active":"2024-03-29T11:31:41.878649+0000","last_peered":"2024-03-29T11:31:41.878649+0000","last_clean":"2024-03-29T11:31:41.878649+0000","last_became_active":"2024-03-29T11:31:40.904390+0000","last_became_peered":"2024-03-29T11:31:40.904390+0000","last_unstale":"2024-03-29T11:31:41.878649+0000","last_undegraded":"2024-03-29T11:31:41.878649+0000","last_fullsized":"2024-03-29T11:31:41.878649+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:14:20.031001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875645+0000","last_change":"2024-03-29T11:31:26.848377+0000","last_active":"2024-03-29T11:31:40.875645+0000","last_peered":"2024-03-29T11:31:40.875645+0000","last_clean":"2024-03-29T11:31:40.875645+0000","last_became_active":"2024-03-29T11:31:26.847564+0000","last_became_peered":"2024-03-29T11:31:26.847564+0000","last_unstale":"2024-03-29T11:31:40.875645+0000","last_undegraded":"2024-03-29T11:31:40.875645+0000","last_fullsized":"2024-03-29T11:31:40.875645+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:40:17.980944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949830+0000","last_change":"2024-03-29T11:31:40.898642+0000","last_active":"2024-03-29T11:31:41.949830+0000","last_peered":"2024-03-29T11:31:41.949830+0000","last_clean":"2024-03-29T11:31:41.949830+0000","last_became_active":"2024-03-29T11:31:40.898101+0000","last_became_peered":"2024-03-29T11:31:40.898101+0000","last_unstale":"2024-03-29T11:31:41.949830+0000","last_undegraded":"2024-03-29T11:31:41.949830+0000","last_fullsized":"2024-03-29T11:31:41.949830+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:32:36.867108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876930+0000","last_change":"2024-03-29T11:31:26.848052+0000","last_active":"2024-03-29T11:31:40.876930+0000","last_peered":"2024-03-29T11:31:40.876930+0000","last_clean":"2024-03-29T11:31:40.876930+0000","last_became_active":"2024-03-29T11:31:26.847789+0000","last_became_peered":"2024-03-29T11:31:26.847789+0000","last_unstale":"2024-03-29T11:31:40.876930+0000","last_undegraded":"2024-03-29T11:31:40.876930+0000","last_fullsized":"2024-03-29T11:31:40.876930+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:23:23.236057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949909+0000","last_change":"2024-03-29T11:31:40.898685+0000","last_active":"2024-03-29T11:31:41.949909+0000","last_peered":"2024-03-29T11:31:41.949909+0000","last_clean":"2024-03-29T11:31:41.949909+0000","last_became_active":"2024-03-29T11:31:40.898419+0000","last_became_peered":"2024-03-29T11:31:40.898419+0000","last_unstale":"2024-03-29T11:31:41.949909+0000","last_undegraded":"2024-03-29T11:31:41.949909+0000","last_fullsized":"2024-03-29T11:31:41.949909+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:46.656502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884150+0000","last_change":"2024-03-29T11:31:26.841262+0000","last_active":"2024-03-29T11:31:40.884150+0000","last_peered":"2024-03-29T11:31:40.884150+0000","last_clean":"2024-03-29T11:31:40.884150+0000","last_became_active":"2024-03-29T11:31:26.841004+0000","last_became_peered":"2024-03-29T11:31:26.841004+0000","last_unstale":"2024-03-29T11:31:40.884150+0000","last_undegraded":"2024-03-29T11:31:40.884150+0000","last_fullsized":"2024-03-29T11:31:40.884150+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:54.107881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949636+0000","last_change":"2024-03-29T11:31:40.903665+0000","last_active":"2024-03-29T11:31:41.949636+0000","last_peered":"2024-03-29T11:31:41.949636+0000","last_clean":"2024-03-29T11:31:41.949636+0000","last_became_active":"2024-03-29T11:31:40.903409+0000","last_became_peered":"2024-03-29T11:31:40.903409+0000","last_unstale":"2024-03-29T11:31:41.949636+0000","last_undegraded":"2024-03-29T11:31:41.949636+0000","last_fullsized":"2024-03-29T11:31:41.949636+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:13.874339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876591+0000","last_change":"2024-03-29T11:31:26.853942+0000","last_active":"2024-03-29T11:31:40.876591+0000","last_peered":"2024-03-29T11:31:40.876591+0000","last_clean":"2024-03-29T11:31:40.876591+0000","last_became_active":"2024-03-29T11:31:26.853648+0000","last_became_peered":"2024-03-29T11:31:26.853648+0000","last_unstale":"2024-03-29T11:31:40.876591+0000","last_undegraded":"2024-03-29T11:31:40.876591+0000","last_fullsized":"2024-03-29T11:31:40.876591+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:24.230359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949341+0000","last_change":"2024-03-29T11:31:40.887123+0000","last_active":"2024-03-29T11:31:41.949341+0000","last_peered":"2024-03-29T11:31:41.949341+0000","last_clean":"2024-03-29T11:31:41.949341+0000","last_became_active":"2024-03-29T11:31:40.886835+0000","last_became_peered":"2024-03-29T11:31:40.886835+0000","last_unstale":"2024-03-29T11:31:41.949341+0000","last_undegraded":"2024-03-29T11:31:41.949341+0000","last_fullsized":"2024-03-29T11:31:41.949341+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:01:54.265645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.889378+0000","last_change":"2024-03-29T11:31:26.851850+0000","last_active":"2024-03-29T11:31:40.889378+0000","last_peered":"2024-03-29T11:31:40.889378+0000","last_clean":"2024-03-29T11:31:40.889378+0000","last_became_active":"2024-03-29T11:31:26.848888+0000","last_became_peered":"2024-03-29T11:31:26.848888+0000","last_unstale":"2024-03-29T11:31:40.889378+0000","last_undegraded":"2024-03-29T11:31:40.889378+0000","last_fullsized":"2024-03-29T11:31:40.889378+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:26:36.183185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951339+0000","last_change":"2024-03-29T11:31:40.901556+0000","last_active":"2024-03-29T11:31:41.951339+0000","last_peered":"2024-03-29T11:31:41.951339+0000","last_clean":"2024-03-29T11:31:41.951339+0000","last_became_active":"2024-03-29T11:31:40.901278+0000","last_became_peered":"2024-03-29T11:31:40.901278+0000","last_unstale":"2024-03-29T11:31:41.951339+0000","last_undegraded":"2024-03-29T11:31:41.951339+0000","last_fullsized":"2024-03-29T11:31:41.951339+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:09:49.649528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884142+0000","last_change":"2024-03-29T11:31:26.833794+0000","last_active":"2024-03-29T11:31:40.884142+0000","last_peered":"2024-03-29T11:31:40.884142+0000","last_clean":"2024-03-29T11:31:40.884142+0000","last_became_active":"2024-03-29T11:31:26.833440+0000","last_became_peered":"2024-03-29T11:31:26.833440+0000","last_unstale":"2024-03-29T11:31:40.884142+0000","last_undegraded":"2024-03-29T11:31:40.884142+0000","last_fullsized":"2024-03-29T11:31:40.884142+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:07.336827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880321+0000","last_change":"2024-03-29T11:31:40.904700+0000","last_active":"2024-03-29T11:31:41.880321+0000","last_peered":"2024-03-29T11:31:41.880321+0000","last_clean":"2024-03-29T11:31:41.880321+0000","last_became_active":"2024-03-29T11:31:40.904009+0000","last_became_peered":"2024-03-29T11:31:40.904009+0000","last_unstale":"2024-03-29T11:31:41.880321+0000","last_undegraded":"2024-03-29T11:31:41.880321+0000","last_fullsized":"2024-03-29T11:31:41.880321+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:46.539890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870188+0000","last_change":"2024-03-29T11:31:26.836892+0000","last_active":"2024-03-29T11:31:40.870188+0000","last_peered":"2024-03-29T11:31:40.870188+0000","last_clean":"2024-03-29T11:31:40.870188+0000","last_became_active":"2024-03-29T11:31:26.832344+0000","last_became_peered":"2024-03-29T11:31:26.832344+0000","last_unstale":"2024-03-29T11:31:40.870188+0000","last_undegraded":"2024-03-29T11:31:40.870188+0000","last_fullsized":"2024-03-29T11:31:40.870188+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:27:29.490312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950322+0000","last_change":"2024-03-29T11:31:40.903357+0000","last_active":"2024-03-29T11:31:41.950322+0000","last_peered":"2024-03-29T11:31:41.950322+0000","last_clean":"2024-03-29T11:31:41.950322+0000","last_became_active":"2024-03-29T11:31:40.903091+0000","last_became_peered":"2024-03-29T11:31:40.903091+0000","last_unstale":"2024-03-29T11:31:41.950322+0000","last_undegraded":"2024-03-29T11:31:41.950322+0000","last_fullsized":"2024-03-29T11:31:41.950322+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:26:19.372364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.892440+0000","last_change":"2024-03-29T11:31:26.836146+0000","last_active":"2024-03-29T11:31:40.892440+0000","last_peered":"2024-03-29T11:31:40.892440+0000","last_clean":"2024-03-29T11:31:40.892440+0000","last_became_active":"2024-03-29T11:31:26.835185+0000","last_became_peered":"2024-03-29T11:31:26.835185+0000","last_unstale":"2024-03-29T11:31:40.892440+0000","last_undegraded":"2024-03-29T11:31:40.892440+0000","last_fullsized":"2024-03-29T11:31:40.892440+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:15:49.012984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880903+0000","last_change":"2024-03-29T11:31:40.905024+0000","last_active":"2024-03-29T11:31:41.880903+0000","last_peered":"2024-03-29T11:31:41.880903+0000","last_clean":"2024-03-29T11:31:41.880903+0000","last_became_active":"2024-03-29T11:31:40.904676+0000","last_became_peered":"2024-03-29T11:31:40.904676+0000","last_unstale":"2024-03-29T11:31:41.880903+0000","last_undegraded":"2024-03-29T11:31:41.880903+0000","last_fullsized":"2024-03-29T11:31:41.880903+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:29:11.480583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.892264+0000","last_change":"2024-03-29T11:31:26.841559+0000","last_active":"2024-03-29T11:31:40.892264+0000","last_peered":"2024-03-29T11:31:40.892264+0000","last_clean":"2024-03-29T11:31:40.892264+0000","last_became_active":"2024-03-29T11:31:26.841323+0000","last_became_peered":"2024-03-29T11:31:26.841323+0000","last_unstale":"2024-03-29T11:31:40.892264+0000","last_undegraded":"2024-03-29T11:31:40.892264+0000","last_fullsized":"2024-03-29T11:31:40.892264+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:40:17.980944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.873796+0000","last_change":"2024-03-29T11:31:40.896558+0000","last_active":"2024-03-29T11:31:41.873796+0000","last_peered":"2024-03-29T11:31:41.873796+0000","last_clean":"2024-03-29T11:31:41.873796+0000","last_became_active":"2024-03-29T11:31:40.896296+0000","last_became_peered":"2024-03-29T11:31:40.896296+0000","last_unstale":"2024-03-29T11:31:41.873796+0000","last_undegraded":"2024-03-29T11:31:41.873796+0000","last_fullsized":"2024-03-29T11:31:41.873796+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:27.695748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.874842+0000","last_change":"2024-03-29T11:31:26.852752+0000","last_active":"2024-03-29T11:31:40.874842+0000","last_peered":"2024-03-29T11:31:40.874842+0000","last_clean":"2024-03-29T11:31:40.874842+0000","last_became_active":"2024-03-29T11:31:26.852491+0000","last_became_peered":"2024-03-29T11:31:26.852491+0000","last_unstale":"2024-03-29T11:31:40.874842+0000","last_undegraded":"2024-03-29T11:31:40.874842+0000","last_fullsized":"2024-03-29T11:31:40.874842+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:32.613551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950067+0000","last_change":"2024-03-29T11:31:40.885411+0000","last_active":"2024-03-29T11:31:41.950067+0000","last_peered":"2024-03-29T11:31:41.950067+0000","last_clean":"2024-03-29T11:31:41.950067+0000","last_became_active":"2024-03-29T11:31:40.885167+0000","last_became_peered":"2024-03-29T11:31:40.885167+0000","last_unstale":"2024-03-29T11:31:41.950067+0000","last_undegraded":"2024-03-29T11:31:41.950067+0000","last_fullsized":"2024-03-29T11:31:41.950067+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:16:12.619950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884071+0000","last_change":"2024-03-29T11:31:26.851998+0000","last_active":"2024-03-29T11:31:40.884071+0000","last_peered":"2024-03-29T11:31:40.884071+0000","last_clean":"2024-03-29T11:31:40.884071+0000","last_became_active":"2024-03-29T11:31:26.850781+0000","last_became_peered":"2024-03-29T11:31:26.850781+0000","last_unstale":"2024-03-29T11:31:40.884071+0000","last_undegraded":"2024-03-29T11:31:40.884071+0000","last_fullsized":"2024-03-29T11:31:40.884071+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:57.876055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949090+0000","last_change":"2024-03-29T11:31:40.903133+0000","last_active":"2024-03-29T11:31:41.949090+0000","last_peered":"2024-03-29T11:31:41.949090+0000","last_clean":"2024-03-29T11:31:41.949090+0000","last_became_active":"2024-03-29T11:31:40.902862+0000","last_became_peered":"2024-03-29T11:31:40.902862+0000","last_unstale":"2024-03-29T11:31:41.949090+0000","last_undegraded":"2024-03-29T11:31:41.949090+0000","last_fullsized":"2024-03-29T11:31:41.949090+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:43.064595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884404+0000","last_change":"2024-03-29T11:31:26.848512+0000","last_active":"2024-03-29T11:31:40.884404+0000","last_peered":"2024-03-29T11:31:40.884404+0000","last_clean":"2024-03-29T11:31:40.884404+0000","last_became_active":"2024-03-29T11:31:26.846361+0000","last_became_peered":"2024-03-29T11:31:26.846361+0000","last_unstale":"2024-03-29T11:31:40.884404+0000","last_undegraded":"2024-03-29T11:31:40.884404+0000","last_fullsized":"2024-03-29T11:31:40.884404+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:49:25.037675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879746+0000","last_change":"2024-03-29T11:31:40.882526+0000","last_active":"2024-03-29T11:31:41.879746+0000","last_peered":"2024-03-29T11:31:41.879746+0000","last_clean":"2024-03-29T11:31:41.879746+0000","last_became_active":"2024-03-29T11:31:40.882222+0000","last_became_peered":"2024-03-29T11:31:40.882222+0000","last_unstale":"2024-03-29T11:31:41.879746+0000","last_undegraded":"2024-03-29T11:31:41.879746+0000","last_fullsized":"2024-03-29T11:31:41.879746+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:07.759471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872754+0000","last_change":"2024-03-29T11:31:26.850587+0000","last_active":"2024-03-29T11:31:40.872754+0000","last_peered":"2024-03-29T11:31:40.872754+0000","last_clean":"2024-03-29T11:31:40.872754+0000","last_became_active":"2024-03-29T11:31:26.850150+0000","last_became_peered":"2024-03-29T11:31:26.850150+0000","last_unstale":"2024-03-29T11:31:40.872754+0000","last_undegraded":"2024-03-29T11:31:40.872754+0000","last_fullsized":"2024-03-29T11:31:40.872754+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:49.204368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575997,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52580,"kb_used_data":3412,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3493888,"data_stored":2001990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":50330060},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.51200000000000001,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":0.88400000000000001,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.747},"min":{"1min":0.56499999999999995,"5min":0.46800000000000003,"15min":0.40300000000000002},"max":{"1min":0.98899999999999999,"5min":1.119,"15min":1.119},"last":0.66600000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.875,"5min":1.002,"15min":1.478},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74299999999999999,"15min":0.76900000000000002},"min":{"1min":0.55600000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91200000000000003,"5min":0.98699999999999999,"15min":1.393},"last":0.68300000000000005}]},{"osd":2,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72299999999999998,"15min":0.75700000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.89800000000000002,"5min":1.089,"15min":1.284},"last":0.628},{"interface":"front","average":{"1min":0.747,"5min":0.754,"15min":0.77500000000000002},"min":{"1min":0.57099999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.92700000000000005,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.73799999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.74399999999999999,"15min":0.76200000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.85499999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.77600000000000002,"15min":0.79000000000000004},"min":{"1min":0.57699999999999996,"5min":0.55900000000000005,"15min":0.55300000000000005},"max":{"1min":0.92200000000000004,"5min":1.071,"15min":1.3169999999999999},"last":0.78600000000000003}]},{"osd":4,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83299999999999996,"5min":1.024,"15min":1.3779999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91500000000000004,"5min":0.93500000000000005,"15min":1.335},"last":0.70399999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69499999999999995,"15min":0.68000000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":0.95899999999999996,"5min":0.98399999999999999,"15min":1.347},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73499999999999999,"15min":0.71699999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93600000000000005,"5min":1.0169999999999999,"15min":1.1759999999999999},"last":0.88}]},{"osd":6,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0309999999999999,"5min":1.0549999999999999,"15min":1.2070000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76200000000000001,"15min":0.73899999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.435},"max":{"1min":0.97899999999999998,"5min":1.071,"15min":1.071},"last":0.77200000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575998,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46256,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325584,"statfs":{"total":96636764160,"available":96589398016,"internally_reserved":0,"allocated":2908160,"data_stored":1416624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44431815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71699999999999997,"15min":0.73699999999999999},"min":{"1min":0.45900000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86199999999999999,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.73199999999999998,"15min":0.76100000000000001},"min":{"1min":0.52700000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.84399999999999997,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.71099999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.71099999999999997,"15min":0.73699999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":0.83399999999999996,"5min":1.98,"15min":1.98},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.745,"15min":0.77300000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.92500000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.5,"5min":0.498,"15min":0.497},"max":{"1min":0.88700000000000001,"5min":1.669,"15min":1.669},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.55700000000000005,"5min":0.49099999999999999,"15min":0.46200000000000002},"max":{"1min":0.879,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.58299999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.75600000000000001,"15min":0.77300000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.91000000000000003,"5min":2.52,"15min":2.52},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.78800000000000003,"15min":0.80300000000000005},"min":{"1min":0.57099999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.996,"5min":1.877,"15min":1.877},"last":0.64000000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.62,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":0.76500000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.70899999999999996,"15min":0.70699999999999996},"min":{"1min":0.504,"5min":0.47399999999999998,"15min":0.45900000000000002},"max":{"1min":0.90000000000000002,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":0.66600000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.47299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90200000000000002,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70999999999999996,"15min":0.71299999999999997},"min":{"1min":0.54300000000000004,"5min":0.46999999999999997,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":1.925,"15min":1.925},"last":0.752}]},{"osd":7,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77100000000000002,"15min":0.76700000000000002},"min":{"1min":0.58099999999999996,"5min":0.54800000000000004,"15min":0.52700000000000002},"max":{"1min":1.0149999999999999,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.74299999999999999,"15min":0.74099999999999999},"min":{"1min":0.54600000000000004,"5min":0.504,"15min":0.47499999999999998},"max":{"1min":0.95099999999999996,"5min":1.726,"15min":1.726},"last":0.82099999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575996,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60204,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311636,"statfs":{"total":96636764160,"available":96575115264,"internally_reserved":0,"allocated":2908160,"data_stored":1416132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.46899999999999997},"min":{"1min":0.33900000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.81100000000000005},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.505}]},{"osd":2,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.41999999999999998,"5min":0.35299999999999998,"15min":0.34799999999999998},"max":{"1min":0.75900000000000001,"5min":0.81399999999999995,"15min":0.90300000000000002},"last":0.48099999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.42999999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77000000000000002,"5min":0.86199999999999999,"15min":0.995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.49199999999999999,"5min":0.38900000000000001,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.83999999999999997,"15min":0.878},"last":0.45900000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75900000000000001,"15min":0.79200000000000004},"min":{"1min":0.55100000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.995,"5min":1.153,"15min":1.3660000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.78300000000000003,"15min":0.80500000000000005},"min":{"1min":0.60199999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.96399999999999997,"5min":1.304,"15min":1.304},"last":0.64900000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80100000000000005,"15min":0.82199999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.48199999999999998},"max":{"1min":0.97799999999999998,"5min":1.3380000000000001,"15min":1.3720000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.80900000000000005,"15min":0.83799999999999997},"min":{"1min":0.65300000000000002,"5min":0.59199999999999997,"15min":0.59099999999999997},"max":{"1min":1.0549999999999999,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.81699999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.79000000000000004,"15min":0.82299999999999995},"min":{"1min":0.67100000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.95599999999999996,"5min":1.1890000000000001,"15min":1.1919999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.84199999999999997,"15min":0.85499999999999998},"min":{"1min":0.61899999999999999,"5min":0.60999999999999999,"15min":0.53100000000000003},"max":{"1min":1.0569999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"last":0.92500000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.65900000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.034,"5min":1.327,"15min":1.3540000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.84399999999999997,"15min":0.87},"min":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.94199999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575998,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60764,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311076,"statfs":{"total":96636764160,"available":96574541824,"internally_reserved":0,"allocated":3485696,"data_stored":1998858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.45300000000000001,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65300000000000002,"5min":0.95899999999999996,"15min":1.087},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77400000000000002,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84999999999999998,"5min":1.028,"15min":1.5289999999999999},"last":0.5},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51100000000000001,"15min":0.52000000000000002},"min":{"1min":0.372,"5min":0.317,"15min":0.317},"max":{"1min":0.58299999999999996,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81000000000000005,"5min":1.222,"15min":1.222},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.42199999999999999},"max":{"1min":0.83499999999999996,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.83599999999999997,"15min":0.84999999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.091,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.028,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81000000000000005,"15min":0.82199999999999995},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1499999999999999,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.69799999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84999999999999998,"15min":0.85899999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0700000000000001,"5min":1.343,"15min":1.6240000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.873,"15min":0.88800000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.171,"5min":1.3240000000000001,"15min":1.337},"last":0.66500000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97899999999999998,"5min":1.115,"15min":1.214},"last":0.752},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.85799999999999998,"15min":0.874},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.204,"5min":1.365,"15min":1.365},"last":0.63200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575997,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47912,"kb_used_data":2836,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323928,"statfs":{"total":96636764160,"available":96587702272,"internally_reserved":0,"allocated":2904064,"data_stored":1410367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.377,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67800000000000005,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73999999999999999,"5min":0.75600000000000001,"15min":0.96799999999999997},"last":0.52900000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.79000000000000004,"15min":0.80700000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.872,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76700000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.44,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76300000000000001,"5min":1.038,"15min":1.038},"last":0.44}]},{"osd":4,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79100000000000004,"15min":0.81100000000000005},"min":{"1min":0.61599999999999999,"5min":0.57899999999999996,"15min":0.55100000000000005},"max":{"1min":0.89200000000000002,"5min":1.169,"15min":1.1970000000000001},"last":0.626},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78500000000000003,"15min":0.81599999999999995},"min":{"1min":0.64800000000000002,"5min":0.57099999999999995,"15min":0.52400000000000002},"max":{"1min":1.157,"5min":1.244,"15min":1.244},"last":0.61199999999999999}]},{"osd":5,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.61799999999999999,"5min":0.59799999999999998,"15min":0.53400000000000003},"max":{"1min":1.1200000000000001,"5min":1.256,"15min":1.256},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81299999999999994,"15min":0.82999999999999996},"min":{"1min":0.72799999999999998,"5min":0.57799999999999996,"15min":0.56000000000000005},"max":{"1min":0.97499999999999998,"5min":1.24,"15min":1.3140000000000001},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82799999999999996,"15min":0.85199999999999998},"min":{"1min":0.64800000000000002,"5min":0.58999999999999997,"15min":0.57799999999999996},"max":{"1min":1.105,"5min":1.1459999999999999,"15min":1.24},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.623,"5min":0.58599999999999997,"15min":0.58199999999999996},"max":{"1min":1.0880000000000001,"5min":1.1879999999999999,"15min":1.329},"last":0.64800000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.877,"15min":0.90000000000000002},"min":{"1min":0.67400000000000004,"5min":0.66800000000000004,"15min":0.64600000000000002},"max":{"1min":1.0209999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88900000000000001,"15min":0.91200000000000003},"min":{"1min":0.68300000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.071,"5min":1.304,"15min":1.304},"last":0.59699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575997,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47916,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323924,"statfs":{"total":96636764160,"available":96587698176,"internally_reserved":0,"allocated":2912256,"data_stored":1410563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.47499999999999998,"15min":0.47099999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.36899999999999999,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70099999999999996,"5min":0.72099999999999997,"15min":0.748},"last":0.373},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54400000000000004,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":1.046},"last":0.56499999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.96699999999999997},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.66200000000000003},"min":{"1min":0.53200000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86399999999999999,"5min":1.0049999999999999,"15min":1.01},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.80200000000000005},"min":{"1min":0.56299999999999994,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.2549999999999999,"5min":1.2609999999999999,"15min":1.2889999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81799999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1599999999999999,"5min":1.306,"15min":1.306},"last":0.68999999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80400000000000005,"15min":0.81599999999999995},"min":{"1min":0.56999999999999995,"5min":0.53800000000000003,"15min":0.502},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.62},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.56000000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.214,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.79600000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64900000000000002,"5min":0.64400000000000002,"15min":0.57799999999999996},"max":{"1min":1.196,"5min":1.282,"15min":1.288},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.84099999999999997,"15min":0.84699999999999998},"min":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.86399999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.85799999999999998,"15min":0.86699999999999999},"min":{"1min":0.68999999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3100000000000001,"5min":1.331,"15min":1.331},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.89500000000000002,"15min":0.89600000000000002},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.57899999999999996},"max":{"1min":1.246,"5min":1.286,"15min":1.286},"last":0.71799999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575998,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60204,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311636,"statfs":{"total":96636764160,"available":96575115264,"internally_reserved":0,"allocated":2908160,"data_stored":1412238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.68999999999999995,"15min":0.72199999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.39900000000000002},"max":{"1min":0.875,"5min":0.95499999999999996,"15min":1.0509999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69199999999999995,"15min":0.72499999999999998},"min":{"1min":0.45000000000000001,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.89500000000000002,"5min":0.99299999999999999,"15min":1.1459999999999999},"last":0.59399999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.70699999999999996,"15min":0.748},"min":{"1min":0.497,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":0.89100000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"last":0.436},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.70899999999999996,"15min":0.745},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.877,"5min":0.94899999999999995,"15min":1.002},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.79400000000000004,"15min":0.82799999999999996},"min":{"1min":0.52100000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.89500000000000002,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.79100000000000004,"15min":0.81399999999999995},"min":{"1min":0.54100000000000004,"5min":0.496,"15min":0.496},"max":{"1min":0.93799999999999994,"5min":1.224,"15min":1.224},"last":0.626}]},{"osd":3,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.72599999999999998,"15min":0.76400000000000001},"min":{"1min":0.52200000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.96699999999999997,"5min":1.0580000000000001,"15min":1.181},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76500000000000001,"15min":0.77100000000000002},"min":{"1min":0.55600000000000005,"5min":0.55300000000000005,"15min":0.48699999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.2490000000000001},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.63500000000000001,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.47199999999999998,"5min":0.41599999999999998,"15min":0.39700000000000002},"max":{"1min":0.86299999999999999,"5min":1.0920000000000001,"15min":1.169},"last":0.52200000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":1.056,"5min":1.056,"15min":1.0800000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.53200000000000003,"5min":0.48199999999999998,"15min":0.47699999999999998},"max":{"1min":0.93500000000000005,"5min":1.0089999999999999,"15min":1.028},"last":0.66800000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.56200000000000006,"5min":0.501,"15min":0.436},"max":{"1min":0.92100000000000004,"5min":1.0389999999999999,"15min":1.518},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.81999999999999995,"15min":0.82599999999999996},"min":{"1min":0.60299999999999998,"5min":0.58499999999999996,"15min":0.54400000000000004},"max":{"1min":1.0900000000000001,"5min":1.1619999999999999,"15min":1.3440000000000001},"last":0.65900000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575997,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42140,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329700,"statfs":{"total":96636764160,"available":96593612800,"internally_reserved":0,"allocated":2899968,"data_stored":1416568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70699999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83899999999999997,"5min":0.94199999999999995,"15min":0.95999999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.73199999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90000000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.80000000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.75900000000000001,"15min":0.76500000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.73699999999999999,"15min":0.74199999999999999},"min":{"1min":0.52200000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.66600000000000004}]},{"osd":2,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.76800000000000002,"15min":0.75900000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.72899999999999998,"15min":0.748},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.88400000000000001,"5min":0.98499999999999999,"15min":1.0129999999999999},"last":0.52500000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.77900000000000003,"15min":0.77400000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.444,"5min":3.444,"15min":3.444},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.046},"last":0.71299999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.60599999999999998},"min":{"1min":0.44400000000000001,"5min":0.41799999999999998,"15min":0.38700000000000001},"max":{"1min":0.88,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.40600000000000003},"max":{"1min":0.84799999999999998,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.64600000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68799999999999994,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90300000000000002,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.748},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72099999999999997,"15min":0.69799999999999995},"min":{"1min":0.55900000000000005,"5min":0.51000000000000001,"15min":0.42499999999999999},"max":{"1min":0.83499999999999996,"5min":1.107,"15min":1.107},"last":0.68500000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.74199999999999999,"15min":0.72199999999999998},"min":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.42299999999999999},"max":{"1min":0.92200000000000004,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.79300000000000004,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.98199999999999998,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.81100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:31:45.336 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:31:45.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.571+0000 7f050a800700 1 Processor -- start 2024-03-29T11:31:45.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.571+0000 7f050a800700 1 -- start start 2024-03-29T11:31:45.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.572+0000 7f050a800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05040d18b0 0x7f05040cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.572+0000 7f050a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0504161590 0x7f0504161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.572+0000 7f050a800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05040ceda0 con 0x7f0504160f10 2024-03-29T11:31:45.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.572+0000 7f050a800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05040d1d30 con 0x7f05040d18b0 2024-03-29T11:31:45.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.572+0000 7f050a800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0504161f60 con 0x7f0504161590 2024-03-29T11:31:45.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.572+0000 7f0502a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05040d18b0 0x7f05040cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.572+0000 7f0502a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05040d18b0 0x7f05040cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39576/0 (socket says 172.21.15.12:39576) 2024-03-29T11:31:45.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.572+0000 7f0502a00700 1 -- 172.21.15.12:0/2249624462 learned_addr learned my addr 172.21.15.12:0/2249624462 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:45.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.573+0000 7f0502a00700 1 -- 172.21.15.12:0/2249624462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0504161590 msgr2=0x7f0504161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.573+0000 7f0502a00700 1 --2- 172.21.15.12:0/2249624462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0504161590 0x7f0504161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.573+0000 7f0502a00700 1 -- 172.21.15.12:0/2249624462 >> v1:172.21.15.159:6789/0 conn(0x7f0504160f10 legacy=0x7f05040d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.573+0000 7f0502a00700 1 -- 172.21.15.12:0/2249624462 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05041612f0 con 0x7f05040d18b0 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.573+0000 7f0502a00700 1 --2- 172.21.15.12:0/2249624462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 0x7f05040cf230 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009920 tx=0x7f04f0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641d9c021765cd84 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.573+0000 7f0502000700 1 -- 172.21.15.12:0/2249624462 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0004230 con 0x7f05040d18b0 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.573+0000 7f0502000700 1 -- 172.21.15.12:0/2249624462 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f0004390 con 0x7f05040d18b0 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.573+0000 7f0502000700 1 -- 172.21.15.12:0/2249624462 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0009e70 con 0x7f05040d18b0 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.574+0000 7f050a800700 1 -- 172.21.15.12:0/2249624462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 msgr2=0x7f05040cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.574+0000 7f050a800700 1 --2- 172.21.15.12:0/2249624462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 0x7f05040cf230 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009920 tx=0x7f04f0009c30 comp rx=0 tx=0).stop 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.574+0000 7f050a800700 1 -- 172.21.15.12:0/2249624462 shutdown_connections 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.574+0000 7f050a800700 1 --2- 172.21.15.12:0/2249624462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0504161590 0x7f0504161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.574+0000 7f050a800700 1 --2- 172.21.15.12:0/2249624462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 0x7f05040cf230 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.574+0000 7f050a800700 1 -- 172.21.15.12:0/2249624462 >> 172.21.15.12:0/2249624462 conn(0x7f050408aff0 msgr2=0x7f05040d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:45.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.574+0000 7f050a800700 1 -- 172.21.15.12:0/2249624462 shutdown_connections 2024-03-29T11:31:45.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.574+0000 7f050a800700 1 -- 172.21.15.12:0/2249624462 wait complete. 2024-03-29T11:31:45.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.576+0000 7f050a800700 1 Processor -- start 2024-03-29T11:31:45.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.577+0000 7f050a800700 1 -- start start 2024-03-29T11:31:45.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f050a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 0x7f05040c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f050a800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0504160f10 0x7f05040b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f050a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0504161590 0x7f050414a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f050a800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05040b05c0 con 0x7f05040d18b0 2024-03-29T11:31:45.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f050a800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050414a1e0 con 0x7f0504160f10 2024-03-29T11:31:45.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f050a800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05041495d0 con 0x7f0504161590 2024-03-29T11:31:45.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f0503400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 0x7f05040c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f0503400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 0x7f05040c9b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39590/0 (socket says 172.21.15.12:39590) 2024-03-29T11:31:45.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f0503400700 1 -- 172.21.15.12:0/4018554071 learned_addr learned my addr 172.21.15.12:0/4018554071 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:45.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f0503e00700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0504161590 0x7f050414a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.578+0000 7f0502a00700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0504160f10 0x7f05040b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f0503400700 1 -- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0504161590 msgr2=0x7f050414a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f0503400700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0504161590 0x7f050414a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f0503400700 1 -- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0504160f10 msgr2=0x7f05040b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f0503400700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0504160f10 0x7f05040b1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f0503400700 1 -- 172.21.15.12:0/4018554071 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f0009550 con 0x7f05040d18b0 2024-03-29T11:31:45.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f0503400700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 0x7f05040c9b10 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f04f400b470 tx=0x7f04f400b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:45.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f04fb400700 1 -- 172.21.15.12:0/4018554071 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f4011830 con 0x7f05040d18b0 2024-03-29T11:31:45.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f04fb400700 1 -- 172.21.15.12:0/4018554071 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f40119b0 con 0x7f05040d18b0 2024-03-29T11:31:45.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f04fb400700 1 -- 172.21.15.12:0/4018554071 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f4011d60 con 0x7f05040d18b0 2024-03-29T11:31:45.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f050a800700 1 -- 172.21.15.12:0/4018554071 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05041498b0 con 0x7f05040d18b0 2024-03-29T11:31:45.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.579+0000 7f050a800700 1 -- 172.21.15.12:0/4018554071 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0504149e70 con 0x7f05040d18b0 2024-03-29T11:31:45.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.580+0000 7f050a800700 1 -- 172.21.15.12:0/4018554071 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050414a4c0 con 0x7f05040d18b0 2024-03-29T11:31:45.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.582+0000 7f04fb400700 1 -- 172.21.15.12:0/4018554071 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f04f401a400 con 0x7f05040d18b0 2024-03-29T11:31:45.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.582+0000 7f04fb400700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04e406adb0 0x7f04e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:45.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.583+0000 7f04fb400700 1 -- 172.21.15.12:0/4018554071 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f04f4089780 con 0x7f05040d18b0 2024-03-29T11:31:45.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.583+0000 7f0502a00700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04e406adb0 0x7f04e406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:45.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.584+0000 7f0502a00700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04e406adb0 0x7f04e406d270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009130 tx=0x7f04f0008030 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:45.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.587+0000 7f04fb400700 1 -- 172.21.15.12:0/4018554071 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f04f4054400 con 0x7f05040d18b0 2024-03-29T11:31:45.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.801+0000 7f050a800700 1 -- 172.21.15.12:0/4018554071 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05040ce750 con 0x7f04e406adb0 2024-03-29T11:31:45.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.816+0000 7f04fb400700 1 -- 172.21.15.12:0/4018554071 <== mgr.10459 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528658 (secure 0 0 0) 0x7f04f000b7e0 con 0x7f04e406adb0 2024-03-29T11:31:45.818 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:45.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.822+0000 7f050a800700 1 -- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04e406adb0 msgr2=0x7f04e406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.822+0000 7f050a800700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04e406adb0 0x7f04e406d270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009130 tx=0x7f04f0008030 comp rx=0 tx=0).stop 2024-03-29T11:31:45.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.822+0000 7f050a800700 1 -- 172.21.15.12:0/4018554071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 msgr2=0x7f05040c9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:45.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.822+0000 7f050a800700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 0x7f05040c9b10 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f04f400b470 tx=0x7f04f400b780 comp rx=0 tx=0).stop 2024-03-29T11:31:45.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.822+0000 7f050a800700 1 -- 172.21.15.12:0/4018554071 shutdown_connections 2024-03-29T11:31:45.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.822+0000 7f050a800700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04e406adb0 0x7f04e406d270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.822+0000 7f050a800700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0504161590 0x7f050414a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.822+0000 7f050a800700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05040d18b0 0x7f05040c9b10 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.822+0000 7f050a800700 1 --2- 172.21.15.12:0/4018554071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0504160f10 0x7f05040b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:45.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.822+0000 7f050a800700 1 -- 172.21.15.12:0/4018554071 >> 172.21.15.12:0/4018554071 conn(0x7f050408aff0 msgr2=0x7f05040cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:45.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.823+0000 7f050a800700 1 -- 172.21.15.12:0/4018554071 shutdown_connections 2024-03-29T11:31:45.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:45.823+0000 7f050a800700 1 -- 172.21.15.12:0/4018554071 wait complete. 2024-03-29T11:31:45.824 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:31:45.836 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-29T11:31:45.171760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596333,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":702,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":417976,"kb_used_data":23848,"kb_used_omap":12,"kb_used_meta":393971,"kb_avail":754556744,"statfs":{"total":773094113280,"available":772666105856,"internally_reserved":0,"allocated":24420352,"data_stored":12483340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":403426873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.835751"},"pg_stats":[{"pgid":"1.0","version":"215'366","reported_seq":1275,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.871423+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:31:40.871423+0000","last_peered":"2024-03-29T11:31:40.871423+0000","last_clean":"2024-03-29T11:31:40.871423+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:31:40.871423+0000","last_undegraded":"2024-03-29T11:31:40.871423+0000","last_fullsized":"2024-03-29T11:31:40.871423+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.874693+0000","last_change":"2024-03-29T11:31:26.318923+0000","last_active":"2024-03-29T11:31:40.874693+0000","last_peered":"2024-03-29T11:31:40.874693+0000","last_clean":"2024-03-29T11:31:40.874693+0000","last_became_active":"2024-03-29T11:31:26.318658+0000","last_became_peered":"2024-03-29T11:31:26.318658+0000","last_unstale":"2024-03-29T11:31:40.874693+0000","last_undegraded":"2024-03-29T11:31:40.874693+0000","last_fullsized":"2024-03-29T11:31:40.874693+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:25:12.671618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873178+0000","last_change":"2024-03-29T11:31:26.844835+0000","last_active":"2024-03-29T11:31:40.873178+0000","last_peered":"2024-03-29T11:31:40.873178+0000","last_clean":"2024-03-29T11:31:40.873178+0000","last_became_active":"2024-03-29T11:31:26.844570+0000","last_became_peered":"2024-03-29T11:31:26.844570+0000","last_unstale":"2024-03-29T11:31:40.873178+0000","last_undegraded":"2024-03-29T11:31:40.873178+0000","last_fullsized":"2024-03-29T11:31:40.873178+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:53:43.020271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950330+0000","last_change":"2024-03-29T11:31:40.368551+0000","last_active":"2024-03-29T11:31:41.950330+0000","last_peered":"2024-03-29T11:31:41.950330+0000","last_clean":"2024-03-29T11:31:41.950330+0000","last_became_active":"2024-03-29T11:31:40.368254+0000","last_became_peered":"2024-03-29T11:31:40.368254+0000","last_unstale":"2024-03-29T11:31:41.950330+0000","last_undegraded":"2024-03-29T11:31:41.950330+0000","last_fullsized":"2024-03-29T11:31:41.950330+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:35:21.805809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.878868+0000","last_change":"2024-03-29T11:31:40.887274+0000","last_active":"2024-03-29T11:31:41.878868+0000","last_peered":"2024-03-29T11:31:41.878868+0000","last_clean":"2024-03-29T11:31:41.878868+0000","last_became_active":"2024-03-29T11:31:40.886918+0000","last_became_peered":"2024-03-29T11:31:40.886918+0000","last_unstale":"2024-03-29T11:31:41.878868+0000","last_undegraded":"2024-03-29T11:31:41.878868+0000","last_fullsized":"2024-03-29T11:31:41.878868+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:50:30.787108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.886076+0000","last_change":"2024-03-29T11:31:26.318569+0000","last_active":"2024-03-29T11:31:40.886076+0000","last_peered":"2024-03-29T11:31:40.886076+0000","last_clean":"2024-03-29T11:31:40.886076+0000","last_became_active":"2024-03-29T11:31:26.318298+0000","last_became_peered":"2024-03-29T11:31:26.318298+0000","last_unstale":"2024-03-29T11:31:40.886076+0000","last_undegraded":"2024-03-29T11:31:40.886076+0000","last_fullsized":"2024-03-29T11:31:40.886076+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:05:48.527536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885182+0000","last_change":"2024-03-29T11:31:26.843285+0000","last_active":"2024-03-29T11:31:40.885182+0000","last_peered":"2024-03-29T11:31:40.885182+0000","last_clean":"2024-03-29T11:31:40.885182+0000","last_became_active":"2024-03-29T11:31:26.841561+0000","last_became_peered":"2024-03-29T11:31:26.841561+0000","last_unstale":"2024-03-29T11:31:40.885182+0000","last_undegraded":"2024-03-29T11:31:40.885182+0000","last_fullsized":"2024-03-29T11:31:40.885182+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:47.872748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879251+0000","last_change":"2024-03-29T11:31:40.367954+0000","last_active":"2024-03-29T11:31:41.879251+0000","last_peered":"2024-03-29T11:31:41.879251+0000","last_clean":"2024-03-29T11:31:41.879251+0000","last_became_active":"2024-03-29T11:31:40.367669+0000","last_became_peered":"2024-03-29T11:31:40.367669+0000","last_unstale":"2024-03-29T11:31:41.879251+0000","last_undegraded":"2024-03-29T11:31:41.879251+0000","last_fullsized":"2024-03-29T11:31:41.879251+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:58:10.780397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.881307+0000","last_change":"2024-03-29T11:31:40.897163+0000","last_active":"2024-03-29T11:31:41.881307+0000","last_peered":"2024-03-29T11:31:41.881307+0000","last_clean":"2024-03-29T11:31:41.881307+0000","last_became_active":"2024-03-29T11:31:40.896742+0000","last_became_peered":"2024-03-29T11:31:40.896742+0000","last_unstale":"2024-03-29T11:31:41.881307+0000","last_undegraded":"2024-03-29T11:31:41.881307+0000","last_fullsized":"2024-03-29T11:31:41.881307+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:56:07.738932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885453+0000","last_change":"2024-03-29T11:31:26.310072+0000","last_active":"2024-03-29T11:31:40.885453+0000","last_peered":"2024-03-29T11:31:40.885453+0000","last_clean":"2024-03-29T11:31:40.885453+0000","last_became_active":"2024-03-29T11:31:26.309254+0000","last_became_peered":"2024-03-29T11:31:26.309254+0000","last_unstale":"2024-03-29T11:31:40.885453+0000","last_undegraded":"2024-03-29T11:31:40.885453+0000","last_fullsized":"2024-03-29T11:31:40.885453+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:00:36.402074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882760+0000","last_change":"2024-03-29T11:31:26.842129+0000","last_active":"2024-03-29T11:31:40.882760+0000","last_peered":"2024-03-29T11:31:40.882760+0000","last_clean":"2024-03-29T11:31:40.882760+0000","last_became_active":"2024-03-29T11:31:26.841888+0000","last_became_peered":"2024-03-29T11:31:26.841888+0000","last_unstale":"2024-03-29T11:31:40.882760+0000","last_undegraded":"2024-03-29T11:31:40.882760+0000","last_fullsized":"2024-03-29T11:31:40.882760+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:32.997118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951488+0000","last_change":"2024-03-29T11:31:40.372014+0000","last_active":"2024-03-29T11:31:41.951488+0000","last_peered":"2024-03-29T11:31:41.951488+0000","last_clean":"2024-03-29T11:31:41.951488+0000","last_became_active":"2024-03-29T11:31:40.369598+0000","last_became_peered":"2024-03-29T11:31:40.369598+0000","last_unstale":"2024-03-29T11:31:41.951488+0000","last_undegraded":"2024-03-29T11:31:41.951488+0000","last_fullsized":"2024-03-29T11:31:41.951488+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:55:10.199622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951120+0000","last_change":"2024-03-29T11:31:40.901391+0000","last_active":"2024-03-29T11:31:41.951120+0000","last_peered":"2024-03-29T11:31:41.951120+0000","last_clean":"2024-03-29T11:31:41.951120+0000","last_became_active":"2024-03-29T11:31:40.899158+0000","last_became_peered":"2024-03-29T11:31:40.899158+0000","last_unstale":"2024-03-29T11:31:41.951120+0000","last_undegraded":"2024-03-29T11:31:41.951120+0000","last_fullsized":"2024-03-29T11:31:41.951120+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:00:19.784413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882738+0000","last_change":"2024-03-29T11:31:26.322425+0000","last_active":"2024-03-29T11:31:40.882738+0000","last_peered":"2024-03-29T11:31:40.882738+0000","last_clean":"2024-03-29T11:31:40.882738+0000","last_became_active":"2024-03-29T11:31:26.321981+0000","last_became_peered":"2024-03-29T11:31:26.321981+0000","last_unstale":"2024-03-29T11:31:40.882738+0000","last_undegraded":"2024-03-29T11:31:40.882738+0000","last_fullsized":"2024-03-29T11:31:40.882738+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:36.434947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876100+0000","last_change":"2024-03-29T11:31:26.838532+0000","last_active":"2024-03-29T11:31:40.876100+0000","last_peered":"2024-03-29T11:31:40.876100+0000","last_clean":"2024-03-29T11:31:40.876100+0000","last_became_active":"2024-03-29T11:31:26.838267+0000","last_became_peered":"2024-03-29T11:31:26.838267+0000","last_unstale":"2024-03-29T11:31:40.876100+0000","last_undegraded":"2024-03-29T11:31:40.876100+0000","last_fullsized":"2024-03-29T11:31:40.876100+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:17:41.239491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.881113+0000","last_change":"2024-03-29T11:31:40.369144+0000","last_active":"2024-03-29T11:31:41.881113+0000","last_peered":"2024-03-29T11:31:41.881113+0000","last_clean":"2024-03-29T11:31:41.881113+0000","last_became_active":"2024-03-29T11:31:40.368341+0000","last_became_peered":"2024-03-29T11:31:40.368341+0000","last_unstale":"2024-03-29T11:31:41.881113+0000","last_undegraded":"2024-03-29T11:31:41.881113+0000","last_fullsized":"2024-03-29T11:31:41.881113+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:41.276410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950821+0000","last_change":"2024-03-29T11:31:40.903386+0000","last_active":"2024-03-29T11:31:41.950821+0000","last_peered":"2024-03-29T11:31:41.950821+0000","last_clean":"2024-03-29T11:31:41.950821+0000","last_became_active":"2024-03-29T11:31:40.901220+0000","last_became_peered":"2024-03-29T11:31:40.901220+0000","last_unstale":"2024-03-29T11:31:41.950821+0000","last_undegraded":"2024-03-29T11:31:41.950821+0000","last_fullsized":"2024-03-29T11:31:41.950821+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:22.511071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872900+0000","last_change":"2024-03-29T11:31:26.320428+0000","last_active":"2024-03-29T11:31:40.872900+0000","last_peered":"2024-03-29T11:31:40.872900+0000","last_clean":"2024-03-29T11:31:40.872900+0000","last_became_active":"2024-03-29T11:31:26.320099+0000","last_became_peered":"2024-03-29T11:31:26.320099+0000","last_unstale":"2024-03-29T11:31:40.872900+0000","last_undegraded":"2024-03-29T11:31:40.872900+0000","last_fullsized":"2024-03-29T11:31:40.872900+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:56.873408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873231+0000","last_change":"2024-03-29T11:31:26.848609+0000","last_active":"2024-03-29T11:31:40.873231+0000","last_peered":"2024-03-29T11:31:40.873231+0000","last_clean":"2024-03-29T11:31:40.873231+0000","last_became_active":"2024-03-29T11:31:26.848202+0000","last_became_peered":"2024-03-29T11:31:26.848202+0000","last_unstale":"2024-03-29T11:31:40.873231+0000","last_undegraded":"2024-03-29T11:31:40.873231+0000","last_fullsized":"2024-03-29T11:31:40.873231+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:28.240696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879309+0000","last_change":"2024-03-29T11:31:40.367052+0000","last_active":"2024-03-29T11:31:41.879309+0000","last_peered":"2024-03-29T11:31:41.879309+0000","last_clean":"2024-03-29T11:31:41.879309+0000","last_became_active":"2024-03-29T11:31:40.365737+0000","last_became_peered":"2024-03-29T11:31:40.365737+0000","last_unstale":"2024-03-29T11:31:41.879309+0000","last_undegraded":"2024-03-29T11:31:41.879309+0000","last_fullsized":"2024-03-29T11:31:41.879309+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:48:45.870078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880460+0000","last_change":"2024-03-29T11:31:40.889850+0000","last_active":"2024-03-29T11:31:41.880460+0000","last_peered":"2024-03-29T11:31:41.880460+0000","last_clean":"2024-03-29T11:31:41.880460+0000","last_became_active":"2024-03-29T11:31:40.889554+0000","last_became_peered":"2024-03-29T11:31:40.889554+0000","last_unstale":"2024-03-29T11:31:41.880460+0000","last_undegraded":"2024-03-29T11:31:41.880460+0000","last_fullsized":"2024-03-29T11:31:41.880460+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:09:49.649528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883808+0000","last_change":"2024-03-29T11:31:26.311780+0000","last_active":"2024-03-29T11:31:40.883808+0000","last_peered":"2024-03-29T11:31:40.883808+0000","last_clean":"2024-03-29T11:31:40.883808+0000","last_became_active":"2024-03-29T11:31:26.311128+0000","last_became_peered":"2024-03-29T11:31:26.311128+0000","last_unstale":"2024-03-29T11:31:40.883808+0000","last_undegraded":"2024-03-29T11:31:40.883808+0000","last_fullsized":"2024-03-29T11:31:40.883808+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:46.870101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884996+0000","last_change":"2024-03-29T11:31:26.844018+0000","last_active":"2024-03-29T11:31:40.884996+0000","last_peered":"2024-03-29T11:31:40.884996+0000","last_clean":"2024-03-29T11:31:40.884996+0000","last_became_active":"2024-03-29T11:31:26.843237+0000","last_became_peered":"2024-03-29T11:31:26.843237+0000","last_unstale":"2024-03-29T11:31:40.884996+0000","last_undegraded":"2024-03-29T11:31:40.884996+0000","last_fullsized":"2024-03-29T11:31:40.884996+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:12:52.041908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949932+0000","last_change":"2024-03-29T11:31:40.364392+0000","last_active":"2024-03-29T11:31:41.949932+0000","last_peered":"2024-03-29T11:31:41.949932+0000","last_clean":"2024-03-29T11:31:41.949932+0000","last_became_active":"2024-03-29T11:31:40.364108+0000","last_became_peered":"2024-03-29T11:31:40.364108+0000","last_unstale":"2024-03-29T11:31:41.949932+0000","last_undegraded":"2024-03-29T11:31:41.949932+0000","last_fullsized":"2024-03-29T11:31:41.949932+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:54.274308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880566+0000","last_change":"2024-03-29T11:31:40.889918+0000","last_active":"2024-03-29T11:31:41.880566+0000","last_peered":"2024-03-29T11:31:41.880566+0000","last_clean":"2024-03-29T11:31:41.880566+0000","last_became_active":"2024-03-29T11:31:40.889132+0000","last_became_peered":"2024-03-29T11:31:40.889132+0000","last_unstale":"2024-03-29T11:31:41.880566+0000","last_undegraded":"2024-03-29T11:31:41.880566+0000","last_fullsized":"2024-03-29T11:31:41.880566+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:44:04.775529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.892399+0000","last_change":"2024-03-29T11:31:26.317291+0000","last_active":"2024-03-29T11:31:40.892399+0000","last_peered":"2024-03-29T11:31:40.892399+0000","last_clean":"2024-03-29T11:31:40.892399+0000","last_became_active":"2024-03-29T11:31:26.317003+0000","last_became_peered":"2024-03-29T11:31:26.317003+0000","last_unstale":"2024-03-29T11:31:40.892399+0000","last_undegraded":"2024-03-29T11:31:40.892399+0000","last_fullsized":"2024-03-29T11:31:40.892399+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:55:21.607497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.887195+0000","last_change":"2024-03-29T11:31:26.849336+0000","last_active":"2024-03-29T11:31:40.887195+0000","last_peered":"2024-03-29T11:31:40.887195+0000","last_clean":"2024-03-29T11:31:40.887195+0000","last_became_active":"2024-03-29T11:31:26.849042+0000","last_became_peered":"2024-03-29T11:31:26.849042+0000","last_unstale":"2024-03-29T11:31:40.887195+0000","last_undegraded":"2024-03-29T11:31:40.887195+0000","last_fullsized":"2024-03-29T11:31:40.887195+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:58.653895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949597+0000","last_change":"2024-03-29T11:31:40.367967+0000","last_active":"2024-03-29T11:31:41.949597+0000","last_peered":"2024-03-29T11:31:41.949597+0000","last_clean":"2024-03-29T11:31:41.949597+0000","last_became_active":"2024-03-29T11:31:40.367700+0000","last_became_peered":"2024-03-29T11:31:40.367700+0000","last_unstale":"2024-03-29T11:31:41.949597+0000","last_undegraded":"2024-03-29T11:31:41.949597+0000","last_fullsized":"2024-03-29T11:31:41.949597+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:12.867102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950152+0000","last_change":"2024-03-29T11:31:40.894878+0000","last_active":"2024-03-29T11:31:41.950152+0000","last_peered":"2024-03-29T11:31:41.950152+0000","last_clean":"2024-03-29T11:31:41.950152+0000","last_became_active":"2024-03-29T11:31:40.894613+0000","last_became_peered":"2024-03-29T11:31:40.894613+0000","last_unstale":"2024-03-29T11:31:41.950152+0000","last_undegraded":"2024-03-29T11:31:41.950152+0000","last_fullsized":"2024-03-29T11:31:41.950152+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:41.165586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.6","version":"227'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.877289+0000","last_change":"2024-03-29T11:31:26.316463+0000","last_active":"2024-03-29T11:31:40.877289+0000","last_peered":"2024-03-29T11:31:40.877289+0000","last_clean":"2024-03-29T11:31:40.877289+0000","last_became_active":"2024-03-29T11:31:26.315640+0000","last_became_peered":"2024-03-29T11:31:26.315640+0000","last_unstale":"2024-03-29T11:31:40.877289+0000","last_undegraded":"2024-03-29T11:31:40.877289+0000","last_fullsized":"2024-03-29T11:31:40.877289+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:31.994471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872913+0000","last_change":"2024-03-29T11:31:26.845919+0000","last_active":"2024-03-29T11:31:40.872913+0000","last_peered":"2024-03-29T11:31:40.872913+0000","last_clean":"2024-03-29T11:31:40.872913+0000","last_became_active":"2024-03-29T11:31:26.845630+0000","last_became_peered":"2024-03-29T11:31:26.845630+0000","last_unstale":"2024-03-29T11:31:40.872913+0000","last_undegraded":"2024-03-29T11:31:40.872913+0000","last_fullsized":"2024-03-29T11:31:40.872913+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:47.195933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879707+0000","last_change":"2024-03-29T11:31:40.366767+0000","last_active":"2024-03-29T11:31:41.879707+0000","last_peered":"2024-03-29T11:31:41.879707+0000","last_clean":"2024-03-29T11:31:41.879707+0000","last_became_active":"2024-03-29T11:31:40.366265+0000","last_became_peered":"2024-03-29T11:31:40.366265+0000","last_unstale":"2024-03-29T11:31:41.879707+0000","last_undegraded":"2024-03-29T11:31:41.879707+0000","last_fullsized":"2024-03-29T11:31:41.879707+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:17:10.911769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.878471+0000","last_change":"2024-03-29T11:31:40.889807+0000","last_active":"2024-03-29T11:31:41.878471+0000","last_peered":"2024-03-29T11:31:41.878471+0000","last_clean":"2024-03-29T11:31:41.878471+0000","last_became_active":"2024-03-29T11:31:40.889492+0000","last_became_peered":"2024-03-29T11:31:40.889492+0000","last_unstale":"2024-03-29T11:31:41.878471+0000","last_undegraded":"2024-03-29T11:31:41.878471+0000","last_fullsized":"2024-03-29T11:31:41.878471+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:21:21.715338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.7","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885315+0000","last_change":"2024-03-29T11:31:26.310244+0000","last_active":"2024-03-29T11:31:40.885315+0000","last_peered":"2024-03-29T11:31:40.885315+0000","last_clean":"2024-03-29T11:31:40.885315+0000","last_became_active":"2024-03-29T11:31:26.309981+0000","last_became_peered":"2024-03-29T11:31:26.309981+0000","last_unstale":"2024-03-29T11:31:40.885315+0000","last_undegraded":"2024-03-29T11:31:40.885315+0000","last_fullsized":"2024-03-29T11:31:40.885315+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:16.350729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.879270+0000","last_change":"2024-03-29T11:31:26.851615+0000","last_active":"2024-03-29T11:31:40.879270+0000","last_peered":"2024-03-29T11:31:40.879270+0000","last_clean":"2024-03-29T11:31:40.879270+0000","last_became_active":"2024-03-29T11:31:26.851328+0000","last_became_peered":"2024-03-29T11:31:26.851328+0000","last_unstale":"2024-03-29T11:31:40.879270+0000","last_undegraded":"2024-03-29T11:31:40.879270+0000","last_fullsized":"2024-03-29T11:31:40.879270+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:57.906301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949694+0000","last_change":"2024-03-29T11:31:40.367420+0000","last_active":"2024-03-29T11:31:41.949694+0000","last_peered":"2024-03-29T11:31:41.949694+0000","last_clean":"2024-03-29T11:31:41.949694+0000","last_became_active":"2024-03-29T11:31:40.367079+0000","last_became_peered":"2024-03-29T11:31:40.367079+0000","last_unstale":"2024-03-29T11:31:41.949694+0000","last_undegraded":"2024-03-29T11:31:41.949694+0000","last_fullsized":"2024-03-29T11:31:41.949694+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:36:00.231647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879030+0000","last_change":"2024-03-29T11:31:40.905299+0000","last_active":"2024-03-29T11:31:41.879030+0000","last_peered":"2024-03-29T11:31:41.879030+0000","last_clean":"2024-03-29T11:31:41.879030+0000","last_became_active":"2024-03-29T11:31:40.904494+0000","last_became_peered":"2024-03-29T11:31:40.904494+0000","last_unstale":"2024-03-29T11:31:41.879030+0000","last_undegraded":"2024-03-29T11:31:41.879030+0000","last_fullsized":"2024-03-29T11:31:41.879030+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:30.611972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884063+0000","last_change":"2024-03-29T11:31:26.321619+0000","last_active":"2024-03-29T11:31:40.884063+0000","last_peered":"2024-03-29T11:31:40.884063+0000","last_clean":"2024-03-29T11:31:40.884063+0000","last_became_active":"2024-03-29T11:31:26.321349+0000","last_became_peered":"2024-03-29T11:31:26.321349+0000","last_unstale":"2024-03-29T11:31:40.884063+0000","last_undegraded":"2024-03-29T11:31:40.884063+0000","last_fullsized":"2024-03-29T11:31:40.884063+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:09.446187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885595+0000","last_change":"2024-03-29T11:31:26.843420+0000","last_active":"2024-03-29T11:31:40.885595+0000","last_peered":"2024-03-29T11:31:40.885595+0000","last_clean":"2024-03-29T11:31:40.885595+0000","last_became_active":"2024-03-29T11:31:26.842734+0000","last_became_peered":"2024-03-29T11:31:26.842734+0000","last_unstale":"2024-03-29T11:31:40.885595+0000","last_undegraded":"2024-03-29T11:31:40.885595+0000","last_fullsized":"2024-03-29T11:31:40.885595+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:46.006140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950604+0000","last_change":"2024-03-29T11:31:40.371935+0000","last_active":"2024-03-29T11:31:41.950604+0000","last_peered":"2024-03-29T11:31:41.950604+0000","last_clean":"2024-03-29T11:31:41.950604+0000","last_became_active":"2024-03-29T11:31:40.371172+0000","last_became_peered":"2024-03-29T11:31:40.371172+0000","last_unstale":"2024-03-29T11:31:41.950604+0000","last_undegraded":"2024-03-29T11:31:41.950604+0000","last_fullsized":"2024-03-29T11:31:41.950604+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:10:45.255586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874890+0000","last_change":"2024-03-29T11:31:40.895452+0000","last_active":"2024-03-29T11:31:41.874890+0000","last_peered":"2024-03-29T11:31:41.874890+0000","last_clean":"2024-03-29T11:31:41.874890+0000","last_became_active":"2024-03-29T11:31:40.895193+0000","last_became_peered":"2024-03-29T11:31:40.895193+0000","last_unstale":"2024-03-29T11:31:41.874890+0000","last_undegraded":"2024-03-29T11:31:41.874890+0000","last_fullsized":"2024-03-29T11:31:41.874890+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:50:48.434922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870436+0000","last_change":"2024-03-29T11:31:26.312962+0000","last_active":"2024-03-29T11:31:40.870436+0000","last_peered":"2024-03-29T11:31:40.870436+0000","last_clean":"2024-03-29T11:31:40.870436+0000","last_became_active":"2024-03-29T11:31:26.312668+0000","last_became_peered":"2024-03-29T11:31:26.312668+0000","last_unstale":"2024-03-29T11:31:40.870436+0000","last_undegraded":"2024-03-29T11:31:40.870436+0000","last_fullsized":"2024-03-29T11:31:40.870436+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:46.870101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.881073+0000","last_change":"2024-03-29T11:31:26.835978+0000","last_active":"2024-03-29T11:31:40.881073+0000","last_peered":"2024-03-29T11:31:40.881073+0000","last_clean":"2024-03-29T11:31:40.881073+0000","last_became_active":"2024-03-29T11:31:26.835400+0000","last_became_peered":"2024-03-29T11:31:26.835400+0000","last_unstale":"2024-03-29T11:31:40.881073+0000","last_undegraded":"2024-03-29T11:31:40.881073+0000","last_fullsized":"2024-03-29T11:31:40.881073+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:31.783000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.881318+0000","last_change":"2024-03-29T11:31:40.368134+0000","last_active":"2024-03-29T11:31:41.881318+0000","last_peered":"2024-03-29T11:31:41.881318+0000","last_clean":"2024-03-29T11:31:41.881318+0000","last_became_active":"2024-03-29T11:31:40.367120+0000","last_became_peered":"2024-03-29T11:31:40.367120+0000","last_unstale":"2024-03-29T11:31:41.881318+0000","last_undegraded":"2024-03-29T11:31:41.881318+0000","last_fullsized":"2024-03-29T11:31:41.881318+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:55:58.832455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951430+0000","last_change":"2024-03-29T11:31:40.886320+0000","last_active":"2024-03-29T11:31:41.951430+0000","last_peered":"2024-03-29T11:31:41.951430+0000","last_clean":"2024-03-29T11:31:41.951430+0000","last_became_active":"2024-03-29T11:31:40.885837+0000","last_became_peered":"2024-03-29T11:31:40.885837+0000","last_unstale":"2024-03-29T11:31:41.951430+0000","last_undegraded":"2024-03-29T11:31:41.951430+0000","last_fullsized":"2024-03-29T11:31:41.951430+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:10:46.262823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873091+0000","last_change":"2024-03-29T11:31:26.312485+0000","last_active":"2024-03-29T11:31:40.873091+0000","last_peered":"2024-03-29T11:31:40.873091+0000","last_clean":"2024-03-29T11:31:40.873091+0000","last_became_active":"2024-03-29T11:31:26.311989+0000","last_became_peered":"2024-03-29T11:31:26.311989+0000","last_unstale":"2024-03-29T11:31:40.873091+0000","last_undegraded":"2024-03-29T11:31:40.873091+0000","last_fullsized":"2024-03-29T11:31:40.873091+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:42:56.190784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883080+0000","last_change":"2024-03-29T11:31:26.836881+0000","last_active":"2024-03-29T11:31:40.883080+0000","last_peered":"2024-03-29T11:31:40.883080+0000","last_clean":"2024-03-29T11:31:40.883080+0000","last_became_active":"2024-03-29T11:31:26.836613+0000","last_became_peered":"2024-03-29T11:31:26.836613+0000","last_unstale":"2024-03-29T11:31:40.883080+0000","last_undegraded":"2024-03-29T11:31:40.883080+0000","last_fullsized":"2024-03-29T11:31:40.883080+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:23.565406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874497+0000","last_change":"2024-03-29T11:31:40.357898+0000","last_active":"2024-03-29T11:31:41.874497+0000","last_peered":"2024-03-29T11:31:41.874497+0000","last_clean":"2024-03-29T11:31:41.874497+0000","last_became_active":"2024-03-29T11:31:40.357355+0000","last_became_peered":"2024-03-29T11:31:40.357355+0000","last_unstale":"2024-03-29T11:31:41.874497+0000","last_undegraded":"2024-03-29T11:31:41.874497+0000","last_fullsized":"2024-03-29T11:31:41.874497+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:54.052209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880921+0000","last_change":"2024-03-29T11:31:40.897342+0000","last_active":"2024-03-29T11:31:41.880921+0000","last_peered":"2024-03-29T11:31:41.880921+0000","last_clean":"2024-03-29T11:31:41.880921+0000","last_became_active":"2024-03-29T11:31:40.896970+0000","last_became_peered":"2024-03-29T11:31:40.896970+0000","last_unstale":"2024-03-29T11:31:41.880921+0000","last_undegraded":"2024-03-29T11:31:41.880921+0000","last_fullsized":"2024-03-29T11:31:41.880921+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:32:36.867108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875822+0000","last_change":"2024-03-29T11:31:26.316435+0000","last_active":"2024-03-29T11:31:40.875822+0000","last_peered":"2024-03-29T11:31:40.875822+0000","last_clean":"2024-03-29T11:31:40.875822+0000","last_became_active":"2024-03-29T11:31:26.316124+0000","last_became_peered":"2024-03-29T11:31:26.316124+0000","last_unstale":"2024-03-29T11:31:40.875822+0000","last_undegraded":"2024-03-29T11:31:40.875822+0000","last_fullsized":"2024-03-29T11:31:40.875822+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:15:10.280553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875924+0000","last_change":"2024-03-29T11:31:26.844232+0000","last_active":"2024-03-29T11:31:40.875924+0000","last_peered":"2024-03-29T11:31:40.875924+0000","last_clean":"2024-03-29T11:31:40.875924+0000","last_became_active":"2024-03-29T11:31:26.843977+0000","last_became_peered":"2024-03-29T11:31:26.843977+0000","last_unstale":"2024-03-29T11:31:40.875924+0000","last_undegraded":"2024-03-29T11:31:40.875924+0000","last_fullsized":"2024-03-29T11:31:40.875924+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879088+0000","last_change":"2024-03-29T11:31:40.364785+0000","last_active":"2024-03-29T11:31:41.879088+0000","last_peered":"2024-03-29T11:31:41.879088+0000","last_clean":"2024-03-29T11:31:41.879088+0000","last_became_active":"2024-03-29T11:31:40.360188+0000","last_became_peered":"2024-03-29T11:31:40.360188+0000","last_unstale":"2024-03-29T11:31:41.879088+0000","last_undegraded":"2024-03-29T11:31:41.879088+0000","last_fullsized":"2024-03-29T11:31:41.879088+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:54:08.167891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880604+0000","last_change":"2024-03-29T11:31:40.901353+0000","last_active":"2024-03-29T11:31:41.880604+0000","last_peered":"2024-03-29T11:31:41.880604+0000","last_clean":"2024-03-29T11:31:41.880604+0000","last_became_active":"2024-03-29T11:31:40.901082+0000","last_became_peered":"2024-03-29T11:31:40.901082+0000","last_unstale":"2024-03-29T11:31:41.880604+0000","last_undegraded":"2024-03-29T11:31:41.880604+0000","last_fullsized":"2024-03-29T11:31:41.880604+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:00:19.784413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883643+0000","last_change":"2024-03-29T11:31:26.313158+0000","last_active":"2024-03-29T11:31:40.883643+0000","last_peered":"2024-03-29T11:31:40.883643+0000","last_clean":"2024-03-29T11:31:40.883643+0000","last_became_active":"2024-03-29T11:31:26.311492+0000","last_became_peered":"2024-03-29T11:31:26.311492+0000","last_unstale":"2024-03-29T11:31:40.883643+0000","last_undegraded":"2024-03-29T11:31:40.883643+0000","last_fullsized":"2024-03-29T11:31:40.883643+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:40.347554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876039+0000","last_change":"2024-03-29T11:31:26.848108+0000","last_active":"2024-03-29T11:31:40.876039+0000","last_peered":"2024-03-29T11:31:40.876039+0000","last_clean":"2024-03-29T11:31:40.876039+0000","last_became_active":"2024-03-29T11:31:26.847705+0000","last_became_peered":"2024-03-29T11:31:26.847705+0000","last_unstale":"2024-03-29T11:31:40.876039+0000","last_undegraded":"2024-03-29T11:31:40.876039+0000","last_fullsized":"2024-03-29T11:31:40.876039+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:53:43.020271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949828+0000","last_change":"2024-03-29T11:31:40.363049+0000","last_active":"2024-03-29T11:31:41.949828+0000","last_peered":"2024-03-29T11:31:41.949828+0000","last_clean":"2024-03-29T11:31:41.949828+0000","last_became_active":"2024-03-29T11:31:40.361460+0000","last_became_peered":"2024-03-29T11:31:40.361460+0000","last_unstale":"2024-03-29T11:31:41.949828+0000","last_undegraded":"2024-03-29T11:31:41.949828+0000","last_fullsized":"2024-03-29T11:31:41.949828+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:37.266073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880725+0000","last_change":"2024-03-29T11:31:40.882355+0000","last_active":"2024-03-29T11:31:41.880725+0000","last_peered":"2024-03-29T11:31:41.880725+0000","last_clean":"2024-03-29T11:31:41.880725+0000","last_became_active":"2024-03-29T11:31:40.882036+0000","last_became_peered":"2024-03-29T11:31:40.882036+0000","last_unstale":"2024-03-29T11:31:41.880725+0000","last_undegraded":"2024-03-29T11:31:41.880725+0000","last_fullsized":"2024-03-29T11:31:41.880725+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:26:19.372364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.d","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876933+0000","last_change":"2024-03-29T11:31:26.319951+0000","last_active":"2024-03-29T11:31:40.876933+0000","last_peered":"2024-03-29T11:31:40.876933+0000","last_clean":"2024-03-29T11:31:40.876933+0000","last_became_active":"2024-03-29T11:31:26.318226+0000","last_became_peered":"2024-03-29T11:31:26.318226+0000","last_unstale":"2024-03-29T11:31:40.876933+0000","last_undegraded":"2024-03-29T11:31:40.876933+0000","last_fullsized":"2024-03-29T11:31:40.876933+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:20:22.562759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883723+0000","last_change":"2024-03-29T11:31:26.833306+0000","last_active":"2024-03-29T11:31:40.883723+0000","last_peered":"2024-03-29T11:31:40.883723+0000","last_clean":"2024-03-29T11:31:40.883723+0000","last_became_active":"2024-03-29T11:31:26.833041+0000","last_became_peered":"2024-03-29T11:31:26.833041+0000","last_unstale":"2024-03-29T11:31:40.883723+0000","last_undegraded":"2024-03-29T11:31:40.883723+0000","last_fullsized":"2024-03-29T11:31:40.883723+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:15:11.283200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.881180+0000","last_change":"2024-03-29T11:31:40.365002+0000","last_active":"2024-03-29T11:31:41.881180+0000","last_peered":"2024-03-29T11:31:41.881180+0000","last_clean":"2024-03-29T11:31:41.881180+0000","last_became_active":"2024-03-29T11:31:40.364715+0000","last_became_peered":"2024-03-29T11:31:40.364715+0000","last_unstale":"2024-03-29T11:31:41.881180+0000","last_undegraded":"2024-03-29T11:31:41.881180+0000","last_fullsized":"2024-03-29T11:31:41.881180+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:47:07.143595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950062+0000","last_change":"2024-03-29T11:31:40.896742+0000","last_active":"2024-03-29T11:31:41.950062+0000","last_peered":"2024-03-29T11:31:41.950062+0000","last_clean":"2024-03-29T11:31:41.950062+0000","last_became_active":"2024-03-29T11:31:40.896478+0000","last_became_peered":"2024-03-29T11:31:40.896478+0000","last_unstale":"2024-03-29T11:31:41.950062+0000","last_undegraded":"2024-03-29T11:31:41.950062+0000","last_fullsized":"2024-03-29T11:31:41.950062+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:23.215173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884317+0000","last_change":"2024-03-29T11:31:26.317437+0000","last_active":"2024-03-29T11:31:40.884317+0000","last_peered":"2024-03-29T11:31:40.884317+0000","last_clean":"2024-03-29T11:31:40.884317+0000","last_became_active":"2024-03-29T11:31:26.317128+0000","last_became_peered":"2024-03-29T11:31:26.317128+0000","last_unstale":"2024-03-29T11:31:40.884317+0000","last_undegraded":"2024-03-29T11:31:40.884317+0000","last_fullsized":"2024-03-29T11:31:40.884317+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:57.806178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.889312+0000","last_change":"2024-03-29T11:31:26.833432+0000","last_active":"2024-03-29T11:31:40.889312+0000","last_peered":"2024-03-29T11:31:40.889312+0000","last_clean":"2024-03-29T11:31:40.889312+0000","last_became_active":"2024-03-29T11:31:26.833078+0000","last_became_peered":"2024-03-29T11:31:26.833078+0000","last_unstale":"2024-03-29T11:31:40.889312+0000","last_undegraded":"2024-03-29T11:31:40.889312+0000","last_fullsized":"2024-03-29T11:31:40.889312+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:17.353377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879603+0000","last_change":"2024-03-29T11:31:40.366329+0000","last_active":"2024-03-29T11:31:41.879603+0000","last_peered":"2024-03-29T11:31:41.879603+0000","last_clean":"2024-03-29T11:31:41.879603+0000","last_became_active":"2024-03-29T11:31:40.365971+0000","last_became_peered":"2024-03-29T11:31:40.365971+0000","last_unstale":"2024-03-29T11:31:41.879603+0000","last_undegraded":"2024-03-29T11:31:41.879603+0000","last_fullsized":"2024-03-29T11:31:41.879603+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:03.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.948828+0000","last_change":"2024-03-29T11:31:40.896076+0000","last_active":"2024-03-29T11:31:41.948828+0000","last_peered":"2024-03-29T11:31:41.948828+0000","last_clean":"2024-03-29T11:31:41.948828+0000","last_became_active":"2024-03-29T11:31:40.895757+0000","last_became_peered":"2024-03-29T11:31:40.895757+0000","last_unstale":"2024-03-29T11:31:41.948828+0000","last_undegraded":"2024-03-29T11:31:41.948828+0000","last_fullsized":"2024-03-29T11:31:41.948828+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:28:36.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.f","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.893768+0000","last_change":"2024-03-29T11:31:26.320455+0000","last_active":"2024-03-29T11:31:40.893768+0000","last_peered":"2024-03-29T11:31:40.893768+0000","last_clean":"2024-03-29T11:31:40.893768+0000","last_became_active":"2024-03-29T11:31:26.320175+0000","last_became_peered":"2024-03-29T11:31:26.320175+0000","last_unstale":"2024-03-29T11:31:40.893768+0000","last_undegraded":"2024-03-29T11:31:40.893768+0000","last_fullsized":"2024-03-29T11:31:40.893768+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:25:12.671618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884065+0000","last_change":"2024-03-29T11:31:26.854468+0000","last_active":"2024-03-29T11:31:40.884065+0000","last_peered":"2024-03-29T11:31:40.884065+0000","last_clean":"2024-03-29T11:31:40.884065+0000","last_became_active":"2024-03-29T11:31:26.851644+0000","last_became_peered":"2024-03-29T11:31:26.851644+0000","last_unstale":"2024-03-29T11:31:40.884065+0000","last_undegraded":"2024-03-29T11:31:40.884065+0000","last_fullsized":"2024-03-29T11:31:40.884065+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:54:16.143243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949199+0000","last_change":"2024-03-29T11:31:40.366948+0000","last_active":"2024-03-29T11:31:41.949199+0000","last_peered":"2024-03-29T11:31:41.949199+0000","last_clean":"2024-03-29T11:31:41.949199+0000","last_became_active":"2024-03-29T11:31:40.366626+0000","last_became_peered":"2024-03-29T11:31:40.366626+0000","last_unstale":"2024-03-29T11:31:41.949199+0000","last_undegraded":"2024-03-29T11:31:41.949199+0000","last_fullsized":"2024-03-29T11:31:41.949199+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:40.158349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949088+0000","last_change":"2024-03-29T11:31:40.888358+0000","last_active":"2024-03-29T11:31:41.949088+0000","last_peered":"2024-03-29T11:31:41.949088+0000","last_clean":"2024-03-29T11:31:41.949088+0000","last_became_active":"2024-03-29T11:31:40.888044+0000","last_became_peered":"2024-03-29T11:31:40.888044+0000","last_unstale":"2024-03-29T11:31:41.949088+0000","last_undegraded":"2024-03-29T11:31:41.949088+0000","last_fullsized":"2024-03-29T11:31:41.949088+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:11:35.554749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.10","version":"226'8","reported_seq":23,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882877+0000","last_change":"2024-03-29T11:31:26.320392+0000","last_active":"2024-03-29T11:31:40.882877+0000","last_peered":"2024-03-29T11:31:40.882877+0000","last_clean":"2024-03-29T11:31:40.882877+0000","last_became_active":"2024-03-29T11:31:26.319139+0000","last_became_peered":"2024-03-29T11:31:26.319139+0000","last_unstale":"2024-03-29T11:31:40.882877+0000","last_undegraded":"2024-03-29T11:31:40.882877+0000","last_fullsized":"2024-03-29T11:31:40.882877+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:57.651247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870353+0000","last_change":"2024-03-29T11:31:26.843461+0000","last_active":"2024-03-29T11:31:40.870353+0000","last_peered":"2024-03-29T11:31:40.870353+0000","last_clean":"2024-03-29T11:31:40.870353+0000","last_became_active":"2024-03-29T11:31:26.839816+0000","last_became_peered":"2024-03-29T11:31:26.839816+0000","last_unstale":"2024-03-29T11:31:40.870353+0000","last_undegraded":"2024-03-29T11:31:40.870353+0000","last_fullsized":"2024-03-29T11:31:40.870353+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:32.997118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950691+0000","last_change":"2024-03-29T11:31:40.372288+0000","last_active":"2024-03-29T11:31:41.950691+0000","last_peered":"2024-03-29T11:31:41.950691+0000","last_clean":"2024-03-29T11:31:41.950691+0000","last_became_active":"2024-03-29T11:31:40.370805+0000","last_became_peered":"2024-03-29T11:31:40.370805+0000","last_unstale":"2024-03-29T11:31:41.950691+0000","last_undegraded":"2024-03-29T11:31:41.950691+0000","last_fullsized":"2024-03-29T11:31:41.950691+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:54.052209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880516+0000","last_change":"2024-03-29T11:31:40.896839+0000","last_active":"2024-03-29T11:31:41.880516+0000","last_peered":"2024-03-29T11:31:41.880516+0000","last_clean":"2024-03-29T11:31:41.880516+0000","last_became_active":"2024-03-29T11:31:40.896543+0000","last_became_peered":"2024-03-29T11:31:40.896543+0000","last_unstale":"2024-03-29T11:31:41.880516+0000","last_undegraded":"2024-03-29T11:31:41.880516+0000","last_fullsized":"2024-03-29T11:31:41.880516+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:58:21.739901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875590+0000","last_change":"2024-03-29T11:31:26.314471+0000","last_active":"2024-03-29T11:31:40.875590+0000","last_peered":"2024-03-29T11:31:40.875590+0000","last_clean":"2024-03-29T11:31:40.875590+0000","last_became_active":"2024-03-29T11:31:26.314204+0000","last_became_peered":"2024-03-29T11:31:26.314204+0000","last_unstale":"2024-03-29T11:31:40.875590+0000","last_undegraded":"2024-03-29T11:31:40.875590+0000","last_fullsized":"2024-03-29T11:31:40.875590+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:49:13.676544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882678+0000","last_change":"2024-03-29T11:31:26.846893+0000","last_active":"2024-03-29T11:31:40.882678+0000","last_peered":"2024-03-29T11:31:40.882678+0000","last_clean":"2024-03-29T11:31:40.882678+0000","last_became_active":"2024-03-29T11:31:26.845341+0000","last_became_peered":"2024-03-29T11:31:26.845341+0000","last_unstale":"2024-03-29T11:31:40.882678+0000","last_undegraded":"2024-03-29T11:31:40.882678+0000","last_fullsized":"2024-03-29T11:31:40.882678+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:28.240696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880418+0000","last_change":"2024-03-29T11:31:40.367705+0000","last_active":"2024-03-29T11:31:41.880418+0000","last_peered":"2024-03-29T11:31:41.880418+0000","last_clean":"2024-03-29T11:31:41.880418+0000","last_became_active":"2024-03-29T11:31:40.367405+0000","last_became_peered":"2024-03-29T11:31:40.367405+0000","last_unstale":"2024-03-29T11:31:41.880418+0000","last_undegraded":"2024-03-29T11:31:41.880418+0000","last_fullsized":"2024-03-29T11:31:41.880418+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:54.052209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951245+0000","last_change":"2024-03-29T11:31:40.901915+0000","last_active":"2024-03-29T11:31:41.951245+0000","last_peered":"2024-03-29T11:31:41.951245+0000","last_clean":"2024-03-29T11:31:41.951245+0000","last_became_active":"2024-03-29T11:31:40.900676+0000","last_became_peered":"2024-03-29T11:31:40.900676+0000","last_unstale":"2024-03-29T11:31:41.951245+0000","last_undegraded":"2024-03-29T11:31:41.951245+0000","last_fullsized":"2024-03-29T11:31:41.951245+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:16:12.619950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885741+0000","last_change":"2024-03-29T11:31:26.315775+0000","last_active":"2024-03-29T11:31:40.885741+0000","last_peered":"2024-03-29T11:31:40.885741+0000","last_clean":"2024-03-29T11:31:40.885741+0000","last_became_active":"2024-03-29T11:31:26.315493+0000","last_became_peered":"2024-03-29T11:31:26.315493+0000","last_unstale":"2024-03-29T11:31:40.885741+0000","last_undegraded":"2024-03-29T11:31:40.885741+0000","last_fullsized":"2024-03-29T11:31:40.885741+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:32:47.003355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876143+0000","last_change":"2024-03-29T11:31:26.833527+0000","last_active":"2024-03-29T11:31:40.876143+0000","last_peered":"2024-03-29T11:31:40.876143+0000","last_clean":"2024-03-29T11:31:40.876143+0000","last_became_active":"2024-03-29T11:31:26.833280+0000","last_became_peered":"2024-03-29T11:31:26.833280+0000","last_unstale":"2024-03-29T11:31:40.876143+0000","last_undegraded":"2024-03-29T11:31:40.876143+0000","last_fullsized":"2024-03-29T11:31:40.876143+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:35:08.770095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"230'6","reported_seq":17,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874583+0000","last_change":"2024-03-29T11:31:40.367200+0000","last_active":"2024-03-29T11:31:41.874583+0000","last_peered":"2024-03-29T11:31:41.874583+0000","last_clean":"2024-03-29T11:31:41.874583+0000","last_became_active":"2024-03-29T11:31:40.366932+0000","last_became_peered":"2024-03-29T11:31:40.366932+0000","last_unstale":"2024-03-29T11:31:41.874583+0000","last_undegraded":"2024-03-29T11:31:41.874583+0000","last_fullsized":"2024-03-29T11:31:41.874583+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:12:33.891267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879198+0000","last_change":"2024-03-29T11:31:40.889854+0000","last_active":"2024-03-29T11:31:41.879198+0000","last_peered":"2024-03-29T11:31:41.879198+0000","last_clean":"2024-03-29T11:31:41.879198+0000","last_became_active":"2024-03-29T11:31:40.889514+0000","last_became_peered":"2024-03-29T11:31:40.889514+0000","last_unstale":"2024-03-29T11:31:41.879198+0000","last_undegraded":"2024-03-29T11:31:41.879198+0000","last_fullsized":"2024-03-29T11:31:41.879198+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:22:49.143818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.13","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885809+0000","last_change":"2024-03-29T11:31:26.313768+0000","last_active":"2024-03-29T11:31:40.885809+0000","last_peered":"2024-03-29T11:31:40.885809+0000","last_clean":"2024-03-29T11:31:40.885809+0000","last_became_active":"2024-03-29T11:31:26.313458+0000","last_became_peered":"2024-03-29T11:31:26.313458+0000","last_unstale":"2024-03-29T11:31:40.885809+0000","last_undegraded":"2024-03-29T11:31:40.885809+0000","last_fullsized":"2024-03-29T11:31:40.885809+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:06.334180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883806+0000","last_change":"2024-03-29T11:31:26.840312+0000","last_active":"2024-03-29T11:31:40.883806+0000","last_peered":"2024-03-29T11:31:40.883806+0000","last_clean":"2024-03-29T11:31:40.883806+0000","last_became_active":"2024-03-29T11:31:26.840020+0000","last_became_peered":"2024-03-29T11:31:26.840020+0000","last_unstale":"2024-03-29T11:31:40.883806+0000","last_undegraded":"2024-03-29T11:31:40.883806+0000","last_fullsized":"2024-03-29T11:31:40.883806+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:31:51.171001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"230'3","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950922+0000","last_change":"2024-03-29T11:31:40.370725+0000","last_active":"2024-03-29T11:31:41.950922+0000","last_peered":"2024-03-29T11:31:41.950922+0000","last_clean":"2024-03-29T11:31:41.950922+0000","last_became_active":"2024-03-29T11:31:40.370456+0000","last_became_peered":"2024-03-29T11:31:40.370456+0000","last_unstale":"2024-03-29T11:31:41.950922+0000","last_undegraded":"2024-03-29T11:31:41.950922+0000","last_fullsized":"2024-03-29T11:31:41.950922+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:16:46.032832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949653+0000","last_change":"2024-03-29T11:31:40.902923+0000","last_active":"2024-03-29T11:31:41.949653+0000","last_peered":"2024-03-29T11:31:41.949653+0000","last_clean":"2024-03-29T11:31:41.949653+0000","last_became_active":"2024-03-29T11:31:40.902661+0000","last_became_peered":"2024-03-29T11:31:40.902661+0000","last_unstale":"2024-03-29T11:31:41.949653+0000","last_undegraded":"2024-03-29T11:31:41.949653+0000","last_fullsized":"2024-03-29T11:31:41.949653+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:26:19.372364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.14","version":"227'8","reported_seq":23,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.874187+0000","last_change":"2024-03-29T11:31:26.319966+0000","last_active":"2024-03-29T11:31:40.874187+0000","last_peered":"2024-03-29T11:31:40.874187+0000","last_clean":"2024-03-29T11:31:40.874187+0000","last_became_active":"2024-03-29T11:31:26.319700+0000","last_became_peered":"2024-03-29T11:31:26.319700+0000","last_unstale":"2024-03-29T11:31:40.874187+0000","last_undegraded":"2024-03-29T11:31:40.874187+0000","last_fullsized":"2024-03-29T11:31:40.874187+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:56.161261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873111+0000","last_change":"2024-03-29T11:31:26.844802+0000","last_active":"2024-03-29T11:31:40.873111+0000","last_peered":"2024-03-29T11:31:40.873111+0000","last_clean":"2024-03-29T11:31:40.873111+0000","last_became_active":"2024-03-29T11:31:26.844538+0000","last_became_peered":"2024-03-29T11:31:26.844538+0000","last_unstale":"2024-03-29T11:31:40.873111+0000","last_undegraded":"2024-03-29T11:31:40.873111+0000","last_fullsized":"2024-03-29T11:31:40.873111+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:05.909043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949757+0000","last_change":"2024-03-29T11:31:40.369428+0000","last_active":"2024-03-29T11:31:41.949757+0000","last_peered":"2024-03-29T11:31:41.949757+0000","last_clean":"2024-03-29T11:31:41.949757+0000","last_became_active":"2024-03-29T11:31:40.367133+0000","last_became_peered":"2024-03-29T11:31:40.367133+0000","last_unstale":"2024-03-29T11:31:41.949757+0000","last_undegraded":"2024-03-29T11:31:41.949757+0000","last_fullsized":"2024-03-29T11:31:41.949757+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:11:34.547512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880456+0000","last_change":"2024-03-29T11:31:40.889479+0000","last_active":"2024-03-29T11:31:41.880456+0000","last_peered":"2024-03-29T11:31:41.880456+0000","last_clean":"2024-03-29T11:31:41.880456+0000","last_became_active":"2024-03-29T11:31:40.884735+0000","last_became_peered":"2024-03-29T11:31:40.884735+0000","last_unstale":"2024-03-29T11:31:41.880456+0000","last_undegraded":"2024-03-29T11:31:41.880456+0000","last_fullsized":"2024-03-29T11:31:41.880456+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:48:28.663360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.15","version":"226'3","reported_seq":18,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876826+0000","last_change":"2024-03-29T11:31:26.320127+0000","last_active":"2024-03-29T11:31:40.876826+0000","last_peered":"2024-03-29T11:31:40.876826+0000","last_clean":"2024-03-29T11:31:40.876826+0000","last_became_active":"2024-03-29T11:31:26.319346+0000","last_became_peered":"2024-03-29T11:31:26.319346+0000","last_unstale":"2024-03-29T11:31:40.876826+0000","last_undegraded":"2024-03-29T11:31:40.876826+0000","last_fullsized":"2024-03-29T11:31:40.876826+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:32:50.637683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.874388+0000","last_change":"2024-03-29T11:31:26.857148+0000","last_active":"2024-03-29T11:31:40.874388+0000","last_peered":"2024-03-29T11:31:40.874388+0000","last_clean":"2024-03-29T11:31:40.874388+0000","last_became_active":"2024-03-29T11:31:26.856870+0000","last_became_peered":"2024-03-29T11:31:26.856870+0000","last_unstale":"2024-03-29T11:31:40.874388+0000","last_undegraded":"2024-03-29T11:31:40.874388+0000","last_fullsized":"2024-03-29T11:31:40.874388+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951631+0000","last_change":"2024-03-29T11:31:40.367974+0000","last_active":"2024-03-29T11:31:41.951631+0000","last_peered":"2024-03-29T11:31:41.951631+0000","last_clean":"2024-03-29T11:31:41.951631+0000","last_became_active":"2024-03-29T11:31:40.367528+0000","last_became_peered":"2024-03-29T11:31:40.367528+0000","last_unstale":"2024-03-29T11:31:41.951631+0000","last_undegraded":"2024-03-29T11:31:41.951631+0000","last_fullsized":"2024-03-29T11:31:41.951631+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:00:18.777176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949596+0000","last_change":"2024-03-29T11:31:40.883630+0000","last_active":"2024-03-29T11:31:41.949596+0000","last_peered":"2024-03-29T11:31:41.949596+0000","last_clean":"2024-03-29T11:31:41.949596+0000","last_became_active":"2024-03-29T11:31:40.883371+0000","last_became_peered":"2024-03-29T11:31:40.883371+0000","last_unstale":"2024-03-29T11:31:41.949596+0000","last_undegraded":"2024-03-29T11:31:41.949596+0000","last_fullsized":"2024-03-29T11:31:41.949596+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:24:49.255586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885781+0000","last_change":"2024-03-29T11:31:26.316324+0000","last_active":"2024-03-29T11:31:40.885781+0000","last_peered":"2024-03-29T11:31:40.885781+0000","last_clean":"2024-03-29T11:31:40.885781+0000","last_became_active":"2024-03-29T11:31:26.316066+0000","last_became_peered":"2024-03-29T11:31:26.316066+0000","last_unstale":"2024-03-29T11:31:40.885781+0000","last_undegraded":"2024-03-29T11:31:40.885781+0000","last_fullsized":"2024-03-29T11:31:40.885781+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:24:19.700336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884501+0000","last_change":"2024-03-29T11:31:26.846344+0000","last_active":"2024-03-29T11:31:40.884501+0000","last_peered":"2024-03-29T11:31:40.884501+0000","last_clean":"2024-03-29T11:31:40.884501+0000","last_became_active":"2024-03-29T11:31:26.846066+0000","last_became_peered":"2024-03-29T11:31:26.846066+0000","last_unstale":"2024-03-29T11:31:40.884501+0000","last_undegraded":"2024-03-29T11:31:40.884501+0000","last_fullsized":"2024-03-29T11:31:40.884501+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:57.163908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"230'8","reported_seq":19,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949749+0000","last_change":"2024-03-29T11:31:40.361408+0000","last_active":"2024-03-29T11:31:41.949749+0000","last_peered":"2024-03-29T11:31:41.949749+0000","last_clean":"2024-03-29T11:31:41.949749+0000","last_became_active":"2024-03-29T11:31:40.360360+0000","last_became_peered":"2024-03-29T11:31:40.360360+0000","last_unstale":"2024-03-29T11:31:41.949749+0000","last_undegraded":"2024-03-29T11:31:41.949749+0000","last_fullsized":"2024-03-29T11:31:41.949749+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:47:07.143595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949367+0000","last_change":"2024-03-29T11:31:40.891301+0000","last_active":"2024-03-29T11:31:41.949367+0000","last_peered":"2024-03-29T11:31:41.949367+0000","last_clean":"2024-03-29T11:31:41.949367+0000","last_became_active":"2024-03-29T11:31:40.891017+0000","last_became_peered":"2024-03-29T11:31:40.891017+0000","last_unstale":"2024-03-29T11:31:41.949367+0000","last_undegraded":"2024-03-29T11:31:41.949367+0000","last_fullsized":"2024-03-29T11:31:41.949367+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:07.759471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.891837+0000","last_change":"2024-03-29T11:31:26.317297+0000","last_active":"2024-03-29T11:31:40.891837+0000","last_peered":"2024-03-29T11:31:40.891837+0000","last_clean":"2024-03-29T11:31:40.891837+0000","last_became_active":"2024-03-29T11:31:26.316956+0000","last_became_peered":"2024-03-29T11:31:26.316956+0000","last_unstale":"2024-03-29T11:31:40.891837+0000","last_undegraded":"2024-03-29T11:31:40.891837+0000","last_fullsized":"2024-03-29T11:31:40.891837+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:18.302941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.877171+0000","last_change":"2024-03-29T11:31:26.855805+0000","last_active":"2024-03-29T11:31:40.877171+0000","last_peered":"2024-03-29T11:31:40.877171+0000","last_clean":"2024-03-29T11:31:40.877171+0000","last_became_active":"2024-03-29T11:31:26.854618+0000","last_became_peered":"2024-03-29T11:31:26.854618+0000","last_unstale":"2024-03-29T11:31:40.877171+0000","last_undegraded":"2024-03-29T11:31:40.877171+0000","last_fullsized":"2024-03-29T11:31:40.877171+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.831388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949535+0000","last_change":"2024-03-29T11:31:40.360159+0000","last_active":"2024-03-29T11:31:41.949535+0000","last_peered":"2024-03-29T11:31:41.949535+0000","last_clean":"2024-03-29T11:31:41.949535+0000","last_became_active":"2024-03-29T11:31:40.359689+0000","last_became_peered":"2024-03-29T11:31:40.359689+0000","last_unstale":"2024-03-29T11:31:41.949535+0000","last_undegraded":"2024-03-29T11:31:41.949535+0000","last_fullsized":"2024-03-29T11:31:41.949535+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:41.276410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879003+0000","last_change":"2024-03-29T11:31:40.905540+0000","last_active":"2024-03-29T11:31:41.879003+0000","last_peered":"2024-03-29T11:31:41.879003+0000","last_clean":"2024-03-29T11:31:41.879003+0000","last_became_active":"2024-03-29T11:31:40.904961+0000","last_became_peered":"2024-03-29T11:31:40.904961+0000","last_unstale":"2024-03-29T11:31:41.879003+0000","last_undegraded":"2024-03-29T11:31:41.879003+0000","last_fullsized":"2024-03-29T11:31:41.879003+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:43.064595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.894268+0000","last_change":"2024-03-29T11:31:26.320631+0000","last_active":"2024-03-29T11:31:40.894268+0000","last_peered":"2024-03-29T11:31:40.894268+0000","last_clean":"2024-03-29T11:31:40.894268+0000","last_became_active":"2024-03-29T11:31:26.320350+0000","last_became_peered":"2024-03-29T11:31:26.320350+0000","last_unstale":"2024-03-29T11:31:40.894268+0000","last_undegraded":"2024-03-29T11:31:40.894268+0000","last_fullsized":"2024-03-29T11:31:40.894268+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:17:57.806178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872517+0000","last_change":"2024-03-29T11:31:26.847903+0000","last_active":"2024-03-29T11:31:40.872517+0000","last_peered":"2024-03-29T11:31:40.872517+0000","last_clean":"2024-03-29T11:31:40.872517+0000","last_became_active":"2024-03-29T11:31:26.846171+0000","last_became_peered":"2024-03-29T11:31:26.846171+0000","last_unstale":"2024-03-29T11:31:40.872517+0000","last_undegraded":"2024-03-29T11:31:40.872517+0000","last_fullsized":"2024-03-29T11:31:40.872517+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:24.230359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951196+0000","last_change":"2024-03-29T11:31:40.372638+0000","last_active":"2024-03-29T11:31:41.951196+0000","last_peered":"2024-03-29T11:31:41.951196+0000","last_clean":"2024-03-29T11:31:41.951196+0000","last_became_active":"2024-03-29T11:31:40.372368+0000","last_became_peered":"2024-03-29T11:31:40.372368+0000","last_unstale":"2024-03-29T11:31:41.951196+0000","last_undegraded":"2024-03-29T11:31:41.951196+0000","last_fullsized":"2024-03-29T11:31:41.951196+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:27:42.526026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951223+0000","last_change":"2024-03-29T11:31:40.900805+0000","last_active":"2024-03-29T11:31:41.951223+0000","last_peered":"2024-03-29T11:31:41.951223+0000","last_clean":"2024-03-29T11:31:41.951223+0000","last_became_active":"2024-03-29T11:31:40.900541+0000","last_became_peered":"2024-03-29T11:31:40.900541+0000","last_unstale":"2024-03-29T11:31:41.951223+0000","last_undegraded":"2024-03-29T11:31:41.951223+0000","last_fullsized":"2024-03-29T11:31:41.951223+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:30.611972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882346+0000","last_change":"2024-03-29T11:31:26.310235+0000","last_active":"2024-03-29T11:31:40.882346+0000","last_peered":"2024-03-29T11:31:40.882346+0000","last_clean":"2024-03-29T11:31:40.882346+0000","last_became_active":"2024-03-29T11:31:26.309932+0000","last_became_peered":"2024-03-29T11:31:26.309932+0000","last_unstale":"2024-03-29T11:31:40.882346+0000","last_undegraded":"2024-03-29T11:31:40.882346+0000","last_fullsized":"2024-03-29T11:31:40.882346+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:31:30.532498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883923+0000","last_change":"2024-03-29T11:31:26.850244+0000","last_active":"2024-03-29T11:31:40.883923+0000","last_peered":"2024-03-29T11:31:40.883923+0000","last_clean":"2024-03-29T11:31:40.883923+0000","last_became_active":"2024-03-29T11:31:26.849736+0000","last_became_peered":"2024-03-29T11:31:26.849736+0000","last_unstale":"2024-03-29T11:31:40.883923+0000","last_undegraded":"2024-03-29T11:31:40.883923+0000","last_fullsized":"2024-03-29T11:31:40.883923+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:41:57.906301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"230'4","reported_seq":15,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950525+0000","last_change":"2024-03-29T11:31:40.366981+0000","last_active":"2024-03-29T11:31:41.950525+0000","last_peered":"2024-03-29T11:31:41.950525+0000","last_clean":"2024-03-29T11:31:41.950525+0000","last_became_active":"2024-03-29T11:31:40.366421+0000","last_became_peered":"2024-03-29T11:31:40.366421+0000","last_unstale":"2024-03-29T11:31:41.950525+0000","last_undegraded":"2024-03-29T11:31:41.950525+0000","last_fullsized":"2024-03-29T11:31:41.950525+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:42.057358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880833+0000","last_change":"2024-03-29T11:31:40.898286+0000","last_active":"2024-03-29T11:31:41.880833+0000","last_peered":"2024-03-29T11:31:41.880833+0000","last_clean":"2024-03-29T11:31:41.880833+0000","last_became_active":"2024-03-29T11:31:40.897891+0000","last_became_peered":"2024-03-29T11:31:40.897891+0000","last_unstale":"2024-03-29T11:31:41.880833+0000","last_undegraded":"2024-03-29T11:31:41.880833+0000","last_fullsized":"2024-03-29T11:31:41.880833+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:38:17.107942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1a","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.882899+0000","last_change":"2024-03-29T11:31:26.311383+0000","last_active":"2024-03-29T11:31:40.882899+0000","last_peered":"2024-03-29T11:31:40.882899+0000","last_clean":"2024-03-29T11:31:40.882899+0000","last_became_active":"2024-03-29T11:31:26.310630+0000","last_became_peered":"2024-03-29T11:31:26.310630+0000","last_unstale":"2024-03-29T11:31:40.882899+0000","last_undegraded":"2024-03-29T11:31:40.882899+0000","last_fullsized":"2024-03-29T11:31:40.882899+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:51:16.023346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872991+0000","last_change":"2024-03-29T11:31:26.853179+0000","last_active":"2024-03-29T11:31:40.872991+0000","last_peered":"2024-03-29T11:31:40.872991+0000","last_clean":"2024-03-29T11:31:40.872991+0000","last_became_active":"2024-03-29T11:31:26.848186+0000","last_became_peered":"2024-03-29T11:31:26.848186+0000","last_unstale":"2024-03-29T11:31:40.872991+0000","last_undegraded":"2024-03-29T11:31:40.872991+0000","last_fullsized":"2024-03-29T11:31:40.872991+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:35:08.770095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950726+0000","last_change":"2024-03-29T11:31:40.366984+0000","last_active":"2024-03-29T11:31:41.950726+0000","last_peered":"2024-03-29T11:31:41.950726+0000","last_clean":"2024-03-29T11:31:41.950726+0000","last_became_active":"2024-03-29T11:31:40.366706+0000","last_became_peered":"2024-03-29T11:31:40.366706+0000","last_unstale":"2024-03-29T11:31:41.950726+0000","last_undegraded":"2024-03-29T11:31:41.950726+0000","last_fullsized":"2024-03-29T11:31:41.950726+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:03.997350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950153+0000","last_change":"2024-03-29T11:31:40.904479+0000","last_active":"2024-03-29T11:31:41.950153+0000","last_peered":"2024-03-29T11:31:41.950153+0000","last_clean":"2024-03-29T11:31:41.950153+0000","last_became_active":"2024-03-29T11:31:40.904022+0000","last_became_peered":"2024-03-29T11:31:40.904022+0000","last_unstale":"2024-03-29T11:31:41.950153+0000","last_undegraded":"2024-03-29T11:31:41.950153+0000","last_fullsized":"2024-03-29T11:31:41.950153+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:44:04.775529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1c","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.874299+0000","last_change":"2024-03-29T11:31:26.321235+0000","last_active":"2024-03-29T11:31:40.874299+0000","last_peered":"2024-03-29T11:31:40.874299+0000","last_clean":"2024-03-29T11:31:40.874299+0000","last_became_active":"2024-03-29T11:31:26.320454+0000","last_became_peered":"2024-03-29T11:31:26.320454+0000","last_unstale":"2024-03-29T11:31:40.874299+0000","last_undegraded":"2024-03-29T11:31:40.874299+0000","last_fullsized":"2024-03-29T11:31:40.874299+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:54:27.238049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875962+0000","last_change":"2024-03-29T11:31:26.840358+0000","last_active":"2024-03-29T11:31:40.875962+0000","last_peered":"2024-03-29T11:31:40.875962+0000","last_clean":"2024-03-29T11:31:40.875962+0000","last_became_active":"2024-03-29T11:31:26.838477+0000","last_became_peered":"2024-03-29T11:31:26.838477+0000","last_unstale":"2024-03-29T11:31:40.875962+0000","last_undegraded":"2024-03-29T11:31:40.875962+0000","last_fullsized":"2024-03-29T11:31:40.875962+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:27.122635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880281+0000","last_change":"2024-03-29T11:31:40.366196+0000","last_active":"2024-03-29T11:31:41.880281+0000","last_peered":"2024-03-29T11:31:41.880281+0000","last_clean":"2024-03-29T11:31:41.880281+0000","last_became_active":"2024-03-29T11:31:40.365887+0000","last_became_peered":"2024-03-29T11:31:40.365887+0000","last_unstale":"2024-03-29T11:31:41.880281+0000","last_undegraded":"2024-03-29T11:31:41.880281+0000","last_fullsized":"2024-03-29T11:31:41.880281+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:27:42.526026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950426+0000","last_change":"2024-03-29T11:31:40.897244+0000","last_active":"2024-03-29T11:31:41.950426+0000","last_peered":"2024-03-29T11:31:41.950426+0000","last_clean":"2024-03-29T11:31:41.950426+0000","last_became_active":"2024-03-29T11:31:40.896998+0000","last_became_peered":"2024-03-29T11:31:40.896998+0000","last_unstale":"2024-03-29T11:31:41.950426+0000","last_undegraded":"2024-03-29T11:31:41.950426+0000","last_fullsized":"2024-03-29T11:31:41.950426+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:46.656502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1d","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883866+0000","last_change":"2024-03-29T11:31:26.319707+0000","last_active":"2024-03-29T11:31:40.883866+0000","last_peered":"2024-03-29T11:31:40.883866+0000","last_clean":"2024-03-29T11:31:40.883866+0000","last_became_active":"2024-03-29T11:31:26.316121+0000","last_became_peered":"2024-03-29T11:31:26.316121+0000","last_unstale":"2024-03-29T11:31:40.883866+0000","last_undegraded":"2024-03-29T11:31:40.883866+0000","last_fullsized":"2024-03-29T11:31:40.883866+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:12:51.039260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872617+0000","last_change":"2024-03-29T11:31:26.850407+0000","last_active":"2024-03-29T11:31:40.872617+0000","last_peered":"2024-03-29T11:31:40.872617+0000","last_clean":"2024-03-29T11:31:40.872617+0000","last_became_active":"2024-03-29T11:31:26.849457+0000","last_became_peered":"2024-03-29T11:31:26.849457+0000","last_unstale":"2024-03-29T11:31:40.872617+0000","last_undegraded":"2024-03-29T11:31:40.872617+0000","last_fullsized":"2024-03-29T11:31:40.872617+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:38:50.961636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951718+0000","last_change":"2024-03-29T11:31:40.367779+0000","last_active":"2024-03-29T11:31:41.951718+0000","last_peered":"2024-03-29T11:31:41.951718+0000","last_clean":"2024-03-29T11:31:41.951718+0000","last_became_active":"2024-03-29T11:31:40.367158+0000","last_became_peered":"2024-03-29T11:31:40.367158+0000","last_unstale":"2024-03-29T11:31:41.951718+0000","last_undegraded":"2024-03-29T11:31:41.951718+0000","last_fullsized":"2024-03-29T11:31:41.951718+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:38:16.100705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879473+0000","last_change":"2024-03-29T11:31:40.891827+0000","last_active":"2024-03-29T11:31:41.879473+0000","last_peered":"2024-03-29T11:31:41.879473+0000","last_clean":"2024-03-29T11:31:41.879473+0000","last_became_active":"2024-03-29T11:31:40.891565+0000","last_became_peered":"2024-03-29T11:31:40.891565+0000","last_unstale":"2024-03-29T11:31:41.879473+0000","last_undegraded":"2024-03-29T11:31:41.879473+0000","last_fullsized":"2024-03-29T11:31:41.879473+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:58.211565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1e","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883958+0000","last_change":"2024-03-29T11:31:26.320463+0000","last_active":"2024-03-29T11:31:40.883958+0000","last_peered":"2024-03-29T11:31:40.883958+0000","last_clean":"2024-03-29T11:31:40.883958+0000","last_became_active":"2024-03-29T11:31:26.319713+0000","last_became_peered":"2024-03-29T11:31:26.319713+0000","last_unstale":"2024-03-29T11:31:40.883958+0000","last_undegraded":"2024-03-29T11:31:40.883958+0000","last_fullsized":"2024-03-29T11:31:40.883958+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:32:47.003355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885062+0000","last_change":"2024-03-29T11:31:26.843531+0000","last_active":"2024-03-29T11:31:40.885062+0000","last_peered":"2024-03-29T11:31:40.885062+0000","last_clean":"2024-03-29T11:31:40.885062+0000","last_became_active":"2024-03-29T11:31:26.843164+0000","last_became_peered":"2024-03-29T11:31:26.843164+0000","last_unstale":"2024-03-29T11:31:40.885062+0000","last_undegraded":"2024-03-29T11:31:40.885062+0000","last_fullsized":"2024-03-29T11:31:40.885062+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:29:23.515574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.873648+0000","last_change":"2024-03-29T11:31:40.366096+0000","last_active":"2024-03-29T11:31:41.873648+0000","last_peered":"2024-03-29T11:31:41.873648+0000","last_clean":"2024-03-29T11:31:41.873648+0000","last_became_active":"2024-03-29T11:31:40.365769+0000","last_became_peered":"2024-03-29T11:31:40.365769+0000","last_unstale":"2024-03-29T11:31:41.873648+0000","last_undegraded":"2024-03-29T11:31:41.873648+0000","last_fullsized":"2024-03-29T11:31:41.873648+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:10:45.255586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949865+0000","last_change":"2024-03-29T11:31:40.893845+0000","last_active":"2024-03-29T11:31:41.949865+0000","last_peered":"2024-03-29T11:31:41.949865+0000","last_clean":"2024-03-29T11:31:41.949865+0000","last_became_active":"2024-03-29T11:31:40.891813+0000","last_became_peered":"2024-03-29T11:31:40.891813+0000","last_unstale":"2024-03-29T11:31:41.949865+0000","last_undegraded":"2024-03-29T11:31:41.949865+0000","last_fullsized":"2024-03-29T11:31:41.949865+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:38.273310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1f","version":"226'4","reported_seq":19,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875640+0000","last_change":"2024-03-29T11:31:26.315269+0000","last_active":"2024-03-29T11:31:40.875640+0000","last_peered":"2024-03-29T11:31:40.875640+0000","last_clean":"2024-03-29T11:31:40.875640+0000","last_became_active":"2024-03-29T11:31:26.314956+0000","last_became_peered":"2024-03-29T11:31:26.314956+0000","last_unstale":"2024-03-29T11:31:40.875640+0000","last_undegraded":"2024-03-29T11:31:40.875640+0000","last_fullsized":"2024-03-29T11:31:40.875640+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:25.289417+0000","last_clean_scrub_stamp":"2024-03-29T11:31:25.289417+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:18.302941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884150+0000","last_change":"2024-03-29T11:31:26.844474+0000","last_active":"2024-03-29T11:31:40.884150+0000","last_peered":"2024-03-29T11:31:40.884150+0000","last_clean":"2024-03-29T11:31:40.884150+0000","last_became_active":"2024-03-29T11:31:26.844195+0000","last_became_peered":"2024-03-29T11:31:26.844195+0000","last_unstale":"2024-03-29T11:31:40.884150+0000","last_undegraded":"2024-03-29T11:31:40.884150+0000","last_fullsized":"2024-03-29T11:31:40.884150+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:38:37.668282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949371+0000","last_change":"2024-03-29T11:31:40.368937+0000","last_active":"2024-03-29T11:31:41.949371+0000","last_peered":"2024-03-29T11:31:41.949371+0000","last_clean":"2024-03-29T11:31:41.949371+0000","last_became_active":"2024-03-29T11:31:40.368617+0000","last_became_peered":"2024-03-29T11:31:40.368617+0000","last_unstale":"2024-03-29T11:31:41.949371+0000","last_undegraded":"2024-03-29T11:31:41.949371+0000","last_fullsized":"2024-03-29T11:31:41.949371+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:39.327778+0000","last_clean_scrub_stamp":"2024-03-29T11:31:39.327778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:22.207936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.873719+0000","last_change":"2024-03-29T11:31:40.898328+0000","last_active":"2024-03-29T11:31:41.873719+0000","last_peered":"2024-03-29T11:31:41.873719+0000","last_clean":"2024-03-29T11:31:41.873719+0000","last_became_active":"2024-03-29T11:31:40.898058+0000","last_became_peered":"2024-03-29T11:31:40.898058+0000","last_unstale":"2024-03-29T11:31:41.873719+0000","last_undegraded":"2024-03-29T11:31:41.873719+0000","last_fullsized":"2024-03-29T11:31:41.873719+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:58:55.577093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951337+0000","last_change":"2024-03-29T11:31:40.891542+0000","last_active":"2024-03-29T11:31:41.951337+0000","last_peered":"2024-03-29T11:31:41.951337+0000","last_clean":"2024-03-29T11:31:41.951337+0000","last_became_active":"2024-03-29T11:31:40.889902+0000","last_became_peered":"2024-03-29T11:31:40.889902+0000","last_unstale":"2024-03-29T11:31:41.951337+0000","last_undegraded":"2024-03-29T11:31:41.951337+0000","last_fullsized":"2024-03-29T11:31:41.951337+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:55.281545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870195+0000","last_change":"2024-03-29T11:31:26.839138+0000","last_active":"2024-03-29T11:31:40.870195+0000","last_peered":"2024-03-29T11:31:40.870195+0000","last_clean":"2024-03-29T11:31:40.870195+0000","last_became_active":"2024-03-29T11:31:26.838904+0000","last_became_peered":"2024-03-29T11:31:26.838904+0000","last_unstale":"2024-03-29T11:31:40.870195+0000","last_undegraded":"2024-03-29T11:31:40.870195+0000","last_fullsized":"2024-03-29T11:31:40.870195+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:54.107881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880429+0000","last_change":"2024-03-29T11:31:40.904048+0000","last_active":"2024-03-29T11:31:41.880429+0000","last_peered":"2024-03-29T11:31:41.880429+0000","last_clean":"2024-03-29T11:31:41.880429+0000","last_became_active":"2024-03-29T11:31:40.903770+0000","last_became_peered":"2024-03-29T11:31:40.903770+0000","last_unstale":"2024-03-29T11:31:41.880429+0000","last_undegraded":"2024-03-29T11:31:41.880429+0000","last_fullsized":"2024-03-29T11:31:41.880429+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:17.301421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875501+0000","last_change":"2024-03-29T11:31:26.847737+0000","last_active":"2024-03-29T11:31:40.875501+0000","last_peered":"2024-03-29T11:31:40.875501+0000","last_clean":"2024-03-29T11:31:40.875501+0000","last_became_active":"2024-03-29T11:31:26.847473+0000","last_became_peered":"2024-03-29T11:31:26.847473+0000","last_unstale":"2024-03-29T11:31:40.875501+0000","last_undegraded":"2024-03-29T11:31:40.875501+0000","last_fullsized":"2024-03-29T11:31:40.875501+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:36:05.541971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.891946+0000","last_change":"2024-03-29T11:31:26.842077+0000","last_active":"2024-03-29T11:31:40.891946+0000","last_peered":"2024-03-29T11:31:40.891946+0000","last_clean":"2024-03-29T11:31:40.891946+0000","last_became_active":"2024-03-29T11:31:26.841818+0000","last_became_peered":"2024-03-29T11:31:26.841818+0000","last_unstale":"2024-03-29T11:31:40.891946+0000","last_undegraded":"2024-03-29T11:31:40.891946+0000","last_fullsized":"2024-03-29T11:31:40.891946+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:42:57.193431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874594+0000","last_change":"2024-03-29T11:31:40.904727+0000","last_active":"2024-03-29T11:31:41.874594+0000","last_peered":"2024-03-29T11:31:41.874594+0000","last_clean":"2024-03-29T11:31:41.874594+0000","last_became_active":"2024-03-29T11:31:40.904081+0000","last_became_peered":"2024-03-29T11:31:40.904081+0000","last_unstale":"2024-03-29T11:31:41.874594+0000","last_undegraded":"2024-03-29T11:31:41.874594+0000","last_fullsized":"2024-03-29T11:31:41.874594+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:42.265960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.878841+0000","last_change":"2024-03-29T11:31:40.907166+0000","last_active":"2024-03-29T11:31:41.878841+0000","last_peered":"2024-03-29T11:31:41.878841+0000","last_clean":"2024-03-29T11:31:41.878841+0000","last_became_active":"2024-03-29T11:31:40.905090+0000","last_became_peered":"2024-03-29T11:31:40.905090+0000","last_unstale":"2024-03-29T11:31:41.878841+0000","last_undegraded":"2024-03-29T11:31:41.878841+0000","last_fullsized":"2024-03-29T11:31:41.878841+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:36.816868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875554+0000","last_change":"2024-03-29T11:31:26.848961+0000","last_active":"2024-03-29T11:31:40.875554+0000","last_peered":"2024-03-29T11:31:40.875554+0000","last_clean":"2024-03-29T11:31:40.875554+0000","last_became_active":"2024-03-29T11:31:26.848637+0000","last_became_peered":"2024-03-29T11:31:26.848637+0000","last_unstale":"2024-03-29T11:31:40.875554+0000","last_undegraded":"2024-03-29T11:31:40.875554+0000","last_fullsized":"2024-03-29T11:31:40.875554+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:31:51.171001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.883480+0000","last_change":"2024-03-29T11:31:26.838393+0000","last_active":"2024-03-29T11:31:40.883480+0000","last_peered":"2024-03-29T11:31:40.883480+0000","last_clean":"2024-03-29T11:31:40.883480+0000","last_became_active":"2024-03-29T11:31:26.838003+0000","last_became_peered":"2024-03-29T11:31:26.838003+0000","last_unstale":"2024-03-29T11:31:40.883480+0000","last_undegraded":"2024-03-29T11:31:40.883480+0000","last_fullsized":"2024-03-29T11:31:40.883480+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:35:47.195933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879316+0000","last_change":"2024-03-29T11:31:40.892720+0000","last_active":"2024-03-29T11:31:41.879316+0000","last_peered":"2024-03-29T11:31:41.879316+0000","last_clean":"2024-03-29T11:31:41.879316+0000","last_became_active":"2024-03-29T11:31:40.892425+0000","last_became_peered":"2024-03-29T11:31:40.892425+0000","last_unstale":"2024-03-29T11:31:41.879316+0000","last_undegraded":"2024-03-29T11:31:41.879316+0000","last_fullsized":"2024-03-29T11:31:41.879316+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:46.155852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.879257+0000","last_change":"2024-03-29T11:31:26.852411+0000","last_active":"2024-03-29T11:31:40.879257+0000","last_peered":"2024-03-29T11:31:40.879257+0000","last_clean":"2024-03-29T11:31:40.879257+0000","last_became_active":"2024-03-29T11:31:26.852100+0000","last_became_peered":"2024-03-29T11:31:26.852100+0000","last_unstale":"2024-03-29T11:31:40.879257+0000","last_undegraded":"2024-03-29T11:31:40.879257+0000","last_fullsized":"2024-03-29T11:31:40.879257+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:15:49.012984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949498+0000","last_change":"2024-03-29T11:31:40.904368+0000","last_active":"2024-03-29T11:31:41.949498+0000","last_peered":"2024-03-29T11:31:41.949498+0000","last_clean":"2024-03-29T11:31:41.949498+0000","last_became_active":"2024-03-29T11:31:40.903294+0000","last_became_peered":"2024-03-29T11:31:40.903294+0000","last_unstale":"2024-03-29T11:31:41.949498+0000","last_undegraded":"2024-03-29T11:31:41.949498+0000","last_fullsized":"2024-03-29T11:31:41.949498+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:38:17.107942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872844+0000","last_change":"2024-03-29T11:31:26.843275+0000","last_active":"2024-03-29T11:31:40.872844+0000","last_peered":"2024-03-29T11:31:40.872844+0000","last_clean":"2024-03-29T11:31:40.872844+0000","last_became_active":"2024-03-29T11:31:26.842588+0000","last_became_peered":"2024-03-29T11:31:26.842588+0000","last_unstale":"2024-03-29T11:31:40.872844+0000","last_undegraded":"2024-03-29T11:31:40.872844+0000","last_fullsized":"2024-03-29T11:31:40.872844+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:57.876055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.948965+0000","last_change":"2024-03-29T11:31:40.890546+0000","last_active":"2024-03-29T11:31:41.948965+0000","last_peered":"2024-03-29T11:31:41.948965+0000","last_clean":"2024-03-29T11:31:41.948965+0000","last_became_active":"2024-03-29T11:31:40.890297+0000","last_became_peered":"2024-03-29T11:31:40.890297+0000","last_unstale":"2024-03-29T11:31:41.948965+0000","last_undegraded":"2024-03-29T11:31:41.948965+0000","last_fullsized":"2024-03-29T11:31:41.948965+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:36:29.757453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.889285+0000","last_change":"2024-03-29T11:31:26.847195+0000","last_active":"2024-03-29T11:31:40.889285+0000","last_peered":"2024-03-29T11:31:40.889285+0000","last_clean":"2024-03-29T11:31:40.889285+0000","last_became_active":"2024-03-29T11:31:26.846876+0000","last_became_peered":"2024-03-29T11:31:26.846876+0000","last_unstale":"2024-03-29T11:31:40.889285+0000","last_undegraded":"2024-03-29T11:31:40.889285+0000","last_fullsized":"2024-03-29T11:31:40.889285+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:14:41.350201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.948680+0000","last_change":"2024-03-29T11:31:40.890331+0000","last_active":"2024-03-29T11:31:41.948680+0000","last_peered":"2024-03-29T11:31:41.948680+0000","last_clean":"2024-03-29T11:31:41.948680+0000","last_became_active":"2024-03-29T11:31:40.886395+0000","last_became_peered":"2024-03-29T11:31:40.886395+0000","last_unstale":"2024-03-29T11:31:41.948680+0000","last_undegraded":"2024-03-29T11:31:41.948680+0000","last_fullsized":"2024-03-29T11:31:41.948680+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:38.273310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874691+0000","last_change":"2024-03-29T11:31:40.893585+0000","last_active":"2024-03-29T11:31:41.874691+0000","last_peered":"2024-03-29T11:31:41.874691+0000","last_clean":"2024-03-29T11:31:41.874691+0000","last_became_active":"2024-03-29T11:31:40.893333+0000","last_became_peered":"2024-03-29T11:31:40.893333+0000","last_unstale":"2024-03-29T11:31:41.874691+0000","last_undegraded":"2024-03-29T11:31:41.874691+0000","last_fullsized":"2024-03-29T11:31:41.874691+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:37.010261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870753+0000","last_change":"2024-03-29T11:31:26.855506+0000","last_active":"2024-03-29T11:31:40.870753+0000","last_peered":"2024-03-29T11:31:40.870753+0000","last_clean":"2024-03-29T11:31:40.870753+0000","last_became_active":"2024-03-29T11:31:26.852090+0000","last_became_peered":"2024-03-29T11:31:26.852090+0000","last_unstale":"2024-03-29T11:31:40.870753+0000","last_undegraded":"2024-03-29T11:31:40.870753+0000","last_fullsized":"2024-03-29T11:31:40.870753+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:09.172222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951113+0000","last_change":"2024-03-29T11:31:40.896412+0000","last_active":"2024-03-29T11:31:41.951113+0000","last_peered":"2024-03-29T11:31:41.951113+0000","last_clean":"2024-03-29T11:31:41.951113+0000","last_became_active":"2024-03-29T11:31:40.896123+0000","last_became_peered":"2024-03-29T11:31:40.896123+0000","last_unstale":"2024-03-29T11:31:41.951113+0000","last_undegraded":"2024-03-29T11:31:41.951113+0000","last_fullsized":"2024-03-29T11:31:41.951113+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:48:28.663360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.886751+0000","last_change":"2024-03-29T11:31:26.842715+0000","last_active":"2024-03-29T11:31:40.886751+0000","last_peered":"2024-03-29T11:31:40.886751+0000","last_clean":"2024-03-29T11:31:40.886751+0000","last_became_active":"2024-03-29T11:31:26.842354+0000","last_became_peered":"2024-03-29T11:31:26.842354+0000","last_unstale":"2024-03-29T11:31:40.886751+0000","last_undegraded":"2024-03-29T11:31:40.886751+0000","last_fullsized":"2024-03-29T11:31:40.886751+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:58.653895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874662+0000","last_change":"2024-03-29T11:31:40.878065+0000","last_active":"2024-03-29T11:31:41.874662+0000","last_peered":"2024-03-29T11:31:41.874662+0000","last_clean":"2024-03-29T11:31:41.874662+0000","last_became_active":"2024-03-29T11:31:40.877784+0000","last_became_peered":"2024-03-29T11:31:40.877784+0000","last_unstale":"2024-03-29T11:31:41.874662+0000","last_undegraded":"2024-03-29T11:31:41.874662+0000","last_fullsized":"2024-03-29T11:31:41.874662+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:13:00.089904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.885512+0000","last_change":"2024-03-29T11:31:26.834482+0000","last_active":"2024-03-29T11:31:40.885512+0000","last_peered":"2024-03-29T11:31:40.885512+0000","last_clean":"2024-03-29T11:31:40.885512+0000","last_became_active":"2024-03-29T11:31:26.833118+0000","last_became_peered":"2024-03-29T11:31:26.833118+0000","last_unstale":"2024-03-29T11:31:40.885512+0000","last_undegraded":"2024-03-29T11:31:40.885512+0000","last_fullsized":"2024-03-29T11:31:40.885512+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:52:42.629666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950997+0000","last_change":"2024-03-29T11:31:40.902550+0000","last_active":"2024-03-29T11:31:41.950997+0000","last_peered":"2024-03-29T11:31:41.950997+0000","last_clean":"2024-03-29T11:31:41.950997+0000","last_became_active":"2024-03-29T11:31:40.899855+0000","last_became_peered":"2024-03-29T11:31:40.899855+0000","last_unstale":"2024-03-29T11:31:41.950997+0000","last_undegraded":"2024-03-29T11:31:41.950997+0000","last_fullsized":"2024-03-29T11:31:41.950997+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:36:51.759675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873153+0000","last_change":"2024-03-29T11:31:26.856449+0000","last_active":"2024-03-29T11:31:40.873153+0000","last_peered":"2024-03-29T11:31:40.873153+0000","last_clean":"2024-03-29T11:31:40.873153+0000","last_became_active":"2024-03-29T11:31:26.856166+0000","last_became_peered":"2024-03-29T11:31:26.856166+0000","last_unstale":"2024-03-29T11:31:40.873153+0000","last_undegraded":"2024-03-29T11:31:40.873153+0000","last_fullsized":"2024-03-29T11:31:40.873153+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:57:27.122635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949406+0000","last_change":"2024-03-29T11:31:40.890141+0000","last_active":"2024-03-29T11:31:41.949406+0000","last_peered":"2024-03-29T11:31:41.949406+0000","last_clean":"2024-03-29T11:31:41.949406+0000","last_became_active":"2024-03-29T11:31:40.889862+0000","last_became_peered":"2024-03-29T11:31:40.889862+0000","last_unstale":"2024-03-29T11:31:41.949406+0000","last_undegraded":"2024-03-29T11:31:41.949406+0000","last_fullsized":"2024-03-29T11:31:41.949406+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:28:27.804989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.892869+0000","last_change":"2024-03-29T11:31:26.843448+0000","last_active":"2024-03-29T11:31:40.892869+0000","last_peered":"2024-03-29T11:31:40.892869+0000","last_clean":"2024-03-29T11:31:40.892869+0000","last_became_active":"2024-03-29T11:31:26.843183+0000","last_became_peered":"2024-03-29T11:31:26.843183+0000","last_unstale":"2024-03-29T11:31:40.892869+0000","last_undegraded":"2024-03-29T11:31:40.892869+0000","last_fullsized":"2024-03-29T11:31:40.892869+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:23:23.236057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950159+0000","last_change":"2024-03-29T11:31:40.901765+0000","last_active":"2024-03-29T11:31:41.950159+0000","last_peered":"2024-03-29T11:31:41.950159+0000","last_clean":"2024-03-29T11:31:41.950159+0000","last_became_active":"2024-03-29T11:31:40.899905+0000","last_became_peered":"2024-03-29T11:31:40.899905+0000","last_unstale":"2024-03-29T11:31:41.950159+0000","last_undegraded":"2024-03-29T11:31:41.950159+0000","last_fullsized":"2024-03-29T11:31:41.950159+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:13.874339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875882+0000","last_change":"2024-03-29T11:31:26.854890+0000","last_active":"2024-03-29T11:31:40.875882+0000","last_peered":"2024-03-29T11:31:40.875882+0000","last_clean":"2024-03-29T11:31:40.875882+0000","last_became_active":"2024-03-29T11:31:26.854625+0000","last_became_peered":"2024-03-29T11:31:26.854625+0000","last_unstale":"2024-03-29T11:31:40.875882+0000","last_undegraded":"2024-03-29T11:31:40.875882+0000","last_fullsized":"2024-03-29T11:31:40.875882+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.831388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.948696+0000","last_change":"2024-03-29T11:31:40.885062+0000","last_active":"2024-03-29T11:31:41.948696+0000","last_peered":"2024-03-29T11:31:41.948696+0000","last_clean":"2024-03-29T11:31:41.948696+0000","last_became_active":"2024-03-29T11:31:40.884786+0000","last_became_peered":"2024-03-29T11:31:40.884786+0000","last_unstale":"2024-03-29T11:31:41.948696+0000","last_undegraded":"2024-03-29T11:31:41.948696+0000","last_fullsized":"2024-03-29T11:31:41.948696+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:09:49.649528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876629+0000","last_change":"2024-03-29T11:31:26.840048+0000","last_active":"2024-03-29T11:31:40.876629+0000","last_peered":"2024-03-29T11:31:40.876629+0000","last_clean":"2024-03-29T11:31:40.876629+0000","last_became_active":"2024-03-29T11:31:26.838725+0000","last_became_peered":"2024-03-29T11:31:26.838725+0000","last_unstale":"2024-03-29T11:31:40.876629+0000","last_undegraded":"2024-03-29T11:31:40.876629+0000","last_fullsized":"2024-03-29T11:31:40.876629+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:09.172222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.878352+0000","last_change":"2024-03-29T11:31:40.884649+0000","last_active":"2024-03-29T11:31:41.878352+0000","last_peered":"2024-03-29T11:31:41.878352+0000","last_clean":"2024-03-29T11:31:41.878352+0000","last_became_active":"2024-03-29T11:31:40.884374+0000","last_became_peered":"2024-03-29T11:31:40.884374+0000","last_unstale":"2024-03-29T11:31:41.878352+0000","last_undegraded":"2024-03-29T11:31:41.878352+0000","last_fullsized":"2024-03-29T11:31:41.878352+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:57:25.207829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872907+0000","last_change":"2024-03-29T11:31:26.850388+0000","last_active":"2024-03-29T11:31:40.872907+0000","last_peered":"2024-03-29T11:31:40.872907+0000","last_clean":"2024-03-29T11:31:40.872907+0000","last_became_active":"2024-03-29T11:31:26.849826+0000","last_became_peered":"2024-03-29T11:31:26.849826+0000","last_unstale":"2024-03-29T11:31:40.872907+0000","last_undegraded":"2024-03-29T11:31:40.872907+0000","last_fullsized":"2024-03-29T11:31:40.872907+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:55:45.796741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.874662+0000","last_change":"2024-03-29T11:31:40.884799+0000","last_active":"2024-03-29T11:31:41.874662+0000","last_peered":"2024-03-29T11:31:41.874662+0000","last_clean":"2024-03-29T11:31:41.874662+0000","last_became_active":"2024-03-29T11:31:40.884544+0000","last_became_peered":"2024-03-29T11:31:40.884544+0000","last_unstale":"2024-03-29T11:31:41.874662+0000","last_undegraded":"2024-03-29T11:31:41.874662+0000","last_fullsized":"2024-03-29T11:31:41.874662+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:51.749739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875157+0000","last_change":"2024-03-29T11:31:26.843040+0000","last_active":"2024-03-29T11:31:40.875157+0000","last_peered":"2024-03-29T11:31:40.875157+0000","last_clean":"2024-03-29T11:31:40.875157+0000","last_became_active":"2024-03-29T11:31:26.841805+0000","last_became_peered":"2024-03-29T11:31:26.841805+0000","last_unstale":"2024-03-29T11:31:40.875157+0000","last_undegraded":"2024-03-29T11:31:40.875157+0000","last_fullsized":"2024-03-29T11:31:40.875157+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:05.909043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.881086+0000","last_change":"2024-03-29T11:31:40.890262+0000","last_active":"2024-03-29T11:31:41.881086+0000","last_peered":"2024-03-29T11:31:41.881086+0000","last_clean":"2024-03-29T11:31:41.881086+0000","last_became_active":"2024-03-29T11:31:40.889385+0000","last_became_peered":"2024-03-29T11:31:40.889385+0000","last_unstale":"2024-03-29T11:31:41.881086+0000","last_undegraded":"2024-03-29T11:31:41.881086+0000","last_fullsized":"2024-03-29T11:31:41.881086+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:28:36.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870273+0000","last_change":"2024-03-29T11:31:26.843972+0000","last_active":"2024-03-29T11:31:40.870273+0000","last_peered":"2024-03-29T11:31:40.870273+0000","last_clean":"2024-03-29T11:31:40.870273+0000","last_became_active":"2024-03-29T11:31:26.843713+0000","last_became_peered":"2024-03-29T11:31:26.843713+0000","last_unstale":"2024-03-29T11:31:40.870273+0000","last_undegraded":"2024-03-29T11:31:40.870273+0000","last_fullsized":"2024-03-29T11:31:40.870273+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:38.270854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950667+0000","last_change":"2024-03-29T11:31:40.908373+0000","last_active":"2024-03-29T11:31:41.950667+0000","last_peered":"2024-03-29T11:31:41.950667+0000","last_clean":"2024-03-29T11:31:41.950667+0000","last_became_active":"2024-03-29T11:31:40.906929+0000","last_became_peered":"2024-03-29T11:31:40.906929+0000","last_unstale":"2024-03-29T11:31:41.950667+0000","last_undegraded":"2024-03-29T11:31:41.950667+0000","last_fullsized":"2024-03-29T11:31:41.950667+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:28:36.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.873322+0000","last_change":"2024-03-29T11:31:26.848452+0000","last_active":"2024-03-29T11:31:40.873322+0000","last_peered":"2024-03-29T11:31:40.873322+0000","last_clean":"2024-03-29T11:31:40.873322+0000","last_became_active":"2024-03-29T11:31:26.846605+0000","last_became_peered":"2024-03-29T11:31:26.846605+0000","last_unstale":"2024-03-29T11:31:40.873322+0000","last_undegraded":"2024-03-29T11:31:40.873322+0000","last_fullsized":"2024-03-29T11:31:40.873322+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:57.163908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.878649+0000","last_change":"2024-03-29T11:31:40.905176+0000","last_active":"2024-03-29T11:31:41.878649+0000","last_peered":"2024-03-29T11:31:41.878649+0000","last_clean":"2024-03-29T11:31:41.878649+0000","last_became_active":"2024-03-29T11:31:40.904390+0000","last_became_peered":"2024-03-29T11:31:40.904390+0000","last_unstale":"2024-03-29T11:31:41.878649+0000","last_undegraded":"2024-03-29T11:31:41.878649+0000","last_fullsized":"2024-03-29T11:31:41.878649+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:14:20.031001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.875645+0000","last_change":"2024-03-29T11:31:26.848377+0000","last_active":"2024-03-29T11:31:40.875645+0000","last_peered":"2024-03-29T11:31:40.875645+0000","last_clean":"2024-03-29T11:31:40.875645+0000","last_became_active":"2024-03-29T11:31:26.847564+0000","last_became_peered":"2024-03-29T11:31:26.847564+0000","last_unstale":"2024-03-29T11:31:40.875645+0000","last_undegraded":"2024-03-29T11:31:40.875645+0000","last_fullsized":"2024-03-29T11:31:40.875645+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:40:17.980944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949830+0000","last_change":"2024-03-29T11:31:40.898642+0000","last_active":"2024-03-29T11:31:41.949830+0000","last_peered":"2024-03-29T11:31:41.949830+0000","last_clean":"2024-03-29T11:31:41.949830+0000","last_became_active":"2024-03-29T11:31:40.898101+0000","last_became_peered":"2024-03-29T11:31:40.898101+0000","last_unstale":"2024-03-29T11:31:41.949830+0000","last_undegraded":"2024-03-29T11:31:41.949830+0000","last_fullsized":"2024-03-29T11:31:41.949830+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:32:36.867108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876930+0000","last_change":"2024-03-29T11:31:26.848052+0000","last_active":"2024-03-29T11:31:40.876930+0000","last_peered":"2024-03-29T11:31:40.876930+0000","last_clean":"2024-03-29T11:31:40.876930+0000","last_became_active":"2024-03-29T11:31:26.847789+0000","last_became_peered":"2024-03-29T11:31:26.847789+0000","last_unstale":"2024-03-29T11:31:40.876930+0000","last_undegraded":"2024-03-29T11:31:40.876930+0000","last_fullsized":"2024-03-29T11:31:40.876930+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:23:23.236057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949909+0000","last_change":"2024-03-29T11:31:40.898685+0000","last_active":"2024-03-29T11:31:41.949909+0000","last_peered":"2024-03-29T11:31:41.949909+0000","last_clean":"2024-03-29T11:31:41.949909+0000","last_became_active":"2024-03-29T11:31:40.898419+0000","last_became_peered":"2024-03-29T11:31:40.898419+0000","last_unstale":"2024-03-29T11:31:41.949909+0000","last_undegraded":"2024-03-29T11:31:41.949909+0000","last_fullsized":"2024-03-29T11:31:41.949909+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:46.656502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884150+0000","last_change":"2024-03-29T11:31:26.841262+0000","last_active":"2024-03-29T11:31:40.884150+0000","last_peered":"2024-03-29T11:31:40.884150+0000","last_clean":"2024-03-29T11:31:40.884150+0000","last_became_active":"2024-03-29T11:31:26.841004+0000","last_became_peered":"2024-03-29T11:31:26.841004+0000","last_unstale":"2024-03-29T11:31:40.884150+0000","last_undegraded":"2024-03-29T11:31:40.884150+0000","last_fullsized":"2024-03-29T11:31:40.884150+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:54.107881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949636+0000","last_change":"2024-03-29T11:31:40.903665+0000","last_active":"2024-03-29T11:31:41.949636+0000","last_peered":"2024-03-29T11:31:41.949636+0000","last_clean":"2024-03-29T11:31:41.949636+0000","last_became_active":"2024-03-29T11:31:40.903409+0000","last_became_peered":"2024-03-29T11:31:40.903409+0000","last_unstale":"2024-03-29T11:31:41.949636+0000","last_undegraded":"2024-03-29T11:31:41.949636+0000","last_fullsized":"2024-03-29T11:31:41.949636+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:13.874339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.876591+0000","last_change":"2024-03-29T11:31:26.853942+0000","last_active":"2024-03-29T11:31:40.876591+0000","last_peered":"2024-03-29T11:31:40.876591+0000","last_clean":"2024-03-29T11:31:40.876591+0000","last_became_active":"2024-03-29T11:31:26.853648+0000","last_became_peered":"2024-03-29T11:31:26.853648+0000","last_unstale":"2024-03-29T11:31:40.876591+0000","last_undegraded":"2024-03-29T11:31:40.876591+0000","last_fullsized":"2024-03-29T11:31:40.876591+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:24.230359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949341+0000","last_change":"2024-03-29T11:31:40.887123+0000","last_active":"2024-03-29T11:31:41.949341+0000","last_peered":"2024-03-29T11:31:41.949341+0000","last_clean":"2024-03-29T11:31:41.949341+0000","last_became_active":"2024-03-29T11:31:40.886835+0000","last_became_peered":"2024-03-29T11:31:40.886835+0000","last_unstale":"2024-03-29T11:31:41.949341+0000","last_undegraded":"2024-03-29T11:31:41.949341+0000","last_fullsized":"2024-03-29T11:31:41.949341+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:01:54.265645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.889378+0000","last_change":"2024-03-29T11:31:26.851850+0000","last_active":"2024-03-29T11:31:40.889378+0000","last_peered":"2024-03-29T11:31:40.889378+0000","last_clean":"2024-03-29T11:31:40.889378+0000","last_became_active":"2024-03-29T11:31:26.848888+0000","last_became_peered":"2024-03-29T11:31:26.848888+0000","last_unstale":"2024-03-29T11:31:40.889378+0000","last_undegraded":"2024-03-29T11:31:40.889378+0000","last_fullsized":"2024-03-29T11:31:40.889378+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:26:36.183185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.951339+0000","last_change":"2024-03-29T11:31:40.901556+0000","last_active":"2024-03-29T11:31:41.951339+0000","last_peered":"2024-03-29T11:31:41.951339+0000","last_clean":"2024-03-29T11:31:41.951339+0000","last_became_active":"2024-03-29T11:31:40.901278+0000","last_became_peered":"2024-03-29T11:31:40.901278+0000","last_unstale":"2024-03-29T11:31:41.951339+0000","last_undegraded":"2024-03-29T11:31:41.951339+0000","last_fullsized":"2024-03-29T11:31:41.951339+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:09:49.649528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884142+0000","last_change":"2024-03-29T11:31:26.833794+0000","last_active":"2024-03-29T11:31:40.884142+0000","last_peered":"2024-03-29T11:31:40.884142+0000","last_clean":"2024-03-29T11:31:40.884142+0000","last_became_active":"2024-03-29T11:31:26.833440+0000","last_became_peered":"2024-03-29T11:31:26.833440+0000","last_unstale":"2024-03-29T11:31:40.884142+0000","last_undegraded":"2024-03-29T11:31:40.884142+0000","last_fullsized":"2024-03-29T11:31:40.884142+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:07.336827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880321+0000","last_change":"2024-03-29T11:31:40.904700+0000","last_active":"2024-03-29T11:31:41.880321+0000","last_peered":"2024-03-29T11:31:41.880321+0000","last_clean":"2024-03-29T11:31:41.880321+0000","last_became_active":"2024-03-29T11:31:40.904009+0000","last_became_peered":"2024-03-29T11:31:40.904009+0000","last_unstale":"2024-03-29T11:31:41.880321+0000","last_undegraded":"2024-03-29T11:31:41.880321+0000","last_fullsized":"2024-03-29T11:31:41.880321+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:19:46.539890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.870188+0000","last_change":"2024-03-29T11:31:26.836892+0000","last_active":"2024-03-29T11:31:40.870188+0000","last_peered":"2024-03-29T11:31:40.870188+0000","last_clean":"2024-03-29T11:31:40.870188+0000","last_became_active":"2024-03-29T11:31:26.832344+0000","last_became_peered":"2024-03-29T11:31:26.832344+0000","last_unstale":"2024-03-29T11:31:40.870188+0000","last_undegraded":"2024-03-29T11:31:40.870188+0000","last_fullsized":"2024-03-29T11:31:40.870188+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:27:29.490312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950322+0000","last_change":"2024-03-29T11:31:40.903357+0000","last_active":"2024-03-29T11:31:41.950322+0000","last_peered":"2024-03-29T11:31:41.950322+0000","last_clean":"2024-03-29T11:31:41.950322+0000","last_became_active":"2024-03-29T11:31:40.903091+0000","last_became_peered":"2024-03-29T11:31:40.903091+0000","last_unstale":"2024-03-29T11:31:41.950322+0000","last_undegraded":"2024-03-29T11:31:41.950322+0000","last_fullsized":"2024-03-29T11:31:41.950322+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:26:19.372364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.892440+0000","last_change":"2024-03-29T11:31:26.836146+0000","last_active":"2024-03-29T11:31:40.892440+0000","last_peered":"2024-03-29T11:31:40.892440+0000","last_clean":"2024-03-29T11:31:40.892440+0000","last_became_active":"2024-03-29T11:31:26.835185+0000","last_became_peered":"2024-03-29T11:31:26.835185+0000","last_unstale":"2024-03-29T11:31:40.892440+0000","last_undegraded":"2024-03-29T11:31:40.892440+0000","last_fullsized":"2024-03-29T11:31:40.892440+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:15:49.012984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.880903+0000","last_change":"2024-03-29T11:31:40.905024+0000","last_active":"2024-03-29T11:31:41.880903+0000","last_peered":"2024-03-29T11:31:41.880903+0000","last_clean":"2024-03-29T11:31:41.880903+0000","last_became_active":"2024-03-29T11:31:40.904676+0000","last_became_peered":"2024-03-29T11:31:40.904676+0000","last_unstale":"2024-03-29T11:31:41.880903+0000","last_undegraded":"2024-03-29T11:31:41.880903+0000","last_fullsized":"2024-03-29T11:31:41.880903+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:29:11.480583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.892264+0000","last_change":"2024-03-29T11:31:26.841559+0000","last_active":"2024-03-29T11:31:40.892264+0000","last_peered":"2024-03-29T11:31:40.892264+0000","last_clean":"2024-03-29T11:31:40.892264+0000","last_became_active":"2024-03-29T11:31:26.841323+0000","last_became_peered":"2024-03-29T11:31:26.841323+0000","last_unstale":"2024-03-29T11:31:40.892264+0000","last_undegraded":"2024-03-29T11:31:40.892264+0000","last_fullsized":"2024-03-29T11:31:40.892264+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:40:17.980944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.873796+0000","last_change":"2024-03-29T11:31:40.896558+0000","last_active":"2024-03-29T11:31:41.873796+0000","last_peered":"2024-03-29T11:31:41.873796+0000","last_clean":"2024-03-29T11:31:41.873796+0000","last_became_active":"2024-03-29T11:31:40.896296+0000","last_became_peered":"2024-03-29T11:31:40.896296+0000","last_unstale":"2024-03-29T11:31:41.873796+0000","last_undegraded":"2024-03-29T11:31:41.873796+0000","last_fullsized":"2024-03-29T11:31:41.873796+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:07:27.695748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.874842+0000","last_change":"2024-03-29T11:31:26.852752+0000","last_active":"2024-03-29T11:31:40.874842+0000","last_peered":"2024-03-29T11:31:40.874842+0000","last_clean":"2024-03-29T11:31:40.874842+0000","last_became_active":"2024-03-29T11:31:26.852491+0000","last_became_peered":"2024-03-29T11:31:26.852491+0000","last_unstale":"2024-03-29T11:31:40.874842+0000","last_undegraded":"2024-03-29T11:31:40.874842+0000","last_fullsized":"2024-03-29T11:31:40.874842+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:32.613551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.950067+0000","last_change":"2024-03-29T11:31:40.885411+0000","last_active":"2024-03-29T11:31:41.950067+0000","last_peered":"2024-03-29T11:31:41.950067+0000","last_clean":"2024-03-29T11:31:41.950067+0000","last_became_active":"2024-03-29T11:31:40.885167+0000","last_became_peered":"2024-03-29T11:31:40.885167+0000","last_unstale":"2024-03-29T11:31:41.950067+0000","last_undegraded":"2024-03-29T11:31:41.950067+0000","last_fullsized":"2024-03-29T11:31:41.950067+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:16:12.619950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884071+0000","last_change":"2024-03-29T11:31:26.851998+0000","last_active":"2024-03-29T11:31:40.884071+0000","last_peered":"2024-03-29T11:31:40.884071+0000","last_clean":"2024-03-29T11:31:40.884071+0000","last_became_active":"2024-03-29T11:31:26.850781+0000","last_became_peered":"2024-03-29T11:31:26.850781+0000","last_unstale":"2024-03-29T11:31:40.884071+0000","last_undegraded":"2024-03-29T11:31:40.884071+0000","last_fullsized":"2024-03-29T11:31:40.884071+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:16:57.876055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.949090+0000","last_change":"2024-03-29T11:31:40.903133+0000","last_active":"2024-03-29T11:31:41.949090+0000","last_peered":"2024-03-29T11:31:41.949090+0000","last_clean":"2024-03-29T11:31:41.949090+0000","last_became_active":"2024-03-29T11:31:40.902862+0000","last_became_peered":"2024-03-29T11:31:40.902862+0000","last_unstale":"2024-03-29T11:31:41.949090+0000","last_undegraded":"2024-03-29T11:31:41.949090+0000","last_fullsized":"2024-03-29T11:31:41.949090+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:14:43.064595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.884404+0000","last_change":"2024-03-29T11:31:26.848512+0000","last_active":"2024-03-29T11:31:40.884404+0000","last_peered":"2024-03-29T11:31:40.884404+0000","last_clean":"2024-03-29T11:31:40.884404+0000","last_became_active":"2024-03-29T11:31:26.846361+0000","last_became_peered":"2024-03-29T11:31:26.846361+0000","last_unstale":"2024-03-29T11:31:40.884404+0000","last_undegraded":"2024-03-29T11:31:40.884404+0000","last_fullsized":"2024-03-29T11:31:40.884404+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:49:25.037675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-29T11:31:41.879746+0000","last_change":"2024-03-29T11:31:40.882526+0000","last_active":"2024-03-29T11:31:41.879746+0000","last_peered":"2024-03-29T11:31:41.879746+0000","last_clean":"2024-03-29T11:31:41.879746+0000","last_became_active":"2024-03-29T11:31:40.882222+0000","last_became_peered":"2024-03-29T11:31:40.882222+0000","last_unstale":"2024-03-29T11:31:41.879746+0000","last_undegraded":"2024-03-29T11:31:41.879746+0000","last_fullsized":"2024-03-29T11:31:41.879746+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:40.335015+0000","last_clean_scrub_stamp":"2024-03-29T11:31:40.335015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:44:07.759471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-03-29T11:31:40.872754+0000","last_change":"2024-03-29T11:31:26.850587+0000","last_active":"2024-03-29T11:31:40.872754+0000","last_peered":"2024-03-29T11:31:40.872754+0000","last_clean":"2024-03-29T11:31:40.872754+0000","last_became_active":"2024-03-29T11:31:26.850150+0000","last_became_peered":"2024-03-29T11:31:26.850150+0000","last_unstale":"2024-03-29T11:31:40.872754+0000","last_undegraded":"2024-03-29T11:31:40.872754+0000","last_fullsized":"2024-03-29T11:31:40.872754+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:31:26.292064+0000","last_clean_scrub_stamp":"2024-03-29T11:31:26.292064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:49.204368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575997,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52580,"kb_used_data":3412,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3493888,"data_stored":2001990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":50330060},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.51200000000000001,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":0.88400000000000001,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.747},"min":{"1min":0.56499999999999995,"5min":0.46800000000000003,"15min":0.40300000000000002},"max":{"1min":0.98899999999999999,"5min":1.119,"15min":1.119},"last":0.66600000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.875,"5min":1.002,"15min":1.478},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74299999999999999,"15min":0.76900000000000002},"min":{"1min":0.55600000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91200000000000003,"5min":0.98699999999999999,"15min":1.393},"last":0.68300000000000005}]},{"osd":2,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72299999999999998,"15min":0.75700000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.89800000000000002,"5min":1.089,"15min":1.284},"last":0.628},{"interface":"front","average":{"1min":0.747,"5min":0.754,"15min":0.77500000000000002},"min":{"1min":0.57099999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.92700000000000005,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.73799999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.74399999999999999,"15min":0.76200000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.85499999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.77600000000000002,"15min":0.79000000000000004},"min":{"1min":0.57699999999999996,"5min":0.55900000000000005,"15min":0.55300000000000005},"max":{"1min":0.92200000000000004,"5min":1.071,"15min":1.3169999999999999},"last":0.78600000000000003}]},{"osd":4,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83299999999999996,"5min":1.024,"15min":1.3779999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91500000000000004,"5min":0.93500000000000005,"15min":1.335},"last":0.70399999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69499999999999995,"15min":0.68000000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":0.95899999999999996,"5min":0.98399999999999999,"15min":1.347},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73499999999999999,"15min":0.71699999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93600000000000005,"5min":1.0169999999999999,"15min":1.1759999999999999},"last":0.88}]},{"osd":6,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0309999999999999,"5min":1.0549999999999999,"15min":1.2070000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76200000000000001,"15min":0.73899999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.435},"max":{"1min":0.97899999999999998,"5min":1.071,"15min":1.071},"last":0.77200000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575998,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46256,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325584,"statfs":{"total":96636764160,"available":96589398016,"internally_reserved":0,"allocated":2908160,"data_stored":1416624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44431815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71699999999999997,"15min":0.73699999999999999},"min":{"1min":0.45900000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86199999999999999,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.73199999999999998,"15min":0.76100000000000001},"min":{"1min":0.52700000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.84399999999999997,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.71099999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.71099999999999997,"15min":0.73699999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":0.83399999999999996,"5min":1.98,"15min":1.98},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.745,"15min":0.77300000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.92500000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.5,"5min":0.498,"15min":0.497},"max":{"1min":0.88700000000000001,"5min":1.669,"15min":1.669},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.55700000000000005,"5min":0.49099999999999999,"15min":0.46200000000000002},"max":{"1min":0.879,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.58299999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.75600000000000001,"15min":0.77300000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.91000000000000003,"5min":2.52,"15min":2.52},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.78800000000000003,"15min":0.80300000000000005},"min":{"1min":0.57099999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.996,"5min":1.877,"15min":1.877},"last":0.64000000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.62,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":0.76500000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.70899999999999996,"15min":0.70699999999999996},"min":{"1min":0.504,"5min":0.47399999999999998,"15min":0.45900000000000002},"max":{"1min":0.90000000000000002,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":0.66600000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.47299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90200000000000002,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70999999999999996,"15min":0.71299999999999997},"min":{"1min":0.54300000000000004,"5min":0.46999999999999997,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":1.925,"15min":1.925},"last":0.752}]},{"osd":7,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77100000000000002,"15min":0.76700000000000002},"min":{"1min":0.58099999999999996,"5min":0.54800000000000004,"15min":0.52700000000000002},"max":{"1min":1.0149999999999999,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.74299999999999999,"15min":0.74099999999999999},"min":{"1min":0.54600000000000004,"5min":0.504,"15min":0.47499999999999998},"max":{"1min":0.95099999999999996,"5min":1.726,"15min":1.726},"last":0.82099999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575996,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60204,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311636,"statfs":{"total":96636764160,"available":96575115264,"internally_reserved":0,"allocated":2908160,"data_stored":1416132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.46899999999999997},"min":{"1min":0.33900000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.81100000000000005},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.505}]},{"osd":2,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.41999999999999998,"5min":0.35299999999999998,"15min":0.34799999999999998},"max":{"1min":0.75900000000000001,"5min":0.81399999999999995,"15min":0.90300000000000002},"last":0.48099999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.42999999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77000000000000002,"5min":0.86199999999999999,"15min":0.995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.49199999999999999,"5min":0.38900000000000001,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.83999999999999997,"15min":0.878},"last":0.45900000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75900000000000001,"15min":0.79200000000000004},"min":{"1min":0.55100000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.995,"5min":1.153,"15min":1.3660000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.78300000000000003,"15min":0.80500000000000005},"min":{"1min":0.60199999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.96399999999999997,"5min":1.304,"15min":1.304},"last":0.64900000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80100000000000005,"15min":0.82199999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.48199999999999998},"max":{"1min":0.97799999999999998,"5min":1.3380000000000001,"15min":1.3720000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.80900000000000005,"15min":0.83799999999999997},"min":{"1min":0.65300000000000002,"5min":0.59199999999999997,"15min":0.59099999999999997},"max":{"1min":1.0549999999999999,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.81699999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.79000000000000004,"15min":0.82299999999999995},"min":{"1min":0.67100000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.95599999999999996,"5min":1.1890000000000001,"15min":1.1919999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.84199999999999997,"15min":0.85499999999999998},"min":{"1min":0.61899999999999999,"5min":0.60999999999999999,"15min":0.53100000000000003},"max":{"1min":1.0569999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"last":0.92500000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.65900000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.034,"5min":1.327,"15min":1.3540000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.84399999999999997,"15min":0.87},"min":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.94199999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575998,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60764,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311076,"statfs":{"total":96636764160,"available":96574541824,"internally_reserved":0,"allocated":3485696,"data_stored":1998858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.45300000000000001,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65300000000000002,"5min":0.95899999999999996,"15min":1.087},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77400000000000002,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84999999999999998,"5min":1.028,"15min":1.5289999999999999},"last":0.5},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51100000000000001,"15min":0.52000000000000002},"min":{"1min":0.372,"5min":0.317,"15min":0.317},"max":{"1min":0.58299999999999996,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81000000000000005,"5min":1.222,"15min":1.222},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.42199999999999999},"max":{"1min":0.83499999999999996,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.83599999999999997,"15min":0.84999999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.091,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.028,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81000000000000005,"15min":0.82199999999999995},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1499999999999999,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.69799999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84999999999999998,"15min":0.85899999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0700000000000001,"5min":1.343,"15min":1.6240000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.873,"15min":0.88800000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.171,"5min":1.3240000000000001,"15min":1.337},"last":0.66500000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97899999999999998,"5min":1.115,"15min":1.214},"last":0.752},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.85799999999999998,"15min":0.874},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.204,"5min":1.365,"15min":1.365},"last":0.63200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575997,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47912,"kb_used_data":2836,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323928,"statfs":{"total":96636764160,"available":96587702272,"internally_reserved":0,"allocated":2904064,"data_stored":1410367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.377,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67800000000000005,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73999999999999999,"5min":0.75600000000000001,"15min":0.96799999999999997},"last":0.52900000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.79000000000000004,"15min":0.80700000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.872,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76700000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.44,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76300000000000001,"5min":1.038,"15min":1.038},"last":0.44}]},{"osd":4,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79100000000000004,"15min":0.81100000000000005},"min":{"1min":0.61599999999999999,"5min":0.57899999999999996,"15min":0.55100000000000005},"max":{"1min":0.89200000000000002,"5min":1.169,"15min":1.1970000000000001},"last":0.626},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78500000000000003,"15min":0.81599999999999995},"min":{"1min":0.64800000000000002,"5min":0.57099999999999995,"15min":0.52400000000000002},"max":{"1min":1.157,"5min":1.244,"15min":1.244},"last":0.61199999999999999}]},{"osd":5,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.61799999999999999,"5min":0.59799999999999998,"15min":0.53400000000000003},"max":{"1min":1.1200000000000001,"5min":1.256,"15min":1.256},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81299999999999994,"15min":0.82999999999999996},"min":{"1min":0.72799999999999998,"5min":0.57799999999999996,"15min":0.56000000000000005},"max":{"1min":0.97499999999999998,"5min":1.24,"15min":1.3140000000000001},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82799999999999996,"15min":0.85199999999999998},"min":{"1min":0.64800000000000002,"5min":0.58999999999999997,"15min":0.57799999999999996},"max":{"1min":1.105,"5min":1.1459999999999999,"15min":1.24},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.623,"5min":0.58599999999999997,"15min":0.58199999999999996},"max":{"1min":1.0880000000000001,"5min":1.1879999999999999,"15min":1.329},"last":0.64800000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:31:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.877,"15min":0.90000000000000002},"min":{"1min":0.67400000000000004,"5min":0.66800000000000004,"15min":0.64600000000000002},"max":{"1min":1.0209999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88900000000000001,"15min":0.91200000000000003},"min":{"1min":0.68300000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.071,"5min":1.304,"15min":1.304},"last":0.59699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575997,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47916,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323924,"statfs":{"total":96636764160,"available":96587698176,"internally_reserved":0,"allocated":2912256,"data_stored":1410563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.47499999999999998,"15min":0.47099999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.36899999999999999,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70099999999999996,"5min":0.72099999999999997,"15min":0.748},"last":0.373},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54400000000000004,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":1.046},"last":0.56499999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.96699999999999997},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.66200000000000003},"min":{"1min":0.53200000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86399999999999999,"5min":1.0049999999999999,"15min":1.01},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.80200000000000005},"min":{"1min":0.56299999999999994,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.2549999999999999,"5min":1.2609999999999999,"15min":1.2889999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81799999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1599999999999999,"5min":1.306,"15min":1.306},"last":0.68999999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80400000000000005,"15min":0.81599999999999995},"min":{"1min":0.56999999999999995,"5min":0.53800000000000003,"15min":0.502},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.62},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.56000000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.214,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.79600000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64900000000000002,"5min":0.64400000000000002,"15min":0.57799999999999996},"max":{"1min":1.196,"5min":1.282,"15min":1.288},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.84099999999999997,"15min":0.84699999999999998},"min":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.86399999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.85799999999999998,"15min":0.86699999999999999},"min":{"1min":0.68999999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3100000000000001,"5min":1.331,"15min":1.331},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.89500000000000002,"15min":0.89600000000000002},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.57899999999999996},"max":{"1min":1.246,"5min":1.286,"15min":1.286},"last":0.71799999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575998,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60204,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311636,"statfs":{"total":96636764160,"available":96575115264,"internally_reserved":0,"allocated":2908160,"data_stored":1412238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.68999999999999995,"15min":0.72199999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.39900000000000002},"max":{"1min":0.875,"5min":0.95499999999999996,"15min":1.0509999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69199999999999995,"15min":0.72499999999999998},"min":{"1min":0.45000000000000001,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.89500000000000002,"5min":0.99299999999999999,"15min":1.1459999999999999},"last":0.59399999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.70699999999999996,"15min":0.748},"min":{"1min":0.497,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":0.89100000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"last":0.436},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.70899999999999996,"15min":0.745},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.877,"5min":0.94899999999999995,"15min":1.002},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.79400000000000004,"15min":0.82799999999999996},"min":{"1min":0.52100000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.89500000000000002,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.79100000000000004,"15min":0.81399999999999995},"min":{"1min":0.54100000000000004,"5min":0.496,"15min":0.496},"max":{"1min":0.93799999999999994,"5min":1.224,"15min":1.224},"last":0.626}]},{"osd":3,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.72599999999999998,"15min":0.76400000000000001},"min":{"1min":0.52200000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.96699999999999997,"5min":1.0580000000000001,"15min":1.181},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76500000000000001,"15min":0.77100000000000002},"min":{"1min":0.55600000000000005,"5min":0.55300000000000005,"15min":0.48699999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.2490000000000001},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.63500000000000001,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.47199999999999998,"5min":0.41599999999999998,"15min":0.39700000000000002},"max":{"1min":0.86299999999999999,"5min":1.0920000000000001,"15min":1.169},"last":0.52200000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":1.056,"5min":1.056,"15min":1.0800000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.53200000000000003,"5min":0.48199999999999998,"15min":0.47699999999999998},"max":{"1min":0.93500000000000005,"5min":1.0089999999999999,"15min":1.028},"last":0.66800000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.56200000000000006,"5min":0.501,"15min":0.436},"max":{"1min":0.92100000000000004,"5min":1.0389999999999999,"15min":1.518},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.81999999999999995,"15min":0.82599999999999996},"min":{"1min":0.60299999999999998,"5min":0.58499999999999996,"15min":0.54400000000000004},"max":{"1min":1.0900000000000001,"5min":1.1619999999999999,"15min":1.3440000000000001},"last":0.65900000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575997,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42140,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329700,"statfs":{"total":96636764160,"available":96593612800,"internally_reserved":0,"allocated":2899968,"data_stored":1416568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70699999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83899999999999997,"5min":0.94199999999999995,"15min":0.95999999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.73199999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90000000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.80000000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.75900000000000001,"15min":0.76500000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.73699999999999999,"15min":0.74199999999999999},"min":{"1min":0.52200000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.66600000000000004}]},{"osd":2,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.76800000000000002,"15min":0.75900000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.72899999999999998,"15min":0.748},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.88400000000000001,"5min":0.98499999999999999,"15min":1.0129999999999999},"last":0.52500000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.77900000000000003,"15min":0.77400000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.444,"5min":3.444,"15min":3.444},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.046},"last":0.71299999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.60599999999999998},"min":{"1min":0.44400000000000001,"5min":0.41799999999999998,"15min":0.38700000000000001},"max":{"1min":0.88,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.40600000000000003},"max":{"1min":0.84799999999999998,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.64600000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68799999999999994,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90300000000000002,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.748},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72099999999999997,"15min":0.69799999999999995},"min":{"1min":0.55900000000000005,"5min":0.51000000000000001,"15min":0.42499999999999999},"max":{"1min":0.83499999999999996,"5min":1.107,"15min":1.107},"last":0.68500000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.74199999999999999,"15min":0.72199999999999998},"min":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.42299999999999999},"max":{"1min":0.92200000000000004,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.79300000000000004,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.98199999999999998,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.81100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:31:45.844 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T11:31:45.844 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:31:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.081+0000 7fd767400700 1 Processor -- start 2024-03-29T11:31:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.081+0000 7fd767400700 1 -- start start 2024-03-29T11:31:46.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.082+0000 7fd767400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd76015d920 0x7fd76006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.082+0000 7fd767400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd76009a910 0x7fd7600cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.082+0000 7fd767400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd760099a60 0x7fd760099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.082+0000 7fd767400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd76009a3f0 con 0x7fd76015d920 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.082+0000 7fd767400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7600d08d0 con 0x7fd760099a60 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.082+0000 7fd767400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76006c300 con 0x7fd76009a910 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.082+0000 7fd764a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd76015d920 0x7fd76006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.082+0000 7fd764a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd76015d920 0x7fd76006c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39598/0 (socket says 172.21.15.12:39598) 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.082+0000 7fd764a00700 1 -- 172.21.15.12:0/1279152209 learned_addr learned my addr 172.21.15.12:0/1279152209 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.083+0000 7fd764a00700 1 -- 172.21.15.12:0/1279152209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd76009a910 msgr2=0x7fd7600cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.083+0000 7fd764a00700 1 --2- 172.21.15.12:0/1279152209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd76009a910 0x7fd7600cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.083+0000 7fd764a00700 1 -- 172.21.15.12:0/1279152209 >> v2:172.21.15.159:3300/0 conn(0x7fd760099a60 msgr2=0x7fd760099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.083+0000 7fd764a00700 1 --2- 172.21.15.12:0/1279152209 >> v2:172.21.15.159:3300/0 conn(0x7fd760099a60 0x7fd760099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.083+0000 7fd764a00700 1 -- 172.21.15.12:0/1279152209 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76006b9d0 con 0x7fd76015d920 2024-03-29T11:31:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.083+0000 7fd764a00700 1 --2- 172.21.15.12:0/1279152209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76015d920 0x7fd76006c790 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fd760074590 tx=0x7fd754009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b7470e38585851 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.083+0000 7fd75f400700 1 -- 172.21.15.12:0/1279152209 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75400b870 con 0x7fd76015d920 2024-03-29T11:31:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.083+0000 7fd75f400700 1 -- 172.21.15.12:0/1279152209 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75400b9d0 con 0x7fd76015d920 2024-03-29T11:31:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.083+0000 7fd75f400700 1 -- 172.21.15.12:0/1279152209 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75400bd60 con 0x7fd76015d920 2024-03-29T11:31:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.084+0000 7fd767400700 1 -- 172.21.15.12:0/1279152209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76015d920 msgr2=0x7fd76006c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.084+0000 7fd767400700 1 --2- 172.21.15.12:0/1279152209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76015d920 0x7fd76006c790 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fd760074590 tx=0x7fd754009bd0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.084+0000 7fd767400700 1 -- 172.21.15.12:0/1279152209 shutdown_connections 2024-03-29T11:31:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.084+0000 7fd767400700 1 --2- 172.21.15.12:0/1279152209 >> v2:172.21.15.159:3300/0 conn(0x7fd760099a60 0x7fd760099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.084+0000 7fd767400700 1 --2- 172.21.15.12:0/1279152209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76015d920 0x7fd76006c790 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.084+0000 7fd767400700 1 --2- 172.21.15.12:0/1279152209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd76009a910 0x7fd7600cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.084+0000 7fd767400700 1 -- 172.21.15.12:0/1279152209 >> 172.21.15.12:0/1279152209 conn(0x7fd7600d1350 msgr2=0x7fd7600d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.084+0000 7fd767400700 1 -- 172.21.15.12:0/1279152209 shutdown_connections 2024-03-29T11:31:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.084+0000 7fd767400700 1 -- 172.21.15.12:0/1279152209 wait complete. 2024-03-29T11:31:46.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.086+0000 7fd767400700 1 Processor -- start 2024-03-29T11:31:46.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.087+0000 7fd767400700 1 -- start start 2024-03-29T11:31:46.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd767400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd760099a60 0x7fd760151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd767400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd76009a910 0x7fd76015ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd767400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd76015d920 0x7fd76015f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd767400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76015ec60 con 0x7fd760099a60 2024-03-29T11:31:46.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd767400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd760152760 con 0x7fd76009a910 2024-03-29T11:31:46.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd767400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76015dea0 con 0x7fd76015d920 2024-03-29T11:31:46.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd764a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd760099a60 0x7fd760151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd764a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd760099a60 0x7fd760151cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39610/0 (socket says 172.21.15.12:39610) 2024-03-29T11:31:46.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd764a00700 1 -- 172.21.15.12:0/3616048189 learned_addr learned my addr 172.21.15.12:0/3616048189 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:46.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd75fe00700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd76009a910 0x7fd76015ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd765400700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd76015d920 0x7fd76015f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd764a00700 1 -- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd76015d920 msgr2=0x7fd76015f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd764a00700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd76015d920 0x7fd76015f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd764a00700 1 -- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd76009a910 msgr2=0x7fd76015ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd764a00700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd76009a910 0x7fd76015ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd764a00700 1 -- 172.21.15.12:0/3616048189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd754009550 con 0x7fd760099a60 2024-03-29T11:31:46.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd75fe00700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd76009a910 0x7fd76015ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:46.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.088+0000 7fd765400700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd76015d920 0x7fd76015f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:46.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.089+0000 7fd764a00700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd760099a60 0x7fd760151cb0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fd754014040 tx=0x7fd754004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:46.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.089+0000 7fd75cc00700 1 -- 172.21.15.12:0/3616048189 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754004520 con 0x7fd760099a60 2024-03-29T11:31:46.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.089+0000 7fd75cc00700 1 -- 172.21.15.12:0/3616048189 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75401e070 con 0x7fd760099a60 2024-03-29T11:31:46.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.089+0000 7fd75cc00700 1 -- 172.21.15.12:0/3616048189 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7540038c0 con 0x7fd760099a60 2024-03-29T11:31:46.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.089+0000 7fd767400700 1 -- 172.21.15.12:0/3616048189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76015e180 con 0x7fd760099a60 2024-03-29T11:31:46.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.089+0000 7fd767400700 1 -- 172.21.15.12:0/3616048189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76015e740 con 0x7fd760099a60 2024-03-29T11:31:46.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.090+0000 7fd767400700 1 -- 172.21.15.12:0/3616048189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd760067720 con 0x7fd760099a60 2024-03-29T11:31:46.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.091+0000 7fd75cc00700 1 -- 172.21.15.12:0/3616048189 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd754003a20 con 0x7fd760099a60 2024-03-29T11:31:46.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.092+0000 7fd75cc00700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd74006adb0 0x7fd74006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.092+0000 7fd75cc00700 1 -- 172.21.15.12:0/3616048189 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7fd754020070 con 0x7fd760099a60 2024-03-29T11:31:46.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.092+0000 7fd75fe00700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd74006adb0 0x7fd74006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.093+0000 7fd75fe00700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd74006adb0 0x7fd74006d270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd74c006fd0 tx=0x7fd74c008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:46.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.096+0000 7fd75cc00700 1 -- 172.21.15.12:0/3616048189 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd754055c40 con 0x7fd760099a60 2024-03-29T11:31:46.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.353+0000 7fd767400700 1 -- 172.21.15.12:0/3616048189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7600d0de0 con 0x7fd760099a60 2024-03-29T11:31:46.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.354+0000 7fd75cc00700 1 -- 172.21.15.12:0/3616048189 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5971 (secure 0 0 0) 0x7fd75402a020 con 0x7fd760099a60 2024-03-29T11:31:46.348 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:46.348 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":112,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:26.816217+0000","modified":"2024-03-29T11:31:30.290952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/2722218561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2722218561},{"type":"v1","addr":"172.21.15.12:6833","nonce":2722218561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:40.863222+0000","modified":"2024-03-29T11:31:43.883199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-29T11:31:46.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 -- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd74006adb0 msgr2=0x7fd74006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd74006adb0 0x7fd74006d270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd74c006fd0 tx=0x7fd74c008040 comp rx=0 tx=0).stop 2024-03-29T11:31:46.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 -- 172.21.15.12:0/3616048189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd760099a60 msgr2=0x7fd760151cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd760099a60 0x7fd760151cb0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fd754014040 tx=0x7fd754004750 comp rx=0 tx=0).stop 2024-03-29T11:31:46.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 -- 172.21.15.12:0/3616048189 shutdown_connections 2024-03-29T11:31:46.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd74006adb0 0x7fd74006d270 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd76015d920 0x7fd76015f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd760099a60 0x7fd760151cb0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 --2- 172.21.15.12:0/3616048189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd76009a910 0x7fd76015ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 -- 172.21.15.12:0/3616048189 >> 172.21.15.12:0/3616048189 conn(0x7fd7600d1350 msgr2=0x7fd7600b0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:46.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 -- 172.21.15.12:0/3616048189 shutdown_connections 2024-03-29T11:31:46.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.356+0000 7fd767400700 1 -- 172.21.15.12:0/3616048189 wait complete. 2024-03-29T11:31:46.352 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 112 2024-03-29T11:31:46.460 INFO:tasks.ceph.mds.b.smithi159.stderr:2024-03-29T11:31:46.465+0000 7fd80e400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:31:46.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 112, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T11:31:40.863222+0000', 'modified': '2024-03-29T11:31:43.883199+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11025}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11025': {'gid': 11025, 'name': 'b', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.159:6837/2157276692', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6835', 'nonce': 2157276692}, {'type': 'v1', 'addr': '172.21.15.159:6837', 'nonce': 2157276692}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T11:31:46.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T11:31:46.772 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-29T11:31:46.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.983+0000 7f8b82200700 1 Processor -- start 2024-03-29T11:31:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.983+0000 7f8b82200700 1 -- start start 2024-03-29T11:31:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.984+0000 7f8b82200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8b7c15d9d0 0x7f8b7c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.984+0000 7f8b82200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8b7c0d2540 0x7f8b7c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.984+0000 7f8b82200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b7c161420 con 0x7f8b7c0d16a0 2024-03-29T11:31:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.984+0000 7f8b82200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b7c15edc0 con 0x7f8b7c0d2540 2024-03-29T11:31:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.984+0000 7f8b82200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b7c161700 con 0x7f8b7c15d9d0 2024-03-29T11:31:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.984+0000 7f8b7be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8b7c0d2540 0x7f8b7c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.984+0000 7f8b7be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8b7c0d2540 0x7f8b7c0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39616/0 (socket says 172.21.15.12:39616) 2024-03-29T11:31:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.984+0000 7f8b7be00700 1 -- 172.21.15.12:0/2066715658 learned_addr learned my addr 172.21.15.12:0/2066715658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.985+0000 7f8b7b400700 1 --2- 172.21.15.12:0/2066715658 >> v2:172.21.15.159:3300/0 conn(0x7f8b7c15d9d0 0x7f8b7c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.985+0000 7f8b7be00700 1 -- 172.21.15.12:0/2066715658 >> v2:172.21.15.159:3300/0 conn(0x7f8b7c15d9d0 msgr2=0x7f8b7c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.985+0000 7f8b7be00700 1 --2- 172.21.15.12:0/2066715658 >> v2:172.21.15.159:3300/0 conn(0x7f8b7c15d9d0 0x7f8b7c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.985+0000 7f8b7be00700 1 -- 172.21.15.12:0/2066715658 >> v1:172.21.15.12:6789/0 conn(0x7f8b7c0d16a0 legacy=0x7f8b7c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.985+0000 7f8b7be00700 1 -- 172.21.15.12:0/2066715658 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c1619e0 con 0x7f8b7c0d2540 2024-03-29T11:31:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.985+0000 7f8b7a000700 1 -- 172.21.15.12:0/2066715658 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975599754 0 0) 0x7f8b7c161420 con 0x7f8b7c0d16a0 2024-03-29T11:31:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.985+0000 7f8b7be00700 1 --2- 172.21.15.12:0/2066715658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c0d2540 0x7f8b7c0ce360 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c0d0ff0 tx=0x7f8b7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1e39ea2c63319d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.985+0000 7f8b7a000700 1 -- 172.21.15.12:0/2066715658 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7400e070 con 0x7f8b7c0d2540 2024-03-29T11:31:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.985+0000 7f8b7a000700 1 -- 172.21.15.12:0/2066715658 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7400b9c0 con 0x7f8b7c0d2540 2024-03-29T11:31:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.985+0000 7f8b7a000700 1 -- 172.21.15.12:0/2066715658 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7400bd70 con 0x7f8b7c0d2540 2024-03-29T11:31:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.986+0000 7f8b82200700 1 -- 172.21.15.12:0/2066715658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c0d2540 msgr2=0x7f8b7c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.986+0000 7f8b82200700 1 --2- 172.21.15.12:0/2066715658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c0d2540 0x7f8b7c0ce360 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c0d0ff0 tx=0x7f8b7400b330 comp rx=0 tx=0).stop 2024-03-29T11:31:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.986+0000 7f8b82200700 1 -- 172.21.15.12:0/2066715658 shutdown_connections 2024-03-29T11:31:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.986+0000 7f8b82200700 1 --2- 172.21.15.12:0/2066715658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c0d2540 0x7f8b7c0ce360 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.986+0000 7f8b82200700 1 --2- 172.21.15.12:0/2066715658 >> v2:172.21.15.159:3300/0 conn(0x7f8b7c15d9d0 0x7f8b7c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.986+0000 7f8b82200700 1 -- 172.21.15.12:0/2066715658 >> 172.21.15.12:0/2066715658 conn(0x7f8b7c08aff0 msgr2=0x7f8b7c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.986+0000 7f8b82200700 1 -- 172.21.15.12:0/2066715658 shutdown_connections 2024-03-29T11:31:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.986+0000 7f8b82200700 1 -- 172.21.15.12:0/2066715658 wait complete. 2024-03-29T11:31:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.988+0000 7f8b82200700 1 Processor -- start 2024-03-29T11:31:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.988+0000 7f8b82200700 1 -- start start 2024-03-29T11:31:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b82200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b7c0d16a0 0x7f8b7c0cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b82200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b7c0d2540 0x7f8b7c15fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b82200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c15d9d0 0x7f8b7c14e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b82200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c1605c0 con 0x7f8b7c15d9d0 2024-03-29T11:31:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b82200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c1608a0 con 0x7f8b7c0d16a0 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b82200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c160b80 con 0x7f8b7c0d2540 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b7be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c15d9d0 0x7f8b7c14e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b7b400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b7c0d16a0 0x7f8b7c0cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b7be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c15d9d0 0x7f8b7c14e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39626/0 (socket says 172.21.15.12:39626) 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b7be00700 1 -- 172.21.15.12:0/771747323 learned_addr learned my addr 172.21.15.12:0/771747323 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.989+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b7c0d2540 0x7f8b7c15fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b7be00700 1 -- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b7c0d2540 msgr2=0x7f8b7c15fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b7be00700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b7c0d2540 0x7f8b7c15fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b7be00700 1 -- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b7c0d16a0 msgr2=0x7f8b7c0cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b7be00700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b7c0d16a0 0x7f8b7c0cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b7be00700 1 -- 172.21.15.12:0/771747323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b74009b30 con 0x7f8b7c15d9d0 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b7be00700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c15d9d0 0x7f8b7c14e8e0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f8b74012000 tx=0x7f8b74004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b73400700 1 -- 172.21.15.12:0/771747323 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74004880 con 0x7f8b7c15d9d0 2024-03-29T11:31:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b73400700 1 -- 172.21.15.12:0/771747323 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b740092e0 con 0x7f8b7c15d9d0 2024-03-29T11:31:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b73400700 1 -- 172.21.15.12:0/771747323 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7401c660 con 0x7f8b7c15d9d0 2024-03-29T11:31:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b82200700 1 -- 172.21.15.12:0/771747323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7c160e60 con 0x7f8b7c15d9d0 2024-03-29T11:31:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.990+0000 7f8b82200700 1 -- 172.21.15.12:0/771747323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7c14ee20 con 0x7f8b7c15d9d0 2024-03-29T11:31:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.992+0000 7f8b73400700 1 -- 172.21.15.12:0/771747323 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8b7401c7c0 con 0x7f8b7c15d9d0 2024-03-29T11:31:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.993+0000 7f8b73400700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8b5c06ade0 0x7f8b5c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.993+0000 7f8b73400700 1 -- 172.21.15.12:0/771747323 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f8b7408b1c0 con 0x7f8b7c15d9d0 2024-03-29T11:31:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.994+0000 7f8b7b400700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8b5c06ade0 0x7f8b5c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.995+0000 7f8b7b400700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8b5c06ade0 0x7f8b5c06d2a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c006b20 tx=0x7f8b6c008040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.995+0000 7f8b95d55b80 1 Processor -- start 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.996+0000 7f8b95d55b80 1 -- start start 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.996+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad415830 0x55e3ad415c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.996+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad3eba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.996+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad412a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.996+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad360650 con 0x55e3ad3e6b60 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.996+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad341c30 con 0x55e3ad415830 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.996+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad33f430 con 0x55e3ad3e8be0 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad412a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad412a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39628/0 (socket says 172.21.15.12:39628) 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7be00700 1 -- 172.21.15.12:0/686407058 learned_addr learned my addr 172.21.15.12:0/686407058 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7b400700 1 --2- 172.21.15.12:0/686407058 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad415830 0x55e3ad415c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/686407058 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad3eba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7be00700 1 -- 172.21.15.12:0/686407058 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 msgr2=0x55e3ad3eba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7be00700 1 --2- 172.21.15.12:0/686407058 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad3eba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7be00700 1 -- 172.21.15.12:0/686407058 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad415830 msgr2=0x55e3ad415c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7be00700 1 --2- 172.21.15.12:0/686407058 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad415830 0x55e3ad415c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7be00700 1 -- 172.21.15.12:0/686407058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3ad12e790 con 0x55e3ad3e6b60 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/686407058 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad3eba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b7be00700 1 --2- 172.21.15.12:0/686407058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad412a20 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009fd0 tx=0x7f8b740598d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3c9454bf6c784a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b67400700 1 -- 172.21.15.12:0/686407058 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74059a80 con 0x55e3ad3e6b60 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b67400700 1 -- 172.21.15.12:0/686407058 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7405d8b0 con 0x55e3ad3e6b60 2024-03-29T11:31:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.997+0000 7f8b67400700 1 -- 172.21.15.12:0/686407058 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74061e70 con 0x55e3ad3e6b60 2024-03-29T11:31:46.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.998+0000 7f8b95d55b80 1 -- 172.21.15.12:0/686407058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 msgr2=0x55e3ad412a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.998+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/686407058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad412a20 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009fd0 tx=0x7f8b740598d0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.998+0000 7f8b95d55b80 1 -- 172.21.15.12:0/686407058 shutdown_connections 2024-03-29T11:31:46.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.998+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/686407058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad412a20 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.998+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/686407058 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad415830 0x55e3ad415c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.998+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/686407058 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad3eba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:46.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.998+0000 7f8b95d55b80 1 -- 172.21.15.12:0/686407058 >> 172.21.15.12:0/686407058 conn(0x55e3ad3496a0 msgr2=0x55e3ad430640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:46.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.998+0000 7f8b95d55b80 1 -- 172.21.15.12:0/686407058 shutdown_connections 2024-03-29T11:31:46.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.998+0000 7f8b95d55b80 1 -- 172.21.15.12:0/686407058 wait complete. 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.999+0000 7f8b95d55b80 1 -- start start 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:46.999+0000 7f8b95d55b80 1 Processor -- start 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.000+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad413200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.000+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad40b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.000+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad415830 0x55e3ad433100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.000+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad33ad20 con 0x55e3ad3e6b60 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.000+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad2d6770 con 0x55e3ad415830 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.000+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad2f96f0 con 0x55e3ad3e8be0 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.000+0000 7f8b95d55b80 10 client.0 _ll_register_callbacks cb 0x55e3ad2e05e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.000+0000 7f8b7b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad413200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.000+0000 7f8b7b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad413200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39630/0 (socket says 172.21.15.12:39630) 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.000+0000 7f8b7b400700 1 -- 172.21.15.12:0/1419447269 learned_addr learned my addr 172.21.15.12:0/1419447269 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b7be00700 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad415830 0x55e3ad433100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad40b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b7b400700 1 -- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 msgr2=0x55e3ad40b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b7b400700 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad40b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b7b400700 1 -- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad415830 msgr2=0x55e3ad433100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b7b400700 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad415830 0x55e3ad433100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b7b400700 1 -- 172.21.15.12:0/1419447269 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7c14ee20 con 0x55e3ad3e6b60 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b7b400700 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad413200 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c016760 tx=0x7f8b6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b8c7bce9a1579 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b5be00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b5be00700 1 -- 172.21.15.12:0/1419447269 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c014640 con 0x55e3ad3e6b60 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.001+0000 7f8b95d55b80 1 -- 172.21.15.12:0/1419447269 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e3ad3dd920 con 0x55e3ad3e6b60 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b5be00700 1 -- 172.21.15.12:0/1419447269 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f8b6c0147a0 con 0x55e3ad3e6b60 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b5be00700 1 -- 172.21.15.12:0/1419447269 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b6c02cea0 con 0x55e3ad3e6b60 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b95d55b80 10 client.11322 fetch_fsmap learned FSMap version 112 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b95d55b80 1 -- 172.21.15.12:0/1419447269 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) v3 -- 0x55e3ad21b0a0 con 0x55e3ad3e6b60 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b5be00700 1 -- 172.21.15.12:0/1419447269 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e3ad3dd920 con 0x55e3ad3e6b60 2024-03-29T11:31:47.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b5be00700 1 -- 172.21.15.12:0/1419447269 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 112) v1 ==== 3681+0+0 (secure 0 0 0) 0x7f8b6c014960 con 0x55e3ad3e6b60 2024-03-29T11:31:47.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b95d55b80 10 client.11322 fetch_fsmap finished waiting for FSMap version 112 2024-03-29T11:31:47.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b95d55b80 10 client.11322 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-29T11:31:47.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b95d55b80 20 client.11322 populate_metadata read hostname 'smithi012' 2024-03-29T11:31:47.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:6835/2157276692,v1:172.21.15.159:6837/2157276692] conn(0x55e3ad430530 0x55e3ad430910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b95d55b80 4 client.11322 mds_command: new command op to 11025 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:47.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.002+0000 7f8b95d55b80 1 -- 172.21.15.12:0/1419447269 --> [v2:172.21.15.159:6835/2157276692,v1:172.21.15.159:6837/2157276692] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3ad3ba6f0 con 0x55e3ad430530 2024-03-29T11:31:47.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.003+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:6835/2157276692,v1:172.21.15.159:6837/2157276692] conn(0x55e3ad430530 0x55e3ad430910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.003+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:6835/2157276692,v1:172.21.15.159:6837/2157276692] conn(0x55e3ad430530 0x55e3ad430910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4b699fa8d976ea6 server_cookie=11a3d1b4edaa9111 in_seq=0 out_seq=0 2024-03-29T11:31:47.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.003+0000 7f8b5be00700 10 client.11322 ms_handle_connect on v2:172.21.15.159:6835/2157276692 2024-03-29T11:31:47.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.004+0000 7f8b5be00700 1 -- 172.21.15.12:0/1419447269 <== mds.0 v2:172.21.15.159:6835/2157276692 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e3ad3ba6f0 con 0x55e3ad430530 2024-03-29T11:31:47.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.004+0000 7f8b5be00700 10 client.11322 handle_command_reply: tid=0 2024-03-29T11:31:47.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.004+0000 7f8b95d55b80 1 client.11322 shutdown 2024-03-29T11:31:47.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 1 -- 172.21.15.12:0/1419447269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 msgr2=0x55e3ad413200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad413200 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c016760 tx=0x7f8b6c004750 comp rx=0 tx=0).stop 2024-03-29T11:31:47.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 1 -- 172.21.15.12:0/1419447269 shutdown_connections 2024-03-29T11:31:47.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:6835/2157276692,v1:172.21.15.159:6837/2157276692] conn(0x55e3ad430530 0x55e3ad430910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad415830 0x55e3ad433100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad413200 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/1419447269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad40b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b5be00700 0 client.11322 ms_handle_reset on v2:172.21.15.159:6835/2157276692 2024-03-29T11:31:47.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 1 -- 172.21.15.12:0/1419447269 >> 172.21.15.12:0/1419447269 conn(0x55e3ad3496a0 msgr2=0x55e3ad3eac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:47.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 1 -- 172.21.15.12:0/1419447269 shutdown_connections 2024-03-29T11:31:47.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 1 -- 172.21.15.12:0/1419447269 wait complete. 2024-03-29T11:31:47.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.005+0000 7f8b95d55b80 20 client.11322 trim_cache size 0 max 16384 2024-03-29T11:31:47.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.023+0000 7f8b95d55b80 1 Processor -- start 2024-03-29T11:31:47.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.023+0000 7f8b95d55b80 1 -- start start 2024-03-29T11:31:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.024+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3e58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.024+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad3eab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.024+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad415830 0x55e3ad3eb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.024+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad40ea20 con 0x55e3ad415830 2024-03-29T11:31:47.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.024+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad12f390 con 0x55e3ad3e6b60 2024-03-29T11:31:47.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.024+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad40b820 con 0x55e3ad3e8be0 2024-03-29T11:31:47.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.024+0000 7f8b7be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad415830 0x55e3ad3eb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b7be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad415830 0x55e3ad3eb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39644/0 (socket says 172.21.15.12:39644) 2024-03-29T11:31:47.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b7be00700 1 -- 172.21.15.12:0/3693556894 learned_addr learned my addr 172.21.15.12:0/3693556894 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:47.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b7be00700 1 -- 172.21.15.12:0/3693556894 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 msgr2=0x55e3ad3eab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:47.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b7be00700 1 --2- 172.21.15.12:0/3693556894 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad3eab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b7be00700 1 -- 172.21.15.12:0/3693556894 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e6b60 msgr2=0x55e3ad3e58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:47.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b7b400700 1 --2- 172.21.15.12:0/3693556894 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3e58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b7be00700 1 --2- 172.21.15.12:0/3693556894 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3e58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b7be00700 1 -- 172.21.15.12:0/3693556894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3ad3ebaa0 con 0x55e3ad415830 2024-03-29T11:31:47.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b7b400700 1 --2- 172.21.15.12:0/3693556894 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3e58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:47.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b7be00700 1 --2- 172.21.15.12:0/3693556894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad415830 0x55e3ad3eb0c0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f8b74061940 tx=0x7f8b74027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce343717d34c98f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b67e00700 1 -- 172.21.15.12:0/3693556894 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74059a80 con 0x55e3ad415830 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b67e00700 1 -- 172.21.15.12:0/3693556894 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b74061e70 con 0x55e3ad415830 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.025+0000 7f8b67e00700 1 -- 172.21.15.12:0/3693556894 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74089290 con 0x55e3ad415830 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.026+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3693556894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad415830 msgr2=0x55e3ad3eb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.026+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/3693556894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad415830 0x55e3ad3eb0c0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f8b74061940 tx=0x7f8b74027000 comp rx=0 tx=0).stop 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.026+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3693556894 shutdown_connections 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.026+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/3693556894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad415830 0x55e3ad3eb0c0 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.026+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/3693556894 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3e58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.026+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/3693556894 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad3e8be0 0x55e3ad3eab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.026+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3693556894 >> 172.21.15.12:0/3693556894 conn(0x55e3ad3496a0 msgr2=0x55e3ad4523d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:47.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.026+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3693556894 shutdown_connections 2024-03-29T11:31:47.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.026+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3693556894 wait complete. 2024-03-29T11:31:47.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.027+0000 7f8b95d55b80 1 -- start start 2024-03-29T11:31:47.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.027+0000 7f8b95d55b80 1 Processor -- start 2024-03-29T11:31:47.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3eb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e8be0 0x55e3ad409760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b95d55b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad415830 0x55e3ad409ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad433310 con 0x55e3ad3e6b60 2024-03-29T11:31:47.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad497bd0 con 0x55e3ad3e8be0 2024-03-29T11:31:47.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b95d55b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3ad4330c0 con 0x55e3ad415830 2024-03-29T11:31:47.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b95d55b80 10 client.0 _ll_register_callbacks cb 0x7f8b6c010ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:31:47.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b7b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3eb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b7b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3eb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39660/0 (socket says 172.21.15.12:39660) 2024-03-29T11:31:47.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b7b400700 1 -- 172.21.15.12:0/3537590479 learned_addr learned my addr 172.21.15.12:0/3537590479 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:47.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b7be00700 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad415830 0x55e3ad409ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.028+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e8be0 0x55e3ad409760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b7b400700 1 -- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad415830 msgr2=0x55e3ad409ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:47.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b7b400700 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad415830 0x55e3ad409ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b7b400700 1 -- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e8be0 msgr2=0x55e3ad409760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:47.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b7b400700 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e8be0 0x55e3ad409760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b7b400700 1 -- 172.21.15.12:0/3537590479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3ad12e790 con 0x55e3ad3e6b60 2024-03-29T11:31:47.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b7be00700 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad415830 0x55e3ad409ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:47.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e8be0 0x55e3ad409760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:47.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b7b400700 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3eb0c0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c010c80 tx=0x7f8b6c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da42aabdb4a8054b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:47.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b5be00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:31:47.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b5be00700 1 -- 172.21.15.12:0/3537590479 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c016d50 con 0x55e3ad3e6b60 2024-03-29T11:31:47.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b5be00700 1 -- 172.21.15.12:0/3537590479 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f8b6c014e20 con 0x55e3ad3e6b60 2024-03-29T11:31:47.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3537590479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e3ad3e84c0 con 0x55e3ad3e6b60 2024-03-29T11:31:47.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b5be00700 1 -- 172.21.15.12:0/3537590479 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b6c014850 con 0x55e3ad3e6b60 2024-03-29T11:31:47.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b95d55b80 10 client.11328 fetch_fsmap learned FSMap version 112 2024-03-29T11:31:47.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.029+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3537590479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) v3 -- 0x55e3ad301010 con 0x55e3ad3e6b60 2024-03-29T11:31:47.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.030+0000 7f8b5be00700 1 -- 172.21.15.12:0/3537590479 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e3ad3e84c0 con 0x55e3ad3e6b60 2024-03-29T11:31:47.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.030+0000 7f8b5be00700 1 -- 172.21.15.12:0/3537590479 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 112) v1 ==== 3681+0+0 (secure 0 0 0) 0x7f8b6c03e450 con 0x55e3ad3e6b60 2024-03-29T11:31:47.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.030+0000 7f8b95d55b80 10 client.11328 fetch_fsmap finished waiting for FSMap version 112 2024-03-29T11:31:47.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.030+0000 7f8b95d55b80 10 client.11328 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-29T11:31:47.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.030+0000 7f8b95d55b80 20 client.11328 populate_metadata read hostname 'smithi012' 2024-03-29T11:31:47.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.030+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:6835/2157276692,v1:172.21.15.159:6837/2157276692] conn(0x55e3ad3eb600 0x55e3ad40fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.030+0000 7f8b95d55b80 4 client.11328 mds_command: new command op to 11025 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T11:31:47.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.030+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3537590479 --> [v2:172.21.15.159:6835/2157276692,v1:172.21.15.159:6837/2157276692] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f8b7405d8b0 con 0x55e3ad3eb600 2024-03-29T11:31:47.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.030+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:6835/2157276692,v1:172.21.15.159:6837/2157276692] conn(0x55e3ad3eb600 0x55e3ad40fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:6835/2157276692,v1:172.21.15.159:6837/2157276692] conn(0x55e3ad3eb600 0x55e3ad40fb80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1111af69fbe045cd server_cookie=8ad5622552e9a7a5 in_seq=0 out_seq=0 2024-03-29T11:31:47.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b5be00700 10 client.11328 ms_handle_connect on v2:172.21.15.159:6835/2157276692 2024-03-29T11:31:47.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b5be00700 1 -- 172.21.15.12:0/3537590479 <== mds.0 v2:172.21.15.159:6835/2157276692 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7f8b7405d8b0 con 0x55e3ad3eb600 2024-03-29T11:31:47.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b5be00700 10 client.11328 handle_command_reply: tid=0 2024-03-29T11:31:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 client.11328 shutdown 2024-03-29T11:31:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3537590479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 msgr2=0x55e3ad3eb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3eb0c0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c010c80 tx=0x7f8b6c017000 comp rx=0 tx=0).stop 2024-03-29T11:31:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3537590479 shutdown_connections 2024-03-29T11:31:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:6835/2157276692,v1:172.21.15.159:6837/2157276692] conn(0x55e3ad3eb600 0x55e3ad40fb80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e3ad415830 0x55e3ad409ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e3ad3e6b60 0x55e3ad3eb0c0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b5be00700 0 client.11328 ms_handle_reset on v2:172.21.15.159:6835/2157276692 2024-03-29T11:31:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 --2- 172.21.15.12:0/3537590479 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e3ad3e8be0 0x55e3ad409760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3537590479 >> 172.21.15.12:0/3537590479 conn(0x55e3ad3496a0 msgr2=0x55e3ad3e6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:47.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3537590479 shutdown_connections 2024-03-29T11:31:47.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.031+0000 7f8b95d55b80 1 -- 172.21.15.12:0/3537590479 wait complete. 2024-03-29T11:31:47.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b95d55b80 20 client.11328 trim_cache size 0 max 16384 2024-03-29T11:31:47.036 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:31:47.036 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T11:31:47.036 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T11:31:47.037 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 11025, 2024-03-29T11:31:47.037 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T11:31:47.037 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T11:31:47.037 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "backup_fs", 2024-03-29T11:31:47.037 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 6.165687159, 2024-03-29T11:31:47.037 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 112, 2024-03-29T11:31:47.037 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 231, 2024-03-29T11:31:47.037 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 231, 2024-03-29T11:31:47.037 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 26.019468058000001 2024-03-29T11:31:47.037 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:31:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 -- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8b5c06ade0 msgr2=0x7f8b5c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8b5c06ade0 0x7f8b5c06d2a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c006b20 tx=0x7f8b6c008040 comp rx=0 tx=0).stop 2024-03-29T11:31:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 -- 172.21.15.12:0/771747323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c15d9d0 msgr2=0x7f8b7c14e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c15d9d0 0x7f8b7c14e8e0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f8b74012000 tx=0x7f8b74004ad0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 -- 172.21.15.12:0/771747323 shutdown_connections 2024-03-29T11:31:47.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8b5c06ade0 0x7f8b5c06d2a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b7c15d9d0 0x7f8b7c14e8e0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b7c0d16a0 0x7f8b7c0cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 --2- 172.21.15.12:0/771747323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b7c0d2540 0x7f8b7c15fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 -- 172.21.15.12:0/771747323 >> 172.21.15.12:0/771747323 conn(0x7f8b7c08aff0 msgr2=0x7f8b7c06c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:47.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 -- 172.21.15.12:0/771747323 shutdown_connections 2024-03-29T11:31:47.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.032+0000 7f8b82200700 1 -- 172.21.15.12:0/771747323 wait complete. 2024-03-29T11:31:47.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T11:31:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.249+0000 7f2d31c00700 1 Processor -- start 2024-03-29T11:31:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.249+0000 7f2d31c00700 1 -- start start 2024-03-29T11:31:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.250+0000 7f2d31c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d2c15d9d0 0x7f2d2c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.250+0000 7f2d31c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2d2c0d16a0 0x7f2d2c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.250+0000 7f2d31c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c161420 con 0x7f2d2c0d2370 2024-03-29T11:31:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.250+0000 7f2d31c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c0d2820 con 0x7f2d2c0d16a0 2024-03-29T11:31:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.250+0000 7f2d31c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c161700 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.250+0000 7f2d2b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2d2c0d2370 0x7f2d2c15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47280/0 (socket says 172.21.15.12:47280) 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.250+0000 7f2d2b400700 1 -- 172.21.15.12:0/3660822744 learned_addr learned my addr 172.21.15.12:0/3660822744 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.251+0000 7f2d2aa00700 1 --2- 172.21.15.12:0/3660822744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d2c15d9d0 0x7f2d2c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.251+0000 7f2d2a000700 1 --2- 172.21.15.12:0/3660822744 >> v2:172.21.15.159:3300/0 conn(0x7f2d2c0d16a0 0x7f2d2c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.251+0000 7f2d29600700 1 -- 172.21.15.12:0/3660822744 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833775353 0 0) 0x7f2d2c161420 con 0x7f2d2c0d2370 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.251+0000 7f2d29600700 1 -- 172.21.15.12:0/3660822744 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d100035b0 con 0x7f2d2c0d2370 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.251+0000 7f2d2a000700 1 -- 172.21.15.12:0/3660822744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d2c15d9d0 msgr2=0x7f2d2c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.251+0000 7f2d2a000700 1 --2- 172.21.15.12:0/3660822744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d2c15d9d0 0x7f2d2c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.251+0000 7f2d2a000700 1 -- 172.21.15.12:0/3660822744 >> v1:172.21.15.12:6789/0 conn(0x7f2d2c0d2370 legacy=0x7f2d2c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.251+0000 7f2d2a000700 1 -- 172.21.15.12:0/3660822744 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c1619e0 con 0x7f2d2c0d16a0 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.251+0000 7f2d2a000700 1 --2- 172.21.15.12:0/3660822744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d2c0d16a0 0x7f2d2c0d1e30 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009820 tx=0x7f2d18009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97c753b3df75b543 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d29600700 1 -- 172.21.15.12:0/3660822744 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d18005e30 con 0x7f2d2c0d16a0 2024-03-29T11:31:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d29600700 1 -- 172.21.15.12:0/3660822744 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d18006030 con 0x7f2d2c0d16a0 2024-03-29T11:31:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d29600700 1 -- 172.21.15.12:0/3660822744 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d180063c0 con 0x7f2d2c0d16a0 2024-03-29T11:31:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d31c00700 1 -- 172.21.15.12:0/3660822744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d2c0d16a0 msgr2=0x7f2d2c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d31c00700 1 --2- 172.21.15.12:0/3660822744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d2c0d16a0 0x7f2d2c0d1e30 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009820 tx=0x7f2d18009b30 comp rx=0 tx=0).stop 2024-03-29T11:31:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d31c00700 1 -- 172.21.15.12:0/3660822744 shutdown_connections 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d31c00700 1 --2- 172.21.15.12:0/3660822744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d2c15d9d0 0x7f2d2c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d31c00700 1 --2- 172.21.15.12:0/3660822744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d2c0d16a0 0x7f2d2c0d1e30 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d31c00700 1 -- 172.21.15.12:0/3660822744 >> 172.21.15.12:0/3660822744 conn(0x7f2d2c08aff0 msgr2=0x7f2d2c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d31c00700 1 -- 172.21.15.12:0/3660822744 shutdown_connections 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.252+0000 7f2d31c00700 1 -- 172.21.15.12:0/3660822744 wait complete. 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.254+0000 7f2d31c00700 1 Processor -- start 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.254+0000 7f2d31c00700 1 -- start start 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d31c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d2c0d16a0 0x7f2d2c158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d31c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d2c0d2370 0x7f2d2c14ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d31c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2c15d9d0 0x7f2d2c0b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d31c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c14ef40 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d31c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0b6f70 con 0x7f2d2c0d2370 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d31c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0b74e0 con 0x7f2d2c0d16a0 2024-03-29T11:31:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d2b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2c15d9d0 0x7f2d2c0b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d2b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2c15d9d0 0x7f2d2c0b6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39674/0 (socket says 172.21.15.12:39674) 2024-03-29T11:31:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d2b400700 1 -- 172.21.15.12:0/3103959365 learned_addr learned my addr 172.21.15.12:0/3103959365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d2aa00700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d2c0d16a0 0x7f2d2c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.255+0000 7f2d2a000700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d2c0d2370 0x7f2d2c14ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d2b400700 1 -- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d2c0d16a0 msgr2=0x7f2d2c158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d2b400700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d2c0d16a0 0x7f2d2c158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d2b400700 1 -- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d2c0d2370 msgr2=0x7f2d2c14ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d2b400700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d2c0d2370 0x7f2d2c14ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d2b400700 1 -- 172.21.15.12:0/3103959365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18009430 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d2b400700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2c15d9d0 0x7f2d2c0b6a30 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400cb40 tx=0x7f2d2400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d22a00700 1 -- 172.21.15.12:0/3103959365 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2400a9e0 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d22a00700 1 -- 172.21.15.12:0/3103959365 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2400ab40 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d31c00700 1 -- 172.21.15.12:0/3103959365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c0b77c0 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d22a00700 1 -- 172.21.15.12:0/3103959365 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24012400 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.256+0000 7f2d31c00700 1 -- 172.21.15.12:0/3103959365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c0b7d80 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.257+0000 7f2d31c00700 1 -- 172.21.15.12:0/3103959365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c0d1ca0 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.258+0000 7f2d22a00700 1 -- 172.21.15.12:0/3103959365 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2d2401b3b0 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.259+0000 7f2d22a00700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2d0c06adb0 0x7f2d0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.259+0000 7f2d22a00700 1 -- 172.21.15.12:0/3103959365 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f2d24089430 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.260+0000 7f2d2aa00700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2d0c06adb0 0x7f2d0c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.261+0000 7f2d2aa00700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2d0c06adb0 0x7f2d0c06d270 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0c9bc0 tx=0x7f2d1c009210 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:47.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.264+0000 7f2d22a00700 1 -- 172.21.15.12:0/3103959365 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d24054160 con 0x7f2d2c15d9d0 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.509+0000 7f2d31c00700 1 -- 172.21.15.12:0/3103959365 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d2c161fa0 con 0x7f2d0c06adb0 2024-03-29T11:31:47.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.512+0000 7f2d22a00700 1 -- 172.21.15.12:0/3103959365 <== mgr.10459 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f2d1c011790 con 0x7f2d0c06adb0 2024-03-29T11:31:47.505 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:31:47.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:47.506 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T11:31:47.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:31:47.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:31:47.506 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "backup_fs" 2024-03-29T11:31:47.506 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:31:47.506 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:31:47.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 -- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2d0c06adb0 msgr2=0x7f2d0c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2d0c06adb0 0x7f2d0c06d270 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0c9bc0 tx=0x7f2d1c009210 comp rx=0 tx=0).stop 2024-03-29T11:31:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 -- 172.21.15.12:0/3103959365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2c15d9d0 msgr2=0x7f2d2c0b6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2c15d9d0 0x7f2d2c0b6a30 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400cb40 tx=0x7f2d2400ce50 comp rx=0 tx=0).stop 2024-03-29T11:31:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 -- 172.21.15.12:0/3103959365 shutdown_connections 2024-03-29T11:31:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2d0c06adb0 0x7f2d0c06d270 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2c15d9d0 0x7f2d2c0b6a30 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d2c0d16a0 0x7f2d2c158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 --2- 172.21.15.12:0/3103959365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d2c0d2370 0x7f2d2c14ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 -- 172.21.15.12:0/3103959365 >> 172.21.15.12:0/3103959365 conn(0x7f2d2c08aff0 msgr2=0x7f2d2c0b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:47.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 -- 172.21.15.12:0/3103959365 shutdown_connections 2024-03-29T11:31:47.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.513+0000 7f2d31c00700 1 -- 172.21.15.12:0/3103959365 wait complete. 2024-03-29T11:31:47.526 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T11:31:47.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.731+0000 7f48c4200700 1 Processor -- start 2024-03-29T11:31:47.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.731+0000 7f48c4200700 1 -- start start 2024-03-29T11:31:47.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 0x7f48bc0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c4200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f48bc0b3f80 0x7f48bc0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c4200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0cef00 con 0x7f48bc0d3f20 2024-03-29T11:31:47.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c4200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48bc161230 con 0x7f48bc0b3f80 2024-03-29T11:31:47.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc0d2a80 con 0x7f48bc160e50 2024-03-29T11:31:47.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c2200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f48bc0b3f80 0x7f48bc0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c0e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f48bc0d3f20 0x7f48bc0b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47290/0 (socket says 172.21.15.12:47290) 2024-03-29T11:31:47.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c0e00700 1 -- 172.21.15.12:0/1193939977 learned_addr learned my addr 172.21.15.12:0/1193939977 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:47.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c1800700 1 --2- 172.21.15.12:0/1193939977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 0x7f48bc0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c2200700 1 -- 172.21.15.12:0/1193939977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 msgr2=0x7f48bc0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:47.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c2200700 1 --2- 172.21.15.12:0/1193939977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 0x7f48bc0b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c2200700 1 -- 172.21.15.12:0/1193939977 >> v1:172.21.15.12:6789/0 conn(0x7f48bc0d3f20 legacy=0x7f48bc0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.733+0000 7f48c2200700 1 -- 172.21.15.12:0/1193939977 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc0d43b0 con 0x7f48bc0b3f80 2024-03-29T11:31:47.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.734+0000 7f48c2200700 1 --2- 172.21.15.12:0/1193939977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 0x7f48bc0b53d0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f48b4009d90 tx=0x7f48b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345b1f6dc36d18cd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:47.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.734+0000 7f48bbe00700 1 -- 172.21.15.12:0/1193939977 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b400e070 con 0x7f48bc0b3f80 2024-03-29T11:31:47.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.734+0000 7f48bbe00700 1 -- 172.21.15.12:0/1193939977 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b400bb20 con 0x7f48bc0b3f80 2024-03-29T11:31:47.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.734+0000 7f48bbe00700 1 -- 172.21.15.12:0/1193939977 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b4012430 con 0x7f48bc0b3f80 2024-03-29T11:31:47.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.734+0000 7f48c4200700 1 -- 172.21.15.12:0/1193939977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 msgr2=0x7f48bc0b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.734+0000 7f48c4200700 1 --2- 172.21.15.12:0/1193939977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 0x7f48bc0b53d0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f48b4009d90 tx=0x7f48b400b330 comp rx=0 tx=0).stop 2024-03-29T11:31:47.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.734+0000 7f48c4200700 1 -- 172.21.15.12:0/1193939977 shutdown_connections 2024-03-29T11:31:47.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.734+0000 7f48c4200700 1 --2- 172.21.15.12:0/1193939977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 0x7f48bc0b53d0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.734+0000 7f48c4200700 1 --2- 172.21.15.12:0/1193939977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 0x7f48bc0b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.734+0000 7f48c4200700 1 -- 172.21.15.12:0/1193939977 >> 172.21.15.12:0/1193939977 conn(0x7f48bc08aff0 msgr2=0x7f48bc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:47.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.735+0000 7f48c4200700 1 -- 172.21.15.12:0/1193939977 shutdown_connections 2024-03-29T11:31:47.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.735+0000 7f48c4200700 1 -- 172.21.15.12:0/1193939977 wait complete. 2024-03-29T11:31:47.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.737+0000 7f48c4200700 1 Processor -- start 2024-03-29T11:31:47.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.737+0000 7f48c4200700 1 -- start start 2024-03-29T11:31:47.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.738+0000 7f48c4200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 0x7f48bc158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.738+0000 7f48c4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 0x7f48bc0cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.738+0000 7f48c4200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0cb250 0x7f48bc1f6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.738+0000 7f48c4200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc1f7180 con 0x7f48bc0b3f80 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.738+0000 7f48c4200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc1f7460 con 0x7f48bc0cb250 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.738+0000 7f48c4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc0c6fa0 con 0x7f48bc160e50 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 0x7f48bc158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 0x7f48bc158b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39688/0 (socket says 172.21.15.12:39688) 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c1800700 1 -- 172.21.15.12:0/3202767673 learned_addr learned my addr 172.21.15.12:0/3202767673 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c1800700 1 -- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 msgr2=0x7f48bc0cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c2200700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0cb250 0x7f48bc1f6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c0e00700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 0x7f48bc0cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c1800700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 0x7f48bc0cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c1800700 1 -- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0cb250 msgr2=0x7f48bc1f6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c1800700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0cb250 0x7f48bc1f6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:47.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c1800700 1 -- 172.21.15.12:0/3202767673 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b4009a70 con 0x7f48bc0b3f80 2024-03-29T11:31:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c2200700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0cb250 0x7f48bc1f6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c0e00700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 0x7f48bc0cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:31:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.739+0000 7f48c1800700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 0x7f48bc158b00 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f48b000e9b0 tx=0x7f48b000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.740+0000 7f48b9600700 1 -- 172.21.15.12:0/3202767673 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b0009710 con 0x7f48bc0b3f80 2024-03-29T11:31:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.740+0000 7f48b9600700 1 -- 172.21.15.12:0/3202767673 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b000c850 con 0x7f48bc0b3f80 2024-03-29T11:31:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.740+0000 7f48b9600700 1 -- 172.21.15.12:0/3202767673 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b000cbc0 con 0x7f48bc0b3f80 2024-03-29T11:31:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.740+0000 7f48c4200700 1 -- 172.21.15.12:0/3202767673 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc0c7280 con 0x7f48bc0b3f80 2024-03-29T11:31:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.740+0000 7f48c4200700 1 -- 172.21.15.12:0/3202767673 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc0b19a0 con 0x7f48bc0b3f80 2024-03-29T11:31:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.741+0000 7f48c4200700 1 -- 172.21.15.12:0/3202767673 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc0c75f0 con 0x7f48bc0b3f80 2024-03-29T11:31:47.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.742+0000 7f48b9600700 1 -- 172.21.15.12:0/3202767673 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 129) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f48b000cd20 con 0x7f48bc0b3f80 2024-03-29T11:31:47.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.743+0000 7f48b9600700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489c06adb0 0x7f489c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:47.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.744+0000 7f48b9600700 1 -- 172.21.15.12:0/3202767673 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f48b0089a00 con 0x7f48bc0b3f80 2024-03-29T11:31:47.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.744+0000 7f48c0e00700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489c06adb0 0x7f489c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:47.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.745+0000 7f48c0e00700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489c06adb0 0x7f489c06d270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f48a8000f60 tx=0x7f48a8009040 comp rx=0 tx=0).ready entity=mgr.10459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:47.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:47.748+0000 7f48b9600700 1 -- 172.21.15.12:0/3202767673 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f48b0054710 con 0x7f48bc0b3f80 2024-03-29T11:31:48.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.025+0000 7f48c4200700 1 -- 172.21.15.12:0/3202767673 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f48bc0c7410 con 0x7f48bc0b3f80 2024-03-29T11:31:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.921+0000 7f48b9600700 1 -- 172.21.15.12:0/3202767673 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 130) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f48b0054ae0 con 0x7f48bc0b3f80 2024-03-29T11:31:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.923+0000 7f48b9600700 1 -- 172.21.15.12:0/3202767673 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v130) v1 ==== 92+0+0 (secure 0 0 0) 0x7f48b0087700 con 0x7f48bc0b3f80 2024-03-29T11:31:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 -- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489c06adb0 msgr2=0x7f489c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489c06adb0 0x7f489c06d270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f48a8000f60 tx=0x7f48a8009040 comp rx=0 tx=0).stop 2024-03-29T11:31:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 -- 172.21.15.12:0/3202767673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 msgr2=0x7f48bc158b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 0x7f48bc158b00 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f48b000e9b0 tx=0x7f48b000ed70 comp rx=0 tx=0).stop 2024-03-29T11:31:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 -- 172.21.15.12:0/3202767673 shutdown_connections 2024-03-29T11:31:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f489c06adb0 0x7f489c06d270 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0cb250 0x7f48bc1f6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48bc0b3f80 0x7f48bc158b00 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 --2- 172.21.15.12:0/3202767673 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc160e50 0x7f48bc0cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 -- 172.21.15.12:0/3202767673 >> 172.21.15.12:0/3202767673 conn(0x7f48bc08aff0 msgr2=0x7f48bc15e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 -- 172.21.15.12:0/3202767673 shutdown_connections 2024-03-29T11:31:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:48.927+0000 7f48c4200700 1 -- 172.21.15.12:0/3202767673 wait complete. 2024-03-29T11:31:48.944 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T11:31:49.142 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:49.147+0000 7f6031657240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:31:49.151 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:49.156+0000 7f2b70037240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:31:49.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.160+0000 7f693d400700 1 Processor -- start 2024-03-29T11:31:49.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.160+0000 7f693d400700 1 -- start start 2024-03-29T11:31:49.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.161+0000 7f693d400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6938160e50 0x7f69380b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.161+0000 7f693d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f69380b3f80 0x7f69380b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.161+0000 7f693d400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69380cef00 con 0x7f69380d3f20 2024-03-29T11:31:49.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.161+0000 7f693d400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6938161230 con 0x7f69380b3f80 2024-03-29T11:31:49.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.161+0000 7f693d400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69380d2a80 con 0x7f6938160e50 2024-03-29T11:31:49.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.161+0000 7f6937200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f69380b3f80 0x7f69380b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:49.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.161+0000 7f6935e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f69380d3f20 0x7f69380b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37332/0 (socket says 172.21.15.12:37332) 2024-03-29T11:31:49.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.161+0000 7f6935e00700 1 -- 172.21.15.12:0/1350017800 learned_addr learned my addr 172.21.15.12:0/1350017800 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:49.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.162+0000 7f6936800700 1 --2- 172.21.15.12:0/1350017800 >> v2:172.21.15.159:3300/0 conn(0x7f6938160e50 0x7f69380b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:49.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.162+0000 7f6937200700 1 -- 172.21.15.12:0/1350017800 >> v2:172.21.15.159:3300/0 conn(0x7f6938160e50 msgr2=0x7f69380b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:49.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.162+0000 7f6937200700 1 --2- 172.21.15.12:0/1350017800 >> v2:172.21.15.159:3300/0 conn(0x7f6938160e50 0x7f69380b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.162+0000 7f6937200700 1 -- 172.21.15.12:0/1350017800 >> v1:172.21.15.12:6789/0 conn(0x7f69380d3f20 legacy=0x7f69380b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:49.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.162+0000 7f6937200700 1 -- 172.21.15.12:0/1350017800 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69380d43b0 con 0x7f69380b3f80 2024-03-29T11:31:49.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.162+0000 7f6937200700 1 --2- 172.21.15.12:0/1350017800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 0x7f69380b53d0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f69380cff10 tx=0x7f693000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9710736116016ba server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:49.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f6935400700 1 -- 172.21.15.12:0/1350017800 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693000e070 con 0x7f69380b3f80 2024-03-29T11:31:49.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f6935400700 1 -- 172.21.15.12:0/1350017800 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693000bb20 con 0x7f69380b3f80 2024-03-29T11:31:49.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f6935400700 1 -- 172.21.15.12:0/1350017800 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6930012430 con 0x7f69380b3f80 2024-03-29T11:31:49.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f693d400700 1 -- 172.21.15.12:0/1350017800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 msgr2=0x7f69380b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:49.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f693d400700 1 --2- 172.21.15.12:0/1350017800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 0x7f69380b53d0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f69380cff10 tx=0x7f693000b330 comp rx=0 tx=0).stop 2024-03-29T11:31:49.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f693d400700 1 -- 172.21.15.12:0/1350017800 shutdown_connections 2024-03-29T11:31:49.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f693d400700 1 --2- 172.21.15.12:0/1350017800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 0x7f69380b53d0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f693d400700 1 --2- 172.21.15.12:0/1350017800 >> v2:172.21.15.159:3300/0 conn(0x7f6938160e50 0x7f69380b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f693d400700 1 -- 172.21.15.12:0/1350017800 >> 172.21.15.12:0/1350017800 conn(0x7f693808aff0 msgr2=0x7f69380d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:49.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f693d400700 1 -- 172.21.15.12:0/1350017800 shutdown_connections 2024-03-29T11:31:49.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.163+0000 7f693d400700 1 -- 172.21.15.12:0/1350017800 wait complete. 2024-03-29T11:31:49.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.165+0000 7f693d400700 1 Processor -- start 2024-03-29T11:31:49.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.165+0000 7f693d400700 1 -- start start 2024-03-29T11:31:49.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.166+0000 7f693d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 0x7f69380c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.166+0000 7f693d400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6938160e50 0x7f693815ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.166+0000 7f693d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69380ca600 0x7f69380caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.166+0000 7f693d400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69380b7de0 con 0x7f69380b3f80 2024-03-29T11:31:49.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.166+0000 7f693d400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69380cb7e0 con 0x7f6938160e50 2024-03-29T11:31:49.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.166+0000 7f693d400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693815b280 con 0x7f69380ca600 2024-03-29T11:31:49.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6936800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 0x7f69380c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:49.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6936800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 0x7f69380c6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53768/0 (socket says 172.21.15.12:53768) 2024-03-29T11:31:49.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6936800700 1 -- 172.21.15.12:0/2586425013 learned_addr learned my addr 172.21.15.12:0/2586425013 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:49.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6937200700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69380ca600 0x7f69380caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:49.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6935e00700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6938160e50 0x7f693815ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:49.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6936800700 1 -- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69380ca600 msgr2=0x7f69380caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:49.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6936800700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69380ca600 0x7f69380caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6936800700 1 -- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6938160e50 msgr2=0x7f693815ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:49.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6936800700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6938160e50 0x7f693815ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6936800700 1 -- 172.21.15.12:0/2586425013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6930009a70 con 0x7f69380b3f80 2024-03-29T11:31:49.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.167+0000 7f6936800700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 0x7f69380c6e00 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f692800e9e0 tx=0x7f692800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:49.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.168+0000 7f692ea00700 1 -- 172.21.15.12:0/2586425013 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6928009710 con 0x7f69380b3f80 2024-03-29T11:31:49.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.168+0000 7f692ea00700 1 -- 172.21.15.12:0/2586425013 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692800ee80 con 0x7f69380b3f80 2024-03-29T11:31:49.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.168+0000 7f692ea00700 1 -- 172.21.15.12:0/2586425013 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692800ca50 con 0x7f69380b3f80 2024-03-29T11:31:49.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.168+0000 7f693d400700 1 -- 172.21.15.12:0/2586425013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69380b7440 con 0x7f69380b3f80 2024-03-29T11:31:49.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.168+0000 7f693d400700 1 -- 172.21.15.12:0/2586425013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69380b7a00 con 0x7f69380b3f80 2024-03-29T11:31:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.169+0000 7f693d400700 1 -- 172.21.15.12:0/2586425013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69380b7f20 con 0x7f69380b3f80 2024-03-29T11:31:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.170+0000 7f692ea00700 1 -- 172.21.15.12:0/2586425013 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 130) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f692800cbb0 con 0x7f69380b3f80 2024-03-29T11:31:49.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.171+0000 7f692ea00700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f691806ae30 0x7f691806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.171+0000 7f692ea00700 1 -- 172.21.15.12:0/2586425013 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f69280898a0 con 0x7f69380b3f80 2024-03-29T11:31:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.171+0000 7f6935e00700 1 -- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f691806ae30 msgr2=0x7f691806d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:49.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.171+0000 7f6935e00700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f691806ae30 0x7f691806d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:49.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.180+0000 7f692ea00700 1 -- 172.21.15.12:0/2586425013 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69380b7f20 con 0x7f69380b3f80 2024-03-29T11:31:49.330 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:49.335+0000 7f6031657240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:31:49.362 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:49.367+0000 7f2b70037240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:31:49.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.372+0000 7f6935e00700 1 -- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f691806ae30 msgr2=0x7f691806d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:49.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.372+0000 7f6935e00700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f691806ae30 0x7f691806d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:49.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.435+0000 7f693d400700 1 -- 172.21.15.12:0/2586425013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f69380b7f20 con 0x7f69380b3f80 2024-03-29T11:31:49.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.439+0000 7f692ea00700 1 -- 172.21.15.12:0/2586425013 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f69380b7f20 con 0x7f69380b3f80 2024-03-29T11:31:49.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 -- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f691806ae30 msgr2=0x7f691806d2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:49.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f691806ae30 0x7f691806d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 -- 172.21.15.12:0/2586425013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 msgr2=0x7f69380c6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:49.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 0x7f69380c6e00 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f692800e9e0 tx=0x7f692800ecf0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 -- 172.21.15.12:0/2586425013 shutdown_connections 2024-03-29T11:31:49.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f691806ae30 0x7f691806d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69380ca600 0x7f69380caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69380b3f80 0x7f69380c6e00 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 --2- 172.21.15.12:0/2586425013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6938160e50 0x7f693815ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 -- 172.21.15.12:0/2586425013 >> 172.21.15.12:0/2586425013 conn(0x7f693808aff0 msgr2=0x7f69380b2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:49.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 -- 172.21.15.12:0/2586425013 shutdown_connections 2024-03-29T11:31:49.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.442+0000 7f693d400700 1 -- 172.21.15.12:0/2586425013 wait complete. 2024-03-29T11:31:49.458 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T11:31:49.647 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:49.652+0000 7f6031657240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:31:49.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.668+0000 7fa8d4200700 1 Processor -- start 2024-03-29T11:31:49.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.668+0000 7fa8d4200700 1 -- start start 2024-03-29T11:31:49.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.669+0000 7fa8d4200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0d21f0 0x7fa8cc0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.669+0000 7fa8d4200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa8cc0cec40 0x7fa8cc161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.669+0000 7fa8d4200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8cc0ce270 con 0x7fa8cc1611b0 2024-03-29T11:31:49.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.669+0000 7fa8d4200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8cc15b600 con 0x7fa8cc0d21f0 2024-03-29T11:31:49.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.669+0000 7fa8d4200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8cc0d2760 con 0x7fa8cc0cec40 2024-03-29T11:31:49.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.670+0000 7fa8d1800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa8cc1611b0 0x7fa8cc15ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37346/0 (socket says 172.21.15.12:37346) 2024-03-29T11:31:49.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.670+0000 7fa8d0e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0d21f0 0x7fa8cc0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:49.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.670+0000 7fa8d1800700 1 -- 172.21.15.12:0/347977928 learned_addr learned my addr 172.21.15.12:0/347977928 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:49.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.670+0000 7fa8d2200700 1 --2- 172.21.15.12:0/347977928 >> v2:172.21.15.159:3300/0 conn(0x7fa8cc0cec40 0x7fa8cc161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:49.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.670+0000 7fa8cbe00700 1 -- 172.21.15.12:0/347977928 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776786437 0 0) 0x7fa8cc0ce270 con 0x7fa8cc1611b0 2024-03-29T11:31:49.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.670+0000 7fa8cbe00700 1 -- 172.21.15.12:0/347977928 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b40035b0 con 0x7fa8cc1611b0 2024-03-29T11:31:49.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.670+0000 7fa8d0e00700 1 -- 172.21.15.12:0/347977928 >> v2:172.21.15.159:3300/0 conn(0x7fa8cc0cec40 msgr2=0x7fa8cc161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:49.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.670+0000 7fa8d0e00700 1 --2- 172.21.15.12:0/347977928 >> v2:172.21.15.159:3300/0 conn(0x7fa8cc0cec40 0x7fa8cc161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.670+0000 7fa8d0e00700 1 -- 172.21.15.12:0/347977928 >> v1:172.21.15.12:6789/0 conn(0x7fa8cc1611b0 legacy=0x7fa8cc15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.670+0000 7fa8d0e00700 1 -- 172.21.15.12:0/347977928 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc0d2a40 con 0x7fa8cc0d21f0 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.671+0000 7fa8d0e00700 1 --2- 172.21.15.12:0/347977928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc0d21f0 0x7fa8cc0ce700 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0b5cb0 tx=0x7fa8b8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d4f9947f49dbc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.671+0000 7fa8cbe00700 1 -- 172.21.15.12:0/347977928 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8011830 con 0x7fa8cc0d21f0 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.671+0000 7fa8cbe00700 1 -- 172.21.15.12:0/347977928 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b8011990 con 0x7fa8cc0d21f0 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.671+0000 7fa8cbe00700 1 -- 172.21.15.12:0/347977928 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8011d00 con 0x7fa8cc0d21f0 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.671+0000 7fa8d4200700 1 -- 172.21.15.12:0/347977928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc0d21f0 msgr2=0x7fa8cc0ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.671+0000 7fa8d4200700 1 --2- 172.21.15.12:0/347977928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc0d21f0 0x7fa8cc0ce700 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0b5cb0 tx=0x7fa8b8009b20 comp rx=0 tx=0).stop 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.672+0000 7fa8d4200700 1 -- 172.21.15.12:0/347977928 shutdown_connections 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.672+0000 7fa8d4200700 1 --2- 172.21.15.12:0/347977928 >> v2:172.21.15.159:3300/0 conn(0x7fa8cc0cec40 0x7fa8cc161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.672+0000 7fa8d4200700 1 --2- 172.21.15.12:0/347977928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc0d21f0 0x7fa8cc0ce700 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.672+0000 7fa8d4200700 1 -- 172.21.15.12:0/347977928 >> 172.21.15.12:0/347977928 conn(0x7fa8cc08aff0 msgr2=0x7fa8cc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:49.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.672+0000 7fa8d4200700 1 -- 172.21.15.12:0/347977928 shutdown_connections 2024-03-29T11:31:49.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.672+0000 7fa8d4200700 1 -- 172.21.15.12:0/347977928 wait complete. 2024-03-29T11:31:49.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.673+0000 7fa8d4200700 1 Processor -- start 2024-03-29T11:31:49.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.674+0000 7fa8d4200700 1 -- start start 2024-03-29T11:31:49.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.674+0000 7fa8d4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8cc0cec40 0x7fa8cc0ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.674+0000 7fa8d4200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8cc0d21f0 0x7fa8cc0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.674+0000 7fa8d4200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc1611b0 0x7fa8cc0c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.674+0000 7fa8d4200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc0c9230 con 0x7fa8cc1611b0 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.674+0000 7fa8d4200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc0c8620 con 0x7fa8cc0d21f0 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.674+0000 7fa8d4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc0c8900 con 0x7fa8cc0cec40 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc1611b0 0x7fa8cc0c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc1611b0 0x7fa8cc0c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53778/0 (socket says 172.21.15.12:53778) 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d2200700 1 -- 172.21.15.12:0/4063016364 learned_addr learned my addr 172.21.15.12:0/4063016364 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d1800700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8cc0cec40 0x7fa8cc0ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d0e00700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8cc0d21f0 0x7fa8cc0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d2200700 1 -- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8cc0cec40 msgr2=0x7fa8cc0ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d2200700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8cc0cec40 0x7fa8cc0ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d2200700 1 -- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8cc0d21f0 msgr2=0x7fa8cc0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d2200700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8cc0d21f0 0x7fa8cc0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d2200700 1 -- 172.21.15.12:0/4063016364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b80094a0 con 0x7fa8cc1611b0 2024-03-29T11:31:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d1800700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8cc0cec40 0x7fa8cc0ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d0e00700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8cc0d21f0 0x7fa8cc0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.675+0000 7fa8d2200700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc1611b0 0x7fa8cc0c96c0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400efc0 tx=0x7fa8c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.676+0000 7fa8c9600700 1 -- 172.21.15.12:0/4063016364 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c400cd40 con 0x7fa8cc1611b0 2024-03-29T11:31:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.676+0000 7fa8c9600700 1 -- 172.21.15.12:0/4063016364 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c400f040 con 0x7fa8cc1611b0 2024-03-29T11:31:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.676+0000 7fa8c9600700 1 -- 172.21.15.12:0/4063016364 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c40146c0 con 0x7fa8cc1611b0 2024-03-29T11:31:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.676+0000 7fa8d4200700 1 -- 172.21.15.12:0/4063016364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc0c8be0 con 0x7fa8cc1611b0 2024-03-29T11:31:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.676+0000 7fa8d4200700 1 -- 172.21.15.12:0/4063016364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc0cd290 con 0x7fa8cc1611b0 2024-03-29T11:31:49.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.677+0000 7fa8d4200700 1 -- 172.21.15.12:0/4063016364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc0c93a0 con 0x7fa8cc1611b0 2024-03-29T11:31:49.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.678+0000 7fa8c9600700 1 -- 172.21.15.12:0/4063016364 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 130) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fa8c4014820 con 0x7fa8cc1611b0 2024-03-29T11:31:49.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.679+0000 7fa8c9600700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa8ac06ae00 0x7fa8ac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:49.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.679+0000 7fa8c9600700 1 -- 172.21.15.12:0/4063016364 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7fa8c408a3b0 con 0x7fa8cc1611b0 2024-03-29T11:31:49.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.680+0000 7fa8d1800700 1 -- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa8ac06ae00 msgr2=0x7fa8ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:49.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.680+0000 7fa8d1800700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa8ac06ae00 0x7fa8ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:49.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.684+0000 7fa8c9600700 1 -- 172.21.15.12:0/4063016364 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa8c4054f20 con 0x7fa8cc1611b0 2024-03-29T11:31:49.705 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:49.710+0000 7f2b70037240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:31:49.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.881+0000 7fa8d1800700 1 -- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa8ac06ae00 msgr2=0x7fa8ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:49.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.881+0000 7fa8d1800700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa8ac06ae00 0x7fa8ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.898+0000 7fa8d4200700 1 -- 172.21.15.12:0/4063016364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa8cc067720 con 0x7fa8cc1611b0 2024-03-29T11:31:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.902+0000 7fa8c9600700 1 -- 172.21.15.12:0/4063016364 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa8c400cea0 con 0x7fa8cc1611b0 2024-03-29T11:31:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.904+0000 7fa8d4200700 1 -- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa8ac06ae00 msgr2=0x7fa8ac06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.904+0000 7fa8d4200700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa8ac06ae00 0x7fa8ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.904+0000 7fa8d4200700 1 -- 172.21.15.12:0/4063016364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc1611b0 msgr2=0x7fa8cc0c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.904+0000 7fa8d4200700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc1611b0 0x7fa8cc0c96c0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400efc0 tx=0x7fa8c400c3a0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.904+0000 7fa8d4200700 1 -- 172.21.15.12:0/4063016364 shutdown_connections 2024-03-29T11:31:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.904+0000 7fa8d4200700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa8ac06ae00 0x7fa8ac06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.904+0000 7fa8d4200700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8cc1611b0 0x7fa8cc0c96c0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.904+0000 7fa8d4200700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8cc0cec40 0x7fa8cc0ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.904+0000 7fa8d4200700 1 --2- 172.21.15.12:0/4063016364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8cc0d21f0 0x7fa8cc0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.904+0000 7fa8d4200700 1 -- 172.21.15.12:0/4063016364 >> 172.21.15.12:0/4063016364 conn(0x7fa8cc08aff0 msgr2=0x7fa8cc15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.905+0000 7fa8d4200700 1 -- 172.21.15.12:0/4063016364 shutdown_connections 2024-03-29T11:31:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:49.905+0000 7fa8d4200700 1 -- 172.21.15.12:0/4063016364 wait complete. 2024-03-29T11:31:49.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:31:49.918 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-03-29T11:31:49.973 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-29T11:31:50.175 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:50.180+0000 7f6031657240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:31:50.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.186+0000 7fcc1b600700 1 Processor -- start 2024-03-29T11:31:50.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.186+0000 7fcc1b600700 1 -- start start 2024-03-29T11:31:50.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.187+0000 7fcc1b600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcc1415ddf0 0x7fcc140ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.187+0000 7fcc1b600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415e1d0 0x7fcc140d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.187+0000 7fcc1b600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc140ce8a0 con 0x7fcc140d2710 2024-03-29T11:31:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.187+0000 7fcc1b600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc140d2b80 con 0x7fcc1415ddf0 2024-03-29T11:31:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.187+0000 7fcc1b600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1415ec60 con 0x7fcc1415e1d0 2024-03-29T11:31:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.188+0000 7fcc18c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcc1415ddf0 0x7fcc140ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.188+0000 7fcc19600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcc140d2710 0x7fcc141612f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37362/0 (socket says 172.21.15.12:37362) 2024-03-29T11:31:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.188+0000 7fcc19600700 1 -- 172.21.15.12:0/1718121504 learned_addr learned my addr 172.21.15.12:0/1718121504 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.188+0000 7fcc13e00700 1 --2- 172.21.15.12:0/1718121504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415e1d0 0x7fcc140d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.188+0000 7fcc18c00700 1 -- 172.21.15.12:0/1718121504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415e1d0 msgr2=0x7fcc140d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.188+0000 7fcc18c00700 1 --2- 172.21.15.12:0/1718121504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415e1d0 0x7fcc140d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.188+0000 7fcc18c00700 1 -- 172.21.15.12:0/1718121504 >> v1:172.21.15.12:6789/0 conn(0x7fcc140d2710 legacy=0x7fcc141612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.188+0000 7fcc18c00700 1 -- 172.21.15.12:0/1718121504 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc140cdc90 con 0x7fcc1415ddf0 2024-03-29T11:31:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.188+0000 7fcc13e00700 1 --2- 172.21.15.12:0/1718121504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415e1d0 0x7fcc140d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.188+0000 7fcc13400700 1 -- 172.21.15.12:0/1718121504 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055422762 0 0) 0x7fcc140ce8a0 con 0x7fcc140d2710 2024-03-29T11:31:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.189+0000 7fcc18c00700 1 --2- 172.21.15.12:0/1718121504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc1415ddf0 0x7fcc140ced30 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcc08009840 tx=0x7fcc08009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63f3a1cc1039d7bd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.189+0000 7fcc13400700 1 -- 172.21.15.12:0/1718121504 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08011830 con 0x7fcc1415ddf0 2024-03-29T11:31:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.189+0000 7fcc13400700 1 -- 172.21.15.12:0/1718121504 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc08011990 con 0x7fcc1415ddf0 2024-03-29T11:31:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.189+0000 7fcc13400700 1 -- 172.21.15.12:0/1718121504 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08011d20 con 0x7fcc1415ddf0 2024-03-29T11:31:50.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.189+0000 7fcc1b600700 1 -- 172.21.15.12:0/1718121504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc1415ddf0 msgr2=0x7fcc140ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:50.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.189+0000 7fcc1b600700 1 --2- 172.21.15.12:0/1718121504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc1415ddf0 0x7fcc140ced30 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcc08009840 tx=0x7fcc08009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:50.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.189+0000 7fcc1b600700 1 -- 172.21.15.12:0/1718121504 shutdown_connections 2024-03-29T11:31:50.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.189+0000 7fcc1b600700 1 --2- 172.21.15.12:0/1718121504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc1415ddf0 0x7fcc140ced30 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:50.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.189+0000 7fcc1b600700 1 --2- 172.21.15.12:0/1718121504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415e1d0 0x7fcc140d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:50.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.189+0000 7fcc1b600700 1 -- 172.21.15.12:0/1718121504 >> 172.21.15.12:0/1718121504 conn(0x7fcc1408aff0 msgr2=0x7fcc140d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:50.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.190+0000 7fcc1b600700 1 -- 172.21.15.12:0/1718121504 shutdown_connections 2024-03-29T11:31:50.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.190+0000 7fcc1b600700 1 -- 172.21.15.12:0/1718121504 wait complete. 2024-03-29T11:31:50.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.191+0000 7fcc1b600700 1 Processor -- start 2024-03-29T11:31:50.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.192+0000 7fcc1b600700 1 -- start start 2024-03-29T11:31:50.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.192+0000 7fcc1b600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc140d2710 0x7fcc1414c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:50.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc1b600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415ddf0 0x7fcc1415f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:50.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc1b600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc1415e1d0 0x7fcc140cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:50.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc1b600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1414cb30 con 0x7fcc1415e1d0 2024-03-29T11:31:50.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc1b600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc14160e60 con 0x7fcc140d2710 2024-03-29T11:31:50.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc1b600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc14161140 con 0x7fcc1415ddf0 2024-03-29T11:31:50.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc19600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc1415e1d0 0x7fcc140cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:50.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc18c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc140d2710 0x7fcc1414c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:50.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc18c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc140d2710 0x7fcc1414c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:33920/0 (socket says 172.21.15.12:33920) 2024-03-29T11:31:50.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc19600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc1415e1d0 0x7fcc140cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53792/0 (socket says 172.21.15.12:53792) 2024-03-29T11:31:50.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc13e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415ddf0 0x7fcc1415f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:50.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc18c00700 1 -- 172.21.15.12:0/2338447398 learned_addr learned my addr 172.21.15.12:0/2338447398 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:50.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc19600700 1 -- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415ddf0 msgr2=0x7fcc1415f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:50.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc19600700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415ddf0 0x7fcc1415f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:50.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc19600700 1 -- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc140d2710 msgr2=0x7fcc1414c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:50.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc19600700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc140d2710 0x7fcc1414c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:50.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc19600700 1 -- 172.21.15.12:0/2338447398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc000094a0 con 0x7fcc1415e1d0 2024-03-29T11:31:50.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.193+0000 7fcc18c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc140d2710 0x7fcc1414c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:50.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.194+0000 7fcc19600700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc1415e1d0 0x7fcc140cc7b0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00cbe0 tx=0x7fcc0c00cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:50.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.194+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00aa40 con 0x7fcc1415e1d0 2024-03-29T11:31:50.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.194+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0c00abc0 con 0x7fcc1415e1d0 2024-03-29T11:31:50.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.194+0000 7fcc1b600700 1 -- 172.21.15.12:0/2338447398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc080094a0 con 0x7fcc1415e1d0 2024-03-29T11:31:50.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.194+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0070d0 con 0x7fcc1415e1d0 2024-03-29T11:31:50.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.194+0000 7fcc1b600700 1 -- 172.21.15.12:0/2338447398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc14161700 con 0x7fcc1415e1d0 2024-03-29T11:31:50.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.195+0000 7fcc1b600700 1 -- 172.21.15.12:0/2338447398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc14068240 con 0x7fcc1415e1d0 2024-03-29T11:31:50.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.196+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 130) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fcc0c007230 con 0x7fcc1415e1d0 2024-03-29T11:31:50.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.198+0000 7fcc10c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:50.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.198+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6645+0+0 (secure 0 0 0) 0x7fcc0c0892c0 con 0x7fcc1415e1d0 2024-03-29T11:31:50.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.198+0000 7fcc18c00700 1 -- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 msgr2=0x7fcbf406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:50.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.198+0000 7fcc18c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:31:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.202+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcc0c053f40 con 0x7fcc1415e1d0 2024-03-29T11:31:50.275 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:50.281+0000 7f2b70037240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:31:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.399+0000 7fcc18c00700 1 -- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 msgr2=0x7fcbf406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.399+0000 7fcc18c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:31:50.436 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:50.441+0000 7f6031657240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:31:50.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.447+0000 7fcc1b600700 1 -- 172.21.15.12:0/2338447398 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc140ce530 con 0x7fcbf406ae00 2024-03-29T11:31:50.543 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:50.549+0000 7f2b70037240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:31:50.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.800+0000 7fcc18c00700 1 -- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 msgr2=0x7fcbf406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:50.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:50.800+0000 7fcc18c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T11:31:50.854 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:31:50.858 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:31:50.899 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:50.904+0000 7f6031657240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:31:51.027 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:51.032+0000 7f2b70037240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:31:51.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:51.601+0000 7fcc18c00700 1 -- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 msgr2=0x7fcbf406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:51.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:51.601+0000 7fcc18c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T11:31:51.616 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:51.620+0000 7f6031657240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:31:51.708 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:51.713+0000 7f6031657240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:31:51.774 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:51.779+0000 7f2b70037240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:31:51.867 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:51.872+0000 7f6031657240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:31:51.869 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:51.875+0000 7f2b70037240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:31:52.038 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:52.044+0000 7f2b70037240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:31:52.082 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:52.087+0000 7f6031657240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:31:52.174 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:52.179+0000 7f6031657240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:31:52.258 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:52.263+0000 7f2b70037240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:31:52.273 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:52.279+0000 7f6031657240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:31:52.354 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:52.359+0000 7f2b70037240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:31:52.371 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:52.377+0000 7f6031657240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:31:52.457 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:52.463+0000 7f2b70037240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:31:52.470 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:52.475+0000 7f6031657240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:31:52.560 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:52.566+0000 7f2b70037240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:31:52.663 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:52.669+0000 7f2b70037240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:31:52.681 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:52.686+0000 7f6031657240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:31:52.886 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:52.892+0000 7f2b70037240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:31:52.938 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:52.943+0000 7f6031657240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:31:53.131 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:53.137+0000 7f6031657240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:31:53.151 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:53.157+0000 7f2b70037240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:31:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:53.203+0000 7fcc18c00700 1 -- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 msgr2=0x7fcbf406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:31:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:53.203+0000 7fcc18c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T11:31:53.224 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:53.230+0000 7f6031657240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:53.350 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:53.356+0000 7f2b70037240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:31:53.448 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:53.453+0000 7f2b70037240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:53.955 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:53.961+0000 7f6031657240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:31:54.169 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:54.174+0000 7f6031657240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:31:54.212 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:54.218+0000 7f2b70037240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:31:54.377 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:54.383+0000 7f6031657240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:54.434 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:54.440+0000 7f2b70037240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:31:54.652 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:54.656+0000 7f2b70037240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:54.853 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:54.858+0000 7f6031657240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:31:55.062 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.066+0000 7f6031657240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:55.152 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:55.157+0000 7f2b70037240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:31:55.369 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:55.374+0000 7f2b70037240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:55.505 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.510+0000 7f6031657240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:31:55.617 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.623+0000 7f6031657240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:55.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:55.636+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 131) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fcc0c054310 con 0x7fcc1415e1d0 2024-03-29T11:31:55.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:55.636+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 msgr2=0x7fcbf406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:55.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:55.636+0000 7fcc10c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:55.845 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:55.850+0000 7f2b70037240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:31:55.922 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.927+0000 7f5f9d000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.922 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.927+0000 7f5f9d000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.922 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.927+0000 7f5f9d000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.922 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.927+0000 7f5f9d000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.922 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.927+0000 7f5f9d000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.922 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.927+0000 7f5f99400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.922 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.927+0000 7f5f99400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.923 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.927+0000 7f5f99400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.923 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.927+0000 7f5f99400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.923 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.927+0000 7f5f99400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.935 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.939+0000 7f5f9d000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.935 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.939+0000 7f5f9d000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.935 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.939+0000 7f5f9d000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.935 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.939+0000 7f5f9d000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.935 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.939+0000 7f5f9d000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.936 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.941+0000 7f5f99400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.936 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.941+0000 7f5f99400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.936 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.941+0000 7f5f99400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.936 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.941+0000 7f5f99400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.936 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:31:55.941+0000 7f5f99400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:31:55.973 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:31:55.977+0000 7f2b70037240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:56.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.654+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 132) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fcc0c087390 con 0x7fcc1415e1d0 2024-03-29T11:31:56.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.655+0000 7fcc10c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:56.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.655+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbf401d2d0 con 0x7fcbf406ae00 2024-03-29T11:31:56.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.658+0000 7fcc18c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:56.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.661+0000 7fcc18c00700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcc08000c00 tx=0x7fcc08008030 comp rx=0 tx=0).ready entity=mgr.10918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:56.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.729+0000 7fcc10c00700 1 -- 172.21.15.12:0/2338447398 <== mgr.10918 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7fcbf401d2d0 con 0x7fcbf406ae00 2024-03-29T11:31:56.724 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path mandatory_fs_argument_test_dir 2024-03-29T11:31:56.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 -- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 msgr2=0x7fcbf406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcc08000c00 tx=0x7fcc08008030 comp rx=0 tx=0).stop 2024-03-29T11:31:56.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 -- 172.21.15.12:0/2338447398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc1415e1d0 msgr2=0x7fcc140cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc1415e1d0 0x7fcc140cc7b0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00cbe0 tx=0x7fcc0c00cef0 comp rx=0 tx=0).stop 2024-03-29T11:31:56.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 -- 172.21.15.12:0/2338447398 shutdown_connections 2024-03-29T11:31:56.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcbf406ae00 0x7fcbf406d2c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:56.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc1415e1d0 0x7fcc140cc7b0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:56.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc140d2710 0x7fcc1414c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:56.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 --2- 172.21.15.12:0/2338447398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc1415ddf0 0x7fcc1415f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:56.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 -- 172.21.15.12:0/2338447398 >> 172.21.15.12:0/2338447398 conn(0x7fcc1408aff0 msgr2=0x7fcc140b8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:56.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.731+0000 7fcc1b600700 1 -- 172.21.15.12:0/2338447398 shutdown_connections 2024-03-29T11:31:56.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.732+0000 7fcc1b600700 1 -- 172.21.15.12:0/2338447398 wait complete. 2024-03-29T11:31:56.747 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2024-03-29T11:31:56.747 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-29T11:31:56.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.959+0000 7f5ec8a00700 1 Processor -- start 2024-03-29T11:31:56.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.959+0000 7f5ec8a00700 1 -- start start 2024-03-29T11:31:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.960+0000 7f5ec8a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5ec4155ac0 0x7f5ec415c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.960+0000 7f5ec8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec406c910 0x7f5ec40adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.960+0000 7f5ec8a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec415dcc0 con 0x7f5ec40ae4a0 2024-03-29T11:31:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.960+0000 7f5ec8a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec406dd00 con 0x7f5ec4155ac0 2024-03-29T11:31:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.960+0000 7f5ec8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec406d0f0 con 0x7f5ec406c910 2024-03-29T11:31:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.960+0000 7f5ec2800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5ec40ae4a0 0x7f5ec415f0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37364/0 (socket says 172.21.15.12:37364) 2024-03-29T11:31:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.960+0000 7f5ec2800700 1 -- 172.21.15.12:0/1247215253 learned_addr learned my addr 172.21.15.12:0/1247215253 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:56.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec1400700 1 --2- 172.21.15.12:0/1247215253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec406c910 0x7f5ec40adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:56.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec1e00700 1 --2- 172.21.15.12:0/1247215253 >> v2:172.21.15.159:3300/0 conn(0x7f5ec4155ac0 0x7f5ec415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:56.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec0a00700 1 -- 172.21.15.12:0/1247215253 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995386332 0 0) 0x7f5ec415dcc0 con 0x7f5ec40ae4a0 2024-03-29T11:31:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec0a00700 1 -- 172.21.15.12:0/1247215253 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea80035b0 con 0x7f5ec40ae4a0 2024-03-29T11:31:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec0a00700 1 -- 172.21.15.12:0/1247215253 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3767053100 0 0) 0x7f5ea80035b0 con 0x7f5ec40ae4a0 2024-03-29T11:31:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec1e00700 1 -- 172.21.15.12:0/1247215253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec406c910 msgr2=0x7f5ec40adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec1e00700 1 --2- 172.21.15.12:0/1247215253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec406c910 0x7f5ec40adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec1e00700 1 -- 172.21.15.12:0/1247215253 >> v1:172.21.15.12:6789/0 conn(0x7f5ec40ae4a0 legacy=0x7f5ec415f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec1e00700 1 -- 172.21.15.12:0/1247215253 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec406d3d0 con 0x7f5ec4155ac0 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec1400700 1 --2- 172.21.15.12:0/1247215253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec406c910 0x7f5ec40adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.961+0000 7f5ec1e00700 1 --2- 172.21.15.12:0/1247215253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ec4155ac0 0x7f5ec415c700 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009840 tx=0x7f5eb4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecf60650645cc9f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec0a00700 1 -- 172.21.15.12:0/1247215253 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc0019f0 con 0x7f5ec4155ac0 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec0a00700 1 -- 172.21.15.12:0/1247215253 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb4004030 con 0x7f5ec4155ac0 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec0a00700 1 -- 172.21.15.12:0/1247215253 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb40043c0 con 0x7f5ec4155ac0 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec8a00700 1 -- 172.21.15.12:0/1247215253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ec4155ac0 msgr2=0x7f5ec415c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec8a00700 1 --2- 172.21.15.12:0/1247215253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ec4155ac0 0x7f5ec415c700 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009840 tx=0x7f5eb4009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec8a00700 1 -- 172.21.15.12:0/1247215253 shutdown_connections 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec8a00700 1 --2- 172.21.15.12:0/1247215253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ec4155ac0 0x7f5ec415c700 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec8a00700 1 --2- 172.21.15.12:0/1247215253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec406c910 0x7f5ec40adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec8a00700 1 -- 172.21.15.12:0/1247215253 >> 172.21.15.12:0/1247215253 conn(0x7f5ec40cf210 msgr2=0x7f5ec40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec8a00700 1 -- 172.21.15.12:0/1247215253 shutdown_connections 2024-03-29T11:31:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.962+0000 7f5ec8a00700 1 -- 172.21.15.12:0/1247215253 wait complete. 2024-03-29T11:31:56.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.965+0000 7f5ec8a00700 1 Processor -- start 2024-03-29T11:31:56.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.965+0000 7f5ec8a00700 1 -- start start 2024-03-29T11:31:56.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.966+0000 7f5ec8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ec406c910 0x7f5ec414f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:56.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.966+0000 7f5ec8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec40ae4a0 0x7f5ec40b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:56.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.966+0000 7f5ec8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ec4155ac0 0x7f5ec415f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:56.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.966+0000 7f5ec8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec4150920 con 0x7f5ec406c910 2024-03-29T11:31:56.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.966+0000 7f5ec8a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec41f33e0 con 0x7f5ec4155ac0 2024-03-29T11:31:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.966+0000 7f5ec8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec414f880 con 0x7f5ec40ae4a0 2024-03-29T11:31:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.966+0000 7f5ec1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ec406c910 0x7f5ec414f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.966+0000 7f5ec1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ec406c910 0x7f5ec414f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53794/0 (socket says 172.21.15.12:53794) 2024-03-29T11:31:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.966+0000 7f5ec1e00700 1 -- 172.21.15.12:0/572009035 learned_addr learned my addr 172.21.15.12:0/572009035 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5ec1400700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec40ae4a0 0x7f5ec40b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5ec2800700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ec4155ac0 0x7f5ec415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5ec1e00700 1 -- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec40ae4a0 msgr2=0x7f5ec40b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5ec1e00700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec40ae4a0 0x7f5ec40b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5ec1e00700 1 -- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ec4155ac0 msgr2=0x7f5ec415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5ec1e00700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ec4155ac0 0x7f5ec415f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5ec1e00700 1 -- 172.21.15.12:0/572009035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00094a0 con 0x7f5ec406c910 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5ec1e00700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ec406c910 0x7f5ec414f1a0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400a010 tx=0x7f5eb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5eba000700 1 -- 172.21.15.12:0/572009035 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4003b90 con 0x7f5ec406c910 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5eba000700 1 -- 172.21.15.12:0/572009035 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb401f070 con 0x7f5ec406c910 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5ec8a00700 1 -- 172.21.15.12:0/572009035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb40094a0 con 0x7f5ec406c910 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.967+0000 7f5eba000700 1 -- 172.21.15.12:0/572009035 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4011630 con 0x7f5ec406c910 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.968+0000 7f5ec8a00700 1 -- 172.21.15.12:0/572009035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec414fe40 con 0x7f5ec406c910 2024-03-29T11:31:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.969+0000 7f5ec8a00700 1 -- 172.21.15.12:0/572009035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec40b3400 con 0x7f5ec406c910 2024-03-29T11:31:56.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.969+0000 7f5eba000700 1 -- 172.21.15.12:0/572009035 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 132) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f5eb4021030 con 0x7f5ec406c910 2024-03-29T11:31:56.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.970+0000 7f5eba000700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ea406ab30 0x7f5ea406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:56.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.971+0000 7f5eba000700 1 -- 172.21.15.12:0/572009035 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f5eb408a7e0 con 0x7f5ec406c910 2024-03-29T11:31:56.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.971+0000 7f5ec1400700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ea406ab30 0x7f5ea406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:56.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.971+0000 7f5ec1400700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ea406ab30 0x7f5ea406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0009d70 tx=0x7f5eb0009210 comp rx=0 tx=0).ready entity=mgr.10918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:56.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:56.975+0000 7f5eba000700 1 -- 172.21.15.12:0/572009035 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5eb4055730 con 0x7f5ec406c910 2024-03-29T11:31:57.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.218+0000 7f5ec8a00700 1 -- 172.21.15.12:0/572009035 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ec406d990 con 0x7f5ea406ab30 2024-03-29T11:31:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.269+0000 7f5eba000700 1 -- 172.21.15.12:0/572009035 <== mgr.10918 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f5eb00099c0 con 0x7f5ea406ab30 2024-03-29T11:31:57.265 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2024-03-29T11:31:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.271+0000 7f5ec8a00700 1 -- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ea406ab30 msgr2=0x7f5ea406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.271+0000 7f5ec8a00700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ea406ab30 0x7f5ea406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0009d70 tx=0x7f5eb0009210 comp rx=0 tx=0).stop 2024-03-29T11:31:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.271+0000 7f5ec8a00700 1 -- 172.21.15.12:0/572009035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ec406c910 msgr2=0x7f5ec414f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.271+0000 7f5ec8a00700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ec406c910 0x7f5ec414f1a0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400a010 tx=0x7f5eb4004750 comp rx=0 tx=0).stop 2024-03-29T11:31:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.271+0000 7f5ec8a00700 1 -- 172.21.15.12:0/572009035 shutdown_connections 2024-03-29T11:31:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.271+0000 7f5ec8a00700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ea406ab30 0x7f5ea406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.271+0000 7f5ec8a00700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ec4155ac0 0x7f5ec415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.271+0000 7f5ec8a00700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ec406c910 0x7f5ec414f1a0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.271+0000 7f5ec8a00700 1 --2- 172.21.15.12:0/572009035 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ec40ae4a0 0x7f5ec40b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.271+0000 7f5ec8a00700 1 -- 172.21.15.12:0/572009035 >> 172.21.15.12:0/572009035 conn(0x7f5ec40cf210 msgr2=0x7f5ec415c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.272+0000 7f5ec8a00700 1 -- 172.21.15.12:0/572009035 shutdown_connections 2024-03-29T11:31:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.272+0000 7f5ec8a00700 1 -- 172.21.15.12:0/572009035 wait complete. 2024-03-29T11:31:57.285 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2024-03-29T11:31:57.285 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-29T11:31:57.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.496+0000 7f2620400700 1 Processor -- start 2024-03-29T11:31:57.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.496+0000 7f2620400700 1 -- start start 2024-03-29T11:31:57.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.497+0000 7f2620400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f26180cbb30 0x7f26180cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:57.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.497+0000 7f2620400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180cd020 0x7f26180cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:57.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.497+0000 7f2620400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26180cc450 con 0x7f26180b2070 2024-03-29T11:31:57.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.497+0000 7f2620400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26180cdae0 con 0x7f26180cbb30 2024-03-29T11:31:57.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.497+0000 7f2620400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26180cddc0 con 0x7f26180cd020 2024-03-29T11:31:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.498+0000 7f261d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180cd020 0x7f26180cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.498+0000 7f261d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180cd020 0x7f26180cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:38986/0 (socket says 172.21.15.12:38986) 2024-03-29T11:31:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.498+0000 7f261e400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f26180b2070 0x7f261806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:38234/0 (socket says 172.21.15.12:38234) 2024-03-29T11:31:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.498+0000 7f261d000700 1 -- 172.21.15.12:0/92766674 learned_addr learned my addr 172.21.15.12:0/92766674 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.498+0000 7f261da00700 1 --2- 172.21.15.12:0/92766674 >> v2:172.21.15.159:3300/0 conn(0x7f26180cbb30 0x7f26180cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.498+0000 7f261d000700 1 -- 172.21.15.12:0/92766674 >> v2:172.21.15.159:3300/0 conn(0x7f26180cbb30 msgr2=0x7f26180cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.498+0000 7f261d000700 1 --2- 172.21.15.12:0/92766674 >> v2:172.21.15.159:3300/0 conn(0x7f26180cbb30 0x7f26180cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.498+0000 7f2617e00700 1 -- 172.21.15.12:0/92766674 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284020141 0 0) 0x7f26180cc450 con 0x7f26180b2070 2024-03-29T11:31:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.498+0000 7f261d000700 1 -- 172.21.15.12:0/92766674 >> v1:172.21.15.159:6789/0 conn(0x7f26180b2070 legacy=0x7f261806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:57.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.498+0000 7f261d000700 1 -- 172.21.15.12:0/92766674 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26180ce0a0 con 0x7f26180cd020 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.499+0000 7f261d000700 1 --2- 172.21.15.12:0/92766674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180cd020 0x7f26180cd480 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f26040098f0 tx=0x7f2604009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7da619ded1426097 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.499+0000 7f2617e00700 1 -- 172.21.15.12:0/92766674 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2604011830 con 0x7f26180cd020 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.499+0000 7f2617e00700 1 -- 172.21.15.12:0/92766674 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2604011990 con 0x7f26180cd020 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.499+0000 7f2617e00700 1 -- 172.21.15.12:0/92766674 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2604011d20 con 0x7f26180cd020 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.499+0000 7f2620400700 1 -- 172.21.15.12:0/92766674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180cd020 msgr2=0x7f26180cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.499+0000 7f2620400700 1 --2- 172.21.15.12:0/92766674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180cd020 0x7f26180cd480 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f26040098f0 tx=0x7f2604009c00 comp rx=0 tx=0).stop 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.499+0000 7f2620400700 1 -- 172.21.15.12:0/92766674 shutdown_connections 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.499+0000 7f2620400700 1 --2- 172.21.15.12:0/92766674 >> v2:172.21.15.159:3300/0 conn(0x7f26180cbb30 0x7f26180cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.499+0000 7f2620400700 1 --2- 172.21.15.12:0/92766674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180cd020 0x7f26180cd480 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.499+0000 7f2620400700 1 -- 172.21.15.12:0/92766674 >> 172.21.15.12:0/92766674 conn(0x7f26180b0d70 msgr2=0x7f26180b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.500+0000 7f2620400700 1 -- 172.21.15.12:0/92766674 shutdown_connections 2024-03-29T11:31:57.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.500+0000 7f2620400700 1 -- 172.21.15.12:0/92766674 wait complete. 2024-03-29T11:31:57.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.501+0000 7f2620400700 1 Processor -- start 2024-03-29T11:31:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.501+0000 7f2620400700 1 -- start start 2024-03-29T11:31:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.502+0000 7f2620400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180b2070 0x7f26180989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.502+0000 7f2620400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26180cbb30 0x7f261815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.502+0000 7f2620400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26180cd020 0x7f2618155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.502+0000 7f2620400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26181503b0 con 0x7f26180cd020 2024-03-29T11:31:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.502+0000 7f2620400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2618098ef0 con 0x7f26180cbb30 2024-03-29T11:31:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.502+0000 7f2620400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2618154090 con 0x7f26180b2070 2024-03-29T11:31:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26180cd020 0x7f2618155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261da00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180b2070 0x7f26180989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26180cbb30 0x7f261815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26180cd020 0x7f2618155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53800/0 (socket says 172.21.15.12:53800) 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261e400700 1 -- 172.21.15.12:0/2186690480 learned_addr learned my addr 172.21.15.12:0/2186690480 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261e400700 1 -- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180b2070 msgr2=0x7f26180989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261e400700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180b2070 0x7f26180989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261e400700 1 -- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26180cbb30 msgr2=0x7f261815b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261e400700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26180cbb30 0x7f261815b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261e400700 1 -- 172.21.15.12:0/2186690480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2604009550 con 0x7f26180cd020 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261da00700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180b2070 0x7f26180989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.503+0000 7f261e400700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26180cd020 0x7f2618155130 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f261000cba0 tx=0x7f261000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.504+0000 7f2615600700 1 -- 172.21.15.12:0/2186690480 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261000aaa0 con 0x7f26180cd020 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.504+0000 7f2615600700 1 -- 172.21.15.12:0/2186690480 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261000ac20 con 0x7f26180cd020 2024-03-29T11:31:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.504+0000 7f2615600700 1 -- 172.21.15.12:0/2186690480 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26100070d0 con 0x7f26180cd020 2024-03-29T11:31:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.504+0000 7f2620400700 1 -- 172.21.15.12:0/2186690480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2618154370 con 0x7f26180cd020 2024-03-29T11:31:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.504+0000 7f2620400700 1 -- 172.21.15.12:0/2186690480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2618154930 con 0x7f26180cd020 2024-03-29T11:31:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.505+0000 7f2620400700 1 -- 172.21.15.12:0/2186690480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2618067720 con 0x7f26180cd020 2024-03-29T11:31:57.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.506+0000 7f2615600700 1 -- 172.21.15.12:0/2186690480 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 132) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f2610007230 con 0x7f26180cd020 2024-03-29T11:31:57.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.507+0000 7f2615600700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f25f806ab30 0x7f25f806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:57.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.507+0000 7f2615600700 1 -- 172.21.15.12:0/2186690480 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f2610088ff0 con 0x7f26180cd020 2024-03-29T11:31:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.507+0000 7f261da00700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f25f806ab30 0x7f25f806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.508+0000 7f261da00700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f25f806ab30 0x7f25f806cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f260c009f30 tx=0x7f260c009210 comp rx=0 tx=0).ready entity=mgr.10918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.511+0000 7f2615600700 1 -- 172.21.15.12:0/2186690480 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26100bc580 con 0x7f26180cd020 2024-03-29T11:31:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.693+0000 7f2615600700 1 -- 172.21.15.12:0/2186690480 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 133) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f2610054310 con 0x7f26180cd020 2024-03-29T11:31:57.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.758+0000 7f2620400700 1 -- 172.21.15.12:0/2186690480 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f26181594a0 con 0x7f25f806ab30 2024-03-29T11:31:57.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.813+0000 7f2615600700 1 -- 172.21.15.12:0/2186690480 <== mgr.10918 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f260c011bf0 con 0x7f25f806ab30 2024-03-29T11:31:57.811 INFO:teuthology.orchestra.run.smithi012.stdout:Retention added to path mandatory_fs_argument_test_dir 2024-03-29T11:31:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 -- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f25f806ab30 msgr2=0x7f25f806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f25f806ab30 0x7f25f806cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f260c009f30 tx=0x7f260c009210 comp rx=0 tx=0).stop 2024-03-29T11:31:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 -- 172.21.15.12:0/2186690480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26180cd020 msgr2=0x7f2618155130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26180cd020 0x7f2618155130 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f261000cba0 tx=0x7f261000ceb0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 -- 172.21.15.12:0/2186690480 shutdown_connections 2024-03-29T11:31:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f25f806ab30 0x7f25f806cff0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26180cd020 0x7f2618155130 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26180b2070 0x7f26180989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 --2- 172.21.15.12:0/2186690480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26180cbb30 0x7f261815b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:57.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 -- 172.21.15.12:0/2186690480 >> 172.21.15.12:0/2186690480 conn(0x7f26180b0d70 msgr2=0x7f261806cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:57.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 -- 172.21.15.12:0/2186690480 shutdown_connections 2024-03-29T11:31:57.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:57.816+0000 7f2620400700 1 -- 172.21.15.12:0/2186690480 wait complete. 2024-03-29T11:31:57.832 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2024-03-29T11:31:57.832 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-03-29T11:31:58.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.040+0000 7f51f7600700 1 Processor -- start 2024-03-29T11:31:58.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.040+0000 7f51f7600700 1 -- start start 2024-03-29T11:31:58.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.041+0000 7f51f7600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51f015d930 0x7f51f00d3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.041+0000 7f51f7600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f51f00b4fa0 0x7f51f00b5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.041+0000 7f51f7600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f01623a0 con 0x7f51f015dd10 2024-03-29T11:31:58.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.041+0000 7f51f7600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f00b5960 con 0x7f51f00b4fa0 2024-03-29T11:31:58.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.041+0000 7f51f7600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f01614b0 con 0x7f51f015d930 2024-03-29T11:31:58.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.041+0000 7f51f5600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f51f015dd10 0x7f51f015e1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37370/0 (socket says 172.21.15.12:37370) 2024-03-29T11:31:58.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.042+0000 7f51f5600700 1 -- 172.21.15.12:0/760236235 learned_addr learned my addr 172.21.15.12:0/760236235 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:58.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.041+0000 7f51efe00700 1 --2- 172.21.15.12:0/760236235 >> v2:172.21.15.12:3300/0 conn(0x7f51f00b4fa0 0x7f51f00b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:58.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.042+0000 7f51f4c00700 1 --2- 172.21.15.12:0/760236235 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51f015d930 0x7f51f00d3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:58.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.042+0000 7f51ef400700 1 -- 172.21.15.12:0/760236235 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303979988 0 0) 0x7f51f01623a0 con 0x7f51f015dd10 2024-03-29T11:31:58.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.042+0000 7f51efe00700 1 -- 172.21.15.12:0/760236235 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51f015d930 msgr2=0x7f51f00d3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:58.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.042+0000 7f51efe00700 1 --2- 172.21.15.12:0/760236235 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51f015d930 0x7f51f00d3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.042+0000 7f51efe00700 1 -- 172.21.15.12:0/760236235 >> v1:172.21.15.12:6789/0 conn(0x7f51f015dd10 legacy=0x7f51f015e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.042+0000 7f51efe00700 1 -- 172.21.15.12:0/760236235 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0161790 con 0x7f51f00b4fa0 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.042+0000 7f51efe00700 1 --2- 172.21.15.12:0/760236235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f00b4fa0 0x7f51f00b5420 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009840 tx=0x7f51dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ec454913c41f0e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51ef400700 1 -- 172.21.15.12:0/760236235 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc011830 con 0x7f51f00b4fa0 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51ef400700 1 -- 172.21.15.12:0/760236235 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc011990 con 0x7f51f00b4fa0 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51ef400700 1 -- 172.21.15.12:0/760236235 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc011d20 con 0x7f51f00b4fa0 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51f7600700 1 -- 172.21.15.12:0/760236235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f00b4fa0 msgr2=0x7f51f00b5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51f7600700 1 --2- 172.21.15.12:0/760236235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f00b4fa0 0x7f51f00b5420 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009840 tx=0x7f51dc009b50 comp rx=0 tx=0).stop 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51f7600700 1 -- 172.21.15.12:0/760236235 shutdown_connections 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51f7600700 1 --2- 172.21.15.12:0/760236235 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51f015d930 0x7f51f00d3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51f7600700 1 --2- 172.21.15.12:0/760236235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f00b4fa0 0x7f51f00b5420 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51f7600700 1 -- 172.21.15.12:0/760236235 >> 172.21.15.12:0/760236235 conn(0x7f51f008aff0 msgr2=0x7f51f00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51f7600700 1 -- 172.21.15.12:0/760236235 shutdown_connections 2024-03-29T11:31:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.043+0000 7f51f7600700 1 -- 172.21.15.12:0/760236235 wait complete. 2024-03-29T11:31:58.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.045+0000 7f51f7600700 1 Processor -- start 2024-03-29T11:31:58.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.045+0000 7f51f7600700 1 -- start start 2024-03-29T11:31:58.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.046+0000 7f51f7600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51f00b4fa0 0x7f51f00b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.046+0000 7f51f7600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f015d930 0x7f51f0160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.046+0000 7f51f7600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51f015dd10 0x7f51f01f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.046+0000 7f51f7600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f01f6ad0 con 0x7f51f015d930 2024-03-29T11:31:58.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.046+0000 7f51f7600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f0160ae0 con 0x7f51f015dd10 2024-03-29T11:31:58.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.046+0000 7f51f7600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f0160dc0 con 0x7f51f00b4fa0 2024-03-29T11:31:58.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.046+0000 7f51efe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f015d930 0x7f51f0160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.046+0000 7f51efe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f015d930 0x7f51f0160270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53828/0 (socket says 172.21.15.12:53828) 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.046+0000 7f51efe00700 1 -- 172.21.15.12:0/2668972887 learned_addr learned my addr 172.21.15.12:0/2668972887 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.046+0000 7f51f4c00700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51f00b4fa0 0x7f51f00b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51f5600700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51f015dd10 0x7f51f01f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51efe00700 1 -- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51f00b4fa0 msgr2=0x7f51f00b8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51efe00700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51f00b4fa0 0x7f51f00b8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51efe00700 1 -- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51f015dd10 msgr2=0x7f51f01f6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51efe00700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51f015dd10 0x7f51f01f6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51efe00700 1 -- 172.21.15.12:0/2668972887 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc0094a0 con 0x7f51f015d930 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51f5600700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51f015dd10 0x7f51f01f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51efe00700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f015d930 0x7f51f0160270 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009fd0 tx=0x7f51dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:58.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51ecc00700 1 -- 172.21.15.12:0/2668972887 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0036a0 con 0x7f51f015d930 2024-03-29T11:31:58.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51ecc00700 1 -- 172.21.15.12:0/2668972887 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc01f070 con 0x7f51f015d930 2024-03-29T11:31:58.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51f7600700 1 -- 172.21.15.12:0/2668972887 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f01f6760 con 0x7f51f015d930 2024-03-29T11:31:58.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.047+0000 7f51ecc00700 1 -- 172.21.15.12:0/2668972887 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc003a20 con 0x7f51f015d930 2024-03-29T11:31:58.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.048+0000 7f51f7600700 1 -- 172.21.15.12:0/2668972887 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f01610a0 con 0x7f51f015d930 2024-03-29T11:31:58.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.049+0000 7f51f7600700 1 -- 172.21.15.12:0/2668972887 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f01f6c40 con 0x7f51f015d930 2024-03-29T11:31:58.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.050+0000 7f51ecc00700 1 -- 172.21.15.12:0/2668972887 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 133) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f51dc003b80 con 0x7f51f015d930 2024-03-29T11:31:58.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.051+0000 7f51ecc00700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f51d006af20 0x7f51d006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.051+0000 7f51ecc00700 1 -- 172.21.15.12:0/2668972887 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f51dc021070 con 0x7f51f015d930 2024-03-29T11:31:58.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.051+0000 7f51f4c00700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f51d006af20 0x7f51d006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:58.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.052+0000 7f51f4c00700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f51d006af20 0x7f51d006d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f51e4009be0 tx=0x7f51e4009210 comp rx=0 tx=0).ready entity=mgr.10918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.055+0000 7f51ecc00700 1 -- 172.21.15.12:0/2668972887 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51dc056030 con 0x7f51f015d930 2024-03-29T11:31:58.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.290+0000 7f51f7600700 1 -- 172.21.15.12:0/2668972887 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51f0162030 con 0x7f51d006af20 2024-03-29T11:31:58.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.297+0000 7f51ecc00700 1 -- 172.21.15.12:0/2668972887 <== mgr.10918 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7f51e4011860 con 0x7f51d006af20 2024-03-29T11:31:58.294 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:58.294 INFO:teuthology.orchestra.run.smithi012.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-03-29T11:31:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 -- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f51d006af20 msgr2=0x7f51d006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f51d006af20 0x7f51d006d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f51e4009be0 tx=0x7f51e4009210 comp rx=0 tx=0).stop 2024-03-29T11:31:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 -- 172.21.15.12:0/2668972887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f015d930 msgr2=0x7f51f0160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f015d930 0x7f51f0160270 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009fd0 tx=0x7f51dc004750 comp rx=0 tx=0).stop 2024-03-29T11:31:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 -- 172.21.15.12:0/2668972887 shutdown_connections 2024-03-29T11:31:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f51d006af20 0x7f51d006d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51f015dd10 0x7f51f01f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51f00b4fa0 0x7f51f00b8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 --2- 172.21.15.12:0/2668972887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f015d930 0x7f51f0160270 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 -- 172.21.15.12:0/2668972887 >> 172.21.15.12:0/2668972887 conn(0x7f51f008aff0 msgr2=0x7f51f00d1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 -- 172.21.15.12:0/2668972887 shutdown_connections 2024-03-29T11:31:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.300+0000 7f51f7600700 1 -- 172.21.15.12:0/2668972887 wait complete. 2024-03-29T11:31:58.316 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-03-29T11:31:58.316 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-03-29T11:31:58.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.523+0000 7fd317e00700 1 Processor -- start 2024-03-29T11:31:58.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.523+0000 7fd317e00700 1 -- start start 2024-03-29T11:31:58.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.524+0000 7fd317e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3100ae670 0x7fd310166470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.524+0000 7fd317e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31015dcc0 con 0x7fd31015c700 2024-03-29T11:31:58.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.524+0000 7fd317e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31006dd00 con 0x7fd310155ac0 2024-03-29T11:31:58.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.524+0000 7fd317e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31006d0f0 con 0x7fd3100ae670 2024-03-29T11:31:58.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.524+0000 7fd314a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd31015c700 0x7fd3100adf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37386/0 (socket says 172.21.15.12:37386) 2024-03-29T11:31:58.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.524+0000 7fd314a00700 1 -- 172.21.15.12:0/3396864938 learned_addr learned my addr 172.21.15.12:0/3396864938 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:58.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.525+0000 7fd315e00700 1 --2- 172.21.15.12:0/3396864938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3100ae670 0x7fd310166470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.525+0000 7fd30fe00700 1 -- 172.21.15.12:0/3396864938 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791550384 0 0) 0x7fd31015dcc0 con 0x7fd31015c700 2024-03-29T11:31:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.525+0000 7fd30fe00700 1 -- 172.21.15.12:0/3396864938 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f80035b0 con 0x7fd31015c700 2024-03-29T11:31:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.525+0000 7fd30fe00700 1 -- 172.21.15.12:0/3396864938 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107137335 0 0) 0x7fd31006dd00 con 0x7fd310155ac0 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.525+0000 7fd30fe00700 1 -- 172.21.15.12:0/3396864938 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd31015dcc0 con 0x7fd310155ac0 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.525+0000 7fd30fe00700 1 -- 172.21.15.12:0/3396864938 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1160522335 0 0) 0x7fd2f80035b0 con 0x7fd31015c700 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.525+0000 7fd315e00700 1 -- 172.21.15.12:0/3396864938 >> v1:172.21.15.159:6789/0 conn(0x7fd310155ac0 legacy=0x7fd31006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.525+0000 7fd315e00700 1 -- 172.21.15.12:0/3396864938 >> v1:172.21.15.12:6789/0 conn(0x7fd31015c700 legacy=0x7fd3100adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.525+0000 7fd315e00700 1 -- 172.21.15.12:0/3396864938 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31006d3d0 con 0x7fd3100ae670 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.526+0000 7fd315e00700 1 --2- 172.21.15.12:0/3396864938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3100ae670 0x7fd310166470 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd308009dc0 tx=0x7fd30800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fefa24da352e4e57 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.526+0000 7fd30fe00700 1 -- 172.21.15.12:0/3396864938 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30800e070 con 0x7fd3100ae670 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.526+0000 7fd30fe00700 1 -- 172.21.15.12:0/3396864938 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd308003a20 con 0x7fd3100ae670 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.526+0000 7fd30fe00700 1 -- 172.21.15.12:0/3396864938 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc001670 con 0x7fd3100ae670 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.526+0000 7fd317e00700 1 -- 172.21.15.12:0/3396864938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3100ae670 msgr2=0x7fd310166470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.526+0000 7fd317e00700 1 --2- 172.21.15.12:0/3396864938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3100ae670 0x7fd310166470 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd308009dc0 tx=0x7fd30800b330 comp rx=0 tx=0).stop 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.527+0000 7fd317e00700 1 -- 172.21.15.12:0/3396864938 shutdown_connections 2024-03-29T11:31:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.527+0000 7fd317e00700 1 --2- 172.21.15.12:0/3396864938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3100ae670 0x7fd310166470 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.527+0000 7fd317e00700 1 -- 172.21.15.12:0/3396864938 >> 172.21.15.12:0/3396864938 conn(0x7fd3100cf210 msgr2=0x7fd3100cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.527+0000 7fd317e00700 1 -- 172.21.15.12:0/3396864938 shutdown_connections 2024-03-29T11:31:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.527+0000 7fd317e00700 1 -- 172.21.15.12:0/3396864938 wait complete. 2024-03-29T11:31:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.528+0000 7fd317e00700 1 Processor -- start 2024-03-29T11:31:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.528+0000 7fd317e00700 1 -- start start 2024-03-29T11:31:58.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.529+0000 7fd317e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3100ae670 0x7fd310152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.529+0000 7fd317e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd310155ac0 0x7fd3101561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.529+0000 7fd317e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd31015c700 0x7fd3101566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.529+0000 7fd317e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3101579a0 con 0x7fd3100ae670 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.529+0000 7fd317e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd310157dc0 con 0x7fd31015c700 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.529+0000 7fd317e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd310156d20 con 0x7fd310155ac0 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd315400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3100ae670 0x7fd310152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd315400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3100ae670 0x7fd310152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53840/0 (socket says 172.21.15.12:53840) 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd315400700 1 -- 172.21.15.12:0/1425253988 learned_addr learned my addr 172.21.15.12:0/1425253988 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd315400700 1 -- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd310155ac0 msgr2=0x7fd3101561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd315400700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd310155ac0 0x7fd3101561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd315400700 1 -- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd31015c700 msgr2=0x7fd3101566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd315400700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd31015c700 0x7fd3101566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd315400700 1 -- 172.21.15.12:0/1425253988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd308009a70 con 0x7fd3100ae670 2024-03-29T11:31:58.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd315400700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3100ae670 0x7fd310152060 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fd30400c520 tx=0x7fd30400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:58.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd30d600700 1 -- 172.21.15.12:0/1425253988 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3040052d0 con 0x7fd3100ae670 2024-03-29T11:31:58.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd30d600700 1 -- 172.21.15.12:0/1425253988 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd304005430 con 0x7fd3100ae670 2024-03-29T11:31:58.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.530+0000 7fd30d600700 1 -- 172.21.15.12:0/1425253988 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3040038c0 con 0x7fd3100ae670 2024-03-29T11:31:58.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.531+0000 7fd317e00700 1 -- 172.21.15.12:0/1425253988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd310156020 con 0x7fd3100ae670 2024-03-29T11:31:58.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.531+0000 7fd317e00700 1 -- 172.21.15.12:0/1425253988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3101572e0 con 0x7fd3100ae670 2024-03-29T11:31:58.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.532+0000 7fd317e00700 1 -- 172.21.15.12:0/1425253988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd310067720 con 0x7fd3100ae670 2024-03-29T11:31:58.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.533+0000 7fd30d600700 1 -- 172.21.15.12:0/1425253988 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 133) v1 ==== 89359+0+0 (secure 0 0 0) 0x7fd304003a20 con 0x7fd3100ae670 2024-03-29T11:31:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.534+0000 7fd30d600700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd2f006af20 0x7fd2f006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.534+0000 7fd30d600700 1 -- 172.21.15.12:0/1425253988 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7fd3040891c0 con 0x7fd3100ae670 2024-03-29T11:31:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.534+0000 7fd314a00700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd2f006af20 0x7fd2f006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.535+0000 7fd314a00700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd2f006af20 0x7fd2f006d3e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0011e0 tx=0x7fd2fc001230 comp rx=0 tx=0).ready entity=mgr.10918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:58.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.539+0000 7fd30d600700 1 -- 172.21.15.12:0/1425253988 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd304053c10 con 0x7fd3100ae670 2024-03-29T11:31:58.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.777+0000 7fd317e00700 1 -- 172.21.15.12:0/1425253988 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd310155ea0 con 0x7fd2f006af20 2024-03-29T11:31:58.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.785+0000 7fd30d600700 1 -- 172.21.15.12:0/1425253988 <== mgr.10918 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+353 (secure 0 0 0) 0x7fd2fc004870 con 0x7fd2f006af20 2024-03-29T11:31:58.781 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:31:58.781 INFO:teuthology.orchestra.run.smithi012.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T11:31:56", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-03-29T11:31:58.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.787+0000 7fd317e00700 1 -- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd2f006af20 msgr2=0x7fd2f006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.787+0000 7fd317e00700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd2f006af20 0x7fd2f006d3e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0011e0 tx=0x7fd2fc001230 comp rx=0 tx=0).stop 2024-03-29T11:31:58.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.787+0000 7fd317e00700 1 -- 172.21.15.12:0/1425253988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3100ae670 msgr2=0x7fd310152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:58.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.787+0000 7fd317e00700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3100ae670 0x7fd310152060 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fd30400c520 tx=0x7fd30400c8e0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.787+0000 7fd317e00700 1 -- 172.21.15.12:0/1425253988 shutdown_connections 2024-03-29T11:31:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.787+0000 7fd317e00700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd2f006af20 0x7fd2f006d3e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.787+0000 7fd317e00700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd31015c700 0x7fd3101566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.787+0000 7fd317e00700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3100ae670 0x7fd310152060 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.787+0000 7fd317e00700 1 --2- 172.21.15.12:0/1425253988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd310155ac0 0x7fd3101561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.787+0000 7fd317e00700 1 -- 172.21.15.12:0/1425253988 >> 172.21.15.12:0/1425253988 conn(0x7fd3100cf210 msgr2=0x7fd31015e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.788+0000 7fd317e00700 1 -- 172.21.15.12:0/1425253988 shutdown_connections 2024-03-29T11:31:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:58.788+0000 7fd317e00700 1 -- 172.21.15.12:0/1425253988 wait complete. 2024-03-29T11:31:58.802 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-03-29T00:00:00", "created": "2024-03-29T11:31:56", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-03-29T11:31:58.803 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-29T11:31:59.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.014+0000 7f7da2a00700 1 Processor -- start 2024-03-29T11:31:59.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.014+0000 7f7da2a00700 1 -- start start 2024-03-29T11:31:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.016+0000 7f7da2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce6a0 0x7f7d9c15dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.016+0000 7f7da2a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9c0709c0 con 0x7f7d9c15d800 2024-03-29T11:31:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.016+0000 7f7da2a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9c070c80 con 0x7f7d9c0ce230 2024-03-29T11:31:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.016+0000 7f7da2a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c0b40f0 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.016+0000 7f7d9be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7d9c15d800 0x7f7d9c0725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43110/0 (socket says 172.21.15.12:43110) 2024-03-29T11:31:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.016+0000 7f7d9be00700 1 -- 172.21.15.12:0/2635913232 learned_addr learned my addr 172.21.15.12:0/2635913232 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:31:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.016+0000 7f7da0a00700 1 --2- 172.21.15.12:0/2635913232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce6a0 0x7f7d9c15dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.016+0000 7f7d9aa00700 1 -- 172.21.15.12:0/2635913232 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069304427 0 0) 0x7f7d9c0709c0 con 0x7f7d9c15d800 2024-03-29T11:31:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.016+0000 7f7d9aa00700 1 -- 172.21.15.12:0/2635913232 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d800035b0 con 0x7f7d9c15d800 2024-03-29T11:31:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.016+0000 7f7d9aa00700 1 -- 172.21.15.12:0/2635913232 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329205983 0 0) 0x7f7d9c070c80 con 0x7f7d9c0ce230 2024-03-29T11:31:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.017+0000 7f7d9aa00700 1 -- 172.21.15.12:0/2635913232 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d9c0709c0 con 0x7f7d9c0ce230 2024-03-29T11:31:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.017+0000 7f7da0a00700 1 -- 172.21.15.12:0/2635913232 >> v1:172.21.15.159:6789/0 conn(0x7f7d9c0ce230 legacy=0x7f7d9c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.017+0000 7f7d9aa00700 1 -- 172.21.15.12:0/2635913232 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538398959 0 0) 0x7f7d800035b0 con 0x7f7d9c15d800 2024-03-29T11:31:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.017+0000 7f7da0a00700 1 -- 172.21.15.12:0/2635913232 >> v1:172.21.15.12:6789/0 conn(0x7f7d9c15d800 legacy=0x7f7d9c0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.017+0000 7f7da0a00700 1 -- 172.21.15.12:0/2635913232 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c070f60 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.017+0000 7f7da0a00700 1 --2- 172.21.15.12:0/2635913232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce6a0 0x7f7d9c15dbe0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7d94009dc0 tx=0x7f7d9400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94932f2943b04271 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.017+0000 7f7d9aa00700 1 -- 172.21.15.12:0/2635913232 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9400e070 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.017+0000 7f7d9aa00700 1 -- 172.21.15.12:0/2635913232 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d94003a20 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.017+0000 7f7d9aa00700 1 -- 172.21.15.12:0/2635913232 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c001670 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.018+0000 7f7da2a00700 1 -- 172.21.15.12:0/2635913232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce6a0 msgr2=0x7f7d9c15dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.018+0000 7f7da2a00700 1 --2- 172.21.15.12:0/2635913232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce6a0 0x7f7d9c15dbe0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7d94009dc0 tx=0x7f7d9400b330 comp rx=0 tx=0).stop 2024-03-29T11:31:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.018+0000 7f7da2a00700 1 -- 172.21.15.12:0/2635913232 shutdown_connections 2024-03-29T11:31:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.018+0000 7f7da2a00700 1 --2- 172.21.15.12:0/2635913232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce6a0 0x7f7d9c15dbe0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.018+0000 7f7da2a00700 1 -- 172.21.15.12:0/2635913232 >> 172.21.15.12:0/2635913232 conn(0x7f7d9c0b0d70 msgr2=0x7f7d9c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.018+0000 7f7da2a00700 1 -- 172.21.15.12:0/2635913232 shutdown_connections 2024-03-29T11:31:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.018+0000 7f7da2a00700 1 -- 172.21.15.12:0/2635913232 wait complete. 2024-03-29T11:31:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.020+0000 7f7da2a00700 1 Processor -- start 2024-03-29T11:31:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.020+0000 7f7da2a00700 1 -- start start 2024-03-29T11:31:59.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7da2a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce230 0x7f7d9c0ac960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7da2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d9c0ce6a0 0x7f7d9c1495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7da2a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d9c15d800 0x7f7d9c0ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7da2a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c149b30 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7da2a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c0ad220 con 0x7f7d9c15d800 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7da2a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c0ac590 con 0x7f7d9c0ce230 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7d9b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d9c0ce6a0 0x7f7d9c1495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7d9b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d9c0ce6a0 0x7f7d9c1495f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59928/0 (socket says 172.21.15.12:59928) 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7d9b400700 1 -- 172.21.15.12:0/2303220396 learned_addr learned my addr 172.21.15.12:0/2303220396 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7d9be00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce230 0x7f7d9c0ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.021+0000 7f7da0a00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d9c15d800 0x7f7d9c0ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7d9b400700 1 -- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce230 msgr2=0x7f7d9c0ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7d9b400700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce230 0x7f7d9c0ac960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7d9b400700 1 -- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d9c15d800 msgr2=0x7f7d9c0ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7d9b400700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d9c15d800 0x7f7d9c0ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7d9b400700 1 -- 172.21.15.12:0/2303220396 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d94009a70 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7d9b400700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d9c0ce6a0 0x7f7d9c1495f0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800c500 tx=0x7f7d8800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7d93e00700 1 -- 172.21.15.12:0/2303220396 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d880052d0 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7d93e00700 1 -- 172.21.15.12:0/2303220396 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d88005450 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7d93e00700 1 -- 172.21.15.12:0/2303220396 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d880038c0 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7da2a00700 1 -- 172.21.15.12:0/2303220396 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9c0acf40 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.022+0000 7f7da2a00700 1 -- 172.21.15.12:0/2303220396 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9c0c9260 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.023+0000 7f7da2a00700 1 -- 172.21.15.12:0/2303220396 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9c067720 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.024+0000 7f7d93e00700 1 -- 172.21.15.12:0/2303220396 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 133) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f7d88003a20 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.025+0000 7f7d93e00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7d7c06af50 0x7f7d7c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.025+0000 7f7d93e00700 1 -- 172.21.15.12:0/2303220396 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f7d88088fd0 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.026+0000 7f7d9be00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7d7c06af50 0x7f7d7c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.026+0000 7f7d9be00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7d7c06af50 0x7f7d7c06d410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c001eb0 tx=0x7f7d8c001d60 comp rx=0 tx=0).ready entity=mgr.10918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:59.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.029+0000 7f7d93e00700 1 -- 172.21.15.12:0/2303220396 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7d88053990 con 0x7f7d9c0ce6a0 2024-03-29T11:31:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.269+0000 7f7da2a00700 1 -- 172.21.15.12:0/2303220396 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d9c0c8e00 con 0x7f7d7c06af50 2024-03-29T11:31:59.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.333+0000 7f7d93e00700 1 -- 172.21.15.12:0/2303220396 <== mgr.10918 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f7d8c0015f0 con 0x7f7d7c06af50 2024-03-29T11:31:59.329 INFO:teuthology.orchestra.run.smithi012.stdout:Retention removed from path mandatory_fs_argument_test_dir 2024-03-29T11:31:59.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 -- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7d7c06af50 msgr2=0x7f7d7c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7d7c06af50 0x7f7d7c06d410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c001eb0 tx=0x7f7d8c001d60 comp rx=0 tx=0).stop 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 -- 172.21.15.12:0/2303220396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d9c0ce6a0 msgr2=0x7f7d9c1495f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d9c0ce6a0 0x7f7d9c1495f0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800c500 tx=0x7f7d8800c810 comp rx=0 tx=0).stop 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 -- 172.21.15.12:0/2303220396 shutdown_connections 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7d7c06af50 0x7f7d7c06d410 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d9c15d800 0x7f7d9c0ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d9c0ce230 0x7f7d9c0ac960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 --2- 172.21.15.12:0/2303220396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d9c0ce6a0 0x7f7d9c1495f0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 -- 172.21.15.12:0/2303220396 >> 172.21.15.12:0/2303220396 conn(0x7f7d9c0b0d70 msgr2=0x7f7d9c0ca860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 -- 172.21.15.12:0/2303220396 shutdown_connections 2024-03-29T11:31:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.336+0000 7f7da2a00700 1 -- 172.21.15.12:0/2303220396 wait complete. 2024-03-29T11:31:59.349 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2024-03-29T11:31:59.349 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-29T11:31:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.559+0000 7f9ed7e00700 1 Processor -- start 2024-03-29T11:31:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.559+0000 7f9ed7e00700 1 -- start start 2024-03-29T11:31:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.560+0000 7f9ed7e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9ed015d930 0x7f9ed00d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.560+0000 7f9ed7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ed0162030 0x7f9ed00d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.560+0000 7f9ed7e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed00d2520 con 0x7f9ed00d20d0 2024-03-29T11:31:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.560+0000 7f9ed7e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed015ec40 con 0x7f9ed0162030 2024-03-29T11:31:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.560+0000 7f9ed7e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed015dd10 con 0x7f9ed015d930 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.561+0000 7f9ed4a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ed0162030 0x7f9ed00d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.561+0000 7f9ed4a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ed0162030 0x7f9ed00d1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59930/0 (socket says 172.21.15.12:59930) 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.561+0000 7f9ed4a00700 1 -- 172.21.15.12:0/3457254258 learned_addr learned my addr 172.21.15.12:0/3457254258 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.561+0000 7f9ed5400700 1 --2- 172.21.15.12:0/3457254258 >> v2:172.21.15.159:3300/0 conn(0x7f9ed015d930 0x7f9ed00d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.561+0000 7f9ed4a00700 1 -- 172.21.15.12:0/3457254258 >> v2:172.21.15.159:3300/0 conn(0x7f9ed015d930 msgr2=0x7f9ed00d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.561+0000 7f9ed4a00700 1 --2- 172.21.15.12:0/3457254258 >> v2:172.21.15.159:3300/0 conn(0x7f9ed015d930 0x7f9ed00d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.561+0000 7f9ed4a00700 1 -- 172.21.15.12:0/3457254258 >> v1:172.21.15.159:6789/0 conn(0x7f9ed00d20d0 legacy=0x7f9ed00cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.561+0000 7f9ed4a00700 1 -- 172.21.15.12:0/3457254258 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed015dfc0 con 0x7f9ed0162030 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.561+0000 7f9ed4a00700 1 --2- 172.21.15.12:0/3457254258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 0x7f9ed00d1b90 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00a1430 tx=0x7f9ebc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7027909aab153a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ecfe00700 1 -- 172.21.15.12:0/3457254258 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc011830 con 0x7f9ed0162030 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ecfe00700 1 -- 172.21.15.12:0/3457254258 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ebc011990 con 0x7f9ed0162030 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ecfe00700 1 -- 172.21.15.12:0/3457254258 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc011d20 con 0x7f9ed0162030 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ed7e00700 1 -- 172.21.15.12:0/3457254258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 msgr2=0x7f9ed00d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ed7e00700 1 --2- 172.21.15.12:0/3457254258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 0x7f9ed00d1b90 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00a1430 tx=0x7f9ebc009bd0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ed7e00700 1 -- 172.21.15.12:0/3457254258 shutdown_connections 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ed7e00700 1 --2- 172.21.15.12:0/3457254258 >> v2:172.21.15.159:3300/0 conn(0x7f9ed015d930 0x7f9ed00d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ed7e00700 1 --2- 172.21.15.12:0/3457254258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 0x7f9ed00d1b90 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ed7e00700 1 -- 172.21.15.12:0/3457254258 >> 172.21.15.12:0/3457254258 conn(0x7f9ed008aff0 msgr2=0x7f9ed00d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ed7e00700 1 -- 172.21.15.12:0/3457254258 shutdown_connections 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.562+0000 7f9ed7e00700 1 -- 172.21.15.12:0/3457254258 wait complete. 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.564+0000 7f9ed7e00700 1 Processor -- start 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.564+0000 7f9ed7e00700 1 -- start start 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.565+0000 7f9ed7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d20d0 0x7f9ed00cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.565+0000 7f9ed7e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015d930 0x7f9ed015ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.565+0000 7f9ed7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 0x7f9ed0160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.565+0000 7f9ed7e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00cabf0 con 0x7f9ed0162030 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.565+0000 7f9ed7e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00caed0 con 0x7f9ed015d930 2024-03-29T11:31:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.565+0000 7f9ed7e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00cb1b0 con 0x7f9ed00d20d0 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.565+0000 7f9ed5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 0x7f9ed0160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.565+0000 7f9ed5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 0x7f9ed0160190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59938/0 (socket says 172.21.15.12:59938) 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed5e00700 1 -- 172.21.15.12:0/2608294957 learned_addr learned my addr 172.21.15.12:0/2608294957 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed4a00700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015d930 0x7f9ed015ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed5400700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d20d0 0x7f9ed00cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed5e00700 1 -- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d20d0 msgr2=0x7f9ed00cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed5e00700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d20d0 0x7f9ed00cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed5e00700 1 -- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015d930 msgr2=0x7f9ed015ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed5e00700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015d930 0x7f9ed015ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed5e00700 1 -- 172.21.15.12:0/2608294957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc009550 con 0x7f9ed0162030 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed5400700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d20d0 0x7f9ed00cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed5e00700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 0x7f9ed0160190 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f9ec800cb50 tx=0x7f9ec800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ecd600700 1 -- 172.21.15.12:0/2608294957 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8003c00 con 0x7f9ed0162030 2024-03-29T11:31:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ecd600700 1 -- 172.21.15.12:0/2608294957 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec8003d60 con 0x7f9ed0162030 2024-03-29T11:31:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ecd600700 1 -- 172.21.15.12:0/2608294957 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec800aac0 con 0x7f9ed0162030 2024-03-29T11:31:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.566+0000 7f9ed7e00700 1 -- 172.21.15.12:0/2608294957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed00cb490 con 0x7f9ed0162030 2024-03-29T11:31:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.567+0000 7f9ed7e00700 1 -- 172.21.15.12:0/2608294957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed00cd200 con 0x7f9ed0162030 2024-03-29T11:31:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.568+0000 7f9ed7e00700 1 -- 172.21.15.12:0/2608294957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed00cb9b0 con 0x7f9ed0162030 2024-03-29T11:31:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.568+0000 7f9ecd600700 1 -- 172.21.15.12:0/2608294957 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 133) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f9ec800ac20 con 0x7f9ed0162030 2024-03-29T11:31:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.569+0000 7f9ecd600700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9eb006af20 0x7f9eb006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:31:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.569+0000 7f9ecd600700 1 -- 172.21.15.12:0/2608294957 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f9ec8089ae0 con 0x7f9ed0162030 2024-03-29T11:31:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.570+0000 7f9ed5400700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9eb006af20 0x7f9eb006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:31:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.570+0000 7f9ed5400700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9eb006af20 0x7f9eb006d3e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009c30 tx=0x7f9ec4008040 comp rx=0 tx=0).ready entity=mgr.10918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:31:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.574+0000 7f9ecd600700 1 -- 172.21.15.12:0/2608294957 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ed00cb9b0 con 0x7f9ed0162030 2024-03-29T11:31:59.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.818+0000 7f9ed7e00700 1 -- 172.21.15.12:0/2608294957 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ed015e580 con 0x7f9eb006af20 2024-03-29T11:31:59.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.881+0000 7f9ecd600700 1 -- 172.21.15.12:0/2608294957 <== mgr.10918 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7f9ec4009960 con 0x7f9eb006af20 2024-03-29T11:31:59.877 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2024-03-29T11:31:59.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.883+0000 7f9ed7e00700 1 -- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9eb006af20 msgr2=0x7f9eb006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.883+0000 7f9ed7e00700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9eb006af20 0x7f9eb006d3e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009c30 tx=0x7f9ec4008040 comp rx=0 tx=0).stop 2024-03-29T11:31:59.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.883+0000 7f9ed7e00700 1 -- 172.21.15.12:0/2608294957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 msgr2=0x7f9ed0160190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:59.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.883+0000 7f9ed7e00700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 0x7f9ed0160190 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f9ec800cb50 tx=0x7f9ec800ce60 comp rx=0 tx=0).stop 2024-03-29T11:31:59.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.883+0000 7f9ed7e00700 1 -- 172.21.15.12:0/2608294957 shutdown_connections 2024-03-29T11:31:59.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.883+0000 7f9ed7e00700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9eb006af20 0x7f9eb006d3e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.883+0000 7f9ed7e00700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ed0162030 0x7f9ed0160190 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.883+0000 7f9ed7e00700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ed00d20d0 0x7f9ed00cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.883+0000 7f9ed7e00700 1 --2- 172.21.15.12:0/2608294957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ed015d930 0x7f9ed015ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:31:59.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.883+0000 7f9ed7e00700 1 -- 172.21.15.12:0/2608294957 >> 172.21.15.12:0/2608294957 conn(0x7f9ed008aff0 msgr2=0x7f9ed015c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:59.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.884+0000 7f9ed7e00700 1 -- 172.21.15.12:0/2608294957 shutdown_connections 2024-03-29T11:31:59.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:31:59.884+0000 7f9ed7e00700 1 -- 172.21.15.12:0/2608294957 wait complete. 2024-03-29T11:31:59.897 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2024-03-29T11:31:59.898 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-29T11:32:00.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.106+0000 7f4351a00700 1 Processor -- start 2024-03-29T11:32:00.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.106+0000 7f4351a00700 1 -- start start 2024-03-29T11:32:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.107+0000 7f4351a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d0bd0 0x7f434c15e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.107+0000 7f4351a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f434c0b3120 0x7f434c06a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.107+0000 7f4351a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f434c06acd0 0x7f434c06dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.107+0000 7f4351a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434c06b140 con 0x7f434c0b3120 2024-03-29T11:32:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.107+0000 7f4351a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434c15ee10 con 0x7f434c06acd0 2024-03-29T11:32:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.107+0000 7f4351a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c15de10 con 0x7f434c0d0bd0 2024-03-29T11:32:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.107+0000 7f434a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f434c0b3120 0x7f434c06a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.107+0000 7f434a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f434c0b3120 0x7f434c06a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59952/0 (socket says 172.21.15.12:59952) 2024-03-29T11:32:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.107+0000 7f434a000700 1 -- 172.21.15.12:0/3994253980 learned_addr learned my addr 172.21.15.12:0/3994253980 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f434b400700 1 --2- 172.21.15.12:0/3994253980 >> v2:172.21.15.159:3300/0 conn(0x7f434c06acd0 0x7f434c06dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f434aa00700 1 --2- 172.21.15.12:0/3994253980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d0bd0 0x7f434c15e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f434a000700 1 -- 172.21.15.12:0/3994253980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d0bd0 msgr2=0x7f434c15e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f434a000700 1 --2- 172.21.15.12:0/3994253980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d0bd0 0x7f434c15e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f434a000700 1 -- 172.21.15.12:0/3994253980 >> v2:172.21.15.159:3300/0 conn(0x7f434c06acd0 msgr2=0x7f434c06dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f434a000700 1 --2- 172.21.15.12:0/3994253980 >> v2:172.21.15.159:3300/0 conn(0x7f434c06acd0 0x7f434c06dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f434a000700 1 -- 172.21.15.12:0/3994253980 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c0b4fd0 con 0x7f434c0b3120 2024-03-29T11:32:00.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f434b400700 1 --2- 172.21.15.12:0/3994253980 >> v2:172.21.15.159:3300/0 conn(0x7f434c06acd0 0x7f434c06dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:00.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f434a000700 1 --2- 172.21.15.12:0/3994253980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c0b3120 0x7f434c06a790 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f434c0b2ea0 tx=0x7f4338009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f664d3e4d1bae34 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:00.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f4349600700 1 -- 172.21.15.12:0/3994253980 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4338011830 con 0x7f434c0b3120 2024-03-29T11:32:00.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f4349600700 1 -- 172.21.15.12:0/3994253980 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4338011990 con 0x7f434c0b3120 2024-03-29T11:32:00.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.108+0000 7f4349600700 1 -- 172.21.15.12:0/3994253980 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4338011d20 con 0x7f434c0b3120 2024-03-29T11:32:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.109+0000 7f4351a00700 1 -- 172.21.15.12:0/3994253980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c0b3120 msgr2=0x7f434c06a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.109+0000 7f4351a00700 1 --2- 172.21.15.12:0/3994253980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c0b3120 0x7f434c06a790 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f434c0b2ea0 tx=0x7f4338009bd0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.109+0000 7f4351a00700 1 -- 172.21.15.12:0/3994253980 shutdown_connections 2024-03-29T11:32:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.109+0000 7f4351a00700 1 --2- 172.21.15.12:0/3994253980 >> v2:172.21.15.159:3300/0 conn(0x7f434c06acd0 0x7f434c06dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.109+0000 7f4351a00700 1 --2- 172.21.15.12:0/3994253980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d0bd0 0x7f434c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.109+0000 7f4351a00700 1 --2- 172.21.15.12:0/3994253980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c0b3120 0x7f434c06a790 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.109+0000 7f4351a00700 1 -- 172.21.15.12:0/3994253980 >> 172.21.15.12:0/3994253980 conn(0x7f434c0b5640 msgr2=0x7f434c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.109+0000 7f4351a00700 1 -- 172.21.15.12:0/3994253980 shutdown_connections 2024-03-29T11:32:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.109+0000 7f4351a00700 1 -- 172.21.15.12:0/3994253980 wait complete. 2024-03-29T11:32:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.111+0000 7f4351a00700 1 Processor -- start 2024-03-29T11:32:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.111+0000 7f4351a00700 1 -- start start 2024-03-29T11:32:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f4351a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c06acd0 0x7f434c09cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f4351a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f434c0b3120 0x7f434c15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f4351a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d0bd0 0x7f434c1f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f4351a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c09a760 con 0x7f434c06acd0 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f4351a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c09b290 con 0x7f434c0b3120 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f4351a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c15a700 con 0x7f434c0d0bd0 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f434aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c06acd0 0x7f434c09cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f434aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c06acd0 0x7f434c09cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59954/0 (socket says 172.21.15.12:59954) 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f434aa00700 1 -- 172.21.15.12:0/3144501297 learned_addr learned my addr 172.21.15.12:0/3144501297 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f434aa00700 1 -- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d0bd0 msgr2=0x7f434c1f6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f434aa00700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d0bd0 0x7f434c1f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f434aa00700 1 -- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f434c0b3120 msgr2=0x7f434c15a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f434aa00700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f434c0b3120 0x7f434c15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.112+0000 7f434aa00700 1 -- 172.21.15.12:0/3144501297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4338009550 con 0x7f434c06acd0 2024-03-29T11:32:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.113+0000 7f434aa00700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c06acd0 0x7f434c09cc50 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f433c00e9b0 tx=0x7f433c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.113+0000 7f4342a00700 1 -- 172.21.15.12:0/3144501297 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c009710 con 0x7f434c06acd0 2024-03-29T11:32:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.113+0000 7f4342a00700 1 -- 172.21.15.12:0/3144501297 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c00c850 con 0x7f434c06acd0 2024-03-29T11:32:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.113+0000 7f4351a00700 1 -- 172.21.15.12:0/3144501297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c09a960 con 0x7f434c06acd0 2024-03-29T11:32:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.113+0000 7f4342a00700 1 -- 172.21.15.12:0/3144501297 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c00cbc0 con 0x7f434c06acd0 2024-03-29T11:32:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.113+0000 7f4351a00700 1 -- 172.21.15.12:0/3144501297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c09af20 con 0x7f434c06acd0 2024-03-29T11:32:00.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.114+0000 7f4351a00700 1 -- 172.21.15.12:0/3144501297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434c0b3970 con 0x7f434c06acd0 2024-03-29T11:32:00.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.115+0000 7f4342a00700 1 -- 172.21.15.12:0/3144501297 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 133) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f433c00cd20 con 0x7f434c06acd0 2024-03-29T11:32:00.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.116+0000 7f4342a00700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f432c06af20 0x7f432c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.116+0000 7f4342a00700 1 -- 172.21.15.12:0/3144501297 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f433c089f20 con 0x7f434c06acd0 2024-03-29T11:32:00.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.117+0000 7f434a000700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f432c06af20 0x7f432c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.117+0000 7f434a000700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f432c06af20 0x7f432c06d3e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f434c1f68c0 tx=0x7f4338010000 comp rx=0 tx=0).ready entity=mgr.10918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.120+0000 7f4342a00700 1 -- 172.21.15.12:0/3144501297 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f433c054770 con 0x7f434c06acd0 2024-03-29T11:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.308+0000 7f4342a00700 1 -- 172.21.15.12:0/3144501297 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 134) v1 ==== 89415+0+0 (secure 0 0 0) 0x7f433c08d030 con 0x7f434c06acd0 2024-03-29T11:32:00.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.364+0000 7f4351a00700 1 -- 172.21.15.12:0/3144501297 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f434c15e3b0 con 0x7f432c06af20 2024-03-29T11:32:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.434+0000 7f4342a00700 1 -- 172.21.15.12:0/3144501297 <== mgr.10918 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f433801f070 con 0x7f432c06af20 2024-03-29T11:32:00.430 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 -- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f432c06af20 msgr2=0x7f432c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f432c06af20 0x7f432c06d3e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f434c1f68c0 tx=0x7f4338010000 comp rx=0 tx=0).stop 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 -- 172.21.15.12:0/3144501297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c06acd0 msgr2=0x7f434c09cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c06acd0 0x7f434c09cc50 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f433c00e9b0 tx=0x7f433c00ed70 comp rx=0 tx=0).stop 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 -- 172.21.15.12:0/3144501297 shutdown_connections 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f432c06af20 0x7f432c06d3e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d0bd0 0x7f434c1f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f434c06acd0 0x7f434c09cc50 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 --2- 172.21.15.12:0/3144501297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f434c0b3120 0x7f434c15a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 -- 172.21.15.12:0/3144501297 >> 172.21.15.12:0/3144501297 conn(0x7f434c0b5640 msgr2=0x7f434c160290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 -- 172.21.15.12:0/3144501297 shutdown_connections 2024-03-29T11:32:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.437+0000 7f4351a00700 1 -- 172.21.15.12:0/3144501297 wait complete. 2024-03-29T11:32:00.451 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2024-03-29T11:32:00.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:00.451 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-03-29T11:32:00.499 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T11:32:00.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.705+0000 7fd8c4600700 1 Processor -- start 2024-03-29T11:32:00.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.706+0000 7fd8c4600700 1 -- start start 2024-03-29T11:32:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.707+0000 7fd8c4600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd8bc0d1b70 0x7fd8bc0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.707+0000 7fd8c4600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc0cff10 0x7fd8bc0d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.707+0000 7fd8c4600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc0cf640 con 0x7fd8bc15d710 2024-03-29T11:32:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.707+0000 7fd8c4600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8bc0cf220 con 0x7fd8bc0d1b70 2024-03-29T11:32:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.707+0000 7fd8c4600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc0d2560 con 0x7fd8bc0cff10 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.707+0000 7fd8c2600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc0cff10 0x7fd8bc0d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.707+0000 7fd8c1c00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fd8bc15d710 0x7fd8bc0d12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:59078/0 (socket says 172.21.15.12:59078) 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.707+0000 7fd8c1c00700 1 -- 172.21.15.12:0/3623440241 learned_addr learned my addr 172.21.15.12:0/3623440241 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.707+0000 7fd8c1200700 1 --2- 172.21.15.12:0/3623440241 >> v2:172.21.15.159:3300/0 conn(0x7fd8bc0d1b70 0x7fd8bc0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.708+0000 7fd8bbe00700 1 -- 172.21.15.12:0/3623440241 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759330657 0 0) 0x7fd8bc0cf640 con 0x7fd8bc15d710 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.708+0000 7fd8bbe00700 1 -- 172.21.15.12:0/3623440241 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a40035b0 con 0x7fd8bc15d710 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.708+0000 7fd8c2600700 1 -- 172.21.15.12:0/3623440241 >> v2:172.21.15.159:3300/0 conn(0x7fd8bc0d1b70 msgr2=0x7fd8bc0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.708+0000 7fd8c2600700 1 --2- 172.21.15.12:0/3623440241 >> v2:172.21.15.159:3300/0 conn(0x7fd8bc0d1b70 0x7fd8bc0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.708+0000 7fd8c2600700 1 -- 172.21.15.12:0/3623440241 >> v1:172.21.15.159:6789/0 conn(0x7fd8bc15d710 legacy=0x7fd8bc0d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.708+0000 7fd8c2600700 1 -- 172.21.15.12:0/3623440241 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0ce5a0 con 0x7fd8bc0cff10 2024-03-29T11:32:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.708+0000 7fd8c1200700 1 --2- 172.21.15.12:0/3623440241 >> v2:172.21.15.159:3300/0 conn(0x7fd8bc0d1b70 0x7fd8bc0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.708+0000 7fd8c2600700 1 --2- 172.21.15.12:0/3623440241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc0cff10 0x7fd8bc0d0330 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009d90 tx=0x7fd8b400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6374d04f865e93be server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8bbe00700 1 -- 172.21.15.12:0/3623440241 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b400e070 con 0x7fd8bc0cff10 2024-03-29T11:32:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8bbe00700 1 -- 172.21.15.12:0/3623440241 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b400bb20 con 0x7fd8bc0cff10 2024-03-29T11:32:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8bbe00700 1 -- 172.21.15.12:0/3623440241 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4012430 con 0x7fd8bc0cff10 2024-03-29T11:32:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8c4600700 1 -- 172.21.15.12:0/3623440241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc0cff10 msgr2=0x7fd8bc0d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8c4600700 1 --2- 172.21.15.12:0/3623440241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc0cff10 0x7fd8bc0d0330 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009d90 tx=0x7fd8b400b330 comp rx=0 tx=0).stop 2024-03-29T11:32:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8c4600700 1 -- 172.21.15.12:0/3623440241 shutdown_connections 2024-03-29T11:32:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8c4600700 1 --2- 172.21.15.12:0/3623440241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc0cff10 0x7fd8bc0d0330 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8c4600700 1 --2- 172.21.15.12:0/3623440241 >> v2:172.21.15.159:3300/0 conn(0x7fd8bc0d1b70 0x7fd8bc0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8c4600700 1 -- 172.21.15.12:0/3623440241 >> 172.21.15.12:0/3623440241 conn(0x7fd8bc08aff0 msgr2=0x7fd8bc0d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:00.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8c4600700 1 -- 172.21.15.12:0/3623440241 shutdown_connections 2024-03-29T11:32:00.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.709+0000 7fd8c4600700 1 -- 172.21.15.12:0/3623440241 wait complete. 2024-03-29T11:32:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.711+0000 7fd8c4600700 1 Processor -- start 2024-03-29T11:32:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.711+0000 7fd8c4600700 1 -- start start 2024-03-29T11:32:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.712+0000 7fd8c4600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8bc0cff10 0x7fd8bc0b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.712+0000 7fd8c4600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8bc0d1b70 0x7fd8bc0b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.712+0000 7fd8c4600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc15d710 0x7fd8bc0b5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.712+0000 7fd8c4600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc0b4bb0 con 0x7fd8bc0cff10 2024-03-29T11:32:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.712+0000 7fd8c4600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc0b6b20 con 0x7fd8bc0d1b70 2024-03-29T11:32:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.712+0000 7fd8c4600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc0b5e70 con 0x7fd8bc15d710 2024-03-29T11:32:00.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.712+0000 7fd8c1c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8bc0cff10 0x7fd8bc0b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8c1c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8bc0cff10 0x7fd8bc0b39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59966/0 (socket says 172.21.15.12:59966) 2024-03-29T11:32:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8c1c00700 1 -- 172.21.15.12:0/1048912360 learned_addr learned my addr 172.21.15.12:0/1048912360 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8c1200700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8bc0d1b70 0x7fd8bc0b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8c2600700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc15d710 0x7fd8bc0b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8c1c00700 1 -- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc15d710 msgr2=0x7fd8bc0b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8c1c00700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc15d710 0x7fd8bc0b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8c1c00700 1 -- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8bc0d1b70 msgr2=0x7fd8bc0b8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8c1c00700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8bc0d1b70 0x7fd8bc0b8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8c1c00700 1 -- 172.21.15.12:0/1048912360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a8009430 con 0x7fd8bc0cff10 2024-03-29T11:32:00.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8c1c00700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8bc0cff10 0x7fd8bc0b39e0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0001eb0 tx=0x7fd8b000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:00.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8b9600700 1 -- 172.21.15.12:0/1048912360 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0005d70 con 0x7fd8bc0cff10 2024-03-29T11:32:00.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8b9600700 1 -- 172.21.15.12:0/1048912360 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b00070d0 con 0x7fd8bc0cff10 2024-03-29T11:32:00.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.713+0000 7fd8b9600700 1 -- 172.21.15.12:0/1048912360 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0007460 con 0x7fd8bc0cff10 2024-03-29T11:32:00.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.714+0000 7fd8c4600700 1 -- 172.21.15.12:0/1048912360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b4009a70 con 0x7fd8bc0cff10 2024-03-29T11:32:00.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.714+0000 7fd8c4600700 1 -- 172.21.15.12:0/1048912360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc0b4310 con 0x7fd8bc0cff10 2024-03-29T11:32:00.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.715+0000 7fd8c4600700 1 -- 172.21.15.12:0/1048912360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc0b6cf0 con 0x7fd8bc0cff10 2024-03-29T11:32:00.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.716+0000 7fd8b9600700 1 -- 172.21.15.12:0/1048912360 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 134) v1 ==== 89415+0+0 (secure 0 0 0) 0x7fd8b00075c0 con 0x7fd8bc0cff10 2024-03-29T11:32:00.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.716+0000 7fd8b9600700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd89c06afb0 0x7fd89c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:00.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.717+0000 7fd8b9600700 1 -- 172.21.15.12:0/1048912360 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7fd8b0089220 con 0x7fd8bc0cff10 2024-03-29T11:32:00.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.717+0000 7fd8c1200700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd89c06afb0 0x7fd89c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:00.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.718+0000 7fd8c1200700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd89c06afb0 0x7fd89c06d470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0d1270 tx=0x7fd8a8008040 comp rx=0 tx=0).ready entity=mgr.10918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:00.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.721+0000 7fd8b9600700 1 -- 172.21.15.12:0/1048912360 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8b0053bb0 con 0x7fd8bc0cff10 2024-03-29T11:32:00.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:00.996+0000 7fd8c4600700 1 -- 172.21.15.12:0/1048912360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd8bc067e30 con 0x7fd8bc0cff10 2024-03-29T11:32:01.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.708+0000 7fd8b9600700 1 -- 172.21.15.12:0/1048912360 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fd8b0053f80 con 0x7fd8bc0cff10 2024-03-29T11:32:01.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.709+0000 7fd8b9600700 1 -- 172.21.15.12:0/1048912360 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v135) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd8b0071ed0 con 0x7fd8bc0cff10 2024-03-29T11:32:01.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 -- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd89c06afb0 msgr2=0x7fd89c06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:01.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd89c06afb0 0x7fd89c06d470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0d1270 tx=0x7fd8a8008040 comp rx=0 tx=0).stop 2024-03-29T11:32:01.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 -- 172.21.15.12:0/1048912360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8bc0cff10 msgr2=0x7fd8bc0b39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:01.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8bc0cff10 0x7fd8bc0b39e0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0001eb0 tx=0x7fd8b000c860 comp rx=0 tx=0).stop 2024-03-29T11:32:01.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 -- 172.21.15.12:0/1048912360 shutdown_connections 2024-03-29T11:32:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd89c06afb0 0x7fd89c06d470 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8bc15d710 0x7fd8bc0b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8bc0cff10 0x7fd8bc0b39e0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 --2- 172.21.15.12:0/1048912360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8bc0d1b70 0x7fd8bc0b8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 -- 172.21.15.12:0/1048912360 >> 172.21.15.12:0/1048912360 conn(0x7fd8bc08aff0 msgr2=0x7fd8bc0b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 -- 172.21.15.12:0/1048912360 shutdown_connections 2024-03-29T11:32:01.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:01.712+0000 7fd8c4600700 1 -- 172.21.15.12:0/1048912360 wait complete. 2024-03-29T11:32:01.729 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:32:01.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:32:01.826 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:32:01.912 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:32:01.912 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:32:01.912 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:32:01.938 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:01.943+0000 7fe61ff2c240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:32:01.939 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:01.943+0000 7f2910d5b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:32:01.958 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:32:01.958 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:32:02.049 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:02.125 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:02.131+0000 7fe61ff2c240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:32:02.153 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:02.157+0000 7f2910d5b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:32:02.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.290+0000 7f54abe00700 1 Processor -- start 2024-03-29T11:32:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.290+0000 7f54abe00700 1 -- start start 2024-03-29T11:32:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.292+0000 7f54abe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f54a4099a60 0x7f54a4099e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.292+0000 7f54abe00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a409a3d0 con 0x7f54a409aa70 2024-03-29T11:32:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.292+0000 7f54abe00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a40cefc0 con 0x7f54a415b560 2024-03-29T11:32:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.292+0000 7f54abe00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a40cf230 con 0x7f54a4099a60 2024-03-29T11:32:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.292+0000 7f54a8a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f54a409aa70 0x7f54a40cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43134/0 (socket says 172.21.15.12:43134) 2024-03-29T11:32:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.292+0000 7f54a8a00700 1 -- 172.21.15.12:0/2761602599 learned_addr learned my addr 172.21.15.12:0/2761602599 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.292+0000 7f54a9e00700 1 --2- 172.21.15.12:0/2761602599 >> v2:172.21.15.159:3300/0 conn(0x7f54a4099a60 0x7f54a4099e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.292+0000 7f54a3e00700 1 -- 172.21.15.12:0/2761602599 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203609584 0 0) 0x7f54a409a3d0 con 0x7f54a409aa70 2024-03-29T11:32:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.292+0000 7f54a3e00700 1 -- 172.21.15.12:0/2761602599 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f548c0035b0 con 0x7f54a409aa70 2024-03-29T11:32:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.293+0000 7f54a3e00700 1 -- 172.21.15.12:0/2761602599 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247129528 0 0) 0x7f54a40cefc0 con 0x7f54a415b560 2024-03-29T11:32:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.293+0000 7f54a9e00700 1 -- 172.21.15.12:0/2761602599 >> v1:172.21.15.159:6789/0 conn(0x7f54a415b560 legacy=0x7f54a40cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.293+0000 7f54a9e00700 1 -- 172.21.15.12:0/2761602599 >> v1:172.21.15.12:6789/0 conn(0x7f54a409aa70 legacy=0x7f54a40cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.293+0000 7f54a9e00700 1 -- 172.21.15.12:0/2761602599 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a40cf510 con 0x7f54a4099a60 2024-03-29T11:32:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.293+0000 7f54a9e00700 1 --2- 172.21.15.12:0/2761602599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a4099a60 0x7f54a4099e90 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f549c009d90 tx=0x7f549c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69c4f73bdd74e015 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.293+0000 7f54a3e00700 1 -- 172.21.15.12:0/2761602599 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c00e070 con 0x7f54a4099a60 2024-03-29T11:32:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.293+0000 7f54a3e00700 1 -- 172.21.15.12:0/2761602599 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549c00bb20 con 0x7f54a4099a60 2024-03-29T11:32:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.293+0000 7f54a3e00700 1 -- 172.21.15.12:0/2761602599 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c00be90 con 0x7f54a4099a60 2024-03-29T11:32:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.294+0000 7f54abe00700 1 -- 172.21.15.12:0/2761602599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a4099a60 msgr2=0x7f54a4099e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.294+0000 7f54abe00700 1 --2- 172.21.15.12:0/2761602599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a4099a60 0x7f54a4099e90 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f549c009d90 tx=0x7f549c00b330 comp rx=0 tx=0).stop 2024-03-29T11:32:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.294+0000 7f54abe00700 1 -- 172.21.15.12:0/2761602599 shutdown_connections 2024-03-29T11:32:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.294+0000 7f54abe00700 1 --2- 172.21.15.12:0/2761602599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a4099a60 0x7f54a4099e90 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.294+0000 7f54abe00700 1 -- 172.21.15.12:0/2761602599 >> 172.21.15.12:0/2761602599 conn(0x7f54a40d1350 msgr2=0x7f54a40d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.294+0000 7f54abe00700 1 -- 172.21.15.12:0/2761602599 shutdown_connections 2024-03-29T11:32:02.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.294+0000 7f54abe00700 1 -- 172.21.15.12:0/2761602599 wait complete. 2024-03-29T11:32:02.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.296+0000 7f54abe00700 1 Processor -- start 2024-03-29T11:32:02.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.296+0000 7f54abe00700 1 -- start start 2024-03-29T11:32:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54abe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a4099a60 0x7f54a40caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54abe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a409aa70 0x7f54a406fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54abe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a415b560 0x7f54a40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54abe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a40b6360 con 0x7f54a415b560 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54abe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a40b5750 con 0x7f54a4099a60 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54abe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a40b5a30 con 0x7f54a409aa70 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54a9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a415b560 0x7f54a40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54a9400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a4099a60 0x7f54a40caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54a8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a409aa70 0x7f54a406fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54a9400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a4099a60 0x7f54a40caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:48816/0 (socket says 172.21.15.12:48816) 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54a9400700 1 -- 172.21.15.12:0/2845888668 learned_addr learned my addr 172.21.15.12:0/2845888668 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.297+0000 7f54a8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a409aa70 0x7f54a406fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:44394/0 (socket says 172.21.15.12:44394) 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.298+0000 7f54a9e00700 1 -- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a409aa70 msgr2=0x7f54a406fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.298+0000 7f54a9e00700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a409aa70 0x7f54a406fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.298+0000 7f54a9e00700 1 -- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a4099a60 msgr2=0x7f54a40caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.298+0000 7f54a9e00700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a4099a60 0x7f54a40caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.298+0000 7f54a9e00700 1 -- 172.21.15.12:0/2845888668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549c009a70 con 0x7f54a415b560 2024-03-29T11:32:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.298+0000 7f54a9e00700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a415b560 0x7f54a40702c0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f549c016000 tx=0x7f549c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.298+0000 7f54a1600700 1 -- 172.21.15.12:0/2845888668 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c003c20 con 0x7f54a415b560 2024-03-29T11:32:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.298+0000 7f54a1600700 1 -- 172.21.15.12:0/2845888668 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549c003da0 con 0x7f54a415b560 2024-03-29T11:32:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.298+0000 7f54a1600700 1 -- 172.21.15.12:0/2845888668 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c012ae0 con 0x7f54a415b560 2024-03-29T11:32:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.298+0000 7f54abe00700 1 -- 172.21.15.12:0/2845888668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a40b5d10 con 0x7f54a415b560 2024-03-29T11:32:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.299+0000 7f54abe00700 1 -- 172.21.15.12:0/2845888668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a40caff0 con 0x7f54a415b560 2024-03-29T11:32:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.300+0000 7f54abe00700 1 -- 172.21.15.12:0/2845888668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a40b64d0 con 0x7f54a415b560 2024-03-29T11:32:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.300+0000 7f54a1600700 1 -- 172.21.15.12:0/2845888668 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f549c023070 con 0x7f54a415b560 2024-03-29T11:32:02.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.301+0000 7f54a1600700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f548406af60 0x7f548406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.301+0000 7f54a1600700 1 -- 172.21.15.12:0/2845888668 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f549c08ac30 con 0x7f54a415b560 2024-03-29T11:32:02.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.301+0000 7f54a9400700 1 -- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f548406af60 msgr2=0x7f548406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:02.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.302+0000 7f54a9400700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f548406af60 0x7f548406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:02.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.306+0000 7f54a1600700 1 -- 172.21.15.12:0/2845888668 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f549c055500 con 0x7f54a415b560 2024-03-29T11:32:02.447 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:02.453+0000 7fe61ff2c240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:32:02.490 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:02.494+0000 7f2910d5b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:32:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.503+0000 7f54a9400700 1 -- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f548406af60 msgr2=0x7f548406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.503+0000 7f54a9400700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f548406af60 0x7f548406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.520+0000 7f54abe00700 1 -- 172.21.15.12:0/2845888668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54a40b66f0 con 0x7f54a415b560 2024-03-29T11:32:02.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.522+0000 7f54a1600700 1 -- 172.21.15.12:0/2845888668 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+20451 (secure 0 0 0) 0x7f549c058b20 con 0x7f54a415b560 2024-03-29T11:32:02.518 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:02.518 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":232,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:55.626816+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:31:39.075407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:31:39.845704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1069287393":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/353804238":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6833/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6832/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/235474344":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3837208470":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/1750949668":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/629137162":"2024-03-30T11:31:55.626759+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:02.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 -- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f548406af60 msgr2=0x7f548406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:02.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f548406af60 0x7f548406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 -- 172.21.15.12:0/2845888668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a415b560 msgr2=0x7f54a40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a415b560 0x7f54a40702c0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f549c016000 tx=0x7f549c004ad0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 -- 172.21.15.12:0/2845888668 shutdown_connections 2024-03-29T11:32:02.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f548406af60 0x7f548406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54a415b560 0x7f54a40702c0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54a4099a60 0x7f54a40caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 --2- 172.21.15.12:0/2845888668 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54a409aa70 0x7f54a406fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 -- 172.21.15.12:0/2845888668 >> 172.21.15.12:0/2845888668 conn(0x7f54a40d1350 msgr2=0x7f54a40cc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:02.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 -- 172.21.15.12:0/2845888668 shutdown_connections 2024-03-29T11:32:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.525+0000 7f54abe00700 1 -- 172.21.15.12:0/2845888668 wait complete. 2024-03-29T11:32:02.541 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:31:24.485419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:31:25.823318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-29T11:31:39.075407+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-29T11:31:39.845704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:32:02.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:32:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.751+0000 7f4100800700 1 Processor -- start 2024-03-29T11:32:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.751+0000 7f4100800700 1 -- start start 2024-03-29T11:32:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.752+0000 7f4100800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80d21f0 0x7f40f80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.752+0000 7f4100800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f40f80cec40 0x7f40f8161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.752+0000 7f4100800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f80ce270 con 0x7f40f81611b0 2024-03-29T11:32:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.752+0000 7f4100800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f815b600 con 0x7f40f80cec40 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.752+0000 7f4100800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80cf0b0 con 0x7f40f80d21f0 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.753+0000 7f40fd400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80d21f0 0x7f40f80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.753+0000 7f40fe800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f40f80cec40 0x7f40f8161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.753+0000 7f40fde00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f40f81611b0 0x7f40f815ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:59096/0 (socket says 172.21.15.12:59096) 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.753+0000 7f40fde00700 1 -- 172.21.15.12:0/132907853 learned_addr learned my addr 172.21.15.12:0/132907853 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.753+0000 7f40fd400700 1 -- 172.21.15.12:0/132907853 >> v2:172.21.15.159:3300/0 conn(0x7f40f80cec40 msgr2=0x7f40f8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.753+0000 7f40fd400700 1 --2- 172.21.15.12:0/132907853 >> v2:172.21.15.159:3300/0 conn(0x7f40f80cec40 0x7f40f8161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.753+0000 7f40fd400700 1 -- 172.21.15.12:0/132907853 >> v1:172.21.15.159:6789/0 conn(0x7f40f81611b0 legacy=0x7f40f815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.753+0000 7f40fd400700 1 -- 172.21.15.12:0/132907853 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f80d2760 con 0x7f40f80d21f0 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.753+0000 7f40fca00700 1 -- 172.21.15.12:0/132907853 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259077506 0 0) 0x7f40f80ce270 con 0x7f40f81611b0 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.754+0000 7f40fd400700 1 --2- 172.21.15.12:0/132907853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80d21f0 0x7f40f80ce700 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f40e8009840 tx=0x7f40e8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35127d54cd7b734f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.754+0000 7f40fca00700 1 -- 172.21.15.12:0/132907853 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8011830 con 0x7f40f80d21f0 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.754+0000 7f40fca00700 1 -- 172.21.15.12:0/132907853 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e8011990 con 0x7f40f80d21f0 2024-03-29T11:32:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.754+0000 7f40fca00700 1 -- 172.21.15.12:0/132907853 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8011d20 con 0x7f40f80d21f0 2024-03-29T11:32:02.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.754+0000 7f4100800700 1 -- 172.21.15.12:0/132907853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80d21f0 msgr2=0x7f40f80ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.754+0000 7f4100800700 1 --2- 172.21.15.12:0/132907853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80d21f0 0x7f40f80ce700 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f40e8009840 tx=0x7f40e8009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:02.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.754+0000 7f4100800700 1 -- 172.21.15.12:0/132907853 shutdown_connections 2024-03-29T11:32:02.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.754+0000 7f4100800700 1 --2- 172.21.15.12:0/132907853 >> v2:172.21.15.159:3300/0 conn(0x7f40f80cec40 0x7f40f8161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.754+0000 7f4100800700 1 --2- 172.21.15.12:0/132907853 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80d21f0 0x7f40f80ce700 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.754+0000 7f4100800700 1 -- 172.21.15.12:0/132907853 >> 172.21.15.12:0/132907853 conn(0x7f40f808aff0 msgr2=0x7f40f80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:02.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.755+0000 7f4100800700 1 -- 172.21.15.12:0/132907853 shutdown_connections 2024-03-29T11:32:02.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.755+0000 7f4100800700 1 -- 172.21.15.12:0/132907853 wait complete. 2024-03-29T11:32:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.756+0000 7f4100800700 1 Processor -- start 2024-03-29T11:32:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.756+0000 7f4100800700 1 -- start start 2024-03-29T11:32:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.757+0000 7f4100800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80cec40 0x7f40f80b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.757+0000 7f4100800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40f80d21f0 0x7f40f8158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.757+0000 7f4100800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40f81611b0 0x7f40f80ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.757+0000 7f4100800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8158cb0 con 0x7f40f81611b0 2024-03-29T11:32:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.757+0000 7f4100800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80b6d90 con 0x7f40f80d21f0 2024-03-29T11:32:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.757+0000 7f4100800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80b7070 con 0x7f40f80cec40 2024-03-29T11:32:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.758+0000 7f40fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40f81611b0 0x7f40f80ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.758+0000 7f40fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40f81611b0 0x7f40f80ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59984/0 (socket says 172.21.15.12:59984) 2024-03-29T11:32:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.758+0000 7f40fe800700 1 -- 172.21.15.12:0/3242358019 learned_addr learned my addr 172.21.15.12:0/3242358019 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.758+0000 7f40fd400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40f80d21f0 0x7f40f8158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.758+0000 7f40fde00700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80cec40 0x7f40f80b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.758+0000 7f40fe800700 1 -- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80cec40 msgr2=0x7f40f80b67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.758+0000 7f40fe800700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80cec40 0x7f40f80b67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.758+0000 7f40fe800700 1 -- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40f80d21f0 msgr2=0x7f40f8158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.758+0000 7f40fe800700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40f80d21f0 0x7f40f8158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.758+0000 7f40fe800700 1 -- 172.21.15.12:0/3242358019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e80094a0 con 0x7f40f81611b0 2024-03-29T11:32:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.759+0000 7f40fe800700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40f81611b0 0x7f40f80ca870 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f40f400efc0 tx=0x7f40f400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.759+0000 7f40f2000700 1 -- 172.21.15.12:0/3242358019 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4009ea0 con 0x7f40f81611b0 2024-03-29T11:32:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.759+0000 7f40f2000700 1 -- 172.21.15.12:0/3242358019 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f400f040 con 0x7f40f81611b0 2024-03-29T11:32:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.759+0000 7f40f2000700 1 -- 172.21.15.12:0/3242358019 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f400ced0 con 0x7f40f81611b0 2024-03-29T11:32:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.759+0000 7f4100800700 1 -- 172.21.15.12:0/3242358019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f80b7350 con 0x7f40f81611b0 2024-03-29T11:32:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.759+0000 7f4100800700 1 -- 172.21.15.12:0/3242358019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f80b7910 con 0x7f40f81611b0 2024-03-29T11:32:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.760+0000 7f4100800700 1 -- 172.21.15.12:0/3242358019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f80b7b00 con 0x7f40f81611b0 2024-03-29T11:32:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.762+0000 7f40f2000700 1 -- 172.21.15.12:0/3242358019 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f40f4014480 con 0x7f40f81611b0 2024-03-29T11:32:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.763+0000 7f40f2000700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f40dc06af60 0x7f40dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.763+0000 7f40f2000700 1 -- 172.21.15.12:0/3242358019 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f40f408a580 con 0x7f40f81611b0 2024-03-29T11:32:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.763+0000 7f40fde00700 1 -- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f40dc06af60 msgr2=0x7f40dc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.763+0000 7f40fde00700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f40dc06af60 0x7f40dc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.767+0000 7f40f2000700 1 -- 172.21.15.12:0/3242358019 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40f4054e00 con 0x7f40f81611b0 2024-03-29T11:32:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.964+0000 7f40fde00700 1 -- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f40dc06af60 msgr2=0x7f40dc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.964+0000 7f40fde00700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f40dc06af60 0x7f40dc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.986+0000 7f4100800700 1 -- 172.21.15.12:0/3242358019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f40f8067720 con 0x7f40f81611b0 2024-03-29T11:32:02.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.987+0000 7f40f2000700 1 -- 172.21.15.12:0/3242358019 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v232) v1 ==== 93+0+10 (secure 0 0 0) 0x7f40f4058420 con 0x7f40f81611b0 2024-03-29T11:32:02.984 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:32:02.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.989+0000 7f4100800700 1 -- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f40dc06af60 msgr2=0x7f40dc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:02.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.989+0000 7f4100800700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f40dc06af60 0x7f40dc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.989+0000 7f4100800700 1 -- 172.21.15.12:0/3242358019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40f81611b0 msgr2=0x7f40f80ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:02.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.989+0000 7f4100800700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40f81611b0 0x7f40f80ca870 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f40f400efc0 tx=0x7f40f400c3a0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.989+0000 7f4100800700 1 -- 172.21.15.12:0/3242358019 shutdown_connections 2024-03-29T11:32:02.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.989+0000 7f4100800700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f40dc06af60 0x7f40dc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.989+0000 7f4100800700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40f81611b0 0x7f40f80ca870 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.989+0000 7f4100800700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40f80cec40 0x7f40f80b67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.989+0000 7f4100800700 1 --2- 172.21.15.12:0/3242358019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40f80d21f0 0x7f40f8158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:02.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.989+0000 7f4100800700 1 -- 172.21.15.12:0/3242358019 >> 172.21.15.12:0/3242358019 conn(0x7f40f808aff0 msgr2=0x7f40f806d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:02.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.990+0000 7f4100800700 1 -- 172.21.15.12:0/3242358019 shutdown_connections 2024-03-29T11:32:02.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:02.990+0000 7f4100800700 1 -- 172.21.15.12:0/3242358019 wait complete. 2024-03-29T11:32:02.987 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:02.991+0000 7fe61ff2c240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:32:03.005 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:32:03.062 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:03.066+0000 7f2910d5b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:32:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.217+0000 7f946a600700 1 Processor -- start 2024-03-29T11:32:03.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.217+0000 7f946a600700 1 -- start start 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f946a600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f946415d800 0x7f94640725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f946a600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94640ce230 0x7f94640ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f946a600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f94640b2070 0x7f94640b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f946a600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94640b40f0 con 0x7f94640b2070 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f946a600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94640709c0 con 0x7f946415d800 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f946a600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9464070c80 con 0x7f94640ce230 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f9463e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f94640b2070 0x7f94640b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f9463e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f94640b2070 0x7f94640b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59998/0 (socket says 172.21.15.12:59998) 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f9463e00700 1 -- 172.21.15.12:0/676022031 learned_addr learned my addr 172.21.15.12:0/676022031 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f9462a00700 1 --2- 172.21.15.12:0/676022031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94640ce230 0x7f94640ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f9463400700 1 --2- 172.21.15.12:0/676022031 >> v2:172.21.15.159:3300/0 conn(0x7f946415d800 0x7f94640725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f9463e00700 1 -- 172.21.15.12:0/676022031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94640ce230 msgr2=0x7f94640ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:03.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f9463e00700 1 --2- 172.21.15.12:0/676022031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94640ce230 0x7f94640ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f9463e00700 1 -- 172.21.15.12:0/676022031 >> v2:172.21.15.159:3300/0 conn(0x7f946415d800 msgr2=0x7f94640725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f9463e00700 1 --2- 172.21.15.12:0/676022031 >> v2:172.21.15.159:3300/0 conn(0x7f946415d800 0x7f94640725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.219+0000 7f9463e00700 1 -- 172.21.15.12:0/676022031 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9464070f60 con 0x7f94640b2070 2024-03-29T11:32:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.220+0000 7f9463e00700 1 --2- 172.21.15.12:0/676022031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 0x7f94640b2500 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f946406c100 tx=0x7f945c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda9a0932036b4e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.220+0000 7f9462000700 1 -- 172.21.15.12:0/676022031 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00e070 con 0x7f94640b2070 2024-03-29T11:32:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.220+0000 7f9462000700 1 -- 172.21.15.12:0/676022031 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c00b910 con 0x7f94640b2070 2024-03-29T11:32:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.220+0000 7f9462000700 1 -- 172.21.15.12:0/676022031 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00bcc0 con 0x7f94640b2070 2024-03-29T11:32:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.221+0000 7f946a600700 1 -- 172.21.15.12:0/676022031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 msgr2=0x7f94640b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.221+0000 7f946a600700 1 --2- 172.21.15.12:0/676022031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 0x7f94640b2500 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f946406c100 tx=0x7f945c00b330 comp rx=0 tx=0).stop 2024-03-29T11:32:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.221+0000 7f946a600700 1 -- 172.21.15.12:0/676022031 shutdown_connections 2024-03-29T11:32:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.221+0000 7f946a600700 1 --2- 172.21.15.12:0/676022031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 0x7f94640b2500 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.221+0000 7f946a600700 1 --2- 172.21.15.12:0/676022031 >> v2:172.21.15.159:3300/0 conn(0x7f946415d800 0x7f94640725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.221+0000 7f946a600700 1 --2- 172.21.15.12:0/676022031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94640ce230 0x7f94640ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.221+0000 7f946a600700 1 -- 172.21.15.12:0/676022031 >> 172.21.15.12:0/676022031 conn(0x7f94640b0d70 msgr2=0x7f94640b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.221+0000 7f946a600700 1 -- 172.21.15.12:0/676022031 shutdown_connections 2024-03-29T11:32:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.221+0000 7f946a600700 1 -- 172.21.15.12:0/676022031 wait complete. 2024-03-29T11:32:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.222+0000 7f946a600700 1 Processor -- start 2024-03-29T11:32:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.223+0000 7f946a600700 1 -- start start 2024-03-29T11:32:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.223+0000 7f946a600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 0x7f9464150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f946a600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94640ce230 0x7f9464156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f946a600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f946415d800 0x7f94640af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f946a600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9464157280 con 0x7f94640b2070 2024-03-29T11:32:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f946a600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946406cf00 con 0x7f94640ce230 2024-03-29T11:32:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f946a600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94640afec0 con 0x7f946415d800 2024-03-29T11:32:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f9463400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 0x7f9464150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f9463400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 0x7f9464150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60006/0 (socket says 172.21.15.12:60006) 2024-03-29T11:32:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f9463400700 1 -- 172.21.15.12:0/23759346 learned_addr learned my addr 172.21.15.12:0/23759346 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f9463e00700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f946415d800 0x7f94640af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f9462a00700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94640ce230 0x7f9464156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f9463400700 1 -- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f946415d800 msgr2=0x7f94640af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f9463400700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f946415d800 0x7f94640af980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f9463400700 1 -- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94640ce230 msgr2=0x7f9464156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f9463400700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94640ce230 0x7f9464156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.224+0000 7f9463400700 1 -- 172.21.15.12:0/23759346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c009b30 con 0x7f94640b2070 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.225+0000 7f9463400700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 0x7f9464150960 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f945400e870 tx=0x7f945400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.225+0000 7f945b400700 1 -- 172.21.15.12:0/23759346 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9454009590 con 0x7f94640b2070 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.225+0000 7f945b400700 1 -- 172.21.15.12:0/23759346 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9454013680 con 0x7f94640b2070 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.225+0000 7f946a600700 1 -- 172.21.15.12:0/23759346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9464156700 con 0x7f94640b2070 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.225+0000 7f945b400700 1 -- 172.21.15.12:0/23759346 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94540139f0 con 0x7f94640b2070 2024-03-29T11:32:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.225+0000 7f946a600700 1 -- 172.21.15.12:0/23759346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9464156cc0 con 0x7f94640b2070 2024-03-29T11:32:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.226+0000 7f946a600700 1 -- 172.21.15.12:0/23759346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9464067720 con 0x7f94640b2070 2024-03-29T11:32:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.227+0000 7f945b400700 1 -- 172.21.15.12:0/23759346 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f9454013b50 con 0x7f94640b2070 2024-03-29T11:32:03.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.228+0000 7f945b400700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f944406af60 0x7f944406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.228+0000 7f945b400700 1 -- 172.21.15.12:0/23759346 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f9454089b10 con 0x7f94640b2070 2024-03-29T11:32:03.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.228+0000 7f9462a00700 1 -- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f944406af60 msgr2=0x7f944406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:03.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.228+0000 7f9462a00700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f944406af60 0x7f944406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:03.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.233+0000 7f945b400700 1 -- 172.21.15.12:0/23759346 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9454054460 con 0x7f94640b2070 2024-03-29T11:32:03.243 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:03.249+0000 7fe61ff2c240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:32:03.331 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:03.336+0000 7f2910d5b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:32:03.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.429+0000 7f9462a00700 1 -- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f944406af60 msgr2=0x7f944406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:03.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.429+0000 7f9462a00700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f944406af60 0x7f944406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.448+0000 7f946a600700 1 -- 172.21.15.12:0/23759346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f94640cb520 con 0x7f94640b2070 2024-03-29T11:32:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.449+0000 7f945b400700 1 -- 172.21.15.12:0/23759346 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9454057a80 con 0x7f94640b2070 2024-03-29T11:32:03.444 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:32:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 -- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f944406af60 msgr2=0x7f944406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f944406af60 0x7f944406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 -- 172.21.15.12:0/23759346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 msgr2=0x7f9464150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 0x7f9464150960 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f945400e870 tx=0x7f945400eb80 comp rx=0 tx=0).stop 2024-03-29T11:32:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 -- 172.21.15.12:0/23759346 shutdown_connections 2024-03-29T11:32:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f944406af60 0x7f944406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f946415d800 0x7f94640af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94640b2070 0x7f9464150960 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 --2- 172.21.15.12:0/23759346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94640ce230 0x7f9464156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 -- 172.21.15.12:0/23759346 >> 172.21.15.12:0/23759346 conn(0x7f94640b0d70 msgr2=0x7f94640cbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 -- 172.21.15.12:0/23759346 shutdown_connections 2024-03-29T11:32:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.451+0000 7f946a600700 1 -- 172.21.15.12:0/23759346 wait complete. 2024-03-29T11:32:03.466 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:32:03.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.673+0000 7fc730800700 1 Processor -- start 2024-03-29T11:32:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.673+0000 7fc730800700 1 -- start start 2024-03-29T11:32:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.674+0000 7fc730800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc7280cbb30 0x7fc7280cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.674+0000 7fc730800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc7280cd480 0x7fc72806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.674+0000 7fc730800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7280cc450 con 0x7fc7280cd020 2024-03-29T11:32:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.674+0000 7fc730800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7280cd8f0 con 0x7fc7280cd480 2024-03-29T11:32:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.674+0000 7fc730800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7280cdae0 con 0x7fc7280cbb30 2024-03-29T11:32:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc7280cd480 0x7fc72806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72de00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc7280cbb30 0x7fc7280cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc7280cd480 0x7fc72806e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60010/0 (socket says 172.21.15.12:60010) 2024-03-29T11:32:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72e800700 1 -- 172.21.15.12:0/2074544364 learned_addr learned my addr 172.21.15.12:0/2074544364 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72d400700 1 --1- 172.21.15.12:0/2074544364 >> v1:172.21.15.159:6789/0 conn(0x7fc7280cd020 0x7fc7280b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:59102/0 (socket says 172.21.15.12:59102) 2024-03-29T11:32:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72e800700 1 -- 172.21.15.12:0/2074544364 >> v2:172.21.15.159:3300/0 conn(0x7fc7280cbb30 msgr2=0x7fc7280cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72e800700 1 --2- 172.21.15.12:0/2074544364 >> v2:172.21.15.159:3300/0 conn(0x7fc7280cbb30 0x7fc7280cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72e800700 1 -- 172.21.15.12:0/2074544364 >> v1:172.21.15.159:6789/0 conn(0x7fc7280cd020 legacy=0x7fc7280b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72ca00700 1 -- 172.21.15.12:0/2074544364 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557629590 0 0) 0x7fc7280cc450 con 0x7fc7280cd020 2024-03-29T11:32:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72e800700 1 -- 172.21.15.12:0/2074544364 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7280cddc0 con 0x7fc7280cd480 2024-03-29T11:32:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.675+0000 7fc72e800700 1 --2- 172.21.15.12:0/2074544364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cd480 0x7fc72806e420 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fc72806fd00 tx=0x7fc72400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ae85e766fbfb9d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.676+0000 7fc72ca00700 1 -- 172.21.15.12:0/2074544364 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72400e070 con 0x7fc7280cd480 2024-03-29T11:32:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.676+0000 7fc72ca00700 1 -- 172.21.15.12:0/2074544364 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72400b910 con 0x7fc7280cd480 2024-03-29T11:32:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.676+0000 7fc72ca00700 1 -- 172.21.15.12:0/2074544364 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72400bcc0 con 0x7fc7280cd480 2024-03-29T11:32:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.676+0000 7fc730800700 1 -- 172.21.15.12:0/2074544364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cd480 msgr2=0x7fc72806e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.676+0000 7fc730800700 1 --2- 172.21.15.12:0/2074544364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cd480 0x7fc72806e420 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fc72806fd00 tx=0x7fc72400b330 comp rx=0 tx=0).stop 2024-03-29T11:32:03.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.676+0000 7fc730800700 1 -- 172.21.15.12:0/2074544364 shutdown_connections 2024-03-29T11:32:03.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.676+0000 7fc730800700 1 --2- 172.21.15.12:0/2074544364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cd480 0x7fc72806e420 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.677+0000 7fc730800700 1 --2- 172.21.15.12:0/2074544364 >> v2:172.21.15.159:3300/0 conn(0x7fc7280cbb30 0x7fc7280cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.677+0000 7fc730800700 1 -- 172.21.15.12:0/2074544364 >> 172.21.15.12:0/2074544364 conn(0x7fc7280b0d70 msgr2=0x7fc7280b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:03.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.677+0000 7fc730800700 1 -- 172.21.15.12:0/2074544364 shutdown_connections 2024-03-29T11:32:03.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.677+0000 7fc730800700 1 -- 172.21.15.12:0/2074544364 wait complete. 2024-03-29T11:32:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.678+0000 7fc730800700 1 Processor -- start 2024-03-29T11:32:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.679+0000 7fc730800700 1 -- start start 2024-03-29T11:32:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.679+0000 7fc730800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cbb30 0x7fc728154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc730800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7280cd020 0x7fc7280b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc730800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7280cd480 0x7fc72814d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc730800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc728156650 con 0x7fc7280cbb30 2024-03-29T11:32:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc730800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc728156930 con 0x7fc7280cd020 2024-03-29T11:32:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc730800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc728156c10 con 0x7fc7280cd480 2024-03-29T11:32:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc72de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cbb30 0x7fc728154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc72de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cbb30 0x7fc728154220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60020/0 (socket says 172.21.15.12:60020) 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc72de00700 1 -- 172.21.15.12:0/1797070136 learned_addr learned my addr 172.21.15.12:0/1797070136 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc72d400700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7280cd020 0x7fc7280b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc72e800700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7280cd480 0x7fc72814d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc72de00700 1 -- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7280cd480 msgr2=0x7fc72814d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc72de00700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7280cd480 0x7fc72814d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc72de00700 1 -- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7280cd020 msgr2=0x7fc7280b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc72de00700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7280cd020 0x7fc7280b3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.680+0000 7fc72de00700 1 -- 172.21.15.12:0/1797070136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724009b30 con 0x7fc7280cbb30 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.681+0000 7fc72de00700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cbb30 0x7fc728154220 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00e870 tx=0x7fc71c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.681+0000 7fc722000700 1 -- 172.21.15.12:0/1797070136 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c009590 con 0x7fc7280cbb30 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.681+0000 7fc722000700 1 -- 172.21.15.12:0/1797070136 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c013680 con 0x7fc7280cbb30 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.681+0000 7fc730800700 1 -- 172.21.15.12:0/1797070136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc728156ef0 con 0x7fc7280cbb30 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.681+0000 7fc722000700 1 -- 172.21.15.12:0/1797070136 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c0139f0 con 0x7fc7280cbb30 2024-03-29T11:32:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.681+0000 7fc730800700 1 -- 172.21.15.12:0/1797070136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc728154760 con 0x7fc7280cbb30 2024-03-29T11:32:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.682+0000 7fc730800700 1 -- 172.21.15.12:0/1797070136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc728067720 con 0x7fc7280cbb30 2024-03-29T11:32:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.683+0000 7fc722000700 1 -- 172.21.15.12:0/1797070136 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fc71c013b50 con 0x7fc7280cbb30 2024-03-29T11:32:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.684+0000 7fc722000700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc70c06afb0 0x7fc70c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.684+0000 7fc722000700 1 -- 172.21.15.12:0/1797070136 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7fc71c089a70 con 0x7fc7280cbb30 2024-03-29T11:32:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.684+0000 7fc72d400700 1 -- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc70c06afb0 msgr2=0x7fc70c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.684+0000 7fc72d400700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc70c06afb0 0x7fc70c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:03.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.688+0000 7fc722000700 1 -- 172.21.15.12:0/1797070136 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc71c054440 con 0x7fc7280cbb30 2024-03-29T11:32:03.705 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:03.710+0000 7fe61ff2c240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:32:03.818 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:03.822+0000 7f2910d5b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:32:03.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.885+0000 7fc72d400700 1 -- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc70c06afb0 msgr2=0x7fc70c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:03.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.885+0000 7fc72d400700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc70c06afb0 0x7fc70c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.906+0000 7fc730800700 1 -- 172.21.15.12:0/1797070136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc72815eaf0 con 0x7fc7280cbb30 2024-03-29T11:32:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.907+0000 7fc722000700 1 -- 172.21.15.12:0/1797070136 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v232) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc71c0170a0 con 0x7fc7280cbb30 2024-03-29T11:32:03.903 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:32:03.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.909+0000 7fc730800700 1 -- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc70c06afb0 msgr2=0x7fc70c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:03.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.909+0000 7fc730800700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc70c06afb0 0x7fc70c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.909+0000 7fc730800700 1 -- 172.21.15.12:0/1797070136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cbb30 msgr2=0x7fc728154220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.909+0000 7fc730800700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cbb30 0x7fc728154220 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00e870 tx=0x7fc71c00eb80 comp rx=0 tx=0).stop 2024-03-29T11:32:03.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.909+0000 7fc730800700 1 -- 172.21.15.12:0/1797070136 shutdown_connections 2024-03-29T11:32:03.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.909+0000 7fc730800700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc70c06afb0 0x7fc70c06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.909+0000 7fc730800700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7280cd480 0x7fc72814d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.909+0000 7fc730800700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7280cbb30 0x7fc728154220 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.909+0000 7fc730800700 1 --2- 172.21.15.12:0/1797070136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7280cd020 0x7fc7280b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:03.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.909+0000 7fc730800700 1 -- 172.21.15.12:0/1797070136 >> 172.21.15.12:0/1797070136 conn(0x7fc7280b0d70 msgr2=0x7fc72806e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:03.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.910+0000 7fc730800700 1 -- 172.21.15.12:0/1797070136 shutdown_connections 2024-03-29T11:32:03.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:03.910+0000 7fc730800700 1 -- 172.21.15.12:0/1797070136 wait complete. 2024-03-29T11:32:03.925 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-29T11:32:04.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.133+0000 7f6f4ee00700 1 Processor -- start 2024-03-29T11:32:04.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.133+0000 7f6f4ee00700 1 -- start start 2024-03-29T11:32:04.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.134+0000 7f6f4ee00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f4815b740 0x7f6f480cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:04.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.134+0000 7f6f4ee00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4809adf0 con 0x7f6f480cf440 2024-03-29T11:32:04.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.134+0000 7f6f4ee00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4809a830 con 0x7f6f480cce30 2024-03-29T11:32:04.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.134+0000 7f6f4ee00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4815bb20 con 0x7f6f4815b740 2024-03-29T11:32:04.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.134+0000 7f6f47e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f4815b740 0x7f6f480cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:04.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.135+0000 7f6f47e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f4815b740 0x7f6f480cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60036/0 (socket says 172.21.15.12:60036) 2024-03-29T11:32:04.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.135+0000 7f6f47e00700 1 -- 172.21.15.12:0/400139839 learned_addr learned my addr 172.21.15.12:0/400139839 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:04.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.135+0000 7f6f47400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6f480cf440 0x7f6f480c57b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43144/0 (socket says 172.21.15.12:43144) 2024-03-29T11:32:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.135+0000 7f6f47e00700 1 -- 172.21.15.12:0/400139839 >> v1:172.21.15.159:6789/0 conn(0x7f6f480cce30 legacy=0x7f6f480c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.135+0000 7f6f47e00700 1 -- 172.21.15.12:0/400139839 >> v1:172.21.15.12:6789/0 conn(0x7f6f480cf440 legacy=0x7f6f480c57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.135+0000 7f6f47e00700 1 -- 172.21.15.12:0/400139839 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4809ab10 con 0x7f6f4815b740 2024-03-29T11:32:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.135+0000 7f6f46a00700 1 -- 172.21.15.12:0/400139839 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544363160 0 0) 0x7f6f4809adf0 con 0x7f6f480cf440 2024-03-29T11:32:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.135+0000 7f6f47e00700 1 --2- 172.21.15.12:0/400139839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f4815b740 0x7f6f480cef00 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f6f380098f0 tx=0x7f6f38009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd320caa0cf65c2b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.136+0000 7f6f46a00700 1 -- 172.21.15.12:0/400139839 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3800b8a0 con 0x7f6f4815b740 2024-03-29T11:32:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.136+0000 7f6f46a00700 1 -- 172.21.15.12:0/400139839 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3800ba00 con 0x7f6f4815b740 2024-03-29T11:32:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.136+0000 7f6f46a00700 1 -- 172.21.15.12:0/400139839 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3800bd90 con 0x7f6f4815b740 2024-03-29T11:32:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.136+0000 7f6f4ee00700 1 -- 172.21.15.12:0/400139839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f4815b740 msgr2=0x7f6f480cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.136+0000 7f6f4ee00700 1 --2- 172.21.15.12:0/400139839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f4815b740 0x7f6f480cef00 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f6f380098f0 tx=0x7f6f38009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.136+0000 7f6f4ee00700 1 -- 172.21.15.12:0/400139839 shutdown_connections 2024-03-29T11:32:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.136+0000 7f6f4ee00700 1 --2- 172.21.15.12:0/400139839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f4815b740 0x7f6f480cef00 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.136+0000 7f6f4ee00700 1 -- 172.21.15.12:0/400139839 >> 172.21.15.12:0/400139839 conn(0x7f6f480b2e20 msgr2=0x7f6f480b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.136+0000 7f6f4ee00700 1 -- 172.21.15.12:0/400139839 shutdown_connections 2024-03-29T11:32:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.136+0000 7f6f4ee00700 1 -- 172.21.15.12:0/400139839 wait complete. 2024-03-29T11:32:04.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.138+0000 7f6f4ee00700 1 Processor -- start 2024-03-29T11:32:04.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.138+0000 7f6f4ee00700 1 -- start start 2024-03-29T11:32:04.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.139+0000 7f6f4ee00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f480cce30 0x7f6f480c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:04.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.139+0000 7f6f4ee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f480cf440 0x7f6f480b4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.139+0000 7f6f4ee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f4815b740 0x7f6f480b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.139+0000 7f6f4ee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480b4cf0 con 0x7f6f480cf440 2024-03-29T11:32:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.139+0000 7f6f4ee00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480b4fd0 con 0x7f6f480cce30 2024-03-29T11:32:04.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.139+0000 7f6f4ee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480b52b0 con 0x7f6f4815b740 2024-03-29T11:32:04.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f480cf440 0x7f6f480b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:04.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f480cce30 0x7f6f480c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:04.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f480cce30 0x7f6f480c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:48878/0 (socket says 172.21.15.12:48878) 2024-03-29T11:32:04.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f480cf440 0x7f6f480b4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60044/0 (socket says 172.21.15.12:60044) 2024-03-29T11:32:04.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47e00700 1 -- 172.21.15.12:0/3037852923 learned_addr learned my addr 172.21.15.12:0/3037852923 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:04.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f4ce00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f4815b740 0x7f6f480b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:04.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47e00700 1 -- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f4815b740 msgr2=0x7f6f480b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47e00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f4815b740 0x7f6f480b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47e00700 1 -- 172.21.15.12:0/3037852923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f480cf440 msgr2=0x7f6f480b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47e00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f480cf440 0x7f6f480b4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47e00700 1 -- 172.21.15.12:0/3037852923 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f38009550 con 0x7f6f480cce30 2024-03-29T11:32:04.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47400700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f480cf440 0x7f6f480b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:04.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.140+0000 7f6f47e00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f480cce30 0x7f6f480c6ac0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f6f480724f0 tx=0x7f6f38004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:04.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.141+0000 7f6f3fe00700 1 -- 172.21.15.12:0/3037852923 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3801a900 con 0x7f6f480cce30 2024-03-29T11:32:04.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.141+0000 7f6f3fe00700 1 -- 172.21.15.12:0/3037852923 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3801f070 con 0x7f6f480cce30 2024-03-29T11:32:04.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.141+0000 7f6f4ee00700 1 -- 172.21.15.12:0/3037852923 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f480b5590 con 0x7f6f480cce30 2024-03-29T11:32:04.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.141+0000 7f6f3fe00700 1 -- 172.21.15.12:0/3037852923 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3801ac60 con 0x7f6f480cce30 2024-03-29T11:32:04.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.141+0000 7f6f4ee00700 1 -- 172.21.15.12:0/3037852923 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f480c77a0 con 0x7f6f480cce30 2024-03-29T11:32:04.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.142+0000 7f6f4ee00700 1 -- 172.21.15.12:0/3037852923 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f48067720 con 0x7f6f480cce30 2024-03-29T11:32:04.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.143+0000 7f6f3fe00700 1 -- 172.21.15.12:0/3037852923 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f6f38026480 con 0x7f6f480cce30 2024-03-29T11:32:04.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.144+0000 7f6f3fe00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f2806af60 0x7f6f2806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:04.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.144+0000 7f6f3fe00700 1 -- 172.21.15.12:0/3037852923 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f6f38021070 con 0x7f6f480cce30 2024-03-29T11:32:04.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.144+0000 7f6f47400700 1 -- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f2806af60 msgr2=0x7f6f2806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:04.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.144+0000 7f6f47400700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f2806af60 0x7f6f2806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:04.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.149+0000 7f6f3fe00700 1 -- 172.21.15.12:0/3037852923 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f38055f40 con 0x7f6f480cce30 2024-03-29T11:32:04.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.345+0000 7f6f47400700 1 -- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f2806af60 msgr2=0x7f6f2806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.345+0000 7f6f47400700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f2806af60 0x7f6f2806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:04.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.366+0000 7f6f4ee00700 1 -- 172.21.15.12:0/3037852923 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f480b5110 con 0x7f6f480cce30 2024-03-29T11:32:04.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.367+0000 7f6f3fe00700 1 -- 172.21.15.12:0/3037852923 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6f480b5110 con 0x7f6f480cce30 2024-03-29T11:32:04.365 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:32:04.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 -- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f2806af60 msgr2=0x7f6f2806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:04.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f2806af60 0x7f6f2806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 -- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f480cce30 msgr2=0x7f6f480c6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f480cce30 0x7f6f480c6ac0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f6f480724f0 tx=0x7f6f38004750 comp rx=0 tx=0).stop 2024-03-29T11:32:04.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 -- 172.21.15.12:0/3037852923 shutdown_connections 2024-03-29T11:32:04.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f2806af60 0x7f6f2806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f4815b740 0x7f6f480b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f480cce30 0x7f6f480c6ac0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 --2- 172.21.15.12:0/3037852923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f480cf440 0x7f6f480b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 -- 172.21.15.12:0/3037852923 >> 172.21.15.12:0/3037852923 conn(0x7f6f480b2e20 msgr2=0x7f6f480cbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:04.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 -- 172.21.15.12:0/3037852923 shutdown_connections 2024-03-29T11:32:04.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.370+0000 7f6f4ee00700 1 -- 172.21.15.12:0/3037852923 wait complete. 2024-03-29T11:32:04.386 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-29T11:32:04.544 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:04.432+0000 7fe61ff2c240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:32:04.544 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:04.523+0000 7fe61ff2c240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:32:04.576 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:04.580+0000 7f2910d5b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:32:04.677 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:04.682+0000 7f2910d5b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:32:04.679 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:04.685+0000 7fe61ff2c240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:32:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.751+0000 7f1b3d400700 1 Processor -- start 2024-03-29T11:32:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.751+0000 7f1b3d400700 1 -- start start 2024-03-29T11:32:04.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.753+0000 7f1b3d400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1b380d16a0 0x7f1b380d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.753+0000 7f1b3d400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b38161420 con 0x7f1b380d2370 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.753+0000 7f1b3d400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b38161700 con 0x7f1b3815d9d0 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.753+0000 7f1b3d400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b380d4340 con 0x7f1b380d16a0 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.753+0000 7f1b37200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1b380d2370 0x7f1b380ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43154/0 (socket says 172.21.15.12:43154) 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.753+0000 7f1b37200700 1 -- 172.21.15.12:0/3077406901 learned_addr learned my addr 172.21.15.12:0/3077406901 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.753+0000 7f1b35e00700 1 --2- 172.21.15.12:0/3077406901 >> v2:172.21.15.159:3300/0 conn(0x7f1b380d16a0 0x7f1b380d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.754+0000 7f1b35400700 1 -- 172.21.15.12:0/3077406901 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611232659 0 0) 0x7f1b38161420 con 0x7f1b380d2370 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.754+0000 7f1b35400700 1 -- 172.21.15.12:0/3077406901 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b1c0035b0 con 0x7f1b380d2370 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.754+0000 7f1b35400700 1 -- 172.21.15.12:0/3077406901 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108938526 0 0) 0x7f1b38161700 con 0x7f1b3815d9d0 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.754+0000 7f1b35400700 1 -- 172.21.15.12:0/3077406901 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b38161420 con 0x7f1b3815d9d0 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.754+0000 7f1b35e00700 1 -- 172.21.15.12:0/3077406901 >> v1:172.21.15.159:6789/0 conn(0x7f1b3815d9d0 legacy=0x7f1b380d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.754+0000 7f1b35e00700 1 -- 172.21.15.12:0/3077406901 >> v1:172.21.15.12:6789/0 conn(0x7f1b380d2370 legacy=0x7f1b380ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.754+0000 7f1b35e00700 1 -- 172.21.15.12:0/3077406901 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b381619e0 con 0x7f1b380d16a0 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.754+0000 7f1b35e00700 1 --2- 172.21.15.12:0/3077406901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b380d16a0 0x7f1b380d1e30 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1b24009820 tx=0x7f1b24009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11837543d5020546 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.755+0000 7f1b35400700 1 -- 172.21.15.12:0/3077406901 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24005e30 con 0x7f1b380d16a0 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.755+0000 7f1b35400700 1 -- 172.21.15.12:0/3077406901 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b24006030 con 0x7f1b380d16a0 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.755+0000 7f1b35400700 1 -- 172.21.15.12:0/3077406901 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240063c0 con 0x7f1b380d16a0 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.755+0000 7f1b3d400700 1 -- 172.21.15.12:0/3077406901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b380d16a0 msgr2=0x7f1b380d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.755+0000 7f1b3d400700 1 --2- 172.21.15.12:0/3077406901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b380d16a0 0x7f1b380d1e30 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1b24009820 tx=0x7f1b24009b30 comp rx=0 tx=0).stop 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.756+0000 7f1b3d400700 1 -- 172.21.15.12:0/3077406901 shutdown_connections 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.756+0000 7f1b3d400700 1 --2- 172.21.15.12:0/3077406901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b380d16a0 0x7f1b380d1e30 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.756+0000 7f1b3d400700 1 -- 172.21.15.12:0/3077406901 >> 172.21.15.12:0/3077406901 conn(0x7f1b3808aff0 msgr2=0x7f1b380d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.756+0000 7f1b3d400700 1 -- 172.21.15.12:0/3077406901 shutdown_connections 2024-03-29T11:32:04.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.756+0000 7f1b3d400700 1 -- 172.21.15.12:0/3077406901 wait complete. 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.757+0000 7f1b3d400700 1 Processor -- start 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.758+0000 7f1b3d400700 1 -- start start 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.758+0000 7f1b3d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b380d16a0 0x7f1b3815c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.758+0000 7f1b3d400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b380d2370 0x7f1b3815b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.758+0000 7f1b3d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3815d9d0 0x7f1b380b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.758+0000 7f1b3d400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b381611a0 con 0x7f1b3815d9d0 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.758+0000 7f1b3d400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3815bed0 con 0x7f1b380d2370 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.758+0000 7f1b3d400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b380ca790 con 0x7f1b380d16a0 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b37200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3815d9d0 0x7f1b380b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b37200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3815d9d0 0x7f1b380b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60058/0 (socket says 172.21.15.12:60058) 2024-03-29T11:32:04.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b35e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b380d2370 0x7f1b3815b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:04.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b37200700 1 -- 172.21.15.12:0/1478559159 learned_addr learned my addr 172.21.15.12:0/1478559159 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:04.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b36800700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b380d16a0 0x7f1b3815c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:04.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b35e00700 1 -- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b380d16a0 msgr2=0x7f1b3815c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b35e00700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b380d16a0 0x7f1b3815c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b35e00700 1 -- 172.21.15.12:0/1478559159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3815d9d0 msgr2=0x7f1b380b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b35e00700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3815d9d0 0x7f1b380b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b35e00700 1 -- 172.21.15.12:0/1478559159 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b24009430 con 0x7f1b380d2370 2024-03-29T11:32:04.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.759+0000 7f1b37200700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3815d9d0 0x7f1b380b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:04.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.760+0000 7f1b35e00700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b380d2370 0x7f1b3815b990 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f1b3808ca10 tx=0x7f1b24004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:04.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.760+0000 7f1b2ea00700 1 -- 172.21.15.12:0/1478559159 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24006830 con 0x7f1b380d2370 2024-03-29T11:32:04.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.760+0000 7f1b2ea00700 1 -- 172.21.15.12:0/1478559159 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b24020070 con 0x7f1b380d2370 2024-03-29T11:32:04.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.760+0000 7f1b2ea00700 1 -- 172.21.15.12:0/1478559159 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24006990 con 0x7f1b380d2370 2024-03-29T11:32:04.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.760+0000 7f1b3d400700 1 -- 172.21.15.12:0/1478559159 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b380caa70 con 0x7f1b380d2370 2024-03-29T11:32:04.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.760+0000 7f1b3d400700 1 -- 172.21.15.12:0/1478559159 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b380cb030 con 0x7f1b380d2370 2024-03-29T11:32:04.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.761+0000 7f1b3d400700 1 -- 172.21.15.12:0/1478559159 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b380d1ca0 con 0x7f1b380d2370 2024-03-29T11:32:04.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.762+0000 7f1b2ea00700 1 -- 172.21.15.12:0/1478559159 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f1b24022030 con 0x7f1b380d2370 2024-03-29T11:32:04.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.763+0000 7f1b2ea00700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b1806af60 0x7f1b1806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:04.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.763+0000 7f1b2ea00700 1 -- 172.21.15.12:0/1478559159 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f1b2408af60 con 0x7f1b380d2370 2024-03-29T11:32:04.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.763+0000 7f1b36800700 1 -- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b1806af60 msgr2=0x7f1b1806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:04.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.763+0000 7f1b36800700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b1806af60 0x7f1b1806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:04.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.767+0000 7f1b2ea00700 1 -- 172.21.15.12:0/1478559159 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b24055940 con 0x7f1b380d2370 2024-03-29T11:32:04.852 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:04.856+0000 7f2910d5b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:32:04.901 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:04.907+0000 7fe61ff2c240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:32:04.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.964+0000 7f1b36800700 1 -- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b1806af60 msgr2=0x7f1b1806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:04.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.964+0000 7f1b36800700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b1806af60 0x7f1b1806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:04.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.988+0000 7f1b3d400700 1 -- 172.21.15.12:0/1478559159 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f1b38067720 con 0x7f1b380d2370 2024-03-29T11:32:04.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.988+0000 7f1b2ea00700 1 -- 172.21.15.12:0/1478559159 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v232) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1b24058f60 con 0x7f1b380d2370 2024-03-29T11:32:04.985 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:32:04.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 -- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b1806af60 msgr2=0x7f1b1806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:04.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b1806af60 0x7f1b1806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 -- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b380d2370 msgr2=0x7f1b3815b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b380d2370 0x7f1b3815b990 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f1b3808ca10 tx=0x7f1b24004750 comp rx=0 tx=0).stop 2024-03-29T11:32:04.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 -- 172.21.15.12:0/1478559159 shutdown_connections 2024-03-29T11:32:04.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b1806af60 0x7f1b1806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3815d9d0 0x7f1b380b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b380d16a0 0x7f1b3815c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 --2- 172.21.15.12:0/1478559159 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b380d2370 0x7f1b3815b990 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:04.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 -- 172.21.15.12:0/1478559159 >> 172.21.15.12:0/1478559159 conn(0x7f1b3808aff0 msgr2=0x7f1b380b5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:04.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 -- 172.21.15.12:0/1478559159 shutdown_connections 2024-03-29T11:32:04.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:04.991+0000 7f1b3d400700 1 -- 172.21.15.12:0/1478559159 wait complete. 2024-03-29T11:32:04.995 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:05.002+0000 7fe61ff2c240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:32:05.007 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T11:32:05.080 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:05.084+0000 7f2910d5b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:32:05.095 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:05.101+0000 7fe61ff2c240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:32:05.181 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:05.185+0000 7f2910d5b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:32:05.195 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:05.201+0000 7fe61ff2c240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:32:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.258+0000 7f17a4e00700 1 Processor -- start 2024-03-29T11:32:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.258+0000 7f17a4e00700 1 -- start start 2024-03-29T11:32:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.259+0000 7f17a4e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f17a00d21f0 0x7f17a00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.259+0000 7f17a4e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a00ce270 con 0x7f17a00cec40 2024-03-29T11:32:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.259+0000 7f17a4e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a015b600 con 0x7f17a01611b0 2024-03-29T11:32:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.259+0000 7f17a4e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a00d2760 con 0x7f17a00d21f0 2024-03-29T11:32:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.259+0000 7f179d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f17a00d21f0 0x7f17a00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.259+0000 7f179e800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f17a00cec40 0x7f17a0161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43162/0 (socket says 172.21.15.12:43162) 2024-03-29T11:32:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.259+0000 7f179d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f17a00d21f0 0x7f17a00ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60066/0 (socket says 172.21.15.12:60066) 2024-03-29T11:32:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.259+0000 7f179e800700 1 -- 172.21.15.12:0/4168895858 learned_addr learned my addr 172.21.15.12:0/4168895858 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.260+0000 7f179d400700 1 -- 172.21.15.12:0/4168895858 >> v1:172.21.15.159:6789/0 conn(0x7f17a01611b0 legacy=0x7f17a015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.260+0000 7f179d400700 1 -- 172.21.15.12:0/4168895858 >> v1:172.21.15.12:6789/0 conn(0x7f17a00cec40 legacy=0x7f17a0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.260+0000 7f179d400700 1 -- 172.21.15.12:0/4168895858 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a00d2a40 con 0x7f17a00d21f0 2024-03-29T11:32:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.260+0000 7f179d400700 1 --2- 172.21.15.12:0/4168895858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00d21f0 0x7f17a00ce700 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f178c009840 tx=0x7f178c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d17806caf9518a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.260+0000 7f179ca00700 1 -- 172.21.15.12:0/4168895858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c011830 con 0x7f17a00d21f0 2024-03-29T11:32:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.261+0000 7f179ca00700 1 -- 172.21.15.12:0/4168895858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c011990 con 0x7f17a00d21f0 2024-03-29T11:32:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.261+0000 7f179ca00700 1 -- 172.21.15.12:0/4168895858 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c011d20 con 0x7f17a00d21f0 2024-03-29T11:32:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.261+0000 7f17a4e00700 1 -- 172.21.15.12:0/4168895858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00d21f0 msgr2=0x7f17a00ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.261+0000 7f17a4e00700 1 --2- 172.21.15.12:0/4168895858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00d21f0 0x7f17a00ce700 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f178c009840 tx=0x7f178c009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.261+0000 7f17a4e00700 1 -- 172.21.15.12:0/4168895858 shutdown_connections 2024-03-29T11:32:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.261+0000 7f17a4e00700 1 --2- 172.21.15.12:0/4168895858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00d21f0 0x7f17a00ce700 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.261+0000 7f17a4e00700 1 -- 172.21.15.12:0/4168895858 >> 172.21.15.12:0/4168895858 conn(0x7f17a008aff0 msgr2=0x7f17a00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.261+0000 7f17a4e00700 1 -- 172.21.15.12:0/4168895858 shutdown_connections 2024-03-29T11:32:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.261+0000 7f17a4e00700 1 -- 172.21.15.12:0/4168895858 wait complete. 2024-03-29T11:32:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.263+0000 7f17a4e00700 1 Processor -- start 2024-03-29T11:32:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.263+0000 7f17a4e00700 1 -- start start 2024-03-29T11:32:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.264+0000 7f17a4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00cec40 0x7f17a0153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.264+0000 7f17a4e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17a00d21f0 0x7f17a00b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.264+0000 7f17a4e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17a01611b0 0x7f17a00b8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.264+0000 7f17a4e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00b6bd0 con 0x7f17a00cec40 2024-03-29T11:32:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.264+0000 7f17a4e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00b6eb0 con 0x7f17a00d21f0 2024-03-29T11:32:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.264+0000 7f17a4e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00b7190 con 0x7f17a01611b0 2024-03-29T11:32:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.264+0000 7f179de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00cec40 0x7f17a0153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.264+0000 7f179de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00cec40 0x7f17a0153bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60080/0 (socket says 172.21.15.12:60080) 2024-03-29T11:32:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f179de00700 1 -- 172.21.15.12:0/4159218236 learned_addr learned my addr 172.21.15.12:0/4159218236 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f179e800700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17a01611b0 0x7f17a00b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f179d400700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17a00d21f0 0x7f17a00b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f179de00700 1 -- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17a01611b0 msgr2=0x7f17a00b8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f179de00700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17a01611b0 0x7f17a00b8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f179de00700 1 -- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17a00d21f0 msgr2=0x7f17a00b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f179de00700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17a00d21f0 0x7f17a00b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f179de00700 1 -- 172.21.15.12:0/4159218236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0094a0 con 0x7f17a00cec40 2024-03-29T11:32:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f179de00700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00cec40 0x7f17a0153bf0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f179000c700 tx=0x7f179000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f1796000700 1 -- 172.21.15.12:0/4159218236 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17900052d0 con 0x7f17a00cec40 2024-03-29T11:32:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.265+0000 7f1796000700 1 -- 172.21.15.12:0/4159218236 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1790005430 con 0x7f17a00cec40 2024-03-29T11:32:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.266+0000 7f17a4e00700 1 -- 172.21.15.12:0/4159218236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a00b7470 con 0x7f17a00cec40 2024-03-29T11:32:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.266+0000 7f1796000700 1 -- 172.21.15.12:0/4159218236 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790003880 con 0x7f17a00cec40 2024-03-29T11:32:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.266+0000 7f17a4e00700 1 -- 172.21.15.12:0/4159218236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a0154130 con 0x7f17a00cec40 2024-03-29T11:32:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.267+0000 7f17a4e00700 1 -- 172.21.15.12:0/4159218236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a00b7990 con 0x7f17a00cec40 2024-03-29T11:32:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.268+0000 7f1796000700 1 -- 172.21.15.12:0/4159218236 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f17900039e0 con 0x7f17a00cec40 2024-03-29T11:32:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.269+0000 7f1796000700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f178006af60 0x7f178006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.269+0000 7f1796000700 1 -- 172.21.15.12:0/4159218236 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f1790088e90 con 0x7f17a00cec40 2024-03-29T11:32:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.269+0000 7f179d400700 1 -- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f178006af60 msgr2=0x7f178006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.269+0000 7f179d400700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f178006af60 0x7f178006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.273+0000 7f1796000700 1 -- 172.21.15.12:0/4159218236 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17a00b7990 con 0x7f17a00cec40 2024-03-29T11:32:05.289 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:05.294+0000 7f2910d5b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:32:05.295 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:05.301+0000 7fe61ff2c240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:32:05.394 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:05.398+0000 7f2910d5b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:32:05.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.470+0000 7f179d400700 1 -- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f178006af60 msgr2=0x7f178006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:05.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.470+0000 7f179d400700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f178006af60 0x7f178006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:05.498 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:05.502+0000 7f2910d5b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:32:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.505+0000 7f17a4e00700 1 -- 172.21.15.12:0/4159218236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f17a00b7990 con 0x7f17a00cec40 2024-03-29T11:32:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.505+0000 7f1796000700 1 -- 172.21.15.12:0/4159218236 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v232) v1 ==== 80+0+465 (secure 0 0 0) 0x7f17a00b7990 con 0x7f17a00cec40 2024-03-29T11:32:05.501 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1069287393 2024-03-30T11:31:55.626759+0000 2024-03-29T11:32:05.501 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/353804238 2024-03-30T11:31:55.626759+0000 2024-03-29T11:32:05.502 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/62646 2024-03-30T11:31:55.626759+0000 2024-03-29T11:32:05.502 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/62646 2024-03-30T11:31:55.626759+0000 2024-03-29T11:32:05.502 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/235474344 2024-03-30T11:31:55.626759+0000 2024-03-29T11:32:05.502 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3837208470 2024-03-30T11:31:55.626759+0000 2024-03-29T11:32:05.502 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1750949668 2024-03-30T11:31:55.626759+0000 2024-03-29T11:32:05.502 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/629137162 2024-03-30T11:31:55.626759+0000 2024-03-29T11:32:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 -- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f178006af60 msgr2=0x7f178006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f178006af60 0x7f178006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 -- 172.21.15.12:0/4159218236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00cec40 msgr2=0x7f17a0153bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00cec40 0x7f17a0153bf0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f179000c700 tx=0x7f179000ca10 comp rx=0 tx=0).stop 2024-03-29T11:32:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 -- 172.21.15.12:0/4159218236 shutdown_connections 2024-03-29T11:32:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f178006af60 0x7f178006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17a01611b0 0x7f17a00b8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a00cec40 0x7f17a0153bf0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 --2- 172.21.15.12:0/4159218236 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17a00d21f0 0x7f17a00b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 -- 172.21.15.12:0/4159218236 >> 172.21.15.12:0/4159218236 conn(0x7f17a008aff0 msgr2=0x7f17a006bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 -- 172.21.15.12:0/4159218236 shutdown_connections 2024-03-29T11:32:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:05.508+0000 7f17a4e00700 1 -- 172.21.15.12:0/4159218236 wait complete. 2024-03-29T11:32:05.506 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T11:32:05.506 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:05.512+0000 7fe61ff2c240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:32:05.522 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T11:32:05.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:05.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T11:32:05.644 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:32:05.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:05.644 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T11:32:05.689 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:32:05.690 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:32:05.731 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:05.735+0000 7f2910d5b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:32:05.736 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:32:05.737 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:32:05.737 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:32:05.774 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:05.779+0000 7fe61ff2c240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:32:05.788 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:32:05.788 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:32:05.788 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:32:05.840 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:32:05.840 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:05.972 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:05.978+0000 7fe61ff2c240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:32:06.005 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:06.010+0000 7f2910d5b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:32:06.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.050+0000 7fa25d200700 1 Processor -- start 2024-03-29T11:32:06.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.050+0000 7fa25d200700 1 -- start start 2024-03-29T11:32:06.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.051+0000 7fa25d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2580d18b0 0x7fa2580cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.051+0000 7fa25d200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa258161590 0x7fa258161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.051+0000 7fa25d200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2580ceda0 con 0x7fa258160f10 2024-03-29T11:32:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.051+0000 7fa25d200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2580d1d30 con 0x7fa258161590 2024-03-29T11:32:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.051+0000 7fa25d200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa258161f60 con 0x7fa2580d18b0 2024-03-29T11:32:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.052+0000 7fa256800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa258161590 0x7fa258161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.052+0000 7fa256800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa258161590 0x7fa258161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60082/0 (socket says 172.21.15.12:60082) 2024-03-29T11:32:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.052+0000 7fa256800700 1 -- 172.21.15.12:0/2926915733 learned_addr learned my addr 172.21.15.12:0/2926915733 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:06.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.052+0000 7fa255400700 1 --2- 172.21.15.12:0/2926915733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2580d18b0 0x7fa2580cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:06.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.052+0000 7fa256800700 1 -- 172.21.15.12:0/2926915733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2580d18b0 msgr2=0x7fa2580cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:06.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.052+0000 7fa256800700 1 --2- 172.21.15.12:0/2926915733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2580d18b0 0x7fa2580cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.052+0000 7fa256800700 1 -- 172.21.15.12:0/2926915733 >> v1:172.21.15.159:6789/0 conn(0x7fa258160f10 legacy=0x7fa2580d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:06.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.052+0000 7fa256800700 1 -- 172.21.15.12:0/2926915733 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2581612f0 con 0x7fa258161590 2024-03-29T11:32:06.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.052+0000 7fa254a00700 1 -- 172.21.15.12:0/2926915733 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213590636 0 0) 0x7fa2580ceda0 con 0x7fa258160f10 2024-03-29T11:32:06.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.053+0000 7fa256800700 1 --2- 172.21.15.12:0/2926915733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258161590 0x7fa258161a20 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fa250009e80 tx=0x7fa25000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9dc43e4107b8c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:06.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.053+0000 7fa254a00700 1 -- 172.21.15.12:0/2926915733 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000e070 con 0x7fa258161590 2024-03-29T11:32:06.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.053+0000 7fa254a00700 1 -- 172.21.15.12:0/2926915733 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25000b8d0 con 0x7fa258161590 2024-03-29T11:32:06.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.053+0000 7fa254a00700 1 -- 172.21.15.12:0/2926915733 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000bc80 con 0x7fa258161590 2024-03-29T11:32:06.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.053+0000 7fa25d200700 1 -- 172.21.15.12:0/2926915733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258161590 msgr2=0x7fa258161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:06.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.053+0000 7fa25d200700 1 --2- 172.21.15.12:0/2926915733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258161590 0x7fa258161a20 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fa250009e80 tx=0x7fa25000b330 comp rx=0 tx=0).stop 2024-03-29T11:32:06.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.054+0000 7fa25d200700 1 -- 172.21.15.12:0/2926915733 shutdown_connections 2024-03-29T11:32:06.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.054+0000 7fa25d200700 1 --2- 172.21.15.12:0/2926915733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258161590 0x7fa258161a20 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.054+0000 7fa25d200700 1 --2- 172.21.15.12:0/2926915733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2580d18b0 0x7fa2580cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.054+0000 7fa25d200700 1 -- 172.21.15.12:0/2926915733 >> 172.21.15.12:0/2926915733 conn(0x7fa25808aff0 msgr2=0x7fa2580d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:06.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.054+0000 7fa25d200700 1 -- 172.21.15.12:0/2926915733 shutdown_connections 2024-03-29T11:32:06.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.054+0000 7fa25d200700 1 -- 172.21.15.12:0/2926915733 wait complete. 2024-03-29T11:32:06.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.056+0000 7fa25d200700 1 Processor -- start 2024-03-29T11:32:06.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.057+0000 7fa25d200700 1 -- start start 2024-03-29T11:32:06.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.058+0000 7fa25d200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2580d18b0 0x7fa2580b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.058+0000 7fa25d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258160f10 0x7fa25815da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.058+0000 7fa25d200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa258161590 0x7fa258158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.058+0000 7fa25d200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580a0d40 con 0x7fa258160f10 2024-03-29T11:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.058+0000 7fa25d200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25815df70 con 0x7fa2580d18b0 2024-03-29T11:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.058+0000 7fa25d200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25815e240 con 0x7fa258161590 2024-03-29T11:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa255400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258160f10 0x7fa25815da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa255e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2580d18b0 0x7fa2580b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa255400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258160f10 0x7fa25815da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60098/0 (socket says 172.21.15.12:60098) 2024-03-29T11:32:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa255400700 1 -- 172.21.15.12:0/2441897249 learned_addr learned my addr 172.21.15.12:0/2441897249 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa256800700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa258161590 0x7fa258158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa255400700 1 -- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa258161590 msgr2=0x7fa258158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa255400700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa258161590 0x7fa258158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa255400700 1 -- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2580d18b0 msgr2=0x7fa2580b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa255400700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2580d18b0 0x7fa2580b7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa255400700 1 -- 172.21.15.12:0/2441897249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa250009b30 con 0x7fa258160f10 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.059+0000 7fa255400700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258160f10 0x7fa25815da30 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fa24400e870 tx=0x7fa24400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.060+0000 7fa24e000700 1 -- 172.21.15.12:0/2441897249 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244009590 con 0x7fa258160f10 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.060+0000 7fa24e000700 1 -- 172.21.15.12:0/2441897249 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa244013680 con 0x7fa258160f10 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.060+0000 7fa25d200700 1 -- 172.21.15.12:0/2441897249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25815e520 con 0x7fa258160f10 2024-03-29T11:32:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.060+0000 7fa24e000700 1 -- 172.21.15.12:0/2441897249 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2440139f0 con 0x7fa258160f10 2024-03-29T11:32:06.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.060+0000 7fa25d200700 1 -- 172.21.15.12:0/2441897249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25815eae0 con 0x7fa258160f10 2024-03-29T11:32:06.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.061+0000 7fa25d200700 1 -- 172.21.15.12:0/2441897249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2580ce120 con 0x7fa258160f10 2024-03-29T11:32:06.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.062+0000 7fa24e000700 1 -- 172.21.15.12:0/2441897249 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fa244013b50 con 0x7fa258160f10 2024-03-29T11:32:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.064+0000 7fa24e000700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa23806af60 0x7fa23806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.064+0000 7fa24e000700 1 -- 172.21.15.12:0/2441897249 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7fa244089ba0 con 0x7fa258160f10 2024-03-29T11:32:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.064+0000 7fa255e00700 1 -- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa23806af60 msgr2=0x7fa23806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.064+0000 7fa255e00700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa23806af60 0x7fa23806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:06.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.067+0000 7fa24e000700 1 -- 172.21.15.12:0/2441897249 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2440544f0 con 0x7fa258160f10 2024-03-29T11:32:06.067 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:06.073+0000 7fe61ff2c240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:32:06.214 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:06.219+0000 7f2910d5b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:32:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.265+0000 7fa255e00700 1 -- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa23806af60 msgr2=0x7fa23806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.265+0000 7fa255e00700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa23806af60 0x7fa23806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:06.313 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:06.317+0000 7f2910d5b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:32:06.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.330+0000 7fa25d200700 1 -- 172.21.15.12:0/2441897249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2580cd930 con 0x7fa258160f10 2024-03-29T11:32:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.331+0000 7fa24e000700 1 -- 172.21.15.12:0/2441897249 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5971 (secure 0 0 0) 0x7fa244057b10 con 0x7fa258160f10 2024-03-29T11:32:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:06.327 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":112,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:26.816217+0000","modified":"2024-03-29T11:31:30.290952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/2722218561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2722218561},{"type":"v1","addr":"172.21.15.12:6833","nonce":2722218561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:40.863222+0000","modified":"2024-03-29T11:31:43.883199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-29T11:32:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 -- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa23806af60 msgr2=0x7fa23806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa23806af60 0x7fa23806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 -- 172.21.15.12:0/2441897249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258160f10 msgr2=0x7fa25815da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258160f10 0x7fa25815da30 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fa24400e870 tx=0x7fa24400eb80 comp rx=0 tx=0).stop 2024-03-29T11:32:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 -- 172.21.15.12:0/2441897249 shutdown_connections 2024-03-29T11:32:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa23806af60 0x7fa23806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa258161590 0x7fa258158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2580d18b0 0x7fa2580b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 --2- 172.21.15.12:0/2441897249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa258160f10 0x7fa25815da30 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 -- 172.21.15.12:0/2441897249 >> 172.21.15.12:0/2441897249 conn(0x7fa25808aff0 msgr2=0x7fa25815b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.333+0000 7fa25d200700 1 -- 172.21.15.12:0/2441897249 shutdown_connections 2024-03-29T11:32:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.334+0000 7fa25d200700 1 -- 172.21.15.12:0/2441897249 wait complete. 2024-03-29T11:32:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 112 2024-03-29T11:32:06.349 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:06.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.553+0000 7fe59e400700 1 Processor -- start 2024-03-29T11:32:06.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.553+0000 7fe59e400700 1 -- start start 2024-03-29T11:32:06.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.554+0000 7fe59e400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe59809a7c0 0x7fe59806c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.554+0000 7fe59e400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe59806c7f0 0x7fe59815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.554+0000 7fe59e400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5980cf550 con 0x7fe5980cd350 2024-03-29T11:32:06.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.554+0000 7fe59e400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59806cc80 con 0x7fe59809a7c0 2024-03-29T11:32:06.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.554+0000 7fe59e400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5980d0390 con 0x7fe59806c7f0 2024-03-29T11:32:06.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.554+0000 7fe596a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe59809a7c0 0x7fe59806c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:06.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.554+0000 7fe597400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe5980cd350 0x7fe59806ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43174/0 (socket says 172.21.15.12:43174) 2024-03-29T11:32:06.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.554+0000 7fe597400700 1 -- 172.21.15.12:0/2301533832 learned_addr learned my addr 172.21.15.12:0/2301533832 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.555+0000 7fe597e00700 1 --2- 172.21.15.12:0/2301533832 >> v2:172.21.15.159:3300/0 conn(0x7fe59806c7f0 0x7fe59815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.555+0000 7fe596a00700 1 -- 172.21.15.12:0/2301533832 >> v2:172.21.15.159:3300/0 conn(0x7fe59806c7f0 msgr2=0x7fe59815cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.555+0000 7fe596a00700 1 --2- 172.21.15.12:0/2301533832 >> v2:172.21.15.159:3300/0 conn(0x7fe59806c7f0 0x7fe59815cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.555+0000 7fe596a00700 1 -- 172.21.15.12:0/2301533832 >> v1:172.21.15.12:6789/0 conn(0x7fe5980cd350 legacy=0x7fe59806ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.555+0000 7fe596a00700 1 -- 172.21.15.12:0/2301533832 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5980cc970 con 0x7fe59809a7c0 2024-03-29T11:32:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.555+0000 7fe596000700 1 -- 172.21.15.12:0/2301533832 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410301632 0 0) 0x7fe5980cf550 con 0x7fe5980cd350 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.555+0000 7fe597e00700 1 --2- 172.21.15.12:0/2301533832 >> v2:172.21.15.159:3300/0 conn(0x7fe59806c7f0 0x7fe59815cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.555+0000 7fe596a00700 1 --2- 172.21.15.12:0/2301533832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59809a7c0 0x7fe59806c2b0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fe5980d0300 tx=0x7fe584009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1677710b1d5fa6e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe596000700 1 -- 172.21.15.12:0/2301533832 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584011830 con 0x7fe59809a7c0 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe596000700 1 -- 172.21.15.12:0/2301533832 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe584011990 con 0x7fe59809a7c0 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe596000700 1 -- 172.21.15.12:0/2301533832 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584011d20 con 0x7fe59809a7c0 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe59e400700 1 -- 172.21.15.12:0/2301533832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59809a7c0 msgr2=0x7fe59806c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe59e400700 1 --2- 172.21.15.12:0/2301533832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59809a7c0 0x7fe59806c2b0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fe5980d0300 tx=0x7fe584009b20 comp rx=0 tx=0).stop 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe59e400700 1 -- 172.21.15.12:0/2301533832 shutdown_connections 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe59e400700 1 --2- 172.21.15.12:0/2301533832 >> v2:172.21.15.159:3300/0 conn(0x7fe59806c7f0 0x7fe59815cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe59e400700 1 --2- 172.21.15.12:0/2301533832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe59809a7c0 0x7fe59806c2b0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe59e400700 1 -- 172.21.15.12:0/2301533832 >> 172.21.15.12:0/2301533832 conn(0x7fe5980d1310 msgr2=0x7fe5980d1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe59e400700 1 -- 172.21.15.12:0/2301533832 shutdown_connections 2024-03-29T11:32:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.556+0000 7fe59e400700 1 -- 172.21.15.12:0/2301533832 wait complete. 2024-03-29T11:32:06.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.558+0000 7fe59e400700 1 Processor -- start 2024-03-29T11:32:06.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.558+0000 7fe59e400700 1 -- start start 2024-03-29T11:32:06.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.559+0000 7fe59e400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe59806c7f0 0x7fe59814b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.559+0000 7fe59e400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59809a7c0 0x7fe59814b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.559+0000 7fe59e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5980cd350 0x7fe59814bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.559+0000 7fe59e400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59815dab0 con 0x7fe5980cd350 2024-03-29T11:32:06.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.559+0000 7fe59e400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59814ae70 con 0x7fe59809a7c0 2024-03-29T11:32:06.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.559+0000 7fe59e400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59814a540 con 0x7fe59806c7f0 2024-03-29T11:32:06.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe597e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5980cd350 0x7fe59814bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:06.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe596a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59809a7c0 0x7fe59814b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:06.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe597e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5980cd350 0x7fe59814bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60112/0 (socket says 172.21.15.12:60112) 2024-03-29T11:32:06.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe597e00700 1 -- 172.21.15.12:0/2967590949 learned_addr learned my addr 172.21.15.12:0/2967590949 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:06.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe597400700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe59806c7f0 0x7fe59814b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:06.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe597e00700 1 -- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe59806c7f0 msgr2=0x7fe59814b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:06.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe597e00700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe59806c7f0 0x7fe59814b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe597e00700 1 -- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59809a7c0 msgr2=0x7fe59814b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:06.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe597e00700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59809a7c0 0x7fe59814b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe597e00700 1 -- 172.21.15.12:0/2967590949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5840094a0 con 0x7fe5980cd350 2024-03-29T11:32:06.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.560+0000 7fe596a00700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59809a7c0 0x7fe59814b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:06.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.561+0000 7fe597e00700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5980cd350 0x7fe59814bd80 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fe59000cd50 tx=0x7fe59000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:06.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.561+0000 7fe58f400700 1 -- 172.21.15.12:0/2967590949 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59000cea0 con 0x7fe5980cd350 2024-03-29T11:32:06.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.561+0000 7fe58f400700 1 -- 172.21.15.12:0/2967590949 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59000f480 con 0x7fe5980cd350 2024-03-29T11:32:06.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.561+0000 7fe58f400700 1 -- 172.21.15.12:0/2967590949 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59000f830 con 0x7fe5980cd350 2024-03-29T11:32:06.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.561+0000 7fe59e400700 1 -- 172.21.15.12:0/2967590949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59814a820 con 0x7fe5980cd350 2024-03-29T11:32:06.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.561+0000 7fe59e400700 1 -- 172.21.15.12:0/2967590949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59814c2c0 con 0x7fe5980cd350 2024-03-29T11:32:06.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.562+0000 7fe59e400700 1 -- 172.21.15.12:0/2967590949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59814afe0 con 0x7fe5980cd350 2024-03-29T11:32:06.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.563+0000 7fe58f400700 1 -- 172.21.15.12:0/2967590949 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fe59000b040 con 0x7fe5980cd350 2024-03-29T11:32:06.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.564+0000 7fe58f400700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57806af60 0x7fe57806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:06.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.564+0000 7fe58f400700 1 -- 172.21.15.12:0/2967590949 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7fe59008a4e0 con 0x7fe5980cd350 2024-03-29T11:32:06.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.564+0000 7fe597400700 1 -- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57806af60 msgr2=0x7fe57806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:06.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.564+0000 7fe597400700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57806af60 0x7fe57806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:06.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.573+0000 7fe58f400700 1 -- 172.21.15.12:0/2967590949 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe590054ca0 con 0x7fe5980cd350 2024-03-29T11:32:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.765+0000 7fe597400700 1 -- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57806af60 msgr2=0x7fe57806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.765+0000 7fe597400700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57806af60 0x7fe57806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:06.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.793+0000 7fe59e400700 1 -- 172.21.15.12:0/2967590949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe598067720 con 0x7fe5980cd350 2024-03-29T11:32:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.796+0000 7fe58f400700 1 -- 172.21.15.12:0/2967590949 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+20451 (secure 0 0 0) 0x7fe5900582c0 con 0x7fe5980cd350 2024-03-29T11:32:06.792 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:06.792 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":232,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:31:55.626816+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:31:39.075407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:31:39.845704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/1069287393":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/353804238":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6833/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6832/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/235474344":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3837208470":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/1750949668":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/629137162":"2024-03-30T11:31:55.626759+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 -- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57806af60 msgr2=0x7fe57806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57806af60 0x7fe57806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 -- 172.21.15.12:0/2967590949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5980cd350 msgr2=0x7fe59814bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5980cd350 0x7fe59814bd80 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fe59000cd50 tx=0x7fe59000a560 comp rx=0 tx=0).stop 2024-03-29T11:32:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 -- 172.21.15.12:0/2967590949 shutdown_connections 2024-03-29T11:32:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe57806af60 0x7fe57806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5980cd350 0x7fe59814bd80 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe59806c7f0 0x7fe59814b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 --2- 172.21.15.12:0/2967590949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe59809a7c0 0x7fe59814b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 -- 172.21.15.12:0/2967590949 >> 172.21.15.12:0/2967590949 conn(0x7fe5980d1310 msgr2=0x7fe5980cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 -- 172.21.15.12:0/2967590949 shutdown_connections 2024-03-29T11:32:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:06.799+0000 7fe59e400700 1 -- 172.21.15.12:0/2967590949 wait complete. 2024-03-29T11:32:06.808 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:06.814+0000 7fe61ff2c240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:32:06.813 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:31:24.485419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:31:25.823318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-29T11:31:39.075407+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-29T11:31:39.845704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:32:06.814 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:32:07.026 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:07.032+0000 7fe61ff2c240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:32:07.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.033+0000 7f0d0d800700 1 Processor -- start 2024-03-29T11:32:07.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.034+0000 7f0d0d800700 1 -- start start 2024-03-29T11:32:07.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.035+0000 7f0d0d800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d08160d70 0x7f0d080b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.035+0000 7f0d0d800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d08161150 con 0x7f0d0815ef70 2024-03-29T11:32:07.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.035+0000 7f0d0d800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0806dba0 con 0x7f0d0806a790 2024-03-29T11:32:07.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.035+0000 7f0d0d800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080b4cd0 con 0x7f0d08160d70 2024-03-29T11:32:07.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.035+0000 7f0d06000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0d0815ef70 0x7f0d0815e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43184/0 (socket says 172.21.15.12:43184) 2024-03-29T11:32:07.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.035+0000 7f0d06000700 1 -- 172.21.15.12:0/3800469 learned_addr learned my addr 172.21.15.12:0/3800469 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:07.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.035+0000 7f0d06a00700 1 --2- 172.21.15.12:0/3800469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d08160d70 0x7f0d080b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.036+0000 7f0d05600700 1 -- 172.21.15.12:0/3800469 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930991979 0 0) 0x7f0d08161150 con 0x7f0d0815ef70 2024-03-29T11:32:07.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.036+0000 7f0d05600700 1 -- 172.21.15.12:0/3800469 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cec0035b0 con 0x7f0d0815ef70 2024-03-29T11:32:07.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.036+0000 7f0d05600700 1 -- 172.21.15.12:0/3800469 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117729657 0 0) 0x7f0d0806dba0 con 0x7f0d0806a790 2024-03-29T11:32:07.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.036+0000 7f0d05600700 1 -- 172.21.15.12:0/3800469 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d08161150 con 0x7f0d0806a790 2024-03-29T11:32:07.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.036+0000 7f0d06a00700 1 -- 172.21.15.12:0/3800469 >> v1:172.21.15.159:6789/0 conn(0x7f0d0806a790 legacy=0x7f0d0806e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.036+0000 7f0d06a00700 1 -- 172.21.15.12:0/3800469 >> v1:172.21.15.12:6789/0 conn(0x7f0d0815ef70 legacy=0x7f0d0815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.036+0000 7f0d06a00700 1 -- 172.21.15.12:0/3800469 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080b4ee0 con 0x7f0d08160d70 2024-03-29T11:32:07.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.036+0000 7f0d06a00700 1 --2- 172.21.15.12:0/3800469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d08160d70 0x7f0d080b3200 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8009870 tx=0x7f0cf8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e68c47d16d50b5ae server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:07.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.037+0000 7f0d05600700 1 -- 172.21.15.12:0/3800469 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8011830 con 0x7f0d08160d70 2024-03-29T11:32:07.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.037+0000 7f0d05600700 1 -- 172.21.15.12:0/3800469 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf8011990 con 0x7f0d08160d70 2024-03-29T11:32:07.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.037+0000 7f0d05600700 1 -- 172.21.15.12:0/3800469 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8011d20 con 0x7f0d08160d70 2024-03-29T11:32:07.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.037+0000 7f0d0d800700 1 -- 172.21.15.12:0/3800469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d08160d70 msgr2=0x7f0d080b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.037+0000 7f0d0d800700 1 --2- 172.21.15.12:0/3800469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d08160d70 0x7f0d080b3200 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8009870 tx=0x7f0cf8009b80 comp rx=0 tx=0).stop 2024-03-29T11:32:07.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.038+0000 7f0d0d800700 1 -- 172.21.15.12:0/3800469 shutdown_connections 2024-03-29T11:32:07.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.038+0000 7f0d0d800700 1 --2- 172.21.15.12:0/3800469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d08160d70 0x7f0d080b3200 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.038+0000 7f0d0d800700 1 -- 172.21.15.12:0/3800469 >> 172.21.15.12:0/3800469 conn(0x7f0d080b5720 msgr2=0x7f0d080b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:07.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.038+0000 7f0d0d800700 1 -- 172.21.15.12:0/3800469 shutdown_connections 2024-03-29T11:32:07.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.038+0000 7f0d0d800700 1 -- 172.21.15.12:0/3800469 wait complete. 2024-03-29T11:32:07.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.039+0000 7f0d0d800700 1 Processor -- start 2024-03-29T11:32:07.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.040+0000 7f0d0d800700 1 -- start start 2024-03-29T11:32:07.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d0d800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d0806a790 0x7f0d081328c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d0d800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0815ef70 0x7f0d080afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d0d800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d08160d70 0x7f0d080b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d0d800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08134000 con 0x7f0d0806a790 2024-03-29T11:32:07.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d0d800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08132e00 con 0x7f0d08160d70 2024-03-29T11:32:07.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d0d800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080b00d0 con 0x7f0d0815ef70 2024-03-29T11:32:07.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d06a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d0806a790 0x7f0d081328c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d06a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d0806a790 0x7f0d081328c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60118/0 (socket says 172.21.15.12:60118) 2024-03-29T11:32:07.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d06a00700 1 -- 172.21.15.12:0/1772126337 learned_addr learned my addr 172.21.15.12:0/1772126337 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:07.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d06000700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0815ef70 0x7f0d080afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d07400700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d08160d70 0x7f0d080b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d06a00700 1 -- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0815ef70 msgr2=0x7f0d080afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d06a00700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0815ef70 0x7f0d080afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d06a00700 1 -- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d08160d70 msgr2=0x7f0d080b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d06a00700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d08160d70 0x7f0d080b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d06a00700 1 -- 172.21.15.12:0/1772126337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80094a0 con 0x7f0d0806a790 2024-03-29T11:32:07.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.041+0000 7f0d07400700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d08160d70 0x7f0d080b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:07.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.042+0000 7f0d06a00700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d0806a790 0x7f0d081328c0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80090b0 tx=0x7f0cf8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:07.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.042+0000 7f0cfea00700 1 -- 172.21.15.12:0/1772126337 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf801a6e0 con 0x7f0d0806a790 2024-03-29T11:32:07.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.042+0000 7f0cfea00700 1 -- 172.21.15.12:0/1772126337 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf801a840 con 0x7f0d0806a790 2024-03-29T11:32:07.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.042+0000 7f0d0d800700 1 -- 172.21.15.12:0/1772126337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d08133110 con 0x7f0d0806a790 2024-03-29T11:32:07.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.042+0000 7f0cfea00700 1 -- 172.21.15.12:0/1772126337 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf801abf0 con 0x7f0d0806a790 2024-03-29T11:32:07.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.042+0000 7f0d0d800700 1 -- 172.21.15.12:0/1772126337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d081336d0 con 0x7f0d0806a790 2024-03-29T11:32:07.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.043+0000 7f0d0d800700 1 -- 172.21.15.12:0/1772126337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d08067720 con 0x7f0d0806a790 2024-03-29T11:32:07.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.044+0000 7f0cfea00700 1 -- 172.21.15.12:0/1772126337 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f0cf8022070 con 0x7f0d0806a790 2024-03-29T11:32:07.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.045+0000 7f0cfea00700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ce806af60 0x7f0ce806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.045+0000 7f0cfea00700 1 -- 172.21.15.12:0/1772126337 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f0cf808a890 con 0x7f0d0806a790 2024-03-29T11:32:07.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.045+0000 7f0d06000700 1 -- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ce806af60 msgr2=0x7f0ce806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:07.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.045+0000 7f0d06000700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ce806af60 0x7f0ce806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:07.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.050+0000 7f0cfea00700 1 -- 172.21.15.12:0/1772126337 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0cf80551e0 con 0x7f0d0806a790 2024-03-29T11:32:07.113 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:07.118+0000 7f2910d5b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:32:07.238 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:07.244+0000 7fe61ff2c240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:32:07.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.246+0000 7f0d06000700 1 -- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ce806af60 msgr2=0x7f0ce806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:07.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.246+0000 7f0d06000700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ce806af60 0x7f0ce806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.266+0000 7f0d0d800700 1 -- 172.21.15.12:0/1772126337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d0809a400 con 0x7f0d0806a790 2024-03-29T11:32:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.266+0000 7f0cfea00700 1 -- 172.21.15.12:0/1772126337 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v232) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0cf8058800 con 0x7f0d0806a790 2024-03-29T11:32:07.263 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:32:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 -- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ce806af60 msgr2=0x7f0ce806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ce806af60 0x7f0ce806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 -- 172.21.15.12:0/1772126337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d0806a790 msgr2=0x7f0d081328c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d0806a790 0x7f0d081328c0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80090b0 tx=0x7f0cf8004750 comp rx=0 tx=0).stop 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 -- 172.21.15.12:0/1772126337 shutdown_connections 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ce806af60 0x7f0ce806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d08160d70 0x7f0d080b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d0806a790 0x7f0d081328c0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 --2- 172.21.15.12:0/1772126337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0815ef70 0x7f0d080afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 -- 172.21.15.12:0/1772126337 >> 172.21.15.12:0/1772126337 conn(0x7f0d080b5720 msgr2=0x7f0d08144de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 -- 172.21.15.12:0/1772126337 shutdown_connections 2024-03-29T11:32:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.269+0000 7f0d0d800700 1 -- 172.21.15.12:0/1772126337 wait complete. 2024-03-29T11:32:07.284 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:32:07.339 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:07.343+0000 7f2910d5b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:32:07.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.495+0000 7f169a000700 1 Processor -- start 2024-03-29T11:32:07.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.495+0000 7f169a000700 1 -- start start 2024-03-29T11:32:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.496+0000 7f169a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f169415dad0 0x7f16940ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.496+0000 7f169a000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f16940d1630 0x7f16940d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.496+0000 7f169a000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169415f820 con 0x7f16940d1fd0 2024-03-29T11:32:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.496+0000 7f169a000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16940cfcc0 con 0x7f16940d1630 2024-03-29T11:32:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.496+0000 7f169a000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16940cd9f0 con 0x7f169415dad0 2024-03-29T11:32:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.497+0000 7f1692a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f16940d1630 0x7f16940d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.497+0000 7f1693400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f169415dad0 0x7f16940ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.497+0000 7f1692a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f16940d1630 0x7f16940d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:48952/0 (socket says 172.21.15.12:48952) 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.497+0000 7f1692a00700 1 -- 172.21.15.12:0/3841578052 learned_addr learned my addr 172.21.15.12:0/3841578052 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.497+0000 7f1693e00700 1 --1- 172.21.15.12:0/3841578052 >> v1:172.21.15.159:6789/0 conn(0x7f16940d1fd0 0x7f169415deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:59148/0 (socket says 172.21.15.12:59148) 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.497+0000 7f1692a00700 1 -- 172.21.15.12:0/3841578052 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f169415dad0 msgr2=0x7f16940ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.497+0000 7f1692a00700 1 --2- 172.21.15.12:0/3841578052 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f169415dad0 0x7f16940ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.497+0000 7f1692a00700 1 -- 172.21.15.12:0/3841578052 >> v1:172.21.15.159:6789/0 conn(0x7f16940d1fd0 legacy=0x7f169415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.497+0000 7f1692a00700 1 -- 172.21.15.12:0/3841578052 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16940d2460 con 0x7f16940d1630 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.498+0000 7f1692a00700 1 --2- 172.21.15.12:0/3841578052 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16940d1630 0x7f16940d1a90 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f16800098f0 tx=0x7f1680009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f445e95d62a41a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.498+0000 7f1692000700 1 -- 172.21.15.12:0/3841578052 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680011830 con 0x7f16940d1630 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.498+0000 7f1692000700 1 -- 172.21.15.12:0/3841578052 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1680011990 con 0x7f16940d1630 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.498+0000 7f1692000700 1 -- 172.21.15.12:0/3841578052 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680011d20 con 0x7f16940d1630 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.498+0000 7f169a000700 1 -- 172.21.15.12:0/3841578052 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16940d1630 msgr2=0x7f16940d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.498+0000 7f169a000700 1 --2- 172.21.15.12:0/3841578052 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16940d1630 0x7f16940d1a90 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f16800098f0 tx=0x7f1680009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.499+0000 7f169a000700 1 -- 172.21.15.12:0/3841578052 shutdown_connections 2024-03-29T11:32:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.499+0000 7f169a000700 1 --2- 172.21.15.12:0/3841578052 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f169415dad0 0x7f16940ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.499+0000 7f169a000700 1 --2- 172.21.15.12:0/3841578052 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16940d1630 0x7f16940d1a90 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.499+0000 7f169a000700 1 -- 172.21.15.12:0/3841578052 >> 172.21.15.12:0/3841578052 conn(0x7f169408aff0 msgr2=0x7f16940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:07.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.499+0000 7f169a000700 1 -- 172.21.15.12:0/3841578052 shutdown_connections 2024-03-29T11:32:07.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.499+0000 7f169a000700 1 -- 172.21.15.12:0/3841578052 wait complete. 2024-03-29T11:32:07.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.500+0000 7f169a000700 1 Processor -- start 2024-03-29T11:32:07.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.501+0000 7f169a000700 1 -- start start 2024-03-29T11:32:07.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.501+0000 7f169a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16940d1630 0x7f16941548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f169a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16940d1fd0 0x7f16940b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f169a000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f169415dad0 0x7f169415e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f169a000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16940b8240 con 0x7f16940d1630 2024-03-29T11:32:07.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f169a000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16940b8470 con 0x7f169415dad0 2024-03-29T11:32:07.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f169a000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16940b8750 con 0x7f16940d1fd0 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f1693400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16940d1630 0x7f16941548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f1693400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16940d1630 0x7f16941548b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60134/0 (socket says 172.21.15.12:60134) 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f1693400700 1 -- 172.21.15.12:0/826240645 learned_addr learned my addr 172.21.15.12:0/826240645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f1692a00700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16940d1fd0 0x7f16940b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f1693e00700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f169415dad0 0x7f169415e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f1693400700 1 -- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16940d1fd0 msgr2=0x7f16940b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f1693400700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16940d1fd0 0x7f16940b7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f1693400700 1 -- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f169415dad0 msgr2=0x7f169415e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f1693400700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f169415dad0 0x7f169415e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.502+0000 7f1693400700 1 -- 172.21.15.12:0/826240645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680009550 con 0x7f16940d1630 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.503+0000 7f1693400700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16940d1630 0x7f16941548b0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f168400e870 tx=0x7f168400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.503+0000 7f168b400700 1 -- 172.21.15.12:0/826240645 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684009590 con 0x7f16940d1630 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.503+0000 7f168b400700 1 -- 172.21.15.12:0/826240645 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684013680 con 0x7f16940d1630 2024-03-29T11:32:07.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.503+0000 7f169a000700 1 -- 172.21.15.12:0/826240645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16940c71f0 con 0x7f16940d1630 2024-03-29T11:32:07.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.503+0000 7f168b400700 1 -- 172.21.15.12:0/826240645 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16840139f0 con 0x7f16940d1630 2024-03-29T11:32:07.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.503+0000 7f169a000700 1 -- 172.21.15.12:0/826240645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169415eb70 con 0x7f16940d1630 2024-03-29T11:32:07.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.504+0000 7f169a000700 1 -- 172.21.15.12:0/826240645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16940c7050 con 0x7f16940d1630 2024-03-29T11:32:07.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.505+0000 7f168b400700 1 -- 172.21.15.12:0/826240645 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f1684013b50 con 0x7f16940d1630 2024-03-29T11:32:07.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.506+0000 7f168b400700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f167406af60 0x7f167406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.506+0000 7f168b400700 1 -- 172.21.15.12:0/826240645 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7f1684089b60 con 0x7f16940d1630 2024-03-29T11:32:07.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.506+0000 7f1692a00700 1 -- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f167406af60 msgr2=0x7f167406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:07.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.506+0000 7f1692a00700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f167406af60 0x7f167406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:07.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.511+0000 7f168b400700 1 -- 172.21.15.12:0/826240645 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f16940c7050 con 0x7f16940d1630 2024-03-29T11:32:07.562 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:07.566+0000 7f2910d5b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:32:07.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.707+0000 7f1692a00700 1 -- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f167406af60 msgr2=0x7f167406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:07.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.707+0000 7f1692a00700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f167406af60 0x7f167406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:07.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.723+0000 7f169a000700 1 -- 172.21.15.12:0/826240645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f16940c7050 con 0x7f16940d1630 2024-03-29T11:32:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.724+0000 7f168b400700 1 -- 172.21.15.12:0/826240645 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 104+0+11 (secure 0 0 0) 0x7f16940c7050 con 0x7f16940d1630 2024-03-29T11:32:07.720 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:32:07.720 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:07.726+0000 7fe61ff2c240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:32:07.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 -- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f167406af60 msgr2=0x7f167406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:07.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f167406af60 0x7f167406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 -- 172.21.15.12:0/826240645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16940d1630 msgr2=0x7f16941548b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16940d1630 0x7f16941548b0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f168400e870 tx=0x7f168400eb80 comp rx=0 tx=0).stop 2024-03-29T11:32:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 -- 172.21.15.12:0/826240645 shutdown_connections 2024-03-29T11:32:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f167406af60 0x7f167406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f169415dad0 0x7f169415e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16940d1630 0x7f16941548b0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 --2- 172.21.15.12:0/826240645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16940d1fd0 0x7f16940b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 -- 172.21.15.12:0/826240645 >> 172.21.15.12:0/826240645 conn(0x7f169408aff0 msgr2=0x7f169406b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 -- 172.21.15.12:0/826240645 shutdown_connections 2024-03-29T11:32:07.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.726+0000 7f169a000700 1 -- 172.21.15.12:0/826240645 wait complete. 2024-03-29T11:32:07.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:32:07.930 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:07.936+0000 7fe61ff2c240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:32:07.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.949+0000 7ff9dc400700 1 Processor -- start 2024-03-29T11:32:07.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.949+0000 7ff9dc400700 1 -- start start 2024-03-29T11:32:07.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.950+0000 7ff9dc400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff9d40d3eb0 0x7ff9d40b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.950+0000 7ff9dc400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d40ced20 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.950+0000 7ff9dc400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d40d4340 con 0x7ff9d415d8a0 2024-03-29T11:32:07.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.950+0000 7ff9dc400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d40ce6d0 con 0x7ff9d40d3eb0 2024-03-29T11:32:07.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.950+0000 7ff9d9000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff9d40d3eb0 0x7ff9d40b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.950+0000 7ff9da400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff9d40cf1b0 0x7ff9d40d1b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43198/0 (socket says 172.21.15.12:43198) 2024-03-29T11:32:07.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.950+0000 7ff9da400700 1 -- 172.21.15.12:0/3205570967 learned_addr learned my addr 172.21.15.12:0/3205570967 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.951+0000 7ff9d9000700 1 -- 172.21.15.12:0/3205570967 >> v1:172.21.15.159:6789/0 conn(0x7ff9d415d8a0 legacy=0x7ff9d406b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.951+0000 7ff9d3e00700 1 -- 172.21.15.12:0/3205570967 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496850196 0 0) 0x7ff9d40ced20 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.951+0000 7ff9d9000700 1 -- 172.21.15.12:0/3205570967 >> v1:172.21.15.12:6789/0 conn(0x7ff9d40cf1b0 legacy=0x7ff9d40d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.951+0000 7ff9d9000700 1 -- 172.21.15.12:0/3205570967 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d40ce9b0 con 0x7ff9d40d3eb0 2024-03-29T11:32:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.951+0000 7ff9d9000700 1 --2- 172.21.15.12:0/3205570967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d40d3eb0 0x7ff9d40b2990 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009820 tx=0x7ff9c0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb5b21ce4ac1ea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.951+0000 7ff9d3e00700 1 -- 172.21.15.12:0/3205570967 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0005e30 con 0x7ff9d40d3eb0 2024-03-29T11:32:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.951+0000 7ff9d3e00700 1 -- 172.21.15.12:0/3205570967 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c0006030 con 0x7ff9d40d3eb0 2024-03-29T11:32:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.951+0000 7ff9d3e00700 1 -- 172.21.15.12:0/3205570967 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c00063e0 con 0x7ff9d40d3eb0 2024-03-29T11:32:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.952+0000 7ff9dc400700 1 -- 172.21.15.12:0/3205570967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d40d3eb0 msgr2=0x7ff9d40b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.952+0000 7ff9dc400700 1 --2- 172.21.15.12:0/3205570967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d40d3eb0 0x7ff9d40b2990 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009820 tx=0x7ff9c0009b30 comp rx=0 tx=0).stop 2024-03-29T11:32:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.952+0000 7ff9dc400700 1 -- 172.21.15.12:0/3205570967 shutdown_connections 2024-03-29T11:32:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.952+0000 7ff9dc400700 1 --2- 172.21.15.12:0/3205570967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d40d3eb0 0x7ff9d40b2990 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.952+0000 7ff9dc400700 1 -- 172.21.15.12:0/3205570967 >> 172.21.15.12:0/3205570967 conn(0x7ff9d408aff0 msgr2=0x7ff9d40d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.952+0000 7ff9dc400700 1 -- 172.21.15.12:0/3205570967 shutdown_connections 2024-03-29T11:32:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.952+0000 7ff9dc400700 1 -- 172.21.15.12:0/3205570967 wait complete. 2024-03-29T11:32:07.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.954+0000 7ff9dc400700 1 Processor -- start 2024-03-29T11:32:07.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.954+0000 7ff9dc400700 1 -- start start 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9dc400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9d40cf1b0 0x7ff9d40b5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9dc400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d415d8a0 0x7ff9d41f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9dc400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9d40cc8a0 0x7ff9d40b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9dc400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40b5ed0 con 0x7ff9d415d8a0 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9dc400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40b6c20 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9dc400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40b6f00 con 0x7ff9d40cc8a0 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9d9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d415d8a0 0x7ff9d41f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9d9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d415d8a0 0x7ff9d41f3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60152/0 (socket says 172.21.15.12:60152) 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9d9000700 1 -- 172.21.15.12:0/2158219773 learned_addr learned my addr 172.21.15.12:0/2158219773 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9d9a00700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9d40cf1b0 0x7ff9d40b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.955+0000 7ff9da400700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9d40cc8a0 0x7ff9d40b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9d9a00700 1 -- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9d40cc8a0 msgr2=0x7ff9d40b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9d9a00700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9d40cc8a0 0x7ff9d40b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9d9a00700 1 -- 172.21.15.12:0/2158219773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d415d8a0 msgr2=0x7ff9d41f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:07.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9d9a00700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d415d8a0 0x7ff9d41f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:07.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9d9a00700 1 -- 172.21.15.12:0/2158219773 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c0009430 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9d9000700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d415d8a0 0x7ff9d41f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:07.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9d9a00700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9d40cf1b0 0x7ff9d40b5990 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8000f60 tx=0x7ff9c800c730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9d1600700 1 -- 172.21.15.12:0/2158219773 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80052d0 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9d1600700 1 -- 172.21.15.12:0/2158219773 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8005430 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9dc400700 1 -- 172.21.15.12:0/2158219773 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d40b71e0 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.956+0000 7ff9d1600700 1 -- 172.21.15.12:0/2158219773 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80038c0 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.957+0000 7ff9dc400700 1 -- 172.21.15.12:0/2158219773 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d40b77a0 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.958+0000 7ff9dc400700 1 -- 172.21.15.12:0/2158219773 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d406bf80 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.958+0000 7ff9d1600700 1 -- 172.21.15.12:0/2158219773 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7ff9c8003a20 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.959+0000 7ff9d1600700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9b406af60 0x7ff9b406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:07.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.959+0000 7ff9d1600700 1 -- 172.21.15.12:0/2158219773 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7ff9c8089000 con 0x7ff9d40cf1b0 2024-03-29T11:32:07.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.960+0000 7ff9d9000700 1 -- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9b406af60 msgr2=0x7ff9b406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:07.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.960+0000 7ff9d9000700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9b406af60 0x7ff9b406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:07.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:07.964+0000 7ff9d1600700 1 -- 172.21.15.12:0/2158219773 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff9d406bf80 con 0x7ff9d40cf1b0 2024-03-29T11:32:08.080 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:08.084+0000 7f2910d5b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:32:08.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.161+0000 7ff9d9000700 1 -- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9b406af60 msgr2=0x7ff9b406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:08.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.161+0000 7ff9d9000700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9b406af60 0x7ff9b406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.178+0000 7ff9dc400700 1 -- 172.21.15.12:0/2158219773 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff9d406bf80 con 0x7ff9d40cf1b0 2024-03-29T11:32:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.178+0000 7ff9d1600700 1 -- 172.21.15.12:0/2158219773 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v232) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff9d406bf80 con 0x7ff9d40cf1b0 2024-03-29T11:32:08.174 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:32:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 -- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9b406af60 msgr2=0x7ff9b406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9b406af60 0x7ff9b406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 -- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9d40cf1b0 msgr2=0x7ff9d40b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9d40cf1b0 0x7ff9d40b5990 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8000f60 tx=0x7ff9c800c730 comp rx=0 tx=0).stop 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 -- 172.21.15.12:0/2158219773 shutdown_connections 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff9b406af60 0x7ff9b406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9d40cc8a0 0x7ff9d40b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff9d40cf1b0 0x7ff9d40b5990 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 --2- 172.21.15.12:0/2158219773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9d415d8a0 0x7ff9d41f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 -- 172.21.15.12:0/2158219773 >> 172.21.15.12:0/2158219773 conn(0x7ff9d408aff0 msgr2=0x7ff9d406c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 -- 172.21.15.12:0/2158219773 shutdown_connections 2024-03-29T11:32:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.181+0000 7ff9dc400700 1 -- 172.21.15.12:0/2158219773 wait complete. 2024-03-29T11:32:08.195 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-29T11:32:08.302 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:08.307+0000 7f2910d5b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:32:08.376 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.382+0000 7fe61ff2c240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:32:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.402+0000 7fac57000700 1 Processor -- start 2024-03-29T11:32:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.402+0000 7fac57000700 1 -- start start 2024-03-29T11:32:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.403+0000 7fac57000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5015a5c0 0x7fac500709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.403+0000 7fac57000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac500ce840 0x7fac500cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac57000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac500cb310 con 0x7fac5015dfe0 2024-03-29T11:32:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac57000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac50070f00 con 0x7fac500ce840 2024-03-29T11:32:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac57000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac500711a0 con 0x7fac5015a5c0 2024-03-29T11:32:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac55000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac500ce840 0x7fac500cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac55000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac500ce840 0x7fac500cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60158/0 (socket says 172.21.15.12:60158) 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac55000700 1 -- 172.21.15.12:0/1850210819 learned_addr learned my addr 172.21.15.12:0/1850210819 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac4fe00700 1 --2- 172.21.15.12:0/1850210819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5015a5c0 0x7fac500709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac55000700 1 -- 172.21.15.12:0/1850210819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5015a5c0 msgr2=0x7fac500709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac55000700 1 --2- 172.21.15.12:0/1850210819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5015a5c0 0x7fac500709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac55000700 1 -- 172.21.15.12:0/1850210819 >> v1:172.21.15.159:6789/0 conn(0x7fac5015dfe0 legacy=0x7fac500ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.404+0000 7fac55000700 1 -- 172.21.15.12:0/1850210819 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac50071480 con 0x7fac500ce840 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.405+0000 7fac55000700 1 --2- 172.21.15.12:0/1850210819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac500ce840 0x7fac500cba80 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fac48009e50 tx=0x7fac4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5fc84ded2ff4d53 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.405+0000 7fac4ea00700 1 -- 172.21.15.12:0/1850210819 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4800e070 con 0x7fac500ce840 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.405+0000 7fac4ea00700 1 -- 172.21.15.12:0/1850210819 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4800b910 con 0x7fac500ce840 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.405+0000 7fac4ea00700 1 -- 172.21.15.12:0/1850210819 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4800bcc0 con 0x7fac500ce840 2024-03-29T11:32:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.405+0000 7fac57000700 1 -- 172.21.15.12:0/1850210819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac500ce840 msgr2=0x7fac500cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.405+0000 7fac57000700 1 --2- 172.21.15.12:0/1850210819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac500ce840 0x7fac500cba80 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fac48009e50 tx=0x7fac4800b330 comp rx=0 tx=0).stop 2024-03-29T11:32:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.405+0000 7fac57000700 1 -- 172.21.15.12:0/1850210819 shutdown_connections 2024-03-29T11:32:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.405+0000 7fac57000700 1 --2- 172.21.15.12:0/1850210819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac500ce840 0x7fac500cba80 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.405+0000 7fac57000700 1 --2- 172.21.15.12:0/1850210819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5015a5c0 0x7fac500709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.405+0000 7fac57000700 1 -- 172.21.15.12:0/1850210819 >> 172.21.15.12:0/1850210819 conn(0x7fac500b0d70 msgr2=0x7fac500b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.406+0000 7fac57000700 1 -- 172.21.15.12:0/1850210819 shutdown_connections 2024-03-29T11:32:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.406+0000 7fac57000700 1 -- 172.21.15.12:0/1850210819 wait complete. 2024-03-29T11:32:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.407+0000 7fac57000700 1 Processor -- start 2024-03-29T11:32:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.408+0000 7fac57000700 1 -- start start 2024-03-29T11:32:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac57000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac500ce840 0x7fac500af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac57000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac5015a5c0 0x7fac500c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac57000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac5015dfe0 0x7fac50153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac57000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac500c99c0 con 0x7fac5015dfe0 2024-03-29T11:32:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac57000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac500c9ca0 con 0x7fac5015a5c0 2024-03-29T11:32:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac57000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac500c9f80 con 0x7fac500ce840 2024-03-29T11:32:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac55000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac5015dfe0 0x7fac50153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac55000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac5015dfe0 0x7fac50153600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60168/0 (socket says 172.21.15.12:60168) 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac55000700 1 -- 172.21.15.12:0/1615799637 learned_addr learned my addr 172.21.15.12:0/1615799637 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac4f400700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac5015a5c0 0x7fac500c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac4fe00700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac500ce840 0x7fac500af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac55000700 1 -- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac500ce840 msgr2=0x7fac500af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac55000700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac500ce840 0x7fac500af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac55000700 1 -- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac5015a5c0 msgr2=0x7fac500c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac55000700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac5015a5c0 0x7fac500c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.409+0000 7fac55000700 1 -- 172.21.15.12:0/1615799637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac48009b30 con 0x7fac5015dfe0 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.410+0000 7fac55000700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac5015dfe0 0x7fac50153600 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fac5006c130 tx=0x7fac48004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.410+0000 7fac47e00700 1 -- 172.21.15.12:0/1615799637 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48004800 con 0x7fac5015dfe0 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.410+0000 7fac47e00700 1 -- 172.21.15.12:0/1615799637 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac48004960 con 0x7fac5015dfe0 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.410+0000 7fac47e00700 1 -- 172.21.15.12:0/1615799637 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4801c680 con 0x7fac5015dfe0 2024-03-29T11:32:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.410+0000 7fac57000700 1 -- 172.21.15.12:0/1615799637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac500ca260 con 0x7fac5015dfe0 2024-03-29T11:32:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.410+0000 7fac57000700 1 -- 172.21.15.12:0/1615799637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5015d9f0 con 0x7fac5015dfe0 2024-03-29T11:32:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.412+0000 7fac57000700 1 -- 172.21.15.12:0/1615799637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac50067720 con 0x7fac5015dfe0 2024-03-29T11:32:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.412+0000 7fac47e00700 1 -- 172.21.15.12:0/1615799637 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 135) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fac4801c7e0 con 0x7fac5015dfe0 2024-03-29T11:32:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.413+0000 7fac47e00700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fac3006af60 0x7fac3006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.413+0000 7fac47e00700 1 -- 172.21.15.12:0/1615799637 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6989+0+0 (secure 0 0 0) 0x7fac4808b930 con 0x7fac5015dfe0 2024-03-29T11:32:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.413+0000 7fac4fe00700 1 -- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fac3006af60 msgr2=0x7fac3006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.413+0000 7fac4fe00700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fac3006af60 0x7fac3006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.417+0000 7fac47e00700 1 -- 172.21.15.12:0/1615799637 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac48056200 con 0x7fac5015dfe0 2024-03-29T11:32:08.490 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.496+0000 7fe61ff2c240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:32:08.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.507+0000 7fac47e00700 1 -- 172.21.15.12:0/1615799637 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 136) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fac480565d0 con 0x7fac5015dfe0 2024-03-29T11:32:08.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.507+0000 7fac47e00700 1 -- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fac3006af60 msgr2=0x7fac3006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:08.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.507+0000 7fac47e00700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fac3006af60 0x7fac3006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.580 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.585+0000 7fe58ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.580 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.585+0000 7fe58ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.580 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.585+0000 7fe58ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.580 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.585+0000 7fe58ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.580 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.585+0000 7fe58ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.581 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.587+0000 7fe593000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.581 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.587+0000 7fe593000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.581 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.587+0000 7fe593000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.582 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.587+0000 7fe593000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.582 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.587+0000 7fe593000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.594 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.600+0000 7fe593000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.594 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.600+0000 7fe593000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.594 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.600+0000 7fe593000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.594 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.600+0000 7fe593000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.594 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.600+0000 7fe593000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.595 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.600+0000 7fe58ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.595 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.600+0000 7fe58ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.595 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.600+0000 7fe58ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.595 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.600+0000 7fe58ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.595 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:08.600+0000 7fe58ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:32:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.645+0000 7fac57000700 1 -- 172.21.15.12:0/1615799637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fac5006d2d0 con 0x7fac5015dfe0 2024-03-29T11:32:08.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.646+0000 7fac47e00700 1 -- 172.21.15.12:0/1615799637 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v233) v1 ==== 107+0+11 (secure 0 0 0) 0x7fac4806fae0 con 0x7fac5015dfe0 2024-03-29T11:32:08.642 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:32:08.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.650+0000 7fac57000700 1 -- 172.21.15.12:0/1615799637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac5015dfe0 msgr2=0x7fac50153600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.650+0000 7fac57000700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac5015dfe0 0x7fac50153600 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fac5006c130 tx=0x7fac48004ad0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.650+0000 7fac57000700 1 -- 172.21.15.12:0/1615799637 shutdown_connections 2024-03-29T11:32:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.650+0000 7fac57000700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fac3006af60 0x7fac3006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.650+0000 7fac57000700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac5015dfe0 0x7fac50153600 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.650+0000 7fac57000700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac500ce840 0x7fac500af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.650+0000 7fac57000700 1 --2- 172.21.15.12:0/1615799637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac5015a5c0 0x7fac500c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.650+0000 7fac57000700 1 -- 172.21.15.12:0/1615799637 >> 172.21.15.12:0/1615799637 conn(0x7fac500b0d70 msgr2=0x7fac500b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.650+0000 7fac57000700 1 -- 172.21.15.12:0/1615799637 shutdown_connections 2024-03-29T11:32:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.650+0000 7fac57000700 1 -- 172.21.15.12:0/1615799637 wait complete. 2024-03-29T11:32:08.675 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-29T11:32:08.784 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:08.788+0000 7f2910d5b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:32:08.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.883+0000 7f0631a00700 1 Processor -- start 2024-03-29T11:32:08.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.883+0000 7f0631a00700 1 -- start start 2024-03-29T11:32:08.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.884+0000 7f0631a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f062c0cbb30 0x7f062c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.884+0000 7f0631a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f062c0cd4a0 0x7f062c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.884+0000 7f0631a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f062c0cc450 con 0x7f062c0cd020 2024-03-29T11:32:08.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.884+0000 7f0631a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f062c0cdae0 con 0x7f062c0cd4a0 2024-03-29T11:32:08.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.884+0000 7f0631a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062c0cddc0 con 0x7f062c0cbb30 2024-03-29T11:32:08.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.885+0000 7f062b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f062c0cd4a0 0x7f062c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:08.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.885+0000 7f062b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f062c0cd4a0 0x7f062c06e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38692/0 (socket says 172.21.15.12:38692) 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.885+0000 7f062b400700 1 -- 172.21.15.12:0/169752280 learned_addr learned my addr 172.21.15.12:0/169752280 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.885+0000 7f062aa00700 1 --2- 172.21.15.12:0/169752280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f062c0cbb30 0x7f062c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.885+0000 7f062b400700 1 -- 172.21.15.12:0/169752280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f062c0cbb30 msgr2=0x7f062c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.885+0000 7f062b400700 1 --2- 172.21.15.12:0/169752280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f062c0cbb30 0x7f062c0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.885+0000 7f062b400700 1 -- 172.21.15.12:0/169752280 >> v1:172.21.15.159:6789/0 conn(0x7f062c0cd020 legacy=0x7f062c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.885+0000 7f062b400700 1 -- 172.21.15.12:0/169752280 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062c0ce0a0 con 0x7f062c0cd4a0 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.885+0000 7f0629600700 1 -- 172.21.15.12:0/169752280 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557960022 0 0) 0x7f062c0cc450 con 0x7f062c0cd020 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.885+0000 7f062b400700 1 --2- 172.21.15.12:0/169752280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 0x7f062c06e420 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f0624009e50 tx=0x7f062400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e64d3c8419f8bb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.886+0000 7f0629600700 1 -- 172.21.15.12:0/169752280 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062400e070 con 0x7f062c0cd4a0 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.886+0000 7f0629600700 1 -- 172.21.15.12:0/169752280 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062400b910 con 0x7f062c0cd4a0 2024-03-29T11:32:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.886+0000 7f0629600700 1 -- 172.21.15.12:0/169752280 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062400bcc0 con 0x7f062c0cd4a0 2024-03-29T11:32:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.886+0000 7f0631a00700 1 -- 172.21.15.12:0/169752280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 msgr2=0x7f062c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.886+0000 7f0631a00700 1 --2- 172.21.15.12:0/169752280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 0x7f062c06e420 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f0624009e50 tx=0x7f062400b330 comp rx=0 tx=0).stop 2024-03-29T11:32:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.887+0000 7f0631a00700 1 -- 172.21.15.12:0/169752280 shutdown_connections 2024-03-29T11:32:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.887+0000 7f0631a00700 1 --2- 172.21.15.12:0/169752280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 0x7f062c06e420 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.887+0000 7f0631a00700 1 --2- 172.21.15.12:0/169752280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f062c0cbb30 0x7f062c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.887+0000 7f0631a00700 1 -- 172.21.15.12:0/169752280 >> 172.21.15.12:0/169752280 conn(0x7f062c0b0d70 msgr2=0x7f062c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.887+0000 7f0631a00700 1 -- 172.21.15.12:0/169752280 shutdown_connections 2024-03-29T11:32:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.887+0000 7f0631a00700 1 -- 172.21.15.12:0/169752280 wait complete. 2024-03-29T11:32:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.889+0000 7f0631a00700 1 Processor -- start 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.889+0000 7f0631a00700 1 -- start start 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f0631a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f062c0cbb30 0x7f062c0cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f0631a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f062c0cd020 0x7f062c14c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f0631a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 0x7f062c0713a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f0631a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062c14cb60 con 0x7f062c0cd4a0 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f0631a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062c072430 con 0x7f062c0cbb30 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f0631a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062c15c800 con 0x7f062c0cd020 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 0x7f062c0713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 0x7f062c0713a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38696/0 (socket says 172.21.15.12:38696) 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062b400700 1 -- 172.21.15.12:0/1528000873 learned_addr learned my addr 172.21.15.12:0/1528000873 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062aa00700 1 --2- 172.21.15.12:0/1528000873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f062c0cbb30 0x7f062c0cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062a000700 1 --2- 172.21.15.12:0/1528000873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f062c0cd020 0x7f062c14c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062aa00700 1 -- 172.21.15.12:0/1528000873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f062c0cd020 msgr2=0x7f062c14c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062aa00700 1 --2- 172.21.15.12:0/1528000873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f062c0cd020 0x7f062c14c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062aa00700 1 -- 172.21.15.12:0/1528000873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 msgr2=0x7f062c0713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062aa00700 1 --2- 172.21.15.12:0/1528000873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 0x7f062c0713a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062aa00700 1 -- 172.21.15.12:0/1528000873 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0624009b30 con 0x7f062c0cbb30 2024-03-29T11:32:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.891+0000 7f062b400700 1 --2- 172.21.15.12:0/1528000873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 0x7f062c0713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.892+0000 7f062aa00700 1 --2- 172.21.15.12:0/1528000873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f062c0cbb30 0x7f062c0cab00 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f061c00e870 tx=0x7f061c00eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.892+0000 7f0622a00700 1 -- 172.21.15.12:0/1528000873 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c013680 con 0x7f062c0cbb30 2024-03-29T11:32:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.892+0000 7f0622a00700 1 -- 172.21.15.12:0/1528000873 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c0137e0 con 0x7f062c0cbb30 2024-03-29T11:32:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.892+0000 7f0631a00700 1 -- 172.21.15.12:0/1528000873 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062c0718e0 con 0x7f062c0cbb30 2024-03-29T11:32:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.892+0000 7f0622a00700 1 -- 172.21.15.12:0/1528000873 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c013b90 con 0x7f062c0cbb30 2024-03-29T11:32:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.892+0000 7f0631a00700 1 -- 172.21.15.12:0/1528000873 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f062c071e70 con 0x7f062c0cbb30 2024-03-29T11:32:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.893+0000 7f0631a00700 1 -- 172.21.15.12:0/1528000873 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f062c15c660 con 0x7f062c0cbb30 2024-03-29T11:32:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.894+0000 7f0622a00700 1 -- 172.21.15.12:0/1528000873 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 136) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f061c013cf0 con 0x7f062c0cbb30 2024-03-29T11:32:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.895+0000 7f0622a00700 1 -- 172.21.15.12:0/1528000873 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7376+0+0 (secure 0 0 0) 0x7f061c089230 con 0x7f062c0cbb30 2024-03-29T11:32:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:08.900+0000 7f0622a00700 1 -- 172.21.15.12:0/1528000873 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f062c15c660 con 0x7f062c0cbb30 2024-03-29T11:32:08.905 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:08.910+0000 7f2910d5b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:32:09.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.113+0000 7f0631a00700 1 -- 172.21.15.12:0/1528000873 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f062c15c660 con 0x7f062c0cbb30 2024-03-29T11:32:09.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.115+0000 7f0622a00700 1 -- 172.21.15.12:0/1528000873 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v233) v1 ==== 103+0+11 (secure 0 0 0) 0x7f061c017020 con 0x7f062c0cbb30 2024-03-29T11:32:09.110 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:32:09.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.116+0000 7f0631a00700 1 -- 172.21.15.12:0/1528000873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f062c0cbb30 msgr2=0x7f062c0cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.116+0000 7f0631a00700 1 --2- 172.21.15.12:0/1528000873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f062c0cbb30 0x7f062c0cab00 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f061c00e870 tx=0x7f061c00eb80 comp rx=0 tx=0).stop 2024-03-29T11:32:09.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.116+0000 7f0631a00700 1 -- 172.21.15.12:0/1528000873 shutdown_connections 2024-03-29T11:32:09.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.116+0000 7f0631a00700 1 --2- 172.21.15.12:0/1528000873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f062c0cd4a0 0x7f062c0713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.116+0000 7f0631a00700 1 --2- 172.21.15.12:0/1528000873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f062c0cbb30 0x7f062c0cab00 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.116+0000 7f0631a00700 1 --2- 172.21.15.12:0/1528000873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f062c0cd020 0x7f062c14c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.116+0000 7f0631a00700 1 -- 172.21.15.12:0/1528000873 >> 172.21.15.12:0/1528000873 conn(0x7f062c0b0d70 msgr2=0x7f062c159620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:09.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.116+0000 7f0631a00700 1 -- 172.21.15.12:0/1528000873 shutdown_connections 2024-03-29T11:32:09.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.116+0000 7f0631a00700 1 -- 172.21.15.12:0/1528000873 wait complete. 2024-03-29T11:32:09.131 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.334+0000 7fb4e4600700 1 Processor -- start 2024-03-29T11:32:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.334+0000 7fb4e4600700 1 -- start start 2024-03-29T11:32:09.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.336+0000 7fb4e4600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb4dc161ca0 0x7fb4dc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.336+0000 7fb4e4600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc0ceda0 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.336+0000 7fb4e4600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc1612f0 con 0x7fb4dc160f10 2024-03-29T11:32:09.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.336+0000 7fb4e4600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4dc0d1d30 con 0x7fb4dc161ca0 2024-03-29T11:32:09.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.336+0000 7fb4e2600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb4dc161ca0 0x7fb4dc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.336+0000 7fb4e1200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb4dc0d18b0 0x7fb4dc161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46720/0 (socket says 172.21.15.12:46720) 2024-03-29T11:32:09.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.336+0000 7fb4e1200700 1 -- 172.21.15.12:0/686830618 learned_addr learned my addr 172.21.15.12:0/686830618 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:09.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.336+0000 7fb4e2600700 1 -- 172.21.15.12:0/686830618 >> v1:172.21.15.159:6789/0 conn(0x7fb4dc160f10 legacy=0x7fb4dc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.336+0000 7fb4e2600700 1 -- 172.21.15.12:0/686830618 >> v1:172.21.15.12:6789/0 conn(0x7fb4dc0d18b0 legacy=0x7fb4dc161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.336+0000 7fb4e2600700 1 -- 172.21.15.12:0/686830618 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc0ce470 con 0x7fb4dc161ca0 2024-03-29T11:32:09.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.337+0000 7fb4e2600700 1 --2- 172.21.15.12:0/686830618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc161ca0 0x7fb4dc0cf230 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009d90 tx=0x7fb4d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5908b750325490b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.337+0000 7fb4dbe00700 1 -- 172.21.15.12:0/686830618 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d400e070 con 0x7fb4dc161ca0 2024-03-29T11:32:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.337+0000 7fb4dbe00700 1 -- 172.21.15.12:0/686830618 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d400bbd0 con 0x7fb4dc161ca0 2024-03-29T11:32:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.337+0000 7fb4dbe00700 1 -- 172.21.15.12:0/686830618 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4012430 con 0x7fb4dc161ca0 2024-03-29T11:32:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.337+0000 7fb4e4600700 1 -- 172.21.15.12:0/686830618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc161ca0 msgr2=0x7fb4dc0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.337+0000 7fb4e4600700 1 --2- 172.21.15.12:0/686830618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc161ca0 0x7fb4dc0cf230 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009d90 tx=0x7fb4d400b330 comp rx=0 tx=0).stop 2024-03-29T11:32:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.338+0000 7fb4e4600700 1 -- 172.21.15.12:0/686830618 shutdown_connections 2024-03-29T11:32:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.338+0000 7fb4e4600700 1 --2- 172.21.15.12:0/686830618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc161ca0 0x7fb4dc0cf230 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.338+0000 7fb4e4600700 1 -- 172.21.15.12:0/686830618 >> 172.21.15.12:0/686830618 conn(0x7fb4dc08aff0 msgr2=0x7fb4dc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.338+0000 7fb4e4600700 1 -- 172.21.15.12:0/686830618 shutdown_connections 2024-03-29T11:32:09.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.338+0000 7fb4e4600700 1 -- 172.21.15.12:0/686830618 wait complete. 2024-03-29T11:32:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.339+0000 7fb4e4600700 1 Processor -- start 2024-03-29T11:32:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.340+0000 7fb4e4600700 1 -- start start 2024-03-29T11:32:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e4600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc0d18b0 0x7fb4dc1f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e4600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4dc160f10 0x7fb4dc1f5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e4600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4dc161ca0 0x7fb4dc157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e4600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc1592e0 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e4600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc156db0 con 0x7fb4dc160f10 2024-03-29T11:32:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e4600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc156fa0 con 0x7fb4dc161ca0 2024-03-29T11:32:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc0d18b0 0x7fb4dc1f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc0d18b0 0x7fb4dc1f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38710/0 (socket says 172.21.15.12:38710) 2024-03-29T11:32:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1c00700 1 -- 172.21.15.12:0/3975530327 learned_addr learned my addr 172.21.15.12:0/3975530327 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1200700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4dc160f10 0x7fb4dc1f5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e2600700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4dc161ca0 0x7fb4dc157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1c00700 1 -- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4dc161ca0 msgr2=0x7fb4dc157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1c00700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4dc161ca0 0x7fb4dc157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1c00700 1 -- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4dc160f10 msgr2=0x7fb4dc1f5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1c00700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4dc160f10 0x7fb4dc1f5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1c00700 1 -- 172.21.15.12:0/3975530327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d4009a70 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1200700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4dc160f10 0x7fb4dc1f5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:09.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.341+0000 7fb4e1c00700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc0d18b0 0x7fb4dc1f3240 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000c480 tx=0x7fb4d000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:09.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.342+0000 7fb4d9600700 1 -- 172.21.15.12:0/3975530327 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d00052d0 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.342+0000 7fb4d9600700 1 -- 172.21.15.12:0/3975530327 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d0005430 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.342+0000 7fb4e4600700 1 -- 172.21.15.12:0/3975530327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4dc157280 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.342+0000 7fb4d9600700 1 -- 172.21.15.12:0/3975530327 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d00038c0 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.342+0000 7fb4e4600700 1 -- 172.21.15.12:0/3975530327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4dc157840 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.343+0000 7fb4e4600700 1 -- 172.21.15.12:0/3975530327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4dc067720 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.344+0000 7fb4d9600700 1 -- 172.21.15.12:0/3975530327 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 136) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fb4d0003a20 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.345+0000 7fb4d9600700 1 -- 172.21.15.12:0/3975530327 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7376+0+0 (secure 0 0 0) 0x7fb4d0088950 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.349+0000 7fb4d9600700 1 -- 172.21.15.12:0/3975530327 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb4d0053830 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.509+0000 7fb4d9600700 1 -- 172.21.15.12:0/3975530327 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 137) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fb4d00532e0 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.509+0000 7fb4d9600700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb4bc072be0 0x7fb4bc074fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.513+0000 7fb4e1200700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb4bc072be0 0x7fb4bc074fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.517+0000 7fb4e1200700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb4bc072be0 0x7fb4bc074fd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8007fd0 tx=0x7fb4c8009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.612+0000 7fb4e4600700 1 -- 172.21.15.12:0/3975530327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4dc08b760 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.613+0000 7fb4d9600700 1 -- 172.21.15.12:0/3975530327 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5971 (secure 0 0 0) 0x7fb4d0072580 con 0x7fb4dc0d18b0 2024-03-29T11:32:09.609 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:09.610 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":112,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:26.816217+0000","modified":"2024-03-29T11:31:30.290952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/2722218561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2722218561},{"type":"v1","addr":"172.21.15.12:6833","nonce":2722218561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:40.863222+0000","modified":"2024-03-29T11:31:43.883199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-29T11:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 -- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb4bc072be0 msgr2=0x7fb4bc074fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb4bc072be0 0x7fb4bc074fd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8007fd0 tx=0x7fb4c8009040 comp rx=0 tx=0).stop 2024-03-29T11:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 -- 172.21.15.12:0/3975530327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc0d18b0 msgr2=0x7fb4dc1f3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc0d18b0 0x7fb4dc1f3240 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000c480 tx=0x7fb4d000c840 comp rx=0 tx=0).stop 2024-03-29T11:32:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 -- 172.21.15.12:0/3975530327 shutdown_connections 2024-03-29T11:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb4bc072be0 0x7fb4bc074fd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4dc161ca0 0x7fb4dc157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4dc0d18b0 0x7fb4dc1f3240 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 --2- 172.21.15.12:0/3975530327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4dc160f10 0x7fb4dc1f5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 -- 172.21.15.12:0/3975530327 >> 172.21.15.12:0/3975530327 conn(0x7fb4dc08aff0 msgr2=0x7fb4dc0d2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 -- 172.21.15.12:0/3975530327 shutdown_connections 2024-03-29T11:32:09.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.616+0000 7fb4e4600700 1 -- 172.21.15.12:0/3975530327 wait complete. 2024-03-29T11:32:09.615 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 112 2024-03-29T11:32:09.632 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:09.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.844+0000 7fd686000700 1 Processor -- start 2024-03-29T11:32:09.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.844+0000 7fd686000700 1 -- start start 2024-03-29T11:32:09.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.845+0000 7fd686000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd6800b3af0 0x7fd68015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.845+0000 7fd686000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 0x7fd680070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.845+0000 7fd686000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6800b2180 con 0x7fd680072580 2024-03-29T11:32:09.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.845+0000 7fd686000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68015b100 con 0x7fd6800b3af0 2024-03-29T11:32:09.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.845+0000 7fd686000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68015b3e0 con 0x7fd6800709c0 2024-03-29T11:32:09.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67f400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd6800b3af0 0x7fd68015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67fe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 0x7fd680070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67ea00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fd680072580 0x7fd6800b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:57638/0 (socket says 172.21.15.12:57638) 2024-03-29T11:32:09.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67ea00700 1 -- 172.21.15.12:0/1361618756 learned_addr learned my addr 172.21.15.12:0/1361618756 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:09.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67e000700 1 -- 172.21.15.12:0/1361618756 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085207877 0 0) 0x7fd6800b2180 con 0x7fd680072580 2024-03-29T11:32:09.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67e000700 1 -- 172.21.15.12:0/1361618756 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6640035b0 con 0x7fd680072580 2024-03-29T11:32:09.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67fe00700 1 -- 172.21.15.12:0/1361618756 >> v2:172.21.15.159:3300/0 conn(0x7fd6800b3af0 msgr2=0x7fd68015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:09.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67fe00700 1 --2- 172.21.15.12:0/1361618756 >> v2:172.21.15.159:3300/0 conn(0x7fd6800b3af0 0x7fd68015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67fe00700 1 -- 172.21.15.12:0/1361618756 >> v1:172.21.15.159:6789/0 conn(0x7fd680072580 legacy=0x7fd6800b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67fe00700 1 -- 172.21.15.12:0/1361618756 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6800b23f0 con 0x7fd6800709c0 2024-03-29T11:32:09.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.846+0000 7fd67fe00700 1 --2- 172.21.15.12:0/1361618756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 0x7fd680070e50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd678009d90 tx=0x7fd67800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2938085196b0067 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:09.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd67e000700 1 -- 172.21.15.12:0/1361618756 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67800e070 con 0x7fd6800709c0 2024-03-29T11:32:09.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd67e000700 1 -- 172.21.15.12:0/1361618756 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67800bb20 con 0x7fd6800709c0 2024-03-29T11:32:09.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd67e000700 1 -- 172.21.15.12:0/1361618756 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67800be90 con 0x7fd6800709c0 2024-03-29T11:32:09.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd686000700 1 -- 172.21.15.12:0/1361618756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 msgr2=0x7fd680070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd686000700 1 --2- 172.21.15.12:0/1361618756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 0x7fd680070e50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd678009d90 tx=0x7fd67800b330 comp rx=0 tx=0).stop 2024-03-29T11:32:09.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd686000700 1 -- 172.21.15.12:0/1361618756 shutdown_connections 2024-03-29T11:32:09.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd686000700 1 --2- 172.21.15.12:0/1361618756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 0x7fd680070e50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd686000700 1 --2- 172.21.15.12:0/1361618756 >> v2:172.21.15.159:3300/0 conn(0x7fd6800b3af0 0x7fd68015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd686000700 1 -- 172.21.15.12:0/1361618756 >> 172.21.15.12:0/1361618756 conn(0x7fd6800cf190 msgr2=0x7fd6800cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:09.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd686000700 1 -- 172.21.15.12:0/1361618756 shutdown_connections 2024-03-29T11:32:09.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.847+0000 7fd686000700 1 -- 172.21.15.12:0/1361618756 wait complete. 2024-03-29T11:32:09.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.849+0000 7fd686000700 1 Processor -- start 2024-03-29T11:32:09.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.849+0000 7fd686000700 1 -- start start 2024-03-29T11:32:09.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.850+0000 7fd686000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 0x7fd6800c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.850+0000 7fd686000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd680072580 0x7fd680145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.850+0000 7fd686000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6800b3af0 0x7fd680159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.850+0000 7fd686000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd680157c90 con 0x7fd680072580 2024-03-29T11:32:09.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.850+0000 7fd686000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6801579b0 con 0x7fd6800b3af0 2024-03-29T11:32:09.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.850+0000 7fd686000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800caee0 con 0x7fd6800709c0 2024-03-29T11:32:09.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd680072580 0x7fd680145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd680072580 0x7fd680145500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38716/0 (socket says 172.21.15.12:38716) 2024-03-29T11:32:09.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67ea00700 1 -- 172.21.15.12:0/2219923504 learned_addr learned my addr 172.21.15.12:0/2219923504 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:09.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67f400700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 0x7fd6800c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67fe00700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6800b3af0 0x7fd680159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67ea00700 1 -- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 msgr2=0x7fd6800c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67ea00700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 0x7fd6800c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67ea00700 1 -- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6800b3af0 msgr2=0x7fd680159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:09.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67ea00700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6800b3af0 0x7fd680159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:09.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67ea00700 1 -- 172.21.15.12:0/2219923504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd678009a70 con 0x7fd680072580 2024-03-29T11:32:09.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.851+0000 7fd67ea00700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd680072580 0x7fd680145500 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7fd66c001eb0 tx=0x7fd66c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:09.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.852+0000 7fd677400700 1 -- 172.21.15.12:0/2219923504 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c005d70 con 0x7fd680072580 2024-03-29T11:32:09.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.852+0000 7fd677400700 1 -- 172.21.15.12:0/2219923504 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c0070d0 con 0x7fd680072580 2024-03-29T11:32:09.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.852+0000 7fd677400700 1 -- 172.21.15.12:0/2219923504 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c007480 con 0x7fd680072580 2024-03-29T11:32:09.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.852+0000 7fd686000700 1 -- 172.21.15.12:0/2219923504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6800cb1c0 con 0x7fd680072580 2024-03-29T11:32:09.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.852+0000 7fd686000700 1 -- 172.21.15.12:0/2219923504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd680145a40 con 0x7fd680072580 2024-03-29T11:32:09.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.853+0000 7fd686000700 1 -- 172.21.15.12:0/2219923504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd680067720 con 0x7fd680072580 2024-03-29T11:32:09.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.854+0000 7fd677400700 1 -- 172.21.15.12:0/2219923504 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 137) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fd66c0075e0 con 0x7fd680072580 2024-03-29T11:32:09.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.855+0000 7fd677400700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd66006aae0 0x7fd66006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:09.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.855+0000 7fd677400700 1 -- 172.21.15.12:0/2219923504 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7376+0+0 (secure 0 0 0) 0x7fd66c088a70 con 0x7fd680072580 2024-03-29T11:32:09.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.855+0000 7fd67f400700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd66006aae0 0x7fd66006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:09.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.856+0000 7fd67f400700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd66006aae0 0x7fd66006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd670004490 tx=0x7fd67000b420 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:09.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:09.859+0000 7fd677400700 1 -- 172.21.15.12:0/2219923504 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd66c053920 con 0x7fd680072580 2024-03-29T11:32:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.079+0000 7fd686000700 1 -- 172.21.15.12:0/2219923504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6800cef70 con 0x7fd680072580 2024-03-29T11:32:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.081+0000 7fd677400700 1 -- 172.21.15.12:0/2219923504 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) v1 ==== 74+0+21018 (secure 0 0 0) 0x7fd66c017090 con 0x7fd680072580 2024-03-29T11:32:10.078 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:10.079 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":233,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:08.498811+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:31:39.075407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:31:39.845704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/4178225398":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1623528933":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1432915605":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/2328654185":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3030066758":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/2640388148":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1069287393":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/353804238":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6833/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6832/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3242066681":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3562663283":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3628749078":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/235474344":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3837208470":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/1750949668":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/629137162":"2024-03-30T11:31:55.626759+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 -- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd66006aae0 msgr2=0x7fd66006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd66006aae0 0x7fd66006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd670004490 tx=0x7fd67000b420 comp rx=0 tx=0).stop 2024-03-29T11:32:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 -- 172.21.15.12:0/2219923504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd680072580 msgr2=0x7fd680145500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd680072580 0x7fd680145500 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7fd66c001eb0 tx=0x7fd66c00c7b0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 -- 172.21.15.12:0/2219923504 shutdown_connections 2024-03-29T11:32:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd66006aae0 0x7fd66006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6800b3af0 0x7fd680159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6800709c0 0x7fd6800c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 --2- 172.21.15.12:0/2219923504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd680072580 0x7fd680145500 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 -- 172.21.15.12:0/2219923504 >> 172.21.15.12:0/2219923504 conn(0x7fd6800cf190 msgr2=0x7fd680153700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 -- 172.21.15.12:0/2219923504 shutdown_connections 2024-03-29T11:32:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.084+0000 7fd686000700 1 -- 172.21.15.12:0/2219923504 wait complete. 2024-03-29T11:32:10.100 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T11:32:10.100 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:10.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.304+0000 7f2dd1a00700 1 Processor -- start 2024-03-29T11:32:10.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.304+0000 7f2dd1a00700 1 -- start start 2024-03-29T11:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.304+0000 7f2dd1a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc15d710 0x7f2dcc0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.304+0000 7f2dd1a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2dcc09aba0 0x7f2dcc0d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.304+0000 7f2dd1a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc06bb40 con 0x7f2dcc06bf40 2024-03-29T11:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.304+0000 7f2dd1a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dcc099b00 con 0x7f2dcc09aba0 2024-03-29T11:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.304+0000 7f2dd1a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc099de0 con 0x7f2dcc15d710 2024-03-29T11:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.304+0000 7f2dcb400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2dcc06bf40 0x7f2dcc0ccbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46726/0 (socket says 172.21.15.12:46726) 2024-03-29T11:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.304+0000 7f2dcb400700 1 -- 172.21.15.12:0/1838467207 learned_addr learned my addr 172.21.15.12:0/1838467207 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.304+0000 7f2dca000700 1 --2- 172.21.15.12:0/1838467207 >> v2:172.21.15.159:3300/0 conn(0x7f2dcc09aba0 0x7f2dcc0d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.305+0000 7f2dcaa00700 1 --2- 172.21.15.12:0/1838467207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc15d710 0x7f2dcc0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.305+0000 7f2dc9600700 1 -- 172.21.15.12:0/1838467207 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476538182 0 0) 0x7f2dcc06bb40 con 0x7f2dcc06bf40 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.305+0000 7f2dc9600700 1 -- 172.21.15.12:0/1838467207 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db00035b0 con 0x7f2dcc06bf40 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.305+0000 7f2dca000700 1 -- 172.21.15.12:0/1838467207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc15d710 msgr2=0x7f2dcc0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.305+0000 7f2dca000700 1 --2- 172.21.15.12:0/1838467207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc15d710 0x7f2dcc0d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.305+0000 7f2dca000700 1 -- 172.21.15.12:0/1838467207 >> v1:172.21.15.12:6789/0 conn(0x7f2dcc06bf40 legacy=0x7f2dcc0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.305+0000 7f2dca000700 1 -- 172.21.15.12:0/1838467207 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc09a0c0 con 0x7f2dcc09aba0 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.305+0000 7f2dcaa00700 1 --2- 172.21.15.12:0/1838467207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc15d710 0x7f2dcc0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.305+0000 7f2dca000700 1 --2- 172.21.15.12:0/1838467207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dcc09aba0 0x7f2dcc0d0670 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009840 tx=0x7f2db8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58d9f2c7b63de78d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dc9600700 1 -- 172.21.15.12:0/1838467207 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8011830 con 0x7f2dcc09aba0 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dc9600700 1 -- 172.21.15.12:0/1838467207 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db8011990 con 0x7f2dcc09aba0 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dc9600700 1 -- 172.21.15.12:0/1838467207 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8011d20 con 0x7f2dcc09aba0 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1838467207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dcc09aba0 msgr2=0x7f2dcc0d0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dd1a00700 1 --2- 172.21.15.12:0/1838467207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dcc09aba0 0x7f2dcc0d0670 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009840 tx=0x7f2db8009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1838467207 shutdown_connections 2024-03-29T11:32:10.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dd1a00700 1 --2- 172.21.15.12:0/1838467207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc15d710 0x7f2dcc0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dd1a00700 1 --2- 172.21.15.12:0/1838467207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dcc09aba0 0x7f2dcc0d0670 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1838467207 >> 172.21.15.12:0/1838467207 conn(0x7f2dcc0d13e0 msgr2=0x7f2dcc0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:10.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1838467207 shutdown_connections 2024-03-29T11:32:10.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.306+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1838467207 wait complete. 2024-03-29T11:32:10.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.308+0000 7f2dd1a00700 1 Processor -- start 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.308+0000 7f2dd1a00700 1 -- start start 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dd1a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc06bf40 0x7f2dcc09dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dd1a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dcc09aba0 0x7f2dcc14ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dd1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc15d710 0x7f2dcc09bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dd1a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc14d3c0 con 0x7f2dcc15d710 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dd1a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc09c100 con 0x7f2dcc09aba0 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dd1a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc09c3b0 con 0x7f2dcc06bf40 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dcb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc15d710 0x7f2dcc09bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dcb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc15d710 0x7f2dcc09bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38718/0 (socket says 172.21.15.12:38718) 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dcb400700 1 -- 172.21.15.12:0/1576649481 learned_addr learned my addr 172.21.15.12:0/1576649481 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dcaa00700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc06bf40 0x7f2dcc09dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.309+0000 7f2dca000700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dcc09aba0 0x7f2dcc14ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.310+0000 7f2dcb400700 1 -- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc06bf40 msgr2=0x7f2dcc09dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.310+0000 7f2dcb400700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc06bf40 0x7f2dcc09dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.310+0000 7f2dcb400700 1 -- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dcc09aba0 msgr2=0x7f2dcc14ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.310+0000 7f2dcb400700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dcc09aba0 0x7f2dcc14ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.310+0000 7f2dcb400700 1 -- 172.21.15.12:0/1576649481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0094a0 con 0x7f2dcc15d710 2024-03-29T11:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.310+0000 7f2dcb400700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc15d710 0x7f2dcc09bbc0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400cb40 tx=0x7f2dc400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.310+0000 7f2dc2a00700 1 -- 172.21.15.12:0/1576649481 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc400a9e0 con 0x7f2dcc15d710 2024-03-29T11:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.310+0000 7f2dc2a00700 1 -- 172.21.15.12:0/1576649481 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc400ab40 con 0x7f2dcc15d710 2024-03-29T11:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.310+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1576649481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db80094a0 con 0x7f2dcc15d710 2024-03-29T11:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.310+0000 7f2dc2a00700 1 -- 172.21.15.12:0/1576649481 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4012400 con 0x7f2dcc15d710 2024-03-29T11:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.311+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1576649481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc09c970 con 0x7f2dcc15d710 2024-03-29T11:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.312+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1576649481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dcc067720 con 0x7f2dcc15d710 2024-03-29T11:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.313+0000 7f2dc2a00700 1 -- 172.21.15.12:0/1576649481 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 137) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f2dc401b3b0 con 0x7f2dcc15d710 2024-03-29T11:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.314+0000 7f2dc2a00700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2dac06aae0 0x7f2dac06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.314+0000 7f2dc2a00700 1 -- 172.21.15.12:0/1576649481 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7376+0+0 (secure 0 0 0) 0x7f2dc40891e0 con 0x7f2dcc15d710 2024-03-29T11:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.314+0000 7f2dcaa00700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2dac06aae0 0x7f2dac06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.315+0000 7f2dcaa00700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2dac06aae0 0x7f2dac06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009cd0 tx=0x7f2dbc009210 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:10.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.319+0000 7f2dc2a00700 1 -- 172.21.15.12:0/1576649481 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2dc4053ff0 con 0x7f2dcc15d710 2024-03-29T11:32:10.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.582+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1576649481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dcc0d0f20 con 0x7f2dcc15d710 2024-03-29T11:32:10.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.583+0000 7f2dc2a00700 1 -- 172.21.15.12:0/1576649481 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5971 (secure 0 0 0) 0x7f2dc4017090 con 0x7f2dcc15d710 2024-03-29T11:32:10.579 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:10.579 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":112,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:26.816217+0000","modified":"2024-03-29T11:31:30.290952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/2722218561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2722218561},{"type":"v1","addr":"172.21.15.12:6833","nonce":2722218561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:40.863222+0000","modified":"2024-03-29T11:31:43.883199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-29T11:32:10.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2dac06aae0 msgr2=0x7f2dac06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2dac06aae0 0x7f2dac06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009cd0 tx=0x7f2dbc009210 comp rx=0 tx=0).stop 2024-03-29T11:32:10.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1576649481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc15d710 msgr2=0x7f2dcc09bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc15d710 0x7f2dcc09bbc0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400cb40 tx=0x7f2dc400ce50 comp rx=0 tx=0).stop 2024-03-29T11:32:10.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1576649481 shutdown_connections 2024-03-29T11:32:10.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2dac06aae0 0x7f2dac06cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc15d710 0x7f2dcc09bbc0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dcc06bf40 0x7f2dcc09dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 --2- 172.21.15.12:0/1576649481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dcc09aba0 0x7f2dcc14ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1576649481 >> 172.21.15.12:0/1576649481 conn(0x7f2dcc0d13e0 msgr2=0x7f2dcc0b5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:10.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.585+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1576649481 shutdown_connections 2024-03-29T11:32:10.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.586+0000 7f2dd1a00700 1 -- 172.21.15.12:0/1576649481 wait complete. 2024-03-29T11:32:10.584 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 112 2024-03-29T11:32:10.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:10.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.807+0000 7fa5d9c00700 1 Processor -- start 2024-03-29T11:32:10.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.807+0000 7fa5d9c00700 1 -- start start 2024-03-29T11:32:10.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d9c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa5d415d640 0x7fa5d40d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d9c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d40d30d0 con 0x7fa5d40ceec0 2024-03-29T11:32:10.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d9c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d40b6840 con 0x7fa5d41619d0 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d9c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d40d2880 con 0x7fa5d415d640 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa5d415d640 0x7fa5d40d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa5d415d640 0x7fa5d40d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38726/0 (socket says 172.21.15.12:38726) 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d2a00700 1 -- 172.21.15.12:0/644518727 learned_addr learned my addr 172.21.15.12:0/644518727 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d2a00700 1 -- 172.21.15.12:0/644518727 >> v1:172.21.15.159:6789/0 conn(0x7fa5d41619d0 legacy=0x7fa5d40b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d2a00700 1 -- 172.21.15.12:0/644518727 >> v1:172.21.15.12:6789/0 conn(0x7fa5d40ceec0 legacy=0x7fa5d41612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d2a00700 1 -- 172.21.15.12:0/644518727 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d40d2b30 con 0x7fa5d415d640 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d2a00700 1 --2- 172.21.15.12:0/644518727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d415d640 0x7fa5d40d2340 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fa5c40098f0 tx=0x7fa5c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff290295be34aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d1600700 1 -- 172.21.15.12:0/644518727 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c400b8a0 con 0x7fa5d415d640 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d1600700 1 -- 172.21.15.12:0/644518727 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c400ba00 con 0x7fa5d415d640 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.808+0000 7fa5d1600700 1 -- 172.21.15.12:0/644518727 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c400bd90 con 0x7fa5d415d640 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.809+0000 7fa5d9c00700 1 -- 172.21.15.12:0/644518727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d415d640 msgr2=0x7fa5d40d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.809+0000 7fa5d9c00700 1 --2- 172.21.15.12:0/644518727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d415d640 0x7fa5d40d2340 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fa5c40098f0 tx=0x7fa5c4009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:10.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.809+0000 7fa5d9c00700 1 -- 172.21.15.12:0/644518727 shutdown_connections 2024-03-29T11:32:10.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.809+0000 7fa5d9c00700 1 --2- 172.21.15.12:0/644518727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d415d640 0x7fa5d40d2340 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.809+0000 7fa5d9c00700 1 -- 172.21.15.12:0/644518727 >> 172.21.15.12:0/644518727 conn(0x7fa5d408aff0 msgr2=0x7fa5d40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:10.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.809+0000 7fa5d9c00700 1 -- 172.21.15.12:0/644518727 shutdown_connections 2024-03-29T11:32:10.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.809+0000 7fa5d9c00700 1 -- 172.21.15.12:0/644518727 wait complete. 2024-03-29T11:32:10.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.811+0000 7fa5d9c00700 1 Processor -- start 2024-03-29T11:32:10.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.811+0000 7fa5d9c00700 1 -- start start 2024-03-29T11:32:10.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d9c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d40ceec0 0x7fa5d41f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d9c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d415d640 0x7fa5d415eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d41619d0 0x7fa5d415f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d9c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d409e3e0 con 0x7fa5d41619d0 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d9c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d409fe00 con 0x7fa5d415d640 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d9c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d40a00e0 con 0x7fa5d40ceec0 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d41619d0 0x7fa5d415f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d41619d0 0x7fa5d415f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38742/0 (socket says 172.21.15.12:38742) 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d3400700 1 -- 172.21.15.12:0/3111907644 learned_addr learned my addr 172.21.15.12:0/3111907644 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d2a00700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d40ceec0 0x7fa5d41f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d2000700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d415d640 0x7fa5d415eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d3400700 1 -- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d40ceec0 msgr2=0x7fa5d41f3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d3400700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d40ceec0 0x7fa5d41f3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d3400700 1 -- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d415d640 msgr2=0x7fa5d415eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d3400700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d415d640 0x7fa5d415eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d3400700 1 -- 172.21.15.12:0/3111907644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c4009550 con 0x7fa5d41619d0 2024-03-29T11:32:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d3400700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d41619d0 0x7fa5d415f2e0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00cb50 tx=0x7fa5cc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:10.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5caa00700 1 -- 172.21.15.12:0/3111907644 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc003c00 con 0x7fa5d41619d0 2024-03-29T11:32:10.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5caa00700 1 -- 172.21.15.12:0/3111907644 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5cc003d60 con 0x7fa5d41619d0 2024-03-29T11:32:10.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5caa00700 1 -- 172.21.15.12:0/3111907644 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc00aa20 con 0x7fa5d41619d0 2024-03-29T11:32:10.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d9c00700 1 -- 172.21.15.12:0/3111907644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d40a03c0 con 0x7fa5d41619d0 2024-03-29T11:32:10.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.813+0000 7fa5d9c00700 1 -- 172.21.15.12:0/3111907644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d41f44a0 con 0x7fa5d41619d0 2024-03-29T11:32:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.814+0000 7fa5d9c00700 1 -- 172.21.15.12:0/3111907644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d409fc20 con 0x7fa5d41619d0 2024-03-29T11:32:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.815+0000 7fa5caa00700 1 -- 172.21.15.12:0/3111907644 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 138) v1 ==== 89285+0+0 (secure 0 0 0) 0x7fa5cc00ab80 con 0x7fa5d41619d0 2024-03-29T11:32:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.816+0000 7fa5caa00700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5b406ae40 0x7fa5b406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.816+0000 7fa5caa00700 1 -- 172.21.15.12:0/3111907644 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7376+0+0 (secure 0 0 0) 0x7fa5cc089ce0 con 0x7fa5d41619d0 2024-03-29T11:32:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.816+0000 7fa5d2a00700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5b406ae40 0x7fa5b406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:10.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.817+0000 7fa5d2a00700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5b406ae40 0x7fa5b406d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009c40 tx=0x7fa5c4008030 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:10.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:10.821+0000 7fa5caa00700 1 -- 172.21.15.12:0/3111907644 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa5cc054540 con 0x7fa5d41619d0 2024-03-29T11:32:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.083+0000 7fa5d9c00700 1 -- 172.21.15.12:0/3111907644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5d40a05b0 con 0x7fa5d41619d0 2024-03-29T11:32:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.083+0000 7fa5caa00700 1 -- 172.21.15.12:0/3111907644 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5971 (secure 0 0 0) 0x7fa5cc057b60 con 0x7fa5d41619d0 2024-03-29T11:32:11.080 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:11.080 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":112,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:26.816217+0000","modified":"2024-03-29T11:31:30.290952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6833/2722218561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2722218561},{"type":"v1","addr":"172.21.15.12:6833","nonce":2722218561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:40.863222+0000","modified":"2024-03-29T11:31:43.883199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-29T11:32:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 -- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5b406ae40 msgr2=0x7fa5b406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5b406ae40 0x7fa5b406d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009c40 tx=0x7fa5c4008030 comp rx=0 tx=0).stop 2024-03-29T11:32:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 -- 172.21.15.12:0/3111907644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d41619d0 msgr2=0x7fa5d415f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d41619d0 0x7fa5d415f2e0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00cb50 tx=0x7fa5cc00ce60 comp rx=0 tx=0).stop 2024-03-29T11:32:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 -- 172.21.15.12:0/3111907644 shutdown_connections 2024-03-29T11:32:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa5b406ae40 0x7fa5b406d300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5d41619d0 0x7fa5d415f2e0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d40ceec0 0x7fa5d41f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 --2- 172.21.15.12:0/3111907644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d415d640 0x7fa5d415eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 -- 172.21.15.12:0/3111907644 >> 172.21.15.12:0/3111907644 conn(0x7fa5d408aff0 msgr2=0x7fa5d40b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 -- 172.21.15.12:0/3111907644 shutdown_connections 2024-03-29T11:32:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.087+0000 7fa5d9c00700 1 -- 172.21.15.12:0/3111907644 wait complete. 2024-03-29T11:32:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 112 2024-03-29T11:32:11.100 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:11.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.303+0000 7f6b97000700 1 Processor -- start 2024-03-29T11:32:11.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.303+0000 7f6b97000700 1 -- start start 2024-03-29T11:32:11.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.304+0000 7f6b97000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6b9015b5d0 0x7f6b900ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.304+0000 7f6b97000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b900cb3f0 0x7f6b900ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.304+0000 7f6b97000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9015fd50 con 0x7f6b900ae930 2024-03-29T11:32:11.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.304+0000 7f6b97000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9015b9b0 con 0x7f6b9015b5d0 2024-03-29T11:32:11.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.304+0000 7f6b97000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9015f140 con 0x7f6b900cb3f0 2024-03-29T11:32:11.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.304+0000 7f6b8fe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6b9015b5d0 0x7f6b900ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b95000700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f6b900ae930 0x7f6b90140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:57654/0 (socket says 172.21.15.12:57654) 2024-03-29T11:32:11.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b95000700 1 -- 172.21.15.12:0/1474137341 learned_addr learned my addr 172.21.15.12:0/1474137341 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:11.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b87e00700 1 --2- 172.21.15.12:0/1474137341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b900cb3f0 0x7f6b900ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b8f400700 1 -- 172.21.15.12:0/1474137341 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216778753 0 0) 0x7f6b9015fd50 con 0x7f6b900ae930 2024-03-29T11:32:11.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b8f400700 1 -- 172.21.15.12:0/1474137341 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b740035b0 con 0x7f6b900ae930 2024-03-29T11:32:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b87e00700 1 -- 172.21.15.12:0/1474137341 >> v2:172.21.15.159:3300/0 conn(0x7f6b9015b5d0 msgr2=0x7f6b900ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b87e00700 1 --2- 172.21.15.12:0/1474137341 >> v2:172.21.15.159:3300/0 conn(0x7f6b9015b5d0 0x7f6b900ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b87e00700 1 -- 172.21.15.12:0/1474137341 >> v1:172.21.15.159:6789/0 conn(0x7f6b900ae930 legacy=0x7f6b90140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b87e00700 1 -- 172.21.15.12:0/1474137341 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9015f420 con 0x7f6b900cb3f0 2024-03-29T11:32:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b8fe00700 1 --2- 172.21.15.12:0/1474137341 >> v2:172.21.15.159:3300/0 conn(0x7f6b9015b5d0 0x7f6b900ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b87e00700 1 --2- 172.21.15.12:0/1474137341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b900cb3f0 0x7f6b900ac0c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009840 tx=0x7f6b78009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48567275a5908743 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:11.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b8f400700 1 -- 172.21.15.12:0/1474137341 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78011830 con 0x7f6b900cb3f0 2024-03-29T11:32:11.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b8f400700 1 -- 172.21.15.12:0/1474137341 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b78011990 con 0x7f6b900cb3f0 2024-03-29T11:32:11.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b8f400700 1 -- 172.21.15.12:0/1474137341 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78011d20 con 0x7f6b900cb3f0 2024-03-29T11:32:11.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b97000700 1 -- 172.21.15.12:0/1474137341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b900cb3f0 msgr2=0x7f6b900ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b97000700 1 --2- 172.21.15.12:0/1474137341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b900cb3f0 0x7f6b900ac0c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009840 tx=0x7f6b78009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:11.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b97000700 1 -- 172.21.15.12:0/1474137341 shutdown_connections 2024-03-29T11:32:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b97000700 1 --2- 172.21.15.12:0/1474137341 >> v2:172.21.15.159:3300/0 conn(0x7f6b9015b5d0 0x7f6b900ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b97000700 1 --2- 172.21.15.12:0/1474137341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b900cb3f0 0x7f6b900ac0c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b97000700 1 -- 172.21.15.12:0/1474137341 >> 172.21.15.12:0/1474137341 conn(0x7f6b900cf190 msgr2=0x7f6b900cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b97000700 1 -- 172.21.15.12:0/1474137341 shutdown_connections 2024-03-29T11:32:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.306+0000 7f6b97000700 1 -- 172.21.15.12:0/1474137341 wait complete. 2024-03-29T11:32:11.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.308+0000 7f6b97000700 1 Processor -- start 2024-03-29T11:32:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.308+0000 7f6b97000700 1 -- start start 2024-03-29T11:32:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.309+0000 7f6b97000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b900ae930 0x7f6b900cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.309+0000 7f6b97000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b9015b5d0 0x7f6b9006ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.309+0000 7f6b97000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b9009cf40 0x7f6b9006b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.309+0000 7f6b97000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900cd000 con 0x7f6b900ae930 2024-03-29T11:32:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.309+0000 7f6b97000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900cc3f0 con 0x7f6b9009cf40 2024-03-29T11:32:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.309+0000 7f6b97000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900cc6d0 con 0x7f6b9015b5d0 2024-03-29T11:32:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.309+0000 7f6b8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b900ae930 0x7f6b900cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b900ae930 0x7f6b900cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38748/0 (socket says 172.21.15.12:38748) 2024-03-29T11:32:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b8fe00700 1 -- 172.21.15.12:0/1611534541 learned_addr learned my addr 172.21.15.12:0/1611534541 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b95000700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b9009cf40 0x7f6b9006b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b87e00700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b9015b5d0 0x7f6b9006ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b8fe00700 1 -- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b9015b5d0 msgr2=0x7f6b9006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b8fe00700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b9015b5d0 0x7f6b9006ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b8fe00700 1 -- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b9009cf40 msgr2=0x7f6b9006b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b8fe00700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b9009cf40 0x7f6b9006b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b8fe00700 1 -- 172.21.15.12:0/1611534541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b800094a0 con 0x7f6b900ae930 2024-03-29T11:32:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b87e00700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b9015b5d0 0x7f6b9006ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b95000700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b9009cf40 0x7f6b9006b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b8fe00700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b900ae930 0x7f6b900cbba0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000ebe0 tx=0x7f6b80009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.310+0000 7f6b8cc00700 1 -- 172.21.15.12:0/1611534541 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8000c8b0 con 0x7f6b900ae930 2024-03-29T11:32:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.311+0000 7f6b8cc00700 1 -- 172.21.15.12:0/1611534541 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8000ca10 con 0x7f6b900ae930 2024-03-29T11:32:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.311+0000 7f6b97000700 1 -- 172.21.15.12:0/1611534541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b780094a0 con 0x7f6b900ae930 2024-03-29T11:32:11.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.311+0000 7f6b8cc00700 1 -- 172.21.15.12:0/1611534541 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8000cda0 con 0x7f6b900ae930 2024-03-29T11:32:11.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.311+0000 7f6b97000700 1 -- 172.21.15.12:0/1611534541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b900ccc90 con 0x7f6b900ae930 2024-03-29T11:32:11.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.312+0000 7f6b97000700 1 -- 172.21.15.12:0/1611534541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b900cd2e0 con 0x7f6b900ae930 2024-03-29T11:32:11.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.313+0000 7f6b8cc00700 1 -- 172.21.15.12:0/1611534541 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 138) v1 ==== 89285+0+0 (secure 0 0 0) 0x7f6b80013470 con 0x7f6b900ae930 2024-03-29T11:32:11.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.313+0000 7f6b8cc00700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6b6c06ae40 0x7f6b6c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.314+0000 7f6b8cc00700 1 -- 172.21.15.12:0/1611534541 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7376+0+0 (secure 0 0 0) 0x7f6b8008a220 con 0x7f6b900ae930 2024-03-29T11:32:11.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.314+0000 7f6b87e00700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6b6c06ae40 0x7f6b6c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.315+0000 7f6b87e00700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6b6c06ae40 0x7f6b6c06d300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b78000c00 tx=0x7f6b78008030 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:11.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.318+0000 7f6b8cc00700 1 -- 172.21.15.12:0/1611534541 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6b80054a80 con 0x7f6b900ae930 2024-03-29T11:32:11.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.537+0000 7f6b97000700 1 -- 172.21.15.12:0/1611534541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b90067720 con 0x7f6b900ae930 2024-03-29T11:32:11.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.539+0000 7f6b8cc00700 1 -- 172.21.15.12:0/1611534541 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) v1 ==== 74+0+21018 (secure 0 0 0) 0x7f6b800580a0 con 0x7f6b900ae930 2024-03-29T11:32:11.535 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:11.536 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":233,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:08.498811+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:31:24.485419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-03-29T11:31:25.823318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:31:39.075407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:31:39.845704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/4178225398":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1623528933":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1432915605":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/2328654185":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3030066758":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/2640388148":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1069287393":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/353804238":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6833/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6832/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3242066681":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3562663283":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3628749078":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/235474344":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3837208470":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/1750949668":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/629137162":"2024-03-30T11:31:55.626759+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:11.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.541+0000 7f6b97000700 1 -- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6b6c06ae40 msgr2=0x7f6b6c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.541+0000 7f6b97000700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6b6c06ae40 0x7f6b6c06d300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b78000c00 tx=0x7f6b78008030 comp rx=0 tx=0).stop 2024-03-29T11:32:11.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.541+0000 7f6b97000700 1 -- 172.21.15.12:0/1611534541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b900ae930 msgr2=0x7f6b900cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.541+0000 7f6b97000700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b900ae930 0x7f6b900cbba0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000ebe0 tx=0x7f6b80009e20 comp rx=0 tx=0).stop 2024-03-29T11:32:11.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.541+0000 7f6b97000700 1 -- 172.21.15.12:0/1611534541 shutdown_connections 2024-03-29T11:32:11.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.541+0000 7f6b97000700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6b6c06ae40 0x7f6b6c06d300 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.541+0000 7f6b97000700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b9009cf40 0x7f6b9006b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.541+0000 7f6b97000700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b900ae930 0x7f6b900cbba0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.541+0000 7f6b97000700 1 --2- 172.21.15.12:0/1611534541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b9015b5d0 0x7f6b9006ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.541+0000 7f6b97000700 1 -- 172.21.15.12:0/1611534541 >> 172.21.15.12:0/1611534541 conn(0x7f6b900cf190 msgr2=0x7f6b900aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.542+0000 7f6b97000700 1 -- 172.21.15.12:0/1611534541 shutdown_connections 2024-03-29T11:32:11.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.542+0000 7f6b97000700 1 -- 172.21.15.12:0/1611534541 wait complete. 2024-03-29T11:32:11.556 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T11:32:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.762+0000 7f0a64600700 1 Processor -- start 2024-03-29T11:32:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.762+0000 7f0a64600700 1 -- start start 2024-03-29T11:32:11.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.762+0000 7f0a64600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 0x7f0a5c0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.762+0000 7f0a64600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a5c0d3f20 0x7f0a5c0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.762+0000 7f0a64600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a5c0cef00 con 0x7f0a5c0b3f80 2024-03-29T11:32:11.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.762+0000 7f0a64600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a5c161230 con 0x7f0a5c0d3f20 2024-03-29T11:32:11.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.762+0000 7f0a64600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c0d2a80 con 0x7f0a5c160e50 2024-03-29T11:32:11.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.762+0000 7f0a61200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a5c0d3f20 0x7f0a5c0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.763+0000 7f0a61200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a5c0d3f20 0x7f0a5c0b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38756/0 (socket says 172.21.15.12:38756) 2024-03-29T11:32:11.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.763+0000 7f0a61200700 1 -- 172.21.15.12:0/2409411214 learned_addr learned my addr 172.21.15.12:0/2409411214 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:11.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.763+0000 7f0a61c00700 1 --2- 172.21.15.12:0/2409411214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 0x7f0a5c0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.763+0000 7f0a61200700 1 -- 172.21.15.12:0/2409411214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 msgr2=0x7f0a5c0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:11.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.763+0000 7f0a61200700 1 --2- 172.21.15.12:0/2409411214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 0x7f0a5c0b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.763+0000 7f0a61200700 1 -- 172.21.15.12:0/2409411214 >> v1:172.21.15.12:6789/0 conn(0x7f0a5c0b3f80 legacy=0x7f0a5c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.763+0000 7f0a61200700 1 -- 172.21.15.12:0/2409411214 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c0d43b0 con 0x7f0a5c0d3f20 2024-03-29T11:32:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.763+0000 7f0a61200700 1 --2- 172.21.15.12:0/2409411214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0d3f20 0x7f0a5c0b53d0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f0a480098f0 tx=0x7f0a48009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933db9b425e12c9b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.763+0000 7f0a5be00700 1 -- 172.21.15.12:0/2409411214 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a48011830 con 0x7f0a5c0d3f20 2024-03-29T11:32:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.764+0000 7f0a5be00700 1 -- 172.21.15.12:0/2409411214 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a48011990 con 0x7f0a5c0d3f20 2024-03-29T11:32:11.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.764+0000 7f0a5be00700 1 -- 172.21.15.12:0/2409411214 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a48011d20 con 0x7f0a5c0d3f20 2024-03-29T11:32:11.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.764+0000 7f0a64600700 1 -- 172.21.15.12:0/2409411214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0d3f20 msgr2=0x7f0a5c0b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.764+0000 7f0a64600700 1 --2- 172.21.15.12:0/2409411214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0d3f20 0x7f0a5c0b53d0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f0a480098f0 tx=0x7f0a48009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:11.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.764+0000 7f0a64600700 1 -- 172.21.15.12:0/2409411214 shutdown_connections 2024-03-29T11:32:11.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.764+0000 7f0a64600700 1 --2- 172.21.15.12:0/2409411214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 0x7f0a5c0b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.764+0000 7f0a64600700 1 --2- 172.21.15.12:0/2409411214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0d3f20 0x7f0a5c0b53d0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.764+0000 7f0a64600700 1 -- 172.21.15.12:0/2409411214 >> 172.21.15.12:0/2409411214 conn(0x7f0a5c08aff0 msgr2=0x7f0a5c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.764+0000 7f0a64600700 1 -- 172.21.15.12:0/2409411214 shutdown_connections 2024-03-29T11:32:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.764+0000 7f0a64600700 1 -- 172.21.15.12:0/2409411214 wait complete. 2024-03-29T11:32:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.766+0000 7f0a64600700 1 Processor -- start 2024-03-29T11:32:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.766+0000 7f0a64600700 1 -- start start 2024-03-29T11:32:11.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a64600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c0b3f80 0x7f0a5c0c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a64600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 0x7f0a5c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a64600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0b8170 0x7f0a5c0ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a64600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c0caad0 con 0x7f0a5c0b8170 2024-03-29T11:32:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a64600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c0ca780 con 0x7f0a5c0b3f80 2024-03-29T11:32:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a64600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c0b86d0 con 0x7f0a5c160e50 2024-03-29T11:32:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a62600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0b8170 0x7f0a5c0ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a62600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0b8170 0x7f0a5c0ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38770/0 (socket says 172.21.15.12:38770) 2024-03-29T11:32:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a62600700 1 -- 172.21.15.12:0/785031757 learned_addr learned my addr 172.21.15.12:0/785031757 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a61c00700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c0b3f80 0x7f0a5c0c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.767+0000 7f0a61200700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 0x7f0a5c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a62600700 1 -- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 msgr2=0x7f0a5c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a62600700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 0x7f0a5c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a62600700 1 -- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c0b3f80 msgr2=0x7f0a5c0c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a62600700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c0b3f80 0x7f0a5c0c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a62600700 1 -- 172.21.15.12:0/785031757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a48009550 con 0x7f0a5c0b8170 2024-03-29T11:32:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a61200700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 0x7f0a5c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a62600700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0b8170 0x7f0a5c0ccf50 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400cb10 tx=0x7f0a5400ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a59600700 1 -- 172.21.15.12:0/785031757 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a54003c20 con 0x7f0a5c0b8170 2024-03-29T11:32:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a59600700 1 -- 172.21.15.12:0/785031757 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a54003d80 con 0x7f0a5c0b8170 2024-03-29T11:32:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a64600700 1 -- 172.21.15.12:0/785031757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5c0b89b0 con 0x7f0a5c0b8170 2024-03-29T11:32:11.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a59600700 1 -- 172.21.15.12:0/785031757 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5400aaa0 con 0x7f0a5c0b8170 2024-03-29T11:32:11.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.768+0000 7f0a64600700 1 -- 172.21.15.12:0/785031757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5c15aef0 con 0x7f0a5c0b8170 2024-03-29T11:32:11.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.770+0000 7f0a64600700 1 -- 172.21.15.12:0/785031757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5c0cac40 con 0x7f0a5c0b8170 2024-03-29T11:32:11.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.770+0000 7f0a59600700 1 -- 172.21.15.12:0/785031757 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 138) v1 ==== 89285+0+0 (secure 0 0 0) 0x7f0a5400ac00 con 0x7f0a5c0b8170 2024-03-29T11:32:11.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.771+0000 7f0a59600700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a3c06ae40 0x7f0a3c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:11.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.771+0000 7f0a59600700 1 -- 172.21.15.12:0/785031757 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7376+0+0 (secure 0 0 0) 0x7f0a54089c60 con 0x7f0a5c0b8170 2024-03-29T11:32:11.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.772+0000 7f0a61c00700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a3c06ae40 0x7f0a3c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:11.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.772+0000 7f0a61c00700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a3c06ae40 0x7f0a3c06d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009be0 tx=0x7f0a50009210 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:11.776+0000 7f0a59600700 1 -- 172.21.15.12:0/785031757 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0a540544c0 con 0x7f0a5c0b8170 2024-03-29T11:32:12.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.036+0000 7f0a64600700 1 -- 172.21.15.12:0/785031757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0a5c067720 con 0x7f0a5c0b8170 2024-03-29T11:32:12.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T11:32:12.037+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T11:32:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.047+0000 7f0a59600700 1 -- 172.21.15.12:0/785031757 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v113) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0a54057ae0 con 0x7f0a5c0b8170 2024-03-29T11:32:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 -- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a3c06ae40 msgr2=0x7f0a3c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a3c06ae40 0x7f0a3c06d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009be0 tx=0x7f0a50009210 comp rx=0 tx=0).stop 2024-03-29T11:32:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 -- 172.21.15.12:0/785031757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0b8170 msgr2=0x7f0a5c0ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0b8170 0x7f0a5c0ccf50 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400cb10 tx=0x7f0a5400ce20 comp rx=0 tx=0).stop 2024-03-29T11:32:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 -- 172.21.15.12:0/785031757 shutdown_connections 2024-03-29T11:32:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a3c06ae40 0x7f0a3c06d300 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a5c0b8170 0x7f0a5c0ccf50 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a5c0b3f80 0x7f0a5c0c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 --2- 172.21.15.12:0/785031757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a5c160e50 0x7f0a5c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 -- 172.21.15.12:0/785031757 >> 172.21.15.12:0/785031757 conn(0x7f0a5c08aff0 msgr2=0x7f0a5c0d3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 -- 172.21.15.12:0/785031757 shutdown_connections 2024-03-29T11:32:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.050+0000 7f0a64600700 1 -- 172.21.15.12:0/785031757 wait complete. 2024-03-29T11:32:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T11:32:12.054 INFO:tasks.ceph.mds.a.smithi012.stderr: -5882> 2024-03-29T11:31:31.464+0000 7f6d0f800700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:32:12.065 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T11:32:12.107 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:32:12.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.272+0000 7f2376000700 1 Processor -- start 2024-03-29T11:32:12.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.273+0000 7f2376000700 1 -- start start 2024-03-29T11:32:12.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f2376000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f23700d16a0 0x7f23700d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:12.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f2376000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2370161420 con 0x7f23700d2370 2024-03-29T11:32:12.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f2376000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2370161700 con 0x7f237015d9d0 2024-03-29T11:32:12.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f2376000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23700d4340 con 0x7f23700d16a0 2024-03-29T11:32:12.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f23700d16a0 0x7f23700d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:12.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f23700d2370 0x7f23700ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46752/0 (socket says 172.21.15.12:46752) 2024-03-29T11:32:12.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236fe00700 1 -- 172.21.15.12:0/326080141 learned_addr learned my addr 172.21.15.12:0/326080141 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:12.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236ea00700 1 -- 172.21.15.12:0/326080141 >> v1:172.21.15.159:6789/0 conn(0x7f237015d9d0 legacy=0x7f23700d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236ea00700 1 -- 172.21.15.12:0/326080141 >> v1:172.21.15.12:6789/0 conn(0x7f23700d2370 legacy=0x7f23700ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236ea00700 1 -- 172.21.15.12:0/326080141 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23701619e0 con 0x7f23700d16a0 2024-03-29T11:32:12.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236e000700 1 -- 172.21.15.12:0/326080141 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013325912 0 0) 0x7f2370161420 con 0x7f23700d2370 2024-03-29T11:32:12.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236ea00700 1 --2- 172.21.15.12:0/326080141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d16a0 0x7f23700d1e30 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f235c009830 tx=0x7f235c009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb194e9b088cd6e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:12.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236e000700 1 -- 172.21.15.12:0/326080141 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c011830 con 0x7f23700d16a0 2024-03-29T11:32:12.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236e000700 1 -- 172.21.15.12:0/326080141 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c011990 con 0x7f23700d16a0 2024-03-29T11:32:12.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f236e000700 1 -- 172.21.15.12:0/326080141 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c011d40 con 0x7f23700d16a0 2024-03-29T11:32:12.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f2376000700 1 -- 172.21.15.12:0/326080141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d16a0 msgr2=0x7f23700d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.274+0000 7f2376000700 1 --2- 172.21.15.12:0/326080141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d16a0 0x7f23700d1e30 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f235c009830 tx=0x7f235c009b40 comp rx=0 tx=0).stop 2024-03-29T11:32:12.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.276+0000 7f2376000700 1 -- 172.21.15.12:0/326080141 shutdown_connections 2024-03-29T11:32:12.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.276+0000 7f2376000700 1 --2- 172.21.15.12:0/326080141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d16a0 0x7f23700d1e30 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:12.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.276+0000 7f2376000700 1 -- 172.21.15.12:0/326080141 >> 172.21.15.12:0/326080141 conn(0x7f237008aff0 msgr2=0x7f23700d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.276+0000 7f2376000700 1 -- 172.21.15.12:0/326080141 shutdown_connections 2024-03-29T11:32:12.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.276+0000 7f2376000700 1 -- 172.21.15.12:0/326080141 wait complete. 2024-03-29T11:32:12.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.277+0000 7f2376000700 1 Processor -- start 2024-03-29T11:32:12.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.277+0000 7f2376000700 1 -- start start 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.278+0000 7f2376000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23700d16a0 0x7f237014e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.278+0000 7f2376000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d2370 0x7f23700cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.278+0000 7f2376000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f237015d9d0 0x7f23701606d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.278+0000 7f2376000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237014ef20 con 0x7f23700d2370 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.278+0000 7f2376000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23700cc2f0 con 0x7f23700d16a0 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.278+0000 7f2376000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23700cba70 con 0x7f237015d9d0 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d2370 0x7f23700cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d2370 0x7f23700cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38812/0 (socket says 172.21.15.12:38812) 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236ea00700 1 -- 172.21.15.12:0/480349696 learned_addr learned my addr 172.21.15.12:0/480349696 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236f400700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23700d16a0 0x7f237014e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236fe00700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f237015d9d0 0x7f23701606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236ea00700 1 -- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f237015d9d0 msgr2=0x7f23701606d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236ea00700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f237015d9d0 0x7f23701606d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236ea00700 1 -- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23700d16a0 msgr2=0x7f237014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236ea00700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23700d16a0 0x7f237014e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236ea00700 1 -- 172.21.15.12:0/480349696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c009440 con 0x7f23700d2370 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f236ea00700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d2370 0x7f23700cbdb0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f235c0090b0 tx=0x7f235c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f2367400700 1 -- 172.21.15.12:0/480349696 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c0042f0 con 0x7f23700d2370 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f2367400700 1 -- 172.21.15.12:0/480349696 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c004450 con 0x7f23700d2370 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f2376000700 1 -- 172.21.15.12:0/480349696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23701f39d0 con 0x7f23700d2370 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f2367400700 1 -- 172.21.15.12:0/480349696 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c0038e0 con 0x7f23700d2370 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.280+0000 7f2376000700 1 -- 172.21.15.12:0/480349696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23701f3f90 con 0x7f23700d2370 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.282+0000 7f2376000700 1 -- 172.21.15.12:0/480349696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23700d1ca0 con 0x7f23700d2370 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.282+0000 7f2367400700 1 -- 172.21.15.12:0/480349696 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 138) v1 ==== 89285+0+0 (secure 0 0 0) 0x7f235c022030 con 0x7f23700d2370 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.283+0000 7f2367400700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f235006ae40 0x7f235006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:12.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.283+0000 7f2367400700 1 -- 172.21.15.12:0/480349696 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7462+0+0 (secure 0 0 0) 0x7f235c08b650 con 0x7f23700d2370 2024-03-29T11:32:12.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.283+0000 7f236f400700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f235006ae40 0x7f235006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:12.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.284+0000 7f236f400700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f235006ae40 0x7f235006d300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2360000f60 tx=0x7f2360009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:12.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.288+0000 7f2367400700 1 -- 172.21.15.12:0/480349696 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f235c056080 con 0x7f23700d2370 2024-03-29T11:32:12.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:12.554+0000 7f2376000700 1 -- 172.21.15.12:0/480349696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2370067720 con 0x7f23700d2370 2024-03-29T11:32:13.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.058+0000 7f2367400700 1 -- 172.21.15.12:0/480349696 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f235c056450 con 0x7f23700d2370 2024-03-29T11:32:13.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.059+0000 7f2367400700 1 -- 172.21.15.12:0/480349696 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v114) v1 ==== 104+0+0 (secure 0 0 0) 0x7f235c06ecd0 con 0x7f23700d2370 2024-03-29T11:32:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 -- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f235006ae40 msgr2=0x7f235006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f235006ae40 0x7f235006d300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2360000f60 tx=0x7f2360009040 comp rx=0 tx=0).stop 2024-03-29T11:32:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 -- 172.21.15.12:0/480349696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d2370 msgr2=0x7f23700cbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d2370 0x7f23700cbdb0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f235c0090b0 tx=0x7f235c004750 comp rx=0 tx=0).stop 2024-03-29T11:32:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 -- 172.21.15.12:0/480349696 shutdown_connections 2024-03-29T11:32:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f235006ae40 0x7f235006d300 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f237015d9d0 0x7f23701606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23700d16a0 0x7f237014e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 --2- 172.21.15.12:0/480349696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23700d2370 0x7f23700cbdb0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 -- 172.21.15.12:0/480349696 >> 172.21.15.12:0/480349696 conn(0x7f237008aff0 msgr2=0x7f23700d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 -- 172.21.15.12:0/480349696 shutdown_connections 2024-03-29T11:32:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.062+0000 7f2376000700 1 -- 172.21.15.12:0/480349696 wait complete. 2024-03-29T11:32:13.078 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T11:32:13.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.328+0000 7f8311600700 1 Processor -- start 2024-03-29T11:32:13.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.329+0000 7f8311600700 1 -- start start 2024-03-29T11:32:13.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f8311600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f830c155ac0 0x7f830c15c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:13.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f8311600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c0ae670 0x7f830c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f8311600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c06dd00 con 0x7f830c06c910 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f8311600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830c06d0f0 con 0x7f830c155ac0 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f8311600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830c06d3d0 con 0x7f830c0ae670 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f830a000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f830c06c910 0x7f830c0adf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46768/0 (socket says 172.21.15.12:46768) 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f830a000700 1 -- 172.21.15.12:0/1682651418 learned_addr learned my addr 172.21.15.12:0/1682651418 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f830aa00700 1 --2- 172.21.15.12:0/1682651418 >> v2:172.21.15.159:3300/0 conn(0x7f830c155ac0 0x7f830c15c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f830b400700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c0ae670 0x7f830c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172844166 0 0) 0x7f830c06dd00 con 0x7f830c06c910 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.330+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f00035b0 con 0x7f830c06c910 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3852362029 0 0) 0x7f82f00035b0 con 0x7f830c06c910 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c0ae670 msgr2=0x7f830c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c0ae670 0x7f830c15ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 >> v2:172.21.15.159:3300/0 conn(0x7f830c155ac0 msgr2=0x7f830c15c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:13.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 --2- 172.21.15.12:0/1682651418 >> v2:172.21.15.159:3300/0 conn(0x7f830c155ac0 0x7f830c15c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:13.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c06d6b0 con 0x7f830c06c910 2024-03-29T11:32:13.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f82f8001670 con 0x7f830c06c910 2024-03-29T11:32:13.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 >> v1:172.21.15.12:6789/0 conn(0x7f830c06c910 legacy=0x7f830c0adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c0ae670 0x7f82f0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82f0003f80 0x7f82f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f0004d90 0x7f82f0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c06d0f0 con 0x7f82f0004d90 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c06d3d0 con 0x7f82f0003f80 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f000ba10 con 0x7f830c0ae670 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f830b400700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f0004d90 0x7f82f0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f830a000700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82f0003f80 0x7f82f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.331+0000 7f830aa00700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c0ae670 0x7f82f0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.332+0000 7f830b400700 1 -- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c0ae670 msgr2=0x7f82f0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.332+0000 7f830b400700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c0ae670 0x7f82f0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.332+0000 7f830b400700 1 -- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82f0003f80 msgr2=0x7f82f0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.332+0000 7f830b400700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82f0003f80 0x7f82f0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.332+0000 7f830b400700 1 -- 172.21.15.12:0/1682651418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00035b0 con 0x7f82f0004d90 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.332+0000 7f830b400700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f0004d90 0x7f82f0005210 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f8304009fc0 tx=0x7f82f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e721cf761cf31954 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.332+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830400ebd0 con 0x7f82f0004d90 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.332+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830400f040 con 0x7f82f0004d90 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.332+0000 7f8309600700 1 -- 172.21.15.12:0/1682651418 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304014480 con 0x7f82f0004d90 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.333+0000 7f8311600700 1 -- 172.21.15.12:0/1682651418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f0004d90 msgr2=0x7f82f0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.333+0000 7f8311600700 1 --2- 172.21.15.12:0/1682651418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f0004d90 0x7f82f0005210 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f8304009fc0 tx=0x7f82f0005880 comp rx=0 tx=0).stop 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.333+0000 7f830b400700 1 -- 172.21.15.12:0/1682651418 reap_dead start 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.333+0000 7f8311600700 1 -- 172.21.15.12:0/1682651418 shutdown_connections 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.333+0000 7f8311600700 1 -- 172.21.15.12:0/1682651418 >> 172.21.15.12:0/1682651418 conn(0x7f830c0cf210 msgr2=0x7f830c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.333+0000 7f8311600700 1 -- 172.21.15.12:0/1682651418 shutdown_connections 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.333+0000 7f8311600700 1 -- 172.21.15.12:0/1682651418 wait complete. 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.335+0000 7f8311600700 1 Processor -- start 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.335+0000 7f8311600700 1 -- start start 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f8311600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f830c14a380 0x7f830c1f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f8311600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830c150a00 0x7f830c0719d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f8311600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c071f10 0x7f830c1f1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f8311600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c150500 con 0x7f830c150a00 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f8311600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0723a0 con 0x7f830c14a380 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f8311600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830c151270 con 0x7f830c071f10 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f830a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830c150a00 0x7f830c0719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f830aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f830c14a380 0x7f830c1f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f830aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f830c14a380 0x7f830c1f51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:43904/0 (socket says 172.21.15.12:43904) 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f830a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830c150a00 0x7f830c0719d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38832/0 (socket says 172.21.15.12:38832) 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f830aa00700 1 -- 172.21.15.12:0/609383516 learned_addr learned my addr 172.21.15.12:0/609383516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.336+0000 7f830b400700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c071f10 0x7f830c1f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f830a000700 1 -- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c071f10 msgr2=0x7f830c1f1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f830a000700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c071f10 0x7f830c1f1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f830a000700 1 -- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f830c14a380 msgr2=0x7f830c1f51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f830a000700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f830c14a380 0x7f830c1f51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f830a000700 1 -- 172.21.15.12:0/609383516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c156020 con 0x7f830c150a00 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f830a000700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830c150a00 0x7f830c0719d0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f82f8007340 tx=0x7f82f8007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f8302a00700 1 -- 172.21.15.12:0/609383516 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f800f660 con 0x7f830c150a00 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f8302a00700 1 -- 172.21.15.12:0/609383516 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f800f800 con 0x7f830c150a00 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f8311600700 1 -- 172.21.15.12:0/609383516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830c14a760 con 0x7f830c150a00 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f8302a00700 1 -- 172.21.15.12:0/609383516 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f800fbb0 con 0x7f830c150a00 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.337+0000 7f8311600700 1 -- 172.21.15.12:0/609383516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830c14b400 con 0x7f830c150a00 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.338+0000 7f8311600700 1 -- 172.21.15.12:0/609383516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830c14b590 con 0x7f830c150a00 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.339+0000 7f8302a00700 1 -- 172.21.15.12:0/609383516 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f82f800fd10 con 0x7f830c150a00 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.340+0000 7f8302a00700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f82ec06aed0 0x7f82ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.340+0000 7f8302a00700 1 -- 172.21.15.12:0/609383516 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7462+0+0 (secure 0 0 0) 0x7f82f8089de0 con 0x7f830c150a00 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.340+0000 7f830aa00700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f82ec06aed0 0x7f82ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:13.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.341+0000 7f830aa00700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f82ec06aed0 0x7f82ec06d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f830c14de60 tx=0x7f82fc009210 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:13.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.345+0000 7f8302a00700 1 -- 172.21.15.12:0/609383516 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82f8054660 con 0x7f830c150a00 2024-03-29T11:32:13.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:13.559+0000 7f8311600700 1 -- 172.21.15.12:0/609383516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f830c067720 con 0x7f830c150a00 2024-03-29T11:32:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.058+0000 7f8302a00700 1 -- 172.21.15.12:0/609383516 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v235) v1 ==== 181+0+0 (secure 0 0 0) 0x7f82f8057c80 con 0x7f830c150a00 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.061+0000 7f8311600700 1 -- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f82ec06aed0 msgr2=0x7f82ec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.061+0000 7f8311600700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f82ec06aed0 0x7f82ec06d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f830c14de60 tx=0x7f82fc009210 comp rx=0 tx=0).stop 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.061+0000 7f8311600700 1 -- 172.21.15.12:0/609383516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830c150a00 msgr2=0x7f830c0719d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.061+0000 7f8311600700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830c150a00 0x7f830c0719d0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f82f8007340 tx=0x7f82f8007650 comp rx=0 tx=0).stop 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.062+0000 7f8311600700 1 -- 172.21.15.12:0/609383516 shutdown_connections 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.062+0000 7f8311600700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f82ec06aed0 0x7f82ec06d390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.062+0000 7f8311600700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f830c071f10 0x7f830c1f1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.062+0000 7f8311600700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f830c14a380 0x7f830c1f51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.062+0000 7f8311600700 1 --2- 172.21.15.12:0/609383516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830c150a00 0x7f830c0719d0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.062+0000 7f8311600700 1 -- 172.21.15.12:0/609383516 >> 172.21.15.12:0/609383516 conn(0x7f830c0cf210 msgr2=0x7f830c072750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.062+0000 7f8311600700 1 -- 172.21.15.12:0/609383516 shutdown_connections 2024-03-29T11:32:14.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.062+0000 7f8311600700 1 -- 172.21.15.12:0/609383516 wait complete. 2024-03-29T11:32:14.058 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T11:32:14.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T11:32:14.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.285+0000 7f01b0e00700 1 Processor -- start 2024-03-29T11:32:14.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.285+0000 7f01b0e00700 1 -- start start 2024-03-29T11:32:14.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.286+0000 7f01b0e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f01ac15db10 0x7f01ac099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:14.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.286+0000 7f01b0e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01ac0b3930 0x7f01ac099fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:14.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.286+0000 7f01b0e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f01ac09a4e0 0x7f01ac0cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:14.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.286+0000 7f01b0e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ac0cbdf0 con 0x7f01ac09a4e0 2024-03-29T11:32:14.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.286+0000 7f01b0e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ac06bec0 con 0x7f01ac15db10 2024-03-29T11:32:14.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.286+0000 7f01b0e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0b27a0 con 0x7f01ac0b3930 2024-03-29T11:32:14.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.286+0000 7f01aa800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f01ac09a4e0 0x7f01ac0cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:14.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.286+0000 7f01aa800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f01ac09a4e0 0x7f01ac0cf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38844/0 (socket says 172.21.15.12:38844) 2024-03-29T11:32:14.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.286+0000 7f01aa800700 1 -- 172.21.15.12:0/3218744689 learned_addr learned my addr 172.21.15.12:0/3218744689 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:14.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01a9400700 1 --2- 172.21.15.12:0/3218744689 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01ac0b3930 0x7f01ac099fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:14.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01a9e00700 1 --2- 172.21.15.12:0/3218744689 >> v2:172.21.15.159:3300/0 conn(0x7f01ac15db10 0x7f01ac099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:14.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01aa800700 1 -- 172.21.15.12:0/3218744689 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01ac0b3930 msgr2=0x7f01ac099fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:14.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01aa800700 1 --2- 172.21.15.12:0/3218744689 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01ac0b3930 0x7f01ac099fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01aa800700 1 -- 172.21.15.12:0/3218744689 >> v2:172.21.15.159:3300/0 conn(0x7f01ac15db10 msgr2=0x7f01ac099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:14.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01aa800700 1 --2- 172.21.15.12:0/3218744689 >> v2:172.21.15.159:3300/0 conn(0x7f01ac15db10 0x7f01ac099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01aa800700 1 -- 172.21.15.12:0/3218744689 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac06bbe0 con 0x7f01ac09a4e0 2024-03-29T11:32:14.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01aa800700 1 --2- 172.21.15.12:0/3218744689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 0x7f01ac0cf4b0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f01ac072480 tx=0x7f01a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e72691cd8dc9a44 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:14.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01a8a00700 1 -- 172.21.15.12:0/3218744689 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a400e070 con 0x7f01ac09a4e0 2024-03-29T11:32:14.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01a8a00700 1 -- 172.21.15.12:0/3218744689 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a400b9c0 con 0x7f01ac09a4e0 2024-03-29T11:32:14.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.287+0000 7f01a8a00700 1 -- 172.21.15.12:0/3218744689 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a400bd70 con 0x7f01ac09a4e0 2024-03-29T11:32:14.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.288+0000 7f01b0e00700 1 -- 172.21.15.12:0/3218744689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 msgr2=0x7f01ac0cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.288+0000 7f01b0e00700 1 --2- 172.21.15.12:0/3218744689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 0x7f01ac0cf4b0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f01ac072480 tx=0x7f01a400b330 comp rx=0 tx=0).stop 2024-03-29T11:32:14.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.288+0000 7f01b0e00700 1 -- 172.21.15.12:0/3218744689 shutdown_connections 2024-03-29T11:32:14.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.288+0000 7f01b0e00700 1 --2- 172.21.15.12:0/3218744689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 0x7f01ac0cf4b0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.288+0000 7f01b0e00700 1 --2- 172.21.15.12:0/3218744689 >> v2:172.21.15.159:3300/0 conn(0x7f01ac15db10 0x7f01ac099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.288+0000 7f01b0e00700 1 --2- 172.21.15.12:0/3218744689 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01ac0b3930 0x7f01ac099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.288+0000 7f01b0e00700 1 -- 172.21.15.12:0/3218744689 >> 172.21.15.12:0/3218744689 conn(0x7f01ac0b2e10 msgr2=0x7f01ac0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:14.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.288+0000 7f01b0e00700 1 -- 172.21.15.12:0/3218744689 shutdown_connections 2024-03-29T11:32:14.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.288+0000 7f01b0e00700 1 -- 172.21.15.12:0/3218744689 wait complete. 2024-03-29T11:32:14.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.290+0000 7f01b0e00700 1 Processor -- start 2024-03-29T11:32:14.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.291+0000 7f01b0e00700 1 -- start start 2024-03-29T11:32:14.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01b0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 0x7f01ac14e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:14.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01b0e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01ac0b3930 0x7f01ac0c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:14.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01b0e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01ac15db10 0x7f01ac0b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:14.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01b0e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac14ed40 con 0x7f01ac09a4e0 2024-03-29T11:32:14.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01b0e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac15e750 con 0x7f01ac0b3930 2024-03-29T11:32:14.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01b0e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0ca6a0 con 0x7f01ac15db10 2024-03-29T11:32:14.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01a9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 0x7f01ac14e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:14.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01a9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 0x7f01ac14e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38852/0 (socket says 172.21.15.12:38852) 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01a9e00700 1 -- 172.21.15.12:0/71258177 learned_addr learned my addr 172.21.15.12:0/71258177 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01a9400700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01ac0b3930 0x7f01ac0c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.292+0000 7f01aa800700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01ac15db10 0x7f01ac0b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01a9e00700 1 -- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01ac15db10 msgr2=0x7f01ac0b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01a9e00700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01ac15db10 0x7f01ac0b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01a9e00700 1 -- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01ac0b3930 msgr2=0x7f01ac0c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01a9e00700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01ac0b3930 0x7f01ac0c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01a9e00700 1 -- 172.21.15.12:0/71258177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a4009b30 con 0x7f01ac09a4e0 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01a9400700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01ac0b3930 0x7f01ac0c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01a9e00700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 0x7f01ac14e800 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f019c00d800 tx=0x7f019c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01a2000700 1 -- 172.21.15.12:0/71258177 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c0125f0 con 0x7f01ac09a4e0 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01a2000700 1 -- 172.21.15.12:0/71258177 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c012750 con 0x7f01ac09a4e0 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01a2000700 1 -- 172.21.15.12:0/71258177 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c012b00 con 0x7f01ac09a4e0 2024-03-29T11:32:14.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01b0e00700 1 -- 172.21.15.12:0/71258177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ac0ca980 con 0x7f01ac09a4e0 2024-03-29T11:32:14.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.293+0000 7f01b0e00700 1 -- 172.21.15.12:0/71258177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ac15e9f0 con 0x7f01ac09a4e0 2024-03-29T11:32:14.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.294+0000 7f01b0e00700 1 -- 172.21.15.12:0/71258177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ac15ebb0 con 0x7f01ac09a4e0 2024-03-29T11:32:14.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.295+0000 7f01a2000700 1 -- 172.21.15.12:0/71258177 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f019c012c60 con 0x7f01ac09a4e0 2024-03-29T11:32:14.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.296+0000 7f01a2000700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f018c06aed0 0x7f018c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:14.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.296+0000 7f01a2000700 1 -- 172.21.15.12:0/71258177 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7024+0+0 (secure 0 0 0) 0x7f019c089ee0 con 0x7f01ac09a4e0 2024-03-29T11:32:14.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.297+0000 7f01a9400700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f018c06aed0 0x7f018c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:14.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.297+0000 7f01a9400700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f018c06aed0 0x7f018c06d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f01ac15e6e0 tx=0x7f0198008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:14.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.300+0000 7f01a2000700 1 -- 172.21.15.12:0/71258177 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f019c0548d0 con 0x7f01ac09a4e0 2024-03-29T11:32:14.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:14.526+0000 7f01b0e00700 1 -- 172.21.15.12:0/71258177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f01ac067720 con 0x7f01ac09a4e0 2024-03-29T11:32:15.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.315+0000 7f01a2000700 1 -- 172.21.15.12:0/71258177 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v236) v1 ==== 169+0+0 (secure 0 0 0) 0x7f019c057ef0 con 0x7f01ac09a4e0 2024-03-29T11:32:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 -- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f018c06aed0 msgr2=0x7f018c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f018c06aed0 0x7f018c06d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f01ac15e6e0 tx=0x7f0198008040 comp rx=0 tx=0).stop 2024-03-29T11:32:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 -- 172.21.15.12:0/71258177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 msgr2=0x7f01ac14e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 0x7f01ac14e800 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f019c00d800 tx=0x7f019c00dbc0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 -- 172.21.15.12:0/71258177 shutdown_connections 2024-03-29T11:32:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f018c06aed0 0x7f018c06d390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01ac15db10 0x7f01ac0b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01ac09a4e0 0x7f01ac14e800 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 --2- 172.21.15.12:0/71258177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01ac0b3930 0x7f01ac0c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 -- 172.21.15.12:0/71258177 >> 172.21.15.12:0/71258177 conn(0x7f01ac0b2e10 msgr2=0x7f01ac15bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 -- 172.21.15.12:0/71258177 shutdown_connections 2024-03-29T11:32:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.319+0000 7f01b0e00700 1 -- 172.21.15.12:0/71258177 wait complete. 2024-03-29T11:32:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T11:32:15.336 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:15.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.548+0000 7fe0e4200700 1 Processor -- start 2024-03-29T11:32:15.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.549+0000 7fe0e4200700 1 -- start start 2024-03-29T11:32:15.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.550+0000 7fe0e4200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe0dc0cb3f0 0x7fe0dc0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:15.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.550+0000 7fe0e4200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0dc15fd50 con 0x7fe0dc0ac600 2024-03-29T11:32:15.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.550+0000 7fe0e4200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0dc0aca10 con 0x7fe0dc15b5d0 2024-03-29T11:32:15.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.550+0000 7fe0e4200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0dc15b9b0 con 0x7fe0dc0cb3f0 2024-03-29T11:32:15.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.550+0000 7fe0e0e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe0dc0cb3f0 0x7fe0dc0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:15.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.550+0000 7fe0e2200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe0dc0ac600 0x7fe0dc06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46778/0 (socket says 172.21.15.12:46778) 2024-03-29T11:32:15.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.550+0000 7fe0e2200700 1 -- 172.21.15.12:0/2201908994 learned_addr learned my addr 172.21.15.12:0/2201908994 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:15.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.550+0000 7fe0e0e00700 1 -- 172.21.15.12:0/2201908994 >> v1:172.21.15.159:6789/0 conn(0x7fe0dc15b5d0 legacy=0x7fe0dc0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.550+0000 7fe0e0e00700 1 -- 172.21.15.12:0/2201908994 >> v1:172.21.15.12:6789/0 conn(0x7fe0dc0ac600 legacy=0x7fe0dc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.551+0000 7fe0e0e00700 1 -- 172.21.15.12:0/2201908994 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc15f140 con 0x7fe0dc0cb3f0 2024-03-29T11:32:15.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.551+0000 7fe0dbe00700 1 -- 172.21.15.12:0/2201908994 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232028753 0 0) 0x7fe0dc15fd50 con 0x7fe0dc0ac600 2024-03-29T11:32:15.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.551+0000 7fe0e0e00700 1 --2- 172.21.15.12:0/2201908994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc0cb3f0 0x7fe0dc0ac0c0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009840 tx=0x7fe0c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc04626bb7a4d755 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:15.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.551+0000 7fe0dbe00700 1 -- 172.21.15.12:0/2201908994 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8011830 con 0x7fe0dc0cb3f0 2024-03-29T11:32:15.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.551+0000 7fe0dbe00700 1 -- 172.21.15.12:0/2201908994 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c8011990 con 0x7fe0dc0cb3f0 2024-03-29T11:32:15.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.551+0000 7fe0dbe00700 1 -- 172.21.15.12:0/2201908994 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8011d20 con 0x7fe0dc0cb3f0 2024-03-29T11:32:15.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.552+0000 7fe0e4200700 1 -- 172.21.15.12:0/2201908994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc0cb3f0 msgr2=0x7fe0dc0ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.552+0000 7fe0e4200700 1 --2- 172.21.15.12:0/2201908994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc0cb3f0 0x7fe0dc0ac0c0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009840 tx=0x7fe0c8009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:15.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.552+0000 7fe0e4200700 1 -- 172.21.15.12:0/2201908994 shutdown_connections 2024-03-29T11:32:15.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.552+0000 7fe0e4200700 1 --2- 172.21.15.12:0/2201908994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc0cb3f0 0x7fe0dc0ac0c0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.552+0000 7fe0e4200700 1 -- 172.21.15.12:0/2201908994 >> 172.21.15.12:0/2201908994 conn(0x7fe0dc0cf190 msgr2=0x7fe0dc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:15.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.552+0000 7fe0e4200700 1 -- 172.21.15.12:0/2201908994 shutdown_connections 2024-03-29T11:32:15.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.552+0000 7fe0e4200700 1 -- 172.21.15.12:0/2201908994 wait complete. 2024-03-29T11:32:15.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.553+0000 7fe0e4200700 1 Processor -- start 2024-03-29T11:32:15.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.554+0000 7fe0e4200700 1 -- start start 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.554+0000 7fe0e4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0dc0ac600 0x7fe0dc1f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.554+0000 7fe0e4200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0dc15b5d0 0x7fe0dc0cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e4200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc1f39f0 0x7fe0dc0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e4200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc1f3f40 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e4200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc1f6700 con 0x7fe0dc15b5d0 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc1f69e0 con 0x7fe0dc0ac600 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc1f39f0 0x7fe0dc0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc1f39f0 0x7fe0dc0cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38868/0 (socket says 172.21.15.12:38868) 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e2200700 1 -- 172.21.15.12:0/594932386 learned_addr learned my addr 172.21.15.12:0/594932386 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e0e00700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0dc15b5d0 0x7fe0dc0cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e1800700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0dc0ac600 0x7fe0dc1f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e2200700 1 -- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0dc0ac600 msgr2=0x7fe0dc1f03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e2200700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0dc0ac600 0x7fe0dc1f03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e2200700 1 -- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0dc15b5d0 msgr2=0x7fe0dc0cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e2200700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0dc15b5d0 0x7fe0dc0cd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e2200700 1 -- 172.21.15.12:0/594932386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80094a0 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e1800700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0dc0ac600 0x7fe0dc1f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.555+0000 7fe0e0e00700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0dc15b5d0 0x7fe0dc0cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.556+0000 7fe0e2200700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc1f39f0 0x7fe0dc0cda50 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400cb60 tx=0x7fe0d400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.556+0000 7fe0d9600700 1 -- 172.21.15.12:0/594932386 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d400ab50 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.556+0000 7fe0d9600700 1 -- 172.21.15.12:0/594932386 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d400acb0 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.556+0000 7fe0e4200700 1 -- 172.21.15.12:0/594932386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0dc1f6cc0 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.556+0000 7fe0d9600700 1 -- 172.21.15.12:0/594932386 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4007310 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.556+0000 7fe0e4200700 1 -- 172.21.15.12:0/594932386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0dc1f08f0 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.557+0000 7fe0e4200700 1 -- 172.21.15.12:0/594932386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0dc1f71e0 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.558+0000 7fe0d9600700 1 -- 172.21.15.12:0/594932386 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe0d4007470 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.559+0000 7fe0d9600700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0bc06aed0 0x7fe0bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:15.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.559+0000 7fe0d9600700 1 -- 172.21.15.12:0/594932386 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6610+0+0 (secure 0 0 0) 0x7fe0d4089940 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.559+0000 7fe0e1800700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0bc06aed0 0x7fe0bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:15.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.560+0000 7fe0e1800700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0bc06aed0 0x7fe0bc06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0000f60 tx=0x7fe0d0009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:15.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.564+0000 7fe0d9600700 1 -- 172.21.15.12:0/594932386 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe0d4054390 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.775+0000 7fe0e4200700 1 -- 172.21.15.12:0/594932386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0dc067720 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.777+0000 7fe0d9600700 1 -- 172.21.15.12:0/594932386 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+17432 (secure 0 0 0) 0x7fe0d4087e60 con 0x7fe0dc1f39f0 2024-03-29T11:32:15.772 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:15.772 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":236,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:15.304574+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:31:39.075407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:31:39.845704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6833/2722218561":"2024-03-30T11:32:12.038796+0000","172.21.15.159:0/4178225398":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1623528933":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1432915605":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/2328654185":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3030066758":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/2640388148":"2024-03-30T11:32:08.498738+0000","172.21.15.12:6832/2722218561":"2024-03-30T11:32:12.038796+0000","172.21.15.159:0/1069287393":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/353804238":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6833/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6832/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3242066681":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3562663283":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3628749078":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/235474344":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3837208470":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/1750949668":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/629137162":"2024-03-30T11:31:55.626759+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:15.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.779+0000 7fe0e4200700 1 -- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0bc06aed0 msgr2=0x7fe0bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.779+0000 7fe0e4200700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0bc06aed0 0x7fe0bc06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0000f60 tx=0x7fe0d0009040 comp rx=0 tx=0).stop 2024-03-29T11:32:15.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.779+0000 7fe0e4200700 1 -- 172.21.15.12:0/594932386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc1f39f0 msgr2=0x7fe0dc0cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.779+0000 7fe0e4200700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc1f39f0 0x7fe0dc0cda50 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400cb60 tx=0x7fe0d400cf20 comp rx=0 tx=0).stop 2024-03-29T11:32:15.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.779+0000 7fe0e4200700 1 -- 172.21.15.12:0/594932386 shutdown_connections 2024-03-29T11:32:15.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.779+0000 7fe0e4200700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0bc06aed0 0x7fe0bc06d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.779+0000 7fe0e4200700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0dc1f39f0 0x7fe0dc0cda50 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.779+0000 7fe0e4200700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0dc0ac600 0x7fe0dc1f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.779+0000 7fe0e4200700 1 --2- 172.21.15.12:0/594932386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0dc15b5d0 0x7fe0dc0cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.780+0000 7fe0e4200700 1 -- 172.21.15.12:0/594932386 >> 172.21.15.12:0/594932386 conn(0x7fe0dc0cf190 msgr2=0x7fe0dc0aefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:15.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.780+0000 7fe0e4200700 1 -- 172.21.15.12:0/594932386 shutdown_connections 2024-03-29T11:32:15.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.780+0000 7fe0e4200700 1 -- 172.21.15.12:0/594932386 wait complete. 2024-03-29T11:32:15.794 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-29T11:31:39.075407+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-29T11:31:39.845704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:32:15.794 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:32:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:15.999+0000 7f39b4c00700 1 Processor -- start 2024-03-29T11:32:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.000+0000 7f39b4c00700 1 -- start start 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.001+0000 7f39b4c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f39b00d21f0 0x7f39b00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.001+0000 7f39b4c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39b00cec40 0x7f39b0161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.001+0000 7f39b4c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00ce270 con 0x7f39b01611b0 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.001+0000 7f39b4c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b015b600 con 0x7f39b00cec40 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.001+0000 7f39b4c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b00d2760 con 0x7f39b00d21f0 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.001+0000 7f39ae800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39b00cec40 0x7f39b0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.001+0000 7f39ade00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f39b01611b0 0x7f39b015ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46784/0 (socket says 172.21.15.12:46784) 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.001+0000 7f39ade00700 1 -- 172.21.15.12:0/363695193 learned_addr learned my addr 172.21.15.12:0/363695193 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.001+0000 7f39ad400700 1 --2- 172.21.15.12:0/363695193 >> v2:172.21.15.159:3300/0 conn(0x7f39b00d21f0 0x7f39b00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.002+0000 7f39ae800700 1 -- 172.21.15.12:0/363695193 >> v2:172.21.15.159:3300/0 conn(0x7f39b00d21f0 msgr2=0x7f39b00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:15.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.002+0000 7f39ae800700 1 --2- 172.21.15.12:0/363695193 >> v2:172.21.15.159:3300/0 conn(0x7f39b00d21f0 0x7f39b00ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.002+0000 7f39ae800700 1 -- 172.21.15.12:0/363695193 >> v1:172.21.15.12:6789/0 conn(0x7f39b01611b0 legacy=0x7f39b015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.002+0000 7f39ae800700 1 -- 172.21.15.12:0/363695193 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b00d2a40 con 0x7f39b00cec40 2024-03-29T11:32:15.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.002+0000 7f39aca00700 1 -- 172.21.15.12:0/363695193 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245256879 0 0) 0x7f39b00ce270 con 0x7f39b01611b0 2024-03-29T11:32:15.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.002+0000 7f39ae800700 1 --2- 172.21.15.12:0/363695193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00cec40 0x7f39b0161590 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f39b008c030 tx=0x7f39a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c9bcbbc5e6d88d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:15.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.002+0000 7f39aca00700 1 -- 172.21.15.12:0/363695193 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a800e070 con 0x7f39b00cec40 2024-03-29T11:32:15.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.002+0000 7f39aca00700 1 -- 172.21.15.12:0/363695193 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a800bb20 con 0x7f39b00cec40 2024-03-29T11:32:15.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.002+0000 7f39aca00700 1 -- 172.21.15.12:0/363695193 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8012430 con 0x7f39b00cec40 2024-03-29T11:32:15.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.003+0000 7f39b4c00700 1 -- 172.21.15.12:0/363695193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00cec40 msgr2=0x7f39b0161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.003+0000 7f39b4c00700 1 --2- 172.21.15.12:0/363695193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00cec40 0x7f39b0161590 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f39b008c030 tx=0x7f39a800b330 comp rx=0 tx=0).stop 2024-03-29T11:32:15.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.003+0000 7f39b4c00700 1 -- 172.21.15.12:0/363695193 shutdown_connections 2024-03-29T11:32:15.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.003+0000 7f39b4c00700 1 --2- 172.21.15.12:0/363695193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00cec40 0x7f39b0161590 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.003+0000 7f39b4c00700 1 --2- 172.21.15.12:0/363695193 >> v2:172.21.15.159:3300/0 conn(0x7f39b00d21f0 0x7f39b00ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:15.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.003+0000 7f39b4c00700 1 -- 172.21.15.12:0/363695193 >> 172.21.15.12:0/363695193 conn(0x7f39b008aff0 msgr2=0x7f39b00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:15.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.003+0000 7f39b4c00700 1 -- 172.21.15.12:0/363695193 shutdown_connections 2024-03-29T11:32:15.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.003+0000 7f39b4c00700 1 -- 172.21.15.12:0/363695193 wait complete. 2024-03-29T11:32:16.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.005+0000 7f39b4c00700 1 Processor -- start 2024-03-29T11:32:16.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.005+0000 7f39b4c00700 1 -- start start 2024-03-29T11:32:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39b4c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39b00cec40 0x7f39b0156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39b4c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00d21f0 0x7f39b006c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39b4c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39b01611b0 0x7f39b006c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39b4c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b00b72d0 con 0x7f39b00d21f0 2024-03-29T11:32:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39b4c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b0157020 con 0x7f39b00cec40 2024-03-29T11:32:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39b4c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b00b6650 con 0x7f39b01611b0 2024-03-29T11:32:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39ad400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00d21f0 0x7f39b006c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39ad400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00d21f0 0x7f39b006c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38890/0 (socket says 172.21.15.12:38890) 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39ad400700 1 -- 172.21.15.12:0/4196006664 learned_addr learned my addr 172.21.15.12:0/4196006664 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39ade00700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39b00cec40 0x7f39b0156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39ae800700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39b01611b0 0x7f39b006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39ad400700 1 -- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39b01611b0 msgr2=0x7f39b006c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39ad400700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39b01611b0 0x7f39b006c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39ad400700 1 -- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39b00cec40 msgr2=0x7f39b0156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39ad400700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39b00cec40 0x7f39b0156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.006+0000 7f39ad400700 1 -- 172.21.15.12:0/4196006664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8009a70 con 0x7f39b00d21f0 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.007+0000 7f39ad400700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00d21f0 0x7f39b006c320 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f399c00e9b0 tx=0x7f399c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.007+0000 7f39a6000700 1 -- 172.21.15.12:0/4196006664 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c009710 con 0x7f39b00d21f0 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.007+0000 7f39a6000700 1 -- 172.21.15.12:0/4196006664 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c00c850 con 0x7f39b00d21f0 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.007+0000 7f39b4c00700 1 -- 172.21.15.12:0/4196006664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b00b6930 con 0x7f39b00d21f0 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.007+0000 7f39a6000700 1 -- 172.21.15.12:0/4196006664 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c00cbc0 con 0x7f39b00d21f0 2024-03-29T11:32:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.007+0000 7f39b4c00700 1 -- 172.21.15.12:0/4196006664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b00b6ef0 con 0x7f39b00d21f0 2024-03-29T11:32:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.008+0000 7f39b4c00700 1 -- 172.21.15.12:0/4196006664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b00b7410 con 0x7f39b00d21f0 2024-03-29T11:32:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.009+0000 7f39a6000700 1 -- 172.21.15.12:0/4196006664 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f399c00cd20 con 0x7f39b00d21f0 2024-03-29T11:32:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.010+0000 7f39a6000700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f399006aed0 0x7f399006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.010+0000 7f39a6000700 1 -- 172.21.15.12:0/4196006664 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6610+0+0 (secure 0 0 0) 0x7f399c089b50 con 0x7f39b00d21f0 2024-03-29T11:32:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.011+0000 7f39ade00700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f399006aed0 0x7f399006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.011+0000 7f39ade00700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f399006aed0 0x7f399006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f39a0000f60 tx=0x7f39a0007c90 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:16.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.014+0000 7f39a6000700 1 -- 172.21.15.12:0/4196006664 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39b00b7410 con 0x7f39b00d21f0 2024-03-29T11:32:16.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.229+0000 7f39b4c00700 1 -- 172.21.15.12:0/4196006664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39b00b7410 con 0x7f39b00d21f0 2024-03-29T11:32:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.229+0000 7f39a6000700 1 -- 172.21.15.12:0/4196006664 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v236) v1 ==== 93+0+10 (secure 0 0 0) 0x7f399c021020 con 0x7f39b00d21f0 2024-03-29T11:32:16.246 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:32:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 -- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f399006aed0 msgr2=0x7f399006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f399006aed0 0x7f399006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f39a0000f60 tx=0x7f39a0007c90 comp rx=0 tx=0).stop 2024-03-29T11:32:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 -- 172.21.15.12:0/4196006664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00d21f0 msgr2=0x7f39b006c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00d21f0 0x7f39b006c320 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f399c00e9b0 tx=0x7f399c00ed70 comp rx=0 tx=0).stop 2024-03-29T11:32:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 -- 172.21.15.12:0/4196006664 shutdown_connections 2024-03-29T11:32:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f399006aed0 0x7f399006d390 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39b01611b0 0x7f39b006c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39b00cec40 0x7f39b0156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 --2- 172.21.15.12:0/4196006664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39b00d21f0 0x7f39b006c320 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 -- 172.21.15.12:0/4196006664 >> 172.21.15.12:0/4196006664 conn(0x7f39b008aff0 msgr2=0x7f39b006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 -- 172.21.15.12:0/4196006664 shutdown_connections 2024-03-29T11:32:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.232+0000 7f39b4c00700 1 -- 172.21.15.12:0/4196006664 wait complete. 2024-03-29T11:32:16.252 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-29T11:32:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.464+0000 7fd0ff200700 1 Processor -- start 2024-03-29T11:32:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.464+0000 7fd0ff200700 1 -- start start 2024-03-29T11:32:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.465+0000 7fd0ff200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd0f815d9d0 0x7fd0f80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.465+0000 7fd0ff200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f80d2540 0x7fd0f80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.465+0000 7fd0ff200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f8161420 con 0x7fd0f80d16a0 2024-03-29T11:32:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.465+0000 7fd0ff200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f815edc0 con 0x7fd0f815d9d0 2024-03-29T11:32:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.465+0000 7fd0ff200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f8161700 con 0x7fd0f80d2540 2024-03-29T11:32:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.466+0000 7fd0fd200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f80d2540 0x7fd0f80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.466+0000 7fd0f7e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd0f815d9d0 0x7fd0f80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.466+0000 7fd0fd200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f80d2540 0x7fd0f80ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:50636/0 (socket says 172.21.15.12:50636) 2024-03-29T11:32:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.466+0000 7fd0fd200700 1 -- 172.21.15.12:0/3777682479 learned_addr learned my addr 172.21.15.12:0/3777682479 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.466+0000 7fd0fd200700 1 -- 172.21.15.12:0/3777682479 >> v2:172.21.15.159:3300/0 conn(0x7fd0f815d9d0 msgr2=0x7fd0f80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.466+0000 7fd0fd200700 1 --2- 172.21.15.12:0/3777682479 >> v2:172.21.15.159:3300/0 conn(0x7fd0f815d9d0 0x7fd0f80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.466+0000 7fd0fd200700 1 -- 172.21.15.12:0/3777682479 >> v1:172.21.15.159:6789/0 conn(0x7fd0f80d16a0 legacy=0x7fd0f80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.466+0000 7fd0fd200700 1 -- 172.21.15.12:0/3777682479 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f81619e0 con 0x7fd0f80d2540 2024-03-29T11:32:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.466+0000 7fd0fd200700 1 --2- 172.21.15.12:0/3777682479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f80d2540 0x7fd0f80ce360 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0009e50 tx=0x7fd0f000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12e335683a023ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0f6a00700 1 -- 172.21.15.12:0/3777682479 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f000e070 con 0x7fd0f80d2540 2024-03-29T11:32:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0f6a00700 1 -- 172.21.15.12:0/3777682479 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f000b910 con 0x7fd0f80d2540 2024-03-29T11:32:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0f6a00700 1 -- 172.21.15.12:0/3777682479 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f000bcc0 con 0x7fd0f80d2540 2024-03-29T11:32:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0ff200700 1 -- 172.21.15.12:0/3777682479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f80d2540 msgr2=0x7fd0f80ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0ff200700 1 --2- 172.21.15.12:0/3777682479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f80d2540 0x7fd0f80ce360 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0009e50 tx=0x7fd0f000b330 comp rx=0 tx=0).stop 2024-03-29T11:32:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0ff200700 1 -- 172.21.15.12:0/3777682479 shutdown_connections 2024-03-29T11:32:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0ff200700 1 --2- 172.21.15.12:0/3777682479 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f80d2540 0x7fd0f80ce360 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0ff200700 1 --2- 172.21.15.12:0/3777682479 >> v2:172.21.15.159:3300/0 conn(0x7fd0f815d9d0 0x7fd0f80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0ff200700 1 -- 172.21.15.12:0/3777682479 >> 172.21.15.12:0/3777682479 conn(0x7fd0f808aff0 msgr2=0x7fd0f80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0ff200700 1 -- 172.21.15.12:0/3777682479 shutdown_connections 2024-03-29T11:32:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.467+0000 7fd0ff200700 1 -- 172.21.15.12:0/3777682479 wait complete. 2024-03-29T11:32:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.469+0000 7fd0ff200700 1 Processor -- start 2024-03-29T11:32:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.469+0000 7fd0ff200700 1 -- start start 2024-03-29T11:32:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.470+0000 7fd0ff200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0f80d16a0 0x7fd0f815f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.470+0000 7fd0ff200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0f80d2540 0x7fd0f8152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.470+0000 7fd0ff200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f815d9d0 0x7fd0f81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.470+0000 7fd0ff200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f80b90d0 con 0x7fd0f80d16a0 2024-03-29T11:32:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.470+0000 7fd0ff200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f80b80c0 con 0x7fd0f80d2540 2024-03-29T11:32:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.470+0000 7fd0ff200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f80b8310 con 0x7fd0f815d9d0 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0f80d16a0 0x7fd0f815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0f80d16a0 0x7fd0f815f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38906/0 (socket says 172.21.15.12:38906) 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7e00700 1 -- 172.21.15.12:0/1348754986 learned_addr learned my addr 172.21.15.12:0/1348754986 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0fd200700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f815d9d0 0x7fd0f81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7e00700 1 -- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f815d9d0 msgr2=0x7fd0f81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7e00700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f815d9d0 0x7fd0f81529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7e00700 1 -- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0f80d2540 msgr2=0x7fd0f8152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7400700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0f80d2540 0x7fd0f8152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7e00700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0f80d2540 0x7fd0f8152480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7e00700 1 -- 172.21.15.12:0/1348754986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f0009b30 con 0x7fd0f80d16a0 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0fd200700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f815d9d0 0x7fd0f81529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7400700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0f80d2540 0x7fd0f8152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.471+0000 7fd0f7e00700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0f80d16a0 0x7fd0f815f920 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800e870 tx=0x7fd0e800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.472+0000 7fd0efe00700 1 -- 172.21.15.12:0/1348754986 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8009590 con 0x7fd0f80d16a0 2024-03-29T11:32:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.472+0000 7fd0efe00700 1 -- 172.21.15.12:0/1348754986 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e800ee70 con 0x7fd0f80d16a0 2024-03-29T11:32:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.472+0000 7fd0efe00700 1 -- 172.21.15.12:0/1348754986 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e800ca50 con 0x7fd0f80d16a0 2024-03-29T11:32:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.472+0000 7fd0ff200700 1 -- 172.21.15.12:0/1348754986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f80b85f0 con 0x7fd0f80d16a0 2024-03-29T11:32:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.472+0000 7fd0ff200700 1 -- 172.21.15.12:0/1348754986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f80b8bb0 con 0x7fd0f80d16a0 2024-03-29T11:32:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.473+0000 7fd0ff200700 1 -- 172.21.15.12:0/1348754986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f80d1ca0 con 0x7fd0f80d16a0 2024-03-29T11:32:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.474+0000 7fd0efe00700 1 -- 172.21.15.12:0/1348754986 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd0e800cbb0 con 0x7fd0f80d16a0 2024-03-29T11:32:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.475+0000 7fd0efe00700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0d806aed0 0x7fd0d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.475+0000 7fd0efe00700 1 -- 172.21.15.12:0/1348754986 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6610+0+0 (secure 0 0 0) 0x7fd0e8089a50 con 0x7fd0f80d16a0 2024-03-29T11:32:16.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.475+0000 7fd0f7400700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0d806aed0 0x7fd0d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.476+0000 7fd0f7400700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0d806aed0 0x7fd0d806d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd0f815d3e0 tx=0x7fd0e4009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:16.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.479+0000 7fd0efe00700 1 -- 172.21.15.12:0/1348754986 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0e8054530 con 0x7fd0f80d16a0 2024-03-29T11:32:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.699+0000 7fd0ff200700 1 -- 172.21.15.12:0/1348754986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd0f8067720 con 0x7fd0f80d16a0 2024-03-29T11:32:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.699+0000 7fd0efe00700 1 -- 172.21.15.12:0/1348754986 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v236) v1 ==== 107+0+11 (secure 0 0 0) 0x7fd0e800ce60 con 0x7fd0f80d16a0 2024-03-29T11:32:16.694 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:32:16.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 -- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0d806aed0 msgr2=0x7fd0d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0d806aed0 0x7fd0d806d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd0f815d3e0 tx=0x7fd0e4009040 comp rx=0 tx=0).stop 2024-03-29T11:32:16.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 -- 172.21.15.12:0/1348754986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0f80d16a0 msgr2=0x7fd0f815f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0f80d16a0 0x7fd0f815f920 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800e870 tx=0x7fd0e800ec30 comp rx=0 tx=0).stop 2024-03-29T11:32:16.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 -- 172.21.15.12:0/1348754986 shutdown_connections 2024-03-29T11:32:16.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0d806aed0 0x7fd0d806d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0f815d9d0 0x7fd0f81529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0f80d16a0 0x7fd0f815f920 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 --2- 172.21.15.12:0/1348754986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0f80d2540 0x7fd0f8152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 -- 172.21.15.12:0/1348754986 >> 172.21.15.12:0/1348754986 conn(0x7fd0f808aff0 msgr2=0x7fd0f815ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 -- 172.21.15.12:0/1348754986 shutdown_connections 2024-03-29T11:32:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.702+0000 7fd0ff200700 1 -- 172.21.15.12:0/1348754986 wait complete. 2024-03-29T11:32:16.715 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-29T11:32:16.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.962+0000 7fd781800700 1 Processor -- start 2024-03-29T11:32:16.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.962+0000 7fd781800700 1 -- start start 2024-03-29T11:32:16.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.964+0000 7fd781800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c15ef70 0x7fd77c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.964+0000 7fd781800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd77c06a790 0x7fd77c06ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.964+0000 7fd781800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77c161150 con 0x7fd77c160d70 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.964+0000 7fd781800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77c06b180 con 0x7fd77c06a790 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.964+0000 7fd781800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c0b4cd0 con 0x7fd77c15ef70 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.964+0000 7fd77aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd77c160d70 0x7fd77c15e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46798/0 (socket says 172.21.15.12:46798) 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.964+0000 7fd77aa00700 1 -- 172.21.15.12:0/2167913474 learned_addr learned my addr 172.21.15.12:0/2167913474 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.964+0000 7fd773e00700 1 --2- 172.21.15.12:0/2167913474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c15ef70 0x7fd77c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.964+0000 7fd77b400700 1 --2- 172.21.15.12:0/2167913474 >> v2:172.21.15.159:3300/0 conn(0x7fd77c06a790 0x7fd77c06ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.964+0000 7fd77a000700 1 -- 172.21.15.12:0/2167913474 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128843528 0 0) 0x7fd77c161150 con 0x7fd77c160d70 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.965+0000 7fd77a000700 1 -- 172.21.15.12:0/2167913474 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7600035b0 con 0x7fd77c160d70 2024-03-29T11:32:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.965+0000 7fd77b400700 1 -- 172.21.15.12:0/2167913474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c15ef70 msgr2=0x7fd77c0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:16.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.965+0000 7fd77b400700 1 --2- 172.21.15.12:0/2167913474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c15ef70 0x7fd77c0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.965+0000 7fd77b400700 1 -- 172.21.15.12:0/2167913474 >> v1:172.21.15.12:6789/0 conn(0x7fd77c160d70 legacy=0x7fd77c15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.965+0000 7fd77b400700 1 -- 172.21.15.12:0/2167913474 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c06dba0 con 0x7fd77c06a790 2024-03-29T11:32:16.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.965+0000 7fd773e00700 1 --2- 172.21.15.12:0/2167913474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c15ef70 0x7fd77c0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.965+0000 7fd77b400700 1 --2- 172.21.15.12:0/2167913474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77c06a790 0x7fd77c06ac40 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd774009d90 tx=0x7fd77400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6998584f7b3bea0a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.965+0000 7fd77a000700 1 -- 172.21.15.12:0/2167913474 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77400e070 con 0x7fd77c06a790 2024-03-29T11:32:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.966+0000 7fd77a000700 1 -- 172.21.15.12:0/2167913474 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77400bb20 con 0x7fd77c06a790 2024-03-29T11:32:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.966+0000 7fd77a000700 1 -- 172.21.15.12:0/2167913474 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd774012430 con 0x7fd77c06a790 2024-03-29T11:32:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.966+0000 7fd781800700 1 -- 172.21.15.12:0/2167913474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77c06a790 msgr2=0x7fd77c06ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.966+0000 7fd781800700 1 --2- 172.21.15.12:0/2167913474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77c06a790 0x7fd77c06ac40 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd774009d90 tx=0x7fd77400b330 comp rx=0 tx=0).stop 2024-03-29T11:32:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.966+0000 7fd781800700 1 -- 172.21.15.12:0/2167913474 shutdown_connections 2024-03-29T11:32:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.966+0000 7fd781800700 1 --2- 172.21.15.12:0/2167913474 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77c06a790 0x7fd77c06ac40 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.966+0000 7fd781800700 1 --2- 172.21.15.12:0/2167913474 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c15ef70 0x7fd77c0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.966+0000 7fd781800700 1 -- 172.21.15.12:0/2167913474 >> 172.21.15.12:0/2167913474 conn(0x7fd77c0b5720 msgr2=0x7fd77c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.966+0000 7fd781800700 1 -- 172.21.15.12:0/2167913474 shutdown_connections 2024-03-29T11:32:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.966+0000 7fd781800700 1 -- 172.21.15.12:0/2167913474 wait complete. 2024-03-29T11:32:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.968+0000 7fd781800700 1 Processor -- start 2024-03-29T11:32:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.968+0000 7fd781800700 1 -- start start 2024-03-29T11:32:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd781800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77c06a790 0x7fd77c13c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd781800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77c15ef70 0x7fd77c142c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd781800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c160d70 0x7fd77c15fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd781800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c143160 con 0x7fd77c06a790 2024-03-29T11:32:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd781800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c13cf10 con 0x7fd77c15ef70 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd781800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c13d1f0 con 0x7fd77c160d70 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd77aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77c06a790 0x7fd77c13c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd77aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77c06a790 0x7fd77c13c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38922/0 (socket says 172.21.15.12:38922) 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd77aa00700 1 -- 172.21.15.12:0/1918340346 learned_addr learned my addr 172.21.15.12:0/1918340346 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd773e00700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77c15ef70 0x7fd77c142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.969+0000 7fd77b400700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c160d70 0x7fd77c15fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd77aa00700 1 -- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c160d70 msgr2=0x7fd77c15fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd77aa00700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c160d70 0x7fd77c15fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd77aa00700 1 -- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77c15ef70 msgr2=0x7fd77c142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd77aa00700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77c15ef70 0x7fd77c142c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd77aa00700 1 -- 172.21.15.12:0/1918340346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7640094a0 con 0x7fd77c06a790 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd77aa00700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77c06a790 0x7fd77c13c880 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fd76c001eb0 tx=0x7fd76c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd772a00700 1 -- 172.21.15.12:0/1918340346 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c005d70 con 0x7fd77c06a790 2024-03-29T11:32:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd772a00700 1 -- 172.21.15.12:0/1918340346 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76c0070d0 con 0x7fd77c06a790 2024-03-29T11:32:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd781800700 1 -- 172.21.15.12:0/1918340346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd774009a70 con 0x7fd77c06a790 2024-03-29T11:32:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd772a00700 1 -- 172.21.15.12:0/1918340346 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c007460 con 0x7fd77c06a790 2024-03-29T11:32:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.970+0000 7fd781800700 1 -- 172.21.15.12:0/1918340346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77c13d7b0 con 0x7fd77c06a790 2024-03-29T11:32:16.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.971+0000 7fd781800700 1 -- 172.21.15.12:0/1918340346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77c067720 con 0x7fd77c06a790 2024-03-29T11:32:16.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.972+0000 7fd772a00700 1 -- 172.21.15.12:0/1918340346 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd76c0075c0 con 0x7fd77c06a790 2024-03-29T11:32:16.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.973+0000 7fd772a00700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd75806aed0 0x7fd75806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:16.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.973+0000 7fd772a00700 1 -- 172.21.15.12:0/1918340346 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6610+0+0 (secure 0 0 0) 0x7fd76c088f60 con 0x7fd77c06a790 2024-03-29T11:32:16.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.974+0000 7fd773e00700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd75806aed0 0x7fd75806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:16.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.975+0000 7fd773e00700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd75806aed0 0x7fd75806d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd77c066b70 tx=0x7fd764009210 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:16.978+0000 7fd772a00700 1 -- 172.21.15.12:0/1918340346 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd76c0539c0 con 0x7fd77c06a790 2024-03-29T11:32:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.191+0000 7fd781800700 1 -- 172.21.15.12:0/1918340346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fd77c09a390 con 0x7fd77c06a790 2024-03-29T11:32:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.192+0000 7fd772a00700 1 -- 172.21.15.12:0/1918340346 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v236) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd76c056fe0 con 0x7fd77c06a790 2024-03-29T11:32:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:32:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 -- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd75806aed0 msgr2=0x7fd75806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd75806aed0 0x7fd75806d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd77c066b70 tx=0x7fd764009210 comp rx=0 tx=0).stop 2024-03-29T11:32:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 -- 172.21.15.12:0/1918340346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77c06a790 msgr2=0x7fd77c13c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77c06a790 0x7fd77c13c880 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fd76c001eb0 tx=0x7fd76c00c7b0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 -- 172.21.15.12:0/1918340346 shutdown_connections 2024-03-29T11:32:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd75806aed0 0x7fd75806d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd77c160d70 0x7fd77c15fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77c06a790 0x7fd77c13c880 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 --2- 172.21.15.12:0/1918340346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd77c15ef70 0x7fd77c142c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 -- 172.21.15.12:0/1918340346 >> 172.21.15.12:0/1918340346 conn(0x7fd77c0b5720 msgr2=0x7fd77c09aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.194+0000 7fd781800700 1 -- 172.21.15.12:0/1918340346 shutdown_connections 2024-03-29T11:32:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.195+0000 7fd781800700 1 -- 172.21.15.12:0/1918340346 wait complete. 2024-03-29T11:32:17.208 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.415+0000 7fa676400700 1 Processor -- start 2024-03-29T11:32:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.415+0000 7fa676400700 1 -- start start 2024-03-29T11:32:17.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.416+0000 7fa676400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6700d18b0 0x7fa6700cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.416+0000 7fa676400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa670161590 0x7fa670161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.416+0000 7fa676400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6700ceda0 con 0x7fa670160f10 2024-03-29T11:32:17.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.417+0000 7fa676400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6700d1d30 con 0x7fa670161590 2024-03-29T11:32:17.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.417+0000 7fa676400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa670161f60 con 0x7fa6700d18b0 2024-03-29T11:32:17.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.417+0000 7fa66f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa670160f10 0x7fa6700d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46804/0 (socket says 172.21.15.12:46804) 2024-03-29T11:32:17.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.417+0000 7fa66f400700 1 -- 172.21.15.12:0/1826483097 learned_addr learned my addr 172.21.15.12:0/1826483097 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.417+0000 7fa66ea00700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6700d18b0 0x7fa6700cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.417+0000 7fa66fe00700 1 --2- 172.21.15.12:0/1826483097 >> v2:172.21.15.159:3300/0 conn(0x7fa670161590 0x7fa670161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.417+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585358113 0 0) 0x7fa6700ceda0 con 0x7fa670160f10 2024-03-29T11:32:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.417+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6540035b0 con 0x7fa670160f10 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3851875776 0 0) 0x7fa6540035b0 con 0x7fa670160f10 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6700d18b0 msgr2=0x7fa6700cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6700d18b0 0x7fa6700cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 >> v2:172.21.15.159:3300/0 conn(0x7fa670161590 msgr2=0x7fa670161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 --2- 172.21.15.12:0/1826483097 >> v2:172.21.15.159:3300/0 conn(0x7fa670161590 0x7fa670161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6701612f0 con 0x7fa670160f10 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fa660001670 con 0x7fa670160f10 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66fe00700 1 --2- 172.21.15.12:0/1826483097 >> v2:172.21.15.159:3300/0 conn(0x7fa670161590 0x7fa670161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 >> v1:172.21.15.12:6789/0 conn(0x7fa670160f10 legacy=0x7fa6700d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 0x7fa654003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6700d18b0 0x7fa654004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa654004d90 0x7fa654005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6700d1d30 con 0x7fa654003a00 2024-03-29T11:32:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa670161f60 con 0x7fa654004d90 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65400ba40 con 0x7fa6700d18b0 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66f400700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 0x7fa654003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66ea00700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6700d18b0 0x7fa654004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66fe00700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa654004d90 0x7fa654005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66f400700 1 -- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6700d18b0 msgr2=0x7fa654004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66f400700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6700d18b0 0x7fa654004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66f400700 1 -- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa654004d90 msgr2=0x7fa654005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66f400700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa654004d90 0x7fa654005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.418+0000 7fa66f400700 1 -- 172.21.15.12:0/1826483097 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6540035b0 con 0x7fa654003a00 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.419+0000 7fa66f400700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 0x7fa654003e20 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fa66000c160 tx=0x7fa654005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3560a5cde663bd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.419+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660007b70 con 0x7fa654003a00 2024-03-29T11:32:17.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.419+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66000cd20 con 0x7fa654003a00 2024-03-29T11:32:17.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.419+0000 7fa66e000700 1 -- 172.21.15.12:0/1826483097 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660005ba0 con 0x7fa654003a00 2024-03-29T11:32:17.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.419+0000 7fa676400700 1 -- 172.21.15.12:0/1826483097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 msgr2=0x7fa654003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.419+0000 7fa676400700 1 --2- 172.21.15.12:0/1826483097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 0x7fa654003e20 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fa66000c160 tx=0x7fa654005880 comp rx=0 tx=0).stop 2024-03-29T11:32:17.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.419+0000 7fa66fe00700 1 -- 172.21.15.12:0/1826483097 reap_dead start 2024-03-29T11:32:17.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.420+0000 7fa676400700 1 -- 172.21.15.12:0/1826483097 shutdown_connections 2024-03-29T11:32:17.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.420+0000 7fa676400700 1 -- 172.21.15.12:0/1826483097 >> 172.21.15.12:0/1826483097 conn(0x7fa67008aff0 msgr2=0x7fa6700d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:17.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.420+0000 7fa676400700 1 -- 172.21.15.12:0/1826483097 shutdown_connections 2024-03-29T11:32:17.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.420+0000 7fa676400700 1 -- 172.21.15.12:0/1826483097 wait complete. 2024-03-29T11:32:17.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.422+0000 7fa676400700 1 Processor -- start 2024-03-29T11:32:17.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.422+0000 7fa676400700 1 -- start start 2024-03-29T11:32:17.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.423+0000 7fa676400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 0x7fa67015f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.423+0000 7fa676400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa67015cc70 0x7fa67015d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa676400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6701f5c50 0x7fa6701f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa676400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67015e4d0 con 0x7fa654003a00 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa676400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67015fd80 con 0x7fa67015cc70 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa676400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa67015d610 con 0x7fa6701f5c50 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 0x7fa67015f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 0x7fa67015f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38932/0 (socket says 172.21.15.12:38932) 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66f400700 1 -- 172.21.15.12:0/3609508948 learned_addr learned my addr 172.21.15.12:0/3609508948 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66ea00700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa67015cc70 0x7fa67015d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66fe00700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6701f5c50 0x7fa6701f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66f400700 1 -- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6701f5c50 msgr2=0x7fa6701f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66f400700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6701f5c50 0x7fa6701f60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66f400700 1 -- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa67015cc70 msgr2=0x7fa67015d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66f400700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa67015cc70 0x7fa67015d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66f400700 1 -- 172.21.15.12:0/3609508948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67015d850 con 0x7fa654003a00 2024-03-29T11:32:17.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.424+0000 7fa66f400700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 0x7fa67015f150 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fa66000c750 tx=0x7fa660001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:17.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.425+0000 7fa667400700 1 -- 172.21.15.12:0/3609508948 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660004c80 con 0x7fa654003a00 2024-03-29T11:32:17.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.425+0000 7fa667400700 1 -- 172.21.15.12:0/3609508948 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa660004de0 con 0x7fa654003a00 2024-03-29T11:32:17.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.425+0000 7fa667400700 1 -- 172.21.15.12:0/3609508948 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660007310 con 0x7fa654003a00 2024-03-29T11:32:17.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.425+0000 7fa676400700 1 -- 172.21.15.12:0/3609508948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67015db30 con 0x7fa654003a00 2024-03-29T11:32:17.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.425+0000 7fa676400700 1 -- 172.21.15.12:0/3609508948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa67015e0f0 con 0x7fa654003a00 2024-03-29T11:32:17.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.426+0000 7fa676400700 1 -- 172.21.15.12:0/3609508948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa670067720 con 0x7fa654003a00 2024-03-29T11:32:17.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.427+0000 7fa667400700 1 -- 172.21.15.12:0/3609508948 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa660019030 con 0x7fa654003a00 2024-03-29T11:32:17.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.428+0000 7fa667400700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa65006aed0 0x7fa65006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.428+0000 7fa667400700 1 -- 172.21.15.12:0/3609508948 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6610+0+0 (secure 0 0 0) 0x7fa66008a930 con 0x7fa654003a00 2024-03-29T11:32:17.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.428+0000 7fa66ea00700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa65006aed0 0x7fa65006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.429+0000 7fa66ea00700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa65006aed0 0x7fa65006d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009ba0 tx=0x7fa65c008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:17.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.432+0000 7fa667400700 1 -- 172.21.15.12:0/3609508948 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa660055410 con 0x7fa654003a00 2024-03-29T11:32:17.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.690+0000 7fa676400700 1 -- 172.21.15.12:0/3609508948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa67008b760 con 0x7fa654003a00 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.691+0000 7fa667400700 1 -- 172.21.15.12:0/3609508948 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4758 (secure 0 0 0) 0x7fa660058a30 con 0x7fa654003a00 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":114,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":11676,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/1497670214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1497670214},{"type":"v1","addr":"172.21.15.12:6833","nonce":1497670214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:40.863222+0000","modified":"2024-03-29T11:31:43.883199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 -- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa65006aed0 msgr2=0x7fa65006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa65006aed0 0x7fa65006d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009ba0 tx=0x7fa65c008040 comp rx=0 tx=0).stop 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 -- 172.21.15.12:0/3609508948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 msgr2=0x7fa67015f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 0x7fa67015f150 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fa66000c750 tx=0x7fa660001930 comp rx=0 tx=0).stop 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 -- 172.21.15.12:0/3609508948 shutdown_connections 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa65006aed0 0x7fa65006d390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6701f5c50 0x7fa6701f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa654003a00 0x7fa67015f150 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 --2- 172.21.15.12:0/3609508948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa67015cc70 0x7fa67015d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 -- 172.21.15.12:0/3609508948 >> 172.21.15.12:0/3609508948 conn(0x7fa67008aff0 msgr2=0x7fa6700ca790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 -- 172.21.15.12:0/3609508948 shutdown_connections 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.693+0000 7fa676400700 1 -- 172.21.15.12:0/3609508948 wait complete. 2024-03-29T11:32:17.690 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 114 2024-03-29T11:32:17.707 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.915+0000 7f6fba200700 1 Processor -- start 2024-03-29T11:32:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.915+0000 7f6fba200700 1 -- start start 2024-03-29T11:32:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.916+0000 7f6fba200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6fb415b5d0 0x7f6fb40ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.916+0000 7f6fba200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb415fd50 con 0x7f6fb40cb3f0 2024-03-29T11:32:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.916+0000 7f6fba200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb415f140 con 0x7f6fb40ac7d0 2024-03-29T11:32:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.916+0000 7f6fba200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb415b9b0 con 0x7f6fb415b5d0 2024-03-29T11:32:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.916+0000 7f6fb2a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6fb40cb3f0 0x7f6fb40ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46818/0 (socket says 172.21.15.12:46818) 2024-03-29T11:32:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.916+0000 7f6fb2a00700 1 -- 172.21.15.12:0/3015332077 learned_addr learned my addr 172.21.15.12:0/3015332077 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb3400700 1 --2- 172.21.15.12:0/3015332077 >> v2:172.21.15.159:3300/0 conn(0x7f6fb415b5d0 0x7f6fb40ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912298776 0 0) 0x7f6fb415fd50 con 0x7f6fb40cb3f0 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f980035b0 con 0x7f6fb40cb3f0 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377431046 0 0) 0x7f6fb415f140 con 0x7f6fb40ac7d0 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb415fd50 con 0x7f6fb40ac7d0 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3399497297 0 0) 0x7f6f980035b0 con 0x7f6fb40cb3f0 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 >> v2:172.21.15.159:3300/0 conn(0x7f6fb415b5d0 msgr2=0x7f6fb40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 --2- 172.21.15.12:0/3015332077 >> v2:172.21.15.159:3300/0 conn(0x7f6fb415b5d0 0x7f6fb40ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 >> v1:172.21.15.159:6789/0 conn(0x7f6fb40ac7d0 legacy=0x7f6fb406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb415f420 con 0x7f6fb40cb3f0 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f6fa0001670 con 0x7f6fb40cb3f0 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 >> v1:172.21.15.12:6789/0 conn(0x7f6fb40cb3f0 legacy=0x7f6fb40ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f98003b90 0x7f6f98003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f98004850 0x7f6f98004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f98005210 0x7f6f9800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb415b9b0 con 0x7f6f98005210 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9800c140 con 0x7f6f98004850 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.917+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9800c300 con 0x7f6f98003b90 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.918+0000 7f6fb3e00700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f98005210 0x7f6f9800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.918+0000 7f6fb3400700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f98003b90 0x7f6f98003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.918+0000 7f6fb2a00700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f98004850 0x7f6f98004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.918+0000 7f6fb3e00700 1 -- 172.21.15.12:0/3015332077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f98003b90 msgr2=0x7f6f98003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.918+0000 7f6fb3e00700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f98003b90 0x7f6f98003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.918+0000 7f6fb3e00700 1 -- 172.21.15.12:0/3015332077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f98004850 msgr2=0x7f6f98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.918+0000 7f6fb3e00700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f98004850 0x7f6f98004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.918+0000 7f6fb3e00700 1 -- 172.21.15.12:0/3015332077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980035b0 con 0x7f6f98005210 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.918+0000 7f6fb3e00700 1 -- 172.21.15.12:0/3015332077 reap_dead start 2024-03-29T11:32:17.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.918+0000 7f6fb3e00700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f98005210 0x7f6f9800ba10 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00c2f0 tx=0x7f6f98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b310a75d4877fe65 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:17.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.919+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac001f30 con 0x7f6f98005210 2024-03-29T11:32:17.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.919+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac003a00 con 0x7f6f98005210 2024-03-29T11:32:17.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.919+0000 7f6fb2000700 1 -- 172.21.15.12:0/3015332077 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac003db0 con 0x7f6f98005210 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.919+0000 7f6fba200700 1 -- 172.21.15.12:0/3015332077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f98005210 msgr2=0x7f6f9800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.919+0000 7f6fba200700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f98005210 0x7f6f9800ba10 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00c2f0 tx=0x7f6f98005880 comp rx=0 tx=0).stop 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.919+0000 7f6fba200700 1 -- 172.21.15.12:0/3015332077 shutdown_connections 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.919+0000 7f6fba200700 1 --2- 172.21.15.12:0/3015332077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f98005210 0x7f6f9800ba10 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.919+0000 7f6fba200700 1 -- 172.21.15.12:0/3015332077 >> 172.21.15.12:0/3015332077 conn(0x7f6fb40cf190 msgr2=0x7f6fb40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.919+0000 7f6fba200700 1 -- 172.21.15.12:0/3015332077 shutdown_connections 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.919+0000 7f6fba200700 1 -- 172.21.15.12:0/3015332077 wait complete. 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.921+0000 7f6fba200700 1 Processor -- start 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.921+0000 7f6fba200700 1 -- start start 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fba200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f98005210 0x7f6fb4159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fba200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6fb4157850 0x7f6fb4159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fba200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb4156ba0 0x7f6fb40ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fba200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb40cbd40 con 0x7f6fb4156ba0 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fba200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb4159ff0 con 0x7f6f98005210 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fba200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb415a2d0 con 0x7f6fb4157850 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fb3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb4156ba0 0x7f6fb40ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fb3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb4156ba0 0x7f6fb40ae260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38962/0 (socket says 172.21.15.12:38962) 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fb3e00700 1 -- 172.21.15.12:0/3896002767 learned_addr learned my addr 172.21.15.12:0/3896002767 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fb3400700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f98005210 0x7f6fb4159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.922+0000 7f6fb2a00700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6fb4157850 0x7f6fb4159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fb3e00700 1 -- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6fb4157850 msgr2=0x7f6fb4159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fb3e00700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6fb4157850 0x7f6fb4159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fb3e00700 1 -- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f98005210 msgr2=0x7f6fb4159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fb3e00700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f98005210 0x7f6fb4159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fb3e00700 1 -- 172.21.15.12:0/3896002767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb415a5b0 con 0x7f6fb4156ba0 2024-03-29T11:32:17.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fb3e00700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb4156ba0 0x7f6fb40ae260 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f6fb415fcc0 tx=0x7f6fac001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:17.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fab400700 1 -- 172.21.15.12:0/3896002767 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac005070 con 0x7f6fb4156ba0 2024-03-29T11:32:17.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fab400700 1 -- 172.21.15.12:0/3896002767 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac0051d0 con 0x7f6fb4156ba0 2024-03-29T11:32:17.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fba200700 1 -- 172.21.15.12:0/3896002767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb415a890 con 0x7f6fb4156ba0 2024-03-29T11:32:17.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fab400700 1 -- 172.21.15.12:0/3896002767 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac005560 con 0x7f6fb4156ba0 2024-03-29T11:32:17.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.923+0000 7f6fba200700 1 -- 172.21.15.12:0/3896002767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb40ae7a0 con 0x7f6fb4156ba0 2024-03-29T11:32:17.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.924+0000 7f6fba200700 1 -- 172.21.15.12:0/3896002767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb4067720 con 0x7f6fb4156ba0 2024-03-29T11:32:17.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.925+0000 7f6fab400700 1 -- 172.21.15.12:0/3896002767 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6fac0056c0 con 0x7f6fb4156ba0 2024-03-29T11:32:17.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.927+0000 7f6fab400700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f9406aed0 0x7f6f9406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:17.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.927+0000 7f6fab400700 1 -- 172.21.15.12:0/3896002767 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6610+0+0 (secure 0 0 0) 0x7f6fac08a900 con 0x7f6fb4156ba0 2024-03-29T11:32:17.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.927+0000 7f6fb3400700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f9406aed0 0x7f6f9406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:17.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.928+0000 7f6fb3400700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f9406aed0 0x7f6f9406d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4009be0 tx=0x7f6fa4009210 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:17.931+0000 7f6fab400700 1 -- 172.21.15.12:0/3896002767 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6fac055550 con 0x7f6fb4156ba0 2024-03-29T11:32:18.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.143+0000 7f6fba200700 1 -- 172.21.15.12:0/3896002767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fb406ea60 con 0x7f6fb4156ba0 2024-03-29T11:32:18.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.144+0000 7f6fab400700 1 -- 172.21.15.12:0/3896002767 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+17432 (secure 0 0 0) 0x7f6fac058b70 con 0x7f6fb4156ba0 2024-03-29T11:32:18.140 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:18.140 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":236,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:15.304574+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:31:39.075407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:31:39.845704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6833/2722218561":"2024-03-30T11:32:12.038796+0000","172.21.15.159:0/4178225398":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1623528933":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1432915605":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/2328654185":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3030066758":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/2640388148":"2024-03-30T11:32:08.498738+0000","172.21.15.12:6832/2722218561":"2024-03-30T11:32:12.038796+0000","172.21.15.159:0/1069287393":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/353804238":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6833/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6832/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3242066681":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3562663283":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3628749078":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/235474344":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3837208470":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/1750949668":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/629137162":"2024-03-30T11:31:55.626759+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.146+0000 7f6fba200700 1 -- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f9406aed0 msgr2=0x7f6f9406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.146+0000 7f6fba200700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f9406aed0 0x7f6f9406d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4009be0 tx=0x7f6fa4009210 comp rx=0 tx=0).stop 2024-03-29T11:32:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.146+0000 7f6fba200700 1 -- 172.21.15.12:0/3896002767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb4156ba0 msgr2=0x7f6fb40ae260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.146+0000 7f6fba200700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb4156ba0 0x7f6fb40ae260 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f6fb415fcc0 tx=0x7f6fac001650 comp rx=0 tx=0).stop 2024-03-29T11:32:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.147+0000 7f6fba200700 1 -- 172.21.15.12:0/3896002767 shutdown_connections 2024-03-29T11:32:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.147+0000 7f6fba200700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f9406aed0 0x7f6f9406d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.147+0000 7f6fba200700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb4156ba0 0x7f6fb40ae260 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.147+0000 7f6fba200700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f98005210 0x7f6fb4159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.147+0000 7f6fba200700 1 --2- 172.21.15.12:0/3896002767 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6fb4157850 0x7f6fb4159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.147+0000 7f6fba200700 1 -- 172.21.15.12:0/3896002767 >> 172.21.15.12:0/3896002767 conn(0x7f6fb40cf190 msgr2=0x7f6fb4072140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.147+0000 7f6fba200700 1 -- 172.21.15.12:0/3896002767 shutdown_connections 2024-03-29T11:32:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.147+0000 7f6fba200700 1 -- 172.21.15.12:0/3896002767 wait complete. 2024-03-29T11:32:18.160 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-03-29T11:32:18.160 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:18.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.364+0000 7fbf0c400700 1 Processor -- start 2024-03-29T11:32:18.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.364+0000 7fbf0c400700 1 -- start start 2024-03-29T11:32:18.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.365+0000 7fbf0c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf04161ca0 0x7fbf040cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.365+0000 7fbf0c400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf040ceda0 con 0x7fbf040d18b0 2024-03-29T11:32:18.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.365+0000 7fbf0c400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf041612f0 con 0x7fbf04160f10 2024-03-29T11:32:18.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.365+0000 7fbf0c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040d1d30 con 0x7fbf04161ca0 2024-03-29T11:32:18.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.365+0000 7fbf09000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbf040d18b0 0x7fbf04161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46828/0 (socket says 172.21.15.12:46828) 2024-03-29T11:32:18.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.365+0000 7fbf09000700 1 -- 172.21.15.12:0/3659720399 learned_addr learned my addr 172.21.15.12:0/3659720399 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf0a400700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf04161ca0 0x7fbf040cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304995463 0 0) 0x7fbf040ceda0 con 0x7fbf040d18b0 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeec0035b0 con 0x7fbf040d18b0 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677257577 0 0) 0x7fbf041612f0 con 0x7fbf04160f10 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf040ceda0 con 0x7fbf04160f10 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2034322352 0 0) 0x7fbeec0035b0 con 0x7fbf040d18b0 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf04161ca0 msgr2=0x7fbf040cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf04161ca0 0x7fbf040cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 >> v1:172.21.15.159:6789/0 conn(0x7fbf04160f10 legacy=0x7fbf040d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf040ce470 con 0x7fbf040d18b0 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fbef0001670 con 0x7fbf040d18b0 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 >> v1:172.21.15.12:6789/0 conn(0x7fbf040d18b0 legacy=0x7fbf04161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbeec003bc0 0x7fbeec003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbeec004850 0x7fbeec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeec005210 0x7fbeec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.366+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040d1d30 con 0x7fbeec005210 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec00c0e0 con 0x7fbeec003bc0 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec00c2a0 con 0x7fbeec004850 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf0a400700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf04161ca0 0x7fbf040cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf0a400700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeec005210 0x7fbeec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf09a00700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbeec003bc0 0x7fbeec003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf09000700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbeec004850 0x7fbeec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf0a400700 1 -- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbeec004850 msgr2=0x7fbeec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf0a400700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbeec004850 0x7fbeec004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf0a400700 1 -- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbeec003bc0 msgr2=0x7fbeec003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf0a400700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbeec003bc0 0x7fbeec003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf0a400700 1 -- 172.21.15.12:0/3659720399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec0035b0 con 0x7fbeec005210 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.367+0000 7fbf0a400700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeec005210 0x7fbeec00ba10 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00e220 tx=0x7fbeec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d047e7010002b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.368+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc00ecc0 con 0x7fbeec005210 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.368+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc00f040 con 0x7fbeec005210 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.368+0000 7fbf03e00700 1 -- 172.21.15.12:0/3659720399 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc003a20 con 0x7fbeec005210 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.368+0000 7fbf0c400700 1 -- 172.21.15.12:0/3659720399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeec005210 msgr2=0x7fbeec00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.368+0000 7fbf0c400700 1 --2- 172.21.15.12:0/3659720399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeec005210 0x7fbeec00ba10 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00e220 tx=0x7fbeec005880 comp rx=0 tx=0).stop 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.368+0000 7fbf0a400700 1 -- 172.21.15.12:0/3659720399 reap_dead start 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.368+0000 7fbf0c400700 1 -- 172.21.15.12:0/3659720399 shutdown_connections 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.368+0000 7fbf0c400700 1 -- 172.21.15.12:0/3659720399 >> 172.21.15.12:0/3659720399 conn(0x7fbf0408aff0 msgr2=0x7fbf040d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.368+0000 7fbf0c400700 1 -- 172.21.15.12:0/3659720399 shutdown_connections 2024-03-29T11:32:18.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.368+0000 7fbf0c400700 1 -- 172.21.15.12:0/3659720399 wait complete. 2024-03-29T11:32:18.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.370+0000 7fbf0c400700 1 Processor -- start 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.370+0000 7fbf0c400700 1 -- start start 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.371+0000 7fbf0c400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbeec005210 0x7fbf0409e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.371+0000 7fbf0c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf0409f320 0x7fbf04158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.371+0000 7fbf0c400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf04158e90 0x7fbf041f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.371+0000 7fbf0c400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040a07c0 con 0x7fbf04158e90 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.371+0000 7fbf0c400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0409fbb0 con 0x7fbeec005210 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.371+0000 7fbf0c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0409fe90 con 0x7fbf0409f320 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.371+0000 7fbf0a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf04158e90 0x7fbf041f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf0a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf04158e90 0x7fbf041f3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38966/0 (socket says 172.21.15.12:38966) 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf0a400700 1 -- 172.21.15.12:0/349753560 learned_addr learned my addr 172.21.15.12:0/349753560 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:18.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf09a00700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbeec005210 0x7fbf0409e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf09000700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf0409f320 0x7fbf04158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf0a400700 1 -- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf0409f320 msgr2=0x7fbf04158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf0a400700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf0409f320 0x7fbf04158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf0a400700 1 -- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbeec005210 msgr2=0x7fbf0409e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf0a400700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbeec005210 0x7fbf0409e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf0a400700 1 -- 172.21.15.12:0/349753560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf040a0170 con 0x7fbf04158e90 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf0a400700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf04158e90 0x7fbf041f3610 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00e1f0 tx=0x7fbefc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf01600700 1 -- 172.21.15.12:0/349753560 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc004230 con 0x7fbf04158e90 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.372+0000 7fbf01600700 1 -- 172.21.15.12:0/349753560 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc023070 con 0x7fbf04158e90 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.373+0000 7fbf0c400700 1 -- 172.21.15.12:0/349753560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf040a0450 con 0x7fbf04158e90 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.373+0000 7fbf01600700 1 -- 172.21.15.12:0/349753560 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0045d0 con 0x7fbf04158e90 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.373+0000 7fbf0c400700 1 -- 172.21.15.12:0/349753560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0409f980 con 0x7fbf04158e90 2024-03-29T11:32:18.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.374+0000 7fbf0c400700 1 -- 172.21.15.12:0/349753560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf04067720 con 0x7fbf04158e90 2024-03-29T11:32:18.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.375+0000 7fbf01600700 1 -- 172.21.15.12:0/349753560 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fbefc025030 con 0x7fbf04158e90 2024-03-29T11:32:18.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.376+0000 7fbf01600700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbee406aed0 0x7fbee406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.376+0000 7fbf01600700 1 -- 172.21.15.12:0/349753560 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6610+0+0 (secure 0 0 0) 0x7fbefc08b2f0 con 0x7fbf04158e90 2024-03-29T11:32:18.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.376+0000 7fbf09a00700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbee406aed0 0x7fbee406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.377+0000 7fbf09a00700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbee406aed0 0x7fbee406d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbef8000c00 tx=0x7fbef8009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:18.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.381+0000 7fbf01600700 1 -- 172.21.15.12:0/349753560 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbefc055ef0 con 0x7fbf04158e90 2024-03-29T11:32:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.646+0000 7fbf0c400700 1 -- 172.21.15.12:0/349753560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf0408b760 con 0x7fbf04158e90 2024-03-29T11:32:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.646+0000 7fbf01600700 1 -- 172.21.15.12:0/349753560 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4758 (secure 0 0 0) 0x7fbefc059510 con 0x7fbf04158e90 2024-03-29T11:32:18.641 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:18.642 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":114,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":11676,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/1497670214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1497670214},{"type":"v1","addr":"172.21.15.12:6833","nonce":1497670214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:40.863222+0000","modified":"2024-03-29T11:31:43.883199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-29T11:32:18.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.648+0000 7fbf0c400700 1 -- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbee406aed0 msgr2=0x7fbee406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbee406aed0 0x7fbee406d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbef8000c00 tx=0x7fbef8009040 comp rx=0 tx=0).stop 2024-03-29T11:32:18.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 -- 172.21.15.12:0/349753560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf04158e90 msgr2=0x7fbf041f3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf04158e90 0x7fbf041f3610 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00e1f0 tx=0x7fbefc009750 comp rx=0 tx=0).stop 2024-03-29T11:32:18.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 -- 172.21.15.12:0/349753560 shutdown_connections 2024-03-29T11:32:18.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbee406aed0 0x7fbee406d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf04158e90 0x7fbf041f3610 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbeec005210 0x7fbf0409e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 --2- 172.21.15.12:0/349753560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf0409f320 0x7fbf04158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 -- 172.21.15.12:0/349753560 >> 172.21.15.12:0/349753560 conn(0x7fbf0408aff0 msgr2=0x7fbf040cab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:18.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 -- 172.21.15.12:0/349753560 shutdown_connections 2024-03-29T11:32:18.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.649+0000 7fbf0c400700 1 -- 172.21.15.12:0/349753560 wait complete. 2024-03-29T11:32:18.645 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 114 2024-03-29T11:32:18.663 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:18.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.865+0000 7f4c47a00700 1 Processor -- start 2024-03-29T11:32:18.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.866+0000 7f4c47a00700 1 -- start start 2024-03-29T11:32:18.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.867+0000 7f4c47a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c40160f10 0x7f4c400d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.867+0000 7f4c47a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4c40161590 0x7f4c40161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.867+0000 7f4c47a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c400ceda0 con 0x7f4c400cf230 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.867+0000 7f4c47a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c40161f80 con 0x7f4c40161590 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.867+0000 7f4c47a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c401612f0 con 0x7f4c40160f10 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.867+0000 7f4c45000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c40160f10 0x7f4c400d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.867+0000 7f4c45a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4c40161590 0x7f4c40161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.867+0000 7f4c45a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4c40161590 0x7f4c40161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:56038/0 (socket says 172.21.15.12:56038) 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.867+0000 7f4c3fe00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f4c400cf230 0x7f4c400d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:58940/0 (socket says 172.21.15.12:58940) 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.867+0000 7f4c45a00700 1 -- 172.21.15.12:0/3508960278 learned_addr learned my addr 172.21.15.12:0/3508960278 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.868+0000 7f4c45a00700 1 -- 172.21.15.12:0/3508960278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c40160f10 msgr2=0x7f4c400d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.868+0000 7f4c45a00700 1 --2- 172.21.15.12:0/3508960278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c40160f10 0x7f4c400d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.868+0000 7f4c45a00700 1 -- 172.21.15.12:0/3508960278 >> v1:172.21.15.159:6789/0 conn(0x7f4c400cf230 legacy=0x7f4c400d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.868+0000 7f4c45a00700 1 -- 172.21.15.12:0/3508960278 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c400ce470 con 0x7f4c40161590 2024-03-29T11:32:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.868+0000 7f4c45a00700 1 --2- 172.21.15.12:0/3508960278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c40161590 0x7f4c40161a40 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009e80 tx=0x7f4c3800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f90987d5dd51147 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:18.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.868+0000 7f4c3f400700 1 -- 172.21.15.12:0/3508960278 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3800e070 con 0x7f4c40161590 2024-03-29T11:32:18.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.868+0000 7f4c3f400700 1 -- 172.21.15.12:0/3508960278 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3800b8d0 con 0x7f4c40161590 2024-03-29T11:32:18.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.868+0000 7f4c3f400700 1 -- 172.21.15.12:0/3508960278 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3800bc80 con 0x7f4c40161590 2024-03-29T11:32:18.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.869+0000 7f4c47a00700 1 -- 172.21.15.12:0/3508960278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c40161590 msgr2=0x7f4c40161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.869+0000 7f4c47a00700 1 --2- 172.21.15.12:0/3508960278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c40161590 0x7f4c40161a40 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009e80 tx=0x7f4c3800b330 comp rx=0 tx=0).stop 2024-03-29T11:32:18.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.869+0000 7f4c47a00700 1 -- 172.21.15.12:0/3508960278 shutdown_connections 2024-03-29T11:32:18.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.869+0000 7f4c47a00700 1 --2- 172.21.15.12:0/3508960278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c40161590 0x7f4c40161a40 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.869+0000 7f4c47a00700 1 --2- 172.21.15.12:0/3508960278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c40160f10 0x7f4c400d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.869+0000 7f4c47a00700 1 -- 172.21.15.12:0/3508960278 >> 172.21.15.12:0/3508960278 conn(0x7f4c4008aff0 msgr2=0x7f4c400d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:18.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.869+0000 7f4c47a00700 1 -- 172.21.15.12:0/3508960278 shutdown_connections 2024-03-29T11:32:18.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.869+0000 7f4c47a00700 1 -- 172.21.15.12:0/3508960278 wait complete. 2024-03-29T11:32:18.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.871+0000 7f4c47a00700 1 Processor -- start 2024-03-29T11:32:18.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.871+0000 7f4c47a00700 1 -- start start 2024-03-29T11:32:18.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c47a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c400cf230 0x7f4c401f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c47a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c40160f10 0x7f4c401f0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c47a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c40161590 0x7f4c400b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c47a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c401f4060 con 0x7f4c40161590 2024-03-29T11:32:18.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c47a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c401f0ad0 con 0x7f4c40160f10 2024-03-29T11:32:18.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c47a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c400b64b0 con 0x7f4c400cf230 2024-03-29T11:32:18.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c45a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c40161590 0x7f4c400b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c45000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c400cf230 0x7f4c401f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c45a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c40161590 0x7f4c400b7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48596/0 (socket says 172.21.15.12:48596) 2024-03-29T11:32:18.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c45a00700 1 -- 172.21.15.12:0/3112246475 learned_addr learned my addr 172.21.15.12:0/3112246475 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:18.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.872+0000 7f4c3fe00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c40160f10 0x7f4c401f0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c45a00700 1 -- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c400cf230 msgr2=0x7f4c401f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c45a00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c400cf230 0x7f4c401f3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c45a00700 1 -- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c40160f10 msgr2=0x7f4c401f0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c45a00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c40160f10 0x7f4c401f0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:18.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c45a00700 1 -- 172.21.15.12:0/3112246475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38009b30 con 0x7f4c40161590 2024-03-29T11:32:18.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c45a00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c40161590 0x7f4c400b7eb0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f4c4008c570 tx=0x7f4c38004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:18.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c3cc00700 1 -- 172.21.15.12:0/3112246475 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c38004800 con 0x7f4c40161590 2024-03-29T11:32:18.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c3cc00700 1 -- 172.21.15.12:0/3112246475 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c38004960 con 0x7f4c40161590 2024-03-29T11:32:18.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c3cc00700 1 -- 172.21.15.12:0/3112246475 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3801c680 con 0x7f4c40161590 2024-03-29T11:32:18.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c47a00700 1 -- 172.21.15.12:0/3112246475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c400b6790 con 0x7f4c40161590 2024-03-29T11:32:18.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.873+0000 7f4c47a00700 1 -- 172.21.15.12:0/3112246475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c400b6d50 con 0x7f4c40161590 2024-03-29T11:32:18.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.874+0000 7f4c47a00700 1 -- 172.21.15.12:0/3112246475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c40067720 con 0x7f4c40161590 2024-03-29T11:32:18.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.875+0000 7f4c3cc00700 1 -- 172.21.15.12:0/3112246475 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4c3801c7e0 con 0x7f4c40161590 2024-03-29T11:32:18.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.876+0000 7f4c3cc00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c2006aed0 0x7f4c2006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:18.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.876+0000 7f4c3cc00700 1 -- 172.21.15.12:0/3112246475 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6610+0+0 (secure 0 0 0) 0x7f4c3808b4d0 con 0x7f4c40161590 2024-03-29T11:32:18.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.877+0000 7f4c45000700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c2006aed0 0x7f4c2006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:18.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.877+0000 7f4c45000700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c2006aed0 0x7f4c2006d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c40160290 tx=0x7f4c34009210 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:18.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:18.881+0000 7f4c3cc00700 1 -- 172.21.15.12:0/3112246475 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c38055ea0 con 0x7f4c40161590 2024-03-29T11:32:19.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.142+0000 7f4c47a00700 1 -- 172.21.15.12:0/3112246475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c400cf8d0 con 0x7f4c40161590 2024-03-29T11:32:19.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.142+0000 7f4c3cc00700 1 -- 172.21.15.12:0/3112246475 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4758 (secure 0 0 0) 0x7f4c380594c0 con 0x7f4c40161590 2024-03-29T11:32:19.137 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:19.138 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":114,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":11676,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/1497670214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1497670214},{"type":"v1","addr":"172.21.15.12:6833","nonce":1497670214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:31:40.863222+0000","modified":"2024-03-29T11:31:43.883199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/2157276692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2157276692},{"type":"v1","addr":"172.21.15.159:6837","nonce":2157276692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-29T11:32:19.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 -- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c2006aed0 msgr2=0x7f4c2006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c2006aed0 0x7f4c2006d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c40160290 tx=0x7f4c34009210 comp rx=0 tx=0).stop 2024-03-29T11:32:19.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 -- 172.21.15.12:0/3112246475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c40161590 msgr2=0x7f4c400b7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c40161590 0x7f4c400b7eb0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f4c4008c570 tx=0x7f4c38004ad0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 -- 172.21.15.12:0/3112246475 shutdown_connections 2024-03-29T11:32:19.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4c2006aed0 0x7f4c2006d390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c40161590 0x7f4c400b7eb0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c400cf230 0x7f4c401f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 --2- 172.21.15.12:0/3112246475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c40160f10 0x7f4c401f0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 -- 172.21.15.12:0/3112246475 >> 172.21.15.12:0/3112246475 conn(0x7f4c4008aff0 msgr2=0x7f4c4015b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:19.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 -- 172.21.15.12:0/3112246475 shutdown_connections 2024-03-29T11:32:19.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.145+0000 7f4c47a00700 1 -- 172.21.15.12:0/3112246475 wait complete. 2024-03-29T11:32:19.141 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 114 2024-03-29T11:32:19.159 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:19.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.363+0000 7f489e600700 1 Processor -- start 2024-03-29T11:32:19.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.364+0000 7f489e600700 1 -- start start 2024-03-29T11:32:19.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.365+0000 7f489e600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 0x7f48980ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.365+0000 7f489e600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f48980cb3f0 0x7f48980ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.365+0000 7f489e600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489815fd50 con 0x7f48980ae930 2024-03-29T11:32:19.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.365+0000 7f489e600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489815b9b0 con 0x7f48980cb3f0 2024-03-29T11:32:19.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.365+0000 7f489e600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489815f140 con 0x7f489815b5d0 2024-03-29T11:32:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.365+0000 7f4896a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f48980cb3f0 0x7f48980ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.365+0000 7f4897400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 0x7f48980ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.365+0000 7f4897e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f48980ae930 0x7f4898140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:58950/0 (socket says 172.21.15.12:58950) 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.365+0000 7f4897e00700 1 -- 172.21.15.12:0/4263209294 learned_addr learned my addr 172.21.15.12:0/4263209294 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.366+0000 7f4896000700 1 -- 172.21.15.12:0/4263209294 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546454170 0 0) 0x7f489815fd50 con 0x7f48980ae930 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.366+0000 7f4896000700 1 -- 172.21.15.12:0/4263209294 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f487c0035b0 con 0x7f48980ae930 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.366+0000 7f4897400700 1 -- 172.21.15.12:0/4263209294 >> v2:172.21.15.159:3300/0 conn(0x7f48980cb3f0 msgr2=0x7f48980ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.366+0000 7f4897400700 1 --2- 172.21.15.12:0/4263209294 >> v2:172.21.15.159:3300/0 conn(0x7f48980cb3f0 0x7f48980ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.366+0000 7f4897400700 1 -- 172.21.15.12:0/4263209294 >> v1:172.21.15.159:6789/0 conn(0x7f48980ae930 legacy=0x7f4898140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.366+0000 7f4897400700 1 -- 172.21.15.12:0/4263209294 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489815f420 con 0x7f489815b5d0 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.366+0000 7f4896a00700 1 --2- 172.21.15.12:0/4263209294 >> v2:172.21.15.159:3300/0 conn(0x7f48980cb3f0 0x7f48980ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.366+0000 7f4897400700 1 --2- 172.21.15.12:0/4263209294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 0x7f48980ae3f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4888009840 tx=0x7f4888009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3afcac9c5986fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f4896000700 1 -- 172.21.15.12:0/4263209294 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888011830 con 0x7f489815b5d0 2024-03-29T11:32:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f4896000700 1 -- 172.21.15.12:0/4263209294 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4888011990 con 0x7f489815b5d0 2024-03-29T11:32:19.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f4896000700 1 -- 172.21.15.12:0/4263209294 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888011d20 con 0x7f489815b5d0 2024-03-29T11:32:19.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f489e600700 1 -- 172.21.15.12:0/4263209294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 msgr2=0x7f48980ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f489e600700 1 --2- 172.21.15.12:0/4263209294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 0x7f48980ae3f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4888009840 tx=0x7f4888009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:19.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f489e600700 1 -- 172.21.15.12:0/4263209294 shutdown_connections 2024-03-29T11:32:19.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f489e600700 1 --2- 172.21.15.12:0/4263209294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 0x7f48980ae3f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f489e600700 1 --2- 172.21.15.12:0/4263209294 >> v2:172.21.15.159:3300/0 conn(0x7f48980cb3f0 0x7f48980ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f489e600700 1 -- 172.21.15.12:0/4263209294 >> 172.21.15.12:0/4263209294 conn(0x7f48980cf190 msgr2=0x7f48980cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:19.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f489e600700 1 -- 172.21.15.12:0/4263209294 shutdown_connections 2024-03-29T11:32:19.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.367+0000 7f489e600700 1 -- 172.21.15.12:0/4263209294 wait complete. 2024-03-29T11:32:19.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.369+0000 7f489e600700 1 Processor -- start 2024-03-29T11:32:19.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.369+0000 7f489e600700 1 -- start start 2024-03-29T11:32:19.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.370+0000 7f489e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48980ae930 0x7f48980713d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.370+0000 7f489e600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 0x7f489815a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.370+0000 7f489e600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f489814f1e0 0x7f4898155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.370+0000 7f489e600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489814fcc0 con 0x7f48980ae930 2024-03-29T11:32:19.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.370+0000 7f489e600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4898071a60 con 0x7f489814f1e0 2024-03-29T11:32:19.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.370+0000 7f489e600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4898071d40 con 0x7f489815b5d0 2024-03-29T11:32:19.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.370+0000 7f4897400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48980ae930 0x7f48980713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.370+0000 7f4897400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48980ae930 0x7f48980713d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48600/0 (socket says 172.21.15.12:48600) 2024-03-29T11:32:19.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.370+0000 7f4897400700 1 -- 172.21.15.12:0/237224519 learned_addr learned my addr 172.21.15.12:0/237224519 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:19.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f4896a00700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 0x7f489815a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f4897e00700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f489814f1e0 0x7f4898155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f4897400700 1 -- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 msgr2=0x7f489815a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f4897400700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 0x7f489815a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f4897400700 1 -- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f489814f1e0 msgr2=0x7f4898155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f4897400700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f489814f1e0 0x7f4898155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f4897400700 1 -- 172.21.15.12:0/237224519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48840094a0 con 0x7f48980ae930 2024-03-29T11:32:19.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f4897e00700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f489814f1e0 0x7f4898155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:19.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f4897400700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48980ae930 0x7f48980713d0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f4888009b90 tx=0x7f4888004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:19.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f488f400700 1 -- 172.21.15.12:0/237224519 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48880036a0 con 0x7f48980ae930 2024-03-29T11:32:19.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f488f400700 1 -- 172.21.15.12:0/237224519 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4888003800 con 0x7f48980ae930 2024-03-29T11:32:19.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.371+0000 7f489e600700 1 -- 172.21.15.12:0/237224519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48880094a0 con 0x7f48980ae930 2024-03-29T11:32:19.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.372+0000 7f488f400700 1 -- 172.21.15.12:0/237224519 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888003b90 con 0x7f48980ae930 2024-03-29T11:32:19.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.372+0000 7f489e600700 1 -- 172.21.15.12:0/237224519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4898072300 con 0x7f48980ae930 2024-03-29T11:32:19.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.373+0000 7f489e600700 1 -- 172.21.15.12:0/237224519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48980c8460 con 0x7f48980ae930 2024-03-29T11:32:19.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.373+0000 7f488f400700 1 -- 172.21.15.12:0/237224519 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4888003cf0 con 0x7f48980ae930 2024-03-29T11:32:19.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.375+0000 7f488f400700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f487806aed0 0x7f487806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.375+0000 7f488f400700 1 -- 172.21.15.12:0/237224519 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6610+0+0 (secure 0 0 0) 0x7f488808aea0 con 0x7f48980ae930 2024-03-29T11:32:19.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.375+0000 7f4896a00700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f487806aed0 0x7f487806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.376+0000 7f4896a00700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f487806aed0 0x7f487806d390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f48980b4420 tx=0x7f4884008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:19.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.380+0000 7f488f400700 1 -- 172.21.15.12:0/237224519 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4888055980 con 0x7f48980ae930 2024-03-29T11:32:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.594+0000 7f489e600700 1 -- 172.21.15.12:0/237224519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4898067720 con 0x7f48980ae930 2024-03-29T11:32:19.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.596+0000 7f488f400700 1 -- 172.21.15.12:0/237224519 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+17432 (secure 0 0 0) 0x7f4888015090 con 0x7f48980ae930 2024-03-29T11:32:19.591 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:19.591 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":236,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:15.304574+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:31:39.075407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:31:39.845704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6833/2722218561":"2024-03-30T11:32:12.038796+0000","172.21.15.159:0/4178225398":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1623528933":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/1432915605":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/2328654185":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3030066758":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/2640388148":"2024-03-30T11:32:08.498738+0000","172.21.15.12:6832/2722218561":"2024-03-30T11:32:12.038796+0000","172.21.15.159:0/1069287393":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/353804238":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6833/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:6832/62646":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3242066681":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3562663283":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/3628749078":"2024-03-30T11:32:08.498738+0000","172.21.15.159:0/235474344":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/3837208470":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/1750949668":"2024-03-30T11:31:55.626759+0000","172.21.15.159:0/629137162":"2024-03-30T11:31:55.626759+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:19.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 -- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f487806aed0 msgr2=0x7f487806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f487806aed0 0x7f487806d390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f48980b4420 tx=0x7f4884008040 comp rx=0 tx=0).stop 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 -- 172.21.15.12:0/237224519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48980ae930 msgr2=0x7f48980713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48980ae930 0x7f48980713d0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f4888009b90 tx=0x7f4888004750 comp rx=0 tx=0).stop 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 -- 172.21.15.12:0/237224519 shutdown_connections 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f487806aed0 0x7f487806d390 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f489814f1e0 0x7f4898155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48980ae930 0x7f48980713d0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 --2- 172.21.15.12:0/237224519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f489815b5d0 0x7f489815a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 -- 172.21.15.12:0/237224519 >> 172.21.15.12:0/237224519 conn(0x7f48980cf190 msgr2=0x7f48980aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 -- 172.21.15.12:0/237224519 shutdown_connections 2024-03-29T11:32:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.598+0000 7f489e600700 1 -- 172.21.15.12:0/237224519 wait complete. 2024-03-29T11:32:19.612 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-03-29T11:32:19.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.816+0000 7f4171000700 1 Processor -- start 2024-03-29T11:32:19.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.816+0000 7f4171000700 1 -- start start 2024-03-29T11:32:19.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.817+0000 7f4171000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f416c1611b0 0x7f416c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.817+0000 7f4171000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f416c0cee10 0x7f416c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.817+0000 7f4171000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0ce270 con 0x7f416c15f4d0 2024-03-29T11:32:19.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.817+0000 7f4171000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416c15b600 con 0x7f416c1611b0 2024-03-29T11:32:19.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.817+0000 7f4171000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416c0d2760 con 0x7f416c0cee10 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f4169e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f416c1611b0 0x7f416c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f4169400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f416c15f4d0 0x7f416c0ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37396/0 (socket says 172.21.15.12:37396) 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f4169400700 1 -- 172.21.15.12:0/2871473448 learned_addr learned my addr 172.21.15.12:0/2871473448 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f416a800700 1 --2- 172.21.15.12:0/2871473448 >> v2:172.21.15.159:3300/0 conn(0x7f416c0cee10 0x7f416c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f4169e00700 1 -- 172.21.15.12:0/2871473448 >> v2:172.21.15.159:3300/0 conn(0x7f416c0cee10 msgr2=0x7f416c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f4169e00700 1 --2- 172.21.15.12:0/2871473448 >> v2:172.21.15.159:3300/0 conn(0x7f416c0cee10 0x7f416c161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f4169e00700 1 -- 172.21.15.12:0/2871473448 >> v1:172.21.15.12:6789/0 conn(0x7f416c15f4d0 legacy=0x7f416c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f4169e00700 1 -- 172.21.15.12:0/2871473448 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c0d2a40 con 0x7f416c1611b0 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f416a800700 1 --2- 172.21.15.12:0/2871473448 >> v2:172.21.15.159:3300/0 conn(0x7f416c0cee10 0x7f416c161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f4168a00700 1 -- 172.21.15.12:0/2871473448 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516343673 0 0) 0x7f416c0ce270 con 0x7f416c15f4d0 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.818+0000 7f4169e00700 1 --2- 172.21.15.12:0/2871473448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c1611b0 0x7f416c15ef90 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f416c0d2f60 tx=0x7f415c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920b9cec28aa8191 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:19.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4168a00700 1 -- 172.21.15.12:0/2871473448 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c011830 con 0x7f416c1611b0 2024-03-29T11:32:19.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4168a00700 1 -- 172.21.15.12:0/2871473448 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415c011990 con 0x7f416c1611b0 2024-03-29T11:32:19.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4168a00700 1 -- 172.21.15.12:0/2871473448 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c011d20 con 0x7f416c1611b0 2024-03-29T11:32:19.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4171000700 1 -- 172.21.15.12:0/2871473448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c1611b0 msgr2=0x7f416c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4171000700 1 --2- 172.21.15.12:0/2871473448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c1611b0 0x7f416c15ef90 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f416c0d2f60 tx=0x7f415c009b20 comp rx=0 tx=0).stop 2024-03-29T11:32:19.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4171000700 1 -- 172.21.15.12:0/2871473448 shutdown_connections 2024-03-29T11:32:19.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4171000700 1 --2- 172.21.15.12:0/2871473448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c1611b0 0x7f416c15ef90 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4171000700 1 --2- 172.21.15.12:0/2871473448 >> v2:172.21.15.159:3300/0 conn(0x7f416c0cee10 0x7f416c161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4171000700 1 -- 172.21.15.12:0/2871473448 >> 172.21.15.12:0/2871473448 conn(0x7f416c08aff0 msgr2=0x7f416c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:19.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4171000700 1 -- 172.21.15.12:0/2871473448 shutdown_connections 2024-03-29T11:32:19.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.819+0000 7f4171000700 1 -- 172.21.15.12:0/2871473448 wait complete. 2024-03-29T11:32:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.821+0000 7f4171000700 1 Processor -- start 2024-03-29T11:32:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.821+0000 7f4171000700 1 -- start start 2024-03-29T11:32:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.822+0000 7f4171000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f416c0cee10 0x7f416c156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.822+0000 7f4171000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c15f4d0 0x7f416c0c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.822+0000 7f4171000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f416c1611b0 0x7f416c158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.822+0000 7f4171000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c0b65c0 con 0x7f416c15f4d0 2024-03-29T11:32:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.822+0000 7f4171000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c0b68a0 con 0x7f416c0cee10 2024-03-29T11:32:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.822+0000 7f4171000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416c0b6b80 con 0x7f416c1611b0 2024-03-29T11:32:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.822+0000 7f4169400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c15f4d0 0x7f416c0c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.822+0000 7f4169400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c15f4d0 0x7f416c0c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48620/0 (socket says 172.21.15.12:48620) 2024-03-29T11:32:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.822+0000 7f4169400700 1 -- 172.21.15.12:0/1109053247 learned_addr learned my addr 172.21.15.12:0/1109053247 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.822+0000 7f4169e00700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f416c0cee10 0x7f416c156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f416a800700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f416c1611b0 0x7f416c158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f4169400700 1 -- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f416c1611b0 msgr2=0x7f416c158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f4169400700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f416c1611b0 0x7f416c158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f4169400700 1 -- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f416c0cee10 msgr2=0x7f416c156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f4169400700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f416c0cee10 0x7f416c156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f4169400700 1 -- 172.21.15.12:0/1109053247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c0094a0 con 0x7f416c15f4d0 2024-03-29T11:32:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f4169400700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c15f4d0 0x7f416c0c7d10 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f415800c520 tx=0x7f415800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f4162000700 1 -- 172.21.15.12:0/1109053247 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158004c80 con 0x7f416c15f4d0 2024-03-29T11:32:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f4162000700 1 -- 172.21.15.12:0/1109053247 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4158004e00 con 0x7f416c15f4d0 2024-03-29T11:32:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f4162000700 1 -- 172.21.15.12:0/1109053247 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158007310 con 0x7f416c15f4d0 2024-03-29T11:32:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.823+0000 7f4171000700 1 -- 172.21.15.12:0/1109053247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c0b6e60 con 0x7f416c15f4d0 2024-03-29T11:32:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.824+0000 7f4171000700 1 -- 172.21.15.12:0/1109053247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c0b6150 con 0x7f416c15f4d0 2024-03-29T11:32:19.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.824+0000 7f4171000700 1 -- 172.21.15.12:0/1109053247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c0b7380 con 0x7f416c15f4d0 2024-03-29T11:32:19.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.825+0000 7f4162000700 1 -- 172.21.15.12:0/1109053247 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4158007470 con 0x7f416c15f4d0 2024-03-29T11:32:19.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.826+0000 7f4162000700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f414c06aed0 0x7f414c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:19.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.826+0000 7f4162000700 1 -- 172.21.15.12:0/1109053247 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6610+0+0 (secure 0 0 0) 0x7f41580896f0 con 0x7f416c15f4d0 2024-03-29T11:32:19.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.827+0000 7f4169e00700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f414c06aed0 0x7f414c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:19.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.828+0000 7f4169e00700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f414c06aed0 0x7f414c06d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f415c0090b0 tx=0x7f415c01c040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:19.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:19.831+0000 7f4162000700 1 -- 172.21.15.12:0/1109053247 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f416c0b7380 con 0x7f416c15f4d0 2024-03-29T11:32:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.091+0000 7f4171000700 1 -- 172.21.15.12:0/1109053247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f416c0b7380 con 0x7f416c15f4d0 2024-03-29T11:32:20.087 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T11:32:20.092+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T11:32:20.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.100+0000 7f4162000700 1 -- 172.21.15.12:0/1109053247 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v115) v1 ==== 167+0+0 (secure 0 0 0) 0x7f416c0b7380 con 0x7f416c15f4d0 2024-03-29T11:32:20.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 -- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f414c06aed0 msgr2=0x7f414c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f414c06aed0 0x7f414c06d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f415c0090b0 tx=0x7f415c01c040 comp rx=0 tx=0).stop 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 -- 172.21.15.12:0/1109053247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c15f4d0 msgr2=0x7f416c0c7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c15f4d0 0x7f416c0c7d10 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f415800c520 tx=0x7f415800c830 comp rx=0 tx=0).stop 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 -- 172.21.15.12:0/1109053247 shutdown_connections 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f414c06aed0 0x7f414c06d390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f416c1611b0 0x7f416c158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f416c0cee10 0x7f416c156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 --2- 172.21.15.12:0/1109053247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f416c15f4d0 0x7f416c0c7d10 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 -- 172.21.15.12:0/1109053247 >> 172.21.15.12:0/1109053247 conn(0x7f416c08aff0 msgr2=0x7f416c161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 -- 172.21.15.12:0/1109053247 shutdown_connections 2024-03-29T11:32:20.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.103+0000 7f4171000700 1 -- 172.21.15.12:0/1109053247 wait complete. 2024-03-29T11:32:20.100 INFO:teuthology.orchestra.run.smithi012.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T11:32:20.103 INFO:tasks.ceph.mds.b.smithi159.stderr: -1984> 2024-03-29T11:31:46.465+0000 7fd80e400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:32:20.118 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-03-29T11:32:20.150 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T11:32:20.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.325+0000 7fa786000700 1 Processor -- start 2024-03-29T11:32:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.325+0000 7fa786000700 1 -- start start 2024-03-29T11:32:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.326+0000 7fa786000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa7800b2b30 0x7fa7800b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.326+0000 7fa786000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7800b3960 0x7fa7800cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.326+0000 7fa786000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7800b3d70 con 0x7fa780157f90 2024-03-29T11:32:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.326+0000 7fa786000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa780158370 con 0x7fa7800b3960 2024-03-29T11:32:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.326+0000 7fa786000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7800b2710 con 0x7fa7800b2b30 2024-03-29T11:32:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.327+0000 7fa77ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa7800b2b30 0x7fa7800b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.327+0000 7fa77fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7800b3960 0x7fa7800cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.327+0000 7fa77f400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa780157f90 0x7fa7801613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:58958/0 (socket says 172.21.15.12:58958) 2024-03-29T11:32:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.327+0000 7fa77f400700 1 -- 172.21.15.12:0/1689190872 learned_addr learned my addr 172.21.15.12:0/1689190872 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.327+0000 7fa77fe00700 1 -- 172.21.15.12:0/1689190872 >> v2:172.21.15.159:3300/0 conn(0x7fa7800b2b30 msgr2=0x7fa7800b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.327+0000 7fa77fe00700 1 --2- 172.21.15.12:0/1689190872 >> v2:172.21.15.159:3300/0 conn(0x7fa7800b2b30 0x7fa7800b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.327+0000 7fa77fe00700 1 -- 172.21.15.12:0/1689190872 >> v1:172.21.15.159:6789/0 conn(0x7fa780157f90 legacy=0x7fa7801613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.327+0000 7fa77fe00700 1 -- 172.21.15.12:0/1689190872 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800b2070 con 0x7fa7800b3960 2024-03-29T11:32:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.327+0000 7fa77fe00700 1 --2- 172.21.15.12:0/1689190872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b3960 0x7fa7800cd020 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fa7800b4210 tx=0x7fa77800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab1de3c7c257c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.328+0000 7fa77e000700 1 -- 172.21.15.12:0/1689190872 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77800e070 con 0x7fa7800b3960 2024-03-29T11:32:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.328+0000 7fa77e000700 1 -- 172.21.15.12:0/1689190872 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77800bb20 con 0x7fa7800b3960 2024-03-29T11:32:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.328+0000 7fa77e000700 1 -- 172.21.15.12:0/1689190872 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778012430 con 0x7fa7800b3960 2024-03-29T11:32:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.329+0000 7fa786000700 1 -- 172.21.15.12:0/1689190872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b3960 msgr2=0x7fa7800cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.329+0000 7fa786000700 1 --2- 172.21.15.12:0/1689190872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b3960 0x7fa7800cd020 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fa7800b4210 tx=0x7fa77800b330 comp rx=0 tx=0).stop 2024-03-29T11:32:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.329+0000 7fa786000700 1 -- 172.21.15.12:0/1689190872 shutdown_connections 2024-03-29T11:32:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.329+0000 7fa786000700 1 --2- 172.21.15.12:0/1689190872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b3960 0x7fa7800cd020 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.329+0000 7fa786000700 1 --2- 172.21.15.12:0/1689190872 >> v2:172.21.15.159:3300/0 conn(0x7fa7800b2b30 0x7fa7800b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.329+0000 7fa786000700 1 -- 172.21.15.12:0/1689190872 >> 172.21.15.12:0/1689190872 conn(0x7fa7800b0d70 msgr2=0x7fa7800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.329+0000 7fa786000700 1 -- 172.21.15.12:0/1689190872 shutdown_connections 2024-03-29T11:32:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.329+0000 7fa786000700 1 -- 172.21.15.12:0/1689190872 wait complete. 2024-03-29T11:32:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.331+0000 7fa786000700 1 Processor -- start 2024-03-29T11:32:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.331+0000 7fa786000700 1 -- start start 2024-03-29T11:32:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.332+0000 7fa786000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b2b30 0x7fa7800aca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.332+0000 7fa786000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7800b3960 0x7fa78006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.332+0000 7fa786000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa780157f90 0x7fa78006d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.332+0000 7fa786000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78015f050 con 0x7fa7800b2b30 2024-03-29T11:32:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.332+0000 7fa786000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78014c720 con 0x7fa780157f90 2024-03-29T11:32:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.332+0000 7fa786000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800ad040 con 0x7fa7800b3960 2024-03-29T11:32:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.332+0000 7fa77f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b2b30 0x7fa7800aca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.332+0000 7fa77f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b2b30 0x7fa7800aca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48638/0 (socket says 172.21.15.12:48638) 2024-03-29T11:32:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.332+0000 7fa77f400700 1 -- 172.21.15.12:0/1090910154 learned_addr learned my addr 172.21.15.12:0/1090910154 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa77fe00700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa780157f90 0x7fa78006d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa77ea00700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7800b3960 0x7fa78006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa77f400700 1 -- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7800b3960 msgr2=0x7fa78006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa77f400700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7800b3960 0x7fa78006e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa77f400700 1 -- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa780157f90 msgr2=0x7fa78006d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa77f400700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa780157f90 0x7fa78006d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa77f400700 1 -- 172.21.15.12:0/1090910154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa778009a70 con 0x7fa7800b2b30 2024-03-29T11:32:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa77f400700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b2b30 0x7fa7800aca60 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fa77000c550 tx=0x7fa77000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa777400700 1 -- 172.21.15.12:0/1090910154 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa770005d70 con 0x7fa7800b2b30 2024-03-29T11:32:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa777400700 1 -- 172.21.15.12:0/1090910154 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7700070d0 con 0x7fa7800b2b30 2024-03-29T11:32:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa777400700 1 -- 172.21.15.12:0/1090910154 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa770007440 con 0x7fa7800b2b30 2024-03-29T11:32:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.333+0000 7fa786000700 1 -- 172.21.15.12:0/1090910154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7800ad320 con 0x7fa7800b2b30 2024-03-29T11:32:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.334+0000 7fa786000700 1 -- 172.21.15.12:0/1090910154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7800ad8e0 con 0x7fa7800b2b30 2024-03-29T11:32:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.335+0000 7fa786000700 1 -- 172.21.15.12:0/1090910154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa780067720 con 0x7fa7800b2b30 2024-03-29T11:32:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.336+0000 7fa777400700 1 -- 172.21.15.12:0/1090910154 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa7700075a0 con 0x7fa7800b2b30 2024-03-29T11:32:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.336+0000 7fa777400700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa76006aed0 0x7fa76006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.336+0000 7fa777400700 1 -- 172.21.15.12:0/1090910154 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6696+0+0 (secure 0 0 0) 0x7fa770088d50 con 0x7fa7800b2b30 2024-03-29T11:32:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.337+0000 7fa77ea00700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa76006aed0 0x7fa76006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.337+0000 7fa77ea00700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa76006aed0 0x7fa76006d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009b40 tx=0x7fa76c009210 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.341+0000 7fa777400700 1 -- 172.21.15.12:0/1090910154 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa77008c030 con 0x7fa7800b2b30 2024-03-29T11:32:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:20.610+0000 7fa786000700 1 -- 172.21.15.12:0/1090910154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa780070fe0 con 0x7fa7800b2b30 2024-03-29T11:32:20.961 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:32:20.968 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:32:21.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.102+0000 7fa777400700 1 -- 172.21.15.12:0/1090910154 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v116) v1 ==== 107+0+0 (secure 0 0 0) 0x7fa7700170b0 con 0x7fa7800b2b30 2024-03-29T11:32:21.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.105+0000 7fa786000700 1 -- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa76006aed0 msgr2=0x7fa76006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.105+0000 7fa786000700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa76006aed0 0x7fa76006d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009b40 tx=0x7fa76c009210 comp rx=0 tx=0).stop 2024-03-29T11:32:21.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.105+0000 7fa786000700 1 -- 172.21.15.12:0/1090910154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b2b30 msgr2=0x7fa7800aca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.105+0000 7fa786000700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b2b30 0x7fa7800aca60 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fa77000c550 tx=0x7fa77000c860 comp rx=0 tx=0).stop 2024-03-29T11:32:21.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.105+0000 7fa786000700 1 -- 172.21.15.12:0/1090910154 shutdown_connections 2024-03-29T11:32:21.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.105+0000 7fa786000700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa76006aed0 0x7fa76006d390 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:21.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.105+0000 7fa786000700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa780157f90 0x7fa78006d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:21.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.105+0000 7fa786000700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7800b2b30 0x7fa7800aca60 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:21.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.105+0000 7fa786000700 1 --2- 172.21.15.12:0/1090910154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7800b3960 0x7fa78006e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:21.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.105+0000 7fa786000700 1 -- 172.21.15.12:0/1090910154 >> 172.21.15.12:0/1090910154 conn(0x7fa7800b0d70 msgr2=0x7fa7800cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:21.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.106+0000 7fa786000700 1 -- 172.21.15.12:0/1090910154 shutdown_connections 2024-03-29T11:32:21.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.106+0000 7fa786000700 1 -- 172.21.15.12:0/1090910154 wait complete. 2024-03-29T11:32:21.120 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-03-29T11:32:21.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.370+0000 7f51ae000700 1 Processor -- start 2024-03-29T11:32:21.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.370+0000 7f51ae000700 1 -- start start 2024-03-29T11:32:21.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.371+0000 7f51ae000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f51a80ced30 0x7f51a80d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.371+0000 7f51ae000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f51a80d2710 0x7f51a815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51ae000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a80ce8a0 con 0x7f51a815ddf0 2024-03-29T11:32:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51ae000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a815ec60 con 0x7f51a80d2710 2024-03-29T11:32:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51ae000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a80d2b60 con 0x7f51a80ced30 2024-03-29T11:32:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51a7400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f51a815ddf0 0x7f51a815e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37398/0 (socket says 172.21.15.12:37398) 2024-03-29T11:32:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51a7400700 1 -- 172.21.15.12:0/686662725 learned_addr learned my addr 172.21.15.12:0/686662725 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51a6a00700 1 --2- 172.21.15.12:0/686662725 >> v2:172.21.15.159:3300/0 conn(0x7f51a80ced30 0x7f51a80d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51a7e00700 1 --2- 172.21.15.12:0/686662725 >> v2:172.21.15.12:3300/0 conn(0x7f51a80d2710 0x7f51a815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:21.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51a6000700 1 -- 172.21.15.12:0/686662725 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384029649 0 0) 0x7f51a80ce8a0 con 0x7f51a815ddf0 2024-03-29T11:32:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51a6000700 1 -- 172.21.15.12:0/686662725 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c0035b0 con 0x7f51a815ddf0 2024-03-29T11:32:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51a7e00700 1 -- 172.21.15.12:0/686662725 >> v2:172.21.15.159:3300/0 conn(0x7f51a80ced30 msgr2=0x7f51a80d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51a7e00700 1 --2- 172.21.15.12:0/686662725 >> v2:172.21.15.159:3300/0 conn(0x7f51a80ced30 0x7f51a80d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.372+0000 7f51a7e00700 1 -- 172.21.15.12:0/686662725 >> v1:172.21.15.12:6789/0 conn(0x7f51a815ddf0 legacy=0x7f51a815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.373+0000 7f51a7e00700 1 -- 172.21.15.12:0/686662725 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a80cdc90 con 0x7f51a80d2710 2024-03-29T11:32:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.373+0000 7f51a6a00700 1 --2- 172.21.15.12:0/686662725 >> v2:172.21.15.159:3300/0 conn(0x7f51a80ced30 0x7f51a80d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.373+0000 7f51a7e00700 1 --2- 172.21.15.12:0/686662725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80d2710 0x7f51a815f0f0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f51a80d0fe0 tx=0x7f51a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f361a73352e2826 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:21.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.373+0000 7f51a6000700 1 -- 172.21.15.12:0/686662725 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a000e070 con 0x7f51a80d2710 2024-03-29T11:32:21.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.373+0000 7f51a6000700 1 -- 172.21.15.12:0/686662725 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a000bb20 con 0x7f51a80d2710 2024-03-29T11:32:21.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.373+0000 7f51a6000700 1 -- 172.21.15.12:0/686662725 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a000be90 con 0x7f51a80d2710 2024-03-29T11:32:21.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.374+0000 7f51ae000700 1 -- 172.21.15.12:0/686662725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80d2710 msgr2=0x7f51a815f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.374+0000 7f51ae000700 1 --2- 172.21.15.12:0/686662725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80d2710 0x7f51a815f0f0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f51a80d0fe0 tx=0x7f51a000b330 comp rx=0 tx=0).stop 2024-03-29T11:32:21.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.374+0000 7f51ae000700 1 -- 172.21.15.12:0/686662725 shutdown_connections 2024-03-29T11:32:21.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.374+0000 7f51ae000700 1 --2- 172.21.15.12:0/686662725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80d2710 0x7f51a815f0f0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:21.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.374+0000 7f51ae000700 1 --2- 172.21.15.12:0/686662725 >> v2:172.21.15.159:3300/0 conn(0x7f51a80ced30 0x7f51a80d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:21.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.374+0000 7f51ae000700 1 -- 172.21.15.12:0/686662725 >> 172.21.15.12:0/686662725 conn(0x7f51a808aff0 msgr2=0x7f51a80d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:21.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.374+0000 7f51ae000700 1 -- 172.21.15.12:0/686662725 shutdown_connections 2024-03-29T11:32:21.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.374+0000 7f51ae000700 1 -- 172.21.15.12:0/686662725 wait complete. 2024-03-29T11:32:21.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.376+0000 7f51ae000700 1 Processor -- start 2024-03-29T11:32:21.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.376+0000 7f51ae000700 1 -- start start 2024-03-29T11:32:21.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.377+0000 7f51ae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80ced30 0x7f51a814b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:21.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51ae000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51a80d2710 0x7f51a81f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:21.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51ae000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51a815ddf0 0x7f51a81f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:21.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51ae000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a80cc150 con 0x7f51a80ced30 2024-03-29T11:32:21.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51ae000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a80ca1a0 con 0x7f51a815ddf0 2024-03-29T11:32:21.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51ae000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a80cc3a0 con 0x7f51a80d2710 2024-03-29T11:32:21.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80ced30 0x7f51a814b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:21.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80ced30 0x7f51a814b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48666/0 (socket says 172.21.15.12:48666) 2024-03-29T11:32:21.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a7400700 1 -- 172.21.15.12:0/62948261 learned_addr learned my addr 172.21.15.12:0/62948261 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:21.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a6a00700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51a80d2710 0x7f51a81f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:21.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a7e00700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51a815ddf0 0x7f51a81f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:21.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a7400700 1 -- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51a80d2710 msgr2=0x7f51a81f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a7400700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51a80d2710 0x7f51a81f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:21.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a7400700 1 -- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51a815ddf0 msgr2=0x7f51a81f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a7400700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51a815ddf0 0x7f51a81f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:21.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a7400700 1 -- 172.21.15.12:0/62948261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51940094a0 con 0x7f51a80ced30 2024-03-29T11:32:21.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a6a00700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51a80d2710 0x7f51a81f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:21.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.378+0000 7f51a7e00700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51a815ddf0 0x7f51a81f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:21.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.379+0000 7f51a7400700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80ced30 0x7f51a814b040 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f519800c420 tx=0x7f519800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:21.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.379+0000 7f519f400700 1 -- 172.21.15.12:0/62948261 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5198005d70 con 0x7f51a80ced30 2024-03-29T11:32:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.379+0000 7f519f400700 1 -- 172.21.15.12:0/62948261 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51980070d0 con 0x7f51a80ced30 2024-03-29T11:32:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.379+0000 7f51ae000700 1 -- 172.21.15.12:0/62948261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a0009a70 con 0x7f51a80ced30 2024-03-29T11:32:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.379+0000 7f519f400700 1 -- 172.21.15.12:0/62948261 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5198007440 con 0x7f51a80ced30 2024-03-29T11:32:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.379+0000 7f51ae000700 1 -- 172.21.15.12:0/62948261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a80cc960 con 0x7f51a80ced30 2024-03-29T11:32:21.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.380+0000 7f51ae000700 1 -- 172.21.15.12:0/62948261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a8067720 con 0x7f51a80ced30 2024-03-29T11:32:21.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.381+0000 7f519f400700 1 -- 172.21.15.12:0/62948261 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f51980075a0 con 0x7f51a80ced30 2024-03-29T11:32:21.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.382+0000 7f519f400700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f518806aed0 0x7f518806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:21.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.382+0000 7f519f400700 1 -- 172.21.15.12:0/62948261 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6696+0+0 (secure 0 0 0) 0x7f5198088f60 con 0x7f51a80ced30 2024-03-29T11:32:21.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.382+0000 7f51a6a00700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f518806aed0 0x7f518806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:21.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.383+0000 7f51a6a00700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f518806aed0 0x7f518806d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f51a80b8260 tx=0x7f5194008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:21.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.387+0000 7f519f400700 1 -- 172.21.15.12:0/62948261 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5198053a10 con 0x7f51a80ced30 2024-03-29T11:32:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:21.600+0000 7f51ae000700 1 -- 172.21.15.12:0/62948261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f51a80d3050 con 0x7f51a80ced30 2024-03-29T11:32:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.343+0000 7f519f400700 1 -- 172.21.15.12:0/62948261 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v238) v1 ==== 190+0+0 (secure 0 0 0) 0x7f5198017020 con 0x7f51a80ced30 2024-03-29T11:32:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.347+0000 7f51ae000700 1 -- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f518806aed0 msgr2=0x7f518806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.347+0000 7f51ae000700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f518806aed0 0x7f518806d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f51a80b8260 tx=0x7f5194008040 comp rx=0 tx=0).stop 2024-03-29T11:32:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.347+0000 7f51ae000700 1 -- 172.21.15.12:0/62948261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80ced30 msgr2=0x7f51a814b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.347+0000 7f51ae000700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80ced30 0x7f51a814b040 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f519800c420 tx=0x7f519800c7e0 comp rx=0 tx=0).stop 2024-03-29T11:32:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.348+0000 7f51ae000700 1 -- 172.21.15.12:0/62948261 shutdown_connections 2024-03-29T11:32:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.348+0000 7f51ae000700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f518806aed0 0x7f518806d390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.348+0000 7f51ae000700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51a815ddf0 0x7f51a81f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.348+0000 7f51ae000700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51a80ced30 0x7f51a814b040 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.348+0000 7f51ae000700 1 --2- 172.21.15.12:0/62948261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51a80d2710 0x7f51a81f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.348+0000 7f51ae000700 1 -- 172.21.15.12:0/62948261 >> 172.21.15.12:0/62948261 conn(0x7f51a808aff0 msgr2=0x7f51a815e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.348+0000 7f51ae000700 1 -- 172.21.15.12:0/62948261 shutdown_connections 2024-03-29T11:32:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.348+0000 7f51ae000700 1 -- 172.21.15.12:0/62948261 wait complete. 2024-03-29T11:32:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'backup_fs_metadata' removed 2024-03-29T11:32:22.365 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-03-29T11:32:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.583+0000 7f993a400700 1 Processor -- start 2024-03-29T11:32:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.583+0000 7f993a400700 1 -- start start 2024-03-29T11:32:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.584+0000 7f993a400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f993415ddf0 0x7f99340ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.584+0000 7f993a400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340ce8a0 con 0x7f993415e1d0 2024-03-29T11:32:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.584+0000 7f993a400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340cdc90 con 0x7f99340d28e0 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.584+0000 7f993a400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f993415ec60 con 0x7f993415ddf0 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.585+0000 7f9932a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f993415e1d0 0x7f99340d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37408/0 (socket says 172.21.15.12:37408) 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.585+0000 7f9932a00700 1 -- 172.21.15.12:0/4129699328 learned_addr learned my addr 172.21.15.12:0/4129699328 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.585+0000 7f9933400700 1 --2- 172.21.15.12:0/4129699328 >> v2:172.21.15.159:3300/0 conn(0x7f993415ddf0 0x7f99340ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.585+0000 7f9932000700 1 -- 172.21.15.12:0/4129699328 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510073487 0 0) 0x7f99340ce8a0 con 0x7f993415e1d0 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.585+0000 7f9932000700 1 -- 172.21.15.12:0/4129699328 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99180035b0 con 0x7f993415e1d0 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.585+0000 7f9933400700 1 -- 172.21.15.12:0/4129699328 >> v1:172.21.15.159:6789/0 conn(0x7f99340d28e0 legacy=0x7f993415f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.585+0000 7f9933400700 1 -- 172.21.15.12:0/4129699328 >> v1:172.21.15.12:6789/0 conn(0x7f993415e1d0 legacy=0x7f99340d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.585+0000 7f9932000700 1 -- 172.21.15.12:0/4129699328 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 520466867 0 0) 0x7f99180035b0 con 0x7f993415e1d0 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.585+0000 7f9933400700 1 -- 172.21.15.12:0/4129699328 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99340cdf70 con 0x7f993415ddf0 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.586+0000 7f9933400700 1 --2- 172.21.15.12:0/4129699328 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f993415ddf0 0x7f99340ced30 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9924009840 tx=0x7f9924009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1238f8926bab2631 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.586+0000 7f9932000700 1 -- 172.21.15.12:0/4129699328 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9920001670 con 0x7f993415ddf0 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.586+0000 7f9932000700 1 -- 172.21.15.12:0/4129699328 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9924004030 con 0x7f993415ddf0 2024-03-29T11:32:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.586+0000 7f9932000700 1 -- 172.21.15.12:0/4129699328 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99240043c0 con 0x7f993415ddf0 2024-03-29T11:32:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.586+0000 7f993a400700 1 -- 172.21.15.12:0/4129699328 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f993415ddf0 msgr2=0x7f99340ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.586+0000 7f993a400700 1 --2- 172.21.15.12:0/4129699328 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f993415ddf0 0x7f99340ced30 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9924009840 tx=0x7f9924009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.587+0000 7f993a400700 1 -- 172.21.15.12:0/4129699328 shutdown_connections 2024-03-29T11:32:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.587+0000 7f993a400700 1 --2- 172.21.15.12:0/4129699328 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f993415ddf0 0x7f99340ced30 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.587+0000 7f993a400700 1 -- 172.21.15.12:0/4129699328 >> 172.21.15.12:0/4129699328 conn(0x7f993408aff0 msgr2=0x7f99340d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.587+0000 7f993a400700 1 -- 172.21.15.12:0/4129699328 shutdown_connections 2024-03-29T11:32:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.587+0000 7f993a400700 1 -- 172.21.15.12:0/4129699328 wait complete. 2024-03-29T11:32:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.588+0000 7f993a400700 1 Processor -- start 2024-03-29T11:32:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.589+0000 7f993a400700 1 -- start start 2024-03-29T11:32:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.589+0000 7f993a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f993415ddf0 0x7f993415c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.589+0000 7f993a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f993415e1d0 0x7f9934160760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.589+0000 7f993a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99340c7a00 0x7f99341e9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.589+0000 7f993a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993415c980 con 0x7f993415ddf0 2024-03-29T11:32:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.589+0000 7f993a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99340c7ec0 con 0x7f99340c7a00 2024-03-29T11:32:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.589+0000 7f993a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99340c81a0 con 0x7f993415e1d0 2024-03-29T11:32:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.590+0000 7f9933400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f993415ddf0 0x7f993415c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.590+0000 7f9933400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f993415ddf0 0x7f993415c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48678/0 (socket says 172.21.15.12:48678) 2024-03-29T11:32:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.590+0000 7f9933400700 1 -- 172.21.15.12:0/2747986467 learned_addr learned my addr 172.21.15.12:0/2747986467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.590+0000 7f9933400700 1 -- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f993415e1d0 msgr2=0x7f9934160760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.590+0000 7f9933400700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f993415e1d0 0x7f9934160760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.590+0000 7f9933400700 1 -- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99340c7a00 msgr2=0x7f99341e9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.590+0000 7f9933400700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99340c7a00 0x7f99341e9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.590+0000 7f9933400700 1 -- 172.21.15.12:0/2747986467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99240094a0 con 0x7f993415ddf0 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.590+0000 7f9933400700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f993415ddf0 0x7f993415c440 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f9924009b90 tx=0x7f9924004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.591+0000 7f992b400700 1 -- 172.21.15.12:0/2747986467 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924011830 con 0x7f993415ddf0 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.591+0000 7f992b400700 1 -- 172.21.15.12:0/2747986467 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992401f070 con 0x7f993415ddf0 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.591+0000 7f992b400700 1 -- 172.21.15.12:0/2747986467 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924011bd0 con 0x7f993415ddf0 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.591+0000 7f993a400700 1 -- 172.21.15.12:0/2747986467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99340c8480 con 0x7f993415ddf0 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.591+0000 7f993a400700 1 -- 172.21.15.12:0/2747986467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99340c8a40 con 0x7f993415ddf0 2024-03-29T11:32:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.592+0000 7f993a400700 1 -- 172.21.15.12:0/2747986467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993415e640 con 0x7f993415ddf0 2024-03-29T11:32:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.593+0000 7f992b400700 1 -- 172.21.15.12:0/2747986467 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f9924011d30 con 0x7f993415ddf0 2024-03-29T11:32:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.594+0000 7f992b400700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f991406aed0 0x7f991406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.594+0000 7f992b400700 1 -- 172.21.15.12:0/2747986467 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6252+0+0 (secure 0 0 0) 0x7f9924021070 con 0x7f993415ddf0 2024-03-29T11:32:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.594+0000 7f9932a00700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f991406aed0 0x7f991406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.595+0000 7f9932a00700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f991406aed0 0x7f991406d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9920001eb0 tx=0x7f9920001a20 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.598+0000 7f992b400700 1 -- 172.21.15.12:0/2747986467 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f993415e640 con 0x7f993415ddf0 2024-03-29T11:32:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:22.811+0000 7f993a400700 1 -- 172.21.15.12:0/2747986467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f993415e640 con 0x7f993415ddf0 2024-03-29T11:32:23.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.352+0000 7f992b400700 1 -- 172.21.15.12:0/2747986467 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v239) v1 ==== 178+0+0 (secure 0 0 0) 0x7f993415e640 con 0x7f993415ddf0 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 -- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f991406aed0 msgr2=0x7f991406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f991406aed0 0x7f991406d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9920001eb0 tx=0x7f9920001a20 comp rx=0 tx=0).stop 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 -- 172.21.15.12:0/2747986467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f993415ddf0 msgr2=0x7f993415c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f993415ddf0 0x7f993415c440 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f9924009b90 tx=0x7f9924004750 comp rx=0 tx=0).stop 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 -- 172.21.15.12:0/2747986467 shutdown_connections 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f991406aed0 0x7f991406d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99340c7a00 0x7f99341e9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f993415ddf0 0x7f993415c440 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 --2- 172.21.15.12:0/2747986467 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f993415e1d0 0x7f9934160760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 -- 172.21.15.12:0/2747986467 >> 172.21.15.12:0/2747986467 conn(0x7f993408aff0 msgr2=0x7f99340b8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 -- 172.21.15.12:0/2747986467 shutdown_connections 2024-03-29T11:32:23.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.355+0000 7f993a400700 1 -- 172.21.15.12:0/2747986467 wait complete. 2024-03-29T11:32:23.352 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'backup_fs_data' removed 2024-03-29T11:32:23.372 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T11:32:23.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.588+0000 7fad78a00700 1 Processor -- start 2024-03-29T11:32:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.588+0000 7fad78a00700 1 -- start start 2024-03-29T11:32:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.589+0000 7fad78a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fad7415d710 0x7fad740d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.590+0000 7fad78a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad740cf640 con 0x7fad740d1b70 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.590+0000 7fad78a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad740d0d30 con 0x7fad740d2020 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.590+0000 7fad78a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad740cf220 con 0x7fad7415d710 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.590+0000 7fad71400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fad740d1b70 0x7fad740cff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37418/0 (socket says 172.21.15.12:37418) 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.590+0000 7fad71400700 1 -- 172.21.15.12:0/3999563816 learned_addr learned my addr 172.21.15.12:0/3999563816 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.590+0000 7fad71e00700 1 --2- 172.21.15.12:0/3999563816 >> v2:172.21.15.159:3300/0 conn(0x7fad7415d710 0x7fad740d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.590+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654906047 0 0) 0x7fad740cf640 con 0x7fad740d1b70 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.590+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad580035b0 con 0x7fad740d1b70 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 589807726 0 0) 0x7fad580035b0 con 0x7fad740d1b70 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 >> v2:172.21.15.159:3300/0 conn(0x7fad7415d710 msgr2=0x7fad740d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 --2- 172.21.15.12:0/3999563816 >> v2:172.21.15.159:3300/0 conn(0x7fad7415d710 0x7fad740d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 >> v1:172.21.15.159:6789/0 conn(0x7fad740d2020 legacy=0x7fad740d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad740ce5a0 con 0x7fad740d1b70 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fad600017f0 con 0x7fad740d1b70 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 >> v1:172.21.15.12:6789/0 conn(0x7fad740d1b70 legacy=0x7fad740cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 0x7fad58003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad58004850 0x7fad58004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad58005210 0x7fad5800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad740cf220 con 0x7fad58005210 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5800c1b0 con 0x7fad58003a00 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5800c370 con 0x7fad58004850 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad71e00700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 0x7fad58003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad72800700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad58005210 0x7fad5800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.591+0000 7fad71400700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad58004850 0x7fad58004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad71e00700 1 -- 172.21.15.12:0/3999563816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad58004850 msgr2=0x7fad58004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad71e00700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad58004850 0x7fad58004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad71e00700 1 -- 172.21.15.12:0/3999563816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad58005210 msgr2=0x7fad5800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad71e00700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad58005210 0x7fad5800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad71e00700 1 -- 172.21.15.12:0/3999563816 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad580035b0 con 0x7fad58003a00 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad72800700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad58005210 0x7fad5800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad72800700 1 -- 172.21.15.12:0/3999563816 reap_dead start 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad71e00700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 0x7fad58003e20 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fad64009b60 tx=0x7fad58005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a35fd234424613cd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6400e6c0 con 0x7fad58003a00 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6400e820 con 0x7fad58003a00 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.592+0000 7fad70a00700 1 -- 172.21.15.12:0/3999563816 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6400eb90 con 0x7fad58003a00 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.593+0000 7fad78a00700 1 -- 172.21.15.12:0/3999563816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 msgr2=0x7fad58003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.593+0000 7fad78a00700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 0x7fad58003e20 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fad64009b60 tx=0x7fad58005880 comp rx=0 tx=0).stop 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.593+0000 7fad78a00700 1 -- 172.21.15.12:0/3999563816 shutdown_connections 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.593+0000 7fad78a00700 1 --2- 172.21.15.12:0/3999563816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 0x7fad58003e20 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.593+0000 7fad78a00700 1 -- 172.21.15.12:0/3999563816 >> 172.21.15.12:0/3999563816 conn(0x7fad7408aff0 msgr2=0x7fad740d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.593+0000 7fad78a00700 1 -- 172.21.15.12:0/3999563816 shutdown_connections 2024-03-29T11:32:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.593+0000 7fad78a00700 1 -- 172.21.15.12:0/3999563816 wait complete. 2024-03-29T11:32:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.595+0000 7fad78a00700 1 Processor -- start 2024-03-29T11:32:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.595+0000 7fad78a00700 1 -- start start 2024-03-29T11:32:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad78a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 0x7fad740b5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad78a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad74160460 0x7fad740cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad78a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad740cc490 0x7fad740cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad78a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7415e210 con 0x7fad740cc490 2024-03-29T11:32:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad78a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad740b5cc0 con 0x7fad58003a00 2024-03-29T11:32:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad78a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7415dec0 con 0x7fad74160460 2024-03-29T11:32:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad72800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad740cc490 0x7fad740cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad72800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad740cc490 0x7fad740cc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48696/0 (socket says 172.21.15.12:48696) 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad72800700 1 -- 172.21.15.12:0/1634262395 learned_addr learned my addr 172.21.15.12:0/1634262395 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad71e00700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 0x7fad740b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad71400700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad74160460 0x7fad740cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad72800700 1 -- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad74160460 msgr2=0x7fad740cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad72800700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad74160460 0x7fad740cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad72800700 1 -- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 msgr2=0x7fad740b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad72800700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 0x7fad740b5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.596+0000 7fad72800700 1 -- 172.21.15.12:0/1634262395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad740ccf60 con 0x7fad740cc490 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.597+0000 7fad72800700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad740cc490 0x7fad740cc910 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fad740cdd90 tx=0x7fad6c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.597+0000 7fad6a000700 1 -- 172.21.15.12:0/1634262395 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c003c10 con 0x7fad740cc490 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.597+0000 7fad6a000700 1 -- 172.21.15.12:0/1634262395 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c003d70 con 0x7fad740cc490 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.597+0000 7fad78a00700 1 -- 172.21.15.12:0/1634262395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad740cd240 con 0x7fad740cc490 2024-03-29T11:32:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.597+0000 7fad6a000700 1 -- 172.21.15.12:0/1634262395 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c00db50 con 0x7fad740cc490 2024-03-29T11:32:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.597+0000 7fad78a00700 1 -- 172.21.15.12:0/1634262395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad740ca910 con 0x7fad740cc490 2024-03-29T11:32:23.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.598+0000 7fad78a00700 1 -- 172.21.15.12:0/1634262395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7415e030 con 0x7fad740cc490 2024-03-29T11:32:23.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.599+0000 7fad6a000700 1 -- 172.21.15.12:0/1634262395 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fad6c00dcb0 con 0x7fad740cc490 2024-03-29T11:32:23.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.600+0000 7fad6a000700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fad5406aed0 0x7fad5406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:23.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.600+0000 7fad6a000700 1 -- 172.21.15.12:0/1634262395 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5832+0+0 (secure 0 0 0) 0x7fad6c089900 con 0x7fad740cc490 2024-03-29T11:32:23.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.601+0000 7fad71e00700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fad5406aed0 0x7fad5406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:23.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.602+0000 7fad71e00700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fad5406aed0 0x7fad5406d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad640090b0 tx=0x7fad6401e040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:23.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.604+0000 7fad6a000700 1 -- 172.21.15.12:0/1634262395 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fad6c054700 con 0x7fad740cc490 2024-03-29T11:32:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.816+0000 7fad78a00700 1 -- 172.21.15.12:0/1634262395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fad74067e30 con 0x7fad740cc490 2024-03-29T11:32:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.820+0000 7fad6a000700 1 -- 172.21.15.12:0/1634262395 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v79) v1 ==== 115+0+0 (secure 0 0 0) 0x7fad6c057d20 con 0x7fad740cc490 2024-03-29T11:32:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 -- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fad5406aed0 msgr2=0x7fad5406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fad5406aed0 0x7fad5406d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad640090b0 tx=0x7fad6401e040 comp rx=0 tx=0).stop 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 -- 172.21.15.12:0/1634262395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad740cc490 msgr2=0x7fad740cc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad740cc490 0x7fad740cc910 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fad740cdd90 tx=0x7fad6c00d430 comp rx=0 tx=0).stop 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 -- 172.21.15.12:0/1634262395 shutdown_connections 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fad5406aed0 0x7fad5406d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad740cc490 0x7fad740cc910 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad58003a00 0x7fad740b5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 --2- 172.21.15.12:0/1634262395 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad74160460 0x7fad740cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 -- 172.21.15.12:0/1634262395 >> 172.21.15.12:0/1634262395 conn(0x7fad7408aff0 msgr2=0x7fad740b1860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 -- 172.21.15.12:0/1634262395 shutdown_connections 2024-03-29T11:32:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:23.823+0000 7fad78a00700 1 -- 172.21.15.12:0/1634262395 wait complete. 2024-03-29T11:32:23.836 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T11:32:24.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.042+0000 7f3b29800700 1 Processor -- start 2024-03-29T11:32:24.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.042+0000 7f3b29800700 1 -- start start 2024-03-29T11:32:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.043+0000 7f3b29800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b2415d8a0 0x7f3b240b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.043+0000 7f3b29800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b240cf1b0 0x7f3b240cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.043+0000 7f3b29800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240ced20 con 0x7f3b240d3eb0 2024-03-29T11:32:24.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.043+0000 7f3b29800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b240d4340 con 0x7f3b2415d8a0 2024-03-29T11:32:24.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.043+0000 7f3b29800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240ce6d0 con 0x7f3b240cf1b0 2024-03-29T11:32:24.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.043+0000 7f3b22a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b2415d8a0 0x7f3b240b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.043+0000 7f3b22a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b2415d8a0 0x7f3b240b2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48704/0 (socket says 172.21.15.12:48704) 2024-03-29T11:32:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.043+0000 7f3b22a00700 1 -- 172.21.15.12:0/1936624147 learned_addr learned my addr 172.21.15.12:0/1936624147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.043+0000 7f3b23400700 1 --2- 172.21.15.12:0/1936624147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b240cf1b0 0x7f3b240cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.044+0000 7f3b22a00700 1 -- 172.21.15.12:0/1936624147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b240cf1b0 msgr2=0x7f3b240cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.044+0000 7f3b22a00700 1 --2- 172.21.15.12:0/1936624147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b240cf1b0 0x7f3b240cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.044+0000 7f3b22a00700 1 -- 172.21.15.12:0/1936624147 >> v1:172.21.15.159:6789/0 conn(0x7f3b240d3eb0 legacy=0x7f3b2406b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.044+0000 7f3b22a00700 1 -- 172.21.15.12:0/1936624147 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240ce9b0 con 0x7f3b2415d8a0 2024-03-29T11:32:24.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.044+0000 7f3b22a00700 1 --2- 172.21.15.12:0/1936624147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b2415d8a0 0x7f3b240b2990 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f3b140098f0 tx=0x7f3b14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ccfe8aab8526b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:24.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.044+0000 7f3b21600700 1 -- 172.21.15.12:0/1936624147 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14011830 con 0x7f3b2415d8a0 2024-03-29T11:32:24.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.044+0000 7f3b21600700 1 -- 172.21.15.12:0/1936624147 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b14011990 con 0x7f3b2415d8a0 2024-03-29T11:32:24.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.044+0000 7f3b21600700 1 -- 172.21.15.12:0/1936624147 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14011d20 con 0x7f3b2415d8a0 2024-03-29T11:32:24.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.045+0000 7f3b29800700 1 -- 172.21.15.12:0/1936624147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b2415d8a0 msgr2=0x7f3b240b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.045+0000 7f3b29800700 1 --2- 172.21.15.12:0/1936624147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b2415d8a0 0x7f3b240b2990 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f3b140098f0 tx=0x7f3b14009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:24.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.045+0000 7f3b29800700 1 -- 172.21.15.12:0/1936624147 shutdown_connections 2024-03-29T11:32:24.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.045+0000 7f3b29800700 1 --2- 172.21.15.12:0/1936624147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b240cf1b0 0x7f3b240cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.045+0000 7f3b29800700 1 --2- 172.21.15.12:0/1936624147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b2415d8a0 0x7f3b240b2990 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.045+0000 7f3b29800700 1 -- 172.21.15.12:0/1936624147 >> 172.21.15.12:0/1936624147 conn(0x7f3b2408aff0 msgr2=0x7f3b240d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:24.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.045+0000 7f3b29800700 1 -- 172.21.15.12:0/1936624147 shutdown_connections 2024-03-29T11:32:24.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.045+0000 7f3b29800700 1 -- 172.21.15.12:0/1936624147 wait complete. 2024-03-29T11:32:24.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.047+0000 7f3b29800700 1 Processor -- start 2024-03-29T11:32:24.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.047+0000 7f3b29800700 1 -- start start 2024-03-29T11:32:24.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b29800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b240cf1b0 0x7f3b24150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b29800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b2415d8a0 0x7f3b240b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b29800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b240c9400 0x7f3b240b6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b29800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240c8110 con 0x7f3b240c9400 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b29800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240c8360 con 0x7f3b2415d8a0 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b29800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240c8640 con 0x7f3b240cf1b0 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b23400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b240c9400 0x7f3b240b6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b23400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b240c9400 0x7f3b240b6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48720/0 (socket says 172.21.15.12:48720) 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b23400700 1 -- 172.21.15.12:0/805770939 learned_addr learned my addr 172.21.15.12:0/805770939 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b22a00700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b240cf1b0 0x7f3b24150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b22000700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b2415d8a0 0x7f3b240b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.048+0000 7f3b23400700 1 -- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b240cf1b0 msgr2=0x7f3b24150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.049+0000 7f3b23400700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b240cf1b0 0x7f3b24150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.049+0000 7f3b23400700 1 -- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b2415d8a0 msgr2=0x7f3b240b8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.049+0000 7f3b23400700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b2415d8a0 0x7f3b240b8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.049+0000 7f3b23400700 1 -- 172.21.15.12:0/805770939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b14009550 con 0x7f3b240c9400 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.049+0000 7f3b23400700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b240c9400 0x7f3b240b6100 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00efc0 tx=0x7f3b1c00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.049+0000 7f3b1aa00700 1 -- 172.21.15.12:0/805770939 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c00cad0 con 0x7f3b240c9400 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.049+0000 7f3b1aa00700 1 -- 172.21.15.12:0/805770939 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c00f040 con 0x7f3b240c9400 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.049+0000 7f3b1aa00700 1 -- 172.21.15.12:0/805770939 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c00ce70 con 0x7f3b240c9400 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.049+0000 7f3b29800700 1 -- 172.21.15.12:0/805770939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b240c8920 con 0x7f3b240c9400 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.049+0000 7f3b29800700 1 -- 172.21.15.12:0/805770939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b240c8ee0 con 0x7f3b240c9400 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.050+0000 7f3b29800700 1 -- 172.21.15.12:0/805770939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b240c9100 con 0x7f3b240c9400 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.051+0000 7f3b1aa00700 1 -- 172.21.15.12:0/805770939 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f3b1c014480 con 0x7f3b240c9400 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.052+0000 7f3b1aa00700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3b0406aed0 0x7f3b0406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.052+0000 7f3b1aa00700 1 -- 172.21.15.12:0/805770939 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5832+0+0 (secure 0 0 0) 0x7f3b1c08a380 con 0x7f3b240c9400 2024-03-29T11:32:24.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.053+0000 7f3b22a00700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3b0406aed0 0x7f3b0406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.053+0000 7f3b22a00700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3b0406aed0 0x7f3b0406d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009c40 tx=0x7f3b14008030 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:24.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.057+0000 7f3b1aa00700 1 -- 172.21.15.12:0/805770939 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3b1c054fd0 con 0x7f3b240c9400 2024-03-29T11:32:24.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.270+0000 7f3b29800700 1 -- 172.21.15.12:0/805770939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3b24067720 con 0x7f3b240c9400 2024-03-29T11:32:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.276+0000 7f3b1aa00700 1 -- 172.21.15.12:0/805770939 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v80) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3b1c0585f0 con 0x7f3b240c9400 2024-03-29T11:32:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 -- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3b0406aed0 msgr2=0x7f3b0406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3b0406aed0 0x7f3b0406d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009c40 tx=0x7f3b14008030 comp rx=0 tx=0).stop 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 -- 172.21.15.12:0/805770939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b240c9400 msgr2=0x7f3b240b6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b240c9400 0x7f3b240b6100 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00efc0 tx=0x7f3b1c00c0c0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 -- 172.21.15.12:0/805770939 shutdown_connections 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3b0406aed0 0x7f3b0406d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b240c9400 0x7f3b240b6100 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b240cf1b0 0x7f3b24150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 --2- 172.21.15.12:0/805770939 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b2415d8a0 0x7f3b240b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 -- 172.21.15.12:0/805770939 >> 172.21.15.12:0/805770939 conn(0x7f3b2408aff0 msgr2=0x7f3b240b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 -- 172.21.15.12:0/805770939 shutdown_connections 2024-03-29T11:32:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.279+0000 7f3b29800700 1 -- 172.21.15.12:0/805770939 wait complete. 2024-03-29T11:32:24.292 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T11:32:24.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.499+0000 7f91e2200700 1 Processor -- start 2024-03-29T11:32:24.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.500+0000 7f91e2200700 1 -- start start 2024-03-29T11:32:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91e2200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91dc15d9d0 0x7f91dc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91e2200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f91dc0d16a0 0x7f91dc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91e2200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91dc161420 con 0x7f91dc0d2370 2024-03-29T11:32:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91e2200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91dc0d27e0 con 0x7f91dc15d9d0 2024-03-29T11:32:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91e2200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91dc161700 con 0x7f91dc0d16a0 2024-03-29T11:32:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91db400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91dc15d9d0 0x7f91dc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91dbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f91dc0d2370 0x7f91dc15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37430/0 (socket says 172.21.15.12:37430) 2024-03-29T11:32:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91dbe00700 1 -- 172.21.15.12:0/1255070489 learned_addr learned my addr 172.21.15.12:0/1255070489 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91daa00700 1 --2- 172.21.15.12:0/1255070489 >> v2:172.21.15.159:3300/0 conn(0x7f91dc0d16a0 0x7f91dc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91db400700 1 -- 172.21.15.12:0/1255070489 >> v2:172.21.15.159:3300/0 conn(0x7f91dc0d16a0 msgr2=0x7f91dc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91db400700 1 --2- 172.21.15.12:0/1255070489 >> v2:172.21.15.159:3300/0 conn(0x7f91dc0d16a0 0x7f91dc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91db400700 1 -- 172.21.15.12:0/1255070489 >> v1:172.21.15.12:6789/0 conn(0x7f91dc0d2370 legacy=0x7f91dc15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.502+0000 7f91db400700 1 -- 172.21.15.12:0/1255070489 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc1619e0 con 0x7f91dc15d9d0 2024-03-29T11:32:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.501+0000 7f91da000700 1 -- 172.21.15.12:0/1255070489 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599971318 0 0) 0x7f91dc161420 con 0x7f91dc0d2370 2024-03-29T11:32:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.502+0000 7f91db400700 1 --2- 172.21.15.12:0/1255070489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc15d9d0 0x7f91dc0d3c30 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0d12f0 tx=0x7f91cc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a9a50cca806510 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.502+0000 7f91da000700 1 -- 172.21.15.12:0/1255070489 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc011830 con 0x7f91dc15d9d0 2024-03-29T11:32:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.502+0000 7f91da000700 1 -- 172.21.15.12:0/1255070489 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc011990 con 0x7f91dc15d9d0 2024-03-29T11:32:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.502+0000 7f91da000700 1 -- 172.21.15.12:0/1255070489 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc011d20 con 0x7f91dc15d9d0 2024-03-29T11:32:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.502+0000 7f91e2200700 1 -- 172.21.15.12:0/1255070489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc15d9d0 msgr2=0x7f91dc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.502+0000 7f91e2200700 1 --2- 172.21.15.12:0/1255070489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc15d9d0 0x7f91dc0d3c30 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0d12f0 tx=0x7f91cc009b20 comp rx=0 tx=0).stop 2024-03-29T11:32:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.503+0000 7f91e2200700 1 -- 172.21.15.12:0/1255070489 shutdown_connections 2024-03-29T11:32:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.503+0000 7f91e2200700 1 --2- 172.21.15.12:0/1255070489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc15d9d0 0x7f91dc0d3c30 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.503+0000 7f91e2200700 1 --2- 172.21.15.12:0/1255070489 >> v2:172.21.15.159:3300/0 conn(0x7f91dc0d16a0 0x7f91dc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.503+0000 7f91e2200700 1 -- 172.21.15.12:0/1255070489 >> 172.21.15.12:0/1255070489 conn(0x7f91dc08aff0 msgr2=0x7f91dc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.503+0000 7f91e2200700 1 -- 172.21.15.12:0/1255070489 shutdown_connections 2024-03-29T11:32:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.503+0000 7f91e2200700 1 -- 172.21.15.12:0/1255070489 wait complete. 2024-03-29T11:32:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.504+0000 7f91e2200700 1 Processor -- start 2024-03-29T11:32:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.505+0000 7f91e2200700 1 -- start start 2024-03-29T11:32:24.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.505+0000 7f91e2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc0d16a0 0x7f91dc15ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.505+0000 7f91e2200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91dc0d2370 0x7f91dc0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.505+0000 7f91e2200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91dc15d9d0 0x7f91dc14cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.505+0000 7f91e2200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc14b820 con 0x7f91dc0d16a0 2024-03-29T11:32:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.505+0000 7f91e2200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0cd2e0 con 0x7f91dc0d2370 2024-03-29T11:32:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.505+0000 7f91e2200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0cd550 con 0x7f91dc15d9d0 2024-03-29T11:32:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc0d16a0 0x7f91dc15ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc0d16a0 0x7f91dc15ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48750/0 (socket says 172.21.15.12:48750) 2024-03-29T11:32:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91db400700 1 -- 172.21.15.12:0/1374063975 learned_addr learned my addr 172.21.15.12:0/1374063975 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91daa00700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91dc0d2370 0x7f91dc0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91dbe00700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91dc15d9d0 0x7f91dc14cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91db400700 1 -- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91dc15d9d0 msgr2=0x7f91dc14cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91db400700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91dc15d9d0 0x7f91dc14cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91db400700 1 -- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91dc0d2370 msgr2=0x7f91dc0ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91db400700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91dc0d2370 0x7f91dc0ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91db400700 1 -- 172.21.15.12:0/1374063975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91cc0094a0 con 0x7f91dc0d16a0 2024-03-29T11:32:24.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.506+0000 7f91db400700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc0d16a0 0x7f91dc15ef30 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0090b0 tx=0x7f91cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:24.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.507+0000 7f91d3400700 1 -- 172.21.15.12:0/1374063975 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc0038a0 con 0x7f91dc0d16a0 2024-03-29T11:32:24.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.507+0000 7f91d3400700 1 -- 172.21.15.12:0/1374063975 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc01e070 con 0x7f91dc0d16a0 2024-03-29T11:32:24.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.507+0000 7f91e2200700 1 -- 172.21.15.12:0/1374063975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91dc0cd830 con 0x7f91dc0d16a0 2024-03-29T11:32:24.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.507+0000 7f91d3400700 1 -- 172.21.15.12:0/1374063975 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc003c20 con 0x7f91dc0d16a0 2024-03-29T11:32:24.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.507+0000 7f91e2200700 1 -- 172.21.15.12:0/1374063975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91dc0cab30 con 0x7f91dc0d16a0 2024-03-29T11:32:24.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.508+0000 7f91e2200700 1 -- 172.21.15.12:0/1374063975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91dc0d1ca0 con 0x7f91dc0d16a0 2024-03-29T11:32:24.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.509+0000 7f91d3400700 1 -- 172.21.15.12:0/1374063975 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f91cc01a420 con 0x7f91dc0d16a0 2024-03-29T11:32:24.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.510+0000 7f91d3400700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f91bc06aed0 0x7f91bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.510+0000 7f91d3400700 1 -- 172.21.15.12:0/1374063975 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5832+0+0 (secure 0 0 0) 0x7f91cc08a910 con 0x7f91dc0d16a0 2024-03-29T11:32:24.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.510+0000 7f91daa00700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f91bc06aed0 0x7f91bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.511+0000 7f91daa00700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f91bc06aed0 0x7f91bc06d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f91dc15b220 tx=0x7f91c8008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:24.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.515+0000 7f91d3400700 1 -- 172.21.15.12:0/1374063975 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91cc055710 con 0x7f91dc0d16a0 2024-03-29T11:32:24.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.732+0000 7f91e2200700 1 -- 172.21.15.12:0/1374063975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f91dc067720 con 0x7f91dc0d16a0 2024-03-29T11:32:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.738+0000 7f91d3400700 1 -- 172.21.15.12:0/1374063975 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v81) v1 ==== 120+0+0 (secure 0 0 0) 0x7f91cc0200b0 con 0x7f91dc0d16a0 2024-03-29T11:32:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 -- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f91bc06aed0 msgr2=0x7f91bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f91bc06aed0 0x7f91bc06d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f91dc15b220 tx=0x7f91c8008040 comp rx=0 tx=0).stop 2024-03-29T11:32:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 -- 172.21.15.12:0/1374063975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc0d16a0 msgr2=0x7f91dc15ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc0d16a0 0x7f91dc15ef30 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0090b0 tx=0x7f91cc004750 comp rx=0 tx=0).stop 2024-03-29T11:32:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 -- 172.21.15.12:0/1374063975 shutdown_connections 2024-03-29T11:32:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f91bc06aed0 0x7f91bc06d390 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f91dc15b220 tx=0x7f91c8008040 comp rx=0 tx=0).stop 2024-03-29T11:32:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91dc15d9d0 0x7f91dc14cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91dc0d16a0 0x7f91dc15ef30 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 --2- 172.21.15.12:0/1374063975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91dc0d2370 0x7f91dc0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 -- 172.21.15.12:0/1374063975 >> 172.21.15.12:0/1374063975 conn(0x7f91dc08aff0 msgr2=0x7f91dc0b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 -- 172.21.15.12:0/1374063975 shutdown_connections 2024-03-29T11:32:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.740+0000 7f91e2200700 1 -- 172.21.15.12:0/1374063975 wait complete. 2024-03-29T11:32:24.755 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-03-29T11:32:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.958+0000 7f2149000700 1 Processor -- start 2024-03-29T11:32:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.958+0000 7f2149000700 1 -- start start 2024-03-29T11:32:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.959+0000 7f2149000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2144072580 0x7f21440b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.959+0000 7f2149000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21440b2e50 0x7f21440709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.959+0000 7f2149000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21440b2180 con 0x7f21440b3af0 2024-03-29T11:32:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.959+0000 7f2149000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214415b100 con 0x7f21440b2e50 2024-03-29T11:32:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.959+0000 7f2149000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214415b3e0 con 0x7f2144072580 2024-03-29T11:32:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.959+0000 7f2141e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f21440b3af0 0x7f214415a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37446/0 (socket says 172.21.15.12:37446) 2024-03-29T11:32:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.959+0000 7f2141e00700 1 -- 172.21.15.12:0/1977235010 learned_addr learned my addr 172.21.15.12:0/1977235010 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.960+0000 7f2142800700 1 --2- 172.21.15.12:0/1977235010 >> v2:172.21.15.12:3300/0 conn(0x7f21440b2e50 0x7f21440709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.960+0000 7f2141400700 1 --2- 172.21.15.12:0/1977235010 >> v2:172.21.15.159:3300/0 conn(0x7f2144072580 0x7f21440b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.960+0000 7f2142800700 1 -- 172.21.15.12:0/1977235010 >> v2:172.21.15.159:3300/0 conn(0x7f2144072580 msgr2=0x7f21440b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.960+0000 7f2140a00700 1 -- 172.21.15.12:0/1977235010 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616608564 0 0) 0x7f21440b2180 con 0x7f21440b3af0 2024-03-29T11:32:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.960+0000 7f2142800700 1 --2- 172.21.15.12:0/1977235010 >> v2:172.21.15.159:3300/0 conn(0x7f2144072580 0x7f21440b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.960+0000 7f2142800700 1 -- 172.21.15.12:0/1977235010 >> v1:172.21.15.12:6789/0 conn(0x7f21440b3af0 legacy=0x7f214415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.960+0000 7f2142800700 1 -- 172.21.15.12:0/1977235010 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21440b23f0 con 0x7f21440b2e50 2024-03-29T11:32:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.960+0000 7f2142800700 1 --2- 172.21.15.12:0/1977235010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440b2e50 0x7f21440709c0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f214415dd90 tx=0x7f213c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f8d487a2b01617 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2140a00700 1 -- 172.21.15.12:0/1977235010 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c00e070 con 0x7f21440b2e50 2024-03-29T11:32:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2140a00700 1 -- 172.21.15.12:0/1977235010 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c00bb20 con 0x7f21440b2e50 2024-03-29T11:32:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2140a00700 1 -- 172.21.15.12:0/1977235010 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c012430 con 0x7f21440b2e50 2024-03-29T11:32:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2149000700 1 -- 172.21.15.12:0/1977235010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440b2e50 msgr2=0x7f21440709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2149000700 1 --2- 172.21.15.12:0/1977235010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440b2e50 0x7f21440709c0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f214415dd90 tx=0x7f213c00b330 comp rx=0 tx=0).stop 2024-03-29T11:32:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2149000700 1 -- 172.21.15.12:0/1977235010 shutdown_connections 2024-03-29T11:32:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2149000700 1 --2- 172.21.15.12:0/1977235010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440b2e50 0x7f21440709c0 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2149000700 1 --2- 172.21.15.12:0/1977235010 >> v2:172.21.15.159:3300/0 conn(0x7f2144072580 0x7f21440b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2149000700 1 -- 172.21.15.12:0/1977235010 >> 172.21.15.12:0/1977235010 conn(0x7f21440cf190 msgr2=0x7f21440cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2149000700 1 -- 172.21.15.12:0/1977235010 shutdown_connections 2024-03-29T11:32:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.961+0000 7f2149000700 1 -- 172.21.15.12:0/1977235010 wait complete. 2024-03-29T11:32:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.963+0000 7f2149000700 1 Processor -- start 2024-03-29T11:32:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.963+0000 7f2149000700 1 -- start start 2024-03-29T11:32:24.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.964+0000 7f2149000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2144072580 0x7f2144148550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.964+0000 7f2149000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440b2e50 0x7f2144153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.964+0000 7f2149000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21440b3af0 0x7f2144159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.964+0000 7f2149000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440ce780 con 0x7f2144072580 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.964+0000 7f2149000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440cdb00 con 0x7f21440b3af0 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.964+0000 7f2149000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21440cdde0 con 0x7f21440b2e50 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.964+0000 7f2141e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2144072580 0x7f2144148550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.964+0000 7f2141e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2144072580 0x7f2144148550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48768/0 (socket says 172.21.15.12:48768) 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.964+0000 7f2141e00700 1 -- 172.21.15.12:0/69241521 learned_addr learned my addr 172.21.15.12:0/69241521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.964+0000 7f2141400700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440b2e50 0x7f2144153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2142800700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21440b3af0 0x7f2144159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2141e00700 1 -- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440b2e50 msgr2=0x7f2144153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2141e00700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440b2e50 0x7f2144153ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2141e00700 1 -- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21440b3af0 msgr2=0x7f2144159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2141e00700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21440b3af0 0x7f2144159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2141e00700 1 -- 172.21.15.12:0/69241521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c009a70 con 0x7f2144072580 2024-03-29T11:32:24.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2141400700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440b2e50 0x7f2144153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:24.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2142800700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21440b3af0 0x7f2144159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:24.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2141e00700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2144072580 0x7f2144148550 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f213400c550 tx=0x7f213400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:24.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f213a000700 1 -- 172.21.15.12:0/69241521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134004c80 con 0x7f2144072580 2024-03-29T11:32:24.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f213a000700 1 -- 172.21.15.12:0/69241521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2134004e00 con 0x7f2144072580 2024-03-29T11:32:24.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2149000700 1 -- 172.21.15.12:0/69241521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21440ce0c0 con 0x7f2144072580 2024-03-29T11:32:24.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f213a000700 1 -- 172.21.15.12:0/69241521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134007310 con 0x7f2144072580 2024-03-29T11:32:24.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.965+0000 7f2149000700 1 -- 172.21.15.12:0/69241521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2144148a90 con 0x7f2144072580 2024-03-29T11:32:24.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.966+0000 7f2149000700 1 -- 172.21.15.12:0/69241521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2144067720 con 0x7f2144072580 2024-03-29T11:32:24.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.967+0000 7f213a000700 1 -- 172.21.15.12:0/69241521 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2134007470 con 0x7f2144072580 2024-03-29T11:32:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.969+0000 7f213a000700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 0x7f212406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.969+0000 7f213a000700 1 -- 172.21.15.12:0/69241521 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5832+0+0 (secure 0 0 0) 0x7f2134089450 con 0x7f2144072580 2024-03-29T11:32:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.969+0000 7f2141400700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 0x7f212406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.970+0000 7f2141400700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 0x7f212406d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f21440cf5e0 tx=0x7f2130008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:24.973+0000 7f213a000700 1 -- 172.21.15.12:0/69241521 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21340542e0 con 0x7f2144072580 2024-03-29T11:32:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.271+0000 7f2149000700 1 -- 172.21.15.12:0/69241521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f21440cef70 con 0x7f2144072580 2024-03-29T11:32:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.364+0000 7f213a000700 1 -- 172.21.15.12:0/69241521 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3203) v1 ==== 186+0+0 (secure 0 0 0) 0x7f2134057900 con 0x7f2144072580 2024-03-29T11:32:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 -- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 msgr2=0x7f212406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 0x7f212406d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f21440cf5e0 tx=0x7f2130008040 comp rx=0 tx=0).stop 2024-03-29T11:32:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 -- 172.21.15.12:0/69241521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2144072580 msgr2=0x7f2144148550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2144072580 0x7f2144148550 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f213400c550 tx=0x7f213400c910 comp rx=0 tx=0).stop 2024-03-29T11:32:25.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 -- 172.21.15.12:0/69241521 shutdown_connections 2024-03-29T11:32:25.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 0x7f212406d390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:25.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21440b3af0 0x7f2144159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:25.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2144072580 0x7f2144148550 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:25.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 --2- 172.21.15.12:0/69241521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440b2e50 0x7f2144153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:25.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 -- 172.21.15.12:0/69241521 >> 172.21.15.12:0/69241521 conn(0x7f21440cf190 msgr2=0x7f214415ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:25.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 -- 172.21.15.12:0/69241521 shutdown_connections 2024-03-29T11:32:25.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.366+0000 7f2149000700 1 -- 172.21.15.12:0/69241521 wait complete. 2024-03-29T11:32:25.380 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2024-03-29T11:32:25.381 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2024-03-29T11:32:25.381 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-03-29T11:32:25.381 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-03-29T11:32:25.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.585+0000 7f9476a00700 1 Processor -- start 2024-03-29T11:32:25.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.585+0000 7f9476a00700 1 -- start start 2024-03-29T11:32:25.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.586+0000 7f9476a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f947015b5d0 0x7f94700ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:25.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.586+0000 7f9476a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f94700cb3f0 0x7f94700ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:25.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.586+0000 7f9476a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947015fd50 con 0x7f94700ae930 2024-03-29T11:32:25.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.586+0000 7f9476a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947015b9b0 con 0x7f94700cb3f0 2024-03-29T11:32:25.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.586+0000 7f9476a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947015f140 con 0x7f947015b5d0 2024-03-29T11:32:25.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.586+0000 7f9474a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f94700ae930 0x7f9470140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37460/0 (socket says 172.21.15.12:37460) 2024-03-29T11:32:25.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.586+0000 7f9474a00700 1 -- 172.21.15.12:0/3060075002 learned_addr learned my addr 172.21.15.12:0/3060075002 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:25.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.586+0000 7f946f400700 1 --2- 172.21.15.12:0/3060075002 >> v2:172.21.15.12:3300/0 conn(0x7f94700cb3f0 0x7f94700ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:25.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.586+0000 7f946fe00700 1 --2- 172.21.15.12:0/3060075002 >> v2:172.21.15.159:3300/0 conn(0x7f947015b5d0 0x7f94700ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:25.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.587+0000 7f946f400700 1 -- 172.21.15.12:0/3060075002 >> v2:172.21.15.159:3300/0 conn(0x7f947015b5d0 msgr2=0x7f94700ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:25.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.587+0000 7f946f400700 1 --2- 172.21.15.12:0/3060075002 >> v2:172.21.15.159:3300/0 conn(0x7f947015b5d0 0x7f94700ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:25.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.587+0000 7f946f400700 1 -- 172.21.15.12:0/3060075002 >> v1:172.21.15.12:6789/0 conn(0x7f94700ae930 legacy=0x7f9470140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.587+0000 7f946f400700 1 -- 172.21.15.12:0/3060075002 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947015f420 con 0x7f94700cb3f0 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.587+0000 7f946ea00700 1 -- 172.21.15.12:0/3060075002 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584983276 0 0) 0x7f947015fd50 con 0x7f94700ae930 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.587+0000 7f946f400700 1 --2- 172.21.15.12:0/3060075002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94700cb3f0 0x7f94700ac0c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f947006b8b0 tx=0x7f945c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67a78990b20834e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.587+0000 7f946ea00700 1 -- 172.21.15.12:0/3060075002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c011830 con 0x7f94700cb3f0 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.587+0000 7f946ea00700 1 -- 172.21.15.12:0/3060075002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c011990 con 0x7f94700cb3f0 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.587+0000 7f946ea00700 1 -- 172.21.15.12:0/3060075002 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c011d20 con 0x7f94700cb3f0 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.588+0000 7f9476a00700 1 -- 172.21.15.12:0/3060075002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94700cb3f0 msgr2=0x7f94700ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.588+0000 7f9476a00700 1 --2- 172.21.15.12:0/3060075002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94700cb3f0 0x7f94700ac0c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f947006b8b0 tx=0x7f945c009b20 comp rx=0 tx=0).stop 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.588+0000 7f9476a00700 1 -- 172.21.15.12:0/3060075002 shutdown_connections 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.588+0000 7f9476a00700 1 --2- 172.21.15.12:0/3060075002 >> v2:172.21.15.159:3300/0 conn(0x7f947015b5d0 0x7f94700ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.588+0000 7f9476a00700 1 --2- 172.21.15.12:0/3060075002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94700cb3f0 0x7f94700ac0c0 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.588+0000 7f9476a00700 1 -- 172.21.15.12:0/3060075002 >> 172.21.15.12:0/3060075002 conn(0x7f94700cf190 msgr2=0x7f94700cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.588+0000 7f9476a00700 1 -- 172.21.15.12:0/3060075002 shutdown_connections 2024-03-29T11:32:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.588+0000 7f9476a00700 1 -- 172.21.15.12:0/3060075002 wait complete. 2024-03-29T11:32:25.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.590+0000 7f9476a00700 1 Processor -- start 2024-03-29T11:32:25.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.590+0000 7f9476a00700 1 -- start start 2024-03-29T11:32:25.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f9476a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94700ae930 0x7f94701f0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:25.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f9476a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f947015b5d0 0x7f94701573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f9476a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f947015a630 0x7f9470157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f9476a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9470157fa0 con 0x7f947015a630 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f9476a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94700cc580 con 0x7f94700ae930 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f9476a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947015db60 con 0x7f947015b5d0 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f9474a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f947015a630 0x7f9470157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f9474a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f947015a630 0x7f9470157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48782/0 (socket says 172.21.15.12:48782) 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f9474a00700 1 -- 172.21.15.12:0/1636180318 learned_addr learned my addr 172.21.15.12:0/1636180318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f946f400700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f947015b5d0 0x7f94701573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.591+0000 7f946fe00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94700ae930 0x7f94701f0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9474a00700 1 -- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f947015b5d0 msgr2=0x7f94701573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9474a00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f947015b5d0 0x7f94701573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9474a00700 1 -- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94700ae930 msgr2=0x7f94701f0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9474a00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94700ae930 0x7f94701f0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9474a00700 1 -- 172.21.15.12:0/1636180318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0094a0 con 0x7f947015a630 2024-03-29T11:32:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9474a00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f947015a630 0x7f9470157920 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f946800cb60 tx=0x7f946800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9467e00700 1 -- 172.21.15.12:0/1636180318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946800a9e0 con 0x7f947015a630 2024-03-29T11:32:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9467e00700 1 -- 172.21.15.12:0/1636180318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946800ab40 con 0x7f947015a630 2024-03-29T11:32:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9467e00700 1 -- 172.21.15.12:0/1636180318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94680070d0 con 0x7f947015a630 2024-03-29T11:32:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9476a00700 1 -- 172.21.15.12:0/1636180318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94700af460 con 0x7f947015a630 2024-03-29T11:32:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.592+0000 7f9476a00700 1 -- 172.21.15.12:0/1636180318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947015acd0 con 0x7f947015a630 2024-03-29T11:32:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.593+0000 7f9476a00700 1 -- 172.21.15.12:0/1636180318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94700af2c0 con 0x7f947015a630 2024-03-29T11:32:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.595+0000 7f9467e00700 1 -- 172.21.15.12:0/1636180318 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f9468007230 con 0x7f947015a630 2024-03-29T11:32:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.596+0000 7f9467e00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f945006aed0 0x7f945006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.596+0000 7f9467e00700 1 -- 172.21.15.12:0/1636180318 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5832+0+0 (secure 0 0 0) 0x7f94680892c0 con 0x7f947015a630 2024-03-29T11:32:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.596+0000 7f946fe00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f945006aed0 0x7f945006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.597+0000 7f946fe00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f945006aed0 0x7f945006d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9460009be0 tx=0x7f9460009210 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.600+0000 7f9467e00700 1 -- 172.21.15.12:0/1636180318 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9468054130 con 0x7f947015a630 2024-03-29T11:32:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:25.899+0000 7f9476a00700 1 -- 172.21.15.12:0/1636180318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f94700af2c0 con 0x7f947015a630 2024-03-29T11:32:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.364+0000 7f9467e00700 1 -- 172.21.15.12:0/1636180318 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3204) v1 ==== 190+0+0 (secure 0 0 0) 0x7f94700af2c0 con 0x7f947015a630 2024-03-29T11:32:26.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.367+0000 7f9476a00700 1 -- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f945006aed0 msgr2=0x7f945006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:26.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.367+0000 7f9476a00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f945006aed0 0x7f945006d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9460009be0 tx=0x7f9460009210 comp rx=0 tx=0).stop 2024-03-29T11:32:26.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.367+0000 7f9476a00700 1 -- 172.21.15.12:0/1636180318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f947015a630 msgr2=0x7f9470157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:26.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.367+0000 7f9476a00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f947015a630 0x7f9470157920 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f946800cb60 tx=0x7f946800ce70 comp rx=0 tx=0).stop 2024-03-29T11:32:26.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.367+0000 7f9476a00700 1 -- 172.21.15.12:0/1636180318 shutdown_connections 2024-03-29T11:32:26.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.367+0000 7f9476a00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f945006aed0 0x7f945006d390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.367+0000 7f9476a00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f947015a630 0x7f9470157920 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.367+0000 7f9476a00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94700ae930 0x7f94701f0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.368+0000 7f9476a00700 1 --2- 172.21.15.12:0/1636180318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f947015b5d0 0x7f94701573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.368+0000 7f9476a00700 1 -- 172.21.15.12:0/1636180318 >> 172.21.15.12:0/1636180318 conn(0x7f94700cf190 msgr2=0x7f94700b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:26.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.368+0000 7f9476a00700 1 -- 172.21.15.12:0/1636180318 shutdown_connections 2024-03-29T11:32:26.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.368+0000 7f9476a00700 1 -- 172.21.15.12:0/1636180318 wait complete. 2024-03-29T11:32:26.384 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T11:32:26.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.593+0000 7f6815600700 1 Processor -- start 2024-03-29T11:32:26.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.593+0000 7f6815600700 1 -- start start 2024-03-29T11:32:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.594+0000 7f6815600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f68100cd020 0x7f68100cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.594+0000 7f6815600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100b2070 0x7f68100b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.594+0000 7f6815600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68100cdae0 con 0x7f68100cbb30 2024-03-29T11:32:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.594+0000 7f6815600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68100cddc0 con 0x7f68100cd020 2024-03-29T11:32:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.594+0000 7f6815600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68100ce0a0 con 0x7f68100b2070 2024-03-29T11:32:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.594+0000 7f680ea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f68100cbb30 0x7f68100cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37476/0 (socket says 172.21.15.12:37476) 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.594+0000 7f680ea00700 1 -- 172.21.15.12:0/3617023427 learned_addr learned my addr 172.21.15.12:0/3617023427 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680e000700 1 --2- 172.21.15.12:0/3617023427 >> v2:172.21.15.159:3300/0 conn(0x7f68100cd020 0x7f68100cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680f400700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100b2070 0x7f68100b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771251414 0 0) 0x7f68100cdae0 con 0x7f68100cbb30 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f40035b0 con 0x7f68100cbb30 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1367984872 0 0) 0x7f67f40035b0 con 0x7f68100cbb30 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100b2070 msgr2=0x7f68100b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100b2070 0x7f68100b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 >> v2:172.21.15.159:3300/0 conn(0x7f68100cd020 msgr2=0x7f68100cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 --2- 172.21.15.12:0/3617023427 >> v2:172.21.15.159:3300/0 conn(0x7f68100cd020 0x7f68100cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681006dc20 con 0x7f68100cbb30 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f6800001670 con 0x7f68100cbb30 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 >> v1:172.21.15.12:6789/0 conn(0x7f68100cbb30 legacy=0x7f68100cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f4003a00 0x7f67f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f4004850 0x7f67f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100b2070 0x7f67f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68100cddc0 con 0x7f67f4004850 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68100ce0a0 con 0x7f67f4003a00 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.595+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f400ba10 con 0x7f68100b2070 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680e000700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f4004850 0x7f67f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680f400700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100b2070 0x7f67f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680ea00700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f4003a00 0x7f67f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680e000700 1 -- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100b2070 msgr2=0x7f67f4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680e000700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100b2070 0x7f67f4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680e000700 1 -- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f4003a00 msgr2=0x7f67f4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680e000700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f4003a00 0x7f67f4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680e000700 1 -- 172.21.15.12:0/3617023427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f40035b0 con 0x7f67f4004850 2024-03-29T11:32:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680f400700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100b2070 0x7f67f4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680e000700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f4004850 0x7f67f4004cd0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f67fc009b60 tx=0x7f67f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff51d672ae9eca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.596+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc00e6c0 con 0x7f67f4004850 2024-03-29T11:32:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.597+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc00e820 con 0x7f67f4004850 2024-03-29T11:32:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.597+0000 7f680d600700 1 -- 172.21.15.12:0/3617023427 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc00eb90 con 0x7f67f4004850 2024-03-29T11:32:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.597+0000 7f6815600700 1 -- 172.21.15.12:0/3617023427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f4004850 msgr2=0x7f67f4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.597+0000 7f6815600700 1 --2- 172.21.15.12:0/3617023427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f4004850 0x7f67f4004cd0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f67fc009b60 tx=0x7f67f4005880 comp rx=0 tx=0).stop 2024-03-29T11:32:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.597+0000 7f680f400700 1 -- 172.21.15.12:0/3617023427 reap_dead start 2024-03-29T11:32:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.597+0000 7f6815600700 1 -- 172.21.15.12:0/3617023427 shutdown_connections 2024-03-29T11:32:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.597+0000 7f6815600700 1 -- 172.21.15.12:0/3617023427 >> 172.21.15.12:0/3617023427 conn(0x7f68100b0d70 msgr2=0x7f68100b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.597+0000 7f6815600700 1 -- 172.21.15.12:0/3617023427 shutdown_connections 2024-03-29T11:32:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.597+0000 7f6815600700 1 -- 172.21.15.12:0/3617023427 wait complete. 2024-03-29T11:32:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.599+0000 7f6815600700 1 Processor -- start 2024-03-29T11:32:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.599+0000 7f6815600700 1 -- start start 2024-03-29T11:32:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f6815600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f4004850 0x7f681014c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f6815600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100c84d0 0x7f68100cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f6815600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681015a630 0x7f681015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f6815600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681015a210 con 0x7f681015a630 2024-03-29T11:32:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f6815600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6810159620 con 0x7f67f4004850 2024-03-29T11:32:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f6815600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6810159870 con 0x7f68100c84d0 2024-03-29T11:32:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f680f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681015a630 0x7f681015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f680f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681015a630 0x7f681015aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48798/0 (socket says 172.21.15.12:48798) 2024-03-29T11:32:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f680f400700 1 -- 172.21.15.12:0/3508257506 learned_addr learned my addr 172.21.15.12:0/3508257506 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f680e000700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100c84d0 0x7f68100cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.600+0000 7f680ea00700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f4004850 0x7f681014c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.601+0000 7f680f400700 1 -- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100c84d0 msgr2=0x7f68100cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.601+0000 7f680f400700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100c84d0 0x7f68100cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.601+0000 7f680f400700 1 -- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f4004850 msgr2=0x7f681014c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.601+0000 7f680f400700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f4004850 0x7f681014c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.601+0000 7f680f400700 1 -- 172.21.15.12:0/3508257506 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6810159b50 con 0x7f681015a630 2024-03-29T11:32:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.601+0000 7f680f400700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681015a630 0x7f681015aab0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f68100aecb0 tx=0x7f68080104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.601+0000 7f6806a00700 1 -- 172.21.15.12:0/3508257506 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680800e050 con 0x7f681015a630 2024-03-29T11:32:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.601+0000 7f6806a00700 1 -- 172.21.15.12:0/3508257506 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680800f040 con 0x7f681015a630 2024-03-29T11:32:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.601+0000 7f6806a00700 1 -- 172.21.15.12:0/3508257506 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808010da0 con 0x7f681015a630 2024-03-29T11:32:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.601+0000 7f6815600700 1 -- 172.21.15.12:0/3508257506 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6810159e30 con 0x7f681015a630 2024-03-29T11:32:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.602+0000 7f6815600700 1 -- 172.21.15.12:0/3508257506 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681014ca40 con 0x7f681015a630 2024-03-29T11:32:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.602+0000 7f6815600700 1 -- 172.21.15.12:0/3508257506 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681015a350 con 0x7f681015a630 2024-03-29T11:32:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.604+0000 7f6806a00700 1 -- 172.21.15.12:0/3508257506 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6808003a20 con 0x7f681015a630 2024-03-29T11:32:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.604+0000 7f6806a00700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67f006aed0 0x7f67f006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.604+0000 7f6806a00700 1 -- 172.21.15.12:0/3508257506 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5832+0+0 (secure 0 0 0) 0x7f6808089b00 con 0x7f681015a630 2024-03-29T11:32:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.605+0000 7f680ea00700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67f006aed0 0x7f67f006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.606+0000 7f680ea00700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67f006aed0 0x7f67f006d390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6800001eb0 tx=0x7f6800007040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.609+0000 7f6806a00700 1 -- 172.21.15.12:0/3508257506 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f681015a350 con 0x7f681015a630 2024-03-29T11:32:26.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.823+0000 7f6815600700 1 -- 172.21.15.12:0/3508257506 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f681015a350 con 0x7f681015a630 2024-03-29T11:32:26.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.829+0000 7f6806a00700 1 -- 172.21.15.12:0/3508257506 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7f681015a350 con 0x7f681015a630 2024-03-29T11:32:26.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 -- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67f006aed0 msgr2=0x7f67f006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:26.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67f006aed0 0x7f67f006d390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6800001eb0 tx=0x7f6800007040 comp rx=0 tx=0).stop 2024-03-29T11:32:26.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 -- 172.21.15.12:0/3508257506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681015a630 msgr2=0x7f681015aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:26.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681015a630 0x7f681015aab0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f68100aecb0 tx=0x7f68080104a0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 -- 172.21.15.12:0/3508257506 shutdown_connections 2024-03-29T11:32:26.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67f006aed0 0x7f67f006d390 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681015a630 0x7f681015aab0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67f4004850 0x7f681014c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 --2- 172.21.15.12:0/3508257506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68100c84d0 0x7f68100cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:26.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 -- 172.21.15.12:0/3508257506 >> 172.21.15.12:0/3508257506 conn(0x7f68100b0d70 msgr2=0x7f68100ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:26.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 -- 172.21.15.12:0/3508257506 shutdown_connections 2024-03-29T11:32:26.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:26.832+0000 7f6815600700 1 -- 172.21.15.12:0/3508257506 wait complete. 2024-03-29T11:32:26.847 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:32:26.848 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:32:26.934 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:32:27.029 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:32:27.029 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:32:27.029 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:32:27.075 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:32:27.075 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:32:27.075 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:32:27.122 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:32:27.122 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:32:27.123 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:32:27.167 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:32:27.167 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:32:27.167 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:32:27.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:32:27.213 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T11:32:27.213 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:27.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.419+0000 7fe488c00700 1 Processor -- start 2024-03-29T11:32:27.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.419+0000 7fe488c00700 1 -- start start 2024-03-29T11:32:27.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.420+0000 7fe488c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4840d18b0 0x7fe4840cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:27.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.420+0000 7fe488c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4840ceda0 con 0x7fe484160f10 2024-03-29T11:32:27.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.420+0000 7fe488c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4841612f0 con 0x7fe484161590 2024-03-29T11:32:27.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.420+0000 7fe488c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4840d1d30 con 0x7fe4840d18b0 2024-03-29T11:32:27.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.420+0000 7fe481e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe484160f10 0x7fe4840d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37482/0 (socket says 172.21.15.12:37482) 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.420+0000 7fe481e00700 1 -- 172.21.15.12:0/1717531845 learned_addr learned my addr 172.21.15.12:0/1717531845 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.420+0000 7fe481400700 1 --2- 172.21.15.12:0/1717531845 >> v2:172.21.15.12:3300/0 conn(0x7fe4840d18b0 0x7fe4840cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.421+0000 7fe480a00700 1 -- 172.21.15.12:0/1717531845 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872616238 0 0) 0x7fe4840ceda0 con 0x7fe484160f10 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.421+0000 7fe480a00700 1 -- 172.21.15.12:0/1717531845 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4680035b0 con 0x7fe484160f10 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.421+0000 7fe481400700 1 -- 172.21.15.12:0/1717531845 >> v1:172.21.15.159:6789/0 conn(0x7fe484161590 legacy=0x7fe4840b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.421+0000 7fe481400700 1 -- 172.21.15.12:0/1717531845 >> v1:172.21.15.12:6789/0 conn(0x7fe484160f10 legacy=0x7fe4840d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.421+0000 7fe481400700 1 -- 172.21.15.12:0/1717531845 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4840ce470 con 0x7fe4840d18b0 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.421+0000 7fe481400700 1 --2- 172.21.15.12:0/1717531845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4840d18b0 0x7fe4840cf230 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fe470009840 tx=0x7fe470009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9f634b09e5db95 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.421+0000 7fe480a00700 1 -- 172.21.15.12:0/1717531845 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470011830 con 0x7fe4840d18b0 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.421+0000 7fe480a00700 1 -- 172.21.15.12:0/1717531845 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe470011990 con 0x7fe4840d18b0 2024-03-29T11:32:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.421+0000 7fe480a00700 1 -- 172.21.15.12:0/1717531845 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470011d20 con 0x7fe4840d18b0 2024-03-29T11:32:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.422+0000 7fe488c00700 1 -- 172.21.15.12:0/1717531845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4840d18b0 msgr2=0x7fe4840cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.422+0000 7fe488c00700 1 --2- 172.21.15.12:0/1717531845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4840d18b0 0x7fe4840cf230 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fe470009840 tx=0x7fe470009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.422+0000 7fe488c00700 1 -- 172.21.15.12:0/1717531845 shutdown_connections 2024-03-29T11:32:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.422+0000 7fe488c00700 1 --2- 172.21.15.12:0/1717531845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4840d18b0 0x7fe4840cf230 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.422+0000 7fe488c00700 1 -- 172.21.15.12:0/1717531845 >> 172.21.15.12:0/1717531845 conn(0x7fe48408aff0 msgr2=0x7fe4840d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.422+0000 7fe488c00700 1 -- 172.21.15.12:0/1717531845 shutdown_connections 2024-03-29T11:32:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.422+0000 7fe488c00700 1 -- 172.21.15.12:0/1717531845 wait complete. 2024-03-29T11:32:27.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.424+0000 7fe488c00700 1 Processor -- start 2024-03-29T11:32:27.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.425+0000 7fe488c00700 1 -- start start 2024-03-29T11:32:27.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe488c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4840d18b0 0x7fe4841f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:27.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe488c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe484160f10 0x7fe4841e66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe488c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe484161590 0x7fe4841f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe488c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4841e7d10 con 0x7fe484160f10 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe488c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4841e7090 con 0x7fe4840d18b0 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe488c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4841e7370 con 0x7fe484161590 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe481400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe484160f10 0x7fe4841e66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe481400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe484160f10 0x7fe4841e66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48812/0 (socket says 172.21.15.12:48812) 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe481400700 1 -- 172.21.15.12:0/3359179483 learned_addr learned my addr 172.21.15.12:0/3359179483 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe481e00700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4840d18b0 0x7fe4841f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.426+0000 7fe482800700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe484161590 0x7fe4841f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe481400700 1 -- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe484161590 msgr2=0x7fe4841f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:27.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe481400700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe484161590 0x7fe4841f28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe481400700 1 -- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4840d18b0 msgr2=0x7fe4841f6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe481400700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4840d18b0 0x7fe4841f6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe481400700 1 -- 172.21.15.12:0/3359179483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4700094a0 con 0x7fe484160f10 2024-03-29T11:32:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe481400700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe484160f10 0x7fe4841e66b0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fe470014040 tx=0x7fe470004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe47a000700 1 -- 172.21.15.12:0/3359179483 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4700036a0 con 0x7fe484160f10 2024-03-29T11:32:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe47a000700 1 -- 172.21.15.12:0/3359179483 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47001e070 con 0x7fe484160f10 2024-03-29T11:32:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe488c00700 1 -- 172.21.15.12:0/3359179483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4841e7650 con 0x7fe484160f10 2024-03-29T11:32:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe47a000700 1 -- 172.21.15.12:0/3359179483 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470003a40 con 0x7fe484160f10 2024-03-29T11:32:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.427+0000 7fe488c00700 1 -- 172.21.15.12:0/3359179483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4841f7180 con 0x7fe484160f10 2024-03-29T11:32:27.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.428+0000 7fe488c00700 1 -- 172.21.15.12:0/3359179483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4841e7e50 con 0x7fe484160f10 2024-03-29T11:32:27.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.429+0000 7fe47a000700 1 -- 172.21.15.12:0/3359179483 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe470003ba0 con 0x7fe484160f10 2024-03-29T11:32:27.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.430+0000 7fe47a000700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe46406aed0 0x7fe46406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:27.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.431+0000 7fe47a000700 1 -- 172.21.15.12:0/3359179483 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5832+0+0 (secure 0 0 0) 0x7fe47008a710 con 0x7fe484160f10 2024-03-29T11:32:27.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.431+0000 7fe481e00700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe46406aed0 0x7fe46406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:27.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.432+0000 7fe481e00700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe46406aed0 0x7fe46406d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe474007fd0 tx=0x7fe474007dc0 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:27.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.435+0000 7fe47a000700 1 -- 172.21.15.12:0/3359179483 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe4841e7e50 con 0x7fe484160f10 2024-03-29T11:32:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.701+0000 7fe488c00700 1 -- 172.21.15.12:0/3359179483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4841e7e50 con 0x7fe484160f10 2024-03-29T11:32:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.701+0000 7fe47a000700 1 -- 172.21.15.12:0/3359179483 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3543 (secure 0 0 0) 0x7fe470020df0 con 0x7fe484160f10 2024-03-29T11:32:27.697 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:27.697 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":116,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/2862427794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2862427794},{"type":"v1","addr":"172.21.15.159:6836","nonce":2862427794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3470029955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":3470029955},{"type":"v1","addr":"172.21.15.12:6835","nonce":3470029955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":11676,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6833/1497670214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1497670214},{"type":"v1","addr":"172.21.15.12:6833","nonce":1497670214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":11763,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3596798179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3596798179},{"type":"v1","addr":"172.21.15.159:6837","nonce":3596798179}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":116}],"filesystems":[]} 2024-03-29T11:32:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 -- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe46406aed0 msgr2=0x7fe46406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe46406aed0 0x7fe46406d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe474007fd0 tx=0x7fe474007dc0 comp rx=0 tx=0).stop 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 -- 172.21.15.12:0/3359179483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe484160f10 msgr2=0x7fe4841e66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe484160f10 0x7fe4841e66b0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fe470014040 tx=0x7fe470004750 comp rx=0 tx=0).stop 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 -- 172.21.15.12:0/3359179483 shutdown_connections 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe46406aed0 0x7fe46406d390 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe484161590 0x7fe4841f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4840d18b0 0x7fe4841f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 --2- 172.21.15.12:0/3359179483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe484160f10 0x7fe4841e66b0 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 -- 172.21.15.12:0/3359179483 >> 172.21.15.12:0/3359179483 conn(0x7fe48408aff0 msgr2=0x7fe484155af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 -- 172.21.15.12:0/3359179483 shutdown_connections 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:27.704+0000 7fe488c00700 1 -- 172.21.15.12:0/3359179483 wait complete. 2024-03-29T11:32:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 116 2024-03-29T11:32:27.718 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T11:32:27.718 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T11:32:27.719 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T11:32:27.719 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:32:27.719 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T11:32:27.719 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T11:32:27.719 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T11:32:27.719 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:32:27.720 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T11:32:27.720 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T11:32:27.720 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T11:32:27.720 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:32:27.720 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T11:32:27.720 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T11:32:27.720 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T11:32:27.721 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:32:27.735 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T11:32:27.736 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T11:32:27.737 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T11:32:27.737 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T11:32:27.739 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T11:32:27.739 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T11:32:27.740 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T11:32:27.740 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T11:32:27.779 INFO:tasks.ceph.mds.a:Started 2024-03-29T11:32:27.780 DEBUG:teuthology.parallel:result is None 2024-03-29T11:32:27.782 INFO:tasks.ceph.mds.c:Started 2024-03-29T11:32:27.782 DEBUG:teuthology.parallel:result is None 2024-03-29T11:32:27.783 INFO:tasks.ceph.mds.b:Started 2024-03-29T11:32:27.784 DEBUG:teuthology.parallel:result is None 2024-03-29T11:32:27.786 INFO:tasks.ceph.mds.d:Started 2024-03-29T11:32:27.786 DEBUG:teuthology.parallel:result is None 2024-03-29T11:32:27.786 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T11:32:27.930 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:32:27.931 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T11:32:27.933 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T11:32:27.933 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T11:32:27.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-29T11:32:27.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-29T11:32:27.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-29T11:32:28.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.005+0000 7fe4eb800700 1 Processor -- start 2024-03-29T11:32:28.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.005+0000 7fe4eb800700 1 -- start start 2024-03-29T11:32:28.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4eb800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe4e40d21f0 0x7fe4e40ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4eb800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4e40cec40 0x7fe4e4161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4eb800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e40ce270 con 0x7fe4e41611b0 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4eb800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e415b600 con 0x7fe4e40cec40 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4eb800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e40d2760 con 0x7fe4e40d21f0 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4e9800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4e40cec40 0x7fe4e4161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4e3e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe4e40d21f0 0x7fe4e40ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4e9800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4e40cec40 0x7fe4e4161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48850/0 (socket says 172.21.15.12:48850) 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4e9800700 1 -- 172.21.15.12:0/3811798437 learned_addr learned my addr 172.21.15.12:0/3811798437 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4e9800700 1 -- 172.21.15.12:0/3811798437 >> v2:172.21.15.159:3300/0 conn(0x7fe4e40d21f0 msgr2=0x7fe4e40ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4e9800700 1 --2- 172.21.15.12:0/3811798437 >> v2:172.21.15.159:3300/0 conn(0x7fe4e40d21f0 0x7fe4e40ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4e9800700 1 -- 172.21.15.12:0/3811798437 >> v1:172.21.15.159:6789/0 conn(0x7fe4e41611b0 legacy=0x7fe4e415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4e9800700 1 -- 172.21.15.12:0/3811798437 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e40d2a40 con 0x7fe4e40cec40 2024-03-29T11:32:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.007+0000 7fe4e3e00700 1 --2- 172.21.15.12:0/3811798437 >> v2:172.21.15.159:3300/0 conn(0x7fe4e40d21f0 0x7fe4e40ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:28.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.008+0000 7fe4e9800700 1 --2- 172.21.15.12:0/3811798437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 0x7fe4e4161590 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fe4e408c030 tx=0x7fe4dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09fcbb3d371c6fd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.008+0000 7fe4e3400700 1 -- 172.21.15.12:0/3811798437 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc00e070 con 0x7fe4e40cec40 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.008+0000 7fe4e3400700 1 -- 172.21.15.12:0/3811798437 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4dc00b910 con 0x7fe4e40cec40 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.008+0000 7fe4e3400700 1 -- 172.21.15.12:0/3811798437 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc00bcc0 con 0x7fe4e40cec40 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.008+0000 7fe4eb800700 1 -- 172.21.15.12:0/3811798437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 msgr2=0x7fe4e4161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.008+0000 7fe4eb800700 1 --2- 172.21.15.12:0/3811798437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 0x7fe4e4161590 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fe4e408c030 tx=0x7fe4dc00b330 comp rx=0 tx=0).stop 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.008+0000 7fe4eb800700 1 -- 172.21.15.12:0/3811798437 shutdown_connections 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.008+0000 7fe4eb800700 1 --2- 172.21.15.12:0/3811798437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 0x7fe4e4161590 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.009+0000 7fe4eb800700 1 --2- 172.21.15.12:0/3811798437 >> v2:172.21.15.159:3300/0 conn(0x7fe4e40d21f0 0x7fe4e40ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.009+0000 7fe4eb800700 1 -- 172.21.15.12:0/3811798437 >> 172.21.15.12:0/3811798437 conn(0x7fe4e408aff0 msgr2=0x7fe4e40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.009+0000 7fe4eb800700 1 -- 172.21.15.12:0/3811798437 shutdown_connections 2024-03-29T11:32:28.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.009+0000 7fe4eb800700 1 -- 172.21.15.12:0/3811798437 wait complete. 2024-03-29T11:32:28.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.010+0000 7fe4eb800700 1 Processor -- start 2024-03-29T11:32:28.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.010+0000 7fe4eb800700 1 -- start start 2024-03-29T11:32:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.011+0000 7fe4eb800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 0x7fe4e4156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.011+0000 7fe4eb800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e40d21f0 0x7fe4e40ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.011+0000 7fe4eb800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4e41611b0 0x7fe4e41e7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.011+0000 7fe4eb800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e40c74f0 con 0x7fe4e40cec40 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.011+0000 7fe4eb800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e40c77d0 con 0x7fe4e41611b0 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.011+0000 7fe4eb800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e40c7ab0 con 0x7fe4e40d21f0 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 0x7fe4e4156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 0x7fe4e4156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48862/0 (socket says 172.21.15.12:48862) 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e8e00700 1 -- 172.21.15.12:0/3335975305 learned_addr learned my addr 172.21.15.12:0/3335975305 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e9800700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4e41611b0 0x7fe4e41e7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e3e00700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e40d21f0 0x7fe4e40ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e8e00700 1 -- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e40d21f0 msgr2=0x7fe4e40ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e8e00700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e40d21f0 0x7fe4e40ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e8e00700 1 -- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4e41611b0 msgr2=0x7fe4e41e7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e8e00700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4e41611b0 0x7fe4e41e7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e8e00700 1 -- 172.21.15.12:0/3335975305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d00094a0 con 0x7fe4e40cec40 2024-03-29T11:32:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.012+0000 7fe4e8e00700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 0x7fe4e4156380 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800c450 tx=0x7fe4d800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:28.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.013+0000 7fe4e0c00700 1 -- 172.21.15.12:0/3335975305 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d80052d0 con 0x7fe4e40cec40 2024-03-29T11:32:28.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.013+0000 7fe4e0c00700 1 -- 172.21.15.12:0/3335975305 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d8005430 con 0x7fe4e40cec40 2024-03-29T11:32:28.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.013+0000 7fe4e0c00700 1 -- 172.21.15.12:0/3335975305 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d80038c0 con 0x7fe4e40cec40 2024-03-29T11:32:28.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.013+0000 7fe4eb800700 1 -- 172.21.15.12:0/3335975305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4dc009b30 con 0x7fe4e40cec40 2024-03-29T11:32:28.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.013+0000 7fe4eb800700 1 -- 172.21.15.12:0/3335975305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e40c8070 con 0x7fe4e40cec40 2024-03-29T11:32:28.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.014+0000 7fe4eb800700 1 -- 172.21.15.12:0/3335975305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e4067720 con 0x7fe4e40cec40 2024-03-29T11:32:28.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.015+0000 7fe4e0c00700 1 -- 172.21.15.12:0/3335975305 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe4d8003a20 con 0x7fe4e40cec40 2024-03-29T11:32:28.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.016+0000 7fe4e0c00700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe4c406aed0 0x7fe4c406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:28.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.016+0000 7fe4e0c00700 1 -- 172.21.15.12:0/3335975305 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5832+0+0 (secure 0 0 0) 0x7fe4d8088b40 con 0x7fe4e40cec40 2024-03-29T11:32:28.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.016+0000 7fe4e3e00700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe4c406aed0 0x7fe4c406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:28.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.017+0000 7fe4e3e00700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe4c406aed0 0x7fe4c406d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe4e415d660 tx=0x7fe4d0008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:28.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.020+0000 7fe4e0c00700 1 -- 172.21.15.12:0/3335975305 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe4d8053970 con 0x7fe4e40cec40 2024-03-29T11:32:28.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.246+0000 7fe4eb800700 1 -- 172.21.15.12:0/3335975305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe4e40cd960 con 0x7fe4e40cec40 2024-03-29T11:32:28.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.975+0000 7fe4e0c00700 1 -- 172.21.15.12:0/3335975305 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v244) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe4d8056f90 con 0x7fe4e40cec40 2024-03-29T11:32:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 -- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe4c406aed0 msgr2=0x7fe4c406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe4c406aed0 0x7fe4c406d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe4e415d660 tx=0x7fe4d0008040 comp rx=0 tx=0).stop 2024-03-29T11:32:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 -- 172.21.15.12:0/3335975305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 msgr2=0x7fe4e4156380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 0x7fe4e4156380 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800c450 tx=0x7fe4d800c810 comp rx=0 tx=0).stop 2024-03-29T11:32:28.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 -- 172.21.15.12:0/3335975305 shutdown_connections 2024-03-29T11:32:28.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe4c406aed0 0x7fe4c406d390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4e41611b0 0x7fe4e41e7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4e40cec40 0x7fe4e4156380 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 --2- 172.21.15.12:0/3335975305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e40d21f0 0x7fe4e40ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 -- 172.21.15.12:0/3335975305 >> 172.21.15.12:0/3335975305 conn(0x7fe4e408aff0 msgr2=0x7fe4e406c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 -- 172.21.15.12:0/3335975305 shutdown_connections 2024-03-29T11:32:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:28.978+0000 7fe4eb800700 1 -- 172.21.15.12:0/3335975305 wait complete. 2024-03-29T11:32:28.979 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T11:32:28.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T11:32:29.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.203+0000 7eff97200700 1 Processor -- start 2024-03-29T11:32:29.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.203+0000 7eff97200700 1 -- start start 2024-03-29T11:32:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.204+0000 7eff97200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7eff900b3af0 0x7eff9015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.204+0000 7eff97200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff90072580 0x7eff900b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.204+0000 7eff97200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff900b2e50 0x7eff900709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.204+0000 7eff97200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff900b2180 con 0x7eff90072580 2024-03-29T11:32:29.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.204+0000 7eff97200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff9015b100 con 0x7eff900b3af0 2024-03-29T11:32:29.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.204+0000 7eff97200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9015b3e0 con 0x7eff900b2e50 2024-03-29T11:32:29.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.205+0000 7eff8f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff90072580 0x7eff900b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.205+0000 7eff8f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff90072580 0x7eff900b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34746/0 (socket says 172.21.15.12:34746) 2024-03-29T11:32:29.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.205+0000 7eff8f400700 1 -- 172.21.15.12:0/3556316772 learned_addr learned my addr 172.21.15.12:0/3556316772 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:29.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.205+0000 7eff8fe00700 1 --2- 172.21.15.12:0/3556316772 >> v2:172.21.15.159:3300/0 conn(0x7eff900b3af0 0x7eff9015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.205+0000 7eff95200700 1 --2- 172.21.15.12:0/3556316772 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff900b2e50 0x7eff900709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.205+0000 7eff8f400700 1 -- 172.21.15.12:0/3556316772 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff900b2e50 msgr2=0x7eff900709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.205+0000 7eff8f400700 1 --2- 172.21.15.12:0/3556316772 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff900b2e50 0x7eff900709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.205+0000 7eff8f400700 1 -- 172.21.15.12:0/3556316772 >> v2:172.21.15.159:3300/0 conn(0x7eff900b3af0 msgr2=0x7eff9015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.205+0000 7eff8f400700 1 --2- 172.21.15.12:0/3556316772 >> v2:172.21.15.159:3300/0 conn(0x7eff900b3af0 0x7eff9015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.205+0000 7eff8f400700 1 -- 172.21.15.12:0/3556316772 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff900b23f0 con 0x7eff90072580 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.206+0000 7eff8f400700 1 --2- 172.21.15.12:0/3556316772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff90072580 0x7eff900b2910 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7eff9006b860 tx=0x7eff7c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad94b59145e86e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.206+0000 7eff8ea00700 1 -- 172.21.15.12:0/3556316772 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c011830 con 0x7eff90072580 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.206+0000 7eff8ea00700 1 -- 172.21.15.12:0/3556316772 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7c011990 con 0x7eff90072580 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.206+0000 7eff8ea00700 1 -- 172.21.15.12:0/3556316772 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c011d20 con 0x7eff90072580 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.206+0000 7eff97200700 1 -- 172.21.15.12:0/3556316772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff90072580 msgr2=0x7eff900b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.206+0000 7eff97200700 1 --2- 172.21.15.12:0/3556316772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff90072580 0x7eff900b2910 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7eff9006b860 tx=0x7eff7c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.207+0000 7eff97200700 1 -- 172.21.15.12:0/3556316772 shutdown_connections 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.207+0000 7eff97200700 1 --2- 172.21.15.12:0/3556316772 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff900b2e50 0x7eff900709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.207+0000 7eff97200700 1 --2- 172.21.15.12:0/3556316772 >> v2:172.21.15.159:3300/0 conn(0x7eff900b3af0 0x7eff9015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.207+0000 7eff97200700 1 --2- 172.21.15.12:0/3556316772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff90072580 0x7eff900b2910 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.207+0000 7eff97200700 1 -- 172.21.15.12:0/3556316772 >> 172.21.15.12:0/3556316772 conn(0x7eff900cf190 msgr2=0x7eff900cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.207+0000 7eff97200700 1 -- 172.21.15.12:0/3556316772 shutdown_connections 2024-03-29T11:32:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.207+0000 7eff97200700 1 -- 172.21.15.12:0/3556316772 wait complete. 2024-03-29T11:32:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.208+0000 7eff97200700 1 Processor -- start 2024-03-29T11:32:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.209+0000 7eff97200700 1 -- start start 2024-03-29T11:32:29.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff97200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff90072580 0x7eff901495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff97200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff900b2e50 0x7eff9015c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff97200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff900b3af0 0x7eff9015caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff97200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff901500a0 con 0x7eff900b3af0 2024-03-29T11:32:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff97200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff90150310 con 0x7eff90072580 2024-03-29T11:32:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff97200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff901505f0 con 0x7eff900b2e50 2024-03-29T11:32:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff95200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff900b3af0 0x7eff9015caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff95200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff900b3af0 0x7eff9015caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34756/0 (socket says 172.21.15.12:34756) 2024-03-29T11:32:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff95200700 1 -- 172.21.15.12:0/2769862948 learned_addr learned my addr 172.21.15.12:0/2769862948 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff8f400700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff900b2e50 0x7eff9015c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff8fe00700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff90072580 0x7eff901495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff95200700 1 -- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff900b2e50 msgr2=0x7eff9015c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff95200700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff900b2e50 0x7eff9015c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff95200700 1 -- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff90072580 msgr2=0x7eff901495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff95200700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff90072580 0x7eff901495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.210+0000 7eff95200700 1 -- 172.21.15.12:0/2769862948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c009550 con 0x7eff900b3af0 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.211+0000 7eff95200700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff900b3af0 0x7eff9015caa0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7eff8800efc0 tx=0x7eff8800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.211+0000 7eff87e00700 1 -- 172.21.15.12:0/2769862948 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8800cd40 con 0x7eff900b3af0 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.211+0000 7eff87e00700 1 -- 172.21.15.12:0/2769862948 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8800f040 con 0x7eff900b3af0 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.211+0000 7eff97200700 1 -- 172.21.15.12:0/2769862948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff901508d0 con 0x7eff900b3af0 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.211+0000 7eff87e00700 1 -- 172.21.15.12:0/2769862948 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff880146c0 con 0x7eff900b3af0 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.211+0000 7eff97200700 1 -- 172.21.15.12:0/2769862948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff90149b10 con 0x7eff900b3af0 2024-03-29T11:32:29.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.212+0000 7eff97200700 1 -- 172.21.15.12:0/2769862948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff90150df0 con 0x7eff900b3af0 2024-03-29T11:32:29.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.213+0000 7eff87e00700 1 -- 172.21.15.12:0/2769862948 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7eff88014820 con 0x7eff900b3af0 2024-03-29T11:32:29.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.214+0000 7eff87e00700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff7006aed0 0x7eff7006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.214+0000 7eff87e00700 1 -- 172.21.15.12:0/2769862948 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 4929+0+0 (secure 0 0 0) 0x7eff88089db0 con 0x7eff900b3af0 2024-03-29T11:32:29.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.215+0000 7eff8fe00700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff7006aed0 0x7eff7006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.215+0000 7eff8fe00700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff7006aed0 0x7eff7006d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7eff80009a50 tx=0x7eff80008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.218+0000 7eff87e00700 1 -- 172.21.15.12:0/2769862948 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff90150df0 con 0x7eff900b3af0 2024-03-29T11:32:29.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.519+0000 7eff97200700 1 -- 172.21.15.12:0/2769862948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7eff90150df0 con 0x7eff900b3af0 2024-03-29T11:32:29.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.519+0000 7eff87e00700 1 -- 172.21.15.12:0/2769862948 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) v1 ==== 80+0+5304 (secure 0 0 0) 0x7eff90150df0 con 0x7eff900b3af0 2024-03-29T11:32:29.515 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:29.515 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:32:29.515 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T11:32:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T11:32:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T11:32:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:32:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:32:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:32:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T11:32:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T11:32:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:32:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:32:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T11:32:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:32:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T11:32:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:32:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:32:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:32:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T11:32:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T11:32:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:32:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:32:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:32:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:32:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:32:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T11:32:29.530 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:32:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 -- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff7006aed0 msgr2=0x7eff7006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff7006aed0 0x7eff7006d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7eff80009a50 tx=0x7eff80008040 comp rx=0 tx=0).stop 2024-03-29T11:32:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 -- 172.21.15.12:0/2769862948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff900b3af0 msgr2=0x7eff9015caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff900b3af0 0x7eff9015caa0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7eff8800efc0 tx=0x7eff8800c3a0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 -- 172.21.15.12:0/2769862948 shutdown_connections 2024-03-29T11:32:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff7006aed0 0x7eff7006d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff900b3af0 0x7eff9015caa0 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff90072580 0x7eff901495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 --2- 172.21.15.12:0/2769862948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff900b2e50 0x7eff9015c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 -- 172.21.15.12:0/2769862948 >> 172.21.15.12:0/2769862948 conn(0x7eff900cf190 msgr2=0x7eff900cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:29.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 -- 172.21.15.12:0/2769862948 shutdown_connections 2024-03-29T11:32:29.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.522+0000 7eff97200700 1 -- 172.21.15.12:0/2769862948 wait complete. 2024-03-29T11:32:29.535 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.738+0000 7f1360a00700 1 Processor -- start 2024-03-29T11:32:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.738+0000 7f1360a00700 1 -- start start 2024-03-29T11:32:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.739+0000 7f1360a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c160be0 0x7f135c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.739+0000 7f1360a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f135c0ce0e0 0x7f135c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.739+0000 7f1360a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135c160fc0 con 0x7f135c0cead0 2024-03-29T11:32:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.739+0000 7f1360a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135c0cef60 con 0x7f135c0ce0e0 2024-03-29T11:32:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.739+0000 7f1360a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0b65d0 con 0x7f135c160be0 2024-03-29T11:32:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.740+0000 7f1359e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c160be0 0x7f135c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.740+0000 7f1359400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f135c0ce0e0 0x7f135c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.740+0000 7f1359e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c160be0 0x7f135c0cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:35224/0 (socket says 172.21.15.12:35224) 2024-03-29T11:32:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.740+0000 7f1359e00700 1 -- 172.21.15.12:0/2977862582 learned_addr learned my addr 172.21.15.12:0/2977862582 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.740+0000 7f1359e00700 1 -- 172.21.15.12:0/2977862582 >> v2:172.21.15.159:3300/0 conn(0x7f135c0ce0e0 msgr2=0x7f135c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.740+0000 7f1359e00700 1 --2- 172.21.15.12:0/2977862582 >> v2:172.21.15.159:3300/0 conn(0x7f135c0ce0e0 0x7f135c0ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.740+0000 7f1359e00700 1 -- 172.21.15.12:0/2977862582 >> v1:172.21.15.159:6789/0 conn(0x7f135c0cead0 legacy=0x7f135c161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.740+0000 7f1359e00700 1 -- 172.21.15.12:0/2977862582 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c0d21e0 con 0x7f135c160be0 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.740+0000 7f1359400700 1 --2- 172.21.15.12:0/2977862582 >> v2:172.21.15.159:3300/0 conn(0x7f135c0ce0e0 0x7f135c0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.741+0000 7f1359e00700 1 --2- 172.21.15.12:0/2977862582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c160be0 0x7f135c0cdba0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f134c0098f0 tx=0x7f134c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71e5f672999791ac server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.741+0000 7f1358a00700 1 -- 172.21.15.12:0/2977862582 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c011830 con 0x7f135c160be0 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.741+0000 7f1358a00700 1 -- 172.21.15.12:0/2977862582 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c011990 con 0x7f135c160be0 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.741+0000 7f1358a00700 1 -- 172.21.15.12:0/2977862582 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c011d20 con 0x7f135c160be0 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.741+0000 7f1360a00700 1 -- 172.21.15.12:0/2977862582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c160be0 msgr2=0x7f135c0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.741+0000 7f1360a00700 1 --2- 172.21.15.12:0/2977862582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c160be0 0x7f135c0cdba0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f134c0098f0 tx=0x7f134c009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.742+0000 7f1360a00700 1 -- 172.21.15.12:0/2977862582 shutdown_connections 2024-03-29T11:32:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.742+0000 7f1360a00700 1 --2- 172.21.15.12:0/2977862582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c160be0 0x7f135c0cdba0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.742+0000 7f1360a00700 1 --2- 172.21.15.12:0/2977862582 >> v2:172.21.15.159:3300/0 conn(0x7f135c0ce0e0 0x7f135c0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.742+0000 7f1360a00700 1 -- 172.21.15.12:0/2977862582 >> 172.21.15.12:0/2977862582 conn(0x7f135c08aff0 msgr2=0x7f135c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.742+0000 7f1360a00700 1 -- 172.21.15.12:0/2977862582 shutdown_connections 2024-03-29T11:32:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.742+0000 7f1360a00700 1 -- 172.21.15.12:0/2977862582 wait complete. 2024-03-29T11:32:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.743+0000 7f1360a00700 1 Processor -- start 2024-03-29T11:32:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.744+0000 7f1360a00700 1 -- start start 2024-03-29T11:32:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.744+0000 7f1360a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135c0ce0e0 0x7f135c0c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.744+0000 7f1360a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c0cead0 0x7f135c09e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.744+0000 7f1360a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f135c160be0 0x7f135c15d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.744+0000 7f1360a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c1f31c0 con 0x7f135c160be0 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.744+0000 7f1360a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c1f3430 con 0x7f135c0ce0e0 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.744+0000 7f1360a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c1f3710 con 0x7f135c0cead0 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f135a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f135c160be0 0x7f135c15d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f1359e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135c0ce0e0 0x7f135c0c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f1359400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c0cead0 0x7f135c09e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f135a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f135c160be0 0x7f135c15d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34772/0 (socket says 172.21.15.12:34772) 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f1359e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135c0ce0e0 0x7f135c0c8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:49460/0 (socket says 172.21.15.12:49460) 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f1359e00700 1 -- 172.21.15.12:0/2986212861 learned_addr learned my addr 172.21.15.12:0/2986212861 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f1359e00700 1 -- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c0cead0 msgr2=0x7f135c09e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f1359e00700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c0cead0 0x7f135c09e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f1359e00700 1 -- 172.21.15.12:0/2986212861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f135c160be0 msgr2=0x7f135c15d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f1359e00700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f135c160be0 0x7f135c15d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f1359e00700 1 -- 172.21.15.12:0/2986212861 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13480094a0 con 0x7f135c0ce0e0 2024-03-29T11:32:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.745+0000 7f135a800700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f135c160be0 0x7f135c15d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.746+0000 7f1359e00700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135c0ce0e0 0x7f135c0c8c30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f134c009c40 tx=0x7f134c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.746+0000 7f1352000700 1 -- 172.21.15.12:0/2986212861 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c004510 con 0x7f135c0ce0e0 2024-03-29T11:32:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.746+0000 7f1352000700 1 -- 172.21.15.12:0/2986212861 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c01e070 con 0x7f135c0ce0e0 2024-03-29T11:32:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.746+0000 7f1352000700 1 -- 172.21.15.12:0/2986212861 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c0038e0 con 0x7f135c0ce0e0 2024-03-29T11:32:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.746+0000 7f1360a00700 1 -- 172.21.15.12:0/2986212861 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c009550 con 0x7f135c0ce0e0 2024-03-29T11:32:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.746+0000 7f1360a00700 1 -- 172.21.15.12:0/2986212861 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c1f3cd0 con 0x7f135c0ce0e0 2024-03-29T11:32:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.747+0000 7f1360a00700 1 -- 172.21.15.12:0/2986212861 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c162120 con 0x7f135c0ce0e0 2024-03-29T11:32:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.748+0000 7f1352000700 1 -- 172.21.15.12:0/2986212861 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f134c003a40 con 0x7f135c0ce0e0 2024-03-29T11:32:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.749+0000 7f1352000700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f133c06aed0 0x7f133c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.749+0000 7f1352000700 1 -- 172.21.15.12:0/2986212861 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f134c08a580 con 0x7f135c0ce0e0 2024-03-29T11:32:29.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.750+0000 7f1359400700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f133c06aed0 0x7f133c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:29.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.750+0000 7f1359400700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f133c06aed0 0x7f133c06d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f135c0cd7b0 tx=0x7f1348009210 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:29.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.753+0000 7f1352000700 1 -- 172.21.15.12:0/2986212861 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f134c0557c0 con 0x7f135c0ce0e0 2024-03-29T11:32:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.970+0000 7f1360a00700 1 -- 172.21.15.12:0/2986212861 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f135c1f3570 con 0x7f135c0ce0e0 2024-03-29T11:32:29.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.971+0000 7f1352000700 1 -- 172.21.15.12:0/2986212861 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v244) v1 ==== 74+0+12515 (secure 0 0 0) 0x7f135c1f3570 con 0x7f135c0ce0e0 2024-03-29T11:32:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:29.966 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":244,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:28.970885+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 -- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f133c06aed0 msgr2=0x7f133c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f133c06aed0 0x7f133c06d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f135c0cd7b0 tx=0x7f1348009210 comp rx=0 tx=0).stop 2024-03-29T11:32:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 -- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135c0ce0e0 msgr2=0x7f135c0c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135c0ce0e0 0x7f135c0c8c30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f134c009c40 tx=0x7f134c004750 comp rx=0 tx=0).stop 2024-03-29T11:32:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 -- 172.21.15.12:0/2986212861 shutdown_connections 2024-03-29T11:32:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f133c06aed0 0x7f133c06d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f135c160be0 0x7f135c15d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135c0ce0e0 0x7f135c0c8c30 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 --2- 172.21.15.12:0/2986212861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f135c0cead0 0x7f135c09e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:29.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 -- 172.21.15.12:0/2986212861 >> 172.21.15.12:0/2986212861 conn(0x7f135c08aff0 msgr2=0x7f135c0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:29.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 -- 172.21.15.12:0/2986212861 shutdown_connections 2024-03-29T11:32:29.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:29.973+0000 7f1360a00700 1 -- 172.21.15.12:0/2986212861 wait complete. 2024-03-29T11:32:29.987 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:32:29.987 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:32:30.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.193+0000 7fecda200700 1 Processor -- start 2024-03-29T11:32:30.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.193+0000 7fecda200700 1 -- start start 2024-03-29T11:32:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecda200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecd40d16a0 0x7fecd40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecda200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 0x7fecd40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecda200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd4161420 con 0x7fecd415d9d0 2024-03-29T11:32:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecda200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd40d2820 con 0x7fecd40d16a0 2024-03-29T11:32:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecda200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd40d4340 con 0x7fecd40d2370 2024-03-29T11:32:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecd2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecd40d16a0 0x7fecd40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecd3400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fecd415d9d0 0x7fecd40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35782/0 (socket says 172.21.15.12:35782) 2024-03-29T11:32:30.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecd3400700 1 -- 172.21.15.12:0/2741260845 learned_addr learned my addr 172.21.15.12:0/2741260845 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecd2a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecd40d16a0 0x7fecd40d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34780/0 (socket says 172.21.15.12:34780) 2024-03-29T11:32:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecd3e00700 1 --2- 172.21.15.12:0/2741260845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 0x7fecd40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecd2a00700 1 -- 172.21.15.12:0/2741260845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 msgr2=0x7fecd40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecd2a00700 1 --2- 172.21.15.12:0/2741260845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 0x7fecd40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecd2a00700 1 -- 172.21.15.12:0/2741260845 >> v1:172.21.15.12:6789/0 conn(0x7fecd415d9d0 legacy=0x7fecd40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecd2a00700 1 -- 172.21.15.12:0/2741260845 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd4161700 con 0x7fecd40d16a0 2024-03-29T11:32:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.195+0000 7fecd2000700 1 -- 172.21.15.12:0/2741260845 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101203843 0 0) 0x7fecd4161420 con 0x7fecd415d9d0 2024-03-29T11:32:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.196+0000 7fecd2a00700 1 --2- 172.21.15.12:0/2741260845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 0x7fecd40d1e30 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009820 tx=0x7fecc0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ab62fc4f332cdc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.196+0000 7fecd2000700 1 -- 172.21.15.12:0/2741260845 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc0005e30 con 0x7fecd40d16a0 2024-03-29T11:32:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.196+0000 7fecd2000700 1 -- 172.21.15.12:0/2741260845 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc0006030 con 0x7fecd40d16a0 2024-03-29T11:32:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.196+0000 7fecd2000700 1 -- 172.21.15.12:0/2741260845 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc00063c0 con 0x7fecd40d16a0 2024-03-29T11:32:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.196+0000 7fecda200700 1 -- 172.21.15.12:0/2741260845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 msgr2=0x7fecd40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.196+0000 7fecda200700 1 --2- 172.21.15.12:0/2741260845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 0x7fecd40d1e30 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009820 tx=0x7fecc0009b30 comp rx=0 tx=0).stop 2024-03-29T11:32:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.196+0000 7fecda200700 1 -- 172.21.15.12:0/2741260845 shutdown_connections 2024-03-29T11:32:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.196+0000 7fecda200700 1 --2- 172.21.15.12:0/2741260845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 0x7fecd40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.196+0000 7fecda200700 1 --2- 172.21.15.12:0/2741260845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 0x7fecd40d1e30 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.196+0000 7fecda200700 1 -- 172.21.15.12:0/2741260845 >> 172.21.15.12:0/2741260845 conn(0x7fecd408aff0 msgr2=0x7fecd40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.197+0000 7fecda200700 1 -- 172.21.15.12:0/2741260845 shutdown_connections 2024-03-29T11:32:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.197+0000 7fecda200700 1 -- 172.21.15.12:0/2741260845 wait complete. 2024-03-29T11:32:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.198+0000 7fecda200700 1 Processor -- start 2024-03-29T11:32:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.198+0000 7fecda200700 1 -- start start 2024-03-29T11:32:30.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.199+0000 7fecda200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 0x7fecd41559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.199+0000 7fecda200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 0x7fecd40b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.199+0000 7fecda200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecd415d9d0 0x7fecd40b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.199+0000 7fecda200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd40b8690 con 0x7fecd40d16a0 2024-03-29T11:32:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.199+0000 7fecda200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd40b8b20 con 0x7fecd415d9d0 2024-03-29T11:32:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.199+0000 7fecda200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd40b7a80 con 0x7fecd40d2370 2024-03-29T11:32:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 0x7fecd41559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 0x7fecd41559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34786/0 (socket says 172.21.15.12:34786) 2024-03-29T11:32:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3400700 1 -- 172.21.15.12:0/1541070318 learned_addr learned my addr 172.21.15.12:0/1541070318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd2a00700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 0x7fecd40b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3e00700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecd415d9d0 0x7fecd40b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3400700 1 -- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 msgr2=0x7fecd40b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3400700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 0x7fecd40b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3400700 1 -- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecd415d9d0 msgr2=0x7fecd40b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3400700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecd415d9d0 0x7fecd40b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3400700 1 -- 172.21.15.12:0/1541070318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc0009430 con 0x7fecd40d16a0 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd2a00700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 0x7fecd40b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3e00700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecd415d9d0 0x7fecd40b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.200+0000 7fecd3400700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 0x7fecd41559e0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fecc400c550 tx=0x7fecc400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.201+0000 7feccb400700 1 -- 172.21.15.12:0/1541070318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4004c80 con 0x7fecd40d16a0 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.201+0000 7feccb400700 1 -- 172.21.15.12:0/1541070318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc4004e00 con 0x7fecd40d16a0 2024-03-29T11:32:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.201+0000 7feccb400700 1 -- 172.21.15.12:0/1541070318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4007310 con 0x7fecd40d16a0 2024-03-29T11:32:30.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.201+0000 7fecda200700 1 -- 172.21.15.12:0/1541070318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd40b7d60 con 0x7fecd40d16a0 2024-03-29T11:32:30.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.201+0000 7fecda200700 1 -- 172.21.15.12:0/1541070318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd40b8320 con 0x7fecd40d16a0 2024-03-29T11:32:30.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.202+0000 7fecda200700 1 -- 172.21.15.12:0/1541070318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd40d1ca0 con 0x7fecd40d16a0 2024-03-29T11:32:30.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.203+0000 7feccb400700 1 -- 172.21.15.12:0/1541070318 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fecc4007470 con 0x7fecd40d16a0 2024-03-29T11:32:30.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.204+0000 7feccb400700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fecb406aed0 0x7fecb406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.204+0000 7feccb400700 1 -- 172.21.15.12:0/1541070318 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fecc4089180 con 0x7fecd40d16a0 2024-03-29T11:32:30.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.204+0000 7fecd2a00700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fecb406aed0 0x7fecb406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.205+0000 7fecd2a00700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fecb406aed0 0x7fecb406d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fecd40d1ae0 tx=0x7fecc0008070 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:30.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.208+0000 7feccb400700 1 -- 172.21.15.12:0/1541070318 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fecc4054300 con 0x7fecd40d16a0 2024-03-29T11:32:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.421+0000 7fecda200700 1 -- 172.21.15.12:0/1541070318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fecd4067720 con 0x7fecd40d16a0 2024-03-29T11:32:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.422+0000 7feccb400700 1 -- 172.21.15.12:0/1541070318 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v244) v1 ==== 93+0+10 (secure 0 0 0) 0x7fecc4057920 con 0x7fecd40d16a0 2024-03-29T11:32:30.417 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:32:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.424+0000 7fecda200700 1 -- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fecb406aed0 msgr2=0x7fecb406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.424+0000 7fecda200700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fecb406aed0 0x7fecb406d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fecd40d1ae0 tx=0x7fecc0008070 comp rx=0 tx=0).stop 2024-03-29T11:32:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.424+0000 7fecda200700 1 -- 172.21.15.12:0/1541070318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 msgr2=0x7fecd41559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.424+0000 7fecda200700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 0x7fecd41559e0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fecc400c550 tx=0x7fecc400c910 comp rx=0 tx=0).stop 2024-03-29T11:32:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.424+0000 7fecda200700 1 -- 172.21.15.12:0/1541070318 shutdown_connections 2024-03-29T11:32:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.424+0000 7fecda200700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fecb406aed0 0x7fecb406d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.424+0000 7fecda200700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fecd415d9d0 0x7fecd40b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.424+0000 7fecda200700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecd40d16a0 0x7fecd41559e0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.424+0000 7fecda200700 1 --2- 172.21.15.12:0/1541070318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fecd40d2370 0x7fecd40b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.424+0000 7fecda200700 1 -- 172.21.15.12:0/1541070318 >> 172.21.15.12:0/1541070318 conn(0x7fecd408aff0 msgr2=0x7fecd40b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.425+0000 7fecda200700 1 -- 172.21.15.12:0/1541070318 shutdown_connections 2024-03-29T11:32:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.425+0000 7fecda200700 1 -- 172.21.15.12:0/1541070318 wait complete. 2024-03-29T11:32:30.438 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T11:32:30.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.641+0000 7f9514200700 1 Processor -- start 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.642+0000 7f9514200700 1 -- start start 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.643+0000 7f9514200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 0x7f950c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.643+0000 7f9514200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f950c161800 0x7f950c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.643+0000 7f9514200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c0d30d0 con 0x7f950c15d640 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.643+0000 7f9514200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950c161c90 con 0x7f950c161800 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.643+0000 7f9514200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0b6840 con 0x7f950c0ceec0 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.643+0000 7f9512200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f950c161800 0x7f950c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.643+0000 7f9512200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f950c161800 0x7f950c0b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34792/0 (socket says 172.21.15.12:34792) 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.643+0000 7f9512200700 1 -- 172.21.15.12:0/1256256485 learned_addr learned my addr 172.21.15.12:0/1256256485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.643+0000 7f9510e00700 1 --2- 172.21.15.12:0/1256256485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 0x7f950c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f9512200700 1 -- 172.21.15.12:0/1256256485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 msgr2=0x7f950c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f9512200700 1 --2- 172.21.15.12:0/1256256485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 0x7f950c1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f9512200700 1 -- 172.21.15.12:0/1256256485 >> v1:172.21.15.159:6789/0 conn(0x7f950c15d640 legacy=0x7f950c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f9512200700 1 -- 172.21.15.12:0/1256256485 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c0d2b30 con 0x7f950c161800 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f9510e00700 1 --2- 172.21.15.12:0/1256256485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 0x7f950c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f9512200700 1 --2- 172.21.15.12:0/1256256485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c161800 0x7f950c0b6cd0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f9504009e50 tx=0x7f950400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92d4ceb1577bb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f950be00700 1 -- 172.21.15.12:0/1256256485 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950400e070 con 0x7f950c161800 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f950be00700 1 -- 172.21.15.12:0/1256256485 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950400b9c0 con 0x7f950c161800 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f950be00700 1 -- 172.21.15.12:0/1256256485 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950400bd70 con 0x7f950c161800 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f9514200700 1 -- 172.21.15.12:0/1256256485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c161800 msgr2=0x7f950c0b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.644+0000 7f9514200700 1 --2- 172.21.15.12:0/1256256485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c161800 0x7f950c0b6cd0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f9504009e50 tx=0x7f950400b330 comp rx=0 tx=0).stop 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.645+0000 7f9514200700 1 -- 172.21.15.12:0/1256256485 shutdown_connections 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.645+0000 7f9514200700 1 --2- 172.21.15.12:0/1256256485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c161800 0x7f950c0b6cd0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.645+0000 7f9514200700 1 --2- 172.21.15.12:0/1256256485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 0x7f950c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.645+0000 7f9514200700 1 -- 172.21.15.12:0/1256256485 >> 172.21.15.12:0/1256256485 conn(0x7f950c08aff0 msgr2=0x7f950c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:30.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.645+0000 7f9514200700 1 -- 172.21.15.12:0/1256256485 shutdown_connections 2024-03-29T11:32:30.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.645+0000 7f9514200700 1 -- 172.21.15.12:0/1256256485 wait complete. 2024-03-29T11:32:30.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.646+0000 7f9514200700 1 Processor -- start 2024-03-29T11:32:30.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.647+0000 7f9514200700 1 -- start start 2024-03-29T11:32:30.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.647+0000 7f9514200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 0x7f950c15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.647+0000 7f9514200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c15d640 0x7f950c0ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.647+0000 7f9514200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f950c161800 0x7f950c0cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9514200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c15f670 con 0x7f950c15d640 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9514200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0ce7e0 con 0x7f950c161800 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9514200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0cad20 con 0x7f950c0ceec0 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9510e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c15d640 0x7f950c0ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9510e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c15d640 0x7f950c0ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34800/0 (socket says 172.21.15.12:34800) 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9510e00700 1 -- 172.21.15.12:0/3429132560 learned_addr learned my addr 172.21.15.12:0/3429132560 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9512200700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f950c161800 0x7f950c0cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9511800700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 0x7f950c15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9510e00700 1 -- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 msgr2=0x7f950c15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9510e00700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 0x7f950c15ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9510e00700 1 -- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f950c161800 msgr2=0x7f950c0cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9510e00700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f950c161800 0x7f950c0cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.648+0000 7f9510e00700 1 -- 172.21.15.12:0/3429132560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9504009b30 con 0x7f950c15d640 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.649+0000 7f9510e00700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c15d640 0x7f950c0ca7e0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f94f800e760 tx=0x7f94f800ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.649+0000 7f9509600700 1 -- 172.21.15.12:0/3429132560 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8013610 con 0x7f950c15d640 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.649+0000 7f9509600700 1 -- 172.21.15.12:0/3429132560 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f8013770 con 0x7f950c15d640 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.649+0000 7f9514200700 1 -- 172.21.15.12:0/3429132560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c0cde40 con 0x7f950c15d640 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.649+0000 7f9509600700 1 -- 172.21.15.12:0/3429132560 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8013b20 con 0x7f950c15d640 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.649+0000 7f9514200700 1 -- 172.21.15.12:0/3429132560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c0ce400 con 0x7f950c15d640 2024-03-29T11:32:30.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.650+0000 7f9514200700 1 -- 172.21.15.12:0/3429132560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c0cea50 con 0x7f950c15d640 2024-03-29T11:32:30.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.651+0000 7f9509600700 1 -- 172.21.15.12:0/3429132560 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f94f8013c80 con 0x7f950c15d640 2024-03-29T11:32:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.652+0000 7f9509600700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f94ec06aed0 0x7f94ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.652+0000 7f9509600700 1 -- 172.21.15.12:0/3429132560 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f94f8089110 con 0x7f950c15d640 2024-03-29T11:32:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.652+0000 7f9511800700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f94ec06aed0 0x7f94ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.653+0000 7f9511800700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f94ec06aed0 0x7f94ec06d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9500000f60 tx=0x7f9500009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.656+0000 7f9509600700 1 -- 172.21.15.12:0/3429132560 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f94f8054170 con 0x7f950c15d640 2024-03-29T11:32:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.872+0000 7f9514200700 1 -- 172.21.15.12:0/3429132560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f950c067720 con 0x7f950c15d640 2024-03-29T11:32:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.872+0000 7f9509600700 1 -- 172.21.15.12:0/3429132560 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v244) v1 ==== 84+0+65 (secure 0 0 0) 0x7f94f8057790 con 0x7f950c15d640 2024-03-29T11:32:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:30.868 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:32:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T11:32:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T11:32:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:30.869 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:32:30.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.874+0000 7f9514200700 1 -- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f94ec06aed0 msgr2=0x7f94ec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.874+0000 7f9514200700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f94ec06aed0 0x7f94ec06d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9500000f60 tx=0x7f9500009040 comp rx=0 tx=0).stop 2024-03-29T11:32:30.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.874+0000 7f9514200700 1 -- 172.21.15.12:0/3429132560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c15d640 msgr2=0x7f950c0ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.874+0000 7f9514200700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c15d640 0x7f950c0ca7e0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f94f800e760 tx=0x7f94f800ea70 comp rx=0 tx=0).stop 2024-03-29T11:32:30.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.874+0000 7f9514200700 1 -- 172.21.15.12:0/3429132560 shutdown_connections 2024-03-29T11:32:30.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.874+0000 7f9514200700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f94ec06aed0 0x7f94ec06d390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.874+0000 7f9514200700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f950c161800 0x7f950c0cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.874+0000 7f9514200700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f950c0ceec0 0x7f950c15ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.874+0000 7f9514200700 1 --2- 172.21.15.12:0/3429132560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f950c15d640 0x7f950c0ca7e0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:30.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.874+0000 7f9514200700 1 -- 172.21.15.12:0/3429132560 >> 172.21.15.12:0/3429132560 conn(0x7f950c08aff0 msgr2=0x7f950c06cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:30.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.875+0000 7f9514200700 1 -- 172.21.15.12:0/3429132560 shutdown_connections 2024-03-29T11:32:30.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:30.875+0000 7f9514200700 1 -- 172.21.15.12:0/3429132560 wait complete. 2024-03-29T11:32:30.889 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T11:32:30.889 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T11:32:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.095+0000 7f660ea00700 1 Processor -- start 2024-03-29T11:32:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.096+0000 7f660ea00700 1 -- start start 2024-03-29T11:32:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f660815d740 0x7f66080d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66080d1fe0 0x7f66080b0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66080ced30 con 0x7f660815db20 2024-03-29T11:32:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ea00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66080cef80 con 0x7f66080d1fe0 2024-03-29T11:32:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ea00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66080cf260 con 0x7f660815d740 2024-03-29T11:32:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66080d1fe0 0x7f66080b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f66080d1fe0 0x7f66080b0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34806/0 (socket says 172.21.15.12:34806) 2024-03-29T11:32:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ca00700 1 -- 172.21.15.12:0/3172003403 learned_addr learned my addr 172.21.15.12:0/3172003403 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f6607e00700 1 --2- 172.21.15.12:0/3172003403 >> v2:172.21.15.159:3300/0 conn(0x7f660815d740 0x7f66080d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ca00700 1 -- 172.21.15.12:0/3172003403 >> v2:172.21.15.159:3300/0 conn(0x7f660815d740 msgr2=0x7f66080d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ca00700 1 --2- 172.21.15.12:0/3172003403 >> v2:172.21.15.159:3300/0 conn(0x7f660815d740 0x7f66080d1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ca00700 1 -- 172.21.15.12:0/3172003403 >> v1:172.21.15.12:6789/0 conn(0x7f660815db20 legacy=0x7f66080d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.097+0000 7f660ca00700 1 -- 172.21.15.12:0/3172003403 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66080cf540 con 0x7f66080d1fe0 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.098+0000 7f660ca00700 1 --2- 172.21.15.12:0/3172003403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 0x7f66080b0320 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f660808bf00 tx=0x7f660000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d3c7ad5668f094 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.098+0000 7f6606a00700 1 -- 172.21.15.12:0/3172003403 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660000e070 con 0x7f66080d1fe0 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.098+0000 7f6606a00700 1 -- 172.21.15.12:0/3172003403 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660000b9c0 con 0x7f66080d1fe0 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.098+0000 7f6606a00700 1 -- 172.21.15.12:0/3172003403 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660000bd70 con 0x7f66080d1fe0 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.099+0000 7f660ea00700 1 -- 172.21.15.12:0/3172003403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 msgr2=0x7f66080b0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.099+0000 7f660ea00700 1 --2- 172.21.15.12:0/3172003403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 0x7f66080b0320 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f660808bf00 tx=0x7f660000b330 comp rx=0 tx=0).stop 2024-03-29T11:32:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.099+0000 7f660ea00700 1 -- 172.21.15.12:0/3172003403 shutdown_connections 2024-03-29T11:32:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.099+0000 7f660ea00700 1 --2- 172.21.15.12:0/3172003403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 0x7f66080b0320 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.099+0000 7f660ea00700 1 --2- 172.21.15.12:0/3172003403 >> v2:172.21.15.159:3300/0 conn(0x7f660815d740 0x7f66080d1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.099+0000 7f660ea00700 1 -- 172.21.15.12:0/3172003403 >> 172.21.15.12:0/3172003403 conn(0x7f660808aff0 msgr2=0x7f66080d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.099+0000 7f660ea00700 1 -- 172.21.15.12:0/3172003403 shutdown_connections 2024-03-29T11:32:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.099+0000 7f660ea00700 1 -- 172.21.15.12:0/3172003403 wait complete. 2024-03-29T11:32:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.100+0000 7f660ea00700 1 Processor -- start 2024-03-29T11:32:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.101+0000 7f660ea00700 1 -- start start 2024-03-29T11:32:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.101+0000 7f660ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 0x7f660814b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.101+0000 7f660ea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f660815d740 0x7f660809d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.101+0000 7f660ea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660815db20 0x7f660809ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.101+0000 7f660ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660809dc10 con 0x7f66080d1fe0 2024-03-29T11:32:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.101+0000 7f660ea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660809def0 con 0x7f660815d740 2024-03-29T11:32:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.101+0000 7f660ea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f660809e1d0 con 0x7f660815db20 2024-03-29T11:32:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f6607e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 0x7f660814b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f6607e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 0x7f660814b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34816/0 (socket says 172.21.15.12:34816) 2024-03-29T11:32:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f6607e00700 1 -- 172.21.15.12:0/2277923657 learned_addr learned my addr 172.21.15.12:0/2277923657 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f660ca00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660815db20 0x7f660809ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f6607400700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f660815d740 0x7f660809d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f6607e00700 1 -- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660815db20 msgr2=0x7f660809ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f6607e00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660815db20 0x7f660809ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f6607e00700 1 -- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f660815d740 msgr2=0x7f660809d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f6607e00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f660815d740 0x7f660809d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f6607e00700 1 -- 172.21.15.12:0/2277923657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6600009b30 con 0x7f66080d1fe0 2024-03-29T11:32:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.102+0000 7f6607e00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 0x7f660814b950 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f65f800e810 tx=0x7f65f800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.103+0000 7f65ffe00700 1 -- 172.21.15.12:0/2277923657 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f8013610 con 0x7f66080d1fe0 2024-03-29T11:32:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.103+0000 7f65ffe00700 1 -- 172.21.15.12:0/2277923657 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f8013770 con 0x7f66080d1fe0 2024-03-29T11:32:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.103+0000 7f65ffe00700 1 -- 172.21.15.12:0/2277923657 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f8013b20 con 0x7f66080d1fe0 2024-03-29T11:32:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.103+0000 7f660ea00700 1 -- 172.21.15.12:0/2277923657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f660809e4b0 con 0x7f66080d1fe0 2024-03-29T11:32:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.103+0000 7f660ea00700 1 -- 172.21.15.12:0/2277923657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f660814be90 con 0x7f66080d1fe0 2024-03-29T11:32:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.104+0000 7f660ea00700 1 -- 172.21.15.12:0/2277923657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6608067720 con 0x7f66080d1fe0 2024-03-29T11:32:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.105+0000 7f65ffe00700 1 -- 172.21.15.12:0/2277923657 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f65f8013c80 con 0x7f66080d1fe0 2024-03-29T11:32:31.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.106+0000 7f65ffe00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f65e806aed0 0x7f65e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.106+0000 7f65ffe00700 1 -- 172.21.15.12:0/2277923657 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f65f8089240 con 0x7f66080d1fe0 2024-03-29T11:32:31.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.106+0000 7f6607400700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f65e806aed0 0x7f65e806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.107+0000 7f6607400700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f65e806aed0 0x7f65e806d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f660806d660 tx=0x7f65f4009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:31.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.110+0000 7f65ffe00700 1 -- 172.21.15.12:0/2277923657 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65f80542c0 con 0x7f66080d1fe0 2024-03-29T11:32:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.321+0000 7f660ea00700 1 -- 172.21.15.12:0/2277923657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f66080b5d00 con 0x7f66080d1fe0 2024-03-29T11:32:31.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.396+0000 7f65ffe00700 1 -- 172.21.15.12:0/2277923657 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v245) v1 ==== 138+0+0 (secure 0 0 0) 0x7f65f80578e0 con 0x7f66080d1fe0 2024-03-29T11:32:31.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.398+0000 7f660ea00700 1 -- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f65e806aed0 msgr2=0x7f65e806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.398+0000 7f660ea00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f65e806aed0 0x7f65e806d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f660806d660 tx=0x7f65f4009040 comp rx=0 tx=0).stop 2024-03-29T11:32:31.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.398+0000 7f660ea00700 1 -- 172.21.15.12:0/2277923657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 msgr2=0x7f660814b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.398+0000 7f660ea00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 0x7f660814b950 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f65f800e810 tx=0x7f65f800ebd0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.398+0000 7f660ea00700 1 -- 172.21.15.12:0/2277923657 shutdown_connections 2024-03-29T11:32:31.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.398+0000 7f660ea00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f65e806aed0 0x7f65e806d390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.398+0000 7f660ea00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f660815db20 0x7f660809ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.398+0000 7f660ea00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66080d1fe0 0x7f660814b950 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.398+0000 7f660ea00700 1 --2- 172.21.15.12:0/2277923657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f660815d740 0x7f660809d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.398+0000 7f660ea00700 1 -- 172.21.15.12:0/2277923657 >> 172.21.15.12:0/2277923657 conn(0x7f660808aff0 msgr2=0x7f660806b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:31.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.399+0000 7f660ea00700 1 -- 172.21.15.12:0/2277923657 shutdown_connections 2024-03-29T11:32:31.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.399+0000 7f660ea00700 1 -- 172.21.15.12:0/2277923657 wait complete. 2024-03-29T11:32:31.395 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T11:32:31.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T11:32:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.626+0000 7f03e7c00700 1 Processor -- start 2024-03-29T11:32:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.627+0000 7f03e7c00700 1 -- start start 2024-03-29T11:32:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03e7c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f03e00cf8c0 0x7f03e00b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03e7c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e00ced20 con 0x7f03e00d3eb0 2024-03-29T11:32:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03e7c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e00d4340 con 0x7f03e015d8a0 2024-03-29T11:32:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03e7c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e00ce6d0 con 0x7f03e00cf8c0 2024-03-29T11:32:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03dfe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f03e00d3eb0 0x7f03e00cf1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35794/0 (socket says 172.21.15.12:35794) 2024-03-29T11:32:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03dfe00700 1 -- 172.21.15.12:0/3050019519 learned_addr learned my addr 172.21.15.12:0/3050019519 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03e5c00700 1 --2- 172.21.15.12:0/3050019519 >> v2:172.21.15.12:3300/0 conn(0x7f03e00cf8c0 0x7f03e00b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03df400700 1 -- 172.21.15.12:0/3050019519 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173869768 0 0) 0x7f03e00ced20 con 0x7f03e00d3eb0 2024-03-29T11:32:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03df400700 1 -- 172.21.15.12:0/3050019519 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c80035b0 con 0x7f03e00d3eb0 2024-03-29T11:32:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03e5c00700 1 -- 172.21.15.12:0/3050019519 >> v1:172.21.15.159:6789/0 conn(0x7f03e015d8a0 legacy=0x7f03e006b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.628+0000 7f03e5c00700 1 -- 172.21.15.12:0/3050019519 >> v1:172.21.15.12:6789/0 conn(0x7f03e00d3eb0 legacy=0x7f03e00cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.629+0000 7f03e5c00700 1 -- 172.21.15.12:0/3050019519 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e00ce9b0 con 0x7f03e00cf8c0 2024-03-29T11:32:31.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.629+0000 7f03e5c00700 1 --2- 172.21.15.12:0/3050019519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e00cf8c0 0x7f03e00b2990 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009d90 tx=0x7f03d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f176eff4292f0259 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.629+0000 7f03df400700 1 -- 172.21.15.12:0/3050019519 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d800e070 con 0x7f03e00cf8c0 2024-03-29T11:32:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.629+0000 7f03df400700 1 -- 172.21.15.12:0/3050019519 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d800bae0 con 0x7f03e00cf8c0 2024-03-29T11:32:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.629+0000 7f03df400700 1 -- 172.21.15.12:0/3050019519 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d800be90 con 0x7f03e00cf8c0 2024-03-29T11:32:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.629+0000 7f03e7c00700 1 -- 172.21.15.12:0/3050019519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e00cf8c0 msgr2=0x7f03e00b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.629+0000 7f03e7c00700 1 --2- 172.21.15.12:0/3050019519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e00cf8c0 0x7f03e00b2990 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009d90 tx=0x7f03d800b330 comp rx=0 tx=0).stop 2024-03-29T11:32:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.630+0000 7f03e7c00700 1 -- 172.21.15.12:0/3050019519 shutdown_connections 2024-03-29T11:32:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.630+0000 7f03e7c00700 1 --2- 172.21.15.12:0/3050019519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e00cf8c0 0x7f03e00b2990 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.630+0000 7f03e7c00700 1 -- 172.21.15.12:0/3050019519 >> 172.21.15.12:0/3050019519 conn(0x7f03e008aff0 msgr2=0x7f03e00d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.630+0000 7f03e7c00700 1 -- 172.21.15.12:0/3050019519 shutdown_connections 2024-03-29T11:32:31.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.630+0000 7f03e7c00700 1 -- 172.21.15.12:0/3050019519 wait complete. 2024-03-29T11:32:31.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.631+0000 7f03e7c00700 1 Processor -- start 2024-03-29T11:32:31.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.632+0000 7f03e7c00700 1 -- start start 2024-03-29T11:32:31.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.632+0000 7f03e7c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e00cf8c0 0x7f03e00ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.632+0000 7f03e7c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e015d8a0 0x7f03e00c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.632+0000 7f03e7c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e0159680 0x7f03e00c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.632+0000 7f03e7c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e00c70c0 con 0x7f03e015d8a0 2024-03-29T11:32:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.632+0000 7f03e7c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e00cd330 con 0x7f03e00cf8c0 2024-03-29T11:32:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.632+0000 7f03e7c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e00c7310 con 0x7f03e0159680 2024-03-29T11:32:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.633+0000 7f03dfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e015d8a0 0x7f03e00c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.633+0000 7f03dfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e015d8a0 0x7f03e00c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34846/0 (socket says 172.21.15.12:34846) 2024-03-29T11:32:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.633+0000 7f03dfe00700 1 -- 172.21.15.12:0/45216203 learned_addr learned my addr 172.21.15.12:0/45216203 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.633+0000 7f03e5200700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e00cf8c0 0x7f03e00ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.633+0000 7f03e5c00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e0159680 0x7f03e00c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.633+0000 7f03dfe00700 1 -- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e0159680 msgr2=0x7f03e00c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.633+0000 7f03dfe00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e0159680 0x7f03e00c8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.633+0000 7f03dfe00700 1 -- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e00cf8c0 msgr2=0x7f03e00ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.633+0000 7f03dfe00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e00cf8c0 0x7f03e00ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:31.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.633+0000 7f03dfe00700 1 -- 172.21.15.12:0/45216203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d8009a70 con 0x7f03e015d8a0 2024-03-29T11:32:31.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.634+0000 7f03dfe00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e015d8a0 0x7f03e00c7df0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00c700 tx=0x7f03cc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:31.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.634+0000 7f03dcc00700 1 -- 172.21.15.12:0/45216203 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0074e0 con 0x7f03e015d8a0 2024-03-29T11:32:31.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.634+0000 7f03dcc00700 1 -- 172.21.15.12:0/45216203 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc007660 con 0x7f03e015d8a0 2024-03-29T11:32:31.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.634+0000 7f03dcc00700 1 -- 172.21.15.12:0/45216203 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc005a70 con 0x7f03e015d8a0 2024-03-29T11:32:31.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.634+0000 7f03e7c00700 1 -- 172.21.15.12:0/45216203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e00c75f0 con 0x7f03e015d8a0 2024-03-29T11:32:31.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.634+0000 7f03e7c00700 1 -- 172.21.15.12:0/45216203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e00ccf20 con 0x7f03e015d8a0 2024-03-29T11:32:31.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.635+0000 7f03e7c00700 1 -- 172.21.15.12:0/45216203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e00c7b10 con 0x7f03e015d8a0 2024-03-29T11:32:31.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.637+0000 7f03dcc00700 1 -- 172.21.15.12:0/45216203 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f03cc005bd0 con 0x7f03e015d8a0 2024-03-29T11:32:31.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.637+0000 7f03dcc00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f03c006aed0 0x7f03c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:31.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.637+0000 7f03dcc00700 1 -- 172.21.15.12:0/45216203 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 5299+0+0 (secure 0 0 0) 0x7f03cc088bc0 con 0x7f03e015d8a0 2024-03-29T11:32:31.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.638+0000 7f03e5200700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f03c006aed0 0x7f03c006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:31.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.638+0000 7f03e5200700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f03c006aed0 0x7f03c006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009960 tx=0x7f03d40090e0 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:31.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.642+0000 7f03dcc00700 1 -- 172.21.15.12:0/45216203 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f03cc053cb0 con 0x7f03e015d8a0 2024-03-29T11:32:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:31.854+0000 7f03e7c00700 1 -- 172.21.15.12:0/45216203 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f03e00c7b10 con 0x7f03e015d8a0 2024-03-29T11:32:32.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.404+0000 7f03dcc00700 1 -- 172.21.15.12:0/45216203 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v246) v1 ==== 170+0+0 (secure 0 0 0) 0x7f03e00c7b10 con 0x7f03e015d8a0 2024-03-29T11:32:32.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.410+0000 7f03e7c00700 1 -- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f03c006aed0 msgr2=0x7f03c006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:32.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.410+0000 7f03e7c00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f03c006aed0 0x7f03c006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009960 tx=0x7f03d40090e0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.410+0000 7f03e7c00700 1 -- 172.21.15.12:0/45216203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e015d8a0 msgr2=0x7f03e00c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:32.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.410+0000 7f03e7c00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e015d8a0 0x7f03e00c7df0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00c700 tx=0x7f03cc00ca10 comp rx=0 tx=0).stop 2024-03-29T11:32:32.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.411+0000 7f03e7c00700 1 -- 172.21.15.12:0/45216203 shutdown_connections 2024-03-29T11:32:32.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.411+0000 7f03e7c00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f03c006aed0 0x7f03c006d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.411+0000 7f03e7c00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e0159680 0x7f03e00c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.411+0000 7f03e7c00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e00cf8c0 0x7f03e00ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.411+0000 7f03e7c00700 1 --2- 172.21.15.12:0/45216203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e015d8a0 0x7f03e00c7df0 secure :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00c700 tx=0x7f03cc00ca10 comp rx=0 tx=0).stop 2024-03-29T11:32:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.411+0000 7f03e7c00700 1 -- 172.21.15.12:0/45216203 >> 172.21.15.12:0/45216203 conn(0x7f03e008aff0 msgr2=0x7f03e00b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.412+0000 7f03e7c00700 1 -- 172.21.15.12:0/45216203 shutdown_connections 2024-03-29T11:32:32.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.413+0000 7f03e7c00700 1 -- 172.21.15.12:0/45216203 wait complete. 2024-03-29T11:32:32.414 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T11:32:32.438 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T11:32:32.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.657+0000 7fb527400700 1 Processor -- start 2024-03-29T11:32:32.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.657+0000 7fb527400700 1 -- start start 2024-03-29T11:32:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.659+0000 7fb527400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb5200d2540 0x7fb5201687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.659+0000 7fb527400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb520161420 con 0x7fb52015d9d0 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.659+0000 7fb527400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb520161700 con 0x7fb5200d16a0 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.659+0000 7fb527400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5200d4340 con 0x7fb5200d2540 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.659+0000 7fb524a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb52015d9d0 0x7fb5200d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35804/0 (socket says 172.21.15.12:35804) 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.659+0000 7fb524a00700 1 -- 172.21.15.12:0/297968790 learned_addr learned my addr 172.21.15.12:0/297968790 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.660+0000 7fb525400700 1 --2- 172.21.15.12:0/297968790 >> v2:172.21.15.159:3300/0 conn(0x7fb5200d2540 0x7fb5201687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.660+0000 7fb51f400700 1 -- 172.21.15.12:0/297968790 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833479607 0 0) 0x7fb520161420 con 0x7fb52015d9d0 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.660+0000 7fb51f400700 1 -- 172.21.15.12:0/297968790 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5080035b0 con 0x7fb52015d9d0 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.660+0000 7fb51f400700 1 -- 172.21.15.12:0/297968790 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347025014 0 0) 0x7fb520161700 con 0x7fb5200d16a0 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.660+0000 7fb51f400700 1 -- 172.21.15.12:0/297968790 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb520161420 con 0x7fb5200d16a0 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.660+0000 7fb525400700 1 -- 172.21.15.12:0/297968790 >> v1:172.21.15.159:6789/0 conn(0x7fb5200d16a0 legacy=0x7fb5200d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.660+0000 7fb525400700 1 -- 172.21.15.12:0/297968790 >> v1:172.21.15.12:6789/0 conn(0x7fb52015d9d0 legacy=0x7fb5200d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.660+0000 7fb525400700 1 -- 172.21.15.12:0/297968790 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5201619e0 con 0x7fb5200d2540 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.661+0000 7fb525400700 1 --2- 172.21.15.12:0/297968790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb5200d2540 0x7fb5201687b0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb518009d90 tx=0x7fb51800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=336d49a4ca4770bf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.661+0000 7fb51f400700 1 -- 172.21.15.12:0/297968790 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51800e070 con 0x7fb5200d2540 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.661+0000 7fb51f400700 1 -- 172.21.15.12:0/297968790 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51800bb20 con 0x7fb5200d2540 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.661+0000 7fb51f400700 1 -- 172.21.15.12:0/297968790 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb518012430 con 0x7fb5200d2540 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.661+0000 7fb527400700 1 -- 172.21.15.12:0/297968790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb5200d2540 msgr2=0x7fb5201687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.661+0000 7fb527400700 1 --2- 172.21.15.12:0/297968790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb5200d2540 0x7fb5201687b0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb518009d90 tx=0x7fb51800b330 comp rx=0 tx=0).stop 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.661+0000 7fb527400700 1 -- 172.21.15.12:0/297968790 shutdown_connections 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.661+0000 7fb527400700 1 --2- 172.21.15.12:0/297968790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb5200d2540 0x7fb5201687b0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.661+0000 7fb527400700 1 -- 172.21.15.12:0/297968790 >> 172.21.15.12:0/297968790 conn(0x7fb52008aff0 msgr2=0x7fb5200d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.662+0000 7fb527400700 1 -- 172.21.15.12:0/297968790 shutdown_connections 2024-03-29T11:32:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.662+0000 7fb527400700 1 -- 172.21.15.12:0/297968790 wait complete. 2024-03-29T11:32:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.663+0000 7fb527400700 1 Processor -- start 2024-03-29T11:32:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.663+0000 7fb527400700 1 -- start start 2024-03-29T11:32:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.664+0000 7fb527400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb5200d16a0 0x7fb52015c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.664+0000 7fb527400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5200d2540 0x7fb5201e9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.664+0000 7fb527400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb52015d9d0 0x7fb5200b7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.664+0000 7fb527400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5201e7c90 con 0x7fb5200d2540 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.664+0000 7fb527400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52015cf50 con 0x7fb5200d16a0 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.664+0000 7fb527400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5201e7ea0 con 0x7fb52015d9d0 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb51fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5200d2540 0x7fb5201e9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb51fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5200d2540 0x7fb5201e9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34854/0 (socket says 172.21.15.12:34854) 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb51fe00700 1 -- 172.21.15.12:0/2229908940 learned_addr learned my addr 172.21.15.12:0/2229908940 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb524a00700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb5200d16a0 0x7fb52015c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb525400700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb52015d9d0 0x7fb5200b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb51fe00700 1 -- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb52015d9d0 msgr2=0x7fb5200b7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb51fe00700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb52015d9d0 0x7fb5200b7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb51fe00700 1 -- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb5200d16a0 msgr2=0x7fb52015c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb51fe00700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb5200d16a0 0x7fb52015c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb51fe00700 1 -- 172.21.15.12:0/2229908940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb518009a70 con 0x7fb5200d2540 2024-03-29T11:32:32.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.665+0000 7fb51fe00700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5200d2540 0x7fb5201e9160 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00c4f0 tx=0x7fb50c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:32.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.666+0000 7fb51cc00700 1 -- 172.21.15.12:0/2229908940 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c0052d0 con 0x7fb5200d2540 2024-03-29T11:32:32.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.666+0000 7fb51cc00700 1 -- 172.21.15.12:0/2229908940 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50c005430 con 0x7fb5200d2540 2024-03-29T11:32:32.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.666+0000 7fb527400700 1 -- 172.21.15.12:0/2229908940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5201e8180 con 0x7fb5200d2540 2024-03-29T11:32:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.666+0000 7fb51cc00700 1 -- 172.21.15.12:0/2229908940 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c0038f0 con 0x7fb5200d2540 2024-03-29T11:32:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.666+0000 7fb527400700 1 -- 172.21.15.12:0/2229908940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5201e8740 con 0x7fb5200d2540 2024-03-29T11:32:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.667+0000 7fb527400700 1 -- 172.21.15.12:0/2229908940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5200d1ca0 con 0x7fb5200d2540 2024-03-29T11:32:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.668+0000 7fb51cc00700 1 -- 172.21.15.12:0/2229908940 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb50c003a50 con 0x7fb5200d2540 2024-03-29T11:32:32.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.669+0000 7fb51cc00700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb50006aed0 0x7fb50006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:32.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.669+0000 7fb51cc00700 1 -- 172.21.15.12:0/2229908940 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5681+0+0 (secure 0 0 0) 0x7fb50c088f90 con 0x7fb5200d2540 2024-03-29T11:32:32.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.669+0000 7fb524a00700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb50006aed0 0x7fb50006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:32.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.670+0000 7fb524a00700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb50006aed0 0x7fb50006d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb514007fd0 tx=0x7fb514007dc0 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.673+0000 7fb51cc00700 1 -- 172.21.15.12:0/2229908940 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb50c053ed0 con 0x7fb5200d2540 2024-03-29T11:32:32.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.932+0000 7fb527400700 1 -- 172.21.15.12:0/2229908940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb520161660 con 0x7fb5200d2540 2024-03-29T11:32:32.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.942+0000 7fb51cc00700 1 -- 172.21.15.12:0/2229908940 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 30 and data pool 31 v118) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb50c0574f0 con 0x7fb5200d2540 2024-03-29T11:32:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.950+0000 7fb527400700 1 -- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb50006aed0 msgr2=0x7fb50006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.950+0000 7fb527400700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb50006aed0 0x7fb50006d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb514007fd0 tx=0x7fb514007dc0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.950+0000 7fb527400700 1 -- 172.21.15.12:0/2229908940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5200d2540 msgr2=0x7fb5201e9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.950+0000 7fb527400700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5200d2540 0x7fb5201e9160 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00c4f0 tx=0x7fb50c00c800 comp rx=0 tx=0).stop 2024-03-29T11:32:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.950+0000 7fb527400700 1 -- 172.21.15.12:0/2229908940 shutdown_connections 2024-03-29T11:32:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.950+0000 7fb527400700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb50006aed0 0x7fb50006d390 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb514007fd0 tx=0x7fb514007dc0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.950+0000 7fb527400700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb52015d9d0 0x7fb5200b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.950+0000 7fb527400700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb5200d16a0 0x7fb52015c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:32.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.950+0000 7fb527400700 1 --2- 172.21.15.12:0/2229908940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5200d2540 0x7fb5201e9160 secure :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00c4f0 tx=0x7fb50c00c800 comp rx=0 tx=0).stop 2024-03-29T11:32:32.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.950+0000 7fb527400700 1 -- 172.21.15.12:0/2229908940 >> 172.21.15.12:0/2229908940 conn(0x7fb52008aff0 msgr2=0x7fb5200ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:32.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.953+0000 7fb527400700 1 -- 172.21.15.12:0/2229908940 shutdown_connections 2024-03-29T11:32:32.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:32.955+0000 7fb527400700 1 -- 172.21.15.12:0/2229908940 wait complete. 2024-03-29T11:32:32.956 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 30 and data pool 31 2024-03-29T11:32:32.983 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:33.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.192+0000 7fdb01c00700 1 Processor -- start 2024-03-29T11:32:33.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.192+0000 7fdb01c00700 1 -- start start 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.193+0000 7fdb01c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdafc072580 0x7fdafc0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.193+0000 7fdb01c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdafc0b2e50 0x7fdafc0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.193+0000 7fdb01c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc0b2180 con 0x7fdafc0b3af0 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.193+0000 7fdb01c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdafc15b100 con 0x7fdafc072580 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.193+0000 7fdb01c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc15b3e0 con 0x7fdafc0b2e50 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.194+0000 7fdafa000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdafc072580 0x7fdafc0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.194+0000 7fdafa000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdafc072580 0x7fdafc0b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34860/0 (socket says 172.21.15.12:34860) 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.194+0000 7fdafa000700 1 -- 172.21.15.12:0/3077038749 learned_addr learned my addr 172.21.15.12:0/3077038749 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.194+0000 7fdafb400700 1 --2- 172.21.15.12:0/3077038749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdafc0b2e50 0x7fdafc0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.194+0000 7fdafa000700 1 -- 172.21.15.12:0/3077038749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdafc0b2e50 msgr2=0x7fdafc0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.194+0000 7fdafa000700 1 --2- 172.21.15.12:0/3077038749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdafc0b2e50 0x7fdafc0709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.194+0000 7fdafa000700 1 -- 172.21.15.12:0/3077038749 >> v1:172.21.15.159:6789/0 conn(0x7fdafc0b3af0 legacy=0x7fdafc15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.194+0000 7fdafa000700 1 -- 172.21.15.12:0/3077038749 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc0b23f0 con 0x7fdafc072580 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.194+0000 7fdafa000700 1 --2- 172.21.15.12:0/3077038749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 0x7fdafc0b2910 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fdae800bb90 tx=0x7fdae800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d334ba4d0916e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.194+0000 7fdaf9600700 1 -- 172.21.15.12:0/3077038749 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae800dd50 con 0x7fdafc072580 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.195+0000 7fdaf9600700 1 -- 172.21.15.12:0/3077038749 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae8010470 con 0x7fdafc072580 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.195+0000 7fdaf9600700 1 -- 172.21.15.12:0/3077038749 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8010800 con 0x7fdafc072580 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.195+0000 7fdb01c00700 1 -- 172.21.15.12:0/3077038749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 msgr2=0x7fdafc0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.195+0000 7fdb01c00700 1 --2- 172.21.15.12:0/3077038749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 0x7fdafc0b2910 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fdae800bb90 tx=0x7fdae800bea0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.195+0000 7fdb01c00700 1 -- 172.21.15.12:0/3077038749 shutdown_connections 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.195+0000 7fdb01c00700 1 --2- 172.21.15.12:0/3077038749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdafc0b2e50 0x7fdafc0709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.195+0000 7fdb01c00700 1 --2- 172.21.15.12:0/3077038749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 0x7fdafc0b2910 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.195+0000 7fdb01c00700 1 -- 172.21.15.12:0/3077038749 >> 172.21.15.12:0/3077038749 conn(0x7fdafc0cf190 msgr2=0x7fdafc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.195+0000 7fdb01c00700 1 -- 172.21.15.12:0/3077038749 shutdown_connections 2024-03-29T11:32:33.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.195+0000 7fdb01c00700 1 -- 172.21.15.12:0/3077038749 wait complete. 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.197+0000 7fdb01c00700 1 Processor -- start 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.197+0000 7fdb01c00700 1 -- start start 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdb01c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 0x7fdafc15a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdb01c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdafc0b2e50 0x7fdafc157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdb01c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdafc0b3af0 0x7fdafc15cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdb01c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc15a710 con 0x7fdafc072580 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdb01c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc14dbd0 con 0x7fdafc0b2e50 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdb01c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc14cfc0 con 0x7fdafc0b3af0 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdafaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 0x7fdafc15a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdafaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 0x7fdafc15a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34868/0 (socket says 172.21.15.12:34868) 2024-03-29T11:32:33.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdafaa00700 1 -- 172.21.15.12:0/692189151 learned_addr learned my addr 172.21.15.12:0/692189151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdafa000700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdafc0b2e50 0x7fdafc157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.198+0000 7fdafb400700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdafc0b3af0 0x7fdafc15cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdafaa00700 1 -- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdafc0b3af0 msgr2=0x7fdafc15cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdafaa00700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdafc0b3af0 0x7fdafc15cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdafaa00700 1 -- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdafc0b2e50 msgr2=0x7fdafc157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdafaa00700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdafc0b2e50 0x7fdafc157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdafaa00700 1 -- 172.21.15.12:0/692189151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae800b7f0 con 0x7fdafc072580 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdafaa00700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 0x7fdafc15a1d0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00c450 tx=0x7fdaec00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdaf2a00700 1 -- 172.21.15.12:0/692189151 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec0052d0 con 0x7fdafc072580 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdaf2a00700 1 -- 172.21.15.12:0/692189151 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec005430 con 0x7fdafc072580 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdaf2a00700 1 -- 172.21.15.12:0/692189151 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec0038c0 con 0x7fdafc072580 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdb01c00700 1 -- 172.21.15.12:0/692189151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdafc14d2a0 con 0x7fdafc072580 2024-03-29T11:32:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.199+0000 7fdb01c00700 1 -- 172.21.15.12:0/692189151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdafc14d860 con 0x7fdafc072580 2024-03-29T11:32:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.200+0000 7fdb01c00700 1 -- 172.21.15.12:0/692189151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdafc071ee0 con 0x7fdafc072580 2024-03-29T11:32:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.201+0000 7fdaf2a00700 1 -- 172.21.15.12:0/692189151 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fdaec003a20 con 0x7fdafc072580 2024-03-29T11:32:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.202+0000 7fdaf2a00700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdadc06aed0 0x7fdadc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.202+0000 7fdaf2a00700 1 -- 172.21.15.12:0/692189151 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fdaec088980 con 0x7fdafc072580 2024-03-29T11:32:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.203+0000 7fdafa000700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdadc06aed0 0x7fdadc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.203+0000 7fdafa000700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdadc06aed0 0x7fdadc06d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdae800bb90 tx=0x7fdae8013000 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:33.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.207+0000 7fdaf2a00700 1 -- 172.21.15.12:0/692189151 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdaec053880 con 0x7fdafc072580 2024-03-29T11:32:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.423+0000 7fdb01c00700 1 -- 172.21.15.12:0/692189151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdafc0ae240 con 0x7fdafc072580 2024-03-29T11:32:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.424+0000 7fdaf2a00700 1 -- 172.21.15.12:0/692189151 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) v1 ==== 74+0+16180 (secure 0 0 0) 0x7fdaec017090 con 0x7fdafc072580 2024-03-29T11:32:33.430 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:33.431 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":247,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:32.933414+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.426+0000 7fdb01c00700 1 -- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdadc06aed0 msgr2=0x7fdadc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.426+0000 7fdb01c00700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdadc06aed0 0x7fdadc06d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdae800bb90 tx=0x7fdae8013000 comp rx=0 tx=0).stop 2024-03-29T11:32:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.427+0000 7fdb01c00700 1 -- 172.21.15.12:0/692189151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 msgr2=0x7fdafc15a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.427+0000 7fdb01c00700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 0x7fdafc15a1d0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00c450 tx=0x7fdaec00c760 comp rx=0 tx=0).stop 2024-03-29T11:32:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.427+0000 7fdb01c00700 1 -- 172.21.15.12:0/692189151 shutdown_connections 2024-03-29T11:32:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.427+0000 7fdb01c00700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdadc06aed0 0x7fdadc06d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.427+0000 7fdb01c00700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdafc0b3af0 0x7fdafc15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.427+0000 7fdb01c00700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdafc072580 0x7fdafc15a1d0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.427+0000 7fdb01c00700 1 --2- 172.21.15.12:0/692189151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdafc0b2e50 0x7fdafc157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.427+0000 7fdb01c00700 1 -- 172.21.15.12:0/692189151 >> 172.21.15.12:0/692189151 conn(0x7fdafc0cf190 msgr2=0x7fdafc0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.427+0000 7fdb01c00700 1 -- 172.21.15.12:0/692189151 shutdown_connections 2024-03-29T11:32:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.427+0000 7fdb01c00700 1 -- 172.21.15.12:0/692189151 wait complete. 2024-03-29T11:32:33.442 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:33.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.650+0000 7f6c4d600700 1 Processor -- start 2024-03-29T11:32:33.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.650+0000 7f6c4d600700 1 -- start start 2024-03-29T11:32:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.652+0000 7f6c4d600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c481619d0 0x7f6c480d3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.652+0000 7f6c4d600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6c480ce9c0 0x7f6c4815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.652+0000 7f6c4d600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c480d2af0 con 0x7f6c481615f0 2024-03-29T11:32:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.652+0000 7f6c4d600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c480cdc00 con 0x7f6c480ce9c0 2024-03-29T11:32:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.652+0000 7f6c4d600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480cdee0 con 0x7f6c481619d0 2024-03-29T11:32:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.652+0000 7f6c47400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6c480ce9c0 0x7f6c4815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.652+0000 7f6c46000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c481619d0 0x7f6c480d3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.652+0000 7f6c47400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6c480ce9c0 0x7f6c4815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:49554/0 (socket says 172.21.15.12:49554) 2024-03-29T11:32:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.652+0000 7f6c47400700 1 -- 172.21.15.12:0/896351312 learned_addr learned my addr 172.21.15.12:0/896351312 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.652+0000 7f6c46a00700 1 --1- 172.21.15.12:0/896351312 >> v1:172.21.15.159:6789/0 conn(0x7f6c481615f0 0x7f6c480d2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:49440/0 (socket says 172.21.15.12:49440) 2024-03-29T11:32:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.653+0000 7f6c46000700 1 -- 172.21.15.12:0/896351312 >> v2:172.21.15.159:3300/0 conn(0x7f6c480ce9c0 msgr2=0x7f6c4815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.653+0000 7f6c46000700 1 --2- 172.21.15.12:0/896351312 >> v2:172.21.15.159:3300/0 conn(0x7f6c480ce9c0 0x7f6c4815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.653+0000 7f6c46000700 1 -- 172.21.15.12:0/896351312 >> v1:172.21.15.159:6789/0 conn(0x7f6c481615f0 legacy=0x7f6c480d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.653+0000 7f6c46000700 1 -- 172.21.15.12:0/896351312 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c480ce1c0 con 0x7f6c481619d0 2024-03-29T11:32:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.653+0000 7f6c47400700 1 --2- 172.21.15.12:0/896351312 >> v2:172.21.15.159:3300/0 conn(0x7f6c480ce9c0 0x7f6c4815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.653+0000 7f6c45600700 1 -- 172.21.15.12:0/896351312 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935648014 0 0) 0x7f6c480d2af0 con 0x7f6c481615f0 2024-03-29T11:32:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.653+0000 7f6c46000700 1 --2- 172.21.15.12:0/896351312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c481619d0 0x7f6c480d3fb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009840 tx=0x7f6c34009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c0a2b508f25fb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.653+0000 7f6c45600700 1 -- 172.21.15.12:0/896351312 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c34011830 con 0x7f6c481619d0 2024-03-29T11:32:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.653+0000 7f6c45600700 1 -- 172.21.15.12:0/896351312 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c34011990 con 0x7f6c481619d0 2024-03-29T11:32:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.653+0000 7f6c45600700 1 -- 172.21.15.12:0/896351312 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c34011d20 con 0x7f6c481619d0 2024-03-29T11:32:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.654+0000 7f6c4d600700 1 -- 172.21.15.12:0/896351312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c481619d0 msgr2=0x7f6c480d3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.654+0000 7f6c4d600700 1 --2- 172.21.15.12:0/896351312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c481619d0 0x7f6c480d3fb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009840 tx=0x7f6c34009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.654+0000 7f6c4d600700 1 -- 172.21.15.12:0/896351312 shutdown_connections 2024-03-29T11:32:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.654+0000 7f6c4d600700 1 --2- 172.21.15.12:0/896351312 >> v2:172.21.15.159:3300/0 conn(0x7f6c480ce9c0 0x7f6c4815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.654+0000 7f6c4d600700 1 --2- 172.21.15.12:0/896351312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c481619d0 0x7f6c480d3fb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.654+0000 7f6c4d600700 1 -- 172.21.15.12:0/896351312 >> 172.21.15.12:0/896351312 conn(0x7f6c4808aff0 msgr2=0x7f6c480d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.654+0000 7f6c4d600700 1 -- 172.21.15.12:0/896351312 shutdown_connections 2024-03-29T11:32:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.654+0000 7f6c4d600700 1 -- 172.21.15.12:0/896351312 wait complete. 2024-03-29T11:32:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.656+0000 7f6c4d600700 1 Processor -- start 2024-03-29T11:32:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.656+0000 7f6c4d600700 1 -- start start 2024-03-29T11:32:33.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c4d600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c480ce9c0 0x7f6c4815b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c4d600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c481615f0 0x7f6c48151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c4d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c481619d0 0x7f6c480caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c4d600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480b3330 con 0x7f6c481619d0 2024-03-29T11:32:33.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c4d600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480b3610 con 0x7f6c481615f0 2024-03-29T11:32:33.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c4d600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480b38f0 con 0x7f6c480ce9c0 2024-03-29T11:32:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c47400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c481619d0 0x7f6c480caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c47400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c481619d0 0x7f6c480caaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34882/0 (socket says 172.21.15.12:34882) 2024-03-29T11:32:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c47400700 1 -- 172.21.15.12:0/1969974779 learned_addr learned my addr 172.21.15.12:0/1969974779 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c46a00700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c480ce9c0 0x7f6c4815b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.657+0000 7f6c46000700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c481615f0 0x7f6c48151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c47400700 1 -- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c480ce9c0 msgr2=0x7f6c4815b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c47400700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c480ce9c0 0x7f6c4815b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c47400700 1 -- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c481615f0 msgr2=0x7f6c48151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c47400700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c481615f0 0x7f6c48151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c47400700 1 -- 172.21.15.12:0/1969974779 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c40009b30 con 0x7f6c481619d0 2024-03-29T11:32:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c47400700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c481619d0 0x7f6c480caaa0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000efc0 tx=0x7f6c4000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c3ea00700 1 -- 172.21.15.12:0/1969974779 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40018070 con 0x7f6c481619d0 2024-03-29T11:32:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c3ea00700 1 -- 172.21.15.12:0/1969974779 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4000f040 con 0x7f6c481619d0 2024-03-29T11:32:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c3ea00700 1 -- 172.21.15.12:0/1969974779 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40003c20 con 0x7f6c481619d0 2024-03-29T11:32:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c4d600700 1 -- 172.21.15.12:0/1969974779 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c340094a0 con 0x7f6c481619d0 2024-03-29T11:32:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.658+0000 7f6c4d600700 1 -- 172.21.15.12:0/1969974779 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c480b3eb0 con 0x7f6c481619d0 2024-03-29T11:32:33.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.659+0000 7f6c4d600700 1 -- 172.21.15.12:0/1969974779 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c480cf750 con 0x7f6c481619d0 2024-03-29T11:32:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.660+0000 7f6c3ea00700 1 -- 172.21.15.12:0/1969974779 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6c4000c880 con 0x7f6c481619d0 2024-03-29T11:32:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.661+0000 7f6c3ea00700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c2806aed0 0x7f6c2806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:33.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.661+0000 7f6c3ea00700 1 -- 172.21.15.12:0/1969974779 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f6c40089a50 con 0x7f6c481619d0 2024-03-29T11:32:33.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.662+0000 7f6c46a00700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c2806aed0 0x7f6c2806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:33.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.663+0000 7f6c46a00700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c2806aed0 0x7f6c2806d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c38007fd0 tx=0x7f6c38007de0 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:33.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.666+0000 7f6c3ea00700 1 -- 172.21.15.12:0/1969974779 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c40054760 con 0x7f6c481619d0 2024-03-29T11:32:33.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.880+0000 7f6c4d600700 1 -- 172.21.15.12:0/1969974779 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c48067720 con 0x7f6c481619d0 2024-03-29T11:32:33.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.881+0000 7f6c3ea00700 1 -- 172.21.15.12:0/1969974779 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) v1 ==== 74+0+16180 (secure 0 0 0) 0x7f6c4005e030 con 0x7f6c481619d0 2024-03-29T11:32:33.878 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:33.878 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":247,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:32.933414+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 -- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c2806aed0 msgr2=0x7f6c2806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c2806aed0 0x7f6c2806d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c38007fd0 tx=0x7f6c38007de0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 -- 172.21.15.12:0/1969974779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c481619d0 msgr2=0x7f6c480caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c481619d0 0x7f6c480caaa0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000efc0 tx=0x7f6c4000c3a0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 -- 172.21.15.12:0/1969974779 shutdown_connections 2024-03-29T11:32:33.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c2806aed0 0x7f6c2806d390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c481619d0 0x7f6c480caaa0 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c480ce9c0 0x7f6c4815b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 --2- 172.21.15.12:0/1969974779 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c481615f0 0x7f6c48151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 -- 172.21.15.12:0/1969974779 >> 172.21.15.12:0/1969974779 conn(0x7f6c4808aff0 msgr2=0x7f6c480d2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 -- 172.21.15.12:0/1969974779 shutdown_connections 2024-03-29T11:32:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:33.884+0000 7f6c4d600700 1 -- 172.21.15.12:0/1969974779 wait complete. 2024-03-29T11:32:33.899 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T11:32:34.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.107+0000 7fdce2000700 1 Processor -- start 2024-03-29T11:32:34.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.107+0000 7fdce2000700 1 -- start start 2024-03-29T11:32:34.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.108+0000 7fdce2000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcdc15a5c0 0x7fdcdc0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:34.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.108+0000 7fdce2000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc0cb310 con 0x7fdcdc0ce840 2024-03-29T11:32:34.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.108+0000 7fdce2000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc070f00 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.108+0000 7fdce2000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcdc0711a0 con 0x7fdcdc15a5c0 2024-03-29T11:32:34.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.108+0000 7fdcdb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcdc15a5c0 0x7fdcdc0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:34.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.108+0000 7fdcdbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdcdc0ce840 0x7fdcdc0cba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35812/0 (socket says 172.21.15.12:35812) 2024-03-29T11:32:34.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.108+0000 7fdcdbe00700 1 -- 172.21.15.12:0/2047746747 learned_addr learned my addr 172.21.15.12:0/2047746747 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:34.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.109+0000 7fdcdb400700 1 -- 172.21.15.12:0/2047746747 >> v1:172.21.15.159:6789/0 conn(0x7fdcdc15dfe0 legacy=0x7fdcdc0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:34.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.109+0000 7fdcdb400700 1 -- 172.21.15.12:0/2047746747 >> v1:172.21.15.12:6789/0 conn(0x7fdcdc0ce840 legacy=0x7fdcdc0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:34.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.109+0000 7fdcdb400700 1 -- 172.21.15.12:0/2047746747 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc071480 con 0x7fdcdc15a5c0 2024-03-29T11:32:34.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.109+0000 7fdcda000700 1 -- 172.21.15.12:0/2047746747 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384544236 0 0) 0x7fdcdc0cb310 con 0x7fdcdc0ce840 2024-03-29T11:32:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.109+0000 7fdcdb400700 1 --2- 172.21.15.12:0/2047746747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15a5c0 0x7fdcdc0709c0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009840 tx=0x7fdccc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385bfe43ddb508c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.109+0000 7fdcda000700 1 -- 172.21.15.12:0/2047746747 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc011830 con 0x7fdcdc15a5c0 2024-03-29T11:32:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.109+0000 7fdcda000700 1 -- 172.21.15.12:0/2047746747 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc011990 con 0x7fdcdc15a5c0 2024-03-29T11:32:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.109+0000 7fdcda000700 1 -- 172.21.15.12:0/2047746747 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc011d20 con 0x7fdcdc15a5c0 2024-03-29T11:32:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.110+0000 7fdce2000700 1 -- 172.21.15.12:0/2047746747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15a5c0 msgr2=0x7fdcdc0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.110+0000 7fdce2000700 1 --2- 172.21.15.12:0/2047746747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15a5c0 0x7fdcdc0709c0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009840 tx=0x7fdccc009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.110+0000 7fdce2000700 1 -- 172.21.15.12:0/2047746747 shutdown_connections 2024-03-29T11:32:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.110+0000 7fdce2000700 1 --2- 172.21.15.12:0/2047746747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15a5c0 0x7fdcdc0709c0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.110+0000 7fdce2000700 1 -- 172.21.15.12:0/2047746747 >> 172.21.15.12:0/2047746747 conn(0x7fdcdc0b0d70 msgr2=0x7fdcdc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.110+0000 7fdce2000700 1 -- 172.21.15.12:0/2047746747 shutdown_connections 2024-03-29T11:32:34.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.110+0000 7fdce2000700 1 -- 172.21.15.12:0/2047746747 wait complete. 2024-03-29T11:32:34.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.112+0000 7fdce2000700 1 Processor -- start 2024-03-29T11:32:34.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.112+0000 7fdce2000700 1 -- start start 2024-03-29T11:32:34.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.112+0000 7fdce2000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcdc0ce840 0x7fdcdc0ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdce2000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcdc15a5c0 0x7fdcdc1f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdce2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15dfe0 0x7fdcdc0c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdce2000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc0c8930 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdce2000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc0ac980 con 0x7fdcdc15a5c0 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdce2000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc0afd70 con 0x7fdcdc0ce840 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdcdbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15dfe0 0x7fdcdc0c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdcdbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15dfe0 0x7fdcdc0c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34898/0 (socket says 172.21.15.12:34898) 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdcdbe00700 1 -- 172.21.15.12:0/2639488586 learned_addr learned my addr 172.21.15.12:0/2639488586 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdcdb400700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcdc0ce840 0x7fdcdc0ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdcdaa00700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcdc15a5c0 0x7fdcdc1f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdcdbe00700 1 -- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcdc0ce840 msgr2=0x7fdcdc0ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdcdbe00700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcdc0ce840 0x7fdcdc0ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:34.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdcdbe00700 1 -- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcdc15a5c0 msgr2=0x7fdcdc1f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:34.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdcdbe00700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcdc15a5c0 0x7fdcdc1f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:34.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.113+0000 7fdcdbe00700 1 -- 172.21.15.12:0/2639488586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc0094a0 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.114+0000 7fdcdbe00700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15dfe0 0x7fdcdc0c83f0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7fdcd400cbe0 tx=0x7fdcd400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:34.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.114+0000 7fdcd3400700 1 -- 172.21.15.12:0/2639488586 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd400aaa0 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.114+0000 7fdcd3400700 1 -- 172.21.15.12:0/2639488586 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd400ac20 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.114+0000 7fdcd3400700 1 -- 172.21.15.12:0/2639488586 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40070d0 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.114+0000 7fdce2000700 1 -- 172.21.15.12:0/2639488586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc0b0050 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.114+0000 7fdce2000700 1 -- 172.21.15.12:0/2639488586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcdc0afa40 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.115+0000 7fdce2000700 1 -- 172.21.15.12:0/2639488586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcdc0b0570 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.116+0000 7fdcd3400700 1 -- 172.21.15.12:0/2639488586 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fdcd4007230 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.117+0000 7fdcd3400700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcbc06aed0 0x7fdcbc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:34.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.117+0000 7fdcd3400700 1 -- 172.21.15.12:0/2639488586 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fdcd40892d0 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.117+0000 7fdcdb400700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcbc06aed0 0x7fdcbc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:34.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.118+0000 7fdcdb400700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcbc06aed0 0x7fdcbc06d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0090b0 tx=0x7fdccc01c040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:34.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.121+0000 7fdcd3400700 1 -- 172.21.15.12:0/2639488586 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdcdc0b0570 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:34.389+0000 7fdce2000700 1 -- 172.21.15.12:0/2639488586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdcdc0b0570 con 0x7fdcdc15dfe0 2024-03-29T11:32:34.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-03-29T11:32:34.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-29T11:32:34.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-29T11:32:35.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.313+0000 7fdcd3400700 1 -- 172.21.15.12:0/2639488586 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v121) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdcdc0b0570 con 0x7fdcdc15dfe0 2024-03-29T11:32:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.316+0000 7fdce2000700 1 -- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcbc06aed0 msgr2=0x7fdcbc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.316+0000 7fdce2000700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcbc06aed0 0x7fdcbc06d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0090b0 tx=0x7fdccc01c040 comp rx=0 tx=0).stop 2024-03-29T11:32:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.316+0000 7fdce2000700 1 -- 172.21.15.12:0/2639488586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15dfe0 msgr2=0x7fdcdc0c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.316+0000 7fdce2000700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15dfe0 0x7fdcdc0c83f0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7fdcd400cbe0 tx=0x7fdcd400cef0 comp rx=0 tx=0).stop 2024-03-29T11:32:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.316+0000 7fdce2000700 1 -- 172.21.15.12:0/2639488586 shutdown_connections 2024-03-29T11:32:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.316+0000 7fdce2000700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcbc06aed0 0x7fdcbc06d390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.316+0000 7fdce2000700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc15dfe0 0x7fdcdc0c83f0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:35.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.316+0000 7fdce2000700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcdc0ce840 0x7fdcdc0ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.316+0000 7fdce2000700 1 --2- 172.21.15.12:0/2639488586 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcdc15a5c0 0x7fdcdc1f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.316+0000 7fdce2000700 1 -- 172.21.15.12:0/2639488586 >> 172.21.15.12:0/2639488586 conn(0x7fdcdc0b0d70 msgr2=0x7fdcdc06e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.317+0000 7fdce2000700 1 -- 172.21.15.12:0/2639488586 shutdown_connections 2024-03-29T11:32:35.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.317+0000 7fdce2000700 1 -- 172.21.15.12:0/2639488586 wait complete. 2024-03-29T11:32:35.334 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T11:32:35.334 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T11:32:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.548+0000 7fe6e7200700 1 Processor -- start 2024-03-29T11:32:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.548+0000 7fe6e7200700 1 -- start start 2024-03-29T11:32:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.549+0000 7fe6e7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e00ce6a0 0x7fe6e015dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.550+0000 7fe6e7200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e00709c0 con 0x7fe6e015d800 2024-03-29T11:32:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.550+0000 7fe6e7200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e0070c80 con 0x7fe6e00ce230 2024-03-29T11:32:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.550+0000 7fe6e7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e00b40f0 con 0x7fe6e00ce6a0 2024-03-29T11:32:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.550+0000 7fe6dfe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe6e015d800 0x7fe6e00725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35822/0 (socket says 172.21.15.12:35822) 2024-03-29T11:32:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.550+0000 7fe6dfe00700 1 -- 172.21.15.12:0/1852202380 learned_addr learned my addr 172.21.15.12:0/1852202380 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.550+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872285214 0 0) 0x7fe6e00709c0 con 0x7fe6e015d800 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.550+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c40034d0 con 0x7fe6e015d800 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.550+0000 7fe6e5200700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e00ce6a0 0x7fe6e015dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3114780803 0 0) 0x7fe6c40034d0 con 0x7fe6e015d800 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e00ce6a0 msgr2=0x7fe6e015dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e00ce6a0 0x7fe6e015dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 >> v1:172.21.15.159:6789/0 conn(0x7fe6e00ce230 legacy=0x7fe6e00b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e0070f60 con 0x7fe6e015d800 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fe6d0001670 con 0x7fe6e015d800 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 >> v1:172.21.15.12:6789/0 conn(0x7fe6e015d800 legacy=0x7fe6e00725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 0x7fe6c4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e00ce6a0 0x7fe6c400a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6c400b090 0x7fe6c400d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e00b40f0 con 0x7fe6c4003b20 2024-03-29T11:32:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c400dd00 con 0x7fe6c400b090 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c400dec0 con 0x7fe6e00ce6a0 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dfe00700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 0x7fe6c4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6e5200700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6c400b090 0x7fe6c400d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6df400700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e00ce6a0 0x7fe6c400a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dfe00700 1 -- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e00ce6a0 msgr2=0x7fe6c400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dfe00700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e00ce6a0 0x7fe6c400a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dfe00700 1 -- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6c400b090 msgr2=0x7fe6c400d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dfe00700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6c400b090 0x7fe6c400d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6dfe00700 1 -- 172.21.15.12:0/1852202380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c40034d0 con 0x7fe6c4003b20 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.551+0000 7fe6e5200700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6c400b090 0x7fe6c400d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.552+0000 7fe6dfe00700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 0x7fe6c4003f40 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000c160 tx=0x7fe6c4003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38dfb17ee8a503f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.552+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0007b70 con 0x7fe6c4003b20 2024-03-29T11:32:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.552+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d000cd20 con 0x7fe6c4003b20 2024-03-29T11:32:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.552+0000 7fe6dea00700 1 -- 172.21.15.12:0/1852202380 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0005ba0 con 0x7fe6c4003b20 2024-03-29T11:32:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.552+0000 7fe6e7200700 1 -- 172.21.15.12:0/1852202380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 msgr2=0x7fe6c4003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.552+0000 7fe6e7200700 1 --2- 172.21.15.12:0/1852202380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 0x7fe6c4003f40 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000c160 tx=0x7fe6c4003780 comp rx=0 tx=0).stop 2024-03-29T11:32:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.552+0000 7fe6e5200700 1 -- 172.21.15.12:0/1852202380 reap_dead start 2024-03-29T11:32:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.552+0000 7fe6e7200700 1 -- 172.21.15.12:0/1852202380 shutdown_connections 2024-03-29T11:32:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.552+0000 7fe6e7200700 1 -- 172.21.15.12:0/1852202380 >> 172.21.15.12:0/1852202380 conn(0x7fe6e00b0d70 msgr2=0x7fe6e00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.553+0000 7fe6e7200700 1 -- 172.21.15.12:0/1852202380 shutdown_connections 2024-03-29T11:32:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.553+0000 7fe6e7200700 1 -- 172.21.15.12:0/1852202380 wait complete. 2024-03-29T11:32:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.554+0000 7fe6e7200700 1 Processor -- start 2024-03-29T11:32:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.554+0000 7fe6e7200700 1 -- start start 2024-03-29T11:32:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.555+0000 7fe6e7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 0x7fe6e01579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.555+0000 7fe6e7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e015a800 0x7fe6e0148550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.555+0000 7fe6e7200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6e0158630 0x7fe6e0158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.555+0000 7fe6e7200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e0157f10 con 0x7fe6c4003b20 2024-03-29T11:32:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.555+0000 7fe6e7200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e0158ff0 con 0x7fe6e0158630 2024-03-29T11:32:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.555+0000 7fe6e7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e015ac80 con 0x7fe6e015a800 2024-03-29T11:32:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6dfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 0x7fe6e01579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6dfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 0x7fe6e01579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34924/0 (socket says 172.21.15.12:34924) 2024-03-29T11:32:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6dfe00700 1 -- 172.21.15.12:0/1190251715 learned_addr learned my addr 172.21.15.12:0/1190251715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6df400700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e015a800 0x7fe6e0148550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6dfe00700 1 -- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e015a800 msgr2=0x7fe6e0148550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6dfe00700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e015a800 0x7fe6e0148550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6dfe00700 1 -- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6e0158630 msgr2=0x7fe6e0158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6e5200700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6e0158630 0x7fe6e0158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6dfe00700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6e0158630 0x7fe6e0158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6dfe00700 1 -- 172.21.15.12:0/1190251715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e006e0f0 con 0x7fe6c4003b20 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6df400700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e015a800 0x7fe6e0148550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6e5200700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6e0158630 0x7fe6e0158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.556+0000 7fe6dfe00700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 0x7fe6e01579d0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400dcd0 tx=0x7fe6d0001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.557+0000 7fe6d7e00700 1 -- 172.21.15.12:0/1190251715 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0004d40 con 0x7fe6c4003b20 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.557+0000 7fe6d7e00700 1 -- 172.21.15.12:0/1190251715 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d00070d0 con 0x7fe6c4003b20 2024-03-29T11:32:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.557+0000 7fe6d7e00700 1 -- 172.21.15.12:0/1190251715 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0004ea0 con 0x7fe6c4003b20 2024-03-29T11:32:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.557+0000 7fe6e7200700 1 -- 172.21.15.12:0/1190251715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e006e3d0 con 0x7fe6c4003b20 2024-03-29T11:32:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.557+0000 7fe6e7200700 1 -- 172.21.15.12:0/1190251715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e006e990 con 0x7fe6c4003b20 2024-03-29T11:32:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.558+0000 7fe6e7200700 1 -- 172.21.15.12:0/1190251715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e006ed00 con 0x7fe6c4003b20 2024-03-29T11:32:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.559+0000 7fe6d7e00700 1 -- 172.21.15.12:0/1190251715 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe6d0019070 con 0x7fe6c4003b20 2024-03-29T11:32:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.560+0000 7fe6d7e00700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe6c006aed0 0x7fe6c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.560+0000 7fe6d7e00700 1 -- 172.21.15.12:0/1190251715 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fe6d008ad70 con 0x7fe6c4003b20 2024-03-29T11:32:35.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.560+0000 7fe6df400700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe6c006aed0 0x7fe6c006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:35.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.561+0000 7fe6df400700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe6c006aed0 0x7fe6c006d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc007fd0 tx=0x7fe6cc009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:35.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.564+0000 7fe6d7e00700 1 -- 172.21.15.12:0/1190251715 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6d0055b90 con 0x7fe6c4003b20 2024-03-29T11:32:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:35.829+0000 7fe6e7200700 1 -- 172.21.15.12:0/1190251715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe6e0067720 con 0x7fe6c4003b20 2024-03-29T11:32:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.423+0000 7fe6d7e00700 1 -- 172.21.15.12:0/1190251715 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v122) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe6d00170c0 con 0x7fe6c4003b20 2024-03-29T11:32:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.426+0000 7fe6e7200700 1 -- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe6c006aed0 msgr2=0x7fe6c006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.426+0000 7fe6e7200700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe6c006aed0 0x7fe6c006d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc007fd0 tx=0x7fe6cc009040 comp rx=0 tx=0).stop 2024-03-29T11:32:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.426+0000 7fe6e7200700 1 -- 172.21.15.12:0/1190251715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 msgr2=0x7fe6e01579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.426+0000 7fe6e7200700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 0x7fe6e01579d0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400dcd0 tx=0x7fe6d0001930 comp rx=0 tx=0).stop 2024-03-29T11:32:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.426+0000 7fe6e7200700 1 -- 172.21.15.12:0/1190251715 shutdown_connections 2024-03-29T11:32:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.426+0000 7fe6e7200700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe6c006aed0 0x7fe6c006d390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.426+0000 7fe6e7200700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6e0158630 0x7fe6e0158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.426+0000 7fe6e7200700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6c4003b20 0x7fe6e01579d0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.426+0000 7fe6e7200700 1 --2- 172.21.15.12:0/1190251715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6e015a800 0x7fe6e0148550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.426+0000 7fe6e7200700 1 -- 172.21.15.12:0/1190251715 >> 172.21.15.12:0/1190251715 conn(0x7fe6e00b0d70 msgr2=0x7fe6e00cbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.427+0000 7fe6e7200700 1 -- 172.21.15.12:0/1190251715 shutdown_connections 2024-03-29T11:32:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.427+0000 7fe6e7200700 1 -- 172.21.15.12:0/1190251715 wait complete. 2024-03-29T11:32:36.443 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T11:32:36.443 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.649+0000 7f55d1800700 1 Processor -- start 2024-03-29T11:32:36.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.649+0000 7f55d1800700 1 -- start start 2024-03-29T11:32:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.650+0000 7f55d1800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f55cc0d18b0 0x7f55cc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.650+0000 7f55d1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 0x7f55cc161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.650+0000 7f55d1800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc0ceda0 con 0x7f55cc160f10 2024-03-29T11:32:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.650+0000 7f55d1800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc0d1d30 con 0x7f55cc0d18b0 2024-03-29T11:32:36.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.650+0000 7f55d1800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc161f60 con 0x7f55cc161590 2024-03-29T11:32:36.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.651+0000 7f55ca000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f55cc0d18b0 0x7f55cc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:36.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.651+0000 7f55cb400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 0x7f55cc161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:36.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.651+0000 7f55caa00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f55cc160f10 0x7f55cc0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:49460/0 (socket says 172.21.15.12:49460) 2024-03-29T11:32:36.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.651+0000 7f55caa00700 1 -- 172.21.15.12:0/3455316978 learned_addr learned my addr 172.21.15.12:0/3455316978 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:36.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.651+0000 7f55c9600700 1 -- 172.21.15.12:0/3455316978 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566304235 0 0) 0x7f55cc0ceda0 con 0x7f55cc160f10 2024-03-29T11:32:36.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.652+0000 7f55c9600700 1 -- 172.21.15.12:0/3455316978 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b00035b0 con 0x7f55cc160f10 2024-03-29T11:32:36.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.652+0000 7f55cb400700 1 -- 172.21.15.12:0/3455316978 >> v2:172.21.15.159:3300/0 conn(0x7f55cc0d18b0 msgr2=0x7f55cc0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.652+0000 7f55cb400700 1 --2- 172.21.15.12:0/3455316978 >> v2:172.21.15.159:3300/0 conn(0x7f55cc0d18b0 0x7f55cc0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.652+0000 7f55cb400700 1 -- 172.21.15.12:0/3455316978 >> v1:172.21.15.159:6789/0 conn(0x7f55cc160f10 legacy=0x7f55cc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.652+0000 7f55cb400700 1 -- 172.21.15.12:0/3455316978 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc1612f0 con 0x7f55cc161590 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.652+0000 7f55ca000700 1 --2- 172.21.15.12:0/3455316978 >> v2:172.21.15.159:3300/0 conn(0x7f55cc0d18b0 0x7f55cc0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.652+0000 7f55cb400700 1 --2- 172.21.15.12:0/3455316978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 0x7f55cc161a20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009d90 tx=0x7f55c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e08630b211b8e1e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.652+0000 7f55c9600700 1 -- 172.21.15.12:0/3455316978 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c400e070 con 0x7f55cc161590 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.652+0000 7f55c9600700 1 -- 172.21.15.12:0/3455316978 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c400bb20 con 0x7f55cc161590 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.652+0000 7f55c9600700 1 -- 172.21.15.12:0/3455316978 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4012430 con 0x7f55cc161590 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.653+0000 7f55d1800700 1 -- 172.21.15.12:0/3455316978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 msgr2=0x7f55cc161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.653+0000 7f55d1800700 1 --2- 172.21.15.12:0/3455316978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 0x7f55cc161a20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009d90 tx=0x7f55c400b330 comp rx=0 tx=0).stop 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.653+0000 7f55d1800700 1 -- 172.21.15.12:0/3455316978 shutdown_connections 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.653+0000 7f55d1800700 1 --2- 172.21.15.12:0/3455316978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 0x7f55cc161a20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.653+0000 7f55d1800700 1 --2- 172.21.15.12:0/3455316978 >> v2:172.21.15.159:3300/0 conn(0x7f55cc0d18b0 0x7f55cc0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.653+0000 7f55d1800700 1 -- 172.21.15.12:0/3455316978 >> 172.21.15.12:0/3455316978 conn(0x7f55cc08aff0 msgr2=0x7f55cc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.653+0000 7f55d1800700 1 -- 172.21.15.12:0/3455316978 shutdown_connections 2024-03-29T11:32:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.653+0000 7f55d1800700 1 -- 172.21.15.12:0/3455316978 wait complete. 2024-03-29T11:32:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.655+0000 7f55d1800700 1 Processor -- start 2024-03-29T11:32:36.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.655+0000 7f55d1800700 1 -- start start 2024-03-29T11:32:36.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55d1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55cc0d18b0 0x7f55cc1f0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:36.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55d1800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55cc160f10 0x7f55cc1f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55d1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 0x7f55cc15f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55d1800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc1593b0 con 0x7f55cc0d18b0 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55d1800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc1f0b50 con 0x7f55cc160f10 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55d1800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0b7170 con 0x7f55cc161590 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55caa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55cc0d18b0 0x7f55cc1f0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55caa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55cc0d18b0 0x7f55cc1f0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34936/0 (socket says 172.21.15.12:34936) 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55caa00700 1 -- 172.21.15.12:0/3060673692 learned_addr learned my addr 172.21.15.12:0/3060673692 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55ca000700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55cc160f10 0x7f55cc1f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55cb400700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 0x7f55cc15f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55caa00700 1 -- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 msgr2=0x7f55cc15f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55caa00700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 0x7f55cc15f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55caa00700 1 -- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55cc160f10 msgr2=0x7f55cc1f30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55caa00700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55cc160f10 0x7f55cc1f30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55caa00700 1 -- 172.21.15.12:0/3060673692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b80094a0 con 0x7f55cc0d18b0 2024-03-29T11:32:36.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.656+0000 7f55ca000700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55cc160f10 0x7f55cc1f30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:36.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.657+0000 7f55caa00700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55cc0d18b0 0x7f55cc1f0610 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f55bc00c4d0 tx=0x7f55bc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:36.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.657+0000 7f55c2a00700 1 -- 172.21.15.12:0/3060673692 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc005d70 con 0x7f55cc0d18b0 2024-03-29T11:32:36.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.657+0000 7f55c2a00700 1 -- 172.21.15.12:0/3060673692 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc0070d0 con 0x7f55cc0d18b0 2024-03-29T11:32:36.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.657+0000 7f55c2a00700 1 -- 172.21.15.12:0/3060673692 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc007460 con 0x7f55cc0d18b0 2024-03-29T11:32:36.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.657+0000 7f55d1800700 1 -- 172.21.15.12:0/3060673692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c4009a70 con 0x7f55cc0d18b0 2024-03-29T11:32:36.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.657+0000 7f55d1800700 1 -- 172.21.15.12:0/3060673692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc1f3bf0 con 0x7f55cc0d18b0 2024-03-29T11:32:36.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.658+0000 7f55d1800700 1 -- 172.21.15.12:0/3060673692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc067720 con 0x7f55cc0d18b0 2024-03-29T11:32:36.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.659+0000 7f55c2a00700 1 -- 172.21.15.12:0/3060673692 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f55bc0075c0 con 0x7f55cc0d18b0 2024-03-29T11:32:36.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.660+0000 7f55c2a00700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55ac06aed0 0x7f55ac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:36.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.660+0000 7f55c2a00700 1 -- 172.21.15.12:0/3060673692 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f55bc088d30 con 0x7f55cc0d18b0 2024-03-29T11:32:36.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.660+0000 7f55ca000700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55ac06aed0 0x7f55ac06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:36.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.661+0000 7f55ca000700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55ac06aed0 0x7f55ac06d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0b4b40 tx=0x7f55b8008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:36.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.664+0000 7f55c2a00700 1 -- 172.21.15.12:0/3060673692 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55bc053ac0 con 0x7f55cc0d18b0 2024-03-29T11:32:36.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.922+0000 7f55d1800700 1 -- 172.21.15.12:0/3060673692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55cc0b7500 con 0x7f55cc0d18b0 2024-03-29T11:32:36.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.922+0000 7f55c2a00700 1 -- 172.21.15.12:0/3060673692 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f55bc017090 con 0x7f55cc0d18b0 2024-03-29T11:32:36.918 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:36.918 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11416,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:32.933737+0000","modified":"2024-03-29T11:32:36.420698+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11853":{"gid":11853,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3711953177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3711953177},{"type":"v1","addr":"172.21.15.12:6834","nonce":3711953177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-03-29T11:32:36.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 -- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55ac06aed0 msgr2=0x7f55ac06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55ac06aed0 0x7f55ac06d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0b4b40 tx=0x7f55b8008040 comp rx=0 tx=0).stop 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 -- 172.21.15.12:0/3060673692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55cc0d18b0 msgr2=0x7f55cc1f0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55cc0d18b0 0x7f55cc1f0610 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f55bc00c4d0 tx=0x7f55bc00c890 comp rx=0 tx=0).stop 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 -- 172.21.15.12:0/3060673692 shutdown_connections 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f55ac06aed0 0x7f55ac06d390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55cc161590 0x7f55cc15f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55cc0d18b0 0x7f55cc1f0610 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 --2- 172.21.15.12:0/3060673692 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55cc160f10 0x7f55cc1f30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 -- 172.21.15.12:0/3060673692 >> 172.21.15.12:0/3060673692 conn(0x7f55cc08aff0 msgr2=0x7f55cc0ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 -- 172.21.15.12:0/3060673692 shutdown_connections 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:36.925+0000 7f55d1800700 1 -- 172.21.15.12:0/3060673692 wait complete. 2024-03-29T11:32:36.921 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 122 2024-03-29T11:32:36.939 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:37.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.141+0000 7fdf28a00700 1 Processor -- start 2024-03-29T11:32:37.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.141+0000 7fdf28a00700 1 -- start start 2024-03-29T11:32:37.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.142+0000 7fdf28a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf240d18b0 0x7fdf240cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.142+0000 7fdf28a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf240ceda0 con 0x7fdf24161590 2024-03-29T11:32:37.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf28a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf241612f0 con 0x7fdf24160f10 2024-03-29T11:32:37.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf28a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf240d1d30 con 0x7fdf240d18b0 2024-03-29T11:32:37.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf22800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdf24161590 0x7fdf240b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35834/0 (socket says 172.21.15.12:35834) 2024-03-29T11:32:37.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf22800700 1 -- 172.21.15.12:0/4248202869 learned_addr learned my addr 172.21.15.12:0/4248202869 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:37.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf21400700 1 --2- 172.21.15.12:0/4248202869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf240d18b0 0x7fdf240cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf20a00700 1 -- 172.21.15.12:0/4248202869 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672167189 0 0) 0x7fdf240ceda0 con 0x7fdf24161590 2024-03-29T11:32:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf20a00700 1 -- 172.21.15.12:0/4248202869 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf080035b0 con 0x7fdf24161590 2024-03-29T11:32:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf20a00700 1 -- 172.21.15.12:0/4248202869 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253008361 0 0) 0x7fdf241612f0 con 0x7fdf24160f10 2024-03-29T11:32:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf20a00700 1 -- 172.21.15.12:0/4248202869 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf240ceda0 con 0x7fdf24160f10 2024-03-29T11:32:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf21400700 1 -- 172.21.15.12:0/4248202869 >> v1:172.21.15.159:6789/0 conn(0x7fdf24160f10 legacy=0x7fdf240d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.143+0000 7fdf21400700 1 -- 172.21.15.12:0/4248202869 >> v1:172.21.15.12:6789/0 conn(0x7fdf24161590 legacy=0x7fdf240b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.144+0000 7fdf21400700 1 -- 172.21.15.12:0/4248202869 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf240ce470 con 0x7fdf240d18b0 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.144+0000 7fdf21400700 1 --2- 172.21.15.12:0/4248202869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf240d18b0 0x7fdf240cf230 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdf100098a0 tx=0x7fdf10009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fb850563c64412a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.144+0000 7fdf20a00700 1 -- 172.21.15.12:0/4248202869 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10011830 con 0x7fdf240d18b0 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.144+0000 7fdf20a00700 1 -- 172.21.15.12:0/4248202869 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf10011990 con 0x7fdf240d18b0 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.144+0000 7fdf20a00700 1 -- 172.21.15.12:0/4248202869 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10011d20 con 0x7fdf240d18b0 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.145+0000 7fdf28a00700 1 -- 172.21.15.12:0/4248202869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf240d18b0 msgr2=0x7fdf240cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.145+0000 7fdf28a00700 1 --2- 172.21.15.12:0/4248202869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf240d18b0 0x7fdf240cf230 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdf100098a0 tx=0x7fdf10009bb0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.145+0000 7fdf28a00700 1 -- 172.21.15.12:0/4248202869 shutdown_connections 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.145+0000 7fdf28a00700 1 --2- 172.21.15.12:0/4248202869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf240d18b0 0x7fdf240cf230 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.145+0000 7fdf28a00700 1 -- 172.21.15.12:0/4248202869 >> 172.21.15.12:0/4248202869 conn(0x7fdf2408aff0 msgr2=0x7fdf240d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.145+0000 7fdf28a00700 1 -- 172.21.15.12:0/4248202869 shutdown_connections 2024-03-29T11:32:37.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.145+0000 7fdf28a00700 1 -- 172.21.15.12:0/4248202869 wait complete. 2024-03-29T11:32:37.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.147+0000 7fdf28a00700 1 Processor -- start 2024-03-29T11:32:37.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.147+0000 7fdf28a00700 1 -- start start 2024-03-29T11:32:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf28a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf240d18b0 0x7fdf24151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf28a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf24160f10 0x7fdf2415bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf28a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf24161590 0x7fdf240b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf28a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2415c880 con 0x7fdf240d18b0 2024-03-29T11:32:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf28a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24154280 con 0x7fdf24160f10 2024-03-29T11:32:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf28a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24154b20 con 0x7fdf24161590 2024-03-29T11:32:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf21e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf240d18b0 0x7fdf24151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf21e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf240d18b0 0x7fdf24151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34946/0 (socket says 172.21.15.12:34946) 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf21e00700 1 -- 172.21.15.12:0/1198700137 learned_addr learned my addr 172.21.15.12:0/1198700137 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf22800700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf24161590 0x7fdf240b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf21400700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf24160f10 0x7fdf2415bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf21e00700 1 -- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf24161590 msgr2=0x7fdf240b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf21e00700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf24161590 0x7fdf240b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf21e00700 1 -- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf24160f10 msgr2=0x7fdf2415bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf21e00700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf24160f10 0x7fdf2415bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.148+0000 7fdf21e00700 1 -- 172.21.15.12:0/1198700137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf100094a0 con 0x7fdf240d18b0 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.149+0000 7fdf21e00700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf240d18b0 0x7fdf24151590 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400c530 tx=0x7fdf1400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.149+0000 7fdf1a000700 1 -- 172.21.15.12:0/1198700137 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf140052d0 con 0x7fdf240d18b0 2024-03-29T11:32:37.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.149+0000 7fdf1a000700 1 -- 172.21.15.12:0/1198700137 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf14005450 con 0x7fdf240d18b0 2024-03-29T11:32:37.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.149+0000 7fdf1a000700 1 -- 172.21.15.12:0/1198700137 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf140038c0 con 0x7fdf240d18b0 2024-03-29T11:32:37.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.149+0000 7fdf28a00700 1 -- 172.21.15.12:0/1198700137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf24151ad0 con 0x7fdf240d18b0 2024-03-29T11:32:37.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.149+0000 7fdf28a00700 1 -- 172.21.15.12:0/1198700137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf241547b0 con 0x7fdf240d18b0 2024-03-29T11:32:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.150+0000 7fdf28a00700 1 -- 172.21.15.12:0/1198700137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf24067720 con 0x7fdf240d18b0 2024-03-29T11:32:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.151+0000 7fdf1a000700 1 -- 172.21.15.12:0/1198700137 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fdf14003a20 con 0x7fdf240d18b0 2024-03-29T11:32:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.152+0000 7fdf1a000700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf0406af20 0x7fdf0406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.152+0000 7fdf1a000700 1 -- 172.21.15.12:0/1198700137 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fdf14088b00 con 0x7fdf240d18b0 2024-03-29T11:32:37.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.152+0000 7fdf21400700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf0406af20 0x7fdf0406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.153+0000 7fdf21400700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf0406af20 0x7fdf0406d3e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf100097c0 tx=0x7fdf1001c040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.156+0000 7fdf1a000700 1 -- 172.21.15.12:0/1198700137 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf14053ae0 con 0x7fdf240d18b0 2024-03-29T11:32:37.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.367+0000 7fdf28a00700 1 -- 172.21.15.12:0/1198700137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf2408b760 con 0x7fdf240d18b0 2024-03-29T11:32:37.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.369+0000 7fdf1a000700 1 -- 172.21.15.12:0/1198700137 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+16167 (secure 0 0 0) 0x7fdf14057100 con 0x7fdf240d18b0 2024-03-29T11:32:37.365 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:37.365 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":248,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:33.936359+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:37.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.371+0000 7fdf28a00700 1 -- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf0406af20 msgr2=0x7fdf0406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.371+0000 7fdf28a00700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf0406af20 0x7fdf0406d3e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf100097c0 tx=0x7fdf1001c040 comp rx=0 tx=0).stop 2024-03-29T11:32:37.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.371+0000 7fdf28a00700 1 -- 172.21.15.12:0/1198700137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf240d18b0 msgr2=0x7fdf24151590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.371+0000 7fdf28a00700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf240d18b0 0x7fdf24151590 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400c530 tx=0x7fdf1400c8f0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.371+0000 7fdf28a00700 1 -- 172.21.15.12:0/1198700137 shutdown_connections 2024-03-29T11:32:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.371+0000 7fdf28a00700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf0406af20 0x7fdf0406d3e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.371+0000 7fdf28a00700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf24161590 0x7fdf240b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.371+0000 7fdf28a00700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf240d18b0 0x7fdf24151590 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.371+0000 7fdf28a00700 1 --2- 172.21.15.12:0/1198700137 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf24160f10 0x7fdf2415bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.371+0000 7fdf28a00700 1 -- 172.21.15.12:0/1198700137 >> 172.21.15.12:0/1198700137 conn(0x7fdf2408aff0 msgr2=0x7fdf240cb140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.372+0000 7fdf28a00700 1 -- 172.21.15.12:0/1198700137 shutdown_connections 2024-03-29T11:32:37.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.372+0000 7fdf28a00700 1 -- 172.21.15.12:0/1198700137 wait complete. 2024-03-29T11:32:37.387 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T11:32:37.387 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:32:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.591+0000 7f2f2f600700 1 Processor -- start 2024-03-29T11:32:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.591+0000 7f2f2f600700 1 -- start start 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.592+0000 7f2f2f600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2f28160f10 0x7f2f280d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.592+0000 7f2f2f600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f280cf230 0x7f2f280d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.592+0000 7f2f2f600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2f28161590 0x7f2f28161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.592+0000 7f2f2f600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f28161f60 con 0x7f2f28160f10 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.592+0000 7f2f2f600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f280ceda0 con 0x7f2f28161590 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.592+0000 7f2f2f600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280d23d0 con 0x7f2f280cf230 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.592+0000 7f2f2cc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2f28160f10 0x7f2f280d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.592+0000 7f2f2cc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2f28160f10 0x7f2f280d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34956/0 (socket says 172.21.15.12:34956) 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.592+0000 7f2f2cc00700 1 -- 172.21.15.12:0/3105086841 learned_addr learned my addr 172.21.15.12:0/3105086841 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f2cc00700 1 -- 172.21.15.12:0/3105086841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f280cf230 msgr2=0x7f2f280d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f2d600700 1 --2- 172.21.15.12:0/3105086841 >> v2:172.21.15.159:3300/0 conn(0x7f2f28161590 0x7f2f28161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f27e00700 1 --2- 172.21.15.12:0/3105086841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f280cf230 0x7f2f280d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f2cc00700 1 --2- 172.21.15.12:0/3105086841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f280cf230 0x7f2f280d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f2cc00700 1 -- 172.21.15.12:0/3105086841 >> v2:172.21.15.159:3300/0 conn(0x7f2f28161590 msgr2=0x7f2f28161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f2cc00700 1 --2- 172.21.15.12:0/3105086841 >> v2:172.21.15.159:3300/0 conn(0x7f2f28161590 0x7f2f28161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f2cc00700 1 -- 172.21.15.12:0/3105086841 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f281612f0 con 0x7f2f28160f10 2024-03-29T11:32:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f2d600700 1 --2- 172.21.15.12:0/3105086841 >> v2:172.21.15.159:3300/0 conn(0x7f2f28161590 0x7f2f28161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f27e00700 1 --2- 172.21.15.12:0/3105086841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f280cf230 0x7f2f280d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f2cc00700 1 --2- 172.21.15.12:0/3105086841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28160f10 0x7f2f280d18b0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f2f280cfad0 tx=0x7f2f1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ecf96918238dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f27400700 1 -- 172.21.15.12:0/3105086841 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c00b870 con 0x7f2f28160f10 2024-03-29T11:32:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f27400700 1 -- 172.21.15.12:0/3105086841 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1c00b9d0 con 0x7f2f28160f10 2024-03-29T11:32:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.593+0000 7f2f27400700 1 -- 172.21.15.12:0/3105086841 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c00bd60 con 0x7f2f28160f10 2024-03-29T11:32:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.594+0000 7f2f2f600700 1 -- 172.21.15.12:0/3105086841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28160f10 msgr2=0x7f2f280d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.594+0000 7f2f2f600700 1 --2- 172.21.15.12:0/3105086841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28160f10 0x7f2f280d18b0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f2f280cfad0 tx=0x7f2f1c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.594+0000 7f2f2f600700 1 -- 172.21.15.12:0/3105086841 shutdown_connections 2024-03-29T11:32:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.594+0000 7f2f2f600700 1 --2- 172.21.15.12:0/3105086841 >> v2:172.21.15.159:3300/0 conn(0x7f2f28161590 0x7f2f28161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.594+0000 7f2f2f600700 1 --2- 172.21.15.12:0/3105086841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28160f10 0x7f2f280d18b0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.594+0000 7f2f2f600700 1 --2- 172.21.15.12:0/3105086841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f280cf230 0x7f2f280d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.594+0000 7f2f2f600700 1 -- 172.21.15.12:0/3105086841 >> 172.21.15.12:0/3105086841 conn(0x7f2f2808aff0 msgr2=0x7f2f280d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.594+0000 7f2f2f600700 1 -- 172.21.15.12:0/3105086841 shutdown_connections 2024-03-29T11:32:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.594+0000 7f2f2f600700 1 -- 172.21.15.12:0/3105086841 wait complete. 2024-03-29T11:32:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.597+0000 7f2f2f600700 1 Processor -- start 2024-03-29T11:32:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.597+0000 7f2f2f600700 1 -- start start 2024-03-29T11:32:37.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.598+0000 7f2f2f600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f280cf230 0x7f2f2815fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.598+0000 7f2f2f600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f28160f10 0x7f2f2815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.598+0000 7f2f2f600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28161590 0x7f2f2809ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.598+0000 7f2f2f600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2809e9c0 con 0x7f2f28161590 2024-03-29T11:32:37.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.598+0000 7f2f2f600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2815f630 con 0x7f2f280cf230 2024-03-29T11:32:37.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.598+0000 7f2f2f600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280a1430 con 0x7f2f28160f10 2024-03-29T11:32:37.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.598+0000 7f2f2d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28161590 0x7f2f2809ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.598+0000 7f2f2d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28161590 0x7f2f2809ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34972/0 (socket says 172.21.15.12:34972) 2024-03-29T11:32:37.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.598+0000 7f2f2d600700 1 -- 172.21.15.12:0/1439988609 learned_addr learned my addr 172.21.15.12:0/1439988609 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:37.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f2cc00700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f280cf230 0x7f2f2815fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f27e00700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f28160f10 0x7f2f2815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f2d600700 1 -- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f28160f10 msgr2=0x7f2f2815baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f2d600700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f28160f10 0x7f2f2815baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f2d600700 1 -- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f280cf230 msgr2=0x7f2f2815fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f2d600700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f280cf230 0x7f2f2815fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f2d600700 1 -- 172.21.15.12:0/1439988609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c009550 con 0x7f2f28161590 2024-03-29T11:32:37.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f2cc00700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f280cf230 0x7f2f2815fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:37.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f2d600700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28161590 0x7f2f2809ec40 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000cdb0 tx=0x7f2f2000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:37.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f24c00700 1 -- 172.21.15.12:0/1439988609 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2000f480 con 0x7f2f28161590 2024-03-29T11:32:37.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f24c00700 1 -- 172.21.15.12:0/1439988609 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2000f600 con 0x7f2f28161590 2024-03-29T11:32:37.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f2f600700 1 -- 172.21.15.12:0/1439988609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f280a1150 con 0x7f2f28161590 2024-03-29T11:32:37.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f24c00700 1 -- 172.21.15.12:0/1439988609 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2000f9b0 con 0x7f2f28161590 2024-03-29T11:32:37.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.599+0000 7f2f2f600700 1 -- 172.21.15.12:0/1439988609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2809e1c0 con 0x7f2f28161590 2024-03-29T11:32:37.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.601+0000 7f2f2f600700 1 -- 172.21.15.12:0/1439988609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f28067720 con 0x7f2f28161590 2024-03-29T11:32:37.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.601+0000 7f2f24c00700 1 -- 172.21.15.12:0/1439988609 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2f2000b040 con 0x7f2f28161590 2024-03-29T11:32:37.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.602+0000 7f2f24c00700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2f0806aed0 0x7f2f0806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:37.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.602+0000 7f2f24c00700 1 -- 172.21.15.12:0/1439988609 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f2f20089f80 con 0x7f2f28161590 2024-03-29T11:32:37.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.603+0000 7f2f2cc00700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2f0806aed0 0x7f2f0806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:37.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.603+0000 7f2f2cc00700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2f0806aed0 0x7f2f0806d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c014040 tx=0x7f2f1c008030 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:37.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.607+0000 7f2f24c00700 1 -- 172.21.15.12:0/1439988609 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f20054c90 con 0x7f2f28161590 2024-03-29T11:32:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.816+0000 7f2f2f600700 1 -- 172.21.15.12:0/1439988609 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f2809dd70 con 0x7f2f0806aed0 2024-03-29T11:32:37.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.828+0000 7f2f24c00700 1 -- 172.21.15.12:0/1439988609 <== mgr.11050 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289505 (secure 0 0 0) 0x7f2f2809dd70 con 0x7f2f0806aed0 2024-03-29T11:32:37.829 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.832+0000 7f2f2f600700 1 -- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2f0806aed0 msgr2=0x7f2f0806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.832+0000 7f2f2f600700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2f0806aed0 0x7f2f0806d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c014040 tx=0x7f2f1c008030 comp rx=0 tx=0).stop 2024-03-29T11:32:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.832+0000 7f2f2f600700 1 -- 172.21.15.12:0/1439988609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28161590 msgr2=0x7f2f2809ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.832+0000 7f2f2f600700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28161590 0x7f2f2809ec40 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000cdb0 tx=0x7f2f2000a560 comp rx=0 tx=0).stop 2024-03-29T11:32:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.833+0000 7f2f2f600700 1 -- 172.21.15.12:0/1439988609 shutdown_connections 2024-03-29T11:32:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.833+0000 7f2f2f600700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2f0806aed0 0x7f2f0806d390 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.833+0000 7f2f2f600700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f28161590 0x7f2f2809ec40 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.833+0000 7f2f2f600700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f280cf230 0x7f2f2815fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.833+0000 7f2f2f600700 1 --2- 172.21.15.12:0/1439988609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f28160f10 0x7f2f2815baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.833+0000 7f2f2f600700 1 -- 172.21.15.12:0/1439988609 >> 172.21.15.12:0/1439988609 conn(0x7f2f2808aff0 msgr2=0x7f2f2806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:37.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.833+0000 7f2f2f600700 1 -- 172.21.15.12:0/1439988609 shutdown_connections 2024-03-29T11:32:37.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:37.833+0000 7f2f2f600700 1 -- 172.21.15.12:0/1439988609 wait complete. 2024-03-29T11:32:37.834 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:32:37.846 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-03-29T11:32:36.529073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":695,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":433604,"kb_used_data":25520,"kb_used_omap":12,"kb_used_meta":407923,"kb_avail":754541116,"statfs":{"total":773094113280,"available":772650102784,"internally_reserved":0,"allocated":26132480,"data_stored":13379925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":417713721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.132860"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426029+0000","last_change":"2024-03-29T11:32:32.953487+0000","last_active":"2024-03-29T11:32:34.426029+0000","last_peered":"2024-03-29T11:32:34.426029+0000","last_clean":"2024-03-29T11:32:34.426029+0000","last_became_active":"2024-03-29T11:32:32.953243+0000","last_became_peered":"2024-03-29T11:32:32.953243+0000","last_unstale":"2024-03-29T11:32:34.426029+0000","last_undegraded":"2024-03-29T11:32:34.426029+0000","last_fullsized":"2024-03-29T11:32:34.426029+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:11.369063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426361+0000","last_change":"2024-03-29T11:32:32.956013+0000","last_active":"2024-03-29T11:32:34.426361+0000","last_peered":"2024-03-29T11:32:34.426361+0000","last_clean":"2024-03-29T11:32:34.426361+0000","last_became_active":"2024-03-29T11:32:32.954012+0000","last_became_peered":"2024-03-29T11:32:32.954012+0000","last_unstale":"2024-03-29T11:32:34.426361+0000","last_undegraded":"2024-03-29T11:32:34.426361+0000","last_fullsized":"2024-03-29T11:32:34.426361+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:02:36.221843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426669+0000","last_change":"2024-03-29T11:32:32.966647+0000","last_active":"2024-03-29T11:32:34.426669+0000","last_peered":"2024-03-29T11:32:34.426669+0000","last_clean":"2024-03-29T11:32:34.426669+0000","last_became_active":"2024-03-29T11:32:32.966394+0000","last_became_peered":"2024-03-29T11:32:32.966394+0000","last_unstale":"2024-03-29T11:32:34.426669+0000","last_undegraded":"2024-03-29T11:32:34.426669+0000","last_fullsized":"2024-03-29T11:32:34.426669+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:22.600133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425470+0000","last_change":"2024-03-29T11:32:32.955745+0000","last_active":"2024-03-29T11:32:34.425470+0000","last_peered":"2024-03-29T11:32:34.425470+0000","last_clean":"2024-03-29T11:32:34.425470+0000","last_became_active":"2024-03-29T11:32:32.955142+0000","last_became_peered":"2024-03-29T11:32:32.955142+0000","last_unstale":"2024-03-29T11:32:34.425470+0000","last_undegraded":"2024-03-29T11:32:34.425470+0000","last_fullsized":"2024-03-29T11:32:34.425470+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:42.190121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.946994+0000","last_change":"2024-03-29T11:32:32.950908+0000","last_active":"2024-03-29T11:32:33.946994+0000","last_peered":"2024-03-29T11:32:33.946994+0000","last_clean":"2024-03-29T11:32:33.946994+0000","last_became_active":"2024-03-29T11:32:32.950581+0000","last_became_peered":"2024-03-29T11:32:32.950581+0000","last_unstale":"2024-03-29T11:32:33.946994+0000","last_undegraded":"2024-03-29T11:32:33.946994+0000","last_fullsized":"2024-03-29T11:32:33.946994+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:11.495727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425397+0000","last_change":"2024-03-29T11:32:32.966829+0000","last_active":"2024-03-29T11:32:34.425397+0000","last_peered":"2024-03-29T11:32:34.425397+0000","last_clean":"2024-03-29T11:32:34.425397+0000","last_became_active":"2024-03-29T11:32:32.966561+0000","last_became_peered":"2024-03-29T11:32:32.966561+0000","last_unstale":"2024-03-29T11:32:34.425397+0000","last_undegraded":"2024-03-29T11:32:34.425397+0000","last_fullsized":"2024-03-29T11:32:34.425397+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947638+0000","last_change":"2024-03-29T11:32:32.956044+0000","last_active":"2024-03-29T11:32:33.947638+0000","last_peered":"2024-03-29T11:32:33.947638+0000","last_clean":"2024-03-29T11:32:33.947638+0000","last_became_active":"2024-03-29T11:32:32.955298+0000","last_became_peered":"2024-03-29T11:32:32.955298+0000","last_unstale":"2024-03-29T11:32:33.947638+0000","last_undegraded":"2024-03-29T11:32:33.947638+0000","last_fullsized":"2024-03-29T11:32:33.947638+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:36.536991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426215+0000","last_change":"2024-03-29T11:32:32.956054+0000","last_active":"2024-03-29T11:32:34.426215+0000","last_peered":"2024-03-29T11:32:34.426215+0000","last_clean":"2024-03-29T11:32:34.426215+0000","last_became_active":"2024-03-29T11:32:32.952773+0000","last_became_peered":"2024-03-29T11:32:32.952773+0000","last_unstale":"2024-03-29T11:32:34.426215+0000","last_undegraded":"2024-03-29T11:32:34.426215+0000","last_fullsized":"2024-03-29T11:32:34.426215+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:59:43.666251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425918+0000","last_change":"2024-03-29T11:32:32.963463+0000","last_active":"2024-03-29T11:32:34.425918+0000","last_peered":"2024-03-29T11:32:34.425918+0000","last_clean":"2024-03-29T11:32:34.425918+0000","last_became_active":"2024-03-29T11:32:32.962970+0000","last_became_peered":"2024-03-29T11:32:32.962970+0000","last_unstale":"2024-03-29T11:32:34.425918+0000","last_undegraded":"2024-03-29T11:32:34.425918+0000","last_fullsized":"2024-03-29T11:32:34.425918+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:14.637128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948201+0000","last_change":"2024-03-29T11:32:32.953703+0000","last_active":"2024-03-29T11:32:33.948201+0000","last_peered":"2024-03-29T11:32:33.948201+0000","last_clean":"2024-03-29T11:32:33.948201+0000","last_became_active":"2024-03-29T11:32:32.952942+0000","last_became_peered":"2024-03-29T11:32:32.952942+0000","last_unstale":"2024-03-29T11:32:33.948201+0000","last_undegraded":"2024-03-29T11:32:33.948201+0000","last_fullsized":"2024-03-29T11:32:33.948201+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:23:44.759018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947820+0000","last_change":"2024-03-29T11:32:32.970743+0000","last_active":"2024-03-29T11:32:33.947820+0000","last_peered":"2024-03-29T11:32:33.947820+0000","last_clean":"2024-03-29T11:32:33.947820+0000","last_became_active":"2024-03-29T11:32:32.967319+0000","last_became_peered":"2024-03-29T11:32:32.967319+0000","last_unstale":"2024-03-29T11:32:33.947820+0000","last_undegraded":"2024-03-29T11:32:33.947820+0000","last_fullsized":"2024-03-29T11:32:33.947820+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:52.668591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426478+0000","last_change":"2024-03-29T11:32:32.957788+0000","last_active":"2024-03-29T11:32:34.426478+0000","last_peered":"2024-03-29T11:32:34.426478+0000","last_clean":"2024-03-29T11:32:34.426478+0000","last_became_active":"2024-03-29T11:32:32.957519+0000","last_became_peered":"2024-03-29T11:32:32.957519+0000","last_unstale":"2024-03-29T11:32:34.426478+0000","last_undegraded":"2024-03-29T11:32:34.426478+0000","last_fullsized":"2024-03-29T11:32:34.426478+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:21.664133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948137+0000","last_change":"2024-03-29T11:32:32.967536+0000","last_active":"2024-03-29T11:32:33.948137+0000","last_peered":"2024-03-29T11:32:33.948137+0000","last_clean":"2024-03-29T11:32:33.948137+0000","last_became_active":"2024-03-29T11:32:32.966686+0000","last_became_peered":"2024-03-29T11:32:32.966686+0000","last_unstale":"2024-03-29T11:32:33.948137+0000","last_undegraded":"2024-03-29T11:32:33.948137+0000","last_fullsized":"2024-03-29T11:32:33.948137+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:24.736585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426215+0000","last_change":"2024-03-29T11:32:32.952674+0000","last_active":"2024-03-29T11:32:34.426215+0000","last_peered":"2024-03-29T11:32:34.426215+0000","last_clean":"2024-03-29T11:32:34.426215+0000","last_became_active":"2024-03-29T11:32:32.952387+0000","last_became_peered":"2024-03-29T11:32:32.952387+0000","last_unstale":"2024-03-29T11:32:34.426215+0000","last_undegraded":"2024-03-29T11:32:34.426215+0000","last_fullsized":"2024-03-29T11:32:34.426215+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:30.121534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425648+0000","last_change":"2024-03-29T11:32:32.966028+0000","last_active":"2024-03-29T11:32:34.425648+0000","last_peered":"2024-03-29T11:32:34.425648+0000","last_clean":"2024-03-29T11:32:34.425648+0000","last_became_active":"2024-03-29T11:32:32.965642+0000","last_became_peered":"2024-03-29T11:32:32.965642+0000","last_unstale":"2024-03-29T11:32:34.425648+0000","last_undegraded":"2024-03-29T11:32:34.425648+0000","last_fullsized":"2024-03-29T11:32:34.425648+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:28:31.977713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425206+0000","last_change":"2024-03-29T11:32:32.960337+0000","last_active":"2024-03-29T11:32:34.425206+0000","last_peered":"2024-03-29T11:32:34.425206+0000","last_clean":"2024-03-29T11:32:34.425206+0000","last_became_active":"2024-03-29T11:32:32.958824+0000","last_became_peered":"2024-03-29T11:32:32.958824+0000","last_unstale":"2024-03-29T11:32:34.425206+0000","last_undegraded":"2024-03-29T11:32:34.425206+0000","last_fullsized":"2024-03-29T11:32:34.425206+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:48:40.247306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426632+0000","last_change":"2024-03-29T11:32:32.960497+0000","last_active":"2024-03-29T11:32:34.426632+0000","last_peered":"2024-03-29T11:32:34.426632+0000","last_clean":"2024-03-29T11:32:34.426632+0000","last_became_active":"2024-03-29T11:32:32.958333+0000","last_became_peered":"2024-03-29T11:32:32.958333+0000","last_unstale":"2024-03-29T11:32:34.426632+0000","last_undegraded":"2024-03-29T11:32:34.426632+0000","last_fullsized":"2024-03-29T11:32:34.426632+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:27:28.513987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426366+0000","last_change":"2024-03-29T11:32:32.949294+0000","last_active":"2024-03-29T11:32:34.426366+0000","last_peered":"2024-03-29T11:32:34.426366+0000","last_clean":"2024-03-29T11:32:34.426366+0000","last_became_active":"2024-03-29T11:32:32.949025+0000","last_became_peered":"2024-03-29T11:32:32.949025+0000","last_unstale":"2024-03-29T11:32:34.426366+0000","last_undegraded":"2024-03-29T11:32:34.426366+0000","last_fullsized":"2024-03-29T11:32:34.426366+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426685+0000","last_change":"2024-03-29T11:32:32.965153+0000","last_active":"2024-03-29T11:32:34.426685+0000","last_peered":"2024-03-29T11:32:34.426685+0000","last_clean":"2024-03-29T11:32:34.426685+0000","last_became_active":"2024-03-29T11:32:32.964935+0000","last_became_peered":"2024-03-29T11:32:32.964935+0000","last_unstale":"2024-03-29T11:32:34.426685+0000","last_undegraded":"2024-03-29T11:32:34.426685+0000","last_fullsized":"2024-03-29T11:32:34.426685+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:02.769642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426529+0000","last_change":"2024-03-29T11:32:32.953083+0000","last_active":"2024-03-29T11:32:34.426529+0000","last_peered":"2024-03-29T11:32:34.426529+0000","last_clean":"2024-03-29T11:32:34.426529+0000","last_became_active":"2024-03-29T11:32:32.952823+0000","last_became_peered":"2024-03-29T11:32:32.952823+0000","last_unstale":"2024-03-29T11:32:34.426529+0000","last_undegraded":"2024-03-29T11:32:34.426529+0000","last_fullsized":"2024-03-29T11:32:34.426529+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:09.626524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426292+0000","last_change":"2024-03-29T11:32:32.954755+0000","last_active":"2024-03-29T11:32:34.426292+0000","last_peered":"2024-03-29T11:32:34.426292+0000","last_clean":"2024-03-29T11:32:34.426292+0000","last_became_active":"2024-03-29T11:32:32.954467+0000","last_became_peered":"2024-03-29T11:32:32.954467+0000","last_unstale":"2024-03-29T11:32:34.426292+0000","last_undegraded":"2024-03-29T11:32:34.426292+0000","last_fullsized":"2024-03-29T11:32:34.426292+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:10.303092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947741+0000","last_change":"2024-03-29T11:32:32.955216+0000","last_active":"2024-03-29T11:32:33.947741+0000","last_peered":"2024-03-29T11:32:33.947741+0000","last_clean":"2024-03-29T11:32:33.947741+0000","last_became_active":"2024-03-29T11:32:32.954774+0000","last_became_peered":"2024-03-29T11:32:32.954774+0000","last_unstale":"2024-03-29T11:32:33.947741+0000","last_undegraded":"2024-03-29T11:32:33.947741+0000","last_fullsized":"2024-03-29T11:32:33.947741+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:44.576542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426250+0000","last_change":"2024-03-29T11:32:32.964085+0000","last_active":"2024-03-29T11:32:34.426250+0000","last_peered":"2024-03-29T11:32:34.426250+0000","last_clean":"2024-03-29T11:32:34.426250+0000","last_became_active":"2024-03-29T11:32:32.963753+0000","last_became_peered":"2024-03-29T11:32:32.963753+0000","last_unstale":"2024-03-29T11:32:34.426250+0000","last_undegraded":"2024-03-29T11:32:34.426250+0000","last_fullsized":"2024-03-29T11:32:34.426250+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:34.956119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425416+0000","last_change":"2024-03-29T11:32:32.957520+0000","last_active":"2024-03-29T11:32:34.425416+0000","last_peered":"2024-03-29T11:32:34.425416+0000","last_clean":"2024-03-29T11:32:34.425416+0000","last_became_active":"2024-03-29T11:32:32.957245+0000","last_became_peered":"2024-03-29T11:32:32.957245+0000","last_unstale":"2024-03-29T11:32:34.425416+0000","last_undegraded":"2024-03-29T11:32:34.425416+0000","last_fullsized":"2024-03-29T11:32:34.425416+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:45.368395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426117+0000","last_change":"2024-03-29T11:32:32.955473+0000","last_active":"2024-03-29T11:32:34.426117+0000","last_peered":"2024-03-29T11:32:34.426117+0000","last_clean":"2024-03-29T11:32:34.426117+0000","last_became_active":"2024-03-29T11:32:32.953854+0000","last_became_peered":"2024-03-29T11:32:32.953854+0000","last_unstale":"2024-03-29T11:32:34.426117+0000","last_undegraded":"2024-03-29T11:32:34.426117+0000","last_fullsized":"2024-03-29T11:32:34.426117+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:21.819962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948298+0000","last_change":"2024-03-29T11:32:32.967040+0000","last_active":"2024-03-29T11:32:33.948298+0000","last_peered":"2024-03-29T11:32:33.948298+0000","last_clean":"2024-03-29T11:32:33.948298+0000","last_became_active":"2024-03-29T11:32:32.966729+0000","last_became_peered":"2024-03-29T11:32:32.966729+0000","last_unstale":"2024-03-29T11:32:33.948298+0000","last_undegraded":"2024-03-29T11:32:33.948298+0000","last_fullsized":"2024-03-29T11:32:33.948298+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:27.515858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947925+0000","last_change":"2024-03-29T11:32:32.970545+0000","last_active":"2024-03-29T11:32:33.947925+0000","last_peered":"2024-03-29T11:32:33.947925+0000","last_clean":"2024-03-29T11:32:33.947925+0000","last_became_active":"2024-03-29T11:32:32.966917+0000","last_became_peered":"2024-03-29T11:32:32.966917+0000","last_unstale":"2024-03-29T11:32:33.947925+0000","last_undegraded":"2024-03-29T11:32:33.947925+0000","last_fullsized":"2024-03-29T11:32:33.947925+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:11.611059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426064+0000","last_change":"2024-03-29T11:32:32.950772+0000","last_active":"2024-03-29T11:32:34.426064+0000","last_peered":"2024-03-29T11:32:34.426064+0000","last_clean":"2024-03-29T11:32:34.426064+0000","last_became_active":"2024-03-29T11:32:32.950518+0000","last_became_peered":"2024-03-29T11:32:32.950518+0000","last_unstale":"2024-03-29T11:32:34.426064+0000","last_undegraded":"2024-03-29T11:32:34.426064+0000","last_fullsized":"2024-03-29T11:32:34.426064+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:34.956119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947739+0000","last_change":"2024-03-29T11:32:32.956094+0000","last_active":"2024-03-29T11:32:33.947739+0000","last_peered":"2024-03-29T11:32:33.947739+0000","last_clean":"2024-03-29T11:32:33.947739+0000","last_became_active":"2024-03-29T11:32:32.955788+0000","last_became_peered":"2024-03-29T11:32:32.955788+0000","last_unstale":"2024-03-29T11:32:33.947739+0000","last_undegraded":"2024-03-29T11:32:33.947739+0000","last_fullsized":"2024-03-29T11:32:33.947739+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:05.005123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947552+0000","last_change":"2024-03-29T11:32:32.951190+0000","last_active":"2024-03-29T11:32:33.947552+0000","last_peered":"2024-03-29T11:32:33.947552+0000","last_clean":"2024-03-29T11:32:33.947552+0000","last_became_active":"2024-03-29T11:32:32.950830+0000","last_became_peered":"2024-03-29T11:32:32.950830+0000","last_unstale":"2024-03-29T11:32:33.947552+0000","last_undegraded":"2024-03-29T11:32:33.947552+0000","last_fullsized":"2024-03-29T11:32:33.947552+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:12.191813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425220+0000","last_change":"2024-03-29T11:32:32.949171+0000","last_active":"2024-03-29T11:32:34.425220+0000","last_peered":"2024-03-29T11:32:34.425220+0000","last_clean":"2024-03-29T11:32:34.425220+0000","last_became_active":"2024-03-29T11:32:32.948898+0000","last_became_peered":"2024-03-29T11:32:32.948898+0000","last_unstale":"2024-03-29T11:32:34.425220+0000","last_undegraded":"2024-03-29T11:32:34.425220+0000","last_fullsized":"2024-03-29T11:32:34.425220+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:02:36.221843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425655+0000","last_change":"2024-03-29T11:32:32.956902+0000","last_active":"2024-03-29T11:32:34.425655+0000","last_peered":"2024-03-29T11:32:34.425655+0000","last_clean":"2024-03-29T11:32:34.425655+0000","last_became_active":"2024-03-29T11:32:32.955472+0000","last_became_peered":"2024-03-29T11:32:32.955472+0000","last_unstale":"2024-03-29T11:32:34.425655+0000","last_undegraded":"2024-03-29T11:32:34.425655+0000","last_fullsized":"2024-03-29T11:32:34.425655+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:29.628408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426593+0000","last_change":"2024-03-29T11:32:32.959493+0000","last_active":"2024-03-29T11:32:34.426593+0000","last_peered":"2024-03-29T11:32:34.426593+0000","last_clean":"2024-03-29T11:32:34.426593+0000","last_became_active":"2024-03-29T11:32:32.959199+0000","last_became_peered":"2024-03-29T11:32:32.959199+0000","last_unstale":"2024-03-29T11:32:34.426593+0000","last_undegraded":"2024-03-29T11:32:34.426593+0000","last_fullsized":"2024-03-29T11:32:34.426593+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:56.148657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"247'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948400+0000","last_change":"2024-03-29T11:32:32.419758+0000","last_active":"2024-03-29T11:32:33.948400+0000","last_peered":"2024-03-29T11:32:33.948400+0000","last_clean":"2024-03-29T11:32:33.948400+0000","last_became_active":"2024-03-29T11:32:32.419044+0000","last_became_peered":"2024-03-29T11:32:32.419044+0000","last_unstale":"2024-03-29T11:32:33.948400+0000","last_undegraded":"2024-03-29T11:32:33.948400+0000","last_fullsized":"2024-03-29T11:32:33.948400+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:29.373413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947625+0000","last_change":"2024-03-29T11:32:32.964445+0000","last_active":"2024-03-29T11:32:33.947625+0000","last_peered":"2024-03-29T11:32:33.947625+0000","last_clean":"2024-03-29T11:32:33.947625+0000","last_became_active":"2024-03-29T11:32:32.963758+0000","last_became_peered":"2024-03-29T11:32:32.963758+0000","last_unstale":"2024-03-29T11:32:33.947625+0000","last_undegraded":"2024-03-29T11:32:33.947625+0000","last_fullsized":"2024-03-29T11:32:33.947625+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:24.556045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426574+0000","last_change":"2024-03-29T11:32:32.419690+0000","last_active":"2024-03-29T11:32:34.426574+0000","last_peered":"2024-03-29T11:32:34.426574+0000","last_clean":"2024-03-29T11:32:34.426574+0000","last_became_active":"2024-03-29T11:32:32.418508+0000","last_became_peered":"2024-03-29T11:32:32.418508+0000","last_unstale":"2024-03-29T11:32:34.426574+0000","last_undegraded":"2024-03-29T11:32:34.426574+0000","last_fullsized":"2024-03-29T11:32:34.426574+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:34.818177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426719+0000","last_change":"2024-03-29T11:32:32.960733+0000","last_active":"2024-03-29T11:32:34.426719+0000","last_peered":"2024-03-29T11:32:34.426719+0000","last_clean":"2024-03-29T11:32:34.426719+0000","last_became_active":"2024-03-29T11:32:32.959254+0000","last_became_peered":"2024-03-29T11:32:32.959254+0000","last_unstale":"2024-03-29T11:32:34.426719+0000","last_undegraded":"2024-03-29T11:32:34.426719+0000","last_fullsized":"2024-03-29T11:32:34.426719+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:45:42.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426629+0000","last_change":"2024-03-29T11:32:32.422622+0000","last_active":"2024-03-29T11:32:34.426629+0000","last_peered":"2024-03-29T11:32:34.426629+0000","last_clean":"2024-03-29T11:32:34.426629+0000","last_became_active":"2024-03-29T11:32:32.421864+0000","last_became_peered":"2024-03-29T11:32:32.421864+0000","last_unstale":"2024-03-29T11:32:34.426629+0000","last_undegraded":"2024-03-29T11:32:34.426629+0000","last_fullsized":"2024-03-29T11:32:34.426629+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:23.546712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947481+0000","last_change":"2024-03-29T11:32:32.965844+0000","last_active":"2024-03-29T11:32:33.947481+0000","last_peered":"2024-03-29T11:32:33.947481+0000","last_clean":"2024-03-29T11:32:33.947481+0000","last_became_active":"2024-03-29T11:32:32.965511+0000","last_became_peered":"2024-03-29T11:32:32.965511+0000","last_unstale":"2024-03-29T11:32:33.947481+0000","last_undegraded":"2024-03-29T11:32:33.947481+0000","last_fullsized":"2024-03-29T11:32:33.947481+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:23.043448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"247'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425803+0000","last_change":"2024-03-29T11:32:32.419740+0000","last_active":"2024-03-29T11:32:34.425803+0000","last_peered":"2024-03-29T11:32:34.425803+0000","last_clean":"2024-03-29T11:32:34.425803+0000","last_became_active":"2024-03-29T11:32:32.419471+0000","last_became_peered":"2024-03-29T11:32:32.419471+0000","last_unstale":"2024-03-29T11:32:34.425803+0000","last_undegraded":"2024-03-29T11:32:34.425803+0000","last_fullsized":"2024-03-29T11:32:34.425803+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:01.760308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425793+0000","last_change":"2024-03-29T11:32:32.963417+0000","last_active":"2024-03-29T11:32:34.425793+0000","last_peered":"2024-03-29T11:32:34.425793+0000","last_clean":"2024-03-29T11:32:34.425793+0000","last_became_active":"2024-03-29T11:32:32.963109+0000","last_became_peered":"2024-03-29T11:32:32.963109+0000","last_unstale":"2024-03-29T11:32:34.425793+0000","last_undegraded":"2024-03-29T11:32:34.425793+0000","last_fullsized":"2024-03-29T11:32:34.425793+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:24:49.870800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"247'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426294+0000","last_change":"2024-03-29T11:32:32.417496+0000","last_active":"2024-03-29T11:32:34.426294+0000","last_peered":"2024-03-29T11:32:34.426294+0000","last_clean":"2024-03-29T11:32:34.426294+0000","last_became_active":"2024-03-29T11:32:32.417165+0000","last_became_peered":"2024-03-29T11:32:32.417165+0000","last_unstale":"2024-03-29T11:32:34.426294+0000","last_undegraded":"2024-03-29T11:32:34.426294+0000","last_fullsized":"2024-03-29T11:32:34.426294+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:57.952240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425479+0000","last_change":"2024-03-29T11:32:32.954909+0000","last_active":"2024-03-29T11:32:34.425479+0000","last_peered":"2024-03-29T11:32:34.425479+0000","last_clean":"2024-03-29T11:32:34.425479+0000","last_became_active":"2024-03-29T11:32:32.954609+0000","last_became_peered":"2024-03-29T11:32:32.954609+0000","last_unstale":"2024-03-29T11:32:34.425479+0000","last_undegraded":"2024-03-29T11:32:34.425479+0000","last_fullsized":"2024-03-29T11:32:34.425479+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:21:04.890763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947743+0000","last_change":"2024-03-29T11:32:32.424318+0000","last_active":"2024-03-29T11:32:33.947743+0000","last_peered":"2024-03-29T11:32:33.947743+0000","last_clean":"2024-03-29T11:32:33.947743+0000","last_became_active":"2024-03-29T11:32:32.424027+0000","last_became_peered":"2024-03-29T11:32:32.424027+0000","last_unstale":"2024-03-29T11:32:33.947743+0000","last_undegraded":"2024-03-29T11:32:33.947743+0000","last_fullsized":"2024-03-29T11:32:33.947743+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:20.654799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425702+0000","last_change":"2024-03-29T11:32:32.950803+0000","last_active":"2024-03-29T11:32:34.425702+0000","last_peered":"2024-03-29T11:32:34.425702+0000","last_clean":"2024-03-29T11:32:34.425702+0000","last_became_active":"2024-03-29T11:32:32.950515+0000","last_became_peered":"2024-03-29T11:32:32.950515+0000","last_unstale":"2024-03-29T11:32:34.425702+0000","last_undegraded":"2024-03-29T11:32:34.425702+0000","last_fullsized":"2024-03-29T11:32:34.425702+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:42:17.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425113+0000","last_change":"2024-03-29T11:32:32.419260+0000","last_active":"2024-03-29T11:32:34.425113+0000","last_peered":"2024-03-29T11:32:34.425113+0000","last_clean":"2024-03-29T11:32:34.425113+0000","last_became_active":"2024-03-29T11:32:32.418975+0000","last_became_peered":"2024-03-29T11:32:32.418975+0000","last_unstale":"2024-03-29T11:32:34.425113+0000","last_undegraded":"2024-03-29T11:32:34.425113+0000","last_fullsized":"2024-03-29T11:32:34.425113+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:41.904116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426465+0000","last_change":"2024-03-29T11:32:32.949825+0000","last_active":"2024-03-29T11:32:34.426465+0000","last_peered":"2024-03-29T11:32:34.426465+0000","last_clean":"2024-03-29T11:32:34.426465+0000","last_became_active":"2024-03-29T11:32:32.949562+0000","last_became_peered":"2024-03-29T11:32:32.949562+0000","last_unstale":"2024-03-29T11:32:34.426465+0000","last_undegraded":"2024-03-29T11:32:34.426465+0000","last_fullsized":"2024-03-29T11:32:34.426465+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:23.557124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426747+0000","last_change":"2024-03-29T11:32:32.426409+0000","last_active":"2024-03-29T11:32:34.426747+0000","last_peered":"2024-03-29T11:32:34.426747+0000","last_clean":"2024-03-29T11:32:34.426747+0000","last_became_active":"2024-03-29T11:32:32.426142+0000","last_became_peered":"2024-03-29T11:32:32.426142+0000","last_unstale":"2024-03-29T11:32:34.426747+0000","last_undegraded":"2024-03-29T11:32:34.426747+0000","last_fullsized":"2024-03-29T11:32:34.426747+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:51:51.755610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948181+0000","last_change":"2024-03-29T11:32:32.974753+0000","last_active":"2024-03-29T11:32:33.948181+0000","last_peered":"2024-03-29T11:32:33.948181+0000","last_clean":"2024-03-29T11:32:33.948181+0000","last_became_active":"2024-03-29T11:32:32.971332+0000","last_became_peered":"2024-03-29T11:32:32.971332+0000","last_unstale":"2024-03-29T11:32:33.948181+0000","last_undegraded":"2024-03-29T11:32:33.948181+0000","last_fullsized":"2024-03-29T11:32:33.948181+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:33.815472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426620+0000","last_change":"2024-03-29T11:32:32.420533+0000","last_active":"2024-03-29T11:32:34.426620+0000","last_peered":"2024-03-29T11:32:34.426620+0000","last_clean":"2024-03-29T11:32:34.426620+0000","last_became_active":"2024-03-29T11:32:32.420284+0000","last_became_peered":"2024-03-29T11:32:32.420284+0000","last_unstale":"2024-03-29T11:32:34.426620+0000","last_undegraded":"2024-03-29T11:32:34.426620+0000","last_fullsized":"2024-03-29T11:32:34.426620+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:31.173444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426544+0000","last_change":"2024-03-29T11:32:32.969812+0000","last_active":"2024-03-29T11:32:34.426544+0000","last_peered":"2024-03-29T11:32:34.426544+0000","last_clean":"2024-03-29T11:32:34.426544+0000","last_became_active":"2024-03-29T11:32:32.969479+0000","last_became_peered":"2024-03-29T11:32:32.969479+0000","last_unstale":"2024-03-29T11:32:34.426544+0000","last_undegraded":"2024-03-29T11:32:34.426544+0000","last_fullsized":"2024-03-29T11:32:34.426544+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:57.232582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"247'3","reported_seq":14,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426228+0000","last_change":"2024-03-29T11:32:32.418264+0000","last_active":"2024-03-29T11:32:34.426228+0000","last_peered":"2024-03-29T11:32:34.426228+0000","last_clean":"2024-03-29T11:32:34.426228+0000","last_became_active":"2024-03-29T11:32:32.417013+0000","last_became_peered":"2024-03-29T11:32:32.417013+0000","last_unstale":"2024-03-29T11:32:34.426228+0000","last_undegraded":"2024-03-29T11:32:34.426228+0000","last_fullsized":"2024-03-29T11:32:34.426228+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:51.960387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948106+0000","last_change":"2024-03-29T11:32:32.955257+0000","last_active":"2024-03-29T11:32:33.948106+0000","last_peered":"2024-03-29T11:32:33.948106+0000","last_clean":"2024-03-29T11:32:33.948106+0000","last_became_active":"2024-03-29T11:32:32.952017+0000","last_became_peered":"2024-03-29T11:32:32.952017+0000","last_unstale":"2024-03-29T11:32:33.948106+0000","last_undegraded":"2024-03-29T11:32:33.948106+0000","last_fullsized":"2024-03-29T11:32:33.948106+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:45.368395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"247'6","reported_seq":17,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426158+0000","last_change":"2024-03-29T11:32:32.418422+0000","last_active":"2024-03-29T11:32:34.426158+0000","last_peered":"2024-03-29T11:32:34.426158+0000","last_clean":"2024-03-29T11:32:34.426158+0000","last_became_active":"2024-03-29T11:32:32.418119+0000","last_became_peered":"2024-03-29T11:32:32.418119+0000","last_unstale":"2024-03-29T11:32:34.426158+0000","last_undegraded":"2024-03-29T11:32:34.426158+0000","last_fullsized":"2024-03-29T11:32:34.426158+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:34.164745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947987+0000","last_change":"2024-03-29T11:32:32.955062+0000","last_active":"2024-03-29T11:32:33.947987+0000","last_peered":"2024-03-29T11:32:33.947987+0000","last_clean":"2024-03-29T11:32:33.947987+0000","last_became_active":"2024-03-29T11:32:32.954362+0000","last_became_peered":"2024-03-29T11:32:32.954362+0000","last_unstale":"2024-03-29T11:32:33.947987+0000","last_undegraded":"2024-03-29T11:32:33.947987+0000","last_fullsized":"2024-03-29T11:32:33.947987+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:35:32.510118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425947+0000","last_change":"2024-03-29T11:32:32.427566+0000","last_active":"2024-03-29T11:32:34.425947+0000","last_peered":"2024-03-29T11:32:34.425947+0000","last_clean":"2024-03-29T11:32:34.425947+0000","last_became_active":"2024-03-29T11:32:32.426512+0000","last_became_peered":"2024-03-29T11:32:32.426512+0000","last_unstale":"2024-03-29T11:32:34.425947+0000","last_undegraded":"2024-03-29T11:32:34.425947+0000","last_fullsized":"2024-03-29T11:32:34.425947+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:49:23.345901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425929+0000","last_change":"2024-03-29T11:32:32.956252+0000","last_active":"2024-03-29T11:32:34.425929+0000","last_peered":"2024-03-29T11:32:34.425929+0000","last_clean":"2024-03-29T11:32:34.425929+0000","last_became_active":"2024-03-29T11:32:32.955859+0000","last_became_peered":"2024-03-29T11:32:32.955859+0000","last_unstale":"2024-03-29T11:32:34.425929+0000","last_undegraded":"2024-03-29T11:32:34.425929+0000","last_fullsized":"2024-03-29T11:32:34.425929+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:29.934653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425919+0000","last_change":"2024-03-29T11:32:32.420246+0000","last_active":"2024-03-29T11:32:34.425919+0000","last_peered":"2024-03-29T11:32:34.425919+0000","last_clean":"2024-03-29T11:32:34.425919+0000","last_became_active":"2024-03-29T11:32:32.419983+0000","last_became_peered":"2024-03-29T11:32:32.419983+0000","last_unstale":"2024-03-29T11:32:34.425919+0000","last_undegraded":"2024-03-29T11:32:34.425919+0000","last_fullsized":"2024-03-29T11:32:34.425919+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:35.527658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425040+0000","last_change":"2024-03-29T11:32:32.946933+0000","last_active":"2024-03-29T11:32:34.425040+0000","last_peered":"2024-03-29T11:32:34.425040+0000","last_clean":"2024-03-29T11:32:34.425040+0000","last_became_active":"2024-03-29T11:32:32.946663+0000","last_became_peered":"2024-03-29T11:32:32.946663+0000","last_unstale":"2024-03-29T11:32:34.425040+0000","last_undegraded":"2024-03-29T11:32:34.425040+0000","last_fullsized":"2024-03-29T11:32:34.425040+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:23.557124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947784+0000","last_change":"2024-03-29T11:32:32.419238+0000","last_active":"2024-03-29T11:32:33.947784+0000","last_peered":"2024-03-29T11:32:33.947784+0000","last_clean":"2024-03-29T11:32:33.947784+0000","last_became_active":"2024-03-29T11:32:32.416581+0000","last_became_peered":"2024-03-29T11:32:32.416581+0000","last_unstale":"2024-03-29T11:32:33.947784+0000","last_undegraded":"2024-03-29T11:32:33.947784+0000","last_fullsized":"2024-03-29T11:32:33.947784+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:09.293759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947691+0000","last_change":"2024-03-29T11:32:32.960659+0000","last_active":"2024-03-29T11:32:33.947691+0000","last_peered":"2024-03-29T11:32:33.947691+0000","last_clean":"2024-03-29T11:32:33.947691+0000","last_became_active":"2024-03-29T11:32:32.960186+0000","last_became_peered":"2024-03-29T11:32:32.960186+0000","last_unstale":"2024-03-29T11:32:33.947691+0000","last_undegraded":"2024-03-29T11:32:33.947691+0000","last_fullsized":"2024-03-29T11:32:33.947691+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:11.369063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"247'8","reported_seq":19,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947504+0000","last_change":"2024-03-29T11:32:32.420232+0000","last_active":"2024-03-29T11:32:33.947504+0000","last_peered":"2024-03-29T11:32:33.947504+0000","last_clean":"2024-03-29T11:32:33.947504+0000","last_became_active":"2024-03-29T11:32:32.419973+0000","last_became_peered":"2024-03-29T11:32:32.419973+0000","last_unstale":"2024-03-29T11:32:33.947504+0000","last_undegraded":"2024-03-29T11:32:33.947504+0000","last_fullsized":"2024-03-29T11:32:33.947504+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:20.810629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425646+0000","last_change":"2024-03-29T11:32:32.952991+0000","last_active":"2024-03-29T11:32:34.425646+0000","last_peered":"2024-03-29T11:32:34.425646+0000","last_clean":"2024-03-29T11:32:34.425646+0000","last_became_active":"2024-03-29T11:32:32.952683+0000","last_became_peered":"2024-03-29T11:32:32.952683+0000","last_unstale":"2024-03-29T11:32:34.425646+0000","last_undegraded":"2024-03-29T11:32:34.425646+0000","last_fullsized":"2024-03-29T11:32:34.425646+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:30.121534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"247'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948397+0000","last_change":"2024-03-29T11:32:32.419741+0000","last_active":"2024-03-29T11:32:33.948397+0000","last_peered":"2024-03-29T11:32:33.948397+0000","last_clean":"2024-03-29T11:32:33.948397+0000","last_became_active":"2024-03-29T11:32:32.418896+0000","last_became_peered":"2024-03-29T11:32:32.418896+0000","last_unstale":"2024-03-29T11:32:33.948397+0000","last_undegraded":"2024-03-29T11:32:33.948397+0000","last_fullsized":"2024-03-29T11:32:33.948397+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:59:42.656917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426704+0000","last_change":"2024-03-29T11:32:32.963050+0000","last_active":"2024-03-29T11:32:34.426704+0000","last_peered":"2024-03-29T11:32:34.426704+0000","last_clean":"2024-03-29T11:32:34.426704+0000","last_became_active":"2024-03-29T11:32:32.962777+0000","last_became_peered":"2024-03-29T11:32:32.962777+0000","last_unstale":"2024-03-29T11:32:34.426704+0000","last_undegraded":"2024-03-29T11:32:34.426704+0000","last_fullsized":"2024-03-29T11:32:34.426704+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:28.065549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426654+0000","last_change":"2024-03-29T11:32:32.424196+0000","last_active":"2024-03-29T11:32:34.426654+0000","last_peered":"2024-03-29T11:32:34.426654+0000","last_clean":"2024-03-29T11:32:34.426654+0000","last_became_active":"2024-03-29T11:32:32.423924+0000","last_became_peered":"2024-03-29T11:32:32.423924+0000","last_unstale":"2024-03-29T11:32:34.426654+0000","last_undegraded":"2024-03-29T11:32:34.426654+0000","last_fullsized":"2024-03-29T11:32:34.426654+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:51.659258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426048+0000","last_change":"2024-03-29T11:32:32.958521+0000","last_active":"2024-03-29T11:32:34.426048+0000","last_peered":"2024-03-29T11:32:34.426048+0000","last_clean":"2024-03-29T11:32:34.426048+0000","last_became_active":"2024-03-29T11:32:32.958260+0000","last_became_peered":"2024-03-29T11:32:32.958260+0000","last_unstale":"2024-03-29T11:32:34.426048+0000","last_undegraded":"2024-03-29T11:32:34.426048+0000","last_fullsized":"2024-03-29T11:32:34.426048+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:34.744111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"247'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425788+0000","last_change":"2024-03-29T11:32:32.421703+0000","last_active":"2024-03-29T11:32:34.425788+0000","last_peered":"2024-03-29T11:32:34.425788+0000","last_clean":"2024-03-29T11:32:34.425788+0000","last_became_active":"2024-03-29T11:32:32.421417+0000","last_became_peered":"2024-03-29T11:32:32.421417+0000","last_unstale":"2024-03-29T11:32:34.425788+0000","last_undegraded":"2024-03-29T11:32:34.425788+0000","last_fullsized":"2024-03-29T11:32:34.425788+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:51.659258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426771+0000","last_change":"2024-03-29T11:32:32.960765+0000","last_active":"2024-03-29T11:32:34.426771+0000","last_peered":"2024-03-29T11:32:34.426771+0000","last_clean":"2024-03-29T11:32:34.426771+0000","last_became_active":"2024-03-29T11:32:32.959952+0000","last_became_peered":"2024-03-29T11:32:32.959952+0000","last_unstale":"2024-03-29T11:32:34.426771+0000","last_undegraded":"2024-03-29T11:32:34.426771+0000","last_fullsized":"2024-03-29T11:32:34.426771+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:18.696940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947714+0000","last_change":"2024-03-29T11:32:32.423658+0000","last_active":"2024-03-29T11:32:33.947714+0000","last_peered":"2024-03-29T11:32:33.947714+0000","last_clean":"2024-03-29T11:32:33.947714+0000","last_became_active":"2024-03-29T11:32:32.423195+0000","last_became_peered":"2024-03-29T11:32:32.423195+0000","last_unstale":"2024-03-29T11:32:33.947714+0000","last_undegraded":"2024-03-29T11:32:33.947714+0000","last_fullsized":"2024-03-29T11:32:33.947714+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:22.547790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426369+0000","last_change":"2024-03-29T11:32:32.958080+0000","last_active":"2024-03-29T11:32:34.426369+0000","last_peered":"2024-03-29T11:32:34.426369+0000","last_clean":"2024-03-29T11:32:34.426369+0000","last_became_active":"2024-03-29T11:32:32.957784+0000","last_became_peered":"2024-03-29T11:32:32.957784+0000","last_unstale":"2024-03-29T11:32:34.426369+0000","last_undegraded":"2024-03-29T11:32:34.426369+0000","last_fullsized":"2024-03-29T11:32:34.426369+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:48.762233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426512+0000","last_change":"2024-03-29T11:32:32.421797+0000","last_active":"2024-03-29T11:32:34.426512+0000","last_peered":"2024-03-29T11:32:34.426512+0000","last_clean":"2024-03-29T11:32:34.426512+0000","last_became_active":"2024-03-29T11:32:32.418956+0000","last_became_peered":"2024-03-29T11:32:32.418956+0000","last_unstale":"2024-03-29T11:32:34.426512+0000","last_undegraded":"2024-03-29T11:32:34.426512+0000","last_fullsized":"2024-03-29T11:32:34.426512+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:32.806138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426049+0000","last_change":"2024-03-29T11:32:32.951479+0000","last_active":"2024-03-29T11:32:34.426049+0000","last_peered":"2024-03-29T11:32:34.426049+0000","last_clean":"2024-03-29T11:32:34.426049+0000","last_became_active":"2024-03-29T11:32:32.951234+0000","last_became_peered":"2024-03-29T11:32:32.951234+0000","last_unstale":"2024-03-29T11:32:34.426049+0000","last_undegraded":"2024-03-29T11:32:34.426049+0000","last_fullsized":"2024-03-29T11:32:34.426049+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:48.762233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947833+0000","last_change":"2024-03-29T11:32:32.422506+0000","last_active":"2024-03-29T11:32:33.947833+0000","last_peered":"2024-03-29T11:32:33.947833+0000","last_clean":"2024-03-29T11:32:33.947833+0000","last_became_active":"2024-03-29T11:32:32.422249+0000","last_became_peered":"2024-03-29T11:32:32.422249+0000","last_unstale":"2024-03-29T11:32:33.947833+0000","last_undegraded":"2024-03-29T11:32:33.947833+0000","last_fullsized":"2024-03-29T11:32:33.947833+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:29.112200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947425+0000","last_change":"2024-03-29T11:32:32.955109+0000","last_active":"2024-03-29T11:32:33.947425+0000","last_peered":"2024-03-29T11:32:33.947425+0000","last_clean":"2024-03-29T11:32:33.947425+0000","last_became_active":"2024-03-29T11:32:32.954862+0000","last_became_peered":"2024-03-29T11:32:32.954862+0000","last_unstale":"2024-03-29T11:32:33.947425+0000","last_undegraded":"2024-03-29T11:32:33.947425+0000","last_fullsized":"2024-03-29T11:32:33.947425+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:18.702968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425964+0000","last_change":"2024-03-29T11:32:32.429057+0000","last_active":"2024-03-29T11:32:34.425964+0000","last_peered":"2024-03-29T11:32:34.425964+0000","last_clean":"2024-03-29T11:32:34.425964+0000","last_became_active":"2024-03-29T11:32:32.428782+0000","last_became_peered":"2024-03-29T11:32:32.428782+0000","last_unstale":"2024-03-29T11:32:34.425964+0000","last_undegraded":"2024-03-29T11:32:34.425964+0000","last_fullsized":"2024-03-29T11:32:34.425964+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:23.546712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426259+0000","last_change":"2024-03-29T11:32:32.954161+0000","last_active":"2024-03-29T11:32:34.426259+0000","last_peered":"2024-03-29T11:32:34.426259+0000","last_clean":"2024-03-29T11:32:34.426259+0000","last_became_active":"2024-03-29T11:32:32.953891+0000","last_became_peered":"2024-03-29T11:32:32.953891+0000","last_unstale":"2024-03-29T11:32:34.426259+0000","last_undegraded":"2024-03-29T11:32:34.426259+0000","last_fullsized":"2024-03-29T11:32:34.426259+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:42:17.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947669+0000","last_change":"2024-03-29T11:32:32.418585+0000","last_active":"2024-03-29T11:32:33.947669+0000","last_peered":"2024-03-29T11:32:33.947669+0000","last_clean":"2024-03-29T11:32:33.947669+0000","last_became_active":"2024-03-29T11:32:32.418297+0000","last_became_peered":"2024-03-29T11:32:32.418297+0000","last_unstale":"2024-03-29T11:32:33.947669+0000","last_undegraded":"2024-03-29T11:32:33.947669+0000","last_fullsized":"2024-03-29T11:32:33.947669+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:01.760308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948314+0000","last_change":"2024-03-29T11:32:32.953710+0000","last_active":"2024-03-29T11:32:33.948314+0000","last_peered":"2024-03-29T11:32:33.948314+0000","last_clean":"2024-03-29T11:32:33.948314+0000","last_became_active":"2024-03-29T11:32:32.953447+0000","last_became_peered":"2024-03-29T11:32:32.953447+0000","last_unstale":"2024-03-29T11:32:33.948314+0000","last_undegraded":"2024-03-29T11:32:33.948314+0000","last_fullsized":"2024-03-29T11:32:33.948314+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426207+0000","last_change":"2024-03-29T11:32:32.427167+0000","last_active":"2024-03-29T11:32:34.426207+0000","last_peered":"2024-03-29T11:32:34.426207+0000","last_clean":"2024-03-29T11:32:34.426207+0000","last_became_active":"2024-03-29T11:32:32.426833+0000","last_became_peered":"2024-03-29T11:32:32.426833+0000","last_unstale":"2024-03-29T11:32:34.426207+0000","last_undegraded":"2024-03-29T11:32:34.426207+0000","last_fullsized":"2024-03-29T11:32:34.426207+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:29.373413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426051+0000","last_change":"2024-03-29T11:32:32.967461+0000","last_active":"2024-03-29T11:32:34.426051+0000","last_peered":"2024-03-29T11:32:34.426051+0000","last_clean":"2024-03-29T11:32:34.426051+0000","last_became_active":"2024-03-29T11:32:32.967155+0000","last_became_peered":"2024-03-29T11:32:32.967155+0000","last_unstale":"2024-03-29T11:32:34.426051+0000","last_undegraded":"2024-03-29T11:32:34.426051+0000","last_fullsized":"2024-03-29T11:32:34.426051+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:24.736585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947411+0000","last_change":"2024-03-29T11:32:32.421259+0000","last_active":"2024-03-29T11:32:33.947411+0000","last_peered":"2024-03-29T11:32:33.947411+0000","last_clean":"2024-03-29T11:32:33.947411+0000","last_became_active":"2024-03-29T11:32:32.420971+0000","last_became_peered":"2024-03-29T11:32:32.420971+0000","last_unstale":"2024-03-29T11:32:33.947411+0000","last_undegraded":"2024-03-29T11:32:33.947411+0000","last_fullsized":"2024-03-29T11:32:33.947411+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:34.164745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947642+0000","last_change":"2024-03-29T11:32:32.970626+0000","last_active":"2024-03-29T11:32:33.947642+0000","last_peered":"2024-03-29T11:32:33.947642+0000","last_clean":"2024-03-29T11:32:33.947642+0000","last_became_active":"2024-03-29T11:32:32.970154+0000","last_became_peered":"2024-03-29T11:32:32.970154+0000","last_unstale":"2024-03-29T11:32:33.947642+0000","last_undegraded":"2024-03-29T11:32:33.947642+0000","last_fullsized":"2024-03-29T11:32:33.947642+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:35.827510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425993+0000","last_change":"2024-03-29T11:32:32.429420+0000","last_active":"2024-03-29T11:32:34.425993+0000","last_peered":"2024-03-29T11:32:34.425993+0000","last_clean":"2024-03-29T11:32:34.425993+0000","last_became_active":"2024-03-29T11:32:32.429133+0000","last_became_peered":"2024-03-29T11:32:32.429133+0000","last_unstale":"2024-03-29T11:32:34.425993+0000","last_undegraded":"2024-03-29T11:32:34.425993+0000","last_fullsized":"2024-03-29T11:32:34.425993+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:03.995789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948023+0000","last_change":"2024-03-29T11:32:32.971255+0000","last_active":"2024-03-29T11:32:33.948023+0000","last_peered":"2024-03-29T11:32:33.948023+0000","last_clean":"2024-03-29T11:32:33.948023+0000","last_became_active":"2024-03-29T11:32:32.968877+0000","last_became_peered":"2024-03-29T11:32:32.968877+0000","last_unstale":"2024-03-29T11:32:33.948023+0000","last_undegraded":"2024-03-29T11:32:33.948023+0000","last_fullsized":"2024-03-29T11:32:33.948023+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:42.913450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947742+0000","last_change":"2024-03-29T11:32:32.424671+0000","last_active":"2024-03-29T11:32:33.947742+0000","last_peered":"2024-03-29T11:32:33.947742+0000","last_clean":"2024-03-29T11:32:33.947742+0000","last_became_active":"2024-03-29T11:32:32.424378+0000","last_became_peered":"2024-03-29T11:32:32.424378+0000","last_unstale":"2024-03-29T11:32:33.947742+0000","last_undegraded":"2024-03-29T11:32:33.947742+0000","last_fullsized":"2024-03-29T11:32:33.947742+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:41.180787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948098+0000","last_change":"2024-03-29T11:32:32.952424+0000","last_active":"2024-03-29T11:32:33.948098+0000","last_peered":"2024-03-29T11:32:33.948098+0000","last_clean":"2024-03-29T11:32:33.948098+0000","last_became_active":"2024-03-29T11:32:32.951804+0000","last_became_peered":"2024-03-29T11:32:32.951804+0000","last_unstale":"2024-03-29T11:32:33.948098+0000","last_undegraded":"2024-03-29T11:32:33.948098+0000","last_fullsized":"2024-03-29T11:32:33.948098+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:22:05.943330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948021+0000","last_change":"2024-03-29T11:32:32.418931+0000","last_active":"2024-03-29T11:32:33.948021+0000","last_peered":"2024-03-29T11:32:33.948021+0000","last_clean":"2024-03-29T11:32:33.948021+0000","last_became_active":"2024-03-29T11:32:32.418646+0000","last_became_peered":"2024-03-29T11:32:32.418646+0000","last_unstale":"2024-03-29T11:32:33.948021+0000","last_undegraded":"2024-03-29T11:32:33.948021+0000","last_fullsized":"2024-03-29T11:32:33.948021+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:17:12.443004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426346+0000","last_change":"2024-03-29T11:32:32.971041+0000","last_active":"2024-03-29T11:32:34.426346+0000","last_peered":"2024-03-29T11:32:34.426346+0000","last_clean":"2024-03-29T11:32:34.426346+0000","last_became_active":"2024-03-29T11:32:32.970781+0000","last_became_peered":"2024-03-29T11:32:32.970781+0000","last_unstale":"2024-03-29T11:32:34.426346+0000","last_undegraded":"2024-03-29T11:32:34.426346+0000","last_fullsized":"2024-03-29T11:32:34.426346+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:30.382747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425487+0000","last_change":"2024-03-29T11:32:32.414137+0000","last_active":"2024-03-29T11:32:34.425487+0000","last_peered":"2024-03-29T11:32:34.425487+0000","last_clean":"2024-03-29T11:32:34.425487+0000","last_became_active":"2024-03-29T11:32:32.413846+0000","last_became_peered":"2024-03-29T11:32:32.413846+0000","last_unstale":"2024-03-29T11:32:34.425487+0000","last_undegraded":"2024-03-29T11:32:34.425487+0000","last_fullsized":"2024-03-29T11:32:34.425487+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:09.293759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426118+0000","last_change":"2024-03-29T11:32:32.956222+0000","last_active":"2024-03-29T11:32:34.426118+0000","last_peered":"2024-03-29T11:32:34.426118+0000","last_clean":"2024-03-29T11:32:34.426118+0000","last_became_active":"2024-03-29T11:32:32.955784+0000","last_became_peered":"2024-03-29T11:32:32.955784+0000","last_unstale":"2024-03-29T11:32:34.426118+0000","last_undegraded":"2024-03-29T11:32:34.426118+0000","last_fullsized":"2024-03-29T11:32:34.426118+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:52.969721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425735+0000","last_change":"2024-03-29T11:32:32.421490+0000","last_active":"2024-03-29T11:32:34.425735+0000","last_peered":"2024-03-29T11:32:34.425735+0000","last_clean":"2024-03-29T11:32:34.425735+0000","last_became_active":"2024-03-29T11:32:32.421236+0000","last_became_peered":"2024-03-29T11:32:32.421236+0000","last_unstale":"2024-03-29T11:32:34.425735+0000","last_undegraded":"2024-03-29T11:32:34.425735+0000","last_fullsized":"2024-03-29T11:32:34.425735+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:17.693634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425886+0000","last_change":"2024-03-29T11:32:32.950145+0000","last_active":"2024-03-29T11:32:34.425886+0000","last_peered":"2024-03-29T11:32:34.425886+0000","last_clean":"2024-03-29T11:32:34.425886+0000","last_became_active":"2024-03-29T11:32:32.949444+0000","last_became_peered":"2024-03-29T11:32:32.949444+0000","last_unstale":"2024-03-29T11:32:34.425886+0000","last_undegraded":"2024-03-29T11:32:34.425886+0000","last_fullsized":"2024-03-29T11:32:34.425886+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:42.190121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425393+0000","last_change":"2024-03-29T11:32:32.422243+0000","last_active":"2024-03-29T11:32:34.425393+0000","last_peered":"2024-03-29T11:32:34.425393+0000","last_clean":"2024-03-29T11:32:34.425393+0000","last_became_active":"2024-03-29T11:32:32.421949+0000","last_became_peered":"2024-03-29T11:32:32.421949+0000","last_unstale":"2024-03-29T11:32:34.425393+0000","last_undegraded":"2024-03-29T11:32:34.425393+0000","last_fullsized":"2024-03-29T11:32:34.425393+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:03.995789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947429+0000","last_change":"2024-03-29T11:32:32.962313+0000","last_active":"2024-03-29T11:32:33.947429+0000","last_peered":"2024-03-29T11:32:33.947429+0000","last_clean":"2024-03-29T11:32:33.947429+0000","last_became_active":"2024-03-29T11:32:32.961887+0000","last_became_peered":"2024-03-29T11:32:32.961887+0000","last_unstale":"2024-03-29T11:32:33.947429+0000","last_undegraded":"2024-03-29T11:32:33.947429+0000","last_fullsized":"2024-03-29T11:32:33.947429+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:18.696940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"247'4","reported_seq":15,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426703+0000","last_change":"2024-03-29T11:32:32.419906+0000","last_active":"2024-03-29T11:32:34.426703+0000","last_peered":"2024-03-29T11:32:34.426703+0000","last_clean":"2024-03-29T11:32:34.426703+0000","last_became_active":"2024-03-29T11:32:32.419653+0000","last_became_peered":"2024-03-29T11:32:32.419653+0000","last_unstale":"2024-03-29T11:32:34.426703+0000","last_undegraded":"2024-03-29T11:32:34.426703+0000","last_fullsized":"2024-03-29T11:32:34.426703+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:22.034114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"233'388","reported_seq":1367,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-29T11:32:32.944695+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:32:32.944695+0000","last_peered":"2024-03-29T11:32:32.944695+0000","last_clean":"2024-03-29T11:32:32.944695+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:32:32.944695+0000","last_undegraded":"2024-03-29T11:32:32.944695+0000","last_fullsized":"2024-03-29T11:32:32.944695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":388,"ondisk_log_size":388,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56888,"kb_used_data":3620,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314952,"statfs":{"total":96636764160,"available":96578510848,"internally_reserved":0,"allocated":3706880,"data_stored":2114077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.70799999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.73899999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42899999999999999},"max":{"1min":0.89900000000000002,"5min":0.98899999999999999,"15min":1.119},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.69299999999999995,"15min":0.72199999999999998},"min":{"1min":0.437,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90500000000000003,"5min":0.97399999999999998,"15min":1.478},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72799999999999998,"15min":0.75800000000000001},"min":{"1min":0.47399999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.393},"last":0.82699999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70599999999999996,"15min":0.748},"min":{"1min":0.504,"5min":0.436,"15min":0.436},"max":{"1min":0.85899999999999999,"5min":0.998,"15min":1.089},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92100000000000004,"5min":1.032,"15min":1.0720000000000001},"last":0.71199999999999997}]},{"osd":3,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.72299999999999998,"15min":0.753},"min":{"1min":0.54400000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90300000000000002,"5min":0.97499999999999998,"15min":1.0580000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77500000000000002,"15min":0.78300000000000003},"min":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.55300000000000005},"max":{"1min":1.0309999999999999,"5min":1.071,"15min":1.071},"last":0.92500000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":1.3779999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65900000000000003,"15min":0.66900000000000004},"min":{"1min":0.50900000000000001,"5min":0.41599999999999998,"15min":0.40899999999999997},"max":{"1min":0.85099999999999998,"5min":0.93500000000000005,"15min":1.335},"last":0.74299999999999999}]},{"osd":5,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69799999999999995,"15min":0.68100000000000005},"min":{"1min":0.434,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.1160000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.73899999999999999,"15min":0.72199999999999998},"min":{"1min":0.55600000000000005,"5min":0.50900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95499999999999996,"5min":1.0169999999999999,"15min":1.0249999999999999},"last":0.91000000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.76900000000000002,"15min":0.76700000000000002},"min":{"1min":0.52400000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0549999999999999,"15min":1.1319999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.755,"15min":0.74099999999999999},"min":{"1min":0.46899999999999997,"5min":0.439,"15min":0.435},"max":{"1min":1.0580000000000001,"5min":1.071,"15min":1.071},"last":0.78900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834576007,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48120,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323720,"statfs":{"total":96636764160,"available":96587489280,"internally_reserved":0,"allocated":3117056,"data_stored":1524243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71699999999999997,"15min":0.73699999999999999},"min":{"1min":0.45900000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86199999999999999,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.73199999999999998,"15min":0.76100000000000001},"min":{"1min":0.52700000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.84399999999999997,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.627}]},{"osd":1,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.71099999999999997,"15min":0.73699999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":0.83399999999999996,"5min":1.98,"15min":1.98},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.745,"15min":0.77300000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.92500000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.75600000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.5,"5min":0.498,"15min":0.497},"max":{"1min":0.88700000000000001,"5min":1.669,"15min":1.669},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.55700000000000005,"5min":0.49099999999999999,"15min":0.46200000000000002},"max":{"1min":0.879,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.75600000000000001,"15min":0.77300000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.91000000000000003,"5min":2.52,"15min":2.52},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.78800000000000003,"15min":0.80300000000000005},"min":{"1min":0.57099999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.996,"5min":1.877,"15min":1.877},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.62,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":0.76500000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.70899999999999996,"15min":0.70699999999999996},"min":{"1min":0.504,"5min":0.47399999999999998,"15min":0.45900000000000002},"max":{"1min":0.90000000000000002,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":0.81999999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.47299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90200000000000002,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70999999999999996,"15min":0.71299999999999997},"min":{"1min":0.54300000000000004,"5min":0.46999999999999997,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":1.925,"15min":1.925},"last":0.80500000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77100000000000002,"15min":0.76700000000000002},"min":{"1min":0.58099999999999996,"5min":0.54800000000000004,"15min":0.52700000000000002},"max":{"1min":1.0149999999999999,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.74299999999999999,"15min":0.74099999999999999},"min":{"1min":0.54600000000000004,"5min":0.504,"15min":0.47499999999999998},"max":{"1min":0.95099999999999996,"5min":1.726,"15min":1.726},"last":0.72799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576005,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60420,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311420,"statfs":{"total":96636764160,"available":96574894080,"internally_reserved":0,"allocated":3129344,"data_stored":1525327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.44500000000000001,"15min":0.46700000000000003},"min":{"1min":0.29299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69799999999999995,"5min":0.79100000000000004,"15min":0.80300000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.50900000000000001,"15min":0.53200000000000003},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.57199999999999995,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.439}]},{"osd":2,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.56200000000000006},"min":{"1min":0.36499999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.52300000000000002,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.34799999999999998},"max":{"1min":0.61599999999999999,"5min":0.81399999999999995,"15min":0.90300000000000002},"last":0.32600000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.59599999999999997},"min":{"1min":0.433,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73099999999999998,"5min":0.86199999999999999,"15min":0.995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56799999999999995,"15min":0.58099999999999996},"min":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.83999999999999997,"15min":0.878},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.75,"15min":0.78700000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.056,"5min":1.153,"15min":1.3660000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.76200000000000001,"15min":0.80000000000000004},"min":{"1min":0.56299999999999994,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97699999999999998,"5min":1.226,"15min":1.304},"last":0.95299999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.79300000000000004,"15min":0.81999999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":0.997,"5min":1.276,"15min":1.3720000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.099,"5min":1.1299999999999999,"15min":1.3520000000000001},"last":1.044}]},{"osd":6,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.78300000000000003,"15min":0.81899999999999995},"min":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.026,"5min":1.1890000000000001,"15min":1.1919999999999999},"last":1.103},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.042,"5min":1.2250000000000001,"15min":1.3069999999999999},"last":1.012}]},{"osd":7,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.83999999999999997,"15min":0.86899999999999999},"min":{"1min":0.54800000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0649999999999999,"5min":1.327,"15min":1.3540000000000001},"last":1.129},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83399999999999996,"15min":0.86799999999999999},"min":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0700000000000001,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.88700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834576007,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60980,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310860,"statfs":{"total":96636764160,"available":96574320640,"internally_reserved":0,"allocated":3702784,"data_stored":2113022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.45300000000000001,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65300000000000002,"5min":0.95899999999999996,"15min":1.087},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77400000000000002,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84999999999999998,"5min":1.028,"15min":1.5289999999999999},"last":0.502},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51100000000000001,"15min":0.52000000000000002},"min":{"1min":0.372,"5min":0.317,"15min":0.317},"max":{"1min":0.58299999999999996,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.33800000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81000000000000005,"5min":1.222,"15min":1.222},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.42199999999999999},"max":{"1min":0.83499999999999996,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.623}]},{"osd":4,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.747},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.83599999999999997,"15min":0.84999999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.091,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.70599999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.028,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81000000000000005,"15min":0.82199999999999995},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1499999999999999,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.80700000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84999999999999998,"15min":0.85899999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0700000000000001,"5min":1.343,"15min":1.6240000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.873,"15min":0.88800000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.171,"5min":1.3240000000000001,"15min":1.337},"last":0.65000000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97899999999999998,"5min":1.115,"15min":1.214},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.85799999999999998,"15min":0.874},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.204,"5min":1.365,"15min":1.365},"last":0.68600000000000005}]}]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52208,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319632,"statfs":{"total":96636764160,"available":96583303168,"internally_reserved":0,"allocated":3112960,"data_stored":1523729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47799999999999998,"15min":0.47599999999999998},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73199999999999998,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.54100000000000004},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67600000000000005,"5min":0.73999999999999999,"15min":0.96799999999999997},"last":0.377}]},{"osd":1,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74299999999999999,"5min":0.77400000000000002,"15min":0.80700000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55500000000000005,"15min":0.57699999999999996},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68300000000000005,"5min":0.872,"15min":0.88800000000000001},"last":0.42799999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.42299999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86799999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76800000000000002,"5min":1.038,"15min":1.038},"last":0.51300000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.76300000000000001,"15min":0.80300000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.93700000000000006,"5min":1.091,"15min":1.1970000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.755,"15min":0.80800000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.52400000000000002},"max":{"1min":1.026,"5min":1.244,"15min":1.244},"last":0.79500000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.81299999999999994},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.53400000000000003},"max":{"1min":0.94599999999999995,"5min":1.1599999999999999,"15min":1.256},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.78000000000000003,"15min":0.82199999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.995,"5min":1.143,"15min":1.3140000000000001},"last":0.78100000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.79300000000000004,"15min":0.84199999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98099999999999998,"5min":1.1459999999999999,"15min":1.24},"last":0.876},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79700000000000004,"15min":0.83999999999999997},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.97399999999999998,"5min":1.099,"15min":1.329},"last":0.88400000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.83899999999999997,"15min":0.88800000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":0.95899999999999996,"5min":1.1910000000000001,"15min":1.2889999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85499999999999998,"15min":0.90200000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.56899999999999995},"max":{"1min":1.095,"5min":1.2969999999999999,"15min":1.304},"last":0.81699999999999995}]}]},{"osd":3,"up_from":13,"seq":55834576006,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48116,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323724,"statfs":{"total":96636764160,"available":96587493376,"internally_reserved":0,"allocated":3112960,"data_stored":1524221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.47499999999999998,"15min":0.47099999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.36899999999999999,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.36499999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70099999999999996,"5min":0.72099999999999997,"15min":0.748},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54400000000000004,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":1.046},"last":0.60799999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.96699999999999997},"last":0.498},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.66200000000000003},"min":{"1min":0.53200000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86399999999999999,"5min":1.0049999999999999,"15min":1.01},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.80200000000000005},"min":{"1min":0.56299999999999994,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.2549999999999999,"5min":1.2609999999999999,"15min":1.2889999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81799999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1599999999999999,"5min":1.306,"15min":1.306},"last":0.623}]},{"osd":5,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80400000000000005,"15min":0.81599999999999995},"min":{"1min":0.56999999999999995,"5min":0.53800000000000003,"15min":0.502},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.56000000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.214,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.80600000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64900000000000002,"5min":0.64400000000000002,"15min":0.57799999999999996},"max":{"1min":1.196,"5min":1.282,"15min":1.288},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.84099999999999997,"15min":0.84699999999999998},"min":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.78800000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.85799999999999998,"15min":0.86699999999999999},"min":{"1min":0.68999999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3100000000000001,"5min":1.331,"15min":1.331},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.89500000000000002,"15min":0.89600000000000002},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.57899999999999996},"max":{"1min":1.246,"5min":1.286,"15min":1.286},"last":0.69599999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576007,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60416,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311424,"statfs":{"total":96636764160,"available":96574898176,"internally_reserved":0,"allocated":3129344,"data_stored":1525295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.66400000000000003,"15min":0.71499999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83899999999999997,"5min":0.95499999999999996,"15min":1.0509999999999999},"last":0.628},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.67500000000000004,"15min":0.72099999999999997},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.93600000000000005,"5min":0.99299999999999999,"15min":1.1459999999999999},"last":0.65800000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.68300000000000005,"15min":0.73699999999999999},"min":{"1min":0.436,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"last":0.749},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69199999999999995,"15min":0.73799999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91400000000000003,"5min":0.94899999999999995,"15min":1.002},"last":0.86899999999999999}]},{"osd":2,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.77900000000000003,"15min":0.82199999999999995},"min":{"1min":0.55400000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.004,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.76500000000000001,"15min":0.80600000000000005},"min":{"1min":0.503,"5min":0.496,"15min":0.496},"max":{"1min":1.0029999999999999,"5min":1.224,"15min":1.224},"last":0.80000000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71399999999999997,"15min":0.75900000000000001},"min":{"1min":0.54000000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.92600000000000005,"5min":1.0580000000000001,"15min":1.181},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.74399999999999999,"15min":0.76600000000000001},"min":{"1min":0.499,"5min":0.499,"15min":0.48699999999999999},"max":{"1min":0.95199999999999996,"5min":1.01,"15min":1.2490000000000001},"last":0.89200000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63800000000000001,"15min":0.625},"min":{"1min":0.45400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64500000000000002,"15min":0.65300000000000002},"min":{"1min":0.52000000000000002,"5min":0.41599999999999998,"15min":0.39700000000000002},"max":{"1min":0.81499999999999995,"5min":1.0920000000000001,"15min":1.169},"last":0.92200000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.50600000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":0.93100000000000005,"5min":1.056,"15min":1.0800000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.72199999999999998,"15min":0.71999999999999997},"min":{"1min":0.59199999999999997,"5min":0.48199999999999998,"15min":0.47699999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.94299999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73599999999999999,"15min":0.73299999999999998},"min":{"1min":0.60599999999999998,"5min":0.501,"15min":0.436},"max":{"1min":0.97999999999999998,"5min":1.0389999999999999,"15min":1.2669999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81200000000000006,"15min":0.82199999999999995},"min":{"1min":0.65900000000000003,"5min":0.58499999999999996,"15min":0.54400000000000004},"max":{"1min":1.03,"5min":1.1619999999999999,"15min":1.2969999999999999},"last":1.0880000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576006,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46456,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325384,"statfs":{"total":96636764160,"available":96589193216,"internally_reserved":0,"allocated":3121152,"data_stored":1530011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70699999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83899999999999997,"5min":0.94199999999999995,"15min":0.95999999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.73199999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90000000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.51800000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.75900000000000001,"15min":0.76500000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.73699999999999999,"15min":0.74199999999999999},"min":{"1min":0.52200000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.76800000000000002,"15min":0.75900000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.72899999999999998,"15min":0.748},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.88400000000000001,"5min":0.98499999999999999,"15min":1.0129999999999999},"last":0.62}]},{"osd":3,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.77900000000000003,"15min":0.77400000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.444,"5min":3.444,"15min":3.444},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.046},"last":0.73499999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.60599999999999998},"min":{"1min":0.44400000000000001,"5min":0.41799999999999998,"15min":0.38700000000000001},"max":{"1min":0.88,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.40600000000000003},"max":{"1min":0.84799999999999998,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.78100000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68799999999999994,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90300000000000002,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72099999999999997,"15min":0.69799999999999995},"min":{"1min":0.55900000000000005,"5min":0.51000000000000001,"15min":0.42499999999999999},"max":{"1min":0.83499999999999996,"5min":1.107,"15min":1.107},"last":0.745}]},{"osd":7,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.74199999999999999,"15min":0.72199999999999998},"min":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.42299999999999999},"max":{"1min":0.92200000000000004,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.79300000000000004,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.98199999999999998,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.76900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:32:37.852 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:32:38.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.090+0000 7fbc65400700 1 Processor -- start 2024-03-29T11:32:38.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.090+0000 7fbc65400700 1 -- start start 2024-03-29T11:32:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.091+0000 7fbc65400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc60160f10 0x7fbc600d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.091+0000 7fbc65400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbc600cf230 0x7fbc600d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.091+0000 7fbc65400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600ceda0 con 0x7fbc60161590 2024-03-29T11:32:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.091+0000 7fbc65400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc600d23d0 con 0x7fbc600cf230 2024-03-29T11:32:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.091+0000 7fbc65400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc601612f0 con 0x7fbc60160f10 2024-03-29T11:32:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.091+0000 7fbc5f200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbc60161590 0x7fbc600b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35850/0 (socket says 172.21.15.12:35850) 2024-03-29T11:32:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.091+0000 7fbc5f200700 1 -- 172.21.15.12:0/2834989367 learned_addr learned my addr 172.21.15.12:0/2834989367 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5de00700 1 --2- 172.21.15.12:0/2834989367 >> v2:172.21.15.159:3300/0 conn(0x7fbc600cf230 0x7fbc600d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5e800700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc60160f10 0x7fbc600d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308425915 0 0) 0x7fbc600ceda0 con 0x7fbc60161590 2024-03-29T11:32:38.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc440035b0 con 0x7fbc60161590 2024-03-29T11:32:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3563801219 0 0) 0x7fbc440035b0 con 0x7fbc60161590 2024-03-29T11:32:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc60160f10 msgr2=0x7fbc600d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc60160f10 0x7fbc600d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 >> v2:172.21.15.159:3300/0 conn(0x7fbc600cf230 msgr2=0x7fbc600d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 --2- 172.21.15.12:0/2834989367 >> v2:172.21.15.159:3300/0 conn(0x7fbc600cf230 0x7fbc600d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600ce470 con 0x7fbc60161590 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fbc580019f0 con 0x7fbc60161590 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 >> v1:172.21.15.12:6789/0 conn(0x7fbc60161590 legacy=0x7fbc600b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 0x7fbc44003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc44004850 0x7fbc44004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc60160f10 0x7fbc44005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600d23d0 con 0x7fbc44003a00 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc601612f0 con 0x7fbc44004850 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.092+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4400ba10 con 0x7fbc60160f10 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5e800700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 0x7fbc44003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5de00700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc44004850 0x7fbc44004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5f200700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc60160f10 0x7fbc44005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5e800700 1 -- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc60160f10 msgr2=0x7fbc44005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5e800700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc60160f10 0x7fbc44005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5e800700 1 -- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc44004850 msgr2=0x7fbc44004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5e800700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc44004850 0x7fbc44004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5e800700 1 -- 172.21.15.12:0/2834989367 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440035b0 con 0x7fbc44003a00 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5de00700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc44004850 0x7fbc44004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5e800700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 0x7fbc44003e20 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009b90 tx=0x7fbc44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4161ddbefac41fe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5000e6c0 con 0x7fbc44003a00 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5000e820 con 0x7fbc44003a00 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.093+0000 7fbc5d400700 1 -- 172.21.15.12:0/2834989367 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5000eb90 con 0x7fbc44003a00 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.094+0000 7fbc65400700 1 -- 172.21.15.12:0/2834989367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 msgr2=0x7fbc44003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.094+0000 7fbc65400700 1 --2- 172.21.15.12:0/2834989367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 0x7fbc44003e20 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009b90 tx=0x7fbc44005880 comp rx=0 tx=0).stop 2024-03-29T11:32:38.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.094+0000 7fbc5f200700 1 -- 172.21.15.12:0/2834989367 reap_dead start 2024-03-29T11:32:38.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.094+0000 7fbc65400700 1 -- 172.21.15.12:0/2834989367 shutdown_connections 2024-03-29T11:32:38.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.094+0000 7fbc65400700 1 -- 172.21.15.12:0/2834989367 >> 172.21.15.12:0/2834989367 conn(0x7fbc6008aff0 msgr2=0x7fbc600d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:38.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.094+0000 7fbc65400700 1 -- 172.21.15.12:0/2834989367 shutdown_connections 2024-03-29T11:32:38.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.094+0000 7fbc65400700 1 -- 172.21.15.12:0/2834989367 wait complete. 2024-03-29T11:32:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.096+0000 7fbc65400700 1 Processor -- start 2024-03-29T11:32:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.096+0000 7fbc65400700 1 -- start start 2024-03-29T11:32:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc65400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 0x7fbc601f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc65400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600b8920 0x7fbc600b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc65400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc600b8da0 0x7fbc600b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc65400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60158110 con 0x7fbc44003a00 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc65400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc601f3fa0 con 0x7fbc600b8da0 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc65400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600b6aa0 con 0x7fbc600b8920 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 0x7fbc601f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 0x7fbc601f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34992/0 (socket says 172.21.15.12:34992) 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5e800700 1 -- 172.21.15.12:0/1342582168 learned_addr learned my addr 172.21.15.12:0/1342582168 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5de00700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600b8920 0x7fbc600b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5f200700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc600b8da0 0x7fbc600b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5e800700 1 -- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600b8920 msgr2=0x7fbc600b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5e800700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600b8920 0x7fbc600b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5e800700 1 -- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc600b8da0 msgr2=0x7fbc600b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5e800700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc600b8da0 0x7fbc600b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5e800700 1 -- 172.21.15.12:0/1342582168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600b6520 con 0x7fbc44003a00 2024-03-29T11:32:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.097+0000 7fbc5de00700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600b8920 0x7fbc600b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.098+0000 7fbc5e800700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 0x7fbc601f3a60 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fbc44004740 tx=0x7fbc500047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.098+0000 7fbc56a00700 1 -- 172.21.15.12:0/1342582168 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5000e6c0 con 0x7fbc44003a00 2024-03-29T11:32:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.098+0000 7fbc56a00700 1 -- 172.21.15.12:0/1342582168 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc50003ad0 con 0x7fbc44003a00 2024-03-29T11:32:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.098+0000 7fbc56a00700 1 -- 172.21.15.12:0/1342582168 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50003e80 con 0x7fbc44003a00 2024-03-29T11:32:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.098+0000 7fbc65400700 1 -- 172.21.15.12:0/1342582168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc600b9320 con 0x7fbc44003a00 2024-03-29T11:32:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.098+0000 7fbc65400700 1 -- 172.21.15.12:0/1342582168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc60151eb0 con 0x7fbc44003a00 2024-03-29T11:32:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.099+0000 7fbc65400700 1 -- 172.21.15.12:0/1342582168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc60067720 con 0x7fbc44003a00 2024-03-29T11:32:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.100+0000 7fbc56a00700 1 -- 172.21.15.12:0/1342582168 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fbc50013470 con 0x7fbc44003a00 2024-03-29T11:32:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.101+0000 7fbc56a00700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006aed0 0x7fbc4006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.102+0000 7fbc56a00700 1 -- 172.21.15.12:0/1342582168 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fbc5008b1d0 con 0x7fbc44003a00 2024-03-29T11:32:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.102+0000 7fbc5de00700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006aed0 0x7fbc4006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.103+0000 7fbc5de00700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006aed0 0x7fbc4006d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbc60151bc0 tx=0x7fbc4c008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:38.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.106+0000 7fbc56a00700 1 -- 172.21.15.12:0/1342582168 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc50056110 con 0x7fbc44003a00 2024-03-29T11:32:38.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.315+0000 7fbc65400700 1 -- 172.21.15.12:0/1342582168 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc600cf6b0 con 0x7fbc4006aed0 2024-03-29T11:32:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.324+0000 7fbc56a00700 1 -- 172.21.15.12:0/1342582168 <== mgr.11050 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289505 (secure 0 0 0) 0x7fbc4c0138b0 con 0x7fbc4006aed0 2024-03-29T11:32:38.324 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 -- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006aed0 msgr2=0x7fbc4006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006aed0 0x7fbc4006d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbc60151bc0 tx=0x7fbc4c008040 comp rx=0 tx=0).stop 2024-03-29T11:32:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 -- 172.21.15.12:0/1342582168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 msgr2=0x7fbc601f3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 0x7fbc601f3a60 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fbc44004740 tx=0x7fbc500047c0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 -- 172.21.15.12:0/1342582168 shutdown_connections 2024-03-29T11:32:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc4006aed0 0x7fbc4006d390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc600b8da0 0x7fbc600b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc44003a00 0x7fbc601f3a60 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 --2- 172.21.15.12:0/1342582168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc600b8920 0x7fbc600b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 -- 172.21.15.12:0/1342582168 >> 172.21.15.12:0/1342582168 conn(0x7fbc6008aff0 msgr2=0x7fbc60152650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 -- 172.21.15.12:0/1342582168 shutdown_connections 2024-03-29T11:32:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.329+0000 7fbc65400700 1 -- 172.21.15.12:0/1342582168 wait complete. 2024-03-29T11:32:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:32:38.343 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-03-29T11:32:36.529073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":695,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":433604,"kb_used_data":25520,"kb_used_omap":12,"kb_used_meta":407923,"kb_avail":754541116,"statfs":{"total":773094113280,"available":772650102784,"internally_reserved":0,"allocated":26132480,"data_stored":13379925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":417713721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.132860"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426029+0000","last_change":"2024-03-29T11:32:32.953487+0000","last_active":"2024-03-29T11:32:34.426029+0000","last_peered":"2024-03-29T11:32:34.426029+0000","last_clean":"2024-03-29T11:32:34.426029+0000","last_became_active":"2024-03-29T11:32:32.953243+0000","last_became_peered":"2024-03-29T11:32:32.953243+0000","last_unstale":"2024-03-29T11:32:34.426029+0000","last_undegraded":"2024-03-29T11:32:34.426029+0000","last_fullsized":"2024-03-29T11:32:34.426029+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:11.369063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426361+0000","last_change":"2024-03-29T11:32:32.956013+0000","last_active":"2024-03-29T11:32:34.426361+0000","last_peered":"2024-03-29T11:32:34.426361+0000","last_clean":"2024-03-29T11:32:34.426361+0000","last_became_active":"2024-03-29T11:32:32.954012+0000","last_became_peered":"2024-03-29T11:32:32.954012+0000","last_unstale":"2024-03-29T11:32:34.426361+0000","last_undegraded":"2024-03-29T11:32:34.426361+0000","last_fullsized":"2024-03-29T11:32:34.426361+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:02:36.221843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426669+0000","last_change":"2024-03-29T11:32:32.966647+0000","last_active":"2024-03-29T11:32:34.426669+0000","last_peered":"2024-03-29T11:32:34.426669+0000","last_clean":"2024-03-29T11:32:34.426669+0000","last_became_active":"2024-03-29T11:32:32.966394+0000","last_became_peered":"2024-03-29T11:32:32.966394+0000","last_unstale":"2024-03-29T11:32:34.426669+0000","last_undegraded":"2024-03-29T11:32:34.426669+0000","last_fullsized":"2024-03-29T11:32:34.426669+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:22.600133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425470+0000","last_change":"2024-03-29T11:32:32.955745+0000","last_active":"2024-03-29T11:32:34.425470+0000","last_peered":"2024-03-29T11:32:34.425470+0000","last_clean":"2024-03-29T11:32:34.425470+0000","last_became_active":"2024-03-29T11:32:32.955142+0000","last_became_peered":"2024-03-29T11:32:32.955142+0000","last_unstale":"2024-03-29T11:32:34.425470+0000","last_undegraded":"2024-03-29T11:32:34.425470+0000","last_fullsized":"2024-03-29T11:32:34.425470+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:42.190121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.946994+0000","last_change":"2024-03-29T11:32:32.950908+0000","last_active":"2024-03-29T11:32:33.946994+0000","last_peered":"2024-03-29T11:32:33.946994+0000","last_clean":"2024-03-29T11:32:33.946994+0000","last_became_active":"2024-03-29T11:32:32.950581+0000","last_became_peered":"2024-03-29T11:32:32.950581+0000","last_unstale":"2024-03-29T11:32:33.946994+0000","last_undegraded":"2024-03-29T11:32:33.946994+0000","last_fullsized":"2024-03-29T11:32:33.946994+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:11.495727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425397+0000","last_change":"2024-03-29T11:32:32.966829+0000","last_active":"2024-03-29T11:32:34.425397+0000","last_peered":"2024-03-29T11:32:34.425397+0000","last_clean":"2024-03-29T11:32:34.425397+0000","last_became_active":"2024-03-29T11:32:32.966561+0000","last_became_peered":"2024-03-29T11:32:32.966561+0000","last_unstale":"2024-03-29T11:32:34.425397+0000","last_undegraded":"2024-03-29T11:32:34.425397+0000","last_fullsized":"2024-03-29T11:32:34.425397+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947638+0000","last_change":"2024-03-29T11:32:32.956044+0000","last_active":"2024-03-29T11:32:33.947638+0000","last_peered":"2024-03-29T11:32:33.947638+0000","last_clean":"2024-03-29T11:32:33.947638+0000","last_became_active":"2024-03-29T11:32:32.955298+0000","last_became_peered":"2024-03-29T11:32:32.955298+0000","last_unstale":"2024-03-29T11:32:33.947638+0000","last_undegraded":"2024-03-29T11:32:33.947638+0000","last_fullsized":"2024-03-29T11:32:33.947638+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:36.536991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426215+0000","last_change":"2024-03-29T11:32:32.956054+0000","last_active":"2024-03-29T11:32:34.426215+0000","last_peered":"2024-03-29T11:32:34.426215+0000","last_clean":"2024-03-29T11:32:34.426215+0000","last_became_active":"2024-03-29T11:32:32.952773+0000","last_became_peered":"2024-03-29T11:32:32.952773+0000","last_unstale":"2024-03-29T11:32:34.426215+0000","last_undegraded":"2024-03-29T11:32:34.426215+0000","last_fullsized":"2024-03-29T11:32:34.426215+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:59:43.666251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425918+0000","last_change":"2024-03-29T11:32:32.963463+0000","last_active":"2024-03-29T11:32:34.425918+0000","last_peered":"2024-03-29T11:32:34.425918+0000","last_clean":"2024-03-29T11:32:34.425918+0000","last_became_active":"2024-03-29T11:32:32.962970+0000","last_became_peered":"2024-03-29T11:32:32.962970+0000","last_unstale":"2024-03-29T11:32:34.425918+0000","last_undegraded":"2024-03-29T11:32:34.425918+0000","last_fullsized":"2024-03-29T11:32:34.425918+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:14.637128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948201+0000","last_change":"2024-03-29T11:32:32.953703+0000","last_active":"2024-03-29T11:32:33.948201+0000","last_peered":"2024-03-29T11:32:33.948201+0000","last_clean":"2024-03-29T11:32:33.948201+0000","last_became_active":"2024-03-29T11:32:32.952942+0000","last_became_peered":"2024-03-29T11:32:32.952942+0000","last_unstale":"2024-03-29T11:32:33.948201+0000","last_undegraded":"2024-03-29T11:32:33.948201+0000","last_fullsized":"2024-03-29T11:32:33.948201+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:23:44.759018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947820+0000","last_change":"2024-03-29T11:32:32.970743+0000","last_active":"2024-03-29T11:32:33.947820+0000","last_peered":"2024-03-29T11:32:33.947820+0000","last_clean":"2024-03-29T11:32:33.947820+0000","last_became_active":"2024-03-29T11:32:32.967319+0000","last_became_peered":"2024-03-29T11:32:32.967319+0000","last_unstale":"2024-03-29T11:32:33.947820+0000","last_undegraded":"2024-03-29T11:32:33.947820+0000","last_fullsized":"2024-03-29T11:32:33.947820+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:52.668591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426478+0000","last_change":"2024-03-29T11:32:32.957788+0000","last_active":"2024-03-29T11:32:34.426478+0000","last_peered":"2024-03-29T11:32:34.426478+0000","last_clean":"2024-03-29T11:32:34.426478+0000","last_became_active":"2024-03-29T11:32:32.957519+0000","last_became_peered":"2024-03-29T11:32:32.957519+0000","last_unstale":"2024-03-29T11:32:34.426478+0000","last_undegraded":"2024-03-29T11:32:34.426478+0000","last_fullsized":"2024-03-29T11:32:34.426478+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:21.664133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948137+0000","last_change":"2024-03-29T11:32:32.967536+0000","last_active":"2024-03-29T11:32:33.948137+0000","last_peered":"2024-03-29T11:32:33.948137+0000","last_clean":"2024-03-29T11:32:33.948137+0000","last_became_active":"2024-03-29T11:32:32.966686+0000","last_became_peered":"2024-03-29T11:32:32.966686+0000","last_unstale":"2024-03-29T11:32:33.948137+0000","last_undegraded":"2024-03-29T11:32:33.948137+0000","last_fullsized":"2024-03-29T11:32:33.948137+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:24.736585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426215+0000","last_change":"2024-03-29T11:32:32.952674+0000","last_active":"2024-03-29T11:32:34.426215+0000","last_peered":"2024-03-29T11:32:34.426215+0000","last_clean":"2024-03-29T11:32:34.426215+0000","last_became_active":"2024-03-29T11:32:32.952387+0000","last_became_peered":"2024-03-29T11:32:32.952387+0000","last_unstale":"2024-03-29T11:32:34.426215+0000","last_undegraded":"2024-03-29T11:32:34.426215+0000","last_fullsized":"2024-03-29T11:32:34.426215+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:30.121534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425648+0000","last_change":"2024-03-29T11:32:32.966028+0000","last_active":"2024-03-29T11:32:34.425648+0000","last_peered":"2024-03-29T11:32:34.425648+0000","last_clean":"2024-03-29T11:32:34.425648+0000","last_became_active":"2024-03-29T11:32:32.965642+0000","last_became_peered":"2024-03-29T11:32:32.965642+0000","last_unstale":"2024-03-29T11:32:34.425648+0000","last_undegraded":"2024-03-29T11:32:34.425648+0000","last_fullsized":"2024-03-29T11:32:34.425648+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:28:31.977713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425206+0000","last_change":"2024-03-29T11:32:32.960337+0000","last_active":"2024-03-29T11:32:34.425206+0000","last_peered":"2024-03-29T11:32:34.425206+0000","last_clean":"2024-03-29T11:32:34.425206+0000","last_became_active":"2024-03-29T11:32:32.958824+0000","last_became_peered":"2024-03-29T11:32:32.958824+0000","last_unstale":"2024-03-29T11:32:34.425206+0000","last_undegraded":"2024-03-29T11:32:34.425206+0000","last_fullsized":"2024-03-29T11:32:34.425206+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:48:40.247306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426632+0000","last_change":"2024-03-29T11:32:32.960497+0000","last_active":"2024-03-29T11:32:34.426632+0000","last_peered":"2024-03-29T11:32:34.426632+0000","last_clean":"2024-03-29T11:32:34.426632+0000","last_became_active":"2024-03-29T11:32:32.958333+0000","last_became_peered":"2024-03-29T11:32:32.958333+0000","last_unstale":"2024-03-29T11:32:34.426632+0000","last_undegraded":"2024-03-29T11:32:34.426632+0000","last_fullsized":"2024-03-29T11:32:34.426632+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:27:28.513987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426366+0000","last_change":"2024-03-29T11:32:32.949294+0000","last_active":"2024-03-29T11:32:34.426366+0000","last_peered":"2024-03-29T11:32:34.426366+0000","last_clean":"2024-03-29T11:32:34.426366+0000","last_became_active":"2024-03-29T11:32:32.949025+0000","last_became_peered":"2024-03-29T11:32:32.949025+0000","last_unstale":"2024-03-29T11:32:34.426366+0000","last_undegraded":"2024-03-29T11:32:34.426366+0000","last_fullsized":"2024-03-29T11:32:34.426366+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426685+0000","last_change":"2024-03-29T11:32:32.965153+0000","last_active":"2024-03-29T11:32:34.426685+0000","last_peered":"2024-03-29T11:32:34.426685+0000","last_clean":"2024-03-29T11:32:34.426685+0000","last_became_active":"2024-03-29T11:32:32.964935+0000","last_became_peered":"2024-03-29T11:32:32.964935+0000","last_unstale":"2024-03-29T11:32:34.426685+0000","last_undegraded":"2024-03-29T11:32:34.426685+0000","last_fullsized":"2024-03-29T11:32:34.426685+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:02.769642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426529+0000","last_change":"2024-03-29T11:32:32.953083+0000","last_active":"2024-03-29T11:32:34.426529+0000","last_peered":"2024-03-29T11:32:34.426529+0000","last_clean":"2024-03-29T11:32:34.426529+0000","last_became_active":"2024-03-29T11:32:32.952823+0000","last_became_peered":"2024-03-29T11:32:32.952823+0000","last_unstale":"2024-03-29T11:32:34.426529+0000","last_undegraded":"2024-03-29T11:32:34.426529+0000","last_fullsized":"2024-03-29T11:32:34.426529+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:09.626524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426292+0000","last_change":"2024-03-29T11:32:32.954755+0000","last_active":"2024-03-29T11:32:34.426292+0000","last_peered":"2024-03-29T11:32:34.426292+0000","last_clean":"2024-03-29T11:32:34.426292+0000","last_became_active":"2024-03-29T11:32:32.954467+0000","last_became_peered":"2024-03-29T11:32:32.954467+0000","last_unstale":"2024-03-29T11:32:34.426292+0000","last_undegraded":"2024-03-29T11:32:34.426292+0000","last_fullsized":"2024-03-29T11:32:34.426292+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:10.303092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947741+0000","last_change":"2024-03-29T11:32:32.955216+0000","last_active":"2024-03-29T11:32:33.947741+0000","last_peered":"2024-03-29T11:32:33.947741+0000","last_clean":"2024-03-29T11:32:33.947741+0000","last_became_active":"2024-03-29T11:32:32.954774+0000","last_became_peered":"2024-03-29T11:32:32.954774+0000","last_unstale":"2024-03-29T11:32:33.947741+0000","last_undegraded":"2024-03-29T11:32:33.947741+0000","last_fullsized":"2024-03-29T11:32:33.947741+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:44.576542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426250+0000","last_change":"2024-03-29T11:32:32.964085+0000","last_active":"2024-03-29T11:32:34.426250+0000","last_peered":"2024-03-29T11:32:34.426250+0000","last_clean":"2024-03-29T11:32:34.426250+0000","last_became_active":"2024-03-29T11:32:32.963753+0000","last_became_peered":"2024-03-29T11:32:32.963753+0000","last_unstale":"2024-03-29T11:32:34.426250+0000","last_undegraded":"2024-03-29T11:32:34.426250+0000","last_fullsized":"2024-03-29T11:32:34.426250+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:34.956119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425416+0000","last_change":"2024-03-29T11:32:32.957520+0000","last_active":"2024-03-29T11:32:34.425416+0000","last_peered":"2024-03-29T11:32:34.425416+0000","last_clean":"2024-03-29T11:32:34.425416+0000","last_became_active":"2024-03-29T11:32:32.957245+0000","last_became_peered":"2024-03-29T11:32:32.957245+0000","last_unstale":"2024-03-29T11:32:34.425416+0000","last_undegraded":"2024-03-29T11:32:34.425416+0000","last_fullsized":"2024-03-29T11:32:34.425416+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:45.368395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426117+0000","last_change":"2024-03-29T11:32:32.955473+0000","last_active":"2024-03-29T11:32:34.426117+0000","last_peered":"2024-03-29T11:32:34.426117+0000","last_clean":"2024-03-29T11:32:34.426117+0000","last_became_active":"2024-03-29T11:32:32.953854+0000","last_became_peered":"2024-03-29T11:32:32.953854+0000","last_unstale":"2024-03-29T11:32:34.426117+0000","last_undegraded":"2024-03-29T11:32:34.426117+0000","last_fullsized":"2024-03-29T11:32:34.426117+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:21.819962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948298+0000","last_change":"2024-03-29T11:32:32.967040+0000","last_active":"2024-03-29T11:32:33.948298+0000","last_peered":"2024-03-29T11:32:33.948298+0000","last_clean":"2024-03-29T11:32:33.948298+0000","last_became_active":"2024-03-29T11:32:32.966729+0000","last_became_peered":"2024-03-29T11:32:32.966729+0000","last_unstale":"2024-03-29T11:32:33.948298+0000","last_undegraded":"2024-03-29T11:32:33.948298+0000","last_fullsized":"2024-03-29T11:32:33.948298+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:27.515858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947925+0000","last_change":"2024-03-29T11:32:32.970545+0000","last_active":"2024-03-29T11:32:33.947925+0000","last_peered":"2024-03-29T11:32:33.947925+0000","last_clean":"2024-03-29T11:32:33.947925+0000","last_became_active":"2024-03-29T11:32:32.966917+0000","last_became_peered":"2024-03-29T11:32:32.966917+0000","last_unstale":"2024-03-29T11:32:33.947925+0000","last_undegraded":"2024-03-29T11:32:33.947925+0000","last_fullsized":"2024-03-29T11:32:33.947925+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:11.611059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426064+0000","last_change":"2024-03-29T11:32:32.950772+0000","last_active":"2024-03-29T11:32:34.426064+0000","last_peered":"2024-03-29T11:32:34.426064+0000","last_clean":"2024-03-29T11:32:34.426064+0000","last_became_active":"2024-03-29T11:32:32.950518+0000","last_became_peered":"2024-03-29T11:32:32.950518+0000","last_unstale":"2024-03-29T11:32:34.426064+0000","last_undegraded":"2024-03-29T11:32:34.426064+0000","last_fullsized":"2024-03-29T11:32:34.426064+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:34.956119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947739+0000","last_change":"2024-03-29T11:32:32.956094+0000","last_active":"2024-03-29T11:32:33.947739+0000","last_peered":"2024-03-29T11:32:33.947739+0000","last_clean":"2024-03-29T11:32:33.947739+0000","last_became_active":"2024-03-29T11:32:32.955788+0000","last_became_peered":"2024-03-29T11:32:32.955788+0000","last_unstale":"2024-03-29T11:32:33.947739+0000","last_undegraded":"2024-03-29T11:32:33.947739+0000","last_fullsized":"2024-03-29T11:32:33.947739+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:05.005123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947552+0000","last_change":"2024-03-29T11:32:32.951190+0000","last_active":"2024-03-29T11:32:33.947552+0000","last_peered":"2024-03-29T11:32:33.947552+0000","last_clean":"2024-03-29T11:32:33.947552+0000","last_became_active":"2024-03-29T11:32:32.950830+0000","last_became_peered":"2024-03-29T11:32:32.950830+0000","last_unstale":"2024-03-29T11:32:33.947552+0000","last_undegraded":"2024-03-29T11:32:33.947552+0000","last_fullsized":"2024-03-29T11:32:33.947552+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:12.191813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425220+0000","last_change":"2024-03-29T11:32:32.949171+0000","last_active":"2024-03-29T11:32:34.425220+0000","last_peered":"2024-03-29T11:32:34.425220+0000","last_clean":"2024-03-29T11:32:34.425220+0000","last_became_active":"2024-03-29T11:32:32.948898+0000","last_became_peered":"2024-03-29T11:32:32.948898+0000","last_unstale":"2024-03-29T11:32:34.425220+0000","last_undegraded":"2024-03-29T11:32:34.425220+0000","last_fullsized":"2024-03-29T11:32:34.425220+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:02:36.221843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425655+0000","last_change":"2024-03-29T11:32:32.956902+0000","last_active":"2024-03-29T11:32:34.425655+0000","last_peered":"2024-03-29T11:32:34.425655+0000","last_clean":"2024-03-29T11:32:34.425655+0000","last_became_active":"2024-03-29T11:32:32.955472+0000","last_became_peered":"2024-03-29T11:32:32.955472+0000","last_unstale":"2024-03-29T11:32:34.425655+0000","last_undegraded":"2024-03-29T11:32:34.425655+0000","last_fullsized":"2024-03-29T11:32:34.425655+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:29.628408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426593+0000","last_change":"2024-03-29T11:32:32.959493+0000","last_active":"2024-03-29T11:32:34.426593+0000","last_peered":"2024-03-29T11:32:34.426593+0000","last_clean":"2024-03-29T11:32:34.426593+0000","last_became_active":"2024-03-29T11:32:32.959199+0000","last_became_peered":"2024-03-29T11:32:32.959199+0000","last_unstale":"2024-03-29T11:32:34.426593+0000","last_undegraded":"2024-03-29T11:32:34.426593+0000","last_fullsized":"2024-03-29T11:32:34.426593+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:56.148657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"247'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948400+0000","last_change":"2024-03-29T11:32:32.419758+0000","last_active":"2024-03-29T11:32:33.948400+0000","last_peered":"2024-03-29T11:32:33.948400+0000","last_clean":"2024-03-29T11:32:33.948400+0000","last_became_active":"2024-03-29T11:32:32.419044+0000","last_became_peered":"2024-03-29T11:32:32.419044+0000","last_unstale":"2024-03-29T11:32:33.948400+0000","last_undegraded":"2024-03-29T11:32:33.948400+0000","last_fullsized":"2024-03-29T11:32:33.948400+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:29.373413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947625+0000","last_change":"2024-03-29T11:32:32.964445+0000","last_active":"2024-03-29T11:32:33.947625+0000","last_peered":"2024-03-29T11:32:33.947625+0000","last_clean":"2024-03-29T11:32:33.947625+0000","last_became_active":"2024-03-29T11:32:32.963758+0000","last_became_peered":"2024-03-29T11:32:32.963758+0000","last_unstale":"2024-03-29T11:32:33.947625+0000","last_undegraded":"2024-03-29T11:32:33.947625+0000","last_fullsized":"2024-03-29T11:32:33.947625+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:24.556045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426574+0000","last_change":"2024-03-29T11:32:32.419690+0000","last_active":"2024-03-29T11:32:34.426574+0000","last_peered":"2024-03-29T11:32:34.426574+0000","last_clean":"2024-03-29T11:32:34.426574+0000","last_became_active":"2024-03-29T11:32:32.418508+0000","last_became_peered":"2024-03-29T11:32:32.418508+0000","last_unstale":"2024-03-29T11:32:34.426574+0000","last_undegraded":"2024-03-29T11:32:34.426574+0000","last_fullsized":"2024-03-29T11:32:34.426574+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:34.818177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426719+0000","last_change":"2024-03-29T11:32:32.960733+0000","last_active":"2024-03-29T11:32:34.426719+0000","last_peered":"2024-03-29T11:32:34.426719+0000","last_clean":"2024-03-29T11:32:34.426719+0000","last_became_active":"2024-03-29T11:32:32.959254+0000","last_became_peered":"2024-03-29T11:32:32.959254+0000","last_unstale":"2024-03-29T11:32:34.426719+0000","last_undegraded":"2024-03-29T11:32:34.426719+0000","last_fullsized":"2024-03-29T11:32:34.426719+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:45:42.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426629+0000","last_change":"2024-03-29T11:32:32.422622+0000","last_active":"2024-03-29T11:32:34.426629+0000","last_peered":"2024-03-29T11:32:34.426629+0000","last_clean":"2024-03-29T11:32:34.426629+0000","last_became_active":"2024-03-29T11:32:32.421864+0000","last_became_peered":"2024-03-29T11:32:32.421864+0000","last_unstale":"2024-03-29T11:32:34.426629+0000","last_undegraded":"2024-03-29T11:32:34.426629+0000","last_fullsized":"2024-03-29T11:32:34.426629+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:23.546712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947481+0000","last_change":"2024-03-29T11:32:32.965844+0000","last_active":"2024-03-29T11:32:33.947481+0000","last_peered":"2024-03-29T11:32:33.947481+0000","last_clean":"2024-03-29T11:32:33.947481+0000","last_became_active":"2024-03-29T11:32:32.965511+0000","last_became_peered":"2024-03-29T11:32:32.965511+0000","last_unstale":"2024-03-29T11:32:33.947481+0000","last_undegraded":"2024-03-29T11:32:33.947481+0000","last_fullsized":"2024-03-29T11:32:33.947481+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:23.043448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"247'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425803+0000","last_change":"2024-03-29T11:32:32.419740+0000","last_active":"2024-03-29T11:32:34.425803+0000","last_peered":"2024-03-29T11:32:34.425803+0000","last_clean":"2024-03-29T11:32:34.425803+0000","last_became_active":"2024-03-29T11:32:32.419471+0000","last_became_peered":"2024-03-29T11:32:32.419471+0000","last_unstale":"2024-03-29T11:32:34.425803+0000","last_undegraded":"2024-03-29T11:32:34.425803+0000","last_fullsized":"2024-03-29T11:32:34.425803+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:01.760308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425793+0000","last_change":"2024-03-29T11:32:32.963417+0000","last_active":"2024-03-29T11:32:34.425793+0000","last_peered":"2024-03-29T11:32:34.425793+0000","last_clean":"2024-03-29T11:32:34.425793+0000","last_became_active":"2024-03-29T11:32:32.963109+0000","last_became_peered":"2024-03-29T11:32:32.963109+0000","last_unstale":"2024-03-29T11:32:34.425793+0000","last_undegraded":"2024-03-29T11:32:34.425793+0000","last_fullsized":"2024-03-29T11:32:34.425793+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:24:49.870800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"247'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426294+0000","last_change":"2024-03-29T11:32:32.417496+0000","last_active":"2024-03-29T11:32:34.426294+0000","last_peered":"2024-03-29T11:32:34.426294+0000","last_clean":"2024-03-29T11:32:34.426294+0000","last_became_active":"2024-03-29T11:32:32.417165+0000","last_became_peered":"2024-03-29T11:32:32.417165+0000","last_unstale":"2024-03-29T11:32:34.426294+0000","last_undegraded":"2024-03-29T11:32:34.426294+0000","last_fullsized":"2024-03-29T11:32:34.426294+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:57.952240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425479+0000","last_change":"2024-03-29T11:32:32.954909+0000","last_active":"2024-03-29T11:32:34.425479+0000","last_peered":"2024-03-29T11:32:34.425479+0000","last_clean":"2024-03-29T11:32:34.425479+0000","last_became_active":"2024-03-29T11:32:32.954609+0000","last_became_peered":"2024-03-29T11:32:32.954609+0000","last_unstale":"2024-03-29T11:32:34.425479+0000","last_undegraded":"2024-03-29T11:32:34.425479+0000","last_fullsized":"2024-03-29T11:32:34.425479+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:21:04.890763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947743+0000","last_change":"2024-03-29T11:32:32.424318+0000","last_active":"2024-03-29T11:32:33.947743+0000","last_peered":"2024-03-29T11:32:33.947743+0000","last_clean":"2024-03-29T11:32:33.947743+0000","last_became_active":"2024-03-29T11:32:32.424027+0000","last_became_peered":"2024-03-29T11:32:32.424027+0000","last_unstale":"2024-03-29T11:32:33.947743+0000","last_undegraded":"2024-03-29T11:32:33.947743+0000","last_fullsized":"2024-03-29T11:32:33.947743+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:20.654799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425702+0000","last_change":"2024-03-29T11:32:32.950803+0000","last_active":"2024-03-29T11:32:34.425702+0000","last_peered":"2024-03-29T11:32:34.425702+0000","last_clean":"2024-03-29T11:32:34.425702+0000","last_became_active":"2024-03-29T11:32:32.950515+0000","last_became_peered":"2024-03-29T11:32:32.950515+0000","last_unstale":"2024-03-29T11:32:34.425702+0000","last_undegraded":"2024-03-29T11:32:34.425702+0000","last_fullsized":"2024-03-29T11:32:34.425702+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:42:17.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425113+0000","last_change":"2024-03-29T11:32:32.419260+0000","last_active":"2024-03-29T11:32:34.425113+0000","last_peered":"2024-03-29T11:32:34.425113+0000","last_clean":"2024-03-29T11:32:34.425113+0000","last_became_active":"2024-03-29T11:32:32.418975+0000","last_became_peered":"2024-03-29T11:32:32.418975+0000","last_unstale":"2024-03-29T11:32:34.425113+0000","last_undegraded":"2024-03-29T11:32:34.425113+0000","last_fullsized":"2024-03-29T11:32:34.425113+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:41.904116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426465+0000","last_change":"2024-03-29T11:32:32.949825+0000","last_active":"2024-03-29T11:32:34.426465+0000","last_peered":"2024-03-29T11:32:34.426465+0000","last_clean":"2024-03-29T11:32:34.426465+0000","last_became_active":"2024-03-29T11:32:32.949562+0000","last_became_peered":"2024-03-29T11:32:32.949562+0000","last_unstale":"2024-03-29T11:32:34.426465+0000","last_undegraded":"2024-03-29T11:32:34.426465+0000","last_fullsized":"2024-03-29T11:32:34.426465+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:23.557124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426747+0000","last_change":"2024-03-29T11:32:32.426409+0000","last_active":"2024-03-29T11:32:34.426747+0000","last_peered":"2024-03-29T11:32:34.426747+0000","last_clean":"2024-03-29T11:32:34.426747+0000","last_became_active":"2024-03-29T11:32:32.426142+0000","last_became_peered":"2024-03-29T11:32:32.426142+0000","last_unstale":"2024-03-29T11:32:34.426747+0000","last_undegraded":"2024-03-29T11:32:34.426747+0000","last_fullsized":"2024-03-29T11:32:34.426747+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:51:51.755610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948181+0000","last_change":"2024-03-29T11:32:32.974753+0000","last_active":"2024-03-29T11:32:33.948181+0000","last_peered":"2024-03-29T11:32:33.948181+0000","last_clean":"2024-03-29T11:32:33.948181+0000","last_became_active":"2024-03-29T11:32:32.971332+0000","last_became_peered":"2024-03-29T11:32:32.971332+0000","last_unstale":"2024-03-29T11:32:33.948181+0000","last_undegraded":"2024-03-29T11:32:33.948181+0000","last_fullsized":"2024-03-29T11:32:33.948181+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:33.815472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426620+0000","last_change":"2024-03-29T11:32:32.420533+0000","last_active":"2024-03-29T11:32:34.426620+0000","last_peered":"2024-03-29T11:32:34.426620+0000","last_clean":"2024-03-29T11:32:34.426620+0000","last_became_active":"2024-03-29T11:32:32.420284+0000","last_became_peered":"2024-03-29T11:32:32.420284+0000","last_unstale":"2024-03-29T11:32:34.426620+0000","last_undegraded":"2024-03-29T11:32:34.426620+0000","last_fullsized":"2024-03-29T11:32:34.426620+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:31.173444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426544+0000","last_change":"2024-03-29T11:32:32.969812+0000","last_active":"2024-03-29T11:32:34.426544+0000","last_peered":"2024-03-29T11:32:34.426544+0000","last_clean":"2024-03-29T11:32:34.426544+0000","last_became_active":"2024-03-29T11:32:32.969479+0000","last_became_peered":"2024-03-29T11:32:32.969479+0000","last_unstale":"2024-03-29T11:32:34.426544+0000","last_undegraded":"2024-03-29T11:32:34.426544+0000","last_fullsized":"2024-03-29T11:32:34.426544+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:57.232582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"247'3","reported_seq":14,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426228+0000","last_change":"2024-03-29T11:32:32.418264+0000","last_active":"2024-03-29T11:32:34.426228+0000","last_peered":"2024-03-29T11:32:34.426228+0000","last_clean":"2024-03-29T11:32:34.426228+0000","last_became_active":"2024-03-29T11:32:32.417013+0000","last_became_peered":"2024-03-29T11:32:32.417013+0000","last_unstale":"2024-03-29T11:32:34.426228+0000","last_undegraded":"2024-03-29T11:32:34.426228+0000","last_fullsized":"2024-03-29T11:32:34.426228+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:51.960387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948106+0000","last_change":"2024-03-29T11:32:32.955257+0000","last_active":"2024-03-29T11:32:33.948106+0000","last_peered":"2024-03-29T11:32:33.948106+0000","last_clean":"2024-03-29T11:32:33.948106+0000","last_became_active":"2024-03-29T11:32:32.952017+0000","last_became_peered":"2024-03-29T11:32:32.952017+0000","last_unstale":"2024-03-29T11:32:33.948106+0000","last_undegraded":"2024-03-29T11:32:33.948106+0000","last_fullsized":"2024-03-29T11:32:33.948106+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:45.368395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"247'6","reported_seq":17,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426158+0000","last_change":"2024-03-29T11:32:32.418422+0000","last_active":"2024-03-29T11:32:34.426158+0000","last_peered":"2024-03-29T11:32:34.426158+0000","last_clean":"2024-03-29T11:32:34.426158+0000","last_became_active":"2024-03-29T11:32:32.418119+0000","last_became_peered":"2024-03-29T11:32:32.418119+0000","last_unstale":"2024-03-29T11:32:34.426158+0000","last_undegraded":"2024-03-29T11:32:34.426158+0000","last_fullsized":"2024-03-29T11:32:34.426158+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:34.164745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947987+0000","last_change":"2024-03-29T11:32:32.955062+0000","last_active":"2024-03-29T11:32:33.947987+0000","last_peered":"2024-03-29T11:32:33.947987+0000","last_clean":"2024-03-29T11:32:33.947987+0000","last_became_active":"2024-03-29T11:32:32.954362+0000","last_became_peered":"2024-03-29T11:32:32.954362+0000","last_unstale":"2024-03-29T11:32:33.947987+0000","last_undegraded":"2024-03-29T11:32:33.947987+0000","last_fullsized":"2024-03-29T11:32:33.947987+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:35:32.510118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425947+0000","last_change":"2024-03-29T11:32:32.427566+0000","last_active":"2024-03-29T11:32:34.425947+0000","last_peered":"2024-03-29T11:32:34.425947+0000","last_clean":"2024-03-29T11:32:34.425947+0000","last_became_active":"2024-03-29T11:32:32.426512+0000","last_became_peered":"2024-03-29T11:32:32.426512+0000","last_unstale":"2024-03-29T11:32:34.425947+0000","last_undegraded":"2024-03-29T11:32:34.425947+0000","last_fullsized":"2024-03-29T11:32:34.425947+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:49:23.345901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425929+0000","last_change":"2024-03-29T11:32:32.956252+0000","last_active":"2024-03-29T11:32:34.425929+0000","last_peered":"2024-03-29T11:32:34.425929+0000","last_clean":"2024-03-29T11:32:34.425929+0000","last_became_active":"2024-03-29T11:32:32.955859+0000","last_became_peered":"2024-03-29T11:32:32.955859+0000","last_unstale":"2024-03-29T11:32:34.425929+0000","last_undegraded":"2024-03-29T11:32:34.425929+0000","last_fullsized":"2024-03-29T11:32:34.425929+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:29.934653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425919+0000","last_change":"2024-03-29T11:32:32.420246+0000","last_active":"2024-03-29T11:32:34.425919+0000","last_peered":"2024-03-29T11:32:34.425919+0000","last_clean":"2024-03-29T11:32:34.425919+0000","last_became_active":"2024-03-29T11:32:32.419983+0000","last_became_peered":"2024-03-29T11:32:32.419983+0000","last_unstale":"2024-03-29T11:32:34.425919+0000","last_undegraded":"2024-03-29T11:32:34.425919+0000","last_fullsized":"2024-03-29T11:32:34.425919+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:35.527658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425040+0000","last_change":"2024-03-29T11:32:32.946933+0000","last_active":"2024-03-29T11:32:34.425040+0000","last_peered":"2024-03-29T11:32:34.425040+0000","last_clean":"2024-03-29T11:32:34.425040+0000","last_became_active":"2024-03-29T11:32:32.946663+0000","last_became_peered":"2024-03-29T11:32:32.946663+0000","last_unstale":"2024-03-29T11:32:34.425040+0000","last_undegraded":"2024-03-29T11:32:34.425040+0000","last_fullsized":"2024-03-29T11:32:34.425040+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:23.557124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947784+0000","last_change":"2024-03-29T11:32:32.419238+0000","last_active":"2024-03-29T11:32:33.947784+0000","last_peered":"2024-03-29T11:32:33.947784+0000","last_clean":"2024-03-29T11:32:33.947784+0000","last_became_active":"2024-03-29T11:32:32.416581+0000","last_became_peered":"2024-03-29T11:32:32.416581+0000","last_unstale":"2024-03-29T11:32:33.947784+0000","last_undegraded":"2024-03-29T11:32:33.947784+0000","last_fullsized":"2024-03-29T11:32:33.947784+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:09.293759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947691+0000","last_change":"2024-03-29T11:32:32.960659+0000","last_active":"2024-03-29T11:32:33.947691+0000","last_peered":"2024-03-29T11:32:33.947691+0000","last_clean":"2024-03-29T11:32:33.947691+0000","last_became_active":"2024-03-29T11:32:32.960186+0000","last_became_peered":"2024-03-29T11:32:32.960186+0000","last_unstale":"2024-03-29T11:32:33.947691+0000","last_undegraded":"2024-03-29T11:32:33.947691+0000","last_fullsized":"2024-03-29T11:32:33.947691+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:11.369063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"247'8","reported_seq":19,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947504+0000","last_change":"2024-03-29T11:32:32.420232+0000","last_active":"2024-03-29T11:32:33.947504+0000","last_peered":"2024-03-29T11:32:33.947504+0000","last_clean":"2024-03-29T11:32:33.947504+0000","last_became_active":"2024-03-29T11:32:32.419973+0000","last_became_peered":"2024-03-29T11:32:32.419973+0000","last_unstale":"2024-03-29T11:32:33.947504+0000","last_undegraded":"2024-03-29T11:32:33.947504+0000","last_fullsized":"2024-03-29T11:32:33.947504+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:20.810629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425646+0000","last_change":"2024-03-29T11:32:32.952991+0000","last_active":"2024-03-29T11:32:34.425646+0000","last_peered":"2024-03-29T11:32:34.425646+0000","last_clean":"2024-03-29T11:32:34.425646+0000","last_became_active":"2024-03-29T11:32:32.952683+0000","last_became_peered":"2024-03-29T11:32:32.952683+0000","last_unstale":"2024-03-29T11:32:34.425646+0000","last_undegraded":"2024-03-29T11:32:34.425646+0000","last_fullsized":"2024-03-29T11:32:34.425646+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:30.121534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"247'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948397+0000","last_change":"2024-03-29T11:32:32.419741+0000","last_active":"2024-03-29T11:32:33.948397+0000","last_peered":"2024-03-29T11:32:33.948397+0000","last_clean":"2024-03-29T11:32:33.948397+0000","last_became_active":"2024-03-29T11:32:32.418896+0000","last_became_peered":"2024-03-29T11:32:32.418896+0000","last_unstale":"2024-03-29T11:32:33.948397+0000","last_undegraded":"2024-03-29T11:32:33.948397+0000","last_fullsized":"2024-03-29T11:32:33.948397+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:59:42.656917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426704+0000","last_change":"2024-03-29T11:32:32.963050+0000","last_active":"2024-03-29T11:32:34.426704+0000","last_peered":"2024-03-29T11:32:34.426704+0000","last_clean":"2024-03-29T11:32:34.426704+0000","last_became_active":"2024-03-29T11:32:32.962777+0000","last_became_peered":"2024-03-29T11:32:32.962777+0000","last_unstale":"2024-03-29T11:32:34.426704+0000","last_undegraded":"2024-03-29T11:32:34.426704+0000","last_fullsized":"2024-03-29T11:32:34.426704+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:28.065549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426654+0000","last_change":"2024-03-29T11:32:32.424196+0000","last_active":"2024-03-29T11:32:34.426654+0000","last_peered":"2024-03-29T11:32:34.426654+0000","last_clean":"2024-03-29T11:32:34.426654+0000","last_became_active":"2024-03-29T11:32:32.423924+0000","last_became_peered":"2024-03-29T11:32:32.423924+0000","last_unstale":"2024-03-29T11:32:34.426654+0000","last_undegraded":"2024-03-29T11:32:34.426654+0000","last_fullsized":"2024-03-29T11:32:34.426654+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:51.659258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426048+0000","last_change":"2024-03-29T11:32:32.958521+0000","last_active":"2024-03-29T11:32:34.426048+0000","last_peered":"2024-03-29T11:32:34.426048+0000","last_clean":"2024-03-29T11:32:34.426048+0000","last_became_active":"2024-03-29T11:32:32.958260+0000","last_became_peered":"2024-03-29T11:32:32.958260+0000","last_unstale":"2024-03-29T11:32:34.426048+0000","last_undegraded":"2024-03-29T11:32:34.426048+0000","last_fullsized":"2024-03-29T11:32:34.426048+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:34.744111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"247'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425788+0000","last_change":"2024-03-29T11:32:32.421703+0000","last_active":"2024-03-29T11:32:34.425788+0000","last_peered":"2024-03-29T11:32:34.425788+0000","last_clean":"2024-03-29T11:32:34.425788+0000","last_became_active":"2024-03-29T11:32:32.421417+0000","last_became_peered":"2024-03-29T11:32:32.421417+0000","last_unstale":"2024-03-29T11:32:34.425788+0000","last_undegraded":"2024-03-29T11:32:34.425788+0000","last_fullsized":"2024-03-29T11:32:34.425788+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:51.659258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426771+0000","last_change":"2024-03-29T11:32:32.960765+0000","last_active":"2024-03-29T11:32:34.426771+0000","last_peered":"2024-03-29T11:32:34.426771+0000","last_clean":"2024-03-29T11:32:34.426771+0000","last_became_active":"2024-03-29T11:32:32.959952+0000","last_became_peered":"2024-03-29T11:32:32.959952+0000","last_unstale":"2024-03-29T11:32:34.426771+0000","last_undegraded":"2024-03-29T11:32:34.426771+0000","last_fullsized":"2024-03-29T11:32:34.426771+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:18.696940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947714+0000","last_change":"2024-03-29T11:32:32.423658+0000","last_active":"2024-03-29T11:32:33.947714+0000","last_peered":"2024-03-29T11:32:33.947714+0000","last_clean":"2024-03-29T11:32:33.947714+0000","last_became_active":"2024-03-29T11:32:32.423195+0000","last_became_peered":"2024-03-29T11:32:32.423195+0000","last_unstale":"2024-03-29T11:32:33.947714+0000","last_undegraded":"2024-03-29T11:32:33.947714+0000","last_fullsized":"2024-03-29T11:32:33.947714+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:22.547790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426369+0000","last_change":"2024-03-29T11:32:32.958080+0000","last_active":"2024-03-29T11:32:34.426369+0000","last_peered":"2024-03-29T11:32:34.426369+0000","last_clean":"2024-03-29T11:32:34.426369+0000","last_became_active":"2024-03-29T11:32:32.957784+0000","last_became_peered":"2024-03-29T11:32:32.957784+0000","last_unstale":"2024-03-29T11:32:34.426369+0000","last_undegraded":"2024-03-29T11:32:34.426369+0000","last_fullsized":"2024-03-29T11:32:34.426369+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:48.762233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426512+0000","last_change":"2024-03-29T11:32:32.421797+0000","last_active":"2024-03-29T11:32:34.426512+0000","last_peered":"2024-03-29T11:32:34.426512+0000","last_clean":"2024-03-29T11:32:34.426512+0000","last_became_active":"2024-03-29T11:32:32.418956+0000","last_became_peered":"2024-03-29T11:32:32.418956+0000","last_unstale":"2024-03-29T11:32:34.426512+0000","last_undegraded":"2024-03-29T11:32:34.426512+0000","last_fullsized":"2024-03-29T11:32:34.426512+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:32.806138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426049+0000","last_change":"2024-03-29T11:32:32.951479+0000","last_active":"2024-03-29T11:32:34.426049+0000","last_peered":"2024-03-29T11:32:34.426049+0000","last_clean":"2024-03-29T11:32:34.426049+0000","last_became_active":"2024-03-29T11:32:32.951234+0000","last_became_peered":"2024-03-29T11:32:32.951234+0000","last_unstale":"2024-03-29T11:32:34.426049+0000","last_undegraded":"2024-03-29T11:32:34.426049+0000","last_fullsized":"2024-03-29T11:32:34.426049+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:48.762233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947833+0000","last_change":"2024-03-29T11:32:32.422506+0000","last_active":"2024-03-29T11:32:33.947833+0000","last_peered":"2024-03-29T11:32:33.947833+0000","last_clean":"2024-03-29T11:32:33.947833+0000","last_became_active":"2024-03-29T11:32:32.422249+0000","last_became_peered":"2024-03-29T11:32:32.422249+0000","last_unstale":"2024-03-29T11:32:33.947833+0000","last_undegraded":"2024-03-29T11:32:33.947833+0000","last_fullsized":"2024-03-29T11:32:33.947833+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:29.112200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947425+0000","last_change":"2024-03-29T11:32:32.955109+0000","last_active":"2024-03-29T11:32:33.947425+0000","last_peered":"2024-03-29T11:32:33.947425+0000","last_clean":"2024-03-29T11:32:33.947425+0000","last_became_active":"2024-03-29T11:32:32.954862+0000","last_became_peered":"2024-03-29T11:32:32.954862+0000","last_unstale":"2024-03-29T11:32:33.947425+0000","last_undegraded":"2024-03-29T11:32:33.947425+0000","last_fullsized":"2024-03-29T11:32:33.947425+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:18.702968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425964+0000","last_change":"2024-03-29T11:32:32.429057+0000","last_active":"2024-03-29T11:32:34.425964+0000","last_peered":"2024-03-29T11:32:34.425964+0000","last_clean":"2024-03-29T11:32:34.425964+0000","last_became_active":"2024-03-29T11:32:32.428782+0000","last_became_peered":"2024-03-29T11:32:32.428782+0000","last_unstale":"2024-03-29T11:32:34.425964+0000","last_undegraded":"2024-03-29T11:32:34.425964+0000","last_fullsized":"2024-03-29T11:32:34.425964+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:23.546712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426259+0000","last_change":"2024-03-29T11:32:32.954161+0000","last_active":"2024-03-29T11:32:34.426259+0000","last_peered":"2024-03-29T11:32:34.426259+0000","last_clean":"2024-03-29T11:32:34.426259+0000","last_became_active":"2024-03-29T11:32:32.953891+0000","last_became_peered":"2024-03-29T11:32:32.953891+0000","last_unstale":"2024-03-29T11:32:34.426259+0000","last_undegraded":"2024-03-29T11:32:34.426259+0000","last_fullsized":"2024-03-29T11:32:34.426259+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:42:17.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947669+0000","last_change":"2024-03-29T11:32:32.418585+0000","last_active":"2024-03-29T11:32:33.947669+0000","last_peered":"2024-03-29T11:32:33.947669+0000","last_clean":"2024-03-29T11:32:33.947669+0000","last_became_active":"2024-03-29T11:32:32.418297+0000","last_became_peered":"2024-03-29T11:32:32.418297+0000","last_unstale":"2024-03-29T11:32:33.947669+0000","last_undegraded":"2024-03-29T11:32:33.947669+0000","last_fullsized":"2024-03-29T11:32:33.947669+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:01.760308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948314+0000","last_change":"2024-03-29T11:32:32.953710+0000","last_active":"2024-03-29T11:32:33.948314+0000","last_peered":"2024-03-29T11:32:33.948314+0000","last_clean":"2024-03-29T11:32:33.948314+0000","last_became_active":"2024-03-29T11:32:32.953447+0000","last_became_peered":"2024-03-29T11:32:32.953447+0000","last_unstale":"2024-03-29T11:32:33.948314+0000","last_undegraded":"2024-03-29T11:32:33.948314+0000","last_fullsized":"2024-03-29T11:32:33.948314+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"247'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426207+0000","last_change":"2024-03-29T11:32:32.427167+0000","last_active":"2024-03-29T11:32:34.426207+0000","last_peered":"2024-03-29T11:32:34.426207+0000","last_clean":"2024-03-29T11:32:34.426207+0000","last_became_active":"2024-03-29T11:32:32.426833+0000","last_became_peered":"2024-03-29T11:32:32.426833+0000","last_unstale":"2024-03-29T11:32:34.426207+0000","last_undegraded":"2024-03-29T11:32:34.426207+0000","last_fullsized":"2024-03-29T11:32:34.426207+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:29.373413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426051+0000","last_change":"2024-03-29T11:32:32.967461+0000","last_active":"2024-03-29T11:32:34.426051+0000","last_peered":"2024-03-29T11:32:34.426051+0000","last_clean":"2024-03-29T11:32:34.426051+0000","last_became_active":"2024-03-29T11:32:32.967155+0000","last_became_peered":"2024-03-29T11:32:32.967155+0000","last_unstale":"2024-03-29T11:32:34.426051+0000","last_undegraded":"2024-03-29T11:32:34.426051+0000","last_fullsized":"2024-03-29T11:32:34.426051+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:24.736585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947411+0000","last_change":"2024-03-29T11:32:32.421259+0000","last_active":"2024-03-29T11:32:33.947411+0000","last_peered":"2024-03-29T11:32:33.947411+0000","last_clean":"2024-03-29T11:32:33.947411+0000","last_became_active":"2024-03-29T11:32:32.420971+0000","last_became_peered":"2024-03-29T11:32:32.420971+0000","last_unstale":"2024-03-29T11:32:33.947411+0000","last_undegraded":"2024-03-29T11:32:33.947411+0000","last_fullsized":"2024-03-29T11:32:33.947411+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:34.164745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947642+0000","last_change":"2024-03-29T11:32:32.970626+0000","last_active":"2024-03-29T11:32:33.947642+0000","last_peered":"2024-03-29T11:32:33.947642+0000","last_clean":"2024-03-29T11:32:33.947642+0000","last_became_active":"2024-03-29T11:32:32.970154+0000","last_became_peered":"2024-03-29T11:32:32.970154+0000","last_unstale":"2024-03-29T11:32:33.947642+0000","last_undegraded":"2024-03-29T11:32:33.947642+0000","last_fullsized":"2024-03-29T11:32:33.947642+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:35.827510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425993+0000","last_change":"2024-03-29T11:32:32.429420+0000","last_active":"2024-03-29T11:32:34.425993+0000","last_peered":"2024-03-29T11:32:34.425993+0000","last_clean":"2024-03-29T11:32:34.425993+0000","last_became_active":"2024-03-29T11:32:32.429133+0000","last_became_peered":"2024-03-29T11:32:32.429133+0000","last_unstale":"2024-03-29T11:32:34.425993+0000","last_undegraded":"2024-03-29T11:32:34.425993+0000","last_fullsized":"2024-03-29T11:32:34.425993+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:03.995789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948023+0000","last_change":"2024-03-29T11:32:32.971255+0000","last_active":"2024-03-29T11:32:33.948023+0000","last_peered":"2024-03-29T11:32:33.948023+0000","last_clean":"2024-03-29T11:32:33.948023+0000","last_became_active":"2024-03-29T11:32:32.968877+0000","last_became_peered":"2024-03-29T11:32:32.968877+0000","last_unstale":"2024-03-29T11:32:33.948023+0000","last_undegraded":"2024-03-29T11:32:33.948023+0000","last_fullsized":"2024-03-29T11:32:33.948023+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:42.913450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947742+0000","last_change":"2024-03-29T11:32:32.424671+0000","last_active":"2024-03-29T11:32:33.947742+0000","last_peered":"2024-03-29T11:32:33.947742+0000","last_clean":"2024-03-29T11:32:33.947742+0000","last_became_active":"2024-03-29T11:32:32.424378+0000","last_became_peered":"2024-03-29T11:32:32.424378+0000","last_unstale":"2024-03-29T11:32:33.947742+0000","last_undegraded":"2024-03-29T11:32:33.947742+0000","last_fullsized":"2024-03-29T11:32:33.947742+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:41.180787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948098+0000","last_change":"2024-03-29T11:32:32.952424+0000","last_active":"2024-03-29T11:32:33.948098+0000","last_peered":"2024-03-29T11:32:33.948098+0000","last_clean":"2024-03-29T11:32:33.948098+0000","last_became_active":"2024-03-29T11:32:32.951804+0000","last_became_peered":"2024-03-29T11:32:32.951804+0000","last_unstale":"2024-03-29T11:32:33.948098+0000","last_undegraded":"2024-03-29T11:32:33.948098+0000","last_fullsized":"2024-03-29T11:32:33.948098+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:22:05.943330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.948021+0000","last_change":"2024-03-29T11:32:32.418931+0000","last_active":"2024-03-29T11:32:33.948021+0000","last_peered":"2024-03-29T11:32:33.948021+0000","last_clean":"2024-03-29T11:32:33.948021+0000","last_became_active":"2024-03-29T11:32:32.418646+0000","last_became_peered":"2024-03-29T11:32:32.418646+0000","last_unstale":"2024-03-29T11:32:33.948021+0000","last_undegraded":"2024-03-29T11:32:33.948021+0000","last_fullsized":"2024-03-29T11:32:33.948021+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:17:12.443004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426346+0000","last_change":"2024-03-29T11:32:32.971041+0000","last_active":"2024-03-29T11:32:34.426346+0000","last_peered":"2024-03-29T11:32:34.426346+0000","last_clean":"2024-03-29T11:32:34.426346+0000","last_became_active":"2024-03-29T11:32:32.970781+0000","last_became_peered":"2024-03-29T11:32:32.970781+0000","last_unstale":"2024-03-29T11:32:34.426346+0000","last_undegraded":"2024-03-29T11:32:34.426346+0000","last_fullsized":"2024-03-29T11:32:34.426346+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:30.382747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425487+0000","last_change":"2024-03-29T11:32:32.414137+0000","last_active":"2024-03-29T11:32:34.425487+0000","last_peered":"2024-03-29T11:32:34.425487+0000","last_clean":"2024-03-29T11:32:34.425487+0000","last_became_active":"2024-03-29T11:32:32.413846+0000","last_became_peered":"2024-03-29T11:32:32.413846+0000","last_unstale":"2024-03-29T11:32:34.425487+0000","last_undegraded":"2024-03-29T11:32:34.425487+0000","last_fullsized":"2024-03-29T11:32:34.425487+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:09.293759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426118+0000","last_change":"2024-03-29T11:32:32.956222+0000","last_active":"2024-03-29T11:32:34.426118+0000","last_peered":"2024-03-29T11:32:34.426118+0000","last_clean":"2024-03-29T11:32:34.426118+0000","last_became_active":"2024-03-29T11:32:32.955784+0000","last_became_peered":"2024-03-29T11:32:32.955784+0000","last_unstale":"2024-03-29T11:32:34.426118+0000","last_undegraded":"2024-03-29T11:32:34.426118+0000","last_fullsized":"2024-03-29T11:32:34.426118+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:52.969721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425735+0000","last_change":"2024-03-29T11:32:32.421490+0000","last_active":"2024-03-29T11:32:34.425735+0000","last_peered":"2024-03-29T11:32:34.425735+0000","last_clean":"2024-03-29T11:32:34.425735+0000","last_became_active":"2024-03-29T11:32:32.421236+0000","last_became_peered":"2024-03-29T11:32:32.421236+0000","last_unstale":"2024-03-29T11:32:34.425735+0000","last_undegraded":"2024-03-29T11:32:34.425735+0000","last_fullsized":"2024-03-29T11:32:34.425735+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:17.693634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425886+0000","last_change":"2024-03-29T11:32:32.950145+0000","last_active":"2024-03-29T11:32:34.425886+0000","last_peered":"2024-03-29T11:32:34.425886+0000","last_clean":"2024-03-29T11:32:34.425886+0000","last_became_active":"2024-03-29T11:32:32.949444+0000","last_became_peered":"2024-03-29T11:32:32.949444+0000","last_unstale":"2024-03-29T11:32:34.425886+0000","last_undegraded":"2024-03-29T11:32:34.425886+0000","last_fullsized":"2024-03-29T11:32:34.425886+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:42.190121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.425393+0000","last_change":"2024-03-29T11:32:32.422243+0000","last_active":"2024-03-29T11:32:34.425393+0000","last_peered":"2024-03-29T11:32:34.425393+0000","last_clean":"2024-03-29T11:32:34.425393+0000","last_became_active":"2024-03-29T11:32:32.421949+0000","last_became_peered":"2024-03-29T11:32:32.421949+0000","last_unstale":"2024-03-29T11:32:34.425393+0000","last_undegraded":"2024-03-29T11:32:34.425393+0000","last_fullsized":"2024-03-29T11:32:34.425393+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:03.995789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:33.947429+0000","last_change":"2024-03-29T11:32:32.962313+0000","last_active":"2024-03-29T11:32:33.947429+0000","last_peered":"2024-03-29T11:32:33.947429+0000","last_clean":"2024-03-29T11:32:33.947429+0000","last_became_active":"2024-03-29T11:32:32.961887+0000","last_became_peered":"2024-03-29T11:32:32.961887+0000","last_unstale":"2024-03-29T11:32:33.947429+0000","last_undegraded":"2024-03-29T11:32:33.947429+0000","last_fullsized":"2024-03-29T11:32:33.947429+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:18.696940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"247'4","reported_seq":15,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-29T11:32:34.426703+0000","last_change":"2024-03-29T11:32:32.419906+0000","last_active":"2024-03-29T11:32:34.426703+0000","last_peered":"2024-03-29T11:32:34.426703+0000","last_clean":"2024-03-29T11:32:34.426703+0000","last_became_active":"2024-03-29T11:32:32.419653+0000","last_became_peered":"2024-03-29T11:32:32.419653+0000","last_unstale":"2024-03-29T11:32:34.426703+0000","last_undegraded":"2024-03-29T11:32:34.426703+0000","last_fullsized":"2024-03-29T11:32:34.426703+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:22.034114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"233'388","reported_seq":1367,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-29T11:32:32.944695+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:32:32.944695+0000","last_peered":"2024-03-29T11:32:32.944695+0000","last_clean":"2024-03-29T11:32:32.944695+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:32:32.944695+0000","last_undegraded":"2024-03-29T11:32:32.944695+0000","last_fullsized":"2024-03-29T11:32:32.944695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":388,"ondisk_log_size":388,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56888,"kb_used_data":3620,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314952,"statfs":{"total":96636764160,"available":96578510848,"internally_reserved":0,"allocated":3706880,"data_stored":2114077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.70799999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.73899999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42899999999999999},"max":{"1min":0.89900000000000002,"5min":0.98899999999999999,"15min":1.119},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.69299999999999995,"15min":0.72199999999999998},"min":{"1min":0.437,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90500000000000003,"5min":0.97399999999999998,"15min":1.478},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72799999999999998,"15min":0.75800000000000001},"min":{"1min":0.47399999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.393},"last":0.82699999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70599999999999996,"15min":0.748},"min":{"1min":0.504,"5min":0.436,"15min":0.436},"max":{"1min":0.85899999999999999,"5min":0.998,"15min":1.089},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92100000000000004,"5min":1.032,"15min":1.0720000000000001},"last":0.71199999999999997}]},{"osd":3,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.72299999999999998,"15min":0.753},"min":{"1min":0.54400000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90300000000000002,"5min":0.97499999999999998,"15min":1.0580000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77500000000000002,"15min":0.78300000000000003},"min":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.55300000000000005},"max":{"1min":1.0309999999999999,"5min":1.071,"15min":1.071},"last":0.92500000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":1.3779999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65900000000000003,"15min":0.66900000000000004},"min":{"1min":0.50900000000000001,"5min":0.41599999999999998,"15min":0.40899999999999997},"max":{"1min":0.85099999999999998,"5min":0.93500000000000005,"15min":1.335},"last":0.74299999999999999}]},{"osd":5,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69799999999999995,"15min":0.68100000000000005},"min":{"1min":0.434,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.1160000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.73899999999999999,"15min":0.72199999999999998},"min":{"1min":0.55600000000000005,"5min":0.50900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95499999999999996,"5min":1.0169999999999999,"15min":1.0249999999999999},"last":0.91000000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.76900000000000002,"15min":0.76700000000000002},"min":{"1min":0.52400000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0549999999999999,"15min":1.1319999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.755,"15min":0.74099999999999999},"min":{"1min":0.46899999999999997,"5min":0.439,"15min":0.435},"max":{"1min":1.0580000000000001,"5min":1.071,"15min":1.071},"last":0.78900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834576007,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48120,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323720,"statfs":{"total":96636764160,"available":96587489280,"internally_reserved":0,"allocated":3117056,"data_stored":1524243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71699999999999997,"15min":0.73699999999999999},"min":{"1min":0.45900000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86199999999999999,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.73199999999999998,"15min":0.76100000000000001},"min":{"1min":0.52700000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.84399999999999997,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.627}]},{"osd":1,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.71099999999999997,"15min":0.73699999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":0.83399999999999996,"5min":1.98,"15min":1.98},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.745,"15min":0.77300000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.92500000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.75600000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.5,"5min":0.498,"15min":0.497},"max":{"1min":0.88700000000000001,"5min":1.669,"15min":1.669},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.55700000000000005,"5min":0.49099999999999999,"15min":0.46200000000000002},"max":{"1min":0.879,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.75600000000000001,"15min":0.77300000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.91000000000000003,"5min":2.52,"15min":2.52},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.78800000000000003,"15min":0.80300000000000005},"min":{"1min":0.57099999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.996,"5min":1.877,"15min":1.877},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.62,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":0.76500000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.70899999999999996,"15min":0.70699999999999996},"min":{"1min":0.504,"5min":0.47399999999999998,"15min":0.45900000000000002},"max":{"1min":0.90000000000000002,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":0.81999999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.47299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90200000000000002,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70999999999999996,"15min":0.71299999999999997},"min":{"1min":0.54300000000000004,"5min":0.46999999999999997,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":1.925,"15min":1.925},"last":0.80500000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77100000000000002,"15min":0.76700000000000002},"min":{"1min":0.58099999999999996,"5min":0.54800000000000004,"15min":0.52700000000000002},"max":{"1min":1.0149999999999999,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.74299999999999999,"15min":0.74099999999999999},"min":{"1min":0.54600000000000004,"5min":0.504,"15min":0.47499999999999998},"max":{"1min":0.95099999999999996,"5min":1.726,"15min":1.726},"last":0.72799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576005,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60420,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311420,"statfs":{"total":96636764160,"available":96574894080,"internally_reserved":0,"allocated":3129344,"data_stored":1525327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.44500000000000001,"15min":0.46700000000000003},"min":{"1min":0.29299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69799999999999995,"5min":0.79100000000000004,"15min":0.80300000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.50900000000000001,"15min":0.53200000000000003},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.57199999999999995,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.439}]},{"osd":2,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.56200000000000006},"min":{"1min":0.36499999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.52300000000000002,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.34799999999999998},"max":{"1min":0.61599999999999999,"5min":0.81399999999999995,"15min":0.90300000000000002},"last":0.32600000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.59599999999999997},"min":{"1min":0.433,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73099999999999998,"5min":0.86199999999999999,"15min":0.995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56799999999999995,"15min":0.58099999999999996},"min":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.83999999999999997,"15min":0.878},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.75,"15min":0.78700000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.056,"5min":1.153,"15min":1.3660000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.76200000000000001,"15min":0.80000000000000004},"min":{"1min":0.56299999999999994,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97699999999999998,"5min":1.226,"15min":1.304},"last":0.95299999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.79300000000000004,"15min":0.81999999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":0.997,"5min":1.276,"15min":1.3720000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.099,"5min":1.1299999999999999,"15min":1.3520000000000001},"last":1.044}]},{"osd":6,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.78300000000000003,"15min":0.81899999999999995},"min":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.026,"5min":1.1890000000000001,"15min":1.1919999999999999},"last":1.103},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.042,"5min":1.2250000000000001,"15min":1.3069999999999999},"last":1.012}]},{"osd":7,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.83999999999999997,"15min":0.86899999999999999},"min":{"1min":0.54800000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0649999999999999,"5min":1.327,"15min":1.3540000000000001},"last":1.129},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83399999999999996,"15min":0.86799999999999999},"min":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0700000000000001,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.88700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834576007,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60980,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310860,"statfs":{"total":96636764160,"available":96574320640,"internally_reserved":0,"allocated":3702784,"data_stored":2113022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.45300000000000001,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65300000000000002,"5min":0.95899999999999996,"15min":1.087},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77400000000000002,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84999999999999998,"5min":1.028,"15min":1.5289999999999999},"last":0.502},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51100000000000001,"15min":0.52000000000000002},"min":{"1min":0.372,"5min":0.317,"15min":0.317},"max":{"1min":0.58299999999999996,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.33800000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81000000000000005,"5min":1.222,"15min":1.222},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.42199999999999999},"max":{"1min":0.83499999999999996,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.623}]},{"osd":4,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.747},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.83599999999999997,"15min":0.84999999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.091,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.70599999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.028,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81000000000000005,"15min":0.82199999999999995},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1499999999999999,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.80700000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84999999999999998,"15min":0.85899999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0700000000000001,"5min":1.343,"15min":1.6240000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.873,"15min":0.88800000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.171,"5min":1.3240000000000001,"15min":1.337},"last":0.65000000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97899999999999998,"5min":1.115,"15min":1.214},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.85799999999999998,"15min":0.874},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.204,"5min":1.365,"15min":1.365},"last":0.68600000000000005}]}]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52208,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319632,"statfs":{"total":96636764160,"available":96583303168,"internally_reserved":0,"allocated":3112960,"data_stored":1523729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47799999999999998,"15min":0.47599999999999998},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73199999999999998,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.54100000000000004},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67600000000000005,"5min":0.73999999999999999,"15min":0.96799999999999997},"last":0.377}]},{"osd":1,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74299999999999999,"5min":0.77400000000000002,"15min":0.80700000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55500000000000005,"15min":0.57699999999999996},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68300000000000005,"5min":0.872,"15min":0.88800000000000001},"last":0.42799999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.42299999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86799999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76800000000000002,"5min":1.038,"15min":1.038},"last":0.51300000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.76300000000000001,"15min":0.80300000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.93700000000000006,"5min":1.091,"15min":1.1970000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.755,"15min":0.80800000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.52400000000000002},"max":{"1min":1.026,"5min":1.244,"15min":1.244},"last":0.79500000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.81299999999999994},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.53400000000000003},"max":{"1min":0.94599999999999995,"5min":1.1599999999999999,"15min":1.256},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.78000000000000003,"15min":0.82199999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.995,"5min":1.143,"15min":1.3140000000000001},"last":0.78100000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.79300000000000004,"15min":0.84199999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98099999999999998,"5min":1.1459999999999999,"15min":1.24},"last":0.876},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79700000000000004,"15min":0.83999999999999997},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.97399999999999998,"5min":1.099,"15min":1.329},"last":0.88400000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.83899999999999997,"15min":0.88800000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":0.95899999999999996,"5min":1.1910000000000001,"15min":1.2889999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85499999999999998,"15min":0.90200000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.56899999999999995},"max":{"1min":1.095,"5min":1.2969999999999999,"15min":1.304},"last":0.81699999999999995}]}]},{"osd":3,"up_from":13,"seq":55834576006,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48116,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323724,"statfs":{"total":96636764160,"available":96587493376,"internally_reserved":0,"allocated":3112960,"data_stored":1524221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.47499999999999998,"15min":0.47099999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.36899999999999999,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.36499999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70099999999999996,"5min":0.72099999999999997,"15min":0.748},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54400000000000004,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":1.046},"last":0.60799999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.96699999999999997},"last":0.498},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.66200000000000003},"min":{"1min":0.53200000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86399999999999999,"5min":1.0049999999999999,"15min":1.01},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.80200000000000005},"min":{"1min":0.56299999999999994,"5min":0.54700000000000004,"15min":0.53600000000000003},"max":{"1min":1.2549999999999999,"5min":1.2609999999999999,"15min":1.2889999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81799999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1599999999999999,"5min":1.306,"15min":1.306},"last":0.623}]},{"osd":5,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80400000000000005,"15min":0.81599999999999995},"min":{"1min":0.56999999999999995,"5min":0.53800000000000003,"15min":0.502},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.56000000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.214,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.80600000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64900000000000002,"5min":0.64400000000000002,"15min":0.57799999999999996},"max":{"1min":1.196,"5min":1.282,"15min":1.288},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.84099999999999997,"15min":0.84699999999999998},"min":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.78800000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.85799999999999998,"15min":0.86699999999999999},"min":{"1min":0.68999999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3100000000000001,"5min":1.331,"15min":1.331},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.89500000000000002,"15min":0.89600000000000002},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.57899999999999996},"max":{"1min":1.246,"5min":1.286,"15min":1.286},"last":0.69599999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576007,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60416,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311424,"statfs":{"total":96636764160,"available":96574898176,"internally_reserved":0,"allocated":3129344,"data_stored":1525295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.66400000000000003,"15min":0.71499999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83899999999999997,"5min":0.95499999999999996,"15min":1.0509999999999999},"last":0.628},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.67500000000000004,"15min":0.72099999999999997},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.93600000000000005,"5min":0.99299999999999999,"15min":1.1459999999999999},"last":0.65800000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.68300000000000005,"15min":0.73699999999999999},"min":{"1min":0.436,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"last":0.749},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69199999999999995,"15min":0.73799999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91400000000000003,"5min":0.94899999999999995,"15min":1.002},"last":0.86899999999999999}]},{"osd":2,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.77900000000000003,"15min":0.82199999999999995},"min":{"1min":0.55400000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.004,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.76500000000000001,"15min":0.80600000000000005},"min":{"1min":0.503,"5min":0.496,"15min":0.496},"max":{"1min":1.0029999999999999,"5min":1.224,"15min":1.224},"last":0.80000000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71399999999999997,"15min":0.75900000000000001},"min":{"1min":0.54000000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.92600000000000005,"5min":1.0580000000000001,"15min":1.181},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.74399999999999999,"15min":0.76600000000000001},"min":{"1min":0.499,"5min":0.499,"15min":0.48699999999999999},"max":{"1min":0.95199999999999996,"5min":1.01,"15min":1.2490000000000001},"last":0.89200000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63800000000000001,"15min":0.625},"min":{"1min":0.45400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64500000000000002,"15min":0.65300000000000002},"min":{"1min":0.52000000000000002,"5min":0.41599999999999998,"15min":0.39700000000000002},"max":{"1min":0.81499999999999995,"5min":1.0920000000000001,"15min":1.169},"last":0.92200000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.50600000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":0.93100000000000005,"5min":1.056,"15min":1.0800000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.72199999999999998,"15min":0.71999999999999997},"min":{"1min":0.59199999999999997,"5min":0.48199999999999998,"15min":0.47699999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.94299999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73599999999999999,"15min":0.73299999999999998},"min":{"1min":0.60599999999999998,"5min":0.501,"15min":0.436},"max":{"1min":0.97999999999999998,"5min":1.0389999999999999,"15min":1.2669999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81200000000000006,"15min":0.82199999999999995},"min":{"1min":0.65900000000000003,"5min":0.58499999999999996,"15min":0.54400000000000004},"max":{"1min":1.03,"5min":1.1619999999999999,"15min":1.2969999999999999},"last":1.0880000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576006,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46456,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325384,"statfs":{"total":96636764160,"available":96589193216,"internally_reserved":0,"allocated":3121152,"data_stored":1530011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70699999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83899999999999997,"5min":0.94199999999999995,"15min":0.95999999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.73199999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90000000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.51800000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.75900000000000001,"15min":0.76500000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.73699999999999999,"15min":0.74199999999999999},"min":{"1min":0.52200000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.76800000000000002,"15min":0.75900000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.72899999999999998,"15min":0.748},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.88400000000000001,"5min":0.98499999999999999,"15min":1.0129999999999999},"last":0.62}]},{"osd":3,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.77900000000000003,"15min":0.77400000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.444,"5min":3.444,"15min":3.444},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.046},"last":0.73499999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.60599999999999998},"min":{"1min":0.44400000000000001,"5min":0.41799999999999998,"15min":0.38700000000000001},"max":{"1min":0.88,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.40600000000000003},"max":{"1min":0.84799999999999998,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.78100000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68799999999999994,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90300000000000002,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72099999999999997,"15min":0.69799999999999995},"min":{"1min":0.55900000000000005,"5min":0.51000000000000001,"15min":0.42499999999999999},"max":{"1min":0.83499999999999996,"5min":1.107,"15min":1.107},"last":0.745}]},{"osd":7,"last update":"Fri Mar 29 11:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.74199999999999999,"15min":0.72199999999999998},"min":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.42299999999999999},"max":{"1min":0.92200000000000004,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.79300000000000004,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.98199999999999998,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.76900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:32:38.348 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T11:32:38.348 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T11:32:38.372 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-29T11:32:38.376+0000 7f5f30400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:32:38.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.582+0000 7fa23cc00700 1 Processor -- start 2024-03-29T11:32:38.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.582+0000 7fa23cc00700 1 -- start start 2024-03-29T11:32:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.583+0000 7fa23cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 0x7fa2380d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.583+0000 7fa23cc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2380d16a0 0x7fa2380d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.583+0000 7fa23cc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa238161420 con 0x7fa2380d2370 2024-03-29T11:32:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.583+0000 7fa23cc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2380d2820 con 0x7fa2380d16a0 2024-03-29T11:32:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.583+0000 7fa23cc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa238161700 con 0x7fa23815d9d0 2024-03-29T11:32:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa235e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 0x7fa2380d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa235400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2380d16a0 0x7fa2380d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa235e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 0x7fa2380d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:35436/0 (socket says 172.21.15.12:35436) 2024-03-29T11:32:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa235e00700 1 -- 172.21.15.12:0/140974245 learned_addr learned my addr 172.21.15.12:0/140974245 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa236800700 1 --1- 172.21.15.12:0/140974245 >> v1:172.21.15.159:6789/0 conn(0x7fa2380d2370 0x7fa23815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:49470/0 (socket says 172.21.15.12:49470) 2024-03-29T11:32:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa235e00700 1 -- 172.21.15.12:0/140974245 >> v2:172.21.15.159:3300/0 conn(0x7fa2380d16a0 msgr2=0x7fa2380d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:38.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa235e00700 1 --2- 172.21.15.12:0/140974245 >> v2:172.21.15.159:3300/0 conn(0x7fa2380d16a0 0x7fa2380d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa235e00700 1 -- 172.21.15.12:0/140974245 >> v1:172.21.15.159:6789/0 conn(0x7fa2380d2370 legacy=0x7fa23815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa235e00700 1 -- 172.21.15.12:0/140974245 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2381619e0 con 0x7fa23815d9d0 2024-03-29T11:32:38.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa234a00700 1 -- 172.21.15.12:0/140974245 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576822646 0 0) 0x7fa238161420 con 0x7fa2380d2370 2024-03-29T11:32:38.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.584+0000 7fa235e00700 1 --2- 172.21.15.12:0/140974245 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 0x7fa2380d3c30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa2280098f0 tx=0x7fa228009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a14bedc67aa1bfdf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:38.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.585+0000 7fa234a00700 1 -- 172.21.15.12:0/140974245 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa228011830 con 0x7fa23815d9d0 2024-03-29T11:32:38.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.585+0000 7fa234a00700 1 -- 172.21.15.12:0/140974245 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa228011990 con 0x7fa23815d9d0 2024-03-29T11:32:38.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.585+0000 7fa234a00700 1 -- 172.21.15.12:0/140974245 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa228011d20 con 0x7fa23815d9d0 2024-03-29T11:32:38.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.585+0000 7fa23cc00700 1 -- 172.21.15.12:0/140974245 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 msgr2=0x7fa2380d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.585+0000 7fa23cc00700 1 --2- 172.21.15.12:0/140974245 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 0x7fa2380d3c30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa2280098f0 tx=0x7fa228009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:38.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.585+0000 7fa23cc00700 1 -- 172.21.15.12:0/140974245 shutdown_connections 2024-03-29T11:32:38.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.585+0000 7fa23cc00700 1 --2- 172.21.15.12:0/140974245 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 0x7fa2380d3c30 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.585+0000 7fa23cc00700 1 --2- 172.21.15.12:0/140974245 >> v2:172.21.15.159:3300/0 conn(0x7fa2380d16a0 0x7fa2380d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.585+0000 7fa23cc00700 1 -- 172.21.15.12:0/140974245 >> 172.21.15.12:0/140974245 conn(0x7fa23808aff0 msgr2=0x7fa2380d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:38.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.585+0000 7fa23cc00700 1 -- 172.21.15.12:0/140974245 shutdown_connections 2024-03-29T11:32:38.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.586+0000 7fa23cc00700 1 -- 172.21.15.12:0/140974245 wait complete. 2024-03-29T11:32:38.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.587+0000 7fa23cc00700 1 Processor -- start 2024-03-29T11:32:38.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.587+0000 7fa23cc00700 1 -- start start 2024-03-29T11:32:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.588+0000 7fa23cc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2380d16a0 0x7fa23815e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.588+0000 7fa23cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2380d2370 0x7fa2381589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.588+0000 7fa23cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 0x7fa2380b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.588+0000 7fa23cc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2380b7a70 con 0x7fa2380d2370 2024-03-29T11:32:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.588+0000 7fa23cc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2380b71d0 con 0x7fa2380d16a0 2024-03-29T11:32:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.588+0000 7fa23cc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2380b74b0 con 0x7fa23815d9d0 2024-03-29T11:32:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.588+0000 7fa235400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2380d2370 0x7fa2381589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa235400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2380d2370 0x7fa2381589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35004/0 (socket says 172.21.15.12:35004) 2024-03-29T11:32:38.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa235400700 1 -- 172.21.15.12:0/2225651928 learned_addr learned my addr 172.21.15.12:0/2225651928 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:38.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa235e00700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2380d16a0 0x7fa23815e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa236800700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 0x7fa2380b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa235400700 1 -- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 msgr2=0x7fa2380b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa235400700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 0x7fa2380b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa235400700 1 -- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2380d16a0 msgr2=0x7fa23815e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa235400700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2380d16a0 0x7fa23815e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa235400700 1 -- 172.21.15.12:0/2225651928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228009550 con 0x7fa2380d2370 2024-03-29T11:32:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa235400700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2380d2370 0x7fa2381589a0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fa22400e960 tx=0x7fa22400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.589+0000 7fa22e000700 1 -- 172.21.15.12:0/2225651928 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2240096f0 con 0x7fa2380d2370 2024-03-29T11:32:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.590+0000 7fa22e000700 1 -- 172.21.15.12:0/2225651928 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa224013680 con 0x7fa2380d2370 2024-03-29T11:32:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.590+0000 7fa23cc00700 1 -- 172.21.15.12:0/2225651928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2380b7790 con 0x7fa2380d2370 2024-03-29T11:32:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.590+0000 7fa22e000700 1 -- 172.21.15.12:0/2225651928 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224013a30 con 0x7fa2380d2370 2024-03-29T11:32:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.590+0000 7fa23cc00700 1 -- 172.21.15.12:0/2225651928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2380b9250 con 0x7fa2380d2370 2024-03-29T11:32:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.591+0000 7fa23cc00700 1 -- 172.21.15.12:0/2225651928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2380d1ca0 con 0x7fa2380d2370 2024-03-29T11:32:38.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.592+0000 7fa22e000700 1 -- 172.21.15.12:0/2225651928 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa224013b90 con 0x7fa2380d2370 2024-03-29T11:32:38.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.593+0000 7fa22e000700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa21806aed0 0x7fa21806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:38.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.593+0000 7fa22e000700 1 -- 172.21.15.12:0/2225651928 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fa2240899b0 con 0x7fa2380d2370 2024-03-29T11:32:38.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.593+0000 7fa235e00700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa21806aed0 0x7fa21806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:38.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.594+0000 7fa235e00700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa21806aed0 0x7fa21806d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa228014040 tx=0x7fa228008030 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:38.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.597+0000 7fa22e000700 1 -- 172.21.15.12:0/2225651928 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa224054870 con 0x7fa2380d2370 2024-03-29T11:32:38.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.899+0000 7fa23cc00700 1 -- 172.21.15.12:0/2225651928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa238067720 con 0x7fa2380d2370 2024-03-29T11:32:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.904+0000 7fa22e000700 1 -- 172.21.15.12:0/2225651928 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v51) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa2240210e0 con 0x7fa2380d2370 2024-03-29T11:32:38.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.906+0000 7fa23cc00700 1 -- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa21806aed0 msgr2=0x7fa21806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.906+0000 7fa23cc00700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa21806aed0 0x7fa21806d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa228014040 tx=0x7fa228008030 comp rx=0 tx=0).stop 2024-03-29T11:32:38.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.906+0000 7fa23cc00700 1 -- 172.21.15.12:0/2225651928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2380d2370 msgr2=0x7fa2381589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.906+0000 7fa23cc00700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2380d2370 0x7fa2381589a0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fa22400e960 tx=0x7fa22400ec70 comp rx=0 tx=0).stop 2024-03-29T11:32:38.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.907+0000 7fa23cc00700 1 -- 172.21.15.12:0/2225651928 shutdown_connections 2024-03-29T11:32:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.907+0000 7fa23cc00700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa21806aed0 0x7fa21806d390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.907+0000 7fa23cc00700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa23815d9d0 0x7fa2380b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.907+0000 7fa23cc00700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2380d16a0 0x7fa23815e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.907+0000 7fa23cc00700 1 --2- 172.21.15.12:0/2225651928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2380d2370 0x7fa2381589a0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.907+0000 7fa23cc00700 1 -- 172.21.15.12:0/2225651928 >> 172.21.15.12:0/2225651928 conn(0x7fa23808aff0 msgr2=0x7fa2380cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.907+0000 7fa23cc00700 1 -- 172.21.15.12:0/2225651928 shutdown_connections 2024-03-29T11:32:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:38.907+0000 7fa23cc00700 1 -- 172.21.15.12:0/2225651928 wait complete. 2024-03-29T11:32:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T11:32:38.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:39.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.123+0000 7f2bdbe00700 1 Processor -- start 2024-03-29T11:32:39.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.123+0000 7f2bdbe00700 1 -- start start 2024-03-29T11:32:39.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.124+0000 7f2bdbe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b3af0 0x7f2bd415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.124+0000 7f2bdbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2bd4072580 0x7f2bd40b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.124+0000 7f2bdbe00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd40b2180 con 0x7f2bd40b2e50 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.124+0000 7f2bdbe00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd415b100 con 0x7f2bd4072580 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.124+0000 7f2bdbe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd415b3e0 con 0x7f2bd40b3af0 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.124+0000 7f2bd8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2bd4072580 0x7f2bd40b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.124+0000 7f2bd8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2bd4072580 0x7f2bd40b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43250/0 (socket says 172.21.15.12:43250) 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.124+0000 7f2bd8a00700 1 -- 172.21.15.12:0/125483582 learned_addr learned my addr 172.21.15.12:0/125483582 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.125+0000 7f2bd9400700 1 --2- 172.21.15.12:0/125483582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b3af0 0x7f2bd415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.125+0000 7f2bd8a00700 1 -- 172.21.15.12:0/125483582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b3af0 msgr2=0x7f2bd415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.125+0000 7f2bd8a00700 1 --2- 172.21.15.12:0/125483582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b3af0 0x7f2bd415a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.125+0000 7f2bd8a00700 1 -- 172.21.15.12:0/125483582 >> v1:172.21.15.159:6789/0 conn(0x7f2bd40b2e50 legacy=0x7f2bd40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.125+0000 7f2bd8a00700 1 -- 172.21.15.12:0/125483582 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd40b23f0 con 0x7f2bd4072580 2024-03-29T11:32:39.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.125+0000 7f2bd9400700 1 --2- 172.21.15.12:0/125483582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b3af0 0x7f2bd415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:39.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.125+0000 7f2bd8a00700 1 --2- 172.21.15.12:0/125483582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd4072580 0x7f2bd40b2910 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f2bc00098f0 tx=0x7f2bc0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cedff9146be145 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:39.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.125+0000 7f2bd3e00700 1 -- 172.21.15.12:0/125483582 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0004230 con 0x7f2bd4072580 2024-03-29T11:32:39.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.125+0000 7f2bd3e00700 1 -- 172.21.15.12:0/125483582 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc0004390 con 0x7f2bd4072580 2024-03-29T11:32:39.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.125+0000 7f2bd3e00700 1 -- 172.21.15.12:0/125483582 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc00036a0 con 0x7f2bd4072580 2024-03-29T11:32:39.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.126+0000 7f2bdbe00700 1 -- 172.21.15.12:0/125483582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd4072580 msgr2=0x7f2bd40b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:39.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.126+0000 7f2bdbe00700 1 --2- 172.21.15.12:0/125483582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd4072580 0x7f2bd40b2910 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f2bc00098f0 tx=0x7f2bc0009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:39.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.126+0000 7f2bdbe00700 1 -- 172.21.15.12:0/125483582 shutdown_connections 2024-03-29T11:32:39.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.126+0000 7f2bdbe00700 1 --2- 172.21.15.12:0/125483582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b3af0 0x7f2bd415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:39.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.126+0000 7f2bdbe00700 1 --2- 172.21.15.12:0/125483582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd4072580 0x7f2bd40b2910 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:39.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.126+0000 7f2bdbe00700 1 -- 172.21.15.12:0/125483582 >> 172.21.15.12:0/125483582 conn(0x7f2bd40cf190 msgr2=0x7f2bd40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:39.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.126+0000 7f2bdbe00700 1 -- 172.21.15.12:0/125483582 shutdown_connections 2024-03-29T11:32:39.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.126+0000 7f2bdbe00700 1 -- 172.21.15.12:0/125483582 wait complete. 2024-03-29T11:32:39.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.128+0000 7f2bdbe00700 1 Processor -- start 2024-03-29T11:32:39.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.128+0000 7f2bdbe00700 1 -- start start 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bdbe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2bd4072580 0x7f2bd4154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bdbe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b2e50 0x7f2bd41564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bdbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd40b3af0 0x7f2bd40ce330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bdbe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd4155450 con 0x7f2bd40b3af0 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bdbe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd4155730 con 0x7f2bd4072580 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bdbe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd4155a10 con 0x7f2bd40b2e50 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bd9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd40b3af0 0x7f2bd40ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bd9400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2bd4072580 0x7f2bd4154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bd9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd40b3af0 0x7f2bd40ce330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43252/0 (socket says 172.21.15.12:43252) 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bd9e00700 1 -- 172.21.15.12:0/4071814178 learned_addr learned my addr 172.21.15.12:0/4071814178 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bd8a00700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b2e50 0x7f2bd41564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bd9e00700 1 -- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b2e50 msgr2=0x7f2bd41564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:39.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bd9e00700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b2e50 0x7f2bd41564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:39.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bd9e00700 1 -- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2bd4072580 msgr2=0x7f2bd4154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:39.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bd9e00700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2bd4072580 0x7f2bd4154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:39.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.129+0000 7f2bd9e00700 1 -- 172.21.15.12:0/4071814178 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc0009550 con 0x7f2bd40b3af0 2024-03-29T11:32:39.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.130+0000 7f2bd9e00700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd40b3af0 0x7f2bd40ce330 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00b410 tx=0x7f2bcc00b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:39.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.130+0000 7f2bd1600700 1 -- 172.21.15.12:0/4071814178 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc011830 con 0x7f2bd40b3af0 2024-03-29T11:32:39.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.130+0000 7f2bd1600700 1 -- 172.21.15.12:0/4071814178 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bcc011990 con 0x7f2bd40b3af0 2024-03-29T11:32:39.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.130+0000 7f2bdbe00700 1 -- 172.21.15.12:0/4071814178 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd4155cf0 con 0x7f2bd40b3af0 2024-03-29T11:32:39.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.130+0000 7f2bd1600700 1 -- 172.21.15.12:0/4071814178 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc011d40 con 0x7f2bd40b3af0 2024-03-29T11:32:39.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.130+0000 7f2bdbe00700 1 -- 172.21.15.12:0/4071814178 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd40ce870 con 0x7f2bd40b3af0 2024-03-29T11:32:39.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.131+0000 7f2bdbe00700 1 -- 172.21.15.12:0/4071814178 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd4156210 con 0x7f2bd40b3af0 2024-03-29T11:32:39.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.132+0000 7f2bd1600700 1 -- 172.21.15.12:0/4071814178 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2bcc01a400 con 0x7f2bd40b3af0 2024-03-29T11:32:39.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.133+0000 7f2bd1600700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2bb406aed0 0x7f2bb406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:39.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.133+0000 7f2bd1600700 1 -- 172.21.15.12:0/4071814178 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f2bcc089370 con 0x7f2bd40b3af0 2024-03-29T11:32:39.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.133+0000 7f2bd9400700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2bb406aed0 0x7f2bb406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:39.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.134+0000 7f2bd9400700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2bb406aed0 0x7f2bb406d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400d620 tx=0x7f2bc4007810 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:39.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.137+0000 7f2bd1600700 1 -- 172.21.15.12:0/4071814178 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2bcc054100 con 0x7f2bd40b3af0 2024-03-29T11:32:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.402+0000 7f2bdbe00700 1 -- 172.21.15.12:0/4071814178 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bd40b3ed0 con 0x7f2bd40b3af0 2024-03-29T11:32:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.403+0000 7f2bd1600700 1 -- 172.21.15.12:0/4071814178 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f2bcc057720 con 0x7f2bd40b3af0 2024-03-29T11:32:39.398 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:39.398 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11416,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:32.933737+0000","modified":"2024-03-29T11:32:36.420698+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11853":{"gid":11853,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3711953177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3711953177},{"type":"v1","addr":"172.21.15.12:6834","nonce":3711953177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-03-29T11:32:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.405+0000 7f2bdbe00700 1 -- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2bb406aed0 msgr2=0x7f2bb406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.405+0000 7f2bdbe00700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2bb406aed0 0x7f2bb406d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400d620 tx=0x7f2bc4007810 comp rx=0 tx=0).stop 2024-03-29T11:32:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.405+0000 7f2bdbe00700 1 -- 172.21.15.12:0/4071814178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd40b3af0 msgr2=0x7f2bd40ce330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.405+0000 7f2bdbe00700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd40b3af0 0x7f2bd40ce330 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00b410 tx=0x7f2bcc00b720 comp rx=0 tx=0).stop 2024-03-29T11:32:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.405+0000 7f2bdbe00700 1 -- 172.21.15.12:0/4071814178 shutdown_connections 2024-03-29T11:32:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.405+0000 7f2bdbe00700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2bb406aed0 0x7f2bb406d390 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.405+0000 7f2bdbe00700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bd40b3af0 0x7f2bd40ce330 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.405+0000 7f2bdbe00700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2bd4072580 0x7f2bd4154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.405+0000 7f2bdbe00700 1 --2- 172.21.15.12:0/4071814178 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40b2e50 0x7f2bd41564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.405+0000 7f2bdbe00700 1 -- 172.21.15.12:0/4071814178 >> 172.21.15.12:0/4071814178 conn(0x7f2bd40cf190 msgr2=0x7f2bd406daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.406+0000 7f2bdbe00700 1 -- 172.21.15.12:0/4071814178 shutdown_connections 2024-03-29T11:32:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:39.406+0000 7f2bdbe00700 1 -- 172.21.15.12:0/4071814178 wait complete. 2024-03-29T11:32:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 122 2024-03-29T11:32:39.821 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T11:32:32.933737+0000', 'modified': '2024-03-29T11:32:36.420698+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11853}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11853': {'gid': 11853, 'name': 'a', 'rank': 0, 'incarnation': 118, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6834/3711953177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6832', 'nonce': 3711953177}, {'type': 'v1', 'addr': '172.21.15.12:6834', 'nonce': 3711953177}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T11:32:39.821 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T11:32:39.822 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-29T11:32:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.030+0000 7fd4f1400700 1 Processor -- start 2024-03-29T11:32:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.030+0000 7fd4f1400700 1 -- start start 2024-03-29T11:32:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.031+0000 7fd4f1400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd4ec15d9d0 0x7fd4ec0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.031+0000 7fd4f1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d16a0 0x7fd4ec0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.031+0000 7fd4f1400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ec161420 con 0x7fd4ec0d2370 2024-03-29T11:32:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.031+0000 7fd4f1400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ec0d2820 con 0x7fd4ec15d9d0 2024-03-29T11:32:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.031+0000 7fd4f1400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec161700 con 0x7fd4ec0d16a0 2024-03-29T11:32:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.031+0000 7fd4eb200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd4ec0d2370 0x7fd4ec15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53738/0 (socket says 172.21.15.12:53738) 2024-03-29T11:32:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.031+0000 7fd4eb200700 1 -- 172.21.15.12:0/2673952030 learned_addr learned my addr 172.21.15.12:0/2673952030 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.031+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/2673952030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d16a0 0x7fd4ec0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.031+0000 7fd4ea800700 1 --2- 172.21.15.12:0/2673952030 >> v2:172.21.15.159:3300/0 conn(0x7fd4ec15d9d0 0x7fd4ec0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.032+0000 7fd4e9400700 1 -- 172.21.15.12:0/2673952030 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887922666 0 0) 0x7fd4ec161420 con 0x7fd4ec0d2370 2024-03-29T11:32:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.032+0000 7fd4e9400700 1 -- 172.21.15.12:0/2673952030 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d00035b0 con 0x7fd4ec0d2370 2024-03-29T11:32:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.032+0000 7fd4e9e00700 1 -- 172.21.15.12:0/2673952030 >> v2:172.21.15.159:3300/0 conn(0x7fd4ec15d9d0 msgr2=0x7fd4ec0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.032+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/2673952030 >> v2:172.21.15.159:3300/0 conn(0x7fd4ec15d9d0 0x7fd4ec0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.032+0000 7fd4e9e00700 1 -- 172.21.15.12:0/2673952030 >> v1:172.21.15.12:6789/0 conn(0x7fd4ec0d2370 legacy=0x7fd4ec15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.032+0000 7fd4e9e00700 1 -- 172.21.15.12:0/2673952030 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec1619e0 con 0x7fd4ec0d16a0 2024-03-29T11:32:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.032+0000 7fd4ea800700 1 --2- 172.21.15.12:0/2673952030 >> v2:172.21.15.159:3300/0 conn(0x7fd4ec15d9d0 0x7fd4ec0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.032+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/2673952030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d16a0 0x7fd4ec0d1e30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009820 tx=0x7fd4d8009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63e93169b5b2f3b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4e9400700 1 -- 172.21.15.12:0/2673952030 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8005e30 con 0x7fd4ec0d16a0 2024-03-29T11:32:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4e9400700 1 -- 172.21.15.12:0/2673952030 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d8006030 con 0x7fd4ec0d16a0 2024-03-29T11:32:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4e9400700 1 -- 172.21.15.12:0/2673952030 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80063c0 con 0x7fd4ec0d16a0 2024-03-29T11:32:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4f1400700 1 -- 172.21.15.12:0/2673952030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d16a0 msgr2=0x7fd4ec0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4f1400700 1 --2- 172.21.15.12:0/2673952030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d16a0 0x7fd4ec0d1e30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009820 tx=0x7fd4d8009b30 comp rx=0 tx=0).stop 2024-03-29T11:32:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4f1400700 1 -- 172.21.15.12:0/2673952030 shutdown_connections 2024-03-29T11:32:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4f1400700 1 --2- 172.21.15.12:0/2673952030 >> v2:172.21.15.159:3300/0 conn(0x7fd4ec15d9d0 0x7fd4ec0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4f1400700 1 --2- 172.21.15.12:0/2673952030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d16a0 0x7fd4ec0d1e30 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4f1400700 1 -- 172.21.15.12:0/2673952030 >> 172.21.15.12:0/2673952030 conn(0x7fd4ec08aff0 msgr2=0x7fd4ec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4f1400700 1 -- 172.21.15.12:0/2673952030 shutdown_connections 2024-03-29T11:32:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.033+0000 7fd4f1400700 1 -- 172.21.15.12:0/2673952030 wait complete. 2024-03-29T11:32:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.035+0000 7fd4f1400700 1 Processor -- start 2024-03-29T11:32:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.035+0000 7fd4f1400700 1 -- start start 2024-03-29T11:32:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.036+0000 7fd4f1400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4ec0d16a0 0x7fd4ec151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.036+0000 7fd4f1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d2370 0x7fd4ec15e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.036+0000 7fd4f1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4ec15d9d0 0x7fd4ec1604a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.036+0000 7fd4f1400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec15e400 con 0x7fd4ec15d9d0 2024-03-29T11:32:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.036+0000 7fd4f1400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0b80f0 con 0x7fd4ec0d16a0 2024-03-29T11:32:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.036+0000 7fd4f1400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0b83d0 con 0x7fd4ec0d2370 2024-03-29T11:32:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.036+0000 7fd4eb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4ec15d9d0 0x7fd4ec1604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.036+0000 7fd4eb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4ec15d9d0 0x7fd4ec1604a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43260/0 (socket says 172.21.15.12:43260) 2024-03-29T11:32:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.036+0000 7fd4eb200700 1 -- 172.21.15.12:0/4070334475 learned_addr learned my addr 172.21.15.12:0/4070334475 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4ea800700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4ec0d16a0 0x7fd4ec151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d2370 0x7fd4ec15e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4eb200700 1 -- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d2370 msgr2=0x7fd4ec15e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4eb200700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d2370 0x7fd4ec15e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4eb200700 1 -- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4ec0d16a0 msgr2=0x7fd4ec151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4eb200700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4ec0d16a0 0x7fd4ec151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4eb200700 1 -- 172.21.15.12:0/4070334475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc0094a0 con 0x7fd4ec15d9d0 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4ea800700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4ec0d16a0 0x7fd4ec151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4eb200700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4ec15d9d0 0x7fd4ec1604a0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400cb40 tx=0x7fd4e400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4e2a00700 1 -- 172.21.15.12:0/4070334475 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e400aaa0 con 0x7fd4ec15d9d0 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4e2a00700 1 -- 172.21.15.12:0/4070334475 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e400ac00 con 0x7fd4ec15d9d0 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4e2a00700 1 -- 172.21.15.12:0/4070334475 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4012400 con 0x7fd4ec15d9d0 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.037+0000 7fd4f1400700 1 -- 172.21.15.12:0/4070334475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d8009430 con 0x7fd4ec15d9d0 2024-03-29T11:32:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.038+0000 7fd4f1400700 1 -- 172.21.15.12:0/4070334475 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ec15e090 con 0x7fd4ec15d9d0 2024-03-29T11:32:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.040+0000 7fd4e2a00700 1 -- 172.21.15.12:0/4070334475 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd4e401b3b0 con 0x7fd4ec15d9d0 2024-03-29T11:32:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.040+0000 7fd4e2a00700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd4cc06aed0 0x7fd4cc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.040+0000 7fd4e2a00700 1 -- 172.21.15.12:0/4070334475 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fd4e4089590 con 0x7fd4ec15d9d0 2024-03-29T11:32:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.041+0000 7fd4ea800700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd4cc06aed0 0x7fd4cc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.042+0000 7fd4ea800700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd4cc06aed0 0x7fd4cc06d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009d20 tx=0x7fd4dc008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.043+0000 7fd504fe7b80 1 Processor -- start 2024-03-29T11:32:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.043+0000 7fd504fe7b80 1 -- start start 2024-03-29T11:32:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.044+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590d2cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.044+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cffbe0 0x559590d02a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.044+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cfdb60 0x559590d29a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.044+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559590c77650 con 0x559590d2c830 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.044+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x559590c58c30 con 0x559590cffbe0 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.044+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x559590c56430 con 0x559590cfdb60 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.044+0000 7fd4ea800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590d2cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.044+0000 7fd4ea800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590d2cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43268/0 (socket says 172.21.15.12:43268) 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.044+0000 7fd4ea800700 1 -- 172.21.15.12:0/2068577136 learned_addr learned my addr 172.21.15.12:0/2068577136 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cffbe0 0x559590d02a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4eb200700 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cfdb60 0x559590d29a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4ea800700 1 -- 172.21.15.12:0/2068577136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cfdb60 msgr2=0x559590d29a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4ea800700 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cfdb60 0x559590d29a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4ea800700 1 -- 172.21.15.12:0/2068577136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cffbe0 msgr2=0x559590d02a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4ea800700 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cffbe0 0x559590d02a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4ea800700 1 -- 172.21.15.12:0/2068577136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559590a45790 con 0x559590d2c830 2024-03-29T11:32:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4eb200700 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cfdb60 0x559590d29a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cffbe0 0x559590d02a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4ea800700 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590d2cc10 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc011de0 tx=0x7fd4dc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb095f56fc8274e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4d7000700 1 -- 172.21.15.12:0/2068577136 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc011e10 con 0x559590d2c830 2024-03-29T11:32:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4d7000700 1 -- 172.21.15.12:0/2068577136 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc015c60 con 0x559590d2c830 2024-03-29T11:32:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.045+0000 7fd4d7000700 1 -- 172.21.15.12:0/2068577136 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc021640 con 0x559590d2c830 2024-03-29T11:32:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.046+0000 7fd504fe7b80 1 -- 172.21.15.12:0/2068577136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 msgr2=0x559590d2cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.046+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590d2cc10 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc011de0 tx=0x7fd4dc014000 comp rx=0 tx=0).stop 2024-03-29T11:32:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.046+0000 7fd504fe7b80 1 -- 172.21.15.12:0/2068577136 shutdown_connections 2024-03-29T11:32:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.046+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cfdb60 0x559590d29a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.046+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590d2cc10 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.046+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/2068577136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cffbe0 0x559590d02a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.046+0000 7fd504fe7b80 1 -- 172.21.15.12:0/2068577136 >> 172.21.15.12:0/2068577136 conn(0x559590c606a0 msgr2=0x559590d47640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.046+0000 7fd504fe7b80 1 -- 172.21.15.12:0/2068577136 shutdown_connections 2024-03-29T11:32:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.046+0000 7fd504fe7b80 1 -- 172.21.15.12:0/2068577136 wait complete. 2024-03-29T11:32:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.047+0000 7fd504fe7b80 1 -- start start 2024-03-29T11:32:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.047+0000 7fd504fe7b80 1 Processor -- start 2024-03-29T11:32:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 0x559590d28d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cffbe0 0x559590d2fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590d2c830 0x559590d48ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559590c51d20 con 0x559590cffbe0 2024-03-29T11:32:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x559590bed770 con 0x559590cfdb60 2024-03-29T11:32:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x559590c106f0 con 0x559590d2c830 2024-03-29T11:32:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd504fe7b80 10 client.0 _ll_register_callbacks cb 0x559590bf75e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:32:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd4e9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cffbe0 0x559590d2fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd4e9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cffbe0 0x559590d2fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43278/0 (socket says 172.21.15.12:43278) 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd4e9e00700 1 -- 172.21.15.12:0/3290183687 learned_addr learned my addr 172.21.15.12:0/3290183687 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd4ea800700 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 0x559590d28d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.049+0000 7fd4eb200700 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590d2c830 0x559590d48ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd4e9e00700 1 -- 172.21.15.12:0/3290183687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590d2c830 msgr2=0x559590d48ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590d2c830 0x559590d48ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd4e9e00700 1 -- 172.21.15.12:0/3290183687 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 msgr2=0x559590d28d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 0x559590d28d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd4e9e00700 1 -- 172.21.15.12:0/3290183687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc011ae0 con 0x559590cffbe0 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cffbe0 0x559590d2fed0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8013040 tx=0x7fd4d80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7d420dab03b6e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd4cb400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:32:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd4cb400700 1 -- 172.21.15.12:0/3290183687 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8022070 con 0x559590cffbe0 2024-03-29T11:32:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd4cb400700 1 -- 172.21.15.12:0/3290183687 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fd4d8004250 con 0x559590cffbe0 2024-03-29T11:32:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3290183687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559590cf4920 con 0x559590cffbe0 2024-03-29T11:32:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd4cb400700 1 -- 172.21.15.12:0/3290183687 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4d8004410 con 0x559590cffbe0 2024-03-29T11:32:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd504fe7b80 10 client.11982 fetch_fsmap learned FSMap version 122 2024-03-29T11:32:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.050+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3290183687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x559590b320a0 con 0x559590cffbe0 2024-03-29T11:32:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.051+0000 7fd4cb400700 1 -- 172.21.15.12:0/3290183687 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559590cf4920 con 0x559590cffbe0 2024-03-29T11:32:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.051+0000 7fd4cb400700 1 -- 172.21.15.12:0/3290183687 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd4d80061a0 con 0x559590cffbe0 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.051+0000 7fd504fe7b80 10 client.11982 fetch_fsmap finished waiting for FSMap version 122 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.051+0000 7fd504fe7b80 10 client.11982 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.051+0000 7fd504fe7b80 20 client.11982 populate_metadata read hostname 'smithi012' 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.051+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.12:6832/3711953177,v1:172.21.15.12:6834/3711953177] conn(0x559590d5a2c0 0x559590d5a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.051+0000 7fd504fe7b80 4 client.11982 mds_command: new command op to 11853 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.051+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3290183687 --> [v2:172.21.15.12:6832/3711953177,v1:172.21.15.12:6834/3711953177] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559590cd16f0 con 0x559590d5a2c0 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.051+0000 7fd4ea800700 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.12:6832/3711953177,v1:172.21.15.12:6834/3711953177] conn(0x559590d5a2c0 0x559590d5a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.052+0000 7fd4cb400700 10 client.11982 ms_handle_connect on v2:172.21.15.12:6832/3711953177 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.052+0000 7fd4ea800700 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.12:6832/3711953177,v1:172.21.15.12:6834/3711953177] conn(0x559590d5a2c0 0x559590d5a6a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d323c42f55be6d4c server_cookie=f21d21d464bf4f69 in_seq=0 out_seq=0 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.052+0000 7fd4cb400700 1 -- 172.21.15.12:0/3290183687 <== mds.0 v2:172.21.15.12:6832/3711953177 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559590cd16f0 con 0x559590d5a2c0 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.052+0000 7fd4cb400700 10 client.11982 handle_command_reply: tid=0 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.052+0000 7fd504fe7b80 1 client.11982 shutdown 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3290183687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cffbe0 msgr2=0x559590d2fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cffbe0 0x559590d2fed0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8013040 tx=0x7fd4d80047c0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3290183687 shutdown_connections 2024-03-29T11:32:40.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.12:6832/3711953177,v1:172.21.15.12:6834/3711953177] conn(0x559590d5a2c0 0x559590d5a6a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590d2c830 0x559590d48ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 0x559590d28d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd4cb400700 0 client.11982 ms_handle_reset on v2:172.21.15.12:6832/3711953177 2024-03-29T11:32:40.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3290183687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cffbe0 0x559590d2fed0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3290183687 >> 172.21.15.12:0/3290183687 conn(0x559590c606a0 msgr2=0x559590d01c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:40.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3290183687 shutdown_connections 2024-03-29T11:32:40.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3290183687 wait complete. 2024-03-29T11:32:40.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.053+0000 7fd504fe7b80 20 client.11982 trim_cache size 0 max 16384 2024-03-29T11:32:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.071+0000 7fd504fe7b80 1 Processor -- start 2024-03-29T11:32:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.071+0000 7fd504fe7b80 1 -- start start 2024-03-29T11:32:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.072+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cfdb60 0x559590d02a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.072+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cffbe0 0x559590d2b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.072+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590d2c830 0x559590d56010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.072+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559590d49da0 con 0x559590cfdb60 2024-03-29T11:32:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.072+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x559590a46390 con 0x559590d2c830 2024-03-29T11:32:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.072+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x559590d4a8c0 con 0x559590cffbe0 2024-03-29T11:32:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4ea800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cfdb60 0x559590d02a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4ea800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cfdb60 0x559590d02a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43294/0 (socket says 172.21.15.12:43294) 2024-03-29T11:32:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4ea800700 1 -- 172.21.15.12:0/3403372486 learned_addr learned my addr 172.21.15.12:0/3403372486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/3403372486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cffbe0 0x559590d2b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4eb200700 1 --2- 172.21.15.12:0/3403372486 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590d2c830 0x559590d56010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4ea800700 1 -- 172.21.15.12:0/3403372486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cffbe0 msgr2=0x559590d2b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4ea800700 1 --2- 172.21.15.12:0/3403372486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cffbe0 0x559590d2b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4ea800700 1 -- 172.21.15.12:0/3403372486 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590d2c830 msgr2=0x559590d56010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4ea800700 1 --2- 172.21.15.12:0/3403372486 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590d2c830 0x559590d56010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4ea800700 1 -- 172.21.15.12:0/3403372486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559590c18010 con 0x559590cfdb60 2024-03-29T11:32:40.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.073+0000 7fd4ea800700 1 --2- 172.21.15.12:0/3403372486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cfdb60 0x559590d02a60 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009b00 tx=0x7fd4dc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74aedae0348de654 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd4d7a00700 1 -- 172.21.15.12:0/3403372486 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc030040 con 0x559590cfdb60 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd4d7a00700 1 -- 172.21.15.12:0/3403372486 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc032040 con 0x559590cfdb60 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd4d7a00700 1 -- 172.21.15.12:0/3403372486 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc0375c0 con 0x559590cfdb60 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3403372486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cfdb60 msgr2=0x559590d02a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3403372486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cfdb60 0x559590d02a60 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009b00 tx=0x7fd4dc016000 comp rx=0 tx=0).stop 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3403372486 shutdown_connections 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3403372486 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590d2c830 0x559590d56010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3403372486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590cfdb60 0x559590d02a60 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3403372486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cffbe0 0x559590d2b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3403372486 >> 172.21.15.12:0/3403372486 conn(0x559590c606a0 msgr2=0x559590d28b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3403372486 shutdown_connections 2024-03-29T11:32:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.074+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3403372486 wait complete. 2024-03-29T11:32:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.075+0000 7fd504fe7b80 1 -- start start 2024-03-29T11:32:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.075+0000 7fd504fe7b80 1 Processor -- start 2024-03-29T11:32:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.076+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 0x559590d029e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.076+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cffbe0 0x559590dab510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.076+0000 7fd504fe7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590cfc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.076+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559590d3d700 con 0x559590d2c830 2024-03-29T11:32:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.076+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x559590db1590 con 0x559590cfdb60 2024-03-29T11:32:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.076+0000 7fd504fe7b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x559590db17d0 con 0x559590cffbe0 2024-03-29T11:32:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.076+0000 7fd504fe7b80 10 client.0 _ll_register_callbacks cb 0x7fd4d8009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:32:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.076+0000 7fd4eb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590cfc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.076+0000 7fd4eb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590cfc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43302/0 (socket says 172.21.15.12:43302) 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.076+0000 7fd4eb200700 1 -- 172.21.15.12:0/3364946345 learned_addr learned my addr 172.21.15.12:0/3364946345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4e9e00700 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cffbe0 0x559590dab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4ea800700 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 0x559590d029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4eb200700 1 -- 172.21.15.12:0/3364946345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cffbe0 msgr2=0x559590dab510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4eb200700 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cffbe0 0x559590dab510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4eb200700 1 -- 172.21.15.12:0/3364946345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 msgr2=0x559590d029e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4eb200700 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 0x559590d029e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4eb200700 1 -- 172.21.15.12:0/3364946345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559590a45790 con 0x559590d2c830 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4ea800700 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 0x559590d029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4eb200700 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590cfc060 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4057f70 tx=0x7fd4e4057a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b99866caa2b10e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4cb400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4cb400700 1 -- 172.21.15.12:0/3364946345 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4060140 con 0x559590d2c830 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4cb400700 1 -- 172.21.15.12:0/3364946345 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fd4e40603d0 con 0x559590d2c830 2024-03-29T11:32:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3364946345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4dc015c60 con 0x559590d2c830 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.077+0000 7fd4cb400700 1 -- 172.21.15.12:0/3364946345 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4e405e0b0 con 0x559590d2c830 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd504fe7b80 10 client.11988 fetch_fsmap learned FSMap version 122 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3364946345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x559590aa61d0 con 0x559590d2c830 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd4cb400700 1 -- 172.21.15.12:0/3364946345 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc015c60 con 0x559590d2c830 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd4cb400700 1 -- 172.21.15.12:0/3364946345 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd4e4087250 con 0x559590d2c830 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd504fe7b80 10 client.11988 fetch_fsmap finished waiting for FSMap version 122 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd504fe7b80 10 client.11988 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd504fe7b80 20 client.11988 populate_metadata read hostname 'smithi012' 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.12:6832/3711953177,v1:172.21.15.12:6834/3711953177] conn(0x559590dae640 0x559590daea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd504fe7b80 4 client.11988 mds_command: new command op to 11853 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3364946345 --> [v2:172.21.15.12:6832/3711953177,v1:172.21.15.12:6834/3711953177] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559590a8b040 con 0x559590dae640 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.078+0000 7fd4ea800700 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.12:6832/3711953177,v1:172.21.15.12:6834/3711953177] conn(0x559590dae640 0x559590daea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd4cb400700 10 client.11988 ms_handle_connect on v2:172.21.15.12:6832/3711953177 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd4ea800700 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.12:6832/3711953177,v1:172.21.15.12:6834/3711953177] conn(0x559590dae640 0x559590daea20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3b7a21d2b0d1d0a server_cookie=77376fc354391b8b in_seq=0 out_seq=0 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd4cb400700 1 -- 172.21.15.12:0/3364946345 <== mds.0 v2:172.21.15.12:6832/3711953177 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x559590a8b040 con 0x559590dae640 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd4cb400700 10 client.11988 handle_command_reply: tid=0 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 client.11988 shutdown 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3364946345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 msgr2=0x559590cfc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590cfc060 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4057f70 tx=0x7fd4e4057a70 comp rx=0 tx=0).stop 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3364946345 shutdown_connections 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.12:6832/3711953177,v1:172.21.15.12:6834/3711953177] conn(0x559590dae640 0x559590daea20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559590d2c830 0x559590cfc060 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559590cfdb60 0x559590d029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd4cb400700 0 client.11988 ms_handle_reset on v2:172.21.15.12:6832/3711953177 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 --2- 172.21.15.12:0/3364946345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559590cffbe0 0x559590dab510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3364946345 >> 172.21.15.12:0/3364946345 conn(0x559590c606a0 msgr2=0x559590d2f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3364946345 shutdown_connections 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.079+0000 7fd504fe7b80 1 -- 172.21.15.12:0/3364946345 wait complete. 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd504fe7b80 20 client.11988 trim_cache size 0 max 16384 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 11853, 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T11:32:40.079 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1434783980000001, 2024-03-29T11:32:40.080 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 122, 2024-03-29T11:32:40.080 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 248, 2024-03-29T11:32:40.080 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 248, 2024-03-29T11:32:40.080 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12.139907947999999 2024-03-29T11:32:40.080 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 -- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd4cc06aed0 msgr2=0x7fd4cc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd4cc06aed0 0x7fd4cc06d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009d20 tx=0x7fd4dc008040 comp rx=0 tx=0).stop 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 -- 172.21.15.12:0/4070334475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4ec15d9d0 msgr2=0x7fd4ec1604a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4ec15d9d0 0x7fd4ec1604a0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400cb40 tx=0x7fd4e400ce50 comp rx=0 tx=0).stop 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 -- 172.21.15.12:0/4070334475 shutdown_connections 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd4cc06aed0 0x7fd4cc06d390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4ec15d9d0 0x7fd4ec1604a0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4ec0d16a0 0x7fd4ec151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 --2- 172.21.15.12:0/4070334475 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4ec0d2370 0x7fd4ec15e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 -- 172.21.15.12:0/4070334475 >> 172.21.15.12:0/4070334475 conn(0x7fd4ec08aff0 msgr2=0x7fd4ec0b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 -- 172.21.15.12:0/4070334475 shutdown_connections 2024-03-29T11:32:40.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.080+0000 7fd4f1400700 1 -- 172.21.15.12:0/4070334475 wait complete. 2024-03-29T11:32:40.093 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T11:32:40.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.295+0000 7f36ad200700 1 Processor -- start 2024-03-29T11:32:40.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.296+0000 7f36ad200700 1 -- start start 2024-03-29T11:32:40.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.297+0000 7f36ad200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a80709c0 0x7f36a8070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.297+0000 7f36ad200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a80b2180 con 0x7f36a80b3af0 2024-03-29T11:32:40.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.297+0000 7f36ad200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a815b100 con 0x7f36a8072580 2024-03-29T11:32:40.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.297+0000 7f36ad200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a815b3e0 con 0x7f36a80709c0 2024-03-29T11:32:40.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.297+0000 7f36a5e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f36a80b3af0 0x7f36a815a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53742/0 (socket says 172.21.15.12:53742) 2024-03-29T11:32:40.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.297+0000 7f36a5e00700 1 -- 172.21.15.12:0/1106961404 learned_addr learned my addr 172.21.15.12:0/1106961404 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:40.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.297+0000 7f36a6800700 1 --2- 172.21.15.12:0/1106961404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a80709c0 0x7f36a8070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.298+0000 7f36a4a00700 1 -- 172.21.15.12:0/1106961404 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442056818 0 0) 0x7f36a80b2180 con 0x7f36a80b3af0 2024-03-29T11:32:40.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.298+0000 7f36a4a00700 1 -- 172.21.15.12:0/1106961404 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f368c0035b0 con 0x7f36a80b3af0 2024-03-29T11:32:40.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.298+0000 7f36a4a00700 1 -- 172.21.15.12:0/1106961404 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633219040 0 0) 0x7f36a815b100 con 0x7f36a8072580 2024-03-29T11:32:40.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.298+0000 7f36a6800700 1 -- 172.21.15.12:0/1106961404 >> v1:172.21.15.159:6789/0 conn(0x7f36a8072580 legacy=0x7f36a80b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.298+0000 7f36a6800700 1 -- 172.21.15.12:0/1106961404 >> v1:172.21.15.12:6789/0 conn(0x7f36a80b3af0 legacy=0x7f36a815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.298+0000 7f36a6800700 1 -- 172.21.15.12:0/1106961404 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a80b23f0 con 0x7f36a80709c0 2024-03-29T11:32:40.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.298+0000 7f36a6800700 1 --2- 172.21.15.12:0/1106961404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a80709c0 0x7f36a8070e50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009dc0 tx=0x7f36a000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da6d7274253823b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:40.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.298+0000 7f36a4a00700 1 -- 172.21.15.12:0/1106961404 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a000e070 con 0x7f36a80709c0 2024-03-29T11:32:40.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.299+0000 7f36a4a00700 1 -- 172.21.15.12:0/1106961404 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a000bb20 con 0x7f36a80709c0 2024-03-29T11:32:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.299+0000 7f36a4a00700 1 -- 172.21.15.12:0/1106961404 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a000bed0 con 0x7f36a80709c0 2024-03-29T11:32:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.299+0000 7f36ad200700 1 -- 172.21.15.12:0/1106961404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a80709c0 msgr2=0x7f36a8070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.299+0000 7f36ad200700 1 --2- 172.21.15.12:0/1106961404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a80709c0 0x7f36a8070e50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009dc0 tx=0x7f36a000b330 comp rx=0 tx=0).stop 2024-03-29T11:32:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.299+0000 7f36ad200700 1 -- 172.21.15.12:0/1106961404 shutdown_connections 2024-03-29T11:32:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.299+0000 7f36ad200700 1 --2- 172.21.15.12:0/1106961404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a80709c0 0x7f36a8070e50 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.299+0000 7f36ad200700 1 -- 172.21.15.12:0/1106961404 >> 172.21.15.12:0/1106961404 conn(0x7f36a80cf190 msgr2=0x7f36a80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.299+0000 7f36ad200700 1 -- 172.21.15.12:0/1106961404 shutdown_connections 2024-03-29T11:32:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.299+0000 7f36ad200700 1 -- 172.21.15.12:0/1106961404 wait complete. 2024-03-29T11:32:40.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.301+0000 7f36ad200700 1 Processor -- start 2024-03-29T11:32:40.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.301+0000 7f36ad200700 1 -- start start 2024-03-29T11:32:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.302+0000 7f36ad200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36a80709c0 0x7f36a81f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.302+0000 7f36ad200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a8072580 0x7f36a814f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.302+0000 7f36ad200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36a80b3af0 0x7f36a814f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.302+0000 7f36ad200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a8154440 con 0x7f36a80709c0 2024-03-29T11:32:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.302+0000 7f36ad200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a81583f0 con 0x7f36a80b3af0 2024-03-29T11:32:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.302+0000 7f36ad200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a8158c90 con 0x7f36a8072580 2024-03-29T11:32:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.302+0000 7f36a5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36a80709c0 0x7f36a81f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.302+0000 7f36a5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36a80709c0 0x7f36a81f3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43306/0 (socket says 172.21.15.12:43306) 2024-03-29T11:32:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.302+0000 7f36a5e00700 1 -- 172.21.15.12:0/2006550188 learned_addr learned my addr 172.21.15.12:0/2006550188 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f36a5400700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a8072580 0x7f36a814f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f36a6800700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36a80b3af0 0x7f36a814f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f36a5e00700 1 -- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a8072580 msgr2=0x7f36a814f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f36a5e00700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a8072580 0x7f36a814f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f36a5e00700 1 -- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36a80b3af0 msgr2=0x7f36a814f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f36a5e00700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36a80b3af0 0x7f36a814f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f36a5e00700 1 -- 172.21.15.12:0/2006550188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0009a70 con 0x7f36a80709c0 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f36a6800700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36a80b3af0 0x7f36a814f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f36a5e00700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36a80709c0 0x7f36a81f3f70 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f369800c4d0 tx=0x7f369800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f369e000700 1 -- 172.21.15.12:0/2006550188 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698005d70 con 0x7f36a80709c0 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f369e000700 1 -- 172.21.15.12:0/2006550188 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36980070d0 con 0x7f36a80709c0 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f369e000700 1 -- 172.21.15.12:0/2006550188 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698007440 con 0x7f36a80709c0 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.303+0000 7f36ad200700 1 -- 172.21.15.12:0/2006550188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a8153690 con 0x7f36a80709c0 2024-03-29T11:32:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.304+0000 7f36ad200700 1 -- 172.21.15.12:0/2006550188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a8158640 con 0x7f36a80709c0 2024-03-29T11:32:40.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.304+0000 7f36ad200700 1 -- 172.21.15.12:0/2006550188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a8067720 con 0x7f36a80709c0 2024-03-29T11:32:40.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.305+0000 7f369e000700 1 -- 172.21.15.12:0/2006550188 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f36980075a0 con 0x7f36a80709c0 2024-03-29T11:32:40.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.307+0000 7f369e000700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f368806aed0 0x7f368806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:40.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.307+0000 7f369e000700 1 -- 172.21.15.12:0/2006550188 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f3698088c40 con 0x7f36a80709c0 2024-03-29T11:32:40.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.307+0000 7f36a5400700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f368806aed0 0x7f368806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:40.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.308+0000 7f36a5400700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f368806aed0 0x7f368806d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f36a8151180 tx=0x7f3694009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:40.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.312+0000 7f369e000700 1 -- 172.21.15.12:0/2006550188 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3698053a80 con 0x7f36a80709c0 2024-03-29T11:32:40.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.571+0000 7f36ad200700 1 -- 172.21.15.12:0/2006550188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f36a80cb980 con 0x7f36a80709c0 2024-03-29T11:32:40.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.572+0000 7f369e000700 1 -- 172.21.15.12:0/2006550188 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v122) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3698017020 con 0x7f36a80709c0 2024-03-29T11:32:40.567 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T11:32:40.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 -- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f368806aed0 msgr2=0x7f368806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f368806aed0 0x7f368806d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f36a8151180 tx=0x7f3694009040 comp rx=0 tx=0).stop 2024-03-29T11:32:40.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 -- 172.21.15.12:0/2006550188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36a80709c0 msgr2=0x7f36a81f3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36a80709c0 0x7f36a81f3f70 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f369800c4d0 tx=0x7f369800c890 comp rx=0 tx=0).stop 2024-03-29T11:32:40.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 -- 172.21.15.12:0/2006550188 shutdown_connections 2024-03-29T11:32:40.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f368806aed0 0x7f368806d390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36a80b3af0 0x7f36a814f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36a80709c0 0x7f36a81f3f70 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 --2- 172.21.15.12:0/2006550188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36a8072580 0x7f36a814f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:40.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 -- 172.21.15.12:0/2006550188 >> 172.21.15.12:0/2006550188 conn(0x7f36a80cf190 msgr2=0x7f36a80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:40.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 -- 172.21.15.12:0/2006550188 shutdown_connections 2024-03-29T11:32:40.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:40.574+0000 7f36ad200700 1 -- 172.21.15.12:0/2006550188 wait complete. 2024-03-29T11:32:40.588 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T11:32:40.589 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T11:32:40.589 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T11:32:40.589 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T11:32:40.589 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T11:32:40.589 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T11:32:40.589 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T11:32:40.589 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T11:32:40.589 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:32:40.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:40.589 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:32:40.635 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 35557sec preferred_lft 35557sec 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T11:32:40.636 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:40.638 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T11:32:40.740 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T11:32:40.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:40.743 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T11:32:40.791 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T11:32:40.791 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T11:32:40.791 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T11:32:40.792 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T11:32:40.792 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T11:32:40.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:40.792 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T11:32:40.793 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T11:32:40.793 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T11:32:40.793 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T11:32:40.793 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T11:32:40.793 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T11:32:40.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:40.960 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T11:32:41.005 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T11:32:41.005 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T11:32:41.005 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:32:41.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:41.006 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T11:32:41.114 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:32:41.114 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T11:32:41.159 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:32:41.159 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:32:41.159 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:32:41.243 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T11:32:41.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:41.243 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:32:41.386 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:32:41.386 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:32:41.412 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:32:41.413 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:32:41.413 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T11:32:41.413 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T11:32:41.413 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T11:32:41.418 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T11:32:41.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:41.418 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:32:41.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:41.486 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T11:32:41.590 INFO:teuthology.orchestra.run.smithi012.stdout:client11997 2024-03-29T11:32:41.590 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:32:41.590 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:32:41.636 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-03-29T11:32:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.884+0000 7fb050c00700 1 Processor -- start 2024-03-29T11:32:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.884+0000 7fb050c00700 1 -- start start 2024-03-29T11:32:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.885+0000 7fb050c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb04c0ced30 0x7fb04c0d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.885+0000 7fb050c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 0x7fb04c15f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.885+0000 7fb050c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c0ce8a0 con 0x7fb04c15ddf0 2024-03-29T11:32:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.885+0000 7fb050c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb04c15ec60 con 0x7fb04c0ced30 2024-03-29T11:32:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.885+0000 7fb050c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0cdc90 con 0x7fb04c0d2710 2024-03-29T11:32:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.885+0000 7fb049400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb04c0ced30 0x7fb04c0d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.885+0000 7fb049e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb04c15ddf0 0x7fb04c15e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53754/0 (socket says 172.21.15.12:53754) 2024-03-29T11:32:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.885+0000 7fb049e00700 1 -- 172.21.15.12:0/2386990414 learned_addr learned my addr 172.21.15.12:0/2386990414 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb04a800700 1 --2- 172.21.15.12:0/2386990414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 0x7fb04c15f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb049400700 1 -- 172.21.15.12:0/2386990414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 msgr2=0x7fb04c15f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb049400700 1 --2- 172.21.15.12:0/2386990414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 0x7fb04c15f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb049400700 1 -- 172.21.15.12:0/2386990414 >> v1:172.21.15.12:6789/0 conn(0x7fb04c15ddf0 legacy=0x7fb04c15e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb049400700 1 -- 172.21.15.12:0/2386990414 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c0cdf70 con 0x7fb04c0ced30 2024-03-29T11:32:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb048a00700 1 -- 172.21.15.12:0/2386990414 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290156752 0 0) 0x7fb04c0ce8a0 con 0x7fb04c15ddf0 2024-03-29T11:32:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb04a800700 1 --2- 172.21.15.12:0/2386990414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 0x7fb04c15f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb049400700 1 --2- 172.21.15.12:0/2386990414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 0x7fb04c0d21d0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fb038009840 tx=0x7fb038009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56eb97e5af0c57c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb048a00700 1 -- 172.21.15.12:0/2386990414 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb038011830 con 0x7fb04c0ced30 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb048a00700 1 -- 172.21.15.12:0/2386990414 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb038011990 con 0x7fb04c0ced30 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.886+0000 7fb048a00700 1 -- 172.21.15.12:0/2386990414 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb038011d20 con 0x7fb04c0ced30 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.887+0000 7fb050c00700 1 -- 172.21.15.12:0/2386990414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 msgr2=0x7fb04c0d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.887+0000 7fb050c00700 1 --2- 172.21.15.12:0/2386990414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 0x7fb04c0d21d0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fb038009840 tx=0x7fb038009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.887+0000 7fb050c00700 1 -- 172.21.15.12:0/2386990414 shutdown_connections 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.887+0000 7fb050c00700 1 --2- 172.21.15.12:0/2386990414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 0x7fb04c15f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.887+0000 7fb050c00700 1 --2- 172.21.15.12:0/2386990414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 0x7fb04c0d21d0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.887+0000 7fb050c00700 1 -- 172.21.15.12:0/2386990414 >> 172.21.15.12:0/2386990414 conn(0x7fb04c08aff0 msgr2=0x7fb04c0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.887+0000 7fb050c00700 1 -- 172.21.15.12:0/2386990414 shutdown_connections 2024-03-29T11:32:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.887+0000 7fb050c00700 1 -- 172.21.15.12:0/2386990414 wait complete. 2024-03-29T11:32:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.889+0000 7fb050c00700 1 Processor -- start 2024-03-29T11:32:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.890+0000 7fb050c00700 1 -- start start 2024-03-29T11:32:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb050c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 0x7fb04c0b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb050c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 0x7fb04c0c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb050c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c15ddf0 0x7fb04c1f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb050c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c1f7400 con 0x7fb04c0ced30 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb050c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0c80a0 con 0x7fb04c15ddf0 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb050c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0c7490 con 0x7fb04c0d2710 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb049e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 0x7fb04c0b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb049e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 0x7fb04c0b4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43334/0 (socket says 172.21.15.12:43334) 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb049e00700 1 -- 172.21.15.12:0/1149477576 learned_addr learned my addr 172.21.15.12:0/1149477576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb049400700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 0x7fb04c0c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.891+0000 7fb04a800700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c15ddf0 0x7fb04c1f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb049e00700 1 -- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 msgr2=0x7fb04c0c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb049e00700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 0x7fb04c0c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb049e00700 1 -- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c15ddf0 msgr2=0x7fb04c1f6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb049e00700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c15ddf0 0x7fb04c1f6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:41.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb049e00700 1 -- 172.21.15.12:0/1149477576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0380094a0 con 0x7fb04c0ced30 2024-03-29T11:32:41.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb049e00700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 0x7fb04c0b4620 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00c550 tx=0x7fb03c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:41.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb042000700 1 -- 172.21.15.12:0/1149477576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c004c80 con 0x7fb04c0ced30 2024-03-29T11:32:41.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb042000700 1 -- 172.21.15.12:0/1149477576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c004e00 con 0x7fb04c0ced30 2024-03-29T11:32:41.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb042000700 1 -- 172.21.15.12:0/1149477576 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c007310 con 0x7fb04c0ced30 2024-03-29T11:32:41.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb050c00700 1 -- 172.21.15.12:0/1149477576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04c0c7770 con 0x7fb04c0ced30 2024-03-29T11:32:41.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.892+0000 7fb050c00700 1 -- 172.21.15.12:0/1149477576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c0c7d30 con 0x7fb04c0ced30 2024-03-29T11:32:41.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.893+0000 7fb050c00700 1 -- 172.21.15.12:0/1149477576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04c067720 con 0x7fb04c0ced30 2024-03-29T11:32:41.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.894+0000 7fb042000700 1 -- 172.21.15.12:0/1149477576 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb03c007470 con 0x7fb04c0ced30 2024-03-29T11:32:41.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.895+0000 7fb042000700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02c06aed0 0x7fb02c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:41.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.895+0000 7fb042000700 1 -- 172.21.15.12:0/1149477576 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fb03c089510 con 0x7fb04c0ced30 2024-03-29T11:32:41.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.896+0000 7fb049400700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02c06aed0 0x7fb02c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:41.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.896+0000 7fb049400700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02c06aed0 0x7fb02c06d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb0380090b0 tx=0x7fb038008030 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:41.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:41.900+0000 7fb042000700 1 -- 172.21.15.12:0/1149477576 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb03c0542a0 con 0x7fb04c0ced30 2024-03-29T11:32:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.159+0000 7fb050c00700 1 -- 172.21.15.12:0/1149477576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb04c0b54d0 con 0x7fb04c0ced30 2024-03-29T11:32:42.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.634+0000 7fb042000700 1 -- 172.21.15.12:0/1149477576 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v123) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb03c0578c0 con 0x7fb04c0ced30 2024-03-29T11:32:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 -- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02c06aed0 msgr2=0x7fb02c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02c06aed0 0x7fb02c06d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb0380090b0 tx=0x7fb038008030 comp rx=0 tx=0).stop 2024-03-29T11:32:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 -- 172.21.15.12:0/1149477576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 msgr2=0x7fb04c0b4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 0x7fb04c0b4620 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00c550 tx=0x7fb03c00c910 comp rx=0 tx=0).stop 2024-03-29T11:32:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 -- 172.21.15.12:0/1149477576 shutdown_connections 2024-03-29T11:32:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02c06aed0 0x7fb02c06d390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04c15ddf0 0x7fb04c1f6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04c0ced30 0x7fb04c0b4620 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:42.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 --2- 172.21.15.12:0/1149477576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb04c0d2710 0x7fb04c0c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 -- 172.21.15.12:0/1149477576 >> 172.21.15.12:0/1149477576 conn(0x7fb04c08aff0 msgr2=0x7fb04c0b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 -- 172.21.15.12:0/1149477576 shutdown_connections 2024-03-29T11:32:42.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.637+0000 7fb050c00700 1 -- 172.21.15.12:0/1149477576 wait complete. 2024-03-29T11:32:42.653 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.858+0000 7f9d11200700 1 Processor -- start 2024-03-29T11:32:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.858+0000 7f9d11200700 1 -- start start 2024-03-29T11:32:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.859+0000 7f9d11200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c0ac7d0 0x7f9d0c166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.859+0000 7f9d11200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c15fd50 con 0x7f9d0c15b5d0 2024-03-29T11:32:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.859+0000 7f9d11200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c15f140 con 0x7f9d0c0cb3f0 2024-03-29T11:32:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.859+0000 7f9d11200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c15b9b0 con 0x7f9d0c0ac7d0 2024-03-29T11:32:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d09e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9d0c15b5d0 0x7f9d0c0ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53770/0 (socket says 172.21.15.12:53770) 2024-03-29T11:32:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d09e00700 1 -- 172.21.15.12:0/2337672671 learned_addr learned my addr 172.21.15.12:0/2337672671 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d0a800700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c0ac7d0 0x7f9d0c166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887961444 0 0) 0x7f9d0c15fd50 con 0x7f9d0c15b5d0 2024-03-29T11:32:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf00035b0 con 0x7f9d0c15b5d0 2024-03-29T11:32:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261928114 0 0) 0x7f9d0c15f140 con 0x7f9d0c0cb3f0 2024-03-29T11:32:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d0c15fd50 con 0x7f9d0c0cb3f0 2024-03-29T11:32:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3563163652 0 0) 0x7f9cf00035b0 con 0x7f9d0c15b5d0 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c0ac7d0 msgr2=0x7f9d0c166430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d08a00700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c0ac7d0 0x7f9d0c166430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 >> v1:172.21.15.159:6789/0 conn(0x7f9d0c0cb3f0 legacy=0x7f9d0c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c15f420 con 0x7f9d0c15b5d0 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.860+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f9cfc001670 con 0x7f9d0c15b5d0 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 >> v1:172.21.15.12:6789/0 conn(0x7f9d0c15b5d0 legacy=0x7f9d0c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d08a00700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c0ac7d0 0x7f9cf0003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d08a00700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cf0004120 0x7f9cf0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d08a00700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9cf0004d90 0x7f9cf0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c15b9b0 con 0x7f9cf0004d90 2024-03-29T11:32:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf000bba0 con 0x7f9cf0004120 2024-03-29T11:32:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf000bd30 con 0x7f9d0c0ac7d0 2024-03-29T11:32:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d09e00700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c0ac7d0 0x7f9cf0003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d09400700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cf0004120 0x7f9cf0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d0a800700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9cf0004d90 0x7f9cf0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d0a800700 1 -- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c0ac7d0 msgr2=0x7f9cf0003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d0a800700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c0ac7d0 0x7f9cf0003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d0a800700 1 -- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cf0004120 msgr2=0x7f9cf0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d0a800700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cf0004120 0x7f9cf0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.861+0000 7f9d0a800700 1 -- 172.21.15.12:0/2337672671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf00035b0 con 0x7f9cf0004d90 2024-03-29T11:32:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.862+0000 7f9d0a800700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9cf0004d90 0x7f9cf0005210 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009f90 tx=0x7f9cf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6758e10e5adcea4c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.862+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0400ec30 con 0x7f9cf0004d90 2024-03-29T11:32:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.862+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0400f040 con 0x7f9cf0004d90 2024-03-29T11:32:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.862+0000 7f9d08a00700 1 -- 172.21.15.12:0/2337672671 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04014480 con 0x7f9cf0004d90 2024-03-29T11:32:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.863+0000 7f9d11200700 1 -- 172.21.15.12:0/2337672671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9cf0004d90 msgr2=0x7f9cf0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.863+0000 7f9d11200700 1 --2- 172.21.15.12:0/2337672671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9cf0004d90 0x7f9cf0005210 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009f90 tx=0x7f9cf0005880 comp rx=0 tx=0).stop 2024-03-29T11:32:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.863+0000 7f9d0a800700 1 -- 172.21.15.12:0/2337672671 reap_dead start 2024-03-29T11:32:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.863+0000 7f9d11200700 1 -- 172.21.15.12:0/2337672671 shutdown_connections 2024-03-29T11:32:42.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.863+0000 7f9d11200700 1 -- 172.21.15.12:0/2337672671 >> 172.21.15.12:0/2337672671 conn(0x7f9d0c0cf190 msgr2=0x7f9d0c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:42.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.863+0000 7f9d11200700 1 -- 172.21.15.12:0/2337672671 shutdown_connections 2024-03-29T11:32:42.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.863+0000 7f9d11200700 1 -- 172.21.15.12:0/2337672671 wait complete. 2024-03-29T11:32:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.864+0000 7f9d11200700 1 Processor -- start 2024-03-29T11:32:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.865+0000 7f9d11200700 1 -- start start 2024-03-29T11:32:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.865+0000 7f9d11200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c156fa0 0x7f9d0c15cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.865+0000 7f9d11200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d0c15d190 0x7f9d0c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d11200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0c0cc0e0 0x7f9d0c0c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d11200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c06d220 con 0x7f9d0c0cc0e0 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d11200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c15d610 con 0x7f9d0c15d190 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d11200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c0cc560 con 0x7f9d0c156fa0 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d0a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0c0cc0e0 0x7f9d0c0c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d0a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0c0cc0e0 0x7f9d0c0c94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43342/0 (socket says 172.21.15.12:43342) 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d0a800700 1 -- 172.21.15.12:0/3201174303 learned_addr learned my addr 172.21.15.12:0/3201174303 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d09e00700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c156fa0 0x7f9d0c15cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d09400700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d0c15d190 0x7f9d0c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d0a800700 1 -- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c156fa0 msgr2=0x7f9d0c15cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d0a800700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c156fa0 0x7f9d0c15cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d0a800700 1 -- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d0c15d190 msgr2=0x7f9d0c0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d0a800700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d0c15d190 0x7f9d0c0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.866+0000 7f9d0a800700 1 -- 172.21.15.12:0/3201174303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c157380 con 0x7f9d0c0cc0e0 2024-03-29T11:32:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.867+0000 7f9d0a800700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0c0cc0e0 0x7f9d0c0c94b0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f9d04016000 tx=0x7f9d04009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.867+0000 7f9d02000700 1 -- 172.21.15.12:0/3201174303 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04007000 con 0x7f9d0c0cc0e0 2024-03-29T11:32:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.867+0000 7f9d02000700 1 -- 172.21.15.12:0/3201174303 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04003e50 con 0x7f9d0c0cc0e0 2024-03-29T11:32:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.867+0000 7f9d02000700 1 -- 172.21.15.12:0/3201174303 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04015a00 con 0x7f9d0c0cc0e0 2024-03-29T11:32:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.867+0000 7f9d11200700 1 -- 172.21.15.12:0/3201174303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c0cc7b0 con 0x7f9d0c0cc0e0 2024-03-29T11:32:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.867+0000 7f9d11200700 1 -- 172.21.15.12:0/3201174303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c0ccd70 con 0x7f9d0c0cc0e0 2024-03-29T11:32:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.868+0000 7f9d11200700 1 -- 172.21.15.12:0/3201174303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0c067720 con 0x7f9d0c0cc0e0 2024-03-29T11:32:42.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.869+0000 7f9d02000700 1 -- 172.21.15.12:0/3201174303 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f9d04024070 con 0x7f9d0c0cc0e0 2024-03-29T11:32:42.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.870+0000 7f9d02000700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9cec06aed0 0x7f9cec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:42.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.870+0000 7f9d02000700 1 -- 172.21.15.12:0/3201174303 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f9d0408ad80 con 0x7f9d0c0cc0e0 2024-03-29T11:32:42.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.871+0000 7f9d09e00700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9cec06aed0 0x7f9cec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:42.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.871+0000 7f9d09e00700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9cec06aed0 0x7f9cec06d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc001200 tx=0x7f9cfc001e20 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:42.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:42.875+0000 7f9d02000700 1 -- 172.21.15.12:0/3201174303 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d04055ba0 con 0x7f9d0c0cc0e0 2024-03-29T11:32:43.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.092+0000 7f9d11200700 1 -- 172.21.15.12:0/3201174303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d0c0cda60 con 0x7f9d0c0cc0e0 2024-03-29T11:32:43.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.093+0000 7f9d02000700 1 -- 172.21.15.12:0/3201174303 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+16167 (secure 0 0 0) 0x7f9d04018090 con 0x7f9d0c0cc0e0 2024-03-29T11:32:43.089 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:43.089 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":248,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:33.936359+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 -- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9cec06aed0 msgr2=0x7f9cec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9cec06aed0 0x7f9cec06d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc001200 tx=0x7f9cfc001e20 comp rx=0 tx=0).stop 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 -- 172.21.15.12:0/3201174303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0c0cc0e0 msgr2=0x7f9d0c0c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0c0cc0e0 0x7f9d0c0c94b0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f9d04016000 tx=0x7f9d04009750 comp rx=0 tx=0).stop 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 -- 172.21.15.12:0/3201174303 shutdown_connections 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9cec06aed0 0x7f9cec06d390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d0c0cc0e0 0x7f9d0c0c94b0 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0c156fa0 0x7f9d0c15cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 --2- 172.21.15.12:0/3201174303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d0c15d190 0x7f9d0c0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 -- 172.21.15.12:0/3201174303 >> 172.21.15.12:0/3201174303 conn(0x7f9d0c0cf190 msgr2=0x7f9d0c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:43.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 -- 172.21.15.12:0/3201174303 shutdown_connections 2024-03-29T11:32:43.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.096+0000 7f9d11200700 1 -- 172.21.15.12:0/3201174303 wait complete. 2024-03-29T11:32:43.111 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:32:31.368651+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:32:31.900686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:32:43.112 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:32:43.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.317+0000 7fdd19400700 1 Processor -- start 2024-03-29T11:32:43.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.317+0000 7fdd19400700 1 -- start start 2024-03-29T11:32:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.318+0000 7fdd19400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd140cbb30 0x7fdd140cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.318+0000 7fdd19400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdd140cd020 0x7fdd140cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.318+0000 7fdd19400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd140cc450 con 0x7fdd140b2070 2024-03-29T11:32:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.318+0000 7fdd19400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd140cdae0 con 0x7fdd140cd020 2024-03-29T11:32:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.318+0000 7fdd19400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140cddc0 con 0x7fdd140cbb30 2024-03-29T11:32:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.318+0000 7fdd11e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdd140cd020 0x7fdd140cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.318+0000 7fdd11e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdd140cd020 0x7fdd140cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43356/0 (socket says 172.21.15.12:43356) 2024-03-29T11:32:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.318+0000 7fdd11e00700 1 -- 172.21.15.12:0/3221079358 learned_addr learned my addr 172.21.15.12:0/3221079358 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.318+0000 7fdd12800700 1 --2- 172.21.15.12:0/3221079358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd140cbb30 0x7fdd140cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.319+0000 7fdd11e00700 1 -- 172.21.15.12:0/3221079358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd140cbb30 msgr2=0x7fdd140cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:43.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.319+0000 7fdd11e00700 1 --2- 172.21.15.12:0/3221079358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd140cbb30 0x7fdd140cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.319+0000 7fdd11e00700 1 -- 172.21.15.12:0/3221079358 >> v1:172.21.15.159:6789/0 conn(0x7fdd140b2070 legacy=0x7fdd1406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.319+0000 7fdd11400700 1 -- 172.21.15.12:0/3221079358 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422318415 0 0) 0x7fdd140cc450 con 0x7fdd140b2070 2024-03-29T11:32:43.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.319+0000 7fdd11e00700 1 -- 172.21.15.12:0/3221079358 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd140ce0a0 con 0x7fdd140cd020 2024-03-29T11:32:43.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.319+0000 7fdd11e00700 1 --2- 172.21.15.12:0/3221079358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cd020 0x7fdd140cd4a0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fdd000098f0 tx=0x7fdd00009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6537635419d86259 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.319+0000 7fdd11400700 1 -- 172.21.15.12:0/3221079358 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00011830 con 0x7fdd140cd020 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.319+0000 7fdd11400700 1 -- 172.21.15.12:0/3221079358 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd00011990 con 0x7fdd140cd020 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.319+0000 7fdd11400700 1 -- 172.21.15.12:0/3221079358 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00011d20 con 0x7fdd140cd020 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.320+0000 7fdd19400700 1 -- 172.21.15.12:0/3221079358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cd020 msgr2=0x7fdd140cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.320+0000 7fdd19400700 1 --2- 172.21.15.12:0/3221079358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cd020 0x7fdd140cd4a0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fdd000098f0 tx=0x7fdd00009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.320+0000 7fdd19400700 1 -- 172.21.15.12:0/3221079358 shutdown_connections 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.320+0000 7fdd19400700 1 --2- 172.21.15.12:0/3221079358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd140cbb30 0x7fdd140cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.320+0000 7fdd19400700 1 --2- 172.21.15.12:0/3221079358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cd020 0x7fdd140cd4a0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.320+0000 7fdd19400700 1 -- 172.21.15.12:0/3221079358 >> 172.21.15.12:0/3221079358 conn(0x7fdd140b0d70 msgr2=0x7fdd140b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.320+0000 7fdd19400700 1 -- 172.21.15.12:0/3221079358 shutdown_connections 2024-03-29T11:32:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.320+0000 7fdd19400700 1 -- 172.21.15.12:0/3221079358 wait complete. 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.322+0000 7fdd19400700 1 Processor -- start 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.322+0000 7fdd19400700 1 -- start start 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd19400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd140b2070 0x7fdd14144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd19400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cbb30 0x7fdd141f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd19400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd140cd020 0x7fdd1415dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd19400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd14071870 con 0x7fdd140cbb30 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd19400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd14144890 con 0x7fdd140b2070 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd19400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140709c0 con 0x7fdd140cd020 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd11e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cbb30 0x7fdd141f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd11e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cbb30 0x7fdd141f30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43362/0 (socket says 172.21.15.12:43362) 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd11e00700 1 -- 172.21.15.12:0/4114481318 learned_addr learned my addr 172.21.15.12:0/4114481318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd12800700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd140b2070 0x7fdd14144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.323+0000 7fdd13200700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd140cd020 0x7fdd1415dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd11e00700 1 -- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd140cd020 msgr2=0x7fdd1415dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd11e00700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd140cd020 0x7fdd1415dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd11e00700 1 -- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd140b2070 msgr2=0x7fdd14144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd11e00700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd140b2070 0x7fdd14144350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd11e00700 1 -- 172.21.15.12:0/4114481318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd00009550 con 0x7fdd140cbb30 2024-03-29T11:32:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd11e00700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cbb30 0x7fdd141f30b0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009c40 tx=0x7fdd00004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd0aa00700 1 -- 172.21.15.12:0/4114481318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00004510 con 0x7fdd140cbb30 2024-03-29T11:32:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd0aa00700 1 -- 172.21.15.12:0/4114481318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0001e070 con 0x7fdd140cbb30 2024-03-29T11:32:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd0aa00700 1 -- 172.21.15.12:0/4114481318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd000038c0 con 0x7fdd140cbb30 2024-03-29T11:32:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd19400700 1 -- 172.21.15.12:0/4114481318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd14070c60 con 0x7fdd140cbb30 2024-03-29T11:32:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.324+0000 7fdd19400700 1 -- 172.21.15.12:0/4114481318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd14071220 con 0x7fdd140cbb30 2024-03-29T11:32:43.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.325+0000 7fdd19400700 1 -- 172.21.15.12:0/4114481318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd14067720 con 0x7fdd140cbb30 2024-03-29T11:32:43.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.326+0000 7fdd0aa00700 1 -- 172.21.15.12:0/4114481318 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fdd00003a20 con 0x7fdd140cbb30 2024-03-29T11:32:43.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.327+0000 7fdd0aa00700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcf406aed0 0x7fdcf406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.327+0000 7fdd0aa00700 1 -- 172.21.15.12:0/4114481318 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fdd0008a5b0 con 0x7fdd140cbb30 2024-03-29T11:32:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.327+0000 7fdd12800700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcf406aed0 0x7fdcf406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.328+0000 7fdd12800700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcf406aed0 0x7fdcf406d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009b40 tx=0x7fdd04009210 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:43.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.331+0000 7fdd0aa00700 1 -- 172.21.15.12:0/4114481318 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd00055420 con 0x7fdd140cbb30 2024-03-29T11:32:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.543+0000 7fdd19400700 1 -- 172.21.15.12:0/4114481318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd140719b0 con 0x7fdd140cbb30 2024-03-29T11:32:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.544+0000 7fdd0aa00700 1 -- 172.21.15.12:0/4114481318 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v248) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd140719b0 con 0x7fdd140cbb30 2024-03-29T11:32:43.539 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:32:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 -- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcf406aed0 msgr2=0x7fdcf406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcf406aed0 0x7fdcf406d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009b40 tx=0x7fdd04009210 comp rx=0 tx=0).stop 2024-03-29T11:32:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 -- 172.21.15.12:0/4114481318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cbb30 msgr2=0x7fdd141f30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cbb30 0x7fdd141f30b0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009c40 tx=0x7fdd00004750 comp rx=0 tx=0).stop 2024-03-29T11:32:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 -- 172.21.15.12:0/4114481318 shutdown_connections 2024-03-29T11:32:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdcf406aed0 0x7fdcf406d390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd140cd020 0x7fdd1415dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd140b2070 0x7fdd14144350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 --2- 172.21.15.12:0/4114481318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd140cbb30 0x7fdd141f30b0 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 -- 172.21.15.12:0/4114481318 >> 172.21.15.12:0/4114481318 conn(0x7fdd140b0d70 msgr2=0x7fdd140af990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 -- 172.21.15.12:0/4114481318 shutdown_connections 2024-03-29T11:32:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.546+0000 7fdd19400700 1 -- 172.21.15.12:0/4114481318 wait complete. 2024-03-29T11:32:43.560 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:32:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.766+0000 7f67f1000700 1 Processor -- start 2024-03-29T11:32:43.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.766+0000 7f67f1000700 1 -- start start 2024-03-29T11:32:43.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.767+0000 7f67f1000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec15d9d0 0x7f67ec0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.767+0000 7f67f1000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67ec0d2540 0x7f67ec0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.767+0000 7f67f1000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ec161420 con 0x7f67ec0d16a0 2024-03-29T11:32:43.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.767+0000 7f67f1000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67ec15edc0 con 0x7f67ec0d2540 2024-03-29T11:32:43.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.767+0000 7f67f1000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec161700 con 0x7f67ec15d9d0 2024-03-29T11:32:43.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.767+0000 7f67ea800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67ec0d2540 0x7f67ec0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.767+0000 7f67ea800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67ec0d2540 0x7f67ec0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43372/0 (socket says 172.21.15.12:43372) 2024-03-29T11:32:43.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.767+0000 7f67ea800700 1 -- 172.21.15.12:0/1075725023 learned_addr learned my addr 172.21.15.12:0/1075725023 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:43.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.768+0000 7f67e9e00700 1 --2- 172.21.15.12:0/1075725023 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec15d9d0 0x7f67ec0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.768+0000 7f67ea800700 1 -- 172.21.15.12:0/1075725023 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec15d9d0 msgr2=0x7f67ec0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.768+0000 7f67ea800700 1 --2- 172.21.15.12:0/1075725023 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec15d9d0 0x7f67ec0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.768+0000 7f67ea800700 1 -- 172.21.15.12:0/1075725023 >> v1:172.21.15.159:6789/0 conn(0x7f67ec0d16a0 legacy=0x7f67ec0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.768+0000 7f67ea800700 1 -- 172.21.15.12:0/1075725023 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec1619e0 con 0x7f67ec0d2540 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.768+0000 7f67e9e00700 1 --2- 172.21.15.12:0/1075725023 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec15d9d0 0x7f67ec0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.768+0000 7f67ea800700 1 --2- 172.21.15.12:0/1075725023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 0x7f67ec0ce360 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f67e4009e80 tx=0x7f67e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a504e06935b501 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.768+0000 7f67e8a00700 1 -- 172.21.15.12:0/1075725023 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e400e070 con 0x7f67ec0d2540 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.768+0000 7f67e8a00700 1 -- 172.21.15.12:0/1075725023 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e400b980 con 0x7f67ec0d2540 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.768+0000 7f67e8a00700 1 -- 172.21.15.12:0/1075725023 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e400bd30 con 0x7f67ec0d2540 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.769+0000 7f67f1000700 1 -- 172.21.15.12:0/1075725023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 msgr2=0x7f67ec0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.769+0000 7f67f1000700 1 --2- 172.21.15.12:0/1075725023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 0x7f67ec0ce360 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f67e4009e80 tx=0x7f67e400b330 comp rx=0 tx=0).stop 2024-03-29T11:32:43.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.769+0000 7f67f1000700 1 -- 172.21.15.12:0/1075725023 shutdown_connections 2024-03-29T11:32:43.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.769+0000 7f67f1000700 1 --2- 172.21.15.12:0/1075725023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 0x7f67ec0ce360 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.769+0000 7f67f1000700 1 --2- 172.21.15.12:0/1075725023 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec15d9d0 0x7f67ec0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.769+0000 7f67f1000700 1 -- 172.21.15.12:0/1075725023 >> 172.21.15.12:0/1075725023 conn(0x7f67ec08aff0 msgr2=0x7f67ec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:43.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.769+0000 7f67f1000700 1 -- 172.21.15.12:0/1075725023 shutdown_connections 2024-03-29T11:32:43.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.769+0000 7f67f1000700 1 -- 172.21.15.12:0/1075725023 wait complete. 2024-03-29T11:32:43.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.771+0000 7f67f1000700 1 Processor -- start 2024-03-29T11:32:43.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.771+0000 7f67f1000700 1 -- start start 2024-03-29T11:32:43.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67f1000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec0d16a0 0x7f67ec15c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67f1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 0x7f67ec0cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67f1000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67ec15d9d0 0x7f67ec0b78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67f1000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec0cc4b0 con 0x7f67ec0d2540 2024-03-29T11:32:43.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67f1000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec15c8f0 con 0x7f67ec15d9d0 2024-03-29T11:32:43.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67f1000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec0cb5c0 con 0x7f67ec0d16a0 2024-03-29T11:32:43.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67e9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 0x7f67ec0cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67e9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 0x7f67ec0cc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43382/0 (socket says 172.21.15.12:43382) 2024-03-29T11:32:43.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67e9400700 1 -- 172.21.15.12:0/40157997 learned_addr learned my addr 172.21.15.12:0/40157997 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:43.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67e9e00700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec0d16a0 0x7f67ec15c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.772+0000 7f67ea800700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67ec15d9d0 0x7f67ec0b78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67e9400700 1 -- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec0d16a0 msgr2=0x7f67ec15c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67e9400700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec0d16a0 0x7f67ec15c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67e9400700 1 -- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67ec15d9d0 msgr2=0x7f67ec0b78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67e9400700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67ec15d9d0 0x7f67ec0b78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67e9400700 1 -- 172.21.15.12:0/40157997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e4009b30 con 0x7f67ec0d2540 2024-03-29T11:32:43.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67e9400700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 0x7f67ec0cc940 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f67d800c500 tx=0x7f67d800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:43.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67e2000700 1 -- 172.21.15.12:0/40157997 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d80052d0 con 0x7f67ec0d2540 2024-03-29T11:32:43.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67e2000700 1 -- 172.21.15.12:0/40157997 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d8005450 con 0x7f67ec0d2540 2024-03-29T11:32:43.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67e2000700 1 -- 172.21.15.12:0/40157997 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8003910 con 0x7f67ec0d2540 2024-03-29T11:32:43.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67f1000700 1 -- 172.21.15.12:0/40157997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ec0cb8a0 con 0x7f67ec0d2540 2024-03-29T11:32:43.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.773+0000 7f67f1000700 1 -- 172.21.15.12:0/40157997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ec0cbe60 con 0x7f67ec0d2540 2024-03-29T11:32:43.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.774+0000 7f67f1000700 1 -- 172.21.15.12:0/40157997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ec0d1ca0 con 0x7f67ec0d2540 2024-03-29T11:32:43.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.775+0000 7f67e2000700 1 -- 172.21.15.12:0/40157997 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f67d8003a70 con 0x7f67ec0d2540 2024-03-29T11:32:43.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.776+0000 7f67e2000700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67cc06aed0 0x7f67cc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:43.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.776+0000 7f67e2000700 1 -- 172.21.15.12:0/40157997 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f67d8088a40 con 0x7f67ec0d2540 2024-03-29T11:32:43.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.777+0000 7f67e9e00700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67cc06aed0 0x7f67cc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:43.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.777+0000 7f67e9e00700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67cc06aed0 0x7f67cc06d390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0b2910 tx=0x7f67dc008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:43.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.781+0000 7f67e2000700 1 -- 172.21.15.12:0/40157997 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67d808ae60 con 0x7f67ec0d2540 2024-03-29T11:32:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.994+0000 7f67f1000700 1 -- 172.21.15.12:0/40157997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f67ec067720 con 0x7f67ec0d2540 2024-03-29T11:32:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.994+0000 7f67e2000700 1 -- 172.21.15.12:0/40157997 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v248) v1 ==== 104+0+11 (secure 0 0 0) 0x7f67d8017020 con 0x7f67ec0d2540 2024-03-29T11:32:43.990 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:32:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 -- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67cc06aed0 msgr2=0x7f67cc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67cc06aed0 0x7f67cc06d390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0b2910 tx=0x7f67dc008040 comp rx=0 tx=0).stop 2024-03-29T11:32:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 -- 172.21.15.12:0/40157997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 msgr2=0x7f67ec0cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 0x7f67ec0cc940 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f67d800c500 tx=0x7f67d800c810 comp rx=0 tx=0).stop 2024-03-29T11:32:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 -- 172.21.15.12:0/40157997 shutdown_connections 2024-03-29T11:32:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f67cc06aed0 0x7f67cc06d390 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f67ec15d9d0 0x7f67ec0b78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67ec0d16a0 0x7f67ec15c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 --2- 172.21.15.12:0/40157997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67ec0d2540 0x7f67ec0cc940 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 -- 172.21.15.12:0/40157997 >> 172.21.15.12:0/40157997 conn(0x7f67ec08aff0 msgr2=0x7f67ec15fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 -- 172.21.15.12:0/40157997 shutdown_connections 2024-03-29T11:32:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:43.997+0000 7f67f1000700 1 -- 172.21.15.12:0/40157997 wait complete. 2024-03-29T11:32:44.010 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:32:44.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.215+0000 7f8977c00700 1 Processor -- start 2024-03-29T11:32:44.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.215+0000 7f8977c00700 1 -- start start 2024-03-29T11:32:44.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.216+0000 7f8977c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f89700b3f80 0x7f89700ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.216+0000 7f8977c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8970161230 con 0x7f8970160e50 2024-03-29T11:32:44.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.216+0000 7f8977c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89700d2a80 con 0x7f89700d3f20 2024-03-29T11:32:44.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.216+0000 7f8977c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89700d43b0 con 0x7f89700b3f80 2024-03-29T11:32:44.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.217+0000 7f8975c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f89700b3f80 0x7f89700ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.217+0000 7f8975200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8970160e50 0x7f89700b4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53772/0 (socket says 172.21.15.12:53772) 2024-03-29T11:32:44.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.217+0000 7f8975200700 1 -- 172.21.15.12:0/125225673 learned_addr learned my addr 172.21.15.12:0/125225673 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:44.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.217+0000 7f8975c00700 1 -- 172.21.15.12:0/125225673 >> v1:172.21.15.159:6789/0 conn(0x7f89700d3f20 legacy=0x7f89700cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.217+0000 7f8975c00700 1 -- 172.21.15.12:0/125225673 >> v1:172.21.15.12:6789/0 conn(0x7f8970160e50 legacy=0x7f89700b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.217+0000 7f8975c00700 1 -- 172.21.15.12:0/125225673 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89700d21e0 con 0x7f89700b3f80 2024-03-29T11:32:44.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.217+0000 7f896f400700 1 -- 172.21.15.12:0/125225673 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655851713 0 0) 0x7f8970161230 con 0x7f8970160e50 2024-03-29T11:32:44.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.217+0000 7f8975c00700 1 --2- 172.21.15.12:0/125225673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89700b3f80 0x7f89700ce600 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f8968009d90 tx=0x7f896800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d760df1ad684c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:44.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.218+0000 7f896f400700 1 -- 172.21.15.12:0/125225673 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896800e070 con 0x7f89700b3f80 2024-03-29T11:32:44.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.218+0000 7f896f400700 1 -- 172.21.15.12:0/125225673 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896800bbd0 con 0x7f89700b3f80 2024-03-29T11:32:44.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.218+0000 7f896f400700 1 -- 172.21.15.12:0/125225673 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968012430 con 0x7f89700b3f80 2024-03-29T11:32:44.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.218+0000 7f8977c00700 1 -- 172.21.15.12:0/125225673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89700b3f80 msgr2=0x7f89700ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.218+0000 7f8977c00700 1 --2- 172.21.15.12:0/125225673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89700b3f80 0x7f89700ce600 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f8968009d90 tx=0x7f896800b330 comp rx=0 tx=0).stop 2024-03-29T11:32:44.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.218+0000 7f8977c00700 1 -- 172.21.15.12:0/125225673 shutdown_connections 2024-03-29T11:32:44.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.218+0000 7f8977c00700 1 --2- 172.21.15.12:0/125225673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89700b3f80 0x7f89700ce600 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.218+0000 7f8977c00700 1 -- 172.21.15.12:0/125225673 >> 172.21.15.12:0/125225673 conn(0x7f897008aff0 msgr2=0x7f89700d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:44.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.218+0000 7f8977c00700 1 -- 172.21.15.12:0/125225673 shutdown_connections 2024-03-29T11:32:44.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.218+0000 7f8977c00700 1 -- 172.21.15.12:0/125225673 wait complete. 2024-03-29T11:32:44.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.221+0000 7f8977c00700 1 Processor -- start 2024-03-29T11:32:44.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.221+0000 7f8977c00700 1 -- start start 2024-03-29T11:32:44.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.222+0000 7f8977c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89700b3f80 0x7f89700b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.222+0000 7f8977c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8970160e50 0x7f89700ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.222+0000 7f8977c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f897006c100 0x7f897006c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.222+0000 7f8977c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89701f4240 con 0x7f897006c100 2024-03-29T11:32:44.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.222+0000 7f8977c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897006cad0 con 0x7f89700b3f80 2024-03-29T11:32:44.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.222+0000 7f8977c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89700b7870 con 0x7f8970160e50 2024-03-29T11:32:44.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.222+0000 7f8975200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89700b3f80 0x7f89700b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f8975c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f897006c100 0x7f897006c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f8975200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89700b3f80 0x7f89700b7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:54234/0 (socket says 172.21.15.12:54234) 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f8975c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f897006c100 0x7f897006c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43398/0 (socket says 172.21.15.12:43398) 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f8975200700 1 -- 172.21.15.12:0/3566296560 learned_addr learned my addr 172.21.15.12:0/3566296560 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f896fe00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8970160e50 0x7f89700ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f8975c00700 1 -- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8970160e50 msgr2=0x7f89700ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f8975c00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8970160e50 0x7f89700ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f8975c00700 1 -- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89700b3f80 msgr2=0x7f89700b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f8975c00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89700b3f80 0x7f89700b7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f8975c00700 1 -- 172.21.15.12:0/3566296560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8968009a70 con 0x7f897006c100 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f8975c00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f897006c100 0x7f897006c590 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f8968016000 tx=0x7f8968004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.223+0000 7f896cc00700 1 -- 172.21.15.12:0/3566296560 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896801b810 con 0x7f897006c100 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.224+0000 7f896cc00700 1 -- 172.21.15.12:0/3566296560 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8968020070 con 0x7f897006c100 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.224+0000 7f896cc00700 1 -- 172.21.15.12:0/3566296560 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896801bbf0 con 0x7f897006c100 2024-03-29T11:32:44.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.224+0000 7f8977c00700 1 -- 172.21.15.12:0/3566296560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89701f3f60 con 0x7f897006c100 2024-03-29T11:32:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.224+0000 7f8977c00700 1 -- 172.21.15.12:0/3566296560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89700b7500 con 0x7f897006c100 2024-03-29T11:32:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.225+0000 7f8977c00700 1 -- 172.21.15.12:0/3566296560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89700b7690 con 0x7f897006c100 2024-03-29T11:32:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.226+0000 7f896cc00700 1 -- 172.21.15.12:0/3566296560 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f896801bd50 con 0x7f897006c100 2024-03-29T11:32:44.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.226+0000 7f896cc00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f895006aed0 0x7f895006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.227+0000 7f896cc00700 1 -- 172.21.15.12:0/3566296560 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f8968022070 con 0x7f897006c100 2024-03-29T11:32:44.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.227+0000 7f8975200700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f895006aed0 0x7f895006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.228+0000 7f8975200700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f895006aed0 0x7f895006d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8964000f60 tx=0x7f8964004a50 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.231+0000 7f896cc00700 1 -- 172.21.15.12:0/3566296560 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8968056950 con 0x7f897006c100 2024-03-29T11:32:44.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.445+0000 7f8977c00700 1 -- 172.21.15.12:0/3566296560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f89700681a0 con 0x7f897006c100 2024-03-29T11:32:44.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.446+0000 7f896cc00700 1 -- 172.21.15.12:0/3566296560 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v248) v1 ==== 100+0+11 (secure 0 0 0) 0x7f896802c0a0 con 0x7f897006c100 2024-03-29T11:32:44.441 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:32:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 -- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f895006aed0 msgr2=0x7f895006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f895006aed0 0x7f895006d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8964000f60 tx=0x7f8964004a50 comp rx=0 tx=0).stop 2024-03-29T11:32:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 -- 172.21.15.12:0/3566296560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f897006c100 msgr2=0x7f897006c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f897006c100 0x7f897006c590 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f8968016000 tx=0x7f8968004ad0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 -- 172.21.15.12:0/3566296560 shutdown_connections 2024-03-29T11:32:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f895006aed0 0x7f895006d390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f897006c100 0x7f897006c590 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89700b3f80 0x7f89700b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 --2- 172.21.15.12:0/3566296560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8970160e50 0x7f89700ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 -- 172.21.15.12:0/3566296560 >> 172.21.15.12:0/3566296560 conn(0x7f897008aff0 msgr2=0x7f89700d3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.448+0000 7f8977c00700 1 -- 172.21.15.12:0/3566296560 shutdown_connections 2024-03-29T11:32:44.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.449+0000 7f8977c00700 1 -- 172.21.15.12:0/3566296560 wait complete. 2024-03-29T11:32:44.462 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T11:32:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.667+0000 7fb877a00700 1 Processor -- start 2024-03-29T11:32:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.667+0000 7fb877a00700 1 -- start start 2024-03-29T11:32:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.668+0000 7fb877a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb8700b3af0 0x7fb87015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.668+0000 7fb877a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb8700709c0 0x7fb870070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.668+0000 7fb877a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8700b2180 con 0x7fb870072580 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.668+0000 7fb877a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87015b100 con 0x7fb8700709c0 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.668+0000 7fb877a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87015b3e0 con 0x7fb8700b3af0 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.669+0000 7fb875a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb8700709c0 0x7fb870070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.669+0000 7fb875a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb8700709c0 0x7fb870070e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43404/0 (socket says 172.21.15.12:43404) 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.669+0000 7fb875a00700 1 -- 172.21.15.12:0/3127696732 learned_addr learned my addr 172.21.15.12:0/3127696732 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.669+0000 7fb875000700 1 --2- 172.21.15.12:0/3127696732 >> v2:172.21.15.159:3300/0 conn(0x7fb8700b3af0 0x7fb87015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.669+0000 7fb875a00700 1 -- 172.21.15.12:0/3127696732 >> v2:172.21.15.159:3300/0 conn(0x7fb8700b3af0 msgr2=0x7fb87015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.669+0000 7fb875a00700 1 --2- 172.21.15.12:0/3127696732 >> v2:172.21.15.159:3300/0 conn(0x7fb8700b3af0 0x7fb87015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.669+0000 7fb875a00700 1 -- 172.21.15.12:0/3127696732 >> v1:172.21.15.159:6789/0 conn(0x7fb870072580 legacy=0x7fb8700b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.669+0000 7fb875a00700 1 -- 172.21.15.12:0/3127696732 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8700b23f0 con 0x7fb8700709c0 2024-03-29T11:32:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.669+0000 7fb875a00700 1 --2- 172.21.15.12:0/3127696732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 0x7fb870070e00 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fb87006b860 tx=0x7fb86800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea25a896742b8151 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.669+0000 7fb86f400700 1 -- 172.21.15.12:0/3127696732 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86800e070 con 0x7fb8700709c0 2024-03-29T11:32:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.670+0000 7fb86f400700 1 -- 172.21.15.12:0/3127696732 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868003a20 con 0x7fb8700709c0 2024-03-29T11:32:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.670+0000 7fb86f400700 1 -- 172.21.15.12:0/3127696732 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868003dd0 con 0x7fb8700709c0 2024-03-29T11:32:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.670+0000 7fb877a00700 1 -- 172.21.15.12:0/3127696732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 msgr2=0x7fb870070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.670+0000 7fb877a00700 1 --2- 172.21.15.12:0/3127696732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 0x7fb870070e00 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fb87006b860 tx=0x7fb86800b330 comp rx=0 tx=0).stop 2024-03-29T11:32:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.670+0000 7fb877a00700 1 -- 172.21.15.12:0/3127696732 shutdown_connections 2024-03-29T11:32:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.670+0000 7fb877a00700 1 --2- 172.21.15.12:0/3127696732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 0x7fb870070e00 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.670+0000 7fb877a00700 1 --2- 172.21.15.12:0/3127696732 >> v2:172.21.15.159:3300/0 conn(0x7fb8700b3af0 0x7fb87015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.670+0000 7fb877a00700 1 -- 172.21.15.12:0/3127696732 >> 172.21.15.12:0/3127696732 conn(0x7fb8700cf190 msgr2=0x7fb8700cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.670+0000 7fb877a00700 1 -- 172.21.15.12:0/3127696732 shutdown_connections 2024-03-29T11:32:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.670+0000 7fb877a00700 1 -- 172.21.15.12:0/3127696732 wait complete. 2024-03-29T11:32:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.672+0000 7fb877a00700 1 Processor -- start 2024-03-29T11:32:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.672+0000 7fb877a00700 1 -- start start 2024-03-29T11:32:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.673+0000 7fb877a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 0x7fb870148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.673+0000 7fb877a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb870072580 0x7fb8701f1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.673+0000 7fb877a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8700b3af0 0x7fb8700ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.673+0000 7fb877a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8701f15b0 con 0x7fb8700709c0 2024-03-29T11:32:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.673+0000 7fb877a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb870156df0 con 0x7fb8700b3af0 2024-03-29T11:32:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.673+0000 7fb877a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8701570d0 con 0x7fb870072580 2024-03-29T11:32:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.673+0000 7fb875000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 0x7fb870148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.673+0000 7fb875000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 0x7fb870148650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43418/0 (socket says 172.21.15.12:43418) 2024-03-29T11:32:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.673+0000 7fb875000700 1 -- 172.21.15.12:0/1817703204 learned_addr learned my addr 172.21.15.12:0/1817703204 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb875a00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8700b3af0 0x7fb8700ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb86fe00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb870072580 0x7fb8701f1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb875000700 1 -- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb870072580 msgr2=0x7fb8701f1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb875000700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb870072580 0x7fb8701f1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb875000700 1 -- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8700b3af0 msgr2=0x7fb8700ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb875000700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8700b3af0 0x7fb8700ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb875000700 1 -- 172.21.15.12:0/1817703204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb868009b30 con 0x7fb8700709c0 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb875a00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8700b3af0 0x7fb8700ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb875000700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 0x7fb870148650 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fb86400d800 tx=0x7fb86400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb86cc00700 1 -- 172.21.15.12:0/1817703204 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8640125f0 con 0x7fb8700709c0 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb86cc00700 1 -- 172.21.15.12:0/1817703204 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb864012750 con 0x7fb8700709c0 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb86cc00700 1 -- 172.21.15.12:0/1817703204 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb864012b00 con 0x7fb8700709c0 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb877a00700 1 -- 172.21.15.12:0/1817703204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8701573b0 con 0x7fb8700709c0 2024-03-29T11:32:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.674+0000 7fb877a00700 1 -- 172.21.15.12:0/1817703204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8700cbe80 con 0x7fb8700709c0 2024-03-29T11:32:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.675+0000 7fb877a00700 1 -- 172.21.15.12:0/1817703204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8700cc040 con 0x7fb8700709c0 2024-03-29T11:32:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.677+0000 7fb86cc00700 1 -- 172.21.15.12:0/1817703204 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb864012c60 con 0x7fb8700709c0 2024-03-29T11:32:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.677+0000 7fb86cc00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb85006aed0 0x7fb85006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.677+0000 7fb86cc00700 1 -- 172.21.15.12:0/1817703204 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fb864089be0 con 0x7fb8700709c0 2024-03-29T11:32:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.678+0000 7fb86fe00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb85006aed0 0x7fb85006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.679+0000 7fb86fe00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb85006aed0 0x7fb85006d390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb87015a7f0 tx=0x7fb85c009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.682+0000 7fb86cc00700 1 -- 172.21.15.12:0/1817703204 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb864054970 con 0x7fb8700709c0 2024-03-29T11:32:44.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.896+0000 7fb877a00700 1 -- 172.21.15.12:0/1817703204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb8700cc1f0 con 0x7fb8700709c0 2024-03-29T11:32:44.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.896+0000 7fb86cc00700 1 -- 172.21.15.12:0/1817703204 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v248) v1 ==== 84+0+209 (secure 0 0 0) 0x7fb864057f90 con 0x7fb8700709c0 2024-03-29T11:32:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:44.892 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:32:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T11:32:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T11:32:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 30, 2024-03-29T11:32:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": "cephfs_metadata" 2024-03-29T11:32:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 31, 2024-03-29T11:32:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": "cephfs_data" 2024-03-29T11:32:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:44.893 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:32:44.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.898+0000 7fb877a00700 1 -- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb85006aed0 msgr2=0x7fb85006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.898+0000 7fb877a00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb85006aed0 0x7fb85006d390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb87015a7f0 tx=0x7fb85c009040 comp rx=0 tx=0).stop 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.899+0000 7fb877a00700 1 -- 172.21.15.12:0/1817703204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 msgr2=0x7fb870148650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.899+0000 7fb877a00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 0x7fb870148650 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fb86400d800 tx=0x7fb86400dbc0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.899+0000 7fb877a00700 1 -- 172.21.15.12:0/1817703204 shutdown_connections 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.899+0000 7fb877a00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb85006aed0 0x7fb85006d390 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.899+0000 7fb877a00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8700b3af0 0x7fb8700ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.899+0000 7fb877a00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb8700709c0 0x7fb870148650 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.899+0000 7fb877a00700 1 --2- 172.21.15.12:0/1817703204 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb870072580 0x7fb8701f1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.899+0000 7fb877a00700 1 -- 172.21.15.12:0/1817703204 >> 172.21.15.12:0/1817703204 conn(0x7fb8700cf190 msgr2=0x7fb8700713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.899+0000 7fb877a00700 1 -- 172.21.15.12:0/1817703204 shutdown_connections 2024-03-29T11:32:44.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:44.899+0000 7fb877a00700 1 -- 172.21.15.12:0/1817703204 wait complete. 2024-03-29T11:32:44.912 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-03-29T11:32:44.913 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-03-29T11:32:45.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.121+0000 7f766ec00700 1 Processor -- start 2024-03-29T11:32:45.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.121+0000 7f766ec00700 1 -- start start 2024-03-29T11:32:45.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.122+0000 7f766ec00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 0x7f7668166470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:45.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.122+0000 7f766ec00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766815dcc0 con 0x7f7668155ac0 2024-03-29T11:32:45.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.122+0000 7f766ec00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766806dd00 con 0x7f766815c700 2024-03-29T11:32:45.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.122+0000 7f766ec00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766806d0f0 con 0x7f76680ae670 2024-03-29T11:32:45.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.122+0000 7f7667e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7668155ac0 0x7f766806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53782/0 (socket says 172.21.15.12:53782) 2024-03-29T11:32:45.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.122+0000 7f7667e00700 1 -- 172.21.15.12:0/2951144296 learned_addr learned my addr 172.21.15.12:0/2951144296 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:45.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.122+0000 7f766cc00700 1 --2- 172.21.15.12:0/2951144296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 0x7f7668166470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:45.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.123+0000 7f7666a00700 1 -- 172.21.15.12:0/2951144296 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247135183 0 0) 0x7f766815dcc0 con 0x7f7668155ac0 2024-03-29T11:32:45.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.123+0000 7f7666a00700 1 -- 172.21.15.12:0/2951144296 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f764c0035b0 con 0x7f7668155ac0 2024-03-29T11:32:45.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.123+0000 7f7666a00700 1 -- 172.21.15.12:0/2951144296 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187395935 0 0) 0x7f766806dd00 con 0x7f766815c700 2024-03-29T11:32:45.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.123+0000 7f7666a00700 1 -- 172.21.15.12:0/2951144296 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f766815dcc0 con 0x7f766815c700 2024-03-29T11:32:45.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.123+0000 7f766cc00700 1 -- 172.21.15.12:0/2951144296 >> v1:172.21.15.159:6789/0 conn(0x7f766815c700 legacy=0x7f76680adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.123+0000 7f766cc00700 1 -- 172.21.15.12:0/2951144296 >> v1:172.21.15.12:6789/0 conn(0x7f7668155ac0 legacy=0x7f766806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.123+0000 7f7666a00700 1 -- 172.21.15.12:0/2951144296 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2052600620 0 0) 0x7f764c0035b0 con 0x7f7668155ac0 2024-03-29T11:32:45.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.123+0000 7f766cc00700 1 -- 172.21.15.12:0/2951144296 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766806d3d0 con 0x7f76680ae670 2024-03-29T11:32:45.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.123+0000 7f766cc00700 1 --2- 172.21.15.12:0/2951144296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 0x7f7668166470 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7660009dc0 tx=0x7f766000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1347dbcf5878720 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:45.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.124+0000 7f7666a00700 1 -- 172.21.15.12:0/2951144296 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766000e070 con 0x7f76680ae670 2024-03-29T11:32:45.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.124+0000 7f7666a00700 1 -- 172.21.15.12:0/2951144296 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7660003a20 con 0x7f76680ae670 2024-03-29T11:32:45.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.124+0000 7f7666a00700 1 -- 172.21.15.12:0/2951144296 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658001670 con 0x7f76680ae670 2024-03-29T11:32:45.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.124+0000 7f766ec00700 1 -- 172.21.15.12:0/2951144296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 msgr2=0x7f7668166470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.124+0000 7f766ec00700 1 --2- 172.21.15.12:0/2951144296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 0x7f7668166470 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7660009dc0 tx=0x7f766000b330 comp rx=0 tx=0).stop 2024-03-29T11:32:45.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.124+0000 7f766ec00700 1 -- 172.21.15.12:0/2951144296 shutdown_connections 2024-03-29T11:32:45.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.124+0000 7f766ec00700 1 --2- 172.21.15.12:0/2951144296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 0x7f7668166470 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:45.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.124+0000 7f766ec00700 1 -- 172.21.15.12:0/2951144296 >> 172.21.15.12:0/2951144296 conn(0x7f76680cf210 msgr2=0x7f76680cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:45.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.124+0000 7f766ec00700 1 -- 172.21.15.12:0/2951144296 shutdown_connections 2024-03-29T11:32:45.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.124+0000 7f766ec00700 1 -- 172.21.15.12:0/2951144296 wait complete. 2024-03-29T11:32:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.126+0000 7f766ec00700 1 Processor -- start 2024-03-29T11:32:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.126+0000 7f766ec00700 1 -- start start 2024-03-29T11:32:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.127+0000 7f766ec00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 0x7f7668143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.127+0000 7f766ec00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7668155ac0 0x7f7668146560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.127+0000 7f766ec00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f766815c700 0x7f76681431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.127+0000 7f766ec00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7668146aa0 con 0x7f7668155ac0 2024-03-29T11:32:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.127+0000 7f766ec00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76681440f0 con 0x7f766815c700 2024-03-29T11:32:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.127+0000 7f766ec00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76681443d0 con 0x7f76680ae670 2024-03-29T11:32:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.127+0000 7f7667400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7668155ac0 0x7f7668146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.127+0000 7f7667400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7668155ac0 0x7f7668146560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43430/0 (socket says 172.21.15.12:43430) 2024-03-29T11:32:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.127+0000 7f7667400700 1 -- 172.21.15.12:0/2912628940 learned_addr learned my addr 172.21.15.12:0/2912628940 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:45.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f7667e00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 0x7f7668143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f766cc00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f766815c700 0x7f76681431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f7667400700 1 -- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 msgr2=0x7f7668143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f7667400700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 0x7f7668143b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f7667400700 1 -- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f766815c700 msgr2=0x7f76681431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f7667400700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f766815c700 0x7f76681431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f7667400700 1 -- 172.21.15.12:0/2912628940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7660009a70 con 0x7f7668155ac0 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f766cc00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f766815c700 0x7f76681431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f7667e00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 0x7f7668143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f7667400700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7668155ac0 0x7f7668146560 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f765400c500 tx=0x7f765400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f765fe00700 1 -- 172.21.15.12:0/2912628940 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76540052d0 con 0x7f7668155ac0 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f765fe00700 1 -- 172.21.15.12:0/2912628940 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7654005450 con 0x7f7668155ac0 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f766ec00700 1 -- 172.21.15.12:0/2912628940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7668156020 con 0x7f7668155ac0 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.128+0000 7f765fe00700 1 -- 172.21.15.12:0/2912628940 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76540038c0 con 0x7f7668155ac0 2024-03-29T11:32:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.129+0000 7f766ec00700 1 -- 172.21.15.12:0/2912628940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7668144990 con 0x7f7668155ac0 2024-03-29T11:32:45.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.130+0000 7f766ec00700 1 -- 172.21.15.12:0/2912628940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7668072e50 con 0x7f7668155ac0 2024-03-29T11:32:45.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.131+0000 7f765fe00700 1 -- 172.21.15.12:0/2912628940 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f7654003a20 con 0x7f7668155ac0 2024-03-29T11:32:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.131+0000 7f765fe00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f764806aed0 0x7f764806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.132+0000 7f765fe00700 1 -- 172.21.15.12:0/2912628940 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f7654088c10 con 0x7f7668155ac0 2024-03-29T11:32:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.132+0000 7f7667e00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f764806aed0 0x7f764806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.133+0000 7f7667e00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f764806aed0 0x7f764806d390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7658001eb0 tx=0x7f7658009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.136+0000 7f765fe00700 1 -- 172.21.15.12:0/2912628940 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7654053a90 con 0x7f7668155ac0 2024-03-29T11:32:45.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:45.353+0000 7f766ec00700 1 -- 172.21.15.12:0/2912628940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7668067720 con 0x7f7668155ac0 2024-03-29T11:32:46.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.317+0000 7f765fe00700 1 -- 172.21.15.12:0/2912628940 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v249) v1 ==== 144+0+0 (secure 0 0 0) 0x7f76540570b0 con 0x7f7668155ac0 2024-03-29T11:32:46.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.322+0000 7f766ec00700 1 -- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f764806aed0 msgr2=0x7f764806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:46.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.322+0000 7f766ec00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f764806aed0 0x7f764806d390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7658001eb0 tx=0x7f7658009040 comp rx=0 tx=0).stop 2024-03-29T11:32:46.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.322+0000 7f766ec00700 1 -- 172.21.15.12:0/2912628940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7668155ac0 msgr2=0x7f7668146560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:46.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.322+0000 7f766ec00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7668155ac0 0x7f7668146560 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f765400c500 tx=0x7f765400c8c0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.322+0000 7f766ec00700 1 -- 172.21.15.12:0/2912628940 shutdown_connections 2024-03-29T11:32:46.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.322+0000 7f766ec00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f764806aed0 0x7f764806d390 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.322+0000 7f766ec00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f766815c700 0x7f76681431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.322+0000 7f766ec00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76680ae670 0x7f7668143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.322+0000 7f766ec00700 1 --2- 172.21.15.12:0/2912628940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7668155ac0 0x7f7668146560 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.322+0000 7f766ec00700 1 -- 172.21.15.12:0/2912628940 >> 172.21.15.12:0/2912628940 conn(0x7f76680cf210 msgr2=0x7f76680aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:46.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.323+0000 7f766ec00700 1 -- 172.21.15.12:0/2912628940 shutdown_connections 2024-03-29T11:32:46.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.323+0000 7f766ec00700 1 -- 172.21.15.12:0/2912628940 wait complete. 2024-03-29T11:32:46.322 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'backup_fs_metadata' created 2024-03-29T11:32:46.346 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T11:32:46.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.560+0000 7f6cf5400700 1 Processor -- start 2024-03-29T11:32:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.560+0000 7f6cf5400700 1 -- start start 2024-03-29T11:32:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.561+0000 7f6cf5400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6cf015d9d0 0x7f6cf00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.561+0000 7f6cf5400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6cf00d16a0 0x7f6cf00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.561+0000 7f6cf5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf00d2370 0x7f6cf015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.561+0000 7f6cf5400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf00d2820 con 0x7f6cf015d9d0 2024-03-29T11:32:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.561+0000 7f6cf5400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf0161420 con 0x7f6cf00d16a0 2024-03-29T11:32:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.561+0000 7f6cf5400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf0161700 con 0x7f6cf00d2370 2024-03-29T11:32:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.561+0000 7f6cee800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6cf015d9d0 0x7f6cf00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.562+0000 7f6cee800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6cf015d9d0 0x7f6cf00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43440/0 (socket says 172.21.15.12:43440) 2024-03-29T11:32:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.562+0000 7f6cee800700 1 -- 172.21.15.12:0/817443300 learned_addr learned my addr 172.21.15.12:0/817443300 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.562+0000 7f6cede00700 1 --2- 172.21.15.12:0/817443300 >> v2:172.21.15.159:3300/0 conn(0x7f6cf00d16a0 0x7f6cf00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.562+0000 7f6cef200700 1 --2- 172.21.15.12:0/817443300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf00d2370 0x7f6cf015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.562+0000 7f6cee800700 1 -- 172.21.15.12:0/817443300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf00d2370 msgr2=0x7f6cf015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.562+0000 7f6cee800700 1 --2- 172.21.15.12:0/817443300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf00d2370 0x7f6cf015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.562+0000 7f6cee800700 1 -- 172.21.15.12:0/817443300 >> v2:172.21.15.159:3300/0 conn(0x7f6cf00d16a0 msgr2=0x7f6cf00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.562+0000 7f6cee800700 1 --2- 172.21.15.12:0/817443300 >> v2:172.21.15.159:3300/0 conn(0x7f6cf00d16a0 0x7f6cf00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.562+0000 7f6cee800700 1 -- 172.21.15.12:0/817443300 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf01619e0 con 0x7f6cf015d9d0 2024-03-29T11:32:46.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.562+0000 7f6cee800700 1 --2- 172.21.15.12:0/817443300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf015d9d0 0x7f6cf00d3c30 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00d12f0 tx=0x7f6ce0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616fe05f6a2924a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:46.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6ced400700 1 -- 172.21.15.12:0/817443300 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce000b870 con 0x7f6cf015d9d0 2024-03-29T11:32:46.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6ced400700 1 -- 172.21.15.12:0/817443300 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce000b9d0 con 0x7f6cf015d9d0 2024-03-29T11:32:46.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6ced400700 1 -- 172.21.15.12:0/817443300 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce000bd60 con 0x7f6cf015d9d0 2024-03-29T11:32:46.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6cf5400700 1 -- 172.21.15.12:0/817443300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf015d9d0 msgr2=0x7f6cf00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:46.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6cf5400700 1 --2- 172.21.15.12:0/817443300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf015d9d0 0x7f6cf00d3c30 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00d12f0 tx=0x7f6ce0009bd0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6cf5400700 1 -- 172.21.15.12:0/817443300 shutdown_connections 2024-03-29T11:32:46.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6cf5400700 1 --2- 172.21.15.12:0/817443300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf00d2370 0x7f6cf015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6cf5400700 1 --2- 172.21.15.12:0/817443300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf015d9d0 0x7f6cf00d3c30 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6cf5400700 1 --2- 172.21.15.12:0/817443300 >> v2:172.21.15.159:3300/0 conn(0x7f6cf00d16a0 0x7f6cf00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6cf5400700 1 -- 172.21.15.12:0/817443300 >> 172.21.15.12:0/817443300 conn(0x7f6cf008aff0 msgr2=0x7f6cf00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:46.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6cf5400700 1 -- 172.21.15.12:0/817443300 shutdown_connections 2024-03-29T11:32:46.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.563+0000 7f6cf5400700 1 -- 172.21.15.12:0/817443300 wait complete. 2024-03-29T11:32:46.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.565+0000 7f6cf5400700 1 Processor -- start 2024-03-29T11:32:46.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.565+0000 7f6cf5400700 1 -- start start 2024-03-29T11:32:46.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cf5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf00d16a0 0x7f6cf00ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cf5400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cf00d2370 0x7f6cf014a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cf5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf015d9d0 0x7f6cf014ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cf5400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf00cac80 con 0x7f6cf00d16a0 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cf5400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf00cd390 con 0x7f6cf00d2370 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cf5400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf00cd670 con 0x7f6cf015d9d0 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cee800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf00d16a0 0x7f6cf00ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cee800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf00d16a0 0x7f6cf00ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43448/0 (socket says 172.21.15.12:43448) 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cee800700 1 -- 172.21.15.12:0/1061279329 learned_addr learned my addr 172.21.15.12:0/1061279329 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cef200700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf015d9d0 0x7f6cf014ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.566+0000 7f6cede00700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cf00d2370 0x7f6cf014a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.567+0000 7f6cee800700 1 -- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf015d9d0 msgr2=0x7f6cf014ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.567+0000 7f6cee800700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf015d9d0 0x7f6cf014ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.567+0000 7f6cee800700 1 -- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cf00d2370 msgr2=0x7f6cf014a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:46.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.567+0000 7f6cee800700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cf00d2370 0x7f6cf014a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:46.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.567+0000 7f6cee800700 1 -- 172.21.15.12:0/1061279329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0009550 con 0x7f6cf00d16a0 2024-03-29T11:32:46.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.567+0000 7f6cee800700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf00d16a0 0x7f6cf00ca5f0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0000c00 tx=0x7f6ce0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:46.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.567+0000 7f6ce6a00700 1 -- 172.21.15.12:0/1061279329 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0004310 con 0x7f6cf00d16a0 2024-03-29T11:32:46.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.567+0000 7f6ce6a00700 1 -- 172.21.15.12:0/1061279329 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce001e070 con 0x7f6cf00d16a0 2024-03-29T11:32:46.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.567+0000 7f6ce6a00700 1 -- 172.21.15.12:0/1061279329 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00036a0 con 0x7f6cf00d16a0 2024-03-29T11:32:46.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.567+0000 7f6cf5400700 1 -- 172.21.15.12:0/1061279329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf00cd950 con 0x7f6cf00d16a0 2024-03-29T11:32:46.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.568+0000 7f6cf5400700 1 -- 172.21.15.12:0/1061279329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf014a7f0 con 0x7f6cf00d16a0 2024-03-29T11:32:46.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.569+0000 7f6cf5400700 1 -- 172.21.15.12:0/1061279329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf00d1ca0 con 0x7f6cf00d16a0 2024-03-29T11:32:46.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.570+0000 7f6ce6a00700 1 -- 172.21.15.12:0/1061279329 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6ce0020030 con 0x7f6cf00d16a0 2024-03-29T11:32:46.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.571+0000 7f6ce6a00700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6cd006aed0 0x7f6cd006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:46.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.571+0000 7f6ce6a00700 1 -- 172.21.15.12:0/1061279329 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6154+0+0 (secure 0 0 0) 0x7f6ce008a670 con 0x7f6cf00d16a0 2024-03-29T11:32:46.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.571+0000 7f6cede00700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6cd006aed0 0x7f6cd006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:46.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.572+0000 7f6cede00700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6cd006aed0 0x7f6cd006d390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6cf006d770 tx=0x7f6cdc008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:46.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.576+0000 7f6ce6a00700 1 -- 172.21.15.12:0/1061279329 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ce0055490 con 0x7f6cf00d16a0 2024-03-29T11:32:46.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:46.788+0000 7f6cf5400700 1 -- 172.21.15.12:0/1061279329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6cf00d3310 con 0x7f6cf00d16a0 2024-03-29T11:32:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.328+0000 7f6ce6a00700 1 -- 172.21.15.12:0/1061279329 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v250) v1 ==== 176+0+0 (secure 0 0 0) 0x7f6ce0058ab0 con 0x7f6cf00d16a0 2024-03-29T11:32:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.333+0000 7f6cf5400700 1 -- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6cd006aed0 msgr2=0x7f6cd006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.333+0000 7f6cf5400700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6cd006aed0 0x7f6cd006d390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6cf006d770 tx=0x7f6cdc008040 comp rx=0 tx=0).stop 2024-03-29T11:32:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.333+0000 7f6cf5400700 1 -- 172.21.15.12:0/1061279329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf00d16a0 msgr2=0x7f6cf00ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.333+0000 7f6cf5400700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf00d16a0 0x7f6cf00ca5f0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0000c00 tx=0x7f6ce0004750 comp rx=0 tx=0).stop 2024-03-29T11:32:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.335+0000 7f6cf5400700 1 -- 172.21.15.12:0/1061279329 shutdown_connections 2024-03-29T11:32:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.335+0000 7f6cf5400700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6cd006aed0 0x7f6cd006d390 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.335+0000 7f6cf5400700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf015d9d0 0x7f6cf014ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.335+0000 7f6cf5400700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cf00d16a0 0x7f6cf00ca5f0 secure :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0000c00 tx=0x7f6ce0004750 comp rx=0 tx=0).stop 2024-03-29T11:32:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.335+0000 7f6cf5400700 1 --2- 172.21.15.12:0/1061279329 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cf00d2370 0x7f6cf014a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.335+0000 7f6cf5400700 1 -- 172.21.15.12:0/1061279329 >> 172.21.15.12:0/1061279329 conn(0x7f6cf008aff0 msgr2=0x7f6cf00ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.338+0000 7f6cf5400700 1 -- 172.21.15.12:0/1061279329 shutdown_connections 2024-03-29T11:32:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.338+0000 7f6cf5400700 1 -- 172.21.15.12:0/1061279329 wait complete. 2024-03-29T11:32:47.337 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'backup_fs_data' created 2024-03-29T11:32:47.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-03-29T11:32:47.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.569+0000 7f4fb4200700 1 Processor -- start 2024-03-29T11:32:47.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.569+0000 7f4fb4200700 1 -- start start 2024-03-29T11:32:47.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.570+0000 7f4fb4200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4fac0ced80 0x7f4fac0d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:47.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.570+0000 7f4fb4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fac15fcb0 0x7f4fac160120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:47.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.570+0000 7f4fb4200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fac15f820 con 0x7f4fac15dad0 2024-03-29T11:32:47.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.570+0000 7f4fb4200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fac0cfcc0 con 0x7f4fac0ced80 2024-03-29T11:32:47.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.570+0000 7f4fb4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac0cd9f0 con 0x7f4fac15fcb0 2024-03-29T11:32:47.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.570+0000 7f4fb0e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4fac0ced80 0x7f4fac0d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:47.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.570+0000 7f4fb1800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4fac15dad0 0x7f4fac0d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53794/0 (socket says 172.21.15.12:53794) 2024-03-29T11:32:47.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.570+0000 7f4fb1800700 1 -- 172.21.15.12:0/1773148861 learned_addr learned my addr 172.21.15.12:0/1773148861 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:47.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fb2200700 1 --2- 172.21.15.12:0/1773148861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fac15fcb0 0x7f4fac160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:47.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fabe00700 1 -- 172.21.15.12:0/1773148861 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859506171 0 0) 0x7f4fac15f820 con 0x7f4fac15dad0 2024-03-29T11:32:47.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fabe00700 1 -- 172.21.15.12:0/1773148861 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f940035b0 con 0x7f4fac15dad0 2024-03-29T11:32:47.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fb0e00700 1 -- 172.21.15.12:0/1773148861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fac15fcb0 msgr2=0x7f4fac160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:47.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fb0e00700 1 --2- 172.21.15.12:0/1773148861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fac15fcb0 0x7f4fac160120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fb0e00700 1 -- 172.21.15.12:0/1773148861 >> v1:172.21.15.12:6789/0 conn(0x7f4fac15dad0 legacy=0x7f4fac0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:47.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fb0e00700 1 -- 172.21.15.12:0/1773148861 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac0cf420 con 0x7f4fac0ced80 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fb0e00700 1 --2- 172.21.15.12:0/1773148861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac0ced80 0x7f4fac0d1d40 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009840 tx=0x7f4f98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cdd034930a5ced server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fabe00700 1 -- 172.21.15.12:0/1773148861 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98011830 con 0x7f4fac0ced80 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fabe00700 1 -- 172.21.15.12:0/1773148861 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f98011990 con 0x7f4fac0ced80 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.571+0000 7f4fabe00700 1 -- 172.21.15.12:0/1773148861 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98011d20 con 0x7f4fac0ced80 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.572+0000 7f4fb4200700 1 -- 172.21.15.12:0/1773148861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac0ced80 msgr2=0x7f4fac0d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.572+0000 7f4fb4200700 1 --2- 172.21.15.12:0/1773148861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac0ced80 0x7f4fac0d1d40 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009840 tx=0x7f4f98009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.572+0000 7f4fb4200700 1 -- 172.21.15.12:0/1773148861 shutdown_connections 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.572+0000 7f4fb4200700 1 --2- 172.21.15.12:0/1773148861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fac15fcb0 0x7f4fac160120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.572+0000 7f4fb4200700 1 --2- 172.21.15.12:0/1773148861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac0ced80 0x7f4fac0d1d40 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.572+0000 7f4fb4200700 1 -- 172.21.15.12:0/1773148861 >> 172.21.15.12:0/1773148861 conn(0x7f4fac08aff0 msgr2=0x7f4fac0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.572+0000 7f4fb4200700 1 -- 172.21.15.12:0/1773148861 shutdown_connections 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.572+0000 7f4fb4200700 1 -- 172.21.15.12:0/1773148861 wait complete. 2024-03-29T11:32:47.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.574+0000 7f4fb4200700 1 Processor -- start 2024-03-29T11:32:47.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.574+0000 7f4fb4200700 1 -- start start 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.575+0000 7f4fb4200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fac0ced80 0x7f4fac1f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.575+0000 7f4fb4200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac15dad0 0x7f4fac1f0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.575+0000 7f4fb4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fac15fcb0 0x7f4fac0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.575+0000 7f4fb4200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac0b8db0 con 0x7f4fac15dad0 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.575+0000 7f4fb4200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac1f0950 con 0x7f4fac0ced80 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.575+0000 7f4fb4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac1f2fc0 con 0x7f4fac15fcb0 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.575+0000 7f4fb0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac15dad0 0x7f4fac1f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.575+0000 7f4fb0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac15dad0 0x7f4fac1f0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43478/0 (socket says 172.21.15.12:43478) 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.575+0000 7f4fb0e00700 1 -- 172.21.15.12:0/271493334 learned_addr learned my addr 172.21.15.12:0/271493334 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.575+0000 7f4fb1800700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fac0ced80 0x7f4fac1f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fb2200700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fac15fcb0 0x7f4fac0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fb0e00700 1 -- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fac15fcb0 msgr2=0x7f4fac0c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fb0e00700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fac15fcb0 0x7f4fac0c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fb0e00700 1 -- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fac0ced80 msgr2=0x7f4fac1f64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fb0e00700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fac0ced80 0x7f4fac1f64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fb0e00700 1 -- 172.21.15.12:0/271493334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f980094a0 con 0x7f4fac15dad0 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fb1800700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fac0ced80 0x7f4fac1f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fb0e00700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac15dad0 0x7f4fac1f0410 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f4f98000c00 tx=0x7f4f98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fa9600700 1 -- 172.21.15.12:0/271493334 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f980036a0 con 0x7f4fac15dad0 2024-03-29T11:32:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fa9600700 1 -- 172.21.15.12:0/271493334 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9801f070 con 0x7f4fac15dad0 2024-03-29T11:32:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fa9600700 1 -- 172.21.15.12:0/271493334 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98003a60 con 0x7f4fac15dad0 2024-03-29T11:32:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fb4200700 1 -- 172.21.15.12:0/271493334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fac1f6a80 con 0x7f4fac15dad0 2024-03-29T11:32:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.576+0000 7f4fb4200700 1 -- 172.21.15.12:0/271493334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fac1f7040 con 0x7f4fac15dad0 2024-03-29T11:32:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.577+0000 7f4fb4200700 1 -- 172.21.15.12:0/271493334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fac1f3190 con 0x7f4fac15dad0 2024-03-29T11:32:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.579+0000 7f4fa9600700 1 -- 172.21.15.12:0/271493334 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4f98003bc0 con 0x7f4fac15dad0 2024-03-29T11:32:47.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.580+0000 7f4fa9600700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f8c06aed0 0x7f4f8c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:47.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.580+0000 7f4fa9600700 1 -- 172.21.15.12:0/271493334 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f4f98021070 con 0x7f4fac15dad0 2024-03-29T11:32:47.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.580+0000 7f4fb1800700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f8c06aed0 0x7f4f8c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:47.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.581+0000 7f4fb1800700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f8c06aed0 0x7f4f8c06d390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00011e0 tx=0x7f4fa0009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.584+0000 7f4fa9600700 1 -- 172.21.15.12:0/271493334 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4f98055f20 con 0x7f4fac15dad0 2024-03-29T11:32:47.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.847+0000 7f4fb4200700 1 -- 172.21.15.12:0/271493334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f4fac1f3190 con 0x7f4fac15dad0 2024-03-29T11:32:47.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.853+0000 7f4fa9600700 1 -- 172.21.15.12:0/271493334 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 32 and data pool 33 v124) v1 ==== 183+0+0 (secure 0 0 0) 0x7f4fac1f3190 con 0x7f4fac15dad0 2024-03-29T11:32:47.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.859+0000 7f4fb4200700 1 -- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f8c06aed0 msgr2=0x7f4f8c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:47.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.859+0000 7f4fb4200700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f8c06aed0 0x7f4f8c06d390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00011e0 tx=0x7f4fa0009040 comp rx=0 tx=0).stop 2024-03-29T11:32:47.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.859+0000 7f4fb4200700 1 -- 172.21.15.12:0/271493334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac15dad0 msgr2=0x7f4fac1f0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:47.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.859+0000 7f4fb4200700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac15dad0 0x7f4fac1f0410 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f4f98000c00 tx=0x7f4f98004750 comp rx=0 tx=0).stop 2024-03-29T11:32:47.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.862+0000 7f4fb4200700 1 -- 172.21.15.12:0/271493334 shutdown_connections 2024-03-29T11:32:47.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.862+0000 7f4fb4200700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f8c06aed0 0x7f4f8c06d390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.862+0000 7f4fb4200700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fac15fcb0 0x7f4fac0c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.862+0000 7f4fb4200700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fac0ced80 0x7f4fac1f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.862+0000 7f4fb4200700 1 --2- 172.21.15.12:0/271493334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fac15dad0 0x7f4fac1f0410 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:47.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.862+0000 7f4fb4200700 1 -- 172.21.15.12:0/271493334 >> 172.21.15.12:0/271493334 conn(0x7f4fac08aff0 msgr2=0x7f4fac0d04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.862+0000 7f4fb4200700 1 -- 172.21.15.12:0/271493334 shutdown_connections 2024-03-29T11:32:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:47.863+0000 7f4fb4200700 1 -- 172.21.15.12:0/271493334 wait complete. 2024-03-29T11:32:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 32 and data pool 33 2024-03-29T11:32:47.891 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:48.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.576+0000 7fbb8f000700 1 Processor -- start 2024-03-29T11:32:48.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.576+0000 7fbb8f000700 1 -- start start 2024-03-29T11:32:48.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.577+0000 7fbb8f000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbb880ceec0 0x7fbb881612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:48.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.577+0000 7fbb8f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb88161800 0x7fbb880b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:48.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.577+0000 7fbb8f000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb880d30d0 con 0x7fbb8815d640 2024-03-29T11:32:48.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.577+0000 7fbb8f000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb88161c90 con 0x7fbb880ceec0 2024-03-29T11:32:48.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.577+0000 7fbb8f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880b6840 con 0x7fbb88161800 2024-03-29T11:32:48.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.578+0000 7fbb87400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbb880ceec0 0x7fbb881612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:48.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.578+0000 7fbb87400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbb880ceec0 0x7fbb881612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43492/0 (socket says 172.21.15.12:43492) 2024-03-29T11:32:48.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.578+0000 7fbb87400700 1 -- 172.21.15.12:0/2728894915 learned_addr learned my addr 172.21.15.12:0/2728894915 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:48.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.578+0000 7fbb8d000700 1 --2- 172.21.15.12:0/2728894915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb88161800 0x7fbb880b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:48.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.578+0000 7fbb87400700 1 -- 172.21.15.12:0/2728894915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb88161800 msgr2=0x7fbb880b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:48.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.578+0000 7fbb87400700 1 --2- 172.21.15.12:0/2728894915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb88161800 0x7fbb880b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:48.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.578+0000 7fbb87400700 1 -- 172.21.15.12:0/2728894915 >> v1:172.21.15.159:6789/0 conn(0x7fbb8815d640 legacy=0x7fbb880d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.578+0000 7fbb87400700 1 -- 172.21.15.12:0/2728894915 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb880d2b30 con 0x7fbb880ceec0 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb87400700 1 --2- 172.21.15.12:0/2728894915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb880ceec0 0x7fbb881612c0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fbb740098f0 tx=0x7fbb74009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab21ccedc3ba477 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb86a00700 1 -- 172.21.15.12:0/2728894915 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb74011830 con 0x7fbb880ceec0 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb86a00700 1 -- 172.21.15.12:0/2728894915 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb74011990 con 0x7fbb880ceec0 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb86a00700 1 -- 172.21.15.12:0/2728894915 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb74011d20 con 0x7fbb880ceec0 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb8f000700 1 -- 172.21.15.12:0/2728894915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb880ceec0 msgr2=0x7fbb881612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb8f000700 1 --2- 172.21.15.12:0/2728894915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb880ceec0 0x7fbb881612c0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fbb740098f0 tx=0x7fbb74009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb8f000700 1 -- 172.21.15.12:0/2728894915 shutdown_connections 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb8f000700 1 --2- 172.21.15.12:0/2728894915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb88161800 0x7fbb880b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb8f000700 1 --2- 172.21.15.12:0/2728894915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb880ceec0 0x7fbb881612c0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb8f000700 1 -- 172.21.15.12:0/2728894915 >> 172.21.15.12:0/2728894915 conn(0x7fbb8808aff0 msgr2=0x7fbb880d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.579+0000 7fbb8f000700 1 -- 172.21.15.12:0/2728894915 shutdown_connections 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.580+0000 7fbb8f000700 1 -- 172.21.15.12:0/2728894915 wait complete. 2024-03-29T11:32:48.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.581+0000 7fbb8f000700 1 Processor -- start 2024-03-29T11:32:48.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.581+0000 7fbb8f000700 1 -- start start 2024-03-29T11:32:48.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.582+0000 7fbb8f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb880ceec0 0x7fbb8815b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:48.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.582+0000 7fbb8f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb8815d640 0x7fbb880ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:48.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.582+0000 7fbb8f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88161800 0x7fbb880ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:48.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.582+0000 7fbb8f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880cae10 con 0x7fbb88161800 2024-03-29T11:32:48.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.582+0000 7fbb8f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880cdb10 con 0x7fbb880ceec0 2024-03-29T11:32:48.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.582+0000 7fbb8f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880cddf0 con 0x7fbb8815d640 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.582+0000 7fbb8d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88161800 0x7fbb880ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.582+0000 7fbb8d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88161800 0x7fbb880ccb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43504/0 (socket says 172.21.15.12:43504) 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.582+0000 7fbb8d000700 1 -- 172.21.15.12:0/324018744 learned_addr learned my addr 172.21.15.12:0/324018744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb87400700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb8815d640 0x7fbb880ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb87e00700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb880ceec0 0x7fbb8815b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb8d000700 1 -- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb8815d640 msgr2=0x7fbb880ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb8d000700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb8815d640 0x7fbb880ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb8d000700 1 -- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb880ceec0 msgr2=0x7fbb8815b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb8d000700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb880ceec0 0x7fbb8815b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb8d000700 1 -- 172.21.15.12:0/324018744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb74009550 con 0x7fbb88161800 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb87e00700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb880ceec0 0x7fbb8815b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb8d000700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88161800 0x7fbb880ccb20 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000efc0 tx=0x7fbb8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb7fe00700 1 -- 172.21.15.12:0/324018744 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb80009ea0 con 0x7fbb88161800 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb7fe00700 1 -- 172.21.15.12:0/324018744 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8000f040 con 0x7fbb88161800 2024-03-29T11:32:48.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb8f000700 1 -- 172.21.15.12:0/324018744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb880cb0d0 con 0x7fbb88161800 2024-03-29T11:32:48.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.583+0000 7fbb7fe00700 1 -- 172.21.15.12:0/324018744 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb80014480 con 0x7fbb88161800 2024-03-29T11:32:48.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.584+0000 7fbb8f000700 1 -- 172.21.15.12:0/324018744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8815bb60 con 0x7fbb88161800 2024-03-29T11:32:48.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.585+0000 7fbb8f000700 1 -- 172.21.15.12:0/324018744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb880cb5f0 con 0x7fbb88161800 2024-03-29T11:32:48.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.586+0000 7fbb7fe00700 1 -- 172.21.15.12:0/324018744 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fbb800145e0 con 0x7fbb88161800 2024-03-29T11:32:48.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.586+0000 7fbb7fe00700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbb6806aed0 0x7fbb6806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:48.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.587+0000 7fbb7fe00700 1 -- 172.21.15.12:0/324018744 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 6645+0+0 (secure 0 0 0) 0x7fbb8008a520 con 0x7fbb88161800 2024-03-29T11:32:48.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.587+0000 7fbb87e00700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbb6806aed0 0x7fbb6806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:48.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.588+0000 7fbb87e00700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbb6806aed0 0x7fbb6806d390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbb78000f60 tx=0x7fbb78009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:48.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.591+0000 7fbb7fe00700 1 -- 172.21.15.12:0/324018744 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbb880cb5f0 con 0x7fbb88161800 2024-03-29T11:32:48.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.806+0000 7fbb8f000700 1 -- 172.21.15.12:0/324018744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb880cb5f0 con 0x7fbb88161800 2024-03-29T11:32:48.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.808+0000 7fbb7fe00700 1 -- 172.21.15.12:0/324018744 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) v1 ==== 74+0+19848 (secure 0 0 0) 0x7fbb880cb5f0 con 0x7fbb88161800 2024-03-29T11:32:48.804 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:48.804 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":251,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:47.848714+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:32:45.400233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:32:46.836046+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:48.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 -- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbb6806aed0 msgr2=0x7fbb6806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbb6806aed0 0x7fbb6806d390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbb78000f60 tx=0x7fbb78009040 comp rx=0 tx=0).stop 2024-03-29T11:32:48.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 -- 172.21.15.12:0/324018744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88161800 msgr2=0x7fbb880ccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88161800 0x7fbb880ccb20 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000efc0 tx=0x7fbb8000c3a0 comp rx=0 tx=0).stop 2024-03-29T11:32:48.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 -- 172.21.15.12:0/324018744 shutdown_connections 2024-03-29T11:32:48.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbb6806aed0 0x7fbb6806d390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:48.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88161800 0x7fbb880ccb20 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:48.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb880ceec0 0x7fbb8815b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:48.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 --2- 172.21.15.12:0/324018744 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb8815d640 0x7fbb880ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:48.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 -- 172.21.15.12:0/324018744 >> 172.21.15.12:0/324018744 conn(0x7fbb8808aff0 msgr2=0x7fbb880b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:48.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 -- 172.21.15.12:0/324018744 shutdown_connections 2024-03-29T11:32:48.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:48.810+0000 7fbb8f000700 1 -- 172.21.15.12:0/324018744 wait complete. 2024-03-29T11:32:48.825 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.030+0000 7f3098600700 1 Processor -- start 2024-03-29T11:32:49.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.030+0000 7f3098600700 1 -- start start 2024-03-29T11:32:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.032+0000 7f3098600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f309015d640 0x7f30900d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.032+0000 7f3098600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30900d30d0 con 0x7f30900ceec0 2024-03-29T11:32:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.032+0000 7f3098600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30900b6840 con 0x7f30901619d0 2024-03-29T11:32:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.032+0000 7f3098600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30900d2880 con 0x7f309015d640 2024-03-29T11:32:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.032+0000 7f3095200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f30900ceec0 0x7f30901612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59552/0 (socket says 172.21.15.12:59552) 2024-03-29T11:32:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.032+0000 7f3095200700 1 -- 172.21.15.12:0/2411621064 learned_addr learned my addr 172.21.15.12:0/2411621064 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.032+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630755702 0 0) 0x7f30900d30d0 con 0x7f30900ceec0 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.032+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30780035b0 con 0x7f30900ceec0 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.032+0000 7f3095c00700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f309015d640 0x7f30900d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3735028747 0 0) 0x7f30780035b0 con 0x7f30900ceec0 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f309015d640 msgr2=0x7f30900d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f309015d640 0x7f30900d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 >> v1:172.21.15.159:6789/0 conn(0x7f30901619d0 legacy=0x7f30900b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30900d2b30 con 0x7f30900ceec0 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f307c001670 con 0x7f30900ceec0 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 >> v1:172.21.15.12:6789/0 conn(0x7f30900ceec0 legacy=0x7f30901612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3078003a30 0x7f3078003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f309015d640 0x7f3078004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 0x7f3078005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30900d2880 con 0x7f3078004d90 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307800bbf0 con 0x7f3078003a30 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307800bdb0 con 0x7f309015d640 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f3095200700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f309015d640 0x7f3078004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f3095c00700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3078003a30 0x7f3078003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.033+0000 7f3096600700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 0x7f3078005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3096600700 1 -- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f309015d640 msgr2=0x7f3078004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3096600700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f309015d640 0x7f3078004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3096600700 1 -- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3078003a30 msgr2=0x7f3078003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3096600700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3078003a30 0x7f3078003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3096600700 1 -- 172.21.15.12:0/2411621064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30780035b0 con 0x7f3078004d90 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3095c00700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3078003a30 0x7f3078003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3095200700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f309015d640 0x7f3078004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3096600700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 0x7f3078005210 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f30900cffa0 tx=0x7f3078005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5854e7fafb7fefb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088001db0 con 0x7f3078004d90 2024-03-29T11:32:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308800cc70 con 0x7f3078004d90 2024-03-29T11:32:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f308fe00700 1 -- 172.21.15.12:0/2411621064 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088003a00 con 0x7f3078004d90 2024-03-29T11:32:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3098600700 1 -- 172.21.15.12:0/2411621064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 msgr2=0x7f3078005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3098600700 1 --2- 172.21.15.12:0/2411621064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 0x7f3078005210 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f30900cffa0 tx=0x7f3078005880 comp rx=0 tx=0).stop 2024-03-29T11:32:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.034+0000 7f3096600700 1 -- 172.21.15.12:0/2411621064 reap_dead start 2024-03-29T11:32:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.035+0000 7f3098600700 1 -- 172.21.15.12:0/2411621064 shutdown_connections 2024-03-29T11:32:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.035+0000 7f3098600700 1 -- 172.21.15.12:0/2411621064 >> 172.21.15.12:0/2411621064 conn(0x7f309008aff0 msgr2=0x7f30900d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.035+0000 7f3098600700 1 -- 172.21.15.12:0/2411621064 shutdown_connections 2024-03-29T11:32:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.035+0000 7f3098600700 1 -- 172.21.15.12:0/2411621064 wait complete. 2024-03-29T11:32:49.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.036+0000 7f3098600700 1 Processor -- start 2024-03-29T11:32:49.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.037+0000 7f3098600700 1 -- start start 2024-03-29T11:32:49.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.037+0000 7f3098600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 0x7f3090159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.037+0000 7f3098600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30900cdfa0 0x7f30900c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.037+0000 7f3098600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30900c7fd0 0x7f30900c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.037+0000 7f3098600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30900caa30 con 0x7f3078004d90 2024-03-29T11:32:49.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.037+0000 7f3098600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309015a100 con 0x7f30900c7fd0 2024-03-29T11:32:49.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.037+0000 7f3098600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30900c8450 con 0x7f30900cdfa0 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3095c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 0x7f3090159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3095c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 0x7f3090159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46254/0 (socket says 172.21.15.12:46254) 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3095c00700 1 -- 172.21.15.12:0/3312315114 learned_addr learned my addr 172.21.15.12:0/3312315114 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3095200700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30900cdfa0 0x7f30900c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3096600700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30900c7fd0 0x7f30900c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3095c00700 1 -- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30900cdfa0 msgr2=0x7f30900c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3095c00700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30900cdfa0 0x7f30900c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3095c00700 1 -- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30900c7fd0 msgr2=0x7f30900c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3095c00700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30900c7fd0 0x7f30900c9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3095c00700 1 -- 172.21.15.12:0/3312315114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30900c8730 con 0x7f3078004d90 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3096600700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30900c7fd0 0x7f30900c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.038+0000 7f3095c00700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 0x7f3090159bc0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f3084011680 tx=0x7f3084011a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.039+0000 7f308d600700 1 -- 172.21.15.12:0/3312315114 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308400c070 con 0x7f3078004d90 2024-03-29T11:32:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.039+0000 7f308d600700 1 -- 172.21.15.12:0/3312315114 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3084015870 con 0x7f3078004d90 2024-03-29T11:32:49.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.039+0000 7f308d600700 1 -- 172.21.15.12:0/3312315114 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3084015c20 con 0x7f3078004d90 2024-03-29T11:32:49.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.039+0000 7f3098600700 1 -- 172.21.15.12:0/3312315114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30900ca6c0 con 0x7f3078004d90 2024-03-29T11:32:49.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.039+0000 7f3098600700 1 -- 172.21.15.12:0/3312315114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30900c97d0 con 0x7f3078004d90 2024-03-29T11:32:49.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.040+0000 7f3098600700 1 -- 172.21.15.12:0/3312315114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3090067720 con 0x7f3078004d90 2024-03-29T11:32:49.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.041+0000 7f308d600700 1 -- 172.21.15.12:0/3312315114 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f3084013400 con 0x7f3078004d90 2024-03-29T11:32:49.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.042+0000 7f308d600700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f307006aed0 0x7f307006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.042+0000 7f308d600700 1 -- 172.21.15.12:0/3312315114 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f3084089ff0 con 0x7f3078004d90 2024-03-29T11:32:49.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.042+0000 7f3095200700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f307006aed0 0x7f307006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.043+0000 7f3095200700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f307006aed0 0x7f307006d390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f307c007fd0 tx=0x7f307c009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.046+0000 7f308d600700 1 -- 172.21.15.12:0/3312315114 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f30840549c0 con 0x7f3078004d90 2024-03-29T11:32:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.263+0000 7f3098600700 1 -- 172.21.15.12:0/3312315114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30900b5c70 con 0x7f3078004d90 2024-03-29T11:32:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.264+0000 7f308d600700 1 -- 172.21.15.12:0/3312315114 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+19835 (secure 0 0 0) 0x7f3084057fe0 con 0x7f3078004d90 2024-03-29T11:32:49.260 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:49.260 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":252,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:48.851660+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:32:45.400233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:32:46.836046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:49.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 -- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f307006aed0 msgr2=0x7f307006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f307006aed0 0x7f307006d390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f307c007fd0 tx=0x7f307c009040 comp rx=0 tx=0).stop 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 -- 172.21.15.12:0/3312315114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 msgr2=0x7f3090159bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 0x7f3090159bc0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f3084011680 tx=0x7f3084011a40 comp rx=0 tx=0).stop 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 -- 172.21.15.12:0/3312315114 shutdown_connections 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f307006aed0 0x7f307006d390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30900c7fd0 0x7f30900c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3078004d90 0x7f3090159bc0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 --2- 172.21.15.12:0/3312315114 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30900cdfa0 0x7f30900c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 -- 172.21.15.12:0/3312315114 >> 172.21.15.12:0/3312315114 conn(0x7f309008aff0 msgr2=0x7f309006cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 -- 172.21.15.12:0/3312315114 shutdown_connections 2024-03-29T11:32:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.267+0000 7f3098600700 1 -- 172.21.15.12:0/3312315114 wait complete. 2024-03-29T11:32:49.281 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-03-29T11:32:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.490+0000 7f7a4f000700 1 Processor -- start 2024-03-29T11:32:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.490+0000 7f7a4f000700 1 -- start start 2024-03-29T11:32:49.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.491+0000 7f7a4f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a480d16a0 0x7f7a480d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.491+0000 7f7a4f000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7a480d2370 0x7f7a480ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.491+0000 7f7a4f000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a48161420 con 0x7f7a4815d9d0 2024-03-29T11:32:49.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.491+0000 7f7a4f000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a480d2820 con 0x7f7a480d2370 2024-03-29T11:32:49.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.491+0000 7f7a4f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a480d4340 con 0x7f7a480d16a0 2024-03-29T11:32:49.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.491+0000 7f7a4d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7a480d2370 0x7f7a480ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.491+0000 7f7a4d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7a480d2370 0x7f7a480ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46256/0 (socket says 172.21.15.12:46256) 2024-03-29T11:32:49.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.491+0000 7f7a4d000700 1 -- 172.21.15.12:0/2175641943 learned_addr learned my addr 172.21.15.12:0/2175641943 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:49.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.492+0000 7f7a47400700 1 --2- 172.21.15.12:0/2175641943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a480d16a0 0x7f7a480d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.492+0000 7f7a4d000700 1 -- 172.21.15.12:0/2175641943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a480d16a0 msgr2=0x7f7a480d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.492+0000 7f7a4d000700 1 --2- 172.21.15.12:0/2175641943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a480d16a0 0x7f7a480d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.492+0000 7f7a4d000700 1 -- 172.21.15.12:0/2175641943 >> v1:172.21.15.159:6789/0 conn(0x7f7a4815d9d0 legacy=0x7f7a480d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.492+0000 7f7a4d000700 1 -- 172.21.15.12:0/2175641943 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48161700 con 0x7f7a480d2370 2024-03-29T11:32:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.492+0000 7f7a4d000700 1 --2- 172.21.15.12:0/2175641943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 0x7f7a480ce360 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009e80 tx=0x7f7a4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b2dd688503d0e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.492+0000 7f7a46a00700 1 -- 172.21.15.12:0/2175641943 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4000e070 con 0x7f7a480d2370 2024-03-29T11:32:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.492+0000 7f7a46a00700 1 -- 172.21.15.12:0/2175641943 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4000b980 con 0x7f7a480d2370 2024-03-29T11:32:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.492+0000 7f7a46a00700 1 -- 172.21.15.12:0/2175641943 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4000bd30 con 0x7f7a480d2370 2024-03-29T11:32:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.493+0000 7f7a4f000700 1 -- 172.21.15.12:0/2175641943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 msgr2=0x7f7a480ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.493+0000 7f7a4f000700 1 --2- 172.21.15.12:0/2175641943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 0x7f7a480ce360 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009e80 tx=0x7f7a4000b330 comp rx=0 tx=0).stop 2024-03-29T11:32:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.493+0000 7f7a4f000700 1 -- 172.21.15.12:0/2175641943 shutdown_connections 2024-03-29T11:32:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.493+0000 7f7a4f000700 1 --2- 172.21.15.12:0/2175641943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 0x7f7a480ce360 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.493+0000 7f7a4f000700 1 --2- 172.21.15.12:0/2175641943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a480d16a0 0x7f7a480d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.493+0000 7f7a4f000700 1 -- 172.21.15.12:0/2175641943 >> 172.21.15.12:0/2175641943 conn(0x7f7a4808aff0 msgr2=0x7f7a480d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.493+0000 7f7a4f000700 1 -- 172.21.15.12:0/2175641943 shutdown_connections 2024-03-29T11:32:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.493+0000 7f7a4f000700 1 -- 172.21.15.12:0/2175641943 wait complete. 2024-03-29T11:32:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.495+0000 7f7a4f000700 1 Processor -- start 2024-03-29T11:32:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.495+0000 7f7a4f000700 1 -- start start 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a4f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a480d16a0 0x7f7a480cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a4f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 0x7f7a48159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a4f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a4815d9d0 0x7f7a480cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a4f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a480b6020 con 0x7f7a480d2370 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a4f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4815a440 con 0x7f7a4815d9d0 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a4f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a480cb2e0 con 0x7f7a480d16a0 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a47400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 0x7f7a48159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a47400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 0x7f7a48159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46270/0 (socket says 172.21.15.12:46270) 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a47400700 1 -- 172.21.15.12:0/4268044901 learned_addr learned my addr 172.21.15.12:0/4268044901 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a47e00700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a480d16a0 0x7f7a480cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.496+0000 7f7a4d000700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a4815d9d0 0x7f7a480cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a47400700 1 -- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a480d16a0 msgr2=0x7f7a480cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a47400700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a480d16a0 0x7f7a480cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a47400700 1 -- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a4815d9d0 msgr2=0x7f7a480cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a47400700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a4815d9d0 0x7f7a480cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a47400700 1 -- 172.21.15.12:0/4268044901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a40009b30 con 0x7f7a480d2370 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a47400700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 0x7f7a48159f00 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400e810 tx=0x7f7a3400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a3fe00700 1 -- 172.21.15.12:0/4268044901 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3400c070 con 0x7f7a480d2370 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a3fe00700 1 -- 172.21.15.12:0/4268044901 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a34013610 con 0x7f7a480d2370 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a4f000700 1 -- 172.21.15.12:0/4268044901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a480cb5c0 con 0x7f7a480d2370 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a3fe00700 1 -- 172.21.15.12:0/4268044901 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a340139c0 con 0x7f7a480d2370 2024-03-29T11:32:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.497+0000 7f7a4f000700 1 -- 172.21.15.12:0/4268044901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4815f490 con 0x7f7a480d2370 2024-03-29T11:32:49.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.498+0000 7f7a4f000700 1 -- 172.21.15.12:0/4268044901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a480d1ca0 con 0x7f7a480d2370 2024-03-29T11:32:49.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.499+0000 7f7a3fe00700 1 -- 172.21.15.12:0/4268044901 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f7a34013b20 con 0x7f7a480d2370 2024-03-29T11:32:49.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.500+0000 7f7a3fe00700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a2806af00 0x7f7a2806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:49.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.500+0000 7f7a3fe00700 1 -- 172.21.15.12:0/4268044901 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f7a34089920 con 0x7f7a480d2370 2024-03-29T11:32:49.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.501+0000 7f7a47e00700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a2806af00 0x7f7a2806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:49.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.501+0000 7f7a47e00700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a2806af00 0x7f7a2806d3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7a38007fd0 tx=0x7f7a38009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:49.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.505+0000 7f7a3fe00700 1 -- 172.21.15.12:0/4268044901 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7a34054240 con 0x7f7a480d2370 2024-03-29T11:32:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:49.765+0000 7f7a4f000700 1 -- 172.21.15.12:0/4268044901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7a4815f6b0 con 0x7f7a480d2370 2024-03-29T11:32:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.341+0000 7f7a3fe00700 1 -- 172.21.15.12:0/4268044901 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v127) v1 ==== 121+0+0 (secure 0 0 0) 0x7f7a34057860 con 0x7f7a480d2370 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 -- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a2806af00 msgr2=0x7f7a2806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a2806af00 0x7f7a2806d3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7a38007fd0 tx=0x7f7a38009040 comp rx=0 tx=0).stop 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 -- 172.21.15.12:0/4268044901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 msgr2=0x7f7a48159f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 0x7f7a48159f00 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400e810 tx=0x7f7a3400eb20 comp rx=0 tx=0).stop 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 -- 172.21.15.12:0/4268044901 shutdown_connections 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a2806af00 0x7f7a2806d3c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a4815d9d0 0x7f7a480cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a480d16a0 0x7f7a480cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 --2- 172.21.15.12:0/4268044901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a480d2370 0x7f7a48159f00 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 -- 172.21.15.12:0/4268044901 >> 172.21.15.12:0/4268044901 conn(0x7f7a4808aff0 msgr2=0x7f7a480b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 -- 172.21.15.12:0/4268044901 shutdown_connections 2024-03-29T11:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.344+0000 7f7a4f000700 1 -- 172.21.15.12:0/4268044901 wait complete. 2024-03-29T11:32:50.359 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T11:32:50.359 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-03-29T11:32:50.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.566+0000 7f9fe8200700 1 Processor -- start 2024-03-29T11:32:50.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.566+0000 7f9fe8200700 1 -- start start 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.567+0000 7f9fe8200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9fe015dfe0 0x7f9fe00ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.567+0000 7f9fe8200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe00ce670 0x7f9fe00cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.567+0000 7f9fe8200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe00cb310 con 0x7f9fe015a5c0 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.567+0000 7f9fe8200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe00ceb00 con 0x7f9fe015dfe0 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.567+0000 7f9fe8200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe00711a0 con 0x7f9fe00ce670 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.567+0000 7f9fe5800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9fe015a5c0 0x7f9fe00709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59566/0 (socket says 172.21.15.12:59566) 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.567+0000 7f9fe5800700 1 -- 172.21.15.12:0/1991145839 learned_addr learned my addr 172.21.15.12:0/1991145839 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.568+0000 7f9fe4e00700 1 --2- 172.21.15.12:0/1991145839 >> v2:172.21.15.12:3300/0 conn(0x7f9fe015dfe0 0x7f9fe00ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.568+0000 7f9fe6200700 1 --2- 172.21.15.12:0/1991145839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe00ce670 0x7f9fe00cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.568+0000 7f9fdfe00700 1 -- 172.21.15.12:0/1991145839 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692728956 0 0) 0x7f9fe00cb310 con 0x7f9fe015a5c0 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.568+0000 7f9fdfe00700 1 -- 172.21.15.12:0/1991145839 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc80035b0 con 0x7f9fe015a5c0 2024-03-29T11:32:50.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.568+0000 7f9fe4e00700 1 -- 172.21.15.12:0/1991145839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe00ce670 msgr2=0x7f9fe00cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:50.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.568+0000 7f9fe4e00700 1 --2- 172.21.15.12:0/1991145839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe00ce670 0x7f9fe00cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:50.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.568+0000 7f9fe4e00700 1 -- 172.21.15.12:0/1991145839 >> v1:172.21.15.12:6789/0 conn(0x7f9fe015a5c0 legacy=0x7f9fe00709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:50.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.568+0000 7f9fe4e00700 1 -- 172.21.15.12:0/1991145839 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0071480 con 0x7f9fe015dfe0 2024-03-29T11:32:50.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.568+0000 7f9fe4e00700 1 --2- 172.21.15.12:0/1991145839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015dfe0 0x7f9fe00ce130 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc009840 tx=0x7f9fcc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970666a86e644caa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fdfe00700 1 -- 172.21.15.12:0/1991145839 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc011830 con 0x7f9fe015dfe0 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fdfe00700 1 -- 172.21.15.12:0/1991145839 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fcc011990 con 0x7f9fe015dfe0 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fdfe00700 1 -- 172.21.15.12:0/1991145839 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc011d20 con 0x7f9fe015dfe0 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fe8200700 1 -- 172.21.15.12:0/1991145839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015dfe0 msgr2=0x7f9fe00ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fe8200700 1 --2- 172.21.15.12:0/1991145839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015dfe0 0x7f9fe00ce130 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc009840 tx=0x7f9fcc009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fe8200700 1 -- 172.21.15.12:0/1991145839 shutdown_connections 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fe8200700 1 --2- 172.21.15.12:0/1991145839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe00ce670 0x7f9fe00cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fe8200700 1 --2- 172.21.15.12:0/1991145839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015dfe0 0x7f9fe00ce130 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fe8200700 1 -- 172.21.15.12:0/1991145839 >> 172.21.15.12:0/1991145839 conn(0x7f9fe00b0d70 msgr2=0x7f9fe00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fe8200700 1 -- 172.21.15.12:0/1991145839 shutdown_connections 2024-03-29T11:32:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.569+0000 7f9fe8200700 1 -- 172.21.15.12:0/1991145839 wait complete. 2024-03-29T11:32:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.571+0000 7f9fe8200700 1 Processor -- start 2024-03-29T11:32:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.571+0000 7f9fe8200700 1 -- start start 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.572+0000 7f9fe8200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9fe00ce670 0x7f9fe0155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.572+0000 7f9fe8200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015a5c0 0x7f9fe01f06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.572+0000 7f9fe8200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe015dfe0 0x7f9fe014dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.572+0000 7f9fe8200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe014eea0 con 0x7f9fe015a5c0 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.572+0000 7f9fe8200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe0148e90 con 0x7f9fe00ce670 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.572+0000 7f9fe8200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe0155580 con 0x7f9fe015dfe0 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015a5c0 0x7f9fe01f06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015a5c0 0x7f9fe01f06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46282/0 (socket says 172.21.15.12:46282) 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe4e00700 1 -- 172.21.15.12:0/3562335072 learned_addr learned my addr 172.21.15.12:0/3562335072 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe6200700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe015dfe0 0x7f9fe014dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe5800700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9fe00ce670 0x7f9fe0155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe4e00700 1 -- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe015dfe0 msgr2=0x7f9fe014dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:50.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe4e00700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe015dfe0 0x7f9fe014dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:50.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe4e00700 1 -- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9fe00ce670 msgr2=0x7f9fe0155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:50.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe4e00700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9fe00ce670 0x7f9fe0155040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:50.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe4e00700 1 -- 172.21.15.12:0/3562335072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc0094a0 con 0x7f9fe015a5c0 2024-03-29T11:32:50.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe6200700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe015dfe0 0x7f9fe014dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:50.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.573+0000 7f9fe4e00700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015a5c0 0x7f9fe01f06b0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc009fd0 tx=0x7f9fcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:50.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.574+0000 7f9fdd600700 1 -- 172.21.15.12:0/3562335072 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc0036a0 con 0x7f9fe015a5c0 2024-03-29T11:32:50.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.574+0000 7f9fdd600700 1 -- 172.21.15.12:0/3562335072 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fcc01e070 con 0x7f9fe015a5c0 2024-03-29T11:32:50.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.574+0000 7f9fdd600700 1 -- 172.21.15.12:0/3562335072 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc003a60 con 0x7f9fe015a5c0 2024-03-29T11:32:50.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.574+0000 7f9fe8200700 1 -- 172.21.15.12:0/3562335072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe0155820 con 0x7f9fe015a5c0 2024-03-29T11:32:50.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.574+0000 7f9fe8200700 1 -- 172.21.15.12:0/3562335072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe0155de0 con 0x7f9fe015a5c0 2024-03-29T11:32:50.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.575+0000 7f9fe8200700 1 -- 172.21.15.12:0/3562335072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe0067720 con 0x7f9fe015a5c0 2024-03-29T11:32:50.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.576+0000 7f9fdd600700 1 -- 172.21.15.12:0/3562335072 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f9fcc003bc0 con 0x7f9fe015a5c0 2024-03-29T11:32:50.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.576+0000 7f9fdd600700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9fc006aed0 0x7f9fc006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:50.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.577+0000 7f9fdd600700 1 -- 172.21.15.12:0/3562335072 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f9fcc08a9a0 con 0x7f9fe015a5c0 2024-03-29T11:32:50.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.577+0000 7f9fe5800700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9fc006aed0 0x7f9fc006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.578+0000 7f9fe5800700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9fc006aed0 0x7f9fc006d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9fd40011e0 tx=0x7f9fd4009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:50.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.581+0000 7f9fdd600700 1 -- 172.21.15.12:0/3562335072 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9fcc055500 con 0x7f9fe015a5c0 2024-03-29T11:32:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:50.842+0000 7f9fe8200700 1 -- 172.21.15.12:0/3562335072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9fe00b1ad0 con 0x7f9fe015a5c0 2024-03-29T11:32:51.337 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:32:51.346 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:32:51.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.348+0000 7f9fdd600700 1 -- 172.21.15.12:0/3562335072 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v128) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9fcc058b20 con 0x7f9fe015a5c0 2024-03-29T11:32:51.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 -- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9fc006aed0 msgr2=0x7f9fc006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:51.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9fc006aed0 0x7f9fc006d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9fd40011e0 tx=0x7f9fd4009040 comp rx=0 tx=0).stop 2024-03-29T11:32:51.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 -- 172.21.15.12:0/3562335072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015a5c0 msgr2=0x7f9fe01f06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:51.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015a5c0 0x7f9fe01f06b0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc009fd0 tx=0x7f9fcc004750 comp rx=0 tx=0).stop 2024-03-29T11:32:51.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 -- 172.21.15.12:0/3562335072 shutdown_connections 2024-03-29T11:32:51.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9fc006aed0 0x7f9fc006d390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9fe015dfe0 0x7f9fe014dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9fe00ce670 0x7f9fe0155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 --2- 172.21.15.12:0/3562335072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fe015a5c0 0x7f9fe01f06b0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 -- 172.21.15.12:0/3562335072 >> 172.21.15.12:0/3562335072 conn(0x7f9fe00b0d70 msgr2=0x7f9fe00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:51.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 -- 172.21.15.12:0/3562335072 shutdown_connections 2024-03-29T11:32:51.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.351+0000 7f9fe8200700 1 -- 172.21.15.12:0/3562335072 wait complete. 2024-03-29T11:32:51.368 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-03-29T11:32:51.368 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:51.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.621+0000 7fb131e00700 1 Processor -- start 2024-03-29T11:32:51.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.622+0000 7fb131e00700 1 -- start start 2024-03-29T11:32:51.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb131e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb12c15d640 0x7fb12c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:51.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb131e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb12c0ceec0 0x7fb12c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:51.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb131e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb12c0d30d0 con 0x7fb12c161800 2024-03-29T11:32:51.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb131e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb12c161c90 con 0x7fb12c0ceec0 2024-03-29T11:32:51.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb131e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c0b6840 con 0x7fb12c15d640 2024-03-29T11:32:51.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb12a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb12c0ceec0 0x7fb12c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:51.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb12a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb12c0ceec0 0x7fb12c1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46296/0 (socket says 172.21.15.12:46296) 2024-03-29T11:32:51.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb12a000700 1 -- 172.21.15.12:0/3922184264 learned_addr learned my addr 172.21.15.12:0/3922184264 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:51.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb12a000700 1 -- 172.21.15.12:0/3922184264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb12c15d640 msgr2=0x7fb12c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:51.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb12a000700 1 --2- 172.21.15.12:0/3922184264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb12c15d640 0x7fb12c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb12a000700 1 -- 172.21.15.12:0/3922184264 >> v1:172.21.15.159:6789/0 conn(0x7fb12c161800 legacy=0x7fb12c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.623+0000 7fb12a000700 1 -- 172.21.15.12:0/3922184264 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c0d2880 con 0x7fb12c0ceec0 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.624+0000 7fb12a000700 1 --2- 172.21.15.12:0/3922184264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c0ceec0 0x7fb12c1612c0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fb1180098f0 tx=0x7fb118009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d12ebf5164ecca server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.624+0000 7fb129600700 1 -- 172.21.15.12:0/3922184264 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118004230 con 0x7fb12c0ceec0 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.624+0000 7fb129600700 1 -- 172.21.15.12:0/3922184264 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb118004390 con 0x7fb12c0ceec0 2024-03-29T11:32:51.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.624+0000 7fb129600700 1 -- 172.21.15.12:0/3922184264 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1180036a0 con 0x7fb12c0ceec0 2024-03-29T11:32:51.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.624+0000 7fb131e00700 1 -- 172.21.15.12:0/3922184264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c0ceec0 msgr2=0x7fb12c1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:51.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.624+0000 7fb131e00700 1 --2- 172.21.15.12:0/3922184264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c0ceec0 0x7fb12c1612c0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fb1180098f0 tx=0x7fb118009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:51.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.625+0000 7fb131e00700 1 -- 172.21.15.12:0/3922184264 shutdown_connections 2024-03-29T11:32:51.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.625+0000 7fb131e00700 1 --2- 172.21.15.12:0/3922184264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb12c15d640 0x7fb12c0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.625+0000 7fb131e00700 1 --2- 172.21.15.12:0/3922184264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c0ceec0 0x7fb12c1612c0 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.625+0000 7fb131e00700 1 -- 172.21.15.12:0/3922184264 >> 172.21.15.12:0/3922184264 conn(0x7fb12c08aff0 msgr2=0x7fb12c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:51.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.625+0000 7fb131e00700 1 -- 172.21.15.12:0/3922184264 shutdown_connections 2024-03-29T11:32:51.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.625+0000 7fb131e00700 1 -- 172.21.15.12:0/3922184264 wait complete. 2024-03-29T11:32:51.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.627+0000 7fb131e00700 1 Processor -- start 2024-03-29T11:32:51.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.627+0000 7fb131e00700 1 -- start start 2024-03-29T11:32:51.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.628+0000 7fb131e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb12c0ceec0 0x7fb12c09dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.628+0000 7fb131e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb12c15d640 0x7fb12c1f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.628+0000 7fb131e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c161800 0x7fb12c0c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.628+0000 7fb131e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c0c7080 con 0x7fb12c161800 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.628+0000 7fb131e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c0c7c00 con 0x7fb12c0ceec0 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.628+0000 7fb131e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c0c72d0 con 0x7fb12c15d640 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c161800 0x7fb12c0c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12a000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb12c15d640 0x7fb12c1f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c161800 0x7fb12c0c8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46312/0 (socket says 172.21.15.12:46312) 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12b400700 1 -- 172.21.15.12:0/517958669 learned_addr learned my addr 172.21.15.12:0/517958669 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12aa00700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb12c0ceec0 0x7fb12c09dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12b400700 1 -- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb12c15d640 msgr2=0x7fb12c1f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12b400700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb12c15d640 0x7fb12c1f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12b400700 1 -- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb12c0ceec0 msgr2=0x7fb12c09dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:51.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12b400700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb12c0ceec0 0x7fb12c09dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12b400700 1 -- 172.21.15.12:0/517958669 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118009550 con 0x7fb12c161800 2024-03-29T11:32:51.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.629+0000 7fb12b400700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c161800 0x7fb12c0c8090 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7fb12400cc10 tx=0x7fb12400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:51.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.630+0000 7fb123400700 1 -- 172.21.15.12:0/517958669 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12400aa60 con 0x7fb12c161800 2024-03-29T11:32:51.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.630+0000 7fb123400700 1 -- 172.21.15.12:0/517958669 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12400abc0 con 0x7fb12c161800 2024-03-29T11:32:51.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.630+0000 7fb131e00700 1 -- 172.21.15.12:0/517958669 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12c0c75b0 con 0x7fb12c161800 2024-03-29T11:32:51.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.630+0000 7fb123400700 1 -- 172.21.15.12:0/517958669 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb124012420 con 0x7fb12c161800 2024-03-29T11:32:51.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.630+0000 7fb131e00700 1 -- 172.21.15.12:0/517958669 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12c159f30 con 0x7fb12c161800 2024-03-29T11:32:51.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.631+0000 7fb131e00700 1 -- 172.21.15.12:0/517958669 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12c0c7d70 con 0x7fb12c161800 2024-03-29T11:32:51.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.632+0000 7fb123400700 1 -- 172.21.15.12:0/517958669 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb124012580 con 0x7fb12c161800 2024-03-29T11:32:51.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.633+0000 7fb123400700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb10c06aed0 0x7fb10c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:51.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.633+0000 7fb123400700 1 -- 172.21.15.12:0/517958669 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7fb124089e80 con 0x7fb12c161800 2024-03-29T11:32:51.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.633+0000 7fb12aa00700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb10c06aed0 0x7fb10c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:51.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.634+0000 7fb12aa00700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb10c06aed0 0x7fb10c06d390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009a10 tx=0x7fb11c009190 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:51.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.637+0000 7fb123400700 1 -- 172.21.15.12:0/517958669 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb1240548b0 con 0x7fb12c161800 2024-03-29T11:32:51.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.893+0000 7fb131e00700 1 -- 172.21.15.12:0/517958669 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb12c067720 con 0x7fb12c161800 2024-03-29T11:32:51.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.893+0000 7fb123400700 1 -- 172.21.15.12:0/517958669 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5972 (secure 0 0 0) 0x7fb124057ed0 con 0x7fb12c161800 2024-03-29T11:32:51.889 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:51.889 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":128,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:32.933737+0000","modified":"2024-03-29T11:32:36.420698+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11853":{"gid":11853,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3711953177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3711953177},{"type":"v1","addr":"172.21.15.12:6834","nonce":3711953177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:47.849181+0000","modified":"2024-03-29T11:32:51.340904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11416":{"gid":11416,"name":"d","rank":0,"incarnation":124,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-29T11:32:51.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 -- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb10c06aed0 msgr2=0x7fb10c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:51.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb10c06aed0 0x7fb10c06d390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009a10 tx=0x7fb11c009190 comp rx=0 tx=0).stop 2024-03-29T11:32:51.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 -- 172.21.15.12:0/517958669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c161800 msgr2=0x7fb12c0c8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:51.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c161800 0x7fb12c0c8090 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7fb12400cc10 tx=0x7fb12400cf20 comp rx=0 tx=0).stop 2024-03-29T11:32:51.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 -- 172.21.15.12:0/517958669 shutdown_connections 2024-03-29T11:32:51.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb10c06aed0 0x7fb10c06d390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb12c161800 0x7fb12c0c8090 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb12c0ceec0 0x7fb12c09dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 --2- 172.21.15.12:0/517958669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb12c15d640 0x7fb12c1f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:51.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 -- 172.21.15.12:0/517958669 >> 172.21.15.12:0/517958669 conn(0x7fb12c08aff0 msgr2=0x7fb12c0b7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:51.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 -- 172.21.15.12:0/517958669 shutdown_connections 2024-03-29T11:32:51.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:51.896+0000 7fb131e00700 1 -- 172.21.15.12:0/517958669 wait complete. 2024-03-29T11:32:51.893 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 128 2024-03-29T11:32:51.911 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:52.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.119+0000 7f469ba00700 1 Processor -- start 2024-03-29T11:32:52.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.119+0000 7f469ba00700 1 -- start start 2024-03-29T11:32:52.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.120+0000 7f469ba00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f469415d640 0x7f46940d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:52.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.121+0000 7f469ba00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46940d30d0 con 0x7f46941619d0 2024-03-29T11:32:52.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.121+0000 7f469ba00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46940b6840 con 0x7f46940ceec0 2024-03-29T11:32:52.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.121+0000 7f469ba00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46940d2880 con 0x7f469415d640 2024-03-29T11:32:52.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.121+0000 7f4699000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f469415d640 0x7f46940d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:52.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.121+0000 7f4699a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f46941619d0 0x7f46940b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59570/0 (socket says 172.21.15.12:59570) 2024-03-29T11:32:52.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.121+0000 7f4699a00700 1 -- 172.21.15.12:0/756685678 learned_addr learned my addr 172.21.15.12:0/756685678 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:52.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.121+0000 7f4699000700 1 -- 172.21.15.12:0/756685678 >> v1:172.21.15.159:6789/0 conn(0x7f46940ceec0 legacy=0x7f46941612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.121+0000 7f4699000700 1 -- 172.21.15.12:0/756685678 >> v1:172.21.15.12:6789/0 conn(0x7f46941619d0 legacy=0x7f46940b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.121+0000 7f4699000700 1 -- 172.21.15.12:0/756685678 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46940d2b30 con 0x7f469415d640 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.121+0000 7f4693400700 1 -- 172.21.15.12:0/756685678 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528938930 0 0) 0x7f46940d30d0 con 0x7f46941619d0 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.122+0000 7f4699000700 1 --2- 172.21.15.12:0/756685678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 0x7f46940d2340 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f4688009840 tx=0x7f4688009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8684be404fbc1406 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.122+0000 7f4693400700 1 -- 172.21.15.12:0/756685678 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688011830 con 0x7f469415d640 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.122+0000 7f4693400700 1 -- 172.21.15.12:0/756685678 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4688011990 con 0x7f469415d640 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.122+0000 7f4693400700 1 -- 172.21.15.12:0/756685678 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688011d20 con 0x7f469415d640 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.122+0000 7f469ba00700 1 -- 172.21.15.12:0/756685678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 msgr2=0x7f46940d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.122+0000 7f469ba00700 1 --2- 172.21.15.12:0/756685678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 0x7f46940d2340 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f4688009840 tx=0x7f4688009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.123+0000 7f469ba00700 1 -- 172.21.15.12:0/756685678 shutdown_connections 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.123+0000 7f469ba00700 1 --2- 172.21.15.12:0/756685678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 0x7f46940d2340 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.123+0000 7f469ba00700 1 -- 172.21.15.12:0/756685678 >> 172.21.15.12:0/756685678 conn(0x7f469408aff0 msgr2=0x7f46940d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:52.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.123+0000 7f469ba00700 1 -- 172.21.15.12:0/756685678 shutdown_connections 2024-03-29T11:32:52.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.123+0000 7f469ba00700 1 -- 172.21.15.12:0/756685678 wait complete. 2024-03-29T11:32:52.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.124+0000 7f469ba00700 1 Processor -- start 2024-03-29T11:32:52.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.124+0000 7f469ba00700 1 -- start start 2024-03-29T11:32:52.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.125+0000 7f469ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46940ceec0 0x7f4694158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:52.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.125+0000 7f469ba00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 0x7f46941e8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:52.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.125+0000 7f469ba00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46941619d0 0x7f46940c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:52.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.125+0000 7f469ba00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46941e7510 con 0x7f469415d640 2024-03-29T11:32:52.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.125+0000 7f469ba00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46941e8370 con 0x7f46941619d0 2024-03-29T11:32:52.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.125+0000 7f469ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4694158cc0 con 0x7f46940ceec0 2024-03-29T11:32:52.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4693e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 0x7f46941e8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:52.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4693e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 0x7f46941e8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46338/0 (socket says 172.21.15.12:46338) 2024-03-29T11:32:52.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4693e00700 1 -- 172.21.15.12:0/1653459007 learned_addr learned my addr 172.21.15.12:0/1653459007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:52.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4699000700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46940ceec0 0x7f4694158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:52.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4699a00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46941619d0 0x7f46940c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:52.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4693e00700 1 -- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46940ceec0 msgr2=0x7f4694158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4693e00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46940ceec0 0x7f4694158780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4693e00700 1 -- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46941619d0 msgr2=0x7f46940c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4693e00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46941619d0 0x7f46940c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4693e00700 1 -- 172.21.15.12:0/1653459007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46880094a0 con 0x7f469415d640 2024-03-29T11:32:52.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.126+0000 7f4693e00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 0x7f46941e8f60 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f468000c4e0 tx=0x7f468000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:52.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.127+0000 7f4690c00700 1 -- 172.21.15.12:0/1653459007 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46800052d0 con 0x7f469415d640 2024-03-29T11:32:52.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.127+0000 7f4690c00700 1 -- 172.21.15.12:0/1653459007 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4680005430 con 0x7f469415d640 2024-03-29T11:32:52.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.127+0000 7f469ba00700 1 -- 172.21.15.12:0/1653459007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46941e7760 con 0x7f469415d640 2024-03-29T11:32:52.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.127+0000 7f4690c00700 1 -- 172.21.15.12:0/1653459007 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46800038c0 con 0x7f469415d640 2024-03-29T11:32:52.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.127+0000 7f469ba00700 1 -- 172.21.15.12:0/1653459007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46941e7d20 con 0x7f469415d640 2024-03-29T11:32:52.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.128+0000 7f469ba00700 1 -- 172.21.15.12:0/1653459007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46941e84e0 con 0x7f469415d640 2024-03-29T11:32:52.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.129+0000 7f4690c00700 1 -- 172.21.15.12:0/1653459007 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4680003a20 con 0x7f469415d640 2024-03-29T11:32:52.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.130+0000 7f4690c00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f467406aed0 0x7f467406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:52.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.130+0000 7f4690c00700 1 -- 172.21.15.12:0/1653459007 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f4680088ee0 con 0x7f469415d640 2024-03-29T11:32:52.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.130+0000 7f4699000700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f467406aed0 0x7f467406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:52.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.131+0000 7f4699000700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f467406aed0 0x7f467406d390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f46880090b0 tx=0x7f4688008030 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:52.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.134+0000 7f4690c00700 1 -- 172.21.15.12:0/1653459007 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4680053a80 con 0x7f469415d640 2024-03-29T11:32:52.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.351+0000 7f469ba00700 1 -- 172.21.15.12:0/1653459007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4694067720 con 0x7f469415d640 2024-03-29T11:32:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.352+0000 7f4690c00700 1 -- 172.21.15.12:0/1653459007 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+19835 (secure 0 0 0) 0x7f4680017090 con 0x7f469415d640 2024-03-29T11:32:52.348 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:52.349 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":252,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:32:48.851660+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:32:45.400233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:32:46.836046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 -- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f467406aed0 msgr2=0x7f467406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f467406aed0 0x7f467406d390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f46880090b0 tx=0x7f4688008030 comp rx=0 tx=0).stop 2024-03-29T11:32:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 -- 172.21.15.12:0/1653459007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 msgr2=0x7f46941e8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 0x7f46941e8f60 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f468000c4e0 tx=0x7f468000c8a0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 -- 172.21.15.12:0/1653459007 shutdown_connections 2024-03-29T11:32:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f467406aed0 0x7f467406d390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46941619d0 0x7f46940c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46940ceec0 0x7f4694158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 --2- 172.21.15.12:0/1653459007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f469415d640 0x7f46941e8f60 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 -- 172.21.15.12:0/1653459007 >> 172.21.15.12:0/1653459007 conn(0x7f469408aff0 msgr2=0x7f469415c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 -- 172.21.15.12:0/1653459007 shutdown_connections 2024-03-29T11:32:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.355+0000 7f469ba00700 1 -- 172.21.15.12:0/1653459007 wait complete. 2024-03-29T11:32:52.371 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T11:32:52.371 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:32:52.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.578+0000 7f32af000700 1 Processor -- start 2024-03-29T11:32:52.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.578+0000 7f32af000700 1 -- start start 2024-03-29T11:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.579+0000 7f32af000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f32a8161ca0 0x7f32a80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.579+0000 7f32af000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80ceda0 con 0x7f32a8160f10 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.579+0000 7f32af000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a81612f0 con 0x7f32a80d18b0 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.579+0000 7f32af000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a80d1d30 con 0x7f32a8161ca0 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.579+0000 7f32a7e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f32a8160f10 0x7f32a80d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59572/0 (socket says 172.21.15.12:59572) 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.579+0000 7f32a7e00700 1 -- 172.21.15.12:0/2124387555 learned_addr learned my addr 172.21.15.12:0/2124387555 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.580+0000 7f32ad000700 1 --2- 172.21.15.12:0/2124387555 >> v2:172.21.15.12:3300/0 conn(0x7f32a8161ca0 0x7f32a80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.580+0000 7f32a6a00700 1 -- 172.21.15.12:0/2124387555 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010208456 0 0) 0x7f32a80ceda0 con 0x7f32a8160f10 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.580+0000 7f32a6a00700 1 -- 172.21.15.12:0/2124387555 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f328c0035b0 con 0x7f32a8160f10 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.580+0000 7f32ad000700 1 -- 172.21.15.12:0/2124387555 >> v1:172.21.15.159:6789/0 conn(0x7f32a80d18b0 legacy=0x7f32a8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.580+0000 7f32ad000700 1 -- 172.21.15.12:0/2124387555 >> v1:172.21.15.12:6789/0 conn(0x7f32a8160f10 legacy=0x7f32a80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.580+0000 7f32ad000700 1 -- 172.21.15.12:0/2124387555 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a80ce470 con 0x7f32a8161ca0 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.580+0000 7f32ad000700 1 --2- 172.21.15.12:0/2124387555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a8161ca0 0x7f32a80cf230 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009d90 tx=0x7f32a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bccf3b4a9f5d88e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.580+0000 7f32a6a00700 1 -- 172.21.15.12:0/2124387555 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a000e070 con 0x7f32a8161ca0 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.581+0000 7f32a6a00700 1 -- 172.21.15.12:0/2124387555 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a000bb20 con 0x7f32a8161ca0 2024-03-29T11:32:52.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.581+0000 7f32a6a00700 1 -- 172.21.15.12:0/2124387555 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0012430 con 0x7f32a8161ca0 2024-03-29T11:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.581+0000 7f32af000700 1 -- 172.21.15.12:0/2124387555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a8161ca0 msgr2=0x7f32a80cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.581+0000 7f32af000700 1 --2- 172.21.15.12:0/2124387555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a8161ca0 0x7f32a80cf230 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009d90 tx=0x7f32a000b330 comp rx=0 tx=0).stop 2024-03-29T11:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.581+0000 7f32af000700 1 -- 172.21.15.12:0/2124387555 shutdown_connections 2024-03-29T11:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.581+0000 7f32af000700 1 --2- 172.21.15.12:0/2124387555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a8161ca0 0x7f32a80cf230 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.581+0000 7f32af000700 1 -- 172.21.15.12:0/2124387555 >> 172.21.15.12:0/2124387555 conn(0x7f32a808aff0 msgr2=0x7f32a80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.581+0000 7f32af000700 1 -- 172.21.15.12:0/2124387555 shutdown_connections 2024-03-29T11:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.581+0000 7f32af000700 1 -- 172.21.15.12:0/2124387555 wait complete. 2024-03-29T11:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.583+0000 7f32af000700 1 Processor -- start 2024-03-29T11:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.583+0000 7f32af000700 1 -- start start 2024-03-29T11:32:52.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.584+0000 7f32af000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a80d18b0 0x7f32a814ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:52.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.584+0000 7f32af000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32a8160f10 0x7f32a815d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:52.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.584+0000 7f32af000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a8161ca0 0x7f32a815f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:52.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.584+0000 7f32af000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a814bf70 con 0x7f32a80d18b0 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.584+0000 7f32af000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a815da20 con 0x7f32a8160f10 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.584+0000 7f32af000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a815dd00 con 0x7f32a8161ca0 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.584+0000 7f32a7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a80d18b0 0x7f32a814ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.584+0000 7f32a7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a80d18b0 0x7f32a814ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46352/0 (socket says 172.21.15.12:46352) 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.584+0000 7f32a7e00700 1 -- 172.21.15.12:0/3546000865 learned_addr learned my addr 172.21.15.12:0/3546000865 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.585+0000 7f32a7400700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32a8160f10 0x7f32a815d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.585+0000 7f32ad000700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a8161ca0 0x7f32a815f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.585+0000 7f32a7e00700 1 -- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a8161ca0 msgr2=0x7f32a815f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.585+0000 7f32a7e00700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a8161ca0 0x7f32a815f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.585+0000 7f32a7e00700 1 -- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32a8160f10 msgr2=0x7f32a815d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.585+0000 7f32a7e00700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32a8160f10 0x7f32a815d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.585+0000 7f32a7e00700 1 -- 172.21.15.12:0/3546000865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0009a70 con 0x7f32a80d18b0 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.585+0000 7f32a7e00700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a80d18b0 0x7f32a814ba30 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f3298001eb0 tx=0x7f329800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.585+0000 7f329fe00700 1 -- 172.21.15.12:0/3546000865 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298005d70 con 0x7f32a80d18b0 2024-03-29T11:32:52.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.586+0000 7f329fe00700 1 -- 172.21.15.12:0/3546000865 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32980070d0 con 0x7f32a80d18b0 2024-03-29T11:32:52.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.586+0000 7f329fe00700 1 -- 172.21.15.12:0/3546000865 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298007460 con 0x7f32a80d18b0 2024-03-29T11:32:52.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.586+0000 7f32af000700 1 -- 172.21.15.12:0/3546000865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a815dfe0 con 0x7f32a80d18b0 2024-03-29T11:32:52.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.586+0000 7f32af000700 1 -- 172.21.15.12:0/3546000865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a815e5a0 con 0x7f32a80d18b0 2024-03-29T11:32:52.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.587+0000 7f32af000700 1 -- 172.21.15.12:0/3546000865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a8067720 con 0x7f32a80d18b0 2024-03-29T11:32:52.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.588+0000 7f329fe00700 1 -- 172.21.15.12:0/3546000865 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f32980075c0 con 0x7f32a80d18b0 2024-03-29T11:32:52.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.589+0000 7f329fe00700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f328806aed0 0x7f328806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:52.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.589+0000 7f329fe00700 1 -- 172.21.15.12:0/3546000865 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f3298088ee0 con 0x7f32a80d18b0 2024-03-29T11:32:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.589+0000 7f32a7400700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f328806aed0 0x7f328806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.590+0000 7f32a7400700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f328806aed0 0x7f328806d390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f32a815f090 tx=0x7f3294009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:52.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.593+0000 7f329fe00700 1 -- 172.21.15.12:0/3546000865 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32980539b0 con 0x7f32a80d18b0 2024-03-29T11:32:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.803+0000 7f32af000700 1 -- 172.21.15.12:0/3546000865 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32a80cea30 con 0x7f328806aed0 2024-03-29T11:32:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.819+0000 7f329fe00700 1 -- 172.21.15.12:0/3546000865 <== mgr.11050 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528549 (secure 0 0 0) 0x7f3294001cc0 con 0x7f328806aed0 2024-03-29T11:32:52.816 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:52.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.824+0000 7f32af000700 1 -- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f328806aed0 msgr2=0x7f328806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.824+0000 7f32af000700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f328806aed0 0x7f328806d390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f32a815f090 tx=0x7f3294009040 comp rx=0 tx=0).stop 2024-03-29T11:32:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.824+0000 7f32af000700 1 -- 172.21.15.12:0/3546000865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a80d18b0 msgr2=0x7f32a814ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.824+0000 7f32af000700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a80d18b0 0x7f32a814ba30 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f3298001eb0 tx=0x7f329800c7b0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.825+0000 7f32af000700 1 -- 172.21.15.12:0/3546000865 shutdown_connections 2024-03-29T11:32:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.825+0000 7f32af000700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f328806aed0 0x7f328806d390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.825+0000 7f32af000700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a8161ca0 0x7f32a815f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.825+0000 7f32af000700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a80d18b0 0x7f32a814ba30 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.825+0000 7f32af000700 1 --2- 172.21.15.12:0/3546000865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32a8160f10 0x7f32a815d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:52.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.825+0000 7f32af000700 1 -- 172.21.15.12:0/3546000865 >> 172.21.15.12:0/3546000865 conn(0x7f32a808aff0 msgr2=0x7f32a80d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:52.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.825+0000 7f32af000700 1 -- 172.21.15.12:0/3546000865 shutdown_connections 2024-03-29T11:32:52.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:52.825+0000 7f32af000700 1 -- 172.21.15.12:0/3546000865 wait complete. 2024-03-29T11:32:52.833 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:32:52.838 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-03-29T11:32:52.533751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596333,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":742,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":467,"ondisk_log_size":467,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":321,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":434036,"kb_used_data":25968,"kb_used_omap":12,"kb_used_meta":407923,"kb_avail":754540684,"statfs":{"total":773094113280,"available":772649660416,"internally_reserved":0,"allocated":26591232,"data_stored":13612350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":417713721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001980"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193329+0000","last_change":"2024-03-29T11:32:47.880663+0000","last_active":"2024-03-29T11:32:49.193329+0000","last_peered":"2024-03-29T11:32:49.193329+0000","last_clean":"2024-03-29T11:32:49.193329+0000","last_became_active":"2024-03-29T11:32:47.879936+0000","last_became_peered":"2024-03-29T11:32:47.879936+0000","last_unstale":"2024-03-29T11:32:49.193329+0000","last_undegraded":"2024-03-29T11:32:49.193329+0000","last_fullsized":"2024-03-29T11:32:49.193329+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:57.835090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.337907+0000","last_change":"2024-03-29T11:32:47.340805+0000","last_active":"2024-03-29T11:32:49.337907+0000","last_peered":"2024-03-29T11:32:49.337907+0000","last_clean":"2024-03-29T11:32:49.337907+0000","last_became_active":"2024-03-29T11:32:47.340542+0000","last_became_peered":"2024-03-29T11:32:47.340542+0000","last_unstale":"2024-03-29T11:32:49.337907+0000","last_undegraded":"2024-03-29T11:32:49.337907+0000","last_fullsized":"2024-03-29T11:32:49.337907+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:51:06.069753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854251+0000","last_change":"2024-03-29T11:32:32.953487+0000","last_active":"2024-03-29T11:32:47.854251+0000","last_peered":"2024-03-29T11:32:47.854251+0000","last_clean":"2024-03-29T11:32:47.854251+0000","last_became_active":"2024-03-29T11:32:32.953243+0000","last_became_peered":"2024-03-29T11:32:32.953243+0000","last_unstale":"2024-03-29T11:32:47.854251+0000","last_undegraded":"2024-03-29T11:32:47.854251+0000","last_fullsized":"2024-03-29T11:32:47.854251+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:11.369063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.861659+0000","last_change":"2024-03-29T11:32:47.881569+0000","last_active":"2024-03-29T11:32:48.861659+0000","last_peered":"2024-03-29T11:32:48.861659+0000","last_clean":"2024-03-29T11:32:48.861659+0000","last_became_active":"2024-03-29T11:32:47.881243+0000","last_became_peered":"2024-03-29T11:32:47.881243+0000","last_unstale":"2024-03-29T11:32:48.861659+0000","last_undegraded":"2024-03-29T11:32:48.861659+0000","last_fullsized":"2024-03-29T11:32:48.861659+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:08.340482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862041+0000","last_change":"2024-03-29T11:32:47.345546+0000","last_active":"2024-03-29T11:32:48.862041+0000","last_peered":"2024-03-29T11:32:48.862041+0000","last_clean":"2024-03-29T11:32:48.862041+0000","last_became_active":"2024-03-29T11:32:47.344388+0000","last_became_peered":"2024-03-29T11:32:47.344388+0000","last_unstale":"2024-03-29T11:32:48.862041+0000","last_undegraded":"2024-03-29T11:32:48.862041+0000","last_fullsized":"2024-03-29T11:32:48.862041+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:25:03.789446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.856105+0000","last_change":"2024-03-29T11:32:32.956013+0000","last_active":"2024-03-29T11:32:47.856105+0000","last_peered":"2024-03-29T11:32:47.856105+0000","last_clean":"2024-03-29T11:32:47.856105+0000","last_became_active":"2024-03-29T11:32:32.954012+0000","last_became_peered":"2024-03-29T11:32:32.954012+0000","last_unstale":"2024-03-29T11:32:47.856105+0000","last_undegraded":"2024-03-29T11:32:47.856105+0000","last_fullsized":"2024-03-29T11:32:47.856105+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:02:36.221843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862151+0000","last_change":"2024-03-29T11:32:47.878275+0000","last_active":"2024-03-29T11:32:48.862151+0000","last_peered":"2024-03-29T11:32:48.862151+0000","last_clean":"2024-03-29T11:32:48.862151+0000","last_became_active":"2024-03-29T11:32:47.878041+0000","last_became_peered":"2024-03-29T11:32:47.878041+0000","last_unstale":"2024-03-29T11:32:48.862151+0000","last_undegraded":"2024-03-29T11:32:48.862151+0000","last_fullsized":"2024-03-29T11:32:48.862151+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:19:11.800767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":10,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.860528+0000","last_change":"2024-03-29T11:32:47.351818+0000","last_active":"2024-03-29T11:32:47.860528+0000","last_peered":"2024-03-29T11:32:47.860528+0000","last_clean":"2024-03-29T11:32:47.860528+0000","last_became_active":"2024-03-29T11:32:47.351082+0000","last_became_peered":"2024-03-29T11:32:47.351082+0000","last_unstale":"2024-03-29T11:32:47.860528+0000","last_undegraded":"2024-03-29T11:32:47.860528+0000","last_fullsized":"2024-03-29T11:32:47.860528+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:08:10.182655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.867842+0000","last_change":"2024-03-29T11:32:32.966647+0000","last_active":"2024-03-29T11:32:47.867842+0000","last_peered":"2024-03-29T11:32:47.867842+0000","last_clean":"2024-03-29T11:32:47.867842+0000","last_became_active":"2024-03-29T11:32:32.966394+0000","last_became_peered":"2024-03-29T11:32:32.966394+0000","last_unstale":"2024-03-29T11:32:47.867842+0000","last_undegraded":"2024-03-29T11:32:47.867842+0000","last_fullsized":"2024-03-29T11:32:47.867842+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:22.600133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862318+0000","last_change":"2024-03-29T11:32:47.884486+0000","last_active":"2024-03-29T11:32:48.862318+0000","last_peered":"2024-03-29T11:32:48.862318+0000","last_clean":"2024-03-29T11:32:48.862318+0000","last_became_active":"2024-03-29T11:32:47.884250+0000","last_became_peered":"2024-03-29T11:32:47.884250+0000","last_unstale":"2024-03-29T11:32:48.862318+0000","last_undegraded":"2024-03-29T11:32:48.862318+0000","last_fullsized":"2024-03-29T11:32:48.862318+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:30.885593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":10,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861560+0000","last_change":"2024-03-29T11:32:47.350253+0000","last_active":"2024-03-29T11:32:47.861560+0000","last_peered":"2024-03-29T11:32:47.861560+0000","last_clean":"2024-03-29T11:32:47.861560+0000","last_became_active":"2024-03-29T11:32:47.349972+0000","last_became_peered":"2024-03-29T11:32:47.349972+0000","last_unstale":"2024-03-29T11:32:47.861560+0000","last_undegraded":"2024-03-29T11:32:47.861560+0000","last_fullsized":"2024-03-29T11:32:47.861560+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:18.636533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.865924+0000","last_change":"2024-03-29T11:32:32.955745+0000","last_active":"2024-03-29T11:32:47.865924+0000","last_peered":"2024-03-29T11:32:47.865924+0000","last_clean":"2024-03-29T11:32:47.865924+0000","last_became_active":"2024-03-29T11:32:32.955142+0000","last_became_peered":"2024-03-29T11:32:32.955142+0000","last_unstale":"2024-03-29T11:32:47.865924+0000","last_undegraded":"2024-03-29T11:32:47.865924+0000","last_fullsized":"2024-03-29T11:32:47.865924+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:42.190121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193940+0000","last_change":"2024-03-29T11:32:47.869332+0000","last_active":"2024-03-29T11:32:49.193940+0000","last_peered":"2024-03-29T11:32:49.193940+0000","last_clean":"2024-03-29T11:32:49.193940+0000","last_became_active":"2024-03-29T11:32:47.868665+0000","last_became_peered":"2024-03-29T11:32:47.868665+0000","last_unstale":"2024-03-29T11:32:49.193940+0000","last_undegraded":"2024-03-29T11:32:49.193940+0000","last_fullsized":"2024-03-29T11:32:49.193940+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:37.965088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338792+0000","last_change":"2024-03-29T11:32:47.340906+0000","last_active":"2024-03-29T11:32:49.338792+0000","last_peered":"2024-03-29T11:32:49.338792+0000","last_clean":"2024-03-29T11:32:49.338792+0000","last_became_active":"2024-03-29T11:32:47.340587+0000","last_became_peered":"2024-03-29T11:32:47.340587+0000","last_unstale":"2024-03-29T11:32:49.338792+0000","last_undegraded":"2024-03-29T11:32:49.338792+0000","last_fullsized":"2024-03-29T11:32:49.338792+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:34:59.271199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861395+0000","last_change":"2024-03-29T11:32:32.950908+0000","last_active":"2024-03-29T11:32:47.861395+0000","last_peered":"2024-03-29T11:32:47.861395+0000","last_clean":"2024-03-29T11:32:47.861395+0000","last_became_active":"2024-03-29T11:32:32.950581+0000","last_became_peered":"2024-03-29T11:32:32.950581+0000","last_unstale":"2024-03-29T11:32:47.861395+0000","last_undegraded":"2024-03-29T11:32:47.861395+0000","last_fullsized":"2024-03-29T11:32:47.861395+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:11.495727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193750+0000","last_change":"2024-03-29T11:32:47.869312+0000","last_active":"2024-03-29T11:32:49.193750+0000","last_peered":"2024-03-29T11:32:49.193750+0000","last_clean":"2024-03-29T11:32:49.193750+0000","last_became_active":"2024-03-29T11:32:47.868448+0000","last_became_peered":"2024-03-29T11:32:47.868448+0000","last_unstale":"2024-03-29T11:32:49.193750+0000","last_undegraded":"2024-03-29T11:32:49.193750+0000","last_fullsized":"2024-03-29T11:32:49.193750+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:25:04.792440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867909+0000","last_change":"2024-03-29T11:32:47.341720+0000","last_active":"2024-03-29T11:32:48.867909+0000","last_peered":"2024-03-29T11:32:48.867909+0000","last_clean":"2024-03-29T11:32:48.867909+0000","last_became_active":"2024-03-29T11:32:47.341458+0000","last_became_peered":"2024-03-29T11:32:47.341458+0000","last_unstale":"2024-03-29T11:32:48.867909+0000","last_undegraded":"2024-03-29T11:32:48.867909+0000","last_fullsized":"2024-03-29T11:32:48.867909+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:36.518778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.863047+0000","last_change":"2024-03-29T11:32:32.966829+0000","last_active":"2024-03-29T11:32:47.863047+0000","last_peered":"2024-03-29T11:32:47.863047+0000","last_clean":"2024-03-29T11:32:47.863047+0000","last_became_active":"2024-03-29T11:32:32.966561+0000","last_became_peered":"2024-03-29T11:32:32.966561+0000","last_unstale":"2024-03-29T11:32:47.863047+0000","last_undegraded":"2024-03-29T11:32:47.863047+0000","last_fullsized":"2024-03-29T11:32:47.863047+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.339167+0000","last_change":"2024-03-29T11:32:47.870380+0000","last_active":"2024-03-29T11:32:49.339167+0000","last_peered":"2024-03-29T11:32:49.339167+0000","last_clean":"2024-03-29T11:32:49.339167+0000","last_became_active":"2024-03-29T11:32:47.869954+0000","last_became_peered":"2024-03-29T11:32:47.869954+0000","last_unstale":"2024-03-29T11:32:49.339167+0000","last_undegraded":"2024-03-29T11:32:49.339167+0000","last_fullsized":"2024-03-29T11:32:49.339167+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:52:34.148222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193942+0000","last_change":"2024-03-29T11:32:47.337485+0000","last_active":"2024-03-29T11:32:49.193942+0000","last_peered":"2024-03-29T11:32:49.193942+0000","last_clean":"2024-03-29T11:32:49.193942+0000","last_became_active":"2024-03-29T11:32:47.336790+0000","last_became_peered":"2024-03-29T11:32:47.336790+0000","last_unstale":"2024-03-29T11:32:49.193942+0000","last_undegraded":"2024-03-29T11:32:49.193942+0000","last_fullsized":"2024-03-29T11:32:49.193942+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:41.434503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862285+0000","last_change":"2024-03-29T11:32:32.956044+0000","last_active":"2024-03-29T11:32:47.862285+0000","last_peered":"2024-03-29T11:32:47.862285+0000","last_clean":"2024-03-29T11:32:47.862285+0000","last_became_active":"2024-03-29T11:32:32.955298+0000","last_became_peered":"2024-03-29T11:32:32.955298+0000","last_unstale":"2024-03-29T11:32:47.862285+0000","last_undegraded":"2024-03-29T11:32:47.862285+0000","last_fullsized":"2024-03-29T11:32:47.862285+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:36.536991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862599+0000","last_change":"2024-03-29T11:32:47.880947+0000","last_active":"2024-03-29T11:32:48.862599+0000","last_peered":"2024-03-29T11:32:48.862599+0000","last_clean":"2024-03-29T11:32:48.862599+0000","last_became_active":"2024-03-29T11:32:47.879746+0000","last_became_peered":"2024-03-29T11:32:47.879746+0000","last_unstale":"2024-03-29T11:32:48.862599+0000","last_undegraded":"2024-03-29T11:32:48.862599+0000","last_fullsized":"2024-03-29T11:32:48.862599+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:01.324372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862895+0000","last_change":"2024-03-29T11:32:47.350151+0000","last_active":"2024-03-29T11:32:48.862895+0000","last_peered":"2024-03-29T11:32:48.862895+0000","last_clean":"2024-03-29T11:32:48.862895+0000","last_became_active":"2024-03-29T11:32:47.349871+0000","last_became_peered":"2024-03-29T11:32:47.349871+0000","last_unstale":"2024-03-29T11:32:48.862895+0000","last_undegraded":"2024-03-29T11:32:48.862895+0000","last_fullsized":"2024-03-29T11:32:48.862895+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:53.214603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.337071+0000","last_change":"2024-03-29T11:32:32.956054+0000","last_active":"2024-03-29T11:32:47.337071+0000","last_peered":"2024-03-29T11:32:47.337071+0000","last_clean":"2024-03-29T11:32:47.337071+0000","last_became_active":"2024-03-29T11:32:32.952773+0000","last_became_peered":"2024-03-29T11:32:32.952773+0000","last_unstale":"2024-03-29T11:32:47.337071+0000","last_undegraded":"2024-03-29T11:32:47.337071+0000","last_fullsized":"2024-03-29T11:32:47.337071+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:59:43.666251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.870319+0000","last_change":"2024-03-29T11:32:47.866242+0000","last_active":"2024-03-29T11:32:47.870319+0000","last_peered":"2024-03-29T11:32:47.870319+0000","last_clean":"2024-03-29T11:32:47.870319+0000","last_became_active":"2024-03-29T11:32:47.865604+0000","last_became_peered":"2024-03-29T11:32:47.865604+0000","last_unstale":"2024-03-29T11:32:47.870319+0000","last_undegraded":"2024-03-29T11:32:47.870319+0000","last_fullsized":"2024-03-29T11:32:47.870319+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:16.654138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861122+0000","last_change":"2024-03-29T11:32:47.340544+0000","last_active":"2024-03-29T11:32:47.861122+0000","last_peered":"2024-03-29T11:32:47.861122+0000","last_clean":"2024-03-29T11:32:47.861122+0000","last_became_active":"2024-03-29T11:32:47.340286+0000","last_became_peered":"2024-03-29T11:32:47.340286+0000","last_unstale":"2024-03-29T11:32:47.861122+0000","last_undegraded":"2024-03-29T11:32:47.861122+0000","last_fullsized":"2024-03-29T11:32:47.861122+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:56.832095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.879554+0000","last_change":"2024-03-29T11:32:32.963463+0000","last_active":"2024-03-29T11:32:47.879554+0000","last_peered":"2024-03-29T11:32:47.879554+0000","last_clean":"2024-03-29T11:32:47.879554+0000","last_became_active":"2024-03-29T11:32:32.962970+0000","last_became_peered":"2024-03-29T11:32:32.962970+0000","last_unstale":"2024-03-29T11:32:47.879554+0000","last_undegraded":"2024-03-29T11:32:47.879554+0000","last_fullsized":"2024-03-29T11:32:47.879554+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:14.637128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338023+0000","last_change":"2024-03-29T11:32:47.870166+0000","last_active":"2024-03-29T11:32:49.338023+0000","last_peered":"2024-03-29T11:32:49.338023+0000","last_clean":"2024-03-29T11:32:49.338023+0000","last_became_active":"2024-03-29T11:32:47.869868+0000","last_became_peered":"2024-03-29T11:32:47.869868+0000","last_unstale":"2024-03-29T11:32:49.338023+0000","last_undegraded":"2024-03-29T11:32:49.338023+0000","last_fullsized":"2024-03-29T11:32:49.338023+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:18:25.353859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868745+0000","last_change":"2024-03-29T11:32:47.347326+0000","last_active":"2024-03-29T11:32:48.868745+0000","last_peered":"2024-03-29T11:32:48.868745+0000","last_clean":"2024-03-29T11:32:48.868745+0000","last_became_active":"2024-03-29T11:32:47.347007+0000","last_became_peered":"2024-03-29T11:32:47.347007+0000","last_unstale":"2024-03-29T11:32:48.868745+0000","last_undegraded":"2024-03-29T11:32:48.868745+0000","last_fullsized":"2024-03-29T11:32:48.868745+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:00.321378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.871023+0000","last_change":"2024-03-29T11:32:32.953703+0000","last_active":"2024-03-29T11:32:47.871023+0000","last_peered":"2024-03-29T11:32:47.871023+0000","last_clean":"2024-03-29T11:32:47.871023+0000","last_became_active":"2024-03-29T11:32:32.952942+0000","last_became_peered":"2024-03-29T11:32:32.952942+0000","last_unstale":"2024-03-29T11:32:47.871023+0000","last_undegraded":"2024-03-29T11:32:47.871023+0000","last_fullsized":"2024-03-29T11:32:47.871023+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:23:44.759018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218814+0000","last_change":"2024-03-29T11:32:47.873899+0000","last_active":"2024-03-29T11:32:49.218814+0000","last_peered":"2024-03-29T11:32:49.218814+0000","last_clean":"2024-03-29T11:32:49.218814+0000","last_became_active":"2024-03-29T11:32:47.873620+0000","last_became_peered":"2024-03-29T11:32:47.873620+0000","last_unstale":"2024-03-29T11:32:49.218814+0000","last_undegraded":"2024-03-29T11:32:49.218814+0000","last_fullsized":"2024-03-29T11:32:49.218814+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:28.748316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.859706+0000","last_change":"2024-03-29T11:32:47.342809+0000","last_active":"2024-03-29T11:32:47.859706+0000","last_peered":"2024-03-29T11:32:47.859706+0000","last_clean":"2024-03-29T11:32:47.859706+0000","last_became_active":"2024-03-29T11:32:47.341749+0000","last_became_peered":"2024-03-29T11:32:47.341749+0000","last_unstale":"2024-03-29T11:32:47.859706+0000","last_undegraded":"2024-03-29T11:32:47.859706+0000","last_fullsized":"2024-03-29T11:32:47.859706+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:36.962093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.346070+0000","last_change":"2024-03-29T11:32:32.970743+0000","last_active":"2024-03-29T11:32:47.346070+0000","last_peered":"2024-03-29T11:32:47.346070+0000","last_clean":"2024-03-29T11:32:47.346070+0000","last_became_active":"2024-03-29T11:32:32.967319+0000","last_became_peered":"2024-03-29T11:32:32.967319+0000","last_unstale":"2024-03-29T11:32:47.346070+0000","last_undegraded":"2024-03-29T11:32:47.346070+0000","last_fullsized":"2024-03-29T11:32:47.346070+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:52.668591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218485+0000","last_change":"2024-03-29T11:32:47.863502+0000","last_active":"2024-03-29T11:32:49.218485+0000","last_peered":"2024-03-29T11:32:49.218485+0000","last_clean":"2024-03-29T11:32:49.218485+0000","last_became_active":"2024-03-29T11:32:47.862432+0000","last_became_peered":"2024-03-29T11:32:47.862432+0000","last_unstale":"2024-03-29T11:32:49.218485+0000","last_undegraded":"2024-03-29T11:32:49.218485+0000","last_fullsized":"2024-03-29T11:32:49.218485+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:19:11.800767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193591+0000","last_change":"2024-03-29T11:32:47.339115+0000","last_active":"2024-03-29T11:32:49.193591+0000","last_peered":"2024-03-29T11:32:49.193591+0000","last_clean":"2024-03-29T11:32:49.193591+0000","last_became_active":"2024-03-29T11:32:47.338823+0000","last_became_peered":"2024-03-29T11:32:47.338823+0000","last_unstale":"2024-03-29T11:32:49.193591+0000","last_undegraded":"2024-03-29T11:32:49.193591+0000","last_fullsized":"2024-03-29T11:32:49.193591+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:58.495188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.339142+0000","last_change":"2024-03-29T11:32:32.957788+0000","last_active":"2024-03-29T11:32:47.339142+0000","last_peered":"2024-03-29T11:32:47.339142+0000","last_clean":"2024-03-29T11:32:47.339142+0000","last_became_active":"2024-03-29T11:32:32.957519+0000","last_became_peered":"2024-03-29T11:32:32.957519+0000","last_unstale":"2024-03-29T11:32:47.339142+0000","last_undegraded":"2024-03-29T11:32:47.339142+0000","last_fullsized":"2024-03-29T11:32:47.339142+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:21.664133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.194043+0000","last_change":"2024-03-29T11:32:47.871212+0000","last_active":"2024-03-29T11:32:49.194043+0000","last_peered":"2024-03-29T11:32:49.194043+0000","last_clean":"2024-03-29T11:32:49.194043+0000","last_became_active":"2024-03-29T11:32:47.869237+0000","last_became_peered":"2024-03-29T11:32:47.869237+0000","last_unstale":"2024-03-29T11:32:49.194043+0000","last_undegraded":"2024-03-29T11:32:49.194043+0000","last_fullsized":"2024-03-29T11:32:49.194043+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:49.665750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862336+0000","last_change":"2024-03-29T11:32:47.352473+0000","last_active":"2024-03-29T11:32:48.862336+0000","last_peered":"2024-03-29T11:32:48.862336+0000","last_clean":"2024-03-29T11:32:48.862336+0000","last_became_active":"2024-03-29T11:32:47.347349+0000","last_became_peered":"2024-03-29T11:32:47.347349+0000","last_unstale":"2024-03-29T11:32:48.862336+0000","last_undegraded":"2024-03-29T11:32:48.862336+0000","last_fullsized":"2024-03-29T11:32:48.862336+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:28:45.896358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.867210+0000","last_change":"2024-03-29T11:32:32.967536+0000","last_active":"2024-03-29T11:32:47.867210+0000","last_peered":"2024-03-29T11:32:47.867210+0000","last_clean":"2024-03-29T11:32:47.867210+0000","last_became_active":"2024-03-29T11:32:32.966686+0000","last_became_peered":"2024-03-29T11:32:32.966686+0000","last_unstale":"2024-03-29T11:32:47.867210+0000","last_undegraded":"2024-03-29T11:32:47.867210+0000","last_fullsized":"2024-03-29T11:32:47.867210+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:24.736585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862432+0000","last_change":"2024-03-29T11:32:47.873240+0000","last_active":"2024-03-29T11:32:48.862432+0000","last_peered":"2024-03-29T11:32:48.862432+0000","last_clean":"2024-03-29T11:32:48.862432+0000","last_became_active":"2024-03-29T11:32:47.871973+0000","last_became_peered":"2024-03-29T11:32:47.871973+0000","last_unstale":"2024-03-29T11:32:48.862432+0000","last_undegraded":"2024-03-29T11:32:48.862432+0000","last_fullsized":"2024-03-29T11:32:48.862432+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:52:10.675709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868306+0000","last_change":"2024-03-29T11:32:47.347470+0000","last_active":"2024-03-29T11:32:48.868306+0000","last_peered":"2024-03-29T11:32:48.868306+0000","last_clean":"2024-03-29T11:32:48.868306+0000","last_became_active":"2024-03-29T11:32:47.347097+0000","last_became_peered":"2024-03-29T11:32:47.347097+0000","last_unstale":"2024-03-29T11:32:48.868306+0000","last_undegraded":"2024-03-29T11:32:48.868306+0000","last_fullsized":"2024-03-29T11:32:48.868306+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:15.651144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854388+0000","last_change":"2024-03-29T11:32:32.952674+0000","last_active":"2024-03-29T11:32:47.854388+0000","last_peered":"2024-03-29T11:32:47.854388+0000","last_clean":"2024-03-29T11:32:47.854388+0000","last_became_active":"2024-03-29T11:32:32.952387+0000","last_became_peered":"2024-03-29T11:32:32.952387+0000","last_unstale":"2024-03-29T11:32:47.854388+0000","last_undegraded":"2024-03-29T11:32:47.854388+0000","last_fullsized":"2024-03-29T11:32:47.854388+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:30.121534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868873+0000","last_change":"2024-03-29T11:32:47.888074+0000","last_active":"2024-03-29T11:32:48.868873+0000","last_peered":"2024-03-29T11:32:48.868873+0000","last_clean":"2024-03-29T11:32:48.868873+0000","last_became_active":"2024-03-29T11:32:47.887753+0000","last_became_peered":"2024-03-29T11:32:47.887753+0000","last_unstale":"2024-03-29T11:32:48.868873+0000","last_undegraded":"2024-03-29T11:32:48.868873+0000","last_fullsized":"2024-03-29T11:32:48.868873+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:16:25.816476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.219106+0000","last_change":"2024-03-29T11:32:47.348457+0000","last_active":"2024-03-29T11:32:49.219106+0000","last_peered":"2024-03-29T11:32:49.219106+0000","last_clean":"2024-03-29T11:32:49.219106+0000","last_became_active":"2024-03-29T11:32:47.348162+0000","last_became_peered":"2024-03-29T11:32:47.348162+0000","last_unstale":"2024-03-29T11:32:49.219106+0000","last_undegraded":"2024-03-29T11:32:49.219106+0000","last_fullsized":"2024-03-29T11:32:49.219106+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:41.984195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.878359+0000","last_change":"2024-03-29T11:32:32.966028+0000","last_active":"2024-03-29T11:32:47.878359+0000","last_peered":"2024-03-29T11:32:47.878359+0000","last_clean":"2024-03-29T11:32:47.878359+0000","last_became_active":"2024-03-29T11:32:32.965642+0000","last_became_peered":"2024-03-29T11:32:32.965642+0000","last_unstale":"2024-03-29T11:32:47.878359+0000","last_undegraded":"2024-03-29T11:32:47.878359+0000","last_fullsized":"2024-03-29T11:32:47.878359+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:28:31.977713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862565+0000","last_change":"2024-03-29T11:32:47.887949+0000","last_active":"2024-03-29T11:32:48.862565+0000","last_peered":"2024-03-29T11:32:48.862565+0000","last_clean":"2024-03-29T11:32:48.862565+0000","last_became_active":"2024-03-29T11:32:47.887377+0000","last_became_peered":"2024-03-29T11:32:47.887377+0000","last_unstale":"2024-03-29T11:32:48.862565+0000","last_undegraded":"2024-03-29T11:32:48.862565+0000","last_fullsized":"2024-03-29T11:32:48.862565+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:15.233472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"251'1","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.880873+0000","last_change":"2024-03-29T11:32:47.345394+0000","last_active":"2024-03-29T11:32:47.880873+0000","last_peered":"2024-03-29T11:32:47.880873+0000","last_clean":"2024-03-29T11:32:47.880873+0000","last_became_active":"2024-03-29T11:32:47.345100+0000","last_became_peered":"2024-03-29T11:32:47.345100+0000","last_unstale":"2024-03-29T11:32:47.880873+0000","last_undegraded":"2024-03-29T11:32:47.880873+0000","last_fullsized":"2024-03-29T11:32:47.880873+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:49.746156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.860050+0000","last_change":"2024-03-29T11:32:32.960337+0000","last_active":"2024-03-29T11:32:47.860050+0000","last_peered":"2024-03-29T11:32:47.860050+0000","last_clean":"2024-03-29T11:32:47.860050+0000","last_became_active":"2024-03-29T11:32:32.958824+0000","last_became_peered":"2024-03-29T11:32:32.958824+0000","last_unstale":"2024-03-29T11:32:47.860050+0000","last_undegraded":"2024-03-29T11:32:47.860050+0000","last_fullsized":"2024-03-29T11:32:47.860050+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:48:40.247306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.337903+0000","last_change":"2024-03-29T11:32:47.874778+0000","last_active":"2024-03-29T11:32:49.337903+0000","last_peered":"2024-03-29T11:32:49.337903+0000","last_clean":"2024-03-29T11:32:49.337903+0000","last_became_active":"2024-03-29T11:32:47.872247+0000","last_became_peered":"2024-03-29T11:32:47.872247+0000","last_unstale":"2024-03-29T11:32:49.337903+0000","last_undegraded":"2024-03-29T11:32:49.337903+0000","last_fullsized":"2024-03-29T11:32:49.337903+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:22:15.580093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"251'8","reported_seq":19,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218395+0000","last_change":"2024-03-29T11:32:47.353083+0000","last_active":"2024-03-29T11:32:49.218395+0000","last_peered":"2024-03-29T11:32:49.218395+0000","last_clean":"2024-03-29T11:32:49.218395+0000","last_became_active":"2024-03-29T11:32:47.352595+0000","last_became_peered":"2024-03-29T11:32:47.352595+0000","last_unstale":"2024-03-29T11:32:49.218395+0000","last_undegraded":"2024-03-29T11:32:49.218395+0000","last_fullsized":"2024-03-29T11:32:49.218395+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:21:18.809408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.341757+0000","last_change":"2024-03-29T11:32:32.960497+0000","last_active":"2024-03-29T11:32:47.341757+0000","last_peered":"2024-03-29T11:32:47.341757+0000","last_clean":"2024-03-29T11:32:47.341757+0000","last_became_active":"2024-03-29T11:32:32.958333+0000","last_became_peered":"2024-03-29T11:32:32.958333+0000","last_unstale":"2024-03-29T11:32:47.341757+0000","last_undegraded":"2024-03-29T11:32:47.341757+0000","last_fullsized":"2024-03-29T11:32:47.341757+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:27:28.513987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.337900+0000","last_change":"2024-03-29T11:32:47.882397+0000","last_active":"2024-03-29T11:32:49.337900+0000","last_peered":"2024-03-29T11:32:49.337900+0000","last_clean":"2024-03-29T11:32:49.337900+0000","last_became_active":"2024-03-29T11:32:47.881886+0000","last_became_peered":"2024-03-29T11:32:47.881886+0000","last_unstale":"2024-03-29T11:32:49.337900+0000","last_undegraded":"2024-03-29T11:32:49.337900+0000","last_fullsized":"2024-03-29T11:32:49.337900+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:21:15.130504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868839+0000","last_change":"2024-03-29T11:32:47.348169+0000","last_active":"2024-03-29T11:32:48.868839+0000","last_peered":"2024-03-29T11:32:48.868839+0000","last_clean":"2024-03-29T11:32:48.868839+0000","last_became_active":"2024-03-29T11:32:47.347899+0000","last_became_peered":"2024-03-29T11:32:47.347899+0000","last_unstale":"2024-03-29T11:32:48.868839+0000","last_undegraded":"2024-03-29T11:32:48.868839+0000","last_fullsized":"2024-03-29T11:32:48.868839+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:27.533294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854490+0000","last_change":"2024-03-29T11:32:32.949294+0000","last_active":"2024-03-29T11:32:47.854490+0000","last_peered":"2024-03-29T11:32:47.854490+0000","last_clean":"2024-03-29T11:32:47.854490+0000","last_became_active":"2024-03-29T11:32:32.949025+0000","last_became_peered":"2024-03-29T11:32:32.949025+0000","last_unstale":"2024-03-29T11:32:47.854490+0000","last_undegraded":"2024-03-29T11:32:47.854490+0000","last_fullsized":"2024-03-29T11:32:47.854490+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218794+0000","last_change":"2024-03-29T11:32:47.873323+0000","last_active":"2024-03-29T11:32:49.218794+0000","last_peered":"2024-03-29T11:32:49.218794+0000","last_clean":"2024-03-29T11:32:49.218794+0000","last_became_active":"2024-03-29T11:32:47.873061+0000","last_became_peered":"2024-03-29T11:32:47.873061+0000","last_unstale":"2024-03-29T11:32:49.218794+0000","last_undegraded":"2024-03-29T11:32:49.218794+0000","last_fullsized":"2024-03-29T11:32:49.218794+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:15.233472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":10,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.858998+0000","last_change":"2024-03-29T11:32:47.352021+0000","last_active":"2024-03-29T11:32:47.858998+0000","last_peered":"2024-03-29T11:32:47.858998+0000","last_clean":"2024-03-29T11:32:47.858998+0000","last_became_active":"2024-03-29T11:32:47.351478+0000","last_became_peered":"2024-03-29T11:32:47.351478+0000","last_unstale":"2024-03-29T11:32:47.858998+0000","last_undegraded":"2024-03-29T11:32:47.858998+0000","last_fullsized":"2024-03-29T11:32:47.858998+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:44.616340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.341112+0000","last_change":"2024-03-29T11:32:32.965153+0000","last_active":"2024-03-29T11:32:47.341112+0000","last_peered":"2024-03-29T11:32:47.341112+0000","last_clean":"2024-03-29T11:32:47.341112+0000","last_became_active":"2024-03-29T11:32:32.964935+0000","last_became_peered":"2024-03-29T11:32:32.964935+0000","last_unstale":"2024-03-29T11:32:47.341112+0000","last_undegraded":"2024-03-29T11:32:47.341112+0000","last_fullsized":"2024-03-29T11:32:47.341112+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:02.769642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338031+0000","last_change":"2024-03-29T11:32:47.880871+0000","last_active":"2024-03-29T11:32:49.338031+0000","last_peered":"2024-03-29T11:32:49.338031+0000","last_clean":"2024-03-29T11:32:49.338031+0000","last_became_active":"2024-03-29T11:32:47.880621+0000","last_became_peered":"2024-03-29T11:32:47.880621+0000","last_unstale":"2024-03-29T11:32:49.338031+0000","last_undegraded":"2024-03-29T11:32:49.338031+0000","last_fullsized":"2024-03-29T11:32:49.338031+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:24:56.799955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"251'2","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.885870+0000","last_change":"2024-03-29T11:32:47.352084+0000","last_active":"2024-03-29T11:32:47.885870+0000","last_peered":"2024-03-29T11:32:47.885870+0000","last_clean":"2024-03-29T11:32:47.885870+0000","last_became_active":"2024-03-29T11:32:47.351764+0000","last_became_peered":"2024-03-29T11:32:47.351764+0000","last_unstale":"2024-03-29T11:32:47.885870+0000","last_undegraded":"2024-03-29T11:32:47.885870+0000","last_fullsized":"2024-03-29T11:32:47.885870+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:47.734117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.866498+0000","last_change":"2024-03-29T11:32:32.953083+0000","last_active":"2024-03-29T11:32:47.866498+0000","last_peered":"2024-03-29T11:32:47.866498+0000","last_clean":"2024-03-29T11:32:47.866498+0000","last_became_active":"2024-03-29T11:32:32.952823+0000","last_became_peered":"2024-03-29T11:32:32.952823+0000","last_unstale":"2024-03-29T11:32:47.866498+0000","last_undegraded":"2024-03-29T11:32:47.866498+0000","last_fullsized":"2024-03-29T11:32:47.866498+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:09.626524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338902+0000","last_change":"2024-03-29T11:32:47.875386+0000","last_active":"2024-03-29T11:32:49.338902+0000","last_peered":"2024-03-29T11:32:49.338902+0000","last_clean":"2024-03-29T11:32:49.338902+0000","last_became_active":"2024-03-29T11:32:47.875130+0000","last_became_peered":"2024-03-29T11:32:47.875130+0000","last_unstale":"2024-03-29T11:32:49.338902+0000","last_undegraded":"2024-03-29T11:32:49.338902+0000","last_fullsized":"2024-03-29T11:32:49.338902+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:06.010602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"251'6","reported_seq":17,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338195+0000","last_change":"2024-03-29T11:32:47.344656+0000","last_active":"2024-03-29T11:32:49.338195+0000","last_peered":"2024-03-29T11:32:49.338195+0000","last_clean":"2024-03-29T11:32:49.338195+0000","last_became_active":"2024-03-29T11:32:47.344357+0000","last_became_peered":"2024-03-29T11:32:47.344357+0000","last_unstale":"2024-03-29T11:32:49.338195+0000","last_undegraded":"2024-03-29T11:32:49.338195+0000","last_fullsized":"2024-03-29T11:32:49.338195+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:36:44.081864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.339447+0000","last_change":"2024-03-29T11:32:32.954755+0000","last_active":"2024-03-29T11:32:47.339447+0000","last_peered":"2024-03-29T11:32:47.339447+0000","last_clean":"2024-03-29T11:32:47.339447+0000","last_became_active":"2024-03-29T11:32:32.954467+0000","last_became_peered":"2024-03-29T11:32:32.954467+0000","last_unstale":"2024-03-29T11:32:47.339447+0000","last_undegraded":"2024-03-29T11:32:47.339447+0000","last_fullsized":"2024-03-29T11:32:47.339447+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:10.303092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862462+0000","last_change":"2024-03-29T11:32:47.881806+0000","last_active":"2024-03-29T11:32:48.862462+0000","last_peered":"2024-03-29T11:32:48.862462+0000","last_clean":"2024-03-29T11:32:48.862462+0000","last_became_active":"2024-03-29T11:32:47.872495+0000","last_became_peered":"2024-03-29T11:32:47.872495+0000","last_unstale":"2024-03-29T11:32:48.862462+0000","last_undegraded":"2024-03-29T11:32:48.862462+0000","last_fullsized":"2024-03-29T11:32:48.862462+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:16:25.816476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"251'3","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.219036+0000","last_change":"2024-03-29T11:32:47.347470+0000","last_active":"2024-03-29T11:32:49.219036+0000","last_peered":"2024-03-29T11:32:49.219036+0000","last_clean":"2024-03-29T11:32:49.219036+0000","last_became_active":"2024-03-29T11:32:47.347198+0000","last_became_peered":"2024-03-29T11:32:47.347198+0000","last_unstale":"2024-03-29T11:32:49.219036+0000","last_undegraded":"2024-03-29T11:32:49.219036+0000","last_fullsized":"2024-03-29T11:32:49.219036+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:19.718112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861971+0000","last_change":"2024-03-29T11:32:32.955216+0000","last_active":"2024-03-29T11:32:47.861971+0000","last_peered":"2024-03-29T11:32:47.861971+0000","last_clean":"2024-03-29T11:32:47.861971+0000","last_became_active":"2024-03-29T11:32:32.954774+0000","last_became_peered":"2024-03-29T11:32:32.954774+0000","last_unstale":"2024-03-29T11:32:47.861971+0000","last_undegraded":"2024-03-29T11:32:47.861971+0000","last_fullsized":"2024-03-29T11:32:47.861971+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:44.576542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338992+0000","last_change":"2024-03-29T11:32:47.874012+0000","last_active":"2024-03-29T11:32:49.338992+0000","last_peered":"2024-03-29T11:32:49.338992+0000","last_clean":"2024-03-29T11:32:49.338992+0000","last_became_active":"2024-03-29T11:32:47.873755+0000","last_became_peered":"2024-03-29T11:32:47.873755+0000","last_unstale":"2024-03-29T11:32:49.338992+0000","last_undegraded":"2024-03-29T11:32:49.338992+0000","last_fullsized":"2024-03-29T11:32:49.338992+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:40.369123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.219123+0000","last_change":"2024-03-29T11:32:47.347700+0000","last_active":"2024-03-29T11:32:49.219123+0000","last_peered":"2024-03-29T11:32:49.219123+0000","last_clean":"2024-03-29T11:32:49.219123+0000","last_became_active":"2024-03-29T11:32:47.347431+0000","last_became_peered":"2024-03-29T11:32:47.347431+0000","last_unstale":"2024-03-29T11:32:49.219123+0000","last_undegraded":"2024-03-29T11:32:49.219123+0000","last_fullsized":"2024-03-29T11:32:49.219123+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:23.545169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.338716+0000","last_change":"2024-03-29T11:32:32.964085+0000","last_active":"2024-03-29T11:32:47.338716+0000","last_peered":"2024-03-29T11:32:47.338716+0000","last_clean":"2024-03-29T11:32:47.338716+0000","last_became_active":"2024-03-29T11:32:32.963753+0000","last_became_peered":"2024-03-29T11:32:32.963753+0000","last_unstale":"2024-03-29T11:32:47.338716+0000","last_undegraded":"2024-03-29T11:32:47.338716+0000","last_fullsized":"2024-03-29T11:32:47.338716+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:34.956119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867875+0000","last_change":"2024-03-29T11:32:47.874694+0000","last_active":"2024-03-29T11:32:48.867875+0000","last_peered":"2024-03-29T11:32:48.867875+0000","last_clean":"2024-03-29T11:32:48.867875+0000","last_became_active":"2024-03-29T11:32:47.874365+0000","last_became_peered":"2024-03-29T11:32:47.874365+0000","last_unstale":"2024-03-29T11:32:48.867875+0000","last_undegraded":"2024-03-29T11:32:48.867875+0000","last_fullsized":"2024-03-29T11:32:48.867875+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:30.885593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.339110+0000","last_change":"2024-03-29T11:32:47.333902+0000","last_active":"2024-03-29T11:32:49.339110+0000","last_peered":"2024-03-29T11:32:49.339110+0000","last_clean":"2024-03-29T11:32:49.339110+0000","last_became_active":"2024-03-29T11:32:47.333598+0000","last_became_peered":"2024-03-29T11:32:47.333598+0000","last_unstale":"2024-03-29T11:32:49.339110+0000","last_undegraded":"2024-03-29T11:32:49.339110+0000","last_fullsized":"2024-03-29T11:32:49.339110+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:37:48.411120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854010+0000","last_change":"2024-03-29T11:32:32.957520+0000","last_active":"2024-03-29T11:32:47.854010+0000","last_peered":"2024-03-29T11:32:47.854010+0000","last_clean":"2024-03-29T11:32:47.854010+0000","last_became_active":"2024-03-29T11:32:32.957245+0000","last_became_peered":"2024-03-29T11:32:32.957245+0000","last_unstale":"2024-03-29T11:32:47.854010+0000","last_undegraded":"2024-03-29T11:32:47.854010+0000","last_fullsized":"2024-03-29T11:32:47.854010+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:45.368395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.866798+0000","last_change":"2024-03-29T11:32:47.873501+0000","last_active":"2024-03-29T11:32:48.866798+0000","last_peered":"2024-03-29T11:32:48.866798+0000","last_clean":"2024-03-29T11:32:48.866798+0000","last_became_active":"2024-03-29T11:32:47.870502+0000","last_became_peered":"2024-03-29T11:32:47.870502+0000","last_unstale":"2024-03-29T11:32:48.866798+0000","last_undegraded":"2024-03-29T11:32:48.866798+0000","last_fullsized":"2024-03-29T11:32:48.866798+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:59:56.436348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.866887+0000","last_change":"2024-03-29T11:32:47.346085+0000","last_active":"2024-03-29T11:32:48.866887+0000","last_peered":"2024-03-29T11:32:48.866887+0000","last_clean":"2024-03-29T11:32:48.866887+0000","last_became_active":"2024-03-29T11:32:47.345724+0000","last_became_peered":"2024-03-29T11:32:47.345724+0000","last_unstale":"2024-03-29T11:32:48.866887+0000","last_undegraded":"2024-03-29T11:32:48.866887+0000","last_fullsized":"2024-03-29T11:32:48.866887+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:48:54.165952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.866234+0000","last_change":"2024-03-29T11:32:32.955473+0000","last_active":"2024-03-29T11:32:47.866234+0000","last_peered":"2024-03-29T11:32:47.866234+0000","last_clean":"2024-03-29T11:32:47.866234+0000","last_became_active":"2024-03-29T11:32:32.953854+0000","last_became_peered":"2024-03-29T11:32:32.953854+0000","last_unstale":"2024-03-29T11:32:47.866234+0000","last_undegraded":"2024-03-29T11:32:47.866234+0000","last_fullsized":"2024-03-29T11:32:47.866234+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:21.819962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218281+0000","last_change":"2024-03-29T11:32:47.868864+0000","last_active":"2024-03-29T11:32:49.218281+0000","last_peered":"2024-03-29T11:32:49.218281+0000","last_clean":"2024-03-29T11:32:49.218281+0000","last_became_active":"2024-03-29T11:32:47.868572+0000","last_became_peered":"2024-03-29T11:32:47.868572+0000","last_unstale":"2024-03-29T11:32:49.218281+0000","last_undegraded":"2024-03-29T11:32:49.218281+0000","last_fullsized":"2024-03-29T11:32:49.218281+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:01.324372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193400+0000","last_change":"2024-03-29T11:32:47.337570+0000","last_active":"2024-03-29T11:32:49.193400+0000","last_peered":"2024-03-29T11:32:49.193400+0000","last_clean":"2024-03-29T11:32:49.193400+0000","last_became_active":"2024-03-29T11:32:47.337288+0000","last_became_peered":"2024-03-29T11:32:47.337288+0000","last_unstale":"2024-03-29T11:32:49.193400+0000","last_undegraded":"2024-03-29T11:32:49.193400+0000","last_fullsized":"2024-03-29T11:32:49.193400+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:22:19.861976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.870940+0000","last_change":"2024-03-29T11:32:32.967040+0000","last_active":"2024-03-29T11:32:47.870940+0000","last_peered":"2024-03-29T11:32:47.870940+0000","last_clean":"2024-03-29T11:32:47.870940+0000","last_became_active":"2024-03-29T11:32:32.966729+0000","last_became_peered":"2024-03-29T11:32:32.966729+0000","last_unstale":"2024-03-29T11:32:47.870940+0000","last_undegraded":"2024-03-29T11:32:47.870940+0000","last_fullsized":"2024-03-29T11:32:47.870940+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:27.515858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218710+0000","last_change":"2024-03-29T11:32:47.885165+0000","last_active":"2024-03-29T11:32:49.218710+0000","last_peered":"2024-03-29T11:32:49.218710+0000","last_clean":"2024-03-29T11:32:49.218710+0000","last_became_active":"2024-03-29T11:32:47.884897+0000","last_became_peered":"2024-03-29T11:32:47.884897+0000","last_unstale":"2024-03-29T11:32:49.218710+0000","last_undegraded":"2024-03-29T11:32:49.218710+0000","last_fullsized":"2024-03-29T11:32:49.218710+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:52:10.675709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338304+0000","last_change":"2024-03-29T11:32:47.340767+0000","last_active":"2024-03-29T11:32:49.338304+0000","last_peered":"2024-03-29T11:32:49.338304+0000","last_clean":"2024-03-29T11:32:49.338304+0000","last_became_active":"2024-03-29T11:32:47.339903+0000","last_became_peered":"2024-03-29T11:32:47.339903+0000","last_unstale":"2024-03-29T11:32:49.338304+0000","last_undegraded":"2024-03-29T11:32:49.338304+0000","last_fullsized":"2024-03-29T11:32:49.338304+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:33:56.275080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.346156+0000","last_change":"2024-03-29T11:32:32.970545+0000","last_active":"2024-03-29T11:32:47.346156+0000","last_peered":"2024-03-29T11:32:47.346156+0000","last_clean":"2024-03-29T11:32:47.346156+0000","last_became_active":"2024-03-29T11:32:32.966917+0000","last_became_peered":"2024-03-29T11:32:32.966917+0000","last_unstale":"2024-03-29T11:32:47.346156+0000","last_undegraded":"2024-03-29T11:32:47.346156+0000","last_fullsized":"2024-03-29T11:32:47.346156+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:11.611059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.876653+0000","last_change":"2024-03-29T11:32:47.871148+0000","last_active":"2024-03-29T11:32:47.876653+0000","last_peered":"2024-03-29T11:32:47.876653+0000","last_clean":"2024-03-29T11:32:47.876653+0000","last_became_active":"2024-03-29T11:32:47.870896+0000","last_became_peered":"2024-03-29T11:32:47.870896+0000","last_unstale":"2024-03-29T11:32:47.876653+0000","last_undegraded":"2024-03-29T11:32:47.876653+0000","last_fullsized":"2024-03-29T11:32:47.876653+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:15:14.194005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.866970+0000","last_change":"2024-03-29T11:32:47.343168+0000","last_active":"2024-03-29T11:32:48.866970+0000","last_peered":"2024-03-29T11:32:48.866970+0000","last_clean":"2024-03-29T11:32:48.866970+0000","last_became_active":"2024-03-29T11:32:47.342862+0000","last_became_peered":"2024-03-29T11:32:47.342862+0000","last_unstale":"2024-03-29T11:32:48.866970+0000","last_undegraded":"2024-03-29T11:32:48.866970+0000","last_fullsized":"2024-03-29T11:32:48.866970+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:34:11.151228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.855906+0000","last_change":"2024-03-29T11:32:32.950772+0000","last_active":"2024-03-29T11:32:47.855906+0000","last_peered":"2024-03-29T11:32:47.855906+0000","last_clean":"2024-03-29T11:32:47.855906+0000","last_became_active":"2024-03-29T11:32:32.950518+0000","last_became_peered":"2024-03-29T11:32:32.950518+0000","last_unstale":"2024-03-29T11:32:47.855906+0000","last_undegraded":"2024-03-29T11:32:47.855906+0000","last_fullsized":"2024-03-29T11:32:47.855906+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:34.956119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.879431+0000","last_change":"2024-03-29T11:32:47.878315+0000","last_active":"2024-03-29T11:32:47.879431+0000","last_peered":"2024-03-29T11:32:47.879431+0000","last_clean":"2024-03-29T11:32:47.879431+0000","last_became_active":"2024-03-29T11:32:47.878010+0000","last_became_peered":"2024-03-29T11:32:47.878010+0000","last_unstale":"2024-03-29T11:32:47.879431+0000","last_undegraded":"2024-03-29T11:32:47.879431+0000","last_fullsized":"2024-03-29T11:32:47.879431+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:04.012912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218788+0000","last_change":"2024-03-29T11:32:47.348418+0000","last_active":"2024-03-29T11:32:49.218788+0000","last_peered":"2024-03-29T11:32:49.218788+0000","last_clean":"2024-03-29T11:32:49.218788+0000","last_became_active":"2024-03-29T11:32:47.348122+0000","last_became_peered":"2024-03-29T11:32:47.348122+0000","last_unstale":"2024-03-29T11:32:49.218788+0000","last_undegraded":"2024-03-29T11:32:49.218788+0000","last_fullsized":"2024-03-29T11:32:49.218788+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:13:43.623127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.343181+0000","last_change":"2024-03-29T11:32:32.956094+0000","last_active":"2024-03-29T11:32:47.343181+0000","last_peered":"2024-03-29T11:32:47.343181+0000","last_clean":"2024-03-29T11:32:47.343181+0000","last_became_active":"2024-03-29T11:32:32.955788+0000","last_became_peered":"2024-03-29T11:32:32.955788+0000","last_unstale":"2024-03-29T11:32:47.343181+0000","last_undegraded":"2024-03-29T11:32:47.343181+0000","last_fullsized":"2024-03-29T11:32:47.343181+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:05.005123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867401+0000","last_change":"2024-03-29T11:32:47.870943+0000","last_active":"2024-03-29T11:32:48.867401+0000","last_peered":"2024-03-29T11:32:48.867401+0000","last_clean":"2024-03-29T11:32:48.867401+0000","last_became_active":"2024-03-29T11:32:47.863639+0000","last_became_peered":"2024-03-29T11:32:47.863639+0000","last_unstale":"2024-03-29T11:32:48.867401+0000","last_undegraded":"2024-03-29T11:32:48.867401+0000","last_fullsized":"2024-03-29T11:32:48.867401+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:28.536288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218877+0000","last_change":"2024-03-29T11:32:47.341878+0000","last_active":"2024-03-29T11:32:49.218877+0000","last_peered":"2024-03-29T11:32:49.218877+0000","last_clean":"2024-03-29T11:32:49.218877+0000","last_became_active":"2024-03-29T11:32:47.341594+0000","last_became_peered":"2024-03-29T11:32:47.341594+0000","last_unstale":"2024-03-29T11:32:49.218877+0000","last_undegraded":"2024-03-29T11:32:49.218877+0000","last_fullsized":"2024-03-29T11:32:49.218877+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:03.009918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861574+0000","last_change":"2024-03-29T11:32:32.951190+0000","last_active":"2024-03-29T11:32:47.861574+0000","last_peered":"2024-03-29T11:32:47.861574+0000","last_clean":"2024-03-29T11:32:47.861574+0000","last_became_active":"2024-03-29T11:32:32.950830+0000","last_became_peered":"2024-03-29T11:32:32.950830+0000","last_unstale":"2024-03-29T11:32:47.861574+0000","last_undegraded":"2024-03-29T11:32:47.861574+0000","last_fullsized":"2024-03-29T11:32:47.861574+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:12.191813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862773+0000","last_change":"2024-03-29T11:32:47.867607+0000","last_active":"2024-03-29T11:32:48.862773+0000","last_peered":"2024-03-29T11:32:48.862773+0000","last_clean":"2024-03-29T11:32:48.862773+0000","last_became_active":"2024-03-29T11:32:47.867338+0000","last_became_peered":"2024-03-29T11:32:47.867338+0000","last_unstale":"2024-03-29T11:32:48.862773+0000","last_undegraded":"2024-03-29T11:32:48.862773+0000","last_fullsized":"2024-03-29T11:32:48.862773+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:28.536288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862692+0000","last_change":"2024-03-29T11:32:47.351153+0000","last_active":"2024-03-29T11:32:48.862692+0000","last_peered":"2024-03-29T11:32:48.862692+0000","last_clean":"2024-03-29T11:32:48.862692+0000","last_became_active":"2024-03-29T11:32:47.350841+0000","last_became_peered":"2024-03-29T11:32:47.350841+0000","last_unstale":"2024-03-29T11:32:48.862692+0000","last_undegraded":"2024-03-29T11:32:48.862692+0000","last_fullsized":"2024-03-29T11:32:48.862692+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:28.555774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862635+0000","last_change":"2024-03-29T11:32:32.949171+0000","last_active":"2024-03-29T11:32:47.862635+0000","last_peered":"2024-03-29T11:32:47.862635+0000","last_clean":"2024-03-29T11:32:47.862635+0000","last_became_active":"2024-03-29T11:32:32.948898+0000","last_became_peered":"2024-03-29T11:32:32.948898+0000","last_unstale":"2024-03-29T11:32:47.862635+0000","last_undegraded":"2024-03-29T11:32:47.862635+0000","last_fullsized":"2024-03-29T11:32:47.862635+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:02:36.221843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867700+0000","last_change":"2024-03-29T11:32:47.872107+0000","last_active":"2024-03-29T11:32:48.867700+0000","last_peered":"2024-03-29T11:32:48.867700+0000","last_clean":"2024-03-29T11:32:48.867700+0000","last_became_active":"2024-03-29T11:32:47.871812+0000","last_became_peered":"2024-03-29T11:32:47.871812+0000","last_unstale":"2024-03-29T11:32:48.867700+0000","last_undegraded":"2024-03-29T11:32:48.867700+0000","last_fullsized":"2024-03-29T11:32:48.867700+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:15:14.194005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"251'4","reported_seq":15,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.339066+0000","last_change":"2024-03-29T11:32:47.348086+0000","last_active":"2024-03-29T11:32:49.339066+0000","last_peered":"2024-03-29T11:32:49.339066+0000","last_clean":"2024-03-29T11:32:49.339066+0000","last_became_active":"2024-03-29T11:32:47.347790+0000","last_became_peered":"2024-03-29T11:32:47.347790+0000","last_unstale":"2024-03-29T11:32:49.339066+0000","last_undegraded":"2024-03-29T11:32:49.339066+0000","last_fullsized":"2024-03-29T11:32:49.339066+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:26.775222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854118+0000","last_change":"2024-03-29T11:32:32.956902+0000","last_active":"2024-03-29T11:32:47.854118+0000","last_peered":"2024-03-29T11:32:47.854118+0000","last_clean":"2024-03-29T11:32:47.854118+0000","last_became_active":"2024-03-29T11:32:32.955472+0000","last_became_peered":"2024-03-29T11:32:32.955472+0000","last_unstale":"2024-03-29T11:32:47.854118+0000","last_undegraded":"2024-03-29T11:32:47.854118+0000","last_fullsized":"2024-03-29T11:32:47.854118+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:29.628408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.878650+0000","last_change":"2024-03-29T11:32:47.877511+0000","last_active":"2024-03-29T11:32:47.878650+0000","last_peered":"2024-03-29T11:32:47.878650+0000","last_clean":"2024-03-29T11:32:47.878650+0000","last_became_active":"2024-03-29T11:32:47.877188+0000","last_became_peered":"2024-03-29T11:32:47.877188+0000","last_unstale":"2024-03-29T11:32:47.878650+0000","last_undegraded":"2024-03-29T11:32:47.878650+0000","last_fullsized":"2024-03-29T11:32:47.878650+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:20.721106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.869118+0000","last_change":"2024-03-29T11:32:32.959493+0000","last_active":"2024-03-29T11:32:47.869118+0000","last_peered":"2024-03-29T11:32:47.869118+0000","last_clean":"2024-03-29T11:32:47.869118+0000","last_became_active":"2024-03-29T11:32:32.959199+0000","last_became_peered":"2024-03-29T11:32:32.959199+0000","last_unstale":"2024-03-29T11:32:47.869118+0000","last_undegraded":"2024-03-29T11:32:47.869118+0000","last_fullsized":"2024-03-29T11:32:47.869118+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:56.148657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.870724+0000","last_change":"2024-03-29T11:32:32.419758+0000","last_active":"2024-03-29T11:32:47.870724+0000","last_peered":"2024-03-29T11:32:47.870724+0000","last_clean":"2024-03-29T11:32:47.870724+0000","last_became_active":"2024-03-29T11:32:32.419044+0000","last_became_peered":"2024-03-29T11:32:32.419044+0000","last_unstale":"2024-03-29T11:32:47.870724+0000","last_undegraded":"2024-03-29T11:32:47.870724+0000","last_fullsized":"2024-03-29T11:32:47.870724+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:29.373413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.861959+0000","last_change":"2024-03-29T11:32:47.883962+0000","last_active":"2024-03-29T11:32:48.861959+0000","last_peered":"2024-03-29T11:32:48.861959+0000","last_clean":"2024-03-29T11:32:48.861959+0000","last_became_active":"2024-03-29T11:32:47.873744+0000","last_became_peered":"2024-03-29T11:32:47.873744+0000","last_unstale":"2024-03-29T11:32:48.861959+0000","last_undegraded":"2024-03-29T11:32:48.861959+0000","last_fullsized":"2024-03-29T11:32:48.861959+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:15:14.194005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.345188+0000","last_change":"2024-03-29T11:32:32.964445+0000","last_active":"2024-03-29T11:32:47.345188+0000","last_peered":"2024-03-29T11:32:47.345188+0000","last_clean":"2024-03-29T11:32:47.345188+0000","last_became_active":"2024-03-29T11:32:32.963758+0000","last_became_peered":"2024-03-29T11:32:32.963758+0000","last_unstale":"2024-03-29T11:32:47.345188+0000","last_undegraded":"2024-03-29T11:32:47.345188+0000","last_fullsized":"2024-03-29T11:32:47.345188+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:24.556045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.868333+0000","last_change":"2024-03-29T11:32:32.419690+0000","last_active":"2024-03-29T11:32:47.868333+0000","last_peered":"2024-03-29T11:32:47.868333+0000","last_clean":"2024-03-29T11:32:47.868333+0000","last_became_active":"2024-03-29T11:32:32.418508+0000","last_became_peered":"2024-03-29T11:32:32.418508+0000","last_unstale":"2024-03-29T11:32:47.868333+0000","last_undegraded":"2024-03-29T11:32:47.868333+0000","last_fullsized":"2024-03-29T11:32:47.868333+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:34.818177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338384+0000","last_change":"2024-03-29T11:32:47.878409+0000","last_active":"2024-03-29T11:32:49.338384+0000","last_peered":"2024-03-29T11:32:49.338384+0000","last_clean":"2024-03-29T11:32:49.338384+0000","last_became_active":"2024-03-29T11:32:47.877957+0000","last_became_peered":"2024-03-29T11:32:47.877957+0000","last_unstale":"2024-03-29T11:32:49.338384+0000","last_undegraded":"2024-03-29T11:32:49.338384+0000","last_fullsized":"2024-03-29T11:32:49.338384+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:24.877946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.341200+0000","last_change":"2024-03-29T11:32:32.960733+0000","last_active":"2024-03-29T11:32:47.341200+0000","last_peered":"2024-03-29T11:32:47.341200+0000","last_clean":"2024-03-29T11:32:47.341200+0000","last_became_active":"2024-03-29T11:32:32.959254+0000","last_became_peered":"2024-03-29T11:32:32.959254+0000","last_unstale":"2024-03-29T11:32:47.341200+0000","last_undegraded":"2024-03-29T11:32:47.341200+0000","last_fullsized":"2024-03-29T11:32:47.341200+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:45:42.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.866772+0000","last_change":"2024-03-29T11:32:32.422622+0000","last_active":"2024-03-29T11:32:47.866772+0000","last_peered":"2024-03-29T11:32:47.866772+0000","last_clean":"2024-03-29T11:32:47.866772+0000","last_became_active":"2024-03-29T11:32:32.421864+0000","last_became_peered":"2024-03-29T11:32:32.421864+0000","last_unstale":"2024-03-29T11:32:47.866772+0000","last_undegraded":"2024-03-29T11:32:47.866772+0000","last_fullsized":"2024-03-29T11:32:47.866772+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:23.546712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862252+0000","last_change":"2024-03-29T11:32:47.878011+0000","last_active":"2024-03-29T11:32:48.862252+0000","last_peered":"2024-03-29T11:32:48.862252+0000","last_clean":"2024-03-29T11:32:48.862252+0000","last_became_active":"2024-03-29T11:32:47.872989+0000","last_became_peered":"2024-03-29T11:32:47.872989+0000","last_unstale":"2024-03-29T11:32:48.862252+0000","last_undegraded":"2024-03-29T11:32:48.862252+0000","last_fullsized":"2024-03-29T11:32:48.862252+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:28.748316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.341735+0000","last_change":"2024-03-29T11:32:32.965844+0000","last_active":"2024-03-29T11:32:47.341735+0000","last_peered":"2024-03-29T11:32:47.341735+0000","last_clean":"2024-03-29T11:32:47.341735+0000","last_became_active":"2024-03-29T11:32:32.965511+0000","last_became_peered":"2024-03-29T11:32:32.965511+0000","last_unstale":"2024-03-29T11:32:47.341735+0000","last_undegraded":"2024-03-29T11:32:47.341735+0000","last_fullsized":"2024-03-29T11:32:47.341735+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:23.043448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862714+0000","last_change":"2024-03-29T11:32:32.419740+0000","last_active":"2024-03-29T11:32:47.862714+0000","last_peered":"2024-03-29T11:32:47.862714+0000","last_clean":"2024-03-29T11:32:47.862714+0000","last_became_active":"2024-03-29T11:32:32.419471+0000","last_became_peered":"2024-03-29T11:32:32.419471+0000","last_unstale":"2024-03-29T11:32:47.862714+0000","last_undegraded":"2024-03-29T11:32:47.862714+0000","last_fullsized":"2024-03-29T11:32:47.862714+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:01.760308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868017+0000","last_change":"2024-03-29T11:32:47.873163+0000","last_active":"2024-03-29T11:32:48.868017+0000","last_peered":"2024-03-29T11:32:48.868017+0000","last_clean":"2024-03-29T11:32:48.868017+0000","last_became_active":"2024-03-29T11:32:47.872867+0000","last_became_peered":"2024-03-29T11:32:47.872867+0000","last_unstale":"2024-03-29T11:32:48.868017+0000","last_undegraded":"2024-03-29T11:32:48.868017+0000","last_fullsized":"2024-03-29T11:32:48.868017+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:32.717628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.874734+0000","last_change":"2024-03-29T11:32:32.963417+0000","last_active":"2024-03-29T11:32:47.874734+0000","last_peered":"2024-03-29T11:32:47.874734+0000","last_clean":"2024-03-29T11:32:47.874734+0000","last_became_active":"2024-03-29T11:32:32.963109+0000","last_became_peered":"2024-03-29T11:32:32.963109+0000","last_unstale":"2024-03-29T11:32:47.874734+0000","last_undegraded":"2024-03-29T11:32:47.874734+0000","last_fullsized":"2024-03-29T11:32:47.874734+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:24:49.870800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854420+0000","last_change":"2024-03-29T11:32:32.417496+0000","last_active":"2024-03-29T11:32:47.854420+0000","last_peered":"2024-03-29T11:32:47.854420+0000","last_clean":"2024-03-29T11:32:47.854420+0000","last_became_active":"2024-03-29T11:32:32.417165+0000","last_became_peered":"2024-03-29T11:32:32.417165+0000","last_unstale":"2024-03-29T11:32:47.854420+0000","last_undegraded":"2024-03-29T11:32:47.854420+0000","last_fullsized":"2024-03-29T11:32:47.854420+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:57.952240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.194116+0000","last_change":"2024-03-29T11:32:47.876196+0000","last_active":"2024-03-29T11:32:49.194116+0000","last_peered":"2024-03-29T11:32:49.194116+0000","last_clean":"2024-03-29T11:32:49.194116+0000","last_became_active":"2024-03-29T11:32:47.875871+0000","last_became_peered":"2024-03-29T11:32:47.875871+0000","last_unstale":"2024-03-29T11:32:49.194116+0000","last_undegraded":"2024-03-29T11:32:49.194116+0000","last_fullsized":"2024-03-29T11:32:49.194116+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:47.104417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.859799+0000","last_change":"2024-03-29T11:32:32.954909+0000","last_active":"2024-03-29T11:32:47.859799+0000","last_peered":"2024-03-29T11:32:47.859799+0000","last_clean":"2024-03-29T11:32:47.859799+0000","last_became_active":"2024-03-29T11:32:32.954609+0000","last_became_peered":"2024-03-29T11:32:32.954609+0000","last_unstale":"2024-03-29T11:32:47.859799+0000","last_undegraded":"2024-03-29T11:32:47.859799+0000","last_fullsized":"2024-03-29T11:32:47.859799+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:21:04.890763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862402+0000","last_change":"2024-03-29T11:32:32.424318+0000","last_active":"2024-03-29T11:32:47.862402+0000","last_peered":"2024-03-29T11:32:47.862402+0000","last_clean":"2024-03-29T11:32:47.862402+0000","last_became_active":"2024-03-29T11:32:32.424027+0000","last_became_peered":"2024-03-29T11:32:32.424027+0000","last_unstale":"2024-03-29T11:32:47.862402+0000","last_undegraded":"2024-03-29T11:32:47.862402+0000","last_fullsized":"2024-03-29T11:32:47.862402+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:20.654799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862679+0000","last_change":"2024-03-29T11:32:47.878560+0000","last_active":"2024-03-29T11:32:48.862679+0000","last_peered":"2024-03-29T11:32:48.862679+0000","last_clean":"2024-03-29T11:32:48.862679+0000","last_became_active":"2024-03-29T11:32:47.877643+0000","last_became_peered":"2024-03-29T11:32:47.877643+0000","last_unstale":"2024-03-29T11:32:48.862679+0000","last_undegraded":"2024-03-29T11:32:48.862679+0000","last_fullsized":"2024-03-29T11:32:48.862679+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:16.654138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.853523+0000","last_change":"2024-03-29T11:32:32.950803+0000","last_active":"2024-03-29T11:32:47.853523+0000","last_peered":"2024-03-29T11:32:47.853523+0000","last_clean":"2024-03-29T11:32:47.853523+0000","last_became_active":"2024-03-29T11:32:32.950515+0000","last_became_peered":"2024-03-29T11:32:32.950515+0000","last_unstale":"2024-03-29T11:32:47.853523+0000","last_undegraded":"2024-03-29T11:32:47.853523+0000","last_fullsized":"2024-03-29T11:32:47.853523+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:42:17.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.859898+0000","last_change":"2024-03-29T11:32:32.419260+0000","last_active":"2024-03-29T11:32:47.859898+0000","last_peered":"2024-03-29T11:32:47.859898+0000","last_clean":"2024-03-29T11:32:47.859898+0000","last_became_active":"2024-03-29T11:32:32.418975+0000","last_became_peered":"2024-03-29T11:32:32.418975+0000","last_unstale":"2024-03-29T11:32:47.859898+0000","last_undegraded":"2024-03-29T11:32:47.859898+0000","last_fullsized":"2024-03-29T11:32:47.859898+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:41.904116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.337792+0000","last_change":"2024-03-29T11:32:47.873019+0000","last_active":"2024-03-29T11:32:49.337792+0000","last_peered":"2024-03-29T11:32:49.337792+0000","last_clean":"2024-03-29T11:32:49.337792+0000","last_became_active":"2024-03-29T11:32:47.872719+0000","last_became_peered":"2024-03-29T11:32:47.872719+0000","last_unstale":"2024-03-29T11:32:49.337792+0000","last_undegraded":"2024-03-29T11:32:49.337792+0000","last_fullsized":"2024-03-29T11:32:49.337792+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:11:39.133021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854999+0000","last_change":"2024-03-29T11:32:32.949825+0000","last_active":"2024-03-29T11:32:47.854999+0000","last_peered":"2024-03-29T11:32:47.854999+0000","last_clean":"2024-03-29T11:32:47.854999+0000","last_became_active":"2024-03-29T11:32:32.949562+0000","last_became_peered":"2024-03-29T11:32:32.949562+0000","last_unstale":"2024-03-29T11:32:47.854999+0000","last_undegraded":"2024-03-29T11:32:47.854999+0000","last_fullsized":"2024-03-29T11:32:47.854999+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:23.557124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.342055+0000","last_change":"2024-03-29T11:32:32.426409+0000","last_active":"2024-03-29T11:32:47.342055+0000","last_peered":"2024-03-29T11:32:47.342055+0000","last_clean":"2024-03-29T11:32:47.342055+0000","last_became_active":"2024-03-29T11:32:32.426142+0000","last_became_peered":"2024-03-29T11:32:32.426142+0000","last_unstale":"2024-03-29T11:32:47.342055+0000","last_undegraded":"2024-03-29T11:32:47.342055+0000","last_fullsized":"2024-03-29T11:32:47.342055+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:51:51.755610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.866667+0000","last_change":"2024-03-29T11:32:47.874116+0000","last_active":"2024-03-29T11:32:48.866667+0000","last_peered":"2024-03-29T11:32:48.866667+0000","last_clean":"2024-03-29T11:32:48.866667+0000","last_became_active":"2024-03-29T11:32:47.871106+0000","last_became_peered":"2024-03-29T11:32:47.871106+0000","last_unstale":"2024-03-29T11:32:48.866667+0000","last_undegraded":"2024-03-29T11:32:48.866667+0000","last_fullsized":"2024-03-29T11:32:48.866667+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:37:49.414115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.346666+0000","last_change":"2024-03-29T11:32:32.974753+0000","last_active":"2024-03-29T11:32:47.346666+0000","last_peered":"2024-03-29T11:32:47.346666+0000","last_clean":"2024-03-29T11:32:47.346666+0000","last_became_active":"2024-03-29T11:32:32.971332+0000","last_became_peered":"2024-03-29T11:32:32.971332+0000","last_unstale":"2024-03-29T11:32:47.346666+0000","last_undegraded":"2024-03-29T11:32:47.346666+0000","last_fullsized":"2024-03-29T11:32:47.346666+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:33.815472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.868697+0000","last_change":"2024-03-29T11:32:32.420533+0000","last_active":"2024-03-29T11:32:47.868697+0000","last_peered":"2024-03-29T11:32:47.868697+0000","last_clean":"2024-03-29T11:32:47.868697+0000","last_became_active":"2024-03-29T11:32:32.420284+0000","last_became_peered":"2024-03-29T11:32:32.420284+0000","last_unstale":"2024-03-29T11:32:47.868697+0000","last_undegraded":"2024-03-29T11:32:47.868697+0000","last_fullsized":"2024-03-29T11:32:47.868697+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:31.173444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.869540+0000","last_change":"2024-03-29T11:32:47.864600+0000","last_active":"2024-03-29T11:32:47.869540+0000","last_peered":"2024-03-29T11:32:47.869540+0000","last_clean":"2024-03-29T11:32:47.869540+0000","last_became_active":"2024-03-29T11:32:47.864345+0000","last_became_peered":"2024-03-29T11:32:47.864345+0000","last_unstale":"2024-03-29T11:32:47.869540+0000","last_undegraded":"2024-03-29T11:32:47.869540+0000","last_fullsized":"2024-03-29T11:32:47.869540+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:28.748316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.866587+0000","last_change":"2024-03-29T11:32:32.969812+0000","last_active":"2024-03-29T11:32:47.866587+0000","last_peered":"2024-03-29T11:32:47.866587+0000","last_clean":"2024-03-29T11:32:47.866587+0000","last_became_active":"2024-03-29T11:32:32.969479+0000","last_became_peered":"2024-03-29T11:32:32.969479+0000","last_unstale":"2024-03-29T11:32:47.866587+0000","last_undegraded":"2024-03-29T11:32:47.866587+0000","last_fullsized":"2024-03-29T11:32:47.866587+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:57.232582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"247'3","reported_seq":18,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.856072+0000","last_change":"2024-03-29T11:32:32.418264+0000","last_active":"2024-03-29T11:32:47.856072+0000","last_peered":"2024-03-29T11:32:47.856072+0000","last_clean":"2024-03-29T11:32:47.856072+0000","last_became_active":"2024-03-29T11:32:32.417013+0000","last_became_peered":"2024-03-29T11:32:32.417013+0000","last_unstale":"2024-03-29T11:32:47.856072+0000","last_undegraded":"2024-03-29T11:32:47.856072+0000","last_fullsized":"2024-03-29T11:32:47.856072+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:51.960387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.878388+0000","last_change":"2024-03-29T11:32:47.876221+0000","last_active":"2024-03-29T11:32:47.878388+0000","last_peered":"2024-03-29T11:32:47.878388+0000","last_clean":"2024-03-29T11:32:47.878388+0000","last_became_active":"2024-03-29T11:32:47.873772+0000","last_became_peered":"2024-03-29T11:32:47.873772+0000","last_unstale":"2024-03-29T11:32:47.878388+0000","last_undegraded":"2024-03-29T11:32:47.878388+0000","last_fullsized":"2024-03-29T11:32:47.878388+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:08.340482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.867521+0000","last_change":"2024-03-29T11:32:32.955257+0000","last_active":"2024-03-29T11:32:47.867521+0000","last_peered":"2024-03-29T11:32:47.867521+0000","last_clean":"2024-03-29T11:32:47.867521+0000","last_became_active":"2024-03-29T11:32:32.952017+0000","last_became_peered":"2024-03-29T11:32:32.952017+0000","last_unstale":"2024-03-29T11:32:47.867521+0000","last_undegraded":"2024-03-29T11:32:47.867521+0000","last_fullsized":"2024-03-29T11:32:47.867521+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:45.368395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"248'8","reported_seq":23,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854751+0000","last_change":"2024-03-29T11:32:32.418422+0000","last_active":"2024-03-29T11:32:47.854751+0000","last_peered":"2024-03-29T11:32:47.854751+0000","last_clean":"2024-03-29T11:32:47.854751+0000","last_became_active":"2024-03-29T11:32:32.418119+0000","last_became_peered":"2024-03-29T11:32:32.418119+0000","last_unstale":"2024-03-29T11:32:47.854751+0000","last_undegraded":"2024-03-29T11:32:47.854751+0000","last_fullsized":"2024-03-29T11:32:47.854751+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:34.164745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218793+0000","last_change":"2024-03-29T11:32:47.863185+0000","last_active":"2024-03-29T11:32:49.218793+0000","last_peered":"2024-03-29T11:32:49.218793+0000","last_clean":"2024-03-29T11:32:49.218793+0000","last_became_active":"2024-03-29T11:32:47.862499+0000","last_became_peered":"2024-03-29T11:32:47.862499+0000","last_unstale":"2024-03-29T11:32:49.218793+0000","last_undegraded":"2024-03-29T11:32:49.218793+0000","last_fullsized":"2024-03-29T11:32:49.218793+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:18.229652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862221+0000","last_change":"2024-03-29T11:32:32.955062+0000","last_active":"2024-03-29T11:32:47.862221+0000","last_peered":"2024-03-29T11:32:47.862221+0000","last_clean":"2024-03-29T11:32:47.862221+0000","last_became_active":"2024-03-29T11:32:32.954362+0000","last_became_peered":"2024-03-29T11:32:32.954362+0000","last_unstale":"2024-03-29T11:32:47.862221+0000","last_undegraded":"2024-03-29T11:32:47.862221+0000","last_fullsized":"2024-03-29T11:32:47.862221+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:35:32.510118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.855404+0000","last_change":"2024-03-29T11:32:32.427566+0000","last_active":"2024-03-29T11:32:47.855404+0000","last_peered":"2024-03-29T11:32:47.855404+0000","last_clean":"2024-03-29T11:32:47.855404+0000","last_became_active":"2024-03-29T11:32:32.426512+0000","last_became_peered":"2024-03-29T11:32:32.426512+0000","last_unstale":"2024-03-29T11:32:47.855404+0000","last_undegraded":"2024-03-29T11:32:47.855404+0000","last_fullsized":"2024-03-29T11:32:47.855404+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:49:23.345901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867565+0000","last_change":"2024-03-29T11:32:47.879992+0000","last_active":"2024-03-29T11:32:48.867565+0000","last_peered":"2024-03-29T11:32:48.867565+0000","last_clean":"2024-03-29T11:32:48.867565+0000","last_became_active":"2024-03-29T11:32:47.879694+0000","last_became_peered":"2024-03-29T11:32:47.879694+0000","last_unstale":"2024-03-29T11:32:48.867565+0000","last_undegraded":"2024-03-29T11:32:48.867565+0000","last_fullsized":"2024-03-29T11:32:48.867565+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:15.233472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.874820+0000","last_change":"2024-03-29T11:32:32.956252+0000","last_active":"2024-03-29T11:32:47.874820+0000","last_peered":"2024-03-29T11:32:47.874820+0000","last_clean":"2024-03-29T11:32:47.874820+0000","last_became_active":"2024-03-29T11:32:32.955859+0000","last_became_peered":"2024-03-29T11:32:32.955859+0000","last_unstale":"2024-03-29T11:32:47.874820+0000","last_undegraded":"2024-03-29T11:32:47.874820+0000","last_fullsized":"2024-03-29T11:32:47.874820+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:29.934653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.860117+0000","last_change":"2024-03-29T11:32:32.420246+0000","last_active":"2024-03-29T11:32:47.860117+0000","last_peered":"2024-03-29T11:32:47.860117+0000","last_clean":"2024-03-29T11:32:47.860117+0000","last_became_active":"2024-03-29T11:32:32.419983+0000","last_became_peered":"2024-03-29T11:32:32.419983+0000","last_unstale":"2024-03-29T11:32:47.860117+0000","last_undegraded":"2024-03-29T11:32:47.860117+0000","last_fullsized":"2024-03-29T11:32:47.860117+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:35.527658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867670+0000","last_change":"2024-03-29T11:32:47.883672+0000","last_active":"2024-03-29T11:32:48.867670+0000","last_peered":"2024-03-29T11:32:48.867670+0000","last_clean":"2024-03-29T11:32:48.867670+0000","last_became_active":"2024-03-29T11:32:47.883391+0000","last_became_peered":"2024-03-29T11:32:47.883391+0000","last_unstale":"2024-03-29T11:32:48.867670+0000","last_undegraded":"2024-03-29T11:32:48.867670+0000","last_fullsized":"2024-03-29T11:32:48.867670+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:29.841029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862101+0000","last_change":"2024-03-29T11:32:32.946933+0000","last_active":"2024-03-29T11:32:47.862101+0000","last_peered":"2024-03-29T11:32:47.862101+0000","last_clean":"2024-03-29T11:32:47.862101+0000","last_became_active":"2024-03-29T11:32:32.946663+0000","last_became_peered":"2024-03-29T11:32:32.946663+0000","last_unstale":"2024-03-29T11:32:47.862101+0000","last_undegraded":"2024-03-29T11:32:47.862101+0000","last_fullsized":"2024-03-29T11:32:47.862101+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:23.557124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862467+0000","last_change":"2024-03-29T11:32:32.419238+0000","last_active":"2024-03-29T11:32:47.862467+0000","last_peered":"2024-03-29T11:32:47.862467+0000","last_clean":"2024-03-29T11:32:47.862467+0000","last_became_active":"2024-03-29T11:32:32.416581+0000","last_became_peered":"2024-03-29T11:32:32.416581+0000","last_unstale":"2024-03-29T11:32:47.862467+0000","last_undegraded":"2024-03-29T11:32:47.862467+0000","last_fullsized":"2024-03-29T11:32:47.862467+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:09.293759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338972+0000","last_change":"2024-03-29T11:32:47.870210+0000","last_active":"2024-03-29T11:32:49.338972+0000","last_peered":"2024-03-29T11:32:49.338972+0000","last_clean":"2024-03-29T11:32:49.338972+0000","last_became_active":"2024-03-29T11:32:47.869589+0000","last_became_peered":"2024-03-29T11:32:47.869589+0000","last_unstale":"2024-03-29T11:32:49.338972+0000","last_undegraded":"2024-03-29T11:32:49.338972+0000","last_fullsized":"2024-03-29T11:32:49.338972+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:09:46.377915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862352+0000","last_change":"2024-03-29T11:32:32.960659+0000","last_active":"2024-03-29T11:32:47.862352+0000","last_peered":"2024-03-29T11:32:47.862352+0000","last_clean":"2024-03-29T11:32:47.862352+0000","last_became_active":"2024-03-29T11:32:32.960186+0000","last_became_peered":"2024-03-29T11:32:32.960186+0000","last_unstale":"2024-03-29T11:32:47.862352+0000","last_undegraded":"2024-03-29T11:32:47.862352+0000","last_fullsized":"2024-03-29T11:32:47.862352+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:11.369063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"247'8","reported_seq":22,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.345480+0000","last_change":"2024-03-29T11:32:32.420232+0000","last_active":"2024-03-29T11:32:47.345480+0000","last_peered":"2024-03-29T11:32:47.345480+0000","last_clean":"2024-03-29T11:32:47.345480+0000","last_became_active":"2024-03-29T11:32:32.419973+0000","last_became_peered":"2024-03-29T11:32:32.419973+0000","last_unstale":"2024-03-29T11:32:47.345480+0000","last_undegraded":"2024-03-29T11:32:47.345480+0000","last_fullsized":"2024-03-29T11:32:47.345480+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:20.810629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218276+0000","last_change":"2024-03-29T11:32:47.879307+0000","last_active":"2024-03-29T11:32:49.218276+0000","last_peered":"2024-03-29T11:32:49.218276+0000","last_clean":"2024-03-29T11:32:49.218276+0000","last_became_active":"2024-03-29T11:32:47.874529+0000","last_became_peered":"2024-03-29T11:32:47.874529+0000","last_unstale":"2024-03-29T11:32:49.218276+0000","last_undegraded":"2024-03-29T11:32:49.218276+0000","last_fullsized":"2024-03-29T11:32:49.218276+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:16.654138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.867507+0000","last_change":"2024-03-29T11:32:32.952991+0000","last_active":"2024-03-29T11:32:47.867507+0000","last_peered":"2024-03-29T11:32:47.867507+0000","last_clean":"2024-03-29T11:32:47.867507+0000","last_became_active":"2024-03-29T11:32:32.952683+0000","last_became_peered":"2024-03-29T11:32:32.952683+0000","last_unstale":"2024-03-29T11:32:47.867507+0000","last_undegraded":"2024-03-29T11:32:47.867507+0000","last_fullsized":"2024-03-29T11:32:47.867507+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:30.121534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.872169+0000","last_change":"2024-03-29T11:32:32.419741+0000","last_active":"2024-03-29T11:32:47.872169+0000","last_peered":"2024-03-29T11:32:47.872169+0000","last_clean":"2024-03-29T11:32:47.872169+0000","last_became_active":"2024-03-29T11:32:32.418896+0000","last_became_peered":"2024-03-29T11:32:32.418896+0000","last_unstale":"2024-03-29T11:32:47.872169+0000","last_undegraded":"2024-03-29T11:32:47.872169+0000","last_fullsized":"2024-03-29T11:32:47.872169+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:59:42.656917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.861925+0000","last_change":"2024-03-29T11:32:47.885402+0000","last_active":"2024-03-29T11:32:48.861925+0000","last_peered":"2024-03-29T11:32:48.861925+0000","last_clean":"2024-03-29T11:32:48.861925+0000","last_became_active":"2024-03-29T11:32:47.885130+0000","last_became_peered":"2024-03-29T11:32:47.885130+0000","last_unstale":"2024-03-29T11:32:48.861925+0000","last_undegraded":"2024-03-29T11:32:48.861925+0000","last_fullsized":"2024-03-29T11:32:48.861925+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:18.229652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.869035+0000","last_change":"2024-03-29T11:32:32.963050+0000","last_active":"2024-03-29T11:32:47.869035+0000","last_peered":"2024-03-29T11:32:47.869035+0000","last_clean":"2024-03-29T11:32:47.869035+0000","last_became_active":"2024-03-29T11:32:32.962777+0000","last_became_peered":"2024-03-29T11:32:32.962777+0000","last_unstale":"2024-03-29T11:32:47.869035+0000","last_undegraded":"2024-03-29T11:32:47.869035+0000","last_fullsized":"2024-03-29T11:32:47.869035+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:28.065549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.868784+0000","last_change":"2024-03-29T11:32:32.424196+0000","last_active":"2024-03-29T11:32:47.868784+0000","last_peered":"2024-03-29T11:32:47.868784+0000","last_clean":"2024-03-29T11:32:47.868784+0000","last_became_active":"2024-03-29T11:32:32.423924+0000","last_became_peered":"2024-03-29T11:32:32.423924+0000","last_unstale":"2024-03-29T11:32:47.868784+0000","last_undegraded":"2024-03-29T11:32:47.868784+0000","last_fullsized":"2024-03-29T11:32:47.868784+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:51.659258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.861865+0000","last_change":"2024-03-29T11:32:47.888252+0000","last_active":"2024-03-29T11:32:48.861865+0000","last_peered":"2024-03-29T11:32:48.861865+0000","last_clean":"2024-03-29T11:32:48.861865+0000","last_became_active":"2024-03-29T11:32:47.887169+0000","last_became_peered":"2024-03-29T11:32:47.887169+0000","last_unstale":"2024-03-29T11:32:48.861865+0000","last_undegraded":"2024-03-29T11:32:48.861865+0000","last_fullsized":"2024-03-29T11:32:48.861865+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:05:30.468540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.877255+0000","last_change":"2024-03-29T11:32:32.958521+0000","last_active":"2024-03-29T11:32:47.877255+0000","last_peered":"2024-03-29T11:32:47.877255+0000","last_clean":"2024-03-29T11:32:47.877255+0000","last_became_active":"2024-03-29T11:32:32.958260+0000","last_became_peered":"2024-03-29T11:32:32.958260+0000","last_unstale":"2024-03-29T11:32:47.877255+0000","last_undegraded":"2024-03-29T11:32:47.877255+0000","last_fullsized":"2024-03-29T11:32:47.877255+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:34.744111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.878893+0000","last_change":"2024-03-29T11:32:32.421703+0000","last_active":"2024-03-29T11:32:47.878893+0000","last_peered":"2024-03-29T11:32:47.878893+0000","last_clean":"2024-03-29T11:32:47.878893+0000","last_became_active":"2024-03-29T11:32:32.421417+0000","last_became_peered":"2024-03-29T11:32:32.421417+0000","last_unstale":"2024-03-29T11:32:47.878893+0000","last_undegraded":"2024-03-29T11:32:47.878893+0000","last_fullsized":"2024-03-29T11:32:47.878893+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:51.659258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862130+0000","last_change":"2024-03-29T11:32:47.885332+0000","last_active":"2024-03-29T11:32:48.862130+0000","last_peered":"2024-03-29T11:32:48.862130+0000","last_clean":"2024-03-29T11:32:48.862130+0000","last_became_active":"2024-03-29T11:32:47.883499+0000","last_became_peered":"2024-03-29T11:32:47.883499+0000","last_unstale":"2024-03-29T11:32:48.862130+0000","last_undegraded":"2024-03-29T11:32:48.862130+0000","last_fullsized":"2024-03-29T11:32:48.862130+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:59:56.436348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.342104+0000","last_change":"2024-03-29T11:32:32.960765+0000","last_active":"2024-03-29T11:32:47.342104+0000","last_peered":"2024-03-29T11:32:47.342104+0000","last_clean":"2024-03-29T11:32:47.342104+0000","last_became_active":"2024-03-29T11:32:32.959952+0000","last_became_peered":"2024-03-29T11:32:32.959952+0000","last_unstale":"2024-03-29T11:32:47.342104+0000","last_undegraded":"2024-03-29T11:32:47.342104+0000","last_fullsized":"2024-03-29T11:32:47.342104+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:18.696940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.345358+0000","last_change":"2024-03-29T11:32:32.423658+0000","last_active":"2024-03-29T11:32:47.345358+0000","last_peered":"2024-03-29T11:32:47.345358+0000","last_clean":"2024-03-29T11:32:47.345358+0000","last_became_active":"2024-03-29T11:32:32.423195+0000","last_became_peered":"2024-03-29T11:32:32.423195+0000","last_unstale":"2024-03-29T11:32:47.345358+0000","last_undegraded":"2024-03-29T11:32:47.345358+0000","last_fullsized":"2024-03-29T11:32:47.345358+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:22.547790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.888067+0000","last_change":"2024-03-29T11:32:47.887314+0000","last_active":"2024-03-29T11:32:47.888067+0000","last_peered":"2024-03-29T11:32:47.888067+0000","last_clean":"2024-03-29T11:32:47.888067+0000","last_became_active":"2024-03-29T11:32:47.887045+0000","last_became_peered":"2024-03-29T11:32:47.887045+0000","last_unstale":"2024-03-29T11:32:47.888067+0000","last_undegraded":"2024-03-29T11:32:47.888067+0000","last_fullsized":"2024-03-29T11:32:47.888067+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:13:44.626121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.339860+0000","last_change":"2024-03-29T11:32:32.958080+0000","last_active":"2024-03-29T11:32:47.339860+0000","last_peered":"2024-03-29T11:32:47.339860+0000","last_clean":"2024-03-29T11:32:47.339860+0000","last_became_active":"2024-03-29T11:32:32.957784+0000","last_became_peered":"2024-03-29T11:32:32.957784+0000","last_unstale":"2024-03-29T11:32:47.339860+0000","last_undegraded":"2024-03-29T11:32:47.339860+0000","last_fullsized":"2024-03-29T11:32:47.339860+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:48.762233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.863931+0000","last_change":"2024-03-29T11:32:32.421797+0000","last_active":"2024-03-29T11:32:47.863931+0000","last_peered":"2024-03-29T11:32:47.863931+0000","last_clean":"2024-03-29T11:32:47.863931+0000","last_became_active":"2024-03-29T11:32:32.418956+0000","last_became_peered":"2024-03-29T11:32:32.418956+0000","last_unstale":"2024-03-29T11:32:47.863931+0000","last_undegraded":"2024-03-29T11:32:47.863931+0000","last_fullsized":"2024-03-29T11:32:47.863931+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:32.806138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867893+0000","last_change":"2024-03-29T11:32:47.874664+0000","last_active":"2024-03-29T11:32:48.867893+0000","last_peered":"2024-03-29T11:32:48.867893+0000","last_clean":"2024-03-29T11:32:48.867893+0000","last_became_active":"2024-03-29T11:32:47.873804+0000","last_became_peered":"2024-03-29T11:32:47.873804+0000","last_unstale":"2024-03-29T11:32:48.867893+0000","last_undegraded":"2024-03-29T11:32:48.867893+0000","last_fullsized":"2024-03-29T11:32:48.867893+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:05:30.468540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854360+0000","last_change":"2024-03-29T11:32:32.951479+0000","last_active":"2024-03-29T11:32:47.854360+0000","last_peered":"2024-03-29T11:32:47.854360+0000","last_clean":"2024-03-29T11:32:47.854360+0000","last_became_active":"2024-03-29T11:32:32.951234+0000","last_became_peered":"2024-03-29T11:32:32.951234+0000","last_unstale":"2024-03-29T11:32:47.854360+0000","last_undegraded":"2024-03-29T11:32:47.854360+0000","last_fullsized":"2024-03-29T11:32:47.854360+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:48.762233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.341905+0000","last_change":"2024-03-29T11:32:32.422506+0000","last_active":"2024-03-29T11:32:47.341905+0000","last_peered":"2024-03-29T11:32:47.341905+0000","last_clean":"2024-03-29T11:32:47.341905+0000","last_became_active":"2024-03-29T11:32:32.422249+0000","last_became_peered":"2024-03-29T11:32:32.422249+0000","last_unstale":"2024-03-29T11:32:47.341905+0000","last_undegraded":"2024-03-29T11:32:47.341905+0000","last_fullsized":"2024-03-29T11:32:47.341905+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:29.112200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868675+0000","last_change":"2024-03-29T11:32:47.882727+0000","last_active":"2024-03-29T11:32:48.868675+0000","last_peered":"2024-03-29T11:32:48.868675+0000","last_clean":"2024-03-29T11:32:48.868675+0000","last_became_active":"2024-03-29T11:32:47.882455+0000","last_became_peered":"2024-03-29T11:32:47.882455+0000","last_unstale":"2024-03-29T11:32:48.868675+0000","last_undegraded":"2024-03-29T11:32:48.868675+0000","last_fullsized":"2024-03-29T11:32:48.868675+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:27.778216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.343117+0000","last_change":"2024-03-29T11:32:32.955109+0000","last_active":"2024-03-29T11:32:47.343117+0000","last_peered":"2024-03-29T11:32:47.343117+0000","last_clean":"2024-03-29T11:32:47.343117+0000","last_became_active":"2024-03-29T11:32:32.954862+0000","last_became_peered":"2024-03-29T11:32:32.954862+0000","last_unstale":"2024-03-29T11:32:47.343117+0000","last_undegraded":"2024-03-29T11:32:47.343117+0000","last_fullsized":"2024-03-29T11:32:47.343117+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:18.702968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.877964+0000","last_change":"2024-03-29T11:32:32.429057+0000","last_active":"2024-03-29T11:32:47.877964+0000","last_peered":"2024-03-29T11:32:47.877964+0000","last_clean":"2024-03-29T11:32:47.877964+0000","last_became_active":"2024-03-29T11:32:32.428782+0000","last_became_peered":"2024-03-29T11:32:32.428782+0000","last_unstale":"2024-03-29T11:32:47.877964+0000","last_undegraded":"2024-03-29T11:32:47.877964+0000","last_fullsized":"2024-03-29T11:32:47.877964+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:23.546712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.877786+0000","last_change":"2024-03-29T11:32:47.871820+0000","last_active":"2024-03-29T11:32:47.877786+0000","last_peered":"2024-03-29T11:32:47.877786+0000","last_clean":"2024-03-29T11:32:47.877786+0000","last_became_active":"2024-03-29T11:32:47.871052+0000","last_became_peered":"2024-03-29T11:32:47.871052+0000","last_unstale":"2024-03-29T11:32:47.877786+0000","last_undegraded":"2024-03-29T11:32:47.877786+0000","last_fullsized":"2024-03-29T11:32:47.877786+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:28.536288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.338800+0000","last_change":"2024-03-29T11:32:32.954161+0000","last_active":"2024-03-29T11:32:47.338800+0000","last_peered":"2024-03-29T11:32:47.338800+0000","last_clean":"2024-03-29T11:32:47.338800+0000","last_became_active":"2024-03-29T11:32:32.953891+0000","last_became_peered":"2024-03-29T11:32:32.953891+0000","last_unstale":"2024-03-29T11:32:47.338800+0000","last_undegraded":"2024-03-29T11:32:47.338800+0000","last_fullsized":"2024-03-29T11:32:47.338800+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:42:17.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862271+0000","last_change":"2024-03-29T11:32:32.418585+0000","last_active":"2024-03-29T11:32:47.862271+0000","last_peered":"2024-03-29T11:32:47.862271+0000","last_clean":"2024-03-29T11:32:47.862271+0000","last_became_active":"2024-03-29T11:32:32.418297+0000","last_became_peered":"2024-03-29T11:32:32.418297+0000","last_unstale":"2024-03-29T11:32:47.862271+0000","last_undegraded":"2024-03-29T11:32:47.862271+0000","last_fullsized":"2024-03-29T11:32:47.862271+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:01.760308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.883867+0000","last_change":"2024-03-29T11:32:47.882656+0000","last_active":"2024-03-29T11:32:47.883867+0000","last_peered":"2024-03-29T11:32:47.883867+0000","last_clean":"2024-03-29T11:32:47.883867+0000","last_became_active":"2024-03-29T11:32:47.882267+0000","last_became_peered":"2024-03-29T11:32:47.882267+0000","last_unstale":"2024-03-29T11:32:47.883867+0000","last_undegraded":"2024-03-29T11:32:47.883867+0000","last_fullsized":"2024-03-29T11:32:47.883867+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:54.217597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.870826+0000","last_change":"2024-03-29T11:32:32.953710+0000","last_active":"2024-03-29T11:32:47.870826+0000","last_peered":"2024-03-29T11:32:47.870826+0000","last_clean":"2024-03-29T11:32:47.870826+0000","last_became_active":"2024-03-29T11:32:32.953447+0000","last_became_peered":"2024-03-29T11:32:32.953447+0000","last_unstale":"2024-03-29T11:32:47.870826+0000","last_undegraded":"2024-03-29T11:32:47.870826+0000","last_fullsized":"2024-03-29T11:32:47.870826+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.866441+0000","last_change":"2024-03-29T11:32:32.427167+0000","last_active":"2024-03-29T11:32:47.866441+0000","last_peered":"2024-03-29T11:32:47.866441+0000","last_clean":"2024-03-29T11:32:47.866441+0000","last_became_active":"2024-03-29T11:32:32.426833+0000","last_became_peered":"2024-03-29T11:32:32.426833+0000","last_unstale":"2024-03-29T11:32:47.866441+0000","last_undegraded":"2024-03-29T11:32:47.866441+0000","last_fullsized":"2024-03-29T11:32:47.866441+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:29.373413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867133+0000","last_change":"2024-03-29T11:32:47.882648+0000","last_active":"2024-03-29T11:32:48.867133+0000","last_peered":"2024-03-29T11:32:48.867133+0000","last_clean":"2024-03-29T11:32:48.867133+0000","last_became_active":"2024-03-29T11:32:47.882342+0000","last_became_peered":"2024-03-29T11:32:47.882342+0000","last_unstale":"2024-03-29T11:32:48.867133+0000","last_undegraded":"2024-03-29T11:32:48.867133+0000","last_fullsized":"2024-03-29T11:32:48.867133+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:16:25.816476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.856238+0000","last_change":"2024-03-29T11:32:32.967461+0000","last_active":"2024-03-29T11:32:47.856238+0000","last_peered":"2024-03-29T11:32:47.856238+0000","last_clean":"2024-03-29T11:32:47.856238+0000","last_became_active":"2024-03-29T11:32:32.967155+0000","last_became_peered":"2024-03-29T11:32:32.967155+0000","last_unstale":"2024-03-29T11:32:47.856238+0000","last_undegraded":"2024-03-29T11:32:47.856238+0000","last_fullsized":"2024-03-29T11:32:47.856238+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:24.736585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"248'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.863975+0000","last_change":"2024-03-29T11:32:32.421259+0000","last_active":"2024-03-29T11:32:47.863975+0000","last_peered":"2024-03-29T11:32:47.863975+0000","last_clean":"2024-03-29T11:32:47.863975+0000","last_became_active":"2024-03-29T11:32:32.420971+0000","last_became_peered":"2024-03-29T11:32:32.420971+0000","last_unstale":"2024-03-29T11:32:47.863975+0000","last_undegraded":"2024-03-29T11:32:47.863975+0000","last_fullsized":"2024-03-29T11:32:47.863975+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:34.164745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218433+0000","last_change":"2024-03-29T11:32:47.869320+0000","last_active":"2024-03-29T11:32:49.218433+0000","last_peered":"2024-03-29T11:32:49.218433+0000","last_clean":"2024-03-29T11:32:49.218433+0000","last_became_active":"2024-03-29T11:32:47.869067+0000","last_became_peered":"2024-03-29T11:32:47.869067+0000","last_unstale":"2024-03-29T11:32:49.218433+0000","last_undegraded":"2024-03-29T11:32:49.218433+0000","last_fullsized":"2024-03-29T11:32:49.218433+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:19.639527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.345276+0000","last_change":"2024-03-29T11:32:32.970626+0000","last_active":"2024-03-29T11:32:47.345276+0000","last_peered":"2024-03-29T11:32:47.345276+0000","last_clean":"2024-03-29T11:32:47.345276+0000","last_became_active":"2024-03-29T11:32:32.970154+0000","last_became_peered":"2024-03-29T11:32:32.970154+0000","last_unstale":"2024-03-29T11:32:47.345276+0000","last_undegraded":"2024-03-29T11:32:47.345276+0000","last_fullsized":"2024-03-29T11:32:47.345276+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:35.827510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.879058+0000","last_change":"2024-03-29T11:32:32.429420+0000","last_active":"2024-03-29T11:32:47.879058+0000","last_peered":"2024-03-29T11:32:47.879058+0000","last_clean":"2024-03-29T11:32:47.879058+0000","last_became_active":"2024-03-29T11:32:32.429133+0000","last_became_peered":"2024-03-29T11:32:32.429133+0000","last_unstale":"2024-03-29T11:32:47.879058+0000","last_undegraded":"2024-03-29T11:32:47.879058+0000","last_fullsized":"2024-03-29T11:32:47.879058+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:03.995789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193741+0000","last_change":"2024-03-29T11:32:47.874467+0000","last_active":"2024-03-29T11:32:49.193741+0000","last_peered":"2024-03-29T11:32:49.193741+0000","last_clean":"2024-03-29T11:32:49.193741+0000","last_became_active":"2024-03-29T11:32:47.874198+0000","last_became_peered":"2024-03-29T11:32:47.874198+0000","last_unstale":"2024-03-29T11:32:49.193741+0000","last_undegraded":"2024-03-29T11:32:49.193741+0000","last_fullsized":"2024-03-29T11:32:49.193741+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:44.856293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.346235+0000","last_change":"2024-03-29T11:32:32.971255+0000","last_active":"2024-03-29T11:32:47.346235+0000","last_peered":"2024-03-29T11:32:47.346235+0000","last_clean":"2024-03-29T11:32:47.346235+0000","last_became_active":"2024-03-29T11:32:32.968877+0000","last_became_peered":"2024-03-29T11:32:32.968877+0000","last_unstale":"2024-03-29T11:32:47.346235+0000","last_undegraded":"2024-03-29T11:32:47.346235+0000","last_fullsized":"2024-03-29T11:32:47.346235+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:42.913450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.345435+0000","last_change":"2024-03-29T11:32:32.424671+0000","last_active":"2024-03-29T11:32:47.345435+0000","last_peered":"2024-03-29T11:32:47.345435+0000","last_clean":"2024-03-29T11:32:47.345435+0000","last_became_active":"2024-03-29T11:32:32.424378+0000","last_became_peered":"2024-03-29T11:32:32.424378+0000","last_unstale":"2024-03-29T11:32:47.345435+0000","last_undegraded":"2024-03-29T11:32:47.345435+0000","last_fullsized":"2024-03-29T11:32:47.345435+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:41.180787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193746+0000","last_change":"2024-03-29T11:32:47.873238+0000","last_active":"2024-03-29T11:32:49.193746+0000","last_peered":"2024-03-29T11:32:49.193746+0000","last_clean":"2024-03-29T11:32:49.193746+0000","last_became_active":"2024-03-29T11:32:47.872942+0000","last_became_peered":"2024-03-29T11:32:47.872942+0000","last_unstale":"2024-03-29T11:32:49.193746+0000","last_undegraded":"2024-03-29T11:32:49.193746+0000","last_fullsized":"2024-03-29T11:32:49.193746+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:11:54.332535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862059+0000","last_change":"2024-03-29T11:32:32.952424+0000","last_active":"2024-03-29T11:32:47.862059+0000","last_peered":"2024-03-29T11:32:47.862059+0000","last_clean":"2024-03-29T11:32:47.862059+0000","last_became_active":"2024-03-29T11:32:32.951804+0000","last_became_peered":"2024-03-29T11:32:32.951804+0000","last_unstale":"2024-03-29T11:32:47.862059+0000","last_undegraded":"2024-03-29T11:32:47.862059+0000","last_fullsized":"2024-03-29T11:32:47.862059+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:22:05.943330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.867307+0000","last_change":"2024-03-29T11:32:32.418931+0000","last_active":"2024-03-29T11:32:47.867307+0000","last_peered":"2024-03-29T11:32:47.867307+0000","last_clean":"2024-03-29T11:32:47.867307+0000","last_became_active":"2024-03-29T11:32:32.418646+0000","last_became_peered":"2024-03-29T11:32:32.418646+0000","last_unstale":"2024-03-29T11:32:47.867307+0000","last_undegraded":"2024-03-29T11:32:47.867307+0000","last_fullsized":"2024-03-29T11:32:47.867307+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:17:12.443004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193907+0000","last_change":"2024-03-29T11:32:47.869516+0000","last_active":"2024-03-29T11:32:49.193907+0000","last_peered":"2024-03-29T11:32:49.193907+0000","last_clean":"2024-03-29T11:32:49.193907+0000","last_became_active":"2024-03-29T11:32:47.868816+0000","last_became_peered":"2024-03-29T11:32:47.868816+0000","last_unstale":"2024-03-29T11:32:49.193907+0000","last_undegraded":"2024-03-29T11:32:49.193907+0000","last_fullsized":"2024-03-29T11:32:49.193907+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:48.737111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.339534+0000","last_change":"2024-03-29T11:32:32.971041+0000","last_active":"2024-03-29T11:32:47.339534+0000","last_peered":"2024-03-29T11:32:47.339534+0000","last_clean":"2024-03-29T11:32:47.339534+0000","last_became_active":"2024-03-29T11:32:32.970781+0000","last_became_peered":"2024-03-29T11:32:32.970781+0000","last_unstale":"2024-03-29T11:32:47.339534+0000","last_undegraded":"2024-03-29T11:32:47.339534+0000","last_fullsized":"2024-03-29T11:32:47.339534+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:30.382747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.863169+0000","last_change":"2024-03-29T11:32:32.414137+0000","last_active":"2024-03-29T11:32:47.863169+0000","last_peered":"2024-03-29T11:32:47.863169+0000","last_clean":"2024-03-29T11:32:47.863169+0000","last_became_active":"2024-03-29T11:32:32.413846+0000","last_became_peered":"2024-03-29T11:32:32.413846+0000","last_unstale":"2024-03-29T11:32:47.863169+0000","last_undegraded":"2024-03-29T11:32:47.863169+0000","last_fullsized":"2024-03-29T11:32:47.863169+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:09.293759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868742+0000","last_change":"2024-03-29T11:32:47.884754+0000","last_active":"2024-03-29T11:32:48.868742+0000","last_peered":"2024-03-29T11:32:48.868742+0000","last_clean":"2024-03-29T11:32:48.868742+0000","last_became_active":"2024-03-29T11:32:47.884252+0000","last_became_peered":"2024-03-29T11:32:47.884252+0000","last_unstale":"2024-03-29T11:32:48.868742+0000","last_undegraded":"2024-03-29T11:32:48.868742+0000","last_fullsized":"2024-03-29T11:32:48.868742+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:59:56.436348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.338171+0000","last_change":"2024-03-29T11:32:32.956222+0000","last_active":"2024-03-29T11:32:47.338171+0000","last_peered":"2024-03-29T11:32:47.338171+0000","last_clean":"2024-03-29T11:32:47.338171+0000","last_became_active":"2024-03-29T11:32:32.955784+0000","last_became_peered":"2024-03-29T11:32:32.955784+0000","last_unstale":"2024-03-29T11:32:47.338171+0000","last_undegraded":"2024-03-29T11:32:47.338171+0000","last_fullsized":"2024-03-29T11:32:47.338171+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:52.969721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.863793+0000","last_change":"2024-03-29T11:32:32.421490+0000","last_active":"2024-03-29T11:32:47.863793+0000","last_peered":"2024-03-29T11:32:47.863793+0000","last_clean":"2024-03-29T11:32:47.863793+0000","last_became_active":"2024-03-29T11:32:32.421236+0000","last_became_peered":"2024-03-29T11:32:32.421236+0000","last_unstale":"2024-03-29T11:32:47.863793+0000","last_undegraded":"2024-03-29T11:32:47.863793+0000","last_fullsized":"2024-03-29T11:32:47.863793+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:17.693634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.881856+0000","last_change":"2024-03-29T11:32:47.879901+0000","last_active":"2024-03-29T11:32:47.881856+0000","last_peered":"2024-03-29T11:32:47.881856+0000","last_clean":"2024-03-29T11:32:47.881856+0000","last_became_active":"2024-03-29T11:32:47.879634+0000","last_became_peered":"2024-03-29T11:32:47.879634+0000","last_unstale":"2024-03-29T11:32:47.881856+0000","last_undegraded":"2024-03-29T11:32:47.881856+0000","last_fullsized":"2024-03-29T11:32:47.881856+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:28:46.899353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.853526+0000","last_change":"2024-03-29T11:32:32.950145+0000","last_active":"2024-03-29T11:32:47.853526+0000","last_peered":"2024-03-29T11:32:47.853526+0000","last_clean":"2024-03-29T11:32:47.853526+0000","last_became_active":"2024-03-29T11:32:32.949444+0000","last_became_peered":"2024-03-29T11:32:32.949444+0000","last_unstale":"2024-03-29T11:32:47.853526+0000","last_undegraded":"2024-03-29T11:32:47.853526+0000","last_fullsized":"2024-03-29T11:32:47.853526+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:42.190121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.860138+0000","last_change":"2024-03-29T11:32:32.422243+0000","last_active":"2024-03-29T11:32:47.860138+0000","last_peered":"2024-03-29T11:32:47.860138+0000","last_clean":"2024-03-29T11:32:47.860138+0000","last_became_active":"2024-03-29T11:32:32.421949+0000","last_became_peered":"2024-03-29T11:32:32.421949+0000","last_unstale":"2024-03-29T11:32:47.860138+0000","last_undegraded":"2024-03-29T11:32:47.860138+0000","last_fullsized":"2024-03-29T11:32:47.860138+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:03.995789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.866667+0000","last_change":"2024-03-29T11:32:47.883873+0000","last_active":"2024-03-29T11:32:48.866667+0000","last_peered":"2024-03-29T11:32:48.866667+0000","last_clean":"2024-03-29T11:32:48.866667+0000","last_became_active":"2024-03-29T11:32:47.883381+0000","last_became_peered":"2024-03-29T11:32:47.883381+0000","last_unstale":"2024-03-29T11:32:48.866667+0000","last_undegraded":"2024-03-29T11:32:48.866667+0000","last_fullsized":"2024-03-29T11:32:48.866667+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:08.340482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861202+0000","last_change":"2024-03-29T11:32:32.962313+0000","last_active":"2024-03-29T11:32:47.861202+0000","last_peered":"2024-03-29T11:32:47.861202+0000","last_clean":"2024-03-29T11:32:47.861202+0000","last_became_active":"2024-03-29T11:32:32.961887+0000","last_became_peered":"2024-03-29T11:32:32.961887+0000","last_unstale":"2024-03-29T11:32:47.861202+0000","last_undegraded":"2024-03-29T11:32:47.861202+0000","last_fullsized":"2024-03-29T11:32:47.861202+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:18.696940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"247'4","reported_seq":19,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.868945+0000","last_change":"2024-03-29T11:32:32.419906+0000","last_active":"2024-03-29T11:32:47.868945+0000","last_peered":"2024-03-29T11:32:47.868945+0000","last_clean":"2024-03-29T11:32:47.868945+0000","last_became_active":"2024-03-29T11:32:32.419653+0000","last_became_peered":"2024-03-29T11:32:32.419653+0000","last_unstale":"2024-03-29T11:32:47.868945+0000","last_undegraded":"2024-03-29T11:32:47.868945+0000","last_fullsized":"2024-03-29T11:32:47.868945+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:22.034114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"233'388","reported_seq":1371,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.877658+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:32:47.877658+0000","last_peered":"2024-03-29T11:32:47.877658+0000","last_clean":"2024-03-29T11:32:47.877658+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:32:47.877658+0000","last_undegraded":"2024-03-29T11:32:47.877658+0000","last_fullsized":"2024-03-29T11:32:47.877658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":388,"ondisk_log_size":388,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576009,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56944,"kb_used_data":3684,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314896,"statfs":{"total":96636764160,"available":96578453504,"internally_reserved":0,"allocated":3772416,"data_stored":2149430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.70799999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.73899999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42899999999999999},"max":{"1min":0.89900000000000002,"5min":0.98899999999999999,"15min":1.119},"last":0.76800000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.69299999999999995,"15min":0.72199999999999998},"min":{"1min":0.437,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90500000000000003,"5min":0.97399999999999998,"15min":1.478},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72799999999999998,"15min":0.75800000000000001},"min":{"1min":0.47399999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.393},"last":0.75}]},{"osd":2,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70599999999999996,"15min":0.748},"min":{"1min":0.504,"5min":0.436,"15min":0.436},"max":{"1min":0.85899999999999999,"5min":0.998,"15min":1.089},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92100000000000004,"5min":1.032,"15min":1.0720000000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.72299999999999998,"15min":0.753},"min":{"1min":0.54400000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90300000000000002,"5min":0.97499999999999998,"15min":1.0580000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77500000000000002,"15min":0.78300000000000003},"min":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.55300000000000005},"max":{"1min":1.0309999999999999,"5min":1.071,"15min":1.071},"last":0.79200000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":1.3779999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65900000000000003,"15min":0.66900000000000004},"min":{"1min":0.50900000000000001,"5min":0.41599999999999998,"15min":0.40899999999999997},"max":{"1min":0.85099999999999998,"5min":0.93500000000000005,"15min":1.335},"last":0.81699999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69799999999999995,"15min":0.68100000000000005},"min":{"1min":0.434,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.1160000000000001},"last":0.495},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.73899999999999999,"15min":0.72199999999999998},"min":{"1min":0.55600000000000005,"5min":0.50900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95499999999999996,"5min":1.0169999999999999,"15min":1.0249999999999999},"last":0.54200000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.76900000000000002,"15min":0.76700000000000002},"min":{"1min":0.52400000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0549999999999999,"15min":1.1319999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.755,"15min":0.74099999999999999},"min":{"1min":0.46899999999999997,"5min":0.439,"15min":0.435},"max":{"1min":1.0580000000000001,"5min":1.071,"15min":1.071},"last":0.64500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834576010,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48176,"kb_used_data":3100,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323664,"statfs":{"total":96636764160,"available":96587431936,"internally_reserved":0,"allocated":3174400,"data_stored":1552399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69699999999999995,"15min":0.72899999999999998},"min":{"1min":0.48699999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.74299999999999999,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70899999999999996,"15min":0.751},"min":{"1min":0.40200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95199999999999996,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.70399999999999996,"15min":0.73099999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.022,"5min":1.98,"15min":1.98},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72899999999999998,"15min":0.76400000000000001},"min":{"1min":0.505,"5min":0.435,"15min":0.435},"max":{"1min":1.012,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.745}]},{"osd":2,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.72599999999999998,"15min":0.746},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92800000000000005,"5min":1.669,"15min":1.669},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.73599999999999999,"15min":0.75800000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.46200000000000002},"max":{"1min":1,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.505}]},{"osd":3,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.73599999999999999,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.997,"5min":2.52,"15min":2.52},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77900000000000003,"15min":0.79800000000000004},"min":{"1min":0.53800000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.105,"5min":1.877,"15min":1.877},"last":0.70199999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63600000000000001,"15min":0.61399999999999999},"min":{"1min":0.45200000000000001,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":0.92300000000000004,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.62},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.56100000000000005,"5min":0.47399999999999998,"15min":0.45900000000000002},"max":{"1min":1.2709999999999999,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":0.72499999999999998}]},{"osd":5,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.66000000000000003},"min":{"1min":0.42799999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0900000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.72199999999999998,"15min":0.71799999999999997},"min":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.42199999999999999},"max":{"1min":1.0429999999999999,"5min":1.925,"15min":1.925},"last":0.56399999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.78000000000000003,"15min":0.76900000000000002},"min":{"1min":0.66800000000000004,"5min":0.54800000000000004,"15min":0.52700000000000002},"max":{"1min":1.0509999999999999,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.751,"15min":0.74399999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1220000000000001,"5min":1.726,"15min":1.726},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576008,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60484,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311356,"statfs":{"total":96636764160,"available":96574828544,"internally_reserved":0,"allocated":3194880,"data_stored":1555388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.44500000000000001,"15min":0.46700000000000003},"min":{"1min":0.29299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69799999999999995,"5min":0.79100000000000004,"15min":0.80300000000000005},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.50900000000000001,"15min":0.53200000000000003},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.57199999999999995,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.56200000000000006},"min":{"1min":0.36499999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.52300000000000002,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.34799999999999998},"max":{"1min":0.61599999999999999,"5min":0.81399999999999995,"15min":0.90300000000000002},"last":0.45400000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.59599999999999997},"min":{"1min":0.433,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73099999999999998,"5min":0.86199999999999999,"15min":0.995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56799999999999995,"15min":0.58099999999999996},"min":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.83999999999999997,"15min":0.878},"last":0.42699999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.75,"15min":0.78700000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.056,"5min":1.153,"15min":1.3660000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.76200000000000001,"15min":0.80000000000000004},"min":{"1min":0.56299999999999994,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97699999999999998,"5min":1.226,"15min":1.304},"last":0.81899999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.79300000000000004,"15min":0.81999999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":0.997,"5min":1.276,"15min":1.3720000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.099,"5min":1.1299999999999999,"15min":1.3520000000000001},"last":0.90600000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.78300000000000003,"15min":0.81899999999999995},"min":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.026,"5min":1.1890000000000001,"15min":1.1919999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.042,"5min":1.2250000000000001,"15min":1.3069999999999999},"last":0.877}]},{"osd":7,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.83999999999999997,"15min":0.86899999999999999},"min":{"1min":0.54800000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0649999999999999,"5min":1.327,"15min":1.3540000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83399999999999996,"15min":0.86799999999999999},"min":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0700000000000001,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.78300000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576010,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61028,"kb_used_data":3664,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310812,"statfs":{"total":96636764160,"available":96574271488,"internally_reserved":0,"allocated":3751936,"data_stored":2141066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44900000000000001,"15min":0.46100000000000002},"min":{"1min":0.317,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.623,"5min":0.95899999999999996,"15min":1.087},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.52200000000000002,"15min":0.53500000000000003},"min":{"1min":0.32400000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67200000000000004,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.374}]},{"osd":2,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56000000000000005,"15min":0.57899999999999996},"min":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66600000000000004,"5min":1.028,"15min":1.5289999999999999},"last":0.437},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.499,"15min":0.51600000000000001},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.629,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55600000000000005,"15min":0.56200000000000006},"min":{"1min":0.38500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77600000000000002,"5min":1.222,"15min":1.222},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.622,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.44,"15min":0.42199999999999999},"max":{"1min":0.81799999999999995,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.56599999999999995}]},{"osd":4,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80500000000000005,"15min":0.81399999999999995},"min":{"1min":0.56899999999999995,"5min":0.55600000000000005,"15min":0.495},"max":{"1min":0.93400000000000005,"5min":1.248,"15min":1.248},"last":0.876},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82399999999999995,"15min":0.84599999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0309999999999999,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.89800000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76800000000000002,"15min":0.78500000000000003},"min":{"1min":0.54500000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.88900000000000001,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80400000000000005,"15min":0.81899999999999995},"min":{"1min":0.58999999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94899999999999995,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.80100000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.83399999999999996,"15min":0.85299999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.97599999999999998,"5min":1.343,"15min":1.6240000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.85699999999999998,"15min":0.88200000000000001},"min":{"1min":0.62,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.004,"5min":1.3240000000000001,"15min":1.337},"last":0.96399999999999997}]},{"osd":7,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79900000000000004,"15min":0.82399999999999995},"min":{"1min":0.56899999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.99099999999999999,"5min":1.115,"15min":1.214},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.84099999999999997,"15min":0.86899999999999999},"min":{"1min":0.55500000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.054,"5min":1.365,"15min":1.365},"last":0.93500000000000005}]}]},{"osd":2,"up_from":13,"seq":55834576009,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3162112,"data_stored":1551773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47799999999999998,"15min":0.47599999999999998},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73199999999999998,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.502},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.54100000000000004},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67600000000000005,"5min":0.73999999999999999,"15min":0.96799999999999997},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74299999999999999,"5min":0.77400000000000002,"15min":0.80700000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55500000000000005,"15min":0.57699999999999996},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68300000000000005,"5min":0.872,"15min":0.88800000000000001},"last":0.54100000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.42299999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86799999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76800000000000002,"5min":1.038,"15min":1.038},"last":0.47099999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.76300000000000001,"15min":0.80300000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.93700000000000006,"5min":1.091,"15min":1.1970000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.755,"15min":0.80800000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.52400000000000002},"max":{"1min":1.026,"5min":1.244,"15min":1.244},"last":0.64500000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.81299999999999994},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.53400000000000003},"max":{"1min":0.94599999999999995,"5min":1.1599999999999999,"15min":1.256},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.78000000000000003,"15min":0.82199999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.995,"5min":1.143,"15min":1.3140000000000001},"last":0.66000000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.79300000000000004,"15min":0.84199999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98099999999999998,"5min":1.1459999999999999,"15min":1.24},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79700000000000004,"15min":0.83999999999999997},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.97399999999999998,"5min":1.099,"15min":1.329},"last":0.63500000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.83899999999999997,"15min":0.88800000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":0.95899999999999996,"5min":1.1910000000000001,"15min":1.2889999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85499999999999998,"15min":0.90200000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.56899999999999995},"max":{"1min":1.095,"5min":1.2969999999999999,"15min":1.304},"last":0.81399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834576009,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48180,"kb_used_data":3104,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323660,"statfs":{"total":96636764160,"available":96587427840,"internally_reserved":0,"allocated":3178496,"data_stored":1554411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.29599999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.33200000000000002,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48399999999999999,"15min":0.49399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.57199999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.38700000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.503}]},{"osd":4,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79600000000000004,"15min":0.79700000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.80300000000000005,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80500000000000005,"15min":0.81299999999999994},"min":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.502},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.83099999999999996,"15min":0.82699999999999996},"min":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.54600000000000004},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":0.73399999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86599999999999999,"15min":0.85899999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.84099999999999997,"15min":0.84299999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":0.66800000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86699999999999999,"15min":0.86599999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.58099999999999996},"max":{"1min":2.444,"5min":2.444,"15min":2.444},"last":0.85399999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576010,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60484,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311356,"statfs":{"total":96636764160,"available":96574828544,"internally_reserved":0,"allocated":3198976,"data_stored":1556200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.66400000000000003,"15min":0.71499999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83899999999999997,"5min":0.95499999999999996,"15min":1.0509999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.67500000000000004,"15min":0.72099999999999997},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.93600000000000005,"5min":0.99299999999999999,"15min":1.1459999999999999},"last":0.42299999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.68300000000000005,"15min":0.73699999999999999},"min":{"1min":0.436,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69199999999999995,"15min":0.73799999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91400000000000003,"5min":0.94899999999999995,"15min":1.002},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.77900000000000003,"15min":0.82199999999999995},"min":{"1min":0.55400000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.004,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.76500000000000001,"15min":0.80600000000000005},"min":{"1min":0.503,"5min":0.496,"15min":0.496},"max":{"1min":1.0029999999999999,"5min":1.224,"15min":1.224},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71399999999999997,"15min":0.75900000000000001},"min":{"1min":0.54000000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.92600000000000005,"5min":1.0580000000000001,"15min":1.181},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.74399999999999999,"15min":0.76600000000000001},"min":{"1min":0.499,"5min":0.499,"15min":0.48699999999999999},"max":{"1min":0.95199999999999996,"5min":1.01,"15min":1.2490000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63800000000000001,"15min":0.625},"min":{"1min":0.45400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64500000000000002,"15min":0.65300000000000002},"min":{"1min":0.52000000000000002,"5min":0.41599999999999998,"15min":0.39700000000000002},"max":{"1min":0.81499999999999995,"5min":1.0920000000000001,"15min":1.169},"last":0.5}]},{"osd":6,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.50600000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":0.93100000000000005,"5min":1.056,"15min":1.0800000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.72199999999999998,"15min":0.71999999999999997},"min":{"1min":0.59199999999999997,"5min":0.48199999999999998,"15min":0.47699999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.56399999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73599999999999999,"15min":0.73299999999999998},"min":{"1min":0.60599999999999998,"5min":0.501,"15min":0.436},"max":{"1min":0.97999999999999998,"5min":1.0389999999999999,"15min":1.2669999999999999},"last":0.626},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81200000000000006,"15min":0.82199999999999995},"min":{"1min":0.65900000000000003,"5min":0.58499999999999996,"15min":0.54400000000000004},"max":{"1min":1.03,"5min":1.1619999999999999,"15min":1.2969999999999999},"last":0.61299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576009,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46488,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325352,"statfs":{"total":96636764160,"available":96589160448,"internally_reserved":0,"allocated":3158016,"data_stored":1551683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.69799999999999995},"min":{"1min":0.47199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.72299999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.95599999999999996,"15min":1.0680000000000001},"last":0.67300000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73699999999999999,"15min":0.75700000000000001},"min":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98099999999999998,"5min":1.839,"15min":1.839},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.71899999999999997,"15min":0.73599999999999999},"min":{"1min":0.504,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.282,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.64800000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.753,"15min":0.755},"min":{"1min":0.51000000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.625},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.71499999999999997,"15min":0.74399999999999999},"min":{"1min":0.54400000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.76100000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.48499999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.95099999999999996,"5min":3.444,"15min":3.444},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.75800000000000001,"15min":0.77200000000000002},"min":{"1min":0.54400000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.72899999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.629,"15min":0.60899999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38700000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67800000000000005,"15min":0.65300000000000002},"min":{"1min":0.47799999999999998,"5min":0.41699999999999998,"15min":0.40600000000000003},"max":{"1min":0.999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.48999999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.69699999999999995,"15min":0.67000000000000004},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72199999999999998,"15min":0.70299999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.107},"last":0.51700000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.746,"15min":0.72599999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.42299999999999999},"max":{"1min":1.04,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79600000000000004,"15min":0.77400000000000002},"min":{"1min":0.53500000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:32:52.847 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:32:53.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.082+0000 7fe271800700 1 Processor -- start 2024-03-29T11:32:53.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.082+0000 7fe271800700 1 -- start start 2024-03-29T11:32:53.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.083+0000 7fe271800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe26c15d640 0x7fe26c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.083+0000 7fe271800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe26c0ceec0 0x7fe26c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.083+0000 7fe271800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe26c161800 0x7fe26c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.083+0000 7fe271800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c161c90 con 0x7fe26c15d640 2024-03-29T11:32:53.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.083+0000 7fe271800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c0d30d0 con 0x7fe26c161800 2024-03-29T11:32:53.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.083+0000 7fe271800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c0b6840 con 0x7fe26c0ceec0 2024-03-29T11:32:53.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.083+0000 7fe26aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe26c15d640 0x7fe26c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe26aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe26c15d640 0x7fe26c0d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46358/0 (socket says 172.21.15.12:46358) 2024-03-29T11:32:53.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe26aa00700 1 -- 172.21.15.12:0/3299817977 learned_addr learned my addr 172.21.15.12:0/3299817977 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:53.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe26b400700 1 --2- 172.21.15.12:0/3299817977 >> v2:172.21.15.159:3300/0 conn(0x7fe26c161800 0x7fe26c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe26a000700 1 --2- 172.21.15.12:0/3299817977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe26c0ceec0 0x7fe26c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe26aa00700 1 -- 172.21.15.12:0/3299817977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe26c0ceec0 msgr2=0x7fe26c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:53.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe26aa00700 1 --2- 172.21.15.12:0/3299817977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe26c0ceec0 0x7fe26c1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe26aa00700 1 -- 172.21.15.12:0/3299817977 >> v2:172.21.15.159:3300/0 conn(0x7fe26c161800 msgr2=0x7fe26c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:53.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe26aa00700 1 --2- 172.21.15.12:0/3299817977 >> v2:172.21.15.159:3300/0 conn(0x7fe26c161800 0x7fe26c0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe26aa00700 1 -- 172.21.15.12:0/3299817977 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c0d2880 con 0x7fe26c15d640 2024-03-29T11:32:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe26aa00700 1 --2- 172.21.15.12:0/3299817977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c15d640 0x7fe26c0d2340 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0c7050 tx=0x7fe25c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554be269fccd0ebb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.084+0000 7fe269600700 1 -- 172.21.15.12:0/3299817977 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00b870 con 0x7fe26c15d640 2024-03-29T11:32:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe269600700 1 -- 172.21.15.12:0/3299817977 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c00b9d0 con 0x7fe26c15d640 2024-03-29T11:32:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe269600700 1 -- 172.21.15.12:0/3299817977 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00bd60 con 0x7fe26c15d640 2024-03-29T11:32:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe271800700 1 -- 172.21.15.12:0/3299817977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c15d640 msgr2=0x7fe26c0d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe271800700 1 --2- 172.21.15.12:0/3299817977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c15d640 0x7fe26c0d2340 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0c7050 tx=0x7fe25c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe271800700 1 -- 172.21.15.12:0/3299817977 shutdown_connections 2024-03-29T11:32:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe271800700 1 --2- 172.21.15.12:0/3299817977 >> v2:172.21.15.159:3300/0 conn(0x7fe26c161800 0x7fe26c0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe271800700 1 --2- 172.21.15.12:0/3299817977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c15d640 0x7fe26c0d2340 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe271800700 1 --2- 172.21.15.12:0/3299817977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe26c0ceec0 0x7fe26c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe271800700 1 -- 172.21.15.12:0/3299817977 >> 172.21.15.12:0/3299817977 conn(0x7fe26c08aff0 msgr2=0x7fe26c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:53.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe271800700 1 -- 172.21.15.12:0/3299817977 shutdown_connections 2024-03-29T11:32:53.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.085+0000 7fe271800700 1 -- 172.21.15.12:0/3299817977 wait complete. 2024-03-29T11:32:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.087+0000 7fe271800700 1 Processor -- start 2024-03-29T11:32:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.087+0000 7fe271800700 1 -- start start 2024-03-29T11:32:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.088+0000 7fe271800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c0ceec0 0x7fe26c1600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.088+0000 7fe271800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe26c15d640 0x7fe26c1f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.088+0000 7fe271800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe26c161800 0x7fe26c148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.088+0000 7fe271800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c1484d0 con 0x7fe26c0ceec0 2024-03-29T11:32:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.088+0000 7fe271800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c1478c0 con 0x7fe26c161800 2024-03-29T11:32:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.088+0000 7fe271800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c147ba0 con 0x7fe26c15d640 2024-03-29T11:32:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.088+0000 7fe26aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c0ceec0 0x7fe26c1600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.088+0000 7fe26aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c0ceec0 0x7fe26c1600b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46374/0 (socket says 172.21.15.12:46374) 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.088+0000 7fe26aa00700 1 -- 172.21.15.12:0/3839993315 learned_addr learned my addr 172.21.15.12:0/3839993315 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.088+0000 7fe26a000700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe26c15d640 0x7fe26c1f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe26b400700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe26c161800 0x7fe26c148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe26aa00700 1 -- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe26c15d640 msgr2=0x7fe26c1f2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe26aa00700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe26c15d640 0x7fe26c1f2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe26aa00700 1 -- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe26c161800 msgr2=0x7fe26c148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe26aa00700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe26c161800 0x7fe26c148960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe26aa00700 1 -- 172.21.15.12:0/3839993315 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c009550 con 0x7fe26c0ceec0 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe26b400700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe26c161800 0x7fe26c148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe26aa00700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c0ceec0 0x7fe26c1600b0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fe25c000c00 tx=0x7fe25c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe262a00700 1 -- 172.21.15.12:0/3839993315 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c004540 con 0x7fe26c0ceec0 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe262a00700 1 -- 172.21.15.12:0/3839993315 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c0036a0 con 0x7fe26c0ceec0 2024-03-29T11:32:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe271800700 1 -- 172.21.15.12:0/3839993315 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c147e80 con 0x7fe26c0ceec0 2024-03-29T11:32:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe262a00700 1 -- 172.21.15.12:0/3839993315 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c003a50 con 0x7fe26c0ceec0 2024-03-29T11:32:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.089+0000 7fe271800700 1 -- 172.21.15.12:0/3839993315 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c1605f0 con 0x7fe26c0ceec0 2024-03-29T11:32:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.091+0000 7fe271800700 1 -- 172.21.15.12:0/3839993315 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c067720 con 0x7fe26c0ceec0 2024-03-29T11:32:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.091+0000 7fe262a00700 1 -- 172.21.15.12:0/3839993315 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe25c021070 con 0x7fe26c0ceec0 2024-03-29T11:32:53.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.092+0000 7fe262a00700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24c06aed0 0x7fe24c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.092+0000 7fe262a00700 1 -- 172.21.15.12:0/3839993315 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7fe25c08b0f0 con 0x7fe26c0ceec0 2024-03-29T11:32:53.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.093+0000 7fe26a000700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24c06aed0 0x7fe24c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.093+0000 7fe26a000700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24c06aed0 0x7fe24c06d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe26c14da80 tx=0x7fe258008040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.097+0000 7fe262a00700 1 -- 172.21.15.12:0/3839993315 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe25c055d70 con 0x7fe26c0ceec0 2024-03-29T11:32:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.306+0000 7fe271800700 1 -- 172.21.15.12:0/3839993315 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe26c0b64d0 con 0x7fe24c06aed0 2024-03-29T11:32:53.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.321+0000 7fe262a00700 1 -- 172.21.15.12:0/3839993315 <== mgr.11050 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528549 (secure 0 0 0) 0x7fe25800be50 con 0x7fe24c06aed0 2024-03-29T11:32:53.318 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:53.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 -- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24c06aed0 msgr2=0x7fe24c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24c06aed0 0x7fe24c06d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe26c14da80 tx=0x7fe258008040 comp rx=0 tx=0).stop 2024-03-29T11:32:53.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 -- 172.21.15.12:0/3839993315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c0ceec0 msgr2=0x7fe26c1600b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c0ceec0 0x7fe26c1600b0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fe25c000c00 tx=0x7fe25c004750 comp rx=0 tx=0).stop 2024-03-29T11:32:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 -- 172.21.15.12:0/3839993315 shutdown_connections 2024-03-29T11:32:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe24c06aed0 0x7fe24c06d390 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe26c161800 0x7fe26c148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe26c0ceec0 0x7fe26c1600b0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 --2- 172.21.15.12:0/3839993315 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe26c15d640 0x7fe26c1f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 -- 172.21.15.12:0/3839993315 >> 172.21.15.12:0/3839993315 conn(0x7fe26c08aff0 msgr2=0x7fe26c0d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 -- 172.21.15.12:0/3839993315 shutdown_connections 2024-03-29T11:32:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.327+0000 7fe271800700 1 -- 172.21.15.12:0/3839993315 wait complete. 2024-03-29T11:32:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:32:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-03-29T11:32:52.533751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596333,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":742,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":467,"ondisk_log_size":467,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":321,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":434036,"kb_used_data":25968,"kb_used_omap":12,"kb_used_meta":407923,"kb_avail":754540684,"statfs":{"total":773094113280,"available":772649660416,"internally_reserved":0,"allocated":26591232,"data_stored":13612350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":417713721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001980"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193329+0000","last_change":"2024-03-29T11:32:47.880663+0000","last_active":"2024-03-29T11:32:49.193329+0000","last_peered":"2024-03-29T11:32:49.193329+0000","last_clean":"2024-03-29T11:32:49.193329+0000","last_became_active":"2024-03-29T11:32:47.879936+0000","last_became_peered":"2024-03-29T11:32:47.879936+0000","last_unstale":"2024-03-29T11:32:49.193329+0000","last_undegraded":"2024-03-29T11:32:49.193329+0000","last_fullsized":"2024-03-29T11:32:49.193329+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:57.835090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.337907+0000","last_change":"2024-03-29T11:32:47.340805+0000","last_active":"2024-03-29T11:32:49.337907+0000","last_peered":"2024-03-29T11:32:49.337907+0000","last_clean":"2024-03-29T11:32:49.337907+0000","last_became_active":"2024-03-29T11:32:47.340542+0000","last_became_peered":"2024-03-29T11:32:47.340542+0000","last_unstale":"2024-03-29T11:32:49.337907+0000","last_undegraded":"2024-03-29T11:32:49.337907+0000","last_fullsized":"2024-03-29T11:32:49.337907+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:51:06.069753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854251+0000","last_change":"2024-03-29T11:32:32.953487+0000","last_active":"2024-03-29T11:32:47.854251+0000","last_peered":"2024-03-29T11:32:47.854251+0000","last_clean":"2024-03-29T11:32:47.854251+0000","last_became_active":"2024-03-29T11:32:32.953243+0000","last_became_peered":"2024-03-29T11:32:32.953243+0000","last_unstale":"2024-03-29T11:32:47.854251+0000","last_undegraded":"2024-03-29T11:32:47.854251+0000","last_fullsized":"2024-03-29T11:32:47.854251+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:11.369063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.861659+0000","last_change":"2024-03-29T11:32:47.881569+0000","last_active":"2024-03-29T11:32:48.861659+0000","last_peered":"2024-03-29T11:32:48.861659+0000","last_clean":"2024-03-29T11:32:48.861659+0000","last_became_active":"2024-03-29T11:32:47.881243+0000","last_became_peered":"2024-03-29T11:32:47.881243+0000","last_unstale":"2024-03-29T11:32:48.861659+0000","last_undegraded":"2024-03-29T11:32:48.861659+0000","last_fullsized":"2024-03-29T11:32:48.861659+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:08.340482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862041+0000","last_change":"2024-03-29T11:32:47.345546+0000","last_active":"2024-03-29T11:32:48.862041+0000","last_peered":"2024-03-29T11:32:48.862041+0000","last_clean":"2024-03-29T11:32:48.862041+0000","last_became_active":"2024-03-29T11:32:47.344388+0000","last_became_peered":"2024-03-29T11:32:47.344388+0000","last_unstale":"2024-03-29T11:32:48.862041+0000","last_undegraded":"2024-03-29T11:32:48.862041+0000","last_fullsized":"2024-03-29T11:32:48.862041+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:25:03.789446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.856105+0000","last_change":"2024-03-29T11:32:32.956013+0000","last_active":"2024-03-29T11:32:47.856105+0000","last_peered":"2024-03-29T11:32:47.856105+0000","last_clean":"2024-03-29T11:32:47.856105+0000","last_became_active":"2024-03-29T11:32:32.954012+0000","last_became_peered":"2024-03-29T11:32:32.954012+0000","last_unstale":"2024-03-29T11:32:47.856105+0000","last_undegraded":"2024-03-29T11:32:47.856105+0000","last_fullsized":"2024-03-29T11:32:47.856105+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:02:36.221843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862151+0000","last_change":"2024-03-29T11:32:47.878275+0000","last_active":"2024-03-29T11:32:48.862151+0000","last_peered":"2024-03-29T11:32:48.862151+0000","last_clean":"2024-03-29T11:32:48.862151+0000","last_became_active":"2024-03-29T11:32:47.878041+0000","last_became_peered":"2024-03-29T11:32:47.878041+0000","last_unstale":"2024-03-29T11:32:48.862151+0000","last_undegraded":"2024-03-29T11:32:48.862151+0000","last_fullsized":"2024-03-29T11:32:48.862151+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:19:11.800767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":10,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.860528+0000","last_change":"2024-03-29T11:32:47.351818+0000","last_active":"2024-03-29T11:32:47.860528+0000","last_peered":"2024-03-29T11:32:47.860528+0000","last_clean":"2024-03-29T11:32:47.860528+0000","last_became_active":"2024-03-29T11:32:47.351082+0000","last_became_peered":"2024-03-29T11:32:47.351082+0000","last_unstale":"2024-03-29T11:32:47.860528+0000","last_undegraded":"2024-03-29T11:32:47.860528+0000","last_fullsized":"2024-03-29T11:32:47.860528+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:08:10.182655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.867842+0000","last_change":"2024-03-29T11:32:32.966647+0000","last_active":"2024-03-29T11:32:47.867842+0000","last_peered":"2024-03-29T11:32:47.867842+0000","last_clean":"2024-03-29T11:32:47.867842+0000","last_became_active":"2024-03-29T11:32:32.966394+0000","last_became_peered":"2024-03-29T11:32:32.966394+0000","last_unstale":"2024-03-29T11:32:47.867842+0000","last_undegraded":"2024-03-29T11:32:47.867842+0000","last_fullsized":"2024-03-29T11:32:47.867842+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:22.600133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862318+0000","last_change":"2024-03-29T11:32:47.884486+0000","last_active":"2024-03-29T11:32:48.862318+0000","last_peered":"2024-03-29T11:32:48.862318+0000","last_clean":"2024-03-29T11:32:48.862318+0000","last_became_active":"2024-03-29T11:32:47.884250+0000","last_became_peered":"2024-03-29T11:32:47.884250+0000","last_unstale":"2024-03-29T11:32:48.862318+0000","last_undegraded":"2024-03-29T11:32:48.862318+0000","last_fullsized":"2024-03-29T11:32:48.862318+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:30.885593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":10,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861560+0000","last_change":"2024-03-29T11:32:47.350253+0000","last_active":"2024-03-29T11:32:47.861560+0000","last_peered":"2024-03-29T11:32:47.861560+0000","last_clean":"2024-03-29T11:32:47.861560+0000","last_became_active":"2024-03-29T11:32:47.349972+0000","last_became_peered":"2024-03-29T11:32:47.349972+0000","last_unstale":"2024-03-29T11:32:47.861560+0000","last_undegraded":"2024-03-29T11:32:47.861560+0000","last_fullsized":"2024-03-29T11:32:47.861560+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:18.636533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.865924+0000","last_change":"2024-03-29T11:32:32.955745+0000","last_active":"2024-03-29T11:32:47.865924+0000","last_peered":"2024-03-29T11:32:47.865924+0000","last_clean":"2024-03-29T11:32:47.865924+0000","last_became_active":"2024-03-29T11:32:32.955142+0000","last_became_peered":"2024-03-29T11:32:32.955142+0000","last_unstale":"2024-03-29T11:32:47.865924+0000","last_undegraded":"2024-03-29T11:32:47.865924+0000","last_fullsized":"2024-03-29T11:32:47.865924+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:42.190121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193940+0000","last_change":"2024-03-29T11:32:47.869332+0000","last_active":"2024-03-29T11:32:49.193940+0000","last_peered":"2024-03-29T11:32:49.193940+0000","last_clean":"2024-03-29T11:32:49.193940+0000","last_became_active":"2024-03-29T11:32:47.868665+0000","last_became_peered":"2024-03-29T11:32:47.868665+0000","last_unstale":"2024-03-29T11:32:49.193940+0000","last_undegraded":"2024-03-29T11:32:49.193940+0000","last_fullsized":"2024-03-29T11:32:49.193940+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:37.965088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338792+0000","last_change":"2024-03-29T11:32:47.340906+0000","last_active":"2024-03-29T11:32:49.338792+0000","last_peered":"2024-03-29T11:32:49.338792+0000","last_clean":"2024-03-29T11:32:49.338792+0000","last_became_active":"2024-03-29T11:32:47.340587+0000","last_became_peered":"2024-03-29T11:32:47.340587+0000","last_unstale":"2024-03-29T11:32:49.338792+0000","last_undegraded":"2024-03-29T11:32:49.338792+0000","last_fullsized":"2024-03-29T11:32:49.338792+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:34:59.271199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861395+0000","last_change":"2024-03-29T11:32:32.950908+0000","last_active":"2024-03-29T11:32:47.861395+0000","last_peered":"2024-03-29T11:32:47.861395+0000","last_clean":"2024-03-29T11:32:47.861395+0000","last_became_active":"2024-03-29T11:32:32.950581+0000","last_became_peered":"2024-03-29T11:32:32.950581+0000","last_unstale":"2024-03-29T11:32:47.861395+0000","last_undegraded":"2024-03-29T11:32:47.861395+0000","last_fullsized":"2024-03-29T11:32:47.861395+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:11.495727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193750+0000","last_change":"2024-03-29T11:32:47.869312+0000","last_active":"2024-03-29T11:32:49.193750+0000","last_peered":"2024-03-29T11:32:49.193750+0000","last_clean":"2024-03-29T11:32:49.193750+0000","last_became_active":"2024-03-29T11:32:47.868448+0000","last_became_peered":"2024-03-29T11:32:47.868448+0000","last_unstale":"2024-03-29T11:32:49.193750+0000","last_undegraded":"2024-03-29T11:32:49.193750+0000","last_fullsized":"2024-03-29T11:32:49.193750+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:25:04.792440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867909+0000","last_change":"2024-03-29T11:32:47.341720+0000","last_active":"2024-03-29T11:32:48.867909+0000","last_peered":"2024-03-29T11:32:48.867909+0000","last_clean":"2024-03-29T11:32:48.867909+0000","last_became_active":"2024-03-29T11:32:47.341458+0000","last_became_peered":"2024-03-29T11:32:47.341458+0000","last_unstale":"2024-03-29T11:32:48.867909+0000","last_undegraded":"2024-03-29T11:32:48.867909+0000","last_fullsized":"2024-03-29T11:32:48.867909+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:36.518778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.863047+0000","last_change":"2024-03-29T11:32:32.966829+0000","last_active":"2024-03-29T11:32:47.863047+0000","last_peered":"2024-03-29T11:32:47.863047+0000","last_clean":"2024-03-29T11:32:47.863047+0000","last_became_active":"2024-03-29T11:32:32.966561+0000","last_became_peered":"2024-03-29T11:32:32.966561+0000","last_unstale":"2024-03-29T11:32:47.863047+0000","last_undegraded":"2024-03-29T11:32:47.863047+0000","last_fullsized":"2024-03-29T11:32:47.863047+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.339167+0000","last_change":"2024-03-29T11:32:47.870380+0000","last_active":"2024-03-29T11:32:49.339167+0000","last_peered":"2024-03-29T11:32:49.339167+0000","last_clean":"2024-03-29T11:32:49.339167+0000","last_became_active":"2024-03-29T11:32:47.869954+0000","last_became_peered":"2024-03-29T11:32:47.869954+0000","last_unstale":"2024-03-29T11:32:49.339167+0000","last_undegraded":"2024-03-29T11:32:49.339167+0000","last_fullsized":"2024-03-29T11:32:49.339167+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:52:34.148222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193942+0000","last_change":"2024-03-29T11:32:47.337485+0000","last_active":"2024-03-29T11:32:49.193942+0000","last_peered":"2024-03-29T11:32:49.193942+0000","last_clean":"2024-03-29T11:32:49.193942+0000","last_became_active":"2024-03-29T11:32:47.336790+0000","last_became_peered":"2024-03-29T11:32:47.336790+0000","last_unstale":"2024-03-29T11:32:49.193942+0000","last_undegraded":"2024-03-29T11:32:49.193942+0000","last_fullsized":"2024-03-29T11:32:49.193942+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:41.434503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862285+0000","last_change":"2024-03-29T11:32:32.956044+0000","last_active":"2024-03-29T11:32:47.862285+0000","last_peered":"2024-03-29T11:32:47.862285+0000","last_clean":"2024-03-29T11:32:47.862285+0000","last_became_active":"2024-03-29T11:32:32.955298+0000","last_became_peered":"2024-03-29T11:32:32.955298+0000","last_unstale":"2024-03-29T11:32:47.862285+0000","last_undegraded":"2024-03-29T11:32:47.862285+0000","last_fullsized":"2024-03-29T11:32:47.862285+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:36.536991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862599+0000","last_change":"2024-03-29T11:32:47.880947+0000","last_active":"2024-03-29T11:32:48.862599+0000","last_peered":"2024-03-29T11:32:48.862599+0000","last_clean":"2024-03-29T11:32:48.862599+0000","last_became_active":"2024-03-29T11:32:47.879746+0000","last_became_peered":"2024-03-29T11:32:47.879746+0000","last_unstale":"2024-03-29T11:32:48.862599+0000","last_undegraded":"2024-03-29T11:32:48.862599+0000","last_fullsized":"2024-03-29T11:32:48.862599+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:01.324372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862895+0000","last_change":"2024-03-29T11:32:47.350151+0000","last_active":"2024-03-29T11:32:48.862895+0000","last_peered":"2024-03-29T11:32:48.862895+0000","last_clean":"2024-03-29T11:32:48.862895+0000","last_became_active":"2024-03-29T11:32:47.349871+0000","last_became_peered":"2024-03-29T11:32:47.349871+0000","last_unstale":"2024-03-29T11:32:48.862895+0000","last_undegraded":"2024-03-29T11:32:48.862895+0000","last_fullsized":"2024-03-29T11:32:48.862895+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:53.214603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.337071+0000","last_change":"2024-03-29T11:32:32.956054+0000","last_active":"2024-03-29T11:32:47.337071+0000","last_peered":"2024-03-29T11:32:47.337071+0000","last_clean":"2024-03-29T11:32:47.337071+0000","last_became_active":"2024-03-29T11:32:32.952773+0000","last_became_peered":"2024-03-29T11:32:32.952773+0000","last_unstale":"2024-03-29T11:32:47.337071+0000","last_undegraded":"2024-03-29T11:32:47.337071+0000","last_fullsized":"2024-03-29T11:32:47.337071+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:59:43.666251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.870319+0000","last_change":"2024-03-29T11:32:47.866242+0000","last_active":"2024-03-29T11:32:47.870319+0000","last_peered":"2024-03-29T11:32:47.870319+0000","last_clean":"2024-03-29T11:32:47.870319+0000","last_became_active":"2024-03-29T11:32:47.865604+0000","last_became_peered":"2024-03-29T11:32:47.865604+0000","last_unstale":"2024-03-29T11:32:47.870319+0000","last_undegraded":"2024-03-29T11:32:47.870319+0000","last_fullsized":"2024-03-29T11:32:47.870319+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:16.654138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861122+0000","last_change":"2024-03-29T11:32:47.340544+0000","last_active":"2024-03-29T11:32:47.861122+0000","last_peered":"2024-03-29T11:32:47.861122+0000","last_clean":"2024-03-29T11:32:47.861122+0000","last_became_active":"2024-03-29T11:32:47.340286+0000","last_became_peered":"2024-03-29T11:32:47.340286+0000","last_unstale":"2024-03-29T11:32:47.861122+0000","last_undegraded":"2024-03-29T11:32:47.861122+0000","last_fullsized":"2024-03-29T11:32:47.861122+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:56.832095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.879554+0000","last_change":"2024-03-29T11:32:32.963463+0000","last_active":"2024-03-29T11:32:47.879554+0000","last_peered":"2024-03-29T11:32:47.879554+0000","last_clean":"2024-03-29T11:32:47.879554+0000","last_became_active":"2024-03-29T11:32:32.962970+0000","last_became_peered":"2024-03-29T11:32:32.962970+0000","last_unstale":"2024-03-29T11:32:47.879554+0000","last_undegraded":"2024-03-29T11:32:47.879554+0000","last_fullsized":"2024-03-29T11:32:47.879554+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:14.637128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338023+0000","last_change":"2024-03-29T11:32:47.870166+0000","last_active":"2024-03-29T11:32:49.338023+0000","last_peered":"2024-03-29T11:32:49.338023+0000","last_clean":"2024-03-29T11:32:49.338023+0000","last_became_active":"2024-03-29T11:32:47.869868+0000","last_became_peered":"2024-03-29T11:32:47.869868+0000","last_unstale":"2024-03-29T11:32:49.338023+0000","last_undegraded":"2024-03-29T11:32:49.338023+0000","last_fullsized":"2024-03-29T11:32:49.338023+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:18:25.353859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868745+0000","last_change":"2024-03-29T11:32:47.347326+0000","last_active":"2024-03-29T11:32:48.868745+0000","last_peered":"2024-03-29T11:32:48.868745+0000","last_clean":"2024-03-29T11:32:48.868745+0000","last_became_active":"2024-03-29T11:32:47.347007+0000","last_became_peered":"2024-03-29T11:32:47.347007+0000","last_unstale":"2024-03-29T11:32:48.868745+0000","last_undegraded":"2024-03-29T11:32:48.868745+0000","last_fullsized":"2024-03-29T11:32:48.868745+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:00.321378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.871023+0000","last_change":"2024-03-29T11:32:32.953703+0000","last_active":"2024-03-29T11:32:47.871023+0000","last_peered":"2024-03-29T11:32:47.871023+0000","last_clean":"2024-03-29T11:32:47.871023+0000","last_became_active":"2024-03-29T11:32:32.952942+0000","last_became_peered":"2024-03-29T11:32:32.952942+0000","last_unstale":"2024-03-29T11:32:47.871023+0000","last_undegraded":"2024-03-29T11:32:47.871023+0000","last_fullsized":"2024-03-29T11:32:47.871023+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:23:44.759018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218814+0000","last_change":"2024-03-29T11:32:47.873899+0000","last_active":"2024-03-29T11:32:49.218814+0000","last_peered":"2024-03-29T11:32:49.218814+0000","last_clean":"2024-03-29T11:32:49.218814+0000","last_became_active":"2024-03-29T11:32:47.873620+0000","last_became_peered":"2024-03-29T11:32:47.873620+0000","last_unstale":"2024-03-29T11:32:49.218814+0000","last_undegraded":"2024-03-29T11:32:49.218814+0000","last_fullsized":"2024-03-29T11:32:49.218814+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:28.748316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.859706+0000","last_change":"2024-03-29T11:32:47.342809+0000","last_active":"2024-03-29T11:32:47.859706+0000","last_peered":"2024-03-29T11:32:47.859706+0000","last_clean":"2024-03-29T11:32:47.859706+0000","last_became_active":"2024-03-29T11:32:47.341749+0000","last_became_peered":"2024-03-29T11:32:47.341749+0000","last_unstale":"2024-03-29T11:32:47.859706+0000","last_undegraded":"2024-03-29T11:32:47.859706+0000","last_fullsized":"2024-03-29T11:32:47.859706+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:36.962093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.346070+0000","last_change":"2024-03-29T11:32:32.970743+0000","last_active":"2024-03-29T11:32:47.346070+0000","last_peered":"2024-03-29T11:32:47.346070+0000","last_clean":"2024-03-29T11:32:47.346070+0000","last_became_active":"2024-03-29T11:32:32.967319+0000","last_became_peered":"2024-03-29T11:32:32.967319+0000","last_unstale":"2024-03-29T11:32:47.346070+0000","last_undegraded":"2024-03-29T11:32:47.346070+0000","last_fullsized":"2024-03-29T11:32:47.346070+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:52.668591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218485+0000","last_change":"2024-03-29T11:32:47.863502+0000","last_active":"2024-03-29T11:32:49.218485+0000","last_peered":"2024-03-29T11:32:49.218485+0000","last_clean":"2024-03-29T11:32:49.218485+0000","last_became_active":"2024-03-29T11:32:47.862432+0000","last_became_peered":"2024-03-29T11:32:47.862432+0000","last_unstale":"2024-03-29T11:32:49.218485+0000","last_undegraded":"2024-03-29T11:32:49.218485+0000","last_fullsized":"2024-03-29T11:32:49.218485+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:19:11.800767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193591+0000","last_change":"2024-03-29T11:32:47.339115+0000","last_active":"2024-03-29T11:32:49.193591+0000","last_peered":"2024-03-29T11:32:49.193591+0000","last_clean":"2024-03-29T11:32:49.193591+0000","last_became_active":"2024-03-29T11:32:47.338823+0000","last_became_peered":"2024-03-29T11:32:47.338823+0000","last_unstale":"2024-03-29T11:32:49.193591+0000","last_undegraded":"2024-03-29T11:32:49.193591+0000","last_fullsized":"2024-03-29T11:32:49.193591+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:58.495188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.339142+0000","last_change":"2024-03-29T11:32:32.957788+0000","last_active":"2024-03-29T11:32:47.339142+0000","last_peered":"2024-03-29T11:32:47.339142+0000","last_clean":"2024-03-29T11:32:47.339142+0000","last_became_active":"2024-03-29T11:32:32.957519+0000","last_became_peered":"2024-03-29T11:32:32.957519+0000","last_unstale":"2024-03-29T11:32:47.339142+0000","last_undegraded":"2024-03-29T11:32:47.339142+0000","last_fullsized":"2024-03-29T11:32:47.339142+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:21.664133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.194043+0000","last_change":"2024-03-29T11:32:47.871212+0000","last_active":"2024-03-29T11:32:49.194043+0000","last_peered":"2024-03-29T11:32:49.194043+0000","last_clean":"2024-03-29T11:32:49.194043+0000","last_became_active":"2024-03-29T11:32:47.869237+0000","last_became_peered":"2024-03-29T11:32:47.869237+0000","last_unstale":"2024-03-29T11:32:49.194043+0000","last_undegraded":"2024-03-29T11:32:49.194043+0000","last_fullsized":"2024-03-29T11:32:49.194043+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:49.665750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862336+0000","last_change":"2024-03-29T11:32:47.352473+0000","last_active":"2024-03-29T11:32:48.862336+0000","last_peered":"2024-03-29T11:32:48.862336+0000","last_clean":"2024-03-29T11:32:48.862336+0000","last_became_active":"2024-03-29T11:32:47.347349+0000","last_became_peered":"2024-03-29T11:32:47.347349+0000","last_unstale":"2024-03-29T11:32:48.862336+0000","last_undegraded":"2024-03-29T11:32:48.862336+0000","last_fullsized":"2024-03-29T11:32:48.862336+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:28:45.896358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.867210+0000","last_change":"2024-03-29T11:32:32.967536+0000","last_active":"2024-03-29T11:32:47.867210+0000","last_peered":"2024-03-29T11:32:47.867210+0000","last_clean":"2024-03-29T11:32:47.867210+0000","last_became_active":"2024-03-29T11:32:32.966686+0000","last_became_peered":"2024-03-29T11:32:32.966686+0000","last_unstale":"2024-03-29T11:32:47.867210+0000","last_undegraded":"2024-03-29T11:32:47.867210+0000","last_fullsized":"2024-03-29T11:32:47.867210+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:24.736585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862432+0000","last_change":"2024-03-29T11:32:47.873240+0000","last_active":"2024-03-29T11:32:48.862432+0000","last_peered":"2024-03-29T11:32:48.862432+0000","last_clean":"2024-03-29T11:32:48.862432+0000","last_became_active":"2024-03-29T11:32:47.871973+0000","last_became_peered":"2024-03-29T11:32:47.871973+0000","last_unstale":"2024-03-29T11:32:48.862432+0000","last_undegraded":"2024-03-29T11:32:48.862432+0000","last_fullsized":"2024-03-29T11:32:48.862432+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:52:10.675709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868306+0000","last_change":"2024-03-29T11:32:47.347470+0000","last_active":"2024-03-29T11:32:48.868306+0000","last_peered":"2024-03-29T11:32:48.868306+0000","last_clean":"2024-03-29T11:32:48.868306+0000","last_became_active":"2024-03-29T11:32:47.347097+0000","last_became_peered":"2024-03-29T11:32:47.347097+0000","last_unstale":"2024-03-29T11:32:48.868306+0000","last_undegraded":"2024-03-29T11:32:48.868306+0000","last_fullsized":"2024-03-29T11:32:48.868306+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:15.651144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854388+0000","last_change":"2024-03-29T11:32:32.952674+0000","last_active":"2024-03-29T11:32:47.854388+0000","last_peered":"2024-03-29T11:32:47.854388+0000","last_clean":"2024-03-29T11:32:47.854388+0000","last_became_active":"2024-03-29T11:32:32.952387+0000","last_became_peered":"2024-03-29T11:32:32.952387+0000","last_unstale":"2024-03-29T11:32:47.854388+0000","last_undegraded":"2024-03-29T11:32:47.854388+0000","last_fullsized":"2024-03-29T11:32:47.854388+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:30.121534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868873+0000","last_change":"2024-03-29T11:32:47.888074+0000","last_active":"2024-03-29T11:32:48.868873+0000","last_peered":"2024-03-29T11:32:48.868873+0000","last_clean":"2024-03-29T11:32:48.868873+0000","last_became_active":"2024-03-29T11:32:47.887753+0000","last_became_peered":"2024-03-29T11:32:47.887753+0000","last_unstale":"2024-03-29T11:32:48.868873+0000","last_undegraded":"2024-03-29T11:32:48.868873+0000","last_fullsized":"2024-03-29T11:32:48.868873+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:16:25.816476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.219106+0000","last_change":"2024-03-29T11:32:47.348457+0000","last_active":"2024-03-29T11:32:49.219106+0000","last_peered":"2024-03-29T11:32:49.219106+0000","last_clean":"2024-03-29T11:32:49.219106+0000","last_became_active":"2024-03-29T11:32:47.348162+0000","last_became_peered":"2024-03-29T11:32:47.348162+0000","last_unstale":"2024-03-29T11:32:49.219106+0000","last_undegraded":"2024-03-29T11:32:49.219106+0000","last_fullsized":"2024-03-29T11:32:49.219106+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:41.984195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.878359+0000","last_change":"2024-03-29T11:32:32.966028+0000","last_active":"2024-03-29T11:32:47.878359+0000","last_peered":"2024-03-29T11:32:47.878359+0000","last_clean":"2024-03-29T11:32:47.878359+0000","last_became_active":"2024-03-29T11:32:32.965642+0000","last_became_peered":"2024-03-29T11:32:32.965642+0000","last_unstale":"2024-03-29T11:32:47.878359+0000","last_undegraded":"2024-03-29T11:32:47.878359+0000","last_fullsized":"2024-03-29T11:32:47.878359+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:28:31.977713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862565+0000","last_change":"2024-03-29T11:32:47.887949+0000","last_active":"2024-03-29T11:32:48.862565+0000","last_peered":"2024-03-29T11:32:48.862565+0000","last_clean":"2024-03-29T11:32:48.862565+0000","last_became_active":"2024-03-29T11:32:47.887377+0000","last_became_peered":"2024-03-29T11:32:47.887377+0000","last_unstale":"2024-03-29T11:32:48.862565+0000","last_undegraded":"2024-03-29T11:32:48.862565+0000","last_fullsized":"2024-03-29T11:32:48.862565+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:15.233472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"251'1","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.880873+0000","last_change":"2024-03-29T11:32:47.345394+0000","last_active":"2024-03-29T11:32:47.880873+0000","last_peered":"2024-03-29T11:32:47.880873+0000","last_clean":"2024-03-29T11:32:47.880873+0000","last_became_active":"2024-03-29T11:32:47.345100+0000","last_became_peered":"2024-03-29T11:32:47.345100+0000","last_unstale":"2024-03-29T11:32:47.880873+0000","last_undegraded":"2024-03-29T11:32:47.880873+0000","last_fullsized":"2024-03-29T11:32:47.880873+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:49.746156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.860050+0000","last_change":"2024-03-29T11:32:32.960337+0000","last_active":"2024-03-29T11:32:47.860050+0000","last_peered":"2024-03-29T11:32:47.860050+0000","last_clean":"2024-03-29T11:32:47.860050+0000","last_became_active":"2024-03-29T11:32:32.958824+0000","last_became_peered":"2024-03-29T11:32:32.958824+0000","last_unstale":"2024-03-29T11:32:47.860050+0000","last_undegraded":"2024-03-29T11:32:47.860050+0000","last_fullsized":"2024-03-29T11:32:47.860050+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:48:40.247306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.337903+0000","last_change":"2024-03-29T11:32:47.874778+0000","last_active":"2024-03-29T11:32:49.337903+0000","last_peered":"2024-03-29T11:32:49.337903+0000","last_clean":"2024-03-29T11:32:49.337903+0000","last_became_active":"2024-03-29T11:32:47.872247+0000","last_became_peered":"2024-03-29T11:32:47.872247+0000","last_unstale":"2024-03-29T11:32:49.337903+0000","last_undegraded":"2024-03-29T11:32:49.337903+0000","last_fullsized":"2024-03-29T11:32:49.337903+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:22:15.580093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"251'8","reported_seq":19,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218395+0000","last_change":"2024-03-29T11:32:47.353083+0000","last_active":"2024-03-29T11:32:49.218395+0000","last_peered":"2024-03-29T11:32:49.218395+0000","last_clean":"2024-03-29T11:32:49.218395+0000","last_became_active":"2024-03-29T11:32:47.352595+0000","last_became_peered":"2024-03-29T11:32:47.352595+0000","last_unstale":"2024-03-29T11:32:49.218395+0000","last_undegraded":"2024-03-29T11:32:49.218395+0000","last_fullsized":"2024-03-29T11:32:49.218395+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:21:18.809408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.341757+0000","last_change":"2024-03-29T11:32:32.960497+0000","last_active":"2024-03-29T11:32:47.341757+0000","last_peered":"2024-03-29T11:32:47.341757+0000","last_clean":"2024-03-29T11:32:47.341757+0000","last_became_active":"2024-03-29T11:32:32.958333+0000","last_became_peered":"2024-03-29T11:32:32.958333+0000","last_unstale":"2024-03-29T11:32:47.341757+0000","last_undegraded":"2024-03-29T11:32:47.341757+0000","last_fullsized":"2024-03-29T11:32:47.341757+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:27:28.513987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.337900+0000","last_change":"2024-03-29T11:32:47.882397+0000","last_active":"2024-03-29T11:32:49.337900+0000","last_peered":"2024-03-29T11:32:49.337900+0000","last_clean":"2024-03-29T11:32:49.337900+0000","last_became_active":"2024-03-29T11:32:47.881886+0000","last_became_peered":"2024-03-29T11:32:47.881886+0000","last_unstale":"2024-03-29T11:32:49.337900+0000","last_undegraded":"2024-03-29T11:32:49.337900+0000","last_fullsized":"2024-03-29T11:32:49.337900+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:21:15.130504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868839+0000","last_change":"2024-03-29T11:32:47.348169+0000","last_active":"2024-03-29T11:32:48.868839+0000","last_peered":"2024-03-29T11:32:48.868839+0000","last_clean":"2024-03-29T11:32:48.868839+0000","last_became_active":"2024-03-29T11:32:47.347899+0000","last_became_peered":"2024-03-29T11:32:47.347899+0000","last_unstale":"2024-03-29T11:32:48.868839+0000","last_undegraded":"2024-03-29T11:32:48.868839+0000","last_fullsized":"2024-03-29T11:32:48.868839+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:27.533294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854490+0000","last_change":"2024-03-29T11:32:32.949294+0000","last_active":"2024-03-29T11:32:47.854490+0000","last_peered":"2024-03-29T11:32:47.854490+0000","last_clean":"2024-03-29T11:32:47.854490+0000","last_became_active":"2024-03-29T11:32:32.949025+0000","last_became_peered":"2024-03-29T11:32:32.949025+0000","last_unstale":"2024-03-29T11:32:47.854490+0000","last_undegraded":"2024-03-29T11:32:47.854490+0000","last_fullsized":"2024-03-29T11:32:47.854490+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218794+0000","last_change":"2024-03-29T11:32:47.873323+0000","last_active":"2024-03-29T11:32:49.218794+0000","last_peered":"2024-03-29T11:32:49.218794+0000","last_clean":"2024-03-29T11:32:49.218794+0000","last_became_active":"2024-03-29T11:32:47.873061+0000","last_became_peered":"2024-03-29T11:32:47.873061+0000","last_unstale":"2024-03-29T11:32:49.218794+0000","last_undegraded":"2024-03-29T11:32:49.218794+0000","last_fullsized":"2024-03-29T11:32:49.218794+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:15.233472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":10,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.858998+0000","last_change":"2024-03-29T11:32:47.352021+0000","last_active":"2024-03-29T11:32:47.858998+0000","last_peered":"2024-03-29T11:32:47.858998+0000","last_clean":"2024-03-29T11:32:47.858998+0000","last_became_active":"2024-03-29T11:32:47.351478+0000","last_became_peered":"2024-03-29T11:32:47.351478+0000","last_unstale":"2024-03-29T11:32:47.858998+0000","last_undegraded":"2024-03-29T11:32:47.858998+0000","last_fullsized":"2024-03-29T11:32:47.858998+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:15:44.616340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.341112+0000","last_change":"2024-03-29T11:32:32.965153+0000","last_active":"2024-03-29T11:32:47.341112+0000","last_peered":"2024-03-29T11:32:47.341112+0000","last_clean":"2024-03-29T11:32:47.341112+0000","last_became_active":"2024-03-29T11:32:32.964935+0000","last_became_peered":"2024-03-29T11:32:32.964935+0000","last_unstale":"2024-03-29T11:32:47.341112+0000","last_undegraded":"2024-03-29T11:32:47.341112+0000","last_fullsized":"2024-03-29T11:32:47.341112+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:02.769642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338031+0000","last_change":"2024-03-29T11:32:47.880871+0000","last_active":"2024-03-29T11:32:49.338031+0000","last_peered":"2024-03-29T11:32:49.338031+0000","last_clean":"2024-03-29T11:32:49.338031+0000","last_became_active":"2024-03-29T11:32:47.880621+0000","last_became_peered":"2024-03-29T11:32:47.880621+0000","last_unstale":"2024-03-29T11:32:49.338031+0000","last_undegraded":"2024-03-29T11:32:49.338031+0000","last_fullsized":"2024-03-29T11:32:49.338031+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:24:56.799955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"251'2","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.885870+0000","last_change":"2024-03-29T11:32:47.352084+0000","last_active":"2024-03-29T11:32:47.885870+0000","last_peered":"2024-03-29T11:32:47.885870+0000","last_clean":"2024-03-29T11:32:47.885870+0000","last_became_active":"2024-03-29T11:32:47.351764+0000","last_became_peered":"2024-03-29T11:32:47.351764+0000","last_unstale":"2024-03-29T11:32:47.885870+0000","last_undegraded":"2024-03-29T11:32:47.885870+0000","last_fullsized":"2024-03-29T11:32:47.885870+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:47.734117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.866498+0000","last_change":"2024-03-29T11:32:32.953083+0000","last_active":"2024-03-29T11:32:47.866498+0000","last_peered":"2024-03-29T11:32:47.866498+0000","last_clean":"2024-03-29T11:32:47.866498+0000","last_became_active":"2024-03-29T11:32:32.952823+0000","last_became_peered":"2024-03-29T11:32:32.952823+0000","last_unstale":"2024-03-29T11:32:47.866498+0000","last_undegraded":"2024-03-29T11:32:47.866498+0000","last_fullsized":"2024-03-29T11:32:47.866498+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:09.626524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338902+0000","last_change":"2024-03-29T11:32:47.875386+0000","last_active":"2024-03-29T11:32:49.338902+0000","last_peered":"2024-03-29T11:32:49.338902+0000","last_clean":"2024-03-29T11:32:49.338902+0000","last_became_active":"2024-03-29T11:32:47.875130+0000","last_became_peered":"2024-03-29T11:32:47.875130+0000","last_unstale":"2024-03-29T11:32:49.338902+0000","last_undegraded":"2024-03-29T11:32:49.338902+0000","last_fullsized":"2024-03-29T11:32:49.338902+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:54:06.010602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"251'6","reported_seq":17,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338195+0000","last_change":"2024-03-29T11:32:47.344656+0000","last_active":"2024-03-29T11:32:49.338195+0000","last_peered":"2024-03-29T11:32:49.338195+0000","last_clean":"2024-03-29T11:32:49.338195+0000","last_became_active":"2024-03-29T11:32:47.344357+0000","last_became_peered":"2024-03-29T11:32:47.344357+0000","last_unstale":"2024-03-29T11:32:49.338195+0000","last_undegraded":"2024-03-29T11:32:49.338195+0000","last_fullsized":"2024-03-29T11:32:49.338195+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:36:44.081864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.339447+0000","last_change":"2024-03-29T11:32:32.954755+0000","last_active":"2024-03-29T11:32:47.339447+0000","last_peered":"2024-03-29T11:32:47.339447+0000","last_clean":"2024-03-29T11:32:47.339447+0000","last_became_active":"2024-03-29T11:32:32.954467+0000","last_became_peered":"2024-03-29T11:32:32.954467+0000","last_unstale":"2024-03-29T11:32:47.339447+0000","last_undegraded":"2024-03-29T11:32:47.339447+0000","last_fullsized":"2024-03-29T11:32:47.339447+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:10.303092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862462+0000","last_change":"2024-03-29T11:32:47.881806+0000","last_active":"2024-03-29T11:32:48.862462+0000","last_peered":"2024-03-29T11:32:48.862462+0000","last_clean":"2024-03-29T11:32:48.862462+0000","last_became_active":"2024-03-29T11:32:47.872495+0000","last_became_peered":"2024-03-29T11:32:47.872495+0000","last_unstale":"2024-03-29T11:32:48.862462+0000","last_undegraded":"2024-03-29T11:32:48.862462+0000","last_fullsized":"2024-03-29T11:32:48.862462+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:16:25.816476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"251'3","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.219036+0000","last_change":"2024-03-29T11:32:47.347470+0000","last_active":"2024-03-29T11:32:49.219036+0000","last_peered":"2024-03-29T11:32:49.219036+0000","last_clean":"2024-03-29T11:32:49.219036+0000","last_became_active":"2024-03-29T11:32:47.347198+0000","last_became_peered":"2024-03-29T11:32:47.347198+0000","last_unstale":"2024-03-29T11:32:49.219036+0000","last_undegraded":"2024-03-29T11:32:49.219036+0000","last_fullsized":"2024-03-29T11:32:49.219036+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:19.718112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861971+0000","last_change":"2024-03-29T11:32:32.955216+0000","last_active":"2024-03-29T11:32:47.861971+0000","last_peered":"2024-03-29T11:32:47.861971+0000","last_clean":"2024-03-29T11:32:47.861971+0000","last_became_active":"2024-03-29T11:32:32.954774+0000","last_became_peered":"2024-03-29T11:32:32.954774+0000","last_unstale":"2024-03-29T11:32:47.861971+0000","last_undegraded":"2024-03-29T11:32:47.861971+0000","last_fullsized":"2024-03-29T11:32:47.861971+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:49:44.576542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338992+0000","last_change":"2024-03-29T11:32:47.874012+0000","last_active":"2024-03-29T11:32:49.338992+0000","last_peered":"2024-03-29T11:32:49.338992+0000","last_clean":"2024-03-29T11:32:49.338992+0000","last_became_active":"2024-03-29T11:32:47.873755+0000","last_became_peered":"2024-03-29T11:32:47.873755+0000","last_unstale":"2024-03-29T11:32:49.338992+0000","last_undegraded":"2024-03-29T11:32:49.338992+0000","last_fullsized":"2024-03-29T11:32:49.338992+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:40.369123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.219123+0000","last_change":"2024-03-29T11:32:47.347700+0000","last_active":"2024-03-29T11:32:49.219123+0000","last_peered":"2024-03-29T11:32:49.219123+0000","last_clean":"2024-03-29T11:32:49.219123+0000","last_became_active":"2024-03-29T11:32:47.347431+0000","last_became_peered":"2024-03-29T11:32:47.347431+0000","last_unstale":"2024-03-29T11:32:49.219123+0000","last_undegraded":"2024-03-29T11:32:49.219123+0000","last_fullsized":"2024-03-29T11:32:49.219123+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:43:23.545169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.338716+0000","last_change":"2024-03-29T11:32:32.964085+0000","last_active":"2024-03-29T11:32:47.338716+0000","last_peered":"2024-03-29T11:32:47.338716+0000","last_clean":"2024-03-29T11:32:47.338716+0000","last_became_active":"2024-03-29T11:32:32.963753+0000","last_became_peered":"2024-03-29T11:32:32.963753+0000","last_unstale":"2024-03-29T11:32:47.338716+0000","last_undegraded":"2024-03-29T11:32:47.338716+0000","last_fullsized":"2024-03-29T11:32:47.338716+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:34.956119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867875+0000","last_change":"2024-03-29T11:32:47.874694+0000","last_active":"2024-03-29T11:32:48.867875+0000","last_peered":"2024-03-29T11:32:48.867875+0000","last_clean":"2024-03-29T11:32:48.867875+0000","last_became_active":"2024-03-29T11:32:47.874365+0000","last_became_peered":"2024-03-29T11:32:47.874365+0000","last_unstale":"2024-03-29T11:32:48.867875+0000","last_undegraded":"2024-03-29T11:32:48.867875+0000","last_fullsized":"2024-03-29T11:32:48.867875+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:30.885593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.339110+0000","last_change":"2024-03-29T11:32:47.333902+0000","last_active":"2024-03-29T11:32:49.339110+0000","last_peered":"2024-03-29T11:32:49.339110+0000","last_clean":"2024-03-29T11:32:49.339110+0000","last_became_active":"2024-03-29T11:32:47.333598+0000","last_became_peered":"2024-03-29T11:32:47.333598+0000","last_unstale":"2024-03-29T11:32:49.339110+0000","last_undegraded":"2024-03-29T11:32:49.339110+0000","last_fullsized":"2024-03-29T11:32:49.339110+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:37:48.411120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854010+0000","last_change":"2024-03-29T11:32:32.957520+0000","last_active":"2024-03-29T11:32:47.854010+0000","last_peered":"2024-03-29T11:32:47.854010+0000","last_clean":"2024-03-29T11:32:47.854010+0000","last_became_active":"2024-03-29T11:32:32.957245+0000","last_became_peered":"2024-03-29T11:32:32.957245+0000","last_unstale":"2024-03-29T11:32:47.854010+0000","last_undegraded":"2024-03-29T11:32:47.854010+0000","last_fullsized":"2024-03-29T11:32:47.854010+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:45.368395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.866798+0000","last_change":"2024-03-29T11:32:47.873501+0000","last_active":"2024-03-29T11:32:48.866798+0000","last_peered":"2024-03-29T11:32:48.866798+0000","last_clean":"2024-03-29T11:32:48.866798+0000","last_became_active":"2024-03-29T11:32:47.870502+0000","last_became_peered":"2024-03-29T11:32:47.870502+0000","last_unstale":"2024-03-29T11:32:48.866798+0000","last_undegraded":"2024-03-29T11:32:48.866798+0000","last_fullsized":"2024-03-29T11:32:48.866798+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:59:56.436348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.866887+0000","last_change":"2024-03-29T11:32:47.346085+0000","last_active":"2024-03-29T11:32:48.866887+0000","last_peered":"2024-03-29T11:32:48.866887+0000","last_clean":"2024-03-29T11:32:48.866887+0000","last_became_active":"2024-03-29T11:32:47.345724+0000","last_became_peered":"2024-03-29T11:32:47.345724+0000","last_unstale":"2024-03-29T11:32:48.866887+0000","last_undegraded":"2024-03-29T11:32:48.866887+0000","last_fullsized":"2024-03-29T11:32:48.866887+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:48:54.165952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.866234+0000","last_change":"2024-03-29T11:32:32.955473+0000","last_active":"2024-03-29T11:32:47.866234+0000","last_peered":"2024-03-29T11:32:47.866234+0000","last_clean":"2024-03-29T11:32:47.866234+0000","last_became_active":"2024-03-29T11:32:32.953854+0000","last_became_peered":"2024-03-29T11:32:32.953854+0000","last_unstale":"2024-03-29T11:32:47.866234+0000","last_undegraded":"2024-03-29T11:32:47.866234+0000","last_fullsized":"2024-03-29T11:32:47.866234+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:21.819962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218281+0000","last_change":"2024-03-29T11:32:47.868864+0000","last_active":"2024-03-29T11:32:49.218281+0000","last_peered":"2024-03-29T11:32:49.218281+0000","last_clean":"2024-03-29T11:32:49.218281+0000","last_became_active":"2024-03-29T11:32:47.868572+0000","last_became_peered":"2024-03-29T11:32:47.868572+0000","last_unstale":"2024-03-29T11:32:49.218281+0000","last_undegraded":"2024-03-29T11:32:49.218281+0000","last_fullsized":"2024-03-29T11:32:49.218281+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:45:01.324372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193400+0000","last_change":"2024-03-29T11:32:47.337570+0000","last_active":"2024-03-29T11:32:49.193400+0000","last_peered":"2024-03-29T11:32:49.193400+0000","last_clean":"2024-03-29T11:32:49.193400+0000","last_became_active":"2024-03-29T11:32:47.337288+0000","last_became_peered":"2024-03-29T11:32:47.337288+0000","last_unstale":"2024-03-29T11:32:49.193400+0000","last_undegraded":"2024-03-29T11:32:49.193400+0000","last_fullsized":"2024-03-29T11:32:49.193400+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:22:19.861976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.870940+0000","last_change":"2024-03-29T11:32:32.967040+0000","last_active":"2024-03-29T11:32:47.870940+0000","last_peered":"2024-03-29T11:32:47.870940+0000","last_clean":"2024-03-29T11:32:47.870940+0000","last_became_active":"2024-03-29T11:32:32.966729+0000","last_became_peered":"2024-03-29T11:32:32.966729+0000","last_unstale":"2024-03-29T11:32:47.870940+0000","last_undegraded":"2024-03-29T11:32:47.870940+0000","last_fullsized":"2024-03-29T11:32:47.870940+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:12:27.515858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218710+0000","last_change":"2024-03-29T11:32:47.885165+0000","last_active":"2024-03-29T11:32:49.218710+0000","last_peered":"2024-03-29T11:32:49.218710+0000","last_clean":"2024-03-29T11:32:49.218710+0000","last_became_active":"2024-03-29T11:32:47.884897+0000","last_became_peered":"2024-03-29T11:32:47.884897+0000","last_unstale":"2024-03-29T11:32:49.218710+0000","last_undegraded":"2024-03-29T11:32:49.218710+0000","last_fullsized":"2024-03-29T11:32:49.218710+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:52:10.675709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338304+0000","last_change":"2024-03-29T11:32:47.340767+0000","last_active":"2024-03-29T11:32:49.338304+0000","last_peered":"2024-03-29T11:32:49.338304+0000","last_clean":"2024-03-29T11:32:49.338304+0000","last_became_active":"2024-03-29T11:32:47.339903+0000","last_became_peered":"2024-03-29T11:32:47.339903+0000","last_unstale":"2024-03-29T11:32:49.338304+0000","last_undegraded":"2024-03-29T11:32:49.338304+0000","last_fullsized":"2024-03-29T11:32:49.338304+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:33:56.275080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.346156+0000","last_change":"2024-03-29T11:32:32.970545+0000","last_active":"2024-03-29T11:32:47.346156+0000","last_peered":"2024-03-29T11:32:47.346156+0000","last_clean":"2024-03-29T11:32:47.346156+0000","last_became_active":"2024-03-29T11:32:32.966917+0000","last_became_peered":"2024-03-29T11:32:32.966917+0000","last_unstale":"2024-03-29T11:32:47.346156+0000","last_undegraded":"2024-03-29T11:32:47.346156+0000","last_fullsized":"2024-03-29T11:32:47.346156+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:11.611059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.876653+0000","last_change":"2024-03-29T11:32:47.871148+0000","last_active":"2024-03-29T11:32:47.876653+0000","last_peered":"2024-03-29T11:32:47.876653+0000","last_clean":"2024-03-29T11:32:47.876653+0000","last_became_active":"2024-03-29T11:32:47.870896+0000","last_became_peered":"2024-03-29T11:32:47.870896+0000","last_unstale":"2024-03-29T11:32:47.876653+0000","last_undegraded":"2024-03-29T11:32:47.876653+0000","last_fullsized":"2024-03-29T11:32:47.876653+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:15:14.194005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.866970+0000","last_change":"2024-03-29T11:32:47.343168+0000","last_active":"2024-03-29T11:32:48.866970+0000","last_peered":"2024-03-29T11:32:48.866970+0000","last_clean":"2024-03-29T11:32:48.866970+0000","last_became_active":"2024-03-29T11:32:47.342862+0000","last_became_peered":"2024-03-29T11:32:47.342862+0000","last_unstale":"2024-03-29T11:32:48.866970+0000","last_undegraded":"2024-03-29T11:32:48.866970+0000","last_fullsized":"2024-03-29T11:32:48.866970+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:34:11.151228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.855906+0000","last_change":"2024-03-29T11:32:32.950772+0000","last_active":"2024-03-29T11:32:47.855906+0000","last_peered":"2024-03-29T11:32:47.855906+0000","last_clean":"2024-03-29T11:32:47.855906+0000","last_became_active":"2024-03-29T11:32:32.950518+0000","last_became_peered":"2024-03-29T11:32:32.950518+0000","last_unstale":"2024-03-29T11:32:47.855906+0000","last_undegraded":"2024-03-29T11:32:47.855906+0000","last_fullsized":"2024-03-29T11:32:47.855906+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:34.956119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.879431+0000","last_change":"2024-03-29T11:32:47.878315+0000","last_active":"2024-03-29T11:32:47.879431+0000","last_peered":"2024-03-29T11:32:47.879431+0000","last_clean":"2024-03-29T11:32:47.879431+0000","last_became_active":"2024-03-29T11:32:47.878010+0000","last_became_peered":"2024-03-29T11:32:47.878010+0000","last_unstale":"2024-03-29T11:32:47.879431+0000","last_undegraded":"2024-03-29T11:32:47.879431+0000","last_fullsized":"2024-03-29T11:32:47.879431+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:04.012912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218788+0000","last_change":"2024-03-29T11:32:47.348418+0000","last_active":"2024-03-29T11:32:49.218788+0000","last_peered":"2024-03-29T11:32:49.218788+0000","last_clean":"2024-03-29T11:32:49.218788+0000","last_became_active":"2024-03-29T11:32:47.348122+0000","last_became_peered":"2024-03-29T11:32:47.348122+0000","last_unstale":"2024-03-29T11:32:49.218788+0000","last_undegraded":"2024-03-29T11:32:49.218788+0000","last_fullsized":"2024-03-29T11:32:49.218788+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:13:43.623127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.343181+0000","last_change":"2024-03-29T11:32:32.956094+0000","last_active":"2024-03-29T11:32:47.343181+0000","last_peered":"2024-03-29T11:32:47.343181+0000","last_clean":"2024-03-29T11:32:47.343181+0000","last_became_active":"2024-03-29T11:32:32.955788+0000","last_became_peered":"2024-03-29T11:32:32.955788+0000","last_unstale":"2024-03-29T11:32:47.343181+0000","last_undegraded":"2024-03-29T11:32:47.343181+0000","last_fullsized":"2024-03-29T11:32:47.343181+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:05.005123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867401+0000","last_change":"2024-03-29T11:32:47.870943+0000","last_active":"2024-03-29T11:32:48.867401+0000","last_peered":"2024-03-29T11:32:48.867401+0000","last_clean":"2024-03-29T11:32:48.867401+0000","last_became_active":"2024-03-29T11:32:47.863639+0000","last_became_peered":"2024-03-29T11:32:47.863639+0000","last_unstale":"2024-03-29T11:32:48.867401+0000","last_undegraded":"2024-03-29T11:32:48.867401+0000","last_fullsized":"2024-03-29T11:32:48.867401+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:28.536288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218877+0000","last_change":"2024-03-29T11:32:47.341878+0000","last_active":"2024-03-29T11:32:49.218877+0000","last_peered":"2024-03-29T11:32:49.218877+0000","last_clean":"2024-03-29T11:32:49.218877+0000","last_became_active":"2024-03-29T11:32:47.341594+0000","last_became_peered":"2024-03-29T11:32:47.341594+0000","last_unstale":"2024-03-29T11:32:49.218877+0000","last_undegraded":"2024-03-29T11:32:49.218877+0000","last_fullsized":"2024-03-29T11:32:49.218877+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:03.009918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861574+0000","last_change":"2024-03-29T11:32:32.951190+0000","last_active":"2024-03-29T11:32:47.861574+0000","last_peered":"2024-03-29T11:32:47.861574+0000","last_clean":"2024-03-29T11:32:47.861574+0000","last_became_active":"2024-03-29T11:32:32.950830+0000","last_became_peered":"2024-03-29T11:32:32.950830+0000","last_unstale":"2024-03-29T11:32:47.861574+0000","last_undegraded":"2024-03-29T11:32:47.861574+0000","last_fullsized":"2024-03-29T11:32:47.861574+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:12.191813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862773+0000","last_change":"2024-03-29T11:32:47.867607+0000","last_active":"2024-03-29T11:32:48.862773+0000","last_peered":"2024-03-29T11:32:48.862773+0000","last_clean":"2024-03-29T11:32:48.862773+0000","last_became_active":"2024-03-29T11:32:47.867338+0000","last_became_peered":"2024-03-29T11:32:47.867338+0000","last_unstale":"2024-03-29T11:32:48.862773+0000","last_undegraded":"2024-03-29T11:32:48.862773+0000","last_fullsized":"2024-03-29T11:32:48.862773+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:28.536288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862692+0000","last_change":"2024-03-29T11:32:47.351153+0000","last_active":"2024-03-29T11:32:48.862692+0000","last_peered":"2024-03-29T11:32:48.862692+0000","last_clean":"2024-03-29T11:32:48.862692+0000","last_became_active":"2024-03-29T11:32:47.350841+0000","last_became_peered":"2024-03-29T11:32:47.350841+0000","last_unstale":"2024-03-29T11:32:48.862692+0000","last_undegraded":"2024-03-29T11:32:48.862692+0000","last_fullsized":"2024-03-29T11:32:48.862692+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:28.555774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862635+0000","last_change":"2024-03-29T11:32:32.949171+0000","last_active":"2024-03-29T11:32:47.862635+0000","last_peered":"2024-03-29T11:32:47.862635+0000","last_clean":"2024-03-29T11:32:47.862635+0000","last_became_active":"2024-03-29T11:32:32.948898+0000","last_became_peered":"2024-03-29T11:32:32.948898+0000","last_unstale":"2024-03-29T11:32:47.862635+0000","last_undegraded":"2024-03-29T11:32:47.862635+0000","last_fullsized":"2024-03-29T11:32:47.862635+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:02:36.221843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867700+0000","last_change":"2024-03-29T11:32:47.872107+0000","last_active":"2024-03-29T11:32:48.867700+0000","last_peered":"2024-03-29T11:32:48.867700+0000","last_clean":"2024-03-29T11:32:48.867700+0000","last_became_active":"2024-03-29T11:32:47.871812+0000","last_became_peered":"2024-03-29T11:32:47.871812+0000","last_unstale":"2024-03-29T11:32:48.867700+0000","last_undegraded":"2024-03-29T11:32:48.867700+0000","last_fullsized":"2024-03-29T11:32:48.867700+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:15:14.194005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"251'4","reported_seq":15,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.339066+0000","last_change":"2024-03-29T11:32:47.348086+0000","last_active":"2024-03-29T11:32:49.339066+0000","last_peered":"2024-03-29T11:32:49.339066+0000","last_clean":"2024-03-29T11:32:49.339066+0000","last_became_active":"2024-03-29T11:32:47.347790+0000","last_became_peered":"2024-03-29T11:32:47.347790+0000","last_unstale":"2024-03-29T11:32:49.339066+0000","last_undegraded":"2024-03-29T11:32:49.339066+0000","last_fullsized":"2024-03-29T11:32:49.339066+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:46.313431+0000","last_clean_scrub_stamp":"2024-03-29T11:32:46.313431+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:26.775222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854118+0000","last_change":"2024-03-29T11:32:32.956902+0000","last_active":"2024-03-29T11:32:47.854118+0000","last_peered":"2024-03-29T11:32:47.854118+0000","last_clean":"2024-03-29T11:32:47.854118+0000","last_became_active":"2024-03-29T11:32:32.955472+0000","last_became_peered":"2024-03-29T11:32:32.955472+0000","last_unstale":"2024-03-29T11:32:47.854118+0000","last_undegraded":"2024-03-29T11:32:47.854118+0000","last_fullsized":"2024-03-29T11:32:47.854118+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:54:29.628408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.878650+0000","last_change":"2024-03-29T11:32:47.877511+0000","last_active":"2024-03-29T11:32:47.878650+0000","last_peered":"2024-03-29T11:32:47.878650+0000","last_clean":"2024-03-29T11:32:47.878650+0000","last_became_active":"2024-03-29T11:32:47.877188+0000","last_became_peered":"2024-03-29T11:32:47.877188+0000","last_unstale":"2024-03-29T11:32:47.878650+0000","last_undegraded":"2024-03-29T11:32:47.878650+0000","last_fullsized":"2024-03-29T11:32:47.878650+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:13:20.721106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.869118+0000","last_change":"2024-03-29T11:32:32.959493+0000","last_active":"2024-03-29T11:32:47.869118+0000","last_peered":"2024-03-29T11:32:47.869118+0000","last_clean":"2024-03-29T11:32:47.869118+0000","last_became_active":"2024-03-29T11:32:32.959199+0000","last_became_peered":"2024-03-29T11:32:32.959199+0000","last_unstale":"2024-03-29T11:32:47.869118+0000","last_undegraded":"2024-03-29T11:32:47.869118+0000","last_fullsized":"2024-03-29T11:32:47.869118+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:56.148657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.870724+0000","last_change":"2024-03-29T11:32:32.419758+0000","last_active":"2024-03-29T11:32:47.870724+0000","last_peered":"2024-03-29T11:32:47.870724+0000","last_clean":"2024-03-29T11:32:47.870724+0000","last_became_active":"2024-03-29T11:32:32.419044+0000","last_became_peered":"2024-03-29T11:32:32.419044+0000","last_unstale":"2024-03-29T11:32:47.870724+0000","last_undegraded":"2024-03-29T11:32:47.870724+0000","last_fullsized":"2024-03-29T11:32:47.870724+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:29.373413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.861959+0000","last_change":"2024-03-29T11:32:47.883962+0000","last_active":"2024-03-29T11:32:48.861959+0000","last_peered":"2024-03-29T11:32:48.861959+0000","last_clean":"2024-03-29T11:32:48.861959+0000","last_became_active":"2024-03-29T11:32:47.873744+0000","last_became_peered":"2024-03-29T11:32:47.873744+0000","last_unstale":"2024-03-29T11:32:48.861959+0000","last_undegraded":"2024-03-29T11:32:48.861959+0000","last_fullsized":"2024-03-29T11:32:48.861959+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:15:14.194005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.345188+0000","last_change":"2024-03-29T11:32:32.964445+0000","last_active":"2024-03-29T11:32:47.345188+0000","last_peered":"2024-03-29T11:32:47.345188+0000","last_clean":"2024-03-29T11:32:47.345188+0000","last_became_active":"2024-03-29T11:32:32.963758+0000","last_became_peered":"2024-03-29T11:32:32.963758+0000","last_unstale":"2024-03-29T11:32:47.345188+0000","last_undegraded":"2024-03-29T11:32:47.345188+0000","last_fullsized":"2024-03-29T11:32:47.345188+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:24.556045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.868333+0000","last_change":"2024-03-29T11:32:32.419690+0000","last_active":"2024-03-29T11:32:47.868333+0000","last_peered":"2024-03-29T11:32:47.868333+0000","last_clean":"2024-03-29T11:32:47.868333+0000","last_became_active":"2024-03-29T11:32:32.418508+0000","last_became_peered":"2024-03-29T11:32:32.418508+0000","last_unstale":"2024-03-29T11:32:47.868333+0000","last_undegraded":"2024-03-29T11:32:47.868333+0000","last_fullsized":"2024-03-29T11:32:47.868333+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:34.818177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338384+0000","last_change":"2024-03-29T11:32:47.878409+0000","last_active":"2024-03-29T11:32:49.338384+0000","last_peered":"2024-03-29T11:32:49.338384+0000","last_clean":"2024-03-29T11:32:49.338384+0000","last_became_active":"2024-03-29T11:32:47.877957+0000","last_became_peered":"2024-03-29T11:32:47.877957+0000","last_unstale":"2024-03-29T11:32:49.338384+0000","last_undegraded":"2024-03-29T11:32:49.338384+0000","last_fullsized":"2024-03-29T11:32:49.338384+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:24.877946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.341200+0000","last_change":"2024-03-29T11:32:32.960733+0000","last_active":"2024-03-29T11:32:47.341200+0000","last_peered":"2024-03-29T11:32:47.341200+0000","last_clean":"2024-03-29T11:32:47.341200+0000","last_became_active":"2024-03-29T11:32:32.959254+0000","last_became_peered":"2024-03-29T11:32:32.959254+0000","last_unstale":"2024-03-29T11:32:47.341200+0000","last_undegraded":"2024-03-29T11:32:47.341200+0000","last_fullsized":"2024-03-29T11:32:47.341200+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:45:42.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.866772+0000","last_change":"2024-03-29T11:32:32.422622+0000","last_active":"2024-03-29T11:32:47.866772+0000","last_peered":"2024-03-29T11:32:47.866772+0000","last_clean":"2024-03-29T11:32:47.866772+0000","last_became_active":"2024-03-29T11:32:32.421864+0000","last_became_peered":"2024-03-29T11:32:32.421864+0000","last_unstale":"2024-03-29T11:32:47.866772+0000","last_undegraded":"2024-03-29T11:32:47.866772+0000","last_fullsized":"2024-03-29T11:32:47.866772+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:23.546712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862252+0000","last_change":"2024-03-29T11:32:47.878011+0000","last_active":"2024-03-29T11:32:48.862252+0000","last_peered":"2024-03-29T11:32:48.862252+0000","last_clean":"2024-03-29T11:32:48.862252+0000","last_became_active":"2024-03-29T11:32:47.872989+0000","last_became_peered":"2024-03-29T11:32:47.872989+0000","last_unstale":"2024-03-29T11:32:48.862252+0000","last_undegraded":"2024-03-29T11:32:48.862252+0000","last_fullsized":"2024-03-29T11:32:48.862252+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:28.748316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.341735+0000","last_change":"2024-03-29T11:32:32.965844+0000","last_active":"2024-03-29T11:32:47.341735+0000","last_peered":"2024-03-29T11:32:47.341735+0000","last_clean":"2024-03-29T11:32:47.341735+0000","last_became_active":"2024-03-29T11:32:32.965511+0000","last_became_peered":"2024-03-29T11:32:32.965511+0000","last_unstale":"2024-03-29T11:32:47.341735+0000","last_undegraded":"2024-03-29T11:32:47.341735+0000","last_fullsized":"2024-03-29T11:32:47.341735+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:23.043448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862714+0000","last_change":"2024-03-29T11:32:32.419740+0000","last_active":"2024-03-29T11:32:47.862714+0000","last_peered":"2024-03-29T11:32:47.862714+0000","last_clean":"2024-03-29T11:32:47.862714+0000","last_became_active":"2024-03-29T11:32:32.419471+0000","last_became_peered":"2024-03-29T11:32:32.419471+0000","last_unstale":"2024-03-29T11:32:47.862714+0000","last_undegraded":"2024-03-29T11:32:47.862714+0000","last_fullsized":"2024-03-29T11:32:47.862714+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:01.760308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868017+0000","last_change":"2024-03-29T11:32:47.873163+0000","last_active":"2024-03-29T11:32:48.868017+0000","last_peered":"2024-03-29T11:32:48.868017+0000","last_clean":"2024-03-29T11:32:48.868017+0000","last_became_active":"2024-03-29T11:32:47.872867+0000","last_became_peered":"2024-03-29T11:32:47.872867+0000","last_unstale":"2024-03-29T11:32:48.868017+0000","last_undegraded":"2024-03-29T11:32:48.868017+0000","last_fullsized":"2024-03-29T11:32:48.868017+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:32.717628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.874734+0000","last_change":"2024-03-29T11:32:32.963417+0000","last_active":"2024-03-29T11:32:47.874734+0000","last_peered":"2024-03-29T11:32:47.874734+0000","last_clean":"2024-03-29T11:32:47.874734+0000","last_became_active":"2024-03-29T11:32:32.963109+0000","last_became_peered":"2024-03-29T11:32:32.963109+0000","last_unstale":"2024-03-29T11:32:47.874734+0000","last_undegraded":"2024-03-29T11:32:47.874734+0000","last_fullsized":"2024-03-29T11:32:47.874734+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:24:49.870800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854420+0000","last_change":"2024-03-29T11:32:32.417496+0000","last_active":"2024-03-29T11:32:47.854420+0000","last_peered":"2024-03-29T11:32:47.854420+0000","last_clean":"2024-03-29T11:32:47.854420+0000","last_became_active":"2024-03-29T11:32:32.417165+0000","last_became_peered":"2024-03-29T11:32:32.417165+0000","last_unstale":"2024-03-29T11:32:47.854420+0000","last_undegraded":"2024-03-29T11:32:47.854420+0000","last_fullsized":"2024-03-29T11:32:47.854420+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:57.952240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.194116+0000","last_change":"2024-03-29T11:32:47.876196+0000","last_active":"2024-03-29T11:32:49.194116+0000","last_peered":"2024-03-29T11:32:49.194116+0000","last_clean":"2024-03-29T11:32:49.194116+0000","last_became_active":"2024-03-29T11:32:47.875871+0000","last_became_peered":"2024-03-29T11:32:47.875871+0000","last_unstale":"2024-03-29T11:32:49.194116+0000","last_undegraded":"2024-03-29T11:32:49.194116+0000","last_fullsized":"2024-03-29T11:32:49.194116+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:47.104417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.859799+0000","last_change":"2024-03-29T11:32:32.954909+0000","last_active":"2024-03-29T11:32:47.859799+0000","last_peered":"2024-03-29T11:32:47.859799+0000","last_clean":"2024-03-29T11:32:47.859799+0000","last_became_active":"2024-03-29T11:32:32.954609+0000","last_became_peered":"2024-03-29T11:32:32.954609+0000","last_unstale":"2024-03-29T11:32:47.859799+0000","last_undegraded":"2024-03-29T11:32:47.859799+0000","last_fullsized":"2024-03-29T11:32:47.859799+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:21:04.890763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862402+0000","last_change":"2024-03-29T11:32:32.424318+0000","last_active":"2024-03-29T11:32:47.862402+0000","last_peered":"2024-03-29T11:32:47.862402+0000","last_clean":"2024-03-29T11:32:47.862402+0000","last_became_active":"2024-03-29T11:32:32.424027+0000","last_became_peered":"2024-03-29T11:32:32.424027+0000","last_unstale":"2024-03-29T11:32:47.862402+0000","last_undegraded":"2024-03-29T11:32:47.862402+0000","last_fullsized":"2024-03-29T11:32:47.862402+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:20.654799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862679+0000","last_change":"2024-03-29T11:32:47.878560+0000","last_active":"2024-03-29T11:32:48.862679+0000","last_peered":"2024-03-29T11:32:48.862679+0000","last_clean":"2024-03-29T11:32:48.862679+0000","last_became_active":"2024-03-29T11:32:47.877643+0000","last_became_peered":"2024-03-29T11:32:47.877643+0000","last_unstale":"2024-03-29T11:32:48.862679+0000","last_undegraded":"2024-03-29T11:32:48.862679+0000","last_fullsized":"2024-03-29T11:32:48.862679+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:16.654138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.853523+0000","last_change":"2024-03-29T11:32:32.950803+0000","last_active":"2024-03-29T11:32:47.853523+0000","last_peered":"2024-03-29T11:32:47.853523+0000","last_clean":"2024-03-29T11:32:47.853523+0000","last_became_active":"2024-03-29T11:32:32.950515+0000","last_became_peered":"2024-03-29T11:32:32.950515+0000","last_unstale":"2024-03-29T11:32:47.853523+0000","last_undegraded":"2024-03-29T11:32:47.853523+0000","last_fullsized":"2024-03-29T11:32:47.853523+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:42:17.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.859898+0000","last_change":"2024-03-29T11:32:32.419260+0000","last_active":"2024-03-29T11:32:47.859898+0000","last_peered":"2024-03-29T11:32:47.859898+0000","last_clean":"2024-03-29T11:32:47.859898+0000","last_became_active":"2024-03-29T11:32:32.418975+0000","last_became_peered":"2024-03-29T11:32:32.418975+0000","last_unstale":"2024-03-29T11:32:47.859898+0000","last_undegraded":"2024-03-29T11:32:47.859898+0000","last_fullsized":"2024-03-29T11:32:47.859898+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:41.904116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.337792+0000","last_change":"2024-03-29T11:32:47.873019+0000","last_active":"2024-03-29T11:32:49.337792+0000","last_peered":"2024-03-29T11:32:49.337792+0000","last_clean":"2024-03-29T11:32:49.337792+0000","last_became_active":"2024-03-29T11:32:47.872719+0000","last_became_peered":"2024-03-29T11:32:47.872719+0000","last_unstale":"2024-03-29T11:32:49.337792+0000","last_undegraded":"2024-03-29T11:32:49.337792+0000","last_fullsized":"2024-03-29T11:32:49.337792+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:11:39.133021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854999+0000","last_change":"2024-03-29T11:32:32.949825+0000","last_active":"2024-03-29T11:32:47.854999+0000","last_peered":"2024-03-29T11:32:47.854999+0000","last_clean":"2024-03-29T11:32:47.854999+0000","last_became_active":"2024-03-29T11:32:32.949562+0000","last_became_peered":"2024-03-29T11:32:32.949562+0000","last_unstale":"2024-03-29T11:32:47.854999+0000","last_undegraded":"2024-03-29T11:32:47.854999+0000","last_fullsized":"2024-03-29T11:32:47.854999+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:23.557124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.342055+0000","last_change":"2024-03-29T11:32:32.426409+0000","last_active":"2024-03-29T11:32:47.342055+0000","last_peered":"2024-03-29T11:32:47.342055+0000","last_clean":"2024-03-29T11:32:47.342055+0000","last_became_active":"2024-03-29T11:32:32.426142+0000","last_became_peered":"2024-03-29T11:32:32.426142+0000","last_unstale":"2024-03-29T11:32:47.342055+0000","last_undegraded":"2024-03-29T11:32:47.342055+0000","last_fullsized":"2024-03-29T11:32:47.342055+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:51:51.755610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.866667+0000","last_change":"2024-03-29T11:32:47.874116+0000","last_active":"2024-03-29T11:32:48.866667+0000","last_peered":"2024-03-29T11:32:48.866667+0000","last_clean":"2024-03-29T11:32:48.866667+0000","last_became_active":"2024-03-29T11:32:47.871106+0000","last_became_peered":"2024-03-29T11:32:47.871106+0000","last_unstale":"2024-03-29T11:32:48.866667+0000","last_undegraded":"2024-03-29T11:32:48.866667+0000","last_fullsized":"2024-03-29T11:32:48.866667+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:37:49.414115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.346666+0000","last_change":"2024-03-29T11:32:32.974753+0000","last_active":"2024-03-29T11:32:47.346666+0000","last_peered":"2024-03-29T11:32:47.346666+0000","last_clean":"2024-03-29T11:32:47.346666+0000","last_became_active":"2024-03-29T11:32:32.971332+0000","last_became_peered":"2024-03-29T11:32:32.971332+0000","last_unstale":"2024-03-29T11:32:47.346666+0000","last_undegraded":"2024-03-29T11:32:47.346666+0000","last_fullsized":"2024-03-29T11:32:47.346666+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:33.815472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.868697+0000","last_change":"2024-03-29T11:32:32.420533+0000","last_active":"2024-03-29T11:32:47.868697+0000","last_peered":"2024-03-29T11:32:47.868697+0000","last_clean":"2024-03-29T11:32:47.868697+0000","last_became_active":"2024-03-29T11:32:32.420284+0000","last_became_peered":"2024-03-29T11:32:32.420284+0000","last_unstale":"2024-03-29T11:32:47.868697+0000","last_undegraded":"2024-03-29T11:32:47.868697+0000","last_fullsized":"2024-03-29T11:32:47.868697+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:05:31.173444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.869540+0000","last_change":"2024-03-29T11:32:47.864600+0000","last_active":"2024-03-29T11:32:47.869540+0000","last_peered":"2024-03-29T11:32:47.869540+0000","last_clean":"2024-03-29T11:32:47.869540+0000","last_became_active":"2024-03-29T11:32:47.864345+0000","last_became_peered":"2024-03-29T11:32:47.864345+0000","last_unstale":"2024-03-29T11:32:47.869540+0000","last_undegraded":"2024-03-29T11:32:47.869540+0000","last_fullsized":"2024-03-29T11:32:47.869540+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:28.748316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.866587+0000","last_change":"2024-03-29T11:32:32.969812+0000","last_active":"2024-03-29T11:32:47.866587+0000","last_peered":"2024-03-29T11:32:47.866587+0000","last_clean":"2024-03-29T11:32:47.866587+0000","last_became_active":"2024-03-29T11:32:32.969479+0000","last_became_peered":"2024-03-29T11:32:32.969479+0000","last_unstale":"2024-03-29T11:32:47.866587+0000","last_undegraded":"2024-03-29T11:32:47.866587+0000","last_fullsized":"2024-03-29T11:32:47.866587+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:33:57.232582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"247'3","reported_seq":18,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.856072+0000","last_change":"2024-03-29T11:32:32.418264+0000","last_active":"2024-03-29T11:32:47.856072+0000","last_peered":"2024-03-29T11:32:47.856072+0000","last_clean":"2024-03-29T11:32:47.856072+0000","last_became_active":"2024-03-29T11:32:32.417013+0000","last_became_peered":"2024-03-29T11:32:32.417013+0000","last_unstale":"2024-03-29T11:32:47.856072+0000","last_undegraded":"2024-03-29T11:32:47.856072+0000","last_fullsized":"2024-03-29T11:32:47.856072+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:51.960387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.878388+0000","last_change":"2024-03-29T11:32:47.876221+0000","last_active":"2024-03-29T11:32:47.878388+0000","last_peered":"2024-03-29T11:32:47.878388+0000","last_clean":"2024-03-29T11:32:47.878388+0000","last_became_active":"2024-03-29T11:32:47.873772+0000","last_became_peered":"2024-03-29T11:32:47.873772+0000","last_unstale":"2024-03-29T11:32:47.878388+0000","last_undegraded":"2024-03-29T11:32:47.878388+0000","last_fullsized":"2024-03-29T11:32:47.878388+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:08.340482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.867521+0000","last_change":"2024-03-29T11:32:32.955257+0000","last_active":"2024-03-29T11:32:47.867521+0000","last_peered":"2024-03-29T11:32:47.867521+0000","last_clean":"2024-03-29T11:32:47.867521+0000","last_became_active":"2024-03-29T11:32:32.952017+0000","last_became_peered":"2024-03-29T11:32:32.952017+0000","last_unstale":"2024-03-29T11:32:47.867521+0000","last_undegraded":"2024-03-29T11:32:47.867521+0000","last_fullsized":"2024-03-29T11:32:47.867521+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:45.368395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"248'8","reported_seq":23,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854751+0000","last_change":"2024-03-29T11:32:32.418422+0000","last_active":"2024-03-29T11:32:47.854751+0000","last_peered":"2024-03-29T11:32:47.854751+0000","last_clean":"2024-03-29T11:32:47.854751+0000","last_became_active":"2024-03-29T11:32:32.418119+0000","last_became_peered":"2024-03-29T11:32:32.418119+0000","last_unstale":"2024-03-29T11:32:47.854751+0000","last_undegraded":"2024-03-29T11:32:47.854751+0000","last_fullsized":"2024-03-29T11:32:47.854751+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:34.164745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218793+0000","last_change":"2024-03-29T11:32:47.863185+0000","last_active":"2024-03-29T11:32:49.218793+0000","last_peered":"2024-03-29T11:32:49.218793+0000","last_clean":"2024-03-29T11:32:49.218793+0000","last_became_active":"2024-03-29T11:32:47.862499+0000","last_became_peered":"2024-03-29T11:32:47.862499+0000","last_unstale":"2024-03-29T11:32:49.218793+0000","last_undegraded":"2024-03-29T11:32:49.218793+0000","last_fullsized":"2024-03-29T11:32:49.218793+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:18.229652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862221+0000","last_change":"2024-03-29T11:32:32.955062+0000","last_active":"2024-03-29T11:32:47.862221+0000","last_peered":"2024-03-29T11:32:47.862221+0000","last_clean":"2024-03-29T11:32:47.862221+0000","last_became_active":"2024-03-29T11:32:32.954362+0000","last_became_peered":"2024-03-29T11:32:32.954362+0000","last_unstale":"2024-03-29T11:32:47.862221+0000","last_undegraded":"2024-03-29T11:32:47.862221+0000","last_fullsized":"2024-03-29T11:32:47.862221+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:35:32.510118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.855404+0000","last_change":"2024-03-29T11:32:32.427566+0000","last_active":"2024-03-29T11:32:47.855404+0000","last_peered":"2024-03-29T11:32:47.855404+0000","last_clean":"2024-03-29T11:32:47.855404+0000","last_became_active":"2024-03-29T11:32:32.426512+0000","last_became_peered":"2024-03-29T11:32:32.426512+0000","last_unstale":"2024-03-29T11:32:47.855404+0000","last_undegraded":"2024-03-29T11:32:47.855404+0000","last_fullsized":"2024-03-29T11:32:47.855404+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:49:23.345901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867565+0000","last_change":"2024-03-29T11:32:47.879992+0000","last_active":"2024-03-29T11:32:48.867565+0000","last_peered":"2024-03-29T11:32:48.867565+0000","last_clean":"2024-03-29T11:32:48.867565+0000","last_became_active":"2024-03-29T11:32:47.879694+0000","last_became_peered":"2024-03-29T11:32:47.879694+0000","last_unstale":"2024-03-29T11:32:48.867565+0000","last_undegraded":"2024-03-29T11:32:48.867565+0000","last_fullsized":"2024-03-29T11:32:48.867565+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:15.233472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.874820+0000","last_change":"2024-03-29T11:32:32.956252+0000","last_active":"2024-03-29T11:32:47.874820+0000","last_peered":"2024-03-29T11:32:47.874820+0000","last_clean":"2024-03-29T11:32:47.874820+0000","last_became_active":"2024-03-29T11:32:32.955859+0000","last_became_peered":"2024-03-29T11:32:32.955859+0000","last_unstale":"2024-03-29T11:32:47.874820+0000","last_undegraded":"2024-03-29T11:32:47.874820+0000","last_fullsized":"2024-03-29T11:32:47.874820+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:29.934653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.860117+0000","last_change":"2024-03-29T11:32:32.420246+0000","last_active":"2024-03-29T11:32:47.860117+0000","last_peered":"2024-03-29T11:32:47.860117+0000","last_clean":"2024-03-29T11:32:47.860117+0000","last_became_active":"2024-03-29T11:32:32.419983+0000","last_became_peered":"2024-03-29T11:32:32.419983+0000","last_unstale":"2024-03-29T11:32:47.860117+0000","last_undegraded":"2024-03-29T11:32:47.860117+0000","last_fullsized":"2024-03-29T11:32:47.860117+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:06:35.527658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867670+0000","last_change":"2024-03-29T11:32:47.883672+0000","last_active":"2024-03-29T11:32:48.867670+0000","last_peered":"2024-03-29T11:32:48.867670+0000","last_clean":"2024-03-29T11:32:48.867670+0000","last_became_active":"2024-03-29T11:32:47.883391+0000","last_became_peered":"2024-03-29T11:32:47.883391+0000","last_unstale":"2024-03-29T11:32:48.867670+0000","last_undegraded":"2024-03-29T11:32:48.867670+0000","last_fullsized":"2024-03-29T11:32:48.867670+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:29.841029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862101+0000","last_change":"2024-03-29T11:32:32.946933+0000","last_active":"2024-03-29T11:32:47.862101+0000","last_peered":"2024-03-29T11:32:47.862101+0000","last_clean":"2024-03-29T11:32:47.862101+0000","last_became_active":"2024-03-29T11:32:32.946663+0000","last_became_peered":"2024-03-29T11:32:32.946663+0000","last_unstale":"2024-03-29T11:32:47.862101+0000","last_undegraded":"2024-03-29T11:32:47.862101+0000","last_fullsized":"2024-03-29T11:32:47.862101+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:23.557124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862467+0000","last_change":"2024-03-29T11:32:32.419238+0000","last_active":"2024-03-29T11:32:47.862467+0000","last_peered":"2024-03-29T11:32:47.862467+0000","last_clean":"2024-03-29T11:32:47.862467+0000","last_became_active":"2024-03-29T11:32:32.416581+0000","last_became_peered":"2024-03-29T11:32:32.416581+0000","last_unstale":"2024-03-29T11:32:47.862467+0000","last_undegraded":"2024-03-29T11:32:47.862467+0000","last_fullsized":"2024-03-29T11:32:47.862467+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:09.293759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.338972+0000","last_change":"2024-03-29T11:32:47.870210+0000","last_active":"2024-03-29T11:32:49.338972+0000","last_peered":"2024-03-29T11:32:49.338972+0000","last_clean":"2024-03-29T11:32:49.338972+0000","last_became_active":"2024-03-29T11:32:47.869589+0000","last_became_peered":"2024-03-29T11:32:47.869589+0000","last_unstale":"2024-03-29T11:32:49.338972+0000","last_undegraded":"2024-03-29T11:32:49.338972+0000","last_fullsized":"2024-03-29T11:32:49.338972+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:09:46.377915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862352+0000","last_change":"2024-03-29T11:32:32.960659+0000","last_active":"2024-03-29T11:32:47.862352+0000","last_peered":"2024-03-29T11:32:47.862352+0000","last_clean":"2024-03-29T11:32:47.862352+0000","last_became_active":"2024-03-29T11:32:32.960186+0000","last_became_peered":"2024-03-29T11:32:32.960186+0000","last_unstale":"2024-03-29T11:32:47.862352+0000","last_undegraded":"2024-03-29T11:32:47.862352+0000","last_fullsized":"2024-03-29T11:32:47.862352+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:08:11.369063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"247'8","reported_seq":22,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.345480+0000","last_change":"2024-03-29T11:32:32.420232+0000","last_active":"2024-03-29T11:32:47.345480+0000","last_peered":"2024-03-29T11:32:47.345480+0000","last_clean":"2024-03-29T11:32:47.345480+0000","last_became_active":"2024-03-29T11:32:32.419973+0000","last_became_peered":"2024-03-29T11:32:32.419973+0000","last_unstale":"2024-03-29T11:32:47.345480+0000","last_undegraded":"2024-03-29T11:32:47.345480+0000","last_fullsized":"2024-03-29T11:32:47.345480+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:38:20.810629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218276+0000","last_change":"2024-03-29T11:32:47.879307+0000","last_active":"2024-03-29T11:32:49.218276+0000","last_peered":"2024-03-29T11:32:49.218276+0000","last_clean":"2024-03-29T11:32:49.218276+0000","last_became_active":"2024-03-29T11:32:47.874529+0000","last_became_peered":"2024-03-29T11:32:47.874529+0000","last_unstale":"2024-03-29T11:32:49.218276+0000","last_undegraded":"2024-03-29T11:32:49.218276+0000","last_fullsized":"2024-03-29T11:32:49.218276+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:16.654138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.867507+0000","last_change":"2024-03-29T11:32:32.952991+0000","last_active":"2024-03-29T11:32:47.867507+0000","last_peered":"2024-03-29T11:32:47.867507+0000","last_clean":"2024-03-29T11:32:47.867507+0000","last_became_active":"2024-03-29T11:32:32.952683+0000","last_became_peered":"2024-03-29T11:32:32.952683+0000","last_unstale":"2024-03-29T11:32:47.867507+0000","last_undegraded":"2024-03-29T11:32:47.867507+0000","last_fullsized":"2024-03-29T11:32:47.867507+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:30.121534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.872169+0000","last_change":"2024-03-29T11:32:32.419741+0000","last_active":"2024-03-29T11:32:47.872169+0000","last_peered":"2024-03-29T11:32:47.872169+0000","last_clean":"2024-03-29T11:32:47.872169+0000","last_became_active":"2024-03-29T11:32:32.418896+0000","last_became_peered":"2024-03-29T11:32:32.418896+0000","last_unstale":"2024-03-29T11:32:47.872169+0000","last_undegraded":"2024-03-29T11:32:47.872169+0000","last_fullsized":"2024-03-29T11:32:47.872169+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:59:42.656917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.861925+0000","last_change":"2024-03-29T11:32:47.885402+0000","last_active":"2024-03-29T11:32:48.861925+0000","last_peered":"2024-03-29T11:32:48.861925+0000","last_clean":"2024-03-29T11:32:48.861925+0000","last_became_active":"2024-03-29T11:32:47.885130+0000","last_became_peered":"2024-03-29T11:32:47.885130+0000","last_unstale":"2024-03-29T11:32:48.861925+0000","last_undegraded":"2024-03-29T11:32:48.861925+0000","last_fullsized":"2024-03-29T11:32:48.861925+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:18.229652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.869035+0000","last_change":"2024-03-29T11:32:32.963050+0000","last_active":"2024-03-29T11:32:47.869035+0000","last_peered":"2024-03-29T11:32:47.869035+0000","last_clean":"2024-03-29T11:32:47.869035+0000","last_became_active":"2024-03-29T11:32:32.962777+0000","last_became_peered":"2024-03-29T11:32:32.962777+0000","last_unstale":"2024-03-29T11:32:47.869035+0000","last_undegraded":"2024-03-29T11:32:47.869035+0000","last_fullsized":"2024-03-29T11:32:47.869035+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:31:28.065549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.868784+0000","last_change":"2024-03-29T11:32:32.424196+0000","last_active":"2024-03-29T11:32:47.868784+0000","last_peered":"2024-03-29T11:32:47.868784+0000","last_clean":"2024-03-29T11:32:47.868784+0000","last_became_active":"2024-03-29T11:32:32.423924+0000","last_became_peered":"2024-03-29T11:32:32.423924+0000","last_unstale":"2024-03-29T11:32:47.868784+0000","last_undegraded":"2024-03-29T11:32:47.868784+0000","last_fullsized":"2024-03-29T11:32:47.868784+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:51.659258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.861865+0000","last_change":"2024-03-29T11:32:47.888252+0000","last_active":"2024-03-29T11:32:48.861865+0000","last_peered":"2024-03-29T11:32:48.861865+0000","last_clean":"2024-03-29T11:32:48.861865+0000","last_became_active":"2024-03-29T11:32:47.887169+0000","last_became_peered":"2024-03-29T11:32:47.887169+0000","last_unstale":"2024-03-29T11:32:48.861865+0000","last_undegraded":"2024-03-29T11:32:48.861865+0000","last_fullsized":"2024-03-29T11:32:48.861865+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:05:30.468540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.877255+0000","last_change":"2024-03-29T11:32:32.958521+0000","last_active":"2024-03-29T11:32:47.877255+0000","last_peered":"2024-03-29T11:32:47.877255+0000","last_clean":"2024-03-29T11:32:47.877255+0000","last_became_active":"2024-03-29T11:32:32.958260+0000","last_became_peered":"2024-03-29T11:32:32.958260+0000","last_unstale":"2024-03-29T11:32:47.877255+0000","last_undegraded":"2024-03-29T11:32:47.877255+0000","last_fullsized":"2024-03-29T11:32:47.877255+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:34.744111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.878893+0000","last_change":"2024-03-29T11:32:32.421703+0000","last_active":"2024-03-29T11:32:47.878893+0000","last_peered":"2024-03-29T11:32:47.878893+0000","last_clean":"2024-03-29T11:32:47.878893+0000","last_became_active":"2024-03-29T11:32:32.421417+0000","last_became_peered":"2024-03-29T11:32:32.421417+0000","last_unstale":"2024-03-29T11:32:47.878893+0000","last_undegraded":"2024-03-29T11:32:47.878893+0000","last_fullsized":"2024-03-29T11:32:47.878893+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:51.659258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.862130+0000","last_change":"2024-03-29T11:32:47.885332+0000","last_active":"2024-03-29T11:32:48.862130+0000","last_peered":"2024-03-29T11:32:48.862130+0000","last_clean":"2024-03-29T11:32:48.862130+0000","last_became_active":"2024-03-29T11:32:47.883499+0000","last_became_peered":"2024-03-29T11:32:47.883499+0000","last_unstale":"2024-03-29T11:32:48.862130+0000","last_undegraded":"2024-03-29T11:32:48.862130+0000","last_fullsized":"2024-03-29T11:32:48.862130+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:59:56.436348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.342104+0000","last_change":"2024-03-29T11:32:32.960765+0000","last_active":"2024-03-29T11:32:47.342104+0000","last_peered":"2024-03-29T11:32:47.342104+0000","last_clean":"2024-03-29T11:32:47.342104+0000","last_became_active":"2024-03-29T11:32:32.959952+0000","last_became_peered":"2024-03-29T11:32:32.959952+0000","last_unstale":"2024-03-29T11:32:47.342104+0000","last_undegraded":"2024-03-29T11:32:47.342104+0000","last_fullsized":"2024-03-29T11:32:47.342104+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:18.696940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.345358+0000","last_change":"2024-03-29T11:32:32.423658+0000","last_active":"2024-03-29T11:32:47.345358+0000","last_peered":"2024-03-29T11:32:47.345358+0000","last_clean":"2024-03-29T11:32:47.345358+0000","last_became_active":"2024-03-29T11:32:32.423195+0000","last_became_peered":"2024-03-29T11:32:32.423195+0000","last_unstale":"2024-03-29T11:32:47.345358+0000","last_undegraded":"2024-03-29T11:32:47.345358+0000","last_fullsized":"2024-03-29T11:32:47.345358+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:28:22.547790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.888067+0000","last_change":"2024-03-29T11:32:47.887314+0000","last_active":"2024-03-29T11:32:47.888067+0000","last_peered":"2024-03-29T11:32:47.888067+0000","last_clean":"2024-03-29T11:32:47.888067+0000","last_became_active":"2024-03-29T11:32:47.887045+0000","last_became_peered":"2024-03-29T11:32:47.887045+0000","last_unstale":"2024-03-29T11:32:47.888067+0000","last_undegraded":"2024-03-29T11:32:47.888067+0000","last_fullsized":"2024-03-29T11:32:47.888067+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:13:44.626121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.339860+0000","last_change":"2024-03-29T11:32:32.958080+0000","last_active":"2024-03-29T11:32:47.339860+0000","last_peered":"2024-03-29T11:32:47.339860+0000","last_clean":"2024-03-29T11:32:47.339860+0000","last_became_active":"2024-03-29T11:32:32.957784+0000","last_became_peered":"2024-03-29T11:32:32.957784+0000","last_unstale":"2024-03-29T11:32:47.339860+0000","last_undegraded":"2024-03-29T11:32:47.339860+0000","last_fullsized":"2024-03-29T11:32:47.339860+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:48.762233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.863931+0000","last_change":"2024-03-29T11:32:32.421797+0000","last_active":"2024-03-29T11:32:47.863931+0000","last_peered":"2024-03-29T11:32:47.863931+0000","last_clean":"2024-03-29T11:32:47.863931+0000","last_became_active":"2024-03-29T11:32:32.418956+0000","last_became_peered":"2024-03-29T11:32:32.418956+0000","last_unstale":"2024-03-29T11:32:47.863931+0000","last_undegraded":"2024-03-29T11:32:47.863931+0000","last_fullsized":"2024-03-29T11:32:47.863931+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:32.806138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867893+0000","last_change":"2024-03-29T11:32:47.874664+0000","last_active":"2024-03-29T11:32:48.867893+0000","last_peered":"2024-03-29T11:32:48.867893+0000","last_clean":"2024-03-29T11:32:48.867893+0000","last_became_active":"2024-03-29T11:32:47.873804+0000","last_became_peered":"2024-03-29T11:32:47.873804+0000","last_unstale":"2024-03-29T11:32:48.867893+0000","last_undegraded":"2024-03-29T11:32:48.867893+0000","last_fullsized":"2024-03-29T11:32:48.867893+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:05:30.468540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.854360+0000","last_change":"2024-03-29T11:32:32.951479+0000","last_active":"2024-03-29T11:32:47.854360+0000","last_peered":"2024-03-29T11:32:47.854360+0000","last_clean":"2024-03-29T11:32:47.854360+0000","last_became_active":"2024-03-29T11:32:32.951234+0000","last_became_peered":"2024-03-29T11:32:32.951234+0000","last_unstale":"2024-03-29T11:32:47.854360+0000","last_undegraded":"2024-03-29T11:32:47.854360+0000","last_fullsized":"2024-03-29T11:32:47.854360+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:18:48.762233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.341905+0000","last_change":"2024-03-29T11:32:32.422506+0000","last_active":"2024-03-29T11:32:47.341905+0000","last_peered":"2024-03-29T11:32:47.341905+0000","last_clean":"2024-03-29T11:32:47.341905+0000","last_became_active":"2024-03-29T11:32:32.422249+0000","last_became_peered":"2024-03-29T11:32:32.422249+0000","last_unstale":"2024-03-29T11:32:47.341905+0000","last_undegraded":"2024-03-29T11:32:47.341905+0000","last_fullsized":"2024-03-29T11:32:47.341905+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:29.112200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868675+0000","last_change":"2024-03-29T11:32:47.882727+0000","last_active":"2024-03-29T11:32:48.868675+0000","last_peered":"2024-03-29T11:32:48.868675+0000","last_clean":"2024-03-29T11:32:48.868675+0000","last_became_active":"2024-03-29T11:32:47.882455+0000","last_became_peered":"2024-03-29T11:32:47.882455+0000","last_unstale":"2024-03-29T11:32:48.868675+0000","last_undegraded":"2024-03-29T11:32:48.868675+0000","last_fullsized":"2024-03-29T11:32:48.868675+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:46:27.778216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.343117+0000","last_change":"2024-03-29T11:32:32.955109+0000","last_active":"2024-03-29T11:32:47.343117+0000","last_peered":"2024-03-29T11:32:47.343117+0000","last_clean":"2024-03-29T11:32:47.343117+0000","last_became_active":"2024-03-29T11:32:32.954862+0000","last_became_peered":"2024-03-29T11:32:32.954862+0000","last_unstale":"2024-03-29T11:32:47.343117+0000","last_undegraded":"2024-03-29T11:32:47.343117+0000","last_fullsized":"2024-03-29T11:32:47.343117+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:18.702968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.877964+0000","last_change":"2024-03-29T11:32:32.429057+0000","last_active":"2024-03-29T11:32:47.877964+0000","last_peered":"2024-03-29T11:32:47.877964+0000","last_clean":"2024-03-29T11:32:47.877964+0000","last_became_active":"2024-03-29T11:32:32.428782+0000","last_became_peered":"2024-03-29T11:32:32.428782+0000","last_unstale":"2024-03-29T11:32:47.877964+0000","last_undegraded":"2024-03-29T11:32:47.877964+0000","last_fullsized":"2024-03-29T11:32:47.877964+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:46:23.546712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.877786+0000","last_change":"2024-03-29T11:32:47.871820+0000","last_active":"2024-03-29T11:32:47.877786+0000","last_peered":"2024-03-29T11:32:47.877786+0000","last_clean":"2024-03-29T11:32:47.877786+0000","last_became_active":"2024-03-29T11:32:47.871052+0000","last_became_peered":"2024-03-29T11:32:47.871052+0000","last_unstale":"2024-03-29T11:32:47.877786+0000","last_undegraded":"2024-03-29T11:32:47.877786+0000","last_fullsized":"2024-03-29T11:32:47.877786+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:28.536288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.338800+0000","last_change":"2024-03-29T11:32:32.954161+0000","last_active":"2024-03-29T11:32:47.338800+0000","last_peered":"2024-03-29T11:32:47.338800+0000","last_clean":"2024-03-29T11:32:47.338800+0000","last_became_active":"2024-03-29T11:32:32.953891+0000","last_became_peered":"2024-03-29T11:32:32.953891+0000","last_unstale":"2024-03-29T11:32:47.338800+0000","last_undegraded":"2024-03-29T11:32:47.338800+0000","last_fullsized":"2024-03-29T11:32:47.338800+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:42:17.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862271+0000","last_change":"2024-03-29T11:32:32.418585+0000","last_active":"2024-03-29T11:32:47.862271+0000","last_peered":"2024-03-29T11:32:47.862271+0000","last_clean":"2024-03-29T11:32:47.862271+0000","last_became_active":"2024-03-29T11:32:32.418297+0000","last_became_peered":"2024-03-29T11:32:32.418297+0000","last_unstale":"2024-03-29T11:32:47.862271+0000","last_undegraded":"2024-03-29T11:32:47.862271+0000","last_fullsized":"2024-03-29T11:32:47.862271+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:01.760308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.883867+0000","last_change":"2024-03-29T11:32:47.882656+0000","last_active":"2024-03-29T11:32:47.883867+0000","last_peered":"2024-03-29T11:32:47.883867+0000","last_clean":"2024-03-29T11:32:47.883867+0000","last_became_active":"2024-03-29T11:32:47.882267+0000","last_became_peered":"2024-03-29T11:32:47.882267+0000","last_unstale":"2024-03-29T11:32:47.883867+0000","last_undegraded":"2024-03-29T11:32:47.883867+0000","last_fullsized":"2024-03-29T11:32:47.883867+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:30:54.217597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.870826+0000","last_change":"2024-03-29T11:32:32.953710+0000","last_active":"2024-03-29T11:32:47.870826+0000","last_peered":"2024-03-29T11:32:47.870826+0000","last_clean":"2024-03-29T11:32:47.870826+0000","last_became_active":"2024-03-29T11:32:32.953447+0000","last_became_peered":"2024-03-29T11:32:32.953447+0000","last_unstale":"2024-03-29T11:32:47.870826+0000","last_undegraded":"2024-03-29T11:32:47.870826+0000","last_fullsized":"2024-03-29T11:32:47.870826+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:07.581087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.866441+0000","last_change":"2024-03-29T11:32:32.427167+0000","last_active":"2024-03-29T11:32:47.866441+0000","last_peered":"2024-03-29T11:32:47.866441+0000","last_clean":"2024-03-29T11:32:47.866441+0000","last_became_active":"2024-03-29T11:32:32.426833+0000","last_became_peered":"2024-03-29T11:32:32.426833+0000","last_unstale":"2024-03-29T11:32:47.866441+0000","last_undegraded":"2024-03-29T11:32:47.866441+0000","last_fullsized":"2024-03-29T11:32:47.866441+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:29.373413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.867133+0000","last_change":"2024-03-29T11:32:47.882648+0000","last_active":"2024-03-29T11:32:48.867133+0000","last_peered":"2024-03-29T11:32:48.867133+0000","last_clean":"2024-03-29T11:32:48.867133+0000","last_became_active":"2024-03-29T11:32:47.882342+0000","last_became_peered":"2024-03-29T11:32:47.882342+0000","last_unstale":"2024-03-29T11:32:48.867133+0000","last_undegraded":"2024-03-29T11:32:48.867133+0000","last_fullsized":"2024-03-29T11:32:48.867133+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:16:25.816476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.856238+0000","last_change":"2024-03-29T11:32:32.967461+0000","last_active":"2024-03-29T11:32:47.856238+0000","last_peered":"2024-03-29T11:32:47.856238+0000","last_clean":"2024-03-29T11:32:47.856238+0000","last_became_active":"2024-03-29T11:32:32.967155+0000","last_became_peered":"2024-03-29T11:32:32.967155+0000","last_unstale":"2024-03-29T11:32:47.856238+0000","last_undegraded":"2024-03-29T11:32:47.856238+0000","last_fullsized":"2024-03-29T11:32:47.856238+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:45:24.736585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"248'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.863975+0000","last_change":"2024-03-29T11:32:32.421259+0000","last_active":"2024-03-29T11:32:47.863975+0000","last_peered":"2024-03-29T11:32:47.863975+0000","last_clean":"2024-03-29T11:32:47.863975+0000","last_became_active":"2024-03-29T11:32:32.420971+0000","last_became_peered":"2024-03-29T11:32:32.420971+0000","last_unstale":"2024-03-29T11:32:47.863975+0000","last_undegraded":"2024-03-29T11:32:47.863975+0000","last_fullsized":"2024-03-29T11:32:47.863975+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:34.164745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.218433+0000","last_change":"2024-03-29T11:32:47.869320+0000","last_active":"2024-03-29T11:32:49.218433+0000","last_peered":"2024-03-29T11:32:49.218433+0000","last_clean":"2024-03-29T11:32:49.218433+0000","last_became_active":"2024-03-29T11:32:47.869067+0000","last_became_peered":"2024-03-29T11:32:47.869067+0000","last_unstale":"2024-03-29T11:32:49.218433+0000","last_undegraded":"2024-03-29T11:32:49.218433+0000","last_fullsized":"2024-03-29T11:32:49.218433+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:05:19.639527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.345276+0000","last_change":"2024-03-29T11:32:32.970626+0000","last_active":"2024-03-29T11:32:47.345276+0000","last_peered":"2024-03-29T11:32:47.345276+0000","last_clean":"2024-03-29T11:32:47.345276+0000","last_became_active":"2024-03-29T11:32:32.970154+0000","last_became_peered":"2024-03-29T11:32:32.970154+0000","last_unstale":"2024-03-29T11:32:47.345276+0000","last_undegraded":"2024-03-29T11:32:47.345276+0000","last_fullsized":"2024-03-29T11:32:47.345276+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:13:35.827510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.879058+0000","last_change":"2024-03-29T11:32:32.429420+0000","last_active":"2024-03-29T11:32:47.879058+0000","last_peered":"2024-03-29T11:32:47.879058+0000","last_clean":"2024-03-29T11:32:47.879058+0000","last_became_active":"2024-03-29T11:32:32.429133+0000","last_became_peered":"2024-03-29T11:32:32.429133+0000","last_unstale":"2024-03-29T11:32:47.879058+0000","last_undegraded":"2024-03-29T11:32:47.879058+0000","last_fullsized":"2024-03-29T11:32:47.879058+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:03.995789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193741+0000","last_change":"2024-03-29T11:32:47.874467+0000","last_active":"2024-03-29T11:32:49.193741+0000","last_peered":"2024-03-29T11:32:49.193741+0000","last_clean":"2024-03-29T11:32:49.193741+0000","last_became_active":"2024-03-29T11:32:47.874198+0000","last_became_peered":"2024-03-29T11:32:47.874198+0000","last_unstale":"2024-03-29T11:32:49.193741+0000","last_undegraded":"2024-03-29T11:32:49.193741+0000","last_fullsized":"2024-03-29T11:32:49.193741+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:07:44.856293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.346235+0000","last_change":"2024-03-29T11:32:32.971255+0000","last_active":"2024-03-29T11:32:47.346235+0000","last_peered":"2024-03-29T11:32:47.346235+0000","last_clean":"2024-03-29T11:32:47.346235+0000","last_became_active":"2024-03-29T11:32:32.968877+0000","last_became_peered":"2024-03-29T11:32:32.968877+0000","last_unstale":"2024-03-29T11:32:47.346235+0000","last_undegraded":"2024-03-29T11:32:47.346235+0000","last_fullsized":"2024-03-29T11:32:47.346235+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:35:42.913450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.345435+0000","last_change":"2024-03-29T11:32:32.424671+0000","last_active":"2024-03-29T11:32:47.345435+0000","last_peered":"2024-03-29T11:32:47.345435+0000","last_clean":"2024-03-29T11:32:47.345435+0000","last_became_active":"2024-03-29T11:32:32.424378+0000","last_became_peered":"2024-03-29T11:32:32.424378+0000","last_unstale":"2024-03-29T11:32:47.345435+0000","last_undegraded":"2024-03-29T11:32:47.345435+0000","last_fullsized":"2024-03-29T11:32:47.345435+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:41.180787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193746+0000","last_change":"2024-03-29T11:32:47.873238+0000","last_active":"2024-03-29T11:32:49.193746+0000","last_peered":"2024-03-29T11:32:49.193746+0000","last_clean":"2024-03-29T11:32:49.193746+0000","last_became_active":"2024-03-29T11:32:47.872942+0000","last_became_peered":"2024-03-29T11:32:47.872942+0000","last_unstale":"2024-03-29T11:32:49.193746+0000","last_undegraded":"2024-03-29T11:32:49.193746+0000","last_fullsized":"2024-03-29T11:32:49.193746+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:11:54.332535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.862059+0000","last_change":"2024-03-29T11:32:32.952424+0000","last_active":"2024-03-29T11:32:47.862059+0000","last_peered":"2024-03-29T11:32:47.862059+0000","last_clean":"2024-03-29T11:32:47.862059+0000","last_became_active":"2024-03-29T11:32:32.951804+0000","last_became_peered":"2024-03-29T11:32:32.951804+0000","last_unstale":"2024-03-29T11:32:47.862059+0000","last_undegraded":"2024-03-29T11:32:47.862059+0000","last_fullsized":"2024-03-29T11:32:47.862059+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:22:05.943330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.867307+0000","last_change":"2024-03-29T11:32:32.418931+0000","last_active":"2024-03-29T11:32:47.867307+0000","last_peered":"2024-03-29T11:32:47.867307+0000","last_clean":"2024-03-29T11:32:47.867307+0000","last_became_active":"2024-03-29T11:32:32.418646+0000","last_became_peered":"2024-03-29T11:32:32.418646+0000","last_unstale":"2024-03-29T11:32:47.867307+0000","last_undegraded":"2024-03-29T11:32:47.867307+0000","last_fullsized":"2024-03-29T11:32:47.867307+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:17:12.443004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:49.193907+0000","last_change":"2024-03-29T11:32:47.869516+0000","last_active":"2024-03-29T11:32:49.193907+0000","last_peered":"2024-03-29T11:32:49.193907+0000","last_clean":"2024-03-29T11:32:49.193907+0000","last_became_active":"2024-03-29T11:32:47.868816+0000","last_became_peered":"2024-03-29T11:32:47.868816+0000","last_unstale":"2024-03-29T11:32:49.193907+0000","last_undegraded":"2024-03-29T11:32:49.193907+0000","last_fullsized":"2024-03-29T11:32:49.193907+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:48.737111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.339534+0000","last_change":"2024-03-29T11:32:32.971041+0000","last_active":"2024-03-29T11:32:47.339534+0000","last_peered":"2024-03-29T11:32:47.339534+0000","last_clean":"2024-03-29T11:32:47.339534+0000","last_became_active":"2024-03-29T11:32:32.970781+0000","last_became_peered":"2024-03-29T11:32:32.970781+0000","last_unstale":"2024-03-29T11:32:47.339534+0000","last_undegraded":"2024-03-29T11:32:47.339534+0000","last_fullsized":"2024-03-29T11:32:47.339534+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:31:30.382747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.863169+0000","last_change":"2024-03-29T11:32:32.414137+0000","last_active":"2024-03-29T11:32:47.863169+0000","last_peered":"2024-03-29T11:32:47.863169+0000","last_clean":"2024-03-29T11:32:47.863169+0000","last_became_active":"2024-03-29T11:32:32.413846+0000","last_became_peered":"2024-03-29T11:32:32.413846+0000","last_unstale":"2024-03-29T11:32:47.863169+0000","last_undegraded":"2024-03-29T11:32:47.863169+0000","last_fullsized":"2024-03-29T11:32:47.863169+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:52:09.293759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.868742+0000","last_change":"2024-03-29T11:32:47.884754+0000","last_active":"2024-03-29T11:32:48.868742+0000","last_peered":"2024-03-29T11:32:48.868742+0000","last_clean":"2024-03-29T11:32:48.868742+0000","last_became_active":"2024-03-29T11:32:47.884252+0000","last_became_peered":"2024-03-29T11:32:47.884252+0000","last_unstale":"2024-03-29T11:32:48.868742+0000","last_undegraded":"2024-03-29T11:32:48.868742+0000","last_fullsized":"2024-03-29T11:32:48.868742+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:59:56.436348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.338171+0000","last_change":"2024-03-29T11:32:32.956222+0000","last_active":"2024-03-29T11:32:47.338171+0000","last_peered":"2024-03-29T11:32:47.338171+0000","last_clean":"2024-03-29T11:32:47.338171+0000","last_became_active":"2024-03-29T11:32:32.955784+0000","last_became_peered":"2024-03-29T11:32:32.955784+0000","last_unstale":"2024-03-29T11:32:47.338171+0000","last_undegraded":"2024-03-29T11:32:47.338171+0000","last_fullsized":"2024-03-29T11:32:47.338171+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:52.969721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.863793+0000","last_change":"2024-03-29T11:32:32.421490+0000","last_active":"2024-03-29T11:32:47.863793+0000","last_peered":"2024-03-29T11:32:47.863793+0000","last_clean":"2024-03-29T11:32:47.863793+0000","last_became_active":"2024-03-29T11:32:32.421236+0000","last_became_peered":"2024-03-29T11:32:32.421236+0000","last_unstale":"2024-03-29T11:32:47.863793+0000","last_undegraded":"2024-03-29T11:32:47.863793+0000","last_fullsized":"2024-03-29T11:32:47.863793+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:17.693634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.881856+0000","last_change":"2024-03-29T11:32:47.879901+0000","last_active":"2024-03-29T11:32:47.881856+0000","last_peered":"2024-03-29T11:32:47.881856+0000","last_clean":"2024-03-29T11:32:47.881856+0000","last_became_active":"2024-03-29T11:32:47.879634+0000","last_became_peered":"2024-03-29T11:32:47.879634+0000","last_unstale":"2024-03-29T11:32:47.881856+0000","last_undegraded":"2024-03-29T11:32:47.881856+0000","last_fullsized":"2024-03-29T11:32:47.881856+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:28:46.899353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.853526+0000","last_change":"2024-03-29T11:32:32.950145+0000","last_active":"2024-03-29T11:32:47.853526+0000","last_peered":"2024-03-29T11:32:47.853526+0000","last_clean":"2024-03-29T11:32:47.853526+0000","last_became_active":"2024-03-29T11:32:32.949444+0000","last_became_peered":"2024-03-29T11:32:32.949444+0000","last_unstale":"2024-03-29T11:32:47.853526+0000","last_undegraded":"2024-03-29T11:32:47.853526+0000","last_fullsized":"2024-03-29T11:32:47.853526+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:57:42.190121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.860138+0000","last_change":"2024-03-29T11:32:32.422243+0000","last_active":"2024-03-29T11:32:47.860138+0000","last_peered":"2024-03-29T11:32:47.860138+0000","last_clean":"2024-03-29T11:32:47.860138+0000","last_became_active":"2024-03-29T11:32:32.421949+0000","last_became_peered":"2024-03-29T11:32:32.421949+0000","last_unstale":"2024-03-29T11:32:47.860138+0000","last_undegraded":"2024-03-29T11:32:47.860138+0000","last_fullsized":"2024-03-29T11:32:47.860138+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:03.995789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-29T11:32:48.866667+0000","last_change":"2024-03-29T11:32:47.883873+0000","last_active":"2024-03-29T11:32:48.866667+0000","last_peered":"2024-03-29T11:32:48.866667+0000","last_clean":"2024-03-29T11:32:48.866667+0000","last_became_active":"2024-03-29T11:32:47.883381+0000","last_became_peered":"2024-03-29T11:32:47.883381+0000","last_unstale":"2024-03-29T11:32:48.866667+0000","last_undegraded":"2024-03-29T11:32:48.866667+0000","last_fullsized":"2024-03-29T11:32:48.866667+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:47.316425+0000","last_clean_scrub_stamp":"2024-03-29T11:32:47.316425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:24:08.340482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.861202+0000","last_change":"2024-03-29T11:32:32.962313+0000","last_active":"2024-03-29T11:32:47.861202+0000","last_peered":"2024-03-29T11:32:47.861202+0000","last_clean":"2024-03-29T11:32:47.861202+0000","last_became_active":"2024-03-29T11:32:32.961887+0000","last_became_peered":"2024-03-29T11:32:32.961887+0000","last_unstale":"2024-03-29T11:32:47.861202+0000","last_undegraded":"2024-03-29T11:32:47.861202+0000","last_fullsized":"2024-03-29T11:32:47.861202+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:32.394785+0000","last_clean_scrub_stamp":"2024-03-29T11:32:32.394785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:18.696940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"247'4","reported_seq":19,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.868945+0000","last_change":"2024-03-29T11:32:32.419906+0000","last_active":"2024-03-29T11:32:47.868945+0000","last_peered":"2024-03-29T11:32:47.868945+0000","last_clean":"2024-03-29T11:32:47.868945+0000","last_became_active":"2024-03-29T11:32:32.419653+0000","last_became_peered":"2024-03-29T11:32:32.419653+0000","last_unstale":"2024-03-29T11:32:47.868945+0000","last_undegraded":"2024-03-29T11:32:47.868945+0000","last_fullsized":"2024-03-29T11:32:47.868945+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:32:31.385452+0000","last_clean_scrub_stamp":"2024-03-29T11:32:31.385452+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:25:22.034114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"233'388","reported_seq":1371,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-29T11:32:47.877658+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:32:47.877658+0000","last_peered":"2024-03-29T11:32:47.877658+0000","last_clean":"2024-03-29T11:32:47.877658+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:32:47.877658+0000","last_undegraded":"2024-03-29T11:32:47.877658+0000","last_fullsized":"2024-03-29T11:32:47.877658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":388,"ondisk_log_size":388,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576009,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56944,"kb_used_data":3684,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314896,"statfs":{"total":96636764160,"available":96578453504,"internally_reserved":0,"allocated":3772416,"data_stored":2149430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.70799999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.73899999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42899999999999999},"max":{"1min":0.89900000000000002,"5min":0.98899999999999999,"15min":1.119},"last":0.76800000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.69299999999999995,"15min":0.72199999999999998},"min":{"1min":0.437,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90500000000000003,"5min":0.97399999999999998,"15min":1.478},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72799999999999998,"15min":0.75800000000000001},"min":{"1min":0.47399999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.393},"last":0.75}]},{"osd":2,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70599999999999996,"15min":0.748},"min":{"1min":0.504,"5min":0.436,"15min":0.436},"max":{"1min":0.85899999999999999,"5min":0.998,"15min":1.089},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92100000000000004,"5min":1.032,"15min":1.0720000000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.72299999999999998,"15min":0.753},"min":{"1min":0.54400000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90300000000000002,"5min":0.97499999999999998,"15min":1.0580000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77500000000000002,"15min":0.78300000000000003},"min":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.55300000000000005},"max":{"1min":1.0309999999999999,"5min":1.071,"15min":1.071},"last":0.79200000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":1.3779999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65900000000000003,"15min":0.66900000000000004},"min":{"1min":0.50900000000000001,"5min":0.41599999999999998,"15min":0.40899999999999997},"max":{"1min":0.85099999999999998,"5min":0.93500000000000005,"15min":1.335},"last":0.81699999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69799999999999995,"15min":0.68100000000000005},"min":{"1min":0.434,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.1160000000000001},"last":0.495},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.73899999999999999,"15min":0.72199999999999998},"min":{"1min":0.55600000000000005,"5min":0.50900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95499999999999996,"5min":1.0169999999999999,"15min":1.0249999999999999},"last":0.54200000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.76900000000000002,"15min":0.76700000000000002},"min":{"1min":0.52400000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0549999999999999,"15min":1.1319999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.755,"15min":0.74099999999999999},"min":{"1min":0.46899999999999997,"5min":0.439,"15min":0.435},"max":{"1min":1.0580000000000001,"5min":1.071,"15min":1.071},"last":0.64500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834576010,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48176,"kb_used_data":3100,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323664,"statfs":{"total":96636764160,"available":96587431936,"internally_reserved":0,"allocated":3174400,"data_stored":1552399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69699999999999995,"15min":0.72899999999999998},"min":{"1min":0.48699999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.74299999999999999,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70899999999999996,"15min":0.751},"min":{"1min":0.40200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95199999999999996,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.70399999999999996,"15min":0.73099999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.022,"5min":1.98,"15min":1.98},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72899999999999998,"15min":0.76400000000000001},"min":{"1min":0.505,"5min":0.435,"15min":0.435},"max":{"1min":1.012,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.745}]},{"osd":2,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.72599999999999998,"15min":0.746},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92800000000000005,"5min":1.669,"15min":1.669},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.73599999999999999,"15min":0.75800000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.46200000000000002},"max":{"1min":1,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.505}]},{"osd":3,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.73599999999999999,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.997,"5min":2.52,"15min":2.52},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77900000000000003,"15min":0.79800000000000004},"min":{"1min":0.53800000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.105,"5min":1.877,"15min":1.877},"last":0.70199999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63600000000000001,"15min":0.61399999999999999},"min":{"1min":0.45200000000000001,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":0.92300000000000004,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.62},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.56100000000000005,"5min":0.47399999999999998,"15min":0.45900000000000002},"max":{"1min":1.2709999999999999,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":0.72499999999999998}]},{"osd":5,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.66000000000000003},"min":{"1min":0.42799999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0900000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.72199999999999998,"15min":0.71799999999999997},"min":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.42199999999999999},"max":{"1min":1.0429999999999999,"5min":1.925,"15min":1.925},"last":0.56399999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.78000000000000003,"15min":0.76900000000000002},"min":{"1min":0.66800000000000004,"5min":0.54800000000000004,"15min":0.52700000000000002},"max":{"1min":1.0509999999999999,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.751,"15min":0.74399999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1220000000000001,"5min":1.726,"15min":1.726},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576008,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60484,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311356,"statfs":{"total":96636764160,"available":96574828544,"internally_reserved":0,"allocated":3194880,"data_stored":1555388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.44500000000000001,"15min":0.46700000000000003},"min":{"1min":0.29299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69799999999999995,"5min":0.79100000000000004,"15min":0.80300000000000005},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.50900000000000001,"15min":0.53200000000000003},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.57199999999999995,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.56200000000000006},"min":{"1min":0.36499999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.52300000000000002,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.34799999999999998},"max":{"1min":0.61599999999999999,"5min":0.81399999999999995,"15min":0.90300000000000002},"last":0.45400000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.59599999999999997},"min":{"1min":0.433,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73099999999999998,"5min":0.86199999999999999,"15min":0.995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56799999999999995,"15min":0.58099999999999996},"min":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.83999999999999997,"15min":0.878},"last":0.42699999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.75,"15min":0.78700000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.056,"5min":1.153,"15min":1.3660000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.76200000000000001,"15min":0.80000000000000004},"min":{"1min":0.56299999999999994,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97699999999999998,"5min":1.226,"15min":1.304},"last":0.81899999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.79300000000000004,"15min":0.81999999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":0.997,"5min":1.276,"15min":1.3720000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.099,"5min":1.1299999999999999,"15min":1.3520000000000001},"last":0.90600000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.78300000000000003,"15min":0.81899999999999995},"min":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.026,"5min":1.1890000000000001,"15min":1.1919999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.042,"5min":1.2250000000000001,"15min":1.3069999999999999},"last":0.877}]},{"osd":7,"last update":"Fri Mar 29 11:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.83999999999999997,"15min":0.86899999999999999},"min":{"1min":0.54800000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0649999999999999,"5min":1.327,"15min":1.3540000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83399999999999996,"15min":0.86799999999999999},"min":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0700000000000001,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.78300000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576010,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61028,"kb_used_data":3664,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310812,"statfs":{"total":96636764160,"available":96574271488,"internally_reserved":0,"allocated":3751936,"data_stored":2141066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44900000000000001,"15min":0.46100000000000002},"min":{"1min":0.317,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.623,"5min":0.95899999999999996,"15min":1.087},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.52200000000000002,"15min":0.53500000000000003},"min":{"1min":0.32400000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67200000000000004,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.374}]},{"osd":2,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56000000000000005,"15min":0.57899999999999996},"min":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66600000000000004,"5min":1.028,"15min":1.5289999999999999},"last":0.437},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.499,"15min":0.51600000000000001},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.629,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55600000000000005,"15min":0.56200000000000006},"min":{"1min":0.38500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77600000000000002,"5min":1.222,"15min":1.222},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.622,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.44,"15min":0.42199999999999999},"max":{"1min":0.81799999999999995,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.56599999999999995}]},{"osd":4,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80500000000000005,"15min":0.81399999999999995},"min":{"1min":0.56899999999999995,"5min":0.55600000000000005,"15min":0.495},"max":{"1min":0.93400000000000005,"5min":1.248,"15min":1.248},"last":0.876},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82399999999999995,"15min":0.84599999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0309999999999999,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.89800000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76800000000000002,"15min":0.78500000000000003},"min":{"1min":0.54500000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.88900000000000001,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80400000000000005,"15min":0.81899999999999995},"min":{"1min":0.58999999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94899999999999995,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.80100000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.83399999999999996,"15min":0.85299999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.97599999999999998,"5min":1.343,"15min":1.6240000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.85699999999999998,"15min":0.88200000000000001},"min":{"1min":0.62,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.004,"5min":1.3240000000000001,"15min":1.337},"last":0.96399999999999997}]},{"osd":7,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79900000000000004,"15min":0.82399999999999995},"min":{"1min":0.56899999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.99099999999999999,"5min":1.115,"15min":1.214},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.84099999999999997,"15min":0.86899999999999999},"min":{"1min":0.55500000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.054,"5min":1.365,"15min":1.365},"last":0.93500000000000005}]}]},{"osd":2,"up_from":13,"seq":55834576009,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3162112,"data_stored":1551773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47799999999999998,"15min":0.47599999999999998},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73199999999999998,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.502},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.54100000000000004},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67600000000000005,"5min":0.73999999999999999,"15min":0.96799999999999997},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74299999999999999,"5min":0.77400000000000002,"15min":0.80700000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55500000000000005,"15min":0.57699999999999996},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68300000000000005,"5min":0.872,"15min":0.88800000000000001},"last":0.54100000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.42299999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86799999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76800000000000002,"5min":1.038,"15min":1.038},"last":0.47099999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.76300000000000001,"15min":0.80300000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.93700000000000006,"5min":1.091,"15min":1.1970000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.755,"15min":0.80800000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.52400000000000002},"max":{"1min":1.026,"5min":1.244,"15min":1.244},"last":0.64500000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.81299999999999994},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.53400000000000003},"max":{"1min":0.94599999999999995,"5min":1.1599999999999999,"15min":1.256},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.78000000000000003,"15min":0.82199999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.995,"5min":1.143,"15min":1.3140000000000001},"last":0.66000000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.79300000000000004,"15min":0.84199999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98099999999999998,"5min":1.1459999999999999,"15min":1.24},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79700000000000004,"15min":0.83999999999999997},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.97399999999999998,"5min":1.099,"15min":1.329},"last":0.63500000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.83899999999999997,"15min":0.88800000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":0.95899999999999996,"5min":1.1910000000000001,"15min":1.2889999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85499999999999998,"15min":0.90200000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.56899999999999995},"max":{"1min":1.095,"5min":1.2969999999999999,"15min":1.304},"last":0.81399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834576009,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48180,"kb_used_data":3104,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323660,"statfs":{"total":96636764160,"available":96587427840,"internally_reserved":0,"allocated":3178496,"data_stored":1554411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.29599999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.33200000000000002,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48399999999999999,"15min":0.49399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.57199999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.38700000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.503}]},{"osd":4,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79600000000000004,"15min":0.79700000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.80300000000000005,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80500000000000005,"15min":0.81299999999999994},"min":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.502},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.83099999999999996,"15min":0.82699999999999996},"min":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.54600000000000004},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":0.73399999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86599999999999999,"15min":0.85899999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.84099999999999997,"15min":0.84299999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":0.66800000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86699999999999999,"15min":0.86599999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.58099999999999996},"max":{"1min":2.444,"5min":2.444,"15min":2.444},"last":0.85399999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576010,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60484,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311356,"statfs":{"total":96636764160,"available":96574828544,"internally_reserved":0,"allocated":3198976,"data_stored":1556200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.66400000000000003,"15min":0.71499999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83899999999999997,"5min":0.95499999999999996,"15min":1.0509999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.67500000000000004,"15min":0.72099999999999997},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.93600000000000005,"5min":0.99299999999999999,"15min":1.1459999999999999},"last":0.42299999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.68300000000000005,"15min":0.73699999999999999},"min":{"1min":0.436,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69199999999999995,"15min":0.73799999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91400000000000003,"5min":0.94899999999999995,"15min":1.002},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.77900000000000003,"15min":0.82199999999999995},"min":{"1min":0.55400000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.004,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.76500000000000001,"15min":0.80600000000000005},"min":{"1min":0.503,"5min":0.496,"15min":0.496},"max":{"1min":1.0029999999999999,"5min":1.224,"15min":1.224},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71399999999999997,"15min":0.75900000000000001},"min":{"1min":0.54000000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.92600000000000005,"5min":1.0580000000000001,"15min":1.181},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.74399999999999999,"15min":0.76600000000000001},"min":{"1min":0.499,"5min":0.499,"15min":0.48699999999999999},"max":{"1min":0.95199999999999996,"5min":1.01,"15min":1.2490000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63800000000000001,"15min":0.625},"min":{"1min":0.45400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64500000000000002,"15min":0.65300000000000002},"min":{"1min":0.52000000000000002,"5min":0.41599999999999998,"15min":0.39700000000000002},"max":{"1min":0.81499999999999995,"5min":1.0920000000000001,"15min":1.169},"last":0.5}]},{"osd":6,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.50600000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":0.93100000000000005,"5min":1.056,"15min":1.0800000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.72199999999999998,"15min":0.71999999999999997},"min":{"1min":0.59199999999999997,"5min":0.48199999999999998,"15min":0.47699999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.56399999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73599999999999999,"15min":0.73299999999999998},"min":{"1min":0.60599999999999998,"5min":0.501,"15min":0.436},"max":{"1min":0.97999999999999998,"5min":1.0389999999999999,"15min":1.2669999999999999},"last":0.626},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81200000000000006,"15min":0.82199999999999995},"min":{"1min":0.65900000000000003,"5min":0.58499999999999996,"15min":0.54400000000000004},"max":{"1min":1.03,"5min":1.1619999999999999,"15min":1.2969999999999999},"last":0.61299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576009,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46488,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325352,"statfs":{"total":96636764160,"available":96589160448,"internally_reserved":0,"allocated":3158016,"data_stored":1551683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.69799999999999995},"min":{"1min":0.47199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.72299999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.95599999999999996,"15min":1.0680000000000001},"last":0.67300000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73699999999999999,"15min":0.75700000000000001},"min":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98099999999999998,"5min":1.839,"15min":1.839},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.71899999999999997,"15min":0.73599999999999999},"min":{"1min":0.504,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.282,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.64800000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.753,"15min":0.755},"min":{"1min":0.51000000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.625},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.71499999999999997,"15min":0.74399999999999999},"min":{"1min":0.54400000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.76100000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.48499999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.95099999999999996,"5min":3.444,"15min":3.444},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.75800000000000001,"15min":0.77200000000000002},"min":{"1min":0.54400000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.72899999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.629,"15min":0.60899999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38700000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67800000000000005,"15min":0.65300000000000002},"min":{"1min":0.47799999999999998,"5min":0.41699999999999998,"15min":0.40600000000000003},"max":{"1min":0.999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.48999999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.69699999999999995,"15min":0.67000000000000004},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72199999999999998,"15min":0.70299999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.107},"last":0.51700000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.746,"15min":0.72599999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.42299999999999999},"max":{"1min":1.04,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79600000000000004,"15min":0.77400000000000002},"min":{"1min":0.53500000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:32:53.349 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T11:32:53.350 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:32:53.378 INFO:tasks.ceph.mds.d.smithi159.stderr:2024-03-29T11:32:53.383+0000 7efd3a600700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.584+0000 7f3498400700 1 Processor -- start 2024-03-29T11:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.584+0000 7f3498400700 1 -- start start 2024-03-29T11:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.585+0000 7f3498400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f349009aba0 0x7f349006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.585+0000 7f3498400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f349006c480 0x7f34900ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.585+0000 7f3498400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349006c8d0 con 0x7f349015d710 2024-03-29T11:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.585+0000 7f3498400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349006bb40 con 0x7f349009aba0 2024-03-29T11:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.585+0000 7f3498400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3490099b00 con 0x7f349006c480 2024-03-29T11:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.585+0000 7f3495a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f349015d710 0x7f34900d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59586/0 (socket says 172.21.15.12:59586) 2024-03-29T11:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.585+0000 7f3495a00700 1 -- 172.21.15.12:0/2971477062 learned_addr learned my addr 172.21.15.12:0/2971477062 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.585+0000 7f3495000700 1 --2- 172.21.15.12:0/2971477062 >> v2:172.21.15.12:3300/0 conn(0x7f349009aba0 0x7f349006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f3496400700 1 --2- 172.21.15.12:0/2971477062 >> v2:172.21.15.159:3300/0 conn(0x7f349006c480 0x7f34900ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f348fe00700 1 -- 172.21.15.12:0/2971477062 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139580308 0 0) 0x7f349006c8d0 con 0x7f349015d710 2024-03-29T11:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f3495000700 1 -- 172.21.15.12:0/2971477062 >> v2:172.21.15.159:3300/0 conn(0x7f349006c480 msgr2=0x7f34900ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f3495000700 1 --2- 172.21.15.12:0/2971477062 >> v2:172.21.15.159:3300/0 conn(0x7f349006c480 0x7f34900ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f3495000700 1 -- 172.21.15.12:0/2971477062 >> v1:172.21.15.12:6789/0 conn(0x7f349015d710 legacy=0x7f34900d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f3495000700 1 -- 172.21.15.12:0/2971477062 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3490099de0 con 0x7f349009aba0 2024-03-29T11:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f3496400700 1 --2- 172.21.15.12:0/2971477062 >> v2:172.21.15.159:3300/0 conn(0x7f349006c480 0x7f34900ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f3495000700 1 --2- 172.21.15.12:0/2971477062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349009aba0 0x7f349006bf40 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f34900722c0 tx=0x7f347c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43023cfe03dce6c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f348fe00700 1 -- 172.21.15.12:0/2971477062 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c011830 con 0x7f349009aba0 2024-03-29T11:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f348fe00700 1 -- 172.21.15.12:0/2971477062 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347c011990 con 0x7f349009aba0 2024-03-29T11:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.586+0000 7f348fe00700 1 -- 172.21.15.12:0/2971477062 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c011d20 con 0x7f349009aba0 2024-03-29T11:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.587+0000 7f3498400700 1 -- 172.21.15.12:0/2971477062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349009aba0 msgr2=0x7f349006bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.587+0000 7f3498400700 1 --2- 172.21.15.12:0/2971477062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349009aba0 0x7f349006bf40 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f34900722c0 tx=0x7f347c009b20 comp rx=0 tx=0).stop 2024-03-29T11:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.587+0000 7f3498400700 1 -- 172.21.15.12:0/2971477062 shutdown_connections 2024-03-29T11:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.587+0000 7f3498400700 1 --2- 172.21.15.12:0/2971477062 >> v2:172.21.15.159:3300/0 conn(0x7f349006c480 0x7f34900ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.587+0000 7f3498400700 1 --2- 172.21.15.12:0/2971477062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349009aba0 0x7f349006bf40 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.587+0000 7f3498400700 1 -- 172.21.15.12:0/2971477062 >> 172.21.15.12:0/2971477062 conn(0x7f34900d13e0 msgr2=0x7f34900d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.587+0000 7f3498400700 1 -- 172.21.15.12:0/2971477062 shutdown_connections 2024-03-29T11:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.587+0000 7f3498400700 1 -- 172.21.15.12:0/2971477062 wait complete. 2024-03-29T11:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.589+0000 7f3498400700 1 Processor -- start 2024-03-29T11:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.589+0000 7f3498400700 1 -- start start 2024-03-29T11:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3498400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f349006c480 0x7f34901551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3498400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f349009aba0 0x7f349006f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3498400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349015d710 0x7f349006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3498400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34901f2e80 con 0x7f349015d710 2024-03-29T11:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3498400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349015db90 con 0x7f349009aba0 2024-03-29T11:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3498400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349015de70 con 0x7f349006c480 2024-03-29T11:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3496400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349015d710 0x7f349006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3496400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349015d710 0x7f349006fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46394/0 (socket says 172.21.15.12:46394) 2024-03-29T11:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3496400700 1 -- 172.21.15.12:0/481680861 learned_addr learned my addr 172.21.15.12:0/481680861 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3495a00700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f349006c480 0x7f34901551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.590+0000 7f3495000700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f349009aba0 0x7f349006f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f3496400700 1 -- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f349006c480 msgr2=0x7f34901551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f3496400700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f349006c480 0x7f34901551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f3496400700 1 -- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f349009aba0 msgr2=0x7f349006f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f3496400700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f349009aba0 0x7f349006f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f3496400700 1 -- 172.21.15.12:0/481680861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c0094a0 con 0x7f349015d710 2024-03-29T11:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f3496400700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349015d710 0x7f349006fcd0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f3488007ef0 tx=0x7f348800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f348d600700 1 -- 172.21.15.12:0/481680861 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348800dd80 con 0x7f349015d710 2024-03-29T11:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f348d600700 1 -- 172.21.15.12:0/481680861 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348800f040 con 0x7f349015d710 2024-03-29T11:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f348d600700 1 -- 172.21.15.12:0/481680861 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488003c60 con 0x7f349015d710 2024-03-29T11:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f3498400700 1 -- 172.21.15.12:0/481680861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349015e150 con 0x7f349015d710 2024-03-29T11:32:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.591+0000 7f3498400700 1 -- 172.21.15.12:0/481680861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3490155b50 con 0x7f349015d710 2024-03-29T11:32:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.592+0000 7f3498400700 1 -- 172.21.15.12:0/481680861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3490067720 con 0x7f349015d710 2024-03-29T11:32:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.593+0000 7f348d600700 1 -- 172.21.15.12:0/481680861 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f3488004030 con 0x7f349015d710 2024-03-29T11:32:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.594+0000 7f348d600700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f347006aed0 0x7f347006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.594+0000 7f348d600700 1 -- 172.21.15.12:0/481680861 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f3488089d00 con 0x7f349015d710 2024-03-29T11:32:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.594+0000 7f3495a00700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f347006aed0 0x7f347006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.595+0000 7f3495a00700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f347006aed0 0x7f347006d390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3484000f60 tx=0x7f3484007e70 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.598+0000 7f348d600700 1 -- 172.21.15.12:0/481680861 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3488054620 con 0x7f349015d710 2024-03-29T11:32:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.861+0000 7f3498400700 1 -- 172.21.15.12:0/481680861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34900d0f20 con 0x7f349015d710 2024-03-29T11:32:53.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.861+0000 7f348d600700 1 -- 172.21.15.12:0/481680861 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5972 (secure 0 0 0) 0x7f348801a090 con 0x7f349015d710 2024-03-29T11:32:53.857 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:32:53.857 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":128,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:32.933737+0000","modified":"2024-03-29T11:32:36.420698+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11853":{"gid":11853,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3711953177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3711953177},{"type":"v1","addr":"172.21.15.12:6834","nonce":3711953177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:47.849181+0000","modified":"2024-03-29T11:32:51.340904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11416":{"gid":11416,"name":"d","rank":0,"incarnation":124,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 -- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f347006aed0 msgr2=0x7f347006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f347006aed0 0x7f347006d390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3484000f60 tx=0x7f3484007e70 comp rx=0 tx=0).stop 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 -- 172.21.15.12:0/481680861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349015d710 msgr2=0x7f349006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349015d710 0x7f349006fcd0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f3488007ef0 tx=0x7f348800d3b0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 -- 172.21.15.12:0/481680861 shutdown_connections 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f347006aed0 0x7f347006d390 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f349015d710 0x7f349006fcd0 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f349006c480 0x7f34901551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 --2- 172.21.15.12:0/481680861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f349009aba0 0x7f349006f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 -- 172.21.15.12:0/481680861 >> 172.21.15.12:0/481680861 conn(0x7f34900d13e0 msgr2=0x7f34900af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 -- 172.21.15.12:0/481680861 shutdown_connections 2024-03-29T11:32:53.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:53.864+0000 7f3498400700 1 -- 172.21.15.12:0/481680861 wait complete. 2024-03-29T11:32:53.861 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 128 2024-03-29T11:32:54.281 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 128, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T11:32:47.849181+0000', 'modified': '2024-03-29T11:32:51.340904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11416}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11416': {'gid': 11416, 'name': 'd', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.159:6837/1262061639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6835', 'nonce': 1262061639}, {'type': 'v1', 'addr': '172.21.15.159:6837', 'nonce': 1262061639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T11:32:54.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T11:32:54.282 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-29T11:32:54.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.491+0000 7feb4ca00700 1 Processor -- start 2024-03-29T11:32:54.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.491+0000 7feb4ca00700 1 -- start start 2024-03-29T11:32:54.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.492+0000 7feb4ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb48099a60 0x7feb480d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.492+0000 7feb4ca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb4809a750 0x7feb4806c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.492+0000 7feb4ca00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb480d0970 con 0x7feb4815d990 2024-03-29T11:32:54.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.492+0000 7feb4ca00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4806c020 con 0x7feb4809a750 2024-03-29T11:32:54.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.492+0000 7feb4ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb480c81a0 con 0x7feb48099a60 2024-03-29T11:32:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.492+0000 7feb45400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7feb4815d990 0x7feb48099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59598/0 (socket says 172.21.15.12:59598) 2024-03-29T11:32:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.492+0000 7feb45400700 1 -- 172.21.15.12:0/475934329 learned_addr learned my addr 172.21.15.12:0/475934329 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.492+0000 7feb46800700 1 --2- 172.21.15.12:0/475934329 >> v2:172.21.15.12:3300/0 conn(0x7feb4809a750 0x7feb4806c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.493+0000 7feb45e00700 1 --2- 172.21.15.12:0/475934329 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb48099a60 0x7feb480d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.493+0000 7feb44a00700 1 -- 172.21.15.12:0/475934329 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425291557 0 0) 0x7feb480d0970 con 0x7feb4815d990 2024-03-29T11:32:54.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.493+0000 7feb44a00700 1 -- 172.21.15.12:0/475934329 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb2c0035b0 con 0x7feb4815d990 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.493+0000 7feb46800700 1 -- 172.21.15.12:0/475934329 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb48099a60 msgr2=0x7feb480d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.493+0000 7feb46800700 1 --2- 172.21.15.12:0/475934329 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb48099a60 0x7feb480d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.493+0000 7feb46800700 1 -- 172.21.15.12:0/475934329 >> v1:172.21.15.12:6789/0 conn(0x7feb4815d990 legacy=0x7feb48099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.493+0000 7feb46800700 1 -- 172.21.15.12:0/475934329 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb480d0690 con 0x7feb4809a750 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.493+0000 7feb46800700 1 --2- 172.21.15.12:0/475934329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb4809a750 0x7feb4806c4b0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7feb40009d90 tx=0x7feb4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb197f45218227b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.493+0000 7feb44a00700 1 -- 172.21.15.12:0/475934329 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4000e070 con 0x7feb4809a750 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.493+0000 7feb44a00700 1 -- 172.21.15.12:0/475934329 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4000bb20 con 0x7feb4809a750 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.494+0000 7feb44a00700 1 -- 172.21.15.12:0/475934329 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40012430 con 0x7feb4809a750 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.494+0000 7feb4ca00700 1 -- 172.21.15.12:0/475934329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb4809a750 msgr2=0x7feb4806c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.494+0000 7feb4ca00700 1 --2- 172.21.15.12:0/475934329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb4809a750 0x7feb4806c4b0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7feb40009d90 tx=0x7feb4000b330 comp rx=0 tx=0).stop 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.494+0000 7feb4ca00700 1 -- 172.21.15.12:0/475934329 shutdown_connections 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.494+0000 7feb4ca00700 1 --2- 172.21.15.12:0/475934329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb4809a750 0x7feb4806c4b0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.494+0000 7feb4ca00700 1 --2- 172.21.15.12:0/475934329 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb48099a60 0x7feb480d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.494+0000 7feb4ca00700 1 -- 172.21.15.12:0/475934329 >> 172.21.15.12:0/475934329 conn(0x7feb480b2e20 msgr2=0x7feb480b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:54.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.494+0000 7feb4ca00700 1 -- 172.21.15.12:0/475934329 shutdown_connections 2024-03-29T11:32:54.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.494+0000 7feb4ca00700 1 -- 172.21.15.12:0/475934329 wait complete. 2024-03-29T11:32:54.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.496+0000 7feb4ca00700 1 Processor -- start 2024-03-29T11:32:54.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.496+0000 7feb4ca00700 1 -- start start 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.497+0000 7feb4ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb48099a60 0x7feb48151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.497+0000 7feb4ca00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feb4809a750 0x7feb48160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.497+0000 7feb4ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4815d990 0x7feb480b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.497+0000 7feb4ca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb48160590 con 0x7feb48099a60 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.497+0000 7feb4ca00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb480b6120 con 0x7feb4809a750 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.497+0000 7feb4ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb480c6fa0 con 0x7feb4815d990 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.497+0000 7feb45e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb48099a60 0x7feb48151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.497+0000 7feb45e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb48099a60 0x7feb48151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46402/0 (socket says 172.21.15.12:46402) 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.497+0000 7feb45e00700 1 -- 172.21.15.12:0/2544451931 learned_addr learned my addr 172.21.15.12:0/2544451931 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb45400700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feb4809a750 0x7feb48160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb46800700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4815d990 0x7feb480b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb45e00700 1 -- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4815d990 msgr2=0x7feb480b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb45e00700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4815d990 0x7feb480b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb45e00700 1 -- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feb4809a750 msgr2=0x7feb48160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb45e00700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feb4809a750 0x7feb48160050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb45e00700 1 -- 172.21.15.12:0/2544451931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40009a70 con 0x7feb48099a60 2024-03-29T11:32:54.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb45e00700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb48099a60 0x7feb48151110 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7feb3800e870 tx=0x7feb3800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:54.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb3e000700 1 -- 172.21.15.12:0/2544451931 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb38009590 con 0x7feb48099a60 2024-03-29T11:32:54.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb3e000700 1 -- 172.21.15.12:0/2544451931 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3800ee70 con 0x7feb48099a60 2024-03-29T11:32:54.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb3e000700 1 -- 172.21.15.12:0/2544451931 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800ca50 con 0x7feb48099a60 2024-03-29T11:32:54.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb4ca00700 1 -- 172.21.15.12:0/2544451931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb480b63a0 con 0x7feb48099a60 2024-03-29T11:32:54.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.498+0000 7feb4ca00700 1 -- 172.21.15.12:0/2544451931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb480b6960 con 0x7feb48099a60 2024-03-29T11:32:54.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.501+0000 7feb3e000700 1 -- 172.21.15.12:0/2544451931 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7feb3800cbb0 con 0x7feb48099a60 2024-03-29T11:32:54.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.502+0000 7feb3e000700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7feb2806aed0 0x7feb2806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.502+0000 7feb3e000700 1 -- 172.21.15.12:0/2544451931 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7feb38089b50 con 0x7feb48099a60 2024-03-29T11:32:54.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.502+0000 7feb45400700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7feb2806aed0 0x7feb2806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.503+0000 7feb45400700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7feb2806aed0 0x7feb2806d390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feb480cb730 tx=0x7feb34009040 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:54.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.504+0000 7feb604c4b80 1 Processor -- start 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.504+0000 7feb604c4b80 1 -- start start 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba239830 0x561eba239c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba20fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 0x561eba236a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb604c4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eba184650 con 0x561eba20cbe0 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb604c4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eba165c30 con 0x561eba239830 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb604c4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x561eba163430 con 0x561eba20ab60 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb45400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba20fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb45e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba239830 0x561eba239c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb45400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba20fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46410/0 (socket says 172.21.15.12:46410) 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb45400700 1 -- 172.21.15.12:0/3530331009 learned_addr learned my addr 172.21.15.12:0/3530331009 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.505+0000 7feb46800700 1 --2- 172.21.15.12:0/3530331009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 0x561eba236a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.506+0000 7feb45400700 1 -- 172.21.15.12:0/3530331009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 msgr2=0x561eba236a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.506+0000 7feb45400700 1 --2- 172.21.15.12:0/3530331009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 0x561eba236a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.506+0000 7feb45400700 1 -- 172.21.15.12:0/3530331009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba239830 msgr2=0x561eba239c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.506+0000 7feb45400700 1 --2- 172.21.15.12:0/3530331009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba239830 0x561eba239c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.506+0000 7feb45400700 1 -- 172.21.15.12:0/3530331009 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561eb9f52790 con 0x561eba20cbe0 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.506+0000 7feb45400700 1 --2- 172.21.15.12:0/3530331009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba20fa10 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7feb34005af0 tx=0x7feb34011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12691fed9dba17b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.506+0000 7feb32600700 1 -- 172.21.15.12:0/3530331009 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34005e00 con 0x561eba20cbe0 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.506+0000 7feb32600700 1 -- 172.21.15.12:0/3530331009 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb34007770 con 0x561eba20cbe0 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.506+0000 7feb32600700 1 -- 172.21.15.12:0/3530331009 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34021670 con 0x561eba20cbe0 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.507+0000 7feb604c4b80 1 -- 172.21.15.12:0/3530331009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 msgr2=0x561eba20fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.507+0000 7feb604c4b80 1 --2- 172.21.15.12:0/3530331009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba20fa10 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7feb34005af0 tx=0x7feb34011000 comp rx=0 tx=0).stop 2024-03-29T11:32:54.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.507+0000 7feb604c4b80 1 -- 172.21.15.12:0/3530331009 shutdown_connections 2024-03-29T11:32:54.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.507+0000 7feb604c4b80 1 --2- 172.21.15.12:0/3530331009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 0x561eba236a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.507+0000 7feb604c4b80 1 --2- 172.21.15.12:0/3530331009 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba239830 0x561eba239c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.507+0000 7feb604c4b80 1 --2- 172.21.15.12:0/3530331009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba20fa10 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.507+0000 7feb604c4b80 1 -- 172.21.15.12:0/3530331009 >> 172.21.15.12:0/3530331009 conn(0x561eba16d6a0 msgr2=0x561eba254640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:54.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.507+0000 7feb604c4b80 1 -- 172.21.15.12:0/3530331009 shutdown_connections 2024-03-29T11:32:54.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.507+0000 7feb604c4b80 1 -- 172.21.15.12:0/3530331009 wait complete. 2024-03-29T11:32:54.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.508+0000 7feb604c4b80 1 -- start start 2024-03-29T11:32:54.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.508+0000 7feb604c4b80 1 Processor -- start 2024-03-29T11:32:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20ab60 0x561eba263ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba237170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba239830 0x561eba25bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb604c4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eba15ed20 con 0x561eba20cbe0 2024-03-29T11:32:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb604c4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eba0fa770 con 0x561eba20ab60 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb604c4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x561eba11d6f0 con 0x561eba239830 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb604c4b80 10 client.0 _ll_register_callbacks cb 0x561eba1045e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb45400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba237170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb45400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba237170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46418/0 (socket says 172.21.15.12:46418) 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb45400700 1 -- 172.21.15.12:0/3051309263 learned_addr learned my addr 172.21.15.12:0/3051309263 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb46800700 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba239830 0x561eba25bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb45e00700 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20ab60 0x561eba263ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb45400700 1 -- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba239830 msgr2=0x561eba25bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb45400700 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba239830 0x561eba25bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb45400700 1 -- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20ab60 msgr2=0x561eba263ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb45400700 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20ab60 0x561eba263ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.509+0000 7feb45400700 1 -- 172.21.15.12:0/3051309263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3400f040 con 0x561eba20cbe0 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.510+0000 7feb45400700 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba237170 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7feb34007ef0 tx=0x7feb34007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cba58e0298f0d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.510+0000 7feb26a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.510+0000 7feb26a00700 1 -- 172.21.15.12:0/3051309263 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34028040 con 0x561eba20cbe0 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.510+0000 7feb26a00700 1 -- 172.21.15.12:0/3051309263 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7feb34005e00 con 0x561eba20cbe0 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.510+0000 7feb604c4b80 1 -- 172.21.15.12:0/3051309263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561eba201920 con 0x561eba20cbe0 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.510+0000 7feb26a00700 1 -- 172.21.15.12:0/3051309263 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=128) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb34021a20 con 0x561eba20cbe0 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.510+0000 7feb604c4b80 10 client.12141 fetch_fsmap learned FSMap version 128 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.510+0000 7feb604c4b80 1 -- 172.21.15.12:0/3051309263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=128,monmap=2+}) v3 -- 0x561eba03f0a0 con 0x561eba20cbe0 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.510+0000 7feb26a00700 1 -- 172.21.15.12:0/3051309263 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb34031070 con 0x561eba20cbe0 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.511+0000 7feb26a00700 1 -- 172.21.15.12:0/3051309263 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 128) v1 ==== 3681+0+0 (secure 0 0 0) 0x7feb34021b80 con 0x561eba20cbe0 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.511+0000 7feb604c4b80 10 client.12141 fetch_fsmap finished waiting for FSMap version 128 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.511+0000 7feb604c4b80 10 client.12141 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.511+0000 7feb604c4b80 20 client.12141 populate_metadata read hostname 'smithi012' 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.511+0000 7feb604c4b80 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:6835/1262061639,v1:172.21.15.159:6837/1262061639] conn(0x561eba23cae0 0x561eba23cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.511+0000 7feb604c4b80 4 client.12141 mds_command: new command op to 11416 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.511+0000 7feb604c4b80 1 -- 172.21.15.12:0/3051309263 --> [v2:172.21.15.159:6835/1262061639,v1:172.21.15.159:6837/1262061639] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561eba1de6f0 con 0x561eba23cae0 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.511+0000 7feb45e00700 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:6835/1262061639,v1:172.21.15.159:6837/1262061639] conn(0x561eba23cae0 0x561eba23cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.512+0000 7feb45e00700 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:6835/1262061639,v1:172.21.15.159:6837/1262061639] conn(0x561eba23cae0 0x561eba23cec0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95908b483310c16b server_cookie=499a482cb988f475 in_seq=0 out_seq=0 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.512+0000 7feb26a00700 10 client.12141 ms_handle_connect on v2:172.21.15.159:6835/1262061639 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb26a00700 1 -- 172.21.15.12:0/3051309263 <== mds.0 v2:172.21.15.159:6835/1262061639 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561eba1de6f0 con 0x561eba23cae0 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb26a00700 10 client.12141 handle_command_reply: tid=0 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 client.12141 shutdown 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 -- 172.21.15.12:0/3051309263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 msgr2=0x561eba237170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba237170 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7feb34007ef0 tx=0x7feb34007000 comp rx=0 tx=0).stop 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 -- 172.21.15.12:0/3051309263 shutdown_connections 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:6835/1262061639,v1:172.21.15.159:6837/1262061639] conn(0x561eba23cae0 0x561eba23cec0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba239830 0x561eba25bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20ab60 0x561eba263ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 --2- 172.21.15.12:0/3051309263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba20cbe0 0x561eba237170 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb26a00700 0 client.12141 ms_handle_reset on v2:172.21.15.159:6835/1262061639 2024-03-29T11:32:54.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 -- 172.21.15.12:0/3051309263 >> 172.21.15.12:0/3051309263 conn(0x561eba16d6a0 msgr2=0x561eba20ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:54.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 -- 172.21.15.12:0/3051309263 shutdown_connections 2024-03-29T11:32:54.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 1 -- 172.21.15.12:0/3051309263 wait complete. 2024-03-29T11:32:54.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.513+0000 7feb604c4b80 20 client.12141 trim_cache size 0 max 16384 2024-03-29T11:32:54.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.532+0000 7feb604c4b80 1 Processor -- start 2024-03-29T11:32:54.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.532+0000 7feb604c4b80 1 -- start start 2024-03-29T11:32:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.533+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20ab60 0x561eba238a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.533+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20cbe0 0x561eba277350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.533+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.533+0000 7feb604c4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eb9f53390 con 0x561eba239830 2024-03-29T11:32:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.533+0000 7feb604c4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eba205a20 con 0x561eba20ab60 2024-03-29T11:32:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.533+0000 7feb604c4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x561eba210630 con 0x561eba20cbe0 2024-03-29T11:32:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.533+0000 7feb46800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.533+0000 7feb46800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46422/0 (socket says 172.21.15.12:46422) 2024-03-29T11:32:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.534+0000 7feb46800700 1 -- 172.21.15.12:0/1494904664 learned_addr learned my addr 172.21.15.12:0/1494904664 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.534+0000 7feb45400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20cbe0 0x561eba277350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.534+0000 7feb45e00700 1 --2- 172.21.15.12:0/1494904664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20ab60 0x561eba238a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.534+0000 7feb46800700 1 -- 172.21.15.12:0/1494904664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20cbe0 msgr2=0x561eba277350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.534+0000 7feb46800700 1 --2- 172.21.15.12:0/1494904664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20cbe0 0x561eba277350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.534+0000 7feb46800700 1 -- 172.21.15.12:0/1494904664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20ab60 msgr2=0x561eba238a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.534+0000 7feb46800700 1 --2- 172.21.15.12:0/1494904664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20ab60 0x561eba238a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.534+0000 7feb46800700 1 -- 172.21.15.12:0/1494904664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561eba125010 con 0x561eba239830 2024-03-29T11:32:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.534+0000 7feb45400700 1 --2- 172.21.15.12:0/1494904664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20cbe0 0x561eba277350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.534+0000 7feb46800700 1 --2- 172.21.15.12:0/1494904664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7feb480724f0 tx=0x7feb40011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e7e7bc7ea025d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb33000700 1 -- 172.21.15.12:0/1494904664 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4002a070 con 0x561eba239830 2024-03-29T11:32:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb33000700 1 -- 172.21.15.12:0/1494904664 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb40003a20 con 0x561eba239830 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb33000700 1 -- 172.21.15.12:0/1494904664 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40003dd0 con 0x561eba239830 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb604c4b80 1 -- 172.21.15.12:0/1494904664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 msgr2=0x561eba23cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb604c4b80 1 --2- 172.21.15.12:0/1494904664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7feb480724f0 tx=0x7feb40011000 comp rx=0 tx=0).stop 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb604c4b80 1 -- 172.21.15.12:0/1494904664 shutdown_connections 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb604c4b80 1 --2- 172.21.15.12:0/1494904664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb604c4b80 1 --2- 172.21.15.12:0/1494904664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20ab60 0x561eba238a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb604c4b80 1 --2- 172.21.15.12:0/1494904664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20cbe0 0x561eba277350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb604c4b80 1 -- 172.21.15.12:0/1494904664 >> 172.21.15.12:0/1494904664 conn(0x561eba16d6a0 msgr2=0x561eba236f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb604c4b80 1 -- 172.21.15.12:0/1494904664 shutdown_connections 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.535+0000 7feb604c4b80 1 -- 172.21.15.12:0/1494904664 wait complete. 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.536+0000 7feb604c4b80 1 -- start start 2024-03-29T11:32:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.536+0000 7feb604c4b80 1 Processor -- start 2024-03-29T11:32:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.537+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 0x561eba2bae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.537+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20cbe0 0x561eba24a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.537+0000 7feb604c4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.537+0000 7feb604c4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eba266480 con 0x561eba239830 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.537+0000 7feb604c4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eba20f1b0 con 0x561eba20cbe0 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.537+0000 7feb604c4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x561eba20f400 con 0x561eba20ab60 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.537+0000 7feb604c4b80 10 client.0 _ll_register_callbacks cb 0x7feb340059c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.537+0000 7feb46800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb46800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46428/0 (socket says 172.21.15.12:46428) 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb46800700 1 -- 172.21.15.12:0/2058358903 learned_addr learned my addr 172.21.15.12:0/2058358903 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb45400700 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20cbe0 0x561eba24a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb45e00700 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 0x561eba2bae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb46800700 1 -- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 msgr2=0x561eba2bae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb46800700 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 0x561eba2bae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb46800700 1 -- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20cbe0 msgr2=0x561eba24a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb46800700 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20cbe0 0x561eba24a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb46800700 1 -- 172.21.15.12:0/2058358903 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561eba03f0a0 con 0x561eba239830 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb45e00700 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 0x561eba2bae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb46800700 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7feb40014040 tx=0x7feb40004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f80c12efe39ff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.538+0000 7feb26a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb26a00700 1 -- 172.21.15.12:0/2058358903 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4002a040 con 0x561eba239830 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb26a00700 1 -- 172.21.15.12:0/2058358903 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7feb40020040 con 0x561eba239830 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb604c4b80 1 -- 172.21.15.12:0/2058358903 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb34007770 con 0x561eba239830 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb26a00700 1 -- 172.21.15.12:0/2058358903 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=128) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb4002f640 con 0x561eba239830 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb604c4b80 10 client.12147 fetch_fsmap learned FSMap version 128 2024-03-29T11:32:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb604c4b80 1 -- 172.21.15.12:0/2058358903 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=128,monmap=2+}) v3 -- 0x7feb480d0690 con 0x561eba239830 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb26a00700 1 -- 172.21.15.12:0/2058358903 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb34007770 con 0x561eba239830 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb26a00700 1 -- 172.21.15.12:0/2058358903 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 128) v1 ==== 3681+0+0 (secure 0 0 0) 0x7feb40024050 con 0x561eba239830 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb604c4b80 10 client.12147 fetch_fsmap finished waiting for FSMap version 128 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb604c4b80 10 client.12147 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb604c4b80 20 client.12147 populate_metadata read hostname 'smithi012' 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb604c4b80 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:6835/1262061639,v1:172.21.15.159:6837/1262061639] conn(0x561eba265270 0x561eba265650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb604c4b80 4 client.12147 mds_command: new command op to 11416 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.539+0000 7feb604c4b80 1 -- 172.21.15.12:0/2058358903 --> [v2:172.21.15.159:6835/1262061639,v1:172.21.15.159:6837/1262061639] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561eb9f98040 con 0x561eba265270 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.540+0000 7feb45e00700 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:6835/1262061639,v1:172.21.15.159:6837/1262061639] conn(0x561eba265270 0x561eba265650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.540+0000 7feb45e00700 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:6835/1262061639,v1:172.21.15.159:6837/1262061639] conn(0x561eba265270 0x561eba265650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4132c3c3d31396d server_cookie=d21418450240cec in_seq=0 out_seq=0 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.540+0000 7feb26a00700 10 client.12147 ms_handle_connect on v2:172.21.15.159:6835/1262061639 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb26a00700 1 -- 172.21.15.12:0/2058358903 <== mds.0 v2:172.21.15.159:6835/1262061639 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x561eb9f98040 con 0x561eba265270 2024-03-29T11:32:54.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb26a00700 10 client.12147 handle_command_reply: tid=0 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 client.12147 shutdown 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 -- 172.21.15.12:0/2058358903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 msgr2=0x561eba23cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7feb40014040 tx=0x7feb40004b40 comp rx=0 tx=0).stop 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 -- 172.21.15.12:0/2058358903 shutdown_connections 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:6835/1262061639,v1:172.21.15.159:6837/1262061639] conn(0x561eba265270 0x561eba265650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561eba239830 0x561eba23cf50 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561eba20ab60 0x561eba2bae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 --2- 172.21.15.12:0/2058358903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561eba20cbe0 0x561eba24a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb26a00700 0 client.12147 ms_handle_reset on v2:172.21.15.159:6835/1262061639 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 -- 172.21.15.12:0/2058358903 >> 172.21.15.12:0/2058358903 conn(0x561eba16d6a0 msgr2=0x561eba2772c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 -- 172.21.15.12:0/2058358903 shutdown_connections 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 1 -- 172.21.15.12:0/2058358903 wait complete. 2024-03-29T11:32:54.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.541+0000 7feb604c4b80 20 client.12147 trim_cache size 0 max 16384 2024-03-29T11:32:54.542 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:32:54.542 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T11:32:54.542 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T11:32:54.542 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 11416, 2024-03-29T11:32:54.542 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T11:32:54.542 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T11:32:54.542 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "backup_fs", 2024-03-29T11:32:54.542 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 6.6888303459999996, 2024-03-29T11:32:54.542 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 128, 2024-03-29T11:32:54.542 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 252, 2024-03-29T11:32:54.543 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 252, 2024-03-29T11:32:54.543 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 26.596462590000002 2024-03-29T11:32:54.543 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 -- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7feb2806aed0 msgr2=0x7feb2806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7feb2806aed0 0x7feb2806d390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feb480cb730 tx=0x7feb34009040 comp rx=0 tx=0).stop 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 -- 172.21.15.12:0/2544451931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb48099a60 msgr2=0x7feb48151110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb48099a60 0x7feb48151110 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7feb3800e870 tx=0x7feb3800ec30 comp rx=0 tx=0).stop 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 -- 172.21.15.12:0/2544451931 shutdown_connections 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7feb2806aed0 0x7feb2806d390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4815d990 0x7feb480b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb48099a60 0x7feb48151110 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 --2- 172.21.15.12:0/2544451931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feb4809a750 0x7feb48160050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 -- 172.21.15.12:0/2544451931 >> 172.21.15.12:0/2544451931 conn(0x7feb480b2e20 msgr2=0x7feb4815cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 -- 172.21.15.12:0/2544451931 shutdown_connections 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.542+0000 7feb4ca00700 1 -- 172.21.15.12:0/2544451931 wait complete. 2024-03-29T11:32:54.557 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T11:32:54.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.761+0000 7f4ed7800700 1 Processor -- start 2024-03-29T11:32:54.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.761+0000 7f4ed7800700 1 -- start start 2024-03-29T11:32:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.762+0000 7f4ed7800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ed0160e50 0x7f4ed00b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed7800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ed00d3f20 0x7f4ed00b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed7800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00cef00 con 0x7f4ed00b3f80 2024-03-29T11:32:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed7800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed0161230 con 0x7f4ed0160e50 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed7800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed00d2a80 con 0x7f4ed00d3f20 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed4e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ed0160e50 0x7f4ed00b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed4e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ed0160e50 0x7f4ed00b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46444/0 (socket says 172.21.15.12:46444) 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed4e00700 1 -- 172.21.15.12:0/392612831 learned_addr learned my addr 172.21.15.12:0/392612831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ecfe00700 1 --2- 172.21.15.12:0/392612831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ed00d3f20 0x7f4ed00b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed4e00700 1 -- 172.21.15.12:0/392612831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ed00d3f20 msgr2=0x7f4ed00b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed4e00700 1 --2- 172.21.15.12:0/392612831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ed00d3f20 0x7f4ed00b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed4e00700 1 -- 172.21.15.12:0/392612831 >> v1:172.21.15.159:6789/0 conn(0x7f4ed00b3f80 legacy=0x7f4ed00b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.763+0000 7f4ed4e00700 1 -- 172.21.15.12:0/392612831 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed00d43b0 con 0x7f4ed0160e50 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.764+0000 7f4ed4e00700 1 --2- 172.21.15.12:0/392612831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 0x7f4ed00b4e90 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40098f0 tx=0x7f4ec4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8ea16153d0dec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.764+0000 7f4ecf400700 1 -- 172.21.15.12:0/392612831 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec400b8a0 con 0x7f4ed0160e50 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.764+0000 7f4ecf400700 1 -- 172.21.15.12:0/392612831 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec400ba00 con 0x7f4ed0160e50 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.764+0000 7f4ecf400700 1 -- 172.21.15.12:0/392612831 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec400bd90 con 0x7f4ed0160e50 2024-03-29T11:32:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.764+0000 7f4ed7800700 1 -- 172.21.15.12:0/392612831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 msgr2=0x7f4ed00b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.764+0000 7f4ed7800700 1 --2- 172.21.15.12:0/392612831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 0x7f4ed00b4e90 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40098f0 tx=0x7f4ec4009c00 comp rx=0 tx=0).stop 2024-03-29T11:32:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.764+0000 7f4ed7800700 1 -- 172.21.15.12:0/392612831 shutdown_connections 2024-03-29T11:32:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.764+0000 7f4ed7800700 1 --2- 172.21.15.12:0/392612831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 0x7f4ed00b4e90 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.764+0000 7f4ed7800700 1 --2- 172.21.15.12:0/392612831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ed00d3f20 0x7f4ed00b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.764+0000 7f4ed7800700 1 -- 172.21.15.12:0/392612831 >> 172.21.15.12:0/392612831 conn(0x7f4ed008aff0 msgr2=0x7f4ed00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.765+0000 7f4ed7800700 1 -- 172.21.15.12:0/392612831 shutdown_connections 2024-03-29T11:32:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.765+0000 7f4ed7800700 1 -- 172.21.15.12:0/392612831 wait complete. 2024-03-29T11:32:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.766+0000 7f4ed7800700 1 Processor -- start 2024-03-29T11:32:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.766+0000 7f4ed7800700 1 -- start start 2024-03-29T11:32:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.767+0000 7f4ed7800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ed00b3f80 0x7f4ed01f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.767+0000 7f4ed7800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 0x7f4ed00b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.767+0000 7f4ed7800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ed00b6150 0x7f4ed01f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.767+0000 7f4ed7800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed00b8310 con 0x7f4ed0160e50 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.767+0000 7f4ed7800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed00b92b0 con 0x7f4ed00b3f80 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.767+0000 7f4ed7800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed00b7700 con 0x7f4ed00b6150 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ecfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 0x7f4ed00b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ecfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 0x7f4ed00b7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46448/0 (socket says 172.21.15.12:46448) 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ecfe00700 1 -- 172.21.15.12:0/1954325679 learned_addr learned my addr 172.21.15.12:0/1954325679 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ecfe00700 1 -- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ed00b6150 msgr2=0x7f4ed01f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ecfe00700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ed00b6150 0x7f4ed01f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ed4e00700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ed00b3f80 0x7f4ed01f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ecfe00700 1 -- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ed00b3f80 msgr2=0x7f4ed01f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ecfe00700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ed00b3f80 0x7f4ed01f5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ecfe00700 1 -- 172.21.15.12:0/1954325679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec4009550 con 0x7f4ed0160e50 2024-03-29T11:32:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ed4e00700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ed00b3f80 0x7f4ed01f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:32:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.768+0000 7f4ecfe00700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 0x7f4ed00b7040 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00e810 tx=0x7f4ebc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.769+0000 7f4eccc00700 1 -- 172.21.15.12:0/1954325679 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc013610 con 0x7f4ed0160e50 2024-03-29T11:32:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.769+0000 7f4eccc00700 1 -- 172.21.15.12:0/1954325679 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc013770 con 0x7f4ed0160e50 2024-03-29T11:32:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.769+0000 7f4eccc00700 1 -- 172.21.15.12:0/1954325679 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc013b20 con 0x7f4ed0160e50 2024-03-29T11:32:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.769+0000 7f4ed7800700 1 -- 172.21.15.12:0/1954325679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed00b79e0 con 0x7f4ed0160e50 2024-03-29T11:32:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.769+0000 7f4ed7800700 1 -- 172.21.15.12:0/1954325679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed00b7fa0 con 0x7f4ed0160e50 2024-03-29T11:32:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.770+0000 7f4ed7800700 1 -- 172.21.15.12:0/1954325679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed00b8160 con 0x7f4ed0160e50 2024-03-29T11:32:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.771+0000 7f4eccc00700 1 -- 172.21.15.12:0/1954325679 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4ebc013c80 con 0x7f4ed0160e50 2024-03-29T11:32:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.772+0000 7f4eccc00700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4eb006aed0 0x7f4eb006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.772+0000 7f4eccc00700 1 -- 172.21.15.12:0/1954325679 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f4ebc0898d0 con 0x7f4ed0160e50 2024-03-29T11:32:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.772+0000 7f4ed4e00700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4eb006aed0 0x7f4eb006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.773+0000 7f4ed4e00700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4eb006aed0 0x7f4eb006d390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00d3580 tx=0x7f4ec4008030 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:54.777+0000 7f4eccc00700 1 -- 172.21.15.12:0/1954325679 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ebc054320 con 0x7f4ed0160e50 2024-03-29T11:32:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.023+0000 7f4ed7800700 1 -- 172.21.15.12:0/1954325679 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ed01f2e60 con 0x7f4eb006aed0 2024-03-29T11:32:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.025+0000 7f4eccc00700 1 -- 172.21.15.12:0/1954325679 <== mgr.11050 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f4ed01f2e60 con 0x7f4eb006aed0 2024-03-29T11:32:55.020 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:32:55.020 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:55.021 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T11:32:55.021 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:32:55.021 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:32:55.021 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "backup_fs" 2024-03-29T11:32:55.021 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:32:55.021 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:32:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 -- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4eb006aed0 msgr2=0x7f4eb006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4eb006aed0 0x7f4eb006d390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00d3580 tx=0x7f4ec4008030 comp rx=0 tx=0).stop 2024-03-29T11:32:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 -- 172.21.15.12:0/1954325679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 msgr2=0x7f4ed00b7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 0x7f4ed00b7040 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00e810 tx=0x7f4ebc00ebd0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 -- 172.21.15.12:0/1954325679 shutdown_connections 2024-03-29T11:32:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4eb006aed0 0x7f4eb006d390 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ed00b6150 0x7f4ed01f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ed00b3f80 0x7f4ed01f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 --2- 172.21.15.12:0/1954325679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed0160e50 0x7f4ed00b7040 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 -- 172.21.15.12:0/1954325679 >> 172.21.15.12:0/1954325679 conn(0x7f4ed008aff0 msgr2=0x7f4ed006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 -- 172.21.15.12:0/1954325679 shutdown_connections 2024-03-29T11:32:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.027+0000 7f4ed7800700 1 -- 172.21.15.12:0/1954325679 wait complete. 2024-03-29T11:32:55.042 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T11:32:55.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.246+0000 7ff272800700 1 Processor -- start 2024-03-29T11:32:55.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.246+0000 7ff272800700 1 -- start start 2024-03-29T11:32:55.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.247+0000 7ff272800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff26c15d9d0 0x7ff26c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:55.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.247+0000 7ff272800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff26c0d16a0 0x7ff26c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:55.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.247+0000 7ff272800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c161420 con 0x7ff26c0d2370 2024-03-29T11:32:55.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.247+0000 7ff272800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26c0d2820 con 0x7ff26c0d16a0 2024-03-29T11:32:55.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.247+0000 7ff272800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c161700 con 0x7ff26c15d9d0 2024-03-29T11:32:55.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.248+0000 7ff26be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff26c0d2370 0x7ff26c15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59610/0 (socket says 172.21.15.12:59610) 2024-03-29T11:32:55.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.248+0000 7ff26be00700 1 -- 172.21.15.12:0/3748285658 learned_addr learned my addr 172.21.15.12:0/3748285658 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:55.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.248+0000 7ff26b400700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff26c15d9d0 0x7ff26c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.248+0000 7ff26aa00700 1 --2- 172.21.15.12:0/3748285658 >> v2:172.21.15.159:3300/0 conn(0x7ff26c0d16a0 0x7ff26c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.248+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498830090 0 0) 0x7ff26c161420 con 0x7ff26c0d2370 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.248+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2500035b0 con 0x7ff26c0d2370 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2985255266 0 0) 0x7ff2500035b0 con 0x7ff26c0d2370 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff26c15d9d0 msgr2=0x7ff26c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff26c15d9d0 0x7ff26c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 >> v2:172.21.15.159:3300/0 conn(0x7ff26c0d16a0 msgr2=0x7ff26c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 --2- 172.21.15.12:0/3748285658 >> v2:172.21.15.159:3300/0 conn(0x7ff26c0d16a0 0x7ff26c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c1619e0 con 0x7ff26c0d2370 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7ff2640019f0 con 0x7ff26c0d2370 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26b400700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff26c15d9d0 0x7ff26c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:55.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26aa00700 1 --2- 172.21.15.12:0/3748285658 >> v2:172.21.15.159:3300/0 conn(0x7ff26c0d16a0 0x7ff26c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 >> v1:172.21.15.12:6789/0 conn(0x7ff26c0d2370 legacy=0x7ff26c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff250003a00 0x7ff250003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff26c15d9d0 0x7ff250004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff250004d90 0x7ff250005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c0d2820 con 0x7ff250003a00 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c161700 con 0x7ff250004d90 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25000ba10 con 0x7ff26c15d9d0 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26b400700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff250003a00 0x7ff250003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26aa00700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff26c15d9d0 0x7ff250004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26be00700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff250004d90 0x7ff250005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26b400700 1 -- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff26c15d9d0 msgr2=0x7ff250004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26b400700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff26c15d9d0 0x7ff250004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26b400700 1 -- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff250004d90 msgr2=0x7ff250005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:55.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26b400700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff250004d90 0x7ff250005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26b400700 1 -- 172.21.15.12:0/3748285658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0094a0 con 0x7ff250003a00 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.249+0000 7ff26aa00700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff26c15d9d0 0x7ff250004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.250+0000 7ff26b400700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff250003a00 0x7ff250003e20 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009d30 tx=0x7ff250005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c186ed68fc80b25a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.250+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c009e80 con 0x7ff250003a00 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.250+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c00e9a0 con 0x7ff250003a00 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.250+0000 7ff26a000700 1 -- 172.21.15.12:0/3748285658 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c00ed10 con 0x7ff250003a00 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.250+0000 7ff272800700 1 -- 172.21.15.12:0/3748285658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff250003a00 msgr2=0x7ff250003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.250+0000 7ff272800700 1 --2- 172.21.15.12:0/3748285658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff250003a00 0x7ff250003e20 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009d30 tx=0x7ff250005880 comp rx=0 tx=0).stop 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.250+0000 7ff26be00700 1 -- 172.21.15.12:0/3748285658 reap_dead start 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.251+0000 7ff272800700 1 -- 172.21.15.12:0/3748285658 shutdown_connections 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.251+0000 7ff272800700 1 -- 172.21.15.12:0/3748285658 >> 172.21.15.12:0/3748285658 conn(0x7ff26c08aff0 msgr2=0x7ff26c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.251+0000 7ff272800700 1 -- 172.21.15.12:0/3748285658 shutdown_connections 2024-03-29T11:32:55.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.251+0000 7ff272800700 1 -- 172.21.15.12:0/3748285658 wait complete. 2024-03-29T11:32:55.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.252+0000 7ff272800700 1 Processor -- start 2024-03-29T11:32:55.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.253+0000 7ff272800700 1 -- start start 2024-03-29T11:32:55.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.253+0000 7ff272800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff250003a00 0x7ff26c155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:55.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.253+0000 7ff272800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff26c1529f0 0x7ff26c0b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:55.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff272800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff26c15aaa0 0x7ff26c15af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff272800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c15b490 con 0x7ff26c1529f0 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff272800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c152eb0 con 0x7ff26c15aaa0 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff272800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c153190 con 0x7ff250003a00 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff26aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff26c1529f0 0x7ff26c0b7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff26aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff26c1529f0 0x7ff26c0b7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46472/0 (socket says 172.21.15.12:46472) 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff26aa00700 1 -- 172.21.15.12:0/2678490967 learned_addr learned my addr 172.21.15.12:0/2678490967 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff26be00700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff26c15aaa0 0x7ff26c15af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff26b400700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff250003a00 0x7ff26c155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff26aa00700 1 -- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff250003a00 msgr2=0x7ff26c155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff26aa00700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff250003a00 0x7ff26c155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff26aa00700 1 -- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff26c15aaa0 msgr2=0x7ff26c15af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff26aa00700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff26c15aaa0 0x7ff26c15af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:55.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.254+0000 7ff26aa00700 1 -- 172.21.15.12:0/2678490967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c153470 con 0x7ff26c1529f0 2024-03-29T11:32:55.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.255+0000 7ff26aa00700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff26c1529f0 0x7ff26c0b7ef0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7ff2580118d0 tx=0x7ff258011be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:55.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.255+0000 7ff263400700 1 -- 172.21.15.12:0/2678490967 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258015870 con 0x7ff26c1529f0 2024-03-29T11:32:55.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.255+0000 7ff263400700 1 -- 172.21.15.12:0/2678490967 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2580159d0 con 0x7ff26c1529f0 2024-03-29T11:32:55.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.255+0000 7ff272800700 1 -- 172.21.15.12:0/2678490967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26c153750 con 0x7ff26c1529f0 2024-03-29T11:32:55.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.255+0000 7ff263400700 1 -- 172.21.15.12:0/2678490967 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258015d40 con 0x7ff26c1529f0 2024-03-29T11:32:55.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.255+0000 7ff272800700 1 -- 172.21.15.12:0/2678490967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26c154c10 con 0x7ff26c1529f0 2024-03-29T11:32:55.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.256+0000 7ff272800700 1 -- 172.21.15.12:0/2678490967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26c0d1ca0 con 0x7ff26c1529f0 2024-03-29T11:32:55.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.257+0000 7ff263400700 1 -- 172.21.15.12:0/2678490967 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 139) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff2580143e0 con 0x7ff26c1529f0 2024-03-29T11:32:55.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.258+0000 7ff263400700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff24c06aed0 0x7ff24c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:55.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.258+0000 7ff263400700 1 -- 172.21.15.12:0/2678490967 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7ff25808a240 con 0x7ff26c1529f0 2024-03-29T11:32:55.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.258+0000 7ff26b400700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff24c06aed0 0x7ff24c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:55.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.259+0000 7ff26b400700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff24c06aed0 0x7ff24c06d390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000c00 tx=0x7ff25c008030 comp rx=0 tx=0).ready entity=mgr.11050 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:55.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.262+0000 7ff263400700 1 -- 172.21.15.12:0/2678490967 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff258054d00 con 0x7ff26c1529f0 2024-03-29T11:32:55.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:55.538+0000 7ff272800700 1 -- 172.21.15.12:0/2678490967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ff26c067720 con 0x7ff26c1529f0 2024-03-29T11:32:56.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.378+0000 7ff263400700 1 -- 172.21.15.12:0/2678490967 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 140) v1 ==== 89358+0+0 (secure 0 0 0) 0x7ff2580550d0 con 0x7ff26c1529f0 2024-03-29T11:32:56.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.380+0000 7ff263400700 1 -- 172.21.15.12:0/2678490967 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v140) v1 ==== 92+0+0 (secure 0 0 0) 0x7ff258085420 con 0x7ff26c1529f0 2024-03-29T11:32:56.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.383+0000 7ff272800700 1 -- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff24c06aed0 msgr2=0x7ff24c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:56.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.383+0000 7ff272800700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff24c06aed0 0x7ff24c06d390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000c00 tx=0x7ff25c008030 comp rx=0 tx=0).stop 2024-03-29T11:32:56.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.384+0000 7ff272800700 1 -- 172.21.15.12:0/2678490967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff26c1529f0 msgr2=0x7ff26c0b7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:56.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.384+0000 7ff272800700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff26c1529f0 0x7ff26c0b7ef0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7ff2580118d0 tx=0x7ff258011be0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.384+0000 7ff272800700 1 -- 172.21.15.12:0/2678490967 shutdown_connections 2024-03-29T11:32:56.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.384+0000 7ff272800700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff24c06aed0 0x7ff24c06d390 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.384+0000 7ff272800700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff26c15aaa0 0x7ff26c15af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.384+0000 7ff272800700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff250003a00 0x7ff26c155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.384+0000 7ff272800700 1 --2- 172.21.15.12:0/2678490967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff26c1529f0 0x7ff26c0b7ef0 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.384+0000 7ff272800700 1 -- 172.21.15.12:0/2678490967 >> 172.21.15.12:0/2678490967 conn(0x7ff26c08aff0 msgr2=0x7ff26c15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:56.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.384+0000 7ff272800700 1 -- 172.21.15.12:0/2678490967 shutdown_connections 2024-03-29T11:32:56.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.384+0000 7ff272800700 1 -- 172.21.15.12:0/2678490967 wait complete. 2024-03-29T11:32:56.403 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T11:32:56.603 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:56.608+0000 7f9cb0274240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:32:56.609 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:56.613+0000 7eff153b1240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:32:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.616+0000 7fce1d000700 1 Processor -- start 2024-03-29T11:32:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.616+0000 7fce1d000700 1 -- start start 2024-03-29T11:32:56.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.618+0000 7fce1d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 0x7fce180d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:56.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.618+0000 7fce1d000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce18161420 con 0x7fce1815d9d0 2024-03-29T11:32:56.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.618+0000 7fce1d000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce18161700 con 0x7fce180d2370 2024-03-29T11:32:56.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.618+0000 7fce1d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce180d2820 con 0x7fce180d16a0 2024-03-29T11:32:56.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.619+0000 7fce15e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fce1815d9d0 0x7fce180d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59628/0 (socket says 172.21.15.12:59628) 2024-03-29T11:32:56.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.619+0000 7fce15e00700 1 -- 172.21.15.12:0/3717934348 learned_addr learned my addr 172.21.15.12:0/3717934348 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:56.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.619+0000 7fce15400700 1 --2- 172.21.15.12:0/3717934348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 0x7fce180d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:56.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.619+0000 7fce14a00700 1 -- 172.21.15.12:0/3717934348 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177224955 0 0) 0x7fce18161420 con 0x7fce1815d9d0 2024-03-29T11:32:56.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.619+0000 7fce14a00700 1 -- 172.21.15.12:0/3717934348 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdfc0035b0 con 0x7fce1815d9d0 2024-03-29T11:32:56.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.619+0000 7fce15400700 1 -- 172.21.15.12:0/3717934348 >> v1:172.21.15.159:6789/0 conn(0x7fce180d2370 legacy=0x7fce180ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:56.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.619+0000 7fce15400700 1 -- 172.21.15.12:0/3717934348 >> v1:172.21.15.12:6789/0 conn(0x7fce1815d9d0 legacy=0x7fce180d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:56.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.619+0000 7fce15400700 1 -- 172.21.15.12:0/3717934348 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce181619e0 con 0x7fce180d16a0 2024-03-29T11:32:56.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.620+0000 7fce15400700 1 --2- 172.21.15.12:0/3717934348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 0x7fce180d1e30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fce04009850 tx=0x7fce04009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=273b948a543c396e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:56.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.620+0000 7fce14a00700 1 -- 172.21.15.12:0/3717934348 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce04005e30 con 0x7fce180d16a0 2024-03-29T11:32:56.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.620+0000 7fce14a00700 1 -- 172.21.15.12:0/3717934348 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce04006030 con 0x7fce180d16a0 2024-03-29T11:32:56.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.620+0000 7fce14a00700 1 -- 172.21.15.12:0/3717934348 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce040063e0 con 0x7fce180d16a0 2024-03-29T11:32:56.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.621+0000 7fce1d000700 1 -- 172.21.15.12:0/3717934348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 msgr2=0x7fce180d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:56.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.621+0000 7fce1d000700 1 --2- 172.21.15.12:0/3717934348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 0x7fce180d1e30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fce04009850 tx=0x7fce04009b60 comp rx=0 tx=0).stop 2024-03-29T11:32:56.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.621+0000 7fce1d000700 1 -- 172.21.15.12:0/3717934348 shutdown_connections 2024-03-29T11:32:56.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.621+0000 7fce1d000700 1 --2- 172.21.15.12:0/3717934348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 0x7fce180d1e30 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.621+0000 7fce1d000700 1 -- 172.21.15.12:0/3717934348 >> 172.21.15.12:0/3717934348 conn(0x7fce1808aff0 msgr2=0x7fce180d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:56.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.621+0000 7fce1d000700 1 -- 172.21.15.12:0/3717934348 shutdown_connections 2024-03-29T11:32:56.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.621+0000 7fce1d000700 1 -- 172.21.15.12:0/3717934348 wait complete. 2024-03-29T11:32:56.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.624+0000 7fce1d000700 1 Processor -- start 2024-03-29T11:32:56.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.624+0000 7fce1d000700 1 -- start start 2024-03-29T11:32:56.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.625+0000 7fce1d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 0x7fce18155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:56.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce1d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce180d2370 0x7fce180b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:56.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce1d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce1815d9d0 0x7fce180b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:56.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce1d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce181e7320 con 0x7fce180d2370 2024-03-29T11:32:56.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce1d000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce181e75c0 con 0x7fce1815d9d0 2024-03-29T11:32:56.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce1d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce181e78a0 con 0x7fce180d16a0 2024-03-29T11:32:56.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce15400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce180d2370 0x7fce180b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:56.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce15400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce180d2370 0x7fce180b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46492/0 (socket says 172.21.15.12:46492) 2024-03-29T11:32:56.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce15400700 1 -- 172.21.15.12:0/3026470503 learned_addr learned my addr 172.21.15.12:0/3026470503 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce15e00700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 0x7fce18155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce16800700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce1815d9d0 0x7fce180b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce15400700 1 -- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 msgr2=0x7fce18155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce15400700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 0x7fce18155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce15400700 1 -- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce1815d9d0 msgr2=0x7fce180b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce15400700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce1815d9d0 0x7fce180b5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.626+0000 7fce15400700 1 -- 172.21.15.12:0/3026470503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce04009430 con 0x7fce180d2370 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.627+0000 7fce15400700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce180d2370 0x7fce180b4c20 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fce04009ba0 tx=0x7fce04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.627+0000 7fce0e000700 1 -- 172.21.15.12:0/3026470503 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce040068d0 con 0x7fce180d2370 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.627+0000 7fce0e000700 1 -- 172.21.15.12:0/3026470503 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce04020070 con 0x7fce180d2370 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.627+0000 7fce0e000700 1 -- 172.21.15.12:0/3026470503 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce04025650 con 0x7fce180d2370 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.627+0000 7fce1d000700 1 -- 172.21.15.12:0/3026470503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce181e7b80 con 0x7fce180d2370 2024-03-29T11:32:56.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.627+0000 7fce1d000700 1 -- 172.21.15.12:0/3026470503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce18155e90 con 0x7fce180d2370 2024-03-29T11:32:56.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.628+0000 7fce1d000700 1 -- 172.21.15.12:0/3026470503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce180d1ca0 con 0x7fce180d2370 2024-03-29T11:32:56.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.629+0000 7fce0e000700 1 -- 172.21.15.12:0/3026470503 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 140) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fce04022030 con 0x7fce180d2370 2024-03-29T11:32:56.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.630+0000 7fce0e000700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcdf806af50 0x7fcdf806d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:56.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.630+0000 7fce0e000700 1 -- 172.21.15.12:0/3026470503 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7fce0408afd0 con 0x7fce180d2370 2024-03-29T11:32:56.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.630+0000 7fce15e00700 1 -- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcdf806af50 msgr2=0x7fcdf806d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:56.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.631+0000 7fce15e00700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcdf806af50 0x7fcdf806d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:56.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.635+0000 7fce0e000700 1 -- 172.21.15.12:0/3026470503 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce04055b30 con 0x7fce180d2370 2024-03-29T11:32:56.790 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:56.795+0000 7f9cb0274240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:32:56.821 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:56.823+0000 7eff153b1240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:32:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.832+0000 7fce15e00700 1 -- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcdf806af50 msgr2=0x7fcdf806d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.832+0000 7fce15e00700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcdf806af50 0x7fcdf806d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.888+0000 7fce1d000700 1 -- 172.21.15.12:0/3026470503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fce181e80a0 con 0x7fce180d2370 2024-03-29T11:32:56.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.892+0000 7fce0e000700 1 -- 172.21.15.12:0/3026470503 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7fce181e80a0 con 0x7fce180d2370 2024-03-29T11:32:56.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.894+0000 7fce1d000700 1 -- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcdf806af50 msgr2=0x7fcdf806d410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:56.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.894+0000 7fce1d000700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcdf806af50 0x7fcdf806d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.894+0000 7fce1d000700 1 -- 172.21.15.12:0/3026470503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce180d2370 msgr2=0x7fce180b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.894+0000 7fce1d000700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce180d2370 0x7fce180b4c20 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fce04009ba0 tx=0x7fce04004750 comp rx=0 tx=0).stop 2024-03-29T11:32:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.894+0000 7fce1d000700 1 -- 172.21.15.12:0/3026470503 shutdown_connections 2024-03-29T11:32:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.894+0000 7fce1d000700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcdf806af50 0x7fcdf806d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.895+0000 7fce1d000700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce1815d9d0 0x7fce180b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.895+0000 7fce1d000700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce180d16a0 0x7fce18155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.895+0000 7fce1d000700 1 --2- 172.21.15.12:0/3026470503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce180d2370 0x7fce180b4c20 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.895+0000 7fce1d000700 1 -- 172.21.15.12:0/3026470503 >> 172.21.15.12:0/3026470503 conn(0x7fce1808aff0 msgr2=0x7fce180b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.895+0000 7fce1d000700 1 -- 172.21.15.12:0/3026470503 shutdown_connections 2024-03-29T11:32:56.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:56.895+0000 7fce1d000700 1 -- 172.21.15.12:0/3026470503 wait complete. 2024-03-29T11:32:56.909 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T11:32:57.112 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:57.117+0000 7f9cb0274240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:32:57.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.120+0000 7f3385000700 1 Processor -- start 2024-03-29T11:32:57.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.121+0000 7f3385000700 1 -- start start 2024-03-29T11:32:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f3385000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3380099a60 0x7f33800d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f3385000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f338015d990 0x7f3380099e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f3385000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33800d0970 con 0x7f338009a750 2024-03-29T11:32:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f3385000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338006c020 con 0x7f338015d990 2024-03-29T11:32:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f3385000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338009a380 con 0x7f3380099a60 2024-03-29T11:32:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f337de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3380099a60 0x7f33800d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f337d400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f338015d990 0x7f3380099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f337e800700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f338009a750 0x7f338006c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:38536/0 (socket says 172.21.15.12:38536) 2024-03-29T11:32:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f337de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3380099a60 0x7f33800d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:37044/0 (socket says 172.21.15.12:37044) 2024-03-29T11:32:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f337e800700 1 -- 172.21.15.12:0/809596989 learned_addr learned my addr 172.21.15.12:0/809596989 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.122+0000 7f337d400700 1 --2- 172.21.15.12:0/809596989 >> v2:172.21.15.159:3300/0 conn(0x7f338015d990 0x7f3380099e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:55442/0 (socket says 172.21.15.12:55442) 2024-03-29T11:32:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.123+0000 7f337ca00700 1 -- 172.21.15.12:0/809596989 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843156218 0 0) 0x7f33800d0970 con 0x7f338009a750 2024-03-29T11:32:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.123+0000 7f337de00700 1 -- 172.21.15.12:0/809596989 >> v2:172.21.15.159:3300/0 conn(0x7f338015d990 msgr2=0x7f3380099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.123+0000 7f337de00700 1 --2- 172.21.15.12:0/809596989 >> v2:172.21.15.159:3300/0 conn(0x7f338015d990 0x7f3380099e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.123+0000 7f337de00700 1 -- 172.21.15.12:0/809596989 >> v1:172.21.15.159:6789/0 conn(0x7f338009a750 legacy=0x7f338006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.123+0000 7f337de00700 1 -- 172.21.15.12:0/809596989 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33800c81a0 con 0x7f3380099a60 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.123+0000 7f337de00700 1 --2- 172.21.15.12:0/809596989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3380099a60 0x7f33800d0060 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3370009840 tx=0x7f3370009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d12a6749293734d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.123+0000 7f337ca00700 1 -- 172.21.15.12:0/809596989 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370011830 con 0x7f3380099a60 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.123+0000 7f337ca00700 1 -- 172.21.15.12:0/809596989 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3370011990 con 0x7f3380099a60 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.123+0000 7f337ca00700 1 -- 172.21.15.12:0/809596989 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370011d20 con 0x7f3380099a60 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.124+0000 7f3385000700 1 -- 172.21.15.12:0/809596989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3380099a60 msgr2=0x7f33800d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.124+0000 7f3385000700 1 --2- 172.21.15.12:0/809596989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3380099a60 0x7f33800d0060 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3370009840 tx=0x7f3370009b50 comp rx=0 tx=0).stop 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.124+0000 7f3385000700 1 -- 172.21.15.12:0/809596989 shutdown_connections 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.124+0000 7f3385000700 1 --2- 172.21.15.12:0/809596989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3380099a60 0x7f33800d0060 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.124+0000 7f3385000700 1 --2- 172.21.15.12:0/809596989 >> v2:172.21.15.159:3300/0 conn(0x7f338015d990 0x7f3380099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.124+0000 7f3385000700 1 -- 172.21.15.12:0/809596989 >> 172.21.15.12:0/809596989 conn(0x7f33800b2e20 msgr2=0x7f33800b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.124+0000 7f3385000700 1 -- 172.21.15.12:0/809596989 shutdown_connections 2024-03-29T11:32:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.124+0000 7f3385000700 1 -- 172.21.15.12:0/809596989 wait complete. 2024-03-29T11:32:57.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.126+0000 7f3385000700 1 Processor -- start 2024-03-29T11:32:57.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.126+0000 7f3385000700 1 -- start start 2024-03-29T11:32:57.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f3385000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3380099a60 0x7f338015dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f3385000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f338009a750 0x7f338015f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f3385000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f338015d990 0x7f33800b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f3385000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338015e380 con 0x7f338009a750 2024-03-29T11:32:57.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f3385000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33800c4e30 con 0x7f3380099a60 2024-03-29T11:32:57.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f3385000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33800c5110 con 0x7f338015d990 2024-03-29T11:32:57.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f337de00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3380099a60 0x7f338015dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f337d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f338009a750 0x7f338015f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f337de00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3380099a60 0x7f338015dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:55456/0 (socket says 172.21.15.12:55456) 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f337e800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f338015d990 0x7f33800b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.127+0000 7f337de00700 1 -- 172.21.15.12:0/307088576 learned_addr learned my addr 172.21.15.12:0/307088576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f337d400700 1 -- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f338015d990 msgr2=0x7f33800b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f337d400700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f338015d990 0x7f33800b6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f337d400700 1 -- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3380099a60 msgr2=0x7f338015dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f337d400700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3380099a60 0x7f338015dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f337d400700 1 -- 172.21.15.12:0/307088576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33700094a0 con 0x7f338009a750 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f337de00700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3380099a60 0x7f338015dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f337d400700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f338009a750 0x7f338015f4e0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f336c00e840 tx=0x7f336c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f3376000700 1 -- 172.21.15.12:0/307088576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c013680 con 0x7f338009a750 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f3376000700 1 -- 172.21.15.12:0/307088576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c0137e0 con 0x7f338009a750 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f3385000700 1 -- 172.21.15.12:0/307088576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33800c53f0 con 0x7f338009a750 2024-03-29T11:32:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f3376000700 1 -- 172.21.15.12:0/307088576 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c013b90 con 0x7f338009a750 2024-03-29T11:32:57.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.128+0000 7f3385000700 1 -- 172.21.15.12:0/307088576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338014fa90 con 0x7f338009a750 2024-03-29T11:32:57.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.130+0000 7f3385000700 1 -- 172.21.15.12:0/307088576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3380067720 con 0x7f338009a750 2024-03-29T11:32:57.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.131+0000 7f3376000700 1 -- 172.21.15.12:0/307088576 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 140) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f336c013cf0 con 0x7f338009a750 2024-03-29T11:32:57.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.132+0000 7f3376000700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f336006af20 0x7f336006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.132+0000 7f3376000700 1 -- 172.21.15.12:0/307088576 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7f336c089800 con 0x7f338009a750 2024-03-29T11:32:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.132+0000 7f337de00700 1 -- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f336006af20 msgr2=0x7f336006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.132+0000 7f337de00700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f336006af20 0x7f336006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:57.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.140+0000 7f3376000700 1 -- 172.21.15.12:0/307088576 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f336c054360 con 0x7f338009a750 2024-03-29T11:32:57.172 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:57.176+0000 7eff153b1240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:32:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.333+0000 7f337de00700 1 -- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f336006af20 msgr2=0x7f336006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.333+0000 7f337de00700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f336006af20 0x7f336006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:57.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.355+0000 7f3385000700 1 -- 172.21.15.12:0/307088576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f33800d0b70 con 0x7f338009a750 2024-03-29T11:32:57.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.361+0000 7f3376000700 1 -- 172.21.15.12:0/307088576 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7f336c057980 con 0x7f338009a750 2024-03-29T11:32:57.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 -- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f336006af20 msgr2=0x7f336006d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:57.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f336006af20 0x7f336006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 -- 172.21.15.12:0/307088576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f338009a750 msgr2=0x7f338015f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:57.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f338009a750 0x7f338015f4e0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f336c00e840 tx=0x7f336c00eb50 comp rx=0 tx=0).stop 2024-03-29T11:32:57.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 -- 172.21.15.12:0/307088576 shutdown_connections 2024-03-29T11:32:57.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f336006af20 0x7f336006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f338015d990 0x7f33800b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3380099a60 0x7f338015dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 --2- 172.21.15.12:0/307088576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f338009a750 0x7f338015f4e0 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 -- 172.21.15.12:0/307088576 >> 172.21.15.12:0/307088576 conn(0x7f33800b2e20 msgr2=0x7f338006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:57.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 -- 172.21.15.12:0/307088576 shutdown_connections 2024-03-29T11:32:57.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.364+0000 7f3385000700 1 -- 172.21.15.12:0/307088576 wait complete. 2024-03-29T11:32:57.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:32:57.379 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-03-29T11:32:57.428 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2024-03-29T11:32:57.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.637+0000 7fb36fc00700 1 Processor -- start 2024-03-29T11:32:57.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.637+0000 7fb36fc00700 1 -- start start 2024-03-29T11:32:57.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb36fc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb368160cc0 0x7fb368168730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb36fc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36815f820 con 0x7fb3680ced80 2024-03-29T11:32:57.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb36fc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680d2450 con 0x7fb36815dad0 2024-03-29T11:32:57.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb36fc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3680cfcc0 con 0x7fb368160cc0 2024-03-29T11:32:57.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb367e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb3680ced80 0x7fb3680d1d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59644/0 (socket says 172.21.15.12:59644) 2024-03-29T11:32:57.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb367e00700 1 -- 172.21.15.12:0/1517149238 learned_addr learned my addr 172.21.15.12:0/1517149238 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:32:57.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb36dc00700 1 --2- 172.21.15.12:0/1517149238 >> v2:172.21.15.12:3300/0 conn(0x7fb368160cc0 0x7fb368168730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb367400700 1 -- 172.21.15.12:0/1517149238 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444996951 0 0) 0x7fb36815f820 con 0x7fb3680ced80 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb367400700 1 -- 172.21.15.12:0/1517149238 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3500035b0 con 0x7fb3680ced80 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb367400700 1 -- 172.21.15.12:0/1517149238 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903669862 0 0) 0x7fb3680d2450 con 0x7fb36815dad0 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.639+0000 7fb367400700 1 -- 172.21.15.12:0/1517149238 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb36815f820 con 0x7fb36815dad0 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.640+0000 7fb36dc00700 1 -- 172.21.15.12:0/1517149238 >> v1:172.21.15.159:6789/0 conn(0x7fb36815dad0 legacy=0x7fb3680d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.640+0000 7fb36dc00700 1 -- 172.21.15.12:0/1517149238 >> v1:172.21.15.12:6789/0 conn(0x7fb3680ced80 legacy=0x7fb3680d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.640+0000 7fb36dc00700 1 -- 172.21.15.12:0/1517149238 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3680cf420 con 0x7fb368160cc0 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.640+0000 7fb36dc00700 1 --2- 172.21.15.12:0/1517149238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb368160cc0 0x7fb368168730 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fb360009d90 tx=0x7fb36000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac6c5f594c59c6c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.640+0000 7fb367400700 1 -- 172.21.15.12:0/1517149238 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36000e070 con 0x7fb368160cc0 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.640+0000 7fb367400700 1 -- 172.21.15.12:0/1517149238 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36000bb20 con 0x7fb368160cc0 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.640+0000 7fb367400700 1 -- 172.21.15.12:0/1517149238 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb360012430 con 0x7fb368160cc0 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.641+0000 7fb36fc00700 1 -- 172.21.15.12:0/1517149238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb368160cc0 msgr2=0x7fb368168730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:57.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.641+0000 7fb36fc00700 1 --2- 172.21.15.12:0/1517149238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb368160cc0 0x7fb368168730 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fb360009d90 tx=0x7fb36000b330 comp rx=0 tx=0).stop 2024-03-29T11:32:57.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.641+0000 7fb36fc00700 1 -- 172.21.15.12:0/1517149238 shutdown_connections 2024-03-29T11:32:57.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.641+0000 7fb36fc00700 1 --2- 172.21.15.12:0/1517149238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb368160cc0 0x7fb368168730 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.641+0000 7fb36fc00700 1 -- 172.21.15.12:0/1517149238 >> 172.21.15.12:0/1517149238 conn(0x7fb36808aff0 msgr2=0x7fb3680d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:57.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.641+0000 7fb36fc00700 1 -- 172.21.15.12:0/1517149238 shutdown_connections 2024-03-29T11:32:57.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.641+0000 7fb36fc00700 1 -- 172.21.15.12:0/1517149238 wait complete. 2024-03-29T11:32:57.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.643+0000 7fb36fc00700 1 Processor -- start 2024-03-29T11:32:57.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.643+0000 7fb36fc00700 1 -- start start 2024-03-29T11:32:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb36fc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3680ced80 0x7fb368150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb36fc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb36815dad0 0x7fb368153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb36fc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb368160cc0 0x7fb368154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb36fc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb368150f40 con 0x7fb36815dad0 2024-03-29T11:32:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb36fc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680c9860 con 0x7fb3680ced80 2024-03-29T11:32:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb36fc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680c9b40 con 0x7fb368160cc0 2024-03-29T11:32:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb36d200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3680ced80 0x7fb368150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb367e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb36815dad0 0x7fb368153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb36d200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3680ced80 0x7fb368150a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:55464/0 (socket says 172.21.15.12:55464) 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb36d200700 1 -- 172.21.15.12:0/3592602519 learned_addr learned my addr 172.21.15.12:0/3592602519 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.644+0000 7fb36dc00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb368160cc0 0x7fb368154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb36d200700 1 -- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb368160cc0 msgr2=0x7fb368154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb36d200700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb368160cc0 0x7fb368154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb36d200700 1 -- 172.21.15.12:0/3592602519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb36815dad0 msgr2=0x7fb368153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb36d200700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb36815dad0 0x7fb368153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb36d200700 1 -- 172.21.15.12:0/3592602519 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb360009a70 con 0x7fb3680ced80 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb36d200700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3680ced80 0x7fb368150a00 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00c520 tx=0x7fb35c00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c0052d0 con 0x7fb3680ced80 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c005430 con 0x7fb3680ced80 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c0038c0 con 0x7fb3680ced80 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb36fc00700 1 -- 172.21.15.12:0/3592602519 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3680c9e20 con 0x7fb3680ced80 2024-03-29T11:32:57.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.645+0000 7fb36fc00700 1 -- 172.21.15.12:0/3592602519 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3680b9110 con 0x7fb3680ced80 2024-03-29T11:32:57.643 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:57.646+0000 7f9cb0274240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:32:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.647+0000 7fb36fc00700 1 -- 172.21.15.12:0/3592602519 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3680ca440 con 0x7fb3680ced80 2024-03-29T11:32:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.647+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 140) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fb35c003a20 con 0x7fb3680ced80 2024-03-29T11:32:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.648+0000 7fb364c00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 0x7fb34806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:32:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.648+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6645+0+0 (secure 0 0 0) 0x7fb35c088f30 con 0x7fb3680ced80 2024-03-29T11:32:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.648+0000 7fb367e00700 1 -- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 msgr2=0x7fb34806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.649+0000 7fb367e00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 0x7fb34806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:32:57.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.653+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb35c053a30 con 0x7fb3680ced80 2024-03-29T11:32:57.748 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:57.753+0000 7eff153b1240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:32:57.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.850+0000 7fb367e00700 1 -- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 msgr2=0x7fb34806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:57.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.850+0000 7fb367e00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 0x7fb34806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:32:57.897 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:57.902+0000 7f9cb0274240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:32:57.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:57.916+0000 7fb36fc00700 1 -- 172.21.15.12:0/3592602519 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3680cf9e0 con 0x7fb34806af20 2024-03-29T11:32:58.017 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:58.021+0000 7eff153b1240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:32:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:58.251+0000 7fb367e00700 1 -- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 msgr2=0x7fb34806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:58.251+0000 7fb367e00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 0x7fb34806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T11:32:58.356 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:58.361+0000 7f9cb0274240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:32:58.497 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:58.501+0000 7eff153b1240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:32:59.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:59.052+0000 7fb367e00700 1 -- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 msgr2=0x7fb34806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:32:59.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:32:59.052+0000 7fb367e00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 0x7fb34806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T11:32:59.070 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:59.075+0000 7f9cb0274240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:32:59.163 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:59.168+0000 7f9cb0274240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:32:59.246 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:59.250+0000 7eff153b1240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:32:59.326 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:59.331+0000 7f9cb0274240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:32:59.343 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:59.347+0000 7eff153b1240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:32:59.512 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:59.516+0000 7eff153b1240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:32:59.542 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:59.547+0000 7f9cb0274240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:32:59.634 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:59.639+0000 7f9cb0274240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:32:59.732 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:59.736+0000 7eff153b1240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:32:59.733 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:59.738+0000 7f9cb0274240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:32:59.828 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:59.833+0000 7eff153b1240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:32:59.832 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:59.837+0000 7f9cb0274240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:32:59.930 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:32:59.936+0000 7f9cb0274240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:32:59.932 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:32:59.937+0000 7eff153b1240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:33:00.036 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:00.041+0000 7eff153b1240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:33:00.140 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:00.144+0000 7eff153b1240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:33:00.144 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:00.150+0000 7f9cb0274240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:33:00.364 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:00.368+0000 7eff153b1240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:33:00.404 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:00.409+0000 7f9cb0274240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:33:00.596 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:00.601+0000 7f9cb0274240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:33:00.628 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:00.633+0000 7eff153b1240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:33:00.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:00.654+0000 7fb367e00700 1 -- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 msgr2=0x7fb34806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:00.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:00.654+0000 7fb367e00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 0x7fb34806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T11:33:00.689 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:00.694+0000 7f9cb0274240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:00.831 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:00.835+0000 7eff153b1240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:33:00.929 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:00.933+0000 7eff153b1240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:01.425 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:01.430+0000 7f9cb0274240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:33:01.636 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:01.641+0000 7f9cb0274240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:33:01.697 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:01.701+0000 7eff153b1240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:33:01.842 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:01.847+0000 7f9cb0274240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:01.920 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:01.924+0000 7eff153b1240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:33:02.141 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:02.145+0000 7eff153b1240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:02.316 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:02.321+0000 7f9cb0274240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:33:02.525 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:02.530+0000 7f9cb0274240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:02.647 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:02.651+0000 7eff153b1240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:33:02.867 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:02.871+0000 7eff153b1240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:02.969 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:02.974+0000 7f9cb0274240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:33:03.080 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.085+0000 7f9cb0274240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:03.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:03.099+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mgrmap(e 141) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fb35c053e00 con 0x7fb3680ced80 2024-03-29T11:33:03.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:03.100+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 msgr2=0x7fb34806d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:03.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:03.100+0000 7fb364c00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 0x7fb34806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:03.343 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:03.347+0000 7eff153b1240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:33:03.359 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.363+0000 7f9c18000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.359 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.363+0000 7f9c18000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.359 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.363+0000 7f9c18000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.359 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.363+0000 7f9c18000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.359 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.363+0000 7f9c18000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.359 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.364+0000 7f9c1bc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.360 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.364+0000 7f9c1bc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.360 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.364+0000 7f9c1bc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.360 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.364+0000 7f9c1bc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.360 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.364+0000 7f9c1bc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.376+0000 7f9c18000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.376+0000 7f9c18000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.376+0000 7f9c18000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.376+0000 7f9c18000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.376+0000 7f9c18000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.377+0000 7f9c1bc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.377+0000 7f9c1bc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.377+0000 7f9c1bc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.377+0000 7f9c1bc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.372 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:03.377+0000 7f9c1bc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:03.466 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:03.470+0000 7eff153b1240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:04.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.104+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 <== mon.1 v2:172.21.15.159:3300/0 8 ==== mgrmap(e 142) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fb35c07e5b0 con 0x7fb3680ced80 2024-03-29T11:33:04.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.104+0000 7fb364c00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 0x7fb34806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:04.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.104+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb348035800 con 0x7fb34806af20 2024-03-29T11:33:04.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.107+0000 7fb367e00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 0x7fb34806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:04.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.110+0000 7fb367e00700 1 --2- 172.21.15.12:0/3592602519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb34806af20 0x7fb34806d3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb36806d530 tx=0x7fb354007de0 comp rx=0 tx=0).ready entity=mgr.11665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:04.108 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:04.112+0000 7f9c24800700 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2024-03-29T11:33:04.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.113+0000 7fb364c00700 1 -- 172.21.15.12:0/3592602519 <== mgr.11665 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7fb348035800 con 0x7fb34806af20 2024-03-29T11:33:04.111 INFO:teuthology.orchestra.run.smithi012.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2024-03-29T11:33:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-29T11:33:04.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:04.121 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-03-29T11:33:04.171 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T11:33:04.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.376+0000 7f53c7400700 1 Processor -- start 2024-03-29T11:33:04.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.377+0000 7f53c7400700 1 -- start start 2024-03-29T11:33:04.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.378+0000 7f53c7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f53c0160d70 0x7f53c00b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:04.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.378+0000 7f53c7400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c0161150 con 0x7f53c015ef70 2024-03-29T11:33:04.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.378+0000 7f53c7400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c006dba0 con 0x7f53c006a790 2024-03-29T11:33:04.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.378+0000 7f53c7400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c00b4cd0 con 0x7f53c0160d70 2024-03-29T11:33:04.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.378+0000 7f53c4a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f53c0160d70 0x7f53c00b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:04.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.378+0000 7f53bfe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f53c015ef70 0x7f53c015e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43104/0 (socket says 172.21.15.12:43104) 2024-03-29T11:33:04.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.378+0000 7f53bfe00700 1 -- 172.21.15.12:0/1553232877 learned_addr learned my addr 172.21.15.12:0/1553232877 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:04.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.379+0000 7f53c4a00700 1 -- 172.21.15.12:0/1553232877 >> v1:172.21.15.159:6789/0 conn(0x7f53c006a790 legacy=0x7f53c006e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:04.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.378+0000 7f53bf400700 1 -- 172.21.15.12:0/1553232877 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701501135 0 0) 0x7f53c0161150 con 0x7f53c015ef70 2024-03-29T11:33:04.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.379+0000 7f53c4a00700 1 -- 172.21.15.12:0/1553232877 >> v1:172.21.15.12:6789/0 conn(0x7f53c015ef70 legacy=0x7f53c015e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:04.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.379+0000 7f53c4a00700 1 -- 172.21.15.12:0/1553232877 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c00b4ee0 con 0x7f53c0160d70 2024-03-29T11:33:04.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.379+0000 7f53c4a00700 1 --2- 172.21.15.12:0/1553232877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c0160d70 0x7f53c00b3200 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009840 tx=0x7f53b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c56075d96a44e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:04.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.379+0000 7f53bf400700 1 -- 172.21.15.12:0/1553232877 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4011830 con 0x7f53c0160d70 2024-03-29T11:33:04.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.379+0000 7f53bf400700 1 -- 172.21.15.12:0/1553232877 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b4011990 con 0x7f53c0160d70 2024-03-29T11:33:04.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.379+0000 7f53bf400700 1 -- 172.21.15.12:0/1553232877 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4011d20 con 0x7f53c0160d70 2024-03-29T11:33:04.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.380+0000 7f53c7400700 1 -- 172.21.15.12:0/1553232877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c0160d70 msgr2=0x7f53c00b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:04.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.380+0000 7f53c7400700 1 --2- 172.21.15.12:0/1553232877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c0160d70 0x7f53c00b3200 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009840 tx=0x7f53b4009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:04.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.380+0000 7f53c7400700 1 -- 172.21.15.12:0/1553232877 shutdown_connections 2024-03-29T11:33:04.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.380+0000 7f53c7400700 1 --2- 172.21.15.12:0/1553232877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c0160d70 0x7f53c00b3200 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:04.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.380+0000 7f53c7400700 1 -- 172.21.15.12:0/1553232877 >> 172.21.15.12:0/1553232877 conn(0x7f53c00b5720 msgr2=0x7f53c00b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:04.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.380+0000 7f53c7400700 1 -- 172.21.15.12:0/1553232877 shutdown_connections 2024-03-29T11:33:04.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.380+0000 7f53c7400700 1 -- 172.21.15.12:0/1553232877 wait complete. 2024-03-29T11:33:04.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.381+0000 7f53c7400700 1 Processor -- start 2024-03-29T11:33:04.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.382+0000 7f53c7400700 1 -- start start 2024-03-29T11:33:04.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.382+0000 7f53c7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53c006a790 0x7f53c014cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:04.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.382+0000 7f53c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c015ef70 0x7f53c01f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:04.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.382+0000 7f53c7400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53c0160d70 0x7f53c01f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:04.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.382+0000 7f53c7400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c0149630 con 0x7f53c015ef70 2024-03-29T11:33:04.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.382+0000 7f53c7400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c014aee0 con 0x7f53c0160d70 2024-03-29T11:33:04.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53c7400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c014db80 con 0x7f53c006a790 2024-03-29T11:33:04.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c015ef70 0x7f53c01f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c015ef70 0x7f53c01f65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49102/0 (socket says 172.21.15.12:49102) 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53bfe00700 1 -- 172.21.15.12:0/2593462263 learned_addr learned my addr 172.21.15.12:0/2593462263 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53bfe00700 1 -- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53c006a790 msgr2=0x7f53c014cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53bfe00700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53c006a790 0x7f53c014cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53bfe00700 1 -- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53c0160d70 msgr2=0x7f53c01f6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53c5400700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53c0160d70 0x7f53c01f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53bfe00700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53c0160d70 0x7f53c01f6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53bfe00700 1 -- 172.21.15.12:0/2593462263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40094a0 con 0x7f53c015ef70 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53c5400700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53c0160d70 0x7f53c01f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.383+0000 7f53bfe00700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c015ef70 0x7f53c01f65d0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f53ac001eb0 tx=0x7f53ac00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.384+0000 7f53bcc00700 1 -- 172.21.15.12:0/2593462263 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac004c80 con 0x7f53c015ef70 2024-03-29T11:33:04.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.384+0000 7f53bcc00700 1 -- 172.21.15.12:0/2593462263 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac004de0 con 0x7f53c015ef70 2024-03-29T11:33:04.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.384+0000 7f53bcc00700 1 -- 172.21.15.12:0/2593462263 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac007310 con 0x7f53c015ef70 2024-03-29T11:33:04.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.384+0000 7f53c7400700 1 -- 172.21.15.12:0/2593462263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c0148d90 con 0x7f53c015ef70 2024-03-29T11:33:04.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.384+0000 7f53c7400700 1 -- 172.21.15.12:0/2593462263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c0149350 con 0x7f53c015ef70 2024-03-29T11:33:04.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.385+0000 7f53c7400700 1 -- 172.21.15.12:0/2593462263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c014acb0 con 0x7f53c015ef70 2024-03-29T11:33:04.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.386+0000 7f53bcc00700 1 -- 172.21.15.12:0/2593462263 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 142) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f53ac007470 con 0x7f53c015ef70 2024-03-29T11:33:04.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.387+0000 7f53bcc00700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f53a006ab30 0x7f53a006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:04.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.387+0000 7f53bcc00700 1 -- 172.21.15.12:0/2593462263 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7f53ac0891f0 con 0x7f53c015ef70 2024-03-29T11:33:04.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.387+0000 7f53c4a00700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f53a006ab30 0x7f53a006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:04.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.388+0000 7f53c4a00700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f53a006ab30 0x7f53a006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009b90 tx=0x7f53b4010000 comp rx=0 tx=0).ready entity=mgr.11665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:04.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.392+0000 7f53bcc00700 1 -- 172.21.15.12:0/2593462263 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f53ac0541b0 con 0x7f53c015ef70 2024-03-29T11:33:04.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:04.672+0000 7f53c7400700 1 -- 172.21.15.12:0/2593462263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f53c0067720 con 0x7f53c015ef70 2024-03-29T11:33:05.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.325+0000 7f53bcc00700 1 -- 172.21.15.12:0/2593462263 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f53ac054580 con 0x7f53c015ef70 2024-03-29T11:33:05.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.327+0000 7f53bcc00700 1 -- 172.21.15.12:0/2593462263 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v143) v1 ==== 93+0+0 (secure 0 0 0) 0x7f53ac072f00 con 0x7f53c015ef70 2024-03-29T11:33:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 -- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f53a006ab30 msgr2=0x7f53a006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f53a006ab30 0x7f53a006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009b90 tx=0x7f53b4010000 comp rx=0 tx=0).stop 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 -- 172.21.15.12:0/2593462263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c015ef70 msgr2=0x7f53c01f65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c015ef70 0x7f53c01f65d0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f53ac001eb0 tx=0x7f53ac00c8b0 comp rx=0 tx=0).stop 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 -- 172.21.15.12:0/2593462263 shutdown_connections 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f53a006ab30 0x7f53a006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53c0160d70 0x7f53c01f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53c006a790 0x7f53c014cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 --2- 172.21.15.12:0/2593462263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53c015ef70 0x7f53c01f65d0 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 -- 172.21.15.12:0/2593462263 >> 172.21.15.12:0/2593462263 conn(0x7f53c00b5720 msgr2=0x7f53c015fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.330+0000 7f53c7400700 1 -- 172.21.15.12:0/2593462263 shutdown_connections 2024-03-29T11:33:05.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.331+0000 7f53c7400700 1 -- 172.21.15.12:0/2593462263 wait complete. 2024-03-29T11:33:05.349 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:33:05.349 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:33:05.450 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:33:05.537 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:33:05.538 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:33:05.538 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:05.553 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:05.557+0000 7f4cdf48f240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:33:05.554 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:05.559+0000 7f8811038240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:33:05.582 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:33:05.582 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:05.668 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:05.741 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:05.745+0000 7f8811038240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:33:05.759 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:05.763+0000 7f4cdf48f240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:33:05.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.936+0000 7fa72ee00700 1 Processor -- start 2024-03-29T11:33:05.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.937+0000 7fa72ee00700 1 -- start start 2024-03-29T11:33:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.938+0000 7fa72ee00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7280afbe0 0x7fa72806dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.938+0000 7fa72ee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7280cebd0 0x7fa7280cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.938+0000 7fa72ee00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72806f850 con 0x7fa72806a790 2024-03-29T11:33:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.938+0000 7fa72ee00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72806fad0 con 0x7fa7280afbe0 2024-03-29T11:33:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.938+0000 7fa72ee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280af970 con 0x7fa7280cebd0 2024-03-29T11:33:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.938+0000 7fa727e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa72806a790 0x7fa72806ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43114/0 (socket says 172.21.15.12:43114) 2024-03-29T11:33:05.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.938+0000 7fa727400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7280afbe0 0x7fa72806dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.938+0000 7fa727e00700 1 -- 172.21.15.12:0/2054523140 learned_addr learned my addr 172.21.15.12:0/2054523140 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.938+0000 7fa72ce00700 1 --2- 172.21.15.12:0/2054523140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7280cebd0 0x7fa7280cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.939+0000 7fa727400700 1 -- 172.21.15.12:0/2054523140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7280cebd0 msgr2=0x7fa7280cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.939+0000 7fa727400700 1 --2- 172.21.15.12:0/2054523140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7280cebd0 0x7fa7280cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.939+0000 7fa727400700 1 -- 172.21.15.12:0/2054523140 >> v1:172.21.15.12:6789/0 conn(0x7fa72806a790 legacy=0x7fa72806ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.939+0000 7fa727400700 1 -- 172.21.15.12:0/2054523140 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7280af690 con 0x7fa7280afbe0 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.939+0000 7fa726a00700 1 -- 172.21.15.12:0/2054523140 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985306844 0 0) 0x7fa72806f850 con 0x7fa72806a790 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.939+0000 7fa727400700 1 --2- 172.21.15.12:0/2054523140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 0x7fa72806dce0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fa714009840 tx=0x7fa714009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a427dc37dc21b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.939+0000 7fa726a00700 1 -- 172.21.15.12:0/2054523140 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714011830 con 0x7fa7280afbe0 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.939+0000 7fa726a00700 1 -- 172.21.15.12:0/2054523140 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa714011990 con 0x7fa7280afbe0 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.939+0000 7fa726a00700 1 -- 172.21.15.12:0/2054523140 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714011d20 con 0x7fa7280afbe0 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.940+0000 7fa72ee00700 1 -- 172.21.15.12:0/2054523140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 msgr2=0x7fa72806dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.940+0000 7fa72ee00700 1 --2- 172.21.15.12:0/2054523140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 0x7fa72806dce0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fa714009840 tx=0x7fa714009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.940+0000 7fa72ee00700 1 -- 172.21.15.12:0/2054523140 shutdown_connections 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.940+0000 7fa72ee00700 1 --2- 172.21.15.12:0/2054523140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7280cebd0 0x7fa7280cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.940+0000 7fa72ee00700 1 --2- 172.21.15.12:0/2054523140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 0x7fa72806dce0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.940+0000 7fa72ee00700 1 -- 172.21.15.12:0/2054523140 >> 172.21.15.12:0/2054523140 conn(0x7fa7280b5ab0 msgr2=0x7fa7280b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.940+0000 7fa72ee00700 1 -- 172.21.15.12:0/2054523140 shutdown_connections 2024-03-29T11:33:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.940+0000 7fa72ee00700 1 -- 172.21.15.12:0/2054523140 wait complete. 2024-03-29T11:33:05.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.942+0000 7fa72ee00700 1 Processor -- start 2024-03-29T11:33:05.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.942+0000 7fa72ee00700 1 -- start start 2024-03-29T11:33:05.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa72ee00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa72806a790 0x7fa7280cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa72ee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 0x7fa72814d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa72ee00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280cebd0 0x7fa728146cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa72ee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280b2d40 con 0x7fa7280afbe0 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa72ee00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7281f7660 con 0x7fa7280cebd0 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa72ee00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280b2f90 con 0x7fa72806a790 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa727400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 0x7fa72814d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa727400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 0x7fa72814d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49138/0 (socket says 172.21.15.12:49138) 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa727400700 1 -- 172.21.15.12:0/3389538472 learned_addr learned my addr 172.21.15.12:0/3389538472 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa727e00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa72806a790 0x7fa7280cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.943+0000 7fa72ce00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280cebd0 0x7fa728146cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa727400700 1 -- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa72806a790 msgr2=0x7fa7280cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa727400700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa72806a790 0x7fa7280cb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:05.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa727400700 1 -- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280cebd0 msgr2=0x7fa728146cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa727400700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280cebd0 0x7fa728146cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:05.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa727400700 1 -- 172.21.15.12:0/3389538472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7140094a0 con 0x7fa7280afbe0 2024-03-29T11:33:05.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa727400700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 0x7fa72814d6c0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fa714009fd0 tx=0x7fa714004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:05.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa71fe00700 1 -- 172.21.15.12:0/3389538472 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714004550 con 0x7fa7280afbe0 2024-03-29T11:33:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa71fe00700 1 -- 172.21.15.12:0/3389538472 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71401f070 con 0x7fa7280afbe0 2024-03-29T11:33:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa72ee00700 1 -- 172.21.15.12:0/3389538472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7280b3270 con 0x7fa7280afbe0 2024-03-29T11:33:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa71fe00700 1 -- 172.21.15.12:0/3389538472 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7140038e0 con 0x7fa7280afbe0 2024-03-29T11:33:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.944+0000 7fa72ee00700 1 -- 172.21.15.12:0/3389538472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7280cbbe0 con 0x7fa7280afbe0 2024-03-29T11:33:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.946+0000 7fa72ee00700 1 -- 172.21.15.12:0/3389538472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7280cf560 con 0x7fa7280afbe0 2024-03-29T11:33:05.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.946+0000 7fa71fe00700 1 -- 172.21.15.12:0/3389538472 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fa714003a40 con 0x7fa7280afbe0 2024-03-29T11:33:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.947+0000 7fa71fe00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa70806aed0 0x7fa70806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.947+0000 7fa71fe00700 1 -- 172.21.15.12:0/3389538472 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7fa714021070 con 0x7fa7280afbe0 2024-03-29T11:33:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.948+0000 7fa727e00700 1 -- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa70806aed0 msgr2=0x7fa70806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:05.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.948+0000 7fa727e00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa70806aed0 0x7fa70806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:05.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:05.952+0000 7fa71fe00700 1 -- 172.21.15.12:0/3389538472 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7280cf560 con 0x7fa7280afbe0 2024-03-29T11:33:06.059 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:06.062+0000 7f8811038240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:33:06.104 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:06.108+0000 7f4cdf48f240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:33:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.149+0000 7fa727e00700 1 -- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa70806aed0 msgr2=0x7fa70806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.149+0000 7fa727e00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa70806aed0 0x7fa70806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.170+0000 7fa72ee00700 1 -- 172.21.15.12:0/3389538472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7280b3790 con 0x7fa7280afbe0 2024-03-29T11:33:06.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.172+0000 7fa71fe00700 1 -- 172.21.15.12:0/3389538472 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+20456 (secure 0 0 0) 0x7fa7280b3790 con 0x7fa7280afbe0 2024-03-29T11:33:06.168 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:06.168 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":253,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:03.088324+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:32:45.400233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:32:46.836046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2916339927":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/962755797":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1966802084":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/110096794":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/449562747":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/180562283":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/2535009377":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1057841974":"2024-03-30T11:33:03.088269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:06.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 -- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa70806aed0 msgr2=0x7fa70806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:06.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa70806aed0 0x7fa70806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 -- 172.21.15.12:0/3389538472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 msgr2=0x7fa72814d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 0x7fa72814d6c0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fa714009fd0 tx=0x7fa714004750 comp rx=0 tx=0).stop 2024-03-29T11:33:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 -- 172.21.15.12:0/3389538472 shutdown_connections 2024-03-29T11:33:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa70806aed0 0x7fa70806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7280cebd0 0x7fa728146cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa72806a790 0x7fa7280cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 --2- 172.21.15.12:0/3389538472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7280afbe0 0x7fa72814d6c0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 -- 172.21.15.12:0/3389538472 >> 172.21.15.12:0/3389538472 conn(0x7fa7280b5ab0 msgr2=0x7fa72815c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 -- 172.21.15.12:0/3389538472 shutdown_connections 2024-03-29T11:33:06.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.174+0000 7fa72ee00700 1 -- 172.21.15.12:0/3389538472 wait complete. 2024-03-29T11:33:06.191 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:32:31.368651+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:32:31.900686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-29T11:32:45.400233+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-29T11:32:46.836046+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:33:06.191 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:33:06.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.439+0000 7fac1b400700 1 Processor -- start 2024-03-29T11:33:06.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.439+0000 7fac1b400700 1 -- start start 2024-03-29T11:33:06.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.440+0000 7fac1b400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fac140d21f0 0x7fac140ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.440+0000 7fac1b400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac140cec40 0x7fac14161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.440+0000 7fac1b400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac140ce270 con 0x7fac141611b0 2024-03-29T11:33:06.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.440+0000 7fac1b400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1415b600 con 0x7fac140cec40 2024-03-29T11:33:06.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.440+0000 7fac1b400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac140d2760 con 0x7fac140d21f0 2024-03-29T11:33:06.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.440+0000 7fac19400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac140cec40 0x7fac14161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.440+0000 7fac19400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac140cec40 0x7fac14161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49142/0 (socket says 172.21.15.12:49142) 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.440+0000 7fac19400700 1 -- 172.21.15.12:0/1906972676 learned_addr learned my addr 172.21.15.12:0/1906972676 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.440+0000 7fac13e00700 1 --2- 172.21.15.12:0/1906972676 >> v2:172.21.15.159:3300/0 conn(0x7fac140d21f0 0x7fac140ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.441+0000 7fac19400700 1 -- 172.21.15.12:0/1906972676 >> v2:172.21.15.159:3300/0 conn(0x7fac140d21f0 msgr2=0x7fac140ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.441+0000 7fac19400700 1 --2- 172.21.15.12:0/1906972676 >> v2:172.21.15.159:3300/0 conn(0x7fac140d21f0 0x7fac140ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.441+0000 7fac19400700 1 -- 172.21.15.12:0/1906972676 >> v1:172.21.15.159:6789/0 conn(0x7fac141611b0 legacy=0x7fac1415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.441+0000 7fac19400700 1 -- 172.21.15.12:0/1906972676 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac140d2a40 con 0x7fac140cec40 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.441+0000 7fac19400700 1 --2- 172.21.15.12:0/1906972676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac140cec40 0x7fac14161590 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fac1408c030 tx=0x7fac0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1a644960f66e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.441+0000 7fac13400700 1 -- 172.21.15.12:0/1906972676 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c00e070 con 0x7fac140cec40 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.441+0000 7fac13400700 1 -- 172.21.15.12:0/1906972676 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c00b910 con 0x7fac140cec40 2024-03-29T11:33:06.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.441+0000 7fac13400700 1 -- 172.21.15.12:0/1906972676 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c00bcc0 con 0x7fac140cec40 2024-03-29T11:33:06.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.442+0000 7fac1b400700 1 -- 172.21.15.12:0/1906972676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac140cec40 msgr2=0x7fac14161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.442+0000 7fac1b400700 1 --2- 172.21.15.12:0/1906972676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac140cec40 0x7fac14161590 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fac1408c030 tx=0x7fac0c00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:06.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.442+0000 7fac1b400700 1 -- 172.21.15.12:0/1906972676 shutdown_connections 2024-03-29T11:33:06.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.442+0000 7fac1b400700 1 --2- 172.21.15.12:0/1906972676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac140cec40 0x7fac14161590 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.442+0000 7fac1b400700 1 --2- 172.21.15.12:0/1906972676 >> v2:172.21.15.159:3300/0 conn(0x7fac140d21f0 0x7fac140ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.442+0000 7fac1b400700 1 -- 172.21.15.12:0/1906972676 >> 172.21.15.12:0/1906972676 conn(0x7fac1408aff0 msgr2=0x7fac140d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:06.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.442+0000 7fac1b400700 1 -- 172.21.15.12:0/1906972676 shutdown_connections 2024-03-29T11:33:06.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.442+0000 7fac1b400700 1 -- 172.21.15.12:0/1906972676 wait complete. 2024-03-29T11:33:06.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.445+0000 7fac1b400700 1 Processor -- start 2024-03-29T11:33:06.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.445+0000 7fac1b400700 1 -- start start 2024-03-29T11:33:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.446+0000 7fac1b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac140cec40 0x7fac14156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.446+0000 7fac1b400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac140d21f0 0x7fac140ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.446+0000 7fac1b400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac141611b0 0x7fac141e7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.446+0000 7fac1b400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac140c74f0 con 0x7fac140cec40 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.446+0000 7fac1b400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac140c77d0 con 0x7fac140d21f0 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.446+0000 7fac1b400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac140c7ab0 con 0x7fac141611b0 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac13e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac140d21f0 0x7fac140ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac13e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac140d21f0 0x7fac140ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:51994/0 (socket says 172.21.15.12:51994) 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac13e00700 1 -- 172.21.15.12:0/4197141186 learned_addr learned my addr 172.21.15.12:0/4197141186 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac18a00700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac140cec40 0x7fac14156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac19400700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac141611b0 0x7fac141e7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac13e00700 1 -- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac141611b0 msgr2=0x7fac141e7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac13e00700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac141611b0 0x7fac141e7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac13e00700 1 -- 172.21.15.12:0/4197141186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac140cec40 msgr2=0x7fac14156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac13e00700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac140cec40 0x7fac14156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac13e00700 1 -- 172.21.15.12:0/4197141186 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c009b30 con 0x7fac140d21f0 2024-03-29T11:33:06.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.447+0000 7fac13e00700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac140d21f0 0x7fac140ccd30 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fac0000e9b0 tx=0x7fac0000ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:06.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.448+0000 7fac10c00700 1 -- 172.21.15.12:0/4197141186 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac00009710 con 0x7fac140d21f0 2024-03-29T11:33:06.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.448+0000 7fac10c00700 1 -- 172.21.15.12:0/4197141186 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0000ee50 con 0x7fac140d21f0 2024-03-29T11:33:06.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.448+0000 7fac10c00700 1 -- 172.21.15.12:0/4197141186 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0000ca50 con 0x7fac140d21f0 2024-03-29T11:33:06.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.448+0000 7fac1b400700 1 -- 172.21.15.12:0/4197141186 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac140c7d90 con 0x7fac140d21f0 2024-03-29T11:33:06.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.448+0000 7fac1b400700 1 -- 172.21.15.12:0/4197141186 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac141568c0 con 0x7fac140d21f0 2024-03-29T11:33:06.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.449+0000 7fac1b400700 1 -- 172.21.15.12:0/4197141186 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac140c82b0 con 0x7fac140d21f0 2024-03-29T11:33:06.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.450+0000 7fac10c00700 1 -- 172.21.15.12:0/4197141186 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fac0000cbb0 con 0x7fac140d21f0 2024-03-29T11:33:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.451+0000 7fac10c00700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabf406af20 0x7fabf406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.451+0000 7fac10c00700 1 -- 172.21.15.12:0/4197141186 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7fac00089c60 con 0x7fac140d21f0 2024-03-29T11:33:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.451+0000 7fac18a00700 1 -- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabf406af20 msgr2=0x7fabf406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.451+0000 7fac18a00700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabf406af20 0x7fabf406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:06.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.456+0000 7fac10c00700 1 -- 172.21.15.12:0/4197141186 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac140c82b0 con 0x7fac140d21f0 2024-03-29T11:33:06.588 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:06.592+0000 7f8811038240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:33:06.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.652+0000 7fac18a00700 1 -- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabf406af20 msgr2=0x7fabf406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:06.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.652+0000 7fac18a00700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabf406af20 0x7fabf406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:06.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.671+0000 7fac1b400700 1 -- 172.21.15.12:0/4197141186 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac140c82b0 con 0x7fac140d21f0 2024-03-29T11:33:06.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.672+0000 7fac10c00700 1 -- 172.21.15.12:0/4197141186 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v253) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac140c82b0 con 0x7fac140d21f0 2024-03-29T11:33:06.668 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:33:06.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 -- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabf406af20 msgr2=0x7fabf406d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabf406af20 0x7fabf406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 -- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac140d21f0 msgr2=0x7fac140ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac140d21f0 0x7fac140ccd30 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fac0000e9b0 tx=0x7fac0000ecc0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 -- 172.21.15.12:0/4197141186 shutdown_connections 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabf406af20 0x7fabf406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac141611b0 0x7fac141e7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac140cec40 0x7fac14156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 --2- 172.21.15.12:0/4197141186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac140d21f0 0x7fac140ccd30 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 -- 172.21.15.12:0/4197141186 >> 172.21.15.12:0/4197141186 conn(0x7fac1408aff0 msgr2=0x7fac1406c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 -- 172.21.15.12:0/4197141186 shutdown_connections 2024-03-29T11:33:06.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.674+0000 7fac1b400700 1 -- 172.21.15.12:0/4197141186 wait complete. 2024-03-29T11:33:06.672 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:06.675+0000 7f4cdf48f240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:33:06.690 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:33:06.843 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:06.847+0000 7f8811038240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:33:06.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.899+0000 7f02d7c00700 1 Processor -- start 2024-03-29T11:33:06.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.899+0000 7f02d7c00700 1 -- start start 2024-03-29T11:33:06.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.901+0000 7f02d7c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f02d00cd5c0 0x7f02d015e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.901+0000 7f02d7c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02d006f310 0x7f02d006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.901+0000 7f02d7c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d015eda0 con 0x7f02d015cfe0 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.901+0000 7f02d7c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d006fce0 con 0x7f02d00cd5c0 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.901+0000 7f02d7c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d0099dd0 con 0x7f02d006f310 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.901+0000 7f02d5200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f02d00cd5c0 0x7f02d015e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.901+0000 7f02d5c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f02d015cfe0 0x7f02d015d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43118/0 (socket says 172.21.15.12:43118) 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.901+0000 7f02d5c00700 1 -- 172.21.15.12:0/3159526261 learned_addr learned my addr 172.21.15.12:0/3159526261 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.901+0000 7f02cfe00700 1 --2- 172.21.15.12:0/3159526261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02d006f310 0x7f02d006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.902+0000 7f02d5200700 1 -- 172.21.15.12:0/3159526261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02d006f310 msgr2=0x7f02d006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.902+0000 7f02d5200700 1 --2- 172.21.15.12:0/3159526261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02d006f310 0x7f02d006f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.902+0000 7f02d5200700 1 -- 172.21.15.12:0/3159526261 >> v1:172.21.15.12:6789/0 conn(0x7f02d015cfe0 legacy=0x7f02d015d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.902+0000 7f02d5200700 1 -- 172.21.15.12:0/3159526261 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d009a0b0 con 0x7f02d00cd5c0 2024-03-29T11:33:06.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.902+0000 7f02d5200700 1 --2- 172.21.15.12:0/3159526261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d00cd5c0 0x7f02d015e860 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009840 tx=0x7f02c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03ecb25941eb1e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.902+0000 7f02cf400700 1 -- 172.21.15.12:0/3159526261 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4011830 con 0x7f02d00cd5c0 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.902+0000 7f02cf400700 1 -- 172.21.15.12:0/3159526261 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c4011990 con 0x7f02d00cd5c0 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.902+0000 7f02cf400700 1 -- 172.21.15.12:0/3159526261 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4011d20 con 0x7f02d00cd5c0 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.903+0000 7f02d7c00700 1 -- 172.21.15.12:0/3159526261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d00cd5c0 msgr2=0x7f02d015e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.903+0000 7f02d7c00700 1 --2- 172.21.15.12:0/3159526261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d00cd5c0 0x7f02d015e860 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009840 tx=0x7f02c4009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.903+0000 7f02d7c00700 1 -- 172.21.15.12:0/3159526261 shutdown_connections 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.903+0000 7f02d7c00700 1 --2- 172.21.15.12:0/3159526261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d00cd5c0 0x7f02d015e860 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.903+0000 7f02d7c00700 1 --2- 172.21.15.12:0/3159526261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02d006f310 0x7f02d006f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.903+0000 7f02d7c00700 1 -- 172.21.15.12:0/3159526261 >> 172.21.15.12:0/3159526261 conn(0x7f02d00d1370 msgr2=0x7f02d00d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.903+0000 7f02d7c00700 1 -- 172.21.15.12:0/3159526261 shutdown_connections 2024-03-29T11:33:06.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.903+0000 7f02d7c00700 1 -- 172.21.15.12:0/3159526261 wait complete. 2024-03-29T11:33:06.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.906+0000 7f02d7c00700 1 Processor -- start 2024-03-29T11:33:06.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.906+0000 7f02d7c00700 1 -- start start 2024-03-29T11:33:06.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02d7c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d006f310 0x7f02d0160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02d7c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02d00cd5c0 0x7f02d015c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02d7c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02d015cfe0 0x7f02d015c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02d7c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d0155180 con 0x7f02d006f310 2024-03-29T11:33:06.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02d7c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d00b5c20 con 0x7f02d015cfe0 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02d7c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d0160030 con 0x7f02d00cd5c0 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02d5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d006f310 0x7f02d0160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02d5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d006f310 0x7f02d0160440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49172/0 (socket says 172.21.15.12:49172) 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02d5200700 1 -- 172.21.15.12:0/771611884 learned_addr learned my addr 172.21.15.12:0/771611884 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02cfe00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02d00cd5c0 0x7f02d015c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.907+0000 7f02d5c00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02d015cfe0 0x7f02d015c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02d5200700 1 -- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02d00cd5c0 msgr2=0x7f02d015c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02d5200700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02d00cd5c0 0x7f02d015c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02d5200700 1 -- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02d015cfe0 msgr2=0x7f02d015c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02d5200700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02d015cfe0 0x7f02d015c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02d5200700 1 -- 172.21.15.12:0/771611884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c40094a0 con 0x7f02d006f310 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02d5c00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02d015cfe0 0x7f02d015c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02d5200700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d006f310 0x7f02d0160440 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f02c4000c00 tx=0x7f02c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:06.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02ccc00700 1 -- 172.21.15.12:0/771611884 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c40036a0 con 0x7f02d006f310 2024-03-29T11:33:06.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02ccc00700 1 -- 172.21.15.12:0/771611884 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c401f070 con 0x7f02d006f310 2024-03-29T11:33:06.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02d7c00700 1 -- 172.21.15.12:0/771611884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d015f420 con 0x7f02d006f310 2024-03-29T11:33:06.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.908+0000 7f02ccc00700 1 -- 172.21.15.12:0/771611884 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4003a00 con 0x7f02d006f310 2024-03-29T11:33:06.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.909+0000 7f02d7c00700 1 -- 172.21.15.12:0/771611884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d015f9e0 con 0x7f02d006f310 2024-03-29T11:33:06.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.910+0000 7f02d7c00700 1 -- 172.21.15.12:0/771611884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d0067720 con 0x7f02d006f310 2024-03-29T11:33:06.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.910+0000 7f02ccc00700 1 -- 172.21.15.12:0/771611884 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f02c4003b60 con 0x7f02d006f310 2024-03-29T11:33:06.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.911+0000 7f02ccc00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f02b006aed0 0x7f02b006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:06.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.911+0000 7f02ccc00700 1 -- 172.21.15.12:0/771611884 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7f02c4021070 con 0x7f02d006f310 2024-03-29T11:33:06.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.912+0000 7f02cfe00700 1 -- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f02b006aed0 msgr2=0x7f02b006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:06.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.912+0000 7f02cfe00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f02b006aed0 0x7f02b006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:06.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:06.916+0000 7f02ccc00700 1 -- 172.21.15.12:0/771611884 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f02c4055f10 con 0x7f02d006f310 2024-03-29T11:33:06.944 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:06.948+0000 7f4cdf48f240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:33:07.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.113+0000 7f02cfe00700 1 -- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f02b006aed0 msgr2=0x7f02b006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:07.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.113+0000 7f02cfe00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f02b006aed0 0x7f02b006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.132+0000 7f02d7c00700 1 -- 172.21.15.12:0/771611884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f02d00d0e20 con 0x7f02d006f310 2024-03-29T11:33:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.132+0000 7f02ccc00700 1 -- 172.21.15.12:0/771611884 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v253) v1 ==== 104+0+11 (secure 0 0 0) 0x7f02c4059530 con 0x7f02d006f310 2024-03-29T11:33:07.128 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:33:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 -- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f02b006aed0 msgr2=0x7f02b006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f02b006aed0 0x7f02b006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 -- 172.21.15.12:0/771611884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d006f310 msgr2=0x7f02d0160440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d006f310 0x7f02d0160440 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f02c4000c00 tx=0x7f02c4004750 comp rx=0 tx=0).stop 2024-03-29T11:33:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 -- 172.21.15.12:0/771611884 shutdown_connections 2024-03-29T11:33:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f02b006aed0 0x7f02b006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02d015cfe0 0x7f02d015c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02d006f310 0x7f02d0160440 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 --2- 172.21.15.12:0/771611884 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02d00cd5c0 0x7f02d015c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 -- 172.21.15.12:0/771611884 >> 172.21.15.12:0/771611884 conn(0x7f02d00d1370 msgr2=0x7f02d006b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:07.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 -- 172.21.15.12:0/771611884 shutdown_connections 2024-03-29T11:33:07.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.135+0000 7f02d7c00700 1 -- 172.21.15.12:0/771611884 wait complete. 2024-03-29T11:33:07.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:33:07.302 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:07.306+0000 7f8811038240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:33:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.361+0000 7fabfb800700 1 Processor -- start 2024-03-29T11:33:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.361+0000 7fabfb800700 1 -- start start 2024-03-29T11:33:07.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.363+0000 7fabfb800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 0x7fabf40b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.363+0000 7fabfb800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf40b3db0 con 0x7fabf40b3960 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.363+0000 7fabfb800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf40b2070 con 0x7fabf4157f90 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.363+0000 7fabfb800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf4158370 con 0x7fabf40b2b30 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.363+0000 7fabf9800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fabf40b3960 0x7fabf40cd020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43124/0 (socket says 172.21.15.12:43124) 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.363+0000 7fabf9800700 1 -- 172.21.15.12:0/1807357146 learned_addr learned my addr 172.21.15.12:0/1807357146 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.363+0000 7fabf3e00700 1 --2- 172.21.15.12:0/1807357146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 0x7fabf40b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.363+0000 7fabf3400700 1 -- 172.21.15.12:0/1807357146 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596285553 0 0) 0x7fabec001490 con 0x7fabf40b3960 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.364+0000 7fabf3400700 1 -- 172.21.15.12:0/1807357146 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabdc0035b0 con 0x7fabf40b3960 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.364+0000 7fabf3400700 1 -- 172.21.15.12:0/1807357146 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210983376 0 0) 0x7fabf40b2070 con 0x7fabf4157f90 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.364+0000 7fabf3400700 1 -- 172.21.15.12:0/1807357146 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabec001490 con 0x7fabf4157f90 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.364+0000 7fabf3e00700 1 -- 172.21.15.12:0/1807357146 >> v1:172.21.15.159:6789/0 conn(0x7fabf4157f90 legacy=0x7fabf41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.364+0000 7fabf3e00700 1 -- 172.21.15.12:0/1807357146 >> v1:172.21.15.12:6789/0 conn(0x7fabf40b3960 legacy=0x7fabf40cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.364+0000 7fabf3e00700 1 -- 172.21.15.12:0/1807357146 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf415e0f0 con 0x7fabf40b2b30 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.364+0000 7fabf3e00700 1 --2- 172.21.15.12:0/1807357146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 0x7fabf40b3420 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009870 tx=0x7fabe0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb661df36cfc250 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.364+0000 7fabf3400700 1 -- 172.21.15.12:0/1807357146 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0011830 con 0x7fabf40b2b30 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.364+0000 7fabf3400700 1 -- 172.21.15.12:0/1807357146 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe0011990 con 0x7fabf40b2b30 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.364+0000 7fabf3400700 1 -- 172.21.15.12:0/1807357146 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0011d00 con 0x7fabf40b2b30 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.365+0000 7fabfb800700 1 -- 172.21.15.12:0/1807357146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 msgr2=0x7fabf40b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.365+0000 7fabfb800700 1 --2- 172.21.15.12:0/1807357146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 0x7fabf40b3420 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009870 tx=0x7fabe0009b80 comp rx=0 tx=0).stop 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.365+0000 7fabfb800700 1 -- 172.21.15.12:0/1807357146 shutdown_connections 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.365+0000 7fabfb800700 1 --2- 172.21.15.12:0/1807357146 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 0x7fabf40b3420 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.365+0000 7fabfb800700 1 -- 172.21.15.12:0/1807357146 >> 172.21.15.12:0/1807357146 conn(0x7fabf40b0d70 msgr2=0x7fabf40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.365+0000 7fabfb800700 1 -- 172.21.15.12:0/1807357146 shutdown_connections 2024-03-29T11:33:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.365+0000 7fabfb800700 1 -- 172.21.15.12:0/1807357146 wait complete. 2024-03-29T11:33:07.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.367+0000 7fabfb800700 1 Processor -- start 2024-03-29T11:33:07.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.367+0000 7fabfb800700 1 -- start start 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.368+0000 7fabfb800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 0x7fabf4153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.368+0000 7fabfb800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf40b3960 0x7fabf41f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.368+0000 7fabfb800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabf4157f90 0x7fabf4156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.368+0000 7fabfb800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf40b2710 con 0x7fabf4157f90 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.368+0000 7fabfb800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf4154180 con 0x7fabf40b3960 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.368+0000 7fabfb800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf4154420 con 0x7fabf40b2b30 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.368+0000 7fabf9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabf4157f90 0x7fabf4156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.368+0000 7fabf3e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf40b3960 0x7fabf41f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.369+0000 7fabf3e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf40b3960 0x7fabf41f46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:52012/0 (socket says 172.21.15.12:52012) 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.369+0000 7fabf3e00700 1 -- 172.21.15.12:0/3210400371 learned_addr learned my addr 172.21.15.12:0/3210400371 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:07.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.369+0000 7fabf8e00700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 0x7fabf4153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:07.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.369+0000 7fabf3e00700 1 -- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 msgr2=0x7fabf4153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.369+0000 7fabf3e00700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 0x7fabf4153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.369+0000 7fabf3e00700 1 -- 172.21.15.12:0/3210400371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabf4157f90 msgr2=0x7fabf4156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.369+0000 7fabf3e00700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabf4157f90 0x7fabf4156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.369+0000 7fabf3e00700 1 -- 172.21.15.12:0/3210400371 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe00094a0 con 0x7fabf40b3960 2024-03-29T11:33:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.369+0000 7fabf3e00700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf40b3960 0x7fabf41f46d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fabe00097c0 tx=0x7fabe0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.370+0000 7fabf0c00700 1 -- 172.21.15.12:0/3210400371 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0004450 con 0x7fabf40b3960 2024-03-29T11:33:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.370+0000 7fabf0c00700 1 -- 172.21.15.12:0/3210400371 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe001e070 con 0x7fabf40b3960 2024-03-29T11:33:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.370+0000 7fabf0c00700 1 -- 172.21.15.12:0/3210400371 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe00038e0 con 0x7fabf40b3960 2024-03-29T11:33:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.370+0000 7fabfb800700 1 -- 172.21.15.12:0/3210400371 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf4154700 con 0x7fabf40b3960 2024-03-29T11:33:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.370+0000 7fabfb800700 1 -- 172.21.15.12:0/3210400371 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf4153b40 con 0x7fabf40b3960 2024-03-29T11:33:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.371+0000 7fabfb800700 1 -- 172.21.15.12:0/3210400371 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf4154c20 con 0x7fabf40b3960 2024-03-29T11:33:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.372+0000 7fabf0c00700 1 -- 172.21.15.12:0/3210400371 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fabe0003a40 con 0x7fabf40b3960 2024-03-29T11:33:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.373+0000 7fabf0c00700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabd406aed0 0x7fabd406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.373+0000 7fabf0c00700 1 -- 172.21.15.12:0/3210400371 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7fabe008aa90 con 0x7fabf40b3960 2024-03-29T11:33:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.373+0000 7fabf8e00700 1 -- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabd406aed0 msgr2=0x7fabd406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.373+0000 7fabf8e00700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabd406aed0 0x7fabd406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.377+0000 7fabf0c00700 1 -- 172.21.15.12:0/3210400371 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fabf4154c20 con 0x7fabf40b3960 2024-03-29T11:33:07.460 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:07.464+0000 7f4cdf48f240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:33:07.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.574+0000 7fabf8e00700 1 -- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabd406aed0 msgr2=0x7fabd406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:07.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.574+0000 7fabf8e00700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabd406aed0 0x7fabd406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:07.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.625+0000 7fabfb800700 1 -- 172.21.15.12:0/3210400371 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fabf4154c20 con 0x7fabf40b3960 2024-03-29T11:33:07.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.625+0000 7fabf0c00700 1 -- 172.21.15.12:0/3210400371 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v253) v1 ==== 100+0+11 (secure 0 0 0) 0x7fabf4154c20 con 0x7fabf40b3960 2024-03-29T11:33:07.621 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.629+0000 7fabfb800700 1 -- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabd406aed0 msgr2=0x7fabd406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.629+0000 7fabfb800700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabd406aed0 0x7fabd406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.629+0000 7fabfb800700 1 -- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf40b3960 msgr2=0x7fabf41f46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.629+0000 7fabfb800700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf40b3960 0x7fabf41f46d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fabe00097c0 tx=0x7fabe0004750 comp rx=0 tx=0).stop 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.629+0000 7fabfb800700 1 -- 172.21.15.12:0/3210400371 shutdown_connections 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.629+0000 7fabfb800700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fabd406aed0 0x7fabd406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.629+0000 7fabfb800700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabf4157f90 0x7fabf4156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.629+0000 7fabfb800700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf40b2b30 0x7fabf4153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.629+0000 7fabfb800700 1 --2- 172.21.15.12:0/3210400371 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf40b3960 0x7fabf41f46d0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.629+0000 7fabfb800700 1 -- 172.21.15.12:0/3210400371 >> 172.21.15.12:0/3210400371 conn(0x7fabf40b0d70 msgr2=0x7fabf415d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.630+0000 7fabfb800700 1 -- 172.21.15.12:0/3210400371 shutdown_connections 2024-03-29T11:33:07.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.630+0000 7fabfb800700 1 -- 172.21.15.12:0/3210400371 wait complete. 2024-03-29T11:33:07.644 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-29T11:33:07.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.851+0000 7f0328200700 1 Processor -- start 2024-03-29T11:33:07.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.851+0000 7f0328200700 1 -- start start 2024-03-29T11:33:07.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.852+0000 7f0328200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0320157f90 0x7f03200b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:07.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.853+0000 7f0328200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03200b3070 con 0x7f03201613e0 2024-03-29T11:33:07.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.853+0000 7f0328200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03200b2070 con 0x7f03200b3b30 2024-03-29T11:33:07.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.853+0000 7f0328200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0320158370 con 0x7f0320157f90 2024-03-29T11:33:07.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.853+0000 7f0324e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f03201613e0 0x7f03200b3420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43136/0 (socket says 172.21.15.12:43136) 2024-03-29T11:33:07.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.853+0000 7f0324e00700 1 -- 172.21.15.12:0/1201389781 learned_addr learned my addr 172.21.15.12:0/1201389781 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:07.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.853+0000 7f0325800700 1 --2- 172.21.15.12:0/1201389781 >> v2:172.21.15.12:3300/0 conn(0x7f0320157f90 0x7f03200b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:07.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.853+0000 7f031fe00700 1 -- 172.21.15.12:0/1201389781 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621182469 0 0) 0x7f030c001110 con 0x7f03201613e0 2024-03-29T11:33:07.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.853+0000 7f031fe00700 1 -- 172.21.15.12:0/1201389781 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03080035b0 con 0x7f03201613e0 2024-03-29T11:33:07.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.854+0000 7f031fe00700 1 -- 172.21.15.12:0/1201389781 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577211189 0 0) 0x7f03200b2070 con 0x7f03200b3b30 2024-03-29T11:33:07.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.854+0000 7f031fe00700 1 -- 172.21.15.12:0/1201389781 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f030c001110 con 0x7f03200b3b30 2024-03-29T11:33:07.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.854+0000 7f0325800700 1 -- 172.21.15.12:0/1201389781 >> v1:172.21.15.159:6789/0 conn(0x7f03200b3b30 legacy=0x7f03200cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.854+0000 7f0325800700 1 -- 172.21.15.12:0/1201389781 >> v1:172.21.15.12:6789/0 conn(0x7f03201613e0 legacy=0x7f03200b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.854+0000 7f0325800700 1 -- 172.21.15.12:0/1201389781 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032015e0f0 con 0x7f0320157f90 2024-03-29T11:33:07.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.854+0000 7f0325800700 1 --2- 172.21.15.12:0/1201389781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320157f90 0x7f03200b2b30 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f0314009840 tx=0x7f0314009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935fbb9b16aeee84 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:07.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.854+0000 7f031fe00700 1 -- 172.21.15.12:0/1201389781 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314011830 con 0x7f0320157f90 2024-03-29T11:33:07.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.854+0000 7f031fe00700 1 -- 172.21.15.12:0/1201389781 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0314011990 con 0x7f0320157f90 2024-03-29T11:33:07.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.854+0000 7f031fe00700 1 -- 172.21.15.12:0/1201389781 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314011d00 con 0x7f0320157f90 2024-03-29T11:33:07.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.855+0000 7f0328200700 1 -- 172.21.15.12:0/1201389781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320157f90 msgr2=0x7f03200b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.855+0000 7f0328200700 1 --2- 172.21.15.12:0/1201389781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320157f90 0x7f03200b2b30 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f0314009840 tx=0x7f0314009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:07.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.855+0000 7f0328200700 1 -- 172.21.15.12:0/1201389781 shutdown_connections 2024-03-29T11:33:07.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.855+0000 7f0328200700 1 --2- 172.21.15.12:0/1201389781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0320157f90 0x7f03200b2b30 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.855+0000 7f0328200700 1 -- 172.21.15.12:0/1201389781 >> 172.21.15.12:0/1201389781 conn(0x7f03200b0d70 msgr2=0x7f03200b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:07.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.855+0000 7f0328200700 1 -- 172.21.15.12:0/1201389781 shutdown_connections 2024-03-29T11:33:07.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.855+0000 7f0328200700 1 -- 172.21.15.12:0/1201389781 wait complete. 2024-03-29T11:33:07.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.857+0000 7f0328200700 1 Processor -- start 2024-03-29T11:33:07.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.858+0000 7f0328200700 1 -- start start 2024-03-29T11:33:07.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0328200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03200b3b30 0x7f03201ed7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:07.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0328200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0320157f90 0x7f03201f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:07.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0328200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03201613e0 0x7f0320155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:07.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0328200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03200b2710 con 0x7f03200b3b30 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0328200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0320153fb0 con 0x7f0320157f90 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0328200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03201f4650 con 0x7f03201613e0 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0325800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03200b3b30 0x7f03201ed7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0325800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03200b3b30 0x7f03201ed7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49198/0 (socket says 172.21.15.12:49198) 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0325800700 1 -- 172.21.15.12:0/1908861363 learned_addr learned my addr 172.21.15.12:0/1908861363 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0324e00700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0320157f90 0x7f03201f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.859+0000 7f0326200700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03201613e0 0x7f0320155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f0325800700 1 -- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03201613e0 msgr2=0x7f0320155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f0325800700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03201613e0 0x7f0320155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f0325800700 1 -- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0320157f90 msgr2=0x7f03201f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f0325800700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0320157f90 0x7f03201f4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f0325800700 1 -- 172.21.15.12:0/1908861363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03140094a0 con 0x7f03200b3b30 2024-03-29T11:33:07.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f0324e00700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0320157f90 0x7f03201f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:07.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f0325800700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03200b3b30 0x7f03201ed7d0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f03140090b0 tx=0x7f0314004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:07.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f031d600700 1 -- 172.21.15.12:0/1908861363 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03140044d0 con 0x7f03200b3b30 2024-03-29T11:33:07.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f031d600700 1 -- 172.21.15.12:0/1908861363 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031401f070 con 0x7f03200b3b30 2024-03-29T11:33:07.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f0328200700 1 -- 172.21.15.12:0/1908861363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0320154200 con 0x7f03200b3b30 2024-03-29T11:33:07.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f031d600700 1 -- 172.21.15.12:0/1908861363 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03140038e0 con 0x7f03200b3b30 2024-03-29T11:33:07.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.860+0000 7f0328200700 1 -- 172.21.15.12:0/1908861363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03201547c0 con 0x7f03200b3b30 2024-03-29T11:33:07.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.862+0000 7f0328200700 1 -- 172.21.15.12:0/1908861363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03201550f0 con 0x7f03200b3b30 2024-03-29T11:33:07.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.862+0000 7f031d600700 1 -- 172.21.15.12:0/1908861363 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f0314003a40 con 0x7f03200b3b30 2024-03-29T11:33:07.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.863+0000 7f031d600700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f030006aed0 0x7f030006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:07.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.863+0000 7f031d600700 1 -- 172.21.15.12:0/1908861363 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7f0314021070 con 0x7f03200b3b30 2024-03-29T11:33:07.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.863+0000 7f0324e00700 1 -- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f030006aed0 msgr2=0x7f030006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.863+0000 7f0324e00700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f030006aed0 0x7f030006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:07.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:07.868+0000 7f031d600700 1 -- 172.21.15.12:0/1908861363 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0314055e10 con 0x7f03200b3b30 2024-03-29T11:33:08.015 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:08.019+0000 7f8811038240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:33:08.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.064+0000 7f0324e00700 1 -- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f030006aed0 msgr2=0x7f030006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:08.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.064+0000 7f0324e00700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f030006aed0 0x7f030006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.084+0000 7f0328200700 1 -- 172.21.15.12:0/1908861363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0320067720 con 0x7f03200b3b30 2024-03-29T11:33:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.085+0000 7f031d600700 1 -- 172.21.15.12:0/1908861363 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v253) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0314059430 con 0x7f03200b3b30 2024-03-29T11:33:08.081 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 -- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f030006aed0 msgr2=0x7f030006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f030006aed0 0x7f030006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 -- 172.21.15.12:0/1908861363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03200b3b30 msgr2=0x7f03201ed7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03200b3b30 0x7f03201ed7d0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f03140090b0 tx=0x7f0314004750 comp rx=0 tx=0).stop 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 -- 172.21.15.12:0/1908861363 shutdown_connections 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f030006aed0 0x7f030006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03201613e0 0x7f0320155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03200b3b30 0x7f03201ed7d0 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 --2- 172.21.15.12:0/1908861363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0320157f90 0x7f03201f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 -- 172.21.15.12:0/1908861363 >> 172.21.15.12:0/1908861363 conn(0x7f03200b0d70 msgr2=0x7f03200b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 -- 172.21.15.12:0/1908861363 shutdown_connections 2024-03-29T11:33:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.087+0000 7f0328200700 1 -- 172.21.15.12:0/1908861363 wait complete. 2024-03-29T11:33:08.102 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-29T11:33:08.106 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:08.110+0000 7f8811038240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:33:08.234 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:08.239+0000 7f4cdf48f240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:33:08.265 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:08.268+0000 7f8811038240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:33:08.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.315+0000 7f0a6cc00700 1 Processor -- start 2024-03-29T11:33:08.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.315+0000 7f0a6cc00700 1 -- start start 2024-03-29T11:33:08.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.316+0000 7f0a6cc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a680cd020 0x7f0a680cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.316+0000 7f0a6cc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680cdae0 con 0x7f0a680cbb30 2024-03-29T11:33:08.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.316+0000 7f0a6cc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680cddc0 con 0x7f0a680b2070 2024-03-29T11:33:08.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.316+0000 7f0a6cc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a680ce0a0 con 0x7f0a680cd020 2024-03-29T11:33:08.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.316+0000 7f0a65400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a680cd020 0x7f0a680cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:08.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.316+0000 7f0a65e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0a680cbb30 0x7f0a680cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43140/0 (socket says 172.21.15.12:43140) 2024-03-29T11:33:08.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.316+0000 7f0a65e00700 1 -- 172.21.15.12:0/2760199522 learned_addr learned my addr 172.21.15.12:0/2760199522 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.317+0000 7f0a65400700 1 -- 172.21.15.12:0/2760199522 >> v1:172.21.15.159:6789/0 conn(0x7f0a680b2070 legacy=0x7f0a6806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.317+0000 7f0a65400700 1 -- 172.21.15.12:0/2760199522 >> v1:172.21.15.12:6789/0 conn(0x7f0a680cbb30 legacy=0x7f0a680cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.317+0000 7f0a65400700 1 -- 172.21.15.12:0/2760199522 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6806dc20 con 0x7f0a680cd020 2024-03-29T11:33:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.317+0000 7f0a65400700 1 --2- 172.21.15.12:0/2760199522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 0x7f0a680cd480 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009840 tx=0x7f0a54009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfabb73f466b48a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.317+0000 7f0a64a00700 1 -- 172.21.15.12:0/2760199522 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a54011830 con 0x7f0a680cd020 2024-03-29T11:33:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.317+0000 7f0a64a00700 1 -- 172.21.15.12:0/2760199522 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a54011990 con 0x7f0a680cd020 2024-03-29T11:33:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.317+0000 7f0a64a00700 1 -- 172.21.15.12:0/2760199522 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a54011d20 con 0x7f0a680cd020 2024-03-29T11:33:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.318+0000 7f0a6cc00700 1 -- 172.21.15.12:0/2760199522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 msgr2=0x7f0a680cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.318+0000 7f0a6cc00700 1 --2- 172.21.15.12:0/2760199522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 0x7f0a680cd480 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009840 tx=0x7f0a54009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.318+0000 7f0a6cc00700 1 -- 172.21.15.12:0/2760199522 shutdown_connections 2024-03-29T11:33:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.318+0000 7f0a6cc00700 1 --2- 172.21.15.12:0/2760199522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 0x7f0a680cd480 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.318+0000 7f0a6cc00700 1 -- 172.21.15.12:0/2760199522 >> 172.21.15.12:0/2760199522 conn(0x7f0a680b0d70 msgr2=0x7f0a680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.318+0000 7f0a6cc00700 1 -- 172.21.15.12:0/2760199522 shutdown_connections 2024-03-29T11:33:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.318+0000 7f0a6cc00700 1 -- 172.21.15.12:0/2760199522 wait complete. 2024-03-29T11:33:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.320+0000 7f0a6cc00700 1 Processor -- start 2024-03-29T11:33:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.320+0000 7f0a6cc00700 1 -- start start 2024-03-29T11:33:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.321+0000 7f0a6cc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a680b2070 0x7f0a680adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.321+0000 7f0a6cc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a680cbb30 0x7f0a68152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.321+0000 7f0a6cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 0x7f0a6815a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.321+0000 7f0a6cc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a68152780 con 0x7f0a680cd020 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.321+0000 7f0a6cc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a68070bf0 con 0x7f0a680b2070 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.321+0000 7f0a6cc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a68152c30 con 0x7f0a680cbb30 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.321+0000 7f0a66800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 0x7f0a6815a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.321+0000 7f0a65e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a680b2070 0x7f0a680adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a65400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a680cbb30 0x7f0a68152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a65e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a680b2070 0x7f0a680adc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:52038/0 (socket says 172.21.15.12:52038) 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a66800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 0x7f0a6815a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49202/0 (socket says 172.21.15.12:49202) 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a65e00700 1 -- 172.21.15.12:0/1816184005 learned_addr learned my addr 172.21.15.12:0/1816184005 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a66800700 1 -- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a680cbb30 msgr2=0x7f0a68152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a66800700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a680cbb30 0x7f0a68152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a66800700 1 -- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a680b2070 msgr2=0x7f0a680adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a66800700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a680b2070 0x7f0a680adc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a66800700 1 -- 172.21.15.12:0/1816184005 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a540094a0 con 0x7f0a680cd020 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a65e00700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a680b2070 0x7f0a680adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.322+0000 7f0a66800700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 0x7f0a6815a8e0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000cb90 tx=0x7f0a6000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.323+0000 7f0a5e000700 1 -- 172.21.15.12:0/1816184005 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60003c00 con 0x7f0a680cd020 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.323+0000 7f0a5e000700 1 -- 172.21.15.12:0/1816184005 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a60003d60 con 0x7f0a680cd020 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.323+0000 7f0a6cc00700 1 -- 172.21.15.12:0/1816184005 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a680ac7b0 con 0x7f0a680cd020 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.323+0000 7f0a5e000700 1 -- 172.21.15.12:0/1816184005 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6000aa80 con 0x7f0a680cd020 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.323+0000 7f0a6cc00700 1 -- 172.21.15.12:0/1816184005 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a680ac4d0 con 0x7f0a680cd020 2024-03-29T11:33:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.324+0000 7f0a6cc00700 1 -- 172.21.15.12:0/1816184005 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a680709c0 con 0x7f0a680cd020 2024-03-29T11:33:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.325+0000 7f0a5e000700 1 -- 172.21.15.12:0/1816184005 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f0a6000abe0 con 0x7f0a680cd020 2024-03-29T11:33:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.327+0000 7f0a5e000700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a4806aed0 0x7f0a4806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.327+0000 7f0a5e000700 1 -- 172.21.15.12:0/1816184005 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7f0a60089b30 con 0x7f0a680cd020 2024-03-29T11:33:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.327+0000 7f0a65e00700 1 -- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a4806aed0 msgr2=0x7f0a4806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.327+0000 7f0a65e00700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a4806aed0 0x7f0a4806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.331+0000 7f0a5e000700 1 -- 172.21.15.12:0/1816184005 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0a680709c0 con 0x7f0a680cd020 2024-03-29T11:33:08.337 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:08.341+0000 7f4cdf48f240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:33:08.481 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:08.485+0000 7f8811038240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:33:08.509 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:08.513+0000 7f4cdf48f240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:33:08.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.528+0000 7f0a65e00700 1 -- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a4806aed0 msgr2=0x7f0a4806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:08.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.528+0000 7f0a65e00700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a4806aed0 0x7f0a4806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:08.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.547+0000 7f0a6cc00700 1 -- 172.21.15.12:0/1816184005 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f0a680709c0 con 0x7f0a680cd020 2024-03-29T11:33:08.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.548+0000 7f0a5e000700 1 -- 172.21.15.12:0/1816184005 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v253) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0a680709c0 con 0x7f0a680cd020 2024-03-29T11:33:08.544 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:33:08.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 -- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a4806aed0 msgr2=0x7f0a4806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a4806aed0 0x7f0a4806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 -- 172.21.15.12:0/1816184005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 msgr2=0x7f0a6815a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 0x7f0a6815a8e0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000cb90 tx=0x7f0a6000cea0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 -- 172.21.15.12:0/1816184005 shutdown_connections 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0a4806aed0 0x7f0a4806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a680cd020 0x7f0a6815a8e0 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a680b2070 0x7f0a680adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 --2- 172.21.15.12:0/1816184005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a680cbb30 0x7f0a68152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 -- 172.21.15.12:0/1816184005 >> 172.21.15.12:0/1816184005 conn(0x7f0a680b0d70 msgr2=0x7f0a680cbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 -- 172.21.15.12:0/1816184005 shutdown_connections 2024-03-29T11:33:08.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.550+0000 7f0a6cc00700 1 -- 172.21.15.12:0/1816184005 wait complete. 2024-03-29T11:33:08.565 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T11:33:08.573 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:08.577+0000 7f8811038240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:33:08.671 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:08.675+0000 7f8811038240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:33:08.738 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:08.742+0000 7f4cdf48f240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:33:08.769 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:08.773+0000 7f8811038240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:33:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.774+0000 7f2739800700 1 Processor -- start 2024-03-29T11:33:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.775+0000 7f2739800700 1 -- start start 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.776+0000 7f2739800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f27340cbb30 0x7f27340cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.776+0000 7f2739800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f27340cd480 0x7f273406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.776+0000 7f2739800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27340cc450 con 0x7f27340cd020 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.776+0000 7f2739800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27340cd8f0 con 0x7f27340cbb30 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.776+0000 7f2739800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27340cdae0 con 0x7f27340cd480 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.776+0000 7f2732a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f27340cbb30 0x7f27340cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.776+0000 7f2732a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f27340cbb30 0x7f27340cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49210/0 (socket says 172.21.15.12:49210) 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.776+0000 7f2732a00700 1 -- 172.21.15.12:0/1318335179 learned_addr learned my addr 172.21.15.12:0/1318335179 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.776+0000 7f2733400700 1 --2- 172.21.15.12:0/1318335179 >> v2:172.21.15.159:3300/0 conn(0x7f27340cd480 0x7f273406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.777+0000 7f2732a00700 1 -- 172.21.15.12:0/1318335179 >> v2:172.21.15.159:3300/0 conn(0x7f27340cd480 msgr2=0x7f273406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.777+0000 7f2732a00700 1 --2- 172.21.15.12:0/1318335179 >> v2:172.21.15.159:3300/0 conn(0x7f27340cd480 0x7f273406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.777+0000 7f2732a00700 1 -- 172.21.15.12:0/1318335179 >> v1:172.21.15.159:6789/0 conn(0x7f27340cd020 legacy=0x7f27340b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.777+0000 7f2732a00700 1 -- 172.21.15.12:0/1318335179 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27340cddc0 con 0x7f27340cbb30 2024-03-29T11:33:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.777+0000 7f2732a00700 1 --2- 172.21.15.12:0/1318335179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cbb30 0x7f27340cbf10 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f273406b6d0 tx=0x7f2724009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c18448d0a18563a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.777+0000 7f2731600700 1 -- 172.21.15.12:0/1318335179 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724011830 con 0x7f27340cbb30 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.777+0000 7f2731600700 1 -- 172.21.15.12:0/1318335179 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2724011990 con 0x7f27340cbb30 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.777+0000 7f2731600700 1 -- 172.21.15.12:0/1318335179 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724011d20 con 0x7f27340cbb30 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.778+0000 7f2739800700 1 -- 172.21.15.12:0/1318335179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cbb30 msgr2=0x7f27340cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.778+0000 7f2739800700 1 --2- 172.21.15.12:0/1318335179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cbb30 0x7f27340cbf10 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f273406b6d0 tx=0x7f2724009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.778+0000 7f2739800700 1 -- 172.21.15.12:0/1318335179 shutdown_connections 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.778+0000 7f2739800700 1 --2- 172.21.15.12:0/1318335179 >> v2:172.21.15.159:3300/0 conn(0x7f27340cd480 0x7f273406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.778+0000 7f2739800700 1 --2- 172.21.15.12:0/1318335179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cbb30 0x7f27340cbf10 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.778+0000 7f2739800700 1 -- 172.21.15.12:0/1318335179 >> 172.21.15.12:0/1318335179 conn(0x7f27340b0d70 msgr2=0x7f27340b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.778+0000 7f2739800700 1 -- 172.21.15.12:0/1318335179 shutdown_connections 2024-03-29T11:33:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.778+0000 7f2739800700 1 -- 172.21.15.12:0/1318335179 wait complete. 2024-03-29T11:33:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.780+0000 7f2739800700 1 Processor -- start 2024-03-29T11:33:08.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.780+0000 7f2739800700 1 -- start start 2024-03-29T11:33:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2739800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27340cbb30 0x7f27341540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2739800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27340cd020 0x7f2734154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2739800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cd480 0x7f27341552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2739800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27340b2b10 con 0x7f27340cd480 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2739800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27340b2df0 con 0x7f27340cbb30 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2739800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27340b30d0 con 0x7f27340cd020 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2733400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cd480 0x7f27341552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2732a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27340cbb30 0x7f27341540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2733400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cd480 0x7f27341552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49212/0 (socket says 172.21.15.12:49212) 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2733400700 1 -- 172.21.15.12:0/3006701630 learned_addr learned my addr 172.21.15.12:0/3006701630 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.781+0000 7f2732000700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27340cd020 0x7f2734154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f2733400700 1 -- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27340cd020 msgr2=0x7f2734154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f2733400700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27340cd020 0x7f2734154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f2733400700 1 -- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27340cbb30 msgr2=0x7f27341540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f2733400700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27340cbb30 0x7f27341540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f2733400700 1 -- 172.21.15.12:0/3006701630 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2724009550 con 0x7f27340cd480 2024-03-29T11:33:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f2732a00700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27340cbb30 0x7f27341540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f2733400700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cd480 0x7f27341552e0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f272c00efc0 tx=0x7f272c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f272aa00700 1 -- 172.21.15.12:0/3006701630 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c009ea0 con 0x7f27340cd480 2024-03-29T11:33:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f272aa00700 1 -- 172.21.15.12:0/3006701630 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272c00f040 con 0x7f27340cd480 2024-03-29T11:33:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f272aa00700 1 -- 172.21.15.12:0/3006701630 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c00ced0 con 0x7f27340cd480 2024-03-29T11:33:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f2739800700 1 -- 172.21.15.12:0/3006701630 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27340b33b0 con 0x7f27340cd480 2024-03-29T11:33:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.782+0000 7f2739800700 1 -- 172.21.15.12:0/3006701630 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27341545f0 con 0x7f27340cd480 2024-03-29T11:33:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.783+0000 7f2739800700 1 -- 172.21.15.12:0/3006701630 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27340b35d0 con 0x7f27340cd480 2024-03-29T11:33:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.784+0000 7f272aa00700 1 -- 172.21.15.12:0/3006701630 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f272c014480 con 0x7f27340cd480 2024-03-29T11:33:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.785+0000 7f272aa00700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f271406aed0 0x7f271406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.785+0000 7f272aa00700 1 -- 172.21.15.12:0/3006701630 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7f272c08a6d0 con 0x7f27340cd480 2024-03-29T11:33:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.785+0000 7f2732a00700 1 -- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f271406aed0 msgr2=0x7f271406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.785+0000 7f2732a00700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f271406aed0 0x7f271406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:08.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.794+0000 7f272aa00700 1 -- 172.21.15.12:0/3006701630 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f272c054e40 con 0x7f27340cd480 2024-03-29T11:33:08.843 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:08.847+0000 7f4cdf48f240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:33:08.870 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:08.874+0000 7f8811038240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:33:08.949 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:08.953+0000 7f4cdf48f240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:33:08.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.986+0000 7f2732a00700 1 -- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f271406aed0 msgr2=0x7f271406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:08.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:08.986+0000 7f2732a00700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f271406aed0 0x7f271406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:09.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.034+0000 7f2739800700 1 -- 172.21.15.12:0/3006701630 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f27340b2c50 con 0x7f27340cd480 2024-03-29T11:33:09.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.034+0000 7f272aa00700 1 -- 172.21.15.12:0/3006701630 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v253) v1 ==== 81+0+582 (secure 0 0 0) 0x7f27340b2c50 con 0x7f27340cd480 2024-03-29T11:33:09.030 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2916339927 2024-03-30T11:33:03.088269+0000 2024-03-29T11:33:09.030 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/962755797 2024-03-30T11:33:03.088269+0000 2024-03-29T11:33:09.030 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/62646 2024-03-30T11:33:03.088269+0000 2024-03-29T11:33:09.030 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1966802084 2024-03-30T11:33:03.088269+0000 2024-03-29T11:33:09.031 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/110096794 2024-03-30T11:33:03.088269+0000 2024-03-29T11:33:09.031 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/449562747 2024-03-30T11:33:03.088269+0000 2024-03-29T11:33:09.031 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/180562283 2024-03-30T11:33:03.088269+0000 2024-03-29T11:33:09.031 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2535009377 2024-03-30T11:33:03.088269+0000 2024-03-29T11:33:09.031 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/62646 2024-03-30T11:33:03.088269+0000 2024-03-29T11:33:09.031 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1057841974 2024-03-30T11:33:03.088269+0000 2024-03-29T11:33:09.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.036+0000 7f2739800700 1 -- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f271406aed0 msgr2=0x7f271406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:09.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.036+0000 7f2739800700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f271406aed0 0x7f271406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.036+0000 7f2739800700 1 -- 172.21.15.12:0/3006701630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cd480 msgr2=0x7f27341552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:09.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.036+0000 7f2739800700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cd480 0x7f27341552e0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f272c00efc0 tx=0x7f272c00c3a0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.036+0000 7f2739800700 1 -- 172.21.15.12:0/3006701630 shutdown_connections 2024-03-29T11:33:09.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.037+0000 7f2739800700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f271406aed0 0x7f271406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.037+0000 7f2739800700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27340cd480 0x7f27341552e0 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.037+0000 7f2739800700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27340cbb30 0x7f27341540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.037+0000 7f2739800700 1 --2- 172.21.15.12:0/3006701630 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27340cd020 0x7f2734154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.037+0000 7f2739800700 1 -- 172.21.15.12:0/3006701630 >> 172.21.15.12:0/3006701630 conn(0x7f27340b0d70 msgr2=0x7f273415d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:09.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.037+0000 7f2739800700 1 -- 172.21.15.12:0/3006701630 shutdown_connections 2024-03-29T11:33:09.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.037+0000 7f2739800700 1 -- 172.21.15.12:0/3006701630 wait complete. 2024-03-29T11:33:09.034 INFO:teuthology.orchestra.run.smithi012.stderr:listed 10 entries 2024-03-29T11:33:09.051 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T11:33:09.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:09.051 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T11:33:09.058 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:09.063+0000 7f4cdf48f240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:33:09.085 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:09.089+0000 7f8811038240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:33:09.126 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:33:09.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:09.126 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T11:33:09.166 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:09.171+0000 7f4cdf48f240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:33:09.211 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:33:09.211 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:09.257 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:33:09.258 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:33:09.258 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:09.302 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:33:09.303 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:33:09.303 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:09.352 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:33:09.352 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:33:09.358 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:09.362+0000 7f8811038240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:33:09.395 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:09.399+0000 7f4cdf48f240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:33:09.550 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:09.553+0000 7f8811038240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:33:09.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.562+0000 7f9e03400700 1 Processor -- start 2024-03-29T11:33:09.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.562+0000 7f9e03400700 1 -- start start 2024-03-29T11:33:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.564+0000 7f9e03400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc06a790 0x7f9dfc06ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.564+0000 7f9e03400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc06f850 con 0x7f9dfc0afbe0 2024-03-29T11:33:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.564+0000 7f9e03400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc06e3f0 con 0x7f9dfc0cebd0 2024-03-29T11:33:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.564+0000 7f9e03400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc06b0b0 con 0x7f9dfc06a790 2024-03-29T11:33:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.564+0000 7f9dfbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9dfc0afbe0 0x7f9dfc06dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59432/0 (socket says 172.21.15.12:59432) 2024-03-29T11:33:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.564+0000 7f9dfbe00700 1 -- 172.21.15.12:0/3146455079 learned_addr learned my addr 172.21.15.12:0/3146455079 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.564+0000 7f9e00a00700 1 --2- 172.21.15.12:0/3146455079 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc06a790 0x7f9dfc06ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:09.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.564+0000 7f9dfb400700 1 -- 172.21.15.12:0/3146455079 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583193013 0 0) 0x7f9dfc06f850 con 0x7f9dfc0afbe0 2024-03-29T11:33:09.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.565+0000 7f9dfb400700 1 -- 172.21.15.12:0/3146455079 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de40035b0 con 0x7f9dfc0afbe0 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.565+0000 7f9dfb400700 1 -- 172.21.15.12:0/3146455079 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437196966 0 0) 0x7f9dfc06e3f0 con 0x7f9dfc0cebd0 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.565+0000 7f9dfb400700 1 -- 172.21.15.12:0/3146455079 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dfc06f850 con 0x7f9dfc0cebd0 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.565+0000 7f9e00a00700 1 -- 172.21.15.12:0/3146455079 >> v1:172.21.15.159:6789/0 conn(0x7f9dfc0cebd0 legacy=0x7f9dfc0cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.565+0000 7f9e00a00700 1 -- 172.21.15.12:0/3146455079 >> v1:172.21.15.12:6789/0 conn(0x7f9dfc0afbe0 legacy=0x7f9dfc06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.565+0000 7f9e00a00700 1 -- 172.21.15.12:0/3146455079 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc06fad0 con 0x7f9dfc06a790 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.565+0000 7f9e00a00700 1 --2- 172.21.15.12:0/3146455079 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc06a790 0x7f9dfc06ab70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009870 tx=0x7f9df0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5faa7604e1ac152c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.565+0000 7f9dfb400700 1 -- 172.21.15.12:0/3146455079 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0011830 con 0x7f9dfc06a790 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.565+0000 7f9dfb400700 1 -- 172.21.15.12:0/3146455079 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df0011990 con 0x7f9dfc06a790 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.565+0000 7f9dfb400700 1 -- 172.21.15.12:0/3146455079 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0011d20 con 0x7f9dfc06a790 2024-03-29T11:33:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.566+0000 7f9e03400700 1 -- 172.21.15.12:0/3146455079 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc06a790 msgr2=0x7f9dfc06ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.566+0000 7f9e03400700 1 --2- 172.21.15.12:0/3146455079 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc06a790 0x7f9dfc06ab70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009870 tx=0x7f9df0009b80 comp rx=0 tx=0).stop 2024-03-29T11:33:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.566+0000 7f9e03400700 1 -- 172.21.15.12:0/3146455079 shutdown_connections 2024-03-29T11:33:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.566+0000 7f9e03400700 1 --2- 172.21.15.12:0/3146455079 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc06a790 0x7f9dfc06ab70 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.566+0000 7f9e03400700 1 -- 172.21.15.12:0/3146455079 >> 172.21.15.12:0/3146455079 conn(0x7f9dfc0b5ab0 msgr2=0x7f9dfc0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.566+0000 7f9e03400700 1 -- 172.21.15.12:0/3146455079 shutdown_connections 2024-03-29T11:33:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.566+0000 7f9e03400700 1 -- 172.21.15.12:0/3146455079 wait complete. 2024-03-29T11:33:09.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.568+0000 7f9e03400700 1 Processor -- start 2024-03-29T11:33:09.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.568+0000 7f9e03400700 1 -- start start 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.569+0000 7f9e03400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dfc06a790 0x7f9dfc09e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.569+0000 7f9e03400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc0afbe0 0x7f9dfc0cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.569+0000 7f9e03400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dfc0cebd0 0x7f9dfc0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.569+0000 7f9e03400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc153fc0 con 0x7f9dfc06a790 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.569+0000 7f9e03400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc1474d0 con 0x7f9dfc0cebd0 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.569+0000 7f9e03400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc153440 con 0x7f9dfc0afbe0 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.569+0000 7f9e00a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dfc06a790 0x7f9dfc09e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.569+0000 7f9e00a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dfc06a790 0x7f9dfc09e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54284/0 (socket says 172.21.15.12:54284) 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.569+0000 7f9e00a00700 1 -- 172.21.15.12:0/3163625155 learned_addr learned my addr 172.21.15.12:0/3163625155 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.569+0000 7f9e01400700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dfc0cebd0 0x7f9dfc0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.570+0000 7f9dfbe00700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc0afbe0 0x7f9dfc0cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:09.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.570+0000 7f9e00a00700 1 -- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc0afbe0 msgr2=0x7f9dfc0cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.570+0000 7f9e00a00700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc0afbe0 0x7f9dfc0cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.570+0000 7f9e00a00700 1 -- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dfc0cebd0 msgr2=0x7f9dfc0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.570+0000 7f9e00a00700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dfc0cebd0 0x7f9dfc0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.570+0000 7f9e00a00700 1 -- 172.21.15.12:0/3163625155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df00094a0 con 0x7f9dfc06a790 2024-03-29T11:33:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.570+0000 7f9e00a00700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dfc06a790 0x7f9dfc09e0a0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f9df0014040 tx=0x7f9df0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.571+0000 7f9df8c00700 1 -- 172.21.15.12:0/3163625155 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df001a6e0 con 0x7f9dfc06a790 2024-03-29T11:33:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.571+0000 7f9df8c00700 1 -- 172.21.15.12:0/3163625155 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df001a840 con 0x7f9dfc06a790 2024-03-29T11:33:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.571+0000 7f9df8c00700 1 -- 172.21.15.12:0/3163625155 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df001abb0 con 0x7f9dfc06a790 2024-03-29T11:33:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.571+0000 7f9e03400700 1 -- 172.21.15.12:0/3163625155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc153720 con 0x7f9dfc06a790 2024-03-29T11:33:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.571+0000 7f9e03400700 1 -- 172.21.15.12:0/3163625155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dfc153ce0 con 0x7f9dfc06a790 2024-03-29T11:33:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.572+0000 7f9e03400700 1 -- 172.21.15.12:0/3163625155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dfc067720 con 0x7f9dfc06a790 2024-03-29T11:33:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.574+0000 7f9df8c00700 1 -- 172.21.15.12:0/3163625155 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f9df0022070 con 0x7f9dfc06a790 2024-03-29T11:33:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.575+0000 7f9df8c00700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ddc06aed0 0x7f9ddc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.575+0000 7f9df8c00700 1 -- 172.21.15.12:0/3163625155 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7f9df008a730 con 0x7f9dfc06a790 2024-03-29T11:33:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.575+0000 7f9dfbe00700 1 -- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ddc06aed0 msgr2=0x7f9ddc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.575+0000 7f9dfbe00700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ddc06aed0 0x7f9ddc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:09.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.580+0000 7f9df8c00700 1 -- 172.21.15.12:0/3163625155 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9df00550f0 con 0x7f9dfc06a790 2024-03-29T11:33:09.644 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:09.648+0000 7f8811038240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:09.666 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:09.670+0000 7f4cdf48f240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:33:09.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.776+0000 7f9dfbe00700 1 -- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ddc06aed0 msgr2=0x7f9ddc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:09.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.776+0000 7f9dfbe00700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ddc06aed0 0x7f9ddc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.843+0000 7f9e03400700 1 -- 172.21.15.12:0/3163625155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dfc0cdac0 con 0x7f9dfc06a790 2024-03-29T11:33:09.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.844+0000 7f9df8c00700 1 -- 172.21.15.12:0/3163625155 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5972 (secure 0 0 0) 0x7f9df0015090 con 0x7f9dfc06a790 2024-03-29T11:33:09.840 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:09.840 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":128,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:32.933737+0000","modified":"2024-03-29T11:32:36.420698+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11853":{"gid":11853,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3711953177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3711953177},{"type":"v1","addr":"172.21.15.12:6834","nonce":3711953177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:47.849181+0000","modified":"2024-03-29T11:32:51.340904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11416":{"gid":11416,"name":"d","rank":0,"incarnation":124,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-29T11:33:09.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.846+0000 7f9e03400700 1 -- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ddc06aed0 msgr2=0x7f9ddc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:09.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.846+0000 7f9e03400700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ddc06aed0 0x7f9ddc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.846+0000 7f9e03400700 1 -- 172.21.15.12:0/3163625155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dfc06a790 msgr2=0x7f9dfc09e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:09.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.846+0000 7f9e03400700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dfc06a790 0x7f9dfc09e0a0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f9df0014040 tx=0x7f9df0004750 comp rx=0 tx=0).stop 2024-03-29T11:33:09.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.847+0000 7f9e03400700 1 -- 172.21.15.12:0/3163625155 shutdown_connections 2024-03-29T11:33:09.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.847+0000 7f9e03400700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ddc06aed0 0x7f9ddc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.847+0000 7f9e03400700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dfc0cebd0 0x7f9dfc0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.847+0000 7f9e03400700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dfc06a790 0x7f9dfc09e0a0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.847+0000 7f9e03400700 1 --2- 172.21.15.12:0/3163625155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dfc0afbe0 0x7f9dfc0cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:09.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.847+0000 7f9e03400700 1 -- 172.21.15.12:0/3163625155 >> 172.21.15.12:0/3163625155 conn(0x7f9dfc0b5ab0 msgr2=0x7f9dfc0cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:09.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.847+0000 7f9e03400700 1 -- 172.21.15.12:0/3163625155 shutdown_connections 2024-03-29T11:33:09.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:09.847+0000 7f9e03400700 1 -- 172.21.15.12:0/3163625155 wait complete. 2024-03-29T11:33:09.844 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 128 2024-03-29T11:33:09.862 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:09.871 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:09.876+0000 7f4cdf48f240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:33:09.976 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:09.981+0000 7f4cdf48f240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.076+0000 7fb33a600700 1 Processor -- start 2024-03-29T11:33:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.076+0000 7fb33a600700 1 -- start start 2024-03-29T11:33:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.077+0000 7fb33a600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb334099a60 0x7fb33406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.077+0000 7fb33a600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb334072440 con 0x7fb33415a5a0 2024-03-29T11:33:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.077+0000 7fb33a600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33406f320 con 0x7fb3340cc6f0 2024-03-29T11:33:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.077+0000 7fb33a600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb334099ef0 con 0x7fb334099a60 2024-03-29T11:33:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.077+0000 7fb333400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb33415a5a0 0x7fb33406dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59444/0 (socket says 172.21.15.12:59444) 2024-03-29T11:33:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.077+0000 7fb333400700 1 -- 172.21.15.12:0/3424509019 learned_addr learned my addr 172.21.15.12:0/3424509019 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.078+0000 7fb333e00700 1 --2- 172.21.15.12:0/3424509019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb334099a60 0x7fb33406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.078+0000 7fb332000700 1 -- 172.21.15.12:0/3424509019 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820175738 0 0) 0x7fb334072440 con 0x7fb33415a5a0 2024-03-29T11:33:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.078+0000 7fb332000700 1 -- 172.21.15.12:0/3424509019 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3180035b0 con 0x7fb33415a5a0 2024-03-29T11:33:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.078+0000 7fb332000700 1 -- 172.21.15.12:0/3424509019 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886145092 0 0) 0x7fb33406f320 con 0x7fb3340cc6f0 2024-03-29T11:33:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.078+0000 7fb333e00700 1 -- 172.21.15.12:0/3424509019 >> v1:172.21.15.159:6789/0 conn(0x7fb3340cc6f0 legacy=0x7fb3340ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.078+0000 7fb333e00700 1 -- 172.21.15.12:0/3424509019 >> v1:172.21.15.12:6789/0 conn(0x7fb33415a5a0 legacy=0x7fb33406dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.078+0000 7fb333e00700 1 -- 172.21.15.12:0/3424509019 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33406e9f0 con 0x7fb334099a60 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.078+0000 7fb333e00700 1 --2- 172.21.15.12:0/3424509019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb334099a60 0x7fb33406e2f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009dc0 tx=0x7fb32c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe6261b36072ef75 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.079+0000 7fb332000700 1 -- 172.21.15.12:0/3424509019 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c00e070 con 0x7fb334099a60 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.079+0000 7fb332000700 1 -- 172.21.15.12:0/3424509019 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32c00bb20 con 0x7fb334099a60 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.079+0000 7fb332000700 1 -- 172.21.15.12:0/3424509019 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c00be90 con 0x7fb334099a60 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.079+0000 7fb33a600700 1 -- 172.21.15.12:0/3424509019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb334099a60 msgr2=0x7fb33406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.079+0000 7fb33a600700 1 --2- 172.21.15.12:0/3424509019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb334099a60 0x7fb33406e2f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009dc0 tx=0x7fb32c00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.079+0000 7fb33a600700 1 -- 172.21.15.12:0/3424509019 shutdown_connections 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.079+0000 7fb33a600700 1 --2- 172.21.15.12:0/3424509019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb334099a60 0x7fb33406e2f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.079+0000 7fb33a600700 1 -- 172.21.15.12:0/3424509019 >> 172.21.15.12:0/3424509019 conn(0x7fb3340b5ab0 msgr2=0x7fb3340b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.079+0000 7fb33a600700 1 -- 172.21.15.12:0/3424509019 shutdown_connections 2024-03-29T11:33:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.079+0000 7fb33a600700 1 -- 172.21.15.12:0/3424509019 wait complete. 2024-03-29T11:33:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.081+0000 7fb33a600700 1 Processor -- start 2024-03-29T11:33:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.081+0000 7fb33a600700 1 -- start start 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.082+0000 7fb33a600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb334099a60 0x7fb334151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.082+0000 7fb33a600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3340cc6f0 0x7fb33415b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.082+0000 7fb33a600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33415a5a0 0x7fb3340cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.082+0000 7fb33a600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33415bd40 con 0x7fb334099a60 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.082+0000 7fb33a600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb334157c00 con 0x7fb3340cc6f0 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.082+0000 7fb33a600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb334152160 con 0x7fb33415a5a0 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb333400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb334099a60 0x7fb334151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb332a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3340cc6f0 0x7fb33415b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb333e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33415a5a0 0x7fb3340cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb332a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3340cc6f0 0x7fb33415b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:39284/0 (socket says 172.21.15.12:39284) 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb332a00700 1 -- 172.21.15.12:0/1753027527 learned_addr learned my addr 172.21.15.12:0/1753027527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb333400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb334099a60 0x7fb334151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54298/0 (socket says 172.21.15.12:54298) 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb332a00700 1 -- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33415a5a0 msgr2=0x7fb3340cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb332a00700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33415a5a0 0x7fb3340cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb332a00700 1 -- 172.21.15.12:0/1753027527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb334099a60 msgr2=0x7fb334151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb332a00700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb334099a60 0x7fb334151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb332a00700 1 -- 172.21.15.12:0/1753027527 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c009a70 con 0x7fb3340cc6f0 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb333400700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb334099a60 0x7fb334151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb333e00700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33415a5a0 0x7fb3340cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.083+0000 7fb332a00700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3340cc6f0 0x7fb33415b4e0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fb32000c500 tx=0x7fb32000c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.084+0000 7fb32b400700 1 -- 172.21.15.12:0/1753027527 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3200054e0 con 0x7fb3340cc6f0 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.084+0000 7fb32b400700 1 -- 172.21.15.12:0/1753027527 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb320003680 con 0x7fb3340cc6f0 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.084+0000 7fb32b400700 1 -- 172.21.15.12:0/1753027527 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320003a30 con 0x7fb3340cc6f0 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.084+0000 7fb33a600700 1 -- 172.21.15.12:0/1753027527 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb334157e10 con 0x7fb3340cc6f0 2024-03-29T11:33:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.084+0000 7fb33a600700 1 -- 172.21.15.12:0/1753027527 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3341583d0 con 0x7fb3340cc6f0 2024-03-29T11:33:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.085+0000 7fb33a600700 1 -- 172.21.15.12:0/1753027527 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb334067720 con 0x7fb3340cc6f0 2024-03-29T11:33:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.086+0000 7fb32b400700 1 -- 172.21.15.12:0/1753027527 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fb320003b90 con 0x7fb3340cc6f0 2024-03-29T11:33:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.087+0000 7fb32b400700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31406aed0 0x7fb31406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.087+0000 7fb32b400700 1 -- 172.21.15.12:0/1753027527 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7fb3200890e0 con 0x7fb3340cc6f0 2024-03-29T11:33:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.087+0000 7fb333400700 1 -- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31406aed0 msgr2=0x7fb31406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.087+0000 7fb333400700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31406aed0 0x7fb31406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.091+0000 7fb32b400700 1 -- 172.21.15.12:0/1753027527 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb320053960 con 0x7fb3340cc6f0 2024-03-29T11:33:10.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.288+0000 7fb333400700 1 -- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31406aed0 msgr2=0x7fb31406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:10.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.288+0000 7fb333400700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31406aed0 0x7fb31406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.369+0000 7fb33a600700 1 -- 172.21.15.12:0/1753027527 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3340cf6d0 con 0x7fb3340cc6f0 2024-03-29T11:33:10.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.370+0000 7fb32b400700 1 -- 172.21.15.12:0/1753027527 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+20456 (secure 0 0 0) 0x7fb320017090 con 0x7fb3340cc6f0 2024-03-29T11:33:10.367 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:10.367 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":253,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:03.088324+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:32:45.400233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:32:46.836046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2916339927":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/962755797":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1966802084":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/110096794":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/449562747":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/180562283":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/2535009377":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1057841974":"2024-03-30T11:33:03.088269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:10.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 -- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31406aed0 msgr2=0x7fb31406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31406aed0 0x7fb31406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 -- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3340cc6f0 msgr2=0x7fb33415b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3340cc6f0 0x7fb33415b4e0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fb32000c500 tx=0x7fb32000c810 comp rx=0 tx=0).stop 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 -- 172.21.15.12:0/1753027527 shutdown_connections 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31406aed0 0x7fb31406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33415a5a0 0x7fb3340cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb334099a60 0x7fb334151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 --2- 172.21.15.12:0/1753027527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3340cc6f0 0x7fb33415b4e0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 -- 172.21.15.12:0/1753027527 >> 172.21.15.12:0/1753027527 conn(0x7fb3340b5ab0 msgr2=0x7fb3340affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 -- 172.21.15.12:0/1753027527 shutdown_connections 2024-03-29T11:33:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.373+0000 7fb33a600700 1 -- 172.21.15.12:0/1753027527 wait complete. 2024-03-29T11:33:10.379 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:10.383+0000 7f8811038240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:33:10.388 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:32:31.368651+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:32:31.900686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-29T11:32:45.400233+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-29T11:32:46.836046+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:33:10.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:33:10.590 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:10.594+0000 7f8811038240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:33:10.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.594+0000 7ff7ad400700 1 Processor -- start 2024-03-29T11:33:10.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.595+0000 7ff7ad400700 1 -- start start 2024-03-29T11:33:10.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.596+0000 7ff7ad400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7a815dad0 0x7ff7a80ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.596+0000 7ff7ad400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7a80d21a0 0x7ff7a815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.596+0000 7ff7ad400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a815f820 con 0x7ff7a80d1630 2024-03-29T11:33:10.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.596+0000 7ff7ad400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a80cfcc0 con 0x7ff7a80d21a0 2024-03-29T11:33:10.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.596+0000 7ff7ad400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a80cd9f0 con 0x7ff7a815dad0 2024-03-29T11:33:10.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.596+0000 7ff7a7200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7a80d21a0 0x7ff7a815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:10.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.596+0000 7ff7a7200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7a80d21a0 0x7ff7a815fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54312/0 (socket says 172.21.15.12:54312) 2024-03-29T11:33:10.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.596+0000 7ff7a7200700 1 -- 172.21.15.12:0/4188285638 learned_addr learned my addr 172.21.15.12:0/4188285638 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.596+0000 7ff7a6800700 1 --2- 172.21.15.12:0/4188285638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7a815dad0 0x7ff7a80ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.597+0000 7ff7a7200700 1 -- 172.21.15.12:0/4188285638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7a815dad0 msgr2=0x7ff7a80ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.597+0000 7ff7a7200700 1 --2- 172.21.15.12:0/4188285638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7a815dad0 0x7ff7a80ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.597+0000 7ff7a7200700 1 -- 172.21.15.12:0/4188285638 >> v1:172.21.15.12:6789/0 conn(0x7ff7a80d1630 legacy=0x7ff7a80d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.597+0000 7ff7a7200700 1 -- 172.21.15.12:0/4188285638 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a80cf420 con 0x7ff7a80d21a0 2024-03-29T11:33:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.597+0000 7ff7a7200700 1 --2- 172.21.15.12:0/4188285638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d21a0 0x7ff7a815fcb0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009e50 tx=0x7ff7a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1388c55222825564 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.597+0000 7ff7a5400700 1 -- 172.21.15.12:0/4188285638 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a000e070 con 0x7ff7a80d21a0 2024-03-29T11:33:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.597+0000 7ff7a5400700 1 -- 172.21.15.12:0/4188285638 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a000b910 con 0x7ff7a80d21a0 2024-03-29T11:33:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.597+0000 7ff7a5400700 1 -- 172.21.15.12:0/4188285638 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a000bcc0 con 0x7ff7a80d21a0 2024-03-29T11:33:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.598+0000 7ff7ad400700 1 -- 172.21.15.12:0/4188285638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d21a0 msgr2=0x7ff7a815fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.598+0000 7ff7ad400700 1 --2- 172.21.15.12:0/4188285638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d21a0 0x7ff7a815fcb0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009e50 tx=0x7ff7a000b330 comp rx=0 tx=0).stop 2024-03-29T11:33:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.598+0000 7ff7ad400700 1 -- 172.21.15.12:0/4188285638 shutdown_connections 2024-03-29T11:33:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.598+0000 7ff7ad400700 1 --2- 172.21.15.12:0/4188285638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d21a0 0x7ff7a815fcb0 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.598+0000 7ff7ad400700 1 --2- 172.21.15.12:0/4188285638 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7a815dad0 0x7ff7a80ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.598+0000 7ff7ad400700 1 -- 172.21.15.12:0/4188285638 >> 172.21.15.12:0/4188285638 conn(0x7ff7a808aff0 msgr2=0x7ff7a80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.598+0000 7ff7ad400700 1 -- 172.21.15.12:0/4188285638 shutdown_connections 2024-03-29T11:33:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.598+0000 7ff7ad400700 1 -- 172.21.15.12:0/4188285638 wait complete. 2024-03-29T11:33:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.600+0000 7ff7ad400700 1 Processor -- start 2024-03-29T11:33:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.600+0000 7ff7ad400700 1 -- start start 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7ad400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d1630 0x7ff7a8151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7ad400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7a80d21a0 0x7ff7a80cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7ad400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7a815dad0 0x7ff7a80b8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7ad400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a80cbf90 con 0x7ff7a80d1630 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7ad400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8151fe0 con 0x7ff7a80d21a0 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7ad400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a81e6c70 con 0x7ff7a815dad0 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7a6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d1630 0x7ff7a8151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7a6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d1630 0x7ff7a8151aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54318/0 (socket says 172.21.15.12:54318) 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7a6800700 1 -- 172.21.15.12:0/1844293473 learned_addr learned my addr 172.21.15.12:0/1844293473 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7a5e00700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7a80d21a0 0x7ff7a80cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.601+0000 7ff7a7200700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7a815dad0 0x7ff7a80b8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff7a6800700 1 -- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7a815dad0 msgr2=0x7ff7a80b8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff7a6800700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7a815dad0 0x7ff7a80b8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff7a6800700 1 -- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7a80d21a0 msgr2=0x7ff7a80cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff7a6800700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7a80d21a0 0x7ff7a80cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff7a6800700 1 -- 172.21.15.12:0/1844293473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a0009b30 con 0x7ff7a80d1630 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff7a5e00700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7a80d21a0 0x7ff7a80cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff7a6800700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d1630 0x7ff7a8151aa0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7ff79800e870 tx=0x7ff79800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff79ea00700 1 -- 172.21.15.12:0/1844293473 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff798009590 con 0x7ff7a80d1630 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff79ea00700 1 -- 172.21.15.12:0/1844293473 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff798013680 con 0x7ff7a80d1630 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff79ea00700 1 -- 172.21.15.12:0/1844293473 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7980139f0 con 0x7ff7a80d1630 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff7ad400700 1 -- 172.21.15.12:0/1844293473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a81e6f50 con 0x7ff7a80d1630 2024-03-29T11:33:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.602+0000 7ff7ad400700 1 -- 172.21.15.12:0/1844293473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a80cc8f0 con 0x7ff7a80d1630 2024-03-29T11:33:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.603+0000 7ff7ad400700 1 -- 172.21.15.12:0/1844293473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a80cc200 con 0x7ff7a80d1630 2024-03-29T11:33:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.604+0000 7ff79ea00700 1 -- 172.21.15.12:0/1844293473 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7ff798013b50 con 0x7ff7a80d1630 2024-03-29T11:33:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.605+0000 7ff79ea00700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff78806aed0 0x7ff78806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:10.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.605+0000 7ff79ea00700 1 -- 172.21.15.12:0/1844293473 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7ff798089a40 con 0x7ff7a80d1630 2024-03-29T11:33:10.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.605+0000 7ff7a5e00700 1 -- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff78806aed0 msgr2=0x7ff78806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:10.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.605+0000 7ff7a5e00700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff78806aed0 0x7ff78806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:10.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.609+0000 7ff79ea00700 1 -- 172.21.15.12:0/1844293473 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff79808be60 con 0x7ff7a80d1630 2024-03-29T11:33:10.800 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:10.804+0000 7f8811038240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:10.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.806+0000 7ff7a5e00700 1 -- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff78806aed0 msgr2=0x7ff78806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:10.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.806+0000 7ff7a5e00700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff78806aed0 0x7ff78806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:10.810 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:10.815+0000 7f4cdf48f240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:33:10.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.825+0000 7ff7ad400700 1 -- 172.21.15.12:0/1844293473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff7a8067720 con 0x7ff7a80d1630 2024-03-29T11:33:10.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.826+0000 7ff79ea00700 1 -- 172.21.15.12:0/1844293473 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v253) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff7980543f0 con 0x7ff7a80d1630 2024-03-29T11:33:10.822 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 -- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff78806aed0 msgr2=0x7ff78806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff78806aed0 0x7ff78806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 -- 172.21.15.12:0/1844293473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d1630 msgr2=0x7ff7a8151aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d1630 0x7ff7a8151aa0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7ff79800e870 tx=0x7ff79800eb80 comp rx=0 tx=0).stop 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 -- 172.21.15.12:0/1844293473 shutdown_connections 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff78806aed0 0x7ff78806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7a815dad0 0x7ff7a80b8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7a80d1630 0x7ff7a8151aa0 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 --2- 172.21.15.12:0/1844293473 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7a80d21a0 0x7ff7a80cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 -- 172.21.15.12:0/1844293473 >> 172.21.15.12:0/1844293473 conn(0x7ff7a808aff0 msgr2=0x7ff7a80d1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.828+0000 7ff7ad400700 1 -- 172.21.15.12:0/1844293473 shutdown_connections 2024-03-29T11:33:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:10.829+0000 7ff7ad400700 1 -- 172.21.15.12:0/1844293473 wait complete. 2024-03-29T11:33:10.843 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:33:11.046 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:11.050+0000 7f4cdf48f240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:33:11.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.055+0000 7f1b90400700 1 Processor -- start 2024-03-29T11:33:11.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.055+0000 7f1b90400700 1 -- start start 2024-03-29T11:33:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.056+0000 7f1b90400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b880b2b30 0x7f1b880b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.056+0000 7f1b90400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 0x7f1b880cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.056+0000 7f1b90400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b880b3d90 con 0x7f1b88157f90 2024-03-29T11:33:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b90400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b88158370 con 0x7f1b880b2b30 2024-03-29T11:33:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b90400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880b2710 con 0x7f1b880b3960 2024-03-29T11:33:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b8d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b880b2b30 0x7f1b880b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b8da00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1b88157f90 0x7f1b881613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59448/0 (socket says 172.21.15.12:59448) 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b8da00700 1 -- 172.21.15.12:0/1309981977 learned_addr learned my addr 172.21.15.12:0/1309981977 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b8e400700 1 --2- 172.21.15.12:0/1309981977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 0x7f1b880cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b87e00700 1 -- 172.21.15.12:0/1309981977 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400742967 0 0) 0x7f1b880b3d90 con 0x7f1b88157f90 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b87e00700 1 -- 172.21.15.12:0/1309981977 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b700035b0 con 0x7f1b88157f90 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b8d000700 1 -- 172.21.15.12:0/1309981977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 msgr2=0x7f1b880cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b8d000700 1 --2- 172.21.15.12:0/1309981977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 0x7f1b880cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b8d000700 1 -- 172.21.15.12:0/1309981977 >> v1:172.21.15.12:6789/0 conn(0x7f1b88157f90 legacy=0x7f1b881613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.057+0000 7f1b8d000700 1 -- 172.21.15.12:0/1309981977 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b880b2070 con 0x7f1b880b2b30 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.058+0000 7f1b8d000700 1 --2- 172.21.15.12:0/1309981977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b880b2b30 0x7f1b880b3420 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009840 tx=0x7f1b74009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a31ae325ea3e1c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.058+0000 7f1b87e00700 1 -- 172.21.15.12:0/1309981977 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74011830 con 0x7f1b880b2b30 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.058+0000 7f1b87e00700 1 -- 172.21.15.12:0/1309981977 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b74011990 con 0x7f1b880b2b30 2024-03-29T11:33:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.058+0000 7f1b87e00700 1 -- 172.21.15.12:0/1309981977 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74011d20 con 0x7f1b880b2b30 2024-03-29T11:33:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.059+0000 7f1b90400700 1 -- 172.21.15.12:0/1309981977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b880b2b30 msgr2=0x7f1b880b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.059+0000 7f1b90400700 1 --2- 172.21.15.12:0/1309981977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b880b2b30 0x7f1b880b3420 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009840 tx=0x7f1b74009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.059+0000 7f1b90400700 1 -- 172.21.15.12:0/1309981977 shutdown_connections 2024-03-29T11:33:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.059+0000 7f1b90400700 1 --2- 172.21.15.12:0/1309981977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 0x7f1b880cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.059+0000 7f1b90400700 1 --2- 172.21.15.12:0/1309981977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b880b2b30 0x7f1b880b3420 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.059+0000 7f1b90400700 1 -- 172.21.15.12:0/1309981977 >> 172.21.15.12:0/1309981977 conn(0x7f1b880b0d70 msgr2=0x7f1b880b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.059+0000 7f1b90400700 1 -- 172.21.15.12:0/1309981977 shutdown_connections 2024-03-29T11:33:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.059+0000 7f1b90400700 1 -- 172.21.15.12:0/1309981977 wait complete. 2024-03-29T11:33:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.061+0000 7f1b90400700 1 Processor -- start 2024-03-29T11:33:11.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.061+0000 7f1b90400700 1 -- start start 2024-03-29T11:33:11.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b90400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b880b2b30 0x7f1b881f0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b90400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 0x7f1b88148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b90400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b88157f90 0x7f1b880cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b90400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880cac50 con 0x7f1b88157f90 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b90400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880caf30 con 0x7f1b880b2b30 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b90400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880cb210 con 0x7f1b880b3960 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b8e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b88157f90 0x7f1b880cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b8d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 0x7f1b88148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b8da00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b880b2b30 0x7f1b881f0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b8e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b88157f90 0x7f1b880cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54342/0 (socket says 172.21.15.12:54342) 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.062+0000 7f1b8e400700 1 -- 172.21.15.12:0/1002212965 learned_addr learned my addr 172.21.15.12:0/1002212965 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b8d000700 1 -- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b880b2b30 msgr2=0x7f1b881f0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b8d000700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b880b2b30 0x7f1b881f0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b8d000700 1 -- 172.21.15.12:0/1002212965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b88157f90 msgr2=0x7f1b880cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b8d000700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b88157f90 0x7f1b880cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b8d000700 1 -- 172.21.15.12:0/1002212965 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b740094a0 con 0x7f1b880b3960 2024-03-29T11:33:11.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b8d000700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 0x7f1b88148ef0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1b74016000 tx=0x7f1b74004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:11.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b85600700 1 -- 172.21.15.12:0/1002212965 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74004590 con 0x7f1b880b3960 2024-03-29T11:33:11.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b85600700 1 -- 172.21.15.12:0/1002212965 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7401e070 con 0x7f1b880b3960 2024-03-29T11:33:11.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b85600700 1 -- 172.21.15.12:0/1002212965 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b740038e0 con 0x7f1b880b3960 2024-03-29T11:33:11.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b90400700 1 -- 172.21.15.12:0/1002212965 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b880cb4f0 con 0x7f1b880b3960 2024-03-29T11:33:11.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.063+0000 7f1b90400700 1 -- 172.21.15.12:0/1002212965 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b880cc230 con 0x7f1b880b3960 2024-03-29T11:33:11.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.064+0000 7f1b90400700 1 -- 172.21.15.12:0/1002212965 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b88067720 con 0x7f1b880b3960 2024-03-29T11:33:11.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.065+0000 7f1b85600700 1 -- 172.21.15.12:0/1002212965 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f1b74003a40 con 0x7f1b880b3960 2024-03-29T11:33:11.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.066+0000 7f1b85600700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b6806aed0 0x7f1b6806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.066+0000 7f1b85600700 1 -- 172.21.15.12:0/1002212965 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7f1b7408ac00 con 0x7f1b880b3960 2024-03-29T11:33:11.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.066+0000 7f1b8da00700 1 -- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b6806aed0 msgr2=0x7f1b6806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:11.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.066+0000 7f1b8da00700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b6806aed0 0x7f1b6806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.071+0000 7f1b85600700 1 -- 172.21.15.12:0/1002212965 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b74055490 con 0x7f1b880b3960 2024-03-29T11:33:11.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.267+0000 7f1b8da00700 1 -- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b6806aed0 msgr2=0x7f1b6806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:11.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.267+0000 7f1b8da00700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b6806aed0 0x7f1b6806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:11.269 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:11.274+0000 7f4cdf48f240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:11.274 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:11.278+0000 7f8811038240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:33:11.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.287+0000 7f1b90400700 1 -- 172.21.15.12:0/1002212965 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1b880ce2e0 con 0x7f1b880b3960 2024-03-29T11:33:11.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.288+0000 7f1b85600700 1 -- 172.21.15.12:0/1002212965 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v253) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1b74058ab0 con 0x7f1b880b3960 2024-03-29T11:33:11.284 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:33:11.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 -- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b6806aed0 msgr2=0x7f1b6806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b6806aed0 0x7f1b6806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 -- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 msgr2=0x7f1b88148ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 0x7f1b88148ef0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1b74016000 tx=0x7f1b74004750 comp rx=0 tx=0).stop 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 -- 172.21.15.12:0/1002212965 shutdown_connections 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b6806aed0 0x7f1b6806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b88157f90 0x7f1b880cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b880b2b30 0x7f1b881f0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 --2- 172.21.15.12:0/1002212965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b880b3960 0x7f1b88148ef0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 -- 172.21.15.12:0/1002212965 >> 172.21.15.12:0/1002212965 conn(0x7f1b880b0d70 msgr2=0x7f1b880cdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 -- 172.21.15.12:0/1002212965 shutdown_connections 2024-03-29T11:33:11.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.290+0000 7f1b90400700 1 -- 172.21.15.12:0/1002212965 wait complete. 2024-03-29T11:33:11.305 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:33:11.487 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:11.491+0000 7f8811038240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.515+0000 7f73bfa00700 1 Processor -- start 2024-03-29T11:33:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.515+0000 7f73bfa00700 1 -- start start 2024-03-29T11:33:11.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.516+0000 7f73bfa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b80b0840 0x7f73b815e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.517+0000 7f73bfa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b815ed30 con 0x7f73b806eec0 2024-03-29T11:33:11.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.517+0000 7f73bfa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b809a7a0 con 0x7f73b80cf490 2024-03-29T11:33:11.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.517+0000 7f73bfa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b8099b90 con 0x7f73b80b0840 2024-03-29T11:33:11.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.517+0000 7f73bd000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f73b806eec0 0x7f73b806f2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59452/0 (socket says 172.21.15.12:59452) 2024-03-29T11:33:11.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.517+0000 7f73bd000700 1 -- 172.21.15.12:0/3579432941 learned_addr learned my addr 172.21.15.12:0/3579432941 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:11.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.517+0000 7f73b7e00700 1 --2- 172.21.15.12:0/3579432941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b80b0840 0x7f73b815e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.517+0000 7f73b7400700 1 -- 172.21.15.12:0/3579432941 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620729260 0 0) 0x7f73ac001110 con 0x7f73b806eec0 2024-03-29T11:33:11.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.517+0000 7f73b7400700 1 -- 172.21.15.12:0/3579432941 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a00035b0 con 0x7f73b806eec0 2024-03-29T11:33:11.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.517+0000 7f73b7400700 1 -- 172.21.15.12:0/3579432941 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087727304 0 0) 0x7f73b809a7a0 con 0x7f73b80cf490 2024-03-29T11:33:11.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.517+0000 7f73b7400700 1 -- 172.21.15.12:0/3579432941 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73ac001110 con 0x7f73b80cf490 2024-03-29T11:33:11.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.518+0000 7f73b7e00700 1 -- 172.21.15.12:0/3579432941 >> v1:172.21.15.159:6789/0 conn(0x7f73b80cf490 legacy=0x7f73b80cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.518+0000 7f73b7e00700 1 -- 172.21.15.12:0/3579432941 >> v1:172.21.15.12:6789/0 conn(0x7f73b806eec0 legacy=0x7f73b806f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.518+0000 7f73b7e00700 1 -- 172.21.15.12:0/3579432941 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b8099e70 con 0x7f73b80b0840 2024-03-29T11:33:11.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.518+0000 7f73b7e00700 1 --2- 172.21.15.12:0/3579432941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b80b0840 0x7f73b815e7f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21921f54fa46334e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:11.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.518+0000 7f73b7400700 1 -- 172.21.15.12:0/3579432941 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4011830 con 0x7f73b80b0840 2024-03-29T11:33:11.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.518+0000 7f73b7400700 1 -- 172.21.15.12:0/3579432941 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a4011990 con 0x7f73b80b0840 2024-03-29T11:33:11.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.518+0000 7f73b7400700 1 -- 172.21.15.12:0/3579432941 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4011d20 con 0x7f73b80b0840 2024-03-29T11:33:11.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.519+0000 7f73bfa00700 1 -- 172.21.15.12:0/3579432941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b80b0840 msgr2=0x7f73b815e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.519+0000 7f73bfa00700 1 --2- 172.21.15.12:0/3579432941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b80b0840 0x7f73b815e7f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a4009b80 comp rx=0 tx=0).stop 2024-03-29T11:33:11.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.519+0000 7f73bfa00700 1 -- 172.21.15.12:0/3579432941 shutdown_connections 2024-03-29T11:33:11.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.519+0000 7f73bfa00700 1 --2- 172.21.15.12:0/3579432941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b80b0840 0x7f73b815e7f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.519+0000 7f73bfa00700 1 -- 172.21.15.12:0/3579432941 >> 172.21.15.12:0/3579432941 conn(0x7f73b80d1370 msgr2=0x7f73b80d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:11.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.519+0000 7f73bfa00700 1 -- 172.21.15.12:0/3579432941 shutdown_connections 2024-03-29T11:33:11.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.519+0000 7f73bfa00700 1 -- 172.21.15.12:0/3579432941 wait complete. 2024-03-29T11:33:11.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.521+0000 7f73bfa00700 1 Processor -- start 2024-03-29T11:33:11.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.521+0000 7f73bfa00700 1 -- start start 2024-03-29T11:33:11.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73bfa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b806eec0 0x7f73b80b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73bfa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73b80b0840 0x7f73b80b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73bfa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73b80cf490 0x7f73b80b6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73bfa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b8155020 con 0x7f73b80cf490 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73bfa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b80b4fb0 con 0x7f73b80b0840 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73bfa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b80c5be0 con 0x7f73b806eec0 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73bda00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73b80cf490 0x7f73b80b6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73bd000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b806eec0 0x7f73b80b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73b7e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73b80b0840 0x7f73b80b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73bda00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73b80cf490 0x7f73b80b6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54346/0 (socket says 172.21.15.12:54346) 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.522+0000 7f73bda00700 1 -- 172.21.15.12:0/3193689462 learned_addr learned my addr 172.21.15.12:0/3193689462 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73bda00700 1 -- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b806eec0 msgr2=0x7f73b80b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73bda00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b806eec0 0x7f73b80b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73bda00700 1 -- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73b80b0840 msgr2=0x7f73b80b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73bda00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73b80b0840 0x7f73b80b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73bda00700 1 -- 172.21.15.12:0/3193689462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40094a0 con 0x7f73b80cf490 2024-03-29T11:33:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73b7e00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73b80b0840 0x7f73b80b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:11.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73bda00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73b80cf490 0x7f73b80b6620 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f73b000cc20 tx=0x7f73b000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:11.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73b4c00700 1 -- 172.21.15.12:0/3193689462 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0003c00 con 0x7f73b80cf490 2024-03-29T11:33:11.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73b4c00700 1 -- 172.21.15.12:0/3193689462 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0003d60 con 0x7f73b80cf490 2024-03-29T11:33:11.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73bfa00700 1 -- 172.21.15.12:0/3193689462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b80c4fd0 con 0x7f73b80cf490 2024-03-29T11:33:11.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73b4c00700 1 -- 172.21.15.12:0/3193689462 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b000aae0 con 0x7f73b80cf490 2024-03-29T11:33:11.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.523+0000 7f73bfa00700 1 -- 172.21.15.12:0/3193689462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b80c5590 con 0x7f73b80cf490 2024-03-29T11:33:11.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.524+0000 7f73bfa00700 1 -- 172.21.15.12:0/3193689462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b8067720 con 0x7f73b80cf490 2024-03-29T11:33:11.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.526+0000 7f73b4c00700 1 -- 172.21.15.12:0/3193689462 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f73b000ac40 con 0x7f73b80cf490 2024-03-29T11:33:11.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.527+0000 7f73b4c00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f739806aed0 0x7f739806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.527+0000 7f73b4c00700 1 -- 172.21.15.12:0/3193689462 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7f73b0089d00 con 0x7f73b80cf490 2024-03-29T11:33:11.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.528+0000 7f73bd000700 1 -- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f739806aed0 msgr2=0x7f739806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:11.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.528+0000 7f73bd000700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f739806aed0 0x7f739806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:11.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.532+0000 7f73b4c00700 1 -- 172.21.15.12:0/3193689462 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f73b0054610 con 0x7f73b80cf490 2024-03-29T11:33:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.729+0000 7f73bd000700 1 -- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f739806aed0 msgr2=0x7f739806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.729+0000 7f73bd000700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f739806aed0 0x7f739806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.747+0000 7f73bfa00700 1 -- 172.21.15.12:0/3193689462 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f73b80c4c50 con 0x7f73b80cf490 2024-03-29T11:33:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.747+0000 7f73b4c00700 1 -- 172.21.15.12:0/3193689462 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v253) v1 ==== 100+0+11 (secure 0 0 0) 0x7f73b0057c30 con 0x7f73b80cf490 2024-03-29T11:33:11.743 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:33:11.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 -- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f739806aed0 msgr2=0x7f739806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:11.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f739806aed0 0x7f739806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 -- 172.21.15.12:0/3193689462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73b80cf490 msgr2=0x7f73b80b6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73b80cf490 0x7f73b80b6620 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f73b000cc20 tx=0x7f73b000cf30 comp rx=0 tx=0).stop 2024-03-29T11:33:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 -- 172.21.15.12:0/3193689462 shutdown_connections 2024-03-29T11:33:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f739806aed0 0x7f739806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73b80cf490 0x7f73b80b6620 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f73b806eec0 0x7f73b80b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 --2- 172.21.15.12:0/3193689462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f73b80b0840 0x7f73b80b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 -- 172.21.15.12:0/3193689462 >> 172.21.15.12:0/3193689462 conn(0x7f73b80d1370 msgr2=0x7f73b80b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 -- 172.21.15.12:0/3193689462 shutdown_connections 2024-03-29T11:33:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.750+0000 7f73bfa00700 1 -- 172.21.15.12:0/3193689462 wait complete. 2024-03-29T11:33:11.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-29T11:33:11.794 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:11.798+0000 7f4cdf48f240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:33:11.936 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:11.940+0000 7f8811038240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:33:11.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.986+0000 7f62b8a00700 1 Processor -- start 2024-03-29T11:33:11.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.986+0000 7f62b8a00700 1 -- start start 2024-03-29T11:33:11.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.987+0000 7f62b8a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f62b415dad0 0x7f62b40ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.987+0000 7f62b8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 0x7f62b40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.987+0000 7f62b8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f62b40d1fd0 0x7f62b415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.987+0000 7f62b8a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b40cfcc0 con 0x7f62b40d1fd0 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.987+0000 7f62b8a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b40cd9f0 con 0x7f62b415dad0 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.987+0000 7f62b8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b415f820 con 0x7f62b40d1630 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b2800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f62b40d1fd0 0x7f62b415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b1400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 0x7f62b40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b2800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f62b40d1fd0 0x7f62b415deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54358/0 (socket says 172.21.15.12:54358) 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b2800700 1 -- 172.21.15.12:0/566122701 learned_addr learned my addr 172.21.15.12:0/566122701 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b1e00700 1 --2- 172.21.15.12:0/566122701 >> v2:172.21.15.159:3300/0 conn(0x7f62b415dad0 0x7f62b40ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b2800700 1 -- 172.21.15.12:0/566122701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 msgr2=0x7f62b40d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b2800700 1 --2- 172.21.15.12:0/566122701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 0x7f62b40d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b2800700 1 -- 172.21.15.12:0/566122701 >> v2:172.21.15.159:3300/0 conn(0x7f62b415dad0 msgr2=0x7f62b40ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b2800700 1 --2- 172.21.15.12:0/566122701 >> v2:172.21.15.159:3300/0 conn(0x7f62b415dad0 0x7f62b40ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b2800700 1 -- 172.21.15.12:0/566122701 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b40d2460 con 0x7f62b40d1fd0 2024-03-29T11:33:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.988+0000 7f62b2800700 1 --2- 172.21.15.12:0/566122701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b40d1fd0 0x7f62b415deb0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f62b408d380 tx=0x7f62ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5230c19bb52be45a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b0a00700 1 -- 172.21.15.12:0/566122701 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac00e070 con 0x7f62b40d1fd0 2024-03-29T11:33:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b0a00700 1 -- 172.21.15.12:0/566122701 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ac00b910 con 0x7f62b40d1fd0 2024-03-29T11:33:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b0a00700 1 -- 172.21.15.12:0/566122701 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac00bcc0 con 0x7f62b40d1fd0 2024-03-29T11:33:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b8a00700 1 -- 172.21.15.12:0/566122701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b40d1fd0 msgr2=0x7f62b415deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b8a00700 1 --2- 172.21.15.12:0/566122701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b40d1fd0 0x7f62b415deb0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f62b408d380 tx=0x7f62ac00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b8a00700 1 -- 172.21.15.12:0/566122701 shutdown_connections 2024-03-29T11:33:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b8a00700 1 --2- 172.21.15.12:0/566122701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b40d1fd0 0x7f62b415deb0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b8a00700 1 --2- 172.21.15.12:0/566122701 >> v2:172.21.15.159:3300/0 conn(0x7f62b415dad0 0x7f62b40ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b8a00700 1 --2- 172.21.15.12:0/566122701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 0x7f62b40d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b8a00700 1 -- 172.21.15.12:0/566122701 >> 172.21.15.12:0/566122701 conn(0x7f62b408aff0 msgr2=0x7f62b40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b8a00700 1 -- 172.21.15.12:0/566122701 shutdown_connections 2024-03-29T11:33:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.989+0000 7f62b8a00700 1 -- 172.21.15.12:0/566122701 wait complete. 2024-03-29T11:33:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.991+0000 7f62b8a00700 1 Processor -- start 2024-03-29T11:33:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.991+0000 7f62b8a00700 1 -- start start 2024-03-29T11:33:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.992+0000 7f62b8a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 0x7f62b4154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.992+0000 7f62b8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62b40d1fd0 0x7f62b40b8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.992+0000 7f62b8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b415dad0 0x7f62b40c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.992+0000 7f62b8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b41569e0 con 0x7f62b415dad0 2024-03-29T11:33:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.992+0000 7f62b8a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b40c7fb0 con 0x7f62b40d1fd0 2024-03-29T11:33:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.992+0000 7f62b8a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4156bf0 con 0x7f62b40d1630 2024-03-29T11:33:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 0x7f62b4154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b415dad0 0x7f62b40c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b1e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 0x7f62b4154f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:53490/0 (socket says 172.21.15.12:53490) 2024-03-29T11:33:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b1e00700 1 -- 172.21.15.12:0/2342001784 learned_addr learned my addr 172.21.15.12:0/2342001784 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b1400700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62b40d1fd0 0x7f62b40b8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b1e00700 1 -- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62b40d1fd0 msgr2=0x7f62b40b8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b1e00700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62b40d1fd0 0x7f62b40b8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b1e00700 1 -- 172.21.15.12:0/2342001784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b415dad0 msgr2=0x7f62b40c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b1e00700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b415dad0 0x7f62b40c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b1e00700 1 -- 172.21.15.12:0/2342001784 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac009b30 con 0x7f62b40d1630 2024-03-29T11:33:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b2800700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b415dad0 0x7f62b40c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.993+0000 7f62b1e00700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 0x7f62b4154f10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f62a400e870 tx=0x7f62a400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.994+0000 7f62aa000700 1 -- 172.21.15.12:0/2342001784 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a4009590 con 0x7f62b40d1630 2024-03-29T11:33:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.994+0000 7f62aa000700 1 -- 172.21.15.12:0/2342001784 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a4013680 con 0x7f62b40d1630 2024-03-29T11:33:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.994+0000 7f62aa000700 1 -- 172.21.15.12:0/2342001784 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a40139f0 con 0x7f62b40d1630 2024-03-29T11:33:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.994+0000 7f62b8a00700 1 -- 172.21.15.12:0/2342001784 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b4156ed0 con 0x7f62b40d1630 2024-03-29T11:33:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.994+0000 7f62b8a00700 1 -- 172.21.15.12:0/2342001784 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b4157490 con 0x7f62b40d1630 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.995+0000 7f62b8a00700 1 -- 172.21.15.12:0/2342001784 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b4067720 con 0x7f62b40d1630 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.996+0000 7f62aa000700 1 -- 172.21.15.12:0/2342001784 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 143) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f62a4013b50 con 0x7f62b40d1630 2024-03-29T11:33:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.997+0000 7f62aa000700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f629406aed0 0x7f629406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.997+0000 7f62aa000700 1 -- 172.21.15.12:0/2342001784 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7075+0+0 (secure 0 0 0) 0x7f62a4089ba0 con 0x7f62b40d1630 2024-03-29T11:33:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.997+0000 7f62b1400700 1 -- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f629406aed0 msgr2=0x7f629406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:11.997+0000 7f62b1400700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f629406aed0 0x7f629406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:11.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.001+0000 7f62aa000700 1 -- 172.21.15.12:0/2342001784 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f62a40544b0 con 0x7f62b40d1630 2024-03-29T11:33:12.041 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:12.045+0000 7f4cdf48f240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:12.051 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.055+0000 7f8811038240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:12.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.067+0000 7f62aa000700 1 -- 172.21.15.12:0/2342001784 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mgrmap(e 144) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f62a4054880 con 0x7f62b40d1630 2024-03-29T11:33:12.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.067+0000 7f62aa000700 1 -- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f629406aed0 msgr2=0x7f629406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:12.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.067+0000 7f62aa000700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f629406aed0 0x7f629406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.139 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.142+0000 7f877fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.139 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.142+0000 7f877fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.139 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.142+0000 7f877fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.139 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.142+0000 7f877fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.139 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.142+0000 7f877fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.139 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.143+0000 7f8783a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.139 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.143+0000 7f8783a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.139 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.143+0000 7f8783a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.140 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.143+0000 7f8783a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.140 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.143+0000 7f8783a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.152 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.156+0000 7f877fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.152 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.156+0000 7f877fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.152 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.156+0000 7f877fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.152 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.156+0000 7f877fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.152 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.156+0000 7f877fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.153 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.157+0000 7f8783a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.153 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.157+0000 7f8783a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.154 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.157+0000 7f8783a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.154 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.157+0000 7f8783a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.154 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:12.157+0000 7f8783a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:12.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.238+0000 7f62b8a00700 1 -- 172.21.15.12:0/2342001784 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f62b40b6130 con 0x7f62b40d1630 2024-03-29T11:33:12.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.238+0000 7f62aa000700 1 -- 172.21.15.12:0/2342001784 <== mon.2 v2:172.21.15.159:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v254) v1 ==== 107+0+11 (secure 0 0 0) 0x7f62a4057f40 con 0x7f62b40d1630 2024-03-29T11:33:12.234 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:33:12.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.241+0000 7f62b8a00700 1 -- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 msgr2=0x7f62b4154f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.241+0000 7f62b8a00700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 0x7f62b4154f10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f62a400e870 tx=0x7f62a400eb80 comp rx=0 tx=0).stop 2024-03-29T11:33:12.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.241+0000 7f62b8a00700 1 -- 172.21.15.12:0/2342001784 shutdown_connections 2024-03-29T11:33:12.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.241+0000 7f62b8a00700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f629406aed0 0x7f629406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.241+0000 7f62b8a00700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b415dad0 0x7f62b40c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.241+0000 7f62b8a00700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62b40d1630 0x7f62b4154f10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.241+0000 7f62b8a00700 1 --2- 172.21.15.12:0/2342001784 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62b40d1fd0 0x7f62b40b8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.241+0000 7f62b8a00700 1 -- 172.21.15.12:0/2342001784 >> 172.21.15.12:0/2342001784 conn(0x7f62b408aff0 msgr2=0x7f62b415caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:12.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.241+0000 7f62b8a00700 1 -- 172.21.15.12:0/2342001784 shutdown_connections 2024-03-29T11:33:12.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.241+0000 7f62b8a00700 1 -- 172.21.15.12:0/2342001784 wait complete. 2024-03-29T11:33:12.257 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-29T11:33:12.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.514+0000 7fe311200700 1 Processor -- start 2024-03-29T11:33:12.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.514+0000 7fe311200700 1 -- start start 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.516+0000 7fe311200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe30c15d800 0x7fe30c0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.516+0000 7fe311200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 0x7fe30c15dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.516+0000 7fe311200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30c0709c0 con 0x7fe30c0ce230 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.516+0000 7fe311200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe30c0b40f0 con 0x7fe30c15d800 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.516+0000 7fe311200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c070c80 con 0x7fe30c0ce6a0 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.516+0000 7fe309e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe30c15d800 0x7fe30c0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.516+0000 7fe309e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe30c15d800 0x7fe30c0725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54366/0 (socket says 172.21.15.12:54366) 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.516+0000 7fe309e00700 1 -- 172.21.15.12:0/954724651 learned_addr learned my addr 172.21.15.12:0/954724651 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.516+0000 7fe309400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe30c0ce230 0x7fe30c0b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59468/0 (socket says 172.21.15.12:59468) 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.517+0000 7fe30a800700 1 --2- 172.21.15.12:0/954724651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 0x7fe30c15dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.517+0000 7fe309e00700 1 -- 172.21.15.12:0/954724651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 msgr2=0x7fe30c15dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:12.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.517+0000 7fe309e00700 1 --2- 172.21.15.12:0/954724651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 0x7fe30c15dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.517+0000 7fe309e00700 1 -- 172.21.15.12:0/954724651 >> v1:172.21.15.12:6789/0 conn(0x7fe30c0ce230 legacy=0x7fe30c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.517+0000 7fe309e00700 1 -- 172.21.15.12:0/954724651 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c070f60 con 0x7fe30c15d800 2024-03-29T11:33:12.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.517+0000 7fe308a00700 1 -- 172.21.15.12:0/954724651 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255860513 0 0) 0x7fe30c0709c0 con 0x7fe30c0ce230 2024-03-29T11:33:12.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.517+0000 7fe309e00700 1 --2- 172.21.15.12:0/954724651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 0x7fe30c0725c0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0098f0 tx=0x7fe2fc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef3466bcf5dfd65 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:12.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.517+0000 7fe308a00700 1 -- 172.21.15.12:0/954724651 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc00f8a0 con 0x7fe30c15d800 2024-03-29T11:33:12.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.518+0000 7fe308a00700 1 -- 172.21.15.12:0/954724651 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc00fa00 con 0x7fe30c15d800 2024-03-29T11:33:12.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.518+0000 7fe308a00700 1 -- 172.21.15.12:0/954724651 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc00fd90 con 0x7fe30c15d800 2024-03-29T11:33:12.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.518+0000 7fe311200700 1 -- 172.21.15.12:0/954724651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 msgr2=0x7fe30c0725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.518+0000 7fe311200700 1 --2- 172.21.15.12:0/954724651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 0x7fe30c0725c0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0098f0 tx=0x7fe2fc009c00 comp rx=0 tx=0).stop 2024-03-29T11:33:12.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.518+0000 7fe311200700 1 -- 172.21.15.12:0/954724651 shutdown_connections 2024-03-29T11:33:12.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.518+0000 7fe311200700 1 --2- 172.21.15.12:0/954724651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 0x7fe30c15dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.518+0000 7fe311200700 1 --2- 172.21.15.12:0/954724651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 0x7fe30c0725c0 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.518+0000 7fe311200700 1 -- 172.21.15.12:0/954724651 >> 172.21.15.12:0/954724651 conn(0x7fe30c0b0d70 msgr2=0x7fe30c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:12.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.518+0000 7fe311200700 1 -- 172.21.15.12:0/954724651 shutdown_connections 2024-03-29T11:33:12.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.518+0000 7fe311200700 1 -- 172.21.15.12:0/954724651 wait complete. 2024-03-29T11:33:12.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.520+0000 7fe311200700 1 Processor -- start 2024-03-29T11:33:12.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.520+0000 7fe311200700 1 -- start start 2024-03-29T11:33:12.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.521+0000 7fe311200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe30c0ce230 0x7fe30c159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:12.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.521+0000 7fe311200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 0x7fe30c0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:12.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.521+0000 7fe311200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 0x7fe30c13e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:12.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.521+0000 7fe311200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c140590 con 0x7fe30c15d800 2024-03-29T11:33:12.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.521+0000 7fe311200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c15a150 con 0x7fe30c0ce230 2024-03-29T11:33:12.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.521+0000 7fe311200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c15a430 con 0x7fe30c0ce6a0 2024-03-29T11:33:12.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.521+0000 7fe30a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 0x7fe30c13e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:12.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.521+0000 7fe30a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 0x7fe30c13e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54368/0 (socket says 172.21.15.12:54368) 2024-03-29T11:33:12.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.521+0000 7fe30a800700 1 -- 172.21.15.12:0/3754593939 learned_addr learned my addr 172.21.15.12:0/3754593939 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:12.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.522+0000 7fe309e00700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe30c0ce230 0x7fe30c159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:12.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.522+0000 7fe309400700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 0x7fe30c0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.522+0000 7fe30a800700 1 -- 172.21.15.12:0/3754593939 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 msgr2=0x7fe30c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.522+0000 7fe30a800700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 0x7fe30c0aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.522+0000 7fe30a800700 1 -- 172.21.15.12:0/3754593939 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe30c0ce230 msgr2=0x7fe30c159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.522+0000 7fe30a800700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe30c0ce230 0x7fe30c159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.522+0000 7fe30a800700 1 -- 172.21.15.12:0/3754593939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc009550 con 0x7fe30c15d800 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.522+0000 7fe309e00700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe30c0ce230 0x7fe30c159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.522+0000 7fe309400700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 0x7fe30c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.522+0000 7fe30a800700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 0x7fe30c13e3d0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fe30400efc0 tx=0x7fe30400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.523+0000 7fe302000700 1 -- 172.21.15.12:0/3754593939 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe304009ea0 con 0x7fe30c15d800 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.523+0000 7fe302000700 1 -- 172.21.15.12:0/3754593939 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30400f040 con 0x7fe30c15d800 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.523+0000 7fe302000700 1 -- 172.21.15.12:0/3754593939 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30400ced0 con 0x7fe30c15d800 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.523+0000 7fe311200700 1 -- 172.21.15.12:0/3754593939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30c15a710 con 0x7fe30c15d800 2024-03-29T11:33:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.523+0000 7fe311200700 1 -- 172.21.15.12:0/3754593939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30c0b04a0 con 0x7fe30c15d800 2024-03-29T11:33:12.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.524+0000 7fe311200700 1 -- 172.21.15.12:0/3754593939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30c140700 con 0x7fe30c15d800 2024-03-29T11:33:12.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.525+0000 7fe302000700 1 -- 172.21.15.12:0/3754593939 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 144) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fe304014480 con 0x7fe30c15d800 2024-03-29T11:33:12.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.526+0000 7fe302000700 1 -- 172.21.15.12:0/3754593939 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7419+0+0 (secure 0 0 0) 0x7fe304089f80 con 0x7fe30c15d800 2024-03-29T11:33:12.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.530+0000 7fe302000700 1 -- 172.21.15.12:0/3754593939 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe304054f00 con 0x7fe30c15d800 2024-03-29T11:33:12.528 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:12.532+0000 7f4cdf48f240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:33:12.664 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:12.668+0000 7f4cdf48f240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:12.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.763+0000 7fe311200700 1 -- 172.21.15.12:0/3754593939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fe30c067720 con 0x7fe30c15d800 2024-03-29T11:33:12.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.764+0000 7fe302000700 1 -- 172.21.15.12:0/3754593939 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v254) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe304018020 con 0x7fe30c15d800 2024-03-29T11:33:12.760 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:33:12.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.766+0000 7fe311200700 1 -- 172.21.15.12:0/3754593939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 msgr2=0x7fe30c13e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.766+0000 7fe311200700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 0x7fe30c13e3d0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fe30400efc0 tx=0x7fe30400c3a0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.766+0000 7fe311200700 1 -- 172.21.15.12:0/3754593939 shutdown_connections 2024-03-29T11:33:12.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.766+0000 7fe311200700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30c15d800 0x7fe30c13e3d0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.766+0000 7fe311200700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe30c0ce230 0x7fe30c159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.766+0000 7fe311200700 1 --2- 172.21.15.12:0/3754593939 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe30c0ce6a0 0x7fe30c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.766+0000 7fe311200700 1 -- 172.21.15.12:0/3754593939 >> 172.21.15.12:0/3754593939 conn(0x7fe30c0b0d70 msgr2=0x7fe30c0ac050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:12.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.766+0000 7fe311200700 1 -- 172.21.15.12:0/3754593939 shutdown_connections 2024-03-29T11:33:12.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.767+0000 7fe311200700 1 -- 172.21.15.12:0/3754593939 wait complete. 2024-03-29T11:33:12.782 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:33:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.989+0000 7f0dc4200700 1 Processor -- start 2024-03-29T11:33:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.989+0000 7f0dc4200700 1 -- start start 2024-03-29T11:33:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.990+0000 7f0dc4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dbc15d710 0x7f0dbc0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.990+0000 7f0dc4200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0dbc06c650 0x7f0dbc0d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.990+0000 7f0dc4200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc06bb40 con 0x7f0dbc09aba0 2024-03-29T11:33:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.990+0000 7f0dc4200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dbc099b00 con 0x7f0dbc06c650 2024-03-29T11:33:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.990+0000 7f0dc4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc099de0 con 0x7f0dbc15d710 2024-03-29T11:33:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.990+0000 7f0dc2200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0dbc06c650 0x7f0dbc0d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.990+0000 7f0dc2200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0dbc06c650 0x7f0dbc0d0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54380/0 (socket says 172.21.15.12:54380) 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.990+0000 7f0dc2200700 1 -- 172.21.15.12:0/3949034580 learned_addr learned my addr 172.21.15.12:0/3949034580 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.990+0000 7f0dc1800700 1 --2- 172.21.15.12:0/3949034580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dbc15d710 0x7f0dbc0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.991+0000 7f0dc2200700 1 -- 172.21.15.12:0/3949034580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dbc15d710 msgr2=0x7f0dbc0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.991+0000 7f0dc2200700 1 --2- 172.21.15.12:0/3949034580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dbc15d710 0x7f0dbc0d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.991+0000 7f0dc2200700 1 -- 172.21.15.12:0/3949034580 >> v1:172.21.15.159:6789/0 conn(0x7f0dbc09aba0 legacy=0x7f0dbc06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.991+0000 7f0dc2200700 1 -- 172.21.15.12:0/3949034580 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc09a0c0 con 0x7f0dbc06c650 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.991+0000 7f0dc2200700 1 --2- 172.21.15.12:0/3949034580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 0x7f0dbc0d0670 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009e50 tx=0x7f0db400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d841d60833dcca server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.991+0000 7f0dbbe00700 1 -- 172.21.15.12:0/3949034580 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db400e070 con 0x7f0dbc06c650 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.991+0000 7f0dbbe00700 1 -- 172.21.15.12:0/3949034580 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db400b910 con 0x7f0dbc06c650 2024-03-29T11:33:12.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.991+0000 7f0dbbe00700 1 -- 172.21.15.12:0/3949034580 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db400bcc0 con 0x7f0dbc06c650 2024-03-29T11:33:12.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.992+0000 7f0dc4200700 1 -- 172.21.15.12:0/3949034580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 msgr2=0x7f0dbc0d0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.992+0000 7f0dc4200700 1 --2- 172.21.15.12:0/3949034580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 0x7f0dbc0d0670 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009e50 tx=0x7f0db400b330 comp rx=0 tx=0).stop 2024-03-29T11:33:12.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.992+0000 7f0dc4200700 1 -- 172.21.15.12:0/3949034580 shutdown_connections 2024-03-29T11:33:12.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.992+0000 7f0dc4200700 1 --2- 172.21.15.12:0/3949034580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 0x7f0dbc0d0670 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.992+0000 7f0dc4200700 1 --2- 172.21.15.12:0/3949034580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dbc15d710 0x7f0dbc0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.992+0000 7f0dc4200700 1 -- 172.21.15.12:0/3949034580 >> 172.21.15.12:0/3949034580 conn(0x7f0dbc0d13e0 msgr2=0x7f0dbc0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:12.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.992+0000 7f0dc4200700 1 -- 172.21.15.12:0/3949034580 shutdown_connections 2024-03-29T11:33:12.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.992+0000 7f0dc4200700 1 -- 172.21.15.12:0/3949034580 wait complete. 2024-03-29T11:33:12.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.994+0000 7f0dc4200700 1 Processor -- start 2024-03-29T11:33:12.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.994+0000 7f0dc4200700 1 -- start start 2024-03-29T11:33:12.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc4200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 0x7f0dbc0c6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:12.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dbc09aba0 0x7f0dbc0b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:12.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc4200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dbc15d710 0x7f0dbc070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:12.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc4200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc0b6f20 con 0x7f0dbc06c650 2024-03-29T11:33:12.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc4200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc15aa30 con 0x7f0dbc15d710 2024-03-29T11:33:12.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc0b3fd0 con 0x7f0dbc09aba0 2024-03-29T11:33:12.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 0x7f0dbc0c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:12.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 0x7f0dbc0c6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54396/0 (socket says 172.21.15.12:54396) 2024-03-29T11:33:12.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc1800700 1 -- 172.21.15.12:0/1388553885 learned_addr learned my addr 172.21.15.12:0/1388553885 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:12.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc2200700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dbc15d710 0x7f0dbc070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc0e00700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dbc09aba0 0x7f0dbc0b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc1800700 1 -- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dbc09aba0 msgr2=0x7f0dbc0b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc1800700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dbc09aba0 0x7f0dbc0b69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc1800700 1 -- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dbc15d710 msgr2=0x7f0dbc070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc1800700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dbc15d710 0x7f0dbc070150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.995+0000 7f0dc1800700 1 -- 172.21.15.12:0/1388553885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db4009b30 con 0x7f0dbc06c650 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.996+0000 7f0dc1800700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 0x7f0dbc0c6330 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f0db000e8a0 tx=0x7f0db000ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.996+0000 7f0db9600700 1 -- 172.21.15.12:0/1388553885 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0009590 con 0x7f0dbc06c650 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.996+0000 7f0db9600700 1 -- 172.21.15.12:0/1388553885 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db0013680 con 0x7f0dbc06c650 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.996+0000 7f0dc4200700 1 -- 172.21.15.12:0/1388553885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc1f6ea0 con 0x7f0dbc06c650 2024-03-29T11:33:12.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.996+0000 7f0db9600700 1 -- 172.21.15.12:0/1388553885 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db00139f0 con 0x7f0dbc06c650 2024-03-29T11:33:12.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.996+0000 7f0dc4200700 1 -- 172.21.15.12:0/1388553885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc1f7460 con 0x7f0dbc06c650 2024-03-29T11:33:12.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.997+0000 7f0dc4200700 1 -- 172.21.15.12:0/1388553885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dbc0b42b0 con 0x7f0dbc06c650 2024-03-29T11:33:12.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:12.998+0000 7f0db9600700 1 -- 172.21.15.12:0/1388553885 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 144) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f0db0013b50 con 0x7f0dbc06c650 2024-03-29T11:33:12.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.000+0000 7f0db9600700 1 -- 172.21.15.12:0/1388553885 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7419+0+0 (secure 0 0 0) 0x7f0db0089400 con 0x7f0dbc06c650 2024-03-29T11:33:13.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.004+0000 7f0db9600700 1 -- 172.21.15.12:0/1388553885 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0db0054290 con 0x7f0dbc06c650 2024-03-29T11:33:13.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.069+0000 7f0db9600700 1 -- 172.21.15.12:0/1388553885 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 145) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f0db0053d40 con 0x7f0dbc06c650 2024-03-29T11:33:13.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.069+0000 7f0db9600700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0d9c036150 0x7f0d9c074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.071+0000 7f0dc0e00700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0d9c036150 0x7f0d9c074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.075+0000 7f0dc0e00700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0d9c036150 0x7f0d9c074c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc15a190 tx=0x7f0da8009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.270+0000 7f0dc4200700 1 -- 172.21.15.12:0/1388553885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dbc067720 con 0x7f0dbc06c650 2024-03-29T11:33:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.271+0000 7f0db9600700 1 -- 172.21.15.12:0/1388553885 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5972 (secure 0 0 0) 0x7f0db0087230 con 0x7f0dbc06c650 2024-03-29T11:33:13.267 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:13.267 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":128,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:32.933737+0000","modified":"2024-03-29T11:32:36.420698+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11853":{"gid":11853,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3711953177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3711953177},{"type":"v1","addr":"172.21.15.12:6834","nonce":3711953177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:47.849181+0000","modified":"2024-03-29T11:32:51.340904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11416":{"gid":11416,"name":"d","rank":0,"incarnation":124,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-29T11:33:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 -- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0d9c036150 msgr2=0x7f0d9c074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0d9c036150 0x7f0d9c074c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc15a190 tx=0x7f0da8009040 comp rx=0 tx=0).stop 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 -- 172.21.15.12:0/1388553885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 msgr2=0x7f0dbc0c6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 0x7f0dbc0c6330 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f0db000e8a0 tx=0x7f0db000ebb0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 -- 172.21.15.12:0/1388553885 shutdown_connections 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0d9c036150 0x7f0d9c074c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dbc15d710 0x7f0dbc070150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dbc06c650 0x7f0dbc0c6330 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 --2- 172.21.15.12:0/1388553885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dbc09aba0 0x7f0dbc0b69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 -- 172.21.15.12:0/1388553885 >> 172.21.15.12:0/1388553885 conn(0x7f0dbc0d13e0 msgr2=0x7f0dbc0ccc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 -- 172.21.15.12:0/1388553885 shutdown_connections 2024-03-29T11:33:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.273+0000 7f0dc4200700 1 -- 172.21.15.12:0/1388553885 wait complete. 2024-03-29T11:33:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 128 2024-03-29T11:33:13.287 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:13.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.490+0000 7fd0f0600700 1 Processor -- start 2024-03-29T11:33:13.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.490+0000 7fd0f0600700 1 -- start start 2024-03-29T11:33:13.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.491+0000 7fd0f0600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd0e8160f10 0x7fd0e80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.491+0000 7fd0f0600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e80cf230 0x7fd0e80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.491+0000 7fd0f0600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e80ceda0 con 0x7fd0e8161590 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.491+0000 7fd0f0600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e80d23d0 con 0x7fd0e8160f10 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.491+0000 7fd0f0600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e81612f0 con 0x7fd0e80cf230 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0ee600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd0e8161590 0x7fd0e80b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59484/0 (socket says 172.21.15.12:59484) 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0ee600700 1 -- 172.21.15.12:0/3813054967 learned_addr learned my addr 172.21.15.12:0/3813054967 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0edc00700 1 --2- 172.21.15.12:0/3813054967 >> v2:172.21.15.159:3300/0 conn(0x7fd0e8160f10 0x7fd0e80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0ed200700 1 --2- 172.21.15.12:0/3813054967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e80cf230 0x7fd0e80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0e7e00700 1 -- 172.21.15.12:0/3813054967 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521509736 0 0) 0x7fd0e80ceda0 con 0x7fd0e8161590 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0e7e00700 1 -- 172.21.15.12:0/3813054967 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d00035b0 con 0x7fd0e8161590 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0edc00700 1 -- 172.21.15.12:0/3813054967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e80cf230 msgr2=0x7fd0e80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0edc00700 1 --2- 172.21.15.12:0/3813054967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e80cf230 0x7fd0e80d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0edc00700 1 -- 172.21.15.12:0/3813054967 >> v1:172.21.15.12:6789/0 conn(0x7fd0e8161590 legacy=0x7fd0e80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0edc00700 1 -- 172.21.15.12:0/3813054967 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e80ce470 con 0x7fd0e8160f10 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.492+0000 7fd0ed200700 1 --2- 172.21.15.12:0/3813054967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e80cf230 0x7fd0e80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0edc00700 1 --2- 172.21.15.12:0/3813054967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0e8160f10 0x7fd0e80d18b0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009840 tx=0x7fd0dc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed9712b17c4001a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:13.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0e7e00700 1 -- 172.21.15.12:0/3813054967 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc011830 con 0x7fd0e8160f10 2024-03-29T11:33:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0e7e00700 1 -- 172.21.15.12:0/3813054967 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc011990 con 0x7fd0e8160f10 2024-03-29T11:33:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0e7e00700 1 -- 172.21.15.12:0/3813054967 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc011d20 con 0x7fd0e8160f10 2024-03-29T11:33:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0f0600700 1 -- 172.21.15.12:0/3813054967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0e8160f10 msgr2=0x7fd0e80d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0f0600700 1 --2- 172.21.15.12:0/3813054967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0e8160f10 0x7fd0e80d18b0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009840 tx=0x7fd0dc009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0f0600700 1 -- 172.21.15.12:0/3813054967 shutdown_connections 2024-03-29T11:33:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0f0600700 1 --2- 172.21.15.12:0/3813054967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0e8160f10 0x7fd0e80d18b0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0f0600700 1 --2- 172.21.15.12:0/3813054967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e80cf230 0x7fd0e80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0f0600700 1 -- 172.21.15.12:0/3813054967 >> 172.21.15.12:0/3813054967 conn(0x7fd0e808aff0 msgr2=0x7fd0e80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.493+0000 7fd0f0600700 1 -- 172.21.15.12:0/3813054967 shutdown_connections 2024-03-29T11:33:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.494+0000 7fd0f0600700 1 -- 172.21.15.12:0/3813054967 wait complete. 2024-03-29T11:33:13.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.495+0000 7fd0f0600700 1 Processor -- start 2024-03-29T11:33:13.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.495+0000 7fd0f0600700 1 -- start start 2024-03-29T11:33:13.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.496+0000 7fd0f0600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0e80cf230 0x7fd0e80b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.496+0000 7fd0f0600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0e8160f10 0x7fd0e809efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.496+0000 7fd0f0600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e8161590 0x7fd0e81f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.496+0000 7fd0f0600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e814c250 con 0x7fd0e80cf230 2024-03-29T11:33:13.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.496+0000 7fd0f0600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e81f3580 con 0x7fd0e8160f10 2024-03-29T11:33:13.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.496+0000 7fd0f0600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e809f5f0 con 0x7fd0e8161590 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0edc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0e80cf230 0x7fd0e80b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0edc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0e80cf230 0x7fd0e80b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54408/0 (socket says 172.21.15.12:54408) 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0edc00700 1 -- 172.21.15.12:0/3335744242 learned_addr learned my addr 172.21.15.12:0/3335744242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0edc00700 1 -- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e8161590 msgr2=0x7fd0e81f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0ee600700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e8161590 0x7fd0e81f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0edc00700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e8161590 0x7fd0e81f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0edc00700 1 -- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0e8160f10 msgr2=0x7fd0e809efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0edc00700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0e8160f10 0x7fd0e809efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0edc00700 1 -- 172.21.15.12:0/3335744242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40094a0 con 0x7fd0e80cf230 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0ee600700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e8161590 0x7fd0e81f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.497+0000 7fd0edc00700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0e80cf230 0x7fd0e80b14c0 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009fd0 tx=0x7fd0dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.498+0000 7fd0e5600700 1 -- 172.21.15.12:0/3335744242 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc0036a0 con 0x7fd0e80cf230 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.498+0000 7fd0e5600700 1 -- 172.21.15.12:0/3335744242 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc01e070 con 0x7fd0e80cf230 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.498+0000 7fd0e5600700 1 -- 172.21.15.12:0/3335744242 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc003a00 con 0x7fd0e80cf230 2024-03-29T11:33:13.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.498+0000 7fd0f0600700 1 -- 172.21.15.12:0/3335744242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc0094a0 con 0x7fd0e80cf230 2024-03-29T11:33:13.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.498+0000 7fd0f0600700 1 -- 172.21.15.12:0/3335744242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e809fbb0 con 0x7fd0e80cf230 2024-03-29T11:33:13.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.499+0000 7fd0f0600700 1 -- 172.21.15.12:0/3335744242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e8067720 con 0x7fd0e80cf230 2024-03-29T11:33:13.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.500+0000 7fd0e5600700 1 -- 172.21.15.12:0/3335744242 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 145) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fd0dc003b60 con 0x7fd0e80cf230 2024-03-29T11:33:13.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.501+0000 7fd0e5600700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0c806aae0 0x7fd0c806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.501+0000 7fd0e5600700 1 -- 172.21.15.12:0/3335744242 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7419+0+0 (secure 0 0 0) 0x7fd0dc08a5f0 con 0x7fd0e80cf230 2024-03-29T11:33:13.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.501+0000 7fd0ed200700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0c806aae0 0x7fd0c806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.502+0000 7fd0ed200700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0c806aae0 0x7fd0c806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4006f10 tx=0x7fd0d4006e60 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:13.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.505+0000 7fd0e5600700 1 -- 172.21.15.12:0/3335744242 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0dc055460 con 0x7fd0e80cf230 2024-03-29T11:33:13.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.717+0000 7fd0f0600700 1 -- 172.21.15.12:0/3335744242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0e80c9ee0 con 0x7fd0e80cf230 2024-03-29T11:33:13.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.719+0000 7fd0e5600700 1 -- 172.21.15.12:0/3335744242 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) v1 ==== 74+0+20956 (secure 0 0 0) 0x7fd0dc058a80 con 0x7fd0e80cf230 2024-03-29T11:33:13.716 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:13.716 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":254,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:12.058977+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:32:45.400233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:32:46.836046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/117966312":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/1298574278":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/449301407":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/2167892315":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/122983726":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/2916339927":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/962755797":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1966802084":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/737492091":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/110096794":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/449562747":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1567850739":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/3214276897":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/180562283":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/2535009377":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1057841974":"2024-03-30T11:33:03.088269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 -- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0c806aae0 msgr2=0x7fd0c806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0c806aae0 0x7fd0c806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4006f10 tx=0x7fd0d4006e60 comp rx=0 tx=0).stop 2024-03-29T11:33:13.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 -- 172.21.15.12:0/3335744242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0e80cf230 msgr2=0x7fd0e80b14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0e80cf230 0x7fd0e80b14c0 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009fd0 tx=0x7fd0dc004750 comp rx=0 tx=0).stop 2024-03-29T11:33:13.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 -- 172.21.15.12:0/3335744242 shutdown_connections 2024-03-29T11:33:13.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0c806aae0 0x7fd0c806cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0e8161590 0x7fd0e81f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0e80cf230 0x7fd0e80b14c0 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 --2- 172.21.15.12:0/3335744242 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0e8160f10 0x7fd0e809efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 -- 172.21.15.12:0/3335744242 >> 172.21.15.12:0/3335744242 conn(0x7fd0e808aff0 msgr2=0x7fd0e815b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:13.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 -- 172.21.15.12:0/3335744242 shutdown_connections 2024-03-29T11:33:13.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.722+0000 7fd0f0600700 1 -- 172.21.15.12:0/3335744242 wait complete. 2024-03-29T11:33:13.736 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T11:33:13.736 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:33:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.938+0000 7f6996600700 1 Processor -- start 2024-03-29T11:33:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.938+0000 7f6996600700 1 -- start start 2024-03-29T11:33:13.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.939+0000 7f6996600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f699015d920 0x7f699006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.939+0000 7f6996600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f699009a910 0x7f69900cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.939+0000 7f6996600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69900d08d0 con 0x7f6990099a60 2024-03-29T11:33:13.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.939+0000 7f6996600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699006c300 con 0x7f699009a910 2024-03-29T11:33:13.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.939+0000 7f6996600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699006b9d0 con 0x7f699015d920 2024-03-29T11:33:13.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.940+0000 7f698ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f699009a910 0x7f69900cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.940+0000 7f698ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f699009a910 0x7f69900cfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54416/0 (socket says 172.21.15.12:54416) 2024-03-29T11:33:13.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.940+0000 7f698ea00700 1 -- 172.21.15.12:0/1556832367 learned_addr learned my addr 172.21.15.12:0/1556832367 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:13.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.940+0000 7f698fe00700 1 --1- 172.21.15.12:0/1556832367 >> v1:172.21.15.12:6789/0 conn(0x7f6990099a60 0x7f6990099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59486/0 (socket says 172.21.15.12:59486) 2024-03-29T11:33:13.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.940+0000 7f698f400700 1 --2- 172.21.15.12:0/1556832367 >> v2:172.21.15.159:3300/0 conn(0x7f699015d920 0x7f699006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.940+0000 7f698ea00700 1 -- 172.21.15.12:0/1556832367 >> v2:172.21.15.159:3300/0 conn(0x7f699015d920 msgr2=0x7f699006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:13.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.940+0000 7f698ea00700 1 --2- 172.21.15.12:0/1556832367 >> v2:172.21.15.159:3300/0 conn(0x7f699015d920 0x7f699006c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.940+0000 7f698ea00700 1 -- 172.21.15.12:0/1556832367 >> v1:172.21.15.12:6789/0 conn(0x7f6990099a60 legacy=0x7f6990099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.940+0000 7f698ea00700 1 -- 172.21.15.12:0/1556832367 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699006bcb0 con 0x7f699009a910 2024-03-29T11:33:13.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.940+0000 7f698ea00700 1 --2- 172.21.15.12:0/1556832367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f699009a910 0x7f69900cfff0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f699009a6b0 tx=0x7f697c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64143b3f4866c5a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:13.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f698e000700 1 -- 172.21.15.12:0/1556832367 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c00b870 con 0x7f699009a910 2024-03-29T11:33:13.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f698e000700 1 -- 172.21.15.12:0/1556832367 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697c00b9d0 con 0x7f699009a910 2024-03-29T11:33:13.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f698e000700 1 -- 172.21.15.12:0/1556832367 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c00bd60 con 0x7f699009a910 2024-03-29T11:33:13.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f6996600700 1 -- 172.21.15.12:0/1556832367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f699009a910 msgr2=0x7f69900cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f6996600700 1 --2- 172.21.15.12:0/1556832367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f699009a910 0x7f69900cfff0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f699009a6b0 tx=0x7f697c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f6996600700 1 -- 172.21.15.12:0/1556832367 shutdown_connections 2024-03-29T11:33:13.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f6996600700 1 --2- 172.21.15.12:0/1556832367 >> v2:172.21.15.159:3300/0 conn(0x7f699015d920 0x7f699006c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f6996600700 1 --2- 172.21.15.12:0/1556832367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f699009a910 0x7f69900cfff0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f6996600700 1 -- 172.21.15.12:0/1556832367 >> 172.21.15.12:0/1556832367 conn(0x7f69900d1350 msgr2=0x7f69900d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:13.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f6996600700 1 -- 172.21.15.12:0/1556832367 shutdown_connections 2024-03-29T11:33:13.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.941+0000 7f6996600700 1 -- 172.21.15.12:0/1556832367 wait complete. 2024-03-29T11:33:13.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.943+0000 7f6996600700 1 Processor -- start 2024-03-29T11:33:13.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.943+0000 7f6996600700 1 -- start start 2024-03-29T11:33:13.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.944+0000 7f6996600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6990099a60 0x7f69900c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.944+0000 7f6996600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f699009a910 0x7f69901581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.944+0000 7f6996600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f699015d920 0x7f6990158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.944+0000 7f6996600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69900c8b30 con 0x7f6990099a60 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.944+0000 7f6996600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69900c94b0 con 0x7f699009a910 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.944+0000 7f6996600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69900b6c90 con 0x7f699015d920 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.944+0000 7f698f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6990099a60 0x7f69900c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.944+0000 7f698f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6990099a60 0x7f69900c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54424/0 (socket says 172.21.15.12:54424) 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.944+0000 7f698f400700 1 -- 172.21.15.12:0/1486524066 learned_addr learned my addr 172.21.15.12:0/1486524066 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f698ea00700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f699009a910 0x7f69901581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f698fe00700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f699015d920 0x7f6990158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f698f400700 1 -- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f699015d920 msgr2=0x7f6990158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f698f400700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f699015d920 0x7f6990158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f698f400700 1 -- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f699009a910 msgr2=0x7f69901581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f698f400700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f699009a910 0x7f69901581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f698f400700 1 -- 172.21.15.12:0/1486524066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c009550 con 0x7f6990099a60 2024-03-29T11:33:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f698f400700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6990099a60 0x7f69900c8420 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f698000d800 tx=0x7f698000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f6987400700 1 -- 172.21.15.12:0/1486524066 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69800125f0 con 0x7f6990099a60 2024-03-29T11:33:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f6987400700 1 -- 172.21.15.12:0/1486524066 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6980012750 con 0x7f6990099a60 2024-03-29T11:33:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f6996600700 1 -- 172.21.15.12:0/1486524066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6990158c70 con 0x7f6990099a60 2024-03-29T11:33:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.945+0000 7f6987400700 1 -- 172.21.15.12:0/1486524066 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6980012b00 con 0x7f6990099a60 2024-03-29T11:33:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.946+0000 7f6996600700 1 -- 172.21.15.12:0/1486524066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69900b6490 con 0x7f6990099a60 2024-03-29T11:33:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.947+0000 7f6996600700 1 -- 172.21.15.12:0/1486524066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6990067720 con 0x7f6990099a60 2024-03-29T11:33:13.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.948+0000 7f6987400700 1 -- 172.21.15.12:0/1486524066 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 145) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f6980012c60 con 0x7f6990099a60 2024-03-29T11:33:13.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.948+0000 7f6987400700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f697006aae0 0x7f697006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:13.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.948+0000 7f6987400700 1 -- 172.21.15.12:0/1486524066 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7419+0+0 (secure 0 0 0) 0x7f6980089a60 con 0x7f6990099a60 2024-03-29T11:33:13.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.949+0000 7f698ea00700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f697006aae0 0x7f697006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:13.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.950+0000 7f698ea00700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f697006aae0 0x7f697006cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f699009ad30 tx=0x7f697c008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:13.953+0000 7f6987400700 1 -- 172.21.15.12:0/1486524066 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6980054800 con 0x7f6990099a60 2024-03-29T11:33:14.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.214+0000 7f6996600700 1 -- 172.21.15.12:0/1486524066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69900b72d0 con 0x7f6990099a60 2024-03-29T11:33:14.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.214+0000 7f6987400700 1 -- 172.21.15.12:0/1486524066 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5972 (secure 0 0 0) 0x7f6980057e20 con 0x7f6990099a60 2024-03-29T11:33:14.211 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:14.211 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":128,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:32.933737+0000","modified":"2024-03-29T11:32:36.420698+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11853":{"gid":11853,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3711953177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3711953177},{"type":"v1","addr":"172.21.15.12:6834","nonce":3711953177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:47.849181+0000","modified":"2024-03-29T11:32:51.340904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11416":{"gid":11416,"name":"d","rank":0,"incarnation":124,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-29T11:33:14.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 -- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f697006aae0 msgr2=0x7f697006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f697006aae0 0x7f697006cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f699009ad30 tx=0x7f697c008030 comp rx=0 tx=0).stop 2024-03-29T11:33:14.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 -- 172.21.15.12:0/1486524066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6990099a60 msgr2=0x7f69900c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6990099a60 0x7f69900c8420 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f698000d800 tx=0x7f698000dbc0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 -- 172.21.15.12:0/1486524066 shutdown_connections 2024-03-29T11:33:14.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f697006aae0 0x7f697006cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f699015d920 0x7f6990158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6990099a60 0x7f69900c8420 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 --2- 172.21.15.12:0/1486524066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f699009a910 0x7f69901581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 -- 172.21.15.12:0/1486524066 >> 172.21.15.12:0/1486524066 conn(0x7f69900d1350 msgr2=0x7f69900cdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:14.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 -- 172.21.15.12:0/1486524066 shutdown_connections 2024-03-29T11:33:14.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.217+0000 7f6996600700 1 -- 172.21.15.12:0/1486524066 wait complete. 2024-03-29T11:33:14.214 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 128 2024-03-29T11:33:14.231 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:33:14.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.435+0000 7f1514a00700 1 Processor -- start 2024-03-29T11:33:14.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.435+0000 7f1514a00700 1 -- start start 2024-03-29T11:33:14.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.436+0000 7f1514a00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f151015d640 0x7f15100d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.436+0000 7f1514a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f15101612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.436+0000 7f1514a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15100d30d0 con 0x7f1510161800 2024-03-29T11:33:14.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.436+0000 7f1514a00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1510161c90 con 0x7f151015d640 2024-03-29T11:33:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.436+0000 7f1514a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15100b6840 con 0x7f15100ceec0 2024-03-29T11:33:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f15101612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150de00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f151015d640 0x7f15100d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150de00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f151015d640 0x7f15100d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:39358/0 (socket says 172.21.15.12:39358) 2024-03-29T11:33:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150de00700 1 -- 172.21.15.12:0/209511595 learned_addr learned my addr 172.21.15.12:0/209511595 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150d400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f15101612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:53572/0 (socket says 172.21.15.12:53572) 2024-03-29T11:33:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150d400700 1 -- 172.21.15.12:0/209511595 >> v2:172.21.15.159:3300/0 conn(0x7f151015d640 msgr2=0x7f15100d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:14.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150d400700 1 --2- 172.21.15.12:0/209511595 >> v2:172.21.15.159:3300/0 conn(0x7f151015d640 0x7f15100d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150d400700 1 -- 172.21.15.12:0/209511595 >> v1:172.21.15.159:6789/0 conn(0x7f1510161800 legacy=0x7f15100cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150d400700 1 -- 172.21.15.12:0/209511595 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15100d2880 con 0x7f15100ceec0 2024-03-29T11:33:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150de00700 1 --2- 172.21.15.12:0/209511595 >> v2:172.21.15.159:3300/0 conn(0x7f151015d640 0x7f15100d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.437+0000 7f150ca00700 1 -- 172.21.15.12:0/209511595 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932947129 0 0) 0x7f15100d30d0 con 0x7f1510161800 2024-03-29T11:33:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.438+0000 7f150d400700 1 --2- 172.21.15.12:0/209511595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f15101612c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009840 tx=0x7f14fc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a7f5dd8f5d8e136 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.438+0000 7f150ca00700 1 -- 172.21.15.12:0/209511595 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc011830 con 0x7f15100ceec0 2024-03-29T11:33:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.438+0000 7f150ca00700 1 -- 172.21.15.12:0/209511595 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc011990 con 0x7f15100ceec0 2024-03-29T11:33:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.438+0000 7f150ca00700 1 -- 172.21.15.12:0/209511595 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc011d20 con 0x7f15100ceec0 2024-03-29T11:33:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.438+0000 7f1514a00700 1 -- 172.21.15.12:0/209511595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 msgr2=0x7f15101612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.438+0000 7f1514a00700 1 --2- 172.21.15.12:0/209511595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f15101612c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009840 tx=0x7f14fc009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.439+0000 7f1514a00700 1 -- 172.21.15.12:0/209511595 shutdown_connections 2024-03-29T11:33:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.439+0000 7f1514a00700 1 --2- 172.21.15.12:0/209511595 >> v2:172.21.15.159:3300/0 conn(0x7f151015d640 0x7f15100d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.439+0000 7f1514a00700 1 --2- 172.21.15.12:0/209511595 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f15101612c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.439+0000 7f1514a00700 1 -- 172.21.15.12:0/209511595 >> 172.21.15.12:0/209511595 conn(0x7f151008aff0 msgr2=0x7f15100d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.439+0000 7f1514a00700 1 -- 172.21.15.12:0/209511595 shutdown_connections 2024-03-29T11:33:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.439+0000 7f1514a00700 1 -- 172.21.15.12:0/209511595 wait complete. 2024-03-29T11:33:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.440+0000 7f1514a00700 1 Processor -- start 2024-03-29T11:33:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.441+0000 7f1514a00700 1 -- start start 2024-03-29T11:33:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.441+0000 7f1514a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f1510155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.441+0000 7f1514a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151015d640 0x7f15101f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.441+0000 7f1514a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1510161800 0x7f15100c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.441+0000 7f1514a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1510155fe0 con 0x7f151015d640 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.441+0000 7f1514a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1510156380 con 0x7f1510161800 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.441+0000 7f1514a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1510156660 con 0x7f15100ceec0 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151015d640 0x7f15101f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151015d640 0x7f15101f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54438/0 (socket says 172.21.15.12:54438) 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150d400700 1 -- 172.21.15.12:0/3147231140 learned_addr learned my addr 172.21.15.12:0/3147231140 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150de00700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f1510155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150e800700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1510161800 0x7f15100c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150d400700 1 -- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 msgr2=0x7f1510155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150d400700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f1510155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150d400700 1 -- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1510161800 msgr2=0x7f15100c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150d400700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1510161800 0x7f15100c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150d400700 1 -- 172.21.15.12:0/3147231140 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1500009550 con 0x7f151015d640 2024-03-29T11:33:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.442+0000 7f150de00700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f1510155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.443+0000 7f150d400700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151015d640 0x7f15101f2c60 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009fd0 tx=0x7f14fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.443+0000 7f1506000700 1 -- 172.21.15.12:0/3147231140 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc0036a0 con 0x7f151015d640 2024-03-29T11:33:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.443+0000 7f1506000700 1 -- 172.21.15.12:0/3147231140 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc01e070 con 0x7f151015d640 2024-03-29T11:33:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.443+0000 7f1514a00700 1 -- 172.21.15.12:0/3147231140 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14fc0094a0 con 0x7f151015d640 2024-03-29T11:33:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.443+0000 7f1506000700 1 -- 172.21.15.12:0/3147231140 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc003a60 con 0x7f151015d640 2024-03-29T11:33:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.443+0000 7f1514a00700 1 -- 172.21.15.12:0/3147231140 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1510156c20 con 0x7f151015d640 2024-03-29T11:33:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.444+0000 7f1514a00700 1 -- 172.21.15.12:0/3147231140 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1510067720 con 0x7f151015d640 2024-03-29T11:33:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.445+0000 7f1506000700 1 -- 172.21.15.12:0/3147231140 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 146) v1 ==== 89285+0+0 (secure 0 0 0) 0x7f14fc003bc0 con 0x7f151015d640 2024-03-29T11:33:14.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.446+0000 7f1506000700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f14f006ae40 0x7f14f006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.446+0000 7f1506000700 1 -- 172.21.15.12:0/3147231140 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7419+0+0 (secure 0 0 0) 0x7f14fc08ad00 con 0x7f151015d640 2024-03-29T11:33:14.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.447+0000 7f150de00700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f14f006ae40 0x7f14f006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.447+0000 7f150de00700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f14f006ae40 0x7f14f006d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1500009d10 tx=0x7f1500008040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:14.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.451+0000 7f1506000700 1 -- 172.21.15.12:0/3147231140 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f14fc0555a0 con 0x7f151015d640 2024-03-29T11:33:14.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.713+0000 7f1514a00700 1 -- 172.21.15.12:0/3147231140 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f151015cdf0 con 0x7f151015d640 2024-03-29T11:33:14.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.713+0000 7f1506000700 1 -- 172.21.15.12:0/3147231140 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5972 (secure 0 0 0) 0x7f14fc058bc0 con 0x7f151015d640 2024-03-29T11:33:14.710 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:14.710 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":128,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:32.933737+0000","modified":"2024-03-29T11:32:36.420698+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11853":{"gid":11853,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/3711953177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3711953177},{"type":"v1","addr":"172.21.15.12:6834","nonce":3711953177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:47.849181+0000","modified":"2024-03-29T11:32:51.340904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11416":{"gid":11416,"name":"d","rank":0,"incarnation":124,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-29T11:33:14.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 -- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f14f006ae40 msgr2=0x7f14f006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f14f006ae40 0x7f14f006d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1500009d10 tx=0x7f1500008040 comp rx=0 tx=0).stop 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 -- 172.21.15.12:0/3147231140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151015d640 msgr2=0x7f15101f2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151015d640 0x7f15101f2c60 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009fd0 tx=0x7f14fc004750 comp rx=0 tx=0).stop 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 -- 172.21.15.12:0/3147231140 shutdown_connections 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f14f006ae40 0x7f14f006d300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1510161800 0x7f15100c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ceec0 0x7f1510155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 --2- 172.21.15.12:0/3147231140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151015d640 0x7f15101f2c60 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 -- 172.21.15.12:0/3147231140 >> 172.21.15.12:0/3147231140 conn(0x7f151008aff0 msgr2=0x7f151015b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 -- 172.21.15.12:0/3147231140 shutdown_connections 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.716+0000 7f1514a00700 1 -- 172.21.15.12:0/3147231140 wait complete. 2024-03-29T11:33:14.713 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 128 2024-03-29T11:33:14.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:14.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.932+0000 7f6ca2200700 1 Processor -- start 2024-03-29T11:33:14.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.932+0000 7f6ca2200700 1 -- start start 2024-03-29T11:33:14.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.933+0000 7f6ca2200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6c9c0d18b0 0x7f6c9c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6ca2200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6c9c161590 0x7f6c9c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6ca2200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9c0ceda0 con 0x7f6c9c160f10 2024-03-29T11:33:14.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6ca2200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9c0d1d30 con 0x7f6c9c161590 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6ca2200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9c161f60 con 0x7f6c9c0d18b0 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6c9be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6c9c161590 0x7f6c9c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6c9be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6c9c161590 0x7f6c9c161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54444/0 (socket says 172.21.15.12:54444) 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6c9b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6c9c160f10 0x7f6c9c0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59492/0 (socket says 172.21.15.12:59492) 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6c9be00700 1 -- 172.21.15.12:0/6346300 learned_addr learned my addr 172.21.15.12:0/6346300 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6c93e00700 1 --2- 172.21.15.12:0/6346300 >> v2:172.21.15.159:3300/0 conn(0x7f6c9c0d18b0 0x7f6c9c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6c9be00700 1 -- 172.21.15.12:0/6346300 >> v2:172.21.15.159:3300/0 conn(0x7f6c9c0d18b0 msgr2=0x7f6c9c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6c9be00700 1 --2- 172.21.15.12:0/6346300 >> v2:172.21.15.159:3300/0 conn(0x7f6c9c0d18b0 0x7f6c9c0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6c9be00700 1 -- 172.21.15.12:0/6346300 >> v1:172.21.15.12:6789/0 conn(0x7f6c9c160f10 legacy=0x7f6c9c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6c9be00700 1 -- 172.21.15.12:0/6346300 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c1612f0 con 0x7f6c9c161590 2024-03-29T11:33:14.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.934+0000 7f6c9aa00700 1 -- 172.21.15.12:0/6346300 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084633838 0 0) 0x7f6c9c0ceda0 con 0x7f6c9c160f10 2024-03-29T11:33:14.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.935+0000 7f6c9be00700 1 --2- 172.21.15.12:0/6346300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 0x7f6c9c161a20 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c08ca30 tx=0x7f6c9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0676a89c96bd9f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.935+0000 7f6c9aa00700 1 -- 172.21.15.12:0/6346300 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9400e070 con 0x7f6c9c161590 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.935+0000 7f6c9aa00700 1 -- 172.21.15.12:0/6346300 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9400b9c0 con 0x7f6c9c161590 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.935+0000 7f6c9aa00700 1 -- 172.21.15.12:0/6346300 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9400bd70 con 0x7f6c9c161590 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.935+0000 7f6ca2200700 1 -- 172.21.15.12:0/6346300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 msgr2=0x7f6c9c161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.935+0000 7f6ca2200700 1 --2- 172.21.15.12:0/6346300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 0x7f6c9c161a20 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c08ca30 tx=0x7f6c9400b330 comp rx=0 tx=0).stop 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.936+0000 7f6ca2200700 1 -- 172.21.15.12:0/6346300 shutdown_connections 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.936+0000 7f6ca2200700 1 --2- 172.21.15.12:0/6346300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 0x7f6c9c161a20 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.936+0000 7f6ca2200700 1 --2- 172.21.15.12:0/6346300 >> v2:172.21.15.159:3300/0 conn(0x7f6c9c0d18b0 0x7f6c9c0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.936+0000 7f6ca2200700 1 -- 172.21.15.12:0/6346300 >> 172.21.15.12:0/6346300 conn(0x7f6c9c08aff0 msgr2=0x7f6c9c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.936+0000 7f6ca2200700 1 -- 172.21.15.12:0/6346300 shutdown_connections 2024-03-29T11:33:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.936+0000 7f6ca2200700 1 -- 172.21.15.12:0/6346300 wait complete. 2024-03-29T11:33:14.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.937+0000 7f6ca2200700 1 Processor -- start 2024-03-29T11:33:14.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.938+0000 7f6ca2200700 1 -- start start 2024-03-29T11:33:14.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.938+0000 7f6ca2200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9c0d18b0 0x7f6c9c158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6ca2200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c9c160f10 0x7f6c9c0b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6ca2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 0x7f6c9c0cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6ca2200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c0b8160 con 0x7f6c9c161590 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6ca2200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c0b7ba0 con 0x7f6c9c160f10 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6ca2200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c0b7e80 con 0x7f6c9c0d18b0 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6c9be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 0x7f6c9c0cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6c9be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 0x7f6c9c0cc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54458/0 (socket says 172.21.15.12:54458) 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6c9be00700 1 -- 172.21.15.12:0/3317938901 learned_addr learned my addr 172.21.15.12:0/3317938901 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6c9b400700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9c0d18b0 0x7f6c9c158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6c93e00700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c9c160f10 0x7f6c9c0b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6c9be00700 1 -- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9c0d18b0 msgr2=0x7f6c9c158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6c9be00700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9c0d18b0 0x7f6c9c158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6c9be00700 1 -- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c9c160f10 msgr2=0x7f6c9c0b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6c9be00700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c9c160f10 0x7f6c9c0b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.939+0000 7f6c9be00700 1 -- 172.21.15.12:0/3317938901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c94009b30 con 0x7f6c9c161590 2024-03-29T11:33:14.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.940+0000 7f6c9be00700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 0x7f6c9c0cc110 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c08c570 tx=0x7f6c94004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:14.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.940+0000 7f6c93400700 1 -- 172.21.15.12:0/3317938901 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c940099b0 con 0x7f6c9c161590 2024-03-29T11:33:14.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.940+0000 7f6c93400700 1 -- 172.21.15.12:0/3317938901 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c940092e0 con 0x7f6c9c161590 2024-03-29T11:33:14.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.940+0000 7f6c93400700 1 -- 172.21.15.12:0/3317938901 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9401c640 con 0x7f6c9c161590 2024-03-29T11:33:14.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.940+0000 7f6ca2200700 1 -- 172.21.15.12:0/3317938901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9c1563b0 con 0x7f6c9c161590 2024-03-29T11:33:14.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.940+0000 7f6ca2200700 1 -- 172.21.15.12:0/3317938901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c158e40 con 0x7f6c9c161590 2024-03-29T11:33:14.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.941+0000 7f6ca2200700 1 -- 172.21.15.12:0/3317938901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9c1568d0 con 0x7f6c9c161590 2024-03-29T11:33:14.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.942+0000 7f6c93400700 1 -- 172.21.15.12:0/3317938901 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 146) v1 ==== 89285+0+0 (secure 0 0 0) 0x7f6c9401c7a0 con 0x7f6c9c161590 2024-03-29T11:33:14.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.943+0000 7f6c93400700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7806ae40 0x7f6c7806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:14.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.943+0000 7f6c93400700 1 -- 172.21.15.12:0/3317938901 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7419+0+0 (secure 0 0 0) 0x7f6c9408b7c0 con 0x7f6c9c161590 2024-03-29T11:33:14.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.943+0000 7f6c9b400700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7806ae40 0x7f6c7806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:14.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.944+0000 7f6c9b400700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7806ae40 0x7f6c7806d300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c000f60 tx=0x7f6c8c007c90 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:14.947+0000 7f6c93400700 1 -- 172.21.15.12:0/3317938901 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c9c1568d0 con 0x7f6c9c161590 2024-03-29T11:33:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.162+0000 7f6ca2200700 1 -- 172.21.15.12:0/3317938901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c9c1568d0 con 0x7f6c9c161590 2024-03-29T11:33:15.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.164+0000 7f6c93400700 1 -- 172.21.15.12:0/3317938901 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) v1 ==== 74+0+20956 (secure 0 0 0) 0x7f6c9c1568d0 con 0x7f6c9c161590 2024-03-29T11:33:15.160 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:15.161 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":254,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:12.058977+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:32:31.368651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-03-29T11:32:31.900686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:32:45.400233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:32:46.836046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/117966312":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/1298574278":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/449301407":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/2167892315":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/122983726":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/2916339927":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/962755797":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1966802084":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/737492091":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/110096794":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/449562747":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1567850739":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/3214276897":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/180562283":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/2535009377":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1057841974":"2024-03-30T11:33:03.088269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 -- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7806ae40 msgr2=0x7f6c7806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7806ae40 0x7f6c7806d300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c000f60 tx=0x7f6c8c007c90 comp rx=0 tx=0).stop 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 -- 172.21.15.12:0/3317938901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 msgr2=0x7f6c9c0cc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 0x7f6c9c0cc110 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c08c570 tx=0x7f6c94004ad0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 -- 172.21.15.12:0/3317938901 shutdown_connections 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6c7806ae40 0x7f6c7806d300 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c9c161590 0x7f6c9c0cc110 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c9c0d18b0 0x7f6c9c158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 --2- 172.21.15.12:0/3317938901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c9c160f10 0x7f6c9c0b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 -- 172.21.15.12:0/3317938901 >> 172.21.15.12:0/3317938901 conn(0x7f6c9c08aff0 msgr2=0x7f6c9c0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 -- 172.21.15.12:0/3317938901 shutdown_connections 2024-03-29T11:33:15.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.166+0000 7f6ca2200700 1 -- 172.21.15.12:0/3317938901 wait complete. 2024-03-29T11:33:15.182 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T11:33:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.386+0000 7f32c0000700 1 Processor -- start 2024-03-29T11:33:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.386+0000 7f32c0000700 1 -- start start 2024-03-29T11:33:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.387+0000 7f32c0000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b80cd020 0x7f32b80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.388+0000 7f32c0000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b80cdae0 con 0x7f32b80b2070 2024-03-29T11:33:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.388+0000 7f32c0000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b80cddc0 con 0x7f32b80cbb30 2024-03-29T11:33:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.388+0000 7f32c0000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b80ce0a0 con 0x7f32b80cd020 2024-03-29T11:33:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.388+0000 7f32be000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f32b80b2070 0x7f32b806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59500/0 (socket says 172.21.15.12:59500) 2024-03-29T11:33:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.388+0000 7f32be000700 1 -- 172.21.15.12:0/2528162196 learned_addr learned my addr 172.21.15.12:0/2528162196 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.388+0000 7f32bcc00700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b80cd020 0x7f32b80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.388+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075369020 0 0) 0x7f32b80cdae0 con 0x7f32b80b2070 2024-03-29T11:33:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.388+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a00035b0 con 0x7f32b80b2070 2024-03-29T11:33:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1236064660 0 0) 0x7f32a00035b0 con 0x7f32b80b2070 2024-03-29T11:33:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b80cd020 msgr2=0x7f32b80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b80cd020 0x7f32b80cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 >> v1:172.21.15.159:6789/0 conn(0x7f32b80cbb30 legacy=0x7f32b80cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b806dc20 con 0x7f32b80b2070 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f32b00019f0 con 0x7f32b80b2070 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 >> v1:172.21.15.12:6789/0 conn(0x7f32b80b2070 legacy=0x7f32b806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32a0003a30 0x7f32a0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a0004850 0x7f32a0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b80cd020 0x7f32a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b80ce0a0 con 0x7f32a0004850 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a000bac0 con 0x7f32a0003a30 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a000bc30 con 0x7f32b80cd020 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32bcc00700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a0004850 0x7f32a0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32bd600700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32a0003a30 0x7f32a0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32be000700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b80cd020 0x7f32a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32bcc00700 1 -- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b80cd020 msgr2=0x7f32a0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32bcc00700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b80cd020 0x7f32a0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32bcc00700 1 -- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32a0003a30 msgr2=0x7f32a0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32bcc00700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32a0003a30 0x7f32a0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32bcc00700 1 -- 172.21.15.12:0/2528162196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a00035b0 con 0x7f32a0004850 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.389+0000 7f32be000700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b80cd020 0x7f32a0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.390+0000 7f32bcc00700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a0004850 0x7f32a0004cd0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f32a4009b30 tx=0x7f32a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36dbf9d3ec6e6eda server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.390+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a400e660 con 0x7f32a0004850 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.390+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a400e7c0 con 0x7f32a0004850 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.390+0000 7f32b7e00700 1 -- 172.21.15.12:0/2528162196 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a400eb30 con 0x7f32a0004850 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.390+0000 7f32c0000700 1 -- 172.21.15.12:0/2528162196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a0004850 msgr2=0x7f32a0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.390+0000 7f32c0000700 1 --2- 172.21.15.12:0/2528162196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32a0004850 0x7f32a0004cd0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f32a4009b30 tx=0x7f32a0005880 comp rx=0 tx=0).stop 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.390+0000 7f32be000700 1 -- 172.21.15.12:0/2528162196 reap_dead start 2024-03-29T11:33:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.391+0000 7f32c0000700 1 -- 172.21.15.12:0/2528162196 shutdown_connections 2024-03-29T11:33:15.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.391+0000 7f32c0000700 1 -- 172.21.15.12:0/2528162196 >> 172.21.15.12:0/2528162196 conn(0x7f32b80b0d70 msgr2=0x7f32b80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:15.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.391+0000 7f32c0000700 1 -- 172.21.15.12:0/2528162196 shutdown_connections 2024-03-29T11:33:15.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.391+0000 7f32c0000700 1 -- 172.21.15.12:0/2528162196 wait complete. 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.392+0000 7f32c0000700 1 Processor -- start 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.393+0000 7f32c0000700 1 -- start start 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.393+0000 7f32c0000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a0004850 0x7f32b8071820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.393+0000 7f32c0000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b8159620 0x7f32b8072470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.393+0000 7f32c0000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b815db70 0x7f32b80c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.393+0000 7f32c0000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b815b650 con 0x7f32b815db70 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.393+0000 7f32c0000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b8071d60 con 0x7f32b8159620 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.393+0000 7f32c0000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b8071390 con 0x7f32a0004850 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32be000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b815db70 0x7f32b80c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32be000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b815db70 0x7f32b80c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54472/0 (socket says 172.21.15.12:54472) 2024-03-29T11:33:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32be000700 1 -- 172.21.15.12:0/865579319 learned_addr learned my addr 172.21.15.12:0/865579319 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32bd600700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a0004850 0x7f32b8071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32be000700 1 -- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a0004850 msgr2=0x7f32b8071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32bcc00700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b8159620 0x7f32b8072470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32be000700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a0004850 0x7f32b8071820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32be000700 1 -- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b8159620 msgr2=0x7f32b8072470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32be000700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b8159620 0x7f32b8072470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32be000700 1 -- 172.21.15.12:0/865579319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b80729b0 con 0x7f32b815db70 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32bd600700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a0004850 0x7f32b8071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32bcc00700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b8159620 0x7f32b8072470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.394+0000 7f32be000700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b815db70 0x7f32b80c83f0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f32b806c9f0 tx=0x7f32b000b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.395+0000 7f32b5600700 1 -- 172.21.15.12:0/865579319 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b000bb80 con 0x7f32b815db70 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.395+0000 7f32b5600700 1 -- 172.21.15.12:0/865579319 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b000bce0 con 0x7f32b815db70 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.395+0000 7f32b5600700 1 -- 172.21.15.12:0/865579319 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0007270 con 0x7f32b815db70 2024-03-29T11:33:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.395+0000 7f32c0000700 1 -- 172.21.15.12:0/865579319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b8159b90 con 0x7f32b815db70 2024-03-29T11:33:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.395+0000 7f32c0000700 1 -- 172.21.15.12:0/865579319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b8070d40 con 0x7f32b815db70 2024-03-29T11:33:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.396+0000 7f32c0000700 1 -- 172.21.15.12:0/865579319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b8071540 con 0x7f32b815db70 2024-03-29T11:33:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.397+0000 7f32b5600700 1 -- 172.21.15.12:0/865579319 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 146) v1 ==== 89285+0+0 (secure 0 0 0) 0x7f32b00073d0 con 0x7f32b815db70 2024-03-29T11:33:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.398+0000 7f32b5600700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f329806ae40 0x7f329806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.398+0000 7f32b5600700 1 -- 172.21.15.12:0/865579319 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7419+0+0 (secure 0 0 0) 0x7f32b0089a60 con 0x7f32b815db70 2024-03-29T11:33:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.398+0000 7f32bd600700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f329806ae40 0x7f329806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.399+0000 7f32bd600700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f329806ae40 0x7f329806d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f32ac000c00 tx=0x7f32ac009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:15.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.402+0000 7f32b5600700 1 -- 172.21.15.12:0/865579319 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32b8071540 con 0x7f32b815db70 2024-03-29T11:33:15.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.668+0000 7f32c0000700 1 -- 172.21.15.12:0/865579319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f32b8071500 con 0x7f32b815db70 2024-03-29T11:33:15.665 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T11:33:15.669+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T11:33:15.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.684+0000 7f32b5600700 1 -- 172.21.15.12:0/865579319 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v129) v1 ==== 161+0+0 (secure 0 0 0) 0x7f32b0019090 con 0x7f32b815db70 2024-03-29T11:33:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 -- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f329806ae40 msgr2=0x7f329806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f329806ae40 0x7f329806d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f32ac000c00 tx=0x7f32ac009040 comp rx=0 tx=0).stop 2024-03-29T11:33:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 -- 172.21.15.12:0/865579319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b815db70 msgr2=0x7f32b80c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b815db70 0x7f32b80c83f0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f32b806c9f0 tx=0x7f32b000b0d0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 -- 172.21.15.12:0/865579319 shutdown_connections 2024-03-29T11:33:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f329806ae40 0x7f329806d300 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b815db70 0x7f32b80c83f0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32a0004850 0x7f32b8071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 --2- 172.21.15.12:0/865579319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b8159620 0x7f32b8072470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 -- 172.21.15.12:0/865579319 >> 172.21.15.12:0/865579319 conn(0x7f32b80b0d70 msgr2=0x7f32b80ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 -- 172.21.15.12:0/865579319 shutdown_connections 2024-03-29T11:33:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.687+0000 7f32c0000700 1 -- 172.21.15.12:0/865579319 wait complete. 2024-03-29T11:33:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T11:33:15.690 INFO:tasks.ceph.mds.a.smithi012.stderr: -5641> 2024-03-29T11:32:38.376+0000 7f5f30400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:33:15.703 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T11:33:15.741 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:33:15.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.910+0000 7f045dc00700 1 Processor -- start 2024-03-29T11:33:15.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.911+0000 7f045dc00700 1 -- start start 2024-03-29T11:33:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.912+0000 7f045dc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f04580b3f80 0x7f04580ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.912+0000 7f045dc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0458161230 con 0x7f04580d3f20 2024-03-29T11:33:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.912+0000 7f045dc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04580d2a80 con 0x7f0458160e50 2024-03-29T11:33:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.912+0000 7f045dc00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04580d43b0 con 0x7f04580b3f80 2024-03-29T11:33:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.912+0000 7f0456000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f04580d3f20 0x7f04580cdef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59506/0 (socket says 172.21.15.12:59506) 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.912+0000 7f0456000700 1 -- 172.21.15.12:0/1295865360 learned_addr learned my addr 172.21.15.12:0/1295865360 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.912+0000 7f0457400700 1 --2- 172.21.15.12:0/1295865360 >> v2:172.21.15.12:3300/0 conn(0x7f04580b3f80 0x7f04580ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.912+0000 7f0455600700 1 -- 172.21.15.12:0/1295865360 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882480548 0 0) 0x7f0458161230 con 0x7f04580d3f20 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.912+0000 7f0455600700 1 -- 172.21.15.12:0/1295865360 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f043c0035b0 con 0x7f04580d3f20 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.913+0000 7f0457400700 1 -- 172.21.15.12:0/1295865360 >> v1:172.21.15.159:6789/0 conn(0x7f0458160e50 legacy=0x7f04580b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.913+0000 7f0457400700 1 -- 172.21.15.12:0/1295865360 >> v1:172.21.15.12:6789/0 conn(0x7f04580d3f20 legacy=0x7f04580cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.913+0000 7f0457400700 1 -- 172.21.15.12:0/1295865360 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04580d21e0 con 0x7f04580b3f80 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.913+0000 7f0457400700 1 --2- 172.21.15.12:0/1295865360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04580b3f80 0x7f04580ce600 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f0450009d90 tx=0x7f045000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44e79b606fabadf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.913+0000 7f0455600700 1 -- 172.21.15.12:0/1295865360 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045000e070 con 0x7f04580b3f80 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.913+0000 7f0455600700 1 -- 172.21.15.12:0/1295865360 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045000bb20 con 0x7f04580b3f80 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.913+0000 7f0455600700 1 -- 172.21.15.12:0/1295865360 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450012430 con 0x7f04580b3f80 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.914+0000 7f045dc00700 1 -- 172.21.15.12:0/1295865360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04580b3f80 msgr2=0x7f04580ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.914+0000 7f045dc00700 1 --2- 172.21.15.12:0/1295865360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04580b3f80 0x7f04580ce600 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f0450009d90 tx=0x7f045000b330 comp rx=0 tx=0).stop 2024-03-29T11:33:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.914+0000 7f045dc00700 1 -- 172.21.15.12:0/1295865360 shutdown_connections 2024-03-29T11:33:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.914+0000 7f045dc00700 1 --2- 172.21.15.12:0/1295865360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04580b3f80 0x7f04580ce600 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.914+0000 7f045dc00700 1 -- 172.21.15.12:0/1295865360 >> 172.21.15.12:0/1295865360 conn(0x7f045808aff0 msgr2=0x7f04580d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.914+0000 7f045dc00700 1 -- 172.21.15.12:0/1295865360 shutdown_connections 2024-03-29T11:33:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.914+0000 7f045dc00700 1 -- 172.21.15.12:0/1295865360 wait complete. 2024-03-29T11:33:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.915+0000 7f045dc00700 1 Processor -- start 2024-03-29T11:33:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.916+0000 7f045dc00700 1 -- start start 2024-03-29T11:33:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.916+0000 7f045dc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04580b3f80 0x7f04580b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f045dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0458160e50 0x7f04581509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f045dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045815d050 0x7f04580c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f045dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04580cafd0 con 0x7f045815d050 2024-03-29T11:33:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f045dc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04580cb270 con 0x7f04580b3f80 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f045dc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04580b7fc0 con 0x7f0458160e50 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0457400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045815d050 0x7f04580c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0457400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045815d050 0x7f04580c6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54500/0 (socket says 172.21.15.12:54500) 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0457400700 1 -- 172.21.15.12:0/2761037377 learned_addr learned my addr 172.21.15.12:0/2761037377 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0456a00700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04580b3f80 0x7f04580b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0456000700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0458160e50 0x7f04581509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0457400700 1 -- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0458160e50 msgr2=0x7f04581509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0457400700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0458160e50 0x7f04581509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0457400700 1 -- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04580b3f80 msgr2=0x7f04580b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0457400700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04580b3f80 0x7f04580b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0457400700 1 -- 172.21.15.12:0/2761037377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0450009a70 con 0x7f045815d050 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0456000700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0458160e50 0x7f04581509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.917+0000 7f0457400700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045815d050 0x7f04580c6f90 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f045808ba30 tx=0x7f0450004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.918+0000 7f044ea00700 1 -- 172.21.15.12:0/2761037377 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450012430 con 0x7f045815d050 2024-03-29T11:33:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.918+0000 7f044ea00700 1 -- 172.21.15.12:0/2761037377 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0450020070 con 0x7f045815d050 2024-03-29T11:33:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.918+0000 7f045dc00700 1 -- 172.21.15.12:0/2761037377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04580b8290 con 0x7f045815d050 2024-03-29T11:33:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.918+0000 7f044ea00700 1 -- 172.21.15.12:0/2761037377 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450004280 con 0x7f045815d050 2024-03-29T11:33:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.918+0000 7f045dc00700 1 -- 172.21.15.12:0/2761037377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04580b8850 con 0x7f045815d050 2024-03-29T11:33:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.919+0000 7f045dc00700 1 -- 172.21.15.12:0/2761037377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04580cddf0 con 0x7f045815d050 2024-03-29T11:33:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.920+0000 7f044ea00700 1 -- 172.21.15.12:0/2761037377 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 146) v1 ==== 89285+0+0 (secure 0 0 0) 0x7f04500043e0 con 0x7f045815d050 2024-03-29T11:33:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.921+0000 7f044ea00700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f043806ae40 0x7f043806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.921+0000 7f044ea00700 1 -- 172.21.15.12:0/2761037377 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7505+0+0 (secure 0 0 0) 0x7f0450022070 con 0x7f045815d050 2024-03-29T11:33:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.921+0000 7f0456a00700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f043806ae40 0x7f043806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.922+0000 7f0456a00700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f043806ae40 0x7f043806d300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0448000f60 tx=0x7f0448009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:15.925+0000 7f044ea00700 1 -- 172.21.15.12:0/2761037377 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0450056300 con 0x7f045815d050 2024-03-29T11:33:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.187+0000 7f045dc00700 1 -- 172.21.15.12:0/2761037377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0458067720 con 0x7f045815d050 2024-03-29T11:33:16.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.696+0000 7f044ea00700 1 -- 172.21.15.12:0/2761037377 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f04500566d0 con 0x7f045815d050 2024-03-29T11:33:16.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.697+0000 7f044ea00700 1 -- 172.21.15.12:0/2761037377 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v130) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0450059d90 con 0x7f045815d050 2024-03-29T11:33:16.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 -- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f043806ae40 msgr2=0x7f043806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:16.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f043806ae40 0x7f043806d300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0448000f60 tx=0x7f0448009040 comp rx=0 tx=0).stop 2024-03-29T11:33:16.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 -- 172.21.15.12:0/2761037377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045815d050 msgr2=0x7f04580c6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045815d050 0x7f04580c6f90 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f045808ba30 tx=0x7f0450004ad0 comp rx=0 tx=0).stop 2024-03-29T11:33:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 -- 172.21.15.12:0/2761037377 shutdown_connections 2024-03-29T11:33:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f043806ae40 0x7f043806d300 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045815d050 0x7f04580c6f90 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04580b3f80 0x7f04580b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 --2- 172.21.15.12:0/2761037377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0458160e50 0x7f04581509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 -- 172.21.15.12:0/2761037377 >> 172.21.15.12:0/2761037377 conn(0x7f045808aff0 msgr2=0x7f045815f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 -- 172.21.15.12:0/2761037377 shutdown_connections 2024-03-29T11:33:16.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.701+0000 7f045dc00700 1 -- 172.21.15.12:0/2761037377 wait complete. 2024-03-29T11:33:16.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T11:33:16.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.965+0000 7f0de8200700 1 Processor -- start 2024-03-29T11:33:16.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.965+0000 7f0de8200700 1 -- start start 2024-03-29T11:33:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.966+0000 7f0de8200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0de00b3930 0x7f0de009a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.966+0000 7f0de8200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de00cbdf0 con 0x7f0de015db10 2024-03-29T11:33:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.966+0000 7f0de8200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de006bec0 con 0x7f0de00cc280 2024-03-29T11:33:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.966+0000 7f0de8200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de00b27a0 con 0x7f0de00b3930 2024-03-29T11:33:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.966+0000 7f0de5800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0de015db10 0x7f0de0099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59518/0 (socket says 172.21.15.12:59518) 2024-03-29T11:33:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.966+0000 7f0de5800700 1 -- 172.21.15.12:0/3793674407 learned_addr learned my addr 172.21.15.12:0/3793674407 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0de4e00700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0de00b3930 0x7f0de009a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948764616 0 0) 0x7f0de00cbdf0 con 0x7f0de015db10 2024-03-29T11:33:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc80035b0 con 0x7f0de015db10 2024-03-29T11:33:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773508221 0 0) 0x7f0de006bec0 con 0x7f0de00cc280 2024-03-29T11:33:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de00cbdf0 con 0x7f0de00cc280 2024-03-29T11:33:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3403323703 0 0) 0x7f0dc80035b0 con 0x7f0de015db10 2024-03-29T11:33:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0de00b3930 msgr2=0x7f0de009a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0de00b3930 0x7f0de009a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 >> v1:172.21.15.159:6789/0 conn(0x7f0de00cc280 legacy=0x7f0de00cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de006bbe0 con 0x7f0de015db10 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f0dd4001670 con 0x7f0de015db10 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0de4e00700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0de00b3930 0x7f0de009a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 >> v1:172.21.15.12:6789/0 conn(0x7f0de015db10 legacy=0x7f0de0099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc8003bc0 0x7f0dc8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dc8004850 0x7f0dc8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0de00b3930 0x7f0dc8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de00b27a0 con 0x7f0dc8003bc0 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc800baf0 con 0x7f0dc8004850 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.967+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc800bcb0 con 0x7f0de00b3930 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.968+0000 7f0de5800700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc8003bc0 0x7f0dc8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.968+0000 7f0de4e00700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dc8004850 0x7f0dc8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.968+0000 7f0de6200700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0de00b3930 0x7f0dc8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.968+0000 7f0de5800700 1 -- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0de00b3930 msgr2=0x7f0dc8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.968+0000 7f0de5800700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0de00b3930 0x7f0dc8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.968+0000 7f0de5800700 1 -- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dc8004850 msgr2=0x7f0dc8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.968+0000 7f0de5800700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dc8004850 0x7f0dc8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.968+0000 7f0de5800700 1 -- 172.21.15.12:0/3793674407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc0094a0 con 0x7f0dc8003bc0 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.968+0000 7f0de4e00700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dc8004850 0x7f0dc8004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.968+0000 7f0de5800700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc8003bc0 0x7f0dc8003fc0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4001eb0 tx=0x7f0dc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf6e79df7e79cb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.969+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd400cd80 con 0x7f0dc8003bc0 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.969+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd4001af0 con 0x7f0dc8003bc0 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.969+0000 7f0ddfe00700 1 -- 172.21.15.12:0/3793674407 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4005ba0 con 0x7f0dc8003bc0 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.969+0000 7f0de8200700 1 -- 172.21.15.12:0/3793674407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc8003bc0 msgr2=0x7f0dc8003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.969+0000 7f0de8200700 1 --2- 172.21.15.12:0/3793674407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dc8003bc0 0x7f0dc8003fc0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4001eb0 tx=0x7f0dc8005880 comp rx=0 tx=0).stop 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.969+0000 7f0de6200700 1 -- 172.21.15.12:0/3793674407 reap_dead start 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.969+0000 7f0de8200700 1 -- 172.21.15.12:0/3793674407 shutdown_connections 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.969+0000 7f0de8200700 1 -- 172.21.15.12:0/3793674407 >> 172.21.15.12:0/3793674407 conn(0x7f0de00b2e10 msgr2=0x7f0de00b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.969+0000 7f0de8200700 1 -- 172.21.15.12:0/3793674407 shutdown_connections 2024-03-29T11:33:16.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.969+0000 7f0de8200700 1 -- 172.21.15.12:0/3793674407 wait complete. 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.971+0000 7f0de8200700 1 Processor -- start 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.971+0000 7f0de8200700 1 -- start start 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.972+0000 7f0de8200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc8003bc0 0x7f0de014c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.972+0000 7f0de8200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0de00c4e50 0x7f0de00c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.972+0000 7f0de8200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de00c52d0 0x7f0de00ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.972+0000 7f0de8200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de00c6410 con 0x7f0de00c52d0 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.972+0000 7f0de8200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de014d1c0 con 0x7f0de00c4e50 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.972+0000 7f0de8200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de00c58a0 con 0x7f0dc8003bc0 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.972+0000 7f0de6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de00c52d0 0x7f0de00ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0de6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de00c52d0 0x7f0de00ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54520/0 (socket says 172.21.15.12:54520) 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0de6200700 1 -- 172.21.15.12:0/2775897661 learned_addr learned my addr 172.21.15.12:0/2775897661 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0de4e00700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0de00c4e50 0x7f0de00c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0de5800700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc8003bc0 0x7f0de014c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0de6200700 1 -- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc8003bc0 msgr2=0x7f0de014c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0de6200700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc8003bc0 0x7f0de014c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0de6200700 1 -- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0de00c4e50 msgr2=0x7f0de00c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0de6200700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0de00c4e50 0x7f0de00c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0de6200700 1 -- 172.21.15.12:0/2775897661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de00c5b80 con 0x7f0de00c52d0 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0de6200700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de00c52d0 0x7f0de00ca2a0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f0de0161470 tx=0x7f0dd800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0ddd600700 1 -- 172.21.15.12:0/2775897661 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8003c10 con 0x7f0de00c52d0 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.973+0000 7f0ddd600700 1 -- 172.21.15.12:0/2775897661 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd8003d90 con 0x7f0de00c52d0 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.974+0000 7f0de8200700 1 -- 172.21.15.12:0/2775897661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de00c5e60 con 0x7f0de00c52d0 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.974+0000 7f0ddd600700 1 -- 172.21.15.12:0/2775897661 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8005310 con 0x7f0de00c52d0 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.974+0000 7f0de8200700 1 -- 172.21.15.12:0/2775897661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de014cc40 con 0x7f0de00c52d0 2024-03-29T11:33:16.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.975+0000 7f0de8200700 1 -- 172.21.15.12:0/2775897661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de00c68a0 con 0x7f0de00c52d0 2024-03-29T11:33:16.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.976+0000 7f0ddd600700 1 -- 172.21.15.12:0/2775897661 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0dd8005470 con 0x7f0de00c52d0 2024-03-29T11:33:16.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.976+0000 7f0ddd600700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0dc006aed0 0x7f0dc006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:16.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.977+0000 7f0ddd600700 1 -- 172.21.15.12:0/2775897661 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7505+0+0 (secure 0 0 0) 0x7f0dd8089f30 con 0x7f0de00c52d0 2024-03-29T11:33:16.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.977+0000 7f0de5800700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0dc006aed0 0x7f0dc006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:16.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.978+0000 7f0de5800700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0dc006aed0 0x7f0dc006d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40018b0 tx=0x7f0dd400a000 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:16.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:16.981+0000 7f0ddd600700 1 -- 172.21.15.12:0/2775897661 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0de00c68a0 con 0x7f0de00c52d0 2024-03-29T11:33:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.194+0000 7f0de8200700 1 -- 172.21.15.12:0/2775897661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0de0067720 con 0x7f0de00c52d0 2024-03-29T11:33:17.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.701+0000 7f0ddd600700 1 -- 172.21.15.12:0/2775897661 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v256) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0dd8058090 con 0x7f0de00c52d0 2024-03-29T11:33:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.705+0000 7f0de8200700 1 -- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0dc006aed0 msgr2=0x7f0dc006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.705+0000 7f0de8200700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0dc006aed0 0x7f0dc006d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40018b0 tx=0x7f0dd400a000 comp rx=0 tx=0).stop 2024-03-29T11:33:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.705+0000 7f0de8200700 1 -- 172.21.15.12:0/2775897661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de00c52d0 msgr2=0x7f0de00ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.705+0000 7f0de8200700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de00c52d0 0x7f0de00ca2a0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f0de0161470 tx=0x7f0dd800ef20 comp rx=0 tx=0).stop 2024-03-29T11:33:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.706+0000 7f0de8200700 1 -- 172.21.15.12:0/2775897661 shutdown_connections 2024-03-29T11:33:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.706+0000 7f0de8200700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0dc006aed0 0x7f0dc006d390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.706+0000 7f0de8200700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de00c52d0 0x7f0de00ca2a0 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.706+0000 7f0de8200700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dc8003bc0 0x7f0de014c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.706+0000 7f0de8200700 1 --2- 172.21.15.12:0/2775897661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0de00c4e50 0x7f0de00c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.706+0000 7f0de8200700 1 -- 172.21.15.12:0/2775897661 >> 172.21.15.12:0/2775897661 conn(0x7f0de00b2e10 msgr2=0x7f0de006c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.706+0000 7f0de8200700 1 -- 172.21.15.12:0/2775897661 shutdown_connections 2024-03-29T11:33:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.706+0000 7f0de8200700 1 -- 172.21.15.12:0/2775897661 wait complete. 2024-03-29T11:33:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T11:33:17.721 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T11:33:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.928+0000 7f2b9f000700 1 Processor -- start 2024-03-29T11:33:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.928+0000 7f2b9f000700 1 -- start start 2024-03-29T11:33:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.929+0000 7f2b9f000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2b9815ddf0 0x7f2b980ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.929+0000 7f2b9f000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2b9815e1d0 0x7f2b980d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.929+0000 7f2b9f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b980d2710 0x7f2b981612f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.929+0000 7f2b9f000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b980d2b80 con 0x7f2b9815e1d0 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.929+0000 7f2b9f000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9815ec60 con 0x7f2b9815ddf0 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.929+0000 7f2b9f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b980ce8a0 con 0x7f2b980d2710 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2b9815e1d0 0x7f2b980d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2b9815e1d0 0x7f2b980d21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54524/0 (socket says 172.21.15.12:54524) 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97400700 1 -- 172.21.15.12:0/1490562794 learned_addr learned my addr 172.21.15.12:0/1490562794 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b9d000700 1 --2- 172.21.15.12:0/1490562794 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b980d2710 0x7f2b981612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97e00700 1 --2- 172.21.15.12:0/1490562794 >> v2:172.21.15.159:3300/0 conn(0x7f2b9815ddf0 0x7f2b980ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97400700 1 -- 172.21.15.12:0/1490562794 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b980d2710 msgr2=0x7f2b981612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97400700 1 --2- 172.21.15.12:0/1490562794 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b980d2710 0x7f2b981612f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97400700 1 -- 172.21.15.12:0/1490562794 >> v2:172.21.15.159:3300/0 conn(0x7f2b9815ddf0 msgr2=0x7f2b980ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97400700 1 --2- 172.21.15.12:0/1490562794 >> v2:172.21.15.159:3300/0 conn(0x7f2b9815ddf0 0x7f2b980ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97400700 1 -- 172.21.15.12:0/1490562794 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b980cdc90 con 0x7f2b9815e1d0 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97e00700 1 --2- 172.21.15.12:0/1490562794 >> v2:172.21.15.159:3300/0 conn(0x7f2b9815ddf0 0x7f2b980ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.930+0000 7f2b97400700 1 --2- 172.21.15.12:0/1490562794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b9815e1d0 0x7f2b980d21d0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f2b9806b3e0 tx=0x7f2b84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf80079a5ee913c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b96a00700 1 -- 172.21.15.12:0/1490562794 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8400b870 con 0x7f2b9815e1d0 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b96a00700 1 -- 172.21.15.12:0/1490562794 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8400b9d0 con 0x7f2b9815e1d0 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b96a00700 1 -- 172.21.15.12:0/1490562794 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8400bd60 con 0x7f2b9815e1d0 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b9f000700 1 -- 172.21.15.12:0/1490562794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b9815e1d0 msgr2=0x7f2b980d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b9f000700 1 --2- 172.21.15.12:0/1490562794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b9815e1d0 0x7f2b980d21d0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f2b9806b3e0 tx=0x7f2b84009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b9f000700 1 -- 172.21.15.12:0/1490562794 shutdown_connections 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b9f000700 1 --2- 172.21.15.12:0/1490562794 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b980d2710 0x7f2b981612f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b9f000700 1 --2- 172.21.15.12:0/1490562794 >> v2:172.21.15.159:3300/0 conn(0x7f2b9815ddf0 0x7f2b980ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b9f000700 1 --2- 172.21.15.12:0/1490562794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b9815e1d0 0x7f2b980d21d0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b9f000700 1 -- 172.21.15.12:0/1490562794 >> 172.21.15.12:0/1490562794 conn(0x7f2b9808aff0 msgr2=0x7f2b980d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b9f000700 1 -- 172.21.15.12:0/1490562794 shutdown_connections 2024-03-29T11:33:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.931+0000 7f2b9f000700 1 -- 172.21.15.12:0/1490562794 wait complete. 2024-03-29T11:33:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.933+0000 7f2b9f000700 1 Processor -- start 2024-03-29T11:33:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.933+0000 7f2b9f000700 1 -- start start 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.934+0000 7f2b9f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b980d2710 0x7f2b9814c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.934+0000 7f2b9f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b9815ddf0 0x7f2b980c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.934+0000 7f2b9f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b9815e1d0 0x7f2b980c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.934+0000 7f2b9f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b980c87a0 con 0x7f2b980d2710 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.934+0000 7f2b9f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9814ccf0 con 0x7f2b9815ddf0 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.934+0000 7f2b9f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b980c7b90 con 0x7f2b9815e1d0 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.934+0000 7f2b97e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b980d2710 0x7f2b9814c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.934+0000 7f2b97e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b980d2710 0x7f2b9814c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54536/0 (socket says 172.21.15.12:54536) 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.934+0000 7f2b97e00700 1 -- 172.21.15.12:0/3105665728 learned_addr learned my addr 172.21.15.12:0/3105665728 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.935+0000 7f2b97e00700 1 -- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b9815e1d0 msgr2=0x7f2b980c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.935+0000 7f2b97400700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b9815ddf0 0x7f2b980c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.935+0000 7f2b9d000700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b9815e1d0 0x7f2b980c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.935+0000 7f2b97e00700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b9815e1d0 0x7f2b980c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.935+0000 7f2b97e00700 1 -- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b9815ddf0 msgr2=0x7f2b980c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.935+0000 7f2b97e00700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b9815ddf0 0x7f2b980c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.935+0000 7f2b97e00700 1 -- 172.21.15.12:0/3105665728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b84009550 con 0x7f2b980d2710 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.935+0000 7f2b97400700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b9815ddf0 0x7f2b980c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.935+0000 7f2b9d000700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b9815e1d0 0x7f2b980c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.935+0000 7f2b97e00700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b980d2710 0x7f2b9814c3d0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800c750 tx=0x7f2b8800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.936+0000 7f2b8fe00700 1 -- 172.21.15.12:0/3105665728 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88011800 con 0x7f2b980d2710 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.936+0000 7f2b8fe00700 1 -- 172.21.15.12:0/3105665728 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b88011960 con 0x7f2b980d2710 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.936+0000 7f2b9f000700 1 -- 172.21.15.12:0/3105665728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b980c7e70 con 0x7f2b980d2710 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.936+0000 7f2b8fe00700 1 -- 172.21.15.12:0/3105665728 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88011cd0 con 0x7f2b980d2710 2024-03-29T11:33:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.936+0000 7f2b9f000700 1 -- 172.21.15.12:0/3105665728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b980c8430 con 0x7f2b980d2710 2024-03-29T11:33:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.937+0000 7f2b9f000700 1 -- 172.21.15.12:0/3105665728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b980c8910 con 0x7f2b980d2710 2024-03-29T11:33:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.938+0000 7f2b8fe00700 1 -- 172.21.15.12:0/3105665728 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2b88021350 con 0x7f2b980d2710 2024-03-29T11:33:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.938+0000 7f2b8fe00700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2b7806aed0 0x7f2b7806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.938+0000 7f2b8fe00700 1 -- 172.21.15.12:0/3105665728 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7067+0+0 (secure 0 0 0) 0x7f2b88089d00 con 0x7f2b980d2710 2024-03-29T11:33:17.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.939+0000 7f2b97400700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2b7806aed0 0x7f2b7806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:17.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.940+0000 7f2b97400700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2b7806aed0 0x7f2b7806d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2b980a13d0 tx=0x7f2b84008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:17.943+0000 7f2b8fe00700 1 -- 172.21.15.12:0/3105665728 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2b88054520 con 0x7f2b980d2710 2024-03-29T11:33:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.157+0000 7f2b9f000700 1 -- 172.21.15.12:0/3105665728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2b98067720 con 0x7f2b980d2710 2024-03-29T11:33:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.703+0000 7f2b8fe00700 1 -- 172.21.15.12:0/3105665728 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v257) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2b88057b40 con 0x7f2b980d2710 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.707+0000 7f2b9f000700 1 -- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2b7806aed0 msgr2=0x7f2b7806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.707+0000 7f2b9f000700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2b7806aed0 0x7f2b7806d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2b980a13d0 tx=0x7f2b84008030 comp rx=0 tx=0).stop 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.707+0000 7f2b9f000700 1 -- 172.21.15.12:0/3105665728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b980d2710 msgr2=0x7f2b9814c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.707+0000 7f2b9f000700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b980d2710 0x7f2b9814c3d0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800c750 tx=0x7f2b8800cb10 comp rx=0 tx=0).stop 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.707+0000 7f2b9f000700 1 -- 172.21.15.12:0/3105665728 shutdown_connections 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.707+0000 7f2b9f000700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2b7806aed0 0x7f2b7806d390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.707+0000 7f2b9f000700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b9815e1d0 0x7f2b980c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.707+0000 7f2b9f000700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b980d2710 0x7f2b9814c3d0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.707+0000 7f2b9f000700 1 --2- 172.21.15.12:0/3105665728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b9815ddf0 0x7f2b980c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.707+0000 7f2b9f000700 1 -- 172.21.15.12:0/3105665728 >> 172.21.15.12:0/3105665728 conn(0x7f2b9808aff0 msgr2=0x7f2b980b1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.708+0000 7f2b9f000700 1 -- 172.21.15.12:0/3105665728 shutdown_connections 2024-03-29T11:33:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.708+0000 7f2b9f000700 1 -- 172.21.15.12:0/3105665728 wait complete. 2024-03-29T11:33:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T11:33:18.727 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.942+0000 7fdf9e800700 1 Processor -- start 2024-03-29T11:33:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.942+0000 7fdf9e800700 1 -- start start 2024-03-29T11:33:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.943+0000 7fdf9e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdf98160f10 0x7fdf980d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.943+0000 7fdf9e800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf980cf230 0x7fdf980d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.943+0000 7fdf9e800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf980ceda0 con 0x7fdf98161590 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.943+0000 7fdf9e800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf980d23d0 con 0x7fdf98160f10 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.943+0000 7fdf9e800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf981612f0 con 0x7fdf980cf230 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf97400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdf98160f10 0x7fdf980d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf97400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdf98160f10 0x7fdf980d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55980/0 (socket says 172.21.15.12:55980) 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf97400700 1 -- 172.21.15.12:0/554450348 learned_addr learned my addr 172.21.15.12:0/554450348 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf97400700 1 -- 172.21.15.12:0/554450348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf980cf230 msgr2=0x7fdf980d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf97400700 1 --2- 172.21.15.12:0/554450348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf980cf230 0x7fdf980d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf97400700 1 -- 172.21.15.12:0/554450348 >> v1:172.21.15.159:6789/0 conn(0x7fdf98161590 legacy=0x7fdf980b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf97400700 1 -- 172.21.15.12:0/554450348 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf980ce470 con 0x7fdf98160f10 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf97400700 1 --2- 172.21.15.12:0/554450348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98160f10 0x7fdf980d18b0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fdf880098f0 tx=0x7fdf88009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5caa1a3bd499aefe server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf96000700 1 -- 172.21.15.12:0/554450348 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf88004230 con 0x7fdf98160f10 2024-03-29T11:33:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf96000700 1 -- 172.21.15.12:0/554450348 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf88004390 con 0x7fdf98160f10 2024-03-29T11:33:18.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.944+0000 7fdf96000700 1 -- 172.21.15.12:0/554450348 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf880036a0 con 0x7fdf98160f10 2024-03-29T11:33:18.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.945+0000 7fdf9e800700 1 -- 172.21.15.12:0/554450348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98160f10 msgr2=0x7fdf980d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:18.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.945+0000 7fdf9e800700 1 --2- 172.21.15.12:0/554450348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98160f10 0x7fdf980d18b0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fdf880098f0 tx=0x7fdf88009c00 comp rx=0 tx=0).stop 2024-03-29T11:33:18.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.945+0000 7fdf9e800700 1 -- 172.21.15.12:0/554450348 shutdown_connections 2024-03-29T11:33:18.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.945+0000 7fdf9e800700 1 --2- 172.21.15.12:0/554450348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98160f10 0x7fdf980d18b0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:18.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.945+0000 7fdf9e800700 1 --2- 172.21.15.12:0/554450348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf980cf230 0x7fdf980d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:18.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.945+0000 7fdf9e800700 1 -- 172.21.15.12:0/554450348 >> 172.21.15.12:0/554450348 conn(0x7fdf9808aff0 msgr2=0x7fdf980d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:18.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.945+0000 7fdf9e800700 1 -- 172.21.15.12:0/554450348 shutdown_connections 2024-03-29T11:33:18.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.945+0000 7fdf9e800700 1 -- 172.21.15.12:0/554450348 wait complete. 2024-03-29T11:33:18.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.947+0000 7fdf9e800700 1 Processor -- start 2024-03-29T11:33:18.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.947+0000 7fdf9e800700 1 -- start start 2024-03-29T11:33:18.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf9e800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf980cf230 0x7fdf980b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:18.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf9e800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf98160f10 0x7fdf980a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:18.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf9e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98161590 0x7fdf980a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:18.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf9e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9809f240 con 0x7fdf98161590 2024-03-29T11:33:18.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf9e800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9815b570 con 0x7fdf98160f10 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf9e800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf980b2630 con 0x7fdf980cf230 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf97e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98161590 0x7fdf980a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf97e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98161590 0x7fdf980a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55988/0 (socket says 172.21.15.12:55988) 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf97e00700 1 -- 172.21.15.12:0/2923159289 learned_addr learned my addr 172.21.15.12:0/2923159289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf97400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf980cf230 0x7fdf980b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.948+0000 7fdf96a00700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf98160f10 0x7fdf980a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf97e00700 1 -- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf980cf230 msgr2=0x7fdf980b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf97e00700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf980cf230 0x7fdf980b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf97e00700 1 -- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf98160f10 msgr2=0x7fdf980a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf97e00700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf98160f10 0x7fdf980a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf97e00700 1 -- 172.21.15.12:0/2923159289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf88009550 con 0x7fdf98161590 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf97e00700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98161590 0x7fdf980a0e80 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000b990 tx=0x7fdf9000bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf8f400700 1 -- 172.21.15.12:0/2923159289 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9000d890 con 0x7fdf98161590 2024-03-29T11:33:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf8f400700 1 -- 172.21.15.12:0/2923159289 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9000d9f0 con 0x7fdf98161590 2024-03-29T11:33:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf8f400700 1 -- 172.21.15.12:0/2923159289 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9000dda0 con 0x7fdf98161590 2024-03-29T11:33:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf9e800700 1 -- 172.21.15.12:0/2923159289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9809e5c0 con 0x7fdf98161590 2024-03-29T11:33:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.949+0000 7fdf9e800700 1 -- 172.21.15.12:0/2923159289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9809eb80 con 0x7fdf98161590 2024-03-29T11:33:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.950+0000 7fdf9e800700 1 -- 172.21.15.12:0/2923159289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf980d0b30 con 0x7fdf98161590 2024-03-29T11:33:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.951+0000 7fdf8f400700 1 -- 172.21.15.12:0/2923159289 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fdf90006d50 con 0x7fdf98161590 2024-03-29T11:33:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.952+0000 7fdf8f400700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf7806aed0 0x7fdf7806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.952+0000 7fdf8f400700 1 -- 172.21.15.12:0/2923159289 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6653+0+0 (secure 0 0 0) 0x7fdf90089ab0 con 0x7fdf98161590 2024-03-29T11:33:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.953+0000 7fdf97400700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf7806aed0 0x7fdf7806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:18.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.953+0000 7fdf97400700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf7806aed0 0x7fdf7806d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009130 tx=0x7fdf88008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:18.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:18.957+0000 7fdf8f400700 1 -- 172.21.15.12:0/2923159289 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf90054500 con 0x7fdf98161590 2024-03-29T11:33:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.171+0000 7fdf9e800700 1 -- 172.21.15.12:0/2923159289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf98067720 con 0x7fdf98161590 2024-03-29T11:33:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.173+0000 7fdf8f400700 1 -- 172.21.15.12:0/2923159289 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+17434 (secure 0 0 0) 0x7fdf9001c090 con 0x7fdf98161590 2024-03-29T11:33:19.170 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:19.170 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":257,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:18.697180+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:32:45.400233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:32:46.836046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6834/3711953177":"2024-03-30T11:33:15.669567+0000","172.21.15.159:0/117966312":"2024-03-30T11:33:12.058935+0000","172.21.15.12:6832/3711953177":"2024-03-30T11:33:15.669567+0000","172.21.15.159:0/1298574278":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/449301407":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/2167892315":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/122983726":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/2916339927":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/962755797":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1966802084":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/737492091":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/110096794":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/449562747":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1567850739":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/3214276897":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/180562283":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/2535009377":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1057841974":"2024-03-30T11:33:03.088269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.175+0000 7fdf9e800700 1 -- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf7806aed0 msgr2=0x7fdf7806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.175+0000 7fdf9e800700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf7806aed0 0x7fdf7806d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009130 tx=0x7fdf88008030 comp rx=0 tx=0).stop 2024-03-29T11:33:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.176+0000 7fdf9e800700 1 -- 172.21.15.12:0/2923159289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98161590 msgr2=0x7fdf980a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.176+0000 7fdf9e800700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98161590 0x7fdf980a0e80 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000b990 tx=0x7fdf9000bca0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.176+0000 7fdf9e800700 1 -- 172.21.15.12:0/2923159289 shutdown_connections 2024-03-29T11:33:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.176+0000 7fdf9e800700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdf7806aed0 0x7fdf7806d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.176+0000 7fdf9e800700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf98161590 0x7fdf980a0e80 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.176+0000 7fdf9e800700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf980cf230 0x7fdf980b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.176+0000 7fdf9e800700 1 --2- 172.21.15.12:0/2923159289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf98160f10 0x7fdf980a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.176+0000 7fdf9e800700 1 -- 172.21.15.12:0/2923159289 >> 172.21.15.12:0/2923159289 conn(0x7fdf9808aff0 msgr2=0x7fdf9815b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.176+0000 7fdf9e800700 1 -- 172.21.15.12:0/2923159289 shutdown_connections 2024-03-29T11:33:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.176+0000 7fdf9e800700 1 -- 172.21.15.12:0/2923159289 wait complete. 2024-03-29T11:33:19.190 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-29T11:32:45.400233+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-29T11:32:46.836046+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:33:19.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:33:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.397+0000 7ff1d7e00700 1 Processor -- start 2024-03-29T11:33:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.397+0000 7ff1d7e00700 1 -- start start 2024-03-29T11:33:19.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.398+0000 7ff1d7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff1d01611b0 0x7ff1d015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.398+0000 7ff1d7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1d00cee10 0x7ff1d0161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.398+0000 7ff1d7e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d00ce270 con 0x7ff1d015f4d0 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.398+0000 7ff1d7e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d015b600 con 0x7ff1d01611b0 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.398+0000 7ff1d7e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d00d2760 con 0x7ff1d00cee10 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.398+0000 7ff1d5400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff1d01611b0 0x7ff1d015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.398+0000 7ff1d5400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff1d01611b0 0x7ff1d015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55996/0 (socket says 172.21.15.12:55996) 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.398+0000 7ff1d5400700 1 -- 172.21.15.12:0/1588415156 learned_addr learned my addr 172.21.15.12:0/1588415156 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.399+0000 7ff1d5e00700 1 --2- 172.21.15.12:0/1588415156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1d00cee10 0x7ff1d0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.399+0000 7ff1d5400700 1 -- 172.21.15.12:0/1588415156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1d00cee10 msgr2=0x7ff1d0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.399+0000 7ff1d5400700 1 --2- 172.21.15.12:0/1588415156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1d00cee10 0x7ff1d0161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.399+0000 7ff1d5400700 1 -- 172.21.15.12:0/1588415156 >> v1:172.21.15.159:6789/0 conn(0x7ff1d015f4d0 legacy=0x7ff1d00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.399+0000 7ff1d5400700 1 -- 172.21.15.12:0/1588415156 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d00d2a40 con 0x7ff1d01611b0 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.399+0000 7ff1d5400700 1 --2- 172.21.15.12:0/1588415156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d01611b0 0x7ff1d015ef90 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40098f0 tx=0x7ff1c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a938e42d073d5b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.399+0000 7ff1cfe00700 1 -- 172.21.15.12:0/1588415156 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c400b8a0 con 0x7ff1d01611b0 2024-03-29T11:33:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.399+0000 7ff1cfe00700 1 -- 172.21.15.12:0/1588415156 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c400ba00 con 0x7ff1d01611b0 2024-03-29T11:33:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.399+0000 7ff1cfe00700 1 -- 172.21.15.12:0/1588415156 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c400bd90 con 0x7ff1d01611b0 2024-03-29T11:33:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.400+0000 7ff1d7e00700 1 -- 172.21.15.12:0/1588415156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d01611b0 msgr2=0x7ff1d015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.400+0000 7ff1d7e00700 1 --2- 172.21.15.12:0/1588415156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d01611b0 0x7ff1d015ef90 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40098f0 tx=0x7ff1c4009c00 comp rx=0 tx=0).stop 2024-03-29T11:33:19.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.400+0000 7ff1d7e00700 1 -- 172.21.15.12:0/1588415156 shutdown_connections 2024-03-29T11:33:19.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.400+0000 7ff1d7e00700 1 --2- 172.21.15.12:0/1588415156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1d00cee10 0x7ff1d0161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.400+0000 7ff1d7e00700 1 --2- 172.21.15.12:0/1588415156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d01611b0 0x7ff1d015ef90 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.400+0000 7ff1d7e00700 1 -- 172.21.15.12:0/1588415156 >> 172.21.15.12:0/1588415156 conn(0x7ff1d008aff0 msgr2=0x7ff1d00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:19.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.400+0000 7ff1d7e00700 1 -- 172.21.15.12:0/1588415156 shutdown_connections 2024-03-29T11:33:19.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.400+0000 7ff1d7e00700 1 -- 172.21.15.12:0/1588415156 wait complete. 2024-03-29T11:33:19.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.402+0000 7ff1d7e00700 1 Processor -- start 2024-03-29T11:33:19.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.402+0000 7ff1d7e00700 1 -- start start 2024-03-29T11:33:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.403+0000 7ff1d7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d00cee10 0x7ff1d015ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.403+0000 7ff1d7e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1d015f4d0 0x7ff1d009d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.403+0000 7ff1d7e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1d01611b0 0x7ff1d009e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.403+0000 7ff1d7e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d00ca9c0 con 0x7ff1d00cee10 2024-03-29T11:33:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.403+0000 7ff1d7e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d015b060 con 0x7ff1d01611b0 2024-03-29T11:33:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.403+0000 7ff1d7e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d00ca600 con 0x7ff1d015f4d0 2024-03-29T11:33:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.403+0000 7ff1d5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d00cee10 0x7ff1d015ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.403+0000 7ff1d5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d00cee10 0x7ff1d015ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56004/0 (socket says 172.21.15.12:56004) 2024-03-29T11:33:19.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.403+0000 7ff1d5400700 1 -- 172.21.15.12:0/2090035557 learned_addr learned my addr 172.21.15.12:0/2090035557 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:19.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.403+0000 7ff1d4a00700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1d015f4d0 0x7ff1d009d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.404+0000 7ff1d5e00700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1d01611b0 0x7ff1d009e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.404+0000 7ff1d5400700 1 -- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1d015f4d0 msgr2=0x7ff1d009d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.404+0000 7ff1d5400700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1d015f4d0 0x7ff1d009d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.404+0000 7ff1d5400700 1 -- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1d01611b0 msgr2=0x7ff1d009e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.404+0000 7ff1d5400700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1d01611b0 0x7ff1d009e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.404+0000 7ff1d5400700 1 -- 172.21.15.12:0/2090035557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c4009550 con 0x7ff1d00cee10 2024-03-29T11:33:19.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.404+0000 7ff1d5400700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d00cee10 0x7ff1d015ab20 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009c40 tx=0x7ff1c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:19.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.404+0000 7ff1cd600700 1 -- 172.21.15.12:0/2090035557 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40036a0 con 0x7ff1d00cee10 2024-03-29T11:33:19.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.405+0000 7ff1cd600700 1 -- 172.21.15.12:0/2090035557 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c401e070 con 0x7ff1d00cee10 2024-03-29T11:33:19.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.405+0000 7ff1d7e00700 1 -- 172.21.15.12:0/2090035557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d009db90 con 0x7ff1d00cee10 2024-03-29T11:33:19.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.405+0000 7ff1cd600700 1 -- 172.21.15.12:0/2090035557 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4003a60 con 0x7ff1d00cee10 2024-03-29T11:33:19.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.405+0000 7ff1d7e00700 1 -- 172.21.15.12:0/2090035557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d009e150 con 0x7ff1d00cee10 2024-03-29T11:33:19.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.406+0000 7ff1d7e00700 1 -- 172.21.15.12:0/2090035557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d0162150 con 0x7ff1d00cee10 2024-03-29T11:33:19.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.407+0000 7ff1cd600700 1 -- 172.21.15.12:0/2090035557 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff1c4003bc0 con 0x7ff1d00cee10 2024-03-29T11:33:19.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.407+0000 7ff1cd600700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff1b006aed0 0x7ff1b006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.408+0000 7ff1cd600700 1 -- 172.21.15.12:0/2090035557 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6653+0+0 (secure 0 0 0) 0x7ff1c4020070 con 0x7ff1d00cee10 2024-03-29T11:33:19.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.408+0000 7ff1d4a00700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff1b006aed0 0x7ff1b006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.409+0000 7ff1d4a00700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff1b006aed0 0x7ff1b006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff1d01f7750 tx=0x7ff1bc009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:19.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.412+0000 7ff1cd600700 1 -- 172.21.15.12:0/2090035557 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff1c4055b70 con 0x7ff1d00cee10 2024-03-29T11:33:19.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.629+0000 7ff1d7e00700 1 -- 172.21.15.12:0/2090035557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff1d0067720 con 0x7ff1d00cee10 2024-03-29T11:33:19.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.630+0000 7ff1cd600700 1 -- 172.21.15.12:0/2090035557 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v257) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff1c4015020 con 0x7ff1d00cee10 2024-03-29T11:33:19.626 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:33:19.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.632+0000 7ff1d7e00700 1 -- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff1b006aed0 msgr2=0x7ff1b006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.632+0000 7ff1d7e00700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff1b006aed0 0x7ff1b006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff1d01f7750 tx=0x7ff1bc009040 comp rx=0 tx=0).stop 2024-03-29T11:33:19.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.632+0000 7ff1d7e00700 1 -- 172.21.15.12:0/2090035557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d00cee10 msgr2=0x7ff1d015ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.632+0000 7ff1d7e00700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d00cee10 0x7ff1d015ab20 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009c40 tx=0x7ff1c4004750 comp rx=0 tx=0).stop 2024-03-29T11:33:19.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.633+0000 7ff1d7e00700 1 -- 172.21.15.12:0/2090035557 shutdown_connections 2024-03-29T11:33:19.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.633+0000 7ff1d7e00700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff1b006aed0 0x7ff1b006d390 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.633+0000 7ff1d7e00700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1d01611b0 0x7ff1d009e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.633+0000 7ff1d7e00700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1d00cee10 0x7ff1d015ab20 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.633+0000 7ff1d7e00700 1 --2- 172.21.15.12:0/2090035557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1d015f4d0 0x7ff1d009d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.633+0000 7ff1d7e00700 1 -- 172.21.15.12:0/2090035557 >> 172.21.15.12:0/2090035557 conn(0x7ff1d008aff0 msgr2=0x7ff1d00b8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:19.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.633+0000 7ff1d7e00700 1 -- 172.21.15.12:0/2090035557 shutdown_connections 2024-03-29T11:33:19.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.633+0000 7ff1d7e00700 1 -- 172.21.15.12:0/2090035557 wait complete. 2024-03-29T11:33:19.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-29T11:33:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.851+0000 7fcce3e00700 1 Processor -- start 2024-03-29T11:33:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.851+0000 7fcce3e00700 1 -- start start 2024-03-29T11:33:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.852+0000 7fcce3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc160e50 0x7fccdc0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.852+0000 7fcce3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fccdc0d3f20 0x7fccdc0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.852+0000 7fcce3e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc0cef00 con 0x7fccdc0b3f80 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.852+0000 7fcce3e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccdc161230 con 0x7fccdc0d3f20 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.852+0000 7fcce3e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0d2a80 con 0x7fccdc160e50 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.853+0000 7fcce0a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fccdc0d3f20 0x7fccdc0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.853+0000 7fcce0a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fccdc0d3f20 0x7fccdc0b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56016/0 (socket says 172.21.15.12:56016) 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.853+0000 7fcce0a00700 1 -- 172.21.15.12:0/3391744447 learned_addr learned my addr 172.21.15.12:0/3391744447 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.853+0000 7fcce1400700 1 --2- 172.21.15.12:0/3391744447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc160e50 0x7fccdc0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.853+0000 7fcce0a00700 1 -- 172.21.15.12:0/3391744447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc160e50 msgr2=0x7fccdc0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.853+0000 7fcce0a00700 1 --2- 172.21.15.12:0/3391744447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc160e50 0x7fccdc0b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.853+0000 7fcce0a00700 1 -- 172.21.15.12:0/3391744447 >> v1:172.21.15.159:6789/0 conn(0x7fccdc0b3f80 legacy=0x7fccdc0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.853+0000 7fcce0a00700 1 -- 172.21.15.12:0/3391744447 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc0d43b0 con 0x7fccdc0d3f20 2024-03-29T11:33:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.853+0000 7fcce0a00700 1 --2- 172.21.15.12:0/3391744447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc0d3f20 0x7fccdc0b53d0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7fccc80098f0 tx=0x7fccc8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741b634893ba8bcc server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fccdbe00700 1 -- 172.21.15.12:0/3391744447 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8011830 con 0x7fccdc0d3f20 2024-03-29T11:33:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fccdbe00700 1 -- 172.21.15.12:0/3391744447 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc8011990 con 0x7fccdc0d3f20 2024-03-29T11:33:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fccdbe00700 1 -- 172.21.15.12:0/3391744447 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8011d20 con 0x7fccdc0d3f20 2024-03-29T11:33:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fcce3e00700 1 -- 172.21.15.12:0/3391744447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc0d3f20 msgr2=0x7fccdc0b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fcce3e00700 1 --2- 172.21.15.12:0/3391744447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc0d3f20 0x7fccdc0b53d0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7fccc80098f0 tx=0x7fccc8009c00 comp rx=0 tx=0).stop 2024-03-29T11:33:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fcce3e00700 1 -- 172.21.15.12:0/3391744447 shutdown_connections 2024-03-29T11:33:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fcce3e00700 1 --2- 172.21.15.12:0/3391744447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc160e50 0x7fccdc0b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fcce3e00700 1 --2- 172.21.15.12:0/3391744447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc0d3f20 0x7fccdc0b53d0 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fcce3e00700 1 -- 172.21.15.12:0/3391744447 >> 172.21.15.12:0/3391744447 conn(0x7fccdc08aff0 msgr2=0x7fccdc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fcce3e00700 1 -- 172.21.15.12:0/3391744447 shutdown_connections 2024-03-29T11:33:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.854+0000 7fcce3e00700 1 -- 172.21.15.12:0/3391744447 wait complete. 2024-03-29T11:33:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.856+0000 7fcce3e00700 1 Processor -- start 2024-03-29T11:33:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.856+0000 7fcce3e00700 1 -- start start 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc0b3f80 0x7fccdc0c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc160e50 0x7fccdc15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce3e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccdc0b8170 0x7fccdc0ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0caad0 con 0x7fccdc160e50 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce3e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0ca780 con 0x7fccdc0b8170 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce3e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0b86d0 con 0x7fccdc0b3f80 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc160e50 0x7fccdc15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc160e50 0x7fccdc15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56018/0 (socket says 172.21.15.12:56018) 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce0a00700 1 -- 172.21.15.12:0/215124804 learned_addr learned my addr 172.21.15.12:0/215124804 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce1400700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc0b3f80 0x7fccdc0c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.857+0000 7fcce1e00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccdc0b8170 0x7fccdc0ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fcce0a00700 1 -- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc0b3f80 msgr2=0x7fccdc0c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fcce0a00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc0b3f80 0x7fccdc0c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fcce0a00700 1 -- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccdc0b8170 msgr2=0x7fccdc0ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fcce0a00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccdc0b8170 0x7fccdc0ccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fcce0a00700 1 -- 172.21.15.12:0/215124804 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8009550 con 0x7fccdc160e50 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fcce1400700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc0b3f80 0x7fccdc0c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fcce1e00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccdc0b8170 0x7fccdc0ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fcce0a00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc160e50 0x7fccdc15a9b0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009c40 tx=0x7fccc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fccd9600700 1 -- 172.21.15.12:0/215124804 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc80036a0 con 0x7fccdc160e50 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fccd9600700 1 -- 172.21.15.12:0/215124804 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc801e070 con 0x7fccdc160e50 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fcce3e00700 1 -- 172.21.15.12:0/215124804 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc0b89b0 con 0x7fccdc160e50 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fccd9600700 1 -- 172.21.15.12:0/215124804 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8003a60 con 0x7fccdc160e50 2024-03-29T11:33:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.858+0000 7fcce3e00700 1 -- 172.21.15.12:0/215124804 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc15aef0 con 0x7fccdc160e50 2024-03-29T11:33:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.859+0000 7fcce3e00700 1 -- 172.21.15.12:0/215124804 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc0cac40 con 0x7fccdc160e50 2024-03-29T11:33:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.860+0000 7fccd9600700 1 -- 172.21.15.12:0/215124804 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fccc8003bc0 con 0x7fccdc160e50 2024-03-29T11:33:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.861+0000 7fccd9600700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fccbc06aed0 0x7fccbc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.861+0000 7fccd9600700 1 -- 172.21.15.12:0/215124804 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6653+0+0 (secure 0 0 0) 0x7fccc808a9e0 con 0x7fccdc160e50 2024-03-29T11:33:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.862+0000 7fcce1400700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fccbc06aed0 0x7fccbc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.863+0000 7fcce1400700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fccbc06aed0 0x7fccbc06d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009c30 tx=0x7fccd0008040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:19.866+0000 7fccd9600700 1 -- 172.21.15.12:0/215124804 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fccc8055570 con 0x7fccdc160e50 2024-03-29T11:33:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.086+0000 7fcce3e00700 1 -- 172.21.15.12:0/215124804 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fccdc0b8bc0 con 0x7fccdc160e50 2024-03-29T11:33:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.086+0000 7fccd9600700 1 -- 172.21.15.12:0/215124804 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v257) v1 ==== 107+0+11 (secure 0 0 0) 0x7fccc8058b90 con 0x7fccdc160e50 2024-03-29T11:33:20.083 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:33:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 -- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fccbc06aed0 msgr2=0x7fccbc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fccbc06aed0 0x7fccbc06d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009c30 tx=0x7fccd0008040 comp rx=0 tx=0).stop 2024-03-29T11:33:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 -- 172.21.15.12:0/215124804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc160e50 msgr2=0x7fccdc15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc160e50 0x7fccdc15a9b0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009c40 tx=0x7fccc8004750 comp rx=0 tx=0).stop 2024-03-29T11:33:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 -- 172.21.15.12:0/215124804 shutdown_connections 2024-03-29T11:33:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fccbc06aed0 0x7fccbc06d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccdc0b8170 0x7fccdc0ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccdc0b3f80 0x7fccdc0c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 --2- 172.21.15.12:0/215124804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccdc160e50 0x7fccdc15a9b0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 -- 172.21.15.12:0/215124804 >> 172.21.15.12:0/215124804 conn(0x7fccdc08aff0 msgr2=0x7fccdc0d3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 -- 172.21.15.12:0/215124804 shutdown_connections 2024-03-29T11:33:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.089+0000 7fcce3e00700 1 -- 172.21.15.12:0/215124804 wait complete. 2024-03-29T11:33:20.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-29T11:33:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.307+0000 7f214d000700 1 Processor -- start 2024-03-29T11:33:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.307+0000 7f214d000700 1 -- start start 2024-03-29T11:33:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.308+0000 7f214d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cd020 0x7f21480cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.308+0000 7f214d000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f21480b2070 0x7f21480b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.308+0000 7f214d000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21480cdae0 con 0x7f21480cbb30 2024-03-29T11:33:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.308+0000 7f214d000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21480cddc0 con 0x7f21480b2070 2024-03-29T11:33:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.308+0000 7f214d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21480ce0a0 con 0x7f21480cd020 2024-03-29T11:33:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.309+0000 7f2145400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cd020 0x7f21480cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.309+0000 7f2146800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f21480b2070 0x7f21480b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.309+0000 7f2145e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f21480cbb30 0x7f21480cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:41188/0 (socket says 172.21.15.12:41188) 2024-03-29T11:33:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.309+0000 7f2145e00700 1 -- 172.21.15.12:0/160371311 learned_addr learned my addr 172.21.15.12:0/160371311 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.310+0000 7f2145400700 1 -- 172.21.15.12:0/160371311 >> v2:172.21.15.159:3300/0 conn(0x7f21480b2070 msgr2=0x7f21480b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.310+0000 7f2145400700 1 --2- 172.21.15.12:0/160371311 >> v2:172.21.15.159:3300/0 conn(0x7f21480b2070 0x7f21480b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.310+0000 7f2145400700 1 -- 172.21.15.12:0/160371311 >> v1:172.21.15.159:6789/0 conn(0x7f21480cbb30 legacy=0x7f21480cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.310+0000 7f2145400700 1 -- 172.21.15.12:0/160371311 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214806dc20 con 0x7f21480cd020 2024-03-29T11:33:20.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.310+0000 7f2146800700 1 --2- 172.21.15.12:0/160371311 >> v2:172.21.15.159:3300/0 conn(0x7f21480b2070 0x7f21480b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:20.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.310+0000 7f2144a00700 1 -- 172.21.15.12:0/160371311 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674316247 0 0) 0x7f21480cdae0 con 0x7f21480cbb30 2024-03-29T11:33:20.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.310+0000 7f2145400700 1 --2- 172.21.15.12:0/160371311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cd020 0x7f21480cd480 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2134009840 tx=0x7f2134009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1c70a231dbb8a40 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:20.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.310+0000 7f2144a00700 1 -- 172.21.15.12:0/160371311 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134011830 con 0x7f21480cd020 2024-03-29T11:33:20.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.310+0000 7f2144a00700 1 -- 172.21.15.12:0/160371311 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2134011990 con 0x7f21480cd020 2024-03-29T11:33:20.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.310+0000 7f2144a00700 1 -- 172.21.15.12:0/160371311 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134011d20 con 0x7f21480cd020 2024-03-29T11:33:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.311+0000 7f214d000700 1 -- 172.21.15.12:0/160371311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cd020 msgr2=0x7f21480cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.311+0000 7f214d000700 1 --2- 172.21.15.12:0/160371311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cd020 0x7f21480cd480 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2134009840 tx=0x7f2134009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.311+0000 7f214d000700 1 -- 172.21.15.12:0/160371311 shutdown_connections 2024-03-29T11:33:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.311+0000 7f214d000700 1 --2- 172.21.15.12:0/160371311 >> v2:172.21.15.159:3300/0 conn(0x7f21480b2070 0x7f21480b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.311+0000 7f214d000700 1 --2- 172.21.15.12:0/160371311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cd020 0x7f21480cd480 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.311+0000 7f214d000700 1 -- 172.21.15.12:0/160371311 >> 172.21.15.12:0/160371311 conn(0x7f21480b0d70 msgr2=0x7f21480b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.311+0000 7f214d000700 1 -- 172.21.15.12:0/160371311 shutdown_connections 2024-03-29T11:33:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.311+0000 7f214d000700 1 -- 172.21.15.12:0/160371311 wait complete. 2024-03-29T11:33:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.313+0000 7f214d000700 1 Processor -- start 2024-03-29T11:33:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.313+0000 7f214d000700 1 -- start start 2024-03-29T11:33:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f214d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21480b2070 0x7f2148098920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f214d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cbb30 0x7f21480ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f214d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21480cd020 0x7f2148142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f214d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480ada00 con 0x7f21480b2070 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f214d000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480acff0 con 0x7f21480cd020 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f214d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21480acd10 con 0x7f21480cbb30 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f2145e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21480b2070 0x7f2148098920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f2145e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21480b2070 0x7f2148098920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56026/0 (socket says 172.21.15.12:56026) 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f2145e00700 1 -- 172.21.15.12:0/1781890927 learned_addr learned my addr 172.21.15.12:0/1781890927 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f2145400700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cbb30 0x7f21480ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.314+0000 7f2146800700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21480cd020 0x7f2148142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f2145e00700 1 -- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cbb30 msgr2=0x7f21480ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f2145e00700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cbb30 0x7f21480ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f2145e00700 1 -- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21480cd020 msgr2=0x7f2148142200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f2145e00700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21480cd020 0x7f2148142200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f2145e00700 1 -- 172.21.15.12:0/1781890927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2140009a70 con 0x7f21480b2070 2024-03-29T11:33:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f2145e00700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21480b2070 0x7f2148098920 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f21380011e0 tx=0x7f213800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f213e000700 1 -- 172.21.15.12:0/1781890927 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138004c80 con 0x7f21480b2070 2024-03-29T11:33:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f213e000700 1 -- 172.21.15.12:0/1781890927 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2138004de0 con 0x7f21480b2070 2024-03-29T11:33:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f214d000700 1 -- 172.21.15.12:0/1781890927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21340094a0 con 0x7f21480b2070 2024-03-29T11:33:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f213e000700 1 -- 172.21.15.12:0/1781890927 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138007310 con 0x7f21480b2070 2024-03-29T11:33:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.315+0000 7f214d000700 1 -- 172.21.15.12:0/1781890927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21480ad620 con 0x7f21480b2070 2024-03-29T11:33:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.316+0000 7f214d000700 1 -- 172.21.15.12:0/1781890927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2148067720 con 0x7f21480b2070 2024-03-29T11:33:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.317+0000 7f213e000700 1 -- 172.21.15.12:0/1781890927 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2138007470 con 0x7f21480b2070 2024-03-29T11:33:20.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.318+0000 7f213e000700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212806aed0 0x7f212806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.319+0000 7f213e000700 1 -- 172.21.15.12:0/1781890927 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6653+0+0 (secure 0 0 0) 0x7f2138089730 con 0x7f21480b2070 2024-03-29T11:33:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.319+0000 7f2145400700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212806aed0 0x7f212806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.320+0000 7f2145400700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212806aed0 0x7f212806d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2134000c00 tx=0x7f213401c040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:20.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.322+0000 7f213e000700 1 -- 172.21.15.12:0/1781890927 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21380541f0 con 0x7f21480b2070 2024-03-29T11:33:20.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.540+0000 7f214d000700 1 -- 172.21.15.12:0/1781890927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f214815c380 con 0x7f21480b2070 2024-03-29T11:33:20.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.541+0000 7f213e000700 1 -- 172.21.15.12:0/1781890927 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v257) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2138057810 con 0x7f21480b2070 2024-03-29T11:33:20.537 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:33:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.543+0000 7f214d000700 1 -- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212806aed0 msgr2=0x7f212806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.543+0000 7f214d000700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212806aed0 0x7f212806d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2134000c00 tx=0x7f213401c040 comp rx=0 tx=0).stop 2024-03-29T11:33:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.543+0000 7f214d000700 1 -- 172.21.15.12:0/1781890927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21480b2070 msgr2=0x7f2148098920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.543+0000 7f214d000700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21480b2070 0x7f2148098920 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f21380011e0 tx=0x7f213800ca30 comp rx=0 tx=0).stop 2024-03-29T11:33:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.543+0000 7f214d000700 1 -- 172.21.15.12:0/1781890927 shutdown_connections 2024-03-29T11:33:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.543+0000 7f214d000700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212806aed0 0x7f212806d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.543+0000 7f214d000700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21480cd020 0x7f2148142200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.543+0000 7f214d000700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21480b2070 0x7f2148098920 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.543+0000 7f214d000700 1 --2- 172.21.15.12:0/1781890927 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21480cbb30 0x7f21480ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.543+0000 7f214d000700 1 -- 172.21.15.12:0/1781890927 >> 172.21.15.12:0/1781890927 conn(0x7f21480b0d70 msgr2=0x7f21480cbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.544+0000 7f214d000700 1 -- 172.21.15.12:0/1781890927 shutdown_connections 2024-03-29T11:33:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.544+0000 7f214d000700 1 -- 172.21.15.12:0/1781890927 wait complete. 2024-03-29T11:33:20.557 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:33:20.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.760+0000 7f36bc200700 1 Processor -- start 2024-03-29T11:33:20.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.761+0000 7f36bc200700 1 -- start start 2024-03-29T11:33:20.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.762+0000 7f36bc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b40ac7d0 0x7f36b4166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.762+0000 7f36bc200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b415fd50 con 0x7f36b415b5d0 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.762+0000 7f36bc200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b415f140 con 0x7f36b40cb3f0 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.762+0000 7f36bc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b415b9b0 con 0x7f36b40ac7d0 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.762+0000 7f36b9800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f36b415b5d0 0x7f36b40ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42896/0 (socket says 172.21.15.12:42896) 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.762+0000 7f36b9800700 1 -- 172.21.15.12:0/1822845060 learned_addr learned my addr 172.21.15.12:0/1822845060 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.762+0000 7f36ba200700 1 --2- 172.21.15.12:0/1822845060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b40ac7d0 0x7f36b4166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.762+0000 7f36b3e00700 1 -- 172.21.15.12:0/1822845060 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019148308 0 0) 0x7f36b415fd50 con 0x7f36b415b5d0 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.763+0000 7f36b3e00700 1 -- 172.21.15.12:0/1822845060 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f369c0035b0 con 0x7f36b415b5d0 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.763+0000 7f36b3e00700 1 -- 172.21.15.12:0/1822845060 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147906807 0 0) 0x7f36b415f140 con 0x7f36b40cb3f0 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.763+0000 7f36b3e00700 1 -- 172.21.15.12:0/1822845060 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b415fd50 con 0x7f36b40cb3f0 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.763+0000 7f36ba200700 1 -- 172.21.15.12:0/1822845060 >> v1:172.21.15.159:6789/0 conn(0x7f36b40cb3f0 legacy=0x7f36b40ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.763+0000 7f36ba200700 1 -- 172.21.15.12:0/1822845060 >> v1:172.21.15.12:6789/0 conn(0x7f36b415b5d0 legacy=0x7f36b40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.763+0000 7f36ba200700 1 -- 172.21.15.12:0/1822845060 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b415f420 con 0x7f36b40ac7d0 2024-03-29T11:33:20.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.763+0000 7f36ba200700 1 --2- 172.21.15.12:0/1822845060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b40ac7d0 0x7f36b4166430 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009dc0 tx=0x7f36ac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=445fc580c5c12560 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:20.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.763+0000 7f36b3e00700 1 -- 172.21.15.12:0/1822845060 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac00e070 con 0x7f36b40ac7d0 2024-03-29T11:33:20.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.763+0000 7f36b3e00700 1 -- 172.21.15.12:0/1822845060 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac00bb20 con 0x7f36b40ac7d0 2024-03-29T11:33:20.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.763+0000 7f36b3e00700 1 -- 172.21.15.12:0/1822845060 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac012430 con 0x7f36b40ac7d0 2024-03-29T11:33:20.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.764+0000 7f36bc200700 1 -- 172.21.15.12:0/1822845060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b40ac7d0 msgr2=0x7f36b4166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.764+0000 7f36bc200700 1 --2- 172.21.15.12:0/1822845060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b40ac7d0 0x7f36b4166430 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009dc0 tx=0x7f36ac00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:20.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.764+0000 7f36bc200700 1 -- 172.21.15.12:0/1822845060 shutdown_connections 2024-03-29T11:33:20.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.764+0000 7f36bc200700 1 --2- 172.21.15.12:0/1822845060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b40ac7d0 0x7f36b4166430 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.764+0000 7f36bc200700 1 -- 172.21.15.12:0/1822845060 >> 172.21.15.12:0/1822845060 conn(0x7f36b40cf190 msgr2=0x7f36b40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:20.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.764+0000 7f36bc200700 1 -- 172.21.15.12:0/1822845060 shutdown_connections 2024-03-29T11:33:20.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.764+0000 7f36bc200700 1 -- 172.21.15.12:0/1822845060 wait complete. 2024-03-29T11:33:20.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.766+0000 7f36bc200700 1 Processor -- start 2024-03-29T11:33:20.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.766+0000 7f36bc200700 1 -- start start 2024-03-29T11:33:20.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36bc200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36b40ac7d0 0x7f36b414b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36bc200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b415b5d0 0x7f36b414e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36bc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b4148720 0x7f36b4071db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36bc200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b40b3020 con 0x7f36b415b5d0 2024-03-29T11:33:20.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36bc200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b414ed20 con 0x7f36b40ac7d0 2024-03-29T11:33:20.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36bc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b414b780 con 0x7f36b4148720 2024-03-29T11:33:20.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36b8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b415b5d0 0x7f36b414e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36b8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b415b5d0 0x7f36b414e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56028/0 (socket says 172.21.15.12:56028) 2024-03-29T11:33:20.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36b8e00700 1 -- 172.21.15.12:0/2827959925 learned_addr learned my addr 172.21.15.12:0/2827959925 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:20.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36b9800700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36b40ac7d0 0x7f36b414b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.767+0000 7f36ba200700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b4148720 0x7f36b4071db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36b8e00700 1 -- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b4148720 msgr2=0x7f36b4071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36b8e00700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b4148720 0x7f36b4071db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36b8e00700 1 -- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36b40ac7d0 msgr2=0x7f36b414b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36b8e00700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36b40ac7d0 0x7f36b414b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36b8e00700 1 -- 172.21.15.12:0/2827959925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac009a70 con 0x7f36b415b5d0 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36b8e00700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b415b5d0 0x7f36b414e7e0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f36a000c550 tx=0x7f36a000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36b1600700 1 -- 172.21.15.12:0/2827959925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00052d0 con 0x7f36b415b5d0 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36b1600700 1 -- 172.21.15.12:0/2827959925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a0005430 con 0x7f36b415b5d0 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36b1600700 1 -- 172.21.15.12:0/2827959925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00038c0 con 0x7f36b415b5d0 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36bc200700 1 -- 172.21.15.12:0/2827959925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b40ce370 con 0x7f36b415b5d0 2024-03-29T11:33:20.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.768+0000 7f36bc200700 1 -- 172.21.15.12:0/2827959925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b40ce8f0 con 0x7f36b415b5d0 2024-03-29T11:33:20.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.769+0000 7f36bc200700 1 -- 172.21.15.12:0/2827959925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b40b3160 con 0x7f36b415b5d0 2024-03-29T11:33:20.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.770+0000 7f36b1600700 1 -- 172.21.15.12:0/2827959925 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f36a0003a20 con 0x7f36b415b5d0 2024-03-29T11:33:20.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.771+0000 7f36b1600700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f369406aed0 0x7f369406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:20.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.771+0000 7f36b1600700 1 -- 172.21.15.12:0/2827959925 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6653+0+0 (secure 0 0 0) 0x7f36a0088e40 con 0x7f36b415b5d0 2024-03-29T11:33:20.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.772+0000 7f36b9800700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f369406aed0 0x7f369406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:20.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.772+0000 7f36b9800700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f369406aed0 0x7f369406d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f36a8007fd0 tx=0x7f36a8007dc0 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:20.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:20.775+0000 7f36b1600700 1 -- 172.21.15.12:0/2827959925 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f36b40b3160 con 0x7f36b415b5d0 2024-03-29T11:33:21.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.039+0000 7f36bc200700 1 -- 172.21.15.12:0/2827959925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36b40b3160 con 0x7f36b415b5d0 2024-03-29T11:33:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.040+0000 7f36b1600700 1 -- 172.21.15.12:0/2827959925 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f36b40b3160 con 0x7f36b415b5d0 2024-03-29T11:33:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:21.036 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":130,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":12459,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/839500006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":839500006},{"type":"v1","addr":"172.21.15.12:6834","nonce":839500006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:47.849181+0000","modified":"2024-03-29T11:32:51.340904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11416":{"gid":11416,"name":"d","rank":0,"incarnation":124,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-29T11:33:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.042+0000 7f36bc200700 1 -- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f369406aed0 msgr2=0x7f369406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.042+0000 7f36bc200700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f369406aed0 0x7f369406d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f36a8007fd0 tx=0x7f36a8007dc0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.042+0000 7f36bc200700 1 -- 172.21.15.12:0/2827959925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b415b5d0 msgr2=0x7f36b414e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.043+0000 7f36bc200700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b415b5d0 0x7f36b414e7e0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f36a000c550 tx=0x7f36a000c860 comp rx=0 tx=0).stop 2024-03-29T11:33:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.043+0000 7f36bc200700 1 -- 172.21.15.12:0/2827959925 shutdown_connections 2024-03-29T11:33:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.043+0000 7f36bc200700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f369406aed0 0x7f369406d390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.043+0000 7f36bc200700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36b4148720 0x7f36b4071db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.043+0000 7f36bc200700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36b40ac7d0 0x7f36b414b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.043+0000 7f36bc200700 1 --2- 172.21.15.12:0/2827959925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b415b5d0 0x7f36b414e7e0 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.043+0000 7f36bc200700 1 -- 172.21.15.12:0/2827959925 >> 172.21.15.12:0/2827959925 conn(0x7f36b40cf190 msgr2=0x7f36b40b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.043+0000 7f36bc200700 1 -- 172.21.15.12:0/2827959925 shutdown_connections 2024-03-29T11:33:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.043+0000 7f36bc200700 1 -- 172.21.15.12:0/2827959925 wait complete. 2024-03-29T11:33:21.041 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 130 2024-03-29T11:33:21.057 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:21.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.259+0000 7fe781c00700 1 Processor -- start 2024-03-29T11:33:21.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.259+0000 7fe781c00700 1 -- start start 2024-03-29T11:33:21.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.260+0000 7fe781c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe77c160be0 0x7fe77c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.260+0000 7fe781c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe77c0ce0e0 0x7fe77c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.260+0000 7fe781c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77c160fc0 con 0x7fe77c0cead0 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.260+0000 7fe781c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77c0b65d0 con 0x7fe77c160be0 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.260+0000 7fe781c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77c0d2a80 con 0x7fe77c0ce0e0 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.260+0000 7fe77aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe77c160be0 0x7fe77c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.260+0000 7fe77aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe77c160be0 0x7fe77c0cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56036/0 (socket says 172.21.15.12:56036) 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.260+0000 7fe77aa00700 1 -- 172.21.15.12:0/1093961878 learned_addr learned my addr 172.21.15.12:0/1093961878 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.261+0000 7fe77a000700 1 --2- 172.21.15.12:0/1093961878 >> v2:172.21.15.159:3300/0 conn(0x7fe77c0ce0e0 0x7fe77c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.261+0000 7fe77aa00700 1 -- 172.21.15.12:0/1093961878 >> v2:172.21.15.159:3300/0 conn(0x7fe77c0ce0e0 msgr2=0x7fe77c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.261+0000 7fe77aa00700 1 --2- 172.21.15.12:0/1093961878 >> v2:172.21.15.159:3300/0 conn(0x7fe77c0ce0e0 0x7fe77c0ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.261+0000 7fe77aa00700 1 -- 172.21.15.12:0/1093961878 >> v1:172.21.15.159:6789/0 conn(0x7fe77c0cead0 legacy=0x7fe77c161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.261+0000 7fe77aa00700 1 -- 172.21.15.12:0/1093961878 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77c0d21e0 con 0x7fe77c160be0 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.261+0000 7fe77aa00700 1 --2- 172.21.15.12:0/1093961878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c160be0 0x7fe77c0cdba0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7fe77c0b9600 tx=0x7fe76c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8cb781b694984c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:21.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.261+0000 7fe779600700 1 -- 172.21.15.12:0/1093961878 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c00b870 con 0x7fe77c160be0 2024-03-29T11:33:21.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.261+0000 7fe779600700 1 -- 172.21.15.12:0/1093961878 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76c00b9d0 con 0x7fe77c160be0 2024-03-29T11:33:21.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.261+0000 7fe779600700 1 -- 172.21.15.12:0/1093961878 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c00bd60 con 0x7fe77c160be0 2024-03-29T11:33:21.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.262+0000 7fe781c00700 1 -- 172.21.15.12:0/1093961878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c160be0 msgr2=0x7fe77c0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.262+0000 7fe781c00700 1 --2- 172.21.15.12:0/1093961878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c160be0 0x7fe77c0cdba0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7fe77c0b9600 tx=0x7fe76c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.262+0000 7fe781c00700 1 -- 172.21.15.12:0/1093961878 shutdown_connections 2024-03-29T11:33:21.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.262+0000 7fe781c00700 1 --2- 172.21.15.12:0/1093961878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c160be0 0x7fe77c0cdba0 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.262+0000 7fe781c00700 1 --2- 172.21.15.12:0/1093961878 >> v2:172.21.15.159:3300/0 conn(0x7fe77c0ce0e0 0x7fe77c0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.262+0000 7fe781c00700 1 -- 172.21.15.12:0/1093961878 >> 172.21.15.12:0/1093961878 conn(0x7fe77c08aff0 msgr2=0x7fe77c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:21.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.262+0000 7fe781c00700 1 -- 172.21.15.12:0/1093961878 shutdown_connections 2024-03-29T11:33:21.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.262+0000 7fe781c00700 1 -- 172.21.15.12:0/1093961878 wait complete. 2024-03-29T11:33:21.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.264+0000 7fe781c00700 1 Processor -- start 2024-03-29T11:33:21.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.264+0000 7fe781c00700 1 -- start start 2024-03-29T11:33:21.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe781c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe77c0ce0e0 0x7fe77c0c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe781c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c0cead0 0x7fe77c157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe781c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe77c160be0 0x7fe77c06caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe781c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c0cd4f0 con 0x7fe77c0cead0 2024-03-29T11:33:21.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe781c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c0c7190 con 0x7fe77c0ce0e0 2024-03-29T11:33:21.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe781c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c0c7470 con 0x7fe77c160be0 2024-03-29T11:33:21.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe77a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c0cead0 0x7fe77c157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe77a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c0cead0 0x7fe77c157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56046/0 (socket says 172.21.15.12:56046) 2024-03-29T11:33:21.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe77a000700 1 -- 172.21.15.12:0/61112379 learned_addr learned my addr 172.21.15.12:0/61112379 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:21.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe77b400700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe77c160be0 0x7fe77c06caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe77aa00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe77c0ce0e0 0x7fe77c0c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe77a000700 1 -- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe77c160be0 msgr2=0x7fe77c06caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.265+0000 7fe77a000700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe77c160be0 0x7fe77c06caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.266+0000 7fe77a000700 1 -- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe77c0ce0e0 msgr2=0x7fe77c0c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.266+0000 7fe77a000700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe77c0ce0e0 0x7fe77c0c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.266+0000 7fe77a000700 1 -- 172.21.15.12:0/61112379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c009550 con 0x7fe77c0cead0 2024-03-29T11:33:21.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.266+0000 7fe77aa00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe77c0ce0e0 0x7fe77c0c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:21.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.266+0000 7fe77a000700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c0cead0 0x7fe77c157900 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7fe76800e810 tx=0x7fe76800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:21.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.266+0000 7fe772a00700 1 -- 172.21.15.12:0/61112379 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768013610 con 0x7fe77c0cead0 2024-03-29T11:33:21.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.266+0000 7fe772a00700 1 -- 172.21.15.12:0/61112379 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768013770 con 0x7fe77c0cead0 2024-03-29T11:33:21.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.266+0000 7fe772a00700 1 -- 172.21.15.12:0/61112379 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768013b20 con 0x7fe77c0cead0 2024-03-29T11:33:21.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.266+0000 7fe781c00700 1 -- 172.21.15.12:0/61112379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77c0c7750 con 0x7fe77c0cead0 2024-03-29T11:33:21.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.266+0000 7fe781c00700 1 -- 172.21.15.12:0/61112379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77c0c7d10 con 0x7fe77c0cead0 2024-03-29T11:33:21.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.267+0000 7fe781c00700 1 -- 172.21.15.12:0/61112379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77c0c7ed0 con 0x7fe77c0cead0 2024-03-29T11:33:21.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.268+0000 7fe772a00700 1 -- 172.21.15.12:0/61112379 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe768013c80 con 0x7fe77c0cead0 2024-03-29T11:33:21.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.269+0000 7fe772a00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe75c06aed0 0x7fe75c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.269+0000 7fe772a00700 1 -- 172.21.15.12:0/61112379 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6653+0+0 (secure 0 0 0) 0x7fe768089940 con 0x7fe77c0cead0 2024-03-29T11:33:21.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.270+0000 7fe77aa00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe75c06aed0 0x7fe75c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.270+0000 7fe77aa00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe75c06aed0 0x7fe75c06d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe76c014040 tx=0x7fe76c008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:21.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.273+0000 7fe772a00700 1 -- 172.21.15.12:0/61112379 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe768054270 con 0x7fe77c0cead0 2024-03-29T11:33:21.458 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:33:21.464 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:33:21.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.486+0000 7fe781c00700 1 -- 172.21.15.12:0/61112379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe77c067720 con 0x7fe77c0cead0 2024-03-29T11:33:21.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.487+0000 7fe772a00700 1 -- 172.21.15.12:0/61112379 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+17434 (secure 0 0 0) 0x7fe768057890 con 0x7fe77c0cead0 2024-03-29T11:33:21.484 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:21.484 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":257,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:18.697180+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:32:45.400233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:32:46.836046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6834/3711953177":"2024-03-30T11:33:15.669567+0000","172.21.15.159:0/117966312":"2024-03-30T11:33:12.058935+0000","172.21.15.12:6832/3711953177":"2024-03-30T11:33:15.669567+0000","172.21.15.159:0/1298574278":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/449301407":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/2167892315":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/122983726":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/2916339927":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/962755797":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1966802084":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/737492091":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/110096794":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/449562747":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1567850739":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/3214276897":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/180562283":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/2535009377":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1057841974":"2024-03-30T11:33:03.088269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 -- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe75c06aed0 msgr2=0x7fe75c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe75c06aed0 0x7fe75c06d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe76c014040 tx=0x7fe76c008030 comp rx=0 tx=0).stop 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 -- 172.21.15.12:0/61112379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c0cead0 msgr2=0x7fe77c157900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c0cead0 0x7fe77c157900 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7fe76800e810 tx=0x7fe76800ebd0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 -- 172.21.15.12:0/61112379 shutdown_connections 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe75c06aed0 0x7fe75c06d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe77c160be0 0x7fe77c06caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe77c0ce0e0 0x7fe77c0c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 --2- 172.21.15.12:0/61112379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe77c0cead0 0x7fe77c157900 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 -- 172.21.15.12:0/61112379 >> 172.21.15.12:0/61112379 conn(0x7fe77c08aff0 msgr2=0x7fe77c15f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 -- 172.21.15.12:0/61112379 shutdown_connections 2024-03-29T11:33:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.490+0000 7fe781c00700 1 -- 172.21.15.12:0/61112379 wait complete. 2024-03-29T11:33:21.505 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-03-29T11:33:21.505 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:33:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.757+0000 7f5c77200700 1 Processor -- start 2024-03-29T11:33:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.757+0000 7f5c77200700 1 -- start start 2024-03-29T11:33:21.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.758+0000 7f5c77200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c7015d640 0x7f5c700d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.758+0000 7f5c77200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5c700ceec0 0x7f5c701612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.758+0000 7f5c77200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c700d30d0 con 0x7f5c70161800 2024-03-29T11:33:21.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.758+0000 7f5c77200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c70161c90 con 0x7f5c700ceec0 2024-03-29T11:33:21.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.758+0000 7f5c77200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700b6840 con 0x7f5c7015d640 2024-03-29T11:33:21.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.758+0000 7f5c75200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5c70161800 0x7f5c700cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42898/0 (socket says 172.21.15.12:42898) 2024-03-29T11:33:21.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.758+0000 7f5c75200700 1 -- 172.21.15.12:0/147116518 learned_addr learned my addr 172.21.15.12:0/147116518 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:21.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.758+0000 7f5c6fe00700 1 --2- 172.21.15.12:0/147116518 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c7015d640 0x7f5c700d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.758+0000 7f5c6f400700 1 --2- 172.21.15.12:0/147116518 >> v2:172.21.15.159:3300/0 conn(0x7f5c700ceec0 0x7f5c701612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.759+0000 7f5c6ea00700 1 -- 172.21.15.12:0/147116518 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464457974 0 0) 0x7f5c700d30d0 con 0x7f5c70161800 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.759+0000 7f5c6ea00700 1 -- 172.21.15.12:0/147116518 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c540035b0 con 0x7f5c70161800 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.759+0000 7f5c6fe00700 1 -- 172.21.15.12:0/147116518 >> v2:172.21.15.159:3300/0 conn(0x7f5c700ceec0 msgr2=0x7f5c701612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.759+0000 7f5c6fe00700 1 --2- 172.21.15.12:0/147116518 >> v2:172.21.15.159:3300/0 conn(0x7f5c700ceec0 0x7f5c701612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.759+0000 7f5c6fe00700 1 -- 172.21.15.12:0/147116518 >> v1:172.21.15.12:6789/0 conn(0x7f5c70161800 legacy=0x7f5c700cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.759+0000 7f5c6fe00700 1 -- 172.21.15.12:0/147116518 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c700d2880 con 0x7f5c7015d640 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.759+0000 7f5c6f400700 1 --2- 172.21.15.12:0/147116518 >> v2:172.21.15.159:3300/0 conn(0x7f5c700ceec0 0x7f5c701612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.759+0000 7f5c6fe00700 1 --2- 172.21.15.12:0/147116518 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c7015d640 0x7f5c700d2340 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009840 tx=0x7f5c60009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=211146ebe649a7ca server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c6ea00700 1 -- 172.21.15.12:0/147116518 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60011830 con 0x7f5c7015d640 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c6ea00700 1 -- 172.21.15.12:0/147116518 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c60011990 con 0x7f5c7015d640 2024-03-29T11:33:21.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c6ea00700 1 -- 172.21.15.12:0/147116518 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60011d20 con 0x7f5c7015d640 2024-03-29T11:33:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c77200700 1 -- 172.21.15.12:0/147116518 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c7015d640 msgr2=0x7f5c700d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c77200700 1 --2- 172.21.15.12:0/147116518 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c7015d640 0x7f5c700d2340 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009840 tx=0x7f5c60009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c77200700 1 -- 172.21.15.12:0/147116518 shutdown_connections 2024-03-29T11:33:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c77200700 1 --2- 172.21.15.12:0/147116518 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c7015d640 0x7f5c700d2340 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c77200700 1 --2- 172.21.15.12:0/147116518 >> v2:172.21.15.159:3300/0 conn(0x7f5c700ceec0 0x7f5c701612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c77200700 1 -- 172.21.15.12:0/147116518 >> 172.21.15.12:0/147116518 conn(0x7f5c7008aff0 msgr2=0x7f5c700d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c77200700 1 -- 172.21.15.12:0/147116518 shutdown_connections 2024-03-29T11:33:21.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.760+0000 7f5c77200700 1 -- 172.21.15.12:0/147116518 wait complete. 2024-03-29T11:33:21.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.762+0000 7f5c77200700 1 Processor -- start 2024-03-29T11:33:21.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.762+0000 7f5c77200700 1 -- start start 2024-03-29T11:33:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.763+0000 7f5c77200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c700ceec0 0x7f5c70154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.763+0000 7f5c77200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c7015d640 0x7f5c700c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.763+0000 7f5c77200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c70161800 0x7f5c70143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.763+0000 7f5c77200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70154d60 con 0x7f5c700ceec0 2024-03-29T11:33:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.763+0000 7f5c77200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70155040 con 0x7f5c7015d640 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.763+0000 7f5c77200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70155320 con 0x7f5c70161800 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.763+0000 7f5c6fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c700ceec0 0x7f5c70154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.763+0000 7f5c6fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c700ceec0 0x7f5c70154770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56058/0 (socket says 172.21.15.12:56058) 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.763+0000 7f5c6fe00700 1 -- 172.21.15.12:0/3317615914 learned_addr learned my addr 172.21.15.12:0/3317615914 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c75200700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c70161800 0x7f5c70143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c6f400700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c7015d640 0x7f5c700c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c6fe00700 1 -- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c70161800 msgr2=0x7f5c70143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c6fe00700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c70161800 0x7f5c70143070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c6fe00700 1 -- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c7015d640 msgr2=0x7f5c700c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c6fe00700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c7015d640 0x7f5c700c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c6fe00700 1 -- 172.21.15.12:0/3317615914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c0094a0 con 0x7f5c700ceec0 2024-03-29T11:33:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c6fe00700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c700ceec0 0x7f5c70154770 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f5c600090b0 tx=0x7f5c60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c67e00700 1 -- 172.21.15.12:0/3317615914 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c600036a0 con 0x7f5c700ceec0 2024-03-29T11:33:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c67e00700 1 -- 172.21.15.12:0/3317615914 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6001f070 con 0x7f5c700ceec0 2024-03-29T11:33:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c67e00700 1 -- 172.21.15.12:0/3317615914 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60003a40 con 0x7f5c700ceec0 2024-03-29T11:33:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.764+0000 7f5c77200700 1 -- 172.21.15.12:0/3317615914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c600094a0 con 0x7f5c700ceec0 2024-03-29T11:33:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.765+0000 7f5c77200700 1 -- 172.21.15.12:0/3317615914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c700ca270 con 0x7f5c700ceec0 2024-03-29T11:33:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.765+0000 7f5c77200700 1 -- 172.21.15.12:0/3317615914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c70067720 con 0x7f5c700ceec0 2024-03-29T11:33:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.766+0000 7f5c67e00700 1 -- 172.21.15.12:0/3317615914 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5c60003ba0 con 0x7f5c700ceec0 2024-03-29T11:33:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.767+0000 7f5c67e00700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5c5006aed0 0x7f5c5006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.767+0000 7f5c67e00700 1 -- 172.21.15.12:0/3317615914 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6653+0+0 (secure 0 0 0) 0x7f5c60021070 con 0x7f5c700ceec0 2024-03-29T11:33:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.768+0000 7f5c6f400700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5c5006aed0 0x7f5c5006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.768+0000 7f5c6f400700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5c5006aed0 0x7f5c5006d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5c70160d70 tx=0x7f5c5c008040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:21.772+0000 7f5c67e00700 1 -- 172.21.15.12:0/3317615914 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c60055e60 con 0x7f5c700ceec0 2024-03-29T11:33:22.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.030+0000 7f5c77200700 1 -- 172.21.15.12:0/3317615914 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c700b5c70 con 0x7f5c700ceec0 2024-03-29T11:33:22.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.031+0000 7f5c67e00700 1 -- 172.21.15.12:0/3317615914 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f5c60059480 con 0x7f5c700ceec0 2024-03-29T11:33:22.027 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:22.027 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":130,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":12459,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/839500006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":839500006},{"type":"v1","addr":"172.21.15.12:6834","nonce":839500006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:47.849181+0000","modified":"2024-03-29T11:32:51.340904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11416":{"gid":11416,"name":"d","rank":0,"incarnation":124,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-29T11:33:22.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.033+0000 7f5c77200700 1 -- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5c5006aed0 msgr2=0x7f5c5006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.033+0000 7f5c77200700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5c5006aed0 0x7f5c5006d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5c70160d70 tx=0x7f5c5c008040 comp rx=0 tx=0).stop 2024-03-29T11:33:22.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.033+0000 7f5c77200700 1 -- 172.21.15.12:0/3317615914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c700ceec0 msgr2=0x7f5c70154770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.033+0000 7f5c77200700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c700ceec0 0x7f5c70154770 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f5c600090b0 tx=0x7f5c60004750 comp rx=0 tx=0).stop 2024-03-29T11:33:22.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.033+0000 7f5c77200700 1 -- 172.21.15.12:0/3317615914 shutdown_connections 2024-03-29T11:33:22.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.033+0000 7f5c77200700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5c5006aed0 0x7f5c5006d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.033+0000 7f5c77200700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c70161800 0x7f5c70143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.033+0000 7f5c77200700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c700ceec0 0x7f5c70154770 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.033+0000 7f5c77200700 1 --2- 172.21.15.12:0/3317615914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c7015d640 0x7f5c700c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.033+0000 7f5c77200700 1 -- 172.21.15.12:0/3317615914 >> 172.21.15.12:0/3317615914 conn(0x7f5c7008aff0 msgr2=0x7f5c7006d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:22.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.034+0000 7f5c77200700 1 -- 172.21.15.12:0/3317615914 shutdown_connections 2024-03-29T11:33:22.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.034+0000 7f5c77200700 1 -- 172.21.15.12:0/3317615914 wait complete. 2024-03-29T11:33:22.031 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 130 2024-03-29T11:33:22.048 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:33:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.250+0000 7f428c200700 1 Processor -- start 2024-03-29T11:33:22.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.250+0000 7f428c200700 1 -- start start 2024-03-29T11:33:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.251+0000 7f428c200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4284161ca0 0x7f42840cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.251+0000 7f428c200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42840ceda0 con 0x7f4284160f10 2024-03-29T11:33:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.251+0000 7f428c200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42841612f0 con 0x7f42840d18b0 2024-03-29T11:33:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.251+0000 7f428c200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42840d1d30 con 0x7f4284161ca0 2024-03-29T11:33:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.252+0000 7f4289800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4284160f10 0x7f42840d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42908/0 (socket says 172.21.15.12:42908) 2024-03-29T11:33:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.252+0000 7f4289800700 1 -- 172.21.15.12:0/3548868847 learned_addr learned my addr 172.21.15.12:0/3548868847 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.252+0000 7f428a200700 1 --2- 172.21.15.12:0/3548868847 >> v2:172.21.15.12:3300/0 conn(0x7f4284161ca0 0x7f42840cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.252+0000 7f428a200700 1 -- 172.21.15.12:0/3548868847 >> v1:172.21.15.159:6789/0 conn(0x7f42840d18b0 legacy=0x7f4284161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.252+0000 7f428a200700 1 -- 172.21.15.12:0/3548868847 >> v1:172.21.15.12:6789/0 conn(0x7f4284160f10 legacy=0x7f42840d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.252+0000 7f428a200700 1 -- 172.21.15.12:0/3548868847 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42840ce470 con 0x7f4284161ca0 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.252+0000 7f4283e00700 1 -- 172.21.15.12:0/3548868847 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187078326 0 0) 0x7f42840ceda0 con 0x7f4284160f10 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.253+0000 7f428a200700 1 --2- 172.21.15.12:0/3548868847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4284161ca0 0x7f42840cf230 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f427c009d90 tx=0x7f427c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a5a45c3aff3c42 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.253+0000 7f4283e00700 1 -- 172.21.15.12:0/3548868847 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c00e070 con 0x7f4284161ca0 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.253+0000 7f4283e00700 1 -- 172.21.15.12:0/3548868847 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c00bbd0 con 0x7f4284161ca0 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.253+0000 7f4283e00700 1 -- 172.21.15.12:0/3548868847 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c012430 con 0x7f4284161ca0 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.253+0000 7f428c200700 1 -- 172.21.15.12:0/3548868847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4284161ca0 msgr2=0x7f42840cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.253+0000 7f428c200700 1 --2- 172.21.15.12:0/3548868847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4284161ca0 0x7f42840cf230 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f427c009d90 tx=0x7f427c00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.253+0000 7f428c200700 1 -- 172.21.15.12:0/3548868847 shutdown_connections 2024-03-29T11:33:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.253+0000 7f428c200700 1 --2- 172.21.15.12:0/3548868847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4284161ca0 0x7f42840cf230 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.253+0000 7f428c200700 1 -- 172.21.15.12:0/3548868847 >> 172.21.15.12:0/3548868847 conn(0x7f428408aff0 msgr2=0x7f42840d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.253+0000 7f428c200700 1 -- 172.21.15.12:0/3548868847 shutdown_connections 2024-03-29T11:33:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.254+0000 7f428c200700 1 -- 172.21.15.12:0/3548868847 wait complete. 2024-03-29T11:33:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.255+0000 7f428c200700 1 Processor -- start 2024-03-29T11:33:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.255+0000 7f428c200700 1 -- start start 2024-03-29T11:33:22.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.256+0000 7f428c200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42840d18b0 0x7f42841588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.256+0000 7f428c200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4284160f10 0x7f42841e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.256+0000 7f428c200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4284161ca0 0x7f42840b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.256+0000 7f428c200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42841e87a0 con 0x7f42840d18b0 2024-03-29T11:33:22.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.256+0000 7f428c200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42841e8310 con 0x7f4284161ca0 2024-03-29T11:33:22.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.256+0000 7f428c200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42841e7700 con 0x7f4284160f10 2024-03-29T11:33:22.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.256+0000 7f4289800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42840d18b0 0x7f42841588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.256+0000 7f4289800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42840d18b0 0x7f42841588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56072/0 (socket says 172.21.15.12:56072) 2024-03-29T11:33:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.256+0000 7f4289800700 1 -- 172.21.15.12:0/19967000 learned_addr learned my addr 172.21.15.12:0/19967000 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.257+0000 7f4289800700 1 -- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4284160f10 msgr2=0x7f42841e6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.257+0000 7f4289800700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4284160f10 0x7f42841e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.257+0000 7f4289800700 1 -- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4284161ca0 msgr2=0x7f42840b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.257+0000 7f4289800700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4284161ca0 0x7f42840b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.257+0000 7f4289800700 1 -- 172.21.15.12:0/19967000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c009a70 con 0x7f42840d18b0 2024-03-29T11:33:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.257+0000 7f4289800700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42840d18b0 0x7f42841588d0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f427800c550 tx=0x7f427800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:22.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.257+0000 7f4281600700 1 -- 172.21.15.12:0/19967000 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278004c80 con 0x7f42840d18b0 2024-03-29T11:33:22.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.257+0000 7f4281600700 1 -- 172.21.15.12:0/19967000 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4278004de0 con 0x7f42840d18b0 2024-03-29T11:33:22.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.257+0000 7f428c200700 1 -- 172.21.15.12:0/19967000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42841e79e0 con 0x7f42840d18b0 2024-03-29T11:33:22.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.257+0000 7f4281600700 1 -- 172.21.15.12:0/19967000 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278007310 con 0x7f42840d18b0 2024-03-29T11:33:22.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.258+0000 7f428c200700 1 -- 172.21.15.12:0/19967000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42841e7fa0 con 0x7f42840d18b0 2024-03-29T11:33:22.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.258+0000 7f428c200700 1 -- 172.21.15.12:0/19967000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4284067720 con 0x7f42840d18b0 2024-03-29T11:33:22.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.259+0000 7f4281600700 1 -- 172.21.15.12:0/19967000 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4278007470 con 0x7f42840d18b0 2024-03-29T11:33:22.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.260+0000 7f4281600700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f426406aed0 0x7f426406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.260+0000 7f4281600700 1 -- 172.21.15.12:0/19967000 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6653+0+0 (secure 0 0 0) 0x7f42780897a0 con 0x7f42840d18b0 2024-03-29T11:33:22.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.261+0000 7f4288e00700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f426406aed0 0x7f426406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.262+0000 7f4288e00700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f426406aed0 0x7f426406d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4270000f60 tx=0x7f4270007a40 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.265+0000 7f4281600700 1 -- 172.21.15.12:0/19967000 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4278054260 con 0x7f42840d18b0 2024-03-29T11:33:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.526+0000 7f428c200700 1 -- 172.21.15.12:0/19967000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42841e85f0 con 0x7f42840d18b0 2024-03-29T11:33:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.526+0000 7f4281600700 1 -- 172.21.15.12:0/19967000 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f4278057880 con 0x7f42840d18b0 2024-03-29T11:33:22.523 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:22.523 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":130,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":12459,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/839500006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":839500006},{"type":"v1","addr":"172.21.15.12:6834","nonce":839500006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:32:47.849181+0000","modified":"2024-03-29T11:32:51.340904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11416":{"gid":11416,"name":"d","rank":0,"incarnation":124,"state":"up:active","state_seq":6,"addr":"172.21.15.159:6837/1262061639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1262061639},{"type":"v1","addr":"172.21.15.159:6837","nonce":1262061639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-29T11:33:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 -- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f426406aed0 msgr2=0x7f426406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f426406aed0 0x7f426406d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4270000f60 tx=0x7f4270007a40 comp rx=0 tx=0).stop 2024-03-29T11:33:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 -- 172.21.15.12:0/19967000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42840d18b0 msgr2=0x7f42841588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42840d18b0 0x7f42841588d0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f427800c550 tx=0x7f427800c910 comp rx=0 tx=0).stop 2024-03-29T11:33:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 -- 172.21.15.12:0/19967000 shutdown_connections 2024-03-29T11:33:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f426406aed0 0x7f426406d390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4284161ca0 0x7f42840b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42840d18b0 0x7f42841588d0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 --2- 172.21.15.12:0/19967000 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4284160f10 0x7f42841e6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 -- 172.21.15.12:0/19967000 >> 172.21.15.12:0/19967000 conn(0x7f428408aff0 msgr2=0x7f42840d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 -- 172.21.15.12:0/19967000 shutdown_connections 2024-03-29T11:33:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.529+0000 7f428c200700 1 -- 172.21.15.12:0/19967000 wait complete. 2024-03-29T11:33:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 130 2024-03-29T11:33:22.543 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.748+0000 7f18c4400700 1 Processor -- start 2024-03-29T11:33:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.748+0000 7f18c4400700 1 -- start start 2024-03-29T11:33:22.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.749+0000 7f18c4400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f18bc0cb3f0 0x7f18bc0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.749+0000 7f18c4400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 0x7f18bc06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.749+0000 7f18c4400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18bc15fd50 con 0x7f18bc15b5d0 2024-03-29T11:33:22.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.749+0000 7f18c4400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18bc15b9b0 con 0x7f18bc0cb3f0 2024-03-29T11:33:22.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.749+0000 7f18c4400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18bc0aca50 con 0x7f18bc0ac600 2024-03-29T11:33:22.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.749+0000 7f18c1000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f18bc0cb3f0 0x7f18bc0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:22.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.749+0000 7f18c1000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f18bc0cb3f0 0x7f18bc0ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56082/0 (socket says 172.21.15.12:56082) 2024-03-29T11:33:22.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.749+0000 7f18c1000700 1 -- 172.21.15.12:0/318650031 learned_addr learned my addr 172.21.15.12:0/318650031 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:22.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.750+0000 7f18c1000700 1 -- 172.21.15.12:0/318650031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 msgr2=0x7f18bc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:22.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.750+0000 7f18c2400700 1 --2- 172.21.15.12:0/318650031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 0x7f18bc06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:22.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.750+0000 7f18c1000700 1 --2- 172.21.15.12:0/318650031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 0x7f18bc06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.750+0000 7f18c1000700 1 -- 172.21.15.12:0/318650031 >> v1:172.21.15.159:6789/0 conn(0x7f18bc15b5d0 legacy=0x7f18bc0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:22.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.750+0000 7f18c1000700 1 -- 172.21.15.12:0/318650031 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18bc15f140 con 0x7f18bc0cb3f0 2024-03-29T11:33:22.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.750+0000 7f18c2400700 1 --2- 172.21.15.12:0/318650031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 0x7f18bc06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:22.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.750+0000 7f18c1000700 1 --2- 172.21.15.12:0/318650031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc0cb3f0 0x7f18bc0ac0c0 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f18a80098f0 tx=0x7f18a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba1fbb2e09fb8bd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:22.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.750+0000 7f18bbe00700 1 -- 172.21.15.12:0/318650031 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8004230 con 0x7f18bc0cb3f0 2024-03-29T11:33:22.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.750+0000 7f18bbe00700 1 -- 172.21.15.12:0/318650031 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a8004390 con 0x7f18bc0cb3f0 2024-03-29T11:33:22.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.750+0000 7f18bbe00700 1 -- 172.21.15.12:0/318650031 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a80036a0 con 0x7f18bc0cb3f0 2024-03-29T11:33:22.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.751+0000 7f18c4400700 1 -- 172.21.15.12:0/318650031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc0cb3f0 msgr2=0x7f18bc0ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.751+0000 7f18c4400700 1 --2- 172.21.15.12:0/318650031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc0cb3f0 0x7f18bc0ac0c0 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f18a80098f0 tx=0x7f18a8009c00 comp rx=0 tx=0).stop 2024-03-29T11:33:22.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.751+0000 7f18c4400700 1 -- 172.21.15.12:0/318650031 shutdown_connections 2024-03-29T11:33:22.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.751+0000 7f18c4400700 1 --2- 172.21.15.12:0/318650031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 0x7f18bc06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.751+0000 7f18c4400700 1 --2- 172.21.15.12:0/318650031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc0cb3f0 0x7f18bc0ac0c0 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.751+0000 7f18c4400700 1 -- 172.21.15.12:0/318650031 >> 172.21.15.12:0/318650031 conn(0x7f18bc0cf190 msgr2=0x7f18bc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:22.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.751+0000 7f18c4400700 1 -- 172.21.15.12:0/318650031 shutdown_connections 2024-03-29T11:33:22.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.751+0000 7f18c4400700 1 -- 172.21.15.12:0/318650031 wait complete. 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.753+0000 7f18c4400700 1 Processor -- start 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.753+0000 7f18c4400700 1 -- start start 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c4400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 0x7f18bc0ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c4400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc15b5d0 0x7f18bc1f0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c4400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18bc06ce10 0x7f18bc150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c4400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18bc153900 con 0x7f18bc15b5d0 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c4400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18bc150960 con 0x7f18bc06ce10 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c4400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18bc1532b0 con 0x7f18bc0ac600 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc15b5d0 0x7f18bc1f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc15b5d0 0x7f18bc1f0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56098/0 (socket says 172.21.15.12:56098) 2024-03-29T11:33:22.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c1000700 1 -- 172.21.15.12:0/3405065119 learned_addr learned my addr 172.21.15.12:0/3405065119 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:22.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c1a00700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 0x7f18bc0ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:22.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.754+0000 7f18c2400700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18bc06ce10 0x7f18bc150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:22.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18c1000700 1 -- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 msgr2=0x7f18bc0ad7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18c1000700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 0x7f18bc0ad7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18c1000700 1 -- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18bc06ce10 msgr2=0x7f18bc150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18c1000700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18bc06ce10 0x7f18bc150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18c1000700 1 -- 172.21.15.12:0/3405065119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a8009550 con 0x7f18bc15b5d0 2024-03-29T11:33:22.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18c1000700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc15b5d0 0x7f18bc1f0900 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009130 tx=0x7f18a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:22.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18b9600700 1 -- 172.21.15.12:0/3405065119 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8003cd0 con 0x7f18bc15b5d0 2024-03-29T11:33:22.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18b9600700 1 -- 172.21.15.12:0/3405065119 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a801f070 con 0x7f18bc15b5d0 2024-03-29T11:33:22.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18c4400700 1 -- 172.21.15.12:0/3405065119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18bc153590 con 0x7f18bc15b5d0 2024-03-29T11:33:22.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18b9600700 1 -- 172.21.15.12:0/3405065119 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a800b8a0 con 0x7f18bc15b5d0 2024-03-29T11:33:22.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.755+0000 7f18c4400700 1 -- 172.21.15.12:0/3405065119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18bc1504f0 con 0x7f18bc15b5d0 2024-03-29T11:33:22.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.756+0000 7f18c4400700 1 -- 172.21.15.12:0/3405065119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18bc1581c0 con 0x7f18bc15b5d0 2024-03-29T11:33:22.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.757+0000 7f18b9600700 1 -- 172.21.15.12:0/3405065119 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f18a8021030 con 0x7f18bc15b5d0 2024-03-29T11:33:22.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.758+0000 7f18b9600700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f189c06af00 0x7f189c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:22.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.758+0000 7f18b9600700 1 -- 172.21.15.12:0/3405065119 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6653+0+0 (secure 0 0 0) 0x7f18a808af70 con 0x7f18bc15b5d0 2024-03-29T11:33:22.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.759+0000 7f18c1a00700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f189c06af00 0x7f189c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:22.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.759+0000 7f18c1a00700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f189c06af00 0x7f189c06d3c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009960 tx=0x7f18b00090e0 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.762+0000 7f18b9600700 1 -- 172.21.15.12:0/3405065119 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18bc1581c0 con 0x7f18bc15b5d0 2024-03-29T11:33:22.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.978+0000 7f18c4400700 1 -- 172.21.15.12:0/3405065119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18bc1581c0 con 0x7f18bc15b5d0 2024-03-29T11:33:22.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.979+0000 7f18b9600700 1 -- 172.21.15.12:0/3405065119 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+17434 (secure 0 0 0) 0x7f18bc1581c0 con 0x7f18bc15b5d0 2024-03-29T11:33:22.976 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:22.976 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":257,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:18.697180+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-03-29T11:32:45.400233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-03-29T11:32:46.836046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6834/3711953177":"2024-03-30T11:33:15.669567+0000","172.21.15.159:0/117966312":"2024-03-30T11:33:12.058935+0000","172.21.15.12:6832/3711953177":"2024-03-30T11:33:15.669567+0000","172.21.15.159:0/1298574278":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/449301407":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/2167892315":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/122983726":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/2916339927":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/962755797":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1966802084":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/737492091":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/110096794":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/449562747":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1567850739":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/3214276897":"2024-03-30T11:33:12.058935+0000","172.21.15.159:0/180562283":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/2535009377":"2024-03-30T11:33:03.088269+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:03.088269+0000","172.21.15.159:0/1057841974":"2024-03-30T11:33:03.088269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 -- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f189c06af00 msgr2=0x7f189c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f189c06af00 0x7f189c06d3c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009960 tx=0x7f18b00090e0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 -- 172.21.15.12:0/3405065119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc15b5d0 msgr2=0x7f18bc1f0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc15b5d0 0x7f18bc1f0900 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009130 tx=0x7f18a8004750 comp rx=0 tx=0).stop 2024-03-29T11:33:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 -- 172.21.15.12:0/3405065119 shutdown_connections 2024-03-29T11:33:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f189c06af00 0x7f189c06d3c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18bc06ce10 0x7f18bc150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18bc0ac600 0x7f18bc0ad7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 --2- 172.21.15.12:0/3405065119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18bc15b5d0 0x7f18bc1f0900 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 -- 172.21.15.12:0/3405065119 >> 172.21.15.12:0/3405065119 conn(0x7f18bc0cf190 msgr2=0x7f18bc15d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 -- 172.21.15.12:0/3405065119 shutdown_connections 2024-03-29T11:33:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:22.982+0000 7f18c4400700 1 -- 172.21.15.12:0/3405065119 wait complete. 2024-03-29T11:33:22.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-03-29T11:33:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.200+0000 7fab1ba00700 1 Processor -- start 2024-03-29T11:33:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.200+0000 7fab1ba00700 1 -- start start 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.201+0000 7fab1ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab1415d9d0 0x7fab140d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.201+0000 7fab1ba00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fab140d16a0 0x7fab140d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab1ba00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab14161420 con 0x7fab140d2370 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab1ba00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab140d2820 con 0x7fab140d16a0 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab1ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab14161700 con 0x7fab1415d9d0 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab13e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fab140d16a0 0x7fab140d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab13e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fab140d16a0 0x7fab140d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56108/0 (socket says 172.21.15.12:56108) 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab13e00700 1 -- 172.21.15.12:0/143958908 learned_addr learned my addr 172.21.15.12:0/143958908 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab19000700 1 --2- 172.21.15.12:0/143958908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab1415d9d0 0x7fab140d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab13e00700 1 -- 172.21.15.12:0/143958908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab1415d9d0 msgr2=0x7fab140d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab13e00700 1 --2- 172.21.15.12:0/143958908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab1415d9d0 0x7fab140d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab13e00700 1 -- 172.21.15.12:0/143958908 >> v1:172.21.15.159:6789/0 conn(0x7fab140d2370 legacy=0x7fab1415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.202+0000 7fab13e00700 1 -- 172.21.15.12:0/143958908 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab141619e0 con 0x7fab140d16a0 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.203+0000 7fab13e00700 1 --2- 172.21.15.12:0/143958908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 0x7fab140d1e30 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7fab000098e0 tx=0x7fab00009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32bb7920992d5c3b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.203+0000 7fab13400700 1 -- 172.21.15.12:0/143958908 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0000b8a0 con 0x7fab140d16a0 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.203+0000 7fab13400700 1 -- 172.21.15.12:0/143958908 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0000ba00 con 0x7fab140d16a0 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.203+0000 7fab13400700 1 -- 172.21.15.12:0/143958908 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0000bdb0 con 0x7fab140d16a0 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.203+0000 7fab1ba00700 1 -- 172.21.15.12:0/143958908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 msgr2=0x7fab140d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.203+0000 7fab1ba00700 1 --2- 172.21.15.12:0/143958908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 0x7fab140d1e30 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7fab000098e0 tx=0x7fab00009bf0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.203+0000 7fab1ba00700 1 -- 172.21.15.12:0/143958908 shutdown_connections 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.203+0000 7fab1ba00700 1 --2- 172.21.15.12:0/143958908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab1415d9d0 0x7fab140d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.203+0000 7fab1ba00700 1 --2- 172.21.15.12:0/143958908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 0x7fab140d1e30 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.203+0000 7fab1ba00700 1 -- 172.21.15.12:0/143958908 >> 172.21.15.12:0/143958908 conn(0x7fab1408aff0 msgr2=0x7fab140d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.204+0000 7fab1ba00700 1 -- 172.21.15.12:0/143958908 shutdown_connections 2024-03-29T11:33:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.204+0000 7fab1ba00700 1 -- 172.21.15.12:0/143958908 wait complete. 2024-03-29T11:33:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.205+0000 7fab1ba00700 1 Processor -- start 2024-03-29T11:33:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.206+0000 7fab1ba00700 1 -- start start 2024-03-29T11:33:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.206+0000 7fab1ba00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 0x7fab1415f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.206+0000 7fab1ba00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab140d2370 0x7fab140ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.206+0000 7fab1ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab1415d9d0 0x7fab14151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.206+0000 7fab1ba00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1415e2b0 con 0x7fab140d16a0 2024-03-29T11:33:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.206+0000 7fab1ba00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1415e590 con 0x7fab140d2370 2024-03-29T11:33:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.206+0000 7fab1ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1415e870 con 0x7fab1415d9d0 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab19000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 0x7fab1415f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab19000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 0x7fab1415f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56120/0 (socket says 172.21.15.12:56120) 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab19000700 1 -- 172.21.15.12:0/3186244331 learned_addr learned my addr 172.21.15.12:0/3186244331 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab19a00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab1415d9d0 0x7fab14151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab13e00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab140d2370 0x7fab140ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab19000700 1 -- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab1415d9d0 msgr2=0x7fab14151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab19000700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab1415d9d0 0x7fab14151350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab19000700 1 -- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab140d2370 msgr2=0x7fab140ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab19000700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab140d2370 0x7fab140ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab19000700 1 -- 172.21.15.12:0/3186244331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab000094f0 con 0x7fab140d16a0 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.207+0000 7fab19000700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 0x7fab1415f350 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fab0800d800 tx=0x7fab0800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.208+0000 7fab10c00700 1 -- 172.21.15.12:0/3186244331 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080125f0 con 0x7fab140d16a0 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.208+0000 7fab10c00700 1 -- 172.21.15.12:0/3186244331 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab08012750 con 0x7fab140d16a0 2024-03-29T11:33:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.208+0000 7fab10c00700 1 -- 172.21.15.12:0/3186244331 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08012b00 con 0x7fab140d16a0 2024-03-29T11:33:23.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.208+0000 7fab1ba00700 1 -- 172.21.15.12:0/3186244331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1415eb50 con 0x7fab140d16a0 2024-03-29T11:33:23.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.208+0000 7fab1ba00700 1 -- 172.21.15.12:0/3186244331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1415f890 con 0x7fab140d16a0 2024-03-29T11:33:23.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.209+0000 7fab1ba00700 1 -- 172.21.15.12:0/3186244331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab140d1ca0 con 0x7fab140d16a0 2024-03-29T11:33:23.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.210+0000 7fab10c00700 1 -- 172.21.15.12:0/3186244331 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fab08012c60 con 0x7fab140d16a0 2024-03-29T11:33:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.211+0000 7fab10c00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7faaf406af00 0x7faaf406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.211+0000 7fab10c00700 1 -- 172.21.15.12:0/3186244331 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6653+0+0 (secure 0 0 0) 0x7fab08089f20 con 0x7fab140d16a0 2024-03-29T11:33:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.211+0000 7fab13e00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7faaf406af00 0x7faaf406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.212+0000 7fab13e00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7faaf406af00 0x7faaf406d3c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fab140d1ae0 tx=0x7fab00008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:23.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.215+0000 7fab10c00700 1 -- 172.21.15.12:0/3186244331 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fab08054950 con 0x7fab140d16a0 2024-03-29T11:33:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.480+0000 7fab1ba00700 1 -- 172.21.15.12:0/3186244331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fab1415f070 con 0x7fab140d16a0 2024-03-29T11:33:23.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T11:33:23.481+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T11:33:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.492+0000 7fab10c00700 1 -- 172.21.15.12:0/3186244331 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v131) v1 ==== 167+0+0 (secure 0 0 0) 0x7fab1415f070 con 0x7fab140d16a0 2024-03-29T11:33:23.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 -- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7faaf406af00 msgr2=0x7faaf406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7faaf406af00 0x7faaf406d3c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fab140d1ae0 tx=0x7fab00008030 comp rx=0 tx=0).stop 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 -- 172.21.15.12:0/3186244331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 msgr2=0x7fab1415f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 0x7fab1415f350 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fab0800d800 tx=0x7fab0800dbc0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 -- 172.21.15.12:0/3186244331 shutdown_connections 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7faaf406af00 0x7faaf406d3c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab1415d9d0 0x7fab14151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab140d16a0 0x7fab1415f350 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 --2- 172.21.15.12:0/3186244331 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab140d2370 0x7fab140ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 -- 172.21.15.12:0/3186244331 >> 172.21.15.12:0/3186244331 conn(0x7fab1408aff0 msgr2=0x7fab140b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 -- 172.21.15.12:0/3186244331 shutdown_connections 2024-03-29T11:33:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.495+0000 7fab1ba00700 1 -- 172.21.15.12:0/3186244331 wait complete. 2024-03-29T11:33:23.493 INFO:teuthology.orchestra.run.smithi012.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T11:33:23.496 INFO:tasks.ceph.mds.d.smithi159.stderr: -1777> 2024-03-29T11:32:53.383+0000 7efd3a600700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:33:23.510 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-03-29T11:33:23.541 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T11:33:23.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.713+0000 7f3e32600700 1 Processor -- start 2024-03-29T11:33:23.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.714+0000 7f3e32600700 1 -- start start 2024-03-29T11:33:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e32600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 0x7f3e2c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e32600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3e2c0cd4a0 0x7f3e2c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e32600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c0cc450 con 0x7f3e2c0cd020 2024-03-29T11:33:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e32600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e2c0cdae0 con 0x7f3e2c0cd4a0 2024-03-29T11:33:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e32600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0cddc0 con 0x7f3e2c0cbb30 2024-03-29T11:33:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e2aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3e2c0cd020 0x7f3e2c0b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42916/0 (socket says 172.21.15.12:42916) 2024-03-29T11:33:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e2aa00700 1 -- 172.21.15.12:0/1523127380 learned_addr learned my addr 172.21.15.12:0/1523127380 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e2b400700 1 --2- 172.21.15.12:0/1523127380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 0x7f3e2c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e2be00700 1 --2- 172.21.15.12:0/1523127380 >> v2:172.21.15.159:3300/0 conn(0x7f3e2c0cd4a0 0x7f3e2c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e2a000700 1 -- 172.21.15.12:0/1523127380 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267800244 0 0) 0x7f3e2c0cc450 con 0x7f3e2c0cd020 2024-03-29T11:33:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.715+0000 7f3e2a000700 1 -- 172.21.15.12:0/1523127380 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e100035b0 con 0x7f3e2c0cd020 2024-03-29T11:33:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.716+0000 7f3e2b400700 1 -- 172.21.15.12:0/1523127380 >> v2:172.21.15.159:3300/0 conn(0x7f3e2c0cd4a0 msgr2=0x7f3e2c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.716+0000 7f3e2b400700 1 --2- 172.21.15.12:0/1523127380 >> v2:172.21.15.159:3300/0 conn(0x7f3e2c0cd4a0 0x7f3e2c06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.716+0000 7f3e2b400700 1 -- 172.21.15.12:0/1523127380 >> v1:172.21.15.12:6789/0 conn(0x7f3e2c0cd020 legacy=0x7f3e2c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.716+0000 7f3e2b400700 1 -- 172.21.15.12:0/1523127380 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c0ce0a0 con 0x7f3e2c0cbb30 2024-03-29T11:33:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.716+0000 7f3e2b400700 1 --2- 172.21.15.12:0/1523127380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 0x7f3e2c0cbf10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009840 tx=0x7f3e1c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f9eaf02976eeb05 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.716+0000 7f3e2a000700 1 -- 172.21.15.12:0/1523127380 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c011830 con 0x7f3e2c0cbb30 2024-03-29T11:33:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.716+0000 7f3e2a000700 1 -- 172.21.15.12:0/1523127380 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c011990 con 0x7f3e2c0cbb30 2024-03-29T11:33:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.716+0000 7f3e2a000700 1 -- 172.21.15.12:0/1523127380 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c011d00 con 0x7f3e2c0cbb30 2024-03-29T11:33:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.717+0000 7f3e32600700 1 -- 172.21.15.12:0/1523127380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 msgr2=0x7f3e2c0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.717+0000 7f3e32600700 1 --2- 172.21.15.12:0/1523127380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 0x7f3e2c0cbf10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009840 tx=0x7f3e1c009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.717+0000 7f3e32600700 1 -- 172.21.15.12:0/1523127380 shutdown_connections 2024-03-29T11:33:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.717+0000 7f3e32600700 1 --2- 172.21.15.12:0/1523127380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 0x7f3e2c0cbf10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.717+0000 7f3e32600700 1 --2- 172.21.15.12:0/1523127380 >> v2:172.21.15.159:3300/0 conn(0x7f3e2c0cd4a0 0x7f3e2c06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.717+0000 7f3e32600700 1 -- 172.21.15.12:0/1523127380 >> 172.21.15.12:0/1523127380 conn(0x7f3e2c0b0d70 msgr2=0x7f3e2c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.717+0000 7f3e32600700 1 -- 172.21.15.12:0/1523127380 shutdown_connections 2024-03-29T11:33:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.717+0000 7f3e32600700 1 -- 172.21.15.12:0/1523127380 wait complete. 2024-03-29T11:33:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.719+0000 7f3e32600700 1 Processor -- start 2024-03-29T11:33:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.719+0000 7f3e32600700 1 -- start start 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e32600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 0x7f3e2c14a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e32600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e2c0cd020 0x7f3e2c15ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e32600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e2c0cd4a0 0x7f3e2c0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e32600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c156570 con 0x7f3e2c0cd020 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e32600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c155280 con 0x7f3e2c0cd4a0 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e32600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c156a20 con 0x7f3e2c0cbb30 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e2aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e2c0cd020 0x7f3e2c15ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e2aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e2c0cd020 0x7f3e2c15ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56134/0 (socket says 172.21.15.12:56134) 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e2aa00700 1 -- 172.21.15.12:0/328400915 learned_addr learned my addr 172.21.15.12:0/328400915 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e2b400700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 0x7f3e2c14a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.720+0000 7f3e2be00700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e2c0cd4a0 0x7f3e2c0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e2aa00700 1 -- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 msgr2=0x7f3e2c14a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:23.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e2aa00700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 0x7f3e2c14a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e2aa00700 1 -- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e2c0cd4a0 msgr2=0x7f3e2c0afd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:23.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e2aa00700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e2c0cd4a0 0x7f3e2c0afd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:23.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e2aa00700 1 -- 172.21.15.12:0/328400915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c0094a0 con 0x7f3e2c0cd020 2024-03-29T11:33:23.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e2aa00700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e2c0cd020 0x7f3e2c15ace0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800c450 tx=0x7f3e1800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:23.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e23400700 1 -- 172.21.15.12:0/328400915 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18005d70 con 0x7f3e2c0cd020 2024-03-29T11:33:23.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e23400700 1 -- 172.21.15.12:0/328400915 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1800ce20 con 0x7f3e2c0cd020 2024-03-29T11:33:23.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e32600700 1 -- 172.21.15.12:0/328400915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c1554d0 con 0x7f3e2c0cd020 2024-03-29T11:33:23.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e23400700 1 -- 172.21.15.12:0/328400915 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180072d0 con 0x7f3e2c0cd020 2024-03-29T11:33:23.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.721+0000 7f3e32600700 1 -- 172.21.15.12:0/328400915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2c155d70 con 0x7f3e2c0cd020 2024-03-29T11:33:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.723+0000 7f3e32600700 1 -- 172.21.15.12:0/328400915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2c067720 con 0x7f3e2c0cd020 2024-03-29T11:33:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.723+0000 7f3e23400700 1 -- 172.21.15.12:0/328400915 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f3e18007430 con 0x7f3e2c0cd020 2024-03-29T11:33:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.724+0000 7f3e23400700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e0c06aed0 0x7f3e0c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.724+0000 7f3e23400700 1 -- 172.21.15.12:0/328400915 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6739+0+0 (secure 0 0 0) 0x7f3e18088cd0 con 0x7f3e2c0cd020 2024-03-29T11:33:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.725+0000 7f3e2b400700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e0c06aed0 0x7f3e0c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.725+0000 7f3e2b400700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e0c06aed0 0x7f3e0c06d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009fd0 tx=0x7f3e1c01c040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.729+0000 7f3e23400700 1 -- 172.21.15.12:0/328400915 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e18053880 con 0x7f3e2c0cd020 2024-03-29T11:33:23.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:23.988+0000 7f3e32600700 1 -- 172.21.15.12:0/328400915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3e2c0ce640 con 0x7f3e2c0cd020 2024-03-29T11:33:24.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.496+0000 7f3e23400700 1 -- 172.21.15.12:0/328400915 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v132) v1 ==== 107+0+0 (secure 0 0 0) 0x7f3e18017020 con 0x7f3e2c0cd020 2024-03-29T11:33:24.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 -- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e0c06aed0 msgr2=0x7f3e0c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:24.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e0c06aed0 0x7f3e0c06d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009fd0 tx=0x7f3e1c01c040 comp rx=0 tx=0).stop 2024-03-29T11:33:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 -- 172.21.15.12:0/328400915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e2c0cd020 msgr2=0x7f3e2c15ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e2c0cd020 0x7f3e2c15ace0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800c450 tx=0x7f3e1800c760 comp rx=0 tx=0).stop 2024-03-29T11:33:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 -- 172.21.15.12:0/328400915 shutdown_connections 2024-03-29T11:33:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e0c06aed0 0x7f3e0c06d390 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e2c0cd4a0 0x7f3e2c0afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e2c0cbb30 0x7f3e2c14a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 --2- 172.21.15.12:0/328400915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e2c0cd020 0x7f3e2c15ace0 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 -- 172.21.15.12:0/328400915 >> 172.21.15.12:0/328400915 conn(0x7f3e2c0b0d70 msgr2=0x7f3e2c06e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 -- 172.21.15.12:0/328400915 shutdown_connections 2024-03-29T11:33:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.499+0000 7f3e32600700 1 -- 172.21.15.12:0/328400915 wait complete. 2024-03-29T11:33:24.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-03-29T11:33:24.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.723+0000 7fa316400700 1 Processor -- start 2024-03-29T11:33:24.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.723+0000 7fa316400700 1 -- start start 2024-03-29T11:33:24.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.724+0000 7fa316400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3100b5580 0x7fa31015e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:24.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.724+0000 7fa316400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa31015f260 0x7fa3100d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:24.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.724+0000 7fa316400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa31015f6e0 0x7fa3100d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:24.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.724+0000 7fa316400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31015ee70 con 0x7fa31015f6e0 2024-03-29T11:33:24.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.724+0000 7fa316400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3100ce430 con 0x7fa31015f260 2024-03-29T11:33:24.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.724+0000 7fa316400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100ce710 con 0x7fa3100b5580 2024-03-29T11:33:24.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa31015f6e0 0x7fa3100d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa31015f6e0 0x7fa3100d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56140/0 (socket says 172.21.15.12:56140) 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30fe00700 1 -- 172.21.15.12:0/3970113680 learned_addr learned my addr 172.21.15.12:0/3970113680 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30ea00700 1 --2- 172.21.15.12:0/3970113680 >> v2:172.21.15.159:3300/0 conn(0x7fa31015f260 0x7fa3100d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30f400700 1 --2- 172.21.15.12:0/3970113680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3100b5580 0x7fa31015e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30fe00700 1 -- 172.21.15.12:0/3970113680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3100b5580 msgr2=0x7fa31015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30fe00700 1 --2- 172.21.15.12:0/3970113680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3100b5580 0x7fa31015e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30fe00700 1 -- 172.21.15.12:0/3970113680 >> v2:172.21.15.159:3300/0 conn(0x7fa31015f260 msgr2=0x7fa3100d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30fe00700 1 --2- 172.21.15.12:0/3970113680 >> v2:172.21.15.159:3300/0 conn(0x7fa31015f260 0x7fa3100d1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30fe00700 1 -- 172.21.15.12:0/3970113680 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100ce9f0 con 0x7fa31015f6e0 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30ea00700 1 --2- 172.21.15.12:0/3970113680 >> v2:172.21.15.159:3300/0 conn(0x7fa31015f260 0x7fa3100d1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.725+0000 7fa30fe00700 1 --2- 172.21.15.12:0/3970113680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa31015f6e0 0x7fa3100d2020 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fa31008c490 tx=0x7fa30800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f88a07f87a04700 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa30e000700 1 -- 172.21.15.12:0/3970113680 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30800e070 con 0x7fa31015f6e0 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa30e000700 1 -- 172.21.15.12:0/3970113680 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30800b9c0 con 0x7fa31015f6e0 2024-03-29T11:33:24.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa30e000700 1 -- 172.21.15.12:0/3970113680 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30800bd70 con 0x7fa31015f6e0 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa316400700 1 -- 172.21.15.12:0/3970113680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa31015f6e0 msgr2=0x7fa3100d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa316400700 1 --2- 172.21.15.12:0/3970113680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa31015f6e0 0x7fa3100d2020 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fa31008c490 tx=0x7fa30800b330 comp rx=0 tx=0).stop 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa316400700 1 -- 172.21.15.12:0/3970113680 shutdown_connections 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa316400700 1 --2- 172.21.15.12:0/3970113680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa31015f6e0 0x7fa3100d2020 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa316400700 1 --2- 172.21.15.12:0/3970113680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3100b5580 0x7fa31015e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa316400700 1 --2- 172.21.15.12:0/3970113680 >> v2:172.21.15.159:3300/0 conn(0x7fa31015f260 0x7fa3100d1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa316400700 1 -- 172.21.15.12:0/3970113680 >> 172.21.15.12:0/3970113680 conn(0x7fa31008aff0 msgr2=0x7fa3100d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.726+0000 7fa316400700 1 -- 172.21.15.12:0/3970113680 shutdown_connections 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.727+0000 7fa316400700 1 -- 172.21.15.12:0/3970113680 wait complete. 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.728+0000 7fa316400700 1 Processor -- start 2024-03-29T11:33:24.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.728+0000 7fa316400700 1 -- start start 2024-03-29T11:33:24.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.729+0000 7fa316400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3100b5580 0x7fa3100d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:24.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.729+0000 7fa316400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa31015f260 0x7fa3100cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:24.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.729+0000 7fa316400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa31015f6e0 0x7fa3100cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.729+0000 7fa316400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100cb110 con 0x7fa3100b5580 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.729+0000 7fa316400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100ccc30 con 0x7fa31015f6e0 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.729+0000 7fa316400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100cce30 con 0x7fa31015f260 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.729+0000 7fa30f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3100b5580 0x7fa3100d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3100b5580 0x7fa3100d02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56150/0 (socket says 172.21.15.12:56150) 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30f400700 1 -- 172.21.15.12:0/2181092844 learned_addr learned my addr 172.21.15.12:0/2181092844 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30fe00700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa31015f6e0 0x7fa3100cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30ea00700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa31015f260 0x7fa3100cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30f400700 1 -- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa31015f260 msgr2=0x7fa3100cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30f400700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa31015f260 0x7fa3100cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30f400700 1 -- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa31015f6e0 msgr2=0x7fa3100cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30f400700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa31015f6e0 0x7fa3100cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30f400700 1 -- 172.21.15.12:0/2181092844 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308009b30 con 0x7fa3100b5580 2024-03-29T11:33:24.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30ea00700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa31015f260 0x7fa3100cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:24.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa30f400700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3100b5580 0x7fa3100d02c0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7fa30000e810 tx=0x7fa30000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:24.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.730+0000 7fa307400700 1 -- 172.21.15.12:0/2181092844 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300013610 con 0x7fa3100b5580 2024-03-29T11:33:24.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.731+0000 7fa307400700 1 -- 172.21.15.12:0/2181092844 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa300013790 con 0x7fa3100b5580 2024-03-29T11:33:24.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.731+0000 7fa316400700 1 -- 172.21.15.12:0/2181092844 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3100ca770 con 0x7fa3100b5580 2024-03-29T11:33:24.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.731+0000 7fa307400700 1 -- 172.21.15.12:0/2181092844 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300013b40 con 0x7fa3100b5580 2024-03-29T11:33:24.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.731+0000 7fa316400700 1 -- 172.21.15.12:0/2181092844 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3100cad30 con 0x7fa3100b5580 2024-03-29T11:33:24.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.732+0000 7fa316400700 1 -- 172.21.15.12:0/2181092844 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3100cb380 con 0x7fa3100b5580 2024-03-29T11:33:24.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.733+0000 7fa307400700 1 -- 172.21.15.12:0/2181092844 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa300013ca0 con 0x7fa3100b5580 2024-03-29T11:33:24.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.734+0000 7fa307400700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2f006aed0 0x7fa2f006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:24.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.734+0000 7fa307400700 1 -- 172.21.15.12:0/2181092844 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6739+0+0 (secure 0 0 0) 0x7fa300089800 con 0x7fa3100b5580 2024-03-29T11:33:24.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.734+0000 7fa30ea00700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2f006aed0 0x7fa2f006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:24.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.735+0000 7fa30ea00700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2f006aed0 0x7fa2f006d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa31015da20 tx=0x7fa2fc008040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.739+0000 7fa307400700 1 -- 172.21.15.12:0/2181092844 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa300054150 con 0x7fa3100b5580 2024-03-29T11:33:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:24.960+0000 7fa316400700 1 -- 172.21.15.12:0/2181092844 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa310067720 con 0x7fa3100b5580 2024-03-29T11:33:25.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.757+0000 7fa307400700 1 -- 172.21.15.12:0/2181092844 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v259) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa300009440 con 0x7fa3100b5580 2024-03-29T11:33:25.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 -- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2f006aed0 msgr2=0x7fa2f006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:25.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2f006aed0 0x7fa2f006d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa31015da20 tx=0x7fa2fc008040 comp rx=0 tx=0).stop 2024-03-29T11:33:25.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 -- 172.21.15.12:0/2181092844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3100b5580 msgr2=0x7fa3100d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:25.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3100b5580 0x7fa3100d02c0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7fa30000e810 tx=0x7fa30000eb20 comp rx=0 tx=0).stop 2024-03-29T11:33:25.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 -- 172.21.15.12:0/2181092844 shutdown_connections 2024-03-29T11:33:25.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2f006aed0 0x7fa2f006d390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:25.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa31015f6e0 0x7fa3100cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:25.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3100b5580 0x7fa3100d02c0 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:25.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 --2- 172.21.15.12:0/2181092844 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa31015f260 0x7fa3100cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:25.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 -- 172.21.15.12:0/2181092844 >> 172.21.15.12:0/2181092844 conn(0x7fa31008aff0 msgr2=0x7fa310161620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:25.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 -- 172.21.15.12:0/2181092844 shutdown_connections 2024-03-29T11:33:25.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.762+0000 7fa316400700 1 -- 172.21.15.12:0/2181092844 wait complete. 2024-03-29T11:33:25.760 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'backup_fs_metadata' removed 2024-03-29T11:33:25.780 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-03-29T11:33:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.993+0000 7fb6a5800700 1 Processor -- start 2024-03-29T11:33:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.993+0000 7fb6a5800700 1 -- start start 2024-03-29T11:33:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb6a5800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6a006e9c0 0x7fb6a015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb6a5800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a015b160 0x7fb6a00b2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb6a5800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a0070980 con 0x7fb6a01580c0 2024-03-29T11:33:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb6a5800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a0071570 con 0x7fb6a006e9c0 2024-03-29T11:33:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb6a5800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a0071990 con 0x7fb6a015b160 2024-03-29T11:33:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb69e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6a006e9c0 0x7fb6a015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb69e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6a006e9c0 0x7fb6a015ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56152/0 (socket says 172.21.15.12:56152) 2024-03-29T11:33:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb69e000700 1 -- 172.21.15.12:0/3251424975 learned_addr learned my addr 172.21.15.12:0/3251424975 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb69f400700 1 --2- 172.21.15.12:0/3251424975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a015b160 0x7fb6a00b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb69e000700 1 -- 172.21.15.12:0/3251424975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a015b160 msgr2=0x7fb6a00b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb69e000700 1 --2- 172.21.15.12:0/3251424975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a015b160 0x7fb6a00b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb69e000700 1 -- 172.21.15.12:0/3251424975 >> v1:172.21.15.159:6789/0 conn(0x7fb6a01580c0 legacy=0x7fb6a006e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.995+0000 7fb69e000700 1 -- 172.21.15.12:0/3251424975 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a0070bd0 con 0x7fb6a006e9c0 2024-03-29T11:33:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.996+0000 7fb69e000700 1 --2- 172.21.15.12:0/3251424975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 0x7fb6a015ac20 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fb68c0098f0 tx=0x7fb68c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554b9108d310e5b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.996+0000 7fb69d600700 1 -- 172.21.15.12:0/3251424975 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c011830 con 0x7fb6a006e9c0 2024-03-29T11:33:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.996+0000 7fb69d600700 1 -- 172.21.15.12:0/3251424975 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c011990 con 0x7fb6a006e9c0 2024-03-29T11:33:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.996+0000 7fb69d600700 1 -- 172.21.15.12:0/3251424975 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c011d20 con 0x7fb6a006e9c0 2024-03-29T11:33:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.996+0000 7fb6a5800700 1 -- 172.21.15.12:0/3251424975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 msgr2=0x7fb6a015ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.996+0000 7fb6a5800700 1 --2- 172.21.15.12:0/3251424975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 0x7fb6a015ac20 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fb68c0098f0 tx=0x7fb68c009c00 comp rx=0 tx=0).stop 2024-03-29T11:33:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.997+0000 7fb6a5800700 1 -- 172.21.15.12:0/3251424975 shutdown_connections 2024-03-29T11:33:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.997+0000 7fb6a5800700 1 --2- 172.21.15.12:0/3251424975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a015b160 0x7fb6a00b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.997+0000 7fb6a5800700 1 --2- 172.21.15.12:0/3251424975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 0x7fb6a015ac20 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.997+0000 7fb6a5800700 1 -- 172.21.15.12:0/3251424975 >> 172.21.15.12:0/3251424975 conn(0x7fb6a00cf210 msgr2=0x7fb6a00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.997+0000 7fb6a5800700 1 -- 172.21.15.12:0/3251424975 shutdown_connections 2024-03-29T11:33:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.997+0000 7fb6a5800700 1 -- 172.21.15.12:0/3251424975 wait complete. 2024-03-29T11:33:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.998+0000 7fb6a5800700 1 Processor -- start 2024-03-29T11:33:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.999+0000 7fb6a5800700 1 -- start start 2024-03-29T11:33:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.999+0000 7fb6a5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 0x7fb6a00cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.999+0000 7fb6a5800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a01580c0 0x7fb6a0099550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.999+0000 7fb6a5800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015b160 0x7fb6a0154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.999+0000 7fb6a5800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00cc150 con 0x7fb6a006e9c0 2024-03-29T11:33:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.999+0000 7fb6a5800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00cc430 con 0x7fb6a015b160 2024-03-29T11:33:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:25.999+0000 7fb6a5800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00cc710 con 0x7fb6a01580c0 2024-03-29T11:33:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 0x7fb6a00cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 0x7fb6a00cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56162/0 (socket says 172.21.15.12:56162) 2024-03-29T11:33:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69ea00700 1 -- 172.21.15.12:0/3732686721 learned_addr learned my addr 172.21.15.12:0/3732686721 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69f400700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015b160 0x7fb6a0154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69e000700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a01580c0 0x7fb6a0099550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69ea00700 1 -- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a01580c0 msgr2=0x7fb6a0099550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69ea00700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a01580c0 0x7fb6a0099550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69ea00700 1 -- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015b160 msgr2=0x7fb6a0154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69ea00700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015b160 0x7fb6a0154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69ea00700 1 -- 172.21.15.12:0/3732686721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c009550 con 0x7fb6a006e9c0 2024-03-29T11:33:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69f400700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015b160 0x7fb6a0154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69e000700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a01580c0 0x7fb6a0099550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.000+0000 7fb69ea00700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 0x7fb6a00cb900 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c4e0 tx=0x7fb69000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.001+0000 7fb696a00700 1 -- 172.21.15.12:0/3732686721 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6900052d0 con 0x7fb6a006e9c0 2024-03-29T11:33:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.001+0000 7fb696a00700 1 -- 172.21.15.12:0/3732686721 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb690005430 con 0x7fb6a006e9c0 2024-03-29T11:33:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.001+0000 7fb696a00700 1 -- 172.21.15.12:0/3732686721 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6900038c0 con 0x7fb6a006e9c0 2024-03-29T11:33:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.001+0000 7fb6a5800700 1 -- 172.21.15.12:0/3732686721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a00cc9f0 con 0x7fb6a006e9c0 2024-03-29T11:33:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.001+0000 7fb6a5800700 1 -- 172.21.15.12:0/3732686721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a00cbe40 con 0x7fb6a006e9c0 2024-03-29T11:33:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.002+0000 7fb6a5800700 1 -- 172.21.15.12:0/3732686721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a00b4710 con 0x7fb6a006e9c0 2024-03-29T11:33:26.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.003+0000 7fb696a00700 1 -- 172.21.15.12:0/3732686721 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb690003a20 con 0x7fb6a006e9c0 2024-03-29T11:33:26.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.004+0000 7fb696a00700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb68006aed0 0x7fb68006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:26.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.004+0000 7fb696a00700 1 -- 172.21.15.12:0/3732686721 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6295+0+0 (secure 0 0 0) 0x7fb690088da0 con 0x7fb6a006e9c0 2024-03-29T11:33:26.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.004+0000 7fb69e000700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb68006aed0 0x7fb68006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:26.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.005+0000 7fb69e000700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb68006aed0 0x7fb68006d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb6a006c520 tx=0x7fb68c008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:26.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.008+0000 7fb696a00700 1 -- 172.21.15.12:0/3732686721 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb690053a90 con 0x7fb6a006e9c0 2024-03-29T11:33:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.226+0000 7fb6a5800700 1 -- 172.21.15.12:0/3732686721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb6a0067720 con 0x7fb6a006e9c0 2024-03-29T11:33:26.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.763+0000 7fb696a00700 1 -- 172.21.15.12:0/3732686721 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v260) v1 ==== 178+0+0 (secure 0 0 0) 0x7fb6900170d0 con 0x7fb6a006e9c0 2024-03-29T11:33:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.767+0000 7fb6a5800700 1 -- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb68006aed0 msgr2=0x7fb68006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.767+0000 7fb6a5800700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb68006aed0 0x7fb68006d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb6a006c520 tx=0x7fb68c008030 comp rx=0 tx=0).stop 2024-03-29T11:33:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.767+0000 7fb6a5800700 1 -- 172.21.15.12:0/3732686721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 msgr2=0x7fb6a00cb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.767+0000 7fb6a5800700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 0x7fb6a00cb900 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c4e0 tx=0x7fb69000c8a0 comp rx=0 tx=0).stop 2024-03-29T11:33:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.767+0000 7fb6a5800700 1 -- 172.21.15.12:0/3732686721 shutdown_connections 2024-03-29T11:33:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.767+0000 7fb6a5800700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb68006aed0 0x7fb68006d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.767+0000 7fb6a5800700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015b160 0x7fb6a0154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.767+0000 7fb6a5800700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6a006e9c0 0x7fb6a00cb900 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.767+0000 7fb6a5800700 1 --2- 172.21.15.12:0/3732686721 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a01580c0 0x7fb6a0099550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.767+0000 7fb6a5800700 1 -- 172.21.15.12:0/3732686721 >> 172.21.15.12:0/3732686721 conn(0x7fb6a00cf210 msgr2=0x7fb6a015a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.768+0000 7fb6a5800700 1 -- 172.21.15.12:0/3732686721 shutdown_connections 2024-03-29T11:33:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:26.768+0000 7fb6a5800700 1 -- 172.21.15.12:0/3732686721 wait complete. 2024-03-29T11:33:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'backup_fs_data' removed 2024-03-29T11:33:26.785 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T11:33:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.001+0000 7f214aa00700 1 Processor -- start 2024-03-29T11:33:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.001+0000 7f214aa00700 1 -- start start 2024-03-29T11:33:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.002+0000 7f214aa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f21440d16a0 0x7f21440d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.002+0000 7f214aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21440d2370 0x7f21440ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.002+0000 7f214aa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2144161420 con 0x7f214415d9d0 2024-03-29T11:33:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.002+0000 7f214aa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21440d4340 con 0x7f21440d2370 2024-03-29T11:33:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.002+0000 7f214aa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214415edc0 con 0x7f21440d16a0 2024-03-29T11:33:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.002+0000 7f2148a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21440d2370 0x7f21440ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.002+0000 7f2143e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f214415d9d0 0x7f21440d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42924/0 (socket says 172.21.15.12:42924) 2024-03-29T11:33:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.002+0000 7f2143e00700 1 -- 172.21.15.12:0/4230374717 learned_addr learned my addr 172.21.15.12:0/4230374717 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.002+0000 7f2148a00700 1 --2- 172.21.15.12:0/4230374717 >> v2:172.21.15.12:3300/0 conn(0x7f21440d2370 0x7f21440ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56170/0 (socket says 172.21.15.12:56170) 2024-03-29T11:33:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.003+0000 7f2143400700 1 --2- 172.21.15.12:0/4230374717 >> v2:172.21.15.159:3300/0 conn(0x7f21440d16a0 0x7f21440d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.003+0000 7f2148a00700 1 -- 172.21.15.12:0/4230374717 >> v2:172.21.15.159:3300/0 conn(0x7f21440d16a0 msgr2=0x7f21440d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.003+0000 7f2148a00700 1 --2- 172.21.15.12:0/4230374717 >> v2:172.21.15.159:3300/0 conn(0x7f21440d16a0 0x7f21440d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.003+0000 7f2148a00700 1 -- 172.21.15.12:0/4230374717 >> v1:172.21.15.12:6789/0 conn(0x7f214415d9d0 legacy=0x7f21440d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.003+0000 7f2142a00700 1 -- 172.21.15.12:0/4230374717 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130566635 0 0) 0x7f2144161420 con 0x7f214415d9d0 2024-03-29T11:33:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.003+0000 7f2148a00700 1 -- 172.21.15.12:0/4230374717 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2144161700 con 0x7f21440d2370 2024-03-29T11:33:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.003+0000 7f2148a00700 1 --2- 172.21.15.12:0/4230374717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 0x7f21440ce360 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f21440d12f0 tx=0x7f213c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85227a7b20ded5b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.003+0000 7f2142a00700 1 -- 172.21.15.12:0/4230374717 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c00e070 con 0x7f21440d2370 2024-03-29T11:33:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.003+0000 7f2142a00700 1 -- 172.21.15.12:0/4230374717 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c00bbd0 con 0x7f21440d2370 2024-03-29T11:33:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.003+0000 7f2142a00700 1 -- 172.21.15.12:0/4230374717 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c012430 con 0x7f21440d2370 2024-03-29T11:33:27.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.004+0000 7f214aa00700 1 -- 172.21.15.12:0/4230374717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 msgr2=0x7f21440ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.004+0000 7f214aa00700 1 --2- 172.21.15.12:0/4230374717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 0x7f21440ce360 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f21440d12f0 tx=0x7f213c00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.004+0000 7f214aa00700 1 -- 172.21.15.12:0/4230374717 shutdown_connections 2024-03-29T11:33:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.004+0000 7f214aa00700 1 --2- 172.21.15.12:0/4230374717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 0x7f21440ce360 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.004+0000 7f214aa00700 1 --2- 172.21.15.12:0/4230374717 >> v2:172.21.15.159:3300/0 conn(0x7f21440d16a0 0x7f21440d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.004+0000 7f214aa00700 1 -- 172.21.15.12:0/4230374717 >> 172.21.15.12:0/4230374717 conn(0x7f214408aff0 msgr2=0x7f21440d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.004+0000 7f214aa00700 1 -- 172.21.15.12:0/4230374717 shutdown_connections 2024-03-29T11:33:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.004+0000 7f214aa00700 1 -- 172.21.15.12:0/4230374717 wait complete. 2024-03-29T11:33:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.006+0000 7f214aa00700 1 Processor -- start 2024-03-29T11:33:27.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.006+0000 7f214aa00700 1 -- start start 2024-03-29T11:33:27.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f214aa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440d16a0 0x7f21441529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f214aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 0x7f214406b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f214aa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f214415d9d0 0x7f214406be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f214aa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440ccdf0 con 0x7f21440d2370 2024-03-29T11:33:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f214aa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440ca5f0 con 0x7f214415d9d0 2024-03-29T11:33:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f214aa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2144152ef0 con 0x7f21440d16a0 2024-03-29T11:33:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f2143400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 0x7f214406b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f2143400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 0x7f214406b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56178/0 (socket says 172.21.15.12:56178) 2024-03-29T11:33:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f2143400700 1 -- 172.21.15.12:0/1525631827 learned_addr learned my addr 172.21.15.12:0/1525631827 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f2143e00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440d16a0 0x7f21441529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.007+0000 7f2148a00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f214415d9d0 0x7f214406be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f2143400700 1 -- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440d16a0 msgr2=0x7f21441529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f2143400700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440d16a0 0x7f21441529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f2143400700 1 -- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f214415d9d0 msgr2=0x7f214406be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f2143400700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f214415d9d0 0x7f214406be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f2143400700 1 -- 172.21.15.12:0/1525631827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c009a70 con 0x7f21440d2370 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f2143400700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 0x7f214406b900 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f213000e840 tx=0x7f213000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f213be00700 1 -- 172.21.15.12:0/1525631827 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213000c070 con 0x7f21440d2370 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f213be00700 1 -- 172.21.15.12:0/1525631827 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2130013610 con 0x7f21440d2370 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f213be00700 1 -- 172.21.15.12:0/1525631827 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21300139c0 con 0x7f21440d2370 2024-03-29T11:33:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f214aa00700 1 -- 172.21.15.12:0/1525631827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21440ca840 con 0x7f21440d2370 2024-03-29T11:33:27.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.008+0000 7f214aa00700 1 -- 172.21.15.12:0/1525631827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21440cae00 con 0x7f21440d2370 2024-03-29T11:33:27.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.009+0000 7f214aa00700 1 -- 172.21.15.12:0/1525631827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21440cb450 con 0x7f21440d2370 2024-03-29T11:33:27.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.010+0000 7f213be00700 1 -- 172.21.15.12:0/1525631827 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2130013b20 con 0x7f21440d2370 2024-03-29T11:33:27.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.011+0000 7f213be00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 0x7f212406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.011+0000 7f213be00700 1 -- 172.21.15.12:0/1525631827 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 5875+0+0 (secure 0 0 0) 0x7f2130089640 con 0x7f21440d2370 2024-03-29T11:33:27.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.012+0000 7f2143e00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 0x7f212406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.012+0000 7f2143e00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 0x7f212406d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2134000f60 tx=0x7f2134007e70 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:27.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.015+0000 7f213be00700 1 -- 172.21.15.12:0/1525631827 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2130054260 con 0x7f21440d2370 2024-03-29T11:33:27.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.229+0000 7f214aa00700 1 -- 172.21.15.12:0/1525631827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f21440d1ca0 con 0x7f21440d2370 2024-03-29T11:33:27.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.234+0000 7f213be00700 1 -- 172.21.15.12:0/1525631827 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v85) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2130057880 con 0x7f21440d2370 2024-03-29T11:33:27.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 -- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 msgr2=0x7f212406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 0x7f212406d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2134000f60 tx=0x7f2134007e70 comp rx=0 tx=0).stop 2024-03-29T11:33:27.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 -- 172.21.15.12:0/1525631827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 msgr2=0x7f214406b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 0x7f214406b900 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f213000e840 tx=0x7f213000eb50 comp rx=0 tx=0).stop 2024-03-29T11:33:27.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 -- 172.21.15.12:0/1525631827 shutdown_connections 2024-03-29T11:33:27.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f212406aed0 0x7f212406d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f214415d9d0 0x7f214406be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21440d16a0 0x7f21441529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 --2- 172.21.15.12:0/1525631827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21440d2370 0x7f214406b900 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 -- 172.21.15.12:0/1525631827 >> 172.21.15.12:0/1525631827 conn(0x7f214408aff0 msgr2=0x7f21440b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:27.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 -- 172.21.15.12:0/1525631827 shutdown_connections 2024-03-29T11:33:27.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.236+0000 7f214aa00700 1 -- 172.21.15.12:0/1525631827 wait complete. 2024-03-29T11:33:27.252 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T11:33:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.460+0000 7fb1ffc00700 1 Processor -- start 2024-03-29T11:33:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.461+0000 7fb1ffc00700 1 -- start start 2024-03-29T11:33:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.462+0000 7fb1ffc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb1f8160e50 0x7fb1f80b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.462+0000 7fb1ffc00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f80b3cb0 con 0x7fb1f80d3f20 2024-03-29T11:33:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.462+0000 7fb1ffc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f80cef00 con 0x7fb1f80b3f80 2024-03-29T11:33:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.462+0000 7fb1ffc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f8161230 con 0x7fb1f8160e50 2024-03-29T11:33:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.462+0000 7fb1f7e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb1f80d3f20 0x7fb1f80b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42936/0 (socket says 172.21.15.12:42936) 2024-03-29T11:33:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.462+0000 7fb1f7e00700 1 -- 172.21.15.12:0/4197158346 learned_addr learned my addr 172.21.15.12:0/4197158346 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.462+0000 7fb1fd200700 1 --2- 172.21.15.12:0/4197158346 >> v2:172.21.15.159:3300/0 conn(0x7fb1f8160e50 0x7fb1f80b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.462+0000 7fb1f7400700 1 -- 172.21.15.12:0/4197158346 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446678187 0 0) 0x7fb1f80b3cb0 con 0x7fb1f80d3f20 2024-03-29T11:33:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.462+0000 7fb1f7400700 1 -- 172.21.15.12:0/4197158346 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e00035b0 con 0x7fb1f80d3f20 2024-03-29T11:33:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.463+0000 7fb1fd200700 1 -- 172.21.15.12:0/4197158346 >> v1:172.21.15.159:6789/0 conn(0x7fb1f80b3f80 legacy=0x7fb1f80b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.463+0000 7fb1fd200700 1 -- 172.21.15.12:0/4197158346 >> v1:172.21.15.12:6789/0 conn(0x7fb1f80d3f20 legacy=0x7fb1f80b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.463+0000 7fb1f7400700 1 -- 172.21.15.12:0/4197158346 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163096027 0 0) 0x7fb1f80cef00 con 0x7fb1f80b3f80 2024-03-29T11:33:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.463+0000 7fb1fd200700 1 -- 172.21.15.12:0/4197158346 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f80d2a80 con 0x7fb1f8160e50 2024-03-29T11:33:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.463+0000 7fb1fd200700 1 --2- 172.21.15.12:0/4197158346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1f8160e50 0x7fb1f80b4e90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec009840 tx=0x7fb1ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9735ddc5c610d3e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.463+0000 7fb1f7400700 1 -- 172.21.15.12:0/4197158346 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec004230 con 0x7fb1f8160e50 2024-03-29T11:33:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.463+0000 7fb1f7400700 1 -- 172.21.15.12:0/4197158346 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec004390 con 0x7fb1f8160e50 2024-03-29T11:33:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.463+0000 7fb1f7400700 1 -- 172.21.15.12:0/4197158346 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec0036a0 con 0x7fb1f8160e50 2024-03-29T11:33:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.464+0000 7fb1ffc00700 1 -- 172.21.15.12:0/4197158346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1f8160e50 msgr2=0x7fb1f80b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.464+0000 7fb1ffc00700 1 --2- 172.21.15.12:0/4197158346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1f8160e50 0x7fb1f80b4e90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec009840 tx=0x7fb1ec009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.464+0000 7fb1ffc00700 1 -- 172.21.15.12:0/4197158346 shutdown_connections 2024-03-29T11:33:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.464+0000 7fb1ffc00700 1 --2- 172.21.15.12:0/4197158346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1f8160e50 0x7fb1f80b4e90 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.464+0000 7fb1ffc00700 1 -- 172.21.15.12:0/4197158346 >> 172.21.15.12:0/4197158346 conn(0x7fb1f808aff0 msgr2=0x7fb1f80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.464+0000 7fb1ffc00700 1 -- 172.21.15.12:0/4197158346 shutdown_connections 2024-03-29T11:33:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.464+0000 7fb1ffc00700 1 -- 172.21.15.12:0/4197158346 wait complete. 2024-03-29T11:33:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.466+0000 7fb1ffc00700 1 Processor -- start 2024-03-29T11:33:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.467+0000 7fb1ffc00700 1 -- start start 2024-03-29T11:33:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1ffc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f80b3f80 0x7fb1f80c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1ffc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1f8160e50 0x7fb1f80b87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1ffc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1f80b8d30 0x7fb1f81443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1ffc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f815e680 con 0x7fb1f80b3f80 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1ffc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f815da70 con 0x7fb1f8160e50 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1ffc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f815dce0 con 0x7fb1f80b8d30 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1fd200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f80b3f80 0x7fb1f80c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1fd200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f80b3f80 0x7fb1f80c9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56192/0 (socket says 172.21.15.12:56192) 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1fd200700 1 -- 172.21.15.12:0/618604510 learned_addr learned my addr 172.21.15.12:0/618604510 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1f7e00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1f8160e50 0x7fb1f80b87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1fdc00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1f80b8d30 0x7fb1f81443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1fd200700 1 -- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1f80b8d30 msgr2=0x7fb1f81443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1fd200700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1f80b8d30 0x7fb1f81443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1fd200700 1 -- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1f8160e50 msgr2=0x7fb1f80b87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1fd200700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1f8160e50 0x7fb1f80b87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1fd200700 1 -- 172.21.15.12:0/618604510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec0094a0 con 0x7fb1f80b3f80 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.468+0000 7fb1f7e00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1f8160e50 0x7fb1f80b87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.469+0000 7fb1fd200700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f80b3f80 0x7fb1f80c9870 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec00a010 tx=0x7fb1ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.469+0000 7fb1f4c00700 1 -- 172.21.15.12:0/618604510 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec003e00 con 0x7fb1f80b3f80 2024-03-29T11:33:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.469+0000 7fb1f4c00700 1 -- 172.21.15.12:0/618604510 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec01f070 con 0x7fb1f80b3f80 2024-03-29T11:33:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.469+0000 7fb1ffc00700 1 -- 172.21.15.12:0/618604510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f815dfc0 con 0x7fb1f80b3f80 2024-03-29T11:33:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.469+0000 7fb1f4c00700 1 -- 172.21.15.12:0/618604510 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec011870 con 0x7fb1f80b3f80 2024-03-29T11:33:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.469+0000 7fb1ffc00700 1 -- 172.21.15.12:0/618604510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f8147130 con 0x7fb1f80b3f80 2024-03-29T11:33:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.470+0000 7fb1ffc00700 1 -- 172.21.15.12:0/618604510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f8067720 con 0x7fb1f80b3f80 2024-03-29T11:33:27.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.471+0000 7fb1f4c00700 1 -- 172.21.15.12:0/618604510 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb1ec021030 con 0x7fb1f80b3f80 2024-03-29T11:33:27.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.472+0000 7fb1f4c00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb1d806aed0 0x7fb1d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.472+0000 7fb1f4c00700 1 -- 172.21.15.12:0/618604510 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 5875+0+0 (secure 0 0 0) 0x7fb1ec08ad20 con 0x7fb1f80b3f80 2024-03-29T11:33:27.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.473+0000 7fb1f7e00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb1d806aed0 0x7fb1d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.473+0000 7fb1f7e00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb1d806aed0 0x7fb1d806d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8144280 tx=0x7fb1e4009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.476+0000 7fb1f4c00700 1 -- 172.21.15.12:0/618604510 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb1ec055ae0 con 0x7fb1f80b3f80 2024-03-29T11:33:27.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.691+0000 7fb1ffc00700 1 -- 172.21.15.12:0/618604510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb1f8067720 con 0x7fb1f80b3f80 2024-03-29T11:33:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.696+0000 7fb1f4c00700 1 -- 172.21.15.12:0/618604510 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v86) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb1ec059100 con 0x7fb1f80b3f80 2024-03-29T11:33:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 -- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb1d806aed0 msgr2=0x7fb1d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb1d806aed0 0x7fb1d806d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8144280 tx=0x7fb1e4009040 comp rx=0 tx=0).stop 2024-03-29T11:33:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 -- 172.21.15.12:0/618604510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f80b3f80 msgr2=0x7fb1f80c9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f80b3f80 0x7fb1f80c9870 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec00a010 tx=0x7fb1ec004750 comp rx=0 tx=0).stop 2024-03-29T11:33:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 -- 172.21.15.12:0/618604510 shutdown_connections 2024-03-29T11:33:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb1d806aed0 0x7fb1d806d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1f80b8d30 0x7fb1f81443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f80b3f80 0x7fb1f80c9870 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 --2- 172.21.15.12:0/618604510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1f8160e50 0x7fb1f80b87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 -- 172.21.15.12:0/618604510 >> 172.21.15.12:0/618604510 conn(0x7fb1f808aff0 msgr2=0x7fb1f80ce840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 -- 172.21.15.12:0/618604510 shutdown_connections 2024-03-29T11:33:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.699+0000 7fb1ffc00700 1 -- 172.21.15.12:0/618604510 wait complete. 2024-03-29T11:33:27.714 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T11:33:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.920+0000 7f21e5e00700 1 Processor -- start 2024-03-29T11:33:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.920+0000 7f21e5e00700 1 -- start start 2024-03-29T11:33:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.921+0000 7f21e5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21e015d9d0 0x7f21e00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.921+0000 7f21e5e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f21e00d2540 0x7f21e00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.921+0000 7f21e5e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e0161420 con 0x7f21e00d16a0 2024-03-29T11:33:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.921+0000 7f21e5e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e015edc0 con 0x7f21e015d9d0 2024-03-29T11:33:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.921+0000 7f21e5e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e0161700 con 0x7f21e00d2540 2024-03-29T11:33:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.921+0000 7f21dea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21e015d9d0 0x7f21e00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.921+0000 7f21dea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f21e015d9d0 0x7f21e00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56196/0 (socket says 172.21.15.12:56196) 2024-03-29T11:33:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.921+0000 7f21dea00700 1 -- 172.21.15.12:0/4217710431 learned_addr learned my addr 172.21.15.12:0/4217710431 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.921+0000 7f21df400700 1 --2- 172.21.15.12:0/4217710431 >> v2:172.21.15.159:3300/0 conn(0x7f21e00d2540 0x7f21e00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.922+0000 7f21dea00700 1 -- 172.21.15.12:0/4217710431 >> v2:172.21.15.159:3300/0 conn(0x7f21e00d2540 msgr2=0x7f21e00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.922+0000 7f21dea00700 1 --2- 172.21.15.12:0/4217710431 >> v2:172.21.15.159:3300/0 conn(0x7f21e00d2540 0x7f21e00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.922+0000 7f21dea00700 1 -- 172.21.15.12:0/4217710431 >> v1:172.21.15.159:6789/0 conn(0x7f21e00d16a0 legacy=0x7f21e00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.922+0000 7f21dea00700 1 -- 172.21.15.12:0/4217710431 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e01619e0 con 0x7f21e015d9d0 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.922+0000 7f21dea00700 1 --2- 172.21.15.12:0/4217710431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e015d9d0 0x7f21e00d3c30 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f21e008ca10 tx=0x7f21d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2ac51da53d1f5c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.922+0000 7f21dd600700 1 -- 172.21.15.12:0/4217710431 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0011830 con 0x7f21e015d9d0 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.922+0000 7f21dd600700 1 -- 172.21.15.12:0/4217710431 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d0011990 con 0x7f21e015d9d0 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.922+0000 7f21dd600700 1 -- 172.21.15.12:0/4217710431 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0011d20 con 0x7f21e015d9d0 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.923+0000 7f21e5e00700 1 -- 172.21.15.12:0/4217710431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e015d9d0 msgr2=0x7f21e00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.923+0000 7f21e5e00700 1 --2- 172.21.15.12:0/4217710431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e015d9d0 0x7f21e00d3c30 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f21e008ca10 tx=0x7f21d0009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.923+0000 7f21e5e00700 1 -- 172.21.15.12:0/4217710431 shutdown_connections 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.923+0000 7f21e5e00700 1 --2- 172.21.15.12:0/4217710431 >> v2:172.21.15.159:3300/0 conn(0x7f21e00d2540 0x7f21e00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.923+0000 7f21e5e00700 1 --2- 172.21.15.12:0/4217710431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e015d9d0 0x7f21e00d3c30 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.923+0000 7f21e5e00700 1 -- 172.21.15.12:0/4217710431 >> 172.21.15.12:0/4217710431 conn(0x7f21e008aff0 msgr2=0x7f21e00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.923+0000 7f21e5e00700 1 -- 172.21.15.12:0/4217710431 shutdown_connections 2024-03-29T11:33:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.923+0000 7f21e5e00700 1 -- 172.21.15.12:0/4217710431 wait complete. 2024-03-29T11:33:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.925+0000 7f21e5e00700 1 Processor -- start 2024-03-29T11:33:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.925+0000 7f21e5e00700 1 -- start start 2024-03-29T11:33:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21e5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e00d16a0 0x7f21e01557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21e5e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21e00d2540 0x7f21e00b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21e5e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21e015d9d0 0x7f21e01551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21e5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00b4560 con 0x7f21e00d16a0 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21e5e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00b4840 con 0x7f21e015d9d0 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21e5e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00b4b20 con 0x7f21e00d2540 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21dea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e00d16a0 0x7f21e01557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21dea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e00d16a0 0x7f21e01557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56210/0 (socket says 172.21.15.12:56210) 2024-03-29T11:33:27.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21dea00700 1 -- 172.21.15.12:0/3976860986 learned_addr learned my addr 172.21.15.12:0/3976860986 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:27.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21df400700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21e015d9d0 0x7f21e01551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.926+0000 7f21de000700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21e00d2540 0x7f21e00b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21dea00700 1 -- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21e00d2540 msgr2=0x7f21e00b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21dea00700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21e00d2540 0x7f21e00b5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21dea00700 1 -- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21e015d9d0 msgr2=0x7f21e01551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21dea00700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21e015d9d0 0x7f21e01551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21dea00700 1 -- 172.21.15.12:0/3976860986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d0009550 con 0x7f21e00d16a0 2024-03-29T11:33:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21df400700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21e015d9d0 0x7f21e01551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21dea00700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e00d16a0 0x7f21e01557c0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009c10 tx=0x7f21d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21d7400700 1 -- 172.21.15.12:0/3976860986 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d00036a0 con 0x7f21e00d16a0 2024-03-29T11:33:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21d7400700 1 -- 172.21.15.12:0/3976860986 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d001e070 con 0x7f21e00d16a0 2024-03-29T11:33:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21d7400700 1 -- 172.21.15.12:0/3976860986 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0003a60 con 0x7f21e00d16a0 2024-03-29T11:33:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21e5e00700 1 -- 172.21.15.12:0/3976860986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e00b4e00 con 0x7f21e00d16a0 2024-03-29T11:33:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.927+0000 7f21e5e00700 1 -- 172.21.15.12:0/3976860986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e0155d80 con 0x7f21e00d16a0 2024-03-29T11:33:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.928+0000 7f21e5e00700 1 -- 172.21.15.12:0/3976860986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e00d1ca0 con 0x7f21e00d16a0 2024-03-29T11:33:27.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.929+0000 7f21d7400700 1 -- 172.21.15.12:0/3976860986 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f21d0003bc0 con 0x7f21e00d16a0 2024-03-29T11:33:27.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.931+0000 7f21d7400700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21c006aed0 0x7f21c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:27.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.931+0000 7f21d7400700 1 -- 172.21.15.12:0/3976860986 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 5875+0+0 (secure 0 0 0) 0x7f21d008a670 con 0x7f21e00d16a0 2024-03-29T11:33:27.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.931+0000 7f21de000700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21c006aed0 0x7f21c006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:27.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.932+0000 7f21de000700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21c006aed0 0x7f21c006d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f21e00d1ae0 tx=0x7f21cc009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:27.935+0000 7f21d7400700 1 -- 172.21.15.12:0/3976860986 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21d0055500 con 0x7f21e00d16a0 2024-03-29T11:33:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.149+0000 7f21e5e00700 1 -- 172.21.15.12:0/3976860986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f21e0067720 con 0x7f21e00d16a0 2024-03-29T11:33:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.153+0000 7f21d7400700 1 -- 172.21.15.12:0/3976860986 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v87) v1 ==== 120+0+0 (secure 0 0 0) 0x7f21d0058b20 con 0x7f21e00d16a0 2024-03-29T11:33:28.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 -- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21c006aed0 msgr2=0x7f21c006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21c006aed0 0x7f21c006d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f21e00d1ae0 tx=0x7f21cc009040 comp rx=0 tx=0).stop 2024-03-29T11:33:28.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 -- 172.21.15.12:0/3976860986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e00d16a0 msgr2=0x7f21e01557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e00d16a0 0x7f21e01557c0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009c10 tx=0x7f21d0004750 comp rx=0 tx=0).stop 2024-03-29T11:33:28.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 -- 172.21.15.12:0/3976860986 shutdown_connections 2024-03-29T11:33:28.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f21c006aed0 0x7f21c006d390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21e015d9d0 0x7f21e01551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e00d16a0 0x7f21e01557c0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 --2- 172.21.15.12:0/3976860986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21e00d2540 0x7f21e00b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 -- 172.21.15.12:0/3976860986 >> 172.21.15.12:0/3976860986 conn(0x7f21e008aff0 msgr2=0x7f21e00d1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:28.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 -- 172.21.15.12:0/3976860986 shutdown_connections 2024-03-29T11:33:28.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.156+0000 7f21e5e00700 1 -- 172.21.15.12:0/3976860986 wait complete. 2024-03-29T11:33:28.171 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-03-29T11:33:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.380+0000 7fc956000700 1 Processor -- start 2024-03-29T11:33:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.380+0000 7fc956000700 1 -- start start 2024-03-29T11:33:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.381+0000 7fc956000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 0x7fc9500cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.381+0000 7fc956000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9500ceda0 con 0x7fc950161590 2024-03-29T11:33:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.381+0000 7fc956000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9501612f0 con 0x7fc950160f10 2024-03-29T11:33:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.381+0000 7fc956000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500d1d30 con 0x7fc9500d18b0 2024-03-29T11:33:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.381+0000 7fc94fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc950161590 0x7fc9500b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42944/0 (socket says 172.21.15.12:42944) 2024-03-29T11:33:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.381+0000 7fc94fe00700 1 -- 172.21.15.12:0/4146787973 learned_addr learned my addr 172.21.15.12:0/4146787973 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.381+0000 7fc94ea00700 1 --2- 172.21.15.12:0/4146787973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 0x7fc9500cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.382+0000 7fc94e000700 1 -- 172.21.15.12:0/4146787973 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012700146 0 0) 0x7fc9500ceda0 con 0x7fc950161590 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.382+0000 7fc94e000700 1 -- 172.21.15.12:0/4146787973 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9340035b0 con 0x7fc950161590 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.382+0000 7fc94e000700 1 -- 172.21.15.12:0/4146787973 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713162054 0 0) 0x7fc9501612f0 con 0x7fc950160f10 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.382+0000 7fc94e000700 1 -- 172.21.15.12:0/4146787973 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9500ceda0 con 0x7fc950160f10 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.382+0000 7fc94ea00700 1 -- 172.21.15.12:0/4146787973 >> v1:172.21.15.159:6789/0 conn(0x7fc950160f10 legacy=0x7fc9500d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.382+0000 7fc94e000700 1 -- 172.21.15.12:0/4146787973 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2721532229 0 0) 0x7fc9340035b0 con 0x7fc950161590 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.382+0000 7fc94ea00700 1 -- 172.21.15.12:0/4146787973 >> v1:172.21.15.12:6789/0 conn(0x7fc950161590 legacy=0x7fc9500b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.382+0000 7fc94ea00700 1 -- 172.21.15.12:0/4146787973 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500ce470 con 0x7fc9500d18b0 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.382+0000 7fc94ea00700 1 --2- 172.21.15.12:0/4146787973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 0x7fc9500cf230 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0098a0 tx=0x7fc93c009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fdb86418b28f769 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.383+0000 7fc94e000700 1 -- 172.21.15.12:0/4146787973 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9480019f0 con 0x7fc9500d18b0 2024-03-29T11:33:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.383+0000 7fc94e000700 1 -- 172.21.15.12:0/4146787973 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c004030 con 0x7fc9500d18b0 2024-03-29T11:33:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.383+0000 7fc94e000700 1 -- 172.21.15.12:0/4146787973 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0043a0 con 0x7fc9500d18b0 2024-03-29T11:33:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.383+0000 7fc956000700 1 -- 172.21.15.12:0/4146787973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 msgr2=0x7fc9500cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.383+0000 7fc956000700 1 --2- 172.21.15.12:0/4146787973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 0x7fc9500cf230 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0098a0 tx=0x7fc93c009bb0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.383+0000 7fc956000700 1 -- 172.21.15.12:0/4146787973 shutdown_connections 2024-03-29T11:33:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.383+0000 7fc956000700 1 --2- 172.21.15.12:0/4146787973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 0x7fc9500cf230 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.383+0000 7fc956000700 1 -- 172.21.15.12:0/4146787973 >> 172.21.15.12:0/4146787973 conn(0x7fc95008aff0 msgr2=0x7fc9500d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.383+0000 7fc956000700 1 -- 172.21.15.12:0/4146787973 shutdown_connections 2024-03-29T11:33:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.384+0000 7fc956000700 1 -- 172.21.15.12:0/4146787973 wait complete. 2024-03-29T11:33:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.385+0000 7fc956000700 1 Processor -- start 2024-03-29T11:33:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.385+0000 7fc956000700 1 -- start start 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.386+0000 7fc956000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 0x7fc950155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.386+0000 7fc956000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc950160f10 0x7fc9500b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.386+0000 7fc956000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc950161590 0x7fc9501514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.386+0000 7fc956000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95015d4e0 con 0x7fc950161590 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.386+0000 7fc956000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95015d7c0 con 0x7fc950160f10 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.386+0000 7fc956000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95015daa0 con 0x7fc9500d18b0 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc950161590 0x7fc9501514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc950161590 0x7fc9501514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56216/0 (socket says 172.21.15.12:56216) 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94fe00700 1 -- 172.21.15.12:0/2890276519 learned_addr learned my addr 172.21.15.12:0/2890276519 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94ea00700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc950160f10 0x7fc9500b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94f400700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 0x7fc950155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94fe00700 1 -- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 msgr2=0x7fc950155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94fe00700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 0x7fc950155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94fe00700 1 -- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc950160f10 msgr2=0x7fc9500b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94fe00700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc950160f10 0x7fc9500b57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94fe00700 1 -- 172.21.15.12:0/2890276519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c0094a0 con 0x7fc950161590 2024-03-29T11:33:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94f400700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 0x7fc950155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.387+0000 7fc94fe00700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc950161590 0x7fc9501514f0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7fc94800cc00 tx=0x7fc94800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.388+0000 7fc947400700 1 -- 172.21.15.12:0/2890276519 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94800ab30 con 0x7fc950161590 2024-03-29T11:33:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.388+0000 7fc947400700 1 -- 172.21.15.12:0/2890276519 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94800acb0 con 0x7fc950161590 2024-03-29T11:33:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.388+0000 7fc956000700 1 -- 172.21.15.12:0/2890276519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95015dd80 con 0x7fc950161590 2024-03-29T11:33:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.388+0000 7fc947400700 1 -- 172.21.15.12:0/2890276519 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948007310 con 0x7fc950161590 2024-03-29T11:33:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.388+0000 7fc956000700 1 -- 172.21.15.12:0/2890276519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc950156520 con 0x7fc950161590 2024-03-29T11:33:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.389+0000 7fc956000700 1 -- 172.21.15.12:0/2890276519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95015e2a0 con 0x7fc950161590 2024-03-29T11:33:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.390+0000 7fc947400700 1 -- 172.21.15.12:0/2890276519 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fc948007470 con 0x7fc950161590 2024-03-29T11:33:28.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.391+0000 7fc947400700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc93006aed0 0x7fc93006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:28.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.391+0000 7fc947400700 1 -- 172.21.15.12:0/2890276519 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 5875+0+0 (secure 0 0 0) 0x7fc948089310 con 0x7fc950161590 2024-03-29T11:33:28.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.391+0000 7fc94f400700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc93006aed0 0x7fc93006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.392+0000 7fc94f400700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc93006aed0 0x7fc93006d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9400011e0 tx=0x7fc940009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.395+0000 7fc947400700 1 -- 172.21.15.12:0/2890276519 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc9480540d0 con 0x7fc950161590 2024-03-29T11:33:28.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.706+0000 7fc956000700 1 -- 172.21.15.12:0/2890276519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7fc950067720 con 0x7fc950161590 2024-03-29T11:33:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.766+0000 7fc947400700 1 -- 172.21.15.12:0/2890276519 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3260) v1 ==== 187+0+0 (secure 0 0 0) 0x7fc9480576f0 con 0x7fc950161590 2024-03-29T11:33:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 -- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc93006aed0 msgr2=0x7fc93006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc93006aed0 0x7fc93006d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9400011e0 tx=0x7fc940009040 comp rx=0 tx=0).stop 2024-03-29T11:33:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 -- 172.21.15.12:0/2890276519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc950161590 msgr2=0x7fc9501514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc950161590 0x7fc9501514f0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7fc94800cc00 tx=0x7fc94800cf10 comp rx=0 tx=0).stop 2024-03-29T11:33:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 -- 172.21.15.12:0/2890276519 shutdown_connections 2024-03-29T11:33:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc93006aed0 0x7fc93006d390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc950161590 0x7fc9501514f0 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9500d18b0 0x7fc950155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 --2- 172.21.15.12:0/2890276519 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc950160f10 0x7fc9500b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 -- 172.21.15.12:0/2890276519 >> 172.21.15.12:0/2890276519 conn(0x7fc95008aff0 msgr2=0x7fc950152700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 -- 172.21.15.12:0/2890276519 shutdown_connections 2024-03-29T11:33:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.769+0000 7fc956000700 1 -- 172.21.15.12:0/2890276519 wait complete. 2024-03-29T11:33:28.785 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2024-03-29T11:33:28.785 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2024-03-29T11:33:28.785 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2024-03-29T11:33:28.785 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-03-29T11:33:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.989+0000 7f1aea800700 1 Processor -- start 2024-03-29T11:33:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.989+0000 7f1aea800700 1 -- start start 2024-03-29T11:33:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.990+0000 7f1aea800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ae40ac7d0 0x7f1ae4166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.990+0000 7f1aea800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae415fd50 con 0x7f1ae415b5d0 2024-03-29T11:33:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.990+0000 7f1aea800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae415f140 con 0x7f1ae40cb3f0 2024-03-29T11:33:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.990+0000 7f1aea800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae415b9b0 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.990+0000 7f1ae3e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ae40ac7d0 0x7f1ae4166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.990+0000 7f1ae3400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1ae415b5d0 0x7f1ae40ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40726/0 (socket says 172.21.15.12:40726) 2024-03-29T11:33:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.990+0000 7f1ae3400700 1 -- 172.21.15.12:0/338831637 learned_addr learned my addr 172.21.15.12:0/338831637 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.991+0000 7f1ae3e00700 1 -- 172.21.15.12:0/338831637 >> v1:172.21.15.159:6789/0 conn(0x7f1ae40cb3f0 legacy=0x7f1ae40ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.991+0000 7f1ae2000700 1 -- 172.21.15.12:0/338831637 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405580814 0 0) 0x7f1ae415fd50 con 0x7f1ae415b5d0 2024-03-29T11:33:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.991+0000 7f1ae3e00700 1 -- 172.21.15.12:0/338831637 >> v1:172.21.15.12:6789/0 conn(0x7f1ae415b5d0 legacy=0x7f1ae40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.991+0000 7f1ae3e00700 1 -- 172.21.15.12:0/338831637 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae415f420 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.991+0000 7f1ae3e00700 1 --2- 172.21.15.12:0/338831637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 0x7f1ae4166430 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f1adc009d90 tx=0x7f1adc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe98980e2def36fb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.991+0000 7f1ae2000700 1 -- 172.21.15.12:0/338831637 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc00e070 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.991+0000 7f1ae2000700 1 -- 172.21.15.12:0/338831637 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1adc00bbd0 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.991+0000 7f1ae2000700 1 -- 172.21.15.12:0/338831637 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc012430 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.992+0000 7f1aea800700 1 -- 172.21.15.12:0/338831637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 msgr2=0x7f1ae4166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.992+0000 7f1aea800700 1 --2- 172.21.15.12:0/338831637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 0x7f1ae4166430 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f1adc009d90 tx=0x7f1adc00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.992+0000 7f1aea800700 1 -- 172.21.15.12:0/338831637 shutdown_connections 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.992+0000 7f1aea800700 1 --2- 172.21.15.12:0/338831637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 0x7f1ae4166430 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.992+0000 7f1aea800700 1 -- 172.21.15.12:0/338831637 >> 172.21.15.12:0/338831637 conn(0x7f1ae40cf190 msgr2=0x7f1ae40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.992+0000 7f1aea800700 1 -- 172.21.15.12:0/338831637 shutdown_connections 2024-03-29T11:33:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.992+0000 7f1aea800700 1 -- 172.21.15.12:0/338831637 wait complete. 2024-03-29T11:33:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.994+0000 7f1aea800700 1 Processor -- start 2024-03-29T11:33:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.994+0000 7f1aea800700 1 -- start start 2024-03-29T11:33:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1aea800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 0x7f1ae41f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1aea800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae415b5d0 0x7f1ae40cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1aea800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae40aece0 0x7f1ae40cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1aea800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae41598d0 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1aea800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae4157d90 con 0x7f1ae40aece0 2024-03-29T11:33:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1aea800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae41577d0 con 0x7f1ae415b5d0 2024-03-29T11:33:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 0x7f1ae41f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 0x7f1ae41f38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46512/0 (socket says 172.21.15.12:46512) 2024-03-29T11:33:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae3400700 1 -- 172.21.15.12:0/1528200077 learned_addr learned my addr 172.21.15.12:0/1528200077 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae3e00700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae40aece0 0x7f1ae40cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae2a00700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae415b5d0 0x7f1ae40cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae3400700 1 -- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae415b5d0 msgr2=0x7f1ae40cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae3400700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae415b5d0 0x7f1ae40cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae3400700 1 -- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae40aece0 msgr2=0x7f1ae40cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae3400700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae40aece0 0x7f1ae40cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae3400700 1 -- 172.21.15.12:0/1528200077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc009a70 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.995+0000 7f1ae2a00700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae415b5d0 0x7f1ae40cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.996+0000 7f1ae3400700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 0x7f1ae41f38a0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400c520 tx=0x7f1ad400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.996+0000 7f1adb400700 1 -- 172.21.15.12:0/1528200077 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4004c80 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.996+0000 7f1adb400700 1 -- 172.21.15.12:0/1528200077 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad4004e20 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.996+0000 7f1aea800700 1 -- 172.21.15.12:0/1528200077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae4157ab0 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.996+0000 7f1adb400700 1 -- 172.21.15.12:0/1528200077 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4007310 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.996+0000 7f1aea800700 1 -- 172.21.15.12:0/1528200077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae40ccfc0 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.997+0000 7f1aea800700 1 -- 172.21.15.12:0/1528200077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae40cd1b0 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.998+0000 7f1adb400700 1 -- 172.21.15.12:0/1528200077 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f1ad4007470 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.999+0000 7f1adb400700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ac406aed0 0x7f1ac406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.999+0000 7f1adb400700 1 -- 172.21.15.12:0/1528200077 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 5875+0+0 (secure 0 0 0) 0x7f1ad40893f0 con 0x7f1ae40ac7d0 2024-03-29T11:33:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:28.999+0000 7f1ae2a00700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ac406aed0 0x7f1ac406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.000+0000 7f1ae2a00700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ac406aed0 0x7f1ac406d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4073d30 tx=0x7f1ad0009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:28.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.003+0000 7f1adb400700 1 -- 172.21.15.12:0/1528200077 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ad4054250 con 0x7f1ae40ac7d0 2024-03-29T11:33:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.304+0000 7f1aea800700 1 -- 172.21.15.12:0/1528200077 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f1ae4067720 con 0x7f1ae40ac7d0 2024-03-29T11:33:29.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.768+0000 7f1adb400700 1 -- 172.21.15.12:0/1528200077 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3261) v1 ==== 158+0+0 (secure 0 0 0) 0x7f1ad4057870 con 0x7f1ae40ac7d0 2024-03-29T11:33:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 -- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ac406aed0 msgr2=0x7f1ac406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ac406aed0 0x7f1ac406d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4073d30 tx=0x7f1ad0009040 comp rx=0 tx=0).stop 2024-03-29T11:33:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 -- 172.21.15.12:0/1528200077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 msgr2=0x7f1ae41f38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 0x7f1ae41f38a0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400c520 tx=0x7f1ad400c8e0 comp rx=0 tx=0).stop 2024-03-29T11:33:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 -- 172.21.15.12:0/1528200077 shutdown_connections 2024-03-29T11:33:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ac406aed0 0x7f1ac406d390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae40aece0 0x7f1ae40cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ae40ac7d0 0x7f1ae41f38a0 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 --2- 172.21.15.12:0/1528200077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae415b5d0 0x7f1ae40cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 -- 172.21.15.12:0/1528200077 >> 172.21.15.12:0/1528200077 conn(0x7f1ae40cf190 msgr2=0x7f1ae40b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 -- 172.21.15.12:0/1528200077 shutdown_connections 2024-03-29T11:33:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.771+0000 7f1aea800700 1 -- 172.21.15.12:0/1528200077 wait complete. 2024-03-29T11:33:29.786 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T11:33:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.992+0000 7f20ad400700 1 Processor -- start 2024-03-29T11:33:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.993+0000 7f20ad400700 1 -- start start 2024-03-29T11:33:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.994+0000 7f20ad400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a80ce840 0x7f20a8165370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.994+0000 7f20ad400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a80cb310 con 0x7f20a815dfe0 2024-03-29T11:33:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.994+0000 7f20ad400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a80cca90 con 0x7f20a815a5c0 2024-03-29T11:33:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.994+0000 7f20ad400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80711a0 con 0x7f20a80ce840 2024-03-29T11:33:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.994+0000 7f20a5e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f20a815dfe0 0x7f20a80ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40742/0 (socket says 172.21.15.12:40742) 2024-03-29T11:33:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.994+0000 7f20a5e00700 1 -- 172.21.15.12:0/2578883648 learned_addr learned my addr 172.21.15.12:0/2578883648 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.994+0000 7f20a7200700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a80ce840 0x7f20a8165370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.994+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975517463 0 0) 0x7f20a80cb310 con 0x7f20a815dfe0 2024-03-29T11:33:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.994+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f208c0057c0 con 0x7f20a815dfe0 2024-03-29T11:33:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2218423058 0 0) 0x7f208c0057c0 con 0x7f20a815dfe0 2024-03-29T11:33:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a80ce840 msgr2=0x7f20a8165370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a80ce840 0x7f20a8165370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 >> v1:172.21.15.159:6789/0 conn(0x7f20a815a5c0 legacy=0x7f20a80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a8071480 con 0x7f20a815dfe0 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f2094001670 con 0x7f20a815dfe0 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 >> v1:172.21.15.12:6789/0 conn(0x7f20a815dfe0 legacy=0x7f20a80ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a80ce840 0x7f208c005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 0x7f208c006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f208c006fa0 0x7f208c007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80711a0 con 0x7f208c006180 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c00dc80 con 0x7f208c006fa0 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c00de40 con 0x7f20a80ce840 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a5e00700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 0x7f208c006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a6800700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a80ce840 0x7f208c005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.995+0000 7f20a7200700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f208c006fa0 0x7f208c007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.996+0000 7f20a5e00700 1 -- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a80ce840 msgr2=0x7f208c005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.996+0000 7f20a5e00700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a80ce840 0x7f208c005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.996+0000 7f20a5e00700 1 -- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f208c006fa0 msgr2=0x7f208c007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.996+0000 7f20a5e00700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f208c006fa0 0x7f208c007420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.996+0000 7f20a5e00700 1 -- 172.21.15.12:0/2578883648 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0057c0 con 0x7f208c006180 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.996+0000 7f20a5e00700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 0x7f208c006a60 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f209400c2f0 tx=0x7f208c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f3fe67a7b7d0d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.996+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209400cd70 con 0x7f208c006180 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.996+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2094005960 con 0x7f208c006180 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.996+0000 7f20a5400700 1 -- 172.21.15.12:0/2578883648 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2094005d10 con 0x7f208c006180 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.997+0000 7f20ad400700 1 -- 172.21.15.12:0/2578883648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 msgr2=0x7f208c006a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.997+0000 7f20ad400700 1 --2- 172.21.15.12:0/2578883648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 0x7f208c006a60 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f209400c2f0 tx=0x7f208c007a90 comp rx=0 tx=0).stop 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.997+0000 7f20a7200700 1 -- 172.21.15.12:0/2578883648 reap_dead start 2024-03-29T11:33:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.997+0000 7f20ad400700 1 -- 172.21.15.12:0/2578883648 shutdown_connections 2024-03-29T11:33:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.997+0000 7f20ad400700 1 -- 172.21.15.12:0/2578883648 >> 172.21.15.12:0/2578883648 conn(0x7f20a80b0d70 msgr2=0x7f20a80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.997+0000 7f20ad400700 1 -- 172.21.15.12:0/2578883648 shutdown_connections 2024-03-29T11:33:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.997+0000 7f20ad400700 1 -- 172.21.15.12:0/2578883648 wait complete. 2024-03-29T11:33:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.999+0000 7f20ad400700 1 Processor -- start 2024-03-29T11:33:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:29.999+0000 7f20ad400700 1 -- start start 2024-03-29T11:33:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20ad400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 0x7f20a814c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20ad400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a814e9c0 0x7f20a814cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20ad400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a81554c0 0x7f20a8155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20ad400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a81f6e70 con 0x7f208c006180 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20ad400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a8155e90 con 0x7f20a814e9c0 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20ad400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a814cd60 con 0x7f20a81554c0 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 0x7f20a814c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 0x7f20a814c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46524/0 (socket says 172.21.15.12:46524) 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a6800700 1 -- 172.21.15.12:0/3614813681 learned_addr learned my addr 172.21.15.12:0/3614813681 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a5e00700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a814e9c0 0x7f20a814cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a7200700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a81554c0 0x7f20a8155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a6800700 1 -- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a81554c0 msgr2=0x7f20a8155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a6800700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a81554c0 0x7f20a8155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a6800700 1 -- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a814e9c0 msgr2=0x7f20a814cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a6800700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a814e9c0 0x7f20a814cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a6800700 1 -- 172.21.15.12:0/3614813681 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a81f6560 con 0x7f208c006180 2024-03-29T11:33:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.001+0000 7f20a5e00700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a814e9c0 0x7f20a814cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.002+0000 7f20a6800700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 0x7f20a814c820 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f209800f7a0 tx=0x7f209800fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.002+0000 7f209ea00700 1 -- 172.21.15.12:0/3614813681 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20980052e0 con 0x7f208c006180 2024-03-29T11:33:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.002+0000 7f209ea00700 1 -- 172.21.15.12:0/3614813681 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2098005460 con 0x7f208c006180 2024-03-29T11:33:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.002+0000 7f20ad400700 1 -- 172.21.15.12:0/3614813681 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a80adee0 con 0x7f208c006180 2024-03-29T11:33:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.002+0000 7f209ea00700 1 -- 172.21.15.12:0/3614813681 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20980038c0 con 0x7f208c006180 2024-03-29T11:33:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.002+0000 7f20ad400700 1 -- 172.21.15.12:0/3614813681 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a81f67b0 con 0x7f208c006180 2024-03-29T11:33:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.003+0000 7f20ad400700 1 -- 172.21.15.12:0/3614813681 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a81f6fb0 con 0x7f208c006180 2024-03-29T11:33:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.004+0000 7f209ea00700 1 -- 172.21.15.12:0/3614813681 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2098003a20 con 0x7f208c006180 2024-03-29T11:33:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.005+0000 7f209ea00700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f208806aed0 0x7f208806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.005+0000 7f209ea00700 1 -- 172.21.15.12:0/3614813681 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 5875+0+0 (secure 0 0 0) 0x7f2098089670 con 0x7f208c006180 2024-03-29T11:33:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.005+0000 7f20a5e00700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f208806aed0 0x7f208806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.006+0000 7f20a5e00700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f208806aed0 0x7f208806d390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2094015040 tx=0x7f2094007000 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.010+0000 7f209ea00700 1 -- 172.21.15.12:0/3614813681 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f20a81f6fb0 con 0x7f208c006180 2024-03-29T11:33:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.223+0000 7f20ad400700 1 -- 172.21.15.12:0/3614813681 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f20a81f6fb0 con 0x7f208c006180 2024-03-29T11:33:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.227+0000 7f209ea00700 1 -- 172.21.15.12:0/3614813681 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7f20a81f6fb0 con 0x7f208c006180 2024-03-29T11:33:30.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 -- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f208806aed0 msgr2=0x7f208806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f208806aed0 0x7f208806d390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2094015040 tx=0x7f2094007000 comp rx=0 tx=0).stop 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 -- 172.21.15.12:0/3614813681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 msgr2=0x7f20a814c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 0x7f20a814c820 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f209800f7a0 tx=0x7f209800fb60 comp rx=0 tx=0).stop 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 -- 172.21.15.12:0/3614813681 shutdown_connections 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f208806aed0 0x7f208806d390 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a81554c0 0x7f20a8155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f208c006180 0x7f20a814c820 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 --2- 172.21.15.12:0/3614813681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a814e9c0 0x7f20a814cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 -- 172.21.15.12:0/3614813681 >> 172.21.15.12:0/3614813681 conn(0x7f20a80b0d70 msgr2=0x7f20a80cc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 -- 172.21.15.12:0/3614813681 shutdown_connections 2024-03-29T11:33:30.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.230+0000 7f20ad400700 1 -- 172.21.15.12:0/3614813681 wait complete. 2024-03-29T11:33:30.245 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:33:30.245 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:33:30.328 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:33:30.424 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:33:30.424 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:33:30.424 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:30.468 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:33:30.468 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:33:30.469 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:30.515 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:33:30.515 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:33:30.515 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:30.560 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:33:30.560 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:33:30.560 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:30.605 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:33:30.605 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T11:33:30.606 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:33:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.808+0000 7ff537600700 1 Processor -- start 2024-03-29T11:33:30.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.808+0000 7ff537600700 1 -- start start 2024-03-29T11:33:30.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.809+0000 7ff537600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff5300d18b0 0x7ff5300cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:30.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.809+0000 7ff537600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff530161590 0x7ff530161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:30.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.809+0000 7ff537600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5300ceda0 con 0x7ff530160f10 2024-03-29T11:33:30.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.809+0000 7ff537600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5300d1d30 con 0x7ff5300d18b0 2024-03-29T11:33:30.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.809+0000 7ff537600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff530161f60 con 0x7ff530161590 2024-03-29T11:33:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.809+0000 7ff52fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff5300d18b0 0x7ff5300cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.809+0000 7ff52fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff5300d18b0 0x7ff5300cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46534/0 (socket says 172.21.15.12:46534) 2024-03-29T11:33:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.809+0000 7ff52fe00700 1 -- 172.21.15.12:0/4061179745 learned_addr learned my addr 172.21.15.12:0/4061179745 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.809+0000 7ff535600700 1 --2- 172.21.15.12:0/4061179745 >> v2:172.21.15.159:3300/0 conn(0x7ff530161590 0x7ff530161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.810+0000 7ff52fe00700 1 -- 172.21.15.12:0/4061179745 >> v2:172.21.15.159:3300/0 conn(0x7ff530161590 msgr2=0x7ff530161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.810+0000 7ff52fe00700 1 --2- 172.21.15.12:0/4061179745 >> v2:172.21.15.159:3300/0 conn(0x7ff530161590 0x7ff530161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.810+0000 7ff52fe00700 1 -- 172.21.15.12:0/4061179745 >> v1:172.21.15.159:6789/0 conn(0x7ff530160f10 legacy=0x7ff5300d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.810+0000 7ff52fe00700 1 -- 172.21.15.12:0/4061179745 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5301612f0 con 0x7ff5300d18b0 2024-03-29T11:33:30.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.810+0000 7ff52fe00700 1 --2- 172.21.15.12:0/4061179745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5300d18b0 0x7ff5300cf230 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7ff5300d1030 tx=0x7ff51c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a72c48f9feed92 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.810+0000 7ff52f400700 1 -- 172.21.15.12:0/4061179745 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c011830 con 0x7ff5300d18b0 2024-03-29T11:33:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.810+0000 7ff52f400700 1 -- 172.21.15.12:0/4061179745 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51c011990 con 0x7ff5300d18b0 2024-03-29T11:33:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.810+0000 7ff52f400700 1 -- 172.21.15.12:0/4061179745 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c011d20 con 0x7ff5300d18b0 2024-03-29T11:33:30.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.811+0000 7ff537600700 1 -- 172.21.15.12:0/4061179745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5300d18b0 msgr2=0x7ff5300cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.811+0000 7ff537600700 1 --2- 172.21.15.12:0/4061179745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5300d18b0 0x7ff5300cf230 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7ff5300d1030 tx=0x7ff51c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.811+0000 7ff537600700 1 -- 172.21.15.12:0/4061179745 shutdown_connections 2024-03-29T11:33:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.811+0000 7ff537600700 1 --2- 172.21.15.12:0/4061179745 >> v2:172.21.15.159:3300/0 conn(0x7ff530161590 0x7ff530161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.811+0000 7ff537600700 1 --2- 172.21.15.12:0/4061179745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5300d18b0 0x7ff5300cf230 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.811+0000 7ff537600700 1 -- 172.21.15.12:0/4061179745 >> 172.21.15.12:0/4061179745 conn(0x7ff53008aff0 msgr2=0x7ff5300d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.811+0000 7ff537600700 1 -- 172.21.15.12:0/4061179745 shutdown_connections 2024-03-29T11:33:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.811+0000 7ff537600700 1 -- 172.21.15.12:0/4061179745 wait complete. 2024-03-29T11:33:30.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.813+0000 7ff537600700 1 Processor -- start 2024-03-29T11:33:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.813+0000 7ff537600700 1 -- start start 2024-03-29T11:33:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff537600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5300d18b0 0x7ff53009f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff537600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff530160f10 0x7ff53015bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff537600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff530161590 0x7ff530160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff537600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5300a0790 con 0x7ff530161590 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff537600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53015c070 con 0x7ff5300d18b0 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff537600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53009fb10 con 0x7ff530160f10 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff535600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff530161590 0x7ff530160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff52fe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff530160f10 0x7ff53015bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff52fe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff530160f10 0x7ff53015bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:54822/0 (socket says 172.21.15.12:54822) 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff534c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5300d18b0 0x7ff53009f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.814+0000 7ff52fe00700 1 -- 172.21.15.12:0/3485328880 learned_addr learned my addr 172.21.15.12:0/3485328880 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff52fe00700 1 -- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5300d18b0 msgr2=0x7ff53009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff52fe00700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5300d18b0 0x7ff53009f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff52fe00700 1 -- 172.21.15.12:0/3485328880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff530161590 msgr2=0x7ff530160130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff52fe00700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff530161590 0x7ff530160130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff52fe00700 1 -- 172.21.15.12:0/3485328880 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c009550 con 0x7ff530160f10 2024-03-29T11:33:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff535600700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff530161590 0x7ff530160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff52fe00700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff530160f10 0x7ff53015bb30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009130 tx=0x7ff51c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff52cc00700 1 -- 172.21.15.12:0/3485328880 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c004510 con 0x7ff530160f10 2024-03-29T11:33:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff52cc00700 1 -- 172.21.15.12:0/3485328880 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51c01e070 con 0x7ff530160f10 2024-03-29T11:33:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff52cc00700 1 -- 172.21.15.12:0/3485328880 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c0038a0 con 0x7ff530160f10 2024-03-29T11:33:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff537600700 1 -- 172.21.15.12:0/3485328880 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53009fdf0 con 0x7ff530160f10 2024-03-29T11:33:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.815+0000 7ff537600700 1 -- 172.21.15.12:0/3485328880 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5300a03b0 con 0x7ff530160f10 2024-03-29T11:33:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.816+0000 7ff537600700 1 -- 172.21.15.12:0/3485328880 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5300a08d0 con 0x7ff530160f10 2024-03-29T11:33:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.817+0000 7ff52cc00700 1 -- 172.21.15.12:0/3485328880 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff51c003a00 con 0x7ff530160f10 2024-03-29T11:33:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.818+0000 7ff52cc00700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff51006aed0 0x7ff51006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.818+0000 7ff52cc00700 1 -- 172.21.15.12:0/3485328880 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 5875+0+0 (secure 0 0 0) 0x7ff51c08a620 con 0x7ff530160f10 2024-03-29T11:33:30.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.818+0000 7ff534c00700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff51006aed0 0x7ff51006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:30.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.819+0000 7ff534c00700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff51006aed0 0x7ff51006d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff5300ca330 tx=0x7ff524009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:30.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:30.823+0000 7ff52cc00700 1 -- 172.21.15.12:0/3485328880 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff5300a08d0 con 0x7ff530160f10 2024-03-29T11:33:31.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.080+0000 7ff537600700 1 -- 172.21.15.12:0/3485328880 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5300a08d0 con 0x7ff530160f10 2024-03-29T11:33:31.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.081+0000 7ff52cc00700 1 -- 172.21.15.12:0/3485328880 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+3541 (secure 0 0 0) 0x7ff51c020df0 con 0x7ff530160f10 2024-03-29T11:33:31.077 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:31.077 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":132,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11410,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2295710189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2295710189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2295710189}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11413,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3614770908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3614770908},{"type":"v1","addr":"172.21.15.159:6836","nonce":3614770908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":11986,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3215734067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3215734067},{"type":"v1","addr":"172.21.15.159:6837","nonce":3215734067}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":132},{"gid":12459,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/839500006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":839500006},{"type":"v1","addr":"172.21.15.12:6834","nonce":839500006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[]} 2024-03-29T11:33:31.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 -- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff51006aed0 msgr2=0x7ff51006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff51006aed0 0x7ff51006d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff5300ca330 tx=0x7ff524009040 comp rx=0 tx=0).stop 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 -- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff530160f10 msgr2=0x7ff53015bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff530160f10 0x7ff53015bb30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009130 tx=0x7ff51c004750 comp rx=0 tx=0).stop 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 -- 172.21.15.12:0/3485328880 shutdown_connections 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff51006aed0 0x7ff51006d390 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff530161590 0x7ff530160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5300d18b0 0x7ff53009f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 --2- 172.21.15.12:0/3485328880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff530160f10 0x7ff53015bb30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 -- 172.21.15.12:0/3485328880 >> 172.21.15.12:0/3485328880 conn(0x7ff53008aff0 msgr2=0x7ff5300cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 -- 172.21.15.12:0/3485328880 shutdown_connections 2024-03-29T11:33:31.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.083+0000 7ff537600700 1 -- 172.21.15.12:0/3485328880 wait complete. 2024-03-29T11:33:31.081 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 132 2024-03-29T11:33:31.098 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T11:33:31.098 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T11:33:31.099 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T11:33:31.099 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:33:31.099 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T11:33:31.099 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T11:33:31.099 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T11:33:31.099 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:33:31.099 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T11:33:31.099 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T11:33:31.100 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T11:33:31.100 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:33:31.100 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T11:33:31.100 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T11:33:31.100 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T11:33:31.100 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:33:31.115 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T11:33:31.115 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T11:33:31.116 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T11:33:31.116 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T11:33:31.117 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T11:33:31.117 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T11:33:31.118 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T11:33:31.118 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T11:33:31.159 INFO:tasks.ceph.mds.c:Started 2024-03-29T11:33:31.160 DEBUG:teuthology.parallel:result is None 2024-03-29T11:33:31.161 INFO:tasks.ceph.mds.d:Started 2024-03-29T11:33:31.161 DEBUG:teuthology.parallel:result is None 2024-03-29T11:33:31.162 INFO:tasks.ceph.mds.a:Started 2024-03-29T11:33:31.162 DEBUG:teuthology.parallel:result is None 2024-03-29T11:33:31.163 INFO:tasks.ceph.mds.b:Started 2024-03-29T11:33:31.164 DEBUG:teuthology.parallel:result is None 2024-03-29T11:33:31.164 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T11:33:31.304 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T11:33:31.304 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T11:33:31.310 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:33:31.310 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T11:33:31.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.383+0000 7f91c3c00700 1 Processor -- start 2024-03-29T11:33:31.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.383+0000 7f91c3c00700 1 -- start start 2024-03-29T11:33:31.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.384+0000 7f91c3c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91bc0cbb30 0x7f91bc0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:31.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.384+0000 7f91c3c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f91bc0cd020 0x7f91bc0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:31.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.384+0000 7f91c3c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91bc0cc450 con 0x7f91bc0b2070 2024-03-29T11:33:31.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.384+0000 7f91c3c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91bc0cdae0 con 0x7f91bc0cbb30 2024-03-29T11:33:31.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.384+0000 7f91c3c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91bc0cddc0 con 0x7f91bc0cd020 2024-03-29T11:33:31.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.384+0000 7f91c1200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91bc0cbb30 0x7f91bc0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:31.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91c1200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91bc0cbb30 0x7f91bc0cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46568/0 (socket says 172.21.15.12:46568) 2024-03-29T11:33:31.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91c1200700 1 -- 172.21.15.12:0/3819948039 learned_addr learned my addr 172.21.15.12:0/3819948039 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:31.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91bbe00700 1 --2- 172.21.15.12:0/3819948039 >> v2:172.21.15.159:3300/0 conn(0x7f91bc0cd020 0x7f91bc0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:31.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91c1200700 1 -- 172.21.15.12:0/3819948039 >> v2:172.21.15.159:3300/0 conn(0x7f91bc0cd020 msgr2=0x7f91bc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:31.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91c1200700 1 --2- 172.21.15.12:0/3819948039 >> v2:172.21.15.159:3300/0 conn(0x7f91bc0cd020 0x7f91bc0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:31.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91c1200700 1 -- 172.21.15.12:0/3819948039 >> v1:172.21.15.12:6789/0 conn(0x7f91bc0b2070 legacy=0x7f91bc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:31.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91c1200700 1 -- 172.21.15.12:0/3819948039 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91bc0ce0a0 con 0x7f91bc0cbb30 2024-03-29T11:33:31.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91bb400700 1 -- 172.21.15.12:0/3819948039 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167951247 0 0) 0x7f91bc0cc450 con 0x7f91bc0b2070 2024-03-29T11:33:31.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91c1200700 1 --2- 172.21.15.12:0/3819948039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cbb30 0x7f91bc0cbf10 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f91bc0b1e30 tx=0x7f91b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ed1951b2805f14 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:31.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91bb400700 1 -- 172.21.15.12:0/3819948039 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0011830 con 0x7f91bc0cbb30 2024-03-29T11:33:31.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91bb400700 1 -- 172.21.15.12:0/3819948039 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b0011990 con 0x7f91bc0cbb30 2024-03-29T11:33:31.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.385+0000 7f91bb400700 1 -- 172.21.15.12:0/3819948039 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0011d20 con 0x7f91bc0cbb30 2024-03-29T11:33:31.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.386+0000 7f91c3c00700 1 -- 172.21.15.12:0/3819948039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cbb30 msgr2=0x7f91bc0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:31.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.386+0000 7f91c3c00700 1 --2- 172.21.15.12:0/3819948039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cbb30 0x7f91bc0cbf10 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f91bc0b1e30 tx=0x7f91b0009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:31.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.386+0000 7f91c3c00700 1 -- 172.21.15.12:0/3819948039 shutdown_connections 2024-03-29T11:33:31.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.386+0000 7f91c3c00700 1 --2- 172.21.15.12:0/3819948039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cbb30 0x7f91bc0cbf10 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:31.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.386+0000 7f91c3c00700 1 --2- 172.21.15.12:0/3819948039 >> v2:172.21.15.159:3300/0 conn(0x7f91bc0cd020 0x7f91bc0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:31.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.386+0000 7f91c3c00700 1 -- 172.21.15.12:0/3819948039 >> 172.21.15.12:0/3819948039 conn(0x7f91bc0b0d70 msgr2=0x7f91bc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:31.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.386+0000 7f91c3c00700 1 -- 172.21.15.12:0/3819948039 shutdown_connections 2024-03-29T11:33:31.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.386+0000 7f91c3c00700 1 -- 172.21.15.12:0/3819948039 wait complete. 2024-03-29T11:33:31.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.388+0000 7f91c3c00700 1 Processor -- start 2024-03-29T11:33:31.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.388+0000 7f91c3c00700 1 -- start start 2024-03-29T11:33:31.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91c3c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91bc0b2070 0x7f91bc14f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:31.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91c3c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91bc0cbb30 0x7f91bc0ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:31.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91c3c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cd020 0x7f91bc0af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:31.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91c3c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91bc0adb60 con 0x7f91bc0cd020 2024-03-29T11:33:31.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91c3c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91bc14fe90 con 0x7f91bc0cbb30 2024-03-29T11:33:31.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91c3c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91bc0acf50 con 0x7f91bc0b2070 2024-03-29T11:33:31.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91c1c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cd020 0x7f91bc0af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:31.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91c1c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cd020 0x7f91bc0af920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46572/0 (socket says 172.21.15.12:46572) 2024-03-29T11:33:31.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91c1c00700 1 -- 172.21.15.12:0/424689671 learned_addr learned my addr 172.21.15.12:0/424689671 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:31.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91bbe00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91bc0cbb30 0x7f91bc0ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:31.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.389+0000 7f91c1200700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91bc0b2070 0x7f91bc14f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:31.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91c1c00700 1 -- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91bc0b2070 msgr2=0x7f91bc14f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:31.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91c1c00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91bc0b2070 0x7f91bc14f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:31.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91c1c00700 1 -- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91bc0cbb30 msgr2=0x7f91bc0ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:31.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91c1c00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91bc0cbb30 0x7f91bc0ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:31.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91c1c00700 1 -- 172.21.15.12:0/424689671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b0009550 con 0x7f91bc0cd020 2024-03-29T11:33:31.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91c1c00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cd020 0x7f91bc0af920 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f91b400c7a0 tx=0x7f91b400cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:31.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91b8c00700 1 -- 172.21.15.12:0/424689671 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b4003e30 con 0x7f91bc0cd020 2024-03-29T11:33:31.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91b8c00700 1 -- 172.21.15.12:0/424689671 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b4005070 con 0x7f91bc0cd020 2024-03-29T11:33:31.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91c3c00700 1 -- 172.21.15.12:0/424689671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91bc0ad230 con 0x7f91bc0cd020 2024-03-29T11:33:31.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91b8c00700 1 -- 172.21.15.12:0/424689671 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b4005420 con 0x7f91bc0cd020 2024-03-29T11:33:31.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.390+0000 7f91c3c00700 1 -- 172.21.15.12:0/424689671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91bc0ad7f0 con 0x7f91bc0cd020 2024-03-29T11:33:31.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.391+0000 7f91c3c00700 1 -- 172.21.15.12:0/424689671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91bc0add10 con 0x7f91bc0cd020 2024-03-29T11:33:31.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.392+0000 7f91b8c00700 1 -- 172.21.15.12:0/424689671 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f91b4005580 con 0x7f91bc0cd020 2024-03-29T11:33:31.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.393+0000 7f91b8c00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f919c06aed0 0x7f919c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:31.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.393+0000 7f91b8c00700 1 -- 172.21.15.12:0/424689671 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5875+0+0 (secure 0 0 0) 0x7f91b4088da0 con 0x7f91bc0cd020 2024-03-29T11:33:31.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.394+0000 7f91c1200700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f919c06aed0 0x7f919c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:31.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.394+0000 7f91c1200700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f919c06aed0 0x7f919c06d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f91b0014040 tx=0x7f91b0008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:31.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.398+0000 7f91b8c00700 1 -- 172.21.15.12:0/424689671 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91bc0add10 con 0x7f91bc0cd020 2024-03-29T11:33:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:31.626+0000 7f91c3c00700 1 -- 172.21.15.12:0/424689671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f91bc067720 con 0x7f91bc0cd020 2024-03-29T11:33:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.366+0000 7f91b8c00700 1 -- 172.21.15.12:0/424689671 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v265) v1 ==== 97+0+0 (secure 0 0 0) 0x7f91b4017070 con 0x7f91bc0cd020 2024-03-29T11:33:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 -- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f919c06aed0 msgr2=0x7f919c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f919c06aed0 0x7f919c06d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f91b0014040 tx=0x7f91b0008030 comp rx=0 tx=0).stop 2024-03-29T11:33:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 -- 172.21.15.12:0/424689671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cd020 msgr2=0x7f91bc0af920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cd020 0x7f91bc0af920 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f91b400c7a0 tx=0x7f91b400cab0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 -- 172.21.15.12:0/424689671 shutdown_connections 2024-03-29T11:33:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f919c06aed0 0x7f919c06d390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91bc0cd020 0x7f91bc0af920 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91bc0b2070 0x7f91bc14f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 --2- 172.21.15.12:0/424689671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91bc0cbb30 0x7f91bc0ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 -- 172.21.15.12:0/424689671 >> 172.21.15.12:0/424689671 conn(0x7f91bc0b0d70 msgr2=0x7f91bc0aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 -- 172.21.15.12:0/424689671 shutdown_connections 2024-03-29T11:33:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.369+0000 7f91c3c00700 1 -- 172.21.15.12:0/424689671 wait complete. 2024-03-29T11:33:32.367 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T11:33:32.386 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T11:33:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.592+0000 7fe0cc200700 1 Processor -- start 2024-03-29T11:33:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.592+0000 7fe0cc200700 1 -- start start 2024-03-29T11:33:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.593+0000 7fe0cc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c40ceec0 0x7fe0c41612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.593+0000 7fe0cc200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c40d30d0 con 0x7fe0c4161800 2024-03-29T11:33:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.593+0000 7fe0cc200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c40b6840 con 0x7fe0c415d640 2024-03-29T11:33:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.593+0000 7fe0cc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c4161c90 con 0x7fe0c40ceec0 2024-03-29T11:33:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.594+0000 7fe0ca200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe0c4161800 0x7fe0c40b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40766/0 (socket says 172.21.15.12:40766) 2024-03-29T11:33:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.594+0000 7fe0ca200700 1 -- 172.21.15.12:0/334303237 learned_addr learned my addr 172.21.15.12:0/334303237 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.594+0000 7fe0c8e00700 1 --2- 172.21.15.12:0/334303237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c40ceec0 0x7fe0c41612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.594+0000 7fe0c3e00700 1 -- 172.21.15.12:0/334303237 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964046847 0 0) 0x7fe0c40d30d0 con 0x7fe0c4161800 2024-03-29T11:33:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.594+0000 7fe0c3e00700 1 -- 172.21.15.12:0/334303237 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac0035b0 con 0x7fe0c4161800 2024-03-29T11:33:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.594+0000 7fe0c3e00700 1 -- 172.21.15.12:0/334303237 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482808978 0 0) 0x7fe0c40b6840 con 0x7fe0c415d640 2024-03-29T11:33:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.594+0000 7fe0c3e00700 1 -- 172.21.15.12:0/334303237 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c40d30d0 con 0x7fe0c415d640 2024-03-29T11:33:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.594+0000 7fe0c8e00700 1 -- 172.21.15.12:0/334303237 >> v1:172.21.15.159:6789/0 conn(0x7fe0c415d640 legacy=0x7fe0c40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.594+0000 7fe0c8e00700 1 -- 172.21.15.12:0/334303237 >> v1:172.21.15.12:6789/0 conn(0x7fe0c4161800 legacy=0x7fe0c40b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.594+0000 7fe0c8e00700 1 -- 172.21.15.12:0/334303237 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c40d2b30 con 0x7fe0c40ceec0 2024-03-29T11:33:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.595+0000 7fe0c8e00700 1 --2- 172.21.15.12:0/334303237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c40ceec0 0x7fe0c41612c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009870 tx=0x7fe0b0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72bb15ece64e5b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.595+0000 7fe0c3e00700 1 -- 172.21.15.12:0/334303237 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0011830 con 0x7fe0c40ceec0 2024-03-29T11:33:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.595+0000 7fe0c3e00700 1 -- 172.21.15.12:0/334303237 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b0011990 con 0x7fe0c40ceec0 2024-03-29T11:33:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.595+0000 7fe0c3e00700 1 -- 172.21.15.12:0/334303237 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0011d20 con 0x7fe0c40ceec0 2024-03-29T11:33:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.596+0000 7fe0cc200700 1 -- 172.21.15.12:0/334303237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c40ceec0 msgr2=0x7fe0c41612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.596+0000 7fe0cc200700 1 --2- 172.21.15.12:0/334303237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c40ceec0 0x7fe0c41612c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009870 tx=0x7fe0b0009b80 comp rx=0 tx=0).stop 2024-03-29T11:33:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.596+0000 7fe0cc200700 1 -- 172.21.15.12:0/334303237 shutdown_connections 2024-03-29T11:33:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.596+0000 7fe0cc200700 1 --2- 172.21.15.12:0/334303237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c40ceec0 0x7fe0c41612c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.596+0000 7fe0cc200700 1 -- 172.21.15.12:0/334303237 >> 172.21.15.12:0/334303237 conn(0x7fe0c408aff0 msgr2=0x7fe0c40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.596+0000 7fe0cc200700 1 -- 172.21.15.12:0/334303237 shutdown_connections 2024-03-29T11:33:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.596+0000 7fe0cc200700 1 -- 172.21.15.12:0/334303237 wait complete. 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.597+0000 7fe0cc200700 1 Processor -- start 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.598+0000 7fe0cc200700 1 -- start start 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.598+0000 7fe0cc200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0c40ceec0 0x7fe0c4158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.598+0000 7fe0cc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c415d640 0x7fe0c41f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0cc200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0c4161800 0x7fe0c40cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0cc200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40cb780 con 0x7fe0c4161800 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0cc200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40cd160 con 0x7fe0c40ceec0 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0cc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40cd440 con 0x7fe0c415d640 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0ca200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0c4161800 0x7fe0c40cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0ca200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0c4161800 0x7fe0c40cdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46580/0 (socket says 172.21.15.12:46580) 2024-03-29T11:33:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0ca200700 1 -- 172.21.15.12:0/604212489 learned_addr learned my addr 172.21.15.12:0/604212489 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0c9800700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0c40ceec0 0x7fe0c4158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0c8e00700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c415d640 0x7fe0c41f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0ca200700 1 -- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c415d640 msgr2=0x7fe0c41f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0ca200700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c415d640 0x7fe0c41f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0ca200700 1 -- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0c40ceec0 msgr2=0x7fe0c4158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0ca200700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0c40ceec0 0x7fe0c4158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0ca200700 1 -- 172.21.15.12:0/604212489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b00094a0 con 0x7fe0c4161800 2024-03-29T11:33:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.599+0000 7fe0c8e00700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c415d640 0x7fe0c41f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.600+0000 7fe0ca200700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0c4161800 0x7fe0c40cdc40 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00cba0 tx=0x7fe0bc00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.600+0000 7fe0c1600700 1 -- 172.21.15.12:0/604212489 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc00aa70 con 0x7fe0c4161800 2024-03-29T11:33:32.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.600+0000 7fe0c1600700 1 -- 172.21.15.12:0/604212489 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0bc00abd0 con 0x7fe0c4161800 2024-03-29T11:33:32.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.600+0000 7fe0c1600700 1 -- 172.21.15.12:0/604212489 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc012400 con 0x7fe0c4161800 2024-03-29T11:33:32.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.600+0000 7fe0cc200700 1 -- 172.21.15.12:0/604212489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c40cd720 con 0x7fe0c4161800 2024-03-29T11:33:32.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.600+0000 7fe0cc200700 1 -- 172.21.15.12:0/604212489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c41f4560 con 0x7fe0c4161800 2024-03-29T11:33:32.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.601+0000 7fe0cc200700 1 -- 172.21.15.12:0/604212489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c40ccfa0 con 0x7fe0c4161800 2024-03-29T11:33:32.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.602+0000 7fe0c1600700 1 -- 172.21.15.12:0/604212489 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe0bc01b3b0 con 0x7fe0c4161800 2024-03-29T11:33:32.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.603+0000 7fe0c1600700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0a406aed0 0x7fe0a406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:32.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.603+0000 7fe0c1600700 1 -- 172.21.15.12:0/604212489 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fe0bc089180 con 0x7fe0c4161800 2024-03-29T11:33:32.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.603+0000 7fe0c9800700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0a406aed0 0x7fe0a406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:32.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.604+0000 7fe0c9800700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0a406aed0 0x7fe0a406d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe0b80011e0 tx=0x7fe0b8009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.608+0000 7fe0c1600700 1 -- 172.21.15.12:0/604212489 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe0bc057d90 con 0x7fe0c4161800 2024-03-29T11:33:32.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.904+0000 7fe0cc200700 1 -- 172.21.15.12:0/604212489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe0c41f47b0 con 0x7fe0c4161800 2024-03-29T11:33:32.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.905+0000 7fe0c1600700 1 -- 172.21.15.12:0/604212489 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v52) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe0bc057920 con 0x7fe0c4161800 2024-03-29T11:33:32.901 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:33:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:33:32.907 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T11:33:32.908 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.909 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T11:33:32.910 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:33:32.911 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T11:33:32.912 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:33:32.913 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T11:33:32.914 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:33:32.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.907+0000 7fe0cc200700 1 -- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0a406aed0 msgr2=0x7fe0a406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.907+0000 7fe0cc200700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0a406aed0 0x7fe0a406d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe0b80011e0 tx=0x7fe0b8009040 comp rx=0 tx=0).stop 2024-03-29T11:33:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.907+0000 7fe0cc200700 1 -- 172.21.15.12:0/604212489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0c4161800 msgr2=0x7fe0c40cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.907+0000 7fe0cc200700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0c4161800 0x7fe0c40cdc40 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00cba0 tx=0x7fe0bc00ceb0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.907+0000 7fe0cc200700 1 -- 172.21.15.12:0/604212489 shutdown_connections 2024-03-29T11:33:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.907+0000 7fe0cc200700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe0a406aed0 0x7fe0a406d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.907+0000 7fe0cc200700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0c4161800 0x7fe0c40cdc40 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.907+0000 7fe0cc200700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0c40ceec0 0x7fe0c4158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.908+0000 7fe0cc200700 1 --2- 172.21.15.12:0/604212489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0c415d640 0x7fe0c41f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.908+0000 7fe0cc200700 1 -- 172.21.15.12:0/604212489 >> 172.21.15.12:0/604212489 conn(0x7fe0c408aff0 msgr2=0x7fe0c406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.908+0000 7fe0cc200700 1 -- 172.21.15.12:0/604212489 shutdown_connections 2024-03-29T11:33:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:32.908+0000 7fe0cc200700 1 -- 172.21.15.12:0/604212489 wait complete. 2024-03-29T11:33:32.922 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:33.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.124+0000 7f21ada00700 1 Processor -- start 2024-03-29T11:33:33.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.124+0000 7f21ada00700 1 -- start start 2024-03-29T11:33:33.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.126+0000 7f21ada00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f21a80b3af0 0x7f21a815a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.126+0000 7f21ada00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a80b2180 con 0x7f21a8072580 2024-03-29T11:33:33.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.126+0000 7f21ada00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a815b100 con 0x7f21a80709c0 2024-03-29T11:33:33.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.126+0000 7f21ada00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a815b3e0 con 0x7f21a80b3af0 2024-03-29T11:33:33.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.126+0000 7f21a6000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f21a8072580 0x7f21a80b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40772/0 (socket says 172.21.15.12:40772) 2024-03-29T11:33:33.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.126+0000 7f21a6000700 1 -- 172.21.15.12:0/1029048729 learned_addr learned my addr 172.21.15.12:0/1029048729 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:33.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.126+0000 7f21a6a00700 1 --2- 172.21.15.12:0/1029048729 >> v2:172.21.15.159:3300/0 conn(0x7f21a80b3af0 0x7f21a815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.126+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165251686 0 0) 0x7f21a80b2180 con 0x7f21a8072580 2024-03-29T11:33:33.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.126+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f218c0035b0 con 0x7f21a8072580 2024-03-29T11:33:33.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763159467 0 0) 0x7f21a815b100 con 0x7f21a80709c0 2024-03-29T11:33:33.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21a80b2180 con 0x7f21a80709c0 2024-03-29T11:33:33.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 718246187 0 0) 0x7f218c0035b0 con 0x7f21a8072580 2024-03-29T11:33:33.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 >> v2:172.21.15.159:3300/0 conn(0x7f21a80b3af0 msgr2=0x7f21a815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:33.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 --2- 172.21.15.12:0/1029048729 >> v2:172.21.15.159:3300/0 conn(0x7f21a80b3af0 0x7f21a815a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 >> v1:172.21.15.159:6789/0 conn(0x7f21a80709c0 legacy=0x7f21a806e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80b23f0 con 0x7f21a8072580 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f2194001670 con 0x7f21a8072580 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 >> v1:172.21.15.12:6789/0 conn(0x7f21a8072580 legacy=0x7f21a80b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f218c003be0 0x7f218c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f218c004850 0x7f218c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f218c005210 0x7f218c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a815b3e0 con 0x7f218c003be0 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c00c0e0 con 0x7f218c004850 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218c00c2a0 con 0x7f218c005210 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a6a00700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f218c003be0 0x7f218c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a6000700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f218c004850 0x7f218c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.127+0000 7f21a7400700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f218c005210 0x7f218c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a6a00700 1 -- 172.21.15.12:0/1029048729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f218c005210 msgr2=0x7f218c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:33.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a6a00700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f218c005210 0x7f218c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a6a00700 1 -- 172.21.15.12:0/1029048729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f218c004850 msgr2=0x7f218c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a6a00700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f218c004850 0x7f218c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a6a00700 1 -- 172.21.15.12:0/1029048729 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0035b0 con 0x7f218c003be0 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a6000700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f218c004850 0x7f218c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a7400700 1 -- 172.21.15.12:0/1029048729 reap_dead start 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a6a00700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f218c003be0 0x7f218c003fe0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f2198009b60 tx=0x7f218c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26618c1a02e4782f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219800e6c0 con 0x7f218c003be0 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219800e820 con 0x7f218c003be0 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.128+0000 7f21a5600700 1 -- 172.21.15.12:0/1029048729 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219800ebb0 con 0x7f218c003be0 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.129+0000 7f21ada00700 1 -- 172.21.15.12:0/1029048729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f218c003be0 msgr2=0x7f218c003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.129+0000 7f21ada00700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f218c003be0 0x7f218c003fe0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f2198009b60 tx=0x7f218c005880 comp rx=0 tx=0).stop 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.129+0000 7f21ada00700 1 -- 172.21.15.12:0/1029048729 shutdown_connections 2024-03-29T11:33:33.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.129+0000 7f21ada00700 1 --2- 172.21.15.12:0/1029048729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f218c003be0 0x7f218c003fe0 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.129+0000 7f21ada00700 1 -- 172.21.15.12:0/1029048729 >> 172.21.15.12:0/1029048729 conn(0x7f21a80cf190 msgr2=0x7f21a80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:33.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.129+0000 7f21ada00700 1 -- 172.21.15.12:0/1029048729 shutdown_connections 2024-03-29T11:33:33.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.129+0000 7f21ada00700 1 -- 172.21.15.12:0/1029048729 wait complete. 2024-03-29T11:33:33.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.130+0000 7f21ada00700 1 Processor -- start 2024-03-29T11:33:33.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.131+0000 7f21ada00700 1 -- start start 2024-03-29T11:33:33.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.131+0000 7f21ada00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f218c003be0 0x7f21a81458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21ada00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21a80b2630 0x7f21a80b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21ada00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a815b940 0x7f21a815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21ada00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81562e0 con 0x7f21a815b940 2024-03-29T11:33:33.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21ada00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80b2ff0 con 0x7f218c003be0 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21ada00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81f3d30 con 0x7f21a80b2630 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a815b940 0x7f21a815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a815b940 0x7f21a815bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46596/0 (socket says 172.21.15.12:46596) 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a7400700 1 -- 172.21.15.12:0/1925868704 learned_addr learned my addr 172.21.15.12:0/1925868704 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a6a00700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f218c003be0 0x7f21a81458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a6000700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21a80b2630 0x7f21a80b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a7400700 1 -- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21a80b2630 msgr2=0x7f21a80b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a7400700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21a80b2630 0x7f21a80b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a7400700 1 -- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f218c003be0 msgr2=0x7f21a81458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a7400700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f218c003be0 0x7f21a81458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a7400700 1 -- 172.21.15.12:0/1925868704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a815c670 con 0x7f21a815b940 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.132+0000 7f21a7400700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a815b940 0x7f21a815bdc0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f21a80ac680 tx=0x7f21a000c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:33.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.133+0000 7f219ea00700 1 -- 172.21.15.12:0/1925868704 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0003c10 con 0x7f21a815b940 2024-03-29T11:33:33.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.133+0000 7f219ea00700 1 -- 172.21.15.12:0/1925868704 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a0003d90 con 0x7f21a815b940 2024-03-29T11:33:33.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.133+0000 7f219ea00700 1 -- 172.21.15.12:0/1925868704 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a000ca50 con 0x7f21a815b940 2024-03-29T11:33:33.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.133+0000 7f21ada00700 1 -- 172.21.15.12:0/1925868704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a81f3400 con 0x7f21a815b940 2024-03-29T11:33:33.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.133+0000 7f21ada00700 1 -- 172.21.15.12:0/1925868704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a81f39c0 con 0x7f21a815b940 2024-03-29T11:33:33.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.134+0000 7f21ada00700 1 -- 172.21.15.12:0/1925868704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a8067720 con 0x7f21a815b940 2024-03-29T11:33:33.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.135+0000 7f219ea00700 1 -- 172.21.15.12:0/1925868704 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f21a000cbb0 con 0x7f21a815b940 2024-03-29T11:33:33.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.136+0000 7f219ea00700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f218806aed0 0x7f218806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.136+0000 7f219ea00700 1 -- 172.21.15.12:0/1925868704 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f21a00895b0 con 0x7f21a815b940 2024-03-29T11:33:33.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.137+0000 7f21a6a00700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f218806aed0 0x7f218806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.137+0000 7f21a6a00700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f218806aed0 0x7f218806d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f21980090b0 tx=0x7f2198008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:33.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.141+0000 7f219ea00700 1 -- 172.21.15.12:0/1925868704 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21a0054730 con 0x7f21a815b940 2024-03-29T11:33:33.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.352+0000 7f21ada00700 1 -- 172.21.15.12:0/1925868704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21a81f3e70 con 0x7f21a815b940 2024-03-29T11:33:33.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.353+0000 7f219ea00700 1 -- 172.21.15.12:0/1925868704 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+12515 (secure 0 0 0) 0x7f21a0057d50 con 0x7f21a815b940 2024-03-29T11:33:33.350 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:33.350 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":265,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:32.362331+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:33.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.355+0000 7f21ada00700 1 -- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f218806aed0 msgr2=0x7f218806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.355+0000 7f21ada00700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f218806aed0 0x7f218806d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f21980090b0 tx=0x7f2198008030 comp rx=0 tx=0).stop 2024-03-29T11:33:33.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.356+0000 7f21ada00700 1 -- 172.21.15.12:0/1925868704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a815b940 msgr2=0x7f21a815bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.356+0000 7f21ada00700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a815b940 0x7f21a815bdc0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f21a80ac680 tx=0x7f21a000c0e0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.356+0000 7f21ada00700 1 -- 172.21.15.12:0/1925868704 shutdown_connections 2024-03-29T11:33:33.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.356+0000 7f21ada00700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f218806aed0 0x7f218806d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.356+0000 7f21ada00700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21a815b940 0x7f21a815bdc0 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.356+0000 7f21ada00700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f218c003be0 0x7f21a81458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.356+0000 7f21ada00700 1 --2- 172.21.15.12:0/1925868704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21a80b2630 0x7f21a80b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.356+0000 7f21ada00700 1 -- 172.21.15.12:0/1925868704 >> 172.21.15.12:0/1925868704 conn(0x7f21a80cf190 msgr2=0x7f21a806d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:33.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.356+0000 7f21ada00700 1 -- 172.21.15.12:0/1925868704 shutdown_connections 2024-03-29T11:33:33.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.356+0000 7f21ada00700 1 -- 172.21.15.12:0/1925868704 wait complete. 2024-03-29T11:33:33.371 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:33:33.371 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:33:33.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.575+0000 7f20de800700 1 Processor -- start 2024-03-29T11:33:33.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.575+0000 7f20de800700 1 -- start start 2024-03-29T11:33:33.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.577+0000 7f20de800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f20d80cbb30 0x7f20d80cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.577+0000 7f20de800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cd020 0x7f20d80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.577+0000 7f20de800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d80cc450 con 0x7f20d80b2070 2024-03-29T11:33:33.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.577+0000 7f20de800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d80cdae0 con 0x7f20d80cbb30 2024-03-29T11:33:33.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.577+0000 7f20de800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d80cddc0 con 0x7f20d80cd020 2024-03-29T11:33:33.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.577+0000 7f20d6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cd020 0x7f20d80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.577+0000 7f20d7400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f20d80cbb30 0x7f20d80cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.577+0000 7f20d6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cd020 0x7f20d80cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:54896/0 (socket says 172.21.15.12:54896) 2024-03-29T11:33:33.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.577+0000 7f20d7e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f20d80b2070 0x7f20d806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:42616/0 (socket says 172.21.15.12:42616) 2024-03-29T11:33:33.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.577+0000 7f20d6a00700 1 -- 172.21.15.12:0/1239470831 learned_addr learned my addr 172.21.15.12:0/1239470831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:33.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.578+0000 7f20d6a00700 1 -- 172.21.15.12:0/1239470831 >> v2:172.21.15.159:3300/0 conn(0x7f20d80cbb30 msgr2=0x7f20d80cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:33.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.578+0000 7f20d6a00700 1 --2- 172.21.15.12:0/1239470831 >> v2:172.21.15.159:3300/0 conn(0x7f20d80cbb30 0x7f20d80cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.578+0000 7f20d6a00700 1 -- 172.21.15.12:0/1239470831 >> v1:172.21.15.159:6789/0 conn(0x7f20d80b2070 legacy=0x7f20d806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.578+0000 7f20d6a00700 1 -- 172.21.15.12:0/1239470831 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d80ce0a0 con 0x7f20d80cd020 2024-03-29T11:33:33.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.578+0000 7f20d6a00700 1 --2- 172.21.15.12:0/1239470831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cd020 0x7f20d80cd480 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f20c40098f0 tx=0x7f20c4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42da10f1e9b5c57a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.578+0000 7f20d6000700 1 -- 172.21.15.12:0/1239470831 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4011830 con 0x7f20d80cd020 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.578+0000 7f20d6000700 1 -- 172.21.15.12:0/1239470831 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c4011990 con 0x7f20d80cd020 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.578+0000 7f20d6000700 1 -- 172.21.15.12:0/1239470831 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4011d20 con 0x7f20d80cd020 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.579+0000 7f20de800700 1 -- 172.21.15.12:0/1239470831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cd020 msgr2=0x7f20d80cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.579+0000 7f20de800700 1 --2- 172.21.15.12:0/1239470831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cd020 0x7f20d80cd480 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f20c40098f0 tx=0x7f20c4009c00 comp rx=0 tx=0).stop 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.579+0000 7f20de800700 1 -- 172.21.15.12:0/1239470831 shutdown_connections 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.579+0000 7f20de800700 1 --2- 172.21.15.12:0/1239470831 >> v2:172.21.15.159:3300/0 conn(0x7f20d80cbb30 0x7f20d80cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.579+0000 7f20de800700 1 --2- 172.21.15.12:0/1239470831 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cd020 0x7f20d80cd480 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.579+0000 7f20de800700 1 -- 172.21.15.12:0/1239470831 >> 172.21.15.12:0/1239470831 conn(0x7f20d80b0d70 msgr2=0x7f20d80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.579+0000 7f20de800700 1 -- 172.21.15.12:0/1239470831 shutdown_connections 2024-03-29T11:33:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.579+0000 7f20de800700 1 -- 172.21.15.12:0/1239470831 wait complete. 2024-03-29T11:33:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.581+0000 7f20de800700 1 Processor -- start 2024-03-29T11:33:33.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.581+0000 7f20de800700 1 -- start start 2024-03-29T11:33:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20de800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20d80b2070 0x7f20d80cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20de800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cbb30 0x7f20d8152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20de800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20d80cd020 0x7f20d80c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20de800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d8157160 con 0x7f20d80cd020 2024-03-29T11:33:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20de800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d8158130 con 0x7f20d80b2070 2024-03-29T11:33:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20de800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d8152ff0 con 0x7f20d80cbb30 2024-03-29T11:33:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20d7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20d80cd020 0x7f20d80c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20d7400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20d80b2070 0x7f20d80cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20d7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20d80cd020 0x7f20d80c8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46598/0 (socket says 172.21.15.12:46598) 2024-03-29T11:33:33.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20d7e00700 1 -- 172.21.15.12:0/2309551334 learned_addr learned my addr 172.21.15.12:0/2309551334 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20d7400700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20d80b2070 0x7f20d80cac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:55648/0 (socket says 172.21.15.12:55648) 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.582+0000 7f20d6a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cbb30 0x7f20d8152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.583+0000 7f20d7e00700 1 -- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cbb30 msgr2=0x7f20d8152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.583+0000 7f20d7e00700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cbb30 0x7f20d8152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.583+0000 7f20d7e00700 1 -- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20d80b2070 msgr2=0x7f20d80cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.583+0000 7f20d7e00700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20d80b2070 0x7f20d80cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.583+0000 7f20d7e00700 1 -- 172.21.15.12:0/2309551334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c4009550 con 0x7f20d80cd020 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.583+0000 7f20d7e00700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20d80cd020 0x7f20d80c8920 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f20d000cbc0 tx=0x7f20d000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.583+0000 7f20cf400700 1 -- 172.21.15.12:0/2309551334 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d000a9e0 con 0x7f20d80cd020 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.583+0000 7f20cf400700 1 -- 172.21.15.12:0/2309551334 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d000ab40 con 0x7f20d80cd020 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.583+0000 7f20cf400700 1 -- 172.21.15.12:0/2309551334 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d0007070 con 0x7f20d80cd020 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.583+0000 7f20de800700 1 -- 172.21.15.12:0/2309551334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d8157370 con 0x7f20d80cd020 2024-03-29T11:33:33.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.584+0000 7f20de800700 1 -- 172.21.15.12:0/2309551334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d8157930 con 0x7f20d80cd020 2024-03-29T11:33:33.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.585+0000 7f20de800700 1 -- 172.21.15.12:0/2309551334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d8067720 con 0x7f20d80cd020 2024-03-29T11:33:33.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.585+0000 7f20cf400700 1 -- 172.21.15.12:0/2309551334 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f20d00071d0 con 0x7f20d80cd020 2024-03-29T11:33:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.586+0000 7f20cf400700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f20b806aed0 0x7f20b806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.586+0000 7f20cf400700 1 -- 172.21.15.12:0/2309551334 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f20d0088f80 con 0x7f20d80cd020 2024-03-29T11:33:33.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.587+0000 7f20d7400700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f20b806aed0 0x7f20b806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:33.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.588+0000 7f20d7400700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f20b806aed0 0x7f20b806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009b40 tx=0x7f20c8009210 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:33.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.591+0000 7f20cf400700 1 -- 172.21.15.12:0/2309551334 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f20d00541a0 con 0x7f20d80cd020 2024-03-29T11:33:33.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.805+0000 7f20de800700 1 -- 172.21.15.12:0/2309551334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f20d815bda0 con 0x7f20d80cd020 2024-03-29T11:33:33.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.805+0000 7f20cf400700 1 -- 172.21.15.12:0/2309551334 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v265) v1 ==== 93+0+10 (secure 0 0 0) 0x7f20d0017070 con 0x7f20d80cd020 2024-03-29T11:33:33.802 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:33:33.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 -- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f20b806aed0 msgr2=0x7f20b806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f20b806aed0 0x7f20b806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009b40 tx=0x7f20c8009210 comp rx=0 tx=0).stop 2024-03-29T11:33:33.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 -- 172.21.15.12:0/2309551334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20d80cd020 msgr2=0x7f20d80c8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:33.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20d80cd020 0x7f20d80c8920 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f20d000cbc0 tx=0x7f20d000ced0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 -- 172.21.15.12:0/2309551334 shutdown_connections 2024-03-29T11:33:33.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f20b806aed0 0x7f20b806d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20d80cd020 0x7f20d80c8920 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20d80b2070 0x7f20d80cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 --2- 172.21.15.12:0/2309551334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20d80cbb30 0x7f20d8152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:33.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 -- 172.21.15.12:0/2309551334 >> 172.21.15.12:0/2309551334 conn(0x7f20d80b0d70 msgr2=0x7f20d80af910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:33.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 -- 172.21.15.12:0/2309551334 shutdown_connections 2024-03-29T11:33:33.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:33.808+0000 7f20de800700 1 -- 172.21.15.12:0/2309551334 wait complete. 2024-03-29T11:33:33.823 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T11:33:34.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.024+0000 7f93d6600700 1 Processor -- start 2024-03-29T11:33:34.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.024+0000 7f93d6600700 1 -- start start 2024-03-29T11:33:34.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.025+0000 7f93d6600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f93d0160f10 0x7f93d00d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.025+0000 7f93d6600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d00ceda0 con 0x7f93d00cf230 2024-03-29T11:33:34.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.025+0000 7f93d6600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d01612f0 con 0x7f93d0161590 2024-03-29T11:33:34.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.025+0000 7f93d6600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d00ce470 con 0x7f93d0160f10 2024-03-29T11:33:34.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.025+0000 7f93cea00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f93d00cf230 0x7f93d00d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40788/0 (socket says 172.21.15.12:40788) 2024-03-29T11:33:34.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.025+0000 7f93cea00700 1 -- 172.21.15.12:0/575954233 learned_addr learned my addr 172.21.15.12:0/575954233 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:34.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93cf400700 1 --2- 172.21.15.12:0/575954233 >> v2:172.21.15.159:3300/0 conn(0x7f93d0160f10 0x7f93d00d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135797645 0 0) 0x7f93d00ceda0 con 0x7f93d00cf230 2024-03-29T11:33:34.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b40035b0 con 0x7f93d00cf230 2024-03-29T11:33:34.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796797784 0 0) 0x7f93d01612f0 con 0x7f93d0161590 2024-03-29T11:33:34.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d00ceda0 con 0x7f93d0161590 2024-03-29T11:33:34.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3675650491 0 0) 0x7f93b40035b0 con 0x7f93d00cf230 2024-03-29T11:33:34.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 >> v2:172.21.15.159:3300/0 conn(0x7f93d0160f10 msgr2=0x7f93d00d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 --2- 172.21.15.12:0/575954233 >> v2:172.21.15.159:3300/0 conn(0x7f93d0160f10 0x7f93d00d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 >> v1:172.21.15.159:6789/0 conn(0x7f93d0161590 legacy=0x7f93d00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d00ce750 con 0x7f93d00cf230 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f93bc001670 con 0x7f93d00cf230 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 >> v1:172.21.15.12:6789/0 conn(0x7f93d00cf230 legacy=0x7f93d00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b4003be0 0x7f93b4003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93b4004850 0x7f93b4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.026+0000 7f93ce000700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b4005260 0x7f93b400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d00ce470 con 0x7f93b4003be0 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b400c190 con 0x7f93b4005260 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b400c350 con 0x7f93b4004850 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93cf400700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b4003be0 0x7f93b4003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93cea00700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93b4004850 0x7f93b4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93cfe00700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b4005260 0x7f93b400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93cf400700 1 -- 172.21.15.12:0/575954233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93b4004850 msgr2=0x7f93b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93cf400700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93b4004850 0x7f93b4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93cf400700 1 -- 172.21.15.12:0/575954233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b4005260 msgr2=0x7f93b400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93cf400700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b4005260 0x7f93b400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93cf400700 1 -- 172.21.15.12:0/575954233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b40035b0 con 0x7f93b4003be0 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93cfe00700 1 -- 172.21.15.12:0/575954233 reap_dead start 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.027+0000 7f93cf400700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b4003be0 0x7f93b4003fe0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009b60 tx=0x7f93b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c107039c648014 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.028+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c000e6c0 con 0x7f93b4003be0 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.028+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c000e820 con 0x7f93b4003be0 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.028+0000 7f93ce000700 1 -- 172.21.15.12:0/575954233 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c000eb90 con 0x7f93b4003be0 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.028+0000 7f93d6600700 1 -- 172.21.15.12:0/575954233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b4003be0 msgr2=0x7f93b4003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.028+0000 7f93d6600700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b4003be0 0x7f93b4003fe0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009b60 tx=0x7f93b4005880 comp rx=0 tx=0).stop 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.028+0000 7f93d6600700 1 -- 172.21.15.12:0/575954233 shutdown_connections 2024-03-29T11:33:34.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.028+0000 7f93d6600700 1 --2- 172.21.15.12:0/575954233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b4003be0 0x7f93b4003fe0 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.028+0000 7f93d6600700 1 -- 172.21.15.12:0/575954233 >> 172.21.15.12:0/575954233 conn(0x7f93d008aff0 msgr2=0x7f93d00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.028+0000 7f93d6600700 1 -- 172.21.15.12:0/575954233 shutdown_connections 2024-03-29T11:33:34.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.028+0000 7f93d6600700 1 -- 172.21.15.12:0/575954233 wait complete. 2024-03-29T11:33:34.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.030+0000 7f93d6600700 1 Processor -- start 2024-03-29T11:33:34.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.030+0000 7f93d6600700 1 -- start start 2024-03-29T11:33:34.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.031+0000 7f93d6600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b4003be0 0x7f93d01f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.031+0000 7f93d6600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93d01e66b0 0x7f93d00b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.031+0000 7f93d6600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d01524e0 0x7f93d0152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.031+0000 7f93d6600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0153b40 con 0x7f93d01e66b0 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.031+0000 7f93d6600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0153810 con 0x7f93b4003be0 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.031+0000 7f93d6600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01e6ec0 con 0x7f93d01524e0 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.031+0000 7f93cea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93d01e66b0 0x7f93d00b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.031+0000 7f93cea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93d01e66b0 0x7f93d00b7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46622/0 (socket says 172.21.15.12:46622) 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.031+0000 7f93cea00700 1 -- 172.21.15.12:0/1135465386 learned_addr learned my addr 172.21.15.12:0/1135465386 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93cf400700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b4003be0 0x7f93d01f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93cfe00700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d01524e0 0x7f93d0152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93cea00700 1 -- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d01524e0 msgr2=0x7f93d0152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93cea00700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d01524e0 0x7f93d0152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93cea00700 1 -- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b4003be0 msgr2=0x7f93d01f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93cea00700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b4003be0 0x7f93d01f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93cea00700 1 -- 172.21.15.12:0/1135465386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d01e71a0 con 0x7f93d01e66b0 2024-03-29T11:33:34.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93cfe00700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d01524e0 0x7f93d0152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:34.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93cea00700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93d01e66b0 0x7f93d00b7df0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f93bc001b80 tx=0x7f93bc00ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:34.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93c7400700 1 -- 172.21.15.12:0/1135465386 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc014480 con 0x7f93d01e66b0 2024-03-29T11:33:34.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93c7400700 1 -- 172.21.15.12:0/1135465386 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93bc0145e0 con 0x7f93d01e66b0 2024-03-29T11:33:34.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93c7400700 1 -- 172.21.15.12:0/1135465386 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc014990 con 0x7f93d01e66b0 2024-03-29T11:33:34.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.032+0000 7f93d6600700 1 -- 172.21.15.12:0/1135465386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d01e7480 con 0x7f93d01e66b0 2024-03-29T11:33:34.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.033+0000 7f93d6600700 1 -- 172.21.15.12:0/1135465386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d01e6b30 con 0x7f93d01e66b0 2024-03-29T11:33:34.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.034+0000 7f93d6600700 1 -- 172.21.15.12:0/1135465386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d00ce0e0 con 0x7f93d01e66b0 2024-03-29T11:33:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.035+0000 7f93c7400700 1 -- 172.21.15.12:0/1135465386 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f93bc0114f0 con 0x7f93d01e66b0 2024-03-29T11:33:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.035+0000 7f93c7400700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f93b006aed0 0x7f93b006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.036+0000 7f93c7400700 1 -- 172.21.15.12:0/1135465386 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f93bc088ad0 con 0x7f93d01e66b0 2024-03-29T11:33:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.036+0000 7f93cf400700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f93b006aed0 0x7f93b006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.037+0000 7f93cf400700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f93b006aed0 0x7f93b006d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f93c00090b0 tx=0x7f93c001e040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:34.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.040+0000 7f93c7400700 1 -- 172.21.15.12:0/1135465386 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f93bc053d10 con 0x7f93d01e66b0 2024-03-29T11:33:34.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.260+0000 7f93d6600700 1 -- 172.21.15.12:0/1135465386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f93d0067720 con 0x7f93d01e66b0 2024-03-29T11:33:34.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.260+0000 7f93c7400700 1 -- 172.21.15.12:0/1135465386 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v265) v1 ==== 84+0+65 (secure 0 0 0) 0x7f93bc019070 con 0x7f93d01e66b0 2024-03-29T11:33:34.257 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:34.257 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:33:34.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:34.257 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T11:33:34.257 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T11:33:34.258 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:34.258 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 -- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f93b006aed0 msgr2=0x7f93b006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f93b006aed0 0x7f93b006d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f93c00090b0 tx=0x7f93c001e040 comp rx=0 tx=0).stop 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 -- 172.21.15.12:0/1135465386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93d01e66b0 msgr2=0x7f93d00b7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93d01e66b0 0x7f93d00b7df0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f93bc001b80 tx=0x7f93bc00ea60 comp rx=0 tx=0).stop 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 -- 172.21.15.12:0/1135465386 shutdown_connections 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f93b006aed0 0x7f93b006d390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d01524e0 0x7f93d0152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b4003be0 0x7f93d01f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 --2- 172.21.15.12:0/1135465386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93d01e66b0 0x7f93d00b7df0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 -- 172.21.15.12:0/1135465386 >> 172.21.15.12:0/1135465386 conn(0x7f93d008aff0 msgr2=0x7f93d00cca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 -- 172.21.15.12:0/1135465386 shutdown_connections 2024-03-29T11:33:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.263+0000 7f93d6600700 1 -- 172.21.15.12:0/1135465386 wait complete. 2024-03-29T11:33:34.277 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T11:33:34.277 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T11:33:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.492+0000 7fd6daa00700 1 Processor -- start 2024-03-29T11:33:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.492+0000 7fd6daa00700 1 -- start start 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.493+0000 7fd6daa00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd6d40b5580 0x7fd6d415e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.493+0000 7fd6daa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6d40d21f0 0x7fd6d40cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.493+0000 7fd6daa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d415ee70 con 0x7fd6d415f260 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.493+0000 7fd6daa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d40ce430 con 0x7fd6d40d21f0 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.493+0000 7fd6daa00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d40ce710 con 0x7fd6d40b5580 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.493+0000 7fd6d8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6d40d21f0 0x7fd6d40cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.493+0000 7fd6d8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6d40d21f0 0x7fd6d40cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46628/0 (socket says 172.21.15.12:46628) 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.493+0000 7fd6d8a00700 1 -- 172.21.15.12:0/1968045815 learned_addr learned my addr 172.21.15.12:0/1968045815 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.493+0000 7fd6d3e00700 1 --2- 172.21.15.12:0/1968045815 >> v2:172.21.15.159:3300/0 conn(0x7fd6d40b5580 0x7fd6d415e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.494+0000 7fd6d8a00700 1 -- 172.21.15.12:0/1968045815 >> v2:172.21.15.159:3300/0 conn(0x7fd6d40b5580 msgr2=0x7fd6d415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.494+0000 7fd6d8a00700 1 --2- 172.21.15.12:0/1968045815 >> v2:172.21.15.159:3300/0 conn(0x7fd6d40b5580 0x7fd6d415e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.494+0000 7fd6d8a00700 1 -- 172.21.15.12:0/1968045815 >> v1:172.21.15.159:6789/0 conn(0x7fd6d415f260 legacy=0x7fd6d40d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.494+0000 7fd6d8a00700 1 -- 172.21.15.12:0/1968045815 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d40ce9f0 con 0x7fd6d40d21f0 2024-03-29T11:33:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.494+0000 7fd6d3e00700 1 --2- 172.21.15.12:0/1968045815 >> v2:172.21.15.159:3300/0 conn(0x7fd6d40b5580 0x7fd6d415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.494+0000 7fd6d8a00700 1 --2- 172.21.15.12:0/1968045815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40d21f0 0x7fd6d40cf4d0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fd6d408b8b0 tx=0x7fd6cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b55c8652c7a881 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.494+0000 7fd6d2a00700 1 -- 172.21.15.12:0/1968045815 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc00e070 con 0x7fd6d40d21f0 2024-03-29T11:33:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.494+0000 7fd6d2a00700 1 -- 172.21.15.12:0/1968045815 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc00b9c0 con 0x7fd6d40d21f0 2024-03-29T11:33:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.494+0000 7fd6d2a00700 1 -- 172.21.15.12:0/1968045815 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc00bd70 con 0x7fd6d40d21f0 2024-03-29T11:33:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.495+0000 7fd6daa00700 1 -- 172.21.15.12:0/1968045815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40d21f0 msgr2=0x7fd6d40cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.495+0000 7fd6daa00700 1 --2- 172.21.15.12:0/1968045815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40d21f0 0x7fd6d40cf4d0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fd6d408b8b0 tx=0x7fd6cc00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.495+0000 7fd6daa00700 1 -- 172.21.15.12:0/1968045815 shutdown_connections 2024-03-29T11:33:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.495+0000 7fd6daa00700 1 --2- 172.21.15.12:0/1968045815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40d21f0 0x7fd6d40cf4d0 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.495+0000 7fd6daa00700 1 --2- 172.21.15.12:0/1968045815 >> v2:172.21.15.159:3300/0 conn(0x7fd6d40b5580 0x7fd6d415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.495+0000 7fd6daa00700 1 -- 172.21.15.12:0/1968045815 >> 172.21.15.12:0/1968045815 conn(0x7fd6d408aff0 msgr2=0x7fd6d40d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.495+0000 7fd6daa00700 1 -- 172.21.15.12:0/1968045815 shutdown_connections 2024-03-29T11:33:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.495+0000 7fd6daa00700 1 -- 172.21.15.12:0/1968045815 wait complete. 2024-03-29T11:33:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.497+0000 7fd6daa00700 1 Processor -- start 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.498+0000 7fd6daa00700 1 -- start start 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6daa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40b5580 0x7fd6d4159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6daa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d40d21f0 0x7fd6d40b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6daa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d415f260 0x7fd6d40d1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6daa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d4159950 con 0x7fd6d40b5580 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6daa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d41f4350 con 0x7fd6d40d21f0 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6daa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d41f37d0 con 0x7fd6d415f260 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6d3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40b5580 0x7fd6d4159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6d3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40b5580 0x7fd6d4159410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46642/0 (socket says 172.21.15.12:46642) 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6d3e00700 1 -- 172.21.15.12:0/3873162703 learned_addr learned my addr 172.21.15.12:0/3873162703 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6d8a00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d415f260 0x7fd6d40d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6d3400700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d40d21f0 0x7fd6d40b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6d3e00700 1 -- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d415f260 msgr2=0x7fd6d40d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6d3e00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d415f260 0x7fd6d40d1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6d3e00700 1 -- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d40d21f0 msgr2=0x7fd6d40b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6d3e00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d40d21f0 0x7fd6d40b87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:34.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.499+0000 7fd6d3e00700 1 -- 172.21.15.12:0/3873162703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc009b30 con 0x7fd6d40b5580 2024-03-29T11:33:34.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.500+0000 7fd6d3e00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40b5580 0x7fd6d4159410 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fd6c400e810 tx=0x7fd6c400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:34.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.500+0000 7fd6cbe00700 1 -- 172.21.15.12:0/3873162703 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c400c020 con 0x7fd6d40b5580 2024-03-29T11:33:34.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.500+0000 7fd6cbe00700 1 -- 172.21.15.12:0/3873162703 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c4004250 con 0x7fd6d40b5580 2024-03-29T11:33:34.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.500+0000 7fd6cbe00700 1 -- 172.21.15.12:0/3873162703 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40036a0 con 0x7fd6d40b5580 2024-03-29T11:33:34.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.500+0000 7fd6daa00700 1 -- 172.21.15.12:0/3873162703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d41f3ab0 con 0x7fd6d40b5580 2024-03-29T11:33:34.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.500+0000 7fd6daa00700 1 -- 172.21.15.12:0/3873162703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d41f4070 con 0x7fd6d40b5580 2024-03-29T11:33:34.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.501+0000 7fd6daa00700 1 -- 172.21.15.12:0/3873162703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d4067720 con 0x7fd6d40b5580 2024-03-29T11:33:34.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.502+0000 7fd6cbe00700 1 -- 172.21.15.12:0/3873162703 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd6c4003800 con 0x7fd6d40b5580 2024-03-29T11:33:34.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.503+0000 7fd6cbe00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406aed0 0x7fd6b406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:34.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.503+0000 7fd6cbe00700 1 -- 172.21.15.12:0/3873162703 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fd6c40892d0 con 0x7fd6d40b5580 2024-03-29T11:33:34.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.504+0000 7fd6d3400700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406aed0 0x7fd6b406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:34.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.504+0000 7fd6d3400700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406aed0 0x7fd6b406d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd6d40b8520 tx=0x7fd6c0009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.507+0000 7fd6cbe00700 1 -- 172.21.15.12:0/3873162703 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd6c4054330 con 0x7fd6d40b5580 2024-03-29T11:33:34.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:34.721+0000 7fd6daa00700 1 -- 172.21.15.12:0/3873162703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd6d40d3cc0 con 0x7fd6d40b5580 2024-03-29T11:33:35.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.334+0000 7fd6cbe00700 1 -- 172.21.15.12:0/3873162703 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v266) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd6c4057950 con 0x7fd6d40b5580 2024-03-29T11:33:35.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.338+0000 7fd6daa00700 1 -- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406aed0 msgr2=0x7fd6b406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:35.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.338+0000 7fd6daa00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406aed0 0x7fd6b406d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd6d40b8520 tx=0x7fd6c0009040 comp rx=0 tx=0).stop 2024-03-29T11:33:35.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.338+0000 7fd6daa00700 1 -- 172.21.15.12:0/3873162703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40b5580 msgr2=0x7fd6d4159410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:35.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.338+0000 7fd6daa00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40b5580 0x7fd6d4159410 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fd6c400e810 tx=0x7fd6c400ebd0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.338+0000 7fd6daa00700 1 -- 172.21.15.12:0/3873162703 shutdown_connections 2024-03-29T11:33:35.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.338+0000 7fd6daa00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406aed0 0x7fd6b406d390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.338+0000 7fd6daa00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d415f260 0x7fd6d40d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.338+0000 7fd6daa00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d40b5580 0x7fd6d4159410 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.338+0000 7fd6daa00700 1 --2- 172.21.15.12:0/3873162703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d40d21f0 0x7fd6d40b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.338+0000 7fd6daa00700 1 -- 172.21.15.12:0/3873162703 >> 172.21.15.12:0/3873162703 conn(0x7fd6d408aff0 msgr2=0x7fd6d40d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:35.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.339+0000 7fd6daa00700 1 -- 172.21.15.12:0/3873162703 shutdown_connections 2024-03-29T11:33:35.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.339+0000 7fd6daa00700 1 -- 172.21.15.12:0/3873162703 wait complete. 2024-03-29T11:33:35.339 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T11:33:35.359 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T11:33:35.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.571+0000 7fa50d000700 1 Processor -- start 2024-03-29T11:33:35.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.571+0000 7fa50d000700 1 -- start start 2024-03-29T11:33:35.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa50d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5081611b0 0x7fa50815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:35.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa50d000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa50815f4d0 0x7fa5080ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa50d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa5080d21f0 0x7fa5080cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa50d000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5080cf180 con 0x7fa5080d21f0 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa50d000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5080ce270 con 0x7fa50815f4d0 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa50d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50815b600 con 0x7fa5081611b0 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa506800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa5080d21f0 0x7fa5080cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa506800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa5080d21f0 0x7fa5080cec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46650/0 (socket says 172.21.15.12:46650) 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa506800700 1 -- 172.21.15.12:0/3191837375 learned_addr learned my addr 172.21.15.12:0/3191837375 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa505e00700 1 --2- 172.21.15.12:0/3191837375 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5081611b0 0x7fa50815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa505400700 1 --2- 172.21.15.12:0/3191837375 >> v2:172.21.15.159:3300/0 conn(0x7fa50815f4d0 0x7fa5080ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa506800700 1 -- 172.21.15.12:0/3191837375 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5081611b0 msgr2=0x7fa50815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.573+0000 7fa506800700 1 --2- 172.21.15.12:0/3191837375 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5081611b0 0x7fa50815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.574+0000 7fa506800700 1 -- 172.21.15.12:0/3191837375 >> v2:172.21.15.159:3300/0 conn(0x7fa50815f4d0 msgr2=0x7fa5080ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.574+0000 7fa506800700 1 --2- 172.21.15.12:0/3191837375 >> v2:172.21.15.159:3300/0 conn(0x7fa50815f4d0 0x7fa5080ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.574+0000 7fa506800700 1 -- 172.21.15.12:0/3191837375 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5080d2760 con 0x7fa5080d21f0 2024-03-29T11:33:35.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.574+0000 7fa506800700 1 --2- 172.21.15.12:0/3191837375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5080d21f0 0x7fa5080cec40 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fa50808d540 tx=0x7fa50000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe6f31edf2de17b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:35.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.574+0000 7fa504a00700 1 -- 172.21.15.12:0/3191837375 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50000e070 con 0x7fa5080d21f0 2024-03-29T11:33:35.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.574+0000 7fa504a00700 1 -- 172.21.15.12:0/3191837375 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50000b9c0 con 0x7fa5080d21f0 2024-03-29T11:33:35.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.574+0000 7fa504a00700 1 -- 172.21.15.12:0/3191837375 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50000bd70 con 0x7fa5080d21f0 2024-03-29T11:33:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.575+0000 7fa50d000700 1 -- 172.21.15.12:0/3191837375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5080d21f0 msgr2=0x7fa5080cec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.575+0000 7fa50d000700 1 --2- 172.21.15.12:0/3191837375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5080d21f0 0x7fa5080cec40 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fa50808d540 tx=0x7fa50000b330 comp rx=0 tx=0).stop 2024-03-29T11:33:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.575+0000 7fa50d000700 1 -- 172.21.15.12:0/3191837375 shutdown_connections 2024-03-29T11:33:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.575+0000 7fa50d000700 1 --2- 172.21.15.12:0/3191837375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5080d21f0 0x7fa5080cec40 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.575+0000 7fa50d000700 1 --2- 172.21.15.12:0/3191837375 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5081611b0 0x7fa50815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.575+0000 7fa50d000700 1 --2- 172.21.15.12:0/3191837375 >> v2:172.21.15.159:3300/0 conn(0x7fa50815f4d0 0x7fa5080ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.575+0000 7fa50d000700 1 -- 172.21.15.12:0/3191837375 >> 172.21.15.12:0/3191837375 conn(0x7fa50808aff0 msgr2=0x7fa5080d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.575+0000 7fa50d000700 1 -- 172.21.15.12:0/3191837375 shutdown_connections 2024-03-29T11:33:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.575+0000 7fa50d000700 1 -- 172.21.15.12:0/3191837375 wait complete. 2024-03-29T11:33:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.576+0000 7fa50d000700 1 Processor -- start 2024-03-29T11:33:35.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.577+0000 7fa50d000700 1 -- start start 2024-03-29T11:33:35.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.577+0000 7fa50d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5080d21f0 0x7fa5081f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.577+0000 7fa50d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50815f4d0 0x7fa5080cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.577+0000 7fa50d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5081611b0 0x7fa5080b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa50d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080cd970 con 0x7fa50815f4d0 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa50d000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5081f77b0 con 0x7fa5080d21f0 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa50d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5081f6c30 con 0x7fa5081611b0 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa505400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50815f4d0 0x7fa5080cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa505400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50815f4d0 0x7fa5080cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46658/0 (socket says 172.21.15.12:46658) 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa505400700 1 -- 172.21.15.12:0/1759574130 learned_addr learned my addr 172.21.15.12:0/1759574130 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa506800700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5081611b0 0x7fa5080b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa505e00700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5080d21f0 0x7fa5081f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa505400700 1 -- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5081611b0 msgr2=0x7fa5080b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa505400700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5081611b0 0x7fa5080b78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa505400700 1 -- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5080d21f0 msgr2=0x7fa5081f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa505400700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5080d21f0 0x7fa5081f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa505400700 1 -- 172.21.15.12:0/1759574130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa500009b30 con 0x7fa50815f4d0 2024-03-29T11:33:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.578+0000 7fa505e00700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5080d21f0 0x7fa5081f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.579+0000 7fa505400700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50815f4d0 0x7fa5080cba90 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400e7e0 tx=0x7fa4f400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.579+0000 7fa4fe000700 1 -- 172.21.15.12:0/1759574130 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4013610 con 0x7fa50815f4d0 2024-03-29T11:33:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.579+0000 7fa4fe000700 1 -- 172.21.15.12:0/1759574130 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f4013770 con 0x7fa50815f4d0 2024-03-29T11:33:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.579+0000 7fa4fe000700 1 -- 172.21.15.12:0/1759574130 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4013b00 con 0x7fa50815f4d0 2024-03-29T11:33:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.579+0000 7fa50d000700 1 -- 172.21.15.12:0/1759574130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5081f6f10 con 0x7fa50815f4d0 2024-03-29T11:33:35.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.579+0000 7fa50d000700 1 -- 172.21.15.12:0/1759574130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5081f74d0 con 0x7fa50815f4d0 2024-03-29T11:33:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.580+0000 7fa50d000700 1 -- 172.21.15.12:0/1759574130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa508067720 con 0x7fa50815f4d0 2024-03-29T11:33:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.581+0000 7fa4fe000700 1 -- 172.21.15.12:0/1759574130 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa4f4013c60 con 0x7fa50815f4d0 2024-03-29T11:33:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.582+0000 7fa4fe000700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4e806aed0 0x7fa4e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.582+0000 7fa4fe000700 1 -- 172.21.15.12:0/1759574130 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5299+0+0 (secure 0 0 0) 0x7fa4f4089620 con 0x7fa50815f4d0 2024-03-29T11:33:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.583+0000 7fa505e00700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4e806aed0 0x7fa4e806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:35.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.583+0000 7fa505e00700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4e806aed0 0x7fa4e806d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa5081f0e50 tx=0x7fa4f8008040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:35.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.586+0000 7fa4fe000700 1 -- 172.21.15.12:0/1759574130 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4f4054690 con 0x7fa50815f4d0 2024-03-29T11:33:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:35.802+0000 7fa50d000700 1 -- 172.21.15.12:0/1759574130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa5080b5d00 con 0x7fa50815f4d0 2024-03-29T11:33:36.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.346+0000 7fa4fe000700 1 -- 172.21.15.12:0/1759574130 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v267) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa4f4057cb0 con 0x7fa50815f4d0 2024-03-29T11:33:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.354+0000 7fa50d000700 1 -- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4e806aed0 msgr2=0x7fa4e806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.354+0000 7fa50d000700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4e806aed0 0x7fa4e806d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa5081f0e50 tx=0x7fa4f8008040 comp rx=0 tx=0).stop 2024-03-29T11:33:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.354+0000 7fa50d000700 1 -- 172.21.15.12:0/1759574130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50815f4d0 msgr2=0x7fa5080cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.354+0000 7fa50d000700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50815f4d0 0x7fa5080cba90 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400e7e0 tx=0x7fa4f400eba0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.355+0000 7fa50d000700 1 -- 172.21.15.12:0/1759574130 shutdown_connections 2024-03-29T11:33:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.355+0000 7fa50d000700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4e806aed0 0x7fa4e806d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.355+0000 7fa50d000700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5081611b0 0x7fa5080b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.355+0000 7fa50d000700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5080d21f0 0x7fa5081f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.355+0000 7fa50d000700 1 --2- 172.21.15.12:0/1759574130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa50815f4d0 0x7fa5080cba90 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.355+0000 7fa50d000700 1 -- 172.21.15.12:0/1759574130 >> 172.21.15.12:0/1759574130 conn(0x7fa50808aff0 msgr2=0x7fa50806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.355+0000 7fa50d000700 1 -- 172.21.15.12:0/1759574130 shutdown_connections 2024-03-29T11:33:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.355+0000 7fa50d000700 1 -- 172.21.15.12:0/1759574130 wait complete. 2024-03-29T11:33:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T11:33:36.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T11:33:36.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.585+0000 7f086b600700 1 Processor -- start 2024-03-29T11:33:36.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.585+0000 7f086b600700 1 -- start start 2024-03-29T11:33:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.586+0000 7f086b600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 0x7f08640d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.586+0000 7f086b600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f08640d1b70 0x7f08640d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.586+0000 7f086b600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08640cf640 con 0x7f08640cff10 2024-03-29T11:33:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.586+0000 7f086b600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08640cf220 con 0x7f08640d1b70 2024-03-29T11:33:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.586+0000 7f086b600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08640d2530 con 0x7f086415d710 2024-03-29T11:33:36.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.586+0000 7f0868c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 0x7f08640d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:36.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.586+0000 7f0863e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f08640d1b70 0x7f08640d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:36.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.586+0000 7f0869600700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f08640cff10 0x7f08640d0380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:42654/0 (socket says 172.21.15.12:42654) 2024-03-29T11:33:36.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.587+0000 7f0869600700 1 -- 172.21.15.12:0/2446685422 learned_addr learned my addr 172.21.15.12:0/2446685422 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:36.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.587+0000 7f0863e00700 1 -- 172.21.15.12:0/2446685422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 msgr2=0x7f08640d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:36.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.587+0000 7f0863e00700 1 --2- 172.21.15.12:0/2446685422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 0x7f08640d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.587+0000 7f0863e00700 1 -- 172.21.15.12:0/2446685422 >> v1:172.21.15.159:6789/0 conn(0x7f08640cff10 legacy=0x7f08640d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.587+0000 7f0863e00700 1 -- 172.21.15.12:0/2446685422 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08640ce5a0 con 0x7f08640d1b70 2024-03-29T11:33:36.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.587+0000 7f0868c00700 1 --2- 172.21.15.12:0/2446685422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 0x7f08640d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:36.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.587+0000 7f0863e00700 1 --2- 172.21.15.12:0/2446685422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08640d1b70 0x7f08640d1ff0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f0850009820 tx=0x7f0850009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6da0682b45add61f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:36.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f0863400700 1 -- 172.21.15.12:0/2446685422 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0850005e30 con 0x7f08640d1b70 2024-03-29T11:33:36.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f0863400700 1 -- 172.21.15.12:0/2446685422 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0850006030 con 0x7f08640d1b70 2024-03-29T11:33:36.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f0863400700 1 -- 172.21.15.12:0/2446685422 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500063c0 con 0x7f08640d1b70 2024-03-29T11:33:36.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f086b600700 1 -- 172.21.15.12:0/2446685422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08640d1b70 msgr2=0x7f08640d1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f086b600700 1 --2- 172.21.15.12:0/2446685422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08640d1b70 0x7f08640d1ff0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f0850009820 tx=0x7f0850009b30 comp rx=0 tx=0).stop 2024-03-29T11:33:36.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f086b600700 1 -- 172.21.15.12:0/2446685422 shutdown_connections 2024-03-29T11:33:36.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f086b600700 1 --2- 172.21.15.12:0/2446685422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 0x7f08640d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f086b600700 1 --2- 172.21.15.12:0/2446685422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08640d1b70 0x7f08640d1ff0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f086b600700 1 -- 172.21.15.12:0/2446685422 >> 172.21.15.12:0/2446685422 conn(0x7f086408aff0 msgr2=0x7f08640d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:36.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f086b600700 1 -- 172.21.15.12:0/2446685422 shutdown_connections 2024-03-29T11:33:36.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.588+0000 7f086b600700 1 -- 172.21.15.12:0/2446685422 wait complete. 2024-03-29T11:33:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.590+0000 7f086b600700 1 Processor -- start 2024-03-29T11:33:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.590+0000 7f086b600700 1 -- start start 2024-03-29T11:33:36.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.591+0000 7f086b600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08640cff10 0x7f086414da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:36.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.591+0000 7f086b600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08640d1b70 0x7f08640b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:36.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.591+0000 7f086b600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 0x7f0864149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:36.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.591+0000 7f086b600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0864160380 con 0x7f08640d1b70 2024-03-29T11:33:36.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.591+0000 7f086b600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08640b5da0 con 0x7f08640cff10 2024-03-29T11:33:36.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.591+0000 7f086b600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08640b5910 con 0x7f086415d710 2024-03-29T11:33:36.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.591+0000 7f0863e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08640d1b70 0x7f08640b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:36.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.591+0000 7f0863e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08640d1b70 0x7f08640b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46668/0 (socket says 172.21.15.12:46668) 2024-03-29T11:33:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.591+0000 7f0863e00700 1 -- 172.21.15.12:0/223464740 learned_addr learned my addr 172.21.15.12:0/223464740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.592+0000 7f0868c00700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08640cff10 0x7f086414da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.592+0000 7f0869600700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 0x7f0864149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.592+0000 7f0863e00700 1 -- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 msgr2=0x7f0864149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.592+0000 7f0863e00700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 0x7f0864149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.592+0000 7f0863e00700 1 -- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08640cff10 msgr2=0x7f086414da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.592+0000 7f0863e00700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08640cff10 0x7f086414da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.592+0000 7f0863e00700 1 -- 172.21.15.12:0/223464740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08580094a0 con 0x7f08640d1b70 2024-03-29T11:33:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.592+0000 7f0863e00700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08640d1b70 0x7f08640b2e90 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f0850000c00 tx=0x7f0850004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.593+0000 7f0860c00700 1 -- 172.21.15.12:0/223464740 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500061a0 con 0x7f08640d1b70 2024-03-29T11:33:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.593+0000 7f0860c00700 1 -- 172.21.15.12:0/223464740 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0850020070 con 0x7f08640d1b70 2024-03-29T11:33:36.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.593+0000 7f0860c00700 1 -- 172.21.15.12:0/223464740 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0850006540 con 0x7f08640d1b70 2024-03-29T11:33:36.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.593+0000 7f086b600700 1 -- 172.21.15.12:0/223464740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0850009430 con 0x7f08640d1b70 2024-03-29T11:33:36.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.593+0000 7f086b600700 1 -- 172.21.15.12:0/223464740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08640b4d00 con 0x7f08640d1b70 2024-03-29T11:33:36.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.594+0000 7f086b600700 1 -- 172.21.15.12:0/223464740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08640b5a80 con 0x7f08640d1b70 2024-03-29T11:33:36.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.595+0000 7f0860c00700 1 -- 172.21.15.12:0/223464740 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0850022030 con 0x7f08640d1b70 2024-03-29T11:33:36.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.596+0000 7f0860c00700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f084406aed0 0x7f084406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:36.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.596+0000 7f0860c00700 1 -- 172.21.15.12:0/223464740 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 5681+0+0 (secure 0 0 0) 0x7f085008afe0 con 0x7f08640d1b70 2024-03-29T11:33:36.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.596+0000 7f0868c00700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f084406aed0 0x7f084406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:36.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.597+0000 7f0868c00700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f084406aed0 0x7f084406d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0858009cd0 tx=0x7f0858009210 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.600+0000 7f0860c00700 1 -- 172.21.15.12:0/223464740 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0850055ee0 con 0x7f08640d1b70 2024-03-29T11:33:36.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.861+0000 7f086b600700 1 -- 172.21.15.12:0/223464740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0864067e30 con 0x7f08640d1b70 2024-03-29T11:33:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.867+0000 7f0860c00700 1 -- 172.21.15.12:0/223464740 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 34 and data pool 35 v134) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0850059500 con 0x7f08640d1b70 2024-03-29T11:33:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.874+0000 7f086b600700 1 -- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f084406aed0 msgr2=0x7f084406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.874+0000 7f086b600700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f084406aed0 0x7f084406d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0858009cd0 tx=0x7f0858009210 comp rx=0 tx=0).stop 2024-03-29T11:33:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.874+0000 7f086b600700 1 -- 172.21.15.12:0/223464740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08640d1b70 msgr2=0x7f08640b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.874+0000 7f086b600700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08640d1b70 0x7f08640b2e90 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f0850000c00 tx=0x7f0850004750 comp rx=0 tx=0).stop 2024-03-29T11:33:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.874+0000 7f086b600700 1 -- 172.21.15.12:0/223464740 shutdown_connections 2024-03-29T11:33:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.874+0000 7f086b600700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f084406aed0 0x7f084406d390 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.874+0000 7f086b600700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f086415d710 0x7f0864149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.874+0000 7f086b600700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08640cff10 0x7f086414da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.874+0000 7f086b600700 1 --2- 172.21.15.12:0/223464740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08640d1b70 0x7f08640b2e90 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:36.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.874+0000 7f086b600700 1 -- 172.21.15.12:0/223464740 >> 172.21.15.12:0/223464740 conn(0x7f086408aff0 msgr2=0x7f086415b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:36.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.875+0000 7f086b600700 1 -- 172.21.15.12:0/223464740 shutdown_connections 2024-03-29T11:33:36.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:36.875+0000 7f086b600700 1 -- 172.21.15.12:0/223464740 wait complete. 2024-03-29T11:33:36.875 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 34 and data pool 35 2024-03-29T11:33:36.903 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:37.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.105+0000 7fa218000700 1 Processor -- start 2024-03-29T11:33:37.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.105+0000 7fa218000700 1 -- start start 2024-03-29T11:33:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.106+0000 7fa218000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa21015b5d0 0x7fa2100ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.106+0000 7fa218000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2100cb3f0 0x7fa2100ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.106+0000 7fa218000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2100ae930 0x7fa2100ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.106+0000 7fa218000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21015b9b0 con 0x7fa2100cb3f0 2024-03-29T11:33:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.106+0000 7fa218000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21015fd50 con 0x7fa2100ae930 2024-03-29T11:33:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.106+0000 7fa218000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21015f140 con 0x7fa21015b5d0 2024-03-29T11:33:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.106+0000 7fa214c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2100cb3f0 0x7fa2100ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.107+0000 7fa214c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2100cb3f0 0x7fa2100ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46670/0 (socket says 172.21.15.12:46670) 2024-03-29T11:33:37.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.107+0000 7fa214c00700 1 -- 172.21.15.12:0/1242793454 learned_addr learned my addr 172.21.15.12:0/1242793454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:37.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.107+0000 7fa215600700 1 --2- 172.21.15.12:0/1242793454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa21015b5d0 0x7fa2100ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.107+0000 7fa216000700 1 --2- 172.21.15.12:0/1242793454 >> v2:172.21.15.159:3300/0 conn(0x7fa2100ae930 0x7fa2100ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.107+0000 7fa214c00700 1 -- 172.21.15.12:0/1242793454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa21015b5d0 msgr2=0x7fa2100ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.107+0000 7fa214c00700 1 --2- 172.21.15.12:0/1242793454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa21015b5d0 0x7fa2100ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.107+0000 7fa214c00700 1 -- 172.21.15.12:0/1242793454 >> v2:172.21.15.159:3300/0 conn(0x7fa2100ae930 msgr2=0x7fa2100ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.107+0000 7fa214c00700 1 --2- 172.21.15.12:0/1242793454 >> v2:172.21.15.159:3300/0 conn(0x7fa2100ae930 0x7fa2100ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.107+0000 7fa214c00700 1 -- 172.21.15.12:0/1242793454 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21015f420 con 0x7fa2100cb3f0 2024-03-29T11:33:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.107+0000 7fa214c00700 1 --2- 172.21.15.12:0/1242793454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100cb3f0 0x7fa2100ac0c0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fa2100b0d70 tx=0x7fa1fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ed0eb26d0d4bff server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa20fe00700 1 -- 172.21.15.12:0/1242793454 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc011830 con 0x7fa2100cb3f0 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa20fe00700 1 -- 172.21.15.12:0/1242793454 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc011990 con 0x7fa2100cb3f0 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa20fe00700 1 -- 172.21.15.12:0/1242793454 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc011d20 con 0x7fa2100cb3f0 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa218000700 1 -- 172.21.15.12:0/1242793454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100cb3f0 msgr2=0x7fa2100ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa218000700 1 --2- 172.21.15.12:0/1242793454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100cb3f0 0x7fa2100ac0c0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fa2100b0d70 tx=0x7fa1fc009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa218000700 1 -- 172.21.15.12:0/1242793454 shutdown_connections 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa218000700 1 --2- 172.21.15.12:0/1242793454 >> v2:172.21.15.159:3300/0 conn(0x7fa2100ae930 0x7fa2100ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa218000700 1 --2- 172.21.15.12:0/1242793454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa21015b5d0 0x7fa2100ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa218000700 1 --2- 172.21.15.12:0/1242793454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100cb3f0 0x7fa2100ac0c0 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa218000700 1 -- 172.21.15.12:0/1242793454 >> 172.21.15.12:0/1242793454 conn(0x7fa2100cf190 msgr2=0x7fa2100cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.108+0000 7fa218000700 1 -- 172.21.15.12:0/1242793454 shutdown_connections 2024-03-29T11:33:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.109+0000 7fa218000700 1 -- 172.21.15.12:0/1242793454 wait complete. 2024-03-29T11:33:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.110+0000 7fa218000700 1 Processor -- start 2024-03-29T11:33:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.110+0000 7fa218000700 1 -- start start 2024-03-29T11:33:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.111+0000 7fa218000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100ae930 0x7fa21009caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.111+0000 7fa218000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa21015b5d0 0x7fa2100cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.111+0000 7fa218000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100ccfd0 0x7fa2100cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.111+0000 7fa218000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100b1ed0 con 0x7fa2100ccfd0 2024-03-29T11:33:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.111+0000 7fa218000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa210159250 con 0x7fa2100ae930 2024-03-29T11:33:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.111+0000 7fa218000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa210158640 con 0x7fa21015b5d0 2024-03-29T11:33:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.111+0000 7fa216000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100ccfd0 0x7fa2100cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa216000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100ccfd0 0x7fa2100cd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46682/0 (socket says 172.21.15.12:46682) 2024-03-29T11:33:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa216000700 1 -- 172.21.15.12:0/206101177 learned_addr learned my addr 172.21.15.12:0/206101177 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa214c00700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa21015b5d0 0x7fa2100cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa215600700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100ae930 0x7fa21009caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa216000700 1 -- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa21015b5d0 msgr2=0x7fa2100cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa216000700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa21015b5d0 0x7fa2100cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa216000700 1 -- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100ae930 msgr2=0x7fa21009caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa216000700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100ae930 0x7fa21009caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa216000700 1 -- 172.21.15.12:0/206101177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc009550 con 0x7fa2100ccfd0 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa215600700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100ae930 0x7fa21009caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa216000700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100ccfd0 0x7fa2100cd5c0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fa20800eea0 tx=0x7fa20800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa20d600700 1 -- 172.21.15.12:0/206101177 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208010040 con 0x7fa2100ccfd0 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa20d600700 1 -- 172.21.15.12:0/206101177 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20800bcb0 con 0x7fa2100ccfd0 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa218000700 1 -- 172.21.15.12:0/206101177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa210158920 con 0x7fa2100ccfd0 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.112+0000 7fa20d600700 1 -- 172.21.15.12:0/206101177 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208014640 con 0x7fa2100ccfd0 2024-03-29T11:33:37.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.113+0000 7fa218000700 1 -- 172.21.15.12:0/206101177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa210158ee0 con 0x7fa2100ccfd0 2024-03-29T11:33:37.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.114+0000 7fa218000700 1 -- 172.21.15.12:0/206101177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2101593c0 con 0x7fa2100ccfd0 2024-03-29T11:33:37.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.115+0000 7fa20d600700 1 -- 172.21.15.12:0/206101177 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa2080147a0 con 0x7fa2100ccfd0 2024-03-29T11:33:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.115+0000 7fa20d600700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa1f006aed0 0x7fa1f006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.116+0000 7fa20d600700 1 -- 172.21.15.12:0/206101177 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fa20808a0e0 con 0x7fa2100ccfd0 2024-03-29T11:33:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.116+0000 7fa215600700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa1f006aed0 0x7fa1f006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.117+0000 7fa215600700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa1f006aed0 0x7fa1f006d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa210158290 tx=0x7fa204008040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:37.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.120+0000 7fa20d600700 1 -- 172.21.15.12:0/206101177 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa208054e00 con 0x7fa2100ccfd0 2024-03-29T11:33:37.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.336+0000 7fa218000700 1 -- 172.21.15.12:0/206101177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa210067720 con 0x7fa2100ccfd0 2024-03-29T11:33:37.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.338+0000 7fa20d600700 1 -- 172.21.15.12:0/206101177 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v268) v1 ==== 74+0+16188 (secure 0 0 0) 0x7fa208009ca0 con 0x7fa2100ccfd0 2024-03-29T11:33:37.334 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:37.335 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":268,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:36.862478+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:33:34.767741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-03-29T11:33:35.848901+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 -- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa1f006aed0 msgr2=0x7fa1f006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa1f006aed0 0x7fa1f006d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa210158290 tx=0x7fa204008040 comp rx=0 tx=0).stop 2024-03-29T11:33:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 -- 172.21.15.12:0/206101177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100ccfd0 msgr2=0x7fa2100cd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100ccfd0 0x7fa2100cd5c0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fa20800eea0 tx=0x7fa20800b390 comp rx=0 tx=0).stop 2024-03-29T11:33:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 -- 172.21.15.12:0/206101177 shutdown_connections 2024-03-29T11:33:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa1f006aed0 0x7fa1f006d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2100ccfd0 0x7fa2100cd5c0 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2100ae930 0x7fa21009caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 --2- 172.21.15.12:0/206101177 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa21015b5d0 0x7fa2100cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 -- 172.21.15.12:0/206101177 >> 172.21.15.12:0/206101177 conn(0x7fa2100cf190 msgr2=0x7fa21006d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:37.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 -- 172.21.15.12:0/206101177 shutdown_connections 2024-03-29T11:33:37.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.340+0000 7fa218000700 1 -- 172.21.15.12:0/206101177 wait complete. 2024-03-29T11:33:37.356 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:37.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.559+0000 7fbc0d000700 1 Processor -- start 2024-03-29T11:33:37.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.559+0000 7fbc0d000700 1 -- start start 2024-03-29T11:33:37.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.560+0000 7fbc0d000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc0815b5d0 0x7fbc080ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.560+0000 7fbc0d000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0815fd50 con 0x7fbc080ac7d0 2024-03-29T11:33:37.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.560+0000 7fbc0d000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0815f140 con 0x7fbc080cb3f0 2024-03-29T11:33:37.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.560+0000 7fbc0d000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc0815b9b0 con 0x7fbc0815b5d0 2024-03-29T11:33:37.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.560+0000 7fbc05e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc0815b5d0 0x7fbc080ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.560+0000 7fbc06800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbc080ac7d0 0x7fbc0806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40798/0 (socket says 172.21.15.12:40798) 2024-03-29T11:33:37.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.560+0000 7fbc05e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc0815b5d0 0x7fbc080ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46684/0 (socket says 172.21.15.12:46684) 2024-03-29T11:33:37.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.560+0000 7fbc06800700 1 -- 172.21.15.12:0/1730048235 learned_addr learned my addr 172.21.15.12:0/1730048235 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.561+0000 7fbc05e00700 1 -- 172.21.15.12:0/1730048235 >> v1:172.21.15.159:6789/0 conn(0x7fbc080cb3f0 legacy=0x7fbc080ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.561+0000 7fbc05e00700 1 -- 172.21.15.12:0/1730048235 >> v1:172.21.15.12:6789/0 conn(0x7fbc080ac7d0 legacy=0x7fbc0806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.561+0000 7fbc05e00700 1 -- 172.21.15.12:0/1730048235 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0815f420 con 0x7fbc0815b5d0 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.561+0000 7fbc05e00700 1 --2- 172.21.15.12:0/1730048235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc0815b5d0 0x7fbc080ae3f0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009840 tx=0x7fbbf8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c15f53f5adace81 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.561+0000 7fbc04a00700 1 -- 172.21.15.12:0/1730048235 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8011830 con 0x7fbc0815b5d0 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.561+0000 7fbc04a00700 1 -- 172.21.15.12:0/1730048235 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf8011990 con 0x7fbc0815b5d0 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.561+0000 7fbc04a00700 1 -- 172.21.15.12:0/1730048235 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8011d20 con 0x7fbc0815b5d0 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.562+0000 7fbc0d000700 1 -- 172.21.15.12:0/1730048235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc0815b5d0 msgr2=0x7fbc080ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.562+0000 7fbc0d000700 1 --2- 172.21.15.12:0/1730048235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc0815b5d0 0x7fbc080ae3f0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009840 tx=0x7fbbf8009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.562+0000 7fbc0d000700 1 -- 172.21.15.12:0/1730048235 shutdown_connections 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.562+0000 7fbc0d000700 1 --2- 172.21.15.12:0/1730048235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc0815b5d0 0x7fbc080ae3f0 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.562+0000 7fbc0d000700 1 -- 172.21.15.12:0/1730048235 >> 172.21.15.12:0/1730048235 conn(0x7fbc080cf190 msgr2=0x7fbc080cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:37.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.562+0000 7fbc0d000700 1 -- 172.21.15.12:0/1730048235 shutdown_connections 2024-03-29T11:33:37.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.562+0000 7fbc0d000700 1 -- 172.21.15.12:0/1730048235 wait complete. 2024-03-29T11:33:37.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.564+0000 7fbc0d000700 1 Processor -- start 2024-03-29T11:33:37.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.564+0000 7fbc0d000700 1 -- start start 2024-03-29T11:33:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc0d000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc080ac7d0 0x7fbc080c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc0d000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc0815b5d0 0x7fbc080c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc0d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc080c89a0 0x7fbc080ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc0d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080709c0 con 0x7fbc080c89a0 2024-03-29T11:33:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc0d000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080c8f00 con 0x7fbc080ac7d0 2024-03-29T11:33:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc0d000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080c91e0 con 0x7fbc0815b5d0 2024-03-29T11:33:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc06800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc080c89a0 0x7fbc080ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc05e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc080ac7d0 0x7fbc080c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc06800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc080c89a0 0x7fbc080ca0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46694/0 (socket says 172.21.15.12:46694) 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc06800700 1 -- 172.21.15.12:0/3095917765 learned_addr learned my addr 172.21.15.12:0/3095917765 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.565+0000 7fbc05400700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc0815b5d0 0x7fbc080c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbc06800700 1 -- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc0815b5d0 msgr2=0x7fbc080c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbc06800700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc0815b5d0 0x7fbc080c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbc06800700 1 -- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc080ac7d0 msgr2=0x7fbc080c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbc06800700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc080ac7d0 0x7fbc080c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbc06800700 1 -- 172.21.15.12:0/3095917765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf80094a0 con 0x7fbc080c89a0 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbc06800700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc080c89a0 0x7fbc080ca0c0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000cbc0 tx=0x7fbc0000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbbfe000700 1 -- 172.21.15.12:0/3095917765 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0000a9e0 con 0x7fbc080c89a0 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbbfe000700 1 -- 172.21.15.12:0/3095917765 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0000ab40 con 0x7fbc080c89a0 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbc0d000700 1 -- 172.21.15.12:0/3095917765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc080c94c0 con 0x7fbc080c89a0 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbbfe000700 1 -- 172.21.15.12:0/3095917765 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc00012400 con 0x7fbc080c89a0 2024-03-29T11:33:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.566+0000 7fbc0d000700 1 -- 172.21.15.12:0/3095917765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc08072890 con 0x7fbc080c89a0 2024-03-29T11:33:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.567+0000 7fbc0d000700 1 -- 172.21.15.12:0/3095917765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc080cba10 con 0x7fbc080c89a0 2024-03-29T11:33:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.568+0000 7fbbfe000700 1 -- 172.21.15.12:0/3095917765 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fbc0001b3b0 con 0x7fbc080c89a0 2024-03-29T11:33:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.569+0000 7fbbfe000700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbbe806aed0 0x7fbbe806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.569+0000 7fbbfe000700 1 -- 172.21.15.12:0/3095917765 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fbc00089200 con 0x7fbc080c89a0 2024-03-29T11:33:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.570+0000 7fbc05e00700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbbe806aed0 0x7fbbe806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:37.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.571+0000 7fbc05e00700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbbe806aed0 0x7fbbe806d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbbf80090b0 tx=0x7fbbf801c040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.574+0000 7fbbfe000700 1 -- 172.21.15.12:0/3095917765 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc00054170 con 0x7fbc080c89a0 2024-03-29T11:33:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.790+0000 7fbc0d000700 1 -- 172.21.15.12:0/3095917765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc08067720 con 0x7fbc080c89a0 2024-03-29T11:33:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.792+0000 7fbbfe000700 1 -- 172.21.15.12:0/3095917765 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v268) v1 ==== 74+0+16188 (secure 0 0 0) 0x7fbc00057790 con 0x7fbc080c89a0 2024-03-29T11:33:37.788 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:37.789 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":268,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:36.862478+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:33:34.767741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-03-29T11:33:35.848901+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.794+0000 7fbc0d000700 1 -- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbbe806aed0 msgr2=0x7fbbe806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.794+0000 7fbc0d000700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbbe806aed0 0x7fbbe806d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbbf80090b0 tx=0x7fbbf801c040 comp rx=0 tx=0).stop 2024-03-29T11:33:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.794+0000 7fbc0d000700 1 -- 172.21.15.12:0/3095917765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc080c89a0 msgr2=0x7fbc080ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.794+0000 7fbc0d000700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc080c89a0 0x7fbc080ca0c0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000cbc0 tx=0x7fbc0000ced0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.794+0000 7fbc0d000700 1 -- 172.21.15.12:0/3095917765 shutdown_connections 2024-03-29T11:33:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.794+0000 7fbc0d000700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbbe806aed0 0x7fbbe806d390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.795+0000 7fbc0d000700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc080c89a0 0x7fbc080ca0c0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.795+0000 7fbc0d000700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc080ac7d0 0x7fbc080c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.795+0000 7fbc0d000700 1 --2- 172.21.15.12:0/3095917765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc0815b5d0 0x7fbc080c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.795+0000 7fbc0d000700 1 -- 172.21.15.12:0/3095917765 >> 172.21.15.12:0/3095917765 conn(0x7fbc080cf190 msgr2=0x7fbc0815e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.795+0000 7fbc0d000700 1 -- 172.21.15.12:0/3095917765 shutdown_connections 2024-03-29T11:33:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:37.795+0000 7fbc0d000700 1 -- 172.21.15.12:0/3095917765 wait complete. 2024-03-29T11:33:37.809 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T11:33:38.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.015+0000 7f817e000700 1 Processor -- start 2024-03-29T11:33:38.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.015+0000 7f817e000700 1 -- start start 2024-03-29T11:33:38.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.016+0000 7f817e000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f817815d9d0 0x7f81780d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:38.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.016+0000 7f817e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81780d16a0 0x7f81780d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:38.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.016+0000 7f817e000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f81780d2370 0x7f817815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:38.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.016+0000 7f817e000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81780d2820 con 0x7f817815d9d0 2024-03-29T11:33:38.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.016+0000 7f817e000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8178161420 con 0x7f81780d2370 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.016+0000 7f817e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8178161700 con 0x7f81780d16a0 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8177400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f817815d9d0 0x7f81780d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8177400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f817815d9d0 0x7f81780d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46708/0 (socket says 172.21.15.12:46708) 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8177400700 1 -- 172.21.15.12:0/3357222740 learned_addr learned my addr 172.21.15.12:0/3357222740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8176a00700 1 --2- 172.21.15.12:0/3357222740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81780d16a0 0x7f81780d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8177e00700 1 --2- 172.21.15.12:0/3357222740 >> v2:172.21.15.159:3300/0 conn(0x7f81780d2370 0x7f817815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8177400700 1 -- 172.21.15.12:0/3357222740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81780d16a0 msgr2=0x7f81780d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8177400700 1 --2- 172.21.15.12:0/3357222740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81780d16a0 0x7f81780d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8177400700 1 -- 172.21.15.12:0/3357222740 >> v2:172.21.15.159:3300/0 conn(0x7f81780d2370 msgr2=0x7f817815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8177400700 1 --2- 172.21.15.12:0/3357222740 >> v2:172.21.15.159:3300/0 conn(0x7f81780d2370 0x7f817815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8177400700 1 -- 172.21.15.12:0/3357222740 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81781619e0 con 0x7f817815d9d0 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8176a00700 1 --2- 172.21.15.12:0/3357222740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81780d16a0 0x7f81780d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.017+0000 7f8177e00700 1 --2- 172.21.15.12:0/3357222740 >> v2:172.21.15.159:3300/0 conn(0x7f81780d2370 0x7f817815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:38.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f8177400700 1 --2- 172.21.15.12:0/3357222740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f817815d9d0 0x7f81780d3c30 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f81780cdc00 tx=0x7f8168009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5889cc741b9dd82 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:38.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f8176000700 1 -- 172.21.15.12:0/3357222740 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816800b870 con 0x7f817815d9d0 2024-03-29T11:33:38.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f8176000700 1 -- 172.21.15.12:0/3357222740 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816800b9d0 con 0x7f817815d9d0 2024-03-29T11:33:38.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f8176000700 1 -- 172.21.15.12:0/3357222740 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816800bd60 con 0x7f817815d9d0 2024-03-29T11:33:38.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f817e000700 1 -- 172.21.15.12:0/3357222740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f817815d9d0 msgr2=0x7f81780d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:38.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f817e000700 1 --2- 172.21.15.12:0/3357222740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f817815d9d0 0x7f81780d3c30 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f81780cdc00 tx=0x7f8168009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f817e000700 1 -- 172.21.15.12:0/3357222740 shutdown_connections 2024-03-29T11:33:38.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f817e000700 1 --2- 172.21.15.12:0/3357222740 >> v2:172.21.15.159:3300/0 conn(0x7f81780d2370 0x7f817815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f817e000700 1 --2- 172.21.15.12:0/3357222740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f817815d9d0 0x7f81780d3c30 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f817e000700 1 --2- 172.21.15.12:0/3357222740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81780d16a0 0x7f81780d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.018+0000 7f817e000700 1 -- 172.21.15.12:0/3357222740 >> 172.21.15.12:0/3357222740 conn(0x7f817808aff0 msgr2=0x7f81780d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:38.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.019+0000 7f817e000700 1 -- 172.21.15.12:0/3357222740 shutdown_connections 2024-03-29T11:33:38.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.019+0000 7f817e000700 1 -- 172.21.15.12:0/3357222740 wait complete. 2024-03-29T11:33:38.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.020+0000 7f817e000700 1 Processor -- start 2024-03-29T11:33:38.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.021+0000 7f817e000700 1 -- start start 2024-03-29T11:33:38.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.021+0000 7f817e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81780d16a0 0x7f81781e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:38.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.021+0000 7f817e000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81780d2370 0x7f8178155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:38.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.021+0000 7f817e000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f817815d9d0 0x7f81781e8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f817e000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81781e78d0 con 0x7f81780d16a0 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f817e000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81781e82a0 con 0x7f81780d2370 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f817e000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81781e97b0 con 0x7f817815d9d0 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8177400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81780d16a0 0x7f81781e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8177400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81780d16a0 0x7f81781e7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46712/0 (socket says 172.21.15.12:46712) 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8177400700 1 -- 172.21.15.12:0/1871266367 learned_addr learned my addr 172.21.15.12:0/1871266367 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8177e00700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f817815d9d0 0x7f81781e8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8176a00700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81780d2370 0x7f8178155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8177400700 1 -- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f817815d9d0 msgr2=0x7f81781e8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8177400700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f817815d9d0 0x7f81781e8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8177400700 1 -- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81780d2370 msgr2=0x7f8178155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8177400700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81780d2370 0x7f8178155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8177400700 1 -- 172.21.15.12:0/1871266367 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8168009550 con 0x7f81780d16a0 2024-03-29T11:33:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8176a00700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81780d2370 0x7f8178155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.022+0000 7f8177400700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81780d16a0 0x7f81781e7d60 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f8168014040 tx=0x7f8168004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.023+0000 7f816f400700 1 -- 172.21.15.12:0/1871266367 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168004520 con 0x7f81780d16a0 2024-03-29T11:33:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.023+0000 7f816f400700 1 -- 172.21.15.12:0/1871266367 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816801e070 con 0x7f81780d16a0 2024-03-29T11:33:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.023+0000 7f816f400700 1 -- 172.21.15.12:0/1871266367 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81680038e0 con 0x7f81780d16a0 2024-03-29T11:33:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.023+0000 7f817e000700 1 -- 172.21.15.12:0/1871266367 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81781e9a90 con 0x7f81780d16a0 2024-03-29T11:33:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.023+0000 7f817e000700 1 -- 172.21.15.12:0/1871266367 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81781e7560 con 0x7f81780d16a0 2024-03-29T11:33:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.024+0000 7f817e000700 1 -- 172.21.15.12:0/1871266367 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81780d1ca0 con 0x7f81780d16a0 2024-03-29T11:33:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.025+0000 7f816f400700 1 -- 172.21.15.12:0/1871266367 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8168003a40 con 0x7f81780d16a0 2024-03-29T11:33:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.026+0000 7f816f400700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f815806aed0 0x7f815806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.026+0000 7f816f400700 1 -- 172.21.15.12:0/1871266367 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f8168020070 con 0x7f81780d16a0 2024-03-29T11:33:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.026+0000 7f8176a00700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f815806aed0 0x7f815806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:38.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.027+0000 7f8176a00700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f815806aed0 0x7f815806d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81781e86e0 tx=0x7f8164008040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.030+0000 7f816f400700 1 -- 172.21.15.12:0/1871266367 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8168055c20 con 0x7f81780d16a0 2024-03-29T11:33:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.291+0000 7f817e000700 1 -- 172.21.15.12:0/1871266367 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f81780cfb10 con 0x7f81780d16a0 2024-03-29T11:33:38.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.873+0000 7f816f400700 1 -- 172.21.15.12:0/1871266367 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v137) v1 ==== 118+0+0 (secure 0 0 0) 0x7f816802a020 con 0x7f81780d16a0 2024-03-29T11:33:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.876+0000 7f817e000700 1 -- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f815806aed0 msgr2=0x7f815806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.876+0000 7f817e000700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f815806aed0 0x7f815806d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81781e86e0 tx=0x7f8164008040 comp rx=0 tx=0).stop 2024-03-29T11:33:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.876+0000 7f817e000700 1 -- 172.21.15.12:0/1871266367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81780d16a0 msgr2=0x7f81781e7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.876+0000 7f817e000700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81780d16a0 0x7f81781e7d60 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f8168014040 tx=0x7f8168004750 comp rx=0 tx=0).stop 2024-03-29T11:33:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.876+0000 7f817e000700 1 -- 172.21.15.12:0/1871266367 shutdown_connections 2024-03-29T11:33:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.876+0000 7f817e000700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f815806aed0 0x7f815806d390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.876+0000 7f817e000700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f817815d9d0 0x7f81781e8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.876+0000 7f817e000700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81780d16a0 0x7f81781e7d60 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.876+0000 7f817e000700 1 --2- 172.21.15.12:0/1871266367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81780d2370 0x7f8178155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:38.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.876+0000 7f817e000700 1 -- 172.21.15.12:0/1871266367 >> 172.21.15.12:0/1871266367 conn(0x7f817808aff0 msgr2=0x7f81780b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:38.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.877+0000 7f817e000700 1 -- 172.21.15.12:0/1871266367 shutdown_connections 2024-03-29T11:33:38.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:38.877+0000 7f817e000700 1 -- 172.21.15.12:0/1871266367 wait complete. 2024-03-29T11:33:38.894 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T11:33:38.894 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T11:33:39.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.102+0000 7f4aa0e00700 1 Processor -- start 2024-03-29T11:33:39.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.102+0000 7f4aa0e00700 1 -- start start 2024-03-29T11:33:39.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.103+0000 7f4aa0e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c15d9d0 0x7f4a9c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:39.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.103+0000 7f4aa0e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4a9c0d16a0 0x7f4a9c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:39.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.103+0000 7f4aa0e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9c161420 con 0x7f4a9c0d2370 2024-03-29T11:33:39.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.103+0000 7f4aa0e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a9c0d2820 con 0x7f4a9c0d16a0 2024-03-29T11:33:39.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.103+0000 7f4aa0e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c161700 con 0x7f4a9c15d9d0 2024-03-29T11:33:39.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.103+0000 7f4a99400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4a9c0d16a0 0x7f4a9c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:39.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.103+0000 7f4a9a800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4a9c0d2370 0x7f4a9c15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49660/0 (socket says 172.21.15.12:49660) 2024-03-29T11:33:39.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.103+0000 7f4a9a800700 1 -- 172.21.15.12:0/338504632 learned_addr learned my addr 172.21.15.12:0/338504632 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:39.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.104+0000 7f4a99e00700 1 --2- 172.21.15.12:0/338504632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c15d9d0 0x7f4a9c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:39.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.104+0000 7f4a99400700 1 -- 172.21.15.12:0/338504632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c15d9d0 msgr2=0x7f4a9c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:39.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.104+0000 7f4a99400700 1 --2- 172.21.15.12:0/338504632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c15d9d0 0x7f4a9c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:39.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.104+0000 7f4a99400700 1 -- 172.21.15.12:0/338504632 >> v1:172.21.15.12:6789/0 conn(0x7f4a9c0d2370 legacy=0x7f4a9c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:39.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.104+0000 7f4a99400700 1 -- 172.21.15.12:0/338504632 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c1619e0 con 0x7f4a9c0d16a0 2024-03-29T11:33:39.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.104+0000 7f4a99400700 1 --2- 172.21.15.12:0/338504632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d16a0 0x7f4a9c0d1e30 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009820 tx=0x7f4a88009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f528e1a8c95a90f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:39.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.104+0000 7f4a98a00700 1 -- 172.21.15.12:0/338504632 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88005e30 con 0x7f4a9c0d16a0 2024-03-29T11:33:39.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.104+0000 7f4a98a00700 1 -- 172.21.15.12:0/338504632 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88006030 con 0x7f4a9c0d16a0 2024-03-29T11:33:39.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.104+0000 7f4a98a00700 1 -- 172.21.15.12:0/338504632 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a880063e0 con 0x7f4a9c0d16a0 2024-03-29T11:33:39.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.105+0000 7f4aa0e00700 1 -- 172.21.15.12:0/338504632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d16a0 msgr2=0x7f4a9c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:39.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.105+0000 7f4aa0e00700 1 --2- 172.21.15.12:0/338504632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d16a0 0x7f4a9c0d1e30 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009820 tx=0x7f4a88009b30 comp rx=0 tx=0).stop 2024-03-29T11:33:39.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.105+0000 7f4aa0e00700 1 -- 172.21.15.12:0/338504632 shutdown_connections 2024-03-29T11:33:39.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.105+0000 7f4aa0e00700 1 --2- 172.21.15.12:0/338504632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c15d9d0 0x7f4a9c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:39.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.105+0000 7f4aa0e00700 1 --2- 172.21.15.12:0/338504632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d16a0 0x7f4a9c0d1e30 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:39.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.105+0000 7f4aa0e00700 1 -- 172.21.15.12:0/338504632 >> 172.21.15.12:0/338504632 conn(0x7f4a9c08aff0 msgr2=0x7f4a9c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:39.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.105+0000 7f4aa0e00700 1 -- 172.21.15.12:0/338504632 shutdown_connections 2024-03-29T11:33:39.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.105+0000 7f4aa0e00700 1 -- 172.21.15.12:0/338504632 wait complete. 2024-03-29T11:33:39.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.107+0000 7f4aa0e00700 1 Processor -- start 2024-03-29T11:33:39.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.107+0000 7f4aa0e00700 1 -- start start 2024-03-29T11:33:39.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4aa0e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0d16a0 0x7f4a9c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:39.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4aa0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c0c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:39.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4aa0e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c15d9d0 0x7f4a9c0c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:39.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4aa0e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c15df50 con 0x7f4a9c0d2370 2024-03-29T11:33:39.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4aa0e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0c96f0 con 0x7f4a9c15d9d0 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4aa0e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c1586c0 con 0x7f4a9c0d16a0 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4a99400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c0c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4a99400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c0c8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56432/0 (socket says 172.21.15.12:56432) 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4a99400700 1 -- 172.21.15.12:0/3755615066 learned_addr learned my addr 172.21.15.12:0/3755615066 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4a99e00700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0d16a0 0x7f4a9c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.108+0000 7f4a9a800700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c15d9d0 0x7f4a9c0c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4a99400700 1 -- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0d16a0 msgr2=0x7f4a9c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4a99400700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0d16a0 0x7f4a9c0ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4a99400700 1 -- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c15d9d0 msgr2=0x7f4a9c0c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4a99400700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c15d9d0 0x7f4a9c0c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4a99400700 1 -- 172.21.15.12:0/3755615066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a88009430 con 0x7f4a9c0d2370 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4a9a800700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c15d9d0 0x7f4a9c0c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:39.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4a99400700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c0c8c70 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009b70 tx=0x7f4a88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:39.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4a92000700 1 -- 172.21.15.12:0/3755615066 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88006890 con 0x7f4a9c0d2370 2024-03-29T11:33:39.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4a92000700 1 -- 172.21.15.12:0/3755615066 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88020070 con 0x7f4a9c0d2370 2024-03-29T11:33:39.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4a92000700 1 -- 172.21.15.12:0/3755615066 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88004260 con 0x7f4a9c0d2370 2024-03-29T11:33:39.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4aa0e00700 1 -- 172.21.15.12:0/3755615066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c1589a0 con 0x7f4a9c0d2370 2024-03-29T11:33:39.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.109+0000 7f4aa0e00700 1 -- 172.21.15.12:0/3755615066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c158f60 con 0x7f4a9c0d2370 2024-03-29T11:33:39.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.110+0000 7f4aa0e00700 1 -- 172.21.15.12:0/3755615066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c0d1ca0 con 0x7f4a9c0d2370 2024-03-29T11:33:39.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.111+0000 7f4a92000700 1 -- 172.21.15.12:0/3755615066 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4a88022030 con 0x7f4a9c0d2370 2024-03-29T11:33:39.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.112+0000 7f4a92000700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4a7c06aed0 0x7f4a7c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:39.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.112+0000 7f4a92000700 1 -- 172.21.15.12:0/3755615066 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f4a8808ae00 con 0x7f4a9c0d2370 2024-03-29T11:33:39.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.113+0000 7f4a99e00700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4a7c06aed0 0x7f4a7c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:39.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.114+0000 7f4a99e00700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4a7c06aed0 0x7f4a7c06d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006b20 tx=0x7f4a8c008040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:39.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.117+0000 7f4a92000700 1 -- 172.21.15.12:0/3755615066 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4a88055c30 con 0x7f4a9c0d2370 2024-03-29T11:33:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:39.382+0000 7f4aa0e00700 1 -- 172.21.15.12:0/3755615066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4a9c15e090 con 0x7f4a9c0d2370 2024-03-29T11:33:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.355+0000 7f4a92000700 1 -- 172.21.15.12:0/3755615066 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v138) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4a9c15e090 con 0x7f4a9c0d2370 2024-03-29T11:33:40.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 -- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4a7c06aed0 msgr2=0x7f4a7c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4a7c06aed0 0x7f4a7c06d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006b20 tx=0x7f4a8c008040 comp rx=0 tx=0).stop 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 -- 172.21.15.12:0/3755615066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d2370 msgr2=0x7f4a9c0c8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c0c8c70 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009b70 tx=0x7f4a88004750 comp rx=0 tx=0).stop 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 -- 172.21.15.12:0/3755615066 shutdown_connections 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4a7c06aed0 0x7f4a7c06d390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a9c15d9d0 0x7f4a9c0c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a9c0d16a0 0x7f4a9c0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 --2- 172.21.15.12:0/3755615066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c0c8c70 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 -- 172.21.15.12:0/3755615066 >> 172.21.15.12:0/3755615066 conn(0x7f4a9c08aff0 msgr2=0x7f4a9c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 -- 172.21.15.12:0/3755615066 shutdown_connections 2024-03-29T11:33:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.359+0000 7f4aa0e00700 1 -- 172.21.15.12:0/3755615066 wait complete. 2024-03-29T11:33:40.377 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T11:33:40.377 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:33:40.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.586+0000 7f3549800700 1 Processor -- start 2024-03-29T11:33:40.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.586+0000 7f3549800700 1 -- start start 2024-03-29T11:33:40.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.587+0000 7f3549800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f35440cf3c0 0x7f35440d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.587+0000 7f3549800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f354415db40 0x7f35440cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.587+0000 7f3549800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35440d1330 con 0x7f354415d760 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.587+0000 7f3549800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35440d1fb0 con 0x7f354415db40 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.587+0000 7f3549800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35440d1610 con 0x7f35440cf3c0 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.588+0000 7f3543400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f354415db40 0x7f35440cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.588+0000 7f3543400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f354415db40 0x7f35440cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56434/0 (socket says 172.21.15.12:56434) 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.588+0000 7f3543400700 1 -- 172.21.15.12:0/2598960295 learned_addr learned my addr 172.21.15.12:0/2598960295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.588+0000 7f3542000700 1 --2- 172.21.15.12:0/2598960295 >> v2:172.21.15.159:3300/0 conn(0x7f35440cf3c0 0x7f35440d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.588+0000 7f3543400700 1 -- 172.21.15.12:0/2598960295 >> v2:172.21.15.159:3300/0 conn(0x7f35440cf3c0 msgr2=0x7f35440d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.588+0000 7f3543400700 1 --2- 172.21.15.12:0/2598960295 >> v2:172.21.15.159:3300/0 conn(0x7f35440cf3c0 0x7f35440d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.588+0000 7f3543400700 1 -- 172.21.15.12:0/2598960295 >> v1:172.21.15.159:6789/0 conn(0x7f354415d760 legacy=0x7f35440cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.588+0000 7f3543400700 1 -- 172.21.15.12:0/2598960295 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35440d18f0 con 0x7f354415db40 2024-03-29T11:33:40.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.588+0000 7f3543400700 1 --2- 172.21.15.12:0/2598960295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f354415db40 0x7f35440cffd0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f354408b8f0 tx=0x7f353c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5c7ad5a598799c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3541600700 1 -- 172.21.15.12:0/2598960295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c00e070 con 0x7f354415db40 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3541600700 1 -- 172.21.15.12:0/2598960295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c00b910 con 0x7f354415db40 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3541600700 1 -- 172.21.15.12:0/2598960295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c00bcc0 con 0x7f354415db40 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3549800700 1 -- 172.21.15.12:0/2598960295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f354415db40 msgr2=0x7f35440cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3549800700 1 --2- 172.21.15.12:0/2598960295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f354415db40 0x7f35440cffd0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f354408b8f0 tx=0x7f353c00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3549800700 1 -- 172.21.15.12:0/2598960295 shutdown_connections 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3549800700 1 --2- 172.21.15.12:0/2598960295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f354415db40 0x7f35440cffd0 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3549800700 1 --2- 172.21.15.12:0/2598960295 >> v2:172.21.15.159:3300/0 conn(0x7f35440cf3c0 0x7f35440d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3549800700 1 -- 172.21.15.12:0/2598960295 >> 172.21.15.12:0/2598960295 conn(0x7f354408aff0 msgr2=0x7f35440d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3549800700 1 -- 172.21.15.12:0/2598960295 shutdown_connections 2024-03-29T11:33:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.589+0000 7f3549800700 1 -- 172.21.15.12:0/2598960295 wait complete. 2024-03-29T11:33:40.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.591+0000 7f3549800700 1 Processor -- start 2024-03-29T11:33:40.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.591+0000 7f3549800700 1 -- start start 2024-03-29T11:33:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.592+0000 7f3549800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35440cf3c0 0x7f3544154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.592+0000 7f3549800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f354415d760 0x7f35440b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.592+0000 7f3549800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f354415db40 0x7f354415a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.592+0000 7f3549800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35440b68a0 con 0x7f35440cf3c0 2024-03-29T11:33:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.592+0000 7f3549800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3544155360 con 0x7f354415d760 2024-03-29T11:33:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.592+0000 7f3549800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35440b6200 con 0x7f354415db40 2024-03-29T11:33:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.592+0000 7f3542a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35440cf3c0 0x7f3544154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.592+0000 7f3542a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35440cf3c0 0x7f3544154e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56438/0 (socket says 172.21.15.12:56438) 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.592+0000 7f3542a00700 1 -- 172.21.15.12:0/1115661539 learned_addr learned my addr 172.21.15.12:0/1115661539 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f3542000700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f354415d760 0x7f35440b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f3543400700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f354415db40 0x7f354415a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f3542a00700 1 -- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f354415db40 msgr2=0x7f354415a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f3542a00700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f354415db40 0x7f354415a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f3542a00700 1 -- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f354415d760 msgr2=0x7f35440b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f3542a00700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f354415d760 0x7f35440b85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f3542a00700 1 -- 172.21.15.12:0/1115661539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c009b30 con 0x7f35440cf3c0 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f3542a00700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35440cf3c0 0x7f3544154e20 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f353400c4e0 tx=0x7f353400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f353aa00700 1 -- 172.21.15.12:0/1115661539 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35340052d0 con 0x7f35440cf3c0 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f353aa00700 1 -- 172.21.15.12:0/1115661539 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3534005430 con 0x7f35440cf3c0 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f353aa00700 1 -- 172.21.15.12:0/1115661539 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35340038c0 con 0x7f35440cf3c0 2024-03-29T11:33:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.593+0000 7f3549800700 1 -- 172.21.15.12:0/1115661539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35440b8e80 con 0x7f35440cf3c0 2024-03-29T11:33:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.594+0000 7f3549800700 1 -- 172.21.15.12:0/1115661539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35440b64c0 con 0x7f35440cf3c0 2024-03-29T11:33:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.595+0000 7f3549800700 1 -- 172.21.15.12:0/1115661539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3544067720 con 0x7f35440cf3c0 2024-03-29T11:33:40.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.596+0000 7f353aa00700 1 -- 172.21.15.12:0/1115661539 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f3534003a20 con 0x7f35440cf3c0 2024-03-29T11:33:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.596+0000 7f353aa00700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f352406af50 0x7f352406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.597+0000 7f353aa00700 1 -- 172.21.15.12:0/1115661539 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f3534088ab0 con 0x7f35440cf3c0 2024-03-29T11:33:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.597+0000 7f3542000700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f352406af50 0x7f352406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:40.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.598+0000 7f3542000700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f352406af50 0x7f352406d410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f354415bf80 tx=0x7f3530008040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:40.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.601+0000 7f353aa00700 1 -- 172.21.15.12:0/1115661539 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3534053b20 con 0x7f35440cf3c0 2024-03-29T11:33:40.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.858+0000 7f3549800700 1 -- 172.21.15.12:0/1115661539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35440b28f0 con 0x7f35440cf3c0 2024-03-29T11:33:40.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.858+0000 7f353aa00700 1 -- 172.21.15.12:0/1115661539 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f3534057140 con 0x7f35440cf3c0 2024-03-29T11:33:40.855 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:40.855 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":138,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1546500295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1546500295},{"type":"v1","addr":"172.21.15.159:6836","nonce":1546500295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1627782307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1627782307},{"type":"v1","addr":"172.21.15.12:6834","nonce":1627782307}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3327995506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3327995506},{"type":"v1","addr":"172.21.15.12:6835","nonce":3327995506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:33:36.862806+0000","modified":"2024-03-29T11:33:40.346151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12061":{"gid":12061,"name":"d","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/2746367434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2746367434},{"type":"v1","addr":"172.21.15.159:6837","nonce":2746367434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-29T11:33:40.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 -- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f352406af50 msgr2=0x7f352406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f352406af50 0x7f352406d410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f354415bf80 tx=0x7f3530008040 comp rx=0 tx=0).stop 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 -- 172.21.15.12:0/1115661539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35440cf3c0 msgr2=0x7f3544154e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35440cf3c0 0x7f3544154e20 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f353400c4e0 tx=0x7f353400c8a0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 -- 172.21.15.12:0/1115661539 shutdown_connections 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f352406af50 0x7f352406d410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f354415db40 0x7f354415a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35440cf3c0 0x7f3544154e20 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 --2- 172.21.15.12:0/1115661539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f354415d760 0x7f35440b85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 -- 172.21.15.12:0/1115661539 >> 172.21.15.12:0/1115661539 conn(0x7f354408aff0 msgr2=0x7f354415c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 -- 172.21.15.12:0/1115661539 shutdown_connections 2024-03-29T11:33:40.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:40.861+0000 7f3549800700 1 -- 172.21.15.12:0/1115661539 wait complete. 2024-03-29T11:33:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 138 2024-03-29T11:33:40.876 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:41.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.078+0000 7f5097800700 1 Processor -- start 2024-03-29T11:33:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.078+0000 7f5097800700 1 -- start start 2024-03-29T11:33:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.079+0000 7f5097800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f50900ceec0 0x7f50901612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.079+0000 7f5097800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 0x7f50900b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.079+0000 7f5097800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50900d30d0 con 0x7f509015d640 2024-03-29T11:33:41.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.079+0000 7f5097800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5090161c90 con 0x7f50900ceec0 2024-03-29T11:33:41.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.079+0000 7f5097800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50900b6840 con 0x7f5090161800 2024-03-29T11:33:41.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f5095800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 0x7f50900b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f508fe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f50900ceec0 0x7f50901612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f508fe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f50900ceec0 0x7f50901612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:60686/0 (socket says 172.21.15.12:60686) 2024-03-29T11:33:41.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f5094e00700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f509015d640 0x7f50900d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:56194/0 (socket says 172.21.15.12:56194) 2024-03-29T11:33:41.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f508fe00700 1 -- 172.21.15.12:0/2067078647 learned_addr learned my addr 172.21.15.12:0/2067078647 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f5095800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 0x7f50900b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:55282/0 (socket says 172.21.15.12:55282) 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f508f400700 1 -- 172.21.15.12:0/2067078647 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143112305 0 0) 0x7f50900d30d0 con 0x7f509015d640 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f5095800700 1 -- 172.21.15.12:0/2067078647 >> v2:172.21.15.159:3300/0 conn(0x7f50900ceec0 msgr2=0x7f50901612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f5095800700 1 --2- 172.21.15.12:0/2067078647 >> v2:172.21.15.159:3300/0 conn(0x7f50900ceec0 0x7f50901612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f5095800700 1 -- 172.21.15.12:0/2067078647 >> v1:172.21.15.159:6789/0 conn(0x7f509015d640 legacy=0x7f50900d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f5095800700 1 -- 172.21.15.12:0/2067078647 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50900d2b30 con 0x7f5090161800 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.080+0000 7f508fe00700 1 --2- 172.21.15.12:0/2067078647 >> v2:172.21.15.159:3300/0 conn(0x7f50900ceec0 0x7f50901612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.081+0000 7f5095800700 1 --2- 172.21.15.12:0/2067078647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 0x7f50900b6cd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5088009d90 tx=0x7f508800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2574eb5028369ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.081+0000 7f508f400700 1 -- 172.21.15.12:0/2067078647 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508800e070 con 0x7f5090161800 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.081+0000 7f508f400700 1 -- 172.21.15.12:0/2067078647 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508800bb20 con 0x7f5090161800 2024-03-29T11:33:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.081+0000 7f508f400700 1 -- 172.21.15.12:0/2067078647 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088012430 con 0x7f5090161800 2024-03-29T11:33:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.081+0000 7f5097800700 1 -- 172.21.15.12:0/2067078647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 msgr2=0x7f50900b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.081+0000 7f5097800700 1 --2- 172.21.15.12:0/2067078647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 0x7f50900b6cd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5088009d90 tx=0x7f508800b330 comp rx=0 tx=0).stop 2024-03-29T11:33:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.081+0000 7f5097800700 1 -- 172.21.15.12:0/2067078647 shutdown_connections 2024-03-29T11:33:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.082+0000 7f5097800700 1 --2- 172.21.15.12:0/2067078647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 0x7f50900b6cd0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.082+0000 7f5097800700 1 --2- 172.21.15.12:0/2067078647 >> v2:172.21.15.159:3300/0 conn(0x7f50900ceec0 0x7f50901612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.082+0000 7f5097800700 1 -- 172.21.15.12:0/2067078647 >> 172.21.15.12:0/2067078647 conn(0x7f509008aff0 msgr2=0x7f50900d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.082+0000 7f5097800700 1 -- 172.21.15.12:0/2067078647 shutdown_connections 2024-03-29T11:33:41.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.082+0000 7f5097800700 1 -- 172.21.15.12:0/2067078647 wait complete. 2024-03-29T11:33:41.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.083+0000 7f5097800700 1 Processor -- start 2024-03-29T11:33:41.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.083+0000 7f5097800700 1 -- start start 2024-03-29T11:33:41.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.084+0000 7f5097800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50900ceec0 0x7f5090158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.084+0000 7f5097800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f509015d640 0x7f509009fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.084+0000 7f5097800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 0x7f509015eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.084+0000 7f5097800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50900c98a0 con 0x7f509015d640 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.084+0000 7f5097800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50900c7120 con 0x7f50900ceec0 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.084+0000 7f5097800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50900c7400 con 0x7f5090161800 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f508fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f509015d640 0x7f509009fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f508fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f509015d640 0x7f509009fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56444/0 (socket says 172.21.15.12:56444) 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f508fe00700 1 -- 172.21.15.12:0/349670646 learned_addr learned my addr 172.21.15.12:0/349670646 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f5095800700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 0x7f509015eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f5094e00700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50900ceec0 0x7f5090158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f508fe00700 1 -- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 msgr2=0x7f509015eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f508fe00700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 0x7f509015eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f508fe00700 1 -- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50900ceec0 msgr2=0x7f5090158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f508fe00700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50900ceec0 0x7f5090158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f508fe00700 1 -- 172.21.15.12:0/349670646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c009550 con 0x7f509015d640 2024-03-29T11:33:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.085+0000 7f508fe00700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f509015d640 0x7f509009fc30 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f507c00e9b0 tx=0x7f507c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.086+0000 7f508cc00700 1 -- 172.21.15.12:0/349670646 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c00ee50 con 0x7f509015d640 2024-03-29T11:33:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.086+0000 7f508cc00700 1 -- 172.21.15.12:0/349670646 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507c00c850 con 0x7f509015d640 2024-03-29T11:33:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.086+0000 7f508cc00700 1 -- 172.21.15.12:0/349670646 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c00cbc0 con 0x7f509015d640 2024-03-29T11:33:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.086+0000 7f5097800700 1 -- 172.21.15.12:0/349670646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5088009a70 con 0x7f509015d640 2024-03-29T11:33:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.086+0000 7f5097800700 1 -- 172.21.15.12:0/349670646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50900c9b80 con 0x7f509015d640 2024-03-29T11:33:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.087+0000 7f5097800700 1 -- 172.21.15.12:0/349670646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5090067720 con 0x7f509015d640 2024-03-29T11:33:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.088+0000 7f508cc00700 1 -- 172.21.15.12:0/349670646 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f507c00cd20 con 0x7f509015d640 2024-03-29T11:33:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.089+0000 7f508cc00700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f507006aed0 0x7f507006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.089+0000 7f508cc00700 1 -- 172.21.15.12:0/349670646 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f507c089ae0 con 0x7f509015d640 2024-03-29T11:33:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.089+0000 7f5094e00700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f507006aed0 0x7f507006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.090+0000 7f5094e00700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f507006aed0 0x7f507006d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5084007fd0 tx=0x7f5084007de0 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:41.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.093+0000 7f508cc00700 1 -- 172.21.15.12:0/349670646 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f507c054910 con 0x7f509015d640 2024-03-29T11:33:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.307+0000 7f5097800700 1 -- 172.21.15.12:0/349670646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50900b5c70 con 0x7f509015d640 2024-03-29T11:33:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.309+0000 7f508cc00700 1 -- 172.21.15.12:0/349670646 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v269) v1 ==== 74+0+16175 (secure 0 0 0) 0x7f507c017090 con 0x7f509015d640 2024-03-29T11:33:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:41.306 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":269,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:37.865281+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:33:34.767741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-03-29T11:33:35.848901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.311+0000 7f5097800700 1 -- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f507006aed0 msgr2=0x7f507006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.311+0000 7f5097800700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f507006aed0 0x7f507006d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5084007fd0 tx=0x7f5084007de0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.311+0000 7f5097800700 1 -- 172.21.15.12:0/349670646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f509015d640 msgr2=0x7f509009fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.311+0000 7f5097800700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f509015d640 0x7f509009fc30 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f507c00e9b0 tx=0x7f507c00ecc0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.311+0000 7f5097800700 1 -- 172.21.15.12:0/349670646 shutdown_connections 2024-03-29T11:33:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.311+0000 7f5097800700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f507006aed0 0x7f507006d390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.311+0000 7f5097800700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5090161800 0x7f509015eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.311+0000 7f5097800700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50900ceec0 0x7f5090158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.311+0000 7f5097800700 1 --2- 172.21.15.12:0/349670646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f509015d640 0x7f509009fc30 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.311+0000 7f5097800700 1 -- 172.21.15.12:0/349670646 >> 172.21.15.12:0/349670646 conn(0x7f509008aff0 msgr2=0x7f50900b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.312+0000 7f5097800700 1 -- 172.21.15.12:0/349670646 shutdown_connections 2024-03-29T11:33:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.312+0000 7f5097800700 1 -- 172.21.15.12:0/349670646 wait complete. 2024-03-29T11:33:41.326 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T11:33:41.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:33:41.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.530+0000 7f686c000700 1 Processor -- start 2024-03-29T11:33:41.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.530+0000 7f686c000700 1 -- start start 2024-03-29T11:33:41.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.531+0000 7f686c000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f68641619d0 0x7f68641685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.531+0000 7f686c000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68640d30d0 con 0x7f68640ceec0 2024-03-29T11:33:41.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f686c000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68640b6840 con 0x7f686415d640 2024-03-29T11:33:41.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f686c000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68640d2b30 con 0x7f68641619d0 2024-03-29T11:33:41.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f6868c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f68640ceec0 0x7f68641612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49676/0 (socket says 172.21.15.12:49676) 2024-03-29T11:33:41.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f6868c00700 1 -- 172.21.15.12:0/1848270553 learned_addr learned my addr 172.21.15.12:0/1848270553 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:41.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f686a000700 1 --2- 172.21.15.12:0/1848270553 >> v2:172.21.15.159:3300/0 conn(0x7f68641619d0 0x7f68641685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253804377 0 0) 0x7f68640d30d0 con 0x7f68640ceec0 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684c0035b0 con 0x7f68640ceec0 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432126447 0 0) 0x7f68640b6840 con 0x7f686415d640 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68640d30d0 con 0x7f686415d640 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2368052214 0 0) 0x7f684c0035b0 con 0x7f68640ceec0 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.532+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 >> v2:172.21.15.159:3300/0 conn(0x7f68641619d0 msgr2=0x7f68641685f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 --2- 172.21.15.12:0/1848270553 >> v2:172.21.15.159:3300/0 conn(0x7f68641619d0 0x7f68641685f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 >> v1:172.21.15.159:6789/0 conn(0x7f686415d640 legacy=0x7f68640d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68640d2df0 con 0x7f68640ceec0 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f6850001670 con 0x7f68640ceec0 2024-03-29T11:33:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 >> v1:172.21.15.12:6789/0 conn(0x7f68640ceec0 legacy=0x7f68641612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f684c003b90 0x7f684c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f684c004850 0x7f684c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f684c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68640d2b30 con 0x7f684c005210 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c00c140 con 0x7f684c003b90 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c00c300 con 0x7f684c004850 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f686a000700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f684c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6868c00700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f684c004850 0x7f684c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f6869600700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f684c003b90 0x7f684c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f686a000700 1 -- 172.21.15.12:0/1848270553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f684c004850 msgr2=0x7f684c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f686a000700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f684c004850 0x7f684c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f686a000700 1 -- 172.21.15.12:0/1848270553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f684c003b90 msgr2=0x7f684c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f686a000700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f684c003b90 0x7f684c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.533+0000 7f686a000700 1 -- 172.21.15.12:0/1848270553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c0035b0 con 0x7f684c005210 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.534+0000 7f686a000700 1 -- 172.21.15.12:0/1848270553 reap_dead start 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.534+0000 7f686a000700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f684c00ba10 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f685c009fc0 tx=0x7f684c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee340ae07c9fccde server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.534+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c003a20 con 0x7f684c005210 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.534+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c00f040 con 0x7f684c005210 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.534+0000 7f6863e00700 1 -- 172.21.15.12:0/1848270553 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c003da0 con 0x7f684c005210 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.534+0000 7f686c000700 1 -- 172.21.15.12:0/1848270553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 msgr2=0x7f684c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.534+0000 7f686c000700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f684c00ba10 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f685c009fc0 tx=0x7f684c005880 comp rx=0 tx=0).stop 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.535+0000 7f686c000700 1 -- 172.21.15.12:0/1848270553 shutdown_connections 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.535+0000 7f686c000700 1 --2- 172.21.15.12:0/1848270553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f684c00ba10 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.535+0000 7f686c000700 1 -- 172.21.15.12:0/1848270553 >> 172.21.15.12:0/1848270553 conn(0x7f686408aff0 msgr2=0x7f68640d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.535+0000 7f686c000700 1 -- 172.21.15.12:0/1848270553 shutdown_connections 2024-03-29T11:33:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.535+0000 7f686c000700 1 -- 172.21.15.12:0/1848270553 wait complete. 2024-03-29T11:33:41.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.537+0000 7f686c000700 1 Processor -- start 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.537+0000 7f686c000700 1 -- start start 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.538+0000 7f686c000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f686409d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.538+0000 7f686c000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68640a0e30 0x7f68640c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.538+0000 7f686c000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f686414f510 0x7f686414f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f686c000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686409e670 con 0x7f684c005210 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f686c000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686414fed0 con 0x7f686414f510 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f686c000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686409daf0 con 0x7f68640a0e30 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f6869600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f686409d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f6869600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f686409d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56450/0 (socket says 172.21.15.12:56450) 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f6869600700 1 -- 172.21.15.12:0/267754128 learned_addr learned my addr 172.21.15.12:0/267754128 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f6869600700 1 -- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68640a0e30 msgr2=0x7f68640c9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f6869600700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68640a0e30 0x7f68640c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f6869600700 1 -- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f686414f510 msgr2=0x7f686414f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f6869600700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f686414f510 0x7f686414f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f6869600700 1 -- 172.21.15.12:0/267754128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686409ddd0 con 0x7f684c005210 2024-03-29T11:33:41.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.539+0000 7f6869600700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f686409d230 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f685800f770 tx=0x7f685800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:41.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.540+0000 7f6861600700 1 -- 172.21.15.12:0/267754128 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6858005320 con 0x7f684c005210 2024-03-29T11:33:41.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.540+0000 7f6861600700 1 -- 172.21.15.12:0/267754128 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68580054a0 con 0x7f684c005210 2024-03-29T11:33:41.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.540+0000 7f6861600700 1 -- 172.21.15.12:0/267754128 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6858006aa0 con 0x7f684c005210 2024-03-29T11:33:41.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.540+0000 7f686c000700 1 -- 172.21.15.12:0/267754128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686409e0b0 con 0x7f684c005210 2024-03-29T11:33:41.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.540+0000 7f686c000700 1 -- 172.21.15.12:0/267754128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f686409d770 con 0x7f684c005210 2024-03-29T11:33:41.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.541+0000 7f686c000700 1 -- 172.21.15.12:0/267754128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6864067720 con 0x7f684c005210 2024-03-29T11:33:41.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.542+0000 7f6861600700 1 -- 172.21.15.12:0/267754128 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6858006c00 con 0x7f684c005210 2024-03-29T11:33:41.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.543+0000 7f6861600700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f684406af00 0x7f684406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:41.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.543+0000 7f6861600700 1 -- 172.21.15.12:0/267754128 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f6858088c90 con 0x7f684c005210 2024-03-29T11:33:41.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.543+0000 7f6868c00700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f684406af00 0x7f684406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:41.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.544+0000 7f6868c00700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f684406af00 0x7f684406d3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6850001c40 tx=0x7f6850001af0 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.547+0000 7f6861600700 1 -- 172.21.15.12:0/267754128 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6858053ac0 con 0x7f684c005210 2024-03-29T11:33:41.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.758+0000 7f686c000700 1 -- 172.21.15.12:0/267754128 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68640a0900 con 0x7f684406af00 2024-03-29T11:33:41.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.770+0000 7f6861600700 1 -- 172.21.15.12:0/267754128 <== mgr.11755 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289361 (secure 0 0 0) 0x7f68640a0900 con 0x7f684406af00 2024-03-29T11:33:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.774+0000 7f686c000700 1 -- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f684406af00 msgr2=0x7f684406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.774+0000 7f686c000700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f684406af00 0x7f684406d3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6850001c40 tx=0x7f6850001af0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.774+0000 7f686c000700 1 -- 172.21.15.12:0/267754128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 msgr2=0x7f686409d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.774+0000 7f686c000700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f686409d230 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f685800f770 tx=0x7f685800fb30 comp rx=0 tx=0).stop 2024-03-29T11:33:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.774+0000 7f686c000700 1 -- 172.21.15.12:0/267754128 shutdown_connections 2024-03-29T11:33:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.774+0000 7f686c000700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f684406af00 0x7f684406d3c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.774+0000 7f686c000700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f686414f510 0x7f686414f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.774+0000 7f686c000700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684c005210 0x7f686409d230 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.774+0000 7f686c000700 1 --2- 172.21.15.12:0/267754128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68640a0e30 0x7f68640c9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.774+0000 7f686c000700 1 -- 172.21.15.12:0/267754128 >> 172.21.15.12:0/267754128 conn(0x7f686408aff0 msgr2=0x7f686415c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.775+0000 7f686c000700 1 -- 172.21.15.12:0/267754128 shutdown_connections 2024-03-29T11:33:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:41.775+0000 7f686c000700 1 -- 172.21.15.12:0/267754128 wait complete. 2024-03-29T11:33:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:33:41.785 INFO:tasks.ceph.mds.d.smithi159.stderr:2024-03-29T11:33:41.789+0000 7f4d26400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:33:41.790 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-03-29T11:33:40.087839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2672,"num_read_kb":5632,"num_write":735,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":448,"ondisk_log_size":448,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":455368,"kb_used_data":27652,"kb_used_omap":12,"kb_used_meta":427571,"kb_avail":754519352,"statfs":{"total":773094113280,"available":772627816448,"internally_reserved":0,"allocated":28315648,"data_stored":14522137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":437833273},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.752063"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899776+0000","last_change":"2024-03-29T11:33:36.893003+0000","last_active":"2024-03-29T11:33:36.899776+0000","last_peered":"2024-03-29T11:33:36.899776+0000","last_clean":"2024-03-29T11:33:36.899776+0000","last_became_active":"2024-03-29T11:33:36.892417+0000","last_became_peered":"2024-03-29T11:33:36.892417+0000","last_unstale":"2024-03-29T11:33:36.899776+0000","last_undegraded":"2024-03-29T11:33:36.899776+0000","last_fullsized":"2024-03-29T11:33:36.899776+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:32:46.474810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.900205+0000","last_change":"2024-03-29T11:33:36.898436+0000","last_active":"2024-03-29T11:33:36.900205+0000","last_peered":"2024-03-29T11:33:36.900205+0000","last_clean":"2024-03-29T11:33:36.900205+0000","last_became_active":"2024-03-29T11:33:36.890680+0000","last_became_peered":"2024-03-29T11:33:36.890680+0000","last_unstale":"2024-03-29T11:33:36.900205+0000","last_undegraded":"2024-03-29T11:33:36.900205+0000","last_fullsized":"2024-03-29T11:33:36.900205+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:44:07.664618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.881207+0000","last_change":"2024-03-29T11:33:36.879984+0000","last_active":"2024-03-29T11:33:36.881207+0000","last_peered":"2024-03-29T11:33:36.881207+0000","last_clean":"2024-03-29T11:33:36.881207+0000","last_became_active":"2024-03-29T11:33:36.879646+0000","last_became_peered":"2024-03-29T11:33:36.879646+0000","last_unstale":"2024-03-29T11:33:36.881207+0000","last_undegraded":"2024-03-29T11:33:36.881207+0000","last_fullsized":"2024-03-29T11:33:36.881207+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:01:52.706981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.902858+0000","last_change":"2024-03-29T11:33:36.895719+0000","last_active":"2024-03-29T11:33:36.902858+0000","last_peered":"2024-03-29T11:33:36.902858+0000","last_clean":"2024-03-29T11:33:36.902858+0000","last_became_active":"2024-03-29T11:33:36.895441+0000","last_became_peered":"2024-03-29T11:33:36.895441+0000","last_unstale":"2024-03-29T11:33:36.902858+0000","last_undegraded":"2024-03-29T11:33:36.902858+0000","last_fullsized":"2024-03-29T11:33:36.902858+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:32.153462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899513+0000","last_change":"2024-03-29T11:33:36.896380+0000","last_active":"2024-03-29T11:33:36.899513+0000","last_peered":"2024-03-29T11:33:36.899513+0000","last_clean":"2024-03-29T11:33:36.899513+0000","last_became_active":"2024-03-29T11:33:36.890321+0000","last_became_peered":"2024-03-29T11:33:36.890321+0000","last_unstale":"2024-03-29T11:33:36.899513+0000","last_undegraded":"2024-03-29T11:33:36.899513+0000","last_fullsized":"2024-03-29T11:33:36.899513+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:34.257790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.894232+0000","last_change":"2024-03-29T11:33:36.889542+0000","last_active":"2024-03-29T11:33:36.894232+0000","last_peered":"2024-03-29T11:33:36.894232+0000","last_clean":"2024-03-29T11:33:36.894232+0000","last_became_active":"2024-03-29T11:33:36.889207+0000","last_became_peered":"2024-03-29T11:33:36.889207+0000","last_unstale":"2024-03-29T11:33:36.894232+0000","last_undegraded":"2024-03-29T11:33:36.894232+0000","last_fullsized":"2024-03-29T11:33:36.894232+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:48:58.214029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874299+0000","last_change":"2024-03-29T11:33:36.895133+0000","last_active":"2024-03-29T11:33:37.874299+0000","last_peered":"2024-03-29T11:33:37.874299+0000","last_clean":"2024-03-29T11:33:37.874299+0000","last_became_active":"2024-03-29T11:33:36.894552+0000","last_became_peered":"2024-03-29T11:33:36.894552+0000","last_unstale":"2024-03-29T11:33:37.874299+0000","last_undegraded":"2024-03-29T11:33:37.874299+0000","last_fullsized":"2024-03-29T11:33:37.874299+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:29.041184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874140+0000","last_change":"2024-03-29T11:33:36.881152+0000","last_active":"2024-03-29T11:33:37.874140+0000","last_peered":"2024-03-29T11:33:37.874140+0000","last_clean":"2024-03-29T11:33:37.874140+0000","last_became_active":"2024-03-29T11:33:36.880885+0000","last_became_peered":"2024-03-29T11:33:36.880885+0000","last_unstale":"2024-03-29T11:33:37.874140+0000","last_undegraded":"2024-03-29T11:33:37.874140+0000","last_fullsized":"2024-03-29T11:33:37.874140+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:26.859189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899545+0000","last_change":"2024-03-29T11:33:36.898304+0000","last_active":"2024-03-29T11:33:36.899545+0000","last_peered":"2024-03-29T11:33:36.899545+0000","last_clean":"2024-03-29T11:33:36.899545+0000","last_became_active":"2024-03-29T11:33:36.897344+0000","last_became_peered":"2024-03-29T11:33:36.897344+0000","last_unstale":"2024-03-29T11:33:36.899545+0000","last_undegraded":"2024-03-29T11:33:36.899545+0000","last_fullsized":"2024-03-29T11:33:36.899545+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:08:12.505762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.895702+0000","last_change":"2024-03-29T11:33:36.893590+0000","last_active":"2024-03-29T11:33:36.895702+0000","last_peered":"2024-03-29T11:33:36.895702+0000","last_clean":"2024-03-29T11:33:36.895702+0000","last_became_active":"2024-03-29T11:33:36.893341+0000","last_became_peered":"2024-03-29T11:33:36.893341+0000","last_unstale":"2024-03-29T11:33:36.895702+0000","last_undegraded":"2024-03-29T11:33:36.895702+0000","last_fullsized":"2024-03-29T11:33:36.895702+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:15.073274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.894162+0000","last_change":"2024-03-29T11:33:36.886094+0000","last_active":"2024-03-29T11:33:36.894162+0000","last_peered":"2024-03-29T11:33:36.894162+0000","last_clean":"2024-03-29T11:33:36.894162+0000","last_became_active":"2024-03-29T11:33:36.885781+0000","last_became_peered":"2024-03-29T11:33:36.885781+0000","last_unstale":"2024-03-29T11:33:36.894162+0000","last_undegraded":"2024-03-29T11:33:36.894162+0000","last_fullsized":"2024-03-29T11:33:36.894162+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:30:28.396619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.878893+0000","last_change":"2024-03-29T11:33:36.877933+0000","last_active":"2024-03-29T11:33:36.878893+0000","last_peered":"2024-03-29T11:33:36.878893+0000","last_clean":"2024-03-29T11:33:36.878893+0000","last_became_active":"2024-03-29T11:33:36.877218+0000","last_became_peered":"2024-03-29T11:33:36.877218+0000","last_unstale":"2024-03-29T11:33:36.878893+0000","last_undegraded":"2024-03-29T11:33:36.878893+0000","last_fullsized":"2024-03-29T11:33:36.878893+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:26.859189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.878730+0000","last_change":"2024-03-29T11:33:36.877409+0000","last_active":"2024-03-29T11:33:36.878730+0000","last_peered":"2024-03-29T11:33:36.878730+0000","last_clean":"2024-03-29T11:33:36.878730+0000","last_became_active":"2024-03-29T11:33:36.876887+0000","last_became_peered":"2024-03-29T11:33:36.876887+0000","last_unstale":"2024-03-29T11:33:36.878730+0000","last_undegraded":"2024-03-29T11:33:36.878730+0000","last_fullsized":"2024-03-29T11:33:36.878730+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:51:00.296286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.903646+0000","last_change":"2024-03-29T11:33:36.897953+0000","last_active":"2024-03-29T11:33:36.903646+0000","last_peered":"2024-03-29T11:33:36.903646+0000","last_clean":"2024-03-29T11:33:36.903646+0000","last_became_active":"2024-03-29T11:33:36.897680+0000","last_became_peered":"2024-03-29T11:33:36.897680+0000","last_unstale":"2024-03-29T11:33:36.903646+0000","last_undegraded":"2024-03-29T11:33:36.903646+0000","last_fullsized":"2024-03-29T11:33:36.903646+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:48.820065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874406+0000","last_change":"2024-03-29T11:33:36.891334+0000","last_active":"2024-03-29T11:33:37.874406+0000","last_peered":"2024-03-29T11:33:37.874406+0000","last_clean":"2024-03-29T11:33:37.874406+0000","last_became_active":"2024-03-29T11:33:36.890976+0000","last_became_peered":"2024-03-29T11:33:36.890976+0000","last_unstale":"2024-03-29T11:33:37.874406+0000","last_undegraded":"2024-03-29T11:33:37.874406+0000","last_fullsized":"2024-03-29T11:33:37.874406+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:42:26.424892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874278+0000","last_change":"2024-03-29T11:33:36.885822+0000","last_active":"2024-03-29T11:33:37.874278+0000","last_peered":"2024-03-29T11:33:37.874278+0000","last_clean":"2024-03-29T11:33:37.874278+0000","last_became_active":"2024-03-29T11:33:36.885546+0000","last_became_peered":"2024-03-29T11:33:36.885546+0000","last_unstale":"2024-03-29T11:33:37.874278+0000","last_undegraded":"2024-03-29T11:33:37.874278+0000","last_fullsized":"2024-03-29T11:33:37.874278+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:38:26.882481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.886973+0000","last_change":"2024-03-29T11:33:36.884866+0000","last_active":"2024-03-29T11:33:36.886973+0000","last_peered":"2024-03-29T11:33:36.886973+0000","last_clean":"2024-03-29T11:33:36.886973+0000","last_became_active":"2024-03-29T11:33:36.884516+0000","last_became_peered":"2024-03-29T11:33:36.884516+0000","last_unstale":"2024-03-29T11:33:36.886973+0000","last_undegraded":"2024-03-29T11:33:36.886973+0000","last_fullsized":"2024-03-29T11:33:36.886973+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:30:28.396619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874568+0000","last_change":"2024-03-29T11:33:36.890080+0000","last_active":"2024-03-29T11:33:37.874568+0000","last_peered":"2024-03-29T11:33:37.874568+0000","last_clean":"2024-03-29T11:33:37.874568+0000","last_became_active":"2024-03-29T11:33:36.889714+0000","last_became_peered":"2024-03-29T11:33:36.889714+0000","last_unstale":"2024-03-29T11:33:37.874568+0000","last_undegraded":"2024-03-29T11:33:37.874568+0000","last_fullsized":"2024-03-29T11:33:37.874568+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.148775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875290+0000","last_change":"2024-03-29T11:33:36.898908+0000","last_active":"2024-03-29T11:33:37.875290+0000","last_peered":"2024-03-29T11:33:37.875290+0000","last_clean":"2024-03-29T11:33:37.875290+0000","last_became_active":"2024-03-29T11:33:36.897597+0000","last_became_peered":"2024-03-29T11:33:36.897597+0000","last_unstale":"2024-03-29T11:33:37.875290+0000","last_undegraded":"2024-03-29T11:33:37.875290+0000","last_fullsized":"2024-03-29T11:33:37.875290+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:41:36.893627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875111+0000","last_change":"2024-03-29T11:33:36.888681+0000","last_active":"2024-03-29T11:33:37.875111+0000","last_peered":"2024-03-29T11:33:37.875111+0000","last_clean":"2024-03-29T11:33:37.875111+0000","last_became_active":"2024-03-29T11:33:36.888383+0000","last_became_peered":"2024-03-29T11:33:36.888383+0000","last_unstale":"2024-03-29T11:33:37.875111+0000","last_undegraded":"2024-03-29T11:33:37.875111+0000","last_fullsized":"2024-03-29T11:33:37.875111+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:12.300239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.876668+0000","last_change":"2024-03-29T11:33:36.875193+0000","last_active":"2024-03-29T11:33:36.876668+0000","last_peered":"2024-03-29T11:33:36.876668+0000","last_clean":"2024-03-29T11:33:36.876668+0000","last_became_active":"2024-03-29T11:33:36.874932+0000","last_became_peered":"2024-03-29T11:33:36.874932+0000","last_unstale":"2024-03-29T11:33:36.876668+0000","last_undegraded":"2024-03-29T11:33:36.876668+0000","last_fullsized":"2024-03-29T11:33:36.876668+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:54.747726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.887091+0000","last_change":"2024-03-29T11:33:36.886531+0000","last_active":"2024-03-29T11:33:36.887091+0000","last_peered":"2024-03-29T11:33:36.887091+0000","last_clean":"2024-03-29T11:33:36.887091+0000","last_became_active":"2024-03-29T11:33:36.885653+0000","last_became_peered":"2024-03-29T11:33:36.885653+0000","last_unstale":"2024-03-29T11:33:36.887091+0000","last_undegraded":"2024-03-29T11:33:36.887091+0000","last_fullsized":"2024-03-29T11:33:36.887091+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:10.981880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872834+0000","last_change":"2024-03-29T11:33:36.887478+0000","last_active":"2024-03-29T11:33:37.872834+0000","last_peered":"2024-03-29T11:33:37.872834+0000","last_clean":"2024-03-29T11:33:37.872834+0000","last_became_active":"2024-03-29T11:33:36.886802+0000","last_became_peered":"2024-03-29T11:33:36.886802+0000","last_unstale":"2024-03-29T11:33:37.872834+0000","last_undegraded":"2024-03-29T11:33:37.872834+0000","last_fullsized":"2024-03-29T11:33:37.872834+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:41.212920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.882768+0000","last_change":"2024-03-29T11:33:36.881358+0000","last_active":"2024-03-29T11:33:36.882768+0000","last_peered":"2024-03-29T11:33:36.882768+0000","last_clean":"2024-03-29T11:33:36.882768+0000","last_became_active":"2024-03-29T11:33:36.881103+0000","last_became_peered":"2024-03-29T11:33:36.881103+0000","last_unstale":"2024-03-29T11:33:36.882768+0000","last_undegraded":"2024-03-29T11:33:36.882768+0000","last_fullsized":"2024-03-29T11:33:36.882768+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:16:12.727333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872862+0000","last_change":"2024-03-29T11:33:36.889158+0000","last_active":"2024-03-29T11:33:37.872862+0000","last_peered":"2024-03-29T11:33:37.872862+0000","last_clean":"2024-03-29T11:33:37.872862+0000","last_became_active":"2024-03-29T11:33:36.888904+0000","last_became_peered":"2024-03-29T11:33:36.888904+0000","last_unstale":"2024-03-29T11:33:37.872862+0000","last_undegraded":"2024-03-29T11:33:37.872862+0000","last_fullsized":"2024-03-29T11:33:37.872862+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:06.351839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.878680+0000","last_change":"2024-03-29T11:33:36.877422+0000","last_active":"2024-03-29T11:33:36.878680+0000","last_peered":"2024-03-29T11:33:36.878680+0000","last_clean":"2024-03-29T11:33:36.878680+0000","last_became_active":"2024-03-29T11:33:36.877136+0000","last_became_peered":"2024-03-29T11:33:36.877136+0000","last_unstale":"2024-03-29T11:33:36.878680+0000","last_undegraded":"2024-03-29T11:33:36.878680+0000","last_fullsized":"2024-03-29T11:33:36.878680+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:08:12.505762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875315+0000","last_change":"2024-03-29T11:33:36.886686+0000","last_active":"2024-03-29T11:33:37.875315+0000","last_peered":"2024-03-29T11:33:37.875315+0000","last_clean":"2024-03-29T11:33:37.875315+0000","last_became_active":"2024-03-29T11:33:36.886424+0000","last_became_peered":"2024-03-29T11:33:36.886424+0000","last_unstale":"2024-03-29T11:33:37.875315+0000","last_undegraded":"2024-03-29T11:33:37.875315+0000","last_fullsized":"2024-03-29T11:33:37.875315+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:42.295243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.902879+0000","last_change":"2024-03-29T11:33:36.897242+0000","last_active":"2024-03-29T11:33:36.902879+0000","last_peered":"2024-03-29T11:33:36.902879+0000","last_clean":"2024-03-29T11:33:36.902879+0000","last_became_active":"2024-03-29T11:33:36.896949+0000","last_became_peered":"2024-03-29T11:33:36.896949+0000","last_unstale":"2024-03-29T11:33:36.902879+0000","last_undegraded":"2024-03-29T11:33:36.902879+0000","last_fullsized":"2024-03-29T11:33:36.902879+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:06.950427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875002+0000","last_change":"2024-03-29T11:33:36.894512+0000","last_active":"2024-03-29T11:33:37.875002+0000","last_peered":"2024-03-29T11:33:37.875002+0000","last_clean":"2024-03-29T11:33:37.875002+0000","last_became_active":"2024-03-29T11:33:36.894200+0000","last_became_peered":"2024-03-29T11:33:36.894200+0000","last_unstale":"2024-03-29T11:33:37.875002+0000","last_undegraded":"2024-03-29T11:33:37.875002+0000","last_fullsized":"2024-03-29T11:33:37.875002+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:45.926420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874508+0000","last_change":"2024-03-29T11:33:36.895304+0000","last_active":"2024-03-29T11:33:37.874508+0000","last_peered":"2024-03-29T11:33:37.874508+0000","last_clean":"2024-03-29T11:33:37.874508+0000","last_became_active":"2024-03-29T11:33:36.894890+0000","last_became_peered":"2024-03-29T11:33:36.894890+0000","last_unstale":"2024-03-29T11:33:37.874508+0000","last_undegraded":"2024-03-29T11:33:37.874508+0000","last_fullsized":"2024-03-29T11:33:37.874508+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:01:54.782259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.895081+0000","last_change":"2024-03-29T11:33:36.892935+0000","last_active":"2024-03-29T11:33:36.895081+0000","last_peered":"2024-03-29T11:33:36.895081+0000","last_clean":"2024-03-29T11:33:36.895081+0000","last_became_active":"2024-03-29T11:33:36.892353+0000","last_became_peered":"2024-03-29T11:33:36.892353+0000","last_unstale":"2024-03-29T11:33:36.895081+0000","last_undegraded":"2024-03-29T11:33:36.895081+0000","last_fullsized":"2024-03-29T11:33:36.895081+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:54.747726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.886348+0000","last_change":"2024-03-29T11:33:36.885141+0000","last_active":"2024-03-29T11:33:36.886348+0000","last_peered":"2024-03-29T11:33:36.886348+0000","last_clean":"2024-03-29T11:33:36.886348+0000","last_became_active":"2024-03-29T11:33:36.884840+0000","last_became_peered":"2024-03-29T11:33:36.884840+0000","last_unstale":"2024-03-29T11:33:36.886348+0000","last_undegraded":"2024-03-29T11:33:36.886348+0000","last_fullsized":"2024-03-29T11:33:36.886348+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:15.309041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"268'4","reported_seq":14,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.900582+0000","last_change":"2024-03-29T11:33:36.357638+0000","last_active":"2024-03-29T11:33:36.900582+0000","last_peered":"2024-03-29T11:33:36.900582+0000","last_clean":"2024-03-29T11:33:36.900582+0000","last_became_active":"2024-03-29T11:33:36.357370+0000","last_became_peered":"2024-03-29T11:33:36.357370+0000","last_unstale":"2024-03-29T11:33:36.900582+0000","last_undegraded":"2024-03-29T11:33:36.900582+0000","last_fullsized":"2024-03-29T11:33:36.900582+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:05.348804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.903426+0000","last_change":"2024-03-29T11:33:36.901731+0000","last_active":"2024-03-29T11:33:36.903426+0000","last_peered":"2024-03-29T11:33:36.903426+0000","last_clean":"2024-03-29T11:33:36.903426+0000","last_became_active":"2024-03-29T11:33:36.901150+0000","last_became_peered":"2024-03-29T11:33:36.901150+0000","last_unstale":"2024-03-29T11:33:36.903426+0000","last_undegraded":"2024-03-29T11:33:36.903426+0000","last_fullsized":"2024-03-29T11:33:36.903426+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:54.747726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.900612+0000","last_change":"2024-03-29T11:33:36.365389+0000","last_active":"2024-03-29T11:33:36.900612+0000","last_peered":"2024-03-29T11:33:36.900612+0000","last_clean":"2024-03-29T11:33:36.900612+0000","last_became_active":"2024-03-29T11:33:36.364763+0000","last_became_peered":"2024-03-29T11:33:36.364763+0000","last_unstale":"2024-03-29T11:33:36.900612+0000","last_undegraded":"2024-03-29T11:33:36.900612+0000","last_fullsized":"2024-03-29T11:33:36.900612+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:34:24.861299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.900648+0000","last_change":"2024-03-29T11:33:36.899206+0000","last_active":"2024-03-29T11:33:36.900648+0000","last_peered":"2024-03-29T11:33:36.900648+0000","last_clean":"2024-03-29T11:33:36.900648+0000","last_became_active":"2024-03-29T11:33:36.898932+0000","last_became_peered":"2024-03-29T11:33:36.898932+0000","last_unstale":"2024-03-29T11:33:36.900648+0000","last_undegraded":"2024-03-29T11:33:36.900648+0000","last_fullsized":"2024-03-29T11:33:36.900648+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:19:25.689275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.904867+0000","last_change":"2024-03-29T11:33:36.369096+0000","last_active":"2024-03-29T11:33:36.904867+0000","last_peered":"2024-03-29T11:33:36.904867+0000","last_clean":"2024-03-29T11:33:36.904867+0000","last_became_active":"2024-03-29T11:33:36.367885+0000","last_became_peered":"2024-03-29T11:33:36.367885+0000","last_unstale":"2024-03-29T11:33:36.904867+0000","last_undegraded":"2024-03-29T11:33:36.904867+0000","last_fullsized":"2024-03-29T11:33:36.904867+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:05.348804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.901004+0000","last_change":"2024-03-29T11:33:36.896791+0000","last_active":"2024-03-29T11:33:36.901004+0000","last_peered":"2024-03-29T11:33:36.901004+0000","last_clean":"2024-03-29T11:33:36.901004+0000","last_became_active":"2024-03-29T11:33:36.896502+0000","last_became_peered":"2024-03-29T11:33:36.896502+0000","last_unstale":"2024-03-29T11:33:36.901004+0000","last_undegraded":"2024-03-29T11:33:36.901004+0000","last_fullsized":"2024-03-29T11:33:36.901004+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:53.267539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"268'2","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875253+0000","last_change":"2024-03-29T11:33:36.364750+0000","last_active":"2024-03-29T11:33:37.875253+0000","last_peered":"2024-03-29T11:33:37.875253+0000","last_clean":"2024-03-29T11:33:37.875253+0000","last_became_active":"2024-03-29T11:33:36.364458+0000","last_became_peered":"2024-03-29T11:33:36.364458+0000","last_unstale":"2024-03-29T11:33:37.875253+0000","last_undegraded":"2024-03-29T11:33:37.875253+0000","last_fullsized":"2024-03-29T11:33:37.875253+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:05.348804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872671+0000","last_change":"2024-03-29T11:33:36.889260+0000","last_active":"2024-03-29T11:33:37.872671+0000","last_peered":"2024-03-29T11:33:37.872671+0000","last_clean":"2024-03-29T11:33:37.872671+0000","last_became_active":"2024-03-29T11:33:36.889003+0000","last_became_peered":"2024-03-29T11:33:36.889003+0000","last_unstale":"2024-03-29T11:33:37.872671+0000","last_undegraded":"2024-03-29T11:33:37.872671+0000","last_fullsized":"2024-03-29T11:33:37.872671+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:34:25.864333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.901515+0000","last_change":"2024-03-29T11:33:36.363922+0000","last_active":"2024-03-29T11:33:36.901515+0000","last_peered":"2024-03-29T11:33:36.901515+0000","last_clean":"2024-03-29T11:33:36.901515+0000","last_became_active":"2024-03-29T11:33:36.362834+0000","last_became_peered":"2024-03-29T11:33:36.362834+0000","last_unstale":"2024-03-29T11:33:36.901515+0000","last_undegraded":"2024-03-29T11:33:36.901515+0000","last_fullsized":"2024-03-29T11:33:36.901515+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:53:32.918021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875050+0000","last_change":"2024-03-29T11:33:36.886469+0000","last_active":"2024-03-29T11:33:37.875050+0000","last_peered":"2024-03-29T11:33:37.875050+0000","last_clean":"2024-03-29T11:33:37.875050+0000","last_became_active":"2024-03-29T11:33:36.885981+0000","last_became_peered":"2024-03-29T11:33:36.885981+0000","last_unstale":"2024-03-29T11:33:37.875050+0000","last_undegraded":"2024-03-29T11:33:37.875050+0000","last_fullsized":"2024-03-29T11:33:37.875050+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:50:51.040464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"268'1","reported_seq":12,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874132+0000","last_change":"2024-03-29T11:33:36.364399+0000","last_active":"2024-03-29T11:33:37.874132+0000","last_peered":"2024-03-29T11:33:37.874132+0000","last_clean":"2024-03-29T11:33:37.874132+0000","last_became_active":"2024-03-29T11:33:36.363301+0000","last_became_peered":"2024-03-29T11:33:36.363301+0000","last_unstale":"2024-03-29T11:33:37.874132+0000","last_undegraded":"2024-03-29T11:33:37.874132+0000","last_fullsized":"2024-03-29T11:33:37.874132+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:48:05.424999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.897622+0000","last_change":"2024-03-29T11:33:36.895551+0000","last_active":"2024-03-29T11:33:36.897622+0000","last_peered":"2024-03-29T11:33:36.897622+0000","last_clean":"2024-03-29T11:33:36.897622+0000","last_became_active":"2024-03-29T11:33:36.894886+0000","last_became_peered":"2024-03-29T11:33:36.894886+0000","last_unstale":"2024-03-29T11:33:36.897622+0000","last_undegraded":"2024-03-29T11:33:36.897622+0000","last_fullsized":"2024-03-29T11:33:36.897622+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:51:00.296286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.867066+0000","last_change":"2024-03-29T11:33:36.361389+0000","last_active":"2024-03-29T11:33:36.867066+0000","last_peered":"2024-03-29T11:33:36.867066+0000","last_clean":"2024-03-29T11:33:36.867066+0000","last_became_active":"2024-03-29T11:33:36.361132+0000","last_became_peered":"2024-03-29T11:33:36.361132+0000","last_unstale":"2024-03-29T11:33:36.867066+0000","last_undegraded":"2024-03-29T11:33:36.867066+0000","last_fullsized":"2024-03-29T11:33:36.867066+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:23:36.899065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.873998+0000","last_change":"2024-03-29T11:33:36.881385+0000","last_active":"2024-03-29T11:33:37.873998+0000","last_peered":"2024-03-29T11:33:37.873998+0000","last_clean":"2024-03-29T11:33:37.873998+0000","last_became_active":"2024-03-29T11:33:36.881111+0000","last_became_peered":"2024-03-29T11:33:36.881111+0000","last_unstale":"2024-03-29T11:33:37.873998+0000","last_undegraded":"2024-03-29T11:33:37.873998+0000","last_fullsized":"2024-03-29T11:33:37.873998+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:32:28.625576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.866965+0000","last_change":"2024-03-29T11:33:36.362972+0000","last_active":"2024-03-29T11:33:36.866965+0000","last_peered":"2024-03-29T11:33:36.866965+0000","last_clean":"2024-03-29T11:33:36.866965+0000","last_became_active":"2024-03-29T11:33:36.362704+0000","last_became_peered":"2024-03-29T11:33:36.362704+0000","last_unstale":"2024-03-29T11:33:36.866965+0000","last_undegraded":"2024-03-29T11:33:36.866965+0000","last_fullsized":"2024-03-29T11:33:36.866965+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:47:26.437713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874666+0000","last_change":"2024-03-29T11:33:36.883856+0000","last_active":"2024-03-29T11:33:37.874666+0000","last_peered":"2024-03-29T11:33:37.874666+0000","last_clean":"2024-03-29T11:33:37.874666+0000","last_became_active":"2024-03-29T11:33:36.883495+0000","last_became_peered":"2024-03-29T11:33:36.883495+0000","last_unstale":"2024-03-29T11:33:37.874666+0000","last_undegraded":"2024-03-29T11:33:37.874666+0000","last_fullsized":"2024-03-29T11:33:37.874666+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:53.412422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.887337+0000","last_change":"2024-03-29T11:33:36.365618+0000","last_active":"2024-03-29T11:33:36.887337+0000","last_peered":"2024-03-29T11:33:36.887337+0000","last_clean":"2024-03-29T11:33:36.887337+0000","last_became_active":"2024-03-29T11:33:36.365220+0000","last_became_peered":"2024-03-29T11:33:36.365220+0000","last_unstale":"2024-03-29T11:33:36.887337+0000","last_undegraded":"2024-03-29T11:33:36.887337+0000","last_fullsized":"2024-03-29T11:33:36.887337+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:10.581820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.904486+0000","last_change":"2024-03-29T11:33:36.902403+0000","last_active":"2024-03-29T11:33:36.904486+0000","last_peered":"2024-03-29T11:33:36.904486+0000","last_clean":"2024-03-29T11:33:36.904486+0000","last_became_active":"2024-03-29T11:33:36.899052+0000","last_became_peered":"2024-03-29T11:33:36.899052+0000","last_unstale":"2024-03-29T11:33:36.904486+0000","last_undegraded":"2024-03-29T11:33:36.904486+0000","last_fullsized":"2024-03-29T11:33:36.904486+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:04:50.012882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.876661+0000","last_change":"2024-03-29T11:33:36.369550+0000","last_active":"2024-03-29T11:33:36.876661+0000","last_peered":"2024-03-29T11:33:36.876661+0000","last_clean":"2024-03-29T11:33:36.876661+0000","last_became_active":"2024-03-29T11:33:36.369264+0000","last_became_peered":"2024-03-29T11:33:36.369264+0000","last_unstale":"2024-03-29T11:33:36.876661+0000","last_undegraded":"2024-03-29T11:33:36.876661+0000","last_fullsized":"2024-03-29T11:33:36.876661+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:40.209886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.902596+0000","last_change":"2024-03-29T11:33:36.901191+0000","last_active":"2024-03-29T11:33:36.902596+0000","last_peered":"2024-03-29T11:33:36.902596+0000","last_clean":"2024-03-29T11:33:36.902596+0000","last_became_active":"2024-03-29T11:33:36.899138+0000","last_became_peered":"2024-03-29T11:33:36.899138+0000","last_unstale":"2024-03-29T11:33:36.902596+0000","last_undegraded":"2024-03-29T11:33:36.902596+0000","last_fullsized":"2024-03-29T11:33:36.902596+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:51:00.296286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"268'3","reported_seq":13,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.902055+0000","last_change":"2024-03-29T11:33:36.368872+0000","last_active":"2024-03-29T11:33:36.902055+0000","last_peered":"2024-03-29T11:33:36.902055+0000","last_clean":"2024-03-29T11:33:36.902055+0000","last_became_active":"2024-03-29T11:33:36.367483+0000","last_became_peered":"2024-03-29T11:33:36.367483+0000","last_unstale":"2024-03-29T11:33:36.902055+0000","last_undegraded":"2024-03-29T11:33:36.902055+0000","last_fullsized":"2024-03-29T11:33:36.902055+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:47:26.437713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.895837+0000","last_change":"2024-03-29T11:33:36.893116+0000","last_active":"2024-03-29T11:33:36.895837+0000","last_peered":"2024-03-29T11:33:36.895837+0000","last_clean":"2024-03-29T11:33:36.895837+0000","last_became_active":"2024-03-29T11:33:36.892849+0000","last_became_peered":"2024-03-29T11:33:36.892849+0000","last_unstale":"2024-03-29T11:33:36.895837+0000","last_undegraded":"2024-03-29T11:33:36.895837+0000","last_fullsized":"2024-03-29T11:33:36.895837+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:02:27.050159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"268'6","reported_seq":16,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.905396+0000","last_change":"2024-03-29T11:33:36.359966+0000","last_active":"2024-03-29T11:33:36.905396+0000","last_peered":"2024-03-29T11:33:36.905396+0000","last_clean":"2024-03-29T11:33:36.905396+0000","last_became_active":"2024-03-29T11:33:36.359267+0000","last_became_peered":"2024-03-29T11:33:36.359267+0000","last_unstale":"2024-03-29T11:33:36.905396+0000","last_undegraded":"2024-03-29T11:33:36.905396+0000","last_fullsized":"2024-03-29T11:33:36.905396+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:21.625250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874722+0000","last_change":"2024-03-29T11:33:36.880389+0000","last_active":"2024-03-29T11:33:37.874722+0000","last_peered":"2024-03-29T11:33:37.874722+0000","last_clean":"2024-03-29T11:33:37.874722+0000","last_became_active":"2024-03-29T11:33:36.879945+0000","last_became_peered":"2024-03-29T11:33:36.879945+0000","last_unstale":"2024-03-29T11:33:37.874722+0000","last_undegraded":"2024-03-29T11:33:37.874722+0000","last_fullsized":"2024-03-29T11:33:37.874722+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:15.309041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"268'2","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872900+0000","last_change":"2024-03-29T11:33:36.362533+0000","last_active":"2024-03-29T11:33:37.872900+0000","last_peered":"2024-03-29T11:33:37.872900+0000","last_clean":"2024-03-29T11:33:37.872900+0000","last_became_active":"2024-03-29T11:33:36.361794+0000","last_became_peered":"2024-03-29T11:33:36.361794+0000","last_unstale":"2024-03-29T11:33:37.872900+0000","last_undegraded":"2024-03-29T11:33:37.872900+0000","last_fullsized":"2024-03-29T11:33:37.872900+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:04:49.009848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874088+0000","last_change":"2024-03-29T11:33:36.884480+0000","last_active":"2024-03-29T11:33:37.874088+0000","last_peered":"2024-03-29T11:33:37.874088+0000","last_clean":"2024-03-29T11:33:37.874088+0000","last_became_active":"2024-03-29T11:33:36.884202+0000","last_became_peered":"2024-03-29T11:33:36.884202+0000","last_unstale":"2024-03-29T11:33:37.874088+0000","last_undegraded":"2024-03-29T11:33:37.874088+0000","last_fullsized":"2024-03-29T11:33:37.874088+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:41:49.488639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872652+0000","last_change":"2024-03-29T11:33:36.366389+0000","last_active":"2024-03-29T11:33:37.872652+0000","last_peered":"2024-03-29T11:33:37.872652+0000","last_clean":"2024-03-29T11:33:37.872652+0000","last_became_active":"2024-03-29T11:33:36.366155+0000","last_became_peered":"2024-03-29T11:33:36.366155+0000","last_unstale":"2024-03-29T11:33:37.872652+0000","last_undegraded":"2024-03-29T11:33:37.872652+0000","last_fullsized":"2024-03-29T11:33:37.872652+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:05.947393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.901655+0000","last_change":"2024-03-29T11:33:36.897623+0000","last_active":"2024-03-29T11:33:36.901655+0000","last_peered":"2024-03-29T11:33:36.901655+0000","last_clean":"2024-03-29T11:33:36.901655+0000","last_became_active":"2024-03-29T11:33:36.897366+0000","last_became_peered":"2024-03-29T11:33:36.897366+0000","last_unstale":"2024-03-29T11:33:36.901655+0000","last_undegraded":"2024-03-29T11:33:36.901655+0000","last_fullsized":"2024-03-29T11:33:36.901655+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:43.216913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.884319+0000","last_change":"2024-03-29T11:33:36.362765+0000","last_active":"2024-03-29T11:33:36.884319+0000","last_peered":"2024-03-29T11:33:36.884319+0000","last_clean":"2024-03-29T11:33:36.884319+0000","last_became_active":"2024-03-29T11:33:36.362201+0000","last_became_peered":"2024-03-29T11:33:36.362201+0000","last_unstale":"2024-03-29T11:33:36.884319+0000","last_undegraded":"2024-03-29T11:33:36.884319+0000","last_fullsized":"2024-03-29T11:33:36.884319+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:19:11.386682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874721+0000","last_change":"2024-03-29T11:33:36.881618+0000","last_active":"2024-03-29T11:33:37.874721+0000","last_peered":"2024-03-29T11:33:37.874721+0000","last_clean":"2024-03-29T11:33:37.874721+0000","last_became_active":"2024-03-29T11:33:36.881218+0000","last_became_peered":"2024-03-29T11:33:36.881218+0000","last_unstale":"2024-03-29T11:33:37.874721+0000","last_undegraded":"2024-03-29T11:33:37.874721+0000","last_fullsized":"2024-03-29T11:33:37.874721+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:36:37.933506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"268'8","reported_seq":19,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872655+0000","last_change":"2024-03-29T11:33:36.368694+0000","last_active":"2024-03-29T11:33:37.872655+0000","last_peered":"2024-03-29T11:33:37.872655+0000","last_clean":"2024-03-29T11:33:37.872655+0000","last_became_active":"2024-03-29T11:33:36.368116+0000","last_became_peered":"2024-03-29T11:33:36.368116+0000","last_unstale":"2024-03-29T11:33:37.872655+0000","last_undegraded":"2024-03-29T11:33:37.872655+0000","last_fullsized":"2024-03-29T11:33:37.872655+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:31.150427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872824+0000","last_change":"2024-03-29T11:33:36.877057+0000","last_active":"2024-03-29T11:33:37.872824+0000","last_peered":"2024-03-29T11:33:37.872824+0000","last_clean":"2024-03-29T11:33:37.872824+0000","last_became_active":"2024-03-29T11:33:36.876421+0000","last_became_peered":"2024-03-29T11:33:36.876421+0000","last_unstale":"2024-03-29T11:33:37.872824+0000","last_undegraded":"2024-03-29T11:33:37.872824+0000","last_fullsized":"2024-03-29T11:33:37.872824+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:47:27.440747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.903892+0000","last_change":"2024-03-29T11:33:36.368155+0000","last_active":"2024-03-29T11:33:36.903892+0000","last_peered":"2024-03-29T11:33:36.903892+0000","last_clean":"2024-03-29T11:33:36.903892+0000","last_became_active":"2024-03-29T11:33:36.367899+0000","last_became_peered":"2024-03-29T11:33:36.367899+0000","last_unstale":"2024-03-29T11:33:36.903892+0000","last_undegraded":"2024-03-29T11:33:36.903892+0000","last_fullsized":"2024-03-29T11:33:36.903892+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:24.145740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.886640+0000","last_change":"2024-03-29T11:33:36.885436+0000","last_active":"2024-03-29T11:33:36.886640+0000","last_peered":"2024-03-29T11:33:36.886640+0000","last_clean":"2024-03-29T11:33:36.886640+0000","last_became_active":"2024-03-29T11:33:36.885179+0000","last_became_peered":"2024-03-29T11:33:36.885179+0000","last_unstale":"2024-03-29T11:33:36.886640+0000","last_undegraded":"2024-03-29T11:33:36.886640+0000","last_fullsized":"2024-03-29T11:33:36.886640+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:27:28.295616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.904803+0000","last_change":"2024-03-29T11:33:36.364293+0000","last_active":"2024-03-29T11:33:36.904803+0000","last_peered":"2024-03-29T11:33:36.904803+0000","last_clean":"2024-03-29T11:33:36.904803+0000","last_became_active":"2024-03-29T11:33:36.364045+0000","last_became_peered":"2024-03-29T11:33:36.364045+0000","last_unstale":"2024-03-29T11:33:36.904803+0000","last_undegraded":"2024-03-29T11:33:36.904803+0000","last_fullsized":"2024-03-29T11:33:36.904803+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:05:37.709439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874706+0000","last_change":"2024-03-29T11:33:36.897121+0000","last_active":"2024-03-29T11:33:37.874706+0000","last_peered":"2024-03-29T11:33:37.874706+0000","last_clean":"2024-03-29T11:33:37.874706+0000","last_became_active":"2024-03-29T11:33:36.896843+0000","last_became_peered":"2024-03-29T11:33:36.896843+0000","last_unstale":"2024-03-29T11:33:37.874706+0000","last_undegraded":"2024-03-29T11:33:37.874706+0000","last_fullsized":"2024-03-29T11:33:37.874706+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:48.360209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.896408+0000","last_change":"2024-03-29T11:33:36.366518+0000","last_active":"2024-03-29T11:33:36.896408+0000","last_peered":"2024-03-29T11:33:36.896408+0000","last_clean":"2024-03-29T11:33:36.896408+0000","last_became_active":"2024-03-29T11:33:36.366180+0000","last_became_peered":"2024-03-29T11:33:36.366180+0000","last_unstale":"2024-03-29T11:33:36.896408+0000","last_undegraded":"2024-03-29T11:33:36.896408+0000","last_fullsized":"2024-03-29T11:33:36.896408+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:37.313964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.901023+0000","last_change":"2024-03-29T11:33:36.897048+0000","last_active":"2024-03-29T11:33:36.901023+0000","last_peered":"2024-03-29T11:33:36.901023+0000","last_clean":"2024-03-29T11:33:36.901023+0000","last_became_active":"2024-03-29T11:33:36.890367+0000","last_became_peered":"2024-03-29T11:33:36.890367+0000","last_unstale":"2024-03-29T11:33:36.901023+0000","last_undegraded":"2024-03-29T11:33:36.901023+0000","last_fullsized":"2024-03-29T11:33:36.901023+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:23:35.014690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.884995+0000","last_change":"2024-03-29T11:33:36.367121+0000","last_active":"2024-03-29T11:33:36.884995+0000","last_peered":"2024-03-29T11:33:36.884995+0000","last_clean":"2024-03-29T11:33:36.884995+0000","last_became_active":"2024-03-29T11:33:36.366822+0000","last_became_peered":"2024-03-29T11:33:36.366822+0000","last_unstale":"2024-03-29T11:33:36.884995+0000","last_undegraded":"2024-03-29T11:33:36.884995+0000","last_fullsized":"2024-03-29T11:33:36.884995+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:01:05.143658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899973+0000","last_change":"2024-03-29T11:33:36.898690+0000","last_active":"2024-03-29T11:33:36.899973+0000","last_peered":"2024-03-29T11:33:36.899973+0000","last_clean":"2024-03-29T11:33:36.899973+0000","last_became_active":"2024-03-29T11:33:36.898324+0000","last_became_peered":"2024-03-29T11:33:36.898324+0000","last_unstale":"2024-03-29T11:33:36.899973+0000","last_undegraded":"2024-03-29T11:33:36.899973+0000","last_fullsized":"2024-03-29T11:33:36.899973+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:32:40.653645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.896434+0000","last_change":"2024-03-29T11:33:36.359779+0000","last_active":"2024-03-29T11:33:36.896434+0000","last_peered":"2024-03-29T11:33:36.896434+0000","last_clean":"2024-03-29T11:33:36.896434+0000","last_became_active":"2024-03-29T11:33:36.357360+0000","last_became_peered":"2024-03-29T11:33:36.357360+0000","last_unstale":"2024-03-29T11:33:36.896434+0000","last_undegraded":"2024-03-29T11:33:36.896434+0000","last_fullsized":"2024-03-29T11:33:36.896434+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:36:51.414204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874298+0000","last_change":"2024-03-29T11:33:36.884984+0000","last_active":"2024-03-29T11:33:37.874298+0000","last_peered":"2024-03-29T11:33:37.874298+0000","last_clean":"2024-03-29T11:33:37.874298+0000","last_became_active":"2024-03-29T11:33:36.884714+0000","last_became_peered":"2024-03-29T11:33:36.884714+0000","last_unstale":"2024-03-29T11:33:37.874298+0000","last_undegraded":"2024-03-29T11:33:37.874298+0000","last_fullsized":"2024-03-29T11:33:37.874298+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:19:12.389716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.874650+0000","last_change":"2024-03-29T11:33:36.360935+0000","last_active":"2024-03-29T11:33:36.874650+0000","last_peered":"2024-03-29T11:33:36.874650+0000","last_clean":"2024-03-29T11:33:36.874650+0000","last_became_active":"2024-03-29T11:33:36.360604+0000","last_became_peered":"2024-03-29T11:33:36.360604+0000","last_unstale":"2024-03-29T11:33:36.874650+0000","last_undegraded":"2024-03-29T11:33:36.874650+0000","last_fullsized":"2024-03-29T11:33:36.874650+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:49:20.127472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874514+0000","last_change":"2024-03-29T11:33:36.883467+0000","last_active":"2024-03-29T11:33:37.874514+0000","last_peered":"2024-03-29T11:33:37.874514+0000","last_clean":"2024-03-29T11:33:37.874514+0000","last_became_active":"2024-03-29T11:33:36.883161+0000","last_became_peered":"2024-03-29T11:33:36.883161+0000","last_unstale":"2024-03-29T11:33:37.874514+0000","last_undegraded":"2024-03-29T11:33:37.874514+0000","last_fullsized":"2024-03-29T11:33:37.874514+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:02.895952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.884449+0000","last_change":"2024-03-29T11:33:36.367037+0000","last_active":"2024-03-29T11:33:36.884449+0000","last_peered":"2024-03-29T11:33:36.884449+0000","last_clean":"2024-03-29T11:33:36.884449+0000","last_became_active":"2024-03-29T11:33:36.366741+0000","last_became_peered":"2024-03-29T11:33:36.366741+0000","last_unstale":"2024-03-29T11:33:36.884449+0000","last_undegraded":"2024-03-29T11:33:36.884449+0000","last_fullsized":"2024-03-29T11:33:36.884449+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:47.357175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.896581+0000","last_change":"2024-03-29T11:33:36.895057+0000","last_active":"2024-03-29T11:33:36.896581+0000","last_peered":"2024-03-29T11:33:36.896581+0000","last_clean":"2024-03-29T11:33:36.896581+0000","last_became_active":"2024-03-29T11:33:36.894583+0000","last_became_peered":"2024-03-29T11:33:36.894583+0000","last_unstale":"2024-03-29T11:33:36.896581+0000","last_undegraded":"2024-03-29T11:33:36.896581+0000","last_fullsized":"2024-03-29T11:33:36.896581+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:12:07.903315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.866956+0000","last_change":"2024-03-29T11:33:36.358732+0000","last_active":"2024-03-29T11:33:36.866956+0000","last_peered":"2024-03-29T11:33:36.866956+0000","last_clean":"2024-03-29T11:33:36.866956+0000","last_became_active":"2024-03-29T11:33:36.358474+0000","last_became_peered":"2024-03-29T11:33:36.358474+0000","last_unstale":"2024-03-29T11:33:36.866956+0000","last_undegraded":"2024-03-29T11:33:36.866956+0000","last_fullsized":"2024-03-29T11:33:36.866956+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:34:24.861299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875371+0000","last_change":"2024-03-29T11:33:36.891234+0000","last_active":"2024-03-29T11:33:37.875371+0000","last_peered":"2024-03-29T11:33:37.875371+0000","last_clean":"2024-03-29T11:33:37.875371+0000","last_became_active":"2024-03-29T11:33:36.890942+0000","last_became_peered":"2024-03-29T11:33:36.890942+0000","last_unstale":"2024-03-29T11:33:37.875371+0000","last_undegraded":"2024-03-29T11:33:37.875371+0000","last_fullsized":"2024-03-29T11:33:37.875371+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:12:07.903315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.904913+0000","last_change":"2024-03-29T11:33:36.371487+0000","last_active":"2024-03-29T11:33:36.904913+0000","last_peered":"2024-03-29T11:33:36.904913+0000","last_clean":"2024-03-29T11:33:36.904913+0000","last_became_active":"2024-03-29T11:33:36.371125+0000","last_became_peered":"2024-03-29T11:33:36.371125+0000","last_unstale":"2024-03-29T11:33:36.904913+0000","last_undegraded":"2024-03-29T11:33:36.904913+0000","last_fullsized":"2024-03-29T11:33:36.904913+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:05.348804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874971+0000","last_change":"2024-03-29T11:33:36.887396+0000","last_active":"2024-03-29T11:33:37.874971+0000","last_peered":"2024-03-29T11:33:37.874971+0000","last_clean":"2024-03-29T11:33:37.874971+0000","last_became_active":"2024-03-29T11:33:36.887047+0000","last_became_peered":"2024-03-29T11:33:36.887047+0000","last_unstale":"2024-03-29T11:33:37.874971+0000","last_undegraded":"2024-03-29T11:33:37.874971+0000","last_fullsized":"2024-03-29T11:33:37.874971+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:50.427129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":11,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874755+0000","last_change":"2024-03-29T11:33:36.362642+0000","last_active":"2024-03-29T11:33:37.874755+0000","last_peered":"2024-03-29T11:33:37.874755+0000","last_clean":"2024-03-29T11:33:37.874755+0000","last_became_active":"2024-03-29T11:33:36.360316+0000","last_became_peered":"2024-03-29T11:33:36.360316+0000","last_unstale":"2024-03-29T11:33:37.874755+0000","last_undegraded":"2024-03-29T11:33:37.874755+0000","last_fullsized":"2024-03-29T11:33:37.874755+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:41:00.390407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875130+0000","last_change":"2024-03-29T11:33:36.888685+0000","last_active":"2024-03-29T11:33:37.875130+0000","last_peered":"2024-03-29T11:33:37.875130+0000","last_clean":"2024-03-29T11:33:37.875130+0000","last_became_active":"2024-03-29T11:33:36.888397+0000","last_became_peered":"2024-03-29T11:33:36.888397+0000","last_unstale":"2024-03-29T11:33:37.875130+0000","last_undegraded":"2024-03-29T11:33:37.875130+0000","last_fullsized":"2024-03-29T11:33:37.875130+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:52.609495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.876889+0000","last_change":"2024-03-29T11:33:36.370644+0000","last_active":"2024-03-29T11:33:36.876889+0000","last_peered":"2024-03-29T11:33:36.876889+0000","last_clean":"2024-03-29T11:33:36.876889+0000","last_became_active":"2024-03-29T11:33:36.370302+0000","last_became_peered":"2024-03-29T11:33:36.370302+0000","last_unstale":"2024-03-29T11:33:36.876889+0000","last_undegraded":"2024-03-29T11:33:36.876889+0000","last_fullsized":"2024-03-29T11:33:36.876889+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:34:24.861299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899519+0000","last_change":"2024-03-29T11:33:36.896213+0000","last_active":"2024-03-29T11:33:36.899519+0000","last_peered":"2024-03-29T11:33:36.899519+0000","last_clean":"2024-03-29T11:33:36.899519+0000","last_became_active":"2024-03-29T11:33:36.895319+0000","last_became_peered":"2024-03-29T11:33:36.895319+0000","last_unstale":"2024-03-29T11:33:36.899519+0000","last_undegraded":"2024-03-29T11:33:36.899519+0000","last_fullsized":"2024-03-29T11:33:36.899519+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:51:59.814486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.881508+0000","last_change":"2024-03-29T11:33:36.370316+0000","last_active":"2024-03-29T11:33:36.881508+0000","last_peered":"2024-03-29T11:33:36.881508+0000","last_clean":"2024-03-29T11:33:36.881508+0000","last_became_active":"2024-03-29T11:33:36.369765+0000","last_became_peered":"2024-03-29T11:33:36.369765+0000","last_unstale":"2024-03-29T11:33:36.881508+0000","last_undegraded":"2024-03-29T11:33:36.881508+0000","last_fullsized":"2024-03-29T11:33:36.881508+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:47:26.437713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899447+0000","last_change":"2024-03-29T11:33:36.898119+0000","last_active":"2024-03-29T11:33:36.899447+0000","last_peered":"2024-03-29T11:33:36.899447+0000","last_clean":"2024-03-29T11:33:36.899447+0000","last_became_active":"2024-03-29T11:33:36.897865+0000","last_became_peered":"2024-03-29T11:33:36.897865+0000","last_unstale":"2024-03-29T11:33:36.899447+0000","last_undegraded":"2024-03-29T11:33:36.899447+0000","last_fullsized":"2024-03-29T11:33:36.899447+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:43.216913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875107+0000","last_change":"2024-03-29T11:33:36.361285+0000","last_active":"2024-03-29T11:33:37.875107+0000","last_peered":"2024-03-29T11:33:37.875107+0000","last_clean":"2024-03-29T11:33:37.875107+0000","last_became_active":"2024-03-29T11:33:36.360938+0000","last_became_peered":"2024-03-29T11:33:36.360938+0000","last_unstale":"2024-03-29T11:33:37.875107+0000","last_undegraded":"2024-03-29T11:33:37.875107+0000","last_fullsized":"2024-03-29T11:33:37.875107+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:40.209886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874424+0000","last_change":"2024-03-29T11:33:36.883698+0000","last_active":"2024-03-29T11:33:37.874424+0000","last_peered":"2024-03-29T11:33:37.874424+0000","last_clean":"2024-03-29T11:33:37.874424+0000","last_became_active":"2024-03-29T11:33:36.883417+0000","last_became_peered":"2024-03-29T11:33:36.883417+0000","last_unstale":"2024-03-29T11:33:37.874424+0000","last_undegraded":"2024-03-29T11:33:37.874424+0000","last_fullsized":"2024-03-29T11:33:37.874424+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:01:06.146692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.867059+0000","last_change":"2024-03-29T11:33:36.359437+0000","last_active":"2024-03-29T11:33:36.867059+0000","last_peered":"2024-03-29T11:33:36.867059+0000","last_clean":"2024-03-29T11:33:36.867059+0000","last_became_active":"2024-03-29T11:33:36.359103+0000","last_became_peered":"2024-03-29T11:33:36.359103+0000","last_unstale":"2024-03-29T11:33:36.867059+0000","last_undegraded":"2024-03-29T11:33:36.867059+0000","last_fullsized":"2024-03-29T11:33:36.867059+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:40.209886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874825+0000","last_change":"2024-03-29T11:33:36.879276+0000","last_active":"2024-03-29T11:33:37.874825+0000","last_peered":"2024-03-29T11:33:37.874825+0000","last_clean":"2024-03-29T11:33:37.874825+0000","last_became_active":"2024-03-29T11:33:36.878990+0000","last_became_peered":"2024-03-29T11:33:36.878990+0000","last_unstale":"2024-03-29T11:33:37.874825+0000","last_undegraded":"2024-03-29T11:33:37.874825+0000","last_fullsized":"2024-03-29T11:33:37.874825+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:11:20.442172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.867161+0000","last_change":"2024-03-29T11:33:36.359461+0000","last_active":"2024-03-29T11:33:36.867161+0000","last_peered":"2024-03-29T11:33:36.867161+0000","last_clean":"2024-03-29T11:33:36.867161+0000","last_became_active":"2024-03-29T11:33:36.359171+0000","last_became_peered":"2024-03-29T11:33:36.359171+0000","last_unstale":"2024-03-29T11:33:36.867161+0000","last_undegraded":"2024-03-29T11:33:36.867161+0000","last_fullsized":"2024-03-29T11:33:36.867161+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:40:30.035629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899555+0000","last_change":"2024-03-29T11:33:36.894647+0000","last_active":"2024-03-29T11:33:36.899555+0000","last_peered":"2024-03-29T11:33:36.899555+0000","last_clean":"2024-03-29T11:33:36.899555+0000","last_became_active":"2024-03-29T11:33:36.893163+0000","last_became_peered":"2024-03-29T11:33:36.893163+0000","last_unstale":"2024-03-29T11:33:36.899555+0000","last_undegraded":"2024-03-29T11:33:36.899555+0000","last_fullsized":"2024-03-29T11:33:36.899555+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:17.289090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.874045+0000","last_change":"2024-03-29T11:33:36.359465+0000","last_active":"2024-03-29T11:33:36.874045+0000","last_peered":"2024-03-29T11:33:36.874045+0000","last_clean":"2024-03-29T11:33:36.874045+0000","last_became_active":"2024-03-29T11:33:36.358184+0000","last_became_peered":"2024-03-29T11:33:36.358184+0000","last_unstale":"2024-03-29T11:33:36.874045+0000","last_undegraded":"2024-03-29T11:33:36.874045+0000","last_fullsized":"2024-03-29T11:33:36.874045+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:42:25.421857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.901296+0000","last_change":"2024-03-29T11:33:36.895335+0000","last_active":"2024-03-29T11:33:36.901296+0000","last_peered":"2024-03-29T11:33:36.901296+0000","last_clean":"2024-03-29T11:33:36.901296+0000","last_became_active":"2024-03-29T11:33:36.895052+0000","last_became_peered":"2024-03-29T11:33:36.895052+0000","last_unstale":"2024-03-29T11:33:36.901296+0000","last_undegraded":"2024-03-29T11:33:36.901296+0000","last_fullsized":"2024-03-29T11:33:36.901296+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:32:40.653645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"254'410","reported_seq":1463,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.360414+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:33:36.360414+0000","last_peered":"2024-03-29T11:33:36.360414+0000","last_clean":"2024-03-29T11:33:36.360414+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:33:36.360414+0000","last_undegraded":"2024-03-29T11:33:36.360414+0000","last_fullsized":"2024-03-29T11:33:36.360414+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2672,"num_read_kb":5632,"num_write":691,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2672,"num_read_kb":5632,"num_write":691,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":410,"ondisk_log_size":410,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576017,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57144,"kb_used_data":3884,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314696,"statfs":{"total":96636764160,"available":96578248704,"internally_reserved":0,"allocated":3977216,"data_stored":2257580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.67600000000000005,"15min":0.70799999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86799999999999999,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.68100000000000005,"15min":0.73099999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86899999999999999,"5min":0.98899999999999999,"15min":1.119},"last":0.51600000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.67100000000000004,"15min":0.71599999999999997},"min":{"1min":0.435,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84399999999999997,"5min":0.97399999999999998,"15min":1.478},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.70399999999999996,"15min":0.752},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.82699999999999996,"5min":1.0640000000000001,"15min":1.393},"last":0.623}]},{"osd":2,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70499999999999996,"15min":0.748},"min":{"1min":0.46600000000000003,"5min":0.436,"15min":0.436},"max":{"1min":2.5649999999999999,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.71399999999999997,"15min":0.76100000000000001},"min":{"1min":0.501,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88400000000000001,"5min":1.032,"15min":1.0720000000000001},"last":0.59299999999999997}]},{"osd":3,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.749},"min":{"1min":0.48499999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84299999999999997,"5min":0.97499999999999998,"15min":1.0580000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.77700000000000002,"15min":0.78600000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.6930000000000001,"5min":2.6930000000000001,"15min":2.6930000000000001},"last":0.70099999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88500000000000001,"5min":0.91100000000000003,"15min":1.3779999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66400000000000003,"15min":0.66900000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90800000000000003,"5min":0.93500000000000005,"15min":1.3169999999999999},"last":0.65400000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69999999999999996,"15min":0.68400000000000005},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.1160000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93700000000000006,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.64300000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.76800000000000002,"15min":0.76600000000000001},"min":{"1min":0.48099999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92600000000000005,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.75600000000000001,"15min":0.74199999999999999},"min":{"1min":0.52000000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.91000000000000003,"5min":1.071,"15min":1.071},"last":0.60899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576018,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48380,"kb_used_data":3308,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323460,"statfs":{"total":96636764160,"available":96587223040,"internally_reserved":0,"allocated":3387392,"data_stored":1667302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.67100000000000004,"15min":0.71999999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.94499999999999995,"15min":1.2150000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.68400000000000005,"15min":0.74199999999999999},"min":{"1min":0.42799999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95299999999999996,"5min":0.96399999999999997,"15min":1.4239999999999999},"last":0.64300000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67000000000000004,"15min":0.72199999999999998},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91100000000000003,"5min":1.022,"15min":1.98},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.70699999999999996,"15min":0.75900000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.435},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.8200000000000001},"last":0.80300000000000005}]},{"osd":2,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69699999999999995,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86899999999999999,"5min":0.96199999999999997,"15min":1.669},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71099999999999997,"15min":0.751},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.167,"5min":1.167,"15min":1.7589999999999999},"last":0.747}]},{"osd":3,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70599999999999996,"15min":0.753},"min":{"1min":0.51100000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90200000000000002,"5min":0.997,"15min":2.52},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.75700000000000001,"15min":0.79200000000000004},"min":{"1min":0.51200000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.877},"last":0.877}]},{"osd":4,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.61599999999999999},"min":{"1min":0.435,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93999999999999995,"15min":2.4100000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.71599999999999997,"15min":0.71199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.45900000000000002},"max":{"1min":1.0740000000000001,"5min":1.2709999999999999,"15min":2.6629999999999998},"last":0.91100000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66900000000000004,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99099999999999999,"5min":1.0900000000000001,"15min":1.5960000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.925},"last":0.93300000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77100000000000002,"15min":0.76900000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96699999999999997,"5min":1.0509999999999999,"15min":2.7050000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74299999999999999,"15min":0.74399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0980000000000001,"5min":1.1220000000000001,"15min":1.726},"last":1}]}]},{"osd":1,"up_from":13,"seq":55834576016,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60672,"kb_used_data":3308,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311168,"statfs":{"total":96636764160,"available":96574636032,"internally_reserved":0,"allocated":3387392,"data_stored":1667246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.44400000000000001,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":0.79100000000000004,"15min":0.80300000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.505,"15min":0.52400000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.58899999999999997,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52600000000000002,"15min":0.55400000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.625,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.751,"5min":0.75900000000000001,"15min":0.90300000000000002},"last":0.33800000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56699999999999995,"15min":0.58799999999999997},"min":{"1min":0.38100000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.85099999999999998,"15min":0.995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.54500000000000004,"15min":0.57399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76200000000000001,"5min":0.80100000000000005,"15min":0.83999999999999997},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.74399999999999999,"15min":0.78000000000000003},"min":{"1min":0.52900000000000003,"5min":0.505,"15min":0.505},"max":{"1min":0.997,"5min":1.153,"15min":1.238},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76800000000000002,"15min":0.79300000000000004},"min":{"1min":0.57599999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.097,"5min":1.226,"15min":1.304},"last":0.90100000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80000000000000004,"15min":0.81299999999999994},"min":{"1min":0.57499999999999996,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":0.95899999999999996,"5min":1.276,"15min":1.3720000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.78000000000000003,"15min":0.82499999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.05,"5min":1.1299999999999999,"15min":1.3520000000000001},"last":0.96199999999999997}]},{"osd":6,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.78200000000000003,"15min":0.81100000000000005},"min":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.103,"5min":1.1890000000000001,"15min":1.1919999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.81499999999999995,"15min":0.83999999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.012,"5min":1.2250000000000001,"15min":1.266},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.83899999999999997,"15min":0.86299999999999999},"min":{"1min":0.60899999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.129,"5min":1.327,"15min":1.327},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.82099999999999995,"15min":0.86099999999999999},"min":{"1min":0.57999999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0569999999999999,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.91400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576018,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61256,"kb_used_data":3892,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310584,"statfs":{"total":96636764160,"available":96574038016,"internally_reserved":0,"allocated":3985408,"data_stored":2257199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44900000000000001,"15min":0.46100000000000002},"min":{"1min":0.317,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.623,"5min":0.95899999999999996,"15min":1.087},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.52200000000000002,"15min":0.53500000000000003},"min":{"1min":0.32400000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67200000000000004,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56000000000000005,"15min":0.57899999999999996},"min":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66600000000000004,"5min":1.028,"15min":1.5289999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.499,"15min":0.51600000000000001},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.629,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55600000000000005,"15min":0.56200000000000006},"min":{"1min":0.38500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77600000000000002,"5min":1.222,"15min":1.222},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.622,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.44,"15min":0.42199999999999999},"max":{"1min":0.81799999999999995,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.60899999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80500000000000005,"15min":0.81399999999999995},"min":{"1min":0.56899999999999995,"5min":0.55600000000000005,"15min":0.495},"max":{"1min":0.93400000000000005,"5min":1.248,"15min":1.248},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82399999999999995,"15min":0.84599999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0309999999999999,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.82999999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76800000000000002,"15min":0.78500000000000003},"min":{"1min":0.54500000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.88900000000000001,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80400000000000005,"15min":0.81899999999999995},"min":{"1min":0.58999999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94899999999999995,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.79500000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.83399999999999996,"15min":0.85299999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.97599999999999998,"5min":1.343,"15min":1.6240000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.85699999999999998,"15min":0.88200000000000001},"min":{"1min":0.62,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.004,"5min":1.3240000000000001,"15min":1.337},"last":0.89600000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79900000000000004,"15min":0.82399999999999995},"min":{"1min":0.56899999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.99099999999999999,"5min":1.115,"15min":1.214},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.84099999999999997,"15min":0.86899999999999999},"min":{"1min":0.55500000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.054,"5min":1.365,"15min":1.365},"last":0.98999999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576017,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52476,"kb_used_data":3308,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319364,"statfs":{"total":96636764160,"available":96583028736,"internally_reserved":0,"allocated":3387392,"data_stored":1667302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.47299999999999998,"15min":0.47199999999999998},"min":{"1min":0.32300000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.5,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60199999999999998,"5min":0.73999999999999999,"15min":0.96799999999999997},"last":0.34699999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47899999999999998,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.54900000000000004,"15min":0.56999999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.872,"15min":0.88800000000000001},"last":0.45700000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55800000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64900000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.439},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55700000000000005,"15min":0.57199999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.71099999999999997,"5min":1.038,"15min":1.038},"last":0.55900000000000005}]},{"osd":4,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.754,"15min":0.79800000000000004},"min":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1970000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.747,"15min":0.79900000000000004},"min":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0029999999999999,"5min":1.157,"15min":1.244},"last":0.72599999999999998}]},{"osd":5,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77000000000000002,"15min":0.80900000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.53400000000000003},"max":{"1min":0.96399999999999997,"5min":1.135,"15min":1.256},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.76500000000000001,"15min":0.81399999999999995},"min":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.3140000000000001},"last":0.89100000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.78000000000000003,"15min":0.83099999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93200000000000005,"5min":1.105,"15min":1.24},"last":0.873},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.78000000000000003,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.5,"15min":0.5},"max":{"1min":0.98199999999999998,"5min":1.099,"15min":1.329},"last":0.86199999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82499999999999996,"15min":0.88},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.044,"5min":1.0660000000000001,"15min":1.2889999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.83899999999999997,"15min":0.89400000000000002},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.56899999999999995},"max":{"1min":1.0269999999999999,"5min":1.2969999999999999,"15min":1.304},"last":0.90700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576017,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48384,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323456,"statfs":{"total":96636764160,"available":96587218944,"internally_reserved":0,"allocated":3395584,"data_stored":1668776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.29599999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.33200000000000002,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.29899999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48399999999999999,"15min":0.49399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.57199999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.38700000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.435},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.51400000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79600000000000004,"15min":0.79700000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.80300000000000005,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.86899999999999999}]},{"osd":5,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80500000000000005,"15min":0.81299999999999994},"min":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.502},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.83099999999999996,"15min":0.82699999999999996},"min":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.54600000000000004},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":0.76400000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86599999999999999,"15min":0.85899999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.746},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.84099999999999997,"15min":0.84299999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":0.79500000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86699999999999999,"15min":0.86599999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.58099999999999996},"max":{"1min":2.444,"5min":2.444,"15min":2.444},"last":0.78600000000000003}]}]},{"osd":4,"up_from":13,"seq":55834576018,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78660,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":75326,"kb_avail":94293180,"statfs":{"total":96636764160,"available":96556216320,"internally_reserved":0,"allocated":3391488,"data_stored":1667304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77134278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64400000000000002,"15min":0.70699999999999996},"min":{"1min":0.39700000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":1.0509999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.64000000000000001,"15min":0.70799999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.81000000000000005,"5min":0.93600000000000005,"15min":1.1459999999999999},"last":0.76300000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.65500000000000003,"15min":0.72499999999999998},"min":{"1min":0.48699999999999999,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.1910000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67000000000000004,"15min":0.72999999999999998},"min":{"1min":0.435,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":1.002},"last":0.74299999999999999}]},{"osd":2,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76300000000000001,"15min":0.81399999999999995},"min":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.73899999999999999,"15min":0.80000000000000004},"min":{"1min":0.51900000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.875,"5min":1.224,"15min":1.224},"last":0.84699999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68500000000000005,"15min":0.748},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88100000000000001,"5min":0.96699999999999997,"15min":1.181},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72899999999999998,"15min":0.76300000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.83399999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64900000000000002,"15min":0.628},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.01,"5min":1.0920000000000001,"15min":1.169},"last":0.85999999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69599999999999995,"15min":0.67300000000000004},"min":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72599999999999998,"15min":0.72099999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70899999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.73899999999999999,"15min":0.73399999999999999},"min":{"1min":0.54100000000000004,"5min":0.501,"15min":0.436},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.2669999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.81899999999999995,"15min":0.82299999999999995},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.54400000000000004},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.78600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576017,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48396,"kb_used_data":3324,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323444,"statfs":{"total":96636764160,"available":96587206656,"internally_reserved":0,"allocated":3403776,"data_stored":1669428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.69799999999999995},"min":{"1min":0.47199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.72299999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.95599999999999996,"15min":1.0680000000000001},"last":0.48299999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73699999999999999,"15min":0.75700000000000001},"min":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98099999999999998,"5min":1.839,"15min":1.839},"last":0.434},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.71899999999999997,"15min":0.73599999999999999},"min":{"1min":0.504,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.282,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.753,"15min":0.755},"min":{"1min":0.51000000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.71499999999999997,"15min":0.74399999999999999},"min":{"1min":0.54400000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.48499999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.95099999999999996,"5min":3.444,"15min":3.444},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.75800000000000001,"15min":0.77200000000000002},"min":{"1min":0.54400000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.629,"15min":0.60899999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38700000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67800000000000005,"15min":0.65300000000000002},"min":{"1min":0.47799999999999998,"5min":0.41699999999999998,"15min":0.40600000000000003},"max":{"1min":0.999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.78700000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.69699999999999995,"15min":0.67000000000000004},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72199999999999998,"15min":0.70299999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.107},"last":0.747}]},{"osd":7,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.746,"15min":0.72599999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.42299999999999999},"max":{"1min":1.04,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79600000000000004,"15min":0.77400000000000002},"min":{"1min":0.53500000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.71499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:33:41.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:33:42.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.032+0000 7fb65ce00700 1 Processor -- start 2024-03-29T11:33:42.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.032+0000 7fb65ce00700 1 -- start start 2024-03-29T11:33:42.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.033+0000 7fb65ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb658161ca0 0x7fb6580cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.033+0000 7fb65ce00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6580ceda0 con 0x7fb658160f10 2024-03-29T11:33:42.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.033+0000 7fb65ce00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6581612f0 con 0x7fb6580d18b0 2024-03-29T11:33:42.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.033+0000 7fb65ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580d1d30 con 0x7fb658161ca0 2024-03-29T11:33:42.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.033+0000 7fb655e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb658160f10 0x7fb6580d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49684/0 (socket says 172.21.15.12:49684) 2024-03-29T11:33:42.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.033+0000 7fb655e00700 1 -- 172.21.15.12:0/2105751489 learned_addr learned my addr 172.21.15.12:0/2105751489 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:42.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb656800700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb658161ca0 0x7fb6580cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894820511 0 0) 0x7fb6580ceda0 con 0x7fb658160f10 2024-03-29T11:33:42.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb63c0035b0 con 0x7fb658160f10 2024-03-29T11:33:42.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247601470 0 0) 0x7fb6581612f0 con 0x7fb6580d18b0 2024-03-29T11:33:42.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6580ceda0 con 0x7fb6580d18b0 2024-03-29T11:33:42.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3992496699 0 0) 0x7fb63c0035b0 con 0x7fb658160f10 2024-03-29T11:33:42.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb658161ca0 msgr2=0x7fb6580cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:42.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb658161ca0 0x7fb6580cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 >> v1:172.21.15.159:6789/0 conn(0x7fb6580d18b0 legacy=0x7fb658161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6580ce470 con 0x7fb658160f10 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fb648001670 con 0x7fb658160f10 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 >> v1:172.21.15.12:6789/0 conn(0x7fb658160f10 legacy=0x7fb6580d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb63c003bc0 0x7fb63c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb63c004850 0x7fb63c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb63c005210 0x7fb63c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580d1d30 con 0x7fb63c004850 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c00c080 con 0x7fb63c003bc0 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.034+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c00c240 con 0x7fb63c005210 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb656800700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb658161ca0 0x7fb6580cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb655400700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb63c004850 0x7fb63c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb655e00700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb63c003bc0 0x7fb63c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb656800700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb63c005210 0x7fb63c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb655400700 1 -- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb63c005210 msgr2=0x7fb63c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb655400700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb63c005210 0x7fb63c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb655400700 1 -- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb63c003bc0 msgr2=0x7fb63c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb655400700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb63c003bc0 0x7fb63c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb655400700 1 -- 172.21.15.12:0/2105751489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c0035b0 con 0x7fb63c004850 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb656800700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb63c005210 0x7fb63c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.035+0000 7fb655400700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb63c004850 0x7fb63c004cd0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fb644001980 tx=0x7fb63c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca47e05ec09a96cb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.036+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644001470 con 0x7fb63c004850 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.036+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6440052d0 con 0x7fb63c004850 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.036+0000 7fb654a00700 1 -- 172.21.15.12:0/2105751489 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644003680 con 0x7fb63c004850 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.036+0000 7fb65ce00700 1 -- 172.21.15.12:0/2105751489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb63c004850 msgr2=0x7fb63c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.036+0000 7fb65ce00700 1 --2- 172.21.15.12:0/2105751489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb63c004850 0x7fb63c004cd0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fb644001980 tx=0x7fb63c005880 comp rx=0 tx=0).stop 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.036+0000 7fb656800700 1 -- 172.21.15.12:0/2105751489 reap_dead start 2024-03-29T11:33:42.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.036+0000 7fb65ce00700 1 -- 172.21.15.12:0/2105751489 shutdown_connections 2024-03-29T11:33:42.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.036+0000 7fb65ce00700 1 -- 172.21.15.12:0/2105751489 >> 172.21.15.12:0/2105751489 conn(0x7fb65808aff0 msgr2=0x7fb6580d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:42.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.036+0000 7fb65ce00700 1 -- 172.21.15.12:0/2105751489 shutdown_connections 2024-03-29T11:33:42.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.036+0000 7fb65ce00700 1 -- 172.21.15.12:0/2105751489 wait complete. 2024-03-29T11:33:42.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.038+0000 7fb65ce00700 1 Processor -- start 2024-03-29T11:33:42.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.038+0000 7fb65ce00700 1 -- start start 2024-03-29T11:33:42.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.039+0000 7fb65ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb63c004850 0x7fb65815a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.039+0000 7fb65ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6581e77e0 0x7fb65815e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.039+0000 7fb65ce00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb65815eea0 0x7fb65815f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.039+0000 7fb65ce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65815f890 con 0x7fb6581e77e0 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.039+0000 7fb65ce00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581f35f0 con 0x7fb65815eea0 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.039+0000 7fb65ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581e7ce0 con 0x7fb63c004850 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.039+0000 7fb655400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6581e77e0 0x7fb65815e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.039+0000 7fb655400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6581e77e0 0x7fb65815e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56472/0 (socket says 172.21.15.12:56472) 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.039+0000 7fb655400700 1 -- 172.21.15.12:0/2188884379 learned_addr learned my addr 172.21.15.12:0/2188884379 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb656800700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb65815eea0 0x7fb65815f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb655e00700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb63c004850 0x7fb65815a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb655400700 1 -- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb63c004850 msgr2=0x7fb65815a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb655400700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb63c004850 0x7fb65815a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb655400700 1 -- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb65815eea0 msgr2=0x7fb65815f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb655400700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb65815eea0 0x7fb65815f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb655400700 1 -- 172.21.15.12:0/2188884379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6581e7fc0 con 0x7fb6581e77e0 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb656800700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb65815eea0 0x7fb65815f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb655400700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6581e77e0 0x7fb65815e580 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fb64400c3c0 tx=0x7fb644001310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb64e000700 1 -- 172.21.15.12:0/2188884379 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644001110 con 0x7fb6581e77e0 2024-03-29T11:33:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb64e000700 1 -- 172.21.15.12:0/2188884379 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb644001410 con 0x7fb6581e77e0 2024-03-29T11:33:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb64e000700 1 -- 172.21.15.12:0/2188884379 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644007280 con 0x7fb6581e77e0 2024-03-29T11:33:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.040+0000 7fb65ce00700 1 -- 172.21.15.12:0/2188884379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6581e82a0 con 0x7fb6581e77e0 2024-03-29T11:33:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.041+0000 7fb65ce00700 1 -- 172.21.15.12:0/2188884379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65815eac0 con 0x7fb6581e77e0 2024-03-29T11:33:42.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.042+0000 7fb65ce00700 1 -- 172.21.15.12:0/2188884379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6581e87c0 con 0x7fb6581e77e0 2024-03-29T11:33:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.043+0000 7fb64e000700 1 -- 172.21.15.12:0/2188884379 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb6440073e0 con 0x7fb6581e77e0 2024-03-29T11:33:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.044+0000 7fb64e000700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb63806aed0 0x7fb63806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.044+0000 7fb64e000700 1 -- 172.21.15.12:0/2188884379 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fb64408a7e0 con 0x7fb6581e77e0 2024-03-29T11:33:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.044+0000 7fb655e00700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb63806aed0 0x7fb63806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.045+0000 7fb655e00700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb63806aed0 0x7fb63806d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb648001200 tx=0x7fb648009040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:42.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.048+0000 7fb64e000700 1 -- 172.21.15.12:0/2188884379 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb6440556d0 con 0x7fb6581e77e0 2024-03-29T11:33:42.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.258+0000 7fb65ce00700 1 -- 172.21.15.12:0/2188884379 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6580ce120 con 0x7fb63806aed0 2024-03-29T11:33:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.267+0000 7fb64e000700 1 -- 172.21.15.12:0/2188884379 <== mgr.11755 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289361 (secure 0 0 0) 0x7fb648004dc0 con 0x7fb63806aed0 2024-03-29T11:33:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 -- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb63806aed0 msgr2=0x7fb63806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb63806aed0 0x7fb63806d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb648001200 tx=0x7fb648009040 comp rx=0 tx=0).stop 2024-03-29T11:33:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 -- 172.21.15.12:0/2188884379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6581e77e0 msgr2=0x7fb65815e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6581e77e0 0x7fb65815e580 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fb64400c3c0 tx=0x7fb644001310 comp rx=0 tx=0).stop 2024-03-29T11:33:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 -- 172.21.15.12:0/2188884379 shutdown_connections 2024-03-29T11:33:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb63806aed0 0x7fb63806d390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb65815eea0 0x7fb65815f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb63c004850 0x7fb65815a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 --2- 172.21.15.12:0/2188884379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6581e77e0 0x7fb65815e580 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 -- 172.21.15.12:0/2188884379 >> 172.21.15.12:0/2188884379 conn(0x7fb65808aff0 msgr2=0x7fb65815bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:42.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 -- 172.21.15.12:0/2188884379 shutdown_connections 2024-03-29T11:33:42.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.272+0000 7fb65ce00700 1 -- 172.21.15.12:0/2188884379 wait complete. 2024-03-29T11:33:42.271 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:33:42.286 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-29T11:33:42.088294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2672,"num_read_kb":5632,"num_write":735,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":448,"ondisk_log_size":448,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":455368,"kb_used_data":27652,"kb_used_omap":12,"kb_used_meta":427571,"kb_avail":754519352,"statfs":{"total":773094113280,"available":772627816448,"internally_reserved":0,"allocated":28315648,"data_stored":14522137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":437833273},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001428"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899776+0000","last_change":"2024-03-29T11:33:36.893003+0000","last_active":"2024-03-29T11:33:36.899776+0000","last_peered":"2024-03-29T11:33:36.899776+0000","last_clean":"2024-03-29T11:33:36.899776+0000","last_became_active":"2024-03-29T11:33:36.892417+0000","last_became_peered":"2024-03-29T11:33:36.892417+0000","last_unstale":"2024-03-29T11:33:36.899776+0000","last_undegraded":"2024-03-29T11:33:36.899776+0000","last_fullsized":"2024-03-29T11:33:36.899776+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:32:46.474810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.900205+0000","last_change":"2024-03-29T11:33:36.898436+0000","last_active":"2024-03-29T11:33:36.900205+0000","last_peered":"2024-03-29T11:33:36.900205+0000","last_clean":"2024-03-29T11:33:36.900205+0000","last_became_active":"2024-03-29T11:33:36.890680+0000","last_became_peered":"2024-03-29T11:33:36.890680+0000","last_unstale":"2024-03-29T11:33:36.900205+0000","last_undegraded":"2024-03-29T11:33:36.900205+0000","last_fullsized":"2024-03-29T11:33:36.900205+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:44:07.664618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.881207+0000","last_change":"2024-03-29T11:33:36.879984+0000","last_active":"2024-03-29T11:33:36.881207+0000","last_peered":"2024-03-29T11:33:36.881207+0000","last_clean":"2024-03-29T11:33:36.881207+0000","last_became_active":"2024-03-29T11:33:36.879646+0000","last_became_peered":"2024-03-29T11:33:36.879646+0000","last_unstale":"2024-03-29T11:33:36.881207+0000","last_undegraded":"2024-03-29T11:33:36.881207+0000","last_fullsized":"2024-03-29T11:33:36.881207+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:01:52.706981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.902858+0000","last_change":"2024-03-29T11:33:36.895719+0000","last_active":"2024-03-29T11:33:36.902858+0000","last_peered":"2024-03-29T11:33:36.902858+0000","last_clean":"2024-03-29T11:33:36.902858+0000","last_became_active":"2024-03-29T11:33:36.895441+0000","last_became_peered":"2024-03-29T11:33:36.895441+0000","last_unstale":"2024-03-29T11:33:36.902858+0000","last_undegraded":"2024-03-29T11:33:36.902858+0000","last_fullsized":"2024-03-29T11:33:36.902858+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:32.153462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899513+0000","last_change":"2024-03-29T11:33:36.896380+0000","last_active":"2024-03-29T11:33:36.899513+0000","last_peered":"2024-03-29T11:33:36.899513+0000","last_clean":"2024-03-29T11:33:36.899513+0000","last_became_active":"2024-03-29T11:33:36.890321+0000","last_became_peered":"2024-03-29T11:33:36.890321+0000","last_unstale":"2024-03-29T11:33:36.899513+0000","last_undegraded":"2024-03-29T11:33:36.899513+0000","last_fullsized":"2024-03-29T11:33:36.899513+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:33:34.257790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.894232+0000","last_change":"2024-03-29T11:33:36.889542+0000","last_active":"2024-03-29T11:33:36.894232+0000","last_peered":"2024-03-29T11:33:36.894232+0000","last_clean":"2024-03-29T11:33:36.894232+0000","last_became_active":"2024-03-29T11:33:36.889207+0000","last_became_peered":"2024-03-29T11:33:36.889207+0000","last_unstale":"2024-03-29T11:33:36.894232+0000","last_undegraded":"2024-03-29T11:33:36.894232+0000","last_fullsized":"2024-03-29T11:33:36.894232+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:48:58.214029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874299+0000","last_change":"2024-03-29T11:33:36.895133+0000","last_active":"2024-03-29T11:33:37.874299+0000","last_peered":"2024-03-29T11:33:37.874299+0000","last_clean":"2024-03-29T11:33:37.874299+0000","last_became_active":"2024-03-29T11:33:36.894552+0000","last_became_peered":"2024-03-29T11:33:36.894552+0000","last_unstale":"2024-03-29T11:33:37.874299+0000","last_undegraded":"2024-03-29T11:33:37.874299+0000","last_fullsized":"2024-03-29T11:33:37.874299+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:50:29.041184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874140+0000","last_change":"2024-03-29T11:33:36.881152+0000","last_active":"2024-03-29T11:33:37.874140+0000","last_peered":"2024-03-29T11:33:37.874140+0000","last_clean":"2024-03-29T11:33:37.874140+0000","last_became_active":"2024-03-29T11:33:36.880885+0000","last_became_peered":"2024-03-29T11:33:36.880885+0000","last_unstale":"2024-03-29T11:33:37.874140+0000","last_undegraded":"2024-03-29T11:33:37.874140+0000","last_fullsized":"2024-03-29T11:33:37.874140+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:26.859189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899545+0000","last_change":"2024-03-29T11:33:36.898304+0000","last_active":"2024-03-29T11:33:36.899545+0000","last_peered":"2024-03-29T11:33:36.899545+0000","last_clean":"2024-03-29T11:33:36.899545+0000","last_became_active":"2024-03-29T11:33:36.897344+0000","last_became_peered":"2024-03-29T11:33:36.897344+0000","last_unstale":"2024-03-29T11:33:36.899545+0000","last_undegraded":"2024-03-29T11:33:36.899545+0000","last_fullsized":"2024-03-29T11:33:36.899545+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:08:12.505762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.895702+0000","last_change":"2024-03-29T11:33:36.893590+0000","last_active":"2024-03-29T11:33:36.895702+0000","last_peered":"2024-03-29T11:33:36.895702+0000","last_clean":"2024-03-29T11:33:36.895702+0000","last_became_active":"2024-03-29T11:33:36.893341+0000","last_became_peered":"2024-03-29T11:33:36.893341+0000","last_unstale":"2024-03-29T11:33:36.895702+0000","last_undegraded":"2024-03-29T11:33:36.895702+0000","last_fullsized":"2024-03-29T11:33:36.895702+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:12:15.073274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.894162+0000","last_change":"2024-03-29T11:33:36.886094+0000","last_active":"2024-03-29T11:33:36.894162+0000","last_peered":"2024-03-29T11:33:36.894162+0000","last_clean":"2024-03-29T11:33:36.894162+0000","last_became_active":"2024-03-29T11:33:36.885781+0000","last_became_peered":"2024-03-29T11:33:36.885781+0000","last_unstale":"2024-03-29T11:33:36.894162+0000","last_undegraded":"2024-03-29T11:33:36.894162+0000","last_fullsized":"2024-03-29T11:33:36.894162+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:30:28.396619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.878893+0000","last_change":"2024-03-29T11:33:36.877933+0000","last_active":"2024-03-29T11:33:36.878893+0000","last_peered":"2024-03-29T11:33:36.878893+0000","last_clean":"2024-03-29T11:33:36.878893+0000","last_became_active":"2024-03-29T11:33:36.877218+0000","last_became_peered":"2024-03-29T11:33:36.877218+0000","last_unstale":"2024-03-29T11:33:36.878893+0000","last_undegraded":"2024-03-29T11:33:36.878893+0000","last_fullsized":"2024-03-29T11:33:36.878893+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:26.859189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.878730+0000","last_change":"2024-03-29T11:33:36.877409+0000","last_active":"2024-03-29T11:33:36.878730+0000","last_peered":"2024-03-29T11:33:36.878730+0000","last_clean":"2024-03-29T11:33:36.878730+0000","last_became_active":"2024-03-29T11:33:36.876887+0000","last_became_peered":"2024-03-29T11:33:36.876887+0000","last_unstale":"2024-03-29T11:33:36.878730+0000","last_undegraded":"2024-03-29T11:33:36.878730+0000","last_fullsized":"2024-03-29T11:33:36.878730+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:51:00.296286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.903646+0000","last_change":"2024-03-29T11:33:36.897953+0000","last_active":"2024-03-29T11:33:36.903646+0000","last_peered":"2024-03-29T11:33:36.903646+0000","last_clean":"2024-03-29T11:33:36.903646+0000","last_became_active":"2024-03-29T11:33:36.897680+0000","last_became_peered":"2024-03-29T11:33:36.897680+0000","last_unstale":"2024-03-29T11:33:36.903646+0000","last_undegraded":"2024-03-29T11:33:36.903646+0000","last_fullsized":"2024-03-29T11:33:36.903646+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:48.820065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874406+0000","last_change":"2024-03-29T11:33:36.891334+0000","last_active":"2024-03-29T11:33:37.874406+0000","last_peered":"2024-03-29T11:33:37.874406+0000","last_clean":"2024-03-29T11:33:37.874406+0000","last_became_active":"2024-03-29T11:33:36.890976+0000","last_became_peered":"2024-03-29T11:33:36.890976+0000","last_unstale":"2024-03-29T11:33:37.874406+0000","last_undegraded":"2024-03-29T11:33:37.874406+0000","last_fullsized":"2024-03-29T11:33:37.874406+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:42:26.424892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874278+0000","last_change":"2024-03-29T11:33:36.885822+0000","last_active":"2024-03-29T11:33:37.874278+0000","last_peered":"2024-03-29T11:33:37.874278+0000","last_clean":"2024-03-29T11:33:37.874278+0000","last_became_active":"2024-03-29T11:33:36.885546+0000","last_became_peered":"2024-03-29T11:33:36.885546+0000","last_unstale":"2024-03-29T11:33:37.874278+0000","last_undegraded":"2024-03-29T11:33:37.874278+0000","last_fullsized":"2024-03-29T11:33:37.874278+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:38:26.882481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.886973+0000","last_change":"2024-03-29T11:33:36.884866+0000","last_active":"2024-03-29T11:33:36.886973+0000","last_peered":"2024-03-29T11:33:36.886973+0000","last_clean":"2024-03-29T11:33:36.886973+0000","last_became_active":"2024-03-29T11:33:36.884516+0000","last_became_peered":"2024-03-29T11:33:36.884516+0000","last_unstale":"2024-03-29T11:33:36.886973+0000","last_undegraded":"2024-03-29T11:33:36.886973+0000","last_fullsized":"2024-03-29T11:33:36.886973+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:30:28.396619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874568+0000","last_change":"2024-03-29T11:33:36.890080+0000","last_active":"2024-03-29T11:33:37.874568+0000","last_peered":"2024-03-29T11:33:37.874568+0000","last_clean":"2024-03-29T11:33:37.874568+0000","last_became_active":"2024-03-29T11:33:36.889714+0000","last_became_peered":"2024-03-29T11:33:36.889714+0000","last_unstale":"2024-03-29T11:33:37.874568+0000","last_undegraded":"2024-03-29T11:33:37.874568+0000","last_fullsized":"2024-03-29T11:33:37.874568+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.148775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875290+0000","last_change":"2024-03-29T11:33:36.898908+0000","last_active":"2024-03-29T11:33:37.875290+0000","last_peered":"2024-03-29T11:33:37.875290+0000","last_clean":"2024-03-29T11:33:37.875290+0000","last_became_active":"2024-03-29T11:33:36.897597+0000","last_became_peered":"2024-03-29T11:33:36.897597+0000","last_unstale":"2024-03-29T11:33:37.875290+0000","last_undegraded":"2024-03-29T11:33:37.875290+0000","last_fullsized":"2024-03-29T11:33:37.875290+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:41:36.893627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875111+0000","last_change":"2024-03-29T11:33:36.888681+0000","last_active":"2024-03-29T11:33:37.875111+0000","last_peered":"2024-03-29T11:33:37.875111+0000","last_clean":"2024-03-29T11:33:37.875111+0000","last_became_active":"2024-03-29T11:33:36.888383+0000","last_became_peered":"2024-03-29T11:33:36.888383+0000","last_unstale":"2024-03-29T11:33:37.875111+0000","last_undegraded":"2024-03-29T11:33:37.875111+0000","last_fullsized":"2024-03-29T11:33:37.875111+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:12.300239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.876668+0000","last_change":"2024-03-29T11:33:36.875193+0000","last_active":"2024-03-29T11:33:36.876668+0000","last_peered":"2024-03-29T11:33:36.876668+0000","last_clean":"2024-03-29T11:33:36.876668+0000","last_became_active":"2024-03-29T11:33:36.874932+0000","last_became_peered":"2024-03-29T11:33:36.874932+0000","last_unstale":"2024-03-29T11:33:36.876668+0000","last_undegraded":"2024-03-29T11:33:36.876668+0000","last_fullsized":"2024-03-29T11:33:36.876668+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:54.747726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.887091+0000","last_change":"2024-03-29T11:33:36.886531+0000","last_active":"2024-03-29T11:33:36.887091+0000","last_peered":"2024-03-29T11:33:36.887091+0000","last_clean":"2024-03-29T11:33:36.887091+0000","last_became_active":"2024-03-29T11:33:36.885653+0000","last_became_peered":"2024-03-29T11:33:36.885653+0000","last_unstale":"2024-03-29T11:33:36.887091+0000","last_undegraded":"2024-03-29T11:33:36.887091+0000","last_fullsized":"2024-03-29T11:33:36.887091+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:10.981880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872834+0000","last_change":"2024-03-29T11:33:36.887478+0000","last_active":"2024-03-29T11:33:37.872834+0000","last_peered":"2024-03-29T11:33:37.872834+0000","last_clean":"2024-03-29T11:33:37.872834+0000","last_became_active":"2024-03-29T11:33:36.886802+0000","last_became_peered":"2024-03-29T11:33:36.886802+0000","last_unstale":"2024-03-29T11:33:37.872834+0000","last_undegraded":"2024-03-29T11:33:37.872834+0000","last_fullsized":"2024-03-29T11:33:37.872834+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:41.212920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.882768+0000","last_change":"2024-03-29T11:33:36.881358+0000","last_active":"2024-03-29T11:33:36.882768+0000","last_peered":"2024-03-29T11:33:36.882768+0000","last_clean":"2024-03-29T11:33:36.882768+0000","last_became_active":"2024-03-29T11:33:36.881103+0000","last_became_peered":"2024-03-29T11:33:36.881103+0000","last_unstale":"2024-03-29T11:33:36.882768+0000","last_undegraded":"2024-03-29T11:33:36.882768+0000","last_fullsized":"2024-03-29T11:33:36.882768+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:16:12.727333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872862+0000","last_change":"2024-03-29T11:33:36.889158+0000","last_active":"2024-03-29T11:33:37.872862+0000","last_peered":"2024-03-29T11:33:37.872862+0000","last_clean":"2024-03-29T11:33:37.872862+0000","last_became_active":"2024-03-29T11:33:36.888904+0000","last_became_peered":"2024-03-29T11:33:36.888904+0000","last_unstale":"2024-03-29T11:33:37.872862+0000","last_undegraded":"2024-03-29T11:33:37.872862+0000","last_fullsized":"2024-03-29T11:33:37.872862+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:06.351839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.878680+0000","last_change":"2024-03-29T11:33:36.877422+0000","last_active":"2024-03-29T11:33:36.878680+0000","last_peered":"2024-03-29T11:33:36.878680+0000","last_clean":"2024-03-29T11:33:36.878680+0000","last_became_active":"2024-03-29T11:33:36.877136+0000","last_became_peered":"2024-03-29T11:33:36.877136+0000","last_unstale":"2024-03-29T11:33:36.878680+0000","last_undegraded":"2024-03-29T11:33:36.878680+0000","last_fullsized":"2024-03-29T11:33:36.878680+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:08:12.505762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875315+0000","last_change":"2024-03-29T11:33:36.886686+0000","last_active":"2024-03-29T11:33:37.875315+0000","last_peered":"2024-03-29T11:33:37.875315+0000","last_clean":"2024-03-29T11:33:37.875315+0000","last_became_active":"2024-03-29T11:33:36.886424+0000","last_became_peered":"2024-03-29T11:33:36.886424+0000","last_unstale":"2024-03-29T11:33:37.875315+0000","last_undegraded":"2024-03-29T11:33:37.875315+0000","last_fullsized":"2024-03-29T11:33:37.875315+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:39:42.295243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.902879+0000","last_change":"2024-03-29T11:33:36.897242+0000","last_active":"2024-03-29T11:33:36.902879+0000","last_peered":"2024-03-29T11:33:36.902879+0000","last_clean":"2024-03-29T11:33:36.902879+0000","last_became_active":"2024-03-29T11:33:36.896949+0000","last_became_peered":"2024-03-29T11:33:36.896949+0000","last_unstale":"2024-03-29T11:33:36.902879+0000","last_undegraded":"2024-03-29T11:33:36.902879+0000","last_fullsized":"2024-03-29T11:33:36.902879+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:06.950427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875002+0000","last_change":"2024-03-29T11:33:36.894512+0000","last_active":"2024-03-29T11:33:37.875002+0000","last_peered":"2024-03-29T11:33:37.875002+0000","last_clean":"2024-03-29T11:33:37.875002+0000","last_became_active":"2024-03-29T11:33:36.894200+0000","last_became_peered":"2024-03-29T11:33:36.894200+0000","last_unstale":"2024-03-29T11:33:37.875002+0000","last_undegraded":"2024-03-29T11:33:37.875002+0000","last_fullsized":"2024-03-29T11:33:37.875002+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:54:45.926420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874508+0000","last_change":"2024-03-29T11:33:36.895304+0000","last_active":"2024-03-29T11:33:37.874508+0000","last_peered":"2024-03-29T11:33:37.874508+0000","last_clean":"2024-03-29T11:33:37.874508+0000","last_became_active":"2024-03-29T11:33:36.894890+0000","last_became_peered":"2024-03-29T11:33:36.894890+0000","last_unstale":"2024-03-29T11:33:37.874508+0000","last_undegraded":"2024-03-29T11:33:37.874508+0000","last_fullsized":"2024-03-29T11:33:37.874508+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:01:54.782259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.895081+0000","last_change":"2024-03-29T11:33:36.892935+0000","last_active":"2024-03-29T11:33:36.895081+0000","last_peered":"2024-03-29T11:33:36.895081+0000","last_clean":"2024-03-29T11:33:36.895081+0000","last_became_active":"2024-03-29T11:33:36.892353+0000","last_became_peered":"2024-03-29T11:33:36.892353+0000","last_unstale":"2024-03-29T11:33:36.895081+0000","last_undegraded":"2024-03-29T11:33:36.895081+0000","last_fullsized":"2024-03-29T11:33:36.895081+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:54.747726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.886348+0000","last_change":"2024-03-29T11:33:36.885141+0000","last_active":"2024-03-29T11:33:36.886348+0000","last_peered":"2024-03-29T11:33:36.886348+0000","last_clean":"2024-03-29T11:33:36.886348+0000","last_became_active":"2024-03-29T11:33:36.884840+0000","last_became_peered":"2024-03-29T11:33:36.884840+0000","last_unstale":"2024-03-29T11:33:36.886348+0000","last_undegraded":"2024-03-29T11:33:36.886348+0000","last_fullsized":"2024-03-29T11:33:36.886348+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:15.309041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"268'4","reported_seq":14,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.900582+0000","last_change":"2024-03-29T11:33:36.357638+0000","last_active":"2024-03-29T11:33:36.900582+0000","last_peered":"2024-03-29T11:33:36.900582+0000","last_clean":"2024-03-29T11:33:36.900582+0000","last_became_active":"2024-03-29T11:33:36.357370+0000","last_became_peered":"2024-03-29T11:33:36.357370+0000","last_unstale":"2024-03-29T11:33:36.900582+0000","last_undegraded":"2024-03-29T11:33:36.900582+0000","last_fullsized":"2024-03-29T11:33:36.900582+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:05.348804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.903426+0000","last_change":"2024-03-29T11:33:36.901731+0000","last_active":"2024-03-29T11:33:36.903426+0000","last_peered":"2024-03-29T11:33:36.903426+0000","last_clean":"2024-03-29T11:33:36.903426+0000","last_became_active":"2024-03-29T11:33:36.901150+0000","last_became_peered":"2024-03-29T11:33:36.901150+0000","last_unstale":"2024-03-29T11:33:36.903426+0000","last_undegraded":"2024-03-29T11:33:36.903426+0000","last_fullsized":"2024-03-29T11:33:36.903426+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:54.747726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.900612+0000","last_change":"2024-03-29T11:33:36.365389+0000","last_active":"2024-03-29T11:33:36.900612+0000","last_peered":"2024-03-29T11:33:36.900612+0000","last_clean":"2024-03-29T11:33:36.900612+0000","last_became_active":"2024-03-29T11:33:36.364763+0000","last_became_peered":"2024-03-29T11:33:36.364763+0000","last_unstale":"2024-03-29T11:33:36.900612+0000","last_undegraded":"2024-03-29T11:33:36.900612+0000","last_fullsized":"2024-03-29T11:33:36.900612+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:34:24.861299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.900648+0000","last_change":"2024-03-29T11:33:36.899206+0000","last_active":"2024-03-29T11:33:36.900648+0000","last_peered":"2024-03-29T11:33:36.900648+0000","last_clean":"2024-03-29T11:33:36.900648+0000","last_became_active":"2024-03-29T11:33:36.898932+0000","last_became_peered":"2024-03-29T11:33:36.898932+0000","last_unstale":"2024-03-29T11:33:36.900648+0000","last_undegraded":"2024-03-29T11:33:36.900648+0000","last_fullsized":"2024-03-29T11:33:36.900648+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:19:25.689275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.904867+0000","last_change":"2024-03-29T11:33:36.369096+0000","last_active":"2024-03-29T11:33:36.904867+0000","last_peered":"2024-03-29T11:33:36.904867+0000","last_clean":"2024-03-29T11:33:36.904867+0000","last_became_active":"2024-03-29T11:33:36.367885+0000","last_became_peered":"2024-03-29T11:33:36.367885+0000","last_unstale":"2024-03-29T11:33:36.904867+0000","last_undegraded":"2024-03-29T11:33:36.904867+0000","last_fullsized":"2024-03-29T11:33:36.904867+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:05.348804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.901004+0000","last_change":"2024-03-29T11:33:36.896791+0000","last_active":"2024-03-29T11:33:36.901004+0000","last_peered":"2024-03-29T11:33:36.901004+0000","last_clean":"2024-03-29T11:33:36.901004+0000","last_became_active":"2024-03-29T11:33:36.896502+0000","last_became_peered":"2024-03-29T11:33:36.896502+0000","last_unstale":"2024-03-29T11:33:36.901004+0000","last_undegraded":"2024-03-29T11:33:36.901004+0000","last_fullsized":"2024-03-29T11:33:36.901004+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:53.267539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"268'2","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875253+0000","last_change":"2024-03-29T11:33:36.364750+0000","last_active":"2024-03-29T11:33:37.875253+0000","last_peered":"2024-03-29T11:33:37.875253+0000","last_clean":"2024-03-29T11:33:37.875253+0000","last_became_active":"2024-03-29T11:33:36.364458+0000","last_became_peered":"2024-03-29T11:33:36.364458+0000","last_unstale":"2024-03-29T11:33:37.875253+0000","last_undegraded":"2024-03-29T11:33:37.875253+0000","last_fullsized":"2024-03-29T11:33:37.875253+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:05.348804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872671+0000","last_change":"2024-03-29T11:33:36.889260+0000","last_active":"2024-03-29T11:33:37.872671+0000","last_peered":"2024-03-29T11:33:37.872671+0000","last_clean":"2024-03-29T11:33:37.872671+0000","last_became_active":"2024-03-29T11:33:36.889003+0000","last_became_peered":"2024-03-29T11:33:36.889003+0000","last_unstale":"2024-03-29T11:33:37.872671+0000","last_undegraded":"2024-03-29T11:33:37.872671+0000","last_fullsized":"2024-03-29T11:33:37.872671+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:34:25.864333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.901515+0000","last_change":"2024-03-29T11:33:36.363922+0000","last_active":"2024-03-29T11:33:36.901515+0000","last_peered":"2024-03-29T11:33:36.901515+0000","last_clean":"2024-03-29T11:33:36.901515+0000","last_became_active":"2024-03-29T11:33:36.362834+0000","last_became_peered":"2024-03-29T11:33:36.362834+0000","last_unstale":"2024-03-29T11:33:36.901515+0000","last_undegraded":"2024-03-29T11:33:36.901515+0000","last_fullsized":"2024-03-29T11:33:36.901515+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:53:32.918021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875050+0000","last_change":"2024-03-29T11:33:36.886469+0000","last_active":"2024-03-29T11:33:37.875050+0000","last_peered":"2024-03-29T11:33:37.875050+0000","last_clean":"2024-03-29T11:33:37.875050+0000","last_became_active":"2024-03-29T11:33:36.885981+0000","last_became_peered":"2024-03-29T11:33:36.885981+0000","last_unstale":"2024-03-29T11:33:37.875050+0000","last_undegraded":"2024-03-29T11:33:37.875050+0000","last_fullsized":"2024-03-29T11:33:37.875050+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:50:51.040464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"268'1","reported_seq":12,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874132+0000","last_change":"2024-03-29T11:33:36.364399+0000","last_active":"2024-03-29T11:33:37.874132+0000","last_peered":"2024-03-29T11:33:37.874132+0000","last_clean":"2024-03-29T11:33:37.874132+0000","last_became_active":"2024-03-29T11:33:36.363301+0000","last_became_peered":"2024-03-29T11:33:36.363301+0000","last_unstale":"2024-03-29T11:33:37.874132+0000","last_undegraded":"2024-03-29T11:33:37.874132+0000","last_fullsized":"2024-03-29T11:33:37.874132+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:48:05.424999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.897622+0000","last_change":"2024-03-29T11:33:36.895551+0000","last_active":"2024-03-29T11:33:36.897622+0000","last_peered":"2024-03-29T11:33:36.897622+0000","last_clean":"2024-03-29T11:33:36.897622+0000","last_became_active":"2024-03-29T11:33:36.894886+0000","last_became_peered":"2024-03-29T11:33:36.894886+0000","last_unstale":"2024-03-29T11:33:36.897622+0000","last_undegraded":"2024-03-29T11:33:36.897622+0000","last_fullsized":"2024-03-29T11:33:36.897622+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:51:00.296286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.867066+0000","last_change":"2024-03-29T11:33:36.361389+0000","last_active":"2024-03-29T11:33:36.867066+0000","last_peered":"2024-03-29T11:33:36.867066+0000","last_clean":"2024-03-29T11:33:36.867066+0000","last_became_active":"2024-03-29T11:33:36.361132+0000","last_became_peered":"2024-03-29T11:33:36.361132+0000","last_unstale":"2024-03-29T11:33:36.867066+0000","last_undegraded":"2024-03-29T11:33:36.867066+0000","last_fullsized":"2024-03-29T11:33:36.867066+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:23:36.899065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.873998+0000","last_change":"2024-03-29T11:33:36.881385+0000","last_active":"2024-03-29T11:33:37.873998+0000","last_peered":"2024-03-29T11:33:37.873998+0000","last_clean":"2024-03-29T11:33:37.873998+0000","last_became_active":"2024-03-29T11:33:36.881111+0000","last_became_peered":"2024-03-29T11:33:36.881111+0000","last_unstale":"2024-03-29T11:33:37.873998+0000","last_undegraded":"2024-03-29T11:33:37.873998+0000","last_fullsized":"2024-03-29T11:33:37.873998+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:32:28.625576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.866965+0000","last_change":"2024-03-29T11:33:36.362972+0000","last_active":"2024-03-29T11:33:36.866965+0000","last_peered":"2024-03-29T11:33:36.866965+0000","last_clean":"2024-03-29T11:33:36.866965+0000","last_became_active":"2024-03-29T11:33:36.362704+0000","last_became_peered":"2024-03-29T11:33:36.362704+0000","last_unstale":"2024-03-29T11:33:36.866965+0000","last_undegraded":"2024-03-29T11:33:36.866965+0000","last_fullsized":"2024-03-29T11:33:36.866965+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:47:26.437713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874666+0000","last_change":"2024-03-29T11:33:36.883856+0000","last_active":"2024-03-29T11:33:37.874666+0000","last_peered":"2024-03-29T11:33:37.874666+0000","last_clean":"2024-03-29T11:33:37.874666+0000","last_became_active":"2024-03-29T11:33:36.883495+0000","last_became_peered":"2024-03-29T11:33:36.883495+0000","last_unstale":"2024-03-29T11:33:37.874666+0000","last_undegraded":"2024-03-29T11:33:37.874666+0000","last_fullsized":"2024-03-29T11:33:37.874666+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:53.412422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.887337+0000","last_change":"2024-03-29T11:33:36.365618+0000","last_active":"2024-03-29T11:33:36.887337+0000","last_peered":"2024-03-29T11:33:36.887337+0000","last_clean":"2024-03-29T11:33:36.887337+0000","last_became_active":"2024-03-29T11:33:36.365220+0000","last_became_peered":"2024-03-29T11:33:36.365220+0000","last_unstale":"2024-03-29T11:33:36.887337+0000","last_undegraded":"2024-03-29T11:33:36.887337+0000","last_fullsized":"2024-03-29T11:33:36.887337+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:57:10.581820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.904486+0000","last_change":"2024-03-29T11:33:36.902403+0000","last_active":"2024-03-29T11:33:36.904486+0000","last_peered":"2024-03-29T11:33:36.904486+0000","last_clean":"2024-03-29T11:33:36.904486+0000","last_became_active":"2024-03-29T11:33:36.899052+0000","last_became_peered":"2024-03-29T11:33:36.899052+0000","last_unstale":"2024-03-29T11:33:36.904486+0000","last_undegraded":"2024-03-29T11:33:36.904486+0000","last_fullsized":"2024-03-29T11:33:36.904486+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:04:50.012882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.876661+0000","last_change":"2024-03-29T11:33:36.369550+0000","last_active":"2024-03-29T11:33:36.876661+0000","last_peered":"2024-03-29T11:33:36.876661+0000","last_clean":"2024-03-29T11:33:36.876661+0000","last_became_active":"2024-03-29T11:33:36.369264+0000","last_became_peered":"2024-03-29T11:33:36.369264+0000","last_unstale":"2024-03-29T11:33:36.876661+0000","last_undegraded":"2024-03-29T11:33:36.876661+0000","last_fullsized":"2024-03-29T11:33:36.876661+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:40.209886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.902596+0000","last_change":"2024-03-29T11:33:36.901191+0000","last_active":"2024-03-29T11:33:36.902596+0000","last_peered":"2024-03-29T11:33:36.902596+0000","last_clean":"2024-03-29T11:33:36.902596+0000","last_became_active":"2024-03-29T11:33:36.899138+0000","last_became_peered":"2024-03-29T11:33:36.899138+0000","last_unstale":"2024-03-29T11:33:36.902596+0000","last_undegraded":"2024-03-29T11:33:36.902596+0000","last_fullsized":"2024-03-29T11:33:36.902596+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:51:00.296286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"268'3","reported_seq":13,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.902055+0000","last_change":"2024-03-29T11:33:36.368872+0000","last_active":"2024-03-29T11:33:36.902055+0000","last_peered":"2024-03-29T11:33:36.902055+0000","last_clean":"2024-03-29T11:33:36.902055+0000","last_became_active":"2024-03-29T11:33:36.367483+0000","last_became_peered":"2024-03-29T11:33:36.367483+0000","last_unstale":"2024-03-29T11:33:36.902055+0000","last_undegraded":"2024-03-29T11:33:36.902055+0000","last_fullsized":"2024-03-29T11:33:36.902055+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:47:26.437713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.895837+0000","last_change":"2024-03-29T11:33:36.893116+0000","last_active":"2024-03-29T11:33:36.895837+0000","last_peered":"2024-03-29T11:33:36.895837+0000","last_clean":"2024-03-29T11:33:36.895837+0000","last_became_active":"2024-03-29T11:33:36.892849+0000","last_became_peered":"2024-03-29T11:33:36.892849+0000","last_unstale":"2024-03-29T11:33:36.895837+0000","last_undegraded":"2024-03-29T11:33:36.895837+0000","last_fullsized":"2024-03-29T11:33:36.895837+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:02:27.050159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"268'6","reported_seq":16,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.905396+0000","last_change":"2024-03-29T11:33:36.359966+0000","last_active":"2024-03-29T11:33:36.905396+0000","last_peered":"2024-03-29T11:33:36.905396+0000","last_clean":"2024-03-29T11:33:36.905396+0000","last_became_active":"2024-03-29T11:33:36.359267+0000","last_became_peered":"2024-03-29T11:33:36.359267+0000","last_unstale":"2024-03-29T11:33:36.905396+0000","last_undegraded":"2024-03-29T11:33:36.905396+0000","last_fullsized":"2024-03-29T11:33:36.905396+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:02:21.625250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874722+0000","last_change":"2024-03-29T11:33:36.880389+0000","last_active":"2024-03-29T11:33:37.874722+0000","last_peered":"2024-03-29T11:33:37.874722+0000","last_clean":"2024-03-29T11:33:37.874722+0000","last_became_active":"2024-03-29T11:33:36.879945+0000","last_became_peered":"2024-03-29T11:33:36.879945+0000","last_unstale":"2024-03-29T11:33:37.874722+0000","last_undegraded":"2024-03-29T11:33:37.874722+0000","last_fullsized":"2024-03-29T11:33:37.874722+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:07:15.309041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"268'2","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872900+0000","last_change":"2024-03-29T11:33:36.362533+0000","last_active":"2024-03-29T11:33:37.872900+0000","last_peered":"2024-03-29T11:33:37.872900+0000","last_clean":"2024-03-29T11:33:37.872900+0000","last_became_active":"2024-03-29T11:33:36.361794+0000","last_became_peered":"2024-03-29T11:33:36.361794+0000","last_unstale":"2024-03-29T11:33:37.872900+0000","last_undegraded":"2024-03-29T11:33:37.872900+0000","last_fullsized":"2024-03-29T11:33:37.872900+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:04:49.009848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874088+0000","last_change":"2024-03-29T11:33:36.884480+0000","last_active":"2024-03-29T11:33:37.874088+0000","last_peered":"2024-03-29T11:33:37.874088+0000","last_clean":"2024-03-29T11:33:37.874088+0000","last_became_active":"2024-03-29T11:33:36.884202+0000","last_became_peered":"2024-03-29T11:33:36.884202+0000","last_unstale":"2024-03-29T11:33:37.874088+0000","last_undegraded":"2024-03-29T11:33:37.874088+0000","last_fullsized":"2024-03-29T11:33:37.874088+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:41:49.488639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872652+0000","last_change":"2024-03-29T11:33:36.366389+0000","last_active":"2024-03-29T11:33:37.872652+0000","last_peered":"2024-03-29T11:33:37.872652+0000","last_clean":"2024-03-29T11:33:37.872652+0000","last_became_active":"2024-03-29T11:33:36.366155+0000","last_became_peered":"2024-03-29T11:33:36.366155+0000","last_unstale":"2024-03-29T11:33:37.872652+0000","last_undegraded":"2024-03-29T11:33:37.872652+0000","last_fullsized":"2024-03-29T11:33:37.872652+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:05.947393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.901655+0000","last_change":"2024-03-29T11:33:36.897623+0000","last_active":"2024-03-29T11:33:36.901655+0000","last_peered":"2024-03-29T11:33:36.901655+0000","last_clean":"2024-03-29T11:33:36.901655+0000","last_became_active":"2024-03-29T11:33:36.897366+0000","last_became_peered":"2024-03-29T11:33:36.897366+0000","last_unstale":"2024-03-29T11:33:36.901655+0000","last_undegraded":"2024-03-29T11:33:36.901655+0000","last_fullsized":"2024-03-29T11:33:36.901655+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:43.216913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.884319+0000","last_change":"2024-03-29T11:33:36.362765+0000","last_active":"2024-03-29T11:33:36.884319+0000","last_peered":"2024-03-29T11:33:36.884319+0000","last_clean":"2024-03-29T11:33:36.884319+0000","last_became_active":"2024-03-29T11:33:36.362201+0000","last_became_peered":"2024-03-29T11:33:36.362201+0000","last_unstale":"2024-03-29T11:33:36.884319+0000","last_undegraded":"2024-03-29T11:33:36.884319+0000","last_fullsized":"2024-03-29T11:33:36.884319+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:19:11.386682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874721+0000","last_change":"2024-03-29T11:33:36.881618+0000","last_active":"2024-03-29T11:33:37.874721+0000","last_peered":"2024-03-29T11:33:37.874721+0000","last_clean":"2024-03-29T11:33:37.874721+0000","last_became_active":"2024-03-29T11:33:36.881218+0000","last_became_peered":"2024-03-29T11:33:36.881218+0000","last_unstale":"2024-03-29T11:33:37.874721+0000","last_undegraded":"2024-03-29T11:33:37.874721+0000","last_fullsized":"2024-03-29T11:33:37.874721+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:36:37.933506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"268'8","reported_seq":19,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872655+0000","last_change":"2024-03-29T11:33:36.368694+0000","last_active":"2024-03-29T11:33:37.872655+0000","last_peered":"2024-03-29T11:33:37.872655+0000","last_clean":"2024-03-29T11:33:37.872655+0000","last_became_active":"2024-03-29T11:33:36.368116+0000","last_became_peered":"2024-03-29T11:33:36.368116+0000","last_unstale":"2024-03-29T11:33:37.872655+0000","last_undegraded":"2024-03-29T11:33:37.872655+0000","last_fullsized":"2024-03-29T11:33:37.872655+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:40:31.150427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.872824+0000","last_change":"2024-03-29T11:33:36.877057+0000","last_active":"2024-03-29T11:33:37.872824+0000","last_peered":"2024-03-29T11:33:37.872824+0000","last_clean":"2024-03-29T11:33:37.872824+0000","last_became_active":"2024-03-29T11:33:36.876421+0000","last_became_peered":"2024-03-29T11:33:36.876421+0000","last_unstale":"2024-03-29T11:33:37.872824+0000","last_undegraded":"2024-03-29T11:33:37.872824+0000","last_fullsized":"2024-03-29T11:33:37.872824+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:47:27.440747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.903892+0000","last_change":"2024-03-29T11:33:36.368155+0000","last_active":"2024-03-29T11:33:36.903892+0000","last_peered":"2024-03-29T11:33:36.903892+0000","last_clean":"2024-03-29T11:33:36.903892+0000","last_became_active":"2024-03-29T11:33:36.367899+0000","last_became_peered":"2024-03-29T11:33:36.367899+0000","last_unstale":"2024-03-29T11:33:36.903892+0000","last_undegraded":"2024-03-29T11:33:36.903892+0000","last_fullsized":"2024-03-29T11:33:36.903892+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:24.145740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.886640+0000","last_change":"2024-03-29T11:33:36.885436+0000","last_active":"2024-03-29T11:33:36.886640+0000","last_peered":"2024-03-29T11:33:36.886640+0000","last_clean":"2024-03-29T11:33:36.886640+0000","last_became_active":"2024-03-29T11:33:36.885179+0000","last_became_peered":"2024-03-29T11:33:36.885179+0000","last_unstale":"2024-03-29T11:33:36.886640+0000","last_undegraded":"2024-03-29T11:33:36.886640+0000","last_fullsized":"2024-03-29T11:33:36.886640+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:27:28.295616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.904803+0000","last_change":"2024-03-29T11:33:36.364293+0000","last_active":"2024-03-29T11:33:36.904803+0000","last_peered":"2024-03-29T11:33:36.904803+0000","last_clean":"2024-03-29T11:33:36.904803+0000","last_became_active":"2024-03-29T11:33:36.364045+0000","last_became_peered":"2024-03-29T11:33:36.364045+0000","last_unstale":"2024-03-29T11:33:36.904803+0000","last_undegraded":"2024-03-29T11:33:36.904803+0000","last_fullsized":"2024-03-29T11:33:36.904803+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:05:37.709439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874706+0000","last_change":"2024-03-29T11:33:36.897121+0000","last_active":"2024-03-29T11:33:37.874706+0000","last_peered":"2024-03-29T11:33:37.874706+0000","last_clean":"2024-03-29T11:33:37.874706+0000","last_became_active":"2024-03-29T11:33:36.896843+0000","last_became_peered":"2024-03-29T11:33:36.896843+0000","last_unstale":"2024-03-29T11:33:37.874706+0000","last_undegraded":"2024-03-29T11:33:37.874706+0000","last_fullsized":"2024-03-29T11:33:37.874706+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:48.360209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.896408+0000","last_change":"2024-03-29T11:33:36.366518+0000","last_active":"2024-03-29T11:33:36.896408+0000","last_peered":"2024-03-29T11:33:36.896408+0000","last_clean":"2024-03-29T11:33:36.896408+0000","last_became_active":"2024-03-29T11:33:36.366180+0000","last_became_peered":"2024-03-29T11:33:36.366180+0000","last_unstale":"2024-03-29T11:33:36.896408+0000","last_undegraded":"2024-03-29T11:33:36.896408+0000","last_fullsized":"2024-03-29T11:33:36.896408+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:37.313964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.901023+0000","last_change":"2024-03-29T11:33:36.897048+0000","last_active":"2024-03-29T11:33:36.901023+0000","last_peered":"2024-03-29T11:33:36.901023+0000","last_clean":"2024-03-29T11:33:36.901023+0000","last_became_active":"2024-03-29T11:33:36.890367+0000","last_became_peered":"2024-03-29T11:33:36.890367+0000","last_unstale":"2024-03-29T11:33:36.901023+0000","last_undegraded":"2024-03-29T11:33:36.901023+0000","last_fullsized":"2024-03-29T11:33:36.901023+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:23:35.014690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.884995+0000","last_change":"2024-03-29T11:33:36.367121+0000","last_active":"2024-03-29T11:33:36.884995+0000","last_peered":"2024-03-29T11:33:36.884995+0000","last_clean":"2024-03-29T11:33:36.884995+0000","last_became_active":"2024-03-29T11:33:36.366822+0000","last_became_peered":"2024-03-29T11:33:36.366822+0000","last_unstale":"2024-03-29T11:33:36.884995+0000","last_undegraded":"2024-03-29T11:33:36.884995+0000","last_fullsized":"2024-03-29T11:33:36.884995+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:01:05.143658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899973+0000","last_change":"2024-03-29T11:33:36.898690+0000","last_active":"2024-03-29T11:33:36.899973+0000","last_peered":"2024-03-29T11:33:36.899973+0000","last_clean":"2024-03-29T11:33:36.899973+0000","last_became_active":"2024-03-29T11:33:36.898324+0000","last_became_peered":"2024-03-29T11:33:36.898324+0000","last_unstale":"2024-03-29T11:33:36.899973+0000","last_undegraded":"2024-03-29T11:33:36.899973+0000","last_fullsized":"2024-03-29T11:33:36.899973+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:32:40.653645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.896434+0000","last_change":"2024-03-29T11:33:36.359779+0000","last_active":"2024-03-29T11:33:36.896434+0000","last_peered":"2024-03-29T11:33:36.896434+0000","last_clean":"2024-03-29T11:33:36.896434+0000","last_became_active":"2024-03-29T11:33:36.357360+0000","last_became_peered":"2024-03-29T11:33:36.357360+0000","last_unstale":"2024-03-29T11:33:36.896434+0000","last_undegraded":"2024-03-29T11:33:36.896434+0000","last_fullsized":"2024-03-29T11:33:36.896434+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:36:51.414204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874298+0000","last_change":"2024-03-29T11:33:36.884984+0000","last_active":"2024-03-29T11:33:37.874298+0000","last_peered":"2024-03-29T11:33:37.874298+0000","last_clean":"2024-03-29T11:33:37.874298+0000","last_became_active":"2024-03-29T11:33:36.884714+0000","last_became_peered":"2024-03-29T11:33:36.884714+0000","last_unstale":"2024-03-29T11:33:37.874298+0000","last_undegraded":"2024-03-29T11:33:37.874298+0000","last_fullsized":"2024-03-29T11:33:37.874298+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:19:12.389716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.874650+0000","last_change":"2024-03-29T11:33:36.360935+0000","last_active":"2024-03-29T11:33:36.874650+0000","last_peered":"2024-03-29T11:33:36.874650+0000","last_clean":"2024-03-29T11:33:36.874650+0000","last_became_active":"2024-03-29T11:33:36.360604+0000","last_became_peered":"2024-03-29T11:33:36.360604+0000","last_unstale":"2024-03-29T11:33:36.874650+0000","last_undegraded":"2024-03-29T11:33:36.874650+0000","last_fullsized":"2024-03-29T11:33:36.874650+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:49:20.127472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874514+0000","last_change":"2024-03-29T11:33:36.883467+0000","last_active":"2024-03-29T11:33:37.874514+0000","last_peered":"2024-03-29T11:33:37.874514+0000","last_clean":"2024-03-29T11:33:37.874514+0000","last_became_active":"2024-03-29T11:33:36.883161+0000","last_became_peered":"2024-03-29T11:33:36.883161+0000","last_unstale":"2024-03-29T11:33:37.874514+0000","last_undegraded":"2024-03-29T11:33:37.874514+0000","last_fullsized":"2024-03-29T11:33:37.874514+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:02.895952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.884449+0000","last_change":"2024-03-29T11:33:36.367037+0000","last_active":"2024-03-29T11:33:36.884449+0000","last_peered":"2024-03-29T11:33:36.884449+0000","last_clean":"2024-03-29T11:33:36.884449+0000","last_became_active":"2024-03-29T11:33:36.366741+0000","last_became_peered":"2024-03-29T11:33:36.366741+0000","last_unstale":"2024-03-29T11:33:36.884449+0000","last_undegraded":"2024-03-29T11:33:36.884449+0000","last_fullsized":"2024-03-29T11:33:36.884449+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:45:47.357175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.896581+0000","last_change":"2024-03-29T11:33:36.895057+0000","last_active":"2024-03-29T11:33:36.896581+0000","last_peered":"2024-03-29T11:33:36.896581+0000","last_clean":"2024-03-29T11:33:36.896581+0000","last_became_active":"2024-03-29T11:33:36.894583+0000","last_became_peered":"2024-03-29T11:33:36.894583+0000","last_unstale":"2024-03-29T11:33:36.896581+0000","last_undegraded":"2024-03-29T11:33:36.896581+0000","last_fullsized":"2024-03-29T11:33:36.896581+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:12:07.903315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.866956+0000","last_change":"2024-03-29T11:33:36.358732+0000","last_active":"2024-03-29T11:33:36.866956+0000","last_peered":"2024-03-29T11:33:36.866956+0000","last_clean":"2024-03-29T11:33:36.866956+0000","last_became_active":"2024-03-29T11:33:36.358474+0000","last_became_peered":"2024-03-29T11:33:36.358474+0000","last_unstale":"2024-03-29T11:33:36.866956+0000","last_undegraded":"2024-03-29T11:33:36.866956+0000","last_fullsized":"2024-03-29T11:33:36.866956+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:34:24.861299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875371+0000","last_change":"2024-03-29T11:33:36.891234+0000","last_active":"2024-03-29T11:33:37.875371+0000","last_peered":"2024-03-29T11:33:37.875371+0000","last_clean":"2024-03-29T11:33:37.875371+0000","last_became_active":"2024-03-29T11:33:36.890942+0000","last_became_peered":"2024-03-29T11:33:36.890942+0000","last_unstale":"2024-03-29T11:33:37.875371+0000","last_undegraded":"2024-03-29T11:33:37.875371+0000","last_fullsized":"2024-03-29T11:33:37.875371+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:12:07.903315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.904913+0000","last_change":"2024-03-29T11:33:36.371487+0000","last_active":"2024-03-29T11:33:36.904913+0000","last_peered":"2024-03-29T11:33:36.904913+0000","last_clean":"2024-03-29T11:33:36.904913+0000","last_became_active":"2024-03-29T11:33:36.371125+0000","last_became_peered":"2024-03-29T11:33:36.371125+0000","last_unstale":"2024-03-29T11:33:36.904913+0000","last_undegraded":"2024-03-29T11:33:36.904913+0000","last_fullsized":"2024-03-29T11:33:36.904913+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:05.348804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874971+0000","last_change":"2024-03-29T11:33:36.887396+0000","last_active":"2024-03-29T11:33:37.874971+0000","last_peered":"2024-03-29T11:33:37.874971+0000","last_clean":"2024-03-29T11:33:37.874971+0000","last_became_active":"2024-03-29T11:33:36.887047+0000","last_became_peered":"2024-03-29T11:33:36.887047+0000","last_unstale":"2024-03-29T11:33:37.874971+0000","last_undegraded":"2024-03-29T11:33:37.874971+0000","last_fullsized":"2024-03-29T11:33:37.874971+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:04:50.427129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":11,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874755+0000","last_change":"2024-03-29T11:33:36.362642+0000","last_active":"2024-03-29T11:33:37.874755+0000","last_peered":"2024-03-29T11:33:37.874755+0000","last_clean":"2024-03-29T11:33:37.874755+0000","last_became_active":"2024-03-29T11:33:36.360316+0000","last_became_peered":"2024-03-29T11:33:36.360316+0000","last_unstale":"2024-03-29T11:33:37.874755+0000","last_undegraded":"2024-03-29T11:33:37.874755+0000","last_fullsized":"2024-03-29T11:33:37.874755+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:41:00.390407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875130+0000","last_change":"2024-03-29T11:33:36.888685+0000","last_active":"2024-03-29T11:33:37.875130+0000","last_peered":"2024-03-29T11:33:37.875130+0000","last_clean":"2024-03-29T11:33:37.875130+0000","last_became_active":"2024-03-29T11:33:36.888397+0000","last_became_peered":"2024-03-29T11:33:36.888397+0000","last_unstale":"2024-03-29T11:33:37.875130+0000","last_undegraded":"2024-03-29T11:33:37.875130+0000","last_fullsized":"2024-03-29T11:33:37.875130+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:40:52.609495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.876889+0000","last_change":"2024-03-29T11:33:36.370644+0000","last_active":"2024-03-29T11:33:36.876889+0000","last_peered":"2024-03-29T11:33:36.876889+0000","last_clean":"2024-03-29T11:33:36.876889+0000","last_became_active":"2024-03-29T11:33:36.370302+0000","last_became_peered":"2024-03-29T11:33:36.370302+0000","last_unstale":"2024-03-29T11:33:36.876889+0000","last_undegraded":"2024-03-29T11:33:36.876889+0000","last_fullsized":"2024-03-29T11:33:36.876889+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:34:24.861299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899519+0000","last_change":"2024-03-29T11:33:36.896213+0000","last_active":"2024-03-29T11:33:36.899519+0000","last_peered":"2024-03-29T11:33:36.899519+0000","last_clean":"2024-03-29T11:33:36.899519+0000","last_became_active":"2024-03-29T11:33:36.895319+0000","last_became_peered":"2024-03-29T11:33:36.895319+0000","last_unstale":"2024-03-29T11:33:36.899519+0000","last_undegraded":"2024-03-29T11:33:36.899519+0000","last_fullsized":"2024-03-29T11:33:36.899519+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:51:59.814486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.881508+0000","last_change":"2024-03-29T11:33:36.370316+0000","last_active":"2024-03-29T11:33:36.881508+0000","last_peered":"2024-03-29T11:33:36.881508+0000","last_clean":"2024-03-29T11:33:36.881508+0000","last_became_active":"2024-03-29T11:33:36.369765+0000","last_became_peered":"2024-03-29T11:33:36.369765+0000","last_unstale":"2024-03-29T11:33:36.881508+0000","last_undegraded":"2024-03-29T11:33:36.881508+0000","last_fullsized":"2024-03-29T11:33:36.881508+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:47:26.437713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899447+0000","last_change":"2024-03-29T11:33:36.898119+0000","last_active":"2024-03-29T11:33:36.899447+0000","last_peered":"2024-03-29T11:33:36.899447+0000","last_clean":"2024-03-29T11:33:36.899447+0000","last_became_active":"2024-03-29T11:33:36.897865+0000","last_became_peered":"2024-03-29T11:33:36.897865+0000","last_unstale":"2024-03-29T11:33:36.899447+0000","last_undegraded":"2024-03-29T11:33:36.899447+0000","last_fullsized":"2024-03-29T11:33:36.899447+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:06:43.216913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.875107+0000","last_change":"2024-03-29T11:33:36.361285+0000","last_active":"2024-03-29T11:33:37.875107+0000","last_peered":"2024-03-29T11:33:37.875107+0000","last_clean":"2024-03-29T11:33:37.875107+0000","last_became_active":"2024-03-29T11:33:36.360938+0000","last_became_peered":"2024-03-29T11:33:36.360938+0000","last_unstale":"2024-03-29T11:33:37.875107+0000","last_undegraded":"2024-03-29T11:33:37.875107+0000","last_fullsized":"2024-03-29T11:33:37.875107+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:40.209886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874424+0000","last_change":"2024-03-29T11:33:36.883698+0000","last_active":"2024-03-29T11:33:37.874424+0000","last_peered":"2024-03-29T11:33:37.874424+0000","last_clean":"2024-03-29T11:33:37.874424+0000","last_became_active":"2024-03-29T11:33:36.883417+0000","last_became_peered":"2024-03-29T11:33:36.883417+0000","last_unstale":"2024-03-29T11:33:37.874424+0000","last_undegraded":"2024-03-29T11:33:37.874424+0000","last_fullsized":"2024-03-29T11:33:37.874424+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:01:06.146692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.867059+0000","last_change":"2024-03-29T11:33:36.359437+0000","last_active":"2024-03-29T11:33:36.867059+0000","last_peered":"2024-03-29T11:33:36.867059+0000","last_clean":"2024-03-29T11:33:36.867059+0000","last_became_active":"2024-03-29T11:33:36.359103+0000","last_became_peered":"2024-03-29T11:33:36.359103+0000","last_unstale":"2024-03-29T11:33:36.867059+0000","last_undegraded":"2024-03-29T11:33:36.867059+0000","last_fullsized":"2024-03-29T11:33:36.867059+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:36:40.209886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":10,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-29T11:33:37.874825+0000","last_change":"2024-03-29T11:33:36.879276+0000","last_active":"2024-03-29T11:33:37.874825+0000","last_peered":"2024-03-29T11:33:37.874825+0000","last_clean":"2024-03-29T11:33:37.874825+0000","last_became_active":"2024-03-29T11:33:36.878990+0000","last_became_peered":"2024-03-29T11:33:36.878990+0000","last_unstale":"2024-03-29T11:33:37.874825+0000","last_undegraded":"2024-03-29T11:33:37.874825+0000","last_fullsized":"2024-03-29T11:33:37.874825+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:11:20.442172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.867161+0000","last_change":"2024-03-29T11:33:36.359461+0000","last_active":"2024-03-29T11:33:36.867161+0000","last_peered":"2024-03-29T11:33:36.867161+0000","last_clean":"2024-03-29T11:33:36.867161+0000","last_became_active":"2024-03-29T11:33:36.359171+0000","last_became_peered":"2024-03-29T11:33:36.359171+0000","last_unstale":"2024-03-29T11:33:36.867161+0000","last_undegraded":"2024-03-29T11:33:36.867161+0000","last_fullsized":"2024-03-29T11:33:36.867161+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:40:30.035629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.899555+0000","last_change":"2024-03-29T11:33:36.894647+0000","last_active":"2024-03-29T11:33:36.899555+0000","last_peered":"2024-03-29T11:33:36.899555+0000","last_clean":"2024-03-29T11:33:36.899555+0000","last_became_active":"2024-03-29T11:33:36.893163+0000","last_became_peered":"2024-03-29T11:33:36.893163+0000","last_unstale":"2024-03-29T11:33:36.899555+0000","last_undegraded":"2024-03-29T11:33:36.899555+0000","last_fullsized":"2024-03-29T11:33:36.899555+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:17.289090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.874045+0000","last_change":"2024-03-29T11:33:36.359465+0000","last_active":"2024-03-29T11:33:36.874045+0000","last_peered":"2024-03-29T11:33:36.874045+0000","last_clean":"2024-03-29T11:33:36.874045+0000","last_became_active":"2024-03-29T11:33:36.358184+0000","last_became_peered":"2024-03-29T11:33:36.358184+0000","last_unstale":"2024-03-29T11:33:36.874045+0000","last_undegraded":"2024-03-29T11:33:36.874045+0000","last_fullsized":"2024-03-29T11:33:36.874045+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:35.330015+0000","last_clean_scrub_stamp":"2024-03-29T11:33:35.330015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:42:25.421857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.901296+0000","last_change":"2024-03-29T11:33:36.895335+0000","last_active":"2024-03-29T11:33:36.901296+0000","last_peered":"2024-03-29T11:33:36.901296+0000","last_clean":"2024-03-29T11:33:36.901296+0000","last_became_active":"2024-03-29T11:33:36.895052+0000","last_became_peered":"2024-03-29T11:33:36.895052+0000","last_unstale":"2024-03-29T11:33:36.901296+0000","last_undegraded":"2024-03-29T11:33:36.901296+0000","last_fullsized":"2024-03-29T11:33:36.901296+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:33:36.333049+0000","last_clean_scrub_stamp":"2024-03-29T11:33:36.333049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:32:40.653645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"254'410","reported_seq":1463,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-03-29T11:33:36.360414+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:33:36.360414+0000","last_peered":"2024-03-29T11:33:36.360414+0000","last_clean":"2024-03-29T11:33:36.360414+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:33:36.360414+0000","last_undegraded":"2024-03-29T11:33:36.360414+0000","last_fullsized":"2024-03-29T11:33:36.360414+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2672,"num_read_kb":5632,"num_write":691,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2672,"num_read_kb":5632,"num_write":691,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":410,"ondisk_log_size":410,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576017,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57144,"kb_used_data":3884,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314696,"statfs":{"total":96636764160,"available":96578248704,"internally_reserved":0,"allocated":3977216,"data_stored":2257580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.67600000000000005,"15min":0.70799999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86799999999999999,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.68100000000000005,"15min":0.73099999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86899999999999999,"5min":0.98899999999999999,"15min":1.119},"last":0.51600000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.67100000000000004,"15min":0.71599999999999997},"min":{"1min":0.435,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84399999999999997,"5min":0.97399999999999998,"15min":1.478},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.70399999999999996,"15min":0.752},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.82699999999999996,"5min":1.0640000000000001,"15min":1.393},"last":0.623}]},{"osd":2,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70499999999999996,"15min":0.748},"min":{"1min":0.46600000000000003,"5min":0.436,"15min":0.436},"max":{"1min":2.5649999999999999,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.71399999999999997,"15min":0.76100000000000001},"min":{"1min":0.501,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88400000000000001,"5min":1.032,"15min":1.0720000000000001},"last":0.59299999999999997}]},{"osd":3,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.749},"min":{"1min":0.48499999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84299999999999997,"5min":0.97499999999999998,"15min":1.0580000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.77700000000000002,"15min":0.78600000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.6930000000000001,"5min":2.6930000000000001,"15min":2.6930000000000001},"last":0.70099999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88500000000000001,"5min":0.91100000000000003,"15min":1.3779999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66400000000000003,"15min":0.66900000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90800000000000003,"5min":0.93500000000000005,"15min":1.3169999999999999},"last":0.65400000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69999999999999996,"15min":0.68400000000000005},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.1160000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93700000000000006,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.64300000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.76800000000000002,"15min":0.76600000000000001},"min":{"1min":0.48099999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92600000000000005,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.75600000000000001,"15min":0.74199999999999999},"min":{"1min":0.52000000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.91000000000000003,"5min":1.071,"15min":1.071},"last":0.60899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576018,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48380,"kb_used_data":3308,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323460,"statfs":{"total":96636764160,"available":96587223040,"internally_reserved":0,"allocated":3387392,"data_stored":1667302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.67100000000000004,"15min":0.71999999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.94499999999999995,"15min":1.2150000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.68400000000000005,"15min":0.74199999999999999},"min":{"1min":0.42799999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95299999999999996,"5min":0.96399999999999997,"15min":1.4239999999999999},"last":0.64300000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67000000000000004,"15min":0.72199999999999998},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91100000000000003,"5min":1.022,"15min":1.98},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.70699999999999996,"15min":0.75900000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.435},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.8200000000000001},"last":0.80300000000000005}]},{"osd":2,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69699999999999995,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86899999999999999,"5min":0.96199999999999997,"15min":1.669},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71099999999999997,"15min":0.751},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.167,"5min":1.167,"15min":1.7589999999999999},"last":0.747}]},{"osd":3,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70599999999999996,"15min":0.753},"min":{"1min":0.51100000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90200000000000002,"5min":0.997,"15min":2.52},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.75700000000000001,"15min":0.79200000000000004},"min":{"1min":0.51200000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.877},"last":0.877}]},{"osd":4,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.61599999999999999},"min":{"1min":0.435,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93999999999999995,"15min":2.4100000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.71599999999999997,"15min":0.71199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.45900000000000002},"max":{"1min":1.0740000000000001,"5min":1.2709999999999999,"15min":2.6629999999999998},"last":0.91100000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66900000000000004,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99099999999999999,"5min":1.0900000000000001,"15min":1.5960000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.925},"last":0.93300000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77100000000000002,"15min":0.76900000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96699999999999997,"5min":1.0509999999999999,"15min":2.7050000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74299999999999999,"15min":0.74399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0980000000000001,"5min":1.1220000000000001,"15min":1.726},"last":1}]}]},{"osd":1,"up_from":13,"seq":55834576016,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60672,"kb_used_data":3308,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311168,"statfs":{"total":96636764160,"available":96574636032,"internally_reserved":0,"allocated":3387392,"data_stored":1667246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.44400000000000001,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":0.79100000000000004,"15min":0.80300000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.505,"15min":0.52400000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.58899999999999997,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52600000000000002,"15min":0.55400000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.625,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.751,"5min":0.75900000000000001,"15min":0.90300000000000002},"last":0.33800000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56699999999999995,"15min":0.58799999999999997},"min":{"1min":0.38100000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.85099999999999998,"15min":0.995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.54500000000000004,"15min":0.57399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76200000000000001,"5min":0.80100000000000005,"15min":0.83999999999999997},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.74399999999999999,"15min":0.78000000000000003},"min":{"1min":0.52900000000000003,"5min":0.505,"15min":0.505},"max":{"1min":0.997,"5min":1.153,"15min":1.238},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76800000000000002,"15min":0.79300000000000004},"min":{"1min":0.57599999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.097,"5min":1.226,"15min":1.304},"last":0.90100000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80000000000000004,"15min":0.81299999999999994},"min":{"1min":0.57499999999999996,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":0.95899999999999996,"5min":1.276,"15min":1.3720000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.78000000000000003,"15min":0.82499999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.05,"5min":1.1299999999999999,"15min":1.3520000000000001},"last":0.96199999999999997}]},{"osd":6,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.78200000000000003,"15min":0.81100000000000005},"min":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.103,"5min":1.1890000000000001,"15min":1.1919999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.81499999999999995,"15min":0.83999999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.012,"5min":1.2250000000000001,"15min":1.266},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:33:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.83899999999999997,"15min":0.86299999999999999},"min":{"1min":0.60899999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.129,"5min":1.327,"15min":1.327},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.82099999999999995,"15min":0.86099999999999999},"min":{"1min":0.57999999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0569999999999999,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.91400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576018,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61256,"kb_used_data":3892,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310584,"statfs":{"total":96636764160,"available":96574038016,"internally_reserved":0,"allocated":3985408,"data_stored":2257199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44900000000000001,"15min":0.46100000000000002},"min":{"1min":0.317,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.623,"5min":0.95899999999999996,"15min":1.087},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.52200000000000002,"15min":0.53500000000000003},"min":{"1min":0.32400000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67200000000000004,"5min":0.89100000000000001,"15min":1.3089999999999999},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56000000000000005,"15min":0.57899999999999996},"min":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66600000000000004,"5min":1.028,"15min":1.5289999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.499,"15min":0.51600000000000001},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.629,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55600000000000005,"15min":0.56200000000000006},"min":{"1min":0.38500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77600000000000002,"5min":1.222,"15min":1.222},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.622,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.44,"15min":0.42199999999999999},"max":{"1min":0.81799999999999995,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.60899999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80500000000000005,"15min":0.81399999999999995},"min":{"1min":0.56899999999999995,"5min":0.55600000000000005,"15min":0.495},"max":{"1min":0.93400000000000005,"5min":1.248,"15min":1.248},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82399999999999995,"15min":0.84599999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0309999999999999,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.82999999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76800000000000002,"15min":0.78500000000000003},"min":{"1min":0.54500000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.88900000000000001,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80400000000000005,"15min":0.81899999999999995},"min":{"1min":0.58999999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94899999999999995,"5min":1.2829999999999999,"15min":1.3640000000000001},"last":0.79500000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.83399999999999996,"15min":0.85299999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.97599999999999998,"5min":1.343,"15min":1.6240000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.85699999999999998,"15min":0.88200000000000001},"min":{"1min":0.62,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.004,"5min":1.3240000000000001,"15min":1.337},"last":0.89600000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79900000000000004,"15min":0.82399999999999995},"min":{"1min":0.56899999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.99099999999999999,"5min":1.115,"15min":1.214},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.84099999999999997,"15min":0.86899999999999999},"min":{"1min":0.55500000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.054,"5min":1.365,"15min":1.365},"last":0.98999999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576017,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52476,"kb_used_data":3308,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319364,"statfs":{"total":96636764160,"available":96583028736,"internally_reserved":0,"allocated":3387392,"data_stored":1667302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.47299999999999998,"15min":0.47199999999999998},"min":{"1min":0.32300000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.5,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60199999999999998,"5min":0.73999999999999999,"15min":0.96799999999999997},"last":0.34699999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47899999999999998,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.54900000000000004,"15min":0.56999999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.872,"15min":0.88800000000000001},"last":0.45700000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55800000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64900000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.439},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55700000000000005,"15min":0.57199999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.71099999999999997,"5min":1.038,"15min":1.038},"last":0.55900000000000005}]},{"osd":4,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.754,"15min":0.79800000000000004},"min":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1970000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.747,"15min":0.79900000000000004},"min":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0029999999999999,"5min":1.157,"15min":1.244},"last":0.72599999999999998}]},{"osd":5,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77000000000000002,"15min":0.80900000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.53400000000000003},"max":{"1min":0.96399999999999997,"5min":1.135,"15min":1.256},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.76500000000000001,"15min":0.81399999999999995},"min":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.3140000000000001},"last":0.89100000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.78000000000000003,"15min":0.83099999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93200000000000005,"5min":1.105,"15min":1.24},"last":0.873},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.78000000000000003,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.5,"15min":0.5},"max":{"1min":0.98199999999999998,"5min":1.099,"15min":1.329},"last":0.86199999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82499999999999996,"15min":0.88},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.044,"5min":1.0660000000000001,"15min":1.2889999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.83899999999999997,"15min":0.89400000000000002},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.56899999999999995},"max":{"1min":1.0269999999999999,"5min":1.2969999999999999,"15min":1.304},"last":0.90700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576017,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48384,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323456,"statfs":{"total":96636764160,"available":96587218944,"internally_reserved":0,"allocated":3395584,"data_stored":1668776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.29599999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.33200000000000002,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.29899999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48399999999999999,"15min":0.49399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.57199999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.38700000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.435},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.51400000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79600000000000004,"15min":0.79700000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.80300000000000005,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.86899999999999999}]},{"osd":5,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80500000000000005,"15min":0.81299999999999994},"min":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.502},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.83099999999999996,"15min":0.82699999999999996},"min":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.54600000000000004},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":0.76400000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86599999999999999,"15min":0.85899999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.746},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.84099999999999997,"15min":0.84299999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":0.79500000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86699999999999999,"15min":0.86599999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.58099999999999996},"max":{"1min":2.444,"5min":2.444,"15min":2.444},"last":0.78600000000000003}]}]},{"osd":4,"up_from":13,"seq":55834576018,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78660,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":75326,"kb_avail":94293180,"statfs":{"total":96636764160,"available":96556216320,"internally_reserved":0,"allocated":3391488,"data_stored":1667304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77134278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64400000000000002,"15min":0.70699999999999996},"min":{"1min":0.39700000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":1.0509999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.64000000000000001,"15min":0.70799999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.81000000000000005,"5min":0.93600000000000005,"15min":1.1459999999999999},"last":0.76300000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.65500000000000003,"15min":0.72499999999999998},"min":{"1min":0.48699999999999999,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.1910000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67000000000000004,"15min":0.72999999999999998},"min":{"1min":0.435,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":1.002},"last":0.74299999999999999}]},{"osd":2,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76300000000000001,"15min":0.81399999999999995},"min":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.73899999999999999,"15min":0.80000000000000004},"min":{"1min":0.51900000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.875,"5min":1.224,"15min":1.224},"last":0.84699999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68500000000000005,"15min":0.748},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88100000000000001,"5min":0.96699999999999997,"15min":1.181},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72899999999999998,"15min":0.76300000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.83399999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64900000000000002,"15min":0.628},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.01,"5min":1.0920000000000001,"15min":1.169},"last":0.85999999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69599999999999995,"15min":0.67300000000000004},"min":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72599999999999998,"15min":0.72099999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70899999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.73899999999999999,"15min":0.73399999999999999},"min":{"1min":0.54100000000000004,"5min":0.501,"15min":0.436},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.2669999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.81899999999999995,"15min":0.82299999999999995},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.54400000000000004},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.78600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576017,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48396,"kb_used_data":3324,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323444,"statfs":{"total":96636764160,"available":96587206656,"internally_reserved":0,"allocated":3403776,"data_stored":1669428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.69799999999999995},"min":{"1min":0.47199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.72299999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.95599999999999996,"15min":1.0680000000000001},"last":0.48299999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73699999999999999,"15min":0.75700000000000001},"min":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98099999999999998,"5min":1.839,"15min":1.839},"last":0.434},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.71899999999999997,"15min":0.73599999999999999},"min":{"1min":0.504,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.282,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.753,"15min":0.755},"min":{"1min":0.51000000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.71499999999999997,"15min":0.74399999999999999},"min":{"1min":0.54400000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.48499999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.95099999999999996,"5min":3.444,"15min":3.444},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.75800000000000001,"15min":0.77200000000000002},"min":{"1min":0.54400000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.629,"15min":0.60899999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38700000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67800000000000005,"15min":0.65300000000000002},"min":{"1min":0.47799999999999998,"5min":0.41699999999999998,"15min":0.40600000000000003},"max":{"1min":0.999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.78700000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.69699999999999995,"15min":0.67000000000000004},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72199999999999998,"15min":0.70299999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.107},"last":0.747}]},{"osd":7,"last update":"Fri Mar 29 11:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.746,"15min":0.72599999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.42299999999999999},"max":{"1min":1.04,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79600000000000004,"15min":0.77400000000000002},"min":{"1min":0.53500000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.71499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:33:42.291 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T11:33:42.291 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T11:33:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.536+0000 7f6b19e00700 1 Processor -- start 2024-03-29T11:33:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.536+0000 7f6b19e00700 1 -- start start 2024-03-29T11:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.537+0000 7f6b19e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6b1415d9d0 0x7f6b140d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.537+0000 7f6b19e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d16a0 0x7f6b140d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.537+0000 7f6b19e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6b140d2370 0x7f6b1415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.537+0000 7f6b19e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b140d2820 con 0x7f6b1415d9d0 2024-03-29T11:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.537+0000 7f6b19e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b14161420 con 0x7f6b140d2370 2024-03-29T11:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.537+0000 7f6b19e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b14161700 con 0x7f6b140d16a0 2024-03-29T11:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.537+0000 7f6b12a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6b1415d9d0 0x7f6b140d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.537+0000 7f6b12a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6b1415d9d0 0x7f6b140d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56480/0 (socket says 172.21.15.12:56480) 2024-03-29T11:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.537+0000 7f6b12a00700 1 -- 172.21.15.12:0/1045868888 learned_addr learned my addr 172.21.15.12:0/1045868888 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b12a00700 1 -- 172.21.15.12:0/1045868888 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d16a0 msgr2=0x7f6b140d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b12000700 1 --2- 172.21.15.12:0/1045868888 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d16a0 0x7f6b140d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b13400700 1 --2- 172.21.15.12:0/1045868888 >> v2:172.21.15.159:3300/0 conn(0x7f6b140d2370 0x7f6b1415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b12a00700 1 --2- 172.21.15.12:0/1045868888 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d16a0 0x7f6b140d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b12a00700 1 -- 172.21.15.12:0/1045868888 >> v2:172.21.15.159:3300/0 conn(0x7f6b140d2370 msgr2=0x7f6b1415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b12a00700 1 --2- 172.21.15.12:0/1045868888 >> v2:172.21.15.159:3300/0 conn(0x7f6b140d2370 0x7f6b1415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b12a00700 1 -- 172.21.15.12:0/1045868888 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b141619e0 con 0x7f6b1415d9d0 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b13400700 1 --2- 172.21.15.12:0/1045868888 >> v2:172.21.15.159:3300/0 conn(0x7f6b140d2370 0x7f6b1415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b12000700 1 --2- 172.21.15.12:0/1045868888 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d16a0 0x7f6b140d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b12a00700 1 --2- 172.21.15.12:0/1045868888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 0x7f6b140d3c30 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f6b140cdc00 tx=0x7f6b04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae8c7ff85a55173 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b11600700 1 -- 172.21.15.12:0/1045868888 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0400b870 con 0x7f6b1415d9d0 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b11600700 1 -- 172.21.15.12:0/1045868888 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0400b9d0 con 0x7f6b1415d9d0 2024-03-29T11:33:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.538+0000 7f6b11600700 1 -- 172.21.15.12:0/1045868888 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0400bd60 con 0x7f6b1415d9d0 2024-03-29T11:33:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.539+0000 7f6b19e00700 1 -- 172.21.15.12:0/1045868888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 msgr2=0x7f6b140d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.539+0000 7f6b19e00700 1 --2- 172.21.15.12:0/1045868888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 0x7f6b140d3c30 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f6b140cdc00 tx=0x7f6b04009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.539+0000 7f6b19e00700 1 -- 172.21.15.12:0/1045868888 shutdown_connections 2024-03-29T11:33:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.539+0000 7f6b19e00700 1 --2- 172.21.15.12:0/1045868888 >> v2:172.21.15.159:3300/0 conn(0x7f6b140d2370 0x7f6b1415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.539+0000 7f6b19e00700 1 --2- 172.21.15.12:0/1045868888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 0x7f6b140d3c30 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.539+0000 7f6b19e00700 1 --2- 172.21.15.12:0/1045868888 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d16a0 0x7f6b140d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.539+0000 7f6b19e00700 1 -- 172.21.15.12:0/1045868888 >> 172.21.15.12:0/1045868888 conn(0x7f6b1408aff0 msgr2=0x7f6b140d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.539+0000 7f6b19e00700 1 -- 172.21.15.12:0/1045868888 shutdown_connections 2024-03-29T11:33:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.539+0000 7f6b19e00700 1 -- 172.21.15.12:0/1045868888 wait complete. 2024-03-29T11:33:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.541+0000 7f6b19e00700 1 Processor -- start 2024-03-29T11:33:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.541+0000 7f6b19e00700 1 -- start start 2024-03-29T11:33:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.542+0000 7f6b19e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b140d16a0 0x7f6b141e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.542+0000 7f6b19e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d2370 0x7f6b14155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.542+0000 7f6b19e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 0x7f6b141e8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.542+0000 7f6b19e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b141e78d0 con 0x7f6b1415d9d0 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.542+0000 7f6b19e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b141e82a0 con 0x7f6b140d16a0 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.542+0000 7f6b19e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b141e97b0 con 0x7f6b140d2370 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.542+0000 7f6b13400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 0x7f6b141e8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.542+0000 7f6b13400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 0x7f6b141e8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56490/0 (socket says 172.21.15.12:56490) 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.542+0000 7f6b13400700 1 -- 172.21.15.12:0/720257006 learned_addr learned my addr 172.21.15.12:0/720257006 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.542+0000 7f6b12a00700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b140d16a0 0x7f6b141e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b12000700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d2370 0x7f6b14155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b13400700 1 -- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d2370 msgr2=0x7f6b14155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b13400700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d2370 0x7f6b14155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b13400700 1 -- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b140d16a0 msgr2=0x7f6b141e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b13400700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b140d16a0 0x7f6b141e7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b13400700 1 -- 172.21.15.12:0/720257006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b04009550 con 0x7f6b1415d9d0 2024-03-29T11:33:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b13400700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 0x7f6b141e8eb0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00cef0 tx=0x7f6b0c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b0b400700 1 -- 172.21.15.12:0/720257006 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c00ad00 con 0x7f6b1415d9d0 2024-03-29T11:33:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b0b400700 1 -- 172.21.15.12:0/720257006 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c00ae60 con 0x7f6b1415d9d0 2024-03-29T11:33:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b19e00700 1 -- 172.21.15.12:0/720257006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b141e9a90 con 0x7f6b1415d9d0 2024-03-29T11:33:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.543+0000 7f6b0b400700 1 -- 172.21.15.12:0/720257006 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c015680 con 0x7f6b1415d9d0 2024-03-29T11:33:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.544+0000 7f6b19e00700 1 -- 172.21.15.12:0/720257006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b141e7560 con 0x7f6b1415d9d0 2024-03-29T11:33:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.545+0000 7f6b19e00700 1 -- 172.21.15.12:0/720257006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b140d1ca0 con 0x7f6b1415d9d0 2024-03-29T11:33:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.546+0000 7f6b0b400700 1 -- 172.21.15.12:0/720257006 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6b0c0157e0 con 0x7f6b1415d9d0 2024-03-29T11:33:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.547+0000 7f6b0b400700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6af406aed0 0x7f6af406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.547+0000 7f6b0b400700 1 -- 172.21.15.12:0/720257006 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f6b0c089c40 con 0x7f6b1415d9d0 2024-03-29T11:33:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.547+0000 7f6b12a00700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6af406aed0 0x7f6af406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.548+0000 7f6b12a00700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6af406aed0 0x7f6af406d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b04014040 tx=0x7f6b04008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.551+0000 7f6b0b400700 1 -- 172.21.15.12:0/720257006 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6b0c054a60 con 0x7f6b1415d9d0 2024-03-29T11:33:42.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.848+0000 7f6b19e00700 1 -- 172.21.15.12:0/720257006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6b140cfb10 con 0x7f6b1415d9d0 2024-03-29T11:33:42.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.852+0000 7f6b0b400700 1 -- 172.21.15.12:0/720257006 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v53) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6b0c01a070 con 0x7f6b1415d9d0 2024-03-29T11:33:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 -- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6af406aed0 msgr2=0x7f6af406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6af406aed0 0x7f6af406d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b04014040 tx=0x7f6b04008030 comp rx=0 tx=0).stop 2024-03-29T11:33:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 -- 172.21.15.12:0/720257006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 msgr2=0x7f6b141e8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 0x7f6b141e8eb0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00cef0 tx=0x7f6b0c00cf20 comp rx=0 tx=0).stop 2024-03-29T11:33:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 -- 172.21.15.12:0/720257006 shutdown_connections 2024-03-29T11:33:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6af406aed0 0x7f6af406d390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b1415d9d0 0x7f6b141e8eb0 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b140d16a0 0x7f6b141e7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 --2- 172.21.15.12:0/720257006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b140d2370 0x7f6b14155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 -- 172.21.15.12:0/720257006 >> 172.21.15.12:0/720257006 conn(0x7f6b1408aff0 msgr2=0x7f6b140b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 -- 172.21.15.12:0/720257006 shutdown_connections 2024-03-29T11:33:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:42.855+0000 7f6b19e00700 1 -- 172.21.15.12:0/720257006 wait complete. 2024-03-29T11:33:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T11:33:42.870 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:33:43.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.072+0000 7f8de3200700 1 Processor -- start 2024-03-29T11:33:43.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.072+0000 7f8de3200700 1 -- start start 2024-03-29T11:33:43.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.073+0000 7f8de3200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ddc0ceec0 0x7f8ddc1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.073+0000 7f8de3200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ddc161800 0x7f8ddc0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.073+0000 7f8de3200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc0d30d0 con 0x7f8ddc15d640 2024-03-29T11:33:43.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.073+0000 7f8de3200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc161c90 con 0x7f8ddc0ceec0 2024-03-29T11:33:43.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.073+0000 7f8de3200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc0b6840 con 0x7f8ddc161800 2024-03-29T11:33:43.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8ddb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ddc0ceec0 0x7f8ddc1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8ddb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ddc0ceec0 0x7f8ddc1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56502/0 (socket says 172.21.15.12:56502) 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8ddbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8ddc15d640 0x7f8ddc0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49700/0 (socket says 172.21.15.12:49700) 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8ddb400700 1 -- 172.21.15.12:0/1812988127 learned_addr learned my addr 172.21.15.12:0/1812988127 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8de1200700 1 --2- 172.21.15.12:0/1812988127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ddc161800 0x7f8ddc0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8ddb400700 1 -- 172.21.15.12:0/1812988127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ddc161800 msgr2=0x7f8ddc0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8ddb400700 1 --2- 172.21.15.12:0/1812988127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ddc161800 0x7f8ddc0b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8ddb400700 1 -- 172.21.15.12:0/1812988127 >> v1:172.21.15.12:6789/0 conn(0x7f8ddc15d640 legacy=0x7f8ddc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8ddb400700 1 -- 172.21.15.12:0/1812988127 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc0d2b30 con 0x7f8ddc0ceec0 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8ddaa00700 1 -- 172.21.15.12:0/1812988127 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522979720 0 0) 0x7f8ddc0d30d0 con 0x7f8ddc15d640 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.074+0000 7f8ddb400700 1 --2- 172.21.15.12:0/1812988127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc0ceec0 0x7f8ddc1612c0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80098f0 tx=0x7f8dc8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948633289afae719 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.075+0000 7f8ddaa00700 1 -- 172.21.15.12:0/1812988127 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8011830 con 0x7f8ddc0ceec0 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.075+0000 7f8ddaa00700 1 -- 172.21.15.12:0/1812988127 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc8011990 con 0x7f8ddc0ceec0 2024-03-29T11:33:43.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.075+0000 7f8ddaa00700 1 -- 172.21.15.12:0/1812988127 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8011d20 con 0x7f8ddc0ceec0 2024-03-29T11:33:43.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.075+0000 7f8de3200700 1 -- 172.21.15.12:0/1812988127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc0ceec0 msgr2=0x7f8ddc1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.075+0000 7f8de3200700 1 --2- 172.21.15.12:0/1812988127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc0ceec0 0x7f8ddc1612c0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80098f0 tx=0x7f8dc8009c00 comp rx=0 tx=0).stop 2024-03-29T11:33:43.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.075+0000 7f8de3200700 1 -- 172.21.15.12:0/1812988127 shutdown_connections 2024-03-29T11:33:43.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.075+0000 7f8de3200700 1 --2- 172.21.15.12:0/1812988127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ddc161800 0x7f8ddc0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.075+0000 7f8de3200700 1 --2- 172.21.15.12:0/1812988127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc0ceec0 0x7f8ddc1612c0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.075+0000 7f8de3200700 1 -- 172.21.15.12:0/1812988127 >> 172.21.15.12:0/1812988127 conn(0x7f8ddc08aff0 msgr2=0x7f8ddc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.075+0000 7f8de3200700 1 -- 172.21.15.12:0/1812988127 shutdown_connections 2024-03-29T11:33:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.076+0000 7f8de3200700 1 -- 172.21.15.12:0/1812988127 wait complete. 2024-03-29T11:33:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.077+0000 7f8de3200700 1 Processor -- start 2024-03-29T11:33:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.077+0000 7f8de3200700 1 -- start start 2024-03-29T11:33:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.078+0000 7f8de3200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ddc0ceec0 0x7f8ddc0c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.078+0000 7f8de3200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc15d640 0x7f8ddc15f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.078+0000 7f8de3200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ddc161800 0x7f8ddc09d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.078+0000 7f8de3200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc159030 con 0x7f8ddc15d640 2024-03-29T11:33:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.078+0000 7f8de3200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc09fdf0 con 0x7f8ddc0ceec0 2024-03-29T11:33:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.078+0000 7f8de3200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc09e370 con 0x7f8ddc161800 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8ddb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc15d640 0x7f8ddc15f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8ddb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc15d640 0x7f8ddc15f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56518/0 (socket says 172.21.15.12:56518) 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8ddb400700 1 -- 172.21.15.12:0/3373243376 learned_addr learned my addr 172.21.15.12:0/3373243376 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8de1200700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ddc161800 0x7f8ddc09d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8ddbe00700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ddc0ceec0 0x7f8ddc0c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8ddb400700 1 -- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ddc161800 msgr2=0x7f8ddc09d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8ddb400700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ddc161800 0x7f8ddc09d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8ddb400700 1 -- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ddc0ceec0 msgr2=0x7f8ddc0c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8ddb400700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ddc0ceec0 0x7f8ddc0c8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8ddb400700 1 -- 172.21.15.12:0/3373243376 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc8009550 con 0x7f8ddc15d640 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.079+0000 7f8ddb400700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc15d640 0x7f8ddc15f080 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009c40 tx=0x7f8dc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.080+0000 7f8dd3e00700 1 -- 172.21.15.12:0/3373243376 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8004510 con 0x7f8ddc15d640 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.080+0000 7f8dd3e00700 1 -- 172.21.15.12:0/3373243376 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc801e070 con 0x7f8ddc15d640 2024-03-29T11:33:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.080+0000 7f8de3200700 1 -- 172.21.15.12:0/3373243376 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc09e650 con 0x7f8ddc15d640 2024-03-29T11:33:43.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.080+0000 7f8dd3e00700 1 -- 172.21.15.12:0/3373243376 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc80038c0 con 0x7f8ddc15d640 2024-03-29T11:33:43.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.080+0000 7f8de3200700 1 -- 172.21.15.12:0/3373243376 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc0a0090 con 0x7f8ddc15d640 2024-03-29T11:33:43.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.081+0000 7f8de3200700 1 -- 172.21.15.12:0/3373243376 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ddc0b6490 con 0x7f8ddc15d640 2024-03-29T11:33:43.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.082+0000 7f8dd3e00700 1 -- 172.21.15.12:0/3373243376 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8dc8003a20 con 0x7f8ddc15d640 2024-03-29T11:33:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.083+0000 7f8dd3e00700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8dbc06aed0 0x7f8dbc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.083+0000 7f8dd3e00700 1 -- 172.21.15.12:0/3373243376 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f8dc808a5b0 con 0x7f8ddc15d640 2024-03-29T11:33:43.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.083+0000 7f8ddbe00700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8dbc06aed0 0x7f8dbc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.084+0000 7f8ddbe00700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8dbc06aed0 0x7f8dbc06d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc000f60 tx=0x7f8dcc007e70 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:43.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.087+0000 7f8dd3e00700 1 -- 172.21.15.12:0/3373243376 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8dc8055420 con 0x7f8ddc15d640 2024-03-29T11:33:43.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.346+0000 7f8de3200700 1 -- 172.21.15.12:0/3373243376 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ddc067720 con 0x7f8ddc15d640 2024-03-29T11:33:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.346+0000 7f8dd3e00700 1 -- 172.21.15.12:0/3373243376 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f8dc8058a40 con 0x7f8ddc15d640 2024-03-29T11:33:43.343 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:43.343 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":138,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1546500295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1546500295},{"type":"v1","addr":"172.21.15.159:6836","nonce":1546500295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1627782307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1627782307},{"type":"v1","addr":"172.21.15.12:6834","nonce":1627782307}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3327995506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3327995506},{"type":"v1","addr":"172.21.15.12:6835","nonce":3327995506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:33:36.862806+0000","modified":"2024-03-29T11:33:40.346151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12061":{"gid":12061,"name":"d","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/2746367434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2746367434},{"type":"v1","addr":"172.21.15.159:6837","nonce":2746367434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-29T11:33:43.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 -- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8dbc06aed0 msgr2=0x7f8dbc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8dbc06aed0 0x7f8dbc06d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc000f60 tx=0x7f8dcc007e70 comp rx=0 tx=0).stop 2024-03-29T11:33:43.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 -- 172.21.15.12:0/3373243376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc15d640 msgr2=0x7f8ddc15f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc15d640 0x7f8ddc15f080 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009c40 tx=0x7f8dc8004750 comp rx=0 tx=0).stop 2024-03-29T11:33:43.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 -- 172.21.15.12:0/3373243376 shutdown_connections 2024-03-29T11:33:43.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8dbc06aed0 0x7f8dbc06d390 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ddc161800 0x7f8ddc09d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ddc0ceec0 0x7f8ddc0c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 --2- 172.21.15.12:0/3373243376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ddc15d640 0x7f8ddc15f080 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 -- 172.21.15.12:0/3373243376 >> 172.21.15.12:0/3373243376 conn(0x7f8ddc08aff0 msgr2=0x7f8ddc0b8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:43.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 -- 172.21.15.12:0/3373243376 shutdown_connections 2024-03-29T11:33:43.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.349+0000 7f8de3200700 1 -- 172.21.15.12:0/3373243376 wait complete. 2024-03-29T11:33:43.347 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 138 2024-03-29T11:33:43.767 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 138, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T11:33:36.862806+0000', 'modified': '2024-03-29T11:33:40.346151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12061}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12061': {'gid': 12061, 'name': 'd', 'rank': 0, 'incarnation': 134, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.159:6837/2746367434', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6835', 'nonce': 2746367434}, {'type': 'v1', 'addr': '172.21.15.159:6837', 'nonce': 2746367434}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T11:33:43.767 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T11:33:43.767 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-29T11:33:43.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.972+0000 7f7a88e00700 1 Processor -- start 2024-03-29T11:33:43.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.972+0000 7f7a88e00700 1 -- start start 2024-03-29T11:33:43.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.973+0000 7f7a88e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 0x7f7a84161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.973+0000 7f7a88e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a840ce270 con 0x7f7a841611b0 2024-03-29T11:33:43.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.973+0000 7f7a88e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8415b600 con 0x7f7a840d21f0 2024-03-29T11:33:43.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.973+0000 7f7a88e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a840d2760 con 0x7f7a840cee10 2024-03-29T11:33:43.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.973+0000 7f7a81e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7a841611b0 0x7f7a8415ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49706/0 (socket says 172.21.15.12:49706) 2024-03-29T11:33:43.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.973+0000 7f7a81e00700 1 -- 172.21.15.12:0/4103897851 learned_addr learned my addr 172.21.15.12:0/4103897851 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:43.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.973+0000 7f7a82800700 1 --2- 172.21.15.12:0/4103897851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 0x7f7a84161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.974+0000 7f7a80a00700 1 -- 172.21.15.12:0/4103897851 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739469305 0 0) 0x7f7a840ce270 con 0x7f7a841611b0 2024-03-29T11:33:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.974+0000 7f7a80a00700 1 -- 172.21.15.12:0/4103897851 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a680035b0 con 0x7f7a841611b0 2024-03-29T11:33:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.974+0000 7f7a80a00700 1 -- 172.21.15.12:0/4103897851 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828441942 0 0) 0x7f7a8415b600 con 0x7f7a840d21f0 2024-03-29T11:33:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.974+0000 7f7a80a00700 1 -- 172.21.15.12:0/4103897851 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a840ce270 con 0x7f7a840d21f0 2024-03-29T11:33:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.974+0000 7f7a82800700 1 -- 172.21.15.12:0/4103897851 >> v1:172.21.15.159:6789/0 conn(0x7f7a840d21f0 legacy=0x7f7a840ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.974+0000 7f7a82800700 1 -- 172.21.15.12:0/4103897851 >> v1:172.21.15.12:6789/0 conn(0x7f7a841611b0 legacy=0x7f7a8415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.974+0000 7f7a82800700 1 -- 172.21.15.12:0/4103897851 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a840d2a40 con 0x7f7a840cee10 2024-03-29T11:33:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.974+0000 7f7a82800700 1 --2- 172.21.15.12:0/4103897851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 0x7f7a84161590 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009dc0 tx=0x7f7a7c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e929cfdb4b76e79b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.975+0000 7f7a80a00700 1 -- 172.21.15.12:0/4103897851 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c00e070 con 0x7f7a840cee10 2024-03-29T11:33:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.975+0000 7f7a80a00700 1 -- 172.21.15.12:0/4103897851 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c00bb20 con 0x7f7a840cee10 2024-03-29T11:33:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.975+0000 7f7a80a00700 1 -- 172.21.15.12:0/4103897851 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c012430 con 0x7f7a840cee10 2024-03-29T11:33:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.975+0000 7f7a88e00700 1 -- 172.21.15.12:0/4103897851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 msgr2=0x7f7a84161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.975+0000 7f7a88e00700 1 --2- 172.21.15.12:0/4103897851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 0x7f7a84161590 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009dc0 tx=0x7f7a7c00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.975+0000 7f7a88e00700 1 -- 172.21.15.12:0/4103897851 shutdown_connections 2024-03-29T11:33:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.975+0000 7f7a88e00700 1 --2- 172.21.15.12:0/4103897851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 0x7f7a84161590 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.975+0000 7f7a88e00700 1 -- 172.21.15.12:0/4103897851 >> 172.21.15.12:0/4103897851 conn(0x7f7a8408aff0 msgr2=0x7f7a840d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.975+0000 7f7a88e00700 1 -- 172.21.15.12:0/4103897851 shutdown_connections 2024-03-29T11:33:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.975+0000 7f7a88e00700 1 -- 172.21.15.12:0/4103897851 wait complete. 2024-03-29T11:33:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.977+0000 7f7a88e00700 1 Processor -- start 2024-03-29T11:33:43.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.977+0000 7f7a88e00700 1 -- start start 2024-03-29T11:33:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a88e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 0x7f7a84156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a88e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a840d21f0 0x7f7a84158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a88e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a841611b0 0x7f7a84158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a88e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a841f3120 con 0x7f7a840d21f0 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a88e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a840b8620 con 0x7f7a841611b0 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a88e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a840b8900 con 0x7f7a840cee10 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a81400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a840d21f0 0x7f7a84158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a81400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a840d21f0 0x7f7a84158530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56526/0 (socket says 172.21.15.12:56526) 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a81400700 1 -- 172.21.15.12:0/4161690547 learned_addr learned my addr 172.21.15.12:0/4161690547 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a82800700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a841611b0 0x7f7a84158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.978+0000 7f7a81e00700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 0x7f7a84156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a81400700 1 -- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 msgr2=0x7f7a84156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a81400700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 0x7f7a84156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a81400700 1 -- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a841611b0 msgr2=0x7f7a84158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a81400700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a841611b0 0x7f7a84158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a81400700 1 -- 172.21.15.12:0/4161690547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c009a70 con 0x7f7a840d21f0 2024-03-29T11:33:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a81400700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a840d21f0 0x7f7a84158530 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f7a7000c520 tx=0x7f7a7000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:43.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a7a000700 1 -- 172.21.15.12:0/4161690547 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a700052d0 con 0x7f7a840d21f0 2024-03-29T11:33:43.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a7a000700 1 -- 172.21.15.12:0/4161690547 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a70005430 con 0x7f7a840d21f0 2024-03-29T11:33:43.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a7a000700 1 -- 172.21.15.12:0/4161690547 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a700038c0 con 0x7f7a840d21f0 2024-03-29T11:33:43.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a88e00700 1 -- 172.21.15.12:0/4161690547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a840b8be0 con 0x7f7a840d21f0 2024-03-29T11:33:43.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.979+0000 7f7a88e00700 1 -- 172.21.15.12:0/4161690547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a84156eb0 con 0x7f7a840d21f0 2024-03-29T11:33:43.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.982+0000 7f7a7a000700 1 -- 172.21.15.12:0/4161690547 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f7a70003a20 con 0x7f7a840d21f0 2024-03-29T11:33:43.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.983+0000 7f7a7a000700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a6406aed0 0x7f7a6406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.983+0000 7f7a7a000700 1 -- 172.21.15.12:0/4161690547 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f7a70088b50 con 0x7f7a840d21f0 2024-03-29T11:33:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.983+0000 7f7a81e00700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a6406aed0 0x7f7a6406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.984+0000 7f7a81e00700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a6406aed0 0x7f7a6406d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7a74007fd0 tx=0x7f7a74007dc0 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:43.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.985+0000 7f7a9c8aab80 1 Processor -- start 2024-03-29T11:33:43.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.985+0000 7f7a9c8aab80 1 -- start start 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b706830 0x562e8b706c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b6d9be0 0x562e8b6dca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d7b60 0x562e8b703a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e8b651650 con 0x562e8b706830 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e8b632c30 con 0x562e8b6d9be0 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e8b630430 con 0x562e8b6d7b60 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a81e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b706830 0x562e8b706c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a81e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b706830 0x562e8b706c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56534/0 (socket says 172.21.15.12:56534) 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a81e00700 1 -- 172.21.15.12:0/652411703 learned_addr learned my addr 172.21.15.12:0/652411703 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a81400700 1 --2- 172.21.15.12:0/652411703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b6d9be0 0x562e8b6dca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.986+0000 7f7a82800700 1 --2- 172.21.15.12:0/652411703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d7b60 0x562e8b703a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.987+0000 7f7a81e00700 1 -- 172.21.15.12:0/652411703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d7b60 msgr2=0x562e8b703a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:43.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.987+0000 7f7a81e00700 1 --2- 172.21.15.12:0/652411703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d7b60 0x562e8b703a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.987+0000 7f7a81e00700 1 -- 172.21.15.12:0/652411703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b6d9be0 msgr2=0x562e8b6dca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.987+0000 7f7a81e00700 1 --2- 172.21.15.12:0/652411703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b6d9be0 0x562e8b6dca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.987+0000 7f7a81e00700 1 -- 172.21.15.12:0/652411703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e8b41f790 con 0x562e8b706830 2024-03-29T11:33:43.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.987+0000 7f7a81e00700 1 --2- 172.21.15.12:0/652411703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b706830 0x562e8b706c10 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f7a74007fa0 tx=0x7f7a74012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b264dbdb2fcc94 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.987+0000 7f7a6ea00700 1 -- 172.21.15.12:0/652411703 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a740062c0 con 0x562e8b706830 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.987+0000 7f7a6ea00700 1 -- 172.21.15.12:0/652411703 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a74006420 con 0x562e8b706830 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.987+0000 7f7a6ea00700 1 -- 172.21.15.12:0/652411703 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74006790 con 0x562e8b706830 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.988+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/652411703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b706830 msgr2=0x562e8b706c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.988+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/652411703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b706830 0x562e8b706c10 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f7a74007fa0 tx=0x7f7a74012000 comp rx=0 tx=0).stop 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.988+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/652411703 shutdown_connections 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.988+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/652411703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d7b60 0x562e8b703a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.988+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/652411703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b706830 0x562e8b706c10 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.988+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/652411703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b6d9be0 0x562e8b6dca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.988+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/652411703 >> 172.21.15.12:0/652411703 conn(0x562e8b63a6a0 msgr2=0x562e8b721640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:43.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.988+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/652411703 shutdown_connections 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.988+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/652411703 wait complete. 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.989+0000 7f7a9c8aab80 1 -- start start 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.989+0000 7f7a9c8aab80 1 Processor -- start 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b7092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b6da9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b6daf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e8b62bd20 con 0x562e8b6d7b60 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e8b5c7770 con 0x562e8b706830 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e8b5ea6f0 con 0x562e8b6d9be0 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a9c8aab80 10 client.0 _ll_register_callbacks cb 0x562e8b5d15e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a81e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b7092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a81e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b7092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56538/0 (socket says 172.21.15.12:56538) 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a81e00700 1 -- 172.21.15.12:0/630329417 learned_addr learned my addr 172.21.15.12:0/630329417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:43.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a82800700 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b6daf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.990+0000 7f7a81400700 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b6da9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a81e00700 1 -- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 msgr2=0x562e8b6da9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a81e00700 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b6da9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a81e00700 1 -- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 msgr2=0x562e8b6daf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a81e00700 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b6daf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a81e00700 1 -- 172.21.15.12:0/630329417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a680035b0 con 0x562e8b6d7b60 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a81e00700 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b7092f0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f7a74007f70 tx=0x7f7a74000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6e6f77b8f712fe server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a63400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a63400700 1 -- 172.21.15.12:0/630329417 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a740062c0 con 0x562e8b6d7b60 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a63400700 1 -- 172.21.15.12:0/630329417 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f7a7401b050 con 0x562e8b6d7b60 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/630329417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562e8b6ce920 con 0x562e8b6d7b60 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a63400700 1 -- 172.21.15.12:0/630329417 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a740058e0 con 0x562e8b6d7b60 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.991+0000 7f7a9c8aab80 10 client.12762 fetch_fsmap learned FSMap version 138 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.992+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/630329417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x562e8b50c0a0 con 0x562e8b6d7b60 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.992+0000 7f7a63400700 1 -- 172.21.15.12:0/630329417 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562e8b6ce920 con 0x562e8b6d7b60 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.992+0000 7f7a63400700 1 -- 172.21.15.12:0/630329417 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 138) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7a74012070 con 0x562e8b6d7b60 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.992+0000 7f7a9c8aab80 10 client.12762 fetch_fsmap finished waiting for FSMap version 138 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.992+0000 7f7a9c8aab80 10 client.12762 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.992+0000 7f7a9c8aab80 20 client.12762 populate_metadata read hostname 'smithi012' 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.992+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:6835/2746367434,v1:172.21.15.159:6837/2746367434] conn(0x562e8b71fa80 0x562e8b707c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.992+0000 7f7a9c8aab80 4 client.12762 mds_command: new command op to 12061 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.992+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/630329417 --> [v2:172.21.15.159:6835/2746367434,v1:172.21.15.159:6837/2746367434] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562e8b6ab6f0 con 0x562e8b71fa80 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.992+0000 7f7a81400700 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:6835/2746367434,v1:172.21.15.159:6837/2746367434] conn(0x562e8b71fa80 0x562e8b707c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:43.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.993+0000 7f7a81400700 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:6835/2746367434,v1:172.21.15.159:6837/2746367434] conn(0x562e8b71fa80 0x562e8b707c80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70c9cc1f5dcadb8b server_cookie=c55e7ea768236c36 in_seq=0 out_seq=0 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.993+0000 7f7a63400700 10 client.12762 ms_handle_connect on v2:172.21.15.159:6835/2746367434 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a63400700 1 -- 172.21.15.12:0/630329417 <== mds.0 v2:172.21.15.159:6835/2746367434 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562e8b6ab6f0 con 0x562e8b71fa80 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a63400700 10 client.12762 handle_command_reply: tid=0 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 client.12762 shutdown 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/630329417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 msgr2=0x562e8b7092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b7092f0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f7a74007f70 tx=0x7f7a74000ed0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/630329417 shutdown_connections 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:6835/2746367434,v1:172.21.15.159:6837/2746367434] conn(0x562e8b71fa80 0x562e8b707c80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b6daf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b7092f0 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/630329417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b6da9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a63400700 0 client.12762 ms_handle_reset on v2:172.21.15.159:6835/2746367434 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/630329417 >> 172.21.15.12:0/630329417 conn(0x562e8b63a6a0 msgr2=0x562e8b6dbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:43.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/630329417 shutdown_connections 2024-03-29T11:33:43.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.994+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/630329417 wait complete. 2024-03-29T11:33:43.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:43.995+0000 7f7a9c8aab80 20 client.12762 trim_cache size 0 max 16384 2024-03-29T11:33:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.013+0000 7f7a9c8aab80 1 Processor -- start 2024-03-29T11:33:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.013+0000 7f7a9c8aab80 1 -- start start 2024-03-29T11:33:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.013+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b73a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b6dd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b720050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e8b420390 con 0x562e8b6d7b60 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e8b718ef0 con 0x562e8b706830 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e8b6d2880 con 0x562e8b6d9be0 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b73a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b73a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56542/0 (socket says 172.21.15.12:56542) 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81e00700 1 -- 172.21.15.12:0/772681967 learned_addr learned my addr 172.21.15.12:0/772681967 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81400700 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b6dd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81e00700 1 -- 172.21.15.12:0/772681967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 msgr2=0x562e8b6dd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81e00700 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b6dd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a82800700 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b720050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81e00700 1 -- 172.21.15.12:0/772681967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 msgr2=0x562e8b720050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81e00700 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b720050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81e00700 1 -- 172.21.15.12:0/772681967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e8b5f2010 con 0x562e8b6d7b60 2024-03-29T11:33:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81400700 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b6dd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a82800700 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b720050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.014+0000 7f7a81e00700 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b73a500 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f7a74005ba0 tx=0x7f7a7402d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b683b28029747 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a6f400700 1 -- 172.21.15.12:0/772681967 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74021c80 con 0x562e8b6d7b60 2024-03-29T11:33:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a6f400700 1 -- 172.21.15.12:0/772681967 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a74021a70 con 0x562e8b6d7b60 2024-03-29T11:33:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a6f400700 1 -- 172.21.15.12:0/772681967 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74036670 con 0x562e8b6d7b60 2024-03-29T11:33:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/772681967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 msgr2=0x562e8b73a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b73a500 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f7a74005ba0 tx=0x7f7a7402d000 comp rx=0 tx=0).stop 2024-03-29T11:33:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/772681967 shutdown_connections 2024-03-29T11:33:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b720050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b73a500 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/772681967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b6dd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/772681967 >> 172.21.15.12:0/772681967 conn(0x562e8b63a6a0 msgr2=0x562e8b6da9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/772681967 shutdown_connections 2024-03-29T11:33:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.015+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/772681967 wait complete. 2024-03-29T11:33:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.016+0000 7f7a9c8aab80 1 -- start start 2024-03-29T11:33:44.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.016+0000 7f7a9c8aab80 1 Processor -- start 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.017+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b723740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.017+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b71ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.017+0000 7f7a9c8aab80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b6dbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.017+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e8b71c220 con 0x562e8b6d7b60 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.017+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e8b7045b0 con 0x562e8b706830 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.017+0000 7f7a9c8aab80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e8b6db770 con 0x562e8b6d9be0 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.017+0000 7f7a9c8aab80 10 client.0 _ll_register_callbacks cb 0x7f7a70001310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.017+0000 7f7a81e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b723740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.017+0000 7f7a81e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b723740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56550/0 (socket says 172.21.15.12:56550) 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.017+0000 7f7a81e00700 1 -- 172.21.15.12:0/2352987320 learned_addr learned my addr 172.21.15.12:0/2352987320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.018+0000 7f7a81400700 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b71ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.018+0000 7f7a82800700 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b6dbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.018+0000 7f7a81e00700 1 -- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 msgr2=0x562e8b71ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.018+0000 7f7a81e00700 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b71ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.018+0000 7f7a81e00700 1 -- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 msgr2=0x562e8b6dbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.018+0000 7f7a81e00700 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b6dbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.018+0000 7f7a81e00700 1 -- 172.21.15.12:0/2352987320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562e8b50c0a0 con 0x562e8b6d7b60 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.018+0000 7f7a81e00700 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b723740 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f7a740011e0 tx=0x7f7a7400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b2279c63c5d8e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.018+0000 7f7a63400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.018+0000 7f7a63400700 1 -- 172.21.15.12:0/2352987320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74021c80 con 0x562e8b6d7b60 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a63400700 1 -- 172.21.15.12:0/2352987320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f7a7401b030 con 0x562e8b6d7b60 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/2352987320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a74006420 con 0x562e8b6d7b60 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a63400700 1 -- 172.21.15.12:0/2352987320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a74036e20 con 0x562e8b6d7b60 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a9c8aab80 10 client.12768 fetch_fsmap learned FSMap version 138 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/2352987320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x562e8b4801d0 con 0x562e8b6d7b60 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a63400700 1 -- 172.21.15.12:0/2352987320 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a74006420 con 0x562e8b6d7b60 2024-03-29T11:33:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a63400700 1 -- 172.21.15.12:0/2352987320 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 138) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7a74012070 con 0x562e8b6d7b60 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a9c8aab80 10 client.12768 fetch_fsmap finished waiting for FSMap version 138 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a9c8aab80 10 client.12768 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a9c8aab80 20 client.12768 populate_metadata read hostname 'smithi012' 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:6835/2746367434,v1:172.21.15.159:6837/2746367434] conn(0x562e8b6dc420 0x562e8b781dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a9c8aab80 4 client.12768 mds_command: new command op to 12061 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.019+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/2352987320 --> [v2:172.21.15.159:6835/2746367434,v1:172.21.15.159:6837/2746367434] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562e8b465040 con 0x562e8b6dc420 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.020+0000 7f7a81400700 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:6835/2746367434,v1:172.21.15.159:6837/2746367434] conn(0x562e8b6dc420 0x562e8b781dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.020+0000 7f7a81400700 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:6835/2746367434,v1:172.21.15.159:6837/2746367434] conn(0x562e8b6dc420 0x562e8b781dc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d36122b1ced886c4 server_cookie=de9c23bf3f70c8b0 in_seq=0 out_seq=0 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.020+0000 7f7a63400700 10 client.12768 ms_handle_connect on v2:172.21.15.159:6835/2746367434 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.020+0000 7f7a63400700 1 -- 172.21.15.12:0/2352987320 <== mds.0 v2:172.21.15.159:6835/2746367434 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x562e8b465040 con 0x562e8b6dc420 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.020+0000 7f7a63400700 10 client.12768 handle_command_reply: tid=0 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.020+0000 7f7a9c8aab80 1 client.12768 shutdown 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/2352987320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 msgr2=0x562e8b723740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b723740 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f7a740011e0 tx=0x7f7a7400a000 comp rx=0 tx=0).stop 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/2352987320 shutdown_connections 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:6835/2746367434,v1:172.21.15.159:6837/2746367434] conn(0x562e8b6dc420 0x562e8b781dc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x562e8b706830 0x562e8b6dbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562e8b6d7b60 0x562e8b723740 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a63400700 0 client.12768 ms_handle_reset on v2:172.21.15.159:6835/2746367434 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 1 --2- 172.21.15.12:0/2352987320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x562e8b6d9be0 0x562e8b71ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/2352987320 >> 172.21.15.12:0/2352987320 conn(0x562e8b63a6a0 msgr2=0x562e8b720050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/2352987320 shutdown_connections 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 1 -- 172.21.15.12:0/2352987320 wait complete. 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.021+0000 7f7a9c8aab80 20 client.12768 trim_cache size 0 max 16384 2024-03-29T11:33:44.023 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 12061, 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1548626610000001, 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 138, 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 269, 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 269, 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12.708346647000001 2024-03-29T11:33:44.024 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:33:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 -- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a6406aed0 msgr2=0x7f7a6406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a6406aed0 0x7f7a6406d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7a74007fd0 tx=0x7f7a74007dc0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 -- 172.21.15.12:0/4161690547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a840d21f0 msgr2=0x7f7a84158530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a840d21f0 0x7f7a84158530 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f7a7000c520 tx=0x7f7a7000c830 comp rx=0 tx=0).stop 2024-03-29T11:33:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 -- 172.21.15.12:0/4161690547 shutdown_connections 2024-03-29T11:33:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7a6406aed0 0x7f7a6406d390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a841611b0 0x7f7a84158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a840cee10 0x7f7a84156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 --2- 172.21.15.12:0/4161690547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a840d21f0 0x7f7a84158530 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 -- 172.21.15.12:0/4161690547 >> 172.21.15.12:0/4161690547 conn(0x7f7a8408aff0 msgr2=0x7f7a84159930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 -- 172.21.15.12:0/4161690547 shutdown_connections 2024-03-29T11:33:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.022+0000 7f7a88e00700 1 -- 172.21.15.12:0/4161690547 wait complete. 2024-03-29T11:33:44.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T11:33:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.238+0000 7fe604e00700 1 Processor -- start 2024-03-29T11:33:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.238+0000 7fe604e00700 1 -- start start 2024-03-29T11:33:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.239+0000 7fe604e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe600160f10 0x7fe6000d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.239+0000 7fe604e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 0x7fe6000d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.239+0000 7fe604e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6000ceda0 con 0x7fe600161590 2024-03-29T11:33:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.239+0000 7fe604e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6000d23d0 con 0x7fe600160f10 2024-03-29T11:33:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.239+0000 7fe604e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6001612f0 con 0x7fe6000cf230 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fde00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe600160f10 0x7fe6000d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fde00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe600160f10 0x7fe6000d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56566/0 (socket says 172.21.15.12:56566) 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fe800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe600161590 0x7fe6000b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49710/0 (socket says 172.21.15.12:49710) 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fde00700 1 -- 172.21.15.12:0/3511893047 learned_addr learned my addr 172.21.15.12:0/3511893047 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fd400700 1 --2- 172.21.15.12:0/3511893047 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 0x7fe6000d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fca00700 1 -- 172.21.15.12:0/3511893047 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383266281 0 0) 0x7fe6000ceda0 con 0x7fe600161590 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fca00700 1 -- 172.21.15.12:0/3511893047 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e40035b0 con 0x7fe600161590 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fde00700 1 -- 172.21.15.12:0/3511893047 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 msgr2=0x7fe6000d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fde00700 1 --2- 172.21.15.12:0/3511893047 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 0x7fe6000d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fde00700 1 -- 172.21.15.12:0/3511893047 >> v1:172.21.15.12:6789/0 conn(0x7fe600161590 legacy=0x7fe6000b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.240+0000 7fe5fde00700 1 -- 172.21.15.12:0/3511893047 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6000ce470 con 0x7fe600160f10 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.241+0000 7fe5fde00700 1 --2- 172.21.15.12:0/3511893047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 0x7fe6000d18b0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00098f0 tx=0x7fe5f0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9536571ef04c708f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.241+0000 7fe5fca00700 1 -- 172.21.15.12:0/3511893047 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0011830 con 0x7fe600160f10 2024-03-29T11:33:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.241+0000 7fe5fca00700 1 -- 172.21.15.12:0/3511893047 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f0011990 con 0x7fe600160f10 2024-03-29T11:33:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.241+0000 7fe5fca00700 1 -- 172.21.15.12:0/3511893047 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0011d20 con 0x7fe600160f10 2024-03-29T11:33:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.241+0000 7fe604e00700 1 -- 172.21.15.12:0/3511893047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 msgr2=0x7fe6000d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.241+0000 7fe604e00700 1 --2- 172.21.15.12:0/3511893047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 0x7fe6000d18b0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00098f0 tx=0x7fe5f0009c00 comp rx=0 tx=0).stop 2024-03-29T11:33:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.241+0000 7fe604e00700 1 -- 172.21.15.12:0/3511893047 shutdown_connections 2024-03-29T11:33:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.241+0000 7fe604e00700 1 --2- 172.21.15.12:0/3511893047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 0x7fe6000d18b0 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.241+0000 7fe604e00700 1 --2- 172.21.15.12:0/3511893047 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 0x7fe6000d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.241+0000 7fe604e00700 1 -- 172.21.15.12:0/3511893047 >> 172.21.15.12:0/3511893047 conn(0x7fe60008aff0 msgr2=0x7fe6000d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.242+0000 7fe604e00700 1 -- 172.21.15.12:0/3511893047 shutdown_connections 2024-03-29T11:33:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.242+0000 7fe604e00700 1 -- 172.21.15.12:0/3511893047 wait complete. 2024-03-29T11:33:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.243+0000 7fe604e00700 1 Processor -- start 2024-03-29T11:33:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.243+0000 7fe604e00700 1 -- start start 2024-03-29T11:33:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.244+0000 7fe604e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 0x7fe6000b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.244+0000 7fe604e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 0x7fe60009efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.244+0000 7fe604e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe600161590 0x7fe6001f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.244+0000 7fe604e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60014c250 con 0x7fe600160f10 2024-03-29T11:33:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.244+0000 7fe604e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6001f3580 con 0x7fe600161590 2024-03-29T11:33:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.244+0000 7fe604e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60009f5f0 con 0x7fe6000cf230 2024-03-29T11:33:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 0x7fe60009efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 0x7fe60009efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56572/0 (socket says 172.21.15.12:56572) 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fd400700 1 -- 172.21.15.12:0/56083688 learned_addr learned my addr 172.21.15.12:0/56083688 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fde00700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 0x7fe6000b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fd400700 1 -- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 msgr2=0x7fe6000b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fe800700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe600161590 0x7fe6001f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fd400700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 0x7fe6000b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fd400700 1 -- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe600161590 msgr2=0x7fe6001f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fd400700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe600161590 0x7fe6001f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fd400700 1 -- 172.21.15.12:0/56083688 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f0009550 con 0x7fe600160f10 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fde00700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 0x7fe6000b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fe800700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe600161590 0x7fe6001f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.245+0000 7fe5fd400700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 0x7fe60009efa0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00e870 tx=0x7fe5ec00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.246+0000 7fe5f6000700 1 -- 172.21.15.12:0/56083688 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec009590 con 0x7fe600160f10 2024-03-29T11:33:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.246+0000 7fe5f6000700 1 -- 172.21.15.12:0/56083688 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec00ee70 con 0x7fe600160f10 2024-03-29T11:33:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.246+0000 7fe604e00700 1 -- 172.21.15.12:0/56083688 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60009f8d0 con 0x7fe600160f10 2024-03-29T11:33:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.246+0000 7fe5f6000700 1 -- 172.21.15.12:0/56083688 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec00ca50 con 0x7fe600160f10 2024-03-29T11:33:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.246+0000 7fe604e00700 1 -- 172.21.15.12:0/56083688 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60009fe90 con 0x7fe600160f10 2024-03-29T11:33:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.247+0000 7fe604e00700 1 -- 172.21.15.12:0/56083688 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6000a04e0 con 0x7fe600160f10 2024-03-29T11:33:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.248+0000 7fe5f6000700 1 -- 172.21.15.12:0/56083688 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe5ec00cbb0 con 0x7fe600160f10 2024-03-29T11:33:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.249+0000 7fe5f6000700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5e006aed0 0x7fe5e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.249+0000 7fe5f6000700 1 -- 172.21.15.12:0/56083688 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fe5ec089750 con 0x7fe600160f10 2024-03-29T11:33:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.249+0000 7fe5fde00700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5e006aed0 0x7fe5e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.250+0000 7fe5fde00700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5e006aed0 0x7fe5e006d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0014040 tx=0x7fe5f0008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:44.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.253+0000 7fe5f6000700 1 -- 172.21.15.12:0/56083688 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5ec054580 con 0x7fe600160f10 2024-03-29T11:33:44.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.520+0000 7fe604e00700 1 -- 172.21.15.12:0/56083688 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe600067720 con 0x7fe600160f10 2024-03-29T11:33:44.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.520+0000 7fe5f6000700 1 -- 172.21.15.12:0/56083688 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v138) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe5ec00ce60 con 0x7fe600160f10 2024-03-29T11:33:44.634 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T11:33:44.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 -- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5e006aed0 msgr2=0x7fe5e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5e006aed0 0x7fe5e006d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0014040 tx=0x7fe5f0008030 comp rx=0 tx=0).stop 2024-03-29T11:33:44.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 -- 172.21.15.12:0/56083688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 msgr2=0x7fe60009efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 0x7fe60009efa0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00e870 tx=0x7fe5ec00ec30 comp rx=0 tx=0).stop 2024-03-29T11:33:44.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 -- 172.21.15.12:0/56083688 shutdown_connections 2024-03-29T11:33:44.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe5e006aed0 0x7fe5e006d390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe600161590 0x7fe6001f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6000cf230 0x7fe6000b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 --2- 172.21.15.12:0/56083688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe600160f10 0x7fe60009efa0 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:44.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 -- 172.21.15.12:0/56083688 >> 172.21.15.12:0/56083688 conn(0x7fe60008aff0 msgr2=0x7fe60015b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:44.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 -- 172.21.15.12:0/56083688 shutdown_connections 2024-03-29T11:33:44.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:44.523+0000 7fe604e00700 1 -- 172.21.15.12:0/56083688 wait complete. 2024-03-29T11:33:44.637 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T11:33:44.637 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T11:33:44.637 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T11:33:44.637 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T11:33:44.637 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T11:33:44.637 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T11:33:44.637 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T11:33:44.637 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T11:33:44.637 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:33:44.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:44.638 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T11:33:44.692 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T11:33:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T11:33:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T11:33:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:33:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T11:33:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:33:44.692 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:33:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:33:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 35493sec preferred_lft 35493sec 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T11:33:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:33:44.694 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:33:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T11:33:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T11:33:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:33:44.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:44.696 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T11:33:44.807 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T11:33:44.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:44.810 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T11:33:44.862 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T11:33:44.862 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T11:33:44.862 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T11:33:44.862 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T11:33:44.863 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T11:33:44.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:44.867 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T11:33:44.867 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T11:33:44.867 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T11:33:44.868 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T11:33:44.868 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T11:33:44.868 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T11:33:45.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:45.073 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T11:33:45.118 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T11:33:45.119 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T11:33:45.119 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:33:45.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:45.120 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T11:33:45.229 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:33:45.229 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T11:33:45.275 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:33:45.275 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:33:45.275 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:33:45.360 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T11:33:45.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:45.360 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:33:45.504 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:33:45.504 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:33:45.532 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:33:45.532 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:33:45.532 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T11:33:45.532 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T11:33:45.533 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T11:33:45.537 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T11:33:45.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:45.538 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:33:45.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:45.605 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T11:33:45.712 INFO:teuthology.orchestra.run.smithi012.stdout:client10409 2024-03-29T11:33:45.712 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:33:45.712 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:45.757 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T11:33:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.000+0000 7f3520600700 1 Processor -- start 2024-03-29T11:33:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.000+0000 7f3520600700 1 -- start start 2024-03-29T11:33:45.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.001+0000 7f3520600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 0x7f351815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:45.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.001+0000 7f3520600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f351815f4d0 0x7f35180ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:45.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.001+0000 7f3520600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35180ce270 con 0x7f35180d21f0 2024-03-29T11:33:45.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.001+0000 7f3520600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351815b600 con 0x7f351815f4d0 2024-03-29T11:33:45.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.001+0000 7f3520600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35180d2760 con 0x7f35181611b0 2024-03-29T11:33:46.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.002+0000 7f351d200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f351815f4d0 0x7f35180ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.002+0000 7f351dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 0x7f351815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.002+0000 7f351e600700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f35180d21f0 0x7f3518161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:56232/0 (socket says 172.21.15.12:56232) 2024-03-29T11:33:46.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.002+0000 7f351e600700 1 -- 172.21.15.12:0/368452054 learned_addr learned my addr 172.21.15.12:0/368452054 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:46.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.002+0000 7f3517e00700 1 -- 172.21.15.12:0/368452054 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474697480 0 0) 0x7f35180ce270 con 0x7f35180d21f0 2024-03-29T11:33:46.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.002+0000 7f3517e00700 1 -- 172.21.15.12:0/368452054 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35000035b0 con 0x7f35180d21f0 2024-03-29T11:33:46.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.002+0000 7f351dc00700 1 -- 172.21.15.12:0/368452054 >> v2:172.21.15.159:3300/0 conn(0x7f351815f4d0 msgr2=0x7f35180ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:46.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.002+0000 7f351dc00700 1 --2- 172.21.15.12:0/368452054 >> v2:172.21.15.159:3300/0 conn(0x7f351815f4d0 0x7f35180ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.002+0000 7f351dc00700 1 -- 172.21.15.12:0/368452054 >> v1:172.21.15.159:6789/0 conn(0x7f35180d21f0 legacy=0x7f3518161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.002+0000 7f351dc00700 1 -- 172.21.15.12:0/368452054 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35180d2a40 con 0x7f35181611b0 2024-03-29T11:33:46.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.003+0000 7f351dc00700 1 --2- 172.21.15.12:0/368452054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 0x7f351815ef90 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f350c009840 tx=0x7f350c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=828af2ea567a4716 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:46.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.003+0000 7f3517e00700 1 -- 172.21.15.12:0/368452054 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c011830 con 0x7f35181611b0 2024-03-29T11:33:46.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.003+0000 7f3517e00700 1 -- 172.21.15.12:0/368452054 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c011990 con 0x7f35181611b0 2024-03-29T11:33:46.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.003+0000 7f3517e00700 1 -- 172.21.15.12:0/368452054 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c011d20 con 0x7f35181611b0 2024-03-29T11:33:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.004+0000 7f3520600700 1 -- 172.21.15.12:0/368452054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 msgr2=0x7f351815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.004+0000 7f3520600700 1 --2- 172.21.15.12:0/368452054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 0x7f351815ef90 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f350c009840 tx=0x7f350c009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.004+0000 7f3520600700 1 -- 172.21.15.12:0/368452054 shutdown_connections 2024-03-29T11:33:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.004+0000 7f3520600700 1 --2- 172.21.15.12:0/368452054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 0x7f351815ef90 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.004+0000 7f3520600700 1 --2- 172.21.15.12:0/368452054 >> v2:172.21.15.159:3300/0 conn(0x7f351815f4d0 0x7f35180ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.004+0000 7f3520600700 1 -- 172.21.15.12:0/368452054 >> 172.21.15.12:0/368452054 conn(0x7f351808aff0 msgr2=0x7f35180d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.004+0000 7f3520600700 1 -- 172.21.15.12:0/368452054 shutdown_connections 2024-03-29T11:33:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.004+0000 7f3520600700 1 -- 172.21.15.12:0/368452054 wait complete. 2024-03-29T11:33:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.006+0000 7f3520600700 1 Processor -- start 2024-03-29T11:33:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.006+0000 7f3520600700 1 -- start start 2024-03-29T11:33:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f3520600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35180d21f0 0x7f35180cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f3520600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f351815f4d0 0x7f35180b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f3520600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 0x7f35181f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f3520600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35180b7a30 con 0x7f351815f4d0 2024-03-29T11:33:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f3520600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35180cb940 con 0x7f35180d21f0 2024-03-29T11:33:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f3520600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35180b6b60 con 0x7f35181611b0 2024-03-29T11:33:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f351d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f351815f4d0 0x7f35180b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f351d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f351815f4d0 0x7f35180b7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56582/0 (socket says 172.21.15.12:56582) 2024-03-29T11:33:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f351d200700 1 -- 172.21.15.12:0/473491066 learned_addr learned my addr 172.21.15.12:0/473491066 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f351dc00700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35180d21f0 0x7f35180cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.007+0000 7f351e600700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 0x7f35181f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f351d200700 1 -- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 msgr2=0x7f35181f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f351d200700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 0x7f35181f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f351d200700 1 -- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35180d21f0 msgr2=0x7f35180cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f351d200700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35180d21f0 0x7f35180cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f351d200700 1 -- 172.21.15.12:0/473491066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c0094a0 con 0x7f351815f4d0 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f351d200700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f351815f4d0 0x7f35180b7ed0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f350400e9b0 tx=0x7f350400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f3515600700 1 -- 172.21.15.12:0/473491066 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3504009710 con 0x7f351815f4d0 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f3515600700 1 -- 172.21.15.12:0/473491066 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350400ee50 con 0x7f351815f4d0 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f3520600700 1 -- 172.21.15.12:0/473491066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35180b6db0 con 0x7f351815f4d0 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f3515600700 1 -- 172.21.15.12:0/473491066 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350400ca90 con 0x7f351815f4d0 2024-03-29T11:33:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.008+0000 7f3520600700 1 -- 172.21.15.12:0/473491066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35180b7370 con 0x7f351815f4d0 2024-03-29T11:33:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.009+0000 7f3520600700 1 -- 172.21.15.12:0/473491066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3518067720 con 0x7f351815f4d0 2024-03-29T11:33:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.010+0000 7f3515600700 1 -- 172.21.15.12:0/473491066 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f350400cbf0 con 0x7f351815f4d0 2024-03-29T11:33:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.011+0000 7f3515600700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f34f806aed0 0x7f34f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.011+0000 7f3515600700 1 -- 172.21.15.12:0/473491066 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f3504089630 con 0x7f351815f4d0 2024-03-29T11:33:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.011+0000 7f351dc00700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f34f806aed0 0x7f34f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.012+0000 7f351dc00700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f34f806aed0 0x7f34f806d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f350c000c00 tx=0x7f350c01c040 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.016+0000 7f3515600700 1 -- 172.21.15.12:0/473491066 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f35040544f0 con 0x7f351815f4d0 2024-03-29T11:33:46.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.258+0000 7f3520600700 1 -- 172.21.15.12:0/473491066 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f35180cdf00 con 0x7f34f806aed0 2024-03-29T11:33:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.261+0000 7f3515600700 1 -- 172.21.15.12:0/473491066 <== mgr.11755 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f350c011b00 con 0x7f34f806aed0 2024-03-29T11:33:46.257 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:33:46.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:33:46.257 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T11:33:46.257 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:33:46.257 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:33:46.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.262+0000 7f3520600700 1 -- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f34f806aed0 msgr2=0x7f34f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.262+0000 7f3520600700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f34f806aed0 0x7f34f806d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f350c000c00 tx=0x7f350c01c040 comp rx=0 tx=0).stop 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.262+0000 7f3520600700 1 -- 172.21.15.12:0/473491066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f351815f4d0 msgr2=0x7f35180b7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.262+0000 7f3520600700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f351815f4d0 0x7f35180b7ed0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f350400e9b0 tx=0x7f350400ecc0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.262+0000 7f3520600700 1 -- 172.21.15.12:0/473491066 shutdown_connections 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.262+0000 7f3520600700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f34f806aed0 0x7f34f806d390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.262+0000 7f3520600700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35181611b0 0x7f35181f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.262+0000 7f3520600700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35180d21f0 0x7f35180cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.262+0000 7f3520600700 1 --2- 172.21.15.12:0/473491066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f351815f4d0 0x7f35180b7ed0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.262+0000 7f3520600700 1 -- 172.21.15.12:0/473491066 >> 172.21.15.12:0/473491066 conn(0x7f351808aff0 msgr2=0x7f351815da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.263+0000 7f3520600700 1 -- 172.21.15.12:0/473491066 shutdown_connections 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.263+0000 7f3520600700 1 -- 172.21.15.12:0/473491066 wait complete. 2024-03-29T11:33:46.278 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T11:33:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.483+0000 7f7955800700 1 Processor -- start 2024-03-29T11:33:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.484+0000 7f7955800700 1 -- start start 2024-03-29T11:33:46.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.485+0000 7f7955800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f795015d9d0 0x7f79500d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.485+0000 7f7955800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7950161420 con 0x7f79500d2540 2024-03-29T11:33:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.485+0000 7f7955800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7950161700 con 0x7f79500d16a0 2024-03-29T11:33:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.485+0000 7f7955800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f795015edc0 con 0x7f795015d9d0 2024-03-29T11:33:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.485+0000 7f794ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f795015d9d0 0x7f79500d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.485+0000 7f794ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f795015d9d0 0x7f79500d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56586/0 (socket says 172.21.15.12:56586) 2024-03-29T11:33:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.485+0000 7f794ea00700 1 -- 172.21.15.12:0/3382796832 learned_addr learned my addr 172.21.15.12:0/3382796832 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:46.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.485+0000 7f794f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f79500d2540 0x7f79500ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49714/0 (socket says 172.21.15.12:49714) 2024-03-29T11:33:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.485+0000 7f794ea00700 1 -- 172.21.15.12:0/3382796832 >> v1:172.21.15.159:6789/0 conn(0x7f79500d16a0 legacy=0x7f79500d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.485+0000 7f794ea00700 1 -- 172.21.15.12:0/3382796832 >> v1:172.21.15.12:6789/0 conn(0x7f79500d2540 legacy=0x7f79500ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.486+0000 7f794ea00700 1 -- 172.21.15.12:0/3382796832 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79501619e0 con 0x7f795015d9d0 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.486+0000 7f794ea00700 1 --2- 172.21.15.12:0/3382796832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 0x7f79500d3c30 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f79400098f0 tx=0x7f7940009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efddbcbe4db1d2bf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.486+0000 7f794d600700 1 -- 172.21.15.12:0/3382796832 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794000b8a0 con 0x7f795015d9d0 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.486+0000 7f794d600700 1 -- 172.21.15.12:0/3382796832 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794000ba00 con 0x7f795015d9d0 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.486+0000 7f794d600700 1 -- 172.21.15.12:0/3382796832 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794000bd90 con 0x7f795015d9d0 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.487+0000 7f7955800700 1 -- 172.21.15.12:0/3382796832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 msgr2=0x7f79500d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.487+0000 7f7955800700 1 --2- 172.21.15.12:0/3382796832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 0x7f79500d3c30 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f79400098f0 tx=0x7f7940009c00 comp rx=0 tx=0).stop 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.487+0000 7f7955800700 1 -- 172.21.15.12:0/3382796832 shutdown_connections 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.487+0000 7f7955800700 1 --2- 172.21.15.12:0/3382796832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 0x7f79500d3c30 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.487+0000 7f7955800700 1 -- 172.21.15.12:0/3382796832 >> 172.21.15.12:0/3382796832 conn(0x7f795008aff0 msgr2=0x7f79500d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.487+0000 7f7955800700 1 -- 172.21.15.12:0/3382796832 shutdown_connections 2024-03-29T11:33:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.487+0000 7f7955800700 1 -- 172.21.15.12:0/3382796832 wait complete. 2024-03-29T11:33:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.488+0000 7f7955800700 1 Processor -- start 2024-03-29T11:33:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.489+0000 7f7955800700 1 -- start start 2024-03-29T11:33:46.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f7955800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79500d16a0 0x7f79501e8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:46.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f7955800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79500d2540 0x7f79500b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:46.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f7955800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 0x7f79500b8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f7955800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79501e8470 con 0x7f795015d9d0 2024-03-29T11:33:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f7955800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79501e77f0 con 0x7f79500d2540 2024-03-29T11:33:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f7955800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79501e7ad0 con 0x7f79500d16a0 2024-03-29T11:33:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f794f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 0x7f79500b8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f794f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 0x7f79500b8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56602/0 (socket says 172.21.15.12:56602) 2024-03-29T11:33:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f794f400700 1 -- 172.21.15.12:0/3929074437 learned_addr learned my addr 172.21.15.12:0/3929074437 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f794ea00700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79500d16a0 0x7f79501e8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f794e000700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79500d2540 0x7f79500b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f794f400700 1 -- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79500d16a0 msgr2=0x7f79501e8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f794f400700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79500d16a0 0x7f79501e8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f794f400700 1 -- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79500d2540 msgr2=0x7f79500b6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f794f400700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79500d2540 0x7f79500b6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.490+0000 7f794f400700 1 -- 172.21.15.12:0/3929074437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7940009550 con 0x7f795015d9d0 2024-03-29T11:33:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.491+0000 7f794f400700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 0x7f79500b8200 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f794800cba0 tx=0x7f794800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.491+0000 7f7946a00700 1 -- 172.21.15.12:0/3929074437 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794800ab80 con 0x7f795015d9d0 2024-03-29T11:33:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.491+0000 7f7946a00700 1 -- 172.21.15.12:0/3929074437 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794800ace0 con 0x7f795015d9d0 2024-03-29T11:33:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.491+0000 7f7946a00700 1 -- 172.21.15.12:0/3929074437 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7948007270 con 0x7f795015d9d0 2024-03-29T11:33:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.491+0000 7f7955800700 1 -- 172.21.15.12:0/3929074437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79501e7db0 con 0x7f795015d9d0 2024-03-29T11:33:46.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.491+0000 7f7955800700 1 -- 172.21.15.12:0/3929074437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79500b7220 con 0x7f795015d9d0 2024-03-29T11:33:46.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.492+0000 7f7955800700 1 -- 172.21.15.12:0/3929074437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79500d1ca0 con 0x7f795015d9d0 2024-03-29T11:33:46.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.493+0000 7f7946a00700 1 -- 172.21.15.12:0/3929074437 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 147) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f79480073d0 con 0x7f795015d9d0 2024-03-29T11:33:46.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.494+0000 7f7946a00700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f793006aed0 0x7f793006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:46.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.494+0000 7f7946a00700 1 -- 172.21.15.12:0/3929074437 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f79480894c0 con 0x7f795015d9d0 2024-03-29T11:33:46.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.494+0000 7f794ea00700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f793006aed0 0x7f793006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:46.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.495+0000 7f794ea00700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f793006aed0 0x7f793006d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7940009840 tx=0x7f7940008030 comp rx=0 tx=0).ready entity=mgr.11755 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.498+0000 7f7946a00700 1 -- 172.21.15.12:0/3929074437 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7948054390 con 0x7f795015d9d0 2024-03-29T11:33:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:46.773+0000 7f7955800700 1 -- 172.21.15.12:0/3929074437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7950067720 con 0x7f795015d9d0 2024-03-29T11:33:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.375+0000 7f7946a00700 1 -- 172.21.15.12:0/3929074437 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 148) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f7948054760 con 0x7f795015d9d0 2024-03-29T11:33:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.377+0000 7f7946a00700 1 -- 172.21.15.12:0/3929074437 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v148) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7948073ea0 con 0x7f795015d9d0 2024-03-29T11:33:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 -- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f793006aed0 msgr2=0x7f793006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f793006aed0 0x7f793006d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7940009840 tx=0x7f7940008030 comp rx=0 tx=0).stop 2024-03-29T11:33:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 -- 172.21.15.12:0/3929074437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 msgr2=0x7f79500b8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 0x7f79500b8200 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f794800cba0 tx=0x7f794800cf60 comp rx=0 tx=0).stop 2024-03-29T11:33:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 -- 172.21.15.12:0/3929074437 shutdown_connections 2024-03-29T11:33:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f793006aed0 0x7f793006d390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f795015d9d0 0x7f79500b8200 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79500d16a0 0x7f79501e8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 --2- 172.21.15.12:0/3929074437 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79500d2540 0x7f79500b6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 -- 172.21.15.12:0/3929074437 >> 172.21.15.12:0/3929074437 conn(0x7f795008aff0 msgr2=0x7f79500ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 -- 172.21.15.12:0/3929074437 shutdown_connections 2024-03-29T11:33:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.380+0000 7f7955800700 1 -- 172.21.15.12:0/3929074437 wait complete. 2024-03-29T11:33:47.399 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T11:33:47.602 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:47.606+0000 7f0646470240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:33:47.605 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:47.608+0000 7f01ff120240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:33:47.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.610+0000 7f3586c00700 1 Processor -- start 2024-03-29T11:33:47.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.611+0000 7f3586c00700 1 -- start start 2024-03-29T11:33:47.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.612+0000 7f3586c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35800cd020 0x7f35800cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:47.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.612+0000 7f3586c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35800cdae0 con 0x7f35800b2070 2024-03-29T11:33:47.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.612+0000 7f3586c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35800cddc0 con 0x7f35800cbb30 2024-03-29T11:33:47.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.612+0000 7f3586c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35800ce0a0 con 0x7f35800cd020 2024-03-29T11:33:47.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.612+0000 7f357f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f35800cd020 0x7f35800cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:47.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.612+0000 7f3584c00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f35800b2070 0x7f358006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49730/0 (socket says 172.21.15.12:49730) 2024-03-29T11:33:47.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.612+0000 7f3584c00700 1 -- 172.21.15.12:0/2429286269 learned_addr learned my addr 172.21.15.12:0/2429286269 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:47.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.612+0000 7f357f400700 1 -- 172.21.15.12:0/2429286269 >> v1:172.21.15.159:6789/0 conn(0x7f35800cbb30 legacy=0x7f35800cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:47.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.613+0000 7f357f400700 1 -- 172.21.15.12:0/2429286269 >> v1:172.21.15.12:6789/0 conn(0x7f35800b2070 legacy=0x7f358006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:47.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.613+0000 7f357f400700 1 -- 172.21.15.12:0/2429286269 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358006dc20 con 0x7f35800cd020 2024-03-29T11:33:47.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.613+0000 7f357f400700 1 --2- 172.21.15.12:0/2429286269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800cd020 0x7f35800cd480 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f356c009840 tx=0x7f356c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ae6440adb6b474 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.613+0000 7f357ea00700 1 -- 172.21.15.12:0/2429286269 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c011830 con 0x7f35800cd020 2024-03-29T11:33:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.613+0000 7f357ea00700 1 -- 172.21.15.12:0/2429286269 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356c011990 con 0x7f35800cd020 2024-03-29T11:33:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.613+0000 7f357ea00700 1 -- 172.21.15.12:0/2429286269 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c011d20 con 0x7f35800cd020 2024-03-29T11:33:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.614+0000 7f3586c00700 1 -- 172.21.15.12:0/2429286269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800cd020 msgr2=0x7f35800cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.614+0000 7f3586c00700 1 --2- 172.21.15.12:0/2429286269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800cd020 0x7f35800cd480 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f356c009840 tx=0x7f356c009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.614+0000 7f3586c00700 1 -- 172.21.15.12:0/2429286269 shutdown_connections 2024-03-29T11:33:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.614+0000 7f3586c00700 1 --2- 172.21.15.12:0/2429286269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800cd020 0x7f35800cd480 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.614+0000 7f3586c00700 1 -- 172.21.15.12:0/2429286269 >> 172.21.15.12:0/2429286269 conn(0x7f35800b0d70 msgr2=0x7f35800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:47.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.614+0000 7f3586c00700 1 -- 172.21.15.12:0/2429286269 shutdown_connections 2024-03-29T11:33:47.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.614+0000 7f3586c00700 1 -- 172.21.15.12:0/2429286269 wait complete. 2024-03-29T11:33:47.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.616+0000 7f3586c00700 1 Processor -- start 2024-03-29T11:33:47.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.616+0000 7f3586c00700 1 -- start start 2024-03-29T11:33:47.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f3586c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800b2070 0x7f35800c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:47.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f3586c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35800cbb30 0x7f3580072380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:47.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f3586c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35800cd020 0x7f358015de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f3586c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35800c95e0 con 0x7f35800b2070 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f3586c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35800c8960 con 0x7f35800cbb30 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f3586c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35800c8c40 con 0x7f35800cd020 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f357fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800b2070 0x7f35800c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f357fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800b2070 0x7f35800c9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56630/0 (socket says 172.21.15.12:56630) 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f357fe00700 1 -- 172.21.15.12:0/983726237 learned_addr learned my addr 172.21.15.12:0/983726237 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f3584c00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35800cd020 0x7f358015de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f357f400700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35800cbb30 0x7f3580072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f357fe00700 1 -- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35800cd020 msgr2=0x7f358015de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f357fe00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35800cd020 0x7f358015de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f357fe00700 1 -- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35800cbb30 msgr2=0x7f3580072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f357fe00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35800cbb30 0x7f3580072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.617+0000 7f357fe00700 1 -- 172.21.15.12:0/983726237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0094a0 con 0x7f35800b2070 2024-03-29T11:33:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.618+0000 7f357fe00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800b2070 0x7f35800c9a00 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f357000c4e0 tx=0x7f357000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.618+0000 7f3577e00700 1 -- 172.21.15.12:0/983726237 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35700052d0 con 0x7f35800b2070 2024-03-29T11:33:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.618+0000 7f3577e00700 1 -- 172.21.15.12:0/983726237 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3570005430 con 0x7f35800b2070 2024-03-29T11:33:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.618+0000 7f3577e00700 1 -- 172.21.15.12:0/983726237 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35700038c0 con 0x7f35800b2070 2024-03-29T11:33:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.618+0000 7f3586c00700 1 -- 172.21.15.12:0/983726237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35800c8f20 con 0x7f35800b2070 2024-03-29T11:33:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.618+0000 7f3586c00700 1 -- 172.21.15.12:0/983726237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35800728c0 con 0x7f35800b2070 2024-03-29T11:33:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.620+0000 7f3586c00700 1 -- 172.21.15.12:0/983726237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3580067720 con 0x7f35800b2070 2024-03-29T11:33:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.620+0000 7f3577e00700 1 -- 172.21.15.12:0/983726237 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 148) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f3570003a20 con 0x7f35800b2070 2024-03-29T11:33:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.621+0000 7f3577e00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f356006af20 0x7f356006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.621+0000 7f3577e00700 1 -- 172.21.15.12:0/983726237 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f3570088b60 con 0x7f35800b2070 2024-03-29T11:33:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.621+0000 7f357f400700 1 -- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f356006af20 msgr2=0x7f356006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.621+0000 7f357f400700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f356006af20 0x7f356006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:47.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.625+0000 7f3577e00700 1 -- 172.21.15.12:0/983726237 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f35700539c0 con 0x7f35800b2070 2024-03-29T11:33:47.789 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:47.793+0000 7f0646470240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:33:47.816 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:47.820+0000 7f01ff120240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:33:47.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.822+0000 7f357f400700 1 -- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f356006af20 msgr2=0x7f356006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.822+0000 7f357f400700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f356006af20 0x7f356006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:47.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.882+0000 7f3586c00700 1 -- 172.21.15.12:0/983726237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f35800ced90 con 0x7f35800b2070 2024-03-29T11:33:47.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.887+0000 7f3577e00700 1 -- 172.21.15.12:0/983726237 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7f35700170d0 con 0x7f35800b2070 2024-03-29T11:33:47.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 -- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f356006af20 msgr2=0x7f356006d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:47.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f356006af20 0x7f356006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 -- 172.21.15.12:0/983726237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800b2070 msgr2=0x7f35800c9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:47.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800b2070 0x7f35800c9a00 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f357000c4e0 tx=0x7f357000c7f0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 -- 172.21.15.12:0/983726237 shutdown_connections 2024-03-29T11:33:47.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f356006af20 0x7f356006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35800cd020 0x7f358015de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35800b2070 0x7f35800c9a00 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 --2- 172.21.15.12:0/983726237 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35800cbb30 0x7f3580072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:47.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 -- 172.21.15.12:0/983726237 >> 172.21.15.12:0/983726237 conn(0x7f35800b0d70 msgr2=0x7f35800aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:47.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 -- 172.21.15.12:0/983726237 shutdown_connections 2024-03-29T11:33:47.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:47.890+0000 7f3586c00700 1 -- 172.21.15.12:0/983726237 wait complete. 2024-03-29T11:33:47.915 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T11:33:48.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.123+0000 7fcc6c400700 1 Processor -- start 2024-03-29T11:33:48.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.123+0000 7fcc6c400700 1 -- start start 2024-03-29T11:33:48.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.124+0000 7fcc6c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc640b4420 0x7fcc640ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.124+0000 7fcc6c400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcc640af600 0x7fcc6406dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.124+0000 7fcc6c400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc640ade10 con 0x7fcc6415a680 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.124+0000 7fcc6c400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc640ae040 con 0x7fcc640af600 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.124+0000 7fcc6c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc640ae320 con 0x7fcc640b4420 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.124+0000 7fcc6a400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcc640af600 0x7fcc6406dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.125+0000 7fcc69a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcc6415a680 0x7fcc640b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49734/0 (socket says 172.21.15.12:49734) 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.125+0000 7fcc69a00700 1 -- 172.21.15.12:0/2136825626 learned_addr learned my addr 172.21.15.12:0/2136825626 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.125+0000 7fcc69000700 1 --2- 172.21.15.12:0/2136825626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc640b4420 0x7fcc640ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.125+0000 7fcc6a400700 1 -- 172.21.15.12:0/2136825626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc640b4420 msgr2=0x7fcc640ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.125+0000 7fcc6a400700 1 --2- 172.21.15.12:0/2136825626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc640b4420 0x7fcc640ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.125+0000 7fcc6a400700 1 -- 172.21.15.12:0/2136825626 >> v1:172.21.15.12:6789/0 conn(0x7fcc6415a680 legacy=0x7fcc640b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.125+0000 7fcc6a400700 1 -- 172.21.15.12:0/2136825626 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc640ae600 con 0x7fcc640af600 2024-03-29T11:33:48.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.125+0000 7fcc6a400700 1 --2- 172.21.15.12:0/2136825626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 0x7fcc6406dd40 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009d90 tx=0x7fcc5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4828a05ba5ce7d90 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.126+0000 7fcc63e00700 1 -- 172.21.15.12:0/2136825626 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c00e070 con 0x7fcc640af600 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.126+0000 7fcc63e00700 1 -- 172.21.15.12:0/2136825626 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c00bb20 con 0x7fcc640af600 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.126+0000 7fcc63e00700 1 -- 172.21.15.12:0/2136825626 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c012430 con 0x7fcc640af600 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.126+0000 7fcc6c400700 1 -- 172.21.15.12:0/2136825626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 msgr2=0x7fcc6406dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.126+0000 7fcc6c400700 1 --2- 172.21.15.12:0/2136825626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 0x7fcc6406dd40 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009d90 tx=0x7fcc5c00b330 comp rx=0 tx=0).stop 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.126+0000 7fcc6c400700 1 -- 172.21.15.12:0/2136825626 shutdown_connections 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.126+0000 7fcc6c400700 1 --2- 172.21.15.12:0/2136825626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 0x7fcc6406dd40 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.126+0000 7fcc6c400700 1 --2- 172.21.15.12:0/2136825626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc640b4420 0x7fcc640ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.126+0000 7fcc6c400700 1 -- 172.21.15.12:0/2136825626 >> 172.21.15.12:0/2136825626 conn(0x7fcc640b5720 msgr2=0x7fcc640b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.126+0000 7fcc6c400700 1 -- 172.21.15.12:0/2136825626 shutdown_connections 2024-03-29T11:33:48.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.127+0000 7fcc6c400700 1 -- 172.21.15.12:0/2136825626 wait complete. 2024-03-29T11:33:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.128+0000 7fcc6c400700 1 Processor -- start 2024-03-29T11:33:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.128+0000 7fcc6c400700 1 -- start start 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.129+0000 7fcc6c400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 0x7fcc6415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.129+0000 7fcc6c400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc640b4420 0x7fcc6406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.129+0000 7fcc6c400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6415a680 0x7fcc640b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.129+0000 7fcc6c400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6415b1e0 con 0x7fcc640af600 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.129+0000 7fcc6c400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6415cfd0 con 0x7fcc640b4420 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.129+0000 7fcc6c400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6415ccf0 con 0x7fcc6415a680 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.129+0000 7fcc69a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 0x7fcc6415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.130+0000 7fcc69a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 0x7fcc6415c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56658/0 (socket says 172.21.15.12:56658) 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.130+0000 7fcc69a00700 1 -- 172.21.15.12:0/637104775 learned_addr learned my addr 172.21.15.12:0/637104775 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.130+0000 7fcc6a400700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6415a680 0x7fcc640b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.130+0000 7fcc69000700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc640b4420 0x7fcc6406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.130+0000 7fcc69a00700 1 -- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6415a680 msgr2=0x7fcc640b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:48.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.130+0000 7fcc69a00700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6415a680 0x7fcc640b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.130+0000 7fcc69a00700 1 -- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc640b4420 msgr2=0x7fcc6406f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:48.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.130+0000 7fcc69a00700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc640b4420 0x7fcc6406f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.130+0000 7fcc69a00700 1 -- 172.21.15.12:0/637104775 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c009a70 con 0x7fcc640af600 2024-03-29T11:33:48.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.130+0000 7fcc69a00700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 0x7fcc6415c3a0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800c480 tx=0x7fcc5800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:48.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.131+0000 7fcc61600700 1 -- 172.21.15.12:0/637104775 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58005d70 con 0x7fcc640af600 2024-03-29T11:33:48.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.131+0000 7fcc61600700 1 -- 172.21.15.12:0/637104775 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5800ce90 con 0x7fcc640af600 2024-03-29T11:33:48.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.131+0000 7fcc6c400700 1 -- 172.21.15.12:0/637104775 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6415ab90 con 0x7fcc640af600 2024-03-29T11:33:48.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.131+0000 7fcc61600700 1 -- 172.21.15.12:0/637104775 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc580072d0 con 0x7fcc640af600 2024-03-29T11:33:48.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.131+0000 7fcc6c400700 1 -- 172.21.15.12:0/637104775 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6415c8e0 con 0x7fcc640af600 2024-03-29T11:33:48.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.132+0000 7fcc6c400700 1 -- 172.21.15.12:0/637104775 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6415caa0 con 0x7fcc640af600 2024-03-29T11:33:48.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.133+0000 7fcc61600700 1 -- 172.21.15.12:0/637104775 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 148) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fcc58007430 con 0x7fcc640af600 2024-03-29T11:33:48.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.133+0000 7fcc61600700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc4406af20 0x7fcc4406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.134+0000 7fcc61600700 1 -- 172.21.15.12:0/637104775 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fcc58088ab0 con 0x7fcc640af600 2024-03-29T11:33:48.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.134+0000 7fcc69000700 1 -- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc4406af20 msgr2=0x7fcc4406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:48.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.134+0000 7fcc69000700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc4406af20 0x7fcc4406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:48.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.138+0000 7fcc61600700 1 -- 172.21.15.12:0/637104775 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcc58053900 con 0x7fcc640af600 2024-03-29T11:33:48.135 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:48.139+0000 7f0646470240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:33:48.166 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:48.169+0000 7f01ff120240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:33:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.335+0000 7fcc69000700 1 -- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc4406af20 msgr2=0x7fcc4406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.335+0000 7fcc69000700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc4406af20 0x7fcc4406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.357+0000 7fcc6c400700 1 -- 172.21.15.12:0/637104775 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fcc6415ee30 con 0x7fcc640af600 2024-03-29T11:33:48.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.363+0000 7fcc61600700 1 -- 172.21.15.12:0/637104775 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcc58056f20 con 0x7fcc640af600 2024-03-29T11:33:48.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.365+0000 7fcc6c400700 1 -- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc4406af20 msgr2=0x7fcc4406d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:48.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.365+0000 7fcc6c400700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc4406af20 0x7fcc4406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.365+0000 7fcc6c400700 1 -- 172.21.15.12:0/637104775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 msgr2=0x7fcc6415c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.365+0000 7fcc6c400700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 0x7fcc6415c3a0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800c480 tx=0x7fcc5800c790 comp rx=0 tx=0).stop 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.365+0000 7fcc6c400700 1 -- 172.21.15.12:0/637104775 shutdown_connections 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.365+0000 7fcc6c400700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc4406af20 0x7fcc4406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.365+0000 7fcc6c400700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6415a680 0x7fcc640b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.365+0000 7fcc6c400700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc640af600 0x7fcc6415c3a0 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.365+0000 7fcc6c400700 1 --2- 172.21.15.12:0/637104775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc640b4420 0x7fcc6406f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.365+0000 7fcc6c400700 1 -- 172.21.15.12:0/637104775 >> 172.21.15.12:0/637104775 conn(0x7fcc640b5720 msgr2=0x7fcc640b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.366+0000 7fcc6c400700 1 -- 172.21.15.12:0/637104775 shutdown_connections 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.366+0000 7fcc6c400700 1 -- 172.21.15.12:0/637104775 wait complete. 2024-03-29T11:33:48.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:48.382 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-29T11:33:48.431 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-03-29T11:33:48.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.639+0000 7fd5ebe00700 1 Processor -- start 2024-03-29T11:33:48.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.639+0000 7fd5ebe00700 1 -- start start 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.640+0000 7fd5ebe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5ec15d230 0x7fd5ec0d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.640+0000 7fd5ebe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5ec15d610 0x7fd5ec0d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.640+0000 7fd5ebe00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5ec0b6a40 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.640+0000 7fd5ebe00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ec0b6cf0 con 0x7fd5ec15d610 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.640+0000 7fd5ebe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0b6fd0 con 0x7fd5ec15d230 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.640+0000 7fd5eb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5ec15d610 0x7fd5ec0d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.641+0000 7fd5ea000700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fd5ec0b8fe0 0x7fd5ec0b6330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:56266/0 (socket says 172.21.15.12:56266) 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.641+0000 7fd5eb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5ec15d610 0x7fd5ec0d02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56668/0 (socket says 172.21.15.12:56668) 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.641+0000 7fd5ea000700 1 -- 172.21.15.12:0/1129675422 learned_addr learned my addr 172.21.15.12:0/1129675422 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.641+0000 7fd5eaa00700 1 --2- 172.21.15.12:0/1129675422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5ec15d230 0x7fd5ec0d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:48.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.641+0000 7fd5eb400700 1 -- 172.21.15.12:0/1129675422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5ec15d230 msgr2=0x7fd5ec0d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:48.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.641+0000 7fd5eb400700 1 --2- 172.21.15.12:0/1129675422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5ec15d230 0x7fd5ec0d1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.641+0000 7fd5eb400700 1 -- 172.21.15.12:0/1129675422 >> v1:172.21.15.159:6789/0 conn(0x7fd5ec0b8fe0 legacy=0x7fd5ec0b6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:48.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.641+0000 7fd5e9600700 1 -- 172.21.15.12:0/1129675422 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511891732 0 0) 0x7fd5ec0b6a40 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.641+0000 7fd5eb400700 1 -- 172.21.15.12:0/1129675422 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec0b72b0 con 0x7fd5ec15d610 2024-03-29T11:33:48.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.641+0000 7fd5eb400700 1 --2- 172.21.15.12:0/1129675422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec15d610 0x7fd5ec0d02e0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009d90 tx=0x7fd5e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73fe84bec5b7740 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:48.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.642+0000 7fd5e9600700 1 -- 172.21.15.12:0/1129675422 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400e070 con 0x7fd5ec15d610 2024-03-29T11:33:48.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.642+0000 7fd5e9600700 1 -- 172.21.15.12:0/1129675422 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e400bb20 con 0x7fd5ec15d610 2024-03-29T11:33:48.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.642+0000 7fd5e9600700 1 -- 172.21.15.12:0/1129675422 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4012430 con 0x7fd5ec15d610 2024-03-29T11:33:48.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.642+0000 7fd5ebe00700 1 -- 172.21.15.12:0/1129675422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec15d610 msgr2=0x7fd5ec0d02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:48.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.642+0000 7fd5ebe00700 1 --2- 172.21.15.12:0/1129675422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec15d610 0x7fd5ec0d02e0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009d90 tx=0x7fd5e400b330 comp rx=0 tx=0).stop 2024-03-29T11:33:48.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.642+0000 7fd5ebe00700 1 -- 172.21.15.12:0/1129675422 shutdown_connections 2024-03-29T11:33:48.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.642+0000 7fd5ebe00700 1 --2- 172.21.15.12:0/1129675422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec15d610 0x7fd5ec0d02e0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.642+0000 7fd5ebe00700 1 --2- 172.21.15.12:0/1129675422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5ec15d230 0x7fd5ec0d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.642+0000 7fd5ebe00700 1 -- 172.21.15.12:0/1129675422 >> 172.21.15.12:0/1129675422 conn(0x7fd5ec0668e0 msgr2=0x7fd5ec066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:48.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.642+0000 7fd5ebe00700 1 -- 172.21.15.12:0/1129675422 shutdown_connections 2024-03-29T11:33:48.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.643+0000 7fd5ebe00700 1 -- 172.21.15.12:0/1129675422 wait complete. 2024-03-29T11:33:48.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.645+0000 7fd5ebe00700 1 Processor -- start 2024-03-29T11:33:48.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.645+0000 7fd5ebe00700 1 -- start start 2024-03-29T11:33:48.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.646+0000 7fd5ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec0b8fe0 0x7fd5ec160700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.646+0000 7fd5ebe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5ec15d230 0x7fd5ec14d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.646+0000 7fd5ebe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5ec15d610 0x7fd5ec0cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.646+0000 7fd5ebe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec160c40 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.646+0000 7fd5ebe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec14dcd0 con 0x7fd5ec15d230 2024-03-29T11:33:48.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.646+0000 7fd5ebe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0cbdf0 con 0x7fd5ec15d610 2024-03-29T11:33:48.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec0b8fe0 0x7fd5ec160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:48.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec0b8fe0 0x7fd5ec160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56670/0 (socket says 172.21.15.12:56670) 2024-03-29T11:33:48.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5eaa00700 1 -- 172.21.15.12:0/892649320 learned_addr learned my addr 172.21.15.12:0/892649320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:48.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5eb400700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5ec15d610 0x7fd5ec0cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:48.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5ea000700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5ec15d230 0x7fd5ec14d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:48.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5eaa00700 1 -- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5ec15d610 msgr2=0x7fd5ec0cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:48.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5eaa00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5ec15d610 0x7fd5ec0cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5eaa00700 1 -- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5ec15d230 msgr2=0x7fd5ec14d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:48.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5eaa00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5ec15d230 0x7fd5ec14d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:48.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5eaa00700 1 -- 172.21.15.12:0/892649320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e4009a70 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.647+0000 7fd5eaa00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec0b8fe0 0x7fd5ec160700 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00e9b0 tx=0x7fd5dc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:48.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.648+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc009710 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.648+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5dc00ee50 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.648+0000 7fd5ebe00700 1 -- 172.21.15.12:0/892649320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec0cc0d0 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.648+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc00ca50 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.648+0000 7fd5ebe00700 1 -- 172.21.15.12:0/892649320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec0cc690 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.649+0000 7fd5ebe00700 1 -- 172.21.15.12:0/892649320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ec0ccce0 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.650+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 148) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fd5dc00cbb0 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.651+0000 7fd5e2a00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:48.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.651+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fd5dc089860 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.651+0000 7fd5ea000700 1 -- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 msgr2=0x7fd5cc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:48.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.651+0000 7fd5ea000700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:48.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.655+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd5dc0545f0 con 0x7fd5ec0b8fe0 2024-03-29T11:33:48.669 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:48.673+0000 7f0646470240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:33:48.733 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:48.736+0000 7f01ff120240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:33:48.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.852+0000 7fd5ea000700 1 -- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 msgr2=0x7fd5cc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:48.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.852+0000 7fd5ea000700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:48.895+0000 7fd5ebe00700 1 -- 172.21.15.12:0/892649320 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5ec0b7870 con 0x7fd5cc06af20 2024-03-29T11:33:48.927 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:48.931+0000 7f0646470240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:33:49.000 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:49.003+0000 7f01ff120240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:33:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:49.253+0000 7fd5ea000700 1 -- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 msgr2=0x7fd5cc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:49.253+0000 7fd5ea000700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T11:33:49.387 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:49.391+0000 7f0646470240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:33:49.479 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:49.483+0000 7f01ff120240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:33:50.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:50.054+0000 7fd5ea000700 1 -- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 msgr2=0x7fd5cc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:50.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:50.054+0000 7fd5ea000700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T11:33:50.099 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:50.103+0000 7f0646470240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:33:50.191 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:50.195+0000 7f0646470240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:33:50.223 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:50.225+0000 7f01ff120240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:33:50.320 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:50.322+0000 7f01ff120240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:33:50.350 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:50.354+0000 7f0646470240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:33:50.488 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:50.490+0000 7f01ff120240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:33:50.565 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:50.568+0000 7f0646470240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:33:50.657 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:50.661+0000 7f0646470240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:33:50.708 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:50.711+0000 7f01ff120240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:33:50.756 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:50.760+0000 7f0646470240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:33:50.804 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:50.806+0000 7f01ff120240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:33:50.855 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:50.859+0000 7f0646470240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:33:50.908 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:50.910+0000 7f01ff120240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:33:50.954 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:50.958+0000 7f0646470240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:33:51.012 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:51.014+0000 7f01ff120240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:33:51.115 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:51.117+0000 7f01ff120240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:33:51.166 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:51.170+0000 7f0646470240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:33:51.338 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:51.340+0000 7f01ff120240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:33:51.427 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:51.431+0000 7f0646470240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:33:51.592 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:33:51.596 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:33:51.602 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:51.605+0000 7f01ff120240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:33:51.622 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:51.626+0000 7f0646470240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:33:51.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:51.654+0000 7fd5ea000700 1 -- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 msgr2=0x7fd5cc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:51.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:51.654+0000 7fd5ea000700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T11:33:51.721 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:51.725+0000 7f0646470240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:51.805 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:51.807+0000 7f01ff120240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:33:51.903 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:51.905+0000 7f01ff120240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:52.459 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:52.463+0000 7f0646470240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:33:52.667 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:52.670+0000 7f01ff120240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:33:52.674 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:52.678+0000 7f0646470240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:33:52.880 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:52.884+0000 7f0646470240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:52.892 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:52.895+0000 7f01ff120240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:33:53.109 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:53.112+0000 7f01ff120240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:53.358 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:53.362+0000 7f0646470240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:33:53.566 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:53.570+0000 7f0646470240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:53.613 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:53.616+0000 7f01ff120240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:33:53.831 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:53.834+0000 7f01ff120240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:33:54.012 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.016+0000 7f0646470240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:33:54.126 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.130+0000 7f0646470240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:54.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:54.139+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 149) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fd5dc0549c0 con 0x7fd5ec0b8fe0 2024-03-29T11:33:54.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:54.140+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 msgr2=0x7fd5cc06d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:54.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:54.140+0000 7fd5e2a00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:54.296 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:54.299+0000 7f01ff120240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:33:54.315 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.319+0000 7f05ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:54.315 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.319+0000 7f05ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:54.315 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.319+0000 7f05ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:54.315 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.319+0000 7f05ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:54.315 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.319+0000 7f05ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:54.315 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.319+0000 7f05b2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:54.316 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.319+0000 7f05b2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:54.316 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.319+0000 7f05b2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:54.316 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.319+0000 7f05b2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:54.316 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:54.319+0000 7f05b2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:33:54.417 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:54.420+0000 7f01ff120240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:33:55.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.144+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 150) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fd5dc06fec0 con 0x7fd5ec0b8fe0 2024-03-29T11:33:55.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.144+0000 7fd5e2a00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:55.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.144+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5cc035800 con 0x7fd5cc06af20 2024-03-29T11:33:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.147+0000 7fd5ea000700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.152+0000 7fd5ea000700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0d1fc0 tx=0x7fd5d8007de0 comp rx=0 tx=0).ready entity=mgr.12217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:55.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.220+0000 7fd5e2a00700 1 -- 172.21.15.12:0/892649320 <== mgr.12217 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd5cc035800 con 0x7fd5cc06af20 2024-03-29T11:33:55.218 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path snap_test_dir1 2024-03-29T11:33:55.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 -- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 msgr2=0x7fd5cc06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:55.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0d1fc0 tx=0x7fd5d8007de0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 -- 172.21.15.12:0/892649320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec0b8fe0 msgr2=0x7fd5ec160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:55.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec0b8fe0 0x7fd5ec160700 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00e9b0 tx=0x7fd5dc00ecc0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 -- 172.21.15.12:0/892649320 shutdown_connections 2024-03-29T11:33:55.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd5cc06af20 0x7fd5cc06d3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5ec15d610 0x7fd5ec0cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec0b8fe0 0x7fd5ec160700 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 --2- 172.21.15.12:0/892649320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5ec15d230 0x7fd5ec14d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 -- 172.21.15.12:0/892649320 >> 172.21.15.12:0/892649320 conn(0x7fd5ec0668e0 msgr2=0x7fd5ec0d02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:55.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 -- 172.21.15.12:0/892649320 shutdown_connections 2024-03-29T11:33:55.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.223+0000 7fd5ebe00700 1 -- 172.21.15.12:0/892649320 wait complete. 2024-03-29T11:33:55.242 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-29T11:33:55.242 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-03-29T11:33:55.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.448+0000 7fe362600700 1 Processor -- start 2024-03-29T11:33:55.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.448+0000 7fe362600700 1 -- start start 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.449+0000 7fe362600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe35c155ac0 0x7fe35c15c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.449+0000 7fe362600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe35c0ae670 0x7fe35c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.449+0000 7fe362600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c06dd00 con 0x7fe35c06c910 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.449+0000 7fe362600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35c06d0f0 con 0x7fe35c0ae670 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.449+0000 7fe362600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c06d3d0 con 0x7fe35c155ac0 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.449+0000 7fe35be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe35c0ae670 0x7fe35c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.449+0000 7fe35be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe35c0ae670 0x7fe35c15ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54744/0 (socket says 172.21.15.12:54744) 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.449+0000 7fe35be00700 1 -- 172.21.15.12:0/2857987191 learned_addr learned my addr 172.21.15.12:0/2857987191 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.450+0000 7fe35b400700 1 --2- 172.21.15.12:0/2857987191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe35c155ac0 0x7fe35c15c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.450+0000 7fe35be00700 1 -- 172.21.15.12:0/2857987191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe35c155ac0 msgr2=0x7fe35c15c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.450+0000 7fe35be00700 1 --2- 172.21.15.12:0/2857987191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe35c155ac0 0x7fe35c15c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.450+0000 7fe35be00700 1 -- 172.21.15.12:0/2857987191 >> v1:172.21.15.159:6789/0 conn(0x7fe35c06c910 legacy=0x7fe35c0adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:55.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.450+0000 7fe35be00700 1 -- 172.21.15.12:0/2857987191 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c06d6b0 con 0x7fe35c0ae670 2024-03-29T11:33:55.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.450+0000 7fe35be00700 1 --2- 172.21.15.12:0/2857987191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 0x7fe35c15ccb0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fe354009e50 tx=0x7fe35400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df09e67089cbe4f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.450+0000 7fe35a000700 1 -- 172.21.15.12:0/2857987191 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400e070 con 0x7fe35c0ae670 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.450+0000 7fe35a000700 1 -- 172.21.15.12:0/2857987191 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35400b910 con 0x7fe35c0ae670 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.450+0000 7fe35a000700 1 -- 172.21.15.12:0/2857987191 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400bcc0 con 0x7fe35c0ae670 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.451+0000 7fe362600700 1 -- 172.21.15.12:0/2857987191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 msgr2=0x7fe35c15ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.451+0000 7fe362600700 1 --2- 172.21.15.12:0/2857987191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 0x7fe35c15ccb0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fe354009e50 tx=0x7fe35400b330 comp rx=0 tx=0).stop 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.451+0000 7fe362600700 1 -- 172.21.15.12:0/2857987191 shutdown_connections 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.451+0000 7fe362600700 1 --2- 172.21.15.12:0/2857987191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 0x7fe35c15ccb0 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.451+0000 7fe362600700 1 --2- 172.21.15.12:0/2857987191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe35c155ac0 0x7fe35c15c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.451+0000 7fe362600700 1 -- 172.21.15.12:0/2857987191 >> 172.21.15.12:0/2857987191 conn(0x7fe35c0cf210 msgr2=0x7fe35c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.451+0000 7fe362600700 1 -- 172.21.15.12:0/2857987191 shutdown_connections 2024-03-29T11:33:55.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.451+0000 7fe362600700 1 -- 172.21.15.12:0/2857987191 wait complete. 2024-03-29T11:33:55.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.453+0000 7fe362600700 1 Processor -- start 2024-03-29T11:33:55.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.453+0000 7fe362600700 1 -- start start 2024-03-29T11:33:55.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe362600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe35c06c910 0x7fe35c1f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:55.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe362600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 0x7fe35c0b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe362600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe35c155ac0 0x7fe35c0b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe362600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c14d380 con 0x7fe35c0ae670 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe362600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c1f3a90 con 0x7fe35c06c910 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe362600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c1f3d70 con 0x7fe35c155ac0 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe35aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 0x7fe35c0b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe35aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 0x7fe35c0b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54752/0 (socket says 172.21.15.12:54752) 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe35aa00700 1 -- 172.21.15.12:0/563524790 learned_addr learned my addr 172.21.15.12:0/563524790 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe35be00700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe35c155ac0 0x7fe35c0b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.454+0000 7fe35b400700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe35c06c910 0x7fe35c1f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe35aa00700 1 -- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe35c155ac0 msgr2=0x7fe35c0b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe35aa00700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe35c155ac0 0x7fe35c0b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe35aa00700 1 -- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe35c06c910 msgr2=0x7fe35c1f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe35aa00700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe35c06c910 0x7fe35c1f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe35aa00700 1 -- 172.21.15.12:0/563524790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe354009b30 con 0x7fe35c0ae670 2024-03-29T11:33:55.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe35aa00700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 0x7fe35c0b32f0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c4e0 tx=0x7fe34800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:55.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe353400700 1 -- 172.21.15.12:0/563524790 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3480052d0 con 0x7fe35c0ae670 2024-03-29T11:33:55.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe353400700 1 -- 172.21.15.12:0/563524790 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe348005430 con 0x7fe35c0ae670 2024-03-29T11:33:55.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe362600700 1 -- 172.21.15.12:0/563524790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c156020 con 0x7fe35c0ae670 2024-03-29T11:33:55.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe353400700 1 -- 172.21.15.12:0/563524790 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3480038c0 con 0x7fe35c0ae670 2024-03-29T11:33:55.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.455+0000 7fe362600700 1 -- 172.21.15.12:0/563524790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35c14d010 con 0x7fe35c0ae670 2024-03-29T11:33:55.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.456+0000 7fe362600700 1 -- 172.21.15.12:0/563524790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35c067720 con 0x7fe35c0ae670 2024-03-29T11:33:55.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.457+0000 7fe353400700 1 -- 172.21.15.12:0/563524790 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 150) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fe348003a20 con 0x7fe35c0ae670 2024-03-29T11:33:55.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.459+0000 7fe353400700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe33c06ab30 0x7fe33c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:55.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.459+0000 7fe353400700 1 -- 172.21.15.12:0/563524790 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7fe348088600 con 0x7fe35c0ae670 2024-03-29T11:33:55.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.459+0000 7fe35b400700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe33c06ab30 0x7fe33c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:55.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.460+0000 7fe35b400700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe33c06ab30 0x7fe33c06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0affe0 tx=0x7fe34c009210 comp rx=0 tx=0).ready entity=mgr.12217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:55.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.464+0000 7fe353400700 1 -- 172.21.15.12:0/563524790 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe348053850 con 0x7fe35c0ae670 2024-03-29T11:33:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.706+0000 7fe362600700 1 -- 172.21.15.12:0/563524790 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe35c1f3ee0 con 0x7fe33c06ab30 2024-03-29T11:33:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.767+0000 7fe353400700 1 -- 172.21.15.12:0/563524790 <== mgr.12217 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fe35c1f3ee0 con 0x7fe33c06ab30 2024-03-29T11:33:55.764 INFO:teuthology.orchestra.run.smithi012.stdout:Retention added to path snap_test_dir1 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 -- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe33c06ab30 msgr2=0x7fe33c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe33c06ab30 0x7fe33c06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0affe0 tx=0x7fe34c009210 comp rx=0 tx=0).stop 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 -- 172.21.15.12:0/563524790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 msgr2=0x7fe35c0b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 0x7fe35c0b32f0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c4e0 tx=0x7fe34800c7f0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 -- 172.21.15.12:0/563524790 shutdown_connections 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe33c06ab30 0x7fe33c06cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe35c155ac0 0x7fe35c0b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe35c06c910 0x7fe35c1f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 --2- 172.21.15.12:0/563524790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe35c0ae670 0x7fe35c0b32f0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 -- 172.21.15.12:0/563524790 >> 172.21.15.12:0/563524790 conn(0x7fe35c0cf210 msgr2=0x7fe35c0b02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 -- 172.21.15.12:0/563524790 shutdown_connections 2024-03-29T11:33:55.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:55.769+0000 7fe362600700 1 -- 172.21.15.12:0/563524790 wait complete. 2024-03-29T11:33:55.785 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-03-29T11:33:55.893 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-03-29-11_34_00 in ~5s... 2024-03-29T11:33:55.894 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-03-29T11:33:55.894 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-03-29T11:33:56.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.097+0000 7f9ecde00700 1 Processor -- start 2024-03-29T11:33:56.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.098+0000 7f9ecde00700 1 -- start start 2024-03-29T11:33:56.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.099+0000 7f9ecde00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ec815d9d0 0x7f9ec80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.099+0000 7f9ecde00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ec80d2540 0x7f9ec80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.099+0000 7f9ecde00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec8161420 con 0x7f9ec80d16a0 2024-03-29T11:33:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.099+0000 7f9ecde00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec815edc0 con 0x7f9ec80d2540 2024-03-29T11:33:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.099+0000 7f9ecde00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec8161700 con 0x7f9ec815d9d0 2024-03-29T11:33:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.099+0000 7f9ec6000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9ec80d16a0 0x7f9ec80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59584/0 (socket says 172.21.15.12:59584) 2024-03-29T11:33:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.099+0000 7f9ec6000700 1 -- 172.21.15.12:0/2494809362 learned_addr learned my addr 172.21.15.12:0/2494809362 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.099+0000 7f9ec7400700 1 --2- 172.21.15.12:0/2494809362 >> v2:172.21.15.12:3300/0 conn(0x7f9ec80d2540 0x7f9ec80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.099+0000 7f9ec6a00700 1 --2- 172.21.15.12:0/2494809362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ec815d9d0 0x7f9ec80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.100+0000 7f9ec7400700 1 -- 172.21.15.12:0/2494809362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ec815d9d0 msgr2=0x7f9ec80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.100+0000 7f9ec7400700 1 --2- 172.21.15.12:0/2494809362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ec815d9d0 0x7f9ec80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.100+0000 7f9ec7400700 1 -- 172.21.15.12:0/2494809362 >> v1:172.21.15.12:6789/0 conn(0x7f9ec80d16a0 legacy=0x7f9ec80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.100+0000 7f9ec7400700 1 -- 172.21.15.12:0/2494809362 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec81619e0 con 0x7f9ec80d2540 2024-03-29T11:33:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.100+0000 7f9ec7400700 1 --2- 172.21.15.12:0/2494809362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 0x7f9ec80ce360 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009dc0 tx=0x7f9ec000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93511e8898150f9c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.100+0000 7f9ec5600700 1 -- 172.21.15.12:0/2494809362 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec000e070 con 0x7f9ec80d2540 2024-03-29T11:33:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.100+0000 7f9ec5600700 1 -- 172.21.15.12:0/2494809362 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec000bae0 con 0x7f9ec80d2540 2024-03-29T11:33:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.100+0000 7f9ec5600700 1 -- 172.21.15.12:0/2494809362 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec000be90 con 0x7f9ec80d2540 2024-03-29T11:33:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.101+0000 7f9ecde00700 1 -- 172.21.15.12:0/2494809362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 msgr2=0x7f9ec80ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.101+0000 7f9ecde00700 1 --2- 172.21.15.12:0/2494809362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 0x7f9ec80ce360 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009dc0 tx=0x7f9ec000b330 comp rx=0 tx=0).stop 2024-03-29T11:33:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.101+0000 7f9ecde00700 1 -- 172.21.15.12:0/2494809362 shutdown_connections 2024-03-29T11:33:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.101+0000 7f9ecde00700 1 --2- 172.21.15.12:0/2494809362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 0x7f9ec80ce360 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.101+0000 7f9ecde00700 1 --2- 172.21.15.12:0/2494809362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ec815d9d0 0x7f9ec80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.101+0000 7f9ecde00700 1 -- 172.21.15.12:0/2494809362 >> 172.21.15.12:0/2494809362 conn(0x7f9ec808aff0 msgr2=0x7f9ec80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.101+0000 7f9ecde00700 1 -- 172.21.15.12:0/2494809362 shutdown_connections 2024-03-29T11:33:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.101+0000 7f9ecde00700 1 -- 172.21.15.12:0/2494809362 wait complete. 2024-03-29T11:33:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.103+0000 7f9ecde00700 1 Processor -- start 2024-03-29T11:33:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.103+0000 7f9ecde00700 1 -- start start 2024-03-29T11:33:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ecde00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ec80d16a0 0x7f9ec80b7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ecde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 0x7f9ec815c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ecde00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ec815d9d0 0x7f9ec81606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ecde00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec80cb540 con 0x7f9ec80d2540 2024-03-29T11:33:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ecde00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec80b7f20 con 0x7f9ec80d16a0 2024-03-29T11:33:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ecde00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec80b8200 con 0x7f9ec815d9d0 2024-03-29T11:33:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ec6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 0x7f9ec815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ec6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 0x7f9ec815c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54776/0 (socket says 172.21.15.12:54776) 2024-03-29T11:33:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ec6000700 1 -- 172.21.15.12:0/3588977471 learned_addr learned my addr 172.21.15.12:0/3588977471 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ec6a00700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ec80d16a0 0x7f9ec80b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ec7400700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ec815d9d0 0x7f9ec81606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ec6000700 1 -- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ec815d9d0 msgr2=0x7f9ec81606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ec6000700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ec815d9d0 0x7f9ec81606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ec6000700 1 -- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ec80d16a0 msgr2=0x7f9ec80b7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ec6000700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ec80d16a0 0x7f9ec80b7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.105+0000 7f9ec6000700 1 -- 172.21.15.12:0/3588977471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec0009a70 con 0x7f9ec80d2540 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.106+0000 7f9ec6000700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 0x7f9ec815c090 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400c4e0 tx=0x7f9eb400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.106+0000 7f9ebf400700 1 -- 172.21.15.12:0/3588977471 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb40074e0 con 0x7f9ec80d2540 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.106+0000 7f9ebf400700 1 -- 172.21.15.12:0/3588977471 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb4007640 con 0x7f9ec80d2540 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.106+0000 7f9ebf400700 1 -- 172.21.15.12:0/3588977471 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4005a50 con 0x7f9ec80d2540 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.106+0000 7f9ecde00700 1 -- 172.21.15.12:0/3588977471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec80b84e0 con 0x7f9ec80d2540 2024-03-29T11:33:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.106+0000 7f9ecde00700 1 -- 172.21.15.12:0/3588977471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec80b7b60 con 0x7f9ec80d2540 2024-03-29T11:33:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.107+0000 7f9ecde00700 1 -- 172.21.15.12:0/3588977471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec80d1ca0 con 0x7f9ec80d2540 2024-03-29T11:33:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.108+0000 7f9ebf400700 1 -- 172.21.15.12:0/3588977471 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 150) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f9eb4005bb0 con 0x7f9ec80d2540 2024-03-29T11:33:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.109+0000 7f9ebf400700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ea806ab30 0x7f9ea806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.109+0000 7f9ebf400700 1 -- 172.21.15.12:0/3588977471 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f9eb4088710 con 0x7f9ec80d2540 2024-03-29T11:33:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.109+0000 7f9ec6a00700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ea806ab30 0x7f9ea806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.110+0000 7f9ec6a00700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ea806ab30 0x7f9ea806cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9ec81e9250 tx=0x7f9eb8009210 comp rx=0 tx=0).ready entity=mgr.12217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:56.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.114+0000 7f9ebf400700 1 -- 172.21.15.12:0/3588977471 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9eb40538e0 con 0x7f9ec80d2540 2024-03-29T11:33:56.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.195+0000 7f9ebf400700 1 -- 172.21.15.12:0/3588977471 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 151) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f9eb4053cb0 con 0x7f9ec80d2540 2024-03-29T11:33:56.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.358+0000 7f9ecde00700 1 -- 172.21.15.12:0/3588977471 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9ec8161fa0 con 0x7f9ea806ab30 2024-03-29T11:33:56.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.367+0000 7f9ebf400700 1 -- 172.21.15.12:0/3588977471 <== mgr.12217 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f9eb8011790 con 0x7f9ea806ab30 2024-03-29T11:33:56.364 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:56.365 INFO:teuthology.orchestra.run.smithi012.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-03-29T11:33:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 -- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ea806ab30 msgr2=0x7f9ea806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ea806ab30 0x7f9ea806cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9ec81e9250 tx=0x7f9eb8009210 comp rx=0 tx=0).stop 2024-03-29T11:33:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 -- 172.21.15.12:0/3588977471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 msgr2=0x7f9ec815c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 0x7f9ec815c090 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400c4e0 tx=0x7f9eb400c7f0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 -- 172.21.15.12:0/3588977471 shutdown_connections 2024-03-29T11:33:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9ea806ab30 0x7f9ea806cff0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ec815d9d0 0x7f9ec81606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ec80d16a0 0x7f9ec80b7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 --2- 172.21.15.12:0/3588977471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ec80d2540 0x7f9ec815c090 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 -- 172.21.15.12:0/3588977471 >> 172.21.15.12:0/3588977471 conn(0x7f9ec808aff0 msgr2=0x7f9ec80b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:56.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 -- 172.21.15.12:0/3588977471 shutdown_connections 2024-03-29T11:33:56.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.369+0000 7f9ecde00700 1 -- 172.21.15.12:0/3588977471 wait complete. 2024-03-29T11:33:56.385 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-03-29T11:33:56.385 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-03-29T11:33:56.385 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-03-29T11:33:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.587+0000 7fc84b000700 1 Processor -- start 2024-03-29T11:33:56.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.587+0000 7fc84b000700 1 -- start start 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.588+0000 7fc84b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440cd020 0x7fc8440cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.588+0000 7fc84b000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8440cdae0 con 0x7fc8440b2070 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.588+0000 7fc84b000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8440cddc0 con 0x7fc8440cbb30 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.588+0000 7fc84b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440ce0a0 con 0x7fc8440cd020 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.589+0000 7fc849000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc8440b2070 0x7fc84406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59596/0 (socket says 172.21.15.12:59596) 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.589+0000 7fc849000700 1 -- 172.21.15.12:0/3693345947 learned_addr learned my addr 172.21.15.12:0/3693345947 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.589+0000 7fc843400700 1 --2- 172.21.15.12:0/3693345947 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440cd020 0x7fc8440cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.589+0000 7fc842a00700 1 -- 172.21.15.12:0/3693345947 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280455410 0 0) 0x7fc8440cdae0 con 0x7fc8440b2070 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.589+0000 7fc842a00700 1 -- 172.21.15.12:0/3693345947 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8280035b0 con 0x7fc8440b2070 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.589+0000 7fc842a00700 1 -- 172.21.15.12:0/3693345947 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596795566 0 0) 0x7fc8440cddc0 con 0x7fc8440cbb30 2024-03-29T11:33:56.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.589+0000 7fc842a00700 1 -- 172.21.15.12:0/3693345947 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8440cdae0 con 0x7fc8440cbb30 2024-03-29T11:33:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.590+0000 7fc842a00700 1 -- 172.21.15.12:0/3693345947 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 724557301 0 0) 0x7fc8280035b0 con 0x7fc8440b2070 2024-03-29T11:33:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.590+0000 7fc843400700 1 -- 172.21.15.12:0/3693345947 >> v1:172.21.15.159:6789/0 conn(0x7fc8440cbb30 legacy=0x7fc8440cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.590+0000 7fc843400700 1 -- 172.21.15.12:0/3693345947 >> v1:172.21.15.12:6789/0 conn(0x7fc8440b2070 legacy=0x7fc84406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.590+0000 7fc843400700 1 -- 172.21.15.12:0/3693345947 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84406dc20 con 0x7fc8440cd020 2024-03-29T11:33:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.590+0000 7fc843400700 1 --2- 172.21.15.12:0/3693345947 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440cd020 0x7fc8440cd480 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc830009870 tx=0x7fc830009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc88a1326a0424a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.590+0000 7fc842a00700 1 -- 172.21.15.12:0/3693345947 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0019f0 con 0x7fc8440cd020 2024-03-29T11:33:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.590+0000 7fc842a00700 1 -- 172.21.15.12:0/3693345947 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc830004030 con 0x7fc8440cd020 2024-03-29T11:33:56.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.590+0000 7fc842a00700 1 -- 172.21.15.12:0/3693345947 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8300043c0 con 0x7fc8440cd020 2024-03-29T11:33:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.591+0000 7fc84b000700 1 -- 172.21.15.12:0/3693345947 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440cd020 msgr2=0x7fc8440cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.591+0000 7fc84b000700 1 --2- 172.21.15.12:0/3693345947 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440cd020 0x7fc8440cd480 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc830009870 tx=0x7fc830009b80 comp rx=0 tx=0).stop 2024-03-29T11:33:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.591+0000 7fc84b000700 1 -- 172.21.15.12:0/3693345947 shutdown_connections 2024-03-29T11:33:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.591+0000 7fc84b000700 1 --2- 172.21.15.12:0/3693345947 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440cd020 0x7fc8440cd480 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.591+0000 7fc84b000700 1 -- 172.21.15.12:0/3693345947 >> 172.21.15.12:0/3693345947 conn(0x7fc8440b0d70 msgr2=0x7fc8440b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:56.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.591+0000 7fc84b000700 1 -- 172.21.15.12:0/3693345947 shutdown_connections 2024-03-29T11:33:56.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.591+0000 7fc84b000700 1 -- 172.21.15.12:0/3693345947 wait complete. 2024-03-29T11:33:56.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.593+0000 7fc84b000700 1 Processor -- start 2024-03-29T11:33:56.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.593+0000 7fc84b000700 1 -- start start 2024-03-29T11:33:56.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc84b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440b2070 0x7fc8440ad4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc84b000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8440cbb30 0x7fc844154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc84b000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440cd020 0x7fc844154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc84b000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8441542b0 con 0x7fc8440cd020 2024-03-29T11:33:56.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc84b000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440ada00 con 0x7fc8440cbb30 2024-03-29T11:33:56.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc84b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc844152f30 con 0x7fc8440b2070 2024-03-29T11:33:56.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc849000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440cd020 0x7fc844154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc849000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440cd020 0x7fc844154eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54780/0 (socket says 172.21.15.12:54780) 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc849000700 1 -- 172.21.15.12:0/2562788202 learned_addr learned my addr 172.21.15.12:0/2562788202 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc843e00700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440b2070 0x7fc8440ad4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc843400700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8440cbb30 0x7fc844154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc849000700 1 -- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440b2070 msgr2=0x7fc8440ad4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc849000700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440b2070 0x7fc8440ad4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc849000700 1 -- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8440cbb30 msgr2=0x7fc844154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc849000700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8440cbb30 0x7fc844154970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc849000700 1 -- 172.21.15.12:0/2562788202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8300094a0 con 0x7fc8440cd020 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.594+0000 7fc843e00700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440b2070 0x7fc8440ad4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.595+0000 7fc849000700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440cd020 0x7fc844154eb0 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00cbc0 tx=0x7fc83c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.595+0000 7fc83be00700 1 -- 172.21.15.12:0/2562788202 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c00ab00 con 0x7fc8440cd020 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.595+0000 7fc83be00700 1 -- 172.21.15.12:0/2562788202 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83c00ac60 con 0x7fc8440cd020 2024-03-29T11:33:56.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.595+0000 7fc83be00700 1 -- 172.21.15.12:0/2562788202 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0070d0 con 0x7fc8440cd020 2024-03-29T11:33:56.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.595+0000 7fc84b000700 1 -- 172.21.15.12:0/2562788202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc844153210 con 0x7fc8440cd020 2024-03-29T11:33:56.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.595+0000 7fc84b000700 1 -- 172.21.15.12:0/2562788202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8441537d0 con 0x7fc8440cd020 2024-03-29T11:33:56.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.596+0000 7fc84b000700 1 -- 172.21.15.12:0/2562788202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc844067720 con 0x7fc8440cd020 2024-03-29T11:33:56.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.597+0000 7fc83be00700 1 -- 172.21.15.12:0/2562788202 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 151) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fc83c007230 con 0x7fc8440cd020 2024-03-29T11:33:56.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.598+0000 7fc83be00700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc82406ae00 0x7fc82406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:56.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.598+0000 7fc83be00700 1 -- 172.21.15.12:0/2562788202 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7fc83c089250 con 0x7fc8440cd020 2024-03-29T11:33:56.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.598+0000 7fc843e00700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc82406ae00 0x7fc82406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:56.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.599+0000 7fc843e00700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc82406ae00 0x7fc82406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc844154640 tx=0x7fc834009040 comp rx=0 tx=0).ready entity=mgr.12217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:56.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.603+0000 7fc83be00700 1 -- 172.21.15.12:0/2562788202 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc83c057b10 con 0x7fc8440cd020 2024-03-29T11:33:56.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.846+0000 7fc84b000700 1 -- 172.21.15.12:0/2562788202 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc84415bdd0 con 0x7fc82406ae00 2024-03-29T11:33:56.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.917+0000 7fc83be00700 1 -- 172.21.15.12:0/2562788202 <== mgr.12217 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc8340050d0 con 0x7fc82406ae00 2024-03-29T11:33:56.915 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule removed for path snap_test_dir1 2024-03-29T11:33:56.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 -- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc82406ae00 msgr2=0x7fc82406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc82406ae00 0x7fc82406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc844154640 tx=0x7fc834009040 comp rx=0 tx=0).stop 2024-03-29T11:33:56.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 -- 172.21.15.12:0/2562788202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440cd020 msgr2=0x7fc844154eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440cd020 0x7fc844154eb0 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00cbc0 tx=0x7fc83c00ced0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 -- 172.21.15.12:0/2562788202 shutdown_connections 2024-03-29T11:33:56.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc82406ae00 0x7fc82406d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440cd020 0x7fc844154eb0 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8440b2070 0x7fc8440ad4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 --2- 172.21.15.12:0/2562788202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8440cbb30 0x7fc844154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 -- 172.21.15.12:0/2562788202 >> 172.21.15.12:0/2562788202 conn(0x7fc8440b0d70 msgr2=0x7fc84415d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 -- 172.21.15.12:0/2562788202 shutdown_connections 2024-03-29T11:33:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:56.920+0000 7fc84b000700 1 -- 172.21.15.12:0/2562788202 wait complete. 2024-03-29T11:33:56.937 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-03-29T11:33:56.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:56.937 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-29T11:33:56.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:33:56.985 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-29T11:33:57.032 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T11:33:57.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.234+0000 7f0b1f600700 1 Processor -- start 2024-03-29T11:33:57.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.234+0000 7f0b1f600700 1 -- start start 2024-03-29T11:33:57.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1f600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0b18160e50 0x7f0b180b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:57.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1f600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b180d3f20 0x7f0b180b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:57.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1f600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0b180b3f80 0x7f0b180b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:57.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1f600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b18161230 con 0x7f0b18160e50 2024-03-29T11:33:57.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1f600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b180d2a80 con 0x7f0b180b3f80 2024-03-29T11:33:57.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1f600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180d43b0 con 0x7f0b180d3f20 2024-03-29T11:33:57.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1cc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0b18160e50 0x7f0b180b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:57.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1cc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0b18160e50 0x7f0b180b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54790/0 (socket says 172.21.15.12:54790) 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1cc00700 1 -- 172.21.15.12:0/1061567789 learned_addr learned my addr 172.21.15.12:0/1061567789 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b17e00700 1 --2- 172.21.15.12:0/1061567789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b180d3f20 0x7f0b180b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1cc00700 1 -- 172.21.15.12:0/1061567789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b180d3f20 msgr2=0x7f0b180b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1d600700 1 --2- 172.21.15.12:0/1061567789 >> v2:172.21.15.159:3300/0 conn(0x7f0b180b3f80 0x7f0b180b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1cc00700 1 --2- 172.21.15.12:0/1061567789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b180d3f20 0x7f0b180b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1cc00700 1 -- 172.21.15.12:0/1061567789 >> v2:172.21.15.159:3300/0 conn(0x7f0b180b3f80 msgr2=0x7f0b180b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1cc00700 1 --2- 172.21.15.12:0/1061567789 >> v2:172.21.15.159:3300/0 conn(0x7f0b180b3f80 0x7f0b180b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1cc00700 1 -- 172.21.15.12:0/1061567789 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b180d21e0 con 0x7f0b18160e50 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b17e00700 1 --2- 172.21.15.12:0/1061567789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b180d3f20 0x7f0b180b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.236+0000 7f0b1d600700 1 --2- 172.21.15.12:0/1061567789 >> v2:172.21.15.159:3300/0 conn(0x7f0b180b3f80 0x7f0b180b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:33:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.237+0000 7f0b1cc00700 1 --2- 172.21.15.12:0/1061567789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b18160e50 0x7f0b180b4e90 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f0b180b95b0 tx=0x7f0b0c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652b5af66e161ec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.237+0000 7f0b17400700 1 -- 172.21.15.12:0/1061567789 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c00b870 con 0x7f0b18160e50 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.237+0000 7f0b17400700 1 -- 172.21.15.12:0/1061567789 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c00b9d0 con 0x7f0b18160e50 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.237+0000 7f0b17400700 1 -- 172.21.15.12:0/1061567789 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c00bd60 con 0x7f0b18160e50 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.237+0000 7f0b1f600700 1 -- 172.21.15.12:0/1061567789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b18160e50 msgr2=0x7f0b180b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.237+0000 7f0b1f600700 1 --2- 172.21.15.12:0/1061567789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b18160e50 0x7f0b180b4e90 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f0b180b95b0 tx=0x7f0b0c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.238+0000 7f0b1f600700 1 -- 172.21.15.12:0/1061567789 shutdown_connections 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.238+0000 7f0b1f600700 1 --2- 172.21.15.12:0/1061567789 >> v2:172.21.15.159:3300/0 conn(0x7f0b180b3f80 0x7f0b180b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.238+0000 7f0b1f600700 1 --2- 172.21.15.12:0/1061567789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b18160e50 0x7f0b180b4e90 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.238+0000 7f0b1f600700 1 --2- 172.21.15.12:0/1061567789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b180d3f20 0x7f0b180b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.238+0000 7f0b1f600700 1 -- 172.21.15.12:0/1061567789 >> 172.21.15.12:0/1061567789 conn(0x7f0b1808aff0 msgr2=0x7f0b180d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.238+0000 7f0b1f600700 1 -- 172.21.15.12:0/1061567789 shutdown_connections 2024-03-29T11:33:57.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.238+0000 7f0b1f600700 1 -- 172.21.15.12:0/1061567789 wait complete. 2024-03-29T11:33:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.239+0000 7f0b1f600700 1 Processor -- start 2024-03-29T11:33:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.240+0000 7f0b1f600700 1 -- start start 2024-03-29T11:33:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.240+0000 7f0b1f600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b180b3f80 0x7f0b18151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1f600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b18160e50 0x7f0b180ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1f600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b180cd490 0x7f0b180cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1f600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180b08f0 con 0x7f0b180b3f80 2024-03-29T11:33:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1f600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180b8070 con 0x7f0b180cd490 2024-03-29T11:33:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1f600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180b8350 con 0x7f0b18160e50 2024-03-29T11:33:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b180b3f80 0x7f0b18151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b180b3f80 0x7f0b18151920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54796/0 (socket says 172.21.15.12:54796) 2024-03-29T11:33:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1cc00700 1 -- 172.21.15.12:0/3614560364 learned_addr learned my addr 172.21.15.12:0/3614560364 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b17e00700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b18160e50 0x7f0b180ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1d600700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b180cd490 0x7f0b180cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1cc00700 1 -- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b18160e50 msgr2=0x7f0b180ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:57.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1cc00700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b18160e50 0x7f0b180ccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1cc00700 1 -- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b180cd490 msgr2=0x7f0b180cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1cc00700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b180cd490 0x7f0b180cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1cc00700 1 -- 172.21.15.12:0/3614560364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c009550 con 0x7f0b180b3f80 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.241+0000 7f0b1d600700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b180cd490 0x7f0b180cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.242+0000 7f0b1cc00700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b180b3f80 0x7f0b18151920 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c014040 tx=0x7f0b0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.242+0000 7f0b14c00700 1 -- 172.21.15.12:0/3614560364 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c004310 con 0x7f0b180b3f80 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.242+0000 7f0b14c00700 1 -- 172.21.15.12:0/3614560364 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c01e070 con 0x7f0b180b3f80 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.242+0000 7f0b14c00700 1 -- 172.21.15.12:0/3614560364 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0036a0 con 0x7f0b180b3f80 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.242+0000 7f0b1f600700 1 -- 172.21.15.12:0/3614560364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b180b8630 con 0x7f0b180b3f80 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.242+0000 7f0b1f600700 1 -- 172.21.15.12:0/3614560364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b18151e60 con 0x7f0b180b3f80 2024-03-29T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.243+0000 7f0b1f600700 1 -- 172.21.15.12:0/3614560364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b18067720 con 0x7f0b180b3f80 2024-03-29T11:33:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.244+0000 7f0b14c00700 1 -- 172.21.15.12:0/3614560364 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 151) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f0b0c020030 con 0x7f0b180b3f80 2024-03-29T11:33:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.245+0000 7f0b14c00700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0af806ae00 0x7f0af806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.245+0000 7f0b14c00700 1 -- 172.21.15.12:0/3614560364 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f0b0c08a890 con 0x7f0b180b3f80 2024-03-29T11:33:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.245+0000 7f0b17e00700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0af806ae00 0x7f0af806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.246+0000 7f0b17e00700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0af806ae00 0x7f0af806d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b180b6170 tx=0x7f0b04008040 comp rx=0 tx=0).ready entity=mgr.12217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:57.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.249+0000 7f0b14c00700 1 -- 172.21.15.12:0/3614560364 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b0c055870 con 0x7f0b180b3f80 2024-03-29T11:33:57.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:57.523+0000 7f0b1f600700 1 -- 172.21.15.12:0/3614560364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0b180d2fc0 con 0x7f0b180b3f80 2024-03-29T11:33:58.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.207+0000 7f0b14c00700 1 -- 172.21.15.12:0/3614560364 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f0b0c05f070 con 0x7f0b180b3f80 2024-03-29T11:33:58.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.209+0000 7f0b14c00700 1 -- 172.21.15.12:0/3614560364 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v152) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0b0c06fa80 con 0x7f0b180b3f80 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.212+0000 7f0b1f600700 1 -- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0af806ae00 msgr2=0x7f0af806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.212+0000 7f0b1f600700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0af806ae00 0x7f0af806d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b180b6170 tx=0x7f0b04008040 comp rx=0 tx=0).stop 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.212+0000 7f0b1f600700 1 -- 172.21.15.12:0/3614560364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b180b3f80 msgr2=0x7f0b18151920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.212+0000 7f0b1f600700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b180b3f80 0x7f0b18151920 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c014040 tx=0x7f0b0c004750 comp rx=0 tx=0).stop 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.212+0000 7f0b1f600700 1 -- 172.21.15.12:0/3614560364 shutdown_connections 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.212+0000 7f0b1f600700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0af806ae00 0x7f0af806d2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.212+0000 7f0b1f600700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b180cd490 0x7f0b180cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.212+0000 7f0b1f600700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b180b3f80 0x7f0b18151920 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.212+0000 7f0b1f600700 1 --2- 172.21.15.12:0/3614560364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b18160e50 0x7f0b180ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.213+0000 7f0b1f600700 1 -- 172.21.15.12:0/3614560364 >> 172.21.15.12:0/3614560364 conn(0x7f0b1808aff0 msgr2=0x7f0b180ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.213+0000 7f0b1f600700 1 -- 172.21.15.12:0/3614560364 shutdown_connections 2024-03-29T11:33:58.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.213+0000 7f0b1f600700 1 -- 172.21.15.12:0/3614560364 wait complete. 2024-03-29T11:33:58.233 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:33:58.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:33:58.335 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:33:58.429 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:33:58.430 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:33:58.430 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:58.440 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:58.442+0000 7f6cb3a8a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:33:58.441 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:58.444+0000 7f6d2c2ad240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:33:58.474 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:33:58.474 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:33:58.560 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:58.626 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:58.629+0000 7f6d2c2ad240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:33:58.646 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:58.649+0000 7f6cb3a8a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:33:58.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.824+0000 7f026f600700 1 Processor -- start 2024-03-29T11:33:58.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.824+0000 7f026f600700 1 -- start start 2024-03-29T11:33:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f026f600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0268160c90 0x7f02680b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f026f600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f02680718e0 0x7f02680b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f026f600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 0x7f026806cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f026f600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02680ac4d0 con 0x7f02680718e0 2024-03-29T11:33:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f026f600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0268071e50 con 0x7f0268160c90 2024-03-29T11:33:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f026f600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0268072130 con 0x7f02680709c0 2024-03-29T11:33:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f0267e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f02680718e0 0x7f02680b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f0267e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f02680718e0 0x7f02680b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54812/0 (socket says 172.21.15.12:54812) 2024-03-29T11:33:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f0267e00700 1 -- 172.21.15.12:0/366699187 learned_addr learned my addr 172.21.15.12:0/366699187 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f026cc00700 1 --2- 172.21.15.12:0/366699187 >> v2:172.21.15.159:3300/0 conn(0x7f0268160c90 0x7f02680b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f026d600700 1 --2- 172.21.15.12:0/366699187 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 0x7f026806cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f0267e00700 1 -- 172.21.15.12:0/366699187 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 msgr2=0x7f026806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f0267e00700 1 --2- 172.21.15.12:0/366699187 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 0x7f026806cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f0267e00700 1 -- 172.21.15.12:0/366699187 >> v2:172.21.15.159:3300/0 conn(0x7f0268160c90 msgr2=0x7f02680b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f0267e00700 1 --2- 172.21.15.12:0/366699187 >> v2:172.21.15.159:3300/0 conn(0x7f0268160c90 0x7f02680b20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f0267e00700 1 -- 172.21.15.12:0/366699187 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0268072410 con 0x7f02680718e0 2024-03-29T11:33:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.826+0000 7f026d600700 1 --2- 172.21.15.12:0/366699187 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 0x7f026806cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.827+0000 7f0267e00700 1 --2- 172.21.15.12:0/366699187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 0x7f02680b2620 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f0268070090 tx=0x7f0254009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f345a3b274e974c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.827+0000 7f0267400700 1 -- 172.21.15.12:0/366699187 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254011830 con 0x7f02680718e0 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.827+0000 7f0267400700 1 -- 172.21.15.12:0/366699187 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0254011990 con 0x7f02680718e0 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.827+0000 7f0267400700 1 -- 172.21.15.12:0/366699187 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254011d20 con 0x7f02680718e0 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.828+0000 7f026f600700 1 -- 172.21.15.12:0/366699187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 msgr2=0x7f02680b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.828+0000 7f026f600700 1 --2- 172.21.15.12:0/366699187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 0x7f02680b2620 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f0268070090 tx=0x7f0254009bd0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.828+0000 7f026f600700 1 -- 172.21.15.12:0/366699187 shutdown_connections 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.828+0000 7f026f600700 1 --2- 172.21.15.12:0/366699187 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 0x7f026806cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.828+0000 7f026f600700 1 --2- 172.21.15.12:0/366699187 >> v2:172.21.15.159:3300/0 conn(0x7f0268160c90 0x7f02680b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.828+0000 7f026f600700 1 --2- 172.21.15.12:0/366699187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 0x7f02680b2620 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.828+0000 7f026f600700 1 -- 172.21.15.12:0/366699187 >> 172.21.15.12:0/366699187 conn(0x7f02680cf190 msgr2=0x7f02680cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.828+0000 7f026f600700 1 -- 172.21.15.12:0/366699187 shutdown_connections 2024-03-29T11:33:58.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.828+0000 7f026f600700 1 -- 172.21.15.12:0/366699187 wait complete. 2024-03-29T11:33:58.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.830+0000 7f026f600700 1 Processor -- start 2024-03-29T11:33:58.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.830+0000 7f026f600700 1 -- start start 2024-03-29T11:33:58.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f026f600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 0x7f026809cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:58.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f026f600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 0x7f026814b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:58.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f026f600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0268160c90 0x7f02680ad360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:58.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f026f600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02681f6600 con 0x7f02680718e0 2024-03-29T11:33:58.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f026f600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02681f68e0 con 0x7f0268160c90 2024-03-29T11:33:58.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f026f600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02681f6bc0 con 0x7f02680709c0 2024-03-29T11:33:58.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f0267e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 0x7f026814b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:58.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f0267e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 0x7f026814b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54824/0 (socket says 172.21.15.12:54824) 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f0267e00700 1 -- 172.21.15.12:0/3261861348 learned_addr learned my addr 172.21.15.12:0/3261861348 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f026d600700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0268160c90 0x7f02680ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.831+0000 7f026cc00700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 0x7f026809cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f0267e00700 1 -- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 msgr2=0x7f026809cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f0267e00700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 0x7f026809cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f0267e00700 1 -- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0268160c90 msgr2=0x7f02680ad360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f0267e00700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0268160c90 0x7f02680ad360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f0267e00700 1 -- 172.21.15.12:0/3261861348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0254009550 con 0x7f02680718e0 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f026d600700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0268160c90 0x7f02680ad360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f0267e00700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 0x7f026814b810 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f0254000c00 tx=0x7f0254004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f0264c00700 1 -- 172.21.15.12:0/3261861348 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254004240 con 0x7f02680718e0 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f0264c00700 1 -- 172.21.15.12:0/3261861348 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025401f070 con 0x7f02680718e0 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f0264c00700 1 -- 172.21.15.12:0/3261861348 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02540036a0 con 0x7f02680718e0 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f026f600700 1 -- 172.21.15.12:0/3261861348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02681f6ea0 con 0x7f02680718e0 2024-03-29T11:33:58.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.832+0000 7f026f600700 1 -- 172.21.15.12:0/3261861348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026809c990 con 0x7f02680718e0 2024-03-29T11:33:58.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.833+0000 7f026f600700 1 -- 172.21.15.12:0/3261861348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0268067720 con 0x7f02680718e0 2024-03-29T11:33:58.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.836+0000 7f0264c00700 1 -- 172.21.15.12:0/3261861348 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f0254021030 con 0x7f02680718e0 2024-03-29T11:33:58.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.836+0000 7f0264c00700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f024806ae40 0x7f024806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:58.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.837+0000 7f0264c00700 1 -- 172.21.15.12:0/3261861348 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f025408aef0 con 0x7f02680718e0 2024-03-29T11:33:58.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.837+0000 7f026cc00700 1 -- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f024806ae40 msgr2=0x7f024806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:58.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.837+0000 7f026cc00700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f024806ae40 0x7f024806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:58.839+0000 7f0264c00700 1 -- 172.21.15.12:0/3261861348 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0254055c40 con 0x7f02680718e0 2024-03-29T11:33:58.941 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:58.944+0000 7f6d2c2ad240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:33:58.989 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:58.991+0000 7f6cb3a8a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:33:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.038+0000 7f026cc00700 1 -- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f024806ae40 msgr2=0x7f024806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.038+0000 7f026cc00700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f024806ae40 0x7f024806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:59.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.055+0000 7f026f600700 1 -- 172.21.15.12:0/3261861348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02680b09d0 con 0x7f02680718e0 2024-03-29T11:33:59.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.056+0000 7f0264c00700 1 -- 172.21.15.12:0/3261861348 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+16797 (secure 0 0 0) 0x7f0254015020 con 0x7f02680718e0 2024-03-29T11:33:59.054 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:33:59.054 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":270,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:54.133295+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:33:34.767741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-03-29T11:33:35.848901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2007637492":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2270103540":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/883394797":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/288982228":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2254217683":"2024-03-30T11:33:54.133253+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:54.133253+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2308612503":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/3192194169":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/597141817":"2024-03-30T11:33:54.133253+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:59.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 -- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f024806ae40 msgr2=0x7f024806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:59.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f024806ae40 0x7f024806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 -- 172.21.15.12:0/3261861348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 msgr2=0x7f026814b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 0x7f026814b810 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f0254000c00 tx=0x7f0254004750 comp rx=0 tx=0).stop 2024-03-29T11:33:59.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 -- 172.21.15.12:0/3261861348 shutdown_connections 2024-03-29T11:33:59.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f024806ae40 0x7f024806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0268160c90 0x7f02680ad360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02680709c0 0x7f026809cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 --2- 172.21.15.12:0/3261861348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02680718e0 0x7f026814b810 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 -- 172.21.15.12:0/3261861348 >> 172.21.15.12:0/3261861348 conn(0x7f02680cf190 msgr2=0x7f02680cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:59.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 -- 172.21.15.12:0/3261861348 shutdown_connections 2024-03-29T11:33:59.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.059+0000 7f026f600700 1 -- 172.21.15.12:0/3261861348 wait complete. 2024-03-29T11:33:59.076 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:33:34.767741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '268', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:33:35.848901+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '269', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:33:59.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:33:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.322+0000 7f15ac200700 1 Processor -- start 2024-03-29T11:33:59.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.322+0000 7f15ac200700 1 -- start start 2024-03-29T11:33:59.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.324+0000 7f15ac200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f15a40d21f0 0x7f15a40ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.324+0000 7f15ac200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40ce270 con 0x7f15a41611b0 2024-03-29T11:33:59.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.324+0000 7f15ac200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a415b600 con 0x7f15a40cec40 2024-03-29T11:33:59.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.324+0000 7f15ac200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a40d2760 con 0x7f15a40d21f0 2024-03-29T11:33:59.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.324+0000 7f15a9800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f15a41611b0 0x7f15a415ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49120/0 (socket says 172.21.15.12:49120) 2024-03-29T11:33:59.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.324+0000 7f15a9800700 1 -- 172.21.15.12:0/1756001190 learned_addr learned my addr 172.21.15.12:0/1756001190 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:33:59.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.324+0000 7f15a8e00700 1 --2- 172.21.15.12:0/1756001190 >> v2:172.21.15.159:3300/0 conn(0x7f15a40d21f0 0x7f15a40ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:59.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.324+0000 7f15a3e00700 1 -- 172.21.15.12:0/1756001190 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485209239 0 0) 0x7f15a40ce270 con 0x7f15a41611b0 2024-03-29T11:33:59.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.324+0000 7f15a3e00700 1 -- 172.21.15.12:0/1756001190 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f158c0035b0 con 0x7f15a41611b0 2024-03-29T11:33:59.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.325+0000 7f15a8e00700 1 -- 172.21.15.12:0/1756001190 >> v1:172.21.15.159:6789/0 conn(0x7f15a40cec40 legacy=0x7f15a4161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.325+0000 7f15a3e00700 1 -- 172.21.15.12:0/1756001190 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444781274 0 0) 0x7f15a415b600 con 0x7f15a40cec40 2024-03-29T11:33:59.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.325+0000 7f15a8e00700 1 -- 172.21.15.12:0/1756001190 >> v1:172.21.15.12:6789/0 conn(0x7f15a41611b0 legacy=0x7f15a415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.325+0000 7f15a8e00700 1 -- 172.21.15.12:0/1756001190 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40d2a40 con 0x7f15a40d21f0 2024-03-29T11:33:59.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.325+0000 7f15a8e00700 1 --2- 172.21.15.12:0/1756001190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15a40d21f0 0x7f15a40ce700 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f1590009840 tx=0x7f1590009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=151385c0151640c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:59.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.325+0000 7f15a3e00700 1 -- 172.21.15.12:0/1756001190 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590011830 con 0x7f15a40d21f0 2024-03-29T11:33:59.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.325+0000 7f15a3e00700 1 -- 172.21.15.12:0/1756001190 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1590011990 con 0x7f15a40d21f0 2024-03-29T11:33:59.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.325+0000 7f15a3e00700 1 -- 172.21.15.12:0/1756001190 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590011d00 con 0x7f15a40d21f0 2024-03-29T11:33:59.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.326+0000 7f15ac200700 1 -- 172.21.15.12:0/1756001190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15a40d21f0 msgr2=0x7f15a40ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.326+0000 7f15ac200700 1 --2- 172.21.15.12:0/1756001190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15a40d21f0 0x7f15a40ce700 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f1590009840 tx=0x7f1590009b50 comp rx=0 tx=0).stop 2024-03-29T11:33:59.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.326+0000 7f15ac200700 1 -- 172.21.15.12:0/1756001190 shutdown_connections 2024-03-29T11:33:59.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.326+0000 7f15ac200700 1 --2- 172.21.15.12:0/1756001190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15a40d21f0 0x7f15a40ce700 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.326+0000 7f15ac200700 1 -- 172.21.15.12:0/1756001190 >> 172.21.15.12:0/1756001190 conn(0x7f15a408aff0 msgr2=0x7f15a40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:59.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.326+0000 7f15ac200700 1 -- 172.21.15.12:0/1756001190 shutdown_connections 2024-03-29T11:33:59.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.326+0000 7f15ac200700 1 -- 172.21.15.12:0/1756001190 wait complete. 2024-03-29T11:33:59.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.328+0000 7f15ac200700 1 Processor -- start 2024-03-29T11:33:59.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.328+0000 7f15ac200700 1 -- start start 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.329+0000 7f15ac200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15a40cec40 0x7f15a415a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.329+0000 7f15ac200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15a40d21f0 0x7f15a41603f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.329+0000 7f15ac200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15a41611b0 0x7f15a4160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.329+0000 7f15ac200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40cc570 con 0x7f15a40cec40 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.329+0000 7f15ac200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40cbc10 con 0x7f15a40d21f0 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.329+0000 7f15ac200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40cbeb0 con 0x7f15a41611b0 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15a9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15a40cec40 0x7f15a415a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15a9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15a40cec40 0x7f15a415a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60864/0 (socket says 172.21.15.12:60864) 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15a9800700 1 -- 172.21.15.12:0/883001955 learned_addr learned my addr 172.21.15.12:0/883001955 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15a8e00700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15a40d21f0 0x7f15a41603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15aa200700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15a41611b0 0x7f15a4160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:59.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15a9800700 1 -- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15a41611b0 msgr2=0x7f15a4160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15a9800700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15a41611b0 0x7f15a4160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15a9800700 1 -- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15a40d21f0 msgr2=0x7f15a41603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15a9800700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15a40d21f0 0x7f15a41603f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15a9800700 1 -- 172.21.15.12:0/883001955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15900094a0 con 0x7f15a40cec40 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.330+0000 7f15aa200700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15a41611b0 0x7f15a4160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.331+0000 7f15a9800700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15a40cec40 0x7f15a415a740 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f159800c4d0 tx=0x7f159800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.331+0000 7f15a1600700 1 -- 172.21.15.12:0/883001955 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598005d70 con 0x7f15a40cec40 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.331+0000 7f15a1600700 1 -- 172.21.15.12:0/883001955 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15980070d0 con 0x7f15a40cec40 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.331+0000 7f15ac200700 1 -- 172.21.15.12:0/883001955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a40cc190 con 0x7f15a40cec40 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.331+0000 7f15a1600700 1 -- 172.21.15.12:0/883001955 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598007440 con 0x7f15a40cec40 2024-03-29T11:33:59.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.331+0000 7f15ac200700 1 -- 172.21.15.12:0/883001955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a40b9020 con 0x7f15a40cec40 2024-03-29T11:33:59.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.332+0000 7f15ac200700 1 -- 172.21.15.12:0/883001955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a4067720 con 0x7f15a40cec40 2024-03-29T11:33:59.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.334+0000 7f15a1600700 1 -- 172.21.15.12:0/883001955 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f15980075a0 con 0x7f15a40cec40 2024-03-29T11:33:59.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.335+0000 7f15a1600700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f158406ae40 0x7f158406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.335+0000 7f15a1600700 1 -- 172.21.15.12:0/883001955 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f1598088b30 con 0x7f15a40cec40 2024-03-29T11:33:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.335+0000 7f15a8e00700 1 -- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f158406ae40 msgr2=0x7f158406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.335+0000 7f15a8e00700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f158406ae40 0x7f158406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:59.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.340+0000 7f15a1600700 1 -- 172.21.15.12:0/883001955 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15980539d0 con 0x7f15a40cec40 2024-03-29T11:33:59.471 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:59.474+0000 7f6d2c2ad240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:33:59.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.536+0000 7f15a8e00700 1 -- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f158406ae40 msgr2=0x7f158406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:59.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.536+0000 7f15a8e00700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f158406ae40 0x7f158406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.555+0000 7f15ac200700 1 -- 172.21.15.12:0/883001955 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f15a40b5d00 con 0x7f15a40cec40 2024-03-29T11:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.555+0000 7f15a1600700 1 -- 172.21.15.12:0/883001955 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v270) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1598017020 con 0x7f15a40cec40 2024-03-29T11:33:59.553 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:33:59.554 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:59.556+0000 7f6cb3a8a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 -- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f158406ae40 msgr2=0x7f158406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f158406ae40 0x7f158406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 -- 172.21.15.12:0/883001955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15a40cec40 msgr2=0x7f15a415a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15a40cec40 0x7f15a415a740 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f159800c4d0 tx=0x7f159800c7e0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 -- 172.21.15.12:0/883001955 shutdown_connections 2024-03-29T11:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f158406ae40 0x7f158406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15a41611b0 0x7f15a4160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15a40cec40 0x7f15a415a740 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 --2- 172.21.15.12:0/883001955 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15a40d21f0 0x7f15a41603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 -- 172.21.15.12:0/883001955 >> 172.21.15.12:0/883001955 conn(0x7f15a408aff0 msgr2=0x7f15a415cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 -- 172.21.15.12:0/883001955 shutdown_connections 2024-03-29T11:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.558+0000 7f15ac200700 1 -- 172.21.15.12:0/883001955 wait complete. 2024-03-29T11:33:59.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:33:59.726 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:33:59.729+0000 7f6d2c2ad240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:33:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.800+0000 7f8767200700 1 Processor -- start 2024-03-29T11:33:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.800+0000 7f8767200700 1 -- start start 2024-03-29T11:33:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.802+0000 7f8767200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f87600d16a0 0x7f87600d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.802+0000 7f8767200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f87600d2370 0x7f87600ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.802+0000 7f8767200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8760161420 con 0x7f876015d9d0 2024-03-29T11:33:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.802+0000 7f8767200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87600d4340 con 0x7f87600d2370 2024-03-29T11:33:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.802+0000 7f8767200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876015edc0 con 0x7f87600d16a0 2024-03-29T11:33:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.802+0000 7f8765200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f87600d2370 0x7f87600ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.802+0000 7f8765200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f87600d2370 0x7f87600ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60866/0 (socket says 172.21.15.12:60866) 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.802+0000 7f8765200700 1 -- 172.21.15.12:0/731870825 learned_addr learned my addr 172.21.15.12:0/731870825 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.802+0000 7f875f400700 1 --2- 172.21.15.12:0/731870825 >> v2:172.21.15.159:3300/0 conn(0x7f87600d16a0 0x7f87600d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.803+0000 7f8765200700 1 -- 172.21.15.12:0/731870825 >> v2:172.21.15.159:3300/0 conn(0x7f87600d16a0 msgr2=0x7f87600d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.803+0000 7f8765200700 1 --2- 172.21.15.12:0/731870825 >> v2:172.21.15.159:3300/0 conn(0x7f87600d16a0 0x7f87600d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.803+0000 7f8765200700 1 -- 172.21.15.12:0/731870825 >> v1:172.21.15.159:6789/0 conn(0x7f876015d9d0 legacy=0x7f87600d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.803+0000 7f8765200700 1 -- 172.21.15.12:0/731870825 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8760161700 con 0x7f87600d2370 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.803+0000 7f8765200700 1 --2- 172.21.15.12:0/731870825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d2370 0x7f87600ce360 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f87600d12f0 tx=0x7f875800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8607ba0ab5db6e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.803+0000 7f875ea00700 1 -- 172.21.15.12:0/731870825 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875800e070 con 0x7f87600d2370 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.803+0000 7f875ea00700 1 -- 172.21.15.12:0/731870825 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875800b910 con 0x7f87600d2370 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.803+0000 7f875ea00700 1 -- 172.21.15.12:0/731870825 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875800bcc0 con 0x7f87600d2370 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.804+0000 7f8767200700 1 -- 172.21.15.12:0/731870825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d2370 msgr2=0x7f87600ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.804+0000 7f8767200700 1 --2- 172.21.15.12:0/731870825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d2370 0x7f87600ce360 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f87600d12f0 tx=0x7f875800b330 comp rx=0 tx=0).stop 2024-03-29T11:33:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.804+0000 7f8767200700 1 -- 172.21.15.12:0/731870825 shutdown_connections 2024-03-29T11:33:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.804+0000 7f8767200700 1 --2- 172.21.15.12:0/731870825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d2370 0x7f87600ce360 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.804+0000 7f8767200700 1 --2- 172.21.15.12:0/731870825 >> v2:172.21.15.159:3300/0 conn(0x7f87600d16a0 0x7f87600d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.804+0000 7f8767200700 1 -- 172.21.15.12:0/731870825 >> 172.21.15.12:0/731870825 conn(0x7f876008aff0 msgr2=0x7f87600d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.804+0000 7f8767200700 1 -- 172.21.15.12:0/731870825 shutdown_connections 2024-03-29T11:33:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.804+0000 7f8767200700 1 -- 172.21.15.12:0/731870825 wait complete. 2024-03-29T11:33:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.806+0000 7f8767200700 1 Processor -- start 2024-03-29T11:33:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.806+0000 7f8767200700 1 -- start start 2024-03-29T11:33:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.807+0000 7f8767200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d16a0 0x7f8760159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.807+0000 7f8767200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87600d2370 0x7f87600b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.807+0000 7f8767200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f876015d9d0 0x7f87600b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.807+0000 7f8767200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87600b8130 con 0x7f87600d16a0 2024-03-29T11:33:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.807+0000 7f8767200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87600b7e50 con 0x7f87600d2370 2024-03-29T11:33:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.807+0000 7f8767200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87601e73b0 con 0x7f876015d9d0 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.807+0000 7f875fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d16a0 0x7f8760159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.807+0000 7f875fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d16a0 0x7f8760159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60870/0 (socket says 172.21.15.12:60870) 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.807+0000 7f875f400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87600d2370 0x7f87600b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.807+0000 7f8765200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f876015d9d0 0x7f87600b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.808+0000 7f875fe00700 1 -- 172.21.15.12:0/2041817301 learned_addr learned my addr 172.21.15.12:0/2041817301 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.808+0000 7f875fe00700 1 -- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f876015d9d0 msgr2=0x7f87600b8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.808+0000 7f875fe00700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f876015d9d0 0x7f87600b8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.808+0000 7f875fe00700 1 -- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87600d2370 msgr2=0x7f87600b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.808+0000 7f875fe00700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87600d2370 0x7f87600b84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.808+0000 7f875fe00700 1 -- 172.21.15.12:0/2041817301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8758009b30 con 0x7f87600d16a0 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.808+0000 7f875fe00700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d16a0 0x7f8760159e40 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f875000c450 tx=0x7f875000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.808+0000 7f8757e00700 1 -- 172.21.15.12:0/2041817301 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87500052d0 con 0x7f87600d16a0 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.808+0000 7f8757e00700 1 -- 172.21.15.12:0/2041817301 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8750005430 con 0x7f87600d16a0 2024-03-29T11:33:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.809+0000 7f8767200700 1 -- 172.21.15.12:0/2041817301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87601e7690 con 0x7f87600d16a0 2024-03-29T11:33:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.809+0000 7f8757e00700 1 -- 172.21.15.12:0/2041817301 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87500038c0 con 0x7f87600d16a0 2024-03-29T11:33:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.809+0000 7f8767200700 1 -- 172.21.15.12:0/2041817301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876015a380 con 0x7f87600d16a0 2024-03-29T11:33:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.810+0000 7f8767200700 1 -- 172.21.15.12:0/2041817301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87600d1ca0 con 0x7f87600d16a0 2024-03-29T11:33:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.811+0000 7f8757e00700 1 -- 172.21.15.12:0/2041817301 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f8750003a20 con 0x7f87600d16a0 2024-03-29T11:33:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.812+0000 7f8757e00700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f874006ae40 0x7f874006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:33:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.812+0000 7f8757e00700 1 -- 172.21.15.12:0/2041817301 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f8750088ae0 con 0x7f87600d16a0 2024-03-29T11:33:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.812+0000 7f875f400700 1 -- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f874006ae40 msgr2=0x7f874006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:33:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.812+0000 7f875f400700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f874006ae40 0x7f874006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:33:59.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:33:59.817+0000 7f8757e00700 1 -- 172.21.15.12:0/2041817301 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8750053830 con 0x7f87600d16a0 2024-03-29T11:33:59.847 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:33:59.849+0000 7f6cb3a8a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:34:00.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.013+0000 7f875f400700 1 -- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f874006ae40 msgr2=0x7f874006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:00.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.013+0000 7f875f400700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f874006ae40 0x7f874006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:00.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.051+0000 7f8767200700 1 -- 172.21.15.12:0/2041817301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f876006afa0 con 0x7f87600d16a0 2024-03-29T11:34:00.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.052+0000 7f8757e00700 1 -- 172.21.15.12:0/2041817301 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v270) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8750017020 con 0x7f87600d16a0 2024-03-29T11:34:00.050 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.054+0000 7f8767200700 1 -- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f874006ae40 msgr2=0x7f874006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.054+0000 7f8767200700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f874006ae40 0x7f874006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.054+0000 7f8767200700 1 -- 172.21.15.12:0/2041817301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d16a0 msgr2=0x7f8760159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.054+0000 7f8767200700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d16a0 0x7f8760159e40 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f875000c450 tx=0x7f875000c760 comp rx=0 tx=0).stop 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.054+0000 7f8767200700 1 -- 172.21.15.12:0/2041817301 shutdown_connections 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.054+0000 7f8767200700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f874006ae40 0x7f874006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.054+0000 7f8767200700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f876015d9d0 0x7f87600b8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.054+0000 7f8767200700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87600d16a0 0x7f8760159e40 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.054+0000 7f8767200700 1 --2- 172.21.15.12:0/2041817301 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f87600d2370 0x7f87600b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.054+0000 7f8767200700 1 -- 172.21.15.12:0/2041817301 >> 172.21.15.12:0/2041817301 conn(0x7f876008aff0 msgr2=0x7f876015e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.055+0000 7f8767200700 1 -- 172.21.15.12:0/2041817301 shutdown_connections 2024-03-29T11:34:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.055+0000 7f8767200700 1 -- 172.21.15.12:0/2041817301 wait complete. 2024-03-29T11:34:00.071 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:34:00.185 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:00.188+0000 7f6d2c2ad240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:34:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.280+0000 7f5010c00700 1 Processor -- start 2024-03-29T11:34:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.280+0000 7f5010c00700 1 -- start start 2024-03-29T11:34:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.281+0000 7f5010c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f500c0d2540 0x7f500c1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.281+0000 7f5010c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c161420 con 0x7f500c0d16a0 2024-03-29T11:34:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.281+0000 7f5010c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c161700 con 0x7f500c15d9d0 2024-03-29T11:34:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.281+0000 7f5010c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500c0d4340 con 0x7f500c0d2540 2024-03-29T11:34:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.281+0000 7f5009400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f500c0d16a0 0x7f500c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49136/0 (socket says 172.21.15.12:49136) 2024-03-29T11:34:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.281+0000 7f5009400700 1 -- 172.21.15.12:0/3257540804 learned_addr learned my addr 172.21.15.12:0/3257540804 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.281+0000 7f500a800700 1 --2- 172.21.15.12:0/3257540804 >> v2:172.21.15.159:3300/0 conn(0x7f500c0d2540 0x7f500c1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.282+0000 7f5008a00700 1 -- 172.21.15.12:0/3257540804 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954926304 0 0) 0x7f500c161420 con 0x7f500c0d16a0 2024-03-29T11:34:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.282+0000 7f5008a00700 1 -- 172.21.15.12:0/3257540804 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff00035b0 con 0x7f500c0d16a0 2024-03-29T11:34:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.282+0000 7f500a800700 1 -- 172.21.15.12:0/3257540804 >> v1:172.21.15.159:6789/0 conn(0x7f500c15d9d0 legacy=0x7f500c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.282+0000 7f5008a00700 1 -- 172.21.15.12:0/3257540804 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635270216 0 0) 0x7f500c161700 con 0x7f500c15d9d0 2024-03-29T11:34:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.282+0000 7f500a800700 1 -- 172.21.15.12:0/3257540804 >> v1:172.21.15.12:6789/0 conn(0x7f500c0d16a0 legacy=0x7f500c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.282+0000 7f500a800700 1 -- 172.21.15.12:0/3257540804 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c1619e0 con 0x7f500c0d2540 2024-03-29T11:34:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.282+0000 7f500a800700 1 --2- 172.21.15.12:0/3257540804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f500c0d2540 0x7f500c1687b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f5004009d90 tx=0x7f500400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c9ee99b977bc0b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.283+0000 7f5008a00700 1 -- 172.21.15.12:0/3257540804 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500400e070 con 0x7f500c0d2540 2024-03-29T11:34:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.283+0000 7f5008a00700 1 -- 172.21.15.12:0/3257540804 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500400bb20 con 0x7f500c0d2540 2024-03-29T11:34:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.283+0000 7f5008a00700 1 -- 172.21.15.12:0/3257540804 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500400bed0 con 0x7f500c0d2540 2024-03-29T11:34:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.283+0000 7f5010c00700 1 -- 172.21.15.12:0/3257540804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f500c0d2540 msgr2=0x7f500c1687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.283+0000 7f5010c00700 1 --2- 172.21.15.12:0/3257540804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f500c0d2540 0x7f500c1687b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f5004009d90 tx=0x7f500400b330 comp rx=0 tx=0).stop 2024-03-29T11:34:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.283+0000 7f5010c00700 1 -- 172.21.15.12:0/3257540804 shutdown_connections 2024-03-29T11:34:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.283+0000 7f5010c00700 1 --2- 172.21.15.12:0/3257540804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f500c0d2540 0x7f500c1687b0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.283+0000 7f5010c00700 1 -- 172.21.15.12:0/3257540804 >> 172.21.15.12:0/3257540804 conn(0x7f500c08aff0 msgr2=0x7f500c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.284+0000 7f5010c00700 1 -- 172.21.15.12:0/3257540804 shutdown_connections 2024-03-29T11:34:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.284+0000 7f5010c00700 1 -- 172.21.15.12:0/3257540804 wait complete. 2024-03-29T11:34:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.285+0000 7f5010c00700 1 Processor -- start 2024-03-29T11:34:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.285+0000 7f5010c00700 1 -- start start 2024-03-29T11:34:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.286+0000 7f5010c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f500c0d16a0 0x7f500c1f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.286+0000 7f5010c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f500c0d2540 0x7f500c15c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.286+0000 7f5010c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f500c15d9d0 0x7f500c0cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.286+0000 7f5010c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0b6ff0 con 0x7f500c0d2540 2024-03-29T11:34:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.286+0000 7f5010c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c15ccf0 con 0x7f500c0d16a0 2024-03-29T11:34:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.286+0000 7f5010c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0b63e0 con 0x7f500c15d9d0 2024-03-29T11:34:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f5009400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f500c0d2540 0x7f500c15c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f5009e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f500c0d16a0 0x7f500c1f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f5009400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f500c0d2540 0x7f500c15c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60874/0 (socket says 172.21.15.12:60874) 2024-03-29T11:34:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f5009400700 1 -- 172.21.15.12:0/1508872958 learned_addr learned my addr 172.21.15.12:0/1508872958 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f500a800700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f500c15d9d0 0x7f500c0cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f5009400700 1 -- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f500c15d9d0 msgr2=0x7f500c0cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f5009400700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f500c15d9d0 0x7f500c0cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f5009400700 1 -- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f500c0d16a0 msgr2=0x7f500c1f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f5009400700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f500c0d16a0 0x7f500c1f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f5009400700 1 -- 172.21.15.12:0/1508872958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5004009a70 con 0x7f500c0d2540 2024-03-29T11:34:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.287+0000 7f5009400700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f500c0d2540 0x7f500c15c7b0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800c730 tx=0x7f4ff800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.288+0000 7f5002000700 1 -- 172.21.15.12:0/1508872958 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff80074e0 con 0x7f500c0d2540 2024-03-29T11:34:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.288+0000 7f5002000700 1 -- 172.21.15.12:0/1508872958 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff8007640 con 0x7f500c0d2540 2024-03-29T11:34:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.288+0000 7f5002000700 1 -- 172.21.15.12:0/1508872958 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8005a70 con 0x7f500c0d2540 2024-03-29T11:34:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.288+0000 7f5010c00700 1 -- 172.21.15.12:0/1508872958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500c0b66c0 con 0x7f500c0d2540 2024-03-29T11:34:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.288+0000 7f5010c00700 1 -- 172.21.15.12:0/1508872958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c0b6c80 con 0x7f500c0d2540 2024-03-29T11:34:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.289+0000 7f5010c00700 1 -- 172.21.15.12:0/1508872958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500c0d1ca0 con 0x7f500c0d2540 2024-03-29T11:34:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.290+0000 7f5002000700 1 -- 172.21.15.12:0/1508872958 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f4ff8005bd0 con 0x7f500c0d2540 2024-03-29T11:34:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.291+0000 7f5002000700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4fec06ae40 0x7f4fec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.291+0000 7f5002000700 1 -- 172.21.15.12:0/1508872958 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f4ff8088c20 con 0x7f500c0d2540 2024-03-29T11:34:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.291+0000 7f5009e00700 1 -- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4fec06ae40 msgr2=0x7f4fec06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.291+0000 7f5009e00700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4fec06ae40 0x7f4fec06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.295+0000 7f5002000700 1 -- 172.21.15.12:0/1508872958 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ff8053970 con 0x7f500c0d2540 2024-03-29T11:34:00.360 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:00.362+0000 7f6cb3a8a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:34:00.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.492+0000 7f5009e00700 1 -- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4fec06ae40 msgr2=0x7f4fec06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:00.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.492+0000 7f5009e00700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4fec06ae40 0x7f4fec06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:00.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.512+0000 7f5010c00700 1 -- 172.21.15.12:0/1508872958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f500c0b6e40 con 0x7f500c0d2540 2024-03-29T11:34:00.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.512+0000 7f5002000700 1 -- 172.21.15.12:0/1508872958 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v270) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4ff8017020 con 0x7f500c0d2540 2024-03-29T11:34:00.510 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:34:00.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 -- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4fec06ae40 msgr2=0x7f4fec06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:00.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4fec06ae40 0x7f4fec06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 -- 172.21.15.12:0/1508872958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f500c0d2540 msgr2=0x7f500c15c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f500c0d2540 0x7f500c15c7b0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800c730 tx=0x7f4ff800ca40 comp rx=0 tx=0).stop 2024-03-29T11:34:00.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 -- 172.21.15.12:0/1508872958 shutdown_connections 2024-03-29T11:34:00.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4fec06ae40 0x7f4fec06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f500c15d9d0 0x7f500c0cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f500c0d16a0 0x7f500c1f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 --2- 172.21.15.12:0/1508872958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f500c0d2540 0x7f500c15c7b0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 -- 172.21.15.12:0/1508872958 >> 172.21.15.12:0/1508872958 conn(0x7f500c08aff0 msgr2=0x7f500c0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:00.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 -- 172.21.15.12:0/1508872958 shutdown_connections 2024-03-29T11:34:00.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.515+0000 7f5010c00700 1 -- 172.21.15.12:0/1508872958 wait complete. 2024-03-29T11:34:00.531 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T11:34:00.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.734+0000 7fd6de200700 1 Processor -- start 2024-03-29T11:34:00.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.734+0000 7fd6de200700 1 -- start start 2024-03-29T11:34:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.735+0000 7fd6de200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d815d9d0 0x7fd6d80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.735+0000 7fd6de200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6d80d2540 0x7fd6d80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.735+0000 7fd6de200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d8161420 con 0x7fd6d80d16a0 2024-03-29T11:34:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.735+0000 7fd6de200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d815edc0 con 0x7fd6d80d2540 2024-03-29T11:34:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.735+0000 7fd6de200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8161700 con 0x7fd6d815d9d0 2024-03-29T11:34:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.736+0000 7fd6cfe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd6d80d16a0 0x7fd6d80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49142/0 (socket says 172.21.15.12:49142) 2024-03-29T11:34:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.736+0000 7fd6cfe00700 1 -- 172.21.15.12:0/3933318927 learned_addr learned my addr 172.21.15.12:0/3933318927 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:00.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.736+0000 7fd6d7e00700 1 --2- 172.21.15.12:0/3933318927 >> v2:172.21.15.12:3300/0 conn(0x7fd6d80d2540 0x7fd6d80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:00.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.736+0000 7fd6d7400700 1 --2- 172.21.15.12:0/3933318927 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d815d9d0 0x7fd6d80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:00.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.736+0000 7fd6d6a00700 1 -- 172.21.15.12:0/3933318927 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639014340 0 0) 0x7fd6d8161420 con 0x7fd6d80d16a0 2024-03-29T11:34:00.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.736+0000 7fd6d6a00700 1 -- 172.21.15.12:0/3933318927 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6bc0035b0 con 0x7fd6d80d16a0 2024-03-29T11:34:00.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.736+0000 7fd6d7e00700 1 -- 172.21.15.12:0/3933318927 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d815d9d0 msgr2=0x7fd6d80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:00.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.736+0000 7fd6d7e00700 1 --2- 172.21.15.12:0/3933318927 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d815d9d0 0x7fd6d80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.736+0000 7fd6d7e00700 1 -- 172.21.15.12:0/3933318927 >> v1:172.21.15.12:6789/0 conn(0x7fd6d80d16a0 legacy=0x7fd6d80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.736+0000 7fd6d7e00700 1 -- 172.21.15.12:0/3933318927 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d81619e0 con 0x7fd6d80d2540 2024-03-29T11:34:00.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.737+0000 7fd6d7e00700 1 --2- 172.21.15.12:0/3933318927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d2540 0x7fd6d80ce360 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009dc0 tx=0x7fd6d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399c9333c698716 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:00.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.737+0000 7fd6d6a00700 1 -- 172.21.15.12:0/3933318927 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d000e070 con 0x7fd6d80d2540 2024-03-29T11:34:00.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.737+0000 7fd6d6a00700 1 -- 172.21.15.12:0/3933318927 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d000bae0 con 0x7fd6d80d2540 2024-03-29T11:34:00.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.737+0000 7fd6d6a00700 1 -- 172.21.15.12:0/3933318927 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d000be90 con 0x7fd6d80d2540 2024-03-29T11:34:00.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.737+0000 7fd6de200700 1 -- 172.21.15.12:0/3933318927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d2540 msgr2=0x7fd6d80ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.737+0000 7fd6de200700 1 --2- 172.21.15.12:0/3933318927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d2540 0x7fd6d80ce360 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009dc0 tx=0x7fd6d000b330 comp rx=0 tx=0).stop 2024-03-29T11:34:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.738+0000 7fd6de200700 1 -- 172.21.15.12:0/3933318927 shutdown_connections 2024-03-29T11:34:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.738+0000 7fd6de200700 1 --2- 172.21.15.12:0/3933318927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d2540 0x7fd6d80ce360 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.738+0000 7fd6de200700 1 --2- 172.21.15.12:0/3933318927 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d815d9d0 0x7fd6d80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.738+0000 7fd6de200700 1 -- 172.21.15.12:0/3933318927 >> 172.21.15.12:0/3933318927 conn(0x7fd6d808aff0 msgr2=0x7fd6d80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.738+0000 7fd6de200700 1 -- 172.21.15.12:0/3933318927 shutdown_connections 2024-03-29T11:34:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.738+0000 7fd6de200700 1 -- 172.21.15.12:0/3933318927 wait complete. 2024-03-29T11:34:00.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.739+0000 7fd6de200700 1 Processor -- start 2024-03-29T11:34:00.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.740+0000 7fd6de200700 1 -- start start 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.740+0000 7fd6de200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d16a0 0x7fd6d815f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.740+0000 7fd6de200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d80d2540 0x7fd6d80b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.740+0000 7fd6de200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d815d9d0 0x7fd6d81585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.740+0000 7fd6de200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d80b8f50 con 0x7fd6d80d16a0 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.740+0000 7fd6de200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d81f6b20 con 0x7fd6d815d9d0 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.740+0000 7fd6de200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d81f6e00 con 0x7fd6d80d2540 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6d7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d16a0 0x7fd6d815f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6d7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d16a0 0x7fd6d815f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60880/0 (socket says 172.21.15.12:60880) 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6d7400700 1 -- 172.21.15.12:0/3860268384 learned_addr learned my addr 172.21.15.12:0/3860268384 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6cfe00700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d80d2540 0x7fd6d80b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6d7e00700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d815d9d0 0x7fd6d81585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6d7400700 1 -- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d80d2540 msgr2=0x7fd6d80b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6d7400700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d80d2540 0x7fd6d80b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6d7400700 1 -- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d815d9d0 msgr2=0x7fd6d81585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6d7400700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d815d9d0 0x7fd6d81585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6d7400700 1 -- 172.21.15.12:0/3860268384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d0009a70 con 0x7fd6d80d16a0 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.741+0000 7fd6cfe00700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d80d2540 0x7fd6d80b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.742+0000 7fd6d7400700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d16a0 0x7fd6d815f950 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800e870 tx=0x7fd6c800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.742+0000 7fd6cf400700 1 -- 172.21.15.12:0/3860268384 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8009590 con 0x7fd6d80d16a0 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.742+0000 7fd6cf400700 1 -- 172.21.15.12:0/3860268384 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c8013680 con 0x7fd6d80d16a0 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.742+0000 7fd6cf400700 1 -- 172.21.15.12:0/3860268384 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80139f0 con 0x7fd6d80d16a0 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.742+0000 7fd6de200700 1 -- 172.21.15.12:0/3860268384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d81f70e0 con 0x7fd6d80d16a0 2024-03-29T11:34:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.742+0000 7fd6de200700 1 -- 172.21.15.12:0/3860268384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d81f76a0 con 0x7fd6d80d16a0 2024-03-29T11:34:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.743+0000 7fd6de200700 1 -- 172.21.15.12:0/3860268384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d81f44c0 con 0x7fd6d80d16a0 2024-03-29T11:34:00.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.744+0000 7fd6cf400700 1 -- 172.21.15.12:0/3860268384 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fd6c8013b50 con 0x7fd6d80d16a0 2024-03-29T11:34:00.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.745+0000 7fd6cf400700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406ae70 0x7fd6b406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:00.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.746+0000 7fd6cf400700 1 -- 172.21.15.12:0/3860268384 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7fd6c8089690 con 0x7fd6d80d16a0 2024-03-29T11:34:00.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.746+0000 7fd6cfe00700 1 -- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406ae70 msgr2=0x7fd6b406d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:00.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.746+0000 7fd6cfe00700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406ae70 0x7fd6b406d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:00.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.750+0000 7fd6cf400700 1 -- 172.21.15.12:0/3860268384 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd6c8054480 con 0x7fd6d80d16a0 2024-03-29T11:34:00.905 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:00.908+0000 7f6d2c2ad240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:34:00.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.947+0000 7fd6cfe00700 1 -- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406ae70 msgr2=0x7fd6b406d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:00.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.947+0000 7fd6cfe00700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406ae70 0x7fd6b406d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:00.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.988+0000 7fd6de200700 1 -- 172.21.15.12:0/3860268384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd6d80d1ca0 con 0x7fd6d80d16a0 2024-03-29T11:34:00.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.988+0000 7fd6cf400700 1 -- 172.21.15.12:0/3860268384 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v270) v1 ==== 81+0+583 (secure 0 0 0) 0x7fd6c8057aa0 con 0x7fd6d80d16a0 2024-03-29T11:34:00.986 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2007637492 2024-03-30T11:33:54.133253+0000 2024-03-29T11:34:00.986 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2270103540 2024-03-30T11:33:54.133253+0000 2024-03-29T11:34:00.986 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/883394797 2024-03-30T11:33:54.133253+0000 2024-03-29T11:34:00.986 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/288982228 2024-03-30T11:33:54.133253+0000 2024-03-29T11:34:00.986 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2254217683 2024-03-30T11:33:54.133253+0000 2024-03-29T11:34:00.986 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/62646 2024-03-30T11:33:54.133253+0000 2024-03-29T11:34:00.987 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/62646 2024-03-30T11:33:54.133253+0000 2024-03-29T11:34:00.987 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2308612503 2024-03-30T11:33:54.133253+0000 2024-03-29T11:34:00.987 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3192194169 2024-03-30T11:33:54.133253+0000 2024-03-29T11:34:00.987 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/597141817 2024-03-30T11:33:54.133253+0000 2024-03-29T11:34:00.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 -- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406ae70 msgr2=0x7fd6b406d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:00.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406ae70 0x7fd6b406d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 -- 172.21.15.12:0/3860268384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d16a0 msgr2=0x7fd6d815f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:00.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d16a0 0x7fd6d815f950 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800e870 tx=0x7fd6c800eb80 comp rx=0 tx=0).stop 2024-03-29T11:34:00.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 -- 172.21.15.12:0/3860268384 shutdown_connections 2024-03-29T11:34:00.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd6b406ae70 0x7fd6b406d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6d815d9d0 0x7fd6d81585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80d16a0 0x7fd6d815f950 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 --2- 172.21.15.12:0/3860268384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6d80d2540 0x7fd6d80b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:00.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 -- 172.21.15.12:0/3860268384 >> 172.21.15.12:0/3860268384 conn(0x7fd6d808aff0 msgr2=0x7fd6d80b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:00.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 -- 172.21.15.12:0/3860268384 shutdown_connections 2024-03-29T11:34:00.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:00.991+0000 7fd6de200700 1 -- 172.21.15.12:0/3860268384 wait complete. 2024-03-29T11:34:00.990 INFO:teuthology.orchestra.run.smithi012.stderr:listed 10 entries 2024-03-29T11:34:00.998 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:01.001+0000 7f6d2c2ad240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:34:01.007 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T11:34:01.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:01.007 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T11:34:01.087 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:34:01.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:01.087 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T11:34:01.134 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:34:01.134 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:34:01.138 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:01.141+0000 7f6cb3a8a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:34:01.160 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:01.162+0000 7f6d2c2ad240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:34:01.179 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:34:01.179 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:34:01.179 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:34:01.224 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:34:01.224 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:34:01.224 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:34:01.238 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:01.240+0000 7f6cb3a8a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:34:01.271 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:34:01.271 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:34:01.383 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:01.386+0000 7f6d2c2ad240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:34:01.409 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:01.412+0000 7f6cb3a8a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:34:01.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.476+0000 7fbfc0c00700 1 Processor -- start 2024-03-29T11:34:01.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.476+0000 7fbfc0c00700 1 -- start start 2024-03-29T11:34:01.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.477+0000 7fbfc0c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0cb3f0 0x7fbfbc0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.477+0000 7fbfc0c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfbc15fd50 con 0x7fbfbc0ac600 2024-03-29T11:34:01.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfc0c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfbc15f140 con 0x7fbfbc15b5d0 2024-03-29T11:34:01.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfc0c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc15b9b0 con 0x7fbfbc0cb3f0 2024-03-29T11:34:01.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfba800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbfbc0ac600 0x7fbfbc06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49154/0 (socket says 172.21.15.12:49154) 2024-03-29T11:34:01.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfba800700 1 -- 172.21.15.12:0/2782952893 learned_addr learned my addr 172.21.15.12:0/2782952893 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:01.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfb9400700 1 --2- 172.21.15.12:0/2782952893 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0cb3f0 0x7fbfbc0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:01.477 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:01.478+0000 7f6d2c2ad240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:34:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfb8a00700 1 -- 172.21.15.12:0/2782952893 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914897459 0 0) 0x7fbfbc15fd50 con 0x7fbfbc0ac600 2024-03-29T11:34:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfb8a00700 1 -- 172.21.15.12:0/2782952893 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa00035b0 con 0x7fbfbc0ac600 2024-03-29T11:34:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfb8a00700 1 -- 172.21.15.12:0/2782952893 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657461012 0 0) 0x7fbfbc15f140 con 0x7fbfbc15b5d0 2024-03-29T11:34:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfb8a00700 1 -- 172.21.15.12:0/2782952893 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfbc15fd50 con 0x7fbfbc15b5d0 2024-03-29T11:34:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfb9400700 1 -- 172.21.15.12:0/2782952893 >> v1:172.21.15.159:6789/0 conn(0x7fbfbc15b5d0 legacy=0x7fbfbc0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfb9400700 1 -- 172.21.15.12:0/2782952893 >> v1:172.21.15.12:6789/0 conn(0x7fbfbc0ac600 legacy=0x7fbfbc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.478+0000 7fbfb9400700 1 -- 172.21.15.12:0/2782952893 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc15f420 con 0x7fbfbc0cb3f0 2024-03-29T11:34:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.479+0000 7fbfb9400700 1 --2- 172.21.15.12:0/2782952893 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0cb3f0 0x7fbfbc0ac0c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009870 tx=0x7fbfa8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d53850e273aeb42c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.479+0000 7fbfb8a00700 1 -- 172.21.15.12:0/2782952893 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8011830 con 0x7fbfbc0cb3f0 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.479+0000 7fbfb8a00700 1 -- 172.21.15.12:0/2782952893 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa8011990 con 0x7fbfbc0cb3f0 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.479+0000 7fbfb8a00700 1 -- 172.21.15.12:0/2782952893 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8011d20 con 0x7fbfbc0cb3f0 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.480+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2782952893 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0cb3f0 msgr2=0x7fbfbc0ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.480+0000 7fbfc0c00700 1 --2- 172.21.15.12:0/2782952893 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0cb3f0 0x7fbfbc0ac0c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009870 tx=0x7fbfa8009b80 comp rx=0 tx=0).stop 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.480+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2782952893 shutdown_connections 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.480+0000 7fbfc0c00700 1 --2- 172.21.15.12:0/2782952893 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0cb3f0 0x7fbfbc0ac0c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.480+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2782952893 >> 172.21.15.12:0/2782952893 conn(0x7fbfbc0cf190 msgr2=0x7fbfbc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.480+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2782952893 shutdown_connections 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.480+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2782952893 wait complete. 2024-03-29T11:34:01.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.481+0000 7fbfc0c00700 1 Processor -- start 2024-03-29T11:34:01.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.482+0000 7fbfc0c00700 1 -- start start 2024-03-29T11:34:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.482+0000 7fbfc0c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0ac600 0x7fbfbc0c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.482+0000 7fbfc0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfbc15b5d0 0x7fbfbc0ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfc0c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfbc0aed20 0x7fbfbc0c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfc0c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc1f6330 con 0x7fbfbc15b5d0 2024-03-29T11:34:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfc0c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc1f57b0 con 0x7fbfbc0aed20 2024-03-29T11:34:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfc0c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc1f5a90 con 0x7fbfbc0ac600 2024-03-29T11:34:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfb9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfbc15b5d0 0x7fbfbc0ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfb9e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0ac600 0x7fbfbc0c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfb9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfbc15b5d0 0x7fbfbc0ca230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60896/0 (socket says 172.21.15.12:60896) 2024-03-29T11:34:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfb9400700 1 -- 172.21.15.12:0/2843316147 learned_addr learned my addr 172.21.15.12:0/2843316147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfba800700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfbc0aed20 0x7fbfbc0c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfb9e00700 1 -- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfbc0aed20 msgr2=0x7fbfbc0c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfb9e00700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfbc0aed20 0x7fbfbc0c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfb9e00700 1 -- 172.21.15.12:0/2843316147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfbc15b5d0 msgr2=0x7fbfbc0ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfb9e00700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfbc15b5d0 0x7fbfbc0ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfb9e00700 1 -- 172.21.15.12:0/2843316147 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa80094a0 con 0x7fbfbc0ac600 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.483+0000 7fbfb9400700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfbc15b5d0 0x7fbfbc0ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.484+0000 7fbfb9e00700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0ac600 0x7fbfbc0c9cf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00c450 tx=0x7fbfac00c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.484+0000 7fbfb2000700 1 -- 172.21.15.12:0/2843316147 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac0052d0 con 0x7fbfbc0ac600 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.484+0000 7fbfb2000700 1 -- 172.21.15.12:0/2843316147 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac005430 con 0x7fbfbc0ac600 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.484+0000 7fbfb2000700 1 -- 172.21.15.12:0/2843316147 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac0038a0 con 0x7fbfbc0ac600 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.484+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2843316147 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfbc1f5d70 con 0x7fbfbc0ac600 2024-03-29T11:34:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.484+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2843316147 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfbc1f53d0 con 0x7fbfbc0ac600 2024-03-29T11:34:01.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.485+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2843316147 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfbc1f5560 con 0x7fbfbc0ac600 2024-03-29T11:34:01.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.486+0000 7fbfb2000700 1 -- 172.21.15.12:0/2843316147 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fbfac003a00 con 0x7fbfbc0ac600 2024-03-29T11:34:01.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.487+0000 7fbfb2000700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf9c06ae40 0x7fbf9c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.487+0000 7fbfb2000700 1 -- 172.21.15.12:0/2843316147 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7fbfac088c50 con 0x7fbfbc0ac600 2024-03-29T11:34:01.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.487+0000 7fbfb9400700 1 -- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf9c06ae40 msgr2=0x7fbf9c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:01.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.487+0000 7fbfb9400700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf9c06ae40 0x7fbf9c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:01.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.492+0000 7fbfb2000700 1 -- 172.21.15.12:0/2843316147 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbfac053910 con 0x7fbfbc0ac600 2024-03-29T11:34:01.577 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:01.580+0000 7f6d2c2ad240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:34:01.634 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:01.635+0000 7f6cb3a8a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:34:01.676 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:01.679+0000 7f6d2c2ad240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:01.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.688+0000 7fbfb9400700 1 -- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf9c06ae40 msgr2=0x7fbf9c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:01.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.688+0000 7fbfb9400700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf9c06ae40 0x7fbf9c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:01.730 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:01.732+0000 7f6cb3a8a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:34:01.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.754+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2843316147 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfbc1f5bd0 con 0x7fbfbc0ac600 2024-03-29T11:34:01.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.755+0000 7fbfb2000700 1 -- 172.21.15.12:0/2843316147 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4756 (secure 0 0 0) 0x7fbfbc1f5bd0 con 0x7fbfbc0ac600 2024-03-29T11:34:01.753 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:01.753 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":138,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1546500295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1546500295},{"type":"v1","addr":"172.21.15.159:6836","nonce":1546500295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1627782307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1627782307},{"type":"v1","addr":"172.21.15.12:6834","nonce":1627782307}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3327995506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3327995506},{"type":"v1","addr":"172.21.15.12:6835","nonce":3327995506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:33:36.862806+0000","modified":"2024-03-29T11:33:40.346151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12061":{"gid":12061,"name":"d","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/2746367434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2746367434},{"type":"v1","addr":"172.21.15.159:6837","nonce":2746367434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-29T11:34:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf9c06ae40 msgr2=0x7fbf9c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf9c06ae40 0x7fbf9c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0ac600 msgr2=0x7fbfbc0c9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0ac600 0x7fbfbc0c9cf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00c450 tx=0x7fbfac00c760 comp rx=0 tx=0).stop 2024-03-29T11:34:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2843316147 shutdown_connections 2024-03-29T11:34:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbf9c06ae40 0x7fbf9c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfbc0aed20 0x7fbfbc0c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfbc0ac600 0x7fbfbc0c9cf0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 --2- 172.21.15.12:0/2843316147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfbc15b5d0 0x7fbfbc0ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2843316147 >> 172.21.15.12:0/2843316147 conn(0x7fbfbc0cf190 msgr2=0x7fbfbc15cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2843316147 shutdown_connections 2024-03-29T11:34:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.758+0000 7fbfc0c00700 1 -- 172.21.15.12:0/2843316147 wait complete. 2024-03-29T11:34:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 138 2024-03-29T11:34:01.774 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:01.775 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:01.778+0000 7f6d2c2ad240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:34:01.844 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:01.846+0000 7f6cb3a8a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:34:01.953 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:01.955+0000 7f6cb3a8a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.985+0000 7fb084000700 1 Processor -- start 2024-03-29T11:34:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.985+0000 7fb084000700 1 -- start start 2024-03-29T11:34:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.987+0000 7fb084000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb07c15d710 0x7fb07c0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.987+0000 7fb084000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 0x7fb07c0d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.987+0000 7fb084000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c06bb40 con 0x7fb07c09aba0 2024-03-29T11:34:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.987+0000 7fb084000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb07c099b00 con 0x7fb07c15d710 2024-03-29T11:34:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.987+0000 7fb084000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c099de0 con 0x7fb07c06c650 2024-03-29T11:34:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.987+0000 7fb081600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb07c15d710 0x7fb07c0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.987+0000 7fb081600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb07c15d710 0x7fb07c0d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60900/0 (socket says 172.21.15.12:60900) 2024-03-29T11:34:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.987+0000 7fb081600700 1 -- 172.21.15.12:0/3177913092 learned_addr learned my addr 172.21.15.12:0/3177913092 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.987+0000 7fb082000700 1 --2- 172.21.15.12:0/3177913092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 0x7fb07c0d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.988+0000 7fb081600700 1 -- 172.21.15.12:0/3177913092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 msgr2=0x7fb07c0d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.988+0000 7fb081600700 1 --2- 172.21.15.12:0/3177913092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 0x7fb07c0d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.988+0000 7fb07be00700 1 -- 172.21.15.12:0/3177913092 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117898884 0 0) 0x7fb07c06bb40 con 0x7fb07c09aba0 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.988+0000 7fb081600700 1 -- 172.21.15.12:0/3177913092 >> v1:172.21.15.159:6789/0 conn(0x7fb07c09aba0 legacy=0x7fb07c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.988+0000 7fb081600700 1 -- 172.21.15.12:0/3177913092 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c09a0c0 con 0x7fb07c15d710 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.988+0000 7fb081600700 1 --2- 172.21.15.12:0/3177913092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb07c15d710 0x7fb07c0d0130 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fb0700098f0 tx=0x7fb070009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3d11ae47b3c9a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.988+0000 7fb07be00700 1 -- 172.21.15.12:0/3177913092 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070011830 con 0x7fb07c15d710 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.988+0000 7fb07be00700 1 -- 172.21.15.12:0/3177913092 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb070011990 con 0x7fb07c15d710 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.989+0000 7fb07be00700 1 -- 172.21.15.12:0/3177913092 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070011d20 con 0x7fb07c15d710 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.989+0000 7fb084000700 1 -- 172.21.15.12:0/3177913092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb07c15d710 msgr2=0x7fb07c0d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.989+0000 7fb084000700 1 --2- 172.21.15.12:0/3177913092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb07c15d710 0x7fb07c0d0130 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fb0700098f0 tx=0x7fb070009c00 comp rx=0 tx=0).stop 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.989+0000 7fb084000700 1 -- 172.21.15.12:0/3177913092 shutdown_connections 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.989+0000 7fb084000700 1 --2- 172.21.15.12:0/3177913092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 0x7fb07c0d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.989+0000 7fb084000700 1 --2- 172.21.15.12:0/3177913092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb07c15d710 0x7fb07c0d0130 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.989+0000 7fb084000700 1 -- 172.21.15.12:0/3177913092 >> 172.21.15.12:0/3177913092 conn(0x7fb07c0d13e0 msgr2=0x7fb07c0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.989+0000 7fb084000700 1 -- 172.21.15.12:0/3177913092 shutdown_connections 2024-03-29T11:34:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.989+0000 7fb084000700 1 -- 172.21.15.12:0/3177913092 wait complete. 2024-03-29T11:34:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.991+0000 7fb084000700 1 Processor -- start 2024-03-29T11:34:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.991+0000 7fb084000700 1 -- start start 2024-03-29T11:34:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.992+0000 7fb084000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 0x7fb07c14e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.992+0000 7fb084000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb07c09aba0 0x7fb07c0b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.992+0000 7fb084000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb07c15d710 0x7fb07c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.992+0000 7fb084000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c15e030 con 0x7fb07c09aba0 2024-03-29T11:34:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.992+0000 7fb084000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c0b2370 con 0x7fb07c15d710 2024-03-29T11:34:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.992+0000 7fb084000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c0b17f0 con 0x7fb07c06c650 2024-03-29T11:34:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.992+0000 7fb080c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb07c09aba0 0x7fb07c0b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.993+0000 7fb081600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 0x7fb07c14e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.993+0000 7fb081600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 0x7fb07c14e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:44222/0 (socket says 172.21.15.12:44222) 2024-03-29T11:34:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.993+0000 7fb081600700 1 -- 172.21.15.12:0/1500638208 learned_addr learned my addr 172.21.15.12:0/1500638208 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.993+0000 7fb082000700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb07c15d710 0x7fb07c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.993+0000 7fb081600700 1 -- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb07c15d710 msgr2=0x7fb07c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.993+0000 7fb081600700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb07c15d710 0x7fb07c15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.993+0000 7fb081600700 1 -- 172.21.15.12:0/1500638208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb07c09aba0 msgr2=0x7fb07c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.993+0000 7fb081600700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb07c09aba0 0x7fb07c0b0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.993+0000 7fb081600700 1 -- 172.21.15.12:0/1500638208 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb070009550 con 0x7fb07c06c650 2024-03-29T11:34:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.993+0000 7fb081600700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 0x7fb07c14e9e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb070014040 tx=0x7fb070004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:01.993 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:01.995+0000 7f6d2c2ad240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:34:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.994+0000 7fb079600700 1 -- 172.21.15.12:0/1500638208 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070004590 con 0x7fb07c06c650 2024-03-29T11:34:01.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.994+0000 7fb079600700 1 -- 172.21.15.12:0/1500638208 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07001e070 con 0x7fb07c06c650 2024-03-29T11:34:01.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.994+0000 7fb079600700 1 -- 172.21.15.12:0/1500638208 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0700038c0 con 0x7fb07c06c650 2024-03-29T11:34:01.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.994+0000 7fb084000700 1 -- 172.21.15.12:0/1500638208 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07c0b1ad0 con 0x7fb07c06c650 2024-03-29T11:34:01.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.994+0000 7fb084000700 1 -- 172.21.15.12:0/1500638208 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07c0b2090 con 0x7fb07c06c650 2024-03-29T11:34:01.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.995+0000 7fb084000700 1 -- 172.21.15.12:0/1500638208 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07c067720 con 0x7fb07c06c650 2024-03-29T11:34:01.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.996+0000 7fb079600700 1 -- 172.21.15.12:0/1500638208 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fb070003a20 con 0x7fb07c06c650 2024-03-29T11:34:01.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.997+0000 7fb079600700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb05c06ae40 0x7fb05c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:01.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.997+0000 7fb079600700 1 -- 172.21.15.12:0/1500638208 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7fb07008a6b0 con 0x7fb07c06c650 2024-03-29T11:34:01.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.997+0000 7fb080c00700 1 -- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb05c06ae40 msgr2=0x7fb05c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:01.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:01.997+0000 7fb080c00700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb05c06ae40 0x7fb05c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:01.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.001+0000 7fb079600700 1 -- 172.21.15.12:0/1500638208 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb070055400 con 0x7fb07c06c650 2024-03-29T11:34:02.061 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:02.064+0000 7f6cb3a8a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:34:02.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.198+0000 7fb080c00700 1 -- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb05c06ae40 msgr2=0x7fb05c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:02.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.198+0000 7fb080c00700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb05c06ae40 0x7fb05c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.219+0000 7fb084000700 1 -- 172.21.15.12:0/1500638208 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb07c0d0f20 con 0x7fb07c06c650 2024-03-29T11:34:02.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.220+0000 7fb079600700 1 -- 172.21.15.12:0/1500638208 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+16797 (secure 0 0 0) 0x7fb070058a20 con 0x7fb07c06c650 2024-03-29T11:34:02.219 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:02.219 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":270,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:54.133295+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:33:34.767741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-03-29T11:33:35.848901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2007637492":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2270103540":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/883394797":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/288982228":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2254217683":"2024-03-30T11:33:54.133253+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:54.133253+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2308612503":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/3192194169":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/597141817":"2024-03-30T11:33:54.133253+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 -- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb05c06ae40 msgr2=0x7fb05c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb05c06ae40 0x7fb05c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 -- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 msgr2=0x7fb07c14e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 0x7fb07c14e9e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb070014040 tx=0x7fb070004750 comp rx=0 tx=0).stop 2024-03-29T11:34:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 -- 172.21.15.12:0/1500638208 shutdown_connections 2024-03-29T11:34:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb05c06ae40 0x7fb05c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb07c15d710 0x7fb07c15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb07c06c650 0x7fb07c14e9e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 --2- 172.21.15.12:0/1500638208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb07c09aba0 0x7fb07c0b0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 -- 172.21.15.12:0/1500638208 >> 172.21.15.12:0/1500638208 conn(0x7fb07c0d13e0 msgr2=0x7fb07c0cc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 -- 172.21.15.12:0/1500638208 shutdown_connections 2024-03-29T11:34:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.223+0000 7fb084000700 1 -- 172.21.15.12:0/1500638208 wait complete. 2024-03-29T11:34:02.240 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:33:34.767741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '268', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:33:35.848901+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '269', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:34:02.240 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:34:02.253 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:02.255+0000 7f6d2c2ad240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:34:02.288 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:02.290+0000 7f6cb3a8a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:34:02.446 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:02.449+0000 7f6d2c2ad240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:34:02.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.451+0000 7f3080800700 1 Processor -- start 2024-03-29T11:34:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.451+0000 7f3080800700 1 -- start start 2024-03-29T11:34:02.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.452+0000 7f3080800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f30780cd020 0x7f30780cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.452+0000 7f3080800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30780cdae0 con 0x7f30780cbb30 2024-03-29T11:34:02.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.452+0000 7f3080800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30780cddc0 con 0x7f30780b2070 2024-03-29T11:34:02.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.452+0000 7f3080800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30780ce0a0 con 0x7f30780cd020 2024-03-29T11:34:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.452+0000 7f307de00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f30780cbb30 0x7f30780cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49158/0 (socket says 172.21.15.12:49158) 2024-03-29T11:34:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.452+0000 7f307de00700 1 -- 172.21.15.12:0/3806190543 learned_addr learned my addr 172.21.15.12:0/3806190543 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.453+0000 7f307d400700 1 --2- 172.21.15.12:0/3806190543 >> v2:172.21.15.159:3300/0 conn(0x7f30780cd020 0x7f30780cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.453+0000 7f307ca00700 1 -- 172.21.15.12:0/3806190543 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157315307 0 0) 0x7f30780cdae0 con 0x7f30780cbb30 2024-03-29T11:34:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.453+0000 7f307ca00700 1 -- 172.21.15.12:0/3806190543 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30600035b0 con 0x7f30780cbb30 2024-03-29T11:34:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.453+0000 7f307d400700 1 -- 172.21.15.12:0/3806190543 >> v1:172.21.15.159:6789/0 conn(0x7f30780b2070 legacy=0x7f307806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.453+0000 7f307d400700 1 -- 172.21.15.12:0/3806190543 >> v1:172.21.15.12:6789/0 conn(0x7f30780cbb30 legacy=0x7f30780cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.453+0000 7f307d400700 1 -- 172.21.15.12:0/3806190543 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307806dc20 con 0x7f30780cd020 2024-03-29T11:34:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.453+0000 7f307d400700 1 --2- 172.21.15.12:0/3806190543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cd020 0x7f30780cd480 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f3068009840 tx=0x7f3068009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1a1171b6766d5e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.454+0000 7f307ca00700 1 -- 172.21.15.12:0/3806190543 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068011830 con 0x7f30780cd020 2024-03-29T11:34:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.454+0000 7f307ca00700 1 -- 172.21.15.12:0/3806190543 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3068011990 con 0x7f30780cd020 2024-03-29T11:34:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.454+0000 7f307ca00700 1 -- 172.21.15.12:0/3806190543 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068011d20 con 0x7f30780cd020 2024-03-29T11:34:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.454+0000 7f3080800700 1 -- 172.21.15.12:0/3806190543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cd020 msgr2=0x7f30780cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.454+0000 7f3080800700 1 --2- 172.21.15.12:0/3806190543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cd020 0x7f30780cd480 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f3068009840 tx=0x7f3068009b50 comp rx=0 tx=0).stop 2024-03-29T11:34:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.454+0000 7f3080800700 1 -- 172.21.15.12:0/3806190543 shutdown_connections 2024-03-29T11:34:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.454+0000 7f3080800700 1 --2- 172.21.15.12:0/3806190543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cd020 0x7f30780cd480 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.454+0000 7f3080800700 1 -- 172.21.15.12:0/3806190543 >> 172.21.15.12:0/3806190543 conn(0x7f30780b0d70 msgr2=0x7f30780b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.455+0000 7f3080800700 1 -- 172.21.15.12:0/3806190543 shutdown_connections 2024-03-29T11:34:02.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.455+0000 7f3080800700 1 -- 172.21.15.12:0/3806190543 wait complete. 2024-03-29T11:34:02.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.457+0000 7f3080800700 1 Processor -- start 2024-03-29T11:34:02.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.457+0000 7f3080800700 1 -- start start 2024-03-29T11:34:02.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.458+0000 7f3080800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30780b2070 0x7f307815dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.458+0000 7f3080800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cbb30 0x7f3078155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.458+0000 7f3080800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30780cd020 0x7f30780ad440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.458+0000 7f3080800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3078156370 con 0x7f30780cd020 2024-03-29T11:34:02.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.458+0000 7f3080800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3078156650 con 0x7f30780cbb30 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.458+0000 7f3080800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3078156930 con 0x7f30780b2070 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.459+0000 7f307d400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cbb30 0x7f3078155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.459+0000 7f307e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30780cd020 0x7f30780ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.459+0000 7f307d400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cbb30 0x7f3078155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:33614/0 (socket says 172.21.15.12:33614) 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.459+0000 7f307d400700 1 -- 172.21.15.12:0/622697715 learned_addr learned my addr 172.21.15.12:0/622697715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.459+0000 7f307de00700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30780b2070 0x7f307815dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.459+0000 7f307d400700 1 -- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30780b2070 msgr2=0x7f307815dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.459+0000 7f307d400700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30780b2070 0x7f307815dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.459+0000 7f307d400700 1 -- 172.21.15.12:0/622697715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30780cd020 msgr2=0x7f30780ad440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.459+0000 7f307d400700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30780cd020 0x7f30780ad440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.459+0000 7f307d400700 1 -- 172.21.15.12:0/622697715 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30680094a0 con 0x7f30780cbb30 2024-03-29T11:34:02.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.460+0000 7f307d400700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cbb30 0x7f3078155a00 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3068009b90 tx=0x7f3068004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:02.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.460+0000 7f3072000700 1 -- 172.21.15.12:0/622697715 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30680036a0 con 0x7f30780cbb30 2024-03-29T11:34:02.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.460+0000 7f3072000700 1 -- 172.21.15.12:0/622697715 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306801e070 con 0x7f30780cbb30 2024-03-29T11:34:02.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.460+0000 7f3072000700 1 -- 172.21.15.12:0/622697715 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068003a60 con 0x7f30780cbb30 2024-03-29T11:34:02.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.460+0000 7f3080800700 1 -- 172.21.15.12:0/622697715 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30780acc40 con 0x7f30780cbb30 2024-03-29T11:34:02.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.460+0000 7f3080800700 1 -- 172.21.15.12:0/622697715 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307815e4e0 con 0x7f30780cbb30 2024-03-29T11:34:02.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.461+0000 7f3080800700 1 -- 172.21.15.12:0/622697715 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307815e6d0 con 0x7f30780cbb30 2024-03-29T11:34:02.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.462+0000 7f3072000700 1 -- 172.21.15.12:0/622697715 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f3068003bc0 con 0x7f30780cbb30 2024-03-29T11:34:02.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.463+0000 7f3072000700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f305c06ae40 0x7f305c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.463+0000 7f3072000700 1 -- 172.21.15.12:0/622697715 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f306808a730 con 0x7f30780cbb30 2024-03-29T11:34:02.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.463+0000 7f307de00700 1 -- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f305c06ae40 msgr2=0x7f305c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:02.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.463+0000 7f307de00700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f305c06ae40 0x7f305c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:02.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.468+0000 7f3072000700 1 -- 172.21.15.12:0/622697715 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3068055480 con 0x7f30780cbb30 2024-03-29T11:34:02.541 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:02.543+0000 7f6d2c2ad240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:02.559 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:02.561+0000 7f6cb3a8a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:34:02.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.664+0000 7f307de00700 1 -- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f305c06ae40 msgr2=0x7f305c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:02.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.664+0000 7f307de00700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f305c06ae40 0x7f305c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:02.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.680+0000 7f3080800700 1 -- 172.21.15.12:0/622697715 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30780acaa0 con 0x7f30780cbb30 2024-03-29T11:34:02.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.680+0000 7f3072000700 1 -- 172.21.15.12:0/622697715 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v270) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30780acaa0 con 0x7f30780cbb30 2024-03-29T11:34:02.678 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:34:02.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 -- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f305c06ae40 msgr2=0x7f305c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:02.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f305c06ae40 0x7f305c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 -- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cbb30 msgr2=0x7f3078155a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cbb30 0x7f3078155a00 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3068009b90 tx=0x7f3068004750 comp rx=0 tx=0).stop 2024-03-29T11:34:02.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 -- 172.21.15.12:0/622697715 shutdown_connections 2024-03-29T11:34:02.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f305c06ae40 0x7f305c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30780cd020 0x7f30780ad440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30780b2070 0x7f307815dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 --2- 172.21.15.12:0/622697715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30780cbb30 0x7f3078155a00 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 -- 172.21.15.12:0/622697715 >> 172.21.15.12:0/622697715 conn(0x7f30780b0d70 msgr2=0x7f30780ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 -- 172.21.15.12:0/622697715 shutdown_connections 2024-03-29T11:34:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.683+0000 7f3080800700 1 -- 172.21.15.12:0/622697715 wait complete. 2024-03-29T11:34:02.700 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:34:02.767 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:02.769+0000 7f6cb3a8a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:34:02.867 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:02.869+0000 7f6cb3a8a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.906+0000 7f9d93c00700 1 Processor -- start 2024-03-29T11:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.906+0000 7f9d93c00700 1 -- start start 2024-03-29T11:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.908+0000 7f9d93c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 0x7f9d8c0d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.908+0000 7f9d93c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9d8c15fcb0 0x7f9d8c160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.908+0000 7f9d93c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0cf1e0 con 0x7f9d8c15dad0 2024-03-29T11:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.908+0000 7f9d93c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c0cfcc0 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.908+0000 7f9d93c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0cd9f0 con 0x7f9d8c0ced80 2024-03-29T11:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.908+0000 7f9d8be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 0x7f9d8c0d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.908+0000 7f9d91c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9d8c15fcb0 0x7f9d8c160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.908+0000 7f9d8be00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 0x7f9d8c0d1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:44236/0 (socket says 172.21.15.12:44236) 2024-03-29T11:34:02.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.908+0000 7f9d8be00700 1 -- 172.21.15.12:0/3945224845 learned_addr learned my addr 172.21.15.12:0/3945224845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:02.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.908+0000 7f9d91200700 1 --1- 172.21.15.12:0/3945224845 >> v1:172.21.15.159:6789/0 conn(0x7f9d8c15dad0 0x7f9d8c0d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:46654/0 (socket says 172.21.15.12:46654) 2024-03-29T11:34:02.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.909+0000 7f9d8b400700 1 -- 172.21.15.12:0/3945224845 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797160041 0 0) 0x7f9d80001110 con 0x7f9d8c15dad0 2024-03-29T11:34:02.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.909+0000 7f9d8b400700 1 -- 172.21.15.12:0/3945224845 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d740035b0 con 0x7f9d8c15dad0 2024-03-29T11:34:02.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.909+0000 7f9d8be00700 1 -- 172.21.15.12:0/3945224845 >> v2:172.21.15.159:3300/0 conn(0x7f9d8c15fcb0 msgr2=0x7f9d8c160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:02.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.909+0000 7f9d8be00700 1 --2- 172.21.15.12:0/3945224845 >> v2:172.21.15.159:3300/0 conn(0x7f9d8c15fcb0 0x7f9d8c160140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.909+0000 7f9d8be00700 1 -- 172.21.15.12:0/3945224845 >> v1:172.21.15.159:6789/0 conn(0x7f9d8c15dad0 legacy=0x7f9d8c0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.909+0000 7f9d8be00700 1 -- 172.21.15.12:0/3945224845 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c15f820 con 0x7f9d8c0ced80 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.909+0000 7f9d8be00700 1 --2- 172.21.15.12:0/3945224845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 0x7f9d8c0d1d40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9d780098f0 tx=0x7f9d78009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f624d9620203bea server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.909+0000 7f9d8b400700 1 -- 172.21.15.12:0/3945224845 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d78011830 con 0x7f9d8c0ced80 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.909+0000 7f9d8b400700 1 -- 172.21.15.12:0/3945224845 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d78011990 con 0x7f9d8c0ced80 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.909+0000 7f9d8b400700 1 -- 172.21.15.12:0/3945224845 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d78011d20 con 0x7f9d8c0ced80 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.910+0000 7f9d93c00700 1 -- 172.21.15.12:0/3945224845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 msgr2=0x7f9d8c0d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.910+0000 7f9d93c00700 1 --2- 172.21.15.12:0/3945224845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 0x7f9d8c0d1d40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9d780098f0 tx=0x7f9d78009c00 comp rx=0 tx=0).stop 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.910+0000 7f9d93c00700 1 -- 172.21.15.12:0/3945224845 shutdown_connections 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.910+0000 7f9d93c00700 1 --2- 172.21.15.12:0/3945224845 >> v2:172.21.15.159:3300/0 conn(0x7f9d8c15fcb0 0x7f9d8c160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.910+0000 7f9d93c00700 1 --2- 172.21.15.12:0/3945224845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 0x7f9d8c0d1d40 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.910+0000 7f9d93c00700 1 -- 172.21.15.12:0/3945224845 >> 172.21.15.12:0/3945224845 conn(0x7f9d8c08aff0 msgr2=0x7f9d8c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.910+0000 7f9d93c00700 1 -- 172.21.15.12:0/3945224845 shutdown_connections 2024-03-29T11:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.910+0000 7f9d93c00700 1 -- 172.21.15.12:0/3945224845 wait complete. 2024-03-29T11:34:02.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.912+0000 7f9d93c00700 1 Processor -- start 2024-03-29T11:34:02.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.912+0000 7f9d93c00700 1 -- start start 2024-03-29T11:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d93c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 0x7f9d8c0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d93c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d8c15dad0 0x7f9d8c151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d93c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d8c15fcb0 0x7f9d8c15c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d93c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c14ded0 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d93c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0b8430 con 0x7f9d8c15dad0 2024-03-29T11:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d93c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0b8710 con 0x7f9d8c0ced80 2024-03-29T11:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d91c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d8c15fcb0 0x7f9d8c15c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d91c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d8c15fcb0 0x7f9d8c15c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60926/0 (socket says 172.21.15.12:60926) 2024-03-29T11:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d91c00700 1 -- 172.21.15.12:0/1822865240 learned_addr learned my addr 172.21.15.12:0/1822865240 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:02.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d91200700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 0x7f9d8c0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:02.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.913+0000 7f9d8be00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d8c15dad0 0x7f9d8c151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d91c00700 1 -- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 msgr2=0x7f9d8c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d91c00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 0x7f9d8c0ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d91c00700 1 -- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d8c15dad0 msgr2=0x7f9d8c151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d91c00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d8c15dad0 0x7f9d8c151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d91c00700 1 -- 172.21.15.12:0/1822865240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78009550 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d91c00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d8c15fcb0 0x7f9d8c15c290 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400efc0 tx=0x7f9d8400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d88c00700 1 -- 172.21.15.12:0/1822865240 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8400cad0 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d88c00700 1 -- 172.21.15.12:0/1822865240 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8400f040 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d93c00700 1 -- 172.21.15.12:0/1822865240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c14d880 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d88c00700 1 -- 172.21.15.12:0/1822865240 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8400ce70 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.914+0000 7f9d93c00700 1 -- 172.21.15.12:0/1822865240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c0cd110 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.915+0000 7f9d93c00700 1 -- 172.21.15.12:0/1822865240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c06d010 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.916+0000 7f9d88c00700 1 -- 172.21.15.12:0/1822865240 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f9d84014480 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.917+0000 7f9d88c00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9d6c06ae40 0x7f9d6c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:02.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.917+0000 7f9d88c00700 1 -- 172.21.15.12:0/1822865240 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f9d8408a180 con 0x7f9d8c15fcb0 2024-03-29T11:34:02.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.917+0000 7f9d91200700 1 -- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9d6c06ae40 msgr2=0x7f9d6c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:02.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.917+0000 7f9d91200700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9d6c06ae40 0x7f9d6c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:02.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:02.922+0000 7f9d88c00700 1 -- 172.21.15.12:0/1822865240 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d84054cb0 con 0x7f9d8c15fcb0 2024-03-29T11:34:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.118+0000 7f9d91200700 1 -- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9d6c06ae40 msgr2=0x7f9d6c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.118+0000 7f9d91200700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9d6c06ae40 0x7f9d6c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:03.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.135+0000 7f9d93c00700 1 -- 172.21.15.12:0/1822865240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9d8c067720 con 0x7f9d8c15fcb0 2024-03-29T11:34:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.136+0000 7f9d88c00700 1 -- 172.21.15.12:0/1822865240 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v270) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9d840582d0 con 0x7f9d8c15fcb0 2024-03-29T11:34:03.134 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:34:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 -- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9d6c06ae40 msgr2=0x7f9d6c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9d6c06ae40 0x7f9d6c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 -- 172.21.15.12:0/1822865240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d8c15fcb0 msgr2=0x7f9d8c15c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d8c15fcb0 0x7f9d8c15c290 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400efc0 tx=0x7f9d8400c0c0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 -- 172.21.15.12:0/1822865240 shutdown_connections 2024-03-29T11:34:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9d6c06ae40 0x7f9d6c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d8c15fcb0 0x7f9d8c15c290 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d8c0ced80 0x7f9d8c0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 --2- 172.21.15.12:0/1822865240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d8c15dad0 0x7f9d8c151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 -- 172.21.15.12:0/1822865240 >> 172.21.15.12:0/1822865240 conn(0x7f9d8c08aff0 msgr2=0x7f9d8c06c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 -- 172.21.15.12:0/1822865240 shutdown_connections 2024-03-29T11:34:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.138+0000 7f9d93c00700 1 -- 172.21.15.12:0/1822865240 wait complete. 2024-03-29T11:34:03.154 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:34:03.274 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:03.277+0000 7f6d2c2ad240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:34:03.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.361+0000 7f8220600700 1 Processor -- start 2024-03-29T11:34:03.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.361+0000 7f8220600700 1 -- start start 2024-03-29T11:34:03.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.362+0000 7f8220600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f821815d9d0 0x7f82180d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.362+0000 7f8220600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82180d16a0 0x7f82180d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.362+0000 7f8220600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f82180d2370 0x7f821815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.362+0000 7f8220600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82180d2820 con 0x7f821815d9d0 2024-03-29T11:34:03.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.362+0000 7f8220600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8218161420 con 0x7f82180d2370 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.362+0000 7f8220600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8218161700 con 0x7f82180d16a0 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.362+0000 7f821dc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f821815d9d0 0x7f82180d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.362+0000 7f821dc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f821815d9d0 0x7f82180d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60942/0 (socket says 172.21.15.12:60942) 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.362+0000 7f821dc00700 1 -- 172.21.15.12:0/1590484299 learned_addr learned my addr 172.21.15.12:0/1590484299 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.362+0000 7f821d200700 1 --2- 172.21.15.12:0/1590484299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82180d16a0 0x7f82180d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f821e600700 1 --2- 172.21.15.12:0/1590484299 >> v2:172.21.15.159:3300/0 conn(0x7f82180d2370 0x7f821815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f821dc00700 1 -- 172.21.15.12:0/1590484299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82180d16a0 msgr2=0x7f82180d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f821dc00700 1 --2- 172.21.15.12:0/1590484299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82180d16a0 0x7f82180d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f821dc00700 1 -- 172.21.15.12:0/1590484299 >> v2:172.21.15.159:3300/0 conn(0x7f82180d2370 msgr2=0x7f821815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f821dc00700 1 --2- 172.21.15.12:0/1590484299 >> v2:172.21.15.159:3300/0 conn(0x7f82180d2370 0x7f821815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f821dc00700 1 -- 172.21.15.12:0/1590484299 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82181619e0 con 0x7f821815d9d0 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f821d200700 1 --2- 172.21.15.12:0/1590484299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82180d16a0 0x7f82180d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:03.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f821e600700 1 --2- 172.21.15.12:0/1590484299 >> v2:172.21.15.159:3300/0 conn(0x7f82180d2370 0x7f821815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:03.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f821dc00700 1 --2- 172.21.15.12:0/1590484299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f821815d9d0 0x7f82180d3c30 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f82180cdc00 tx=0x7f820c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634391894502daaa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:03.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f8217e00700 1 -- 172.21.15.12:0/1590484299 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c00b870 con 0x7f821815d9d0 2024-03-29T11:34:03.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f8217e00700 1 -- 172.21.15.12:0/1590484299 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820c00b9d0 con 0x7f821815d9d0 2024-03-29T11:34:03.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.363+0000 7f8217e00700 1 -- 172.21.15.12:0/1590484299 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c00bd60 con 0x7f821815d9d0 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.364+0000 7f8220600700 1 -- 172.21.15.12:0/1590484299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f821815d9d0 msgr2=0x7f82180d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.364+0000 7f8220600700 1 --2- 172.21.15.12:0/1590484299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f821815d9d0 0x7f82180d3c30 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f82180cdc00 tx=0x7f820c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.364+0000 7f8220600700 1 -- 172.21.15.12:0/1590484299 shutdown_connections 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.364+0000 7f8220600700 1 --2- 172.21.15.12:0/1590484299 >> v2:172.21.15.159:3300/0 conn(0x7f82180d2370 0x7f821815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.364+0000 7f8220600700 1 --2- 172.21.15.12:0/1590484299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f821815d9d0 0x7f82180d3c30 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.364+0000 7f8220600700 1 --2- 172.21.15.12:0/1590484299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82180d16a0 0x7f82180d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.364+0000 7f8220600700 1 -- 172.21.15.12:0/1590484299 >> 172.21.15.12:0/1590484299 conn(0x7f821808aff0 msgr2=0x7f82180d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.364+0000 7f8220600700 1 -- 172.21.15.12:0/1590484299 shutdown_connections 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.364+0000 7f8220600700 1 -- 172.21.15.12:0/1590484299 wait complete. 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.366+0000 7f8220600700 1 Processor -- start 2024-03-29T11:34:03.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.366+0000 7f8220600700 1 -- start start 2024-03-29T11:34:03.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f8220600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82180d16a0 0x7f82180b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f8220600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82180d2370 0x7f8218156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f8220600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f821815d9d0 0x7f82181566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f8220600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82180b61c0 con 0x7f82180d16a0 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f8220600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82180b64a0 con 0x7f82180d2370 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f8220600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82180b6780 con 0x7f821815d9d0 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f821dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82180d16a0 0x7f82180b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f821dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82180d16a0 0x7f82180b4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60956/0 (socket says 172.21.15.12:60956) 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f821dc00700 1 -- 172.21.15.12:0/792780915 learned_addr learned my addr 172.21.15.12:0/792780915 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f821d200700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82180d2370 0x7f8218156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.367+0000 7f821e600700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f821815d9d0 0x7f82181566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.368+0000 7f821dc00700 1 -- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f821815d9d0 msgr2=0x7f82181566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.368+0000 7f821dc00700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f821815d9d0 0x7f82181566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.368+0000 7f821dc00700 1 -- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82180d2370 msgr2=0x7f8218156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:03.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.368+0000 7f821dc00700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82180d2370 0x7f8218156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.368+0000 7f821dc00700 1 -- 172.21.15.12:0/792780915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c009550 con 0x7f82180d16a0 2024-03-29T11:34:03.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.368+0000 7f821dc00700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82180d16a0 0x7f82180b4200 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f820c000c00 tx=0x7f820c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:03.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.368+0000 7f8215600700 1 -- 172.21.15.12:0/792780915 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c004310 con 0x7f82180d16a0 2024-03-29T11:34:03.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.368+0000 7f8215600700 1 -- 172.21.15.12:0/792780915 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820c01e070 con 0x7f82180d16a0 2024-03-29T11:34:03.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.368+0000 7f8220600700 1 -- 172.21.15.12:0/792780915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82180b6a60 con 0x7f82180d16a0 2024-03-29T11:34:03.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.368+0000 7f8215600700 1 -- 172.21.15.12:0/792780915 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c0036a0 con 0x7f82180d16a0 2024-03-29T11:34:03.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.369+0000 7f8220600700 1 -- 172.21.15.12:0/792780915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82180b7260 con 0x7f82180d16a0 2024-03-29T11:34:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.370+0000 7f8220600700 1 -- 172.21.15.12:0/792780915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82180d1ca0 con 0x7f82180d16a0 2024-03-29T11:34:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.370+0000 7f8215600700 1 -- 172.21.15.12:0/792780915 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f820c020030 con 0x7f82180d16a0 2024-03-29T11:34:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.371+0000 7f8215600700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81f806ae40 0x7f81f806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.371+0000 7f8215600700 1 -- 172.21.15.12:0/792780915 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f820c08a770 con 0x7f82180d16a0 2024-03-29T11:34:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.372+0000 7f821d200700 1 -- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81f806ae40 msgr2=0x7f81f806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.372+0000 7f821d200700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81f806ae40 0x7f81f806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:03.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.376+0000 7f8215600700 1 -- 172.21.15.12:0/792780915 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f820c0554c0 con 0x7f82180d16a0 2024-03-29T11:34:03.491 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:03.494+0000 7f6d2c2ad240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:34:03.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.573+0000 7f821d200700 1 -- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81f806ae40 msgr2=0x7f81f806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:03.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.573+0000 7f821d200700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81f806ae40 0x7f81f806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:03.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.592+0000 7f8220600700 1 -- 172.21.15.12:0/792780915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f82180b6f80 con 0x7f82180d16a0 2024-03-29T11:34:03.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.592+0000 7f8215600700 1 -- 172.21.15.12:0/792780915 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v270) v1 ==== 100+0+11 (secure 0 0 0) 0x7f82180b6f80 con 0x7f82180d16a0 2024-03-29T11:34:03.591 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:34:03.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 -- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81f806ae40 msgr2=0x7f81f806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:03.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81f806ae40 0x7f81f806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 -- 172.21.15.12:0/792780915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82180d16a0 msgr2=0x7f82180b4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:03.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82180d16a0 0x7f82180b4200 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f820c000c00 tx=0x7f820c004750 comp rx=0 tx=0).stop 2024-03-29T11:34:03.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 -- 172.21.15.12:0/792780915 shutdown_connections 2024-03-29T11:34:03.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81f806ae40 0x7f81f806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f821815d9d0 0x7f82181566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82180d16a0 0x7f82180b4200 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 --2- 172.21.15.12:0/792780915 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82180d2370 0x7f8218156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 -- 172.21.15.12:0/792780915 >> 172.21.15.12:0/792780915 conn(0x7f821808aff0 msgr2=0x7f82180b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:03.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 -- 172.21.15.12:0/792780915 shutdown_connections 2024-03-29T11:34:03.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.595+0000 7f8220600700 1 -- 172.21.15.12:0/792780915 wait complete. 2024-03-29T11:34:03.612 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:34:03.660 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:03.662+0000 7f6cb3a8a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:34:03.700 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:03.702+0000 7f6d2c2ad240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:03.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.815+0000 7fbc53400700 1 Processor -- start 2024-03-29T11:34:03.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.815+0000 7fbc53400700 1 -- start start 2024-03-29T11:34:03.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.817+0000 7fbc53400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc4c15b5d0 0x7fbc4c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.817+0000 7fbc53400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbc4c0ac7d0 0x7fbc4c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.817+0000 7fbc53400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c15fd50 con 0x7fbc4c0cb3f0 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.817+0000 7fbc53400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4c15b9b0 con 0x7fbc4c15b5d0 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.817+0000 7fbc53400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4c15f140 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.817+0000 7fbc50a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc4c15b5d0 0x7fbc4c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.817+0000 7fbc50a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc4c15b5d0 0x7fbc4c0ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60958/0 (socket says 172.21.15.12:60958) 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.817+0000 7fbc50a00700 1 -- 172.21.15.12:0/4264991864 learned_addr learned my addr 172.21.15.12:0/4264991864 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.817+0000 7fbc51400700 1 --2- 172.21.15.12:0/4264991864 >> v2:172.21.15.159:3300/0 conn(0x7fbc4c0ac7d0 0x7fbc4c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.818+0000 7fbc50a00700 1 -- 172.21.15.12:0/4264991864 >> v2:172.21.15.159:3300/0 conn(0x7fbc4c0ac7d0 msgr2=0x7fbc4c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.818+0000 7fbc50a00700 1 --2- 172.21.15.12:0/4264991864 >> v2:172.21.15.159:3300/0 conn(0x7fbc4c0ac7d0 0x7fbc4c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.818+0000 7fbc50a00700 1 -- 172.21.15.12:0/4264991864 >> v1:172.21.15.159:6789/0 conn(0x7fbc4c0cb3f0 legacy=0x7fbc4c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.818+0000 7fbc50a00700 1 -- 172.21.15.12:0/4264991864 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c15f420 con 0x7fbc4c15b5d0 2024-03-29T11:34:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.818+0000 7fbc50a00700 1 --2- 172.21.15.12:0/4264991864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c15b5d0 0x7fbc4c0ae3f0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0cfdc0 tx=0x7fbc40009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d527f0548d8a210f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:03.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.818+0000 7fbc4b400700 1 -- 172.21.15.12:0/4264991864 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc40011830 con 0x7fbc4c15b5d0 2024-03-29T11:34:03.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.818+0000 7fbc4b400700 1 -- 172.21.15.12:0/4264991864 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc40011990 con 0x7fbc4c15b5d0 2024-03-29T11:34:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.818+0000 7fbc4b400700 1 -- 172.21.15.12:0/4264991864 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc40011d20 con 0x7fbc4c15b5d0 2024-03-29T11:34:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.819+0000 7fbc53400700 1 -- 172.21.15.12:0/4264991864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c15b5d0 msgr2=0x7fbc4c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.819+0000 7fbc53400700 1 --2- 172.21.15.12:0/4264991864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c15b5d0 0x7fbc4c0ae3f0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0cfdc0 tx=0x7fbc40009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.819+0000 7fbc53400700 1 -- 172.21.15.12:0/4264991864 shutdown_connections 2024-03-29T11:34:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.819+0000 7fbc53400700 1 --2- 172.21.15.12:0/4264991864 >> v2:172.21.15.159:3300/0 conn(0x7fbc4c0ac7d0 0x7fbc4c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.819+0000 7fbc53400700 1 --2- 172.21.15.12:0/4264991864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c15b5d0 0x7fbc4c0ae3f0 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.819+0000 7fbc53400700 1 -- 172.21.15.12:0/4264991864 >> 172.21.15.12:0/4264991864 conn(0x7fbc4c0cf190 msgr2=0x7fbc4c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.819+0000 7fbc53400700 1 -- 172.21.15.12:0/4264991864 shutdown_connections 2024-03-29T11:34:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.819+0000 7fbc53400700 1 -- 172.21.15.12:0/4264991864 wait complete. 2024-03-29T11:34:03.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.821+0000 7fbc53400700 1 Processor -- start 2024-03-29T11:34:03.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.821+0000 7fbc53400700 1 -- start start 2024-03-29T11:34:03.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc53400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c0ac7d0 0x7fbc4c14b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc53400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc4c15b5d0 0x7fbc4c147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc53400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc4c15ab00 0x7fbc4c157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc53400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0affe0 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc53400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c15a670 con 0x7fbc4c15b5d0 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc53400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c14b7c0 con 0x7fbc4c15ab00 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc50a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c0ac7d0 0x7fbc4c14b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc50a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c0ac7d0 0x7fbc4c14b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60966/0 (socket says 172.21.15.12:60966) 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc50a00700 1 -- 172.21.15.12:0/2770467304 learned_addr learned my addr 172.21.15.12:0/2770467304 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc51400700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc4c15ab00 0x7fbc4c157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.822+0000 7fbc4be00700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc4c15b5d0 0x7fbc4c147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc50a00700 1 -- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc4c15ab00 msgr2=0x7fbc4c157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc50a00700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc4c15ab00 0x7fbc4c157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc50a00700 1 -- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc4c15b5d0 msgr2=0x7fbc4c147f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:03.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc50a00700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc4c15b5d0 0x7fbc4c147f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:03.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc50a00700 1 -- 172.21.15.12:0/2770467304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc40009550 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc4be00700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc4c15b5d0 0x7fbc4c147f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:03.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc50a00700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c0ac7d0 0x7fbc4c14b280 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009c10 tx=0x7fbc40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:03.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc48c00700 1 -- 172.21.15.12:0/2770467304 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc40004590 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc48c00700 1 -- 172.21.15.12:0/2770467304 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4001e070 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc48c00700 1 -- 172.21.15.12:0/2770467304 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc400038e0 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc53400700 1 -- 172.21.15.12:0/2770467304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c159a60 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.823+0000 7fbc53400700 1 -- 172.21.15.12:0/2770467304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4c15a020 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.824+0000 7fbc53400700 1 -- 172.21.15.12:0/2770467304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4c067720 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.825+0000 7fbc48c00700 1 -- 172.21.15.12:0/2770467304 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fbc40003a40 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.827+0000 7fbc48c00700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc2c06ae40 0x7fbc2c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:03.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.827+0000 7fbc48c00700 1 -- 172.21.15.12:0/2770467304 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7fbc4008a700 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.827+0000 7fbc4be00700 1 -- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc2c06ae40 msgr2=0x7fbc2c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:03.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.827+0000 7fbc4be00700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc2c06ae40 0x7fbc2c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:03.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:03.831+0000 7fbc48c00700 1 -- 172.21.15.12:0/2770467304 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc40055450 con 0x7fbc4c0ac7d0 2024-03-29T11:34:03.907 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:03.909+0000 7f6cb3a8a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:34:04.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.028+0000 7fbc4be00700 1 -- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc2c06ae40 msgr2=0x7fbc2c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:04.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.028+0000 7fbc4be00700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc2c06ae40 0x7fbc2c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:04.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.113+0000 7fbc53400700 1 -- 172.21.15.12:0/2770467304 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc4c0711b0 con 0x7fbc4c0ac7d0 2024-03-29T11:34:04.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.114+0000 7fbc48c00700 1 -- 172.21.15.12:0/2770467304 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4756 (secure 0 0 0) 0x7fbc4005de60 con 0x7fbc4c0ac7d0 2024-03-29T11:34:04.112 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:04.112 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":138,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1546500295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1546500295},{"type":"v1","addr":"172.21.15.159:6836","nonce":1546500295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1627782307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1627782307},{"type":"v1","addr":"172.21.15.12:6834","nonce":1627782307}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3327995506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3327995506},{"type":"v1","addr":"172.21.15.12:6835","nonce":3327995506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:33:36.862806+0000","modified":"2024-03-29T11:33:40.346151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12061":{"gid":12061,"name":"d","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/2746367434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2746367434},{"type":"v1","addr":"172.21.15.159:6837","nonce":2746367434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-29T11:34:04.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 -- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc2c06ae40 msgr2=0x7fbc2c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:04.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc2c06ae40 0x7fbc2c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 -- 172.21.15.12:0/2770467304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c0ac7d0 msgr2=0x7fbc4c14b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c0ac7d0 0x7fbc4c14b280 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009c10 tx=0x7fbc40004750 comp rx=0 tx=0).stop 2024-03-29T11:34:04.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 -- 172.21.15.12:0/2770467304 shutdown_connections 2024-03-29T11:34:04.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fbc2c06ae40 0x7fbc2c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc4c15ab00 0x7fbc4c157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c0ac7d0 0x7fbc4c14b280 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 --2- 172.21.15.12:0/2770467304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc4c15b5d0 0x7fbc4c147f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 -- 172.21.15.12:0/2770467304 >> 172.21.15.12:0/2770467304 conn(0x7fbc4c0cf190 msgr2=0x7fbc4c15e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:04.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 -- 172.21.15.12:0/2770467304 shutdown_connections 2024-03-29T11:34:04.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.117+0000 7fbc53400700 1 -- 172.21.15.12:0/2770467304 wait complete. 2024-03-29T11:34:04.116 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 138 2024-03-29T11:34:04.129 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:04.131+0000 7f6cb3a8a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:04.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:04.172 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:04.175+0000 7f6d2c2ad240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:34:04.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.338+0000 7ff075000700 1 Processor -- start 2024-03-29T11:34:04.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.339+0000 7ff075000700 1 -- start start 2024-03-29T11:34:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff075000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff070160f10 0x7ff0700d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff075000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cf230 0x7ff0700d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff075000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff070161590 0x7ff070161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff075000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff070161f60 con 0x7ff070161590 2024-03-29T11:34:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff075000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0700ceda0 con 0x7ff070160f10 2024-03-29T11:34:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff075000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700d23d0 con 0x7ff0700cf230 2024-03-29T11:34:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff06e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff070161590 0x7ff070161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff06e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff070161590 0x7ff070161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60976/0 (socket says 172.21.15.12:60976) 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff06e800700 1 -- 172.21.15.12:0/4015465364 learned_addr learned my addr 172.21.15.12:0/4015465364 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff06de00700 1 --2- 172.21.15.12:0/4015465364 >> v2:172.21.15.159:3300/0 conn(0x7ff070160f10 0x7ff0700d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.340+0000 7ff06d400700 1 --2- 172.21.15.12:0/4015465364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cf230 0x7ff0700d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.341+0000 7ff06e800700 1 -- 172.21.15.12:0/4015465364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cf230 msgr2=0x7ff0700d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.341+0000 7ff06e800700 1 --2- 172.21.15.12:0/4015465364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cf230 0x7ff0700d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.341+0000 7ff06e800700 1 -- 172.21.15.12:0/4015465364 >> v2:172.21.15.159:3300/0 conn(0x7ff070160f10 msgr2=0x7ff0700d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.341+0000 7ff06e800700 1 --2- 172.21.15.12:0/4015465364 >> v2:172.21.15.159:3300/0 conn(0x7ff070160f10 0x7ff0700d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.341+0000 7ff06e800700 1 -- 172.21.15.12:0/4015465364 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0701612f0 con 0x7ff070161590 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.341+0000 7ff06e800700 1 --2- 172.21.15.12:0/4015465364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff070161590 0x7ff070161a20 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7ff0700d1080 tx=0x7ff06800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a6f3913ee94313 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:04.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.341+0000 7ff06ca00700 1 -- 172.21.15.12:0/4015465364 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06800e070 con 0x7ff070161590 2024-03-29T11:34:04.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.341+0000 7ff06ca00700 1 -- 172.21.15.12:0/4015465364 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06800b910 con 0x7ff070161590 2024-03-29T11:34:04.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.341+0000 7ff06ca00700 1 -- 172.21.15.12:0/4015465364 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06800bcc0 con 0x7ff070161590 2024-03-29T11:34:04.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.342+0000 7ff075000700 1 -- 172.21.15.12:0/4015465364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff070161590 msgr2=0x7ff070161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.342+0000 7ff075000700 1 --2- 172.21.15.12:0/4015465364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff070161590 0x7ff070161a20 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7ff0700d1080 tx=0x7ff06800b330 comp rx=0 tx=0).stop 2024-03-29T11:34:04.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.342+0000 7ff075000700 1 -- 172.21.15.12:0/4015465364 shutdown_connections 2024-03-29T11:34:04.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.342+0000 7ff075000700 1 --2- 172.21.15.12:0/4015465364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff070161590 0x7ff070161a20 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.342+0000 7ff075000700 1 --2- 172.21.15.12:0/4015465364 >> v2:172.21.15.159:3300/0 conn(0x7ff070160f10 0x7ff0700d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.342+0000 7ff075000700 1 --2- 172.21.15.12:0/4015465364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cf230 0x7ff0700d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.342+0000 7ff075000700 1 -- 172.21.15.12:0/4015465364 >> 172.21.15.12:0/4015465364 conn(0x7ff07008aff0 msgr2=0x7ff0700d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:04.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.342+0000 7ff075000700 1 -- 172.21.15.12:0/4015465364 shutdown_connections 2024-03-29T11:34:04.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.342+0000 7ff075000700 1 -- 172.21.15.12:0/4015465364 wait complete. 2024-03-29T11:34:04.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.344+0000 7ff075000700 1 Processor -- start 2024-03-29T11:34:04.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.344+0000 7ff075000700 1 -- start start 2024-03-29T11:34:04.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.345+0000 7ff075000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cf230 0x7ff07015a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.345+0000 7ff075000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff070160f10 0x7ff07015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.345+0000 7ff075000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff070161590 0x7ff07015dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.345+0000 7ff075000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0701f85b0 con 0x7ff070161590 2024-03-29T11:34:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.345+0000 7ff075000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0701f7960 con 0x7ff070160f10 2024-03-29T11:34:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.345+0000 7ff075000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0701f7c40 con 0x7ff0700cf230 2024-03-29T11:34:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff070161590 0x7ff07015dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06d400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff070160f10 0x7ff07015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cf230 0x7ff07015a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06d400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff070160f10 0x7ff07015ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:33690/0 (socket says 172.21.15.12:33690) 2024-03-29T11:34:04.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06d400700 1 -- 172.21.15.12:0/2230228817 learned_addr learned my addr 172.21.15.12:0/2230228817 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:04.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06d400700 1 -- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cf230 msgr2=0x7ff07015a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06d400700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cf230 0x7ff07015a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06d400700 1 -- 172.21.15.12:0/2230228817 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff070161590 msgr2=0x7ff07015dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06d400700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff070161590 0x7ff07015dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06d400700 1 -- 172.21.15.12:0/2230228817 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff068009b30 con 0x7ff070160f10 2024-03-29T11:34:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.346+0000 7ff06d400700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff070160f10 0x7ff07015ab70 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7ff05c00e870 tx=0x7ff05c00eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.347+0000 7ff066000700 1 -- 172.21.15.12:0/2230228817 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c009590 con 0x7ff070160f10 2024-03-29T11:34:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.347+0000 7ff066000700 1 -- 172.21.15.12:0/2230228817 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c013680 con 0x7ff070160f10 2024-03-29T11:34:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.347+0000 7ff075000700 1 -- 172.21.15.12:0/2230228817 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0701f7f20 con 0x7ff070160f10 2024-03-29T11:34:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.347+0000 7ff066000700 1 -- 172.21.15.12:0/2230228817 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c0139f0 con 0x7ff070160f10 2024-03-29T11:34:04.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.347+0000 7ff075000700 1 -- 172.21.15.12:0/2230228817 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07015f5b0 con 0x7ff070160f10 2024-03-29T11:34:04.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.348+0000 7ff075000700 1 -- 172.21.15.12:0/2230228817 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0701f8720 con 0x7ff070160f10 2024-03-29T11:34:04.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.349+0000 7ff066000700 1 -- 172.21.15.12:0/2230228817 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7ff05c013b50 con 0x7ff070160f10 2024-03-29T11:34:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.350+0000 7ff066000700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff05006ae40 0x7ff05006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.350+0000 7ff066000700 1 -- 172.21.15.12:0/2230228817 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7ff05c089670 con 0x7ff070160f10 2024-03-29T11:34:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.350+0000 7ff06de00700 1 -- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff05006ae40 msgr2=0x7ff05006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:04.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.350+0000 7ff06de00700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff05006ae40 0x7ff05006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:04.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.354+0000 7ff066000700 1 -- 172.21.15.12:0/2230228817 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff0701f8720 con 0x7ff070160f10 2024-03-29T11:34:04.384 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:04.387+0000 7f6d2c2ad240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:04.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.551+0000 7ff06de00700 1 -- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff05006ae40 msgr2=0x7ff05006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:04.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.551+0000 7ff06de00700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff05006ae40 0x7ff05006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:04.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.572+0000 7ff075000700 1 -- 172.21.15.12:0/2230228817 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0701f8720 con 0x7ff070160f10 2024-03-29T11:34:04.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.573+0000 7ff066000700 1 -- 172.21.15.12:0/2230228817 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+16797 (secure 0 0 0) 0x7ff0701f8720 con 0x7ff070160f10 2024-03-29T11:34:04.571 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:04.571 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":270,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:33:54.133295+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:33:34.767741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-03-29T11:33:35.848901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2007637492":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2270103540":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/883394797":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/288982228":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2254217683":"2024-03-30T11:33:54.133253+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:54.133253+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2308612503":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/3192194169":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/597141817":"2024-03-30T11:33:54.133253+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.575+0000 7ff075000700 1 -- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff05006ae40 msgr2=0x7ff05006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.575+0000 7ff075000700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff05006ae40 0x7ff05006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.575+0000 7ff075000700 1 -- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff070160f10 msgr2=0x7ff07015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.575+0000 7ff075000700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff070160f10 0x7ff07015ab70 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7ff05c00e870 tx=0x7ff05c00eb80 comp rx=0 tx=0).stop 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.576+0000 7ff075000700 1 -- 172.21.15.12:0/2230228817 shutdown_connections 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.576+0000 7ff075000700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff05006ae40 0x7ff05006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.576+0000 7ff075000700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff070161590 0x7ff07015dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.576+0000 7ff075000700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cf230 0x7ff07015a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.576+0000 7ff075000700 1 --2- 172.21.15.12:0/2230228817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff070160f10 0x7ff07015ab70 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.576+0000 7ff075000700 1 -- 172.21.15.12:0/2230228817 >> 172.21.15.12:0/2230228817 conn(0x7ff07008aff0 msgr2=0x7ff0700cad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.576+0000 7ff075000700 1 -- 172.21.15.12:0/2230228817 shutdown_connections 2024-03-29T11:34:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.576+0000 7ff075000700 1 -- 172.21.15.12:0/2230228817 wait complete. 2024-03-29T11:34:04.592 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T11:34:04.592 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:34:04.659 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:04.661+0000 7f6cb3a8a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:34:04.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.794+0000 7fd2c4200700 1 Processor -- start 2024-03-29T11:34:04.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.795+0000 7fd2c4200700 1 -- start start 2024-03-29T11:34:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.796+0000 7fd2c4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 0x7fd2bc0ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.796+0000 7fd2c4200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd2bc06dee0 0x7fd2bc06e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.796+0000 7fd2c4200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc0b3890 con 0x7fd2bc15da50 2024-03-29T11:34:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.796+0000 7fd2c4200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc0acac0 con 0x7fd2bc06dee0 2024-03-29T11:34:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.796+0000 7fd2c4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc159f50 con 0x7fd2bc15d670 2024-03-29T11:34:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.797+0000 7fd2c2200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd2bc06dee0 0x7fd2bc06e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.797+0000 7fd2c1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 0x7fd2bc0ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.797+0000 7fd2c2200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd2bc06dee0 0x7fd2bc06e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:33692/0 (socket says 172.21.15.12:33692) 2024-03-29T11:34:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.797+0000 7fd2c1800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 0x7fd2bc0ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:44324/0 (socket says 172.21.15.12:44324) 2024-03-29T11:34:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.797+0000 7fd2c2200700 1 -- 172.21.15.12:0/1478963722 learned_addr learned my addr 172.21.15.12:0/1478963722 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.797+0000 7fd2c2200700 1 -- 172.21.15.12:0/1478963722 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 msgr2=0x7fd2bc0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.797+0000 7fd2c2200700 1 --2- 172.21.15.12:0/1478963722 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 0x7fd2bc0ac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.797+0000 7fd2c2200700 1 -- 172.21.15.12:0/1478963722 >> v1:172.21.15.159:6789/0 conn(0x7fd2bc15da50 legacy=0x7fd2bc0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.797+0000 7fd2c2200700 1 -- 172.21.15.12:0/1478963722 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc0b35b0 con 0x7fd2bc06dee0 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.798+0000 7fd2c2200700 1 --2- 172.21.15.12:0/1478963722 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2bc06dee0 0x7fd2bc06e370 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400d150 tx=0x7fd2b400d460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de7deab2bf82e574 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.798+0000 7fd2bbe00700 1 -- 172.21.15.12:0/1478963722 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4010070 con 0x7fd2bc06dee0 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.798+0000 7fd2bbe00700 1 -- 172.21.15.12:0/1478963722 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b400db00 con 0x7fd2bc06dee0 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.798+0000 7fd2bbe00700 1 -- 172.21.15.12:0/1478963722 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b400de90 con 0x7fd2bc06dee0 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.798+0000 7fd2c4200700 1 -- 172.21.15.12:0/1478963722 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2bc06dee0 msgr2=0x7fd2bc06e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.798+0000 7fd2c4200700 1 --2- 172.21.15.12:0/1478963722 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2bc06dee0 0x7fd2bc06e370 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400d150 tx=0x7fd2b400d460 comp rx=0 tx=0).stop 2024-03-29T11:34:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.799+0000 7fd2c4200700 1 -- 172.21.15.12:0/1478963722 shutdown_connections 2024-03-29T11:34:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.799+0000 7fd2c4200700 1 --2- 172.21.15.12:0/1478963722 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2bc06dee0 0x7fd2bc06e370 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.799+0000 7fd2c4200700 1 --2- 172.21.15.12:0/1478963722 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 0x7fd2bc0ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.799+0000 7fd2c4200700 1 -- 172.21.15.12:0/1478963722 >> 172.21.15.12:0/1478963722 conn(0x7fd2bc0cf190 msgr2=0x7fd2bc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.799+0000 7fd2c4200700 1 -- 172.21.15.12:0/1478963722 shutdown_connections 2024-03-29T11:34:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.799+0000 7fd2c4200700 1 -- 172.21.15.12:0/1478963722 wait complete. 2024-03-29T11:34:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.800+0000 7fd2c4200700 1 Processor -- start 2024-03-29T11:34:04.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.801+0000 7fd2c4200700 1 -- start start 2024-03-29T11:34:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.801+0000 7fd2c4200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc06dee0 0x7fd2bc09b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.801+0000 7fd2c4200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 0x7fd2bc0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.801+0000 7fd2c4200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2bc15da50 0x7fd2bc0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.801+0000 7fd2c4200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc09b580 con 0x7fd2bc06dee0 2024-03-29T11:34:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.801+0000 7fd2c4200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc1439d0 con 0x7fd2bc15da50 2024-03-29T11:34:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.801+0000 7fd2c4200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc143cb0 con 0x7fd2bc15d670 2024-03-29T11:34:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc06dee0 0x7fd2bc09b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc06dee0 0x7fd2bc09b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32768/0 (socket says 172.21.15.12:32768) 2024-03-29T11:34:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c1800700 1 -- 172.21.15.12:0/2454452732 learned_addr learned my addr 172.21.15.12:0/2454452732 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c2200700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2bc15da50 0x7fd2bc0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c0e00700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 0x7fd2bc0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c2200700 1 -- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 msgr2=0x7fd2bc0b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c2200700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 0x7fd2bc0b21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c2200700 1 -- 172.21.15.12:0/2454452732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc06dee0 msgr2=0x7fd2bc09b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c2200700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc06dee0 0x7fd2bc09b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c2200700 1 -- 172.21.15.12:0/2454452732 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b400bd40 con 0x7fd2bc15da50 2024-03-29T11:34:04.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.802+0000 7fd2c1800700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc06dee0 0x7fd2bc09b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:34:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.803+0000 7fd2c2200700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2bc15da50 0x7fd2bc0991d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4018000 tx=0x7fd2b4004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.803+0000 7fd2b9600700 1 -- 172.21.15.12:0/2454452732 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b40045d0 con 0x7fd2bc15da50 2024-03-29T11:34:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.803+0000 7fd2b9600700 1 -- 172.21.15.12:0/2454452732 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4004750 con 0x7fd2bc15da50 2024-03-29T11:34:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.803+0000 7fd2b9600700 1 -- 172.21.15.12:0/2454452732 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b401f440 con 0x7fd2bc15da50 2024-03-29T11:34:04.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.803+0000 7fd2c4200700 1 -- 172.21.15.12:0/2454452732 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc143f90 con 0x7fd2bc15da50 2024-03-29T11:34:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.803+0000 7fd2c4200700 1 -- 172.21.15.12:0/2454452732 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc144550 con 0x7fd2bc15da50 2024-03-29T11:34:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.804+0000 7fd2c4200700 1 -- 172.21.15.12:0/2454452732 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc067720 con 0x7fd2bc15da50 2024-03-29T11:34:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.805+0000 7fd2b9600700 1 -- 172.21.15.12:0/2454452732 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 152) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fd2b401f5c0 con 0x7fd2bc15da50 2024-03-29T11:34:04.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.806+0000 7fd2b9600700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd29c06ae40 0x7fd29c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:04.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.806+0000 7fd2b9600700 1 -- 172.21.15.12:0/2454452732 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6211+0+0 (secure 0 0 0) 0x7fd2b408d650 con 0x7fd2bc15da50 2024-03-29T11:34:04.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.806+0000 7fd2c1800700 1 -- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd29c06ae40 msgr2=0x7fd29c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:04.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.806+0000 7fd2c1800700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd29c06ae40 0x7fd29c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:04.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.810+0000 7fd2b9600700 1 -- 172.21.15.12:0/2454452732 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd2b40583b0 con 0x7fd2bc15da50 2024-03-29T11:34:04.832 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:04.835+0000 7f6d2c2ad240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:34:04.886 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:04.888+0000 7f6cb3a8a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:04.945 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:04.948+0000 7f6d2c2ad240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:04.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.961+0000 7fd2b9600700 1 -- 172.21.15.12:0/2454452732 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mgrmap(e 153) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fd2b4058780 con 0x7fd2bc15da50 2024-03-29T11:34:04.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.962+0000 7fd2b9600700 1 -- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd29c06ae40 msgr2=0x7fd29c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:04.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:04.962+0000 7fd2b9600700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd29c06ae40 0x7fd29c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.026 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:05.029+0000 7f6c9b800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:05.027 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:05.029+0000 7f6c9b800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:05.027 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:05.029+0000 7f6c9b800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:05.027 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:05.029+0000 7f6c9b800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:05.027 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:05.029+0000 7f6c9b800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:05.027 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:05.029+0000 7f6c9e000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:05.027 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:05.029+0000 7f6c9e000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:05.027 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:05.029+0000 7f6c9e000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:05.027 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:05.029+0000 7f6c9e000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:05.027 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:05.029+0000 7f6c9e000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.081+0000 7fd2c4200700 1 -- 172.21.15.12:0/2454452732 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2bc09d160 con 0x7fd2bc15da50 2024-03-29T11:34:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.082+0000 7fd2b9600700 1 -- 172.21.15.12:0/2454452732 <== mon.1 v2:172.21.15.159:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4756 (secure 0 0 0) 0x7fd2b406feb0 con 0x7fd2bc15da50 2024-03-29T11:34:05.080 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:05.080 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":138,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1546500295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1546500295},{"type":"v1","addr":"172.21.15.159:6836","nonce":1546500295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1627782307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1627782307},{"type":"v1","addr":"172.21.15.12:6834","nonce":1627782307}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3327995506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3327995506},{"type":"v1","addr":"172.21.15.12:6835","nonce":3327995506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:33:36.862806+0000","modified":"2024-03-29T11:33:40.346151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12061":{"gid":12061,"name":"d","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/2746367434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2746367434},{"type":"v1","addr":"172.21.15.159:6837","nonce":2746367434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-29T11:34:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.084+0000 7fd2c4200700 1 -- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2bc15da50 msgr2=0x7fd2bc0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.084+0000 7fd2c4200700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2bc15da50 0x7fd2bc0991d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4018000 tx=0x7fd2b4004ad0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.085+0000 7fd2c4200700 1 -- 172.21.15.12:0/2454452732 shutdown_connections 2024-03-29T11:34:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.085+0000 7fd2c4200700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd29c06ae40 0x7fd29c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.085+0000 7fd2c4200700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2bc15da50 0x7fd2bc0991d0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.085+0000 7fd2c4200700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc06dee0 0x7fd2bc09b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.085+0000 7fd2c4200700 1 --2- 172.21.15.12:0/2454452732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2bc15d670 0x7fd2bc0b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.085+0000 7fd2c4200700 1 -- 172.21.15.12:0/2454452732 >> 172.21.15.12:0/2454452732 conn(0x7fd2bc0cf190 msgr2=0x7fd2bc06e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.085+0000 7fd2c4200700 1 -- 172.21.15.12:0/2454452732 shutdown_connections 2024-03-29T11:34:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.085+0000 7fd2c4200700 1 -- 172.21.15.12:0/2454452732 wait complete. 2024-03-29T11:34:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 138 2024-03-29T11:34:05.102 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:34:05.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.310+0000 7f06dc800700 1 Processor -- start 2024-03-29T11:34:05.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.310+0000 7f06dc800700 1 -- start start 2024-03-29T11:34:05.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.311+0000 7f06dc800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06d40ceec0 0x7f06d41612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.311+0000 7f06dc800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 0x7f06d40b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06dc800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d40d30d0 con 0x7f06d415d640 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06dc800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d4161c90 con 0x7f06d40ceec0 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06dc800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40b6840 con 0x7f06d4161800 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06d9400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06d40ceec0 0x7f06d41612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06d9400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06d40ceec0 0x7f06d41612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32772/0 (socket says 172.21.15.12:32772) 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06d9400700 1 -- 172.21.15.12:0/3621007130 learned_addr learned my addr 172.21.15.12:0/3621007130 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06da800700 1 --2- 172.21.15.12:0/3621007130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 0x7f06d40b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06d9400700 1 -- 172.21.15.12:0/3621007130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 msgr2=0x7f06d40b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06d9400700 1 --2- 172.21.15.12:0/3621007130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 0x7f06d40b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06d9400700 1 -- 172.21.15.12:0/3621007130 >> v1:172.21.15.159:6789/0 conn(0x7f06d415d640 legacy=0x7f06d40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06d9400700 1 -- 172.21.15.12:0/3621007130 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d40d2b30 con 0x7f06d40ceec0 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.312+0000 7f06da800700 1 --2- 172.21.15.12:0/3621007130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 0x7f06d40b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:05.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.313+0000 7f06d9400700 1 --2- 172.21.15.12:0/3621007130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d40ceec0 0x7f06d41612c0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f06c40098f0 tx=0x7f06c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec8b7049baee4dd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:05.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.313+0000 7f06d8a00700 1 -- 172.21.15.12:0/3621007130 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c400b8a0 con 0x7f06d40ceec0 2024-03-29T11:34:05.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.313+0000 7f06d8a00700 1 -- 172.21.15.12:0/3621007130 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c400ba00 con 0x7f06d40ceec0 2024-03-29T11:34:05.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.313+0000 7f06d8a00700 1 -- 172.21.15.12:0/3621007130 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c400bd90 con 0x7f06d40ceec0 2024-03-29T11:34:05.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.313+0000 7f06dc800700 1 -- 172.21.15.12:0/3621007130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d40ceec0 msgr2=0x7f06d41612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.313+0000 7f06dc800700 1 --2- 172.21.15.12:0/3621007130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d40ceec0 0x7f06d41612c0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f06c40098f0 tx=0x7f06c4009c00 comp rx=0 tx=0).stop 2024-03-29T11:34:05.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.314+0000 7f06dc800700 1 -- 172.21.15.12:0/3621007130 shutdown_connections 2024-03-29T11:34:05.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.314+0000 7f06dc800700 1 --2- 172.21.15.12:0/3621007130 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 0x7f06d40b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.314+0000 7f06dc800700 1 --2- 172.21.15.12:0/3621007130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d40ceec0 0x7f06d41612c0 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.314+0000 7f06dc800700 1 -- 172.21.15.12:0/3621007130 >> 172.21.15.12:0/3621007130 conn(0x7f06d408aff0 msgr2=0x7f06d40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:05.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.314+0000 7f06dc800700 1 -- 172.21.15.12:0/3621007130 shutdown_connections 2024-03-29T11:34:05.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.314+0000 7f06dc800700 1 -- 172.21.15.12:0/3621007130 wait complete. 2024-03-29T11:34:05.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.316+0000 7f06dc800700 1 Processor -- start 2024-03-29T11:34:05.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.316+0000 7f06dc800700 1 -- start start 2024-03-29T11:34:05.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06dc800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06d40ceec0 0x7f06d409e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06dc800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d640 0x7f06d41f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06dc800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 0x7f06d40c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06dc800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d41f4110 con 0x7f06d415d640 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06dc800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d41f3180 con 0x7f06d40ceec0 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06dc800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d41f3740 con 0x7f06d4161800 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06d9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d640 0x7f06d41f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06da800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 0x7f06d40c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06d9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d640 0x7f06d41f3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32786/0 (socket says 172.21.15.12:32786) 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06d9400700 1 -- 172.21.15.12:0/2339296868 learned_addr learned my addr 172.21.15.12:0/2339296868 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.318+0000 7f06d9e00700 1 --2- 172.21.15.12:0/2339296868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06d40ceec0 0x7f06d409e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.319+0000 7f06d9400700 1 -- 172.21.15.12:0/2339296868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 msgr2=0x7f06d40c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.319+0000 7f06d9400700 1 --2- 172.21.15.12:0/2339296868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 0x7f06d40c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.319+0000 7f06d9400700 1 -- 172.21.15.12:0/2339296868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06d40ceec0 msgr2=0x7f06d409e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.319+0000 7f06d9400700 1 --2- 172.21.15.12:0/2339296868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06d40ceec0 0x7f06d409e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.319+0000 7f06d9400700 1 -- 172.21.15.12:0/2339296868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c4009550 con 0x7f06d415d640 2024-03-29T11:34:05.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.319+0000 7f06d9400700 1 --2- 172.21.15.12:0/2339296868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d640 0x7f06d41f3bd0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009c40 tx=0x7f06c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:05.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.319+0000 7f06ce000700 1 -- 172.21.15.12:0/2339296868 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4004510 con 0x7f06d415d640 2024-03-29T11:34:05.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.319+0000 7f06ce000700 1 -- 172.21.15.12:0/2339296868 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c401e070 con 0x7f06d415d640 2024-03-29T11:34:05.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.319+0000 7f06dc800700 1 -- 172.21.15.12:0/2339296868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d409f2d0 con 0x7f06d415d640 2024-03-29T11:34:05.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.319+0000 7f06ce000700 1 -- 172.21.15.12:0/2339296868 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c40038e0 con 0x7f06d415d640 2024-03-29T11:34:05.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.320+0000 7f06dc800700 1 -- 172.21.15.12:0/2339296868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d409eff0 con 0x7f06d415d640 2024-03-29T11:34:05.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.321+0000 7f06dc800700 1 -- 172.21.15.12:0/2339296868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d4067720 con 0x7f06d415d640 2024-03-29T11:34:05.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.321+0000 7f06ce000700 1 -- 172.21.15.12:0/2339296868 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 153) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f06c4003a40 con 0x7f06d415d640 2024-03-29T11:34:05.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.323+0000 7f06ce000700 1 -- 172.21.15.12:0/2339296868 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f06c4020070 con 0x7f06d415d640 2024-03-29T11:34:05.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.327+0000 7f06ce000700 1 -- 172.21.15.12:0/2339296868 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06c40556d0 con 0x7f06d415d640 2024-03-29T11:34:05.361 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:05.363+0000 7f6cb3a8a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:34:05.480 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:05.482+0000 7f6cb3a8a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:05.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.591+0000 7f06dc800700 1 -- 172.21.15.12:0/2339296868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06d41f3880 con 0x7f06d415d640 2024-03-29T11:34:05.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.591+0000 7f06ce000700 1 -- 172.21.15.12:0/2339296868 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f06c4058cf0 con 0x7f06d415d640 2024-03-29T11:34:05.589 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:05.589 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":138,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1546500295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1546500295},{"type":"v1","addr":"172.21.15.159:6836","nonce":1546500295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1627782307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1627782307},{"type":"v1","addr":"172.21.15.12:6834","nonce":1627782307}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3327995506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3327995506},{"type":"v1","addr":"172.21.15.12:6835","nonce":3327995506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:33:36.862806+0000","modified":"2024-03-29T11:33:40.346151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12061":{"gid":12061,"name":"d","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6837/2746367434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2746367434},{"type":"v1","addr":"172.21.15.159:6837","nonce":2746367434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-29T11:34:05.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.593+0000 7f06dc800700 1 -- 172.21.15.12:0/2339296868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d640 msgr2=0x7f06d41f3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.593+0000 7f06dc800700 1 --2- 172.21.15.12:0/2339296868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d640 0x7f06d41f3bd0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009c40 tx=0x7f06c4004750 comp rx=0 tx=0).stop 2024-03-29T11:34:05.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.594+0000 7f06dc800700 1 -- 172.21.15.12:0/2339296868 shutdown_connections 2024-03-29T11:34:05.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.594+0000 7f06dc800700 1 --2- 172.21.15.12:0/2339296868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f06d4161800 0x7f06d40c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.594+0000 7f06dc800700 1 --2- 172.21.15.12:0/2339296868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f06d40ceec0 0x7f06d409e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.594+0000 7f06dc800700 1 --2- 172.21.15.12:0/2339296868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06d415d640 0x7f06d41f3bd0 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.594+0000 7f06dc800700 1 -- 172.21.15.12:0/2339296868 >> 172.21.15.12:0/2339296868 conn(0x7f06d408aff0 msgr2=0x7f06d40d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:05.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.594+0000 7f06dc800700 1 -- 172.21.15.12:0/2339296868 shutdown_connections 2024-03-29T11:34:05.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.594+0000 7f06dc800700 1 -- 172.21.15.12:0/2339296868 wait complete. 2024-03-29T11:34:05.593 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 138 2024-03-29T11:34:05.610 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.808+0000 7f32b6800700 1 Processor -- start 2024-03-29T11:34:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.808+0000 7f32b6800700 1 -- start start 2024-03-29T11:34:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.809+0000 7f32b6800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f32b009a910 0x7f32b006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.809+0000 7f32b6800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b0099a60 0x7f32b0099e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.809+0000 7f32b6800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b00d08d0 con 0x7f32b015d920 2024-03-29T11:34:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.809+0000 7f32b6800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b006c300 con 0x7f32b009a910 2024-03-29T11:34:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.809+0000 7f32b6800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b009a3d0 con 0x7f32b0099a60 2024-03-29T11:34:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.809+0000 7f32af400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f32b015d920 0x7f32b00cfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49166/0 (socket says 172.21.15.12:49166) 2024-03-29T11:34:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.809+0000 7f32af400700 1 -- 172.21.15.12:0/761560465 learned_addr learned my addr 172.21.15.12:0/761560465 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.810+0000 7f32aea00700 1 --2- 172.21.15.12:0/761560465 >> v2:172.21.15.159:3300/0 conn(0x7f32b009a910 0x7f32b006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.810+0000 7f32afe00700 1 --2- 172.21.15.12:0/761560465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b0099a60 0x7f32b0099e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.810+0000 7f32ae000700 1 -- 172.21.15.12:0/761560465 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043973972 0 0) 0x7f32b00d08d0 con 0x7f32b015d920 2024-03-29T11:34:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.810+0000 7f32ae000700 1 -- 172.21.15.12:0/761560465 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32940035b0 con 0x7f32b015d920 2024-03-29T11:34:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.810+0000 7f32afe00700 1 -- 172.21.15.12:0/761560465 >> v2:172.21.15.159:3300/0 conn(0x7f32b009a910 msgr2=0x7f32b006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.810+0000 7f32afe00700 1 --2- 172.21.15.12:0/761560465 >> v2:172.21.15.159:3300/0 conn(0x7f32b009a910 0x7f32b006c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.810+0000 7f32afe00700 1 -- 172.21.15.12:0/761560465 >> v1:172.21.15.12:6789/0 conn(0x7f32b015d920 legacy=0x7f32b00cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.810+0000 7f32afe00700 1 -- 172.21.15.12:0/761560465 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b006b9d0 con 0x7f32b0099a60 2024-03-29T11:34:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.810+0000 7f32aea00700 1 --2- 172.21.15.12:0/761560465 >> v2:172.21.15.159:3300/0 conn(0x7f32b009a910 0x7f32b006c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:34:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.810+0000 7f32afe00700 1 --2- 172.21.15.12:0/761560465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b0099a60 0x7f32b0099e90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009d90 tx=0x7f32a800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc870d2a1ac95c5c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32ae000700 1 -- 172.21.15.12:0/761560465 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a800e070 con 0x7f32b0099a60 2024-03-29T11:34:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32ae000700 1 -- 172.21.15.12:0/761560465 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a800bb20 con 0x7f32b0099a60 2024-03-29T11:34:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32ae000700 1 -- 172.21.15.12:0/761560465 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8012430 con 0x7f32b0099a60 2024-03-29T11:34:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32b6800700 1 -- 172.21.15.12:0/761560465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b0099a60 msgr2=0x7f32b0099e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32b6800700 1 --2- 172.21.15.12:0/761560465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b0099a60 0x7f32b0099e90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009d90 tx=0x7f32a800b330 comp rx=0 tx=0).stop 2024-03-29T11:34:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32b6800700 1 -- 172.21.15.12:0/761560465 shutdown_connections 2024-03-29T11:34:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32b6800700 1 --2- 172.21.15.12:0/761560465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b0099a60 0x7f32b0099e90 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32b6800700 1 --2- 172.21.15.12:0/761560465 >> v2:172.21.15.159:3300/0 conn(0x7f32b009a910 0x7f32b006c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32b6800700 1 -- 172.21.15.12:0/761560465 >> 172.21.15.12:0/761560465 conn(0x7f32b00d1350 msgr2=0x7f32b00d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32b6800700 1 -- 172.21.15.12:0/761560465 shutdown_connections 2024-03-29T11:34:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.811+0000 7f32b6800700 1 -- 172.21.15.12:0/761560465 wait complete. 2024-03-29T11:34:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.813+0000 7f32b6800700 1 Processor -- start 2024-03-29T11:34:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.813+0000 7f32b6800700 1 -- start start 2024-03-29T11:34:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.814+0000 7f32b6800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b0099a60 0x7f32b01599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.814+0000 7f32b6800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b009a910 0x7f32b015f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.814+0000 7f32b6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b015d920 0x7f32b00c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.814+0000 7f32b6800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0160d10 con 0x7f32b015d920 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.814+0000 7f32b6800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b015fc70 con 0x7f32b0099a60 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.814+0000 7f32b6800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b015ff50 con 0x7f32b009a910 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.814+0000 7f32afe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b015d920 0x7f32b00c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32afe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b015d920 0x7f32b00c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32790/0 (socket says 172.21.15.12:32790) 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32afe00700 1 -- 172.21.15.12:0/34603879 learned_addr learned my addr 172.21.15.12:0/34603879 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32af400700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b0099a60 0x7f32b01599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32aea00700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b009a910 0x7f32b015f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32afe00700 1 -- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b009a910 msgr2=0x7f32b015f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32afe00700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b009a910 0x7f32b015f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32afe00700 1 -- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b0099a60 msgr2=0x7f32b01599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32afe00700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b0099a60 0x7f32b01599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32afe00700 1 -- 172.21.15.12:0/34603879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c0094a0 con 0x7f32b015d920 2024-03-29T11:34:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32afe00700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b015d920 0x7f32b00c9160 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f32a8014040 tx=0x7f32a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32a7400700 1 -- 172.21.15.12:0/34603879 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8012a90 con 0x7f32b015d920 2024-03-29T11:34:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32a7400700 1 -- 172.21.15.12:0/34603879 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a8020070 con 0x7f32b015d920 2024-03-29T11:34:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.815+0000 7f32b6800700 1 -- 172.21.15.12:0/34603879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a8009a70 con 0x7f32b015d920 2024-03-29T11:34:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.816+0000 7f32a7400700 1 -- 172.21.15.12:0/34603879 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8012e50 con 0x7f32b015d920 2024-03-29T11:34:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.816+0000 7f32b6800700 1 -- 172.21.15.12:0/34603879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b0160510 con 0x7f32b015d920 2024-03-29T11:34:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.817+0000 7f32b6800700 1 -- 172.21.15.12:0/34603879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b0067720 con 0x7f32b015d920 2024-03-29T11:34:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.818+0000 7f32a7400700 1 -- 172.21.15.12:0/34603879 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 153) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f32a8004040 con 0x7f32b015d920 2024-03-29T11:34:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.818+0000 7f32a7400700 1 -- 172.21.15.12:0/34603879 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f32a8022070 con 0x7f32b015d920 2024-03-29T11:34:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.823+0000 7f32a7400700 1 -- 172.21.15.12:0/34603879 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32a8056250 con 0x7f32b015d920 2024-03-29T11:34:05.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.963+0000 7f32a7400700 1 -- 172.21.15.12:0/34603879 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 154) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f32a8055ed0 con 0x7f32b015d920 2024-03-29T11:34:05.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.963+0000 7f32a7400700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3290036150 0x7f3290072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:05.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.966+0000 7f32af400700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3290036150 0x7f3290072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:05.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:05.970+0000 7f32af400700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3290036150 0x7f3290072570 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f32a00011e0 tx=0x7f32a0009040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:06.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.042+0000 7f32b6800700 1 -- 172.21.15.12:0/34603879 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32b00b72d0 con 0x7f32b015d920 2024-03-29T11:34:06.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.043+0000 7f32a7400700 1 -- 172.21.15.12:0/34603879 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) v1 ==== 74+0+17232 (secure 0 0 0) 0x7f32a8083900 con 0x7f32b015d920 2024-03-29T11:34:06.041 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:06.041 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":271,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:34:04.952250+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:33:34.767741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-03-29T11:33:35.848901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/74425":"2024-03-30T11:34:04.952206+0000","172.21.15.159:0/3422160283":"2024-03-30T11:34:04.952206+0000","172.21.15.159:0/4283701383":"2024-03-30T11:34:04.952206+0000","172.21.15.159:0/3418851152":"2024-03-30T11:34:04.952206+0000","172.21.15.159:0/2774416649":"2024-03-30T11:34:04.952206+0000","172.21.15.159:0/799604193":"2024-03-30T11:34:04.952206+0000","172.21.15.159:0/2007637492":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2270103540":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/883394797":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/288982228":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/1757679477":"2024-03-30T11:34:04.952206+0000","172.21.15.159:0/2254217683":"2024-03-30T11:33:54.133253+0000","172.21.15.159:6833/62646":"2024-03-30T11:33:54.133253+0000","172.21.15.159:6832/62646":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/2308612503":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/3192194169":"2024-03-30T11:33:54.133253+0000","172.21.15.159:0/597141817":"2024-03-30T11:33:54.133253+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:06.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.045+0000 7f32b6800700 1 -- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3290036150 msgr2=0x7f3290072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.045+0000 7f32b6800700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3290036150 0x7f3290072570 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f32a00011e0 tx=0x7f32a0009040 comp rx=0 tx=0).stop 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.045+0000 7f32b6800700 1 -- 172.21.15.12:0/34603879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b015d920 msgr2=0x7f32b00c9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.045+0000 7f32b6800700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b015d920 0x7f32b00c9160 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f32a8014040 tx=0x7f32a8004ad0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.045+0000 7f32b6800700 1 -- 172.21.15.12:0/34603879 shutdown_connections 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.045+0000 7f32b6800700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3290036150 0x7f3290072570 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.045+0000 7f32b6800700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32b015d920 0x7f32b00c9160 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.045+0000 7f32b6800700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32b0099a60 0x7f32b01599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.045+0000 7f32b6800700 1 --2- 172.21.15.12:0/34603879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32b009a910 0x7f32b015f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.045+0000 7f32b6800700 1 -- 172.21.15.12:0/34603879 >> 172.21.15.12:0/34603879 conn(0x7f32b00d1350 msgr2=0x7f32b006ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.046+0000 7f32b6800700 1 -- 172.21.15.12:0/34603879 shutdown_connections 2024-03-29T11:34:06.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.046+0000 7f32b6800700 1 -- 172.21.15.12:0/34603879 wait complete. 2024-03-29T11:34:06.062 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T11:34:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.264+0000 7fea30600700 1 Processor -- start 2024-03-29T11:34:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.264+0000 7fea30600700 1 -- start start 2024-03-29T11:34:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.264+0000 7fea30600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281611b0 0x7fea2815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.264+0000 7fea30600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fea280cee10 0x7fea28161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.264+0000 7fea30600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea280ce270 con 0x7fea2815f4d0 2024-03-29T11:34:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.264+0000 7fea30600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea2815b600 con 0x7fea280cee10 2024-03-29T11:34:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.265+0000 7fea30600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea280d2760 con 0x7fea281611b0 2024-03-29T11:34:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.265+0000 7fea2d200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fea2815f4d0 0x7fea280ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49174/0 (socket says 172.21.15.12:49174) 2024-03-29T11:34:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.265+0000 7fea2d200700 1 -- 172.21.15.12:0/326181543 learned_addr learned my addr 172.21.15.12:0/326181543 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.265+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974806364 0 0) 0x7fea280ce270 con 0x7fea2815f4d0 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.265+0000 7fea2e600700 1 --2- 172.21.15.12:0/326181543 >> v2:172.21.15.159:3300/0 conn(0x7fea280cee10 0x7fea28161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.265+0000 7fea2dc00700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281611b0 0x7fea2815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.265+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea100035b0 con 0x7fea2815f4d0 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.265+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 351896935 0 0) 0x7fea100035b0 con 0x7fea2815f4d0 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.265+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281611b0 msgr2=0x7fea2815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281611b0 0x7fea2815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 >> v2:172.21.15.159:3300/0 conn(0x7fea280cee10 msgr2=0x7fea28161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 --2- 172.21.15.12:0/326181543 >> v2:172.21.15.159:3300/0 conn(0x7fea280cee10 0x7fea28161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea280d2a40 con 0x7fea2815f4d0 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fea14001670 con 0x7fea2815f4d0 2024-03-29T11:34:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 >> v1:172.21.15.12:6789/0 conn(0x7fea2815f4d0 legacy=0x7fea280ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea10003a00 0x7fea10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281611b0 0x7fea10004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea10004d90 0x7fea10005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2815b600 con 0x7fea10003a00 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea280d2760 con 0x7fea10004d90 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1000ba10 con 0x7fea281611b0 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea2dc00700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea10003a00 0x7fea10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea2d200700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281611b0 0x7fea10004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea2e600700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea10004d90 0x7fea10005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea2dc00700 1 -- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281611b0 msgr2=0x7fea10004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea2dc00700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281611b0 0x7fea10004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea2dc00700 1 -- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea10004d90 msgr2=0x7fea10005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea2dc00700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea10004d90 0x7fea10005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea2dc00700 1 -- 172.21.15.12:0/326181543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea100035b0 con 0x7fea10003a00 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea2e600700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea10004d90 0x7fea10005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.266+0000 7fea2d200700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281611b0 0x7fea10004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.267+0000 7fea2dc00700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea10003a00 0x7fea10003e20 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009b60 tx=0x7fea10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293e66665a707916 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.267+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c00e6c0 con 0x7fea10003a00 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.267+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1c00e820 con 0x7fea10003a00 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.267+0000 7fea27e00700 1 -- 172.21.15.12:0/326181543 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c00eb90 con 0x7fea10003a00 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.267+0000 7fea30600700 1 -- 172.21.15.12:0/326181543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea10003a00 msgr2=0x7fea10003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.267+0000 7fea30600700 1 --2- 172.21.15.12:0/326181543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea10003a00 0x7fea10003e20 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009b60 tx=0x7fea10005880 comp rx=0 tx=0).stop 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.267+0000 7fea2e600700 1 -- 172.21.15.12:0/326181543 reap_dead start 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.267+0000 7fea30600700 1 -- 172.21.15.12:0/326181543 shutdown_connections 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.267+0000 7fea30600700 1 -- 172.21.15.12:0/326181543 >> 172.21.15.12:0/326181543 conn(0x7fea2808aff0 msgr2=0x7fea280d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.268+0000 7fea30600700 1 -- 172.21.15.12:0/326181543 shutdown_connections 2024-03-29T11:34:06.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.268+0000 7fea30600700 1 -- 172.21.15.12:0/326181543 wait complete. 2024-03-29T11:34:06.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.269+0000 7fea30600700 1 Processor -- start 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.269+0000 7fea30600700 1 -- start start 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.270+0000 7fea30600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea10003a00 0x7fea280c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.270+0000 7fea30600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281542b0 0x7fea28154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.270+0000 7fea30600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea280c9520 0x7fea280c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.270+0000 7fea30600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea280c9ee0 con 0x7fea280c9520 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.270+0000 7fea30600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea28154710 con 0x7fea10003a00 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.270+0000 7fea30600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea280c8010 con 0x7fea281542b0 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea280c9520 0x7fea280c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea280c9520 0x7fea280c99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32804/0 (socket says 172.21.15.12:32804) 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2e600700 1 -- 172.21.15.12:0/2822037423 learned_addr learned my addr 172.21.15.12:0/2822037423 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2dc00700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea10003a00 0x7fea280c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2d200700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281542b0 0x7fea28154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2e600700 1 -- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281542b0 msgr2=0x7fea28154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2e600700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281542b0 0x7fea28154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2e600700 1 -- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea10003a00 msgr2=0x7fea280c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2e600700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea10003a00 0x7fea280c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2e600700 1 -- 172.21.15.12:0/2822037423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea280c82f0 con 0x7fea280c9520 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.271+0000 7fea2e600700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea280c9520 0x7fea280c99a0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fea2808b520 tx=0x7fea200104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.272+0000 7fea25600700 1 -- 172.21.15.12:0/2822037423 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2001b070 con 0x7fea280c9520 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.272+0000 7fea25600700 1 -- 172.21.15.12:0/2822037423 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2000f040 con 0x7fea280c9520 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.272+0000 7fea30600700 1 -- 172.21.15.12:0/2822037423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea280c85d0 con 0x7fea280c9520 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.272+0000 7fea25600700 1 -- 172.21.15.12:0/2822037423 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2000e050 con 0x7fea280c9520 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.272+0000 7fea30600700 1 -- 172.21.15.12:0/2822037423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea281e6ff0 con 0x7fea280c9520 2024-03-29T11:34:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.273+0000 7fea30600700 1 -- 172.21.15.12:0/2822037423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea280cdec0 con 0x7fea280c9520 2024-03-29T11:34:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.275+0000 7fea25600700 1 -- 172.21.15.12:0/2822037423 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 154) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fea20010d00 con 0x7fea280c9520 2024-03-29T11:34:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.275+0000 7fea25600700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea0806ab10 0x7fea0806cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.275+0000 7fea25600700 1 -- 172.21.15.12:0/2822037423 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6512+0+0 (secure 0 0 0) 0x7fea20089240 con 0x7fea280c9520 2024-03-29T11:34:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.275+0000 7fea2dc00700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea0806ab10 0x7fea0806cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.276+0000 7fea2dc00700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea0806ab10 0x7fea0806cfd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0090b0 tx=0x7fea1c01f040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.279+0000 7fea25600700 1 -- 172.21.15.12:0/2822037423 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea20054390 con 0x7fea280c9520 2024-03-29T11:34:06.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.539+0000 7fea30600700 1 -- 172.21.15.12:0/2822037423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fea28067720 con 0x7fea280c9520 2024-03-29T11:34:06.538 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T11:34:06.541+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T11:34:06.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.550+0000 7fea25600700 1 -- 172.21.15.12:0/2822037423 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v139) v1 ==== 161+0+0 (secure 0 0 0) 0x7fea200579b0 con 0x7fea280c9520 2024-03-29T11:34:06.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 -- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea0806ab10 msgr2=0x7fea0806cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea0806ab10 0x7fea0806cfd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0090b0 tx=0x7fea1c01f040 comp rx=0 tx=0).stop 2024-03-29T11:34:06.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 -- 172.21.15.12:0/2822037423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea280c9520 msgr2=0x7fea280c99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea280c9520 0x7fea280c99a0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fea2808b520 tx=0x7fea200104a0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 -- 172.21.15.12:0/2822037423 shutdown_connections 2024-03-29T11:34:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea0806ab10 0x7fea0806cfd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea280c9520 0x7fea280c99a0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea10003a00 0x7fea280c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 --2- 172.21.15.12:0/2822037423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea281542b0 0x7fea28154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 -- 172.21.15.12:0/2822037423 >> 172.21.15.12:0/2822037423 conn(0x7fea2808aff0 msgr2=0x7fea281609b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 -- 172.21.15.12:0/2822037423 shutdown_connections 2024-03-29T11:34:06.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.552+0000 7fea30600700 1 -- 172.21.15.12:0/2822037423 wait complete. 2024-03-29T11:34:06.553 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T11:34:06.561 INFO:tasks.ceph.mds.d.smithi159.stderr: -5250> 2024-03-29T11:33:41.789+0000 7f4d26400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:34:06.569 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T11:34:06.613 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T11:34:06.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.776+0000 7f39aaa00700 1 Processor -- start 2024-03-29T11:34:06.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.776+0000 7f39aaa00700 1 -- start start 2024-03-29T11:34:06.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.777+0000 7f39aaa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39a40d21f0 0x7f39a40ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.777+0000 7f39aaa00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a40ce270 con 0x7f39a40cec40 2024-03-29T11:34:06.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.777+0000 7f39aaa00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a415b600 con 0x7f39a41611b0 2024-03-29T11:34:06.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.777+0000 7f39aaa00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a40d2760 con 0x7f39a40d21f0 2024-03-29T11:34:06.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.777+0000 7f39a3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39a40d21f0 0x7f39a40ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.777+0000 7f39a8a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f39a40cec40 0x7f39a4161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49184/0 (socket says 172.21.15.12:49184) 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.777+0000 7f39a8a00700 1 -- 172.21.15.12:0/3350714537 learned_addr learned my addr 172.21.15.12:0/3350714537 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.778+0000 7f39a3400700 1 -- 172.21.15.12:0/3350714537 >> v1:172.21.15.159:6789/0 conn(0x7f39a41611b0 legacy=0x7f39a415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.778+0000 7f39a3400700 1 -- 172.21.15.12:0/3350714537 >> v1:172.21.15.12:6789/0 conn(0x7f39a40cec40 legacy=0x7f39a4161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.778+0000 7f39a3400700 1 -- 172.21.15.12:0/3350714537 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a40d2a40 con 0x7f39a40d21f0 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.778+0000 7f39a2a00700 1 -- 172.21.15.12:0/3350714537 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438865188 0 0) 0x7f39a40ce270 con 0x7f39a40cec40 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.778+0000 7f39a3400700 1 --2- 172.21.15.12:0/3350714537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40d21f0 0x7f39a40ce700 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f3990009840 tx=0x7f3990009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff20314338be80a server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.778+0000 7f39a2a00700 1 -- 172.21.15.12:0/3350714537 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990011830 con 0x7f39a40d21f0 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.778+0000 7f39a2a00700 1 -- 172.21.15.12:0/3350714537 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3990011990 con 0x7f39a40d21f0 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.778+0000 7f39a2a00700 1 -- 172.21.15.12:0/3350714537 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990011d20 con 0x7f39a40d21f0 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.779+0000 7f39aaa00700 1 -- 172.21.15.12:0/3350714537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40d21f0 msgr2=0x7f39a40ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.779+0000 7f39aaa00700 1 --2- 172.21.15.12:0/3350714537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40d21f0 0x7f39a40ce700 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f3990009840 tx=0x7f3990009b50 comp rx=0 tx=0).stop 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.779+0000 7f39aaa00700 1 -- 172.21.15.12:0/3350714537 shutdown_connections 2024-03-29T11:34:06.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.779+0000 7f39aaa00700 1 --2- 172.21.15.12:0/3350714537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40d21f0 0x7f39a40ce700 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.779+0000 7f39aaa00700 1 -- 172.21.15.12:0/3350714537 >> 172.21.15.12:0/3350714537 conn(0x7f39a408aff0 msgr2=0x7f39a40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:06.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.779+0000 7f39aaa00700 1 -- 172.21.15.12:0/3350714537 shutdown_connections 2024-03-29T11:34:06.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.779+0000 7f39aaa00700 1 -- 172.21.15.12:0/3350714537 wait complete. 2024-03-29T11:34:06.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.781+0000 7f39aaa00700 1 Processor -- start 2024-03-29T11:34:06.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.781+0000 7f39aaa00700 1 -- start start 2024-03-29T11:34:06.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39aaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40cec40 0x7f39a40c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39aaa00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39a40d21f0 0x7f39a4153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39aaa00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39a41611b0 0x7f39a40b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39aaa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4155e80 con 0x7f39a40cec40 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39aaa00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4155200 con 0x7f39a40d21f0 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39aaa00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a41554e0 con 0x7f39a41611b0 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40cec40 0x7f39a40c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40cec40 0x7f39a40c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32824/0 (socket says 172.21.15.12:32824) 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a3e00700 1 -- 172.21.15.12:0/3882745 learned_addr learned my addr 172.21.15.12:0/3882745 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a3400700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39a40d21f0 0x7f39a4153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a8a00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39a41611b0 0x7f39a40b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a3e00700 1 -- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39a41611b0 msgr2=0x7f39a40b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a3e00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39a41611b0 0x7f39a40b72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a3e00700 1 -- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39a40d21f0 msgr2=0x7f39a4153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a3e00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39a40d21f0 0x7f39a4153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a3e00700 1 -- 172.21.15.12:0/3882745 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39900094a0 con 0x7f39a40cec40 2024-03-29T11:34:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39a3e00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40cec40 0x7f39a40c7060 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f399400c700 tx=0x7f399400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f399be00700 1 -- 172.21.15.12:0/3882745 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39940052d0 con 0x7f39a40cec40 2024-03-29T11:34:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f399be00700 1 -- 172.21.15.12:0/3882745 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3994005430 con 0x7f39a40cec40 2024-03-29T11:34:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f399be00700 1 -- 172.21.15.12:0/3882745 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3994003880 con 0x7f39a40cec40 2024-03-29T11:34:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39aaa00700 1 -- 172.21.15.12:0/3882745 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a41557c0 con 0x7f39a40cec40 2024-03-29T11:34:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.783+0000 7f39aaa00700 1 -- 172.21.15.12:0/3882745 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a4153f70 con 0x7f39a40cec40 2024-03-29T11:34:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.784+0000 7f39aaa00700 1 -- 172.21.15.12:0/3882745 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a4067720 con 0x7f39a40cec40 2024-03-29T11:34:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.785+0000 7f399be00700 1 -- 172.21.15.12:0/3882745 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 154) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f39940039e0 con 0x7f39a40cec40 2024-03-29T11:34:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.786+0000 7f399be00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f398406aae0 0x7f398406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.786+0000 7f399be00700 1 -- 172.21.15.12:0/3882745 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f39940887b0 con 0x7f39a40cec40 2024-03-29T11:34:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.786+0000 7f39a3400700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f398406aae0 0x7f398406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:06.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.787+0000 7f39a3400700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f398406aae0 0x7f398406cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f39a41e8220 tx=0x7f399001c040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:06.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:06.791+0000 7f399be00700 1 -- 172.21.15.12:0/3882745 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39940538b0 con 0x7f39a40cec40 2024-03-29T11:34:07.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.054+0000 7f39aaa00700 1 -- 172.21.15.12:0/3882745 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f39a40b5d00 con 0x7f39a40cec40 2024-03-29T11:34:07.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.554+0000 7f399be00700 1 -- 172.21.15.12:0/3882745 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v140) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3994056ed0 con 0x7f39a40cec40 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 -- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f398406aae0 msgr2=0x7f398406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f398406aae0 0x7f398406cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f39a41e8220 tx=0x7f399001c040 comp rx=0 tx=0).stop 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 -- 172.21.15.12:0/3882745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40cec40 msgr2=0x7f39a40c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40cec40 0x7f39a40c7060 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f399400c700 tx=0x7f399400cac0 comp rx=0 tx=0).stop 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 -- 172.21.15.12:0/3882745 shutdown_connections 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f398406aae0 0x7f398406cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39a41611b0 0x7f39a40b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a40cec40 0x7f39a40c7060 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 --2- 172.21.15.12:0/3882745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39a40d21f0 0x7f39a4153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 -- 172.21.15.12:0/3882745 >> 172.21.15.12:0/3882745 conn(0x7f39a408aff0 msgr2=0x7f39a406d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 -- 172.21.15.12:0/3882745 shutdown_connections 2024-03-29T11:34:07.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.557+0000 7f39aaa00700 1 -- 172.21.15.12:0/3882745 wait complete. 2024-03-29T11:34:07.578 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T11:34:07.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.786+0000 7fcc59800700 1 Processor -- start 2024-03-29T11:34:07.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.786+0000 7fcc59800700 1 -- start start 2024-03-29T11:34:07.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.787+0000 7fcc59800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcc540ced30 0x7fcc540d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.787+0000 7fcc59800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcc540d2710 0x7fcc5415f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.787+0000 7fcc59800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc540ce8a0 con 0x7fcc5415ddf0 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.787+0000 7fcc59800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5415ec60 con 0x7fcc540ced30 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.787+0000 7fcc59800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc540d2b60 con 0x7fcc540d2710 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.788+0000 7fcc4be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcc540ced30 0x7fcc540d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.788+0000 7fcc52a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcc5415ddf0 0x7fcc5415e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49192/0 (socket says 172.21.15.12:49192) 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.788+0000 7fcc52a00700 1 -- 172.21.15.12:0/142059959 learned_addr learned my addr 172.21.15.12:0/142059959 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.788+0000 7fcc53400700 1 --2- 172.21.15.12:0/142059959 >> v2:172.21.15.159:3300/0 conn(0x7fcc540d2710 0x7fcc5415f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.788+0000 7fcc4be00700 1 -- 172.21.15.12:0/142059959 >> v2:172.21.15.159:3300/0 conn(0x7fcc540d2710 msgr2=0x7fcc5415f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.788+0000 7fcc4be00700 1 --2- 172.21.15.12:0/142059959 >> v2:172.21.15.159:3300/0 conn(0x7fcc540d2710 0x7fcc5415f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.788+0000 7fcc4be00700 1 -- 172.21.15.12:0/142059959 >> v1:172.21.15.12:6789/0 conn(0x7fcc5415ddf0 legacy=0x7fcc5415e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:07.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.788+0000 7fcc4be00700 1 -- 172.21.15.12:0/142059959 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc540cdc90 con 0x7fcc540ced30 2024-03-29T11:34:07.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.788+0000 7fcc52000700 1 -- 172.21.15.12:0/142059959 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368300889 0 0) 0x7fcc540ce8a0 con 0x7fcc5415ddf0 2024-03-29T11:34:07.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.788+0000 7fcc4be00700 1 --2- 172.21.15.12:0/142059959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc540ced30 0x7fcc540d21d0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fcc540cf9f0 tx=0x7fcc3c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ac4fc7a6a3839e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:07.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc52000700 1 -- 172.21.15.12:0/142059959 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c011830 con 0x7fcc540ced30 2024-03-29T11:34:07.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc52000700 1 -- 172.21.15.12:0/142059959 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3c011990 con 0x7fcc540ced30 2024-03-29T11:34:07.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc52000700 1 -- 172.21.15.12:0/142059959 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c011d20 con 0x7fcc540ced30 2024-03-29T11:34:07.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc59800700 1 -- 172.21.15.12:0/142059959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc540ced30 msgr2=0x7fcc540d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:07.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc59800700 1 --2- 172.21.15.12:0/142059959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc540ced30 0x7fcc540d21d0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fcc540cf9f0 tx=0x7fcc3c009b20 comp rx=0 tx=0).stop 2024-03-29T11:34:07.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc59800700 1 -- 172.21.15.12:0/142059959 shutdown_connections 2024-03-29T11:34:07.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc59800700 1 --2- 172.21.15.12:0/142059959 >> v2:172.21.15.159:3300/0 conn(0x7fcc540d2710 0x7fcc5415f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:07.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc59800700 1 --2- 172.21.15.12:0/142059959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc540ced30 0x7fcc540d21d0 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:07.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc59800700 1 -- 172.21.15.12:0/142059959 >> 172.21.15.12:0/142059959 conn(0x7fcc5408aff0 msgr2=0x7fcc540d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:07.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc59800700 1 -- 172.21.15.12:0/142059959 shutdown_connections 2024-03-29T11:34:07.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.789+0000 7fcc59800700 1 -- 172.21.15.12:0/142059959 wait complete. 2024-03-29T11:34:07.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.791+0000 7fcc59800700 1 Processor -- start 2024-03-29T11:34:07.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.791+0000 7fcc59800700 1 -- start start 2024-03-29T11:34:07.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.792+0000 7fcc59800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc540ced30 0x7fcc541f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:07.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.792+0000 7fcc59800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc540d2710 0x7fcc5415f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:07.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.792+0000 7fcc59800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc5415ddf0 0x7fcc5415f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:07.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.792+0000 7fcc59800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc540cab60 con 0x7fcc5415ddf0 2024-03-29T11:34:07.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.792+0000 7fcc59800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc540cce20 con 0x7fcc540ced30 2024-03-29T11:34:07.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.792+0000 7fcc59800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc540cd070 con 0x7fcc540d2710 2024-03-29T11:34:07.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.792+0000 7fcc53400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc5415ddf0 0x7fcc5415f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:07.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc53400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc5415ddf0 0x7fcc5415f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32852/0 (socket says 172.21.15.12:32852) 2024-03-29T11:34:07.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc53400700 1 -- 172.21.15.12:0/2723737318 learned_addr learned my addr 172.21.15.12:0/2723737318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:07.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc52a00700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc540ced30 0x7fcc541f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:07.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc4be00700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc540d2710 0x7fcc5415f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:07.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc53400700 1 -- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc540d2710 msgr2=0x7fcc5415f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:07.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc53400700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc540d2710 0x7fcc5415f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:07.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc53400700 1 -- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc540ced30 msgr2=0x7fcc541f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:07.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc53400700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc540ced30 0x7fcc541f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:07.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc53400700 1 -- 172.21.15.12:0/2723737318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c0094a0 con 0x7fcc5415ddf0 2024-03-29T11:34:07.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc53400700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc5415ddf0 0x7fcc5415f5b0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c007e40 tx=0x7fcc4c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:07.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc4aa00700 1 -- 172.21.15.12:0/2723737318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00eea0 con 0x7fcc5415ddf0 2024-03-29T11:34:07.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc4aa00700 1 -- 172.21.15.12:0/2723737318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c014420 con 0x7fcc5415ddf0 2024-03-29T11:34:07.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.793+0000 7fcc4aa00700 1 -- 172.21.15.12:0/2723737318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0147b0 con 0x7fcc5415ddf0 2024-03-29T11:34:07.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.794+0000 7fcc59800700 1 -- 172.21.15.12:0/2723737318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc540cd350 con 0x7fcc5415ddf0 2024-03-29T11:34:07.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.794+0000 7fcc59800700 1 -- 172.21.15.12:0/2723737318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc541f2ea0 con 0x7fcc5415ddf0 2024-03-29T11:34:07.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.795+0000 7fcc59800700 1 -- 172.21.15.12:0/2723737318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc540ca5e0 con 0x7fcc5415ddf0 2024-03-29T11:34:07.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.796+0000 7fcc4aa00700 1 -- 172.21.15.12:0/2723737318 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 155) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fcc4c014910 con 0x7fcc5415ddf0 2024-03-29T11:34:07.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.796+0000 7fcc4aa00700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc3006ad20 0x7fcc3006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:07.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.797+0000 7fcc4aa00700 1 -- 172.21.15.12:0/2723737318 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fcc4c089fe0 con 0x7fcc5415ddf0 2024-03-29T11:34:07.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.797+0000 7fcc52a00700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc3006ad20 0x7fcc3006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:07.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.798+0000 7fcc52a00700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc3006ad20 0x7fcc3006d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcc44000f60 tx=0x7fcc44007c90 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:07.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:07.801+0000 7fcc4aa00700 1 -- 172.21.15.12:0/2723737318 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcc540ca5e0 con 0x7fcc5415ddf0 2024-03-29T11:34:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.017+0000 7fcc59800700 1 -- 172.21.15.12:0/2723737318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcc540caca0 con 0x7fcc5415ddf0 2024-03-29T11:34:08.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.969+0000 7fcc4aa00700 1 -- 172.21.15.12:0/2723737318 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v273) v1 ==== 181+0+0 (secure 0 0 0) 0x7fcc540caca0 con 0x7fcc5415ddf0 2024-03-29T11:34:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 -- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc3006ad20 msgr2=0x7fcc3006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc3006ad20 0x7fcc3006d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcc44000f60 tx=0x7fcc44007c90 comp rx=0 tx=0).stop 2024-03-29T11:34:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 -- 172.21.15.12:0/2723737318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc5415ddf0 msgr2=0x7fcc5415f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc5415ddf0 0x7fcc5415f5b0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c007e40 tx=0x7fcc4c00e3c0 comp rx=0 tx=0).stop 2024-03-29T11:34:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 -- 172.21.15.12:0/2723737318 shutdown_connections 2024-03-29T11:34:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcc3006ad20 0x7fcc3006d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc5415ddf0 0x7fcc5415f5b0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc540ced30 0x7fcc541f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 --2- 172.21.15.12:0/2723737318 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc540d2710 0x7fcc5415f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 -- 172.21.15.12:0/2723737318 >> 172.21.15.12:0/2723737318 conn(0x7fcc5408aff0 msgr2=0x7fcc540b8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 -- 172.21.15.12:0/2723737318 shutdown_connections 2024-03-29T11:34:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:08.975+0000 7fcc59800700 1 -- 172.21.15.12:0/2723737318 wait complete. 2024-03-29T11:34:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T11:34:08.994 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T11:34:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.206+0000 7f6765000700 1 Processor -- start 2024-03-29T11:34:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.206+0000 7f6765000700 1 -- start start 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f6765000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f676015d930 0x7f67600d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f6765000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6760162030 0x7f67600d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f6765000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67600d20d0 0x7f67600cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f6765000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676015ec40 con 0x7f6760162030 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f6765000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67600d2560 con 0x7f676015d930 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f6765000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676015dd10 con 0x7f67600d20d0 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f675d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6760162030 0x7f67600d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f675d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6760162030 0x7f67600d1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56048/0 (socket says 172.21.15.12:56048) 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f675d400700 1 -- 172.21.15.12:0/1876236016 learned_addr learned my addr 172.21.15.12:0/1876236016 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f675e800700 1 --2- 172.21.15.12:0/1876236016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67600d20d0 0x7f67600cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.208+0000 7f675de00700 1 --2- 172.21.15.12:0/1876236016 >> v2:172.21.15.159:3300/0 conn(0x7f676015d930 0x7f67600d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.209+0000 7f675d400700 1 -- 172.21.15.12:0/1876236016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67600d20d0 msgr2=0x7f67600cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.209+0000 7f675d400700 1 --2- 172.21.15.12:0/1876236016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67600d20d0 0x7f67600cfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.209+0000 7f675d400700 1 -- 172.21.15.12:0/1876236016 >> v2:172.21.15.159:3300/0 conn(0x7f676015d930 msgr2=0x7f67600d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.209+0000 7f675d400700 1 --2- 172.21.15.12:0/1876236016 >> v2:172.21.15.159:3300/0 conn(0x7f676015d930 0x7f67600d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.209+0000 7f675d400700 1 -- 172.21.15.12:0/1876236016 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676015dfc0 con 0x7f6760162030 2024-03-29T11:34:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.209+0000 7f675d400700 1 --2- 172.21.15.12:0/1876236016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 0x7f67600d1b90 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f67600a1430 tx=0x7f674c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa864f410563e7e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.209+0000 7f675ca00700 1 -- 172.21.15.12:0/1876236016 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c00b870 con 0x7f6760162030 2024-03-29T11:34:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.209+0000 7f675ca00700 1 -- 172.21.15.12:0/1876236016 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c00b9d0 con 0x7f6760162030 2024-03-29T11:34:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.209+0000 7f675ca00700 1 -- 172.21.15.12:0/1876236016 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c00bd60 con 0x7f6760162030 2024-03-29T11:34:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.210+0000 7f6765000700 1 -- 172.21.15.12:0/1876236016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 msgr2=0x7f67600d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.210+0000 7f6765000700 1 --2- 172.21.15.12:0/1876236016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 0x7f67600d1b90 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f67600a1430 tx=0x7f674c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.210+0000 7f6765000700 1 -- 172.21.15.12:0/1876236016 shutdown_connections 2024-03-29T11:34:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.210+0000 7f6765000700 1 --2- 172.21.15.12:0/1876236016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67600d20d0 0x7f67600cfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.210+0000 7f6765000700 1 --2- 172.21.15.12:0/1876236016 >> v2:172.21.15.159:3300/0 conn(0x7f676015d930 0x7f67600d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.210+0000 7f6765000700 1 --2- 172.21.15.12:0/1876236016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 0x7f67600d1b90 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.210+0000 7f6765000700 1 -- 172.21.15.12:0/1876236016 >> 172.21.15.12:0/1876236016 conn(0x7f676008aff0 msgr2=0x7f67600d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:09.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.210+0000 7f6765000700 1 -- 172.21.15.12:0/1876236016 shutdown_connections 2024-03-29T11:34:09.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.210+0000 7f6765000700 1 -- 172.21.15.12:0/1876236016 wait complete. 2024-03-29T11:34:09.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.212+0000 7f6765000700 1 Processor -- start 2024-03-29T11:34:09.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.212+0000 7f6765000700 1 -- start start 2024-03-29T11:34:09.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f6765000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67600d20d0 0x7f67600cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:09.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f6765000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f676015d930 0x7f676015bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:09.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f6765000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 0x7f67601581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f6765000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67600ca080 con 0x7f6760162030 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f6765000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67601586f0 con 0x7f676015d930 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f6765000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67600c9da0 con 0x7f67600d20d0 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f675e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 0x7f67601581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f675de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67600d20d0 0x7f67600cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f675e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 0x7f67601581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56056/0 (socket says 172.21.15.12:56056) 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f675e800700 1 -- 172.21.15.12:0/2253839248 learned_addr learned my addr 172.21.15.12:0/2253839248 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f675d400700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f676015d930 0x7f676015bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f675e800700 1 -- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67600d20d0 msgr2=0x7f67600cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f675e800700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67600d20d0 0x7f67600cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f675e800700 1 -- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f676015d930 msgr2=0x7f676015bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f675e800700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f676015d930 0x7f676015bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.214+0000 7f675e800700 1 -- 172.21.15.12:0/2253839248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c009550 con 0x7f6760162030 2024-03-29T11:34:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.215+0000 7f675e800700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 0x7f67601581b0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f6758007d20 tx=0x7f675800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.215+0000 7f6756000700 1 -- 172.21.15.12:0/2253839248 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675800ecd0 con 0x7f6760162030 2024-03-29T11:34:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.215+0000 7f6756000700 1 -- 172.21.15.12:0/2253839248 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675800ee50 con 0x7f6760162030 2024-03-29T11:34:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.215+0000 7f6765000700 1 -- 172.21.15.12:0/2253839248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67600ca820 con 0x7f6760162030 2024-03-29T11:34:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.215+0000 7f6756000700 1 -- 172.21.15.12:0/2253839248 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758014660 con 0x7f6760162030 2024-03-29T11:34:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.215+0000 7f6765000700 1 -- 172.21.15.12:0/2253839248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67600cade0 con 0x7f6760162030 2024-03-29T11:34:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.216+0000 7f6765000700 1 -- 172.21.15.12:0/2253839248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67600ca680 con 0x7f6760162030 2024-03-29T11:34:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.217+0000 7f6756000700 1 -- 172.21.15.12:0/2253839248 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 155) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f67580147c0 con 0x7f6760162030 2024-03-29T11:34:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.218+0000 7f6756000700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f674006ad20 0x7f674006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.218+0000 7f6756000700 1 -- 172.21.15.12:0/2253839248 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6160+0+0 (secure 0 0 0) 0x7f6758089ac0 con 0x7f6760162030 2024-03-29T11:34:09.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.219+0000 7f675de00700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f674006ad20 0x7f674006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:09.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.219+0000 7f675de00700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f674006ad20 0x7f674006d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6750006b20 tx=0x7f6750008040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:09.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.222+0000 7f6756000700 1 -- 172.21.15.12:0/2253839248 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67600ca680 con 0x7f6760162030 2024-03-29T11:34:09.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.356+0000 7f6756000700 1 -- 172.21.15.12:0/2253839248 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6758012dc0 con 0x7f6760162030 2024-03-29T11:34:09.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:09.442+0000 7f6765000700 1 -- 172.21.15.12:0/2253839248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f67600ca680 con 0x7f6760162030 2024-03-29T11:34:10.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.358+0000 7f6756000700 1 -- 172.21.15.12:0/2253839248 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v274) v1 ==== 169+0+0 (secure 0 0 0) 0x7f67600ca680 con 0x7f6760162030 2024-03-29T11:34:10.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.362+0000 7f6765000700 1 -- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f674006ad20 msgr2=0x7f674006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.362+0000 7f6765000700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f674006ad20 0x7f674006d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6750006b20 tx=0x7f6750008040 comp rx=0 tx=0).stop 2024-03-29T11:34:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.362+0000 7f6765000700 1 -- 172.21.15.12:0/2253839248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 msgr2=0x7f67601581b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.362+0000 7f6765000700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 0x7f67601581b0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f6758007d20 tx=0x7f675800e3c0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.363+0000 7f6765000700 1 -- 172.21.15.12:0/2253839248 shutdown_connections 2024-03-29T11:34:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.363+0000 7f6765000700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f674006ad20 0x7f674006d1e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.363+0000 7f6765000700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760162030 0x7f67601581b0 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.363+0000 7f6765000700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f67600d20d0 0x7f67600cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.363+0000 7f6765000700 1 --2- 172.21.15.12:0/2253839248 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f676015d930 0x7f676015bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.363+0000 7f6765000700 1 -- 172.21.15.12:0/2253839248 >> 172.21.15.12:0/2253839248 conn(0x7f676008aff0 msgr2=0x7f676006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.364+0000 7f6765000700 1 -- 172.21.15.12:0/2253839248 shutdown_connections 2024-03-29T11:34:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.364+0000 7f6765000700 1 -- 172.21.15.12:0/2253839248 wait complete. 2024-03-29T11:34:10.363 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T11:34:10.382 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T11:34:10.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.589+0000 7f81dfe00700 1 Processor -- start 2024-03-29T11:34:10.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.589+0000 7f81dfe00700 1 -- start start 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.590+0000 7f81dfe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f81d8160e50 0x7f81d80b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.590+0000 7f81dfe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81d80d3f20 0x7f81d80b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.590+0000 7f81dfe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f81d80b3f80 0x7f81d80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.590+0000 7f81dfe00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d8161230 con 0x7f81d8160e50 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.590+0000 7f81dfe00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d80d2a80 con 0x7f81d80b3f80 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.590+0000 7f81dfe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80d43b0 con 0x7f81d80d3f20 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.590+0000 7f81dd400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f81d8160e50 0x7f81d80b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.590+0000 7f81dd400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f81d8160e50 0x7f81d80b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56074/0 (socket says 172.21.15.12:56074) 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.590+0000 7f81dd400700 1 -- 172.21.15.12:0/1977580358 learned_addr learned my addr 172.21.15.12:0/1977580358 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81dca00700 1 --2- 172.21.15.12:0/1977580358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81d80d3f20 0x7f81d80b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:10.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81dde00700 1 --2- 172.21.15.12:0/1977580358 >> v2:172.21.15.159:3300/0 conn(0x7f81d80b3f80 0x7f81d80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:10.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81dd400700 1 -- 172.21.15.12:0/1977580358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81d80d3f20 msgr2=0x7f81d80b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:10.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81dd400700 1 --2- 172.21.15.12:0/1977580358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81d80d3f20 0x7f81d80b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81dd400700 1 -- 172.21.15.12:0/1977580358 >> v2:172.21.15.159:3300/0 conn(0x7f81d80b3f80 msgr2=0x7f81d80b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:10.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81dd400700 1 --2- 172.21.15.12:0/1977580358 >> v2:172.21.15.159:3300/0 conn(0x7f81d80b3f80 0x7f81d80b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81dd400700 1 -- 172.21.15.12:0/1977580358 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d80d21e0 con 0x7f81d8160e50 2024-03-29T11:34:10.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81dd400700 1 --2- 172.21.15.12:0/1977580358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 0x7f81d80b4e90 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f81d80b95b0 tx=0x7f81cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d4a012536327a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:10.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81d7e00700 1 -- 172.21.15.12:0/1977580358 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00b870 con 0x7f81d8160e50 2024-03-29T11:34:10.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81d7e00700 1 -- 172.21.15.12:0/1977580358 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc00b9d0 con 0x7f81d8160e50 2024-03-29T11:34:10.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.591+0000 7f81d7e00700 1 -- 172.21.15.12:0/1977580358 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00bd60 con 0x7f81d8160e50 2024-03-29T11:34:10.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.592+0000 7f81dfe00700 1 -- 172.21.15.12:0/1977580358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 msgr2=0x7f81d80b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:10.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.592+0000 7f81dfe00700 1 --2- 172.21.15.12:0/1977580358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 0x7f81d80b4e90 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f81d80b95b0 tx=0x7f81cc009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.592+0000 7f81dfe00700 1 -- 172.21.15.12:0/1977580358 shutdown_connections 2024-03-29T11:34:10.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.592+0000 7f81dfe00700 1 --2- 172.21.15.12:0/1977580358 >> v2:172.21.15.159:3300/0 conn(0x7f81d80b3f80 0x7f81d80b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.592+0000 7f81dfe00700 1 --2- 172.21.15.12:0/1977580358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 0x7f81d80b4e90 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.592+0000 7f81dfe00700 1 --2- 172.21.15.12:0/1977580358 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81d80d3f20 0x7f81d80b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.592+0000 7f81dfe00700 1 -- 172.21.15.12:0/1977580358 >> 172.21.15.12:0/1977580358 conn(0x7f81d808aff0 msgr2=0x7f81d80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:10.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.592+0000 7f81dfe00700 1 -- 172.21.15.12:0/1977580358 shutdown_connections 2024-03-29T11:34:10.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.592+0000 7f81dfe00700 1 -- 172.21.15.12:0/1977580358 wait complete. 2024-03-29T11:34:10.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.594+0000 7f81dfe00700 1 Processor -- start 2024-03-29T11:34:10.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.595+0000 7f81dfe00700 1 -- start start 2024-03-29T11:34:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dfe00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81d80b3f80 0x7f81d80cc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:10.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 0x7f81d80b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dfe00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81d80cc690 0x7f81d80b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dfe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80ca600 con 0x7f81d8160e50 2024-03-29T11:34:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dfe00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80b8f50 con 0x7f81d80cc690 2024-03-29T11:34:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dfe00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80ca850 con 0x7f81d80b3f80 2024-03-29T11:34:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 0x7f81d80b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 0x7f81d80b6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56076/0 (socket says 172.21.15.12:56076) 2024-03-29T11:34:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dca00700 1 -- 172.21.15.12:0/2805350649 learned_addr learned my addr 172.21.15.12:0/2805350649 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dd400700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81d80b3f80 0x7f81d80cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dde00700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81d80cc690 0x7f81d80b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dca00700 1 -- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81d80b3f80 msgr2=0x7f81d80cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dca00700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81d80b3f80 0x7f81d80cc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dca00700 1 -- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81d80cc690 msgr2=0x7f81d80b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.596+0000 7f81dca00700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81d80cc690 0x7f81d80b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.597+0000 7f81dca00700 1 -- 172.21.15.12:0/2805350649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc009550 con 0x7f81d8160e50 2024-03-29T11:34:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.597+0000 7f81dca00700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 0x7f81d80b6150 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f81c400e810 tx=0x7f81c400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.597+0000 7f81d5600700 1 -- 172.21.15.12:0/2805350649 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c400c070 con 0x7f81d8160e50 2024-03-29T11:34:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.597+0000 7f81d5600700 1 -- 172.21.15.12:0/2805350649 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c4013610 con 0x7f81d8160e50 2024-03-29T11:34:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.597+0000 7f81dfe00700 1 -- 172.21.15.12:0/2805350649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d80cab30 con 0x7f81d8160e50 2024-03-29T11:34:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.597+0000 7f81d5600700 1 -- 172.21.15.12:0/2805350649 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c40139c0 con 0x7f81d8160e50 2024-03-29T11:34:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.597+0000 7f81dfe00700 1 -- 172.21.15.12:0/2805350649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d81f4050 con 0x7f81d8160e50 2024-03-29T11:34:10.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.598+0000 7f81dfe00700 1 -- 172.21.15.12:0/2805350649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d80cb050 con 0x7f81d8160e50 2024-03-29T11:34:10.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.599+0000 7f81d5600700 1 -- 172.21.15.12:0/2805350649 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f81c4013b20 con 0x7f81d8160e50 2024-03-29T11:34:10.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.600+0000 7f81d5600700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81b806adb0 0x7f81b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:10.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.600+0000 7f81d5600700 1 -- 172.21.15.12:0/2805350649 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f81c4089000 con 0x7f81d8160e50 2024-03-29T11:34:10.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.601+0000 7f81dd400700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81b806adb0 0x7f81b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:10.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.602+0000 7f81dd400700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81b806adb0 0x7f81b806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f81cc014040 tx=0x7f81cc008030 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:10.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.605+0000 7f81d5600700 1 -- 172.21.15.12:0/2805350649 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81d80cb050 con 0x7f81d8160e50 2024-03-29T11:34:10.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.822+0000 7f81dfe00700 1 -- 172.21.15.12:0/2805350649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f81d80cb050 con 0x7f81d8160e50 2024-03-29T11:34:10.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.826+0000 7f81d5600700 1 -- 172.21.15.12:0/2805350649 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v91) v1 ==== 115+0+0 (secure 0 0 0) 0x7f81d80cb050 con 0x7f81d8160e50 2024-03-29T11:34:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 -- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81b806adb0 msgr2=0x7f81b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81b806adb0 0x7f81b806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f81cc014040 tx=0x7f81cc008030 comp rx=0 tx=0).stop 2024-03-29T11:34:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 -- 172.21.15.12:0/2805350649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 msgr2=0x7f81d80b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 0x7f81d80b6150 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f81c400e810 tx=0x7f81c400eb20 comp rx=0 tx=0).stop 2024-03-29T11:34:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 -- 172.21.15.12:0/2805350649 shutdown_connections 2024-03-29T11:34:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f81b806adb0 0x7f81b806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81d80cc690 0x7f81d80b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81d80b3f80 0x7f81d80cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 --2- 172.21.15.12:0/2805350649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81d8160e50 0x7f81d80b6150 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:10.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 -- 172.21.15.12:0/2805350649 >> 172.21.15.12:0/2805350649 conn(0x7f81d808aff0 msgr2=0x7f81d80d1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:10.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 -- 172.21.15.12:0/2805350649 shutdown_connections 2024-03-29T11:34:10.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:10.829+0000 7f81dfe00700 1 -- 172.21.15.12:0/2805350649 wait complete. 2024-03-29T11:34:10.844 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T11:34:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.048+0000 7f9e5c800700 1 Processor -- start 2024-03-29T11:34:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.048+0000 7f9e5c800700 1 -- start start 2024-03-29T11:34:11.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.049+0000 7f9e5c800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e5415d9d0 0x7f9e540d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.049+0000 7f9e5c800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9e540d2540 0x7f9e540ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.049+0000 7f9e5c800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e54161420 con 0x7f9e540d16a0 2024-03-29T11:34:11.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.049+0000 7f9e5c800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5415edc0 con 0x7f9e5415d9d0 2024-03-29T11:34:11.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.049+0000 7f9e5c800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e54161700 con 0x7f9e540d2540 2024-03-29T11:34:11.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.049+0000 7f9e59e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e5415d9d0 0x7f9e540d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.049+0000 7f9e59e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e5415d9d0 0x7f9e540d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56080/0 (socket says 172.21.15.12:56080) 2024-03-29T11:34:11.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.049+0000 7f9e59e00700 1 -- 172.21.15.12:0/1038207334 learned_addr learned my addr 172.21.15.12:0/1038207334 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:11.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.050+0000 7f9e59e00700 1 -- 172.21.15.12:0/1038207334 >> v2:172.21.15.159:3300/0 conn(0x7f9e540d2540 msgr2=0x7f9e540ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:11.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.050+0000 7f9e59e00700 1 --2- 172.21.15.12:0/1038207334 >> v2:172.21.15.159:3300/0 conn(0x7f9e540d2540 0x7f9e540ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.050+0000 7f9e59e00700 1 -- 172.21.15.12:0/1038207334 >> v1:172.21.15.159:6789/0 conn(0x7f9e540d16a0 legacy=0x7f9e540d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:11.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.050+0000 7f9e59e00700 1 -- 172.21.15.12:0/1038207334 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e541619e0 con 0x7f9e5415d9d0 2024-03-29T11:34:11.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.050+0000 7f9e59e00700 1 --2- 172.21.15.12:0/1038207334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5415d9d0 0x7f9e540d3c30 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f9e5408ca10 tx=0x7f9e48009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8cea9368c6fead server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:11.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.050+0000 7f9e58a00700 1 -- 172.21.15.12:0/1038207334 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48004230 con 0x7f9e5415d9d0 2024-03-29T11:34:11.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.050+0000 7f9e58a00700 1 -- 172.21.15.12:0/1038207334 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e48004390 con 0x7f9e5415d9d0 2024-03-29T11:34:11.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.050+0000 7f9e58a00700 1 -- 172.21.15.12:0/1038207334 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e480036a0 con 0x7f9e5415d9d0 2024-03-29T11:34:11.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.051+0000 7f9e5c800700 1 -- 172.21.15.12:0/1038207334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5415d9d0 msgr2=0x7f9e540d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.051+0000 7f9e5c800700 1 --2- 172.21.15.12:0/1038207334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5415d9d0 0x7f9e540d3c30 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f9e5408ca10 tx=0x7f9e48009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.051+0000 7f9e5c800700 1 -- 172.21.15.12:0/1038207334 shutdown_connections 2024-03-29T11:34:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.051+0000 7f9e5c800700 1 --2- 172.21.15.12:0/1038207334 >> v2:172.21.15.159:3300/0 conn(0x7f9e540d2540 0x7f9e540ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.051+0000 7f9e5c800700 1 --2- 172.21.15.12:0/1038207334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e5415d9d0 0x7f9e540d3c30 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.051+0000 7f9e5c800700 1 -- 172.21.15.12:0/1038207334 >> 172.21.15.12:0/1038207334 conn(0x7f9e5408aff0 msgr2=0x7f9e540d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.051+0000 7f9e5c800700 1 -- 172.21.15.12:0/1038207334 shutdown_connections 2024-03-29T11:34:11.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.051+0000 7f9e5c800700 1 -- 172.21.15.12:0/1038207334 wait complete. 2024-03-29T11:34:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.053+0000 7f9e5c800700 1 Processor -- start 2024-03-29T11:34:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.053+0000 7f9e5c800700 1 -- start start 2024-03-29T11:34:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e5c800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e540d16a0 0x7f9e541581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e5c800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e540d2540 0x7f9e540b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e5c800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5415d9d0 0x7f9e54155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e5c800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54159a80 con 0x7f9e540d16a0 2024-03-29T11:34:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e5c800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e540b82b0 con 0x7f9e540d2540 2024-03-29T11:34:11.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e5c800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e540b8590 con 0x7f9e5415d9d0 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e59e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e540d16a0 0x7f9e541581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e59e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e540d16a0 0x7f9e541581f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56092/0 (socket says 172.21.15.12:56092) 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e59e00700 1 -- 172.21.15.12:0/3438348599 learned_addr learned my addr 172.21.15.12:0/3438348599 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e59400700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e540d2540 0x7f9e540b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e5a800700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5415d9d0 0x7f9e54155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e59e00700 1 -- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5415d9d0 msgr2=0x7f9e54155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e59e00700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5415d9d0 0x7f9e54155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e59e00700 1 -- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e540d2540 msgr2=0x7f9e540b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e59e00700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e540d2540 0x7f9e540b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.054+0000 7f9e59e00700 1 -- 172.21.15.12:0/3438348599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e48009550 con 0x7f9e540d16a0 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.055+0000 7f9e59e00700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e540d16a0 0x7f9e541581f0 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f9e5408c230 tx=0x7f9e48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.055+0000 7f9e4e000700 1 -- 172.21.15.12:0/3438348599 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48011630 con 0x7f9e540d16a0 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.055+0000 7f9e4e000700 1 -- 172.21.15.12:0/3438348599 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e48011790 con 0x7f9e540d16a0 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.055+0000 7f9e4e000700 1 -- 172.21.15.12:0/3438348599 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48011b40 con 0x7f9e540d16a0 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.055+0000 7f9e5c800700 1 -- 172.21.15.12:0/3438348599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e540b8870 con 0x7f9e540d16a0 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.055+0000 7f9e5c800700 1 -- 172.21.15.12:0/3438348599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e540b7fc0 con 0x7f9e540d16a0 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.056+0000 7f9e5c800700 1 -- 172.21.15.12:0/3438348599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e540d1ca0 con 0x7f9e540d16a0 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.057+0000 7f9e4e000700 1 -- 172.21.15.12:0/3438348599 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9e48011ca0 con 0x7f9e540d16a0 2024-03-29T11:34:11.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.058+0000 7f9e4e000700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e3806adb0 0x7f9e3806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.058+0000 7f9e4e000700 1 -- 172.21.15.12:0/3438348599 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f9e4808ac20 con 0x7f9e540d16a0 2024-03-29T11:34:11.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.059+0000 7f9e59400700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e3806adb0 0x7f9e3806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.059+0000 7f9e59400700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e3806adb0 0x7f9e3806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9e5415f920 tx=0x7f9e44009040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:11.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.063+0000 7f9e4e000700 1 -- 172.21.15.12:0/3438348599 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e48055cb0 con 0x7f9e540d16a0 2024-03-29T11:34:11.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.277+0000 7f9e5c800700 1 -- 172.21.15.12:0/3438348599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9e54067720 con 0x7f9e540d16a0 2024-03-29T11:34:11.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.281+0000 7f9e4e000700 1 -- 172.21.15.12:0/3438348599 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v92) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9e480592d0 con 0x7f9e540d16a0 2024-03-29T11:34:11.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 -- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e3806adb0 msgr2=0x7f9e3806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e3806adb0 0x7f9e3806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9e5415f920 tx=0x7f9e44009040 comp rx=0 tx=0).stop 2024-03-29T11:34:11.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 -- 172.21.15.12:0/3438348599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e540d16a0 msgr2=0x7f9e541581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e540d16a0 0x7f9e541581f0 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f9e5408c230 tx=0x7f9e48004750 comp rx=0 tx=0).stop 2024-03-29T11:34:11.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 -- 172.21.15.12:0/3438348599 shutdown_connections 2024-03-29T11:34:11.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e3806adb0 0x7f9e3806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e5415d9d0 0x7f9e54155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e540d16a0 0x7f9e541581f0 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 --2- 172.21.15.12:0/3438348599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e540d2540 0x7f9e540b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 -- 172.21.15.12:0/3438348599 >> 172.21.15.12:0/3438348599 conn(0x7f9e5408aff0 msgr2=0x7f9e540b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:11.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 -- 172.21.15.12:0/3438348599 shutdown_connections 2024-03-29T11:34:11.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.284+0000 7f9e5c800700 1 -- 172.21.15.12:0/3438348599 wait complete. 2024-03-29T11:34:11.299 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T11:34:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.505+0000 7f50f9c00700 1 Processor -- start 2024-03-29T11:34:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.505+0000 7f50f9c00700 1 -- start start 2024-03-29T11:34:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.506+0000 7f50f9c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f50f40ced80 0x7f50f40d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.506+0000 7f50f9c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50f415fcb0 0x7f50f4160120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.506+0000 7f50f9c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f415f820 con 0x7f50f415dad0 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.506+0000 7f50f9c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f40cfcc0 con 0x7f50f40ced80 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.506+0000 7f50f9c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f40cd9f0 con 0x7f50f415fcb0 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.506+0000 7f50f2000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f50f40ced80 0x7f50f40d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.506+0000 7f50f2a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f50f415dad0 0x7f50f40d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53916/0 (socket says 172.21.15.12:53916) 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.506+0000 7f50f2000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f50f40ced80 0x7f50f40d1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56100/0 (socket says 172.21.15.12:56100) 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.506+0000 7f50f2a00700 1 -- 172.21.15.12:0/2360965788 learned_addr learned my addr 172.21.15.12:0/2360965788 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.506+0000 7f50f3400700 1 --2- 172.21.15.12:0/2360965788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50f415fcb0 0x7f50f4160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.507+0000 7f50f2000700 1 -- 172.21.15.12:0/2360965788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50f415fcb0 msgr2=0x7f50f4160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.507+0000 7f50f2000700 1 --2- 172.21.15.12:0/2360965788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50f415fcb0 0x7f50f4160120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.507+0000 7f50f2000700 1 -- 172.21.15.12:0/2360965788 >> v1:172.21.15.12:6789/0 conn(0x7f50f415dad0 legacy=0x7f50f40d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.507+0000 7f50f2000700 1 -- 172.21.15.12:0/2360965788 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f40cf420 con 0x7f50f40ced80 2024-03-29T11:34:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.507+0000 7f50f1600700 1 -- 172.21.15.12:0/2360965788 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626919340 0 0) 0x7f50f415f820 con 0x7f50f415dad0 2024-03-29T11:34:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.507+0000 7f50f2000700 1 --2- 172.21.15.12:0/2360965788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f40ced80 0x7f50f40d1d40 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f50e0009840 tx=0x7f50e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a72177b2718495 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.507+0000 7f50f1600700 1 -- 172.21.15.12:0/2360965788 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e0011830 con 0x7f50f40ced80 2024-03-29T11:34:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.507+0000 7f50f1600700 1 -- 172.21.15.12:0/2360965788 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e0011990 con 0x7f50f40ced80 2024-03-29T11:34:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.507+0000 7f50f1600700 1 -- 172.21.15.12:0/2360965788 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e0011d20 con 0x7f50f40ced80 2024-03-29T11:34:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.508+0000 7f50f9c00700 1 -- 172.21.15.12:0/2360965788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f40ced80 msgr2=0x7f50f40d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.508+0000 7f50f9c00700 1 --2- 172.21.15.12:0/2360965788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f40ced80 0x7f50f40d1d40 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f50e0009840 tx=0x7f50e0009b50 comp rx=0 tx=0).stop 2024-03-29T11:34:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.508+0000 7f50f9c00700 1 -- 172.21.15.12:0/2360965788 shutdown_connections 2024-03-29T11:34:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.508+0000 7f50f9c00700 1 --2- 172.21.15.12:0/2360965788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50f415fcb0 0x7f50f4160120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.508+0000 7f50f9c00700 1 --2- 172.21.15.12:0/2360965788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f40ced80 0x7f50f40d1d40 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.508+0000 7f50f9c00700 1 -- 172.21.15.12:0/2360965788 >> 172.21.15.12:0/2360965788 conn(0x7f50f408aff0 msgr2=0x7f50f40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:11.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.508+0000 7f50f9c00700 1 -- 172.21.15.12:0/2360965788 shutdown_connections 2024-03-29T11:34:11.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.508+0000 7f50f9c00700 1 -- 172.21.15.12:0/2360965788 wait complete. 2024-03-29T11:34:11.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.510+0000 7f50f9c00700 1 Processor -- start 2024-03-29T11:34:11.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.511+0000 7f50f9c00700 1 -- start start 2024-03-29T11:34:11.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f9c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f40ced80 0x7f50f41f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f415dad0 0x7f50f40cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f9c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50f415fcb0 0x7f50f40cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f9c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f41471b0 con 0x7f50f415dad0 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f9c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f4147490 con 0x7f50f40ced80 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f9c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f4147770 con 0x7f50f415fcb0 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f415dad0 0x7f50f40cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f415dad0 0x7f50f40cbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56116/0 (socket says 172.21.15.12:56116) 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f2000700 1 -- 172.21.15.12:0/1382879096 learned_addr learned my addr 172.21.15.12:0/1382879096 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f2a00700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f40ced80 0x7f50f41f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.512+0000 7f50f3400700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50f415fcb0 0x7f50f40cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50f2000700 1 -- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50f415fcb0 msgr2=0x7f50f40cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50f2000700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50f415fcb0 0x7f50f40cc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50f2000700 1 -- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f40ced80 msgr2=0x7f50f41f2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50f2000700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f40ced80 0x7f50f41f2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50f2000700 1 -- 172.21.15.12:0/1382879096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e00094a0 con 0x7f50f415dad0 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50f2000700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f415dad0 0x7f50f40cbd10 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f50e0009b90 tx=0x7f50e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50eaa00700 1 -- 172.21.15.12:0/1382879096 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e0004550 con 0x7f50f415dad0 2024-03-29T11:34:11.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50eaa00700 1 -- 172.21.15.12:0/1382879096 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e001f070 con 0x7f50f415dad0 2024-03-29T11:34:11.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50eaa00700 1 -- 172.21.15.12:0/1382879096 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e00038e0 con 0x7f50f415dad0 2024-03-29T11:34:11.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50f9c00700 1 -- 172.21.15.12:0/1382879096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f4147a50 con 0x7f50f415dad0 2024-03-29T11:34:11.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.513+0000 7f50f9c00700 1 -- 172.21.15.12:0/1382879096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f41f31f0 con 0x7f50f415dad0 2024-03-29T11:34:11.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.514+0000 7f50f9c00700 1 -- 172.21.15.12:0/1382879096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f4147f70 con 0x7f50f415dad0 2024-03-29T11:34:11.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.515+0000 7f50eaa00700 1 -- 172.21.15.12:0/1382879096 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f50e0003a40 con 0x7f50f415dad0 2024-03-29T11:34:11.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.516+0000 7f50eaa00700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f50d406adb0 0x7f50d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.516+0000 7f50eaa00700 1 -- 172.21.15.12:0/1382879096 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f50e0021070 con 0x7f50f415dad0 2024-03-29T11:34:11.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.516+0000 7f50f2a00700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f50d406adb0 0x7f50d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.517+0000 7f50f2a00700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f50d406adb0 0x7f50d406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f50e4007fd0 tx=0x7f50e4007de0 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:11.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.521+0000 7f50eaa00700 1 -- 172.21.15.12:0/1382879096 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f50f4147f70 con 0x7f50f415dad0 2024-03-29T11:34:11.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.737+0000 7f50f9c00700 1 -- 172.21.15.12:0/1382879096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f50f4147f70 con 0x7f50f415dad0 2024-03-29T11:34:11.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.742+0000 7f50eaa00700 1 -- 172.21.15.12:0/1382879096 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v93) v1 ==== 120+0+0 (secure 0 0 0) 0x7f50f4147f70 con 0x7f50f415dad0 2024-03-29T11:34:11.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 -- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f50d406adb0 msgr2=0x7f50d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f50d406adb0 0x7f50d406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f50e4007fd0 tx=0x7f50e4007de0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 -- 172.21.15.12:0/1382879096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f415dad0 msgr2=0x7f50f40cbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f415dad0 0x7f50f40cbd10 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f50e0009b90 tx=0x7f50e0004750 comp rx=0 tx=0).stop 2024-03-29T11:34:11.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 -- 172.21.15.12:0/1382879096 shutdown_connections 2024-03-29T11:34:11.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f50d406adb0 0x7f50d406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50f415fcb0 0x7f50f40cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50f40ced80 0x7f50f41f2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 --2- 172.21.15.12:0/1382879096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50f415dad0 0x7f50f40cbd10 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 -- 172.21.15.12:0/1382879096 >> 172.21.15.12:0/1382879096 conn(0x7f50f408aff0 msgr2=0x7f50f415cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 -- 172.21.15.12:0/1382879096 shutdown_connections 2024-03-29T11:34:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.745+0000 7f50f9c00700 1 -- 172.21.15.12:0/1382879096 wait complete. 2024-03-29T11:34:11.760 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-03-29T11:34:11.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.961+0000 7f86dc400700 1 Processor -- start 2024-03-29T11:34:11.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.961+0000 7f86dc400700 1 -- start start 2024-03-29T11:34:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.962+0000 7f86dc400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f86d4160be0 0x7f86d40cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.962+0000 7f86dc400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f86d40ce0e0 0x7f86d40ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86dc400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d4160fc0 con 0x7f86d40cead0 2024-03-29T11:34:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86dc400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d40b65d0 con 0x7f86d40ce0e0 2024-03-29T11:34:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86dc400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d40d2a80 con 0x7f86d4160be0 2024-03-29T11:34:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86d9000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f86d40ce0e0 0x7f86d40ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86d9000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f86d40ce0e0 0x7f86d40ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56132/0 (socket says 172.21.15.12:56132) 2024-03-29T11:34:11.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86d9000700 1 -- 172.21.15.12:0/3738588152 learned_addr learned my addr 172.21.15.12:0/3738588152 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:11.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86d9a00700 1 --2- 172.21.15.12:0/3738588152 >> v2:172.21.15.159:3300/0 conn(0x7f86d4160be0 0x7f86d40cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86d9000700 1 -- 172.21.15.12:0/3738588152 >> v2:172.21.15.159:3300/0 conn(0x7f86d4160be0 msgr2=0x7f86d40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:11.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86d9000700 1 --2- 172.21.15.12:0/3738588152 >> v2:172.21.15.159:3300/0 conn(0x7f86d4160be0 0x7f86d40cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86d9000700 1 -- 172.21.15.12:0/3738588152 >> v1:172.21.15.159:6789/0 conn(0x7f86d40cead0 legacy=0x7f86d4161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.963+0000 7f86d9000700 1 -- 172.21.15.12:0/3738588152 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d40d21e0 con 0x7f86d40ce0e0 2024-03-29T11:34:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.964+0000 7f86d9000700 1 --2- 172.21.15.12:0/3738588152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40ce0e0 0x7f86d40ce590 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f86d408b590 tx=0x7f86c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad82db0c1e1e713 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.964+0000 7f86d3e00700 1 -- 172.21.15.12:0/3738588152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c000b870 con 0x7f86d40ce0e0 2024-03-29T11:34:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.964+0000 7f86d3e00700 1 -- 172.21.15.12:0/3738588152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c000b9d0 con 0x7f86d40ce0e0 2024-03-29T11:34:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.964+0000 7f86d3e00700 1 -- 172.21.15.12:0/3738588152 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c000bd60 con 0x7f86d40ce0e0 2024-03-29T11:34:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.964+0000 7f86dc400700 1 -- 172.21.15.12:0/3738588152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40ce0e0 msgr2=0x7f86d40ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.964+0000 7f86dc400700 1 --2- 172.21.15.12:0/3738588152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40ce0e0 0x7f86d40ce590 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f86d408b590 tx=0x7f86c0009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.965+0000 7f86dc400700 1 -- 172.21.15.12:0/3738588152 shutdown_connections 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.965+0000 7f86dc400700 1 --2- 172.21.15.12:0/3738588152 >> v2:172.21.15.159:3300/0 conn(0x7f86d4160be0 0x7f86d40cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.965+0000 7f86dc400700 1 --2- 172.21.15.12:0/3738588152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40ce0e0 0x7f86d40ce590 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.965+0000 7f86dc400700 1 -- 172.21.15.12:0/3738588152 >> 172.21.15.12:0/3738588152 conn(0x7f86d408aff0 msgr2=0x7f86d40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.965+0000 7f86dc400700 1 -- 172.21.15.12:0/3738588152 shutdown_connections 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.965+0000 7f86dc400700 1 -- 172.21.15.12:0/3738588152 wait complete. 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.966+0000 7f86dc400700 1 Processor -- start 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.966+0000 7f86dc400700 1 -- start start 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.967+0000 7f86dc400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86d40ce0e0 0x7f86d415c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.967+0000 7f86dc400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40cead0 0x7f86d415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.967+0000 7f86dc400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86d4160be0 0x7f86d41f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.967+0000 7f86dc400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d415f9a0 con 0x7f86d40cead0 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.967+0000 7f86dc400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d415ed90 con 0x7f86d4160be0 2024-03-29T11:34:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.967+0000 7f86dc400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d415f070 con 0x7f86d40ce0e0 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86d9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40cead0 0x7f86d415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86d9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40cead0 0x7f86d415ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56142/0 (socket says 172.21.15.12:56142) 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86d9000700 1 -- 172.21.15.12:0/2317628769 learned_addr learned my addr 172.21.15.12:0/2317628769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86d9a00700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86d40ce0e0 0x7f86d415c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86da400700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86d4160be0 0x7f86d41f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86d9000700 1 -- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86d40ce0e0 msgr2=0x7f86d415c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86d9000700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86d40ce0e0 0x7f86d415c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86d9000700 1 -- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86d4160be0 msgr2=0x7f86d41f6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86d9000700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86d4160be0 0x7f86d41f6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86d9000700 1 -- 172.21.15.12:0/2317628769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c0009550 con 0x7f86d40cead0 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.968+0000 7f86d9000700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40cead0 0x7f86d415ccb0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f86c0000c00 tx=0x7f86c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.969+0000 7f86d1600700 1 -- 172.21.15.12:0/2317628769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0004540 con 0x7f86d40cead0 2024-03-29T11:34:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.969+0000 7f86d1600700 1 -- 172.21.15.12:0/2317628769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c00036a0 con 0x7f86d40cead0 2024-03-29T11:34:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.969+0000 7f86d1600700 1 -- 172.21.15.12:0/2317628769 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0003a30 con 0x7f86d40cead0 2024-03-29T11:34:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.969+0000 7f86dc400700 1 -- 172.21.15.12:0/2317628769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d415f350 con 0x7f86d40cead0 2024-03-29T11:34:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.969+0000 7f86dc400700 1 -- 172.21.15.12:0/2317628769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d414b860 con 0x7f86d40cead0 2024-03-29T11:34:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.970+0000 7f86dc400700 1 -- 172.21.15.12:0/2317628769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d4067720 con 0x7f86d40cead0 2024-03-29T11:34:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.971+0000 7f86d1600700 1 -- 172.21.15.12:0/2317628769 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f86c0021070 con 0x7f86d40cead0 2024-03-29T11:34:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.971+0000 7f86d1600700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f86b406adb0 0x7f86b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.972+0000 7f86d1600700 1 -- 172.21.15.12:0/2317628769 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f86c008ac50 con 0x7f86d40cead0 2024-03-29T11:34:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.972+0000 7f86d9a00700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f86b406adb0 0x7f86b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.973+0000 7f86d9a00700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f86b406adb0 0x7f86b406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b9b0 tx=0x7f86c8008040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:11.976+0000 7f86d1600700 1 -- 172.21.15.12:0/2317628769 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f86c0090020 con 0x7f86d40cead0 2024-03-29T11:34:12.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.285+0000 7f86dc400700 1 -- 172.21.15.12:0/2317628769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f86d4158fb0 con 0x7f86d40cead0 2024-03-29T11:34:12.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.360+0000 7f86d1600700 1 -- 172.21.15.12:0/2317628769 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3294) v1 ==== 155+0+0 (secure 0 0 0) 0x7f86c00290e0 con 0x7f86d40cead0 2024-03-29T11:34:12.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 -- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f86b406adb0 msgr2=0x7f86b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:12.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f86b406adb0 0x7f86b406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b9b0 tx=0x7f86c8008040 comp rx=0 tx=0).stop 2024-03-29T11:34:12.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 -- 172.21.15.12:0/2317628769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40cead0 msgr2=0x7f86d415ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:12.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40cead0 0x7f86d415ccb0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f86c0000c00 tx=0x7f86c0004750 comp rx=0 tx=0).stop 2024-03-29T11:34:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 -- 172.21.15.12:0/2317628769 shutdown_connections 2024-03-29T11:34:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f86b406adb0 0x7f86b406d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86d4160be0 0x7f86d41f6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86d40ce0e0 0x7f86d415c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 --2- 172.21.15.12:0/2317628769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d40cead0 0x7f86d415ccb0 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 -- 172.21.15.12:0/2317628769 >> 172.21.15.12:0/2317628769 conn(0x7f86d408aff0 msgr2=0x7f86d40b7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 -- 172.21.15.12:0/2317628769 shutdown_connections 2024-03-29T11:34:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.363+0000 7f86dc400700 1 -- 172.21.15.12:0/2317628769 wait complete. 2024-03-29T11:34:12.380 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2024-03-29T11:34:12.380 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) Test subvol schedule creation succeeds for default subvolgroup. 2024-03-29T11:34:12.380 INFO:tasks.cephfs_test_runner:test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-29T11:34:12.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-03-29T11:34:12.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.581+0000 7fb57e200700 1 Processor -- start 2024-03-29T11:34:12.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.582+0000 7fb57e200700 1 -- start start 2024-03-29T11:34:12.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb57e200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb57815d640 0x7fb5780d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:12.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb57e200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb5780ceec0 0x7fb5781612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:12.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb57e200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb578161800 0x7fb5780cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:12.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb57e200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb578161c70 con 0x7fb57815d640 2024-03-29T11:34:12.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb57e200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5780d30d0 con 0x7fb5780ceec0 2024-03-29T11:34:12.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb57e200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780b6840 con 0x7fb578161800 2024-03-29T11:34:12.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb577400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb57815d640 0x7fb5780d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:12.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb577400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb57815d640 0x7fb5780d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56148/0 (socket says 172.21.15.12:56148) 2024-03-29T11:34:12.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb577400700 1 -- 172.21.15.12:0/4237738294 learned_addr learned my addr 172.21.15.12:0/4237738294 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:12.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb576a00700 1 --2- 172.21.15.12:0/4237738294 >> v2:172.21.15.159:3300/0 conn(0x7fb5780ceec0 0x7fb5781612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:12.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.583+0000 7fb577e00700 1 --2- 172.21.15.12:0/4237738294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb578161800 0x7fb5780cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:12.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.584+0000 7fb577400700 1 -- 172.21.15.12:0/4237738294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb578161800 msgr2=0x7fb5780cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:12.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.584+0000 7fb577400700 1 --2- 172.21.15.12:0/4237738294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb578161800 0x7fb5780cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.584+0000 7fb577400700 1 -- 172.21.15.12:0/4237738294 >> v2:172.21.15.159:3300/0 conn(0x7fb5780ceec0 msgr2=0x7fb5781612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:12.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.584+0000 7fb577400700 1 --2- 172.21.15.12:0/4237738294 >> v2:172.21.15.159:3300/0 conn(0x7fb5780ceec0 0x7fb5781612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.584+0000 7fb577400700 1 -- 172.21.15.12:0/4237738294 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5780d2880 con 0x7fb57815d640 2024-03-29T11:34:12.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.584+0000 7fb576a00700 1 --2- 172.21.15.12:0/4237738294 >> v2:172.21.15.159:3300/0 conn(0x7fb5780ceec0 0x7fb5781612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:12.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.584+0000 7fb577400700 1 --2- 172.21.15.12:0/4237738294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb57815d640 0x7fb5780d2340 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7fb5780cfbf0 tx=0x7fb568009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912fe6aa7ae44d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:12.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.584+0000 7fb576000700 1 -- 172.21.15.12:0/4237738294 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56800b870 con 0x7fb57815d640 2024-03-29T11:34:12.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.584+0000 7fb576000700 1 -- 172.21.15.12:0/4237738294 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56800b9d0 con 0x7fb57815d640 2024-03-29T11:34:12.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.584+0000 7fb576000700 1 -- 172.21.15.12:0/4237738294 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56800bd60 con 0x7fb57815d640 2024-03-29T11:34:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.585+0000 7fb57e200700 1 -- 172.21.15.12:0/4237738294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb57815d640 msgr2=0x7fb5780d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.585+0000 7fb57e200700 1 --2- 172.21.15.12:0/4237738294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb57815d640 0x7fb5780d2340 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7fb5780cfbf0 tx=0x7fb568009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.585+0000 7fb57e200700 1 -- 172.21.15.12:0/4237738294 shutdown_connections 2024-03-29T11:34:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.585+0000 7fb57e200700 1 --2- 172.21.15.12:0/4237738294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb578161800 0x7fb5780cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.585+0000 7fb57e200700 1 --2- 172.21.15.12:0/4237738294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb57815d640 0x7fb5780d2340 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.585+0000 7fb57e200700 1 --2- 172.21.15.12:0/4237738294 >> v2:172.21.15.159:3300/0 conn(0x7fb5780ceec0 0x7fb5781612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.585+0000 7fb57e200700 1 -- 172.21.15.12:0/4237738294 >> 172.21.15.12:0/4237738294 conn(0x7fb57808aff0 msgr2=0x7fb5780d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.585+0000 7fb57e200700 1 -- 172.21.15.12:0/4237738294 shutdown_connections 2024-03-29T11:34:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.585+0000 7fb57e200700 1 -- 172.21.15.12:0/4237738294 wait complete. 2024-03-29T11:34:12.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.587+0000 7fb57e200700 1 Processor -- start 2024-03-29T11:34:12.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.587+0000 7fb57e200700 1 -- start start 2024-03-29T11:34:12.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb57e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5780ceec0 0x7fb5781f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:12.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb57e200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb57815d640 0x7fb57814c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:12.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb57e200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb578161800 0x7fb5780ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:12.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb57e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780ceb90 con 0x7fb5780ceec0 2024-03-29T11:34:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb57e200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5781f3c70 con 0x7fb578161800 2024-03-29T11:34:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb57e200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780cb840 con 0x7fb57815d640 2024-03-29T11:34:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb577400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5780ceec0 0x7fb5781f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb577400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5780ceec0 0x7fb5781f3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56154/0 (socket says 172.21.15.12:56154) 2024-03-29T11:34:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb577400700 1 -- 172.21.15.12:0/2085760391 learned_addr learned my addr 172.21.15.12:0/2085760391 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb576a00700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb57815d640 0x7fb57814c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb577e00700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb578161800 0x7fb5780ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb577400700 1 -- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb57815d640 msgr2=0x7fb57814c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb577400700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb57815d640 0x7fb57814c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb577400700 1 -- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb578161800 msgr2=0x7fb5780ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb577400700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb578161800 0x7fb5780ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb577400700 1 -- 172.21.15.12:0/2085760391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568009550 con 0x7fb5780ceec0 2024-03-29T11:34:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.588+0000 7fb576a00700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb57815d640 0x7fb57814c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.589+0000 7fb577400700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5780ceec0 0x7fb5781f3730 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7fb568000c00 tx=0x7fb568004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.589+0000 7fb56f400700 1 -- 172.21.15.12:0/2085760391 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568004540 con 0x7fb5780ceec0 2024-03-29T11:34:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.589+0000 7fb56f400700 1 -- 172.21.15.12:0/2085760391 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5680036a0 con 0x7fb5780ceec0 2024-03-29T11:34:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.589+0000 7fb56f400700 1 -- 172.21.15.12:0/2085760391 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568003a50 con 0x7fb5780ceec0 2024-03-29T11:34:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.589+0000 7fb57e200700 1 -- 172.21.15.12:0/2085760391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5780cdaf0 con 0x7fb5780ceec0 2024-03-29T11:34:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.589+0000 7fb57e200700 1 -- 172.21.15.12:0/2085760391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5780ce0b0 con 0x7fb5780ceec0 2024-03-29T11:34:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.590+0000 7fb57e200700 1 -- 172.21.15.12:0/2085760391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb578067720 con 0x7fb5780ceec0 2024-03-29T11:34:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.591+0000 7fb56f400700 1 -- 172.21.15.12:0/2085760391 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb568021070 con 0x7fb5780ceec0 2024-03-29T11:34:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.592+0000 7fb56f400700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb55806adb0 0x7fb55806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.592+0000 7fb56f400700 1 -- 172.21.15.12:0/2085760391 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fb56808ab20 con 0x7fb5780ceec0 2024-03-29T11:34:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.592+0000 7fb576a00700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb55806adb0 0x7fb55806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.593+0000 7fb576a00700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb55806adb0 0x7fb55806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb57815aee0 tx=0x7fb564008040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.596+0000 7fb56f400700 1 -- 172.21.15.12:0/2085760391 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb568055cd0 con 0x7fb5780ceec0 2024-03-29T11:34:12.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:12.895+0000 7fb57e200700 1 -- 172.21.15.12:0/2085760391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) v1 -- 0x7fb5780b5c70 con 0x7fb5780ceec0 2024-03-29T11:34:13.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.366+0000 7fb56f400700 1 -- 172.21.15.12:0/2085760391 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3295) v1 ==== 204+0+0 (secure 0 0 0) 0x7fb568015020 con 0x7fb5780ceec0 2024-03-29T11:34:13.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 -- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb55806adb0 msgr2=0x7fb55806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb55806adb0 0x7fb55806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb57815aee0 tx=0x7fb564008040 comp rx=0 tx=0).stop 2024-03-29T11:34:13.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 -- 172.21.15.12:0/2085760391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5780ceec0 msgr2=0x7fb5781f3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5780ceec0 0x7fb5781f3730 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7fb568000c00 tx=0x7fb568004750 comp rx=0 tx=0).stop 2024-03-29T11:34:13.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 -- 172.21.15.12:0/2085760391 shutdown_connections 2024-03-29T11:34:13.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb55806adb0 0x7fb55806d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb578161800 0x7fb5780ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5780ceec0 0x7fb5781f3730 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 --2- 172.21.15.12:0/2085760391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb57815d640 0x7fb57814c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 -- 172.21.15.12:0/2085760391 >> 172.21.15.12:0/2085760391 conn(0x7fb57808aff0 msgr2=0x7fb57815cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:13.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 -- 172.21.15.12:0/2085760391 shutdown_connections 2024-03-29T11:34:13.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.369+0000 7fb57e200700 1 -- 172.21.15.12:0/2085760391 wait complete. 2024-03-29T11:34:13.386 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-29T11:34:13.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.592+0000 7f2ad7200700 1 Processor -- start 2024-03-29T11:34:13.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.593+0000 7f2ad7200700 1 -- start start 2024-03-29T11:34:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.594+0000 7f2ad7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 0x7f2ad00d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.594+0000 7f2ad7200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad00cf640 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.594+0000 7f2ad7200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad00d0d30 con 0x7f2ad00d1b70 2024-03-29T11:34:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.594+0000 7f2ad7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad00cf220 con 0x7f2ad015d710 2024-03-29T11:34:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.594+0000 7f2ad5200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2ad00d1ff0 0x7f2ad00d0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53922/0 (socket says 172.21.15.12:53922) 2024-03-29T11:34:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.594+0000 7f2ad5200700 1 -- 172.21.15.12:0/3080875819 learned_addr learned my addr 172.21.15.12:0/3080875819 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.594+0000 7f2acfe00700 1 --2- 172.21.15.12:0/3080875819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 0x7f2ad00d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:13.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.595+0000 7f2acea00700 1 -- 172.21.15.12:0/3080875819 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244055350 0 0) 0x7f2ad00cf640 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.595+0000 7f2acea00700 1 -- 172.21.15.12:0/3080875819 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab40035b0 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.595+0000 7f2acea00700 1 -- 172.21.15.12:0/3080875819 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204018062 0 0) 0x7f2ad00d0d30 con 0x7f2ad00d1b70 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.595+0000 7f2acea00700 1 -- 172.21.15.12:0/3080875819 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad00cf640 con 0x7f2ad00d1b70 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.595+0000 7f2acfe00700 1 -- 172.21.15.12:0/3080875819 >> v1:172.21.15.159:6789/0 conn(0x7f2ad00d1b70 legacy=0x7f2ad00cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.595+0000 7f2acfe00700 1 -- 172.21.15.12:0/3080875819 >> v1:172.21.15.12:6789/0 conn(0x7f2ad00d1ff0 legacy=0x7f2ad00d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.595+0000 7f2acea00700 1 -- 172.21.15.12:0/3080875819 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4222335322 0 0) 0x7f2ab40035b0 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.595+0000 7f2acfe00700 1 -- 172.21.15.12:0/3080875819 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad00ce5a0 con 0x7f2ad015d710 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.595+0000 7f2acfe00700 1 --2- 172.21.15.12:0/3080875819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 0x7f2ad00d12f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009870 tx=0x7f2ac0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be6a92d613f50604 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.596+0000 7f2acea00700 1 -- 172.21.15.12:0/3080875819 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac80019f0 con 0x7f2ad015d710 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.596+0000 7f2acea00700 1 -- 172.21.15.12:0/3080875819 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac0004030 con 0x7f2ad015d710 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.596+0000 7f2acea00700 1 -- 172.21.15.12:0/3080875819 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac00043c0 con 0x7f2ad015d710 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.596+0000 7f2ad7200700 1 -- 172.21.15.12:0/3080875819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 msgr2=0x7f2ad00d12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.596+0000 7f2ad7200700 1 --2- 172.21.15.12:0/3080875819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 0x7f2ad00d12f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009870 tx=0x7f2ac0009b80 comp rx=0 tx=0).stop 2024-03-29T11:34:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.596+0000 7f2ad7200700 1 -- 172.21.15.12:0/3080875819 shutdown_connections 2024-03-29T11:34:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.596+0000 7f2ad7200700 1 --2- 172.21.15.12:0/3080875819 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 0x7f2ad00d12f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.596+0000 7f2ad7200700 1 -- 172.21.15.12:0/3080875819 >> 172.21.15.12:0/3080875819 conn(0x7f2ad008aff0 msgr2=0x7f2ad00d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.596+0000 7f2ad7200700 1 -- 172.21.15.12:0/3080875819 shutdown_connections 2024-03-29T11:34:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.596+0000 7f2ad7200700 1 -- 172.21.15.12:0/3080875819 wait complete. 2024-03-29T11:34:13.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.598+0000 7f2ad7200700 1 Processor -- start 2024-03-29T11:34:13.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.598+0000 7f2ad7200700 1 -- start start 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2ad7200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ad00d1b70 0x7f2ad01f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2ad7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad00d1ff0 0x7f2ad00cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2ad7200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 0x7f2ad00cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2ad7200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad0158320 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2ad7200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad00ca790 con 0x7f2ad00d1b70 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2ad7200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad00c9950 con 0x7f2ad015d710 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2acf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad00d1ff0 0x7f2ad00cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2acf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad00d1ff0 0x7f2ad00cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56168/0 (socket says 172.21.15.12:56168) 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2acf400700 1 -- 172.21.15.12:0/745544543 learned_addr learned my addr 172.21.15.12:0/745544543 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2acfe00700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ad00d1b70 0x7f2ad01f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.599+0000 7f2ad5200700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 0x7f2ad00cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2acf400700 1 -- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 msgr2=0x7f2ad00cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2acf400700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 0x7f2ad00cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2acf400700 1 -- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ad00d1b70 msgr2=0x7f2ad01f2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2acf400700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ad00d1b70 0x7f2ad01f2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2acf400700 1 -- 172.21.15.12:0/745544543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac00094a0 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2acf400700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad00d1ff0 0x7f2ad00cd010 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f2abc001f00 tx=0x7f2abc00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2ac7e00700 1 -- 172.21.15.12:0/745544543 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc0052d0 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2ac7e00700 1 -- 172.21.15.12:0/745544543 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc005430 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2ad7200700 1 -- 172.21.15.12:0/745544543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad00c9c30 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2ac7e00700 1 -- 172.21.15.12:0/745544543 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc0038d0 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.600+0000 7f2ad7200700 1 -- 172.21.15.12:0/745544543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad00ca1f0 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.601+0000 7f2ad7200700 1 -- 172.21.15.12:0/745544543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad00ca900 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.602+0000 7f2ac7e00700 1 -- 172.21.15.12:0/745544543 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2abc003a30 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.603+0000 7f2ac7e00700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2ab006adb0 0x7f2ab006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:13.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.603+0000 7f2ac7e00700 1 -- 172.21.15.12:0/745544543 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f2abc088920 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.603+0000 7f2acfe00700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2ab006adb0 0x7f2ab006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:13.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.604+0000 7f2acfe00700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2ab006adb0 0x7f2ab006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00097c0 tx=0x7f2ac0008030 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:13.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.607+0000 7f2ac7e00700 1 -- 172.21.15.12:0/745544543 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ad00ca900 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.824+0000 7f2ad7200700 1 -- 172.21.15.12:0/745544543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2ad00ca900 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.828+0000 7f2ac7e00700 1 -- 172.21.15.12:0/745544543 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v94)=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2ad00ca900 con 0x7f2ad00d1ff0 2024-03-29T11:34:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 -- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2ab006adb0 msgr2=0x7f2ab006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2ab006adb0 0x7f2ab006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00097c0 tx=0x7f2ac0008030 comp rx=0 tx=0).stop 2024-03-29T11:34:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 -- 172.21.15.12:0/745544543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad00d1ff0 msgr2=0x7f2ad00cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad00d1ff0 0x7f2ad00cd010 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f2abc001f00 tx=0x7f2abc00c7b0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 -- 172.21.15.12:0/745544543 shutdown_connections 2024-03-29T11:34:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2ab006adb0 0x7f2ab006d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad015d710 0x7f2ad00cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ad00d1b70 0x7f2ad01f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 --2- 172.21.15.12:0/745544543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad00d1ff0 0x7f2ad00cd010 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 -- 172.21.15.12:0/745544543 >> 172.21.15.12:0/745544543 conn(0x7f2ad008aff0 msgr2=0x7f2ad00b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 -- 172.21.15.12:0/745544543 shutdown_connections 2024-03-29T11:34:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:13.831+0000 7f2ad7200700 1 -- 172.21.15.12:0/745544543 wait complete. 2024-03-29T11:34:13.846 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:34:13.847 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:34:13.932 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:34:14.024 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:34:14.024 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:34:14.024 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:34:14.069 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:34:14.070 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:34:14.070 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:34:14.116 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:34:14.116 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:34:14.116 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:34:14.160 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:34:14.160 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:34:14.161 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:34:14.206 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:34:14.206 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-29T11:34:14.206 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:34:14.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.408+0000 7f9375400700 1 Processor -- start 2024-03-29T11:34:14.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.408+0000 7f9375400700 1 -- start start 2024-03-29T11:34:14.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.409+0000 7f9375400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f937015da50 0x7f93700ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.409+0000 7f9375400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f93700cec70 0x7f93700d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f9375400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93700d24e0 con 0x7f93700d0b20 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f9375400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93700d0fd0 con 0x7f937015da50 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f9375400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93700b95d0 con 0x7f93700cec70 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f936e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f937015da50 0x7f93700ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f936f200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f93700d0b20 0x7f93700b52e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53928/0 (socket says 172.21.15.12:53928) 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f936f200700 1 -- 172.21.15.12:0/3473748764 learned_addr learned my addr 172.21.15.12:0/3473748764 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f936de00700 1 --2- 172.21.15.12:0/3473748764 >> v2:172.21.15.159:3300/0 conn(0x7f93700cec70 0x7f93700d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f936e800700 1 -- 172.21.15.12:0/3473748764 >> v2:172.21.15.159:3300/0 conn(0x7f93700cec70 msgr2=0x7f93700d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f936e800700 1 --2- 172.21.15.12:0/3473748764 >> v2:172.21.15.159:3300/0 conn(0x7f93700cec70 0x7f93700d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f936e800700 1 -- 172.21.15.12:0/3473748764 >> v1:172.21.15.12:6789/0 conn(0x7f93700d0b20 legacy=0x7f93700b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f936e800700 1 -- 172.21.15.12:0/3473748764 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93700d27c0 con 0x7f937015da50 2024-03-29T11:34:14.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.410+0000 7f936d400700 1 -- 172.21.15.12:0/3473748764 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552452781 0 0) 0x7f93700d24e0 con 0x7f93700d0b20 2024-03-29T11:34:14.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.411+0000 7f936e800700 1 --2- 172.21.15.12:0/3473748764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 0x7f93700ce730 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f93700740a0 tx=0x7f9360009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e363c651cac680 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:14.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.411+0000 7f936d400700 1 -- 172.21.15.12:0/3473748764 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9360011830 con 0x7f937015da50 2024-03-29T11:34:14.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.411+0000 7f936d400700 1 -- 172.21.15.12:0/3473748764 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9360011990 con 0x7f937015da50 2024-03-29T11:34:14.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.411+0000 7f936d400700 1 -- 172.21.15.12:0/3473748764 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9360011d20 con 0x7f937015da50 2024-03-29T11:34:14.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.411+0000 7f9375400700 1 -- 172.21.15.12:0/3473748764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 msgr2=0x7f93700ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.411+0000 7f9375400700 1 --2- 172.21.15.12:0/3473748764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 0x7f93700ce730 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f93700740a0 tx=0x7f9360009b20 comp rx=0 tx=0).stop 2024-03-29T11:34:14.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.411+0000 7f9375400700 1 -- 172.21.15.12:0/3473748764 shutdown_connections 2024-03-29T11:34:14.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.411+0000 7f9375400700 1 --2- 172.21.15.12:0/3473748764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 0x7f93700ce730 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.411+0000 7f9375400700 1 --2- 172.21.15.12:0/3473748764 >> v2:172.21.15.159:3300/0 conn(0x7f93700cec70 0x7f93700d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.412+0000 7f9375400700 1 -- 172.21.15.12:0/3473748764 >> 172.21.15.12:0/3473748764 conn(0x7f93700666a0 msgr2=0x7f9370066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:14.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.412+0000 7f9375400700 1 -- 172.21.15.12:0/3473748764 shutdown_connections 2024-03-29T11:34:14.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.412+0000 7f9375400700 1 -- 172.21.15.12:0/3473748764 wait complete. 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.413+0000 7f9375400700 1 Processor -- start 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.413+0000 7f9375400700 1 -- start start 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.414+0000 7f9375400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93700cec70 0x7f93700c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.414+0000 7f9375400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93700d0b20 0x7f9370147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.414+0000 7f9375400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 0x7f9370147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.414+0000 7f9375400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9370148f30 con 0x7f937015da50 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.414+0000 7f9375400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93701493c0 con 0x7f93700cec70 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.414+0000 7f9375400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9370148320 con 0x7f93700d0b20 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 0x7f9370147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 0x7f9370147aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56184/0 (socket says 172.21.15.12:56184) 2024-03-29T11:34:14.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936f200700 1 -- 172.21.15.12:0/2103577435 learned_addr learned my addr 172.21.15.12:0/2103577435 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936e800700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93700cec70 0x7f93700c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936de00700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93700d0b20 0x7f9370147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936f200700 1 -- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93700d0b20 msgr2=0x7f9370147560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936f200700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93700d0b20 0x7f9370147560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936f200700 1 -- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93700cec70 msgr2=0x7f93700c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936f200700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93700cec70 0x7f93700c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936f200700 1 -- 172.21.15.12:0/2103577435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93600094a0 con 0x7f937015da50 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.415+0000 7f936f200700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 0x7f9370147aa0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f936800cc10 tx=0x7f936800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.416+0000 7f9366a00700 1 -- 172.21.15.12:0/2103577435 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936800a9e0 con 0x7f937015da50 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.416+0000 7f9366a00700 1 -- 172.21.15.12:0/2103577435 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936800ab40 con 0x7f937015da50 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.416+0000 7f9375400700 1 -- 172.21.15.12:0/2103577435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9370148600 con 0x7f937015da50 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.416+0000 7f9366a00700 1 -- 172.21.15.12:0/2103577435 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93680070d0 con 0x7f937015da50 2024-03-29T11:34:14.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.416+0000 7f9375400700 1 -- 172.21.15.12:0/2103577435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9370148bc0 con 0x7f937015da50 2024-03-29T11:34:14.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.417+0000 7f9375400700 1 -- 172.21.15.12:0/2103577435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9370067720 con 0x7f937015da50 2024-03-29T11:34:14.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.418+0000 7f9366a00700 1 -- 172.21.15.12:0/2103577435 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9368007230 con 0x7f937015da50 2024-03-29T11:34:14.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.419+0000 7f9366a00700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f935006adb0 0x7f935006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.419+0000 7f9366a00700 1 -- 172.21.15.12:0/2103577435 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f9368088ed0 con 0x7f937015da50 2024-03-29T11:34:14.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.419+0000 7f936e800700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f935006adb0 0x7f935006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.420+0000 7f936e800700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f935006adb0 0x7f935006d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9360000c00 tx=0x7f936001c040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:14.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.423+0000 7f9366a00700 1 -- 172.21.15.12:0/2103577435 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9368053f70 con 0x7f937015da50 2024-03-29T11:34:14.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.686+0000 7f9375400700 1 -- 172.21.15.12:0/2103577435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f937015a7b0 con 0x7f937015da50 2024-03-29T11:34:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.687+0000 7f9366a00700 1 -- 172.21.15.12:0/2103577435 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+3541 (secure 0 0 0) 0x7f9368018070 con 0x7f937015da50 2024-03-29T11:34:14.684 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:14.684 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":140,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10304,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1546500295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1546500295},{"type":"v1","addr":"172.21.15.159:6836","nonce":1546500295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1627782307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1627782307},{"type":"v1","addr":"172.21.15.12:6834","nonce":1627782307}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3327995506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3327995506},{"type":"v1","addr":"172.21.15.12:6835","nonce":3327995506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/822898513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":822898513},{"type":"v1","addr":"172.21.15.159:6837","nonce":822898513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[]} 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.689+0000 7f9375400700 1 -- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f935006adb0 msgr2=0x7f935006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.689+0000 7f9375400700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f935006adb0 0x7f935006d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9360000c00 tx=0x7f936001c040 comp rx=0 tx=0).stop 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.689+0000 7f9375400700 1 -- 172.21.15.12:0/2103577435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 msgr2=0x7f9370147aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.689+0000 7f9375400700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 0x7f9370147aa0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f936800cc10 tx=0x7f936800cf20 comp rx=0 tx=0).stop 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.689+0000 7f9375400700 1 -- 172.21.15.12:0/2103577435 shutdown_connections 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.689+0000 7f9375400700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f935006adb0 0x7f935006d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.689+0000 7f9375400700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f937015da50 0x7f9370147aa0 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.689+0000 7f9375400700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93700cec70 0x7f93700c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.689+0000 7f9375400700 1 --2- 172.21.15.12:0/2103577435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93700d0b20 0x7f9370147560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.690+0000 7f9375400700 1 -- 172.21.15.12:0/2103577435 >> 172.21.15.12:0/2103577435 conn(0x7f93700666a0 msgr2=0x7f93700b32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.690+0000 7f9375400700 1 -- 172.21.15.12:0/2103577435 shutdown_connections 2024-03-29T11:34:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.690+0000 7f9375400700 1 -- 172.21.15.12:0/2103577435 wait complete. 2024-03-29T11:34:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 140 2024-03-29T11:34:14.705 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-29T11:34:14.705 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-29T11:34:14.705 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T11:34:14.706 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:14.706 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-29T11:34:14.706 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-29T11:34:14.706 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T11:34:14.706 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:14.706 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-29T11:34:14.706 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-29T11:34:14.707 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T11:34:14.707 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:14.707 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-29T11:34:14.707 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-29T11:34:14.707 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T11:34:14.707 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:14.719 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T11:34:14.719 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-29T11:34:14.720 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T11:34:14.720 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-29T11:34:14.724 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T11:34:14.724 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-29T11:34:14.725 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T11:34:14.725 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-29T11:34:14.764 INFO:tasks.ceph.mds.a:Started 2024-03-29T11:34:14.765 DEBUG:teuthology.parallel:result is None 2024-03-29T11:34:14.768 INFO:tasks.ceph.mds.c:Started 2024-03-29T11:34:14.768 DEBUG:teuthology.parallel:result is None 2024-03-29T11:34:14.768 INFO:tasks.ceph.mds.d:Started 2024-03-29T11:34:14.769 DEBUG:teuthology.parallel:result is None 2024-03-29T11:34:14.771 INFO:tasks.ceph.mds.b:Started 2024-03-29T11:34:14.771 DEBUG:teuthology.parallel:result is None 2024-03-29T11:34:14.771 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-29T11:34:14.912 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-29T11:34:14.912 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-29T11:34:14.916 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:34:14.916 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-03-29T11:34:14.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.985+0000 7fb046200700 1 Processor -- start 2024-03-29T11:34:14.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.986+0000 7fb046200700 1 -- start start 2024-03-29T11:34:14.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.987+0000 7fb046200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb04015dfe0 0x7fb0400ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.987+0000 7fb046200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0400cb310 con 0x7fb04015a5c0 2024-03-29T11:34:14.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.987+0000 7fb046200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0400711a0 con 0x7fb0400ce670 2024-03-29T11:34:14.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.987+0000 7fb046200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0400ceb00 con 0x7fb04015dfe0 2024-03-29T11:34:14.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.987+0000 7fb03ea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb04015dfe0 0x7fb0400ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.987+0000 7fb03f400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb04015a5c0 0x7fb0400709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53936/0 (socket says 172.21.15.12:53936) 2024-03-29T11:34:14.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.987+0000 7fb03f400700 1 -- 172.21.15.12:0/1068188645 learned_addr learned my addr 172.21.15.12:0/1068188645 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.987+0000 7fb03ea00700 1 -- 172.21.15.12:0/1068188645 >> v1:172.21.15.159:6789/0 conn(0x7fb0400ce670 legacy=0x7fb0400cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.987+0000 7fb03ea00700 1 -- 172.21.15.12:0/1068188645 >> v1:172.21.15.12:6789/0 conn(0x7fb04015a5c0 legacy=0x7fb0400709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.987+0000 7fb03ea00700 1 -- 172.21.15.12:0/1068188645 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb040071480 con 0x7fb04015dfe0 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.988+0000 7fb03ea00700 1 --2- 172.21.15.12:0/1068188645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015dfe0 0x7fb0400ce130 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009840 tx=0x7fb02c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606717201ea3ba61 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.988+0000 7fb03e000700 1 -- 172.21.15.12:0/1068188645 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c004230 con 0x7fb04015dfe0 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.988+0000 7fb03e000700 1 -- 172.21.15.12:0/1068188645 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02c004390 con 0x7fb04015dfe0 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.988+0000 7fb03e000700 1 -- 172.21.15.12:0/1068188645 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0036a0 con 0x7fb04015dfe0 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.988+0000 7fb046200700 1 -- 172.21.15.12:0/1068188645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015dfe0 msgr2=0x7fb0400ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.988+0000 7fb046200700 1 --2- 172.21.15.12:0/1068188645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015dfe0 0x7fb0400ce130 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009840 tx=0x7fb02c009b50 comp rx=0 tx=0).stop 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.988+0000 7fb046200700 1 -- 172.21.15.12:0/1068188645 shutdown_connections 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.988+0000 7fb046200700 1 --2- 172.21.15.12:0/1068188645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015dfe0 0x7fb0400ce130 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.988+0000 7fb046200700 1 -- 172.21.15.12:0/1068188645 >> 172.21.15.12:0/1068188645 conn(0x7fb0400b0d70 msgr2=0x7fb0400b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.989+0000 7fb046200700 1 -- 172.21.15.12:0/1068188645 shutdown_connections 2024-03-29T11:34:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.989+0000 7fb046200700 1 -- 172.21.15.12:0/1068188645 wait complete. 2024-03-29T11:34:14.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.991+0000 7fb046200700 1 Processor -- start 2024-03-29T11:34:14.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.991+0000 7fb046200700 1 -- start start 2024-03-29T11:34:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.992+0000 7fb046200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0400ce670 0x7fb040157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.992+0000 7fb046200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015a5c0 0x7fb0401f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.992+0000 7fb046200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04015dfe0 0x7fb0401594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.992+0000 7fb046200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb040158420 con 0x7fb04015a5c0 2024-03-29T11:34:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.992+0000 7fb046200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb040158700 con 0x7fb04015dfe0 2024-03-29T11:34:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.992+0000 7fb046200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0401589e0 con 0x7fb0400ce670 2024-03-29T11:34:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015a5c0 0x7fb0401f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03f400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0400ce670 0x7fb040157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015a5c0 0x7fb0401f2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56240/0 (socket says 172.21.15.12:56240) 2024-03-29T11:34:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03ea00700 1 -- 172.21.15.12:0/4077923510 learned_addr learned my addr 172.21.15.12:0/4077923510 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03fe00700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04015dfe0 0x7fb0401594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03ea00700 1 -- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0400ce670 msgr2=0x7fb040157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03ea00700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0400ce670 0x7fb040157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03ea00700 1 -- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04015dfe0 msgr2=0x7fb0401594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03ea00700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04015dfe0 0x7fb0401594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03ea00700 1 -- 172.21.15.12:0/4077923510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c0094a0 con 0x7fb04015a5c0 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.993+0000 7fb03ea00700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015a5c0 0x7fb0401f2960 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0090b0 tx=0x7fb02c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.994+0000 7fb037400700 1 -- 172.21.15.12:0/4077923510 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c00f8a0 con 0x7fb04015a5c0 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.994+0000 7fb037400700 1 -- 172.21.15.12:0/4077923510 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02c00fa00 con 0x7fb04015a5c0 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.994+0000 7fb037400700 1 -- 172.21.15.12:0/4077923510 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c00fdb0 con 0x7fb04015a5c0 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.994+0000 7fb046200700 1 -- 172.21.15.12:0/4077923510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb040158cc0 con 0x7fb04015a5c0 2024-03-29T11:34:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.994+0000 7fb046200700 1 -- 172.21.15.12:0/4077923510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0401580f0 con 0x7fb04015a5c0 2024-03-29T11:34:14.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.995+0000 7fb046200700 1 -- 172.21.15.12:0/4077923510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0401591e0 con 0x7fb04015a5c0 2024-03-29T11:34:14.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.996+0000 7fb037400700 1 -- 172.21.15.12:0/4077923510 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb02c0243f0 con 0x7fb04015a5c0 2024-03-29T11:34:14.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.997+0000 7fb037400700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02006adb0 0x7fb02006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:14.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.997+0000 7fb037400700 1 -- 172.21.15.12:0/4077923510 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fb02c08abf0 con 0x7fb04015a5c0 2024-03-29T11:34:14.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.997+0000 7fb03f400700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02006adb0 0x7fb02006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:14.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:14.998+0000 7fb03f400700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02006adb0 0x7fb02006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb030000f60 tx=0x7fb030007f10 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:14.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.001+0000 7fb037400700 1 -- 172.21.15.12:0/4077923510 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0401591e0 con 0x7fb04015a5c0 2024-03-29T11:34:15.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.233+0000 7fb046200700 1 -- 172.21.15.12:0/4077923510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb0401591e0 con 0x7fb04015a5c0 2024-03-29T11:34:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.959+0000 7fb037400700 1 -- 172.21.15.12:0/4077923510 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v279) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb0401591e0 con 0x7fb04015a5c0 2024-03-29T11:34:15.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 -- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02006adb0 msgr2=0x7fb02006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02006adb0 0x7fb02006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb030000f60 tx=0x7fb030007f10 comp rx=0 tx=0).stop 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 -- 172.21.15.12:0/4077923510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015a5c0 msgr2=0x7fb0401f2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015a5c0 0x7fb0401f2960 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0090b0 tx=0x7fb02c004750 comp rx=0 tx=0).stop 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 -- 172.21.15.12:0/4077923510 shutdown_connections 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb02006adb0 0x7fb02006d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb04015dfe0 0x7fb0401594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0400ce670 0x7fb040157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 --2- 172.21.15.12:0/4077923510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb04015a5c0 0x7fb0401f2960 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 -- 172.21.15.12:0/4077923510 >> 172.21.15.12:0/4077923510 conn(0x7fb0400b0d70 msgr2=0x7fb0400cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 -- 172.21.15.12:0/4077923510 shutdown_connections 2024-03-29T11:34:15.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:15.962+0000 7fb046200700 1 -- 172.21.15.12:0/4077923510 wait complete. 2024-03-29T11:34:15.961 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-03-29T11:34:15.981 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-29T11:34:16.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.184+0000 7f7f41800700 1 Processor -- start 2024-03-29T11:34:16.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.185+0000 7f7f41800700 1 -- start start 2024-03-29T11:34:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f41800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f3c0ceec0 0x7f7f3c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f41800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c161800 0x7f7f3c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f41800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3c0d30d0 con 0x7f7f3c15d640 2024-03-29T11:34:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f41800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3c161c90 con 0x7f7f3c0ceec0 2024-03-29T11:34:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f41800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0b6840 con 0x7f7f3c161800 2024-03-29T11:34:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f3a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f3c0ceec0 0x7f7f3c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f3aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7f3c15d640 0x7f7f3c0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53950/0 (socket says 172.21.15.12:53950) 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f3a000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f3c0ceec0 0x7f7f3c1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56256/0 (socket says 172.21.15.12:56256) 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f3aa00700 1 -- 172.21.15.12:0/3947957751 learned_addr learned my addr 172.21.15.12:0/3947957751 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f3b400700 1 --2- 172.21.15.12:0/3947957751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c161800 0x7f7f3c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.186+0000 7f7f3a000700 1 -- 172.21.15.12:0/3947957751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c161800 msgr2=0x7f7f3c0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.187+0000 7f7f3a000700 1 --2- 172.21.15.12:0/3947957751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c161800 0x7f7f3c0b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.187+0000 7f7f3a000700 1 -- 172.21.15.12:0/3947957751 >> v1:172.21.15.12:6789/0 conn(0x7f7f3c15d640 legacy=0x7f7f3c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.187+0000 7f7f3a000700 1 -- 172.21.15.12:0/3947957751 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c0d2b30 con 0x7f7f3c0ceec0 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.187+0000 7f7f39600700 1 -- 172.21.15.12:0/3947957751 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162245296 0 0) 0x7f7f3c0d30d0 con 0x7f7f3c15d640 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.187+0000 7f7f3b400700 1 --2- 172.21.15.12:0/3947957751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c161800 0x7f7f3c0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.187+0000 7f7f3a000700 1 --2- 172.21.15.12:0/3947957751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c0ceec0 0x7f7f3c1612c0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009840 tx=0x7f7f28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4702ebf0059552d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.187+0000 7f7f39600700 1 -- 172.21.15.12:0/3947957751 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28011830 con 0x7f7f3c0ceec0 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.187+0000 7f7f39600700 1 -- 172.21.15.12:0/3947957751 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f28011990 con 0x7f7f3c0ceec0 2024-03-29T11:34:16.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.187+0000 7f7f39600700 1 -- 172.21.15.12:0/3947957751 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28011d20 con 0x7f7f3c0ceec0 2024-03-29T11:34:16.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.188+0000 7f7f41800700 1 -- 172.21.15.12:0/3947957751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c0ceec0 msgr2=0x7f7f3c1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.188+0000 7f7f41800700 1 --2- 172.21.15.12:0/3947957751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c0ceec0 0x7f7f3c1612c0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009840 tx=0x7f7f28009b50 comp rx=0 tx=0).stop 2024-03-29T11:34:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.188+0000 7f7f41800700 1 -- 172.21.15.12:0/3947957751 shutdown_connections 2024-03-29T11:34:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.188+0000 7f7f41800700 1 --2- 172.21.15.12:0/3947957751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c161800 0x7f7f3c0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.188+0000 7f7f41800700 1 --2- 172.21.15.12:0/3947957751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c0ceec0 0x7f7f3c1612c0 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.188+0000 7f7f41800700 1 -- 172.21.15.12:0/3947957751 >> 172.21.15.12:0/3947957751 conn(0x7f7f3c08aff0 msgr2=0x7f7f3c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.188+0000 7f7f41800700 1 -- 172.21.15.12:0/3947957751 shutdown_connections 2024-03-29T11:34:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.188+0000 7f7f41800700 1 -- 172.21.15.12:0/3947957751 wait complete. 2024-03-29T11:34:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.189+0000 7f7f41800700 1 Processor -- start 2024-03-29T11:34:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.190+0000 7f7f41800700 1 -- start start 2024-03-29T11:34:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.190+0000 7f7f41800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c0ceec0 0x7f7f3c0c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f41800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c15d640 0x7f7f3c1f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f41800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f3c161800 0x7f7f3c09ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f41800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c1f3010 con 0x7f7f3c15d640 2024-03-29T11:34:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f41800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c1f3240 con 0x7f7f3c161800 2024-03-29T11:34:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f41800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0c9660 con 0x7f7f3c0ceec0 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c15d640 0x7f7f3c1f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c15d640 0x7f7f3c1f2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56262/0 (socket says 172.21.15.12:56262) 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3a000700 1 -- 172.21.15.12:0/3080162594 learned_addr learned my addr 172.21.15.12:0/3080162594 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3b400700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f3c161800 0x7f7f3c09ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3aa00700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c0ceec0 0x7f7f3c0c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3a000700 1 -- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c0ceec0 msgr2=0x7f7f3c0c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3a000700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c0ceec0 0x7f7f3c0c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3a000700 1 -- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f3c161800 msgr2=0x7f7f3c09ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3a000700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f3c161800 0x7f7f3c09ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3a000700 1 -- 172.21.15.12:0/3080162594 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f280094a0 con 0x7f7f3c15d640 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.191+0000 7f7f3a000700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c15d640 0x7f7f3c1f2ad0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f7f28000c00 tx=0x7f7f28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.192+0000 7f7f32a00700 1 -- 172.21.15.12:0/3080162594 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f280038a0 con 0x7f7f3c15d640 2024-03-29T11:34:16.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.192+0000 7f7f32a00700 1 -- 172.21.15.12:0/3080162594 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2801e070 con 0x7f7f3c15d640 2024-03-29T11:34:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.192+0000 7f7f41800700 1 -- 172.21.15.12:0/3080162594 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c0c9940 con 0x7f7f3c15d640 2024-03-29T11:34:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.192+0000 7f7f32a00700 1 -- 172.21.15.12:0/3080162594 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28003c60 con 0x7f7f3c15d640 2024-03-29T11:34:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.192+0000 7f7f41800700 1 -- 172.21.15.12:0/3080162594 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c09d7e0 con 0x7f7f3c15d640 2024-03-29T11:34:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.193+0000 7f7f41800700 1 -- 172.21.15.12:0/3080162594 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3c1f33b0 con 0x7f7f3c15d640 2024-03-29T11:34:16.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.194+0000 7f7f32a00700 1 -- 172.21.15.12:0/3080162594 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7f2802d450 con 0x7f7f3c15d640 2024-03-29T11:34:16.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.195+0000 7f7f32a00700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7f1c06adb0 0x7f7f1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.195+0000 7f7f32a00700 1 -- 172.21.15.12:0/3080162594 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f7f2808a360 con 0x7f7f3c15d640 2024-03-29T11:34:16.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.195+0000 7f7f3aa00700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7f1c06adb0 0x7f7f1c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.196+0000 7f7f3aa00700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7f1c06adb0 0x7f7f1c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c000f60 tx=0x7f7f2c007be0 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:16.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.199+0000 7f7f32a00700 1 -- 172.21.15.12:0/3080162594 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f28055720 con 0x7f7f3c15d640 2024-03-29T11:34:16.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.502+0000 7f7f41800700 1 -- 172.21.15.12:0/3080162594 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7f3c067720 con 0x7f7f3c15d640 2024-03-29T11:34:16.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.503+0000 7f7f32a00700 1 -- 172.21.15.12:0/3080162594 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v54) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7f28058d40 con 0x7f7f3c15d640 2024-03-29T11:34:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:16.500 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:34:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-03-29T11:34:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-03-29T11:34:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmf25JJxAAnO+JC6jYzZu/stu1f7nWVg==", 2024-03-29T11:34:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.500 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmAeBTAhAADIP9ualMKly2GJ0sNDL7Lw==", 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmxZCkMhAAYRScSIbpnUoAvxkSqpp0Bg==", 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmcjyHDRAAHhuUsI7MP8Fb8T3DfxqKRA==", 2024-03-29T11:34:16.502 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCajwZmrfKlCxAAnpMnGsaHuFvZQ+RS+RvJMw==", 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCdjwZmvmlCEBAAVmA+NRm1jLcrl20hZEMV6w==", 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCgjwZm8ex1FRAAb2YJgMRZOVabMX/7ccXc9A==", 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.504 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjjwZmBHlZGRAA5YQIEugJ8cbco1JdpxVQ4g==", 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.505 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCpjwZmWF68MBAAq/CsQf6Z0JgaoaHvpL0TCQ==", 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCsjwZmAA4XNBAAoBrZ1l/2TEBWl80izejtKw==", 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:34:16.506 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCvjwZmv8AsNxAAHQK/ouFJQ8NLCydmTG2ZuQ==", 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.507 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCyjwZmqg9IOxAAyhjrHeJtj8W2DQ3Haq8r/w==", 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmPg2fFxAAIhAr+rPy8hKtyLLKCnablA==", 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:34:16.508 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow r", 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmh+ltJhAAslVVpgJA9LxcrXs71fzzBA==", 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.509 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZmFgMsHxAAN7st1E0pOsC8PR1jv4iNoQ==", 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCWjwZm2DqwLRAAlzj95JE/j9AhOZoLrIigxw==", 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.510 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCUjwZmQX1GLBAAHKqRLm7F4aS34x9D0RaZ6w==", 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.513 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-03-29T11:34:16.513 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZm/ybFHxAAYu7bxNo0Yd8KeKDCxBamEQ==", 2024-03-29T11:34:16.513 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCVjwZmU/zXFxAAbwoj3az/br9vKM1k3UUVVw==", 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.514 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:16.515 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-29T11:34:16.515 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.505+0000 7f7f41800700 1 -- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7f1c06adb0 msgr2=0x7f7f1c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.505+0000 7f7f41800700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7f1c06adb0 0x7f7f1c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c000f60 tx=0x7f7f2c007be0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.505+0000 7f7f41800700 1 -- 172.21.15.12:0/3080162594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c15d640 msgr2=0x7f7f3c1f2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.505+0000 7f7f41800700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c15d640 0x7f7f3c1f2ad0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f7f28000c00 tx=0x7f7f28004750 comp rx=0 tx=0).stop 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.505+0000 7f7f41800700 1 -- 172.21.15.12:0/3080162594 shutdown_connections 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.505+0000 7f7f41800700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7f1c06adb0 0x7f7f1c06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.505+0000 7f7f41800700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7f3c161800 0x7f7f3c09ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.505+0000 7f7f41800700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7f3c0ceec0 0x7f7f3c0c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.505+0000 7f7f41800700 1 --2- 172.21.15.12:0/3080162594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f3c15d640 0x7f7f3c1f2ad0 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.505+0000 7f7f41800700 1 -- 172.21.15.12:0/3080162594 >> 172.21.15.12:0/3080162594 conn(0x7f7f3c08aff0 msgr2=0x7f7f3c0b6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.506+0000 7f7f41800700 1 -- 172.21.15.12:0/3080162594 shutdown_connections 2024-03-29T11:34:16.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.506+0000 7f7f41800700 1 -- 172.21.15.12:0/3080162594 wait complete. 2024-03-29T11:34:16.521 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.722+0000 7fc374a00700 1 Processor -- start 2024-03-29T11:34:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.722+0000 7fc374a00700 1 -- start start 2024-03-29T11:34:16.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.723+0000 7fc374a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3700d05e0 0x7fc3700d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.723+0000 7fc374a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3700d0f80 con 0x7fc37015da50 2024-03-29T11:34:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.723+0000 7fc374a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3700d24e0 con 0x7fc3700b6990 2024-03-29T11:34:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.723+0000 7fc374a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700b95d0 con 0x7fc3700d05e0 2024-03-29T11:34:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.724+0000 7fc36de00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc37015da50 0x7fc3700ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53956/0 (socket says 172.21.15.12:53956) 2024-03-29T11:34:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.724+0000 7fc36de00700 1 -- 172.21.15.12:0/3211408934 learned_addr learned my addr 172.21.15.12:0/3211408934 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.724+0000 7fc36d400700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3700d05e0 0x7fc3700d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.724+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509160658 0 0) 0x7fc3700d0f80 con 0x7fc37015da50 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.724+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3540035b0 con 0x7fc37015da50 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.724+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513932126 0 0) 0x7fc3700d24e0 con 0x7fc3700b6990 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.724+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3700d0f80 con 0x7fc3700b6990 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.724+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3144366600 0 0) 0x7fc3540035b0 con 0x7fc37015da50 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.724+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3700d05e0 msgr2=0x7fc3700d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3700d05e0 0x7fc3700d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 >> v1:172.21.15.159:6789/0 conn(0x7fc3700b6990 legacy=0x7fc3700b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3700d27c0 con 0x7fc37015da50 2024-03-29T11:34:16.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fc360001670 con 0x7fc37015da50 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 >> v1:172.21.15.12:6789/0 conn(0x7fc37015da50 legacy=0x7fc3700ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc354003bc0 0x7fc354003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc354004850 0x7fc354004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3700d05e0 0x7fc354005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35400bac0 con 0x7fc354003bc0 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35400bc00 con 0x7fc354004850 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35400bdc0 con 0x7fc3700d05e0 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36de00700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc354003bc0 0x7fc354003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36d400700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc354004850 0x7fc354004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36e800700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3700d05e0 0x7fc354005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36de00700 1 -- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3700d05e0 msgr2=0x7fc354005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36de00700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3700d05e0 0x7fc354005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36de00700 1 -- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc354004850 msgr2=0x7fc354004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:16.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36de00700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc354004850 0x7fc354004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.725+0000 7fc36de00700 1 -- 172.21.15.12:0/3211408934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3540035b0 con 0x7fc354003bc0 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.726+0000 7fc36de00700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc354003bc0 0x7fc354003fc0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fc36000c160 tx=0x7fc354005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83bf83d215d2e363 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.726+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36000cca0 con 0x7fc354003bc0 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.726+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36000ce20 con 0x7fc354003bc0 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.726+0000 7fc36ca00700 1 -- 172.21.15.12:0/3211408934 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360005ba0 con 0x7fc354003bc0 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.726+0000 7fc374a00700 1 -- 172.21.15.12:0/3211408934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc354003bc0 msgr2=0x7fc354003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.726+0000 7fc374a00700 1 --2- 172.21.15.12:0/3211408934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc354003bc0 0x7fc354003fc0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fc36000c160 tx=0x7fc354005880 comp rx=0 tx=0).stop 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.726+0000 7fc36e800700 1 -- 172.21.15.12:0/3211408934 reap_dead start 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.727+0000 7fc374a00700 1 -- 172.21.15.12:0/3211408934 shutdown_connections 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.727+0000 7fc374a00700 1 -- 172.21.15.12:0/3211408934 >> 172.21.15.12:0/3211408934 conn(0x7fc3700666a0 msgr2=0x7fc370066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.727+0000 7fc374a00700 1 -- 172.21.15.12:0/3211408934 shutdown_connections 2024-03-29T11:34:16.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.727+0000 7fc374a00700 1 -- 172.21.15.12:0/3211408934 wait complete. 2024-03-29T11:34:16.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.729+0000 7fc374a00700 1 Processor -- start 2024-03-29T11:34:16.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.729+0000 7fc374a00700 1 -- start start 2024-03-29T11:34:16.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.730+0000 7fc374a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc354003bc0 0x7fc3700c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc374a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3701f2b70 0x7fc3700b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc374a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3701f3d10 0x7fc37015b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc374a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc370155550 con 0x7fc3701f2b70 2024-03-29T11:34:16.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc374a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3701f33a0 con 0x7fc3701f3d10 2024-03-29T11:34:16.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc374a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3701f3680 con 0x7fc354003bc0 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc36d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3701f2b70 0x7fc3700b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc36d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3701f2b70 0x7fc3700b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56278/0 (socket says 172.21.15.12:56278) 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc36d400700 1 -- 172.21.15.12:0/2165226665 learned_addr learned my addr 172.21.15.12:0/2165226665 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc36de00700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc354003bc0 0x7fc3700c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc36e800700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3701f3d10 0x7fc37015b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc36d400700 1 -- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc354003bc0 msgr2=0x7fc3700c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc36d400700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc354003bc0 0x7fc3700c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc36d400700 1 -- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3701f3d10 msgr2=0x7fc37015b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc36d400700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3701f3d10 0x7fc37015b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.731+0000 7fc36d400700 1 -- 172.21.15.12:0/2165226665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3701f3960 con 0x7fc3701f2b70 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.732+0000 7fc36d400700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3701f2b70 0x7fc3700b4fe0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0119c0 tx=0x7fc35c011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.732+0000 7fc366000700 1 -- 172.21.15.12:0/2165226665 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c00f940 con 0x7fc3701f2b70 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.732+0000 7fc366000700 1 -- 172.21.15.12:0/2165226665 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c00faa0 con 0x7fc3701f2b70 2024-03-29T11:34:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.732+0000 7fc366000700 1 -- 172.21.15.12:0/2165226665 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c00fe50 con 0x7fc3701f2b70 2024-03-29T11:34:16.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.732+0000 7fc374a00700 1 -- 172.21.15.12:0/2165226665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc370155760 con 0x7fc3701f2b70 2024-03-29T11:34:16.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.732+0000 7fc374a00700 1 -- 172.21.15.12:0/2165226665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3700b5520 con 0x7fc3701f2b70 2024-03-29T11:34:16.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.733+0000 7fc374a00700 1 -- 172.21.15.12:0/2165226665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc370155ad0 con 0x7fc3701f2b70 2024-03-29T11:34:16.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.734+0000 7fc366000700 1 -- 172.21.15.12:0/2165226665 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc35c014440 con 0x7fc3701f2b70 2024-03-29T11:34:16.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.735+0000 7fc366000700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35006adb0 0x7fc35006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:16.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.735+0000 7fc366000700 1 -- 172.21.15.12:0/2165226665 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fc35c089960 con 0x7fc3701f2b70 2024-03-29T11:34:16.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.735+0000 7fc36de00700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35006adb0 0x7fc35006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:16.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.736+0000 7fc36de00700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35006adb0 0x7fc35006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc360000c00 tx=0x7fc360007000 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:16.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.739+0000 7fc366000700 1 -- 172.21.15.12:0/2165226665 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc35c054d20 con 0x7fc3701f2b70 2024-03-29T11:34:16.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.954+0000 7fc374a00700 1 -- 172.21.15.12:0/2165226665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3701f3af0 con 0x7fc3701f2b70 2024-03-29T11:34:16.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.955+0000 7fc366000700 1 -- 172.21.15.12:0/2165226665 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v279) v1 ==== 74+0+12515 (secure 0 0 0) 0x7fc35c022090 con 0x7fc3701f2b70 2024-03-29T11:34:16.952 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:16.952 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":279,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:34:15.952943+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 -- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35006adb0 msgr2=0x7fc35006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35006adb0 0x7fc35006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc360000c00 tx=0x7fc360007000 comp rx=0 tx=0).stop 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 -- 172.21.15.12:0/2165226665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3701f2b70 msgr2=0x7fc3700b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3701f2b70 0x7fc3700b4fe0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0119c0 tx=0x7fc35c011d80 comp rx=0 tx=0).stop 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 -- 172.21.15.12:0/2165226665 shutdown_connections 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc35006adb0 0x7fc35006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3701f3d10 0x7fc37015b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc354003bc0 0x7fc3700c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 --2- 172.21.15.12:0/2165226665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3701f2b70 0x7fc3700b4fe0 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 -- 172.21.15.12:0/2165226665 >> 172.21.15.12:0/2165226665 conn(0x7fc3700666a0 msgr2=0x7fc3700ce150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 -- 172.21.15.12:0/2165226665 shutdown_connections 2024-03-29T11:34:16.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:16.957+0000 7fc374a00700 1 -- 172.21.15.12:0/2165226665 wait complete. 2024-03-29T11:34:16.973 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:34:16.973 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:34:17.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.177+0000 7f502e400700 1 Processor -- start 2024-03-29T11:34:17.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.177+0000 7f502e400700 1 -- start start 2024-03-29T11:34:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f502e400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f50280d3eb0 0x7f50280b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f502e400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50280cf1b0 0x7f50280cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f502e400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50280ced20 con 0x7f502815d8a0 2024-03-29T11:34:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f502e400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50280d4340 con 0x7f50280d3eb0 2024-03-29T11:34:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f502e400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50280ce6d0 con 0x7f50280cf1b0 2024-03-29T11:34:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f5026a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f50280d3eb0 0x7f50280b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f5026a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f50280d3eb0 0x7f50280b2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56294/0 (socket says 172.21.15.12:56294) 2024-03-29T11:34:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f5026a00700 1 -- 172.21.15.12:0/2190762274 learned_addr learned my addr 172.21.15.12:0/2190762274 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f5027e00700 1 --2- 172.21.15.12:0/2190762274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50280cf1b0 0x7f50280cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f5026a00700 1 -- 172.21.15.12:0/2190762274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50280cf1b0 msgr2=0x7f50280cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:17.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f5026a00700 1 --2- 172.21.15.12:0/2190762274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50280cf1b0 0x7f50280cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f5026a00700 1 -- 172.21.15.12:0/2190762274 >> v1:172.21.15.159:6789/0 conn(0x7f502815d8a0 legacy=0x7f502806b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:17.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.179+0000 7f5026a00700 1 -- 172.21.15.12:0/2190762274 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50280ce9b0 con 0x7f50280d3eb0 2024-03-29T11:34:17.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.180+0000 7f5026a00700 1 --2- 172.21.15.12:0/2190762274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50280d3eb0 0x7f50280b2990 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f5014006fd0 tx=0x7f5014009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932405fcf77c949b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:17.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.180+0000 7f5026000700 1 -- 172.21.15.12:0/2190762274 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014005e30 con 0x7f50280d3eb0 2024-03-29T11:34:17.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.180+0000 7f5026000700 1 -- 172.21.15.12:0/2190762274 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5014006030 con 0x7f50280d3eb0 2024-03-29T11:34:17.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.180+0000 7f5026000700 1 -- 172.21.15.12:0/2190762274 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50140063c0 con 0x7f50280d3eb0 2024-03-29T11:34:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.180+0000 7f502e400700 1 -- 172.21.15.12:0/2190762274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50280d3eb0 msgr2=0x7f50280b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.180+0000 7f502e400700 1 --2- 172.21.15.12:0/2190762274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50280d3eb0 0x7f50280b2990 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f5014006fd0 tx=0x7f5014009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.181+0000 7f502e400700 1 -- 172.21.15.12:0/2190762274 shutdown_connections 2024-03-29T11:34:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.181+0000 7f502e400700 1 --2- 172.21.15.12:0/2190762274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f50280cf1b0 0x7f50280cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.181+0000 7f502e400700 1 --2- 172.21.15.12:0/2190762274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50280d3eb0 0x7f50280b2990 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.181+0000 7f502e400700 1 -- 172.21.15.12:0/2190762274 >> 172.21.15.12:0/2190762274 conn(0x7f502808aff0 msgr2=0x7f50280d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.181+0000 7f502e400700 1 -- 172.21.15.12:0/2190762274 shutdown_connections 2024-03-29T11:34:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.181+0000 7f502e400700 1 -- 172.21.15.12:0/2190762274 wait complete. 2024-03-29T11:34:17.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.182+0000 7f502e400700 1 Processor -- start 2024-03-29T11:34:17.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.183+0000 7f502e400700 1 -- start start 2024-03-29T11:34:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.183+0000 7f502e400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50280cf1b0 0x7f5028150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.183+0000 7f502e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f502815d8a0 0x7f50280c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.183+0000 7f502e400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5028159e70 0x7f50280c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.183+0000 7f502e400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50280c9860 con 0x7f502815d8a0 2024-03-29T11:34:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.183+0000 7f502e400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50280c7190 con 0x7f50280cf1b0 2024-03-29T11:34:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.183+0000 7f502e400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50280c7470 con 0x7f5028159e70 2024-03-29T11:34:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5026a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f502815d8a0 0x7f50280c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5026a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f502815d8a0 0x7f50280c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56296/0 (socket says 172.21.15.12:56296) 2024-03-29T11:34:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5026a00700 1 -- 172.21.15.12:0/749854680 learned_addr learned my addr 172.21.15.12:0/749854680 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5027400700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50280cf1b0 0x7f5028150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5027e00700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5028159e70 0x7f50280c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5026a00700 1 -- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5028159e70 msgr2=0x7f50280c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5026a00700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5028159e70 0x7f50280c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5026a00700 1 -- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50280cf1b0 msgr2=0x7f5028150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5026a00700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50280cf1b0 0x7f5028150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5026a00700 1 -- 172.21.15.12:0/749854680 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5014009500 con 0x7f502815d8a0 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.184+0000 7f5026a00700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f502815d8a0 0x7f50280c85e0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f5014000c00 tx=0x7f5014004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.185+0000 7f501f400700 1 -- 172.21.15.12:0/749854680 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014006930 con 0x7f502815d8a0 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.185+0000 7f501f400700 1 -- 172.21.15.12:0/749854680 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5014020070 con 0x7f502815d8a0 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.185+0000 7f501f400700 1 -- 172.21.15.12:0/749854680 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014025670 con 0x7f502815d8a0 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.185+0000 7f502e400700 1 -- 172.21.15.12:0/749854680 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50280c9ab0 con 0x7f502815d8a0 2024-03-29T11:34:17.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.185+0000 7f502e400700 1 -- 172.21.15.12:0/749854680 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50280c8b20 con 0x7f502815d8a0 2024-03-29T11:34:17.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.186+0000 7f502e400700 1 -- 172.21.15.12:0/749854680 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5028067720 con 0x7f502815d8a0 2024-03-29T11:34:17.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.187+0000 7f501f400700 1 -- 172.21.15.12:0/749854680 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5014022030 con 0x7f502815d8a0 2024-03-29T11:34:17.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.188+0000 7f501f400700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f500806adb0 0x7f500806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.188+0000 7f501f400700 1 -- 172.21.15.12:0/749854680 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f501408a810 con 0x7f502815d8a0 2024-03-29T11:34:17.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.188+0000 7f5027400700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f500806adb0 0x7f500806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.189+0000 7f5027400700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f500806adb0 0x7f500806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5018009870 tx=0x7f5018008040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:17.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.192+0000 7f501f400700 1 -- 172.21.15.12:0/749854680 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5014055b40 con 0x7f502815d8a0 2024-03-29T11:34:17.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.409+0000 7f502e400700 1 -- 172.21.15.12:0/749854680 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f50280d1950 con 0x7f502815d8a0 2024-03-29T11:34:17.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.409+0000 7f501f400700 1 -- 172.21.15.12:0/749854680 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v279) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5014006c80 con 0x7f502815d8a0 2024-03-29T11:34:17.407 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:34:17.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 -- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f500806adb0 msgr2=0x7f500806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f500806adb0 0x7f500806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5018009870 tx=0x7f5018008040 comp rx=0 tx=0).stop 2024-03-29T11:34:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 -- 172.21.15.12:0/749854680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f502815d8a0 msgr2=0x7f50280c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f502815d8a0 0x7f50280c85e0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f5014000c00 tx=0x7f5014004750 comp rx=0 tx=0).stop 2024-03-29T11:34:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 -- 172.21.15.12:0/749854680 shutdown_connections 2024-03-29T11:34:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f500806adb0 0x7f500806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5028159e70 0x7f50280c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50280cf1b0 0x7f5028150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 --2- 172.21.15.12:0/749854680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f502815d8a0 0x7f50280c85e0 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 -- 172.21.15.12:0/749854680 >> 172.21.15.12:0/749854680 conn(0x7f502808aff0 msgr2=0x7f502815df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 -- 172.21.15.12:0/749854680 shutdown_connections 2024-03-29T11:34:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.412+0000 7f502e400700 1 -- 172.21.15.12:0/749854680 wait complete. 2024-03-29T11:34:17.427 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-29T11:34:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.629+0000 7fb341c00700 1 Processor -- start 2024-03-29T11:34:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.629+0000 7fb341c00700 1 -- start start 2024-03-29T11:34:17.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.630+0000 7fb341c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb33c160f10 0x7fb33c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.630+0000 7fb341c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33c0cf230 0x7fb33c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.630+0000 7fb341c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33c0ceda0 con 0x7fb33c161590 2024-03-29T11:34:17.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.630+0000 7fb341c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33c0d23d0 con 0x7fb33c160f10 2024-03-29T11:34:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.630+0000 7fb341c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c1612f0 con 0x7fb33c0cf230 2024-03-29T11:34:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.630+0000 7fb33b400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb33c161590 0x7fb33c0b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53968/0 (socket says 172.21.15.12:53968) 2024-03-29T11:34:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.630+0000 7fb33b400700 1 -- 172.21.15.12:0/482193168 learned_addr learned my addr 172.21.15.12:0/482193168 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb33aa00700 1 --2- 172.21.15.12:0/482193168 >> v2:172.21.15.159:3300/0 conn(0x7fb33c160f10 0x7fb33c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb33a000700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33c0cf230 0x7fb33c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548714966 0 0) 0x7fb33c0ceda0 con 0x7fb33c161590 2024-03-29T11:34:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3200035b0 con 0x7fb33c161590 2024-03-29T11:34:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3156593939 0 0) 0x7fb3200035b0 con 0x7fb33c161590 2024-03-29T11:34:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33c0cf230 msgr2=0x7fb33c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33c0cf230 0x7fb33c0d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 >> v2:172.21.15.159:3300/0 conn(0x7fb33c160f10 msgr2=0x7fb33c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 --2- 172.21.15.12:0/482193168 >> v2:172.21.15.159:3300/0 conn(0x7fb33c160f10 0x7fb33c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c0ce470 con 0x7fb33c161590 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fb3340019f0 con 0x7fb33c161590 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 >> v1:172.21.15.12:6789/0 conn(0x7fb33c161590 legacy=0x7fb33c0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb320003a00 0x7fb320003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb320004850 0x7fb320004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33c0cf230 0x7fb320005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0d23d0 con 0x7fb320003a00 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c1612f0 con 0x7fb320004850 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.631+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32000ba10 con 0x7fb33c0cf230 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb33aa00700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb320003a00 0x7fb320003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb33a000700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb320004850 0x7fb320004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb33b400700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33c0cf230 0x7fb320005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb33aa00700 1 -- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33c0cf230 msgr2=0x7fb320005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb33aa00700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33c0cf230 0x7fb320005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb33aa00700 1 -- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb320004850 msgr2=0x7fb320004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb33aa00700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb320004850 0x7fb320004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb33aa00700 1 -- 172.21.15.12:0/482193168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3200035b0 con 0x7fb320003a00 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb33b400700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb33c0cf230 0x7fb320005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb33aa00700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb320003a00 0x7fb320003e20 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009b90 tx=0x7fb320005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca99a3472ef9e7d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.632+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c00e6c0 con 0x7fb320003a00 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.633+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32c00e820 con 0x7fb320003a00 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.633+0000 7fb339600700 1 -- 172.21.15.12:0/482193168 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c00eb90 con 0x7fb320003a00 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.633+0000 7fb341c00700 1 -- 172.21.15.12:0/482193168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb320003a00 msgr2=0x7fb320003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.633+0000 7fb341c00700 1 --2- 172.21.15.12:0/482193168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb320003a00 0x7fb320003e20 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009b90 tx=0x7fb320005880 comp rx=0 tx=0).stop 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.633+0000 7fb33b400700 1 -- 172.21.15.12:0/482193168 reap_dead start 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.633+0000 7fb341c00700 1 -- 172.21.15.12:0/482193168 shutdown_connections 2024-03-29T11:34:17.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.633+0000 7fb341c00700 1 -- 172.21.15.12:0/482193168 >> 172.21.15.12:0/482193168 conn(0x7fb33c08aff0 msgr2=0x7fb33c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:17.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.633+0000 7fb341c00700 1 -- 172.21.15.12:0/482193168 shutdown_connections 2024-03-29T11:34:17.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.633+0000 7fb341c00700 1 -- 172.21.15.12:0/482193168 wait complete. 2024-03-29T11:34:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.636+0000 7fb341c00700 1 Processor -- start 2024-03-29T11:34:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.636+0000 7fb341c00700 1 -- start start 2024-03-29T11:34:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb341c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb320003a00 0x7fb33c150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb341c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb33c0b3be0 0x7fb33c0b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb341c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb33c0cc060 0x7fb33c0cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb341c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0cca50 con 0x7fb33c0b3be0 2024-03-29T11:34:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb341c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0b11e0 con 0x7fb33c0cc060 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb341c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0b4060 con 0x7fb320003a00 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb33c0b3be0 0x7fb33c0b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb33c0b3be0 0x7fb33c0b0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56304/0 (socket says 172.21.15.12:56304) 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33a000700 1 -- 172.21.15.12:0/1193298999 learned_addr learned my addr 172.21.15.12:0/1193298999 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33aa00700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb320003a00 0x7fb33c150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33b400700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb33c0cc060 0x7fb33c0cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33a000700 1 -- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb320003a00 msgr2=0x7fb33c150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33a000700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb320003a00 0x7fb33c150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33a000700 1 -- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb33c0cc060 msgr2=0x7fb33c0cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33a000700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb33c0cc060 0x7fb33c0cc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33a000700 1 -- 172.21.15.12:0/1193298999 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c150a60 con 0x7fb33c0b3be0 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.637+0000 7fb33b400700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb33c0cc060 0x7fb33c0cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.638+0000 7fb33a000700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb33c0b3be0 0x7fb33c0b0ca0 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fb328011aa0 tx=0x7fb328011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:17.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.638+0000 7fb332a00700 1 -- 172.21.15.12:0/1193298999 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32800f940 con 0x7fb33c0b3be0 2024-03-29T11:34:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.638+0000 7fb332a00700 1 -- 172.21.15.12:0/1193298999 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32800faa0 con 0x7fb33c0b3be0 2024-03-29T11:34:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.638+0000 7fb332a00700 1 -- 172.21.15.12:0/1193298999 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32800fe50 con 0x7fb33c0b3be0 2024-03-29T11:34:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.638+0000 7fb341c00700 1 -- 172.21.15.12:0/1193298999 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33c150d40 con 0x7fb33c0b3be0 2024-03-29T11:34:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.638+0000 7fb341c00700 1 -- 172.21.15.12:0/1193298999 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33c151300 con 0x7fb33c0b3be0 2024-03-29T11:34:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.639+0000 7fb341c00700 1 -- 172.21.15.12:0/1193298999 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33c067720 con 0x7fb33c0b3be0 2024-03-29T11:34:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.640+0000 7fb332a00700 1 -- 172.21.15.12:0/1193298999 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb328014440 con 0x7fb33c0b3be0 2024-03-29T11:34:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.641+0000 7fb332a00700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31c06ade0 0x7fb31c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.641+0000 7fb332a00700 1 -- 172.21.15.12:0/1193298999 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fb328089930 con 0x7fb33c0b3be0 2024-03-29T11:34:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.641+0000 7fb33aa00700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31c06ade0 0x7fb31c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.642+0000 7fb33aa00700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31c06ade0 0x7fb31c06d2a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0090b0 tx=0x7fb32c01f040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:17.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.645+0000 7fb332a00700 1 -- 172.21.15.12:0/1193298999 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb328054dd0 con 0x7fb33c0b3be0 2024-03-29T11:34:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.861+0000 7fb341c00700 1 -- 172.21.15.12:0/1193298999 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb33c08b760 con 0x7fb33c0b3be0 2024-03-29T11:34:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.861+0000 7fb332a00700 1 -- 172.21.15.12:0/1193298999 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v279) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb328022020 con 0x7fb33c0b3be0 2024-03-29T11:34:17.859 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:17.859 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:34:17.859 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:17.859 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-29T11:34:17.859 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-29T11:34:17.859 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:17.859 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:34:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.863+0000 7fb341c00700 1 -- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31c06ade0 msgr2=0x7fb31c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.863+0000 7fb341c00700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31c06ade0 0x7fb31c06d2a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0090b0 tx=0x7fb32c01f040 comp rx=0 tx=0).stop 2024-03-29T11:34:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.863+0000 7fb341c00700 1 -- 172.21.15.12:0/1193298999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb33c0b3be0 msgr2=0x7fb33c0b0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.863+0000 7fb341c00700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb33c0b3be0 0x7fb33c0b0ca0 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fb328011aa0 tx=0x7fb328011db0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.863+0000 7fb341c00700 1 -- 172.21.15.12:0/1193298999 shutdown_connections 2024-03-29T11:34:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.863+0000 7fb341c00700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb31c06ade0 0x7fb31c06d2a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.863+0000 7fb341c00700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb33c0cc060 0x7fb33c0cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.864+0000 7fb341c00700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb320003a00 0x7fb33c150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.864+0000 7fb341c00700 1 --2- 172.21.15.12:0/1193298999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb33c0b3be0 0x7fb33c0b0ca0 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.864+0000 7fb341c00700 1 -- 172.21.15.12:0/1193298999 >> 172.21.15.12:0/1193298999 conn(0x7fb33c08aff0 msgr2=0x7fb33c0d3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.864+0000 7fb341c00700 1 -- 172.21.15.12:0/1193298999 shutdown_connections 2024-03-29T11:34:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:17.864+0000 7fb341c00700 1 -- 172.21.15.12:0/1193298999 wait complete. 2024-03-29T11:34:17.878 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-29T11:34:17.879 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-29T11:34:18.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.083+0000 7f1064600700 1 Processor -- start 2024-03-29T11:34:18.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.083+0000 7f1064600700 1 -- start start 2024-03-29T11:34:18.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.084+0000 7f1064600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f105c15ddf0 0x7f105c0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f1064600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105c0ce8a0 con 0x7f105c0d28e0 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f1064600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105c0cdc90 con 0x7f105c15e1d0 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f1064600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f105c15ec60 con 0x7f105c15ddf0 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f1061c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f105c15ddf0 0x7f105c0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f1062600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f105c0d28e0 0x7f105c15f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53984/0 (socket says 172.21.15.12:53984) 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f1061c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f105c15ddf0 0x7f105c0ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56316/0 (socket says 172.21.15.12:56316) 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f1062600700 1 -- 172.21.15.12:0/1313750311 learned_addr learned my addr 172.21.15.12:0/1313750311 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f1061c00700 1 -- 172.21.15.12:0/1313750311 >> v1:172.21.15.159:6789/0 conn(0x7f105c15e1d0 legacy=0x7f105c0d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f1061c00700 1 -- 172.21.15.12:0/1313750311 >> v1:172.21.15.12:6789/0 conn(0x7f105c0d28e0 legacy=0x7f105c15f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f1061c00700 1 -- 172.21.15.12:0/1313750311 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c0cdf70 con 0x7f105c15ddf0 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.085+0000 7f105be00700 1 -- 172.21.15.12:0/1313750311 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124840773 0 0) 0x7f105c0ce8a0 con 0x7f105c0d28e0 2024-03-29T11:34:18.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.086+0000 7f1061c00700 1 --2- 172.21.15.12:0/1313750311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15ddf0 0x7f105c0ced30 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f1050009840 tx=0x7f1050009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57107844987c11b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:18.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.086+0000 7f105be00700 1 -- 172.21.15.12:0/1313750311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050004230 con 0x7f105c15ddf0 2024-03-29T11:34:18.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.086+0000 7f105be00700 1 -- 172.21.15.12:0/1313750311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1050004390 con 0x7f105c15ddf0 2024-03-29T11:34:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.086+0000 7f105be00700 1 -- 172.21.15.12:0/1313750311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10500036a0 con 0x7f105c15ddf0 2024-03-29T11:34:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.086+0000 7f1064600700 1 -- 172.21.15.12:0/1313750311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15ddf0 msgr2=0x7f105c0ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.086+0000 7f1064600700 1 --2- 172.21.15.12:0/1313750311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15ddf0 0x7f105c0ced30 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f1050009840 tx=0x7f1050009b50 comp rx=0 tx=0).stop 2024-03-29T11:34:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.086+0000 7f1064600700 1 -- 172.21.15.12:0/1313750311 shutdown_connections 2024-03-29T11:34:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.086+0000 7f1064600700 1 --2- 172.21.15.12:0/1313750311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15ddf0 0x7f105c0ced30 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.086+0000 7f1064600700 1 -- 172.21.15.12:0/1313750311 >> 172.21.15.12:0/1313750311 conn(0x7f105c08aff0 msgr2=0x7f105c0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.087+0000 7f1064600700 1 -- 172.21.15.12:0/1313750311 shutdown_connections 2024-03-29T11:34:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.087+0000 7f1064600700 1 -- 172.21.15.12:0/1313750311 wait complete. 2024-03-29T11:34:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.088+0000 7f1064600700 1 Processor -- start 2024-03-29T11:34:18.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.088+0000 7f1064600700 1 -- start start 2024-03-29T11:34:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.089+0000 7f1064600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f105c15ddf0 0x7f105c1f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.089+0000 7f1064600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15e1d0 0x7f105c0b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.089+0000 7f1064600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f105c0ca5e0 0x7f105c14f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.089+0000 7f1064600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c0cab50 con 0x7f105c15e1d0 2024-03-29T11:34:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.089+0000 7f1064600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c0cae30 con 0x7f105c0ca5e0 2024-03-29T11:34:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.089+0000 7f1064600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105c0cb110 con 0x7f105c15ddf0 2024-03-29T11:34:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1061200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15e1d0 0x7f105c0b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1061200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15e1d0 0x7f105c0b5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56332/0 (socket says 172.21.15.12:56332) 2024-03-29T11:34:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1061200700 1 -- 172.21.15.12:0/1011307786 learned_addr learned my addr 172.21.15.12:0/1011307786 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1061c00700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f105c15ddf0 0x7f105c1f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1062600700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f105c0ca5e0 0x7f105c14f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1061200700 1 -- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f105c15ddf0 msgr2=0x7f105c1f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1061200700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f105c15ddf0 0x7f105c1f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1061200700 1 -- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f105c0ca5e0 msgr2=0x7f105c14f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1061200700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f105c0ca5e0 0x7f105c14f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1061200700 1 -- 172.21.15.12:0/1011307786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10500094a0 con 0x7f105c15e1d0 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.090+0000 7f1061200700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15e1d0 0x7f105c0b5070 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f104800c4d0 tx=0x7f104800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.091+0000 7f1059600700 1 -- 172.21.15.12:0/1011307786 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048011790 con 0x7f105c15e1d0 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.091+0000 7f1059600700 1 -- 172.21.15.12:0/1011307786 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10480118f0 con 0x7f105c15e1d0 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.091+0000 7f1059600700 1 -- 172.21.15.12:0/1011307786 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048011ca0 con 0x7f105c15e1d0 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.091+0000 7f1064600700 1 -- 172.21.15.12:0/1011307786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105c0cb3f0 con 0x7f105c15e1d0 2024-03-29T11:34:18.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.091+0000 7f1064600700 1 -- 172.21.15.12:0/1011307786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105c1f2f10 con 0x7f105c15e1d0 2024-03-29T11:34:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.092+0000 7f1064600700 1 -- 172.21.15.12:0/1011307786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f105c0cb910 con 0x7f105c15e1d0 2024-03-29T11:34:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.093+0000 7f1059600700 1 -- 172.21.15.12:0/1011307786 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f104801c460 con 0x7f105c15e1d0 2024-03-29T11:34:18.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.094+0000 7f1059600700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f103c06adb0 0x7f103c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.094+0000 7f1059600700 1 -- 172.21.15.12:0/1011307786 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f1048088f20 con 0x7f105c15e1d0 2024-03-29T11:34:18.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.095+0000 7f1061c00700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f103c06adb0 0x7f103c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.096+0000 7f1061c00700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f103c06adb0 0x7f103c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f10500090b0 tx=0x7f105001c040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:18.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.099+0000 7f1059600700 1 -- 172.21.15.12:0/1011307786 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f105c0cb910 con 0x7f105c15e1d0 2024-03-29T11:34:18.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.313+0000 7f1064600700 1 -- 172.21.15.12:0/1011307786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f105c0cb910 con 0x7f105c15e1d0 2024-03-29T11:34:18.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.399+0000 7f1059600700 1 -- 172.21.15.12:0/1011307786 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v280) v1 ==== 138+0+0 (secure 0 0 0) 0x7f105c0cb910 con 0x7f105c15e1d0 2024-03-29T11:34:18.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.402+0000 7f1064600700 1 -- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f103c06adb0 msgr2=0x7f103c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:18.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.402+0000 7f1064600700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f103c06adb0 0x7f103c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f10500090b0 tx=0x7f105001c040 comp rx=0 tx=0).stop 2024-03-29T11:34:18.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.402+0000 7f1064600700 1 -- 172.21.15.12:0/1011307786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15e1d0 msgr2=0x7f105c0b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:18.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.402+0000 7f1064600700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15e1d0 0x7f105c0b5070 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f104800c4d0 tx=0x7f104800c7e0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.403+0000 7f1064600700 1 -- 172.21.15.12:0/1011307786 shutdown_connections 2024-03-29T11:34:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.403+0000 7f1064600700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f103c06adb0 0x7f103c06d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.403+0000 7f1064600700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f105c0ca5e0 0x7f105c14f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.403+0000 7f1064600700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f105c15ddf0 0x7f105c1f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.403+0000 7f1064600700 1 --2- 172.21.15.12:0/1011307786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c15e1d0 0x7f105c0b5070 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.403+0000 7f1064600700 1 -- 172.21.15.12:0/1011307786 >> 172.21.15.12:0/1011307786 conn(0x7f105c08aff0 msgr2=0x7f105c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.403+0000 7f1064600700 1 -- 172.21.15.12:0/1011307786 shutdown_connections 2024-03-29T11:34:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.404+0000 7f1064600700 1 -- 172.21.15.12:0/1011307786 wait complete. 2024-03-29T11:34:18.403 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-29T11:34:18.420 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-29T11:34:18.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.628+0000 7f0b12200700 1 Processor -- start 2024-03-29T11:34:18.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.628+0000 7f0b12200700 1 -- start start 2024-03-29T11:34:18.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b12200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b0c15d800 0x7f0b0c0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b12200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0b0c0ce230 0x7f0b0c0ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b12200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0b0c0b2070 0x7f0b0c0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b12200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c0b40f0 con 0x7f0b0c0b2070 2024-03-29T11:34:18.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b12200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c0709c0 con 0x7f0b0c0ce230 2024-03-29T11:34:18.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b12200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c070c80 con 0x7f0b0c15d800 2024-03-29T11:34:18.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0b0c0b2070 0x7f0b0c0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0be00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0b0c0b2070 0x7f0b0c0b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56346/0 (socket says 172.21.15.12:56346) 2024-03-29T11:34:18.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0be00700 1 -- 172.21.15.12:0/1997334317 learned_addr learned my addr 172.21.15.12:0/1997334317 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:18.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0b400700 1 --2- 172.21.15.12:0/1997334317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b0c15d800 0x7f0b0c0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0aa00700 1 --2- 172.21.15.12:0/1997334317 >> v2:172.21.15.159:3300/0 conn(0x7f0b0c0ce230 0x7f0b0c0ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0be00700 1 -- 172.21.15.12:0/1997334317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b0c15d800 msgr2=0x7f0b0c0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0be00700 1 --2- 172.21.15.12:0/1997334317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b0c15d800 0x7f0b0c0725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0be00700 1 -- 172.21.15.12:0/1997334317 >> v2:172.21.15.159:3300/0 conn(0x7f0b0c0ce230 msgr2=0x7f0b0c0ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0be00700 1 --2- 172.21.15.12:0/1997334317 >> v2:172.21.15.159:3300/0 conn(0x7f0b0c0ce230 0x7f0b0c0ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0be00700 1 -- 172.21.15.12:0/1997334317 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c070f60 con 0x7f0b0c0b2070 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.630+0000 7f0b0aa00700 1 --2- 172.21.15.12:0/1997334317 >> v2:172.21.15.159:3300/0 conn(0x7f0b0c0ce230 0x7f0b0c0ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.631+0000 7f0b0be00700 1 --2- 172.21.15.12:0/1997334317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0b2070 0x7f0b0c0b2500 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c06b600 tx=0x7f0b0400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2211e8560311d11b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.631+0000 7f0b0a000700 1 -- 172.21.15.12:0/1997334317 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0400c9c0 con 0x7f0b0c0b2070 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.631+0000 7f0b0a000700 1 -- 172.21.15.12:0/1997334317 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0400cb20 con 0x7f0b0c0b2070 2024-03-29T11:34:18.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.631+0000 7f0b0a000700 1 -- 172.21.15.12:0/1997334317 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b04008500 con 0x7f0b0c0b2070 2024-03-29T11:34:18.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.631+0000 7f0b12200700 1 -- 172.21.15.12:0/1997334317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0b2070 msgr2=0x7f0b0c0b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:18.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.631+0000 7f0b12200700 1 --2- 172.21.15.12:0/1997334317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0b2070 0x7f0b0c0b2500 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c06b600 tx=0x7f0b0400c330 comp rx=0 tx=0).stop 2024-03-29T11:34:18.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.632+0000 7f0b12200700 1 -- 172.21.15.12:0/1997334317 shutdown_connections 2024-03-29T11:34:18.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.632+0000 7f0b12200700 1 --2- 172.21.15.12:0/1997334317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0b2070 0x7f0b0c0b2500 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.632+0000 7f0b12200700 1 --2- 172.21.15.12:0/1997334317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b0c15d800 0x7f0b0c0725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.632+0000 7f0b12200700 1 --2- 172.21.15.12:0/1997334317 >> v2:172.21.15.159:3300/0 conn(0x7f0b0c0ce230 0x7f0b0c0ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.632+0000 7f0b12200700 1 -- 172.21.15.12:0/1997334317 >> 172.21.15.12:0/1997334317 conn(0x7f0b0c0b0d70 msgr2=0x7f0b0c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:18.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.632+0000 7f0b12200700 1 -- 172.21.15.12:0/1997334317 shutdown_connections 2024-03-29T11:34:18.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.632+0000 7f0b12200700 1 -- 172.21.15.12:0/1997334317 wait complete. 2024-03-29T11:34:18.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.633+0000 7f0b12200700 1 Processor -- start 2024-03-29T11:34:18.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.634+0000 7f0b12200700 1 -- start start 2024-03-29T11:34:18.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.634+0000 7f0b12200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b0c0b2070 0x7f0b0c1f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.634+0000 7f0b12200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0ce230 0x7f0b0c0adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b12200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b0c15d800 0x7f0b0c1531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b12200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c144fb0 con 0x7f0b0c0ce230 2024-03-29T11:34:18.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b12200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c1519b0 con 0x7f0b0c0b2070 2024-03-29T11:34:18.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b12200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c142030 con 0x7f0b0c15d800 2024-03-29T11:34:18.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b0aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0ce230 0x7f0b0c0adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b0aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0ce230 0x7f0b0c0adf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56356/0 (socket says 172.21.15.12:56356) 2024-03-29T11:34:18.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b0aa00700 1 -- 172.21.15.12:0/2557819209 learned_addr learned my addr 172.21.15.12:0/2557819209 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b0b400700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b0c0b2070 0x7f0b0c1f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b0be00700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b0c15d800 0x7f0b0c1531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b0aa00700 1 -- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b0c15d800 msgr2=0x7f0b0c1531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b0aa00700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b0c15d800 0x7f0b0c1531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b0aa00700 1 -- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b0c0b2070 msgr2=0x7f0b0c1f2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b0aa00700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b0c0b2070 0x7f0b0c1f2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.635+0000 7f0b0aa00700 1 -- 172.21.15.12:0/2557819209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0400ac40 con 0x7f0b0c0ce230 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.636+0000 7f0b0aa00700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0ce230 0x7f0b0c0adf40 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f0af800d800 tx=0x7f0af800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.636+0000 7f0b03400700 1 -- 172.21.15.12:0/2557819209 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8004030 con 0x7f0b0c0ce230 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.636+0000 7f0b03400700 1 -- 172.21.15.12:0/2557819209 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af8004190 con 0x7f0b0c0ce230 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.636+0000 7f0b12200700 1 -- 172.21.15.12:0/2557819209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0c0ae480 con 0x7f0b0c0ce230 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.636+0000 7f0b03400700 1 -- 172.21.15.12:0/2557819209 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8004520 con 0x7f0b0c0ce230 2024-03-29T11:34:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.636+0000 7f0b12200700 1 -- 172.21.15.12:0/2557819209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0c151110 con 0x7f0b0c0ce230 2024-03-29T11:34:18.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.637+0000 7f0b12200700 1 -- 172.21.15.12:0/2557819209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0c067720 con 0x7f0b0c0ce230 2024-03-29T11:34:18.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.638+0000 7f0b03400700 1 -- 172.21.15.12:0/2557819209 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0af80036a0 con 0x7f0b0c0ce230 2024-03-29T11:34:18.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.639+0000 7f0b03400700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0aec06adb0 0x7f0aec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:18.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.639+0000 7f0b03400700 1 -- 172.21.15.12:0/2557819209 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 5299+0+0 (secure 0 0 0) 0x7f0af8089000 con 0x7f0b0c0ce230 2024-03-29T11:34:18.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.640+0000 7f0b0b400700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0aec06adb0 0x7f0aec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:18.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.640+0000 7f0b0b400700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0aec06adb0 0x7f0aec06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0afc006b20 tx=0x7f0afc008040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.643+0000 7f0b03400700 1 -- 172.21.15.12:0/2557819209 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0af80542f0 con 0x7f0b0c0ce230 2024-03-29T11:34:18.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:18.856+0000 7f0b12200700 1 -- 172.21.15.12:0/2557819209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0b0c154fc0 con 0x7f0b0c0ce230 2024-03-29T11:34:19.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.410+0000 7f0b03400700 1 -- 172.21.15.12:0/2557819209 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v281) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0af80230d0 con 0x7f0b0c0ce230 2024-03-29T11:34:19.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.416+0000 7f0b12200700 1 -- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0aec06adb0 msgr2=0x7f0aec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.416+0000 7f0b12200700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0aec06adb0 0x7f0aec06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0afc006b20 tx=0x7f0afc008040 comp rx=0 tx=0).stop 2024-03-29T11:34:19.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.416+0000 7f0b12200700 1 -- 172.21.15.12:0/2557819209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0ce230 msgr2=0x7f0b0c0adf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.416+0000 7f0b12200700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0ce230 0x7f0b0c0adf40 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f0af800d800 tx=0x7f0af800db10 comp rx=0 tx=0).stop 2024-03-29T11:34:19.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.416+0000 7f0b12200700 1 -- 172.21.15.12:0/2557819209 shutdown_connections 2024-03-29T11:34:19.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.416+0000 7f0b12200700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0aec06adb0 0x7f0aec06d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.416+0000 7f0b12200700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b0c15d800 0x7f0b0c1531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.416+0000 7f0b12200700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b0c0b2070 0x7f0b0c1f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.416+0000 7f0b12200700 1 --2- 172.21.15.12:0/2557819209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b0c0ce230 0x7f0b0c0adf40 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.416+0000 7f0b12200700 1 -- 172.21.15.12:0/2557819209 >> 172.21.15.12:0/2557819209 conn(0x7f0b0c0b0d70 msgr2=0x7f0b0c0b2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:19.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.418+0000 7f0b12200700 1 -- 172.21.15.12:0/2557819209 shutdown_connections 2024-03-29T11:34:19.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.418+0000 7f0b12200700 1 -- 172.21.15.12:0/2557819209 wait complete. 2024-03-29T11:34:19.418 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-29T11:34:19.443 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-29T11:34:19.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.648+0000 7fd0ca800700 1 Processor -- start 2024-03-29T11:34:19.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.648+0000 7fd0ca800700 1 -- start start 2024-03-29T11:34:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.649+0000 7fd0ca800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd0c415d9d0 0x7fd0c40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.649+0000 7fd0ca800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c4161420 con 0x7fd0c40d16a0 2024-03-29T11:34:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.649+0000 7fd0ca800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c4161700 con 0x7fd0c40d2540 2024-03-29T11:34:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.649+0000 7fd0ca800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c415edc0 con 0x7fd0c415d9d0 2024-03-29T11:34:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.649+0000 7fd0c2a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd0c40d16a0 0x7fd0c40d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60214/0 (socket says 172.21.15.12:60214) 2024-03-29T11:34:19.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.649+0000 7fd0c2a00700 1 -- 172.21.15.12:0/4047621688 learned_addr learned my addr 172.21.15.12:0/4047621688 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:19.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c3400700 1 --2- 172.21.15.12:0/4047621688 >> v2:172.21.15.159:3300/0 conn(0x7fd0c415d9d0 0x7fd0c40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:19.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727113436 0 0) 0x7fd0c4161420 con 0x7fd0c40d16a0 2024-03-29T11:34:19.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0a80035b0 con 0x7fd0c40d16a0 2024-03-29T11:34:19.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3094621844 0 0) 0x7fd0a80035b0 con 0x7fd0c40d16a0 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 >> v2:172.21.15.159:3300/0 conn(0x7fd0c415d9d0 msgr2=0x7fd0c40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 --2- 172.21.15.12:0/4047621688 >> v2:172.21.15.159:3300/0 conn(0x7fd0c415d9d0 0x7fd0c40d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 >> v1:172.21.15.159:6789/0 conn(0x7fd0c40d2540 legacy=0x7fd0c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c41619e0 con 0x7fd0c40d16a0 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fd0b00017f0 con 0x7fd0c40d16a0 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 >> v1:172.21.15.12:6789/0 conn(0x7fd0c40d16a0 legacy=0x7fd0c40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 0x7fd0a8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0a8004400 0x7fd0a80048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8005170 0x7fd0a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a800c080 con 0x7fd0a8005170 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a800c1f0 con 0x7fd0a8003a00 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a800c3b0 con 0x7fd0a8004400 2024-03-29T11:34:19.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.650+0000 7fd0c3400700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 0x7fd0a8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.651+0000 7fd0c3e00700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8005170 0x7fd0a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.651+0000 7fd0c2a00700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0a8004400 0x7fd0a80048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.651+0000 7fd0c3400700 1 -- 172.21.15.12:0/4047621688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0a8004400 msgr2=0x7fd0a80048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.651+0000 7fd0c3400700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0a8004400 0x7fd0a80048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.651+0000 7fd0c3400700 1 -- 172.21.15.12:0/4047621688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8005170 msgr2=0x7fd0a800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.651+0000 7fd0c3400700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8005170 0x7fd0a800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.651+0000 7fd0c3400700 1 -- 172.21.15.12:0/4047621688 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a80035b0 con 0x7fd0a8003a00 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.651+0000 7fd0c3e00700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8005170 0x7fd0a800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.651+0000 7fd0c3e00700 1 -- 172.21.15.12:0/4047621688 reap_dead start 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.651+0000 7fd0c3400700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 0x7fd0a8003de0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009b60 tx=0x7fd0a8005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94cc18a773a1cba server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.652+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b400e6c0 con 0x7fd0a8003a00 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.652+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b400e820 con 0x7fd0a8003a00 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.652+0000 7fd0c2000700 1 -- 172.21.15.12:0/4047621688 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b400eb90 con 0x7fd0a8003a00 2024-03-29T11:34:19.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.652+0000 7fd0ca800700 1 -- 172.21.15.12:0/4047621688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 msgr2=0x7fd0a8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.652+0000 7fd0ca800700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 0x7fd0a8003de0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009b60 tx=0x7fd0a8005880 comp rx=0 tx=0).stop 2024-03-29T11:34:19.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.652+0000 7fd0ca800700 1 -- 172.21.15.12:0/4047621688 shutdown_connections 2024-03-29T11:34:19.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.652+0000 7fd0ca800700 1 --2- 172.21.15.12:0/4047621688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 0x7fd0a8003de0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.652+0000 7fd0ca800700 1 -- 172.21.15.12:0/4047621688 >> 172.21.15.12:0/4047621688 conn(0x7fd0c408aff0 msgr2=0x7fd0c40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:19.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.652+0000 7fd0ca800700 1 -- 172.21.15.12:0/4047621688 shutdown_connections 2024-03-29T11:34:19.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.652+0000 7fd0ca800700 1 -- 172.21.15.12:0/4047621688 wait complete. 2024-03-29T11:34:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.654+0000 7fd0ca800700 1 Processor -- start 2024-03-29T11:34:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.654+0000 7fd0ca800700 1 -- start start 2024-03-29T11:34:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0ca800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 0x7fd0c4152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0ca800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0c415e7c0 0x7fd0c41f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0ca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0c41f3e20 0x7fd0c40ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0ca800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c4152f60 con 0x7fd0c41f3e20 2024-03-29T11:34:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0ca800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c40ca5f0 con 0x7fd0a8003a00 2024-03-29T11:34:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0ca800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c40ca8d0 con 0x7fd0c415e7c0 2024-03-29T11:34:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0c3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0c41f3e20 0x7fd0c40ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0c3400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 0x7fd0c4152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0c3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0c41f3e20 0x7fd0c40ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51432/0 (socket says 172.21.15.12:51432) 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0c3e00700 1 -- 172.21.15.12:0/1278094439 learned_addr learned my addr 172.21.15.12:0/1278094439 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.655+0000 7fd0c2a00700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0c415e7c0 0x7fd0c41f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0c3e00700 1 -- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0c415e7c0 msgr2=0x7fd0c41f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0c3e00700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0c415e7c0 0x7fd0c41f38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0c3e00700 1 -- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 msgr2=0x7fd0c4152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0c3e00700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 0x7fd0c4152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0c3e00700 1 -- 172.21.15.12:0/1278094439 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c40cabb0 con 0x7fd0c41f3e20 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0c3e00700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0c41f3e20 0x7fd0c40ccc00 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40d2f40 tx=0x7fd0bc00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0bb400700 1 -- 172.21.15.12:0/1278094439 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc003cd0 con 0x7fd0c41f3e20 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0bb400700 1 -- 172.21.15.12:0/1278094439 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc003e50 con 0x7fd0c41f3e20 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0ca800700 1 -- 172.21.15.12:0/1278094439 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c40cae90 con 0x7fd0c41f3e20 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0bb400700 1 -- 172.21.15.12:0/1278094439 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc00dbe0 con 0x7fd0c41f3e20 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.656+0000 7fd0ca800700 1 -- 172.21.15.12:0/1278094439 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c415caf0 con 0x7fd0c41f3e20 2024-03-29T11:34:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.657+0000 7fd0ca800700 1 -- 172.21.15.12:0/1278094439 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c415cc80 con 0x7fd0c41f3e20 2024-03-29T11:34:19.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.659+0000 7fd0bb400700 1 -- 172.21.15.12:0/1278094439 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd0bc00dd40 con 0x7fd0c41f3e20 2024-03-29T11:34:19.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.660+0000 7fd0bb400700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0a406adb0 0x7fd0a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:19.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.660+0000 7fd0bb400700 1 -- 172.21.15.12:0/1278094439 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(281..281 src has 1..281) v4 ==== 5681+0+0 (secure 0 0 0) 0x7fd0bc08a1d0 con 0x7fd0c41f3e20 2024-03-29T11:34:19.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.661+0000 7fd0c3400700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0a406adb0 0x7fd0a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:19.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.662+0000 7fd0c3400700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0a406adb0 0x7fd0a406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd0b40090b0 tx=0x7fd0b401e040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.665+0000 7fd0bb400700 1 -- 172.21.15.12:0/1278094439 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0bc055300 con 0x7fd0c41f3e20 2024-03-29T11:34:19.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.923+0000 7fd0ca800700 1 -- 172.21.15.12:0/1278094439 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd0c40cb3b0 con 0x7fd0c41f3e20 2024-03-29T11:34:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.930+0000 7fd0bb400700 1 -- 172.21.15.12:0/1278094439 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 36 and data pool 37 v142) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd0c40cb3b0 con 0x7fd0c41f3e20 2024-03-29T11:34:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.935+0000 7fd0ca800700 1 -- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0a406adb0 msgr2=0x7fd0a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.935+0000 7fd0ca800700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0a406adb0 0x7fd0a406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd0b40090b0 tx=0x7fd0b401e040 comp rx=0 tx=0).stop 2024-03-29T11:34:19.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.935+0000 7fd0ca800700 1 -- 172.21.15.12:0/1278094439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0c41f3e20 msgr2=0x7fd0c40ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.935+0000 7fd0ca800700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0c41f3e20 0x7fd0c40ccc00 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40d2f40 tx=0x7fd0bc00d430 comp rx=0 tx=0).stop 2024-03-29T11:34:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.940+0000 7fd0ca800700 1 -- 172.21.15.12:0/1278094439 shutdown_connections 2024-03-29T11:34:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.940+0000 7fd0ca800700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd0a406adb0 0x7fd0a406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.940+0000 7fd0ca800700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0c41f3e20 0x7fd0c40ccc00 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.940+0000 7fd0ca800700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0a8003a00 0x7fd0c4152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.940+0000 7fd0ca800700 1 --2- 172.21.15.12:0/1278094439 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0c415e7c0 0x7fd0c41f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.940+0000 7fd0ca800700 1 -- 172.21.15.12:0/1278094439 >> 172.21.15.12:0/1278094439 conn(0x7fd0c408aff0 msgr2=0x7fd0c40b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:19.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.941+0000 7fd0ca800700 1 -- 172.21.15.12:0/1278094439 shutdown_connections 2024-03-29T11:34:19.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:19.942+0000 7fd0ca800700 1 -- 172.21.15.12:0/1278094439 wait complete. 2024-03-29T11:34:19.944 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 36 and data pool 37 2024-03-29T11:34:19.970 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:20.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.170+0000 7f8d87c00700 1 Processor -- start 2024-03-29T11:34:20.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.170+0000 7f8d87c00700 1 -- start start 2024-03-29T11:34:20.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.171+0000 7f8d87c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8d800d18b0 0x7f8d800cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.171+0000 7f8d87c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80161590 0x7f8d80161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.171+0000 7f8d87c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d800ceda0 con 0x7f8d80160f10 2024-03-29T11:34:20.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.171+0000 7f8d87c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d800d1d30 con 0x7f8d800d18b0 2024-03-29T11:34:20.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.171+0000 7f8d87c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d80161f60 con 0x7f8d80161590 2024-03-29T11:34:20.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.172+0000 7f8d7fe00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8d800d18b0 0x7f8d800cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.172+0000 7f8d85c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80161590 0x7f8d80161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.172+0000 7f8d85200700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f8d80160f10 0x7f8d800d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:43042/0 (socket says 172.21.15.12:43042) 2024-03-29T11:34:20.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.172+0000 7f8d85200700 1 -- 172.21.15.12:0/1795373713 learned_addr learned my addr 172.21.15.12:0/1795373713 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.172+0000 7f8d7fe00700 1 --2- 172.21.15.12:0/1795373713 >> v2:172.21.15.159:3300/0 conn(0x7f8d800d18b0 0x7f8d800cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:45452/0 (socket says 172.21.15.12:45452) 2024-03-29T11:34:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.172+0000 7f8d7fe00700 1 -- 172.21.15.12:0/1795373713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80161590 msgr2=0x7f8d80161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.172+0000 7f8d7fe00700 1 --2- 172.21.15.12:0/1795373713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80161590 0x7f8d80161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.172+0000 7f8d7fe00700 1 -- 172.21.15.12:0/1795373713 >> v1:172.21.15.159:6789/0 conn(0x7f8d80160f10 legacy=0x7f8d800d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.172+0000 7f8d7fe00700 1 -- 172.21.15.12:0/1795373713 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d801612f0 con 0x7f8d800d18b0 2024-03-29T11:34:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.172+0000 7f8d85c00700 1 --2- 172.21.15.12:0/1795373713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80161590 0x7f8d80161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:34:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.173+0000 7f8d7fe00700 1 --2- 172.21.15.12:0/1795373713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d800d18b0 0x7f8d800cf230 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c009870 tx=0x7f8d6c009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eba02b269264f075 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.173+0000 7f8d7f400700 1 -- 172.21.15.12:0/1795373713 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c011830 con 0x7f8d800d18b0 2024-03-29T11:34:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.173+0000 7f8d7f400700 1 -- 172.21.15.12:0/1795373713 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6c011990 con 0x7f8d800d18b0 2024-03-29T11:34:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.173+0000 7f8d7f400700 1 -- 172.21.15.12:0/1795373713 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c011d20 con 0x7f8d800d18b0 2024-03-29T11:34:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.173+0000 7f8d87c00700 1 -- 172.21.15.12:0/1795373713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d800d18b0 msgr2=0x7f8d800cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.173+0000 7f8d87c00700 1 --2- 172.21.15.12:0/1795373713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d800d18b0 0x7f8d800cf230 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c009870 tx=0x7f8d6c009b80 comp rx=0 tx=0).stop 2024-03-29T11:34:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.174+0000 7f8d87c00700 1 -- 172.21.15.12:0/1795373713 shutdown_connections 2024-03-29T11:34:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.174+0000 7f8d87c00700 1 --2- 172.21.15.12:0/1795373713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80161590 0x7f8d80161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.174+0000 7f8d87c00700 1 --2- 172.21.15.12:0/1795373713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d800d18b0 0x7f8d800cf230 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.174+0000 7f8d87c00700 1 -- 172.21.15.12:0/1795373713 >> 172.21.15.12:0/1795373713 conn(0x7f8d8008aff0 msgr2=0x7f8d800d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.174+0000 7f8d87c00700 1 -- 172.21.15.12:0/1795373713 shutdown_connections 2024-03-29T11:34:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.174+0000 7f8d87c00700 1 -- 172.21.15.12:0/1795373713 wait complete. 2024-03-29T11:34:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.175+0000 7f8d87c00700 1 Processor -- start 2024-03-29T11:34:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.176+0000 7f8d87c00700 1 -- start start 2024-03-29T11:34:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.176+0000 7f8d87c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d800d18b0 0x7f8d8014c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.176+0000 7f8d87c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80160f10 0x7f8d800b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.176+0000 7f8d87c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d80161590 0x7f8d8009d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.176+0000 7f8d87c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8009e990 con 0x7f8d800d18b0 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.176+0000 7f8d87c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8014c930 con 0x7f8d80161590 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.176+0000 7f8d87c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8009dd80 con 0x7f8d80160f10 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d85200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d800d18b0 0x7f8d8014c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d85200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d800d18b0 0x7f8d8014c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51438/0 (socket says 172.21.15.12:51438) 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d85200700 1 -- 172.21.15.12:0/3434120677 learned_addr learned my addr 172.21.15.12:0/3434120677 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d7fe00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80160f10 0x7f8d800b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d85c00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d80161590 0x7f8d8009d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d85200700 1 -- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80160f10 msgr2=0x7f8d800b7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d85200700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80160f10 0x7f8d800b7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d85200700 1 -- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d80161590 msgr2=0x7f8d8009d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d85200700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d80161590 0x7f8d8009d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d85200700 1 -- 172.21.15.12:0/3434120677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d78009a70 con 0x7f8d800d18b0 2024-03-29T11:34:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.177+0000 7f8d85200700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d800d18b0 0x7f8d8014c3f0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400c480 tx=0x7f8d7400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.178+0000 7f8d7cc00700 1 -- 172.21.15.12:0/3434120677 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d74005d70 con 0x7f8d800d18b0 2024-03-29T11:34:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.178+0000 7f8d7cc00700 1 -- 172.21.15.12:0/3434120677 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7400ce80 con 0x7f8d800d18b0 2024-03-29T11:34:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.178+0000 7f8d7cc00700 1 -- 172.21.15.12:0/3434120677 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d740072d0 con 0x7f8d800d18b0 2024-03-29T11:34:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.178+0000 7f8d87c00700 1 -- 172.21.15.12:0/3434120677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6c0094a0 con 0x7f8d800d18b0 2024-03-29T11:34:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.178+0000 7f8d87c00700 1 -- 172.21.15.12:0/3434120677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8009e340 con 0x7f8d800d18b0 2024-03-29T11:34:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.179+0000 7f8d87c00700 1 -- 172.21.15.12:0/3434120677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d80067720 con 0x7f8d800d18b0 2024-03-29T11:34:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.180+0000 7f8d7cc00700 1 -- 172.21.15.12:0/3434120677 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8d74007430 con 0x7f8d800d18b0 2024-03-29T11:34:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.181+0000 7f8d7cc00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d6006adb0 0x7f8d6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.181+0000 7f8d7cc00700 1 -- 172.21.15.12:0/3434120677 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(282..282 src has 1..282) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f8d74088740 con 0x7f8d800d18b0 2024-03-29T11:34:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.182+0000 7f8d7fe00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d6006adb0 0x7f8d6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.183+0000 7f8d7fe00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d6006adb0 0x7f8d6006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0090b0 tx=0x7f8d6c01c040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:20.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.186+0000 7f8d7cc00700 1 -- 172.21.15.12:0/3434120677 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d740537d0 con 0x7f8d800d18b0 2024-03-29T11:34:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.402+0000 7f8d87c00700 1 -- 172.21.15.12:0/3434120677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d8006d670 con 0x7f8d800d18b0 2024-03-29T11:34:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.404+0000 7f8d7cc00700 1 -- 172.21.15.12:0/3434120677 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v282) v1 ==== 74+0+16188 (secure 0 0 0) 0x7f8d74056df0 con 0x7f8d800d18b0 2024-03-29T11:34:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:20.401 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":282,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:34:19.924336+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:34:18.360241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-03-29T11:34:18.903400+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.406+0000 7f8d87c00700 1 -- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d6006adb0 msgr2=0x7f8d6006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.406+0000 7f8d87c00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d6006adb0 0x7f8d6006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0090b0 tx=0x7f8d6c01c040 comp rx=0 tx=0).stop 2024-03-29T11:34:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.406+0000 7f8d87c00700 1 -- 172.21.15.12:0/3434120677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d800d18b0 msgr2=0x7f8d8014c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.406+0000 7f8d87c00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d800d18b0 0x7f8d8014c3f0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400c480 tx=0x7f8d7400c790 comp rx=0 tx=0).stop 2024-03-29T11:34:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.406+0000 7f8d87c00700 1 -- 172.21.15.12:0/3434120677 shutdown_connections 2024-03-29T11:34:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.406+0000 7f8d87c00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f8d6006adb0 0x7f8d6006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.406+0000 7f8d87c00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d80161590 0x7f8d8009d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.406+0000 7f8d87c00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d800d18b0 0x7f8d8014c3f0 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.406+0000 7f8d87c00700 1 --2- 172.21.15.12:0/3434120677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80160f10 0x7f8d800b7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.406+0000 7f8d87c00700 1 -- 172.21.15.12:0/3434120677 >> 172.21.15.12:0/3434120677 conn(0x7f8d8008aff0 msgr2=0x7f8d800cacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:20.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.407+0000 7f8d87c00700 1 -- 172.21.15.12:0/3434120677 shutdown_connections 2024-03-29T11:34:20.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.407+0000 7f8d87c00700 1 -- 172.21.15.12:0/3434120677 wait complete. 2024-03-29T11:34:20.423 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.624+0000 7f1acc200700 1 Processor -- start 2024-03-29T11:34:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.624+0000 7f1acc200700 1 -- start start 2024-03-29T11:34:20.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1acc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 0x7f1ac40cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1acc200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ac40ceca0 0x7f1ac40b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1acc200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac4160fc0 con 0x7f1ac40ce0e0 2024-03-29T11:34:20.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1acc200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac40b65d0 con 0x7f1ac40ceca0 2024-03-29T11:34:20.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1acc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40d2a80 con 0x7f1ac4160be0 2024-03-29T11:34:20.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1aca200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ac40ceca0 0x7f1ac40b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1aca200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ac40ceca0 0x7f1ac40b6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51450/0 (socket says 172.21.15.12:51450) 2024-03-29T11:34:20.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1aca200700 1 -- 172.21.15.12:0/1457229109 learned_addr learned my addr 172.21.15.12:0/1457229109 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:20.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1ac9800700 1 --2- 172.21.15.12:0/1457229109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 0x7f1ac40cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1aca200700 1 -- 172.21.15.12:0/1457229109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 msgr2=0x7f1ac40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:20.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1aca200700 1 --2- 172.21.15.12:0/1457229109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 0x7f1ac40cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1aca200700 1 -- 172.21.15.12:0/1457229109 >> v1:172.21.15.159:6789/0 conn(0x7f1ac40ce0e0 legacy=0x7f1ac40ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.626+0000 7f1aca200700 1 -- 172.21.15.12:0/1457229109 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40d21e0 con 0x7f1ac40ceca0 2024-03-29T11:34:20.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.627+0000 7f1aca200700 1 --2- 172.21.15.12:0/1457229109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 0x7f1ac40b6a20 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009e80 tx=0x7f1abc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42400b966c7265e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:20.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.627+0000 7f1ac3e00700 1 -- 172.21.15.12:0/1457229109 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc00e070 con 0x7f1ac40ceca0 2024-03-29T11:34:20.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.627+0000 7f1ac3e00700 1 -- 172.21.15.12:0/1457229109 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc00b8d0 con 0x7f1ac40ceca0 2024-03-29T11:34:20.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.627+0000 7f1ac3e00700 1 -- 172.21.15.12:0/1457229109 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc00bc80 con 0x7f1ac40ceca0 2024-03-29T11:34:20.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.627+0000 7f1acc200700 1 -- 172.21.15.12:0/1457229109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 msgr2=0x7f1ac40b6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.627+0000 7f1acc200700 1 --2- 172.21.15.12:0/1457229109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 0x7f1ac40b6a20 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009e80 tx=0x7f1abc00b330 comp rx=0 tx=0).stop 2024-03-29T11:34:20.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.628+0000 7f1acc200700 1 -- 172.21.15.12:0/1457229109 shutdown_connections 2024-03-29T11:34:20.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.628+0000 7f1acc200700 1 --2- 172.21.15.12:0/1457229109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 0x7f1ac40b6a20 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.628+0000 7f1acc200700 1 --2- 172.21.15.12:0/1457229109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 0x7f1ac40cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.628+0000 7f1acc200700 1 -- 172.21.15.12:0/1457229109 >> 172.21.15.12:0/1457229109 conn(0x7f1ac408aff0 msgr2=0x7f1ac40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:20.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.628+0000 7f1acc200700 1 -- 172.21.15.12:0/1457229109 shutdown_connections 2024-03-29T11:34:20.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.628+0000 7f1acc200700 1 -- 172.21.15.12:0/1457229109 wait complete. 2024-03-29T11:34:20.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.629+0000 7f1acc200700 1 Processor -- start 2024-03-29T11:34:20.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.630+0000 7f1acc200700 1 -- start start 2024-03-29T11:34:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.630+0000 7f1acc200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ac40ce0e0 0x7f1ac414da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.630+0000 7f1acc200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 0x7f1ac40b8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.630+0000 7f1acc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 0x7f1ac40b8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.630+0000 7f1acc200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40b7c50 con 0x7f1ac40ceca0 2024-03-29T11:34:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.630+0000 7f1acc200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac414df40 con 0x7f1ac40ce0e0 2024-03-29T11:34:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.630+0000 7f1acc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac41f3c60 con 0x7f1ac4160be0 2024-03-29T11:34:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1ac8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 0x7f1ac40b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1ac8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 0x7f1ac40b8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51462/0 (socket says 172.21.15.12:51462) 2024-03-29T11:34:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1ac8e00700 1 -- 172.21.15.12:0/2327542338 learned_addr learned my addr 172.21.15.12:0/2327542338 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1ac9800700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ac40ce0e0 0x7f1ac414da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1aca200700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 0x7f1ac40b8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1ac8e00700 1 -- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 msgr2=0x7f1ac40b8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1ac8e00700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 0x7f1ac40b8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1ac8e00700 1 -- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ac40ce0e0 msgr2=0x7f1ac414da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1ac8e00700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ac40ce0e0 0x7f1ac414da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1ac8e00700 1 -- 172.21.15.12:0/2327542338 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc009b30 con 0x7f1ac40ceca0 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1aca200700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 0x7f1ac40b8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.631+0000 7f1ac8e00700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 0x7f1ac40b8910 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c4e0 tx=0x7f1ab000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.632+0000 7f1ac1600700 1 -- 172.21.15.12:0/2327542338 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00052d0 con 0x7f1ac40ceca0 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.632+0000 7f1ac1600700 1 -- 172.21.15.12:0/2327542338 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab0005430 con 0x7f1ac40ceca0 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.632+0000 7f1acc200700 1 -- 172.21.15.12:0/2327542338 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac41f3980 con 0x7f1ac40ceca0 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.632+0000 7f1ac1600700 1 -- 172.21.15.12:0/2327542338 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00038c0 con 0x7f1ac40ceca0 2024-03-29T11:34:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.632+0000 7f1acc200700 1 -- 172.21.15.12:0/2327542338 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac40caa30 con 0x7f1ac40ceca0 2024-03-29T11:34:20.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.633+0000 7f1acc200700 1 -- 172.21.15.12:0/2327542338 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac4067720 con 0x7f1ac40ceca0 2024-03-29T11:34:20.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.634+0000 7f1ac1600700 1 -- 172.21.15.12:0/2327542338 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1ab0003a20 con 0x7f1ac40ceca0 2024-03-29T11:34:20.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.635+0000 7f1ac1600700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1aa406adb0 0x7f1aa406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:20.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.635+0000 7f1ac1600700 1 -- 172.21.15.12:0/2327542338 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(282..282 src has 1..282) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f1ab00888c0 con 0x7f1ac40ceca0 2024-03-29T11:34:20.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.636+0000 7f1ac9800700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1aa406adb0 0x7f1aa406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.636+0000 7f1ac9800700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1aa406adb0 0x7f1aa406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009b40 tx=0x7f1ab8009210 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:20.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.640+0000 7f1ac1600700 1 -- 172.21.15.12:0/2327542338 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ab00539d0 con 0x7f1ac40ceca0 2024-03-29T11:34:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.855+0000 7f1acc200700 1 -- 172.21.15.12:0/2327542338 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ac40b5cb0 con 0x7f1ac40ceca0 2024-03-29T11:34:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.856+0000 7f1ac1600700 1 -- 172.21.15.12:0/2327542338 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v282) v1 ==== 74+0+16188 (secure 0 0 0) 0x7f1ab0017090 con 0x7f1ac40ceca0 2024-03-29T11:34:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:20.854 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":282,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:34:19.924336+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:34:18.360241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-03-29T11:34:18.903400+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:20.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 -- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1aa406adb0 msgr2=0x7f1aa406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1aa406adb0 0x7f1aa406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009b40 tx=0x7f1ab8009210 comp rx=0 tx=0).stop 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 -- 172.21.15.12:0/2327542338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 msgr2=0x7f1ac40b8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 0x7f1ac40b8910 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c4e0 tx=0x7f1ab000c7f0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 -- 172.21.15.12:0/2327542338 shutdown_connections 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1aa406adb0 0x7f1aa406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ac4160be0 0x7f1ac40b8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ac40ce0e0 0x7f1ac414da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 --2- 172.21.15.12:0/2327542338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ac40ceca0 0x7f1ac40b8910 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 -- 172.21.15.12:0/2327542338 >> 172.21.15.12:0/2327542338 conn(0x7f1ac408aff0 msgr2=0x7f1ac40cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 -- 172.21.15.12:0/2327542338 shutdown_connections 2024-03-29T11:34:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:20.859+0000 7f1acc200700 1 -- 172.21.15.12:0/2327542338 wait complete. 2024-03-29T11:34:20.875 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-29T11:34:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.081+0000 7fccf1600700 1 Processor -- start 2024-03-29T11:34:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.081+0000 7fccf1600700 1 -- start start 2024-03-29T11:34:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.082+0000 7fccf1600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec0d16a0 0x7fccec0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.082+0000 7fccf1600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fccec0d2370 0x7fccec0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.082+0000 7fccf1600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec161420 con 0x7fccec15d9d0 2024-03-29T11:34:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.082+0000 7fccf1600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccec0d2820 con 0x7fccec0d2370 2024-03-29T11:34:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.082+0000 7fccf1600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccec0d4340 con 0x7fccec0d16a0 2024-03-29T11:34:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.083+0000 7fccea000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec0d16a0 0x7fccec0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.083+0000 7fcceb400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fccec0d2370 0x7fccec0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.083+0000 7fccea000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec0d16a0 0x7fccec0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:48688/0 (socket says 172.21.15.12:48688) 2024-03-29T11:34:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.083+0000 7fccea000700 1 -- 172.21.15.12:0/887479940 learned_addr learned my addr 172.21.15.12:0/887479940 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.083+0000 7fcceaa00700 1 --1- 172.21.15.12:0/887479940 >> v1:172.21.15.159:6789/0 conn(0x7fccec15d9d0 0x7fccec0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:43068/0 (socket says 172.21.15.12:43068) 2024-03-29T11:34:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.083+0000 7fccea000700 1 -- 172.21.15.12:0/887479940 >> v2:172.21.15.159:3300/0 conn(0x7fccec0d2370 msgr2=0x7fccec0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.083+0000 7fccea000700 1 --2- 172.21.15.12:0/887479940 >> v2:172.21.15.159:3300/0 conn(0x7fccec0d2370 0x7fccec0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.083+0000 7fccea000700 1 -- 172.21.15.12:0/887479940 >> v1:172.21.15.159:6789/0 conn(0x7fccec15d9d0 legacy=0x7fccec0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.083+0000 7fccea000700 1 -- 172.21.15.12:0/887479940 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec161700 con 0x7fccec0d16a0 2024-03-29T11:34:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.083+0000 7fccea000700 1 --2- 172.21.15.12:0/887479940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec0d16a0 0x7fccec0d1e30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fccd8006fd0 tx=0x7fccd8009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfeacf5b21fe9500 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fcce9600700 1 -- 172.21.15.12:0/887479940 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8005e30 con 0x7fccec0d16a0 2024-03-29T11:34:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fcce9600700 1 -- 172.21.15.12:0/887479940 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd8004030 con 0x7fccec0d16a0 2024-03-29T11:34:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fcce9600700 1 -- 172.21.15.12:0/887479940 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd80043e0 con 0x7fccec0d16a0 2024-03-29T11:34:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fccf1600700 1 -- 172.21.15.12:0/887479940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec0d16a0 msgr2=0x7fccec0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fccf1600700 1 --2- 172.21.15.12:0/887479940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec0d16a0 0x7fccec0d1e30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fccd8006fd0 tx=0x7fccd8009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fccf1600700 1 -- 172.21.15.12:0/887479940 shutdown_connections 2024-03-29T11:34:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fccf1600700 1 --2- 172.21.15.12:0/887479940 >> v2:172.21.15.159:3300/0 conn(0x7fccec0d2370 0x7fccec0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fccf1600700 1 --2- 172.21.15.12:0/887479940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec0d16a0 0x7fccec0d1e30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fccf1600700 1 -- 172.21.15.12:0/887479940 >> 172.21.15.12:0/887479940 conn(0x7fccec08aff0 msgr2=0x7fccec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fccf1600700 1 -- 172.21.15.12:0/887479940 shutdown_connections 2024-03-29T11:34:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.084+0000 7fccf1600700 1 -- 172.21.15.12:0/887479940 wait complete. 2024-03-29T11:34:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.087+0000 7fccf1600700 1 Processor -- start 2024-03-29T11:34:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.087+0000 7fccf1600700 1 -- start start 2024-03-29T11:34:21.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.088+0000 7fccf1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccec0d16a0 0x7fccec0b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:21.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.088+0000 7fccf1600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccec0d2370 0x7fccec1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:21.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.088+0000 7fccf1600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec15d9d0 0x7fccec0cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:21.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.088+0000 7fccf1600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec1f4560 con 0x7fccec0d16a0 2024-03-29T11:34:21.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.088+0000 7fccf1600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec0b6020 con 0x7fccec0d2370 2024-03-29T11:34:21.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.088+0000 7fccf1600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccec0b7570 con 0x7fccec15d9d0 2024-03-29T11:34:21.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.088+0000 7fcceaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccec0d16a0 0x7fccec0b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:21.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.088+0000 7fcceaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccec0d16a0 0x7fccec0b7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51470/0 (socket says 172.21.15.12:51470) 2024-03-29T11:34:21.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.088+0000 7fcceaa00700 1 -- 172.21.15.12:0/2635611295 learned_addr learned my addr 172.21.15.12:0/2635611295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:21.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.088+0000 7fcceb400700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec15d9d0 0x7fccec0cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:21.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.089+0000 7fccea000700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccec0d2370 0x7fccec1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:21.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.089+0000 7fcceaa00700 1 -- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec15d9d0 msgr2=0x7fccec0cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:21.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.089+0000 7fcceaa00700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec15d9d0 0x7fccec0cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:21.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.089+0000 7fcceaa00700 1 -- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccec0d2370 msgr2=0x7fccec1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:21.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.089+0000 7fcceaa00700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccec0d2370 0x7fccec1558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:21.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.089+0000 7fcceaa00700 1 -- 172.21.15.12:0/2635611295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd8009500 con 0x7fccec0d16a0 2024-03-29T11:34:21.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.089+0000 7fcceaa00700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccec0d16a0 0x7fccec0b7030 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00c500 tx=0x7fccdc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:21.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.089+0000 7fcce2a00700 1 -- 172.21.15.12:0/2635611295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc005cb0 con 0x7fccec0d16a0 2024-03-29T11:34:21.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.089+0000 7fcce2a00700 1 -- 172.21.15.12:0/2635611295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc005e10 con 0x7fccec0d16a0 2024-03-29T11:34:21.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.090+0000 7fccf1600700 1 -- 172.21.15.12:0/2635611295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccec0b6270 con 0x7fccec0d16a0 2024-03-29T11:34:21.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.090+0000 7fcce2a00700 1 -- 172.21.15.12:0/2635611295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc0072d0 con 0x7fccec0d16a0 2024-03-29T11:34:21.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.090+0000 7fccf1600700 1 -- 172.21.15.12:0/2635611295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccec0b6830 con 0x7fccec0d16a0 2024-03-29T11:34:21.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.091+0000 7fccf1600700 1 -- 172.21.15.12:0/2635611295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccec0d1ca0 con 0x7fccec0d16a0 2024-03-29T11:34:21.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.091+0000 7fcce2a00700 1 -- 172.21.15.12:0/2635611295 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fccdc007430 con 0x7fccec0d16a0 2024-03-29T11:34:21.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.092+0000 7fcce2a00700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcccc06adb0 0x7fcccc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:21.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.092+0000 7fcce2a00700 1 -- 172.21.15.12:0/2635611295 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fccdc088680 con 0x7fccec0d16a0 2024-03-29T11:34:21.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.093+0000 7fccea000700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcccc06adb0 0x7fcccc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:21.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.093+0000 7fccea000700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcccc06adb0 0x7fcccc06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fccec0d1ae0 tx=0x7fccd8008070 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:21.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.097+0000 7fcce2a00700 1 -- 172.21.15.12:0/2635611295 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fccdc0537c0 con 0x7fccec0d16a0 2024-03-29T11:34:21.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.357+0000 7fccf1600700 1 -- 172.21.15.12:0/2635611295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fccec067720 con 0x7fccec0d16a0 2024-03-29T11:34:21.697 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:34:21.703 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:34:21.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.939+0000 7fcce2a00700 1 -- 172.21.15.12:0/2635611295 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v145) v1 ==== 118+0+0 (secure 0 0 0) 0x7fccdc056de0 con 0x7fccec0d16a0 2024-03-29T11:34:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.941+0000 7fccf1600700 1 -- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcccc06adb0 msgr2=0x7fcccc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.941+0000 7fccf1600700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcccc06adb0 0x7fcccc06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fccec0d1ae0 tx=0x7fccd8008070 comp rx=0 tx=0).stop 2024-03-29T11:34:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.941+0000 7fccf1600700 1 -- 172.21.15.12:0/2635611295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccec0d16a0 msgr2=0x7fccec0b7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.941+0000 7fccf1600700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccec0d16a0 0x7fccec0b7030 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00c500 tx=0x7fccdc00c810 comp rx=0 tx=0).stop 2024-03-29T11:34:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.942+0000 7fccf1600700 1 -- 172.21.15.12:0/2635611295 shutdown_connections 2024-03-29T11:34:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.942+0000 7fccf1600700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fcccc06adb0 0x7fcccc06d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.942+0000 7fccf1600700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccec15d9d0 0x7fccec0cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.942+0000 7fccf1600700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccec0d16a0 0x7fccec0b7030 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:21.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.942+0000 7fccf1600700 1 --2- 172.21.15.12:0/2635611295 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccec0d2370 0x7fccec1558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:21.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.942+0000 7fccf1600700 1 -- 172.21.15.12:0/2635611295 >> 172.21.15.12:0/2635611295 conn(0x7fccec08aff0 msgr2=0x7fccec0b1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:21.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.942+0000 7fccf1600700 1 -- 172.21.15.12:0/2635611295 shutdown_connections 2024-03-29T11:34:21.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:21.942+0000 7fccf1600700 1 -- 172.21.15.12:0/2635611295 wait complete. 2024-03-29T11:34:21.958 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-29T11:34:21.958 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-29T11:34:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.162+0000 7f4275400700 1 Processor -- start 2024-03-29T11:34:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.162+0000 7f4275400700 1 -- start start 2024-03-29T11:34:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.163+0000 7f4275400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f42700d16a0 0x7f42700d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.163+0000 7f4275400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 0x7f42700ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.163+0000 7f4275400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4270161420 con 0x7f427015d9d0 2024-03-29T11:34:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.163+0000 7f4275400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42700d2820 con 0x7f42700d16a0 2024-03-29T11:34:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.163+0000 7f4275400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42700d4340 con 0x7f42700d2370 2024-03-29T11:34:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426de00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f42700d16a0 0x7f42700d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 0x7f42700ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426de00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f42700d16a0 0x7f42700d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:45498/0 (socket says 172.21.15.12:45498) 2024-03-29T11:34:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426f200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 0x7f42700ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:48712/0 (socket says 172.21.15.12:48712) 2024-03-29T11:34:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426e800700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f427015d9d0 0x7f42700d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:43078/0 (socket says 172.21.15.12:43078) 2024-03-29T11:34:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426de00700 1 -- 172.21.15.12:0/3986695032 learned_addr learned my addr 172.21.15.12:0/3986695032 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426de00700 1 -- 172.21.15.12:0/3986695032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 msgr2=0x7f42700ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426de00700 1 --2- 172.21.15.12:0/3986695032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 0x7f42700ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426de00700 1 -- 172.21.15.12:0/3986695032 >> v1:172.21.15.159:6789/0 conn(0x7f427015d9d0 legacy=0x7f42700d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426de00700 1 -- 172.21.15.12:0/3986695032 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4270161700 con 0x7f42700d16a0 2024-03-29T11:34:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.164+0000 7f426d400700 1 -- 172.21.15.12:0/3986695032 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244133371 0 0) 0x7f4270161420 con 0x7f427015d9d0 2024-03-29T11:34:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.165+0000 7f426de00700 1 --2- 172.21.15.12:0/3986695032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42700d16a0 0x7f42700d1e30 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f425c006fd0 tx=0x7f425c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ed2079162970c6b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.165+0000 7f426d400700 1 -- 172.21.15.12:0/3986695032 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c005e30 con 0x7f42700d16a0 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.165+0000 7f426d400700 1 -- 172.21.15.12:0/3986695032 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425c004030 con 0x7f42700d16a0 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.165+0000 7f426d400700 1 -- 172.21.15.12:0/3986695032 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c0043e0 con 0x7f42700d16a0 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.165+0000 7f4275400700 1 -- 172.21.15.12:0/3986695032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42700d16a0 msgr2=0x7f42700d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.165+0000 7f4275400700 1 --2- 172.21.15.12:0/3986695032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42700d16a0 0x7f42700d1e30 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f425c006fd0 tx=0x7f425c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.165+0000 7f4275400700 1 -- 172.21.15.12:0/3986695032 shutdown_connections 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.165+0000 7f4275400700 1 --2- 172.21.15.12:0/3986695032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 0x7f42700ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.165+0000 7f4275400700 1 --2- 172.21.15.12:0/3986695032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42700d16a0 0x7f42700d1e30 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.165+0000 7f4275400700 1 -- 172.21.15.12:0/3986695032 >> 172.21.15.12:0/3986695032 conn(0x7f427008aff0 msgr2=0x7f42700d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.166+0000 7f4275400700 1 -- 172.21.15.12:0/3986695032 shutdown_connections 2024-03-29T11:34:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.166+0000 7f4275400700 1 -- 172.21.15.12:0/3986695032 wait complete. 2024-03-29T11:34:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.167+0000 7f4275400700 1 Processor -- start 2024-03-29T11:34:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.167+0000 7f4275400700 1 -- start start 2024-03-29T11:34:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.168+0000 7f4275400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42700d16a0 0x7f42701e7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.168+0000 7f4275400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 0x7f42700b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.168+0000 7f4275400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f427015d9d0 0x7f42701f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.168+0000 7f4275400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42701e81a0 con 0x7f427015d9d0 2024-03-29T11:34:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.168+0000 7f4275400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4270156d80 con 0x7f42700d16a0 2024-03-29T11:34:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.168+0000 7f4275400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42701e8390 con 0x7f42700d2370 2024-03-29T11:34:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f427015d9d0 0x7f42701f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f427015d9d0 0x7f42701f2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51476/0 (socket says 172.21.15.12:51476) 2024-03-29T11:34:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426f200700 1 -- 172.21.15.12:0/4039164060 learned_addr learned my addr 172.21.15.12:0/4039164060 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426e800700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42700d16a0 0x7f42701e7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426de00700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 0x7f42700b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426f200700 1 -- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 msgr2=0x7f42700b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426f200700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 0x7f42700b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426f200700 1 -- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42700d16a0 msgr2=0x7f42701e7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426f200700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42700d16a0 0x7f42701e7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426f200700 1 -- 172.21.15.12:0/4039164060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c009500 con 0x7f427015d9d0 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426e800700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42700d16a0 0x7f42701e7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.169+0000 7f426f200700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f427015d9d0 0x7f42701f2830 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f426800efc0 tx=0x7f426800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.170+0000 7f4266a00700 1 -- 172.21.15.12:0/4039164060 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426800cd40 con 0x7f427015d9d0 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.170+0000 7f4266a00700 1 -- 172.21.15.12:0/4039164060 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426800f040 con 0x7f427015d9d0 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.170+0000 7f4266a00700 1 -- 172.21.15.12:0/4039164060 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42680146c0 con 0x7f427015d9d0 2024-03-29T11:34:22.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.170+0000 7f4275400700 1 -- 172.21.15.12:0/4039164060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42701e8670 con 0x7f427015d9d0 2024-03-29T11:34:22.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.170+0000 7f4275400700 1 -- 172.21.15.12:0/4039164060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42701e8c30 con 0x7f427015d9d0 2024-03-29T11:34:22.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.171+0000 7f4275400700 1 -- 172.21.15.12:0/4039164060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42700d1ca0 con 0x7f427015d9d0 2024-03-29T11:34:22.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.172+0000 7f4266a00700 1 -- 172.21.15.12:0/4039164060 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4268014820 con 0x7f427015d9d0 2024-03-29T11:34:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.173+0000 7f4266a00700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f425006adb0 0x7f425006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.173+0000 7f4266a00700 1 -- 172.21.15.12:0/4039164060 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f4268089ff0 con 0x7f427015d9d0 2024-03-29T11:34:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.173+0000 7f426e800700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f425006adb0 0x7f425006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.174+0000 7f426e800700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f425006adb0 0x7f425006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4260000f60 tx=0x7f4260007d30 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:22.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.177+0000 7f4266a00700 1 -- 172.21.15.12:0/4039164060 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4268054ec0 con 0x7f427015d9d0 2024-03-29T11:34:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:22.443+0000 7f4275400700 1 -- 172.21.15.12:0/4039164060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4270067720 con 0x7f427015d9d0 2024-03-29T11:34:23.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.422+0000 7f4266a00700 1 -- 172.21.15.12:0/4039164060 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v146) v1 ==== 122+0+0 (secure 0 0 0) 0x7f42680180c0 con 0x7f427015d9d0 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 -- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f425006adb0 msgr2=0x7f425006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f425006adb0 0x7f425006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4260000f60 tx=0x7f4260007d30 comp rx=0 tx=0).stop 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 -- 172.21.15.12:0/4039164060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f427015d9d0 msgr2=0x7f42701f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f427015d9d0 0x7f42701f2830 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f426800efc0 tx=0x7f426800c3a0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 -- 172.21.15.12:0/4039164060 shutdown_connections 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f425006adb0 0x7f425006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f427015d9d0 0x7f42701f2830 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42700d16a0 0x7f42701e7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 --2- 172.21.15.12:0/4039164060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42700d2370 0x7f42700b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 -- 172.21.15.12:0/4039164060 >> 172.21.15.12:0/4039164060 conn(0x7f427008aff0 msgr2=0x7f427006d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 -- 172.21.15.12:0/4039164060 shutdown_connections 2024-03-29T11:34:23.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.425+0000 7f4275400700 1 -- 172.21.15.12:0/4039164060 wait complete. 2024-03-29T11:34:23.444 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-29T11:34:23.444 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:34:23.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.649+0000 7f77b9a00700 1 Processor -- start 2024-03-29T11:34:23.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.649+0000 7f77b9a00700 1 -- start start 2024-03-29T11:34:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.650+0000 7f77b9a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77b41619d0 0x7f77b41685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.650+0000 7f77b9a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b40d30d0 con 0x7f77b40ceec0 2024-03-29T11:34:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.650+0000 7f77b9a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b40b6840 con 0x7f77b415d640 2024-03-29T11:34:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.650+0000 7f77b9a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b40d2b30 con 0x7f77b41619d0 2024-03-29T11:34:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.650+0000 7f77b3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77b41619d0 0x7f77b41685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.650+0000 7f77b3400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77b41619d0 0x7f77b41685f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51486/0 (socket says 172.21.15.12:51486) 2024-03-29T11:34:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.650+0000 7f77b3400700 1 -- 172.21.15.12:0/649589 learned_addr learned my addr 172.21.15.12:0/649589 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.650+0000 7f77b2000700 1 --1- 172.21.15.12:0/649589 >> v1:172.21.15.12:6789/0 conn(0x7f77b40ceec0 0x7f77b41612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60220/0 (socket says 172.21.15.12:60220) 2024-03-29T11:34:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.651+0000 7f77b3400700 1 -- 172.21.15.12:0/649589 >> v1:172.21.15.159:6789/0 conn(0x7f77b415d640 legacy=0x7f77b40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.651+0000 7f77b3400700 1 -- 172.21.15.12:0/649589 >> v1:172.21.15.12:6789/0 conn(0x7f77b40ceec0 legacy=0x7f77b41612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.651+0000 7f77b3400700 1 -- 172.21.15.12:0/649589 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b40d2df0 con 0x7f77b41619d0 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.651+0000 7f77b3400700 1 --2- 172.21.15.12:0/649589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 0x7f77b41685f0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f77ac009e50 tx=0x7f77ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11417c217f766d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.651+0000 7f77b1600700 1 -- 172.21.15.12:0/649589 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac00e070 con 0x7f77b41619d0 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.651+0000 7f77b1600700 1 -- 172.21.15.12:0/649589 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ac003a20 con 0x7f77b41619d0 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.651+0000 7f77b1600700 1 -- 172.21.15.12:0/649589 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac003dd0 con 0x7f77b41619d0 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.652+0000 7f77b9a00700 1 -- 172.21.15.12:0/649589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 msgr2=0x7f77b41685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.652+0000 7f77b9a00700 1 --2- 172.21.15.12:0/649589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 0x7f77b41685f0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f77ac009e50 tx=0x7f77ac00b330 comp rx=0 tx=0).stop 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.652+0000 7f77b9a00700 1 -- 172.21.15.12:0/649589 shutdown_connections 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.652+0000 7f77b9a00700 1 --2- 172.21.15.12:0/649589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 0x7f77b41685f0 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.652+0000 7f77b9a00700 1 -- 172.21.15.12:0/649589 >> 172.21.15.12:0/649589 conn(0x7f77b408aff0 msgr2=0x7f77b40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.652+0000 7f77b9a00700 1 -- 172.21.15.12:0/649589 shutdown_connections 2024-03-29T11:34:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.652+0000 7f77b9a00700 1 -- 172.21.15.12:0/649589 wait complete. 2024-03-29T11:34:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.654+0000 7f77b9a00700 1 Processor -- start 2024-03-29T11:34:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.654+0000 7f77b9a00700 1 -- start start 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b9a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77b40ceec0 0x7f77b41607e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b9a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77b415d640 0x7f77b4155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b9a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 0x7f77b41572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b9a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b4160d20 con 0x7f77b41619d0 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b9a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b409ecd0 con 0x7f77b40ceec0 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b9a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b409efb0 con 0x7f77b415d640 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 0x7f77b41572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 0x7f77b41572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51494/0 (socket says 172.21.15.12:51494) 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b3400700 1 -- 172.21.15.12:0/2994512856 learned_addr learned my addr 172.21.15.12:0/2994512856 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b2000700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77b415d640 0x7f77b4155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b2a00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77b40ceec0 0x7f77b41607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.655+0000 7f77b3400700 1 -- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77b415d640 msgr2=0x7f77b4155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77b3400700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77b415d640 0x7f77b4155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77b3400700 1 -- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77b40ceec0 msgr2=0x7f77b41607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77b3400700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77b40ceec0 0x7f77b41607e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77b3400700 1 -- 172.21.15.12:0/2994512856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac009b30 con 0x7f77b41619d0 2024-03-29T11:34:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77b2a00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77b40ceec0 0x7f77b41607e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77b3400700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 0x7f77b41572f0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f77b408c490 tx=0x7f77ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77aaa00700 1 -- 172.21.15.12:0/2994512856 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac004440 con 0x7f77b41619d0 2024-03-29T11:34:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77aaa00700 1 -- 172.21.15.12:0/2994512856 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ac0045a0 con 0x7f77b41619d0 2024-03-29T11:34:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77b9a00700 1 -- 172.21.15.12:0/2994512856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b409f290 con 0x7f77b41619d0 2024-03-29T11:34:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77aaa00700 1 -- 172.21.15.12:0/2994512856 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac004950 con 0x7f77b41619d0 2024-03-29T11:34:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.656+0000 7f77b9a00700 1 -- 172.21.15.12:0/2994512856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b409f850 con 0x7f77b41619d0 2024-03-29T11:34:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.657+0000 7f77b9a00700 1 -- 172.21.15.12:0/2994512856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b4067720 con 0x7f77b41619d0 2024-03-29T11:34:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.658+0000 7f77aaa00700 1 -- 172.21.15.12:0/2994512856 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f77ac025450 con 0x7f77b41619d0 2024-03-29T11:34:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.659+0000 7f77aaa00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f779406adb0 0x7f779406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.659+0000 7f77aaa00700 1 -- 172.21.15.12:0/2994512856 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f77ac08ae40 con 0x7f77b41619d0 2024-03-29T11:34:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.660+0000 7f77b2a00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f779406adb0 0x7f779406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.660+0000 7f77b2a00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f779406adb0 0x7f779406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77a4003fc0 tx=0x7f77a4009040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:23.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.664+0000 7f77aaa00700 1 -- 172.21.15.12:0/2994512856 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f77ac055dc0 con 0x7f77b41619d0 2024-03-29T11:34:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.926+0000 7f77b9a00700 1 -- 172.21.15.12:0/2994512856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77b40b5c70 con 0x7f77b41619d0 2024-03-29T11:34:23.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.927+0000 7f77aaa00700 1 -- 172.21.15.12:0/2994512856 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4747 (secure 0 0 0) 0x7f77ac0593e0 con 0x7f77b41619d0 2024-03-29T11:34:23.924 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:23.924 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":146,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12538,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/350949303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":350949303},{"type":"v1","addr":"172.21.15.159:6836","nonce":350949303}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1576953466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1576953466},{"type":"v1","addr":"172.21.15.159:6837","nonce":1576953466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/30523791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":30523791},{"type":"v1","addr":"172.21.15.12:6835","nonce":30523791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:34:19.924674+0000","modified":"2024-03-29T11:34:23.413936+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13125":{"gid":13125,"name":"a","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/2115072104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2115072104},{"type":"v1","addr":"172.21.15.12:6834","nonce":2115072104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 -- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f779406adb0 msgr2=0x7f779406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f779406adb0 0x7f779406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77a4003fc0 tx=0x7f77a4009040 comp rx=0 tx=0).stop 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 -- 172.21.15.12:0/2994512856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 msgr2=0x7f77b41572f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 0x7f77b41572f0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f77b408c490 tx=0x7f77ac004ad0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 -- 172.21.15.12:0/2994512856 shutdown_connections 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f779406adb0 0x7f779406d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77b41619d0 0x7f77b41572f0 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f77b40ceec0 0x7f77b41607e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 --2- 172.21.15.12:0/2994512856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77b415d640 0x7f77b4155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 -- 172.21.15.12:0/2994512856 >> 172.21.15.12:0/2994512856 conn(0x7f77b408aff0 msgr2=0x7f77b40b8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 -- 172.21.15.12:0/2994512856 shutdown_connections 2024-03-29T11:34:23.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:23.929+0000 7f77b9a00700 1 -- 172.21.15.12:0/2994512856 wait complete. 2024-03-29T11:34:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 146 2024-03-29T11:34:23.945 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:24.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.145+0000 7f5dc7400700 1 Processor -- start 2024-03-29T11:34:24.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.146+0000 7f5dc7400700 1 -- start start 2024-03-29T11:34:24.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.147+0000 7f5dc7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc0160f10 0x7f5dc00d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.147+0000 7f5dc7400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5dc0161590 0x7f5dc0161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.147+0000 7f5dc7400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc00ceda0 con 0x7f5dc00cf230 2024-03-29T11:34:24.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.147+0000 7f5dc7400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc0161f80 con 0x7f5dc0161590 2024-03-29T11:34:24.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.147+0000 7f5dc7400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc01612f0 con 0x7f5dc0160f10 2024-03-29T11:34:24.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.147+0000 7f5dc5400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5dc0161590 0x7f5dc0161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.147+0000 7f5dc5400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5dc0161590 0x7f5dc0161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51498/0 (socket says 172.21.15.12:51498) 2024-03-29T11:34:24.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.147+0000 7f5dc4a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc0160f10 0x7f5dc00d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.147+0000 7f5dc5400700 1 -- 172.21.15.12:0/1789572093 learned_addr learned my addr 172.21.15.12:0/1789572093 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:24.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.148+0000 7f5dc5400700 1 -- 172.21.15.12:0/1789572093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc0160f10 msgr2=0x7f5dc00d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:24.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.148+0000 7f5dc5400700 1 --2- 172.21.15.12:0/1789572093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc0160f10 0x7f5dc00d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.148+0000 7f5dc5400700 1 -- 172.21.15.12:0/1789572093 >> v1:172.21.15.159:6789/0 conn(0x7f5dc00cf230 legacy=0x7f5dc00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.148+0000 7f5dc5400700 1 -- 172.21.15.12:0/1789572093 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00ce470 con 0x7f5dc0161590 2024-03-29T11:34:24.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.148+0000 7f5dbf400700 1 -- 172.21.15.12:0/1789572093 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374848854 0 0) 0x7f5dc00ceda0 con 0x7f5dc00cf230 2024-03-29T11:34:24.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.148+0000 7f5dc5400700 1 --2- 172.21.15.12:0/1789572093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc0161590 0x7f5dc0161a40 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009e80 tx=0x7f5db800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993a3379932c7656 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.148+0000 7f5dbf400700 1 -- 172.21.15.12:0/1789572093 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db800e070 con 0x7f5dc0161590 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.148+0000 7f5dbf400700 1 -- 172.21.15.12:0/1789572093 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db800b8d0 con 0x7f5dc0161590 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.148+0000 7f5dbf400700 1 -- 172.21.15.12:0/1789572093 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db800bc80 con 0x7f5dc0161590 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.149+0000 7f5dc7400700 1 -- 172.21.15.12:0/1789572093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc0161590 msgr2=0x7f5dc0161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.149+0000 7f5dc7400700 1 --2- 172.21.15.12:0/1789572093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc0161590 0x7f5dc0161a40 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009e80 tx=0x7f5db800b330 comp rx=0 tx=0).stop 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.149+0000 7f5dc7400700 1 -- 172.21.15.12:0/1789572093 shutdown_connections 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.149+0000 7f5dc7400700 1 --2- 172.21.15.12:0/1789572093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc0161590 0x7f5dc0161a40 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.149+0000 7f5dc7400700 1 --2- 172.21.15.12:0/1789572093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc0160f10 0x7f5dc00d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.149+0000 7f5dc7400700 1 -- 172.21.15.12:0/1789572093 >> 172.21.15.12:0/1789572093 conn(0x7f5dc008aff0 msgr2=0x7f5dc00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.149+0000 7f5dc7400700 1 -- 172.21.15.12:0/1789572093 shutdown_connections 2024-03-29T11:34:24.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.149+0000 7f5dc7400700 1 -- 172.21.15.12:0/1789572093 wait complete. 2024-03-29T11:34:24.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.151+0000 7f5dc7400700 1 Processor -- start 2024-03-29T11:34:24.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.151+0000 7f5dc7400700 1 -- start start 2024-03-29T11:34:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc00cf230 0x7f5dc015d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc7400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dc0160f10 0x7f5dc009e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc7400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc0161590 0x7f5dc014e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc7400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc015e2c0 con 0x7f5dc00cf230 2024-03-29T11:34:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc7400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc009e6d0 con 0x7f5dc0160f10 2024-03-29T11:34:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc7400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0159230 con 0x7f5dc0161590 2024-03-29T11:34:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc00cf230 0x7f5dc015d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc00cf230 0x7f5dc015d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51508/0 (socket says 172.21.15.12:51508) 2024-03-29T11:34:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc4a00700 1 -- 172.21.15.12:0/2596845561 learned_addr learned my addr 172.21.15.12:0/2596845561 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc5400700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc0161590 0x7f5dc014e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dbfe00700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dc0160f10 0x7f5dc009e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc4a00700 1 -- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc0161590 msgr2=0x7f5dc014e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc4a00700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc0161590 0x7f5dc014e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc4a00700 1 -- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dc0160f10 msgr2=0x7f5dc009e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc4a00700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dc0160f10 0x7f5dc009e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dc4a00700 1 -- 172.21.15.12:0/2596845561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db8009b30 con 0x7f5dc00cf230 2024-03-29T11:34:24.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.152+0000 7f5dbfe00700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dc0160f10 0x7f5dc009e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:24.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.153+0000 7f5dc4a00700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc00cf230 0x7f5dc015d020 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f5db400e870 tx=0x7f5db400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:24.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.153+0000 7f5dbcc00700 1 -- 172.21.15.12:0/2596845561 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4009590 con 0x7f5dc00cf230 2024-03-29T11:34:24.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.153+0000 7f5dbcc00700 1 -- 172.21.15.12:0/2596845561 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db4013680 con 0x7f5dc00cf230 2024-03-29T11:34:24.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.153+0000 7f5dc7400700 1 -- 172.21.15.12:0/2596845561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc014ede0 con 0x7f5dc00cf230 2024-03-29T11:34:24.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.153+0000 7f5dbcc00700 1 -- 172.21.15.12:0/2596845561 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db40139f0 con 0x7f5dc00cf230 2024-03-29T11:34:24.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.153+0000 7f5dc7400700 1 -- 172.21.15.12:0/2596845561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc015d990 con 0x7f5dc00cf230 2024-03-29T11:34:24.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.154+0000 7f5dc7400700 1 -- 172.21.15.12:0/2596845561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc0067720 con 0x7f5dc00cf230 2024-03-29T11:34:24.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.155+0000 7f5dbcc00700 1 -- 172.21.15.12:0/2596845561 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5db4013b50 con 0x7f5dc00cf230 2024-03-29T11:34:24.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.156+0000 7f5dbcc00700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5da006adb0 0x7f5da006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.157+0000 7f5dbcc00700 1 -- 172.21.15.12:0/2596845561 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f5db40894b0 con 0x7f5dc00cf230 2024-03-29T11:34:24.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.157+0000 7f5dbfe00700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5da006adb0 0x7f5da006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.158+0000 7f5dbfe00700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5da006adb0 0x7f5da006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5dc014f150 tx=0x7f5dac009040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:24.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.161+0000 7f5dbcc00700 1 -- 172.21.15.12:0/2596845561 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5db4054490 con 0x7f5dc00cf230 2024-03-29T11:34:24.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.375+0000 7f5dc7400700 1 -- 172.21.15.12:0/2596845561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dc00d0800 con 0x7f5dc00cf230 2024-03-29T11:34:24.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.376+0000 7f5dbcc00700 1 -- 172.21.15.12:0/2596845561 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v283) v1 ==== 74+0+16175 (secure 0 0 0) 0x7f5db4057ab0 con 0x7f5dc00cf230 2024-03-29T11:34:24.373 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:24.373 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":283,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:34:20.927233+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:34:18.360241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-03-29T11:34:18.903400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"283","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:24.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 -- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5da006adb0 msgr2=0x7f5da006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5da006adb0 0x7f5da006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5dc014f150 tx=0x7f5dac009040 comp rx=0 tx=0).stop 2024-03-29T11:34:24.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 -- 172.21.15.12:0/2596845561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc00cf230 msgr2=0x7f5dc015d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc00cf230 0x7f5dc015d020 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f5db400e870 tx=0x7f5db400eb80 comp rx=0 tx=0).stop 2024-03-29T11:34:24.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 -- 172.21.15.12:0/2596845561 shutdown_connections 2024-03-29T11:34:24.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5da006adb0 0x7f5da006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc0161590 0x7f5dc014e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dc00cf230 0x7f5dc015d020 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 --2- 172.21.15.12:0/2596845561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5dc0160f10 0x7f5dc009e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 -- 172.21.15.12:0/2596845561 >> 172.21.15.12:0/2596845561 conn(0x7f5dc008aff0 msgr2=0x7f5dc00ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:24.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 -- 172.21.15.12:0/2596845561 shutdown_connections 2024-03-29T11:34:24.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.378+0000 7f5dc7400700 1 -- 172.21.15.12:0/2596845561 wait complete. 2024-03-29T11:34:24.394 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-29T11:34:24.394 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:34:24.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.595+0000 7fa4ce200700 1 Processor -- start 2024-03-29T11:34:24.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.595+0000 7fa4ce200700 1 -- start start 2024-03-29T11:34:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.596+0000 7fa4ce200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa4c80cb3f0 0x7fa4c80ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.596+0000 7fa4ce200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa4c80ac600 0x7fa4c806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.596+0000 7fa4ce200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c815fd50 con 0x7fa4c815b5d0 2024-03-29T11:34:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.596+0000 7fa4ce200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c815b9b0 con 0x7fa4c80cb3f0 2024-03-29T11:34:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.596+0000 7fa4ce200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c80aca10 con 0x7fa4c80ac600 2024-03-29T11:34:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.596+0000 7fa4c6a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa4c80cb3f0 0x7fa4c80ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.596+0000 7fa4c7400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa4c815b5d0 0x7fa4c80ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60224/0 (socket says 172.21.15.12:60224) 2024-03-29T11:34:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.596+0000 7fa4c7400700 1 -- 172.21.15.12:0/914295308 learned_addr learned my addr 172.21.15.12:0/914295308 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.597+0000 7fa4c7e00700 1 --2- 172.21.15.12:0/914295308 >> v2:172.21.15.159:3300/0 conn(0x7fa4c80ac600 0x7fa4c806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.597+0000 7fa4c6a00700 1 -- 172.21.15.12:0/914295308 >> v2:172.21.15.159:3300/0 conn(0x7fa4c80ac600 msgr2=0x7fa4c806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.597+0000 7fa4c6a00700 1 --2- 172.21.15.12:0/914295308 >> v2:172.21.15.159:3300/0 conn(0x7fa4c80ac600 0x7fa4c806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.597+0000 7fa4c6a00700 1 -- 172.21.15.12:0/914295308 >> v1:172.21.15.12:6789/0 conn(0x7fa4c815b5d0 legacy=0x7fa4c80ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.597+0000 7fa4c6a00700 1 -- 172.21.15.12:0/914295308 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c815f140 con 0x7fa4c80cb3f0 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.597+0000 7fa4c6a00700 1 --2- 172.21.15.12:0/914295308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80cb3f0 0x7fa4c80ac0c0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80cfdc0 tx=0x7fa4b4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2928cbcbc616847 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.597+0000 7fa4c6000700 1 -- 172.21.15.12:0/914295308 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4011830 con 0x7fa4c80cb3f0 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.597+0000 7fa4c6000700 1 -- 172.21.15.12:0/914295308 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b4011990 con 0x7fa4c80cb3f0 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.597+0000 7fa4c6000700 1 -- 172.21.15.12:0/914295308 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4011d20 con 0x7fa4c80cb3f0 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.598+0000 7fa4ce200700 1 -- 172.21.15.12:0/914295308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80cb3f0 msgr2=0x7fa4c80ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.598+0000 7fa4ce200700 1 --2- 172.21.15.12:0/914295308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80cb3f0 0x7fa4c80ac0c0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80cfdc0 tx=0x7fa4b4009b20 comp rx=0 tx=0).stop 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.598+0000 7fa4ce200700 1 -- 172.21.15.12:0/914295308 shutdown_connections 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.598+0000 7fa4ce200700 1 --2- 172.21.15.12:0/914295308 >> v2:172.21.15.159:3300/0 conn(0x7fa4c80ac600 0x7fa4c806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.598+0000 7fa4ce200700 1 --2- 172.21.15.12:0/914295308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80cb3f0 0x7fa4c80ac0c0 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.598+0000 7fa4ce200700 1 -- 172.21.15.12:0/914295308 >> 172.21.15.12:0/914295308 conn(0x7fa4c80cf190 msgr2=0x7fa4c80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.598+0000 7fa4ce200700 1 -- 172.21.15.12:0/914295308 shutdown_connections 2024-03-29T11:34:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.598+0000 7fa4ce200700 1 -- 172.21.15.12:0/914295308 wait complete. 2024-03-29T11:34:24.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.600+0000 7fa4ce200700 1 Processor -- start 2024-03-29T11:34:24.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.600+0000 7fa4ce200700 1 -- start start 2024-03-29T11:34:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4ce200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ac600 0x7fa4c8157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4ce200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4c815b5d0 0x7fa4c8158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4ce200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c8159030 0x7fa4c80ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4ce200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c80c8600 con 0x7fa4c80ac600 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4ce200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c80c88e0 con 0x7fa4c815b5d0 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4ce200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c80c8bc0 con 0x7fa4c8159030 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ac600 0x7fa4c8157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ac600 0x7fa4c8157000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51526/0 (socket says 172.21.15.12:51526) 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4c7400700 1 -- 172.21.15.12:0/4256955214 learned_addr learned my addr 172.21.15.12:0/4256955214 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4c6a00700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4c815b5d0 0x7fa4c8158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.601+0000 7fa4c7e00700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c8159030 0x7fa4c80ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4c7400700 1 -- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c8159030 msgr2=0x7fa4c80ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4c7400700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c8159030 0x7fa4c80ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4c7400700 1 -- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4c815b5d0 msgr2=0x7fa4c8158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4c7400700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4c815b5d0 0x7fa4c8158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4c7400700 1 -- 172.21.15.12:0/4256955214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b40094a0 con 0x7fa4c80ac600 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4c7400700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ac600 0x7fa4c8157000 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800c490 tx=0x7fa4b800c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4bf400700 1 -- 172.21.15.12:0/4256955214 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8005d70 con 0x7fa4c80ac600 2024-03-29T11:34:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4bf400700 1 -- 172.21.15.12:0/4256955214 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b80070d0 con 0x7fa4c80ac600 2024-03-29T11:34:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4ce200700 1 -- 172.21.15.12:0/4256955214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c80c8ea0 con 0x7fa4c80ac600 2024-03-29T11:34:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4bf400700 1 -- 172.21.15.12:0/4256955214 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8007460 con 0x7fa4c80ac600 2024-03-29T11:34:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.602+0000 7fa4ce200700 1 -- 172.21.15.12:0/4256955214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c80cd000 con 0x7fa4c80ac600 2024-03-29T11:34:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.603+0000 7fa4ce200700 1 -- 172.21.15.12:0/4256955214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c8067720 con 0x7fa4c80ac600 2024-03-29T11:34:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.604+0000 7fa4bf400700 1 -- 172.21.15.12:0/4256955214 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa4b80075c0 con 0x7fa4c80ac600 2024-03-29T11:34:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.606+0000 7fa4bf400700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4a806adb0 0x7fa4a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.606+0000 7fa4bf400700 1 -- 172.21.15.12:0/4256955214 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fa4b8088920 con 0x7fa4c80ac600 2024-03-29T11:34:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.606+0000 7fa4c6a00700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4a806adb0 0x7fa4a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.607+0000 7fa4c6a00700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4a806adb0 0x7fa4a806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40090b0 tx=0x7fa4b4008030 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:24.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.611+0000 7fa4bf400700 1 -- 172.21.15.12:0/4256955214 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4b8053990 con 0x7fa4c80ac600 2024-03-29T11:34:24.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.820+0000 7fa4ce200700 1 -- 172.21.15.12:0/4256955214 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4c80cbf30 con 0x7fa4a806adb0 2024-03-29T11:34:24.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.831+0000 7fa4bf400700 1 -- 172.21.15.12:0/4256955214 <== mgr.10517 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289521 (secure 0 0 0) 0x7fa4b4011cb0 con 0x7fa4a806adb0 2024-03-29T11:34:24.833 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 -- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4a806adb0 msgr2=0x7fa4a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4a806adb0 0x7fa4a806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40090b0 tx=0x7fa4b4008030 comp rx=0 tx=0).stop 2024-03-29T11:34:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 -- 172.21.15.12:0/4256955214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ac600 msgr2=0x7fa4c8157000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ac600 0x7fa4c8157000 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800c490 tx=0x7fa4b800c7a0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 -- 172.21.15.12:0/4256955214 shutdown_connections 2024-03-29T11:34:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa4a806adb0 0x7fa4a806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4c8159030 0x7fa4c80ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c80ac600 0x7fa4c8157000 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 --2- 172.21.15.12:0/4256955214 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4c815b5d0 0x7fa4c8158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 -- 172.21.15.12:0/4256955214 >> 172.21.15.12:0/4256955214 conn(0x7fa4c80cf190 msgr2=0x7fa4c815e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 -- 172.21.15.12:0/4256955214 shutdown_connections 2024-03-29T11:34:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:24.836+0000 7fa4ce200700 1 -- 172.21.15.12:0/4256955214 wait complete. 2024-03-29T11:34:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:34:24.851 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-29T11:34:22.978426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2864,"num_read_kb":6044,"num_write":775,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":470,"ondisk_log_size":470,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":483028,"kb_used_data":29060,"kb_used_omap":12,"kb_used_meta":453811,"kb_avail":754491692,"statfs":{"total":773094113280,"available":772599492608,"internally_reserved":0,"allocated":29757440,"data_stored":15234257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":464703033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.576252"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941411+0000","last_change":"2024-03-29T11:34:19.947211+0000","last_active":"2024-03-29T11:34:20.941411+0000","last_peered":"2024-03-29T11:34:20.941411+0000","last_clean":"2024-03-29T11:34:20.941411+0000","last_became_active":"2024-03-29T11:34:19.946081+0000","last_became_peered":"2024-03-29T11:34:19.946081+0000","last_unstale":"2024-03-29T11:34:20.941411+0000","last_undegraded":"2024-03-29T11:34:20.941411+0000","last_fullsized":"2024-03-29T11:34:20.941411+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:08.753840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.949433+0000","last_change":"2024-03-29T11:34:19.947268+0000","last_active":"2024-03-29T11:34:19.949433+0000","last_peered":"2024-03-29T11:34:19.949433+0000","last_clean":"2024-03-29T11:34:19.949433+0000","last_became_active":"2024-03-29T11:34:19.946610+0000","last_became_peered":"2024-03-29T11:34:19.946610+0000","last_unstale":"2024-03-29T11:34:19.949433+0000","last_undegraded":"2024-03-29T11:34:19.949433+0000","last_fullsized":"2024-03-29T11:34:19.949433+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:50:54.231741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.963132+0000","last_change":"2024-03-29T11:34:19.953262+0000","last_active":"2024-03-29T11:34:19.963132+0000","last_peered":"2024-03-29T11:34:19.963132+0000","last_clean":"2024-03-29T11:34:19.963132+0000","last_became_active":"2024-03-29T11:34:19.952083+0000","last_became_peered":"2024-03-29T11:34:19.952083+0000","last_unstale":"2024-03-29T11:34:19.963132+0000","last_undegraded":"2024-03-29T11:34:19.963132+0000","last_fullsized":"2024-03-29T11:34:19.963132+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:03:26.025761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.965533+0000","last_change":"2024-03-29T11:34:19.957594+0000","last_active":"2024-03-29T11:34:19.965533+0000","last_peered":"2024-03-29T11:34:19.965533+0000","last_clean":"2024-03-29T11:34:19.965533+0000","last_became_active":"2024-03-29T11:34:19.957137+0000","last_became_peered":"2024-03-29T11:34:19.957137+0000","last_unstale":"2024-03-29T11:34:19.965533+0000","last_undegraded":"2024-03-29T11:34:19.965533+0000","last_fullsized":"2024-03-29T11:34:19.965533+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:51:08.112025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966369+0000","last_change":"2024-03-29T11:34:19.963424+0000","last_active":"2024-03-29T11:34:19.966369+0000","last_peered":"2024-03-29T11:34:19.966369+0000","last_clean":"2024-03-29T11:34:19.966369+0000","last_became_active":"2024-03-29T11:34:19.963170+0000","last_became_peered":"2024-03-29T11:34:19.963170+0000","last_unstale":"2024-03-29T11:34:19.966369+0000","last_undegraded":"2024-03-29T11:34:19.966369+0000","last_fullsized":"2024-03-29T11:34:19.966369+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:08:12.571462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.956506+0000","last_change":"2024-03-29T11:34:19.949988+0000","last_active":"2024-03-29T11:34:19.956506+0000","last_peered":"2024-03-29T11:34:19.956506+0000","last_clean":"2024-03-29T11:34:19.956506+0000","last_became_active":"2024-03-29T11:34:19.949550+0000","last_became_peered":"2024-03-29T11:34:19.949550+0000","last_unstale":"2024-03-29T11:34:19.956506+0000","last_undegraded":"2024-03-29T11:34:19.956506+0000","last_fullsized":"2024-03-29T11:34:19.956506+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:20:24.328362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.939980+0000","last_change":"2024-03-29T11:34:19.957036+0000","last_active":"2024-03-29T11:34:20.939980+0000","last_peered":"2024-03-29T11:34:20.939980+0000","last_clean":"2024-03-29T11:34:20.939980+0000","last_became_active":"2024-03-29T11:34:19.956723+0000","last_became_peered":"2024-03-29T11:34:19.956723+0000","last_unstale":"2024-03-29T11:34:20.939980+0000","last_undegraded":"2024-03-29T11:34:20.939980+0000","last_fullsized":"2024-03-29T11:34:20.939980+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:27:25.030286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934403+0000","last_change":"2024-03-29T11:34:19.961721+0000","last_active":"2024-03-29T11:34:20.934403+0000","last_peered":"2024-03-29T11:34:20.934403+0000","last_clean":"2024-03-29T11:34:20.934403+0000","last_became_active":"2024-03-29T11:34:19.961430+0000","last_became_peered":"2024-03-29T11:34:19.961430+0000","last_unstale":"2024-03-29T11:34:20.934403+0000","last_undegraded":"2024-03-29T11:34:20.934403+0000","last_fullsized":"2024-03-29T11:34:20.934403+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:43:25.035704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.958126+0000","last_change":"2024-03-29T11:34:19.955178+0000","last_active":"2024-03-29T11:34:19.958126+0000","last_peered":"2024-03-29T11:34:19.958126+0000","last_clean":"2024-03-29T11:34:19.958126+0000","last_became_active":"2024-03-29T11:34:19.954919+0000","last_became_peered":"2024-03-29T11:34:19.954919+0000","last_unstale":"2024-03-29T11:34:19.958126+0000","last_undegraded":"2024-03-29T11:34:19.958126+0000","last_fullsized":"2024-03-29T11:34:19.958126+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:16.428268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941343+0000","last_change":"2024-03-29T11:34:19.944654+0000","last_active":"2024-03-29T11:34:20.941343+0000","last_peered":"2024-03-29T11:34:20.941343+0000","last_clean":"2024-03-29T11:34:20.941343+0000","last_became_active":"2024-03-29T11:34:19.944304+0000","last_became_peered":"2024-03-29T11:34:19.944304+0000","last_unstale":"2024-03-29T11:34:20.941343+0000","last_undegraded":"2024-03-29T11:34:20.941343+0000","last_fullsized":"2024-03-29T11:34:20.941343+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:36.332104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.935020+0000","last_change":"2024-03-29T11:34:19.958236+0000","last_active":"2024-03-29T11:34:20.935020+0000","last_peered":"2024-03-29T11:34:20.935020+0000","last_clean":"2024-03-29T11:34:20.935020+0000","last_became_active":"2024-03-29T11:34:19.956687+0000","last_became_peered":"2024-03-29T11:34:19.956687+0000","last_unstale":"2024-03-29T11:34:20.935020+0000","last_undegraded":"2024-03-29T11:34:20.935020+0000","last_fullsized":"2024-03-29T11:34:20.935020+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:04:30.046786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.959520+0000","last_change":"2024-03-29T11:34:19.958532+0000","last_active":"2024-03-29T11:34:19.959520+0000","last_peered":"2024-03-29T11:34:19.959520+0000","last_clean":"2024-03-29T11:34:19.959520+0000","last_became_active":"2024-03-29T11:34:19.958273+0000","last_became_peered":"2024-03-29T11:34:19.958273+0000","last_unstale":"2024-03-29T11:34:19.959520+0000","last_undegraded":"2024-03-29T11:34:19.959520+0000","last_fullsized":"2024-03-29T11:34:19.959520+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:01:56.645123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940743+0000","last_change":"2024-03-29T11:34:19.951235+0000","last_active":"2024-03-29T11:34:20.940743+0000","last_peered":"2024-03-29T11:34:20.940743+0000","last_clean":"2024-03-29T11:34:20.940743+0000","last_became_active":"2024-03-29T11:34:19.950916+0000","last_became_peered":"2024-03-29T11:34:19.950916+0000","last_unstale":"2024-03-29T11:34:20.940743+0000","last_undegraded":"2024-03-29T11:34:20.940743+0000","last_fullsized":"2024-03-29T11:34:20.940743+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:01:02.478752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951883+0000","last_change":"2024-03-29T11:34:19.946766+0000","last_active":"2024-03-29T11:34:19.951883+0000","last_peered":"2024-03-29T11:34:19.951883+0000","last_clean":"2024-03-29T11:34:19.951883+0000","last_became_active":"2024-03-29T11:34:19.942919+0000","last_became_peered":"2024-03-29T11:34:19.942919+0000","last_unstale":"2024-03-29T11:34:19.951883+0000","last_undegraded":"2024-03-29T11:34:19.951883+0000","last_fullsized":"2024-03-29T11:34:19.951883+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:34.599290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966295+0000","last_change":"2024-03-29T11:34:19.961549+0000","last_active":"2024-03-29T11:34:19.966295+0000","last_peered":"2024-03-29T11:34:19.966295+0000","last_clean":"2024-03-29T11:34:19.966295+0000","last_became_active":"2024-03-29T11:34:19.961039+0000","last_became_peered":"2024-03-29T11:34:19.961039+0000","last_unstale":"2024-03-29T11:34:19.966295+0000","last_undegraded":"2024-03-29T11:34:19.966295+0000","last_fullsized":"2024-03-29T11:34:19.966295+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:01:02.478752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951834+0000","last_change":"2024-03-29T11:34:19.944494+0000","last_active":"2024-03-29T11:34:19.951834+0000","last_peered":"2024-03-29T11:34:19.951834+0000","last_clean":"2024-03-29T11:34:19.951834+0000","last_became_active":"2024-03-29T11:34:19.944207+0000","last_became_peered":"2024-03-29T11:34:19.944207+0000","last_unstale":"2024-03-29T11:34:19.951834+0000","last_undegraded":"2024-03-29T11:34:19.951834+0000","last_fullsized":"2024-03-29T11:34:19.951834+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:01:04.821333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940953+0000","last_change":"2024-03-29T11:34:19.958166+0000","last_active":"2024-03-29T11:34:20.940953+0000","last_peered":"2024-03-29T11:34:20.940953+0000","last_clean":"2024-03-29T11:34:20.940953+0000","last_became_active":"2024-03-29T11:34:19.956429+0000","last_became_peered":"2024-03-29T11:34:19.956429+0000","last_unstale":"2024-03-29T11:34:20.940953+0000","last_undegraded":"2024-03-29T11:34:20.940953+0000","last_fullsized":"2024-03-29T11:34:20.940953+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:03.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940933+0000","last_change":"2024-03-29T11:34:19.944065+0000","last_active":"2024-03-29T11:34:20.940933+0000","last_peered":"2024-03-29T11:34:20.940933+0000","last_clean":"2024-03-29T11:34:20.940933+0000","last_became_active":"2024-03-29T11:34:19.943800+0000","last_became_peered":"2024-03-29T11:34:19.943800+0000","last_unstale":"2024-03-29T11:34:20.940933+0000","last_undegraded":"2024-03-29T11:34:20.940933+0000","last_fullsized":"2024-03-29T11:34:20.940933+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:41.492915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952415+0000","last_change":"2024-03-29T11:34:19.948092+0000","last_active":"2024-03-29T11:34:19.952415+0000","last_peered":"2024-03-29T11:34:19.952415+0000","last_clean":"2024-03-29T11:34:19.952415+0000","last_became_active":"2024-03-29T11:34:19.947764+0000","last_became_peered":"2024-03-29T11:34:19.947764+0000","last_unstale":"2024-03-29T11:34:19.952415+0000","last_undegraded":"2024-03-29T11:34:19.952415+0000","last_fullsized":"2024-03-29T11:34:19.952415+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:45:56.716334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.960108+0000","last_change":"2024-03-29T11:34:19.959273+0000","last_active":"2024-03-29T11:34:19.960108+0000","last_peered":"2024-03-29T11:34:19.960108+0000","last_clean":"2024-03-29T11:34:19.960108+0000","last_became_active":"2024-03-29T11:34:19.957026+0000","last_became_peered":"2024-03-29T11:34:19.957026+0000","last_unstale":"2024-03-29T11:34:19.960108+0000","last_undegraded":"2024-03-29T11:34:19.960108+0000","last_fullsized":"2024-03-29T11:34:19.960108+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:42.183164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934797+0000","last_change":"2024-03-29T11:34:19.949953+0000","last_active":"2024-03-29T11:34:20.934797+0000","last_peered":"2024-03-29T11:34:20.934797+0000","last_clean":"2024-03-29T11:34:20.934797+0000","last_became_active":"2024-03-29T11:34:19.949671+0000","last_became_peered":"2024-03-29T11:34:19.949671+0000","last_unstale":"2024-03-29T11:34:20.934797+0000","last_undegraded":"2024-03-29T11:34:20.934797+0000","last_fullsized":"2024-03-29T11:34:20.934797+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:44.804649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.965574+0000","last_change":"2024-03-29T11:34:19.960624+0000","last_active":"2024-03-29T11:34:19.965574+0000","last_peered":"2024-03-29T11:34:19.965574+0000","last_clean":"2024-03-29T11:34:19.965574+0000","last_became_active":"2024-03-29T11:34:19.958146+0000","last_became_peered":"2024-03-29T11:34:19.958146+0000","last_unstale":"2024-03-29T11:34:19.965574+0000","last_undegraded":"2024-03-29T11:34:19.965574+0000","last_fullsized":"2024-03-29T11:34:19.965574+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:20:24.328362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940495+0000","last_change":"2024-03-29T11:34:19.940339+0000","last_active":"2024-03-29T11:34:20.940495+0000","last_peered":"2024-03-29T11:34:20.940495+0000","last_clean":"2024-03-29T11:34:20.940495+0000","last_became_active":"2024-03-29T11:34:19.939546+0000","last_became_peered":"2024-03-29T11:34:19.939546+0000","last_unstale":"2024-03-29T11:34:20.940495+0000","last_undegraded":"2024-03-29T11:34:20.940495+0000","last_fullsized":"2024-03-29T11:34:20.940495+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:08:12.571462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.953889+0000","last_change":"2024-03-29T11:34:19.950546+0000","last_active":"2024-03-29T11:34:19.953889+0000","last_peered":"2024-03-29T11:34:19.953889+0000","last_clean":"2024-03-29T11:34:19.953889+0000","last_became_active":"2024-03-29T11:34:19.950027+0000","last_became_peered":"2024-03-29T11:34:19.950027+0000","last_unstale":"2024-03-29T11:34:19.953889+0000","last_undegraded":"2024-03-29T11:34:19.953889+0000","last_fullsized":"2024-03-29T11:34:19.953889+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:55.397955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941047+0000","last_change":"2024-03-29T11:34:19.948467+0000","last_active":"2024-03-29T11:34:20.941047+0000","last_peered":"2024-03-29T11:34:20.941047+0000","last_clean":"2024-03-29T11:34:20.941047+0000","last_became_active":"2024-03-29T11:34:19.947021+0000","last_became_peered":"2024-03-29T11:34:19.947021+0000","last_unstale":"2024-03-29T11:34:20.941047+0000","last_undegraded":"2024-03-29T11:34:20.941047+0000","last_fullsized":"2024-03-29T11:34:20.941047+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:18:34.673238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941389+0000","last_change":"2024-03-29T11:34:19.950495+0000","last_active":"2024-03-29T11:34:20.941389+0000","last_peered":"2024-03-29T11:34:20.941389+0000","last_clean":"2024-03-29T11:34:20.941389+0000","last_became_active":"2024-03-29T11:34:19.950222+0000","last_became_peered":"2024-03-29T11:34:19.950222+0000","last_unstale":"2024-03-29T11:34:20.941389+0000","last_undegraded":"2024-03-29T11:34:20.941389+0000","last_fullsized":"2024-03-29T11:34:20.941389+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:34:17.322355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.954476+0000","last_change":"2024-03-29T11:34:19.949613+0000","last_active":"2024-03-29T11:34:19.954476+0000","last_peered":"2024-03-29T11:34:19.954476+0000","last_clean":"2024-03-29T11:34:19.954476+0000","last_became_active":"2024-03-29T11:34:19.949347+0000","last_became_peered":"2024-03-29T11:34:19.949347+0000","last_unstale":"2024-03-29T11:34:19.954476+0000","last_undegraded":"2024-03-29T11:34:19.954476+0000","last_fullsized":"2024-03-29T11:34:19.954476+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:05:16.963592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.959036+0000","last_change":"2024-03-29T11:34:19.956011+0000","last_active":"2024-03-29T11:34:19.959036+0000","last_peered":"2024-03-29T11:34:19.959036+0000","last_clean":"2024-03-29T11:34:19.959036+0000","last_became_active":"2024-03-29T11:34:19.955710+0000","last_became_peered":"2024-03-29T11:34:19.955710+0000","last_unstale":"2024-03-29T11:34:19.959036+0000","last_undegraded":"2024-03-29T11:34:19.959036+0000","last_fullsized":"2024-03-29T11:34:19.959036+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:39:11.647026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940353+0000","last_change":"2024-03-29T11:34:19.952876+0000","last_active":"2024-03-29T11:34:20.940353+0000","last_peered":"2024-03-29T11:34:20.940353+0000","last_clean":"2024-03-29T11:34:20.940353+0000","last_became_active":"2024-03-29T11:34:19.952595+0000","last_became_peered":"2024-03-29T11:34:19.952595+0000","last_unstale":"2024-03-29T11:34:20.940353+0000","last_undegraded":"2024-03-29T11:34:20.940353+0000","last_fullsized":"2024-03-29T11:34:20.940353+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:22:56.694591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.953144+0000","last_change":"2024-03-29T11:34:19.948122+0000","last_active":"2024-03-29T11:34:19.953144+0000","last_peered":"2024-03-29T11:34:19.953144+0000","last_clean":"2024-03-29T11:34:19.953144+0000","last_became_active":"2024-03-29T11:34:19.947828+0000","last_became_peered":"2024-03-29T11:34:19.947828+0000","last_unstale":"2024-03-29T11:34:19.953144+0000","last_undegraded":"2024-03-29T11:34:19.953144+0000","last_fullsized":"2024-03-29T11:34:19.953144+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:29.853902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.965752+0000","last_change":"2024-03-29T11:34:19.960388+0000","last_active":"2024-03-29T11:34:19.965752+0000","last_peered":"2024-03-29T11:34:19.965752+0000","last_clean":"2024-03-29T11:34:19.965752+0000","last_became_active":"2024-03-29T11:34:19.957211+0000","last_became_peered":"2024-03-29T11:34:19.957211+0000","last_unstale":"2024-03-29T11:34:19.965752+0000","last_undegraded":"2024-03-29T11:34:19.965752+0000","last_fullsized":"2024-03-29T11:34:19.965752+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:03.223791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.953105+0000","last_change":"2024-03-29T11:34:19.948713+0000","last_active":"2024-03-29T11:34:19.953105+0000","last_peered":"2024-03-29T11:34:19.953105+0000","last_clean":"2024-03-29T11:34:19.953105+0000","last_became_active":"2024-03-29T11:34:19.948434+0000","last_became_peered":"2024-03-29T11:34:19.948434+0000","last_unstale":"2024-03-29T11:34:19.953105+0000","last_undegraded":"2024-03-29T11:34:19.953105+0000","last_fullsized":"2024-03-29T11:34:19.953105+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:50:19.403032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"282'4","reported_seq":14,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952910+0000","last_change":"2024-03-29T11:34:19.431952+0000","last_active":"2024-03-29T11:34:19.952910+0000","last_peered":"2024-03-29T11:34:19.952910+0000","last_clean":"2024-03-29T11:34:19.952910+0000","last_became_active":"2024-03-29T11:34:19.431642+0000","last_became_peered":"2024-03-29T11:34:19.431642+0000","last_unstale":"2024-03-29T11:34:19.952910+0000","last_undegraded":"2024-03-29T11:34:19.952910+0000","last_fullsized":"2024-03-29T11:34:19.952910+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:29:32.163286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.954515+0000","last_change":"2024-03-29T11:34:19.950486+0000","last_active":"2024-03-29T11:34:19.954515+0000","last_peered":"2024-03-29T11:34:19.954515+0000","last_clean":"2024-03-29T11:34:19.954515+0000","last_became_active":"2024-03-29T11:34:19.950168+0000","last_became_peered":"2024-03-29T11:34:19.950168+0000","last_unstale":"2024-03-29T11:34:19.954515+0000","last_undegraded":"2024-03-29T11:34:19.954515+0000","last_fullsized":"2024-03-29T11:34:19.954515+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:26.561837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"282'1","reported_seq":11,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.958736+0000","last_change":"2024-03-29T11:34:19.432996+0000","last_active":"2024-03-29T11:34:19.958736+0000","last_peered":"2024-03-29T11:34:19.958736+0000","last_clean":"2024-03-29T11:34:19.958736+0000","last_became_active":"2024-03-29T11:34:19.432616+0000","last_became_peered":"2024-03-29T11:34:19.432616+0000","last_unstale":"2024-03-29T11:34:19.958736+0000","last_undegraded":"2024-03-29T11:34:19.958736+0000","last_fullsized":"2024-03-29T11:34:19.958736+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:25.549951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.954241+0000","last_change":"2024-03-29T11:34:19.946817+0000","last_active":"2024-03-29T11:34:19.954241+0000","last_peered":"2024-03-29T11:34:19.954241+0000","last_clean":"2024-03-29T11:34:19.954241+0000","last_became_active":"2024-03-29T11:34:19.946581+0000","last_became_peered":"2024-03-29T11:34:19.946581+0000","last_unstale":"2024-03-29T11:34:19.954241+0000","last_undegraded":"2024-03-29T11:34:19.954241+0000","last_fullsized":"2024-03-29T11:34:19.954241+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:14:15.546608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"282'2","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.957681+0000","last_change":"2024-03-29T11:34:19.427717+0000","last_active":"2024-03-29T11:34:19.957681+0000","last_peered":"2024-03-29T11:34:19.957681+0000","last_clean":"2024-03-29T11:34:19.957681+0000","last_became_active":"2024-03-29T11:34:19.427476+0000","last_became_peered":"2024-03-29T11:34:19.427476+0000","last_unstale":"2024-03-29T11:34:19.957681+0000","last_undegraded":"2024-03-29T11:34:19.957681+0000","last_fullsized":"2024-03-29T11:34:19.957681+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:11:23.474692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934976+0000","last_change":"2024-03-29T11:34:19.956656+0000","last_active":"2024-03-29T11:34:20.934976+0000","last_peered":"2024-03-29T11:34:20.934976+0000","last_clean":"2024-03-29T11:34:20.934976+0000","last_became_active":"2024-03-29T11:34:19.955318+0000","last_became_peered":"2024-03-29T11:34:19.955318+0000","last_unstale":"2024-03-29T11:34:20.934976+0000","last_undegraded":"2024-03-29T11:34:20.934976+0000","last_fullsized":"2024-03-29T11:34:20.934976+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:26.561837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"282'2","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.963389+0000","last_change":"2024-03-29T11:34:19.429445+0000","last_active":"2024-03-29T11:34:19.963389+0000","last_peered":"2024-03-29T11:34:19.963389+0000","last_clean":"2024-03-29T11:34:19.963389+0000","last_became_active":"2024-03-29T11:34:19.427486+0000","last_became_peered":"2024-03-29T11:34:19.427486+0000","last_unstale":"2024-03-29T11:34:19.963389+0000","last_undegraded":"2024-03-29T11:34:19.963389+0000","last_fullsized":"2024-03-29T11:34:19.963389+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:04:29.034899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.935094+0000","last_change":"2024-03-29T11:34:19.959920+0000","last_active":"2024-03-29T11:34:20.935094+0000","last_peered":"2024-03-29T11:34:20.935094+0000","last_clean":"2024-03-29T11:34:20.935094+0000","last_became_active":"2024-03-29T11:34:19.958971+0000","last_became_peered":"2024-03-29T11:34:19.958971+0000","last_unstale":"2024-03-29T11:34:20.935094+0000","last_undegraded":"2024-03-29T11:34:20.935094+0000","last_fullsized":"2024-03-29T11:34:20.935094+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:15:57.794980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"282'1","reported_seq":11,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.956845+0000","last_change":"2024-03-29T11:34:19.429640+0000","last_active":"2024-03-29T11:34:19.956845+0000","last_peered":"2024-03-29T11:34:19.956845+0000","last_clean":"2024-03-29T11:34:19.956845+0000","last_became_active":"2024-03-29T11:34:19.429417+0000","last_became_peered":"2024-03-29T11:34:19.429417+0000","last_unstale":"2024-03-29T11:34:19.956845+0000","last_undegraded":"2024-03-29T11:34:19.956845+0000","last_fullsized":"2024-03-29T11:34:19.956845+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:33.587404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951198+0000","last_change":"2024-03-29T11:34:19.947653+0000","last_active":"2024-03-29T11:34:19.951198+0000","last_peered":"2024-03-29T11:34:19.951198+0000","last_clean":"2024-03-29T11:34:19.951198+0000","last_became_active":"2024-03-29T11:34:19.947279+0000","last_became_peered":"2024-03-29T11:34:19.947279+0000","last_unstale":"2024-03-29T11:34:19.951198+0000","last_undegraded":"2024-03-29T11:34:19.951198+0000","last_fullsized":"2024-03-29T11:34:19.951198+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:08:57.572725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940458+0000","last_change":"2024-03-29T11:34:19.427504+0000","last_active":"2024-03-29T11:34:20.940458+0000","last_peered":"2024-03-29T11:34:20.940458+0000","last_clean":"2024-03-29T11:34:20.940458+0000","last_became_active":"2024-03-29T11:34:19.426820+0000","last_became_peered":"2024-03-29T11:34:19.426820+0000","last_unstale":"2024-03-29T11:34:20.940458+0000","last_undegraded":"2024-03-29T11:34:20.940458+0000","last_fullsized":"2024-03-29T11:34:20.940458+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:06:37.241290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940656+0000","last_change":"2024-03-29T11:34:19.954850+0000","last_active":"2024-03-29T11:34:20.940656+0000","last_peered":"2024-03-29T11:34:20.940656+0000","last_clean":"2024-03-29T11:34:20.940656+0000","last_became_active":"2024-03-29T11:34:19.953319+0000","last_became_peered":"2024-03-29T11:34:19.953319+0000","last_unstale":"2024-03-29T11:34:20.940656+0000","last_undegraded":"2024-03-29T11:34:20.940656+0000","last_fullsized":"2024-03-29T11:34:20.940656+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:32:52.022464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.941138+0000","last_change":"2024-03-29T11:34:19.428401+0000","last_active":"2024-03-29T11:34:19.941138+0000","last_peered":"2024-03-29T11:34:19.941138+0000","last_clean":"2024-03-29T11:34:19.941138+0000","last_became_active":"2024-03-29T11:34:19.428164+0000","last_became_peered":"2024-03-29T11:34:19.428164+0000","last_unstale":"2024-03-29T11:34:19.941138+0000","last_undegraded":"2024-03-29T11:34:19.941138+0000","last_fullsized":"2024-03-29T11:34:19.941138+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:45:55.704448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951969+0000","last_change":"2024-03-29T11:34:19.946373+0000","last_active":"2024-03-29T11:34:19.951969+0000","last_peered":"2024-03-29T11:34:19.951969+0000","last_clean":"2024-03-29T11:34:19.951969+0000","last_became_active":"2024-03-29T11:34:19.946072+0000","last_became_peered":"2024-03-29T11:34:19.946072+0000","last_unstale":"2024-03-29T11:34:19.951969+0000","last_undegraded":"2024-03-29T11:34:19.951969+0000","last_fullsized":"2024-03-29T11:34:19.951969+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:03.223791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.929819+0000","last_change":"2024-03-29T11:34:19.426889+0000","last_active":"2024-03-29T11:34:19.929819+0000","last_peered":"2024-03-29T11:34:19.929819+0000","last_clean":"2024-03-29T11:34:19.929819+0000","last_became_active":"2024-03-29T11:34:19.426616+0000","last_became_peered":"2024-03-29T11:34:19.426616+0000","last_unstale":"2024-03-29T11:34:19.929819+0000","last_undegraded":"2024-03-29T11:34:19.929819+0000","last_fullsized":"2024-03-29T11:34:19.929819+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:21.911782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.945362+0000","last_change":"2024-03-29T11:34:19.943054+0000","last_active":"2024-03-29T11:34:19.945362+0000","last_peered":"2024-03-29T11:34:19.945362+0000","last_clean":"2024-03-29T11:34:19.945362+0000","last_became_active":"2024-03-29T11:34:19.942728+0000","last_became_peered":"2024-03-29T11:34:19.942728+0000","last_unstale":"2024-03-29T11:34:19.945362+0000","last_undegraded":"2024-03-29T11:34:19.945362+0000","last_fullsized":"2024-03-29T11:34:19.945362+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:01:56.645123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.936914+0000","last_change":"2024-03-29T11:34:19.428484+0000","last_active":"2024-03-29T11:34:19.936914+0000","last_peered":"2024-03-29T11:34:19.936914+0000","last_clean":"2024-03-29T11:34:19.936914+0000","last_became_active":"2024-03-29T11:34:19.427975+0000","last_became_peered":"2024-03-29T11:34:19.427975+0000","last_unstale":"2024-03-29T11:34:19.936914+0000","last_undegraded":"2024-03-29T11:34:19.936914+0000","last_fullsized":"2024-03-29T11:34:19.936914+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:56:10.313615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952850+0000","last_change":"2024-03-29T11:34:19.946891+0000","last_active":"2024-03-29T11:34:19.952850+0000","last_peered":"2024-03-29T11:34:19.952850+0000","last_clean":"2024-03-29T11:34:19.952850+0000","last_became_active":"2024-03-29T11:34:19.946557+0000","last_became_peered":"2024-03-29T11:34:19.946557+0000","last_unstale":"2024-03-29T11:34:19.952850+0000","last_undegraded":"2024-03-29T11:34:19.952850+0000","last_fullsized":"2024-03-29T11:34:19.952850+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:47.091696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.935838+0000","last_change":"2024-03-29T11:34:19.428419+0000","last_active":"2024-03-29T11:34:19.935838+0000","last_peered":"2024-03-29T11:34:19.935838+0000","last_clean":"2024-03-29T11:34:19.935838+0000","last_became_active":"2024-03-29T11:34:19.428100+0000","last_became_peered":"2024-03-29T11:34:19.428100+0000","last_unstale":"2024-03-29T11:34:19.935838+0000","last_undegraded":"2024-03-29T11:34:19.935838+0000","last_fullsized":"2024-03-29T11:34:19.935838+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:35:10.209538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966442+0000","last_change":"2024-03-29T11:34:19.964236+0000","last_active":"2024-03-29T11:34:19.966442+0000","last_peered":"2024-03-29T11:34:19.966442+0000","last_clean":"2024-03-29T11:34:19.966442+0000","last_became_active":"2024-03-29T11:34:19.961413+0000","last_became_peered":"2024-03-29T11:34:19.961413+0000","last_unstale":"2024-03-29T11:34:19.966442+0000","last_undegraded":"2024-03-29T11:34:19.966442+0000","last_fullsized":"2024-03-29T11:34:19.966442+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:23:05.855510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"282'3","reported_seq":13,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.963330+0000","last_change":"2024-03-29T11:34:19.429938+0000","last_active":"2024-03-29T11:34:19.963330+0000","last_peered":"2024-03-29T11:34:19.963330+0000","last_clean":"2024-03-29T11:34:19.963330+0000","last_became_active":"2024-03-29T11:34:19.429114+0000","last_became_peered":"2024-03-29T11:34:19.429114+0000","last_unstale":"2024-03-29T11:34:19.963330+0000","last_undegraded":"2024-03-29T11:34:19.963330+0000","last_fullsized":"2024-03-29T11:34:19.963330+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:51:50.580234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952367+0000","last_change":"2024-03-29T11:34:19.946033+0000","last_active":"2024-03-29T11:34:19.952367+0000","last_peered":"2024-03-29T11:34:19.952367+0000","last_clean":"2024-03-29T11:34:19.952367+0000","last_became_active":"2024-03-29T11:34:19.945746+0000","last_became_peered":"2024-03-29T11:34:19.945746+0000","last_unstale":"2024-03-29T11:34:19.952367+0000","last_undegraded":"2024-03-29T11:34:19.952367+0000","last_fullsized":"2024-03-29T11:34:19.952367+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:23:05.855510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"282'6","reported_seq":17,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940160+0000","last_change":"2024-03-29T11:34:19.426573+0000","last_active":"2024-03-29T11:34:20.940160+0000","last_peered":"2024-03-29T11:34:20.940160+0000","last_clean":"2024-03-29T11:34:20.940160+0000","last_became_active":"2024-03-29T11:34:19.426257+0000","last_became_peered":"2024-03-29T11:34:19.426257+0000","last_unstale":"2024-03-29T11:34:20.940160+0000","last_undegraded":"2024-03-29T11:34:20.940160+0000","last_fullsized":"2024-03-29T11:34:20.940160+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:01:03.809447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951863+0000","last_change":"2024-03-29T11:34:19.943204+0000","last_active":"2024-03-29T11:34:19.951863+0000","last_peered":"2024-03-29T11:34:19.951863+0000","last_clean":"2024-03-29T11:34:19.951863+0000","last_became_active":"2024-03-29T11:34:19.942294+0000","last_became_peered":"2024-03-29T11:34:19.942294+0000","last_unstale":"2024-03-29T11:34:19.951863+0000","last_undegraded":"2024-03-29T11:34:19.951863+0000","last_fullsized":"2024-03-29T11:34:19.951863+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:51:51.592120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934915+0000","last_change":"2024-03-29T11:34:19.434932+0000","last_active":"2024-03-29T11:34:20.934915+0000","last_peered":"2024-03-29T11:34:20.934915+0000","last_clean":"2024-03-29T11:34:20.934915+0000","last_became_active":"2024-03-29T11:34:19.434640+0000","last_became_peered":"2024-03-29T11:34:19.434640+0000","last_unstale":"2024-03-29T11:34:20.934915+0000","last_undegraded":"2024-03-29T11:34:20.934915+0000","last_fullsized":"2024-03-29T11:34:20.934915+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:39.490318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940693+0000","last_change":"2024-03-29T11:34:19.950379+0000","last_active":"2024-03-29T11:34:20.940693+0000","last_peered":"2024-03-29T11:34:20.940693+0000","last_clean":"2024-03-29T11:34:20.940693+0000","last_became_active":"2024-03-29T11:34:19.948304+0000","last_became_peered":"2024-03-29T11:34:19.948304+0000","last_unstale":"2024-03-29T11:34:20.940693+0000","last_undegraded":"2024-03-29T11:34:20.940693+0000","last_fullsized":"2024-03-29T11:34:20.940693+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:02:35.771545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.935906+0000","last_change":"2024-03-29T11:34:19.433087+0000","last_active":"2024-03-29T11:34:19.935906+0000","last_peered":"2024-03-29T11:34:19.935906+0000","last_clean":"2024-03-29T11:34:19.935906+0000","last_became_active":"2024-03-29T11:34:19.432780+0000","last_became_peered":"2024-03-29T11:34:19.432780+0000","last_unstale":"2024-03-29T11:34:19.935906+0000","last_undegraded":"2024-03-29T11:34:19.935906+0000","last_fullsized":"2024-03-29T11:34:19.935906+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:11:23.474692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.960348+0000","last_change":"2024-03-29T11:34:19.959501+0000","last_active":"2024-03-29T11:34:19.960348+0000","last_peered":"2024-03-29T11:34:19.960348+0000","last_clean":"2024-03-29T11:34:19.960348+0000","last_became_active":"2024-03-29T11:34:19.959137+0000","last_became_peered":"2024-03-29T11:34:19.959137+0000","last_unstale":"2024-03-29T11:34:19.960348+0000","last_undegraded":"2024-03-29T11:34:19.960348+0000","last_fullsized":"2024-03-29T11:34:19.960348+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:06:38.253176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940559+0000","last_change":"2024-03-29T11:34:19.422959+0000","last_active":"2024-03-29T11:34:20.940559+0000","last_peered":"2024-03-29T11:34:20.940559+0000","last_clean":"2024-03-29T11:34:20.940559+0000","last_became_active":"2024-03-29T11:34:19.422665+0000","last_became_peered":"2024-03-29T11:34:19.422665+0000","last_unstale":"2024-03-29T11:34:20.940559+0000","last_undegraded":"2024-03-29T11:34:20.940559+0000","last_fullsized":"2024-03-29T11:34:20.940559+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:05:15.951706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940088+0000","last_change":"2024-03-29T11:34:19.939914+0000","last_active":"2024-03-29T11:34:20.940088+0000","last_peered":"2024-03-29T11:34:20.940088+0000","last_clean":"2024-03-29T11:34:20.940088+0000","last_became_active":"2024-03-29T11:34:19.939187+0000","last_became_peered":"2024-03-29T11:34:19.939187+0000","last_unstale":"2024-03-29T11:34:20.940088+0000","last_undegraded":"2024-03-29T11:34:20.940088+0000","last_fullsized":"2024-03-29T11:34:20.940088+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:41.278594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"282'8","reported_seq":18,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966527+0000","last_change":"2024-03-29T11:34:19.432536+0000","last_active":"2024-03-29T11:34:19.966527+0000","last_peered":"2024-03-29T11:34:19.966527+0000","last_clean":"2024-03-29T11:34:19.966527+0000","last_became_active":"2024-03-29T11:34:19.432074+0000","last_became_peered":"2024-03-29T11:34:19.432074+0000","last_unstale":"2024-03-29T11:34:19.966527+0000","last_undegraded":"2024-03-29T11:34:19.966527+0000","last_fullsized":"2024-03-29T11:34:19.966527+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:21.911782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.957538+0000","last_change":"2024-03-29T11:34:19.952465+0000","last_active":"2024-03-29T11:34:19.957538+0000","last_peered":"2024-03-29T11:34:19.957538+0000","last_clean":"2024-03-29T11:34:19.957538+0000","last_became_active":"2024-03-29T11:34:19.951484+0000","last_became_peered":"2024-03-29T11:34:19.951484+0000","last_unstale":"2024-03-29T11:34:19.957538+0000","last_undegraded":"2024-03-29T11:34:19.957538+0000","last_fullsized":"2024-03-29T11:34:19.957538+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:41.492915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934837+0000","last_change":"2024-03-29T11:34:19.428942+0000","last_active":"2024-03-29T11:34:20.934837+0000","last_peered":"2024-03-29T11:34:20.934837+0000","last_clean":"2024-03-29T11:34:20.934837+0000","last_became_active":"2024-03-29T11:34:19.428635+0000","last_became_peered":"2024-03-29T11:34:19.428635+0000","last_unstale":"2024-03-29T11:34:20.934837+0000","last_undegraded":"2024-03-29T11:34:20.934837+0000","last_fullsized":"2024-03-29T11:34:20.934837+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:46.079810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.954963+0000","last_change":"2024-03-29T11:34:19.950578+0000","last_active":"2024-03-29T11:34:19.954963+0000","last_peered":"2024-03-29T11:34:19.954963+0000","last_clean":"2024-03-29T11:34:19.954963+0000","last_became_active":"2024-03-29T11:34:19.949954+0000","last_became_peered":"2024-03-29T11:34:19.949954+0000","last_unstale":"2024-03-29T11:34:19.954963+0000","last_undegraded":"2024-03-29T11:34:19.954963+0000","last_fullsized":"2024-03-29T11:34:19.954963+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:11:24.475618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"282'2","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951861+0000","last_change":"2024-03-29T11:34:19.433823+0000","last_active":"2024-03-29T11:34:19.951861+0000","last_peered":"2024-03-29T11:34:19.951861+0000","last_clean":"2024-03-29T11:34:19.951861+0000","last_became_active":"2024-03-29T11:34:19.433451+0000","last_became_peered":"2024-03-29T11:34:19.433451+0000","last_unstale":"2024-03-29T11:34:19.951861+0000","last_undegraded":"2024-03-29T11:34:19.951861+0000","last_fullsized":"2024-03-29T11:34:19.951861+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:19:25.412060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.958958+0000","last_change":"2024-03-29T11:34:19.955022+0000","last_active":"2024-03-29T11:34:19.958958+0000","last_peered":"2024-03-29T11:34:19.958958+0000","last_clean":"2024-03-29T11:34:19.958958+0000","last_became_active":"2024-03-29T11:34:19.952831+0000","last_became_peered":"2024-03-29T11:34:19.952831+0000","last_unstale":"2024-03-29T11:34:19.958958+0000","last_undegraded":"2024-03-29T11:34:19.958958+0000","last_fullsized":"2024-03-29T11:34:19.958958+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:51:08.112025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"282'1","reported_seq":11,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.959866+0000","last_change":"2024-03-29T11:34:19.426275+0000","last_active":"2024-03-29T11:34:19.959866+0000","last_peered":"2024-03-29T11:34:19.959866+0000","last_clean":"2024-03-29T11:34:19.959866+0000","last_became_active":"2024-03-29T11:34:19.423168+0000","last_became_peered":"2024-03-29T11:34:19.423168+0000","last_unstale":"2024-03-29T11:34:19.959866+0000","last_undegraded":"2024-03-29T11:34:19.959866+0000","last_fullsized":"2024-03-29T11:34:19.959866+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:43.792763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952391+0000","last_change":"2024-03-29T11:34:19.947805+0000","last_active":"2024-03-29T11:34:19.952391+0000","last_peered":"2024-03-29T11:34:19.952391+0000","last_clean":"2024-03-29T11:34:19.952391+0000","last_became_active":"2024-03-29T11:34:19.947415+0000","last_became_peered":"2024-03-29T11:34:19.947415+0000","last_unstale":"2024-03-29T11:34:19.952391+0000","last_undegraded":"2024-03-29T11:34:19.952391+0000","last_fullsized":"2024-03-29T11:34:19.952391+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:22.923669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.931629+0000","last_change":"2024-03-29T11:34:19.425570+0000","last_active":"2024-03-29T11:34:19.931629+0000","last_peered":"2024-03-29T11:34:19.931629+0000","last_clean":"2024-03-29T11:34:19.931629+0000","last_became_active":"2024-03-29T11:34:19.425286+0000","last_became_peered":"2024-03-29T11:34:19.425286+0000","last_unstale":"2024-03-29T11:34:19.931629+0000","last_undegraded":"2024-03-29T11:34:19.931629+0000","last_fullsized":"2024-03-29T11:34:19.931629+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:04:29.034899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952533+0000","last_change":"2024-03-29T11:34:19.945718+0000","last_active":"2024-03-29T11:34:19.952533+0000","last_peered":"2024-03-29T11:34:19.952533+0000","last_clean":"2024-03-29T11:34:19.952533+0000","last_became_active":"2024-03-29T11:34:19.945423+0000","last_became_peered":"2024-03-29T11:34:19.945423+0000","last_unstale":"2024-03-29T11:34:19.952533+0000","last_undegraded":"2024-03-29T11:34:19.952533+0000","last_fullsized":"2024-03-29T11:34:19.952533+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:43:25.035704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"282'2","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.963294+0000","last_change":"2024-03-29T11:34:19.428704+0000","last_active":"2024-03-29T11:34:19.963294+0000","last_peered":"2024-03-29T11:34:19.963294+0000","last_clean":"2024-03-29T11:34:19.963294+0000","last_became_active":"2024-03-29T11:34:19.428125+0000","last_became_peered":"2024-03-29T11:34:19.428125+0000","last_unstale":"2024-03-29T11:34:19.963294+0000","last_undegraded":"2024-03-29T11:34:19.963294+0000","last_fullsized":"2024-03-29T11:34:19.963294+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:45:55.704448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.955691+0000","last_change":"2024-03-29T11:34:19.953791+0000","last_active":"2024-03-29T11:34:19.955691+0000","last_peered":"2024-03-29T11:34:19.955691+0000","last_clean":"2024-03-29T11:34:19.955691+0000","last_became_active":"2024-03-29T11:34:19.952430+0000","last_became_peered":"2024-03-29T11:34:19.952430+0000","last_unstale":"2024-03-29T11:34:19.955691+0000","last_undegraded":"2024-03-29T11:34:19.955691+0000","last_fullsized":"2024-03-29T11:34:19.955691+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:24:08.808042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.935169+0000","last_change":"2024-03-29T11:34:19.424373+0000","last_active":"2024-03-29T11:34:20.935169+0000","last_peered":"2024-03-29T11:34:20.935169+0000","last_clean":"2024-03-29T11:34:20.935169+0000","last_became_active":"2024-03-29T11:34:19.423585+0000","last_became_peered":"2024-03-29T11:34:19.423585+0000","last_unstale":"2024-03-29T11:34:20.935169+0000","last_undegraded":"2024-03-29T11:34:20.935169+0000","last_fullsized":"2024-03-29T11:34:20.935169+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:28.076825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934479+0000","last_change":"2024-03-29T11:34:19.954942+0000","last_active":"2024-03-29T11:34:20.934479+0000","last_peered":"2024-03-29T11:34:20.934479+0000","last_clean":"2024-03-29T11:34:20.934479+0000","last_became_active":"2024-03-29T11:34:19.954660+0000","last_became_peered":"2024-03-29T11:34:19.954660+0000","last_unstale":"2024-03-29T11:34:20.934479+0000","last_undegraded":"2024-03-29T11:34:20.934479+0000","last_fullsized":"2024-03-29T11:34:20.934479+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:50:19.403032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.930332+0000","last_change":"2024-03-29T11:34:19.428156+0000","last_active":"2024-03-29T11:34:19.930332+0000","last_peered":"2024-03-29T11:34:19.930332+0000","last_clean":"2024-03-29T11:34:19.930332+0000","last_became_active":"2024-03-29T11:34:19.427900+0000","last_became_peered":"2024-03-29T11:34:19.427900+0000","last_unstale":"2024-03-29T11:34:19.930332+0000","last_undegraded":"2024-03-29T11:34:19.930332+0000","last_fullsized":"2024-03-29T11:34:19.930332+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:54.386068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934544+0000","last_change":"2024-03-29T11:34:19.947167+0000","last_active":"2024-03-29T11:34:20.934544+0000","last_peered":"2024-03-29T11:34:20.934544+0000","last_clean":"2024-03-29T11:34:20.934544+0000","last_became_active":"2024-03-29T11:34:19.946910+0000","last_became_peered":"2024-03-29T11:34:19.946910+0000","last_unstale":"2024-03-29T11:34:20.934544+0000","last_undegraded":"2024-03-29T11:34:20.934544+0000","last_fullsized":"2024-03-29T11:34:20.934544+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:40.860490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934570+0000","last_change":"2024-03-29T11:34:19.423695+0000","last_active":"2024-03-29T11:34:20.934570+0000","last_peered":"2024-03-29T11:34:20.934570+0000","last_clean":"2024-03-29T11:34:20.934570+0000","last_became_active":"2024-03-29T11:34:19.423386+0000","last_became_peered":"2024-03-29T11:34:19.423386+0000","last_unstale":"2024-03-29T11:34:20.934570+0000","last_undegraded":"2024-03-29T11:34:20.934570+0000","last_fullsized":"2024-03-29T11:34:20.934570+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:14.486166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.959882+0000","last_change":"2024-03-29T11:34:19.957275+0000","last_active":"2024-03-29T11:34:19.959882+0000","last_peered":"2024-03-29T11:34:19.959882+0000","last_clean":"2024-03-29T11:34:19.959882+0000","last_became_active":"2024-03-29T11:34:19.957003+0000","last_became_peered":"2024-03-29T11:34:19.957003+0000","last_unstale":"2024-03-29T11:34:19.959882+0000","last_undegraded":"2024-03-29T11:34:19.959882+0000","last_fullsized":"2024-03-29T11:34:19.959882+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:47:39.749122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"282'2","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.960690+0000","last_change":"2024-03-29T11:34:19.427149+0000","last_active":"2024-03-29T11:34:19.960690+0000","last_peered":"2024-03-29T11:34:19.960690+0000","last_clean":"2024-03-29T11:34:19.960690+0000","last_became_active":"2024-03-29T11:34:19.426789+0000","last_became_peered":"2024-03-29T11:34:19.426789+0000","last_unstale":"2024-03-29T11:34:19.960690+0000","last_undegraded":"2024-03-29T11:34:19.960690+0000","last_fullsized":"2024-03-29T11:34:19.960690+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:33.587404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941051+0000","last_change":"2024-03-29T11:34:19.947080+0000","last_active":"2024-03-29T11:34:20.941051+0000","last_peered":"2024-03-29T11:34:20.941051+0000","last_clean":"2024-03-29T11:34:20.941051+0000","last_became_active":"2024-03-29T11:34:19.946641+0000","last_became_peered":"2024-03-29T11:34:19.946641+0000","last_unstale":"2024-03-29T11:34:20.941051+0000","last_undegraded":"2024-03-29T11:34:20.941051+0000","last_fullsized":"2024-03-29T11:34:20.941051+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:19:49.890182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934989+0000","last_change":"2024-03-29T11:34:19.422163+0000","last_active":"2024-03-29T11:34:20.934989+0000","last_peered":"2024-03-29T11:34:20.934989+0000","last_clean":"2024-03-29T11:34:20.934989+0000","last_became_active":"2024-03-29T11:34:19.421894+0000","last_became_peered":"2024-03-29T11:34:19.421894+0000","last_unstale":"2024-03-29T11:34:20.934989+0000","last_undegraded":"2024-03-29T11:34:20.934989+0000","last_fullsized":"2024-03-29T11:34:20.934989+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:22:00.471992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.959950+0000","last_change":"2024-03-29T11:34:19.958676+0000","last_active":"2024-03-29T11:34:19.959950+0000","last_peered":"2024-03-29T11:34:19.959950+0000","last_clean":"2024-03-29T11:34:19.959950+0000","last_became_active":"2024-03-29T11:34:19.957819+0000","last_became_peered":"2024-03-29T11:34:19.957819+0000","last_unstale":"2024-03-29T11:34:19.959950+0000","last_undegraded":"2024-03-29T11:34:19.959950+0000","last_fullsized":"2024-03-29T11:34:19.959950+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:11:24.486578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.935123+0000","last_change":"2024-03-29T11:34:19.429752+0000","last_active":"2024-03-29T11:34:19.935123+0000","last_peered":"2024-03-29T11:34:19.935123+0000","last_clean":"2024-03-29T11:34:19.935123+0000","last_became_active":"2024-03-29T11:34:19.429508+0000","last_became_peered":"2024-03-29T11:34:19.429508+0000","last_unstale":"2024-03-29T11:34:19.935123+0000","last_undegraded":"2024-03-29T11:34:19.935123+0000","last_fullsized":"2024-03-29T11:34:19.935123+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:25.549951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941150+0000","last_change":"2024-03-29T11:34:19.945264+0000","last_active":"2024-03-29T11:34:20.941150+0000","last_peered":"2024-03-29T11:34:20.941150+0000","last_clean":"2024-03-29T11:34:20.941150+0000","last_became_active":"2024-03-29T11:34:19.944967+0000","last_became_peered":"2024-03-29T11:34:19.944967+0000","last_unstale":"2024-03-29T11:34:20.941150+0000","last_undegraded":"2024-03-29T11:34:20.941150+0000","last_fullsized":"2024-03-29T11:34:20.941150+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:52:42.879051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.936864+0000","last_change":"2024-03-29T11:34:19.434147+0000","last_active":"2024-03-29T11:34:19.936864+0000","last_peered":"2024-03-29T11:34:19.936864+0000","last_clean":"2024-03-29T11:34:19.936864+0000","last_became_active":"2024-03-29T11:34:19.433817+0000","last_became_peered":"2024-03-29T11:34:19.433817+0000","last_unstale":"2024-03-29T11:34:19.936864+0000","last_undegraded":"2024-03-29T11:34:19.936864+0000","last_fullsized":"2024-03-29T11:34:19.936864+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:51:50.580234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.954620+0000","last_change":"2024-03-29T11:34:19.948206+0000","last_active":"2024-03-29T11:34:19.954620+0000","last_peered":"2024-03-29T11:34:19.954620+0000","last_clean":"2024-03-29T11:34:19.954620+0000","last_became_active":"2024-03-29T11:34:19.947867+0000","last_became_peered":"2024-03-29T11:34:19.947867+0000","last_unstale":"2024-03-29T11:34:19.954620+0000","last_undegraded":"2024-03-29T11:34:19.954620+0000","last_fullsized":"2024-03-29T11:34:19.954620+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:03:26.025761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.937323+0000","last_change":"2024-03-29T11:34:19.433271+0000","last_active":"2024-03-29T11:34:19.937323+0000","last_peered":"2024-03-29T11:34:19.937323+0000","last_clean":"2024-03-29T11:34:19.937323+0000","last_became_active":"2024-03-29T11:34:19.432922+0000","last_became_peered":"2024-03-29T11:34:19.432922+0000","last_unstale":"2024-03-29T11:34:19.937323+0000","last_undegraded":"2024-03-29T11:34:19.937323+0000","last_fullsized":"2024-03-29T11:34:19.937323+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:28.842015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966335+0000","last_change":"2024-03-29T11:34:19.963007+0000","last_active":"2024-03-29T11:34:19.966335+0000","last_peered":"2024-03-29T11:34:19.966335+0000","last_clean":"2024-03-29T11:34:19.966335+0000","last_became_active":"2024-03-29T11:34:19.960610+0000","last_became_peered":"2024-03-29T11:34:19.960610+0000","last_unstale":"2024-03-29T11:34:19.966335+0000","last_undegraded":"2024-03-29T11:34:19.966335+0000","last_fullsized":"2024-03-29T11:34:19.966335+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:32:52.022464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.935997+0000","last_change":"2024-03-29T11:34:19.431583+0000","last_active":"2024-03-29T11:34:19.935997+0000","last_peered":"2024-03-29T11:34:19.935997+0000","last_clean":"2024-03-29T11:34:19.935997+0000","last_became_active":"2024-03-29T11:34:19.431102+0000","last_became_peered":"2024-03-29T11:34:19.431102+0000","last_unstale":"2024-03-29T11:34:19.935997+0000","last_undegraded":"2024-03-29T11:34:19.935997+0000","last_fullsized":"2024-03-29T11:34:19.935997+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:43.792763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934500+0000","last_change":"2024-03-29T11:34:19.957073+0000","last_active":"2024-03-29T11:34:20.934500+0000","last_peered":"2024-03-29T11:34:20.934500+0000","last_clean":"2024-03-29T11:34:20.934500+0000","last_became_active":"2024-03-29T11:34:19.956777+0000","last_became_peered":"2024-03-29T11:34:19.956777+0000","last_unstale":"2024-03-29T11:34:20.934500+0000","last_undegraded":"2024-03-29T11:34:20.934500+0000","last_fullsized":"2024-03-29T11:34:20.934500+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:36:08.286066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.934719+0000","last_change":"2024-03-29T11:34:19.417904+0000","last_active":"2024-03-29T11:34:19.934719+0000","last_peered":"2024-03-29T11:34:19.934719+0000","last_clean":"2024-03-29T11:34:19.934719+0000","last_became_active":"2024-03-29T11:34:19.417551+0000","last_became_peered":"2024-03-29T11:34:19.417551+0000","last_unstale":"2024-03-29T11:34:19.934719+0000","last_undegraded":"2024-03-29T11:34:19.934719+0000","last_fullsized":"2024-03-29T11:34:19.934719+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:27:24.018400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940183+0000","last_change":"2024-03-29T11:34:19.940077+0000","last_active":"2024-03-29T11:34:20.940183+0000","last_peered":"2024-03-29T11:34:20.940183+0000","last_clean":"2024-03-29T11:34:20.940183+0000","last_became_active":"2024-03-29T11:34:19.939633+0000","last_became_peered":"2024-03-29T11:34:19.939633+0000","last_unstale":"2024-03-29T11:34:20.940183+0000","last_undegraded":"2024-03-29T11:34:20.940183+0000","last_fullsized":"2024-03-29T11:34:20.940183+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:54.046445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.941263+0000","last_change":"2024-03-29T11:34:19.432321+0000","last_active":"2024-03-29T11:34:19.941263+0000","last_peered":"2024-03-29T11:34:19.941263+0000","last_clean":"2024-03-29T11:34:19.941263+0000","last_became_active":"2024-03-29T11:34:19.431739+0000","last_became_peered":"2024-03-29T11:34:19.431739+0000","last_unstale":"2024-03-29T11:34:19.941263+0000","last_undegraded":"2024-03-29T11:34:19.941263+0000","last_fullsized":"2024-03-29T11:34:19.941263+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:54.386068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966497+0000","last_change":"2024-03-29T11:34:19.964406+0000","last_active":"2024-03-29T11:34:19.966497+0000","last_peered":"2024-03-29T11:34:19.966497+0000","last_clean":"2024-03-29T11:34:19.966497+0000","last_became_active":"2024-03-29T11:34:19.963593+0000","last_became_peered":"2024-03-29T11:34:19.963593+0000","last_unstale":"2024-03-29T11:34:19.966497+0000","last_undegraded":"2024-03-29T11:34:19.966497+0000","last_fullsized":"2024-03-29T11:34:19.966497+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:39:11.647026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"272'432","reported_seq":1552,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.423309+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:34:19.423309+0000","last_peered":"2024-03-29T11:34:19.423309+0000","last_clean":"2024-03-29T11:34:19.423309+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:34:19.423309+0000","last_undegraded":"2024-03-29T11:34:19.423309+0000","last_fullsized":"2024-03-29T11:34:19.423309+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2864,"num_read_kb":6044,"num_write":731,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2864,"num_read_kb":6044,"num_write":731,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576023,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57324,"kb_used_data":4060,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314516,"statfs":{"total":96636764160,"available":96578064384,"internally_reserved":0,"allocated":4157440,"data_stored":2346595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.67600000000000005,"15min":0.70799999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86799999999999999,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.68100000000000005,"15min":0.73099999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86899999999999999,"5min":0.98899999999999999,"15min":1.119},"last":0.81599999999999995}]},{"osd":1,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.67100000000000004,"15min":0.71599999999999997},"min":{"1min":0.435,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84399999999999997,"5min":0.97399999999999998,"15min":1.478},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.70399999999999996,"15min":0.752},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.82699999999999996,"5min":1.0640000000000001,"15min":1.393},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70499999999999996,"15min":0.748},"min":{"1min":0.46600000000000003,"5min":0.436,"15min":0.436},"max":{"1min":2.5649999999999999,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":0.751},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.71399999999999997,"15min":0.76100000000000001},"min":{"1min":0.501,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88400000000000001,"5min":1.032,"15min":1.0720000000000001},"last":0.77600000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.749},"min":{"1min":0.48499999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84299999999999997,"5min":0.97499999999999998,"15min":1.0580000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.77700000000000002,"15min":0.78600000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.6930000000000001,"5min":2.6930000000000001,"15min":2.6930000000000001},"last":0.79100000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88500000000000001,"5min":0.91100000000000003,"15min":1.3779999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66400000000000003,"15min":0.66900000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90800000000000003,"5min":0.93500000000000005,"15min":1.3169999999999999},"last":0.58999999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69999999999999996,"15min":0.68400000000000005},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.1160000000000001},"last":0.62},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93700000000000006,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.65600000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.76800000000000002,"15min":0.76600000000000001},"min":{"1min":0.48099999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92600000000000005,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.75600000000000001,"15min":0.74199999999999999},"min":{"1min":0.52000000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.91000000000000003,"5min":1.071,"15min":1.071},"last":0.68799999999999994}]}]},{"osd":6,"up_from":13,"seq":55834576024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48568,"kb_used_data":3492,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323272,"statfs":{"total":96636764160,"available":96587030528,"internally_reserved":0,"allocated":3575808,"data_stored":1757791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.67100000000000004,"15min":0.71999999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.94499999999999995,"15min":1.2150000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.68400000000000005,"15min":0.74199999999999999},"min":{"1min":0.42799999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95299999999999996,"5min":0.96399999999999997,"15min":1.4239999999999999},"last":0.83899999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67000000000000004,"15min":0.72199999999999998},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91100000000000003,"5min":1.022,"15min":1.98},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.70699999999999996,"15min":0.75900000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.435},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.8200000000000001},"last":0.93999999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69699999999999995,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86899999999999999,"5min":0.96199999999999997,"15min":1.669},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71099999999999997,"15min":0.751},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.167,"5min":1.167,"15min":1.7589999999999999},"last":0.70799999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70599999999999996,"15min":0.753},"min":{"1min":0.51100000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90200000000000002,"5min":0.997,"15min":2.52},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.75700000000000001,"15min":0.79200000000000004},"min":{"1min":0.51200000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.877},"last":1.1060000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.61599999999999999},"min":{"1min":0.435,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93999999999999995,"15min":2.4100000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.71599999999999997,"15min":0.71199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.45900000000000002},"max":{"1min":1.0740000000000001,"5min":1.2709999999999999,"15min":2.6629999999999998},"last":0.91300000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66900000000000004,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99099999999999999,"5min":1.0900000000000001,"15min":1.5960000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.925},"last":0.80600000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77100000000000002,"15min":0.76900000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96699999999999997,"5min":1.0509999999999999,"15min":2.7050000000000001},"last":1.056},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74299999999999999,"15min":0.74399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0980000000000001,"5min":1.1220000000000001,"15min":1.726},"last":0.88100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576022,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64944,"kb_used_data":3488,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306896,"statfs":{"total":96636764160,"available":96570261504,"internally_reserved":0,"allocated":3571712,"data_stored":1756283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.44600000000000001,"15min":0.46100000000000002},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.62,"5min":0.79100000000000004,"15min":0.80300000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.34399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.626,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52700000000000002,"15min":0.55400000000000005},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72599999999999998,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.50700000000000001,"15min":0.52600000000000002},"min":{"1min":0.33800000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66500000000000004,"5min":0.75900000000000001,"15min":0.90300000000000002},"last":0.36099999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56799999999999995,"15min":0.58699999999999997},"min":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.36299999999999999},"max":{"1min":0.77600000000000002,"5min":0.77700000000000002,"15min":0.995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53900000000000003,"15min":0.56999999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75800000000000001,"5min":0.77700000000000002,"15min":0.83999999999999997},"last":0.45900000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0589999999999999,"5min":1.153,"15min":1.238},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77300000000000002,"15min":0.79200000000000004},"min":{"1min":0.60099999999999998,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":1.0109999999999999,"5min":1.222,"15min":1.304},"last":0.74399999999999999}]},{"osd":5,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.80100000000000005,"15min":0.81100000000000005},"min":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":1.1359999999999999,"5min":1.1819999999999999,"15min":1.3720000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78900000000000003,"15min":0.82199999999999995},"min":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.3520000000000001},"last":0.66000000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78700000000000003,"15min":0.81000000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0369999999999999,"5min":1.1559999999999999,"15min":1.1919999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.58999999999999997,"5min":0.501,"15min":0.501},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.266},"last":0.85299999999999998}]},{"osd":7,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.85899999999999999},"min":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.54500000000000004},"max":{"1min":1.0800000000000001,"5min":1.129,"15min":1.327},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82099999999999995,"15min":0.85799999999999998},"min":{"1min":0.622,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.159,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.68300000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576024,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65520,"kb_used_data":4064,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306320,"statfs":{"total":96636764160,"available":96569671680,"internally_reserved":0,"allocated":4161536,"data_stored":2345668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.442,"15min":0.45900000000000002},"min":{"1min":0.28699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":1.087},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.504,"15min":0.53200000000000003},"min":{"1min":0.33600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.85799999999999998,"15min":1.3089999999999999},"last":0.59799999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54700000000000004,"15min":0.57399999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63300000000000001,"5min":0.84999999999999998,"15min":1.5289999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48299999999999998,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66900000000000004,"5min":0.70899999999999996,"15min":1.6140000000000001},"last":0.46800000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.53200000000000003,"15min":0.55600000000000005},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65300000000000002,"5min":0.84699999999999998,"15min":1.222},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.59899999999999998,"15min":0.63200000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.72999999999999998,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.64900000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.78400000000000003,"15min":0.80700000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.495},"max":{"1min":0.93400000000000005,"5min":1.248,"15min":1.248},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.80400000000000005,"15min":0.83899999999999997},"min":{"1min":0.58299999999999996,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.012,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.95299999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.74299999999999999,"15min":0.77900000000000003},"min":{"1min":0.53200000000000003,"5min":0.505,"15min":0.505},"max":{"1min":0.89200000000000002,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78600000000000003,"15min":0.81299999999999994},"min":{"1min":0.52600000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.018,"5min":1.1499999999999999,"15min":1.3640000000000001},"last":0.78500000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.80400000000000005,"15min":0.84499999999999997},"min":{"1min":0.499,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0649999999999999,"5min":1.236,"15min":1.6240000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.83499999999999996,"15min":0.876},"min":{"1min":0.5,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.996,"5min":1.1739999999999999,"15min":1.337},"last":0.88700000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.78500000000000003,"15min":0.81899999999999995},"min":{"1min":0.52300000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.05,"5min":1.111,"15min":1.214},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.81599999999999995,"15min":0.86399999999999999},"min":{"1min":0.56200000000000006,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0329999999999999,"5min":1.2090000000000001,"15min":1.365},"last":0.86699999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576023,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52660,"kb_used_data":3488,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319180,"statfs":{"total":96636764160,"available":96582840320,"internally_reserved":0,"allocated":3571712,"data_stored":1756351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.47299999999999998,"15min":0.47199999999999998},"min":{"1min":0.32300000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.5,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60199999999999998,"5min":0.73999999999999999,"15min":0.96799999999999997},"last":0.377}]},{"osd":1,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47899999999999998,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.54900000000000004,"15min":0.56999999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.872,"15min":0.88800000000000001},"last":0.505}]},{"osd":3,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55800000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64900000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55700000000000005,"15min":0.57199999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.71099999999999997,"5min":1.038,"15min":1.038},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.754,"15min":0.79800000000000004},"min":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1970000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.747,"15min":0.79900000000000004},"min":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0029999999999999,"5min":1.157,"15min":1.244},"last":0.77600000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77000000000000002,"15min":0.80900000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.53400000000000003},"max":{"1min":0.96399999999999997,"5min":1.135,"15min":1.256},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.747,"5min":0.76500000000000001,"15min":0.81399999999999995},"min":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.3140000000000001},"last":0.76700000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.78000000000000003,"15min":0.83099999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93200000000000005,"5min":1.105,"15min":1.24},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.78000000000000003,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.5,"15min":0.5},"max":{"1min":0.98199999999999998,"5min":1.099,"15min":1.329},"last":0.753}]},{"osd":7,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82499999999999996,"15min":0.88},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.044,"5min":1.0660000000000001,"15min":1.2889999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.83899999999999997,"15min":0.89400000000000002},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.56899999999999995},"max":{"1min":1.0269999999999999,"5min":1.2969999999999999,"15min":1.304},"last":0.84399999999999997}]}]},{"osd":3,"up_from":13,"seq":55834576023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48556,"kb_used_data":3484,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323284,"statfs":{"total":96636764160,"available":96587042816,"internally_reserved":0,"allocated":3567616,"data_stored":1756809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.28199999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.69399999999999995,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52800000000000002,"15min":0.54500000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66100000000000003,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.436}]},{"osd":1,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47999999999999998,"15min":0.49099999999999999},"min":{"1min":0.30599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.626,"5min":1.821,"15min":1.821},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.54400000000000004,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.64500000000000002,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.58199999999999996,"15min":0.58699999999999997},"min":{"1min":0.39200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.745,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.64100000000000001,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85699999999999998,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.46200000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.55500000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.014,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.872},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.78800000000000003,"15min":0.81100000000000005},"min":{"1min":0.55600000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95599999999999996,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.02}]},{"osd":5,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79600000000000004,"15min":0.80900000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.032,"5min":2.157,"15min":2.157},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.82599999999999996,"15min":0.82299999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0640000000000001,"5min":2.266,"15min":2.266},"last":1.0389999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.85399999999999998,"15min":0.85599999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.083,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.82299999999999995,"15min":0.83799999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.03,"5min":2.198,"15min":2.198},"last":0.93999999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.85399999999999998,"15min":0.86199999999999999},"min":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":0.98499999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.88,"15min":0.88600000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.149,"5min":2.444,"15min":2.444},"last":1.0549999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96884,"kb_used_data":3488,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274956,"statfs":{"total":96636764160,"available":96537554944,"internally_reserved":0,"allocated":3571712,"data_stored":1756879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64400000000000002,"15min":0.70699999999999996},"min":{"1min":0.39700000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":1.0509999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.64000000000000001,"15min":0.70799999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.81000000000000005,"5min":0.93600000000000005,"15min":1.1459999999999999},"last":0.65500000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.65500000000000003,"15min":0.72499999999999998},"min":{"1min":0.48699999999999999,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.1910000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67000000000000004,"15min":0.72999999999999998},"min":{"1min":0.435,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":1.002},"last":0.70199999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76300000000000001,"15min":0.81399999999999995},"min":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.73899999999999999,"15min":0.80000000000000004},"min":{"1min":0.51900000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.875,"5min":1.224,"15min":1.224},"last":0.753}]},{"osd":3,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68500000000000005,"15min":0.748},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88100000000000001,"5min":0.96699999999999997,"15min":1.181},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72899999999999998,"15min":0.76300000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.84899999999999998}]},{"osd":5,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64900000000000002,"15min":0.628},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.01,"5min":1.0920000000000001,"15min":1.169},"last":0.50900000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69599999999999995,"15min":0.67300000000000004},"min":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72599999999999998,"15min":0.72099999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.53200000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.73899999999999999,"15min":0.73399999999999999},"min":{"1min":0.54100000000000004,"5min":0.501,"15min":0.436},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.2669999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.81899999999999995,"15min":0.82299999999999995},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.54400000000000004},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.878}]}]},{"osd":5,"up_from":13,"seq":55834576023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48572,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323268,"statfs":{"total":96636764160,"available":96587026432,"internally_reserved":0,"allocated":3579904,"data_stored":1757881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.64800000000000002,"15min":0.68999999999999995},"min":{"1min":0.38700000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71299999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.66100000000000003,"15min":0.71599999999999997},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.752,"5min":0.95599999999999996,"15min":1.0680000000000001},"last":0.51600000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.72199999999999998,"15min":0.751},"min":{"1min":0.434,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":1.839,"15min":1.839},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.69999999999999996,"15min":0.72999999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.88600000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.56899999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.73099999999999998,"15min":0.747},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93899999999999995,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.73999999999999999},"min":{"1min":0.54700000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.81299999999999994,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.753,"15min":0.76400000000000001},"min":{"1min":0.49199999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.84699999999999998,"5min":3.444,"15min":3.444},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.748,"15min":0.76900000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.98299999999999998,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63400000000000001,"15min":0.61199999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86099999999999999,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.68200000000000005,"15min":0.65500000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88600000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.76000000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.71299999999999997,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90100000000000002,"5min":1.214,"15min":1.214},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.72399999999999998,"15min":0.70299999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.42499999999999999},"max":{"1min":0.91000000000000003,"5min":1.0900000000000001,"15min":1.107},"last":0.74399999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.748,"15min":0.72799999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92400000000000004,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.79200000000000004,"15min":0.77500000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93999999999999995,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.78300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:34:24.856 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:34:25.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.093+0000 7f0f06400700 1 Processor -- start 2024-03-29T11:34:25.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.093+0000 7f0f06400700 1 -- start start 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.094+0000 7f0f06400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0f000ce590 0x7f0f000ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.094+0000 7f0f06400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f000b6a20 0x7f0f000b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.094+0000 7f0f06400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f000ceef0 con 0x7f0f00160be0 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.094+0000 7f0f06400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f000b65d0 con 0x7f0f000ce590 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.094+0000 7f0f06400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f000d2a80 con 0x7f0f000b6a20 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0efea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0f000ce590 0x7f0f000ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0efea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0f000ce590 0x7f0f000ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51528/0 (socket says 172.21.15.12:51528) 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0efea00700 1 -- 172.21.15.12:0/2551563311 learned_addr learned my addr 172.21.15.12:0/2551563311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0eff400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0f00160be0 0x7f0f000cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60236/0 (socket says 172.21.15.12:60236) 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0effe00700 1 --2- 172.21.15.12:0/2551563311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f000b6a20 0x7f0f000b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0efea00700 1 -- 172.21.15.12:0/2551563311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f000b6a20 msgr2=0x7f0f000b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0efea00700 1 --2- 172.21.15.12:0/2551563311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f000b6a20 0x7f0f000b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0efea00700 1 -- 172.21.15.12:0/2551563311 >> v1:172.21.15.12:6789/0 conn(0x7f0f00160be0 legacy=0x7f0f000cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0efea00700 1 -- 172.21.15.12:0/2551563311 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f00160fc0 con 0x7f0f000ce590 2024-03-29T11:34:25.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0efea00700 1 --2- 172.21.15.12:0/2551563311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000ce590 0x7f0f000ce9b0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0098f0 tx=0x7f0eec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c2dffd8ef62cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:25.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.095+0000 7f0efe000700 1 -- 172.21.15.12:0/2551563311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec00b8a0 con 0x7f0f000ce590 2024-03-29T11:34:25.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.096+0000 7f0efe000700 1 -- 172.21.15.12:0/2551563311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eec00ba00 con 0x7f0f000ce590 2024-03-29T11:34:25.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.096+0000 7f0efe000700 1 -- 172.21.15.12:0/2551563311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec00bd90 con 0x7f0f000ce590 2024-03-29T11:34:25.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.096+0000 7f0f06400700 1 -- 172.21.15.12:0/2551563311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000ce590 msgr2=0x7f0f000ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.096+0000 7f0f06400700 1 --2- 172.21.15.12:0/2551563311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000ce590 0x7f0f000ce9b0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0098f0 tx=0x7f0eec009c00 comp rx=0 tx=0).stop 2024-03-29T11:34:25.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.096+0000 7f0f06400700 1 -- 172.21.15.12:0/2551563311 shutdown_connections 2024-03-29T11:34:25.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.096+0000 7f0f06400700 1 --2- 172.21.15.12:0/2551563311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f000b6a20 0x7f0f000b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.096+0000 7f0f06400700 1 --2- 172.21.15.12:0/2551563311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000ce590 0x7f0f000ce9b0 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.096+0000 7f0f06400700 1 -- 172.21.15.12:0/2551563311 >> 172.21.15.12:0/2551563311 conn(0x7f0f0008aff0 msgr2=0x7f0f000d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:25.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.096+0000 7f0f06400700 1 -- 172.21.15.12:0/2551563311 shutdown_connections 2024-03-29T11:34:25.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.096+0000 7f0f06400700 1 -- 172.21.15.12:0/2551563311 wait complete. 2024-03-29T11:34:25.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.098+0000 7f0f06400700 1 Processor -- start 2024-03-29T11:34:25.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.098+0000 7f0f06400700 1 -- start start 2024-03-29T11:34:25.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.099+0000 7f0f06400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000b6a20 0x7f0f001f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.099+0000 7f0f06400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f000ce590 0x7f0f0006caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.099+0000 7f0f06400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f00160be0 0x7f0f001f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.099+0000 7f0f06400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f001f6d40 con 0x7f0f000b6a20 2024-03-29T11:34:25.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.099+0000 7f0f06400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f000c8cb0 con 0x7f0f00160be0 2024-03-29T11:34:25.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.099+0000 7f0f06400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f001f71d0 con 0x7f0f000ce590 2024-03-29T11:34:25.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.099+0000 7f0eff400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000b6a20 0x7f0f001f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.099+0000 7f0eff400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000b6a20 0x7f0f001f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51534/0 (socket says 172.21.15.12:51534) 2024-03-29T11:34:25.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.099+0000 7f0eff400700 1 -- 172.21.15.12:0/2428173202 learned_addr learned my addr 172.21.15.12:0/2428173202 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0efea00700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f000ce590 0x7f0f0006caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0effe00700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f00160be0 0x7f0f001f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0eff400700 1 -- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f000ce590 msgr2=0x7f0f0006caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0eff400700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f000ce590 0x7f0f0006caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0eff400700 1 -- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f00160be0 msgr2=0x7f0f001f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0eff400700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f00160be0 0x7f0f001f25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0eff400700 1 -- 172.21.15.12:0/2428173202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec009550 con 0x7f0f000b6a20 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0effe00700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f00160be0 0x7f0f001f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0eff400700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000b6a20 0x7f0f001f3760 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000c500 tx=0x7f0ef000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0ef7400700 1 -- 172.21.15.12:0/2428173202 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0005cb0 con 0x7f0f000b6a20 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0ef7400700 1 -- 172.21.15.12:0/2428173202 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef0005e10 con 0x7f0f000b6a20 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0ef7400700 1 -- 172.21.15.12:0/2428173202 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef00072d0 con 0x7f0f000b6a20 2024-03-29T11:34:25.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.100+0000 7f0f06400700 1 -- 172.21.15.12:0/2428173202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f001f6410 con 0x7f0f000b6a20 2024-03-29T11:34:25.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.101+0000 7f0f06400700 1 -- 172.21.15.12:0/2428173202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f001f69d0 con 0x7f0f000b6a20 2024-03-29T11:34:25.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.102+0000 7f0f06400700 1 -- 172.21.15.12:0/2428173202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f00067720 con 0x7f0f000b6a20 2024-03-29T11:34:25.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.102+0000 7f0ef7400700 1 -- 172.21.15.12:0/2428173202 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0ef0007430 con 0x7f0f000b6a20 2024-03-29T11:34:25.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.103+0000 7f0ef7400700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ee006adb0 0x7f0ee006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.103+0000 7f0ef7400700 1 -- 172.21.15.12:0/2428173202 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f0ef0088760 con 0x7f0f000b6a20 2024-03-29T11:34:25.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.104+0000 7f0efea00700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ee006adb0 0x7f0ee006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.105+0000 7f0efea00700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ee006adb0 0x7f0ee006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009c40 tx=0x7f0eec008030 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:25.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.108+0000 7f0ef7400700 1 -- 172.21.15.12:0/2428173202 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ef0053880 con 0x7f0f000b6a20 2024-03-29T11:34:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.316+0000 7f0f06400700 1 -- 172.21.15.12:0/2428173202 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f000d27a0 con 0x7f0ee006adb0 2024-03-29T11:34:25.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.326+0000 7f0ef7400700 1 -- 172.21.15.12:0/2428173202 <== mgr.10517 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289521 (secure 0 0 0) 0x7f0eec00bb70 con 0x7f0ee006adb0 2024-03-29T11:34:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:25.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 -- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ee006adb0 msgr2=0x7f0ee006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ee006adb0 0x7f0ee006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009c40 tx=0x7f0eec008030 comp rx=0 tx=0).stop 2024-03-29T11:34:25.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 -- 172.21.15.12:0/2428173202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000b6a20 msgr2=0x7f0f001f3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000b6a20 0x7f0f001f3760 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000c500 tx=0x7f0ef000c810 comp rx=0 tx=0).stop 2024-03-29T11:34:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 -- 172.21.15.12:0/2428173202 shutdown_connections 2024-03-29T11:34:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f0ee006adb0 0x7f0ee006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f00160be0 0x7f0f001f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f000b6a20 0x7f0f001f3760 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 --2- 172.21.15.12:0/2428173202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f000ce590 0x7f0f0006caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 -- 172.21.15.12:0/2428173202 >> 172.21.15.12:0/2428173202 conn(0x7f0f0008aff0 msgr2=0x7f0f000b6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 -- 172.21.15.12:0/2428173202 shutdown_connections 2024-03-29T11:34:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.330+0000 7f0f06400700 1 -- 172.21.15.12:0/2428173202 wait complete. 2024-03-29T11:34:25.330 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:34:25.346 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-29T11:34:24.979055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2864,"num_read_kb":6044,"num_write":775,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":470,"ondisk_log_size":470,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":483028,"kb_used_data":29060,"kb_used_omap":12,"kb_used_meta":453811,"kb_avail":754491692,"statfs":{"total":773094113280,"available":772599492608,"internally_reserved":0,"allocated":29757440,"data_stored":15234257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":464703033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001528"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941411+0000","last_change":"2024-03-29T11:34:19.947211+0000","last_active":"2024-03-29T11:34:20.941411+0000","last_peered":"2024-03-29T11:34:20.941411+0000","last_clean":"2024-03-29T11:34:20.941411+0000","last_became_active":"2024-03-29T11:34:19.946081+0000","last_became_peered":"2024-03-29T11:34:19.946081+0000","last_unstale":"2024-03-29T11:34:20.941411+0000","last_undegraded":"2024-03-29T11:34:20.941411+0000","last_fullsized":"2024-03-29T11:34:20.941411+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:20:08.753840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.949433+0000","last_change":"2024-03-29T11:34:19.947268+0000","last_active":"2024-03-29T11:34:19.949433+0000","last_peered":"2024-03-29T11:34:19.949433+0000","last_clean":"2024-03-29T11:34:19.949433+0000","last_became_active":"2024-03-29T11:34:19.946610+0000","last_became_peered":"2024-03-29T11:34:19.946610+0000","last_unstale":"2024-03-29T11:34:19.949433+0000","last_undegraded":"2024-03-29T11:34:19.949433+0000","last_fullsized":"2024-03-29T11:34:19.949433+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:50:54.231741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.963132+0000","last_change":"2024-03-29T11:34:19.953262+0000","last_active":"2024-03-29T11:34:19.963132+0000","last_peered":"2024-03-29T11:34:19.963132+0000","last_clean":"2024-03-29T11:34:19.963132+0000","last_became_active":"2024-03-29T11:34:19.952083+0000","last_became_peered":"2024-03-29T11:34:19.952083+0000","last_unstale":"2024-03-29T11:34:19.963132+0000","last_undegraded":"2024-03-29T11:34:19.963132+0000","last_fullsized":"2024-03-29T11:34:19.963132+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:03:26.025761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.965533+0000","last_change":"2024-03-29T11:34:19.957594+0000","last_active":"2024-03-29T11:34:19.965533+0000","last_peered":"2024-03-29T11:34:19.965533+0000","last_clean":"2024-03-29T11:34:19.965533+0000","last_became_active":"2024-03-29T11:34:19.957137+0000","last_became_peered":"2024-03-29T11:34:19.957137+0000","last_unstale":"2024-03-29T11:34:19.965533+0000","last_undegraded":"2024-03-29T11:34:19.965533+0000","last_fullsized":"2024-03-29T11:34:19.965533+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:51:08.112025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966369+0000","last_change":"2024-03-29T11:34:19.963424+0000","last_active":"2024-03-29T11:34:19.966369+0000","last_peered":"2024-03-29T11:34:19.966369+0000","last_clean":"2024-03-29T11:34:19.966369+0000","last_became_active":"2024-03-29T11:34:19.963170+0000","last_became_peered":"2024-03-29T11:34:19.963170+0000","last_unstale":"2024-03-29T11:34:19.966369+0000","last_undegraded":"2024-03-29T11:34:19.966369+0000","last_fullsized":"2024-03-29T11:34:19.966369+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:08:12.571462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.956506+0000","last_change":"2024-03-29T11:34:19.949988+0000","last_active":"2024-03-29T11:34:19.956506+0000","last_peered":"2024-03-29T11:34:19.956506+0000","last_clean":"2024-03-29T11:34:19.956506+0000","last_became_active":"2024-03-29T11:34:19.949550+0000","last_became_peered":"2024-03-29T11:34:19.949550+0000","last_unstale":"2024-03-29T11:34:19.956506+0000","last_undegraded":"2024-03-29T11:34:19.956506+0000","last_fullsized":"2024-03-29T11:34:19.956506+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:20:24.328362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.939980+0000","last_change":"2024-03-29T11:34:19.957036+0000","last_active":"2024-03-29T11:34:20.939980+0000","last_peered":"2024-03-29T11:34:20.939980+0000","last_clean":"2024-03-29T11:34:20.939980+0000","last_became_active":"2024-03-29T11:34:19.956723+0000","last_became_peered":"2024-03-29T11:34:19.956723+0000","last_unstale":"2024-03-29T11:34:20.939980+0000","last_undegraded":"2024-03-29T11:34:20.939980+0000","last_fullsized":"2024-03-29T11:34:20.939980+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:27:25.030286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934403+0000","last_change":"2024-03-29T11:34:19.961721+0000","last_active":"2024-03-29T11:34:20.934403+0000","last_peered":"2024-03-29T11:34:20.934403+0000","last_clean":"2024-03-29T11:34:20.934403+0000","last_became_active":"2024-03-29T11:34:19.961430+0000","last_became_peered":"2024-03-29T11:34:19.961430+0000","last_unstale":"2024-03-29T11:34:20.934403+0000","last_undegraded":"2024-03-29T11:34:20.934403+0000","last_fullsized":"2024-03-29T11:34:20.934403+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:43:25.035704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.958126+0000","last_change":"2024-03-29T11:34:19.955178+0000","last_active":"2024-03-29T11:34:19.958126+0000","last_peered":"2024-03-29T11:34:19.958126+0000","last_clean":"2024-03-29T11:34:19.958126+0000","last_became_active":"2024-03-29T11:34:19.954919+0000","last_became_peered":"2024-03-29T11:34:19.954919+0000","last_unstale":"2024-03-29T11:34:19.958126+0000","last_undegraded":"2024-03-29T11:34:19.958126+0000","last_fullsized":"2024-03-29T11:34:19.958126+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:41:16.428268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941343+0000","last_change":"2024-03-29T11:34:19.944654+0000","last_active":"2024-03-29T11:34:20.941343+0000","last_peered":"2024-03-29T11:34:20.941343+0000","last_clean":"2024-03-29T11:34:20.941343+0000","last_became_active":"2024-03-29T11:34:19.944304+0000","last_became_peered":"2024-03-29T11:34:19.944304+0000","last_unstale":"2024-03-29T11:34:20.941343+0000","last_undegraded":"2024-03-29T11:34:20.941343+0000","last_fullsized":"2024-03-29T11:34:20.941343+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:36.332104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.935020+0000","last_change":"2024-03-29T11:34:19.958236+0000","last_active":"2024-03-29T11:34:20.935020+0000","last_peered":"2024-03-29T11:34:20.935020+0000","last_clean":"2024-03-29T11:34:20.935020+0000","last_became_active":"2024-03-29T11:34:19.956687+0000","last_became_peered":"2024-03-29T11:34:19.956687+0000","last_unstale":"2024-03-29T11:34:20.935020+0000","last_undegraded":"2024-03-29T11:34:20.935020+0000","last_fullsized":"2024-03-29T11:34:20.935020+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:04:30.046786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.959520+0000","last_change":"2024-03-29T11:34:19.958532+0000","last_active":"2024-03-29T11:34:19.959520+0000","last_peered":"2024-03-29T11:34:19.959520+0000","last_clean":"2024-03-29T11:34:19.959520+0000","last_became_active":"2024-03-29T11:34:19.958273+0000","last_became_peered":"2024-03-29T11:34:19.958273+0000","last_unstale":"2024-03-29T11:34:19.959520+0000","last_undegraded":"2024-03-29T11:34:19.959520+0000","last_fullsized":"2024-03-29T11:34:19.959520+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:01:56.645123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940743+0000","last_change":"2024-03-29T11:34:19.951235+0000","last_active":"2024-03-29T11:34:20.940743+0000","last_peered":"2024-03-29T11:34:20.940743+0000","last_clean":"2024-03-29T11:34:20.940743+0000","last_became_active":"2024-03-29T11:34:19.950916+0000","last_became_peered":"2024-03-29T11:34:19.950916+0000","last_unstale":"2024-03-29T11:34:20.940743+0000","last_undegraded":"2024-03-29T11:34:20.940743+0000","last_fullsized":"2024-03-29T11:34:20.940743+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:01:02.478752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951883+0000","last_change":"2024-03-29T11:34:19.946766+0000","last_active":"2024-03-29T11:34:19.951883+0000","last_peered":"2024-03-29T11:34:19.951883+0000","last_clean":"2024-03-29T11:34:19.951883+0000","last_became_active":"2024-03-29T11:34:19.942919+0000","last_became_peered":"2024-03-29T11:34:19.942919+0000","last_unstale":"2024-03-29T11:34:19.951883+0000","last_undegraded":"2024-03-29T11:34:19.951883+0000","last_fullsized":"2024-03-29T11:34:19.951883+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:34.599290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966295+0000","last_change":"2024-03-29T11:34:19.961549+0000","last_active":"2024-03-29T11:34:19.966295+0000","last_peered":"2024-03-29T11:34:19.966295+0000","last_clean":"2024-03-29T11:34:19.966295+0000","last_became_active":"2024-03-29T11:34:19.961039+0000","last_became_peered":"2024-03-29T11:34:19.961039+0000","last_unstale":"2024-03-29T11:34:19.966295+0000","last_undegraded":"2024-03-29T11:34:19.966295+0000","last_fullsized":"2024-03-29T11:34:19.966295+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:01:02.478752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951834+0000","last_change":"2024-03-29T11:34:19.944494+0000","last_active":"2024-03-29T11:34:19.951834+0000","last_peered":"2024-03-29T11:34:19.951834+0000","last_clean":"2024-03-29T11:34:19.951834+0000","last_became_active":"2024-03-29T11:34:19.944207+0000","last_became_peered":"2024-03-29T11:34:19.944207+0000","last_unstale":"2024-03-29T11:34:19.951834+0000","last_undegraded":"2024-03-29T11:34:19.951834+0000","last_fullsized":"2024-03-29T11:34:19.951834+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:01:04.821333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940953+0000","last_change":"2024-03-29T11:34:19.958166+0000","last_active":"2024-03-29T11:34:20.940953+0000","last_peered":"2024-03-29T11:34:20.940953+0000","last_clean":"2024-03-29T11:34:20.940953+0000","last_became_active":"2024-03-29T11:34:19.956429+0000","last_became_peered":"2024-03-29T11:34:19.956429+0000","last_unstale":"2024-03-29T11:34:20.940953+0000","last_undegraded":"2024-03-29T11:34:20.940953+0000","last_fullsized":"2024-03-29T11:34:20.940953+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:00:03.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940933+0000","last_change":"2024-03-29T11:34:19.944065+0000","last_active":"2024-03-29T11:34:20.940933+0000","last_peered":"2024-03-29T11:34:20.940933+0000","last_clean":"2024-03-29T11:34:20.940933+0000","last_became_active":"2024-03-29T11:34:19.943800+0000","last_became_peered":"2024-03-29T11:34:19.943800+0000","last_unstale":"2024-03-29T11:34:20.940933+0000","last_undegraded":"2024-03-29T11:34:20.940933+0000","last_fullsized":"2024-03-29T11:34:20.940933+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:41.492915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952415+0000","last_change":"2024-03-29T11:34:19.948092+0000","last_active":"2024-03-29T11:34:19.952415+0000","last_peered":"2024-03-29T11:34:19.952415+0000","last_clean":"2024-03-29T11:34:19.952415+0000","last_became_active":"2024-03-29T11:34:19.947764+0000","last_became_peered":"2024-03-29T11:34:19.947764+0000","last_unstale":"2024-03-29T11:34:19.952415+0000","last_undegraded":"2024-03-29T11:34:19.952415+0000","last_fullsized":"2024-03-29T11:34:19.952415+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:45:56.716334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.960108+0000","last_change":"2024-03-29T11:34:19.959273+0000","last_active":"2024-03-29T11:34:19.960108+0000","last_peered":"2024-03-29T11:34:19.960108+0000","last_clean":"2024-03-29T11:34:19.960108+0000","last_became_active":"2024-03-29T11:34:19.957026+0000","last_became_peered":"2024-03-29T11:34:19.957026+0000","last_unstale":"2024-03-29T11:34:19.960108+0000","last_undegraded":"2024-03-29T11:34:19.960108+0000","last_fullsized":"2024-03-29T11:34:19.960108+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:42.183164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934797+0000","last_change":"2024-03-29T11:34:19.949953+0000","last_active":"2024-03-29T11:34:20.934797+0000","last_peered":"2024-03-29T11:34:20.934797+0000","last_clean":"2024-03-29T11:34:20.934797+0000","last_became_active":"2024-03-29T11:34:19.949671+0000","last_became_peered":"2024-03-29T11:34:19.949671+0000","last_unstale":"2024-03-29T11:34:20.934797+0000","last_undegraded":"2024-03-29T11:34:20.934797+0000","last_fullsized":"2024-03-29T11:34:20.934797+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:44.804649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.965574+0000","last_change":"2024-03-29T11:34:19.960624+0000","last_active":"2024-03-29T11:34:19.965574+0000","last_peered":"2024-03-29T11:34:19.965574+0000","last_clean":"2024-03-29T11:34:19.965574+0000","last_became_active":"2024-03-29T11:34:19.958146+0000","last_became_peered":"2024-03-29T11:34:19.958146+0000","last_unstale":"2024-03-29T11:34:19.965574+0000","last_undegraded":"2024-03-29T11:34:19.965574+0000","last_fullsized":"2024-03-29T11:34:19.965574+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:20:24.328362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940495+0000","last_change":"2024-03-29T11:34:19.940339+0000","last_active":"2024-03-29T11:34:20.940495+0000","last_peered":"2024-03-29T11:34:20.940495+0000","last_clean":"2024-03-29T11:34:20.940495+0000","last_became_active":"2024-03-29T11:34:19.939546+0000","last_became_peered":"2024-03-29T11:34:19.939546+0000","last_unstale":"2024-03-29T11:34:20.940495+0000","last_undegraded":"2024-03-29T11:34:20.940495+0000","last_fullsized":"2024-03-29T11:34:20.940495+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:08:12.571462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.953889+0000","last_change":"2024-03-29T11:34:19.950546+0000","last_active":"2024-03-29T11:34:19.953889+0000","last_peered":"2024-03-29T11:34:19.953889+0000","last_clean":"2024-03-29T11:34:19.953889+0000","last_became_active":"2024-03-29T11:34:19.950027+0000","last_became_peered":"2024-03-29T11:34:19.950027+0000","last_unstale":"2024-03-29T11:34:19.953889+0000","last_undegraded":"2024-03-29T11:34:19.953889+0000","last_fullsized":"2024-03-29T11:34:19.953889+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:55.397955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941047+0000","last_change":"2024-03-29T11:34:19.948467+0000","last_active":"2024-03-29T11:34:20.941047+0000","last_peered":"2024-03-29T11:34:20.941047+0000","last_clean":"2024-03-29T11:34:20.941047+0000","last_became_active":"2024-03-29T11:34:19.947021+0000","last_became_peered":"2024-03-29T11:34:19.947021+0000","last_unstale":"2024-03-29T11:34:20.941047+0000","last_undegraded":"2024-03-29T11:34:20.941047+0000","last_fullsized":"2024-03-29T11:34:20.941047+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:18:34.673238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941389+0000","last_change":"2024-03-29T11:34:19.950495+0000","last_active":"2024-03-29T11:34:20.941389+0000","last_peered":"2024-03-29T11:34:20.941389+0000","last_clean":"2024-03-29T11:34:20.941389+0000","last_became_active":"2024-03-29T11:34:19.950222+0000","last_became_peered":"2024-03-29T11:34:19.950222+0000","last_unstale":"2024-03-29T11:34:20.941389+0000","last_undegraded":"2024-03-29T11:34:20.941389+0000","last_fullsized":"2024-03-29T11:34:20.941389+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:34:17.322355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.954476+0000","last_change":"2024-03-29T11:34:19.949613+0000","last_active":"2024-03-29T11:34:19.954476+0000","last_peered":"2024-03-29T11:34:19.954476+0000","last_clean":"2024-03-29T11:34:19.954476+0000","last_became_active":"2024-03-29T11:34:19.949347+0000","last_became_peered":"2024-03-29T11:34:19.949347+0000","last_unstale":"2024-03-29T11:34:19.954476+0000","last_undegraded":"2024-03-29T11:34:19.954476+0000","last_fullsized":"2024-03-29T11:34:19.954476+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:05:16.963592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.959036+0000","last_change":"2024-03-29T11:34:19.956011+0000","last_active":"2024-03-29T11:34:19.959036+0000","last_peered":"2024-03-29T11:34:19.959036+0000","last_clean":"2024-03-29T11:34:19.959036+0000","last_became_active":"2024-03-29T11:34:19.955710+0000","last_became_peered":"2024-03-29T11:34:19.955710+0000","last_unstale":"2024-03-29T11:34:19.959036+0000","last_undegraded":"2024-03-29T11:34:19.959036+0000","last_fullsized":"2024-03-29T11:34:19.959036+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:39:11.647026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940353+0000","last_change":"2024-03-29T11:34:19.952876+0000","last_active":"2024-03-29T11:34:20.940353+0000","last_peered":"2024-03-29T11:34:20.940353+0000","last_clean":"2024-03-29T11:34:20.940353+0000","last_became_active":"2024-03-29T11:34:19.952595+0000","last_became_peered":"2024-03-29T11:34:19.952595+0000","last_unstale":"2024-03-29T11:34:20.940353+0000","last_undegraded":"2024-03-29T11:34:20.940353+0000","last_fullsized":"2024-03-29T11:34:20.940353+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:22:56.694591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.953144+0000","last_change":"2024-03-29T11:34:19.948122+0000","last_active":"2024-03-29T11:34:19.953144+0000","last_peered":"2024-03-29T11:34:19.953144+0000","last_clean":"2024-03-29T11:34:19.953144+0000","last_became_active":"2024-03-29T11:34:19.947828+0000","last_became_peered":"2024-03-29T11:34:19.947828+0000","last_unstale":"2024-03-29T11:34:19.953144+0000","last_undegraded":"2024-03-29T11:34:19.953144+0000","last_fullsized":"2024-03-29T11:34:19.953144+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:29.853902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.965752+0000","last_change":"2024-03-29T11:34:19.960388+0000","last_active":"2024-03-29T11:34:19.965752+0000","last_peered":"2024-03-29T11:34:19.965752+0000","last_clean":"2024-03-29T11:34:19.965752+0000","last_became_active":"2024-03-29T11:34:19.957211+0000","last_became_peered":"2024-03-29T11:34:19.957211+0000","last_unstale":"2024-03-29T11:34:19.965752+0000","last_undegraded":"2024-03-29T11:34:19.965752+0000","last_fullsized":"2024-03-29T11:34:19.965752+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:03.223791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.953105+0000","last_change":"2024-03-29T11:34:19.948713+0000","last_active":"2024-03-29T11:34:19.953105+0000","last_peered":"2024-03-29T11:34:19.953105+0000","last_clean":"2024-03-29T11:34:19.953105+0000","last_became_active":"2024-03-29T11:34:19.948434+0000","last_became_peered":"2024-03-29T11:34:19.948434+0000","last_unstale":"2024-03-29T11:34:19.953105+0000","last_undegraded":"2024-03-29T11:34:19.953105+0000","last_fullsized":"2024-03-29T11:34:19.953105+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:50:19.403032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"282'4","reported_seq":14,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952910+0000","last_change":"2024-03-29T11:34:19.431952+0000","last_active":"2024-03-29T11:34:19.952910+0000","last_peered":"2024-03-29T11:34:19.952910+0000","last_clean":"2024-03-29T11:34:19.952910+0000","last_became_active":"2024-03-29T11:34:19.431642+0000","last_became_peered":"2024-03-29T11:34:19.431642+0000","last_unstale":"2024-03-29T11:34:19.952910+0000","last_undegraded":"2024-03-29T11:34:19.952910+0000","last_fullsized":"2024-03-29T11:34:19.952910+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:29:32.163286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.954515+0000","last_change":"2024-03-29T11:34:19.950486+0000","last_active":"2024-03-29T11:34:19.954515+0000","last_peered":"2024-03-29T11:34:19.954515+0000","last_clean":"2024-03-29T11:34:19.954515+0000","last_became_active":"2024-03-29T11:34:19.950168+0000","last_became_peered":"2024-03-29T11:34:19.950168+0000","last_unstale":"2024-03-29T11:34:19.954515+0000","last_undegraded":"2024-03-29T11:34:19.954515+0000","last_fullsized":"2024-03-29T11:34:19.954515+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:26.561837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"282'1","reported_seq":11,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.958736+0000","last_change":"2024-03-29T11:34:19.432996+0000","last_active":"2024-03-29T11:34:19.958736+0000","last_peered":"2024-03-29T11:34:19.958736+0000","last_clean":"2024-03-29T11:34:19.958736+0000","last_became_active":"2024-03-29T11:34:19.432616+0000","last_became_peered":"2024-03-29T11:34:19.432616+0000","last_unstale":"2024-03-29T11:34:19.958736+0000","last_undegraded":"2024-03-29T11:34:19.958736+0000","last_fullsized":"2024-03-29T11:34:19.958736+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:25.549951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.954241+0000","last_change":"2024-03-29T11:34:19.946817+0000","last_active":"2024-03-29T11:34:19.954241+0000","last_peered":"2024-03-29T11:34:19.954241+0000","last_clean":"2024-03-29T11:34:19.954241+0000","last_became_active":"2024-03-29T11:34:19.946581+0000","last_became_peered":"2024-03-29T11:34:19.946581+0000","last_unstale":"2024-03-29T11:34:19.954241+0000","last_undegraded":"2024-03-29T11:34:19.954241+0000","last_fullsized":"2024-03-29T11:34:19.954241+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:14:15.546608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"282'2","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.957681+0000","last_change":"2024-03-29T11:34:19.427717+0000","last_active":"2024-03-29T11:34:19.957681+0000","last_peered":"2024-03-29T11:34:19.957681+0000","last_clean":"2024-03-29T11:34:19.957681+0000","last_became_active":"2024-03-29T11:34:19.427476+0000","last_became_peered":"2024-03-29T11:34:19.427476+0000","last_unstale":"2024-03-29T11:34:19.957681+0000","last_undegraded":"2024-03-29T11:34:19.957681+0000","last_fullsized":"2024-03-29T11:34:19.957681+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:11:23.474692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934976+0000","last_change":"2024-03-29T11:34:19.956656+0000","last_active":"2024-03-29T11:34:20.934976+0000","last_peered":"2024-03-29T11:34:20.934976+0000","last_clean":"2024-03-29T11:34:20.934976+0000","last_became_active":"2024-03-29T11:34:19.955318+0000","last_became_peered":"2024-03-29T11:34:19.955318+0000","last_unstale":"2024-03-29T11:34:20.934976+0000","last_undegraded":"2024-03-29T11:34:20.934976+0000","last_fullsized":"2024-03-29T11:34:20.934976+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:26.561837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"282'2","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.963389+0000","last_change":"2024-03-29T11:34:19.429445+0000","last_active":"2024-03-29T11:34:19.963389+0000","last_peered":"2024-03-29T11:34:19.963389+0000","last_clean":"2024-03-29T11:34:19.963389+0000","last_became_active":"2024-03-29T11:34:19.427486+0000","last_became_peered":"2024-03-29T11:34:19.427486+0000","last_unstale":"2024-03-29T11:34:19.963389+0000","last_undegraded":"2024-03-29T11:34:19.963389+0000","last_fullsized":"2024-03-29T11:34:19.963389+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:04:29.034899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.935094+0000","last_change":"2024-03-29T11:34:19.959920+0000","last_active":"2024-03-29T11:34:20.935094+0000","last_peered":"2024-03-29T11:34:20.935094+0000","last_clean":"2024-03-29T11:34:20.935094+0000","last_became_active":"2024-03-29T11:34:19.958971+0000","last_became_peered":"2024-03-29T11:34:19.958971+0000","last_unstale":"2024-03-29T11:34:20.935094+0000","last_undegraded":"2024-03-29T11:34:20.935094+0000","last_fullsized":"2024-03-29T11:34:20.935094+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:15:57.794980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"282'1","reported_seq":11,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.956845+0000","last_change":"2024-03-29T11:34:19.429640+0000","last_active":"2024-03-29T11:34:19.956845+0000","last_peered":"2024-03-29T11:34:19.956845+0000","last_clean":"2024-03-29T11:34:19.956845+0000","last_became_active":"2024-03-29T11:34:19.429417+0000","last_became_peered":"2024-03-29T11:34:19.429417+0000","last_unstale":"2024-03-29T11:34:19.956845+0000","last_undegraded":"2024-03-29T11:34:19.956845+0000","last_fullsized":"2024-03-29T11:34:19.956845+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:33.587404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951198+0000","last_change":"2024-03-29T11:34:19.947653+0000","last_active":"2024-03-29T11:34:19.951198+0000","last_peered":"2024-03-29T11:34:19.951198+0000","last_clean":"2024-03-29T11:34:19.951198+0000","last_became_active":"2024-03-29T11:34:19.947279+0000","last_became_peered":"2024-03-29T11:34:19.947279+0000","last_unstale":"2024-03-29T11:34:19.951198+0000","last_undegraded":"2024-03-29T11:34:19.951198+0000","last_fullsized":"2024-03-29T11:34:19.951198+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:08:57.572725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940458+0000","last_change":"2024-03-29T11:34:19.427504+0000","last_active":"2024-03-29T11:34:20.940458+0000","last_peered":"2024-03-29T11:34:20.940458+0000","last_clean":"2024-03-29T11:34:20.940458+0000","last_became_active":"2024-03-29T11:34:19.426820+0000","last_became_peered":"2024-03-29T11:34:19.426820+0000","last_unstale":"2024-03-29T11:34:20.940458+0000","last_undegraded":"2024-03-29T11:34:20.940458+0000","last_fullsized":"2024-03-29T11:34:20.940458+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:06:37.241290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940656+0000","last_change":"2024-03-29T11:34:19.954850+0000","last_active":"2024-03-29T11:34:20.940656+0000","last_peered":"2024-03-29T11:34:20.940656+0000","last_clean":"2024-03-29T11:34:20.940656+0000","last_became_active":"2024-03-29T11:34:19.953319+0000","last_became_peered":"2024-03-29T11:34:19.953319+0000","last_unstale":"2024-03-29T11:34:20.940656+0000","last_undegraded":"2024-03-29T11:34:20.940656+0000","last_fullsized":"2024-03-29T11:34:20.940656+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:32:52.022464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.941138+0000","last_change":"2024-03-29T11:34:19.428401+0000","last_active":"2024-03-29T11:34:19.941138+0000","last_peered":"2024-03-29T11:34:19.941138+0000","last_clean":"2024-03-29T11:34:19.941138+0000","last_became_active":"2024-03-29T11:34:19.428164+0000","last_became_peered":"2024-03-29T11:34:19.428164+0000","last_unstale":"2024-03-29T11:34:19.941138+0000","last_undegraded":"2024-03-29T11:34:19.941138+0000","last_fullsized":"2024-03-29T11:34:19.941138+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:45:55.704448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951969+0000","last_change":"2024-03-29T11:34:19.946373+0000","last_active":"2024-03-29T11:34:19.951969+0000","last_peered":"2024-03-29T11:34:19.951969+0000","last_clean":"2024-03-29T11:34:19.951969+0000","last_became_active":"2024-03-29T11:34:19.946072+0000","last_became_peered":"2024-03-29T11:34:19.946072+0000","last_unstale":"2024-03-29T11:34:19.951969+0000","last_undegraded":"2024-03-29T11:34:19.951969+0000","last_fullsized":"2024-03-29T11:34:19.951969+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:49:03.223791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.929819+0000","last_change":"2024-03-29T11:34:19.426889+0000","last_active":"2024-03-29T11:34:19.929819+0000","last_peered":"2024-03-29T11:34:19.929819+0000","last_clean":"2024-03-29T11:34:19.929819+0000","last_became_active":"2024-03-29T11:34:19.426616+0000","last_became_peered":"2024-03-29T11:34:19.426616+0000","last_unstale":"2024-03-29T11:34:19.929819+0000","last_undegraded":"2024-03-29T11:34:19.929819+0000","last_fullsized":"2024-03-29T11:34:19.929819+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:21.911782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.945362+0000","last_change":"2024-03-29T11:34:19.943054+0000","last_active":"2024-03-29T11:34:19.945362+0000","last_peered":"2024-03-29T11:34:19.945362+0000","last_clean":"2024-03-29T11:34:19.945362+0000","last_became_active":"2024-03-29T11:34:19.942728+0000","last_became_peered":"2024-03-29T11:34:19.942728+0000","last_unstale":"2024-03-29T11:34:19.945362+0000","last_undegraded":"2024-03-29T11:34:19.945362+0000","last_fullsized":"2024-03-29T11:34:19.945362+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:01:56.645123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.936914+0000","last_change":"2024-03-29T11:34:19.428484+0000","last_active":"2024-03-29T11:34:19.936914+0000","last_peered":"2024-03-29T11:34:19.936914+0000","last_clean":"2024-03-29T11:34:19.936914+0000","last_became_active":"2024-03-29T11:34:19.427975+0000","last_became_peered":"2024-03-29T11:34:19.427975+0000","last_unstale":"2024-03-29T11:34:19.936914+0000","last_undegraded":"2024-03-29T11:34:19.936914+0000","last_fullsized":"2024-03-29T11:34:19.936914+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:56:10.313615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952850+0000","last_change":"2024-03-29T11:34:19.946891+0000","last_active":"2024-03-29T11:34:19.952850+0000","last_peered":"2024-03-29T11:34:19.952850+0000","last_clean":"2024-03-29T11:34:19.952850+0000","last_became_active":"2024-03-29T11:34:19.946557+0000","last_became_peered":"2024-03-29T11:34:19.946557+0000","last_unstale":"2024-03-29T11:34:19.952850+0000","last_undegraded":"2024-03-29T11:34:19.952850+0000","last_fullsized":"2024-03-29T11:34:19.952850+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:47.091696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.935838+0000","last_change":"2024-03-29T11:34:19.428419+0000","last_active":"2024-03-29T11:34:19.935838+0000","last_peered":"2024-03-29T11:34:19.935838+0000","last_clean":"2024-03-29T11:34:19.935838+0000","last_became_active":"2024-03-29T11:34:19.428100+0000","last_became_peered":"2024-03-29T11:34:19.428100+0000","last_unstale":"2024-03-29T11:34:19.935838+0000","last_undegraded":"2024-03-29T11:34:19.935838+0000","last_fullsized":"2024-03-29T11:34:19.935838+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:35:10.209538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966442+0000","last_change":"2024-03-29T11:34:19.964236+0000","last_active":"2024-03-29T11:34:19.966442+0000","last_peered":"2024-03-29T11:34:19.966442+0000","last_clean":"2024-03-29T11:34:19.966442+0000","last_became_active":"2024-03-29T11:34:19.961413+0000","last_became_peered":"2024-03-29T11:34:19.961413+0000","last_unstale":"2024-03-29T11:34:19.966442+0000","last_undegraded":"2024-03-29T11:34:19.966442+0000","last_fullsized":"2024-03-29T11:34:19.966442+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:23:05.855510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"282'3","reported_seq":13,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.963330+0000","last_change":"2024-03-29T11:34:19.429938+0000","last_active":"2024-03-29T11:34:19.963330+0000","last_peered":"2024-03-29T11:34:19.963330+0000","last_clean":"2024-03-29T11:34:19.963330+0000","last_became_active":"2024-03-29T11:34:19.429114+0000","last_became_peered":"2024-03-29T11:34:19.429114+0000","last_unstale":"2024-03-29T11:34:19.963330+0000","last_undegraded":"2024-03-29T11:34:19.963330+0000","last_fullsized":"2024-03-29T11:34:19.963330+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:51:50.580234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952367+0000","last_change":"2024-03-29T11:34:19.946033+0000","last_active":"2024-03-29T11:34:19.952367+0000","last_peered":"2024-03-29T11:34:19.952367+0000","last_clean":"2024-03-29T11:34:19.952367+0000","last_became_active":"2024-03-29T11:34:19.945746+0000","last_became_peered":"2024-03-29T11:34:19.945746+0000","last_unstale":"2024-03-29T11:34:19.952367+0000","last_undegraded":"2024-03-29T11:34:19.952367+0000","last_fullsized":"2024-03-29T11:34:19.952367+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:23:05.855510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"282'6","reported_seq":17,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940160+0000","last_change":"2024-03-29T11:34:19.426573+0000","last_active":"2024-03-29T11:34:20.940160+0000","last_peered":"2024-03-29T11:34:20.940160+0000","last_clean":"2024-03-29T11:34:20.940160+0000","last_became_active":"2024-03-29T11:34:19.426257+0000","last_became_peered":"2024-03-29T11:34:19.426257+0000","last_unstale":"2024-03-29T11:34:20.940160+0000","last_undegraded":"2024-03-29T11:34:20.940160+0000","last_fullsized":"2024-03-29T11:34:20.940160+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:01:03.809447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951863+0000","last_change":"2024-03-29T11:34:19.943204+0000","last_active":"2024-03-29T11:34:19.951863+0000","last_peered":"2024-03-29T11:34:19.951863+0000","last_clean":"2024-03-29T11:34:19.951863+0000","last_became_active":"2024-03-29T11:34:19.942294+0000","last_became_peered":"2024-03-29T11:34:19.942294+0000","last_unstale":"2024-03-29T11:34:19.951863+0000","last_undegraded":"2024-03-29T11:34:19.951863+0000","last_fullsized":"2024-03-29T11:34:19.951863+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:51:51.592120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934915+0000","last_change":"2024-03-29T11:34:19.434932+0000","last_active":"2024-03-29T11:34:20.934915+0000","last_peered":"2024-03-29T11:34:20.934915+0000","last_clean":"2024-03-29T11:34:20.934915+0000","last_became_active":"2024-03-29T11:34:19.434640+0000","last_became_peered":"2024-03-29T11:34:19.434640+0000","last_unstale":"2024-03-29T11:34:20.934915+0000","last_undegraded":"2024-03-29T11:34:20.934915+0000","last_fullsized":"2024-03-29T11:34:20.934915+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:05:39.490318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940693+0000","last_change":"2024-03-29T11:34:19.950379+0000","last_active":"2024-03-29T11:34:20.940693+0000","last_peered":"2024-03-29T11:34:20.940693+0000","last_clean":"2024-03-29T11:34:20.940693+0000","last_became_active":"2024-03-29T11:34:19.948304+0000","last_became_peered":"2024-03-29T11:34:19.948304+0000","last_unstale":"2024-03-29T11:34:20.940693+0000","last_undegraded":"2024-03-29T11:34:20.940693+0000","last_fullsized":"2024-03-29T11:34:20.940693+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:02:35.771545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.935906+0000","last_change":"2024-03-29T11:34:19.433087+0000","last_active":"2024-03-29T11:34:19.935906+0000","last_peered":"2024-03-29T11:34:19.935906+0000","last_clean":"2024-03-29T11:34:19.935906+0000","last_became_active":"2024-03-29T11:34:19.432780+0000","last_became_peered":"2024-03-29T11:34:19.432780+0000","last_unstale":"2024-03-29T11:34:19.935906+0000","last_undegraded":"2024-03-29T11:34:19.935906+0000","last_fullsized":"2024-03-29T11:34:19.935906+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:11:23.474692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.960348+0000","last_change":"2024-03-29T11:34:19.959501+0000","last_active":"2024-03-29T11:34:19.960348+0000","last_peered":"2024-03-29T11:34:19.960348+0000","last_clean":"2024-03-29T11:34:19.960348+0000","last_became_active":"2024-03-29T11:34:19.959137+0000","last_became_peered":"2024-03-29T11:34:19.959137+0000","last_unstale":"2024-03-29T11:34:19.960348+0000","last_undegraded":"2024-03-29T11:34:19.960348+0000","last_fullsized":"2024-03-29T11:34:19.960348+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:06:38.253176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940559+0000","last_change":"2024-03-29T11:34:19.422959+0000","last_active":"2024-03-29T11:34:20.940559+0000","last_peered":"2024-03-29T11:34:20.940559+0000","last_clean":"2024-03-29T11:34:20.940559+0000","last_became_active":"2024-03-29T11:34:19.422665+0000","last_became_peered":"2024-03-29T11:34:19.422665+0000","last_unstale":"2024-03-29T11:34:20.940559+0000","last_undegraded":"2024-03-29T11:34:20.940559+0000","last_fullsized":"2024-03-29T11:34:20.940559+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T23:05:15.951706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940088+0000","last_change":"2024-03-29T11:34:19.939914+0000","last_active":"2024-03-29T11:34:20.940088+0000","last_peered":"2024-03-29T11:34:20.940088+0000","last_clean":"2024-03-29T11:34:20.940088+0000","last_became_active":"2024-03-29T11:34:19.939187+0000","last_became_peered":"2024-03-29T11:34:19.939187+0000","last_unstale":"2024-03-29T11:34:20.940088+0000","last_undegraded":"2024-03-29T11:34:20.940088+0000","last_fullsized":"2024-03-29T11:34:20.940088+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:49:41.278594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"282'8","reported_seq":18,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966527+0000","last_change":"2024-03-29T11:34:19.432536+0000","last_active":"2024-03-29T11:34:19.966527+0000","last_peered":"2024-03-29T11:34:19.966527+0000","last_clean":"2024-03-29T11:34:19.966527+0000","last_became_active":"2024-03-29T11:34:19.432074+0000","last_became_peered":"2024-03-29T11:34:19.432074+0000","last_unstale":"2024-03-29T11:34:19.966527+0000","last_undegraded":"2024-03-29T11:34:19.966527+0000","last_fullsized":"2024-03-29T11:34:19.966527+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:21.911782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.957538+0000","last_change":"2024-03-29T11:34:19.952465+0000","last_active":"2024-03-29T11:34:19.957538+0000","last_peered":"2024-03-29T11:34:19.957538+0000","last_clean":"2024-03-29T11:34:19.957538+0000","last_became_active":"2024-03-29T11:34:19.951484+0000","last_became_peered":"2024-03-29T11:34:19.951484+0000","last_unstale":"2024-03-29T11:34:19.957538+0000","last_undegraded":"2024-03-29T11:34:19.957538+0000","last_fullsized":"2024-03-29T11:34:19.957538+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:41.492915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934837+0000","last_change":"2024-03-29T11:34:19.428942+0000","last_active":"2024-03-29T11:34:20.934837+0000","last_peered":"2024-03-29T11:34:20.934837+0000","last_clean":"2024-03-29T11:34:20.934837+0000","last_became_active":"2024-03-29T11:34:19.428635+0000","last_became_peered":"2024-03-29T11:34:19.428635+0000","last_unstale":"2024-03-29T11:34:20.934837+0000","last_undegraded":"2024-03-29T11:34:20.934837+0000","last_fullsized":"2024-03-29T11:34:20.934837+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:46:46.079810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.954963+0000","last_change":"2024-03-29T11:34:19.950578+0000","last_active":"2024-03-29T11:34:19.954963+0000","last_peered":"2024-03-29T11:34:19.954963+0000","last_clean":"2024-03-29T11:34:19.954963+0000","last_became_active":"2024-03-29T11:34:19.949954+0000","last_became_peered":"2024-03-29T11:34:19.949954+0000","last_unstale":"2024-03-29T11:34:19.954963+0000","last_undegraded":"2024-03-29T11:34:19.954963+0000","last_fullsized":"2024-03-29T11:34:19.954963+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:11:24.475618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"282'2","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.951861+0000","last_change":"2024-03-29T11:34:19.433823+0000","last_active":"2024-03-29T11:34:19.951861+0000","last_peered":"2024-03-29T11:34:19.951861+0000","last_clean":"2024-03-29T11:34:19.951861+0000","last_became_active":"2024-03-29T11:34:19.433451+0000","last_became_peered":"2024-03-29T11:34:19.433451+0000","last_unstale":"2024-03-29T11:34:19.951861+0000","last_undegraded":"2024-03-29T11:34:19.951861+0000","last_fullsized":"2024-03-29T11:34:19.951861+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:19:25.412060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.958958+0000","last_change":"2024-03-29T11:34:19.955022+0000","last_active":"2024-03-29T11:34:19.958958+0000","last_peered":"2024-03-29T11:34:19.958958+0000","last_clean":"2024-03-29T11:34:19.958958+0000","last_became_active":"2024-03-29T11:34:19.952831+0000","last_became_peered":"2024-03-29T11:34:19.952831+0000","last_unstale":"2024-03-29T11:34:19.958958+0000","last_undegraded":"2024-03-29T11:34:19.958958+0000","last_fullsized":"2024-03-29T11:34:19.958958+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:51:08.112025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"282'1","reported_seq":11,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.959866+0000","last_change":"2024-03-29T11:34:19.426275+0000","last_active":"2024-03-29T11:34:19.959866+0000","last_peered":"2024-03-29T11:34:19.959866+0000","last_clean":"2024-03-29T11:34:19.959866+0000","last_became_active":"2024-03-29T11:34:19.423168+0000","last_became_peered":"2024-03-29T11:34:19.423168+0000","last_unstale":"2024-03-29T11:34:19.959866+0000","last_undegraded":"2024-03-29T11:34:19.959866+0000","last_fullsized":"2024-03-29T11:34:19.959866+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:43.792763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952391+0000","last_change":"2024-03-29T11:34:19.947805+0000","last_active":"2024-03-29T11:34:19.952391+0000","last_peered":"2024-03-29T11:34:19.952391+0000","last_clean":"2024-03-29T11:34:19.952391+0000","last_became_active":"2024-03-29T11:34:19.947415+0000","last_became_peered":"2024-03-29T11:34:19.947415+0000","last_unstale":"2024-03-29T11:34:19.952391+0000","last_undegraded":"2024-03-29T11:34:19.952391+0000","last_fullsized":"2024-03-29T11:34:19.952391+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:02:22.923669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.931629+0000","last_change":"2024-03-29T11:34:19.425570+0000","last_active":"2024-03-29T11:34:19.931629+0000","last_peered":"2024-03-29T11:34:19.931629+0000","last_clean":"2024-03-29T11:34:19.931629+0000","last_became_active":"2024-03-29T11:34:19.425286+0000","last_became_peered":"2024-03-29T11:34:19.425286+0000","last_unstale":"2024-03-29T11:34:19.931629+0000","last_undegraded":"2024-03-29T11:34:19.931629+0000","last_fullsized":"2024-03-29T11:34:19.931629+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:04:29.034899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.952533+0000","last_change":"2024-03-29T11:34:19.945718+0000","last_active":"2024-03-29T11:34:19.952533+0000","last_peered":"2024-03-29T11:34:19.952533+0000","last_clean":"2024-03-29T11:34:19.952533+0000","last_became_active":"2024-03-29T11:34:19.945423+0000","last_became_peered":"2024-03-29T11:34:19.945423+0000","last_unstale":"2024-03-29T11:34:19.952533+0000","last_undegraded":"2024-03-29T11:34:19.952533+0000","last_fullsized":"2024-03-29T11:34:19.952533+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:43:25.035704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"282'2","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.963294+0000","last_change":"2024-03-29T11:34:19.428704+0000","last_active":"2024-03-29T11:34:19.963294+0000","last_peered":"2024-03-29T11:34:19.963294+0000","last_clean":"2024-03-29T11:34:19.963294+0000","last_became_active":"2024-03-29T11:34:19.428125+0000","last_became_peered":"2024-03-29T11:34:19.428125+0000","last_unstale":"2024-03-29T11:34:19.963294+0000","last_undegraded":"2024-03-29T11:34:19.963294+0000","last_fullsized":"2024-03-29T11:34:19.963294+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:45:55.704448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.955691+0000","last_change":"2024-03-29T11:34:19.953791+0000","last_active":"2024-03-29T11:34:19.955691+0000","last_peered":"2024-03-29T11:34:19.955691+0000","last_clean":"2024-03-29T11:34:19.955691+0000","last_became_active":"2024-03-29T11:34:19.952430+0000","last_became_peered":"2024-03-29T11:34:19.952430+0000","last_unstale":"2024-03-29T11:34:19.955691+0000","last_undegraded":"2024-03-29T11:34:19.955691+0000","last_fullsized":"2024-03-29T11:34:19.955691+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:24:08.808042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.935169+0000","last_change":"2024-03-29T11:34:19.424373+0000","last_active":"2024-03-29T11:34:20.935169+0000","last_peered":"2024-03-29T11:34:20.935169+0000","last_clean":"2024-03-29T11:34:20.935169+0000","last_became_active":"2024-03-29T11:34:19.423585+0000","last_became_peered":"2024-03-29T11:34:19.423585+0000","last_unstale":"2024-03-29T11:34:20.935169+0000","last_undegraded":"2024-03-29T11:34:20.935169+0000","last_fullsized":"2024-03-29T11:34:20.935169+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:25:28.076825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934479+0000","last_change":"2024-03-29T11:34:19.954942+0000","last_active":"2024-03-29T11:34:20.934479+0000","last_peered":"2024-03-29T11:34:20.934479+0000","last_clean":"2024-03-29T11:34:20.934479+0000","last_became_active":"2024-03-29T11:34:19.954660+0000","last_became_peered":"2024-03-29T11:34:19.954660+0000","last_unstale":"2024-03-29T11:34:20.934479+0000","last_undegraded":"2024-03-29T11:34:20.934479+0000","last_fullsized":"2024-03-29T11:34:20.934479+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:50:19.403032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.930332+0000","last_change":"2024-03-29T11:34:19.428156+0000","last_active":"2024-03-29T11:34:19.930332+0000","last_peered":"2024-03-29T11:34:19.930332+0000","last_clean":"2024-03-29T11:34:19.930332+0000","last_became_active":"2024-03-29T11:34:19.427900+0000","last_became_peered":"2024-03-29T11:34:19.427900+0000","last_unstale":"2024-03-29T11:34:19.930332+0000","last_undegraded":"2024-03-29T11:34:19.930332+0000","last_fullsized":"2024-03-29T11:34:19.930332+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:54.386068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934544+0000","last_change":"2024-03-29T11:34:19.947167+0000","last_active":"2024-03-29T11:34:20.934544+0000","last_peered":"2024-03-29T11:34:20.934544+0000","last_clean":"2024-03-29T11:34:20.934544+0000","last_became_active":"2024-03-29T11:34:19.946910+0000","last_became_peered":"2024-03-29T11:34:19.946910+0000","last_unstale":"2024-03-29T11:34:20.934544+0000","last_undegraded":"2024-03-29T11:34:20.934544+0000","last_fullsized":"2024-03-29T11:34:20.934544+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:39:40.860490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934570+0000","last_change":"2024-03-29T11:34:19.423695+0000","last_active":"2024-03-29T11:34:20.934570+0000","last_peered":"2024-03-29T11:34:20.934570+0000","last_clean":"2024-03-29T11:34:20.934570+0000","last_became_active":"2024-03-29T11:34:19.423386+0000","last_became_peered":"2024-03-29T11:34:19.423386+0000","last_unstale":"2024-03-29T11:34:20.934570+0000","last_undegraded":"2024-03-29T11:34:20.934570+0000","last_fullsized":"2024-03-29T11:34:20.934570+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:59:14.486166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.959882+0000","last_change":"2024-03-29T11:34:19.957275+0000","last_active":"2024-03-29T11:34:19.959882+0000","last_peered":"2024-03-29T11:34:19.959882+0000","last_clean":"2024-03-29T11:34:19.959882+0000","last_became_active":"2024-03-29T11:34:19.957003+0000","last_became_peered":"2024-03-29T11:34:19.957003+0000","last_unstale":"2024-03-29T11:34:19.959882+0000","last_undegraded":"2024-03-29T11:34:19.959882+0000","last_fullsized":"2024-03-29T11:34:19.959882+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:47:39.749122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"282'2","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.960690+0000","last_change":"2024-03-29T11:34:19.427149+0000","last_active":"2024-03-29T11:34:19.960690+0000","last_peered":"2024-03-29T11:34:19.960690+0000","last_clean":"2024-03-29T11:34:19.960690+0000","last_became_active":"2024-03-29T11:34:19.426789+0000","last_became_peered":"2024-03-29T11:34:19.426789+0000","last_unstale":"2024-03-29T11:34:19.960690+0000","last_undegraded":"2024-03-29T11:34:19.960690+0000","last_fullsized":"2024-03-29T11:34:19.960690+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:17:33.587404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941051+0000","last_change":"2024-03-29T11:34:19.947080+0000","last_active":"2024-03-29T11:34:20.941051+0000","last_peered":"2024-03-29T11:34:20.941051+0000","last_clean":"2024-03-29T11:34:20.941051+0000","last_became_active":"2024-03-29T11:34:19.946641+0000","last_became_peered":"2024-03-29T11:34:19.946641+0000","last_unstale":"2024-03-29T11:34:20.941051+0000","last_undegraded":"2024-03-29T11:34:20.941051+0000","last_fullsized":"2024-03-29T11:34:20.941051+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T21:19:49.890182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934989+0000","last_change":"2024-03-29T11:34:19.422163+0000","last_active":"2024-03-29T11:34:20.934989+0000","last_peered":"2024-03-29T11:34:20.934989+0000","last_clean":"2024-03-29T11:34:20.934989+0000","last_became_active":"2024-03-29T11:34:19.421894+0000","last_became_peered":"2024-03-29T11:34:19.421894+0000","last_unstale":"2024-03-29T11:34:20.934989+0000","last_undegraded":"2024-03-29T11:34:20.934989+0000","last_fullsized":"2024-03-29T11:34:20.934989+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:22:00.471992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.959950+0000","last_change":"2024-03-29T11:34:19.958676+0000","last_active":"2024-03-29T11:34:19.959950+0000","last_peered":"2024-03-29T11:34:19.959950+0000","last_clean":"2024-03-29T11:34:19.959950+0000","last_became_active":"2024-03-29T11:34:19.957819+0000","last_became_peered":"2024-03-29T11:34:19.957819+0000","last_unstale":"2024-03-29T11:34:19.959950+0000","last_undegraded":"2024-03-29T11:34:19.959950+0000","last_fullsized":"2024-03-29T11:34:19.959950+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:11:24.486578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.935123+0000","last_change":"2024-03-29T11:34:19.429752+0000","last_active":"2024-03-29T11:34:19.935123+0000","last_peered":"2024-03-29T11:34:19.935123+0000","last_clean":"2024-03-29T11:34:19.935123+0000","last_became_active":"2024-03-29T11:34:19.429508+0000","last_became_peered":"2024-03-29T11:34:19.429508+0000","last_unstale":"2024-03-29T11:34:19.935123+0000","last_undegraded":"2024-03-29T11:34:19.935123+0000","last_fullsized":"2024-03-29T11:34:19.935123+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:11:25.549951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.941150+0000","last_change":"2024-03-29T11:34:19.945264+0000","last_active":"2024-03-29T11:34:20.941150+0000","last_peered":"2024-03-29T11:34:20.941150+0000","last_clean":"2024-03-29T11:34:20.941150+0000","last_became_active":"2024-03-29T11:34:19.944967+0000","last_became_peered":"2024-03-29T11:34:19.944967+0000","last_unstale":"2024-03-29T11:34:20.941150+0000","last_undegraded":"2024-03-29T11:34:20.941150+0000","last_fullsized":"2024-03-29T11:34:20.941150+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:52:42.879051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.936864+0000","last_change":"2024-03-29T11:34:19.434147+0000","last_active":"2024-03-29T11:34:19.936864+0000","last_peered":"2024-03-29T11:34:19.936864+0000","last_clean":"2024-03-29T11:34:19.936864+0000","last_became_active":"2024-03-29T11:34:19.433817+0000","last_became_peered":"2024-03-29T11:34:19.433817+0000","last_unstale":"2024-03-29T11:34:19.936864+0000","last_undegraded":"2024-03-29T11:34:19.936864+0000","last_fullsized":"2024-03-29T11:34:19.936864+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:51:50.580234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.954620+0000","last_change":"2024-03-29T11:34:19.948206+0000","last_active":"2024-03-29T11:34:19.954620+0000","last_peered":"2024-03-29T11:34:19.954620+0000","last_clean":"2024-03-29T11:34:19.954620+0000","last_became_active":"2024-03-29T11:34:19.947867+0000","last_became_peered":"2024-03-29T11:34:19.947867+0000","last_unstale":"2024-03-29T11:34:19.954620+0000","last_undegraded":"2024-03-29T11:34:19.954620+0000","last_fullsized":"2024-03-29T11:34:19.954620+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T17:03:26.025761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.937323+0000","last_change":"2024-03-29T11:34:19.433271+0000","last_active":"2024-03-29T11:34:19.937323+0000","last_peered":"2024-03-29T11:34:19.937323+0000","last_clean":"2024-03-29T11:34:19.937323+0000","last_became_active":"2024-03-29T11:34:19.432922+0000","last_became_peered":"2024-03-29T11:34:19.432922+0000","last_unstale":"2024-03-29T11:34:19.937323+0000","last_undegraded":"2024-03-29T11:34:19.937323+0000","last_fullsized":"2024-03-29T11:34:19.937323+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:28.842015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966335+0000","last_change":"2024-03-29T11:34:19.963007+0000","last_active":"2024-03-29T11:34:19.966335+0000","last_peered":"2024-03-29T11:34:19.966335+0000","last_clean":"2024-03-29T11:34:19.966335+0000","last_became_active":"2024-03-29T11:34:19.960610+0000","last_became_peered":"2024-03-29T11:34:19.960610+0000","last_unstale":"2024-03-29T11:34:19.966335+0000","last_undegraded":"2024-03-29T11:34:19.966335+0000","last_fullsized":"2024-03-29T11:34:19.966335+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T20:32:52.022464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.935997+0000","last_change":"2024-03-29T11:34:19.431583+0000","last_active":"2024-03-29T11:34:19.935997+0000","last_peered":"2024-03-29T11:34:19.935997+0000","last_clean":"2024-03-29T11:34:19.935997+0000","last_became_active":"2024-03-29T11:34:19.431102+0000","last_became_peered":"2024-03-29T11:34:19.431102+0000","last_unstale":"2024-03-29T11:34:19.935997+0000","last_undegraded":"2024-03-29T11:34:19.935997+0000","last_fullsized":"2024-03-29T11:34:19.935997+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T22:17:43.792763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.934500+0000","last_change":"2024-03-29T11:34:19.957073+0000","last_active":"2024-03-29T11:34:20.934500+0000","last_peered":"2024-03-29T11:34:20.934500+0000","last_clean":"2024-03-29T11:34:20.934500+0000","last_became_active":"2024-03-29T11:34:19.956777+0000","last_became_peered":"2024-03-29T11:34:19.956777+0000","last_unstale":"2024-03-29T11:34:20.934500+0000","last_undegraded":"2024-03-29T11:34:20.934500+0000","last_fullsized":"2024-03-29T11:34:20.934500+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T19:36:08.286066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.934719+0000","last_change":"2024-03-29T11:34:19.417904+0000","last_active":"2024-03-29T11:34:19.934719+0000","last_peered":"2024-03-29T11:34:19.934719+0000","last_clean":"2024-03-29T11:34:19.934719+0000","last_became_active":"2024-03-29T11:34:19.417551+0000","last_became_peered":"2024-03-29T11:34:19.417551+0000","last_unstale":"2024-03-29T11:34:19.934719+0000","last_undegraded":"2024-03-29T11:34:19.934719+0000","last_fullsized":"2024-03-29T11:34:19.934719+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:27:24.018400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-29T11:34:20.940183+0000","last_change":"2024-03-29T11:34:19.940077+0000","last_active":"2024-03-29T11:34:20.940183+0000","last_peered":"2024-03-29T11:34:20.940183+0000","last_clean":"2024-03-29T11:34:20.940183+0000","last_became_active":"2024-03-29T11:34:19.939633+0000","last_became_peered":"2024-03-29T11:34:19.939633+0000","last_unstale":"2024-03-29T11:34:20.940183+0000","last_undegraded":"2024-03-29T11:34:20.940183+0000","last_fullsized":"2024-03-29T11:34:20.940183+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:02:54.046445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":10,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.941263+0000","last_change":"2024-03-29T11:34:19.432321+0000","last_active":"2024-03-29T11:34:19.941263+0000","last_peered":"2024-03-29T11:34:19.941263+0000","last_clean":"2024-03-29T11:34:19.941263+0000","last_became_active":"2024-03-29T11:34:19.431739+0000","last_became_peered":"2024-03-29T11:34:19.431739+0000","last_unstale":"2024-03-29T11:34:19.941263+0000","last_undegraded":"2024-03-29T11:34:19.941263+0000","last_fullsized":"2024-03-29T11:34:19.941263+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:18.385727+0000","last_clean_scrub_stamp":"2024-03-29T11:34:18.385727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:35:54.386068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":9,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.966497+0000","last_change":"2024-03-29T11:34:19.964406+0000","last_active":"2024-03-29T11:34:19.966497+0000","last_peered":"2024-03-29T11:34:19.966497+0000","last_clean":"2024-03-29T11:34:19.966497+0000","last_became_active":"2024-03-29T11:34:19.963593+0000","last_became_peered":"2024-03-29T11:34:19.963593+0000","last_unstale":"2024-03-29T11:34:19.966497+0000","last_undegraded":"2024-03-29T11:34:19.966497+0000","last_fullsized":"2024-03-29T11:34:19.966497+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T11:34:19.397614+0000","last_clean_scrub_stamp":"2024-03-29T11:34:19.397614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T16:39:11.647026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"272'432","reported_seq":1552,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-29T11:34:19.423309+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:34:19.423309+0000","last_peered":"2024-03-29T11:34:19.423309+0000","last_clean":"2024-03-29T11:34:19.423309+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:34:19.423309+0000","last_undegraded":"2024-03-29T11:34:19.423309+0000","last_fullsized":"2024-03-29T11:34:19.423309+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2864,"num_read_kb":6044,"num_write":731,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2864,"num_read_kb":6044,"num_write":731,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576023,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57324,"kb_used_data":4060,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314516,"statfs":{"total":96636764160,"available":96578064384,"internally_reserved":0,"allocated":4157440,"data_stored":2346595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.67600000000000005,"15min":0.70799999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86799999999999999,"5min":1.0349999999999999,"15min":1.1459999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.68100000000000005,"15min":0.73099999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86899999999999999,"5min":0.98899999999999999,"15min":1.119},"last":0.81599999999999995}]},{"osd":1,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.67100000000000004,"15min":0.71599999999999997},"min":{"1min":0.435,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84399999999999997,"5min":0.97399999999999998,"15min":1.478},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.70399999999999996,"15min":0.752},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.82699999999999996,"5min":1.0640000000000001,"15min":1.393},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70499999999999996,"15min":0.748},"min":{"1min":0.46600000000000003,"5min":0.436,"15min":0.436},"max":{"1min":2.5649999999999999,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":0.751},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.71399999999999997,"15min":0.76100000000000001},"min":{"1min":0.501,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88400000000000001,"5min":1.032,"15min":1.0720000000000001},"last":0.77600000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.749},"min":{"1min":0.48499999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84299999999999997,"5min":0.97499999999999998,"15min":1.0580000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.77700000000000002,"15min":0.78600000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.6930000000000001,"5min":2.6930000000000001,"15min":2.6930000000000001},"last":0.79100000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88500000000000001,"5min":0.91100000000000003,"15min":1.3779999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66400000000000003,"15min":0.66900000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90800000000000003,"5min":0.93500000000000005,"15min":1.3169999999999999},"last":0.58999999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69999999999999996,"15min":0.68400000000000005},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.1160000000000001},"last":0.62},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93700000000000006,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.65600000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.76800000000000002,"15min":0.76600000000000001},"min":{"1min":0.48099999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92600000000000005,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.75600000000000001,"15min":0.74199999999999999},"min":{"1min":0.52000000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.91000000000000003,"5min":1.071,"15min":1.071},"last":0.68799999999999994}]}]},{"osd":6,"up_from":13,"seq":55834576024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48568,"kb_used_data":3492,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323272,"statfs":{"total":96636764160,"available":96587030528,"internally_reserved":0,"allocated":3575808,"data_stored":1757791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.67100000000000004,"15min":0.71999999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.94499999999999995,"15min":1.2150000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.68400000000000005,"15min":0.74199999999999999},"min":{"1min":0.42799999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95299999999999996,"5min":0.96399999999999997,"15min":1.4239999999999999},"last":0.83899999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67000000000000004,"15min":0.72199999999999998},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91100000000000003,"5min":1.022,"15min":1.98},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.70699999999999996,"15min":0.75900000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.435},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.8200000000000001},"last":0.93999999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69699999999999995,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86899999999999999,"5min":0.96199999999999997,"15min":1.669},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71099999999999997,"15min":0.751},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.167,"5min":1.167,"15min":1.7589999999999999},"last":0.70799999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70599999999999996,"15min":0.753},"min":{"1min":0.51100000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90200000000000002,"5min":0.997,"15min":2.52},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.75700000000000001,"15min":0.79200000000000004},"min":{"1min":0.51200000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.877},"last":1.1060000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.61599999999999999},"min":{"1min":0.435,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93999999999999995,"15min":2.4100000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.71599999999999997,"15min":0.71199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.45900000000000002},"max":{"1min":1.0740000000000001,"5min":1.2709999999999999,"15min":2.6629999999999998},"last":0.91300000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66900000000000004,"15min":0.66200000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99099999999999999,"5min":1.0900000000000001,"15min":1.5960000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.925},"last":0.80600000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77100000000000002,"15min":0.76900000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96699999999999997,"5min":1.0509999999999999,"15min":2.7050000000000001},"last":1.056},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74299999999999999,"15min":0.74399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0980000000000001,"5min":1.1220000000000001,"15min":1.726},"last":0.88100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576022,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64944,"kb_used_data":3488,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306896,"statfs":{"total":96636764160,"available":96570261504,"internally_reserved":0,"allocated":3571712,"data_stored":1756283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.44600000000000001,"15min":0.46100000000000002},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.62,"5min":0.79100000000000004,"15min":0.80300000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.34399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.626,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52700000000000002,"15min":0.55400000000000005},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72599999999999998,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.50700000000000001,"15min":0.52600000000000002},"min":{"1min":0.33800000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66500000000000004,"5min":0.75900000000000001,"15min":0.90300000000000002},"last":0.36099999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56799999999999995,"15min":0.58699999999999997},"min":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.36299999999999999},"max":{"1min":0.77600000000000002,"5min":0.77700000000000002,"15min":0.995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53900000000000003,"15min":0.56999999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75800000000000001,"5min":0.77700000000000002,"15min":0.83999999999999997},"last":0.45900000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0589999999999999,"5min":1.153,"15min":1.238},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77300000000000002,"15min":0.79200000000000004},"min":{"1min":0.60099999999999998,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":1.0109999999999999,"5min":1.222,"15min":1.304},"last":0.74399999999999999}]},{"osd":5,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.80100000000000005,"15min":0.81100000000000005},"min":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":1.1359999999999999,"5min":1.1819999999999999,"15min":1.3720000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78900000000000003,"15min":0.82199999999999995},"min":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.3520000000000001},"last":0.66000000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78700000000000003,"15min":0.81000000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0369999999999999,"5min":1.1559999999999999,"15min":1.1919999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.58999999999999997,"5min":0.501,"15min":0.501},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.266},"last":0.85299999999999998}]},{"osd":7,"last update":"Fri Mar 29 11:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.85899999999999999},"min":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.54500000000000004},"max":{"1min":1.0800000000000001,"5min":1.129,"15min":1.327},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82099999999999995,"15min":0.85799999999999998},"min":{"1min":0.622,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.159,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.68300000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576024,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65520,"kb_used_data":4064,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306320,"statfs":{"total":96636764160,"available":96569671680,"internally_reserved":0,"allocated":4161536,"data_stored":2345668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.442,"15min":0.45900000000000002},"min":{"1min":0.28699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":1.087},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.504,"15min":0.53200000000000003},"min":{"1min":0.33600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.85799999999999998,"15min":1.3089999999999999},"last":0.59799999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54700000000000004,"15min":0.57399999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63300000000000001,"5min":0.84999999999999998,"15min":1.5289999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48299999999999998,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66900000000000004,"5min":0.70899999999999996,"15min":1.6140000000000001},"last":0.46800000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.53200000000000003,"15min":0.55600000000000005},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65300000000000002,"5min":0.84699999999999998,"15min":1.222},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.59899999999999998,"15min":0.63200000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.72999999999999998,"5min":0.95899999999999996,"15min":1.3899999999999999},"last":0.64900000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.78400000000000003,"15min":0.80700000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.495},"max":{"1min":0.93400000000000005,"5min":1.248,"15min":1.248},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.80400000000000005,"15min":0.83899999999999997},"min":{"1min":0.58299999999999996,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.012,"5min":1.1970000000000001,"15min":1.5820000000000001},"last":0.95299999999999996}]},{"osd":5,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.74299999999999999,"15min":0.77900000000000003},"min":{"1min":0.53200000000000003,"5min":0.505,"15min":0.505},"max":{"1min":0.89200000000000002,"5min":1.1779999999999999,"15min":1.3440000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78600000000000003,"15min":0.81299999999999994},"min":{"1min":0.52600000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.018,"5min":1.1499999999999999,"15min":1.3640000000000001},"last":0.78500000000000003}]},{"osd":6,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.80400000000000005,"15min":0.84499999999999997},"min":{"1min":0.499,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0649999999999999,"5min":1.236,"15min":1.6240000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.83499999999999996,"15min":0.876},"min":{"1min":0.5,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.996,"5min":1.1739999999999999,"15min":1.337},"last":0.88700000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.78500000000000003,"15min":0.81899999999999995},"min":{"1min":0.52300000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.05,"5min":1.111,"15min":1.214},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.81599999999999995,"15min":0.86399999999999999},"min":{"1min":0.56200000000000006,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0329999999999999,"5min":1.2090000000000001,"15min":1.365},"last":0.86699999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576023,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52660,"kb_used_data":3488,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319180,"statfs":{"total":96636764160,"available":96582840320,"internally_reserved":0,"allocated":3571712,"data_stored":1756351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.47299999999999998,"15min":0.47199999999999998},"min":{"1min":0.32300000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.74099999999999999,"15min":0.75900000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.5,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60199999999999998,"5min":0.73999999999999999,"15min":0.96799999999999997},"last":0.377}]},{"osd":1,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47899999999999998,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.54900000000000004,"15min":0.56999999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73599999999999999,"5min":0.872,"15min":0.88800000000000001},"last":0.505}]},{"osd":3,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55800000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64900000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55700000000000005,"15min":0.57199999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.71099999999999997,"5min":1.038,"15min":1.038},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.754,"15min":0.79800000000000004},"min":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1970000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.747,"15min":0.79900000000000004},"min":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0029999999999999,"5min":1.157,"15min":1.244},"last":0.77600000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77000000000000002,"15min":0.80900000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.53400000000000003},"max":{"1min":0.96399999999999997,"5min":1.135,"15min":1.256},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.747,"5min":0.76500000000000001,"15min":0.81399999999999995},"min":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.3140000000000001},"last":0.76700000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.78000000000000003,"15min":0.83099999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93200000000000005,"5min":1.105,"15min":1.24},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.78000000000000003,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.5,"15min":0.5},"max":{"1min":0.98199999999999998,"5min":1.099,"15min":1.329},"last":0.753}]},{"osd":7,"last update":"Fri Mar 29 11:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82499999999999996,"15min":0.88},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.044,"5min":1.0660000000000001,"15min":1.2889999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.83899999999999997,"15min":0.89400000000000002},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.56899999999999995},"max":{"1min":1.0269999999999999,"5min":1.2969999999999999,"15min":1.304},"last":0.84399999999999997}]}]},{"osd":3,"up_from":13,"seq":55834576023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48556,"kb_used_data":3484,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323284,"statfs":{"total":96636764160,"available":96587042816,"internally_reserved":0,"allocated":3567616,"data_stored":1756809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.28199999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.69399999999999995,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52800000000000002,"15min":0.54500000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66100000000000003,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.436}]},{"osd":1,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47999999999999998,"15min":0.49099999999999999},"min":{"1min":0.30599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.626,"5min":1.821,"15min":1.821},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.54400000000000004,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.64500000000000002,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.58199999999999996,"15min":0.58699999999999997},"min":{"1min":0.39200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.745,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.64100000000000001,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85699999999999998,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.46200000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.55500000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.014,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.872},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.78800000000000003,"15min":0.81100000000000005},"min":{"1min":0.55600000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95599999999999996,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.02}]},{"osd":5,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79600000000000004,"15min":0.80900000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.032,"5min":2.157,"15min":2.157},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.82599999999999996,"15min":0.82299999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0640000000000001,"5min":2.266,"15min":2.266},"last":1.0389999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.85399999999999998,"15min":0.85599999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.083,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.82299999999999995,"15min":0.83799999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.03,"5min":2.198,"15min":2.198},"last":0.93999999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.85399999999999998,"15min":0.86199999999999999},"min":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":0.98499999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.88,"15min":0.88600000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.149,"5min":2.444,"15min":2.444},"last":1.0549999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96884,"kb_used_data":3488,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274956,"statfs":{"total":96636764160,"available":96537554944,"internally_reserved":0,"allocated":3571712,"data_stored":1756879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64400000000000002,"15min":0.70699999999999996},"min":{"1min":0.39700000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":1.0509999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.64000000000000001,"15min":0.70799999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.38600000000000001},"max":{"1min":0.81000000000000005,"5min":0.93600000000000005,"15min":1.1459999999999999},"last":0.65500000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.65500000000000003,"15min":0.72499999999999998},"min":{"1min":0.48699999999999999,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.1910000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67000000000000004,"15min":0.72999999999999998},"min":{"1min":0.435,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":1.002},"last":0.70199999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76300000000000001,"15min":0.81399999999999995},"min":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.73899999999999999,"15min":0.80000000000000004},"min":{"1min":0.51900000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.875,"5min":1.224,"15min":1.224},"last":0.753}]},{"osd":3,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68500000000000005,"15min":0.748},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88100000000000001,"5min":0.96699999999999997,"15min":1.181},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72899999999999998,"15min":0.76300000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.84899999999999998}]},{"osd":5,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64900000000000002,"15min":0.628},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.01,"5min":1.0920000000000001,"15min":1.169},"last":0.50900000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69599999999999995,"15min":0.67300000000000004},"min":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72599999999999998,"15min":0.72099999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.53200000000000003}]},{"osd":7,"last update":"Fri Mar 29 11:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.73899999999999999,"15min":0.73399999999999999},"min":{"1min":0.54100000000000004,"5min":0.501,"15min":0.436},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.2669999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.81899999999999995,"15min":0.82299999999999995},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.54400000000000004},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.878}]}]},{"osd":5,"up_from":13,"seq":55834576023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48572,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323268,"statfs":{"total":96636764160,"available":96587026432,"internally_reserved":0,"allocated":3579904,"data_stored":1757881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.64800000000000002,"15min":0.68999999999999995},"min":{"1min":0.38700000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71299999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.66100000000000003,"15min":0.71599999999999997},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.752,"5min":0.95599999999999996,"15min":1.0680000000000001},"last":0.51600000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.72199999999999998,"15min":0.751},"min":{"1min":0.434,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":1.839,"15min":1.839},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.69999999999999996,"15min":0.72999999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.88600000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.56899999999999995}]},{"osd":2,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.73099999999999998,"15min":0.747},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93899999999999995,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.73999999999999999},"min":{"1min":0.54700000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.81299999999999994,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.753,"15min":0.76400000000000001},"min":{"1min":0.49199999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.84699999999999998,"5min":3.444,"15min":3.444},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.748,"15min":0.76900000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.98299999999999998,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63400000000000001,"15min":0.61199999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86099999999999999,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.68200000000000005,"15min":0.65500000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88600000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.76000000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.71299999999999997,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90100000000000002,"5min":1.214,"15min":1.214},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.72399999999999998,"15min":0.70299999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.42499999999999999},"max":{"1min":0.91000000000000003,"5min":1.0900000000000001,"15min":1.107},"last":0.74399999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.748,"15min":0.72799999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92400000000000004,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.79200000000000004,"15min":0.77500000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93999999999999995,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.78300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:34:25.351 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-29T11:34:25.351 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-29T11:34:25.378 INFO:tasks.ceph.mds.a.smithi012.stderr:2024-03-29T11:34:25.380+0000 7f407ba00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-29T11:34:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.587+0000 7fc392600700 1 Processor -- start 2024-03-29T11:34:25.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.587+0000 7fc392600700 1 -- start start 2024-03-29T11:34:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.588+0000 7fc392600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc38c15dbd0 0x7fc38c0719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.588+0000 7fc392600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc38c0cc020 0x7fc38c0cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.588+0000 7fc392600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0cc800 con 0x7fc38c15dfb0 2024-03-29T11:34:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.588+0000 7fc392600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc38c0ccae0 con 0x7fc38c0cc020 2024-03-29T11:34:25.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.588+0000 7fc392600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0ccdc0 con 0x7fc38c15dbd0 2024-03-29T11:34:25.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.588+0000 7fc38be00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc38c15dfb0 0x7fc38c15e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60244/0 (socket says 172.21.15.12:60244) 2024-03-29T11:34:25.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.588+0000 7fc38be00700 1 -- 172.21.15.12:0/80906539 learned_addr learned my addr 172.21.15.12:0/80906539 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:25.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38b400700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc38c15dbd0 0x7fc38c0719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38aa00700 1 --2- 172.21.15.12:0/80906539 >> v2:172.21.15.159:3300/0 conn(0x7fc38c0cc020 0x7fc38c0cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386339743 0 0) 0x7fc38c0cc800 con 0x7fc38c15dfb0 2024-03-29T11:34:25.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3700035b0 con 0x7fc38c15dfb0 2024-03-29T11:34:25.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861670081 0 0) 0x7fc3700035b0 con 0x7fc38c15dfb0 2024-03-29T11:34:25.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc38c15dbd0 msgr2=0x7fc38c0719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:25.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc38c15dbd0 0x7fc38c0719a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 >> v2:172.21.15.159:3300/0 conn(0x7fc38c0cc020 msgr2=0x7fc38c0cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 --2- 172.21.15.12:0/80906539 >> v2:172.21.15.159:3300/0 conn(0x7fc38c0cc020 0x7fc38c0cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c0cd0a0 con 0x7fc38c15dfb0 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fc3840019f0 con 0x7fc38c15dfb0 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 >> v1:172.21.15.12:6789/0 conn(0x7fc38c15dfb0 legacy=0x7fc38c15e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc370003a00 0x7fc370003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc38c15dbd0 0x7fc370004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc370004d90 0x7fc370005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0ccae0 con 0x7fc370003a00 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0ccdc0 con 0x7fc370004d90 2024-03-29T11:34:25.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.589+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37000ba10 con 0x7fc38c15dbd0 2024-03-29T11:34:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.590+0000 7fc38b400700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc370003a00 0x7fc370003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.590+0000 7fc38be00700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc370004d90 0x7fc370005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.590+0000 7fc38aa00700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc38c15dbd0 0x7fc370004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.590+0000 7fc38b400700 1 -- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc38c15dbd0 msgr2=0x7fc370004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.590+0000 7fc38b400700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc38c15dbd0 0x7fc370004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.590+0000 7fc38b400700 1 -- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc370004d90 msgr2=0x7fc370005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.590+0000 7fc38b400700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc370004d90 0x7fc370005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.590+0000 7fc38b400700 1 -- 172.21.15.12:0/80906539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3700035b0 con 0x7fc370003a00 2024-03-29T11:34:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.590+0000 7fc38b400700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc370003a00 0x7fc370003e20 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009b60 tx=0x7fc370005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b1d9db3de81915 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.591+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c00e6c0 con 0x7fc370003a00 2024-03-29T11:34:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.591+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c00e820 con 0x7fc370003a00 2024-03-29T11:34:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.591+0000 7fc38a000700 1 -- 172.21.15.12:0/80906539 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c00eb90 con 0x7fc370003a00 2024-03-29T11:34:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.591+0000 7fc392600700 1 -- 172.21.15.12:0/80906539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc370003a00 msgr2=0x7fc370003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.591+0000 7fc392600700 1 --2- 172.21.15.12:0/80906539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc370003a00 0x7fc370003e20 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009b60 tx=0x7fc370005880 comp rx=0 tx=0).stop 2024-03-29T11:34:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.591+0000 7fc38be00700 1 -- 172.21.15.12:0/80906539 reap_dead start 2024-03-29T11:34:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.591+0000 7fc392600700 1 -- 172.21.15.12:0/80906539 shutdown_connections 2024-03-29T11:34:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.591+0000 7fc392600700 1 -- 172.21.15.12:0/80906539 >> 172.21.15.12:0/80906539 conn(0x7fc38c0664c0 msgr2=0x7fc38c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.591+0000 7fc392600700 1 -- 172.21.15.12:0/80906539 shutdown_connections 2024-03-29T11:34:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.591+0000 7fc392600700 1 -- 172.21.15.12:0/80906539 wait complete. 2024-03-29T11:34:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.593+0000 7fc392600700 1 Processor -- start 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.593+0000 7fc392600700 1 -- start start 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.594+0000 7fc392600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc370003a00 0x7fc38c1490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.594+0000 7fc392600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc38c0cad20 0x7fc38c0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.594+0000 7fc392600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c1e4d50 0x7fc38c1e51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.594+0000 7fc392600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0cb6e0 con 0x7fc38c1e4d50 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.594+0000 7fc392600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c1e6690 con 0x7fc38c0cad20 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.594+0000 7fc392600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c1495e0 con 0x7fc370003a00 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.594+0000 7fc38be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c1e4d50 0x7fc38c1e51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.594+0000 7fc38be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c1e4d50 0x7fc38c1e51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51548/0 (socket says 172.21.15.12:51548) 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.594+0000 7fc38be00700 1 -- 172.21.15.12:0/4203848233 learned_addr learned my addr 172.21.15.12:0/4203848233 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc38b400700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc370003a00 0x7fc38c1490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc38aa00700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc38c0cad20 0x7fc38c0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc38be00700 1 -- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc370003a00 msgr2=0x7fc38c1490a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc38be00700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc370003a00 0x7fc38c1490a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc38be00700 1 -- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc38c0cad20 msgr2=0x7fc38c0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc38be00700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc38c0cad20 0x7fc38c0cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc38be00700 1 -- 172.21.15.12:0/4203848233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c1e5a80 con 0x7fc38c1e4d50 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc38aa00700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc38c0cad20 0x7fc38c0cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc38be00700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c1e4d50 0x7fc38c1e51d0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7fc38c15e390 tx=0x7fc38400a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc383400700 1 -- 172.21.15.12:0/4203848233 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38400ade0 con 0x7fc38c1e4d50 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc383400700 1 -- 172.21.15.12:0/4203848233 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3840133f0 con 0x7fc38c1e4d50 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc392600700 1 -- 172.21.15.12:0/4203848233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c1e5d60 con 0x7fc38c1e4d50 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc383400700 1 -- 172.21.15.12:0/4203848233 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3840137a0 con 0x7fc38c1e4d50 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.595+0000 7fc392600700 1 -- 172.21.15.12:0/4203848233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c1e6320 con 0x7fc38c1e4d50 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.596+0000 7fc392600700 1 -- 172.21.15.12:0/4203848233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c1e64e0 con 0x7fc38c1e4d50 2024-03-29T11:34:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.597+0000 7fc383400700 1 -- 172.21.15.12:0/4203848233 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc384013900 con 0x7fc38c1e4d50 2024-03-29T11:34:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.598+0000 7fc383400700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc36c06adb0 0x7fc36c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.598+0000 7fc383400700 1 -- 172.21.15.12:0/4203848233 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fc384089e70 con 0x7fc38c1e4d50 2024-03-29T11:34:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.599+0000 7fc38b400700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc36c06adb0 0x7fc36c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.599+0000 7fc38b400700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc36c06adb0 0x7fc36c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0090b0 tx=0x7fc37c01f040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:25.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.602+0000 7fc383400700 1 -- 172.21.15.12:0/4203848233 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc384054e50 con 0x7fc38c1e4d50 2024-03-29T11:34:25.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.905+0000 7fc392600700 1 -- 172.21.15.12:0/4203848233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc38c1e6820 con 0x7fc38c1e4d50 2024-03-29T11:34:25.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.909+0000 7fc383400700 1 -- 172.21.15.12:0/4203848233 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v55) v1 ==== 190+0+0 (secure 0 0 0) 0x7fc384058470 con 0x7fc38c1e4d50 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 -- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc36c06adb0 msgr2=0x7fc36c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc36c06adb0 0x7fc36c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0090b0 tx=0x7fc37c01f040 comp rx=0 tx=0).stop 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 -- 172.21.15.12:0/4203848233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c1e4d50 msgr2=0x7fc38c1e51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c1e4d50 0x7fc38c1e51d0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7fc38c15e390 tx=0x7fc38400a550 comp rx=0 tx=0).stop 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 -- 172.21.15.12:0/4203848233 shutdown_connections 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc36c06adb0 0x7fc36c06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c1e4d50 0x7fc38c1e51d0 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc370003a00 0x7fc38c1490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 --2- 172.21.15.12:0/4203848233 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc38c0cad20 0x7fc38c0cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 -- 172.21.15.12:0/4203848233 >> 172.21.15.12:0/4203848233 conn(0x7fc38c0664c0 msgr2=0x7fc38c15a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 -- 172.21.15.12:0/4203848233 shutdown_connections 2024-03-29T11:34:25.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:25.912+0000 7fc392600700 1 -- 172.21.15.12:0/4203848233 wait complete. 2024-03-29T11:34:25.911 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-03-29T11:34:25.928 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:34:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.129+0000 7f90d2400700 1 Processor -- start 2024-03-29T11:34:26.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.129+0000 7f90d2400700 1 -- start start 2024-03-29T11:34:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.130+0000 7f90d2400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f90cc15d670 0x7f90cc0ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:26.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.130+0000 7f90d2400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90cc06dee0 0x7f90cc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.130+0000 7f90d2400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0b3890 con 0x7f90cc15da50 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.130+0000 7f90d2400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90cc0acac0 con 0x7f90cc06dee0 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.130+0000 7f90d2400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90cc159f50 con 0x7f90cc15d670 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.131+0000 7f90cbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90cc06dee0 0x7f90cc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.131+0000 7f90cbe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90cc06dee0 0x7f90cc06e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51562/0 (socket says 172.21.15.12:51562) 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.131+0000 7f90cbe00700 1 -- 172.21.15.12:0/3865487693 learned_addr learned my addr 172.21.15.12:0/3865487693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.131+0000 7f90cb400700 1 --2- 172.21.15.12:0/3865487693 >> v2:172.21.15.159:3300/0 conn(0x7f90cc15d670 0x7f90cc0ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.131+0000 7f90cbe00700 1 -- 172.21.15.12:0/3865487693 >> v2:172.21.15.159:3300/0 conn(0x7f90cc15d670 msgr2=0x7f90cc0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.131+0000 7f90cbe00700 1 --2- 172.21.15.12:0/3865487693 >> v2:172.21.15.159:3300/0 conn(0x7f90cc15d670 0x7f90cc0ac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.131+0000 7f90cbe00700 1 -- 172.21.15.12:0/3865487693 >> v1:172.21.15.159:6789/0 conn(0x7f90cc15da50 legacy=0x7f90cc0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.131+0000 7f90cbe00700 1 -- 172.21.15.12:0/3865487693 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc0b35b0 con 0x7f90cc06dee0 2024-03-29T11:34:26.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.131+0000 7f90cbe00700 1 --2- 172.21.15.12:0/3865487693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc06dee0 0x7f90cc06e350 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f90cc070090 tx=0x7f90c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc6046c03812f1c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90ca000700 1 -- 172.21.15.12:0/3865487693 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c400e070 con 0x7f90cc06dee0 2024-03-29T11:34:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90ca000700 1 -- 172.21.15.12:0/3865487693 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c400b910 con 0x7f90cc06dee0 2024-03-29T11:34:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90ca000700 1 -- 172.21.15.12:0/3865487693 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c400bcc0 con 0x7f90cc06dee0 2024-03-29T11:34:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90d2400700 1 -- 172.21.15.12:0/3865487693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc06dee0 msgr2=0x7f90cc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90d2400700 1 --2- 172.21.15.12:0/3865487693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc06dee0 0x7f90cc06e350 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f90cc070090 tx=0x7f90c400b330 comp rx=0 tx=0).stop 2024-03-29T11:34:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90d2400700 1 -- 172.21.15.12:0/3865487693 shutdown_connections 2024-03-29T11:34:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90d2400700 1 --2- 172.21.15.12:0/3865487693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc06dee0 0x7f90cc06e350 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90d2400700 1 --2- 172.21.15.12:0/3865487693 >> v2:172.21.15.159:3300/0 conn(0x7f90cc15d670 0x7f90cc0ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90d2400700 1 -- 172.21.15.12:0/3865487693 >> 172.21.15.12:0/3865487693 conn(0x7f90cc0cf190 msgr2=0x7f90cc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90d2400700 1 -- 172.21.15.12:0/3865487693 shutdown_connections 2024-03-29T11:34:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.132+0000 7f90d2400700 1 -- 172.21.15.12:0/3865487693 wait complete. 2024-03-29T11:34:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.134+0000 7f90d2400700 1 Processor -- start 2024-03-29T11:34:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.134+0000 7f90d2400700 1 -- start start 2024-03-29T11:34:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90d2400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f90cc06dee0 0x7f90cc0c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90d2400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f90cc15d670 0x7f90cc0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90d2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc15da50 0x7f90cc0c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90d2400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc159b80 con 0x7f90cc15da50 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90d2400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc152650 con 0x7f90cc15d670 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90d2400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc152930 con 0x7f90cc06dee0 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc15da50 0x7f90cc0c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc15da50 0x7f90cc0c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51564/0 (socket says 172.21.15.12:51564) 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90cbe00700 1 -- 172.21.15.12:0/3003346380 learned_addr learned my addr 172.21.15.12:0/3003346380 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90cb400700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f90cc06dee0 0x7f90cc0c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.135+0000 7f90caa00700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f90cc15d670 0x7f90cc0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90cbe00700 1 -- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f90cc06dee0 msgr2=0x7f90cc0c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90cbe00700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f90cc06dee0 0x7f90cc0c86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90cbe00700 1 -- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f90cc15d670 msgr2=0x7f90cc0c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90cbe00700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f90cc15d670 0x7f90cc0c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90cbe00700 1 -- 172.21.15.12:0/3003346380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c4009b30 con 0x7f90cc15da50 2024-03-29T11:34:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90cbe00700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc15da50 0x7f90cc0c98b0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f90cc06b930 tx=0x7f90c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90c3400700 1 -- 172.21.15.12:0/3003346380 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4004880 con 0x7f90cc15da50 2024-03-29T11:34:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90c3400700 1 -- 172.21.15.12:0/3003346380 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c40099b0 con 0x7f90cc15da50 2024-03-29T11:34:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90c3400700 1 -- 172.21.15.12:0/3003346380 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c401c680 con 0x7f90cc15da50 2024-03-29T11:34:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90d2400700 1 -- 172.21.15.12:0/3003346380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc152c10 con 0x7f90cc15da50 2024-03-29T11:34:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.136+0000 7f90d2400700 1 -- 172.21.15.12:0/3003346380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc1531d0 con 0x7f90cc15da50 2024-03-29T11:34:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.138+0000 7f90d2400700 1 -- 172.21.15.12:0/3003346380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90cc1434e0 con 0x7f90cc15da50 2024-03-29T11:34:26.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.138+0000 7f90c3400700 1 -- 172.21.15.12:0/3003346380 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f90c401c7e0 con 0x7f90cc15da50 2024-03-29T11:34:26.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.139+0000 7f90c3400700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f90ac06adb0 0x7f90ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:26.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.139+0000 7f90c3400700 1 -- 172.21.15.12:0/3003346380 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f90c408aeb0 con 0x7f90cc15da50 2024-03-29T11:34:26.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.140+0000 7f90cb400700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f90ac06adb0 0x7f90ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:26.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.140+0000 7f90cb400700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f90ac06adb0 0x7f90ac06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009be0 tx=0x7f90bc009210 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:26.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.144+0000 7f90c3400700 1 -- 172.21.15.12:0/3003346380 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90c4055e10 con 0x7f90cc15da50 2024-03-29T11:34:26.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.406+0000 7f90d2400700 1 -- 172.21.15.12:0/3003346380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90cc066ff0 con 0x7f90cc15da50 2024-03-29T11:34:26.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.407+0000 7f90c3400700 1 -- 172.21.15.12:0/3003346380 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4747 (secure 0 0 0) 0x7f90c4059430 con 0x7f90cc15da50 2024-03-29T11:34:26.406 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:34:26.406 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":146,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12538,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/350949303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":350949303},{"type":"v1","addr":"172.21.15.159:6836","nonce":350949303}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1576953466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1576953466},{"type":"v1","addr":"172.21.15.159:6837","nonce":1576953466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/30523791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":30523791},{"type":"v1","addr":"172.21.15.12:6835","nonce":30523791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:34:19.924674+0000","modified":"2024-03-29T11:34:23.413936+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13125":{"gid":13125,"name":"a","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/2115072104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2115072104},{"type":"v1","addr":"172.21.15.12:6834","nonce":2115072104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-29T11:34:26.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.409+0000 7f90d2400700 1 -- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f90ac06adb0 msgr2=0x7f90ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.409+0000 7f90d2400700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f90ac06adb0 0x7f90ac06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009be0 tx=0x7f90bc009210 comp rx=0 tx=0).stop 2024-03-29T11:34:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.409+0000 7f90d2400700 1 -- 172.21.15.12:0/3003346380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc15da50 msgr2=0x7f90cc0c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.409+0000 7f90d2400700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc15da50 0x7f90cc0c98b0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f90cc06b930 tx=0x7f90c4004ad0 comp rx=0 tx=0).stop 2024-03-29T11:34:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.410+0000 7f90d2400700 1 -- 172.21.15.12:0/3003346380 shutdown_connections 2024-03-29T11:34:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.410+0000 7f90d2400700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f90ac06adb0 0x7f90ac06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.410+0000 7f90d2400700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90cc15da50 0x7f90cc0c98b0 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.410+0000 7f90d2400700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f90cc06dee0 0x7f90cc0c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.410+0000 7f90d2400700 1 --2- 172.21.15.12:0/3003346380 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f90cc15d670 0x7f90cc0c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.410+0000 7f90d2400700 1 -- 172.21.15.12:0/3003346380 >> 172.21.15.12:0/3003346380 conn(0x7f90cc0cf190 msgr2=0x7f90cc071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.410+0000 7f90d2400700 1 -- 172.21.15.12:0/3003346380 shutdown_connections 2024-03-29T11:34:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:26.410+0000 7f90d2400700 1 -- 172.21.15.12:0/3003346380 wait complete. 2024-03-29T11:34:26.411 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 146 2024-03-29T11:34:26.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 146, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-29T11:34:19.924674+0000', 'modified': '2024-03-29T11:34:23.413936+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13125}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13125': {'gid': 13125, 'name': 'a', 'rank': 0, 'incarnation': 142, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.12:6834/2115072104', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6832', 'nonce': 2115072104}, {'type': 'v1', 'addr': '172.21.15.12:6834', 'nonce': 2115072104}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-29T11:34:26.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-29T11:34:26.829 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-29T11:34:27.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.034+0000 7fe4bc200700 1 Processor -- start 2024-03-29T11:34:27.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.035+0000 7fe4bc200700 1 -- start start 2024-03-29T11:34:27.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.036+0000 7fe4bc200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe4b415a5c0 0x7fe4b40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.036+0000 7fe4bc200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4b415dfe0 0x7fe4b40ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.036+0000 7fe4bc200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b40cb310 con 0x7fe4b40ce670 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.036+0000 7fe4bc200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b40ceae0 con 0x7fe4b415dfe0 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.036+0000 7fe4bc200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b4070f00 con 0x7fe4b415a5c0 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.036+0000 7fe4b8e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4b415dfe0 0x7fe4b40ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.036+0000 7fe4ba200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe4b40ce670 0x7fe4b40b2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60250/0 (socket says 172.21.15.12:60250) 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.036+0000 7fe4b8e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4b415dfe0 0x7fe4b40ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51576/0 (socket says 172.21.15.12:51576) 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.036+0000 7fe4ba200700 1 -- 172.21.15.12:0/2718101412 learned_addr learned my addr 172.21.15.12:0/2718101412 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.036+0000 7fe4b9800700 1 --2- 172.21.15.12:0/2718101412 >> v2:172.21.15.159:3300/0 conn(0x7fe4b415a5c0 0x7fe4b40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.037+0000 7fe4b8e00700 1 -- 172.21.15.12:0/2718101412 >> v2:172.21.15.159:3300/0 conn(0x7fe4b415a5c0 msgr2=0x7fe4b40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.037+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/2718101412 >> v2:172.21.15.159:3300/0 conn(0x7fe4b415a5c0 0x7fe4b40709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.037+0000 7fe4b8e00700 1 -- 172.21.15.12:0/2718101412 >> v1:172.21.15.12:6789/0 conn(0x7fe4b40ce670 legacy=0x7fe4b40b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.037+0000 7fe4b8e00700 1 -- 172.21.15.12:0/2718101412 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b40cca90 con 0x7fe4b415dfe0 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.037+0000 7fe4b3e00700 1 -- 172.21.15.12:0/2718101412 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760196747 0 0) 0x7fe4b40cb310 con 0x7fe4b40ce670 2024-03-29T11:34:27.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.037+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/2718101412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 0x7fe4b40ce130 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4072e70 tx=0x7fe4a0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3da7a6829ce55eb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:27.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.037+0000 7fe4b3e00700 1 -- 172.21.15.12:0/2718101412 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0011830 con 0x7fe4b415dfe0 2024-03-29T11:34:27.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.037+0000 7fe4b3e00700 1 -- 172.21.15.12:0/2718101412 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a0011990 con 0x7fe4b415dfe0 2024-03-29T11:34:27.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.037+0000 7fe4b3e00700 1 -- 172.21.15.12:0/2718101412 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0011d20 con 0x7fe4b415dfe0 2024-03-29T11:34:27.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.038+0000 7fe4bc200700 1 -- 172.21.15.12:0/2718101412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 msgr2=0x7fe4b40ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.038+0000 7fe4bc200700 1 --2- 172.21.15.12:0/2718101412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 0x7fe4b40ce130 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4072e70 tx=0x7fe4a0009b20 comp rx=0 tx=0).stop 2024-03-29T11:34:27.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.038+0000 7fe4bc200700 1 -- 172.21.15.12:0/2718101412 shutdown_connections 2024-03-29T11:34:27.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.038+0000 7fe4bc200700 1 --2- 172.21.15.12:0/2718101412 >> v2:172.21.15.159:3300/0 conn(0x7fe4b415a5c0 0x7fe4b40709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.038+0000 7fe4bc200700 1 --2- 172.21.15.12:0/2718101412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 0x7fe4b40ce130 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.038+0000 7fe4bc200700 1 -- 172.21.15.12:0/2718101412 >> 172.21.15.12:0/2718101412 conn(0x7fe4b40b0d70 msgr2=0x7fe4b40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:27.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.038+0000 7fe4bc200700 1 -- 172.21.15.12:0/2718101412 shutdown_connections 2024-03-29T11:34:27.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.038+0000 7fe4bc200700 1 -- 172.21.15.12:0/2718101412 wait complete. 2024-03-29T11:34:27.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.040+0000 7fe4bc200700 1 Processor -- start 2024-03-29T11:34:27.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.040+0000 7fe4bc200700 1 -- start start 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4bc200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4b40ce670 0x7fe4b41464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4bc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4b415a5c0 0x7fe4b41594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4bc200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 0x7fe4b4155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4bc200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b4158660 con 0x7fe4b415dfe0 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4bc200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b4157d30 con 0x7fe4b40ce670 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4bc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b4158010 con 0x7fe4b415a5c0 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4ba200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 0x7fe4b4155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4ba200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 0x7fe4b4155680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51584/0 (socket says 172.21.15.12:51584) 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4ba200700 1 -- 172.21.15.12:0/1946090042 learned_addr learned my addr 172.21.15.12:0/1946090042 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4b9800700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4b40ce670 0x7fe4b41464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.041+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4b415a5c0 0x7fe4b41594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4ba200700 1 -- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4b415a5c0 msgr2=0x7fe4b41594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4ba200700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4b415a5c0 0x7fe4b41594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4ba200700 1 -- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4b40ce670 msgr2=0x7fe4b41464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4ba200700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4b40ce670 0x7fe4b41464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4ba200700 1 -- 172.21.15.12:0/1946090042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a00094a0 con 0x7fe4b415dfe0 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4b9800700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4b40ce670 0x7fe4b41464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4b415a5c0 0x7fe4b41594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4ba200700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 0x7fe4b4155680 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac002230 tx=0x7fe4ac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4b1600700 1 -- 172.21.15.12:0/1946090042 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac00ac20 con 0x7fe4b415dfe0 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4b1600700 1 -- 172.21.15.12:0/1946090042 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ac00ada0 con 0x7fe4b415dfe0 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4b1600700 1 -- 172.21.15.12:0/1946090042 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac007310 con 0x7fe4b415dfe0 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4bc200700 1 -- 172.21.15.12:0/1946090042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b41582f0 con 0x7fe4b415dfe0 2024-03-29T11:34:27.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.042+0000 7fe4bc200700 1 -- 172.21.15.12:0/1946090042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b4159a10 con 0x7fe4b415dfe0 2024-03-29T11:34:27.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.044+0000 7fe4b1600700 1 -- 172.21.15.12:0/1946090042 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe4ac007470 con 0x7fe4b415dfe0 2024-03-29T11:34:27.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.045+0000 7fe4b1600700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe49406adb0 0x7fe49406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.045+0000 7fe4b1600700 1 -- 172.21.15.12:0/1946090042 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fe4ac089160 con 0x7fe4b415dfe0 2024-03-29T11:34:27.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.046+0000 7fe4b9800700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe49406adb0 0x7fe49406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.046+0000 7fe4b9800700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe49406adb0 0x7fe49406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8009c30 tx=0x7fe4a8008040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.047+0000 7fe4cfcf0b80 1 Processor -- start 2024-03-29T11:34:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.047+0000 7fe4cfcf0b80 1 -- start start 2024-03-29T11:34:27.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117f06830 0x55c117f06c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117edca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 0x55c117f03a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c117e51650 con 0x55c117ed9be0 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c117e32c30 con 0x55c117f06830 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c117e30430 con 0x55c117ed7b60 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4b8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117edca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4b8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117edca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51588/0 (socket says 172.21.15.12:51588) 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4b8e00700 1 -- 172.21.15.12:0/3561632636 learned_addr learned my addr 172.21.15.12:0/3561632636 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4b9800700 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117f06830 0x55c117f06c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4ba200700 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 0x55c117f03a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4b8e00700 1 -- 172.21.15.12:0/3561632636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 msgr2=0x55c117f03a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 0x55c117f03a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4b8e00700 1 -- 172.21.15.12:0/3561632636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117f06830 msgr2=0x55c117f06c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117f06830 0x55c117f06c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.048+0000 7fe4b8e00700 1 -- 172.21.15.12:0/3561632636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c117c1f790 con 0x55c117ed9be0 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.049+0000 7fe4ba200700 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 0x55c117f03a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:34:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.049+0000 7fe4b9800700 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117f06830 0x55c117f06c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:34:27.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.049+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117edca10 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009b60 tx=0x7fe4a0008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9965e3017a9fcbd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:27.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.049+0000 7fe4a6000700 1 -- 172.21.15.12:0/3561632636 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0011b00 con 0x55c117ed9be0 2024-03-29T11:34:27.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.049+0000 7fe4a6000700 1 -- 172.21.15.12:0/3561632636 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a0011c60 con 0x55c117ed9be0 2024-03-29T11:34:27.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.049+0000 7fe4a6000700 1 -- 172.21.15.12:0/3561632636 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0004240 con 0x55c117ed9be0 2024-03-29T11:34:27.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.050+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/3561632636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 msgr2=0x55c117edca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.050+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117edca10 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009b60 tx=0x7fe4a0008030 comp rx=0 tx=0).stop 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.050+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/3561632636 shutdown_connections 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.050+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 0x55c117f03a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.050+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117f06830 0x55c117f06c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.050+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/3561632636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117edca10 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.050+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/3561632636 >> 172.21.15.12:0/3561632636 conn(0x55c117e3a6a0 msgr2=0x55c117f21640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.050+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/3561632636 shutdown_connections 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.050+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/3561632636 wait complete. 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.051+0000 7fe4cfcf0b80 1 -- start start 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.051+0000 7fe4cfcf0b80 1 Processor -- start 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 0x55c117f09c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117f084d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117f06830 0x55c117f08a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c117e2bd20 con 0x55c117ed9be0 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c117dc7770 con 0x55c117ed7b60 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c117dea6f0 con 0x55c117f06830 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4cfcf0b80 10 client.0 _ll_register_callbacks cb 0x55c117dd15e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4b8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117f084d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4b8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117f084d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51594/0 (socket says 172.21.15.12:51594) 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4b8e00700 1 -- 172.21.15.12:0/2953735649 learned_addr learned my addr 172.21.15.12:0/2953735649 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4b9800700 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 0x55c117f09c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4ba200700 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117f06830 0x55c117f08a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4b8e00700 1 -- 172.21.15.12:0/2953735649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117f06830 msgr2=0x55c117f08a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117f06830 0x55c117f08a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4b8e00700 1 -- 172.21.15.12:0/2953735649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 msgr2=0x55c117f09c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 0x55c117f09c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4b8e00700 1 -- 172.21.15.12:0/2953735649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b40cca90 con 0x55c117ed9be0 2024-03-29T11:34:27.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.052+0000 7fe4b9800700 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 0x55c117f09c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.053+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117f084d0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fe4b406c130 tx=0x7fe4a00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa01e8621e909f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.053+0000 7fe49aa00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.053+0000 7fe49aa00700 1 -- 172.21.15.12:0/2953735649 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0011b00 con 0x55c117ed9be0 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.053+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/2953735649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c117ece920 con 0x55c117ed9be0 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.053+0000 7fe49aa00700 1 -- 172.21.15.12:0/2953735649 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fe4a001ad40 con 0x55c117ed9be0 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.053+0000 7fe49aa00700 1 -- 172.21.15.12:0/2953735649 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4a0006ca0 con 0x55c117ed9be0 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.053+0000 7fe4cfcf0b80 10 client.13260 fetch_fsmap learned FSMap version 146 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.053+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/2953735649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=146,monmap=2+}) v3 -- 0x55c117d0c0a0 con 0x55c117ed9be0 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.054+0000 7fe49aa00700 1 -- 172.21.15.12:0/2953735649 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c117ece920 con 0x55c117ed9be0 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.054+0000 7fe49aa00700 1 -- 172.21.15.12:0/2953735649 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 146) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe4a002b070 con 0x55c117ed9be0 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.054+0000 7fe4cfcf0b80 10 client.13260 fetch_fsmap finished waiting for FSMap version 146 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.054+0000 7fe4cfcf0b80 10 client.13260 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T11:34:27.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.054+0000 7fe4cfcf0b80 20 client.13260 populate_metadata read hostname 'smithi012' 2024-03-29T11:34:27.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.054+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.12:6832/2115072104,v1:172.21.15.12:6834/2115072104] conn(0x55c117f01370 0x55c117f01750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.054+0000 7fe4cfcf0b80 4 client.13260 mds_command: new command op to 13125 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:27.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.054+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/2953735649 --> [v2:172.21.15.12:6832/2115072104,v1:172.21.15.12:6834/2115072104] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c117eab6f0 con 0x55c117f01370 2024-03-29T11:34:27.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.054+0000 7fe4b9800700 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.12:6832/2115072104,v1:172.21.15.12:6834/2115072104] conn(0x55c117f01370 0x55c117f01750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.055+0000 7fe4b9800700 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.12:6832/2115072104,v1:172.21.15.12:6834/2115072104] conn(0x55c117f01370 0x55c117f01750 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd305cdc8f831033 server_cookie=3422c08cda958f3 in_seq=0 out_seq=0 2024-03-29T11:34:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.055+0000 7fe49aa00700 10 client.13260 ms_handle_connect on v2:172.21.15.12:6832/2115072104 2024-03-29T11:34:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.055+0000 7fe49aa00700 1 -- 172.21.15.12:0/2953735649 <== mds.0 v2:172.21.15.12:6832/2115072104 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c117eab6f0 con 0x55c117f01370 2024-03-29T11:34:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.055+0000 7fe49aa00700 10 client.13260 handle_command_reply: tid=0 2024-03-29T11:34:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.055+0000 7fe4cfcf0b80 1 client.13260 shutdown 2024-03-29T11:34:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/2953735649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 msgr2=0x55c117f084d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117f084d0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fe4b406c130 tx=0x7fe4a00047c0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/2953735649 shutdown_connections 2024-03-29T11:34:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.12:6832/2115072104,v1:172.21.15.12:6834/2115072104] conn(0x55c117f01370 0x55c117f01750 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117f06830 0x55c117f08a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 0x55c117f09c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/2953735649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117f084d0 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe49aa00700 0 client.13260 ms_handle_reset on v2:172.21.15.12:6832/2115072104 2024-03-29T11:34:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/2953735649 >> 172.21.15.12:0/2953735649 conn(0x55c117e3a6a0 msgr2=0x55c117edbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/2953735649 shutdown_connections 2024-03-29T11:34:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/2953735649 wait complete. 2024-03-29T11:34:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.056+0000 7fe4cfcf0b80 20 client.13260 trim_cache size 0 max 16384 2024-03-29T11:34:27.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.074+0000 7fe4cfcf0b80 1 Processor -- start 2024-03-29T11:34:27.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.074+0000 7fe4cfcf0b80 1 -- start start 2024-03-29T11:34:27.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.075+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 0x55c117f01c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.075+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117ed55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.075+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117f06830 0x55c117ed5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.075+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c117c20390 con 0x55c117ed9be0 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.075+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c117edc7d0 con 0x55c117ed7b60 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.075+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c117ed8b60 con 0x55c117f06830 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.075+0000 7fe4b8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117ed55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4b8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117ed55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51610/0 (socket says 172.21.15.12:51610) 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4b8e00700 1 -- 172.21.15.12:0/1019592190 learned_addr learned my addr 172.21.15.12:0/1019592190 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4ba200700 1 --2- 172.21.15.12:0/1019592190 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117f06830 0x55c117ed5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4b9800700 1 --2- 172.21.15.12:0/1019592190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 0x55c117f01c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4b8e00700 1 -- 172.21.15.12:0/1019592190 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117f06830 msgr2=0x55c117ed5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/1019592190 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117f06830 0x55c117ed5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4b8e00700 1 -- 172.21.15.12:0/1019592190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 msgr2=0x55c117f01c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/1019592190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 0x55c117f01c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4b8e00700 1 -- 172.21.15.12:0/1019592190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c117df2010 con 0x55c117ed9be0 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4b9800700 1 --2- 172.21.15.12:0/1019592190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 0x55c117f01c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/1019592190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117ed55a0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0014040 tx=0x7fe4a0026040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2022f8b8af62f318 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4a6a00700 1 -- 172.21.15.12:0/1019592190 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a001ad40 con 0x55c117ed9be0 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4a6a00700 1 -- 172.21.15.12:0/1019592190 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a001a7b0 con 0x55c117ed9be0 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.076+0000 7fe4a6a00700 1 -- 172.21.15.12:0/1019592190 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a001aea0 con 0x55c117ed9be0 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.077+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1019592190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 msgr2=0x55c117ed55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.077+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/1019592190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117ed55a0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0014040 tx=0x7fe4a0026040 comp rx=0 tx=0).stop 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.077+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1019592190 shutdown_connections 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.077+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/1019592190 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117f06830 0x55c117ed5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.077+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/1019592190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed7b60 0x55c117f01c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.077+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/1019592190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117ed9be0 0x55c117ed55a0 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.077+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1019592190 >> 172.21.15.12:0/1019592190 conn(0x55c117e3a6a0 msgr2=0x55c117edd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.077+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1019592190 shutdown_connections 2024-03-29T11:34:27.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.077+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1019592190 wait complete. 2024-03-29T11:34:27.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.078+0000 7fe4cfcf0b80 1 -- start start 2024-03-29T11:34:27.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.078+0000 7fe4cfcf0b80 1 Processor -- start 2024-03-29T11:34:27.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 0x55c117f882b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed9be0 0x55c117f09a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4cfcf0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117f06830 0x55c117f00ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c117f23de0 con 0x55c117f06830 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c117f1f870 con 0x55c117ed9be0 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4cfcf0b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c117f011e0 con 0x55c117ed7b60 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4cfcf0b80 10 client.0 _ll_register_callbacks cb 0x7fe4a80090e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4ba200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117f06830 0x55c117f00ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4ba200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117f06830 0x55c117f00ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51618/0 (socket says 172.21.15.12:51618) 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4ba200700 1 -- 172.21.15.12:0/1452635930 learned_addr learned my addr 172.21.15.12:0/1452635930 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4b8e00700 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed9be0 0x55c117f09a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4b9800700 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 0x55c117f882b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4ba200700 1 -- 172.21.15.12:0/1452635930 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 msgr2=0x55c117f882b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4ba200700 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 0x55c117f882b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4ba200700 1 -- 172.21.15.12:0/1452635930 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed9be0 msgr2=0x55c117f09a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4ba200700 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed9be0 0x55c117f09a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.079+0000 7fe4ba200700 1 -- 172.21.15.12:0/1452635930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c117d0c0a0 con 0x55c117f06830 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe4ba200700 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117f06830 0x55c117f00ca0 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac054300 tx=0x7fe4ac0877e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647f8648ce9599c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe49aa00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe49aa00700 1 -- 172.21.15.12:0/1452635930 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac057e00 con 0x55c117f06830 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe49aa00700 1 -- 172.21.15.12:0/1452635930 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fe4ac05bc30 con 0x55c117f06830 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1452635930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c117c65040 con 0x55c117f06830 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe49aa00700 1 -- 172.21.15.12:0/1452635930 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4ac0601f0 con 0x55c117f06830 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe4cfcf0b80 10 client.13266 fetch_fsmap learned FSMap version 146 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1452635930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=146,monmap=2+}) v3 -- 0x55c117c801d0 con 0x55c117f06830 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe49aa00700 1 -- 172.21.15.12:0/1452635930 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ac074020 con 0x55c117f06830 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe49aa00700 1 -- 172.21.15.12:0/1452635930 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 146) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe4ac087080 con 0x55c117f06830 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe4cfcf0b80 10 client.13266 fetch_fsmap finished waiting for FSMap version 146 2024-03-29T11:34:27.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe4cfcf0b80 10 client.13266 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.080+0000 7fe4cfcf0b80 20 client.13266 populate_metadata read hostname 'smithi012' 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.081+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.12:6832/2115072104,v1:172.21.15.12:6834/2115072104] conn(0x55c117ed55a0 0x55c117ed5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.081+0000 7fe4cfcf0b80 4 client.13266 mds_command: new command op to 13125 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.081+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1452635930 --> [v2:172.21.15.12:6832/2115072104,v1:172.21.15.12:6834/2115072104] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c117ca10c0 con 0x55c117ed55a0 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.081+0000 7fe4b9800700 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.12:6832/2115072104,v1:172.21.15.12:6834/2115072104] conn(0x55c117ed55a0 0x55c117ed5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.081+0000 7fe49aa00700 10 client.13266 ms_handle_connect on v2:172.21.15.12:6832/2115072104 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.081+0000 7fe4b9800700 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.12:6832/2115072104,v1:172.21.15.12:6834/2115072104] conn(0x55c117ed55a0 0x55c117ed5980 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b433c2ce93c2247 server_cookie=d592ada24b43c409 in_seq=0 out_seq=0 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe49aa00700 1 -- 172.21.15.12:0/1452635930 <== mds.0 v2:172.21.15.12:6832/2115072104 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55c117ca10c0 con 0x55c117ed55a0 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe49aa00700 10 client.13266 handle_command_reply: tid=0 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 client.13266 shutdown 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1452635930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117f06830 msgr2=0x55c117f00ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117f06830 0x55c117f00ca0 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac054300 tx=0x7fe4ac0877e0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1452635930 shutdown_connections 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.12:6832/2115072104,v1:172.21.15.12:6834/2115072104] conn(0x55c117ed55a0 0x55c117ed5980 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c117f06830 0x55c117f00ca0 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55c117ed7b60 0x55c117f882b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 --2- 172.21.15.12:0/1452635930 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55c117ed9be0 0x55c117f09a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe49aa00700 0 client.13266 ms_handle_reset on v2:172.21.15.12:6832/2115072104 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1452635930 >> 172.21.15.12:0/1452635930 conn(0x55c117e3a6a0 msgr2=0x55c117f01c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1452635930 shutdown_connections 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 1 -- 172.21.15.12:0/1452635930 wait complete. 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.082+0000 7fe4cfcf0b80 20 client.13266 trim_cache size 0 max 16384 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:34:27.084 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "216a15e0-b7ec-465c-8851-ed202d429a6c", 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 13125, 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.1552401779999997, 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 146, 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 283, 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 283, 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12.158786364999999 2024-03-29T11:34:27.085 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:34:27.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 -- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe49406adb0 msgr2=0x7fe49406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe49406adb0 0x7fe49406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8009c30 tx=0x7fe4a8008040 comp rx=0 tx=0).stop 2024-03-29T11:34:27.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 -- 172.21.15.12:0/1946090042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 msgr2=0x7fe4b4155680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 0x7fe4b4155680 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac002230 tx=0x7fe4ac00a060 comp rx=0 tx=0).stop 2024-03-29T11:34:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 -- 172.21.15.12:0/1946090042 shutdown_connections 2024-03-29T11:34:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe49406adb0 0x7fe49406d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4b415dfe0 0x7fe4b4155680 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4b40ce670 0x7fe4b41464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 --2- 172.21.15.12:0/1946090042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4b415a5c0 0x7fe4b41594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 -- 172.21.15.12:0/1946090042 >> 172.21.15.12:0/1946090042 conn(0x7fe4b40b0d70 msgr2=0x7fe4b40b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 -- 172.21.15.12:0/1946090042 shutdown_connections 2024-03-29T11:34:27.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.083+0000 7fe4bc200700 1 -- 172.21.15.12:0/1946090042 wait complete. 2024-03-29T11:34:27.098 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-29T11:34:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.298+0000 7fd88f000700 1 Processor -- start 2024-03-29T11:34:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.299+0000 7fd88f000700 1 -- start start 2024-03-29T11:34:27.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd88f000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd88815b5d0 0x7fd8880ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd88f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8880ac7d0 0x7fd88806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd88f000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88815fd50 con 0x7fd8880cb3f0 2024-03-29T11:34:27.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd88f000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88815b9b0 con 0x7fd88815b5d0 2024-03-29T11:34:27.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd88f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88815f140 con 0x7fd8880ac7d0 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd887400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd8880cb3f0 0x7fd8880ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60254/0 (socket says 172.21.15.12:60254) 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd887400700 1 -- 172.21.15.12:0/3150306032 learned_addr learned my addr 172.21.15.12:0/3150306032 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd88d000700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8880ac7d0 0x7fd88806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd887e00700 1 --2- 172.21.15.12:0/3150306032 >> v2:172.21.15.159:3300/0 conn(0x7fd88815b5d0 0x7fd8880ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20225141 0 0) 0x7fd88815fd50 con 0x7fd8880cb3f0 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.300+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd86c0035b0 con 0x7fd8880cb3f0 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 210447968 0 0) 0x7fd86c0035b0 con 0x7fd8880cb3f0 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8880ac7d0 msgr2=0x7fd88806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8880ac7d0 0x7fd88806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 >> v2:172.21.15.159:3300/0 conn(0x7fd88815b5d0 msgr2=0x7fd8880ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 --2- 172.21.15.12:0/3150306032 >> v2:172.21.15.159:3300/0 conn(0x7fd88815b5d0 0x7fd8880ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88815f420 con 0x7fd8880cb3f0 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7fd874001670 con 0x7fd8880cb3f0 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 >> v1:172.21.15.12:6789/0 conn(0x7fd8880cb3f0 legacy=0x7fd8880ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8880ac7d0 0x7fd86c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd86c003fd0 0x7fd86c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c004d90 0x7fd86c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88815b9b0 con 0x7fd86c004d90 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88815f140 con 0x7fd86c003fd0 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c00bac0 con 0x7fd8880ac7d0 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd88d000700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c004d90 0x7fd86c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd887e00700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8880ac7d0 0x7fd86c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.301+0000 7fd887400700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd86c003fd0 0x7fd86c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.302+0000 7fd88d000700 1 -- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8880ac7d0 msgr2=0x7fd86c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.302+0000 7fd88d000700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8880ac7d0 0x7fd86c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.302+0000 7fd88d000700 1 -- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd86c003fd0 msgr2=0x7fd86c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.302+0000 7fd88d000700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd86c003fd0 0x7fd86c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.302+0000 7fd88d000700 1 -- 172.21.15.12:0/3150306032 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c0035b0 con 0x7fd86c004d90 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.302+0000 7fd88d000700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c004d90 0x7fd86c005210 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7fd880009f90 tx=0x7fd86c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f156864909bdf5ca server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.302+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88000ec30 con 0x7fd86c004d90 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.302+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88000f040 con 0x7fd86c004d90 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.302+0000 7fd886a00700 1 -- 172.21.15.12:0/3150306032 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880014480 con 0x7fd86c004d90 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.303+0000 7fd88f000700 1 -- 172.21.15.12:0/3150306032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c004d90 msgr2=0x7fd86c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.303+0000 7fd88f000700 1 --2- 172.21.15.12:0/3150306032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c004d90 0x7fd86c005210 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7fd880009f90 tx=0x7fd86c005880 comp rx=0 tx=0).stop 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.303+0000 7fd88d000700 1 -- 172.21.15.12:0/3150306032 reap_dead start 2024-03-29T11:34:27.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.303+0000 7fd88f000700 1 -- 172.21.15.12:0/3150306032 shutdown_connections 2024-03-29T11:34:27.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.303+0000 7fd88f000700 1 -- 172.21.15.12:0/3150306032 >> 172.21.15.12:0/3150306032 conn(0x7fd8880cf190 msgr2=0x7fd8880cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:27.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.303+0000 7fd88f000700 1 -- 172.21.15.12:0/3150306032 shutdown_connections 2024-03-29T11:34:27.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.303+0000 7fd88f000700 1 -- 172.21.15.12:0/3150306032 wait complete. 2024-03-29T11:34:27.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.305+0000 7fd88f000700 1 Processor -- start 2024-03-29T11:34:27.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.305+0000 7fd88f000700 1 -- start start 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd88f000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c004d90 0x7fd8880af160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd88f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd88806dd70 0x7fd8881f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd88f000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8880cc860 0x7fd8880ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd88f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8880ce170 con 0x7fd88806dd70 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd88f000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8880cd530 con 0x7fd8880cc860 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd88f000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8880cd7d0 con 0x7fd86c004d90 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd887400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd88806dd70 0x7fd8881f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd887400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd88806dd70 0x7fd8881f2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51630/0 (socket says 172.21.15.12:51630) 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd887400700 1 -- 172.21.15.12:0/644020398 learned_addr learned my addr 172.21.15.12:0/644020398 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd887e00700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c004d90 0x7fd8880af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd88d000700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8880cc860 0x7fd8880ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd887400700 1 -- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c004d90 msgr2=0x7fd8880af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd887400700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c004d90 0x7fd8880af160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd887400700 1 -- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8880cc860 msgr2=0x7fd8880ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd887400700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8880cc860 0x7fd8880ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.306+0000 7fd887400700 1 -- 172.21.15.12:0/644020398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8880cdab0 con 0x7fd88806dd70 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.307+0000 7fd887400700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd88806dd70 0x7fd8881f2ab0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7fd874007fd0 tx=0x7fd8740075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.307+0000 7fd87fe00700 1 -- 172.21.15.12:0/644020398 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8740077d0 con 0x7fd88806dd70 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.307+0000 7fd87fe00700 1 -- 172.21.15.12:0/644020398 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87400f8d0 con 0x7fd88806dd70 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.307+0000 7fd87fe00700 1 -- 172.21.15.12:0/644020398 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87400fc80 con 0x7fd88806dd70 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.307+0000 7fd88f000700 1 -- 172.21.15.12:0/644020398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8880cdd90 con 0x7fd88806dd70 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.307+0000 7fd88f000700 1 -- 172.21.15.12:0/644020398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8880ce7a0 con 0x7fd88806dd70 2024-03-29T11:34:27.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.308+0000 7fd88f000700 1 -- 172.21.15.12:0/644020398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd888067720 con 0x7fd88806dd70 2024-03-29T11:34:27.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.309+0000 7fd87fe00700 1 -- 172.21.15.12:0/644020398 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd87401d460 con 0x7fd88806dd70 2024-03-29T11:34:27.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.310+0000 7fd87fe00700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd86806adb0 0x7fd86806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:27.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.310+0000 7fd87fe00700 1 -- 172.21.15.12:0/644020398 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fd8740895d0 con 0x7fd88806dd70 2024-03-29T11:34:27.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.310+0000 7fd887e00700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd86806adb0 0x7fd86806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:27.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.311+0000 7fd887e00700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd86806adb0 0x7fd86806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd878009c20 tx=0x7fd878008040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:27.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.314+0000 7fd87fe00700 1 -- 172.21.15.12:0/644020398 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd874054640 con 0x7fd88806dd70 2024-03-29T11:34:27.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.584+0000 7fd88f000700 1 -- 172.21.15.12:0/644020398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd88815e560 con 0x7fd88806dd70 2024-03-29T11:34:27.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.585+0000 7fd87fe00700 1 -- 172.21.15.12:0/644020398 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v146) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd874057c60 con 0x7fd88806dd70 2024-03-29T11:34:27.582 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-29T11:34:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 -- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd86806adb0 msgr2=0x7fd86806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd86806adb0 0x7fd86806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd878009c20 tx=0x7fd878008040 comp rx=0 tx=0).stop 2024-03-29T11:34:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 -- 172.21.15.12:0/644020398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd88806dd70 msgr2=0x7fd8881f2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd88806dd70 0x7fd8881f2ab0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7fd874007fd0 tx=0x7fd8740075f0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 -- 172.21.15.12:0/644020398 shutdown_connections 2024-03-29T11:34:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd86806adb0 0x7fd86806d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8880cc860 0x7fd8880ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c004d90 0x7fd8880af160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 --2- 172.21.15.12:0/644020398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd88806dd70 0x7fd8881f2ab0 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 -- 172.21.15.12:0/644020398 >> 172.21.15.12:0/644020398 conn(0x7fd8880cf190 msgr2=0x7fd88806cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.587+0000 7fd88f000700 1 -- 172.21.15.12:0/644020398 shutdown_connections 2024-03-29T11:34:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:27.588+0000 7fd88f000700 1 -- 172.21.15.12:0/644020398 wait complete. 2024-03-29T11:34:27.603 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-29T11:34:27.603 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-29T11:34:27.603 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-29T11:34:27.603 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-29T11:34:27.603 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-29T11:34:27.603 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-29T11:34:27.603 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-29T11:34:27.604 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-29T11:34:27.604 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:34:27.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:27.604 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:34:27.650 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 35450sec preferred_lft 35450sec 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-29T11:34:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::d81b:e0ff:fe29:a4d7/64 scope link 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run.smithi012.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 66:b2:5e:0c:39:78 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::64b2:5eff:fe0c:3978/64 scope link 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run.smithi012.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 6e:45:3f:9e:e4:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::6c45:3fff:fe9e:e4b9/64 scope link 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-29T11:34:27.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:27.653 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-29T11:34:27.755 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-29T11:34:27.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:27.758 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-29T11:34:27.806 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-29T11:34:27.806 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-29T11:34:27.806 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-29T11:34:27.806 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-29T11:34:27.806 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-29T11:34:27.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:27.807 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-29T11:34:27.807 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-29T11:34:27.807 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-29T11:34:27.807 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-29T11:34:27.807 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-29T11:34:27.807 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-29T11:34:27.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:27.974 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T11:34:28.018 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T11:34:28.018 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T11:34:28.019 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:34:28.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:28.020 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-29T11:34:28.129 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:34:28.129 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-29T11:34:28.174 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-29T11:34:28.174 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-29T11:34:28.174 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:34:28.257 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-29T11:34:28.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:28.258 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:34:28.401 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:34:28.401 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:34:28.428 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-29T11:34:28.428 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-29T11:34:28.428 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-29T11:34:28.428 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@216a15e0-b7ec-465c-8851-ed202d429a6c.cephfs=/ 2024-03-29T11:34:28.428 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.12:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-03-29T11:34:28.433 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-29T11:34:28.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:28.433 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-29T11:34:28.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:34:28.501 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-29T11:34:28.601 INFO:teuthology.orchestra.run.smithi012.stdout:client12673 2024-03-29T11:34:28.602 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:34:28.602 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:34:28.646 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-29T11:34:28.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.888+0000 7fdbaea00700 1 Processor -- start 2024-03-29T11:34:28.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.888+0000 7fdbaea00700 1 -- start start 2024-03-29T11:34:28.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.889+0000 7fdbaea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdba815a5c0 0x7fdba80709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:28.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.889+0000 7fdbaea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdba80ce840 0x7fdba80cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:28.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.889+0000 7fdbaea00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba80cb310 con 0x7fdba815dfe0 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.889+0000 7fdbaea00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba8070f00 con 0x7fdba815a5c0 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.889+0000 7fdbaea00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba80711a0 con 0x7fdba80ce840 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.889+0000 7fdba7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdba815a5c0 0x7fdba80709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.890+0000 7fdba7e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdba815a5c0 0x7fdba80709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46088/0 (socket says 172.21.15.12:46088) 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.890+0000 7fdba7e00700 1 -- 172.21.15.12:0/3407530807 learned_addr learned my addr 172.21.15.12:0/3407530807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.890+0000 7fdba7e00700 1 -- 172.21.15.12:0/3407530807 >> v2:172.21.15.159:3300/0 conn(0x7fdba80ce840 msgr2=0x7fdba80cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.890+0000 7fdbaca00700 1 --2- 172.21.15.12:0/3407530807 >> v2:172.21.15.159:3300/0 conn(0x7fdba80ce840 0x7fdba80cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.890+0000 7fdba7e00700 1 --2- 172.21.15.12:0/3407530807 >> v2:172.21.15.159:3300/0 conn(0x7fdba80ce840 0x7fdba80cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.890+0000 7fdba7e00700 1 -- 172.21.15.12:0/3407530807 >> v1:172.21.15.159:6789/0 conn(0x7fdba815dfe0 legacy=0x7fdba80ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.890+0000 7fdba7e00700 1 -- 172.21.15.12:0/3407530807 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba8071480 con 0x7fdba815a5c0 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.890+0000 7fdbaca00700 1 --2- 172.21.15.12:0/3407530807 >> v2:172.21.15.159:3300/0 conn(0x7fdba80ce840 0x7fdba80cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:34:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.890+0000 7fdba7e00700 1 --2- 172.21.15.12:0/3407530807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba815a5c0 0x7fdba80709c0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7fdba806b630 tx=0x7fdb98009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5c9e0e15a89485 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:28.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.890+0000 7fdba6a00700 1 -- 172.21.15.12:0/3407530807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98004230 con 0x7fdba815a5c0 2024-03-29T11:34:28.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.891+0000 7fdba6a00700 1 -- 172.21.15.12:0/3407530807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb98004390 con 0x7fdba815a5c0 2024-03-29T11:34:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.891+0000 7fdba6a00700 1 -- 172.21.15.12:0/3407530807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb980036a0 con 0x7fdba815a5c0 2024-03-29T11:34:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.891+0000 7fdbaea00700 1 -- 172.21.15.12:0/3407530807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba815a5c0 msgr2=0x7fdba80709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.891+0000 7fdbaea00700 1 --2- 172.21.15.12:0/3407530807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba815a5c0 0x7fdba80709c0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7fdba806b630 tx=0x7fdb98009bd0 comp rx=0 tx=0).stop 2024-03-29T11:34:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.891+0000 7fdbaea00700 1 -- 172.21.15.12:0/3407530807 shutdown_connections 2024-03-29T11:34:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.891+0000 7fdbaea00700 1 --2- 172.21.15.12:0/3407530807 >> v2:172.21.15.159:3300/0 conn(0x7fdba80ce840 0x7fdba80cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.891+0000 7fdbaea00700 1 --2- 172.21.15.12:0/3407530807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba815a5c0 0x7fdba80709c0 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.891+0000 7fdbaea00700 1 -- 172.21.15.12:0/3407530807 >> 172.21.15.12:0/3407530807 conn(0x7fdba80b0d70 msgr2=0x7fdba80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.891+0000 7fdbaea00700 1 -- 172.21.15.12:0/3407530807 shutdown_connections 2024-03-29T11:34:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.891+0000 7fdbaea00700 1 -- 172.21.15.12:0/3407530807 wait complete. 2024-03-29T11:34:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.893+0000 7fdbaea00700 1 Processor -- start 2024-03-29T11:34:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.893+0000 7fdbaea00700 1 -- start start 2024-03-29T11:34:28.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdbaea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba80ce840 0x7fdba814dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdbaea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdba815a5c0 0x7fdba80af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdbaea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdba815dfe0 0x7fdba8148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdbaea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba814d7d0 con 0x7fdba80ce840 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdbaea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba814c720 con 0x7fdba815dfe0 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdbaea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba815c6d0 con 0x7fdba815a5c0 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdba7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba80ce840 0x7fdba814dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdba7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba80ce840 0x7fdba814dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46100/0 (socket says 172.21.15.12:46100) 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdba7e00700 1 -- 172.21.15.12:0/50464429 learned_addr learned my addr 172.21.15.12:0/50464429 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdbaca00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdba815dfe0 0x7fdba8148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.894+0000 7fdba7400700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdba815a5c0 0x7fdba80af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdba7e00700 1 -- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdba815a5c0 msgr2=0x7fdba80af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdba7e00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdba815a5c0 0x7fdba80af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:28.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdba7e00700 1 -- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdba815dfe0 msgr2=0x7fdba8148250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdba7e00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdba815dfe0 0x7fdba8148250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:28.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdba7e00700 1 -- 172.21.15.12:0/50464429 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb98009550 con 0x7fdba80ce840 2024-03-29T11:34:28.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdba7e00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba80ce840 0x7fdba814dbf0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009fd0 tx=0x7fdb98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:28.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdb9fe00700 1 -- 172.21.15.12:0/50464429 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98003d70 con 0x7fdba80ce840 2024-03-29T11:34:28.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdb9fe00700 1 -- 172.21.15.12:0/50464429 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9801f070 con 0x7fdba80ce840 2024-03-29T11:34:28.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdbaea00700 1 -- 172.21.15.12:0/50464429 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba814d110 con 0x7fdba80ce840 2024-03-29T11:34:28.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdb9fe00700 1 -- 172.21.15.12:0/50464429 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98011870 con 0x7fdba80ce840 2024-03-29T11:34:28.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.895+0000 7fdbaea00700 1 -- 172.21.15.12:0/50464429 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba815c9b0 con 0x7fdba80ce840 2024-03-29T11:34:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.896+0000 7fdbaea00700 1 -- 172.21.15.12:0/50464429 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba814da40 con 0x7fdba80ce840 2024-03-29T11:34:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.897+0000 7fdb9fe00700 1 -- 172.21.15.12:0/50464429 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdb98021030 con 0x7fdba80ce840 2024-03-29T11:34:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.898+0000 7fdb9fe00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdb8806adb0 0x7fdb8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.898+0000 7fdb9fe00700 1 -- 172.21.15.12:0/50464429 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fdb9808ab60 con 0x7fdba80ce840 2024-03-29T11:34:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.899+0000 7fdba7400700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdb8806adb0 0x7fdb8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.899+0000 7fdba7400700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdb8806adb0 0x7fdb8806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdba814e680 tx=0x7fdb94008040 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:28.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:28.902+0000 7fdb9fe00700 1 -- 172.21.15.12:0/50464429 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdb98055c70 con 0x7fdba80ce840 2024-03-29T11:34:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.145+0000 7fdbaea00700 1 -- 172.21.15.12:0/50464429 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdba80cb560 con 0x7fdb8806adb0 2024-03-29T11:34:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.146+0000 7fdb9fe00700 1 -- 172.21.15.12:0/50464429 <== mgr.10517 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fdb94009500 con 0x7fdb8806adb0 2024-03-29T11:34:29.144 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-29T11:34:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-29T11:34:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephfs" 2024-03-29T11:34:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-29T11:34:29.144 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 -- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdb8806adb0 msgr2=0x7fdb8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdb8806adb0 0x7fdb8806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdba814e680 tx=0x7fdb94008040 comp rx=0 tx=0).stop 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 -- 172.21.15.12:0/50464429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba80ce840 msgr2=0x7fdba814dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba80ce840 0x7fdba814dbf0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009fd0 tx=0x7fdb98004750 comp rx=0 tx=0).stop 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 -- 172.21.15.12:0/50464429 shutdown_connections 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdb8806adb0 0x7fdb8806d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdba815dfe0 0x7fdba8148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdba80ce840 0x7fdba814dbf0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 --2- 172.21.15.12:0/50464429 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdba815a5c0 0x7fdba80af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 -- 172.21.15.12:0/50464429 >> 172.21.15.12:0/50464429 conn(0x7fdba80b0d70 msgr2=0x7fdba80b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 -- 172.21.15.12:0/50464429 shutdown_connections 2024-03-29T11:34:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.149+0000 7fdbaea00700 1 -- 172.21.15.12:0/50464429 wait complete. 2024-03-29T11:34:29.165 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-29T11:34:29.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.368+0000 7efd4ba00700 1 Processor -- start 2024-03-29T11:34:29.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.368+0000 7efd4ba00700 1 -- start start 2024-03-29T11:34:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.369+0000 7efd4ba00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efd440d2540 0x7efd441687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.369+0000 7efd4ba00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd44161420 con 0x7efd4415d9d0 2024-03-29T11:34:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.369+0000 7efd4ba00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd44161700 con 0x7efd440d16a0 2024-03-29T11:34:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.369+0000 7efd4ba00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd440d4340 con 0x7efd440d2540 2024-03-29T11:34:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.369+0000 7efd49000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7efd4415d9d0 0x7efd440d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58082/0 (socket says 172.21.15.12:58082) 2024-03-29T11:34:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.369+0000 7efd49000700 1 -- 172.21.15.12:0/4143298255 learned_addr learned my addr 172.21.15.12:0/4143298255 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.369+0000 7efd49a00700 1 --2- 172.21.15.12:0/4143298255 >> v2:172.21.15.159:3300/0 conn(0x7efd440d2540 0x7efd441687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.369+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274384510 0 0) 0x7efd44161420 con 0x7efd4415d9d0 2024-03-29T11:34:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.369+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd2c0035b0 con 0x7efd4415d9d0 2024-03-29T11:34:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3093921561 0 0) 0x7efd2c0035b0 con 0x7efd4415d9d0 2024-03-29T11:34:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 >> v2:172.21.15.159:3300/0 conn(0x7efd440d2540 msgr2=0x7efd441687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 --2- 172.21.15.12:0/4143298255 >> v2:172.21.15.159:3300/0 conn(0x7efd440d2540 0x7efd441687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 >> v1:172.21.15.159:6789/0 conn(0x7efd440d16a0 legacy=0x7efd440d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd441619e0 con 0x7efd4415d9d0 2024-03-29T11:34:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7efd38001670 con 0x7efd4415d9d0 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 >> v1:172.21.15.12:6789/0 conn(0x7efd4415d9d0 legacy=0x7efd440d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd2c003a50 0x7efd2c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd2c004850 0x7efd2c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd2c005260 0x7efd2c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c00c180 con 0x7efd2c003a50 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c00c2f0 con 0x7efd2c004850 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c00c4b0 con 0x7efd2c005260 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd49000700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd2c003a50 0x7efd2c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.370+0000 7efd43e00700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd2c004850 0x7efd2c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd49a00700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd2c005260 0x7efd2c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd49000700 1 -- 172.21.15.12:0/4143298255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd2c005260 msgr2=0x7efd2c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd49000700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd2c005260 0x7efd2c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd49000700 1 -- 172.21.15.12:0/4143298255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd2c004850 msgr2=0x7efd2c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:29.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd49000700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd2c004850 0x7efd2c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd49000700 1 -- 172.21.15.12:0/4143298255 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c0035b0 con 0x7efd2c003a50 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd49a00700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd2c005260 0x7efd2c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd43e00700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd2c004850 0x7efd2c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd49a00700 1 -- 172.21.15.12:0/4143298255 reap_dead start 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd49000700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd2c003a50 0x7efd2c003e70 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7efd38001eb0 tx=0x7efd2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b3de6e32d253b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd38007b70 con 0x7efd2c003a50 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3800ccf0 con 0x7efd2c003a50 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.371+0000 7efd43400700 1 -- 172.21.15.12:0/4143298255 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd38005ba0 con 0x7efd2c003a50 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.372+0000 7efd4ba00700 1 -- 172.21.15.12:0/4143298255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd2c003a50 msgr2=0x7efd2c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.372+0000 7efd4ba00700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd2c003a50 0x7efd2c003e70 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7efd38001eb0 tx=0x7efd2c005880 comp rx=0 tx=0).stop 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.372+0000 7efd4ba00700 1 -- 172.21.15.12:0/4143298255 shutdown_connections 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.372+0000 7efd4ba00700 1 --2- 172.21.15.12:0/4143298255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd2c003a50 0x7efd2c003e70 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.372+0000 7efd4ba00700 1 -- 172.21.15.12:0/4143298255 >> 172.21.15.12:0/4143298255 conn(0x7efd4408aff0 msgr2=0x7efd440d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.372+0000 7efd4ba00700 1 -- 172.21.15.12:0/4143298255 shutdown_connections 2024-03-29T11:34:29.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.372+0000 7efd4ba00700 1 -- 172.21.15.12:0/4143298255 wait complete. 2024-03-29T11:34:29.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.373+0000 7efd4ba00700 1 Processor -- start 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.374+0000 7efd4ba00700 1 -- start start 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd4ba00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd2c003a50 0x7efd441f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd4ba00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd44158b80 0x7efd440b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd4ba00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd440b4ad0 0x7efd4415f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd4ba00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd440b6b60 con 0x7efd44158b80 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd4ba00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd441601c0 con 0x7efd2c003a50 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd4ba00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd440b4f50 con 0x7efd440b4ad0 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd43e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd44158b80 0x7efd440b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd43e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd44158b80 0x7efd440b4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46114/0 (socket says 172.21.15.12:46114) 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd43e00700 1 -- 172.21.15.12:0/1920726618 learned_addr learned my addr 172.21.15.12:0/1920726618 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd49000700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd2c003a50 0x7efd441f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd49a00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd440b4ad0 0x7efd4415f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd43e00700 1 -- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd440b4ad0 msgr2=0x7efd4415f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:29.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd43e00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd440b4ad0 0x7efd4415f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd43e00700 1 -- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd2c003a50 msgr2=0x7efd441f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd43e00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd2c003a50 0x7efd441f45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.375+0000 7efd43e00700 1 -- 172.21.15.12:0/1920726618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd440b61c0 con 0x7efd44158b80 2024-03-29T11:34:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.376+0000 7efd43e00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd44158b80 0x7efd440b4590 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7efd3000e480 tx=0x7efd3000e790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.376+0000 7efd40c00700 1 -- 172.21.15.12:0/1920726618 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3000ee70 con 0x7efd44158b80 2024-03-29T11:34:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.376+0000 7efd40c00700 1 -- 172.21.15.12:0/1920726618 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd300052e0 con 0x7efd44158b80 2024-03-29T11:34:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.376+0000 7efd4ba00700 1 -- 172.21.15.12:0/1920726618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd440b64a0 con 0x7efd44158b80 2024-03-29T11:34:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.376+0000 7efd40c00700 1 -- 172.21.15.12:0/1920726618 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd300036d0 con 0x7efd44158b80 2024-03-29T11:34:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.376+0000 7efd4ba00700 1 -- 172.21.15.12:0/1920726618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd44159000 con 0x7efd44158b80 2024-03-29T11:34:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.377+0000 7efd4ba00700 1 -- 172.21.15.12:0/1920726618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd440b6ca0 con 0x7efd44158b80 2024-03-29T11:34:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.378+0000 7efd40c00700 1 -- 172.21.15.12:0/1920726618 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 156) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efd30003830 con 0x7efd44158b80 2024-03-29T11:34:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.379+0000 7efd40c00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7efd2406adb0 0x7efd2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.379+0000 7efd40c00700 1 -- 172.21.15.12:0/1920726618 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7efd30088970 con 0x7efd44158b80 2024-03-29T11:34:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.379+0000 7efd49000700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7efd2406adb0 0x7efd2406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.380+0000 7efd49000700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7efd2406adb0 0x7efd2406d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efd38000c00 tx=0x7efd38007000 comp rx=0 tx=0).ready entity=mgr.10517 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.384+0000 7efd40c00700 1 -- 172.21.15.12:0/1920726618 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd440b6ca0 con 0x7efd44158b80 2024-03-29T11:34:29.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:29.655+0000 7efd4ba00700 1 -- 172.21.15.12:0/1920726618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7efd440b6ca0 con 0x7efd44158b80 2024-03-29T11:34:30.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.455+0000 7efd40c00700 1 -- 172.21.15.12:0/1920726618 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 157) v1 ==== 89256+0+0 (secure 0 0 0) 0x7efd30053e60 con 0x7efd44158b80 2024-03-29T11:34:30.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.457+0000 7efd40c00700 1 -- 172.21.15.12:0/1920726618 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v157) v1 ==== 92+0+0 (secure 0 0 0) 0x7efd30070de0 con 0x7efd44158b80 2024-03-29T11:34:30.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.460+0000 7efd4ba00700 1 -- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7efd2406adb0 msgr2=0x7efd2406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.460+0000 7efd4ba00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7efd2406adb0 0x7efd2406d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efd38000c00 tx=0x7efd38007000 comp rx=0 tx=0).stop 2024-03-29T11:34:30.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.460+0000 7efd4ba00700 1 -- 172.21.15.12:0/1920726618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd44158b80 msgr2=0x7efd440b4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.460+0000 7efd4ba00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd44158b80 0x7efd440b4590 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7efd3000e480 tx=0x7efd3000e790 comp rx=0 tx=0).stop 2024-03-29T11:34:30.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.461+0000 7efd4ba00700 1 -- 172.21.15.12:0/1920726618 shutdown_connections 2024-03-29T11:34:30.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.461+0000 7efd4ba00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7efd2406adb0 0x7efd2406d270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.461+0000 7efd4ba00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd440b4ad0 0x7efd4415f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.461+0000 7efd4ba00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd2c003a50 0x7efd441f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.461+0000 7efd4ba00700 1 --2- 172.21.15.12:0/1920726618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd44158b80 0x7efd440b4590 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.461+0000 7efd4ba00700 1 -- 172.21.15.12:0/1920726618 >> 172.21.15.12:0/1920726618 conn(0x7efd4408aff0 msgr2=0x7efd440b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:30.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.461+0000 7efd4ba00700 1 -- 172.21.15.12:0/1920726618 shutdown_connections 2024-03-29T11:34:30.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.461+0000 7efd4ba00700 1 -- 172.21.15.12:0/1920726618 wait complete. 2024-03-29T11:34:30.487 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-29T11:34:30.676 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:30.678+0000 7f431e91a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:34:30.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.697+0000 7ff6b9400700 1 Processor -- start 2024-03-29T11:34:30.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.697+0000 7ff6b9400700 1 -- start start 2024-03-29T11:34:30.695 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:30.698+0000 7fa0de43c240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.698+0000 7ff6b9400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6b415dad0 0x7ff6b40ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.698+0000 7ff6b9400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff6b40d21d0 0x7ff6b415fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.698+0000 7ff6b9400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b415f820 con 0x7ff6b40d1630 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.698+0000 7ff6b9400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b40cfcc0 con 0x7ff6b415dad0 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.698+0000 7ff6b9400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b40cd9f0 con 0x7ff6b40d21d0 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.699+0000 7ff6b1e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff6b40d1630 0x7ff6b40d1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58092/0 (socket says 172.21.15.12:58092) 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.699+0000 7ff6b2800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6b415dad0 0x7ff6b40ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.699+0000 7ff6b1e00700 1 -- 172.21.15.12:0/237857144 learned_addr learned my addr 172.21.15.12:0/237857144 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:30.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.699+0000 7ff6b3200700 1 --2- 172.21.15.12:0/237857144 >> v2:172.21.15.159:3300/0 conn(0x7ff6b40d21d0 0x7ff6b415fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:30.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.699+0000 7ff6b2800700 1 -- 172.21.15.12:0/237857144 >> v2:172.21.15.159:3300/0 conn(0x7ff6b40d21d0 msgr2=0x7ff6b415fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:30.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.699+0000 7ff6b2800700 1 --2- 172.21.15.12:0/237857144 >> v2:172.21.15.159:3300/0 conn(0x7ff6b40d21d0 0x7ff6b415fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.699+0000 7ff6b2800700 1 -- 172.21.15.12:0/237857144 >> v1:172.21.15.12:6789/0 conn(0x7ff6b40d1630 legacy=0x7ff6b40d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.699+0000 7ff6b2800700 1 -- 172.21.15.12:0/237857144 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b40cf420 con 0x7ff6b415dad0 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.699+0000 7ff6b1400700 1 -- 172.21.15.12:0/237857144 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805892674 0 0) 0x7ff6b415f820 con 0x7ff6b40d1630 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.700+0000 7ff6b2800700 1 --2- 172.21.15.12:0/237857144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 0x7ff6b40ced80 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7ff6b408ccb0 tx=0x7ff6a4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d3f7d6d31b1cbd server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.700+0000 7ff6b1400700 1 -- 172.21.15.12:0/237857144 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4011830 con 0x7ff6b415dad0 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.700+0000 7ff6b1400700 1 -- 172.21.15.12:0/237857144 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a4011990 con 0x7ff6b415dad0 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.700+0000 7ff6b1400700 1 -- 172.21.15.12:0/237857144 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4011d20 con 0x7ff6b415dad0 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.701+0000 7ff6b9400700 1 -- 172.21.15.12:0/237857144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 msgr2=0x7ff6b40ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.701+0000 7ff6b9400700 1 --2- 172.21.15.12:0/237857144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 0x7ff6b40ced80 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7ff6b408ccb0 tx=0x7ff6a4009b20 comp rx=0 tx=0).stop 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.701+0000 7ff6b9400700 1 -- 172.21.15.12:0/237857144 shutdown_connections 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.701+0000 7ff6b9400700 1 --2- 172.21.15.12:0/237857144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 0x7ff6b40ced80 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.701+0000 7ff6b9400700 1 --2- 172.21.15.12:0/237857144 >> v2:172.21.15.159:3300/0 conn(0x7ff6b40d21d0 0x7ff6b415fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.701+0000 7ff6b9400700 1 -- 172.21.15.12:0/237857144 >> 172.21.15.12:0/237857144 conn(0x7ff6b408aff0 msgr2=0x7ff6b40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.701+0000 7ff6b9400700 1 -- 172.21.15.12:0/237857144 shutdown_connections 2024-03-29T11:34:30.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.701+0000 7ff6b9400700 1 -- 172.21.15.12:0/237857144 wait complete. 2024-03-29T11:34:30.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.702+0000 7ff6b9400700 1 Processor -- start 2024-03-29T11:34:30.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.703+0000 7ff6b9400700 1 -- start start 2024-03-29T11:34:30.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.703+0000 7ff6b9400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6b40d1630 0x7ff6b414f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:30.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.703+0000 7ff6b9400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6b40d21d0 0x7ff6b406cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:30.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 0x7ff6b414f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:30.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b9400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b40b6770 con 0x7ff6b415dad0 2024-03-29T11:34:30.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b9400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b406ba90 con 0x7ff6b40d1630 2024-03-29T11:34:30.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b9400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b406bce0 con 0x7ff6b40d21d0 2024-03-29T11:34:30.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b3200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 0x7ff6b414f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:30.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b3200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 0x7ff6b414f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46146/0 (socket says 172.21.15.12:46146) 2024-03-29T11:34:30.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b3200700 1 -- 172.21.15.12:0/104984471 learned_addr learned my addr 172.21.15.12:0/104984471 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:30.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b2800700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6b40d1630 0x7ff6b414f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:30.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b1e00700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6b40d21d0 0x7ff6b406cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:30.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b3200700 1 -- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6b40d21d0 msgr2=0x7ff6b406cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b3200700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6b40d21d0 0x7ff6b406cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b3200700 1 -- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6b40d1630 msgr2=0x7ff6b414f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b3200700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6b40d1630 0x7ff6b414f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b3200700 1 -- 172.21.15.12:0/104984471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a40094a0 con 0x7ff6b415dad0 2024-03-29T11:34:30.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.704+0000 7ff6b2800700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6b40d1630 0x7ff6b414f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:34:30.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.705+0000 7ff6b3200700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 0x7ff6b414f870 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00efc0 tx=0x7ff6ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:30.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.705+0000 7ff6aaa00700 1 -- 172.21.15.12:0/104984471 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac009ea0 con 0x7ff6b415dad0 2024-03-29T11:34:30.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.705+0000 7ff6aaa00700 1 -- 172.21.15.12:0/104984471 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ac00f040 con 0x7ff6b415dad0 2024-03-29T11:34:30.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.705+0000 7ff6aaa00700 1 -- 172.21.15.12:0/104984471 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac00ced0 con 0x7ff6b415dad0 2024-03-29T11:34:30.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.705+0000 7ff6b9400700 1 -- 172.21.15.12:0/104984471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b406bfc0 con 0x7ff6b415dad0 2024-03-29T11:34:30.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.705+0000 7ff6b9400700 1 -- 172.21.15.12:0/104984471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b406c580 con 0x7ff6b415dad0 2024-03-29T11:34:30.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.706+0000 7ff6b9400700 1 -- 172.21.15.12:0/104984471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b4067720 con 0x7ff6b415dad0 2024-03-29T11:34:30.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.707+0000 7ff6aaa00700 1 -- 172.21.15.12:0/104984471 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 157) v1 ==== 89256+0+0 (secure 0 0 0) 0x7ff6ac014480 con 0x7ff6b415dad0 2024-03-29T11:34:30.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.708+0000 7ff6aaa00700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff69406ae00 0x7ff69406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:30.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.708+0000 7ff6aaa00700 1 -- 172.21.15.12:0/104984471 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7ff6ac08a1b0 con 0x7ff6b415dad0 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.708+0000 7ff6b2800700 1 -- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff69406ae00 msgr2=0x7ff69406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.708+0000 7ff6b2800700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff69406ae00 0x7ff69406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.717+0000 7ff6aaa00700 1 -- 172.21.15.12:0/104984471 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6ac055120 con 0x7ff6b415dad0 2024-03-29T11:34:30.863 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:30.865+0000 7f431e91a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:34:30.905 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:30.908+0000 7fa0de43c240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.909+0000 7ff6b2800700 1 -- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff69406ae00 msgr2=0x7ff69406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.909+0000 7ff6b2800700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff69406ae00 0x7ff69406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:30.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.971+0000 7ff6b9400700 1 -- 172.21.15.12:0/104984471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff6b40cef20 con 0x7ff6b415dad0 2024-03-29T11:34:30.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.975+0000 7ff6aaa00700 1 -- 172.21.15.12:0/104984471 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v95)=0 v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff6ac058740 con 0x7ff6b415dad0 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 -- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff69406ae00 msgr2=0x7ff69406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff69406ae00 0x7ff69406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 -- 172.21.15.12:0/104984471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 msgr2=0x7ff6b414f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 0x7ff6b414f870 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00efc0 tx=0x7ff6ac00c3a0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 -- 172.21.15.12:0/104984471 shutdown_connections 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff69406ae00 0x7ff69406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6b415dad0 0x7ff6b414f870 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6b40d1630 0x7ff6b414f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 --2- 172.21.15.12:0/104984471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6b40d21d0 0x7ff6b406cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 -- 172.21.15.12:0/104984471 >> 172.21.15.12:0/104984471 conn(0x7ff6b408aff0 msgr2=0x7ff6b415c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 -- 172.21.15.12:0/104984471 shutdown_connections 2024-03-29T11:34:30.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:30.978+0000 7ff6b9400700 1 -- 172.21.15.12:0/104984471 wait complete. 2024-03-29T11:34:30.995 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-29T11:34:31.184 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:31.186+0000 7f431e91a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:34:31.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.201+0000 7fa29ca00700 1 Processor -- start 2024-03-29T11:34:31.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.201+0000 7fa29ca00700 1 -- start start 2024-03-29T11:34:31.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.202+0000 7fa29ca00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2980cd5c0 0x7fa29815e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.202+0000 7fa29ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa29806f310 0x7fa29806f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa29ca00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29815eda0 con 0x7fa29815cfe0 2024-03-29T11:34:31.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa29ca00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29806fce0 con 0x7fa2980cd5c0 2024-03-29T11:34:31.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa29ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa298099dd0 con 0x7fa29806f310 2024-03-29T11:34:31.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa295e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2980cd5c0 0x7fa29815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:31.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa296800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa29815cfe0 0x7fa29815d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58104/0 (socket says 172.21.15.12:58104) 2024-03-29T11:34:31.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa296800700 1 -- 172.21.15.12:0/1013383699 learned_addr learned my addr 172.21.15.12:0/1013383699 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:31.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa295400700 1 --2- 172.21.15.12:0/1013383699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa29806f310 0x7fa29806f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:31.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa295e00700 1 -- 172.21.15.12:0/1013383699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa29806f310 msgr2=0x7fa29806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:31.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa295e00700 1 --2- 172.21.15.12:0/1013383699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa29806f310 0x7fa29806f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa295e00700 1 -- 172.21.15.12:0/1013383699 >> v1:172.21.15.12:6789/0 conn(0x7fa29815cfe0 legacy=0x7fa29815d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa295e00700 1 -- 172.21.15.12:0/1013383699 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29809a0b0 con 0x7fa2980cd5c0 2024-03-29T11:34:31.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.203+0000 7fa294a00700 1 -- 172.21.15.12:0/1013383699 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019084919 0 0) 0x7fa290001490 con 0x7fa29815cfe0 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.204+0000 7fa295e00700 1 --2- 172.21.15.12:0/1013383699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2980cd5c0 0x7fa29815e860 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fa288009840 tx=0x7fa288009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd05c23936491cf server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.204+0000 7fa294a00700 1 -- 172.21.15.12:0/1013383699 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288011830 con 0x7fa2980cd5c0 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.204+0000 7fa294a00700 1 -- 172.21.15.12:0/1013383699 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa288011990 con 0x7fa2980cd5c0 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.204+0000 7fa294a00700 1 -- 172.21.15.12:0/1013383699 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288011d20 con 0x7fa2980cd5c0 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.205+0000 7fa29ca00700 1 -- 172.21.15.12:0/1013383699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2980cd5c0 msgr2=0x7fa29815e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.205+0000 7fa29ca00700 1 --2- 172.21.15.12:0/1013383699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2980cd5c0 0x7fa29815e860 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fa288009840 tx=0x7fa288009b50 comp rx=0 tx=0).stop 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.205+0000 7fa29ca00700 1 -- 172.21.15.12:0/1013383699 shutdown_connections 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.205+0000 7fa29ca00700 1 --2- 172.21.15.12:0/1013383699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2980cd5c0 0x7fa29815e860 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.205+0000 7fa29ca00700 1 --2- 172.21.15.12:0/1013383699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa29806f310 0x7fa29806f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.205+0000 7fa29ca00700 1 -- 172.21.15.12:0/1013383699 >> 172.21.15.12:0/1013383699 conn(0x7fa2980d1370 msgr2=0x7fa2980d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.205+0000 7fa29ca00700 1 -- 172.21.15.12:0/1013383699 shutdown_connections 2024-03-29T11:34:31.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.205+0000 7fa29ca00700 1 -- 172.21.15.12:0/1013383699 wait complete. 2024-03-29T11:34:31.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.207+0000 7fa29ca00700 1 Processor -- start 2024-03-29T11:34:31.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.207+0000 7fa29ca00700 1 -- start start 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa29ca00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa29806f310 0x7fa2980afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa29ca00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2980cd5c0 0x7fa2980b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa29ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa29815cfe0 0x7fa29806b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa29ca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2981592d0 con 0x7fa29815cfe0 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa29ca00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2980b68c0 con 0x7fa29806f310 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa29ca00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2980b65e0 con 0x7fa2980cd5c0 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa296800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa29815cfe0 0x7fa29806b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa296800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa29815cfe0 0x7fa29806b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46166/0 (socket says 172.21.15.12:46166) 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa296800700 1 -- 172.21.15.12:0/1742930073 learned_addr learned my addr 172.21.15.12:0/1742930073 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa295400700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2980cd5c0 0x7fa2980b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa295e00700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa29806f310 0x7fa2980afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa296800700 1 -- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2980cd5c0 msgr2=0x7fa2980b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa296800700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2980cd5c0 0x7fa2980b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa296800700 1 -- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa29806f310 msgr2=0x7fa2980afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa296800700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa29806f310 0x7fa2980afbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.208+0000 7fa296800700 1 -- 172.21.15.12:0/1742930073 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2880094a0 con 0x7fa29815cfe0 2024-03-29T11:34:31.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.209+0000 7fa296800700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa29815cfe0 0x7fa29806b8a0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7fa29000cb60 tx=0x7fa29000ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:31.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.209+0000 7fa28e000700 1 -- 172.21.15.12:0/1742930073 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29000a9e0 con 0x7fa29815cfe0 2024-03-29T11:34:31.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.209+0000 7fa28e000700 1 -- 172.21.15.12:0/1742930073 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29000ab40 con 0x7fa29815cfe0 2024-03-29T11:34:31.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.209+0000 7fa29ca00700 1 -- 172.21.15.12:0/1742930073 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa298158c80 con 0x7fa29815cfe0 2024-03-29T11:34:31.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.209+0000 7fa28e000700 1 -- 172.21.15.12:0/1742930073 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900070d0 con 0x7fa29815cfe0 2024-03-29T11:34:31.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.209+0000 7fa29ca00700 1 -- 172.21.15.12:0/1742930073 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2980b6160 con 0x7fa29815cfe0 2024-03-29T11:34:31.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.210+0000 7fa29ca00700 1 -- 172.21.15.12:0/1742930073 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2980b6380 con 0x7fa29815cfe0 2024-03-29T11:34:31.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.211+0000 7fa28e000700 1 -- 172.21.15.12:0/1742930073 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 157) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fa290007230 con 0x7fa29815cfe0 2024-03-29T11:34:31.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.213+0000 7fa28e000700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa27806ae00 0x7fa27806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.213+0000 7fa28e000700 1 -- 172.21.15.12:0/1742930073 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fa290088da0 con 0x7fa29815cfe0 2024-03-29T11:34:31.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.213+0000 7fa295e00700 1 -- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa27806ae00 msgr2=0x7fa27806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:31.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.213+0000 7fa295e00700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa27806ae00 0x7fa27806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:31.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.219+0000 7fa28e000700 1 -- 172.21.15.12:0/1742930073 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa290053d90 con 0x7fa29815cfe0 2024-03-29T11:34:31.252 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:31.254+0000 7fa0de43c240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:34:31.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.414+0000 7fa295e00700 1 -- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa27806ae00 msgr2=0x7fa27806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:31.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.414+0000 7fa295e00700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa27806ae00 0x7fa27806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:31.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.435+0000 7fa29ca00700 1 -- 172.21.15.12:0/1742930073 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa29815eda0 con 0x7fa29815cfe0 2024-03-29T11:34:31.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.439+0000 7fa28e000700 1 -- 172.21.15.12:0/1742930073 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v96)=0 v96) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa29815eda0 con 0x7fa29815cfe0 2024-03-29T11:34:31.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.441+0000 7fa29ca00700 1 -- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa27806ae00 msgr2=0x7fa27806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:31.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.441+0000 7fa29ca00700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa27806ae00 0x7fa27806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.441+0000 7fa29ca00700 1 -- 172.21.15.12:0/1742930073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa29815cfe0 msgr2=0x7fa29806b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.441+0000 7fa29ca00700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa29815cfe0 0x7fa29806b8a0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7fa29000cb60 tx=0x7fa29000ce70 comp rx=0 tx=0).stop 2024-03-29T11:34:31.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.442+0000 7fa29ca00700 1 -- 172.21.15.12:0/1742930073 shutdown_connections 2024-03-29T11:34:31.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.442+0000 7fa29ca00700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa27806ae00 0x7fa27806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.442+0000 7fa29ca00700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa29815cfe0 0x7fa29806b8a0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.442+0000 7fa29ca00700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa29806f310 0x7fa2980afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.442+0000 7fa29ca00700 1 --2- 172.21.15.12:0/1742930073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2980cd5c0 0x7fa2980b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.442+0000 7fa29ca00700 1 -- 172.21.15.12:0/1742930073 >> 172.21.15.12:0/1742930073 conn(0x7fa2980d1370 msgr2=0x7fa2980d1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.442+0000 7fa29ca00700 1 -- 172.21.15.12:0/1742930073 shutdown_connections 2024-03-29T11:34:31.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.442+0000 7fa29ca00700 1 -- 172.21.15.12:0/1742930073 wait complete. 2024-03-29T11:34:31.459 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv01 2024-03-29T11:34:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.666+0000 7ff0e3c00700 1 Processor -- start 2024-03-29T11:34:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.667+0000 7ff0e3c00700 1 -- start start 2024-03-29T11:34:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.668+0000 7ff0e3c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff0dc160d70 0x7ff0dc0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.668+0000 7ff0e3c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc06ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.668+0000 7ff0e3c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0dc161150 con 0x7ff0dc15ef70 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.668+0000 7ff0e3c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0dc06b180 con 0x7ff0dc160d70 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.668+0000 7ff0e3c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc0b4cd0 con 0x7ff0dc06a790 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.668+0000 7ff0dbe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff0dc15ef70 0x7ff0dc15e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58112/0 (socket says 172.21.15.12:58112) 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.668+0000 7ff0dbe00700 1 -- 172.21.15.12:0/3214551225 learned_addr learned my addr 172.21.15.12:0/3214551225 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.668+0000 7ff0e1200700 1 --2- 172.21.15.12:0/3214551225 >> v2:172.21.15.159:3300/0 conn(0x7ff0dc160d70 0x7ff0dc0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.668+0000 7ff0e1c00700 1 --2- 172.21.15.12:0/3214551225 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc06ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.669+0000 7ff0db400700 1 -- 172.21.15.12:0/3214551225 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166030723 0 0) 0x7ff0dc161150 con 0x7ff0dc15ef70 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.669+0000 7ff0db400700 1 -- 172.21.15.12:0/3214551225 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c40035b0 con 0x7ff0dc15ef70 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.669+0000 7ff0e1200700 1 -- 172.21.15.12:0/3214551225 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 msgr2=0x7ff0dc06ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.669+0000 7ff0e1200700 1 --2- 172.21.15.12:0/3214551225 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc06ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.669+0000 7ff0e1200700 1 -- 172.21.15.12:0/3214551225 >> v1:172.21.15.12:6789/0 conn(0x7ff0dc15ef70 legacy=0x7ff0dc15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.669+0000 7ff0e1200700 1 -- 172.21.15.12:0/3214551225 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc06dba0 con 0x7ff0dc160d70 2024-03-29T11:34:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.669+0000 7ff0e1c00700 1 --2- 172.21.15.12:0/3214551225 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc06ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:34:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.669+0000 7ff0e1200700 1 --2- 172.21.15.12:0/3214551225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0dc160d70 0x7ff0dc0b3200 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009840 tx=0x7ff0d0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db58acdbc819fd14 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0db400700 1 -- 172.21.15.12:0/3214551225 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0011830 con 0x7ff0dc160d70 2024-03-29T11:34:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0db400700 1 -- 172.21.15.12:0/3214551225 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d0011990 con 0x7ff0dc160d70 2024-03-29T11:34:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0db400700 1 -- 172.21.15.12:0/3214551225 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0011d20 con 0x7ff0dc160d70 2024-03-29T11:34:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0e3c00700 1 -- 172.21.15.12:0/3214551225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0dc160d70 msgr2=0x7ff0dc0b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0e3c00700 1 --2- 172.21.15.12:0/3214551225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0dc160d70 0x7ff0dc0b3200 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009840 tx=0x7ff0d0009b50 comp rx=0 tx=0).stop 2024-03-29T11:34:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0e3c00700 1 -- 172.21.15.12:0/3214551225 shutdown_connections 2024-03-29T11:34:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0e3c00700 1 --2- 172.21.15.12:0/3214551225 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc06ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0e3c00700 1 --2- 172.21.15.12:0/3214551225 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0dc160d70 0x7ff0dc0b3200 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0e3c00700 1 -- 172.21.15.12:0/3214551225 >> 172.21.15.12:0/3214551225 conn(0x7ff0dc0b5720 msgr2=0x7ff0dc0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0e3c00700 1 -- 172.21.15.12:0/3214551225 shutdown_connections 2024-03-29T11:34:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.670+0000 7ff0e3c00700 1 -- 172.21.15.12:0/3214551225 wait complete. 2024-03-29T11:34:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.672+0000 7ff0e3c00700 1 Processor -- start 2024-03-29T11:34:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.672+0000 7ff0e3c00700 1 -- start start 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.673+0000 7ff0e3c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc0b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.673+0000 7ff0e3c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc15ef70 0x7ff0dc1f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.673+0000 7ff0e3c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0dc160d70 0x7ff0dc1f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.673+0000 7ff0e3c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc14b170 con 0x7ff0dc15ef70 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.673+0000 7ff0e3c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc14a630 con 0x7ff0dc160d70 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.673+0000 7ff0e3c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc0b2360 con 0x7ff0dc06a790 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.673+0000 7ff0dbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc15ef70 0x7ff0dc1f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0e1200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc0b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0e1200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc0b1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:58722/0 (socket says 172.21.15.12:58722) 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0dbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc15ef70 0x7ff0dc1f65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46170/0 (socket says 172.21.15.12:46170) 2024-03-29T11:34:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0e1200700 1 -- 172.21.15.12:0/1170232166 learned_addr learned my addr 172.21.15.12:0/1170232166 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0e1c00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0dc160d70 0x7ff0dc1f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0dbe00700 1 -- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 msgr2=0x7ff0dc0b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0dbe00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc0b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0dbe00700 1 -- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0dc160d70 msgr2=0x7ff0dc1f6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0dbe00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0dc160d70 0x7ff0dc1f6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0dbe00700 1 -- 172.21.15.12:0/1170232166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d4009a70 con 0x7ff0dc15ef70 2024-03-29T11:34:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0e1200700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:34:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0dbe00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc15ef70 0x7ff0dc1f65d0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800c460 tx=0x7ff0c800c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.674+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8005d70 con 0x7ff0dc15ef70 2024-03-29T11:34:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.675+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c80070d0 con 0x7ff0dc15ef70 2024-03-29T11:34:31.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.675+0000 7ff0e3c00700 1 -- 172.21.15.12:0/1170232166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d00094a0 con 0x7ff0dc15ef70 2024-03-29T11:34:31.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.675+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8007440 con 0x7ff0dc15ef70 2024-03-29T11:34:31.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.675+0000 7ff0e3c00700 1 -- 172.21.15.12:0/1170232166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0dc1f7420 con 0x7ff0dc15ef70 2024-03-29T11:34:31.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.676+0000 7ff0e3c00700 1 -- 172.21.15.12:0/1170232166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0dc14b320 con 0x7ff0dc15ef70 2024-03-29T11:34:31.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.677+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 157) v1 ==== 89256+0+0 (secure 0 0 0) 0x7ff0c80075a0 con 0x7ff0dc15ef70 2024-03-29T11:34:31.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.678+0000 7ff0d8c00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:31.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.678+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 5781+0+0 (secure 0 0 0) 0x7ff0c8088450 con 0x7ff0dc15ef70 2024-03-29T11:34:31.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.678+0000 7ff0e1200700 1 -- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 msgr2=0x7ff0bc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:31.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.678+0000 7ff0e1200700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:34:31.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.682+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff0dc14b320 con 0x7ff0dc15ef70 2024-03-29T11:34:31.723 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:31.725+0000 7f431e91a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:34:31.842 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:31.845+0000 7fa0de43c240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:34:31.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.879+0000 7ff0e1200700 1 -- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 msgr2=0x7ff0bc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:31.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.879+0000 7ff0e1200700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:34:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:31.955+0000 7ff0e3c00700 1 -- 172.21.15.12:0/1170232166 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0dc0b54a0 con 0x7ff0bc06ae00 2024-03-29T11:34:31.983 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:31.985+0000 7f431e91a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:34:32.110 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:32.112+0000 7fa0de43c240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:34:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:32.280+0000 7ff0e1200700 1 -- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 msgr2=0x7ff0bc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:32.280+0000 7ff0e1200700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-29T11:34:32.446 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:32.448+0000 7f431e91a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:34:32.590 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:32.592+0000 7fa0de43c240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:34:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:33.081+0000 7ff0e1200700 1 -- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 msgr2=0x7ff0bc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:33.081+0000 7ff0e1200700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-29T11:34:33.159 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:33.160+0000 7f431e91a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:34:33.250 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:33.252+0000 7f431e91a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:34:33.336 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:33.338+0000 7fa0de43c240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:34:33.408 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:33.410+0000 7f431e91a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:34:33.433 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:33.435+0000 7fa0de43c240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:34:33.601 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:33.603+0000 7fa0de43c240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:34:33.624 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:33.626+0000 7f431e91a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:34:33.716 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:33.718+0000 7f431e91a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:34:33.815 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:33.817+0000 7f431e91a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:34:33.821 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:33.823+0000 7fa0de43c240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:34:33.914 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:33.917+0000 7f431e91a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:33.918 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:33.921+0000 7fa0de43c240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:34:34.015 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:34.017+0000 7f431e91a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:34:34.022 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:34.025+0000 7fa0de43c240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:34:34.125 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:34.128+0000 7fa0de43c240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:34.228 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:34.231+0000 7fa0de43c240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:34:34.229 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:34.232+0000 7f431e91a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:34:34.454 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:34.456+0000 7fa0de43c240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:34:34.490 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:34.492+0000 7f431e91a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:34:34.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:34.683+0000 7ff0e1200700 1 -- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 msgr2=0x7ff0bc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:34:34.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:34.683+0000 7ff0e1200700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-29T11:34:34.685 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:34.687+0000 7f431e91a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:34:34.719 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:34.722+0000 7fa0de43c240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:34:34.778 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:34.781+0000 7f431e91a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:34.921 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:34.923+0000 7fa0de43c240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:34:35.019 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:35.022+0000 7fa0de43c240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:35.518 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:35.520+0000 7f431e91a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:34:35.730 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:35.733+0000 7f431e91a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:34:35.789 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:35.791+0000 7fa0de43c240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:34:35.939 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:35.941+0000 7f431e91a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:36.016 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:36.018+0000 7fa0de43c240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:34:36.234 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:36.236+0000 7fa0de43c240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:36.422 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:36.424+0000 7f431e91a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:34:36.628 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:36.632+0000 7f431e91a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:36.739 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:36.742+0000 7fa0de43c240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:34:36.958 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:36.961+0000 7fa0de43c240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:37.076 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.079+0000 7f431e91a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:34:37.188 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.191+0000 7f431e91a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:37.204+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 158) v1 ==== 88865+0+0 (secure 0 0 0) 0x7ff0c80538c0 con 0x7ff0dc15ef70 2024-03-29T11:34:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:37.204+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 msgr2=0x7ff0bc06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:37.204+0000 7ff0d8c00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:37.380 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.383+0000 7f428ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:37.380 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.383+0000 7f428ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:37.380 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.383+0000 7f428ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:37.380 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.383+0000 7f428ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:37.380 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.383+0000 7f428ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:37.381 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.384+0000 7f4286600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:37.381 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.384+0000 7f4286600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:37.381 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.384+0000 7f4286600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:37.381 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.384+0000 7f4286600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:37.381 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:37.384+0000 7f4286600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:37.433 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:37.435+0000 7fa0de43c240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:34:37.552 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:34:37.554+0000 7fa0de43c240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:38.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.206+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 159) v1 ==== 88992+0+0 (secure 0 0 0) 0x7ff0c8086880 con 0x7ff0dc15ef70 2024-03-29T11:34:38.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.206+0000 7ff0d8c00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:38.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.206+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0bc01d540 con 0x7ff0bc06ae00 2024-03-29T11:34:38.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.207+0000 7ff0e1200700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.213+0000 7ff0e1200700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009b90 tx=0x7ff0d001c040 comp rx=0 tx=0).ready entity=mgr.12691 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:38.220 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:38.223+0000 7f4283400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:38.220 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:38.223+0000 7f4283400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:38.220 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:38.223+0000 7f4283400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:38.220 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:38.223+0000 7f4283400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:38.220 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:34:38.223+0000 7f4283400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:34:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.266+0000 7ff0d8c00700 1 -- 172.21.15.12:0/1170232166 <== mgr.12691 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff0bc01d540 con 0x7ff0bc06ae00 2024-03-29T11:34:38.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 -- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 msgr2=0x7ff0bc06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009b90 tx=0x7ff0d001c040 comp rx=0 tx=0).stop 2024-03-29T11:34:38.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 -- 172.21.15.12:0/1170232166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc15ef70 msgr2=0x7ff0dc1f65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc15ef70 0x7ff0dc1f65d0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800c460 tx=0x7ff0c800c770 comp rx=0 tx=0).stop 2024-03-29T11:34:38.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 -- 172.21.15.12:0/1170232166 shutdown_connections 2024-03-29T11:34:38.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7ff0bc06ae00 0x7ff0bc06d240 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0dc160d70 0x7ff0dc1f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0dc06a790 0x7ff0dc0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 --2- 172.21.15.12:0/1170232166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc15ef70 0x7ff0dc1f65d0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 -- 172.21.15.12:0/1170232166 >> 172.21.15.12:0/1170232166 conn(0x7ff0dc0b5720 msgr2=0x7ff0dc15b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:38.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 -- 172.21.15.12:0/1170232166 shutdown_connections 2024-03-29T11:34:38.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.269+0000 7ff0e3c00700 1 -- 172.21.15.12:0/1170232166 wait complete. 2024-03-29T11:34:38.288 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv01 . 1m 2024-03-29T11:34:38.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.496+0000 7f6bdce00700 1 Processor -- start 2024-03-29T11:34:38.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.496+0000 7f6bdce00700 1 -- start start 2024-03-29T11:34:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.497+0000 7f6bdce00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6bd815d930 0x7f6bd80d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.497+0000 7f6bdce00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6bd80d22a0 0x7f6bd80ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.497+0000 7f6bdce00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd815dd10 con 0x7f6bd8162030 2024-03-29T11:34:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.497+0000 7f6bdce00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd815ec40 con 0x7f6bd815d930 2024-03-29T11:34:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.498+0000 7f6bdce00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd815dfc0 con 0x7f6bd80d22a0 2024-03-29T11:34:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.498+0000 7f6bd5e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6bd815d930 0x7f6bd80d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.498+0000 7f6bd5400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6bd8162030 0x7f6bd80d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58114/0 (socket says 172.21.15.12:58114) 2024-03-29T11:34:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.498+0000 7f6bd5400700 1 -- 172.21.15.12:0/4101406521 learned_addr learned my addr 172.21.15.12:0/4101406521 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:34:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.498+0000 7f6bd6800700 1 --2- 172.21.15.12:0/4101406521 >> v2:172.21.15.159:3300/0 conn(0x7f6bd80d22a0 0x7f6bd80ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.498+0000 7f6bd5e00700 1 -- 172.21.15.12:0/4101406521 >> v2:172.21.15.159:3300/0 conn(0x7f6bd80d22a0 msgr2=0x7f6bd80ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.498+0000 7f6bd5e00700 1 --2- 172.21.15.12:0/4101406521 >> v2:172.21.15.159:3300/0 conn(0x7f6bd80d22a0 0x7f6bd80ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.498+0000 7f6bd5e00700 1 -- 172.21.15.12:0/4101406521 >> v1:172.21.15.12:6789/0 conn(0x7f6bd8162030 legacy=0x7f6bd80d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.498+0000 7f6bd5e00700 1 -- 172.21.15.12:0/4101406521 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd815e2a0 con 0x7f6bd815d930 2024-03-29T11:34:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.498+0000 7f6bd4a00700 1 -- 172.21.15.12:0/4101406521 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055134091 0 0) 0x7f6bd815dd10 con 0x7f6bd8162030 2024-03-29T11:34:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.499+0000 7f6bd5e00700 1 --2- 172.21.15.12:0/4101406521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 0x7f6bd80d3ed0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80d2800 tx=0x7f6bc8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d8c37a44b1419c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.499+0000 7f6bd4a00700 1 -- 172.21.15.12:0/4101406521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8011830 con 0x7f6bd815d930 2024-03-29T11:34:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.499+0000 7f6bd4a00700 1 -- 172.21.15.12:0/4101406521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc8011990 con 0x7f6bd815d930 2024-03-29T11:34:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.499+0000 7f6bd4a00700 1 -- 172.21.15.12:0/4101406521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8011d20 con 0x7f6bd815d930 2024-03-29T11:34:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.499+0000 7f6bdce00700 1 -- 172.21.15.12:0/4101406521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 msgr2=0x7f6bd80d3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.499+0000 7f6bdce00700 1 --2- 172.21.15.12:0/4101406521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 0x7f6bd80d3ed0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80d2800 tx=0x7f6bc8009b20 comp rx=0 tx=0).stop 2024-03-29T11:34:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.500+0000 7f6bdce00700 1 -- 172.21.15.12:0/4101406521 shutdown_connections 2024-03-29T11:34:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.500+0000 7f6bdce00700 1 --2- 172.21.15.12:0/4101406521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 0x7f6bd80d3ed0 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.500+0000 7f6bdce00700 1 --2- 172.21.15.12:0/4101406521 >> v2:172.21.15.159:3300/0 conn(0x7f6bd80d22a0 0x7f6bd80ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.500+0000 7f6bdce00700 1 -- 172.21.15.12:0/4101406521 >> 172.21.15.12:0/4101406521 conn(0x7f6bd808aff0 msgr2=0x7f6bd80d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.500+0000 7f6bdce00700 1 -- 172.21.15.12:0/4101406521 shutdown_connections 2024-03-29T11:34:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.500+0000 7f6bdce00700 1 -- 172.21.15.12:0/4101406521 wait complete. 2024-03-29T11:34:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.502+0000 7f6bdce00700 1 Processor -- start 2024-03-29T11:34:38.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.502+0000 7f6bdce00700 1 -- start start 2024-03-29T11:34:38.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.503+0000 7f6bdce00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd80d22a0 0x7f6bd81f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:38.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.503+0000 7f6bdce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 0x7f6bd80cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:38.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.503+0000 7f6bdce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd8162030 0x7f6bd81f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:38.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.503+0000 7f6bdce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd81f4100 con 0x7f6bd815d930 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.503+0000 7f6bdce00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd80cc620 con 0x7f6bd80d22a0 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.503+0000 7f6bdce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd81f3580 con 0x7f6bd8162030 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 0x7f6bd80cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 0x7f6bd80cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46186/0 (socket says 172.21.15.12:46186) 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd5400700 1 -- 172.21.15.12:0/3191588796 learned_addr learned my addr 172.21.15.12:0/3191588796 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd6800700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd8162030 0x7f6bd81f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd5e00700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd80d22a0 0x7f6bd81f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd5400700 1 -- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd8162030 msgr2=0x7f6bd81f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd5400700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd8162030 0x7f6bd81f29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd5400700 1 -- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd80d22a0 msgr2=0x7f6bd81f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd5400700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd80d22a0 0x7f6bd81f4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd5400700 1 -- 172.21.15.12:0/3191588796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc80094a0 con 0x7f6bd815d930 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.504+0000 7f6bd5400700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 0x7f6bd80cc0e0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400c550 tx=0x7f6bc400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.505+0000 7f6bce000700 1 -- 172.21.15.12:0/3191588796 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4004c80 con 0x7f6bd815d930 2024-03-29T11:34:38.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.505+0000 7f6bce000700 1 -- 172.21.15.12:0/3191588796 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc4004e00 con 0x7f6bd815d930 2024-03-29T11:34:38.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.505+0000 7f6bce000700 1 -- 172.21.15.12:0/3191588796 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4007310 con 0x7f6bd815d930 2024-03-29T11:34:38.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.505+0000 7f6bdce00700 1 -- 172.21.15.12:0/3191588796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd81f3860 con 0x7f6bd815d930 2024-03-29T11:34:38.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.505+0000 7f6bdce00700 1 -- 172.21.15.12:0/3191588796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd81f3e20 con 0x7f6bd815d930 2024-03-29T11:34:38.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.506+0000 7f6bdce00700 1 -- 172.21.15.12:0/3191588796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd8067720 con 0x7f6bd815d930 2024-03-29T11:34:38.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.507+0000 7f6bce000700 1 -- 172.21.15.12:0/3191588796 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 159) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f6bc4007470 con 0x7f6bd815d930 2024-03-29T11:34:38.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.508+0000 7f6bce000700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6bb806ab30 0x7f6bb806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:34:38.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.508+0000 7f6bce000700 1 -- 172.21.15.12:0/3191588796 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f6bc4088d50 con 0x7f6bd815d930 2024-03-29T11:34:38.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.508+0000 7f6bd5e00700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6bb806ab30 0x7f6bb806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:34:38.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.509+0000 7f6bd5e00700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6bb806ab30 0x7f6bb806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6bc80090b0 tx=0x7f6bc8008030 comp rx=0 tx=0).ready entity=mgr.12691 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:34:38.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.512+0000 7f6bce000700 1 -- 172.21.15.12:0/3191588796 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6bc4054000 con 0x7f6bd815d930 2024-03-29T11:34:38.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.755+0000 7f6bdce00700 1 -- 172.21.15.12:0/3191588796 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv01", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bd80d4410 con 0x7f6bb806ab30 2024-03-29T11:34:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.847+0000 7f6bce000700 1 -- 172.21.15.12:0/3191588796 <== mgr.12691 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+85 (secure 0 0 0) 0x7f6bc8011630 con 0x7f6bb806ab30 2024-03-29T11:34:38.845 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path /volumes/_nogroup/sv01/9c84ed60-22b7-4963-978c-2139b1562842/.. 2024-03-29T11:34:38.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 -- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6bb806ab30 msgr2=0x7f6bb806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6bb806ab30 0x7f6bb806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6bc80090b0 tx=0x7f6bc8008030 comp rx=0 tx=0).stop 2024-03-29T11:34:38.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 -- 172.21.15.12:0/3191588796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 msgr2=0x7f6bd80cc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 0x7f6bd80cc0e0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400c550 tx=0x7f6bc400c860 comp rx=0 tx=0).stop 2024-03-29T11:34:38.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 -- 172.21.15.12:0/3191588796 shutdown_connections 2024-03-29T11:34:38.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6bb806ab30 0x7f6bb806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd8162030 0x7f6bd81f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd80d22a0 0x7f6bd81f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 --2- 172.21.15.12:0/3191588796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bd815d930 0x7f6bd80cc0e0 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:34:38.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 -- 172.21.15.12:0/3191588796 >> 172.21.15.12:0/3191588796 conn(0x7f6bd808aff0 msgr2=0x7f6bd80cebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:38.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 -- 172.21.15.12:0/3191588796 shutdown_connections 2024-03-29T11:34:38.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:34:38.850+0000 7f6bdce00700 1 -- 172.21.15.12:0/3191588796 wait complete. 2024-03-29T11:34:38.866 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /volumes/_nogroup/sv01/9c84ed60-22b7-4963-978c-2139b1562842/.. 2024-03-29T11:34:51.861 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:34:51.866 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:35:00.860 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:00.862+0000 7f4270800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:35:00.861 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:00.862+0000 7f4270800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:35:00.861 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:00.862+0000 7f4270800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:35:00.861 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:00.862+0000 7f4270800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:35:00.861 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:00.862+0000 7f4270800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:35:21.979 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:35:21.983 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:35:52.147 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:35:52.152 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:35:53.867 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-29T11:35:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.079+0000 7f851b000700 1 Processor -- start 2024-03-29T11:35:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.079+0000 7f851b000700 1 -- start start 2024-03-29T11:35:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.080+0000 7f851b000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f85140d21f0 0x7f85140ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.080+0000 7f851b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 0x7f8514161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.080+0000 7f851b000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140ce270 con 0x7f85141611b0 2024-03-29T11:35:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.080+0000 7f851b000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f851415b600 con 0x7f85140d21f0 2024-03-29T11:35:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.080+0000 7f851b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85140d2760 con 0x7f85140cec40 2024-03-29T11:35:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.081+0000 7f8513400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f85140d21f0 0x7f85140ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.081+0000 7f8513400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f85140d21f0 0x7f85140ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52926/0 (socket says 172.21.15.12:52926) 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.081+0000 7f8513400700 1 -- 172.21.15.12:0/2197294213 learned_addr learned my addr 172.21.15.12:0/2197294213 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.081+0000 7f8519000700 1 --2- 172.21.15.12:0/2197294213 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 0x7f8514161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.081+0000 7f8513400700 1 -- 172.21.15.12:0/2197294213 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 msgr2=0x7f8514161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.081+0000 7f8513400700 1 --2- 172.21.15.12:0/2197294213 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 0x7f8514161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.081+0000 7f8513400700 1 -- 172.21.15.12:0/2197294213 >> v1:172.21.15.159:6789/0 conn(0x7f85141611b0 legacy=0x7f851415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.081+0000 7f8513400700 1 -- 172.21.15.12:0/2197294213 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140d2a40 con 0x7f85140d21f0 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.081+0000 7f8513400700 1 --2- 172.21.15.12:0/2197294213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 0x7f85140ce700 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f85000098f0 tx=0x7f8500009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20dbe81687553f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f8512a00700 1 -- 172.21.15.12:0/2197294213 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850000b8a0 con 0x7f85140d21f0 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f8512a00700 1 -- 172.21.15.12:0/2197294213 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850000ba00 con 0x7f85140d21f0 2024-03-29T11:35:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f8512a00700 1 -- 172.21.15.12:0/2197294213 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850000bd90 con 0x7f85140d21f0 2024-03-29T11:35:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f851b000700 1 -- 172.21.15.12:0/2197294213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 msgr2=0x7f85140ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f851b000700 1 --2- 172.21.15.12:0/2197294213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 0x7f85140ce700 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f85000098f0 tx=0x7f8500009c00 comp rx=0 tx=0).stop 2024-03-29T11:35:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f851b000700 1 -- 172.21.15.12:0/2197294213 shutdown_connections 2024-03-29T11:35:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f851b000700 1 --2- 172.21.15.12:0/2197294213 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 0x7f8514161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f851b000700 1 --2- 172.21.15.12:0/2197294213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 0x7f85140ce700 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f851b000700 1 -- 172.21.15.12:0/2197294213 >> 172.21.15.12:0/2197294213 conn(0x7f851408aff0 msgr2=0x7f85140d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f851b000700 1 -- 172.21.15.12:0/2197294213 shutdown_connections 2024-03-29T11:35:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.082+0000 7f851b000700 1 -- 172.21.15.12:0/2197294213 wait complete. 2024-03-29T11:35:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.084+0000 7f851b000700 1 Processor -- start 2024-03-29T11:35:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.084+0000 7f851b000700 1 -- start start 2024-03-29T11:35:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.085+0000 7f851b000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 0x7f85140b8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.085+0000 7f851b000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 0x7f851415a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.085+0000 7f851b000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85141611b0 0x7f851409ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.085+0000 7f851b000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851409fcf0 con 0x7f85140d21f0 2024-03-29T11:35:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.085+0000 7f851b000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851409d710 con 0x7f85141611b0 2024-03-29T11:35:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.085+0000 7f851b000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851415ade0 con 0x7f85140cec40 2024-03-29T11:35:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.085+0000 7f8513400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 0x7f851415a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.085+0000 7f8513400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 0x7f851415a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52942/0 (socket says 172.21.15.12:52942) 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.085+0000 7f8513400700 1 -- 172.21.15.12:0/413760963 learned_addr learned my addr 172.21.15.12:0/413760963 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.085+0000 7f8513e00700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 0x7f85140b8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f8519000700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85141611b0 0x7f851409ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f8513400700 1 -- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 msgr2=0x7f85140b8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f8513400700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 0x7f85140b8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f8513400700 1 -- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85141611b0 msgr2=0x7f851409ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f8513400700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85141611b0 0x7f851409ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f8513400700 1 -- 172.21.15.12:0/413760963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8500009550 con 0x7f85140d21f0 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f8513e00700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 0x7f85140b8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f8513400700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 0x7f851415a8a0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f8500009c40 tx=0x7f8500004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f850be00700 1 -- 172.21.15.12:0/413760963 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85000044d0 con 0x7f85140d21f0 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f850be00700 1 -- 172.21.15.12:0/413760963 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850001f070 con 0x7f85140d21f0 2024-03-29T11:35:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f851b000700 1 -- 172.21.15.12:0/413760963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851409f730 con 0x7f85140d21f0 2024-03-29T11:35:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.086+0000 7f850be00700 1 -- 172.21.15.12:0/413760963 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85000038e0 con 0x7f85140d21f0 2024-03-29T11:35:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.087+0000 7f851b000700 1 -- 172.21.15.12:0/413760963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851409d3a0 con 0x7f85140d21f0 2024-03-29T11:35:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.088+0000 7f851b000700 1 -- 172.21.15.12:0/413760963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851409d530 con 0x7f85140d21f0 2024-03-29T11:35:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.089+0000 7f850be00700 1 -- 172.21.15.12:0/413760963 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 162) v1 ==== 89421+0+0 (secure 0 0 0) 0x7f8500003a40 con 0x7f85140d21f0 2024-03-29T11:35:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.089+0000 7f850be00700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f84f406afb0 0x7f84f406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.090+0000 7f850be00700 1 -- 172.21.15.12:0/413760963 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7f8500021070 con 0x7f85140d21f0 2024-03-29T11:35:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.090+0000 7f8513e00700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f84f406afb0 0x7f84f406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.091+0000 7f8513e00700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f84f406afb0 0x7f84f406d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f85040016e0 tx=0x7f8504009040 comp rx=0 tx=0).ready entity=mgr.12691 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:54.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.094+0000 7f850be00700 1 -- 172.21.15.12:0/413760963 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85000be580 con 0x7f85140d21f0 2024-03-29T11:35:54.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:54.369+0000 7f851b000700 1 -- 172.21.15.12:0/413760963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f851409d880 con 0x7f85140d21f0 2024-03-29T11:35:55.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.367+0000 7f850be00700 1 -- 172.21.15.12:0/413760963 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f8500003cf0 con 0x7f85140d21f0 2024-03-29T11:35:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.369+0000 7f850be00700 1 -- 172.21.15.12:0/413760963 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v163) v1 ==== 93+0+0 (secure 0 0 0) 0x7f850005d490 con 0x7f85140d21f0 2024-03-29T11:35:55.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 -- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f84f406afb0 msgr2=0x7f84f406d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f84f406afb0 0x7f84f406d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f85040016e0 tx=0x7f8504009040 comp rx=0 tx=0).stop 2024-03-29T11:35:55.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 -- 172.21.15.12:0/413760963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 msgr2=0x7f851415a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 0x7f851415a8a0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f8500009c40 tx=0x7f8500004750 comp rx=0 tx=0).stop 2024-03-29T11:35:55.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 -- 172.21.15.12:0/413760963 shutdown_connections 2024-03-29T11:35:55.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f84f406afb0 0x7f84f406d470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:55.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85141611b0 0x7f851409ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:55.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85140cec40 0x7f85140b8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:55.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 --2- 172.21.15.12:0/413760963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85140d21f0 0x7f851415a8a0 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:55.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 -- 172.21.15.12:0/413760963 >> 172.21.15.12:0/413760963 conn(0x7f851408aff0 msgr2=0x7f851406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:55.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 -- 172.21.15.12:0/413760963 shutdown_connections 2024-03-29T11:35:55.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:55.372+0000 7f851b000700 1 -- 172.21.15.12:0/413760963 wait complete. 2024-03-29T11:35:55.391 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-29T11:35:55.392 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:35:55.489 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-29T11:35:55.602 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:55.602+0000 7f3f02537240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:35:55.604 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:55.606+0000 7fc7324ff240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:35:55.624 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-29T11:35:55.624 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:35:55.624 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:35:55.671 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:35:55.671 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:35:55.790 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:55.793 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:55.794+0000 7fc7324ff240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:35:55.811 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:55.811+0000 7f3f02537240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:35:56.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.002+0000 7f3987a00700 1 Processor -- start 2024-03-29T11:35:56.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.002+0000 7f3987a00700 1 -- start start 2024-03-29T11:35:56.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.003+0000 7f3987a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f398015db40 0x7f39800d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.003+0000 7f3987a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39800d1330 con 0x7f398015d760 2024-03-29T11:35:56.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.003+0000 7f3987a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39800d1610 con 0x7f39800cf3c0 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.003+0000 7f3987a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800d1fb0 con 0x7f398015db40 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.004+0000 7f3985000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f398015d760 0x7f39800cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55164/0 (socket says 172.21.15.12:55164) 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.004+0000 7f3985000700 1 -- 172.21.15.12:0/3108820589 learned_addr learned my addr 172.21.15.12:0/3108820589 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.004+0000 7f3985a00700 1 --2- 172.21.15.12:0/3108820589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f398015db40 0x7f39800d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.004+0000 7f397f400700 1 -- 172.21.15.12:0/3108820589 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790722927 0 0) 0x7f39800d1330 con 0x7f398015d760 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.004+0000 7f397f400700 1 -- 172.21.15.12:0/3108820589 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39680035b0 con 0x7f398015d760 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.004+0000 7f397f400700 1 -- 172.21.15.12:0/3108820589 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781019858 0 0) 0x7f39800d1610 con 0x7f39800cf3c0 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.004+0000 7f397f400700 1 -- 172.21.15.12:0/3108820589 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39800d1330 con 0x7f39800cf3c0 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.004+0000 7f3985a00700 1 -- 172.21.15.12:0/3108820589 >> v1:172.21.15.159:6789/0 conn(0x7f39800cf3c0 legacy=0x7f39800cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.004+0000 7f3985a00700 1 -- 172.21.15.12:0/3108820589 >> v1:172.21.15.12:6789/0 conn(0x7f398015d760 legacy=0x7f39800cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.004+0000 7f3985a00700 1 -- 172.21.15.12:0/3108820589 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800d18f0 con 0x7f398015db40 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.005+0000 7f3985a00700 1 --2- 172.21.15.12:0/3108820589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f398015db40 0x7f39800d23d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3978009dc0 tx=0x7f397800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dea6413bef740437 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:56.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.005+0000 7f397f400700 1 -- 172.21.15.12:0/3108820589 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397800e070 con 0x7f398015db40 2024-03-29T11:35:56.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.005+0000 7f397f400700 1 -- 172.21.15.12:0/3108820589 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397800bb20 con 0x7f398015db40 2024-03-29T11:35:56.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.005+0000 7f397f400700 1 -- 172.21.15.12:0/3108820589 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978012430 con 0x7f398015db40 2024-03-29T11:35:56.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.005+0000 7f3987a00700 1 -- 172.21.15.12:0/3108820589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f398015db40 msgr2=0x7f39800d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.005+0000 7f3987a00700 1 --2- 172.21.15.12:0/3108820589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f398015db40 0x7f39800d23d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3978009dc0 tx=0x7f397800b330 comp rx=0 tx=0).stop 2024-03-29T11:35:56.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.005+0000 7f3987a00700 1 -- 172.21.15.12:0/3108820589 shutdown_connections 2024-03-29T11:35:56.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.005+0000 7f3987a00700 1 --2- 172.21.15.12:0/3108820589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f398015db40 0x7f39800d23d0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.005+0000 7f3987a00700 1 -- 172.21.15.12:0/3108820589 >> 172.21.15.12:0/3108820589 conn(0x7f398008aff0 msgr2=0x7f39800d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:56.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.006+0000 7f3987a00700 1 -- 172.21.15.12:0/3108820589 shutdown_connections 2024-03-29T11:35:56.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.006+0000 7f3987a00700 1 -- 172.21.15.12:0/3108820589 wait complete. 2024-03-29T11:35:56.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.007+0000 7f3987a00700 1 Processor -- start 2024-03-29T11:35:56.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.008+0000 7f3987a00700 1 -- start start 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.008+0000 7f3987a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39800cf3c0 0x7f39800cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.008+0000 7f3987a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f398015d760 0x7f3980154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.008+0000 7f3987a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f398015db40 0x7f3980158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.008+0000 7f3987a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800cd560 con 0x7f398015d760 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.008+0000 7f3987a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800b66d0 con 0x7f398015db40 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.008+0000 7f3987a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800b63a0 con 0x7f39800cf3c0 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f397fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f398015d760 0x7f3980154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f3985000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39800cf3c0 0x7f39800cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f397fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f398015d760 0x7f3980154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52966/0 (socket says 172.21.15.12:52966) 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f397fe00700 1 -- 172.21.15.12:0/3749653083 learned_addr learned my addr 172.21.15.12:0/3749653083 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:56.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f3985a00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f398015db40 0x7f3980158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f397fe00700 1 -- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39800cf3c0 msgr2=0x7f39800cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f397fe00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39800cf3c0 0x7f39800cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f397fe00700 1 -- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f398015db40 msgr2=0x7f3980158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f397fe00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f398015db40 0x7f3980158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f397fe00700 1 -- 172.21.15.12:0/3749653083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3978009a70 con 0x7f398015d760 2024-03-29T11:35:56.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.009+0000 7f3985000700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39800cf3c0 0x7f39800cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:35:56.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.010+0000 7f397fe00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f398015d760 0x7f3980154ad0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f396c00c500 tx=0x7f396c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:56.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.010+0000 7f397cc00700 1 -- 172.21.15.12:0/3749653083 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c0052d0 con 0x7f398015d760 2024-03-29T11:35:56.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.010+0000 7f397cc00700 1 -- 172.21.15.12:0/3749653083 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c005430 con 0x7f398015d760 2024-03-29T11:35:56.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.010+0000 7f397cc00700 1 -- 172.21.15.12:0/3749653083 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c0038c0 con 0x7f398015d760 2024-03-29T11:35:56.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.010+0000 7f3987a00700 1 -- 172.21.15.12:0/3749653083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3980155010 con 0x7f398015d760 2024-03-29T11:35:56.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.010+0000 7f3987a00700 1 -- 172.21.15.12:0/3749653083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39800cd280 con 0x7f398015d760 2024-03-29T11:35:56.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.011+0000 7f3987a00700 1 -- 172.21.15.12:0/3749653083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39800b6200 con 0x7f398015d760 2024-03-29T11:35:56.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.012+0000 7f397cc00700 1 -- 172.21.15.12:0/3749653083 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f396c003a20 con 0x7f398015d760 2024-03-29T11:35:56.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.013+0000 7f397cc00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f396006af60 0x7f396006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.013+0000 7f397cc00700 1 -- 172.21.15.12:0/3749653083 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7f396c088cf0 con 0x7f398015d760 2024-03-29T11:35:56.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.013+0000 7f3985000700 1 -- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f396006af60 msgr2=0x7f396006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:56.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.013+0000 7f3985000700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f396006af60 0x7f396006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:35:56.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.018+0000 7f397cc00700 1 -- 172.21.15.12:0/3749653083 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39800b6200 con 0x7f398015d760 2024-03-29T11:35:56.118 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:56.119+0000 7fc7324ff240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:35:56.159 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:56.160+0000 7f3f02537240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:35:56.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.214+0000 7f3985000700 1 -- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f396006af60 msgr2=0x7f396006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:56.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.214+0000 7f3985000700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f396006af60 0x7f396006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:35:56.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.233+0000 7f3987a00700 1 -- 172.21.15.12:0/3749653083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39800b6200 con 0x7f398015d760 2024-03-29T11:35:56.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.234+0000 7f397cc00700 1 -- 172.21.15.12:0/3749653083 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v286) v1 ==== 74+0+16721 (secure 0 0 0) 0x7f39800b6200 con 0x7f398015d760 2024-03-29T11:35:56.234 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:35:56.234 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":286,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:35:38.461906+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:34:18.360241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-03-29T11:34:18.903400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"283","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.d","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2115167177":"2024-03-30T11:34:37.193903+0000","172.21.15.159:6833/62646":"2024-03-30T11:34:37.193903+0000","172.21.15.159:6832/62646":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/411770029":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/3938940279":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/2298284405":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/1795874409":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/547605341":"2024-03-30T11:34:37.193903+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 -- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f396006af60 msgr2=0x7f396006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f396006af60 0x7f396006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 -- 172.21.15.12:0/3749653083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f398015d760 msgr2=0x7f3980154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f398015d760 0x7f3980154ad0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f396c00c500 tx=0x7f396c00c810 comp rx=0 tx=0).stop 2024-03-29T11:35:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 -- 172.21.15.12:0/3749653083 shutdown_connections 2024-03-29T11:35:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f396006af60 0x7f396006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f398015db40 0x7f3980158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39800cf3c0 0x7f39800cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 --2- 172.21.15.12:0/3749653083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f398015d760 0x7f3980154ad0 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 -- 172.21.15.12:0/3749653083 >> 172.21.15.12:0/3749653083 conn(0x7f398008aff0 msgr2=0x7f39800b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 -- 172.21.15.12:0/3749653083 shutdown_connections 2024-03-29T11:35:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.237+0000 7f3987a00700 1 -- 172.21.15.12:0/3749653083 wait complete. 2024-03-29T11:35:56.256 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:34:18.360241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '282', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:34:18.903400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '283', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:35:56.257 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:35:56.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.500+0000 7f5e17400700 1 Processor -- start 2024-03-29T11:35:56.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.501+0000 7f5e17400700 1 -- start start 2024-03-29T11:35:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.502+0000 7f5e17400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100cee10 0x7f5e10161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.502+0000 7f5e17400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e100ce270 con 0x7f5e101611b0 2024-03-29T11:35:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.502+0000 7f5e17400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1015b600 con 0x7f5e100d21f0 2024-03-29T11:35:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.502+0000 7f5e17400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100d2760 con 0x7f5e100cee10 2024-03-29T11:35:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.502+0000 7f5e14a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5e101611b0 0x7f5e1015ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55166/0 (socket says 172.21.15.12:55166) 2024-03-29T11:35:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.502+0000 7f5e14a00700 1 -- 172.21.15.12:0/1571820901 learned_addr learned my addr 172.21.15.12:0/1571820901 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:35:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.502+0000 7f5e15400700 1 --2- 172.21.15.12:0/1571820901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100cee10 0x7f5e10161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.502+0000 7f5e0f400700 1 -- 172.21.15.12:0/1571820901 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387272821 0 0) 0x7f5e100ce270 con 0x7f5e101611b0 2024-03-29T11:35:56.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.503+0000 7f5e0f400700 1 -- 172.21.15.12:0/1571820901 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df80035b0 con 0x7f5e101611b0 2024-03-29T11:35:56.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.503+0000 7f5e0f400700 1 -- 172.21.15.12:0/1571820901 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833467089 0 0) 0x7f5e1015b600 con 0x7f5e100d21f0 2024-03-29T11:35:56.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.503+0000 7f5e0f400700 1 -- 172.21.15.12:0/1571820901 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e100ce270 con 0x7f5e100d21f0 2024-03-29T11:35:56.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.503+0000 7f5e15400700 1 -- 172.21.15.12:0/1571820901 >> v1:172.21.15.159:6789/0 conn(0x7f5e100d21f0 legacy=0x7f5e100ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.503+0000 7f5e15400700 1 -- 172.21.15.12:0/1571820901 >> v1:172.21.15.12:6789/0 conn(0x7f5e101611b0 legacy=0x7f5e1015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.503+0000 7f5e15400700 1 -- 172.21.15.12:0/1571820901 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e100d2a40 con 0x7f5e100cee10 2024-03-29T11:35:56.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.503+0000 7f5e15400700 1 --2- 172.21.15.12:0/1571820901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100cee10 0x7f5e10161590 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009dc0 tx=0x7f5e0800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d97cd144bd2e3c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:56.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.503+0000 7f5e0f400700 1 -- 172.21.15.12:0/1571820901 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800e070 con 0x7f5e100cee10 2024-03-29T11:35:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.503+0000 7f5e0f400700 1 -- 172.21.15.12:0/1571820901 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0800bb20 con 0x7f5e100cee10 2024-03-29T11:35:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.503+0000 7f5e0f400700 1 -- 172.21.15.12:0/1571820901 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08012430 con 0x7f5e100cee10 2024-03-29T11:35:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.504+0000 7f5e17400700 1 -- 172.21.15.12:0/1571820901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100cee10 msgr2=0x7f5e10161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.504+0000 7f5e17400700 1 --2- 172.21.15.12:0/1571820901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100cee10 0x7f5e10161590 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009dc0 tx=0x7f5e0800b330 comp rx=0 tx=0).stop 2024-03-29T11:35:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.504+0000 7f5e17400700 1 -- 172.21.15.12:0/1571820901 shutdown_connections 2024-03-29T11:35:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.504+0000 7f5e17400700 1 --2- 172.21.15.12:0/1571820901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100cee10 0x7f5e10161590 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.504+0000 7f5e17400700 1 -- 172.21.15.12:0/1571820901 >> 172.21.15.12:0/1571820901 conn(0x7f5e1008aff0 msgr2=0x7f5e100d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.504+0000 7f5e17400700 1 -- 172.21.15.12:0/1571820901 shutdown_connections 2024-03-29T11:35:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.504+0000 7f5e17400700 1 -- 172.21.15.12:0/1571820901 wait complete. 2024-03-29T11:35:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.506+0000 7f5e17400700 1 Processor -- start 2024-03-29T11:35:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.506+0000 7f5e17400700 1 -- start start 2024-03-29T11:35:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.507+0000 7f5e17400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e100cee10 0x7f5e101f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.507+0000 7f5e17400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100d21f0 0x7f5e100b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.507+0000 7f5e17400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e101611b0 0x7f5e1006c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.507+0000 7f5e17400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100b74b0 con 0x7f5e100cee10 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.507+0000 7f5e17400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e101f2c30 con 0x7f5e101611b0 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.507+0000 7f5e17400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100b6c10 con 0x7f5e100d21f0 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.507+0000 7f5e14a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e100cee10 0x7f5e101f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.507+0000 7f5e14a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e100cee10 0x7f5e101f26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52978/0 (socket says 172.21.15.12:52978) 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.508+0000 7f5e14a00700 1 -- 172.21.15.12:0/2193297817 learned_addr learned my addr 172.21.15.12:0/2193297817 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.507+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100d21f0 0x7f5e100b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.508+0000 7f5e15400700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e101611b0 0x7f5e1006c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.508+0000 7f5e0fe00700 1 -- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e101611b0 msgr2=0x7f5e1006c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.508+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e101611b0 0x7f5e1006c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.508+0000 7f5e0fe00700 1 -- 172.21.15.12:0/2193297817 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e100cee10 msgr2=0x7f5e101f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.508+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e100cee10 0x7f5e101f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.508+0000 7f5e0fe00700 1 -- 172.21.15.12:0/2193297817 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e08009a70 con 0x7f5e100d21f0 2024-03-29T11:35:56.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.508+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100d21f0 0x7f5e100b6150 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00c480 tx=0x7f5dfc00c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:56.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.509+0000 7f5e0cc00700 1 -- 172.21.15.12:0/2193297817 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc0052d0 con 0x7f5e100d21f0 2024-03-29T11:35:56.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.509+0000 7f5e0cc00700 1 -- 172.21.15.12:0/2193297817 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc005430 con 0x7f5e100d21f0 2024-03-29T11:35:56.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.509+0000 7f5e0cc00700 1 -- 172.21.15.12:0/2193297817 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc0038a0 con 0x7f5e100d21f0 2024-03-29T11:35:56.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.509+0000 7f5e17400700 1 -- 172.21.15.12:0/2193297817 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e100b6ef0 con 0x7f5e100d21f0 2024-03-29T11:35:56.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.509+0000 7f5e17400700 1 -- 172.21.15.12:0/2193297817 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e100b8b90 con 0x7f5e100d21f0 2024-03-29T11:35:56.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.510+0000 7f5e17400700 1 -- 172.21.15.12:0/2193297817 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e100ca600 con 0x7f5e100d21f0 2024-03-29T11:35:56.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.511+0000 7f5e0cc00700 1 -- 172.21.15.12:0/2193297817 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f5dfc003a00 con 0x7f5e100d21f0 2024-03-29T11:35:56.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.512+0000 7f5e0cc00700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5df006af60 0x7f5df006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.512+0000 7f5e0cc00700 1 -- 172.21.15.12:0/2193297817 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7f5dfc088c90 con 0x7f5e100d21f0 2024-03-29T11:35:56.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.512+0000 7f5e14a00700 1 -- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5df006af60 msgr2=0x7f5df006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:56.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.512+0000 7f5e14a00700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5df006af60 0x7f5df006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:35:56.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.516+0000 7f5e0cc00700 1 -- 172.21.15.12:0/2193297817 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e100ca600 con 0x7f5e100d21f0 2024-03-29T11:35:56.653 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:56.654+0000 7fc7324ff240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:35:56.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.713+0000 7f5e14a00700 1 -- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5df006af60 msgr2=0x7f5df006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:56.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.713+0000 7f5e14a00700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5df006af60 0x7f5df006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:35:56.722 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:56.723+0000 7f3f02537240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:35:56.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.731+0000 7f5e17400700 1 -- 172.21.15.12:0/2193297817 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e100ca600 con 0x7f5e100d21f0 2024-03-29T11:35:56.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.732+0000 7f5e0cc00700 1 -- 172.21.15.12:0/2193297817 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v286) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5e100ca600 con 0x7f5e100d21f0 2024-03-29T11:35:56.731 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:35:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 -- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5df006af60 msgr2=0x7f5df006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5df006af60 0x7f5df006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 -- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100d21f0 msgr2=0x7f5e100b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100d21f0 0x7f5e100b6150 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00c480 tx=0x7f5dfc00c790 comp rx=0 tx=0).stop 2024-03-29T11:35:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 -- 172.21.15.12:0/2193297817 shutdown_connections 2024-03-29T11:35:56.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5df006af60 0x7f5df006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e101611b0 0x7f5e1006c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e100cee10 0x7f5e101f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 --2- 172.21.15.12:0/2193297817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e100d21f0 0x7f5e100b6150 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 -- 172.21.15.12:0/2193297817 >> 172.21.15.12:0/2193297817 conn(0x7f5e1008aff0 msgr2=0x7f5e101619e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:56.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.734+0000 7f5e17400700 1 -- 172.21.15.12:0/2193297817 shutdown_connections 2024-03-29T11:35:56.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.735+0000 7f5e17400700 1 -- 172.21.15.12:0/2193297817 wait complete. 2024-03-29T11:35:56.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:35:56.911 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:56.912+0000 7fc7324ff240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:35:56.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.963+0000 7fb137600700 1 Processor -- start 2024-03-29T11:35:56.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.964+0000 7fb137600700 1 -- start start 2024-03-29T11:35:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.965+0000 7fb137600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb13015d8a0 0x7fb1300b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.965+0000 7fb137600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1300cf1b0 0x7fb1300cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.965+0000 7fb137600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1300ced20 con 0x7fb1300d3eb0 2024-03-29T11:35:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.965+0000 7fb137600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1300d4340 con 0x7fb13015d8a0 2024-03-29T11:35:56.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.965+0000 7fb137600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300ce6d0 con 0x7fb1300cf1b0 2024-03-29T11:35:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.965+0000 7fb134c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb13015d8a0 0x7fb1300b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.965+0000 7fb134c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb13015d8a0 0x7fb1300b2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52990/0 (socket says 172.21.15.12:52990) 2024-03-29T11:35:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.965+0000 7fb134c00700 1 -- 172.21.15.12:0/2333066754 learned_addr learned my addr 172.21.15.12:0/2333066754 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.965+0000 7fb135600700 1 --2- 172.21.15.12:0/2333066754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1300cf1b0 0x7fb1300cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.966+0000 7fb134c00700 1 -- 172.21.15.12:0/2333066754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1300cf1b0 msgr2=0x7fb1300cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.966+0000 7fb134c00700 1 --2- 172.21.15.12:0/2333066754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1300cf1b0 0x7fb1300cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.966+0000 7fb134c00700 1 -- 172.21.15.12:0/2333066754 >> v1:172.21.15.159:6789/0 conn(0x7fb1300d3eb0 legacy=0x7fb13006b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.966+0000 7fb134c00700 1 -- 172.21.15.12:0/2333066754 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1300ce9b0 con 0x7fb13015d8a0 2024-03-29T11:35:56.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.966+0000 7fb134c00700 1 --2- 172.21.15.12:0/2333066754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 0x7fb1300b2990 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fb1240098f0 tx=0x7fb124009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6163c87eb6725c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:56.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.966+0000 7fb12f400700 1 -- 172.21.15.12:0/2333066754 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12400b8a0 con 0x7fb13015d8a0 2024-03-29T11:35:56.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.966+0000 7fb12f400700 1 -- 172.21.15.12:0/2333066754 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12400ba00 con 0x7fb13015d8a0 2024-03-29T11:35:56.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.966+0000 7fb12f400700 1 -- 172.21.15.12:0/2333066754 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12400bd90 con 0x7fb13015d8a0 2024-03-29T11:35:56.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.967+0000 7fb137600700 1 -- 172.21.15.12:0/2333066754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 msgr2=0x7fb1300b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.967+0000 7fb137600700 1 --2- 172.21.15.12:0/2333066754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 0x7fb1300b2990 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fb1240098f0 tx=0x7fb124009c00 comp rx=0 tx=0).stop 2024-03-29T11:35:56.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.967+0000 7fb137600700 1 -- 172.21.15.12:0/2333066754 shutdown_connections 2024-03-29T11:35:56.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.967+0000 7fb137600700 1 --2- 172.21.15.12:0/2333066754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1300cf1b0 0x7fb1300cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.967+0000 7fb137600700 1 --2- 172.21.15.12:0/2333066754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 0x7fb1300b2990 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.967+0000 7fb137600700 1 -- 172.21.15.12:0/2333066754 >> 172.21.15.12:0/2333066754 conn(0x7fb13008aff0 msgr2=0x7fb1300d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:56.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.967+0000 7fb137600700 1 -- 172.21.15.12:0/2333066754 shutdown_connections 2024-03-29T11:35:56.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.967+0000 7fb137600700 1 -- 172.21.15.12:0/2333066754 wait complete. 2024-03-29T11:35:56.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.969+0000 7fb137600700 1 Processor -- start 2024-03-29T11:35:56.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.969+0000 7fb137600700 1 -- start start 2024-03-29T11:35:56.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.970+0000 7fb137600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1300cf1b0 0x7fb130159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.970+0000 7fb137600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 0x7fb1300b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.970+0000 7fb137600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1300c8d20 0x7fb1300c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.970+0000 7fb137600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300b50a0 con 0x7fb13015d8a0 2024-03-29T11:35:56.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.970+0000 7fb137600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300c96f0 con 0x7fb1300c8d20 2024-03-29T11:35:56.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.970+0000 7fb137600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1301f3df0 con 0x7fb1300cf1b0 2024-03-29T11:35:56.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.970+0000 7fb12fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 0x7fb1300b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.970+0000 7fb12fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 0x7fb1300b4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53006/0 (socket says 172.21.15.12:53006) 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.970+0000 7fb12fe00700 1 -- 172.21.15.12:0/1771799417 learned_addr learned my addr 172.21.15.12:0/1771799417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.970+0000 7fb134c00700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1300cf1b0 0x7fb130159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb135600700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1300c8d20 0x7fb1300c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb12fe00700 1 -- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1300cf1b0 msgr2=0x7fb130159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb12fe00700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1300cf1b0 0x7fb130159860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb12fe00700 1 -- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1300c8d20 msgr2=0x7fb1300c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb12fe00700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1300c8d20 0x7fb1300c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb12fe00700 1 -- 172.21.15.12:0/1771799417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb124009550 con 0x7fb13015d8a0 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb135600700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1300c8d20 0x7fb1300c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb12fe00700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 0x7fb1300b4b60 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00c4d0 tx=0x7fb11c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb12cc00700 1 -- 172.21.15.12:0/1771799417 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c0052d0 con 0x7fb13015d8a0 2024-03-29T11:35:56.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb12cc00700 1 -- 172.21.15.12:0/1771799417 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c005430 con 0x7fb13015d8a0 2024-03-29T11:35:56.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb137600700 1 -- 172.21.15.12:0/1771799417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1300b3da0 con 0x7fb13015d8a0 2024-03-29T11:35:56.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.971+0000 7fb12cc00700 1 -- 172.21.15.12:0/1771799417 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c0038f0 con 0x7fb13015d8a0 2024-03-29T11:35:56.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.972+0000 7fb137600700 1 -- 172.21.15.12:0/1771799417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1300b4360 con 0x7fb13015d8a0 2024-03-29T11:35:56.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.973+0000 7fb137600700 1 -- 172.21.15.12:0/1771799417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb130067720 con 0x7fb13015d8a0 2024-03-29T11:35:56.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.974+0000 7fb12cc00700 1 -- 172.21.15.12:0/1771799417 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fb11c003a50 con 0x7fb13015d8a0 2024-03-29T11:35:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.975+0000 7fb12cc00700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb11006af60 0x7fb11006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:56.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.975+0000 7fb12cc00700 1 -- 172.21.15.12:0/1771799417 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7fb11c088d30 con 0x7fb13015d8a0 2024-03-29T11:35:56.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.975+0000 7fb134c00700 1 -- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb11006af60 msgr2=0x7fb11006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:56.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.975+0000 7fb134c00700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb11006af60 0x7fb11006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:35:56.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:56.984+0000 7fb12cc00700 1 -- 172.21.15.12:0/1771799417 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb11c053a50 con 0x7fb13015d8a0 2024-03-29T11:35:57.003 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:57.004+0000 7f3f02537240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:35:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.176+0000 7fb134c00700 1 -- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb11006af60 msgr2=0x7fb11006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.176+0000 7fb134c00700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb11006af60 0x7fb11006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:35:57.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.197+0000 7fb137600700 1 -- 172.21.15.12:0/1771799417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1300d1950 con 0x7fb13015d8a0 2024-03-29T11:35:57.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.198+0000 7fb12cc00700 1 -- 172.21.15.12:0/1771799417 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v286) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb11c017020 con 0x7fb13015d8a0 2024-03-29T11:35:57.197 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:35:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 -- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb11006af60 msgr2=0x7fb11006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb11006af60 0x7fb11006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 -- 172.21.15.12:0/1771799417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 msgr2=0x7fb1300b4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 0x7fb1300b4b60 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00c4d0 tx=0x7fb11c00c7e0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 -- 172.21.15.12:0/1771799417 shutdown_connections 2024-03-29T11:35:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb11006af60 0x7fb11006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1300c8d20 0x7fb1300c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1300cf1b0 0x7fb130159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 --2- 172.21.15.12:0/1771799417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13015d8a0 0x7fb1300b4b60 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 -- 172.21.15.12:0/1771799417 >> 172.21.15.12:0/1771799417 conn(0x7fb13008aff0 msgr2=0x7fb13015df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 -- 172.21.15.12:0/1771799417 shutdown_connections 2024-03-29T11:35:57.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.200+0000 7fb137600700 1 -- 172.21.15.12:0/1771799417 wait complete. 2024-03-29T11:35:57.219 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:35:57.374 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:57.375+0000 7fc7324ff240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:35:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.433+0000 7f181de00700 1 Processor -- start 2024-03-29T11:35:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.433+0000 7f181de00700 1 -- start start 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.435+0000 7f181de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f18180ce230 0x7f18180ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.435+0000 7f181de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180b2070 0x7f18180b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.435+0000 7f181de00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18180709c0 con 0x7f181815d800 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.435+0000 7f181de00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18180b40f0 con 0x7f18180ce230 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.435+0000 7f181de00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1818070c80 con 0x7f18180b2070 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.435+0000 7f1816000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f18180ce230 0x7f18180ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.435+0000 7f1816000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f18180ce230 0x7f18180ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53022/0 (socket says 172.21.15.12:53022) 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.435+0000 7f1816000700 1 -- 172.21.15.12:0/3395355641 learned_addr learned my addr 172.21.15.12:0/3395355641 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.435+0000 7f1817400700 1 --2- 172.21.15.12:0/3395355641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180b2070 0x7f18180b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.436+0000 7f1816000700 1 -- 172.21.15.12:0/3395355641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180b2070 msgr2=0x7f18180b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.436+0000 7f1816000700 1 --2- 172.21.15.12:0/3395355641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180b2070 0x7f18180b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.436+0000 7f1816000700 1 -- 172.21.15.12:0/3395355641 >> v1:172.21.15.159:6789/0 conn(0x7f181815d800 legacy=0x7f18180725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.436+0000 7f1816000700 1 -- 172.21.15.12:0/3395355641 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818070f60 con 0x7f18180ce230 2024-03-29T11:35:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.436+0000 7f1816000700 1 --2- 172.21.15.12:0/3395355641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18180ce230 0x7f18180ce6a0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f180400aa90 tx=0x7f180400ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f8ff9fcccd2a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.436+0000 7f1815600700 1 -- 172.21.15.12:0/3395355641 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180400cb70 con 0x7f18180ce230 2024-03-29T11:35:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.436+0000 7f1815600700 1 -- 172.21.15.12:0/3395355641 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180400ccd0 con 0x7f18180ce230 2024-03-29T11:35:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.436+0000 7f1815600700 1 -- 172.21.15.12:0/3395355641 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804018620 con 0x7f18180ce230 2024-03-29T11:35:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.437+0000 7f181de00700 1 -- 172.21.15.12:0/3395355641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18180ce230 msgr2=0x7f18180ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.437+0000 7f181de00700 1 --2- 172.21.15.12:0/3395355641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18180ce230 0x7f18180ce6a0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f180400aa90 tx=0x7f180400ada0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.437+0000 7f181de00700 1 -- 172.21.15.12:0/3395355641 shutdown_connections 2024-03-29T11:35:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.437+0000 7f181de00700 1 --2- 172.21.15.12:0/3395355641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180b2070 0x7f18180b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.437+0000 7f181de00700 1 --2- 172.21.15.12:0/3395355641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18180ce230 0x7f18180ce6a0 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.437+0000 7f181de00700 1 -- 172.21.15.12:0/3395355641 >> 172.21.15.12:0/3395355641 conn(0x7f18180b0d70 msgr2=0x7f18180b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.437+0000 7f181de00700 1 -- 172.21.15.12:0/3395355641 shutdown_connections 2024-03-29T11:35:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.437+0000 7f181de00700 1 -- 172.21.15.12:0/3395355641 wait complete. 2024-03-29T11:35:57.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.439+0000 7f181de00700 1 Processor -- start 2024-03-29T11:35:57.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.440+0000 7f181de00700 1 -- start start 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f181de00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18180b2070 0x7f1818155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f181de00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180ce230 0x7f1818098920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f181de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181815d800 0x7f181806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f181de00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1818144330 con 0x7f181815d800 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f181de00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1818156dd0 con 0x7f18180b2070 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f181de00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1818156530 con 0x7f18180ce230 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f1817400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181815d800 0x7f181806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f1816a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18180b2070 0x7f1818155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f1816000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180ce230 0x7f1818098920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f1816a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18180b2070 0x7f1818155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:45252/0 (socket says 172.21.15.12:45252) 2024-03-29T11:35:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f1817400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181815d800 0x7f181806d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53024/0 (socket says 172.21.15.12:53024) 2024-03-29T11:35:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.441+0000 7f1816a00700 1 -- 172.21.15.12:0/2205369917 learned_addr learned my addr 172.21.15.12:0/2205369917 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f1816000700 1 -- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18180b2070 msgr2=0x7f1818155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f1816000700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18180b2070 0x7f1818155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f1816000700 1 -- 172.21.15.12:0/2205369917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181815d800 msgr2=0x7f181806d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f1816000700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181815d800 0x7f181806d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f1816000700 1 -- 172.21.15.12:0/2205369917 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180400a6f0 con 0x7f18180ce230 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f1817400700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181815d800 0x7f181806d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f1816000700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180ce230 0x7f1818098920 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1804017000 tx=0x7f18040047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f180f400700 1 -- 172.21.15.12:0/2205369917 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804003ab0 con 0x7f18180ce230 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f180f400700 1 -- 172.21.15.12:0/2205369917 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1804003c10 con 0x7f18180ce230 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f181de00700 1 -- 172.21.15.12:0/2205369917 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1818156810 con 0x7f18180ce230 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.442+0000 7f180f400700 1 -- 172.21.15.12:0/2205369917 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804020400 con 0x7f18180ce230 2024-03-29T11:35:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.443+0000 7f181de00700 1 -- 172.21.15.12:0/2205369917 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1818098e60 con 0x7f18180ce230 2024-03-29T11:35:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.444+0000 7f181de00700 1 -- 172.21.15.12:0/2205369917 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18181444a0 con 0x7f18180ce230 2024-03-29T11:35:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.445+0000 7f180f400700 1 -- 172.21.15.12:0/2205369917 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f1804018400 con 0x7f18180ce230 2024-03-29T11:35:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.445+0000 7f180f400700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17f806af60 0x7f17f806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.445+0000 7f180f400700 1 -- 172.21.15.12:0/2205369917 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7f180408acc0 con 0x7f18180ce230 2024-03-29T11:35:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.446+0000 7f1816a00700 1 -- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17f806af60 msgr2=0x7f17f806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.446+0000 7f1816a00700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17f806af60 0x7f17f806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:35:57.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.450+0000 7f180f400700 1 -- 172.21.15.12:0/2205369917 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18181444a0 con 0x7f18180ce230 2024-03-29T11:35:57.506 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:57.506+0000 7f3f02537240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:35:57.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.647+0000 7f1816a00700 1 -- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17f806af60 msgr2=0x7f17f806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:57.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.647+0000 7f1816a00700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17f806af60 0x7f17f806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:35:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.674+0000 7f181de00700 1 -- 172.21.15.12:0/2205369917 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18181444a0 con 0x7f18180ce230 2024-03-29T11:35:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.675+0000 7f180f400700 1 -- 172.21.15.12:0/2205369917 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v286) v1 ==== 100+0+11 (secure 0 0 0) 0x7f18181444a0 con 0x7f18180ce230 2024-03-29T11:35:57.675 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 -- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17f806af60 msgr2=0x7f17f806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17f806af60 0x7f17f806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 -- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180ce230 msgr2=0x7f1818098920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180ce230 0x7f1818098920 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1804017000 tx=0x7f18040047c0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 -- 172.21.15.12:0/2205369917 shutdown_connections 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f17f806af60 0x7f17f806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f181815d800 0x7f181806d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18180b2070 0x7f1818155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 --2- 172.21.15.12:0/2205369917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18180ce230 0x7f1818098920 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 -- 172.21.15.12:0/2205369917 >> 172.21.15.12:0/2205369917 conn(0x7f18180b0d70 msgr2=0x7f18180aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 -- 172.21.15.12:0/2205369917 shutdown_connections 2024-03-29T11:35:57.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.678+0000 7f181de00700 1 -- 172.21.15.12:0/2205369917 wait complete. 2024-03-29T11:35:57.697 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-29T11:35:57.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.898+0000 7f12b9000700 1 Processor -- start 2024-03-29T11:35:57.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.898+0000 7f12b9000700 1 -- start start 2024-03-29T11:35:57.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.899+0000 7f12b9000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b40ce840 0x7f12b4165370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.899+0000 7f12b9000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b40cb310 con 0x7f12b415dfe0 2024-03-29T11:35:57.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.899+0000 7f12b9000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b40cca90 con 0x7f12b415a5c0 2024-03-29T11:35:57.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.899+0000 7f12b9000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40711a0 con 0x7f12b40ce840 2024-03-29T11:35:57.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.899+0000 7f12b1400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f12b415dfe0 0x7f12b40ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55176/0 (socket says 172.21.15.12:55176) 2024-03-29T11:35:57.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.899+0000 7f12b1400700 1 -- 172.21.15.12:0/804366935 learned_addr learned my addr 172.21.15.12:0/804366935 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:35:57.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.899+0000 7f12b2800700 1 --2- 172.21.15.12:0/804366935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b40ce840 0x7f12b4165370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:57.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.900+0000 7f12b0a00700 1 -- 172.21.15.12:0/804366935 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480235582 0 0) 0x7f12b40cb310 con 0x7f12b415dfe0 2024-03-29T11:35:57.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.900+0000 7f12b0a00700 1 -- 172.21.15.12:0/804366935 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12980035b0 con 0x7f12b415dfe0 2024-03-29T11:35:57.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.900+0000 7f12b0a00700 1 -- 172.21.15.12:0/804366935 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849864576 0 0) 0x7f12b40cca90 con 0x7f12b415a5c0 2024-03-29T11:35:57.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.900+0000 7f12b0a00700 1 -- 172.21.15.12:0/804366935 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b40cb310 con 0x7f12b415a5c0 2024-03-29T11:35:57.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.900+0000 7f12b2800700 1 -- 172.21.15.12:0/804366935 >> v1:172.21.15.159:6789/0 conn(0x7f12b415a5c0 legacy=0x7f12b40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.900+0000 7f12b2800700 1 -- 172.21.15.12:0/804366935 >> v1:172.21.15.12:6789/0 conn(0x7f12b415dfe0 legacy=0x7f12b40ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.900+0000 7f12b2800700 1 -- 172.21.15.12:0/804366935 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4071480 con 0x7f12b40ce840 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.900+0000 7f12b2800700 1 --2- 172.21.15.12:0/804366935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b40ce840 0x7f12b4165370 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009dc0 tx=0x7f12ac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64f0ee2df7881315 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.901+0000 7f12b0a00700 1 -- 172.21.15.12:0/804366935 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac00e070 con 0x7f12b40ce840 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.901+0000 7f12b0a00700 1 -- 172.21.15.12:0/804366935 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac00bb20 con 0x7f12b40ce840 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.901+0000 7f12b0a00700 1 -- 172.21.15.12:0/804366935 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac012430 con 0x7f12b40ce840 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.901+0000 7f12b9000700 1 -- 172.21.15.12:0/804366935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b40ce840 msgr2=0x7f12b4165370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.901+0000 7f12b9000700 1 --2- 172.21.15.12:0/804366935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b40ce840 0x7f12b4165370 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009dc0 tx=0x7f12ac00b330 comp rx=0 tx=0).stop 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.901+0000 7f12b9000700 1 -- 172.21.15.12:0/804366935 shutdown_connections 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.901+0000 7f12b9000700 1 --2- 172.21.15.12:0/804366935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b40ce840 0x7f12b4165370 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.901+0000 7f12b9000700 1 -- 172.21.15.12:0/804366935 >> 172.21.15.12:0/804366935 conn(0x7f12b40b0d70 msgr2=0x7f12b40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:57.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.901+0000 7f12b9000700 1 -- 172.21.15.12:0/804366935 shutdown_connections 2024-03-29T11:35:57.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.901+0000 7f12b9000700 1 -- 172.21.15.12:0/804366935 wait complete. 2024-03-29T11:35:57.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.903+0000 7f12b9000700 1 Processor -- start 2024-03-29T11:35:57.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.903+0000 7f12b9000700 1 -- start start 2024-03-29T11:35:57.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.904+0000 7f12b9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b40ce840 0x7f12b4153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.904+0000 7f12b9000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b415a5c0 0x7f12b4158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.904+0000 7f12b9000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b415dfe0 0x7f12b41f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.904+0000 7f12b9000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b4152390 con 0x7f12b40ce840 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.904+0000 7f12b9000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b41520b0 con 0x7f12b415dfe0 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.904+0000 7f12b9000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40ae060 con 0x7f12b415a5c0 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.904+0000 7f12b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b40ce840 0x7f12b4153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.904+0000 7f12b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b40ce840 0x7f12b4153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53040/0 (socket says 172.21.15.12:53040) 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.904+0000 7f12b1e00700 1 -- 172.21.15.12:0/560143543 learned_addr learned my addr 172.21.15.12:0/560143543 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.904+0000 7f12b1400700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b415a5c0 0x7f12b4158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.905+0000 7f12b2800700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b415dfe0 0x7f12b41f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.905+0000 7f12b1e00700 1 -- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b415a5c0 msgr2=0x7f12b4158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.905+0000 7f12b1e00700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b415a5c0 0x7f12b4158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.905+0000 7f12b1e00700 1 -- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b415dfe0 msgr2=0x7f12b41f6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:57.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.905+0000 7f12b1e00700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b415dfe0 0x7f12b41f6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:57.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.905+0000 7f12b1e00700 1 -- 172.21.15.12:0/560143543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac009a70 con 0x7f12b40ce840 2024-03-29T11:35:57.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.905+0000 7f12b1e00700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b40ce840 0x7f12b4153b20 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c520 tx=0x7f12a400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:57.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.905+0000 7f12aa000700 1 -- 172.21.15.12:0/560143543 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40052d0 con 0x7f12b40ce840 2024-03-29T11:35:57.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.906+0000 7f12aa000700 1 -- 172.21.15.12:0/560143543 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a4005430 con 0x7f12b40ce840 2024-03-29T11:35:57.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.906+0000 7f12aa000700 1 -- 172.21.15.12:0/560143543 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40038c0 con 0x7f12b40ce840 2024-03-29T11:35:57.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.906+0000 7f12b9000700 1 -- 172.21.15.12:0/560143543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b4151a60 con 0x7f12b40ce840 2024-03-29T11:35:57.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.906+0000 7f12b9000700 1 -- 172.21.15.12:0/560143543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b41515b0 con 0x7f12b40ce840 2024-03-29T11:35:57.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.907+0000 7f12b9000700 1 -- 172.21.15.12:0/560143543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b4067720 con 0x7f12b40ce840 2024-03-29T11:35:57.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.908+0000 7f12aa000700 1 -- 172.21.15.12:0/560143543 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f12a4003a20 con 0x7f12b40ce840 2024-03-29T11:35:57.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.909+0000 7f12aa000700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f129406af60 0x7f129406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:57.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.909+0000 7f12aa000700 1 -- 172.21.15.12:0/560143543 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7f12a4088df0 con 0x7f12b40ce840 2024-03-29T11:35:57.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.909+0000 7f12b1400700 1 -- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f129406af60 msgr2=0x7f129406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:57.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.909+0000 7f12b1400700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f129406af60 0x7f129406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:35:57.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:57.913+0000 7f12aa000700 1 -- 172.21.15.12:0/560143543 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12a4053a80 con 0x7f12b40ce840 2024-03-29T11:35:58.091 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:58.092+0000 7fc7324ff240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:35:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.110+0000 7f12b1400700 1 -- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f129406af60 msgr2=0x7f129406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:58.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.110+0000 7f12b1400700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f129406af60 0x7f129406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:35:58.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.143+0000 7f12b9000700 1 -- 172.21.15.12:0/560143543 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f12b40b37f0 con 0x7f12b40ce840 2024-03-29T11:35:58.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.144+0000 7f12aa000700 1 -- 172.21.15.12:0/560143543 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v286) v1 ==== 80+0+466 (secure 0 0 0) 0x7f12a4017080 con 0x7f12b40ce840 2024-03-29T11:35:58.144 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2115167177 2024-03-30T11:34:37.193903+0000 2024-03-29T11:35:58.144 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6833/62646 2024-03-30T11:34:37.193903+0000 2024-03-29T11:35:58.144 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:6832/62646 2024-03-30T11:34:37.193903+0000 2024-03-29T11:35:58.144 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/411770029 2024-03-30T11:34:37.193903+0000 2024-03-29T11:35:58.144 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/3938940279 2024-03-30T11:34:37.193903+0000 2024-03-29T11:35:58.144 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/2298284405 2024-03-30T11:34:37.193903+0000 2024-03-29T11:35:58.144 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/1795874409 2024-03-30T11:34:37.193903+0000 2024-03-29T11:35:58.144 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.159:0/547605341 2024-03-30T11:34:37.193903+0000 2024-03-29T11:35:58.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.146+0000 7f12b9000700 1 -- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f129406af60 msgr2=0x7f129406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.146+0000 7f12b9000700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f129406af60 0x7f129406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.146+0000 7f12b9000700 1 -- 172.21.15.12:0/560143543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b40ce840 msgr2=0x7f12b4153b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.146+0000 7f12b9000700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b40ce840 0x7f12b4153b20 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c520 tx=0x7f12a400c830 comp rx=0 tx=0).stop 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.146+0000 7f12b9000700 1 -- 172.21.15.12:0/560143543 shutdown_connections 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.146+0000 7f12b9000700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f129406af60 0x7f129406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.146+0000 7f12b9000700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12b415dfe0 0x7f12b41f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.146+0000 7f12b9000700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12b40ce840 0x7f12b4153b20 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.146+0000 7f12b9000700 1 --2- 172.21.15.12:0/560143543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12b415a5c0 0x7f12b4158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.147+0000 7f12b9000700 1 -- 172.21.15.12:0/560143543 >> 172.21.15.12:0/560143543 conn(0x7f12b40b0d70 msgr2=0x7f12b406db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.147+0000 7f12b9000700 1 -- 172.21.15.12:0/560143543 shutdown_connections 2024-03-29T11:35:58.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.147+0000 7f12b9000700 1 -- 172.21.15.12:0/560143543 wait complete. 2024-03-29T11:35:58.148 INFO:teuthology.orchestra.run.smithi012.stderr:listed 8 entries 2024-03-29T11:35:58.166 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-29T11:35:58.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:35:58.166 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-29T11:35:58.182 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:58.183+0000 7fc7324ff240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:35:58.241 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:35:58.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:35:58.241 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-29T11:35:58.271 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:58.271+0000 7f3f02537240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:35:58.286 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:35:58.287 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:35:58.331 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:35:58.331 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:35:58.331 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:35:58.351 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:58.352+0000 7fc7324ff240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:35:58.372 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:58.372+0000 7f3f02537240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:35:58.376 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:35:58.376 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:35:58.376 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:35:58.421 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:35:58.421 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:35:58.548 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:58.549+0000 7f3f02537240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:35:58.573 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:58.575+0000 7fc7324ff240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:35:58.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.623+0000 7f7bbb400700 1 Processor -- start 2024-03-29T11:35:58.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.623+0000 7f7bbb400700 1 -- start start 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.624+0000 7f7bbb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7bb415d710 0x7f7bb40d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.624+0000 7f7bbb400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7bb409aba0 0x7f7bb40d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.624+0000 7f7bbb400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb406bb40 con 0x7f7bb406bf40 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.624+0000 7f7bbb400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb4099b00 con 0x7f7bb415d710 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.624+0000 7f7bbb400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb4099de0 con 0x7f7bb409aba0 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.624+0000 7f7bb8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7bb415d710 0x7f7bb40d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.625+0000 7f7bb8a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7bb415d710 0x7f7bb40d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53056/0 (socket says 172.21.15.12:53056) 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.625+0000 7f7bb8a00700 1 -- 172.21.15.12:0/1014246645 learned_addr learned my addr 172.21.15.12:0/1014246645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.625+0000 7f7bb3e00700 1 --2- 172.21.15.12:0/1014246645 >> v2:172.21.15.159:3300/0 conn(0x7f7bb409aba0 0x7f7bb40d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.625+0000 7f7bb8a00700 1 -- 172.21.15.12:0/1014246645 >> v2:172.21.15.159:3300/0 conn(0x7f7bb409aba0 msgr2=0x7f7bb40d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.625+0000 7f7bb8a00700 1 --2- 172.21.15.12:0/1014246645 >> v2:172.21.15.159:3300/0 conn(0x7f7bb409aba0 0x7f7bb40d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.625+0000 7f7bb8a00700 1 -- 172.21.15.12:0/1014246645 >> v1:172.21.15.159:6789/0 conn(0x7f7bb406bf40 legacy=0x7f7bb40ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.625+0000 7f7bb8a00700 1 -- 172.21.15.12:0/1014246645 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb409a0c0 con 0x7f7bb415d710 2024-03-29T11:35:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.625+0000 7f7bb8a00700 1 --2- 172.21.15.12:0/1014246645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb415d710 0x7f7bb40d0130 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4072100 tx=0x7f7ba8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a34639d421f7518 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.626+0000 7f7bb3400700 1 -- 172.21.15.12:0/1014246645 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8011830 con 0x7f7bb415d710 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.626+0000 7f7bb3400700 1 -- 172.21.15.12:0/1014246645 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba8011990 con 0x7f7bb415d710 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.626+0000 7f7bb3400700 1 -- 172.21.15.12:0/1014246645 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8011d20 con 0x7f7bb415d710 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.626+0000 7f7bbb400700 1 -- 172.21.15.12:0/1014246645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb415d710 msgr2=0x7f7bb40d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.626+0000 7f7bbb400700 1 --2- 172.21.15.12:0/1014246645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb415d710 0x7f7bb40d0130 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4072100 tx=0x7f7ba8009bd0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.627+0000 7f7bbb400700 1 -- 172.21.15.12:0/1014246645 shutdown_connections 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.627+0000 7f7bbb400700 1 --2- 172.21.15.12:0/1014246645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb415d710 0x7f7bb40d0130 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.627+0000 7f7bbb400700 1 --2- 172.21.15.12:0/1014246645 >> v2:172.21.15.159:3300/0 conn(0x7f7bb409aba0 0x7f7bb40d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.627+0000 7f7bbb400700 1 -- 172.21.15.12:0/1014246645 >> 172.21.15.12:0/1014246645 conn(0x7f7bb40d13e0 msgr2=0x7f7bb40d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.627+0000 7f7bbb400700 1 -- 172.21.15.12:0/1014246645 shutdown_connections 2024-03-29T11:35:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.627+0000 7f7bbb400700 1 -- 172.21.15.12:0/1014246645 wait complete. 2024-03-29T11:35:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.628+0000 7f7bbb400700 1 Processor -- start 2024-03-29T11:35:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.629+0000 7f7bbb400700 1 -- start start 2024-03-29T11:35:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.629+0000 7f7bbb400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb406bf40 0x7f7bb4157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.629+0000 7f7bbb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb409aba0 0x7f7bb41f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.629+0000 7f7bbb400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb415d710 0x7f7bb415f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.629+0000 7f7bbb400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb415efc0 con 0x7f7bb409aba0 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.629+0000 7f7bbb400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb41f6a10 con 0x7f7bb406bf40 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.629+0000 7f7bbb400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb4160cb0 con 0x7f7bb415d710 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb409aba0 0x7f7bb41f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb8a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb406bf40 0x7f7bb4157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb409aba0 0x7f7bb41f3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53070/0 (socket says 172.21.15.12:53070) 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb3e00700 1 -- 172.21.15.12:0/2630347860 learned_addr learned my addr 172.21.15.12:0/2630347860 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb9400700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb415d710 0x7f7bb415f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb3e00700 1 -- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb415d710 msgr2=0x7f7bb415f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb3e00700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb415d710 0x7f7bb415f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb3e00700 1 -- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb406bf40 msgr2=0x7f7bb4157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb3e00700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb406bf40 0x7f7bb4157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb3e00700 1 -- 172.21.15.12:0/2630347860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba8009550 con 0x7f7bb409aba0 2024-03-29T11:35:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb8a00700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb406bf40 0x7f7bb4157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:35:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.630+0000 7f7bb3e00700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb409aba0 0x7f7bb41f3750 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000e9e0 tx=0x7f7ba000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.631+0000 7f7bb0c00700 1 -- 172.21.15.12:0/2630347860 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0009710 con 0x7f7bb409aba0 2024-03-29T11:35:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.631+0000 7f7bb0c00700 1 -- 172.21.15.12:0/2630347860 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba000ee80 con 0x7f7bb409aba0 2024-03-29T11:35:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.631+0000 7f7bbb400700 1 -- 172.21.15.12:0/2630347860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb41609d0 con 0x7f7bb409aba0 2024-03-29T11:35:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.631+0000 7f7bb0c00700 1 -- 172.21.15.12:0/2630347860 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba000ca90 con 0x7f7bb409aba0 2024-03-29T11:35:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.631+0000 7f7bbb400700 1 -- 172.21.15.12:0/2630347860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb415ec50 con 0x7f7bb409aba0 2024-03-29T11:35:58.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.632+0000 7f7bbb400700 1 -- 172.21.15.12:0/2630347860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb415f170 con 0x7f7bb409aba0 2024-03-29T11:35:58.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.633+0000 7f7bb0c00700 1 -- 172.21.15.12:0/2630347860 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f7ba000cbf0 con 0x7f7bb409aba0 2024-03-29T11:35:58.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.634+0000 7f7bb0c00700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b9406af90 0x7f7b9406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:58.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.634+0000 7f7bb0c00700 1 -- 172.21.15.12:0/2630347860 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7f7ba00898c0 con 0x7f7bb409aba0 2024-03-29T11:35:58.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.634+0000 7f7bb8a00700 1 -- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b9406af90 msgr2=0x7f7b9406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:58.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.634+0000 7f7bb8a00700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b9406af90 0x7f7b9406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:35:58.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.639+0000 7f7bb0c00700 1 -- 172.21.15.12:0/2630347860 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7bb415f170 con 0x7f7bb409aba0 2024-03-29T11:35:58.666 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:58.668+0000 7fc7324ff240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:35:58.765 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:58.766+0000 7fc7324ff240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:35:58.772 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:58.773+0000 7f3f02537240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:35:58.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.835+0000 7f7bb8a00700 1 -- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b9406af90 msgr2=0x7f7b9406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:58.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.835+0000 7f7bb8a00700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b9406af90 0x7f7b9406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:35:58.864 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:58.865+0000 7fc7324ff240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:35:58.870 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:58.871+0000 7f3f02537240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:35:58.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.898+0000 7f7bbb400700 1 -- 172.21.15.12:0/2630347860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bb415f170 con 0x7f7bb409aba0 2024-03-29T11:35:58.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.899+0000 7f7bb0c00700 1 -- 172.21.15.12:0/2630347860 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4747 (secure 0 0 0) 0x7f7bb415f170 con 0x7f7bb409aba0 2024-03-29T11:35:58.899 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:35:58.899 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":146,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12538,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/350949303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":350949303},{"type":"v1","addr":"172.21.15.159:6836","nonce":350949303}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1576953466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1576953466},{"type":"v1","addr":"172.21.15.159:6837","nonce":1576953466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/30523791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":30523791},{"type":"v1","addr":"172.21.15.12:6835","nonce":30523791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:34:19.924674+0000","modified":"2024-03-29T11:34:23.413936+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13125":{"gid":13125,"name":"a","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/2115072104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2115072104},{"type":"v1","addr":"172.21.15.12:6834","nonce":2115072104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-29T11:35:58.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 -- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b9406af90 msgr2=0x7f7b9406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:58.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b9406af90 0x7f7b9406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 -- 172.21.15.12:0/2630347860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb409aba0 msgr2=0x7f7bb41f3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:58.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb409aba0 0x7f7bb41f3750 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000e9e0 tx=0x7f7ba000ecf0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 -- 172.21.15.12:0/2630347860 shutdown_connections 2024-03-29T11:35:58.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7b9406af90 0x7f7b9406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb415d710 0x7f7bb415f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb406bf40 0x7f7bb4157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 --2- 172.21.15.12:0/2630347860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7bb409aba0 0x7f7bb41f3750 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:58.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 -- 172.21.15.12:0/2630347860 >> 172.21.15.12:0/2630347860 conn(0x7f7bb40d13e0 msgr2=0x7f7bb40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:58.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.901+0000 7f7bbb400700 1 -- 172.21.15.12:0/2630347860 shutdown_connections 2024-03-29T11:35:58.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:58.902+0000 7f7bbb400700 1 -- 172.21.15.12:0/2630347860 wait complete. 2024-03-29T11:35:58.903 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 146 2024-03-29T11:35:58.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:58.963 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:58.964+0000 7fc7324ff240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:35:58.981 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:58.982+0000 7f3f02537240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:35:59.089 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:59.089+0000 7f3f02537240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:35:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.121+0000 7f4887200700 1 Processor -- start 2024-03-29T11:35:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.121+0000 7f4887200700 1 -- start start 2024-03-29T11:35:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f4887200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f488015d640 0x7f48800d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f4887200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48800ceec0 0x7f48801612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f4887200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4880161800 0x7f48800cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f4887200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4880161c90 con 0x7f488015d640 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f4887200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48800d30d0 con 0x7f4880161800 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f4887200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48800b6840 con 0x7f48800ceec0 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f487fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f488015d640 0x7f48800d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f487fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f488015d640 0x7f48800d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40742/0 (socket says 172.21.15.12:40742) 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f487fe00700 1 -- 172.21.15.12:0/751514816 learned_addr learned my addr 172.21.15.12:0/751514816 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f4885200700 1 --2- 172.21.15.12:0/751514816 >> v2:172.21.15.159:3300/0 conn(0x7f4880161800 0x7f48800cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.122+0000 7f487f400700 1 --2- 172.21.15.12:0/751514816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48800ceec0 0x7f48801612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.123+0000 7f487fe00700 1 -- 172.21.15.12:0/751514816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48800ceec0 msgr2=0x7f48801612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.123+0000 7f487fe00700 1 --2- 172.21.15.12:0/751514816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48800ceec0 0x7f48801612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.123+0000 7f487fe00700 1 -- 172.21.15.12:0/751514816 >> v2:172.21.15.159:3300/0 conn(0x7f4880161800 msgr2=0x7f48800cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.123+0000 7f487fe00700 1 --2- 172.21.15.12:0/751514816 >> v2:172.21.15.159:3300/0 conn(0x7f4880161800 0x7f48800cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.123+0000 7f487fe00700 1 -- 172.21.15.12:0/751514816 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48800d2880 con 0x7f488015d640 2024-03-29T11:35:59.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.123+0000 7f487fe00700 1 --2- 172.21.15.12:0/751514816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f488015d640 0x7f48800d2340 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f48800c7050 tx=0x7f4870009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2619df6840f90eb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.123+0000 7f487ea00700 1 -- 172.21.15.12:0/751514816 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4870011830 con 0x7f488015d640 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.123+0000 7f487ea00700 1 -- 172.21.15.12:0/751514816 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4870011990 con 0x7f488015d640 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.123+0000 7f487ea00700 1 -- 172.21.15.12:0/751514816 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4870011d20 con 0x7f488015d640 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.124+0000 7f4887200700 1 -- 172.21.15.12:0/751514816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f488015d640 msgr2=0x7f48800d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.124+0000 7f4887200700 1 --2- 172.21.15.12:0/751514816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f488015d640 0x7f48800d2340 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f48800c7050 tx=0x7f4870009bd0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.124+0000 7f4887200700 1 -- 172.21.15.12:0/751514816 shutdown_connections 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.124+0000 7f4887200700 1 --2- 172.21.15.12:0/751514816 >> v2:172.21.15.159:3300/0 conn(0x7f4880161800 0x7f48800cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.124+0000 7f4887200700 1 --2- 172.21.15.12:0/751514816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f488015d640 0x7f48800d2340 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.124+0000 7f4887200700 1 --2- 172.21.15.12:0/751514816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48800ceec0 0x7f48801612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.124+0000 7f4887200700 1 -- 172.21.15.12:0/751514816 >> 172.21.15.12:0/751514816 conn(0x7f488008aff0 msgr2=0x7f48800d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:59.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.124+0000 7f4887200700 1 -- 172.21.15.12:0/751514816 shutdown_connections 2024-03-29T11:35:59.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.124+0000 7f4887200700 1 -- 172.21.15.12:0/751514816 wait complete. 2024-03-29T11:35:59.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.126+0000 7f4887200700 1 Processor -- start 2024-03-29T11:35:59.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.126+0000 7f4887200700 1 -- start start 2024-03-29T11:35:59.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f4887200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48800ceec0 0x7f48801f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f4887200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f488015d640 0x7f48801536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f4887200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4880161800 0x7f4880152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f4887200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4880156370 con 0x7f48800ceec0 2024-03-29T11:35:59.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f4887200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4880155b60 con 0x7f4880161800 2024-03-29T11:35:59.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f4887200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4880157d80 con 0x7f488015d640 2024-03-29T11:35:59.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f487fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48800ceec0 0x7f48801f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f487fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48800ceec0 0x7f48801f2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40758/0 (socket says 172.21.15.12:40758) 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f487fe00700 1 -- 172.21.15.12:0/2409407703 learned_addr learned my addr 172.21.15.12:0/2409407703 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f487f400700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f488015d640 0x7f48801536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.127+0000 7f4885200700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4880161800 0x7f4880152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.128+0000 7f487fe00700 1 -- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f488015d640 msgr2=0x7f48801536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.128+0000 7f487fe00700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f488015d640 0x7f48801536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.128+0000 7f487fe00700 1 -- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4880161800 msgr2=0x7f4880152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.128+0000 7f487fe00700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4880161800 0x7f4880152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.128+0000 7f487fe00700 1 -- 172.21.15.12:0/2409407703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4870009550 con 0x7f48800ceec0 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.128+0000 7f487fe00700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48800ceec0 0x7f48801f2a40 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f4870009c10 tx=0x7f4870004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.128+0000 7f4877e00700 1 -- 172.21.15.12:0/2409407703 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4870004240 con 0x7f48800ceec0 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.128+0000 7f4877e00700 1 -- 172.21.15.12:0/2409407703 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487001f070 con 0x7f48800ceec0 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.128+0000 7f4877e00700 1 -- 172.21.15.12:0/2409407703 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700036a0 con 0x7f48800ceec0 2024-03-29T11:35:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.128+0000 7f4887200700 1 -- 172.21.15.12:0/2409407703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4880157790 con 0x7f48800ceec0 2024-03-29T11:35:59.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.129+0000 7f4887200700 1 -- 172.21.15.12:0/2409407703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4880155db0 con 0x7f48800ceec0 2024-03-29T11:35:59.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.130+0000 7f4887200700 1 -- 172.21.15.12:0/2409407703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4880067720 con 0x7f48800ceec0 2024-03-29T11:35:59.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.131+0000 7f4877e00700 1 -- 172.21.15.12:0/2409407703 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f4870021030 con 0x7f48800ceec0 2024-03-29T11:35:59.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.132+0000 7f4877e00700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f486006af60 0x7f486006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.132+0000 7f4877e00700 1 -- 172.21.15.12:0/2409407703 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7f487008aa90 con 0x7f48800ceec0 2024-03-29T11:35:59.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.132+0000 7f487f400700 1 -- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f486006af60 msgr2=0x7f486006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:59.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.132+0000 7f487f400700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f486006af60 0x7f486006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:35:59.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.136+0000 7f4877e00700 1 -- 172.21.15.12:0/2409407703 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f48700557b0 con 0x7f48800ceec0 2024-03-29T11:35:59.177 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:59.179+0000 7fc7324ff240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:35:59.197 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:59.197+0000 7f3f02537240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:35:59.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.333+0000 7f487f400700 1 -- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f486006af60 msgr2=0x7f486006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:59.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.333+0000 7f487f400700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f486006af60 0x7f486006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:35:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.353+0000 7f4887200700 1 -- 172.21.15.12:0/2409407703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48800b5c70 con 0x7f48800ceec0 2024-03-29T11:35:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.354+0000 7f4877e00700 1 -- 172.21.15.12:0/2409407703 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v286) v1 ==== 74+0+16721 (secure 0 0 0) 0x7f4870015090 con 0x7f48800ceec0 2024-03-29T11:35:59.354 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:35:59.354 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":286,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:35:38.461906+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:34:18.360241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-03-29T11:34:18.903400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"283","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.d","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2115167177":"2024-03-30T11:34:37.193903+0000","172.21.15.159:6833/62646":"2024-03-30T11:34:37.193903+0000","172.21.15.159:6832/62646":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/411770029":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/3938940279":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/2298284405":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/1795874409":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/547605341":"2024-03-30T11:34:37.193903+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 -- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f486006af60 msgr2=0x7f486006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f486006af60 0x7f486006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 -- 172.21.15.12:0/2409407703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48800ceec0 msgr2=0x7f48801f2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48800ceec0 0x7f48801f2a40 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f4870009c10 tx=0x7f4870004750 comp rx=0 tx=0).stop 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 -- 172.21.15.12:0/2409407703 shutdown_connections 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f486006af60 0x7f486006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4880161800 0x7f4880152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48800ceec0 0x7f48801f2a40 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 --2- 172.21.15.12:0/2409407703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f488015d640 0x7f48801536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 -- 172.21.15.12:0/2409407703 >> 172.21.15.12:0/2409407703 conn(0x7f488008aff0 msgr2=0x7f488015dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 -- 172.21.15.12:0/2409407703 shutdown_connections 2024-03-29T11:35:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.357+0000 7f4887200700 1 -- 172.21.15.12:0/2409407703 wait complete. 2024-03-29T11:35:59.376 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-29T09:54:16.494788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-29T11:34:18.360241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '282', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2024-03-29T11:34:18.903400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '283', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-29T11:35:59.376 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-29T11:35:59.431 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:59.431+0000 7f3f02537240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:35:59.439 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:59.440+0000 7fc7324ff240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:35:59.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.589+0000 7f7437800700 1 Processor -- start 2024-03-29T11:35:59.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.590+0000 7f7437800700 1 -- start start 2024-03-29T11:35:59.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.591+0000 7f7437800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f743015d9d0 0x7f74300d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.591+0000 7f7437800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74300d2540 0x7f74300ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.591+0000 7f7437800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7430161420 con 0x7f74300d16a0 2024-03-29T11:35:59.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.591+0000 7f7437800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f743015edc0 con 0x7f743015d9d0 2024-03-29T11:35:59.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.591+0000 7f7437800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7430161700 con 0x7f74300d2540 2024-03-29T11:35:59.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.591+0000 7f742fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f74300d16a0 0x7f74300d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40662/0 (socket says 172.21.15.12:40662) 2024-03-29T11:35:59.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.591+0000 7f742fe00700 1 -- 172.21.15.12:0/2788097455 learned_addr learned my addr 172.21.15.12:0/2788097455 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:35:59.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.591+0000 7f7435800700 1 --2- 172.21.15.12:0/2788097455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74300d2540 0x7f74300ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.591+0000 7f7434e00700 1 --2- 172.21.15.12:0/2788097455 >> v2:172.21.15.159:3300/0 conn(0x7f743015d9d0 0x7f74300d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.592+0000 7f742f400700 1 -- 172.21.15.12:0/2788097455 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282019646 0 0) 0x7f7430161420 con 0x7f74300d16a0 2024-03-29T11:35:59.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.592+0000 7f742f400700 1 -- 172.21.15.12:0/2788097455 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74180035b0 con 0x7f74300d16a0 2024-03-29T11:35:59.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.592+0000 7f7435800700 1 -- 172.21.15.12:0/2788097455 >> v2:172.21.15.159:3300/0 conn(0x7f743015d9d0 msgr2=0x7f74300d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:59.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.592+0000 7f7435800700 1 --2- 172.21.15.12:0/2788097455 >> v2:172.21.15.159:3300/0 conn(0x7f743015d9d0 0x7f74300d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.592+0000 7f7435800700 1 -- 172.21.15.12:0/2788097455 >> v1:172.21.15.12:6789/0 conn(0x7f74300d16a0 legacy=0x7f74300d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.592+0000 7f7435800700 1 -- 172.21.15.12:0/2788097455 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74301619e0 con 0x7f74300d2540 2024-03-29T11:35:59.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.592+0000 7f7435800700 1 --2- 172.21.15.12:0/2788097455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74300d2540 0x7f74300ce360 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7428009d90 tx=0x7f742800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=537a5ed2bfe6be38 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:59.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.592+0000 7f742f400700 1 -- 172.21.15.12:0/2788097455 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742800e070 con 0x7f74300d2540 2024-03-29T11:35:59.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.592+0000 7f742f400700 1 -- 172.21.15.12:0/2788097455 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742800bb20 con 0x7f74300d2540 2024-03-29T11:35:59.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.592+0000 7f742f400700 1 -- 172.21.15.12:0/2788097455 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428012430 con 0x7f74300d2540 2024-03-29T11:35:59.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.593+0000 7f7437800700 1 -- 172.21.15.12:0/2788097455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74300d2540 msgr2=0x7f74300ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.593+0000 7f7437800700 1 --2- 172.21.15.12:0/2788097455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74300d2540 0x7f74300ce360 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7428009d90 tx=0x7f742800b330 comp rx=0 tx=0).stop 2024-03-29T11:35:59.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.593+0000 7f7437800700 1 -- 172.21.15.12:0/2788097455 shutdown_connections 2024-03-29T11:35:59.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.593+0000 7f7437800700 1 --2- 172.21.15.12:0/2788097455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74300d2540 0x7f74300ce360 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.593+0000 7f7437800700 1 --2- 172.21.15.12:0/2788097455 >> v2:172.21.15.159:3300/0 conn(0x7f743015d9d0 0x7f74300d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.593+0000 7f7437800700 1 -- 172.21.15.12:0/2788097455 >> 172.21.15.12:0/2788097455 conn(0x7f743008aff0 msgr2=0x7f74300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:59.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.593+0000 7f7437800700 1 -- 172.21.15.12:0/2788097455 shutdown_connections 2024-03-29T11:35:59.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.593+0000 7f7437800700 1 -- 172.21.15.12:0/2788097455 wait complete. 2024-03-29T11:35:59.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.595+0000 7f7437800700 1 Processor -- start 2024-03-29T11:35:59.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.595+0000 7f7437800700 1 -- start start 2024-03-29T11:35:59.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.596+0000 7f7437800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74300d16a0 0x7f74301f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.596+0000 7f7437800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74300d2540 0x7f74300b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.596+0000 7f7437800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f743015d9d0 0x7f74300b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.596+0000 7f7437800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74300ca770 con 0x7f74300d2540 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.596+0000 7f7437800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74300caa50 con 0x7f74300d16a0 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.596+0000 7f7437800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74300cad30 con 0x7f743015d9d0 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.596+0000 7f742fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74300d2540 0x7f74300b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.596+0000 7f742fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74300d2540 0x7f74300b69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40774/0 (socket says 172.21.15.12:40774) 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.596+0000 7f742fe00700 1 -- 172.21.15.12:0/2711832098 learned_addr learned my addr 172.21.15.12:0/2711832098 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.596+0000 7f7434e00700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74300d16a0 0x7f74301f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f7435800700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f743015d9d0 0x7f74300b73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f742fe00700 1 -- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f743015d9d0 msgr2=0x7f74300b73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f742fe00700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f743015d9d0 0x7f74300b73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f742fe00700 1 -- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74300d16a0 msgr2=0x7f74301f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f742fe00700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74300d16a0 0x7f74301f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f742fe00700 1 -- 172.21.15.12:0/2711832098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7428009a70 con 0x7f74300d2540 2024-03-29T11:35:59.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f742fe00700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74300d2540 0x7f74300b69e0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f741c00c700 tx=0x7f741c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:35:59.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f742cc00700 1 -- 172.21.15.12:0/2711832098 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0074e0 con 0x7f74300d2540 2024-03-29T11:35:59.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f742cc00700 1 -- 172.21.15.12:0/2711832098 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c007640 con 0x7f74300d2540 2024-03-29T11:35:59.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f742cc00700 1 -- 172.21.15.12:0/2711832098 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c005a50 con 0x7f74300d2540 2024-03-29T11:35:59.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.597+0000 7f7437800700 1 -- 172.21.15.12:0/2711832098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74300cb010 con 0x7f74300d2540 2024-03-29T11:35:59.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.598+0000 7f7437800700 1 -- 172.21.15.12:0/2711832098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74300b6f20 con 0x7f74300d2540 2024-03-29T11:35:59.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.599+0000 7f7437800700 1 -- 172.21.15.12:0/2711832098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74300d1ca0 con 0x7f74300d2540 2024-03-29T11:35:59.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.600+0000 7f742cc00700 1 -- 172.21.15.12:0/2711832098 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f741c005bb0 con 0x7f74300d2540 2024-03-29T11:35:59.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.601+0000 7f742cc00700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f741006af60 0x7f741006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:35:59.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.601+0000 7f742cc00700 1 -- 172.21.15.12:0/2711832098 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7f741c088e80 con 0x7f74300d2540 2024-03-29T11:35:59.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.601+0000 7f7434e00700 1 -- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f741006af60 msgr2=0x7f741006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:59.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.601+0000 7f7434e00700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f741006af60 0x7f741006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:35:59.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.605+0000 7f742cc00700 1 -- 172.21.15.12:0/2711832098 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f741c053a80 con 0x7f74300d2540 2024-03-29T11:35:59.632 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:59.634+0000 7fc7324ff240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:35:59.705 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:59.705+0000 7f3f02537240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:35:59.727 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:35:59.728+0000 7fc7324ff240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.802+0000 7f7434e00700 1 -- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f741006af60 msgr2=0x7f741006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:35:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.802+0000 7f7434e00700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f741006af60 0x7f741006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:35:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.819+0000 7f7437800700 1 -- 172.21.15.12:0/2711832098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f74300cb530 con 0x7f74300d2540 2024-03-29T11:35:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.820+0000 7f742cc00700 1 -- 172.21.15.12:0/2711832098 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v286) v1 ==== 93+0+10 (secure 0 0 0) 0x7f74300cb530 con 0x7f74300d2540 2024-03-29T11:35:59.820 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-29T11:35:59.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 -- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f741006af60 msgr2=0x7f741006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:59.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f741006af60 0x7f741006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 -- 172.21.15.12:0/2711832098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74300d2540 msgr2=0x7f74300b69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74300d2540 0x7f74300b69e0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f741c00c700 tx=0x7f741c00ca10 comp rx=0 tx=0).stop 2024-03-29T11:35:59.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 -- 172.21.15.12:0/2711832098 shutdown_connections 2024-03-29T11:35:59.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f741006af60 0x7f741006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f743015d9d0 0x7f74300b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74300d16a0 0x7f74301f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 --2- 172.21.15.12:0/2711832098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74300d2540 0x7f74300b69e0 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:35:59.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 -- 172.21.15.12:0/2711832098 >> 172.21.15.12:0/2711832098 conn(0x7f743008aff0 msgr2=0x7f74300b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:59.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.822+0000 7f7437800700 1 -- 172.21.15.12:0/2711832098 shutdown_connections 2024-03-29T11:35:59.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:35:59.823+0000 7f7437800700 1 -- 172.21.15.12:0/2711832098 wait complete. 2024-03-29T11:35:59.840 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-29T11:35:59.922 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:35:59.923+0000 7f3f02537240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:36:00.024 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:36:00.024+0000 7f3f02537240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:36:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.053+0000 7fe23a400700 1 Processor -- start 2024-03-29T11:36:00.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.053+0000 7fe23a400700 1 -- start start 2024-03-29T11:36:00.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.054+0000 7fe23a400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe23415ef70 0x7fe2340b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.055+0000 7fe23a400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe234161150 con 0x7fe234160d70 2024-03-29T11:36:00.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.055+0000 7fe23a400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23406dba0 con 0x7fe23406a790 2024-03-29T11:36:00.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.055+0000 7fe23a400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2340b4cd0 con 0x7fe23415ef70 2024-03-29T11:36:00.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.055+0000 7fe233400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe234160d70 0x7fe23415e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40668/0 (socket says 172.21.15.12:40668) 2024-03-29T11:36:00.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.055+0000 7fe233400700 1 -- 172.21.15.12:0/3327184243 learned_addr learned my addr 172.21.15.12:0/3327184243 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:00.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.055+0000 7fe232a00700 1 --2- 172.21.15.12:0/3327184243 >> v2:172.21.15.159:3300/0 conn(0x7fe23415ef70 0x7fe2340b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.055+0000 7fe232000700 1 -- 172.21.15.12:0/3327184243 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292143079 0 0) 0x7fe234161150 con 0x7fe234160d70 2024-03-29T11:36:00.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.055+0000 7fe232000700 1 -- 172.21.15.12:0/3327184243 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2180035b0 con 0x7fe234160d70 2024-03-29T11:36:00.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.055+0000 7fe232a00700 1 -- 172.21.15.12:0/3327184243 >> v1:172.21.15.159:6789/0 conn(0x7fe23406a790 legacy=0x7fe23406e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.055+0000 7fe232a00700 1 -- 172.21.15.12:0/3327184243 >> v1:172.21.15.12:6789/0 conn(0x7fe234160d70 legacy=0x7fe23415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.056+0000 7fe232a00700 1 -- 172.21.15.12:0/3327184243 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2340b4ee0 con 0x7fe23415ef70 2024-03-29T11:36:00.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.056+0000 7fe232a00700 1 --2- 172.21.15.12:0/3327184243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe23415ef70 0x7fe2340b3200 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fe220009840 tx=0x7fe220009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d38f73af0fad6715 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:00.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.056+0000 7fe232000700 1 -- 172.21.15.12:0/3327184243 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220011830 con 0x7fe23415ef70 2024-03-29T11:36:00.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.056+0000 7fe232000700 1 -- 172.21.15.12:0/3327184243 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe220011990 con 0x7fe23415ef70 2024-03-29T11:36:00.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.056+0000 7fe232000700 1 -- 172.21.15.12:0/3327184243 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220011d00 con 0x7fe23415ef70 2024-03-29T11:36:00.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.057+0000 7fe23a400700 1 -- 172.21.15.12:0/3327184243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe23415ef70 msgr2=0x7fe2340b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.057+0000 7fe23a400700 1 --2- 172.21.15.12:0/3327184243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe23415ef70 0x7fe2340b3200 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fe220009840 tx=0x7fe220009b50 comp rx=0 tx=0).stop 2024-03-29T11:36:00.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.057+0000 7fe23a400700 1 -- 172.21.15.12:0/3327184243 shutdown_connections 2024-03-29T11:36:00.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.057+0000 7fe23a400700 1 --2- 172.21.15.12:0/3327184243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe23415ef70 0x7fe2340b3200 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.057+0000 7fe23a400700 1 -- 172.21.15.12:0/3327184243 >> 172.21.15.12:0/3327184243 conn(0x7fe2340b5720 msgr2=0x7fe2340b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:00.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.057+0000 7fe23a400700 1 -- 172.21.15.12:0/3327184243 shutdown_connections 2024-03-29T11:36:00.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.057+0000 7fe23a400700 1 -- 172.21.15.12:0/3327184243 wait complete. 2024-03-29T11:36:00.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.059+0000 7fe23a400700 1 Processor -- start 2024-03-29T11:36:00.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.059+0000 7fe23a400700 1 -- start start 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe23a400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe23406a790 0x7fe23409b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe23a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe23415ef70 0x7fe2341f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe23a400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe234160d70 0x7fe2341f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe23a400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2341f7530 con 0x7fe23415ef70 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe23a400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2341f6fe0 con 0x7fe23406a790 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe23a400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2341f7250 con 0x7fe234160d70 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe232a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe23415ef70 0x7fe2341f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe232a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe23415ef70 0x7fe2341f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40788/0 (socket says 172.21.15.12:40788) 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe232a00700 1 -- 172.21.15.12:0/4252304808 learned_addr learned my addr 172.21.15.12:0/4252304808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe233400700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe23406a790 0x7fe23409b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe233e00700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe234160d70 0x7fe2341f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe232a00700 1 -- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe234160d70 msgr2=0x7fe2341f6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe232a00700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe234160d70 0x7fe2341f6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe232a00700 1 -- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe23406a790 msgr2=0x7fe23409b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe232a00700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe23406a790 0x7fe23409b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.060+0000 7fe232a00700 1 -- 172.21.15.12:0/4252304808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2200094a0 con 0x7fe23415ef70 2024-03-29T11:36:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.061+0000 7fe232a00700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe23415ef70 0x7fe2341f6560 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7fe220000c00 tx=0x7fe220004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.061+0000 7fe22b400700 1 -- 172.21.15.12:0/4252304808 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220004550 con 0x7fe23415ef70 2024-03-29T11:36:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.061+0000 7fe22b400700 1 -- 172.21.15.12:0/4252304808 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22001f070 con 0x7fe23415ef70 2024-03-29T11:36:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.061+0000 7fe23a400700 1 -- 172.21.15.12:0/4252304808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2341479b0 con 0x7fe23415ef70 2024-03-29T11:36:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.061+0000 7fe22b400700 1 -- 172.21.15.12:0/4252304808 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2200038c0 con 0x7fe23415ef70 2024-03-29T11:36:00.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.061+0000 7fe23a400700 1 -- 172.21.15.12:0/4252304808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe234147f70 con 0x7fe23415ef70 2024-03-29T11:36:00.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.062+0000 7fe23a400700 1 -- 172.21.15.12:0/4252304808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe234147810 con 0x7fe23415ef70 2024-03-29T11:36:00.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.063+0000 7fe22b400700 1 -- 172.21.15.12:0/4252304808 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fe220003a20 con 0x7fe23415ef70 2024-03-29T11:36:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.064+0000 7fe22b400700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe21406af60 0x7fe21406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.064+0000 7fe22b400700 1 -- 172.21.15.12:0/4252304808 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7fe220021070 con 0x7fe23415ef70 2024-03-29T11:36:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.064+0000 7fe233400700 1 -- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe21406af60 msgr2=0x7fe21406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:36:00.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.064+0000 7fe233400700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe21406af60 0x7fe21406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:36:00.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.068+0000 7fe22b400700 1 -- 172.21.15.12:0/4252304808 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe234147810 con 0x7fe23415ef70 2024-03-29T11:36:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.265+0000 7fe233400700 1 -- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe21406af60 msgr2=0x7fe21406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:36:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.265+0000 7fe233400700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe21406af60 0x7fe21406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:36:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.285+0000 7fe23a400700 1 -- 172.21.15.12:0/4252304808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe234147810 con 0x7fe23415ef70 2024-03-29T11:36:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.286+0000 7fe22b400700 1 -- 172.21.15.12:0/4252304808 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v286) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe234147810 con 0x7fe23415ef70 2024-03-29T11:36:00.286 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 -- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe21406af60 msgr2=0x7fe21406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe21406af60 0x7fe21406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 -- 172.21.15.12:0/4252304808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe23415ef70 msgr2=0x7fe2341f6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe23415ef70 0x7fe2341f6560 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7fe220000c00 tx=0x7fe220004750 comp rx=0 tx=0).stop 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 -- 172.21.15.12:0/4252304808 shutdown_connections 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe21406af60 0x7fe21406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe234160d70 0x7fe2341f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe23406a790 0x7fe23409b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 --2- 172.21.15.12:0/4252304808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe23415ef70 0x7fe2341f6560 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 -- 172.21.15.12:0/4252304808 >> 172.21.15.12:0/4252304808 conn(0x7fe2340b5720 msgr2=0x7fe23415bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 -- 172.21.15.12:0/4252304808 shutdown_connections 2024-03-29T11:36:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.290+0000 7fe23a400700 1 -- 172.21.15.12:0/4252304808 wait complete. 2024-03-29T11:36:00.311 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-29T11:36:00.464 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:00.465+0000 7fc7324ff240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:36:00.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.513+0000 7fb333c00700 1 Processor -- start 2024-03-29T11:36:00.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.513+0000 7fb333c00700 1 -- start start 2024-03-29T11:36:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.515+0000 7fb333c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15a5c0 0x7fb32c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.515+0000 7fb333c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb32c15dfe0 0x7fb32c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.515+0000 7fb333c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32c0cb310 con 0x7fb32c0ce670 2024-03-29T11:36:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.515+0000 7fb333c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c0ceb00 con 0x7fb32c15dfe0 2024-03-29T11:36:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.515+0000 7fb333c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c070f00 con 0x7fb32c15a5c0 2024-03-29T11:36:00.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.515+0000 7fb331200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15a5c0 0x7fb32c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.516+0000 7fb32be00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb32c15dfe0 0x7fb32c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.516+0000 7fb331200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15a5c0 0x7fb32c0709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.12:55728/0 (socket says 172.21.15.12:55728) 2024-03-29T11:36:00.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.516+0000 7fb331200700 1 -- 172.21.15.12:0/3198421255 learned_addr learned my addr 172.21.15.12:0/3198421255 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.516+0000 7fb331200700 1 -- 172.21.15.12:0/3198421255 >> v2:172.21.15.159:3300/0 conn(0x7fb32c15dfe0 msgr2=0x7fb32c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.516+0000 7fb331200700 1 --2- 172.21.15.12:0/3198421255 >> v2:172.21.15.159:3300/0 conn(0x7fb32c15dfe0 0x7fb32c0ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.516+0000 7fb331200700 1 -- 172.21.15.12:0/3198421255 >> v1:172.21.15.159:6789/0 conn(0x7fb32c0ce670 legacy=0x7fb32c0b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.516+0000 7fb331200700 1 -- 172.21.15.12:0/3198421255 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0cca90 con 0x7fb32c15a5c0 2024-03-29T11:36:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.516+0000 7fb331200700 1 --2- 172.21.15.12:0/3198421255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15a5c0 0x7fb32c0709c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb32000bb90 tx=0x7fb32000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f49c2dd2a0bc967 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.517+0000 7fb32b400700 1 -- 172.21.15.12:0/3198421255 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32000dd30 con 0x7fb32c15a5c0 2024-03-29T11:36:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.517+0000 7fb32b400700 1 -- 172.21.15.12:0/3198421255 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32000de90 con 0x7fb32c15a5c0 2024-03-29T11:36:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.517+0000 7fb32b400700 1 -- 172.21.15.12:0/3198421255 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320010690 con 0x7fb32c15a5c0 2024-03-29T11:36:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.517+0000 7fb333c00700 1 -- 172.21.15.12:0/3198421255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15a5c0 msgr2=0x7fb32c0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.517+0000 7fb333c00700 1 --2- 172.21.15.12:0/3198421255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15a5c0 0x7fb32c0709c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb32000bb90 tx=0x7fb32000bea0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.517+0000 7fb333c00700 1 -- 172.21.15.12:0/3198421255 shutdown_connections 2024-03-29T11:36:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.517+0000 7fb333c00700 1 --2- 172.21.15.12:0/3198421255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15a5c0 0x7fb32c0709c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.517+0000 7fb333c00700 1 --2- 172.21.15.12:0/3198421255 >> v2:172.21.15.159:3300/0 conn(0x7fb32c15dfe0 0x7fb32c0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.517+0000 7fb333c00700 1 -- 172.21.15.12:0/3198421255 >> 172.21.15.12:0/3198421255 conn(0x7fb32c0b0d70 msgr2=0x7fb32c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.517+0000 7fb333c00700 1 -- 172.21.15.12:0/3198421255 shutdown_connections 2024-03-29T11:36:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.518+0000 7fb333c00700 1 -- 172.21.15.12:0/3198421255 wait complete. 2024-03-29T11:36:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.519+0000 7fb333c00700 1 Processor -- start 2024-03-29T11:36:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.519+0000 7fb333c00700 1 -- start start 2024-03-29T11:36:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.520+0000 7fb333c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb32c0ce670 0x7fb32c14a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.520+0000 7fb333c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c15a5c0 0x7fb32c149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.520+0000 7fb333c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15dfe0 0x7fb32c14f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.520+0000 7fb333c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c15aa30 con 0x7fb32c15a5c0 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.520+0000 7fb333c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c14a140 con 0x7fb32c0ce670 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.520+0000 7fb333c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c149e60 con 0x7fb32c15dfe0 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb331200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb32c0ce670 0x7fb32c14a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb331200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb32c0ce670 0x7fb32c14a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:36812/0 (socket says 172.21.15.12:36812) 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb32be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c15a5c0 0x7fb32c149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb331200700 1 -- 172.21.15.12:0/4031462985 learned_addr learned my addr 172.21.15.12:0/4031462985 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb331c00700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15dfe0 0x7fb32c14f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb331200700 1 -- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15dfe0 msgr2=0x7fb32c14f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb331200700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15dfe0 0x7fb32c14f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb331200700 1 -- 172.21.15.12:0/4031462985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c15a5c0 msgr2=0x7fb32c149640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb331200700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c15a5c0 0x7fb32c149640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb331200700 1 -- 172.21.15.12:0/4031462985 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32000b7f0 con 0x7fb32c0ce670 2024-03-29T11:36:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.521+0000 7fb331200700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb32c0ce670 0x7fb32c14a390 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb320000c00 tx=0x7fb320004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.522+0000 7fb328c00700 1 -- 172.21.15.12:0/4031462985 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32001f070 con 0x7fb32c0ce670 2024-03-29T11:36:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.522+0000 7fb328c00700 1 -- 172.21.15.12:0/4031462985 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb320003970 con 0x7fb32c0ce670 2024-03-29T11:36:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.522+0000 7fb333c00700 1 -- 172.21.15.12:0/4031462985 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c06e460 con 0x7fb32c0ce670 2024-03-29T11:36:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.522+0000 7fb328c00700 1 -- 172.21.15.12:0/4031462985 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320003d40 con 0x7fb32c0ce670 2024-03-29T11:36:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.522+0000 7fb333c00700 1 -- 172.21.15.12:0/4031462985 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c15acd0 con 0x7fb32c0ce670 2024-03-29T11:36:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.523+0000 7fb333c00700 1 -- 172.21.15.12:0/4031462985 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c067720 con 0x7fb32c0ce670 2024-03-29T11:36:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.524+0000 7fb328c00700 1 -- 172.21.15.12:0/4031462985 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fb320010470 con 0x7fb32c0ce670 2024-03-29T11:36:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.525+0000 7fb328c00700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb30c06af60 0x7fb30c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.525+0000 7fb328c00700 1 -- 172.21.15.12:0/4031462985 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7fb32008cec0 con 0x7fb32c0ce670 2024-03-29T11:36:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.525+0000 7fb32be00700 1 -- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb30c06af60 msgr2=0x7fb30c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:36:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.525+0000 7fb32be00700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb30c06af60 0x7fb30c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:36:00.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.532+0000 7fb328c00700 1 -- 172.21.15.12:0/4031462985 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb320057b50 con 0x7fb32c0ce670 2024-03-29T11:36:00.683 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:00.684+0000 7fc7324ff240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:36:00.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.726+0000 7fb32be00700 1 -- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb30c06af60 msgr2=0x7fb30c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:36:00.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.726+0000 7fb32be00700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb30c06af60 0x7fb30c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:36:00.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.746+0000 7fb333c00700 1 -- 172.21.15.12:0/4031462985 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb32c0ae530 con 0x7fb32c0ce670 2024-03-29T11:36:00.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.747+0000 7fb328c00700 1 -- 172.21.15.12:0/4031462985 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v286) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb32005b170 con 0x7fb32c0ce670 2024-03-29T11:36:00.747 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-29T11:36:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 -- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb30c06af60 msgr2=0x7fb30c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb30c06af60 0x7fb30c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 -- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb32c0ce670 msgr2=0x7fb32c14a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb32c0ce670 0x7fb32c14a390 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb320000c00 tx=0x7fb320004750 comp rx=0 tx=0).stop 2024-03-29T11:36:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 -- 172.21.15.12:0/4031462985 shutdown_connections 2024-03-29T11:36:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb30c06af60 0x7fb30c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32c15dfe0 0x7fb32c14f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb32c0ce670 0x7fb32c14a390 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 --2- 172.21.15.12:0/4031462985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb32c15a5c0 0x7fb32c149640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 -- 172.21.15.12:0/4031462985 >> 172.21.15.12:0/4031462985 conn(0x7fb32c0b0d70 msgr2=0x7fb32c0af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:00.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 -- 172.21.15.12:0/4031462985 shutdown_connections 2024-03-29T11:36:00.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.750+0000 7fb333c00700 1 -- 172.21.15.12:0/4031462985 wait complete. 2024-03-29T11:36:00.768 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:36:00.814 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:36:00.814+0000 7f3f02537240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:36:00.893 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:00.894+0000 7fc7324ff240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:36:00.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.970+0000 7fa2eb000700 1 Processor -- start 2024-03-29T11:36:00.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.970+0000 7fa2eb000700 1 -- start start 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.972+0000 7fa2eb000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2e40ceec0 0x7fa2e41612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.972+0000 7fa2eb000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e4161800 0x7fa2e40b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.972+0000 7fa2eb000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e40d30d0 con 0x7fa2e415d640 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.972+0000 7fa2eb000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e4161c90 con 0x7fa2e40ceec0 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.972+0000 7fa2eb000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40b6840 con 0x7fa2e4161800 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.972+0000 7fa2e3e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa2e415d640 0x7fa2e40d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40678/0 (socket says 172.21.15.12:40678) 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.972+0000 7fa2e3e00700 1 -- 172.21.15.12:0/1998639397 learned_addr learned my addr 172.21.15.12:0/1998639397 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.973+0000 7fa2e9000700 1 --2- 172.21.15.12:0/1998639397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e4161800 0x7fa2e40b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.973+0000 7fa2e3400700 1 --2- 172.21.15.12:0/1998639397 >> v2:172.21.15.159:3300/0 conn(0x7fa2e40ceec0 0x7fa2e41612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.973+0000 7fa2e2a00700 1 -- 172.21.15.12:0/1998639397 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834123439 0 0) 0x7fa2e40d30d0 con 0x7fa2e415d640 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.973+0000 7fa2e2a00700 1 -- 172.21.15.12:0/1998639397 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c80035b0 con 0x7fa2e415d640 2024-03-29T11:36:00.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.973+0000 7fa2e9000700 1 -- 172.21.15.12:0/1998639397 >> v2:172.21.15.159:3300/0 conn(0x7fa2e40ceec0 msgr2=0x7fa2e41612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:00.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.973+0000 7fa2e9000700 1 --2- 172.21.15.12:0/1998639397 >> v2:172.21.15.159:3300/0 conn(0x7fa2e40ceec0 0x7fa2e41612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.973+0000 7fa2e9000700 1 -- 172.21.15.12:0/1998639397 >> v1:172.21.15.12:6789/0 conn(0x7fa2e415d640 legacy=0x7fa2e40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.973+0000 7fa2e9000700 1 -- 172.21.15.12:0/1998639397 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40d2b30 con 0x7fa2e4161800 2024-03-29T11:36:00.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.973+0000 7fa2e9000700 1 --2- 172.21.15.12:0/1998639397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e4161800 0x7fa2e40b6cd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009d90 tx=0x7fa2dc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff11ddabdf9cef7e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:00.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.974+0000 7fa2e2a00700 1 -- 172.21.15.12:0/1998639397 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc00e070 con 0x7fa2e4161800 2024-03-29T11:36:00.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.974+0000 7fa2e2a00700 1 -- 172.21.15.12:0/1998639397 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc00bb20 con 0x7fa2e4161800 2024-03-29T11:36:00.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.974+0000 7fa2e2a00700 1 -- 172.21.15.12:0/1998639397 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc012430 con 0x7fa2e4161800 2024-03-29T11:36:00.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.975+0000 7fa2eb000700 1 -- 172.21.15.12:0/1998639397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e4161800 msgr2=0x7fa2e40b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.975+0000 7fa2eb000700 1 --2- 172.21.15.12:0/1998639397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e4161800 0x7fa2e40b6cd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009d90 tx=0x7fa2dc00b330 comp rx=0 tx=0).stop 2024-03-29T11:36:00.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.975+0000 7fa2eb000700 1 -- 172.21.15.12:0/1998639397 shutdown_connections 2024-03-29T11:36:00.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.975+0000 7fa2eb000700 1 --2- 172.21.15.12:0/1998639397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e4161800 0x7fa2e40b6cd0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.975+0000 7fa2eb000700 1 --2- 172.21.15.12:0/1998639397 >> v2:172.21.15.159:3300/0 conn(0x7fa2e40ceec0 0x7fa2e41612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.975+0000 7fa2eb000700 1 -- 172.21.15.12:0/1998639397 >> 172.21.15.12:0/1998639397 conn(0x7fa2e408aff0 msgr2=0x7fa2e40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:00.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.975+0000 7fa2eb000700 1 -- 172.21.15.12:0/1998639397 shutdown_connections 2024-03-29T11:36:00.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.975+0000 7fa2eb000700 1 -- 172.21.15.12:0/1998639397 wait complete. 2024-03-29T11:36:00.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.977+0000 7fa2eb000700 1 Processor -- start 2024-03-29T11:36:00.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.977+0000 7fa2eb000700 1 -- start start 2024-03-29T11:36:00.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2eb000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e40ceec0 0x7fa2e40a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2eb000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2e415d640 0x7fa2e409e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2eb000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2e4161800 0x7fa2e4151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2eb000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40a0d10 con 0x7fa2e415d640 2024-03-29T11:36:00.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2eb000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e409e8d0 con 0x7fa2e4161800 2024-03-29T11:36:00.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2eb000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e41524d0 con 0x7fa2e40ceec0 2024-03-29T11:36:00.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2e3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2e415d640 0x7fa2e409e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2e3e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e40ceec0 0x7fa2e40a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2e3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2e415d640 0x7fa2e409e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40812/0 (socket says 172.21.15.12:40812) 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2e3400700 1 -- 172.21.15.12:0/1501096950 learned_addr learned my addr 172.21.15.12:0/1501096950 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2e9000700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2e4161800 0x7fa2e4151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2e3400700 1 -- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e40ceec0 msgr2=0x7fa2e40a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.978+0000 7fa2e3400700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e40ceec0 0x7fa2e40a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2e3400700 1 -- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2e4161800 msgr2=0x7fa2e4151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2e3400700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2e4161800 0x7fa2e4151f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2e3400700 1 -- 172.21.15.12:0/1501096950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc009a70 con 0x7fa2e415d640 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2e9000700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2e4161800 0x7fa2e4151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2e3e00700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e40ceec0 0x7fa2e40a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2e3400700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2e415d640 0x7fa2e409e390 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000e870 tx=0x7fa2d000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2dbe00700 1 -- 172.21.15.12:0/1501096950 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0009590 con 0x7fa2e415d640 2024-03-29T11:36:00.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2dbe00700 1 -- 172.21.15.12:0/1501096950 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d0013680 con 0x7fa2e415d640 2024-03-29T11:36:00.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2dbe00700 1 -- 172.21.15.12:0/1501096950 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00139f0 con 0x7fa2e415d640 2024-03-29T11:36:00.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2eb000700 1 -- 172.21.15.12:0/1501096950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e40c8250 con 0x7fa2e415d640 2024-03-29T11:36:00.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.979+0000 7fa2eb000700 1 -- 172.21.15.12:0/1501096950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e40c84f0 con 0x7fa2e415d640 2024-03-29T11:36:00.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.980+0000 7fa2eb000700 1 -- 172.21.15.12:0/1501096950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4067720 con 0x7fa2e415d640 2024-03-29T11:36:00.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.982+0000 7fa2dbe00700 1 -- 172.21.15.12:0/1501096950 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fa2d0013b50 con 0x7fa2e415d640 2024-03-29T11:36:00.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.983+0000 7fa2dbe00700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2c406af90 0x7fa2c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:00.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.983+0000 7fa2dbe00700 1 -- 172.21.15.12:0/1501096950 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7fa2d0089eb0 con 0x7fa2e415d640 2024-03-29T11:36:00.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.983+0000 7fa2e3e00700 1 -- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2c406af90 msgr2=0x7fa2c406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:36:00.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.983+0000 7fa2e3e00700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2c406af90 0x7fa2c406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:36:00.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:00.986+0000 7fa2dbe00700 1 -- 172.21.15.12:0/1501096950 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2d0054bc0 con 0x7fa2e415d640 2024-03-29T11:36:01.048 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:36:01.048+0000 7f3f02537240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:36:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.184+0000 7fa2e3e00700 1 -- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2c406af90 msgr2=0x7fa2c406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:36:01.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.184+0000 7fa2e3e00700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2c406af90 0x7fa2c406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:36:01.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.254+0000 7fa2eb000700 1 -- 172.21.15.12:0/1501096950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2e40b5c70 con 0x7fa2e415d640 2024-03-29T11:36:01.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.255+0000 7fa2dbe00700 1 -- 172.21.15.12:0/1501096950 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4747 (secure 0 0 0) 0x7fa2d0017090 con 0x7fa2e415d640 2024-03-29T11:36:01.254 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:01.255 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":146,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12538,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/350949303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":350949303},{"type":"v1","addr":"172.21.15.159:6836","nonce":350949303}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1576953466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1576953466},{"type":"v1","addr":"172.21.15.159:6837","nonce":1576953466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/30523791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":30523791},{"type":"v1","addr":"172.21.15.12:6835","nonce":30523791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:34:19.924674+0000","modified":"2024-03-29T11:34:23.413936+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13125":{"gid":13125,"name":"a","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/2115072104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2115072104},{"type":"v1","addr":"172.21.15.12:6834","nonce":2115072104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-29T11:36:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.257+0000 7fa2eb000700 1 -- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2c406af90 msgr2=0x7fa2c406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.257+0000 7fa2eb000700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2c406af90 0x7fa2c406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.257+0000 7fa2eb000700 1 -- 172.21.15.12:0/1501096950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2e415d640 msgr2=0x7fa2e409e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.257+0000 7fa2eb000700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2e415d640 0x7fa2e409e390 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000e870 tx=0x7fa2d000eb80 comp rx=0 tx=0).stop 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.257+0000 7fa2eb000700 1 -- 172.21.15.12:0/1501096950 shutdown_connections 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.257+0000 7fa2eb000700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa2c406af90 0x7fa2c406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.257+0000 7fa2eb000700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2e4161800 0x7fa2e4151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.257+0000 7fa2eb000700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2e40ceec0 0x7fa2e40a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.257+0000 7fa2eb000700 1 --2- 172.21.15.12:0/1501096950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2e415d640 0x7fa2e409e390 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.258+0000 7fa2eb000700 1 -- 172.21.15.12:0/1501096950 >> 172.21.15.12:0/1501096950 conn(0x7fa2e408aff0 msgr2=0x7fa2e406d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.258+0000 7fa2eb000700 1 -- 172.21.15.12:0/1501096950 shutdown_connections 2024-03-29T11:36:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.258+0000 7fa2eb000700 1 -- 172.21.15.12:0/1501096950 wait complete. 2024-03-29T11:36:01.259 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 146 2024-03-29T11:36:01.271 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:36:01.271+0000 7f3f02537240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:36:01.275 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:01.377 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:01.378+0000 7fc7324ff240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:36:01.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.477+0000 7f72e4e00700 1 Processor -- start 2024-03-29T11:36:01.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.477+0000 7f72e4e00700 1 -- start start 2024-03-29T11:36:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.478+0000 7f72e4e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e015b5d0 0x7f72e00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.478+0000 7f72e4e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e015fd50 con 0x7f72e00cb3f0 2024-03-29T11:36:01.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.478+0000 7f72e4e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e015f140 con 0x7f72e00ac7d0 2024-03-29T11:36:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.478+0000 7f72e4e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e015b9b0 con 0x7f72e015b5d0 2024-03-29T11:36:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.479+0000 7f72dd400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f72e00cb3f0 0x7f72e00ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40688/0 (socket says 172.21.15.12:40688) 2024-03-29T11:36:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.479+0000 7f72dd400700 1 -- 172.21.15.12:0/44829070 learned_addr learned my addr 172.21.15.12:0/44829070 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:01.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.479+0000 7f72dde00700 1 --2- 172.21.15.12:0/44829070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e015b5d0 0x7f72e00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:01.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.479+0000 7f72dca00700 1 -- 172.21.15.12:0/44829070 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927153230 0 0) 0x7f72e015fd50 con 0x7f72e00cb3f0 2024-03-29T11:36:01.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.479+0000 7f72dca00700 1 -- 172.21.15.12:0/44829070 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c40035b0 con 0x7f72e00cb3f0 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.479+0000 7f72dca00700 1 -- 172.21.15.12:0/44829070 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034575645 0 0) 0x7f72e015f140 con 0x7f72e00ac7d0 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.479+0000 7f72dca00700 1 -- 172.21.15.12:0/44829070 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e015fd50 con 0x7f72e00ac7d0 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.479+0000 7f72dde00700 1 -- 172.21.15.12:0/44829070 >> v1:172.21.15.159:6789/0 conn(0x7f72e00ac7d0 legacy=0x7f72e006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.479+0000 7f72dde00700 1 -- 172.21.15.12:0/44829070 >> v1:172.21.15.12:6789/0 conn(0x7f72e00cb3f0 legacy=0x7f72e00ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.479+0000 7f72dde00700 1 -- 172.21.15.12:0/44829070 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e015f420 con 0x7f72e015b5d0 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.480+0000 7f72dde00700 1 --2- 172.21.15.12:0/44829070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e015b5d0 0x7f72e00ae3f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009870 tx=0x7f72d0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1729eb2b25a9e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.480+0000 7f72dca00700 1 -- 172.21.15.12:0/44829070 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0011830 con 0x7f72e015b5d0 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.480+0000 7f72dca00700 1 -- 172.21.15.12:0/44829070 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d0011990 con 0x7f72e015b5d0 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.480+0000 7f72dca00700 1 -- 172.21.15.12:0/44829070 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0011d20 con 0x7f72e015b5d0 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.481+0000 7f72e4e00700 1 -- 172.21.15.12:0/44829070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e015b5d0 msgr2=0x7f72e00ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.481+0000 7f72e4e00700 1 --2- 172.21.15.12:0/44829070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e015b5d0 0x7f72e00ae3f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009870 tx=0x7f72d0009b80 comp rx=0 tx=0).stop 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.481+0000 7f72e4e00700 1 -- 172.21.15.12:0/44829070 shutdown_connections 2024-03-29T11:36:01.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.481+0000 7f72e4e00700 1 --2- 172.21.15.12:0/44829070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e015b5d0 0x7f72e00ae3f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.481+0000 7f72e4e00700 1 -- 172.21.15.12:0/44829070 >> 172.21.15.12:0/44829070 conn(0x7f72e00cf190 msgr2=0x7f72e00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.481+0000 7f72e4e00700 1 -- 172.21.15.12:0/44829070 shutdown_connections 2024-03-29T11:36:01.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.481+0000 7f72e4e00700 1 -- 172.21.15.12:0/44829070 wait complete. 2024-03-29T11:36:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.484+0000 7f72e4e00700 1 Processor -- start 2024-03-29T11:36:01.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.484+0000 7f72e4e00700 1 -- start start 2024-03-29T11:36:01.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.485+0000 7f72e4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72e00ac7d0 0x7f72e00b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.485+0000 7f72e4e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72e015b5d0 0x7f72e01f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.485+0000 7f72e4e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e01f08f0 0x7f72e0152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.485+0000 7f72e4e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e014cdd0 con 0x7f72e00ac7d0 2024-03-29T11:36:01.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.485+0000 7f72e4e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e01f0f40 con 0x7f72e015b5d0 2024-03-29T11:36:01.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.485+0000 7f72e4e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e01f1220 con 0x7f72e01f08f0 2024-03-29T11:36:01.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72e00ac7d0 0x7f72e00b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:01.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72e00ac7d0 0x7f72e00b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40816/0 (socket says 172.21.15.12:40816) 2024-03-29T11:36:01.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dde00700 1 -- 172.21.15.12:0/173099636 learned_addr learned my addr 172.21.15.12:0/173099636 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:01.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dd400700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72e015b5d0 0x7f72e01f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:01.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72de800700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e01f08f0 0x7f72e0152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:01.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dde00700 1 -- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e01f08f0 msgr2=0x7f72e0152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dde00700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e01f08f0 0x7f72e0152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dde00700 1 -- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72e015b5d0 msgr2=0x7f72e01f03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dde00700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72e015b5d0 0x7f72e01f03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dde00700 1 -- 172.21.15.12:0/173099636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d00094a0 con 0x7f72e00ac7d0 2024-03-29T11:36:01.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dd400700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72e015b5d0 0x7f72e01f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:01.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.486+0000 7f72dde00700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72e00ac7d0 0x7f72e00b4220 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009bc0 tx=0x7f72d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:01.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.487+0000 7f72d6000700 1 -- 172.21.15.12:0/173099636 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d001a860 con 0x7f72e00ac7d0 2024-03-29T11:36:01.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.487+0000 7f72d6000700 1 -- 172.21.15.12:0/173099636 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d001a9c0 con 0x7f72e00ac7d0 2024-03-29T11:36:01.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.487+0000 7f72e4e00700 1 -- 172.21.15.12:0/173099636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e01f1500 con 0x7f72e00ac7d0 2024-03-29T11:36:01.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.487+0000 7f72d6000700 1 -- 172.21.15.12:0/173099636 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d001ad70 con 0x7f72e00ac7d0 2024-03-29T11:36:01.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.487+0000 7f72e4e00700 1 -- 172.21.15.12:0/173099636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e006b430 con 0x7f72e00ac7d0 2024-03-29T11:36:01.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.488+0000 7f72e4e00700 1 -- 172.21.15.12:0/173099636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e01f0cd0 con 0x7f72e00ac7d0 2024-03-29T11:36:01.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.489+0000 7f72d6000700 1 -- 172.21.15.12:0/173099636 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f72d0022070 con 0x7f72e00ac7d0 2024-03-29T11:36:01.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.491+0000 7f72d6000700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f72c006af60 0x7f72c006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.491+0000 7f72d6000700 1 -- 172.21.15.12:0/173099636 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7f72d008a980 con 0x7f72e00ac7d0 2024-03-29T11:36:01.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.491+0000 7f72dd400700 1 -- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f72c006af60 msgr2=0x7f72c006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:36:01.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.492+0000 7f72dd400700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f72c006af60 0x7f72c006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:36:01.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.495+0000 7f72d6000700 1 -- 172.21.15.12:0/173099636 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72d00557c0 con 0x7f72e00ac7d0 2024-03-29T11:36:01.589 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:01.590+0000 7fc7324ff240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:36:01.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.693+0000 7f72dd400700 1 -- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f72c006af60 msgr2=0x7f72c006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:36:01.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.693+0000 7f72dd400700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f72c006af60 0x7f72c006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:36:01.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.713+0000 7f72e4e00700 1 -- 172.21.15.12:0/173099636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72e0067720 con 0x7f72e00ac7d0 2024-03-29T11:36:01.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.714+0000 7f72d6000700 1 -- 172.21.15.12:0/173099636 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v286) v1 ==== 74+0+16721 (secure 0 0 0) 0x7f72d0015090 con 0x7f72e00ac7d0 2024-03-29T11:36:01.714 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:01.714 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":286,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:35:38.461906+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:34:18.360241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-03-29T11:34:18.903400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"283","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.d","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/2115167177":"2024-03-30T11:34:37.193903+0000","172.21.15.159:6833/62646":"2024-03-30T11:34:37.193903+0000","172.21.15.159:6832/62646":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/411770029":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/3938940279":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/2298284405":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/1795874409":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/547605341":"2024-03-30T11:34:37.193903+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:01.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.716+0000 7f72e4e00700 1 -- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f72c006af60 msgr2=0x7f72c006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.716+0000 7f72e4e00700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f72c006af60 0x7f72c006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.716+0000 7f72e4e00700 1 -- 172.21.15.12:0/173099636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72e00ac7d0 msgr2=0x7f72e00b4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.716+0000 7f72e4e00700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72e00ac7d0 0x7f72e00b4220 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009bc0 tx=0x7f72d0004750 comp rx=0 tx=0).stop 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.717+0000 7f72e4e00700 1 -- 172.21.15.12:0/173099636 shutdown_connections 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.717+0000 7f72e4e00700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f72c006af60 0x7f72c006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.717+0000 7f72e4e00700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f72e01f08f0 0x7f72e0152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.717+0000 7f72e4e00700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72e00ac7d0 0x7f72e00b4220 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.717+0000 7f72e4e00700 1 --2- 172.21.15.12:0/173099636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f72e015b5d0 0x7f72e01f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.717+0000 7f72e4e00700 1 -- 172.21.15.12:0/173099636 >> 172.21.15.12:0/173099636 conn(0x7f72e00cf190 msgr2=0x7f72e006cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.717+0000 7f72e4e00700 1 -- 172.21.15.12:0/173099636 shutdown_connections 2024-03-29T11:36:01.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.717+0000 7f72e4e00700 1 -- 172.21.15.12:0/173099636 wait complete. 2024-03-29T11:36:01.735 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T11:36:01.735 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:36:01.803 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:36:01.804+0000 7f3f02537240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:36:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.936+0000 7fe7c6600700 1 Processor -- start 2024-03-29T11:36:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.936+0000 7fe7c6600700 1 -- start start 2024-03-29T11:36:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.937+0000 7fe7c6600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe7c0160c90 0x7fe7c00b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.937+0000 7fe7c6600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7c00718e0 0x7fe7c00b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.937+0000 7fe7c6600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c00ac4d0 con 0x7fe7c00709c0 2024-03-29T11:36:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.937+0000 7fe7c6600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c0071e50 con 0x7fe7c00718e0 2024-03-29T11:36:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.937+0000 7fe7c6600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c0072130 con 0x7fe7c0160c90 2024-03-29T11:36:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.937+0000 7fe7bea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7c00718e0 0x7fe7c00b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.938+0000 7fe7bea00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7c00718e0 0x7fe7c00b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40832/0 (socket says 172.21.15.12:40832) 2024-03-29T11:36:01.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.938+0000 7fe7bea00700 1 -- 172.21.15.12:0/1896001148 learned_addr learned my addr 172.21.15.12:0/1896001148 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:01.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.938+0000 7fe7bf400700 1 --2- 172.21.15.12:0/1896001148 >> v2:172.21.15.159:3300/0 conn(0x7fe7c0160c90 0x7fe7c00b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.938+0000 7fe7bea00700 1 -- 172.21.15.12:0/1896001148 >> v2:172.21.15.159:3300/0 conn(0x7fe7c0160c90 msgr2=0x7fe7c00b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.938+0000 7fe7bea00700 1 --2- 172.21.15.12:0/1896001148 >> v2:172.21.15.159:3300/0 conn(0x7fe7c0160c90 0x7fe7c00b20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.938+0000 7fe7bea00700 1 -- 172.21.15.12:0/1896001148 >> v1:172.21.15.12:6789/0 conn(0x7fe7c00709c0 legacy=0x7fe7c006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.938+0000 7fe7bea00700 1 -- 172.21.15.12:0/1896001148 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c0072410 con 0x7fe7c00718e0 2024-03-29T11:36:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.938+0000 7fe7bea00700 1 --2- 172.21.15.12:0/1896001148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c00718e0 0x7fe7c00b2620 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0070090 tx=0x7fe7ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3185f0f6db06c27f server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.938+0000 7fe7be000700 1 -- 172.21.15.12:0/1896001148 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac011830 con 0x7fe7c00718e0 2024-03-29T11:36:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.939+0000 7fe7be000700 1 -- 172.21.15.12:0/1896001148 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ac011990 con 0x7fe7c00718e0 2024-03-29T11:36:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.939+0000 7fe7be000700 1 -- 172.21.15.12:0/1896001148 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac011d20 con 0x7fe7c00718e0 2024-03-29T11:36:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.939+0000 7fe7c6600700 1 -- 172.21.15.12:0/1896001148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c00718e0 msgr2=0x7fe7c00b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.939+0000 7fe7c6600700 1 --2- 172.21.15.12:0/1896001148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c00718e0 0x7fe7c00b2620 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0070090 tx=0x7fe7ac009bd0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.939+0000 7fe7c6600700 1 -- 172.21.15.12:0/1896001148 shutdown_connections 2024-03-29T11:36:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.939+0000 7fe7c6600700 1 --2- 172.21.15.12:0/1896001148 >> v2:172.21.15.159:3300/0 conn(0x7fe7c0160c90 0x7fe7c00b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.939+0000 7fe7c6600700 1 --2- 172.21.15.12:0/1896001148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c00718e0 0x7fe7c00b2620 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.939+0000 7fe7c6600700 1 -- 172.21.15.12:0/1896001148 >> 172.21.15.12:0/1896001148 conn(0x7fe7c00cf190 msgr2=0x7fe7c00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.939+0000 7fe7c6600700 1 -- 172.21.15.12:0/1896001148 shutdown_connections 2024-03-29T11:36:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.939+0000 7fe7c6600700 1 -- 172.21.15.12:0/1896001148 wait complete. 2024-03-29T11:36:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.941+0000 7fe7c6600700 1 Processor -- start 2024-03-29T11:36:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.941+0000 7fe7c6600700 1 -- start start 2024-03-29T11:36:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.942+0000 7fe7c6600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7c00709c0 0x7fe7c01f2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.942+0000 7fe7c6600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c00718e0 0x7fe7c01f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.942+0000 7fe7c6600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c0160c90 0x7fe7c01f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.942+0000 7fe7c6600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c01f4a50 con 0x7fe7c0160c90 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.942+0000 7fe7c6600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c01f3720 con 0x7fe7c00709c0 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.942+0000 7fe7c6600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c01f3e60 con 0x7fe7c00718e0 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c0160c90 0x7fe7c01f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bf400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7c00709c0 0x7fe7c01f2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c0160c90 0x7fe7c01f6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40842/0 (socket says 172.21.15.12:40842) 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bfe00700 1 -- 172.21.15.12:0/3780944873 learned_addr learned my addr 172.21.15.12:0/3780944873 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bea00700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c00718e0 0x7fe7c01f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bfe00700 1 -- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c00718e0 msgr2=0x7fe7c01f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bfe00700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c00718e0 0x7fe7c01f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bfe00700 1 -- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7c00709c0 msgr2=0x7fe7c01f2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bfe00700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7c00709c0 0x7fe7c01f2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bfe00700 1 -- 172.21.15.12:0/3780944873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac009550 con 0x7fe7c0160c90 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7bfe00700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c0160c90 0x7fe7c01f6290 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800cc50 tx=0x7fe7b800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.943+0000 7fe7b7400700 1 -- 172.21.15.12:0/3780944873 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8003c00 con 0x7fe7c0160c90 2024-03-29T11:36:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.944+0000 7fe7b7400700 1 -- 172.21.15.12:0/3780944873 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b8003d60 con 0x7fe7c0160c90 2024-03-29T11:36:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.944+0000 7fe7c6600700 1 -- 172.21.15.12:0/3780944873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c00adeb0 con 0x7fe7c0160c90 2024-03-29T11:36:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.944+0000 7fe7b7400700 1 -- 172.21.15.12:0/3780944873 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b800aab0 con 0x7fe7c0160c90 2024-03-29T11:36:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.944+0000 7fe7c6600700 1 -- 172.21.15.12:0/3780944873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c01f4390 con 0x7fe7c0160c90 2024-03-29T11:36:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.945+0000 7fe7c6600700 1 -- 172.21.15.12:0/3780944873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c01f4b90 con 0x7fe7c0160c90 2024-03-29T11:36:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.947+0000 7fe7b7400700 1 -- 172.21.15.12:0/3780944873 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 163) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fe7b800ac10 con 0x7fe7c0160c90 2024-03-29T11:36:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.947+0000 7fe7b7400700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a006af60 0x7fe7a006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.948+0000 7fe7b7400700 1 -- 172.21.15.12:0/3780944873 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6154+0+0 (secure 0 0 0) 0x7fe7b80898c0 con 0x7fe7c0160c90 2024-03-29T11:36:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.948+0000 7fe7bf400700 1 -- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a006af60 msgr2=0x7fe7a006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:36:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.948+0000 7fe7bf400700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a006af60 0x7fe7a006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-29T11:36:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:01.952+0000 7fe7b7400700 1 -- 172.21.15.12:0/3780944873 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe7c01f4b90 con 0x7fe7c0160c90 2024-03-29T11:36:02.032 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:36:02.033+0000 7f3f02537240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:36:02.035 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.036+0000 7fc7324ff240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:36:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.149+0000 7fe7bf400700 1 -- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a006af60 msgr2=0x7fe7a006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.159:6832/62646 2024-03-29T11:36:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.149+0000 7fe7bf400700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a006af60 0x7fe7a006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-29T11:36:02.149 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.151+0000 7fc7324ff240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:36:02.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.163+0000 7fe7b7400700 1 -- 172.21.15.12:0/3780944873 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 164) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fe7b8054960 con 0x7fe7c0160c90 2024-03-29T11:36:02.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.163+0000 7fe7b7400700 1 -- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a006af60 msgr2=0x7fe7a006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:02.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.163+0000 7fe7b7400700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a006af60 0x7fe7a006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.259+0000 7fe7c6600700 1 -- 172.21.15.12:0/3780944873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7c01f4b90 con 0x7fe7c0160c90 2024-03-29T11:36:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.260+0000 7fe7b7400700 1 -- 172.21.15.12:0/3780944873 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4747 (secure 0 0 0) 0x7fe7b806fe60 con 0x7fe7c0160c90 2024-03-29T11:36:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:02.260 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":146,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12538,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/350949303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":350949303},{"type":"v1","addr":"172.21.15.159:6836","nonce":350949303}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1576953466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1576953466},{"type":"v1","addr":"172.21.15.159:6837","nonce":1576953466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/30523791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":30523791},{"type":"v1","addr":"172.21.15.12:6835","nonce":30523791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:34:19.924674+0000","modified":"2024-03-29T11:34:23.413936+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13125":{"gid":13125,"name":"a","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/2115072104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2115072104},{"type":"v1","addr":"172.21.15.12:6834","nonce":2115072104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-29T11:36:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.262+0000 7fe7c6600700 1 -- 172.21.15.12:0/3780944873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c0160c90 msgr2=0x7fe7c01f6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.262+0000 7fe7c6600700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c0160c90 0x7fe7c01f6290 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800cc50 tx=0x7fe7b800cf60 comp rx=0 tx=0).stop 2024-03-29T11:36:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.263+0000 7fe7c6600700 1 -- 172.21.15.12:0/3780944873 shutdown_connections 2024-03-29T11:36:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.263+0000 7fe7c6600700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe7a006af60 0x7fe7a006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.263+0000 7fe7c6600700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7c0160c90 0x7fe7c01f6290 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.263+0000 7fe7c6600700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7c00709c0 0x7fe7c01f2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.263+0000 7fe7c6600700 1 --2- 172.21.15.12:0/3780944873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7c00718e0 0x7fe7c01f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.263+0000 7fe7c6600700 1 -- 172.21.15.12:0/3780944873 >> 172.21.15.12:0/3780944873 conn(0x7fe7c00cf190 msgr2=0x7fe7c00736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.263+0000 7fe7c6600700 1 -- 172.21.15.12:0/3780944873 shutdown_connections 2024-03-29T11:36:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.263+0000 7fe7c6600700 1 -- 172.21.15.12:0/3780944873 wait complete. 2024-03-29T11:36:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 146 2024-03-29T11:36:02.272 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.274+0000 7fc6a1a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:36:02.273 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.274+0000 7fc6a1a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:36:02.273 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.274+0000 7fc6a1a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:36:02.273 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.274+0000 7fc6a1a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:36:02.273 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.274+0000 7fc6a1a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:36:02.273 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.274+0000 7fc6a4c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:36:02.273 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.274+0000 7fc6a4c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:36:02.273 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.274+0000 7fc6a4c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:36:02.273 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.274+0000 7fc6a4c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:36:02.274 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-29T11:36:02.274+0000 7fc6a4c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-29T11:36:02.284 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:36:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.492+0000 7f9362000700 1 Processor -- start 2024-03-29T11:36:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.492+0000 7f9362000700 1 -- start start 2024-03-29T11:36:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.493+0000 7f9362000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f935c0d0a40 0x7f935c0b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.494+0000 7f9362000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935c0d0ef0 con 0x7f935c0d05e0 2024-03-29T11:36:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.494+0000 7f9362000700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935c0d24e0 con 0x7f935c15da50 2024-03-29T11:36:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.494+0000 7f9362000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c0b95d0 con 0x7f935c0d0a40 2024-03-29T11:36:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.494+0000 7f935aa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f935c0d05e0 0x7f935c0b6990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40710/0 (socket says 172.21.15.12:40710) 2024-03-29T11:36:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.494+0000 7f935aa00700 1 -- 172.21.15.12:0/349107057 learned_addr learned my addr 172.21.15.12:0/349107057 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.494+0000 7f935be00700 1 --2- 172.21.15.12:0/349107057 >> v2:172.21.15.12:3300/0 conn(0x7f935c0d0a40 0x7f935c0b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.494+0000 7f935be00700 1 -- 172.21.15.12:0/349107057 >> v1:172.21.15.159:6789/0 conn(0x7f935c15da50 legacy=0x7f935c0ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.494+0000 7f935be00700 1 -- 172.21.15.12:0/349107057 >> v1:172.21.15.12:6789/0 conn(0x7f935c0d05e0 legacy=0x7f935c0b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.494+0000 7f935be00700 1 -- 172.21.15.12:0/349107057 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0d27c0 con 0x7f935c0d0a40 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.495+0000 7f935be00700 1 --2- 172.21.15.12:0/349107057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c0d0a40 0x7f935c0b70a0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f9354009d90 tx=0x7f935400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c62f62d38c2657 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.495+0000 7f935a000700 1 -- 172.21.15.12:0/349107057 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935400e070 con 0x7f935c0d0a40 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.495+0000 7f935a000700 1 -- 172.21.15.12:0/349107057 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935400bbd0 con 0x7f935c0d0a40 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.495+0000 7f935a000700 1 -- 172.21.15.12:0/349107057 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354012430 con 0x7f935c0d0a40 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.495+0000 7f9362000700 1 -- 172.21.15.12:0/349107057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c0d0a40 msgr2=0x7f935c0b70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.495+0000 7f9362000700 1 --2- 172.21.15.12:0/349107057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c0d0a40 0x7f935c0b70a0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f9354009d90 tx=0x7f935400b330 comp rx=0 tx=0).stop 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.496+0000 7f9362000700 1 -- 172.21.15.12:0/349107057 shutdown_connections 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.496+0000 7f9362000700 1 --2- 172.21.15.12:0/349107057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c0d0a40 0x7f935c0b70a0 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.496+0000 7f9362000700 1 -- 172.21.15.12:0/349107057 >> 172.21.15.12:0/349107057 conn(0x7f935c0666a0 msgr2=0x7f935c066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.496+0000 7f9362000700 1 -- 172.21.15.12:0/349107057 shutdown_connections 2024-03-29T11:36:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.496+0000 7f9362000700 1 -- 172.21.15.12:0/349107057 wait complete. 2024-03-29T11:36:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.497+0000 7f9362000700 1 Processor -- start 2024-03-29T11:36:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.498+0000 7f9362000700 1 -- start start 2024-03-29T11:36:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.498+0000 7f9362000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f935c0d05e0 0x7f935c1f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.498+0000 7f9362000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f935c0d0a40 0x7f935c1f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.498+0000 7f9362000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c15da50 0x7f935c0c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.498+0000 7f9362000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c1f67d0 con 0x7f935c15da50 2024-03-29T11:36:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.498+0000 7f9362000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0c92f0 con 0x7f935c0d05e0 2024-03-29T11:36:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f9362000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0c8d30 con 0x7f935c0d0a40 2024-03-29T11:36:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f935be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c15da50 0x7f935c0c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f935be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c15da50 0x7f935c0c83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40862/0 (socket says 172.21.15.12:40862) 2024-03-29T11:36:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f935be00700 1 -- 172.21.15.12:0/3139998287 learned_addr learned my addr 172.21.15.12:0/3139998287 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f935b400700 1 --2- 172.21.15.12:0/3139998287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f935c0d05e0 0x7f935c1f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f935aa00700 1 --2- 172.21.15.12:0/3139998287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f935c0d0a40 0x7f935c1f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f935be00700 1 -- 172.21.15.12:0/3139998287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f935c0d0a40 msgr2=0x7f935c1f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f935be00700 1 --2- 172.21.15.12:0/3139998287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f935c0d0a40 0x7f935c1f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f935be00700 1 -- 172.21.15.12:0/3139998287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f935c0d05e0 msgr2=0x7f935c1f45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f935be00700 1 --2- 172.21.15.12:0/3139998287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f935c0d05e0 0x7f935c1f45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.499+0000 7f935be00700 1 -- 172.21.15.12:0/3139998287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9354009a70 con 0x7f935c15da50 2024-03-29T11:36:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.500+0000 7f935be00700 1 --2- 172.21.15.12:0/3139998287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c15da50 0x7f935c0c83c0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f9354014040 tx=0x7f9354004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.500+0000 7f9353400700 1 -- 172.21.15.12:0/3139998287 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354012d20 con 0x7f935c15da50 2024-03-29T11:36:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.500+0000 7f9353400700 1 -- 172.21.15.12:0/3139998287 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9354020070 con 0x7f935c15da50 2024-03-29T11:36:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.500+0000 7f9353400700 1 -- 172.21.15.12:0/3139998287 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354012ea0 con 0x7f935c15da50 2024-03-29T11:36:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.500+0000 7f9362000700 1 -- 172.21.15.12:0/3139998287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c0c9010 con 0x7f935c15da50 2024-03-29T11:36:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.500+0000 7f9362000700 1 -- 172.21.15.12:0/3139998287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c150a90 con 0x7f935c15da50 2024-03-29T11:36:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.501+0000 7f9362000700 1 -- 172.21.15.12:0/3139998287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c150610 con 0x7f935c15da50 2024-03-29T11:36:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.502+0000 7f9353400700 1 -- 172.21.15.12:0/3139998287 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 164) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f9354004280 con 0x7f935c15da50 2024-03-29T11:36:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.503+0000 7f9353400700 1 -- 172.21.15.12:0/3139998287 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 6541+0+0 (secure 0 0 0) 0x7f9354022070 con 0x7f935c15da50 2024-03-29T11:36:02.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.508+0000 7f9353400700 1 -- 172.21.15.12:0/3139998287 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f935c150610 con 0x7f935c15da50 2024-03-29T11:36:02.550 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:36:02.550+0000 7f3f02537240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:36:02.675 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-03-29T11:36:02.675+0000 7f3f02537240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:36:02.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.798+0000 7f9362000700 1 -- 172.21.15.12:0/3139998287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f935c150610 con 0x7f935c15da50 2024-03-29T11:36:02.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.798+0000 7f9353400700 1 -- 172.21.15.12:0/3139998287 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4747 (secure 0 0 0) 0x7f935c150610 con 0x7f935c15da50 2024-03-29T11:36:02.798 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:02.798 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":146,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12538,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/350949303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":350949303},{"type":"v1","addr":"172.21.15.159:6836","nonce":350949303}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1576953466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1576953466},{"type":"v1","addr":"172.21.15.159:6837","nonce":1576953466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/30523791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":30523791},{"type":"v1","addr":"172.21.15.12:6835","nonce":30523791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-29T11:34:19.924674+0000","modified":"2024-03-29T11:34:23.413936+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13125":{"gid":13125,"name":"a","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.12:6834/2115072104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2115072104},{"type":"v1","addr":"172.21.15.12:6834","nonce":2115072104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-29T11:36:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.801+0000 7f9362000700 1 -- 172.21.15.12:0/3139998287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c15da50 msgr2=0x7f935c0c83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.801+0000 7f9362000700 1 --2- 172.21.15.12:0/3139998287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c15da50 0x7f935c0c83c0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f9354014040 tx=0x7f9354004ad0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.801+0000 7f9362000700 1 -- 172.21.15.12:0/3139998287 shutdown_connections 2024-03-29T11:36:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.801+0000 7f9362000700 1 --2- 172.21.15.12:0/3139998287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f935c15da50 0x7f935c0c83c0 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.801+0000 7f9362000700 1 --2- 172.21.15.12:0/3139998287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f935c0d05e0 0x7f935c1f45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.801+0000 7f9362000700 1 --2- 172.21.15.12:0/3139998287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f935c0d0a40 0x7f935c1f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.801+0000 7f9362000700 1 -- 172.21.15.12:0/3139998287 >> 172.21.15.12:0/3139998287 conn(0x7f935c0666a0 msgr2=0x7f935c0b7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.801+0000 7f9362000700 1 -- 172.21.15.12:0/3139998287 shutdown_connections 2024-03-29T11:36:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:02.801+0000 7f9362000700 1 -- 172.21.15.12:0/3139998287 wait complete. 2024-03-29T11:36:02.802 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 146 2024-03-29T11:36:02.819 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:03.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.017+0000 7f4e92e00700 1 Processor -- start 2024-03-29T11:36:03.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.017+0000 7f4e92e00700 1 -- start start 2024-03-29T11:36:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.018+0000 7f4e92e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4e8c06a790 0x7f4e8c06ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.018+0000 7f4e92e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4e8c0cebd0 0x7f4e8c0cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.018+0000 7f4e92e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c06f850 con 0x7f4e8c0afbe0 2024-03-29T11:36:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.018+0000 7f4e92e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8c06e3f0 con 0x7f4e8c0cebd0 2024-03-29T11:36:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.018+0000 7f4e92e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8c06b0b0 con 0x7f4e8c06a790 2024-03-29T11:36:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.018+0000 7f4e90e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4e8c0cebd0 0x7f4e8c0cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.018+0000 7f4e90e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4e8c0cebd0 0x7f4e8c0cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40866/0 (socket says 172.21.15.12:40866) 2024-03-29T11:36:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.018+0000 7f4e90e00700 1 -- 172.21.15.12:0/2221103344 learned_addr learned my addr 172.21.15.12:0/2221103344 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.019+0000 7f4e8be00700 1 --2- 172.21.15.12:0/2221103344 >> v2:172.21.15.159:3300/0 conn(0x7f4e8c06a790 0x7f4e8c06ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.019+0000 7f4e90e00700 1 -- 172.21.15.12:0/2221103344 >> v2:172.21.15.159:3300/0 conn(0x7f4e8c06a790 msgr2=0x7f4e8c06ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:03.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.019+0000 7f4e90e00700 1 --2- 172.21.15.12:0/2221103344 >> v2:172.21.15.159:3300/0 conn(0x7f4e8c06a790 0x7f4e8c06ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.019+0000 7f4e90e00700 1 -- 172.21.15.12:0/2221103344 >> v1:172.21.15.12:6789/0 conn(0x7f4e8c0afbe0 legacy=0x7f4e8c06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.019+0000 7f4e90e00700 1 -- 172.21.15.12:0/2221103344 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c06fad0 con 0x7f4e8c0cebd0 2024-03-29T11:36:03.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.019+0000 7f4e90e00700 1 --2- 172.21.15.12:0/2221103344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c0cebd0 0x7f4e8c0cefe0 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c06d9a0 tx=0x7f4e8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60f7b5b8c9c42e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:03.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.019+0000 7f4e8aa00700 1 -- 172.21.15.12:0/2221103344 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8400e070 con 0x7f4e8c0cebd0 2024-03-29T11:36:03.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.019+0000 7f4e8aa00700 1 -- 172.21.15.12:0/2221103344 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8400b9c0 con 0x7f4e8c0cebd0 2024-03-29T11:36:03.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.019+0000 7f4e8aa00700 1 -- 172.21.15.12:0/2221103344 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8400bd70 con 0x7f4e8c0cebd0 2024-03-29T11:36:03.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.020+0000 7f4e92e00700 1 -- 172.21.15.12:0/2221103344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c0cebd0 msgr2=0x7f4e8c0cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.020+0000 7f4e92e00700 1 --2- 172.21.15.12:0/2221103344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c0cebd0 0x7f4e8c0cefe0 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c06d9a0 tx=0x7f4e8400b330 comp rx=0 tx=0).stop 2024-03-29T11:36:03.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.020+0000 7f4e92e00700 1 -- 172.21.15.12:0/2221103344 shutdown_connections 2024-03-29T11:36:03.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.020+0000 7f4e92e00700 1 --2- 172.21.15.12:0/2221103344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c0cebd0 0x7f4e8c0cefe0 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.020+0000 7f4e92e00700 1 --2- 172.21.15.12:0/2221103344 >> v2:172.21.15.159:3300/0 conn(0x7f4e8c06a790 0x7f4e8c06ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.020+0000 7f4e92e00700 1 -- 172.21.15.12:0/2221103344 >> 172.21.15.12:0/2221103344 conn(0x7f4e8c0b5ab0 msgr2=0x7f4e8c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:03.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.020+0000 7f4e92e00700 1 -- 172.21.15.12:0/2221103344 shutdown_connections 2024-03-29T11:36:03.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.020+0000 7f4e92e00700 1 -- 172.21.15.12:0/2221103344 wait complete. 2024-03-29T11:36:03.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.022+0000 7f4e92e00700 1 Processor -- start 2024-03-29T11:36:03.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.022+0000 7f4e92e00700 1 -- start start 2024-03-29T11:36:03.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.023+0000 7f4e92e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c06a790 0x7f4e8c154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.023+0000 7f4e92e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e8c0afbe0 0x7f4e8c0cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.023+0000 7f4e92e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0cebd0 0x7f4e8c0c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.023+0000 7f4e92e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c15bd40 con 0x7f4e8c06a790 2024-03-29T11:36:03.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.023+0000 7f4e92e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c15c550 con 0x7f4e8c0afbe0 2024-03-29T11:36:03.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.023+0000 7f4e92e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c15bf90 con 0x7f4e8c0cebd0 2024-03-29T11:36:03.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.023+0000 7f4e8be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c06a790 0x7f4e8c154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.023+0000 7f4e8be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c06a790 0x7f4e8c154ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40880/0 (socket says 172.21.15.12:40880) 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.023+0000 7f4e8be00700 1 -- 172.21.15.12:0/3398067457 learned_addr learned my addr 172.21.15.12:0/3398067457 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.023+0000 7f4e8b400700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e8c0afbe0 0x7f4e8c0cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e90e00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0cebd0 0x7f4e8c0c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e8be00700 1 -- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0cebd0 msgr2=0x7f4e8c0c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e8be00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0cebd0 0x7f4e8c0c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e8be00700 1 -- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e8c0afbe0 msgr2=0x7f4e8c0cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e8be00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e8c0afbe0 0x7f4e8c0cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e8be00700 1 -- 172.21.15.12:0/3398067457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e84009b30 con 0x7f4e8c06a790 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e8be00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c06a790 0x7f4e8c154ce0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00d800 tx=0x7f4e7c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e83e00700 1 -- 172.21.15.12:0/3398067457 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0125f0 con 0x7f4e8c06a790 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e83e00700 1 -- 172.21.15.12:0/3398067457 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c012750 con 0x7f4e8c06a790 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e92e00700 1 -- 172.21.15.12:0/3398067457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c15c270 con 0x7f4e8c06a790 2024-03-29T11:36:03.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e83e00700 1 -- 172.21.15.12:0/3398067457 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c012b00 con 0x7f4e8c06a790 2024-03-29T11:36:03.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.024+0000 7f4e92e00700 1 -- 172.21.15.12:0/3398067457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c1558a0 con 0x7f4e8c06a790 2024-03-29T11:36:03.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.025+0000 7f4e92e00700 1 -- 172.21.15.12:0/3398067457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8c067720 con 0x7f4e8c06a790 2024-03-29T11:36:03.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.026+0000 7f4e83e00700 1 -- 172.21.15.12:0/3398067457 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 164) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f4e7c012c60 con 0x7f4e8c06a790 2024-03-29T11:36:03.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.027+0000 7f4e83e00700 1 -- 172.21.15.12:0/3398067457 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 6541+0+0 (secure 0 0 0) 0x7f4e7c089470 con 0x7f4e8c06a790 2024-03-29T11:36:03.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.032+0000 7f4e83e00700 1 -- 172.21.15.12:0/3398067457 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e7c054760 con 0x7f4e8c06a790 2024-03-29T11:36:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.165+0000 7f4e83e00700 1 -- 172.21.15.12:0/3398067457 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 165) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f4e7c054210 con 0x7f4e8c06a790 2024-03-29T11:36:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.165+0000 7f4e83e00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e6c070590 0x7f4e6c072980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.166+0000 7f4e8b400700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e6c070590 0x7f4e6c072980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.167+0000 7f4e8b400700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e6c070590 0x7f4e6c072980 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c1f63e0 tx=0x7f4e78009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:03.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.251+0000 7f4e92e00700 1 -- 172.21.15.12:0/3398067457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e8c06de30 con 0x7f4e8c06a790 2024-03-29T11:36:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.253+0000 7f4e83e00700 1 -- 172.21.15.12:0/3398067457 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v287) v1 ==== 74+0+17287 (secure 0 0 0) 0x7f4e7c05e070 con 0x7f4e8c06a790 2024-03-29T11:36:03.253 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:03.253 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":287,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:36:02.154728+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-03-29T11:34:18.360241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-03-29T11:34:18.903400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"283","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.d","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/4093476986":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/3928992099":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/3508779586":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/1359166523":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/922684053":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/3536271161":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/3208784174":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/2115167177":"2024-03-30T11:34:37.193903+0000","172.21.15.159:6833/62646":"2024-03-30T11:34:37.193903+0000","172.21.15.159:6832/62646":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/411770029":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/3938940279":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/2298284405":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/3177306995":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/2440579896":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/1795874409":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/547605341":"2024-03-30T11:34:37.193903+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 -- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e6c070590 msgr2=0x7f4e6c072980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e6c070590 0x7f4e6c072980 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c1f63e0 tx=0x7f4e78009040 comp rx=0 tx=0).stop 2024-03-29T11:36:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 -- 172.21.15.12:0/3398067457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c06a790 msgr2=0x7f4e8c154ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c06a790 0x7f4e8c154ce0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00d800 tx=0x7f4e7c00dbc0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 -- 172.21.15.12:0/3398067457 shutdown_connections 2024-03-29T11:36:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e6c070590 0x7f4e6c072980 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0cebd0 0x7f4e8c0c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8c06a790 0x7f4e8c154ce0 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 --2- 172.21.15.12:0/3398067457 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e8c0afbe0 0x7f4e8c0cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 -- 172.21.15.12:0/3398067457 >> 172.21.15.12:0/3398067457 conn(0x7f4e8c0b5ab0 msgr2=0x7f4e8c0ad350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 -- 172.21.15.12:0/3398067457 shutdown_connections 2024-03-29T11:36:03.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.255+0000 7f4e92e00700 1 -- 172.21.15.12:0/3398067457 wait complete. 2024-03-29T11:36:03.273 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-29T11:36:03.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.473+0000 7fe895600700 1 Processor -- start 2024-03-29T11:36:03.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.474+0000 7fe895600700 1 -- start start 2024-03-29T11:36:03.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.475+0000 7fe895600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe8900cd020 0x7fe8900cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.475+0000 7fe895600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe8900b2070 0x7fe8900b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.475+0000 7fe895600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8900cdae0 con 0x7fe8900cbb30 2024-03-29T11:36:03.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.475+0000 7fe895600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8900cddc0 con 0x7fe8900b2070 2024-03-29T11:36:03.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.475+0000 7fe895600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8900ce0a0 con 0x7fe8900cd020 2024-03-29T11:36:03.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.475+0000 7fe88f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe8900b2070 0x7fe8900b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.475+0000 7fe88f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe8900b2070 0x7fe8900b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40884/0 (socket says 172.21.15.12:40884) 2024-03-29T11:36:03.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.475+0000 7fe88f400700 1 -- 172.21.15.12:0/602291500 learned_addr learned my addr 172.21.15.12:0/602291500 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:03.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.475+0000 7fe88e000700 1 --2- 172.21.15.12:0/602291500 >> v2:172.21.15.159:3300/0 conn(0x7fe8900cd020 0x7fe8900cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.476+0000 7fe88f400700 1 -- 172.21.15.12:0/602291500 >> v2:172.21.15.159:3300/0 conn(0x7fe8900cd020 msgr2=0x7fe8900cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:03.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.476+0000 7fe88f400700 1 --2- 172.21.15.12:0/602291500 >> v2:172.21.15.159:3300/0 conn(0x7fe8900cd020 0x7fe8900cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.476+0000 7fe88f400700 1 -- 172.21.15.12:0/602291500 >> v1:172.21.15.159:6789/0 conn(0x7fe8900cbb30 legacy=0x7fe8900cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.476+0000 7fe88f400700 1 -- 172.21.15.12:0/602291500 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89006dc20 con 0x7fe8900b2070 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.476+0000 7fe88f400700 1 --2- 172.21.15.12:0/602291500 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900b2070 0x7fe8900b2500 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7fe8900b1e30 tx=0x7fe88800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2268bf5b8e1082 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.476+0000 7fe88d600700 1 -- 172.21.15.12:0/602291500 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88800e070 con 0x7fe8900b2070 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.476+0000 7fe88d600700 1 -- 172.21.15.12:0/602291500 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88800b910 con 0x7fe8900b2070 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.476+0000 7fe88d600700 1 -- 172.21.15.12:0/602291500 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88800bcc0 con 0x7fe8900b2070 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.477+0000 7fe895600700 1 -- 172.21.15.12:0/602291500 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900b2070 msgr2=0x7fe8900b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.477+0000 7fe895600700 1 --2- 172.21.15.12:0/602291500 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900b2070 0x7fe8900b2500 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7fe8900b1e30 tx=0x7fe88800b330 comp rx=0 tx=0).stop 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.477+0000 7fe895600700 1 -- 172.21.15.12:0/602291500 shutdown_connections 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.477+0000 7fe895600700 1 --2- 172.21.15.12:0/602291500 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900b2070 0x7fe8900b2500 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.477+0000 7fe895600700 1 --2- 172.21.15.12:0/602291500 >> v2:172.21.15.159:3300/0 conn(0x7fe8900cd020 0x7fe8900cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.477+0000 7fe895600700 1 -- 172.21.15.12:0/602291500 >> 172.21.15.12:0/602291500 conn(0x7fe8900b0d70 msgr2=0x7fe8900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.477+0000 7fe895600700 1 -- 172.21.15.12:0/602291500 shutdown_connections 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.477+0000 7fe895600700 1 -- 172.21.15.12:0/602291500 wait complete. 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe895600700 1 Processor -- start 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe895600700 1 -- start start 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe895600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8900b2070 0x7fe890098940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe895600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8900cbb30 0x7fe8900ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe895600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900cd020 0x7fe890071cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe895600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900cb5a0 con 0x7fe8900cd020 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe895600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900ac520 con 0x7fe8900cbb30 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe895600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900adfb0 con 0x7fe8900b2070 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe88f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900cd020 0x7fe890071cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe88f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900cd020 0x7fe890071cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40888/0 (socket says 172.21.15.12:40888) 2024-03-29T11:36:03.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe88f400700 1 -- 172.21.15.12:0/1246234067 learned_addr learned my addr 172.21.15.12:0/1246234067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:03.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe88ea00700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8900b2070 0x7fe890098940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.480+0000 7fe88e000700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8900cbb30 0x7fe8900ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe88f400700 1 -- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8900b2070 msgr2=0x7fe890098940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe88f400700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8900b2070 0x7fe890098940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe88f400700 1 -- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8900cbb30 msgr2=0x7fe8900ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe88f400700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8900cbb30 0x7fe8900ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe88f400700 1 -- 172.21.15.12:0/1246234067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe888009b30 con 0x7fe8900cd020 2024-03-29T11:36:03.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe88f400700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900cd020 0x7fe890071cb0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fe88800b370 tx=0x7fe888004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:03.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe886a00700 1 -- 172.21.15.12:0/1246234067 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888004880 con 0x7fe8900cd020 2024-03-29T11:36:03.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe886a00700 1 -- 172.21.15.12:0/1246234067 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8880099b0 con 0x7fe8900cd020 2024-03-29T11:36:03.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe886a00700 1 -- 172.21.15.12:0/1246234067 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88801c680 con 0x7fe8900cd020 2024-03-29T11:36:03.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe895600700 1 -- 172.21.15.12:0/1246234067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8900ac1f0 con 0x7fe8900cd020 2024-03-29T11:36:03.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.481+0000 7fe895600700 1 -- 172.21.15.12:0/1246234067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8900cb2c0 con 0x7fe8900cd020 2024-03-29T11:36:03.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.482+0000 7fe895600700 1 -- 172.21.15.12:0/1246234067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8900ac050 con 0x7fe8900cd020 2024-03-29T11:36:03.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.483+0000 7fe886a00700 1 -- 172.21.15.12:0/1246234067 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 165) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fe88801c7e0 con 0x7fe8900cd020 2024-03-29T11:36:03.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.484+0000 7fe886a00700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe87006aae0 0x7fe87006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.484+0000 7fe886a00700 1 -- 172.21.15.12:0/1246234067 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 6541+0+0 (secure 0 0 0) 0x7fe88808af20 con 0x7fe8900cd020 2024-03-29T11:36:03.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.485+0000 7fe88ea00700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe87006aae0 0x7fe87006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.485+0000 7fe88ea00700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe87006aae0 0x7fe87006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe880007fd0 tx=0x7fe880009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:03.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.489+0000 7fe886a00700 1 -- 172.21.15.12:0/1246234067 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe8900ac050 con 0x7fe8900cd020 2024-03-29T11:36:03.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.751+0000 7fe895600700 1 -- 172.21.15.12:0/1246234067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe8900ac050 con 0x7fe8900cd020 2024-03-29T11:36:03.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-03-29T11:36:03.752+0000 7f0f51a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-29T11:36:03.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.765+0000 7fe886a00700 1 -- 172.21.15.12:0/1246234067 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v147) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe8900ac050 con 0x7fe8900cd020 2024-03-29T11:36:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 -- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe87006aae0 msgr2=0x7fe87006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe87006aae0 0x7fe87006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe880007fd0 tx=0x7fe880009040 comp rx=0 tx=0).stop 2024-03-29T11:36:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 -- 172.21.15.12:0/1246234067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900cd020 msgr2=0x7fe890071cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900cd020 0x7fe890071cb0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fe88800b370 tx=0x7fe888004ad0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 -- 172.21.15.12:0/1246234067 shutdown_connections 2024-03-29T11:36:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe87006aae0 0x7fe87006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8900cd020 0x7fe890071cb0 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8900b2070 0x7fe890098940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 --2- 172.21.15.12:0/1246234067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8900cbb30 0x7fe8900ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 -- 172.21.15.12:0/1246234067 >> 172.21.15.12:0/1246234067 conn(0x7fe8900b0d70 msgr2=0x7fe8900aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 -- 172.21.15.12:0/1246234067 shutdown_connections 2024-03-29T11:36:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.768+0000 7fe895600700 1 -- 172.21.15.12:0/1246234067 wait complete. 2024-03-29T11:36:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-29T11:36:03.786 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-29T11:36:03.832 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-03-29T11:36:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.994+0000 7fb16dc00700 1 Processor -- start 2024-03-29T11:36:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.994+0000 7fb16dc00700 1 -- start start 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb16dc00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb1680cd020 0x7fb1680cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb16dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 0x7fb1680b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb16dc00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1680cdae0 con 0x7fb1680cbb30 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb16dc00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1680cddc0 con 0x7fb1680cd020 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb16dc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680ce0a0 con 0x7fb1680b2070 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb166000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb1680cd020 0x7fb1680cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb167400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 0x7fb1680b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb166000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb1680cd020 0x7fb1680cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:36952/0 (socket says 172.21.15.12:36952) 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb166000700 1 -- 172.21.15.12:0/335097277 learned_addr learned my addr 172.21.15.12:0/335097277 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb166a00700 1 --1- 172.21.15.12:0/335097277 >> v1:172.21.15.159:6789/0 conn(0x7fb1680cbb30 0x7fb1680cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:52736/0 (socket says 172.21.15.12:52736) 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb167400700 1 -- 172.21.15.12:0/335097277 >> v2:172.21.15.159:3300/0 conn(0x7fb1680cd020 msgr2=0x7fb1680cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb167400700 1 --2- 172.21.15.12:0/335097277 >> v2:172.21.15.159:3300/0 conn(0x7fb1680cd020 0x7fb1680cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb167400700 1 -- 172.21.15.12:0/335097277 >> v1:172.21.15.159:6789/0 conn(0x7fb1680cbb30 legacy=0x7fb1680cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb167400700 1 -- 172.21.15.12:0/335097277 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16806dc20 con 0x7fb1680b2070 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.995+0000 7fb167400700 1 --2- 172.21.15.12:0/335097277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 0x7fb1680b2500 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb160009d90 tx=0x7fb16000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=303e2fa4445ba7d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb165600700 1 -- 172.21.15.12:0/335097277 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16000e070 con 0x7fb1680b2070 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb165600700 1 -- 172.21.15.12:0/335097277 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16000bb20 con 0x7fb1680b2070 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb165600700 1 -- 172.21.15.12:0/335097277 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160012430 con 0x7fb1680b2070 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb16dc00700 1 -- 172.21.15.12:0/335097277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 msgr2=0x7fb1680b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb16dc00700 1 --2- 172.21.15.12:0/335097277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 0x7fb1680b2500 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb160009d90 tx=0x7fb16000b330 comp rx=0 tx=0).stop 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb16dc00700 1 -- 172.21.15.12:0/335097277 shutdown_connections 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb16dc00700 1 --2- 172.21.15.12:0/335097277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 0x7fb1680b2500 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb16dc00700 1 --2- 172.21.15.12:0/335097277 >> v2:172.21.15.159:3300/0 conn(0x7fb1680cd020 0x7fb1680cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb16dc00700 1 -- 172.21.15.12:0/335097277 >> 172.21.15.12:0/335097277 conn(0x7fb1680b0d70 msgr2=0x7fb1680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb16dc00700 1 -- 172.21.15.12:0/335097277 shutdown_connections 2024-03-29T11:36:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.996+0000 7fb16dc00700 1 -- 172.21.15.12:0/335097277 wait complete. 2024-03-29T11:36:03.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.998+0000 7fb16dc00700 1 Processor -- start 2024-03-29T11:36:03.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.998+0000 7fb16dc00700 1 -- start start 2024-03-29T11:36:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.999+0000 7fb16dc00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 0x7fb16814c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.999+0000 7fb16dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1680cbb30 0x7fb1680c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.999+0000 7fb16dc00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1680cd020 0x7fb1680c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.999+0000 7fb16dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680c8dd0 con 0x7fb1680cbb30 2024-03-29T11:36:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.999+0000 7fb16dc00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680c90b0 con 0x7fb1680cd020 2024-03-29T11:36:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.999+0000 7fb16dc00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680c9390 con 0x7fb1680b2070 2024-03-29T11:36:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.999+0000 7fb166000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1680cbb30 0x7fb1680c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.999+0000 7fb166a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 0x7fb16814c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.999+0000 7fb166000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1680cbb30 0x7fb1680c9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40924/0 (socket says 172.21.15.12:40924) 2024-03-29T11:36:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:03.999+0000 7fb166000700 1 -- 172.21.15.12:0/807000284 learned_addr learned my addr 172.21.15.12:0/807000284 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.000+0000 7fb167400700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1680cd020 0x7fb1680c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.000+0000 7fb166000700 1 -- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 msgr2=0x7fb16814c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.000+0000 7fb166000700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 0x7fb16814c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.000+0000 7fb166000700 1 -- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1680cd020 msgr2=0x7fb1680c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.000+0000 7fb166000700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1680cd020 0x7fb1680c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.000+0000 7fb166000700 1 -- 172.21.15.12:0/807000284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160009a70 con 0x7fb1680cbb30 2024-03-29T11:36:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.000+0000 7fb166000700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1680cbb30 0x7fb1680c9e70 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fb15400e8b0 tx=0x7fb15400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.000+0000 7fb15ea00700 1 -- 172.21.15.12:0/807000284 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154009600 con 0x7fb1680cbb30 2024-03-29T11:36:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.002+0000 7fb15ea00700 1 -- 172.21.15.12:0/807000284 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb154013680 con 0x7fb1680cbb30 2024-03-29T11:36:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.002+0000 7fb15ea00700 1 -- 172.21.15.12:0/807000284 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154013a30 con 0x7fb1680cbb30 2024-03-29T11:36:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.002+0000 7fb16dc00700 1 -- 172.21.15.12:0/807000284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1680c9670 con 0x7fb1680cbb30 2024-03-29T11:36:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.002+0000 7fb16dc00700 1 -- 172.21.15.12:0/807000284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb16814ca20 con 0x7fb1680cbb30 2024-03-29T11:36:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.002+0000 7fb16dc00700 1 -- 172.21.15.12:0/807000284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb168067720 con 0x7fb1680cbb30 2024-03-29T11:36:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.003+0000 7fb15ea00700 1 -- 172.21.15.12:0/807000284 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 165) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fb154013b90 con 0x7fb1680cbb30 2024-03-29T11:36:04.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.004+0000 7fb15ea00700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb14806aae0 0x7fb14806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:04.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.004+0000 7fb15ea00700 1 -- 172.21.15.12:0/807000284 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6627+0+0 (secure 0 0 0) 0x7fb154089170 con 0x7fb1680cbb30 2024-03-29T11:36:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.004+0000 7fb166a00700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb14806aae0 0x7fb14806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.005+0000 7fb166a00700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb14806aae0 0x7fb14806cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb158000f60 tx=0x7fb158007ed0 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.008+0000 7fb15ea00700 1 -- 172.21.15.12:0/807000284 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb154054260 con 0x7fb1680cbb30 2024-03-29T11:36:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.277+0000 7fb16dc00700 1 -- 172.21.15.12:0/807000284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb168158cc0 con 0x7fb1680cbb30 2024-03-29T11:36:04.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.768+0000 7fb15ea00700 1 -- 172.21.15.12:0/807000284 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v148) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb154057880 con 0x7fb1680cbb30 2024-03-29T11:36:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 -- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb14806aae0 msgr2=0x7fb14806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb14806aae0 0x7fb14806cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb158000f60 tx=0x7fb158007ed0 comp rx=0 tx=0).stop 2024-03-29T11:36:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 -- 172.21.15.12:0/807000284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1680cbb30 msgr2=0x7fb1680c9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1680cbb30 0x7fb1680c9e70 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fb15400e8b0 tx=0x7fb15400ebc0 comp rx=0 tx=0).stop 2024-03-29T11:36:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 -- 172.21.15.12:0/807000284 shutdown_connections 2024-03-29T11:36:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fb14806aae0 0x7fb14806cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1680cd020 0x7fb1680c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1680b2070 0x7fb16814c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:04.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 --2- 172.21.15.12:0/807000284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1680cbb30 0x7fb1680c9e70 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 -- 172.21.15.12:0/807000284 >> 172.21.15.12:0/807000284 conn(0x7fb1680b0d70 msgr2=0x7fb1680b3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 -- 172.21.15.12:0/807000284 shutdown_connections 2024-03-29T11:36:04.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.771+0000 7fb16dc00700 1 -- 172.21.15.12:0/807000284 wait complete. 2024-03-29T11:36:04.793 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-29T11:36:04.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.998+0000 7fea5e800700 1 Processor -- start 2024-03-29T11:36:04.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.998+0000 7fea5e800700 1 -- start start 2024-03-29T11:36:04.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.999+0000 7fea5e800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fea580ced30 0x7fea580d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:04.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.999+0000 7fea5e800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea580ce8a0 con 0x7fea580d2710 2024-03-29T11:36:04.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.999+0000 7fea5e800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea580cdc90 con 0x7fea5815ddf0 2024-03-29T11:36:04.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:04.999+0000 7fea5e800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5815ec60 con 0x7fea580ced30 2024-03-29T11:36:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.000+0000 7fea56a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fea580ced30 0x7fea580d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.000+0000 7fea56a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fea580ced30 0x7fea580d21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40936/0 (socket says 172.21.15.12:40936) 2024-03-29T11:36:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.000+0000 7fea57e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fea580d2710 0x7fea5815f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40730/0 (socket says 172.21.15.12:40730) 2024-03-29T11:36:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.000+0000 7fea56a00700 1 -- 172.21.15.12:0/3251550195 learned_addr learned my addr 172.21.15.12:0/3251550195 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.000+0000 7fea56a00700 1 -- 172.21.15.12:0/3251550195 >> v1:172.21.15.159:6789/0 conn(0x7fea5815ddf0 legacy=0x7fea5815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.000+0000 7fea56a00700 1 -- 172.21.15.12:0/3251550195 >> v1:172.21.15.12:6789/0 conn(0x7fea580d2710 legacy=0x7fea5815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.000+0000 7fea56a00700 1 -- 172.21.15.12:0/3251550195 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580cdf70 con 0x7fea580ced30 2024-03-29T11:36:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.000+0000 7fea56a00700 1 --2- 172.21.15.12:0/3251550195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580ced30 0x7fea580d21d0 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7fea440098f0 tx=0x7fea44009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170d14d6db726472 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.001+0000 7fea56000700 1 -- 172.21.15.12:0/3251550195 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4400b8a0 con 0x7fea580ced30 2024-03-29T11:36:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.001+0000 7fea56000700 1 -- 172.21.15.12:0/3251550195 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4400ba00 con 0x7fea580ced30 2024-03-29T11:36:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.001+0000 7fea56000700 1 -- 172.21.15.12:0/3251550195 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4400bd90 con 0x7fea580ced30 2024-03-29T11:36:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.001+0000 7fea5e800700 1 -- 172.21.15.12:0/3251550195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580ced30 msgr2=0x7fea580d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.001+0000 7fea5e800700 1 --2- 172.21.15.12:0/3251550195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580ced30 0x7fea580d21d0 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7fea440098f0 tx=0x7fea44009c00 comp rx=0 tx=0).stop 2024-03-29T11:36:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.001+0000 7fea5e800700 1 -- 172.21.15.12:0/3251550195 shutdown_connections 2024-03-29T11:36:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.001+0000 7fea5e800700 1 --2- 172.21.15.12:0/3251550195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580ced30 0x7fea580d21d0 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.001+0000 7fea5e800700 1 -- 172.21.15.12:0/3251550195 >> 172.21.15.12:0/3251550195 conn(0x7fea5808aff0 msgr2=0x7fea580d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.001+0000 7fea5e800700 1 -- 172.21.15.12:0/3251550195 shutdown_connections 2024-03-29T11:36:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.001+0000 7fea5e800700 1 -- 172.21.15.12:0/3251550195 wait complete. 2024-03-29T11:36:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.003+0000 7fea5e800700 1 Processor -- start 2024-03-29T11:36:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.003+0000 7fea5e800700 1 -- start start 2024-03-29T11:36:05.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea5e800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea580ced30 0x7fea5815edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:05.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea5e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580d2710 0x7fea580ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:05.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea5e800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea5815ddf0 0x7fea581f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:05.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea5e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea58156f30 con 0x7fea580d2710 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea5e800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea580cb090 con 0x7fea580ced30 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea5e800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea580cb330 con 0x7fea5815ddf0 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea56a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580d2710 0x7fea580ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea56a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580d2710 0x7fea580ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40948/0 (socket says 172.21.15.12:40948) 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea56a00700 1 -- 172.21.15.12:0/2989766004 learned_addr learned my addr 172.21.15.12:0/2989766004 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea57400700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea580ced30 0x7fea5815edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.004+0000 7fea57e00700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea5815ddf0 0x7fea581f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea56a00700 1 -- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea5815ddf0 msgr2=0x7fea581f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea56a00700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea5815ddf0 0x7fea581f2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea56a00700 1 -- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea580ced30 msgr2=0x7fea5815edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea56a00700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea580ced30 0x7fea5815edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea56a00700 1 -- 172.21.15.12:0/2989766004 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea44009550 con 0x7fea580d2710 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea57400700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea580ced30 0x7fea5815edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea56a00700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580d2710 0x7fea580ccd50 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fea44000c00 tx=0x7fea44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:05.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea4f400700 1 -- 172.21.15.12:0/2989766004 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea44004570 con 0x7fea580d2710 2024-03-29T11:36:05.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea4f400700 1 -- 172.21.15.12:0/2989766004 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4401f070 con 0x7fea580d2710 2024-03-29T11:36:05.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea4f400700 1 -- 172.21.15.12:0/2989766004 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea440038e0 con 0x7fea580d2710 2024-03-29T11:36:05.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea5e800700 1 -- 172.21.15.12:0/2989766004 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea580cb610 con 0x7fea580d2710 2024-03-29T11:36:05.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.005+0000 7fea5e800700 1 -- 172.21.15.12:0/2989766004 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea580cd440 con 0x7fea580d2710 2024-03-29T11:36:05.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.006+0000 7fea5e800700 1 -- 172.21.15.12:0/2989766004 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea580cbb30 con 0x7fea580d2710 2024-03-29T11:36:05.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.007+0000 7fea4f400700 1 -- 172.21.15.12:0/2989766004 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 166) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fea44021030 con 0x7fea580d2710 2024-03-29T11:36:05.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.008+0000 7fea4f400700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea3806ad20 0x7fea3806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:05.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.008+0000 7fea4f400700 1 -- 172.21.15.12:0/2989766004 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6627+0+0 (secure 0 0 0) 0x7fea4408ab90 con 0x7fea580d2710 2024-03-29T11:36:05.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.009+0000 7fea57400700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea3806ad20 0x7fea3806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:05.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.009+0000 7fea57400700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea3806ad20 0x7fea3806d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea48000f60 tx=0x7fea48009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.013+0000 7fea4f400700 1 -- 172.21.15.12:0/2989766004 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea580cbb30 con 0x7fea580d2710 2024-03-29T11:36:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:05.230+0000 7fea5e800700 1 -- 172.21.15.12:0/2989766004 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fea580cbb30 con 0x7fea580d2710 2024-03-29T11:36:06.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.172+0000 7fea4f400700 1 -- 172.21.15.12:0/2989766004 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v289) v1 ==== 181+0+0 (secure 0 0 0) 0x7fea580cbb30 con 0x7fea580d2710 2024-03-29T11:36:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 -- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea3806ad20 msgr2=0x7fea3806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea3806ad20 0x7fea3806d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea48000f60 tx=0x7fea48009040 comp rx=0 tx=0).stop 2024-03-29T11:36:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 -- 172.21.15.12:0/2989766004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580d2710 msgr2=0x7fea580ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580d2710 0x7fea580ccd50 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fea44000c00 tx=0x7fea44004750 comp rx=0 tx=0).stop 2024-03-29T11:36:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 -- 172.21.15.12:0/2989766004 shutdown_connections 2024-03-29T11:36:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fea3806ad20 0x7fea3806d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea5815ddf0 0x7fea581f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:06.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea580ced30 0x7fea5815edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 --2- 172.21.15.12:0/2989766004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea580d2710 0x7fea580ccd50 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 -- 172.21.15.12:0/2989766004 >> 172.21.15.12:0/2989766004 conn(0x7fea5808aff0 msgr2=0x7fea580b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 -- 172.21.15.12:0/2989766004 shutdown_connections 2024-03-29T11:36:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.176+0000 7fea5e800700 1 -- 172.21.15.12:0/2989766004 wait complete. 2024-03-29T11:36:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-29T11:36:06.197 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-29T11:36:06.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.412+0000 7f4526e00700 1 Processor -- start 2024-03-29T11:36:06.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.412+0000 7f4526e00700 1 -- start start 2024-03-29T11:36:06.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.413+0000 7f4526e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f452006e9c0 0x7f452015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:06.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.413+0000 7f4526e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f452015b160 0x7f45200b2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:06.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.413+0000 7f4526e00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4520070980 con 0x7f45201580c0 2024-03-29T11:36:06.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.413+0000 7f4526e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4520071570 con 0x7f452006e9c0 2024-03-29T11:36:06.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.413+0000 7f4526e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4520071990 con 0x7f452015b160 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.413+0000 7f451f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f452006e9c0 0x7f452015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.413+0000 7f451fe00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f45201580c0 0x7f452006e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40738/0 (socket says 172.21.15.12:40738) 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.413+0000 7f451f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f452006e9c0 0x7f452015ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40962/0 (socket says 172.21.15.12:40962) 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.414+0000 7f451fe00700 1 -- 172.21.15.12:0/3834506685 learned_addr learned my addr 172.21.15.12:0/3834506685 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.414+0000 7f4524e00700 1 --2- 172.21.15.12:0/3834506685 >> v2:172.21.15.159:3300/0 conn(0x7f452015b160 0x7f45200b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.414+0000 7f451f400700 1 -- 172.21.15.12:0/3834506685 >> v2:172.21.15.159:3300/0 conn(0x7f452015b160 msgr2=0x7f45200b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.414+0000 7f451f400700 1 --2- 172.21.15.12:0/3834506685 >> v2:172.21.15.159:3300/0 conn(0x7f452015b160 0x7f45200b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.414+0000 7f451f400700 1 -- 172.21.15.12:0/3834506685 >> v1:172.21.15.12:6789/0 conn(0x7f45201580c0 legacy=0x7f452006e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.414+0000 7f451ea00700 1 -- 172.21.15.12:0/3834506685 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543656686 0 0) 0x7f4520070980 con 0x7f45201580c0 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.414+0000 7f451f400700 1 -- 172.21.15.12:0/3834506685 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4520070bd0 con 0x7f452006e9c0 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.414+0000 7f451f400700 1 --2- 172.21.15.12:0/3834506685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f452006e9c0 0x7f452015ac20 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f452015f080 tx=0x7f450c00bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f68ff226efcda6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.414+0000 7f451ea00700 1 -- 172.21.15.12:0/3834506685 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c00dc70 con 0x7f452006e9c0 2024-03-29T11:36:06.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.415+0000 7f451ea00700 1 -- 172.21.15.12:0/3834506685 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c00ddd0 con 0x7f452006e9c0 2024-03-29T11:36:06.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.415+0000 7f451ea00700 1 -- 172.21.15.12:0/3834506685 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c010690 con 0x7f452006e9c0 2024-03-29T11:36:06.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.415+0000 7f4526e00700 1 -- 172.21.15.12:0/3834506685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f452006e9c0 msgr2=0x7f452015ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.415+0000 7f4526e00700 1 --2- 172.21.15.12:0/3834506685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f452006e9c0 0x7f452015ac20 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f452015f080 tx=0x7f450c00bdc0 comp rx=0 tx=0).stop 2024-03-29T11:36:06.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.415+0000 7f4526e00700 1 -- 172.21.15.12:0/3834506685 shutdown_connections 2024-03-29T11:36:06.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.415+0000 7f4526e00700 1 --2- 172.21.15.12:0/3834506685 >> v2:172.21.15.159:3300/0 conn(0x7f452015b160 0x7f45200b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:06.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.415+0000 7f4526e00700 1 --2- 172.21.15.12:0/3834506685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f452006e9c0 0x7f452015ac20 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:06.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.415+0000 7f4526e00700 1 -- 172.21.15.12:0/3834506685 >> 172.21.15.12:0/3834506685 conn(0x7f45200cf210 msgr2=0x7f45200cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:06.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.415+0000 7f4526e00700 1 -- 172.21.15.12:0/3834506685 shutdown_connections 2024-03-29T11:36:06.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.415+0000 7f4526e00700 1 -- 172.21.15.12:0/3834506685 wait complete. 2024-03-29T11:36:06.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.417+0000 7f4526e00700 1 Processor -- start 2024-03-29T11:36:06.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.417+0000 7f4526e00700 1 -- start start 2024-03-29T11:36:06.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.418+0000 7f4526e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f452006e9c0 0x7f45201435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:06.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.418+0000 7f4526e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45201580c0 0x7f45200ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:06.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.418+0000 7f4526e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f452015b160 0x7f4520159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:06.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.418+0000 7f4526e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4520143b00 con 0x7f45201580c0 2024-03-29T11:36:06.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.418+0000 7f4526e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45201f3300 con 0x7f452006e9c0 2024-03-29T11:36:06.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.418+0000 7f4526e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45200cd040 con 0x7f452015b160 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.418+0000 7f451f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45201580c0 0x7f45200ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.418+0000 7f451f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45201580c0 0x7f45200ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40966/0 (socket says 172.21.15.12:40966) 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.418+0000 7f451f400700 1 -- 172.21.15.12:0/4057018653 learned_addr learned my addr 172.21.15.12:0/4057018653 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f4524e00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f452015b160 0x7f4520159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f451fe00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f452006e9c0 0x7f45201435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f451f400700 1 -- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f452015b160 msgr2=0x7f4520159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f451f400700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f452015b160 0x7f4520159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f451f400700 1 -- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f452006e9c0 msgr2=0x7f45201435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f451f400700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f452006e9c0 0x7f45201435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f451f400700 1 -- 172.21.15.12:0/4057018653 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c00b740 con 0x7f45201580c0 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f451f400700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45201580c0 0x7f45200ce0e0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f450c00b2c0 tx=0x7f450c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f4517e00700 1 -- 172.21.15.12:0/4057018653 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c003cf0 con 0x7f45201580c0 2024-03-29T11:36:06.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f4517e00700 1 -- 172.21.15.12:0/4057018653 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c003e50 con 0x7f45201580c0 2024-03-29T11:36:06.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f4517e00700 1 -- 172.21.15.12:0/4057018653 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c010670 con 0x7f45201580c0 2024-03-29T11:36:06.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f4526e00700 1 -- 172.21.15.12:0/4057018653 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45200cd320 con 0x7f45201580c0 2024-03-29T11:36:06.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.419+0000 7f4526e00700 1 -- 172.21.15.12:0/4057018653 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45200cd8e0 con 0x7f45201580c0 2024-03-29T11:36:06.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.420+0000 7f4526e00700 1 -- 172.21.15.12:0/4057018653 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45200b4710 con 0x7f45201580c0 2024-03-29T11:36:06.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.421+0000 7f4517e00700 1 -- 172.21.15.12:0/4057018653 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f450c019400 con 0x7f45201580c0 2024-03-29T11:36:06.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.422+0000 7f4517e00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f450006adb0 0x7f450006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:06.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.422+0000 7f4517e00700 1 -- 172.21.15.12:0/4057018653 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6189+0+0 (secure 0 0 0) 0x7f450c08b3d0 con 0x7f45201580c0 2024-03-29T11:36:06.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.423+0000 7f451fe00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f450006adb0 0x7f450006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:06.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.423+0000 7f451fe00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f450006adb0 0x7f450006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4510000f60 tx=0x7f4510007be0 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:06.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.427+0000 7f4517e00700 1 -- 172.21.15.12:0/4057018653 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f450c0561a0 con 0x7f45201580c0 2024-03-29T11:36:06.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:06.642+0000 7f4526e00700 1 -- 172.21.15.12:0/4057018653 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4520067720 con 0x7f45201580c0 2024-03-29T11:36:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.387+0000 7f4517e00700 1 -- 172.21.15.12:0/4057018653 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v290) v1 ==== 169+0+0 (secure 0 0 0) 0x7f450c0597c0 con 0x7f45201580c0 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.392+0000 7f4526e00700 1 -- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f450006adb0 msgr2=0x7f450006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.392+0000 7f4526e00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f450006adb0 0x7f450006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4510000f60 tx=0x7f4510007be0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.392+0000 7f4526e00700 1 -- 172.21.15.12:0/4057018653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45201580c0 msgr2=0x7f45200ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.392+0000 7f4526e00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45201580c0 0x7f45200ce0e0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f450c00b2c0 tx=0x7f450c004750 comp rx=0 tx=0).stop 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.393+0000 7f4526e00700 1 -- 172.21.15.12:0/4057018653 shutdown_connections 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.393+0000 7f4526e00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f450006adb0 0x7f450006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.393+0000 7f4526e00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f452015b160 0x7f4520159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.393+0000 7f4526e00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f452006e9c0 0x7f45201435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.393+0000 7f4526e00700 1 --2- 172.21.15.12:0/4057018653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45201580c0 0x7f45200ce0e0 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.393+0000 7f4526e00700 1 -- 172.21.15.12:0/4057018653 >> 172.21.15.12:0/4057018653 conn(0x7f45200cf210 msgr2=0x7f452006d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:07.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.393+0000 7f4526e00700 1 -- 172.21.15.12:0/4057018653 shutdown_connections 2024-03-29T11:36:07.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.393+0000 7f4526e00700 1 -- 172.21.15.12:0/4057018653 wait complete. 2024-03-29T11:36:07.393 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-29T11:36:07.412 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-29T11:36:07.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.626+0000 7f1ba0400700 1 Processor -- start 2024-03-29T11:36:07.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.626+0000 7f1ba0400700 1 -- start start 2024-03-29T11:36:07.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1ba0400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b9815d9d0 0x7f1b980d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1ba0400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b980d2540 0x7f1b980ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1ba0400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b98161420 con 0x7f1b980d16a0 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1ba0400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b9815edc0 con 0x7f1b980d2540 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1ba0400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b98161700 con 0x7f1b9815d9d0 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1b9e400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b980d2540 0x7f1b980ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1b9e400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b980d2540 0x7f1b980ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40972/0 (socket says 172.21.15.12:40972) 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1b9e400700 1 -- 172.21.15.12:0/2074911459 learned_addr learned my addr 172.21.15.12:0/2074911459 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1b9da00700 1 --2- 172.21.15.12:0/2074911459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b9815d9d0 0x7f1b980d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1b9e400700 1 -- 172.21.15.12:0/2074911459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b9815d9d0 msgr2=0x7f1b980d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1b9e400700 1 --2- 172.21.15.12:0/2074911459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b9815d9d0 0x7f1b980d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1b9e400700 1 -- 172.21.15.12:0/2074911459 >> v1:172.21.15.159:6789/0 conn(0x7f1b980d16a0 legacy=0x7f1b980d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.628+0000 7f1b9e400700 1 -- 172.21.15.12:0/2074911459 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b981619e0 con 0x7f1b980d2540 2024-03-29T11:36:07.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.629+0000 7f1b9e400700 1 --2- 172.21.15.12:0/2074911459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b980d2540 0x7f1b980ce360 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009e80 tx=0x7f1b9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a66576c80522798 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:07.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.629+0000 7f1b97e00700 1 -- 172.21.15.12:0/2074911459 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000e070 con 0x7f1b980d2540 2024-03-29T11:36:07.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.629+0000 7f1b97e00700 1 -- 172.21.15.12:0/2074911459 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9000b980 con 0x7f1b980d2540 2024-03-29T11:36:07.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.629+0000 7f1b97e00700 1 -- 172.21.15.12:0/2074911459 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000bd30 con 0x7f1b980d2540 2024-03-29T11:36:07.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.629+0000 7f1ba0400700 1 -- 172.21.15.12:0/2074911459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b980d2540 msgr2=0x7f1b980ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.629+0000 7f1ba0400700 1 --2- 172.21.15.12:0/2074911459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b980d2540 0x7f1b980ce360 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009e80 tx=0x7f1b9000b330 comp rx=0 tx=0).stop 2024-03-29T11:36:07.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.630+0000 7f1ba0400700 1 -- 172.21.15.12:0/2074911459 shutdown_connections 2024-03-29T11:36:07.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.630+0000 7f1ba0400700 1 --2- 172.21.15.12:0/2074911459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b980d2540 0x7f1b980ce360 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.630+0000 7f1ba0400700 1 --2- 172.21.15.12:0/2074911459 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b9815d9d0 0x7f1b980d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.630+0000 7f1ba0400700 1 -- 172.21.15.12:0/2074911459 >> 172.21.15.12:0/2074911459 conn(0x7f1b9808aff0 msgr2=0x7f1b980d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:07.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.630+0000 7f1ba0400700 1 -- 172.21.15.12:0/2074911459 shutdown_connections 2024-03-29T11:36:07.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.630+0000 7f1ba0400700 1 -- 172.21.15.12:0/2074911459 wait complete. 2024-03-29T11:36:07.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.631+0000 7f1ba0400700 1 Processor -- start 2024-03-29T11:36:07.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.631+0000 7f1ba0400700 1 -- start start 2024-03-29T11:36:07.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.632+0000 7f1ba0400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b980d16a0 0x7f1b98154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:07.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.632+0000 7f1ba0400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b980d2540 0x7f1b980b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.632+0000 7f1ba0400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b9815d9d0 0x7f1b980b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.632+0000 7f1ba0400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980b60c0 con 0x7f1b9815d9d0 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.632+0000 7f1ba0400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980b63a0 con 0x7f1b980d16a0 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.632+0000 7f1ba0400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980b6680 con 0x7f1b980d2540 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.633+0000 7f1b9e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b9815d9d0 0x7f1b980b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.633+0000 7f1b9e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b9815d9d0 0x7f1b980b8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40974/0 (socket says 172.21.15.12:40974) 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.633+0000 7f1b9e400700 1 -- 172.21.15.12:0/4279185082 learned_addr learned my addr 172.21.15.12:0/4279185082 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.633+0000 7f1b9e400700 1 -- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b980d2540 msgr2=0x7f1b980b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.633+0000 7f1b9e400700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b980d2540 0x7f1b980b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.633+0000 7f1b9e400700 1 -- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b980d16a0 msgr2=0x7f1b98154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.633+0000 7f1b9e400700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b980d16a0 0x7f1b98154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.633+0000 7f1b9e400700 1 -- 172.21.15.12:0/4279185082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b90009b30 con 0x7f1b9815d9d0 2024-03-29T11:36:07.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.633+0000 7f1b9e400700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b9815d9d0 0x7f1b980b8930 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f1b90012000 tx=0x7f1b90004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:07.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.633+0000 7f1b95600700 1 -- 172.21.15.12:0/4279185082 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b90004960 con 0x7f1b9815d9d0 2024-03-29T11:36:07.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.634+0000 7f1b95600700 1 -- 172.21.15.12:0/4279185082 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b900099b0 con 0x7f1b9815d9d0 2024-03-29T11:36:07.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.634+0000 7f1b95600700 1 -- 172.21.15.12:0/4279185082 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9001c680 con 0x7f1b9815d9d0 2024-03-29T11:36:07.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.634+0000 7f1ba0400700 1 -- 172.21.15.12:0/4279185082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b980b6960 con 0x7f1b9815d9d0 2024-03-29T11:36:07.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.634+0000 7f1ba0400700 1 -- 172.21.15.12:0/4279185082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b981549c0 con 0x7f1b9815d9d0 2024-03-29T11:36:07.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.635+0000 7f1ba0400700 1 -- 172.21.15.12:0/4279185082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b980d1ca0 con 0x7f1b9815d9d0 2024-03-29T11:36:07.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.636+0000 7f1b95600700 1 -- 172.21.15.12:0/4279185082 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1b9001c7e0 con 0x7f1b9815d9d0 2024-03-29T11:36:07.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.636+0000 7f1b95600700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b7806adb0 0x7f1b7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:07.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.636+0000 7f1b95600700 1 -- 172.21.15.12:0/4279185082 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f1b9008ae90 con 0x7f1b9815d9d0 2024-03-29T11:36:07.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.637+0000 7f1b9da00700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b7806adb0 0x7f1b7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:07.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.638+0000 7f1b9da00700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b7806adb0 0x7f1b7806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0098d0 tx=0x7f1b8c008040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:07.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.641+0000 7f1b95600700 1 -- 172.21.15.12:0/4279185082 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b90055eb0 con 0x7f1b9815d9d0 2024-03-29T11:36:07.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.854+0000 7f1ba0400700 1 -- 172.21.15.12:0/4279185082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1b980b6e20 con 0x7f1b9815d9d0 2024-03-29T11:36:07.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.858+0000 7f1b95600700 1 -- 172.21.15.12:0/4279185082 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v97) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1b900594d0 con 0x7f1b9815d9d0 2024-03-29T11:36:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 -- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b7806adb0 msgr2=0x7f1b7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b7806adb0 0x7f1b7806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0098d0 tx=0x7f1b8c008040 comp rx=0 tx=0).stop 2024-03-29T11:36:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 -- 172.21.15.12:0/4279185082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b9815d9d0 msgr2=0x7f1b980b8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b9815d9d0 0x7f1b980b8930 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f1b90012000 tx=0x7f1b90004ad0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 -- 172.21.15.12:0/4279185082 shutdown_connections 2024-03-29T11:36:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1b7806adb0 0x7f1b7806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b9815d9d0 0x7f1b980b8930 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b980d16a0 0x7f1b98154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 --2- 172.21.15.12:0/4279185082 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b980d2540 0x7f1b980b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:07.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 -- 172.21.15.12:0/4279185082 >> 172.21.15.12:0/4279185082 conn(0x7f1b9808aff0 msgr2=0x7f1b9815fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:07.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 -- 172.21.15.12:0/4279185082 shutdown_connections 2024-03-29T11:36:07.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:07.861+0000 7f1ba0400700 1 -- 172.21.15.12:0/4279185082 wait complete. 2024-03-29T11:36:07.878 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-29T11:36:08.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.080+0000 7f9e30600700 1 Processor -- start 2024-03-29T11:36:08.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.080+0000 7f9e30600700 1 -- start start 2024-03-29T11:36:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.081+0000 7f9e30600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e2815a5c0 0x7f9e280709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.081+0000 7f9e30600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9e280ce840 0x7f9e280cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.081+0000 7f9e30600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280cb310 con 0x7f9e2815dfe0 2024-03-29T11:36:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.081+0000 7f9e30600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e28070f00 con 0x7f9e2815a5c0 2024-03-29T11:36:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.081+0000 7f9e30600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e280711a0 con 0x7f9e280ce840 2024-03-29T11:36:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.081+0000 7f9e2dc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e2815a5c0 0x7f9e280709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.081+0000 7f9e2dc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e2815a5c0 0x7f9e280709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40984/0 (socket says 172.21.15.12:40984) 2024-03-29T11:36:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.081+0000 7f9e2dc00700 1 -- 172.21.15.12:0/2746516602 learned_addr learned my addr 172.21.15.12:0/2746516602 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.081+0000 7f9e2d200700 1 --1- 172.21.15.12:0/2746516602 >> v1:172.21.15.12:6789/0 conn(0x7f9e2815dfe0 0x7f9e280ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40754/0 (socket says 172.21.15.12:40754) 2024-03-29T11:36:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e2e600700 1 --2- 172.21.15.12:0/2746516602 >> v2:172.21.15.159:3300/0 conn(0x7f9e280ce840 0x7f9e280cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e2dc00700 1 -- 172.21.15.12:0/2746516602 >> v2:172.21.15.159:3300/0 conn(0x7f9e280ce840 msgr2=0x7f9e280cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e2dc00700 1 --2- 172.21.15.12:0/2746516602 >> v2:172.21.15.159:3300/0 conn(0x7f9e280ce840 0x7f9e280cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e2dc00700 1 -- 172.21.15.12:0/2746516602 >> v1:172.21.15.12:6789/0 conn(0x7f9e2815dfe0 legacy=0x7f9e280ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e2dc00700 1 -- 172.21.15.12:0/2746516602 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e28071480 con 0x7f9e2815a5c0 2024-03-29T11:36:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e2e600700 1 --2- 172.21.15.12:0/2746516602 >> v2:172.21.15.159:3300/0 conn(0x7f9e280ce840 0x7f9e280cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:36:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e27e00700 1 -- 172.21.15.12:0/2746516602 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600894246 0 0) 0x7f9e280cb310 con 0x7f9e2815dfe0 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e2dc00700 1 --2- 172.21.15.12:0/2746516602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e2815a5c0 0x7f9e280709c0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f9e2806b630 tx=0x7f9e1c00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df7fe9c90fa50c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e27e00700 1 -- 172.21.15.12:0/2746516602 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c00dd30 con 0x7f9e2815a5c0 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e27e00700 1 -- 172.21.15.12:0/2746516602 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c00de90 con 0x7f9e2815a5c0 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.082+0000 7f9e27e00700 1 -- 172.21.15.12:0/2746516602 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c010690 con 0x7f9e2815a5c0 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.083+0000 7f9e30600700 1 -- 172.21.15.12:0/2746516602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e2815a5c0 msgr2=0x7f9e280709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.083+0000 7f9e30600700 1 --2- 172.21.15.12:0/2746516602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e2815a5c0 0x7f9e280709c0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f9e2806b630 tx=0x7f9e1c00be70 comp rx=0 tx=0).stop 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.083+0000 7f9e30600700 1 -- 172.21.15.12:0/2746516602 shutdown_connections 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.083+0000 7f9e30600700 1 --2- 172.21.15.12:0/2746516602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e2815a5c0 0x7f9e280709c0 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.083+0000 7f9e30600700 1 --2- 172.21.15.12:0/2746516602 >> v2:172.21.15.159:3300/0 conn(0x7f9e280ce840 0x7f9e280cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.083+0000 7f9e30600700 1 -- 172.21.15.12:0/2746516602 >> 172.21.15.12:0/2746516602 conn(0x7f9e280b0d70 msgr2=0x7f9e280b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.083+0000 7f9e30600700 1 -- 172.21.15.12:0/2746516602 shutdown_connections 2024-03-29T11:36:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.083+0000 7f9e30600700 1 -- 172.21.15.12:0/2746516602 wait complete. 2024-03-29T11:36:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.085+0000 7f9e30600700 1 Processor -- start 2024-03-29T11:36:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.085+0000 7f9e30600700 1 -- start start 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e30600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e280ce840 0x7f9e2814cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e30600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e2815a5c0 0x7f9e28098880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e30600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e2815dfe0 0x7f9e281554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e30600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e281e6bc0 con 0x7f9e280ce840 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e30600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2814c7e0 con 0x7f9e2815dfe0 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e30600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e281e68e0 con 0x7f9e2815a5c0 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e2dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e280ce840 0x7f9e2814cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e2dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e280ce840 0x7f9e2814cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40996/0 (socket says 172.21.15.12:40996) 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e2dc00700 1 -- 172.21.15.12:0/4034118229 learned_addr learned my addr 172.21.15.12:0/4034118229 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e2dc00700 1 -- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e2815a5c0 msgr2=0x7f9e28098880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e2dc00700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e2815a5c0 0x7f9e28098880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e2dc00700 1 -- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e2815dfe0 msgr2=0x7f9e281554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e2dc00700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e2815dfe0 0x7f9e281554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e2dc00700 1 -- 172.21.15.12:0/4034118229 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c00b7f0 con 0x7f9e280ce840 2024-03-29T11:36:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.086+0000 7f9e2dc00700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e280ce840 0x7f9e2814cc70 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00b6f0 tx=0x7f9e1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.087+0000 7f9e25600700 1 -- 172.21.15.12:0/4034118229 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c003e50 con 0x7f9e280ce840 2024-03-29T11:36:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.087+0000 7f9e25600700 1 -- 172.21.15.12:0/4034118229 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c010470 con 0x7f9e280ce840 2024-03-29T11:36:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.087+0000 7f9e30600700 1 -- 172.21.15.12:0/4034118229 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2814c190 con 0x7f9e280ce840 2024-03-29T11:36:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.087+0000 7f9e25600700 1 -- 172.21.15.12:0/4034118229 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c010800 con 0x7f9e280ce840 2024-03-29T11:36:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.087+0000 7f9e30600700 1 -- 172.21.15.12:0/4034118229 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e28098dc0 con 0x7f9e280ce840 2024-03-29T11:36:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.088+0000 7f9e30600700 1 -- 172.21.15.12:0/4034118229 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e28067720 con 0x7f9e280ce840 2024-03-29T11:36:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.089+0000 7f9e25600700 1 -- 172.21.15.12:0/4034118229 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9e1c019400 con 0x7f9e280ce840 2024-03-29T11:36:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.090+0000 7f9e25600700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e0806adb0 0x7f9e0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.090+0000 7f9e25600700 1 -- 172.21.15.12:0/4034118229 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f9e1c08b250 con 0x7f9e280ce840 2024-03-29T11:36:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.090+0000 7f9e2d200700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e0806adb0 0x7f9e0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.091+0000 7f9e2d200700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e0806adb0 0x7f9e0806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9e2809d3a0 tx=0x7f9e140079f0 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:08.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.094+0000 7f9e25600700 1 -- 172.21.15.12:0/4034118229 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e1c056270 con 0x7f9e280ce840 2024-03-29T11:36:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.308+0000 7f9e30600700 1 -- 172.21.15.12:0/4034118229 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9e280b3f90 con 0x7f9e280ce840 2024-03-29T11:36:08.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.313+0000 7f9e25600700 1 -- 172.21.15.12:0/4034118229 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v98) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9e1c059890 con 0x7f9e280ce840 2024-03-29T11:36:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 -- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e0806adb0 msgr2=0x7f9e0806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e0806adb0 0x7f9e0806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9e2809d3a0 tx=0x7f9e140079f0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 -- 172.21.15.12:0/4034118229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e280ce840 msgr2=0x7f9e2814cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e280ce840 0x7f9e2814cc70 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00b6f0 tx=0x7f9e1c004750 comp rx=0 tx=0).stop 2024-03-29T11:36:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 -- 172.21.15.12:0/4034118229 shutdown_connections 2024-03-29T11:36:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9e0806adb0 0x7f9e0806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e2815dfe0 0x7f9e281554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e280ce840 0x7f9e2814cc70 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 --2- 172.21.15.12:0/4034118229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e2815a5c0 0x7f9e28098880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 -- 172.21.15.12:0/4034118229 >> 172.21.15.12:0/4034118229 conn(0x7f9e280b0d70 msgr2=0x7f9e280b2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 -- 172.21.15.12:0/4034118229 shutdown_connections 2024-03-29T11:36:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.315+0000 7f9e30600700 1 -- 172.21.15.12:0/4034118229 wait complete. 2024-03-29T11:36:08.332 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-29T11:36:08.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.533+0000 7f46fea00700 1 Processor -- start 2024-03-29T11:36:08.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.533+0000 7f46fea00700 1 -- start start 2024-03-29T11:36:08.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.534+0000 7f46fea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f815dad0 0x7f46f80ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.534+0000 7f46fea00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f46f80d21a0 0x7f46f815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.534+0000 7f46fea00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f815f820 con 0x7f46f80d1630 2024-03-29T11:36:08.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.534+0000 7f46fea00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f80cfcc0 con 0x7f46f80d21a0 2024-03-29T11:36:08.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.534+0000 7f46fea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80cd9f0 con 0x7f46f815dad0 2024-03-29T11:36:08.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.534+0000 7f46f7400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f46f80d1630 0x7f46f80d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40756/0 (socket says 172.21.15.12:40756) 2024-03-29T11:36:08.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.534+0000 7f46f7400700 1 -- 172.21.15.12:0/2599543266 learned_addr learned my addr 172.21.15.12:0/2599543266 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:08.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f7e00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f815dad0 0x7f46f80ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281300359 0 0) 0x7f46f815f820 con 0x7f46f80d1630 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46fca00700 1 --2- 172.21.15.12:0/2599543266 >> v2:172.21.15.159:3300/0 conn(0x7f46f80d21a0 0x7f46f815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc0035b0 con 0x7f46f80d1630 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1744586727 0 0) 0x7f46dc0035b0 con 0x7f46f80d1630 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f815dad0 msgr2=0x7f46f80ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f815dad0 0x7f46f80ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 >> v2:172.21.15.159:3300/0 conn(0x7f46f80d21a0 msgr2=0x7f46f815fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 --2- 172.21.15.12:0/2599543266 >> v2:172.21.15.159:3300/0 conn(0x7f46f80d21a0 0x7f46f815fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80cf420 con 0x7f46f80d1630 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 270904776 0 0) 0x7f46e4001670 con 0x7f46f80d1630 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 >> v1:172.21.15.12:6789/0 conn(0x7f46f80d1630 legacy=0x7f46f80d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc003a00 0x7f46dc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f815dad0 0x7f46dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46dc004d90 0x7f46dc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.535+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80cfcc0 con 0x7f46dc003a00 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80cd9f0 con 0x7f46dc004d90 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc00ba10 con 0x7f46f815dad0 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46f7e00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc003a00 0x7f46dc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46f7400700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f815dad0 0x7f46dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46fca00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46dc004d90 0x7f46dc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46f7e00700 1 -- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f815dad0 msgr2=0x7f46dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46f7e00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f815dad0 0x7f46dc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46f7e00700 1 -- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46dc004d90 msgr2=0x7f46dc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46f7e00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46dc004d90 0x7f46dc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46f7e00700 1 -- 172.21.15.12:0/2599543266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc0035b0 con 0x7f46dc003a00 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.536+0000 7f46f7e00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc003a00 0x7f46dc003e20 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f46e8009b60 tx=0x7f46dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db19f3ca9d44caea server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.537+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e800e6c0 con 0x7f46dc003a00 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.537+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e800e820 con 0x7f46dc003a00 2024-03-29T11:36:08.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.537+0000 7f46f6a00700 1 -- 172.21.15.12:0/2599543266 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e800eb90 con 0x7f46dc003a00 2024-03-29T11:36:08.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.537+0000 7f46fea00700 1 -- 172.21.15.12:0/2599543266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc003a00 msgr2=0x7f46dc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.537+0000 7f46fea00700 1 --2- 172.21.15.12:0/2599543266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc003a00 0x7f46dc003e20 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f46e8009b60 tx=0x7f46dc005880 comp rx=0 tx=0).stop 2024-03-29T11:36:08.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.537+0000 7f46fca00700 1 -- 172.21.15.12:0/2599543266 reap_dead start 2024-03-29T11:36:08.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.537+0000 7f46fea00700 1 -- 172.21.15.12:0/2599543266 shutdown_connections 2024-03-29T11:36:08.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.537+0000 7f46fea00700 1 -- 172.21.15.12:0/2599543266 >> 172.21.15.12:0/2599543266 conn(0x7f46f808aff0 msgr2=0x7f46f80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:08.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.537+0000 7f46fea00700 1 -- 172.21.15.12:0/2599543266 shutdown_connections 2024-03-29T11:36:08.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.537+0000 7f46fea00700 1 -- 172.21.15.12:0/2599543266 wait complete. 2024-03-29T11:36:08.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.539+0000 7f46fea00700 1 Processor -- start 2024-03-29T11:36:08.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.539+0000 7f46fea00700 1 -- start start 2024-03-29T11:36:08.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.540+0000 7f46fea00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46dc003a00 0x7f46f80b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.540+0000 7f46fea00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f814ebf0 0x7f46f806cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.540+0000 7f46fea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46f806d240 0x7f46f80c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.540+0000 7f46fea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f806d6c0 con 0x7f46f806d240 2024-03-29T11:36:08.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.540+0000 7f46fea00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80b6d40 con 0x7f46dc003a00 2024-03-29T11:36:08.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.540+0000 7f46fea00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f81f2ed0 con 0x7f46f814ebf0 2024-03-29T11:36:08.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.540+0000 7f46fca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46f806d240 0x7f46f80c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.540+0000 7f46fca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46f806d240 0x7f46f80c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41022/0 (socket says 172.21.15.12:41022) 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46fca00700 1 -- 172.21.15.12:0/1636223751 learned_addr learned my addr 172.21.15.12:0/1636223751 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46f7e00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46dc003a00 0x7f46f80b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46f7400700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f814ebf0 0x7f46f806cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46fca00700 1 -- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f814ebf0 msgr2=0x7f46f806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46fca00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f814ebf0 0x7f46f806cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46fca00700 1 -- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46dc003a00 msgr2=0x7f46f80b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46fca00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46dc003a00 0x7f46f80b71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46fca00700 1 -- 172.21.15.12:0/1636223751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f814f070 con 0x7f46f806d240 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46fca00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46f806d240 0x7f46f80c8260 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f46f80d2610 tx=0x7f46f0011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46efe00700 1 -- 172.21.15.12:0/1636223751 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f000e040 con 0x7f46f806d240 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46efe00700 1 -- 172.21.15.12:0/1636223751 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f000f040 con 0x7f46f806d240 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46efe00700 1 -- 172.21.15.12:0/1636223751 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f000fc70 con 0x7f46f806d240 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.541+0000 7f46fea00700 1 -- 172.21.15.12:0/1636223751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f80b6410 con 0x7f46f806d240 2024-03-29T11:36:08.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.542+0000 7f46fea00700 1 -- 172.21.15.12:0/1636223751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f80b69d0 con 0x7f46f806d240 2024-03-29T11:36:08.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.543+0000 7f46fea00700 1 -- 172.21.15.12:0/1636223751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f8067720 con 0x7f46f806d240 2024-03-29T11:36:08.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.543+0000 7f46efe00700 1 -- 172.21.15.12:0/1636223751 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f46f0022460 con 0x7f46f806d240 2024-03-29T11:36:08.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.545+0000 7f46efe00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f46d806adb0 0x7f46d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.545+0000 7f46efe00700 1 -- 172.21.15.12:0/1636223751 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f46f00899d0 con 0x7f46f806d240 2024-03-29T11:36:08.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.545+0000 7f46f7e00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f46d806adb0 0x7f46d806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.546+0000 7f46f7e00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f46d806adb0 0x7f46d806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f46e80090b0 tx=0x7f46e801f040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:08.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.549+0000 7f46efe00700 1 -- 172.21.15.12:0/1636223751 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f46f00548c0 con 0x7f46f806d240 2024-03-29T11:36:08.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.765+0000 7f46fea00700 1 -- 172.21.15.12:0/1636223751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f46f815a380 con 0x7f46f806d240 2024-03-29T11:36:08.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.769+0000 7f46efe00700 1 -- 172.21.15.12:0/1636223751 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v99) v1 ==== 120+0+0 (secure 0 0 0) 0x7f46f0057ee0 con 0x7f46f806d240 2024-03-29T11:36:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 -- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f46d806adb0 msgr2=0x7f46d806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f46d806adb0 0x7f46d806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f46e80090b0 tx=0x7f46e801f040 comp rx=0 tx=0).stop 2024-03-29T11:36:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 -- 172.21.15.12:0/1636223751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46f806d240 msgr2=0x7f46f80c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46f806d240 0x7f46f80c8260 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f46f80d2610 tx=0x7f46f0011ed0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 -- 172.21.15.12:0/1636223751 shutdown_connections 2024-03-29T11:36:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f46d806adb0 0x7f46d806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46f806d240 0x7f46f80c8260 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46dc003a00 0x7f46f80b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 --2- 172.21.15.12:0/1636223751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f814ebf0 0x7f46f806cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 -- 172.21.15.12:0/1636223751 >> 172.21.15.12:0/1636223751 conn(0x7f46f808aff0 msgr2=0x7f46f80cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 -- 172.21.15.12:0/1636223751 shutdown_connections 2024-03-29T11:36:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.772+0000 7f46fea00700 1 -- 172.21.15.12:0/1636223751 wait complete. 2024-03-29T11:36:08.789 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-03-29T11:36:08.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.989+0000 7f44cd000700 1 Processor -- start 2024-03-29T11:36:08.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.989+0000 7f44cd000700 1 -- start start 2024-03-29T11:36:08.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.990+0000 7f44cd000700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f44c80cb3f0 0x7f44c80ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.990+0000 7f44cd000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f44c80ac600 0x7f44c806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.990+0000 7f44cd000700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c815fd50 con 0x7f44c815b5d0 2024-03-29T11:36:08.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.990+0000 7f44cd000700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c815b9b0 con 0x7f44c80ac600 2024-03-29T11:36:08.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.990+0000 7f44cd000700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c80aca10 con 0x7f44c80cb3f0 2024-03-29T11:36:08.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.990+0000 7f44c6800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f44c80ac600 0x7f44c806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.991+0000 7f44c5e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f44c815b5d0 0x7f44c80ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55692/0 (socket says 172.21.15.12:55692) 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.991+0000 7f44c5e00700 1 -- 172.21.15.12:0/2182321258 learned_addr learned my addr 172.21.15.12:0/2182321258 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.991+0000 7f44c5400700 1 --2- 172.21.15.12:0/2182321258 >> v2:172.21.15.159:3300/0 conn(0x7f44c80cb3f0 0x7f44c80ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.991+0000 7f44c4a00700 1 -- 172.21.15.12:0/2182321258 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173781791 0 0) 0x7f44c815fd50 con 0x7f44c815b5d0 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.991+0000 7f44c4a00700 1 -- 172.21.15.12:0/2182321258 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44ac0035b0 con 0x7f44c815b5d0 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.991+0000 7f44c6800700 1 -- 172.21.15.12:0/2182321258 >> v2:172.21.15.159:3300/0 conn(0x7f44c80cb3f0 msgr2=0x7f44c80ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.991+0000 7f44c6800700 1 --2- 172.21.15.12:0/2182321258 >> v2:172.21.15.159:3300/0 conn(0x7f44c80cb3f0 0x7f44c80ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.991+0000 7f44c6800700 1 -- 172.21.15.12:0/2182321258 >> v1:172.21.15.12:6789/0 conn(0x7f44c815b5d0 legacy=0x7f44c80ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.991+0000 7f44c6800700 1 -- 172.21.15.12:0/2182321258 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c815f140 con 0x7f44c80ac600 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.991+0000 7f44c6800700 1 --2- 172.21.15.12:0/2182321258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c80ac600 0x7f44c806c9d0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f44c806c6c0 tx=0x7f44c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878e7f9e9a8155e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44c4a00700 1 -- 172.21.15.12:0/2182321258 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c000e070 con 0x7f44c80ac600 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44c4a00700 1 -- 172.21.15.12:0/2182321258 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c000bb20 con 0x7f44c80ac600 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44c4a00700 1 -- 172.21.15.12:0/2182321258 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0012430 con 0x7f44c80ac600 2024-03-29T11:36:08.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44cd000700 1 -- 172.21.15.12:0/2182321258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c80ac600 msgr2=0x7f44c806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44cd000700 1 --2- 172.21.15.12:0/2182321258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c80ac600 0x7f44c806c9d0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f44c806c6c0 tx=0x7f44c000b330 comp rx=0 tx=0).stop 2024-03-29T11:36:08.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44cd000700 1 -- 172.21.15.12:0/2182321258 shutdown_connections 2024-03-29T11:36:08.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44cd000700 1 --2- 172.21.15.12:0/2182321258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c80ac600 0x7f44c806c9d0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44cd000700 1 --2- 172.21.15.12:0/2182321258 >> v2:172.21.15.159:3300/0 conn(0x7f44c80cb3f0 0x7f44c80ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44cd000700 1 -- 172.21.15.12:0/2182321258 >> 172.21.15.12:0/2182321258 conn(0x7f44c80cf190 msgr2=0x7f44c80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:08.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44cd000700 1 -- 172.21.15.12:0/2182321258 shutdown_connections 2024-03-29T11:36:08.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.992+0000 7f44cd000700 1 -- 172.21.15.12:0/2182321258 wait complete. 2024-03-29T11:36:08.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.994+0000 7f44cd000700 1 Processor -- start 2024-03-29T11:36:08.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.994+0000 7f44cd000700 1 -- start start 2024-03-29T11:36:08.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44cd000700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44c80ac600 0x7f44c80c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44cd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c815b5d0 0x7f44c80c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44cd000700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44c8071e60 0x7f44c80722d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44cd000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c80719d0 con 0x7f44c815b5d0 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44cd000700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c8070dc0 con 0x7f44c8071e60 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44cd000700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c80710a0 con 0x7f44c80ac600 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44c5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c815b5d0 0x7f44c80c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44c5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c815b5d0 0x7f44c80c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33502/0 (socket says 172.21.15.12:33502) 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44c5400700 1 -- 172.21.15.12:0/2184730183 learned_addr learned my addr 172.21.15.12:0/2184730183 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44c5e00700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44c80ac600 0x7f44c80c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.995+0000 7f44c6800700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44c8071e60 0x7f44c80722d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44c5400700 1 -- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44c80ac600 msgr2=0x7f44c80c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44c5400700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44c80ac600 0x7f44c80c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44c5400700 1 -- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44c8071e60 msgr2=0x7f44c80722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44c5400700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44c8071e60 0x7f44c80722d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44c5400700 1 -- 172.21.15.12:0/2184730183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c0009a70 con 0x7f44c815b5d0 2024-03-29T11:36:08.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44c5400700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c815b5d0 0x7f44c80c89a0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f44b400e9b0 tx=0x7f44b400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:08.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44be000700 1 -- 172.21.15.12:0/2184730183 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4009710 con 0x7f44c815b5d0 2024-03-29T11:36:08.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44be000700 1 -- 172.21.15.12:0/2184730183 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b400c850 con 0x7f44c815b5d0 2024-03-29T11:36:08.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44be000700 1 -- 172.21.15.12:0/2184730183 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b400cbc0 con 0x7f44c815b5d0 2024-03-29T11:36:08.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44cd000700 1 -- 172.21.15.12:0/2184730183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c8071380 con 0x7f44c815b5d0 2024-03-29T11:36:08.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.996+0000 7f44cd000700 1 -- 172.21.15.12:0/2184730183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c80709c0 con 0x7f44c815b5d0 2024-03-29T11:36:08.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.997+0000 7f44cd000700 1 -- 172.21.15.12:0/2184730183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c8067720 con 0x7f44c815b5d0 2024-03-29T11:36:08.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.998+0000 7f44be000700 1 -- 172.21.15.12:0/2184730183 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f44b400cd20 con 0x7f44c815b5d0 2024-03-29T11:36:08.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.999+0000 7f44be000700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f44a806adb0 0x7f44a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:08.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:08.999+0000 7f44be000700 1 -- 172.21.15.12:0/2184730183 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f44b4089670 con 0x7f44c815b5d0 2024-03-29T11:36:08.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.000+0000 7f44c5e00700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f44a806adb0 0x7f44a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:08.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.000+0000 7f44c5e00700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f44a806adb0 0x7f44a806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44b80011e0 tx=0x7f44b8009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.003+0000 7f44be000700 1 -- 172.21.15.12:0/2184730183 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f44b4054710 con 0x7f44c815b5d0 2024-03-29T11:36:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.305+0000 7f44cd000700 1 -- 172.21.15.12:0/2184730183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) v1 -- 0x7f44c8071b10 con 0x7f44c815b5d0 2024-03-29T11:36:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.390+0000 7f44be000700 1 -- 172.21.15.12:0/2184730183 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3370) v1 ==== 201+0+0 (secure 0 0 0) 0x7f44b4057d30 con 0x7f44c815b5d0 2024-03-29T11:36:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.392+0000 7f44cd000700 1 -- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f44a806adb0 msgr2=0x7f44a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.392+0000 7f44cd000700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f44a806adb0 0x7f44a806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44b80011e0 tx=0x7f44b8009040 comp rx=0 tx=0).stop 2024-03-29T11:36:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.392+0000 7f44cd000700 1 -- 172.21.15.12:0/2184730183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c815b5d0 msgr2=0x7f44c80c89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.392+0000 7f44cd000700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c815b5d0 0x7f44c80c89a0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f44b400e9b0 tx=0x7f44b400ed70 comp rx=0 tx=0).stop 2024-03-29T11:36:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.393+0000 7f44cd000700 1 -- 172.21.15.12:0/2184730183 shutdown_connections 2024-03-29T11:36:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.393+0000 7f44cd000700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f44a806adb0 0x7f44a806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.393+0000 7f44cd000700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44c8071e60 0x7f44c80722d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.393+0000 7f44cd000700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44c80ac600 0x7f44c80c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.393+0000 7f44cd000700 1 --2- 172.21.15.12:0/2184730183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c815b5d0 0x7f44c80c89a0 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.393+0000 7f44cd000700 1 -- 172.21.15.12:0/2184730183 >> 172.21.15.12:0/2184730183 conn(0x7f44c80cf190 msgr2=0x7f44c8155370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.393+0000 7f44cd000700 1 -- 172.21.15.12:0/2184730183 shutdown_connections 2024-03-29T11:36:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:09.393+0000 7f44cd000700 1 -- 172.21.15.12:0/2184730183 wait complete. 2024-03-29T11:36:09.441 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. ... ERROR 2024-03-29T11:36:09.441 INFO:tasks.cephfs_test_runner: 2024-03-29T11:36:09.441 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa/tasks/cephfs/test_snap_schedules.py", line 678, in test_snap_schedule_subvol_and_group_arguments_01 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner: self._verify_snap_schedule(self.CREATE_VERSION, 'sv01', None) 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa/tasks/cephfs/test_snap_schedules.py", line 660, in _verify_snap_schedule 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner: path = self._get_subvol_snapdir_path(version, subvol, group) 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa/tasks/cephfs/test_snap_schedules.py", line 653, in _get_subvol_snapdir_path 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner: path = self.get_ceph_cmd_stdout("fs", *args).rstrip() 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner:AttributeError: 'TestSnapSchedulesSubvolAndGroupArguments' object has no attribute 'get_ceph_cmd_stdout' 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner: 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner:Ran 15 tests in 6063.014s 2024-03-29T11:36:09.442 INFO:tasks.cephfs_test_runner: 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner: 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa/tasks/cephfs/test_snap_schedules.py", line 678, in test_snap_schedule_subvol_and_group_arguments_01 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner: self._verify_snap_schedule(self.CREATE_VERSION, 'sv01', None) 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa/tasks/cephfs/test_snap_schedules.py", line 660, in _verify_snap_schedule 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner: path = self._get_subvol_snapdir_path(version, subvol, group) 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa/tasks/cephfs/test_snap_schedules.py", line 653, in _get_subvol_snapdir_path 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner: path = self.get_ceph_cmd_stdout("fs", *args).rstrip() 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner:AttributeError: 'TestSnapSchedulesSubvolAndGroupArguments' object has no attribute 'get_ceph_cmd_stdout' 2024-03-29T11:36:09.443 INFO:tasks.cephfs_test_runner: 2024-03-29T11:36:09.444 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-29T11:36:09.695 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a0b9ea7dcb494d7293b452e8c2de435f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0e392d9aee635f2b65768c460c7a151ead7a5e2/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-29T11:36:09.696 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-29T11:36:09.706 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-29T11:36:09.723 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-29T11:36:09.723 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:36:09.723 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:36:09.771 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:36:09.772 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-29T11:36:09.772 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:36:09.856 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348728k,nr_inodes=4087182,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-29T11:36:09.856 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:36:09.856 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:36:09.906 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:36:09.906 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-29T11:36:09.906 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-29T11:36:09.953 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345636k,nr_inodes=4086409,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-29T11:36:09.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:09.953 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T11:36:09.998 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-29T11:36:09.998 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-29T11:36:09.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:09.999 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-29T11:36:10.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:10.108 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-29T11:36:10.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:10.183 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-03-29T11:36:10.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:10.303 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-29T11:36:10.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:10.349 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-03-29T11:36:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:Cannot find device "ceph-brx" 2024-03-29T11:36:10.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:36:10.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:10.466 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-03-29T11:36:10.516 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-29T11:36:10.516 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-29T11:36:10.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:10.517 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-29T11:36:10.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:10.588 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-29T11:36:10.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:10.659 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link delete ceph-brx 2024-03-29T11:36:10.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:10.756 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-03-29T11:36:10.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-29T11:36:10.801 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link delete ceph-brx 2024-03-29T11:36:10.873 INFO:teuthology.orchestra.run.smithi159.stderr:Cannot find device "ceph-brx" 2024-03-29T11:36:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:36:10.876 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:36:10.886 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-29T11:36:10.887 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:36:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.093+0000 7f92f0200700 1 Processor -- start 2024-03-29T11:36:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.093+0000 7f92f0200700 1 -- start start 2024-03-29T11:36:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.094+0000 7f92f0200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 0x7f92e80b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.094+0000 7f92f0200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e80b2180 con 0x7f92e80b2e50 2024-03-29T11:36:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.094+0000 7f92f0200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e815b100 con 0x7f92e80b3af0 2024-03-29T11:36:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.094+0000 7f92f0200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e815b3e0 con 0x7f92e8072580 2024-03-29T11:36:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.094+0000 7f92ee200700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f92e80b2e50 0x7f92e80709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55696/0 (socket says 172.21.15.12:55696) 2024-03-29T11:36:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.094+0000 7f92ee200700 1 -- 172.21.15.12:0/1306883243 learned_addr learned my addr 172.21.15.12:0/1306883243 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.094+0000 7f92ece00700 1 --2- 172.21.15.12:0/1306883243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 0x7f92e80b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.095+0000 7f92e7e00700 1 -- 172.21.15.12:0/1306883243 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161581871 0 0) 0x7f92e815b100 con 0x7f92e80b3af0 2024-03-29T11:36:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.095+0000 7f92e7e00700 1 -- 172.21.15.12:0/1306883243 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d00057c0 con 0x7f92e80b3af0 2024-03-29T11:36:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.095+0000 7f92e7e00700 1 -- 172.21.15.12:0/1306883243 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427115594 0 0) 0x7f92e80b2180 con 0x7f92e80b2e50 2024-03-29T11:36:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.095+0000 7f92ece00700 1 -- 172.21.15.12:0/1306883243 >> v1:172.21.15.159:6789/0 conn(0x7f92e80b3af0 legacy=0x7f92e815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.095+0000 7f92ece00700 1 -- 172.21.15.12:0/1306883243 >> v1:172.21.15.12:6789/0 conn(0x7f92e80b2e50 legacy=0x7f92e80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.095+0000 7f92ece00700 1 -- 172.21.15.12:0/1306883243 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e80b23f0 con 0x7f92e8072580 2024-03-29T11:36:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.095+0000 7f92ece00700 1 --2- 172.21.15.12:0/1306883243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 0x7f92e80b2910 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009870 tx=0x7f92d4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5151e2ac68abea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.096+0000 7f92e7e00700 1 -- 172.21.15.12:0/1306883243 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d4011830 con 0x7f92e8072580 2024-03-29T11:36:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.096+0000 7f92e7e00700 1 -- 172.21.15.12:0/1306883243 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d4011990 con 0x7f92e8072580 2024-03-29T11:36:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.096+0000 7f92e7e00700 1 -- 172.21.15.12:0/1306883243 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d4011d20 con 0x7f92e8072580 2024-03-29T11:36:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.096+0000 7f92f0200700 1 -- 172.21.15.12:0/1306883243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 msgr2=0x7f92e80b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.096+0000 7f92f0200700 1 --2- 172.21.15.12:0/1306883243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 0x7f92e80b2910 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009870 tx=0x7f92d4009b80 comp rx=0 tx=0).stop 2024-03-29T11:36:11.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.096+0000 7f92f0200700 1 -- 172.21.15.12:0/1306883243 shutdown_connections 2024-03-29T11:36:11.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.096+0000 7f92f0200700 1 --2- 172.21.15.12:0/1306883243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 0x7f92e80b2910 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.096+0000 7f92f0200700 1 -- 172.21.15.12:0/1306883243 >> 172.21.15.12:0/1306883243 conn(0x7f92e80cf190 msgr2=0x7f92e80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:11.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.096+0000 7f92f0200700 1 -- 172.21.15.12:0/1306883243 shutdown_connections 2024-03-29T11:36:11.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.096+0000 7f92f0200700 1 -- 172.21.15.12:0/1306883243 wait complete. 2024-03-29T11:36:11.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.098+0000 7f92f0200700 1 Processor -- start 2024-03-29T11:36:11.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.098+0000 7f92f0200700 1 -- start start 2024-03-29T11:36:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.099+0000 7f92f0200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 0x7f92e80ce660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.099+0000 7f92f0200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92e80b2e50 0x7f92e8140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.099+0000 7f92f0200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92e80b3af0 0x7f92e80afca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.099+0000 7f92f0200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e81411c0 con 0x7f92e80b2e50 2024-03-29T11:36:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.099+0000 7f92f0200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e8140230 con 0x7f92e80b3af0 2024-03-29T11:36:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.099+0000 7f92f0200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e81407f0 con 0x7f92e8072580 2024-03-29T11:36:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ece00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92e80b2e50 0x7f92e8140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ece00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92e80b2e50 0x7f92e8140c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33512/0 (socket says 172.21.15.12:33512) 2024-03-29T11:36:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ece00700 1 -- 172.21.15.12:0/2930612901 learned_addr learned my addr 172.21.15.12:0/2930612901 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ed800700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 0x7f92e80ce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ee200700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92e80b3af0 0x7f92e80afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ece00700 1 -- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 msgr2=0x7f92e80ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ece00700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 0x7f92e80ce660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ece00700 1 -- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92e80b3af0 msgr2=0x7f92e80afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ece00700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92e80b3af0 0x7f92e80afca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ece00700 1 -- 172.21.15.12:0/2930612901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d40094a0 con 0x7f92e80b2e50 2024-03-29T11:36:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.100+0000 7f92ece00700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92e80b2e50 0x7f92e8140c80 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f92d40090b0 tx=0x7f92d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.101+0000 7f92e5600700 1 -- 172.21.15.12:0/2930612901 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d401a9f0 con 0x7f92e80b2e50 2024-03-29T11:36:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.101+0000 7f92e5600700 1 -- 172.21.15.12:0/2930612901 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d401ab50 con 0x7f92e80b2e50 2024-03-29T11:36:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.101+0000 7f92e5600700 1 -- 172.21.15.12:0/2930612901 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d40243f0 con 0x7f92e80b2e50 2024-03-29T11:36:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.101+0000 7f92f0200700 1 -- 172.21.15.12:0/2930612901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e81f2f80 con 0x7f92e80b2e50 2024-03-29T11:36:11.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.101+0000 7f92f0200700 1 -- 172.21.15.12:0/2930612901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e80b0940 con 0x7f92e80b2e50 2024-03-29T11:36:11.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.102+0000 7f92f0200700 1 -- 172.21.15.12:0/2930612901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e8067720 con 0x7f92e80b2e50 2024-03-29T11:36:11.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.103+0000 7f92e5600700 1 -- 172.21.15.12:0/2930612901 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f92d402d450 con 0x7f92e80b2e50 2024-03-29T11:36:11.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.104+0000 7f92e5600700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f92c806ade0 0x7f92c806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.104+0000 7f92e5600700 1 -- 172.21.15.12:0/2930612901 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f92d408a770 con 0x7f92e80b2e50 2024-03-29T11:36:11.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.105+0000 7f92ed800700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f92c806ade0 0x7f92c806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.105+0000 7f92ed800700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f92c806ade0 0x7f92c806d2a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f92dc0011e0 tx=0x7f92dc009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:11.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.109+0000 7f92e5600700 1 -- 172.21.15.12:0/2930612901 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92d40559c0 con 0x7f92e80b2e50 2024-03-29T11:36:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.320+0000 7f92f0200700 1 -- 172.21.15.12:0/2930612901 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92e80b3ed0 con 0x7f92c806ade0 2024-03-29T11:36:11.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.323+0000 7f92e5600700 1 -- 172.21.15.12:0/2930612901 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50494 (secure 0 0 0) 0x7f92dc001cb0 con 0x7f92c806ade0 2024-03-29T11:36:11.323 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 -- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f92c806ade0 msgr2=0x7f92c806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f92c806ade0 0x7f92c806d2a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f92dc0011e0 tx=0x7f92dc009040 comp rx=0 tx=0).stop 2024-03-29T11:36:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 -- 172.21.15.12:0/2930612901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92e80b2e50 msgr2=0x7f92e8140c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92e80b2e50 0x7f92e8140c80 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f92d40090b0 tx=0x7f92d4004750 comp rx=0 tx=0).stop 2024-03-29T11:36:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 -- 172.21.15.12:0/2930612901 shutdown_connections 2024-03-29T11:36:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f92c806ade0 0x7f92c806d2a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92e80b3af0 0x7f92e80afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e8072580 0x7f92e80ce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 --2- 172.21.15.12:0/2930612901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92e80b2e50 0x7f92e8140c80 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 -- 172.21.15.12:0/2930612901 >> 172.21.15.12:0/2930612901 conn(0x7f92e80cf190 msgr2=0x7f92e806d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 -- 172.21.15.12:0/2930612901 shutdown_connections 2024-03-29T11:36:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.326+0000 7f92f0200700 1 -- 172.21.15.12:0/2930612901 wait complete. 2024-03-29T11:36:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:36:11.342 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-29T11:36:10.186134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":467344,"kb_used_data":29796,"kb_used_omap":12,"kb_used_meta":437427,"kb_avail":754507376,"statfs":{"total":773094113280,"available":772615553024,"internally_reserved":0,"allocated":30511104,"data_stored":15641827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":447925817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766062,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-289,"num_write_kb":-2297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.426011"},"pg_stats":[{"pgid":"1.0","version":"288'454","reported_seq":1639,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-29T11:36:06.175425+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:36:06.175425+0000","last_peered":"2024-03-29T11:36:06.175425+0000","last_clean":"2024-03-29T11:36:06.175425+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:36:06.175425+0000","last_undegraded":"2024-03-29T11:36:06.175425+0000","last_fullsized":"2024-03-29T11:36:06.175425+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576042,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57424,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314416,"statfs":{"total":96636764160,"available":96577961984,"internally_reserved":0,"allocated":4263936,"data_stored":2399728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.67100000000000004,"15min":0.70899999999999996},"min":{"1min":0.47899999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91600000000000004,"5min":1.155,"15min":1.155},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68200000000000005,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94699999999999995,"5min":1.137,"15min":1.137},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.66500000000000004,"15min":0.71399999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.026,"5min":1.026,"15min":1.478},"last":0.755},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.69399999999999995,"15min":0.745},"min":{"1min":0.57299999999999995,"5min":0.434,"15min":0.434},"max":{"1min":0.96199999999999997,"5min":1.0640000000000001,"15min":1.393},"last":0.503}]},{"osd":2,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.69199999999999995,"15min":0.73899999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.008,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.71699999999999997,"15min":0.75600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.99299999999999999,"5min":1.04,"15min":1.0720000000000001},"last":0.47499999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.69099999999999995,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.0580000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.77000000000000002,"15min":0.78400000000000003},"min":{"1min":0.54000000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.0620000000000001,"5min":2.6930000000000001,"15min":2.6930000000000001},"last":0.71499999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.84799999999999998,"5min":0.95499999999999996,"15min":1.0760000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.46600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":0.93100000000000005,"15min":0.98199999999999998},"last":0.81799999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69199999999999995,"15min":0.68400000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0920000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.48999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0129999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.53700000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.036,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73599999999999999,"15min":0.73799999999999999},"min":{"1min":0.52300000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.98299999999999998,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.79200000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48652,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323188,"statfs":{"total":96636764160,"available":96586944512,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.65300000000000002,"15min":0.71099999999999997},"min":{"1min":0.55700000000000005,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.876,"5min":1.0409999999999999,"15min":1.2150000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.67600000000000005,"15min":0.73499999999999999},"min":{"1min":0.61799999999999999,"5min":0.40200000000000002,"15min":0.39100000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.4239999999999999},"last":0.73399999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65200000000000002,"15min":0.71099999999999997},"min":{"1min":0.54300000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.81399999999999995,"5min":1.022,"15min":1.98},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.71199999999999997,"15min":0.753},"min":{"1min":0.498,"5min":0.435,"15min":0.435},"max":{"1min":0.97699999999999998,"5min":1.0489999999999999,"15min":1.8200000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.69199999999999995,"15min":0.73299999999999998},"min":{"1min":0.57799999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92600000000000005,"5min":0.92800000000000005,"15min":1.669},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.69499999999999995,"15min":0.747},"min":{"1min":0.61199999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.98599999999999999,"5min":1.167,"15min":1.7589999999999999},"last":0.97299999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.69999999999999996,"15min":0.749},"min":{"1min":0.67900000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91900000000000004,"5min":0.997,"15min":2.52},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.78500000000000003},"min":{"1min":0.55600000000000005,"5min":0.51200000000000001,"15min":0.47799999999999998},"max":{"1min":1.0249999999999999,"5min":1.1160000000000001,"15min":1.877},"last":0.82999999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.439,"5min":0.376,"15min":0.376},"max":{"1min":0.87,"5min":0.93799999999999994,"15min":2.4100000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.70599999999999996,"15min":0.71099999999999997},"min":{"1min":0.51700000000000002,"5min":0.46000000000000002,"15min":0.45900000000000002},"max":{"1min":0.93200000000000005,"5min":1.2709999999999999,"15min":2.6629999999999998},"last":1.038}]},{"osd":5,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.438,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72899999999999998,"5min":1.0900000000000001,"15min":1.5960000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88500000000000001,"5min":1.1180000000000001,"15min":1.925},"last":0.89500000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.76400000000000001,"15min":0.76900000000000002},"min":{"1min":0.58499999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88300000000000001,"5min":1.056,"15min":2.7050000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71699999999999997,"15min":0.73899999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81399999999999995,"5min":1.1220000000000001,"15min":1.726},"last":0.93600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576041,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48636,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323204,"statfs":{"total":96636764160,"available":96586960896,"internally_reserved":0,"allocated":3653632,"data_stored":1802975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.379,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56899999999999995,"5min":0.71099999999999997,"15min":0.80300000000000005},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.5,"15min":0.52600000000000002},"min":{"1min":0.33500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.79000000000000004,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72899999999999998,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.503,"15min":0.52100000000000002},"min":{"1min":0.39700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71499999999999997,"5min":0.75900000000000001,"15min":0.90300000000000002},"last":0.58199999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.36299999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.995},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53500000000000003,"15min":0.56699999999999995},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76000000000000001,"5min":0.77700000000000002,"15min":0.83999999999999997},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.77300000000000002},"min":{"1min":0.60699999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.008,"5min":1.0589999999999999,"15min":1.238},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.75600000000000001,"15min":0.78700000000000003},"min":{"1min":0.59499999999999997,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":1.0169999999999999,"5min":1.097,"15min":1.304},"last":0.71199999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78400000000000003,"15min":0.80500000000000005},"min":{"1min":0.60999999999999999,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":1.069,"5min":1.1359999999999999,"15min":1.3720000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.77800000000000002,"15min":0.81499999999999995},"min":{"1min":0.55600000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96199999999999997,"5min":1.2030000000000001,"15min":1.3520000000000001},"last":0.72699999999999998}]},{"osd":6,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.77800000000000002,"15min":0.80400000000000005},"min":{"1min":0.57099999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.03,"5min":1.103,"15min":1.1919999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.80600000000000005,"15min":0.83299999999999996},"min":{"1min":0.63900000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.0309999999999999,"5min":1.2170000000000001,"15min":1.266},"last":0.69499999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81899999999999995,"15min":0.85299999999999998},"min":{"1min":0.63200000000000001,"5min":0.54800000000000004,"15min":0.54500000000000004},"max":{"1min":1.0089999999999999,"5min":1.129,"15min":1.327},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.81599999999999995,"15min":0.85099999999999998},"min":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.57499999999999996},"max":{"1min":1.0549999999999999,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.77200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65616,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306224,"statfs":{"total":96636764160,"available":96569573376,"internally_reserved":0,"allocated":4259840,"data_stored":2398709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.46300000000000002},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.23000000000000001},"max":{"1min":0.65500000000000003,"5min":0.83999999999999997,"15min":1.087},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.499,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70399999999999996,"5min":0.77400000000000002,"15min":1.3089999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.53400000000000003,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.754,"5min":0.84999999999999998,"15min":1.5289999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47599999999999998,"15min":0.51500000000000001},"min":{"1min":0.372,"5min":0.308,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":1.6140000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.52500000000000002,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78300000000000003,"5min":0.81000000000000005,"15min":1.222},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59299999999999997,"15min":0.63500000000000001},"min":{"1min":0.49199999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.876,"5min":0.876,"15min":1.3899999999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.76700000000000002,"15min":0.80300000000000005},"min":{"1min":0.70499999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0800000000000001,"5min":1.248,"15min":1.248},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.79400000000000004,"15min":0.83799999999999997},"min":{"1min":0.626,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.5820000000000001},"last":0.76700000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.72999999999999998,"15min":0.77300000000000002},"min":{"1min":0.63600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.3440000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.77700000000000002,"15min":0.81399999999999995},"min":{"1min":0.749,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.113,"5min":1.1499999999999999,"15min":1.3640000000000001},"last":0.79600000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.78200000000000003,"15min":0.84199999999999997},"min":{"1min":0.60699999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.6240000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.82299999999999995,"15min":0.871},"min":{"1min":0.751,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.337},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.78100000000000003,"15min":0.81599999999999995},"min":{"1min":0.71099999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.214},"last":0.746},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.81000000000000005,"15min":0.86099999999999999},"min":{"1min":0.76300000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.365},"last":0.67500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576042,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52732,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319108,"statfs":{"total":96636764160,"available":96582766592,"internally_reserved":0,"allocated":3653632,"data_stored":1802975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45600000000000002,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.29199999999999998,"15min":0.27800000000000002},"max":{"1min":0.63600000000000001,"5min":0.73199999999999998,"15min":0.75900000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.49099999999999999,"15min":0.52700000000000002},"min":{"1min":0.38500000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.96799999999999997},"last":0.51200000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.47699999999999998},"min":{"1min":0.35899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69599999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67000000000000004,"5min":0.872,"15min":0.88800000000000001},"last":0.625}]},{"osd":3,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58099999999999996},"min":{"1min":0.42999999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.86799999999999999,"15min":0.98499999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.56899999999999995},"min":{"1min":0.38800000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.038},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.75600000000000001,"15min":0.79500000000000004},"min":{"1min":0.54000000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.026,"5min":1.1100000000000001,"15min":1.1970000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.74399999999999999,"15min":0.79000000000000004},"min":{"1min":0.55300000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.0049999999999999,"5min":1.157,"15min":1.244},"last":0.76300000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.024,"5min":1.171,"15min":1.256},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.77100000000000002,"15min":0.81200000000000006},"min":{"1min":0.57399999999999995,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0649999999999999,"5min":1.1180000000000001,"15min":1.3140000000000001},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.79100000000000004,"15min":0.83299999999999996},"min":{"1min":0.62,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0800000000000001,"5min":1.21,"15min":1.24},"last":1},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.78200000000000003,"15min":0.82599999999999996},"min":{"1min":0.60399999999999998,"5min":0.5,"15min":0.5},"max":{"1min":1.036,"5min":1.0920000000000001,"15min":1.329},"last":1.0189999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.83499999999999996,"15min":0.879},"min":{"1min":0.68200000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.087,"5min":1.151,"15min":1.2889999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.84299999999999997,"15min":0.88600000000000001},"min":{"1min":0.63600000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1040000000000001,"5min":1.234,"15min":1.304},"last":1.1499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576042,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48652,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323188,"statfs":{"total":96636764160,"available":96586944512,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.34699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.65200000000000002,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.38},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51800000000000002,"15min":0.54000000000000004},"min":{"1min":0.36399999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48599999999999999,"15min":0.49099999999999999},"min":{"1min":0.39300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80200000000000005,"5min":1.821,"15min":1.821},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.56299999999999994},"min":{"1min":0.48599999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94599999999999995,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.627,"15min":0.65000000000000002},"min":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81499999999999995,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.77500000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.753,"15min":0.78100000000000003},"min":{"1min":0.66200000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0609999999999999,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77100000000000002,"15min":0.80000000000000004},"min":{"1min":0.61199999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0820000000000001,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.56999999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77800000000000002,"15min":0.79900000000000004},"min":{"1min":0.629,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.089,"5min":2.157,"15min":2.157},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1699999999999999,"5min":2.266,"15min":2.266},"last":0.61799999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.82199999999999995,"15min":0.84699999999999998},"min":{"1min":0.68200000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.121,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.872,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.66300000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.137,"5min":2.198,"15min":2.198},"last":0.65000000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.83599999999999997,"15min":0.84999999999999998},"min":{"1min":0.68100000000000005,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":1.111,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.85999999999999999,"15min":0.879},"min":{"1min":0.64400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.153,"5min":2.444,"15min":2.444},"last":0.97399999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96976,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274864,"statfs":{"total":96636764160,"available":96537460736,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.65600000000000003,"15min":0.69899999999999995},"min":{"1min":0.433,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.64900000000000002,"15min":0.69899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88200000000000001,"5min":1.02,"15min":1.02},"last":0.68500000000000005}]},{"osd":1,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.67100000000000004,"15min":0.71899999999999997},"min":{"1min":0.58299999999999996,"5min":0.436,"15min":0.432},"max":{"1min":0.96799999999999997,"5min":1.135,"15min":1.135},"last":0.625},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.68100000000000005,"15min":0.72399999999999998},"min":{"1min":0.54700000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97099999999999997,"5min":1.004,"15min":1.004},"last":0.76600000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.76800000000000002,"15min":0.80700000000000005},"min":{"1min":0.70099999999999996,"5min":0.52100000000000002,"15min":0.49199999999999999},"max":{"1min":1.016,"5min":1.8,"15min":1.8},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.74199999999999999,"15min":0.79100000000000004},"min":{"1min":0.53200000000000003,"5min":0.503,"15min":0.496},"max":{"1min":0.96599999999999997,"5min":1.0369999999999999,"15min":1.224},"last":0.82499999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.69399999999999995,"15min":0.73699999999999999},"min":{"1min":0.55600000000000005,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92300000000000004,"5min":0.96699999999999997,"15min":1.0580000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.71999999999999997,"15min":0.755},"min":{"1min":0.60599999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.88200000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.65200000000000002,"15min":0.63200000000000001},"min":{"1min":0.48099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.872,"5min":1.032,"15min":1.032},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63900000000000001,"15min":0.64500000000000002},"min":{"1min":0.505,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81299999999999994,"5min":1.01,"15min":1.0920000000000001},"last":0.80300000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.69799999999999995,"15min":0.67500000000000004},"min":{"1min":0.48999999999999999,"5min":0.44700000000000001,"15min":0.40899999999999997},"max":{"1min":0.876,"5min":1.722,"15min":1.722},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73199999999999998,"15min":0.72099999999999997},"min":{"1min":0.57599999999999996,"5min":0.439,"15min":0.439},"max":{"1min":0.93300000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.45700000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81299999999999994,"15min":0.81699999999999995},"min":{"1min":0.68400000000000005,"5min":0.57499999999999996,"15min":0.54400000000000004},"max":{"1min":0.98199999999999998,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.78500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576042,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48656,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323184,"statfs":{"total":96636764160,"available":96586940416,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.63400000000000001,"15min":0.68400000000000005},"min":{"1min":0.52200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.66200000000000003,"15min":0.71199999999999997},"min":{"1min":0.54700000000000004,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0680000000000001},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.70199999999999996,"15min":0.746},"min":{"1min":0.58699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0980000000000001,"5min":1.839,"15min":1.839},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.69699999999999995,"15min":0.72399999999999998},"min":{"1min":0.499,"5min":0.438,"15min":0.438},"max":{"1min":0.94599999999999995,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.70899999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.70799999999999996,"15min":0.74299999999999999},"min":{"1min":0.60599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69799999999999995,"15min":0.73499999999999999},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97599999999999998,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.91400000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.75,"15min":0.76400000000000001},"min":{"1min":0.70099999999999996,"5min":0.443,"15min":0.443},"max":{"1min":0.94999999999999996,"5min":3.444,"15min":3.444},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.72299999999999998,"15min":0.76400000000000001},"min":{"1min":0.63800000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.996,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.84999999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.439,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84799999999999998,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.95699999999999996,"5min":0.999,"15min":1.0660000000000001},"last":0.76900000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.67500000000000004},"min":{"1min":0.496,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92300000000000004,"5min":1.214,"15min":1.214},"last":0.751},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.69799999999999995,"15min":0.70399999999999996},"min":{"1min":0.58099999999999996,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":1.0900000000000001,"15min":1.107},"last":0.98799999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71899999999999997,"15min":0.73199999999999998},"min":{"1min":0.48899999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.89000000000000001,"5min":1.04,"15min":1.2010000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.55600000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89100000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.89300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:36:11.344 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:36:11.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.548+0000 7f1f90e00700 1 Processor -- start 2024-03-29T11:36:11.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.548+0000 7f1f90e00700 1 -- start start 2024-03-29T11:36:11.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.549+0000 7f1f90e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f8c0cf3c0 0x7f1f8c0d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.549+0000 7f1f90e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1f8c15db40 0x7f1f8c0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.549+0000 7f1f90e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c0d1330 con 0x7f1f8c15d760 2024-03-29T11:36:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.549+0000 7f1f90e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8c0d1fb0 con 0x7f1f8c0cf3c0 2024-03-29T11:36:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.549+0000 7f1f90e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8c0d1610 con 0x7f1f8c15db40 2024-03-29T11:36:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.549+0000 7f1f89400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f8c0cf3c0 0x7f1f8c0d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.550+0000 7f1f89400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f8c0cf3c0 0x7f1f8c0d23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33528/0 (socket says 172.21.15.12:33528) 2024-03-29T11:36:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.550+0000 7f1f89400700 1 -- 172.21.15.12:0/1413185868 learned_addr learned my addr 172.21.15.12:0/1413185868 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.550+0000 7f1f8a800700 1 --2- 172.21.15.12:0/1413185868 >> v2:172.21.15.159:3300/0 conn(0x7f1f8c15db40 0x7f1f8c0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.550+0000 7f1f89400700 1 -- 172.21.15.12:0/1413185868 >> v2:172.21.15.159:3300/0 conn(0x7f1f8c15db40 msgr2=0x7f1f8c0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:11.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.550+0000 7f1f89400700 1 --2- 172.21.15.12:0/1413185868 >> v2:172.21.15.159:3300/0 conn(0x7f1f8c15db40 0x7f1f8c0cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.550+0000 7f1f89400700 1 -- 172.21.15.12:0/1413185868 >> v1:172.21.15.159:6789/0 conn(0x7f1f8c15d760 legacy=0x7f1f8c0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.550+0000 7f1f89400700 1 -- 172.21.15.12:0/1413185868 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c0d18f0 con 0x7f1f8c0cf3c0 2024-03-29T11:36:11.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.550+0000 7f1f89400700 1 --2- 172.21.15.12:0/1413185868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c0cf3c0 0x7f1f8c0d23d0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0c6de0 tx=0x7f1f78009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd430d58c539da2d server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:11.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.550+0000 7f1f88a00700 1 -- 172.21.15.12:0/1413185868 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7800b870 con 0x7f1f8c0cf3c0 2024-03-29T11:36:11.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.550+0000 7f1f88a00700 1 -- 172.21.15.12:0/1413185868 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7800b9d0 con 0x7f1f8c0cf3c0 2024-03-29T11:36:11.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.551+0000 7f1f88a00700 1 -- 172.21.15.12:0/1413185868 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7800bd60 con 0x7f1f8c0cf3c0 2024-03-29T11:36:11.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.551+0000 7f1f90e00700 1 -- 172.21.15.12:0/1413185868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c0cf3c0 msgr2=0x7f1f8c0d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.551+0000 7f1f90e00700 1 --2- 172.21.15.12:0/1413185868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c0cf3c0 0x7f1f8c0d23d0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0c6de0 tx=0x7f1f78009bd0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.551+0000 7f1f90e00700 1 -- 172.21.15.12:0/1413185868 shutdown_connections 2024-03-29T11:36:11.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.551+0000 7f1f90e00700 1 --2- 172.21.15.12:0/1413185868 >> v2:172.21.15.159:3300/0 conn(0x7f1f8c15db40 0x7f1f8c0cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.551+0000 7f1f90e00700 1 --2- 172.21.15.12:0/1413185868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c0cf3c0 0x7f1f8c0d23d0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.551+0000 7f1f90e00700 1 -- 172.21.15.12:0/1413185868 >> 172.21.15.12:0/1413185868 conn(0x7f1f8c08aff0 msgr2=0x7f1f8c0d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:11.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.551+0000 7f1f90e00700 1 -- 172.21.15.12:0/1413185868 shutdown_connections 2024-03-29T11:36:11.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.551+0000 7f1f90e00700 1 -- 172.21.15.12:0/1413185868 wait complete. 2024-03-29T11:36:11.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.553+0000 7f1f90e00700 1 Processor -- start 2024-03-29T11:36:11.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.553+0000 7f1f90e00700 1 -- start start 2024-03-29T11:36:11.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f90e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f8c0cf3c0 0x7f1f8c160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f90e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f8c15d760 0x7f1f8c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f90e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c15db40 0x7f1f8c15c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f90e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c0c8560 con 0x7f1f8c15db40 2024-03-29T11:36:11.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f90e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c0cc590 con 0x7f1f8c0cf3c0 2024-03-29T11:36:11.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f90e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c0cba10 con 0x7f1f8c15d760 2024-03-29T11:36:11.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f8a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c15db40 0x7f1f8c15c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f89e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f8c0cf3c0 0x7f1f8c160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f8a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c15db40 0x7f1f8c15c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33532/0 (socket says 172.21.15.12:33532) 2024-03-29T11:36:11.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f89400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f8c15d760 0x7f1f8c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.554+0000 7f1f8a800700 1 -- 172.21.15.12:0/1241193505 learned_addr learned my addr 172.21.15.12:0/1241193505 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:11.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f8a800700 1 -- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f8c15d760 msgr2=0x7f1f8c0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f8a800700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f8c15d760 0x7f1f8c0cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f8a800700 1 -- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f8c0cf3c0 msgr2=0x7f1f8c160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f8a800700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f8c0cf3c0 0x7f1f8c160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f8a800700 1 -- 172.21.15.12:0/1241193505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78009550 con 0x7f1f8c15db40 2024-03-29T11:36:11.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f8a800700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c15db40 0x7f1f8c15c090 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f1f84009fd0 tx=0x7f1f8400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:11.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f82000700 1 -- 172.21.15.12:0/1241193505 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84007ea0 con 0x7f1f8c15db40 2024-03-29T11:36:11.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f82000700 1 -- 172.21.15.12:0/1241193505 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8400ed30 con 0x7f1f8c15db40 2024-03-29T11:36:11.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f90e00700 1 -- 172.21.15.12:0/1241193505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8c0cbcf0 con 0x7f1f8c15db40 2024-03-29T11:36:11.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f82000700 1 -- 172.21.15.12:0/1241193505 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8400eea0 con 0x7f1f8c15db40 2024-03-29T11:36:11.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.555+0000 7f1f90e00700 1 -- 172.21.15.12:0/1241193505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8c0cc2b0 con 0x7f1f8c15db40 2024-03-29T11:36:11.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.556+0000 7f1f90e00700 1 -- 172.21.15.12:0/1241193505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8c0c86f0 con 0x7f1f8c15db40 2024-03-29T11:36:11.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.557+0000 7f1f82000700 1 -- 172.21.15.12:0/1241193505 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1f84014660 con 0x7f1f8c15db40 2024-03-29T11:36:11.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.559+0000 7f1f82000700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1f6c06adb0 0x7f1f6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:11.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.559+0000 7f1f82000700 1 -- 172.21.15.12:0/1241193505 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f1f84089bb0 con 0x7f1f8c15db40 2024-03-29T11:36:11.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.559+0000 7f1f89e00700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1f6c06adb0 0x7f1f6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:11.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.560+0000 7f1f89e00700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1f6c06adb0 0x7f1f6c06d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0c6b80 tx=0x7f1f7c009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:11.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.563+0000 7f1f82000700 1 -- 172.21.15.12:0/1241193505 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f8c0c86f0 con 0x7f1f8c15db40 2024-03-29T11:36:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.777+0000 7f1f90e00700 1 -- 172.21.15.12:0/1241193505 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f8c0c86f0 con 0x7f1f6c06adb0 2024-03-29T11:36:11.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.780+0000 7f1f82000700 1 -- 172.21.15.12:0/1241193505 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50494 (secure 0 0 0) 0x7f1f8c0c86f0 con 0x7f1f6c06adb0 2024-03-29T11:36:11.780 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:11.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 -- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1f6c06adb0 msgr2=0x7f1f6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1f6c06adb0 0x7f1f6c06d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0c6b80 tx=0x7f1f7c009040 comp rx=0 tx=0).stop 2024-03-29T11:36:11.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 -- 172.21.15.12:0/1241193505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c15db40 msgr2=0x7f1f8c15c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:11.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c15db40 0x7f1f8c15c090 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f1f84009fd0 tx=0x7f1f8400e3c0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 -- 172.21.15.12:0/1241193505 shutdown_connections 2024-03-29T11:36:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1f6c06adb0 0x7f1f6c06d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c15db40 0x7f1f8c15c090 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f8c0cf3c0 0x7f1f8c160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 --2- 172.21.15.12:0/1241193505 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f8c15d760 0x7f1f8c0cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 -- 172.21.15.12:0/1241193505 >> 172.21.15.12:0/1241193505 conn(0x7f1f8c08aff0 msgr2=0x7f1f8c0b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 -- 172.21.15.12:0/1241193505 shutdown_connections 2024-03-29T11:36:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:11.783+0000 7f1f90e00700 1 -- 172.21.15.12:0/1241193505 wait complete. 2024-03-29T11:36:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:36:11.800 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-29T11:36:10.186134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":467344,"kb_used_data":29796,"kb_used_omap":12,"kb_used_meta":437427,"kb_avail":754507376,"statfs":{"total":773094113280,"available":772615553024,"internally_reserved":0,"allocated":30511104,"data_stored":15641827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":447925817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766062,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-289,"num_write_kb":-2297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.426011"},"pg_stats":[{"pgid":"1.0","version":"288'454","reported_seq":1639,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-29T11:36:06.175425+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:36:06.175425+0000","last_peered":"2024-03-29T11:36:06.175425+0000","last_clean":"2024-03-29T11:36:06.175425+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:36:06.175425+0000","last_undegraded":"2024-03-29T11:36:06.175425+0000","last_fullsized":"2024-03-29T11:36:06.175425+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576042,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57424,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314416,"statfs":{"total":96636764160,"available":96577961984,"internally_reserved":0,"allocated":4263936,"data_stored":2399728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.67100000000000004,"15min":0.70899999999999996},"min":{"1min":0.47899999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91600000000000004,"5min":1.155,"15min":1.155},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68200000000000005,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94699999999999995,"5min":1.137,"15min":1.137},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.66500000000000004,"15min":0.71399999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.026,"5min":1.026,"15min":1.478},"last":0.755},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.69399999999999995,"15min":0.745},"min":{"1min":0.57299999999999995,"5min":0.434,"15min":0.434},"max":{"1min":0.96199999999999997,"5min":1.0640000000000001,"15min":1.393},"last":0.503}]},{"osd":2,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.69199999999999995,"15min":0.73899999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.008,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.71699999999999997,"15min":0.75600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.99299999999999999,"5min":1.04,"15min":1.0720000000000001},"last":0.47499999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.69099999999999995,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.0580000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.77000000000000002,"15min":0.78400000000000003},"min":{"1min":0.54000000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.0620000000000001,"5min":2.6930000000000001,"15min":2.6930000000000001},"last":0.71499999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.84799999999999998,"5min":0.95499999999999996,"15min":1.0760000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.46600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":0.93100000000000005,"15min":0.98199999999999998},"last":0.81799999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69199999999999995,"15min":0.68400000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0920000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.48999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0129999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.53700000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.036,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73599999999999999,"15min":0.73799999999999999},"min":{"1min":0.52300000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.98299999999999998,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.79200000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48652,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323188,"statfs":{"total":96636764160,"available":96586944512,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.65300000000000002,"15min":0.71099999999999997},"min":{"1min":0.55700000000000005,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.876,"5min":1.0409999999999999,"15min":1.2150000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.67600000000000005,"15min":0.73499999999999999},"min":{"1min":0.61799999999999999,"5min":0.40200000000000002,"15min":0.39100000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.4239999999999999},"last":0.73399999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65200000000000002,"15min":0.71099999999999997},"min":{"1min":0.54300000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.81399999999999995,"5min":1.022,"15min":1.98},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.71199999999999997,"15min":0.753},"min":{"1min":0.498,"5min":0.435,"15min":0.435},"max":{"1min":0.97699999999999998,"5min":1.0489999999999999,"15min":1.8200000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.69199999999999995,"15min":0.73299999999999998},"min":{"1min":0.57799999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92600000000000005,"5min":0.92800000000000005,"15min":1.669},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.69499999999999995,"15min":0.747},"min":{"1min":0.61199999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.98599999999999999,"5min":1.167,"15min":1.7589999999999999},"last":0.97299999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.69999999999999996,"15min":0.749},"min":{"1min":0.67900000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91900000000000004,"5min":0.997,"15min":2.52},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.78500000000000003},"min":{"1min":0.55600000000000005,"5min":0.51200000000000001,"15min":0.47799999999999998},"max":{"1min":1.0249999999999999,"5min":1.1160000000000001,"15min":1.877},"last":0.82999999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.439,"5min":0.376,"15min":0.376},"max":{"1min":0.87,"5min":0.93799999999999994,"15min":2.4100000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.70599999999999996,"15min":0.71099999999999997},"min":{"1min":0.51700000000000002,"5min":0.46000000000000002,"15min":0.45900000000000002},"max":{"1min":0.93200000000000005,"5min":1.2709999999999999,"15min":2.6629999999999998},"last":1.038}]},{"osd":5,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.438,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72899999999999998,"5min":1.0900000000000001,"15min":1.5960000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88500000000000001,"5min":1.1180000000000001,"15min":1.925},"last":0.89500000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.76400000000000001,"15min":0.76900000000000002},"min":{"1min":0.58499999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88300000000000001,"5min":1.056,"15min":2.7050000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71699999999999997,"15min":0.73899999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81399999999999995,"5min":1.1220000000000001,"15min":1.726},"last":0.93600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576041,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48636,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323204,"statfs":{"total":96636764160,"available":96586960896,"internally_reserved":0,"allocated":3653632,"data_stored":1802975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.379,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56899999999999995,"5min":0.71099999999999997,"15min":0.80300000000000005},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.5,"15min":0.52600000000000002},"min":{"1min":0.33500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.79000000000000004,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72899999999999998,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.503,"15min":0.52100000000000002},"min":{"1min":0.39700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71499999999999997,"5min":0.75900000000000001,"15min":0.90300000000000002},"last":0.58199999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.36299999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.995},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53500000000000003,"15min":0.56699999999999995},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76000000000000001,"5min":0.77700000000000002,"15min":0.83999999999999997},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.77300000000000002},"min":{"1min":0.60699999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.008,"5min":1.0589999999999999,"15min":1.238},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.75600000000000001,"15min":0.78700000000000003},"min":{"1min":0.59499999999999997,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":1.0169999999999999,"5min":1.097,"15min":1.304},"last":0.71199999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78400000000000003,"15min":0.80500000000000005},"min":{"1min":0.60999999999999999,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":1.069,"5min":1.1359999999999999,"15min":1.3720000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.77800000000000002,"15min":0.81499999999999995},"min":{"1min":0.55600000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96199999999999997,"5min":1.2030000000000001,"15min":1.3520000000000001},"last":0.72699999999999998}]},{"osd":6,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.77800000000000002,"15min":0.80400000000000005},"min":{"1min":0.57099999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.03,"5min":1.103,"15min":1.1919999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.80600000000000005,"15min":0.83299999999999996},"min":{"1min":0.63900000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.0309999999999999,"5min":1.2170000000000001,"15min":1.266},"last":0.69499999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81899999999999995,"15min":0.85299999999999998},"min":{"1min":0.63200000000000001,"5min":0.54800000000000004,"15min":0.54500000000000004},"max":{"1min":1.0089999999999999,"5min":1.129,"15min":1.327},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.81599999999999995,"15min":0.85099999999999998},"min":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.57499999999999996},"max":{"1min":1.0549999999999999,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.77200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65616,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306224,"statfs":{"total":96636764160,"available":96569573376,"internally_reserved":0,"allocated":4259840,"data_stored":2398709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.46300000000000002},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.23000000000000001},"max":{"1min":0.65500000000000003,"5min":0.83999999999999997,"15min":1.087},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.499,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70399999999999996,"5min":0.77400000000000002,"15min":1.3089999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.53400000000000003,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.754,"5min":0.84999999999999998,"15min":1.5289999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47599999999999998,"15min":0.51500000000000001},"min":{"1min":0.372,"5min":0.308,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":1.6140000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.52500000000000002,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78300000000000003,"5min":0.81000000000000005,"15min":1.222},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59299999999999997,"15min":0.63500000000000001},"min":{"1min":0.49199999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.876,"5min":0.876,"15min":1.3899999999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.76700000000000002,"15min":0.80300000000000005},"min":{"1min":0.70499999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0800000000000001,"5min":1.248,"15min":1.248},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.79400000000000004,"15min":0.83799999999999997},"min":{"1min":0.626,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.5820000000000001},"last":0.76700000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.72999999999999998,"15min":0.77300000000000002},"min":{"1min":0.63600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.3440000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.77700000000000002,"15min":0.81399999999999995},"min":{"1min":0.749,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.113,"5min":1.1499999999999999,"15min":1.3640000000000001},"last":0.79600000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.78200000000000003,"15min":0.84199999999999997},"min":{"1min":0.60699999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.6240000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.82299999999999995,"15min":0.871},"min":{"1min":0.751,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.337},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.78100000000000003,"15min":0.81599999999999995},"min":{"1min":0.71099999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.214},"last":0.746},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.81000000000000005,"15min":0.86099999999999999},"min":{"1min":0.76300000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.365},"last":0.67500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576042,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52732,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319108,"statfs":{"total":96636764160,"available":96582766592,"internally_reserved":0,"allocated":3653632,"data_stored":1802975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45600000000000002,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.29199999999999998,"15min":0.27800000000000002},"max":{"1min":0.63600000000000001,"5min":0.73199999999999998,"15min":0.75900000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.49099999999999999,"15min":0.52700000000000002},"min":{"1min":0.38500000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.96799999999999997},"last":0.51200000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.47699999999999998},"min":{"1min":0.35899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69599999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67000000000000004,"5min":0.872,"15min":0.88800000000000001},"last":0.625}]},{"osd":3,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58099999999999996},"min":{"1min":0.42999999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.86799999999999999,"15min":0.98499999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.56899999999999995},"min":{"1min":0.38800000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.038},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.75600000000000001,"15min":0.79500000000000004},"min":{"1min":0.54000000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.026,"5min":1.1100000000000001,"15min":1.1970000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.74399999999999999,"15min":0.79000000000000004},"min":{"1min":0.55300000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.0049999999999999,"5min":1.157,"15min":1.244},"last":0.76300000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.024,"5min":1.171,"15min":1.256},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.77100000000000002,"15min":0.81200000000000006},"min":{"1min":0.57399999999999995,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0649999999999999,"5min":1.1180000000000001,"15min":1.3140000000000001},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.79100000000000004,"15min":0.83299999999999996},"min":{"1min":0.62,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0800000000000001,"5min":1.21,"15min":1.24},"last":1},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.78200000000000003,"15min":0.82599999999999996},"min":{"1min":0.60399999999999998,"5min":0.5,"15min":0.5},"max":{"1min":1.036,"5min":1.0920000000000001,"15min":1.329},"last":1.0189999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.83499999999999996,"15min":0.879},"min":{"1min":0.68200000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.087,"5min":1.151,"15min":1.2889999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.84299999999999997,"15min":0.88600000000000001},"min":{"1min":0.63600000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1040000000000001,"5min":1.234,"15min":1.304},"last":1.1499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576042,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48652,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323188,"statfs":{"total":96636764160,"available":96586944512,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.34699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.65200000000000002,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.38},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51800000000000002,"15min":0.54000000000000004},"min":{"1min":0.36399999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48599999999999999,"15min":0.49099999999999999},"min":{"1min":0.39300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80200000000000005,"5min":1.821,"15min":1.821},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.56299999999999994},"min":{"1min":0.48599999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94599999999999995,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.627,"15min":0.65000000000000002},"min":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81499999999999995,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.77500000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.753,"15min":0.78100000000000003},"min":{"1min":0.66200000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0609999999999999,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77100000000000002,"15min":0.80000000000000004},"min":{"1min":0.61199999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0820000000000001,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.56999999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77800000000000002,"15min":0.79900000000000004},"min":{"1min":0.629,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.089,"5min":2.157,"15min":2.157},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1699999999999999,"5min":2.266,"15min":2.266},"last":0.61799999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.82199999999999995,"15min":0.84699999999999998},"min":{"1min":0.68200000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.121,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.872,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.66300000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.137,"5min":2.198,"15min":2.198},"last":0.65000000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.83599999999999997,"15min":0.84999999999999998},"min":{"1min":0.68100000000000005,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":1.111,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.85999999999999999,"15min":0.879},"min":{"1min":0.64400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.153,"5min":2.444,"15min":2.444},"last":0.97399999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96976,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274864,"statfs":{"total":96636764160,"available":96537460736,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.65600000000000003,"15min":0.69899999999999995},"min":{"1min":0.433,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.64900000000000002,"15min":0.69899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88200000000000001,"5min":1.02,"15min":1.02},"last":0.68500000000000005}]},{"osd":1,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.67100000000000004,"15min":0.71899999999999997},"min":{"1min":0.58299999999999996,"5min":0.436,"15min":0.432},"max":{"1min":0.96799999999999997,"5min":1.135,"15min":1.135},"last":0.625},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.68100000000000005,"15min":0.72399999999999998},"min":{"1min":0.54700000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97099999999999997,"5min":1.004,"15min":1.004},"last":0.76600000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.76800000000000002,"15min":0.80700000000000005},"min":{"1min":0.70099999999999996,"5min":0.52100000000000002,"15min":0.49199999999999999},"max":{"1min":1.016,"5min":1.8,"15min":1.8},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.74199999999999999,"15min":0.79100000000000004},"min":{"1min":0.53200000000000003,"5min":0.503,"15min":0.496},"max":{"1min":0.96599999999999997,"5min":1.0369999999999999,"15min":1.224},"last":0.82499999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.69399999999999995,"15min":0.73699999999999999},"min":{"1min":0.55600000000000005,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92300000000000004,"5min":0.96699999999999997,"15min":1.0580000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.71999999999999997,"15min":0.755},"min":{"1min":0.60599999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.88200000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.65200000000000002,"15min":0.63200000000000001},"min":{"1min":0.48099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.872,"5min":1.032,"15min":1.032},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63900000000000001,"15min":0.64500000000000002},"min":{"1min":0.505,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81299999999999994,"5min":1.01,"15min":1.0920000000000001},"last":0.80300000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.69799999999999995,"15min":0.67500000000000004},"min":{"1min":0.48999999999999999,"5min":0.44700000000000001,"15min":0.40899999999999997},"max":{"1min":0.876,"5min":1.722,"15min":1.722},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73199999999999998,"15min":0.72099999999999997},"min":{"1min":0.57599999999999996,"5min":0.439,"15min":0.439},"max":{"1min":0.93300000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.45700000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81299999999999994,"15min":0.81699999999999995},"min":{"1min":0.68400000000000005,"5min":0.57499999999999996,"15min":0.54400000000000004},"max":{"1min":0.98199999999999998,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.78500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576042,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48656,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323184,"statfs":{"total":96636764160,"available":96586940416,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.63400000000000001,"15min":0.68400000000000005},"min":{"1min":0.52200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.66200000000000003,"15min":0.71199999999999997},"min":{"1min":0.54700000000000004,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0680000000000001},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.70199999999999996,"15min":0.746},"min":{"1min":0.58699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0980000000000001,"5min":1.839,"15min":1.839},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.69699999999999995,"15min":0.72399999999999998},"min":{"1min":0.499,"5min":0.438,"15min":0.438},"max":{"1min":0.94599999999999995,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.70899999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.70799999999999996,"15min":0.74299999999999999},"min":{"1min":0.60599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69799999999999995,"15min":0.73499999999999999},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97599999999999998,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.91400000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.75,"15min":0.76400000000000001},"min":{"1min":0.70099999999999996,"5min":0.443,"15min":0.443},"max":{"1min":0.94999999999999996,"5min":3.444,"15min":3.444},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.72299999999999998,"15min":0.76400000000000001},"min":{"1min":0.63800000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.996,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.84999999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.439,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84799999999999998,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.95699999999999996,"5min":0.999,"15min":1.0660000000000001},"last":0.76900000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.67500000000000004},"min":{"1min":0.496,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92300000000000004,"5min":1.214,"15min":1.214},"last":0.751},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.69799999999999995,"15min":0.70399999999999996},"min":{"1min":0.58099999999999996,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":1.0900000000000001,"15min":1.107},"last":0.98799999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71899999999999997,"15min":0.73199999999999998},"min":{"1min":0.48899999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.89000000000000001,"5min":1.04,"15min":1.2010000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.55600000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89100000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.89300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:36:11.801 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-29T11:36:11.802 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:36:12.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.001+0000 7fec6a800700 1 Processor -- start 2024-03-29T11:36:12.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.001+0000 7fec6a800700 1 -- start start 2024-03-29T11:36:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.002+0000 7fec6a800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec6415b660 0x7fec640d1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.002+0000 7fec6a800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fec640cfb70 0x7fec6409a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec6a800700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec640cce30 con 0x7fec640c4f30 2024-03-29T11:36:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec6a800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec64099a60 con 0x7fec6415b660 2024-03-29T11:36:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec6a800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec640cd110 con 0x7fec640cfb70 2024-03-29T11:36:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec63400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec6415b660 0x7fec640d1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec63400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fec6415b660 0x7fec640d1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33534/0 (socket says 172.21.15.12:33534) 2024-03-29T11:36:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec63400700 1 -- 172.21.15.12:0/2270711237 learned_addr learned my addr 172.21.15.12:0/2270711237 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec63e00700 1 --2- 172.21.15.12:0/2270711237 >> v2:172.21.15.159:3300/0 conn(0x7fec640cfb70 0x7fec6409a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec63400700 1 -- 172.21.15.12:0/2270711237 >> v2:172.21.15.159:3300/0 conn(0x7fec640cfb70 msgr2=0x7fec6409a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec63400700 1 --2- 172.21.15.12:0/2270711237 >> v2:172.21.15.159:3300/0 conn(0x7fec640cfb70 0x7fec6409a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec63400700 1 -- 172.21.15.12:0/2270711237 >> v1:172.21.15.159:6789/0 conn(0x7fec640c4f30 legacy=0x7fec640cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec63400700 1 -- 172.21.15.12:0/2270711237 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec64099c70 con 0x7fec6415b660 2024-03-29T11:36:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.003+0000 7fec63400700 1 --2- 172.21.15.12:0/2270711237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 0x7fec640d1ed0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7fec640c4b60 tx=0x7fec54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c9aee656cffd4c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec62000700 1 -- 172.21.15.12:0/2270711237 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54004230 con 0x7fec6415b660 2024-03-29T11:36:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec62000700 1 -- 172.21.15.12:0/2270711237 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec54004390 con 0x7fec6415b660 2024-03-29T11:36:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec62000700 1 -- 172.21.15.12:0/2270711237 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec540036a0 con 0x7fec6415b660 2024-03-29T11:36:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec6a800700 1 -- 172.21.15.12:0/2270711237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 msgr2=0x7fec640d1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec6a800700 1 --2- 172.21.15.12:0/2270711237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 0x7fec640d1ed0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7fec640c4b60 tx=0x7fec54009bd0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec6a800700 1 -- 172.21.15.12:0/2270711237 shutdown_connections 2024-03-29T11:36:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec6a800700 1 --2- 172.21.15.12:0/2270711237 >> v2:172.21.15.159:3300/0 conn(0x7fec640cfb70 0x7fec6409a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec6a800700 1 --2- 172.21.15.12:0/2270711237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 0x7fec640d1ed0 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec6a800700 1 -- 172.21.15.12:0/2270711237 >> 172.21.15.12:0/2270711237 conn(0x7fec640d13e0 msgr2=0x7fec640d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec6a800700 1 -- 172.21.15.12:0/2270711237 shutdown_connections 2024-03-29T11:36:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.004+0000 7fec6a800700 1 -- 172.21.15.12:0/2270711237 wait complete. 2024-03-29T11:36:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.006+0000 7fec6a800700 1 Processor -- start 2024-03-29T11:36:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.006+0000 7fec6a800700 1 -- start start 2024-03-29T11:36:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.007+0000 7fec6a800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec640c4f30 0x7fec64159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.007+0000 7fec6a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec640cfb70 0x7fec640caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.007+0000 7fec6a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 0x7fec640b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.007+0000 7fec6a800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec640ca6d0 con 0x7fec6415b660 2024-03-29T11:36:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.007+0000 7fec6a800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec640c9e00 con 0x7fec640c4f30 2024-03-29T11:36:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.007+0000 7fec6a800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6415caf0 con 0x7fec640cfb70 2024-03-29T11:36:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.007+0000 7fec63e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 0x7fec640b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.007+0000 7fec63e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 0x7fec640b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33544/0 (socket says 172.21.15.12:33544) 2024-03-29T11:36:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec63e00700 1 -- 172.21.15.12:0/669232199 learned_addr learned my addr 172.21.15.12:0/669232199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec63400700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec640c4f30 0x7fec64159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec63e00700 1 -- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec640cfb70 msgr2=0x7fec640caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec63e00700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec640cfb70 0x7fec640caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec63e00700 1 -- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec640c4f30 msgr2=0x7fec64159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec63e00700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec640c4f30 0x7fec64159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec63e00700 1 -- 172.21.15.12:0/669232199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec54009550 con 0x7fec6415b660 2024-03-29T11:36:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec63400700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec640c4f30 0x7fec64159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:36:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec63e00700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 0x7fec640b41b0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7fec5c007e30 tx=0x7fec5c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec5b400700 1 -- 172.21.15.12:0/669232199 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c00eea0 con 0x7fec6415b660 2024-03-29T11:36:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec5b400700 1 -- 172.21.15.12:0/669232199 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c014420 con 0x7fec6415b660 2024-03-29T11:36:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec5b400700 1 -- 172.21.15.12:0/669232199 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0147d0 con 0x7fec6415b660 2024-03-29T11:36:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.008+0000 7fec6a800700 1 -- 172.21.15.12:0/669232199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec640ca010 con 0x7fec6415b660 2024-03-29T11:36:12.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.009+0000 7fec6a800700 1 -- 172.21.15.12:0/669232199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6415a240 con 0x7fec6415b660 2024-03-29T11:36:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.010+0000 7fec6a800700 1 -- 172.21.15.12:0/669232199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec6406e470 con 0x7fec6415b660 2024-03-29T11:36:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.010+0000 7fec5b400700 1 -- 172.21.15.12:0/669232199 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fec5c014930 con 0x7fec6415b660 2024-03-29T11:36:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.011+0000 7fec5b400700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fec4406adb0 0x7fec4406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.011+0000 7fec5b400700 1 -- 172.21.15.12:0/669232199 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fec5c089c50 con 0x7fec6415b660 2024-03-29T11:36:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.012+0000 7fec63400700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fec4406adb0 0x7fec4406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.012+0000 7fec63400700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fec4406adb0 0x7fec4406d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fec54009c10 tx=0x7fec54008030 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.016+0000 7fec5b400700 1 -- 172.21.15.12:0/669232199 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec5c08f020 con 0x7fec6415b660 2024-03-29T11:36:12.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.225+0000 7fec6a800700 1 -- 172.21.15.12:0/669232199 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fec640d01b0 con 0x7fec4406adb0 2024-03-29T11:36:12.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.228+0000 7fec5b400700 1 -- 172.21.15.12:0/669232199 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50494 (secure 0 0 0) 0x7fec5401f070 con 0x7fec4406adb0 2024-03-29T11:36:12.227 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 -- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fec4406adb0 msgr2=0x7fec4406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fec4406adb0 0x7fec4406d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fec54009c10 tx=0x7fec54008030 comp rx=0 tx=0).stop 2024-03-29T11:36:12.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 -- 172.21.15.12:0/669232199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 msgr2=0x7fec640b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 0x7fec640b41b0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7fec5c007e30 tx=0x7fec5c00e3c0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 -- 172.21.15.12:0/669232199 shutdown_connections 2024-03-29T11:36:12.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fec4406adb0 0x7fec4406d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec6415b660 0x7fec640b41b0 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec640c4f30 0x7fec64159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 --2- 172.21.15.12:0/669232199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec640cfb70 0x7fec640caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 -- 172.21.15.12:0/669232199 >> 172.21.15.12:0/669232199 conn(0x7fec640d13e0 msgr2=0x7fec6406f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:12.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 -- 172.21.15.12:0/669232199 shutdown_connections 2024-03-29T11:36:12.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.230+0000 7fec6a800700 1 -- 172.21.15.12:0/669232199 wait complete. 2024-03-29T11:36:12.231 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:36:12.247 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-03-29T11:36:12.186500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":467344,"kb_used_data":29796,"kb_used_omap":12,"kb_used_meta":437427,"kb_avail":754507376,"statfs":{"total":773094113280,"available":772615553024,"internally_reserved":0,"allocated":30511104,"data_stored":15641827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":447925817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766062,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-289,"num_write_kb":-2297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001687"},"pg_stats":[{"pgid":"1.0","version":"288'454","reported_seq":1639,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-29T11:36:06.175425+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:36:06.175425+0000","last_peered":"2024-03-29T11:36:06.175425+0000","last_clean":"2024-03-29T11:36:06.175425+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:36:06.175425+0000","last_undegraded":"2024-03-29T11:36:06.175425+0000","last_fullsized":"2024-03-29T11:36:06.175425+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576042,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57424,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314416,"statfs":{"total":96636764160,"available":96577961984,"internally_reserved":0,"allocated":4263936,"data_stored":2399728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.67100000000000004,"15min":0.70899999999999996},"min":{"1min":0.47899999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91600000000000004,"5min":1.155,"15min":1.155},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68200000000000005,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94699999999999995,"5min":1.137,"15min":1.137},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.66500000000000004,"15min":0.71399999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.026,"5min":1.026,"15min":1.478},"last":0.755},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.69399999999999995,"15min":0.745},"min":{"1min":0.57299999999999995,"5min":0.434,"15min":0.434},"max":{"1min":0.96199999999999997,"5min":1.0640000000000001,"15min":1.393},"last":0.503}]},{"osd":2,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.69199999999999995,"15min":0.73899999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.008,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.71699999999999997,"15min":0.75600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.99299999999999999,"5min":1.04,"15min":1.0720000000000001},"last":0.47499999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.69099999999999995,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.0580000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.77000000000000002,"15min":0.78400000000000003},"min":{"1min":0.54000000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.0620000000000001,"5min":2.6930000000000001,"15min":2.6930000000000001},"last":0.71499999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.84799999999999998,"5min":0.95499999999999996,"15min":1.0760000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.46600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":0.93100000000000005,"15min":0.98199999999999998},"last":0.81799999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69199999999999995,"15min":0.68400000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0920000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.48999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0129999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.53700000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.036,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73599999999999999,"15min":0.73799999999999999},"min":{"1min":0.52300000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.98299999999999998,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.79200000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48652,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323188,"statfs":{"total":96636764160,"available":96586944512,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.65300000000000002,"15min":0.71099999999999997},"min":{"1min":0.55700000000000005,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.876,"5min":1.0409999999999999,"15min":1.2150000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.67600000000000005,"15min":0.73499999999999999},"min":{"1min":0.61799999999999999,"5min":0.40200000000000002,"15min":0.39100000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.4239999999999999},"last":0.73399999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65200000000000002,"15min":0.71099999999999997},"min":{"1min":0.54300000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.81399999999999995,"5min":1.022,"15min":1.98},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.71199999999999997,"15min":0.753},"min":{"1min":0.498,"5min":0.435,"15min":0.435},"max":{"1min":0.97699999999999998,"5min":1.0489999999999999,"15min":1.8200000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.69199999999999995,"15min":0.73299999999999998},"min":{"1min":0.57799999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92600000000000005,"5min":0.92800000000000005,"15min":1.669},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.69499999999999995,"15min":0.747},"min":{"1min":0.61199999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.98599999999999999,"5min":1.167,"15min":1.7589999999999999},"last":0.97299999999999998}]},{"osd":3,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.69999999999999996,"15min":0.749},"min":{"1min":0.67900000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91900000000000004,"5min":0.997,"15min":2.52},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.78500000000000003},"min":{"1min":0.55600000000000005,"5min":0.51200000000000001,"15min":0.47799999999999998},"max":{"1min":1.0249999999999999,"5min":1.1160000000000001,"15min":1.877},"last":0.82999999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.439,"5min":0.376,"15min":0.376},"max":{"1min":0.87,"5min":0.93799999999999994,"15min":2.4100000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.70599999999999996,"15min":0.71099999999999997},"min":{"1min":0.51700000000000002,"5min":0.46000000000000002,"15min":0.45900000000000002},"max":{"1min":0.93200000000000005,"5min":1.2709999999999999,"15min":2.6629999999999998},"last":1.038}]},{"osd":5,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.438,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72899999999999998,"5min":1.0900000000000001,"15min":1.5960000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88500000000000001,"5min":1.1180000000000001,"15min":1.925},"last":0.89500000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.76400000000000001,"15min":0.76900000000000002},"min":{"1min":0.58499999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88300000000000001,"5min":1.056,"15min":2.7050000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71699999999999997,"15min":0.73899999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81399999999999995,"5min":1.1220000000000001,"15min":1.726},"last":0.93600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576041,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48636,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323204,"statfs":{"total":96636764160,"available":96586960896,"internally_reserved":0,"allocated":3653632,"data_stored":1802975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.379,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56899999999999995,"5min":0.71099999999999997,"15min":0.80300000000000005},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.5,"15min":0.52600000000000002},"min":{"1min":0.33500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.79000000000000004,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72899999999999998,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.503,"15min":0.52100000000000002},"min":{"1min":0.39700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71499999999999997,"5min":0.75900000000000001,"15min":0.90300000000000002},"last":0.58199999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.36299999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.995},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53500000000000003,"15min":0.56699999999999995},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76000000000000001,"5min":0.77700000000000002,"15min":0.83999999999999997},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.77300000000000002},"min":{"1min":0.60699999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.008,"5min":1.0589999999999999,"15min":1.238},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.75600000000000001,"15min":0.78700000000000003},"min":{"1min":0.59499999999999997,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":1.0169999999999999,"5min":1.097,"15min":1.304},"last":0.71199999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78400000000000003,"15min":0.80500000000000005},"min":{"1min":0.60999999999999999,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":1.069,"5min":1.1359999999999999,"15min":1.3720000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.77800000000000002,"15min":0.81499999999999995},"min":{"1min":0.55600000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96199999999999997,"5min":1.2030000000000001,"15min":1.3520000000000001},"last":0.72699999999999998}]},{"osd":6,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.77800000000000002,"15min":0.80400000000000005},"min":{"1min":0.57099999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.03,"5min":1.103,"15min":1.1919999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.80600000000000005,"15min":0.83299999999999996},"min":{"1min":0.63900000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.0309999999999999,"5min":1.2170000000000001,"15min":1.266},"last":0.69499999999999995}]},{"osd":7,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81899999999999995,"15min":0.85299999999999998},"min":{"1min":0.63200000000000001,"5min":0.54800000000000004,"15min":0.54500000000000004},"max":{"1min":1.0089999999999999,"5min":1.129,"15min":1.327},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.81599999999999995,"15min":0.85099999999999998},"min":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.57499999999999996},"max":{"1min":1.0549999999999999,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.77200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65616,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306224,"statfs":{"total":96636764160,"available":96569573376,"internally_reserved":0,"allocated":4259840,"data_stored":2398709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.46300000000000002},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.23000000000000001},"max":{"1min":0.65500000000000003,"5min":0.83999999999999997,"15min":1.087},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.499,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70399999999999996,"5min":0.77400000000000002,"15min":1.3089999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.53400000000000003,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.754,"5min":0.84999999999999998,"15min":1.5289999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47599999999999998,"15min":0.51500000000000001},"min":{"1min":0.372,"5min":0.308,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":1.6140000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.52500000000000002,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78300000000000003,"5min":0.81000000000000005,"15min":1.222},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59299999999999997,"15min":0.63500000000000001},"min":{"1min":0.49199999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.876,"5min":0.876,"15min":1.3899999999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.76700000000000002,"15min":0.80300000000000005},"min":{"1min":0.70499999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0800000000000001,"5min":1.248,"15min":1.248},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.79400000000000004,"15min":0.83799999999999997},"min":{"1min":0.626,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.5820000000000001},"last":0.76700000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.72999999999999998,"15min":0.77300000000000002},"min":{"1min":0.63600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.3440000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.77700000000000002,"15min":0.81399999999999995},"min":{"1min":0.749,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.113,"5min":1.1499999999999999,"15min":1.3640000000000001},"last":0.79600000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.78200000000000003,"15min":0.84199999999999997},"min":{"1min":0.60699999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.6240000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.82299999999999995,"15min":0.871},"min":{"1min":0.751,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.337},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.78100000000000003,"15min":0.81599999999999995},"min":{"1min":0.71099999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.214},"last":0.746},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.81000000000000005,"15min":0.86099999999999999},"min":{"1min":0.76300000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.365},"last":0.67500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576042,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52732,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319108,"statfs":{"total":96636764160,"available":96582766592,"internally_reserved":0,"allocated":3653632,"data_stored":1802975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45600000000000002,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.29199999999999998,"15min":0.27800000000000002},"max":{"1min":0.63600000000000001,"5min":0.73199999999999998,"15min":0.75900000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.49099999999999999,"15min":0.52700000000000002},"min":{"1min":0.38500000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.96799999999999997},"last":0.51200000000000001}]},{"osd":1,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.47699999999999998},"min":{"1min":0.35899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69599999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67000000000000004,"5min":0.872,"15min":0.88800000000000001},"last":0.625}]},{"osd":3,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58099999999999996},"min":{"1min":0.42999999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.86799999999999999,"15min":0.98499999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.56899999999999995},"min":{"1min":0.38800000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.038},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.75600000000000001,"15min":0.79500000000000004},"min":{"1min":0.54000000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.026,"5min":1.1100000000000001,"15min":1.1970000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.74399999999999999,"15min":0.79000000000000004},"min":{"1min":0.55300000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.0049999999999999,"5min":1.157,"15min":1.244},"last":0.76300000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.024,"5min":1.171,"15min":1.256},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.77100000000000002,"15min":0.81200000000000006},"min":{"1min":0.57399999999999995,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0649999999999999,"5min":1.1180000000000001,"15min":1.3140000000000001},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.79100000000000004,"15min":0.83299999999999996},"min":{"1min":0.62,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0800000000000001,"5min":1.21,"15min":1.24},"last":1},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.78200000000000003,"15min":0.82599999999999996},"min":{"1min":0.60399999999999998,"5min":0.5,"15min":0.5},"max":{"1min":1.036,"5min":1.0920000000000001,"15min":1.329},"last":1.0189999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.83499999999999996,"15min":0.879},"min":{"1min":0.68200000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.087,"5min":1.151,"15min":1.2889999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.84299999999999997,"15min":0.88600000000000001},"min":{"1min":0.63600000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1040000000000001,"5min":1.234,"15min":1.304},"last":1.1499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576042,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48652,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323188,"statfs":{"total":96636764160,"available":96586944512,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.34699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.65200000000000002,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.38},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51800000000000002,"15min":0.54000000000000004},"min":{"1min":0.36399999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48599999999999999,"15min":0.49099999999999999},"min":{"1min":0.39300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80200000000000005,"5min":1.821,"15min":1.821},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.56299999999999994},"min":{"1min":0.48599999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94599999999999995,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.627,"15min":0.65000000000000002},"min":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81499999999999995,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.77500000000000002}]},{"osd":4,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.753,"15min":0.78100000000000003},"min":{"1min":0.66200000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0609999999999999,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77100000000000002,"15min":0.80000000000000004},"min":{"1min":0.61199999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0820000000000001,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.56999999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77800000000000002,"15min":0.79900000000000004},"min":{"1min":0.629,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.089,"5min":2.157,"15min":2.157},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1699999999999999,"5min":2.266,"15min":2.266},"last":0.61799999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.82199999999999995,"15min":0.84699999999999998},"min":{"1min":0.68200000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.121,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.872,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.66300000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.137,"5min":2.198,"15min":2.198},"last":0.65000000000000002}]},{"osd":7,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.83599999999999997,"15min":0.84999999999999998},"min":{"1min":0.68100000000000005,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":1.111,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.85999999999999999,"15min":0.879},"min":{"1min":0.64400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.153,"5min":2.444,"15min":2.444},"last":0.97399999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96976,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274864,"statfs":{"total":96636764160,"available":96537460736,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.65600000000000003,"15min":0.69899999999999995},"min":{"1min":0.433,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.64900000000000002,"15min":0.69899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88200000000000001,"5min":1.02,"15min":1.02},"last":0.68500000000000005}]},{"osd":1,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.67100000000000004,"15min":0.71899999999999997},"min":{"1min":0.58299999999999996,"5min":0.436,"15min":0.432},"max":{"1min":0.96799999999999997,"5min":1.135,"15min":1.135},"last":0.625},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.68100000000000005,"15min":0.72399999999999998},"min":{"1min":0.54700000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97099999999999997,"5min":1.004,"15min":1.004},"last":0.76600000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.76800000000000002,"15min":0.80700000000000005},"min":{"1min":0.70099999999999996,"5min":0.52100000000000002,"15min":0.49199999999999999},"max":{"1min":1.016,"5min":1.8,"15min":1.8},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.74199999999999999,"15min":0.79100000000000004},"min":{"1min":0.53200000000000003,"5min":0.503,"15min":0.496},"max":{"1min":0.96599999999999997,"5min":1.0369999999999999,"15min":1.224},"last":0.82499999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.69399999999999995,"15min":0.73699999999999999},"min":{"1min":0.55600000000000005,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92300000000000004,"5min":0.96699999999999997,"15min":1.0580000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.71999999999999997,"15min":0.755},"min":{"1min":0.60599999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.88200000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.65200000000000002,"15min":0.63200000000000001},"min":{"1min":0.48099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.872,"5min":1.032,"15min":1.032},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63900000000000001,"15min":0.64500000000000002},"min":{"1min":0.505,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81299999999999994,"5min":1.01,"15min":1.0920000000000001},"last":0.80300000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.69799999999999995,"15min":0.67500000000000004},"min":{"1min":0.48999999999999999,"5min":0.44700000000000001,"15min":0.40899999999999997},"max":{"1min":0.876,"5min":1.722,"15min":1.722},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73199999999999998,"15min":0.72099999999999997},"min":{"1min":0.57599999999999996,"5min":0.439,"15min":0.439},"max":{"1min":0.93300000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.45700000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81299999999999994,"15min":0.81699999999999995},"min":{"1min":0.68400000000000005,"5min":0.57499999999999996,"15min":0.54400000000000004},"max":{"1min":0.98199999999999998,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.78500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576042,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48656,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323184,"statfs":{"total":96636764160,"available":96586940416,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.63400000000000001,"15min":0.68400000000000005},"min":{"1min":0.52200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.66200000000000003,"15min":0.71199999999999997},"min":{"1min":0.54700000000000004,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0680000000000001},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.70199999999999996,"15min":0.746},"min":{"1min":0.58699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0980000000000001,"5min":1.839,"15min":1.839},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.69699999999999995,"15min":0.72399999999999998},"min":{"1min":0.499,"5min":0.438,"15min":0.438},"max":{"1min":0.94599999999999995,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.70899999999999996}]},{"osd":2,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.70799999999999996,"15min":0.74299999999999999},"min":{"1min":0.60599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69799999999999995,"15min":0.73499999999999999},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97599999999999998,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.91400000000000003}]},{"osd":3,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.75,"15min":0.76400000000000001},"min":{"1min":0.70099999999999996,"5min":0.443,"15min":0.443},"max":{"1min":0.94999999999999996,"5min":3.444,"15min":3.444},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.72299999999999998,"15min":0.76400000000000001},"min":{"1min":0.63800000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.996,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.84999999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.439,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84799999999999998,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.95699999999999996,"5min":0.999,"15min":1.0660000000000001},"last":0.76900000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.67500000000000004},"min":{"1min":0.496,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92300000000000004,"5min":1.214,"15min":1.214},"last":0.751},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.69799999999999995,"15min":0.70399999999999996},"min":{"1min":0.58099999999999996,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":1.0900000000000001,"15min":1.107},"last":0.98799999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71899999999999997,"15min":0.73199999999999998},"min":{"1min":0.48899999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.89000000000000001,"5min":1.04,"15min":1.2010000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.55600000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89100000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.89300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:36:12.249 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:12.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.447+0000 7eff81200700 1 Processor -- start 2024-03-29T11:36:12.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.448+0000 7eff81200700 1 -- start start 2024-03-29T11:36:12.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.449+0000 7eff81200700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7eff7c15b5d0 0x7eff7c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.449+0000 7eff81200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c0ac7d0 0x7eff7c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.449+0000 7eff81200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7c15fd50 con 0x7eff7c0cb3f0 2024-03-29T11:36:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.449+0000 7eff81200700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff7c15b9b0 con 0x7eff7c15b5d0 2024-03-29T11:36:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.449+0000 7eff81200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c15f140 con 0x7eff7c0ac7d0 2024-03-29T11:36:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.449+0000 7eff79e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7eff7c15b5d0 0x7eff7c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.449+0000 7eff7a800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c0ac7d0 0x7eff7c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.449+0000 7eff79e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7eff7c15b5d0 0x7eff7c0ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.12:43216/0 (socket says 172.21.15.12:43216) 2024-03-29T11:36:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.449+0000 7eff79e00700 1 -- 172.21.15.12:0/938217321 learned_addr learned my addr 172.21.15.12:0/938217321 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:12.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.449+0000 7eff79400700 1 --1- 172.21.15.12:0/938217321 >> v1:172.21.15.159:6789/0 conn(0x7eff7c0cb3f0 0x7eff7c0ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:41100/0 (socket says 172.21.15.12:41100) 2024-03-29T11:36:12.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.450+0000 7eff78a00700 1 -- 172.21.15.12:0/938217321 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302807889 0 0) 0x7eff7c15fd50 con 0x7eff7c0cb3f0 2024-03-29T11:36:12.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.450+0000 7eff79e00700 1 -- 172.21.15.12:0/938217321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c0ac7d0 msgr2=0x7eff7c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:12.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.450+0000 7eff79e00700 1 --2- 172.21.15.12:0/938217321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c0ac7d0 0x7eff7c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.450+0000 7eff79e00700 1 -- 172.21.15.12:0/938217321 >> v1:172.21.15.159:6789/0 conn(0x7eff7c0cb3f0 legacy=0x7eff7c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.450+0000 7eff79e00700 1 -- 172.21.15.12:0/938217321 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c15f420 con 0x7eff7c15b5d0 2024-03-29T11:36:12.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.450+0000 7eff7a800700 1 --2- 172.21.15.12:0/938217321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c0ac7d0 0x7eff7c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-29T11:36:12.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.450+0000 7eff79e00700 1 --2- 172.21.15.12:0/938217321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff7c15b5d0 0x7eff7c0ae3f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0098f0 tx=0x7eff6c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=428263b4f2b77a7e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:12.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.450+0000 7eff78a00700 1 -- 172.21.15.12:0/938217321 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c011830 con 0x7eff7c15b5d0 2024-03-29T11:36:12.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.450+0000 7eff78a00700 1 -- 172.21.15.12:0/938217321 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c011990 con 0x7eff7c15b5d0 2024-03-29T11:36:12.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.450+0000 7eff78a00700 1 -- 172.21.15.12:0/938217321 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c011d20 con 0x7eff7c15b5d0 2024-03-29T11:36:12.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.451+0000 7eff81200700 1 -- 172.21.15.12:0/938217321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff7c15b5d0 msgr2=0x7eff7c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.451+0000 7eff81200700 1 --2- 172.21.15.12:0/938217321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff7c15b5d0 0x7eff7c0ae3f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0098f0 tx=0x7eff6c009c00 comp rx=0 tx=0).stop 2024-03-29T11:36:12.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.451+0000 7eff81200700 1 -- 172.21.15.12:0/938217321 shutdown_connections 2024-03-29T11:36:12.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.451+0000 7eff81200700 1 --2- 172.21.15.12:0/938217321 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c0ac7d0 0x7eff7c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.451+0000 7eff81200700 1 --2- 172.21.15.12:0/938217321 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff7c15b5d0 0x7eff7c0ae3f0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.451+0000 7eff81200700 1 -- 172.21.15.12:0/938217321 >> 172.21.15.12:0/938217321 conn(0x7eff7c0cf190 msgr2=0x7eff7c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:12.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.451+0000 7eff81200700 1 -- 172.21.15.12:0/938217321 shutdown_connections 2024-03-29T11:36:12.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.451+0000 7eff81200700 1 -- 172.21.15.12:0/938217321 wait complete. 2024-03-29T11:36:12.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.453+0000 7eff81200700 1 Processor -- start 2024-03-29T11:36:12.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.453+0000 7eff81200700 1 -- start start 2024-03-29T11:36:12.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff81200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff7c0ac7d0 0x7eff7c0ce680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff81200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff7c15b5d0 0x7eff7c1575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff81200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c1479d0 0x7eff7c071b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff81200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c157ae0 con 0x7eff7c15b5d0 2024-03-29T11:36:12.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff81200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0b3a60 con 0x7eff7c0ac7d0 2024-03-29T11:36:12.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff81200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0b3d40 con 0x7eff7c1479d0 2024-03-29T11:36:12.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff79400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff7c15b5d0 0x7eff7c1575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff79400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff7c15b5d0 0x7eff7c1575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33556/0 (socket says 172.21.15.12:33556) 2024-03-29T11:36:12.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff79400700 1 -- 172.21.15.12:0/1594882549 learned_addr learned my addr 172.21.15.12:0/1594882549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:12.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff79e00700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff7c0ac7d0 0x7eff7c0ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff7a800700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c1479d0 0x7eff7c071b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff79400700 1 -- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c1479d0 msgr2=0x7eff7c071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff79400700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c1479d0 0x7eff7c071b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff79400700 1 -- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff7c0ac7d0 msgr2=0x7eff7c0ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff79400700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff7c0ac7d0 0x7eff7c0ce680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff79400700 1 -- 172.21.15.12:0/1594882549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff74009a70 con 0x7eff7c15b5d0 2024-03-29T11:36:12.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.454+0000 7eff79e00700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff7c0ac7d0 0x7eff7c0ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:12.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.455+0000 7eff79400700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff7c15b5d0 0x7eff7c1575a0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7eff680011e0 tx=0x7eff6800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:12.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.455+0000 7eff72000700 1 -- 172.21.15.12:0/1594882549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68004c80 con 0x7eff7c15b5d0 2024-03-29T11:36:12.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.455+0000 7eff72000700 1 -- 172.21.15.12:0/1594882549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff68004e00 con 0x7eff7c15b5d0 2024-03-29T11:36:12.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.455+0000 7eff81200700 1 -- 172.21.15.12:0/1594882549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6c009550 con 0x7eff7c15b5d0 2024-03-29T11:36:12.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.455+0000 7eff72000700 1 -- 172.21.15.12:0/1594882549 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68007310 con 0x7eff7c15b5d0 2024-03-29T11:36:12.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.455+0000 7eff81200700 1 -- 172.21.15.12:0/1594882549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7c159560 con 0x7eff7c15b5d0 2024-03-29T11:36:12.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.456+0000 7eff81200700 1 -- 172.21.15.12:0/1594882549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7c159780 con 0x7eff7c15b5d0 2024-03-29T11:36:12.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.457+0000 7eff72000700 1 -- 172.21.15.12:0/1594882549 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7eff68007470 con 0x7eff7c15b5d0 2024-03-29T11:36:12.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.458+0000 7eff72000700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff5c06adb0 0x7eff5c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:12.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.458+0000 7eff72000700 1 -- 172.21.15.12:0/1594882549 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7eff680892c0 con 0x7eff7c15b5d0 2024-03-29T11:36:12.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.459+0000 7eff79e00700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff5c06adb0 0x7eff5c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:12.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.459+0000 7eff79e00700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff5c06adb0 0x7eff5c06d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7eff6c014040 tx=0x7eff6c008030 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:12.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.462+0000 7eff72000700 1 -- 172.21.15.12:0/1594882549 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff68054360 con 0x7eff7c15b5d0 2024-03-29T11:36:12.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.676+0000 7eff81200700 1 -- 172.21.15.12:0/1594882549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff7c067720 con 0x7eff7c15b5d0 2024-03-29T11:36:12.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.677+0000 7eff72000700 1 -- 172.21.15.12:0/1594882549 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v290) v1 ==== 74+0+13708 (secure 0 0 0) 0x7eff68057980 con 0x7eff7c15b5d0 2024-03-29T11:36:12.680 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:12.680 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":290,"fsid":"216a15e0-b7ec-465c-8851-ed202d429a6c","created":"2024-03-29T09:54:02.796303+0000","modified":"2024-03-29T11:36:07.382771+0000","last_up_change":"2024-03-29T09:54:15.811097+0000","last_in_change":"2024-03-29T09:54:06.330559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T09:54:16.494788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cf9b5216-0622-4882-90da-b5e81467c283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":43908},{"type":"v1","addr":"172.21.15.12:6809","nonce":43908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":43908},{"type":"v1","addr":"172.21.15.12:6811","nonce":43908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":43908},{"type":"v1","addr":"172.21.15.12:6815","nonce":43908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":43908},{"type":"v1","addr":"172.21.15.12:6813","nonce":43908}]},"public_addr":"172.21.15.12:6809/43908","cluster_addr":"172.21.15.12:6811/43908","heartbeat_back_addr":"172.21.15.12:6815/43908","heartbeat_front_addr":"172.21.15.12:6813/43908","state":["exists","up"]},{"osd":1,"uuid":"1607d220-054e-43c6-8e00-45696c8702c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":43909},{"type":"v1","addr":"172.21.15.12:6817","nonce":43909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":43909},{"type":"v1","addr":"172.21.15.12:6819","nonce":43909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6825","nonce":43909},{"type":"v1","addr":"172.21.15.12:6827","nonce":43909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6821","nonce":43909},{"type":"v1","addr":"172.21.15.12:6823","nonce":43909}]},"public_addr":"172.21.15.12:6817/43909","cluster_addr":"172.21.15.12:6819/43909","heartbeat_back_addr":"172.21.15.12:6827/43909","heartbeat_front_addr":"172.21.15.12:6823/43909","state":["exists","up"]},{"osd":2,"uuid":"623142ba-90d6-4b24-8677-0058a5a01056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":43907},{"type":"v1","addr":"172.21.15.12:6801","nonce":43907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":43907},{"type":"v1","addr":"172.21.15.12:6803","nonce":43907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":43907},{"type":"v1","addr":"172.21.15.12:6807","nonce":43907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":43907},{"type":"v1","addr":"172.21.15.12:6805","nonce":43907}]},"public_addr":"172.21.15.12:6801/43907","cluster_addr":"172.21.15.12:6803/43907","heartbeat_back_addr":"172.21.15.12:6807/43907","heartbeat_front_addr":"172.21.15.12:6805/43907","state":["exists","up"]},{"osd":3,"uuid":"82460961-6482-46b5-8419-0bb9824c7a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":43910},{"type":"v1","addr":"172.21.15.12:6822","nonce":43910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":43910},{"type":"v1","addr":"172.21.15.12:6826","nonce":43910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":43910},{"type":"v1","addr":"172.21.15.12:6831","nonce":43910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":43910},{"type":"v1","addr":"172.21.15.12:6829","nonce":43910}]},"public_addr":"172.21.15.12:6822/43910","cluster_addr":"172.21.15.12:6826/43910","heartbeat_back_addr":"172.21.15.12:6831/43910","heartbeat_front_addr":"172.21.15.12:6829/43910","state":["exists","up"]},{"osd":4,"uuid":"0bed68a5-f7af-452a-98b2-f26f1b394375","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":44146},{"type":"v1","addr":"172.21.15.159:6801","nonce":44146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":44146},{"type":"v1","addr":"172.21.15.159:6803","nonce":44146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":44146},{"type":"v1","addr":"172.21.15.159:6807","nonce":44146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":44146},{"type":"v1","addr":"172.21.15.159:6805","nonce":44146}]},"public_addr":"172.21.15.159:6801/44146","cluster_addr":"172.21.15.159:6803/44146","heartbeat_back_addr":"172.21.15.159:6807/44146","heartbeat_front_addr":"172.21.15.159:6805/44146","state":["exists","up"]},{"osd":5,"uuid":"b7c21a7a-17e5-4a72-af50-2200c49f3c88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":44148},{"type":"v1","addr":"172.21.15.159:6818","nonce":44148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":44148},{"type":"v1","addr":"172.21.15.159:6822","nonce":44148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":44148},{"type":"v1","addr":"172.21.15.159:6830","nonce":44148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":44148},{"type":"v1","addr":"172.21.15.159:6826","nonce":44148}]},"public_addr":"172.21.15.159:6818/44148","cluster_addr":"172.21.15.159:6822/44148","heartbeat_back_addr":"172.21.15.159:6830/44148","heartbeat_front_addr":"172.21.15.159:6826/44148","state":["exists","up"]},{"osd":6,"uuid":"7f90aabb-c34c-4700-8bfa-a1f4d4549820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6817","nonce":44159},{"type":"v1","addr":"172.21.15.159:6819","nonce":44159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6821","nonce":44159},{"type":"v1","addr":"172.21.15.159:6823","nonce":44159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6829","nonce":44159},{"type":"v1","addr":"172.21.15.159:6831","nonce":44159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6825","nonce":44159},{"type":"v1","addr":"172.21.15.159:6827","nonce":44159}]},"public_addr":"172.21.15.159:6819/44159","cluster_addr":"172.21.15.159:6823/44159","heartbeat_back_addr":"172.21.15.159:6831/44159","heartbeat_front_addr":"172.21.15.159:6827/44159","state":["exists","up"]},{"osd":7,"uuid":"fa2eb995-510c-426b-9027-c75b1aaa9360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":44147},{"type":"v1","addr":"172.21.15.159:6809","nonce":44147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":44147},{"type":"v1","addr":"172.21.15.159:6811","nonce":44147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":44147},{"type":"v1","addr":"172.21.15.159:6815","nonce":44147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":44147},{"type":"v1","addr":"172.21.15.159:6813","nonce":44147}]},"public_addr":"172.21.15.159:6809/44147","cluster_addr":"172.21.15.159:6811/44147","heartbeat_back_addr":"172.21.15.159:6815/44147","heartbeat_front_addr":"172.21.15.159:6813/44147","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.878826+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.642667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.563844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.670932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.535599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.864102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.601485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-29T09:54:11.813465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6834/2115072104":"2024-03-30T11:36:03.753825+0000","172.21.15.159:0/4093476986":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/3928992099":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/3508779586":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/1359166523":"2024-03-30T11:36:02.154650+0000","172.21.15.12:6832/2115072104":"2024-03-30T11:36:03.753825+0000","172.21.15.159:0/922684053":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/3536271161":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/3208784174":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/2115167177":"2024-03-30T11:34:37.193903+0000","172.21.15.159:6833/62646":"2024-03-30T11:34:37.193903+0000","172.21.15.159:6832/62646":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/411770029":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/3938940279":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/2298284405":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/3177306995":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/2440579896":"2024-03-30T11:36:02.154650+0000","172.21.15.159:0/1795874409":"2024-03-30T11:34:37.193903+0000","172.21.15.159:0/547605341":"2024-03-30T11:34:37.193903+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:12.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 -- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff5c06adb0 msgr2=0x7eff5c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff5c06adb0 0x7eff5c06d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7eff6c014040 tx=0x7eff6c008030 comp rx=0 tx=0).stop 2024-03-29T11:36:12.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 -- 172.21.15.12:0/1594882549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff7c15b5d0 msgr2=0x7eff7c1575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:12.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff7c15b5d0 0x7eff7c1575a0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7eff680011e0 tx=0x7eff6800cae0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 -- 172.21.15.12:0/1594882549 shutdown_connections 2024-03-29T11:36:12.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7eff5c06adb0 0x7eff5c06d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff7c1479d0 0x7eff7c071b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff7c0ac7d0 0x7eff7c0ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 --2- 172.21.15.12:0/1594882549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff7c15b5d0 0x7eff7c1575a0 unknown :-1 s=CLOSED pgs=2713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:12.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 -- 172.21.15.12:0/1594882549 >> 172.21.15.12:0/1594882549 conn(0x7eff7c0cf190 msgr2=0x7eff7c15cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:12.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 -- 172.21.15.12:0/1594882549 shutdown_connections 2024-03-29T11:36:12.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:12.680+0000 7eff81200700 1 -- 172.21.15.12:0/1594882549 wait complete. 2024-03-29T11:36:13.699 INFO:tasks.ceph:Scrubbing osd.0 2024-03-29T11:36:13.700 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-29T11:36:13.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.915+0000 7f4f66c00700 1 Processor -- start 2024-03-29T11:36:13.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.915+0000 7f4f66c00700 1 -- start start 2024-03-29T11:36:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f66c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f6015db10 0x7f4f60099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f66c00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4f600b3930 0x7f4f60099fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f66c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4f6009a4e0 0x7f4f600cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f66c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f600cbdf0 con 0x7f4f6009a4e0 2024-03-29T11:36:13.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f66c00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6006bec0 con 0x7f4f600b3930 2024-03-29T11:36:13.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f66c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f600b27a0 con 0x7f4f6015db10 2024-03-29T11:36:13.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f64c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4f6009a4e0 0x7f4f600cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:13.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f64c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4f6009a4e0 0x7f4f600cf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33558/0 (socket says 172.21.15.12:33558) 2024-03-29T11:36:13.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f64c00700 1 -- 172.21.15.12:0/1093478425 learned_addr learned my addr 172.21.15.12:0/1093478425 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:13.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f5fe00700 1 --2- 172.21.15.12:0/1093478425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f6015db10 0x7f4f60099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:13.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.916+0000 7f4f5f400700 1 --2- 172.21.15.12:0/1093478425 >> v2:172.21.15.159:3300/0 conn(0x7f4f600b3930 0x7f4f60099fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:13.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.917+0000 7f4f64c00700 1 -- 172.21.15.12:0/1093478425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f6015db10 msgr2=0x7f4f60099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.917+0000 7f4f64c00700 1 --2- 172.21.15.12:0/1093478425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f6015db10 0x7f4f60099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.917+0000 7f4f64c00700 1 -- 172.21.15.12:0/1093478425 >> v2:172.21.15.159:3300/0 conn(0x7f4f600b3930 msgr2=0x7f4f60099fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.917+0000 7f4f64c00700 1 --2- 172.21.15.12:0/1093478425 >> v2:172.21.15.159:3300/0 conn(0x7f4f600b3930 0x7f4f60099fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.917+0000 7f4f64c00700 1 -- 172.21.15.12:0/1093478425 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6006bbe0 con 0x7f4f6009a4e0 2024-03-29T11:36:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.917+0000 7f4f64c00700 1 --2- 172.21.15.12:0/1093478425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f6009a4e0 0x7f4f600cf4b0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f4f600712f0 tx=0x7f4f5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e131a88d81a16b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.917+0000 7f4f5ea00700 1 -- 172.21.15.12:0/1093478425 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5800e070 con 0x7f4f6009a4e0 2024-03-29T11:36:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.917+0000 7f4f5ea00700 1 -- 172.21.15.12:0/1093478425 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5800b9c0 con 0x7f4f6009a4e0 2024-03-29T11:36:13.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.917+0000 7f4f5ea00700 1 -- 172.21.15.12:0/1093478425 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5800bd70 con 0x7f4f6009a4e0 2024-03-29T11:36:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.918+0000 7f4f66c00700 1 -- 172.21.15.12:0/1093478425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f6009a4e0 msgr2=0x7f4f600cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.918+0000 7f4f66c00700 1 --2- 172.21.15.12:0/1093478425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f6009a4e0 0x7f4f600cf4b0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f4f600712f0 tx=0x7f4f5800b330 comp rx=0 tx=0).stop 2024-03-29T11:36:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.918+0000 7f4f66c00700 1 -- 172.21.15.12:0/1093478425 shutdown_connections 2024-03-29T11:36:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.918+0000 7f4f66c00700 1 --2- 172.21.15.12:0/1093478425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f6009a4e0 0x7f4f600cf4b0 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.918+0000 7f4f66c00700 1 --2- 172.21.15.12:0/1093478425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f6015db10 0x7f4f60099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.918+0000 7f4f66c00700 1 --2- 172.21.15.12:0/1093478425 >> v2:172.21.15.159:3300/0 conn(0x7f4f600b3930 0x7f4f60099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.918+0000 7f4f66c00700 1 -- 172.21.15.12:0/1093478425 >> 172.21.15.12:0/1093478425 conn(0x7f4f600b2e10 msgr2=0x7f4f600b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.918+0000 7f4f66c00700 1 -- 172.21.15.12:0/1093478425 shutdown_connections 2024-03-29T11:36:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.918+0000 7f4f66c00700 1 -- 172.21.15.12:0/1093478425 wait complete. 2024-03-29T11:36:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.920+0000 7f4f66c00700 1 Processor -- start 2024-03-29T11:36:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.920+0000 7f4f66c00700 1 -- start start 2024-03-29T11:36:13.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f66c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f6009a4e0 0x7f4f600b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:13.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f66c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f600b3930 0x7f4f600c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:13.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f66c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f6015db10 0x7f4f600cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:13.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f66c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f600c6810 con 0x7f4f6009a4e0 2024-03-29T11:36:13.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f66c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f600c6ca0 con 0x7f4f600b3930 2024-03-29T11:36:13.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f66c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f600c5c00 con 0x7f4f6015db10 2024-03-29T11:36:13.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f5fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f6009a4e0 0x7f4f600b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:13.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f5fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f6009a4e0 0x7f4f600b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33562/0 (socket says 172.21.15.12:33562) 2024-03-29T11:36:13.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f5fe00700 1 -- 172.21.15.12:0/170487126 learned_addr learned my addr 172.21.15.12:0/170487126 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:13.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f64c00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f6015db10 0x7f4f600cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:13.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f5f400700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f600b3930 0x7f4f600c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:13.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f5fe00700 1 -- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f6015db10 msgr2=0x7f4f600cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:13.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f5fe00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f6015db10 0x7f4f600cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:13.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f5fe00700 1 -- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f600b3930 msgr2=0x7f4f600c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:13.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f5fe00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f600b3930 0x7f4f600c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:13.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f5fe00700 1 -- 172.21.15.12:0/170487126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f58009b30 con 0x7f4f6009a4e0 2024-03-29T11:36:13.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.921+0000 7f4f64c00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f6015db10 0x7f4f600cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:13.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.922+0000 7f4f5fe00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f6009a4e0 0x7f4f600b6a10 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000d800 tx=0x7f4f5000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:13.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.922+0000 7f4f57e00700 1 -- 172.21.15.12:0/170487126 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500125f0 con 0x7f4f6009a4e0 2024-03-29T11:36:13.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.922+0000 7f4f57e00700 1 -- 172.21.15.12:0/170487126 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f50012750 con 0x7f4f6009a4e0 2024-03-29T11:36:13.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.922+0000 7f4f57e00700 1 -- 172.21.15.12:0/170487126 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50012b00 con 0x7f4f6009a4e0 2024-03-29T11:36:13.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.922+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f600c5ee0 con 0x7f4f6009a4e0 2024-03-29T11:36:13.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.922+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f600c64a0 con 0x7f4f6009a4e0 2024-03-29T11:36:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.923+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f600b2470 con 0x7f4f6009a4e0 2024-03-29T11:36:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.924+0000 7f4f57e00700 1 -- 172.21.15.12:0/170487126 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4f50012c60 con 0x7f4f6009a4e0 2024-03-29T11:36:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.925+0000 7f4f57e00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f4006adb0 0x7f4f4006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.925+0000 7f4f57e00700 1 -- 172.21.15.12:0/170487126 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f4f500897a0 con 0x7f4f6009a4e0 2024-03-29T11:36:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.925+0000 7f4f57e00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] conn(0x7f4f400712c0 0x7f4f400736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.925+0000 7f4f57e00700 1 -- 172.21.15.12:0/170487126 --> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f40073db0 con 0x7f4f400712c0 2024-03-29T11:36:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.925+0000 7f4f57e00700 1 -- 172.21.15.12:0/170487126 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f5008be60 con 0x7f4f6009a4e0 2024-03-29T11:36:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.925+0000 7f4f5f400700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f4006adb0 0x7f4f4006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.925+0000 7f4f64c00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] conn(0x7f4f400712c0 0x7f4f400736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.926+0000 7f4f64c00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] conn(0x7f4f400712c0 0x7f4f400736e0 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.926+0000 7f4f5f400700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f4006adb0 0x7f4f4006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f601f6fd0 tx=0x7f4f4c008040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.927+0000 7f4f57e00700 1 -- 172.21.15.12:0/170487126 <== osd.0 v2:172.21.15.12:6808/43908 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4f40073db0 con 0x7f4f400712c0 2024-03-29T11:36:13.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.942+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 --> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4f600c6950 con 0x7f4f400712c0 2024-03-29T11:36:13.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.942+0000 7f4f57e00700 1 -- 172.21.15.12:0/170487126 <== osd.0 v2:172.21.15.12:6808/43908 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4f5800bb30 con 0x7f4f400712c0 2024-03-29T11:36:13.941 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:36:13.941 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T11:36:13.941 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:36:13.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.942+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 >> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] conn(0x7f4f400712c0 msgr2=0x7f4f400736e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.942+0000 7f4f66c00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.12:6808/43908,v1:172.21.15.12:6809/43908] conn(0x7f4f400712c0 0x7f4f400736e0 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.942+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f4006adb0 msgr2=0x7f4f4006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.942+0000 7f4f66c00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4f4006adb0 0x7f4f4006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f601f6fd0 tx=0x7f4f4c008040 comp rx=0 tx=0).stop 2024-03-29T11:36:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.942+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f6009a4e0 msgr2=0x7f4f600b6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.942+0000 7f4f66c00700 1 --2- 172.21.15.12:0/170487126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f6009a4e0 0x7f4f600b6a10 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000d800 tx=0x7f4f5000dbc0 comp rx=0 tx=0).stop 2024-03-29T11:36:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.942+0000 7f4f64c00700 1 -- 172.21.15.12:0/170487126 reap_dead start 2024-03-29T11:36:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.943+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 shutdown_connections 2024-03-29T11:36:13.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.943+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 >> 172.21.15.12:0/170487126 conn(0x7f4f600b2e10 msgr2=0x7f4f6015cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:13.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.943+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 shutdown_connections 2024-03-29T11:36:13.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:13.943+0000 7f4f66c00700 1 -- 172.21.15.12:0/170487126 wait complete. 2024-03-29T11:36:13.959 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-29T11:36:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.158+0000 7f3e96600700 1 Processor -- start 2024-03-29T11:36:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.159+0000 7f3e96600700 1 -- start start 2024-03-29T11:36:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e96600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3e9015dad0 0x7f3e900ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e96600700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3e900d1630 0x7f3e900d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e96600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9015f820 con 0x7f3e900d2000 2024-03-29T11:36:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e96600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e900cfcc0 con 0x7f3e9015dad0 2024-03-29T11:36:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e96600700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e900cd9f0 con 0x7f3e900d1630 2024-03-29T11:36:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e8f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3e9015dad0 0x7f3e900ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e8f400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3e9015dad0 0x7f3e900ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33568/0 (socket says 172.21.15.12:33568) 2024-03-29T11:36:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e8f400700 1 -- 172.21.15.12:0/857157742 learned_addr learned my addr 172.21.15.12:0/857157742 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e8ea00700 1 --2- 172.21.15.12:0/857157742 >> v2:172.21.15.159:3300/0 conn(0x7f3e900d1630 0x7f3e900d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e8f400700 1 -- 172.21.15.12:0/857157742 >> v2:172.21.15.159:3300/0 conn(0x7f3e900d1630 msgr2=0x7f3e900d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.160+0000 7f3e8f400700 1 --2- 172.21.15.12:0/857157742 >> v2:172.21.15.159:3300/0 conn(0x7f3e900d1630 0x7f3e900d1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.161+0000 7f3e8f400700 1 -- 172.21.15.12:0/857157742 >> v1:172.21.15.159:6789/0 conn(0x7f3e900d2000 legacy=0x7f3e9015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.161+0000 7f3e8f400700 1 -- 172.21.15.12:0/857157742 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e900d2430 con 0x7f3e9015dad0 2024-03-29T11:36:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.161+0000 7f3e8ea00700 1 --2- 172.21.15.12:0/857157742 >> v2:172.21.15.159:3300/0 conn(0x7f3e900d1630 0x7f3e900d1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.161+0000 7f3e8f400700 1 --2- 172.21.15.12:0/857157742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 0x7f3e900ced80 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f3e9008b730 tx=0x7f3e80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d779af598dfc186 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.161+0000 7f3e8e000700 1 -- 172.21.15.12:0/857157742 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8000b870 con 0x7f3e9015dad0 2024-03-29T11:36:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.161+0000 7f3e8e000700 1 -- 172.21.15.12:0/857157742 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8000b9d0 con 0x7f3e9015dad0 2024-03-29T11:36:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.161+0000 7f3e8e000700 1 -- 172.21.15.12:0/857157742 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8000bd60 con 0x7f3e9015dad0 2024-03-29T11:36:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.162+0000 7f3e96600700 1 -- 172.21.15.12:0/857157742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 msgr2=0x7f3e900ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.162+0000 7f3e96600700 1 --2- 172.21.15.12:0/857157742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 0x7f3e900ced80 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f3e9008b730 tx=0x7f3e80009bd0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.162+0000 7f3e96600700 1 -- 172.21.15.12:0/857157742 shutdown_connections 2024-03-29T11:36:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.162+0000 7f3e96600700 1 --2- 172.21.15.12:0/857157742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 0x7f3e900ced80 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.162+0000 7f3e96600700 1 --2- 172.21.15.12:0/857157742 >> v2:172.21.15.159:3300/0 conn(0x7f3e900d1630 0x7f3e900d1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.162+0000 7f3e96600700 1 -- 172.21.15.12:0/857157742 >> 172.21.15.12:0/857157742 conn(0x7f3e9008aff0 msgr2=0x7f3e900d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.162+0000 7f3e96600700 1 -- 172.21.15.12:0/857157742 shutdown_connections 2024-03-29T11:36:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.162+0000 7f3e96600700 1 -- 172.21.15.12:0/857157742 wait complete. 2024-03-29T11:36:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.164+0000 7f3e96600700 1 Processor -- start 2024-03-29T11:36:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.164+0000 7f3e96600700 1 -- start start 2024-03-29T11:36:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e96600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e900d1630 0x7f3e900cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e96600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e900d2000 0x7f3e901f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e96600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 0x7f3e901f4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e96600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e900c73a0 con 0x7f3e9015dad0 2024-03-29T11:36:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e96600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e900cb750 con 0x7f3e900d1630 2024-03-29T11:36:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e96600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e900cba30 con 0x7f3e900d2000 2024-03-29T11:36:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 0x7f3e901f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 0x7f3e901f4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33574/0 (socket says 172.21.15.12:33574) 2024-03-29T11:36:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8fe00700 1 -- 172.21.15.12:0/2657301967 learned_addr learned my addr 172.21.15.12:0/2657301967 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8ea00700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e900d2000 0x7f3e901f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8f400700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e900d1630 0x7f3e900cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8fe00700 1 -- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e900d2000 msgr2=0x7f3e901f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8fe00700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e900d2000 0x7f3e901f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8fe00700 1 -- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e900d1630 msgr2=0x7f3e900cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8fe00700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e900d1630 0x7f3e900cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8fe00700 1 -- 172.21.15.12:0/2657301967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e80009550 con 0x7f3e9015dad0 2024-03-29T11:36:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.165+0000 7f3e8ea00700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e900d2000 0x7f3e901f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.166+0000 7f3e8fe00700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 0x7f3e901f4390 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f3e8800cc50 tx=0x7f3e8800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.166+0000 7f3e87400700 1 -- 172.21.15.12:0/2657301967 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e88003c00 con 0x7f3e9015dad0 2024-03-29T11:36:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.166+0000 7f3e87400700 1 -- 172.21.15.12:0/2657301967 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e88003d60 con 0x7f3e9015dad0 2024-03-29T11:36:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.166+0000 7f3e87400700 1 -- 172.21.15.12:0/2657301967 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8800ab60 con 0x7f3e9015dad0 2024-03-29T11:36:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.166+0000 7f3e96600700 1 -- 172.21.15.12:0/2657301967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e900cbd10 con 0x7f3e9015dad0 2024-03-29T11:36:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.166+0000 7f3e96600700 1 -- 172.21.15.12:0/2657301967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e900cc2d0 con 0x7f3e9015dad0 2024-03-29T11:36:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.167+0000 7f3e96600700 1 -- 172.21.15.12:0/2657301967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e90067720 con 0x7f3e9015dad0 2024-03-29T11:36:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.168+0000 7f3e87400700 1 -- 172.21.15.12:0/2657301967 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3e8800acc0 con 0x7f3e9015dad0 2024-03-29T11:36:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.169+0000 7f3e87400700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e7006adb0 0x7f3e7006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.169+0000 7f3e87400700 1 -- 172.21.15.12:0/2657301967 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f3e880895c0 con 0x7f3e9015dad0 2024-03-29T11:36:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.169+0000 7f3e8f400700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e7006adb0 0x7f3e7006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.170+0000 7f3e8f400700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e7006adb0 0x7f3e7006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3e80014040 tx=0x7f3e80008030 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.173+0000 7f3e87400700 1 -- 172.21.15.12:0/2657301967 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e880545c0 con 0x7f3e9015dad0 2024-03-29T11:36:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.384+0000 7f3e96600700 1 -- 172.21.15.12:0/2657301967 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e900cf9e0 con 0x7f3e7006adb0 2024-03-29T11:36:14.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.385+0000 7f3e87400700 1 -- 172.21.15.12:0/2657301967 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3e8000bb40 con 0x7f3e7006adb0 2024-03-29T11:36:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 -- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e7006adb0 msgr2=0x7f3e7006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e7006adb0 0x7f3e7006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3e80014040 tx=0x7f3e80008030 comp rx=0 tx=0).stop 2024-03-29T11:36:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 -- 172.21.15.12:0/2657301967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 msgr2=0x7f3e901f4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 0x7f3e901f4390 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f3e8800cc50 tx=0x7f3e8800cf60 comp rx=0 tx=0).stop 2024-03-29T11:36:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 -- 172.21.15.12:0/2657301967 shutdown_connections 2024-03-29T11:36:14.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f3e7006adb0 0x7f3e7006d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e9015dad0 0x7f3e901f4390 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e900d1630 0x7f3e900cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 --2- 172.21.15.12:0/2657301967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e900d2000 0x7f3e901f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 -- 172.21.15.12:0/2657301967 >> 172.21.15.12:0/2657301967 conn(0x7f3e9008aff0 msgr2=0x7f3e9006cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:14.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 -- 172.21.15.12:0/2657301967 shutdown_connections 2024-03-29T11:36:14.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.388+0000 7f3e96600700 1 -- 172.21.15.12:0/2657301967 wait complete. 2024-03-29T11:36:14.389 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 0 to deep-scrub 2024-03-29T11:36:14.405 INFO:tasks.ceph:Scrubbing osd.1 2024-03-29T11:36:14.406 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-29T11:36:14.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.610+0000 7f5bcce00700 1 Processor -- start 2024-03-29T11:36:14.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.610+0000 7f5bcce00700 1 -- start start 2024-03-29T11:36:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.611+0000 7f5bcce00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5bc815d930 0x7f5bc80d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.611+0000 7f5bcce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bc80d22a0 0x7f5bc80ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.611+0000 7f5bcce00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc815dd10 con 0x7f5bc8162030 2024-03-29T11:36:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.611+0000 7f5bcce00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc815ec40 con 0x7f5bc815d930 2024-03-29T11:36:14.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.611+0000 7f5bcce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc815dfc0 con 0x7f5bc80d22a0 2024-03-29T11:36:14.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.612+0000 7f5bc5e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5bc815d930 0x7f5bc80d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.612+0000 7f5bc6800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bc80d22a0 0x7f5bc80ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.612+0000 7f5bc5400700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f5bc8162030 0x7f5bc80d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.12:41116/0 (socket says 172.21.15.12:41116) 2024-03-29T11:36:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.612+0000 7f5bc5400700 1 -- 172.21.15.12:0/1231070028 learned_addr learned my addr 172.21.15.12:0/1231070028 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.612+0000 7f5bc4a00700 1 -- 172.21.15.12:0/1231070028 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669676886 0 0) 0x7f5bc815dd10 con 0x7f5bc8162030 2024-03-29T11:36:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.612+0000 7f5bc6800700 1 -- 172.21.15.12:0/1231070028 >> v2:172.21.15.159:3300/0 conn(0x7f5bc815d930 msgr2=0x7f5bc80d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.612+0000 7f5bc6800700 1 --2- 172.21.15.12:0/1231070028 >> v2:172.21.15.159:3300/0 conn(0x7f5bc815d930 0x7f5bc80d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.612+0000 7f5bc6800700 1 -- 172.21.15.12:0/1231070028 >> v1:172.21.15.159:6789/0 conn(0x7f5bc8162030 legacy=0x7f5bc80d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.612+0000 7f5bc6800700 1 -- 172.21.15.12:0/1231070028 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc815e2a0 con 0x7f5bc80d22a0 2024-03-29T11:36:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.612+0000 7f5bc6800700 1 --2- 172.21.15.12:0/1231070028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bc80d22a0 0x7f5bc80ce300 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0009d90 tx=0x7f5bc000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dcf360b068e9697 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bc4a00700 1 -- 172.21.15.12:0/1231070028 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc000e070 con 0x7f5bc80d22a0 2024-03-29T11:36:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bc4a00700 1 -- 172.21.15.12:0/1231070028 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc000bb20 con 0x7f5bc80d22a0 2024-03-29T11:36:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bc4a00700 1 -- 172.21.15.12:0/1231070028 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0012430 con 0x7f5bc80d22a0 2024-03-29T11:36:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bcce00700 1 -- 172.21.15.12:0/1231070028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bc80d22a0 msgr2=0x7f5bc80ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bcce00700 1 --2- 172.21.15.12:0/1231070028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bc80d22a0 0x7f5bc80ce300 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0009d90 tx=0x7f5bc000b330 comp rx=0 tx=0).stop 2024-03-29T11:36:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bcce00700 1 -- 172.21.15.12:0/1231070028 shutdown_connections 2024-03-29T11:36:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bcce00700 1 --2- 172.21.15.12:0/1231070028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bc80d22a0 0x7f5bc80ce300 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bcce00700 1 --2- 172.21.15.12:0/1231070028 >> v2:172.21.15.159:3300/0 conn(0x7f5bc815d930 0x7f5bc80d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bcce00700 1 -- 172.21.15.12:0/1231070028 >> 172.21.15.12:0/1231070028 conn(0x7f5bc808aff0 msgr2=0x7f5bc80d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bcce00700 1 -- 172.21.15.12:0/1231070028 shutdown_connections 2024-03-29T11:36:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.613+0000 7f5bcce00700 1 -- 172.21.15.12:0/1231070028 wait complete. 2024-03-29T11:36:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.615+0000 7f5bcce00700 1 Processor -- start 2024-03-29T11:36:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.615+0000 7f5bcce00700 1 -- start start 2024-03-29T11:36:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.616+0000 7f5bcce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bc80d22a0 0x7f5bc80cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.616+0000 7f5bcce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bc815d930 0x7f5bc80c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.616+0000 7f5bcce00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bc8162030 0x7f5bc80c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.616+0000 7f5bcce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80b8c90 con 0x7f5bc815d930 2024-03-29T11:36:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.616+0000 7f5bcce00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80b6f10 con 0x7f5bc8162030 2024-03-29T11:36:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.616+0000 7f5bcce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80b6300 con 0x7f5bc80d22a0 2024-03-29T11:36:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bc815d930 0x7f5bc80c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bc815d930 0x7f5bc80c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33580/0 (socket says 172.21.15.12:33580) 2024-03-29T11:36:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc5400700 1 -- 172.21.15.12:0/1327051792 learned_addr learned my addr 172.21.15.12:0/1327051792 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc6800700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bc8162030 0x7f5bc80c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc5e00700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bc80d22a0 0x7f5bc80cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc5400700 1 -- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bc80d22a0 msgr2=0x7f5bc80cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc5400700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5bc80d22a0 0x7f5bc80cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc5400700 1 -- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bc8162030 msgr2=0x7f5bc80c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc5400700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5bc8162030 0x7f5bc80c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc5400700 1 -- 172.21.15.12:0/1327051792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc0009a70 con 0x7f5bc815d930 2024-03-29T11:36:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.617+0000 7f5bc5400700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bc815d930 0x7f5bc80c7af0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400c550 tx=0x7f5bb400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.618+0000 7f5bbe000700 1 -- 172.21.15.12:0/1327051792 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4004c80 con 0x7f5bc815d930 2024-03-29T11:36:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.618+0000 7f5bbe000700 1 -- 172.21.15.12:0/1327051792 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb4004e00 con 0x7f5bc815d930 2024-03-29T11:36:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.618+0000 7f5bbe000700 1 -- 172.21.15.12:0/1327051792 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4007310 con 0x7f5bc815d930 2024-03-29T11:36:14.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.618+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc80b65e0 con 0x7f5bc815d930 2024-03-29T11:36:14.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.618+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc80b6ba0 con 0x7f5bc815d930 2024-03-29T11:36:14.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.619+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5bc80d4410 con 0x7f5bc815d930 2024-03-29T11:36:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.620+0000 7f5bbe000700 1 -- 172.21.15.12:0/1327051792 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5bb4007470 con 0x7f5bc815d930 2024-03-29T11:36:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.621+0000 7f5bbe000700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ba806adb0 0x7f5ba806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.621+0000 7f5bbe000700 1 -- 172.21.15.12:0/1327051792 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f5bb4088fd0 con 0x7f5bc815d930 2024-03-29T11:36:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.621+0000 7f5bbe000700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] conn(0x7f5ba80712c0 0x7f5ba80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.621+0000 7f5bbe000700 1 -- 172.21.15.12:0/1327051792 --> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ba8073db0 con 0x7f5ba80712c0 2024-03-29T11:36:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.621+0000 7f5bbe000700 1 -- 172.21.15.12:0/1327051792 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bb408b630 con 0x7f5bc815d930 2024-03-29T11:36:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.621+0000 7f5bc6800700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] conn(0x7f5ba80712c0 0x7f5ba80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.621+0000 7f5bc5e00700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ba806adb0 0x7f5ba806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.622+0000 7f5bc6800700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] conn(0x7f5ba80712c0 0x7f5ba80736e0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:14.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.622+0000 7f5bc5e00700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ba806adb0 0x7f5ba806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009be0 tx=0x7f5bb8009210 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:14.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.622+0000 7f5bbe000700 1 -- 172.21.15.12:0/1327051792 <== osd.1 v2:172.21.15.12:6816/43909 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5ba8073db0 con 0x7f5ba80712c0 2024-03-29T11:36:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.637+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 --> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5bc815e860 con 0x7f5ba80712c0 2024-03-29T11:36:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.638+0000 7f5bbe000700 1 -- 172.21.15.12:0/1327051792 <== osd.1 v2:172.21.15.12:6816/43909 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5bc00098f0 con 0x7f5ba80712c0 2024-03-29T11:36:14.637 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:36:14.637 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T11:36:14.637 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:36:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.638+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 >> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] conn(0x7f5ba80712c0 msgr2=0x7f5ba80736e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.638+0000 7f5bcce00700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.12:6816/43909,v1:172.21.15.12:6817/43909] conn(0x7f5ba80712c0 0x7f5ba80736e0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.638+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ba806adb0 msgr2=0x7f5ba806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.638+0000 7f5bcce00700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f5ba806adb0 0x7f5ba806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009be0 tx=0x7f5bb8009210 comp rx=0 tx=0).stop 2024-03-29T11:36:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.638+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bc815d930 msgr2=0x7f5bc80c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.638+0000 7f5bcce00700 1 --2- 172.21.15.12:0/1327051792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bc815d930 0x7f5bc80c7af0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400c550 tx=0x7f5bb400c860 comp rx=0 tx=0).stop 2024-03-29T11:36:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.638+0000 7f5bc6800700 1 -- 172.21.15.12:0/1327051792 reap_dead start 2024-03-29T11:36:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.638+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 shutdown_connections 2024-03-29T11:36:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.638+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 >> 172.21.15.12:0/1327051792 conn(0x7f5bc808aff0 msgr2=0x7f5bc806d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.639+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 shutdown_connections 2024-03-29T11:36:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.639+0000 7f5bcce00700 1 -- 172.21.15.12:0/1327051792 wait complete. 2024-03-29T11:36:14.654 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-29T11:36:14.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.855+0000 7f6f96600700 1 Processor -- start 2024-03-29T11:36:14.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.855+0000 7f6f96600700 1 -- start start 2024-03-29T11:36:14.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.856+0000 7f6f96600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f900cd480 0x7f6f9006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.856+0000 7f6f96600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f900cdae0 con 0x7f6f900cd020 2024-03-29T11:36:14.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.856+0000 7f6f96600700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f900cddc0 con 0x7f6f900cbb30 2024-03-29T11:36:14.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.856+0000 7f6f96600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f900ce0a0 con 0x7f6f900cd480 2024-03-29T11:36:14.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.857+0000 7f6f8fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f900cd480 0x7f6f9006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.857+0000 7f6f8fe00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f900cd480 0x7f6f9006e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33582/0 (socket says 172.21.15.12:33582) 2024-03-29T11:36:14.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.857+0000 7f6f8fe00700 1 -- 172.21.15.12:0/1573688048 learned_addr learned my addr 172.21.15.12:0/1573688048 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:14.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.857+0000 7f6f8ea00700 1 --1- 172.21.15.12:0/1573688048 >> v1:172.21.15.12:6789/0 conn(0x7f6f900cd020 0x7f6f900b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55704/0 (socket says 172.21.15.12:55704) 2024-03-29T11:36:14.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.857+0000 7f6f8fe00700 1 -- 172.21.15.12:0/1573688048 >> v1:172.21.15.159:6789/0 conn(0x7f6f900cbb30 legacy=0x7f6f900cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.857+0000 7f6f8fe00700 1 -- 172.21.15.12:0/1573688048 >> v1:172.21.15.12:6789/0 conn(0x7f6f900cd020 legacy=0x7f6f900b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.857+0000 7f6f8fe00700 1 -- 172.21.15.12:0/1573688048 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9006dc20 con 0x7f6f900cd480 2024-03-29T11:36:14.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.857+0000 7f6f8fe00700 1 --2- 172.21.15.12:0/1573688048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd480 0x7f6f9006e420 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009e50 tx=0x7f6f8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c9860b8bdc5faa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.858+0000 7f6f8e000700 1 -- 172.21.15.12:0/1573688048 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8800e070 con 0x7f6f900cd480 2024-03-29T11:36:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.858+0000 7f6f8e000700 1 -- 172.21.15.12:0/1573688048 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8800b9c0 con 0x7f6f900cd480 2024-03-29T11:36:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.858+0000 7f6f8e000700 1 -- 172.21.15.12:0/1573688048 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8800bd70 con 0x7f6f900cd480 2024-03-29T11:36:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.858+0000 7f6f96600700 1 -- 172.21.15.12:0/1573688048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd480 msgr2=0x7f6f9006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.858+0000 7f6f96600700 1 --2- 172.21.15.12:0/1573688048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd480 0x7f6f9006e420 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009e50 tx=0x7f6f8800b330 comp rx=0 tx=0).stop 2024-03-29T11:36:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.858+0000 7f6f96600700 1 -- 172.21.15.12:0/1573688048 shutdown_connections 2024-03-29T11:36:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.858+0000 7f6f96600700 1 --2- 172.21.15.12:0/1573688048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd480 0x7f6f9006e420 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.858+0000 7f6f96600700 1 -- 172.21.15.12:0/1573688048 >> 172.21.15.12:0/1573688048 conn(0x7f6f900b0d70 msgr2=0x7f6f900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.858+0000 7f6f96600700 1 -- 172.21.15.12:0/1573688048 shutdown_connections 2024-03-29T11:36:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.858+0000 7f6f96600700 1 -- 172.21.15.12:0/1573688048 wait complete. 2024-03-29T11:36:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.860+0000 7f6f96600700 1 Processor -- start 2024-03-29T11:36:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.860+0000 7f6f96600700 1 -- start start 2024-03-29T11:36:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.861+0000 7f6f96600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f900cbb30 0x7f6f900b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.861+0000 7f6f96600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd020 0x7f6f900c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.861+0000 7f6f96600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f900cd480 0x7f6f901f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.861+0000 7f6f96600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900afe10 con 0x7f6f900cd020 2024-03-29T11:36:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.861+0000 7f6f96600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f901f47e0 con 0x7f6f900cd480 2024-03-29T11:36:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.861+0000 7f6f96600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f901f4220 con 0x7f6f900cbb30 2024-03-29T11:36:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.861+0000 7f6f8ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd020 0x7f6f900c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.861+0000 7f6f8ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd020 0x7f6f900c8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33590/0 (socket says 172.21.15.12:33590) 2024-03-29T11:36:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.861+0000 7f6f8ea00700 1 -- 172.21.15.12:0/2072404201 learned_addr learned my addr 172.21.15.12:0/2072404201 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.861+0000 7f6f8fe00700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f900cd480 0x7f6f901f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f8f400700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f900cbb30 0x7f6f900b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f8ea00700 1 -- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f900cbb30 msgr2=0x7f6f900b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f8ea00700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f900cbb30 0x7f6f900b0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f8ea00700 1 -- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f900cd480 msgr2=0x7f6f901f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f8ea00700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f900cd480 0x7f6f901f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:14.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f8ea00700 1 -- 172.21.15.12:0/2072404201 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f88009b30 con 0x7f6f900cd020 2024-03-29T11:36:14.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f8f400700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f900cbb30 0x7f6f900b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:14.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f8ea00700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd020 0x7f6f900c8660 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00c500 tx=0x7f6f7c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:14.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f87400700 1 -- 172.21.15.12:0/2072404201 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c005cb0 con 0x7f6f900cd020 2024-03-29T11:36:14.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f87400700 1 -- 172.21.15.12:0/2072404201 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c005e10 con 0x7f6f900cd020 2024-03-29T11:36:14.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f96600700 1 -- 172.21.15.12:0/2072404201 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f901f4500 con 0x7f6f900cd020 2024-03-29T11:36:14.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f87400700 1 -- 172.21.15.12:0/2072404201 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c0072d0 con 0x7f6f900cd020 2024-03-29T11:36:14.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.862+0000 7f6f96600700 1 -- 172.21.15.12:0/2072404201 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f900b0770 con 0x7f6f900cd020 2024-03-29T11:36:14.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.864+0000 7f6f96600700 1 -- 172.21.15.12:0/2072404201 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f900aff50 con 0x7f6f900cd020 2024-03-29T11:36:14.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.865+0000 7f6f87400700 1 -- 172.21.15.12:0/2072404201 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6f7c007430 con 0x7f6f900cd020 2024-03-29T11:36:14.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.865+0000 7f6f87400700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f7006adb0 0x7f6f7006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:14.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.866+0000 7f6f87400700 1 -- 172.21.15.12:0/2072404201 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f6f7c0886a0 con 0x7f6f900cd020 2024-03-29T11:36:14.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.866+0000 7f6f8f400700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f7006adb0 0x7f6f7006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:14.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.867+0000 7f6f8f400700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f7006adb0 0x7f6f7006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6f9006eed0 tx=0x7f6f80009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:14.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:14.870+0000 7f6f87400700 1 -- 172.21.15.12:0/2072404201 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f900aff50 con 0x7f6f900cd020 2024-03-29T11:36:15.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.086+0000 7f6f96600700 1 -- 172.21.15.12:0/2072404201 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f9015bdd0 con 0x7f6f7006adb0 2024-03-29T11:36:15.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.086+0000 7f6f87400700 1 -- 172.21.15.12:0/2072404201 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6f800050d0 con 0x7f6f7006adb0 2024-03-29T11:36:15.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 -- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f7006adb0 msgr2=0x7f6f7006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f7006adb0 0x7f6f7006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6f9006eed0 tx=0x7f6f80009040 comp rx=0 tx=0).stop 2024-03-29T11:36:15.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 -- 172.21.15.12:0/2072404201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd020 msgr2=0x7f6f900c8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd020 0x7f6f900c8660 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00c500 tx=0x7f6f7c00c810 comp rx=0 tx=0).stop 2024-03-29T11:36:15.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 -- 172.21.15.12:0/2072404201 shutdown_connections 2024-03-29T11:36:15.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f6f7006adb0 0x7f6f7006d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6f900cd480 0x7f6f901f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6f900cbb30 0x7f6f900b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 --2- 172.21.15.12:0/2072404201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f900cd020 0x7f6f900c8660 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 -- 172.21.15.12:0/2072404201 >> 172.21.15.12:0/2072404201 conn(0x7f6f900b0d70 msgr2=0x7f6f9006e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:15.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 -- 172.21.15.12:0/2072404201 shutdown_connections 2024-03-29T11:36:15.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.089+0000 7f6f96600700 1 -- 172.21.15.12:0/2072404201 wait complete. 2024-03-29T11:36:15.090 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 1 to deep-scrub 2024-03-29T11:36:15.107 INFO:tasks.ceph:Scrubbing osd.2 2024-03-29T11:36:15.107 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-29T11:36:15.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.311+0000 7f4e30600700 1 Processor -- start 2024-03-29T11:36:15.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.311+0000 7f4e30600700 1 -- start start 2024-03-29T11:36:15.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.312+0000 7f4e30600700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4e2815d930 0x7f4e280d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.312+0000 7f4e30600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e28162030 0x7f4e280d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.312+0000 7f4e30600700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2815dd10 con 0x7f4e280d20d0 2024-03-29T11:36:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.312+0000 7f4e30600700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2815ec40 con 0x7f4e2815d930 2024-03-29T11:36:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.312+0000 7f4e30600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280d2560 con 0x7f4e28162030 2024-03-29T11:36:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.313+0000 7f4e2dc00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4e2815d930 0x7f4e280d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.313+0000 7f4e2e600700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4e280d20d0 0x7f4e280cfc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55716/0 (socket says 172.21.15.12:55716) 2024-03-29T11:36:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.313+0000 7f4e2e600700 1 -- 172.21.15.12:0/465155444 learned_addr learned my addr 172.21.15.12:0/465155444 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.313+0000 7f4e2d200700 1 --2- 172.21.15.12:0/465155444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e28162030 0x7f4e280d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.313+0000 7f4e2dc00700 1 -- 172.21.15.12:0/465155444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e28162030 msgr2=0x7f4e280d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.313+0000 7f4e2dc00700 1 --2- 172.21.15.12:0/465155444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e28162030 0x7f4e280d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.313+0000 7f4e2dc00700 1 -- 172.21.15.12:0/465155444 >> v1:172.21.15.12:6789/0 conn(0x7f4e280d20d0 legacy=0x7f4e280cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.313+0000 7f4e2dc00700 1 -- 172.21.15.12:0/465155444 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2815dfc0 con 0x7f4e2815d930 2024-03-29T11:36:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.313+0000 7f4e27e00700 1 -- 172.21.15.12:0/465155444 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532627041 0 0) 0x7f4e2815dd10 con 0x7f4e280d20d0 2024-03-29T11:36:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.313+0000 7f4e2dc00700 1 --2- 172.21.15.12:0/465155444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2815d930 0x7f4e280d3ed0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009840 tx=0x7f4e1c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154804508106c840 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e27e00700 1 -- 172.21.15.12:0/465155444 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c011830 con 0x7f4e2815d930 2024-03-29T11:36:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e27e00700 1 -- 172.21.15.12:0/465155444 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1c011990 con 0x7f4e2815d930 2024-03-29T11:36:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e27e00700 1 -- 172.21.15.12:0/465155444 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c011d20 con 0x7f4e2815d930 2024-03-29T11:36:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e30600700 1 -- 172.21.15.12:0/465155444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2815d930 msgr2=0x7f4e280d3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e30600700 1 --2- 172.21.15.12:0/465155444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2815d930 0x7f4e280d3ed0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009840 tx=0x7f4e1c009b50 comp rx=0 tx=0).stop 2024-03-29T11:36:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e30600700 1 -- 172.21.15.12:0/465155444 shutdown_connections 2024-03-29T11:36:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e30600700 1 --2- 172.21.15.12:0/465155444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e2815d930 0x7f4e280d3ed0 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e30600700 1 --2- 172.21.15.12:0/465155444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e28162030 0x7f4e280d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e30600700 1 -- 172.21.15.12:0/465155444 >> 172.21.15.12:0/465155444 conn(0x7f4e2808aff0 msgr2=0x7f4e280d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e30600700 1 -- 172.21.15.12:0/465155444 shutdown_connections 2024-03-29T11:36:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.314+0000 7f4e30600700 1 -- 172.21.15.12:0/465155444 wait complete. 2024-03-29T11:36:15.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.316+0000 7f4e30600700 1 Processor -- start 2024-03-29T11:36:15.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.316+0000 7f4e30600700 1 -- start start 2024-03-29T11:36:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.317+0000 7f4e30600700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e280d20d0 0x7f4e2815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.317+0000 7f4e30600700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e2815d930 0x7f4e280c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.317+0000 7f4e30600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e28162030 0x7f4e280c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.317+0000 7f4e30600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280cbf50 con 0x7f4e28162030 2024-03-29T11:36:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.317+0000 7f4e30600700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2815a8d0 con 0x7f4e2815d930 2024-03-29T11:36:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.317+0000 7f4e30600700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280cb2d0 con 0x7f4e280d20d0 2024-03-29T11:36:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e28162030 0x7f4e280c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e28162030 0x7f4e280c75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33606/0 (socket says 172.21.15.12:33606) 2024-03-29T11:36:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2e600700 1 -- 172.21.15.12:0/2912776230 learned_addr learned my addr 172.21.15.12:0/2912776230 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2dc00700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e280d20d0 0x7f4e2815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2d200700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e2815d930 0x7f4e280c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2e600700 1 -- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e280d20d0 msgr2=0x7f4e2815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2e600700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e280d20d0 0x7f4e2815a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2e600700 1 -- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e2815d930 msgr2=0x7f4e280c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2e600700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e2815d930 0x7f4e280c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2e600700 1 -- 172.21.15.12:0/2912776230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c0094a0 con 0x7f4e28162030 2024-03-29T11:36:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e2e600700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e28162030 0x7f4e280c75e0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f4e20002230 tx=0x7f4e2000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.318+0000 7f4e25600700 1 -- 172.21.15.12:0/2912776230 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2000aac0 con 0x7f4e28162030 2024-03-29T11:36:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.319+0000 7f4e25600700 1 -- 172.21.15.12:0/2912776230 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2000ac40 con 0x7f4e28162030 2024-03-29T11:36:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.319+0000 7f4e25600700 1 -- 172.21.15.12:0/2912776230 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e200070d0 con 0x7f4e28162030 2024-03-29T11:36:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.319+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e280cb5b0 con 0x7f4e28162030 2024-03-29T11:36:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.319+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e280cbb70 con 0x7f4e28162030 2024-03-29T11:36:15.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.320+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4e2815e860 con 0x7f4e28162030 2024-03-29T11:36:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.321+0000 7f4e25600700 1 -- 172.21.15.12:0/2912776230 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4e20007250 con 0x7f4e28162030 2024-03-29T11:36:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.322+0000 7f4e25600700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e0806adb0 0x7f4e0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.322+0000 7f4e25600700 1 -- 172.21.15.12:0/2912776230 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f4e20088fc0 con 0x7f4e28162030 2024-03-29T11:36:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.322+0000 7f4e25600700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] conn(0x7f4e080712c0 0x7f4e080736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.322+0000 7f4e25600700 1 -- 172.21.15.12:0/2912776230 --> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e08073db0 con 0x7f4e080712c0 2024-03-29T11:36:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.322+0000 7f4e25600700 1 -- 172.21.15.12:0/2912776230 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e2008b5a0 con 0x7f4e28162030 2024-03-29T11:36:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.322+0000 7f4e2dc00700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e0806adb0 0x7f4e0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.322+0000 7f4e2d200700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] conn(0x7f4e080712c0 0x7f4e080736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.323+0000 7f4e2d200700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] conn(0x7f4e080712c0 0x7f4e080736e0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.323+0000 7f4e2dc00700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e0806adb0 0x7f4e0806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0090b0 tx=0x7f4e1c008030 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:15.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.323+0000 7f4e25600700 1 -- 172.21.15.12:0/2912776230 <== osd.2 v2:172.21.15.12:6800/43907 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4e08073db0 con 0x7f4e080712c0 2024-03-29T11:36:15.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.338+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 --> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4e2815e580 con 0x7f4e080712c0 2024-03-29T11:36:15.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e25600700 1 -- 172.21.15.12:0/2912776230 <== osd.2 v2:172.21.15.12:6800/43907 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4e140076b0 con 0x7f4e080712c0 2024-03-29T11:36:15.338 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:36:15.338 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T11:36:15.338 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:36:15.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 >> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] conn(0x7f4e080712c0 msgr2=0x7f4e080736e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e30600700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.12:6800/43907,v1:172.21.15.12:6801/43907] conn(0x7f4e080712c0 0x7f4e080736e0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e0806adb0 msgr2=0x7f4e0806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e30600700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f4e0806adb0 0x7f4e0806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0090b0 tx=0x7f4e1c008030 comp rx=0 tx=0).stop 2024-03-29T11:36:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e28162030 msgr2=0x7f4e280c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e30600700 1 --2- 172.21.15.12:0/2912776230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e28162030 0x7f4e280c75e0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f4e20002230 tx=0x7f4e2000cf50 comp rx=0 tx=0).stop 2024-03-29T11:36:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e2e600700 1 -- 172.21.15.12:0/2912776230 reap_dead start 2024-03-29T11:36:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 shutdown_connections 2024-03-29T11:36:15.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 >> 172.21.15.12:0/2912776230 conn(0x7f4e2808aff0 msgr2=0x7f4e280cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:15.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 shutdown_connections 2024-03-29T11:36:15.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.339+0000 7f4e30600700 1 -- 172.21.15.12:0/2912776230 wait complete. 2024-03-29T11:36:15.355 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-29T11:36:15.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.556+0000 7fa804400700 1 Processor -- start 2024-03-29T11:36:15.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.557+0000 7fa804400700 1 -- start start 2024-03-29T11:36:15.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.558+0000 7fa804400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa7fc0d21f0 0x7fa7fc0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.558+0000 7fa804400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7fc0cec40 0x7fa7fc161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.558+0000 7fa804400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc0ce270 con 0x7fa7fc1611b0 2024-03-29T11:36:15.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.558+0000 7fa804400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7fc15b600 con 0x7fa7fc0cec40 2024-03-29T11:36:15.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.558+0000 7fa804400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7fc0d2760 con 0x7fa7fc0d21f0 2024-03-29T11:36:15.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.558+0000 7fa802400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7fc0cec40 0x7fa7fc161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.558+0000 7fa802400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7fc0cec40 0x7fa7fc161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33614/0 (socket says 172.21.15.12:33614) 2024-03-29T11:36:15.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.558+0000 7fa802400700 1 -- 172.21.15.12:0/2766642417 learned_addr learned my addr 172.21.15.12:0/2766642417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.558+0000 7fa801000700 1 --2- 172.21.15.12:0/2766642417 >> v2:172.21.15.159:3300/0 conn(0x7fa7fc0d21f0 0x7fa7fc0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.559+0000 7fa802400700 1 -- 172.21.15.12:0/2766642417 >> v2:172.21.15.159:3300/0 conn(0x7fa7fc0d21f0 msgr2=0x7fa7fc0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.559+0000 7fa802400700 1 --2- 172.21.15.12:0/2766642417 >> v2:172.21.15.159:3300/0 conn(0x7fa7fc0d21f0 0x7fa7fc0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.559+0000 7fa802400700 1 -- 172.21.15.12:0/2766642417 >> v1:172.21.15.159:6789/0 conn(0x7fa7fc1611b0 legacy=0x7fa7fc15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.559+0000 7fa802400700 1 -- 172.21.15.12:0/2766642417 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc0d2a40 con 0x7fa7fc0cec40 2024-03-29T11:36:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.559+0000 7fa802400700 1 --2- 172.21.15.12:0/2766642417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 0x7fa7fc161590 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc08c030 tx=0x7fa7f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556a4d8f166bf68c server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:15.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.559+0000 7fa7fbe00700 1 -- 172.21.15.12:0/2766642417 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f400e070 con 0x7fa7fc0cec40 2024-03-29T11:36:15.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.559+0000 7fa7fbe00700 1 -- 172.21.15.12:0/2766642417 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f400b9c0 con 0x7fa7fc0cec40 2024-03-29T11:36:15.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.559+0000 7fa7fbe00700 1 -- 172.21.15.12:0/2766642417 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f400bd70 con 0x7fa7fc0cec40 2024-03-29T11:36:15.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.559+0000 7fa804400700 1 -- 172.21.15.12:0/2766642417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 msgr2=0x7fa7fc161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.559+0000 7fa804400700 1 --2- 172.21.15.12:0/2766642417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 0x7fa7fc161590 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc08c030 tx=0x7fa7f400b330 comp rx=0 tx=0).stop 2024-03-29T11:36:15.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.560+0000 7fa804400700 1 -- 172.21.15.12:0/2766642417 shutdown_connections 2024-03-29T11:36:15.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.560+0000 7fa804400700 1 --2- 172.21.15.12:0/2766642417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 0x7fa7fc161590 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.560+0000 7fa804400700 1 --2- 172.21.15.12:0/2766642417 >> v2:172.21.15.159:3300/0 conn(0x7fa7fc0d21f0 0x7fa7fc0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.560+0000 7fa804400700 1 -- 172.21.15.12:0/2766642417 >> 172.21.15.12:0/2766642417 conn(0x7fa7fc08aff0 msgr2=0x7fa7fc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:15.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.560+0000 7fa804400700 1 -- 172.21.15.12:0/2766642417 shutdown_connections 2024-03-29T11:36:15.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.560+0000 7fa804400700 1 -- 172.21.15.12:0/2766642417 wait complete. 2024-03-29T11:36:15.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.561+0000 7fa804400700 1 Processor -- start 2024-03-29T11:36:15.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.562+0000 7fa804400700 1 -- start start 2024-03-29T11:36:15.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.562+0000 7fa804400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 0x7fa7fc0b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa804400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7fc0d21f0 0x7fa7fc0cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa804400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7fc1611b0 0x7fa7fc1e8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa804400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc0c7af0 con 0x7fa7fc0cec40 2024-03-29T11:36:15.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa804400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc0c71e0 con 0x7fa7fc0d21f0 2024-03-29T11:36:15.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa804400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc0c74a0 con 0x7fa7fc1611b0 2024-03-29T11:36:15.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa801a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 0x7fa7fc0b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa801a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 0x7fa7fc0b5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33618/0 (socket says 172.21.15.12:33618) 2024-03-29T11:36:15.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa801a00700 1 -- 172.21.15.12:0/1302434281 learned_addr learned my addr 172.21.15.12:0/1302434281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:15.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa801000700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7fc0d21f0 0x7fa7fc0cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa802400700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7fc1611b0 0x7fa7fc1e8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa801a00700 1 -- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7fc1611b0 msgr2=0x7fa7fc1e8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa801a00700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7fc1611b0 0x7fa7fc1e8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa801a00700 1 -- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7fc0d21f0 msgr2=0x7fa7fc0cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa801a00700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7fc0d21f0 0x7fa7fc0cb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa801a00700 1 -- 172.21.15.12:0/1302434281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f4009b30 con 0x7fa7fc0cec40 2024-03-29T11:36:15.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.563+0000 7fa802400700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7fc1611b0 0x7fa7fc1e8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:15.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.564+0000 7fa801a00700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 0x7fa7fc0b5390 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fa7f000c450 tx=0x7fa7f000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:15.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.564+0000 7fa7f9600700 1 -- 172.21.15.12:0/1302434281 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f00052d0 con 0x7fa7fc0cec40 2024-03-29T11:36:15.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.564+0000 7fa7f9600700 1 -- 172.21.15.12:0/1302434281 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f0005430 con 0x7fa7fc0cec40 2024-03-29T11:36:15.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.564+0000 7fa7f9600700 1 -- 172.21.15.12:0/1302434281 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f00038c0 con 0x7fa7fc0cec40 2024-03-29T11:36:15.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.564+0000 7fa804400700 1 -- 172.21.15.12:0/1302434281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc0c7780 con 0x7fa7fc0cec40 2024-03-29T11:36:15.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.564+0000 7fa804400700 1 -- 172.21.15.12:0/1302434281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc0cb5c0 con 0x7fa7fc0cec40 2024-03-29T11:36:15.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.565+0000 7fa804400700 1 -- 172.21.15.12:0/1302434281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc162150 con 0x7fa7fc0cec40 2024-03-29T11:36:15.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.566+0000 7fa7f9600700 1 -- 172.21.15.12:0/1302434281 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa7f0003a20 con 0x7fa7fc0cec40 2024-03-29T11:36:15.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.567+0000 7fa7f9600700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa7dc06adb0 0x7fa7dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:15.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.567+0000 7fa7f9600700 1 -- 172.21.15.12:0/1302434281 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fa7f0088830 con 0x7fa7fc0cec40 2024-03-29T11:36:15.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.567+0000 7fa801000700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa7dc06adb0 0x7fa7dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:15.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.568+0000 7fa801000700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa7dc06adb0 0x7fa7dc06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc1e83b0 tx=0x7fa7e8008040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:15.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.571+0000 7fa7f9600700 1 -- 172.21.15.12:0/1302434281 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7f00538d0 con 0x7fa7fc0cec40 2024-03-29T11:36:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.788+0000 7fa804400700 1 -- 172.21.15.12:0/1302434281 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7fc15f6a0 con 0x7fa7dc06adb0 2024-03-29T11:36:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.788+0000 7fa7f9600700 1 -- 172.21.15.12:0/1302434281 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa7e8009e40 con 0x7fa7dc06adb0 2024-03-29T11:36:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 -- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa7dc06adb0 msgr2=0x7fa7dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa7dc06adb0 0x7fa7dc06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc1e83b0 tx=0x7fa7e8008040 comp rx=0 tx=0).stop 2024-03-29T11:36:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 -- 172.21.15.12:0/1302434281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 msgr2=0x7fa7fc0b5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 0x7fa7fc0b5390 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fa7f000c450 tx=0x7fa7f000c760 comp rx=0 tx=0).stop 2024-03-29T11:36:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 -- 172.21.15.12:0/1302434281 shutdown_connections 2024-03-29T11:36:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fa7dc06adb0 0x7fa7dc06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa7fc1611b0 0x7fa7fc1e8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7fc0cec40 0x7fa7fc0b5390 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 --2- 172.21.15.12:0/1302434281 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7fc0d21f0 0x7fa7fc0cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 -- 172.21.15.12:0/1302434281 >> 172.21.15.12:0/1302434281 conn(0x7fa7fc08aff0 msgr2=0x7fa7fc06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 -- 172.21.15.12:0/1302434281 shutdown_connections 2024-03-29T11:36:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:15.791+0000 7fa804400700 1 -- 172.21.15.12:0/1302434281 wait complete. 2024-03-29T11:36:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 2 to deep-scrub 2024-03-29T11:36:15.808 INFO:tasks.ceph:Scrubbing osd.3 2024-03-29T11:36:15.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-29T11:36:16.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.011+0000 7f8644c00700 1 Processor -- start 2024-03-29T11:36:16.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.011+0000 7f8644c00700 1 -- start start 2024-03-29T11:36:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.012+0000 7f8644c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f864015d740 0x7f86400d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.012+0000 7f8644c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86400ced30 con 0x7f864015db20 2024-03-29T11:36:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.012+0000 7f8644c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86400cef80 con 0x7f86400d1fe0 2024-03-29T11:36:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.012+0000 7f8644c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86400cf260 con 0x7f864015d740 2024-03-29T11:36:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.013+0000 7f863de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f864015d740 0x7f86400d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.013+0000 7f863d400700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f864015db20 0x7f86400d18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55718/0 (socket says 172.21.15.12:55718) 2024-03-29T11:36:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.013+0000 7f863d400700 1 -- 172.21.15.12:0/3343827664 learned_addr learned my addr 172.21.15.12:0/3343827664 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.013+0000 7f863de00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f864015d740 0x7f86400d1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33628/0 (socket says 172.21.15.12:33628) 2024-03-29T11:36:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.013+0000 7f863de00700 1 -- 172.21.15.12:0/3343827664 >> v1:172.21.15.159:6789/0 conn(0x7f86400d1fe0 legacy=0x7f86400b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.013+0000 7f863de00700 1 -- 172.21.15.12:0/3343827664 >> v1:172.21.15.12:6789/0 conn(0x7f864015db20 legacy=0x7f86400d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.013+0000 7f863ca00700 1 -- 172.21.15.12:0/3343827664 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546960185 0 0) 0x7f86400ced30 con 0x7f864015db20 2024-03-29T11:36:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.013+0000 7f863de00700 1 -- 172.21.15.12:0/3343827664 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86400cf540 con 0x7f864015d740 2024-03-29T11:36:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.013+0000 7f863de00700 1 --2- 172.21.15.12:0/3343827664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864015d740 0x7f86400d1280 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f8630009840 tx=0x7f8630009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f638a9adab38e4ff server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.014+0000 7f863ca00700 1 -- 172.21.15.12:0/3343827664 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630011830 con 0x7f864015d740 2024-03-29T11:36:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.014+0000 7f863ca00700 1 -- 172.21.15.12:0/3343827664 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8630011990 con 0x7f864015d740 2024-03-29T11:36:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.014+0000 7f863ca00700 1 -- 172.21.15.12:0/3343827664 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630011d20 con 0x7f864015d740 2024-03-29T11:36:16.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.014+0000 7f8644c00700 1 -- 172.21.15.12:0/3343827664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864015d740 msgr2=0x7f86400d1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.014+0000 7f8644c00700 1 --2- 172.21.15.12:0/3343827664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864015d740 0x7f86400d1280 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f8630009840 tx=0x7f8630009b50 comp rx=0 tx=0).stop 2024-03-29T11:36:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.014+0000 7f8644c00700 1 -- 172.21.15.12:0/3343827664 shutdown_connections 2024-03-29T11:36:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.014+0000 7f8644c00700 1 --2- 172.21.15.12:0/3343827664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864015d740 0x7f86400d1280 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.014+0000 7f8644c00700 1 -- 172.21.15.12:0/3343827664 >> 172.21.15.12:0/3343827664 conn(0x7f864008aff0 msgr2=0x7f86400d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.014+0000 7f8644c00700 1 -- 172.21.15.12:0/3343827664 shutdown_connections 2024-03-29T11:36:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.014+0000 7f8644c00700 1 -- 172.21.15.12:0/3343827664 wait complete. 2024-03-29T11:36:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.016+0000 7f8644c00700 1 Processor -- start 2024-03-29T11:36:16.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.016+0000 7f8644c00700 1 -- start start 2024-03-29T11:36:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.017+0000 7f8644c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86400d1fe0 0x7f864015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.017+0000 7f8644c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f864015d740 0x7f86400b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.017+0000 7f8644c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864015db20 0x7f86400a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.017+0000 7f8644c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86400a0640 con 0x7f864015db20 2024-03-29T11:36:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.017+0000 7f8644c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86400b7010 con 0x7f864015d740 2024-03-29T11:36:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.017+0000 7f8644c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86400b6120 con 0x7f86400d1fe0 2024-03-29T11:36:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.017+0000 7f863e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864015db20 0x7f86400a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.017+0000 7f863e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864015db20 0x7f86400a0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33636/0 (socket says 172.21.15.12:33636) 2024-03-29T11:36:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.017+0000 7f863e800700 1 -- 172.21.15.12:0/4005770168 learned_addr learned my addr 172.21.15.12:0/4005770168 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.017+0000 7f863d400700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f864015d740 0x7f86400b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f863de00700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86400d1fe0 0x7f864015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f863e800700 1 -- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86400d1fe0 msgr2=0x7f864015b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f863e800700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86400d1fe0 0x7f864015b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f863e800700 1 -- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f864015d740 msgr2=0x7f86400b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f863e800700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f864015d740 0x7f86400b74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f863e800700 1 -- 172.21.15.12:0/4005770168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86300094a0 con 0x7f864015db20 2024-03-29T11:36:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f863e800700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864015db20 0x7f86400a0b40 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f863800cc50 tx=0x7f863800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f8636000700 1 -- 172.21.15.12:0/4005770168 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8638003c00 con 0x7f864015db20 2024-03-29T11:36:16.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f8636000700 1 -- 172.21.15.12:0/4005770168 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8638003d60 con 0x7f864015db20 2024-03-29T11:36:16.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f8636000700 1 -- 172.21.15.12:0/4005770168 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863800aab0 con 0x7f864015db20 2024-03-29T11:36:16.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86400b6400 con 0x7f864015db20 2024-03-29T11:36:16.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.018+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86400b69c0 con 0x7f864015db20 2024-03-29T11:36:16.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.019+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f86400cfb00 con 0x7f864015db20 2024-03-29T11:36:16.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.020+0000 7f8636000700 1 -- 172.21.15.12:0/4005770168 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f863800ac10 con 0x7f864015db20 2024-03-29T11:36:16.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.021+0000 7f8636000700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f862006adb0 0x7f862006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.021+0000 7f8636000700 1 -- 172.21.15.12:0/4005770168 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f86380894f0 con 0x7f864015db20 2024-03-29T11:36:16.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.022+0000 7f8636000700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] conn(0x7f86200712c0 0x7f86200736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.022+0000 7f8636000700 1 -- 172.21.15.12:0/4005770168 --> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8620073db0 con 0x7f86200712c0 2024-03-29T11:36:16.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.022+0000 7f8636000700 1 -- 172.21.15.12:0/4005770168 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7f863808bad0 con 0x7f864015db20 2024-03-29T11:36:16.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.022+0000 7f863d400700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] conn(0x7f86200712c0 0x7f86200736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.022+0000 7f863de00700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f862006adb0 0x7f862006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.022+0000 7f863d400700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] conn(0x7f86200712c0 0x7f86200736e0 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.023+0000 7f863de00700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f862006adb0 0x7f862006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f86300090b0 tx=0x7f863001c040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.023+0000 7f8636000700 1 -- 172.21.15.12:0/4005770168 <== osd.3 v2:172.21.15.12:6820/43910 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f8620073db0 con 0x7f86200712c0 2024-03-29T11:36:16.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.038+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 --> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8640067720 con 0x7f86200712c0 2024-03-29T11:36:16.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.038+0000 7f8636000700 1 -- 172.21.15.12:0/4005770168 <== osd.3 v2:172.21.15.12:6820/43910 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8640067720 con 0x7f86200712c0 2024-03-29T11:36:16.038 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:36:16.038 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T11:36:16.038 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:36:16.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 >> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] conn(0x7f86200712c0 msgr2=0x7f86200736e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f8644c00700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.12:6820/43910,v1:172.21.15.12:6822/43910] conn(0x7f86200712c0 0x7f86200736e0 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f862006adb0 msgr2=0x7f862006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f8644c00700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f862006adb0 0x7f862006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f86300090b0 tx=0x7f863001c040 comp rx=0 tx=0).stop 2024-03-29T11:36:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864015db20 msgr2=0x7f86400a0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f8644c00700 1 --2- 172.21.15.12:0/4005770168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864015db20 0x7f86400a0b40 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f863800cc50 tx=0x7f863800cf60 comp rx=0 tx=0).stop 2024-03-29T11:36:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f863e800700 1 -- 172.21.15.12:0/4005770168 reap_dead start 2024-03-29T11:36:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 shutdown_connections 2024-03-29T11:36:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 >> 172.21.15.12:0/4005770168 conn(0x7f864008aff0 msgr2=0x7f86400b28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 shutdown_connections 2024-03-29T11:36:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.039+0000 7f8644c00700 1 -- 172.21.15.12:0/4005770168 wait complete. 2024-03-29T11:36:16.056 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-29T11:36:16.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.256+0000 7f9678400700 1 Processor -- start 2024-03-29T11:36:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.256+0000 7f9678400700 1 -- start start 2024-03-29T11:36:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.257+0000 7f9678400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 0x7f96700725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.258+0000 7f9678400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96700709c0 con 0x7f96700ce230 2024-03-29T11:36:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.258+0000 7f9678400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9670070c80 con 0x7f96700ce670 2024-03-29T11:36:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.258+0000 7f9678400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96700ceb00 con 0x7f967015d800 2024-03-29T11:36:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.258+0000 7f9675000700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f96700ce230 0x7f96700b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55722/0 (socket says 172.21.15.12:55722) 2024-03-29T11:36:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.258+0000 7f9675000700 1 -- 172.21.15.12:0/1549167450 learned_addr learned my addr 172.21.15.12:0/1549167450 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.258+0000 7f9675a00700 1 --2- 172.21.15.12:0/1549167450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 0x7f96700725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.258+0000 7f966fe00700 1 -- 172.21.15.12:0/1549167450 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086598020 0 0) 0x7f96700709c0 con 0x7f96700ce230 2024-03-29T11:36:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.258+0000 7f966fe00700 1 -- 172.21.15.12:0/1549167450 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96580035b0 con 0x7f96700ce230 2024-03-29T11:36:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.258+0000 7f966fe00700 1 -- 172.21.15.12:0/1549167450 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969272362 0 0) 0x7f9670070c80 con 0x7f96700ce670 2024-03-29T11:36:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.258+0000 7f966fe00700 1 -- 172.21.15.12:0/1549167450 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96700709c0 con 0x7f96700ce670 2024-03-29T11:36:16.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.259+0000 7f9675a00700 1 -- 172.21.15.12:0/1549167450 >> v1:172.21.15.159:6789/0 conn(0x7f96700ce670 legacy=0x7f967015dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.259+0000 7f9675a00700 1 -- 172.21.15.12:0/1549167450 >> v1:172.21.15.12:6789/0 conn(0x7f96700ce230 legacy=0x7f96700b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.259+0000 7f9675a00700 1 -- 172.21.15.12:0/1549167450 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9670070f60 con 0x7f967015d800 2024-03-29T11:36:16.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.259+0000 7f9675a00700 1 --2- 172.21.15.12:0/1549167450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 0x7f96700725c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9664009870 tx=0x7f9664009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f88976c22f8bd46 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.259+0000 7f966fe00700 1 -- 172.21.15.12:0/1549167450 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664011830 con 0x7f967015d800 2024-03-29T11:36:16.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.259+0000 7f966fe00700 1 -- 172.21.15.12:0/1549167450 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9664011990 con 0x7f967015d800 2024-03-29T11:36:16.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.259+0000 7f966fe00700 1 -- 172.21.15.12:0/1549167450 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664011d20 con 0x7f967015d800 2024-03-29T11:36:16.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.260+0000 7f9678400700 1 -- 172.21.15.12:0/1549167450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 msgr2=0x7f96700725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.260+0000 7f9678400700 1 --2- 172.21.15.12:0/1549167450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 0x7f96700725c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9664009870 tx=0x7f9664009b80 comp rx=0 tx=0).stop 2024-03-29T11:36:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.260+0000 7f9678400700 1 -- 172.21.15.12:0/1549167450 shutdown_connections 2024-03-29T11:36:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.260+0000 7f9678400700 1 --2- 172.21.15.12:0/1549167450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 0x7f96700725c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.260+0000 7f9678400700 1 -- 172.21.15.12:0/1549167450 >> 172.21.15.12:0/1549167450 conn(0x7f96700b0d70 msgr2=0x7f96700b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.260+0000 7f9678400700 1 -- 172.21.15.12:0/1549167450 shutdown_connections 2024-03-29T11:36:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.260+0000 7f9678400700 1 -- 172.21.15.12:0/1549167450 wait complete. 2024-03-29T11:36:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.262+0000 7f9678400700 1 Processor -- start 2024-03-29T11:36:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.262+0000 7f9678400700 1 -- start start 2024-03-29T11:36:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9678400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96700ce230 0x7f967014d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9678400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96700ce670 0x7f967015b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9678400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 0x7f967014c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9678400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9670136ef0 con 0x7f96700ce670 2024-03-29T11:36:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9678400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967006ceb0 con 0x7f96700ce230 2024-03-29T11:36:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9678400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967006d190 con 0x7f967015d800 2024-03-29T11:36:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9675000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96700ce670 0x7f967015b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9675000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96700ce670 0x7f967015b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33646/0 (socket says 172.21.15.12:33646) 2024-03-29T11:36:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9675000700 1 -- 172.21.15.12:0/1412348671 learned_addr learned my addr 172.21.15.12:0/1412348671 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9675a00700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96700ce230 0x7f967014d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9676400700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 0x7f967014c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9675000700 1 -- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 msgr2=0x7f967014c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9675000700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 0x7f967014c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9675000700 1 -- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96700ce230 msgr2=0x7f967014d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9675000700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96700ce230 0x7f967014d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9675000700 1 -- 172.21.15.12:0/1412348671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96640094a0 con 0x7f96700ce670 2024-03-29T11:36:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9675a00700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96700ce230 0x7f967014d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.263+0000 7f9676400700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 0x7f967014c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.264+0000 7f9675000700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96700ce670 0x7f967015b810 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f965c001eb0 tx=0x7f965c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.264+0000 7f966d600700 1 -- 172.21.15.12:0/1412348671 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c005d70 con 0x7f96700ce670 2024-03-29T11:36:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.264+0000 7f966d600700 1 -- 172.21.15.12:0/1412348671 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965c0070d0 con 0x7f96700ce670 2024-03-29T11:36:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.264+0000 7f9678400700 1 -- 172.21.15.12:0/1412348671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967006d470 con 0x7f96700ce670 2024-03-29T11:36:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.264+0000 7f966d600700 1 -- 172.21.15.12:0/1412348671 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c007460 con 0x7f96700ce670 2024-03-29T11:36:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.264+0000 7f9678400700 1 -- 172.21.15.12:0/1412348671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967014db80 con 0x7f96700ce670 2024-03-29T11:36:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.265+0000 7f9678400700 1 -- 172.21.15.12:0/1412348671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9670137030 con 0x7f96700ce670 2024-03-29T11:36:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.266+0000 7f966d600700 1 -- 172.21.15.12:0/1412348671 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f965c0075c0 con 0x7f96700ce670 2024-03-29T11:36:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.267+0000 7f966d600700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f965006adb0 0x7f965006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.267+0000 7f966d600700 1 -- 172.21.15.12:0/1412348671 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f965c088a20 con 0x7f96700ce670 2024-03-29T11:36:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.267+0000 7f9675a00700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f965006adb0 0x7f965006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.268+0000 7f9675a00700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f965006adb0 0x7f965006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f96640097c0 tx=0x7f966401c040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.271+0000 7f966d600700 1 -- 172.21.15.12:0/1412348671 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9670137030 con 0x7f96700ce670 2024-03-29T11:36:16.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.485+0000 7f9678400700 1 -- 172.21.15.12:0/1412348671 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f9670072b00 con 0x7f965006adb0 2024-03-29T11:36:16.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.486+0000 7f966d600700 1 -- 172.21.15.12:0/1412348671 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9664011b80 con 0x7f965006adb0 2024-03-29T11:36:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 -- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f965006adb0 msgr2=0x7f965006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f965006adb0 0x7f965006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f96640097c0 tx=0x7f966401c040 comp rx=0 tx=0).stop 2024-03-29T11:36:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 -- 172.21.15.12:0/1412348671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96700ce670 msgr2=0x7f967015b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96700ce670 0x7f967015b810 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f965c001eb0 tx=0x7f965c00c860 comp rx=0 tx=0).stop 2024-03-29T11:36:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 -- 172.21.15.12:0/1412348671 shutdown_connections 2024-03-29T11:36:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f965006adb0 0x7f965006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f967015d800 0x7f967014c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96700ce230 0x7f967014d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 --2- 172.21.15.12:0/1412348671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96700ce670 0x7f967015b810 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 -- 172.21.15.12:0/1412348671 >> 172.21.15.12:0/1412348671 conn(0x7f96700b0d70 msgr2=0x7f967015dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 -- 172.21.15.12:0/1412348671 shutdown_connections 2024-03-29T11:36:16.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.488+0000 7f9678400700 1 -- 172.21.15.12:0/1412348671 wait complete. 2024-03-29T11:36:16.490 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 3 to deep-scrub 2024-03-29T11:36:16.505 INFO:tasks.ceph:Scrubbing osd.4 2024-03-29T11:36:16.505 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-29T11:36:16.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.708+0000 7f9ac5400700 1 Processor -- start 2024-03-29T11:36:16.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.709+0000 7f9ac5400700 1 -- start start 2024-03-29T11:36:16.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.710+0000 7f9ac5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d1fe0 0x7f9ac00d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.710+0000 7f9ac5400700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac00ced30 con 0x7f9ac015d740 2024-03-29T11:36:16.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.710+0000 7f9ac5400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac00cef80 con 0x7f9ac015db20 2024-03-29T11:36:16.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.710+0000 7f9ac5400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00cf260 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.710+0000 7f9abe800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9ac015d740 0x7f9ac00d18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55736/0 (socket says 172.21.15.12:55736) 2024-03-29T11:36:16.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.710+0000 7f9abe800700 1 -- 172.21.15.12:0/1911884991 learned_addr learned my addr 172.21.15.12:0/1911884991 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.710+0000 7f9abde00700 1 --2- 172.21.15.12:0/1911884991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d1fe0 0x7f9ac00d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.710+0000 7f9abd400700 1 -- 172.21.15.12:0/1911884991 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555118013 0 0) 0x7f9ac00ced30 con 0x7f9ac015d740 2024-03-29T11:36:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.711+0000 7f9abd400700 1 -- 172.21.15.12:0/1911884991 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa40035b0 con 0x7f9ac015d740 2024-03-29T11:36:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.711+0000 7f9abd400700 1 -- 172.21.15.12:0/1911884991 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239335610 0 0) 0x7f9ac00cef80 con 0x7f9ac015db20 2024-03-29T11:36:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.711+0000 7f9abd400700 1 -- 172.21.15.12:0/1911884991 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac00ced30 con 0x7f9ac015db20 2024-03-29T11:36:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.711+0000 7f9abde00700 1 -- 172.21.15.12:0/1911884991 >> v1:172.21.15.159:6789/0 conn(0x7f9ac015db20 legacy=0x7f9ac00b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.711+0000 7f9abde00700 1 -- 172.21.15.12:0/1911884991 >> v1:172.21.15.12:6789/0 conn(0x7f9ac015d740 legacy=0x7f9ac00d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.711+0000 7f9abde00700 1 -- 172.21.15.12:0/1911884991 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac00cf540 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.711+0000 7f9abde00700 1 --2- 172.21.15.12:0/1911884991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d1fe0 0x7f9ac00d1280 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009870 tx=0x7f9aac009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68492c1516b73049 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.711+0000 7f9abd400700 1 -- 172.21.15.12:0/1911884991 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac011830 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.711+0000 7f9abd400700 1 -- 172.21.15.12:0/1911884991 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac011990 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.711+0000 7f9abd400700 1 -- 172.21.15.12:0/1911884991 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac011d20 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.712+0000 7f9ac5400700 1 -- 172.21.15.12:0/1911884991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d1fe0 msgr2=0x7f9ac00d1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.712+0000 7f9ac5400700 1 --2- 172.21.15.12:0/1911884991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d1fe0 0x7f9ac00d1280 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009870 tx=0x7f9aac009b80 comp rx=0 tx=0).stop 2024-03-29T11:36:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.712+0000 7f9ac5400700 1 -- 172.21.15.12:0/1911884991 shutdown_connections 2024-03-29T11:36:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.712+0000 7f9ac5400700 1 --2- 172.21.15.12:0/1911884991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac00d1fe0 0x7f9ac00d1280 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.712+0000 7f9ac5400700 1 -- 172.21.15.12:0/1911884991 >> 172.21.15.12:0/1911884991 conn(0x7f9ac008aff0 msgr2=0x7f9ac00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.712+0000 7f9ac5400700 1 -- 172.21.15.12:0/1911884991 shutdown_connections 2024-03-29T11:36:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.712+0000 7f9ac5400700 1 -- 172.21.15.12:0/1911884991 wait complete. 2024-03-29T11:36:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.714+0000 7f9ac5400700 1 Processor -- start 2024-03-29T11:36:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.714+0000 7f9ac5400700 1 -- start start 2024-03-29T11:36:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.715+0000 7f9ac5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac00d1fe0 0x7f9ac00b5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.715+0000 7f9ac5400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac015d740 0x7f9ac00b8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.715+0000 7f9ac5400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac015db20 0x7f9ac00cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.715+0000 7f9ac5400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00b40b0 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.715+0000 7f9ac5400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00b4390 con 0x7f9ac015d740 2024-03-29T11:36:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.715+0000 7f9ac5400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00b4670 con 0x7f9ac015db20 2024-03-29T11:36:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.715+0000 7f9abe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac00d1fe0 0x7f9ac00b5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.715+0000 7f9abe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac00d1fe0 0x7f9ac00b5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33660/0 (socket says 172.21.15.12:33660) 2024-03-29T11:36:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.715+0000 7f9abe800700 1 -- 172.21.15.12:0/2294264125 learned_addr learned my addr 172.21.15.12:0/2294264125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.715+0000 7f9abde00700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac015d740 0x7f9ac00b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9abe800700 1 -- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac015db20 msgr2=0x7f9ac00cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9abf200700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac015db20 0x7f9ac00cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9abe800700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac015db20 0x7f9ac00cc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9abe800700 1 -- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac015d740 msgr2=0x7f9ac00b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9abe800700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac015d740 0x7f9ac00b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9abe800700 1 -- 172.21.15.12:0/2294264125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac0094a0 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9abde00700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ac015d740 0x7f9ac00b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:36:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9abf200700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ac015db20 0x7f9ac00cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:36:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9abe800700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac00d1fe0 0x7f9ac00b5150 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c700 tx=0x7f9ab000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9ab6a00700 1 -- 172.21.15.12:0/2294264125 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0005d70 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9ab6a00700 1 -- 172.21.15.12:0/2294264125 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab00070d0 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9ab6a00700 1 -- 172.21.15.12:0/2294264125 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0007480 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac00b4950 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.716+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac00b3d60 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.717+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ac00cfb00 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.718+0000 7f9ab6a00700 1 -- 172.21.15.12:0/2294264125 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9ab00075e0 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.719+0000 7f9ab6a00700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006ade0 0x7f9aa006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.719+0000 7f9ab6a00700 1 -- 172.21.15.12:0/2294264125 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f9ab00892b0 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.719+0000 7f9ab6a00700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] conn(0x7f9aa00712f0 0x7f9aa0073710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.719+0000 7f9ab6a00700 1 -- 172.21.15.12:0/2294264125 --> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aa0073de0 con 0x7f9aa00712f0 2024-03-29T11:36:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.719+0000 7f9ab6a00700 1 -- 172.21.15.12:0/2294264125 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ab0089700 con 0x7f9ac00d1fe0 2024-03-29T11:36:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.720+0000 7f9abde00700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006ade0 0x7f9aa006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.720+0000 7f9abf200700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] conn(0x7f9aa00712f0 0x7f9aa0073710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.720+0000 7f9abde00700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006ade0 0x7f9aa006d2a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00a11c0 tx=0x7f9aac01c040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.720+0000 7f9abf200700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] conn(0x7f9aa00712f0 0x7f9aa0073710 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.721+0000 7f9ab6a00700 1 -- 172.21.15.12:0/2294264125 <== osd.4 v2:172.21.15.159:6800/44146 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f9aa0073de0 con 0x7f9aa00712f0 2024-03-29T11:36:16.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.737+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 --> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9ac00b4e70 con 0x7f9aa00712f0 2024-03-29T11:36:16.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.737+0000 7f9ab6a00700 1 -- 172.21.15.12:0/2294264125 <== osd.4 v2:172.21.15.159:6800/44146 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9ac00b4e70 con 0x7f9aa00712f0 2024-03-29T11:36:16.736 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:36:16.737 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T11:36:16.737 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:36:16.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.737+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] conn(0x7f9aa00712f0 msgr2=0x7f9aa0073710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.737+0000 7f9ac5400700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:6800/44146,v1:172.21.15.159:6801/44146] conn(0x7f9aa00712f0 0x7f9aa0073710 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.738+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006ade0 msgr2=0x7f9aa006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.738+0000 7f9ac5400700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f9aa006ade0 0x7f9aa006d2a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00a11c0 tx=0x7f9aac01c040 comp rx=0 tx=0).stop 2024-03-29T11:36:16.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.738+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac00d1fe0 msgr2=0x7f9ac00b5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.738+0000 7f9ac5400700 1 --2- 172.21.15.12:0/2294264125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac00d1fe0 0x7f9ac00b5150 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c700 tx=0x7f9ab000cac0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.738+0000 7f9abf200700 1 -- 172.21.15.12:0/2294264125 reap_dead start 2024-03-29T11:36:16.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.738+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 shutdown_connections 2024-03-29T11:36:16.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.738+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 >> 172.21.15.12:0/2294264125 conn(0x7f9ac008aff0 msgr2=0x7f9ac00b0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.738+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 shutdown_connections 2024-03-29T11:36:16.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.738+0000 7f9ac5400700 1 -- 172.21.15.12:0/2294264125 wait complete. 2024-03-29T11:36:16.754 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-29T11:36:16.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.955+0000 7fdee8200700 1 Processor -- start 2024-03-29T11:36:16.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.955+0000 7fdee8200700 1 -- start start 2024-03-29T11:36:16.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.956+0000 7fdee8200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdee015a5c0 0x7fdee00709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.956+0000 7fdee8200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 0x7fdee00cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.956+0000 7fdee8200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee00cb310 con 0x7fdee015dfe0 2024-03-29T11:36:16.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.956+0000 7fdee8200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee0070f00 con 0x7fdee015a5c0 2024-03-29T11:36:16.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.956+0000 7fdee8200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00711a0 con 0x7fdee00ce840 2024-03-29T11:36:16.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.956+0000 7fdee5800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdee015a5c0 0x7fdee00709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.956+0000 7fdee4e00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdee015dfe0 0x7fdee00ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55738/0 (socket says 172.21.15.12:55738) 2024-03-29T11:36:16.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.956+0000 7fdee4e00700 1 -- 172.21.15.12:0/3995578419 learned_addr learned my addr 172.21.15.12:0/3995578419 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:16.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.957+0000 7fdee6200700 1 --2- 172.21.15.12:0/3995578419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 0x7fdee00cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.957+0000 7fdee5800700 1 -- 172.21.15.12:0/3995578419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 msgr2=0x7fdee00cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:16.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.957+0000 7fdee5800700 1 --2- 172.21.15.12:0/3995578419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 0x7fdee00cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.957+0000 7fdee5800700 1 -- 172.21.15.12:0/3995578419 >> v1:172.21.15.12:6789/0 conn(0x7fdee015dfe0 legacy=0x7fdee00ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.957+0000 7fdee5800700 1 -- 172.21.15.12:0/3995578419 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee0071480 con 0x7fdee015a5c0 2024-03-29T11:36:16.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.957+0000 7fdedfe00700 1 -- 172.21.15.12:0/3995578419 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285552812 0 0) 0x7fdee00cb310 con 0x7fdee015dfe0 2024-03-29T11:36:16.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.957+0000 7fdee5800700 1 --2- 172.21.15.12:0/3995578419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015a5c0 0x7fdee00709c0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fded4009840 tx=0x7fded4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c706c9920468fa server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.957+0000 7fdedfe00700 1 -- 172.21.15.12:0/3995578419 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4011830 con 0x7fdee015a5c0 2024-03-29T11:36:16.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.957+0000 7fdedfe00700 1 -- 172.21.15.12:0/3995578419 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded4011990 con 0x7fdee015a5c0 2024-03-29T11:36:16.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.958+0000 7fdedfe00700 1 -- 172.21.15.12:0/3995578419 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4011d20 con 0x7fdee015a5c0 2024-03-29T11:36:16.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.958+0000 7fdee8200700 1 -- 172.21.15.12:0/3995578419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015a5c0 msgr2=0x7fdee00709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.958+0000 7fdee8200700 1 --2- 172.21.15.12:0/3995578419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015a5c0 0x7fdee00709c0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fded4009840 tx=0x7fded4009b50 comp rx=0 tx=0).stop 2024-03-29T11:36:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.958+0000 7fdee8200700 1 -- 172.21.15.12:0/3995578419 shutdown_connections 2024-03-29T11:36:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.958+0000 7fdee8200700 1 --2- 172.21.15.12:0/3995578419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 0x7fdee00cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.958+0000 7fdee8200700 1 --2- 172.21.15.12:0/3995578419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015a5c0 0x7fdee00709c0 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.958+0000 7fdee8200700 1 -- 172.21.15.12:0/3995578419 >> 172.21.15.12:0/3995578419 conn(0x7fdee00b0d70 msgr2=0x7fdee00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.958+0000 7fdee8200700 1 -- 172.21.15.12:0/3995578419 shutdown_connections 2024-03-29T11:36:16.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.958+0000 7fdee8200700 1 -- 172.21.15.12:0/3995578419 wait complete. 2024-03-29T11:36:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.960+0000 7fdee8200700 1 Processor -- start 2024-03-29T11:36:16.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.960+0000 7fdee8200700 1 -- start start 2024-03-29T11:36:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee8200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 0x7fdee01464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee8200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdee015a5c0 0x7fdee01554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee8200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015dfe0 0x7fdee0155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee8200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee0157770 con 0x7fdee015dfe0 2024-03-29T11:36:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee8200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee01566d0 con 0x7fdee015a5c0 2024-03-29T11:36:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee8200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee01569b0 con 0x7fdee00ce840 2024-03-29T11:36:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015dfe0 0x7fdee0155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015dfe0 0x7fdee0155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33678/0 (socket says 172.21.15.12:33678) 2024-03-29T11:36:16.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee6200700 1 -- 172.21.15.12:0/3386050445 learned_addr learned my addr 172.21.15.12:0/3386050445 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:16.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee5800700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 0x7fdee01464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.961+0000 7fdee4e00700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdee015a5c0 0x7fdee01554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdee6200700 1 -- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 msgr2=0x7fdee01464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdee6200700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 0x7fdee01464a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdee6200700 1 -- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdee015a5c0 msgr2=0x7fdee01554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdee6200700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdee015a5c0 0x7fdee01554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:16.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdee6200700 1 -- 172.21.15.12:0/3386050445 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded40094a0 con 0x7fdee015dfe0 2024-03-29T11:36:16.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdee5800700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 0x7fdee01464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:16.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdee6200700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015dfe0 0x7fdee0155a00 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fded800efc0 tx=0x7fded800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdedd600700 1 -- 172.21.15.12:0/3386050445 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded8009ca0 con 0x7fdee015dfe0 2024-03-29T11:36:16.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdedd600700 1 -- 172.21.15.12:0/3386050445 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded800f040 con 0x7fdee015dfe0 2024-03-29T11:36:16.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdee8200700 1 -- 172.21.15.12:0/3386050445 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee0156c90 con 0x7fdee015dfe0 2024-03-29T11:36:16.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdedd600700 1 -- 172.21.15.12:0/3386050445 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded8014480 con 0x7fdee015dfe0 2024-03-29T11:36:16.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.962+0000 7fdee8200700 1 -- 172.21.15.12:0/3386050445 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee0157250 con 0x7fdee015dfe0 2024-03-29T11:36:16.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.963+0000 7fdee8200700 1 -- 172.21.15.12:0/3386050445 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee0067720 con 0x7fdee015dfe0 2024-03-29T11:36:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.964+0000 7fdedd600700 1 -- 172.21.15.12:0/3386050445 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fded80145e0 con 0x7fdee015dfe0 2024-03-29T11:36:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.965+0000 7fdedd600700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdec006adb0 0x7fdec006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:16.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.965+0000 7fdedd600700 1 -- 172.21.15.12:0/3386050445 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fded808a2f0 con 0x7fdee015dfe0 2024-03-29T11:36:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.966+0000 7fdee5800700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdec006adb0 0x7fdec006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:16.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.967+0000 7fdee5800700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdec006adb0 0x7fdec006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fded4000c00 tx=0x7fded4008030 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:16.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:16.970+0000 7fdedd600700 1 -- 172.21.15.12:0/3386050445 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fded8055280 con 0x7fdee015dfe0 2024-03-29T11:36:17.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.182+0000 7fdee8200700 1 -- 172.21.15.12:0/3386050445 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fdee00cb560 con 0x7fdec006adb0 2024-03-29T11:36:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.183+0000 7fdedd600700 1 -- 172.21.15.12:0/3386050445 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fded4011cb0 con 0x7fdec006adb0 2024-03-29T11:36:17.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.185+0000 7fdee8200700 1 -- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdec006adb0 msgr2=0x7fdec006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.185+0000 7fdee8200700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdec006adb0 0x7fdec006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fded4000c00 tx=0x7fded4008030 comp rx=0 tx=0).stop 2024-03-29T11:36:17.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.185+0000 7fdee8200700 1 -- 172.21.15.12:0/3386050445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015dfe0 msgr2=0x7fdee0155a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.185+0000 7fdee8200700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015dfe0 0x7fdee0155a00 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fded800efc0 tx=0x7fded800c3a0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.186+0000 7fdee8200700 1 -- 172.21.15.12:0/3386050445 shutdown_connections 2024-03-29T11:36:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.186+0000 7fdee8200700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fdec006adb0 0x7fdec006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.186+0000 7fdee8200700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee015dfe0 0x7fdee0155a00 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.186+0000 7fdee8200700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdee00ce840 0x7fdee01464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.186+0000 7fdee8200700 1 --2- 172.21.15.12:0/3386050445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdee015a5c0 0x7fdee01554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.186+0000 7fdee8200700 1 -- 172.21.15.12:0/3386050445 >> 172.21.15.12:0/3386050445 conn(0x7fdee00b0d70 msgr2=0x7fdee006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.186+0000 7fdee8200700 1 -- 172.21.15.12:0/3386050445 shutdown_connections 2024-03-29T11:36:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.186+0000 7fdee8200700 1 -- 172.21.15.12:0/3386050445 wait complete. 2024-03-29T11:36:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 4 to deep-scrub 2024-03-29T11:36:17.203 INFO:tasks.ceph:Scrubbing osd.5 2024-03-29T11:36:17.203 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-29T11:36:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.406+0000 7f2ac1c00700 1 Processor -- start 2024-03-29T11:36:17.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.406+0000 7f2ac1c00700 1 -- start start 2024-03-29T11:36:17.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.407+0000 7f2ac1c00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2abc0d1770 0x7f2abc0d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.407+0000 7f2ac1c00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc0ceb70 con 0x7f2abc0b56e0 2024-03-29T11:36:17.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.407+0000 7f2ac1c00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc0cf000 con 0x7f2abc15d7f0 2024-03-29T11:36:17.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.407+0000 7f2ac1c00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2abc15dbd0 con 0x7f2abc0d1770 2024-03-29T11:36:17.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.407+0000 7f2abb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2abc0d1770 0x7f2abc0d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.407+0000 7f2abb400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2abc0d1770 0x7f2abc0d25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33688/0 (socket says 172.21.15.12:33688) 2024-03-29T11:36:17.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.407+0000 7f2abb400700 1 -- 172.21.15.12:0/1094316893 learned_addr learned my addr 172.21.15.12:0/1094316893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:17.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.407+0000 7f2aba000700 1 --1- 172.21.15.12:0/1094316893 >> v1:172.21.15.12:6789/0 conn(0x7f2abc0b56e0 0x7f2abc06ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55744/0 (socket says 172.21.15.12:55744) 2024-03-29T11:36:17.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.408+0000 7f2abb400700 1 -- 172.21.15.12:0/1094316893 >> v1:172.21.15.159:6789/0 conn(0x7f2abc15d7f0 legacy=0x7f2abc0b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.408+0000 7f2abb400700 1 -- 172.21.15.12:0/1094316893 >> v1:172.21.15.12:6789/0 conn(0x7f2abc0b56e0 legacy=0x7f2abc06ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.408+0000 7f2abb400700 1 -- 172.21.15.12:0/1094316893 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc0cdf60 con 0x7f2abc0d1770 2024-03-29T11:36:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.408+0000 7f2ab9600700 1 -- 172.21.15.12:0/1094316893 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708181571 0 0) 0x7f2abc0ceb70 con 0x7f2abc0b56e0 2024-03-29T11:36:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.408+0000 7f2abb400700 1 --2- 172.21.15.12:0/1094316893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2abc0d1770 0x7f2abc0d25d0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009e50 tx=0x7f2ab400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5206117cd48c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.408+0000 7f2ab9600700 1 -- 172.21.15.12:0/1094316893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab400e070 con 0x7f2abc0d1770 2024-03-29T11:36:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.408+0000 7f2ab9600700 1 -- 172.21.15.12:0/1094316893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab400b9c0 con 0x7f2abc0d1770 2024-03-29T11:36:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.408+0000 7f2ab9600700 1 -- 172.21.15.12:0/1094316893 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab400bd70 con 0x7f2abc0d1770 2024-03-29T11:36:17.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.409+0000 7f2ac1c00700 1 -- 172.21.15.12:0/1094316893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2abc0d1770 msgr2=0x7f2abc0d25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.409+0000 7f2ac1c00700 1 --2- 172.21.15.12:0/1094316893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2abc0d1770 0x7f2abc0d25d0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009e50 tx=0x7f2ab400b330 comp rx=0 tx=0).stop 2024-03-29T11:36:17.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.409+0000 7f2ac1c00700 1 -- 172.21.15.12:0/1094316893 shutdown_connections 2024-03-29T11:36:17.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.409+0000 7f2ac1c00700 1 --2- 172.21.15.12:0/1094316893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2abc0d1770 0x7f2abc0d25d0 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.409+0000 7f2ac1c00700 1 -- 172.21.15.12:0/1094316893 >> 172.21.15.12:0/1094316893 conn(0x7f2abc08aff0 msgr2=0x7f2abc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:17.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.409+0000 7f2ac1c00700 1 -- 172.21.15.12:0/1094316893 shutdown_connections 2024-03-29T11:36:17.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.409+0000 7f2ac1c00700 1 -- 172.21.15.12:0/1094316893 wait complete. 2024-03-29T11:36:17.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.411+0000 7f2ac1c00700 1 Processor -- start 2024-03-29T11:36:17.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.411+0000 7f2ac1c00700 1 -- start start 2024-03-29T11:36:17.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2ac1c00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0b56e0 0x7f2abc152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2ac1c00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0d1770 0x7f2abc15c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2ac1c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2abc15d7f0 0x7f2abc155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2ac1c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc1604f0 con 0x7f2abc15d7f0 2024-03-29T11:36:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2ac1c00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc0b8860 con 0x7f2abc0d1770 2024-03-29T11:36:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2ac1c00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc15f8b0 con 0x7f2abc0b56e0 2024-03-29T11:36:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2abb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2abc15d7f0 0x7f2abc155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2abb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2abc15d7f0 0x7f2abc155cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33698/0 (socket says 172.21.15.12:33698) 2024-03-29T11:36:17.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2abb400700 1 -- 172.21.15.12:0/3689988523 learned_addr learned my addr 172.21.15.12:0/3689988523 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2abaa00700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0b56e0 0x7f2abc152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2aba000700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0d1770 0x7f2abc15c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2abb400700 1 -- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0b56e0 msgr2=0x7f2abc152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2abb400700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0b56e0 0x7f2abc152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2abb400700 1 -- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0d1770 msgr2=0x7f2abc15c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2abb400700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0d1770 0x7f2abc15c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2abb400700 1 -- 172.21.15.12:0/3689988523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4009b30 con 0x7f2abc15d7f0 2024-03-29T11:36:17.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.412+0000 7f2aba000700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0d1770 0x7f2abc15c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.413+0000 7f2abb400700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2abc15d7f0 0x7f2abc155cd0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4008dd0 tx=0x7f2ab4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.413+0000 7f2ab2a00700 1 -- 172.21.15.12:0/3689988523 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab401b630 con 0x7f2abc15d7f0 2024-03-29T11:36:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.413+0000 7f2ab2a00700 1 -- 172.21.15.12:0/3689988523 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab401b790 con 0x7f2abc15d7f0 2024-03-29T11:36:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.413+0000 7f2ab2a00700 1 -- 172.21.15.12:0/3689988523 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab401bb40 con 0x7f2abc15d7f0 2024-03-29T11:36:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.413+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2abc15fb50 con 0x7f2abc15d7f0 2024-03-29T11:36:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.413+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2abc160110 con 0x7f2abc15d7f0 2024-03-29T11:36:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.414+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2abc0ce800 con 0x7f2abc15d7f0 2024-03-29T11:36:17.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.415+0000 7f2ab2a00700 1 -- 172.21.15.12:0/3689988523 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2ab401bca0 con 0x7f2abc15d7f0 2024-03-29T11:36:17.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.416+0000 7f2ab2a00700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2a9c06adb0 0x7f2a9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.416+0000 7f2ab2a00700 1 -- 172.21.15.12:0/3689988523 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f2ab408afc0 con 0x7f2abc15d7f0 2024-03-29T11:36:17.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.416+0000 7f2ab2a00700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] conn(0x7f2a9c0712c0 0x7f2a9c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.416+0000 7f2ab2a00700 1 -- 172.21.15.12:0/3689988523 --> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a9c073db0 con 0x7f2a9c0712c0 2024-03-29T11:36:17.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.416+0000 7f2abaa00700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2a9c06adb0 0x7f2a9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.417+0000 7f2aba000700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] conn(0x7f2a9c0712c0 0x7f2a9c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.417+0000 7f2abaa00700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2a9c06adb0 0x7f2a9c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2aac000f60 tx=0x7f2aac009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:17.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.417+0000 7f2aba000700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] conn(0x7f2a9c0712c0 0x7f2a9c0736e0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:17.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.418+0000 7f2ab2a00700 1 -- 172.21.15.12:0/3689988523 <== osd.5 v2:172.21.15.159:6816/44148 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2a9c073db0 con 0x7f2a9c0712c0 2024-03-29T11:36:17.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.420+0000 7f2ab2a00700 1 -- 172.21.15.12:0/3689988523 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ab40099b0 con 0x7f2abc15d7f0 2024-03-29T11:36:17.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.434+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 --> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2abc0ce520 con 0x7f2a9c0712c0 2024-03-29T11:36:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.434+0000 7f2ab2a00700 1 -- 172.21.15.12:0/3689988523 <== osd.5 v2:172.21.15.159:6816/44148 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2aa8007130 con 0x7f2a9c0712c0 2024-03-29T11:36:17.433 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:36:17.434 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T11:36:17.434 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:36:17.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] conn(0x7f2a9c0712c0 msgr2=0x7f2a9c0736e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2ac1c00700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:6816/44148,v1:172.21.15.159:6818/44148] conn(0x7f2a9c0712c0 0x7f2a9c0736e0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2a9c06adb0 msgr2=0x7f2a9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2ac1c00700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f2a9c06adb0 0x7f2a9c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2aac000f60 tx=0x7f2aac009040 comp rx=0 tx=0).stop 2024-03-29T11:36:17.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2abc15d7f0 msgr2=0x7f2abc155cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2ac1c00700 1 --2- 172.21.15.12:0/3689988523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2abc15d7f0 0x7f2abc155cd0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4008dd0 tx=0x7f2ab4004ad0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2abb400700 1 -- 172.21.15.12:0/3689988523 reap_dead start 2024-03-29T11:36:17.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 shutdown_connections 2024-03-29T11:36:17.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 >> 172.21.15.12:0/3689988523 conn(0x7f2abc08aff0 msgr2=0x7f2abc0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:17.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 shutdown_connections 2024-03-29T11:36:17.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.435+0000 7f2ac1c00700 1 -- 172.21.15.12:0/3689988523 wait complete. 2024-03-29T11:36:17.451 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-29T11:36:17.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.651+0000 7f96e2200700 1 Processor -- start 2024-03-29T11:36:17.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.651+0000 7f96e2200700 1 -- start start 2024-03-29T11:36:17.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.652+0000 7f96e2200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96dc15d9d0 0x7f96dc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.652+0000 7f96e2200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96dc0d2540 0x7f96dc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.652+0000 7f96e2200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96dc161420 con 0x7f96dc0d16a0 2024-03-29T11:36:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.652+0000 7f96e2200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96dc15edc0 con 0x7f96dc15d9d0 2024-03-29T11:36:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.652+0000 7f96e2200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc161700 con 0x7f96dc0d2540 2024-03-29T11:36:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.652+0000 7f96db400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96dc15d9d0 0x7f96dc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.652+0000 7f96daa00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f96dc0d16a0 0x7f96dc0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55752/0 (socket says 172.21.15.12:55752) 2024-03-29T11:36:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.652+0000 7f96db400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96dc15d9d0 0x7f96dc0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33710/0 (socket says 172.21.15.12:33710) 2024-03-29T11:36:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.652+0000 7f96daa00700 1 -- 172.21.15.12:0/3342764461 learned_addr learned my addr 172.21.15.12:0/3342764461 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.652+0000 7f96dbe00700 1 --2- 172.21.15.12:0/3342764461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96dc0d2540 0x7f96dc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.653+0000 7f96db400700 1 -- 172.21.15.12:0/3342764461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96dc0d2540 msgr2=0x7f96dc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:17.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.653+0000 7f96db400700 1 --2- 172.21.15.12:0/3342764461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96dc0d2540 0x7f96dc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.653+0000 7f96db400700 1 -- 172.21.15.12:0/3342764461 >> v1:172.21.15.12:6789/0 conn(0x7f96dc0d16a0 legacy=0x7f96dc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.653+0000 7f96db400700 1 -- 172.21.15.12:0/3342764461 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc1619e0 con 0x7f96dc15d9d0 2024-03-29T11:36:17.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.653+0000 7f96da000700 1 -- 172.21.15.12:0/3342764461 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996486396 0 0) 0x7f96dc161420 con 0x7f96dc0d16a0 2024-03-29T11:36:17.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.653+0000 7f96db400700 1 --2- 172.21.15.12:0/3342764461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 0x7f96dc0d3c30 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009840 tx=0x7f96cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e5e78247ed68ae server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:17.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.653+0000 7f96da000700 1 -- 172.21.15.12:0/3342764461 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc011830 con 0x7f96dc15d9d0 2024-03-29T11:36:17.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.653+0000 7f96da000700 1 -- 172.21.15.12:0/3342764461 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96cc011990 con 0x7f96dc15d9d0 2024-03-29T11:36:17.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.653+0000 7f96da000700 1 -- 172.21.15.12:0/3342764461 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc011d20 con 0x7f96dc15d9d0 2024-03-29T11:36:17.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.654+0000 7f96e2200700 1 -- 172.21.15.12:0/3342764461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 msgr2=0x7f96dc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.654+0000 7f96e2200700 1 --2- 172.21.15.12:0/3342764461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 0x7f96dc0d3c30 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009840 tx=0x7f96cc009b50 comp rx=0 tx=0).stop 2024-03-29T11:36:17.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.654+0000 7f96e2200700 1 -- 172.21.15.12:0/3342764461 shutdown_connections 2024-03-29T11:36:17.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.654+0000 7f96e2200700 1 --2- 172.21.15.12:0/3342764461 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96dc0d2540 0x7f96dc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.654+0000 7f96e2200700 1 --2- 172.21.15.12:0/3342764461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 0x7f96dc0d3c30 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.654+0000 7f96e2200700 1 -- 172.21.15.12:0/3342764461 >> 172.21.15.12:0/3342764461 conn(0x7f96dc08aff0 msgr2=0x7f96dc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:17.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.654+0000 7f96e2200700 1 -- 172.21.15.12:0/3342764461 shutdown_connections 2024-03-29T11:36:17.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.654+0000 7f96e2200700 1 -- 172.21.15.12:0/3342764461 wait complete. 2024-03-29T11:36:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.656+0000 7f96e2200700 1 Processor -- start 2024-03-29T11:36:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.656+0000 7f96e2200700 1 -- start start 2024-03-29T11:36:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96e2200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96dc0d16a0 0x7f96dc0b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96e2200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96dc0d2540 0x7f96dc1e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96e2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 0x7f96dc14f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96e2200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc1e9080 con 0x7f96dc15d9d0 2024-03-29T11:36:17.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96e2200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc1e81d0 con 0x7f96dc0d16a0 2024-03-29T11:36:17.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96e2200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc1e8470 con 0x7f96dc0d2540 2024-03-29T11:36:17.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96dbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 0x7f96dc14f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96dbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 0x7f96dc14f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33716/0 (socket says 172.21.15.12:33716) 2024-03-29T11:36:17.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96dbe00700 1 -- 172.21.15.12:0/1188826109 learned_addr learned my addr 172.21.15.12:0/1188826109 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:17.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96db400700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96dc0d16a0 0x7f96dc0b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.657+0000 7f96daa00700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96dc0d2540 0x7f96dc1e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96dbe00700 1 -- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96dc0d2540 msgr2=0x7f96dc1e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96dbe00700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96dc0d2540 0x7f96dc1e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96dbe00700 1 -- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96dc0d16a0 msgr2=0x7f96dc0b8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96dbe00700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96dc0d16a0 0x7f96dc0b8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96dbe00700 1 -- 172.21.15.12:0/1188826109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc0094a0 con 0x7f96dc15d9d0 2024-03-29T11:36:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96dbe00700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 0x7f96dc14f910 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f96d400efc0 tx=0x7f96d400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96d3400700 1 -- 172.21.15.12:0/1188826109 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4009ea0 con 0x7f96dc15d9d0 2024-03-29T11:36:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96d3400700 1 -- 172.21.15.12:0/1188826109 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d400f040 con 0x7f96dc15d9d0 2024-03-29T11:36:17.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96e2200700 1 -- 172.21.15.12:0/1188826109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96dc1e8750 con 0x7f96dc15d9d0 2024-03-29T11:36:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96d3400700 1 -- 172.21.15.12:0/1188826109 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4014480 con 0x7f96dc15d9d0 2024-03-29T11:36:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.658+0000 7f96e2200700 1 -- 172.21.15.12:0/1188826109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96dc1e8d10 con 0x7f96dc15d9d0 2024-03-29T11:36:17.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.659+0000 7f96e2200700 1 -- 172.21.15.12:0/1188826109 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96dc1e91f0 con 0x7f96dc15d9d0 2024-03-29T11:36:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.660+0000 7f96d3400700 1 -- 172.21.15.12:0/1188826109 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f96d40145e0 con 0x7f96dc15d9d0 2024-03-29T11:36:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.661+0000 7f96d3400700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f96bc06adb0 0x7f96bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.661+0000 7f96d3400700 1 -- 172.21.15.12:0/1188826109 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f96d408a190 con 0x7f96dc15d9d0 2024-03-29T11:36:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.662+0000 7f96db400700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f96bc06adb0 0x7f96bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.662+0000 7f96db400700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f96bc06adb0 0x7f96bc06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009fd0 tx=0x7f96cc01c040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:17.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.666+0000 7f96d3400700 1 -- 172.21.15.12:0/1188826109 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f96d4055110 con 0x7f96dc15d9d0 2024-03-29T11:36:17.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.883+0000 7f96e2200700 1 -- 172.21.15.12:0/1188826109 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f96dc161fa0 con 0x7f96bc06adb0 2024-03-29T11:36:17.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.884+0000 7f96d3400700 1 -- 172.21.15.12:0/1188826109 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f96cc011b80 con 0x7f96bc06adb0 2024-03-29T11:36:17.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 -- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f96bc06adb0 msgr2=0x7f96bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f96bc06adb0 0x7f96bc06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009fd0 tx=0x7f96cc01c040 comp rx=0 tx=0).stop 2024-03-29T11:36:17.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 -- 172.21.15.12:0/1188826109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 msgr2=0x7f96dc14f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 0x7f96dc14f910 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f96d400efc0 tx=0x7f96d400c3a0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 -- 172.21.15.12:0/1188826109 shutdown_connections 2024-03-29T11:36:17.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f96bc06adb0 0x7f96bc06d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96dc15d9d0 0x7f96dc14f910 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96dc0d16a0 0x7f96dc0b8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 --2- 172.21.15.12:0/1188826109 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96dc0d2540 0x7f96dc1e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:17.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 -- 172.21.15.12:0/1188826109 >> 172.21.15.12:0/1188826109 conn(0x7f96dc08aff0 msgr2=0x7f96dc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:17.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 -- 172.21.15.12:0/1188826109 shutdown_connections 2024-03-29T11:36:17.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:17.887+0000 7f96e2200700 1 -- 172.21.15.12:0/1188826109 wait complete. 2024-03-29T11:36:17.888 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 5 to deep-scrub 2024-03-29T11:36:17.904 INFO:tasks.ceph:Scrubbing osd.6 2024-03-29T11:36:17.904 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-29T11:36:18.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.107+0000 7f22db400700 1 Processor -- start 2024-03-29T11:36:18.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.107+0000 7f22db400700 1 -- start start 2024-03-29T11:36:18.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.108+0000 7f22db400700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f22d4162030 0x7f22d40d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.108+0000 7f22db400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22d40d20d0 0x7f22d40ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22db400700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d40d2520 con 0x7f22d415d930 2024-03-29T11:36:18.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22db400700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d415ec40 con 0x7f22d40d20d0 2024-03-29T11:36:18.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22db400700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d415dd10 con 0x7f22d4162030 2024-03-29T11:36:18.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22d9400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22d40d20d0 0x7f22d40ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22d9400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22d40d20d0 0x7f22d40ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33722/0 (socket says 172.21.15.12:33722) 2024-03-29T11:36:18.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22d9400700 1 -- 172.21.15.12:0/2684069439 learned_addr learned my addr 172.21.15.12:0/2684069439 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:18.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22d3e00700 1 --2- 172.21.15.12:0/2684069439 >> v2:172.21.15.159:3300/0 conn(0x7f22d4162030 0x7f22d40d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22d9400700 1 -- 172.21.15.12:0/2684069439 >> v2:172.21.15.159:3300/0 conn(0x7f22d4162030 msgr2=0x7f22d40d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22d9400700 1 --2- 172.21.15.12:0/2684069439 >> v2:172.21.15.159:3300/0 conn(0x7f22d4162030 0x7f22d40d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22d9400700 1 -- 172.21.15.12:0/2684069439 >> v1:172.21.15.159:6789/0 conn(0x7f22d415d930 legacy=0x7f22d40d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.109+0000 7f22d9400700 1 -- 172.21.15.12:0/2684069439 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d415dfc0 con 0x7f22d40d20d0 2024-03-29T11:36:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.110+0000 7f22d9400700 1 --2- 172.21.15.12:0/2684069439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22d40d20d0 0x7f22d40ce300 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f22d40d3510 tx=0x7f22cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3fbd00d6243d236 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.110+0000 7f22d3400700 1 -- 172.21.15.12:0/2684069439 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc00e070 con 0x7f22d40d20d0 2024-03-29T11:36:18.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.110+0000 7f22d3400700 1 -- 172.21.15.12:0/2684069439 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22cc003a20 con 0x7f22d40d20d0 2024-03-29T11:36:18.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.110+0000 7f22d3400700 1 -- 172.21.15.12:0/2684069439 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc003dd0 con 0x7f22d40d20d0 2024-03-29T11:36:18.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.110+0000 7f22db400700 1 -- 172.21.15.12:0/2684069439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22d40d20d0 msgr2=0x7f22d40ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.110+0000 7f22db400700 1 --2- 172.21.15.12:0/2684069439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22d40d20d0 0x7f22d40ce300 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f22d40d3510 tx=0x7f22cc00b330 comp rx=0 tx=0).stop 2024-03-29T11:36:18.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.111+0000 7f22db400700 1 -- 172.21.15.12:0/2684069439 shutdown_connections 2024-03-29T11:36:18.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.111+0000 7f22db400700 1 --2- 172.21.15.12:0/2684069439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22d40d20d0 0x7f22d40ce300 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.111+0000 7f22db400700 1 --2- 172.21.15.12:0/2684069439 >> v2:172.21.15.159:3300/0 conn(0x7f22d4162030 0x7f22d40d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.111+0000 7f22db400700 1 -- 172.21.15.12:0/2684069439 >> 172.21.15.12:0/2684069439 conn(0x7f22d408aff0 msgr2=0x7f22d40d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:18.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.111+0000 7f22db400700 1 -- 172.21.15.12:0/2684069439 shutdown_connections 2024-03-29T11:36:18.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.111+0000 7f22db400700 1 -- 172.21.15.12:0/2684069439 wait complete. 2024-03-29T11:36:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.112+0000 7f22db400700 1 Processor -- start 2024-03-29T11:36:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.112+0000 7f22db400700 1 -- start start 2024-03-29T11:36:18.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.113+0000 7f22db400700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f22d40d20d0 0x7f22d4159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.113+0000 7f22db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22d415d930 0x7f22d40b6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.113+0000 7f22db400700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22d4162030 0x7f22d40b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.113+0000 7f22db400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d40b71e0 con 0x7f22d415d930 2024-03-29T11:36:18.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.113+0000 7f22db400700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d40b2d50 con 0x7f22d40d20d0 2024-03-29T11:36:18.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.113+0000 7f22db400700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d41594f0 con 0x7f22d4162030 2024-03-29T11:36:18.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22d415d930 0x7f22d40b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22d415d930 0x7f22d40b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33726/0 (socket says 172.21.15.12:33726) 2024-03-29T11:36:18.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d3e00700 1 -- 172.21.15.12:0/4161347738 learned_addr learned my addr 172.21.15.12:0/4161347738 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:18.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d8a00700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f22d40d20d0 0x7f22d4159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d9400700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22d4162030 0x7f22d40b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d3e00700 1 -- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22d4162030 msgr2=0x7f22d40b6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d3e00700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f22d4162030 0x7f22d40b6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d3e00700 1 -- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f22d40d20d0 msgr2=0x7f22d4159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d3e00700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f22d40d20d0 0x7f22d4159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d3e00700 1 -- 172.21.15.12:0/4161347738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc009b30 con 0x7f22d415d930 2024-03-29T11:36:18.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.114+0000 7f22d3e00700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22d415d930 0x7f22d40b6670 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f22c000d800 tx=0x7f22c000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.115+0000 7f22d0c00700 1 -- 172.21.15.12:0/4161347738 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c00125f0 con 0x7f22d415d930 2024-03-29T11:36:18.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.115+0000 7f22d0c00700 1 -- 172.21.15.12:0/4161347738 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c0012750 con 0x7f22d415d930 2024-03-29T11:36:18.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.115+0000 7f22d0c00700 1 -- 172.21.15.12:0/4161347738 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0012b00 con 0x7f22d415d930 2024-03-29T11:36:18.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.115+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d40b2f60 con 0x7f22d415d930 2024-03-29T11:36:18.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.115+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d40b3520 con 0x7f22d415d930 2024-03-29T11:36:18.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.115+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22d415e860 con 0x7f22d415d930 2024-03-29T11:36:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.117+0000 7f22d0c00700 1 -- 172.21.15.12:0/4161347738 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f22c0012c60 con 0x7f22d415d930 2024-03-29T11:36:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.118+0000 7f22d0c00700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f22b406adb0 0x7f22b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.118+0000 7f22d0c00700 1 -- 172.21.15.12:0/4161347738 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f22c008a2c0 con 0x7f22d415d930 2024-03-29T11:36:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.118+0000 7f22d0c00700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] conn(0x7f22b40712c0 0x7f22b40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.118+0000 7f22d0c00700 1 -- 172.21.15.12:0/4161347738 --> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22b4073db0 con 0x7f22b40712c0 2024-03-29T11:36:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.118+0000 7f22d0c00700 1 -- 172.21.15.12:0/4161347738 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22c008a6e0 con 0x7f22d415d930 2024-03-29T11:36:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.118+0000 7f22d8a00700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f22b406adb0 0x7f22b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.118+0000 7f22d9400700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] conn(0x7f22b40712c0 0x7f22b40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.119+0000 7f22d8a00700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f22b406adb0 0x7f22b406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f22d41f0d50 tx=0x7f22c8009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.119+0000 7f22d9400700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] conn(0x7f22b40712c0 0x7f22b40736e0 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.120+0000 7f22d0c00700 1 -- 172.21.15.12:0/4161347738 <== osd.6 v2:172.21.15.159:6817/44159 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f22b4073db0 con 0x7f22b40712c0 2024-03-29T11:36:18.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.135+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 --> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f22d415e580 con 0x7f22b40712c0 2024-03-29T11:36:18.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.136+0000 7f22d0c00700 1 -- 172.21.15.12:0/4161347738 <== osd.6 v2:172.21.15.159:6817/44159 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f22cc003e30 con 0x7f22b40712c0 2024-03-29T11:36:18.135 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:36:18.135 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T11:36:18.136 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:36:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.136+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] conn(0x7f22b40712c0 msgr2=0x7f22b40736e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.136+0000 7f22db400700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:6817/44159,v1:172.21.15.159:6819/44159] conn(0x7f22b40712c0 0x7f22b40736e0 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.137+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f22b406adb0 msgr2=0x7f22b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.137+0000 7f22db400700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f22b406adb0 0x7f22b406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f22d41f0d50 tx=0x7f22c8009040 comp rx=0 tx=0).stop 2024-03-29T11:36:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.137+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22d415d930 msgr2=0x7f22d40b6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.137+0000 7f22db400700 1 --2- 172.21.15.12:0/4161347738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22d415d930 0x7f22d40b6670 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f22c000d800 tx=0x7f22c000dbc0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.137+0000 7f22d9400700 1 -- 172.21.15.12:0/4161347738 reap_dead start 2024-03-29T11:36:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.137+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 shutdown_connections 2024-03-29T11:36:18.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.137+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 >> 172.21.15.12:0/4161347738 conn(0x7f22d408aff0 msgr2=0x7f22d406d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:18.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.137+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 shutdown_connections 2024-03-29T11:36:18.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.137+0000 7f22db400700 1 -- 172.21.15.12:0/4161347738 wait complete. 2024-03-29T11:36:18.153 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-29T11:36:18.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.353+0000 7f2021800700 1 Processor -- start 2024-03-29T11:36:18.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.353+0000 7f2021800700 1 -- start start 2024-03-29T11:36:18.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.354+0000 7f2021800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f201c15d800 0x7f201c0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.354+0000 7f2021800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f201c0ce6a0 0x7f201c15dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.354+0000 7f2021800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201c0709c0 con 0x7f201c0ce230 2024-03-29T11:36:18.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.354+0000 7f2021800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201c0b40f0 con 0x7f201c15d800 2024-03-29T11:36:18.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.354+0000 7f2021800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201c070c80 con 0x7f201c0ce6a0 2024-03-29T11:36:18.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.354+0000 7f201aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f201c15d800 0x7f201c0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f201aa00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f201c15d800 0x7f201c0725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33742/0 (socket says 172.21.15.12:33742) 2024-03-29T11:36:18.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f201aa00700 1 -- 172.21.15.12:0/530317957 learned_addr learned my addr 172.21.15.12:0/530317957 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:18.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f201b400700 1 --2- 172.21.15.12:0/530317957 >> v2:172.21.15.159:3300/0 conn(0x7f201c0ce6a0 0x7f201c15dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f201aa00700 1 -- 172.21.15.12:0/530317957 >> v2:172.21.15.159:3300/0 conn(0x7f201c0ce6a0 msgr2=0x7f201c15dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:18.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f201aa00700 1 --2- 172.21.15.12:0/530317957 >> v2:172.21.15.159:3300/0 conn(0x7f201c0ce6a0 0x7f201c15dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f201aa00700 1 -- 172.21.15.12:0/530317957 >> v1:172.21.15.12:6789/0 conn(0x7f201c0ce230 legacy=0x7f201c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f201aa00700 1 -- 172.21.15.12:0/530317957 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c070f60 con 0x7f201c15d800 2024-03-29T11:36:18.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f201b400700 1 --2- 172.21.15.12:0/530317957 >> v2:172.21.15.159:3300/0 conn(0x7f201c0ce6a0 0x7f201c15dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:18.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f2019600700 1 -- 172.21.15.12:0/530317957 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862463530 0 0) 0x7f201c0709c0 con 0x7f201c0ce230 2024-03-29T11:36:18.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f201aa00700 1 --2- 172.21.15.12:0/530317957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c15d800 0x7f201c0725c0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f201c06b6d0 tx=0x7f200c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6485c743879a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.355+0000 7f2019600700 1 -- 172.21.15.12:0/530317957 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c00b870 con 0x7f201c15d800 2024-03-29T11:36:18.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.356+0000 7f2019600700 1 -- 172.21.15.12:0/530317957 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200c00b9d0 con 0x7f201c15d800 2024-03-29T11:36:18.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.356+0000 7f2019600700 1 -- 172.21.15.12:0/530317957 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c00bd60 con 0x7f201c15d800 2024-03-29T11:36:18.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.356+0000 7f2021800700 1 -- 172.21.15.12:0/530317957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c15d800 msgr2=0x7f201c0725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.356+0000 7f2021800700 1 --2- 172.21.15.12:0/530317957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c15d800 0x7f201c0725c0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f201c06b6d0 tx=0x7f200c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.356+0000 7f2021800700 1 -- 172.21.15.12:0/530317957 shutdown_connections 2024-03-29T11:36:18.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.356+0000 7f2021800700 1 --2- 172.21.15.12:0/530317957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c15d800 0x7f201c0725c0 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.356+0000 7f2021800700 1 --2- 172.21.15.12:0/530317957 >> v2:172.21.15.159:3300/0 conn(0x7f201c0ce6a0 0x7f201c15dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.356+0000 7f2021800700 1 -- 172.21.15.12:0/530317957 >> 172.21.15.12:0/530317957 conn(0x7f201c0b0d70 msgr2=0x7f201c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:18.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.356+0000 7f2021800700 1 -- 172.21.15.12:0/530317957 shutdown_connections 2024-03-29T11:36:18.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.356+0000 7f2021800700 1 -- 172.21.15.12:0/530317957 wait complete. 2024-03-29T11:36:18.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.358+0000 7f2021800700 1 Processor -- start 2024-03-29T11:36:18.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.358+0000 7f2021800700 1 -- start start 2024-03-29T11:36:18.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.359+0000 7f2021800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f201c0ce230 0x7f201c0cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.359+0000 7f2021800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c0ce6a0 0x7f201c152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.359+0000 7f2021800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f201c15d800 0x7f201c146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.359+0000 7f2021800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201c0caa10 con 0x7f201c0ce6a0 2024-03-29T11:36:18.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.359+0000 7f2021800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201c15c6d0 con 0x7f201c0ce230 2024-03-29T11:36:18.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.359+0000 7f2021800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201c152f90 con 0x7f201c15d800 2024-03-29T11:36:18.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.359+0000 7f201a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c0ce6a0 0x7f201c152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.359+0000 7f201a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c0ce6a0 0x7f201c152890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33746/0 (socket says 172.21.15.12:33746) 2024-03-29T11:36:18.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.359+0000 7f201a000700 1 -- 172.21.15.12:0/496422735 learned_addr learned my addr 172.21.15.12:0/496422735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:18.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f201aa00700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f201c0ce230 0x7f201c0cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f201b400700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f201c15d800 0x7f201c146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f201a000700 1 -- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f201c15d800 msgr2=0x7f201c146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f201a000700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f201c15d800 0x7f201c146850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f201a000700 1 -- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f201c0ce230 msgr2=0x7f201c0cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f201a000700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f201c0ce230 0x7f201c0cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f201a000700 1 -- 172.21.15.12:0/496422735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200c009550 con 0x7f201c0ce6a0 2024-03-29T11:36:18.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f201a000700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c0ce6a0 0x7f201c152890 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f200800c550 tx=0x7f200800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f2012a00700 1 -- 172.21.15.12:0/496422735 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20080054e0 con 0x7f201c0ce6a0 2024-03-29T11:36:18.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f2012a00700 1 -- 172.21.15.12:0/496422735 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2008003680 con 0x7f201c0ce6a0 2024-03-29T11:36:18.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f2021800700 1 -- 172.21.15.12:0/496422735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201c153270 con 0x7f201c0ce6a0 2024-03-29T11:36:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.360+0000 7f2012a00700 1 -- 172.21.15.12:0/496422735 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2008003a30 con 0x7f201c0ce6a0 2024-03-29T11:36:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.361+0000 7f2021800700 1 -- 172.21.15.12:0/496422735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201c15cc00 con 0x7f201c0ce6a0 2024-03-29T11:36:18.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.362+0000 7f2021800700 1 -- 172.21.15.12:0/496422735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f201c071840 con 0x7f201c0ce6a0 2024-03-29T11:36:18.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.362+0000 7f2012a00700 1 -- 172.21.15.12:0/496422735 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2008003b90 con 0x7f201c0ce6a0 2024-03-29T11:36:18.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.363+0000 7f2012a00700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ffc06adb0 0x7f1ffc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.363+0000 7f2012a00700 1 -- 172.21.15.12:0/496422735 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f2008088750 con 0x7f201c0ce6a0 2024-03-29T11:36:18.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.364+0000 7f201aa00700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ffc06adb0 0x7f1ffc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.364+0000 7f201aa00700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ffc06adb0 0x7f1ffc06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f200c009c10 tx=0x7f200c008030 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.368+0000 7f2012a00700 1 -- 172.21.15.12:0/496422735 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2008053880 con 0x7f201c0ce6a0 2024-03-29T11:36:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.585+0000 7f2021800700 1 -- 172.21.15.12:0/496422735 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f201c072b00 con 0x7f1ffc06adb0 2024-03-29T11:36:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.585+0000 7f2012a00700 1 -- 172.21.15.12:0/496422735 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f200c00bb40 con 0x7f1ffc06adb0 2024-03-29T11:36:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.587+0000 7f2021800700 1 -- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ffc06adb0 msgr2=0x7f1ffc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.587+0000 7f2021800700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ffc06adb0 0x7f1ffc06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f200c009c10 tx=0x7f200c008030 comp rx=0 tx=0).stop 2024-03-29T11:36:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.587+0000 7f2021800700 1 -- 172.21.15.12:0/496422735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c0ce6a0 msgr2=0x7f201c152890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.587+0000 7f2021800700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c0ce6a0 0x7f201c152890 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f200800c550 tx=0x7f200800c860 comp rx=0 tx=0).stop 2024-03-29T11:36:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.588+0000 7f2021800700 1 -- 172.21.15.12:0/496422735 shutdown_connections 2024-03-29T11:36:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.588+0000 7f2021800700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f1ffc06adb0 0x7f1ffc06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.588+0000 7f2021800700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f201c15d800 0x7f201c146850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.588+0000 7f2021800700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f201c0ce230 0x7f201c0cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.588+0000 7f2021800700 1 --2- 172.21.15.12:0/496422735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f201c0ce6a0 0x7f201c152890 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.588+0000 7f2021800700 1 -- 172.21.15.12:0/496422735 >> 172.21.15.12:0/496422735 conn(0x7f201c0b0d70 msgr2=0x7f201c0aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.588+0000 7f2021800700 1 -- 172.21.15.12:0/496422735 shutdown_connections 2024-03-29T11:36:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.588+0000 7f2021800700 1 -- 172.21.15.12:0/496422735 wait complete. 2024-03-29T11:36:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 6 to deep-scrub 2024-03-29T11:36:18.604 INFO:tasks.ceph:Scrubbing osd.7 2024-03-29T11:36:18.605 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-29T11:36:18.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.807+0000 7fe854e00700 1 Processor -- start 2024-03-29T11:36:18.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.807+0000 7fe854e00700 1 -- start start 2024-03-29T11:36:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.808+0000 7fe854e00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe8501580c0 0x7fe85006e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.808+0000 7fe854e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe85006e7f0 0x7fe85015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.808+0000 7fe854e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe850070980 con 0x7fe85015b160 2024-03-29T11:36:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.808+0000 7fe854e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe850071570 con 0x7fe85006e7f0 2024-03-29T11:36:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.808+0000 7fe854e00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe850071990 con 0x7fe8501580c0 2024-03-29T11:36:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.809+0000 7fe84d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe85006e7f0 0x7fe85015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.809+0000 7fe84d400700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe85006e7f0 0x7fe85015ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33752/0 (socket says 172.21.15.12:33752) 2024-03-29T11:36:18.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.809+0000 7fe84d400700 1 -- 172.21.15.12:0/1538081638 learned_addr learned my addr 172.21.15.12:0/1538081638 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:18.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.809+0000 7fe84de00700 1 --2- 172.21.15.12:0/1538081638 >> v2:172.21.15.159:3300/0 conn(0x7fe8501580c0 0x7fe85006e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.809+0000 7fe84d400700 1 -- 172.21.15.12:0/1538081638 >> v2:172.21.15.159:3300/0 conn(0x7fe8501580c0 msgr2=0x7fe85006e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:18.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.809+0000 7fe84d400700 1 --2- 172.21.15.12:0/1538081638 >> v2:172.21.15.159:3300/0 conn(0x7fe8501580c0 0x7fe85006e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.809+0000 7fe84d400700 1 -- 172.21.15.12:0/1538081638 >> v1:172.21.15.159:6789/0 conn(0x7fe85015b160 legacy=0x7fe85015df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:18.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.809+0000 7fe84d400700 1 -- 172.21.15.12:0/1538081638 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe850070bd0 con 0x7fe85006e7f0 2024-03-29T11:36:18.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.809+0000 7fe84de00700 1 --2- 172.21.15.12:0/1538081638 >> v2:172.21.15.159:3300/0 conn(0x7fe8501580c0 0x7fe85006e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:36:18.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.809+0000 7fe84d400700 1 --2- 172.21.15.12:0/1538081638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe85006e7f0 0x7fe85015ac20 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7fe85006c750 tx=0x7fe83c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e79509c4b5181ffb server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe84ca00700 1 -- 172.21.15.12:0/1538081638 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c004230 con 0x7fe85006e7f0 2024-03-29T11:36:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe84ca00700 1 -- 172.21.15.12:0/1538081638 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83c004390 con 0x7fe85006e7f0 2024-03-29T11:36:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe84ca00700 1 -- 172.21.15.12:0/1538081638 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c0036a0 con 0x7fe85006e7f0 2024-03-29T11:36:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe854e00700 1 -- 172.21.15.12:0/1538081638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe85006e7f0 msgr2=0x7fe85015ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe854e00700 1 --2- 172.21.15.12:0/1538081638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe85006e7f0 0x7fe85015ac20 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7fe85006c750 tx=0x7fe83c009bd0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe854e00700 1 -- 172.21.15.12:0/1538081638 shutdown_connections 2024-03-29T11:36:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe854e00700 1 --2- 172.21.15.12:0/1538081638 >> v2:172.21.15.159:3300/0 conn(0x7fe8501580c0 0x7fe85006e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe854e00700 1 --2- 172.21.15.12:0/1538081638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe85006e7f0 0x7fe85015ac20 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe854e00700 1 -- 172.21.15.12:0/1538081638 >> 172.21.15.12:0/1538081638 conn(0x7fe8500cf210 msgr2=0x7fe8500cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe854e00700 1 -- 172.21.15.12:0/1538081638 shutdown_connections 2024-03-29T11:36:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.810+0000 7fe854e00700 1 -- 172.21.15.12:0/1538081638 wait complete. 2024-03-29T11:36:18.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.812+0000 7fe854e00700 1 Processor -- start 2024-03-29T11:36:18.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.812+0000 7fe854e00700 1 -- start start 2024-03-29T11:36:18.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.813+0000 7fe854e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe85006e7f0 0x7fe85015bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.813+0000 7fe854e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8501580c0 0x7fe85014d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.813+0000 7fe854e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe85015b160 0x7fe85014d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.813+0000 7fe854e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85015a9c0 con 0x7fe85006e7f0 2024-03-29T11:36:18.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.813+0000 7fe854e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8500cb940 con 0x7fe85015b160 2024-03-29T11:36:18.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.813+0000 7fe854e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe850159e40 con 0x7fe8501580c0 2024-03-29T11:36:18.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.813+0000 7fe84de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe85006e7f0 0x7fe85015bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.813+0000 7fe84de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe85006e7f0 0x7fe85015bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33768/0 (socket says 172.21.15.12:33768) 2024-03-29T11:36:18.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.813+0000 7fe84de00700 1 -- 172.21.15.12:0/2577271488 learned_addr learned my addr 172.21.15.12:0/2577271488 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:18.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe84d400700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8501580c0 0x7fe85014d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe84e800700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe85015b160 0x7fe85014d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe84de00700 1 -- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8501580c0 msgr2=0x7fe85014d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe84de00700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8501580c0 0x7fe85014d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe84de00700 1 -- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe85015b160 msgr2=0x7fe85014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe84de00700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe85015b160 0x7fe85014d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe84de00700 1 -- 172.21.15.12:0/2577271488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c009550 con 0x7fe85006e7f0 2024-03-29T11:36:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe84d400700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8501580c0 0x7fe85014d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe84e800700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe85015b160 0x7fe85014d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe84de00700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe85006e7f0 0x7fe85015bcf0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7fe84000c750 tx=0x7fe84000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe846000700 1 -- 172.21.15.12:0/2577271488 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840011800 con 0x7fe85006e7f0 2024-03-29T11:36:18.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe846000700 1 -- 172.21.15.12:0/2577271488 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe840011960 con 0x7fe85006e7f0 2024-03-29T11:36:18.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.814+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85015a120 con 0x7fe85006e7f0 2024-03-29T11:36:18.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.815+0000 7fe846000700 1 -- 172.21.15.12:0/2577271488 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840011cd0 con 0x7fe85006e7f0 2024-03-29T11:36:18.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.815+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85015a6e0 con 0x7fe85006e7f0 2024-03-29T11:36:18.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.815+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe850071190 con 0x7fe85006e7f0 2024-03-29T11:36:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.817+0000 7fe846000700 1 -- 172.21.15.12:0/2577271488 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe8400223e0 con 0x7fe85006e7f0 2024-03-29T11:36:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.818+0000 7fe846000700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe83006adb0 0x7fe83006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.818+0000 7fe846000700 1 -- 172.21.15.12:0/2577271488 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fe840089b50 con 0x7fe85006e7f0 2024-03-29T11:36:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.818+0000 7fe846000700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] conn(0x7fe8300712c0 0x7fe8300736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.818+0000 7fe846000700 1 -- 172.21.15.12:0/2577271488 --> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe830073db0 con 0x7fe8300712c0 2024-03-29T11:36:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.818+0000 7fe846000700 1 -- 172.21.15.12:0/2577271488 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe840089fa0 con 0x7fe85006e7f0 2024-03-29T11:36:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.818+0000 7fe84d400700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe83006adb0 0x7fe83006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.819+0000 7fe84e800700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] conn(0x7fe8300712c0 0x7fe8300736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:18.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.819+0000 7fe84d400700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe83006adb0 0x7fe83006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe83c009130 tx=0x7fe83c008030 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.819+0000 7fe84e800700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] conn(0x7fe8300712c0 0x7fe8300736e0 crc :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:18.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.820+0000 7fe846000700 1 -- 172.21.15.12:0/2577271488 <== osd.7 v2:172.21.15.159:6808/44147 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe830073db0 con 0x7fe8300712c0 2024-03-29T11:36:18.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.835+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 --> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe8500cbb00 con 0x7fe8300712c0 2024-03-29T11:36:18.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.836+0000 7fe846000700 1 -- 172.21.15.12:0/2577271488 <== osd.7 v2:172.21.15.159:6808/44147 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe8500cbb00 con 0x7fe8300712c0 2024-03-29T11:36:18.835 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-29T11:36:18.835 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-29T11:36:18.836 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-29T11:36:18.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.836+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] conn(0x7fe8300712c0 msgr2=0x7fe8300736e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.837+0000 7fe854e00700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:6808/44147,v1:172.21.15.159:6809/44147] conn(0x7fe8300712c0 0x7fe8300736e0 crc :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:18.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.837+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe83006adb0 msgr2=0x7fe83006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.837+0000 7fe854e00700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fe83006adb0 0x7fe83006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe83c009130 tx=0x7fe83c008030 comp rx=0 tx=0).stop 2024-03-29T11:36:18.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.837+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe85006e7f0 msgr2=0x7fe85015bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.837+0000 7fe854e00700 1 --2- 172.21.15.12:0/2577271488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe85006e7f0 0x7fe85015bcf0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7fe84000c750 tx=0x7fe84000cb10 comp rx=0 tx=0).stop 2024-03-29T11:36:18.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.837+0000 7fe84e800700 1 -- 172.21.15.12:0/2577271488 reap_dead start 2024-03-29T11:36:18.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.837+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 shutdown_connections 2024-03-29T11:36:18.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.837+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 >> 172.21.15.12:0/2577271488 conn(0x7fe8500cf210 msgr2=0x7fe85015e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:18.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.837+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 shutdown_connections 2024-03-29T11:36:18.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:18.837+0000 7fe854e00700 1 -- 172.21.15.12:0/2577271488 wait complete. 2024-03-29T11:36:18.852 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-29T11:36:19.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.053+0000 7f7ae5800700 1 Processor -- start 2024-03-29T11:36:19.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.053+0000 7f7ae5800700 1 -- start start 2024-03-29T11:36:19.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.054+0000 7f7ae5800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ae00d21f0 0x7f7ae00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.054+0000 7f7ae5800700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7ae00cec40 0x7f7ae0161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.054+0000 7f7ae5800700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae00ce270 con 0x7f7ae01611b0 2024-03-29T11:36:19.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.054+0000 7f7ae5800700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae015b600 con 0x7f7ae00d21f0 2024-03-29T11:36:19.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.054+0000 7f7ae5800700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae00d2760 con 0x7f7ae00cec40 2024-03-29T11:36:19.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.054+0000 7f7ade000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ae00d21f0 0x7f7ae00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.054+0000 7f7ade000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ae00d21f0 0x7f7ae00ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60256/0 (socket says 172.21.15.12:60256) 2024-03-29T11:36:19.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.054+0000 7f7ade000700 1 -- 172.21.15.12:0/2518910420 learned_addr learned my addr 172.21.15.12:0/2518910420 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:19.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.054+0000 7f7adea00700 1 --1- 172.21.15.12:0/2518910420 >> v1:172.21.15.12:6789/0 conn(0x7f7ae01611b0 0x7f7ae015ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49174/0 (socket says 172.21.15.12:49174) 2024-03-29T11:36:19.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.055+0000 7f7adf400700 1 --2- 172.21.15.12:0/2518910420 >> v2:172.21.15.159:3300/0 conn(0x7f7ae00cec40 0x7f7ae0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.055+0000 7f7ade000700 1 -- 172.21.15.12:0/2518910420 >> v2:172.21.15.159:3300/0 conn(0x7f7ae00cec40 msgr2=0x7f7ae0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:19.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.055+0000 7f7ade000700 1 --2- 172.21.15.12:0/2518910420 >> v2:172.21.15.159:3300/0 conn(0x7f7ae00cec40 0x7f7ae0161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.055+0000 7f7ade000700 1 -- 172.21.15.12:0/2518910420 >> v1:172.21.15.12:6789/0 conn(0x7f7ae01611b0 legacy=0x7f7ae015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.055+0000 7f7ade000700 1 -- 172.21.15.12:0/2518910420 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae00d2a40 con 0x7f7ae00d21f0 2024-03-29T11:36:19.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.055+0000 7f7add600700 1 -- 172.21.15.12:0/2518910420 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644226503 0 0) 0x7f7ae00ce270 con 0x7f7ae01611b0 2024-03-29T11:36:19.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.055+0000 7f7ade000700 1 --2- 172.21.15.12:0/2518910420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 0x7f7ae00ce700 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00b5cb0 tx=0x7f7acc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141fb8670c885911 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:19.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.055+0000 7f7add600700 1 -- 172.21.15.12:0/2518910420 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc00b870 con 0x7f7ae00d21f0 2024-03-29T11:36:19.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.055+0000 7f7add600700 1 -- 172.21.15.12:0/2518910420 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7acc00b9d0 con 0x7f7ae00d21f0 2024-03-29T11:36:19.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.055+0000 7f7add600700 1 -- 172.21.15.12:0/2518910420 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc00bd60 con 0x7f7ae00d21f0 2024-03-29T11:36:19.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.056+0000 7f7ae5800700 1 -- 172.21.15.12:0/2518910420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 msgr2=0x7f7ae00ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.056+0000 7f7ae5800700 1 --2- 172.21.15.12:0/2518910420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 0x7f7ae00ce700 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00b5cb0 tx=0x7f7acc009bd0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.056+0000 7f7ae5800700 1 -- 172.21.15.12:0/2518910420 shutdown_connections 2024-03-29T11:36:19.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.056+0000 7f7ae5800700 1 --2- 172.21.15.12:0/2518910420 >> v2:172.21.15.159:3300/0 conn(0x7f7ae00cec40 0x7f7ae0161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.056+0000 7f7ae5800700 1 --2- 172.21.15.12:0/2518910420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 0x7f7ae00ce700 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.056+0000 7f7ae5800700 1 -- 172.21.15.12:0/2518910420 >> 172.21.15.12:0/2518910420 conn(0x7f7ae008aff0 msgr2=0x7f7ae00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:19.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.056+0000 7f7ae5800700 1 -- 172.21.15.12:0/2518910420 shutdown_connections 2024-03-29T11:36:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.056+0000 7f7ae5800700 1 -- 172.21.15.12:0/2518910420 wait complete. 2024-03-29T11:36:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.058+0000 7f7ae5800700 1 Processor -- start 2024-03-29T11:36:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.058+0000 7f7ae5800700 1 -- start start 2024-03-29T11:36:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.059+0000 7f7ae5800700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ae00cec40 0x7f7ae00c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.059+0000 7f7ae5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 0x7f7ae015f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.059+0000 7f7ae5800700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ae01611b0 0x7f7ae015fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.059+0000 7f7ae5800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae014a090 con 0x7f7ae00d21f0 2024-03-29T11:36:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.059+0000 7f7ae5800700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae0149760 con 0x7f7ae01611b0 2024-03-29T11:36:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.059+0000 7f7ae5800700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae01499d0 con 0x7f7ae00cec40 2024-03-29T11:36:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.059+0000 7f7ade000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 0x7f7ae015f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.060+0000 7f7ade000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 0x7f7ae015f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60270/0 (socket says 172.21.15.12:60270) 2024-03-29T11:36:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.060+0000 7f7ade000700 1 -- 172.21.15.12:0/1993881119 learned_addr learned my addr 172.21.15.12:0/1993881119 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.060+0000 7f7adea00700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ae00cec40 0x7f7ae00c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.060+0000 7f7adf400700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ae01611b0 0x7f7ae015fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.060+0000 7f7ade000700 1 -- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ae00cec40 msgr2=0x7f7ae00c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.060+0000 7f7ade000700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ae00cec40 0x7f7ae00c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.060+0000 7f7ade000700 1 -- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ae01611b0 msgr2=0x7f7ae015fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.060+0000 7f7ade000700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ae01611b0 0x7f7ae015fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.060+0000 7f7ade000700 1 -- 172.21.15.12:0/1993881119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc009550 con 0x7f7ae00d21f0 2024-03-29T11:36:19.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.060+0000 7f7ade000700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 0x7f7ae015f8b0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f7ae008ce10 tx=0x7f7acc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:19.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.061+0000 7f7ad6a00700 1 -- 172.21.15.12:0/1993881119 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc004250 con 0x7f7ae00d21f0 2024-03-29T11:36:19.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.061+0000 7f7ad6a00700 1 -- 172.21.15.12:0/1993881119 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7acc01f070 con 0x7f7ae00d21f0 2024-03-29T11:36:19.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.061+0000 7f7ad6a00700 1 -- 172.21.15.12:0/1993881119 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc0045b0 con 0x7f7ae00d21f0 2024-03-29T11:36:19.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.061+0000 7f7ae5800700 1 -- 172.21.15.12:0/1993881119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae0149cb0 con 0x7f7ae00d21f0 2024-03-29T11:36:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.061+0000 7f7ae5800700 1 -- 172.21.15.12:0/1993881119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae00cd380 con 0x7f7ae00d21f0 2024-03-29T11:36:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.062+0000 7f7ae5800700 1 -- 172.21.15.12:0/1993881119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae0067720 con 0x7f7ae00d21f0 2024-03-29T11:36:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.063+0000 7f7ad6a00700 1 -- 172.21.15.12:0/1993881119 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7acc021030 con 0x7f7ae00d21f0 2024-03-29T11:36:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.063+0000 7f7ad6a00700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7ac006adb0 0x7f7ac006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.064+0000 7f7ad6a00700 1 -- 172.21.15.12:0/1993881119 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f7acc08a680 con 0x7f7ae00d21f0 2024-03-29T11:36:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.064+0000 7f7adea00700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7ac006adb0 0x7f7ac006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.065+0000 7f7adea00700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7ac006adb0 0x7f7ac006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0000f60 tx=0x7f7ad0007e70 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.068+0000 7f7ad6a00700 1 -- 172.21.15.12:0/1993881119 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7acc055710 con 0x7f7ae00d21f0 2024-03-29T11:36:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.284+0000 7f7ae5800700 1 -- 172.21.15.12:0/1993881119 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ae015f6a0 con 0x7f7ac006adb0 2024-03-29T11:36:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.285+0000 7f7ad6a00700 1 -- 172.21.15.12:0/1993881119 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7ad00133f0 con 0x7f7ac006adb0 2024-03-29T11:36:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 -- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7ac006adb0 msgr2=0x7f7ac006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7ac006adb0 0x7f7ac006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0000f60 tx=0x7f7ad0007e70 comp rx=0 tx=0).stop 2024-03-29T11:36:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 -- 172.21.15.12:0/1993881119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 msgr2=0x7f7ae015f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 0x7f7ae015f8b0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f7ae008ce10 tx=0x7f7acc004750 comp rx=0 tx=0).stop 2024-03-29T11:36:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 -- 172.21.15.12:0/1993881119 shutdown_connections 2024-03-29T11:36:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f7ac006adb0 0x7f7ac006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ae01611b0 0x7f7ae015fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ae00cec40 0x7f7ae00c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 --2- 172.21.15.12:0/1993881119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ae00d21f0 0x7f7ae015f8b0 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 -- 172.21.15.12:0/1993881119 >> 172.21.15.12:0/1993881119 conn(0x7f7ae008aff0 msgr2=0x7f7ae015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 -- 172.21.15.12:0/1993881119 shutdown_connections 2024-03-29T11:36:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.288+0000 7f7ae5800700 1 -- 172.21.15.12:0/1993881119 wait complete. 2024-03-29T11:36:19.289 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 7 to deep-scrub 2024-03-29T11:36:19.305 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:36:19.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.505+0000 7fc55ce00700 1 Processor -- start 2024-03-29T11:36:19.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.505+0000 7fc55ce00700 1 -- start start 2024-03-29T11:36:19.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc55ce00700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc5580ac580 0x7fc5580b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc55ce00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc5580b3000 0x7fc55806dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc55ce00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5580b3890 con 0x7fc55815d670 2024-03-29T11:36:19.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc55ce00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc558159f50 con 0x7fc5580b3000 2024-03-29T11:36:19.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc55ce00700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5580b35b0 con 0x7fc5580ac580 2024-03-29T11:36:19.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc556800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc5580b3000 0x7fc55806dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc556800700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc5580b3000 0x7fc55806dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60276/0 (socket says 172.21.15.12:60276) 2024-03-29T11:36:19.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc556800700 1 -- 172.21.15.12:0/4023227897 learned_addr learned my addr 172.21.15.12:0/4023227897 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:19.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc555400700 1 --2- 172.21.15.12:0/4023227897 >> v2:172.21.15.159:3300/0 conn(0x7fc5580ac580 0x7fc5580b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc556800700 1 -- 172.21.15.12:0/4023227897 >> v2:172.21.15.159:3300/0 conn(0x7fc5580ac580 msgr2=0x7fc5580b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:19.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc556800700 1 --2- 172.21.15.12:0/4023227897 >> v2:172.21.15.159:3300/0 conn(0x7fc5580ac580 0x7fc5580b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc556800700 1 -- 172.21.15.12:0/4023227897 >> v1:172.21.15.159:6789/0 conn(0x7fc55815d670 legacy=0x7fc55815da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:19.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.507+0000 7fc556800700 1 -- 172.21.15.12:0/4023227897 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55806d400 con 0x7fc5580b3000 2024-03-29T11:36:19.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.508+0000 7fc556800700 1 --2- 172.21.15.12:0/4023227897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580b3000 0x7fc55806dee0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7fc5580cb2e0 tx=0x7fc55000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3813af2ac8546040 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:19.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.508+0000 7fc554a00700 1 -- 172.21.15.12:0/4023227897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55000e070 con 0x7fc5580b3000 2024-03-29T11:36:19.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.508+0000 7fc554a00700 1 -- 172.21.15.12:0/4023227897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc550003a20 con 0x7fc5580b3000 2024-03-29T11:36:19.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.508+0000 7fc554a00700 1 -- 172.21.15.12:0/4023227897 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550003dd0 con 0x7fc5580b3000 2024-03-29T11:36:19.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.508+0000 7fc55ce00700 1 -- 172.21.15.12:0/4023227897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580b3000 msgr2=0x7fc55806dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.508+0000 7fc55ce00700 1 --2- 172.21.15.12:0/4023227897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580b3000 0x7fc55806dee0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7fc5580cb2e0 tx=0x7fc55000b330 comp rx=0 tx=0).stop 2024-03-29T11:36:19.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.508+0000 7fc55ce00700 1 -- 172.21.15.12:0/4023227897 shutdown_connections 2024-03-29T11:36:19.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.508+0000 7fc55ce00700 1 --2- 172.21.15.12:0/4023227897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5580b3000 0x7fc55806dee0 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.508+0000 7fc55ce00700 1 --2- 172.21.15.12:0/4023227897 >> v2:172.21.15.159:3300/0 conn(0x7fc5580ac580 0x7fc5580b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.508+0000 7fc55ce00700 1 -- 172.21.15.12:0/4023227897 >> 172.21.15.12:0/4023227897 conn(0x7fc5580cf190 msgr2=0x7fc5580cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:19.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.509+0000 7fc55ce00700 1 -- 172.21.15.12:0/4023227897 shutdown_connections 2024-03-29T11:36:19.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.509+0000 7fc55ce00700 1 -- 172.21.15.12:0/4023227897 wait complete. 2024-03-29T11:36:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.511+0000 7fc55ce00700 1 Processor -- start 2024-03-29T11:36:19.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.511+0000 7fc55ce00700 1 -- start start 2024-03-29T11:36:19.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.512+0000 7fc55ce00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580ac580 0x7fc5581598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.512+0000 7fc55ce00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5580b3000 0x7fc5580c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.512+0000 7fc55ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc55815d670 0x7fc5580b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.512+0000 7fc55ce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580c89a0 con 0x7fc55815d670 2024-03-29T11:36:19.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.512+0000 7fc55ce00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580c8c20 con 0x7fc5580ac580 2024-03-29T11:36:19.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.512+0000 7fc55ce00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580c8f00 con 0x7fc5580b3000 2024-03-29T11:36:19.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc556800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc55815d670 0x7fc5580b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc556800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc55815d670 0x7fc5580b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60278/0 (socket says 172.21.15.12:60278) 2024-03-29T11:36:19.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc556800700 1 -- 172.21.15.12:0/4084778903 learned_addr learned my addr 172.21.15.12:0/4084778903 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:19.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc555e00700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580ac580 0x7fc5581598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc555400700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5580b3000 0x7fc5580c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc556800700 1 -- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5580b3000 msgr2=0x7fc5580c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc556800700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5580b3000 0x7fc5580c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc556800700 1 -- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580ac580 msgr2=0x7fc5581598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc556800700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580ac580 0x7fc5581598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc556800700 1 -- 172.21.15.12:0/4084778903 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc550009b30 con 0x7fc55815d670 2024-03-29T11:36:19.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.513+0000 7fc556800700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc55815d670 0x7fc5580b0490 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7fc55000b370 tx=0x7fc550004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:19.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.514+0000 7fc54e000700 1 -- 172.21.15.12:0/4084778903 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550004440 con 0x7fc55815d670 2024-03-29T11:36:19.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.514+0000 7fc54e000700 1 -- 172.21.15.12:0/4084778903 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5500092e0 con 0x7fc55815d670 2024-03-29T11:36:19.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.514+0000 7fc55ce00700 1 -- 172.21.15.12:0/4084778903 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5580cc5c0 con 0x7fc55815d670 2024-03-29T11:36:19.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.514+0000 7fc54e000700 1 -- 172.21.15.12:0/4084778903 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550004800 con 0x7fc55815d670 2024-03-29T11:36:19.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.514+0000 7fc55ce00700 1 -- 172.21.15.12:0/4084778903 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5580c91e0 con 0x7fc55815d670 2024-03-29T11:36:19.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.515+0000 7fc55ce00700 1 -- 172.21.15.12:0/4084778903 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5580cca80 con 0x7fc55815d670 2024-03-29T11:36:19.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.516+0000 7fc54e000700 1 -- 172.21.15.12:0/4084778903 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc550012430 con 0x7fc55815d670 2024-03-29T11:36:19.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.517+0000 7fc54e000700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc53806adb0 0x7fc53806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:19.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.517+0000 7fc54e000700 1 -- 172.21.15.12:0/4084778903 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fc55008a970 con 0x7fc55815d670 2024-03-29T11:36:19.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.518+0000 7fc555e00700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc53806adb0 0x7fc53806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:19.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.518+0000 7fc555e00700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc53806adb0 0x7fc53806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc5580cb2b0 tx=0x7fc5480090e0 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.521+0000 7fc54e000700 1 -- 172.21.15.12:0/4084778903 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc5500558f0 con 0x7fc55815d670 2024-03-29T11:36:19.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.728+0000 7fc55ce00700 1 -- 172.21.15.12:0/4084778903 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5580ac9e0 con 0x7fc53806adb0 2024-03-29T11:36:19.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.731+0000 7fc54e000700 1 -- 172.21.15.12:0/4084778903 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50308 (secure 0 0 0) 0x7fc548009e20 con 0x7fc53806adb0 2024-03-29T11:36:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:19.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 -- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc53806adb0 msgr2=0x7fc53806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc53806adb0 0x7fc53806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc5580cb2b0 tx=0x7fc5480090e0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 -- 172.21.15.12:0/4084778903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc55815d670 msgr2=0x7fc5580b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:19.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc55815d670 0x7fc5580b0490 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7fc55000b370 tx=0x7fc550004ad0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 -- 172.21.15.12:0/4084778903 shutdown_connections 2024-03-29T11:36:19.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fc53806adb0 0x7fc53806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc55815d670 0x7fc5580b0490 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5580ac580 0x7fc5581598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 --2- 172.21.15.12:0/4084778903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5580b3000 0x7fc5580c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:19.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 -- 172.21.15.12:0/4084778903 >> 172.21.15.12:0/4084778903 conn(0x7fc5580cf190 msgr2=0x7fc5580ad7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:19.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 -- 172.21.15.12:0/4084778903 shutdown_connections 2024-03-29T11:36:19.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:19.734+0000 7fc55ce00700 1 -- 172.21.15.12:0/4084778903 wait complete. 2024-03-29T11:36:19.735 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:36:19.751 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-03-29T11:36:18.187801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":467384,"kb_used_data":29828,"kb_used_omap":12,"kb_used_meta":437427,"kb_avail":754507336,"statfs":{"total":773094113280,"available":772615512064,"internally_reserved":0,"allocated":30543872,"data_stored":15654597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":447925817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.800920"},"pg_stats":[{"pgid":"1.0","version":"288'454","reported_seq":1639,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-29T11:36:06.175425+0000","last_change":"2024-03-29T09:54:17.857004+0000","last_active":"2024-03-29T11:36:06.175425+0000","last_peered":"2024-03-29T11:36:06.175425+0000","last_clean":"2024-03-29T11:36:06.175425+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:36:06.175425+0000","last_undegraded":"2024-03-29T11:36:06.175425+0000","last_fullsized":"2024-03-29T11:36:06.175425+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T09:54:16.825941+0000","last_clean_scrub_stamp":"2024-03-29T09:54:16.825941+0000","objects_scrubbed":0,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:59:09.514451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576044,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57424,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314416,"statfs":{"total":96636764160,"available":96577961984,"internally_reserved":0,"allocated":4263936,"data_stored":2399728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.67100000000000004,"15min":0.70899999999999996},"min":{"1min":0.47899999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91600000000000004,"5min":1.155,"15min":1.155},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68200000000000005,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94699999999999995,"5min":1.137,"15min":1.137},"last":0.74099999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.66500000000000004,"15min":0.71399999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.026,"5min":1.026,"15min":1.478},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.69399999999999995,"15min":0.745},"min":{"1min":0.57299999999999995,"5min":0.434,"15min":0.434},"max":{"1min":0.96199999999999997,"5min":1.0640000000000001,"15min":1.393},"last":0.48199999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.69199999999999995,"15min":0.73899999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.008,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.71699999999999997,"15min":0.75600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.99299999999999999,"5min":1.04,"15min":1.0720000000000001},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.69099999999999995,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.0580000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.77000000000000002,"15min":0.78400000000000003},"min":{"1min":0.54000000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.0620000000000001,"5min":2.6930000000000001,"15min":2.6930000000000001},"last":0.58899999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.84799999999999998,"5min":0.95499999999999996,"15min":1.0760000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.46600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":0.93100000000000005,"15min":0.98199999999999998},"last":0.79800000000000004}]},{"osd":5,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69199999999999995,"15min":0.68400000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0920000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.48999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0129999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.77800000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.53700000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.036,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73599999999999999,"15min":0.73799999999999999},"min":{"1min":0.52300000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.98299999999999998,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.84399999999999997}]}]},{"osd":6,"up_from":13,"seq":55834576045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48652,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323188,"statfs":{"total":96636764160,"available":96586944512,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.65300000000000002,"15min":0.71099999999999997},"min":{"1min":0.55700000000000005,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.876,"5min":1.0409999999999999,"15min":1.2150000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.67600000000000005,"15min":0.73499999999999999},"min":{"1min":0.61799999999999999,"5min":0.40200000000000002,"15min":0.39100000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.4239999999999999},"last":0.48099999999999998}]},{"osd":1,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65200000000000002,"15min":0.71099999999999997},"min":{"1min":0.54300000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.81399999999999995,"5min":1.022,"15min":1.98},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.71199999999999997,"15min":0.753},"min":{"1min":0.498,"5min":0.435,"15min":0.435},"max":{"1min":0.97699999999999998,"5min":1.0489999999999999,"15min":1.8200000000000001},"last":0.73199999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.69199999999999995,"15min":0.73299999999999998},"min":{"1min":0.57799999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92600000000000005,"5min":0.92800000000000005,"15min":1.669},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.69499999999999995,"15min":0.747},"min":{"1min":0.61199999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.98599999999999999,"5min":1.167,"15min":1.7589999999999999},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.69999999999999996,"15min":0.749},"min":{"1min":0.67900000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91900000000000004,"5min":0.997,"15min":2.52},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.78500000000000003},"min":{"1min":0.55600000000000005,"5min":0.51200000000000001,"15min":0.47799999999999998},"max":{"1min":1.0249999999999999,"5min":1.1160000000000001,"15min":1.877},"last":0.78500000000000003}]},{"osd":4,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.439,"5min":0.376,"15min":0.376},"max":{"1min":0.87,"5min":0.93799999999999994,"15min":2.4100000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.70599999999999996,"15min":0.71099999999999997},"min":{"1min":0.51700000000000002,"5min":0.46000000000000002,"15min":0.45900000000000002},"max":{"1min":0.93200000000000005,"5min":1.2709999999999999,"15min":2.6629999999999998},"last":0.69499999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.438,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72899999999999998,"5min":1.0900000000000001,"15min":1.5960000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88500000000000001,"5min":1.1180000000000001,"15min":1.925},"last":0.76100000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.76400000000000001,"15min":0.76900000000000002},"min":{"1min":0.58499999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88300000000000001,"5min":1.056,"15min":2.7050000000000001},"last":0.748},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71699999999999997,"15min":0.73899999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81399999999999995,"5min":1.1220000000000001,"15min":1.726},"last":0.71799999999999997}]}]},{"osd":1,"up_from":13,"seq":55834576043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48656,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323184,"statfs":{"total":96636764160,"available":96586940416,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.379,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56899999999999995,"5min":0.71099999999999997,"15min":0.80300000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.5,"15min":0.52600000000000002},"min":{"1min":0.33500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.79000000000000004,"5min":0.84999999999999998,"15min":0.89000000000000001},"last":0.40400000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72899999999999998,"5min":0.90200000000000002,"15min":0.92600000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.503,"15min":0.52100000000000002},"min":{"1min":0.39700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71499999999999997,"5min":0.75900000000000001,"15min":0.90300000000000002},"last":0.505}]},{"osd":3,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.36299999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53500000000000003,"15min":0.56699999999999995},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76000000000000001,"5min":0.77700000000000002,"15min":0.83999999999999997},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.77300000000000002},"min":{"1min":0.60699999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.008,"5min":1.0589999999999999,"15min":1.238},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.75600000000000001,"15min":0.78700000000000003},"min":{"1min":0.59499999999999997,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":1.0169999999999999,"5min":1.097,"15min":1.304},"last":0.747}]},{"osd":5,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78400000000000003,"15min":0.80500000000000005},"min":{"1min":0.60999999999999999,"5min":0.54900000000000004,"15min":0.48199999999999998},"max":{"1min":1.069,"5min":1.1359999999999999,"15min":1.3720000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.77800000000000002,"15min":0.81499999999999995},"min":{"1min":0.55600000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96199999999999997,"5min":1.2030000000000001,"15min":1.3520000000000001},"last":0.873}]},{"osd":6,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.77800000000000002,"15min":0.80400000000000005},"min":{"1min":0.57099999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.03,"5min":1.103,"15min":1.1919999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.80600000000000005,"15min":0.83299999999999996},"min":{"1min":0.63900000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.0309999999999999,"5min":1.2170000000000001,"15min":1.266},"last":0.92600000000000005}]},{"osd":7,"last update":"Fri Mar 29 11:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81899999999999995,"15min":0.85299999999999998},"min":{"1min":0.63200000000000001,"5min":0.54800000000000004,"15min":0.54500000000000004},"max":{"1min":1.0089999999999999,"5min":1.129,"15min":1.327},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.81599999999999995,"15min":0.85099999999999998},"min":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.57499999999999996},"max":{"1min":1.0549999999999999,"5min":1.2430000000000001,"15min":1.3420000000000001},"last":0.91300000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576045,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65616,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306224,"statfs":{"total":96636764160,"available":96569573376,"internally_reserved":0,"allocated":4259840,"data_stored":2398709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.46300000000000002},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.23000000000000001},"max":{"1min":0.65500000000000003,"5min":0.83999999999999997,"15min":1.087},"last":0.308},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.499,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70399999999999996,"5min":0.77400000000000002,"15min":1.3089999999999999},"last":0.50900000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.53400000000000003,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.754,"5min":0.84999999999999998,"15min":1.5289999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47599999999999998,"15min":0.51500000000000001},"min":{"1min":0.372,"5min":0.308,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":1.6140000000000001},"last":0.36099999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.52500000000000002,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78300000000000003,"5min":0.81000000000000005,"15min":1.222},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59299999999999997,"15min":0.63500000000000001},"min":{"1min":0.49199999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.876,"5min":0.876,"15min":1.3899999999999999},"last":0.46899999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.76700000000000002,"15min":0.80300000000000005},"min":{"1min":0.70499999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0800000000000001,"5min":1.248,"15min":1.248},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.79400000000000004,"15min":0.83799999999999997},"min":{"1min":0.626,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.5820000000000001},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.72999999999999998,"15min":0.77300000000000002},"min":{"1min":0.63600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.3440000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.77700000000000002,"15min":0.81399999999999995},"min":{"1min":0.749,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.113,"5min":1.1499999999999999,"15min":1.3640000000000001},"last":0.61699999999999999}]},{"osd":6,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.78200000000000003,"15min":0.84199999999999997},"min":{"1min":0.60699999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.6240000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.82299999999999995,"15min":0.871},"min":{"1min":0.751,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.337},"last":0.75800000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.78100000000000003,"15min":0.81599999999999995},"min":{"1min":0.71099999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.214},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.81000000000000005,"15min":0.86099999999999999},"min":{"1min":0.76300000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.365},"last":0.64600000000000002}]}]},{"osd":2,"up_from":13,"seq":55834576044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52752,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319088,"statfs":{"total":96636764160,"available":96582746112,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45600000000000002,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.29199999999999998,"15min":0.27800000000000002},"max":{"1min":0.63600000000000001,"5min":0.73199999999999998,"15min":0.75900000000000001},"last":0.437},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.49099999999999999,"15min":0.52700000000000002},"min":{"1min":0.38500000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.96799999999999997},"last":0.45800000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.47699999999999998},"min":{"1min":0.35899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69599999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67000000000000004,"5min":0.872,"15min":0.88800000000000001},"last":0.57899999999999996}]},{"osd":3,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58099999999999996},"min":{"1min":0.42999999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.86799999999999999,"15min":0.98499999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.56899999999999995},"min":{"1min":0.38800000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.038},"last":0.69999999999999996}]},{"osd":4,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.75600000000000001,"15min":0.79500000000000004},"min":{"1min":0.54000000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.026,"5min":1.1100000000000001,"15min":1.1970000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.74399999999999999,"15min":0.79000000000000004},"min":{"1min":0.55300000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.0049999999999999,"5min":1.157,"15min":1.244},"last":0.64300000000000002}]},{"osd":5,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.024,"5min":1.171,"15min":1.256},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.77100000000000002,"15min":0.81200000000000006},"min":{"1min":0.57399999999999995,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0649999999999999,"5min":1.1180000000000001,"15min":1.3140000000000001},"last":0.875}]},{"osd":6,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.79100000000000004,"15min":0.83299999999999996},"min":{"1min":0.62,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0800000000000001,"5min":1.21,"15min":1.24},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.78200000000000003,"15min":0.82599999999999996},"min":{"1min":0.60399999999999998,"5min":0.5,"15min":0.5},"max":{"1min":1.036,"5min":1.0920000000000001,"15min":1.329},"last":0.82999999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.83499999999999996,"15min":0.879},"min":{"1min":0.68200000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.087,"5min":1.151,"15min":1.2889999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.84299999999999997,"15min":0.88600000000000001},"min":{"1min":0.63600000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1040000000000001,"5min":1.234,"15min":1.304},"last":0.78000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48652,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323188,"statfs":{"total":96636764160,"available":96586944512,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.34699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.65200000000000002,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51800000000000002,"15min":0.54000000000000004},"min":{"1min":0.36399999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.432}]},{"osd":1,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48599999999999999,"15min":0.49099999999999999},"min":{"1min":0.39300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80200000000000005,"5min":1.821,"15min":1.821},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.56299999999999994},"min":{"1min":0.48599999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94599999999999995,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.627,"15min":0.65000000000000002},"min":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81499999999999995,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.58899999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.753,"15min":0.78100000000000003},"min":{"1min":0.66200000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0609999999999999,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77100000000000002,"15min":0.80000000000000004},"min":{"1min":0.61199999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0820000000000001,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.745}]},{"osd":5,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77800000000000002,"15min":0.79900000000000004},"min":{"1min":0.629,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.089,"5min":2.157,"15min":2.157},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1699999999999999,"5min":2.266,"15min":2.266},"last":0.84799999999999998}]},{"osd":6,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.82199999999999995,"15min":0.84699999999999998},"min":{"1min":0.68200000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.121,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.872,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.66300000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.137,"5min":2.198,"15min":2.198},"last":0.878}]},{"osd":7,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.83599999999999997,"15min":0.84999999999999998},"min":{"1min":0.68100000000000005,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":1.111,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.85999999999999999,"15min":0.879},"min":{"1min":0.64400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.153,"5min":2.444,"15min":2.444},"last":0.93500000000000005}]}]},{"osd":4,"up_from":13,"seq":55834576045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96976,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274864,"statfs":{"total":96636764160,"available":96537460736,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.65600000000000003,"15min":0.69899999999999995},"min":{"1min":0.433,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.0309999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.64900000000000002,"15min":0.69899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88200000000000001,"5min":1.02,"15min":1.02},"last":0.52800000000000002}]},{"osd":1,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.67100000000000004,"15min":0.71899999999999997},"min":{"1min":0.58299999999999996,"5min":0.436,"15min":0.432},"max":{"1min":0.96799999999999997,"5min":1.135,"15min":1.135},"last":1.046},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.68100000000000005,"15min":0.72399999999999998},"min":{"1min":0.54700000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97099999999999997,"5min":1.004,"15min":1.004},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.76800000000000002,"15min":0.80700000000000005},"min":{"1min":0.70099999999999996,"5min":0.52100000000000002,"15min":0.49199999999999999},"max":{"1min":1.016,"5min":1.8,"15min":1.8},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.74199999999999999,"15min":0.79100000000000004},"min":{"1min":0.53200000000000003,"5min":0.503,"15min":0.496},"max":{"1min":0.96599999999999997,"5min":1.0369999999999999,"15min":1.224},"last":0.65300000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.69399999999999995,"15min":0.73699999999999999},"min":{"1min":0.55600000000000005,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92300000000000004,"5min":0.96699999999999997,"15min":1.0580000000000001},"last":0.996},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.71999999999999997,"15min":0.755},"min":{"1min":0.60599999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.51000000000000001}]},{"osd":5,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.65200000000000002,"15min":0.63200000000000001},"min":{"1min":0.48099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.872,"5min":1.032,"15min":1.032},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63900000000000001,"15min":0.64500000000000002},"min":{"1min":0.505,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81299999999999994,"5min":1.01,"15min":1.0920000000000001},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.69799999999999995,"15min":0.67500000000000004},"min":{"1min":0.48999999999999999,"5min":0.44700000000000001,"15min":0.40899999999999997},"max":{"1min":0.876,"5min":1.722,"15min":1.722},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73199999999999998,"15min":0.72099999999999997},"min":{"1min":0.57599999999999996,"5min":0.439,"15min":0.439},"max":{"1min":0.93300000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:35:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.45700000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.622},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81299999999999994,"15min":0.81699999999999995},"min":{"1min":0.68400000000000005,"5min":0.57499999999999996,"15min":0.54400000000000004},"max":{"1min":0.98199999999999998,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.71799999999999997}]}]},{"osd":5,"up_from":13,"seq":55834576044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48656,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323184,"statfs":{"total":96636764160,"available":96586940416,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.63400000000000001,"15min":0.68400000000000005},"min":{"1min":0.52200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.66200000000000003,"15min":0.71199999999999997},"min":{"1min":0.54700000000000004,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0680000000000001},"last":0.54700000000000004}]},{"osd":1,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.70199999999999996,"15min":0.746},"min":{"1min":0.58699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0980000000000001,"5min":1.839,"15min":1.839},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.69699999999999995,"15min":0.72399999999999998},"min":{"1min":0.499,"5min":0.438,"15min":0.438},"max":{"1min":0.94599999999999995,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.59699999999999998}]},{"osd":2,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.70799999999999996,"15min":0.74299999999999999},"min":{"1min":0.60599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69799999999999995,"15min":0.73499999999999999},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97599999999999998,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.64500000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.75,"15min":0.76400000000000001},"min":{"1min":0.70099999999999996,"5min":0.443,"15min":0.443},"max":{"1min":0.94999999999999996,"5min":3.444,"15min":3.444},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.72299999999999998,"15min":0.76400000000000001},"min":{"1min":0.63800000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.996,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.82299999999999995}]},{"osd":4,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.439,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84799999999999998,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.95699999999999996,"5min":0.999,"15min":1.0660000000000001},"last":0.76700000000000002}]},{"osd":6,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.67500000000000004},"min":{"1min":0.496,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92300000000000004,"5min":1.214,"15min":1.214},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.69799999999999995,"15min":0.70399999999999996},"min":{"1min":0.58099999999999996,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":1.0900000000000001,"15min":1.107},"last":0.755}]},{"osd":7,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71899999999999997,"15min":0.73199999999999998},"min":{"1min":0.48899999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.89000000000000001,"5min":1.04,"15min":1.2010000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.55600000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89100000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.88900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:36:19.755 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-29T09:54:16.825941+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=29, tm_hour=9, tm_min=54, tm_sec=16, tm_wday=4, tm_yday=89, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=29, tm_hour=11, tm_min=36, tm_sec=12, tm_wday=4, tm_yday=89, tm_isdst=0) 2024-03-29T11:36:19.755 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-29T11:36:22.261 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:36:22.266 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:36:39.756 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T11:36:39.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.967+0000 7f62cc200700 1 Processor -- start 2024-03-29T11:36:39.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.968+0000 7f62cc200700 1 -- start start 2024-03-29T11:36:39.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.969+0000 7f62cc200700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f62c40709c0 0x7f62c4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:39.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.969+0000 7f62cc200700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c40b2180 con 0x7f62c40b3af0 2024-03-29T11:36:39.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.969+0000 7f62cc200700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c415b100 con 0x7f62c4072580 2024-03-29T11:36:39.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.969+0000 7f62cc200700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c415b3e0 con 0x7f62c40709c0 2024-03-29T11:36:39.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.969+0000 7f62c9800700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f62c40b3af0 0x7f62c415a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60102/0 (socket says 172.21.15.12:60102) 2024-03-29T11:36:39.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.969+0000 7f62c9800700 1 -- 172.21.15.12:0/1170464179 learned_addr learned my addr 172.21.15.12:0/1170464179 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:39.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.969+0000 7f62ca200700 1 --2- 172.21.15.12:0/1170464179 >> v2:172.21.15.12:3300/0 conn(0x7f62c40709c0 0x7f62c4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:39.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.970+0000 7f62c3e00700 1 -- 172.21.15.12:0/1170464179 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944847880 0 0) 0x7f62c40b2180 con 0x7f62c40b3af0 2024-03-29T11:36:39.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.970+0000 7f62c3e00700 1 -- 172.21.15.12:0/1170464179 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ac0035b0 con 0x7f62c40b3af0 2024-03-29T11:36:39.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.970+0000 7f62ca200700 1 -- 172.21.15.12:0/1170464179 >> v1:172.21.15.159:6789/0 conn(0x7f62c4072580 legacy=0x7f62c40b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:39.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.970+0000 7f62ca200700 1 -- 172.21.15.12:0/1170464179 >> v1:172.21.15.12:6789/0 conn(0x7f62c40b3af0 legacy=0x7f62c415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.970+0000 7f62ca200700 1 -- 172.21.15.12:0/1170464179 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c40b23f0 con 0x7f62c40709c0 2024-03-29T11:36:39.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.970+0000 7f62ca200700 1 --2- 172.21.15.12:0/1170464179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40709c0 0x7f62c4070e00 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009d90 tx=0x7f62bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c044f418dce2e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:39.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.970+0000 7f62c3e00700 1 -- 172.21.15.12:0/1170464179 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc00e070 con 0x7f62c40709c0 2024-03-29T11:36:39.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.970+0000 7f62c3e00700 1 -- 172.21.15.12:0/1170464179 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc00baa0 con 0x7f62c40709c0 2024-03-29T11:36:39.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.970+0000 7f62c3e00700 1 -- 172.21.15.12:0/1170464179 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc00be50 con 0x7f62c40709c0 2024-03-29T11:36:39.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.971+0000 7f62cc200700 1 -- 172.21.15.12:0/1170464179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40709c0 msgr2=0x7f62c4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.971+0000 7f62cc200700 1 --2- 172.21.15.12:0/1170464179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40709c0 0x7f62c4070e00 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009d90 tx=0x7f62bc00b330 comp rx=0 tx=0).stop 2024-03-29T11:36:39.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.971+0000 7f62cc200700 1 -- 172.21.15.12:0/1170464179 shutdown_connections 2024-03-29T11:36:39.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.971+0000 7f62cc200700 1 --2- 172.21.15.12:0/1170464179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40709c0 0x7f62c4070e00 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:39.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.971+0000 7f62cc200700 1 -- 172.21.15.12:0/1170464179 >> 172.21.15.12:0/1170464179 conn(0x7f62c40cf190 msgr2=0x7f62c40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:39.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.971+0000 7f62cc200700 1 -- 172.21.15.12:0/1170464179 shutdown_connections 2024-03-29T11:36:39.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.971+0000 7f62cc200700 1 -- 172.21.15.12:0/1170464179 wait complete. 2024-03-29T11:36:39.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.973+0000 7f62cc200700 1 Processor -- start 2024-03-29T11:36:39.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.973+0000 7f62cc200700 1 -- start start 2024-03-29T11:36:39.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62cc200700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62c40709c0 0x7f62c41f1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:39.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62cc200700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62c4072580 0x7f62c41574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:39.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62cc200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40b3af0 0x7f62c409c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:39.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62cc200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c40cd8b0 con 0x7f62c40b3af0 2024-03-29T11:36:39.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62cc200700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4155bc0 con 0x7f62c4072580 2024-03-29T11:36:39.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62cc200700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4155ea0 con 0x7f62c40709c0 2024-03-29T11:36:39.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62ca200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40b3af0 0x7f62c409c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:39.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62ca200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40b3af0 0x7f62c409c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59314/0 (socket says 172.21.15.12:59314) 2024-03-29T11:36:39.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62ca200700 1 -- 172.21.15.12:0/575892541 learned_addr learned my addr 172.21.15.12:0/575892541 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:39.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62c9800700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62c40709c0 0x7f62c41f1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:39.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62ca200700 1 -- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62c40709c0 msgr2=0x7f62c41f1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62ca200700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62c40709c0 0x7f62c41f1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:39.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62ca200700 1 -- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62c4072580 msgr2=0x7f62c41574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62c8e00700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62c4072580 0x7f62c41574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:39.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62ca200700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62c4072580 0x7f62c41574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:39.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62ca200700 1 -- 172.21.15.12:0/575892541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc009a70 con 0x7f62c40b3af0 2024-03-29T11:36:39.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62c9800700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62c40709c0 0x7f62c41f1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:36:39.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.974+0000 7f62c8e00700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62c4072580 0x7f62c41574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:36:39.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.975+0000 7f62ca200700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40b3af0 0x7f62c409c6c0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f62bc00b370 tx=0x7f62bc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:39.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.975+0000 7f62c1600700 1 -- 172.21.15.12:0/575892541 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc009430 con 0x7f62c40b3af0 2024-03-29T11:36:39.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.975+0000 7f62c1600700 1 -- 172.21.15.12:0/575892541 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc026460 con 0x7f62c40b3af0 2024-03-29T11:36:39.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.975+0000 7f62cc200700 1 -- 172.21.15.12:0/575892541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c40cd260 con 0x7f62c40b3af0 2024-03-29T11:36:39.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.975+0000 7f62c1600700 1 -- 172.21.15.12:0/575892541 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc026810 con 0x7f62c40b3af0 2024-03-29T11:36:39.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.975+0000 7f62cc200700 1 -- 172.21.15.12:0/575892541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c4155820 con 0x7f62c40b3af0 2024-03-29T11:36:39.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.976+0000 7f62cc200700 1 -- 172.21.15.12:0/575892541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c40cda20 con 0x7f62c40b3af0 2024-03-29T11:36:39.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.977+0000 7f62c1600700 1 -- 172.21.15.12:0/575892541 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f62bc012430 con 0x7f62c40b3af0 2024-03-29T11:36:39.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.978+0000 7f62c1600700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f62a406adb0 0x7f62a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:39.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.978+0000 7f62c1600700 1 -- 172.21.15.12:0/575892541 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f62bc08b6e0 con 0x7f62c40b3af0 2024-03-29T11:36:39.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.979+0000 7f62c9800700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f62a406adb0 0x7f62a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:39.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.979+0000 7f62c9800700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f62a406adb0 0x7f62a406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f62b8000f60 tx=0x7f62b8009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:39.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:39.983+0000 7f62c1600700 1 -- 172.21.15.12:0/575892541 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f62bc056660 con 0x7f62c40b3af0 2024-03-29T11:36:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.193+0000 7f62cc200700 1 -- 172.21.15.12:0/575892541 --> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62c40b3ed0 con 0x7f62a406adb0 2024-03-29T11:36:40.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.195+0000 7f62c1600700 1 -- 172.21.15.12:0/575892541 <== mgr.10910 v2:172.21.15.159:6832/62646 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50094 (secure 0 0 0) 0x7f62b80070d0 con 0x7f62a406adb0 2024-03-29T11:36:40.196 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 -- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f62a406adb0 msgr2=0x7f62a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f62a406adb0 0x7f62a406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f62b8000f60 tx=0x7f62b8009040 comp rx=0 tx=0).stop 2024-03-29T11:36:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 -- 172.21.15.12:0/575892541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40b3af0 msgr2=0x7f62c409c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40b3af0 0x7f62c409c6c0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f62bc00b370 tx=0x7f62bc004ad0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 -- 172.21.15.12:0/575892541 shutdown_connections 2024-03-29T11:36:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f62a406adb0 0x7f62a406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62c40b3af0 0x7f62c409c6c0 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62c40709c0 0x7f62c41f1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 --2- 172.21.15.12:0/575892541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62c4072580 0x7f62c41574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 -- 172.21.15.12:0/575892541 >> 172.21.15.12:0/575892541 conn(0x7f62c40cf190 msgr2=0x7f62c406e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 -- 172.21.15.12:0/575892541 shutdown_connections 2024-03-29T11:36:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.198+0000 7f62cc200700 1 -- 172.21.15.12:0/575892541 wait complete. 2024-03-29T11:36:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-29T11:36:40.216 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-29T11:36:40.192571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":455,"ondisk_log_size":455,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":467372,"kb_used_data":29828,"kb_used_omap":12,"kb_used_meta":437427,"kb_avail":754507348,"statfs":{"total":773094113280,"available":772615524352,"internally_reserved":0,"allocated":30543872,"data_stored":15654597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":447925817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002732"},"pg_stats":[{"pgid":"1.0","version":"290'455","reported_seq":1652,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-29T11:36:22.941908+0000","last_change":"2024-03-29T11:36:22.942031+0000","last_active":"2024-03-29T11:36:22.941908+0000","last_peered":"2024-03-29T11:36:22.941908+0000","last_clean":"2024-03-29T11:36:22.941908+0000","last_became_active":"2024-03-29T09:54:17.856641+0000","last_became_peered":"2024-03-29T09:54:17.856641+0000","last_unstale":"2024-03-29T11:36:22.941908+0000","last_undegraded":"2024-03-29T11:36:22.941908+0000","last_fullsized":"2024-03-29T11:36:22.941908+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'455","last_scrub_stamp":"2024-03-29T11:36:22.941882+0000","last_deep_scrub":"290'455","last_deep_scrub_stamp":"2024-03-29T11:36:22.941882+0000","last_clean_scrub_stamp":"2024-03-29T11:36:22.941882+0000","objects_scrubbed":2,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T18:56:48.934688+0000","scrub_duration":0.0066895360000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3056,"num_read_kb":6456,"num_write":771,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":455,"ondisk_log_size":455,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576048,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57420,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314420,"statfs":{"total":96636764160,"available":96577966080,"internally_reserved":0,"allocated":4263936,"data_stored":2399728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":54524364},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.67100000000000004,"15min":0.70899999999999996},"min":{"1min":0.47899999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91600000000000004,"5min":1.155,"15min":1.155},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68200000000000005,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94699999999999995,"5min":1.137,"15min":1.137},"last":0.81499999999999995}]},{"osd":1,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.66500000000000004,"15min":0.71399999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.026,"5min":1.026,"15min":1.478},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.69399999999999995,"15min":0.745},"min":{"1min":0.57299999999999995,"5min":0.434,"15min":0.434},"max":{"1min":0.96199999999999997,"5min":1.0640000000000001,"15min":1.393},"last":0.79100000000000004}]},{"osd":2,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.69199999999999995,"15min":0.73899999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.008,"5min":2.5649999999999999,"15min":2.5649999999999999},"last":0.873},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.71699999999999997,"15min":0.75600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.99299999999999999,"5min":1.04,"15min":1.0720000000000001},"last":0.99099999999999999}]},{"osd":3,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.69099999999999995,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.0580000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.77000000000000002,"15min":0.78400000000000003},"min":{"1min":0.54000000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.0620000000000001,"5min":2.6930000000000001,"15min":2.6930000000000001},"last":0.92100000000000004}]},{"osd":4,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.84799999999999998,"5min":0.95499999999999996,"15min":1.0760000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.46600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":0.93100000000000005,"15min":0.98199999999999998},"last":0.75}]},{"osd":5,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69199999999999995,"15min":0.68400000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0920000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.48999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0129999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.94499999999999995}]},{"osd":6,"last update":"Fri Mar 29 11:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.53700000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.036,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73599999999999999,"15min":0.73799999999999999},"min":{"1min":0.52300000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.98299999999999998,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.90300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834576049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48652,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323188,"statfs":{"total":96636764160,"available":96586944512,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.65300000000000002,"15min":0.71099999999999997},"min":{"1min":0.55700000000000005,"5min":0.38600000000000001,"15min":0.38400000000000001},"max":{"1min":0.876,"5min":1.0409999999999999,"15min":1.2150000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.67600000000000005,"15min":0.73499999999999999},"min":{"1min":0.61799999999999999,"5min":0.40200000000000002,"15min":0.39100000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.4239999999999999},"last":0.747}]},{"osd":1,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65200000000000002,"15min":0.71099999999999997},"min":{"1min":0.54300000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.81399999999999995,"5min":1.022,"15min":1.98},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.71199999999999997,"15min":0.753},"min":{"1min":0.498,"5min":0.435,"15min":0.435},"max":{"1min":0.97699999999999998,"5min":1.0489999999999999,"15min":1.8200000000000001},"last":0.67800000000000005}]},{"osd":2,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.69199999999999995,"15min":0.73299999999999998},"min":{"1min":0.57799999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92600000000000005,"5min":0.92800000000000005,"15min":1.669},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.69499999999999995,"15min":0.747},"min":{"1min":0.61199999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.98599999999999999,"5min":1.167,"15min":1.7589999999999999},"last":0.63}]},{"osd":3,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.69999999999999996,"15min":0.749},"min":{"1min":0.67900000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91900000000000004,"5min":0.997,"15min":2.52},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.78500000000000003},"min":{"1min":0.55600000000000005,"5min":0.51200000000000001,"15min":0.47799999999999998},"max":{"1min":1.0249999999999999,"5min":1.1160000000000001,"15min":1.877},"last":0.71199999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.439,"5min":0.376,"15min":0.376},"max":{"1min":0.87,"5min":0.93799999999999994,"15min":2.4100000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.70599999999999996,"15min":0.71099999999999997},"min":{"1min":0.51700000000000002,"5min":0.46000000000000002,"15min":0.45900000000000002},"max":{"1min":0.93200000000000005,"5min":1.2709999999999999,"15min":2.6629999999999998},"last":0.66000000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.438,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72899999999999998,"5min":1.0900000000000001,"15min":1.5960000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88500000000000001,"5min":1.1180000000000001,"15min":1.925},"last":0.50800000000000001}]},{"osd":7,"last update":"Fri Mar 29 11:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.76400000000000001,"15min":0.76900000000000002},"min":{"1min":0.58499999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88300000000000001,"5min":1.056,"15min":2.7050000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71699999999999997,"15min":0.73899999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81399999999999995,"5min":1.1220000000000001,"15min":1.726},"last":0.64100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48656,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323184,"statfs":{"total":96636764160,"available":96586940416,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:36:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.44800000000000001,"15min":0.45800000000000002},"min":{"1min":0.29399999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.623,"5min":0.71099999999999997,"15min":0.80300000000000005},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.48299999999999998,"15min":0.52200000000000002},"min":{"1min":0.33200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.67400000000000004,"5min":0.79000000000000004,"15min":0.89000000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Fri Mar 29 11:36:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.52600000000000002,"15min":0.55200000000000005},"min":{"1min":0.42599999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.92600000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.502,"15min":0.52100000000000002},"min":{"1min":0.47799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.65300000000000002,"5min":0.751,"15min":0.90300000000000002},"last":0.57299999999999995}]},{"osd":3,"last update":"Fri Mar 29 11:36:19 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56799999999999995,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.38100000000000001,"15min":0.36299999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.995},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.53200000000000003,"15min":0.56699999999999995},"min":{"1min":0.36699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76100000000000001,"5min":0.77700000000000002,"15min":0.83999999999999997},"last":0.626}]},{"osd":4,"last update":"Fri Mar 29 11:36:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.752,"15min":0.77200000000000002},"min":{"1min":0.55800000000000005,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0580000000000001,"5min":1.0589999999999999,"15min":1.153},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.76900000000000002,"15min":0.78700000000000003},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":1.0660000000000001,"5min":1.097,"15min":1.304},"last":0.752}]},{"osd":5,"last update":"Fri Mar 29 11:36:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.78200000000000003,"15min":0.80200000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.48199999999999998},"max":{"1min":1.0429999999999999,"5min":1.1359999999999999,"15min":1.3380000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.79000000000000004,"15min":0.81599999999999995},"min":{"1min":0.70899999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1719999999999999,"5min":1.2030000000000001,"15min":1.3520000000000001},"last":0.79100000000000004}]},{"osd":6,"last update":"Fri Mar 29 11:36:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.78700000000000003,"15min":0.80500000000000005},"min":{"1min":0.63300000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.081,"5min":1.103,"15min":1.1890000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81000000000000005,"15min":0.83299999999999996},"min":{"1min":0.60099999999999998,"5min":0.501,"15min":0.501},"max":{"1min":1.087,"5min":1.2170000000000001,"15min":1.266},"last":0.86299999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:36:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81899999999999995,"15min":0.84999999999999998},"min":{"1min":0.623,"5min":0.54800000000000004,"15min":0.54500000000000004},"max":{"1min":1.0780000000000001,"5min":1.129,"15min":1.327},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.82599999999999996,"15min":0.85099999999999998},"min":{"1min":0.70399999999999996,"5min":0.57699999999999996,"15min":0.57499999999999996},"max":{"1min":1.119,"5min":1.159,"15min":1.2430000000000001},"last":0.85399999999999998}]}]},{"osd":0,"up_from":13,"seq":55834576049,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65608,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306232,"statfs":{"total":96636764160,"available":96569581568,"internally_reserved":0,"allocated":4259840,"data_stored":2398709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.46300000000000002},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.23000000000000001},"max":{"1min":0.65500000000000003,"5min":0.83999999999999997,"15min":1.087},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.499,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70399999999999996,"5min":0.77400000000000002,"15min":1.3089999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.53400000000000003,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.754,"5min":0.84999999999999998,"15min":1.5289999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47599999999999998,"15min":0.51500000000000001},"min":{"1min":0.372,"5min":0.308,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":1.6140000000000001},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.52500000000000002,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78300000000000003,"5min":0.81000000000000005,"15min":1.222},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59299999999999997,"15min":0.63500000000000001},"min":{"1min":0.49199999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.876,"5min":0.876,"15min":1.3899999999999999},"last":0.71399999999999997}]},{"osd":4,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.76700000000000002,"15min":0.80300000000000005},"min":{"1min":0.70499999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0800000000000001,"5min":1.248,"15min":1.248},"last":1.111},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.79400000000000004,"15min":0.83799999999999997},"min":{"1min":0.626,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.5820000000000001},"last":1.026}]},{"osd":5,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.72999999999999998,"15min":0.77300000000000002},"min":{"1min":0.63600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.3440000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.77700000000000002,"15min":0.81399999999999995},"min":{"1min":0.749,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.113,"5min":1.1499999999999999,"15min":1.3640000000000001},"last":1.046}]},{"osd":6,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.78200000000000003,"15min":0.84199999999999997},"min":{"1min":0.60699999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.6240000000000001},"last":1.091},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.82299999999999995,"15min":0.871},"min":{"1min":0.751,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.337},"last":0.997}]},{"osd":7,"last update":"Fri Mar 29 11:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.78100000000000003,"15min":0.81599999999999995},"min":{"1min":0.71099999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.214},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.81000000000000005,"15min":0.86099999999999999},"min":{"1min":0.76300000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.365},"last":0.95199999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52752,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319088,"statfs":{"total":96636764160,"available":96582746112,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:36:34 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":0.80200000000000005,"15min":0.58699999999999997},"min":{"1min":0.35899999999999999,"5min":0.29199999999999998,"15min":0.27800000000000002},"max":{"1min":32.890000000000001,"5min":32.890000000000001,"15min":32.890000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.524,"5min":0.68999999999999995,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.316,"15min":0.316},"max":{"1min":19.241,"5min":19.241,"15min":19.241},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:36:34 2024","interfaces":[{"interface":"back","average":{"1min":1.861,"5min":0.73399999999999999,"15min":0.56799999999999995},"min":{"1min":0.38400000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":26.308,"5min":26.308,"15min":26.308},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.575,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.432,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":19.274999999999999,"5min":19.274999999999999,"15min":19.274999999999999},"last":0.63800000000000001}]},{"osd":3,"last update":"Fri Mar 29 11:36:34 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":0.82799999999999996,"15min":0.66900000000000004},"min":{"1min":0.40899999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":26.259,"5min":26.259,"15min":26.259},"last":0.755},{"interface":"front","average":{"1min":2.278,"5min":0.88400000000000001,"15min":0.68000000000000005},"min":{"1min":0.437,"5min":0.307,"15min":0.307},"max":{"1min":32.496000000000002,"5min":32.496000000000002,"15min":32.496000000000002},"last":0.66000000000000003}]},{"osd":4,"last update":"Fri Mar 29 11:36:34 2024","interfaces":[{"interface":"back","average":{"1min":2.5339999999999998,"5min":1.107,"15min":0.90600000000000003},"min":{"1min":0.64600000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":32.715000000000003,"5min":32.715000000000003,"15min":32.715000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.7549999999999999,"5min":0.93600000000000005,"15min":0.84899999999999998},"min":{"1min":0.59399999999999997,"5min":0.498,"15min":0.498},"max":{"1min":19.120999999999999,"5min":19.120999999999999,"15min":19.120999999999999},"last":0.90400000000000003}]},{"osd":5,"last update":"Fri Mar 29 11:36:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":0.92400000000000004,"15min":0.85199999999999998},"min":{"1min":0.67600000000000005,"5min":0.505,"15min":0.505},"max":{"1min":15.661,"5min":15.661,"15min":15.661},"last":0.84799999999999998},{"interface":"front","average":{"1min":2.5150000000000001,"5min":1.1100000000000001,"15min":0.92000000000000004},"min":{"1min":0.624,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":32.677,"5min":32.677,"15min":32.677},"last":0.97299999999999998}]},{"osd":6,"last update":"Fri Mar 29 11:36:34 2024","interfaces":[{"interface":"back","average":{"1min":1.8400000000000001,"5min":0.99299999999999999,"15min":0.89400000000000002},"min":{"1min":0.64100000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":19.161000000000001,"5min":19.161000000000001,"15min":19.161000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.482,"5min":0.91600000000000004,"15min":0.86499999999999999},"min":{"1min":0.66800000000000004,"5min":0.5,"15min":0.5},"max":{"1min":13.186999999999999,"5min":13.186999999999999,"15min":13.186999999999999},"last":0.95999999999999996}]},{"osd":7,"last update":"Fri Mar 29 11:36:34 2024","interfaces":[{"interface":"back","average":{"1min":1.867,"5min":1.0369999999999999,"15min":0.93999999999999995},"min":{"1min":0.67200000000000004,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":19.202999999999999,"5min":19.202999999999999,"15min":19.202999999999999},"last":0.876},{"interface":"front","average":{"1min":1.677,"5min":1.0009999999999999,"15min":0.93400000000000005},"min":{"1min":0.61799999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":15.726000000000001,"5min":15.726000000000001,"15min":15.726000000000001},"last":0.88500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48652,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323188,"statfs":{"total":96636764160,"available":96586944512,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.34699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.65200000000000002,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51800000000000002,"15min":0.54000000000000004},"min":{"1min":0.36399999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.57699999999999996}]},{"osd":1,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48599999999999999,"15min":0.49099999999999999},"min":{"1min":0.39300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80200000000000005,"5min":1.821,"15min":1.821},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.56299999999999994},"min":{"1min":0.48599999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94599999999999995,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.47999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.627,"15min":0.65000000000000002},"min":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81499999999999995,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.68500000000000005}]},{"osd":4,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.753,"15min":0.78100000000000003},"min":{"1min":0.66200000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0609999999999999,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77100000000000002,"15min":0.80000000000000004},"min":{"1min":0.61199999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0820000000000001,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.81399999999999995}]},{"osd":5,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77800000000000002,"15min":0.79900000000000004},"min":{"1min":0.629,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.089,"5min":2.157,"15min":2.157},"last":1.097},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1699999999999999,"5min":2.266,"15min":2.266},"last":0.76300000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.82199999999999995,"15min":0.84699999999999998},"min":{"1min":0.68200000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.121,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":1.081},{"interface":"front","average":{"1min":0.872,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.66300000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.137,"5min":2.198,"15min":2.198},"last":1.0489999999999999}]},{"osd":7,"last update":"Fri Mar 29 11:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.83599999999999997,"15min":0.84999999999999998},"min":{"1min":0.68100000000000005,"5min":0.53600000000000003,"15min":0.53300000000000003},"max":{"1min":1.111,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.85999999999999999,"15min":0.879},"min":{"1min":0.64400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.153,"5min":2.444,"15min":2.444},"last":1.0369999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96976,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274864,"statfs":{"total":96636764160,"available":96537460736,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:36:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.69499999999999995},"min":{"1min":0.38100000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0309999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.69499999999999995},"min":{"1min":0.45800000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89700000000000002,"5min":1.02,"15min":1.02},"last":0.66300000000000003}]},{"osd":1,"last update":"Fri Mar 29 11:36:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.67400000000000004,"15min":0.71199999999999997},"min":{"1min":0.49399999999999999,"5min":0.436,"15min":0.432},"max":{"1min":1.046,"5min":1.135,"15min":1.135},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69699999999999995,"15min":0.72199999999999998},"min":{"1min":0.54600000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:36:36 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77500000000000002,"15min":0.80300000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.1439999999999999,"5min":1.8,"15min":1.8},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.748,"5min":0.74399999999999999,"15min":0.78500000000000003},"min":{"1min":0.56499999999999995,"5min":0.503,"15min":0.496},"max":{"1min":0.97999999999999998,"5min":1.0369999999999999,"15min":1.224},"last":0.77200000000000002}]},{"osd":3,"last update":"Fri Mar 29 11:36:36 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.70499999999999996,"15min":0.73399999999999999},"min":{"1min":0.55100000000000005,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.996,"5min":0.996,"15min":1.0580000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72399999999999998,"15min":0.754},"min":{"1min":0.51000000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.089,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.84299999999999997}]},{"osd":5,"last update":"Fri Mar 29 11:36:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63900000000000001,"15min":0.63100000000000001},"min":{"1min":0.41299999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79800000000000004,"5min":1.032,"15min":1.032},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.52500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84899999999999998,"5min":1.01,"15min":1.0920000000000001},"last":0.55800000000000005}]},{"osd":6,"last update":"Fri Mar 29 11:36:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.69999999999999996,"15min":0.67500000000000004},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40899999999999997},"max":{"1min":0.88400000000000001,"5min":1.722,"15min":1.722},"last":0.753},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.72799999999999998,"15min":0.71899999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.86499999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.72299999999999998}]},{"osd":7,"last update":"Fri Mar 29 11:36:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71699999999999997,"15min":0.72099999999999997},"min":{"1min":0.501,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91500000000000004,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81299999999999994,"15min":0.81599999999999995},"min":{"1min":0.63900000000000001,"5min":0.57499999999999996,"15min":0.54400000000000004},"max":{"1min":1.0329999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.877}]}]},{"osd":5,"up_from":13,"seq":55834576048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48656,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323184,"statfs":{"total":96636764160,"available":96586940416,"internally_reserved":0,"allocated":3670016,"data_stored":1809360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.63400000000000001,"15min":0.68400000000000005},"min":{"1min":0.52200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.66200000000000003,"15min":0.71199999999999997},"min":{"1min":0.54700000000000004,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0680000000000001},"last":0.61099999999999999}]},{"osd":1,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.70199999999999996,"15min":0.746},"min":{"1min":0.58699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0980000000000001,"5min":1.839,"15min":1.839},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.69699999999999995,"15min":0.72399999999999998},"min":{"1min":0.499,"5min":0.438,"15min":0.438},"max":{"1min":0.94599999999999995,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.50800000000000001}]},{"osd":2,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.70799999999999996,"15min":0.74299999999999999},"min":{"1min":0.60599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69799999999999995,"15min":0.73499999999999999},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97599999999999998,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.58799999999999997}]},{"osd":3,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.75,"15min":0.76400000000000001},"min":{"1min":0.70099999999999996,"5min":0.443,"15min":0.443},"max":{"1min":0.94999999999999996,"5min":3.444,"15min":3.444},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.72299999999999998,"15min":0.76400000000000001},"min":{"1min":0.63800000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.996,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.72699999999999998}]},{"osd":4,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.439,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84799999999999998,"5min":0.96999999999999997,"15min":1.0589999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.95699999999999996,"5min":0.999,"15min":1.0660000000000001},"last":0.75600000000000001}]},{"osd":6,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.67500000000000004},"min":{"1min":0.496,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92300000000000004,"5min":1.214,"15min":1.214},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.69799999999999995,"15min":0.70399999999999996},"min":{"1min":0.58099999999999996,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":1.0900000000000001,"15min":1.107},"last":0.67100000000000004}]},{"osd":7,"last update":"Fri Mar 29 11:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71899999999999997,"15min":0.73199999999999998},"min":{"1min":0.48899999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.89000000000000001,"5min":1.04,"15min":1.2010000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.55600000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89100000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-29T11:36:40.218 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-29T11:36:40.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.412+0000 7f0517a00700 1 Processor -- start 2024-03-29T11:36:40.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.412+0000 7f0517a00700 1 -- start start 2024-03-29T11:36:40.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.413+0000 7f0517a00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f051015dad0 0x7f05100ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.414+0000 7f0517a00700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051015f820 con 0x7f05100d21d0 2024-03-29T11:36:40.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.414+0000 7f0517a00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05100cf420 con 0x7f05100d1630 2024-03-29T11:36:40.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.414+0000 7f0517a00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05100cfcc0 con 0x7f051015dad0 2024-03-29T11:36:40.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.414+0000 7f0515000700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f051015dad0 0x7f05100ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:40.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.414+0000 7f0515a00700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f05100d21d0 0x7f051015fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60118/0 (socket says 172.21.15.12:60118) 2024-03-29T11:36:40.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.414+0000 7f0515a00700 1 -- 172.21.15.12:0/3258421965 learned_addr learned my addr 172.21.15.12:0/3258421965 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-03-29T11:36:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.414+0000 7f0515000700 1 -- 172.21.15.12:0/3258421965 >> v1:172.21.15.159:6789/0 conn(0x7f05100d1630 legacy=0x7f05100d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.414+0000 7f0515000700 1 -- 172.21.15.12:0/3258421965 >> v1:172.21.15.12:6789/0 conn(0x7f05100d21d0 legacy=0x7f051015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.414+0000 7f0515000700 1 -- 172.21.15.12:0/3258421965 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05100cf700 con 0x7f051015dad0 2024-03-29T11:36:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.414+0000 7f050f400700 1 -- 172.21.15.12:0/3258421965 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699193887 0 0) 0x7f051015f820 con 0x7f05100d21d0 2024-03-29T11:36:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.415+0000 7f0515000700 1 --2- 172.21.15.12:0/3258421965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f051015dad0 0x7f05100ced80 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f0504009840 tx=0x7f0504009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc52a808bc0c807 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.415+0000 7f050f400700 1 -- 172.21.15.12:0/3258421965 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504011830 con 0x7f051015dad0 2024-03-29T11:36:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.415+0000 7f050f400700 1 -- 172.21.15.12:0/3258421965 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0504011990 con 0x7f051015dad0 2024-03-29T11:36:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.415+0000 7f050f400700 1 -- 172.21.15.12:0/3258421965 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504011d20 con 0x7f051015dad0 2024-03-29T11:36:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.415+0000 7f0517a00700 1 -- 172.21.15.12:0/3258421965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f051015dad0 msgr2=0x7f05100ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.415+0000 7f0517a00700 1 --2- 172.21.15.12:0/3258421965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f051015dad0 0x7f05100ced80 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f0504009840 tx=0x7f0504009b50 comp rx=0 tx=0).stop 2024-03-29T11:36:40.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.415+0000 7f0517a00700 1 -- 172.21.15.12:0/3258421965 shutdown_connections 2024-03-29T11:36:40.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.415+0000 7f0517a00700 1 --2- 172.21.15.12:0/3258421965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f051015dad0 0x7f05100ced80 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.415+0000 7f0517a00700 1 -- 172.21.15.12:0/3258421965 >> 172.21.15.12:0/3258421965 conn(0x7f051008aff0 msgr2=0x7f05100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:40.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.415+0000 7f0517a00700 1 -- 172.21.15.12:0/3258421965 shutdown_connections 2024-03-29T11:36:40.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.416+0000 7f0517a00700 1 -- 172.21.15.12:0/3258421965 wait complete. 2024-03-29T11:36:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.418+0000 7f0517a00700 1 Processor -- start 2024-03-29T11:36:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.418+0000 7f0517a00700 1 -- start start 2024-03-29T11:36:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.419+0000 7f0517a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05100d1630 0x7f05100c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.419+0000 7f0517a00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05100d21d0 0x7f051009d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.419+0000 7f0517a00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f051015dad0 0x7f051009d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.419+0000 7f0517a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05100b8ce0 con 0x7f05100d1630 2024-03-29T11:36:40.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.419+0000 7f0517a00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05100b8440 con 0x7f05100d21d0 2024-03-29T11:36:40.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.419+0000 7f0517a00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05100b8720 con 0x7f051015dad0 2024-03-29T11:36:40.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.419+0000 7f0515000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05100d1630 0x7f05100c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:40.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f0515000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05100d1630 0x7f05100c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59322/0 (socket says 172.21.15.12:59322) 2024-03-29T11:36:40.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f0515000700 1 -- 172.21.15.12:0/760303807 learned_addr learned my addr 172.21.15.12:0/760303807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:40.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f0515000700 1 -- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f051015dad0 msgr2=0x7f051009d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f0515000700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f051015dad0 0x7f051009d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f0515000700 1 -- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05100d21d0 msgr2=0x7f051009d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f050fe00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05100d21d0 0x7f051009d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f0515000700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05100d21d0 0x7f051009d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f0515000700 1 -- 172.21.15.12:0/760303807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05040094a0 con 0x7f05100d1630 2024-03-29T11:36:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f050fe00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05100d21d0 0x7f051009d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-29T11:36:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f0515000700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05100d1630 0x7f05100c9d10 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f0504009b90 tx=0x7f0504004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f050cc00700 1 -- 172.21.15.12:0/760303807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05040036a0 con 0x7f05100d1630 2024-03-29T11:36:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f050cc00700 1 -- 172.21.15.12:0/760303807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050401e070 con 0x7f05100d1630 2024-03-29T11:36:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f050cc00700 1 -- 172.21.15.12:0/760303807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504003a00 con 0x7f05100d1630 2024-03-29T11:36:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.420+0000 7f0517a00700 1 -- 172.21.15.12:0/760303807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05100b8a00 con 0x7f05100d1630 2024-03-29T11:36:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.421+0000 7f0517a00700 1 -- 172.21.15.12:0/760303807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05100ca250 con 0x7f05100d1630 2024-03-29T11:36:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.422+0000 7f0517a00700 1 -- 172.21.15.12:0/760303807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05100c7ea0 con 0x7f05100d1630 2024-03-29T11:36:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.423+0000 7f050cc00700 1 -- 172.21.15.12:0/760303807 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0504003b60 con 0x7f05100d1630 2024-03-29T11:36:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.424+0000 7f050cc00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04f006adb0 0x7f04f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.424+0000 7f050cc00700 1 -- 172.21.15.12:0/760303807 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f050408a6a0 con 0x7f05100d1630 2024-03-29T11:36:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.424+0000 7f050fe00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04f006adb0 0x7f04f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:40.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.425+0000 7f050fe00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04f006adb0 0x7f04f006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f05101f61a0 tx=0x7f04fc009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:40.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.429+0000 7f050cc00700 1 -- 172.21.15.12:0/760303807 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0504055730 con 0x7f05100d1630 2024-03-29T11:36:40.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.639+0000 7f0517a00700 1 -- 172.21.15.12:0/760303807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f0510067e80 con 0x7f05100d1630 2024-03-29T11:36:40.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.644+0000 7f050cc00700 1 -- 172.21.15.12:0/760303807 <== mon.0 v2:172.21.15.12:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f050405f070 con 0x7f05100d1630 2024-03-29T11:36:40.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.644+0000 7f050cc00700 1 -- 172.21.15.12:0/760303807 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v100)=0 v100) v1 ==== 123+0+0 (secure 0 0 0) 0x7f0504058d50 con 0x7f05100d1630 2024-03-29T11:36:40.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 -- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04f006adb0 msgr2=0x7f04f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04f006adb0 0x7f04f006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f05101f61a0 tx=0x7f04fc009040 comp rx=0 tx=0).stop 2024-03-29T11:36:40.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 -- 172.21.15.12:0/760303807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05100d1630 msgr2=0x7f05100c9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05100d1630 0x7f05100c9d10 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f0504009b90 tx=0x7f0504004750 comp rx=0 tx=0).stop 2024-03-29T11:36:40.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 -- 172.21.15.12:0/760303807 shutdown_connections 2024-03-29T11:36:40.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7f04f006adb0 0x7f04f006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f051015dad0 0x7f051009d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05100d1630 0x7f05100c9d10 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 --2- 172.21.15.12:0/760303807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f05100d21d0 0x7f051009d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 -- 172.21.15.12:0/760303807 >> 172.21.15.12:0/760303807 conn(0x7f051008aff0 msgr2=0x7f051015b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:40.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.647+0000 7f0517a00700 1 -- 172.21.15.12:0/760303807 shutdown_connections 2024-03-29T11:36:40.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.648+0000 7f0517a00700 1 -- 172.21.15.12:0/760303807 wait complete. 2024-03-29T11:36:40.665 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-29T11:36:40.666 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-29T11:36:40.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.868+0000 7fd872e00700 1 Processor -- start 2024-03-29T11:36:40.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.868+0000 7fd872e00700 1 -- start start 2024-03-29T11:36:40.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.869+0000 7fd872e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 0x7fd86c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.869+0000 7fd872e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd86c161800 0x7fd86c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.869+0000 7fd872e00700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86c0d30d0 con 0x7fd86c15d640 2024-03-29T11:36:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.869+0000 7fd872e00700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c161c90 con 0x7fd86c161800 2024-03-29T11:36:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.869+0000 7fd872e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0b6840 con 0x7fd86c0ceec0 2024-03-29T11:36:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.870+0000 7fd870e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd86c161800 0x7fd86c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.870+0000 7fd870e00700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd86c161800 0x7fd86c0b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59338/0 (socket says 172.21.15.12:59338) 2024-03-29T11:36:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.870+0000 7fd870e00700 1 -- 172.21.15.12:0/3684660662 learned_addr learned my addr 172.21.15.12:0/3684660662 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.870+0000 7fd86b400700 1 --2- 172.21.15.12:0/3684660662 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 0x7fd86c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.870+0000 7fd870e00700 1 -- 172.21.15.12:0/3684660662 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 msgr2=0x7fd86c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.870+0000 7fd870e00700 1 --2- 172.21.15.12:0/3684660662 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 0x7fd86c1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.870+0000 7fd870e00700 1 -- 172.21.15.12:0/3684660662 >> v1:172.21.15.159:6789/0 conn(0x7fd86c15d640 legacy=0x7fd86c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.870+0000 7fd870e00700 1 -- 172.21.15.12:0/3684660662 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c0d2b30 con 0x7fd86c161800 2024-03-29T11:36:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.870+0000 7fd86b400700 1 --2- 172.21.15.12:0/3684660662 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 0x7fd86c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-29T11:36:40.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.870+0000 7fd870e00700 1 --2- 172.21.15.12:0/3684660662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 0x7fd86c0b6cd0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7fd864009e50 tx=0x7fd86400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90c2b019789f41e server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:40.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd86aa00700 1 -- 172.21.15.12:0/3684660662 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86400e070 con 0x7fd86c161800 2024-03-29T11:36:40.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd86aa00700 1 -- 172.21.15.12:0/3684660662 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd86400b9c0 con 0x7fd86c161800 2024-03-29T11:36:40.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd86aa00700 1 -- 172.21.15.12:0/3684660662 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86400bd90 con 0x7fd86c161800 2024-03-29T11:36:40.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd872e00700 1 -- 172.21.15.12:0/3684660662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 msgr2=0x7fd86c0b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd872e00700 1 --2- 172.21.15.12:0/3684660662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 0x7fd86c0b6cd0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7fd864009e50 tx=0x7fd86400b330 comp rx=0 tx=0).stop 2024-03-29T11:36:40.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd872e00700 1 -- 172.21.15.12:0/3684660662 shutdown_connections 2024-03-29T11:36:40.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd872e00700 1 --2- 172.21.15.12:0/3684660662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 0x7fd86c0b6cd0 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd872e00700 1 --2- 172.21.15.12:0/3684660662 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 0x7fd86c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd872e00700 1 -- 172.21.15.12:0/3684660662 >> 172.21.15.12:0/3684660662 conn(0x7fd86c08aff0 msgr2=0x7fd86c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:40.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd872e00700 1 -- 172.21.15.12:0/3684660662 shutdown_connections 2024-03-29T11:36:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.871+0000 7fd872e00700 1 -- 172.21.15.12:0/3684660662 wait complete. 2024-03-29T11:36:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.873+0000 7fd872e00700 1 Processor -- start 2024-03-29T11:36:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.873+0000 7fd872e00700 1 -- start start 2024-03-29T11:36:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.874+0000 7fd872e00700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 0x7fd86c09ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.874+0000 7fd872e00700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd86c15d640 0x7fd86c15b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.874+0000 7fd872e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 0x7fd86c15bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.874+0000 7fd872e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c09f4a0 con 0x7fd86c161800 2024-03-29T11:36:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.874+0000 7fd872e00700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c15b1b0 con 0x7fd86c15d640 2024-03-29T11:36:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.874+0000 7fd872e00700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c15a5a0 con 0x7fd86c0ceec0 2024-03-29T11:36:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.874+0000 7fd870e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 0x7fd86c15bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.874+0000 7fd870e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 0x7fd86c15bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59352/0 (socket says 172.21.15.12:59352) 2024-03-29T11:36:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.874+0000 7fd870e00700 1 -- 172.21.15.12:0/2531161199 learned_addr learned my addr 172.21.15.12:0/2531161199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-29T11:36:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.874+0000 7fd86be00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 0x7fd86c09ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.875+0000 7fd86b400700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd86c15d640 0x7fd86c15b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.875+0000 7fd870e00700 1 -- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 msgr2=0x7fd86c09ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.875+0000 7fd870e00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 0x7fd86c09ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.875+0000 7fd870e00700 1 -- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd86c15d640 msgr2=0x7fd86c15b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.875+0000 7fd870e00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd86c15d640 0x7fd86c15b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.875+0000 7fd870e00700 1 -- 172.21.15.12:0/2531161199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd864009b30 con 0x7fd86c161800 2024-03-29T11:36:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.875+0000 7fd870e00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 0x7fd86c15bdc0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7fd86c08c490 tx=0x7fd864004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.875+0000 7fd863e00700 1 -- 172.21.15.12:0/2531161199 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8640092e0 con 0x7fd86c161800 2024-03-29T11:36:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.876+0000 7fd863e00700 1 -- 172.21.15.12:0/2531161199 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd864004250 con 0x7fd86c161800 2024-03-29T11:36:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.876+0000 7fd863e00700 1 -- 172.21.15.12:0/2531161199 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864004620 con 0x7fd86c161800 2024-03-29T11:36:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.876+0000 7fd872e00700 1 -- 172.21.15.12:0/2531161199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c15a880 con 0x7fd86c161800 2024-03-29T11:36:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.876+0000 7fd872e00700 1 -- 172.21.15.12:0/2531161199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c15ae40 con 0x7fd86c161800 2024-03-29T11:36:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.877+0000 7fd872e00700 1 -- 172.21.15.12:0/2531161199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c15b000 con 0x7fd86c161800 2024-03-29T11:36:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.878+0000 7fd863e00700 1 -- 172.21.15.12:0/2531161199 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd864004780 con 0x7fd86c161800 2024-03-29T11:36:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.879+0000 7fd863e00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd84c06adb0 0x7fd84c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-29T11:36:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.879+0000 7fd863e00700 1 -- 172.21.15.12:0/2531161199 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fd86408a840 con 0x7fd86c161800 2024-03-29T11:36:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.879+0000 7fd86be00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd84c06adb0 0x7fd84c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-29T11:36:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.880+0000 7fd86be00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd84c06adb0 0x7fd84c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd85c007fd0 tx=0x7fd85c009040 comp rx=0 tx=0).ready entity=mgr.10910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-29T11:36:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:40.883+0000 7fd863e00700 1 -- 172.21.15.12:0/2531161199 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8640557e0 con 0x7fd86c161800 2024-03-29T11:36:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.147+0000 7fd872e00700 1 -- 172.21.15.12:0/2531161199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd86c15b320 con 0x7fd86c161800 2024-03-29T11:36:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.147+0000 7fd863e00700 1 -- 172.21.15.12:0/2531161199 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+3535 (secure 0 0 0) 0x7fd86401f090 con 0x7fd86c161800 2024-03-29T11:36:41.147 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-29T11:36:41.147 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":148,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12538,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/350949303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":350949303},{"type":"v1","addr":"172.21.15.159:6836","nonce":350949303}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1576953466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1576953466},{"type":"v1","addr":"172.21.15.159:6837","nonce":1576953466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":12547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/30523791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":30523791},{"type":"v1","addr":"172.21.15.12:6835","nonce":30523791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":13539,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/1818678675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1818678675},{"type":"v1","addr":"172.21.15.12:6834","nonce":1818678675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":148}],"filesystems":[]} 2024-03-29T11:36:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 -- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd84c06adb0 msgr2=0x7fd84c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd84c06adb0 0x7fd84c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd85c007fd0 tx=0x7fd85c009040 comp rx=0 tx=0).stop 2024-03-29T11:36:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 -- 172.21.15.12:0/2531161199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 msgr2=0x7fd86c15bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 0x7fd86c15bdc0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7fd86c08c490 tx=0x7fd864004ad0 comp rx=0 tx=0).stop 2024-03-29T11:36:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 -- 172.21.15.12:0/2531161199 shutdown_connections 2024-03-29T11:36:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:6832/62646,v1:172.21.15.159:6833/62646] conn(0x7fd84c06adb0 0x7fd84c06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c161800 0x7fd86c15bdc0 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd86c0ceec0 0x7fd86c09ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 --2- 172.21.15.12:0/2531161199 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd86c15d640 0x7fd86c15b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-29T11:36:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 -- 172.21.15.12:0/2531161199 >> 172.21.15.12:0/2531161199 conn(0x7fd86c08aff0 msgr2=0x7fd86c0b83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 -- 172.21.15.12:0/2531161199 shutdown_connections 2024-03-29T11:36:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-29T11:36:41.150+0000 7fd872e00700 1 -- 172.21.15.12:0/2531161199 wait complete. 2024-03-29T11:36:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 148 2024-03-29T11:36:41.167 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-29T11:36:41.168 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-29T11:36:41.168 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-29T11:36:41.168 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.181 INFO:tasks.ceph.mds.a:Stopped 2024-03-29T11:36:41.181 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-29T11:36:41.181 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.236 INFO:tasks.ceph.mds.c:Stopped 2024-03-29T11:36:41.236 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-29T11:36:41.237 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.252 INFO:tasks.ceph.mds.b:Stopped 2024-03-29T11:36:41.253 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-29T11:36:41.253 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.308 INFO:tasks.ceph.mds.d:Stopped 2024-03-29T11:36:41.309 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:36:41.309 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:36:41.309 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.359 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:36:41.359 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:36:41.359 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.415 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:36:41.416 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:36:41.416 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.463 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:36:41.463 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:36:41.464 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.512 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:36:41.512 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:36:41.512 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.568 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:36:41.569 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:36:41.569 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.624 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:36:41.625 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:36:41.625 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.669 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:36:41.670 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:36:41.670 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.732 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:36:41.732 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:36:41.732 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:36:41.732 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.801 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:36:41.801 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:36:41.801 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.873 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:36:41.874 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:36:41.874 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:36:41.874 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.909 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:36:41.910 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:36:41.910 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:41.935 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:36:41.935 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:36:41.935 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:36:42.001 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:36:42.002 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:36:42.002 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-03-29T11:36:42.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-29T11:36:42.092 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:36:42.092 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:36:42.352 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:36:42.352 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:36:42.491 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:36:42.491 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:36:42.597 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:36:42.598 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:36:42.714 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T11:36:42.715 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:36:42.856 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T11:36:42.857 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:36:42.994 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T11:36:43.571 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:36:43.722 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T11:36:43.723 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:36:43.847 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:36:44.479 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/data/mon.a.tgz 2024-03-29T11:36:44.480 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-03-29T11:36:44.529 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.0nWQbwg5C1 2024-03-29T11:36:44.530 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0nWQbwg5C1 2024-03-29T11:36:45.237 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.0nWQbwg5C1 2024-03-29T11:36:45.353 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.0nWQbwg5C1 is 4MB 2024-03-29T11:36:45.577 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.0nWQbwg5C1 2024-03-29T11:36:45.623 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/data/mon.b.tgz 2024-03-29T11:36:45.623 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-03-29T11:36:45.675 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.n7h3t9Ft6W 2024-03-29T11:36:45.675 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.n7h3t9Ft6W 2024-03-29T11:36:46.358 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.n7h3t9Ft6W 2024-03-29T11:36:46.471 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.n7h3t9Ft6W is 4MB 2024-03-29T11:36:46.627 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.n7h3t9Ft6W 2024-03-29T11:36:46.672 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/data/mon.c.tgz 2024-03-29T11:36:46.673 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-03-29T11:36:46.756 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.R6Zul8vAcJ 2024-03-29T11:36:46.757 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.R6Zul8vAcJ 2024-03-29T11:36:47.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-29T11:36:47.424 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.R6Zul8vAcJ 2024-03-29T11:36:47.535 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.R6Zul8vAcJ is 4MB 2024-03-29T11:36:47.688 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.R6Zul8vAcJ 2024-03-29T11:36:47.734 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:36:47.734 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:36:47.738 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:36:47.840 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-03-29T11:36:47.840 DEBUG:teuthology.orchestra.run.smithi012:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-29T11:36:47.844 DEBUG:teuthology.orchestra.run.smithi159:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-29T11:36:47.919 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-29T11:36:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-29T11:36:47.941 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:36:47.943 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/remote/smithi012/crash 2024-03-29T11:36:47.943 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:36:48.009 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/remote/smithi159/crash 2024-03-29T11:36:48.010 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:36:48.075 INFO:tasks.ceph:Shutting down logrotate 2024-03-29T11:36:48.075 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:36:48.079 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-29T11:36:48.175 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-29T11:36:48.241 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-29T11:36:48.307 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:36:48.308 DEBUG:teuthology.orchestra.run.smithi012:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:36:48.312 DEBUG:teuthology.orchestra.run.smithi159:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:37:10.156 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:37:10.157 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/remote/smithi012/log 2024-03-29T11:37:10.158 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:37:15.446 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/remote/smithi159/log 2024-03-29T11:37:15.447 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:37:17.680 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:37:17.691 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:37:17.691 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:37:17.696 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:37:17.818 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:37:17.827 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:37:17.827 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:37:17.835 DEBUG:teuthology.orchestra.run.smithi159:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:37:17.871 INFO:teuthology.orchestra.run.smithi012.stderr:bash: ntpq: command not found 2024-03-29T11:37:17.877 INFO:teuthology.orchestra.run.smithi159.stderr:bash: ntpq: command not found 2024-03-29T11:37:17.879 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:37:17.879 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-03-29T11:37:17.879 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 41 +451us[ +411us] +/- 96ms 2024-03-29T11:37:17.880 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -161us[ -201us] +/- 59ms 2024-03-29T11:37:17.880 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -366us[ -366us] +/- 90ms 2024-03-29T11:37:17.880 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:37:17.884 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:37:17.884 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-03-29T11:37:17.885 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 42 +402us[ +402us] +/- 96ms 2024-03-29T11:37:17.885 INFO:teuthology.orchestra.run.smithi159.stdout:^* hv02.front.sepia.ceph.com 3 9 377 120 -111us[ -154us] +/- 58ms 2024-03-29T11:37:17.885 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 -372us[ -372us] +/- 90ms 2024-03-29T11:37:17.885 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:37:17.886 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:37:17.896 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:37:17.896 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:37:17.908 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:37:18.094 DEBUG:teuthology.orchestra.run.smithi159:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:37:18.226 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:37:18.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:18.301 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:37:18.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:18.372 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:37:18.381 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:37:18.391 INFO:teuthology.task.internal:Duration was 7830.078377 seconds 2024-03-29T11:37:18.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:37:18.402 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:37:18.402 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:37:18.407 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:37:18.517 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:37:18.519 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:37:18.804 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:37:18.804 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-03-29T11:37:18.804 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:37:18.864 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi159.front.sepia.ceph.com 2024-03-29T11:37:18.864 DEBUG:teuthology.orchestra.run.smithi159:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:37:18.922 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:37:18.923 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:37:18.927 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:37:19.017 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:37:19.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:37:19.021 DEBUG:teuthology.orchestra.run.smithi159:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:37:19.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:37:19.180 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:37:19.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:37:19.248 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:37:19.316 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:37:19.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:37:19.332 DEBUG:teuthology.orchestra.run.smithi159:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:37:19.394 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-03-29T11:37:19.398 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = core 2024-03-29T11:37:19.431 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:37:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:19.473 DEBUG:teuthology.orchestra.run.smithi159:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:37:19.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:19.516 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:37:19.526 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:37:19.527 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/remote/smithi012 2024-03-29T11:37:19.527 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:37:19.610 DEBUG:teuthology.misc:Transferring archived files from smithi159:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/remote/smithi159 2024-03-29T11:37:19.610 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:37:19.688 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:37:19.688 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:37:19.692 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:37:19.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:37:19.748 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:37:19.748 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:37:19.758 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:37:19.758 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:37:19.776 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:37:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:37 /home/ubuntu/cephtest 2024-03-29T11:37:19.821 INFO:teuthology.orchestra.run.smithi159.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:37 /home/ubuntu/cephtest 2024-03-29T11:37:19.824 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:37:19.836 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:37:19.861 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:37:19.889 DEBUG:teuthology.nuke:shortname: smithi012 2024-03-29T11:37:19.889 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:37:19.914 DEBUG:teuthology.nuke:shortname: smithi159 2024-03-29T11:37:19.914 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:37:19.917 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 09:07:13.369330', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIF8rEDuGqw7u1MAn7aIckiCIMAIgcK8ExXmWCwNekr63BcoLpA25vbhmE2qT/oUqU/zqamOdxeKlOruHPG4l8s='} 2024-03-29T11:37:19.937 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 09:07:13.367518', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFpoTDgkUa4ldKvUIt0bpX7pXwcd199Mcl3Y1HX+RceoaNKFSo7mIUmoYKdUUWIiRlhtPgolkPbAw/7/+jHMc64='} 2024-03-29T11:37:19.969 INFO:teuthology.orchestra.console.smithi012:Power off 2024-03-29T11:37:19.970 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:37:19.994 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2024-03-29T11:37:19.994 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:20.010 INFO:teuthology.orchestra.console.smithi159:Power off 2024-03-29T11:37:20.010 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:37:20.020 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-29T11:37:20.032 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off 2024-03-29T11:37:20.032 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:20.055 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-03-29T11:37:24.022 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:24.056 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:24.132 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-29T11:37:24.167 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-03-29T11:37:28.134 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:28.168 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:28.247 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-29T11:37:28.278 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off 2024-03-29T11:37:28.379 INFO:teuthology.orchestra.console.smithi159:Power off completed 2024-03-29T11:37:32.248 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:32.360 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2024-03-29T11:37:32.461 INFO:teuthology.orchestra.console.smithi012:Power off completed 2024-03-29T11:37:32.569 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 client.1-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} duration: 7830.078377485275 failure_reason: 'Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments)' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a0b9ea7dcb494d7293b452e8c2de435f status: fail success: false 2024-03-29T11:37:32.570 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:37:32.642 INFO:teuthology.run:FAIL